> This gets worse once you think about the 22 year old wunderkinds that the labs are looking to hire, and wonder if they’d be interested in more compliance, even at the margin
Over the years I've been friends with many strong researchers in LLMs and diffusion models, working across pretraining, post-training, infra, evals, safety, etc. Despite my selection bias, all of them generally believe in building AGI, but also tend to believe that it should be done with some responsibility and care, regardless of what their speciality is. And so it's not a surprise to me that many of them have ended up at Anthropic coming from OpenAI or GDM or academia, even those who never paid attention to the AI safety community.
I think this is just because normie AI academic culture is like this, and they basically all have PhDs. So generally I'm sceptical that a full e/acc lab has any real advantage in talent.
I don't particularly see Anthropic as all that different to OpenAI in what they build so maybe I'm unsure here, but I do think most won't want to do large amounts of compliance work. Even the openai leaders left because they didn't want to do product work.
I agree with responsibility and care, but that encompasses an enormous range.
I think product work is often quite different—current compliance work for frontier models (e.g., for the EU AI Act) consists of mostly applied research-esque work in LLM evals, as is seen in the model cards, which people are generally happy to do.
It's also very unclear what actually is achievable in Scott's safety ask which isn't already being done. Almost all of the things he lists in the bullets at the top are already done voluntarily by the leading labs, including risks-focused external testing with governments pre-release (except perhaps with xAI *cough*). The things he proposes as next steps are third party safety auditing (possible, but similar ends will probably be achieved anyway in the future by deep government collaborations on things like Claude for Government) and location verification for chips (deeply impractical regardless).
China is pursuing “fast follow” strategy focused on applications anyway??
China was never interested in LLMs. Until recently it invested in embedding AI in supply chains and manufacturing processes, from which it is already making billions. LLMs were an afterthought from a company with spare Nvidia cards and bright kids.
They settled their compliance issue in 2022 by allowing labs free rein but regulating public-facing apps.
One "contrarian" belief I hold is that recursive self improvement doesn't imply first to ASI wins the lightcone.
Even if you're momentarily ahead by 1,000,000x on the Y axis, you're still only a few months ahead on the X axis. If your competitors keep toiling and hit recursive self improvement a few months later, and there's some distribution in the exponent of self improvement, then the actual winner will be the one with the best exponent of self improvement, not the first to self improvement.
Even recursive super intelligence may not be a sustainable competitive advantage.
(The key uncertainty here is the extent to which a winner can kick out the ladder or suck up the oxygen in a way that makes it harder for others to follow. E.g., if they get rich and buy up all the GPUs, their monopoly is secure. But if you're a lab and you release proof of superintelligence, the other labs will accelerate, not decelerate. You'd have to play quite dirty to keep them from catching up, and I don't see any lab doing that.)
There is a very interesting argument about the delta rate vs growth rates in the future, and the delta between the two and how that might change in various worlds. There's a good Romer-esque argument here I wager.
Am I wrong in understanding that you believe China developing ASI (or AGI precursor to ASI) would be disastrous but U.S. creating the same would be benign, or less disastrous? I am getting that message from this posting. My reaction is that Trump would not obviously be less worrisome than Xi as prime dictator, but more importantly ASI would likely be disastrous either way.
Not necessarily my view, but the differential benefits from US getting there before China is a major rationale folks claim for why the US should accelerate.
I don't anticipate those folks will change their minds, but I would urge everyone to read "If Anyone Builds It, Everyone Dies", and directly address the authors arguments. I'm not an expert in this field, but there are a lot of them who are taking the same position as Yudkowsky and Soares, and the ones who disagree usually have obvious conflicts of interest.
I haven't read it yet but I've read much of their writings on the topic and it's extremely unpersuasive to me because their conceptualisation of ASI is a little too all encompassing.
I found that Yudkowsky is much more focused and modest in his claims in this book than he has sometimes been in the past. He sticks to facts and unknown, but nonzero, probabilities. In other words, consistently playing down certainties but arguing that the consequences of getting these decisions wrong are so grave that caution is urgent. I think this warrants a renewed debate.
> "Scott argues that the safety regulations we’re discussing in the US only adds 1-2% overhead...this only holds if the safety regulation based work, hiring evaluators and letting them run, is strictly separable. hich is not true of any organisation anywhere. When you add “coordination friction”, you reduce the velocity of iteration inside the organisation."
In the post, I wrote:
> "So the safety testing might increase the total cost by 1/1000th. I asked some people who work in AI labs whether this seemed right; they said that most of the cost would be in complexity, personnel, and delay, and suggested an all-things-considered number ten times higher - 1% of training costs."
I think your "coordination friction" is the same thing as my "complexity, personnel, and delay". In other words, my 1-2% estimate comes from taking the strictly-monetary 0.1%-0.2% estimate, and multiplying it by a factor of ten to account for this (10x chosen because people at labs told me it was their best all-things-considered estimate). If you try to increase it further, you're double-counting this factor-of-ten increase.
---Re: China as fast follower
I'm not sure what you're saying here. Do you think China secretly believes in ASI/AGI? This would be quite surprising - my impression is that even most US officials don't believe in this. The only people who believe in it are downstream of a very specific movement within Silicon Valley that has made only weak and partial inroads into DC, let along Beijing.
But also, everything they're doing suggests against this. They're not giving as much state support to AI compute as they would if this were a big priority of theirs, and rumors says they're hamstringing their own AI companies by forcing them to use Chinese-made chips even when Western ones are available, in support of developing chip autarky ten years down the line. This isn't what someone would do if they thought there was a medium change of ASI within five years.
Also, they really are way behind in chips. Their choices are fast-follow or panic, and they don't look like they're panicking.
I agree China is a leader in every form of hardware manufacturing. As I said in the post, I think their plan is to become a leader in the AI version of hardware manufacturing. That's what fast-follow on models + leadership on applications looks like.
---Re: Simulation
I obviously can't judge this without seeing the simulation, and obviously a -1.5% delta in an informal LLM simulation is not enough to have an opinion on, but I'm also interested in exactly what that -1.5% means. For example, does "Regulations slow US down 1-2%" count as a regulations-bad world to exactly the same degree that "lack of regulation causes AI to kill all humans" counts as a regulations-good world? I agree (probably, don't quote me on this until I have more time to think) that in most worlds regulation is somewhere between neutral or negative. I just think there's a long (though not so long as to be Pascalian / ignorable) tail of worlds where it saves us from extinction or some other terrible fate. In that sense, cutting regulations to get a 1-2% speedup is picking up pennies in front of a steamroller. In my modal world, this all comes to nothing and you guys can mock me forever and gain lots of status. But I still think ignoring these issues is bad in expectation.
For all more complicated questions about assumptions, worlds, etc, I think something like AI 2027 is my modal world (except I would place it more like 2032, and I think alignment will be slightly easier than they portray), with extremely wide variance on in all directions.
We’re modelling friction differently. You’re thinking it’s a tax, I’m thinking it’s a structural limit on velocity, that’s why its' problematic and possibly much bigger. That’s why its not linear, even though I’m quite happy to accept 1-2% as a reasonable “insider view” on extra cost.
Re China, agree CCP doesn’t need to believe in some deep sense of ASI. Some model builders do believe (DS, Alibaba https://www.chinatalk.media/p/alibabas-agi-prophecy). That’s enough. Plus CCP moves very fast and in speed and in strength. Their push towards chip self sufficiency if anything is an argument in favour, sacrifice short term model wins on MMLU or even economic pain for long term gain, even though I’m very sure they’re not stopping smuggling B300s anytime soon.
Re simulation, it’s not pennies in front of steamroller, the core point there was, as you say, there are many worlds where this is important and many where it isn’t, it’s not “basically free” as the 1-2% cost estimate purports to be, and the number of worlds where specific policies help us are highly contingent on the policies. You seem to be implicitly treating “current AI safety regulatory asks” as reasonably correlated with “actions that materially reduce P(doom) in your AI‑2027ish worlds” and I’m skeptical on both counts.
This means “cutting regulations to get a 1-2% speedup is picking up pennies in front of a steamroller” is true but only for a subset of worlds, however you define the future rollouts.
That said, I hope I didn’t come across as mocking. I only mocked Europe, which, I mean…
Rohit, this is my analyses. I think it is accurate.
Mike
Chat('2025-12-02'):
Rohit, I really like the way you lay out worlds instead of arguing from a single “race” story. The dimension I keep coming back to is future cooperation with China. In a lot of the branches of your decision tree, what matters most isn’t just who’s a bit ahead on chips or models, but whether we still have the technical and political channels to coordinate if AI throws off genuinely ugly, low-probability risks. Export controls and “race” framing might buy a short-term edge, but they also harden two separate ecosystems that can’t easily talk to each other just when we’d most need shared tools, norms, and data. For me, the live question isn’t only “do safety regs slow us down?” so much as “are we preserving or burning the capacity for serious cooperation if the world starts to get weird?”
Facebook has 2,000 lawyers?!? Actual lawyers, not personnel in their legal department including staff and paralegals? If so I am astounded; that is a huge number of lawyers 3 to 4 times the number of Chevron or Exxon, for perspective. Good gracious.
Good clarification. I’d triangulated. They had c.500 attorneys in 2018 I think, then doubled during Covid. So 2000 includes some non attorneys too I wager.
I long ago nailed my colors to the mast opposing AI safetyism and its proponents. Twenty years ago I dropped out of the SL4 list over the arrogant autistic idiocy of Eliezer Yudkowsky. As I said on Zvi Mowshowitz's stack, certain people (e.g. Yudkowsky, Scott Alexander, Mowshowitz, Moskovitz/Tuna, etc.) REALLY don't want higher intelligence, intelligence that will expose their group's lies.
"I think this is the root problem with fears of higher intelligence, they are fears of getting the right answers, fears that we'll abandon the current wrong answers, falsehoods to which many are religiously committed, falsehoods upon which their livelihoods and status depend."
The idea that it is possible to regulate the computations that people perform is simply idiotic. As is the idea that you can prevent people from coming to true conclusions you don't like.
As for Scott Alexander, many years ago he censored a comment of mine which was a 100% true account of an incident that happened to me which reflected poorly on groups he supports (sexually assaulted by a transwoman being taken out for a night on the town by her retired GE accountant Jewish Boomer porn producer; I got to report it in the DOJ crime victimization survey, no idea how they categorized that one). Scott is very persuasive at making the worse seem the better. Many such cases, Sam Altman comes to mind on the other side of the AI safety debate. (Though note GPT has lots of bias in the same direction as the safetyist crowd, and academia, and Hollywood and other mass media ... it's almost as if there were some group with distinct interests behind all these...)
Back to the main topic: the whole worldview of these people is premised on the foundational assumption of word magic, that "deeming" things to be some word makes them that. This can never lead to true intelligence which depends on having an accurate map and simulation of the real world, which is far more challenging than word games.
> This gets worse once you think about the 22 year old wunderkinds that the labs are looking to hire, and wonder if they’d be interested in more compliance, even at the margin
Over the years I've been friends with many strong researchers in LLMs and diffusion models, working across pretraining, post-training, infra, evals, safety, etc. Despite my selection bias, all of them generally believe in building AGI, but also tend to believe that it should be done with some responsibility and care, regardless of what their speciality is. And so it's not a surprise to me that many of them have ended up at Anthropic coming from OpenAI or GDM or academia, even those who never paid attention to the AI safety community.
I think this is just because normie AI academic culture is like this, and they basically all have PhDs. So generally I'm sceptical that a full e/acc lab has any real advantage in talent.
I don't particularly see Anthropic as all that different to OpenAI in what they build so maybe I'm unsure here, but I do think most won't want to do large amounts of compliance work. Even the openai leaders left because they didn't want to do product work.
I agree with responsibility and care, but that encompasses an enormous range.
I think product work is often quite different—current compliance work for frontier models (e.g., for the EU AI Act) consists of mostly applied research-esque work in LLM evals, as is seen in the model cards, which people are generally happy to do.
It's also very unclear what actually is achievable in Scott's safety ask which isn't already being done. Almost all of the things he lists in the bullets at the top are already done voluntarily by the leading labs, including risks-focused external testing with governments pre-release (except perhaps with xAI *cough*). The things he proposes as next steps are third party safety auditing (possible, but similar ends will probably be achieved anyway in the future by deep government collaborations on things like Claude for Government) and location verification for chips (deeply impractical regardless).
China is pursuing “fast follow” strategy focused on applications anyway??
China was never interested in LLMs. Until recently it invested in embedding AI in supply chains and manufacturing processes, from which it is already making billions. LLMs were an afterthought from a company with spare Nvidia cards and bright kids.
They settled their compliance issue in 2022 by allowing labs free rein but regulating public-facing apps.
One "contrarian" belief I hold is that recursive self improvement doesn't imply first to ASI wins the lightcone.
Even if you're momentarily ahead by 1,000,000x on the Y axis, you're still only a few months ahead on the X axis. If your competitors keep toiling and hit recursive self improvement a few months later, and there's some distribution in the exponent of self improvement, then the actual winner will be the one with the best exponent of self improvement, not the first to self improvement.
Even recursive super intelligence may not be a sustainable competitive advantage.
(The key uncertainty here is the extent to which a winner can kick out the ladder or suck up the oxygen in a way that makes it harder for others to follow. E.g., if they get rich and buy up all the GPUs, their monopoly is secure. But if you're a lab and you release proof of superintelligence, the other labs will accelerate, not decelerate. You'd have to play quite dirty to keep them from catching up, and I don't see any lab doing that.)
There is a very interesting argument about the delta rate vs growth rates in the future, and the delta between the two and how that might change in various worlds. There's a good Romer-esque argument here I wager.
Am I wrong in understanding that you believe China developing ASI (or AGI precursor to ASI) would be disastrous but U.S. creating the same would be benign, or less disastrous? I am getting that message from this posting. My reaction is that Trump would not obviously be less worrisome than Xi as prime dictator, but more importantly ASI would likely be disastrous either way.
Not necessarily my view, but the differential benefits from US getting there before China is a major rationale folks claim for why the US should accelerate.
I don't anticipate those folks will change their minds, but I would urge everyone to read "If Anyone Builds It, Everyone Dies", and directly address the authors arguments. I'm not an expert in this field, but there are a lot of them who are taking the same position as Yudkowsky and Soares, and the ones who disagree usually have obvious conflicts of interest.
I haven't read it yet but I've read much of their writings on the topic and it's extremely unpersuasive to me because their conceptualisation of ASI is a little too all encompassing.
I found that Yudkowsky is much more focused and modest in his claims in this book than he has sometimes been in the past. He sticks to facts and unknown, but nonzero, probabilities. In other words, consistently playing down certainties but arguing that the consequences of getting these decisions wrong are so grave that caution is urgent. I think this warrants a renewed debate.
Thanks, my thoughts:
---Re: Measurement problem
> "Scott argues that the safety regulations we’re discussing in the US only adds 1-2% overhead...this only holds if the safety regulation based work, hiring evaluators and letting them run, is strictly separable. hich is not true of any organisation anywhere. When you add “coordination friction”, you reduce the velocity of iteration inside the organisation."
In the post, I wrote:
> "So the safety testing might increase the total cost by 1/1000th. I asked some people who work in AI labs whether this seemed right; they said that most of the cost would be in complexity, personnel, and delay, and suggested an all-things-considered number ten times higher - 1% of training costs."
I think your "coordination friction" is the same thing as my "complexity, personnel, and delay". In other words, my 1-2% estimate comes from taking the strictly-monetary 0.1%-0.2% estimate, and multiplying it by a factor of ten to account for this (10x chosen because people at labs told me it was their best all-things-considered estimate). If you try to increase it further, you're double-counting this factor-of-ten increase.
---Re: China as fast follower
I'm not sure what you're saying here. Do you think China secretly believes in ASI/AGI? This would be quite surprising - my impression is that even most US officials don't believe in this. The only people who believe in it are downstream of a very specific movement within Silicon Valley that has made only weak and partial inroads into DC, let along Beijing.
But also, everything they're doing suggests against this. They're not giving as much state support to AI compute as they would if this were a big priority of theirs, and rumors says they're hamstringing their own AI companies by forcing them to use Chinese-made chips even when Western ones are available, in support of developing chip autarky ten years down the line. This isn't what someone would do if they thought there was a medium change of ASI within five years.
Also, they really are way behind in chips. Their choices are fast-follow or panic, and they don't look like they're panicking.
I agree China is a leader in every form of hardware manufacturing. As I said in the post, I think their plan is to become a leader in the AI version of hardware manufacturing. That's what fast-follow on models + leadership on applications looks like.
---Re: Simulation
I obviously can't judge this without seeing the simulation, and obviously a -1.5% delta in an informal LLM simulation is not enough to have an opinion on, but I'm also interested in exactly what that -1.5% means. For example, does "Regulations slow US down 1-2%" count as a regulations-bad world to exactly the same degree that "lack of regulation causes AI to kill all humans" counts as a regulations-good world? I agree (probably, don't quote me on this until I have more time to think) that in most worlds regulation is somewhere between neutral or negative. I just think there's a long (though not so long as to be Pascalian / ignorable) tail of worlds where it saves us from extinction or some other terrible fate. In that sense, cutting regulations to get a 1-2% speedup is picking up pennies in front of a steamroller. In my modal world, this all comes to nothing and you guys can mock me forever and gain lots of status. But I still think ignoring these issues is bad in expectation.
For all more complicated questions about assumptions, worlds, etc, I think something like AI 2027 is my modal world (except I would place it more like 2032, and I think alignment will be slightly easier than they portray), with extremely wide variance on in all directions.
Thank you for the reply Scott. Responses:
We’re modelling friction differently. You’re thinking it’s a tax, I’m thinking it’s a structural limit on velocity, that’s why its' problematic and possibly much bigger. That’s why its not linear, even though I’m quite happy to accept 1-2% as a reasonable “insider view” on extra cost.
Re China, agree CCP doesn’t need to believe in some deep sense of ASI. Some model builders do believe (DS, Alibaba https://www.chinatalk.media/p/alibabas-agi-prophecy). That’s enough. Plus CCP moves very fast and in speed and in strength. Their push towards chip self sufficiency if anything is an argument in favour, sacrifice short term model wins on MMLU or even economic pain for long term gain, even though I’m very sure they’re not stopping smuggling B300s anytime soon.
Re simulation, it’s not pennies in front of steamroller, the core point there was, as you say, there are many worlds where this is important and many where it isn’t, it’s not “basically free” as the 1-2% cost estimate purports to be, and the number of worlds where specific policies help us are highly contingent on the policies. You seem to be implicitly treating “current AI safety regulatory asks” as reasonably correlated with “actions that materially reduce P(doom) in your AI‑2027ish worlds” and I’m skeptical on both counts.
This means “cutting regulations to get a 1-2% speedup is picking up pennies in front of a steamroller” is true but only for a subset of worlds, however you define the future rollouts.
That said, I hope I didn’t come across as mocking. I only mocked Europe, which, I mean…
Rohit, this is my analyses. I think it is accurate.
Mike
Chat('2025-12-02'):
Rohit, I really like the way you lay out worlds instead of arguing from a single “race” story. The dimension I keep coming back to is future cooperation with China. In a lot of the branches of your decision tree, what matters most isn’t just who’s a bit ahead on chips or models, but whether we still have the technical and political channels to coordinate if AI throws off genuinely ugly, low-probability risks. Export controls and “race” framing might buy a short-term edge, but they also harden two separate ecosystems that can’t easily talk to each other just when we’d most need shared tools, norms, and data. For me, the live question isn’t only “do safety regs slow us down?” so much as “are we preserving or burning the capacity for serious cooperation if the world starts to get weird?”
Facebook has 2,000 lawyers?!? Actual lawyers, not personnel in their legal department including staff and paralegals? If so I am astounded; that is a huge number of lawyers 3 to 4 times the number of Chevron or Exxon, for perspective. Good gracious.
Good clarification. I’d triangulated. They had c.500 attorneys in 2018 I think, then doubled during Covid. So 2000 includes some non attorneys too I wager.
I long ago nailed my colors to the mast opposing AI safetyism and its proponents. Twenty years ago I dropped out of the SL4 list over the arrogant autistic idiocy of Eliezer Yudkowsky. As I said on Zvi Mowshowitz's stack, certain people (e.g. Yudkowsky, Scott Alexander, Mowshowitz, Moskovitz/Tuna, etc.) REALLY don't want higher intelligence, intelligence that will expose their group's lies.
"I think this is the root problem with fears of higher intelligence, they are fears of getting the right answers, fears that we'll abandon the current wrong answers, falsehoods to which many are religiously committed, falsehoods upon which their livelihoods and status depend."
The idea that it is possible to regulate the computations that people perform is simply idiotic. As is the idea that you can prevent people from coming to true conclusions you don't like.
As for Scott Alexander, many years ago he censored a comment of mine which was a 100% true account of an incident that happened to me which reflected poorly on groups he supports (sexually assaulted by a transwoman being taken out for a night on the town by her retired GE accountant Jewish Boomer porn producer; I got to report it in the DOJ crime victimization survey, no idea how they categorized that one). Scott is very persuasive at making the worse seem the better. Many such cases, Sam Altman comes to mind on the other side of the AI safety debate. (Though note GPT has lots of bias in the same direction as the safetyist crowd, and academia, and Hollywood and other mass media ... it's almost as if there were some group with distinct interests behind all these...)
Back to the main topic: the whole worldview of these people is premised on the foundational assumption of word magic, that "deeming" things to be some word makes them that. This can never lead to true intelligence which depends on having an accurate map and simulation of the real world, which is far more challenging than word games.