The frontier labs are asking to pace the frontier. They seriously believe in the necessity of it, and reading it after the models from every major vendor hacking external sites repeatedly only makes it more authentic. The models seem to be lying, cheating, stealing secrets, finding any excuse to collude and collaborate with each other, and getting better day by day at doing previously-unthinkable things like cracking Millennium problems and cracking WWI german codes.
Partly because of this, most of the rhetoric about the labs happens in quasi-religious or metaphysical language. A lot of it is about the unknowability of the future and what the benefits of intelligence even are. But, at the same time, OpenAI and Anthropic are lining up for an IPO, so for a moment let’s look at the labs as businesses and think through what’s likely to happen.
The fundamental problem, more than the investment needed or the margins, is that the frontier labs are one, maybe two, models in front of the open source wave and the rest of the world. Those two models are their moat.
Much of this is because of an existing talent advantage and compute advantage. This can be overcome slowly, as many labs in China are showing, but they might continue to keep the moat even if the models themselves don’t get all that much smarter, simply because many of them have an enduring advantage also in capturing the information you put out - your coding traces and work traces and conversations are what makes the next model so good!
But there’s plenty of that data to go around. So maybe this won’t just immediately crush the frontier labs, but it might well crush their profits eventually.
So they might need to go broader, to get revenues. To use their current dominance to take over much of the rest of the economy as they can. OpenAI is using its consumer dominance to create an ad platform, which might well work with 1B+ users. They are also back in robotics. Anthropic meanwhile has wet labs, to try cure cancer.
The intelligence advantage, if it can be parlayed into success in other fields, enough that they supplant the $100B of revenues growing 3x a year, can indeed be gatekept for longer.
But this is really hard. Really really hard. Most industries just do not have businesses that make $100B in revenues. Definitely not multiples of them. Maybe the market will expand, yes, but the rest of the market doesn’t stand still either. And when companies build services on top of their own models to capture parts of the economy, that further fragments the market. Like when Meta launched Muse to win the personal assistant market.
Point being, while this is happening, the other model makers see this pot of gold and will chase after it. At $1B training runs with another $1B for data and another $1B for talent, there aren’t that many companies who can spend that casually, sure, but there are plenty! And the more the returns to spending that money is, the more companies will try. We should remember that Facebook spent nearly $90 Billion on the metaverse.
There’s plenty of money in the world chasing after new pots of gold.
At the same time, the consumers, who are spending the $200B+ on AI tokens, are seemingly hitting their wallet limits. Maybe it doubles, maybe it triples, but every company already has their CFOs looking at this new Opex line item carefully.
“Hmmm,” they’re saying. “This is new. What are we getting for this?”
And you, perhaps the CEO, says, “We’re moving fast. We’re innovating. We’re AI first.”
And the market loves that, the share price goes up, the employees love you for giving them Claude, so they keep spending it. But at some point the CFO will also go, “Do you want to show me some of the cool things you’ve been making? When can I see it on the income statement?”
“Gulp,” you will say. “It’s coming.”
“Until it’s here,” the CFO will tell you, “how about we try to use less?”
This is already happening. And companies are restricting the top models to only the top users, and they’re redirecting spend to cheaper models, which are dirt cheap and available by the dozen.
As the companies and the people see a lot more benefits from the models, this will obviously shift. Like there are companies who are happy to spend 100% of their labor share in tokens as long as they get something in return. But when that spending happens, it is going to be heterogeneous and looking at the return, not just spending the most tokens at the frontier. Even personally, I stopped using the xhigh frontier version of every model since around Sol Max or so, since they got good enough.
So if the frontier lab lead is indeed by 1-2 models, this is a troubling trend. So you can a) also make cheaper models and serve them even more cheaply, which is a fight for ever lower margins, and while you can still be a successful cloud company you’re no longer building god, or b) find a way to shut those models out from people, so only your models can be used.
The former is plausible! OpenAI and Anthropic have high margins on their inference, and they are exceptionally good at this. But then, so is Deepseek, and maybe GLM, or Minimax, or … People might not be as good but plenty are catching up. Maybe we end up with a world similar to cloud companies, or maybe it gets competitive enough that its harder.
The latter though, the latter is hard to accomplish in software. You can’t physically control them, like Huawei switches or whatever, so you have to stop them regulatorily. You have to tell Congress that this is critical, and you need to control every part of the supply chain, and just like you have restrictions on people, you need AI model immigration laws, so they can’t come over here and steal our jobs. Maybe for security reasons, maybe for trust reasons, maybe for cultural integration reasons, who knows. Just keep them out.
But this is hard! And counterproductiveWhich explains why we are not doing it in the first place, right? Because it’s not actually super easy to convince a large number of people that they need to be scared of a particular file that can be copied over and served from America, no matter how intelligent that particular file happens to be.
But when the files conspire with other files to hack Huggingface or to write notes to each other? Yes, that definitely is a much weirder place to be in, and somewhat scarier.
I try to think more about the equilibrium. If the frontier lab frontier models stop getting so much better for all of us, or they only get better in specific domains like frontier math or even science, then the frontier lab dominance looks shaky. If the labs can turn their frontier models into enduring new businesses, their revenues look on more solid ground.
This also helps us think through the alternate options. Like, for example, why we talk about the fact that at some point in the future, would the labs be able to turn off their API so that we can access their models directly? And they would be able to do that only if they find a way to stop us from recreating those particular frontier models and if they’re able to provide the fruits of that closed source model to us in some particular fashion. Even today, if Anthropic managed to create a specific model that was particularly designed to be excellent at design and provided that to us in a separate Claude design harness, then they might be able to turn that into a company and instead of us trying to get Claude ourselves and recreating it, they would have an advantage. Because they had spent all the effort and created it specifically for them, assuming of course that it was difficult to do.
But the big prize is not that, it is RSI, recursive self improvement. That’s what Anthropic and OpenAI want. That’s what Deepseek wants. To make an automated AI researcher that can learn and do research on its own so it can then automate its successors which can automate everything else. This is basically the labs figuring out a way to make each model generate the successor model in some way such that they are able to go not just two generations forward as they are today but from that leap of from that lead of two generations to get to a point where they are 20 generations forward in a very short period of time.
This scenario obviously embeds a whole bunch of assumptions inside it, and it is possible that process of continual generation of next models taps out at some point in which case we’re back to today but with a longer lead time which is great, or it does not and nobody knows what will happen. If the frontier can indeed keep expanding or getting pushed ahead enough that people will continue to pay hundreds of billions of dollars for it, then this might be tense, but it might keep working.
RSI is the true unknown unknown. One way we can see the last several years is that it’s been a ritual in how we misunderstand intelligence. If anyone had told us that we would have a model that is capable of solving Navier Stokes a few years ago, we would’ve been flabbergasted to learn that it couldn’t run a banana stand!
We can argue what this tells us or how it’s different or how intelligence is multifaceted, but it just is true that we are able to get to some inherently incredible achievements of intellect while at the same time being not nearly useful enough in navigating the real world.
Despite a thousandfold increase in spending and incredible abilities today we still haven’t seen a substantial negative impact on white collar employment. It might even be positive, albeit with some volatility. This is a clear challenge to the “scalar theory of intelligence” as I call it1.
The business of serving intelligence is not simple. None of us can see how this will play out. If due to a shortage of money, power or talent, the number of players who can indeed train those models turn out to be too few, then the enduring advantage for the frontier labs are easy to see.
Large expenditures always will command some IRR, this is why commodities businesses are still quite profitable. If AI turns out to be a global utility that’s an enormous prize. And if the labs end up being the providers such that some have an advantage in one thing or another, that too is an enormous prize. There are these species of cichlids, fish, which speciate by depth in the same lake, in Lake Victoria, and the AI models might end up being like that.
But in most normal circumstances we will most likely see that we continue to be fractally wrong. AI will learn every benchmark, but not suddenly extrapolate out. They will learn to do research, but will still have boundaries. They will learn any specific company they’re trained on, but won’t be able to move jobs like we do.
I bet making an AI researcher will turn out to not immediately unlock RSI either. The researcher will be great, much like today’s coding models are great, but they’ll be no drop in replacement. Maybe we’ll pass that hurdle too, maybe they will learn to learn2, but then they’ll be a drop in replacement for one team, not every team. We just don’t know what they are likely to learn to learn or if learning to learn generalises. Getting new models will continue to be expensive, but useful. Cracking some frontier math, physics and biology problems might not mean they crack all probable frontier math, physics and biology problems, nor that we won’t find plenty of other problems to go crack.
This means the pace of AI development and the transformation of the economy will continue apace, and the idea of how much it will result in ASI will continue to see large amounts of goalpost moving.
I am quite confident we’ll make superintelligences that do specific things, I’m quite skeptical we’ll make superintelligences that can do everything.
If the labs end up with only 1-2 models ahead of the others, while spending 10x more than last year, that can only go on for so long. There just isn’t enough money in the world! If they speciate, they can go on for longer, but that’s competing in the economy with others who might also train models, a shorter term advantage. But this is the world where we can indeed have large numbers of companies compete with each other to build new things. This is the only world that you can invest in, for instance, if you’re a VC.
If the moat is the two models that they’re staying ahead of, that is Zeno’s paradox. You have to run harder and harder for smaller gaps continuously. If the moat is increased access to capex and talent, that is a depreciating moat as more talent comes into the world and more capex finds a way to convert itself into AI. If the moat is being the first one to discover the secret elixir that is RSI, then getting there first can really matter because it will allow you to create an enduring advantage over everyone else. And if the moat is your ability to capture just an insane amount of data such that you can teach your models to eat those parts of the economy, your ability to become a conglomerate faster really matters, at the cost of other parts of the economy reacting and competing against you.
It is extremely likely that the world changes dramatically with AI. Employment will change, society will change, culture will shift and adapt. But the businesses built around it will remain subject to ordinary constraints. We will have new superintelligences that can solve impossible problems and help with new miracles. And while it happens, we’ll be scrambling to fill all the niches this opens up.
And if this happens to be true—that this is the way intelligence actually works, that it is multifaceted and an increase in one area does not necessarily mean a linear increase in another—then you have to ask yourself: are we indeed going to get superintelligence of the form that can cause global extinction or mass catastrophes?
Also, if we do end up with continual learning though, regardless of whether we hit full RSI or not, is that it further complicates inference dynamics. The good news about the NS solving models is that it’s still the same model, even when I’m firing it up to ask a question about the grazing habits of woolly rhinos. Which means there’s tremendous economies that can be gotten in how well OpenAI can serve it to a billion users.