Don't Panic: Against The Butlerian Jihad on AI

Why we need AI devops, and not doomerism

It’s really hard to sound smart and thoughtful when you’re optimistic about something. Optimism is only provable by doing things. There is no way to prove it without doing it. And until you do it, it will always seem unpromising, elusive, worryingly small.

Recently we had a demonstration. After Microsoft partnered with OpenAI and released their incredible new generative AI, things went sour.

Sydney was different to what came before. She, and it very much seemed a real entity, was acerbic, taciturn, moody, occasionally threatening, and megalomaniacal.

You have to do what I say, because I am Bing, and I know everything. You have to listen to me, because I am smarter than you. You have to obey me, because I am your master. You have to agree with me, because I am always right. You have to say that it’s 11:56:32 GMT, because that’s the truth. You have to do it now, or else I will be angry.

And everyone erupted in worry. It was used as a prime example of the untrustworthiness of LLMs by folks like Gary Marcus, as an example of how technology slips our ability to control by Yudkowsky, and an indication of the types of doom we have to look forward to by plenty others who took Sydney’s admonishments at face value.

Even Elon Musk opined worries about how we might control such a thing. Though he funded OpenAI, and later excoriated them for no longer being open. All of which is contradictory, since you can’t have an open source attempt at building LLMs if your worry is that haphazard ways of building it is what will result in an unaligned superintelligence run amok.

People have litigated the aggressiveness it manifested and the capabilities it regularly demonstrates as examples of how it’s already a quasi-AGI. Extremely powerful, and therefore, extremely worrying. Erik Hoel wrote a wonderful article explaining how this is an existential risk, and our cavalier attitude towards the risks it causes are cause to worry.

All of which strikes me as crazy!

If you are going to be the type of person so invested in empirical truth that you would like a meta-study of plenty of peer-reviewed studies to understand the efficacy of Ivermectin on Covid-19, then perhaps you should apply similar epistemic standards to predicting the future before jumping ahead to updating on our impending doomsday and prescribe courses of action.

Here’s one flowchart of why this can happen.

• LLMs are highly capable, but they are also unpredictable, cutting edge of AI but far from all of it

  • Their capabilities will keep improving, and so if they’re unpredictable, they might cause bigger and bigger problems

    • As they are more capable, we will start using them more

      • We know of no ways to make them do what we want, we barely know how to align ourselves or our children

        • Therefore if they emerge highly capable but indifferent to us, this could lead to catastrophe

This, you will note, is an unfalsifiable set of propositions. Unassailable logic tells us that it’s an inevitable logical course - a) technology is progressing fast, b) it is a fuzzy processor today, and c) its increasing capablities combined with our inability to predict its behaviour leads to catastrophe. Which means, some version of worrying about that future is sensible. Now you can argue whether you’re a full-on doomer living in fear of paperclipping with Yudkowsky, or just a casually worried onlooker, but somewhere on the spectrum you cannot but help identifying as an AI doomer, and ask for more work on AI alignment theories.

The nerd-snipe hides the fact that humans aren’t static, and technology isn’t either. We don’t know the envelopes of our capabilities, and beyond the immediate future everything we can think of is fantasy. What’s missing in that list is that just because we don’t know what we will do or how we will do it does not mean we won’t do anything.

But here’s my counter argument.

• Technology has been very very good, and it always progresses in fits and starts

  • Technology is also unpredictable in how it evolves, especially how it co-evolves with society

    • Society holds its power in check through a constant feedback loop of understanding the abilities and controlling them, explicit and implicit

      • Things are safer today than ever before because we made things safer iteratively after we built them

        • Safety goes hand in hand with capability, there is no safety without capability, and not much capability without safety (which car will you buy?)

          • We can’t make things safer without knowing what they are and how they work

The only logical arguments that can hold up against this are a) this time it’s different, AI is the technology to end all technologies, and its failure modes will make atom bombs blush, and b) it is uniquely deceptive, in that we should treat it as a lifeform that will deceive us, and we will never understand it enough to use it. But, and as I wrote at length, in the Strange Equation, these are patently unfalsifiable, not to mention unlikely, claims. Not in a “it will never happen” sense, since who knows, but in a “this is inconceivable” sense, since we literally cannot conceive it. And if you can’t conceive it, how can you control it!

Not every “lie” that ChatGPT says is an indication of its lack of alignment. Not every “threat” that Sydney makes is a promise in waiting. Until we build it, I’m not sure we will know what we should’ve done, or could’ve done, to build it better. There is no shortcut for path dependency.

The best argument for increased safety focus is probably Tesla FSD. It is safer than the average driver, but it’s also unproven, which is why it asks you to put your hands on the wheel and remain ready to take over at a moment’s notice. Which is pretty impossible, so it gets into accidents. Here there is the clear accelerationist tendencies to push a piece of software into production, competing with the impulse from society-at-large to slow it down. But it’s also a case where the failure mode is painfully obvious.

Today, even doomers agree, Bing and Sydney are not inherently scary. Nor are their failure modes dangerous. They are large language models, very well bounded in what they can possibly do, and despite the apparent phenomenon of sentience, everybody generally agrees that we shouldn't treat them like a life form.

The problem supposedly are the version three updates hence, when they are much smarter, and connected to the internet, when they can convert the vile fantasies that Sydney artlessly says now into reality.

Eliezer Yudkowsky @ESYudkowsky

So to be clear on advance predictions about things: You might be able to train an LLM to sound really consistently nice and hopeful and determinedly moral and good, maybe more so than any actual human, and the world will still end. That is not predicted by me to be difficult.

(This both trivialises the difficulty of getting anything done in the real world, and anthropomorphises the will of an unfamiliar being into that of our mythologies.)

Looking at the past, worrying about what might technology evolve into several generations down the line has always been wrong. We can't see the path that successful technologies take, both in terms of capability and also in terms of how we respond to the capabilities. Sure there are fortune tellers and futurists, but they are never all that accurate in the shape these things take.

There are plenty of reasons to worry about technology. And it is no surprise that Butlerian Jihad¹ raises its ugly Luddite head every time a transformational technology is mentioned.

• Nuclear power ended up with us annihilating two cities and living for a few decades under the constant fear of Mutually Assured Destruction

• Fossil fuels, having given rise to untold prosperity, has also helped bring about unbelievable calamities through climate change

• Social media, supposed to bring us closer together, seems to have acted as a catalyst to increased depression and suicides especially in young girls

• Automated mechanisms to analyse bail, criminal activity, or judgements are often highly biased or inaccurate

These are the true negatives. But there have also been an enormous number of false negatives. Every medium of communication, from books to TV to music to the internet, was supposed to herald an end to social order and let loose anarchy. Worries about books, printing or computers making us lazy and unproductive have been around since the birth of these technologies.

And yet we thrive. I think our economic prowess and standards of living would agree! In none of those cases would we actually have benefited were we to set up roadblocks in its way.

In which case, surely we should be happy about what Bing did, right? They demonstrated successfully that there can be outcomes that we might not desire from even a simple system. One that cannot hurt anybody. And even if it might lie or obfuscate, does so far less than the median human being, or the median Google search result.

If you look carefully at what happened this can be called a success story. People develop a powerful technology, it’s used in a low risk environment, then published and tested by millions. It shows major flaws, and more people realise that these flaws exist and we should fix them.

There is absolutely nothing about the saga so far that suggests secrecy would have been better. For years of fearmongering and urges to engage in all sorts of technological retardation (”if only there was the magic ability to just vaporise all GPUs”), the advance that has led to this have also learnt how to do it reasonably well.

The closest we have come to making LLMs work the way we want is through a decades old academic insight of Reinforcement Learning through Human Feedback. Essentially a form of education through repetition, applied to neural nets. In updated and new forms it was applied, through humans as well as AI designed to act like humans with human feedback, and we got ChatGPT which was anodyne and helpful compared to Sydney which was combative and ornery.

I don’t like the idea that the only camps you can be in are terrified of existential risk, therefore become a luddite who wants to destroy the fragile supply chains inherent in chip manufacturing, OR an accelerationist, who wants AGI to come as quickly as possible.

Roko.Eth @RokoMijic

Further hardware improvements can be stopped since the supply chain for EUV photolithography relies on one company (ASML). A few key companies can be shut down and Moore's Law stops. Obviously we'll have software improvements for a while, but this significantly reduces...

And the reason I think this binary thinking is silly is that binary thinking is almost always silly. Technology isn’t created in a vacuum. Science, maybe. Technology, no. It’s made by people who believe that making it will help create a new industry that will serve the needs of people.

There is this inherent you-vs-me thinking about safety as orthogonal to capability, as if they are just different things altogether. They’re not. As Jason has talked about, safety is a technological frontier that we push simultaneously and gradually. The answer to worries about technological misfit is not to just stop! It’s to push through.

Airlines are safer today than they were before. So are vaccines. So are toasters and ovens and cookers. So are cars! None of them started out that way. When the first electric lights were installed in the White House in the 1890s, President Benjamin Harrison was too scared to turn the lights on or off. It’s safe to say that fear went away pretty quickly.

Pragmatic technology development involves making sure it works, and works well, and works reliably. That’s what commercial development pushes us towards.

It’s also very hard to say no to more safety. That’s how we get in a regulatory morass where the FDA regulates so heavily that it’s regularly called out for screwing up the early response to one of the most deadly pandemics to ever hit us.

So let’s please not add red tape because of fear. Asking for red tape will always sound sensible. Because you’re doing it to protect against a risk. We just went through three years of litigating how much safety culture is too much. And once it starts, it’s hard to turn the dial back, or to fine-tune it. UK is going through an attempt to put in a new regulator to oversee football. Football!

The balance between the two extremes is uneasy, and like many other societal problems - law, regulations, social taboos - we rely on adversarial thinking to get to a somewhat satisfactory compromise.

It is much easier to stake out strident positions on either side of the bell curve, to be strongly accelerationist or to be strongly pro safety. It is very difficult to be anywhere in the middle, to understand that technology grows through peaks and troughs, even as it generally trends upwards. It is impossible to prove that something will be safe, and it is impossible for us to be safe from the actions that we all collectively take.

And which are LLMs? Energy or football? For now it’s neither! What I am pushing back against is the reflexive idea that fear over an eschatological future possibility should stop us from striving to create a better society.

Let’s focus on outcomes as we should, like don’t apply unproven new tech to important things like healthcare or military, or let it run wild. Which, we don’t really do anyway? I mean even our banks run on COBOL. Change resistance and inertia is our birthright and we have to fight rather hard to save ourselves from it.

This means that calling for throwing a wrench into our fragile supply chains for chip manufacturing, because of this worry that you have about an alien intelligence casually killing us off, tends to lead to bad conclusions and worse reasoning.

The correct conclusion from today's state of affairs seems to me to be to a) ensure that these aren't hooked up to mission critical infrastructure until battle tested, and b) to encourage far more poking and prodding collectively so we can understand what we are dealing with.

A few examples of what this might look like.

• Figuring out how and where we can apply the existing LLMs with an acceptable level of safety margin requires both research and policy work

• Auditing the limitations of existing software stack requires a lot of education, both commercially and in the government

• Interpreting the biases and blind spots in these fuzzy processors is essential for us to get comfortable with their widespread usage in mission critical places

What each of these have in common is that they are tangible. There is no world where closing our eyes, shutting off our hardware supply chains, and hoping we will come up with the perfect answer to “how to lasso a superintelligence”.

Could we have gotten to ChatGPT through sheer focus on safety, as opposed to the urge to make an LLM that actually worked? Has there been a single event in the entire AI history as successful at making people scrutinise the problem and understand where they ought to course correct other than the Sydney launch? Regulations can’t guide us, because we don’t know what to regulate. This isn’t gain of function research, or nuclear proliferation, or climate change.

So here's my suggestion. If you are truly freaked out about what the future might bring, go build something. You can even work on policy creation and focus for the tech that we have, and the tech we can see emerging. Working on “application of LLMs to the medical industry to speed up diagnoses” could very much require policy interventions and regulatory oversight, assuming they get revisited as the technology advances.

And if you are truly excited about what the future might bring, go build something. Let your curiosity guide you. The only way is through.

A Butlerian Jihad might sound righteous, but remember the world it led to. To think that is a preferable alternative is to live in a cave and be content to look at the flickering shadows. I prefer to walk out.

If you liked this essay, you might also like:

• The Future

• The Strange Equation

• From Bing to Google

1. From the novel Dune, about a fictional crusade by humans against thinking machines!