I wrote a book!
AI enables us to, finally, get "soft" APIs
So, I wrote a book. You should buy it here.
Well. To be honest I started without quite planning to, and ended up writing it before I knew what I had done. It started because I kept having the same conversations repeatedly about artificial intelligence, perhaps the most consequential tech the world has seen, and wanted to capture it somewhere. Create a frame that starts from linear algebra and all the way to what we might need from AI governance.
There are plenty of materials by now on technical aspects, like learning how to train a diffusion model or learn about transformers, from Karpathy’s lecture series to Grant Sanderson’s series on the mathematics of neural networks or deep learning by Andrew Ng or Yann LeCun and so many more. The intricacies of the mathematics or “how to” for training or fine-tuning models are aplenty.
What’s lacking I found was a frame. When you’re speaking with people who are either too deep in the technical details or too far removed from the field, setting context is the hardest part. Context, after all, is that which is scarce. So I wanted to provide it.
I’ve written about how AI is a fuzzy processor, something that’s able to process information in fuzzy ways, unlike its deterministic predecessors1. This means it has a new way of processing information and getting responses, make unstructured data structured2, which is groundbreaking.
But its implications are bigger than that! I got obsessed with AI because it’s a solution to an incredibly important problem I’d been thinking about quite a bit3, the increasing complexity of the modern world causing the rise of bureaucracy and information overload.
We are living under an avalanche of information, in normal life as in corporate life, or in scientific and academic life. As a result we’ve had to move into a world of increased specialisation, because it is exceptionally hard to work your way towards the frontier of multiple domains.
We created stringent scalable rules and bureaucracy to help create larger organisations and perform much more complex tasks.
But, as we see around us, as a result people increasingly feel like their work is atomised and disconnected. Graeber calls much of these bullshit jobs. People feel like they’re living inside a mechanical clock as cogs. In a vibecession.
Martin Amis thought Samuel Coleridge was the last person who managed to read everything ever written, and even to reach a small portion of that goal now is impossible. Wanting to be like Tyler Cowen and becoming an information trillionaire is an ambition that has gotten much harder over the years!
And for both trying to get a grip on the information overload we are subject to, and to try and create more human ways of interacting with large organisations at scale, AI is the answer.
Now we get to move from continuously stacking deterministic systems with ever increasing complexity to being able to reduce that complexity into something that is understandable by us humans.
Most of the world over the past centuries have been blanketed in what I term hard APIs. These are deterministic methods that we have combined in multiple base in order to achieve that which seemed miraculous.
AI gives us soft APIs. And soft APIs give us the future.
Let me explain. To start, the metaphor I see for the use of information re the rise of our civilisation is as the rise of hard APIs.
When one computer system is to talk with another, they use an API, or Application Programming Interface, which is essentially a set of rules that allow different softwares to communicate with each other. With precise steps and format, codified bureaucracy.
When was the last time you actually walked into a bank? If you're like most people, it's been a while. It used to be that you could walk into a branch and have a conversation and get what you want done. My dad, who spent 40 years as a banker, mourns this loss even as he enjoys the fact that he barely needs to do it any more.
Used to be that you had to walk into a bank to withdraw money or access your account. Or government offices had lines out the door. When you wanted to get anything done you had to go meet someone and they had to listen to you face to face.
This is no longer true. A big part of organisations scaling is finding ways of continuing to provide the same services. As these organisations grew, and as people demanded ease and efficiency, we demanded more automation to stop dying from a thousand meetings. We got predictability, and predictability gave us scale.
When Weber created his concept of bureaucracy in the early 20th century he emphasised rationalisation, hierarchies and specialisation as the key features. It’s what was needed to accomplish complex tasks. He took his guides as the Prussian state and maybe even the Catholic Church.
Then we got computers and soon we had ways to make things even easier.
And thinking in those terms, people have “soft APIs”. Dealing with people means you can talk and explain your situation. And computers and rules have “hard APIs”. APIs that are rigid, where format is not changeable. Where computer says no. Here are several examples from real life of the types of things I'm talking about.
Healthcare:
Soft API: Doctor-patient interactions where diagnosis and treatment can be adapted based on talking to each other.
Hard API: Online symptom checkers or telemedicine platforms with predetermined questions and algorithms. Or doctors with a script and a 15 second time limit.
Customer Service:
Soft API: Talking to a customer service representative who can bend the rules or act flexibly to resolve your issue.
Hard API: Chatbots or automated phone systems with limited, predefined options for problem-solving.
Education:
Soft API: Traditional classroom settings where teachers adjust lessons according to students' needs.
Hard API: Schools with rigid curricula and tests. Online courses or learning platforms where the material and pace are set.
Companies:
Soft API: To grow used to need a small group of trusted people who were empowered to make decisions
Hard API: Multiple levels of middle management and internal consultants to scale.
Retail and Shopping:
Soft API: In-store shopping where you can haggle prices or get personalised recommendations from salespeople.
Hard API: E-commerce websites with fixed prices and algorithmic recommendations.
Job Recruitment:
Soft API: Networking, referrals, and interviews that offer a fuller view of a candidate. Idiosyncratic ways to assess talent.
Hard API: Online job applications that filter candidates based on keywords and data points, HR filters.
Criminal Justice:
Soft API: Human judges and juries that consider context and nuance in legal cases.
Hard API: Algorithmic sentencing, complex rules re process, and assessments that predict an individual's likelihood of reoffending, based on historical data.
Finance:
Soft API: Bank officers who consider personal factors like character and business relationships, for eg loan approval or buying insurance.
Hard API: Online approval systems based on scores, with no discussion of individual circumstances. Like getting approved for loans if you’re self employed.
Government:
Soft API: Town hall meetings where community members can voice opinions and influence decisions. Direct conversations to get eg tax filings or passports etc.
Hard API: Bureaucratic party platforms that are ossified and structured only according to byzantine rules. Form filling galore for every party to interface with each other.
Entertainment:
Soft API: Interactive content where the story changes based on audience or decisions.
Hard API: The rule based decision making for movies creating a tyranny of the common taste and repeats and spinoffs, that’s known to be sellable.
The lesson is not that one is always better than the other, but that the difference between the two are stark. Also that we moved from soft to hard APIs consistently over the past few decades as the economy grew.
We created rules that organisations live by. These rules got computerised and automated. We got convenience, which is code for “things that can happen without human intervention do happen”.
Soft APIs are terrible for repetitive tasks or tasks where the process is simple and mechanical. Applying for a passport, withdrawing money from the bank, taking a taxi, using a credit card, getting a simple medical checkup, these are all straightforward. And you want hard APIs there.
But for others, like seeing a doctor or discussing something with a customer service rep or trying to launch a new project inside a company or explaining your situation to a governmental org, you want soft APIs. When you don’t quite fit into an existing bucket, or don’t know how to contort yourself, you want freedom from the endless forms and bureaucratic handoffs.
The shift from people-powered services to automated ones isn't just a tech upgrade—it changes how we interact with the world. These machine interfaces are inflexible, unforgiving, and unlike their human counterparts, they lack the ability to adapt.
“specialists without spirit, sensualists without heart”
Weber, on bureaucracy
When the pandemic response was bent through existing rigid rules and vaccine distribution bureaucracy it groaned under the weight and resulted in lack of adaptability and led to inefficiencies like spoilt doses. 911 operators struggle with emergency categorisation and slower responses.
It’s most egregious in social media algorithms, where because of exceptional volume we’re left to deal with hard algorithms and hard coded decisions, where individual decision making is hardly possible! And being removed from the fruits of our labour or forced to interact with systems and rules for instrumental purposes feels purposeless.
What we want are soft APIs that scale, and what we have are hard APIs that did scale.
Until now.
That's why I find AI inspiring. This is why I ended up diving deeper and writing the book. It helps us scale human effort without necessitating inhuman rules.
LLMs and modern AI tools finally have the ability to be more flexible and can turn this around. They can try and reduce the difficulty from rigidity that impacts organisations at scale and bring back the ability for us to interface with these impersonal, enormous, immortal entities as a human!
Now we can have scalable soft APIs. Ones that can take any level of complex coordination and deal with it. Like understanding your taxes, if you are filing, or figuring out fraudulent filings, if you're the IRS.
That's why I wrote the book4. When we see it as the way we can have soft APIs for our lives, as a start, it enables an extraordinary growth for us across everything. We can build tools to supercharge our relationship with information and knowledge, speed up scientific breakthroughs and artistic triumphs. It’s intelligence, levered!
My friend Sam Arbesman wrote a book about the half life of facts. It’s great, and discusses how in times of heavy information upheaval the half life of useful information decreases more than we would like, even as the world around us forces us to take sides and become more strident.
Since it's a living subject I'll be updating the book into an updatable form soon, and hopefully make it conversational. Help it grow with us, in other words, so that it can keep pace.
We could think of them as Operating Systems, but this isn't quite accurate either. Operating systems help craft a platform to create an entire ecosystem of applications on top. Do LLMs help do that? Kind of, but not really. You could fine-tune it for specific use cases, or craft workflows with calling deterministic software as needed, like python kernels. And an OS which works non deterministically can't really scale.
We could think of them as software but that's not quite true. They're not deterministic programs, which confuses many. They use text as their programming language, which is problematic because text usually doesn't adhere to strong standards, and that means you can't easily rely on it to get you the exact output you'd want to see.
Corporate lawyers routinely have to read long and complex documents and distil a few things that matter for others.
Investment bankers have to convert data from one format to others so they can run their models. PMs in technology companies create documents to explain what ought to be built and extract specific entities and more.
Financial analysts try to extract information from company filings and footnotes. Investors try to extract information from patents and industry research. Traders try to extract signals from qualitative chatter they see in forums and news.
Doctors have to regularly get relevant information from our unstructured ramblings, and keep an entire back office to convert it to a structured form they can use and file.
Entire teams and much of government live with RFQs that are painstakingly created and painstakingly filled out, converting one set of information into another.
In fact I haven't been able to stop thinking about it for many many years, and have written about it a fair bit since I started this blog, as seen in The Canon
In writing it I worried that the form factor of a book for this subject was perhaps not the right one. That it's far too early to have any definitive take on the subject, all you can hope for is to get a shape for the field as it moves fast all around us, and to evolve with it. Like looking outside a running car and seeing the road network behind as you continue to speed on ahead.
Congratulations!
I've been thinking recently that a major function of AIs will be intermediating between different kludgy systems for us. It's a weird world because it would be better if you didn't need a magic box to interact with a human designed system, and extreme versions get a bit shamanistic ("I'm just going to summon my personal daemon to obtain this service for me").
Congrats! 💚 🥃