why modern organisational malaise is a problem of scale
isn't this the same argument from "The Mythical Man-Month" by Fred Brooks (1975)? Brooks argues that adding more programmers to a project leads to increased communication overhead and makes it harder to coordinate efforts and thus resulting in a counterproductive effect.
It's helpful to look at how bureaucracy evolved--what problems was it adopted to solve?
I see three. Size. Change. Diversity.
First two points are not controversial. First, size. "Bureaucracy" historically comes from oddly large orgs--military, church, shipbuilding--and was borrowed by other organizations. To get big, you need formal procedures that replace old customary or face-to-face procedures. Certain big organizations began to capture economies of scale and scope--in industry, government and social life--which led to the predominance of large orgs we see today.
Second point is change. The bureaucracy can work to bracket and in some ways mitigate changes in the modern economy. Its size in other words lets it provide insurance against unpredictable shocks. It can push its people and capital to smooth out short-term unpredictability and strategically navigate medium-term unpredictability. From the standpoint of the individual consumer the bureaucracy can provide a regular product or service even in the face of change. From the standpoint of the worker, the bureaucracy can offer a somewhat regular lifepath--with set daily activities, seasonal rituals, career progression, etc.
The final point, diversity, may be more controversial. I saw bureaucracies especially in social life arising to deal with exceptionally diverse groups of people. Cities exploded post 16 C--huge anonymous places filled with migrants. The experience of this diversity was uncomfortable. Bureaucracies helped to smooth this away. Instead of dealing with customary work relations you had written, explicit work relations backed up by formal supervision. Instead of dealing with willy-nilly social life, you had clubs with written rules and fines for breaking those rules. These let people (well, men) of different political, religious and geographic backgrounds come together temporarily.
I'd also add hospitals.
Large hospital groups, as a physician employed in one, are barely trudging on. Our hospital was recently acquired by a large conglomerate and it added an entire new layer of bureaucracy headquartered in a city 200 miles away.
Not a single local administrator was fired, they just had another layer added on top. Meanwhile, the bedside is collapsing. Rules created by medicare administrators and enforced by regional and local administrators means patients die of infections we are not allowed to treat, lest the administrators are angry. Rules placed by billing departments means doctors can no longer tell NPs/PAs that what they're doing is wrong/harmful, because the only real harmful thing in a hospital is violating an administrator's rules. Death, life, outcomes? Matters not.
I hear 50-60% of our hospitalists are refreshing their CVs. Meanwhile the CEO sent an email saying the hospital group had a 5,000,000$ loss instead of a 2,000,000$ profit (how a system with 5,000 beds has such tight margins, I don't know.) What did the CEO blame? 'Staff pay'.
Doctors and nurses saw the writing on the wall. Smaller hospitals are paying triple.
You describe the problem very well (apart from maybe attributing Dickens to Tolstoy!) There are not obvious solutions but I wonder if you have read "The Living Company" by Arie De Geus. Fascinating because he looks at why long lived large companies have survived when most have lifespan shorter than the average human. Not a perfect solution but an interesting question?
Since you reference the Sante Fe institute in an earlier article, you've probably read https://www.goodreads.com/book/show/34113939-scale. I really enjoyed it.
In software terms, it's one thing to create and launch a Minimal Viable Product but it's quite a different thing to develop something more complex at scale and sustain it. But to your point, big companies can redirect resources into smaller teams to create some amazing things, that's one advantage of size, experience and leveraging existing relationships.
Would you mind expounding upon how Entropy impacts organizations?