Are governments really that inefficient, or do we just not know how to measure efficiency?
I feel like you have overlooked a massive reason why people perceive the government as hopelessly inefficient - because of a massive, decades-long PR campaign pushed by wealthy individuals, big companies, and the Republican Party to convince large amounts of people that the government can’t do anything right. Look at all of the advertising, education and “think tanks” funded by the Koch brothers and the many politicians who have complained about “welfare queens”, and Ronald Reagan’s entire campaign. One of his most quoted speech lines is: “The nine most terrifying words in the English language are: I'm from the Government, and I'm here to help.” I can point out more examples if you like but if you look around just a bit you will see they are everywhere. The Republican Party has made increased privatization and criticism of “wasteful” government spending a central talking point and has pushed this message hard through its media arms.
The fact that a lot of people think government is hopelessly inefficient doesn’t mean there has to be some hidden strong evidence that everything the government turns to garbage. It could just be that they have been propagandized into believing something that doesn’t have good evidentiary support. At one point in history the vast majority of people believed in polytheistic religions but that doesn’t mean there must be gods and goddesses running around someplace.
I’m sure you don’t want to come off as being politically partisan. But I’m not sure you can understand why the perception of vast government inefficiency exists without examining the powerful people and corporations that benefit from that perception and invest lots of resources to prop it up.
I think the simplest reason Govt is considered inefficient is because it is hopelessly inefficient. I fear you are getting lost in statistics, papers and coming to wrong conclusions. This is why I find it helpful to always think deductively to ground my inductive reasoning. Consider the following questions :
Is firing an employee who does not work a good for the organization or not?
Is providing monetary incentives to customer satisfaction good for an organization or not?
Is it better to delegate responsibilities to employees to meet targets or better to define and keep writing sets of rules to reach targets?
Which strategy attracts and keeps young talent, high pay, high rsponsibility and high growth potential (and higher risk) or long term stability, low risk, low growth and mediocre pay? For each of these questions, the private sector does one thing while the government employs the opposite strategy. Which would be more efficient?
If the government in spite of employing the clearly bad strategy keeps getting efficient results, then our priors have to be wrong and the private industry clearly has a lot to learn but I reckon that is not the case. It's much more likely that the government is just as inefficient as it's predicted, and you're just getting lost in statistics. You can easily come up with 6-7 other such organisational deductive questions, where the government clearly uses the wrong strategy. The only times the Government is successful is when it has programs that follow at least some of the above questions (especially the 4th one, which matters more) as in the case of DARPA or early NASA or when it gives money to the private industry which can cause issues like Solyndra but also successes like SpaceX, Tesla, Internet etc. My favourite story of Government Programs done right is the Apollo Moon Mission 1969 (written in a book by Charles Murray)
Rxample: Postal service vs. The Bezos machine; the second one operates on subsidized labor, subsidized real esrate, and tax breaks, yet only extracts massive amounts of money out of the economy. It’s a black hole that never sheds a dollar. Meanwhile the postal service is hamstrung, obstructed and underfunded, yet manages to pay living wages, charge nominal rates and provide essential services that no private entity would go near despite their supposed efficiency and despite refusing to pay a living wage. So imo the private sectir is more efficient at sucking money out of society and not much else.
The problem with most comparisons that are made here is they look at a world in which the government already has a monopoly. No wonder private insurances cost more in such a context. I wonder how it is that you wrote an article about the costs of efficiency comparing public and private efficiency without ever talking about government regulations.