The Algorithmic Leviathan

Imagine a water authority facing a long drought. Its models forecast shrinking reservoirs and estimate how different restrictions would affect households, farms, and industry. The analysis may be better than any one official could produce alone. But when the system recommends which neighborhoods should take the first cuts, who gets to question the answer?

When a problem grows beyond what people can easily understand together, we look for someone or something that can settle it. That impulse is older than software. People have turned to rulers, religious authorities, courts, and experts when factions could not agree. These institutions are not the same, but each can promise a decision from outside the immediate argument.

An advanced AI system can make that promise feel especially convincing. It has no family to favor and no election to win. It can compare more information than a committee can read in a day. Those qualities may help us coordinate. They can also make a recommendation look impartial when it still reflects choices made by people: what to measure, what outcome to optimize, and which costs to accept.

Relief from deciding

Climate, energy, finance, and supply chains cross borders and affect one another. A policy that helps one place can shift costs elsewhere. Every government faces pressure to protect its own residents, even when cooperation would leave everyone better off. It is difficult to ask people to accept a loss now for a benefit spread across many countries and years.

A model can help make those trade-offs visible. It can compare scenarios, point out consequences that a decision-maker missed, and show where the available evidence is weak. But a forecast cannot choose whose needs matter most. That is a political judgment, even when it arrives in the form of a score or ranked list.

There is another temptation. When a decision hurts people, leaders can point to the system and say the numbers left no choice. That explanation may be comforting to the people making the decision. It does little for someone whose water was cut off, whose work disappeared, or whose community was treated as an acceptable cost. The model can take on the appearance of responsibility while the people who set its goals remain difficult to see.

Advice can become authority

The analogy to an oracle is useful, as long as we remember where it stops. A neural network is not a priesthood, and its output is not a message from outside human life. Yet a system that is hard to inspect can acquire authority because only a small group knows how to interpret it. The rest of us are left deciding whether to trust a recommendation we cannot fully examine.

That matters more as automated decisions become tied to essential systems. Software can help balance an electrical grid or route traffic through a supply network. In settings where decisions must happen quickly, a person may not be able to review each one in time. Over years, institutions may build their routines around machine recommendations. Reclaiming control then takes more than switching off a tool; it means having people and procedures ready to keep services running.

The handoff can happen in small steps. A system begins by advising. People follow its advice because it is often useful. Its recommendation becomes the default, and questioning it takes more time than accepting it. No dramatic takeover is needed. Dependence grows through ordinary decisions that each seem reasonable on their own.

Keep a way to disagree

We should use computation where it helps us see more clearly. But a system that shapes shared decisions needs boundaries that people can understand. We should be able to ask what it was asked to optimize, whose information it used, and what uncertainty remains. People affected by its decisions need a way to challenge them, and essential services need a credible fallback when the system fails.

Those safeguards will not make every choice fair or every crisis manageable. They can keep technical skill from quietly becoming political authority. As I wrote in The Architecture of Consensus, a decision process earns trust when people can see how a choice was made and where disagreement can go.

The deeper question is not whether machines can help us govern at a scale no committee can hold in its head. They can. The question is whether we can accept their help without handing them the right to decide what a society should value. A useful system may show us the costs of each path. We still have to decide, together, which costs we are willing to bear.