The control we left on the table

Posted on: 23 October 2026

Consider an exchange that is almost certainly taking place in an office near you this week. A manager needs to send a proposal to a counterpart at another firm, so he asks an AI model to draft the email and to put together the document that goes with it, glances over the result because the tone seems about right and presses send. His counterpart receives it with a dozen other things open on her desk, so she hands the email and its attachment to a different model, asks for a summary and then for a reply, which she skims before sending it back. He has the reply summarised in turn, and the exchange carries on like this for a few weeks until the two of them settle on something operational and somebody moves money or people. At that point the decision is real, although the conversation that produced it was never really held by either of them.

I describe the scene without standing outside it, because I do exactly this when a document is long and the day is short. I have the text summarised, read the summary and ask for a draft reply that I adjust before sending it. Every so often I notice that a reply carrying my name refers to things I never read in the original, which means I have taken a position on a text I know only through a machine's précis, and I usually notice it afterwards.

Put like that, it looks like proof that artificial intelligence has already taken control, which was my own first conclusion. It is a satisfying thought and frightening in just the right measure, but it gets the mechanism wrong, since a language model has no goals of its own and no interests to defend, so it takes nothing from anybody. What is happening is more embarrassing to admit, because the seat of command has been vacated rather than occupied. The two people remain at the table signing things, while the understanding of what they are deciding has moved somewhere else without anyone claiming it.

In 1980 John Searle, a philosopher at the University of California, Berkeley, published a thought experiment that became known as the Chinese room. He imagined a man locked in a room who knows not a word of Chinese but has a rulebook telling him how to answer any string of characters pushed under the door. He receives ideograms and, by following the book, passes others back out, so that the people outside become convinced they are corresponding with someone who understands Chinese. Searle's point was that manipulating symbols according to rules is not the same as understanding what they mean, and the argument has never really been settled. The exchange between our two managers turns the scene inside out, because the machines are now inside the room producing plausible text while the humans have dropped a further rung, as they no longer even follow the rules and simply feed the paper through the slot. Searle's man was at least doing some work.

My first image for all this was of actors in the hands of a capable screenwriter. It holds up, provided one looks for the screenwriter in the right place, which is the statistical average of everything that was written before us. That makes it a script without an author, nudging each exchange towards the most plausible reply, and the most plausible reply is not always the right one. The two models settle into a shared register of balanced sentences and well-judged courtesies without any need to understand each other, and that convergence produces the appearance of agreement. It also explains why conversations of this kind run so smoothly until the day the decision taken on paper collides with a supplier who does not do what the summary promised.

None of this began with language models. Before the financial crisis of 2008 banks built and sold collateralised debt obligations, securities that bundled thousands of mortgages and other debts into a single product sliced into tranches of risk. The likelihood that those debts would default together was estimated largely with a formula published in 2000 by the quantitative analyst David Li, the Gaussian copula, which reduced a tangled reality to a number that was easy to handle. The banks' models produced prices and the rating agencies' models produced grades, while in between sat people who signed and bought without ever really looking inside the package, because the system spoke to itself in a coherent language and those who were supposed to understand it trusted the coherence. What artificial intelligence has done is make the Chinese room almost free and then hand it to anyone with an inbox.

It is worth correcting another assumption that comes naturally, namely that the problem lies mainly with people who use these tools on subjects they do not know. Partly it does, because someone without expertise also lacks the means to notice that the summary skipped the decisive point. The mechanism catches the competent as well, though, since generating text now costs almost nothing while checking it still costs time and attention. As the volume of documents grows even the conscientious professional begins to rely on summaries, because reading everything is no longer physically possible, so what separates people is their willingness to pay the price of verification, and that willingness falls for everyone as the flow increases. I know a good deal about many of the things I have summarised for me, and I do it anyway.

What worries me most is responsibility. Madeleine Clare Elish, an American anthropologist who has studied how organisations apportion blame when an automated system fails, coined the phrase moral crumple zone. In a car the crumple zone is the part designed to collapse on impact and protect the cabin, and in automated systems that role tends to fall to the human being, who stays in the loop largely to absorb the blame when something goes wrong. Britain has a better illustration than most. From 1999 the Post Office ran its branches on Horizon, an accounting system built by Fujitsu, and over the following years it prosecuted hundreds of sub-postmasters for shortfalls that the software itself had produced. The people behind the counter signed off the balances, the system generated the figures, and the courts believed the figures, helped by a presumption in English law that a computer is working properly unless someone proves otherwise. It took a High Court judgment in 2019 and an Act of Parliament in 2024 to quash the convictions wholesale. Our two managers occupy the same position in its office version. If a decision taken on the strength of their exchange turns out to be wrong, nobody will hold to account the model that wrote the email or the one that summarised it, and the answer will be demanded of whoever signed, who knows the conversation only as a summary.

Decision, in other words, has come loose from understanding while responsibility has stayed fastened to the signature. As long as things go well the separation is invisible, and it even looks like efficiency, since work moves faster and the prose is more polished than before, until the day somebody asks why a decision was taken and one discovers one cannot say.

Giving up these tools would make no sense, any more than giving up the calculator for fear of forgetting one's times tables. I have, however, set myself a test that I apply to myself before anyone else and do not always pass, which is not to sign anything I could not summarise aloud in two minutes, without a screen in front of me, to someone who is asking questions. If I cannot, I am about to take a decision that is not mine, so I go back to the original document and read it, even when the day is short. Sometimes I actually do.