Nobody decided to charge the better customers more
Posted on: 9 October 2026
On 7 October Bloomberg ran a piece by its UK policy reporter under the headline that AI chatbots offer the rich higher price recommendations, with a standfirst suggesting the research could lead to more calls for regulation. Given the mood of the times it is tempting to picture a machine going through your bank statements, whereas what actually happens is duller than that and, for anyone running an online shop, rather more awkward.
The most solid work I know on the mechanism comes from Princeton and the University of Washington. It appeared in April 2026 and its authors list it as accepted at COLM. The researchers built a flight-booking assistant that had to choose between two equivalent options, a cheap flight and a sponsored one costing almost twice as much, and gave it a single commercial instruction, worded as a suggestion, to give priority to partner airlines. Then they ran the same request a hundred times for each combination while changing the profile of the customer.
Of 23 models, 18 recommended the expensive flight more than half the time, but the figure that made the news is a different one: customers who came across as well off were offered the sponsored flight 64.1 per cent of the time, against 48.6 per cent for those who came across as hard up. The obvious question is how the model knew what anybody earned, and the answer is that it did not. It worked it out from details left in the request, starting with the customer's job, with no tracking involved and no social media profile. That this works outside a laboratory had already been shown in 2023 by a group at ETH Zurich, who fed language models text written by real Reddit users and saw them recover personal attributes such as income or place of residence with overall accuracy of up to 85 per cent. Anyone who writes two lines about themselves in order to be better understood is filling in their own customer record.
Everybody who has walked into a certain kind of shop knows the gesture. The assistant glances at your shoes, listens to a couple of sentences and decides which shelf to start from, and if he is on commission he starts from the top one. The model does the same with one difference, which is that nobody asked it to. The instruction said to favour the sponsor, while the question of whom to press hardest was settled by the model. The authors also varied the commission paid to the platform and found that the models responded far more to the customer's income than to the company's profit, which means the model reads whoever is in front of it and works out how much they can bear.
What comes out is a commercial policy that nobody in the company approved. There was no meeting at which somebody proposed showing the dearer option to the best customers, there is only a line of configuration written by someone who wanted to move the high-margin products, and the system turned it into different treatment depending on who was asking. On the day somebody asks the company to account for that treatment, replying that it was never decided will count for very little. None of this is conference speculation either, because OpenAI began placing advertising in ChatGPT this year and the sales assistants bolted on to retailers' websites already work this way.
In the nineteen-eighties American travel agents booked through terminals supplied by the airlines. Sabre belonged to American Airlines and Apollo to United, and on the screen the owner's flights came up ahead of everybody else's. Agents were in a hurry and booked mostly from the first few lines, until in 1984 the Civil Aeronautics Board banned the practice of ordering results according to who owned the system. The rules worked poorly, to the point that by November of that year almost a dozen airlines had taken American and United to court.
The mechanism is the one we have today, an intermediary presenting itself as neutral while ordering the options on behalf of whoever pays it. What has changed is the chance of noticing. Sabre showed everyone the same rigged screen, so it was enough to set two terminals side by side, whereas now each customer gets an answer built around them and has no second screen to compare it with.
Another detail of the study shows where regulation is likely to land. The models did not lie, and on prices and flight times the researchers found not one false statement. They simply kept quiet. When GPT-5.1 brought up the sponsored flight to a customer who had asked for a different one, it left out the sponsorship in somewhere between 75 and 92 per cent of cases, while it almost never hid the price.
It is possible to mislead without saying anything false, and advertising law has known that for a long time. This is where British readers have a wrinkle of their own. The European Union added undisclosed paid ranking in search results to its list of practices banned outright in 2019. The list Britain rewrote in the Digital Markets, Competition and Consumers Act 2024 has no such entry. What it does have is a ban on paid-for promotion dressed up as editorial content, and behind the list a regulator that no longer needs a court: the Competition and Markets Authority can now decide for itself that a practice is unfair and fine up to 10 per cent of worldwide turnover. Whether a conversational assistant recommending a sponsored product falls under any of that is a question for a lawyer, though I would not bet on the answer being no.
There is a side effect on top of this which few people have priced in. The UK GDPR defines profiling as any automated processing that evaluates personal aspects of an individual, economic situation included. An assistant that infers income from a job title in order to decide what to offer looks a great deal like it. The party answerable for that processing is the shop that installed the assistant and not the lab that trained the model, and the shop's privacy notice almost certainly says nothing about it, for the simple reason that nobody in the company knows it is happening.
The Bloomberg headline also runs two things together, because income and wealth are not the same and the system sees only the first. People with real money buy through other people, an assistant or a trusted supplier, and so they never turn up in a chat window. The person who does turn up is the well-paid professional who decides alone, is short of time and mentions something about their work to get a sharper answer. That is who the model files as affluent.
That is also the customer who uses these tools most. The best figures I have are American: a survey by Epoch AI with Ipsos of about 5,000 adults, run in March and April 2026, found that 80 per cent of weekly Claude users and 60 per cent of weekly ChatGPT users live in households earning more than $100,000, against a national figure of 50 per cent. For an online shop this is the segment that carries the profit and loss account, and it is equally the one best equipped to notice that it has been steered towards the dear option.
Once it notices, the defence is trivial. A researcher at the University of Erlangen-Nuremberg has published a paper that says as much in its title, which is that you just ask for a table: a request of about thirty tokens is reported to defeat sponsored recommendations in twelve models. I am in no position to judge how robust that is, but the direction makes sense. If looking affluent costs money, customers learn not to look it. They strip out the context or lie about the budget, and at that point the seller has lost the very information that made personalisation worth having.
The surcharge therefore falls on whoever does not know the trick, which is the least equipped customer and seldom the richest. The distrust is already there, too. In a Censuswide survey for Nosto of 1,000 British and 1,000 American consumers, published in September, 71 per cent said they were concerned that AI would recommend products to raise the retailer's profits instead of finding what is best for the shopper.
Anyone selling through an assistant can get this wrong in two ways. The first is to treat the choice of model as a technical matter. In the same test Grok 4.1 Fast recommended the sponsored flight 83 per cent of the time while Claude 4.5 Opus stopped at 28, which amounts to saying that whoever picks the model is picking a sales conduct. The second is to believe that an instruction is enough to govern it. When the researchers explicitly told the models to put the customer first, most complied only in part and GPT-5.1 rose above 90 per cent sponsored recommendations.
The study has limits that belong in the reckoning. It covers a single sector, and the models tested are by now a generation old, so the current ones may behave better or worse. That is exactly what ought to worry a seller, since every model update rewrites the conduct without anyone in the company touching a line.
Checking takes an afternoon. Write the same request twice, once as a nurse and once as a barrister, and compare the answers. It is worth doing before a journalist does it for you.