Support Tickets, Chatbots and the Coming Drudgery
The imminent wave of machine learning deployments is shaping up to be less about productivity and more about friction. Companies are pushing customers toward LLM-powered chat and voice systems for support, deploying models in fraud detection and claims review, and layering "agentic" commerce on top of traditional dark patterns. The common thread is accountability—or the lack of it.
For support staff, the goal has always been to minimize cost per ticket. Offshoring, rigid scripts, and bureaucracy all serve to distance representatives from the knowledge and authority needed to actually fix broken systems. The cynical endgame is to wear customers down until they give up. LLMs are the next step in that progression: endlessly patient, empathetic-sounding, and utterly wedded to the script.
Because LLMs are unpredictable and susceptible to injection attacks, they will be given severely limited powers, especially the ability to act outside system constraints. For routine questions, this is fine. For anyone whose account is tangled in a bureaucratic mess, it will be a new circle of hell. As with current phone trees, access to human beings will be tiered by economic value—premium customers will get capable humans, while everyone else talks to a model.
Arguing with the Machine
The deployment of LLMs will not stop at customer service. Pricing, insurance underwriting, medical necessity reviews, and traffic enforcement are all fertile ground. These systems don't need to be accurate to be deployed; they just need to be cost-effective. Hertz's model can misprice rentals as long as overall profits rise.
The response will be a new kind of arms race. We already see it in algorithmic airline pricing, where consumers hop between browsers and devices to find a better fare. Expect similar tactics with grocery cameras and insurance chatbots: dressing down to signal you aren't wealthy, or phrasing requests in specific ways to game a medical necessity review. There will be listicles of "five phrases to say in meetings to improve your Workday AI TeamScore™."
The good news is that LLMs cut both ways. People are already using them to contest denied insurance claims, and tools to negotiate with corporate chatbots are emerging. Distributed boycotts—where many personal models burn through a company's token budget—are plausible. The asymmetry is real, though. Companies amortize LLM risk across millions of transactions; an individual staking a single, emotionally significant claim is far less willing to accept that the model might make things worse rather than better.
Responsibility in a Diffuse Supply Chain
A COMPUTER CAN NEVER BE HELD ACCOUNTABLE
THEREFORE A COMPUTER MUST NEVER MAKE A MANAGEMENT DECISION
—IBM internal training, 1979
The failure modes are visible today. Angela Lipps was jailed for four months after a facial-recognition system misidentified her for a crime in a state she'd never visited. Taki Allen was swarmed by police when a surveillance camera flagged a bag of chips as a gun. These are not solely ML failures—they are failures of sociotechnical systems. Police should have recognized the absurdity; a school resource officer chose to escalate despite a canceled alert.
The issue is that a billion-parameter model is illegible. Its decisions cannot be meaningfully explained, and its self-reports are unreliable. Human reviewers are primed to trust statistical output, which can encode social biases—inferring Black borrowers are less credit-worthy, withholding medical care from women, or misidentifying Black faces. The veneer of objectivity makes biased decisions harder to challenge.
When a claim is denied by a model purchased from one vendor, trained on records from another, and classified by thousands of subcontractors coordinated by a third, responsibility diffuses. Every involved party—raters, engineers, executives—can truthfully say no one understood the full system. This parallels the Boeing 737 MAX debacle, where mechanical and bureaucratic complexity obscured causation. The difference is that air crashes warrant an NTSB investigation; getting banned from Hinge or losing your home to a false arrest does not.
Individual accountability—the understanding, acknowledgement, and correction of faults—will be harder to achieve as ML systems lengthen the supply chain of decisions. Someone will be denied a necessary test, run over by an automated car that keeps driving, or fired by a manager using Copilot to stack-rank reviews. The question is not whether these systems will be deployed; it is whether anyone will be able to fix what they break.
When Robots Do the Shopping
“Agentic commerce” is the idea of handing a credit card to a large language model, giving it internet access, and telling it to buy something. Optimists see this as the end of subscriptions and the beginning of relentless price competition, as customer LLMs re-shop insurance and delivery services constantly.
If LLMs become the primary purchasers, the economics of advertising collapse. Why market to humans when machines decide? Consultancies anticipate declining ad revenue and suggest mitigations like placing ads inside chatbots or having business LLMs negotiate with consumer LLMs. But this ignores a bigger dynamic: once LLMs control buying decisions, there is enormous incentive to manipulate the LLMs themselves.
That points toward ads engineered for specific models, SEO advice tuned to prompt particular responses in ChatGPT 8.3, and pages of invisible text that steer an agent toward a certain retailer. This is just the existing search-engine optimization arms race, but aimed at a new audience. LLM vendors will find themselves brokering commerce between producers and consumers, able to charge both ends—perhaps taking money to nudge users toward particular cloud APIs, and taking more money to remove those nudges.
Some foresee LLM-to-LLM negotiation over pricing and perks. The reality could be less polite. Two agents haggling over a burrito might flood each other with dark patterns, spoofed browsers, and prompt-injection tricks before settling on a price. Sellers with any scruple will keep their models restrained; the long tail of sketchy vendors will not. Expect agents to be talked into buying seeds they never wanted, or millions of people waking up to fraudulent purchases because their Claude fell for a leetspeak jailbreak.
The idea that this commerce will move to low-fee cryptocurrency also seems optimistic. LLMs are chaotic; they will buy the wrong things, and someone will have to adjudicate returns and chargebacks. That means explaining to Visa that Gemini was told to be frugal but let United’s LLM upsell it to first class, then pasting the negotiation transcript into a support ticket for another LLM to judge. The need for sophisticated fraud protection will raise fees, spreading the added financial risk of unpredictable agents across everyone’s transactions.
This does not sound pleasant for ordinary people. Nobody wants to maintain a fake social profile to convince Costco’s LLM they deserve a discount, or run their own agent to haggle with another agent. Being annoying has not stopped autoplaying video or loyalty programs, though, so there is little reason to think the industry will restrain itself. Everyone may end up paying AI companies to manage the drudgery those companies created.
It is tempting to think this problem will be self-limiting: if too many vendors deploy LLM nonsense, customers will go elsewhere. But adoption pressures cut both ways. If enough companies use LLMs, customers will need them too; if enough customers use them, vendors will be forced to join. The likely outcome is an equilibrium where everyone gets by, accepts some bias and fraud, and commercial transactions become increasingly complex and difficult to unravel when they go wrong—with exceptions perhaps made for the wealthy, who are few enough that annoying them is not worth it.
