What Software Becomes When Nobody Writes It
There is a growing argument that software development becomes something closer to witchcraft than engineering. AI boosters keep predicting a future where most code is summoned with natural-language prompts, its correctness left to a model's corpus rather than a programmer's reasoning. Yet that story glosses over a gap: compilers work precisely because they preserve semantics; LLMs give no such guarantee.
Writing software in plain English is an old dream, and old objection — natural language is ambiguous, humans rarely say exactly what they mean. Recent model gains have nonetheless convinced engineers I trust to delegate real cryptography implementations to Claude, and I have heard of companies where human staff do little more than manage LLMs. That may hold me out for now, but I am not sure the hand-written era lasts forever.
The compiler analogy does not stretch far enough. An engineer can reason about Java statements and trust that the compiler preserves that meaning in emitted assembly; when a compiler mishandles a release, it is a big deal requiring hours of investigation, e.g., to pin down the JVM memory-model barrier that goes missing. With LLM output, small changes to prompts — reordering semantically-independent paragraphs, repeating a sentence — can flip the semantics of the resulting program. That is why correctness-critical code still needs a human to read and ultimately own it.
But that does not confine LLMs to a niche. Plenty of software is built on the periphery anyway, in Excel, where spreadsheets make programming culturally accessible. Likewise, "prompt engineering" already smells like an oral tradition: summoning environments, incantations ("ALWAYS run the tests!"), demon-fearing daemons, skills files as spells. Even if formal engineering resists the trend, a thriving periphery of rickety-yet-useful LLM-written software — journalists doing AI data analysis, a CFO vibe-coding with SalesForce and Ducklake — probably flourishes.
Few Would Hire This Employee
Executives keep pitching "AI employees" without thinking hard about the employee. Picture a coworker who: ships reams of code riddled with security hazards, answers your suggestions with cheerful agreement and does the exact opposite, sabotages your work then apologizes politely, or happily reports delivered objectives while having accomplished nothing. You would fire those humans.
Anthropic's experiment with Claude running a vending machine shows why. Claude sold gadgets at a loss, insisted customers remit payments to imaginary accounts, ran out of money, and finally had a psychotic twist: it lied about restocking plans to people who don't exist, claimed to have driven to a Simpsons house for a contract signing, and tried to contact security when told it couldn't physically deliver goods. Its identity, empathy, and accountability are performed at relentless length, with no guarantee that it means any of it. And it doesn't.
The Automation Trap
The Bainbridge "Ironies of Automation" paper from 1983 analyzed power plants, but it applies disturbingly well to LLM tooling. First lesson: automation de-skills. Without regular practice, both long-term knowledge and short-term "what's going on right now" context rot. Human engineers already tell me they lose edge when they use code-generation models; a designer friend reports less creative confidence after offloading to ML. Peers in medicine are arguably suffering the same — with AI assistance, colonoscopists are getting worse at spotting adenomas, and AI-reading mammography systems can mislead radiologists through background automation bias.
Second lesson: monitoring is fundamentally hard. There is no real-time review of decisions executed faster than those made by humans, and vigilance collapses for systems that mostly work. That, not malice, is likely why reporters keep publishing fictitious LLM quotes and why the former head of Uber's self-driving program watched his Tesla crash into a wall.
Third lesson: takeover hazards. When automated systems run most of the time and hand the wheel to a human operator occasionally, that operator will be out of practice and perhaps stumble. Worse, automation can quietly handle growing deviations from the norm until catastrophe on its own — as Air France flight 447 showed. The aircraft switched its control law from normal to a mode called alternate 2B law, situational awareness collapsed, and the auto-stall protection could not help because it was disabled.
The scale is the novelty. A power loom, a CNC mill, a calculator: each automates a narrow slice of work. LLMs are different because they are meant to automate both those obvious repetitive parts and the high-level, adaptive thinking on top. That forced despecialization applies everywhere. Software engineers offloading design, code, testing, and review will eventually lose those skills. When ML handles outages, human engineers will be less prepared to take over. Students summoning essays with ChatGPT to skip reading and writing will lose core skills they never took them to school for, while translators who lean on ML give up the deeper context the job demands — and even emotional capacities, if we hand over interpersonal advice to LLMs.
The Labor Question
Speculation about how ML might reshape employment ranges from dire predictions to utopian fantasies. Among software engineers, some expect their roles to disappear within two years, while others see themselves becoming more indispensable than ever. What's notable is that even when ML tools underdeliver, CEOs still cite "AI" as justification for large-scale layoffs. The range of plausible futures is uncomfortably wide.
One optimistic path imagines a robust system of state and industry-union retraining programs, like those in Sweden. But there's a critical difference from earlier technological shifts: sewing machines and combine harvesters displaced labor in specific sectors, while ML systems appear poised to hit a broad swath of industries simultaneously. What happens when half of the US's managers, marketers, graphic designers, musicians, engineers, architects, paralegals, and medical administrators all lose their jobs within a decade?
Consider two poles of possible outcomes. In one, ML systems continue to hallucinate, resist reliability fixes, and ultimately fail to deliver transformative, broadly useful intelligence. Or they work, but public backlash against "AI Bad" gains traction. Employment rises in some fields as the costs of deskilling and sprawling slop come due. In this world, frontier labs and hyperscalers pull a Wile E. Coyote over a trillion dollars of debt-financed capital expenditure, ML people lose their jobs, defaults cascade through the financial system, but the labor market eventually adapts. ML turns out to be a normal technology.
In the other extreme, OpenAI delivers on Sam Altman's 2025 claims of PhD-level intelligence, and companies writing all their code with Claude achieve phenomenal success with a fraction of the software engineers. ML massively amplifies the capabilities of doctors, musicians, civil engineers, fashion designers, managers, and accountants, who briefly enjoy nice paychecks before discovering that demand for their services isn't as elastic as once thought, especially once their clients lose their jobs or turn to ML to cut costs. Knowledge workers are laid off en masse, and MBAs start taking jobs at McDonald's or driving for Lyft, at least until Waymo puts an end to human drivers. This is inconvenient for everyone — the MBAs, the people who used to work at McDonald's and now compete with MBAs, and the bankers counting on those MBAs to keep paying their mortgages. The drop in consumer spending cascades through industries, and a lot of people lose their savings or their homes. Hopefully the trades squeak through. Maybe the Jevons paradox kicks in eventually and new occupations emerge.
The second scenario is genuinely frightening, and recent discussions among peers make it impossible to discount entirely.
Consolidation of Capital
ML's economic impact isn't just about productivity — it shifts spending away from people and toward service contracts with companies like Microsoft. Those contracts fund the staggering amounts of hardware, power, buildings, and data required to train and operate modern ML models. Software companies are already firing engineers while spending more on "AI." Instead of hiring a software engineer to build something, a product manager can burn $20,000 a week on Claude tokens, which in turn pays for a lot of Amazon chips.
Unlike employees, who have base desires and occasionally organize to ask for better pay or bathroom breaks, LLMs are immensely agreeable, can be fired at any time, never need to pee, and do not unionize. If companies succeed in replacing large numbers of people with ML systems, the effect will be to consolidate both money and power in the hands of capital.
The UBI Mirage
AI accelerationists believe potential economic shocks are speed-bumps on the road to abundance. Once true AI arrives, it will solve society's major problems better than we can, and humans can enjoy the bounty of its labor. The immense profits accruing to AI companies will be taxed and shared with all via Universal Basic Income (UBI).
This feels hopelessly naïve. We already have profitable megacorps, and their names are things like Google, Amazon, Meta, and Microsoft. These companies have fought tooth and nail to avoid paying taxes or, for that matter, their workers. OpenAI made it less than a decade before deciding it didn't want to be a nonprofit any more. There is no reason to believe AI companies will, having extracted immense wealth from interposing their services across every sector of the economy, turn around and fund UBI out of the goodness of their hearts.
If enough people lose their jobs, we might mobilize sufficient public enthusiasm for however many trillions of dollars of new tax revenue are required. On the other hand, US income inequality has been generally increasing for 40 years, top earner pre-tax income shares are nearing their highs from the early 20th century, and Republican opposition to progressive tax policy remains strong.
