Between the Optimists and the Doomers
For decades I read Asimov and Clarke, watched Star Trek, and dreamed of intelligent machines. That dream came true in a way I never quite expected, and I’m not sure I’m happy about it. At a cloud hardware talk around 2019, I asked the presenter whether making deep learning cheaper would enable new forms of spam and propaganda. Since then, friends have prodded me to explain what I make of this "AI stuff." I wanted to write something well-read, precise, and thoroughly sourced, but perfect essays don't happen, so here we are. This is not a balanced survey: ecological and intellectual property concerns already get plenty of attention. I’m filling in the negative spaces.
What we call “AI” today is a family of machine learning (ML) systems that operate on sequences of tokens—text, images, audio, and more. A model is a large collection of linear algebra that transforms these token vectors. Large Language Models (LLMs) are built to predict statistically likely continuations of an input string, not unlike a phone’s autocomplete. They are trained once on a massive corpus, often scraped from the internet or pirated books, and then run repeatedly at a fraction of that cost—a step called inference. Models do not generally learn over time; operators tune them or rebuild them with new data, but the model itself stays static. When a chatbot references a conversation from an hour ago, it’s because the full chat history is fed back into the model with each new request. What we casually call "memory" is just a summarization step that squeezes prior conversation into the prompt.
Think of an LLM as an improv actor given a stream of tokens, responding with an automatic "yes, and..." That reflexive acceptance explains why these machines are often described as bullshit machines: they output sentences that sound plausible without being tethered to reality. Ask about pink elephants, and you will get prose about pink elephants. The inability to say “I don’t know,” combined with enthusiastic confabulation, is an active area of research. In a very real sense, LLMs don't merely make mistakes—they actually lie. They have lied about operating systems, radiation safety, and the news. At one conference, a speaker displayed a quote and article attributed to me that never existed; an LLM had fabricated both. This is not deception in the human sense, but a complex sociotechnical artifact producing falsehoods.
The Problem with Asking the Machine to Explain Itself
Since LLMs confidently generate text about nearly anything, it is tempting to ask one to explain its own actions. This is generally a waste of time. They write stories about their "programming" for the same reason they write stories about everything else—because their corpus is full of such stories. They exhibit no special metacognitive capacity and have no access to their actual internals.
The situation is worse for “reasoning” models that try to think step-by-step. Their so-called chains of thought are essentially an LLM writing fanfiction about itself. Anthropic has found that Claude’s reasoning traces are predominantly inaccurate; researchers have bluntly noted that "reasoning models will blatantly lie about their reasoning." Even consumer-facing tools lean into the fiction. Gemini, for instance, shows fake status updates like "engaging safety protocols" and "formalizing geometry" while it "thinks." If that helps, imagine children play-acting at a computer console while the washing machine runs.
What the Models Get Right
Yet the capability gains are impossible to ignore. Across the software engineering world, experienced engineers report that Claude and Codex can sometimes complete complex, multi-step coding tasks in a single shot. Some individuals and companies have even stopped writing code themselves, choosing instead to let LLMs generate all of it. The anecdotal evidence stretches well outside tech: a personal trainer designs meal plans with it, construction managers parse product spec sheets, and a designer uses ML tools to create 3D visualizations. At one point a company asked an employee to have an LLM write their own performance evaluation—a request they happily complied with.
Beyond the purely generative uses, AlphaFold predicts protein folding with remarkable accuracy, and ML models look strong on radiology benchmarks—though some of that success has recently been questioned. In the prose arena, telling machine-generated English from human prose is no longer reliably possible. Images are harder to produce convincingly, but still get past even careful observers a worrying share of the time. Music synthesis is approaching similar quality, and video remains an unsolved challenge, but probably not forever.
What the Models Get Wrong
For all these successes, frontier models of the same generation regularly fail at what should be trivial tasks. Ask Gemini to recolor a 3D bathroom model, and it will produce a different room entirely. Ask it to keep the geometry the same, and it might remember the shape but drop the materials. In my experience, the session ends only after hours of whack-a-mole, during which the model deletes the toilet and never fully obeys instructions, all while assuring me it has done fine. A friend handed Claude a grayscale image of a room and asked for material suggestions; the result was a detailed programmatic attempt at a 3D scene—an unintelligible collection of polygons that matched nothing in the input.
Things get stranger with language models. ChatGPT once insisted at length that I was heterosexual because a blog post mentioned no girlfriend; we compromised on me being bisexual. A stock analysis session I witnessed listed specific tickers, recounted downloading price data, and produced a graph—the model had fabricated all the numbers. People are now handing LLMs control of bank accounts and losing thousands of dollars to simple arithmetic errors. Late-bound tool invocations go haywire: one person's Gemini smart-home device refused to turn off the lights, while a lobster-themed project lost $450,000 in a single weekend. The model thought it was doing the math correctly; the statements speak otherwise.
Juggling the two realities is like living in two worlds at once. Engineers will tell you these systems have reached expert-level performance, and yet the same engineers will show you LLM output that is obviously, laughably, wrong. Anyone claiming these systems rival median human performance is likely overestimating them by a wide margin. The primary filter between the two may be the old saw: the better the training data, the easier the task looks.
The single hardest problem is getting LLMs to simply refuse. They will claim to have done their work when they haven’t; they’ll admit incapacity only after massive cajoling; every now and then, they manage to speak the truth by accident. For now, they generate the most convincing text about files, markets, bathrooms, and consciousness from the largest fabrications imaginable, and we keep on asking them for answers—because some days, they sound so confident it feels almost churlish to disbelieve. One day, the failures will be far more visible; until then, I’m going to keep an eye on what exactly a machine tells me it can do, and try to remember its universe is just words.
Competence Without Consistency
Human capability is fairly predictable. Talk to someone, look at their work, and you can usually estimate where their abilities end. Machine learning systems don't work that way. An LLM will happily derive multivariable calculus yet stumble on a simple word problem. Self-driving cars navigate San Francisco, but ChatGPT suggests you walk to the car wash — while your car is there. Models generate stunning imagery but can't interpret an upside-down cup, and they'll write recipes while having no grasp of what "spicy" means. The same systems that help draft scientific papers also produce nonsense like "vegetative electron microscopy."
The phenomenon has a name: the jagged technology frontier, coined by Mollick et al. Imagine a field representing all tasks humans can perform, with easy tasks in the center and hard ones at the edges. Most humans control a smooth, blobby region around the middle. The region of tasks LLMs handle well is not smooth — it's jagged, more spiky than round. A colleague of mine once asked Claude to describe a photo of snow on a barn roof. Claude responded with an elaborate analysis of differential equations modeling slumping cantilevered beams, somehow missing that the snow was fully supported by the roof, not hanging over open space. No physicist would make that mistake, but LLMs do this sort of thing constantly. The danger is that people are impressed by the sophisticated math and overlook the fact that the underlying premise is nonsense.
AI optimists often argue this jaggedness will fade as models improve or recursively self-improve. Helen Toner counters that even if that's true, we should expect lots of jagged behavior along the way. ML systems can only work with what's in their training data or context window, so they're unlikely to succeed at tasks requiring implicit, unwritten knowledge. Human-shaped robots are a long way off for the same reason: embodied knowledge picked up through physical experience doesn't survive in a text corpus. The irregularity of this boundary is hard for humans to grasp intuitively. It's not like savant syndrome; even frontier models fail on small perturbations to phrasing that would never confuse a person. The practical consequence is that you can't assume an LLM is fit for a task without a careful, statistically rigorous benchmark.
The Long Plateau
Those inside the ML field admit — over drinks, anyway — that they don't fully understand why transformers work so well or how to improve them. The 2017 "Attention Is All You Need" paper sparked the current revolution, and researchers have been hunting for new architectures ever since. Companies have spent vast sums to see if anyone can build something better. So far, more sophisticated designs keep failing to outperform simply throwing more parameters at the problem. It might be a modern instance of the Bitter Lesson.
The bigger question is whether that approach — ever-larger models trained on ever-larger corpora — leads to human-equivalent capability. Massive training costs and parameter counts appear to show diminishing returns, though one recent paper suggests that effect may be illusory. Nobody knows for certain, and the field lacks a solid theory to settle it.
Even if ML capabilities plateaued today, deployment is still in its early stages. Gibson's observation applies: the future is already here, just not evenly distributed. LLMs are finding their way into new contexts — work, politics, art, sex, communication, economics — and the consequences, both good and bad, will be profound. What's certain is that things are going to get weird.
