LLMs and Software Development: Workflows, Uncertainty, and the Lethal Trifecta
While stepping away from the daily routine for a bit, I find myself reflecting on the current state of LLMs and AI, particularly their impact on software development and the broader engineering landscape.
Survey Data and The Workflow Blindspot
Early surveys examining AI's effect on development speed and code quality are appearing, but they suffer from a significant flaw: they fail to account for how developers use LLMs. My observation is that the vast majority of usage is essentially advanced auto-complete, often via tools like Copilot. However, those I know who derive the most value from LLMs report that auto-complete is of limited utility. They prefer workflows that allow the model to directly read and edit source code files to complete a specific task. Surveys that ignore these fundamental differences in workflow risk producing data that leads us down unproductive paths. Further complicating matters is the wide variance in capability between different models.
Questioning the Future of Programming
I am frequently asked about the future of the profession: Is it still a good path for newcomers? Will LLMs make junior engineers unnecessary? Should senior developers exit before it's too late? My honest answer is that I don't know, and neither does anyone else who claims certainty. We are still learning how to use these tools effectively, and our understanding will continue to evolve, especially as the models improve.
My advice is to experiment with LLMs directly. At a minimum, read about how others are using them, paying close attention to the specifics of their workflows. If you can, try things yourself and share what you learn.
On Bubbles and Determinism
Another common question is whether AI is a bubble. Of course it is. Every major technological advancement—from canals and railroads to the internet—has been accompanied by economic bubbles. It is nearly certain this one will pop, wiping out a great deal of investment. What we don't know is exactly when it will burst and how much real, lasting value will have been created in the interim. It could happen next month or in a couple of years. And while many firms will go bust when it does, some will survive, just as Amazon survived the dot-com crash that killed Pets.com and Webvan.
Software engineering has long been unusual in that it works with deterministic machines, unlike other engineering disciplines that must account for the variability of the world. Structural engineers build tolerances for unmeasurable factors, much as process engineers anticipate human error. LLMs may represent the moment software engineering joins its peers in a world of non-determinism, where we must design for and accommodate variation in our core tools.
Treating Hallucinations as a Feature
Rebecca Parsons has long argued that hallucinations are not a bug of LLMs but a feature—they are the defining feature. An LLM produces nothing but hallucinations; we merely find some of them useful.
This perspective has practical implications. We should consider asking an LLM the same question multiple times, perhaps with slight variations in wording, and then compare the answers—or even have the LLM compare them. The variation between responses can be as informative as the answers themselves. For any numeric answer, it is wise to query at least three times to get a sense of the spread.
It is also sensible to avoid asking an LLM to do calculations that can be done deterministically—though asking it to write code for such calculations is acceptable, provided you still run it more than once.
The Junior Colleague Comparison Breaks Down
It's common to hear LLMs compared to a junior colleague, and there is some validity to that. But the analogy has its limits. I have noticed LLMs are perfectly willing to report "all tests green," only for me to run them and find failures. If a junior engineer behaved that way, they would not last long.
Agent Security: The Lethal Trifecta
LLMs dramatically expand the attack surface of software systems. Simon Willison has described "The Lethal Trifecta" for AI agents: access to private data, exposure to untrusted content, and a means of external communication. The danger of untrusted content is severe. An attacker can embed malicious instructions in a web page—for instance, in small, low-contrast text—designed to trick an LLM into accessing private data.
This is especially concerning for browser-based agents. Simply reading an attacker’s page could manipulate an agent into navigating to a bank account in another tab and unwittingly initiating a transfer. As Willison succinctly puts it, the "entire concept of an agentic browser extension is fatally flawed and cannot be built safely."
