Essential complexity: a bounding argument under pressure
Fred Brooks’ 1986 essay No Silver Bullet remains one of the most cited pieces in software engineering. His central claim: programming tasks contain a core of essential, conceptual complexity that technology cannot eliminate. Using an Amdahl's law-style argument, he reasoned that if essential complexity dominates, then improvements to tools, languages, and processes can only yield a limited productivity gain — at most a factor of two, since he estimated accidental complexity had already been reduced to around half of the total.
The argument rests on a specific equation: task time is the sum of the frequency of each component times the time it takes. If most time is spent on conceptual work — specifying, designing, and testing the abstract construct of a software entity — then attacking only the representational part (writing code, fixing syntax errors) cannot produce large gains. Brooks viewed this conceptual essence as invariant across representations, and the hard part of building software to be the design and testing of this construct, not the labor of expressing it.
There is a fundamental problem with stating such a bound: it requires knowing what problems programmers would attempt if friction were lower. Without foresight into future tools and practices, estimating the accidental complexity of tasks that no one would reasonably try today is guesswork. Brooks’ approach was to survey existing categories of tools and declare them either ineffective or played out. This strategy failed not only for technologies that didn’t exist yet, but for ones he was familiar with. He wrote off formal methods while never mentioning fuzzing, static analysis without full verification, or tools like Valgrind. He dismissed programming languages as essentially finished, just before scripting languages and garbage-collected runtimes transformed mainstream development. Few people would volunteer to build a web application in C today.
His treatment of AI was similarly premature: speech recognition, image recognition, and expert systems supposedly shared nothing in common, yet neural networks now handle all three effectively. Whether they will serve as programming tools remains open, but a central pillar of his argument has crumbled. Expecting Brooks to have predicted deep learning is unreasonable — but that is precisely why dismissing entire fields of future innovation was unjustified.
What the 1980s couldn’t foresee
Brooks also underestimated improvements from practices and the tooling that enables them. Consider testing. Brooks acknowledged its importance but saw little room for serious gains beyond expert systems that help beginners. Modern correctness-minded teams use techniques producing massive speedups compared to 1980s practices — far beyond 2x. Version control is another case. Systems supporting multi-file commits, branches, and automatic merging are recent inventions. During the 1990s, Microsoft's Windows 2000 development needed version control that could handle a 30-million-line repository with branching support; lacking it, the team simulated branches by copying entire source trees and manually merging. Only about 100 changes could be merged per day across roughly 5,000 people — a decade after Brooks wrote, with an order of magnitude improvement available from better tooling, testing practice, and faster machines that would enable those practices.
CI/CD did not even appear on Brooks' radar. Writing in the mid-1990s, he cited an industry practice of nightly builds as potentially too ambitious, noting that a weekly build schedule seemed reasonable to Bell Northern Research. Soon after, Google engineers built their own infrastructure for continuous merging and building against a monorepo at massive scale. A startup founded in 1995 did the same. For large projects, maintaining a clean build through continuous integration over weekly builds is itself a 2x productivity gain — larger than Brooks' bound would allow. Fortunately, engineers at these companies did not accept that a 2x improvement was impossible and built the tools that delivered it.
This pattern — veterans declaring that progress has reached its ceiling — recurs outside programming as well. Cliff Stoll’s dismissal of the internet's potential in 1995 reads the same way: visionaries imagine telecommuting and electronic commerce, but the truth is no online database will replace the daily newspaper and the local mall does more business in an afternoon than the entire Internet handles in a month. Search and replace the subject matter, and Stoll makes Brooks' argument: technologies changed things before, but these particular changes will not happen.
From the 1980s to today’s workloads
Brooks kept writing about software long after he stopped practicing it. His later work drew on examples from earlier essays, and much of his new material came from designing a house — which programmers who generalize to civil engineering tend to do poorly. His specific suggestions, like running software teams as surgical teams on the model of IBM's 1950s practices, had little relevance to an industry that had moved far beyond them.
A concrete look at routine engineering tasks today shows how thoroughly accidental complexity can dominate — even after order-of-magnitude improvements over 1986 equivalents. One everyday task: pulling logs from several hundred thousand machines via scp, then parsing a few terabytes to understand a problem. The script to parallelize downloads with robust error handling takes about a minute to write in Python. Parsing goes to Rust for performance — a few minutes of effort. In 1986, before modern scripting languages existed (Perl arrived in 1987, Perl 5 in 1994), managing a connection pool with good error handling could have taken an order of magnitude longer than the entire task takes now. Finding and compiling relevant libraries, which today is trivial, would have required writing parsing code by hand. Questions that Stack Overflow answers instantly had no equivalent. And even with all these speedups, the task still feels mostly accidental — if centralized logging exported data differently, or log formats were uniform, rg or ag would have been sufficient.
A more involved example: querying two years of metrics data from thousands of instances to generate plots. The query scans on the order of 100 TB, from a dataset that would be 100 PB without pre-aggregated tables maintained specifically to make such analysis feasible. Writing the SQL takes seconds; the query runs in minutes using about one CPU-year of compute. Plotting happens with ggplot code written while queries run. None of this is portable to 1986 — the very idea of a pipeline collecting comprehensive resource-usage data from consumer software was absurd then. Even assuming storage existed, marshaling a CPU-year of computation into a five-minute query would have required roughly ten Cray-2 supercomputers per query, when only 27 Cray-2s were ever built. The tasks Brooks could not conceive were not limited by essential complexity; they were limited by accidental complexity so large that the problems were unapproachable. He explicitly dismissed faster hardware as unhelpful — asking how many MIPS one can fruitfully use — yet faster machines enabled tools like ggplot that would not exist if wringing performance from slow hardware were a prerequisite to writing usable software.
When a task is broken down today, the essential core is vanishingly small. The SQL dialect has arbitrary quirks: an internal assertion failure triggers under specific combinations of numeric_histogram and cross join unnest, requiring mental heuristics to avoid. Partition elimination fails on certain joins, forcing hand-written query compilers to split work across days. Choosing between a fast cluster with low memory limits and a slow one with high limits requires estimating per-node memory usage. Plotting libraries impose their own constraints: when to use size = [number] versus single-pixel scatterplots, minimum opacity for visible gradients, whether to downsample data. Each of these represents dozens of implicit pieces of knowledge held in a programmer's head, and all of them are accidental. For some queries, the work exists only because of some arbitrary quirk — without the quirk, there would be no work at all.
What “essential” really means
Brooks’ claim that most complexity is essential — and that the field had captured most available gains — resembles an assembly language programmer in 1955 declaring assembly as good as any language could be. When he spoke of conceptual complexity, he meant the building blocks he knew in 1986, applied to problems he would recognize. There is no reason to treat that conception as fundamental. The apocryphal “640k should be enough for anybody” quote gets mocked because it lacks imagination; Brooks’ actual statement, across all categories of possible improvement, that most of what can be done has been done, is directly analogous.
The distinction between essential and accidental complexity is not as clean as the argument requires. Brooks admitted that buying a regular expression package could reduce essential complexity by avoiding the concepts of writing a parser. But if a language bundles regexes in its standard library, or ships concurrency primitives, or provides an entire HTTP server, the line blurs. There is no bright line between what a library can add and what a language provides, which means no bright line between attacks on essential versus accidental complexity. Brooks insisted such a line exists, but provided no argument.
His later claim that no one publicly disputed a 9/10 accidental fraction is notable. For day-to-day engineering work, the fraction sits far above nine-tenths. When Brooks revisited his essay in 1995, he restated the strongest form: even shrinking accidental work to zero, if it were under nine-tenths of the total, would not give an order of magnitude gain. By his own logic, if the fraction is unknowable, so is the bound. His writing is charismatic and locally persuasive — each section sounds reasonable in isolation. Pulled together, the pieces contradict: he both claims most complexity is essential and that no single improvement can yield 10x in a decade, though the first claim alone allows for multiple 2x improvements adding up to more than 10x.
Estimating the fraction of complexity that is accidental is effectively impossible, precisely because the problems programmers work on are shaped by the tools available. Brooks' essential complexity is a limit of imagination, not of physics. Programmers who spend their days with microcode or assembly sometimes believe those languages are as productive as anything that exists — this century, not just in 1955. The trap is easy to fall into when the tools you use define the problems you can see. Brooks’ 1986 conception of programming defined his view of what problems were worth solving, and his bound on productivity gains came from that bounded imagination rather than from any fundamental constraint.
