In Pursuit of Production Minimalism

The Case for Keeping Production Boring

Kelly Johnson, the engineer behind Lockheed’s Skunk Works and aircraft like the U-2 and SR-71, is credited with coining KISS—“keep it simple, stupid.” The principle is easy to state but hard to follow: design systems to be as simple as possible. Complexity is rarely a conscious goal, but it accrues naturally as features are added and components multiply. KISS treats simplicity as an explicit design objective, yielding systems that are more maintainable, more reliable, and more flexible.

Why Stacks Bloat

Engineers are drawn to new technology. News feeds, meetups, and conferences all bias toward the shiny and the new, while older components that quietly do their job fade into the background. Over time, technologies get added but rarely removed. A production stack that survives long enough becomes a patchwork of accumulated choices, and that sprawl has real costs:

• More parts means more cognitive complexity. When a system becomes hard to understand, developers make changes without seeing all the intertwined concerns, raising the risk of bugs and operational mishaps.
• Every component is a candidate for failure. At sufficient scale, something in the stack will be failing at any given time.
• Engineers become jacks of all trades but masters of none. When a genuinely nasty problem appears, few people have the depth to diagnose it quickly.

Despite knowing this, the urge to expand the toolbox is hard to resist. Persuasion is often part of the job, and it can be turned inward: that new key/value store will fix uptime, that event stream will enable an immutable ledger, that orchestration system will transform deployment. A dispassionate pause would often reveal that the new tool could wait until it’s well vetted and its fit with the existing architecture—including what it will replace—is understood.

Ephemeralization as a Goal

Inventor R. Buckminster Fuller described the idea of ephemeralization in his 1938 book Nine Chains to the Moon:

Do more and more with less and less until eventually you can do everything with nothing.

Projected onto technology, it means building a stack that serves more users and more activity while the people and infrastructure supporting it stay fixed. That requires systems that are more robust, more automatic, and less prone to problems—achieved by resisting the natural growth in complexity.

Heroku pursued this idea explicitly. The customer-facing platform was “user space,” and internal infrastructure was “kernel space.” The ambition was to break up the kernel and move it piece by piece into user space, effectively rebuilding Heroku to run on Heroku. In the ultimate manifestation, the kernel would shrink until it vanished, leaving a single uniform stack. Realistic? Probably not. Useful? Yes—even falling short of an ambitious goal tends to leave you somewhere good.

Minimalism in Practice

Heroku’s history offers concrete examples of this approach:

• The core database tracking apps, users, releases, and configuration was once a special snowflake on a custom AWS instance. It was folded into Heroku Postgres, becoming just another managed node.
• Whole products were retired. The ssl:ip add-on, which ran on dedicated servers for SSL/TLS termination, was end-of-lifed when Amazon offered a better and cheaper option. With SNI support widespread, ssl:endpoint will follow.
• All non-ephemeral data was moved out of Redis, leaving Postgres as the sole persistent data store for internal apps. A bonus: stacks could tolerate a downed Redis and stay online.
• After an overzealous turn toward polyglotism, the last Scala component was retired. Fewer languages made the whole system easier to operate, by more engineers.
• The service handling Heroku orgs began as its own microservice. When microservice expansion proved excessive, it was folded back into the hub alongside a few others.

The effort to tear down old technology was often as large as shipping something new. Heroku marked these projects with “burn parties,” symbolically feeding dead components to a flame.

Guiding Principles

Practicing production minimalism starts with recognizing the problem exists. Once that’s done, the mitigations are fairly direct:

• Retire old technology. When introducing something new, look for roughly equivalent older components to remove. If Kafka is coming in, can Rabbit or NSQ go out?
• Standardize. Aim for one database, one language, one job queue, one web server, one reverse proxy. If not one, then as few as possible.
• Favor simplicity. Keep the total number of moving parts small. This may mean reusing a slightly less ideal tool rather than adopting a better-fitting newcomer.
• Let others vet new tech. Don’t adopt on day one—or even year one. Let others find bugs and stabilize the software, and skip it entirely if it doesn’t gain a real community.
• Avoid custom software. Code you write is code you maintain, forever. Don’t succumb to not-invented-here when a supported public solution fits.
• Use services. Software you install is software you operate, with regular maintenance and troubleshooting. A public service is often the better choice.

New technology is not forbidden, but it should be introduced with rational defensiveness and a critical eye for how it fits an evolving architecture.

Perfection as Subtraction

Antoine de Saint Exupéry, the French aviator and poet, put it this way:

It seems that perfection is reached not when there is nothing left to add, but when there is nothing left to take away.

Most engineering organizations would benefit from architecture that is simpler, more conservative, and more directed. A minimal, stable, nearly perfectly operable stack is what frees a team to push forward on new products and ideas.