On the Diverse And Fantastical Shapes of Testing

The Shape of Testing Arguments

Arguments about how teams should distribute their automated testing efforts have resurfaced again, most notably in a recent post by Tim Bray making the case for taking automated testing seriously. The discussion inevitably returns to the familiar images used to represent testing strategy: the Test Pyramid, and its newer challengers, the honeycomb and the trophy. These misshapen blobs are all attempts to express how much effort should go into different kinds of tests, particularly the balance between unit tests and broader integration tests.

The pyramid's classic message is that most testing should happen at the unit level, with progressively less effort spent on broader tests. The honeycomb and trophy shapes argue the opposite: that teams should write relatively few unit tests and concentrate instead on integration tests. But this debate is often muddier than it needs to be, because the participants rarely agree on what the terms "unit test" and "integration test" actually mean.

What "Unit Test" Originally Meant

The terms "unit test" and "integration test" have never been precisely defined, even by the loose standards of software terminology. In the original, waterfall-era usage, they described an organizational workflow. A team would work for months on a large hunk of code, thinking of it as a conceptual unit that could be developed in relative isolation. Once complete, the unit was handed to a separate testing team, which tested it on its own. Only after that passed would the unit be integrated with neighboring code, at which point integration tests would verify that the combined system worked.

The key distinction was structural: unit tests verified your code in isolation, while integration tests verified how your code worked with code developed by others. The introduction of the Xunit testing tools and Extreme Programming brought a different meaning. Kent Beck used "unit test" to mean anything written by programmers as part of their daily work, as opposed to tests written by a separate QA team. At the C3 project, a "unit test" would typically focus on a single class's behavior, but the test fixture would create that object with all its real dependencies, allowing collaborating objects to execute as well. The assumption was simply that the other code was working correctly, often because it had its own tests.

Beck himself drew the distinction this way:

Programmers write unit tests so that their confidence in the operation of the program can become part of the program itself. Customers write functional tests so that their confidence in the operation of the program can become part of the program too.

-- Kent Beck (Extreme Programming Explained, 1st Edition)

This usage drew criticism. One testing expert objected strenuously to Beck's definition, and when asked for his own, replied that his training course covered 24 different definitions of "unit test." Early on, some proposed alternatives like "microtest" or "programmer test" to avoid the confusion.

Solitary Versus Sociable Tests

A major point of contention was the role of dependent objects. If you test an order object that collaborates with a customer object, a bug in the customer object could make your order test fail. That concern led to a different testing style where every collaborator is replaced with a mock, stub, or other test double. These approaches have since been described as solitary (mockist) and sociable (classic). Jay Fields coined the terms "solitary" and "sociable" to capture exactly this difference.

This distinction drove a wedge between two schools of XP unit testing practice. Classic XP unit testing follows the sociable approach, exercising real collaborators. Mockist style favors solitary tests, isolating the unit under test from its dependencies.

Why the Geometry Doesn't Matter

With that background, the pyramid-versus-honeycomb debate starts to dissolve. When advocates of the honeycomb or trophy criticize excessive mocking, they are usually using "unit test" to mean solitary unit tests specifically. And what they call an "integration test" often sounds like what others would call a sociable unit test—a test that exercises a class together with its real collaborators. Any conventional description of the test pyramid, on the other hand, treats unit tests as encompassing both sociable and solitary styles.

The semantic fog thickens further when "integration test" itself enters the picture, a term with even less consistent usage than "unit test." The lesson is straightforward: when someone starts talking about test categories, ask what they mean. Terms like "unit test" and "integration test" are not shared vocabulary, and the shapes drawn to represent testing strategy inherit all that ambiguity.

The more substantive concern, however, is that the shape of the distribution is the wrong thing to argue about in the first place. As Justin Searls put it:

People love debating what percentage of which type of tests to write, but it's a distraction. Nearly zero teams write expressive tests that establish clear boundaries, run quickly & reliably, and only fail for useful reasons. Focus on that instead.

-- Justin Searls

The proportions matter far less than whether the tests a team writes are well-designed, fast, deterministic, and meaningful. A team arguing over pyramid versus honeycomb geometry may simply be talking past each other about what the base layers even contain. Resolving that vocabulary first is the only way the conversation can be productive at all.