Judging Software Teams by Results, Not Deliverables
Anyone who has spent time in software development eventually asks: what makes a team effective? For individuals, managers, and industry observers alike, the answer is rarely straightforward because there is no clean way to measure a development team’s productivity. A central point of confusion is the distinction between output and outcome.
Consider a team building software for an online retailer. Output measures might count the number of features shipped in a quarter or track technical improvements such as reduced page load time. Outcome measures, by contrast, look at whether the work actually moved the business: increased sales revenue, or fewer support calls related to the product. The latter is what ultimately matters. Functionality that does not help users accomplish their goals is wasted effort — worse, it bloats the codebase and makes future changes harder. A team that ships less but delivers greater utility may be improving more than one that produces a high volume of features.
The Measurement Objection
A common argument against outcome-based assessments is that outcomes are harder to measure repeatably than output. That claim is difficult to defend. Measuring software output is notoriously unreliable. Lines of code are a poor metric even without considering how easily they are gamed. Function Points and Story Points suffer from weak replicability — different people assign different scores to the same work. By comparison, financial outcomes are measured routinely and with reasonable precision in most organizations.
To be fair, some outcomes are genuinely tricky to observe, such as customer satisfaction. But none of them are inherently more difficult to gauge than software functionality itself.
Choosing the Right Outcomes
Calling something an outcome does not automatically make it the correct focus. Selecting what to observe takes skill. One useful framing comes from Seiden, who defines an outcome as a change in user, employee, or customer behavior that produces a beneficial result for the organization. Seiden distinguishes these from “impacts,” which are broader organizational effects and typically harder to attribute. Work by Highsmith, Luu, and Robinson in their development of the EDGE operating model adds another suggestion: weigh outcomes tied to customer value (such as dishwasher reliability) more heavily than those tied only to business value (such as warranty repair costs).
The Apportionment Problem
A more legitimate concern is that outcomes are difficult to apportion to a specific development team. Imagine a customer team using software to monitor product quality in its supply chain. Final consumer rejects might decline, but how much of that improvement is due to the software versus the quality control procedures designed by analysts versus a separate initiative to improve raw materials?
This attribution problem becomes a major hurdle when the goal is comparing teams. Suppose an organization wants to judge whether adopting Clojure makes teams more effective. Outcome-based comparisons will be clouded by factors outside each team’s control. Even the inverse case causes trouble: developers may do excellent work and deliver a valuable quality-tracking system, but if the accompanying quality control procedures are flawed, rejects will not drop and the whole initiative may be declared a failure despite the engineering team having done its part well.
Still, the difficulty of apportionment is not a reason to track the wrong thing. “You get what you measure” understates the case — it is closer to “you get what you try to measure.” If success is defined only by output, everyone naturally focuses on increasing output. Even when a team’s contribution to an outcome is hard to isolate, making people think about outcomes and how to improve them is more valuable than any effort to compare teams on productivity measures that incentivize the wrong behavior.
Further Reading and Background
Outcome-oriented thinking has deep roots in the agile community. A preference for outcomes over features was suggested as a fifth agile value by Daniel Terhorst-North as far back as 2006. The idea has since appeared in several Thoughtworks-related books, including Lean Enterprise, EDGE, and the Digital Transformation Game Plan.
For a practical framework on identifying outcomes, Seiden’s book is informed by work with non-profits — organizations that face similarly difficult questions about evaluating the impact of their efforts. The EDGE model, developed by colleagues, treats outcome focus as a core component of its philosophy for transforming businesses to operate in the digital world. Teams that want to pursue this direction can also look to the concept of outcome-oriented team structures. The broader problem of measuring developer productivity itself is addressed separately in discussions of whether productivity can be meaningfully measured at all.
