Why Human-Sourced Data Beats System Data for Productivity
Engineering leaders frequently respond to pressure for productivity visibility by reaching for the same handful of system-derived metrics: lead time, deployment frequency, pull requests per engineer. Dashboards get built, reviews get scheduled, and the doubts quickly surface. Is a two-day lead time actually a problem, or does the benchmark simply not fit the team’s context? Does lower deployment frequency indicate an opportunity or just a different domain reality?
That story repeats across the industry, including at some of the largest technology companies. The failure mode is consistent: system metrics answer what happened but not why, and they are blind to friction that never surfaces in a version control log. A complementary approach focuses on developers themselves as the data source. When organizations treat qualitative measurement — data provided by humans — as a first-class input, they gain visibility that system telemetry cannot produce alone.
A critical distinction up front: “developer productivity” here means the degree to which developers can perform their work without friction, not an individual performance score. Because the term is easily misread, many organizations we work with prefer “developer experience,” which carries less individual-consequence baggage for engineers.
Defining a Qualitative Metric
We define a qualitative metric pragmatically: a measurement built from human-provided data. There is no authoritative definition of “qualitative” to borrow from the social sciences. A 2019 paper in Qualitative Sociology titled “What is Qualitative in Qualitative Research” notes that researchers across social science fields act as though they know what qualitative research is, yet cannot formulate a coherent definition.
Two common alternative definitions fail in practice. Treating “qualitative” as a measure of quality versus quantity collapses because a metric is, by definition, a quantity — quality is commonly captured on ordinal scales converted to numerical scores. The second argument — that sentiment analysis output is quantitative because it produces numbers — misses the point. The metric’s origin determines its nature. Even when analysis yields numeric output, if the raw material is human-provided data, it is a quantity produced qualitatively.
We also caution against the language sometimes used for these measurements. “Soft metric” wrongly implies weaker data compared with system-derived “hard metrics.” “Subjective metric” is equally misleading, since human-sourced responses can yield both objective facts (such as a self-reported frequency) and subjective judgments.
| Type | Definition | Example |
|---|---|---|
| Attitudinal metrics | Subjective feelings, opinions, or attitudes toward a specific subject. | How satisfied are you with your IDE, on a scale of 1–10? |
| Behavioral metrics | Objective facts or events pertaining to an individual's work experience. | How long does it take for you to deploy a change to production? |
A Real-World Implementation: Peloton
Peloton, the American technology company, illustrates how this approach takes shape in practice. Its developer productivity measurement strategy centers on a semi-annual developer experience survey. The effort is led by the Tech Enablement & Developer Experience team, housed in Product Operations.
Thansha Sadacharam, head of tech learning and insights at Peloton, explains the rationale: engineers “aren’t robots, they’re humans,” and basic numbers do not convey the full story. For Peloton, a comprehensive survey became the mechanism to understand the complete developer experience.
Each survey cycle reaches a random sample of roughly half of Peloton’s developers. That cadence means an individual developer participates in at most one survey per year — keeping the time burden low while remaining statistically representative. The Tech Enablement & Developer Experience team owns analysis and distributes findings to leaders across the organization. For a deeper look at their process, an interview with Sadacharam covers the survey design in detail.
Why Qualitative Metrics Deserve a Seat at the Table
Engineering leaders often treat qualitative metrics with skepticism, preferring system telemetry because it feels objective and familiar. But this bias is less about the data itself than about how surveys have historically been designed. The reality is that qualitative and quantitative measures answer different questions, and treating them as competing options ignores what each brings to the table.
In fact, the underlying objections to qualitative data usually stem from a few common misconceptions rather than from inherent flaws in the approach.
Misconception: Surveys Only Capture Subjective Opinions
Many workplace surveys do focus on feelings and attitudes, which leads to the assumption that this is all a survey can do. That is incorrect. Surveys are fully capable of collecting objective, factual information about events and processes.
Two classic examples of objective survey questions are:
- How long does it take to go from code committed to code successfully running in production?
- How often does your organization deploy code to production or release it to end users?
Google’s DevOps Research and Assessment (DORA) program is built entirely on this kind of survey data, proving that structured self-reporting can produce reliable operational benchmarks.
Misconception: Qualitative Data Is Unreliable
Poorly written surveys do produce unreliable data. However, the problem there is survey design, not the survey method itself. Well-constructed surveys with carefully validated questions yield accurate and dependable results.
There is also a concern that respondents might lie. In practice, this is rare when surveys are framed as a diagnostic tool to identify and remove bottlenecks, since developers have no incentive to game a system that is meant to help them. It is also worth remembering that system metrics are far from perfect either. Measuring CI build times from raw pipeline data, for instance, often requires significant cleanup to account for background jobs or parallel execution before it reflects reality.
Two Distinct Categories of Qualitative Metrics
Qualitative measurement is not a single monolithic tool. It splits into two types that serve very different purposes:
- Attitudinal metrics capture subjective feelings, opinions, or attitudes. An example is the numeric response to "How satisfied are you with your IDE, on a scale of 1-10?"
- Behavioral metrics capture objective facts about an individual’s work experience, such as the answer to "How long does it take for you to deploy a change to production?"
It is common for practitioners to overlook behavioral measures when thinking about qualitative methods, even though they appear frequently in software research. DORA’s annual benchmarks for lead time, deployment frequency, and change fail rate are a prominent case in point. They are gathered entirely through survey questions with discrete answer ranges rather than from instrumentation:
- Lead time: How long does it take from code committed to code running in production? Ranges span from "less than one hour" to "more than six months."
- Deployment frequency: How often does your organization deploy to production? Options run from "fewer than once per six months" to "on demand (multiple deploys per day)."
- Change fail percentage: What percentage of production changes result in degraded service requiring remediation? Bands range from 0–15% up to 76–100%.
- Time to restore: How long does it generally take to restore service after an incident? Answers use the same time bands as lead time.
One powerful aspect of qualitative measurement is that attitudinal and behavioral data can be collected simultaneously. Knowing that a release process is objectively fast is only half the picture. Attitudinal data can reveal whether that process is smooth and painless for developers, which is a separate concern with real implications for burnout and retention.
A medical analogy is apt. A doctor can measure your blood pressure, temperature, and heart rate and conclude that nothing is wrong, but if you say you feel sick, the objective numbers are incomplete. The patient’s self-report is not a substitute for the vitals; it is essential diagnostic information in its own right.
Concrete Benefits of Qualitative Metrics
Beyond avoiding the uneasy feeling developers get from being monitored via their Git or Jira activity, qualitative methods offer distinct practical advantages.
Measuring What Systems Cannot See
Pipeline metrics track commits, deployments, and tickets, but a large share of development work never shows up in those systems. Designing features, navigating a codebase, maintaining flow, and shepherding a project’s technical direction are all productivity-relevant activities that are invisible to instrumentation.
Technical debt is a good illustration. Google ran a study looking for a valid objective metric for technical debt and analyzed 117 candidate indicators; none proved reliable. The core difficulty is that technical debt is inherently a comparison between a system’s current state and an imagined ideal state. That judgment cannot be made by a machine, which means human assessment is not merely helpful—it is unavoidable.
Visibility Across Teams Without Heavy Instrumentation
Capturing metrics like time-to-restore from system data is feasible at small scale, but it gets complicated quickly. Normalizing metrics across teams with different ticketing workflows is difficult. Larger organizations often need to consolidate data across multiple planning systems and deployment pipelines to get an end-to-end view, an effort that can easily take a year. In these cases, asking developers directly is a faster and often more accurate route to a baseline number.
Context That Keeps Quantitative Metrics Honest
Numbers look clean until you try to act on them. Take code review: optimizing for speed alone is tempting, since waiting on reviews causes context switching. But a fast review process is not inherently a good one. Rushed reviews, missed security risks, and skipped expert input are all possible side effects. A holistic view requires measuring outcomes like code quality, knowledge sharing among teammates, and whether reviewers are being blocked. Those outcomes are only assessable through qualitative methods.
The same logic applies to developer onboarding. Counting time to first commit misses whether new hires feel safe asking questions, are collaborating with peers, or are actually contributing their ideas. Self-reported measures are the only practical way to capture these signals.
Designing Surveys That Yield Reliable Data
Writing effective survey questions is harder than most tech practitioners expect. It draws on established disciplines like psychometrics and industrial psychology, and organizations should bring in or develop that expertise rather than improvising. Common failure modes include poorly worded questions and altered wording between survey waves that invalidates comparisons.
Some practical guardrails:
- Each question should ask only one thing, with careful wording.
- Don't change wording between surveys if you intend to compare results; any change means you may no longer be measuring the same thing.
- If wording changes are unavoidable, subject them to rigorous statistical testing before drawing conclusions.
In survey terms, a "good" survey is one that is valid (it measures the construct you intend) and reliable (it produces consistent results from your population over time). One useful model for engineers: treat the survey response process as an algorithm executed by a human mind. When a respondent sees a question, a series of mental steps produces an answer. That process can be decomposed into stages, as described in The Psychology of Survey Response (2012):
| Component | Specific Processes |
|---|---|
| Comprehension |
Attend to questions and instructions Represent logical form of question Identify question focus (information sought) Link key terms to relevant concepts |
| Retrieval |
Generate retrieval strategy and cues Retrieve specific, generic memories Fill in missing details |
| Judgment |
Assess completeness and relevance of memories Draw inferences based on accessibility Integrate material retrieved Make estimate based on partial retrieval |
| Response |
Map Judgement onto response category Edit response |
Inspecting each stage helps refine question design to reduce error. Developing survey items requires the same rigor as designing software — iterative testing and analysis.
Beyond item design, success depends on participation, analysis, and follow-through. A few practices make the difference:
- Segment results by team and persona. Company-wide averages hide problems that affect specific groups (for example, mobile developers). Since developer experience is highly contextual, results should be broken down by team, role, tenure, and seniority.
- Treat free-text comments as primary data. Open-ended responses capture friction and workflow detail that scales can't, often surface unanticipated problem areas, and frequently include actionable improvement ideas — including who to follow up with.
- Compare against benchmarks. Some sentiment signals skew negative by default (e.g., code quality). Knowing whether your developers are more frustrated than peers, teams, or industry counterparts gives leaders a more actionable signal than raw scores alone.
- Use transactional surveys at key touchpoints. Platform teams can prompt for feedback while a developer is creating a service in an internal portal. This yields higher-frequency, more granular feedback than periodic surveys alone.
- Guard against survey fatigue. Low participation over time usually stems from a lack of visible follow-up. Leaders must act on results and communicate what changed. Quarterly or semi-annual cadence works for most, but some teams succeed with lightweight surveys integrated into retrospectives.
A starting template
A minimal question set is enough to begin. It can be loaded into any survey tool, or started directly from a ready-to-go Google Forms template. Note that as measurement programs mature, surveys often grow; Shopify's developer survey runs 20 minutes, and Google's exceeds 30.
The starter set includes questions on overall ease of work, lead time for changes, self-assessed productivity frequency, and agreement statements, plus an open textarea for additional feedback.
| Strongly disagree | Disagree | Neutral | Agree | Strongly agree | |
|---|---|---|---|---|---|
| My team follows development best practices | □ | □ | □ | □ | □ |
| I have enough time for deep work. | □ | □ | □ | □ | □ |
| I am satisfied with the amount of automated test coverage in my project. | □ | □ | □ | □ | □ |
| It's easy for me to deploy to production. | □ | □ | □ | □ | □ |
| I'm satisfied with the quality of our CI/CD tooling. | □ | □ | □ | □ | □ |
| My team's codebase is easy for me to contribute to. | □ | □ | □ | □ | □ |
| The amount of technical debt on my team is appropriate based on our goals. | □ | □ | □ | □ | □ |
| Specifications are continuously revisited and reprioritized according to user signals. | □ | □ | □ | □ | □ |
Score multiple-choice items using either mean or top-box scoring. Mean scores assign 1–5 to each option and average them; top-box scores report the percentage choosing one of the two most favorable options. Open responses should be reviewed — if the volume is high, LLM tools can help extract themes and suggestions. Close the loop: share findings with respondents so their time feels worthwhile.
Combining Qualitative and Quantitative Signals
Qualitative and quantitative metrics cover different gaps. Surveys give a holistic, contextual view that includes subjective experience. Quantitative metrics excel where humans fall short:
- Precision. Developers can say whether builds take minutes or hours, but not report millisecond-level durations. When fine granularity matters, logs are necessary.
- Continuity. Surveys run at most once or twice per quarter; systematic data collection enables continuous measurement.
A mixed-methods approach starts with surveys to build context, then uses quantitative metrics to drill into specific suspected problem areas. Engineering leaders favor this order: qualitative gives breadth across the whole process, while quantitative typically covers only a narrower slice of the path to production. Google's engineering researchers advise the same sequence. As researcher Ciera Jaspan puts it, logs data alone won't tell you whether values are good or bad — “the time to make a change… is useless by itself. You don't know, is this a good thing? Is it a bad thing? Do we have a problem?”
The workflow in practice:
- Start with qualitative data to identify the top opportunities.
- For a target area, drill in using quantitative metrics.
- Track progress over time with both quantitative and qualitative measures.
Organizations spend heavily on people who can observe and detect problems that log-based metrics can't register. Survey data is the method for converting those human observations into measurable signals.
