Measuring a Pair Programmer’s Real Impact
When GitHub Copilot launched in technical preview last year, early users reported it quickly became essential to their daily workflows. Now that the tool is generally available, the GitHub research and engineering teams have partnered to test a core hypothesis: does AI-assisted coding actually lead to higher productivity?
The first wave of results comes from a study combining survey responses from more than 2,000 U.S.-based developers with anonymized usage data from the same period. The findings were published in an academic paper presented at PLDI’s MAPS ‘22 conference.
Why Pair Programming Comparisons Fall Short
Working with a colleague typically helps you reach coding goals faster, produce better output, and learn along the way—academic research has supported this for years. But GitHub Copilot is less like a human pair and more like a hybrid: a pair programmer with a calculator attached. It reliably handles the fiddly mechanics of coding, such as closing brackets, but its greater value can be found in larger suggestions.
Developers who used Copilot report missing those flashes of insight when offline—not the bracket-closing precision, but whole patterns, pre-populated boilerplate requiring only light adaptation, and speculative attempts at expressions that offered a starting point. This distinction shapes how productivity gains should be measured.
From Survey to Signals
Productivity in software development is a broad term. It might mean fewer keystrokes, fewer searches across Google and StackOverflow, or staying in a flow state through accurate suggestions on mechanical tasks. Alternatively, it could involve visualising speculative stubs that help unblock stuck developers. Since this territory is uncharted, the research team first asked developers directly, then cross-checked those answers against usage data.
The study set out to answer three questions:
- Do people feel GitHub Copilot makes them more productive?
- Is that feeling reflected in objective usage measurements?
- Which usage measurements best reflect that feeling?
Acceptance Rates Tell the Strongest Story
Survey respondents described multiple ways GitHub Copilot empowers their work. The objective data backed these claims: the number of characters contributed by Copilot, the number of retained suggestions, and the frequency of suggestions in the first place all correlated with reported usefulness and improved productivity.
The strongest connection, however, was found by a simpler calculation: dividing the number of accepted suggestions by the number of shown suggestions. This acceptance rate captures how many of the tool’s suggestions are considered promising enough to adopt.
Developers reporting the highest productivity gains also accept the largest share of shown suggestions.
When users were sorted into quartiles based on how useful they found Copilot, a stark pattern emerged. Those reporting the biggest productivity gains accepted completions at a much higher rate. Interestingly, developers did not mind reworking a suggestion, as long as it provided a suitable starting point.
This aligns with Copilot’s intended role: it is not designed to build software autonomously, but to offer suggestions that help developers stay in the flow—providing parts while leaving assembly and design to the user.
Further Reading
Details on retention rates, language differences, and weekend coding patterns are available in the full academic paper presented at PLDI’s MAPS ‘22. It offers a more systematic look at the data behind these early productivity findings.
