Does GitHub Copilot improve code quality? Here’s what the data says

What a controlled trial says about Copilot code

GitHub Copilot has been widely reported to speed up development—earlier research pegged the productivity gain at up to 55%—and surveys have shown that most developers feel more confident and “in the flow” when using it. But objective evidence about code quality has been thinner. A new randomized controlled trial from GitHub’s customer research team set out to measure whether code written with Copilot is actually more functional, readable, reliable, maintainable, concise, and likely to pass review.

The study recruited 202 developers with at least five years of Python experience and randomly split them into two groups: one with GitHub Copilot access, one instructed to avoid all AI tools. Both groups implemented API endpoints for a web server in a fictional restaurant review application. Submissions were assessed with 10 unit tests, then a separate blind review by 25 developers who rated readability, reliability, maintainability, conciseness, and approval-worthiness using a rubric focused on code errors. Across 1,293 reviews, the verdict was consistent: Copilot-assisted code scored better on every measure.

Functionality: more tests passed

Functionality was measured directly by unit test results. Code authored with Copilot passed significantly more tests, and the gap was wide: developers with Copilot access were 53.2% more likely to pass all 10 unit tests in the study (p < 0.01).

Readability and code errors

In the second phase, reviewers conducted line-by-line analysis to flag code errors—poor practices that make code harder to understand, such as inconsistent naming, unclear identifiers, excessive line length or whitespace, missing documentation, repeated code, excessive branching or loop depth, and insufficient separation of functionality.

Copilot-authored code averaged 18.2 lines of code per code error, versus 16.0 for code written without it—a 13.6% improvement (p = 0.002). That compounds in practice, since each error typically triggers a comment or change request during review. Teams working without Copilot could expect up to 13% more such interruptions over time.

Reviewers’ overall ratings told the same story. Code written with Copilot was rated more readable, reliable, maintainable, and concise by 1–3%, with p-values of 0.003, 0.01, 0.041, and 0.002 respectively. The differences were modest but statistically meaningful and contribute to a cleaner codebase over time.

Finally, reviewers were 5% more likely to approve Copilot-authored code (p = 0.014). In practical terms, that means code is ready to merge sooner, shortening the path from implementation to deployment.

Why quality improved

One notable pattern from the data: the Copilot group produced significantly more commits and changed more lines of code, yet average commit size was slightly smaller. The interpretation is that developers using Copilot spent less time fighting to make code functional and instead iterated more—making smaller, more frequent refinements to improve quality. This dovetails with earlier findings that developers feel more confident with Copilot, and suggests the tool empowers them to revise code without fear of introducing errors.

The study authors also offer a hypothesis for why other research has not always found quality gains from Copilot: the issue may not be the tool but whether developers had the incentive or opportunity to focus on code quality in the first place. In this trial, participants were explicitly encouraged to write high-quality code.

Methodology notes

The trial initially recruited 243 developers, all with at least five years of Python experience. After excluding an invalid submission, valid data came from 202 participants: 104 with Copilot and 98 without. Each submission in the review phase was evaluated by at least 10 different reviewers, who were blind to whether Copilot had been used. Study design and statistical analysis were supported by Lizzie Redford, Ph.D., and Sida Peng, Ph.D.