Copilot Chat’s effect on code quality, measured
GitHub has released new research into how GitHub Copilot Chat affects the code developers write and review. The study’s headline finding: developers reported better code quality across the board when authoring and reviewing with Copilot Chat enabled—even though none of the participants had used the feature before.
Developer surveys backed up the measurement:
- 85% of developers felt more confident in their code quality when authoring code with GitHub Copilot and GitHub Copilot Chat.
- Code reviews were more actionable and completed 15% faster with GitHub Copilot Chat.
- 88% of developers reported maintaining flow state with GitHub Copilot Chat because they felt more focused, less frustrated, and enjoyed coding more.
GitHub’s prior research showed that Copilot users code 55% faster. That finding addressed speed; this study asks whether the AI-assisted work holds up on quality metrics.
What counts as “good” code
To make code quality measurable, GitHub used a rubric of five metrics that align with academic and industry standards. Participants used these to differentiate strong code from code that slows them down.
Readable
Does the code follow the language’s idioms and naming patterns? Code that is difficult to read makes it harder to maintain, improve, and document.
Reusable
Is the code written so that it can be reused? Code reuse is a cornerstone of developer collaboration; it saves time and energy, breaks down silos, and creates consistency.
Concise
Does the code adhere to DRY (don’t repeat yourself)? Less repetitive code is easier to read, understand, and build upon. Complex code can lead to bugs that are tough to remediate.
Maintainable
Is the code written in a way that makes its functionality clear and relevant to the problem at hand? Maintainable code minimizes dependencies and supports code search and reuse.
Resilient
Does the code anticipate and handle errors? Resilient code maintains its functionality, or at least experiences minimal disruption, if errors occur.
Copilot use correlates with higher perceived quality
85% of developers felt more confident in code quality when authoring with Copilot and Copilot Chat
The code was so clean, I could just look at the code, know what was going on, and bring it over fairly easily to my codebase. So, I felt the code quality was very clean and easy to understand. When I was doing the pull request review using Copilot, I saw how good it was at generating proper error handling code.
Developers reported that coding was easier, more error-free, more readable, more reusable, more concise, more maintainable, and more resilient with GitHub Copilot and Copilot Chat than without.
Code reviews were more actionable and 15% faster—for first-time users
GitHub’s data shows it’s possible to get both speed and quality. Developers found that using Copilot Chat for code review improved code quality compared to review without it. Reviews were 15% faster, a higher percentage of comments were accepted, and almost 70% of participants accepted comments from reviewers using Copilot Chat.
One of the more actionable comments I received was related to conceptual understanding of my code–I want other developers to be able to understand what I'm doing, so this was good feedback.
The authors note the potential impact on collaboration and larger engineering teams: less time on pull requests and code reviews frees developers for higher-priority changes, and better quality code from the start reduces the need for rollbacks and additional testing.
88% of developers reported maintaining flow state with Copilot Chat
Last year’s research found that 60–75% of developers using GitHub Copilot reported feeling more fulfilled in their job, less frustrated when coding, and better able to focus on satisfying work. In this year’s study, 88% of participants felt similarly less frustrated and more focused. The ability to stay in the IDE means less time searching and more time in a focused state.
Because I’ve had industry experience for over a decade, I usually prefer coding with a notepad or a simplistic platform and Copilot Chat was a really useful tool for checking basic things without me needing to go to Google or Stack Overflow. And Copilot has a pretty neat syntax edition.
Study design
The study simulated the full authoring workflow in a controlled setting: author code, have it reviewed, then review and incorporate the review suggestions. Each participant was asked to do all three phases.
GitHub recruited 36 participants with five to 10 years of software development experience. Participants authored and reviewed code both with and without Copilot Chat. They had some experience with Copilot but none with Copilot Chat. Participants were asked to author API endpoints for an HTTP service that creates, reads, and deletes objects, and were randomly assigned whether to use Copilot Chat for each portion. Before using Copilot Chat, they watched a brief video on its functionality. They created one pull request for the create endpoint and another for the read and delete portion.
After authoring, participants compared how Copilot Chat impacted quality: whether the task was easier, the code had fewer errors, and was more readable, reusable, concise, maintainable, and resilient.
Then, participants reviewed pull requests authored by other participants, blind to whether Copilot was used. They provided improvement suggestions and later rated the review process with and without Copilot Chat, plus the quality of the code using the rubric above. Finally, original authors reviewed the comments on their pull requests—again blind to which reviewer used Copilot Chat—to judge which comments were helpful and actionable.
Bottom line
The study suggests GitHub Copilot Chat improves the quality of code produced and the speed of reviewing it. As AI-generated code becomes more common, these results offer an early look at how those tools affect the codebase itself, not just developer velocity.
