Why model selection needs more than a benchmark score
The roster of available AI models keeps growing, with proprietary foundation models from OpenAI, Google, and Anthropic competing against smaller open options from Meta and Mistral. It is tempting to jump to whatever was released most recently, but newer does not automatically mean better for a given task. GitHub recently added support for Claude 3.5 Sonnet, Gemini 1.5 Pro, and OpenAI's o1-preview and o1-mini to GitHub Copilot, and made that call only after running the models through a rigorous evaluation process focused on performance, quality, and safety.
Here is a look at how the Copilot team approaches those evaluations, with an emphasis on the offline tests they run before anything touches a production environment.
Automated tests vs. manual review
AI model evaluation is the process of assessing performance, quality, and safety. Automated tests allow evaluation at scale across a large number of tasks, but they require objective criteria for judging output. Manual testing captures subjective quality and accuracy but is time-intensive. Combining both gives a picture of subjective response quality that is not based purely on anecdotal evidence, without requiring the manual review of thousands of individual responses.
Copilot runs more than 4,000 offline tests, most as part of an automated CI pipeline, plus live internal evaluations similar to canary testing where a group of employees switches to a new model. This applies to all major product changes, not just model additions.
Assessing real-world coding ability
A core part of the offline evaluation measures a model's ability to evaluate and modify codebases. The team maintains a collection of around 100 containerized repositories that have passed a battery of CI tests. They are then modified to fail those tests, and the model under evaluation must edit the codebase to make the suite pass again.
The team generates as many scenarios as possible across languages and frameworks, and is constantly expanding the test set, including testing against multiple versions of supported languages. It is time-consuming work, but it remains the best available method for judging a model's code quality.
Using an LLM to judge an LLM
Copilot Chat goes beyond code generation, answering questions about code and suggesting problem-solving approaches. To evaluate chat quality, the team uses a collection of more than 1,000 technical questions. Simple true-or-false questions are graded automatically. For more complex answers, another LLM—one with known good performance—checks the responses so the team can scale beyond what manual testing would allow.
That evaluating LLM is used to keep assessments consistent and is routinely audited to ensure its judgements align with human reviewers and stay reliable across many requests. Production models face these same chat tests daily. If degradation shows up, the team audits to identify why performance slipped, and sometimes modifies prompts or other components to bring quality back to expected levels.
Testing without touching product code
A key part of the infrastructure is a proxy server that the code completion feature uses. Swapping which API the proxy calls enables rapid testing of a new model without any client-side changes or product code modifications.
The evaluation platform is built primarily on GitHub Actions, with results flowing through systems like Apache Kafka and Microsoft Azure and explored through custom dashboards.
Making the adoption call
Deciding what to do with the collected data is not always cut-and-dried. A model that scores poorly across all metrics is easy to reject. The harder cases involve tradeoffs, such as a model that shows a substantial bump in acceptance rates but also increased latency.
Metrics can be inversely related; higher latency can produce higher acceptance rates simply because users see fewer suggestions during a session. The final call is guided by the goal of building the most capable, responsible AI coding assistant possible, and the quality data from these evaluation procedures underpins those decisions. The same approach can be adapted to your own use cases.
GitHub Models lets developers use, compare, experiment, and build with AI models from providers including OpenAI, Cohere, Microsoft, and Mistral.
