A faster, sharper Copilot: what changed under the hood

Code completion is still the most-used GitHub Copilot feature, and the custom models behind it just got a significant upgrade. The latest models are now serving suggestions with 20% more accepted-and-retained characters, a 12% higher acceptance rate, 3x higher token-per-second throughput, and 35% lower latency across all editors and environments.

These gains didn't come from squeezing the old approach. The team at GitHub rebuilt their evaluation criteria, retrained the model with a focus on real-world behavior, and made some key architectural decisions along the way.

Why the old metric didn't cut it

The original Copilot was tuned almost exclusively for acceptance rate. That turned out to be a trap: it favored short, simple suggestions that developers would accept quickly but often delete moments later. The feedback from developers was clear — that didn't reflect how code is actually written.

So the team pivoted. They now optimize for a combination of signals that better measure real value delivered at the cursor:

  • 20% higher accepted-and-retained characters means more of each suggestion survives into the final code, instead of getting accepted briefly and then undone.
  • 12% higher acceptance rate reflects that suggestions are more immediately useful, so developers accept them more often.
  • 3x throughput with 35% lower latency makes the system feel snappier. Throughput measures overall capacity; latency measures how fast each individual suggestion arrives after a keystroke.

Evaluating models on three levels

Before any model ships, GitHub runs it through a three-stage evaluation pipeline. Each layer catches different kinds of problems and tests different aspects of the experience.

Offline: execution and LLM-judge

First, the model is tested against internal and public repositories with unit tests and scenario coverage across all major languages. Each test simulates a real coding task, accepts the suggestion, and measures whether the code still builds and passes its tests. This catches functional errors like the one below, where the model produced dataet instead of dataset:

Screenshot of a Python code editor showing a function named resolve_file inside a file called dataset_utilities.py. The function takes two string arguments, dataset and filename, and returns a string. The purpose, according to the docstring, is to resolve a file from a dataset and assert that only one file is found. The code uses os.path and glob to find files. There’s a highlighted line path = os.path.join(dat... with an error under dat, suggesting a variable name typo (dat instead of dataset). Several red underlines indicate syntax or reference errors in the code.

Execution-based tests have a blind spot, though: code that compiles isn't always code developers want. To close that gap, GitHub runs an independent LLM as a judge, scoring completions on three axes:

  • Quality: Syntax validity, duplication or overlap with existing code, and format/style consistency.
  • Relevance: Does the completion stay on-task, avoiding hallucinations and overreach?
  • Helpfulness: Does it reduce manual effort and steer clear of outdated or deprecated APIs?

Pre-production: dogfooding with experts

The next step involves internal developers and external partners testing models side-by-side in real workflows, with the preview model exposed through Copilot's model picker. Structured feedback is collected on readability, trust, and "taste." A distinctive part of this process is working with language-specific experts to refine quality and style preferences — an area where generic execution tests and LLM judges fall short.

Production: A/B testing at scale

Ultimately, the lived developer experience wins. New models are shipped only when statistically significant improvements hold under real workloads, measured via accepted-and-retained characters, acceptance rates, completion-shown rate, time-to-first-token, latency, and more.

Training the next-generation model

The new model is the product of a three-stage training pipeline, each stage addressing a different weakness in generic code generation.

Mid-training on modern code

Before fine-tuning, GitHub builds a code-specific foundational model via mid-training — the stage after base pretraining, before instruction-tuning. This uses a curated, de-duplicated corpus of modern, idiomatic, public, and internal code spanning nearly 10M repositories and 600-plus programming languages.

The goal is to ensure the model knows current language syntax and recent API versions. The training mixes objectives beyond next-token prediction — span infillings and docstring/function pairs — so the model learns structure, naming, and intent rather than just statistical next-token patterns. The result is a foundation model that is code-fluent, style-consistent, and context-aware, ready for the next stage.

Supervised fine-tuning for fill-in-the-middle

General-purpose chat models can generate code from natural language, but they struggle with fill-in-the-middle (FIM) completion. They produce cursor-misaligned inserts, duplicate code from before the cursor, and overwrite code after it.

To fix this, GitHub used synthetic fine-tuning to train the model as a dedicated FIM engine. This improved two areas specifically:

  • Prefix/suffix awareness: Accurate inserts between tokens, mid-line continuations, full-line and multi-line block completions, without trampling the suffix.
  • Formatting fidelity: Respect for local style — indentation, imports, docstrings — and no prefix duplication.

The impact on FIM benchmarks is substantial. On OpenAI's HumanEval Infilling Benchmarks, the latest completions model outperforms GPT-4.1-mini:

A chart showing HumanEval Infilling Benchmarks for two different AI models. These include a custom model from GitHub named Copilot Completions and OpenAI's GPT-4o-mini. The evaluations show superior performance across single line, multi line, random span, and random span light tests for the Copilot Completions model.

Reinforcement learning with guardrails

The final stage uses a custom reinforcement learning algorithm, training the model through rewards and penalties to internalize what makes a suggestion genuinely useful, along the same three axes used in evaluation:

  • Quality: Syntax-valid, compilable code that follows project style.
  • Relevance: On-task suggestions that respect the file's surrounding context and intent.
  • Helpfulness: Suggestions that reduce manual effort and prefer modern APIs.

The goal is completions that are correct at the cursor — not verbose or superficially helpful.

Three lessons from the trenches

Beyond the technical wins, the team came away with three hard-won lessons about training for real developer workflows:

  • Reward carefully: An early RL version over-optimized for longer completions, adding excessive comments — a classic "reward hacking" failure. The fix was comment guardrails that penalize unnecessary commentary and keep completions focused on advancing the task.
  • Metrics matter: Fixating on a single metric like acceptance rate produces experiences that look good in dashboards but don't make developers happy. Multiple metrics with real-world impact must be tracked together.
  • Train for real-world usage: Aligning synthetic fine-tuning data with actual usage patterns helps identify and remove problematic behaviors during training, improving outcomes where it counts.

What's next

GitHub is already expanding into domain-specific slices — game engines, financial systems, ERP workflows — and refining reward functions for build/test success, semantic usefulness, and API modernity. The goal is ever-faster, cheaper, higher-quality completions across every developer environment.

You can try the improved completions today in GitHub Copilot in VS Code.