Why GitHub Copilot keeps getting better at reading your code
GitHub Copilot's utility hinges on context. Like a human pair programmer, it needs to understand what you're working on to make useful suggestions. The team behind the tool has been investing heavily in improving that contextual understanding, which means figuring out what information to feed the model and how to structure it for the best results.
That work began before Copilot even had a name. When OpenAI released GPT-3 in June 2020, GitHub recognized the potential for a coding-focused product and gave input as OpenAI built Codex, the model that would eventually power Copilot. The tool launched as a technical preview in June 2021 and went generally available in June 2022.
"The hands-on experience we gained from iterating on model and prompt improvements was invaluable," says David Slater, a machine learning engineer at GitHub. That iteration has paid off, says Alice Li, an ML researcher at GitHub: Copilot is often a significant help even when starting new projects from scratch, because it scaffolds a starting point developers can adapt.
The context problem
Developers naturally draw on pull requests, project structure, open issues, and other cues to give their code meaning. A generative AI tool needs to be told which of those cues matter. Today's transformer-based large language models (LLMs) are good at connecting dots between user input and the text they've already generated, but they have limits. Transformers fast enough to power Copilot can process roughly 6,000 characters at a time. That's enough for tasks like code completion, but it means not all of a developer's code fits into the context window.
The resulting challenge is two-fold: determining what data to feed the model, and how to order and format it for the best suggestions. That's the domain of prompt engineering. Algorithms run in the background, selecting relevant snippets and comments from your current file—and increasingly, other sources—then prioritizing and filtering them into a final prompt. Much of this work happens in what GitHub calls a prompt library, where ML experts tune how context is extracted and assembled.
From one file to many: neighboring tabs
Early versions of Copilot could only consider the file you were actively working on. The first major expansion came with neighboring tabs, a technique that lets Copilot process all files open in your IDE, not just the active one. The model combs through the open files, finds code that matches what's around your cursor, and adds those matches to the prompt.
During development, the GitHub Next team ran A/B tests to find the best parameters for matching. Counterintuitively, they found a very low threshold for inclusion produced the best suggestions. Including every bit of context yielded a 5% relative increase in user acceptance of suggestions, and the feature runs in the background without adding latency.
Fill-in-the-Middle looks both ways
Before the Fill-in-the-Middle (FIM) paradigm, Copilot only considered the code before your cursor—the prefix—and ignored the suffix, the code that follows. FIM lets the model know which part of the prompt is the prefix and which is the suffix. That matters because coding isn't linear. Even when you're building a file from a skeleton, you bounce around, and FIM helps Copilot offer better completion for the code that should sit between the prefix and suffix.
A/B testing showed FIM gave a 10% relative boost in performance, meaning developers accepted 10% more completions shown to them, and caching keeps it similarly latency-free.
Moving toward semantic understanding
GitHub is now experimenting with vector databases to create customized experiences for developers working in private repositories or with proprietary code. The approach relies on embeddings—mathematical representations of code as high-dimensional vectors. Because LLMs are trained on vast amounts of both code and natural language, embeddings can capture a snippet's semantics and even its intention, not just its syntax.
The workflow looks like this: algorithms create embeddings for every snippet in a repository (potentially billions), store them in a vector database, then embed snippets from your IDE in real time as you code. The database allows fast searches for approximate matches between your current code and the stored embeddings—unlike hashcodes, which require exact character-by-character matches.
"Embeddings—because they arise from LLMs that were trained on a vast amount of data—develop a sense of semantic closeness between code snippets and natural language prompts," says Alireza Goudarzi, senior ML researcher at GitHub.
The difference between semantic and syntactic similarity is easy to see. Compare "The king moved and captured the pawn" with "The king was crowned in Westminster Abbey." The sentences have the same subject, but mean something entirely different. Add "Both white rooks were still in the game," and the semantic link with the first sentence is clear. Two code snippets can look structurally similar while doing very different things, and vice versa. Embeddings aim to capture that deeper meaning.
This work is still experimental, and GitHub is designing it with enterprise customers in mind—specifically those who would explicitly opt in for a customized experience with private repositories.
Behind the screens
Prior research showed developers code up to 55% faster while using Copilot. Much of what makes that possible is invisible. As you type and edit, Copilot plays catch-up, generating prompts that prioritize relevant information and send it to the model in real time. GitHub's product and R&D teams, including GitHub Next, continue to collaborate with Microsoft Azure AI-Platform to push the contextual understanding further.
