From GPT-3 Experiment to IDE Staple
When GitHub engineers first experimented with OpenAI’s large language models, the experience was revelatory. “As a theoretical AI researcher, my job has been to take apart deep learning models to make sense of them and how they learn, but this was the first time that a model truly astonished me,” says Alireza Goudarzi, a senior machine learning researcher at GitHub. That astonishment quickly translated into action, setting the stage for GitHub Copilot.
The project’s origin traces back to June 2020, when OpenAI released GPT-3. Before that release, discussions about general-purpose code generation within GitHub’s GitHub Next research team routinely hit a dead end. “Every six months or so, someone would ask in our meetings, ‘Should we think about general purpose code generation,’ but the answer was always ‘No, it’s too difficult, the current models just can’t do it,’” recalls Albert Ziegler, a principal machine learning engineer on that team.
GPT-3 changed the calculus. OpenAI provided GitHub with API access, and the GitHub Next team began evaluating the model with crowdsourced, self-contained coding problems. Early on, the model solved about half of them; soon it was solving upwards of 90 percent. The original evaluation method was eventually retired—Ziegler laughs, “The reason we don’t do this anymore is because the models just got too good.”
Initial prototypes explored an AI chatbot that answered coding questions with runnable snippets. That direction was abandoned once the team tried embedding the model inside the IDE. “The moment we did that and saw how well it worked, the whole static question-and-answer modality was forgotten,” Ziegler says. “This new approach was interactive and it was useful in almost every situation.”
Model Evolution and the Codex Era
GitHub’s early collaborations with OpenAI produced a sequence of models: first a Python-only model, then a JavaScript model, followed by a multilingual version. The JavaScript model had distinctive weaknesses that surprised the team. “It came as a surprise to us that the multilingual model could perform so well,” Ziegler notes. “But each time, the models were just getting better and better.”
In 2021, OpenAI released the Codex model, developed in partnership with GitHub. An offshoot of GPT-3, Codex retained natural-language generation but added training on billions of lines of public code, enabling it to produce code suggestions. Businesses could access the model through an API, but GitHub’s internal Model Improvements team still had to refine how Copilot communicated with the underlying LLM to maximize quality.
That team focuses on maximizing completion—the rate at which users accept and keep Copilot’s suggestions—using two primary levers: prompt crafting and fine-tuning.
Prompt Crafting: Building the Right Pseudo-Document
LLMs function fundamentally as document completion models, trained to extend partial documents token by token. “The art of prompt crafting is really all about creating a ‘pseudo-document’ that will lead the model to a completion that benefits the customer,” says John Berryman, a senior machine learning researcher on the Model Improvements team. For code completion, that pseudo-document is the file a user is editing—but GitHub’s engineers discovered the model performs better with additional context from the IDE.
Berryman highlights a pivotal technique: “One of my favorite tricks was when we pulled similar texts in from the user’s neighboring editor tabs. That was a huge lift in our acceptance rate and characters retained.” The model effectively performs the contextual lookup a developer would otherwise do manually. “Where the developer might flip back and forth between tabs to reference code, we just can do that for them, and the completion is exactly what it would be if the user had taken all of the time to look that information up,” Berryman explains.
Fine-Tuning for Individual Codebases
Fine-tuning adapts a pre-trained model to a narrower task by continued training on a smaller, domain-specific dataset. For GitHub Copilot, the goal is to make Codex—which contains upwards of 170 billion parameters—responsive to the nuances of a specific user’s code. “Basically, we’re training the underlying Codex model on a user’s specific codebase to provide more focused, customized completions,” Goudarzi says.
Defining what makes a “good” completion remains a core research challenge. “Our greatest challenge right now is to consider why the user rejects or accepts a suggestion,” says Goudarzi. “There’s no way for us to really troubleshoot in the typical engineering way, but what we can do is figure out how to ask the right questions to get the output we desire.”
Lessons from Iterative Improvement
Not every model upgrade produced immediate, visible gains. “When we received the latest model drops from OpenAI in the past, the improvements were good, but they couldn’t really be felt by the end user,” recalls Johan Rosenkilde, a staff researcher on the GitHub Next team. The third Codex iteration was a different story. Rosenkilde discovered the difference while working on a programming competition in F#: “In the first 24 hours, we evidently had the old model for GitHub Copilot, but then BOOM! Magic happened,” he says. The performance jump was especially noticeable outside the dominant programming languages.
Early Copilot models also had a quirk: suggesting code in the wrong language, particularly near the top of a new file. “You could be working in a C# project, then all of the sudden at the top of a new file, it would suggest Python code,” Rosenkilde explains. Those early models defaulted to popular languages in ambiguous contexts. The team added a headline to the prompt naming the file’s language—a change with little effect deep inside a file but significant one at its start.
Roughly a month later, a stronger improvement emerged: putting the full file path at the top of the document.
“The end of the file name would give away the language in most cases, and in fact the file name could provide crucial, additional information,” Rosenkilde says. A file named connectiondatabase.py hints at database-related code and suggests importing an SQL library; the model could also generate appropriate boilerplate. “That not only solved the language problem, but it also improved the quality and user experience by a surprising margin.”
Another long-discussed capability—cross-file awareness—finally materialized when Ziegler built a component that scans other open editor tabs for text similar to the code at the user’s cursor. “This was a huge boost in code acceptance because suddenly, GitHub Copilot knew about other files,” Rosenkilde says.
Looking Ahead: Copilot X
Three years of hands-on experience with LLMs have convinced GitHub that generative AI’s role in development will keep expanding. GitHub Copilot X, announced in March 2023, extends the AI experience beyond the IDE to additional platform surfaces, including documentation and pull requests. The project illustrates how increasingly capable models, combined with focused engineering on prompting and training, continue to reshape the developer workflow.
