How to build an enterprise LLM application: Lessons from GitHub Copilot

Scoping an LLM product before writing code

GitHub’s path to shipping Copilot took three years and followed three broad phases: find it, nail it, scale it. The first phase is about picking a problem worth solving. The team’s guidance is to be specific about the user, focus on one pain point, and confirm that an LLM is genuinely the right tool for the job.

For Copilot, the target user was a time-constrained developer, and the initial scope was a single slice of the development lifecycle: writing functions inside the IDE. Earlier ambitions around generating whole commits were shelved because model quality couldn't support them. Sizing the feature down to whole-function suggestions was a pragmatic compromise between ambition and output quality.

A second scoping rule: integrate with the user’s existing workflow rather than inventing a new one. The Copilot team’s yardstick was that "it’s a bug if you have to change the way you code when using GitHub Copilot." Code suggestions had to arrive inside the editor without forcing a new process on developers.

Iterating through rapid experimentation

Building on generative AI is less linear than traditional product development, because models change quickly and users are still learning how to interact with them. Idan Gazit, Senior Director of Research at GitHub Next, frames the design problem as two-sided: "We have to design apps not only for models whose outputs need evaluation by humans, but also for humans who are learning how to interact with AI."

GitHub’s primary iteration vehicle was an A/B experimentation platform, but early prototyping began even simpler. The team stood up a web interface to tinker with foundation models and test ideas internally. That exercise revealed a UX problem: asking developers to switch between their editor and a browser was too much friction. The capability had to be modeless, working in the background inside the IDE.

Internal dogfooding surfaced the niche behavior of developers referencing several open editor tabs at once. That observation led to the "neighboring tabs" technique, where Copilot processes multiple open files rather than just the active one. The change lifted suggestion acceptance rates by 5%.

Two lessons governed how the team handled dead ends:

• Regularly reassess assumptions. GitHub and OpenAI initially believed each programming language needed a dedicated fine-tuned model. Rapid progress in base LLMs invalidated that assumption, as a single model handled many languages well.
• Revisit deprioritized features. A chat interface was tested early, but users expected more from it than the models could deliver. After ChatGPT familiarized users with AI chatbots and LLMs advanced, Copilot Chat became viable.

Testing infrastructure also had to evolve. GitHub outgrew its internal tools and moved to the Microsoft Experimentation Platform to handle feedback and interaction at scale.

Reaching general availability

Getting an LLM application to GA requires focus on quality, usability, responsibility, and go-to-market. GitHub’s experience offers a checklist for each.

Guardrail quality and consistency

LLMs are probabilistic, so experimentation must be statistically grounded. To deliver consistent whole-function suggestions, the Copilot team reduced output randomness by tuning model parameters and caching responses. Caching served a double purpose: it cut variability and improved performance on repeated prompts.

During the technical preview, external feedback caught a regression in suggestion quality. In response, the team added a guardrail metric measuring the ratio of multi-line to single-line suggestions and adjusted the model accordingly. The preview also shaped the product’s KPIs: early developer feedback pointed to code acceptance rate, and later to code retention rate—how much of a suggestion a developer keeps or edits.

User feedback drove cost optimization too. Originally, Copilot eagerly generated ten suggestions at once and displayed them all. Most developers chose the first one, meaning the other nine incurred compute costs and dragged users into an evaluation mindset. As Gazit put it, "It was like paying to calculate the results that appear on the second page of a search engine—and making that second page grab your attention—even though most folks end up using the top result." Ghost text delivered fewer suggestions up front, reducing costs while keeping users in flow.

Technical preview management

GitHub capped the technical preview with a waitlist to keep feedback manageable and ensure a mix of developers at different experience levels. The team engaged with preview users early and often, on the platforms those users preferred, to respond to issues in real time. External testers complemented internal dogfooding with diverse, real-world use cases.

Responsible AI guardrails

Preview feedback made it clear that code security had to be a priority. The team integrated filters to block suggestions containing potential vulnerabilities like SQL injections and hard-coded credentials, and used Azure OpenAI Service’s natural language filters for offensive content.

Community concerns also influenced product decisions. Some developers worried Copilot suggestions might mirror public code. GitHub built a filter to block suggestions matching public source code longer than 150 characters. Later, the code reference tool gave developers links to potentially matching public code, including licensing information, so they could make informed decisions.

Scaling infrastructure and go-to-market

Early experiments ran directly against the OpenAI API. As Copilot grew, GitHub leaned on Microsoft Azure infrastructure to get the reliability and enterprise-grade guardrails it needed for a large user base.

The go-to-market play had two deliberate steps. Prior to the 2021 technical preview, the team demonstrated a prototype to influential community members and GitHub Stars, securing an initial base of support. Then, instead of starting with enterprise sales, GitHub sold licenses directly to individual developers through a free trial and monthly subscription model—based on survey data showing individuals prefer simple, predictable pricing. Individual adoption built a foundation that later drove enterprise uptake.

Scaling a Copilot: What Worked and What Didn't

The path from a promising prototype to a production-grade LLM application is rarely linear. For GitHub Copilot, the journey involved not just improving the underlying model, but fundamentally rethinking how the product was evaluated and refined. The core challenge is that LLM outputs are probabilistic by nature; there is no single "correct" answer. This shifts the engineering focus from pure model accuracy to the reliability and usefulness of the end-user experience.

From Heuristics to A/B Testing with Code Referencing

An early decision was to move beyond simple heuristic evaluations. When the team began developing a code referencing feature, they realized that objective metrics like precision and recall were insufficient on their own. These numbers didn't capture whether a suggestion was genuinely helpful to a developer in a real-world workflow.

To solve this, the engineering team integrated formal A/B testing directly into the development process. The feature was rolled out to a small percentage of users, and the team meticulously compared suggestion acceptance rates. Specifically, they measured how often users accepted a suggestion with a reference to public code versus a baseline without one. This experiment yielded a concrete result: a 5% increase in acceptance rates. This data-driven validation was crucial, providing confidence that the feature added tangible value before a wider rollout.

The implementation also required thoughtful engineering around the core model. The key was to give the model enough context to properly cite its sources. By enriching the prompt with code snippets from a public code index, they enabled the model to generate correct attribution. A security filter was also a necessary component, ensuring that suggestions did not include potentially vulnerable code patterns. This combination of prompt engineering, A/B testing, and tight feedback loops became the template for subsequent enhancements.

The Complexity of Model Evaluation

Along the way, the team learned critical lessons about the impracticalities of certain optimization strategies. Fine-tuning the underlying model was a frequent suggestion, but it presented major logistical hurdles. One significant issue was the risk of the model "forgetting" previous learnings—a phenomenon known as regressive memory loss. This makes every subsequent fine-tuning session a potential regression risk.

Furthermore, standard model quality benchmarks proved to be largely unhelpful. Static benchmarks like HumanEval measure code syntax, but they fail to capture the nuance of a real development environment. A more effective strategy involved using a private, internal dataset, but even that had its limits. The team found that the most reliable modern method for evaluation is the "golden dataset." These are curated collections of problems with verified, high-quality solutions.

Using a golden set allows development teams to harness LLMs as judges in an automated evaluation pipeline. Model A (the candidate) generates a solution, and Model B (the judge) scores it against the golden answer. This turns an otherwise subjective review process into a scalable, objective metric. While not a perfect substitute for human review, it eliminates the bottleneck of manual inspection and provides consistent methodology as the product scales.

Structural Fine-Tuning and Human Oversight

While full model fine-tuning was often impractical for product development, the team did find strategic value in a technique known as Low-Rank Adaptation (LoRA). This form of lightweight fine-tuning was used to adjust the model's temperature sampling parameters—essentially tuning the "creativity" of the model's output. Learning to apply these small, surgical updates was a far more agile way to guide behavior than retraining the entire model.

Ultimately, the most critical component proved to be the human in the loop. The team intentionally built "chokepoints" into the AI runtime, where every prompt and response was logged and made available for human review. This practice of manual "red-teaming" is vital for uncovering failures that automated tests cannot catch. It also forms the backbone of an evaluation pipeline that can track performance against user expectations over time.

This human feedback loop ensures that the "ground truth" for an AI application is not a static benchmark, but the evolving, dynamic needs of its user base. It is what separates a genuinely useful tool from a technical novelty.

Key Takeaways for Enterprise LLM Development

The generative AI landscape is still maturing. While every product will require its own unique approach, the GitHub Copilot experience offers several transferable principles for enterprises looking to build their own LLM-powered applications.

• Focus on a specific problem. Discerning where AI genuinely adds value ensures your application has a more significant impact and reaches the market faster.
• Embed experimentation from day one. Integrating tight feedback loops into the design process is critical when working with probabilistic outputs and users who are still learning to interact with AI.
• Continuously prioritize user needs at scale. Leveraging user feedback as you grow is the only way to ensure your product delivers consistent results and long-term value.

These learnings underscore a broader principle for the industry: building a successful LLM application is less about the raw model and more about the disciplined product and engineering processes surrounding it. The model is a raw material, but the value lies in how you refine, evaluate, and ship it to the user.