Customizing and fine-tuning LLMs: What you need to know

From Search Bar to Context Bar

Every developer knows the friction of dropping what they're doing to look up a syntax detail or debugging approach. Those interruptions break concentration and fragment the workday. But when AI coding assistance lives directly in the IDE, the equation changes: researchers have found that integrated assistants reduce context switching and preserve mental energy.

More importantly, an in-IDE assistant has something a general-purpose chatbot lacks: awareness of your actual codebase. Instead of asking how to reverse a string in Python, developers can ask questions grounded in their own work:

With access to open files, project structure, and recent changes, these assistants can answer questions like:

• Can you suggest a better way to structure my code for scalability?
• Can you help me debug this function? It's not returning the expected results.
• Can you help me understand this piece of code in this repository?

The result is answers that fit the specific repository, conventions, and challenges at hand. But how does that customization actually work under the hood? We spoke with Alireza Goudarzi, a senior machine learning researcher at GitHub, to break it down.

How LLMs Use Context to Generate Answers

At the core of today's generative AI tools are large language models (LLMs), sets of algorithms trained on massive amounts of human language and code. Modern LLMs use the transformer architecture, which gives them a notable strength: connecting the dots between data and generating responses that are contextually relevant.

That contextual relevance depends on input. A coding assistant in the IDE can be directed to use data from open files or code surrounding the cursor to understand the current line and suggest completions. As a chatbot, it can pull from indexed repositories, custom knowledge bases, the developer's prompt, and search engine integrations to offer guidance.

All of that data—code, queries, and other input—passes through a context window present in every transformer-based LLM. The context window's size dictates how much data the model can process at once. It's not infinite, so prompt engineers must decide what data to feed the model and in what order to get the most useful response.

Customization vs. Training

It's important to distinguish between training and customizing an LLM. Training means building the scaffold and neural networks for deep learning from the ground up. Customization, the focus here, means adapting a pre-trained model to specific tasks, like generating information about a particular repository or translating an organization's legacy code into a modern language.

Three main approaches are used to customize LLMs: retrieval-augmented generation (RAG), in-context learning, and fine-tuning.

Retrieval-Augmented Generation (RAG)

RAG relies on embeddings to retrieve information from vector databases. A vector database is significant because it transforms source code into retrievable data while preserving semantic complexity. This allows an assistant using RAG to generate relevant answers about a private repository, proprietary code, or—when integrations are configured—external search results.

Here's how it works in practice:

• Vector databases store embeddings of your repository's code and documentation. Embeddings make that content intelligible to an LLM, similar to how code is compiled into binary.
• As developers type in an IDE, algorithms transform code snippets into embeddings and make approximate matches against embeddings already in the vector database.
• When a user asks a natural-language question, that query is also converted into embeddings and matched against stored vectors.

The approximate matching is what matters. If algorithms only looked for exact matches, there would often be nothing to include as context. Embeddings give the LLM better semantic understanding, so it finds code or documents that are relevant to the query and uses them to form a useful answer.

In-Context Learning

Often compared to prompt engineering, in-context learning is giving the model explicit instructions or examples at inference time. The idea is to provide enough guidance in the request itself that the model understands what you need and generates a relevant result.

This approach can take several shapes, including:

• Adding a high-level statement of the goal in your prompt
• Rephrasing your queries
• Providing examples in the prompt

Fine-Tuning

Fine-tuning produces the most customized outcome: an LLM that excels at a highly specific task. There are two primary methods for fine-tuning: supervised learning and reinforcement learning from human feedback (RLHF).

Supervised learning requires a predefined correct answer. The model's output is evaluated against that expected result, and its parameters are adjusted until it generates the right output. The challenge is practical: even smaller models can have well over 300 million parameters to adjust, and the method requires time-intensive labeling for every training sample.

RLHF either relies on direct human feedback or uses a separate reward model trained to predict whether a user would accept or reject an output. The reward model's learnings are passed to the LLM, which adjusts output to increase user acceptance. The advantage is that RLHF doesn't depend on supervised labels, opening the door to subtler criteria. For example, the model might learn that a response with an acceptable with an 80% probability of user acceptance is fine to generate.

Tailoring GitHub Copilot to your organization

GitHub Copilot's context awareness has expanded significantly since its initial release, when it could only consider the file open in your IDE. It now also draws from neighboring tabs, and with GitHub Copilot Enterprise, organizations can index entire repositories in vector databases for improved semantic search. This gives developers a customized experience grounded in the organization's own codebase.

Organizations can also create knowledge bases from Markdown files across collections of repositories, feeding Copilot with unstructured data that provides additional context beyond what's in a database or spreadsheet. In practice, this delivers several operational benefits:

• Faster onboarding and code comprehension. Developers at any seniority level can ask Copilot for code summaries, suggestions, and explanations of code behavior. This streamlines feature implementation, issue resolution, and code modernization efforts.
• Organizational knowledge becomes accessible to everyone. By indexing repositories and documentation, Copilot generates suggestions aligned with internal best practices. Non-developers and cross-functional team members can also use natural language prompts in Copilot Chat on GitHub.com to get answers about relevant documentation or existing solutions, making captured data more accessible and improving collaboration.
• Streamlined pull request workflows. Copilot Enterprise can generate pull request summaries directly in GitHub.com, helping developers communicate changes clearly while saving time. Reviewers can use Copilot to understand proposed changes quickly and focus their effort on providing substantive feedback.

Practical strategies for LLM customization

A customized LLM lets organizations extract value from all available data, even when it's unstructured text like documentation. This approach surfaces insights, supports data-driven decisions, and makes enterprise information easier to locate.

Choose an AI solution with RAG

Much of an organization's data lives outside structured storage — in text like code documentation. GitHub Copilot Enterprise implements retrieval-augmented generation by giving Copilot Chat access to selected repositories and Markdown knowledge bases, enabling it to answer questions grounded in the organization's own content.

Adopt innersource practices

GitHub's chief operating officer Kyle Daigle has highlighted the value of bringing open source communication practices internally, a process called innersource. A central tenant: write things down and make them easily discoverable. More documentation means more context for an AI tool to draw from when generating tailored solutions — a foundation that makes AI adoption effective.

Developers can interact with Copilot Chat in their preferred natural language, from German to Telugu. This means more documentation improves global collaboration, allowing developers to work on shared code while understanding and improving it in their own language.

Daigle recommends these steps for innersource adoption:

1. When you hear something valuable, record it and make it discoverable — many video and productivity tools now offer AI-powered summaries and action items.
2. Share useful solutions with the wider organization so others benefit from what your team has learned.
3. Offer feedback on publicly shared information, focusing critique on the work itself, not the person.
4. When requesting changes to a project or document, explain the reasoning behind the request.

Adding these notes to relevant GitHub repositories in Markdown format makes them accessible for AI tools.

Expanding results through search engine integration

Transformer-based LLMs possess strong semantic understanding even without embeddings or high-dimensional vectors, thanks to training on large volumes of unlabeled natural language and publicly available source code using self-supervised learning. When integrated with a search engine via its API, an LLM can retrieve current, relevant results beyond its training data's cutoff.

Consider a developer asking about the most recent Java version. If the LLM's training data predates that release, a web search integration lets the coding tool retrieve the answer from current sources rather than providing outdated information. An AI assistant that draws from both customized organizational data and publicly available data gives employees comprehensive, context-aware guidance.

What's ahead for customized coding assistants

Gartner projects that 50% of enterprise software engineers will use machine-learning powered coding tools by 2027. Developers already leverage AI assistants for complex code translation, building test coverage, solving novel problems, and answering coding questions without leaving their IDEs. With customization, these tools can also surface solutions aligned with proprietary or private source code and enhance communication between technical and non-technical team members.

The trajectory points toward even more organization-specific customization. Future capabilities could allow fine-tuning a coding assistant to improve code completion suggestions, and integrating AI tools into custom tech stacks could provide richer context from services and data sources extending beyond GitHub itself.