Unlocking the power of unstructured data with RAG

Why unstructured data is hard—and worth the trouble

Most of the data generated around software development doesn't fit neatly into rows and columns. README files, code comments, commit messages, wiki pages, issue descriptions, and review discussions are all written in natural language, following conventions rather than schemas. That's what makes them unstructured: there's no predefined format that dictates what fields they contain or how they're organized.

That lack of structure is exactly what makes these sources valuable—and difficult to analyze. "Without clear boundaries or predefined formats, extracting meaningful information from unstructured data becomes very challenging," says Jessica Guo, a data scientist at GitHub.

Large language models (LLMs) are well-suited to this problem, particularly when the data is text. "When dealing with ambiguous, semi-structured or unstructured data, LLMs dramatically excel at identifying patterns, sentiments, entities, and topics within text data and uncover valuable insights that might otherwise remain hidden," Guo explains.

Where unstructured data lives in a repository

Unstructured data on GitHub takes many forms, each carrying context that structured data can't express:

• README files explain a project's purpose, usage instructions, and contribution guidelines in free-form Markdown.
• Code files follow a language's syntax, but variable names, logic, and structure vary by developer.
• Package documentation includes installation steps, troubleshooting tips, API descriptions, dependencies, and code snippets.
• Code comments clarify intent behind specific blocks in natural language.
• Wiki pages may hold installation instructions, API references, and other project documentation.
• Commit messages describe what changed and why.
• Issue and pull request descriptions capture bug reports, feature requests, and tasks in free text.
• Discussions range from end-user feedback to open-ended technical conversations.
• Review comments reveal reasoning about code quality, decisions, and potential bugs before changes merge.

RAG turns that data into answers

Retrieval-augmented generation (RAG) is a prompting method that pulls additional context into an LLM prompt from a source beyond the model's training data. Developers can then ask questions about their own codebases and get responses grounded in organizational knowledge—without manually assembling context from scattered documents.

RAG systems rely on several retrieval sources:

• Vector databases. AI coding tools generate embeddings from code snippets as you work and store them in a vector database. When you ask a question in GitHub Copilot Chat, your query is converted to an embedding, and the retrieval service finds similar embeddings from the indexed repository. Those embeddings are converted back to text and code, then added to the prompt as context. The pipeline internally uses embeddings, but the underlying data is still unstructured.
• General text search. Under GitHub Copilot Enterprise, repositories can be indexed—code and documentation alike. GitHub Copilot Chat can then retrieve from those indexed sources, including collections of Markdown files across repositories called knowledge bases.
• External or internal search engines. Retrieval can pull from the public web or internal platforms, making text, images, video, and audio available as additional prompt context. For indexed repositories, GitHub's internal search engine finds relevant code or text before the LLM generates a response.

What RAG-powered LLMs actually improve

Development teams have several concrete reasons to apply RAG to unstructured data.

It surfaces organizational best practices and consistency. An LLM can receive context pulled from an organization's repositories and documents, allowing developers to get answers that align with internal conventions without piecing together information manually.

It accelerates codebase comprehension. Understanding code written by someone else is a persistent challenge, complicated by differing coding styles, missing documentation, legacy code, deprecated libraries, and accumulated technical debt. RAG lets developers ask natural-language questions about a specific codebase and receive answers pointing to relevant documentation or existing solutions. That speeds onboarding for junior developers, helps senior developers respond to incidents in unfamiliar services, and makes modernizing legacy code more feasible.

It surfaces product feedback that structured data misses. "Structured data might show a user's decision to upgrade or renew a subscription, or how frequently they use a product or not," says Pam Moriarty, a GitHub data scientist. "While those decisions represent the user's attitude and feelings toward the product, it's not a complete representation. Unstructured data allows for more nuanced and qualitative feedback, making for a more complete picture."

Structured data isn't the problem

None of this is meant to dismiss structured data. Relational databases, Protobuf files, and configuration files follow predefined formats and are relatively straightforward to analyze with SQL and conventional statistical methods. Machine learning is widely and successfully applied to that data.

"Structured data is often numeric, and numbers are simply easier to analyze for patterns than words are," Moriarty says. Methods for analyzing structured data also have the advantage of a longer history and broader familiarity.

The opportunity, Guo argues, lies where those established tools don't reach: "The potential for transformative impact is significantly greater when applied to unstructured data."

The feedback loop of AI-generated context

As developers use AI tools to write more code, they generate more unstructured data—not just source code, but the information used to build, explain, and maintain it. That data holds insights organizations can either leverage or lose.

RAG-powered tools let developers and IT leaders simply ask questions to discover, analyze, and evaluate that wealth of unstructured data, improving productivity, code consistency, and knowledge preservation along the way.

As with any LLM output, verify what you get back. LLMs can produce results beyond what they were explicitly trained to do, so it's important to always evaluate and check their responses.

RAG in Practice: Where Retrieval Beats Fine-Tuning

Retrieval-augmented generation is often described as a way to ground large language models in facts they were never trained on. In practical terms, RAG lets you build a system that answers questions using a specific knowledge base — your company's internal docs, a product catalog, a code repository — by retrieving the relevant pieces and feeding them to the model as context. The model does the talking; the retrieval layer does the remembering.

The key distinction flows from that division of labor. When you need your model to answer questions about data that changes frequently — employee handbooks, API docs, recent releases — RAG keeps the knowledge external and updatable. When you need to change behavior, style, or task competency, fine-tuning adjusts the model's weights directly. In practice, the two are complementary: you can fine-tune for a specific task format, then augment with retrieval to supply the domain facts that change.

As an architecture, RAG splits into two stages. The first is indexing: you chunk your documents, embed the chunks into vectors, and store those vectors in a database that supports similarity search. The second is generation: an incoming query is embedded with the same model, the database returns the nearest chunks, and those chunks are passed to the LLM along with the original prompt. From the model's perspective, the retrieved text simply appears as more prompt context.

To implement this yourself, your stack needs three pieces: a model for embedding, a model for generation, and a toolchain that connects them. Embedding is often handled by a sparse or dense model — bm25 is a common sparse baseline, while dense models include families like e5 and adversarial-trained models. For the generative end, chat-capable instruction-following models work better than base models. Frameworks such as LangChain provide retrieval abstractions, and a few tools, like txtai, bundle embedding and generation into a single package.

The Bottlenecks Nobody Mentions

RAG is conceptually simple, but the engineering reality is harder than the diagrams suggest. The biggest bottleneck is the embedding step, because encoding a large corpus is a one-time, but computationally heavy, operation. Before you run it, though, you need to clean your source files: extraneous encoding characters, HTML formatting, and markup noise will degrade retrieval quality if left in place. Storage is the second concern — in practice, retrieval pipelines for even modest corpora can hit capacity limits on embedding vectors, so plan around your actual vector size and count rather than a generic API.

Then there is the chunking question, which is both problem-specific and decisive. There is no universal "right" split, only what the source documents dictate. The tension is that you want to omit irrelevant context, but if your chunks are too small, critical key-value pairs or connecting sentences can fall outside the window and never reach the model. A sensible approach is a moderate chunk length floor with overlap between segments to keep related phrases together.

Embedding models are trained on what they can encode, which underlines a sharp constraint: if your source content is not in the language or domain the embedding model was trained on, retrieval quality goes down. There are few absolute "best" models per se — raw score rankings often shift with the data, meaning one well-known model might dominate certain datasets while falling short on others. The standard technical recommendation in this space is to run your own offline quality test with your own queries, rather than deferring blindly to preference benchmarks.

Guarding Quality, Not Just Retrieving

Even with a strong stack, latency tends to sit around 2–5 seconds per query, and generated responses carry a stochastic nature. That drives the need for quality checks beyond simple retrieval metrics. Two approaches in common use include input-level filtering — checking that a query on the backend has already been answered with a safe path — and self-reflection loops in which the agent feeds its own output back into the model with relevance feedback. Each approach has parameters to calibrate: insufficient checks can issue risky generations, while excessive constraints on which questions are allowed can kill the usefulness of the feature.

• A common implementation trick is a query paraphrase step that also rewards the memory store when the original question is deemed difficult.
• Contextualized retrieval functions can narrow candidate sets before scoring, helping keep precision high without inflating token count.
• A single general-purpose model can switch behaviors driven by active branches ("if query is legal, do thing A") — a lighter traffic path than switching infrastructure per task.
• Orchestrators should interoperate with a general, integrated action library rather than being bound to niche code specific to a proprietary vector database.

There is a reliability floor to consider. If a retrieval system serves content to a large enterprise user base, the right default is an internal-only, semantically cached fallback with fast response times that does not alter the user-facing answer set. Without that safety net, an untested pipeline launched against live traffic risks exposing empty result sets to a catastrophic degree at scale.