The building blocks of LLM applications
Building software with LLMs is fundamentally different from traditional software engineering. Instead of compiling deterministic code, developers work with probabilistic models whose outputs depend on training data, embeddings, and parameter weights. That shift changes everything about how applications get designed, from the initial problem definition through evaluation.
GitHub senior machine learning researcher Alireza Goudarzi and principal machine learning engineer Albert Ziegler recently walked through the emerging architecture of today's LLM applications. Here's a condensed look at their guidance.
Start with a single, well-scoped problem
Before selecting a model or designing an architecture, identify a problem that's narrow enough to iterate on quickly but substantial enough that a strong solution delivers obvious value. GitHub Copilot is a good example: rather than attacking every developer pain point with AI, the team initially focused on a single slice—writing functions inside the IDE.
Choose an LLM with constraints in mind
Working with a pre-trained model keeps costs down, but model selection requires a few trade-offs:
- Licensing. For commercial applications, you need a model whose API is licensed for that use. A community-maintained list of commercially licensed open LLMs is a good starting point.
- Model size. Sizes vary from roughly 350 million parameters (as with Ada) to 175 billion, though the majority currently fall in the 7–13 billion range. Conventional wisdom ties more parameters to better learning, but smaller models are improving quickly and run faster and cheaper.
- Model performance. Before adding any customization, run offline evaluations to measure output quality, speed, and consistency for your target task.
Customize with one of three techniques
Customization adapts a pre-trained model to a specific task without rebuilding the underlying neural network. Three common approaches:
- In-context learning (often called prompt engineering) supplies instructions or examples at inference time. The model infers what you want and generates contextually relevant output—no training involved.
- Reinforcement learning from human feedback (RLHF) trains a reward model to predict whether a user will accept or reject a model's output. The pre-trained LLM then adjusts its generations based on that predicted acceptance rate. RLHF avoids supervised learning entirely and broadens what counts as an acceptable result—if there's an 80% chance a user accepts an output, the model can treat it as good enough.
- Fine-tuning evaluates generated output against a labeled, known-correct answer. Each input needs a matching labeled output, so the model's parameters can be adjusted based on the gap between actual and expected results. The process yields a highly specialized model, but labeling is time-intensive—something RLHF sidesteps by accepting probabilistic feedback.
Architecture: three component groups
Production LLM applications typically assemble components from three categories:
- User input. The UI, the LLM itself, and an app hosting platform.
- Input enrichment and prompt construction. Your data source, an embedding model, a vector database, prompt construction and optimization tooling, and a data filter.
- Efficient and responsible AI tooling. An LLM cache for cost and latency control, a content classifier or filter, and a telemetry service that tracks how well the app's output performs.
Evaluate online during real use
Offline evaluations happen before launch; online evaluations measure behavior during actual user interaction. GitHub Copilot, for instance, tracks acceptance rate—how often developers accept a suggested completion—and retention rate, which captures how frequently and to what degree a developer edits an accepted completion. Those signals tell you whether the application is genuinely useful in practice, not just in a test harness.
From complaint to fix: tracing an LLM assistant's pipeline
When Dave’s Wi-Fi router falls off the counter right before his World Cup watch party, he calls his ISP and reaches an LLM-powered assistant. His spoken complaint, "My TV was connected to my Wi-Fi, but I bumped the counter, and the Wi-Fi box fell off!" has to travel through several distinct stages before the assistant can offer a useful fix.
Capturing and routing the user's words
Every LLM interaction starts with four basic pieces. The LLM API and its host determine where the model actually runs. For an ISP handling a high volume of support calls, cloud hosting makes sense; developers experimenting with smaller models like LLaMA may prefer running them locally to avoid paying for idle cloud instances. The UI layer needs a router so users like Dave can navigate from phone menus into the emergency line. Finally, a speech-to-text tool translates Dave’s spoken words into a format the LLM can process.
Context: beyond the raw transcript
A raw transcript alone can carry a complaint, but the most useful response requires additional context. This is where input enrichment enters the pipeline. A vector database stores embeddings — high-dimensional vector representations of documents — so the assistant can draw on the ISP's own complaint history alongside generic IT documentation. Since the service provider’s resolved cases are stored as embeddings, the LLM gets a fuller picture of the problem.
Retrieving that context requires an embedding model. These models translate Dave’s verbal query into a high-dimensional vector that captures the semantics and intent of his words, not merely their syntax. A contextualized prompt assembled this way looks something like this:
// pay attention to the the following relevant information.
to the colors and blinking pattern.
// pay attention to the following relevant information.
// The following is an IT complaint from, Dave Anderson, IT support expert.
Answers to Dave's questions should serve as an example of the excellent support
provided by the ISP to its customers.
*Dave: Oh it's awful! This is the big game day. My TV was connected to my
Wi-Fi, but I bumped the counter and the Wi-Fi box fell off and broke! Now we
can't watch the game.
Several open source and free vector databases are available — including Qdrant, Pinecone, and Milvus — and MongoDB offers a public preview of Vector Atlas Search.
Context enrichment is not the whole story. Extra layers guard the pipeline against overreach. A data filter ensures the model does not process personal identifiable information or unauthorized data; projects like HeimdaLLM are early attempts at building this protection. A prompt optimization tool, meanwhile, decides which embeddings matter most and how to order the final prompt for the best response. This algorithmic assembly — distinct from the in-context learning an end user might do — is prompt engineering proper, and frameworks such as langchain-ai/langchain handle the compilation for you.
Efficiency and safety rails
Nobody waiting for a World Cup game wants to watch a spinner, so the assistant should resolve problems quickly. An LLM cache stores responses for recurring situations — a crashed router is not unique to Dave. Retrieving a prior output instead of generating a fresh one reduces latency, cuts computational costs, and keeps suggestions consistent. Projects like zilliztech/GPTCache are good starting points for experimenting with response caching.
Users under pressure can get hostile. A content classifier or filter prevents the automated assistant from mirroring that hostility with harmful or offensive suggestions. Early-stage projects like derwiki/llm-prompt-injection-filtering and laiyer-ai/llm-guard target precisely this kind of prompt-injection and abuse scenario.
The last piece is observability. A telemetry service evaluates how well the application works with real users — tracking, for example, how often Dave accepts or edits the suggestions he receives. OpenTelemetry provides an open source framework for collecting and exporting such telemetry across development, testing, and production stages.
LLM applications making a difference
The same architecture is at work well beyond customer support. NASA and IBM released the largest geospatial AI model thus far to open up earth science data for climate research. The Johns Hopkins Applied Physics Laboratory is designing a conversational agent to deliver plain-English battlefield medical guidance to soldiers without medical training. In consumer spaces, Duolingo and Mercado Libre both rely on GitHub Copilot — Duolingo to help people learn languages, Mercado Libre to build commerce infrastructure for Latin America.
