Why RAG exists

Large language models have some significant blind spots. Their training data has a cutoff date, so anything published after that is unknown to them. They also have no access to private or proprietary documents. And because you pay per token, feeding an entire large corpus into every request gets expensive fast.

Retrieval Augmented Generation (RAG) addresses these limitations with a separate retrieval step: instead of sending the whole document set to the LLM with each query, you first find the most relevant pieces of text, then send only those along with the question as context.

RAG in Go

RAG is often demonstrated in Python, but the core work is really a data-pipeline problem, not a machine-learning one. You are not training models or tuning loss functions; you are gluing textual tools together — and Go is well suited for that. It is fast, handles text processing cleanly, and its concurrency model fits applications that spend most of their time waiting on network calls to LLM APIs.

The motivating example here is asking questions about the Go documentation, including material added after the model's training cutoff. For instance, Go 1.21 introduced the GOTOOLCHAIN environment variable. Asking a model with an older cutoff about it produces plausible-sounding but wrong hallucinations, because the model simply has no knowledge of that feature.

The solution is a RAG pipeline with three stages:

  1. Read the latest Go documentation pages (Markdown files from the Go website repository) and split them into chunks.
  2. When a question comes in, find the most relevant chunks from that corpus.
  3. Append those chunks to the question as context, and send the whole thing to the LLM API.

Step 1: Ingest and chunk the documentation

The first step requires no LLM technology at all. A command-line tool walks a locally cloned _content directory of the Go website source, reads each Markdown file, and splits it into chunks of roughly 1000 tokens, keeping whole paragraphs intact. Each chunk is stored in a SQLite table along with metadata.

CREATE TABLE IF NOT EXISTS chunks (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  path TEXT,
  nchunk INTEGER,
  content TEXT
);

Step 2: Embeddings and retrieval

An embedding model maps arbitrary text to a fixed-size vector of real numbers, such that related texts land close together in that vector space. To make the corpus searchable, a second command-line tool with a --calculate flag reads each chunk from the DB, computes its embedding via the OpenAI API, and stores the vector in a new table keyed by the chunk's id.

CREATE TABLE IF NOT EXISTS embeddings (
  id INTEGER PRIMARY KEY,
  embedding BLOB
)

Because each embedding requires an API call, this step takes time and costs money. The DB serves as a cache, so you only need to pay for this once per chunk.

When a user asks a question, the same process computes the question's embedding. The program then iterates over every chunk in the DB, computes cosine similarity between the question vector and each chunk vector, and keeps the three closest matches (roughly 3000 tokens of context).

For a small corpus, a linear scan is perfectly adequate. In production settings with millions of chunks, you would reach for a vector database; several popular ones, such as Milvus and Weaviate, are implemented in Go.

Step 3: Prompt with context

The final step is simple string interpolation. The relevant chunks are concatenated and placed into a prompt template alongside the user's question.

  query := fmt.Sprintf(`Use the below information to answer the subsequent question.
Information:
%v

Question: %v`, contextInfo, theQuestion)

That is the entire mechanism. Re-asking the same GOTOOLCHAIN question with this RAG setup yields a correct answer, grounded in the actual documentation.

In Go, `GOTOOLCHAIN` is an environment variable that determines which Go
toolchain version should be used. The value of `GOTOOLCHAIN` can be set to
various options:

1. `local`: This indicates that the bundled Go toolchain should be used.
2. `<name>`: This specifies a specific Go toolchain version that should
   be used. If a binary with that name is found in the system PATH, it will be
   used; otherwise, the `go` command will download and use the specified Go
   toolchain.
3. `<name>+auto` or `<name>+path`: This tells the `go` command to
   select and run a newer Go version as needed. It will consult the
   `toolchain` and `go` lines in the current workspace's `go.work` file (or
   the main module's `go.mod` file if there is no workspace) to determine
   which toolchain version to use. If a newer version is specified in those
   files, it will be used; otherwise, it will fallback to the default Go
   toolchain.

Overall, `GOTOOLCHAIN` is used to specify the specific Go toolchain version or
the selection process of a newer Go version that should be used by the `go`
command.

Code availability

All the code for this project is on GitHub. You only need your own OPENAI_API_KEY to run it. The repository ships with a SQLite DB that already contains pre-computed embeddings, so you can skip the --calculate step entirely. A cmd/gemini-rag directory also reimplements the tooling using the Google Gemini model. Full instructions are in the README.