Site Search With Generative AI: Adding Context and Making It Scale

Generative AI can open up a new search interface for your site: instead of a list of links, users ask a question and get a synthesized answer. The foundation is straightforward—send the question to a large language model (LLM) like Gemini, then display the response. The hard part is getting that response right.

Ask an LLM "Which APIs are included in Interop 2024?" and you'll likely get a wrong answer. There are two reasons for this: the question lacks context, making hallucinations more likely, and LLM training data is inherently outdated. Keeping training data current is expensive and time-consuming, so the model may simply not know that Interop 2024 exists.

Grounding Answers in Your Content

Prompt engineering offers a practical remedy. A well-known technique is to supply the relevant content as context in the prompt itself, then ask the question. If you paste the full text of an article into the prompt alongside the user's query, the LLM can answer based on that provided context.

Include the Interop 2024 article as context, and the model will produce an answer grounded in that text. The model isn't retrieving the information from its own training—it's reading the context you supplied. That's a meaningfully better result, but this approach works for only a handful of articles.

From One Article to a Whole Site, With RAG

Applying this to an entire site requires scaling. Sending every article page as context is not practical. Although Gemini 1.5 supports large inputs, prompts with a million tokens are slower and more expensive. LLM pricing is measured in tokens (a representation of character sequences; one word is often more than one token), so limiting input size matters for both latency and cost.

Instead, you can send only the relevant articles in the prompt. That's the idea behind Retrieval-Augmented Generation (RAG), which is a two-step technique:

  1. Find which articles are relevant to the user's question.
  2. Include the content of those articles as context when calling the LLM.

For the Interop question, instead of one article, your search step might select three relevant articles and concatenate their contents into the prompt. The resulting answer is then based on the full context of those articles, not just one.

RAG works with traditional full-text search, but end results can be unreliable. Full-text search hunts for exact keyword matches and ignores the user's intent. A query about a "bank" could match results about both financial institutions and riverbanks. Results may contain the right keywords but satisfy the wrong objective.

Semantic search addresses these failures through three pillars: searcher's intent, contextual meaning, and the relationship between concepts. It identifies what the user wants to accomplish, interprets words in context (including location and history), and uses knowledge graphs and natural language processing to understand the connections between terms. So a semantic search engine can pull documents that are relevant even when they don't include the exact keyword, and skip documents that do but have a different meaning.

You can incorporate semantic search via Vertex AI Search and Algolia AI Search.

The combination is what counts: prompt engineering supplies context so the LLM answers accurately for a single article; RAG scales that contextual approach to a full content library; and semantic search makes the retrieval stage more accurate so the LLM sees genuinely relevant articles. The result is a site search that responds with verifiable answers and drastically reduces hallucinations.