Using LLMs to amplify human labeling and improve Dash search relevance

Why search ranking quality matters for RAG

Dropbox Dash uses a retrieval-augmented generation (RAG) pattern to answer questions. The system first retrieves relevant company information from an enterprise search index, then passes a subset of those results to a large language model (LLM) to generate a grounded answer. Because enterprise indexes can contain millions or even billions of documents, only a small portion of retrieved candidates ever reaches the LLM. The quality of the final answer therefore depends heavily on how well the underlying search ranking model orders those candidates.

Dash's ranking model is trained with XGBoost rather than hand-tuned rules. It learns from labeled examples of query–document pairs, where each document receives a graded relevance score on a 1–5 scale (5 being a close match; 1 being not useful enough to show). Relevance is not a fixed document property—it depends on the query, the user's context, and the moment of search. The core challenge is generating enough high-quality relevance labels to teach the model effectively.

Human labeling: strengths and limits

Historically, search engines relied on teams of third-party human judges to label large datasets. This approach allows systematic evaluation of full result sets, providing consistent relevance coverage that user behavior signals (like clicks or skips) cannot offer on their own—those signals tend to be sparse, biased by existing rankings, and unevenly distributed.

But human labeling has serious drawbacks. It is expensive, hard to scale, and prone to inconsistency across judges. It is also nearly impossible to apply directly to sensitive or proprietary customer data. The diversity of content types—comparing a Slack message to a Jira ticket or a Salesforce record—requires very different contextual judgment, which makes training and consistency even harder.

LLM-assisted labeling at scale

LLMs offer a cheaper, more consistent alternative that can evaluate much larger candidate sets across languages and can operate within defined compliance boundaries. However, an LLM is not a general intelligence system. Its performance depends heavily on the underlying model's quality and the precision of the instructions. Any LLM-generated judgments must be validated and calibrated before being used for training.

Dash's approach pairs automation with human oversight. A small group of human evaluators labels a dataset orders of magnitude smaller than what full training would require, working only with limited, non-sensitive internal datasets (no customer data is reviewed by humans). Those labels are used to tune the LLM's prompt and parameters. Once the LLM meets quality thresholds, it is deployed to generate hundreds of thousands—or millions—of relevance labels to train the ranking model. In this setup, the LLM acts as a force multiplier: humans teach the LLM, and the LLM generates training data at scale.

There is no current path to using LLMs directly at query time to replace the ranking model; context window limitations and latency make that infeasible. Instead, Dash uses LLMs offline as teachers for smaller, more efficient relevance models that can run in production.

Measuring LLM judge accuracy

Improvement starts with evaluation. Dash compares LLM-generated relevance ratings against human judgments on the same 1–5 scale. Exact matches are rewarded; disagreements incur penalties that grow quadratically with the gap. This is captured as mean squared error (MSE), ranging from 0 for exact agreement to 16 for the maximum possible disagreement (a 1 versus a 5).

Focusing evaluation on error-prone cases

Not all evaluation data is equally informative. To improve LLM accuracy efficiently, Dash biases training samples toward cases where mistakes are most likely. These are identified by analyzing discrepancies between user behavior and LLM-predicted relevance—for instance, users clicking documents the LLM rated as low relevance, or consistently skipping documents it rated highly. Such discrepancies are prioritized for human review and prompt refinement, again on small internal datasets only. This process repeats iteratively until major error sources are addressed or improvements plateau.

Adding context before judging relevance

A query and a document alone are often insufficient to make a reliable relevance judgment. Internal terminology, acronyms, and organizational knowledge can be required. Within Dropbox, for example, "diet sprite" refers to an internal performance management tool—not a soft drink. Acronyms frequently have multiple meanings even within a single company. Human evaluators resolve such ambiguity by running additional searches or consulting internal tools.

To automate this, Dash gives the LLM tools to research query context before assigning labels. Once the LLM understands user intent, it can apply consistent, context-aware judgments across large candidate sets—often going deeper than human evaluators would in practice.

Iterative prompt optimization

As evaluation scales, prompt quality matters increasingly. Prompt optimization resembles the way human guidelines are developed: review cases where the model gets relevance wrong, adjust instructions or add context, then test again. Small prompt changes can cause unexpected regressions, and consistency becomes harder as prompts grow longer.

Meta-prompting frameworks like DSPy—a library for programmatically optimizing LLM prompts against defined targets—help manage this complexity. Given a clear objective and a small set of human-labeled examples, DSPy can refine prompts automatically to better match human judgments, enabling reuse of the same optimization approach across tasks and model configurations.

The chart below shows the MSE of the LLM relevance evaluator improving over time through a combination of prompt refinement, a reasoning-optimized model, query context, and automated optimization with DSPy.

Human Judgment as the Anchor for LLM-Based Relevance Labeling

The labeling methodology described here extends beyond document search or any particular model or evaluation framework. The core pattern is what matters: begin with a small corpus of high-quality human judgments, use that data to calibrate LLM-based evaluation, and scale relevance labeling while keeping it measurable, auditable, and correctable.

Because LLM-generated labels are anchored to human-reviewed reference data, they remain open to continuous monitoring, stress-testing, and recalibration as models, prompts, and product requirements evolve. This grounding yields a stable evaluation baseline in which regressions are detectable and improvements are quantifiable, even when the surrounding system changes.

Adapting Across Content Types

As Dash extends to new content types—images, videos, messages, and chat—the evaluation problem grows more complex. Relevance is encoded differently in each domain, and surface-level similarity seldom suffices. A human-calibrated LLM evaluation offers a shared approach for adapting relevance judgments across modalities without rebuilding labeling pipelines or redefining evaluation criteria from scratch.

Even as models advance, human grounding remains a structural necessity. Prompts drift, models shift, and product expectations evolve. A persistent, human-reviewed reference set anchors evaluation over time, letting LLMs scale judgment without compromising correctness. In essence, LLMs make possible the consistent and large-scale application of human judgment—they do not replace it.

Acknowledgments: Eric Wang, Hans Sayyadi, Josh Clemm, Mingming Liu, Andrew Yates, Marta Mendez, Jun Sun, Jay Frank, Angela Li