A practical blueprint for evaluating conversational AI at scale

Why Evaluation Has to Be a First-Class Citizen in LLM Apps

An LLM application looks like a single text box, but underneath it is a chain of probabilistic stages: intent classification, retrieval, ranking, prompt construction, inference, and safety filtering. Change one link and the effect can ripple unpredictably through the whole pipeline. Building Dropbox Dash, a system that understands a user's work context across files and connected data sources, made that lesson concrete: in the foundation-model era, evaluation matters as much as model training.

Early on, Dash's testing was ad hoc. Progress came less from individual model tweaks and more from how the team structured processes around retrieval, prompting, and output consistency. The turning point was treating every experiment like production code. Any change to a prompt, a retriever setting, or a model version had to pass the same kind of testing gate that a code merge would. Evaluation was not bolted on at the end; it was built into each step of development.

Building the Right Datasets First

The evaluation stack starts with datasets. Public benchmarks helped establish a retrieval and question-answering baseline. Google's Natural Questions tests retrieval against large documents, MS MARCO stresses handling multiple relevant hits per query, and MuSiQue challenges multi-hop reasoning. Together they gave early signals on how parameter and system choices would hold up.

But public data does not capture the long tail of real user phrasing. The team built two kinds of internal evaluation sets from Dropbox employees dogfooding Dash. Representative query datasets anonymized and ranked top internal queries with annotations from proxy labels or human annotators. Representative content datasets focused on the material users rely on—widely shared files, documentation, connected sources—and used LLMs to generate synthetic questions covering tables, images, tutorials, and factual lookups.

These curated sets mirror real-world messiness. But datasets alone are inert until scoring logic gives them meaning. The next step is defining what success looks like through metrics, budget limits, and automated checks, before the first experiment runs.

Metrics and Rubrics That Actually Gate a Release

Classic offline metrics like BLEU, ROUGE, METEOR, BERTScore, and embedding cosine similarity are quick and familiar, but they fall short on production tasks like retrieving cited answers or parsing internal wikis. They catch wild drift but miss deployment-critical errors: high ROUGE scores even when an answer omits its source, strong BERTScore alongside hallucinated file names, fluent Markdown burying factual mistakes. These failures are not rare in production, so the team turned to a different grader.

Using an LLM as a judge. Having one model judge another sounds recursive, but it unlocks flexibility. A judge model can check factual correctness against ground truth or context, verify citations, enforce formatting and tone, and scale across dimensions traditional metrics ignore. The insight is that LLMs score natural language well when the evaluation problem is framed clearly.

Judges and rubrics need their own iteration loop. Prompts, instructions, and the choice of judge model all change outcomes. For specialized domains or specific languages, the team relied on specialized models to keep scoring fair. Evaluating the evaluators became part of the quality cycle.

Judges were treated as software modules: designed, calibrated, tested, and versioned. A reusable template takes the query, the model's answer, source context when available, and an optional hidden reference answer. The judge prompt walks through structured checks:

• Does the answer directly address the query?
• Are all factual claims supported by the provided context?
• Is the answer clear, well-formatted, and consistent in voice?

The output is a justification plus a scalar or categorical score. Every few weeks, manually labeled samples were used to tune judge prompts, benchmark human-model agreement, and track drift. Whenever the judge diverged from the gold standard, the prompt or underlying model was updated.

Automation did not eliminate human review. Engineers manually audited 5–10% of the regression suite per release, logging discrepancies back to prompt bugs or model hallucinations. Recurring issues triggered prompt rewrites or finer-grained scoring.

These metric types were wired into every development stage: fast regression tests ran automatically on each pull request, full curated datasets ran in staging, and live traffic was sampled and scored in production. Dashboards made pass/fail rates and shifts over time visible. If performance slipped below thresholds, the change did not move forward.

Tooling: From Spreadsheets to a Central Platform

Managing scattered artifacts quickly becomes unsustainable. The team adopted Braintrust, an evaluation platform that brought four capabilities to the workflow:

• Central store: a unified, versioned repository for datasets and experiment outputs.
• Experiment API: each run is defined by dataset, endpoint, parameters, and scorers, producing an immutable run ID.
• Side-by-side comparisons: dashboards highlight regressions and quantify trade-offs across latency, quality, and cost.
• Trace-level debugging: one click reveals retrieval hits, prompt payloads, generated answers, and judge critiques.

Spreadsheets work for demos, not real experimentation. With scattered files, results are hard to reproduce and nearly impossible to compare. The platform gave the team versioned runs, reproducible results, and automatic regression surfacing in a shared space.

Automating Evaluation in the Dev-to-Prod Pipeline

The team treated prompts, context selection, and model choices like any other code. Every pull request kicked off about 150 canonical queries, judged automatically in under 10 minutes. Merge triggered the full suite plus smoke checks for latency and cost. Any red line blocked the merge.

Those canonical queries were few but deliberately chosen: multiple document connectors, no-answer cases, and non-English requests. Each recorded the exact retriever version, prompt hash, and model choice. If scores dropped below a threshold, such as too many answers missing citations, the build stopped. Regressions that once slipped into staging were caught at the pull-request level.

On-Demand Synthetic Sweeps and Live Scoring

Large refactors can hide subtle regressions, so the team ran end-to-end sweeps across a golden dataset. These dispatches ran as Kubeflow DAGs, executing hundreds of requests in parallel, each logged under a unique run ID for comparison against the last accepted baseline. RAG-specific metrics like binary answer correctness, completeness, source F1 (a precision/recall balance on retrieved sources), and source recall flagged any drift beyond thresholds. LLMOps tools then sliced traces by retrieval quality, prompt version, or model settings to isolate which stage shifted.

Live traffic is the ultimate test. The team continuously sampled production queries and scored them with the same metrics and logic as offline suites, while respecting Dropbox's AI principles. Each response, context, and retrieval trace was logged and judged for accuracy, completeness, citation fidelity, and latency in near real time. Since scoring ran asynchronously, users saw no added latency. Dashboards tracked rolling medians over one-hour, six-hour, and 24-hour windows. A sudden drop in source F1 or latency spike triggered immediate alerts.

Layered Risk Gates

Three gates progressively tightened requirements as changes moved through the pipeline.

• Merge gate: curated regression tests run on every change; only those meeting baseline quality and performance pass.
• Stage gate: expanded coverage on larger, more diverse datasets with stricter thresholds for rare edge cases.
• Production gate: continuously samples real traffic to catch scale-only issues; automated alerts and rollbacks fire if metrics dip.

By scaling dataset size and realism at each gate, regressions were caught early while staging evaluation stayed closely aligned with real-world behavior.

Closing the Loop With Continuous Improvement

Evaluation is not a phase; it is a feedback loop. Low-scoring outputs and flaky regressions are not just red flags, they are chances to improve the system end to end.

Mining low-rated traces from live traffic uncovered patterns synthetic datasets missed: retrieval gaps on rare file formats, prompts truncated by context windows, inconsistent tone in multilingual input, hallucinations triggered by underspecified queries. These hard negatives became labeled examples in the regression suite or spawned new synthetic sweep variants, creating a virtuous cycle that stress-tested the system on its own past failures.

For riskier experiments, like a new chunking policy or reranking model, the team built a structured A/B playground. It allowed controlled tests against consistent baselines with golden datasets, user cohorts, or synthetic clusters as inputs. Variants covered retrieval methods, prompt styles, and model configurations. Outputs spanned trace comparisons, judge scores, and latency or cost budgets. This safe space let tweaks prove their value or fail fast without consuming production bandwidth.

When an answer failed, guessing was costly. Playbooks guided engineers to likely causes: was the document never retrieved, was context included but ignored, or did the judge itself mis-score? These triage paths ensured regressions were traced systematically.

Critically, evaluation was not owned by a single team. Every feature pull request linked to evaluation runs, on-call rotations had dashboards and alert thresholds, negative feedback was triaged, and every engineer owned the quality impact of their changes.

What the Process Taught Us

The biggest takeaway is that regressions often came from prompt edits, not model swaps—a single word change could tank citation accuracy. Human eyeballs were no substitute for formal gates. Judge models and rubrics are not set-and-forget assets; their prompts need versioning and recalibration, and specialized judges were sometimes the only way to score fairly in specific languages or technical domains.

The path forward is making evaluation proactive, not just protective. That means measuring user delight and task success beyond accuracy, building self-healing pipelines that suggest fixes when metrics drop, and extending coverage beyond text to images, audio, and low-resource languages. Evaluation is not a sidecar to development. Treated with the same rigor as production code, it is what turns probabilistic LLMs into dependable products.