Evaluating the evaluators before turning them loose

Dropbox’s chat agent, Dash, answers questions by pulling together knowledge from documents, messages, meetings, and other company sources. Because so much depends on the quality of a single interaction, the agent is assessed along its entire trajectory—not just the final answer. That means judging how it interprets intent, gathers context, uses tools, handles ambiguity, grounds its answer, and completes multi-turn tasks.

Building that evaluation layer was the first step. The harder problem was making sure the judges themselves were trustworthy. If the evaluations aren’t aligned with human judgment, then any improvement signal you derive from them is suspect. So we started with human-labeled examples and a structured rubric that engineers could apply consistently.

Internal chats—including final responses and trace logs—were sampled and reviewed across five dimensions: user intent following, semantic relevance, tool calling, instruction following, and context selection. Each metric was scored on a 1–5 scale. Evaluators also wrote reasoning notes and assigned failure codes for recurring problems such as stale evidence, missing context, unsupported claims, incomplete coverage, or lack of personalization. Those failure codes turned out to be particularly valuable: they give us signal not just on response quality but on where in the pipeline things go wrong.

The annotations served double duty. Beyond optimizing the judge prompts to minimize disagreements with human labelers, they became a benchmark for debugging, error analysis, and roadmap planning.

Using DSPy to calibrate the judges

With labeled data in place, we used DSPy—specifically its GEPA and MIPROv2 optimization algorithms—to tune the judges. These algorithms automatically propose prompt changes and test them against the human-labeled examples. Think of DSPy as the toolkit, and GEPA and MIPROv2 as specific optimization strategies inside it.

Judges didn’t just need a generic scoring prompt. They had to follow a specific retrospective workflow: infer the user’s intent, inspect the conversation, review the trace and supporting evidence, reason about context selection and tool use, and then output a score accompanied by failure codes and reasoning notes. We also needed to preserve certain judge behaviors when adapting them to a different underlying model, so we supported targeted optimization aimed at correcting specific failure modes—like over-scoring outdated information or underweighting missing context—without rewriting the entire rubric.

Throughout, we relied on both the human scores and textual feedback. Scores told us when a judge disagreed; the reasoning notes explained why. For instance, if a judge repeatedly scored answers that relied on stale data too highly, we could update its instructions to recognize and penalize that specific failure mode. Once the judges reliably reflected human judgment, they could serve as the foundation for improving the agent itself.

Automating prompt optimization with replay

Prompt optimization used to be a manual, slow process. Engineers reviewed failures, proposed edits, tested them, and iterated. That workflow was hard to scale, and it was difficult to know whether a change would actually hold up in production. We replaced it with an automated, evaluation-driven loop built on labeled examples, production-aligned scorers, and offline counterfactual replay.

Each GEPA round works roughly like this: a candidate prompt is replayed on representative historical Dropbox internal chats, and the resulting agent outputs are scored by the evaluation pipeline. The scores and structured judge reasoning are fed back to GEPA, which proposes the next prompt update. This grounds prompt optimization in realistic agent behavior rather than abstract examples.

The loop targeted concrete failure modes: wrong context selection, incomplete answers, missed ambiguity, incorrect search-tool use, and loss of multi-turn context. We repeated the cycle—replaying, scoring, and proposing—until the data supported a launch candidate. Operationalizing quality checks required testing each prompt against the same set of examples as the existing production prompt, giving us an apples-to-apples comparison and a way to check whether gains were statistically significant.

Measurable improvements without added cost

The results showed up quickly. In the first two weeks, the optimization loop generated six prompt candidates automatically, compared with five manual prompt changes in the prior month. Launch results within the first 24 hours included a 26% reduction in incomplete answers and a 13% reduction in missed key aspects of a user’s request.

Efficiency also improved. Total token usage dropped by 5.4%, and average completion length decreased by 9.8%—without compromising answer quality. That last point is worth underscoring: production cost went down even as key quality metrics improved.

The whole pipeline now forms a practical feedback loop: human labels calibrate the judges, the judges produce scalable evaluation signals, and those signals drive agent improvements. For teams building agents with visible failure modes, this is a reliable way to identify those modes, generate candidate prompt changes, validate quality gains, and ship with confidence.

Safeguards that kept optimization production-safe

The clearest lesson from this work is that automated prompt optimization requires strong guardrails. The team deliberately restricted most agent prompt edits to small, targeted instruction updates and layered in automated review checks covering prompt structure, completeness, caching behavior, and size limits. These safeguards kept candidate prompts maintainable and safe to deploy even as the optimization pipeline ran with less manual supervision.

Bringing ML discipline to prompt engineering

Stepping back, this experiment demonstrated that prompt optimization can import the habits of traditional machine learning into prompt work. Combining human-labeled evaluations, representative replay data, and GEPA-based optimization inside DSPy let the team treat prompts as measurable, optimizable artifacts rather than frozen instructions. The framework provided a systematic way to search over the instructions, constraints, examples, and policies that shape model behavior, moving the process past intuition and manual iteration toward identifying failure modes, comparing improvements, and validating impact before release.

Toward continuous agent optimization

Longer term, the authors expect agent optimization to look less like iterative prompt handwriting and more like a continuous ML workflow: replay representative data, run optimization jobs, compare candidates against evaluation datasets, review the evidence, and ship validated improvements. As with conventional ML systems, weak evaluation signals produce brittle gains, whereas strong evaluations, representative data, and expert review help changes generalize while keeping regressions controlled.

The broader takeaway: agent optimization works best when the automation rides on top of rigorous evaluation—reliable judges, representative replay data, and unambiguous success metrics form the feedback loop that improves agent behavior without losing sight of measurable quality.

Acknowledgments: Jongmin Baek, Josh Wilson, Akshay Bapat, Gonzalo Garcia, April Liu, Eric Wang, Hans Sayyadi, Prasang Upadhyaya, and Emeka Okafor Jr. The team also thanked the DSPy community for engagement and support, and credited their DSPy collaborators for guidance and responsiveness as they applied DSPy to production systems at Dropbox.