How we optimized Dash's relevance judge with DSPy

Defining "Good" for a Relevance Judge

Before optimizing anything, you need a quantifiable definition of success. For Dash's relevance judge, the core task is to assign a score from 1 to 5 for a given query-document pair. The reference point for quality is human annotation: annotators rate the same pairs and provide brief explanations for their choices. Our optimization target is minimizing the gap between the model's ratings and these human judgments.

We measure this gap using normalized mean squared error (NMSE), which summarizes the average squared deviation between model and human scores on a 0-100 scale. A score of 0 indicates perfect agreement; higher values mean worse alignment. A rating of 5 versus 4 counts as a minor miss, while a 5 versus 1 is a major error, and NMSE weights the latter more heavily.

Accuracy alone isn't enough, though. The judge's output must be usable. Since the system returns JSON, any malformed output is treated as fully incorrect. These formatting failures aren't just cosmetic—they cause dropped examples, failed batches, and unreliable evaluation metrics. Our objective, then, has two prongs: minimize disagreement with human raters while ensuring outputs are consistently parseable.

Adapting to a Cheaper Model with DSPy

Our initial high-quality judge ran on OpenAI's o3, which produced excellent scores but was too expensive for the volume Dash required. We needed to move to gpt-oss-120b, a cheaper open-weight model. The problem was prompt transfer: our carefully tuned o3 prompt didn't work well on the new model, and manual rewriting was a slow, regression-prone process. We used DSPy to systematize that adaptation.

DSPy optimizes prompts against a defined metric. Our setup was clear: the task (1-5 relevance scoring), the data (human-annotated examples), and the metric (NMSE) were all fixed. We used DSPy's GEPA optimizer, which iteratively improves prompts by analyzing specific failure cases. GEPA doesn't just look at a single score; for each example where the model disagrees with a human, it generates structured feedback combining the size and direction of the gap, the human's explanation, and the model's own reasoning.

This textual feedback feeds a reflection loop. The prompt is evaluated, its failure modes are described in plain language (e.g., "underweights recency" or "overvalues keyword matches"), the prompt is revised, and the cycle repeats. The simplified construction of that feedback looks like this:

diff = predicted_rating - expected_rating
direction = "higher" if diff > 0 else "lower"
feedback_parts = [
    f"Predicted rating {int(predicted_rating)} but expected {int(expected_rating)}.",
    f"Model rated {abs(diff):.0f} point(s) {direction} than the expected human rating.",
]

# Include human explanation if available
if gold.explanation:
    feedback_parts.append(f"Human rationale: {gold.explanation}")

# Include model's explanation for comparison
if pred.explanation:
    feedback_parts.append(f"Model's reasoning: {pred.explanation}")

feedback_parts.append(
    "Remember: when adapting the prompt, avoid overfitting to specific 
example(s). Do not include exact examples or keywords from them in the prompt. 
Also ensure you do not change the basic parameters of the task (e.g. changing the 
rating range to be anything but 1-5). Try to add a general rule to an execution 
plan to rate similar documents in the future."
)

feedback = "\n".join(feedback_parts)

We had to add guardrails to prevent overfitting. The optimizer sometimes copied specific keywords, usernames, or verbatim document phrases into prompts, which looked great on training data but didn't generalize. We forbade direct inclusion of example-specific content. We also constrained the task definition itself; without guardrails, the optimizer occasionally altered the rating scale to 1-3 or 1-4. With these constraints in place, we could compare the optimized prompt against our original manual one under identical conditions.

The results were substantial. The best DSPy-optimized prompt reduced NMSE by 45 percent, from 8.83 to 4.86, meaning the judge's scores tracked human ratings much more closely. Model adaptation time dropped from one to two weeks of manual iteration to one to two days. This speed allowed us to evaluate newly released models with less risk and keep the judge aligned with evolving product needs.

The cost benefit was equally important. By running on a much cheaper model than o3, we could label 10-100 times more data for the same budget. That increased coverage and statistical power, enabled larger experiments, and reduced the risk of downstream models overfitting to a small evaluation set.

Stress-Testing Structural Reliability

Optimizing for cost and human alignment left a second question unanswered: could the judge behave reliably when its outputs are consumed by automated pipelines? In Dash, the judge isn't just read by people; it scores large candidate sets, generates training data, and runs offline simulations where outputs are parsed by other components. We stress-tested this with gemma-3-12b, a much smaller and cheaper model that's more brittle when it comes to formatting and instruction-following.

The baseline was stark. More than 40 percent of gemma-3-12b's responses were malformed JSON, which we treated as fully incorrect. After DSPy optimization, malformed outputs dropped by more than 97 percent, and NMSE improved significantly:

This demonstrated that DSPy wasn't just improving alignment with human judgments—it was also directly optimizing structural reliability. Even a much smaller model became operationally dependable when tuned against the right objective.

The experiment also confirmed the value of iteration speed. Although gemma-3-12b wasn't strong enough for our highest-quality production judge paths, we reached that conclusion quickly with measurable evidence, rather than through prolonged debate or manual trial and error. The optimization framework let us test the model directly against our evaluation metrics and make a confident decision.

Constrained Optimization for a Production Model

The strategy differed depending on which model we were targeting. For newer, cheaper models like gpt-oss-120b or gemma-3-12b, we were comfortable with full prompt rewrites, favoring broad exploration and end-to-end optimization. The production o3 judge was another matter. Already strong and deeply integrated into multiple pipelines, it demanded a more cautious approach. Large rewrites were risky: even subtle wording changes could alter behavior in edge cases with a wide blast radius.

Prompts as Small, Testable Changes

To make improvements both targeted and controllable, we introduced an instruction library layer. When the judge's score diverged significantly from a human rating, an evaluator would write a short explanation of what the model misunderstood and what it should have focused on instead. These explanations were distilled into single-line instruction bullets—small, reusable rules of thumb the model could follow.

In this setup, DSPy's role is reduced to selecting the best instruction bullets and deciding how to combine them, rather than rewriting the prompt text. Common error themes were identified and addressed by assembling the most helpful additional guidance. This turned optimization into a series of incremental “small PRs with tests” rather than a large-scale refactor. Regressions were simpler to diagnose, and baseline behavior stayed stable while agreement steadily improved.

For instance, an error explanation like “the document is older than a year, so it's less relevant for this query” was converted into a bullet such as: “Documents older than a year should be rated at least one point lower unless they are clearly evergreen.” DSPy could then learn whether including that specific bullet improved alignment on the eval set without introducing unintended side effects.

Each step shown above is a small, testable change; together they yield a meaningful improvement over the initial prompt.

A Repeatable Optimization Loop

In Dash, relevance scoring is a core capability that underpins ranking, training data generation, and offline simulation. Because these pipelines depend on the judge, even minor scoring fluctuations can propagate outward. Relying on manual prompt surgery for every model update or new prompting idea would make progress slow and error-prone.

With DSPy, the objective is defined upfront: alignment with human relevance judgments. By holding the task and dataset fixed, we can swap in new models and adapt them quickly, backed by measurable evidence rather than intuition. The workflow shifts from rewriting prompts to improving against a clear metric. Crucially, we can calibrate the risk: full end-to-end optimization when exploring new, cost-effective models, or constrained, incremental updates when preserving stability is critical, as with o3.

For a system like Dash, where relevance touches ranking, data generation, offline evaluation, and cost–latency tradeoffs, prompt optimization cannot be a one-off exercise. DSPy makes it a repeatable loop: define the task, evaluate against human labels, optimize, and ship changes with confidence as models continue to evolve.