Generating text at scale with an LLM creates an evaluation gap: nobody can read every output, so teams cannot say which results are strong, which are mediocre, and which cause harm. The usual answer is LLM-as-a-Judge — a second model that scores the first model's output as a stand-in for human raters. It is fast, cheap, and quick to deploy, but it raises its own question: who evaluates the evaluator? A judge that drifts, quietly degrading until it approves bad output, trips no alarm downstream.

The lifecycle described here addresses that risk with one governing rule: an LLM judge must never operate alone. It stays anchored in human judgment through a human-in-the-loop (HITL) presence at every phase — people define "good," produce the labeled data used to train and validate the judge, watch its behavior in production, and approve any update before rollout. A reliable judge is therefore not a fixed artifact but something with a lifecycle: it is built against trusted human labels, aligned with human preference and reasoning, run under human-set policy, and maintained as data and usage shift. Each stage has human owners.

The running example is recommendation explanations. A recommendation explanation is a short natural-language message shown next to a recommended item, shaping how users read and trust a recommender system — human-readable evidence for why a title was recommended given what the member has watched, displayed on the title's detail page. The work focuses on similarity-based explanations, which tie a recommended title to one reference title the member has engaged with through shared attributes such as genre and tone:

A funny, heartfelt holiday romance about love and new beginnings, much like "My Secret Santa."

Quality matters because a misleading or poorly grounded explanation destroys the trust it was meant to create. The findings come from controlled, user-facing online experiments on the explanation feature, including a month-long test on the Netflix mobile app. Those tests relied on an LLM generating hundreds of thousands of explanations a week across a constantly changing catalog — far too many to review by hand, which is why a second LLM was deployed as evaluator. The standards for a good explanation are task-specific, but the build-align-monitor method generalizes to other tasks.

The Four Phases and Their Components

The lifecycle runs from benchmark construction through judge development, deployment, and long-term monitoring.

Figure 1. The four-phase lifecycle of an LLM judge, and the data that flows between the phases.

Birth: Benchmark Data Creation. Human experts author hard, near-boundary examples, and human raters label them with pass/fail decisions and written rationales; LLM-generated gray-area examples enrich the benchmark.

Training: LLM Judge Development. The judge is tuned to match human labels and rationales from the Birth phase. Self-reflection surfaces judge-human mismatches, and rubrics are updated from those mismatches and anchor examples. Iteration continues until metrics show sufficient alignment with human raters.

Deployment. The judge enters the explanation generation system as both gate and critic: the generator writes explanations, the judge scores them, and each one is passed through or blocked with a rationale the generator can use for self-refinement and retries. Passing explanations are served; low-quality ones are gated.

HITL Alignment & Drift Monitoring. The judge is benchmarked against trained raters, and a human approves every change before rollout. During each test, a sample of explanations — passed and gated, with a bias toward newer items — was reviewed weekly by human raters with labels and rationales and compared against the judge. Exceeding the drift threshold triggered retraining and benchmark augmentation, feeding new human-labeled data back into the benchmark and development phases.

Components present in every phase

Four automated components recur throughout, alongside the human roles that define, validate, and supervise them.

  • Generator: writes the explanation, optionally incorporating earlier judge rationales and traces into its prompt for revision.
  • Judge: evaluates an explanation and returns a pass/fail label plus a written rationale. Deployed, it serves as guardrail (gating bad outputs) and critic (feedback for self-refinement).
  • Reflector: revises the judge's rubric after mistakes, using mismatched examples, human rationales, and anchor examples. It tunes the rubric and never evaluates explanations directly.
  • Meta-Judge: checks whether the judge rejected an explanation for the right reason by comparing its rationale with a human rater's.
  • Human experts and raters: the experts who define quality criteria; the raters whose labels and written rationales are the target for tuning and measurement; the reviewers who monitor the running judge and approve changes before rollout.

Phase I: Ground Truth First

Everything downstream rests on a definition of "good," so this phase is the most human-intensive. Internal writing experts specify the must-have criteria, author detailed labeling guidelines, and hand-craft adversarial examples. The guidelines serve double duty as the rulebook for human raters and the seed rubric for the LLM judges tuned later. Off-the-shelf LLM-judge benchmarks target general-purpose text evaluation and miss this task's specifics: explanations are short, item-specific, and context-dependent, each tied to a target title and one or two reference titles the member has watched, and the bar is an internal editorial standard rather than generic language quality. The custom benchmark was therefore assembled from three sources:

  • Expert-crafted adversarial cases: hand-authored pass/fail examples with known failure modes, edge cases nothing else reliably produces.
  • Boundary examples: LLM-synthesized explanations deliberately pushed near the pass/fail boundary and then rated by humans — the ambiguous cases live traffic rarely surfaces.
  • Real sampled explanations: outputs sampled from the LLM generator during test runs.

Two design details carried disproportionate weight. The benchmark is not frozen: the Phase IV monitoring pipeline appends a fresh batch of human-rated explanations weekly, so the benchmark tracked the current catalog and new usage patterns instead of going stale. And every "fail" label carries a free-text rationale explaining why the explanation failed — easy to dismiss as bookkeeping, but those rationales are what make tuning and auditing the judge's reasoning possible.

Phase II: Teaching the Judge to Reason, Not Just Score

The failure mode label agreement cannot see

A judge can agree with humans on the label — passing what they would pass, failing what they would fail — while reaching the right verdict for the wrong reason. On a dashboard it looks perfectly aligned, yet when the pipeline asks it to explain a rejection to steer self-revision, it can hand back misleading guidance. Label agreement is therefore not enough; the target is reasoning alignment. In this setup the judge always emits a label (e.g., PASS/FAIL) and a short natural-language rationale for it.

An illustrative case: the member watched a musical, a family fantasy with a young heroine and a well-known ensemble cast, and is recommended an adventure movie that is also a family fantasy with a young heroine — the basis for the pairing — but that shares neither songs nor ensemble cast.

FAIL is the correct label and the stated reason looks reasonable at first glance, but it proves incomplete in a way label agreement cannot detect.

Reasoning-Aligned Rubric Tuning

Each criterion's judge is tuned by Reasoning-Aligned Rubric Tuning (RART), which combines the Judge, Meta-Judge, and Reflector into a training loop: the Judge scores an explanation against a rubric and gives a rationale, the Meta-Judge compares that rationale to a human-written one, and the Reflector edits the rubric text. Because every judge is a fixed prompt template parameterized by a rubric, tuning does not retrain the model — the rubric text is refined iteratively until judge behavior matches human labels and human rationales. At iteration i, the judge is conditioned on the current rubric text and maps an explanation to a label-rationale pair.

R_i is the current rubric text, y_J(x) the label, and r_J(x) the rationale.

Each RART iteration proceeds in three steps:

  1. Score the training data with the current rubric.
  2. Evaluate weighted alignment metrics on a held-out validation set; stop if all per-criterion metrics clear their thresholds.
  3. Otherwise, build an error-driven focus set of misaligned examples and hand it to the Reflector, which proposes a revised rubric.

The divergence from a standard self-improvement loop lies in how a "mistake" is defined. Most such loops look only at label disagreements. RART adds a second class: the Meta-Judge examines cases where judge and human both said fail, compares the judge's stated reason against the human's rationale, and treats divergence as a right answer, wrong reason error. The focus set handed to the Reflector is the union of two error classes:

  • Label mismatches: examples where the judge's label differs from the human's.
  • Right label, wrong reason: examples where both said FAIL but the Meta-Judge finds their rationales don't match.

Rubric revisions therefore draw on both ordinary label errors and right-label-wrong-reason errors, and that second term is the core of RART: the judge is pushed not merely to match human decisions but to justify them in a way the Meta-Judge recognizes as aligned with human reasoning. In the earlier example, judge and human both said FAIL, so a label-only check would record agreement — placing the two reasons side by side shows what that check misses.

RART routes the case into the focus set, forcing the next rubric revision to account for it. Because the Meta-Judge is itself an LLM, it was validated against humans first: trained raters labeled a sample of agreed-fail rationale pairs, and the Meta-Judge matched their verdicts on nearly all of them.

Per-criterion metrics

One judge runs per criterion, each with its own rubric, tuned and scored independently; all measurements are per criterion on held-out data.

  • Specificity (fail-recall): how often the judge correctly rejects bad explanations — the metric that matters most, since a bad explanation slipping through the gate is a trust hazard.
  • Recall (pass-recall): how often it correctly passes good explanations, keeping coverage high and revision cost low.
  • Reasoning agreement: on explanations both reject, how often they reject for the same reason.

Each criterion's judge carries its own three metrics, and one can be healthy while another slips. This is also why the gate is strict: an explanation must pass every must-have criterion before it is eligible to serve.

Ablation: isolating the reasoning signal

To measure what the reasoning signal contributes, RART was compared against an otherwise identical loop using only label disagreements — same Reflector, same judge, same data, but no Meta-Judge and no human rationales. This baseline is called vanilla.

Figure 2. Improvement over the starting rubric, per criterion, for reasoning-aware tuning (RART) and label-only tuning (vanilla).

On two of the three must-have criteria, vanilla and RART land in the same place: the starting rubric was already near ceiling, leaving little room to gain. The third criterion separates them. There the vanilla, label-only optimizer degrades — its Reflector rewrites the rubric into degenerate forms, and specificity and reasoning agreement fall with each iteration, ending below the rubric it started from. RART stays stable on that criterion and lifts both metrics, with a small recall gain as well. The explanation: a label-only optimizer can reach the right label through unsound reasoning, and unsound reasoning does not generalize to new data, whereas anchoring on the rationale prevents that. Reasoning alignment helps where headroom exists and does no harm where it doesn't.

Phase III: Gating, Critiquing, and Proving Usefulness

Once aligned, the judge sits inside the generation pipeline, gating and critiquing explanations as they are produced. It does two jobs for each generated explanation.

  • The gate. The judge scores each explanation against the must-have criteria and rejects anything that fails.
  • The critic. A failed explanation isn't discarded: the judge's free-text reason is appended to the generator's prompt and the generator tries again.

Together these roles make the pipeline a generate → judge → revise loop, with the judge's critique steering every revision.

Bounded retries, and dropping instead of serving

Retries are capped, since each revision costs another generation-and-judging round and gains flatten quickly. On 1,000 generated explanations run through as many as 12 revisions with four generators, every curve climbs — the judge's critique working as intended — but the steep part comes early, and the spread between generators is wide. Revision amplifies a capable generator rather than rescuing a weak one. The test runs used a budget of 3 retries with the strongest generator, and any explanation still failing after that was dropped rather than served.

Figure 3. Cumulative share of explanations that clear the Judge as we allow more revisions, across four generator models.

The drop-rather-than-serve rule is deliberate, because the two outcomes are asymmetric: a bad explanation is a trust hazard, while a missing explanation only forgoes an opportunity. When in doubt, show nothing. Most explanations cleared the bar within the retry budget.

Do judge-approved explanations actually help members?

Passing the judge proves an explanation is good by the internal standard; it does not prove judge-approved explanations are useful to members. That link was tested with a one-time controlled experiment on the Netflix mobile app — explanations versus no explanations on the title detail page, over roughly a month — while the pipeline generated and judged hundreds of thousands of explanations a week. Members who saw explanations shifted their viewing toward titles they hadn't watched before, and more of their browse sessions ended in a meaningful play; both effects were statistically significant, and no member-initiated takedowns or escalations about explanation quality occurred during the test. The transferable result is not the effect size: offline agreement with raters was necessary for these explanations to work but never sufficient to demonstrate that they help members. Only the online A/B test could establish that judge-aligned explanations are genuinely useful rather than merely quality-compliant.

Phase IV: Monitoring Without an End Date

A judge well aligned at launch will not stay that way — the catalog shifts and so does the definition of high quality, so the lifecycle has to close the loop.

The weekly human review

For each week of a running test, roughly 300 generated explanations were pulled for human review, stratified across the judge's decisions (served, revised, dropped) and weighted toward newer catalog items, where drift is most likely. The stratification stayed fixed week to week so successive weeks could be compared. At least three raters reviewed each explanation, and the majority label served as ground truth — telling the team both whether judge-human agreement was slipping and, by growing the benchmark, feeding the next cycle.

Using human disagreement to set the bar

The panel also reveals how much raters disagreed with each other, which makes the right yardstick for the judge. Instead of a fixed threshold, the judge and each rater are scored against the same majority label, and the judge is required to fall within the raters' own spread.

Take one criterion and one of its metrics, M. That week's sample yields one value M(J) for the judge and a set of per-rater values M(H) for the three or more raters on the panel. The requirement is

so the judge must score no worse than two standard deviations below the average rater. A harder week widens human disagreement and the band along with it, so the judge isn't penalized for difficulty the humans also felt.

Falling below the band on any metric-criterion pair counts as a drift event, triggering re-tuning (back to Phase II) on the augmented benchmark. A re-tuned rubric never goes live automatically: a team member reads the proposed changes and signs off before they reach the running pipeline, and the previous rubric is retained for instant rollback.

What only human review catches

The weekly review answers a second question beyond judge-rater agreement: are the current guidelines still asking for the right things? Reviewers surfaced issues that never appeared in agreement metrics. An explanation can use accurate attributes while overstating how similar two titles really are. A single vague adjective ("exciting," "fun," "emotional") can satisfy the rubric while saying almost nothing. Sometimes the reference itself is the problem — a title the member wouldn't consider comparable at all, so the comparison lands oddly however well the attributes line up. Some genres show the problem at scale: stand-up specials can share metadata tags while differing completely in tone and cultural context, so a comparison looks justified on paper, isn't bad enough for a rater to fail, and still lands poorly. And some explanations meet every must-have criterion while reading as generic "word soup" of little help to the user.

When recurring patterns appeared, the Phase I guidelines were updated — since those guidelines seed both the raters' instructions and the judges' rubrics, one revision updates both sides. In other cases product rules were added outside the judge, for example by blocking certain title types from being used as references. Drift detection catches a judge that has stopped agreeing with people; only continued human review reveals when the definition of a good explanation is too narrow or incomplete. Both are needed:

  • Drift detection keeps the judge aligned with the current rubric.
  • Continuous human review keeps the rubric aligned with what actually works for members.

Transferable Lessons

  1. Invest in the benchmark before the judge. A modest rationale-annotated benchmark beats a large label-only one; those rationales are what make reasoning-aligned tuning possible, and a judge's reasoning cannot be corrected if the human's was never written down.
  2. Tune on reasoning, not just labels. A judge can reach the right verdict for the wrong reason — fine on a dashboard, misleading as feedback, and unstable on new data. In the ablation, the label-only loop degraded a rubric that the reasoning-aware loop kept stable.
  3. One aligned judge can serve many roles. Reusing the same judge as gate and critic amortizes the alignment cost and keeps production behavior consistent across roles; alignment is the expensive part, so get one judge right and use it more than once.
  4. Plan monitoring from day one. A recommender platform's items and members never stop changing, so an LLM judge, like any ML model, must be watched and re-tuned as needed rather than shipped and forgotten.
  5. Keep people in the loop by design, not by exception. Automating evaluation is not the same as delegating responsibility for it. Here humans set the standard (Phase I), supply the rationales the judge is aligned to (Phase II), choose the conservative default of dropping rather than serving a doubtful output (Phase III), and review a standing sample of live decisions while approving every rubric change before it ships (Phase IV). These checkpoints are far harder to retrofit once a judge is already making decisions alone, and the standing review also shows what the rubric is missing, not just when the judge has slipped.

The judge extends the reach of the reviewers, and those reviewers are the basis for trusting the judge — a relationship that makes running this system at scale responsible. An automated evaluator that no one audits provides no assurance about the decisions it makes.

Acknowledgements

This article summarizes the research paper "The Lifecycle of LLM-as-a-Judge for Large-Scale Recommendation Explanations," which contains the full methodology, metrics, and ablations. Thanks go to the internal human writers and review experts who defined what a good explanation is and keep the guidelines current; to the Human Data Operations team and the raters they train and calibrate, whose labels and written rationales are what the judges are aligned to; to the engineers across the team who built and operate the pipeline; and to the UI engineering partners who built the member-facing surfaces that bring these explanations to the homepage.