Why synopsis factuality is hard

A synopsis sits between accuracy and intrigue: it must tell a viewer what to expect without giving the twist away. On a platform with thousands of titles, multiple episodes per season and dozens of languages, human fact-checking does not scale, so the team behind the service built an automated checker that judges whether a synopsis is factual.

The current checker hands a reasoning model everything known about a title, such as metadata and text assets, and asks it to apply a detailed rubric. The Bridgerton examples show how broad "factual" has to be:

  • A synopsis that is plainly wrong (a modern-day crime drama starring a Regency character) is a clear FAIL.
  • A synopsis in which every sentence is technically true but the emphasis misleads a new viewer is also a FAIL.
  • A synopsis that is literally true but implies an immediate, passionate affair when the relationship actually begins as a fake arrangement is a harder FAIL to detect.

The existing pipeline has two weaknesses. It is expensive: each decision takes five calls to a large reasoning model over contexts of 100k+ tokens, and runs are repeated to damp sampling noise, failing the synopsis if any run flags it. It also reasons differently from the annotators — it may fail a synopsis for nitpicking (never literally calling magic "wonderful") while talking itself out of genuinely misleading phrasing. That misalignment makes verdicts less reliable and rationales less usable for downstream correction.

The reworked pipeline addresses both at once. Cost per synopsis drops 52.4%, Recall–FAIL (the share of truly non-factual synopses correctly rejected) rises from 0.77 to 0.86, and rationale overlap with human reasoning climbs from ~69% to ~82%.

Three changes to the pipeline

Gate the repeated calls on confidence

Analysis on a custom benchmark of human-labelled synopses and annotator rationales showed that the five expensive holistic calls were mostly wasted. For blatantly true or false synopses, every run agrees with nearly identical reasoning; disagreements only appear on nuanced errors, where sampling does help. Having the model emit a confidence score alongside its verdict lets the system pass high-confidence cases straight through and escalate low-confidence ones to further checks, much as airport security gives most bags a direct path and pulls flagged ones aside for extra screening.

Use specialised judges instead of identical ones

Extra calls only reduce sampling noise; if every run follows the same flawed rationale, they all arrive at the same wrong verdict. The fix splits the generic prompt into two specialised judges:

  • Literal Judge: is the synopsis literally true?
  • Implicature Judge: does the synopsis give a false picture to a reader who has not seen the show?

Asking each model to hunt for a specific error type flags literal and implicature mistakes more reliably than re-querying a single holistic judge and hoping randomness surfaces them.

Recheck the shaky spots in the rationale

Rationales often contain a point of contention the model brushes aside — an assumption, an unresolved doubt the evidence cannot settle (such as an external critic quote), or a contradiction dismissed as too small to change the verdict. Both judges are asked to critically review their own reasoning and surface these concessions. A router model then rewrites them as self-contained, verifiable atomic claims, each carrying a materiality field describing the evidence needed to settle it, and routes each claim to an appropriate Stage-2 judge: a web judge for external facts (citations, media quotes, prior work) or a source-material recheck focused on the reframed claim. Narrowing the point of contention makes it easier to confirm or refute.

The cascade: how the tiers fit together

Every synopsis passes through both the literal and implicature judges. High-confidence verdicts are kept; low-confidence cases escalate to the router and Stage-2 judges, which can overturn the initial decision in two situations:

  • FAIL → PASS: every concession, including the original failure reasons, is supported by the secondary check.
  • PASS → FAIL: at least one concession is contradicted.

Evaluation setup

Three pipelines are compared: the current holistic baseline, a confidence-gated holistic variant, and the full cascade.

Testing uses two datasets, show-level and episode-level synopses. Each datapoint contains a synopsis, a human gold label (factual / not factual) and the annotator's rationale; episode-level items must also be specific to that episode rather than a generic show synopsis. Quality is measured by Recall-FAIL as the primary metric (higher means fewer missed Gold FAILs), Precision-PASS (how reliably PASS verdicts match Gold PASSes) and alignment with human rationales. Cost is measured by average LLM calls per synopsis and end-to-end cost per verdict. Latency is not a primary target, but the median (P50) should stay within a practical range for judgment.

Results and lessons

Focused on the quality side, the cascade improves the primary metric Recall-FAIL on both show and episode-level synopses with only limited impact on Recall-PASS, and is cheaper at the same time. Cost drops 52.4% on the show-level dataset, and on the episodic dataset the cascade with summary is 15% cheaper than the current no-summary holistic pipeline. Its P50 latency rises relative to the holistic baseline but remains within what the team considers reasonable.

To test whether simple tweaks suffice, the cascade was compared against two holistic ablations: higher reasoning effort, which made the checker too lenient, and a smaller, cheaper model, which made it too strict. Neither addressed the baseline's underlying problems. For episodic synopses, a full-show summary was added to the prompt for context from previous episodes; summaries substantially improve performance over non-summary context variants, so only summary-based variants are reported. Without a summary, the model tends to over-fail because it cannot tell whether an episode detail fits the broader storyline.

Are the verdicts right for the right reasons?

An LLM-as-a-Judge compares each pipeline's reasoning against human rationale on Gold-FAIL synopses, assigning a Reasoning Similarity Score of 0 (dissimilar), 0.5 (partially similar) or 1 (very similar). The cascade is significantly better aligned in all cases, meaning it identifies the same points of contention as a specialist annotator — which also makes its explanations usable for correcting faulty synopses later.

Where this leads

Confidence gating avoids unnecessary calls; diverse judges contribute complementary verdicts; Stage-2 judges recheck and correct the assumptions behind uncertain decisions. The result is a factuality service that is cheaper, more accurate, more interpretable and self-correcting. Future directions include multimodal inputs to verify claims that text assets alone cannot confirm, and using the factuality rationale to repair the synopsis itself.