Catching JiTTests have moved from research prototype to part of Meta's production infrastructure. The company has published the results of its initial deployment in the paper "Just-in-Time Catching Test Generation at Meta," presented at FSE 2026 and featured at Product@Scale. The system targets a simple problem: with hundreds of millions of lines of code evolving continuously, traditional testing cannot keep pace, and catching risks before code lands lets teams respond earlier.

How it differs from AI code review

Most AI-based approaches to bug prevention use LLMs or agentic AI to review code and flag faults based on prior experience — a faster version of the review a human would perform. Catching JiTTests takes a fundamentally different route: it generates executable tests that demonstrate a concrete behavioral difference introduced by a change. That requires more computational steps, but it catches classes of bugs that LLM- or agent-based review might miss.

Table 1. Key differences between traditional LLM-based code review and Catching JiTTests.

Two workflows, one goal

JiTTests are generated automatically and designed to fail, surfacing bugs in diffs before they reach production. At Meta's scale, the hard part is separating strong catches — true-positive failures revealing real bugs — from weak catches that may be false positives. Two tandem workflows address this:

  • An intent-aware workflow infers the intention and risks of a change and targets test generation at likely failure points.
  • A dodgy diff workflow treats every change as if it were a mutant or bug and generates tests exposing its behavioral differences, without reasoning about intent.
Figure 1. The architecture of the dodgy diff and intent-aware workflows for Catching JiTTests.

The pipeline in five stages

1. Targeting and triggering

The system runs overnight on spare capacity and selects diffs submitted the previous day that are deemed risky, using a mechanism similar to Diff Risk Score (DRS) and trained on previous code changes. This focuses generation on diffs most likely to hold severe regressions.

2. Diff-aware test generation

The dodgy diff workflow treats each diff as a mutant and uses mutation-guided LLM-based test generation to produce tests that distinguish the diff's behavior from its parent, maximizing the pool of weak catches and the chance of a strong catch among them. In parallel, the intent-aware workflow uses an LLM to infer the diff's intention from the code changes, diff title and summary, describes the risks — ways an implementation attempt could go wrong — and constructs mutants for each. Tests are generated to kill each mutant, then run against the actual diff; failures are harvested as weak catches.

3. LLM-as-Judge ensemble

To classify weak catches, a collection of SOTA LLMs reviews the test-case code, the execution outcome (failure code, message, execution trace), the parent-to-child code changes and the inferred intent.

4. Rule-based assessor

LLM-as-Judge is complemented by a traditional rule-based system called Rule-Based False-Positive Killing Environment (RubFake). Where the ensemble offers rich semantics but non-deterministic, hard-to-verify judgments, RubFake provides verified, deterministic ones. Together they produce an overall ranking of weak-catch candidates.

5. Human-on-the-loop

Based on that ranking, the system decides whether to reach out to engineers. Instead of a human-in-the-loop model in which a person reviews every output, engineers set criteria and review outcomes at an aggregate level.

Table 2. Operational differences between human-in-the-loop and human-on-the-loop approaches.

Production results

As of this writing, 589 engineers at Meta — 64% of those reached — have engaged with Catching JiTTests, and they report genuinely actionable findings with a 60% true-positive rate. In the first half of 2026, the system helped catch 74 serious (SEV) privacy bugs at the diff stage, before they landed in production, and lowered the overall occurrence of privacy SEVs by 20% during that period.

Figure 2. The impact of Catching JiTTests at Meta in the first half of 2026.

Why the model scales

As agentic development speeds up writing, reviewing and shipping code, testing has to keep up, and every bug caught before production is one end users never encounter. Catching JiTTests shows LLM-generated testing can scale enough to reduce organizational risk. The approach is generalizable: engineers define success criteria early, let AI execute the testing, monitor outcomes in aggregate and intervene at the policy level rather than per item — so they rarely need to read test code and only confirm whether an observed behavioral change was intended.