Testing’s 50-Year Model Is Breaking
For decades, software testing has followed the same recipe: an engineer writes a test when code lands, that test gets committed to a static suite, and then it runs forever. Every future change to the codebase risks breaking that test — either with a false positive that wastes an engineer’s time, or with a test so loosely coupled to the actual code that it catches nothing at all. The rise of agentic development, where code is generated, reviewed, and shipped at machine speed, has turned this maintenance burden and false-positive drag from a nuisance into a bottleneck.
Just-in-Time Tests (JiTTests) flip the model. Instead of maintaining long-lived tests, JiTTests are generated on the fly by LLMs at the moment a pull request lands. They are tailored to that specific change, designed to catch regressions before the code reaches production, and they never need to be updated again — because they don’t live in the codebase.
Inferring Intent, Then Looking for Faults
A Catching JiTTest, as described in the paper A Catching JiTTest, is specifically aimed at regressions introduced by a code modification. The core insight is that an LLM can do more than just run tests: it can infer what the code change was trying to accomplish. With that understanding of intention, it can then deliberately break the code to see what happens.
The process works in a short sequence:
- New code lands in the codebase.
- The system infers the intention of the code change.
- It creates mutants — versions of the code with faults deliberately inserted — to simulate what could go wrong.
- It generates and runs tests designed to catch those faults.
- Ensembles of rule-based and LLM-based assessors separate genuine failures from false positives.
- Engineers receive clear reports only when an unexpected change actually matters.
This mutation-based approach lets the system generate tests that probe the boundaries of what the code intends to do, rather than simply re-running a fixed set of assertions that may have become irrelevant as the code evolves.
What the JIT Model Eliminates
The traditional testing paradigm asks engineers to anticipate not just the current behavior of their code, but all possible future changes to it. That inherent uncertainty is why traditional suites accumulate false positives and tests that assert nothing useful. Agentic development compounds the problem by accelerating the pace of change, making the cost of maintaining those suites prohibitive.
JiTTests remove that forward-looking burden. Because each test is generated fresh at the moment of a change, it can never go stale. The benefits accumulate:
- No maintenance: Tests are generated on-the-fly and don’t reside in the codebase, shifting effort from humans to machines.
- Change-specific: Each test is tailored to the particular diff, so it won’t break when unrelated code is intentionally updated.
- Always current: Tests automatically adapt as the codebase evolves, since they are regenerated for every change.
- Human-in-the-loop only when it matters: Engineers review test output only when a genuine bug is flagged, not when a test fails due to an unrelated update.
Because mutation-based fault analysis drives test generation, the system can keep false positives low. The signal it surfaces focuses on alignment with the code’s inferred purpose, meaning engineers can spend their time on real bugs rather than on triaging alerts generated by outdated test suites.
This represents a shift in testing infrastructure from a generic “does the code pass?” question to a far more targeted one: “does this specific change introduce a behavior that no one intended?” The answer to that question, delivered just-in-time, determines whether the change can go to production with confidence.
The full methodology, including implementation details from deployment inside Meta, is available in the paper Just-in-Time Catching Test Generation at Meta.
