From “More Coverage” to “Find This Bug”
Automated test generation has historically chased a single metric: code coverage. The assumption is that if enough lines of code are exercised, faults will surface. Meta’s Automated Compliance Hardening (ACH) tool starts from a different premise—describe the fault you care about, in plain text, and ACH will both manufacture instances of that fault and produce tests proven to catch them.
ACH is a mutation-guided, LLM-based test generation system. In practice, an engineer provides a natural-language description of a concern—for example, a privacy-related bug. ACH then generates a large set of mutants (faults injected into source code) that match that description, and in turn generates tests that kill those mutants: that is, tests that demonstrably fail when the injected fault is present. The description can be incomplete or even internally contradictory, yet ACH still yields tests with verifiable assurances that they catch the stated class of bug.
The tool is rooted in principles of Assured LLM-based Software Engineering, meaning the results carry guarantees rather than probabilistic hopes. The underlying research is detailed in the paper “Mutation-Guided LLM-based Test Generation at Meta.” While the deployed implementation targets privacy compliance, the mechanism applies to any category of regression testing.
Old Idea, New Execution
Mutation testing itself is decades old. The approach—deliberately introducing faults into source code (via version control, keeping them out of production) to see whether an existing test suite detects them—has long been studied but rarely deployed at scale. Earlier generations of mutation tools suffered from two compounding weaknesses.
First, mutants were frequently generated by rule-based transformations, which produced faults that were not realistic representations of actual concerns. Second, even when mutants were auto-generated, a human had to manually write the tests to catch them—a laborious process with no guarantee the hand-written test would actually kill the generated mutant.
LLMs remove both bottlenecks. ACH uses them to generate realistic mutants tailored to a stated concern, and again to generate tests for those mutants automatically. The workflow is straightforward:
- An engineer describes the kind of bugs they are concerned about.
- ACH generates many bugs based on that description.
- ACH generates many tests that catch the generated bugs.
Why LLMs Made the Difference
Meta operates enormous numbers of data systems across a range of programming languages, frameworks, and services. With thousands of engineers shipping code, the question of reliability—and, in compliance-sensitive areas, risk—becomes a scaling problem. LLM-based test generation and LLM-based mutant generation have each been demonstrated separately, but ACH is the first combination of the two deployed in large-scale industrial systems.
Because LLMs are probabilistic rather than rule-bound, they can straddle both sides of the mutation equation efficiently. Generating a mutant set and the tests to kill them—historically a painstaking process to scale—becomes tractable. The approach lets engineers take concerns from varied sources (previous faults, regulatory requirements, team feedback, user requirements) and turn them from freeform text into executable tests with a guarantee the test targets the described fault.
Deployments at Meta cover privacy testing on platforms including Facebook Feed, Instagram, Messenger, and WhatsApp. Feedback indicates engineers find ACH useful for hardening code against specific concerns—and that even tests that do not directly hit the stated concern provide auxiliary value.
The mechanism is agnostic to fault type, meaning ACH can harden against any class of regression. Planned next steps include expanding deployment areas, building methods to measure how relevant generated mutants are to real faults, and detecting existing faults—work aimed at broader industry adoption of automated test generation in compliance contexts.
