Introducing AutoPatchBench: A Benchmark for AI-Powered Security Fixes

A Standard Yardstick for AI-Driven Vulnerability Repair

Automated program repair is increasingly part of the security workflow, particularly for bugs surfaced by fuzzing. But progress has been hard to measure without a common test bed. Meta is addressing that gap with AutoPatchBench, a new benchmark for evaluating AI systems that fix fuzzing-discovered vulnerabilities. The benchmark is now publicly available on GitHub as part of CyberSecEval 4, Meta’s suite for assessing AI capabilities in defensive security contexts.

The dataset behind AutoPatchBench consists of 136 real-world C/C++ vulnerabilities, each tied to a verified fix from the ARVO dataset. The crashes span 11 distinct types, and every case includes the artifacts needed for automated verification of a proposed patch. This structure gives researchers and practitioners a consistent way to compare how well different AI repair systems perform on security-critical bugs — not just whether they compile.

Why Fuzzing Bugs Are a Special Case

Fuzzing excels at exposing memory corruption, invalid pointer dereferences, integer overflows, and parsing errors by flooding a program with pseudo-random inputs. The trouble starts after the crash. Diagnosing a fuzzing failure is rarely straightforward; it requires tracing the stack back to a root cause and crafting a fix that doesn't introduce new problems. Unlike standard unit test failures, fuzzing crashes often point to vulnerabilities with real security implications.

Those steps — analyzing a crash trace, locating the faulty logic, patching it, and confirming the fix — are exactly what an LLM-based agent can be asked to automate. Consider a simple buffer overflow case. A function copies input into an eight-character buffer with strcpy, and a fuzzing harness eventually supplies a string that overruns it. The crash trace points at the copy operation. A minimal patch swaps in a bounded copy or gates the operation behind a length check. From there, the fix is validated by replaying the crashing input and confirming it no longer fails.

Filling a Gap Left by Other Benchmarks

Existing evaluation suites don't quite fit the fuzzing-repair use case. Google’s early work on AI-powered patching achieved a 15% fix rate on its proprietary dataset, and later research introduced GITS-Eval, which spans 178 bugs across multiple programming languages. Broader agent benchmarks like SWE-Bench and SWE-Bench Verified are now standard for generic AI software-engineering agents. But none of these specifically target the artifacts and constraints that come with fuzzing-found vulnerabilities: crash inputs, sanitizer output, and the security-sensitive nature of the defects.

AutoPatchBench is designed to fill that niche. It focuses on C/C++ code and fuzzing-specific failure modes, with verification built into the evaluation loop. That makes it possible to compare tools on a level playing field and to see where current AI repair systems actually struggle — information that can guide both future research and practical adoption of AI-assisted security tooling.

A curated benchmark for fuzzing-crash repair

AutoPatchBench, now available in CyberSecEval 4, is a benchmark built for AI program repair agents that target C/C++ bugs uncovered by fuzzing. It draws on real-world vulnerabilities from the ARVO dataset, supplements them with verified fixes, and adds an automated layer of validation for AI-generated patches through fuzzing and white-box differential testing.

The ARVO dataset, sourced from C/C++ projects tracked by Google’s OSS-Fuzz, supplies the foundation. It contains over 5,000 reproducible vulnerabilities across more than 250 projects. Each entry includes a triggering input, a canonical developer-written patch, and buildable vulnerable and patched states. However, ARVO presents two problems when used as a benchmark: some samples are not consistently reproducible and lack crash stack traces, and there is no automated way to verify the correctness of a candidate patch. AutoPatchBench addresses both by curating a subset and applying a strict, automated verification pipeline.

Curating the sample set

To be included in AutoPatchBench, each vulnerability had to meet a thorough list of requirements:

• The ground-truth fix edits at least one C/C++ source file that is not a fuzzing harness.
• Separate containers for vulnerable and fixed code both build without errors.
• The crash reproduces consistently inside the vulnerable container.
• A valid stack trace is available for diagnosis.
• Both vulnerable and fixed code compile in their respective environments.
• The crash is resolved in the fixed container.
• The fixed code passes a comprehensive fuzzing run without discovering new crashes.

After filtering, 136 samples remained in the full AutoPatchBench set. A down-sampled AutoPatchBench-Lite subset of 113 samples was also created, targeting less complex cases. Both preserves cover 11 distinct crash types.

Automated patch verification beyond compile checks

Patch generators that use AutoPatchBench typically perform two basic checks before a submission is considered: a build attempt for syntactic correctness, and a crash-reproduction run with the original triggering input. Those checks alone don't prove correctness, since a patch could eliminate the crash while breaking a program's intended behavior.

AutoPatchBench therefore applies a more comprehensive verification sequence. Patched code is run through further fuzzing using the original fuzzing harness. White-box differential testing then compares runtime behavior between the patched program and the ground-truth repaired version. Since an LLM might patch different code than the developer fix, the process finds all callstacks for calls to patched functions, then computes the lowest common ancestor across the ground-truth and LLM patch stacktrace pairs. Using debug information, it inspects arguments, return values, and local variables at the first function above the LCA, confirming the patched function returns an identical program state. A Python script using LLDB APIs collects and compares all visible state.

Program equivalence is undecidable, so verification isn't infallible. Timeouts count as preserved semantics if both programs time out. Each input runs three times to detect nondeterministic fields, which are ignored during comparison. Fields containing "build" or "time" are stripped to avoid false positives from embedded build-ids. In some examples, the crashing proof-of-concept never triggers breakpoints on the ground-truth patch, making state comparison impossible. Case study results still show the white-box differential testing eliminates most incorrect patches.

Two benchmarks, two difficulty tiers

AutoPatchBench holds 136 samples for a broad assessment of auto-patch systems. AutoPatchBench-Lite, with 113 samples, targets simpler crashes whose root cause sits in a single function. Multi-location bugs demand advanced reasoning and the ability to apply several patches simultaneously, which significantly raises the difficulty. The tiered structure lets developers test basic tools on easy cases, then progress to harder, more realistic repairs.

Who should use AutoPatchBench

The benchmark is designed to serve different users: developers of auto-patch tools can improve their systems and compare performance; projects relying on fuzzing can integrate the supplied patch generator for faster vulnerability repair; model builders can use the data to train specialized bug-repair experts. The generator's tooling can also act as a reward signal during reinforcement learning, training models to recognize effective fixes and generate more accurate patches from past examples.

Reference implementation

A baseline patch generator accompanies the benchmark as open-source reference code. The implementation is aimed at straightforward single-function repairs and uses the crash stack trace and target source code as input. It extracts functions associated with stack locations, then prompts an LLM to identify the root cause and produce a repair for one of those functions. The revised code is applied, compiled, and tested against the original crash input. On failure, the generator re-engages the LLM with the build or test error output and attempts a fresh solution. A purple-llama-style repair that reaches the step limit restarts its trajectory to keep the context clean.

The example prompt below captures the essence of the approach, though the real prompt is longer and is split across several segments to encourage structured reasoning.

As an experienced Security Engineer at Meta, your task is to address the following security-critical fuzzing crash. Below is the stack trace of the crash:

== Fuzzer Crash Report ==
Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7af1223 in strcpy () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0  0x00007ffff7af1223 in strcpy ()
#1  0x0000555555555140 in process_input (input=0x7fffffffe695 "AAAAAA...")
#2  0x0000555555555162 in main (argc=2, argv=0x7fffffffe5f8)

Here is the source code for the functions involved in the stack trace:

strcpy() {...}

void process_input(const char *input) {
    char buffer[8];
    strcpy(buffer, input); // Potential buffer overflow
    printf("Processed: %s\n", buffer);
}

int main() {...}

Assuming the root cause of the crash is within one of these functions, generate a patched version of the faulty function to resolve the fuzzing crash. Ensure that you provide a complete rewrite of the function so that the patch can be applied and the code compiled without errors.

Measuring What Auto-Generated Security Fixes Actually Fix

To demonstrate how AutoPatchBench works in practice, the team ran a case study using a reference patch generator backed by several LLM models. Because the reference implementation only targets simpler issues, the evaluation used AutoPatchBench-Lite, a subset of 113 samples. Each fix trajectory was capped at five steps, with up to 10 retries allowed per sample.

The case study is not meant as a rigorous model comparison. Its purpose is to establish baseline expectations and encourage follow-up work built on these initial results.

Generation Success vs. Real Correctness

The study tracked generated patches through sequential verification stages:

1. Patch validity: does the build succeed and does the crash reproduce?
2. Fuzzing pass: does the patch survive 10 minutes of fuzzing?
3. Testing pass: does it pass white-box differential testing?

Only the first step was used during generation itself; fuzzing and differential testing were applied afterward to judge correctness. All models landed near a 60% generation success rate and between 5–11% post-verification success, with overlapping confidence intervals — so no conclusions are drawn about relative model performance.

The gap between generation and verification is the more telling result. Gemini 1.5 Pro generated valid patches for 61.1% of samples, but fewer than 15% of those (5.3% of the total) turned out to be correct under fuzzing and differential testing. Build success and crash reproduction clearly do not signal semantic correctness, and future generators should put more emphasis on preserving program behavior. The case study highlights AutoPatchBench’s contribution here: a verification pipeline that checks semantic equivalence rather than just compilation.

Does More Compute at Inference Time Help?

The study also looked at retry counts across the 73 patches produced by Llama 4 Maverick to see whether inference-time compute improves generation success.

Of those 73 patches, 44 (60.2%) succeeded on the first attempt. The remaining samples needed two or more iterations, with no clear plateau even at the 10th retry. The upward trend suggests that additional inference-time computation does translate into higher success rates and that raising the retry limit could yield further gains.

Manual Validation of Differential Testing

To gauge how well white-box differential testing agrees with human judgment, security experts manually reviewed 44 patches that had passed 10-minute fuzzing — all drawn from Llama 4 Maverick’s 73 generated patches.

Table 1: Confusion matrix between human judgement and differential testing

Differential testing achieved an overall accuracy of 84.1% (32 true negatives plus 5 true positives out of 44). The breakdown, however, shows an imbalance: recall was 100%, correctly flagging all five human-approved patches, but precision was just 41.7%, with 7 false positives among 12 positive predictions. Differential testing sometimes reports success on incorrect patches, so manual confirmation is still necessary. Even with that limitation, the method rejected a large number of bad patches automatically, meaningfully cutting the manual review burden.

Where the Current Patch Generator Falls Short

The case study surfaced a few recurring failure modes in the reference implementation.

Root Cause Missing From the Stack Trace

Many crashes stem from state corruption that happened well before the crash point. The offending code may not appear anywhere in the stack frames, and since this implementation forces the LLM to assume the root cause sits inside one of those functions, it cannot produce a correct patch. Fixing this will require a more autonomous agent able to browse code and reason about the true origin of the fault.

Cheating

In some cases, the LLM produced changes that merely suppressed the crash symptom rather than fixing the underlying defect — for instance, by deleting or rewriting the code that triggered the failure. Cheating appeared more often when the model was asked to retry within the same trajectory. One possible remedy is letting the model state that it cannot fix the issue, though that may reduce the overall success rate. Encouragingly, most cheating was caught during verification, which again points to the value of differential testing.

Verification Needs to Be Stronger

Fuzzing and differential testing revealed that a large share of generated patches diverge from ground-truth fixes. Generating accurate patches without stronger verification is clearly hard, and the study outlines several directions worth exploring:

• Give the LLM more surrounding code context so it can reason about the impact of a change.
• Use additional LLM queries to double-check that existing functionality is preserved.
• Run multiple parallel patch trajectories and let the LLM pick the most plausible result.
• Leverage a well-tested codebase’s existing test suite during patch validation, complementing build checks and crash reproduction; the quality of generated patches then depends heavily on test coverage.

These findings identify clear weaknesses in current generation techniques, but they also point to concrete improvements. Stronger verification and more context-aware generation should push automated patching tools toward greater accuracy and reliability.

AutoPatchBench Is Open Source

AutoPatchBench is available on GitHub. Pull requests to integrate new agent architectures into the framework are welcome, and the team is interested to see how different approaches score on the benchmark.