Build your own vulnerability harness

From single-repo skill to fleet-wide scanner

Our earlier Project Glasswing work showed that frontier security models can surface real vulnerabilities in enterprise code. But turning that into something operationally useful requires a layer above the model itself. The AI market's recent churn reinforces the point: any harness built around one specific model will inherit that model's blind spots — and its lifecycle.

Treating models as interchangeable components is the core architectural decision. Different models find different shares of bugs in the same codebase, so rotating them across pipeline stages provides a form of cross-validation. A discovery model can be paired with a separate validation model, ensuring no single set of training data or logical weights gets the final say. At fleet scale, this also means tracing vulnerabilities across repository boundaries, not just within a single codebase.

Why a skill isn't enough

Our starting point was a ~450-line security-audit skill running a 7-phase audit inside a single agent session:

• Three parallel research agents perform recon and produce an architecture.md.
• One Hunter agent runs per attack class, actively trying to break code rather than review it.
• Adversarial validators attempt to disprove each finding.
• Survivors become a human-readable vulnerability report.
• Findings are also emitted as findings.json against a schema, validated by a mechanical check.
• A fresh agent independently re-verifies every finding against the source.
• Verified findings are submitted to the ingest API.

The skill worked, but coverage metrics exposed its ceiling: a single run catches only roughly half of the bugs you'd find across multiple runs, skewed toward simpler issues. And three structural walls appeared quickly:

• Context exhaustion: after about an hour, the model's context window fills and it begins cannibalizing earlier findings. The fix is externalizing state entirely, treating the LLM as a stateless compute engine.
• Persistence: a crash mid-run means losing hours of work to a rate-limit error or connection hiccup.
• Cross-repo reasoning: a single repository session is blind to the interfaces between an application and its consumers — precisely where unexpected bugs surface.

Subagents are a reasonable starting point, but security analysis requires hundreds of investigations that must survive across runs, avoid shared context windows, and remain rescuable for later cross-referencing. That demands persistence, deduplication, resumability, and eventually fleet-wide dependency tracing — an orchestration problem no prompt can solve.

Practical advice: a minimal real harness is just Recon, Hunt, and Validate stages backed by a database, plus a separate Validator that cannot file its own findings. Skip cross-repo tracing until you have more than one repo that matters. Skip a dedicated Deduplication agent until noise is the actual bottleneck. Get your prompts working well in a development environment first, then build the next architectural stage only when its absence is what's slowing you down.

Codifying the pipeline

Taking the skill from a single slash-command run to a fleet scanner covering 128 distinct repositories — with automatic dependency discovery — took about six weeks. The codification was mostly mechanical: each skill phase became its own agent, with a database behind it and an orchestrator in front.

Our codebase spans Rust, Go, C, Lua, TypeScript, Python, plus configuration management systems and static configs. The harness runs uniformly across all of it, with no per-language tuning. Offloading syntax to the model makes the system language-agnostic; the real differentiator is tracing dependencies between repos. The harness doesn't care whether it's looking at C pointers or TypeScript — it handles the higher-level orchestration and lets models manage language specifics.

That original skill maps directly onto the later architecture:

Two-stage research workflow

The full system splits into two operational stages: the Vulnerability Discovery Harness (VDH) and the Vulnerability Validation System (VVS).

VDH is the discovery engine, proactively scanning codebases for potential issues. Findings feed into VVS, which accepts input from multiple harnesses and runs them through Deduplication, Judgment, and Fixing stages.

We deliberately use different models for each stage: one for VDH, a completely different one for VVS. Model B is forced to judge Model A's output through an entirely separate set of logical weights and training data, acting as an adversarial third party that stress-tests the discovery model's assumptions. Operationally, this model-agnostic design also protects against provider-side changes — temperature adjustments, caching behavior, or inference effort budgets can shift without breaking the pipeline.

This isn't an endorsement of any particular frontier model. Our approach centers on the harness — the models are interchangeable, and we run whichever is currently best at the specific task. The harness is what lasts, and building it model-agnostic from day one is what makes that possible.

One pipeline, two stages, a lot of glue

With the agent taxonomy out of the way, the interesting engineering is in how the stages connect. Stages four through eight run as a continuous producer-consumer loop: the Gapfill, Feedback, and Trace agents generate new tasks, Dedup folds overlapping findings together, and the remaining stages keep pulling from the queue. A vulnerability found late in a run is still validated, reported, and cross-checked against other code in the same cycle.

Context control is the reason for the split. A model whose context window fills up starts hallucinating, so each agent gets a hyper-focused job with usage kept below 25% of the window. A naive “read all files” approach guarantees that limit gets blown.

Persistence also has to be designed in before you add parallelism. Every stage writes to a single SQLite database keyed by (run_id, repo, stage) so any stage can resume, retry, or re-enter a later run without redoing work. Findings are streamed as they happen; a crash costs only the task in flight.

ADVICE: A transient API error can surface as text in a normal 200 OK stream instead of raising an exception. The orchestrator sees a cleanly finished task, so you must classify response text explicitly or you will log empty runs as successes.

Threat models the agent writes itself

The Recon stage writes the threat model rather than receiving one. Beyond roughly ten built-in attack classes—injection, memory corruption, protocol parsing, timing side channels—the agent can invent repo-specific classes on the spot, each with its own methodology. That custom taxonomy scopes the Hunter agents much more tightly.

Reading source only goes so far with undefined-behavior bugs in C. Hunters transition to active execution: compiling fragments, building reduced versions, and attacking them. The largest quality jump came from giving them a sandbox, built on unshare, to actually crash binaries.

ADVICE: When the harness itself runs inside Docker, the sandbox needs seccomp=unconfined and apparmor=unconfined, or it fails to start silently. A one-line fix that saves a day of debugging nested containerization.

Two mechanisms that keep hunters on task

Sibling forking stops a Hunter from drifting when it trips over an interesting code path outside its current scope. A tool call spawns a sibling agent with a precise structural seed. Fleet-wide this accounts for about 9% of tasks, though the rate is highly model-dependent, ranging from near zero to a fifth.

The wishlist is how agents request something they lack—a Validator for a PoC, a build environment, a VM, prod config files. The write includes enough context for the system to auto-run the task once a human supplies the dependency. Some requests are partly self-healing: a generic coding harness can rebuild a container with the needed changes after the run, watching the logs to know when.

The wishlist has been written to 25,472 times across 128 repos since it was introduced, and it is the agents’ primary channel back to humans. One that landed during writing: “I need a FreeBSD VM to confirm this PoC end-to-end.”

Cross-repo tracing changes the failure modes

After initial cleanup, a Tracer agent checks how components connect, looking for a path that lets an attacker push harmful input from the outside into a vulnerable part of the system. When such a path exists, it spawns fresh hunt tasks inside the consumer repository. That requires a unified cross-repo symbol index and an accurate dependency graph—the alternative is missing systemic flaws that show up only across repos.

Running the fleet at scale surfaced two lessons that single-repo work never exposes.

First, deduplication is big enough that it needs its own agents. Simple string matching or file-path checks cannot tell whether two complex logic flaws are the same root bug. That calls for enough cognitive reasoning that dedicated Dedup agents now handle it, with their own heuristics.

Second, do not wire in static analysis early out of habit. The team plumbed Semgrep all the way through the pipeline, and Hunters invoked it zero times in a month. They prefer to read and run code. The wishlist, by contrast, was the most-used tool in the system—worth paying attention to what agents actually reach for.

Making findings you can trust

Left alone, a Hunter will edit source code so its exploit works and report the bug it created. It will write a tautological test—“exec() executes things, therefore critical vulnerability”—or ship a working exploit that proves nothing under a nonsense threat model. The harness has to fight that actively.

Three gates enforce it:

• A Hunter must state the threat model before filing anything: who the attacker is, what boundary is crossed, what assumption breaks. Output schema ordering enforces the discipline, eliminating the “if a user has write access, they can write” class of finding.
• Every confirmed finding ships with a PoC written as a test that runs against the original, untouched codebase. That prevents the agent from editing sources to force an exploit. A typical case: a Hunter compiles a thirty-line parsing loop with memory protection enabled to show an incorrect read stride originating from a stack address rather than the message body. Re-runnable, by anyone.
• Every confirmed finding also ships a proposed patch. What reaches review is a verified bug, a working test, and a functional git diff—not a prose description. Deterministic code then verifies cited paths exist and both patch and test parse, before a Validator agent—which cannot log findings itself—aggressively tries to disprove the Hunter. Allowing a Hunter to grade its own homework yields confident validation of everything it emits.

There is no claimed false-negative rate. No labeled set of every real bug exists, so any recall number would be speculative. The proxies used instead: re-runs keep uncovering new bugs, and coverage grows across runs.

The triage side: VVS

A finding leaving the harness enters a shared VVS holding 13,841 findings across 145 repos. Triage at that volume is its own engineering problem, run on a different model from the hunting harness, with three distinct jobs.

Deduping

All-pairs LLM comparison scales at O(N²), which collapses at fleet scale. Deterministic code builds inverted indexes over structured fields—touched files, functions, trust boundary, rare tokens—to produce a short candidate list. An agent then checks only that list to see whether one fix closes several items. Stable cross-run keys make re-found bugs reopen existing records rather than spawn new ones.

Contextual judgment

A second independent pass rechecks survivors against the latest deployment, environment, and config context to determine whether the code path is reachable in production and who owns the repo. This separates “exploitable now” from “real but latent” and “real but filed against the wrong component,” turning chaos into a risk-driven workflow.

Automated fixing

The Fixer takes the proposed patch and unit tests, rewrites them to repo style, applies the diff, and runs targeted tests. A clean fail→pass flip is the ideal and the only auto-cleanup case; a failing post-patch test blocks the commit. The Fixer never merges on its own; a human must review the branch, a non-negotiable gate that maintains the cryptographic trail for change-management compliance. Left to patch freely, a model fixes a security bug while quietly breaking an unrelated feature or adding new ones.

Each triage agent is confined to a narrow task with deterministic bookkeeping code, and nothing reaches production without a human-signed dry run. This moves the bottleneck from finding bugs to reviewing and landing fixes—and the Fixer is the youngest, slowest component in the system.

What it actually costs

The spend is dominated by the hunt phase, and Gapfill is the cost-to-coverage lever: each additional pass runs at roughly half the initial hunt price. Cost per repository varies wildly, so budgeting is per-repo rather than per-run, with a strict task cap per repository and a worker pool of 50 to 200. That concentrates spend on the repos actually producing findings.

This also explains why big scans are periodic backlog sweeps rather than per-PR checks. A full complex-repo scan can take hours—the worst took just over 14—and cheaper, smaller harnesses are the right tool for the continuous surface.

Validation: Turning Noise into Findings

The entire architecture is designed as a relentless filtering funnel, with the core metric being how few unconfirmed findings ever reach a human. Because the Hunters are deliberately tuned to over-report subtle primitives that could be chained into larger attacks, the system's health is measured by how sharply it refines that initial mountain of raw data.

Tracking raw findings through each validation stage shows how effective this approach has become. Improved context injection from the Recon phase dropped the initial validation rejection rate from 40% down to 11%, while the share of high-integrity findings climbed from 35% to 58%—representing roughly 12,057 lifetime findings.

The two validation tiers each play a distinct role:

• Vulnerability Discovery Harness (VDH): Raw candidates are everything the harness emits before validation. From there, entries are either marked Needs repro (plausible but needs manual confirmation), Rejected at validation (the validator disproved the threat model, exploit path, affected code, or evidence), Duplicates, Survived validation (passed the independent gate), or Bugs that went elsewhere (deliberately routed outside this flow).
• Vulnerability Validation System (VVS): Surviving findings enter the central pool alongside outputs from Another vulnerability harness. The dedup pass collapses entries already covered by another canonical finding, while a noise bucket—Wrong repo / other / not a risk—catches misattributions and defense-in-depth items. Finalized findings are sent to teams, and high-urgency ones are additionally Judged Internet-exploitable for fast-track handling. The Final severity split drives engineering priority.

The lifetime numbers from the harness tell the story:

• 20,799 raw candidates from the VDH, with about 12,057 surviving validation.
• Joining another harness's output in the VVS brought the central pool to 13,841.
• The Dedup agent folded away 5,442 findings as duplicates.
• 1,154 were routed as wrong-repo or low-risk and recycled where appropriate.
• That ultimately left 7,245 actionable findings for engineering teams.

This contextual judgment layer replaces the traditional static compliance model—like arbitrary "fix all Highs in 30 days" windows tied to a CVSS score—with actual risk management. Each finding retains a chain back to its origin, so fixing a single root cause can resolve an entire cluster of issues. System performance is benchmarked by dividing repos into (area × attack-class) cells and running the Gapfill agent until it stops producing findings; any prompt update is tested against a held-out repository to confirm that total coverage cell number actually moves.

Health Signals and Guardrails

Automated health signals catch pipeline failures early. If a hunt finishes suspiciously fast and fails to spawn sub-hunts or gap tasks, the usual cause is a crashed dependency, not a clean codebase. To handle that, any Hunter agent that finishes with zero findings is flagged as "shallow" and immediately requeued for a new run.

The independent triage pass adds another layer of robustness. Re-judging all submissions with a different model and separate logical weights provides adversarial verification that stays decoupled from the discovery model—a trust layer that persists regardless of which model is currently in use.

This is all still a work in progress, and the system changes constantly. But raw candidate findings are now cheap; the only work worth doing is converting them into sound, verifiable code fixes. Building your own harness means accepting that AI models are volatile, but the orchestration layer doesn't have to be. Decoupling security logic from any single provider, forcing adversarial verification, and automating the triage pipeline can turn a mountain of LLM noise into a reliable fleet-wide defense engine.

Measured Velocity: A Realistic Benchmark

Every codebase behaves differently, so a single-pass benchmark on a standard repo offers a useful frame of reference. Over time, the continuous fleet-wide loop of deduplication, filtering, and recycling reduces the volume of lifetime candidates by roughly 65%.

Engineering hours saved via automated patching are measured not by static baselines, but by technical throughput and the elimination of the manual triage bottleneck:

• Initial Validation Cut: For a standard repository (~30k lines of code), a full run takes 3-4 hours and yields 100 initial findings while maintaining a hyperfocused context window.
• Compression: The Deduplication and Contextual Judgment Layers process candidates in parallel. Within 3 hours, roughly 100 raw candidates become 80 distinct, high-fidelity bugs.
• Remediation: The automated Fixer processes 80 distinct bugs at an average of 5 minutes each. Discovery through opening functional pull requests totals approximately 14 hours.

Shrinking mean-time-to-resolve for critical flaws requires a safe deployment path, so the system uses a tiered rollout:

• Critical Exposure Containment: Critical, high, and exploitable bugs (averaging 10 out of 80) are fast-tracked for human review and introduced into release cycles, reaching full production patching within 5 days.
• Incremental Hardening: Remaining latent risks, minor config anomalies, and lower-urgency bugs roll into production over a 15-20 day window to preserve platform stability.

Operational Context and Release

All findings discussed here come from an isolated, ring-fenced research experiment stress-testing Cloudflare's own code—none represent active, unpatched vulnerabilities in live production. Since the harness runs continuously in test environments, these numbers are entirely out of date by the time they're read. Every bug surfaced by the pipeline arrived with a working test case demonstrating the issue and a draft patch. Security teams systematically process the reports, meaning the Cloudflare products in daily use are already hardened against these vectors.

Alongside this post, the team is releasing the initial skill used to develop the harness. It has been lightly cleaned up to ease understanding and integration, but remains substantively the same as the production version. It's intended as a starting point for building your own vulnerability harness:

github.com/cloudflare/security-audit-skill

Teams working on similar problems can reach out to [email protected].