Agentic security at three stages

Figma has spent the past year running security agents across the entire software development lifecycle. The agents operate at three distinct points: when code is generated, when pull requests are reviewed, and when historical code is audited. All of them follow a single, shared policy.

The engineering challenge is not pointing agents at the codebase and asking them to find vulnerabilities—that part is easy. The hard part is achieving acceptable precision and recall. Precision measures the share of reported findings that are real vulnerabilities; high precision means few false positives. Recall measures the share of real vulnerabilities the system catches; high recall means few false negatives.

The team prioritized pull request review as the first capability to build. That decision was strategic: PR review became the proving ground for the policy that all other agents would later follow. The lessons learned there directly enabled secure code generation and historical auditing, because the policy could be developed and automatically improved based on PR review results.

Building trust in agent findings

Rolling out agentic security requires more than just deploying a tool. Engineers need to trust the agent's output, or they will ignore it. Figma's approach centers on iterating on precision and recall metrics, using the PR review stage as the primary feedback loop for refining the underlying policy.

Because all security agents share one policy, improvements made during PR review propagate automatically to the other two stages. This design choice means that a fix discovered while reviewing a new pull request also improves the quality of audits on the decade-old monorepo and the scrutiny applied to newly generated code.

The result is a system that guards code as it is written, reviews every pull request, and continuously audits legacy code—all without requiring separate, stage-specific configurations or duplicated engineering effort.

From prototype to production: How Figma hardened its AI code reviewer

Figma’s journey from a shadow-mode experiment to a merge-blocking security control took roughly four months. The critical early bottleneck wasn’t model capability — it was precision. In the first week of running Anthropic’s Claude Code Security Reviewer internally, only 4 of 27 findings (about 15%) were valid. That trust deficit is exactly the problem OpenAI has argued makes precision more important than raw recall: developers stop reading comments from a tool that cries wolf.

Building a threat model as an agent policy

To drive precision above the 70% bar Figma’s team set (a target based on the intuition that developers will engage with a tool where seven in ten comments are worth reading), the security team replayed the reviewer over eight weeks of historical pull requests and hand-labeled the false positives. From that work, they wrote a policy document built not on abstract rules but on concrete precedents — contextual examples explaining why a finding is or is not valid in a given part of the codebase.

The distinction matters. Rather than a rule like "don’t flag SQL injection in dbops", the policy says "dbops is only run by highly privileged operators who already have direct database access." That phrasing preserves the security reasoning an agent needs.

The resulting document ran 99 lines and roughly 2,560 words containing 68 precedents. It had an unintended side effect: Figma had essentially written a complete threat model of its codebase at a resolution suitable for an agent — something that had previously been scattered across docs, incident learnings, and institutional knowledge.

Iterating on the architecture

With precision above 70% on a two-week lookback, Figma enabled developer-facing comments and, within a month, made agentic review a merge requirement. That forced the tool to become boring, dependable infrastructure. Retry policies ensure a model vendor outage can't let a PR through unreviewed; Datadog and Slack telemetry alert on precision or recall drops; and fix-rate tracking keeps the focus on remediation instead of merely surfacing issues.

By December 2025, Figma had outgrown the stock GitHub Action and rebuilt around it in three ways:

  • Migrated to a TypeScript service. The workflow-based fork became hard to maintain, so Figma pulled it into a small service that runs Claude Code and Codex independently with cleaner observability and retry behavior.
  • Ablated the prompt. Newer models handle context, long-context retrieval, and basic vulnerability knowledge well enough that the original two-pass structure (finding candidates, then adversarially filtering them) collapsed into a single prompt. The models don't need OWASP reminders or instructions on how to hunt bugs.
  • Added an adjudicator. Analysis of missed bugs showed the reviewer was talking itself out of true findings with excuses like "pre-existing pattern" or "low confidence" while chasing precision. Since the first pass already emitted dropped candidates as structured output, a second pass now re-examines those borderline dismissals. This raised pass-rate recall by roughly 30% relative on known-bad commits.

Measuring what matters

Figma deliberately avoids academic-style eval confidence intervals. Instead, it runs two scoring types on a corpus of every real vulnerability that previously slipped past human review and static analysis:

  • Pass-rate scoring measures the percentage of known bugs the reviewer catches.
  • Payout-weighted scoring normalizes results by the dollar value each bug carried (either an actual HackerOne payout or an estimate), reflecting that impact clusters in a few high-value bug classes.

The corpus now holds 66 tasks, each a real historical vulnerability found by bounty hunters, incidents, or audits — 46 from HackerOne alone, 24 of those from a single researcher. Each task is a YAML file containing the root-cause commit, a description, a weight, and tags. The eval framework checks each commit into an isolated worktree and uses a simple LLM-as-judge to score whether the reviewer's finding matches the known issue.

On the latest frontier models, the union of Claude Code (Opus 4.8, extra-high effort) and Codex (GPT-5.6 Sol, high effort) catches 75.8% of all bugs that previously defeated every existing control. Since both tools miss different bug classes, Figma runs each model and surfaces a finding if either one raises it.

The per-PR cost is modest: roughly $0.50 median spend, since most pull requests have nothing to flag. That economics plus avoided bounty payouts makes the system self-funding.

Quantifying the policy's value

To isolate the contribution of its engineering effort, Figma ran the same reviewer with its entire policy ablated — just "find vulnerabilities," the model, and its tools. That empty-prompt version cleared 44.4% of the corpus on payout weight. With the full policy reinstated, a single reviewer climbs to 64.2%. Precision is where the gap widens; with no precedents to lean on, an unconstrained prompt flags every suspicious pattern. The eval corpus can't quantify that effect (it contains only true positives), but developers running unconstrained agents on their own repos have surely lived it.

Precision itself is measured in production, not offline, because real findings are rare enough that any standing benchmark would require hand-checking an impractical number of pull requests. Instead, PR authors give the first thumbs-up or thumbs-down on each finding, and a separate agent periodically re-checks current code behind recent findings. Disputed cases go to security on-call for a final call. On Opus 4.7, Figma held zero disputed findings for a sustained month-long stretch; the move to Opus 4.8 brought a small precision dip, accepted as a tradeoff for deeper multi-step exploit-chain detection.

Improving the agent automatically

Both recall and precision improvements run through self-improvement loops that feed back into the policy under human supervision.

For recall, when a bounty researcher reports a real vulnerability:

  1. A cloud agent triages the report and filters the roughly 75% of submissions that are low quality.
  2. Recognizing a rare exploit chain, the agent validates and writes a fix; security on-call reviews and merges.
  3. The agent traces the root-cause commit and adds it to the eval corpus.
  4. A second agent runs the existing PR reviewer against that commit; if it misses the bug, the second agent reads the reviewer's chat transcripts to determine why (file never read? read but not flagged? flagged but excluded?) and updates the policy until the bug is caught.
  5. Security on-call reviews the policy change before shipping.

This loop converts each new vulnerability into durable coverage for its entire bug class. Humans stay in the loop because automated policy refinement can swallow Goodhart-style overfitting or produce verbose amendments nobody wants to read.

For precision, when a PR author dismisses a finding as a false positive, an agent skill updates the policy to prevent that class of error, incorporating developer feedback and reviewing chat transcripts. Security on-call reviews the change and the eval suite re-runs against the established corpus.

Given that any user-controlled text reaching an agent opens a prompt-injection surface, Figma enforces hard trust boundaries. A read-only validation agent screens for injection; fix-writing agents run in sandboxes with egress controls; CI for agent-generated PRs has no access to deploy credentials; and nothing merges without human review, SAST, and agentic review all passing. For externally reported bugs, Figma considers reading code sufficient for validation — generating an exploit is unnecessary given its monorepo structure.

The fix-rate metrics reveal that getting bugs surfaced isn't enough. Real findings compete with everything else on a developer's plate, so Figma tested hard merge-blocking but rejected it as too heavy. Two lighter changes worked: a bolder comment footer demanding action instead of apologizing for noise, and a strict output format (one-sentence finding, numbered exploit steps, recommendation, code links) that eliminates the wall-of-text habit language models tend to fall into.

Scaling audits across the monorepo

Figma applies the same policy and threat model used for PR review to full repository audits. The difference is scope: the agent and policy are pointed at the entire monorepo rather than a single pull request. This matters both for the ten-plus-year-old codebase's historical code and for the steady state, since models are stochastic and a single pass may not surface everything, and the threat model itself keeps evolving as new model generations catch what previous ones could not.

Auditing is harder than review for both agents and humans. A ten-year-old codebase offers no natural starting point, so Figma made the task tractable by brute-forcing it with many agents. They picked a cost budget, converted it into an agent shard count, and scanned — mostly sharding by file or application route. Two adjustments helped: ablating low-severity bug classes to focus budget where it matters, and adding a lightweight adversarial review pass to refute findings.

The first run found more than a hundred latent vulnerabilities, including two criticals that traditional SAST tools missed, both patched immediately and added to the eval corpus where they became its two highest-scored tasks.

More recently, Figma has experimented with dynamic workflows for repo-wide audits, with promising early results but unpredictable token usage. As these systems mature, they may replace the multi-harness approach, allowing simple prompts like "find vulnerabilities" pointed at the threat model, with parallelized scans and built-in adversarial and adjudication passes.

Guiding secure code generation with hooks

An agent hook is a script that runs automatically at a defined point in an agent's loop — before or after a tool call, after a user prompt, at session start, and so on. Like git hooks, but firing on agent actions rather than git operations, they let you inspect, block, or modify what the agent does.

Figma found hooks steer agents more reliably than the same guidance placed in AGENTS.md. For some bug classes like logging safety, the drop after adding a guidance hook was about 50%. They maintain a vendored, agent-agnostic version of Anthropic's security guidance plugin, which provides two sets of agent hooks:

  • One injects just-in-time guidance the moment an agent touches something risky — adding a route, changing a permissions policy, or logging telemetry.
  • The other hands the diff to an LLM periodically or, on commit, kicks it to a second coding agent for a look, running on the same policy as the reviewer and auditor.

Hooks also lack the high precision bar of PR review because they are mostly invisible to the human author. That lets them nudge toward good practices, not just away from vulnerabilities. When an agent adds a new API route, a hook can steer it to attach the proper secure-by-default authorization decorator and tests, not merely surface authorization bugs.

Lessons learned

The specifics depend on company size, risk profile, and existing feedback loops, but the main lesson is to improve precision before recall. The order is counterintuitive because historical bugs can only measure recall; they barely help with the precision you must fix first.

Precision and recall come from different places. Recall comes from replaying the reviewer against known bugs and counting misses. Precision has no fixed target to replay and shifts with every policy change, so there is no standing offline eval. Judging enough of the rare live findings would require combing through mostly benign PRs at an unsustainable scale. In production, the reviewer runs on every PR anyway, so the author makes the first call, with another agent re-reading the current code behind each finding and sending disputed or unaddressed cases with flagged patterns still present to internal security experts.

A few more findings:

  • Put a triage agent with source access on every incoming bug bounty report. A newer kind of bug bounty researcher chains low-severity vulnerabilities into high-impact exploits. Fast triage builds trust and treats these researchers as valuable security assets. The reviewer alone can't catch novel bug classes outside the threat model — those still fall to internal experts and external sources. Manual triage is too slow to sustain the feedback loop; an agent enables fast triage that feeds better evals and drives better recall.
  • Make agents review for good practice, not just for bugs. Have agents check that code uses the right secure-by-default frameworks and has real test coverage. Skip this and the codebase drifts toward something neither a human nor a model can reason about. Figma runs separate agents across all three stages to guard against anti-patterns and quality issues well beyond bug finding.