A benchmark for defenders: DeepsecBench tests how well models find real vulnerabilities
Security teams and attackers now have access to the same class of AI tools. The difference is that defenders already know their own codebase, which gives them a structural advantage — if they scan before an external actor does. DeepsecBench, released today by Vercel, measures how effectively different models find vulnerabilities in application code, with results broken down by recall, precision, cost, and total scan time.
Rank | Model | Level | Score | Cost | Total time |
|---|---|---|---|---|---|
1 | GPT-5.6 Sol | xhigh | 35.58 | $55.98 | 03:39:00 |
3 | Claude Opus 5 | medium | 28.36 | $31.96 | 00:47:01 |
8 | Kimi K3 | high | 17.56 | $12.38 | 01:59:00 |
10 | Grok 4.5 | high | 15.58 | $5.60 | 01:24:00 |
The benchmark runs against an open-source repository pinned to a commit just before a large batch of vulnerability fixes. From that state, 50 entry-point files were selected and a golden set of 231 human-judged findings was built. Each model's overall score is a recall-weighted F2 metric (Score = 100 × 5PR/(4P+R)), where recall counts twice as heavily as precision. The weighting reflects a simple reality of security work: a missed vulnerability stays unpatched, while a false positive merely wastes time.
Findings outside the golden set are classified by a judge model as real or false, and those classifications feed into the precision score. Recall is measured only against the 231 known findings. Each model runs the benchmark three times, and the published numbers are the median of those runs.
Why scores stay low
The benchmark construction is deliberately opaque. The repository, commit, files, and findings are not disclosed, so models cannot train against the test set. A model that recited memorized fixes would score near-perfect recall; in practice, the best run found 30.7% of the golden set, and 20 of 25 runs came in under 20%.
Stronger results no longer require the most expensive models
When deepsec was first introduced, comprehensive scanning required frontier models that were costly to run at scale. The most capable models from OpenAI and Anthropic still top the leaderboard, but open-weight and more efficient reasoning models have narrowed the gap — and the price difference has shrunk faster than the performance difference.
Kimi K3 from Moonshot AI scores 17.56 on its high setting for $12.38 — roughly half the top score at about a fifth of the cost. Grok 4.5 on high delivers a score of 15.58 for $5.60. GPT-5.6 Sol on medium offers 25.10 at $17.95, the best score-to-cost ratio among the top performers. The full cost figures cover 50 files; a production codebase of typical size — around 100 times larger — would run roughly $1,200 for a Kimi K3 sweep or over $5,000 for the highest-scoring OpenAI model at current pricing.
Not every capable model appears on the board. Anthropic's Fable 5 is absent because it declines security work, including defensive tasks. The benchmark will add security-enabled versions when they become available.
Matching models to scan cadence
Security scanning is recurring work, so the practical question is which model to run and how often. Frontier models work well for periodic deep audits; cheaper options like Kimi K3 or Grok 4.5 can run more frequently. Reasoning settings also change the economics: dropping GPT-5.6 Sol from xhigh to medium moves it from a score of 35.58 over 3 hours 39 minutes to 25.10 in just over 30 minutes — fast enough to review new features before they reach production.
A small team might scan every merge with Grok 4.5 and reserve more expensive audits for milestones. A larger organization could run frontier audits on critical services, use the same model at lower reasoning settings on key pull requests, and keep a continuous Kimi- or Grok-class sweep over the wider codebase.
One endpoint for bursty scans
Security scans are spiky workloads that consume a large burst of tokens in a short window — capacity that is hard to buy from a single provider. Every DeepsecBench run goes through AI Gateway, which provides one endpoint for all models and handles routing, retries, and failover automatically, with no per-provider keys or rate limits to manage.
The same routing applies to production scans. Pass provider/model to deepsec, and a single AI_GATEWAY_API_KEY in your environment covers every model on the leaderboard, or use OIDC on a linked Vercel project.
pnpm deepsec process --project-id my-app --agent pi --model xai/grok-4.5
