Building an agentic harness that outlasts the model (2026) - Shopify

Why the security harness matters more than the model

Shopify’s Application Security team built an agentic code review harness that scans public-facing surfaces for vulnerabilities, confirms them with real integration tests, and opens Shopify-specific pull requests with fixes. Strict verification criteria mean findings are proven before developers see them, not merely reported as candidates.

After five months of evaluating security-tuned frontier models inside that harness, the team reached a counterintuitive conclusion: the best new models outperform baseline frontier models at discovering findings, but improved hunting creates more noise that needs confirmation. In one audit, a model surfaced more than 30 candidate vulnerabilities; after validation, every one was downgraded, rejected as a false positive, or reclassified as defense in depth.

The model improves, but the harness remains the durable asset. The team’s scanning workflows run across Shopify’s most important public-facing applications inside one of the largest Rails monoliths in the world. Over roughly six weeks, thousands of scans produced more than 300 findings ranging from defense-in-depth improvements to resolved security incidents. Two findings would have been rated Critical by the internal severity calculator, and the team conservatively values the total at over $400,000 in equivalent bug bounty payouts.

How the orchestration pipeline works

Dispatch, Shopify’s internal harness orchestrator, is a thin Ruby client that abstracts agentic scanning complexity from scan authors. Authors create category-specific bug-hunting agents, which run in an ordered set of stages inside the pipeline.

On a first run against a target application, the pipeline builds code partitions based on size and scope, then runs a full scan with Hunting agents in parallel per partition. Initial runs also generate reusable artifacts documenting available APIs and data models. These artifacts condense application-specific context, letting downstream agents keep most of their context window focused on the primary task instead of re-discovering application structure. All scan data and artifacts persist in a custom Rails backend.

Follow-up runs compare the latest commit with the previous one and scan only the changes in the diff. That reduces token spend significantly while maintaining coverage, and any new information found in the diff updates persisted artifacts such as partitions and documentation. Both generalist and vulnerability-specific hunting agents participate, each encoded with multiple layers of software security guidance and Shopify context.

Building test oracles for web vulnerabilities

Criticality is harder to confirm for web vulnerabilities than for memory safety issues, which have direct oracles via compiler flags. Shopify’s Environments team maintains local developer tooling that supports mature integration, unit, and functional testing pipelines across target applications. The Verifier agent taps into those existing test primitives to author and embed its validations.

Agents encode strict guidelines defining what counts as a “proven finding.” For the IDOR-focused Verifier, those rules include:

• Work backwards from public call sites, using internal tooling to search across all web clients.
• Create fixtures belonging to two different tenants.
• Exercise as much of the public stack as possible — Model-layer unit tests alone cannot determine exploitability.
• Extract impactful cross-tenant data. Booleans, raw IDs, and similar values are downgraded to “Low.”
• Evaluate above and below the current layer for upstream or downstream controls that reduce severity; do not stub important controls.

Findings that cannot be proven under these constraints are rejected or downgraded to Low or Medium, depending on potential criticality. The lower ratings include findings unlikely to be exploitable or requiring an unusual prerequisite like a secret token.

Consistency comes from specialization

The team’s earliest experiments used security-generalist multi-agent workflows run manually against repositories. Results included some true positives but also a high volume of theoretical issues irrelevant in practice. Proposed fixes were often incorrect or phrased in theoretical terms, and output was very non-deterministic.

Working from first principles, the team found this the least effective approach. Without a clear vulnerability category, a generalist agent fills its context window chasing an unordered list of potential findings, then either abandons the task or gets forced into compaction before locating the real vulnerability. Generalist agents can still close gaps left by category-specific agents if encoded with rigorous guiding principles, but their results require additional scrutiny because accuracy and recall tend to be lower.

Partitioning for cost and recall

Pointing a full agentic pipeline at every file in a repository is the most expensive way to perform risk discovery. Running one agent across the entire repository looking for any vulnerability is the cheapest. Cost and speed trade off against accuracy, recall, and finding quality. Partitioning balances those forces: agents group files into subsets that each contain 20–30% of a model’s context window in token count, leaving room for code exploration and tool use. Files are grouped by domain, purpose, or feature, and each partition includes global dependencies or important shared context.

To validate the approach, the team re-introduced previously confirmed vulnerabilities into local applications and compared partitioned against non-partitioned scanning. Partitioned runs improved accuracy and recall while keeping costs reasonably low. Tuning partition size and scope to the vulnerability category lands on a generally correct scope.

Determinism where it helps

Agents should receive deterministic scripts inline, or dedicated skills that call scripts, whenever structured input or output is needed. For example, a simple skill can enforce JSON document structure programmatically, letting agents populate documents rather than own the whole process. Deterministic scripts mean fewer malformed outputs and verifiable, parseable results.

The harness, not the model, is the investment

Each new frontier model finds more candidate vulnerabilities than the last. But a better hunter also produces more confident-sounding noise, and noise sent to a developer is worse than no finding at all: it wastes time and erodes trust in the security team. The lasting advantage comes from a harness tuned to the software development ecosystem around it:

• A test oracle that proves exploitability with a real integration test
• Partitioning that keeps cost and recall in balance
• Cross-model verification that catches one model’s mistakes with another
• Deterministic code that owns credentials, Git, and storage so agents do not have to

Migrating quickly to newer models is useful, but iterating on the harness itself matters more. Cost for a full application scan with publicly available frontier models ranges from $50 to $300 depending on model choice and application size. Incremental diff scans run roughly $5 to $50. That coverage spans all Shopify surfaces, running unattended at a fraction of the cost of the bugs it prevents. The models will keep changing; the harness is where the innovation lives.