Closing the loop on vulnerability remediation

Shopify’s dependency-upgrade and code-fix systems can draft patches, but drafting isn’t the same as fixing. The repository can drift while a PR waits for review; new developer context can change what the correct fix looks like; and a green CI run on a branch never proves the vulnerability is gone from the repository head. River, Shopify’s AI agent that works in Slack from the root of World, the company’s monorepo, now drives these workflows to a verifiable close.

River uses the same reproducible development environments, skills, and engineering conventions as Shopify’s developers. That shared foundation gives security workflows the needed repository-specific context from the start. Rather than teaching the agent how the codebase works, the security team layered remediation logic on top. Engineers no longer shuffle state between code, CI, Slack, and a tracker; they review risk decisions and product trade-offs, while River verifies the live state of a finding, updates a patch when it can safely do so, escalates the right questions to the right people, and follows up after handoff.

After a merge, River inspects the repository head and the resulting dependency graph before marking work done. The results so far are measurable: in the first 11 days of the dependency workflow, the open-issue backlog fell by about 70%, with roughly two-thirds direct merges and the rest confirmed as obsolete or already resolved elsewhere. Since launch, security merges through the company’s freshness-gated merge queue have gone from about 10% to 80%.

This pipeline-around-the-diff work is what’s least discussed in agentic patching, and it’s where the defensive asymmetry becomes concrete. Attackers can afford repeated failure; defenders must preserve production behavior with every fix.

Where the process starts

Dependency and first-party application workflows take different entry points. Dependency work usually begins with a PR upgrading a vulnerable library. Most link to a vulnerability item, but River must still verify that link and the item’s status. Application vulnerability work starts with an agent-discovered finding and a recommended fix that may need deeper investigation.

From there both follow the same remediation loop. River creates one Slack thread per application finding, but groups dependency PRs affecting the same area into a shared thread. On a recent run, four threads appeared within a minute of each other, spanning 35 findings across several codebases, and each proceeded independently without waiting on the others.

River reads its work ledger, identifies eligible items, and checks them against live repository, PR, and tracker state before acting. Later runs revisit the threads and update the ledger. Beyond serving as a transcript, each thread preserves the investigation and handoff context so future sessions — for River, security engineers, or code owners — can resume the work.

The remediation loop

On paper the pipeline looks like boxes and arrows; the real work is in the transitions between them.

Reconcile against live state

Each run starts by comparing the ledger to the actual repository, PR, and vulnerability-tracker state. The ledger remembers intent, but it isn’t operational truth — it may list a merged PR as open or show progress on a branch that closed without merging. River treats every ledger entry as a claim, not a fact. On a recent pass, reconciliation removed a large share of the apparent dependency backlog: most PRs had actually merged, a few were closed because newer upgrades were available.

Then River confirms the vulnerability is still present before touching anything. For dependencies it checks whether the affected codebase still resolves to the vulnerable version or whether a sibling PR already upgraded it. For application findings it locates the current repository, verifies at head that the code still behaves as reported, and searches each PR system for another fix already in flight. Where a search can’t be exhaustive, the handoff says so. If another change resolved the issue, River records the evidence and stops — a technically correct patch on a stale premise wastes engineering time and risks breaking production for nothing.

Repair against the current head

Only after confirming the vulnerability is live does River edit. For a stale dependency upgrade it replays the upgrade on repository head, regenerates the lockfile with agreed-upon tooling, and repairs mechanical failures the upgrade introduced. For an application finding it prepares a narrowly scoped draft PR.

In one batch of eight dependency PRs, River rebased seven of them, retriggered CI on each new head, and repaired a release-version check the upgrade had tripped. Two details mattered:

  • The lockfile had moved since the PRs were opened, so replaying each branch’s own copy would have reverted two unrelated upgrades. River grafted each upgrade into the current lockfile; three PRs shared that lockfile and all landed without disrupting adjacent work.
  • Every rebase creates a commit that invalidates the prior run’s results. CI notifications that arrived right after each force-push were for the old heads, so River discarded them and waited on new verdicts.

It left the eighth PR alone: a developer had added commits and a second engineer had an open review. Rebasing would have overwritten human work and overridden a pending design question, so the PR went back untouched. Six of the seven rebased PRs came out green and required only approval; the seventh stayed red on a problem needing a decision, not a patch.

Hand off the decision with the evidence

When River can’t resolve a question itself, the handoff carries the current head, what was tried, what the evidence shows, and the most specific question an owner must answer. Developers bring product context, accept risk, review, merge — and don’t ferry state between Git, CI, Slack, and the tracker.

The red PR required two answers: whether the new version was deployable at all, and how its type change would behave at runtime. River could gather context but couldn’t adjudicate the outcome, and when the desired result can’t be specified, another edit is a guess, not a fix. Sometimes the investigation reverses the initial finding. One finding recommended hardening a long-standing feature. River reproduced the behavior, prepared a change, updated tests — then a developer asked what requirement the feature was originally designed to serve. The behavior was deliberate: introduced years earlier for a specific problem, reviewed for security then, and further protected since by platform-level controls that didn’t exist at the time. The prescribed fix would have broken the flow it aimed to preserve — so River recommended dropping its own patch.

That outcome wasn’t a failure; the evidence changed and the recommendation followed. A good stop records the evidence, names the boundary, and routes a question that allows the work to resume. Unexplained pushes and premature handoffs are what destroy developer trust.

Verify closure on the default branch

A stewardship handoff changes ownership but not status: the vulnerability stays open until the fix ships. River monitors dependency PRs after review, because code changes invalidate earlier test results. After a merge it confirms the repository head no longer resolves to the vulnerable dependency and that both the vulnerability record and the ledger reflect reality.

One comparison clarifies this: two green dependency PRs both passed CI. The first merged and removed the vulnerable version from the repository head; the second sat awaiting review, leaving the vulnerable version in place. Both fixes worked, but only the first removed a vulnerability. For this workflow, done requires the source systems to record the outcome and the default branch to reflect it — whether the close comes from a fix, a superseding change, or a rejected finding, each needs the same evidence and attribution.

What prompts can’t guarantee

River behaviors are driven by prompts and skills specifying full enumeration, draft-only behavior, and mandatory human merge authority. Prompts are the right medium for judgment, but they’re not the right medium for guarantees. In the course of a run, an agent may recheck only the oldest in-flight claim, as River once did — proving that “the prompt says so” isn’t an invariant.

Guarantees belong in code. Pagination, deduplication, current-head identity, and accounting should become deterministic as the workflow matures so prompts can be reserved for judgment: whether code still expresses the reported behavior, and why a workflow should stop. This distinction matters most in security, where rules determining vulnerability closure have to be enforced deterministically.

What to take and apply first

The internal tools are Shopify-specific, but three principles transfer:

  • Revalidate before you edit: Security work is a claim about moving code. Check the repository head, live PR state, and the tracker before letting an agent generate a patch.
  • Keep evidence attached to the code: Bind tests and CI to the current SHA. After merge, inspect the default branch and reconcile the tracker. PR counts and green-branch tallies measure activity, not remediation.
  • Design the handoff deliberately: Allow the agent to finish routine work it can verify. When judgment is needed, it should preserve the evidence, identify the owning engineer, and pose one clear question. Track whether each finding reaches an evidence-supported conclusion — fixed, rejected, or escalated — not how many patches the agent produces.

Defining the Remediation Loop

River is Shopify's attempt to take vulnerability discovery and turn it into a closed remediation loop. The system connects specialized tools that find vulnerabilities to a pipeline that produces patches. Importantly, it manages the handoffs between each stage, preserves context over days of work across different systems, and knows when to stop.

The value here is not simply that more patches get produced. River enforces a protocol on every transition in the pipeline. Each stage must have an expected input, a set of conditions that must hold true for the work to continue, a defined set of actions the agent may take, and explicit stopping points. This structure is what makes the difference between automated patching and a remediation process that can be trusted at scale.

Turning a Fix into a Merge

The patch itself is only the beginning. A generated patch is not a merged fix; it needs to be validated, reviewed, and integrated. River's protocol means every handoff carries the context the next stage requires, instead of expecting an agent to re-derive what it needs from scratch.

Explicit stopping points are as critical as the actions themselves. If a stage hits a condition it cannot satisfy, the pipeline halts predictably rather than making assumptions or producing an incomplete result. This containment of failure is what makes the loop safe enough to run autonomously.