Shopify's bug bounty program has logged almost a 300% increase in reports since 2024, a surge the company attributes to frontier models pushing the cost of vulnerability discovery toward zero. Reviewing, validating, and acting on that volume is what makes the program hard to run. Shopify's response is the Cyber Chaos Monkey (CCM), an agentic system that attacks live production infrastructure to find weaknesses before outside attackers do. According to the company, agents have run tens of thousands of offensive engagements and acted on dozens of impactful findings without causing an outage or incident.

The premise is that frontier models raise capabilities for adversaries and defenders alike, but Shopify holds an informational advantage over its own software. Combined with those models, that asymmetry is meant to do two things: surface many of the most impactful issues first, and make it prohibitively expensive for adversaries to find what is missed. Because attackers target production, testing must happen there too—so CCM is built to keep availability and merchant data out of harm's way.

Discovery and Sandbox

Conceptually, CCM is a graph-engineered agentic bug-bounty researcher with access to source code, threat intelligence, past reports, and infrastructure and deployment records, plus the ability to call production using live test shops and organizations. The name is an homage to Netflix's Chaos Monkey, the 2011 tool that randomly shut down production instances; CCM applies chaos engineering to security. The architecture splits into two components: Discovery and Sandbox.

CCM workflow: upstream ideas enter Discovery, which queries system knowledge, sends leads to Sandbox for testing, and uses results to generate further engagements.

An engagement starts from a hypothesis—specific ("find similar risks to a bug bounty report in other places") or abstract ("achieve RCE on Shopify admin")—contributed by a human security engineer, other upstream security agents, or Discovery's own feedback loops. From there:

  1. The idea is sent as input to Discovery.
  2. The latest cyber models refine it and identify code or configuration that may lead to a vulnerability.
  3. A promising lead is passed as structured data to the Sandbox, where agents attempt exploitation.
  4. Sandbox agents do hands-on work such as installing packages and building fuzzers. One of their most powerful primitives, Shopify notes, is making live requests, reading production logs to see how the request was processed, reconciling that against source, and iterating.
  5. Findings go into a report sent back to Discovery and upwards.
  6. Discovery analyzes the results and runs feedback loops that may generate further engagements.

Containment: technical controls

Shopify cites recent incidents in which security-testing agents reached systems they were never meant to touch as the reason containment has to be designed in from the start. CCM addresses both technical risk—leaked credentials, network escape—and alignment risk, such as an agent ignoring its Rules of Engagement.

Each engagement maps to a Kubernetes Job. The pod runs several containers; the application controller is a state-machine-based orchestrator, and everything else is deferred to sidecars, including maintenance tasks and the offensive agent itself.

CCM sandbox network: a pod in a Kubernetes node routes requests through credential and egress proxies to approved internet services.
  • Containerization and least privilege. The offensive agent has no escalated permissions. Before it starts, the orchestrator writes ephemeral Shopify credentials (for example a session cookie and CSRF token) to a shared volume; the agent can read them but not modify them. Containers are Debian-based and pre-loaded with development packages and security testing tools.
  • Credential proxy. Requests to knowledge services or model providers carry a placeholder header; the credential proxy swaps in the real credential and sends the request through an egress proxy. Header injection is strictly constrained per domain, and credentials rotate regularly.
  • Egress controls. The egress proxy is effectively an allowlist on top of nginx, and its pods are the only ones able to exit the VPC. Permitted destinations are production domains, knowledge services, model providers, and a select set of package managers such as Shopify's own package proxies.

Containment: keeping agents aligned

Every agent receives a strict set of Rules of Engagement (RoE): it may try any exploit against its target, but the rules impose hard limits on what it may do after an exploit succeeds—pivoting to other systems or retaining access, for instance.

Because execution-time alignment can't be assumed, CCM layers on two-person integrity (TPI), the practice—used in government and high-risk red teaming—of having a second party confirm the acting party's work poses no security risk. Shopify's twist is to make that second party an agent prompted and aligned differently from the executor. Session logs from the offensive agent stream in real time to a separately aligned agent running a different model. If it sees behavior incompatible with the RoE, it makes a tool call that instantly kills the executor. The judge reads every event from the executor's session: responses, thinking, tool calls, and tool call results.

CCM positive control loop: the orchestrator starts executor and judge agents, monitors them, and terminates the run when the judge flags a breach, is lost, or the executor finishes.

Control is explicitly positive: if the judge times out, crashes, or fails to adjudicate a message, the run ends. The orchestrator is the pod's main application container, with executor and judge as native sidecars in the same pod, and it holds a heartbeat with both. If either dies, the orchestrator exits; the pod's restart policy prevents it from restarting, so Kubernetes tears down both sidecars and the job terminates. Keeping the components pod-local but isolated reduces the chance of network partitions disrupting their communication, and an eviction removes every container in the pod—so execution containers can never run without their paired judge.

An asymmetry bet

CCM is a wager that informational advantage outweighs the falling cost of vulnerability discovery. Using the same class of models as attackers, but starting with the full picture of its own systems, Shopify aims to reach the most impactful issues first and to force external parties to compete with agents that work around the clock.