Why Shopify Beats on Its Own Platform
Shopify’s platform handles millions of merchants, and flash sales—product launches, promotions, or Black Friday Cyber Monday—can draw a sudden rush of buyers. To keep those events from turning into outages, engineering teams regularly simulate heavy traffic against representative test shops, hunting for bottlenecks before real customers hit them.
This work falls under two broad categories. Load testing confirms a service can handle a known volume—say, one million requests per minute for 15 minutes. Stress testing pushes a service until it breaks, revealing its upper limits. Both happen dozens of times per day across Shopify, conditioning the platform much like an athlete trains before a big race.
Each team at Shopify owns performance testing for the services it builds. The Platform Conditioning team supports that work by improving tooling, processes, and culture around high-traffic testing. The goal is to make simulated sales events a routine part of development, not a special occasion.
Building Load That Mirrors the Wild
Performance testing isn't just about request volume. Real traffic is uneven, and different requests tax systems differently. A request for a static CSS file might be served from a CDN in 40ms without touching Shopify's internal network, while a search query could hit three cache layers and query Redis, MySQL, and Elasticsearch in 1.5 seconds or more. Good tests hit a variety of realistic endpoints—especially the expensive ones—because those are where hidden performance problems live.
Traffic shape matters too. Flash sales are spiky by nature, so test load should come in similar bursts and run for realistic durations. Shopify’s internal load generator, written in Go with Lua-scripted "flows," scales to tens of millions of requests per minute. Each worker runs the Go-Lua VM, which lets a single flow coordinate requests across a distributed group of workers. Teams kick off tests through the internal ChatOps tool Spy, which starts the test and posts dashboard links and results back to Slack.
These Lua flows simulate browser behavior: sending headers, storing cookies, and making sequential requests. But they can't execute JavaScript or automatically fetch page assets, so they don't replicate the full request patterns of a real browsing session. That limitation motivated a more realistic approach.
Replaying Real Browsing Sessions
Early attempts to use real browsers through tools like Puppeteer were technically successful but too costly. Headless browsers can drive only thousands of concurrent sessions, while Shopify needs to scale to tens of millions. Browsers bring too much overhead.
A browser session on a typical page generates hundreds of requests, all visible in the Developer Tools Network tab. That led to a new approach: HTTP Archive (HAR) files. These files, exported from browsers or proxy tools like Charles Proxy, contain a detailed JSON record of every network request and response in a session.
Shopify's HAR-based tool, Hardy Har Har (dubbed HHH), turns a HAR file into a load-testing flow. It extracts all requests, lets the author strip out external hostnames (like Google Analytics) or CDN static assets, and produces a Lua flow committed to the load-testing repository. This lets engineers replay a real browsing session at massive scale.
Dynamic steps—customer logins, unique checkout creation—can't be replayed verbatim. HHH recognizes these and swaps in logic to handle them at runtime. Everything else stays in the Lua flow, editable by hand for fine-grained control of the test scenario.
Performance Testing as an Experiment
At Shopify, code ships constantly and anyone can deploy to production at any time. The same applies to load testing—any developer can kick off a massive test from Slack whenever they want. With that kind of freedom, the team relies on structure and repeatability to keep results meaningful.
Developers are expected to treat performance testing like any scientific experiment: form a clear hypothesis about a product or service, run experiments, observe results, and draw a conclusion tied back to the original hypothesis.
To make that formal process practical, the Platform Conditioning team built Cronograma, an internal Rails application that lets anyone set up a load or stress test experiment and track repeated runs. The app’s Experiment model captures essential context: a hypothesis, one or more orchestrations (coordinated load tests run simultaneously at different magnitudes and durations), the Shopify stores targeted during the test, and links to relevant dashboards, tracing, and logs.
Running and Observing Experiments
Once an experiment is defined, it can be run repeatedly. A runner starts a test from Slack with a simple command, which triggers a new experiment run and spins up a dedicated Slack channel for the session. During a run, multiple things can happen: exceptions, elevated traffic levels, or in some cases actual pages. The team wants to record all of it, especially since many colleagues work fully remote. Observations are captured from Slack and logged to a timeline for the run, and anyone can contribute comments. When the experiment finishes, the runner terminates the run and logs a conclusion based on the observations, tying it back to the hypothesis.
Cronograma also pulls in supporting data automatically:
- Monitors and alerts triggered during the experiment, from internal or third-party monitoring applications, are logged to the timeline.
- Metrics are retrieved from Shopify’s data warehouse, letting developers compare metrics across runs of the same experiment.
That includes response times, 5xx error counts, and requests per minute (RPM) generated by the test. Because this data is captured automatically for every run, teams can determine whether a service is improving or degrading over time.
Cronograma gives Shopify a home for all formal performance testing. Developers can browse past experiments, repeat them, and compare hypotheses, observations, and conclusions. Combined with the other tooling the team has built, it has driven numerous performance improvements and helps ensure the platform can handle the next major sales event.
Preparing for the Spike
Shopify’s merchants depend on the platform being fast and stable whether they’re making their first sale or processing tens of thousands of orders per hour. The team prepares for worst-case scenarios by proactively testing the core product, services, and apps using simulations, exposing and fixing problems before they affect merchants. That culture of load testing across teams means the platform is ready for flash sales and Black Friday Cyber Monday. The work may go unnoticed when a large sales event goes smoothly, but that is the goal: strength and conditioning make seamless performance possible.
