Preparing the Platform for Peak Traffic
For Shopify, Black Friday Cyber Monday (BFCM) is the most demanding period of the year, pushing our ecommerce platform to its limits. To ensure stability for our merchants, our engineering teams run a structured program of performance testing. This includes capacity planning, resiliency checks, and single-application performance tests, but the central exercise is full-scale load testing—what we call BFCM Scale Testing. The core process is straightforward: scale the platform to its BFCM configuration, run simulated peak traffic against it, analyze the results, make iterative improvements, and repeat throughout the year.
Shopify’s architecture is a hybrid model. At its center is a monolithic Ruby application, Shopify Core, supported by a range of services such as Storefront Renderer—which serves HTML to browsers and handles storefront API traffic—along with analytics pipelines, fraud analysis, payment processing, and webhook services. A single buyer journey from browsing to checkout interacts with many of these systems. During testing, we must carefully determine which components require scaling and to what degree to accurately simulate the load.
Scaling Up: Resource Planning and Coordination
In 2022, our goal was to run more comprehensive full-scale exercises than in 2021, making them routine events. Planning began in February, using the previous year’s data on instance counts for Shopify Core components and related applications, alongside learnings from prior scale test findings. Securing the required infrastructure from our cloud vendor is a critical early step. Cloud providers have finite resources, and requesting a 200 percent expansion of our already-large footprint demands advance coordination.
Our scaling strategy is shaped by years of observing the platform's behavior under load. Components are scaled up in a specific order to protect dependencies. For load testing, we scale up the nginx request-routing tiers downstream of the core platform, as well as upstream components like the ProxySQL tier and global Key/Value store. Coordination spans multiple teams: some scale their systems a day before the test because they require more time, while others scale on the test day itself to minimize costs.
On the day of a scale test, we bring all remaining components to full scale profiles, a configuration that remains in place for several hours. Often, issues surface simply from operating at this elevated profile, even without additional test load. Identifying these problems early is a core value of the exercise. We frequently encounter connection-count issues, such as an increased number of Storefront Renderer instances overwhelming an upstream application, or additional Shopify Core replicas straining streaming infrastructure. Sometimes we can resolve this by scaling up dependent services; in other cases, we refactor parts of the architecture, introducing intermediary software to multiplex connections.
Running the Load Tests
Once the platform is stable at full scale, we begin the load generation. In 2022, we employed a suite of five core tests, each simulating a distinct traffic class:
- Browsing and buying flow: Simulates users browsing storefronts, adding products to cart, and completing checkout via a mock payment gateway.
- Admin flow: Simulates merchant activities in the admin portal, such as inventory checks and price updates.
- Flash sale flow: Generates a burst of storefront activity as simulated users purchase a single product.
- Storefront API flow: Simulates API calls from headless Shopify stores using Hydrogen, reflecting the growing adoption of that technology.
- Hydrogen and Oxygen flow: Tests real headless storefronts powered by Shopify’s hosted Hydrogen and Oxygen offering.
Each flow approximates a single user’s journey as a sequence of HTTP requests—sometimes 25 to 50 per session—executed repeatedly by our custom load testing tool, Cronograma. We configure each flow’s request rate to precisely control the target RPM for each traffic class. Load is increased gradually, with tests layered sequentially, and engineering teams continuously monitor platform health. If any signs of degradation appear, an integrated "abort switch" in Cronograma allows us to stop all tests immediately.
Target Setting and Single-Region Resilience
For the early scale tests of 2022, we set targets based on projections and actuals from BFCM 2021. Performance is typically measured in Requests Per Minute (RPM) and Checkouts Completed Per Minute (CCPM). Around August, we receive official traffic projections from internal data science teams, enabling us to fine-tune our RPM and CCPM targets. We aim to test at a load comfortably exceeding projected levels, ensuring the platform remains unstressed during the actual peak.
During select exercises, we also simulate a catastrophic regional failure by routing all traffic through one U.S. cloud region while shutting off the other. This verifies that a single region has sufficient capacity to maintain service under full expected load. Our objective with the scale tests is not to push the platform to failure, but to gain confidence in handling target load without degradation. The exercise surfaces components requiring further optimization, and we iterate to eliminate these issues as they are discovered.
Safe Scale-Down and Post-Test Analysis
After the test runs conclude, we scale the platform down in the reverse order of the scale-up. This protects upstream dependencies such as ProxySQL—scaling it down before Shopify Core could overload it with lingering connections. Once scale-down is complete, each involved team collects and analyzes data from their systems. After allowing a day or two for data gathering, we hold a blameless retrospective to review issues requiring remediation, and also to flag components that narrowly avoided failure during the tests.
Why Full-Scale Testing Over a Scaled Model
Load testing a distributed system typically follows one of two strategies: a scaled-model approach or full-scale testing. The scaled-model strategy tests against a reduced subset of the production fleet, using proportionally scaled traffic, with results extrapolated to estimate real-world capacity. While this approach is cost-effective and easier to manage, it has significant drawbacks:
- Databases and other vertically-scaled components are difficult to represent accurately in a scaled model.
- It assumes workloads scale horizontally in a perfectly predictable manner.
We choose a full-scale approach, scaling the entire system to its BFCM profile and generating the full projected load. This approach validates the entire architecture—including vertical components—under the exact conditions expected during the peak weekend. We track all remediation actions from each test, assign owners, and begin planning for the next exercise. This continuous iteration ensures that when the busiest weekend of the year finally arrives, the testing has already made it as non-event as possible.
Why Full-Scale Testing Is Worth the Cost
Scaling a workload up to production-level numbers carries a real price tag in a cloud billing model, and it's a cost Shopify accepts deliberately. BFCM is the single most important traffic event of the year for Shopify's merchants, so confidence in the platform matters more than the expense of simulating it.
One of the less obvious lessons of operating at massive scale is that systems break simply from running at a higher baseline, even without additional load being applied. A frequent culprit is TCP connection count: more connections mean more memory pressure on proxies and databases, and can eventually trigger errors like maximum connection count exceeded. Getting comfortable running the entire infrastructure at this elevated level is the only way to be sure every component can survive the real event.
Full-scale testing also surfaces unexpected interactions in a complex, constantly changing architecture. No single engineer can hold a working map of every dependency in their head at once. Pushing high volumes of realistic traffic through the system exposes dependencies whose problems only appear as a side effect of overall platform load.
Simulating heavy traffic through a handful of core business flows effectively load tests every system on that path for free. This is especially valuable for middleware components like proxies, where designing a synthetic test from scratch is difficult — what load level is realistic, and what kind of traffic mix? If these components can satisfy the full-scale test workload, they're almost certainly sized appropriately for production spikes.
What Synthetic Testing Can't Capture
Some parts of the stack are outside Shopify's control, notably the content delivery network (CDN). Real user traffic arrives from all over the world and enters through the closest CDN point-of-presence (PoP). Load tests, by contrast, generate requests from a small number of clusters, which concentrates millions of requests per minute through just a handful of PoPs — enough to overwhelm them. To compensate, the test scripts route a portion of traffic directly to origins, bypassing the CDN, while the remainder still goes through it.
Cache layers can also skew results artificially. The applications under test use multiple layers of in-memory and key/value store caching. A specified portion of test requests are crafted to "bust through" the cache: the applications watch for these specially-marked HTTP requests and skip their caching layers when they appear.
It's important to remember that surviving a synthetic test at a given magnitude is not the same as being ready for production traffic of the same size. Production traffic patterns are nearly impossible to recreate exactly in a load test; the only true substitute for production traffic is production traffic itself. Shopify treats these exercises as an iterative process, not a guarantee of success. Still, any negative signal during a test — a small latency increase, a slight rise in error rate — gets close attention, as it may be the first warning of a much larger problem.
Two Scaling Problems, Two Fixes
The 2022 BFCM Scale Testing cycle surfaced two infrastructure issues that could have caused serious trouble during the real event.
The first involved a limit in the Kubernetes ingress controller. After Shopify refactored its architecture to concentrate Shopify Core into fewer clusters, it hit a ceiling of 1000 endpoints per cluster per deployment, stemming from an unfixed issue in ingress-nginx. That meant only the first 1000 of roughly 1200 desired Shopfiy Core pods per cluster would actually receive traffic. Rather than patching ingress-nginx or running additional deployments per cluster, Shopify changed the resource profile of the pods: each pod got more CPU, memory, and web worker capacity, allowing it to serve more requests. The resulting pod count per cluster stayed below the 1000 limit while still providing enough capacity for both the scale tests and BFCM itself.
The second issue appeared during single-region tests, which simulate a regional outage by routing all traffic to one location. As load ramped up, latency rose and errors increased, with some ingress-nginx pods in Shopify Core clusters handling far more traffic than others, leading to overload, elevated response times, and occasional HTTP 504 timeouts. Debugging with the cloud vendor traced the problem to the load balancer's distribution among backends. The nginx pods were spread unevenly across Kubernetes node pools — some nodes had a single nginx pod while others had several — so the load balancer's backend list was inherently imbalanced. The limited number of source IPs used for test traffic made things worse, since the low entropy in source addresses provided few distinct inputs for the load balancer's backend selection. The fix was to create dedicated Kubernetes node pools for nginx, ensuring every node always ran the same number of pods and eliminating the hotspots.
Making the Tests More Realistic Over Time
Future BFCM Scale Testing work will focus on increasing the number of load generation locations to better match Shopify's global footprint, and continuing to refine test flows so they more closely approximate real production traffic patterns. The gap between synthetic and real traffic will never fully close, but it can keep narrowing — and the confidence gained from each full-scale iteration is what justifies the cost.
