A Year of Rehearsing for the Biggest Shopping Weekend

Shopify’s BFCM preparation doesn’t start in November. It starts in March with a structured program of capacity planning, risk assessment, and system-wide testing. The goal is to simulate last year’s record traffic—and then push beyond it—so that when the real event arrives, the platform's response is a matter of routine, not reaction.

The scale of the target is substantial. In 2024, the platform processed 57.3 PB of data, executed 10.5 trillion database queries, and served 1.19 trillion edge requests. It peaked at 284 million requests per minute on the edge and 80 million on app servers. For 2025, the preparation has been designed to handle traffic that surpasses these numbers, which the engineering team now considers "a regular day".

Three parallel workstreams drive this year-round readiness:

  1. Capacity planning: Modeling traffic from historical data and merchant growth to determine required infrastructure and geographic placement, with estimates submitted to cloud providers well in advance.
  2. Infrastructure roadmap: Reviewing the tech stack to sequence architectural changes and upgrades before the peak window, rather than treating BFCM as a release deadline.
  3. Risk assessments: Running "What Could Go Wrong" (WCGW) exercises to document failure scenarios and feed inputs into testing cycles.

Chaos Engineering and the Resiliency Matrix

A core pillar of the preparation is the Game Day program—chaos engineering exercises that inject faults into production systems to test responses under failure conditions. These exercises focus on "critical journeys" such as checkout, payment processing, and fulfillment. They simulate cross-system failures by testing search and pages endpoints, randomizing navigation, and injecting network faults to create realistic load patterns.

All findings from these tests are logged in a central Resiliency Matrix, which serves as the roadmap for system hardening. The matrix documents service status, specific failure modes with impact analysis, and recovery procedures, alongside detailed operational playbooks and on-call schedules for rapid incident response.

Load Testing Fundamentals

The Genghis load testing tool runs scripted workflows—browsing, cart adds, checkout—to gradually ramp traffic and identify breaking points. Tests run on production infrastructure from three Google Cloud regions (us-central, us-east, and europe-west4) and include flash sale bursts on top of baseline load. The open-source framework Toxiproxy simulates network partitions and failures during these runs.

When the system hits a limit, teams choose from three options: horizontal scaling (more instances), vertical scaling (more resources per instance), or architecture-layer optimizations such as database query improvements or performance tuning. These decisions set the final capacity targets and drive pre-BFCM optimization work.

New Systems, New Challenges

While most of Shopify’s infrastructure has historical BFCM data to model from, the analytics platform does not. In 2024, the team rebuilt it entirely, creating new ETL pipelines and replacing the legacy system with new APIs.

This migration created an asymmetry: the ETL pipelines ran through BFCM 2024 and have a season of production data, but the new API layer launched after that peak. The prep for these untested APIs involved dedicated Game Days with injected traffic loads, database latency, and cache failures. These exercises uncovered several bottlenecks that they subsequently resolved:

  • Kafka partitions were increased to maintain data freshness during traffic spikes.
  • Memory usage in the API layer was optimized based on profiling results.
  • Connection timeouts were tuned to prevent pool exhaustion.
  • A different load balancer approach offloaded requests to another region, providing extra headroom.

The Scale Testing Program

Load testing and Game Days validate components in isolation, but scale testing confirms the entire platform works in concert at forecasted volumes. From April to October, the team ran five major tests that ramped from 2024 baselines to 2025 projections, which involved simulated the platform's p90 traffic assumptions.

These tests are massive. By the fourth test, the system handled 146 million requests per minute and over 80,000 checkouts per minute. The final test of the series reached p99 levels at 200 million requests per minute. Because of the scale, the tests ran at night and were coordinated with external partners like YouTube. Alongside sheer load, the tests exercised regional failovers, evacuating traffic from core US and EU regions to validate disaster recovery.

Each test cycle revealed unique issues that aren't visible during normal operations:

  • Tests 1-2: Identified core operations throwing errors under heavy load and checkout queues backing up.
  • Test 3: Validated key migrations and confirmed regional routing after infrastructure changes.
  • Test 4: Hit a limit that triggered an unplanned failover, identifying priority issues in test traffic routing and delays during region rebalancing.
  • Test 5: A full dress rehearsal run during NA business hours to accurately simulate BFCM conditions.
  • Mid-program shift: Adding authenticated checkout to test scenarios. Modeling real buyers, even as a small traffic percentage, exposed rate-limit paths that anonymous browsing never touches.

The Operational Plan

When the weekend arrives, the engineering effort shifts from testing to execution. Teams rely on real-time dashboards across all regions with automated alerts, supported by 24/7 incident response coverage. The plan includes dedicated merchant communications for status updates, live system tuning based on actual traffic patterns, and a post-event debrief to correlate monitoring data with merchant outcomes.

The preparation program concluded in October, having tackled all known bottlenecks and armed the response teams with documented procedures. The final test of the year was the closest simulation yet. With the infrastructure hardened and the muscle memory built, BFCM is treated as a validation of the year’s work rather than an unpredictable event.

A Year of Readiness, Compressed Into Four Days

Preparing for Black Friday and Cyber Monday is not a seasonal task at Shopify. The 2025 readiness program spanned nine months and involved thousands of engineers. The work included five major scale tests, regional failovers, and chaos engineering exercises designed to expose weaknesses before merchants feel them.

The goal was straightforward: make sure the platform holds up during the four days of peak commerce starting November 28th. But the process was anything but simple. Teams documented system vulnerabilities, updated runbooks, and hardened infrastructure under realistic failure conditions.

Permanent Improvements, Not Temporary Scaffolding

A key principle behind this year's effort was durability. The tools built for BFCM readiness were not designed as throwaway load-testing utilities. Three initiatives stand out as lasting contributions to platform reliability:

  • Resiliency Matrix: A structured way to track and visualize system resilience across critical paths.
  • Critical Journey Game Days: Exercises that simulate high-impact failure scenarios on essential commerce journeys.
  • Real-time adaptive forecasting: A system that adjusts capacity predictions dynamically as traffic patterns evolve.

These assets remain in place after the peak season ends, providing ongoing protection for the platform.

Testing Under Pressure

Scale tests were a central component of the program. Running five major tests gave teams repeated opportunities to validate capacity limits and identify bottlenecks. Regional failovers confirmed that traffic could be redirected quickly if an entire data center region became unavailable.

Chaos engineering played a complementary role. By deliberately injecting failures into live systems, engineers could observe how components behaved under stress and verify that automated recovery mechanisms worked as expected. Documentation from these exercises fed directly into actionable runbooks, so that when a real incident occurs, responders have tested procedures rather than improvised ones.

What Comes Next

BFCM begins November 28th. The platform enters that window backed by months of preparation, measured capacity, and improved operational tooling. The engineering work done in 2025 is meant to make the system more resilient every day of the year, with the peak season serving as the ultimate validation of those efforts.