Preparing for the Yearly Peak
For Shopify, Black Friday Cyber Monday (BFCM) is both the heaviest traffic period of the year and the moment merchants place the most trust in the platform. Preparation for that weekend begins as soon as the previous one ends. The work falls into two categories: continuous, year-round practices that build resiliency, and the deliberate slowdown of change in the weeks before the event.
Regular Load Testing as a Regression Tool
Load testing is most often described as a way to confirm the system can sustain expected traffic or to discover new limits. Shopify also uses frequent, repeated tests to detect regressions. Running the same experiments on a schedule surfaces trends at low traffic levels before they have the chance to escalate into outages during peak periods.
These same tools can apply identical loads to shops with different configurations, themes, or other distinguishing dimensions. Comparing the results highlights how those variations affect behavior under pressure.
Documenting the User-Facing Failure Modes
Complex systems are well defended against failure, which means catastrophe generally requires multiple, simultaneous failures rather than a single point of failure. To defend the system properly, teams must map dependencies, the ways those dependencies can fail, and the resulting impact on the end-user.
Creating this user-centric resiliency matrix is itself a basic tabletop exercise. Teams are forced to state how well they understand each dependency and what behavior should be expected when that dependency fails. Reading across the rows and columns of the matrix makes it clear which experiences depend on multiple fragile components, and which areas deserve attention first.
Testing the Model Against Reality
Once written, a resiliency matrix is an approximation of the system as it currently stands—but systems change. Without testing, the divergence between the model and reality goes unnoticed until it surfaces during a live incident. Game days exist to run the documented scenarios in a controlled setting and compare the outcomes with expectations.
The game day itself should examine both the external outcomes, such as whether a request works or fails, as well as the internal mechanics: whether the right alerts fire, the proper on-call teams are paged, and those teams have enough context to understand what is happening.
Though the reference is from writing on complex systems, the defenses against failure are technical, human, and organizational. A well-designed game day exercises all three:
- Do any automated systems engage on their own?
- Do the human operators called in have the knowledge, data, and tooling to act?
- Do current processes help or get in the way of responding?
Observed results feed back into the matrix, either by updating it to match the real system or by driving changes to the platform to bring behavior back in line with the intended model.
Learning Without Root Cause Attribution
Year-round incidents are unavoidable, and each one offers a learning opportunity beyond restoration of service. The useful insight is that reducing an incident to a single "root cause" ends the investigation too early. Focusing on an obvious surface issue misses the underlying technical, cultural, and organizational problems that lined up to allow the failure. Until those contributing conditions are corrected, the same incident can repeat even after the immediate problem is "fixed."
Slowing the Rate of Change for the Big Weekend
The most distinctive adjustment in the BFCM period is a deliberate reduction in the pace of change, because any change introduces new ways to fail. Rather than relying on heroic last-minute fixes, engineering effort shifts away from launching features and toward reliability work in the run-up to the event.
This shift means:
- Reviewing the defined resilience matrices.
- Running game days, load tests, and stress exercises more frequently.
- Updating runbooks and refining internal tools based on findings.
- Shipping fixes for defects in capacity, performance, and scaling that the increased testing reveals.
The continuous work of testing, documenting failure modes, running practice disruptions, and learning from real incidents comes together as a platform able to support the significant sales volume of the BFCM weekend.
