Risk Work as a Series of Bets

Shopify’s infrastructure platform is highly complex, interconnected, and globally distributed, with no single person able to oversee the full design and detail. During Black Friday Cyber Monday (BFCM) 2022, the platform served 75.98M requests per minute at peak—1.27M requests per second. At that scale, it is impossible to identify and mitigate every conceivable failure. Risk mitigation therefore becomes a question of prioritization: with finite time and an effectively infinite list of things that can go wrong, how do you decide where to focus?

Engineering Operations at Shopify breaks this down into four questions that can be applied to nearly any scenario.

First: What Are the Risks?

Before making any mitigation decisions, you need visibility into the current state of the system. Shopify runs “what could go wrong” (WCGW) exercises, where anyone building or interested in infrastructure can highlight a risk—technology risks, operational risks, or something else entirely. The unfiltered list that results provides a broad view of what could happen. The goal of this step is visibility.

Second: What Is Worth Mitigating?

Brainstorming leaves you with a large, daunting list. With limited time to fix everything, the key is prioritizing what matters most to the business. Teams vote on risks, then gather technical experts to discuss the highest-ranked ones in detail—including likelihood and severity. Decisions are made about what and how to mitigate, and which team owns each action item. The goal here is optimizing how time is spent.

Third: Who Makes the Decisions?

In any organization, there are times when waiting for perfect consensus is neither possible nor effective. Shopify identifies decision makers and empowers them to gather input, weigh risks and rewards, and come to a conclusion. Frequently, the best decision is made by the subject matter expert, or by whoever bears the most benefit or consequence of the chosen direction. The goal is aligning incentives and accountability.

Fourth: How Do You Communicate?

Even when moving fast, stakeholders and close collaborators need to stay informed. Findings and risks from the WCGW exercises are summarized so everyone lands on the same page about the risk profile—including key risks and single points of failure. Over-communication keeps teams aligned and aware, and gives stakeholders opportunities to interject. The goal here is alignment and awareness.

Dealing With Uncertainty

Underlying all four questions is the inherent uncertainty of the working environment. You never have all the facts, nor do you know exactly which components will fail, when, and how. The most effective response is to work in probabilities.

Expert poker players internalize that great bets don’t always yield great outcomes, and bad bets don’t always yield bad ones. What matters is betting on the probability of outcomes—over enough rounds, results converge to expectation. The same logic applies in engineering, where teams constantly make bets and learn from them. This requires clearly distinguishing the quality of decisions from the quality of outcomes. It means not over-indexing on bad decisions that led to lucky outcomes, or on great decisions that happened to hit very unlucky scenarios.

Acknowledging that you can’t control everything also helps preserve calm, which is vital for good judgment under pressure. Looking back, the question is whether you prioritized the most important things and made thoughtful bets with the information available. Did you facilitate meaningful discussions with the right people? Could you justify your actions to your customers and their customers? No process can predict the future—but a structured approach to risk ensures the bets you place are the ones worth making.