Quantifying Impact Before You Build
Every product initiative carries both an opportunity and an implicit cost: the cost of not investing elsewhere. Yet most organizations pick projects based on intuition, not arithmetic. Intuition has its place, but it is vulnerable to confirmation bias, reliance on readily available information, and the pull of pattern-matching new decisions to old experiences.
Opportunity sizing gives data scientists a structured way to estimate an initiative’s potential impact before committing resources. At Shopify, we use it to help product and business leaders direct effort toward the most consequential work, and to make the assumptions behind those choices explicit and debatable.
An opportunity sizing statement looks something like this: if we build feature X, we will acquire MM (+/- delta) new active users in T timeframe under DD assumptions. The output is not a single number, but a range tied to the assumptions underneath it.
Start With an Annualized Frame
Timing matters when sizing an opportunity. We recommend using an annualized view of impact so initiatives can be compared fairly against one another. In-year estimates can be skewed by launch timing: an initiative that ships early has more months to accrue impact than one that ships late, even if their steady-state effects are identical.
Beyond that, the right method depends on how much you know and how precise you need to be.
Directional T-Shirt Sizing
For quick, early-stage estimates of an existing initiative, t-shirt sizing is the most common approach. It does not require deep data science—subject matter experts supply rough estimates informed by prior experience and industry benchmarks. The underlying assumptions are intentionally generalized, relying on average or median conversion rates rather than initiative-specific modeling.
Suppose your Growth Marketing team wants to refresh an email sequence. Opening rates from your best-performing content and industry averages give you the raw material for a directional estimate. If your top content opens at five percent and the industry benchmark is ten percent, you might assume the opportunity is roughly double—moving from five to ten percent.
This method trades accuracy for speed, so it carries the risk of embedded bias and shallow assumption-checking. Use it for early ideation or as a sanity check, not for growth initiatives where a more rigorous approach is warranted.
Bottom-Up With Comparables
For existing initiatives where more precision is warranted, the bottom-up method anchors estimates to observed performance from a comparable product or system. It calls on a data scientist’s judgment, but produces results with higher confidence than t-shirt sizing because the benchmark is real, not assumed.
Three steps keep the method honest.
1. Understand the comparable. To estimate what an enhancement will do, you need to know how the current system is performing. Identify a similar product or system, and study its audience and processes closely. Key questions include: How many people received the offer? Was anything unusual about the audience selection? What were the participation and conversion rates?
If your team localized a welcome email for France last year and is now considering Italy, the French campaign is your comparable. Suppose Italy’s non-localized email drives a three percent click-through rate (CTR), while France’s localized version hit five percent over one year. That spread gives you a starting point for what localization might achieve.
2. Document assumptions and bound the estimate. Write down not just the numbers, but the why behind each one. From the comparable system’s metrics, derive a base metric and then estimate the positive and negative effects your initiative might have on it. Express the result as a range with an upper and lower bound, not a point estimate.
In the Italy example, the logic runs: localizing content should outperform non-localized content, as it did in France. So sending a localized email to 100,000 Italian leads per year should yield a CTR between three and five percent—the lower bound being the status quo, the upper bound informed by the French comparable.
3. Connect to a top-line metric. A CTR improvement means little in isolation. To compare initiatives fairly and avoid anchoring on large-looking but shallow-funnel numbers, translate the estimate into your business’s primary metric. A one percent lift in sessions can look inflated next to a three percent lift in customers, though the latter is further down the funnel and likely more valuable.
Returning to the email example: if the French localization produced a five percent CTR lift that translated into a three percent annual increase in active users, the Italian initiative—if it performs comparably—might add roughly 3,000 active users per year out of 100,000 leads.
This is also where second-order thinking pays off. Rising CTR could pull lower-intent users into the funnel, dimming downstream performance—or it could orient users to the offer better and improve conversion at every stage. Enumerate those ranges and find evidence for them. The exercise may reshape the proposal: it might not be enough to rewrite an email. The landing page, audience selection, or other adjacent systems may need to change to capture the opportunity you just sized.
Opportunity Sizing for New Initiatives: The Top-Down Approach
When the initiative in question is brand new, there is no existing system to measure and optimize — the top-down method is the appropriate sizing strategy. Unlike bottom-up sizing, which relies on a comparable baseline, this approach begins with a broader set of less precise information and progressively narrows it down into a credible estimate built on assumptions and observations.
Applying the top-down method effectively involves three key steps.
1. Collect Broad Information First
Because there is no comparable internal system to establish a base metric, the first move is to gather as much relevant information as possible from both internal and external sources. For example, sizing the opportunity of expanding a product into a new market would likely benefit from input from a product research team. Their work can inform estimates on total market size, the number of potential users, and the competitive landscape.
2. Document Assumptions and Pressure-Test Them
As with any sizing exercise, each estimate must be clearly stated with supporting evidence. There is a particular risk with new initiatives: assumptions tend to skew optimistic because teams are naturally biased toward believing their project will succeed. Rigorous testing is therefore essential. Two useful techniques include:
- Reviewing the range of impact observed from previous initiative launches to ground expectations in what has actually been achieved.
- Presenting the business case to senior stakeholders and defending it, a process that forces a more critical look at underlying assumptions.
Given the inherent uncertainty of a new venture, it is wise to err on the side of conservatism in initial estimates. Returning to the market-expansion example, documented assumptions should cover at least two areas:
- Market size. Compare the existing served market to the new target market, using external datasets where available. When direct data is scarce, assumptions can be modeled on analogous audiences or regions.
- Reach and conversion. The assumed rate at which new users can be engaged and converted. Conversion rates may borrow from historical performance when a comparable new channel or audience was introduced, applying the same tactics used in bottom-up sizing.
3. Connect the Estimate to Wider Business Goals
The final step is translating the opportunity size into its projected effect on the business’s strategic objectives. In the market-expansion scenario, that means asking what the assumed opportunity implies for the total number of active users.
Opportunity sizing pays for itself by providing a structured way to say yes to the most consequential initiatives. For data science teams, it is also a key mechanism for guiding leadership prioritization and decision-making. Once launched, the initiative should be measured against the original estimates; comparing the forecast to actuals is how sizing accuracy improves over time.
Whether the task is shaping the product roadmap or deciding whether to take on a particular project, these fundamentals help identify where the real opportunity lies — and where it does not.
