Beyond A/B Tests: Causal Methods for Product Decisions
Correlation and descriptive statistics are standard tools for data scientists, but they don’t explain why patterns emerge. At Shopify, understanding causality is central to product development: causal insights can validate or invalidate business strategies. While A/B tests are the best-known tool for this, they aren’t the only one available. Quasi-experiments and counterfactual estimation are powerful alternatives when randomization isn't feasible.
A Ladder of Evidence
One useful mental model for causal inference is the "levels of evidence ladder," which ranks methods by the strength of proof they offer. A/B tests sit at the top, followed by quasi-experiments, with counterfactual estimation at the bottom. Descriptive statistics provide no direct causal evidence.
This is not a strict hierarchy of quality, but rather an indication of the effort required at each stage. Methods higher on the ladder typically demand stronger experimental setup upfront. Lower-rung methods rely more on observational data and therefore require more rigorous robustness checks. There is no free lunch: a powerful causal analysis needs either a solid experiment design or a thorough statistical approach supported by domain expertise.
Rung One: Randomized Experiments
A/B tests are the gold standard for causal inference. By randomly assigning units to treatment and control groups, you ensure the groups are equivalent "on average," and any difference in outcomes can be attributed to the treatment. In practice, caveats apply — self-selection into an experiment is a common pitfall.
Setting up a proper A/B test requires significant infrastructure:
- A system for randomly assigning users to groups in real time.
- Tracking to collect data on all relevant metrics.
- Statistical analysis to compute effect sizes and validate the causal relationship.
Shopify’s internal experimentation platform handles much of this overhead, allowing data scientists to launch experiments with minimal friction.
Rung Two: Quasi-Experiments
Sometimes an A/B test simply isn't possible — due to missing tooling, time constraints, ethical concerns, or retroactive analysis of past launches. In these cases, quasi-experiments offer an alternative: the treatment and control groups are divided by a natural process that isn't truly random but is close enough to yield reasonable estimates. This often happens when features roll out to different user segments at different times or when eligibility depends on external factors.
Two methods are commonly used at Shopify for quasi-experimental analysis. The first is linear regression with fixed effects, which assumes you have data on all the factors that separate treatment and control groups. If that assumption holds, the regression coefficient on the treatment variable provides an unbiased estimate of the causal effect.
The second widely used approach is difference-in-differences, which requires finding a control group with a trend parallel to the treatment group for the metric of interest before treatment. After treatment, any break from the parallel trend is attributed to the causal effect of the treatment.
Rung Three: Counterfactual Estimation
There are cases where no control group exists at all — for example, when a feature ships to every user simultaneously. To estimate its impact, you can build a counterfactual: a model that predicts what would have happened had the feature not existed. The quality of this prediction is the key to the method's validity.
An example from Shopify: when a security update was rolled out to all merchants, data scientists wanted to know if the added friction hurt usage. No control group was available, so they trained a time-series model on data from unrelated features and global platform activity — variables independent of the security update. Comparing the model's forecast to actual usage showed no significant decline, a useful null result confirming the update didn't harm user engagement.
Counterfactual methods are vulnerable to unseen confounding factors. If a competitor launches a similar feature after yours, a model that doesn't account for that shift may misattribute the change. This is why domain knowledge and rigorous testing are essential for this approach.
Test Your Assumptions
Quasi-experiments and counterfactuals come with a cost: confidence intervals are harder to compute, uncertainty is higher, and false positives are more common. The antidote is robustness checking. This involves explicitly stating the assumptions behind your methods and relaxing them one by one to see if the results hold. If a single noisy variable flips your conclusions, your analysis is fragile.
Directed Acyclic Graphs (DAGs), popularized by computer scientist Judea Pearl, are an effective way to formalize assumptions in causal analysis. At Shopify, data scientists often use Dagitty, a browser-based tool that lets you draw a hypothesized causal chain. The tool then suggests robustness checks, such as conditional correlations that should disappear under your assumptions.
Three Takeaways on Causal Methods
- A/B tests are the first-choice tool for measuring causal effects and should be used whenever feasible.
- When randomization isn't possible, look for natural experiments — quasi-experimental settings can still produce strong evidence.
- Counterfactual methods are a last resort and are generally unsuitable for detecting weak signals.
Causal inference has untapped potential in industry product development. Just as the 1990s built a culture of experimentation, the hope is that the 2020s will bring a broader "causal culture" to how companies interpret data.
