Risk-Aware Product Decisions in A/B Tests with Multiple Metrics

Every A/B test is an exercise in risk management

Product development is a gamble. Iterate too slowly and competitors pull ahead; ship changes users dislike and you lose them. Experimentation platforms exist to turn those gambles into calculated bets, but the statistical machinery only helps if it is aimed at the decision you will actually make. At Spotify, that means defining the rules in advance that map test results onto ship/don’t-ship decisions, and then designing and analyzing experiments so the error rates of the whole decision process—not just of individual metrics—stay under control.

This approach to risk-aware decision-making is detailed in a recent paper by Schultzberg, Ankargren, and Frånberg (2024). The key insight is that when an experiment involves multiple metrics of different types, the way you handle multiple testing—and the very definition of what constitutes a false positive—depends on how the metrics are combined into the final product decision.

Why a formal decision rule matters

Making sense of an experiment with a dozen metrics is rarely straightforward. A change might improve one success metric, leave others flat, and cause a warning metric to dip. Without a pre-specified way to weigh those outcomes, the final call tends to become ad hoc, varying from experiment to experiment, and the statistical guarantees from carefully designed tests quietly evaporate.

A formal decision rule solves that on three fronts. First, it forces you to articulate what a successful experiment actually looks like. Second, when built into a centralized platform, it standardizes decisions across teams, so a successful experiment means the same thing everywhere in the company. Third, because it maps every possible set of outcomes to a decision, it lets anyone read the result of an experiment—say, “ship thanks to rule 2”—without needing to interpret confidence intervals themselves.

For that reason, the rule is not just a tool for statisticians. It democratizes experiment results, which is central to running experimentation as a team sport.

Defining the metrics and the shipping rule

Spotify’s default decision rule combines four categories of metrics:

1. Success metrics (superiority tests): what you hope to improve.

2. Guardrail metrics (non-inferiority tests): things you don’t expect to improve, but need to ensure don’t regress by more than a set margin.

3. Deterioration metrics (inferiority tests): metrics that must not get worse; this category can also include success and guardrail metrics.

4. Quality tests: validity checks on the experiment itself, like tests for sample ratio mismatch and pre-exposure bias.

The resulting rule for shipping is strict. The treatment must be significantly superior on at least one success metric, must be significantly non-inferior on all guardrail metrics, must not show deterioration on any success, guardrail, or deterioration metric, and must clear all quality tests. The reasoning is straightforward: a product change should offer a real benefit and do so without bringing bugs, breakage, or unintended side effects along for the ride.

Of course, the decision rule is not a universal fit. Experiments whose outcomes hinge on external, unmodelable factors will not fit this template. But for the majority of experiments, this structure is a pragmatic way to convert deliberately chosen business risk into statistical terms and keep it bounded.

Guardrails flip the multiple-testing story

Multiple testing correction is basic hygiene when you run many hypotheses, because it prevents you from declaring a win too easily. But it actually fights against the purpose of guardrail metrics.

When you are running superiority tests against every success metric, the treatment has multiple ways to show a win—hence the classic rationale for corrections. In contrast, guardrail metrics store multiple ways to fail: the treatment must prove non-inferiority on every one. Fewer comparisons, not more, ought to be treated as chances to succeed.

That asymmetry has concrete implications. Under a shipping rule that requires exactly one significant success metric and non-inferiority across all guardrails, you control the false-positive rate by correcting only for the number of success metrics, not the total metric count. For success metrics, set:

alpha_corrected = alpha / number_of_success_metrics

For guardrail metrics, use alpha_corrected = alpha—the false-positive rate should not be adjusted at all.

Deterioration and quality tests introduce one more key effect: they place additional bounds on the false-negative rate rather than the false-positive rate. Qualitatively, you no longer have multiple ways to pass the quality bar—you have multiple ways to fail it. The design of the experiment, including its powering, has to account for the fact that each guardrail or quality check that returns “inconclusive” pushes you toward not shipping. Missing a genuine improvement in a success metric is no longer the only way to make a wrong decision.

Powering Guardrail Metrics: A Practical Correction

Beta corrections — adjustments to the false-negative rate, or power level — are rarely discussed in online experimentation literature, even though underpowered experiments and their pitfalls are well understood. When a decision rule requires the treatment to be non-inferior on all guardrail metrics and superior on at least one success metric, these conditions must hold simultaneously with probability at least 1 - β.

Consider a simple case where guardrail metrics are independent, and each is powered at 1 - β. As the number of guardrail metrics G grows, the probability that all guardrails are simultaneously significant drops rapidly. With only five guardrail metrics, simultaneous power falls below 40%; with ten, it’s around 11%.

Schultzberg, Ankargren, and Frånberg (2024) show this power loss can be mitigated by adjusting the power level per metric. Powering each guardrail metric for

β* = β / (G + 1),

where G is the number of guardrail metrics, guarantees the overall decision retains at least 1 - β power.

Deterioration and Quality Metrics Add Risk

At Spotify, certain critical business metrics are designated as deterioration metrics and included in every experiment. These metrics are tested for inferiority — that is, whether the treatment shows significant regression relative to control. Inferiority tests on all metrics help detect deteriorations that would undermine an experiment’s success, complementing the superiority and non-inferiority tests in the decision rule.

Quality metrics, such as tests for sample ratio mismatch and pre-exposure bias, are also standard in advanced experimentation platforms to validate data quality. Including deterioration and quality tests makes the decision rule more conservative, adding more opportunities to stop an experiment. The design must be adjusted accordingly to preserve intended power.

A Complete Strategy for Bounding Decision Error Rates

To make the decision-rule error rates explicit, consider the following rule:

Ship if and only if:

• The treatment is significantly superior to control on at least one success metric.
• The treatment is significantly non-inferior to control on all guardrail metrics.
• No success, guardrail, or deterioration metrics show evidence of deterioration.
• No quality test significantly invalidates the experiment’s quality.

Let S be the number of success metrics and G the number of guardrail metrics. All success and guardrail metrics are also tested for deterioration. Let D be the additional number of metrics tested only for deterioration, and Q the number of quality tests. Let α and β be the intended false-positive and false-negative rates for the overall product decision, and α₋ the intended false-positive rate for deterioration and quality tests. To ensure the decision’s actual error rates don’t exceed these intentions:

• Use α₋* = α₋ / (S + G + Q + T) for all deterioration and quality tests.
• Use α* = α / S for superiority tests on success metrics.
• Use α (uncorrected) for non-inferiority tests on guardrail metrics.
• Use the following β* for all non-inferiority and superiority tests:

β* = (β - α₋) / ((1 - α₋)(G + 1)) if S > 0, and β* = (β - α₋) / ((1 - α₋)G) if S = 0.

These corrections bound both false-positive and false-negative rates for the decision under any covariance structure; a formal proof appears in Schultzberg, Ankargren, and Frånberg (2024).

For typical values of α₋ — around 1% — the contribution of deterioration and quality tests to the beta correction is negligible. In practice, omitting their impact from the correction still yields empirical error rates at or below intended levels under commonly observed correlation structures.

Key Takeaways

1. An explicit decision rule that maps all statistical test results to a product decision is essential for controlling the risk of incorrect decisions in A/B tests.
2. Different decision rules require different experimental designs and statistical analyses to bound error rates effectively.
3. Multiple testing corrections for false-positive rates are standard, but guardrail metrics should not be corrected in this way. Conversely, beta corrections are critical for properly powering decision rules that include guardrails.
4. A mismatch between the decision rule and the design/analysis means the actual error rates differ from what you intend.

Unless your design and analysis are matched to how you make shipping decisions, you aren’t actually controlling the risk of wrong decisions. Spotify’s Confidence experimentation platform applies decision-rule-based analysis by default, with multiple testing adjustments aligned to the shipping decision. Each treatment receives a shipping recommendation that explains how each component of the decision rule contributes to the outcome.