Fixed-Power Designs: It’s Not IF You Peek, It’s WHAT You Peek at

When Sample Size Calculation Has to Wait

Running an online experiment usually starts with a power analysis: estimate the variance of the metric you care about, decide on a minimum detectable effect, and compute how many users you need. That number gets fixed in advance, data is collected until it is reached, and then standard inference is applied.

The problem is that pre-experiment variance estimates are often wrong. At Spotify, historical data is a shaky foundation for power calculations. The user base is diverse and constantly shifting, especially in new markets and among new users. Historical data also tells you nothing about how different user segments will respond to a treatment—users with different listening habits are unlikely to react uniformly to the same feature change. When the variance estimate is off, so is the required sample size, and the experiment either drags on or ends underpowered.

An alternative is to let the experiment itself inform the sample size. Start collecting data, periodically re-estimate the variance from what you have, recompute the required sample size, and stop when the current sample exceeds the estimate. This is what researchers at Spotify call a fixed-power design, and it is the subject of a recent paper (Nordin and Schultzberg, 2024).

At first glance this looks like peeking—using outcome data to make a stopping decision—and peeking usually invalidates standard fixed-sample inference. The paper's central claim is that not all peeking is equal. Stopping based on the sample variance, which is what a fixed-power design effectively does, has far milder consequences than stopping based on significance.

Fixed-Sample vs. Sequential Designs

Experimental designs for A/B tests fall into two broad camps.

A fixed-sample design predetermines the sample size via a power analysis. The experiment runs until that number is hit, at which point the analysis is performed. This approach maximizes power for a given effect size but offers no flexibility to stop early or adjust.

A sequential design has no fixed sample size. Users enter over time and the experiment stops based on a rule evaluated on accumulated data—most commonly, stop when the test turns significant. Sequential tests provide valid inference under any stopping rule, which sounds appealing, but they come with real costs:

• Biased estimates. Stopping at the first significant result overestimates the effect size. The very design principle conflicts with the standard advice not to trust experiments that are too underpowered.
• Lost power. Experiments often must run a minimum duration regardless of significance—to observe users long enough to rule out novelty effects, or, at Spotify, to cover a full week and avoid weekday seasonality. A sequential test designed to stop at any time is conservative when early stopping is actually prohibited for most of the experiment's life.

Many companies in practice run a hybrid: the design is sequential in that it stops when the current sample size exceeds an estimate of the required sample size, where that estimate is continuously updated from the data collected so far. But the analysis treats it as if it were a fixed-sample design, using standard tests without correction. The open question is whether that analysis is valid.

Precision-Based Stopping

The fixed-power design stops based on the precision of the treatment effect estimator. Precision can be expressed two equivalent ways: as the width of the confidence interval, or as the required sample size for a given hypothetical effect. The paper shows these are the same stopping rule in disguise, since required sample size is just a transformation of the estimator's variance.

Precision-based stopping rules are already used in practice—some experimentation vendors sell them—but their statistical properties had not been rigorously examined. Intuition says they should require corrections, since they peek at outcome data. The paper demonstrates otherwise.

The graph above illustrates a fixed-power design in miniature. With a small example sample size, the experiment stops very close to the true required sample size. In more realistic large-sample settings with small effects, the estimated required sample size becomes increasingly precise, and the stop point lands in a narrow band around the true value.

The Statistical Guarantees

Nordin and Schultzberg (2024) investigate the difference-in-means estimator of the average treatment effect under precision-based stopping rules, specifically stopping on required sample size or, equivalently, on confidence interval width. Two results stand out:

• The difference-in-means estimator remains consistent for the average treatment effect.
• The standard fixed-sample confidence interval has asymptotically nominal coverage.

In plain terms: for large samples, you can apply ordinary fixed-sample inference after stopping on an updated required sample size, with no corrections. The paper also proposes conservative finite-sample versions of both the fixed-power design and the fixed-width confidence interval design, for cases where asymptotic guarantees are not enough.

The reason this works is that the stopping rule is a function of the sample variance, not of the treatment effect estimate itself. Which aspect of the outcome data you peek at determines whether peeking distorts inference. Stopping on significance, for instance, is far more problematic because it directly conditions on the magnitude of the effect. Stopping on how much noise is left in the estimate is a milder intervention.

Practical Benefits

These guarantees matter operationally. They mean experiments can start without a reliable pre-experiment variance estimate, revise their required sample size as outcome data arrives, and stop at the right moment—all while relying on standard fixed-sample analysis.

The fixed-power design captures the benefit of sequential flexibility without the two main drawbacks of sequential testing discussed above. It delivers unbiased effect estimates and does not sacrifice power when early stopping is restricted by practical constraints. The experimenter can maintain the safety of fixed-sample testing while continuously checking whether the original power calculation was correct.

What Fixed-Power Designs Change About Experiment Stopping

Choosing when to stop an online experiment is a classic trade-off. Fixed-sample designs lock in a sample size up front, but leave no room to react to what the data is actually saying. Sequential designs allow early or late stopping, but at the cost of power and biased estimates unless special inferential procedures are used. Spotify’s engineering team has been exploring a middle path—one that sidesteps the usual compromise.

In Nordin and Schultzberg (2024), the authors propose a “fixed-power design.” The idea is simple: don’t fix the sample size in advance. Instead, estimate the required sample size from ongoing outcome data as it accumulates. The experiment ends when the observed sample size exceeds that estimate. Crucially, the design supports standard nonsequential inference—point estimators remain consistent, and confidence intervals hold their nominal coverage. In effect, you get sequential stopping without surrendering the statistical properties of a fixed-sample analysis.

Why Pre-Run Calculations Fall Short

Spotify’s user base is large, diverse, and continually shifting. Pre-experiment power calculations built on historical data assume that treatment effects are relatively stable across segments and over time. That assumption often breaks down in practice. Different user groups can respond differently to the same change, and newer users may behave unlike the cohorts that informed the original estimate.

With a fixed-power design, those mismatches become visible early. Because the required sample size is recalculated from live data, the experiment gives a running signal when the effect size is smaller (or larger) than initially expected. That early readout matters in an environment where multiple experiments compete for the same traffic.

Operational Realities in a Multi-Experiment Environment

Running tens of thousands of experiments means that no single test can be given unlimited room. If the live data reveals that a statistically adequate sample would be far larger than planned, teams can’t always extend the experiment or increase its traffic share—other tests are in the way. In that situation, the fixed-power design’s value isn’t just the flexible stopping rule; it also offers an early diagnostic. It tells a team whether the observed data is aligned with the original power analysis, well before the experiment would have ended under a fixed-sample plan.

The authors note that this is a practical balance: more adaptable than fixed-sample designs, but without the inferential complications of always-valid sequential tests. Still, knowing the required sample size only mid-experiment can be operationally uncomfortable. Planning for capacity and timelines is harder when a key quantity isn’t pinned down at the start.

Spotify also points to its Confidence product as a tool that exposes the current required sample size and current-powered effect while an experiment is live. With those numbers available, a team can effectively run a fixed-power design by starting a test with a conventional fixed-sample setup and monitoring the evolving estimates.