Building Blocks for Experiment Coordination

When many teams run experiments on the same surfaces, they need to know what their colleagues are doing. Spotify's platform lets experiments be grouped into "domains" that roughly map to different surfaces or systems. Each domain has a timeline of running and upcoming experiments, making it easier for teams to stay aware of activity in their area.

Because experiments running in close proximity risk interacting with one another, many must run exclusively: a user may only be in one experiment from a set of potentially conflicting ones. Today, only experiments within a single domain can be exclusive to each other. The team is planning to decouple exclusivity from domains so experiments can be mutually exclusive across domains as well.

Holdbacks are implemented at the domain level. Each domain can define a set of users who are exempt from the general experimentation in that domain. Spotify follows a quarterly pattern: a new holdback is created at the start of each quarter, and experiments running during that quarter are never assigned to those users. At the end of the quarter, one test measures the compound effect of all successful experiments by giving the entire combined experience to the treatment group in the holdback. Once that evaluation is complete, the holdback is released and those users rejoin the experiment pool.

Randomized Assignment Without Global Halts

Autonomous teams at Spotify start and stop experiments on their own schedules, which makes it hard to maintain randomization when experiments must be exclusive or respect holdbacks. To avoid stopping all experiments whenever assignments need reshuffling, Spotify built a "salt machine" that reassigns users by hashing the user id with a salt into buckets, limiting the user to a fixed set of treatments, without touching other experiments.

Consider an 8-bucket split where a user lands in bucket 1 if HASH(user id, SALT) % 8 = 1. Buckets are allocated to experiments; when one experiment ends and frees buckets, a new experiment cannot simply take the freed buckets. If a new experiment needed 25% of the population and only picked two freed buckets that happened to overlap entirely with an experiment that just finished, carryover effects could bias results.

Instead, the salt machine reshuffles users into new buckets using a new salt. After an experiment E1 (using buckets 0 and 1) ends, the freed users are rehashed into eight new buckets that each contain only 50% of the traffic. To get 25% of the total population for a new experiment, four of those diluted buckets must be allocated. The required over-allocation is the "compensation factor" — in this example, 1/0.50 = 2. When a second experiment ends, that space cannot be reused until the current one finishes, effectively wasting 50% of the users. Spotify has learned to avoid starting new experiments when the compensation factor exceeds 5. Work on a second-generation allocation scheme aims to waste less space while keeping the randomization benefits.

If two experiments are disjoint due to targeting, they don't need to share the same salt tree. Also, for simplicity, per-experiment salts that spread users across treatment arms are handled separately.

Formalizing Hypotheses, Metrics, and Tests

The Experiment Planner requires defining what to measure and test before an experiment starts. Metrics have one of two roles:

  • Success metrics — used to find evidence for the hypothesis.
  • Guardrail metrics — used to find evidence that the experiment introduces no harmful side effects.

For success metrics, experimenters run superiority tests and must specify a relative minimum detectable effect (MDE). The MDE feeds into power calculations for result analysis and the sample size calculator. Guardrail metrics use a non-inferiority test with a specified margin so the platform knows when a change is acceptable. Each success metric can use either a one- or two-sided test.

Experimenters choose between sequential testing, which provides results as data comes in, and a fixed-horizon test, which only yields results after the experiment ends. To avoid weekday biases, tests should run for the full planned period and only be stopped early if a harmful effect is detected. An optional gradual ramp-up of the treatment assignment further reduces weekday effects.

Calculating required sample size across many metrics, targeting rules, and statistical tests is complex. A built-in sample size calculator (required for sequential testing, optional for fixed-horizon tests) shows how many users are needed to power the metric for the specific target population. It automatically queries historical data for that population to proxy control group averages and variances.

Continuous Validity Monitoring

Subtle issues can compromise an experiment's result, so the platform continuously monitors all running tests and alerts the owning team when it detects a problem. The following checks are in place:

  • Sample ratio mismatch: the actual exposure proportions between groups must align with the plan; a statistically significant difference triggers an alarm.
  • Pre-exposure activity: looks for differences in user activity before the experiment begins.
  • Increased crashes: ensures no rise in client crashes.
  • Property collisions: flags when two non-exclusive experiments use the same Remote Configuration properties, marking a risk that exposure is not as expected.

Statistical checks use sequential testing with corrections for multiple comparisons.

Gradual Rollouts With or Without Stats

The platform also supports gradual rollouts for shipping a proven change safely. Rollouts can run with or without statistical testing. When statistical testing is enabled, a set of guardrail metrics is monitored via sequential testing. A daily recommendation is issued to the owning team, with three possible outcomes:

  • No harmful effects detected — recommended to continue.
  • Statistical evidence of harm — recommended to abort.
  • Not yet conclusive — recommended to continue with caution or wait for more data.

Metrics for rollouts is a new capability, and the team is iterating on it, with plans to make it the default rollout mode going forward.