Coordinating Experiments at Scale Without Sacrificing Statistical Validity

Spotify runs hundreds of concurrent experiments across a user base of hundreds of millions. A central engineering challenge is coordination: ensuring the right users receive the right treatments while preventing the same user from being enrolled in mutually exclusive experiments simultaneously. Many tech companies have addressed this problem—Google’s approach was documented by Tang et al. in 2010—but the statistical consequences of different coordination methods have seen less formal investigation. In a recent paper by Schultzberg, Kjellin, and Rydberg, the authors analyze the statistical properties of a common technical solution called “Bucket Reuse” and provide the theoretical grounding for Spotify’s new coordination strategy.

How Bucket Reuse Works

Bucket Reuse relies on hashing to partition users into fixed groups. The process works as follows:

  • Choose a number of buckets, B.
  • Hash each unique user ID with a fixed random salt, assigning every user to exactly one bucket.
  • Perform sampling at the bucket level: if N users are needed, select the number of buckets whose total user count most closely matches N. For example, if N=20 and each bucket holds 3 users, seven buckets (21 users) would be sampled.
Feature Image

The “reuse” aspect comes from keeping this hash map and bucket count constant across all experiments. A bucket is merely a logical grouping defined by the fixed hash. Once established, the same buckets serve every experiment indefinitely. Critically, Bucket Reuse for experimentation always entails two layers of randomization: sampling operates on buckets, while treatment allocation operates on individual users within the selected buckets.

Figure 1: Schematic illustration of a hash map. A user ID is hashed together with a random salt to map each user to a unique bucket.

The Challenge of Exclusive Experiment Programs

Experiments at Spotify fall into two categories: exclusive and nonexclusive. Nonexclusive experiments randomly overlap in their user samples. Exclusive experiments operate on disjoint user sets—a user can be part of at most one experiment in an exclusive group at any moment.

Figure 2: Illustration of exclusive and nonexclusive experiments. Exclusive experiments never overlap with each other in terms of users; nonexclusive experiments randomly overlap with exclusive experiments and other nonexclusive experiments. Note that the allocations in this figure were selected for illustration; in a true random sample we would expect exclusive experiments to also be spread out uniformly.

The statistical difficulty arises with programs of exclusive experiments: sequences of experiments over time where simultaneous experiments never share users. Spotify runs such programs for several surfaces, including Search, the Home screen, and certain backend components. The constraints surface when considering paths—the sequence of experiments a user can follow over time. In a program with five experiments, as shown below, certain paths are impossible. For instance, experiments 3 and 4 overlap in time and are exclusive, so no user can move from one to the other.

Figure 3: Paths of experiences possible during a hypothetical program of 5 exclusive experiments.

The number of possible paths grows combinatorially over time, and any given sampling strategy can only realize a small fraction of them.

Key Statistical Findings

The paper reports two principal results:

  1. Nonexclusive experiments retain valid inference. With unrestricted bucket sampling—Bucket Reuse applied to nonexclusive experiments—the difference-in-means estimator for the average treatment effect behaves approximately as it would under simple random sampling of units. Standard t-tests remain valid. This finding connects Bucket Reuse to the established literature on randomized experiments within complex sampling designs, including work by Horvitz and Thompson (1952) and van den Brakel and Renssen (2005).
  2. Bias in exclusive programs is bounded in time. The bias introduced by restricted bucket sampling in exclusive experiment programs is derived analytically. It is shown to be confined largely to the period immediately preceding an experiment. For any empirical program, the length of this bias-inducing history window—call it D days—can be estimated. In practical terms, the sample at time T is random with respect to events before T-D, but not with respect to the intervening D days. If those recent days involved experiments that systematically differentiate the available bucket population from the general population on the outcome of interest, estimators risk bias. Experimenters can therefore check what ran in the program over the last D days before launching a new experiment.

Why Spotify Adopted Bucket Reuse

Spotify has migrated its experimentation platform to rely on Bucket Reuse for all experiments. The choice stems from several practical advantages:

  • Simplicity: The mechanism is easy to implement and explain.
  • Scalability and speed: New users are uniformly hashed into existing buckets as they join, allowing the system to grow without architectural changes or performance loss.
  • Operational flexibility: A single company-wide bucket structure makes arbitrary coordination straightforward. Programs can be merged into exclusive groups at any time, and samples from problematic past experiments can be excluded from all future experiments since sampling units remain constant over time.

Spotify currently uses 1,000,000 buckets. While the paper shows that even smaller bucket counts support valid inference for average treatment effects, larger counts are statistically preferable—cluster sampling effects on other estimands diminish as the bucket count grows. The 1M figure was selected as the practical maximum that allows the set of selected buckets to be stored in an executable script within a standard database, avoiding the need for BLOB storage.

Spotify does not plan to reshuffle users into new buckets. However, natural periods of low experimentation—such as the winter holidays, when unusual listening behavior reduces general product experiments—provide effective resets, breaking dependencies between samples in exclusive programs.

To support experimenters running exclusive programs, Spotify is building tooling to monitor short-term dependencies. One tool will estimate, per program, the length of the history window that can bias results. Another will display the recent experiment history of available buckets at any point in time, allowing experimenters to judge whether the experiences those users previously received are likely to cause bias in a new experiment.

Figure 4: Dependency graph that shows the experimenter where the free space, and thereby their sample, will come from in terms of previous experiments in the exclusive program. Each rectangle corresponds to a previously run experiment. The numbers in the yellow circles indicate the percentage points of the population that went from one experiment into another, and finally into the proportion of the population that is available (“free space”) for sampling right now. The experimenter can see what effects the previous experiments had on the metrics of interest in their experiment.

Figure above for illustrative purposes only.

Statistical validity depends on proper random sampling and random treatment allocation. But building systems that deliver hundreds of experiments to hundreds of millions of users, while maintaining the ability to coordinate exclusivity without overlap, requires balancing statistical ideals against technical constraints. Bucket Reuse offers speed, simplicity, and flexibility at scale, and the new analysis confirms that its use does not compromise the validity of causal inference. The migration enables Spotify to run more experiments—and more reliable ones—than before.