Spotify’s Search team wants every product decision to be backed by reliable evidence. That ambition requires two pillars: an experimentation platform capable of running tests at scale with accurate results, and a development culture where hypothesis testing is a core habit.

Spotify has spent two years building out its new Experimentation Platform, which covers much of the platform requirement. Documentation and best practices for the platform are mature, and the tooling already offers most of what teams need. The harder part is cultural: integrating those practices into daily workflows across a large, multidisciplinary team. Search consists of several engineering teams, product managers, a product insights group with data scientists, and product area leads. Keeping engineering teams independent from data scientists avoids bottlenecks, but it also means experimentation skills must spread beyond the data science function.

Advice on how to run a proper experiment — power analysis, metric selection, p-value interpretation — is abundant. Turning that advice into habits is another matter. It requires building an environment where adopting new ways of working feels empowering rather than uncomfortable, especially when experience levels vary widely across disciplines.

Two key drivers made the difference for the Search team: a deliberate roadmap for maturing practices, and a sustained injection of energy into the teams adopting them.

A Roadmap in Three Steps

The path to full data-informed development can overwhelm teams if it is pushed too hard. The Search team created a roadmap that respects each team's maturity and bandwidth, with three stages:

  1. Individual experiment quality — raising the standard of each test
  2. Cross-experiments quality — coordinating tests that share user populations
  3. Total business impact — measuring the combined effect of everything shipped

Step 1: Individual Experiment Quality

For an experiment to support a product decision reliably, several elements must be in place: a clear hypothesis, metrics that map to that hypothesis, and decision rules defined before the test starts. These practices can be introduced gradually, so each one is easy to absorb. Over time they compound into experiments that generate trustworthy decisions, and as quality rises, the number of experiments teams run tends to rise too.

At Spotify Search, faster iteration means decisions must be transparent and self-explanatory — for new team members and for teams outside the one running the test. Shared conventions help: consistent naming for experiments, a standard hypothesis format, and a log of decisions after an experiment concludes. The Search data science team supplied simple templates for test specifications and experiment setups, which nudge people toward the important steps while keeping practices uniform across teams.

Step 2: Cross-Experiment Quality

Most of the time, teams run several experiments simultaneously, and some of those tests need coordination so the same user is not enrolled in conflicting treatments. Spotify's experimentation platform has built-in coordination capabilities, but using them at scale still requires alignment between teams. The data science group helps experimenting teams choose the right coordination setup for each test and keeps teams aware of the available user population. Templates established in Step 1 make this easier: pre-selected coordination behaviors in the setup guide team members toward the right configuration.

Step 3: Measuring Total Business Impact

The final stage answers: "What is the combined causal effect of all our product changes on key metrics?" A naive approach would sum the estimated effects from individual shipped experiments. Such an estimate inherits many assumptions and does not capture interactions between changes.

Spotify Search instead relies on quarterly holdbacks. A set of users is held back from all product changes during the quarter. After the quarter ends, a single experiment runs on those users, with one group receiving no changes (control) and another receiving every shipped change (treatment). That design provides an unbiased estimate of the total causal effect of the quarter's work. The holdback has become a frame for the other two steps: it moves evaluation from one team's experiment to several related tests and finally to a whole product initiative.

Notably, these steps grow from within one team outward to many teams, and from a single experiment outward to initiatives. Measuring impact for a whole initiative with good precision is genuinely hard, and it is harder still if product evaluation is not yet natural to the culture. The Search team believes the fastest route to whole-program evaluation is bottom-up: first building reliable practices in each team, then aggregating upward. Along the way, each step adds value beyond measurement alone — better product quality awareness, improved evaluation habits, knowledge sharing, and stronger alignment between product teams.

Sustaining Momentum: The Second Driver

A roadmap only works if teams have the energy to follow it. Building solid experimentation practices is like lighting a fire with damp wood: a spark is not enough; you need steady care and the right fuel before the fire catches.

Early on, new practices frequently conflict with existing habits, and convincing teams of the value takes time. The data science group found it necessary to do substantial work for the engineering teams rather than expecting them to handle everything themselves. Reviewing test designs and setting up experiments on behalf of a team demonstrates that the effort is small day to day and that the return is real. Doing the work also builds trust, reassuring engineers that expert support is available.

Early experiments will not be perfect, and some product decisions will not be fully data-informed. The priority is supporting teams so they can succeed, with enough reinforcement to keep experiments running. Continuous energy beats perfect timing. As one search data scientist put it: "The smallest lighter can make any wet wood burn with sufficient time."

Eventually, the energy must scale past the advocating team. In practice, members of other teams develop their own interest and expertise, taking over roles as advocates and continuing the work in their own squads. But organizational changes and new joiners constantly threaten alignment. The fire may burn on its own after a while, yet it still needs a few dedicated people to sustain it — restarting and boosting it continuously.

From Principles to Practice: Search’s Year of Experimentation

The principles behind Spotify Search’s experimentation program only became meaningful once they were applied systematically. The team’s 2021 roadmap shows how a sequence of modest, incremental actions — each building on the previous one — transformed how the product was built and evaluated.

  1. December 2020: The team unlocked the full capabilities of the existing experimentation platform (e.g. sample size calculation, result pages) by integrating Search-specific metrics directly into the system.
  2. January 2021: A dedicated weekly forum and communication channel was created to improve information flow between search engineering teams and data scientists.
  3. March 2021: Standardized test setup and specification templates with pre-filled parameters made it possible for anyone, regardless of prior experimentation knowledge, to create and launch tests with basic search-specific settings.
  4. June 2021: The first end-of-quarter cumulative experiment ran. With most new user-facing features already gated behind experiments, the team had enough data to measure the aggregate impact of all Search features launched in a quarter.
  5. August 2021: The Experimentation Champion program designated one person from each team to act as a knowledge bridge between the Search data science team and their own, easing the onboarding process for new members.
  6. September 2021: A naming convention for experiments was introduced, making it possible to identify the quarter and initiative associated with a given test simply from its name.
  7. September 2021: The Search team made a formal commitment to ship all product changes gradually using monitoring — a milestone that reflected growing confidence in the new practices.
  8. October 2021: The decision-making process was tightened by requiring three elements in every experiment design: a hypothesis, metrics, and decision rules.

The roadmap succeeded by concentrating on one small step at a time. Each change was manageable on its own, yet the cumulative effect over a single year was a substantial shift in Search’s approach to experimentation — improving the quality of individual tests, coordination between concurrent experiments, and the visibility of overall impact.

Beyond 2021: Broader Evaluation

In 2022, the next stage of this effort is to integrate experimentation into a wider evaluation process encompassing every product-development initiative. Rather than treating A/B tests as isolated events, the Search team intends to make them a component of a holistic, data-informed assessment framework for each initiative.

Culture Change in Practice

The journey demonstrated a fundamental insight: shifting development culture does not require top-down mandate. Instead, it emerged because people across disciplines — engineers, PMs, and data scientists — took ownership of improving the evaluation process. Engineers now regularly make the call to delay or cancel a launch to avoid a negative user experience, and PMs involved with Search work to educate PMs across Spotify on experimentation topics.

That behavioral evolution points to a repeatable recipe for accelerating change in product evaluation: establish a safe environment where teams can refine their workflows, provide steady energy and support, and make the benefits of each incremental improvement visibly obvious to those involved.