Spotify Opens Up Its Homegrown Experimentation Stack
Spotify is preparing to release its internal experimentation platform as a commercial product called Confidence. The platform, currently in private beta, is the result of over a decade of building and refining tools for running controlled product tests at scale — from basic A/B tests to complex, coordinated experiments across mobile, desktop, and web.
The company’s goal with Confidence: make it easy for any team to set up, run, and analyze user experiments without needing deep statistical expertise or manual coordination, while still offering flexibility for teams that need custom experimentation methods.
From Manual Tests to a Company-Wide System
Spotify’s experimentation journey started in the early 2010s, when a few data scientists and engineers ran small A/B tests manually. Recognizing the limitations of that approach, the company built a homegrown platform called ABBA, which provided feature flagging and analysis for standardized metrics. That simple foundation drove adoption from fewer than 20 priority experiments per year to hundreds across multiple squads.
But that growth exposed ABBA’s limits. As experimentation scaled, teams became bogged down restarting tests, manually calculating statistical analyses in notebooks, and coordinating test groups in spreadsheets. The old architecture couldn't handle the growing complexity of experiments — particularly as Spotify began testing machine learning-powered recommendations, running tests across numerous devices, and managing an expanding user base.
In response, Spotify rebuilt its experimentation platform. Two improvements proved central:
- A Metrics Catalog that made metric analysis self-service, eliminating the need for data scientists to run analyses manually.
- A coordination engine that allowed many mutually exclusive experiments to run simultaneously, including the management of holdback groups.
These changes automated scientific best practices and removed the need for manual scheduling, letting teams run experiments independently and reliably. By the time ABBA was retired in 2020, Spotify was running thousands of experiments per year, not just on app features but also on backend services and data pipelines.
Fewer Friction Points, More Reliable Results
The company’s experience taught it a clear lesson: running more tests in itself isn’t progress if the testing methods are flawed. Bottlenecks, workarounds, and errors pile up when teams have to manually manage test groups and statistics. Spotify’s platform approach focused on embedding scientific rigor into the tooling itself, so teams could trust their results and iterate quickly. This "think it, build it, ship it, tweak it" philosophy is what the company says lets it converge on its best ideas faster.
Three Ways to Deploy Confidence
Confidence is designed to serve teams at different stages of their experimentation journey — whether they’re just starting out or have outgrown their current platform. To accommodate that range, Spotify says the platform will be available in three forms:
Managed service. The platform runs as a standalone web service managed by Spotify, for teams that want low technical overhead and quick setup.
Backstage plugin. For companies already running Backstage — Spotify’s open-source developer portal, which it donated to the CNCF — Confidence will be available as a plugin alongside other developer tools. This is how Spotify itself runs its experimentation platform.
APIs. Teams that need maximum customization — like building a bandit or running switchback tests — can integrate Confidence into their own infrastructure using its APIs.
Confidence is currently available as a private beta only, with a waitlist open for interested teams. Spotify has not yet announced a general availability date.
