Spotify’s New Experimentation Platform (Part 1)

Rebuilding Spotify's Experimentation Stack

Spotify has long leaned on A/B testing to validate product ideas. What started as an ad hoc practice in the company's early days grew into a formal system in 2013 with the launch of ABBA, an experimentation platform that gave teams a place to track running tests, evaluate feature flags, and compute results. ABBA became embedded across desktop and mobile clients, backend services, data pipelines, in-app messaging, and email campaigns.

ABBA's model was straightforward: each experiment or rollout mapped one-to-one to a feature flag named after the experiment. When a client resolved a flag, it received the treatment group name—"Control," "Enabled," or sometimes even JSON-encoded configurations that users had creatively stuffed into group names. Every flag resolution logged an event that fed into exposure and results pipelines. But ABBA only computed a small set of metrics, many of which weren't sensitive enough, so most analysis happened manually in notebooks.

Growing Pains

By 2017, the limitations were hard to ignore. Large projects demanding heavy experimentation exposed several structural problems:

• Slow iteration: The one-to-one mapping between experiments and feature flags meant a failed experiment couldn't simply be restarted. Teams had to create a new experiment and update software to reference a new flag—a costly cycle.
• Event bloat: A/B testing events had grown to roughly 25% of Spotify's total event volume, driving up processing costs and causing incidents in the event delivery system.
• Weak analysis: Out-of-the-box metrics weren't sufficient, and data scientists were doing repetitive, inconsistent analyses in notebooks. The platform needed support for custom metrics and a standardized methodology.
• Risky coordination: Allocating users to experiments required manually coordinating bucket ranges between teams—an error-prone process where a mistake could corrupt many experiments at once.

At a hack week in late 2017, a group of senior engineers sketched out a replacement. The result is the Experimentation Platform, built from three components: Remote Configuration, Metrics Catalog, and Experiment Planner.

Remote Configuration: Properties Instead of Flags

Remote Configuration replaces Spotify's feature-flagging service. Rather than "flags," the model centers on properties—typed variables (enum or integer) with default values that control appearance or behavior in a client or backend service. A property might govern the number of shelves on the home page, the color of buttons, or the size of fonts. The same properties serve experiments, rollouts, personalization, and localization.

Properties are defined in a YAML file alongside the code that consumes them. At build time, all properties and their defaults are collected and published via API to the admin interface, along with the client ID and version number. The default value plays a critical role: it gives the system a programmatic guarantee of what users will experience if the client can't fetch or apply new values. Because defaults are known at build time, the system only needs to transfer values that differ from them, minimizing data traffic on client startup. Clients identify themselves by version so the server knows which defaults apply.

Assigning different values to different users happens through policies. A policy consists of filtering criteria plus a property-value mapping that applies when filters match. Policies are implemented as PlanOut scripts executed by the Remote Configuration service.

Fetching property values generates two events:

• Config Assigned—records that a user fetched values, including which policies were applied. This determines which experiments a user was exposed to.
• Config Applied—records that the device actually started using those values and serves as the trigger for exposure.

Values are re-fetched in the background periodically, but only applied when the app relaunches. Spotify deliberately avoids changing the user experience mid-session.

Metrics Catalog: Centralized Metric Management

The Metrics Catalog manages, stores, and serves metrics to the Experimentation Platform. Raw metric data flows through a pipeline that joins it with experiment group assignments, aggregates the result into an OLAP cube, and loads it into a data warehouse. A front-end API lets consumers query data without needing to understand the underlying storage.

Exposure is assembled from Config Assigned and Config Applied events. A user is exposed to an experiment when all of the following are true:

• A Config Assigned event assigns the user to one of the experiment's groups
• A Config Applied event confirms the user started using that configuration
• Optionally, the user exists in a specified "custom exposure source"

Custom exposure sources refine exposure granularity—for example, defining exposure as the moment a user visits a specific mobile page.

Experiment Planner: The Orchestration Layer

Experiment Planner sits atop Remote Configuration and Metrics Catalog, handling experiment creation, launch, stop, and results analysis. The UI is embedded in Backstage, Spotify's developer portal, where all internal teams have access to create experiments freely.

Creating an experiment requires defining test treatments, the property values users should experience in each treatment, and the ingredients of hypothesis testing. Because Remote Configuration exposes programmatic knowledge of available properties and their types, configuration errors are reduced. A single experiment can define values for properties across different systems—so one experiment can run on both Android and iOS even if those clients are implemented differently.

Spotify has spent the last two years rebuilding its experimentation capabilities. The company considers the new platform a step change in usability and functionality, and experimentation remains an area of ongoing evolution and investment.