A Forecast Platform That Keeps Pace With Growth

Spotify operates in more than 180 countries, and its business teams rely on forecasts of key user metrics to monitor performance and guide decisions. Those forecasts have to be accurate, and they have to be delivered on a schedule that matches how the business actually runs. The infrastructure supporting them needed to evolve accordingly.

Work began in Q1 2020 on an upgraded system designed to produce automated, high-quality market-level forecasts on a weekly cadence, with the ability to run on demand as well. By Q4 2020, the models were in place with advanced hyperparameter tuning that compressed work which used to take months on a single machine into a few hours of compute. To keep the system reliable, the team separated core logic from experimental adjustments, and built integration tests and visualizations to catch anomalies from a business perspective.

Pipeline Structure

After quality assurance on input data, the pipeline forks into three workstreams, each tailored to the data history available for a given market.

Mature Markets

Markets with a robust history can support time series models that learn from trend changes, seasonalities, and holiday effects. The first step is data processing, where one-time disruptions are removed so they are not propagated into future forecasts. Model training and inference follow.

New Markets

Newly launched or soon-to-be-launched markets have limited data, so the team built an in-house proxy-based model for these cases. External data, such as macroeconomic and music-related signals, trains clustering models to find “proxy” markets among the already-launched ones. Internal data then helps learn launch and growth patterns from those look-alikes. This cold-start problem is among the hardest to solve for accuracy, but the proxy approach has shown reliable and consistent forecast results after market entry.

Custom Models

Some markets are classified as custom or specialty because they require human involvement. This covers testing new research models, incorporating business input for accurate forecasts, and reviewing model results—for instance, a market that is heavily campaign-driven may need explicit business knowledge to forecast properly.

Compute and Orchestration

For tasks requiring parallelization across large data volumes or many model runs, the platform uses Dataflow (Google-managed Apache Beam). Smaller jobs are automated as dockerized Python workloads on Kubernetes.

The engineering-heavy core is the mature markets forecast, which needs dedicated data processing and labeling before hyperparameter tuning can begin. Its key components are:

  • Parameter-Space Creator — lets users define the hyperparameters evaluated via cross-validation and stores them in Google BigQuery.
  • Data Labeler — labels time series data (train / test / validation) and reformats it along with hyperparameters for parallel execution.
  • Hyperparameter Tuning — runs models in parallel across markets, cross-validation time splits, and hyperparameter combinations, evaluating an error metric for each. Originally done directly through the Apache Beam API, the grid search was later abstracted into an in-house package for time series forecasting at scale.
  • Model Selector — takes tuning output, computes a weighted error for each model to pick the top candidates, and validates them against the final holdout set.
  • Model Applier — runs the best model over the full time period for each market to produce the forecast.
  • QC & Manual Intervention — runs a series of checks on the final output, shared with internal stakeholders through dedicated channels. Easy-to-use tools let business users provide forward-looking input, such as knowledge of a major campaign in an upcoming quarter that models could not anticipate.
  • Publish & Visualization — generates automatic summaries and insights. Internal visibility was a design priority, so automated visualizations are distributed through multiple channels across the company.

Lessons From the Build

The team distilled several takeaways from the effort of building and operating this system.

Quality Control Has a Cost

Strict quality control creates significant development overhead, so the team applied it selectively. Core logic—model implementations—is packaged in a Python library that is unit tested, strictly peer reviewed, and well documented for every release. Peripheral logic, like chart and email creation, enjoys more flexibility. Developers can build extensions or experiment with models without facing the same standards.

Parallelization Isn’t Free

Scaling feels powerful, but it is not always the answer. Choosing models that fit the problem reduces computational load, and research can shrink the hyperparameter space so brute-force searches avoid regions with little impact on the outcome. Given the overhead of parallelization, it is reserved for cases where it is genuinely necessary.

Keep the System Simple

There is a constant tension between simplicity and incremental accuracy gains. Each potential change has to be evaluated: add it to the core, build it as an extension, or reject it because the cost outweighs the benefit. The natural tendency is to keep adding features at the expense of system health, but teams need to consider the whole picture before doing so.