The Rise (and Lessons Learned) of ML Models to Personalize Content on Home (Part I)

Home page recommendations at Spotify

Spotify's Home page connects listeners with creators by recommending music and podcasts tailored to each user. Like most recommendation systems, the Home page relies on two stages: candidate generation, which selects the best albums, playlists, artists, and podcasts for a listener, and ranking, which orders those candidates for maximum relevance.

Part I of this series covers the first stage: the machine learning models that personalize Home page content, with a focus on what we learned building, experimenting, and deploying them.

The models behind the shelves

• The Podcast Model: Predicts podcasts a listener is likely to listen to in the Shows you might like shelf.
• The Shortcuts Model: Predicts the listener's next familiar listen in the Shortcuts feature.
• The Playlists Model: Predicts the playlists a new listener is likely to listen to in the Try something else shelf.

These models sit at different points on a spectrum of listeners — from those who want familiar content to those who want novel recommendations.

Simplifying the ML workflow

Moving a model from experimentation to production is rarely straightforward. Managing training data, running experiments, monitoring performance, and retraining models each introduce their own complexity. Our ML infrastructure has always aimed for simplicity, but the path from our first batch-prediction models to today's fully online serving stack has made deployment and maintenance significantly easier.

An ML workflow generally breaks into three phases: data management, experimentation, and operationalization. In practice, teams iterate on training and evaluation until a model version performs best, then deploy it to production. Once live, the model — and the services and pipelines that support it — must be monitored, retrained, and versioned.

Our current stack automates much of this. Feature logging is instrumented directly in our serving infrastructure, with scheduled Scio pipelines transforming features and Kubeflow pipelines handling weekly retraining. Data validation runs on training and serving features, checking that schemas and distributions stay consistent between the two. The Kubeflow pipelines also evaluate models and automatically push them to production when scores clear our threshold. Alerts on both the validation pipeline and online deployments let us catch issues quickly.

A cautionary tale about training versus serving data

When we start on a new problem, we always begin with the data — what's useful, what's available, and what edge cases exist. We get comfortable with the contents of the data used for training features, but features fetched and transformed at serving time are a different matter.

Historically, training and serving used separate infrastructure. That split worked fine for batch predictions, where the same transformation code could run in both stages. But when the Podcast Model moved to real-time serving, feature processing moved into a new prediction service, separate from where training features were prepared.

Models are opaque, and testing their output directly is hard. So when a subtle discrepancy crept in — one feature was being transformed slightly differently at serving time than at training time — it went undetected for four months, silently degrading recommendations. The fix was a one-line change, but the underlying problem needed a structural solution: either a single source of data for both training and serving, or a guarantee that data is produced and transformed identically in both stages.

Unifying feature transformation

Our first remedy was to ensure all feature processing happens in the same code path, so training and serving features are transformed identically. For the Shortcuts Model, that meant retiring a Python service that ran continuously, checking every day whether it was Monday before requesting data at a rate-limited 5 requests/second and transforming it into features. The process took over 24 hours, so it couldn't be scheduled as a pipeline — and logging features from a service owned by another squad was impractical.

Moving transformation logic into the serving infrastructure gave us automatic feature logging of already-transformed features, which could then be reused for training. We now apply this pattern to all our models — it solves the consistency problem and reduces extra infrastructure.

Validating data to catch drift

The second safeguard compares training and serving data directly. We use TensorFlow Data Validation (TFDV) to compare schemas and feature distributions daily, alerting us to significant differences via the Chebyshev distance metric, which measures the distance between two vectors and flags drift.

The lesson is straightforward: it's easy to introduce mistakes when moving models to production, often because a different processing library ends up in the serving path. Validation doesn't stop every inconsistency from happening, but it ensures we know when something changed and can act before the impact compounds.

Part II of this series will cover offline and online evaluation metrics, why manually inspecting recommendations matters, and the challenges we faced bringing CI/CD to model retraining.