The Rise (and Lessons Learned) of ML Models to Personalize Content on Home (Part II)

From Metrics Only to Full Visibility

Static numbers rarely tell the whole story of how a model recommends content. To get that story, Spotify’s Home team combines its offline evaluation toolbox with purpose-built dashboards that show actual recommendations for specific listener features. The result is a deeper, more actionable view of model behavior than any single metric can provide.

Augmenting metrics with hand-built dashboards

Past incidents have shown the team that evaluation metrics can miss important flaws in how a model recommends content. Sometimes the clearest way to assess a model is to simply look at what it suggests given particular listener attributes. To support this, the team built a dashboard that loads a model from its storage location and lets engineers compare multiple models side by side for a given feature set. Before pushing a new model to production, engineers frequently test it by generating predictions with varied feature values, which builds intuition about the content that different users would receive.

This approach has repeatedly surfaced serious issues that metrics would have missed. For instance, while developing one new model, the team noticed it kept recommending the same popular playlist to listeners across all European countries. That discovery led to a fix before deployment. Similarly, in early work on an album recommendation model based on locality and listening preferences, the dashboard revealed that one album consistently ranked first regardless of input features like country or followed artists — meaning that recommendation would have been served to nearly everyone. Offline metrics might have flagged poor performance in these cases, but they would not have explained why, whereas direct inspection made the problem obvious.

Combining task-specific custom evaluations with dashboards that visualize both metrics and sample recommendations has given the team a much clearer picture of how their models operate — pulling back the curtain and removing some of the “black box” nature of these systems.

Structuring the Experimentation Backbone

Evaluating models is only half the equation; supporting infrastructure must also scale with demand. The team’s experimentation pipeline has evolved significantly, addressing key pain points along the way.

Growing pains with a siloed experimentation platform

For the initial Podcast Model and Shortcuts Model, experiments ran on a dedicated platform used primarily within the team. It could launch hundreds of experiments with ease via configuration files specifying hyperparameters, including grid search support. Because submissions were scripts, the platform handled custom evaluation metrics — a long-standing priority for the team. However, it was not scalable, lacked maintenance, and suffered from an incomplete UI. Occasionally, compute instances would lose their API connection and continue running as unconnected “ghost workers.”

Integrating with the broader ML ecosystem

Spotify’s managed Kubeflow clusters offered scalability, modularity, and compatibility with the company’s wider ML infrastructure, making the migration an obvious one. Training models through Kubeflow pipelines proved efficient, but evaluation presented two major hurdles. First, since Spotify’s Kubeflow SDK depends on Tensorflow Model Analysis (TFMA), comparing a non-ML heuristic to a trained model required significant extra setup. Second, the team regularly needs custom evaluation metrics specific to each task, but implementing these in TFMA is considerably more complex than writing the same logic in regular Python.

Choosing the right evaluation path

Start with a strong baseline

A fundamental question guides this work: what makes recommendations “better,” and compared to what? A solid baseline — usually a quick heuristic — provides the necessary reference point for answering these questions. For the Shortcuts Model, for example, the initial heuristic simply picked the most frequently played items from a listener’s recent history. After many iterations to improve it, the team could confidently compare it to trained models and determine that the ML approach offered a genuine improvement over the heuristic and its maintenance overhead.

Leveling the comparison playing field

The real difficulty emerges when attempting to compare these fundamentally different solutions fairly. In an ideal scenario, one could run extensive A/B tests with many test cells to test solutions against real listeners. Lacking that luxury requires reliable offline metrics as proxies.

Typically, the team uses normalized discounted cumulative gain (NDCG@k) for recommendation models, implementable via Spotify’s Python SDK for Kubeflow pipelines. But how does one apply the same evaluation to a heuristic, which is usually written in a Java service and tested only with unit tests for correctness, not performance? Consistent evaluation logic and shared test data are essential to make such comparisons meaningful.

For simpler heuristics the team found an elegant workaround: in effect “training a model” that replicates the heuristic’s output. This scheme allows the use of the same evaluation script and test set as for trained models. The same method was used for the Try something else shelf recommendations for new users, where a popularity heuristic driven by listener demographics was computed within Tensorflow Transform (TFT) and the model acted as a lookup utility with a fake loss.

But not all problems can be forced into such a simple heuristic. For the Shortcuts feature, the logic proved too intricate to express in Tensorflow, prompting the team to build a fully separate offline evaluation pipeline. This pipeline collects recommendations from both models and heuristics and applies custom evaluation functions for direct comparison.

Freedom and flexibility in evaluation

A second common pain point is custom metric implementation in TFMA. The platform supplies standard measures like precision, recall, and accuracy out of the box, plus NDCG at a raw level, but not NDCG@k — which is the team’s standard. Whereas NDCG@k can be computed in a single line with scikit-learn in Python, the TFMA implementation requires roughly 120 lines. Likewise, for a model predicting playlists new users would try, the team wanted to verify it wasn’t just returning popular content. Their diversity metric — measuring differences in playlist characteristics — was nearly impossible to implement in TFMA. As a result, the team contributed support for arbitrary custom Python evaluation to Spotify’s Kubeflow SDK. Experiments have run this way since October 2020.

Moving to Kubeflow also replaced the previous in-house experiment tracking with Spotify’s internal ML UI, which integrates cleanly with pipeline runs. The interface permits side-by-side comparisons of evaluation scores for both NDCG and custom metrics, while also allowing the team to track model deployments over time.

Ultimately, pairing task-driven evaluations with dashboard views that log both metrics and per-feature recommendations has allowed the team to build a nuanced understanding of their models’ behavior — all while keeping the inference layer genuinely interpretable.

Retraining isn’t optional

Training once and deploying forever is a nice idea, but it doesn’t hold up in practice. The Podcast Model has always had retraining configured, because it only recommends shows it has seen during training — without scheduled retraining, new shows would never surface. But the Shortcuts Model tells a different story. It only recommends content the listener has already played, so retraining wasn’t strictly necessary for the model to function. That turned out to be a costly assumption.

Months after launch, the Shortcuts Model started failing to recommend a newly introduced content type, because the features describing content categories were trained before that type existed. Beyond that, the model’s library versions fell behind, making migrations to newer tools and platforms painful. The lesson: even when retraining seems unnecessary for the task, skipping it accumulates technical debt that eventually surfaces in unpredictable ways.

Short-term serving solutions and their costs

The original Podcast Model ran as a batch pipeline using Scio with Zoltar, generating predictions stored in Bigtable. That worked, but predictions were only as fresh as the batch schedule, and listeners’ features change as they listen to new content or follow artists. To get fresher recommendations, the team built a dedicated recommender service that could serve online predictions on demand — for example, right after a listener follows a new artist.

This service was always meant to be temporary. Spotify’s internal online serving platform wasn’t production-ready yet, and the team decided the benefits of faster iteration outweighed waiting for the ideal solution. The service refreshed its local copy of the model by polling internal storage every 10 minutes for a new revision, then pulling it down when one appeared. That worked, though it meant different machines could briefly run different model versions — acceptable in this case, but not a robust pattern.

Deploying a model manually was the real bottleneck. Each deployment required copying the model to a storage location, generating a pointer to that location, and wiring the pointer into the recommender service along with feature fetch and transform logic. Every retrain meant repeating the entire sequence. Cumbersome as it was, the setup did allow the team to get four models into production and test others via A/B experiments over roughly 10 months.

Moving to a CI/CD pipeline for models

The eventual migration to Spotify’s model serving platform automated most of the manual loop. With the platform handling model version pushes from Kubeflow pipelines and providing feature logging, the team could stop polling for updates and start treating model deployment like standard CI/CD.

Feature transformations were a notable sticking point. The team doesn’t use TensorFlow Transform (TFT) because many of their transformations are too complex to express cleanly in TensorFlow. Instead, they leveraged the serving platform’s feature logging to record already-transformed features, apply labels, and split train and test sets — all in scheduled weekly pipelines.

The migration itself was straightforward: dark-load all prediction traffic to the new deployment, then gradually shift live traffic away from the Zoltar-based service. The serving platform brought lower latencies, feature logging, and less code for the team to maintain.

With models deployed through the platform, retraining and rollout became fully automatable. A scheduled Kubeflow pipeline retrains models weekly, and before any new version is pushed to production, the pipeline checks that its evaluation score clears a configured threshold. Only models that pass are automatically deployed.

Building CI/CD for ML is harder than it should be, but the tooling is improving. And the payoff isn’t just fresher recommendations — it’s a predictable, reliable process for keeping models healthy over time. Even for models whose task doesn’t obviously require retraining, the alternative is silent degradation and growing technical debt.

Still refining the stack

The current infrastructure has cut the time needed to iterate, experiment, and ship quality models. The team still has open problems — data and model versioning, moving feature transformations into TFT, and better ways to compare offline metrics across ML and non-ML solutions. But the core loop of data management, experimentation, and continuous deployment is now in place, supporting models that serve millions of listeners on Home daily.