When Exact Words Aren’t Enough

Traditional search at Spotify has largely depended on term matching. A query like “electric cars climate impact” would return results based on whether indexed metadata—such as podcast episode titles—contains those exact words. While fuzzy matching, normalization, and manual aliases help bridge the gap between user intent and literal text, these approaches fall short when users express themselves in full natural language sentences.

In many cases, the exact-term approach simply fails to surface relevant content. For example, a search for “electric cars climate impact” might return nothing from a term-based index, even though Spotify’s catalog contains episodes that directly address the topic—just not with those precise words in their metadata.

To solve this, Spotify introduced Natural Language Search (also called Semantic Search in the literature). Rather than requiring exact word matches, this technique matches a query to a textual document based on semantic correlation—synonyms, paraphrases, and other variations of natural language that convey the same meaning. The feature is now deployed for most Spotify users, with podcast episode retrieval as the first application.

With this system, a query like the one above can return episodes whose titles don’t contain all query words, yet are clearly relevant to the user’s intent. The implementation leverages recent advances in deep learning and NLP, including self-supervised learning, Transformer neural networks, and Approximate Nearest Neighbor (ANN) techniques for fast online serving.

Dense Retrieval Architecture

The technical approach is Dense Retrieval, a machine learning method where a model is trained to produce both query and episode vectors in a shared embedding space. A vector is an array of float values, and the goal is to ensure that the vectors of a search query and a relevant episode end up close together in that space.

For queries, the query text itself is the model input. For episodes, the model consumes a concatenation of textual metadata fields—including the episode’s title and description, as well as its parent podcast show’s title and description. During live traffic, vector search techniques efficiently retrieve the episodes whose vectors are closest to the query vector.

Choosing the Base Model

Transformer models like BERT dominate NLP tasks, largely thanks to two properties: self-supervised pre-training on large text corpora (where the model learns to predict randomly masked words), and a bidirectional self-attention mechanism that produces high-quality contextual word embeddings.

However, vanilla BERT has limitations for Spotify’s use case:

  • Its pre-training focuses on high-quality word embeddings, but off-the-shelf sentence representations are not strong, as demonstrated in research on SBERT.
  • It was pre-trained on English text only, whereas multilingual support for queries and episodes was required.

After experiments, Spotify chose the Universal Sentence Encoder CMLM model as the base. This model uses Conditional Masked Language Modeling (CMLM), a self-supervised objective designed to produce high-quality sentence embeddings directly. It is also pre-trained on multilingual corpora spanning more than 100 languages and is publicly available on TFHub.

Training Data Preparation

Fine-tuning the pre-trained model for podcast search requires a carefully assembled dataset:

  • Successful past searches: (query, episode) pairs mined from Search logs, where results were returned via Elasticsearch.
  • Query reformulations: From user sessions, cases where an initial search failed and a follow-up succeeded. These yield (query_prior_to_successful_reformulation, episode) pairs, capturing more semantic relationships with no exact word matches.
  • Synthetic queries: Generated from popular episode titles and descriptions, inspired by research on embedding-based zero-shot retrieval through query generation. A BART transformer fine-tuned on the MS MARCO dataset produces these (synthetic_query, episode) pairs.
  • Curated set: A small manually written collection of semantic queries for popular episodes, used for evaluation only, not training.

When splitting data for training and evaluation, episodes in the evaluation set are kept out of the training set to verify the model’s generalization to new content.

Model Training with Hard Negatives

Training uses a siamese network setting where weights are shared between the query encoder and the episode encoder. Cosine similarity measures the distance between query and episode vectors.

Positive pairs are easy to derive from mining, but the model also needs negative pairs—examples where an episode should not be retrieved for a given query. Spotify uses a technique called in-batch negatives: for each (query, episode) positive pair in a training batch of size B, the episodes from other pairs in the same batch serve as negatives for that query. This yields B positive pairs and B² – B negative pairs per batch.

For computational efficiency, query and episode vectors are encoded once per positive pair, then an in-batch cosine similarity matrix is computed. The diagonal represents positive pair similarities; all other values represent negatives. Losses are applied to this matrix, including Mean Squared Error loss with the identity matrix as a label. Later iterations refined this with in-batch hard negative mining and margin loss, which substantially improved offline metrics.

Evaluation Approach

Two metric types are used to assess model quality:

  • In-batch metrics: Recall@1 and Mean Reciprocal Rank (MRR) computed efficiently at the batch level using in-batch negatives.
  • Full-retrieval metrics: Periodically during training, vectors for all episodes in the eval set are computed, and metrics like Recall@30 and MRR@30 are calculated using evaluation queries. Retrieval metrics are also computed on the curated dataset.

Production Integration

With fine-tuned query and episode encoders ready, integration into the production Search workflow requires two paths: offline indexing and online retrieval.

Offline Episode Indexing

Episode vectors are pre-computed for a large set of episodes using the episode encoder in an offline pipeline. These vectors are indexed in the Vespa search engine, which provides native support for ANN Search. ANN keeps retrieval latency acceptable across tens of millions of indexed episodes with minimal impact on retrieval metrics.

Vespa also supports a first-phase ranking function executed on each content node. This re-ranks the top ANN-retrieved episodes using additional features like episode popularity.

Online Query Encoding

When a user submits a query, the query vector is computed on the fly using Google Cloud Vertex AI, where the query encoder is deployed. Vertex AI was chosen primarily for its GPU inference support—Transformer models of this size are significantly more cost-effective on GPUs even during inference, with load tests showing a 6x cost reduction factor between T4 GPU and CPU. Once the query vector is ready, it retrieves the top 30 “semantic podcast episodes” from Vespa via ANN. A vector cache is also in place to avoid recomputing common query vectors.

Complement, Not Replacement

Dense Retrieval offers impressive semantic capabilities but has limitations when compared to traditional IR methods. It can struggle with exact term matching and is more expensive to run on all queries.

As a result, Natural Language Search is treated as an additional retrieval source rather than a replacement for existing ones, including the Elasticsearch cluster. In Spotify’s Search system, a final-stage reranking model takes the top candidates from each retrieval source and produces the final ranking shown to users. To help this model rank semantic candidates appropriately, the (query, episode) cosine similarity value was added to the reranker’s input features.

What the A/B Test Showed

The retrieval and ranking architecture behind Natural Language Search passed its first real-world check: an A/B test produced a statistically significant lift in podcast engagement. That result justified expanding the feature from the initial pilot to most users. With the rollout largely complete, the team is already turning toward the next round of model refinements rather than treating the launch as a finished project.

Where the Work Goes Next

Three areas are on the roadmap for the coming iterations:

  • Model architecture improvements — the underlying encoder and re-ranking layers are the first candidates for further tuning.
  • Better fusion of dense and sparse retrieval — the current hybrid approach works, but the team sees room to make the blending smarter rather than relying on static weights.
  • Coverage expansion — query types and catalog segments that were underrepresented in the initial training and evaluation data are earmarked for additional work.

The launch is treated as a first step, not a final state. The existing stack gives the team a base to iterate on, and the measurement infrastructure from the A/B test provides a clear way to validate whatever ships next.