Advancing Generative Recommendation Inference: our collaboration with NVIDIA on TensorRT LLM

Spotify's personalization surfaces — the Home feed, autoplay, and conversational features — have traditionally been served by multi-stage candidate retrieval and ranking pipelines, sometimes with a separate system per surface. The company has been testing an alternative: a Large Personalized Language Model that produces recommendations in real time across multiple surfaces. Moving that idea from proof of concept to production exposed limits in the available inference engines, which is the problem its collaboration with NVIDIA set out to solve.

Beam search and inference engine trade-offs

Spotify introduced LLM-powered technology to podcast recommendations in January 2026, building on the generative recommendation methods in the PLUM paper and on its own work, "Deploying Semantic ID-based Generative Retrieval for Large-Scale Podcast Discovery at Spotify." That architecture uses beam search to explore several token sequence paths at once for candidate generation, keeping the most promising beams active at each step and converting the highest-scoring sequences into semantic IDs that map back to catalog items.

The initial release runs on vLLM, a widely adopted open source inference engine with strong support for modern model architectures in high-throughput deployments. Its beam search support, however, is weaker than for ordinary completion requests and does not extract maximum performance from GPUs. The implementation operates outside the core GPU engine, creating CPU-side bottlenecks and extra GPU work from recomputing partially filled KV blocks. For the January launch this meant high-end GPUs could not be fully utilized, so Spotify chose lower-tier accelerators to control cost while keeping the system at an optimal load point.

Candidate scoring and personalization

A second technique under exploration is candidate scoring, which uses a model's log probabilities to evaluate and rank a set of candidates. The naive approach fans out inference per candidate per request, then extracts and aggregates the logprobs. This is broadly available in open source engines, but a more efficient variant — scoring multiple candidates in a single request via candidate masking — reached SGLang partly through LinkedIn's engineering team. Spotify has used SGLang to access that feature at inference time.

Personalization has historically relied on dense user embeddings at scale; Spotify explored three ways to carry it into the LLM paradigm.

• Plain user text. Listening and interaction history is converted to text and placed in the prompt. The amount of text needed to represent a user accurately makes prompts extremely large, driving latency beyond the request budget and raising GPU usage. It is used sparingly, mainly for small cases such as a user's most recent real-time streams.
• User semantic IDs. As in PLUM, user history is represented as a semantic ID, extending the model vocabulary with user semantic ID tokens that can be added to prompts with a handful of text tokens. Prompts stay short and need no custom inference requirements; the LLM was fine-tuned on the user semantic ID space and extended vocabulary. Early results were promising, but the approach needs further experimentation and research before production.
• Soft tokens. The most successful method builds on existing user embeddings and is outlined in the Embedding to Prefix white paper from Spotify Research. A projection is learned from an existing embedding space into the LLM's embedding space as a single soft token, and the LLM is fine-tuned on a subset of weights to consume it. At inference time the user vector is projected and inserted into the prompt before the initial forward pass. It has been used successfully for podcast recommendations and is being integrated and evaluated for various tasks with Spotify's internal LLM model.

Unifying features in TensorRT LLM

With the first LLM-powered system rolled out and internal experimentation continuing, the effort widened to bringing generative recommendation across catalog offerings and surfaces. Production use made an efficient, manageable inference stack a priority, and the NVIDIA collaboration began with a concrete goal: combine efficient beam search, scoring, and user soft tokens in a single engine.

TensorRT LLM (TRT-LLM) offered better hardware utilization for beam search, but initially lacked the Score API available in the SGLang-based setup and, compared with vLLM, could not inject soft tokens or return sequences for multiple beams. Working with NVIDIA engineers — Marek, Martin, Stefan, and Alexander — Spotify set out to close those gaps. The work produced several upstream additions:

• support for returning multiple sequences in beam search (v1.2.0rc3);
• support for out-of-tree models in trtllm-serve (v1.2.0rc4);
• support for passing multi-modal embeddings via image_embeds (v1.3.0rc0);
• support for multi-item scoring via the Python API (v1.3.0rc20);
• general bug fixes and performance improvements.

Beam search in TensorRT LLM runs as part of the sampler, reducing host overhead; together with overlap scheduling this keeps GPU utilization high. The C++-based sampler has been replaced with a more flexible design leaning on PyTorch, and beam search on the new sampler delivered a notable performance uplift over the previous implementation. Using TensorRT LLM's out-of-tree model support, and with NVIDIA's help and review, Spotify implemented a customized Qwen3 extension plugin for the soft token approach: it accepts user vectors through multi-modal request parameters and performs the user embedding projection and soft token replacement.

The multi-item scoring extensions to the Python API provide functionality equivalent to SGLang's scoring API. A custom serving layer doing efficient tokenization and request batching is still required, though TensorRT LLM continues to evolve and has recently added support for serving prefill-only embeddings. Multi-item scoring matters as a performance optimization because it avoids the overhead of one request per candidate while keeping computational effort linear in the number of candidates.

These changes have landed in recent TensorRT LLM versions.

The gains translated into real cost reduction for Spotify's generative recommendation applications: for the 1.7B variant of Qwen3, up to 54% lower cost for beam search workloads and up to 7% for scoring workloads.

Next steps

Beyond resolving the immediate inference problems, the collaboration broadened Spotify's view of where further efficiency gains might come from. As generative recommendation models spread across the company, inference performance and cost efficiency stay priorities. Further work with NVIDIA will focus on quantization and NVIDIA Blackwell-specific techniques to draw more performance from the latest GPU architectures, with the aim of supporting increasingly sophisticated personalization at scale while keeping latency and infrastructure costs in check.

Contributors to the collaboration and to the original post from Spotify and NVIDIA: Ivo Senese, Reid Wilbur, Mark VanMiddlesworth, Guy Stone, Marek Pikulski, Stefan Niebler, and Alexander Young.