Scaling Podcast Previews at Spotify
In March 2023, Spotify began serving short audio previews for music, podcasts, and audiobooks on its home feed. For podcasts specifically, this represented a major shift in discovery: instead of relying on cover art and static descriptions, listeners could hear actual audio snippets to decide whether to dive into an episode. Behind this feature lies a pipeline that generates previews for hundreds of thousands of new podcast episodes daily, a significant scale-up from the thousands of episodes the original preview engine could handle.
The groundwork came from Spotify's 2021 acquisition of Podz, a company building 60-second podcast previews using NLP and audio machine learning models. At acquisition time, Podz operated a microservices architecture where each ML model ran as its own service, orchestrated by a central API that transcribed new episodes and processed them via directed acyclic graph (DAG) logic. This setup worked for thousands of episodes per day but hit a wall against the podcast catalog's growth rate of hundreds of thousands of new episodes daily.
Spotify chose Google Dataflow running the Apache Beam Python SDK to build a fully managed, horizontally scaling pipeline. Dataflow handles Dockerized computational graphs over batch or streaming inputs, autoscales based on demand, and optimizes Beam transforms like ParDo and Map into fused pipeline stages for distributed processing. This avoided the operational overhead of running self-managed Kubernetes clusters.
Assembly and infrastructure decisions
The pipeline ingests raw audio and transcription data, applies source transforms for deduplication, and then runs an ensemble of over half a dozen ML models. These models, which include fine-tuned language models and sound event detection, span multiple frameworks: TensorFlow, PyTorch, Scikit-learn, and Gensim. Bringing them together raised three key questions:
- How to structure the models—as different nodes within a single Apache Beam transform or as separate transforms in the DAG.
- How to choose hardware and GPU types for latency and throughput targets.
- How to manage library dependencies across these heterogeneous frameworks.
For structure and hardware, Spotify factored in model sizes and grouped the bulk of models into one transform using NVIDIA T4 GPUs with 16 GB of memory. This reduced code complexity and data transfer between models, but meant swapping models in and out of GPU memory and concentrating logs and errors in one dense ParDo operation. The team inserted fusion breaks between transforms to ensure the GPU loaded only one stage of models at a time.
For dependencies, the pipeline relied on custom containers supported by Dataflow. Resolving dependencies for all these models in a single Docker image proved difficult. The team used Poetry to resolve dependencies ahead of Dockerization and consulted Dataflow SDK dependency guidance for version compatibility. Still, locally resolved and Dockerized dependencies could fail at runtime with resolution errors.
Generalizing to multiple pipelines
The result was an hourly scheduled, custom container Dockerized batch pipeline. To maintain multiple versions of the pipeline—production, staging, and a lightweight fallback for lower-viewership content or episodes the production pipeline missed—Spotify developed a common DAG structure with a shared code template. Dataflow-native timers and failure counters were added to profile code and track latency and error sources.
Running many GPU machines year-round carried a high cost. Initially, autoscaling was disabled because each batch job handled fixed-size hourly input partitions; the team estimated the worker count needed per input and spun up exactly that many machines. But latency from input queuing and machine spin-up and teardown pushed the team to reconsider. Since Apache Beam offers the same API for batch and streaming, switching modes required minimal code change.
The move to streaming brought autoscaling back into play. In streaming mode, Dataflow determines resource needs dynamically based on input traffic, reducing the time and cost of repeatedly spinning up and tearing down machines. This shift delivered the most significant gains in both cost and latency.
Latency: from two hours to minutes
The batch pipeline produced a preview about two hours after an episode was ingested. That delay was problematic for two reasons: time-sensitive content like daily news episodes, and highly anticipated releases from popular shows that listeners expect at specific moments.
Spotify explored using Klio, an open source Spotify project designed for processing audio files at scale, within its Beam and Dataflow pipeline. Klio builds both streaming and batch data pipelines, and its existing track record with music audio suggested it could similarly accelerate podcast preview generation. Within a week, Spotify had a streaming preview pipeline running.
Monitoring came next. Successful and failing inputs were logged to a BigQuery table, with exception messages captured for failures. Dashboards built with Google Metrics Explorer tracked backlog size in Pub/Sub queues and triggered alerts if backlogs grew too large.
The results were dramatic. Measuring median preview latency—time between episode ingestion and completion of preview generation—across two distinct weeks, one with only the batch pipeline and one with only the Klio streaming pipeline, the streaming approach cut median latency from 111.7 minutes to 3.7 minutes. That is roughly a 30-fold improvement. The latency distribution charts below show the streamed and batch performance for the leftmost 80% of processed episodes.
The new streaming pipeline generates previews 30 times faster, medians aside. This enabled near-real-time preview generation for time-sensitive podcast content.
Common pitfalls with custom containers
Both batch and streaming pipelines faced similar operational challenges on Dataflow, especially around dependency management when using custom containers via the --sdk_container_image flag or running jobs inside a VPN—both common at large companies. Debugging those errors is tricky because logs are often insufficient, and cloud workers may crash or restart immediately after an error occurs.
One specific case surfaced after upgrading the Apache Beam SDK from version 2.35.0 to 2.40.0. The pipeline began failing within five minutes of job startup with the error SDK harness disconnected, meaning the process running the pipeline code crashed—but no underlying error logs appeared. Small inputs from a BigQuery source succeeded, but jobs failed when inputs exceeded 10,000 rows, even though the same pipeline handled far more throughput before the upgrade.
After investigation, the issue was traced to a problematic package combination: grpcio==1.34.1 installed together with google-cloud-pubsublite==1.4.2. The failure did not occur when either package was updated to its latest version. Since Spotify didn't use Pub/Sub functionality in this pipeline, the fix was simple: uninstall google-cloud-pubsublite from the Docker image after installing all other dependencies.
This experience highlights a broader lesson. When upgrading the Beam SDK, pipelines using custom containers in VPN environments can fail due to subtle changes in transitive dependency resolution, and not all errors are logged inside workers. Making such issues tractable requires full visibility into version changes for all installed packages, including transitive dependencies, plus manual inspection of prebuilt containers. More logging and tooling around dependency management for custom containers would help, as would recreating a local environment that closely emulates the managed service, like the workflow described in Dataflow's GPU development documentation.
What’s next: finer-grained control and model-serving improvements
With Podcast Previews in production, the team is now watching pipeline behavior closely and tuning generation based on observed listener engagement. Two upcoming Dataflow capabilities are of particular interest.
Dataflow Prime Right Fitting would let Spotify set resource requirements per step or per ParDo, rather than applying one configuration to the whole pipeline. That means less expensive stages—like reading inputs or writing outputs—could run with fewer resources, improving overall utilization.
The RunInference API is another target. It would replace the current dense ParDo with smaller steps, letting model-heavy code run on machines dedicated to individual models over the job’s lifetime. That avoids frequent model swapping in memory and would also standardize inference metrics. Intelligent input batching could further push pipeline throughput.
Lessons from the build
Building a system to generate podcast previews for millions of users took time, but it produced clear takeaways. Managed pipeline infrastructure like Google Dataflow lowered the barrier for building and maintaining the system, letting the team concentrate on algorithms and pipeline logic rather than plumbing.
Streaming proved to be the stronger architecture, even though it required extra upfront effort. And the work doesn’t stop at launch—the team plans to keep improving both the data engineering and the data science behind the feature, with an eye toward better experiences for users and creators.
