A Decade of Listening: Engineering Spotify Wrapped 2019
Wrapped is one of Spotify’s biggest annual pushes, giving users a personalized recap of their listening habits. The 2019 edition carried extra weight: it covered the entire decade, drawing on data stretching back to before 2010. That ambition created a significant engineering challenge, since the underlying analytics had to be processed for more than 248 million monthly active users across ten years of history.
Wrapped is run by a dedicated team that spins up sub-teams for marketing, design, and engineering. For the data engineering side—the group responsible for the heavy processing—the core problem was scale. 2019’s campaign involved roughly five times as much data as 2018’s, which itself had pushed the limits of Google Cloud Platform’s Dataflow. Rather than repeating that approach, the team redesigned the pipeline to be more cost-efficient and easier to iterate on.
Decoupling the Data Stories
The campaign’s statistics—Top Artists, Top Songs, Top Podcasts, and similar summaries—were treated as discrete “data stories.” Each story was independently computable, which allowed the team to break the overall workload into separate jobs. The key architectural change was storing intermediate results in a way that reduced the need for user-level shuffles across jobs.
Spotify’s listening analytics, spanning years of history, live in a time-series data lake backed by Google Cloud Bigtable. That storage layer is optimized for aggregating over arbitrary time ranges, giving the team efficient access to data for each year of the decade as well as specific seasons of 2019.
Rather than running one massive job to compute all stories, separate Dataflow jobs handled individual stories. Their outputs were written to a shared Bigtable instance, with one row per user and each story in its own column family. That design meant data was already grouped and colocated at the user level, eliminating the expensive shuffle operations that had bottlenecked the 2018 pipeline.
This decoupled approach also made development faster. Jobs could run in parallel, and a bug or last-minute requirement change in one story didn’t force a full pipeline re-run. The only exception was the “top of the decade” metrics, which required combining all ten years of data. The team handled that by reading the already-aggregated yearly results from Bigtable—a much cheaper operation than reprocessing the raw history.
QA and Iteration
To validate results quickly, the team built a small Python library that used Cloud Bigtable APIs to pull data directly from both intermediate and final storage. That gave engineers a straightforward way to spot-check numbers and catch bugs before launch, without having to interact with the full pipeline.
Cost and Velocity Payoffs
The design choices paid off in measurable ways. Spotify processed roughly five times the data of the 2018 campaign while spending 25 percent less on processing. The savings came from reducing group-by operations and reusing the per-year aggregations for decade-level statistics.
Breaking the work into smaller, independent data stories also created a more flexible system. Iteration was faster, and changes could be isolated to a single workflow without destabilizing the rest of the campaign. The team credited this modular structure as a major factor in meeting the ambitious scope of a decade-long Wrapped.
