Spotify’s Listening Together: Turning Live Streams into a Global Visual
When Spotify launched its Listening Together campaign in April, the goal was to visualize a simple but powerful idea: at any given second, tens of thousands of people worldwide are pressing play on the same track. The site, which has since logged over a million visits, pairs simultaneous plays in two different locations, turning abstract streaming data into a shared, real-time experience.
Building that experience meant solving a series of engineering challenges, starting with how to capture and process a firehose of global listening events without choking on the volume.
From Play Events to Processable Data
Every play on Spotify generates a message published through the company’s asynchronous messaging backbone, Google Cloud Pub/Sub. For this project, the engineering team tapped into that event stream with authorized access, needing a framework capable of handling the scale of incoming data in real time.
They chose Apache Beam, the open-source stream processing model, running on Google’s Dataflow backend. On top of Beam sits Scio, Spotify’s open-source Scala API, which the company now uses as its default framework for both batch and stream processing. Scio’s functional approach, combined with Dataflow’s managed scalability, gave the team the programming model and throughput needed to handle the incoming plays.
Collating and Cleaning the Stream
The pipeline begins by consuming the Pub/Sub play events one by one. Using Dataflow’s windowing features, the team collates all plays arriving within a short time slice into a single group, treating them as if they occurred simultaneously. From that batch, the pipeline extracts each track’s identifier and the listener’s IP address, then groups the data by track. The result is a set of unique tracks, each paired with a list of IP addresses from its listeners.
Not all plays make the cut. Individual user plays and IP addresses routed through VPNs or anonymizing services are filtered out, since their geographic locations don’t match the listener’s physical position. And because 30,000 simultaneous plays per second would overwhelm the visual, the pipeline samples a fraction of the total events for processing.
For the plays that remain, MaxMind databases handle IP-to-city and IP-to-country conversion, and also provide approximate coordinates used to project locations on the website. The pipeline handles localization of city names as well. Once processed, each Listening Together session is published to a fresh Pub/Sub topic, ready for consumption.
Serving Sessions to the Globe
To deliver those sessions to site visitors, the team built a backend service on Spotify’s Apollo framework. That service subscribes to the new Pub/Sub topic, pulling down Listening Together sessions as they become available. It runs in autoscaled Kubernetes clusters spread across geographically separate data centers.
The distributed setup introduces a minor timing discrepancy: because the service instances are physically far apart, visitors in different regions may briefly see different plays in the same second. The team acknowledges that global synchronization is inherently challenging and accepts the slight deviation from the goal of showing everyone the same play at the same time.
The website fetches the latest session data through a simple endpoint. Since all the heavy computation happens upstream in the streaming pipeline, the endpoint only returns data already held in memory, allowing it to field a large number of requests with minimal compute. The entire journey, from a user pressing play to that moment appearing on the Listening Together globe, typically takes just a few seconds.
