Changing the Wheels on a Moving Bus — Spotify’s Event Delivery Migration

Why Spotify Rebuilt Its Event Pipeline Mid-Flight

At Spotify, every surface running Spotify code emits log data — listening history, A/B test results, page load times — through a system the company calls the Event Delivery Infrastructure (EDI). The EDI serves internal users (engineers, data scientists, product managers, and squads) rather than the end users who consume the audio platform. In 2016, alongside Spotify’s migration to Google Cloud Platform (GCP), the EDI was redesigned from the ground up. Peak traffic has since grown from 1.5M events per second to nearly 8M, with daily ingested volume reaching roughly 70TB.

That growth exposed real bottlenecks. As adoption climbed, internal users hit feature gaps and data quality issues began degrading productivity across Spotify’s data community. Three years of operating the system revealed where the original design assumptions no longer held.

Where the Old System Broke Down

The original EDI was built around a clear mission: reliably collect data and make it available safely and efficiently. Core use cases — music streaming, application monitoring — were well served. But as new use cases emerged, the system’s foundational choices became liabilities in three specific areas.

Data Loss Was Built In

Most events from mobile clients were sent fire-and-forget. That may sound surprising, but offline listening creates complications around deduplication when data is re-sent. If a data point appears missing, it’s unclear whether it was genuinely lost or simply delayed because the user was offline, in a tunnel, or on a flaky connection. The result was a small percentage of loss across nearly all collected data — unacceptable for certain event types. The problem compounded for datasets assembled from multiple event types to reconstruct user journeys, where a single lost event could invalidate an entire journey. Some client-side algorithms could deliver business-critical data exactly once, but the approach never scaled to all 600+ event types in use.

The Control Plane Was Painfully Slow

Getting from instrumentation to insights took roughly a week under normal conditions. A major culprit: multiple EDI components needed to be schema-aware. The receiver service — the infrastructure’s entry point — validates incoming data against schemas, but due to accumulated tech debt, schema propagation took hours. That friction drove some teams to abandon instrumentation efforts entirely; others shoehorned data into existing event types, creating gaps in coverage and a data-quality nightmare.

Backward Compatibility Became a Trap

In 2016, speed of migration to GCP was strategic, and staying backward compatible minimized migration time. But that choice forced the team to retain historical design decisions they would never have made from scratch:

• Tab-separated values (TSV): All data events were sent as TSV strings, with schemas parsed and converted to Avro using a Python library from 2007. This TSV tooling was the root cause of the painful control plane experience.
• Stateful services: Events were written to disk before forwarding to the EDI, providing crash resilience but making the system vulnerable to data loss when a machine went down. It also blocked auto-scaling and Kubernetes adoption without difficult workarounds.
• Legacy perimeter: Because events were forwarded from disk, client-triggered events had to originate from perimeter servers that stored events on disk and were tightly coupled to legacy logging mechanisms. This added complexity and created alignment problems across teams with different goals.

Migrating With a Long Tail

The migration challenge was compounded by mobile app update cycles: a new version takes months to reach a high percentage of users, meaning old and new EDI traffic would coexist for a significant period. Embedded devices — TVs, speakers — posed an even harder problem, since some would likely never be upgraded at all. Spotify calls this the “Long Tail Problem.”

Their partial solution: a data transformation pipeline that reads events from legacy clients, converts them, and feeds them from the legacy EDI into the new infrastructure. Breaking backward compatibility was deliberate — it gave the team license to update the data model. The transformation would inevitably miss or corrupt some fields, but since it applied only to legacy clients, that impact would shrink as users upgraded and eventually become negligible.

Data producers were given two onboarding paths: redesign instrumentation around the new data model, or keep existing instrumentation and export data from the legacy EDI to the new one. Consumers then migrated to read from the new EDI. Producers and consumers agreeing on the exporter route needed to update downstream pipelines before making client-side changes.

Validating With Real Traffic, Early

Early adopters were essential to validate design decisions. The team presented the alpha product’s advantages and limitations to interested teams, making clear that experimentation, breakage, and fast fixes were part of the deal — without risking production systems or critical data. Setting those expectations allowed for breaking changes when assumptions proved wrong.

The team then sought at least one real use case for migration, deliberately choosing one that matched the limited features built so far. Understanding how event owners used their data helped prioritize the roadmap: each added feature unlocked more event types for onboarding. Design decisions and assumptions were revisited periodically to surface problems early.

Once a prototype ran on real production traffic, the team solidified interfaces and the data model, then helped alpha users adapt. That stable abstraction decoupled the work of migrating 600+ legacy event types from the work of building the new EDI itself.

Optimizing Only When Metrics Justified It

Premature optimization, without motivating metrics, is generally a mistake. The team had to balance efficiency against the features needed to meet migration goals. A key lesson from the first EDI: design for target service availability from the start. Transactional collection wasn’t required, so 100% delivery wasn’t the goal. The real question was determining what availability level was acceptable — and understanding the trade-offs that came with it.

The New Control and Data Planes

Spotify's Event Delivery Infrastructure (EDI) separates into two planes. The control plane is where internal users declare their events, design schemas, and bind them to SDKs. The data plane receives those events, routes them by event type, and exposes them as batch datasets or streaming topics. Between ingestion and output, events pass through deduplication, translation to a shared data model, and pseudonymization before reliable storage for downstream consumers.

The gap between legacy operations and current needs forced a redesign. Growing internal adoption, rising operational overhead, and scalability limits in the original components drove the shift to a new infrastructure.

Client Resends and a Wider Deduplication Window

To reduce event loss, the team introduced client resends. These retries can happen instantly, minutes later, or days later — or never, if a user drops their phone in the ocean mid-session. Because transport failures are indistinguishable from client abandonment, the combination of retries and unstable connections naturally produces duplicate events.

The legacy system only deduplicated within a window of hours. As duplicate volume grew, that approach became a bottleneck. The new deduplication job introduces event message identifiers to generate lookup indices and remove duplicates, and it replaces Hadoop with Google's Dataflow. The new design can look back across multiple weeks instead of hours.

From Offline Files to an Online Receiver

Legacy EDI stored events as files on disk before forwarding them to a receiver service. That design leaned on the availability guarantees of Spotify's access points and other backends, reading from disk at some later point. The new receiver service is a highly available API that SDKs call directly, and it carries its own availability guarantees — a paradigm shift for the SRE team. The old file-based approach also blocked auto-scaling fleets, which the new design can leverage.

If the new receiver service goes down, events buffer temporarily on clients and resend according to a predefined retry policy.

Dataflow Over Persistent Clusters

A driving principle for the rebuild was using cloud-managed services wherever possible, offloading management to Google and letting the team focus on product work.

The legacy Hadoop deduplication job had been migrated from on-premises to Google's managed Hadoop, Dataproc, as the path of least resistance. The new job rewrites that work in Scio, the Scala API for Apache Beam, running on Google Dataflow. Spark and Flink were considered, but both would run atop Hadoop, conflicting with the goal of reducing operational burden and cost.

Dataflow eliminates the need for long-lived Hadoop clusters sized for peak loads. Those clusters were large enough to handle the heaviest job without issues, which made them wildly oversized for everything else and expensive to maintain. Dataflow spins up clusters per job and auto-scales, so resources are used — and paid for — only when needed.

Migration Strategy

The redesign followed a phased path. New components were assembled into a proof of concept, which grew quickly into a production-quality prototype. Shipping early validated the end-to-end infrastructure and surfaced issues fast. Onboarding a handful of internal users early kept quality and operational maturity high.

With the prototype's interfaces solid, the team scaled traffic by onboarding noncritical event types in bulk. Stable interfaces meant the internals could be improved or replaced without friction. That decoupling let the mass migration proceed independently of the rebuild, trimming wall-clock time for the whole project.

By the end, nearly all obsolete legacy infrastructure had been retired and replaced with modern components. The migration demonstrated that a data platform can be fundamentally rebuilt while it remains in active service.