Scaling the Collection Layer

Spotify’s data collection platform ingests over 1 trillion events per day. That scale didn't arrive overnight; the event delivery architecture has gone through multiple iterations since its inception and subsequent migration to the cloud. Collection exists to serve three core purposes: understanding content relevance for users, responding directly to feedback, and gaining deeper insight into user interactions to improve the experience.

Figure 1: The event delivery infrastructure is a significant topic that deserves its own dedicated article (coming soon). Nevertheless, here’s an overview of the main components handled by our event delivery infrastructure.

Teams instrumenting new functionality only need to write code with the SDKs and define event schemas. The underlying infrastructure handles the rest, automatically deploying event-specific components — PubSub queues, anonymization pipelines, and streaming jobs — via Kubernetes operators. Schema changes trigger corresponding resource updates automatically. The architecture balances centralized control with distributed ownership, so most updates can be managed by the consumers of the data without involving the platform team. More than 1,800 distinct event types are currently published.

Pipeline Management and Data Integrity

Data Processing is about making data usable; Data Management is about keeping it trustworthy. With over 38,000 actively scheduled pipelines running hourly or daily tasks, scalability is the primary constraint. These functions must maintain lineage for traceability, metadata for searchability, and access controls and retention policies to manage storage costs and regulatory compliance.

Figure 2: These domains, like Event Delivery, warrant their own comprehensive blog posts. This article provides a closer look at the tools we use, and our organizational structure.

Scheduling and orchestration are the backbone of processing. Once scheduled, workflows execute on BigQuery, Flink, or Dataflow clusters. Most pipelines are written with Scio, a Scala API for Beam. Each pipeline produces data endpoints with defined schemas, optional partitions, retention policies, access controls, lineage, and quality checks. Defining a workflow or endpoint relies on custom Kubernetes operators, allowing complex structures to be deployed and maintained easily; the resource definition lives alongside the pipeline code, owned by the codeowners. Monitoring covers data lateness, failing workflows, and endpoint health, while Backstage integration provides resource management, cost analysis, and quality assurance in one place.

Driving Adoption

A data platform only delivers value if people actually use it. The platform must be flexible enough for many use cases while staying cost-effective, easy to onboard to, and reliable enough for business-critical logic. It also needs smooth upgrade paths — nobody wants disruptive breaking changes.

Fostering engagement comes down to a few practical tactics:

  • Documentation that's easy to find. It should be quicker to locate documentation than to wait for an answer in a chat.
  • Onboarding teams directly. Embedding with users reveals real use cases, exposes friction in different environments, and brings concrete learnings back to the platform.
  • Automating change. People rarely want to do manual migrations. Automating toil and handling the associated risks with dedicated support time makes platform evolution painless.
  • Building a community. A dedicated support channel with assigned "goalies" — rather than relying on goodwill — ensures questions get answered seriously. At Spotify, that's the #data-support Slack channel.

Evolution, Not Prescription

The platform's trajectory speaks to the scale of the effort. What began as a small team running what was the largest Hadoop cluster in Europe is now 100+ engineers building the data platform on GCP, spanning collection, management, and processing. There is no formula for constructing a data platform. The practical starting point is aligning investments with organizational needs — those needs become the drivers for the platform's building blocks. Clear goals and expectations secure organizational support and set the stage for success. Proximity to users and a direct feedback loop will naturally foster a community. Starting small is acceptable; the important thing is to iterate and evolve from there.