Turning Annotation from a Bottleneck into a Platform

As ML and GenAI applications expand across Spotify’s catalog—spanning hundreds of millions of tracks and podcast episodes—two core machine-learning needs drive the demand for data: detecting content relations (such as placing a new release on the right Artist Page) and identifying policy violations in audio, video, and metadata. Each of these use cases requires models that cover entire catalogs, which in turn requires large volumes of high-quality training and evaluation annotations. The engineering challenge was not just collecting those annotations, but doing so at scale without letting manual processes become the bottleneck.

The path started with a straightforward ML classification project. The team dissected the end-to-end workflow, isolated the manual steps, and scripted them: sampling predictions for review, serving data for operator review, and feeding the results back into model training and evaluation. That experiment increased the annotation corpus by 10 times while improving annotator productivity by three times. After confirming that the approach translated to other ML tasks, the decision was made to invest seriously in a shared platform. The strategy centered on three pillars: scaling human expertise, building annotation tooling, and establishing foundational infrastructure.

Scaling Human Expertise and Machine Assistance

At the operational level, Spotify established large-scale expert workforces across multiple domains, organized into tiers:

  • Core annotator workforces: Domain experts who perform the first-pass review of all annotation cases.
  • Quality analysts: Top-level experts who resolve ambiguous or complex cases escalated by core annotators.
  • Project managers: The bridge between engineering/product teams and the workforce, responsible for training materials and feedback on data collection strategies.

Running in parallel with human review is a configurable, LLM-based system. This machine layer allows the team to expand the high-quality annotation corpus significantly with comparatively low effort and cost, freeing human experts to focus on cases that genuinely need their judgment.

From Simple Tools to Flexible Interfaces

Although the initial use case was a simple classification task—essentially answering a yes/no question—the platform soon had to accommodate more complex work such as annotating audio/video segments and natural-language processing tasks. That led to building custom interfaces that could quickly spin up new projects without starting from scratch each time.

Behind those interfaces, the team invested in project management and access-control tooling to handle the distribution of annotations across experts. These tools enable dozens of parallel annotation projects while keeping experts productive across multiple efforts. Metrics such as project completion rate, data volumes, and annotations per annotator help project managers and ML engineers track every project.

The team also analyzed the annotation data itself. In nuanced cases—like detecting music overlaid in podcast audio—different experts naturally arrive at different answers. To handle that, they introduced an agreement metric. Data points that lack clear consensus are automatically escalated to quality analysts for final resolution, ensuring that model training data consistently contains the highest-confidence annotations.

Abstractions That Fit Various Workflows

No single tool serves all of Spotify’s needs at scale. The platform’s integrations are built on abstract data models, APIs, and interfaces that are generic enough to work across multiple annotation-tool vendors, letting project teams pick the right tool for the use case rather than being forced into a one-size-fits-all solution.

The platform also provides bindings that connect annotation workflows directly to the ML development lifecycle. For new or ad hoc ML projects, CLIs and UIs are available for quick setup. For production-grade workflows, the binding connects to internal batch orchestration and workflow systems, enabling annotation flows that run reliably at a larger scale.

The Payoff of a Parallel Investment

By democratizing access to high-quality annotations, the platform has dramatically shortened the time it takes to develop new ML models and iterate on existing systems. The investment logic was deliberately twofold: scaling the expert workforce without scaling the technical machinery to support them would create its own bottleneck, while scaling purely technical capabilities would have missed the value that experienced domain experts bring—especially in deciding edge cases.

Moving from ad hoc one-off projects to a full platform was a major undertaking, and the effort continues to evolve. The platform remains a work-stream in active development, adapting as new ML and GenAI requirements appear.