Orchestration at Spotify: Why a Library Duo Had to Go

Spotify’s batch data platform is large by any measure: roughly 20,000 pipelines daily, defined across more than 1,000 repositories, and owned by over 300 teams. Most of those pipelines have historically run on two in-house libraries — Luigi for Python users and Flo for Java users. In 2019, the data orchestration team decided that this foundation had to change.

The Old Model: Libraries Deployed as Black Boxes

With Luigi and Flo, engineers define workflows as recursively dependent tasks. All task code, libraries, and logic are packaged into a Docker image, which is deployed to Styx along with an entry point to the root task. Styx handles scheduling, starting the image in a GKE cluster at the designated time. At runtime, the graph of dependencies is discovered and executed in order; instrumented workflows can emit lineage data, and tasks may launch Dataflow or BigQuery jobs and publish results to various locations.

As a platform team, however, this library-centric design created three structural problems:

  • Maintenance burden: Two separate libraries offering equivalent capabilities had to be implemented, maintained, and updated in parallel.
  • Support burden: Pushing updates across the fleet was difficult because users don’t always upgrade. With more than 1,000 repositories and growing, rolling out fixes and instrumentation meant tracking fragmented versions, which hurt both platform efficiency and data engineer productivity.
  • Lack of insights: From the platform’s perspective, the workflow container is a black box. Insights into task graphs and blocking upstreams are only available after execution, forcing a portion of GKE cluster time to be spent polling for missing dependencies.

Priorities for the Next Generation

The team identified two goals as most important for a replacement system:

  • Frictionless upgrades: Push as much foundational functionality as possible down to the platform teams. This would let features and fixes reach all users automatically, minimizing the maintenance burden on data engineers and letting them focus on their core domain.
  • Platform automation: Enable features like opt-in downstream backfill triggering, event-triggered actions, canceling workflow execution after user-defined alerts, and notifying downstream consumers of failures or delays. These all require visibility into tasks and workflows — something the Docker-image model with Luigi/Flo does not provide.

The vision that emerged was to move away from libraries entirely and build orchestration logic into a managed service:

  • Orchestration of tasks within a workflow is performed by a managed service.
  • Business logic (task code) runs in an environment the platform controls.
  • Users define workflows and tasks using SDKs distributed by platform teams.

How the Choice Was Made

The evaluation criteria covered migration effort for existing workflows, development time and effort (in-house versus ready-made products), maintenance burden (Google-managed versus self-managed, for instance), extensibility, workflow inspectability, and support for multiple languages.

The Move to Flyte

After comparing the state of the art in orchestration, Spotify settled on Flyte. Key draws were its extensibility for integrating Spotify tooling, multi-language support, and scalability. The project was actively developed with strong support from the Flyte team, and the roadmap included an open source release — which made it a good fit for Spotify’s open source contribution ethos.

Several aspects of Flyte’s design aligned well with the platform team’s requirements:

  • Its entity model and nomenclature are similar to Luigi and Flo, which simplifies user migration.
  • Tasks are a first-class citizen, making them easy to share and reuse.
  • The client SDK is thin, with logic pushed to the backend — a much simpler maintenance story than two libraries (Python and Java) holding all the logic.
  • It decouples orchestration from the rest of Spotify’s ecosystem, allowing mix-and-match in the service layer.
  • The backend is extensible to support different task types, and the SDK model can grow to support additional languages.
  • It is an existing platform product, so Spotify doesn’t have to develop the service from scratch.
  • It is battle-hardened and scalable, already running under constraints similar to Spotify’s.
  • As an actively developed open source project, it offers a path for Spotify to contribute back to the community.

Integration into the Spotify Ecosystem

With Flyte, workflow actions are visible to the platform team. Workflows can be registered both in Flyte Admin and in Styx, Spotify’s existing scheduler, which keeps the integration smooth. The platform team gains the visibility needed to build the automation features users have requested.

Status and Next Steps

The Flyte migration at Spotify is still underway. Critical pipelines are already running on Flyte, but all existing Luigi pipelines still need to be migrated — a process the team expects to cover in a follow-up post.

The team acknowledges the Union team behind Flyte and the platform and financial engineers at Spotify who helped refine the early implementation.