Fleet Management at Spotify (Part 1): Spotify’s Shift to a Fleet-First Mindset

From Squad-First to Fleet-First

Spotify’s architecture has long favored many small, independently owned components — mobile features, data pipelines, services, and websites — each designed, built, and operated by its own team. That model enables rapid innovation, but it creates a second-order problem: when you run thousands of components in production, even trivial maintenance tasks become a slow, manual grind.

A single Java runtime upgrade, for instance, once took eight months and roughly 2,000 semiautomated pull requests. Larger migrations — Python 2 to 3, or expanding into new cloud regions — involved hundreds of teams over years. Security and reliability fixes required urgent, company-wide coordination. And for the engineers doing the work, it was repetitive toil, the kind of maintenance that drains motivation.

The alternative Spotify has pursued is what it calls a fleet-first mindset: instead of ten large upgrades a year, run thousands of small, automated ones across the entire codebase. Rather than updating components one by one, apply changes fleet-wide. This shift is as much cultural as technical. Infrastructure teams now make changes to code owned by others, and component owners receive changes merged and deployed without being in the loop beforehand.

Four Questions for Fleet-Wide Change

Safely automating changes across thousands of components and roughly 60 million lines of production code (out of over 1 billion lines in source control) requires answers to four questions.

1. What code are we changing?

Spotify needed precise targeting. Basic code search existed, and all code and configuration is ingested into Google’s BigQuery, alongside production infrastructure data on library dependencies, deployments, containers, and security vulnerabilities. That combination enables flexible, fine-grained querying for change targeting. Semantic indexing of the codebase is under exploration to identify API call sites more precisely.

2. Is everything under version control?

Code and configuration were already in Git, but many cloud resources — storage buckets, service accounts, database instances — were not, because Spotify was only partway through a migration back to fully declarative infrastructure. That work had to be finished before fleet-wide automation could proceed.

3. How do we make the changes?

The mechanism must identify where a change applies (for example, matching a code pattern in a repository), apply the edit via automated refactoring, and verify it with a CI build. It then orchestrates merging and deployment, with monitoring to abort if anything fails. Automated refactoring at this scale is complicated; Hyrum’s law holds that every observable behavior of a system will be depended on by someone, no matter what the contract promises.

Spotify runs a polyrepo layout with thousands of GitHub repositories, though the tooling should apply to a monorepo should that change. The largest repositories already approach 1 million lines of code, where specialized fleet-wide tooling becomes necessary.

4. How can we trust changes nobody reviews?

Completing a fleet-wide change in hours or days without requiring hundreds of developers means changes must be automatically verified, merged, and deployed. Spotify leaned on its test automation, but needed better coverage in some components and, in some cases, container-based integration testing. Many components lacked automated canary-based deployment testing. Manual staging is the exception, limited to partner end-to-end verification or compliance requirements.

To ensure the main branch was always deployable, Spotify implemented a regular rebuild and redeployment of every component at least once per week, reducing the risk of code rot causing build or deployment failures.

Results So Far

Fleet Management now covers more than 80% of production components, especially data pipelines and backend services. Over three years, Spotify completed over 100 automated migrations and updates external and internal library dependencies daily. The automation has authored and merged more than 300,000 changes — roughly 7,500 per week — with 75% automerged.

Developer toil has dropped by an order of magnitude of hundreds of developer years, according to Spotify’s estimates. Sentiment analysis shows more than 95% of developers believe Fleet Management has improved software quality.

The fleet is also healthier. Components stay current with internal and external libraries and frameworks, reducing known security vulnerabilities and reliability incidents. When Log4Shell hit, Spotify deployed a fix to 80% of production backend services within 9 hours; the remaining, then-unmanaged services took a few days.

New framework features reach developers far faster. A new version of the internal service framework once took about 200 days to reach 70% of backend services through organic updates; now that number is under 7 days.

More than 75% of the production environment is repaved — rebuilt and redeployed from source — weekly, mitigating security vulnerabilities and keeping components in sync with Git.

What’s Next

Adoption continues for the long tail of components that don’t yet follow Spotify’s technology standards. Onboarding teams to a fleet-first model, where changes roll out automatically and safely, will remain a focus until it becomes the default.

Spotify aims to take on more complex fleet-wide changes, building on experience with library and container dependencies and simpler refactoring. The goal is to increase confidence in automatic merging and deployment for increasingly intricate migrations.

Standardization is a major lever: the more similar components are — same frameworks, same code patterns — the easier fleet-wide changes become. Spotify doubled the share of components fully using its standard technology stacks over the past year and plans to expand supported stacks and drive adoption further.

Finally, Fleet Management tooling itself will be simplified, and the platform abstraction level exposed to developers will continue to rise. The long-term aim: every developer at Spotify can safely perform a fleet-wide change with ease.

What Fleet-First Changes in Practice

Shifting to a fleet-first mindset isn’t just about changing tooling—it changes how engineering teams make decisions. Instead of treating each service as an isolated project with its own bespoke setup, teams start with the assumption that their service will share patterns, interfaces, and operational practices with the rest of the fleet. This drives a number of concrete changes in day-to-day work.

One of the most visible shifts is in how new services get created. In a fleet-first model, the default path isn’t a blank slate where a team hand-crafts configuration. It’s a standardized template that encodes the organization’s accumulated operational knowledge. When every service starts from the same base, the differences that remain are meaningful ones—business logic and actual variance, rather than incidental configuration drift.

This also transforms ownership and collaboration. When a team runs a service, they’re not just running their own code; they’re operating a member of the fleet. That means the abstractions, APIs, and data contracts exposed by a service matter beyond its immediate consumers. A change that looks local from one team’s perspective can have fleet-wide implications, which pushes teams to design with a broader set of stakeholders in mind.

Adopting a fleet-first mindset also changes how infrastructure teams work. Rather than responding to tickets about individual services, infrastructure engineers can focus on improving the shared platform and the default patterns that all teams use. Improvements to the platform have a compounding effect—a small gain in efficiency or reliability multiplied across the entire fleet becomes a substantial win.

This approach is only as good as the data that backs it. Making fleet-level decisions requires visibility into how services actually behave in production—not anecdotal reports from individual teams. When the default patterns are consistent, it’s far easier to compare performance, spot outliers, and identify which practices are working. Anomalies become signals of a real problem rather than noise from a non-standard setup.

Underlying all of this is a shift in how risk is perceived. In a service-by-service model, a team owns the full burden of its operational decisions. In a fleet-first model, that burden is shared by the platform and the standards it enforces. Teams still make judgment calls, but they do so inside a framework that has already proven itself at scale—making it easier to onboard new engineers, retain knowledge, and keep the overall system healthy as the company grows.

The motivation for this shift at Spotify is rooted in a familiar set of operational challenges. As the number of services grew, the cost of maintaining bespoke implementations multiplied. Standardizing on fleet-level patterns wasn’t a matter of ideology; it was a response to the inefficiencies that scale exposes. The better the platform supports the fleet, the more time engineers spend on the work that differentiates their services.

That said, a fleet-first mindset is a foundation, not a final state. The work of evaluating what patterns deserve to be defaults needs to be ongoing, and the platform needs to evolve along with the teams that use it. The goal is to reach a point where the right thing to do for an individual service and the right thing to do for the fleet are one and the same—and when that becomes true, infrastructure stops being a tax on innovation and starts being an accelerator.

Once this mindset is in place, the next step is figuring out how to put it into practice with declarative infrastructure and the other supporting changes that make fleet management possible at scale. Those implementation details will be covered in future installments of this series.