Fleet Management at Spotify (Part 3): Fleet-wide Refactoring

Removing Repos as a Barrier to Change

Spotify’s backend code lives in thousands of Git repos, with each repo holding one or a small set of related components. That polyrepo layout grew from a desire to give teams strong ownership over their components and let them develop and deploy independently. But a large share of the code in those repos depends on shared frameworks and core libraries that live in other repos. Most backend services run on an internal service framework, and the bulk of batch data pipelines are built with Scio.

Every update or bugfix to those foundational libraries once meant a coordinated effort across hundreds of repos to bump versions — a process that could take six months or more to achieve even 70% adoption for a new backend framework release. As the number of components grew faster than the number of engineers, touching the whole fleet became increasingly costly.

In 2021, Spotify decided to invest in tooling to enable what they call "fleet-wide refactoring": making a change across every repo in the fleet at once, rather than updating components one by one. The goal was to shift the definition of "done" for a framework team from cutting a release to having that release adopted fleet-wide. Instead of asking other teams to upgrade by a deadline, the team owning the framework does the upgrade work themselves, even if that means sending pull requests to hundreds of repos they don't own.

To make that practical, two things had to change:

1. Dependency versions for most components needed to be managed centrally, so a single version bump could propagate across the fleet without editing each repo's config individually.
2. Engineers needed an easy way to send and merge pull requests across hundreds or thousands of repos.

Centralizing Dependency Management

Spotify had already adopted a Maven "Bill of Materials" (BOM) for its backend services and libraries — an artifact that declares which version of many libraries should be imported together. The Java BOM grew out of the pain of upgrading libraries like Guava, Google Cloud libraries, grpc-java, and internal frameworks that depend on each other. A single line bump in pom.xml could bring in a coordinated set of versions that were known to work together, and the BOM repo itself served as the vetting point for compatibility. Running a test suite against every proposed BOM change catches conflicts before they reach services.

The first major move in the fleet-wide refactoring effort was to stop treating the BOM as optional. Instead of asking backend teams to opt in, Spotify generated pull requests for every repo to onboard it automatically, with automated merges when tests passed.

That automation has paid off:

• 96% of production backend services now use the Java BOM.
• 80% are on a BOM version less than seven days old — a streak maintained for 18 months.
• A dependency version update is now a single PR to the BOM repo that rolls out to nearly all backend services within days, as long as the change is backward compatible.

Beyond the Backend

Backend services were the natural first target because of their volume. After that effort, Spotify repeated the pattern for batch data pipelines built on Scio: 97% of pipelines use an sbt plugin that manages library versions, and 70% of pipelines are, on average, on artifact versions less than 30 days old. Node-based web components and Python components are the next targets for the same "BOM-ification" treatment to bring central version control to those ecosystems.

Centralized dependency management has also had security and refactoring benefits. During the Log4j vulnerability disclosure in late 2021, 80% of backend services were patched within nine hours. And when most components run the same versions of common dependencies, writing a code refactoring that touches the entire fleet becomes far simpler — there are fewer version-specific code paths to handle.

One Image, Many Repos

Spotify’s fleet-wide code changes run through an internal tool called Fleetshift. What began as a single engineer’s hack project—a successor to assorted homegrown pull request generators—has since been standardized across the company, much like the Java BOM. A dedicated team owns Fleetshift and related refactoring tools, while other teams use those tools to push changes to their code. This separation matters as the number of automated fleet-wide changes grows; each team that ships a platform, library, SDK, or API is expected to manage its own fleet rollout rather than handing that work to a central refactoring group, which would quickly become a bottleneck.

The core abstraction in Fleetshift is a Docker image. An engineer who wants to change code across the fleet supplies an image that performs the refactoring using whatever tools they prefer — off-the-shelf utilities like sed and grep, or custom code. Fleetshift then clones each targeted repo, runs the image against the checked-out code, diffs the result, creates a Git commit, and opens a pull request back to the original repo. Shift authors specify the commit message and PR title and description in the shift configuration, so reviewers know what is changing and why. Progress is tracked on Backstage.

Fleetshift’s essential innovation is offloading the heavy lifting to Spotify’s existing Kubernetes footprint — thousands of nodes across dozens of clusters. Earlier generation tools ran every step on the engineer’s laptop, which made fleet-wide work prohibitive. Fleetshift instead executes transformations as Kubernetes Jobs, leveraging parallelism and autoscaling to make targeting ten or a thousand repos equally trivial. The jobs run on spare cycles of the same infrastructure that powers Spotify’s backend.

Because transformations are packaged as Docker images, Fleetshift imposes no limits on the kind of change you can make — as long as you can write code to make it. Current refactorings run the gamut from simple shell scripts to AST-based rewrites using OpenRewrite and Scalafix. Shifts can also be scheduled; recurring daily shifts handle continual updates like bumping the Java BOM version in every backend component or refreshing consumption of internal base Docker images.

Shifts are modeled as a custom Kubernetes resource, making Fleetshift itself an operator built on Spotify’s declarative infrastructure tooling. Every shift resource lives as a YAML file in a GitHub repo, subject to the same pull request and review workflow as any other code change.

apiVersion: fleetshift.spotify.com/v1
kind: Shift
metadata:
  name: update-foobar-sdk-to-v2
  namespace: fleetshift
spec:
  container:
    image: example-shift:1.2.3 
  pullRequest:
    title: update foobar-sdk to v2
    commitMessage: update foobar-sdk to v2
    description: |
      The PR updates the foobar-sdk from v1 to v2 and refactors code  calling now-removed methods in v2 to use their counterparts instead.

      You can see a full changelog [here](link).

      If you have any questions please reach out to #foobar-team-channel

  targetRepos: [ ... ]

Choosing target repos is flexible. Authors can use GitHub search for straightforward queries, or BigQuery for deeper analysis — Spotify ingests all source code from GitHub into BigQuery daily, enabling joins across datasets, such as finding repos with production data pipelines that call a particular method and depend on a specific library version. The repo list can be pinned in the shift resource itself, or pulled from a BigQuery table for recurring shifts whose targets drift over time.

To flatten the learning curve, the Fleetshift platform team maintains a library of reusable Docker images for common tasks — YAML mutation, bumping a Docker image to the latest release, and Java AST rewriting via OpenRewrite. A companion knowledge base documents how to approach various refactoring problems in different languages, so teams in need of a fleet-wide change can build on work already done elsewhere at Spotify.

Automerging: removing the review bottleneck

Generating hundreds or thousands of pull requests is only half the battle; getting them merged is the other half. Teams maintaining a few dozen repositories naturally prioritize their own work, meaning automated changes can sit unreviewed for weeks. The long tail of unmerged PRs becomes a significant drag on fleet-wide refactoring efforts. Relying on manual follow-ups to push changes through is simply not scalable.

To solve this, Spotify built a dedicated infrastructure service called automerger. While GitHub offers its own built-in automerge feature, building internally gives Spotify precise control over when and how changes are merged. The key design decision is inverting control: the author of the change, not the repository owner, decides whether a change should be automerged. The service also throttles the merge rate to avoid overwhelming CI/CD systems and only automerges during the owning team’s working hours, never on weekends or holidays.

Automerge is only allowed if all tests pass. If a repo’s test suite isn’t robust enough to catch a bad change, that’s a signal to the owner that more tests are needed. To help teams track readiness, Spotify created new Soundcheck programs and checks in its internal Backstage instance. These checks monitor code health, tracking whether components adopt the centralized Java BOM, have adequate integration tests, or use proper deployment health checks.

Automerge acts as both stick and carrot: platform teams can automate maintenance only where test coverage is sufficient. Seeing this, many teams have been motivated to improve their Soundcheck scores so that more of their code can be automatically managed.

Pre-merge testing alone isn’t enough for fleet-wide changes. Spotify built Firewatch, a service that monitors the health of components after an automerge. Firewatch consumes Pub/Sub events from backend deployment and data pipeline execution systems and correlates failed deployments or executions with the automerged commits that preceded them. If too many failures are associated with a set of automated changes, Firewatch alerts the change owners for investigation.

Rollout and adoption

The initial rollout was deliberately conservative, limited to Java BOM updates. Backend components were phased by criticality, with automerge enabled one phase at a time. This gradual approach built confidence in both the automerger’s safety and Firewatch’s monitoring capabilities, while driving BOM adoption to over 90%. Today, several dozen shifts run daily, covering everything from simple dependency bumps to complex refactorings.

Automerge was positioned as opt-out rather than opt-in from the start. Teams wishing to exclude a repo must document a reason and set an expiration date, encouraging them to fix underlying issues rather than permanently avoiding automation. Critical repos have additional mandatory checks before any automerge. Today, only about 50 of thousands of repos have an active opt-out, and many teams remove theirs before expiration once the underlying issue is resolved.

The response to automerge has surprised the platform team. Once engineers get past the initial apprehension of robots modifying their code without human approval, the prevailing sentiment isn’t reluctance — it’s wanting more. Teams frequently ask why certain changes aren’t automerged, and consuming teams often request that producers enable automerge for their updates. Most engineers don’t want to manually review a patch-release library bump or a Docker image version update.

Not every Fleetshift change is configured for automerge. The system is also used for “recommendation” PRs, such as suggesting better storage classes for Google Cloud Storage buckets or rightsizing Kubernetes pods. Presenting a recommendation as a ready-to-merge PR significantly lowers the barrier to action — the owning team just reviews the diff and merges.

Testing before opening pull requests

Fleetshift offers a “preview” mode for authors iterating on a transformation, allowing them to inspect diffs and logs across all target repositories before sending any PRs. But seeing the diff wasn’t enough; authors also wanted to know whether tests would pass before creating PR noise.

This led to Fleetsweep, a tool for testing a shift’s Docker image against a set of repositories. Running on the same Kubernetes-backed infrastructure as Fleetshift, it scales to large numbers of repos. Instead of creating a PR, Fleetsweep creates a short-lived branch with the automated change, triggers a CI build for that branch, and reports aggregate results back to the author. This allows engineers to inspect which repos would fail and review build logs — particularly useful for Java BOM upgrades where backward compatibility is uncertain.

Gradual rollouts for uncertain changes

Pre-merge tests don’t always catch problems. Batch data pipelines running on Google Dataflow are a case in point: test suites focus on business logic, making it difficult to reliably predict how dozens of runtime dependencies will interact in the cloud. Batch pipelines also run on schedules of hours or days, lengthening the feedback loop between merge and observable results — and a bad change might cause silent data corruption rather than visible failures.

To handle these cases, Fleetshift supports “gradual” rollouts. An author configures a shift to divide target repos into cohorts. The change is applied to each cohort only after it has been successfully applied to enough repos in the previous cohort. To determine cohort health, Fleetshift consults Firewatch, which supplies the maximum version for each cohort where a configurable percentage of previous deployments or executions succeeded. This feedback loop means yesterday’s success or failure informs today’s rollout decisions.

For example, batch data pipeline cohorts are tiered by how critical the produced data is — lower-value tiers receive automated changes first — and by the quality of pre-merge tests and post-merge monitoring. Similarly, the daily Java BOM update shift staggers backend repos so that the most critical systems only receive the new version after it has been successfully deployed across a majority of other components.

What Two Years of Fleet-wide Refactoring Look Like

Looking at the output from 2022, the scale of change is substantial. Fleetshift generated more than 270,000 pull requests, with 77% automerged and another 11% merged manually—figures that represent a 4–6x increase from the prior year. In total, the 241,000 merged PRs accounted for 4.2 million lines of code changed. Survey data backs this up: over 80% of Spotify engineers reported that Fleet Management has had a positive effect on code quality.

Those raw numbers matter less than what they enable. The time for a new release of the internal backend service framework to reach 70%+ of deployed services has fallen from roughly 200 days to under 7 days. For Scio and the internal SBT plugin that manages dependencies in batch pipelines, the comparable figure went from about 300 days down to around 30 days.

These are just two examples of a broader shift. Over the last 12 months, 49 distinct teams used Fleetshift to send PRs to other teams, with 27 of them doing so in the past 30 days alone. The median shift targeted 46 repos, and 22 shifts modified code in a thousand or more repositories.

Why Standardization Matters

Making Fleetshift the default tool for fleet-wide refactoring—with a dedicated full-time team behind it—has been key to getting teams to adopt a fleet-first mindset. Engineers no longer have to figure out which tooling they can use to change code outside their own remit. Instead, teams planning a new release or a breaking change now factor Fleetshift into the rollout from the start.

Where the Work Goes Next

The current state is encouraging, but there is still room to push much further. Lowering the learning curve for non-trivial code refactorings remains a priority, as does making it simpler for teams to get started with Fleetshift and to take on more ambitious API deprecations and code migrations. Searching for the right code to refactor is another bottleneck: writing SQL queries against BigQuery tables is not the most approachable workflow. And a significant portion of the codebase lives in monorepos such as the client and mobile codebases, where many teams share a single repository—a workflow Fleetshift does not yet handle well.

Acknowledgments

This work would not have been possible without the contributions of Andy Beane, Brian Lim, Carolina Almirola, Charlie Rudenstål, Dana Yang, Daniel Norberg, Daynesh Mangal, Diana Ring, Gianluca Brindisi, Henry Stanley, Hongxin Liang, Ilias Chatzidrosos, Jack Edmonds, Jonatan Dahl, Matt Brown, Meera Srinivasan, Niklas Gustavsson, Orcun Berkem, Sanjana Seetharam, Tommy Ulfsparre, and Yavor Paunov.

Apache Maven is a trademark of the Apache Software Foundation. Docker is a registered trademark of Docker, Inc. Kubernetes is a registered trademark of the Linux Foundation in the United States and other countries. Google Cloud Bigtable and Dataflow are trademarks of Google LLC.