Fleet Management at Spotify (Part 2): The Path to Declarative Infrastructure

The shift away from host-centric config

Spotify’s move from on-premise data centers to Google Cloud Platform (GCP) started as a “lift and shift,” preserving existing service architectures and their infrastructure relationships. That approach had a side effect: infrastructure choices became a long tail of snapshots capturing whatever was considered best practice at the time—self-hosted Cassandra clusters, Kafka, Elasticsearch, PostgreSQL, or Memcached on dedicated VMs—alongside the mistakes made along the way.

The problem compounded as the company grew. Engineering headcount rose steadily, but the amount of software and infrastructure grew exponentially relative to developer count. Teams commonly found themselves owning dozens to hundreds of codebases, and acquisitions and reorganizations frequently transferred codebases to teams that no longer understood the architectural or infrastructure decisions behind them. What was missing was a mechanism to bring existing infrastructure up to current standards and reduce fragmentation enough for the platform organization to support the full footprint.

In the on-premise world, where machines were treated as precious pets, host-centric configuration management tools like Puppet and Ansible worked. That model does not hold up in the cloud:

• Cloud-managed alternatives to self-hosted infrastructure are generally preferred.
• Much configuration—IAM, network and firewall rules, relationships between subscriptions, backups, cross-region replication—does not map cleanly to any single service or VM.
• There are no long-running compute instances to support slow, steady host-oriented reconciliation. VMs and Kubernetes Pods can be rescheduled at any time, and the long bootstrap process for Puppet-managed VMs was unacceptable for shifting data center traffic patterns.

Defining the requirements

Spotify needed a new solution for controlling all infrastructure while preserving the ability to apply company-wide policies and guide users toward best practices. Several constraints emerged:

1. GitOps workflow: infrastructure configuration checked in with source code, peer-reviewed, with an audit trail.
2. Runtime introspection: the ability to enumerate all resources, flag policy violations, and identify owning teams. A break-glass mechanism was also needed, allowing SRE teams to alter resource configuration immediately without going through GitOps.
3. Configuration as data: to support automated changes, configuration had to be JSON or YAML rather than code (Starlark, HCL, TypeScript). Changing code to produce desired infrastructure output is computationally hard—the Halting Problem—while changing configuration-as-data is trivial.

Any solution also had to support importing hundreds of thousands of existing cloud resources without affecting their current state. Modeling raw cloud resources directly was seen as a starting point, but the goal was to introduce bespoke custom resources that bundle and compose lower-level resources, eventually eliminating raw cloud resource declarations entirely. Common solutions fell short—Terraform, for instance, fails requirements 2 and 3.

The bet was on Kubernetes for modeling infrastructure resources, which met all requirements. The resulting platform is called “declarative infrastructure.”

Kubernetes as the control plane

Each kind of infrastructure resource—raw cloud primitives and bespoke custom resources alike—is modeled with a custom resource definition (CRD). Operators continually reconcile the declared state against the real world: GCP, the managed cache system Locus (which declaratively deploys Memcached instances), and the Flyte system, among others.

The declarative infrastructure platform runs on dedicated Kubernetes clusters, isolating its API-server-intensive operators from the workload clusters hosting stateless Spotify services. The platform currently manages 3,000-plus GCP projects and approximately 50,000 GCP resources. There are around 20 internally built operators.

Manifests are ingested from source code repositories through CI/CD. A declarative infrastructure build step—a Docker image—performs light transformations and validation on the manifests using kpt, then applies them to the relevant Kubernetes cluster. Each source repository uses a dedicated service account, and each GCP project maps one-to-one to a Kubernetes namespace, limiting repositories to managing resources in explicitly granted GCP projects.

Users generate Kubernetes resource manifests in several ways: importing existing cloud resources through a Backstage plugin that queries the cloud account and generates a pull request with Config Connector manifests; using a Backstage plugin that asks questions and generates best-practice infrastructure manifests; or writing YAML with an IDE plugin offering autocomplete.

During the build, manifests are applied and the declarative infrastructure step waits for Kubernetes resources to reconcile—for example, waiting for a managed cache instance to be created—surfacing success or failure in build logs and pull requests. Review builds perform light validation and a dry run against the cluster, showing users which resources would be added, updated, or removed.

Operators react to all resource changes, handling creation, updates, and deletion of the managed infrastructure when the corresponding Kubernetes resource is deleted.

Default permissions are deliberately limited. Users can only get resources via RBAC; updates and deletes go through a break-glass mechanism that lets owning teams impersonate a per-namespace break-glass account, bypassing the full code review and build process for emergency changes.

Abstractions on top of primitives

The Kubernetes-based approach makes the platform extensible. Infrastructure developers build custom operators that encapsulate high-level abstractions—managed databases, managed caches, data jobs—rather than exposing raw cloud resources.

A managed cache resource, for example, exposes runtime insights through its status field:

 apiVersion: caching.spotify.com/v1alpha2
 kind: Locus
 metadata:
   name: my-locus-instance
   namespace: my-gcp-project
 spec:
   numShards: 6
   podSpec:
     cpu: 10
     memorySizeGb: 16
   regions:
   - europe-west1
   - us-central1
   - asia-east1
   - us-east1
   - europe-west4
 status:
   conditions:
   - lastTransitionTime: "2023-03-17T15:13:46.690585664Z"
     message: this resource is up to date
     observedGeneration: 1
     reason: UpToDate
     status: "True"
     type: Ready
   observedGeneration: 1
   ready: "True"

Another example: a data endpoint resource that defines a dataset can create and manage the lifecycle of the underlying cloud storage resource. Teams creating Flyte projects through CRDs also get Google service accounts created, role bindings attached, and workload identity fixed for multiple namespaces—automation that eliminates a great deal of script-based complexity and enables new platform-level automation.

Extensibility also comes through kpt functions and gatekeeper validating or mutating webhook configurations. Gatekeeper has an advantage over kpt: its logic applies regardless of how resources are applied to the cluster, while kpt functions run only client-side in the declarative infrastructure action container and are easy to bypass.

What comes next

Future work targets two audiences. For end users, the goal is to make creating cloud resources easy without crafting YAML or importing existing resources, and to integrate more tightly with Backstage so runtime resource state is visible without manually querying the cluster. For infrastructure developers, the focus is improving the operator development experience so more teams can build on the platform.

Why Imperative Fleet Management Doesn’t Scale

Managing Cassandra clusters with imperative tooling meant every change — a schema update, a node replacement, a configuration tweak — had to be executed by a human or a script that encoded a specific sequence of steps. As Spotify’s fleet grew, this approach broke down in predictable ways. Operators had to know the exact current state of each cluster before running anything, which made automation fragile and audits difficult. The system worked when clusters were few and changes were rare; it became a bottleneck when fleets grew and iteration speed mattered.

The team’s move to declarative infrastructure was driven by a simple observation: the desired end state of a cluster — its topology, its configuration, its schema — is a far more stable and useful artifact than the series of shell commands needed to reach it. When you describe what you want, the platform can figure out how to get there, and can keep getting there as conditions change. This shift, applied to the storage layer that underpins much of Spotify’s data, has changed how the fleet is operated and evolved.

The Orchestrator Pattern

At the core of the new system is a custom Kubernetes operator, which acts as the control loop for each Cassandra cluster. The operator watches a custom resource definition (CRD) that holds the desired state — the cluster size, the hardware profile, the Cassandra version, and the schema. Whenever that desired state changes, the operator reconciles the live cluster against it, issuing the necessary imperative commands to bring reality in line with intent.

This is not merely automation of previous manual steps. The operator encodes operational knowledge that previously lived in runbooks and senior engineers’ heads: how to safely add a node, how to decommission one without losing data, how to roll a configuration change without triggering a cascading failure.

Reconciliation is idempotent — running it once or a hundred times yields the same result, which is what makes automated remediation and self-healing possible.

Using Kubernetes as the substrate provided a uniform API for teams to declare their needs. While the storage fleet is managed by a central infrastructure team, the declarative model means application teams can express requirements (e.g., “this keyspace needs three replicas across two regions”) without needing to understand the underlying operational procedures. The CRD becomes a contract between the two groups.

Schema Management as Code

One of the most error-prone parts of Cassandra operations is schema changes — ALTER TABLE and similar operations are asynchronous and can have subtle impacts on cluster performance. In the imperative world, a bad schema migration could be incredibly difficult to roll back. In the declarative world, the schema is part of the same desired-state definition as the cluster topology.

The team developed a tool that turns schema definitions into a series of migrations, and the operator applies them in the correct order during reconciliation. If a schema change needs to be reverted, the declared state is reverted, and the system handles the consequences. This makes schema evolution reviewable in the same pull-request workflow as application code, bringing the same rigor to database changes that already applies to software changes.

Operational Benefits and New Failure Modes

The payoff has been a significant decrease in toil. Routine operations like scaling a cluster or upgrading Cassandra are now declarative requests instead of multi-hour manual procedures. The team able to run many more clusters — including smaller, specialized ones — without a corresponding increase in operational headcount. This has improved resource utilization because teams can spin up dedicated clusters for workloads with very different access patterns, rather than forcing everything into a few large shared clusters.

That said, the model introduced its own set of challenges. When reconciliation is automated, errors are amplified in different ways than before. A bug in a schema template could, in theory, be rolled out to many clusters simultaneously. The team has had to invest in robust validation and dry-run capabilities to ensure that the operator fails safely. The declarative API also means that mistakes happen at the level of intent — misconfiguring a CRD — rather than at the level of an individual shell command, which requires a different debugging mindset.

Observability has had to evolve to match. Teams now need to understand not just whether a cluster is healthy, but whether the reconciliation loop is progressing or stuck. Metrics and alerts had to be built around the operator itself and its view of cluster state.

The Road Ahead

The team is moving away from managing the underlying Cassandra nodes with imperative shell commands towards a model where the entire lifecycle is governed by declared state. This includes exploring how to apply the same principles to the other stateful services in the fleet, such as Kafka and Elasticsearch, whose operational burden has historically been even higher than Cassandra’s. The goal is to build a general-purpose declarative substrate for stateful infrastructure.

Shifting from imperative to declarative has also opened the door for a tighter feedback loop with the open-source community. As the team gains experience with the operator pattern on Cassandra at Spotify’s scale, it is able to contribute lessons learned back to the Kubernetes ecosystem, influencing how the next generation of stateful workloads will be managed.