Custom Rollout Control for Large StatefulSets

Running stateful applications on Kubernetes with StatefulSets gives teams stable Pod identity and persistent volumes, but the built-in update strategies leave gaps for operators managing large fleets. Slack's platform engineering group hit those limits and built a Kubernetes operator to close them.

Why Native Strategies Fall Short

Kubernetes ships two update strategies for StatefulSets, configured via .spec.updateStrategy:

  • OnDelete — The controller never updates Pods automatically; users delete Pods manually to trigger replacement with the new template.
  • RollingUpdate — The default strategy; the controller rolls out updates automatically, one Pod at a time.

RollingUpdate supports useful features like Partitions for percent-based rollouts and .spec.minReadySeconds to control pacing. But the maxUnavailable field for StatefulSet is still alpha, gated behind the MaxUnavailableStatefulSet feature flag on the API server, so it isn't available in AWS EKS. That means a RollingUpdate can only touch one Pod at a time — painfully slow for services with hundreds of Pods.

OnDelete gives you direct control but none of the conveniences: no percentage-based rollouts, no pause, no automation. Slack's internal teams wanted more: faster percent-based rollouts, quick rollbacks, pausable deployments, integration with Consul for service discovery, and Slack notifications on rollout progress. The platform team built the Bedrock Rollout Operator to provide those capabilities on top of StatefulSets.

The Bedrock Rollout Operator

Bedrock is Slack's internal platform layer for Kubernetes, providing engineers an opinionated configuration interface and integrations with the rest of the infrastructure: Consul for service discovery, Consul-Template/Vault for secrets, Nebula for encrypted internal traffic, and Envoy for layer-7 routing. The rollout operator — built with Kubebuilder — manages a custom resource named StatefulsetRollout, which carries the StatefulSet spec plus extra parameters for pause, Slack notifications, and other rollout controls. The operator runs in each of Slack's 200-plus Kubernetes clusters, which host over 50 stateless services (Deployments) and nearly 100 stateful services (StatefulSets).

Rollout Flow End to End

A rollout begins when an engineer writes a bedrock.yaml config file in their application repository:

images:
 bedrock-tester:
   dockerfile: Dockerfile

services:
 bedrock-tester-sts:
   notify_settings:
     launch:
       level: "debug"
       channel: "#devel-rollout-operator-notifications"
   kind: StatefulSet
   disruption_policy:
     max_unavailable: 50%
   containers:
     - image: bedrock-tester
   stages:
     dev:
       strategy: OnDelete
       orchestration:
         min_pod_eviction_interval_seconds: 10
         phases:
           - 1
           - 50
           - 100
       clusters:
         - playground
       replicas: 2

The engineer then triggers a deployment from Slack's internal release UI:

The release platform calls the Bedrock API, which parses the config and generates a StatefulsetRollout custom resource:

apiVersion: bedrock.operator.slack.com/v1
kind: StatefulsetRollout
metadata:
 annotations:
   slack.com/bedrock.git.branch: master
   slack.com/bedrock.git.origin: [email protected]:slack/bedrock-tester.git
 labels:
   app: bedrock-tester-sts-dev
   app.kubernetes.io/version: v1.custom-1709074522
 name: bedrock-tester-sts-dev
 namespace: default
spec:
 bapi:
   bapiUrl: http://bedrock-api.internal.url
   stageId: 2dD2a0GTleDCxkfFXD3n0q9msql
 channel: '#devel-rollout-operator-notifications'
 minPodEvictionIntervalSeconds: 10
 pauseRequested: false
 percent: 25
 rolloutIdentity: GbTdWjQYgiiToKdoWDLN
 serviceDiscovery:
   dc: cloud1
   serviceNames:
   - bedrock-tester-sts
 statefulset:
   apiVersion: apps/v1
   kind: StatefulSet
   metadata:
     annotations:
       slack.com/bedrock.git.origin: [email protected]:slack/bedrock-tester.git
     labels:
       app: bedrock-tester-sts-dev
     name: bedrock-tester-sts-dev
     namespace: default
   spec:
     replicas: 4
     selector:
       matchLabels:
         app: bedrock-tester-sts-dev
     template:
       metadata:
         annotations:
           slack.com/bedrock.git.origin: [email protected]:slack/bedrock-tester.git
         labels:
           app: bedrock-tester-sts-dev
       spec:
         containers:
           image: account-id.dkr.ecr.us-east-1.amazonaws.com/bedrock-tester@sha256:SHA
           name: bedrock-tester
     updateStrategy:
       type: OnDelete

The top-level fields of the resource's spec add the custom behavior:

  • bapi — Callback details for the Bedrock API to signal rollout completion or failure
  • channel — Slack channel for notifications
  • minPodEvictionIntervalSeconds — Optional; time to wait between Pod rotations
  • pauseRequested — Optional; pauses an ongoing rollout when set to true
  • percent — Set to 100 for full rollout, lower for percent-based deployments
  • rolloutIdentity — A random string enabling retries when a rollout fails due to transient issues
  • serviceDiscovery — Consul registration details for querying service health during the rollout

Notice the disruption_policy.max_unavailable value from bedrock.yaml doesn't appear in the custom resource — the API uses it to create a Pod disruption policy instead. At runtime, the operator reads that policy to determine how many Pods can roll out in parallel.

The operator then steps in, converging the cluster's current state toward the desired state in the StatefulsetRollout resource. Throughout, it sends rich Slack notifications — designed with Block Kit Builder — tracking version numbers and listing Pods currently being rolled out:

Once converged, the operator calls back to the Bedrock API with success or failure; the API relays that to the release platform so the state shows in the UI.

Reconcile Loop Design

The operator watches StatefulsetRollout resources as the desired state and reconciles them against what actually exists. A typical rollout applies a new StatefulSet spec, then terminates the target number of Pods — half, for a percent: 50 rollout. The reconcile loop:

  1. Reads the expected state from the custom resource spec
  2. Reads the actual state from the StatefulSet and its Pods
  3. Takes action: updates the StatefulSet with the new spec, or evicts Pods so replacement Pods run the new version

Most Kubernetes controllers are event-driven, watching resources and running the reconcile loop on each update. This operator deliberately takes a different approach: it enqueues its own next reconcile, re-requesting future runs as long as change is expected, and exits cleanly once reaching a final state like RolloutDone or RolloutFailed. That drastically cuts the number of reconciliations and guarantees sequential processing per resource, avoiding the race conditions that parallel loops could cause when mutating the same custom resource.

Each loop takes exactly one action, then schedules the next run a few seconds out. Keeping loops small and quick makes the operator resilient to disruption. The key state — the Phase — is stored in the custom resource status:

// StatefulsetRolloutStatus defines the observed state of StatefulsetRollout
type StatefulsetRolloutStatus struct {
 // The Phase is a high level summary of where the StatefulsetRollout is in its lifecycle.
 Phase RolloutPhase `json:"phase,omitempty"`
 // PercentRequested should match Spec.Percent at the end of a rollout
 PercentRequested int `json:"percentDeployed,omitempty"`
 // A human readable message indicating details about why the StatefulsetRollout is in this phase.
 Reason string `json:"reason,omitempty"`
 // The number of Pods currently showing ready in kube
 ReadyReplicas int `json:"readyReplicas"`
 // The number of Pods currently showing ready in service discovery
 ReadyReplicasServiceDiscovey int `json:"readyReplicasRotor,omitempty"`
 // Paused indicates that the rollout has been paused
 Paused bool `json:"paused,omitempty"`
 // Deleted indicates that the statefulset under management has been deleted
 Deleted bool `json:"deleted,omitempty"`
 // The list of Pods owned by the managed sts
 Pods []Pod `json:"Pods,omitempty"`
 // ReconcileAfter indicates if the controller should enqueue a reconcile for a future time
 ReconcileAfter *metav1.Time `json:"reconcileAfter,omitempty"`
 // LastUpdated is the time at which the status was last updated
 LastUpdated *metav1.Time `json:"lastUpdated"`
 // LastCallbackStageId is the BAPI stage ID of the last callback sent
 //+kubebuilder:validation:Optional
 LastCallbackStageId string `json:"lastCallbackStageId,omitempty"`
 // BuildMetadata like branch and commit sha
 BuildMetadata BuildMetadata `json:"buildMetadata,omitempty"`
 // SlackMessage is used to update an existing message in Slack
 SlackMessage *SlackMessage `json:"slackMessage,omitempty"`
 // ConsulServices tracks if the consul services specified in spec.ServiceDiscovery exists
 // will be nil if no services exist in service discovery
 ConsulServices []string `json:"consulServices,omitempty"`
 // StatefulsetName tracks the name of the statefulset under management.
 // If no statefulset exists that matches the expected metadata, this field is left blank
 StatefulsetName string `json:"statefulsetName,omitempty"`
 // True if the statefulset under management's spec matches the sts Spec in StatefulsetRolloutSpec.sts.spec
 StatefulsetSpecCurrent bool `json:"statefulsetSpecCurrent,omitempty"`
 // RolloutIdentity is the identity of the rollout requested by the user
 RolloutIdentity string `json:"rolloutIdentity,omitempty"`
}

The status struct tracks a lot of metadata: the Slack message ID, the list of managed Pods, and which version each Pod runs.

Operational Lessons

Scale Surprises

Slack's scale is large, with significant traffic backed by stateful services on Bedrock:

But some StatefulSets spin up to 1,000 Pods, and the initial design — one Slack message per Pod, with up to 100 Pods rotating in parallel — trip rate limits. The team rewrote the notification stack with pagination, batching up to 50 Pods per message.

Version Leak

Relying on the OnDelete strategy creates what the team calls a "version leak." In a percent-based or paused rollout, some Pods run the new version and some run the old. If a Pod on the old version gets terminated for an unrelated reason — node scaling, compliance rotation, chaos engineering — the controller replaces it with a Pod running the new version. Over time, a deliberately stopped rollout drifts toward fully deployed. That's a known, accepted limitation; Slack teams push their services to 100% before it becomes a problem.

Direction

The operator model proved effective, so Slack now manages all Kubernetes deployments this way — but not necessarily by extending the StatefulSet operator. For Deployment resources, the team is evaluating existing CNCF projects: Argo Rollouts and OpenKruise.

Custom rollout logic inside an operator isn't simple work, and upstream Kubernetes features like the maxUnavailable field for StatefulSet might eventually retire some of that custom code. For now, the operator model delivers what the native strategies can't: direct integration with Slack's notification, service discovery, and release systems, with rollouts that scale past the built-in one-Pod-at-a-time pace.