Slack’s Migration to a Cellular Architecture

Why a Single Failed Availability Zone Caused User-Visible Outages

On June 30, 2021, Slack experienced a notable service outage. A network link connecting one availability zone (AZ) to several others in the company’s primary hosting region failed intermittently. The link was briefly restored, then failed again and was permanently removed from service. At first glance, this reads as a routine hardware failure. Yet Slack’s post-incident review surfaced a more troubling question: why were users affected at all?

Slack operates a multi-regional edge network, but most core compute runs in multiple AZs within a single region, us-east-1. AZs are designed to be isolated datacenters whose underlying components—virtualization, storage, networking—should not fail simultaneously across zones. Architecting across AZs should yield regional availability greater than that of any single AZ. In this case, that strategy failed because detecting failure in distributed systems is hard.

A single user request, such as loading a channel, can fan out into hundreds of RPCs to backend services. Frontends do continuously probe and exclude failed backends, but they must observe some failures before exclusion can happen. The problem is compounded by strongly consistent datastores like Vitess, Slack’s primary datastore. For a given write there must be a single available shard primary. If an application frontend cannot reach that primary, writes fail until the primary recovers or a secondary is promoted.

This incident is a classic example of a gray failure: different components have divergent views of system availability. Systems within the impacted AZ saw internal backends as healthy, but unavailable to the outside, and vice versa. Even clients within the same AZ could see different statuses depending on whether their network flows traversed the faulty equipment. Automating remediation of this kind of ambiguity is extremely complex.

Slack’s answer was to make the system's job easier by putting human judgment in the loop. During the incident, engineers could see clearly that one AZ was unreachable—nearly every graph aggregated by target AZ told the same story. What they lacked was a mechanism to act on that insight. So Slack set out to build a “button” that would drain traffic from a failing AZ.

The AZ Drain Design Goals

Implementing an AZ drain is conceptually simple but operationally intricate. Slack defined four design goals for the mechanism:

1. Remove as much traffic as possible from the failing AZ within five minutes. Slack’s 99.99% SLA allows less than an hour of total downtime per year, so mitigation tools must act quickly.
2. Drains must not introduce user-visible errors. Draining should be usable as a generic mitigation even before the root cause is understood; if it worsens the outage, it ceases to be useful.
3. Traffic shifting must be incremental. When undraining, an operator should be able to route as little as 1% of traffic back to test whether the AZ has truly recovered.
4. The mechanism must not depend on resources within the AZ being drained. It must work even when the AZ is entirely offline.

Siloing: Making the Problem Single-Place

A direct implementation—plumbing a drain signal into every RPC client—was rejected early. Slack’s services span Hack, Go, Java, and C++, requiring separate implementations per language. Internal service discovery also varies, spanning Envoy xDS, Consul, and DNS, and DNS offers no native notion of an AZ or partial draining. Open-source systems like Vitess would require forking or upstream changes to accommodate such a mechanism.

Instead, Slack adopted a different architectural pattern called siloing. A service is siloed when it both receives traffic only from clients within its AZ and sends traffic only to upstream servers in the same AZ. The net effect is that each logical service becomes N virtual services—one per AZ—even while the physical infrastructure remains shared.

With siloing, all services run in all AZs, but each communicates only within its own zone. A failure inside one AZ is contained to that zone. Crucially, to steer traffic away from a problematic AZ, Slack now only needs to redirect at the frontend; internal services in the excluded AZ naturally quiesce for lack of new work.

Envoy and Weighted Clusters as the Drain Mechanism

Siloing concentrated the traffic-shifting logic at the entry point to Slack’s core services. As part of an ongoing migration from HAProxy, all edge load balancers now run Envoy, configured by Rotor, an in-house xDS control plane. That made an AZ drain achievable with two standard Envoy features: weighted clusters and dynamic weight assignment via RTDS.

To drain AZ, the operator sends a signal through Rotor that instructs edge Envoy load balancers to reweight clusters for the target AZ to zero. Envoy finishes in-flight requests but routes all new traffic to the remaining AZs. The approach satisfies Slack’s original goals:

1. Control-plane propagation takes seconds, so new weights apply almost immediately.
2. Draining is graceful; the load balancer does not abandon queries already in flight.
3. Weights allow gradual drain and undrain with 1% granularity.
4. The control plane is regionally replicated and does not depend on the AZ being drained; edge load balancers even run in other regions.

The bandwidth-per-AZ graph shows sharp “knees” as traffic shifts from one AZ to the others, reflecting the combination of fast propagation and precise weight control. The siloing pattern and Envoy-based drain mechanism are foundational steps Slack plans to build on, both in operations and in how future services are designed.