Actor of Last Resort: The Cascading Shard

In October 2022, Slack’s Datastores team faced back-to-back incidents. On two consecutive days, the EMEA on-call rotation was paged with the same symptom: an increase in failed database queries. Both times the trigger was an asynchronous forget user job that purges records for a workspace removing a very large number of users at once. The first incident was contained with a shim that blocked new jobs from being processed; the second was severe enough to impact customers trying to load Slack.

The User Removal Trigger

A single customer deleted thousands of users from their workspace in one operation, which is unusual — removal normally happens in small batches as people leave a company. Removing users spawns the forget user job, which must locate and delete every subscription record for each user: every channel they belong to, every thread they participated in. That means the number of database queries equals the number of deleted users multiplied by their average subscription count.

The dashboard above shows the dramatic spike in query volume that overwhelmed one shard in particular, which held 6% of the user’s subscription data. As the write load mounted, replicas fell behind and the shard’s primary tablet ran out of memory. The kernel OOM-killed MySQL, a replica was promoted, and the cycle began again — each newly promoted primary also ran out of memory, and each replacement replica could not catch up due to the sustained write volume.

Sharded Storage and Replication

Slack’s MySQL clusters are managed with Vitess. Tables are organized into keyspaces — distributed databases that present as a single MySQL instance to clients but are actually split across multiple MySQL schemata. Each keyspace is composed of shards, where each shard owns a key range. For a keyspace split into two shards, one covers keys below (2^64)/2 (the -80 range), the other above (the 80- range). Vitess stores shard metadata internally so VTGates can route queries to the correct shard.

Sharding isn’t only for scale. It also yields several operational benefits: faster and smaller backups, a smaller blast radius when a shard fails, smaller host machines, distribution of query load, and higher write capacity.

Every shard consists of one primary tablet and multiple replica tablets. Primaries handle data-modifying queries (DML); replicas serve read queries. Committed data propagates from the primary to replicas via MySQL replication, which is asynchronous. Under heavy write load, replicas can lag significantly. Slack removes replicas from service if replication lag exceeds one hour.

Replacing Replicas

Replacing a replica tablet is a four-step process: provision a new host with all dependencies; restore the most recent backup to obtain a data copy; catch up on replication; and finally deprovision the old replica. The catch-up step is critical — after backup restore, the copy is stale. The new replica reads changes from MySQL’s binary log and applies them until replication lag falls below the acceptance threshold.

The Infinite Replacement Loop

Here’s how the incident spiraled out of control. The huge volume of writes caused MySQL on the shard primary to be OOM-killed. Automation promoted a replica to primary and started provisioning a replacement. That replacement restored from the last backup but could not catch up: the primary was still executing a massive number of writes, so the replacement lagged far behind. The automation interpreted the prolonged catch-up as a sign of ill health and deprovisioned the new replica. Meanwhile, the new primary also ran out of memory and was killed, triggering another promotion and another doomed replacement cycle.

The shard was caught in an endless loop of primary failure, promotion, replacement provisioning, failed catch-up, and deprovisioning.

Breaking the Cycle

Manual, Larger Replicas

Datastores broke the loop in two ways. First, they manually provisioned larger instance types with more CPU and memory, preventing the OOM-kills that kept toppling primaries. Second, they switched from automation-orchestrated replacement to manual provisioning. This avoided the auto-deprovisioning of healthy replacements that simply needed more time to catch up.

Fixing the “Forget User” Job

The root cause ran deeper than infrastructure. The forget user job had poor performance characteristics. When processing a user, it collected all the channels the user belonged to, then dispatched a leave channel job for each. Normally leave channel handles a single channel when a user manually leaves; during this incident, the job ran for every channel of every deactivated user.

The owning team found and fixed three inefficiencies in leave channel:

  1. Each job run queried for all of the user’s subscriptions across every channel, even though the job processed only one channel.
  2. The UPDATE to mark thread subscriptions as inactive was scoped to the channel being processed, but the query included all of the user’s thread subscription IDs from all channels. For some users, that was tens of thousands of IDs — extremely expensive for MySQL.
  3. After the UPDATEs, the job re-queried all thread subscriptions to notify connected clients about unread thread counts, even for users being deactivated who couldn’t be connected.

The team changed the job to query only subscriptions within the channel being processed and to scope the UPDATEs to those IDs alone. The notification step was skipped entirely in the deactivation scenario, since deactivated users can’t receive updates.

Temporary Client Feature Disable

As a final temporary mitigation, the client team disabled Thread View in the Slack client. This cut the volume of read queries hitting replicas, giving the database cluster room to recover. The feature was re-enabled as soon as it was safe to do so.

Defense in Depth: Throttling, Circuit Breakers, and Smarter Jobs

In the aftermath of the incident, the Datastores team moved quickly to address the immediate edge-case with replacements that had been uncovered, treating it as a top priority. Beyond that specific fix, they began integrating two complementary safeguards into their Vitess-based infrastructure: the tablet throttling mechanism and the circuit breaker pattern. Both are designed to prevent a client from overwhelming the database with excessive queries in the first place.

Throttling controls the rate at which queries are processed, helping the database allocate resources to critical operations while shedding or postponing less essential work. The circuit breaker pattern serves as a fail-safe: it monitors the health and responsiveness of replicas, and if an unhealthy state is detected, it temporarily halts the flow of queries to the affected shard. This isolates the problem, gives the replicas time to recover, and prevents cascading failures across the system.

In practice, when tablets become unhealthy or degrade in performance, the team can limit or cancel queries directed at that shard. Once the tablets are restored to a healthy state, normal query operations resume without compromising overall system performance. The Datastores team has been actively contributing related features and bug fixes upstream to Vitess, a positive outcome of the incident.

The team responsible for the “forget user” job also took further steps to reduce database strain. The existing “leave channel” job is appropriate when a user leaves a single channel, but during a “forget user” operation, firing it for every channel a user belonged to created unnecessary contention. Instead, the team introduced a new job to unsubscribe a user from all of their threads. “Forget user” now enqueues a single “unsubscribe from all threads” job, which significantly lowers contention during these runs.

Additionally, the Forget User job adopted the exponential back-off algorithm alongside the circuit breaker pattern. Failed jobs now cope with the state of their dependencies—such as the database—by backing off and eventually stopping retries rather than hammering an already strained system.

What the Incident Taught Us

The incidents on October 12th and 13th, 2022 laid bare the challenges faced by the Datastores EMEA team and the teams running asynchronous jobs at Slack. A mass removal of users from a workspace triggered a spike in write requests that overwhelmed the Vitess shards. Replicas fell behind the primary, and the primary eventually crashed, setting off an endless loop of replacements that compounded the strain.

Mitigation came from two fronts: the Datastores team manually provisioned replicas with more memory to break the replacement loop, while the team responsible for the Forget User job stopped the job generating the write load and optimized its queries to relieve the primary database.

The preventative measures that followed—throttling mechanisms, the circuit breaker pattern, and exponential back-off—provide a layered defense against query overload. Together, they help ensure the database infrastructure remains stable and responsive, reducing the likelihood that a similar incident can escalate into a full outage.