A Decade of Cache-Driven Outages at Twitter

Major incidents at Twitter that were at least partly attributed to cache, tracked between 2012 and 2022, include six SEV-0s and six SEV-1s, plus one earlier event. There were also 38 less severe incidents in that period. This collection focuses on the major ones, with an eye toward the patterns that emerge when failures are viewed through the specific lens of caching.

Cache is a common culprit in major failures, and Twitter’s experience illustrates why. A former Twitter engineer observed that for a long time, cache was the top source of site-down incidents. In her first six months, restarting a single cache server could cause a SEV-0-level outage. The infrastructure then was a fork of memcached, widely used elsewhere, which makes the lessons here broadly applicable.

The Nature of Cache Failure Modes

Conceptually, a cache server is a high-throughput, low-latency RPC server paired with a data management library. Since the data management side can easily outpace the RPC side for in-memory caches, cache workloads approximate nearly pure RPC, which makes them important in service-oriented architectures. When performance is critical, caches are sharded into distributed clusters, exposing them to the standard pitfalls of distributed systems, though with less emphasis on synchronization issues than, say, a strongly consistent database.

Cache outages easily cascade into total outages built around an assumption that a significant fraction of data is served from cache, not backing stores. Caches are also highly sensitive to anomalies lower in the stack because they have high-volume, low-latency SLOs and spend a large fraction of time in the kernel. With few threads, they are easily starved by co-tenants. High volume and low latency worsen positive feedback loops, producing death spirals. Most of the incidents in this record aren’t logic errors in cache itself, but anomalies that trigger an unmitigated runaway loop.

2011-08: The Case of the Flickering User Names

For a few months, a significant fraction of user changes, like updating a username, screen name, or password, would be reverted. Risk persisted for years.

At that time, the Rails app used single-threaded workers managed by a master that killed and restarted any worker stuck for 30 seconds. Each worker had a cache client that made independent sharding decisions, ejecting hosts it considered unhealthy. This caused data inconsistency when clients disagreed on which cache shard held a key.

After weeks of adding instrumentation, an engineer found that roughly 0.2% of users in cache were out of sync with the database. A check of 150 popular users found that most were stored on two or more caches when should have been in at most one. The first fix prevented writing stale data back to the DB. A second change increased retry effort before ejecting a host and extended the ejection period, intending to keep flapping keys in place. This change, when fully rolled out, immediately took the site down. If a shard was ejected, the prolonged ejection time pushed load onto MySQL, which under high load could take arbitrarily long to respond, so workers were killed, take down the site.

The root cause was eventually traced to packet loss from an overloaded CPU0. Interrupt request affinity wasn’t configured, and CPU0 spent 65-70% of its time handling soft IRQs. Setting IRQ affinity to spread packet processing across cores fixed the loss on the original cluster, but a new cluster then also showed packet loss.

That new loss was attributed to a specific hardware SKU and BIOS version. A BMC firmware health check ran every 20 hours and 40 minutes, interrupting the kernel long enough to drop packets. A kernel-team ticket that documented the problem and proposed a fix was marked resolved months earlier but never widely rolled out. Staggered host reboots were applied as a mitigation, and the BIOS fix was expedited. Total recovery took about two months, but the ejection architecture itself remained unable to tolerate partitions without generating stale or duplicate cache entries. Fixing that took about two years.

  • Fixes: Added visibility into cache consistency; set IRQ affinity; fixed a firmware issue causing packet drops; eventually re-architected cache.
  • Lessons: Need visibility, low-level systems understanding, and isolated changes; mixing a new cluster migration with the same IRQ affinity fix introduced a second instance of packet loss.

Non-personalized trends failed to display for about 10% of users for 10 hours. A recent deploy added a Thrift field to the Trend object, which raised memory usage in the trends cache refresh daemon past its limit, causing segfaults. The alarm monitored the daemon starting a run, but not completing one.

2012-07: Partial Site Outage on Rack Power Loss (SEV-0)

A two-rack power loss caused roughly four hours of significant site impact. When power returned, 37 of 48 cache hosts failed to restart because a startup script expected a directory on a mount that no longer existed. The directory wasn’t needed for running a cache server, but the script wasn’t updated in lockstep with the layout change that removed it. Debugging was slowed because /proc wasn’t mounted correctly on some rebooted hosts, an independent issue. Other compounding issues were a 5-second timeout that never fired because it compared the wrong timestamp, a front-end outage page served with ERB templates still using Ruby wildcards after a recent migration, CSS that wouldn’t load because it was hosted elsewhere, and a user-data service logging so heavily during failures that garbage collection disabled it.

2013-01: Site Outage for 3h30m (SEV-0)

An ordinary increase in load produced a tail-latency spike on cache because new hosts didn’t have IRQ affinity set. Elevated queue lengths caused latency to get worse until the tweet service entered a garbage collection–induced death spiral. Caches for user data and tweet service were co-located; the user data team had added hosts gradually overnight. Peak traffic the following morning precipitated the incident, for which GC configs weren’t adjustable since GC stats weren’t exported. Before this, an alert and analysis indicated that the missing IRQ settings were a known issue. Among the fixes afterward: make IRQ affinity a default for all cache hosts, reduce connection limits to about 2x steady state, and reduce the tweet service’s maximum connections while increasing timeout.

2013-09 / 2014-01: The Trouble with Clients (SEV-1 / SEV-0)

In the 2013-09 incident, a cache serverset change cut off the timeline service from 75% of a cache, since it didn’t use the recommended mechanism to find that serverset’s path and so didn’t know which hosts were the cache.

The 2014-01 event caused a one-datacenter outage with 20 minutes of impact. A recent change had removed the cap on client connections. When caches neared 160k connections, they would begin rejecting new ones, which triggered the monitoring system to restart the affected cache shards. Several tickets preceding the outage described pieces of the problem: an engineer had previously debugged why caches stopped accepting connections at roughly that limit, a kernel fix addressed OOMs tied to 32kB page exhaustion, and other tickets noted the need to standardize hardware settings for IRQ affinity, C-states, and firmware versions.

Client retry settings let tweet service amplify load and initiate a doomed positive feedback loop. With 5 requests before marking a node down, 16 connections of parallelism, and clients talking to multiple servers, the product numbers quickly overcame what any cache could handle. On the server side, a backlog of 1024 allowed queued work to accumulate dangerously without applying backpressure.

Fixes reduced that backlog to 128, limited fd count, pinned the hash table and CPU affinity, and changed client timeout/retry/connection settings. Testing showed caches stayed up even with clients still configured to “attack,” confirming the cache changes were central. On lesson: clients must calculate impact when choosing retry and connection settings, because naive high concurrency can overwhelm any server.

2014-03: Oscars Retweet Storm (SEV-0)

A heavily retweeted celebrity Oscars post drove a 25-minute search outage and prevented many using the site. Cache overload again stemmed from client misbehavior, worse than the 2014-01 case since subsequent cache mitigations weren’t sufficient. Containerized cache on shared infrastructure made some shards vulnerable to throttling death spirals. The major fix was pipelining in the Finagle memcached client, letting clients achieve decent throughput with only one or two open connections. Numerous other service-level fixes followed, including coordinated backpressure, same-DC data flow, dedicated thread pools for critical operations, in-process caching of hot items, degraded, incomplete responses rather than hard failures, and a guide on configuring cache clients.

2016-01: The Boy-Band Panic (SEV-0)

SMAP’s denial of breakup rumors crashed traffic. Load in one DC spiked and caused 39 cache host OOM kills. Because more than a few nodes were dead, the cluster manager wouldn’t automatically remove them for fear of sending catastrophic load to storage. Manual cleanup didn’t match with a puppet change that had removed the service that usually restarts those nodes. The OOMs stemmed from too much kernel socket buffer memory. The initial mitigation of a 4 GB TCP buffer limit failed a stress test due to memory fragmentation on hosts with two years of uptime, so the reboot cadence changed.

2016-02 / 2016-07: Tests and Migrations (SEV-1 / SEV-1)

The 2016-02 incident was essentially the failed stress test above, causing some minutes of mild site degradation. Later that year, a planned cache migration went wrong. A misplaced configuration change added 115 new cache instances to a serverset as fast as possible rather than one every 20 minutes. As hit rate dropped from 99.8% to 84%, spikes hit the backing DB, created long GC pauses, and made request success drop to 0% when missing cache. Operators killed the cluster manager, restored the prior configuration, and did a rolling restart of the DB with strict rate limits. Limits set too tight after failing away from the first DC caused rejection in a second DC.

2018-04 / 2018-06: Failover Tests Expose Cracks (SEV-0 / SEV-1)

A planned test failover in April 2018 caused an hour of partial outage. User data service suffered cache hot keys. A mechanism meant to promote hot keys sampled 1% of events into a FIFO with an associated hash-map count, but worked poorly for caches where individual values were much larger than the tweet service keys (~5KB) it was designed for. The user data cache wasn’t a design consideration for hot-key promotion since the hottest items were served from an in-process cache at the time. The FNV1-32 hash used also ignored least-significant bytes, causing key data variance to land all in a few shards. Hot cache shards saturated 1Gb NIC connectivity. By June, a second test failover surfaced two more issues: cache shards could only sustain half the expected traffic compared to load tests with unrepresentative, mostly-Poisson arrival distributions and far fewer connections, and a periodic ten-second container-based performance counter collection process pushed shards into throttling delays. After a few failed requests, clients marked nodes down, causing request failures as load exceeded the backing DB quota. Fixes included cutting client connections from 4 down to 2, enlarging cache capacity, improving hot-key promotion, changing capacity qualification tests to reflect real request patterns and connection counts, and investigating the cost of multiplexed hardware counter gathering.

Tradeoffs and Continuing Work

Multiple incidents trace back to deliberately configured client retry and connection concurrency set “just in case” high. This increased tail latency and worsened ejection-causing timeouts. Conversely, the introduction of pipelined clients made it possible to achieve the same throughput at much lower connection counts, but only if clients limited themselves. Similarly, attempts to protect storage with rate limits only worked when set low enough to trigger correctly.

Infrastructure hygiene was also a repeated pain point. Unset IRQ affinity (irqbalance or explicit affinity), stale puppet scripts, non-standardized host settings for C-states, turbo boost, or firmware, and missing mounting of /proc all produced major incidents or amplified existing headroom failures. Hardware heterogeneity created additional fragility during peak load events when NIC bandwidth or other host-specific differences mattered.

Knowledge preservation was often a failure mode in itself. Tickets were sometimes marked resolved without the fix being rolled out widely, as in the earlier kernel change that would have prevented packet drops, and root-cause insight from one hardware SKU or version wasn’t always checked against others. The distinction between “daemon started” and “daemon finished” also let some incidents go unnoticed until they’d become full-blown outages. Incident analysis showed that even the adopted mitigation of a UDP packet ring buffer fix was never widely deployed because the knowledge was lost before implementation.

Finally, over the same decade, Twitter developed a replacement cache called Pelikan out of the operational pain of running memcached and Redis at scale. Though not itself a response to any single SEV in this list, Pelikan’s modular design was meant to allow easier iteration and improvement relative to the standard memcached fork that showed such a wide variety of failure signatures.