Four Outages in Twelve Hours: What Broke Figma’s Backend

Between 11:34 PM PDT on June 6 and 10:43 AM PDT on June 7, 2022, Figma experienced four separate service disruptions lasting from seven minutes to one hour and twenty minutes. This was the longest period of instability in the company’s history. During those windows, already-loaded files remained usable, but users could not open new files or collaborate. No data was lost; changes made during the disruptions were saved locally and synced to servers once service resumed.

The incidents traced back to a rare bug in AWS ElastiCache for Redis, specifically in clusters running with Cluster-Mode Enabled (CME). Figma uses ElastiCache—a managed Redis service—to accelerate frequently accessed data and route messages between services. At the time, the company ran two ElastiCache instances: one with CME, which supports horizontal scaling by adding nodes dynamically, and one with Cluster-Mode Disabled (CMD).

How a Redis Cluster Bus Became the Bottleneck

The root cause was sudden, complete saturation of the Engine CPU on one node in the CME instance. AWS later identified the trigger: a high volume of Redis Publish/Subscribe operations on a CME cluster. Figma had shifted its Pub/Sub load from the CMD instance to the CME instance a few weeks before the incidents. The CME instance handled the workload without issue until the June 6 outage, and there were no significant changes in traffic or Redis usage in the days prior.

The underlying problem lies in how Redis Cluster Mode communicates between nodes. A cluster uses a “Cluster Bus” as a control plane for coordinating system messages. Pub/Sub is one of the only data plane operations that also relies on this bus to exchange client data across nodes. The Cluster Bus maintains an outgoing buffer for each node. The buffer grows exponentially when the frequency or size of messages exceeds the rate at which the bus can send data to all other nodes—or if there are network issues with any single node, even if message volume stays constant.

Each time the bus sends data, it truncates the buffer using an internal Redis function called sdsrange, which relies on memmove to shift remaining data to the head of the buffer. The CPU cost of memmove is proportional to the amount of memory moved. If the bus sends 100KB to a client while 10GB remains queued, memmove must process nearly 10GB of data per send. Because buffer growth is exponential, CPU consumption escalates quickly. AWS’s reproduction of the incident showed most Engine CPU time spent in memmove within the sdsrange call stack.

The process is self-reinforcing: as the buffer grows, truncation becomes more expensive, slowing message delivery, which in turn drives further buffer growth. This explains why the Engine CPU remained pinned at 100% even after the original triggering condition subsided. AWS has made enhancements to its managed Redis that worsen the issue compared to open source Redis, though open source is also vulnerable. Figma recommends avoiding Pub/Sub on any Redis instance with Cluster Mode Enabled until the bug is fixed (tracked in the Redis issue tracker).

Chasing the Cause While Keeping Service Up

Figma’s monitoring alerted the team within seconds. Initially, the leading hypotheses were a capacity problem or a bad machine. The team initiated a failover of the impacted node while spinning up a larger replacement cluster. The failover did not complete as quickly as expected, so traffic was redirected to the new cluster (V2). Only later did the team learn that increasing cluster size actually made the problem worse.

Investigation ruled out routine backups or snapshots, sudden increases in Redis command usage, and problematic commands (slow queries, large keys). During the second incident at 1:59 AM PDT, a manual failover worked, restoring service faster. The team black-holed some of the requests waiting longest on Redis commands, but that did not help, and no scheduled background jobs correlated with the incident times.

A third incident occurred at 6:53 AM PDT. Since some non-core functionality had already been disabled, the team could rule that out as a cause. More features were disabled, and a fourth incident struck at 9:36 AM PDT. This time, the team could not fail over because AWS rate-limits manual failovers per cluster. While waiting for AWS to raise the limit, Figma migrated traffic to a new cluster (V3). By the time the limit was raised, the migration was complete.

The V2 cluster—still pegged at 100% CPU but no longer serving core traffic—became a test environment. By disabling the small set of services still connected to it one by one, the team localized the problem to a small set of Redis usages and suspected Pub/Sub, though without definitive proof. Those workloads were isolated and disabled, causing relatively minor user impact.

Figma then partitioned the original cluster into multiple separate clusters, spreading Redis usage across them. Changes were tested in staging before production rollout. At 11:35 PM PDT—exactly 24 hours after the first incident and a period of expected peak utilization—no recurrence occurred. The next morning, AWS’s ElastiCache team confirmed the Pub/Sub bug and Figma moved all Pub/Sub workloads off the clustered ElastiCache instance that same afternoon.

Root Cause and Contributing Factors

The disruption originated from a bug in AWS ElastiCache with Cluster Mode enabled. Under high Publish/Subscribe (Pub/Sub) workloads, the service could enter a state where Engine CPU utilization experienced a sudden and sustained spike. Because the trigger condition was specific and complex, Figma ran Pub/Sub traffic on a Clustered-Mode Enabled instance for weeks without incident before the failure surfaced at 11:30 PM PDT on June 6.

The response to the initial incident inadvertently made things worse. Figma failed over to a new ElastiCache cluster with greater capacity. That larger footprint increased the number of nodes the Cluster Bus had to coordinate, which lowered the Pub/Sub activity threshold required to re-trigger the same AWS bug. The result was a cycle of repeated incidents on the enlarged cluster until the team isolated the Pub/Sub workload onto its own dedicated Redis instance.

The issue is now considered resolved. Figma has moved all Pub/Sub workloads back onto an ElastiCache instance with Cluster Mode disabled, which is not susceptible to this particular defect.

Architectural Response

The incident prompted a permanent change in how Figma maps workloads to ElastiCache. The major use cases now reside on separate ElastiCache instances. This separation delivers two operational benefits: better fault isolation, allowing selective feature disabling during emergencies, and clearer visibility into traffic and performance shifts for each distinct workload.

Resilience Improvements

Figma has outlined a concrete set of follow-up actions to reduce the likelihood and impact of similar incidents in the future.

  • Hardening the core user workflows in Figma against unexpected ElastiCache failures.
  • Expanding tooling to quickly enable or disable non-core functionality when a partial failure is ongoing.
  • Scheduling regular Disaster Recovery Training (DRT) drills to ensure core workflows remain operational when ElastiCache is completely unavailable.
  • Improving the internal libraries that communicate with ElastiCache instances to better handle edge-case failures.
  • Partnering with AWS to review incident response procedures and accelerate identification of service-side defects in the future.