What went wrong on January 21–22

Figma experienced intermittent availability issues and full outages on Tuesday, January 21 and Wednesday, January 22, 2020. The first incident lasted roughly from 6:11 AM to 6:54 AM PST and was traced to a single long-running, expensive query that drove database CPU usage up. Cancelling the query restored normal performance.

Wednesday's problems were more severe and persistent. Elevated write IOPS and lock contention appeared around 9:45 AM PST, with the service becoming unavailable for some users by 11:42 AM. Cancelling inessential queries and increasing allocated IOPS provided only temporary relief. After a second deterioration at 2:00 PM PST and a database restart at 3:00 PM PST, the team took the service down for emergency maintenance at 7:00 PM PST to upgrade PostgreSQL. The upgrade completed at 8:15 PM PST and all metrics returned to normal.

Two compounding root causes

Tuesday's expensive query was a direct cause of Wednesday's difficulties, but not in the way initially suspected. The cancelled query left a vacuuming backlog that crossed the threshold for aggressive autovacuuming—a mode designed to prevent transaction ID wraparound, but one that imposes heavier locking and write loads, particularly on the PostgreSQL version in use at the time.

Autovacuum mitigation helped temporarily but did not fully resolve the issue. The team discovered that a second factor was at play: a complex ORM-generated query whose execution plan had become catastrophically inefficient. On the problematic plan, the query estimated it would return over 20 million rows when the actual result was only 3 rows. This miscalculation triggered full table scans, large writes to temporary buffers, and execution times stretching into minutes.

Investigation of a reduced reproduction of the query on the roles table revealed the trigger. Row estimates for the same query had been reasonable (around 199) on a snapshot from days earlier, but changed after an automatic ANALYZE run. Manually running ANALYZE reproduced the degradation, confirming that a routine statistics update had caused the planner to switch to the bad plan.

The precise mechanism was a deficiency in the PostgreSQL 9 query planner. When the estimated row count for an inner HashAggregate crossed a threshold of 200, the total estimate jumped wildly. The team bisected PostgreSQL versions using official Docker images: the issue replicated in 9.6.16 but not in 10.0. Release notes for 10.0 pointed to a change titled "Use uniqueness guarantees to optimize certain join types," which applied because the join involved the primary key. The upgraded database confirmed the fix was in use, with EXPLAIN VERBOSE output showing "Inner Unique: true."

The team also determined why statistics had crossed the critical threshold in the first place: PostgreSQL's ANALYZE uses a statistical sample that can underestimate the number of unique values in a column when identical values cluster together.

Why the upgrade resolved it

PostgreSQL 11 addressed both contributing factors:

  • The query planner now uses uniqueness guarantees to avoid the problematic plan entirely.
  • Autovacuuming performance characteristics improved significantly between versions 9 and 11.

The team had prepared for this scenario in advance. Staging had been running PostgreSQL 11 for several months, and a detailed production upgrade plan existed. The decision to upgrade during the incident, rather than waiting for the weekend, was based on the risk of ongoing instability.

Prevention and follow-up

The team is taking steps to avoid a repeat of these incidents:

  • Improving monitoring of expensive queries and setting tighter bounds on allowed running time, which would have prevented Tuesday's long-running query.
  • PostgreSQL 10 and later, available through Amazon RDS, offer more sophisticated performance analysis tools for faster diagnosis.
  • Reviewing and optimizing the complex query and similar ones generated by the ORM layer.

The findings and conclusions from this investigation are documented internally to inform ongoing reliability work.

Follow-Up Work

The incidents surfaced several areas where Figma’s infrastructure and tooling need strengthening. Work is already underway on a set of improvements aimed at reducing the risk of similar failures and speeding up diagnosis when they do occur.

  • Revisiting transaction timeouts and limits so that long-running and expensive queries are terminated before they can cause widespread issues.
  • Improving metrics, monitoring, and database settings related to autovacuuming, locking, and query performance.
  • Auditing queries for unnecessary or excessive joins and subqueries that could trip up the query planner.
  • Integrating the new performance tools available in the upgraded version of PostgreSQL into incident response plans; this will help isolate problematic queries faster if a future event occurs.
  • Auditing the structure and access patterns of various types of data currently stored in PostgreSQL. Some data may benefit from being moved into datastores tailored to their access patterns.
  • Improving in-app messaging around unexpected maintenance windows.
  • Moving towards online database upgrades that don’t require future downtime.

Beyond infrastructure, Figma is revisiting product priorities and front-loading some project work that was planned for later. The first initiative in this area is ensuring users can continue to work on already-open files safely after going offline, even if they close the browser before coming back online.

More details on that product work will be shared soon. Questions about the service disruption can be directed to [email protected].