When a connection pooler becomes the bottleneck

Figma’s Postgres fleet sits behind a layered stack that has evolved alongside the product. DBProxy, a request routing system, handles horizontal and vertical sharding by parsing each query, picking the right Postgres instances, and rewriting the query accordingly. The request then passes through a connection pooler that manages the fixed-size connection pool for each database machine. Until recently, that pooler was PgBouncer, with a dedicated set of pooler replicas serving each Postgres node in an n-to-1 arrangement.

That setup worked for years, but growth in both features and user traffic exposed four structural problems with PgBouncer:

  • Vertical scalability limits. PgBouncer’s single-threaded design caps how much one instance can handle. Adding replicas helped, but load distribution skew degraded performance as the fleet grew.
  • No load prioritization. There was no way to shield critical traffic from lower-priority requests, nor any built-in backpressure or support for algorithms like CoDel to shed work gracefully under bursts.
  • Weak connection safeguards. Postgres connections are expensive, and PgBouncer offers no protection against rapid connection creation or churn. After an overload incident, naive recovery efforts could themselves trigger prolonged connection churn that cascades across Postgres nodes.
  • Hard to extend. As the pooler became more central, we needed deep observability, feature-flag-driven rollouts, admission control, and fair resource sharing per traffic type. PgBouncer was not built for that kind of fine-grained shaping, and even small patches carried significant maintenance weight.

Why not just extend an existing proxy?

Embedding connection pooling into DBProxy was tempting but impractical. A typical Postgres instance has a connection pool sized around 100 connections, while DBProxy runs hundreds of stateless replicas. Distributing one small fixed pool across that many replicas either blows past the connection limit or demands complex cross-replica coordination.

PGCat, a modern multi-threaded PostgreSQL proxy, solved the vertical scalability problem but not the extensibility one. Adding the observability, feature flags, and admission control we needed would require deep changes to its core execution paths — the kind of changes unlikely to be accepted upstream, which would mean maintaining a fork indefinitely.

So we built our own: PGKeeper, a Go service that sits between DBProxy and Postgres. The name reflects its role as a goalkeeper — blocking bad traffic before it overloads the database and preventing connections from churning.

PGKeeper deliberately departs from PgBouncer and PGCat by exposing a gRPC interface. Each query is an independent request carrying metadata — traffic tier, user type, request source — that PGKeeper uses to decide how to route or shed it. The service is built on PGX, a mature Go PostgreSQL toolkit that provides connection management and protocol primitives without dictating higher-level behavior, leaving room for custom load management and admission control logic.

Go was the natural choice given our engineering team’s expertise and the broader infrastructure ecosystem. Its concurrency model and efficient resource usage delivered the vertical scalability that PgBouncer lacked, right out of the box.

Connection lifecycle management

PostgreSQL's process-per-connection model means every client connection carries its own memory allocation, session state, and query plan cache. Connections are expensive to create and inherently stateful, making them a resource worth protecting. PGKeeper manages connection lifecycles through a set of mechanisms designed to minimize churn and prevent the database from becoming the bottleneck during recovery events.

Pool warming proactively establishes connections before production traffic is routed to a new instance, eliminating the cold-start latency that would otherwise accompany the first requests. For steady-state operation, that is sufficient. But a massive connection churn event can leave the pool partially hydrated, and without guardrails, the re-creation process could itself trigger another overload cycle.

PGKeeper rate-limits connection creation through a token-bucket mechanism provided by the Go library bradenaw/backpressure. New connections are only established when a creation token is available, spreading the cost of pool growth over time. This prevents bursts of connection creation from overwhelming Postgres, particularly during recovery from churn events, where simultaneous query replanning across many new connections can saturate CPU and prolong overload. The same principle applies to teardown: PGKeeper rate-limits connection destruction as well, ensuring pool shutdown does not destabilize the database host.

In addition to rate limiting, PGKeeper implements three mechanisms to ensure connections are cleaned up and reused rather than discarded:

  1. Bounded exhaust: When a client does not fully consume a result set, the connection may retain server-side state such as open cursors or buffered rows. PGKeeper drains up to 100 remaining rows before returning the connection to the pool. This rescues the vast majority of incomplete reads (P99 of Figma's queries return fewer than 5 rows) while avoiding unbounded work on outliers.
  2. Auto rollback: If a connection is released while a transaction is still in progress, PGKeeper automatically issues a ROLLBACK before returning the connection to the pool, preventing uncommitted transactional state from making the connection unsafe to reuse.
  3. Context cancellation handling: The Postgres wire protocol has no native mechanism for a client to signal disinterest in an in-flight query. PGKeeper accepts a context from the client through its gRPC interface and translates cancellation into Postgres-native operations. Because P95 of queries complete within 2 ms, PGKeeper waits briefly before initiating cancellation, avoiding unnecessary overhead for queries likely to finish on their own. If the query does not complete within that window, PGKeeper issues a pg_cancel_backend() call through a dedicated cancel-only pool, keeping cancellations from monopolizing the main pool. Since the return of that call only guarantees the kill signal has been enqueued, PGKeeper runs a lightweight SELECT 1 to force the target backend to handle the signal before the connection undergoes bounded exhaust and auto rollback.

These mechanisms were born from operational incidents at Figma, including a mass connection closure during an upgrade from PostgreSQL 13.21 to 13.22 that drove significant CPU saturation. Since deploying this connection management approach, Figma has not experienced a single massive connection churn incident.

Admission control design principles

At scale, overload is inevitable. The system will occasionally receive more demand than it can serve within a reasonable time. Blindly admitting requests leads to a vicious cycle where queues grow, latency spikes, and failure cascades across unrelated workloads. Indiscriminate rejection wastes work and degrades the end-user experience.

Figma aligned on four design principles for admission control in PGKeeper:

  • Optimize for the end-user experience: Actions like opening a file, loading a document, or saving edits should remain highly available during overload, while asynchronous batch processing jobs can tolerate temporary degradation.
  • Solve for concurrency-based load: The most frequent source of overload at Figma is an unexpected spike in the volume of concurrent requests.
  • Be adaptive and dynamic: Traffic patterns fluctuate greatly, so static thresholds or manual adjustments are not viable. Mechanisms should be self-balancing.
  • Make decisions fast: Admission control runs in the critical path of every database request, so latency overhead must be negligible.

V1: Priority-based admission control

The first implementation used a semaphore to bound the maximum number of concurrent connections the database serves. When a request arrives, PGKeeper attempts to acquire a slot. If capacity is unavailable, the system must choose between immediately rejecting the request or queuing it.

Immediate rejection is often inefficient—clients are likely to retry soon, so the system pays for the rejection and then again for the retry. Queuing is a more graceful response to short-term spikes, allowing the system to hold excess work while capacity recovers. But queuing has its own failure mode, previously observed with PgBouncer: during sustained overload, the queue fills faster than capacity can drain it, and requests sit long enough that the work they represent stops mattering. A user waiting on a slow page load hits refresh, leaving stale requests in the queue that the system struggles to clear.

PGKeeper adopts the CoDel algorithm combined with adaptive LIFO scheduling to address this. In a healthy state, the queue is usually empty and requests process in standard FIFO order. When the queue remains non-empty for an extended duration during heavy traffic, the system is defined as overloaded. It then switches processing order to LIFO and permits shorter queue residency times, aggressively shedding older backlog requests so they are dropped more quickly than usual.

Priority ordering is enforced through a self-adjusting mechanism called debt, provided by the bradenaw/backpressure library. All traffic at Figma is tagged with a priority, with higher priorities corresponding to critical user flows and lower priorities representing background jobs. Whenever a higher-priority request cannot immediately acquire a semaphore ticket, the system assigns debt to all lower-priority traffic tiers. A lower-priority request can only acquire a token if there is enough capacity for both the request and the accumulated debt against its tier. Debt decays over time and pays down faster when higher-priority traffic is succeeding. This approach favors higher-priority requests during overload without permanently allocating capacity to any traffic class. In production, high-priority traffic remained available during overload incidents while lower-priority requests were shed and mostly retried successfully.

V2: Multi-dimensional fair sharing

The priority-based semaphore left a gap: a single dominant workload could consume the majority of available concurrency within its priority tier and starve out traffic at the same level. An unexpected popular feature or a DDoS attack could exhaust the high-priority budget before admission control rebalanced resources.

Real traffic varies along multiple axes—authentication status, request source, client, user type, and more. PGKeeper implements a weighted min-max fair share algorithm that allocates capacity proportionally across these categories. The resource is database concurrency capacity and the consumers are categories of traffic, such as "authenticated traffic to the comments endpoint" or "unauthenticated traffic to the search endpoint." These categories are represented as a tree, with each level corresponding to a different dimension. Capacity is allocated from the root downward, divided across child categories according to their weights, recursively through the tree.

Fair sharing works by giving each consumer a baseline allocation. Consumers who do not need their full share release the rest into a common pool; consumers who need more than their share can draw from that pool up to a limit. No one starves, and no one hogs the resource.

Every request passes through both controllers in sequence. Priority admission control first uses the semaphore-with-debt mechanism to favor high-priority traffic under contention. If the request clears that, it passes through the fair-sharing tree, which checks whether its category has room within its allocation. Both controllers must admit the request for it to proceed.

Client feedback loop

Admission control improves system stability, but rejection alone cannot prevent overload at sufficiently high request rates. At some point, the only way to prevent sustained pressure is for clients to send less traffic. PGKeeper returns informative error codes indicating overload rejections, and provides an AdaptiveThrottle package that clients can use to back off when encountering these errors, paired with exponential backoff and retry strategies. This feedback loop allows the server and clients to converge toward a stable request rate without central coordination.

Known limitations

PGKeeper does not eliminate every failure mode. Three classes of problems remain:

  • Expensive queries: Admission control primarily protects against excessive concurrency, not the cost of individual queries. An unexpected sequential scan on a large table or a poor query plan can consume significant CPU or I/O even when concurrency is low. Figma mitigates this through an adjacent project, Guardrails, which flags inefficient queries at CI time.
  • Skewed load distribution: Per-instance load management only works if traffic is evenly distributed across PGKeeper pods. Load balancing is rarely perfect. Current mitigation places replicas behind an NLB with clients establishing multiple gRPC connections; a service mesh would provide stronger guarantees.
  • Small connection pools: Multi-dimensional fair sharing needs enough total concurrency to slice across categories. With a small pool, fairness becomes coarse and admission control must fall back on simpler prioritization.

De-risking the migration

Moving production traffic off PgBouncer without compromising performance or reliability demanded a staged rollout plan and explicit disaster-scenario testing.

Testing against known failure modes

Two types of disaster readiness tests were run before production traffic was shifted.

  1. Load testing: PGKeeper was tested at 3× peak production QPS to validate its vertical scalability and to measure the latency overhead introduced by the gRPC layer relative to raw PgBouncer. The added overhead was sub-millisecond, an acceptable cost for the gains in load management and observability.
  2. Synthetic load generators: Since not every production traffic pattern could be predicted in advance, past incidents were replayed as synthetic load generators against PGKeeper. Parameters and algorithm logic were tuned until the system could withstand each of the historical failure scenarios.

Bounding the blast radius

A flawed rollout could have disrupted database availability across the entire platform, so deployment ordering was designed to limit the blast radius of any given failure. Database instance groups were ranked by criticality, and rollout proceeded from the lowest-risk to the highest-risk groups.

The process started with a single replica in each cluster, ordered by criticality. This was inherently low-risk because DBProxy hedges replica requests—if the replica running PGKeeper encountered problems, traffic could still be served by the other replicas still connected to PgBouncer. Coverage was then gradually expanded across all replicas, providing a safe opportunity to validate PGKeeper's behavior under live production traffic before tackling riskier configurations. Only after all replicas were migrated did the rollout move on to non-critical primaries, and finally to all primaries.

Automatic failback

A human operator alone could not react quickly enough to every problem, so a sliding window error detector was built into the DBProxy layer. This detector continuously evaluated the error profile of recent traffic and, if the error rate stayed above a threshold for a sustained number of windows, DBProxy automatically switched traffic back to PgBouncer. The value of this mechanism showed up early: during the first rollout phases it triggered a few times, preventing what would otherwise have become availability incidents.

Flipping all traffic back at once does send a sudden surge of new connections at PgBouncer, which carries its own risk of instability. However, a brief period of unavailability during the switchback was judged preferable to an extended outage caused by a malfunctioning PGKeeper continuing to serve traffic.

PGKeeper in production

Since the full rollout, Figma's database SLO, as measured against core user experience availability, has remained above 99.99%. In Q4 2025 alone, PGKeeper is credited with preventing more than 20 incidents that would otherwise have resulted in user-visible outages.

What began as an effort to modernize a legacy connection pooler has become a foundational piece of Figma's database infrastructure, now serving as the primary defensive layer between applications and PostgreSQL. Owning this layer brings with it ongoing maintenance, on-call duties, and operational overhead—costs that have been outweighed by the improvements in reliability, observability, and control.