Postgres Under Fire: Multiplexing Load Shedding at Cloudflare
Every database cluster running a multi-tenant OLTP workload eventually hits the same wall: one tenant’s traffic spikes and everyone else’s queries crawl. Cloudflare operates production Postgres clusters across multiple regions, using Stolon for high-availability management and failover. In front of Postgres sit HAProxy for load balancing across primaries and replicas, and PgBouncer as the connection pooler that hands out a finite pool of server-side connections to tenants. Because these clusters run on bare metal without containerization, every tenant contends for the same finite machine resources—CPU time, memory, disk IO—as well as database-level resources like Postgres connections and table locks.
The failure modes are familiar. A tenant may fire a burst of short transactions, starving neighboring tenants of CPU or disk IO. Or a tenant may submit long, expensive queries—large table scans for an ETL job, queries holding extended table locks—that either degrade query execution for others or cause transactions to hang. With all available PgBouncer server connections occupied by a misbehaving tenant, innocent tenants can even be locked out of obtaining new database connections entirely.
The Detection Problem
When cluster load spikes, the first job is attribution: which tenant is responsible? With no static global cap that fits every workload, operators historically had to dig through pg_stat_activity, comparing each tenant’s query activity under normal circumstances to hunt for newly introduced expensive queries. It’s an analysis that requires production SQL experience and is stressful to perform live during an incident.
Once a tenant was identified, the fix was to manually enforce a Postgres-level connection cap with a query like the following:
ALTER USER "some_bad-user" WITH CONNECTION LIMIT 123;This “connection squeezing” restricts a single user’s concurrent throughput to their share of connections, and it did deliver measurable load shedding during high production workloads:
Why Manual Throttling Failed
The technique worked, but it came with several drawbacks that make it unsuitable as a repeatable operational tool:
- Setting a new user limit in Postgres does not terminate existing connections, so a tenant can keep issuing bursty or expensive queries until its connections age out naturally.
- Lowering concurrency does not make an already-running expensive query—or its lock waits or disk seeks—any cheaper for everyone else sharing the machine.
- The entire process is manual toil. An SRE may be paged at any hour to apply limits mid-incident.
- Choosing an appropriate per-user connection cap is arbitrary and experimental when workloads vary, requiring deep tenant knowledge each time adjustments are needed.
- Under severe CPU starvation Postgres itself may hang, at which point applying native throttles by connecting to the database becomes impossible.
Throttling at the Gateway
Once a query reaches Postgres for execution, controlling what it consumes at the system level is largely a lost cause. Cloudflare decided instead to enforce isolation one layer up: at PgBouncer, the connection pooler that mediates all queries, where per-user and per-pool control can decouple tenant admission from database connection availability.
Prior to Cloudflare’s work, PgBouncer’s user-level connection limits only prevented the pool from growing past a threshold—they never invalidated excess existing connections. The team’s fork changed that, adding support for eviction and throttling of live connections per user or per pool, either set statically at startup or injected at runtime.
Configuration
[users]
dns_service_user = max_user_connections=60
firewall_service_user = max_user_connections=80
[pools]
user1.database1 = pool_size=90Runtime Operational Commands
SET USER dns_service_user = ‘max_user_connections=40’;
SET POOL dns_service_user.dns_db = ‘pool_size=30’;Supporting these features required heavy refactoring and bug fixes in the fork. The team has raised multiple pull requests to upstream the changes to the PgBouncer open-source project. The key operational outcome: an operator or an automated system can shed load granularly against a misbehaving tenant’s connection pool immediately, without relying on manual Postgres analysis or accepting the lag before old connections die, yielding much stricter performance isolation between tenants under burst conditions.
Automating Tenant Throttling
Beyond policy-based controls, the next step is infrastructure that continuously monitors per-tenant resource consumption and flags misbehaving tenants by comparing system indicators against historical baselines. The goal is to automate connection and query throttling using new administrative commands. Several experimental approaches are being explored to enforce strict performance isolation.
Congestion Avoidance
One promising direction adapts the TCP Vegas congestion avoidance algorithm to estimate and enforce each tenant’s optimal concurrency. The approach requires no profiling of resource consumption, no manual threshold tuning, no knowledge of the underlying hardware, and no expensive computation. In TCP Vegas, the algorithm converges to the unknown optimal congestion window—the max packets that can be sent concurrently. We can treat that window as the optimal connection pool size for database queries.
At the gateway layer (PgBouncer), each tenant begins with a small pool. The system then samples each tenant’s transaction round trip time (RTT) against Postgres, gradually increasing the pool size so long as RTTs do not deteriorate.
When a tenant’s sampled transaction latency rises, the formula’s minimum by sampled request latency ratio decreases, which naturally reduces that tenant’s concurrency and database load.
The algorithm backs off upon observing high query latencies, treating them as a signal of high database load—whether caused by CPU, disk, or network blocking. It converges to an optimal concurrency limit because the latency ratio always trends to 0 with sufficiently large sample request latencies. The square root of the current pool size is chosen as burst headroom because it grows fast for small pools (when latencies are low) but converges as pools shrink (when latencies are high).
This congestion avoidance approach throttles traffic preventatively, before load-induced performance degradation occurs, instead of reactively shedding load. It aims to prevent resource starvation that hangs other queries. One theoretical caveat: if a single tenant misbehaves and causes latency for others, the algorithm might incorrectly throttle all tenants. It may therefore be safer to apply adaptive throttling only to tenants showing a high CPU-to-latency correlation when the system degrades.
Tenant Resource Quotas
Another approach introduces configurable resource quotas per tenant. Each upstream application service tenant is restricted to a defined share of resources, expressed as CPU percent per second and max memory. If a tenant exceeds its share, the PgBouncer gateway throttles concurrency, queries per second, and ingress bytes to force consumption back into the allocated slice.
Throttling must be isolated so it doesn’t spill over onto other tenants sharing the cluster, which could reduce availability of other applications and violate SLOs. Under low traffic, tenants should be allowed to exceed their allocation. But when cluster load degrades overall latency, the gateway must re-enforce limits. Average query latency’s rate of change against a predefined threshold can serve as an indicator of server health. All tenants should accept that a surplus in consumption may result in query throttling under any pattern.
Because each tenant’s workload is unique and variable, quick detection requires near-real-time profiling of each tenant’s (or pooled connection’s) baseline consumption on each local Postgres server. From there, baseline traffic characteristics can be correlated with system-level consumption per instance. Generalizing statistical measures across distributed nodes can be misleading due to high variance between leader and replica traffic. For example, a user should not be throttled on an idle read replica just because they overuse the primary. Tenant consumption should be captured and enforced per Postgres instance, not cluster-wide.
Multivariable regression could model the relationship between independent variables (concurrency, queries per second, ingested bytes) and dependent variables (system resource consumption). This would allow optimal independent variables to be calculated and enforced per tenant under high load. Adjusting the sliding window size—how long profiled data is retained—tunes the tradeoff between regression adaptability and accuracy as workloads change.
Gateway Query Queuing
At the gateway layer, queries can be prioritized before submission to Postgres. With one or more global priority queues, submissions by all tenants are ordered by the current resource consumption of the tenant’s connection pool or the tenant itself. Alternatively, ordering can be based on each query’s profiled historical consumption. The queue is reordered each time the scheduler forwards a query, using changes in tenant resource consumption captured from each Postgres instance.
To prevent starvation—where one tenant’s query waits indefinitely—gateway query queuing can be enabled only during peak load or traffic to the Postgres instance. Enqueue time can also factor into the priority ordering. This approach isolates performance by letting non-offending tenants continue reserving connections and running queries, including critical health checks. Higher latency is limited to tenants consuming more resources. The method is simple, generic, and non-destructive—it does not kill connections, only dropping queries when the in-memory priority queue is full.
Conclusion
Performance isolation in a multi-tenant storage environment spans OS resource management, database internals, queueing theory, congestion algorithms, and statistics. We’ll be watching for how the community tackles the noisy neighbor problem at scale.
