PgBouncer Fork Addresses Authentication Bugs, Adds Runtime Throttling
Cloudflare runs highly available Postgres production clusters across multiple data centers, supporting transactional workloads for services including its DNS Resolver, Firewall, and DDoS Protection. At the front of the gateway layer for each cluster sit multiple PgBouncer instances acting as TCP proxies for Postgres connection pooling. This pooling lets upstream applications connect without constantly opening and closing expensive database-level connections, while reducing the total number of Postgres connections used. Tenants acquire client-side connections from PgBouncer rather than from Postgres directly.
PgBouncer maintains a maximum pool of server-side connections to Postgres, allocating them across tenants to prevent connection starvation. Backend queries are forwarded from PgBouncer to HAProxy, which load balances across Postgres primaries and read replicas. Because the Postgres infrastructure runs in non-containerized bare metal environments, multi-tenant resource contention between Postgres users is a concern. To enforce stricter tenant performance isolation at the database level—covering CPU time, memory consumption, and disk IO—Cloudflare needed to configure and enforce per-user and per-pool connection limits at PgBouncer. Achieving this required adding features and fixing bugs in PgBouncer, and the resulting fork is now being open sourced.
Authentication Bug Blocked User-Level Limits
PgBouncer offers configuration options to enforce per-user server connection pool size limits (effective concurrency). However, an upstream authentication bug prevented these features from working correctly when Postgres was set to use HBA authentication. Administrators relying on server-side authentication could not take advantage of the user-level features. The issue has been independently reported by others in the open-source community:
Root Cause and Fix
When proxying queries from a client connection to a Postgres server connection, PgBouncer needs the user's Postgres password. On first login, PgBouncer compares the provided password against the one defined in userlist.txt (the auth_file). If the user is not in that file, PgBouncer fetches the password from the Postgres pg_shadow system view, and that password is then used for subsequent forwarding of queries to Postgres. The same applies when Postgres is configured with HBA authentication.
After debugging with GDB, the cause emerged: multiple user objects are typically created for a single real user—once from parsing the [users] configuration section and again upon the user's first login. Users requiring a shadow auth query were stored under their respective database struct instance, while users with passwords in userlist.txt were stored globally. Because a non-authenticated user already existed in memory from the [users] section, PgBouncer assumed the user was defined in userlist.txt and skipped the shadow authentication query. The user's password was never fetched or set at first login, resulting in an empty password and authentication failures for subsequent queries at Postgres.
The fix simplifies user storage: all users are now stored globally in a single location, rather than splitting different authentication types across per-database and global stores. Instead of assuming authentication based on mere existence, PgBouncer tracks whether a user requires authentication via the auth query or from userlist.txt, based on how the user was created.
Runtime Connection Throttling and Per-Pool Limits
Beyond the bug fix, the fork adds features for stricter tenant performance isolation. Previously, PgBouncer only prevented tenants from exceeding preconfigured limits—too late once a user is already misbehaving or has consumed too many connections. The updated PgBouncer now supports enforcing or shrinking per-user connection pool limits at runtime, useful for throttling tenants issuing bursts of expensive queries or hogging connections from others. New administrative commands allow throttling maximum connections per user or per pool at runtime.
PgBouncer also supports statically configuring and dynamically enforcing connection limits per individual connection pool. This enables granular throttling of a tenant's misbehaving pool without reducing availability on its other, healthy pools.
Configuration and Runtime Commands
[users]
dns_service_user = max_user_connections=60
firewall_service_user = max_user_connections=80
[pools]
user1.database1 = pool_size=90SET USER dns_service_user = ‘max_user_connections=40’;
SET POOL dns_service_user.dns_db = ‘pool_size=30’;Implementing these features required major refactoring in how PgBouncer stores users, manages weakly referenced and stored passwords for different users, and enforces killing of server-side connections still in use.
Open Source Availability
The Cloudflare fork is available on GitHub for community use and contribution.
