One Tier, Many Benefits: How ZGateway Reorients ZippyDB Traffic
ZippyDB, Meta’s most widely used key value store, handles billions of operations per second from a client fleet numbering over a million hosts. Those hosts belong to hundreds of different teams, making any client-side change slow and painful. ZGateway, a stateless proxy tier, offers a different approach: instead of trying to update countless clients, it sits directly in the traffic path, providing a single point of control that individual clients never could.
The proxy’s value is structural. It collapses a dense network of direct client-to-server connections into two manageable hops, and its position in the stream enables capabilities like admission control, load balancing, cross-region resilience, and request batching that are impractical to build into a million separate binaries. The tradeoff is an extra hop and another tier to operate, but when the client population is numerous, varied, and outside your direct control, that trade pays off handsomely.
The Connective-Tissue Problem
In the original direct-access model, every client established a connection to every database host it needed—potentially tens of thousands of distinct shards spread across hundreds of thousands of hosts. This created an extremely dense mesh of TLS connections. A typical client held tens of thousands of outbound connections, while a database host accepted tens of thousands of inbound ones.
This mesh consumed memory, CPU, and file descriptors on both sides, mostly while idle. It also worsened with scale: since each new client cohort increased the inbound connection count on every database host, a sudden drop in connection reuse—triggered by a restart or deploy—could create a storm of new connections that crashed hosts through file-descriptor exhaustion and OOMs. One routing bug caused every client to open a connection per shard, pushing hosts past their limits and sending the fleet into a reboot loop.
Fixing this on the client side proved nearly impossible because two independent systems were moving at different speeds: client fleets kept expanding and changing policies while the database fleet consolidated on its own schedule. The proxy decouples them. When ZGateway sits in the path, a reconnection storm is contained within a fleet that operators control, observe, and can harden centrally. Direct access made sense in ZippyDB’s early days when the client population was small, but at scale, the resource overhead and reliability risks became unacceptable limits.
How the Gateway Operates
ZGateway runs as regional tiers discovered through ServiceRouter, Meta’s hyperscale service mesh. It handles more than 1 billion operations per second, carries about 40% of all ZippyDB traffic (projected to pass 60%), and adds only about 6% computational overhead. Two flavors share one pipeline: a pure proxy and a read-through cache. Both are built on ZippyDB’s thick C++ client used as a managed server-side engine.
A client sends a request over a sticky connection to a regional gateway host. That host terminates TLS, checks the request against the use case’s ACLs, applies admission control and shaping, resolves the shard, and—on caching tiers—consults the local cache. Misses and writes are batched and coalesced with other in-flight requests for that shard before routing to the correct replicas. Responses flow back to the original callers, with traces, metrics, and quota usage recorded en route.
The defining property is the asymmetry of connection counts. Clients need only a small sticky pool to their regional gateway, and each ZServer sees connections only from the gateway fleet—a size that operators control. Some responsibilities intentionally stay in their existing places: TLS handling persists within the Thrift/ServiceRouter stack, key-to-shard mapping stays in the shard locator, and replica selection remains in the embedded client. ZGateway owns traffic management rather than duplicating the database client’s core logic.
The Scaling Arithmetic
The reduction in connections follows from a straightforward balls-into-bins model. If a host touches B shards across H hosts, the expected number of distinct bins (fan-out) is given by:
A given server is hit with a probability of:
With approximate fleet numbers—20 regions, 500,000 database hosts, 30,000 proxy hosts, 1,000,000 clients, 50,000 shards per client—the model projects the following:
The connections don’t simply vanish; they migrate to the tier that specializes in handling them. End-to-end total persistent connections still drop by roughly 19x, because each backend connection multiplexes many clients.
The permanent win, though, is the change in growth dynamics. Under direct access, fan-in scales linearly with the client population, so every new cohort degrades every database host. With ZGateway, that population term disappears completely. Fan-in now depends only on regions times shard density per host, a quantity the operators own. An unbounded number driven by external teams becomes a bounded, internally controlled figure.
Merging and De-Duplicating Requests
Since ZGateway intercepts traffic from many unrelated clients, it can combine work that no client-side library ever could. A shared batcher on each host groups requests by use case and physical shard, merging them into single backend RPCs instead of many individual ones.
Coalescing goes one step further: if multiple callers need the same key at nearly the same instant, the gateway fetches the value once and distributes the result to everyone. Client-side batching is fundamentally limited to a single process; the gateway collapses cross-client demand.
The efficiency gains compound quickly. Every RPC carries fixed overhead for serialization, authorization, and syscalls regardless of payload size, so larger batched requests amortize that cost across more operations. Fewer backend requests mean lower QPS and CPU usage, and a linger window smooths out micro-bursts into steady traffic. Since use cases are billed by the QPS they generate, batching stretches their rate-limit allowances and reduces throttling.
Two less obvious benefits matter just as much. First, coalescing neutralizes hot-key stampedes: thousands of simultaneous reads become one backend fetch. Second, it allows removal of fragile, CPU-hungry client-side batching libraries that had accumulated over the years and caused recurring incidents because their logic lived in a million-strong binary fleet that ZGateway’s operators could not control. A shared batcher serves the same purpose more reliably and lets teams retire their custom code.
Batching carries inherent risk: holding requests in memory invites OOM conditions. The design includes two safeguards. An idle-eviction mechanism removes batch-map entries that have been empty past a TTL, protecting against slow-memory-growth scenarios. An in-flight cap addresses acute overload—when the backend slows down, coroutines for flushed batches pile up faster than they drain, so the cap rejects new executions once a threshold is crossed. Steady hygiene combined with an acute safety valve is what allows batching to remain safe by default.
The Proxy Tier as a Shared Service
Once traffic funnels through a single tier, that tier becomes the natural place to implement capabilities that every client would otherwise build separately. Beyond batching, ZGateway has grown several such features, each designed around safe operation at scale.
Controlled Rollouts and Tenant Isolation
Moving traffic onto a proxy is inherently risky, so the migration path must be incremental, reversible, and narrowly scoped. Client-side configuration flags control routing to ZGateway, scoped per service and shard prefix. A percentage knob ramps eligible traffic, a region filter limits blast radius, and a global kill switch provides instant rollback. Since this is pure configuration, no client code changes are required, keeping the rollout controllable in real time.
A shared tier serving hundreds of use cases needs strong isolation so one misbehaving tenant cannot starve others. ZGateway relies on Discriminant Load Shedding (DLS) for this. Each request maps to a per-tenant bucket, keyed by use case and split by priority, with buckets draining round-robin. When a tenant floods the tier, its bucket fills and excess requests are shed while other buckets continue draining normally. Isolation is thus a structural property, not a matter of luck. In front of DLS, a CPU concurrency controller uses an AIMD loop to regulate the shared token bucket's admission rate, and a memory handler similarly guards against out-of-memory conditions.
The shedding remains discriminant in practice. In a controlled overload at above 90% CPU across roughly 1,350 active tenant buckets, only six — the actual noisy neighbors — were shedding. The other ~1,344 buckets executed 99.9% of their requests with zero rejections, goodput held near 97–98%, and the machinery cost about 8% of CPU.
Caching, Load Balancing, and Cross-Region Failover
Hot reads on the cache tier are served from an in-process cache. On a miss, the gateway takes a per-key fill lock so a thundering herd for one key collapses into a single backend fetch. Freshness is maintained by a change-data-capture stream of write and checkpoint events that invalidates or refills affected entries, operating within an explicit bounded-staleness contract. Each host owns a slice of the keyspace via consistent hashing. The result is substantial read offload from storage at lower latency, without sacrificing correctness.
Because ZGateway is stateless, any request can be served by any host in a regional tier, making it possible to steer traffic for even load distribution. The tier is not uniform: it mixes hosts from roughly 26-core to 126-core machines, and a large task replacement can reshuffle capacity within minutes. Treating unequal hosts equally produces hot outliers, and a hot ZGateway host translates into error-rate spikes and ServiceRouter throttling. Since ServiceRouter routes by weighted consistent hashing, the lever is choosing the right weight per host.
A control-plane balancer computes these weights on a fixed cadence. It reads each host's recent CPU utilization, normalizes the tier average to 1.0, and nudges each weight opposite to its load. Guardrails keep the system stable: adjustments are damped and clamped, the distribution is recentered on a target median so weights don't drift toward zero, and a change throttle moves only the most imbalanced hosts per run, limiting cost shard reshuffling — expensive on cache tiers, where moving a weight means moving keys. New hosts start with weights scaled to hardware capacity.
A single fixed policy cannot serve both a calm tier and one in shock, so the balancer is becoming adaptive. It classifies each tier's state — steady drift, task churn, flat initial weights, bimodal load, hot outliers, regional skew — and applies a matching policy accordingly.
For most of its life, ZGateway was strictly regional, with failover occurring only within a region. That approach is great for latency but leaves healthy capacity idle next door when an entire region's tier comes under pressure, forcing requests to queue and time out locally. Since ZGateway sits on ServiceRouter, routing can now cross region boundaries in controlled ways via three mechanisms:
- Global routing builds a routing table spanning regions, so a saturated local tier fails over to a healthy one.
- Mega-regions group geographically close regions into one locality, so overflow spills nearby and retains most of the latency benefit.
- Rings declare exactly which regions back each other up, and in what proportion.
Each mechanism is enabled per tier and region behind a percentage knob. The failover signal itself matters as much as the routing: a simple regional CPU average smooths over exactly the hot conditions that need catching, so failover keys off a sharper measure tuned to fire before a region tips into overload.
Transactions Move to the Gateway
Transactions require client-side bookkeeping — read sets, scanned ranges, pending writes — which historically lived in the thick client. When ZGateway moved customers onto a thin client, that bookkeeping had to move onto the gateway itself. The first implementation left two parallel code paths: a bespoke store built for ZGateway alongside the in-memory path the engine already used. Maintaining two versions of the most correctness-critical part of the flow was untenable.
The consolidation ran behind a flag in nine phases up to the highest-volume regions, eventually reaching 100% of transaction traffic with no reliability regression. Sharing that path with the engine keeps ZGateway in lock-step with server-side transaction evolution: a capability is evolved once, inside the tier, and every client inherits it.
Running the Tier in Production
ZGateway runs as a large volume of servers across dozens of regions, organized into a handful of tiers by workload. There is one large general-purpose tier for the long tail of use cases, dedicated tiers for the largest customers, and a separate high-throughput proxy tier. These tiers differ in footprint and size by more than an order of magnitude, and individual tiers are not uniform either, largely due to stacking — multiple tasks packed onto one machine at varying densities alongside full-size dedicated hosts. Across all of this, ZGateway exposes rich per-use-case observability, the visibility that makes the admission control and load balancing described above safe on shared infrastructure.
Toward a Programmable Gateway
The near-term goal — unifying all ZippyDB traffic through ZGateway — is unchanged. The more interesting question is what a universally adopted gateway makes possible. Three directions stand out, each sharing a common theme: ZGateway both sees the most and decides the most.
Agent-operated heuristics. Almost every capability described here is governed by a control loop and hand-tuned knobs: load-shedding bucket sizes and CPU thresholds, balancer parameters, failover triggers, batch flush windows, cache staleness bounds. Today those are tuned by humans and nudged by crons; the adaptive balancer is already an agent in all but name. The next step is to make this explicit, exposing heuristics and internal state as a structured control surface. AI agents could then watch the same telemetry that operators do — diagnosing tier state, attributing an incident to a noisy tenant, and applying remediation behind guardrails faster than any oncall engineer.
Co-location. ZGateway is a distinct tier, which costs an extra network hop and a few percent of overhead. For latency- or efficiency-critical workloads, a portion of the gateway could be pushed down beside the ZServer host, making the gateway↔server leg a local call while the control plane stays central. The challenge is doing this without re-coupling the fleets that were deliberately decoupled. The connection-management and admission-control front would remain a shared regional tier, with only what benefits from data locality moving down.
A multi-process gateway. ZGateway currently runs many distinct responsibilities in one process, so one tenant's memory blowup can threaten everything on the host. Splitting it into cooperating processes — a connection/TLS front-end, request workers, separate cache and transaction components — buys hard fault isolation and an independent lifecycle. This also complements the other two directions: agents could manage the process fleet on a host, and co-location becomes cleaner when the data plane is already its own placeable process.
Taken together, these directions turn ZGateway from a smart tier into a programmable one, where control decisions are made by agents, the footprint moves to where the work is, and failure domains are isolated by construction.
