Geo Key Manager at Scale: When Geography Meets Distributed Systems
Launching in 2017, Geo Key Manager lets Cloudflare customers dictate where their TLS private keys are stored. A US customer can restrict keys to US data centers only. When a request arrives at, say, a Tokyo data center that lacks the key, that edge location contacts a US facility to establish the TLS session. The Tokyo data center then serves subsequent requests from the established session. The service has grown from a research project into a core part of our infrastructure, presenting new networking challenges that test the limits of our original architecture.
The foundation of Geo Key Manager is built on Keyless SSL, an early Cloudflare innovation for keeping private keys on customer infrastructure. Two cryptographic techniques—identity-based and broadcast encryption—enable identity schemes based on geography. Keyless was originally intended for external, customer-hosted keyservers. But eventually, all TLS terminations at Cloudflare switched to an internal version, and Keyless began handling Geo Key Manager requests.
Traffic Trouble in Melbourne
In late March, we started seeing TLS handshake failures in Melbourne, Australia. Initially, we rerouted traffic via anycast to adjacent data centers, but the failures followed to Adelaide and Perth. A spike in timeouts led us to the first service in our TLS termination stack—a process that handles all key-less parts of the TLS handshake. Since this is an Internet-facing process, we aim to keep sensitive information out of it. It forwards signing requests to keynotto, a Rust service built to handle RSA and ECDSA signatures.
Using distributed tracing via the Jaeger UI, we identified a single zone generating massive API traffic. This zone also had Geo Key Manager enabled, with a policy restricting keys to US data centers. Requests routed to the nearest compliant center—San Francisco—added roughly 175ms of network travel time on top of a mere 3ms median signing time. But this explained the slow remote signatures, not why unrelated zones timed out. Something deeper was blocking our TLS termination stack, and a deeper look at the architecture revealed why.
The Life of a TLS Request
Each server runs four core components: the first service, keynotto, and an internal instance of the Keyless keyserver, gokeyless. The first service spawns a worker pool from half its CPU cores (48 workers on a 96-core machine), with each worker maintaining its own keynotto connection. keynotto spawns a thread per connection, but critically, processes all requests on a connection sequentially. Under normal load, where requests were short, this went unnoticed. But with an influx of slow, geographic-bound requests, a single slow request causes head-of-line blocking, stalling every request behind it.
When internally deployed, gokeyless was designed around worker pools based on operation type: RSA, ECDSA, and remote pool for Geo Key Manager requests. Local-only performance benefited from this design. The introduction of blocking remote requests exposed its fragility. A new Rust-based keynotto proxy now sits between the first service and gokeyless, handling ~99% of key signing operations—RSA and ECDSA—while eschewing CPU-heavy Go crypto for BoringSSL's faster implementation.
In Search of a Performance Differential
Recent FedRAMP certification forced gokeyless to switch from Go's crypto library to BoringCrypto, the same library keynotto uses. Turning keynotto off for a few large data centers allowed us to isolate language and implementation differences from library effects. Results showed only marginal benefits from the crypto library change, meaning Rust and implementation details drove the gains. With keynotto on, average maximum memory dropped to 26% and CPU consumption dropped 71%, with significant latency improvements across quantiles.
A Serial Culprit
The custom Keyless protocol, developed in 2014, communicates between services over TCP. It supports multiplexing via unique request IDs, like HTTP/2. The veteran first service and gokeyless handle out-of-order responses. keynotto, however, did not. This forced requests on the same connection to process sequentially, holding fast local jobs hostage to slow remote ones—blocking 3ms signatures behind requests taking 60 times as long.
Our initial mitigation bumped gokeyless's remote worker pool from 200 to 2,000 workers per server. But simple math showed why this couldn't help. In Melbourne's 60-server facility, we were handling 200 remote requests per second—a maximum of 10 rps on any single server, versus 200 workers idle. The real issue was keynotto's serial queue, not a shortage of remote slots.
To resolve this, keynotto moved to a concurrent processing model using a multi-producer, single-consumer queue. Requests read from a connection are handed off to new threads immediately, while a write thread per connection polls a channel for results. Careful timeout values were set—an order of magnitude above p99 latency—so stall and resource consumption quickly end. We also considered more general solutions like gRPC, which has built-in multiplexing, covering many of our custom protocol struggles.
Midwest Mesh: A Second Outage
In early June, TLS timeouts hit a large Midwest data center. This time, keynotto was fine; its head-of-line issue was solved. The problem piling up elsewhere: gokeyless timeouts on remote operations. A single EU-bound zone was now generating 80,000 remote rps—versus 200 in Australia. This traffic maxed out the entire data center's remote worker pool. We raised the workers from 200 to 20,000, and then 200,000, but utilization remained at 100% with timeouts persisting. Increasing concurrency couldn't solve the real bottleneck.
File Descriptors and Outdated Routing
The bottleneck wasn't workers—it was connectivity. Originally, every server in every facility maintained direct connections to other data centers. As the network grew, this consumed too many file descriptors. We moved to "pods" of 32 servers that share connections outward. Only one server per pod maintains active connections to the closest eligible remote data center, with second and third choices as fallbacks.
In the Midwest, roughly 80k rps from 100 servers (590 rps per server) meant only four outbound connections to Germany existed at any time, each carrying up to 1,000 times the load. Even massive worker bumps didn't address this fundamental limit.
Little's Law and the Binding Constraint
Applying Little's Law (capacity = arrival rate × service time) revealed the German side wasn't the limit. With 20k rps at ~5ms processing, Germany needed ~100 workers to handle it—it had 200. Real rps per server peaked at 700, far below calculated capacity.
Investigation post-incident traced bottlenecks to two issues. Slow remote operations, especially retries to invalid hosts from a hardcoded server list, mean p50 spikes well above calculated baselines. As new servers come and go, gokeyless rarely updated its routing table, causing 1-second timeouts followed by retries to other hosts. And while load tests establish a rated upper bound of 7,000 rps per server between US and EU, such hardware and configuration limitations became clear only when remote traffic swelled.
We also recognized that while gokeyless worker pools made sense for local CPU-bound operations, remote network calls turned them into an idle resource drain. We removed them entirely, moving to one goroutine per request. Goroutines are lightweight enough for this scale, and this shift dropped memory usage 7x—no idle goroutines waiting for work.
The Road to Scalability
Our failures underscore the value of deep infrastructure visibility. Distributed tracing is essential, but our spans often didn't cover the entire remote operation path, obscuring critical performance bottlenecks. The load-testing blinds spot: we tested common local key paths extensively but not the more complex, lower-volume remote case. Even more, our static, hardcoded list of data centers became stale, routing traffic to dead ends instead of the nearest available, active nodes.
As Geo Key Manager becomes more central to new offerings, solving these architectural constraints isn't just about recovery—it’s preparing for what comes next. We are already working on making Geo Key Manager far more flexible and scalable, setting the stage for deeper changes in the near future.
