Inside Geo Key Manager v2: re-imagining access control for distributed systems

Why Static Access Control Broke Down

The fundamental problem with the first iteration of Geo Key Manager was not cryptography, but policy. The system was built on a combination of identity-based broadcast encryption and identity-based revocation. It gave customers three dials: a set of regions to encrypt to, a set of locations inside those regions to exclude, and a set of locations outside to include. That worked well enough at a small scale, but the underlying cryptographic parameters—the list of regions, data centers, and their attributes—were frozen at setup time.

When a new data center came online, it could not be added to decryption policies without resurrecting the entire setup. When geopolitical events demanded immediate exclusions, the system could not respond quickly. And when customer needs evolved, such as a new data sovereignty requirement that did not cleanly map to existing region definitions, the predefined static lists were insufficient. These limitations, combined with painful manual key rotation and high tail latency, drove the design of Geo Key Manager v2, which is built on a properly expressive Attribute-Based Encryption (ABE) scheme.

The Hard Problem: Giving Out Keys Without Central Authority

To understand why this matters, consider a European bank that wants its TLS private keys stored only in EU data centers. Cloudflare must have the key on a machine in order to terminate TLS for a customer. The challenge is where to put the encrypted key material without breaking the operational model that makes Cloudflare's global network work.

The default architecture replicates every customer's private key to every machine in every data center via Quicksilver, a globally distributed KV store. The basic design is straightforward: the customer uploads a private key, which is encrypted with a master public key shared across all machines, then pushed to every data center.¹

Four candidate solutions present themselves, and three are dead ends for the global network model:

• Restricted replication: Only send a key to machines that are allowed to use it. This contradicts the architectural assumption that the entire keyset is available on every machine. Machines would have to proxy TLS handshakes to permitted locations, adding latency and complexity.
• Centralized storage: Keep keys in the core, not in Quicksilver, with policy enforcement at lookup time. This reintroduces a single point of failure and forces every handshake across the world to reach a central location, defeating the purpose of an edge network.
• Per-datacenter ciphertexts: Encrypt a customer key with the public key of every datacenter allowed to hold it. With 500 data centers and a set of 200 permitted ones, storing a customer key now consumes 200 to 500 times its original size to avoid distributing it to the others. Space usage, previously a function of the key count only, becomes multiplied by the number of permitted data centers.
• Grouped ciphertexts: A single key pair for all permitted data centers and another for the rest means encrypting twice, not 200 times. This works for one simple policy but shatters the moment policies become more nuanced. A New York data center could hold a country: US key, a country: US or region: EU key, and a not country: RU key, each as a separate key pair. With multiple overlapping policies, each data center needs a combinatorial set of keys, and provisioning a new site forces re-evaluation of every policy for key assignments.

None of these scale with the flexibility that modern regulatory and compliance demands require. The answer, implemented in CIRCL, Cloudflare's open source cryptographic library, is ABE. ABE makes the ciphertext itself carry an arbitrary access policy, which eliminates the quadratic blow-up in storage and keeps distributed authority by not relying on a central server to issue decryption decisions.

Policy-Based Encryption for Distributed Infrastructure

In 2004, Amit Sahai and Brent Waters proposed attribute-based encryption (ABE), a cryptosystem where a message is encrypted under an access policy rather than an identity. Users receive private keys derived from their attributes, and decryption is only possible when those attributes satisfy the policy. This stands in contrast to traditional public key encryption, where keys are bound to specific identities.

ABE comes in two related forms: key-policy (KP-ABE) and ciphertext-policy (CP-ABE). They are duals of each other and functionally equivalent. CP-ABE maps more naturally onto real-world access control, making it the better fit for practical deployments.

ABE's real power is its expressiveness. Consider a hospital where a doctor holds the attributes role: doctor and region: US, while a nurse holds role: nurse and region: EU. A document encrypted under the policy role: doctor or region: EU is readable by both, demonstrating how a single ciphertext can be scoped to many different authorized parties.

Not every ABE scheme meets the needs of a production distributed system, though. Our requirements narrowed the field considerably:

1. Negation — Supporting NOT in addition to AND and OR gates enables blocklisting. This requires care: negation of an attribute compares values for a known key rather than treating the attribute as absent. So "country is not Japan" requires a machine that has a country and can show it isn't Japan. Most schemes only handle monotonic formulas.
2. Repeated Attributes — Policies like organization: executive or (organization: weapons and clearance: top-secret) need support for the same attribute appearing multiple times.
3. Chosen Ciphertext Security — The base form of many ABE schemes is only secure against chosen plaintext attacks. We applied the well-known Boneh-Katz transform to gain security against chosen ciphertext attacks. The full security proof will be presented in a forthcoming paper.

These constraints led us to the scheme by Tomida et al. (2021). Implementing it required moving beyond the discrete log assumption used in traditional public key cryptography, since ABE must secure both ciphertexts and attribute-based keys. The construction depends on bilinear pairings, and the speed of those operations determines baseline performance. Decryption is where pairing efficiency matters most, since that's when keys are combined with ciphertexts. We used our highly optimized pairing implementations from the open source CIRCL library. The keys, attributes, and ciphertext structures also require linear algebra routines for matrix multiplication, transpose, and inversion.

Policy expression presented another design decision. We settled on strings for the API, treating policies as boolean expressions like country: JP or (not region: EU). Users would have needed a parser regardless of the interface we chose; providing one ourselves gives a more stable contract. Internally, the frontend string is converted into a monotonic boolean circuit of wires and gates. NOT gates are handled at the wire level: by De Morgan's Law, any NOT can be pushed down to the wires of an AND or OR gate, so the circuit itself remains monotonic.

The API follows a standard ABE structure. A central authority runs Setup to produce a master public key and master secret key. The public key encrypts messages over policies, while the secret key generates user keys from attributes. Attribute values are supplied and validated out-of-band — in our case, via the machine provisioning database. Generated keys are distributed over TLS, and helper functions check decryption capability or extract policies from ciphertexts.

publicKey, masterSecretKey := cpabe.Setup()

policy := cpabe.Policy{}
policy.FromString("country: US or region: EU")

ciphertext := publicKey.Encrypt(policy, []byte("secret message"))

attrsParisDC := cpabe.Attributes{}
attrsParisDC.FromMap(map[string]string{"country": "FR", "region": "EU"}

secretKeyParisDC := masterSecretKey.KeyGen(attrsParisDC)

plaintext := secretKeyParisDC.Decrypt(ciphertext)

assertEquals(plaintext, "secret message")

From Centralized Broadcast to ABE

Returning to our original motivating scenario, each machine in every data center presents its attributes to the central authority, which validates them and issues a unique attribute-based secret key. Key issuance happens during machine provisioning or whenever attributes change — never in the critical path of a TLS handshake.

ABE is also collusion resistant. Two machines cannot pool their keys to decrypt a ciphertext that neither can decrypt alone. A machine with country: US and one with security: high cannot together decrypt something encrypted under country: US and security: high without one machine holding both attributes.

The scheme also handles infrastructure changes gracefully. New machines are issued keys on demand, since participants do not need to be fixed at setup time — a distinct advantage over the earlier identity-broadcast approach.

The operational flow works like this: a customer uploading a TLS certificate specifies a policy, and the central authority encrypts the private key under the master public key using that policy. The ciphertext is then distributed to all data centers through Quicksilver.

At request time, the TLS termination service that receives a connection fetches the encrypted private key. If that service's attributes do not satisfy the policy, decryption fails and the request is proxied onward to the nearest data center that can. Only a data center that successfully decrypts the key can perform the signature to complete the handshake.

The trade-offs across all candidate solutions are summarized below:

Performance Profile

Benchmarks are inspired by the ECRYPT framework. Measurements ran on an Intel Core i7-10610U CPU @ 1.80GHz with an attribute set size of 50 — far higher than most real-world deployments need, but a useful worst case. Results are compared against RSA with 2048-bit keys, X25519, and our prior scheme.

Different ABE construction optimize for different profiles: some favor fast key generation, others faster encryption or decryption. Our priority is decryption speed, because decryption is the only operation in the critical path of a request. Key generation and encryption both occur as part of out-of-band workflows where added latency is an acceptable trade for stronger security.

ABE as an ABAC Implementation

What we've built is an implementation of Attribute-Based Access Control (ABAC) — an extension of Role-Based Access Control (RBAC), as defined by NIST in 2017. ABAC addresses an evolution in access control thinking that began with Discretionary Access Control (DAC), the model underlying Unix file permissions. DAC fails when resource owners can grant access in ways a central administrator would not permit. Mandatory Access Control (MAC) was introduced to prevent exactly that kind of resharing; DRM is a common example.

Roughly speaking, RBAC implements aspects of MAC by constraining users to pre-defined roles. ABAC extends that to arbitrary attributes like time of day or user agent. Both are specifications, traditionally enforced by a central authority policing a resource. Attribute-based encryption offers a way to implement the same policy model without requiring a central decision point at every access — the encryption itself enforces the policy across the distributed system.

Key Rotation Without the Headaches

Key rotation in Geo Key Manager v1 was manual and error-prone, so v2 was designed from the start to make rotation robust and simple, with no availability impact. The solution introduces an indirection layer in the customer key wrapping process. Instead of encrypting a customer's uploaded TLS private key directly with the Master Public Key, the system generates an X25519 keypair called the policy key. The central authority stores the public half of this keypair along with its associated policy label in a database, and encrypts the private half with the Master Public Key according to the access policy. The customer's private key is then encrypted with the public policy key and saved to Quicksilver.

When a user hits a customer's website, the receiving data center's TLS termination service fetches the encrypted policy key tied to the customer's access policy. If the machine's attributes fail to satisfy the policy, decryption fails and the request is forwarded to the closest data center that can satisfy it. On successful decryption, the policy key unlocks the customer's private key and the handshake completes.

Policy keys are not minted for every certificate upload. If a customer requests a policy that already exists in the system, the existing policy key is reused. Since most customers gravitate toward a small set of policies — one country, the EU, and so on — the number of policy keys is orders of magnitude smaller than the number of customer keys.

That reuse is what makes rotation practical. When master keys are rotated, only those few policy keys need to be re-encrypted instead of every customer key, cutting compute and bandwidth costs. Caching policy keys at the TLS termination service also keeps decryptions out of the critical path, improving performance.

The mechanism is similar to hybrid encryption, but with a twist: the policy keys are X25519 keypairs (asymmetric, elliptic-curve based) rather than symmetric keys. This is slower than schemes like AES but far faster than attribute-based encryption, and it means the central service can encrypt customer keys without ever touching secret key material.

Robust rotation also requires maintaining multiple key generations. The newest generation is used for encryption, while both the latest and previous versions can decrypt. A state system manages key transitions and safe deletion of retired keys, with monitoring in place to catch any machine lagging on the wrong generation.

The Tail at Scale

Geo Key Manager v1 suffered from high tail latency that occasionally threatened availability. Server and client revamps — including switching from worker pools to one goroutine per request and deleting thousands of lines of code — didn't move the p99 numbers. Distributed tracing eventually showed the delays were happening between the client sending a request and the server receiving it, but no further progress was possible at that level.

The key insight was an indirection layer between client and server. Cloudflare's data centers vary greatly in size, so larger data centers used intermediary machines to proxy requests to smaller ones via the Go net/rpc library, avoiding connection overload. Only when the forwarding function was added to the trace did the problem surface: a long delay between issuing a request and processing it, in code that was just a call to a built-in library function.

The root cause was a lock held during request serialization. The net/rpc package doesn't support streams, but Cloudflare's custom packet-oriented application protocol (pre-dating gRPC) does. To bridge that gap, the serialization function executed a request and waited for its response — a functional but serial bottleneck that allowed only one forwarded request at a time. The fix replaced that with channel-based coordination, letting multiple requests execute concurrently while responses arrive. Tail latency dropped dramatically after rollout.

Physical limits remain: customers who restrict keys to the US while their users are in Australia will still pay the trans-pacific round trip, though session tickets limit that cost to new connections. Uptime improved substantially as well. Data centers provisioned after cryptographic initiation could now join the system, and those that didn't satisfy a policy had a wider set of satisfying neighbors to forward to — a redundancy boost that especially helped regions with unreliable internet connectivity. Over a two-day global probe, Geo v1 policies for US and EU regions dipped below 98% availability at points, while Geo v2 rarely dropped below four nines.

Looking Forward

Attribute-based encryption has become markedly more efficient and capable in recent years, and the gap between research and production adoption is worth closing — particularly for distributed systems that can't rely on a highly available central authority. Cloudflare has open-sourced its CP-ABE implementation in CIRCL and plans a paper with more details. Beyond private keys, the ABE-based mechanism is slated for storing other data types, and work is underway to make it more user-friendly and generalizable for internal services.