Why another redesign was necessary
Quicksilver is Cloudflare's key-value database, running on every server in its network across 330 cities and 125+ countries. It holds configuration data for numerous services and is consulted on essentially every request that hits the network. Because it sits directly in the request path, performance requirements are strict: 90% of requests complete in under 1 ms, and 99.9% in under 7 ms. The database currently holds over five billion key-value pairs totaling 1.6 TB, and serves more than three billion keys per second worldwide.
The primary constraint has always been disk space. Since Quicksilver historically stored every key on every server, the dataset size directly limited how much room remained for other uses, such as content caching. The larger the dataset grew, the more expensive it became to store it across the entire fleet. Over the past year alone, the database has grown by roughly 50% to its current 1.6 TB.
From full replication to a hybrid model
The first version of Quicksilver was simple: every key-value pair lived on every server. This offered excellent read performance and trivial lookup logic, but it did not scale in terms of storage. As disk space began running out, the team needed an interim solution, and Quicksilver V1.5 was born.
V1.5 introduced a proxy mode. Instead of holding the full dataset, proxy instances maintain only a cache of keys. Cache misses are resolved by querying another server that runs a replica instance with the complete dataset. Each server runs roughly ten separate Quicksilver instances, each with its own distinct database and key set. In the V1.5 design, half of those instances operated in proxy mode and the other half remained full replicas.
This intermediate approach cut disk usage on each server in half, and it had the side benefit of giving the team practical experience operating Quicksilver in a more distributed configuration. But it was always intended as a stopgap, not the end state.
Why V1.5 could not scale further
Three problems made V1.5 unsuitable as a long-term architecture. First, instance sizes were inherently unstable. The teams that use Quicksilver own their key spaces, and their usage patterns shift frequently. While most instances grew over time, some actually shrank as teams optimized their workloads. A split that was well balanced at deployment quickly became skewed.
Second, the cache sizing assumptions proved incorrect. The team estimated which keys would need to be cached by assuming that all keys accessed over a three-day period would be a sufficient working set. That analysis suggested roughly 20% of the key space needed to be cached. While this worked for most instances, some required far more than 20% of their key space to achieve acceptable hit rates.
The third, and most fundamental, issue was that even a 40% reduction in disk usage does not change the growth trajectory. Quicksilver's dataset keeps expanding, and within about two years of deploying V1.5, disk space was again in short supply. The team concluded that what was needed was an architecture that could actually get ahead of the storage ceiling, rather than another temporary reprieve.
Rethinking the data path
Analysis of access patterns revealed that a considerable portion of the key-value pairs were effectively never used. These cold keys accumulated for various reasons: outdated entries not properly cleaned up, keys relevant only to specific regions or data centers, and values that had not been requested in a very long time—if at all.
Sharding the full dataset across servers was considered as a way to distribute storage, but it introduces significant complexity and fails to optimize for data locality. If the key space is divided into four shards, each server can only serve 25% of requested keys from its local database, and cold keys would still occupy disk space. A cache, by contrast, naturally handles local access patterns and avoids storing unused keys entirely.
The decision was made to keep copies of the full dataset on only a handful of storage servers with large disks, while all other servers maintain only a cache. This was an incremental step from the preceding version: caching infrastructure and inter-data center discovery had been in operation since 2021 and were battle-tested.
Adding a relay
A concern emerged that having every instance connect directly to a small number of storage replicas would overwhelm them with connections. To address this, a Quicksilver relay was introduced. Within each data center, a few servers are elected to run in relay mode, maintaining persistent connections to the storage replicas. All proxies in the data center discover these relays and route cache misses through them.
Reactive prefetching
Initial hit rates still required improvement. The observation was that a key missing on one server in a data center had a high probability of missing on another server in the same data center in the near future. This led to a mechanism where relays publish a stream of all resolved cache misses. All proxies in the data center subscribe to this stream and populate their local caches with the resulting key-values. This strategy is termed reactive prefetching, as it responds to actual misses rather than predicting future ones. Predictive approaches were tried but produced no measurable benefit and were dropped.
With reactive prefetching, the worst performing instance reached a cache hit rate of roughly 99.9%. However, one team required even higher rates due to latency sensitivity. Their instance, dnsv2, serves DNS queries, where a single query may trigger multiple lookups, amplifying any added latency. Achieving the necessary performance for this instance required one more architectural change.
Sharded caching returns
The instance needing the highest hit rate was also the one where cache performance was worst. Cache retention time—the duration a key-value is kept after last access—needed to be longer for this workload, which demanded more disk space than was available.
Another pattern had been noticed: caches performed better in smaller data centers. Larger data centers serve larger and more diverse request populations, leading to bigger cache footprints. But larger data centers also have more total disk capacity. This observation revived the idea of sharding—not the full dataset, but the cache itself.
The key space is divided into 1024 logical shards by hashing keys and splitting by range. These logical shards are then grouped by range into physical shards, with each server assigned one physical shard, determined by the same range-based process applied to server hostnames. This creates a data center-wide sharded cache as a second tier, supplementing the local per-server caches. A request first checks the server's local cache, then the data center-wide sharded cache, and only on a miss there does it fall back to the storage nodes.
Scaling the shard count is straightforward: doubling the number of physical shards re-assigns each server a subset of its previous key range. The server's existing cache simply contains the necessary keys; those no longer needed are evicted over time. Shards stay well balanced as they represent uniform random subsets of a very large key space. Populating the physical shard caches reuses the reactive prefetching stream—keys belonging to a server's assigned shard are simply retained longer in cache than others.
Cutting corners is possible with a cache. Multiversion concurrency control, which requires serving older versions of keys for consistency, is not supported on cache shards. A key-value is only unusable if it was written at a more recent database version than the proxy's current version. Since the proxy and cache shard versions are generally very close, this is a rare occurrence, and deferring such lookups to storage nodes has no noticeable impact on hit rate.
Three tiers of storage
The resulting Quicksilver V2 architecture comprises three distinct storage levels:
- Level 1: The local cache on each server holds the most recently accessed key-values.
- Level 2: The data center-wide sharded cache retains key-values accessed less recently.
- Level 3: Storage replicas on a small set of servers hold the full dataset and serve cold keys.
Measured impact
Adding the second caching layer significantly improved the share of keys resolved entirely within a data center. The worst performing instance achieves a combined cache hit rate above 99.99%, while all other instances exceed 99.999%.
Evolution in practice
The migration from a fully replicated store to the tiered architecture took several years. It involved moving hundreds of thousands of live databases without interruption while handling billions of requests per second. The rollout was iterative, with changes designed to be easily reversible when possible—strategies essential for safely evolving critical infrastructure.
