From full replication to replica/proxy roles
Cloudflare's Quicksilver began as a way to distribute configuration globally at internet scale, but it quickly became the storage backbone for many products. The original design — Quicksilver v1 — gave every server a complete copy of the data, updated through asynchronous replication. That worked for a while, but the cost of storing everything everywhere grew as both the dataset and the data center footprint expanded.
Most of that replication was unnecessary. Data accessed in one region was being copied to every other region, even if it was never read there. The response is a new architecture with two distinct server roles: replicas hold the full dataset, while proxies act as persistent caches that evict unused key-value pairs to save disk space. This interim design, called Quicksilver v1.5, is a stepping stone toward a more scalable system.
Understanding the roles requires some context on Cloudflare's network layout. Core data centers — the control plane — host root nodes with terabytes of storage. Smaller edge data centers have two node types: intermediate nodes, which replicate from roots or other intermediates, and leaf nodes, which serve end-user traffic. Every server runs 10 Quicksilver instances — independent databases backing specific products like DNS, CDN, or WAF.
Rather than hosting ten full datasets per machine, a data center now deploys only a few replicas per instance; the remaining servers run proxies with a hot-key cache that queries replicas on misses.
Data centers range from hundreds of servers down to a single rack, so the first step was simple: split each server evenly, with five replicas of some instances and five proxies of others. This frees disk space immediately, potentially cutting usage by up to 50%, and gives the codebase an early battle test before moving to a more distributed model.
Validation through working-set analysis
Before committing to the design, we verified that proxy caching would actually suit our workloads. If proxies needed the entire dataset cached to perform well, there would be no disk savings. A pipeline pushing accessed keys from across the network to ClickHouse revealed that large data centers use roughly 20% of the keyspace, while small data centers use only about 1%. Those figures made the caching approach viable.
Persistent caching with eviction
In-memory caching was ruled out for two reasons: the sheer number of keys would balloon memory usage, and a restart would leave the cache cold. Instead, the proxy cache is persistent, stored in the same embedded RocksDB used for full datasets. Cache misses trigger a request to a replica over the internal distributed key-value protocol, after which the key is stored locally.
Eviction relies on RocksDB compaction filters, which run custom logic on background threads as files are compacted. A key-value pair is filtered periodically, evicting least recently used data once free disk space crosses a soft limit. An LRU-like in-memory structure tracks access times and informs the filter. A hard limit acts as a safety valve: if disk space drops to a critical level, new keys stop being cached until space frees up.
Replication and consistency challenges
The original Quicksilver delivered sequential consistency: if key A is written before key B, a client can never read B without seeing A. That guarantee is baked into the products that depend on Quicksilver, so v1.5 must preserve it.
Asyncrounous replication complicates that. Replication paths vary in speed across the globe, so machines progress at different rates. With a proxy in front of replicas, this can break monotonic reads. Consider a client writing keys A through K sequentially at the root. Replication propagates unevenly:
- replica_1 has only seen A and B
- a proxy has seen A through E
- replica_2 is ahead, with A through I
If the client reads E through I twice, the first request might hit replica_2 and succeed, while the second lands on replica_1 — returning “not found” for keys F, G, H, and I. The correct answer is actually neither. A v1 server at the proxy's replication index would only have seen A through E, so E should be the only key returned from both calls. Both replicas gave wrong answers — one ahead, one behind.
MVCC for versioned lookups
Locking servers together would reintroduce latency into the replication tree, so the solution is multiversion concurrency control (MVCC) on replicas. Instead of overwriting the latest value in the default RocksDB column family, updates now also land in a new MVCC column family, each tagged with its replication index. Lookups from a proxy at a specific historical index work as follows:
- Check the default column family. If the key's write timestamp is not greater than the requesting proxy's index, return it directly.
- Otherwise, scan the MVCC column family for versions valid at the target timestamp.
In the earlier scenario, replica_2 holds A@1 through K@11. A request for key H at index 5 finds the latest version in the default column family, but its timestamp is 8 — too new for the proxy. The MVCC scan turns up nothing, and the replica responds “not found.” If H were updated at indexes 4 and 8, the version at index 4 would be in MVCC and returned. Key E, with a timestamp of 5, is served straight from the default column family. Deletions are handled with tombstones, marking the period during which a key is absent before a rewrite.
MVCC history doesn't need to be infinite — it only has to cover the maximum replication index gap between machines. A two-hour retention window proves generous in practice, adding about 500 MB of disk usage per replica. Custom compaction filters garbage-collect MVCC records older than two hours.
Handling Proxies Ahead of Replicas
A proxy that lags behind its replicas is easy enough to manage. The harder case is the inverse: when a proxy has already learned about a recent update but the replica it queries has not yet received it. The natural response is to have replicas refuse requests whose target index exceeds their own, and that was indeed the first implementation. Since the situation was expected to be rare, the replicas simply returned an error. Gradual rollout to a few data centers proved otherwise.
Analyzing which keys are affected by this kind of asymmetry narrows the problem considerably. Keys that were written long ago are already replicated, so the only troublesome ones are those updated very recently. Instead of pushing the burden to replicas, the fix belongs on the proxy side: keep every recent update locally so the proxy never needs to query a replica for them. This became the sliding window.
The sliding window preserves all updates made within a short, rolling timeframe. Unlike cached keys, the items in the window cannot be evicted until they fall out of the window's range. Internally, the window maintains lower and upper boundary pointers, which are kept in memory and can be trivially reconstructed from the current database index and the configured window size after a restart.
When the replication layer delivers a new update event, the proxy adds it to the sliding window by moving both boundaries one position upward, preserving a fixed window size. Keys that fall below the lower bound become eligible for eviction by the compaction filter, which is aware of the current window boundaries.
Negative Lookups
A second problem specific to the distributed replica-proxy setup is negative lookups — requests for keys that do not exist. These are not a corner case: in production workloads, negative lookups are roughly ten times more common than positive ones.
The difficulty is that every negative lookup misses the proxy cache and forces a request to a replica. At the observed request volume, that would overwhelm replicas, saturate the data center network, and ruin latency. A fast, proxy-local way to identify nonexistent keys is essential.
In v1, negative lookups are the fastest request type. RocksDB relies on Bloom filters to decide whether a key might exist in a given Sorted Sequence Table (SST) file. About 99% of negative lookups are served entirely from this in-memory structure, avoiding disk I/O entirely. Caching negative lookups on the proxy seems promising but runs into two immediate obstacles:
- The negative keyspace is theoretically infinite and practically unknown. It must fit in the cache to matter.
- Cached negatives would no longer use Bloom filters. The row and block caches in RocksDB have lower hit rates than the SST filters, meaning more negative lookups would reach the disk.
Both concerns proved fatal. The negative keyspace is enormous — for some instances it exceeds the real keyspace by a factor of a thousand. And clients are latency-sensitive enough that lookups need to be served from memory whenever possible. Exploring probabilistic alternatives, Cuckoo filters were eliminated after measurements showed roughly 18 GB of memory to match the false-positive rate of Bloom filters for 5 billion keys (the Bloom filter version needs only 6 GB).
The solution adopted was key and value separation: all keys are stored on every proxy, while values are persisted only for cached keys. When a key is evicted from the cache, its value is removed; the key itself stays. The total size of keys stripped of values in Quicksilver is approximately 11 times smaller than the full dataset. That is larger than any probabilistic representation, but the approach retains two major advantages: Bloom filter lookups in RocksDB continue to work, and the design enables optimizations for distributed range queries.
Service Discovery
Distributed query execution requires proxies to locate replicas. Within a data center this is straightforward: each operates its own Consul cluster, where machines register as services. Consul integrates with internal DNS resolvers, letting a single DNS request return the names of all replicas in a data center, which proxies can connect to directly.
Achieving reliable operation across data centers of varying size, with servers constantly added and removed, requires more than local discovery. Proxies also need to find replicas in nearby data centers. The replication layer had already hit a similar problem. Its topology was originally defined statically in configuration files distributed to every server. Simple as it was, the approach was fragile and rigid — it produced a static replication tree with suboptimal performance and required manual intervention whenever the network changed.
The replacement is the Network Oracle, an overlay network built on a gossip protocol among intermediate nodes in each data center. Every member continuously exchanges status and metadata with peers, giving a near-real-time view of active membership. Each node also runs network probes against its peers, measuring round-trip time so that the closest active intermediate nodes can be selected to form a low-latency replication tree. The result is a replication system that is fully self-organized and self-healing.
Quicksilver reuses the Network Oracle for discovery, separating the problem into two parts: data center discovery and specific service lookup. Rather than joining every Quicksilver instance to the same gossip overlay — which would inflate traffic and message delivery times — intermediate nodes expose network proximity information to leaf nodes. Knowing which data centers are close, proxies send DNS queries there directly to resolve specific services such as Quicksilver replicas.
Proxies keep a connection pool to active replicas and distribute requests across them to avoid hotspots. A health-tracking mechanism monitors connection state and replica errors, temporarily deprioritizing or isolating replicas that appear faulty.
The effect is measurable: after the new discovery system was introduced, errors from replica requests nearly disappeared.
Results
The goal of Quicksilver v1.5 was simple: free up disk space without degrading request latency. The replica-proxy design delivered significant space savings while the 99.9th percentile of request latency over a 24-hour window, shown below for both replicas and proxies, reveals almost no difference between the two. In some cases proxies are even slightly faster, likely because their smaller datasets reduce disk I/O.
Quicksilver v1.5 is released, but the work continues. A subsequent iteration will address the next set of scaling challenges.
