Configuration Distribution at Cloudflare: Building Quicksilver

Cloudflare’s network processes over fourteen million HTTP requests per second at peak, serving users worldwide. A critical component of that performance is how the company distributes configuration changes—DNS records, Workers, and hundreds of other settings—to hardware in 200 cities across 90 countries, all within seconds. More than 26 million Internet properties depend on this system, which has had to scale dramatically as the company has grown.

For years, this infrastructure ran on Kyoto Tycoon (KT), a datastore that worked well for what it was originally designed to handle. But as the network expanded past 100 data centers, KT’s limitations became critical. In 2015, Cloudflare decided to build a replacement from scratch, resulting in Quicksilver, a system that now powers an average of 2.5 trillion reads each day with latency in the microseconds.

The Architecture of the Edge

Every data center must serve requests even if cut off from all central configuration. This means all configuration data must live locally at each edge location. The original system deployed KT to thousands of machines, with centralized web services writing to root nodes that distributed values to management nodes in each data center. Each server then received its own copy of data from its local management node.

Reading from KT was on the critical path of nearly every Cloudflare service. A DNS or HTTP request would trigger multiple KT reads, and every TLS handshake loaded certificates from it. If KT was slow or down, so were the services depending on it. Early fixes included sharding KT into separate instances—Page Rules in one, DNS records in another. By 2015, Cloudflare operated eight such instances storing 100 million key-value (KV) pairs, with about 200 values changing per second, running tens of thousands of KT processes.

Why Kyoto Tycoon Broke Down

KT’s own documentation describes it as "a lightweight datastore server with auto expiration mechanism" designed for cache data and persistent data. Cloudflare was pushing it far beyond that scope, and the problems were significant.

Write Contention and Read Latency

KT's documentation claims no exclusive write lock over the entire database—the hash database uses record locking, and the B+ tree database uses page locking. In practice, this proved not to be true. When heavy write bursts occurred, read latencies spiked dramatically.

Performance testing showed the problem clearly. Reading the same 20 key/value pairs in an infinite loop produced acceptable results with no writes:

  • P99: 9ms
  • P99.9: 15ms

Adding a single writer sequentially appending a 40kB value caused latencies to skyrocket:

  • P99: 154ms
  • P99.9: 250ms

With a second writer, the 99.9th percentile exceeded one second:

  • P99: 701ms
  • P99.9: 1215ms

The root cause was found in the storage engine code: the accept function in kcplandb.h acquires a lock that is also acquired in the synchronize function of kcfile.cc, which flushes data to disk. Flushing to disk blocks all reads, and flushing is slow. KT was simply not designed for concurrent access at this scale.

Initial mitigations included reducing write load and disabling fsync on each write. This meant KT would only flush to disk on shutdown, introducing potential data corruption that required separate tooling to detect and repair.

Data Corruption and Sync Problems

KT's documentation acknowledges that if a database is not closed properly, "some records may be missing and the database may be broken." With kernel panics and processor bugs happening at scale, this was a constant threat. Disabling syncing compounded the issue.

Database corruption became common. KT's auto-repair mechanism worked on smaller databases but often failed to complete on the largest ones. Ultimately, auto-repair was disabled entirely, and corrupted databases had to be manually copied from healthy nodes—a time-consuming process for the SRE team. The lack of disk syncing also meant KT had to flush the entire database on shutdown, a process that sometimes exceeded the systemd grace period and resulted in SIGKILL termination and further corruption. Clean shutdowns sometimes still produced corrupted databases on restart; with thousands of servers, this happened multiple times daily.

Replication Weaknesses

KT's replication protocol is based solely on timestamps. If a transaction fails to replicate and isn't detected, the timestamp advances past it permanently, and the entry is lost. Replication logs are garbage collected over time, so a node down for days could restart and request logs that no longer exist, resulting in an inconsistent database.

The timestamp file itself was another flaw. It was only written when the replication loop terminated, not continuously. If KT terminated unexpectedly, it would replay days of already-applied transaction logs, making old values visible again temporarily. Cloudflare also experienced databases getting out of sync without any identifiable root cause—a problem never fully explained.

Upgrade and Availability Constraints

KT does not allow multiple processes to access the same database file, so zero downtime upgrades were impossible. Stopping KT was slow—often over 15 minutes—with no guarantee about the database state. With full releases only happening quarterly in 2015, upgrades to KT infrastructure were an extremely slow process.

KT's "dual main" replication topology, which supports two servers replicating each other, comes with a critical caveat: updating both servers simultaneously can cause inconsistency. One must be designated active and the other standby, with manual promotion on failure. This violates high availability principles—there was no capability for automatic zero-downtime failover.

Requirements for a Replacement

Fixing KT wasn't feasible. The project had no maintainer—the last official update was from April 2012—and consisted of 100k lines of C++. The fundamental limitations were clear: no high availability, a weak replication protocol, an exclusive write lock, and no support for zero downtime upgrades. Maintaining KT consumed 48 hours of SRE time per week.

The search for an alternative focused on a few key requirements:

  • Ability to take consistent snapshots of a running database, useful for bootstrapping new machines efficiently
  • Zero downtime upgrades
  • No exclusive write lock
  • Crash-proof behavior without recovery tooling

Cloudflare settled on LMDB, a datastore library that meets all these requirements. LMDB is optimized for low read latency—more important than write throughput, since reads are far more frequent. The performance improvement was dramatic: the DNS service's 99th percentile of reads dropped by two orders of magnitude after the switch.

LMDB allows multiple processes to concurrently access the same datastore, enabling upgrades without stopping traffic. It's append-only, never overwrites existing data, and nothing is written to disk in a corrupted state. After any termination, it can be restarted immediately without issue. Its stability has been exceptional—only a single bug and zero data corruption in over three years of production, across 90,000 database instances.

Quicksilver: The Distributed Layer

LMDB provides local storage, but Cloudflare needed a distributed system. The solution was a fan-out topology where nodes query main-nodes, which in turn query top-mains for the latest updates.

To ensure reliable propagation, the protocol includes a monotonically increasing sequence number. By comparing sequence numbers to ensure each message is exactly one higher than the last, it becomes easy to detect lost updates. Since the system only supports writes from a single aggregated location, maintaining this counter is straightforward.

To prevent misconfiguration errors where different database instances could be corrupted with the wrong data, unique IDs were added for each database instance, required during the replication handshake.

Replication performance was improved with batching: all updates in a 500ms window are combined into a single write, reducing the number of disk commits. Transaction logs are stored within the same LMDB database, allowing the log and the update to be committed atomically in one shot.

Optimizations and Operations

Transaction log entries are significantly larger than the values they represent, so Snappy compression is used to prevent disk exhaustion. Logs are also periodically garbage collected. An incremental hash within the logs ensures messages haven't been lost or reordered.

To prevent a node from accidentally replicating from itself, a randomly generated process ID is exchanged during the handshake. Secondary main nodes store a week of history, allowing machines that have been offline to be fully resynchronized on startup.

The fragmentation cost of storing transaction logs within the database was a known trade-off. Large values can eventually be hard to store in available disk regions, causing write latency to spike. Compaction requires rewriting an entire database, a process performed when data centers come offline, with benefits lasting around two months. Chunking transaction logs into page-sized pieces has mitigated this, and the codebase includes a key-value level CRC to detect any bugs arising from fragmentation logic.

Upgrades and Observability

LMDB's support for concurrent processes enables handoff upgrades: a new process starts, and the listening socket is passed over seamlessly. Interestingly, after years of experience, the team now questions whether this complexity is warranted—since a Quicksilver restart takes only milliseconds, allowing momentary connection failures might be simpler and equally reliable.

Cloudflare uses Prometheus for metrics collection and Grafana for dashboards, with dashboards at global, data center, and per-server levels. Detection of availability issues is easy—if Quicksilver is down, countless alerts fire. Detection of replication lag is harder; it's monitored by writing a heartbeat at the top of the replication tree and measuring the time difference on each server.

Quicksilver has become a key enabling technology at Cloudflare, allowing configuration changes to propagate in seconds. The plan is to open source it, following the pattern of other infrastructure tools Cloudflare has shared with the broader engineering community.