Rolling Out a Distributed Data Store Without the Drama
Updating live systems is a core discipline in software engineering. Sometimes it’s easy — spin up a replacement, shift traffic, done. But when the system is the distributed KV store underpinning configuration for millions of websites, the bar is different. Quicksilver, Cloudflare’s data store for billions of key-value pairs, had to reach production without interrupting the very traffic it was built to serve.
Earlier work covered why Quicksilver exists and what it replaced. The harder part came after: getting it into a fault-tolerant network where downtime is not an option. The deployment had to be seamless and reversible. As it turned out, plenty went wrong along the way — and every layer of failure tolerance added to the system earned its keep, because none of it was visible to customers.
Replication Under Production Load
The rollout strategy was deliberately conservative. Quicksilver ran in parallel with the existing Kyoto Tycoon deployment, with writes replicated through a custom bridge service, QSKTBridge. This bridge consumed the Kyoto Tycoon update feed, batched changes, and flushed them to the Quicksilver root node every 500ms. To guard against duplicate entries if multiple bridge nodes were ever live simultaneously, each batch carried a timestamp and was written using a Compare And Swap operation, ensuring only one batch with a given timestamp could succeed.
After a successful internal "dogfooding" test in the DOG data center, traffic was gradually shifted by pointing the loopback reads from the Kyoto Tycoon port to the Quicksilver port. Since both the memcached and KT HTTP protocols were implemented, this was transparent to existing clients. The rollout then expanded to other data centers, a process that would ultimately take over a year.
Bootstrap I/O Contention
Provisioning new data centers required a bootstrap mechanism to pull a full database copy from a remote server. Because the underlying LMDB datastore can copy an entire environment to a file descriptor, Quicksilver could wrap this to send a specific database version over a network socket. It was an easy win that quickly hit a wall: the bootstrap process filled the page cache and caused I/O saturation, impacting production reads from the still-active Kyoto Tycoon. The workaround was to defer bootstrap operations until a data center was offline for maintenance, which pushed the timeline well beyond original estimates.
Validating Against The FL Service
The FL component, which acts as the "brain" for edge requests, was the ideal validation candidate due to its heavy use of Quicksilver. The switchover metrics showed immediate gains: error counts dropped substantially, and average read latency fell to around 500 microseconds, no longer regularly hitting the 1ms threshold common with Kyoto Tycoon. The primary cause of remaining errors was aging SSDs that could occasionally take hundreds of milliseconds to respond. At the time, the metrics tool only supported min, max, and average readings, not percentiles, and all consumers were reading via TCP over loopback before later moving to Unix sockets. Malformed request detection was also enabled after discovering some consumers sent bogus requests without monitoring return values; alerts were added for all such cases.
Wild Discoveries and Fixes
Observing Quicksilver in production revealed that replication delays were driven primarily by saturated I/O, not network issues. A server under I/O contention could take seconds to write a batch, falling out of sync and failing to propagate updates to its own clients. The immediate workaround was simple: if the latest received transaction log was over 30 seconds old, Quicksilver would disconnect and try the next source server. This was typically a precursor to a disk reaching its maximum write cycle count and being queued for hardware replacement.
A cascading failure mode was also addressed. A Quicksilver instance that was not actively replicating would stop its own replication server, forcing clients to disconnect and reconnect elsewhere. Combined with exponential backoff, this caused significant lag in the six-layer replication topology used in some regions. The feature was ultimately removed entirely.
SSD Wear and Write Amplification
I/O errors on aging SSDs were more frequent than anticipated. Since LMDB maps the database file into memory, a kernel I/O error could terminate Quicksilver with a SIGBUS. These errors caused latency spikes and raised system load averages, affecting all services on the host. More unexpected was LMDB's interaction with the filesystem, which produced high write amplification. A 1 megabyte write would result in 30 megabytes flushed to disk due to copy-on-write page copying, with observed amplification factors between 1.5X and 80X, before any internal SSD amplification. Identical key-value writes from producers compounded the issue, so Quicksilver now drops duplicate KV writes on the root node.
Static Topology
Replication topology is hardcoded in Salt, with each node identified by IP address. This avoids a dependency cycle since Quicksilver itself serves DNS for Cloudflare infrastructure. While rigid and requiring careful testing for changes, it works at current scale, though more dynamic topology provisioning is under consideration.
Decommissioning Kyoto Tycoon
Identifying all Kyoto Tycoon consumers proved difficult. Obvious high-read services like FL and DNS were easy to spot, but less-visible consumers required adding logging to Kyoto Tycoon to see who was connecting. This caught permanently connected clients but missed rare, brief connections such as startup-time configuration loads. Exotic setups like stunnel-based remote access to the memcached interface added technical debt that was eliminated by strictly enforcing access methods. The edge migration proceeded methodically, data center by data center, supporting dual configuration and alerting in parallel until ready for cutover.
The Core Migration
Migrating the core data centers was fundamentally different. The edge consumed reads, but the core handled writes, and it all ran on a single physical root node. This long-standing single point of failure had already caused incidents, so its removal was a company-wide effort. A note on process: this work overlapped with the company's migration from Marathon to Kubernetes, so jobs were often started on one system before being moved to the other.
The migration progressed through several stages:
- KTrest, the stateless REST interface for Kyoto Tycoon writes, was moved to Kubernetes. All teams were asked to move their producers there as well; the move required no code changes.
- QSrest, a KTrest-compatible service for Quicksilver, was built with support for batching (aggregating 500ms of updates before flushing) to manage disk load. It also introduced write quotas, which throttled a producer's writes once a limit was reached.
- Many teams were still writing directly to Kyoto Tycoon, largely for large range reads that KTrest did not support. Range requests were known to hurt Quicksilver, so a cluster of Quicksilver consumer nodes was added to serve them. Teams were asked to route writes through KTrest and reads through Quicksilver consumers.
- After producers were migrated off the root node, teams switched from KTrest to QSrest. Once started, the move was one-way since the databases were now inconsistent. Over 50 producers were migrated one by one with close server monitoring.
After all producers were moved, a check of Kyoto Tycoon transaction logs revealed an obsolete heartbeat mechanism, which was shut down. The Kyoto Tycoon root processes were then turned off permanently. The complete removal of Kyoto Tycoon from Cloudflare took four years.
Eliminating the Root Server Bottleneck
With Kyoto Tycoon retired and all Cloudflare services running on Quicksilver, the architecture looked far healthier — but the Quicksilver root server remained a single point of failure. To close that gap, the team built Quicksilver Raft, a Raft-enabled root cluster built on the etcd Raft package.
Integrating Raft into Quicksilver proved harder than expected. The core challenge was synchronizing the Raft snapshot with the live Quicksilver database. In certain situations Quicksilver must temporarily fall back to asynchronous replication to catch up before resuming proper Raft synchronous writes. These changes ran deep in the codebase, which also made building solid unit and integration tests difficult. Still, removing that last hardware single point of failure justified the effort.
Testing the Writer Interface with QSQSBridge
Cloudflare prefers to validate components in an environment close to production before release than testing Quicksilver's core writer interface meant. Under Kyoto Tycoon, a secondary Quicksilver root node was fed from Kyoto Tycoon through the KTQSbridge, allowing tests to be run in specific ways. Once Kyoto Tycoon was deprecated, that capability was lost. The replacement, QSQSbridge, replicates from one Quicksilver instance and writes to a Quicksilver root node, effectively providing that same testing environment with both sides being Quicksilver.
Retiring the Legacy Top-Main
After Quicksilver Raft and QSQSbridge were in place, three small data centers replicated from a test Raft cluster for several weeks. The next step was to promote the high-availability (HA) cluster and redirect the entire world to it, while retaining the ability to roll back at any moment.
To make this possible, the team reworked QSQSbridge so it could generate transaction logs compatible with the legacy feeding root. That allowed data centers to be moved in groups under the Raft cluster, with a safe path back to the legacy top-main at any time.
The migration started with the legacy architecture:
From there, all ten QSrest instances were moved one at a time from the legacy root node to the HA cluster. The process went slowly but smoothly, and eventually the old top-main could be powered down.
That marked a major milestone — Quicksilver was running in production without a hardware single point of failure.
Signs of Wear: Scaling Pain Emerges
The original Quicksilver bootstrap process pulled a full database copy, which worked fine when databases were small. The approach had fundamental problems under load. A long-lived read transaction was opened on the server during bootstrap, keeping the initial DB version intact while applying new log entries. If anything interrupted that read — a network glitch, for example — the read transaction closed, the desired DB version became unavailable, and the entire bootstrap had to restart. On top of that, the process fragmented the source database, requiring manual compaction.
As databases grew over the first six months, both problems worsened. Longer transfers meant greater exposure to network failures, especially for remote data centers, and far more fragmentation on the source side. Bootstrapping increasingly became an operational headache.
One SRE noticed that bootstrapping instances serially performed far better than bootstrapping them in parallel. Investigation showed that parallel bootstraps caused severe page cache contention. When all Quicksilver instances fetched full database copies at once, the combined database size relative to available physical memory overwhelmed the kernel and SSD with page swapping.
The immediate workaround was serialized bootstrapping across instances, enabled by a lock between Quicksilver processes. A client grabs the lock to bootstrap, then releases it for the next client. The lock circulates among clients in round-robin fashion.
A second page cache thrashing issue was traced to readahead configuration. Despite LMDB doing mostly random I/O, readahead had been enabled to maximize the database loaded into memory. As databases grew, that readahead began indiscriminately evicting useful pages. The fix was counterintuitive but effective — disabling readahead.
The deeper, most fundamental issue, however, was disk space and device wear. Every server in Cloudflare carries the same full dataset, meaning a single 1 MB piece of data consumes at least 10 GB of storage globally. Adding or replacing disks could only stretch so far. The Quicksilver design from five years earlier was clearly hitting its ceiling and needed a re-think in both the short and long terms.
Immediate Relief and a Future Direction
On the long-term roadmap, Cloudflare is building a sharded version of Quicksilver that avoids replicating the full dataset to every machine while still guaranteeing that the complete dataset lives within each data center.
For immediate needs, several initiatives addressed storage pressure. First, many internal teams didn't track their Quicksilver usage closely. A service called QSusage was created to match KV data with their owners via QSrest ACL, reporting total disk usage per producer, including Quicksilver metadata. This gave teams visibility into their actual footprint and growth rates.
Second was the question of which KV pairs were actually being used. The QSanalytics service answers that by gathering all keys accessed across Cloudflare, aggregating, and shipping them to a ClickHouse cluster with a 30-day rolling window — with no sampling and all read accesses tracked. Engineering teams can use the reports to identify unused keys and decide whether to delete or retain them.
Partly, the problem traced back to the LMDB storage engine. Cloudflare began exploring alternatives and zeroed in on RocksDB, which offered built-in compression, online defragmentation, and prefix deduplication.
Testing with representative Quicksilver data suggested RocksDB could cut storage needs by roughly 60% compared to LMDB. Read latency was slightly higher in certain cases, though not seriously. CPU usage rose to around 150% relative to LMDB's 70%, but write amplification was much lower, easing SSD load and allowing higher data ingest rates. Subsequently, Cloudflare switched to RocksDB, with plans to share deeper performance comparisons and the migration experience in a follow-up piece.
A Platform Lessons Learned
Moving from Kyoto Tycoon to Quicksilver took company-wide effort and careful scheduling, but eliminated single points of failure and improved database access performance along the way.
The journey surfaced issues that simply weren't knowable at design time — page cache contention, fragmentation costs at scale, and global storage footprints all emerged only when running at production scale. As data and product demands evolve, that real-world insight is steering how Quicksilver will grow. The current version now serves as a solid foundation for what comes next.
