A New Storage Backbone for Workers KV
On June 12, 2025, a major outage at a third-party cloud provider took down critical Cloudflare services that depend on Workers KV. That incident exposed a fundamental vulnerability in the service’s architecture: a reliance on external object storage. In response, Cloudflare has overhauled Workers KV’s backend, moving all data onto its own infrastructure and adding a redundant, in-house storage path that also improves performance.
From Dual-Provider to a Single Point of Failure
Workers KV launched in 2018, before Cloudflare had its own storage services like R2 and Durable Objects. To ensure availability, the original design wrote every key-value pair to two independent third-party object storage providers. A service called Storage Gateway Worker (SGW) handled writes and deletes, racing requests against both providers and returning the fastest response for cache misses. This active-active setup provided resilience and low latency, but it came with operational costs.
Keeping two independent object stores synchronized required complex consistency machinery. As KV scaled, the differences in provider APIs, failure modes, and performance limits became increasingly difficult to manage. The system’s caching layer also became less predictable under heavy traffic, exposing more consistency edge cases. Earlier this year, Cloudflare made the strategic decision to consolidate onto a single third-party provider as an intermediate step while developing its own storage solution. That decision proved costly on June 12, when the remaining provider suffered a global outage, causing over two hours of widespread failures across Access, Gateway, WARP, and dozens of other services.
Design Constraints and the Database Path
The immediate requirement was to bring a second, fully redundant storage backend online so no single provider outage could again take down KV. The scale demands were extreme: hundreds of billions of small objects, petabytes of data, millions of GETs per second, and tens of thousands of steady-state writes. The median object size is just 288 bytes—a workload poorly suited to traditional object storage, which assumes larger files and carries high per-object overhead.
Bringing the previously disabled third-party provider back was not feasible; its integration code had degraded, and it had historically shown high error rates and low throughput limits. Building directly on R2 was also not ideal for this workload, though Cloudflare is working on optimizations like inlining small objects to reduce retrieval hops. Instead, Cloudflare chose a distributed database already running in production behind both R2 and Durable Objects. This database provides in-house expertise, proven reliability, and strong consistency. Data is sharded across multiple database clusters, each with three-way replication, which keeps the blast radius small and avoids the limits of a single massive cluster.
Bridging the Gap with KVSP
One immediate implementation challenge was connectivity. SGW communicates over HTTP across Cloudflare’s global network, while databases typically use binary protocols over persistent TCP connections. To bridge this gap, Cloudflare built a new service called KV Storage Proxy (KVSP). KVSP exposes an HTTP interface to SGW and handles database connectivity, authentication, and shard routing internally. It stripes namespaces across clusters using consistent hashing, preventing hotspotting and noisy-neighbor problems.
A distributed database excels at tiny objects but is not efficient for the largest KV values, which can reach 25 MiB. To handle both cases, KVSP automatically routes larger objects to R2, creating a hybrid storage backend. From SGW’s perspective, the HTTP API is identical regardless of object size, so the complexity is entirely transparent.
Restoring Redundancy with a Faster Write Path
Cloudflare has also restored dual-provider capabilities from the earlier architecture, adapted to work with the new system. Writes are now raced to both backends simultaneously, but success is returned to the client as soon as the first backend confirms. This keeps latency low while ensuring durability across two independent systems. If one backend fails—due to network issues, rate limiting, or degradation—the failed write is queued for background reconciliation, forming part of the synchronization machinery that keeps both backends consistent.
The June 12 outage was a stark reminder that external dependencies can become single points of failure. By moving Workers KV onto Cloudflare-owned infrastructure and layering in a hybrid database-object storage backend with write racing, the service now has a path to eliminate third-party reliance entirely while improving performance.
Rolling Out the Hybrid Architecture
Deployment began cautiously, starting with background writes from SGW to the new Cloudflare backend. This validated write performance and error rates under production load without disturbing read traffic. It also moved the full dataset onto the new infrastructure. Existing data from the third-party provider was then copied through KVSP, marking the point where manual failover to the new backend could be completed within minutes if another outage struck — eliminating the single point of failure behind the June incident.
Once failover capability was confirmed, the first namespaces were enabled in active-active mode, beginning with internal Cloudflare services that had strong monitoring and known workload patterns. Traffic was increased slowly, with results compared between backends after responses had already been returned to users. This asynchronous comparison caught discrepancies without adding user-facing latency.
Testing surfaced a consistency regression relative to the previous dual-provider setup, prompting a brief rollback of the active-active change. Workers KV is eventually consistent by design, with changes taking up to 60 seconds to propagate globally as cached versions expire. But the team had inadvertently degraded read-your-own-write (RYOW) consistency for requests routed through the same Cloudflare point of presence. Previously, RYOW worked within each PoP because PUT operations wrote directly to a local cache. KV throughput had since outscaled the IOPS the caching infrastructure could support, so that approach no longer worked. Although not a documented property of Workers KV, some customers had come to depend on this behavior.
To quantify the issue, an adversarial test framework interspersed rapid reads and writes to a small key set from multiple global locations. It measured the percentage of reads that observed RYOW violations — stale data returned immediately after a write from the same PoP. The results guided a redesigned cache population and invalidation strategy that restored the expected RYOW behavior without sacrificing the performance that makes Workers KV suitable for high-read workloads.
Keeping Data Aligned Across Backends
With writes racing to two independent providers, maintaining consistency requires layered safeguards. Workers KV has always used three complementary mechanisms, though the details have evolved.
The first line of defense operates at write time. SGW sends writes to both backends simultaneously and treats the operation as successful once either provider confirms persistence. If a write succeeds on one side but fails on the other due to network issues, rate limiting, or degradation, the failed key is queued for a background reconciliation system, which deduplicates failed keys and starts synchronization.
The second mechanism engages during reads. When SGW races reads against both providers and observes divergent results, it triggers the same background synchronization process. This pulls inconsistent keys back into alignment on first access rather than letting them diverge indefinitely.
Background crawlers form the third layer, continuously scanning data across both providers and correcting mismatches missed by the reactive mechanisms. They also track drift rates, revealing how often keys slip through and surfacing underlying issues.
Synchronization relies on version metadata attached to every key-value pair. Each write generates a new version from a high-precision timestamp plus a random nonce. When values diverge, the newer timestamp wins and that value is copied to the provider holding the older version. Clock skew could theoretically cause misordering when timestamps land within milliseconds of each other, but the tight bounds maintained through Cloudflare Time Services and typical write latencies mean conflicts would require nearly simultaneous overlapping writes.
To prevent synchronization from overwriting newer data, conditional writes verify the stored timestamp is older before applying a change. Deletes are handled the same way: a tombstone with a newer timestamp is written instead of removing the key outright, since a delete that reached only one backend would otherwise be treated as missing data and copied back from the other side. Only after both providers hold the tombstone is the key physically removed.
This design does not promise strong consistency, but it eliminates most mismatches between backends while sustaining the latency profile that suits high-read workloads. In systems terms, Workers KV favors availability over consistency (AP under the CAP theorem) and further chooses latency over consistency even when no partition exists — PA/EL under the PACELC model. Most inconsistencies resolve in seconds through the reactive mechanisms, and the background crawlers eventually correct remaining edge cases.
The architecture is consistent with the earlier dual-provider setup, but two changes materially improve outcomes. KVSP has a far lower steady-state error rate than the prior third-party providers, reducing the write failures that create inconsistencies. And reads are now always raced against both backends, whereas the old system learned which provider was faster for its region and routed subsequent reads exclusively to that one, only falling back to the other on failure. That preferential routing controlled cost and optimized latency, but it created a blind spot — inconsistencies could persist if reads rarely hit the slower backend.
Measuring the Gains
As active-active operation expanded to internal and external namespaces, the expected availability improvement arrived alongside notable performance gains. The effects were strongest in Europe, where the new storage backend is located, but stretched beyond what geographic proximity alone explains.
Internal p99 latency for reads to KVSP came in below 5 milliseconds. By comparison, non-cached reads to the third-party object store from the closest location — normalized for transit time — typically ran around 80ms at p50 and 200ms at p99.
The graphs above show the closest available apples-to-apples comparison: internal latency for KVSP requests versus cache misses forwarded to the external provider from the closest point of presence, which includes an additional 5-10 milliseconds of request transit time.
These improvements translated directly into faster responses for internal Cloudflare services relying on Workers KV. The database-optimized storage was especially effective for the small object access patterns that dominate KV traffic. Success with internal workloads paved the way for expanding the rollout to external customer namespaces, confirming the value of building critical infrastructure on Cloudflare's own platform.
Next Steps
The immediate roadmap centers on expanding the hybrid architecture into additional locations, building a fully global distributed backend served entirely from Cloudflare infrastructure. Work also continues on reducing the time to reach consistency both between providers and in cache after writes.
The end goal is to eliminate the remaining third-party storage dependency entirely, removing the external single points of failure that triggered the June incident and taking full control over the storage layer's performance and reliability. Beyond Workers KV, the patterns developed here — KVSP as a translation layer, automatic object routing by size, and reuse of existing database expertise — apply to other services balancing global scale with consistency requirements. The path from a single-provider setup to a resilient hybrid architecture running on Cloudflare infrastructure demonstrates how operational challenges can drive architectural improvements that benefit customers across the platform.
