From spoke-hub to peer-to-peer
Cloudflare's first flexible purge system leaned on Quicksilver, the internal configuration distribution network, to carry purges from a core data center out to every other data center. Propagation took under a second, and the purge key lookups in the hot path cost low hundreds of microseconds. That design held for years before three problems surfaced: latency tied to a customer's distance from the centralized ingest point, write bottlenecks at that same ingest point, and storage in every data center that scaled with throughput demand. Kafka queues in front of Quicksilver plus heavier batching bought runway, but added latency.
Purge-by-URL had already moved to peer-to-peer distribution, and that system's results were strong enough that shipping flexible purges through it became the obvious step. Quicksilver's speed depends on its spoke-hub assumption, so spoke data centers cannot write to it — meaning a new storage system was required.
Revisiting the lazy purge
The quote from part 2 explains why flexible purge was never coreless-compatible: these requests can clear many objects or entire zones, through a process separate from the simple single-object path.
The reason flexible purge isn't also fully coreless yet is because it's a more complex task than "purge this object"; flexible purge requests can end up purging multiple objects – or even entire zones – from cache. They do this through an entirely different process that isn't coreless compatible, so to make flexible purge fully coreless we would have needed to come up with an entirely new multi-purge mechanism on top of redesigning distribution. We chose instead to start with just purge by URL, so we could focus purely on the most impactful improvements, revamping distribution, without reworking the logic a data center uses to actually remove an object from cache.
A URL purge request carries everything needed to compute the cache key — the filename pointing at the response object on disk — which is why distributing address-style purges is a tractable problem. Response-based attributes are different: cache-tags are response headers, while disk placement is decided from request attributes alone. Finding every file carrying a given tag would mean scanning every disk on every machine in every data center.
The original system sidestepped that with "lazy purge": matching files were not deleted up front. On a request, the proxy compared the timestamp of recent owner purges against the file's insertion timestamp and treated a fresher purge as a cache MISS. Correctness required retaining purge history longer than a data center's maximum eviction age, so every matching file was either naturally evicted or forced to refresh from origin.
A database to scan on each machine
Storage pressure eventually forced a rethink. Retained purge history, though small and compressible, competed for disk with customer content. The team reconsidered the early conclusion that proactive deletion from disk was impractical. A global index would hit the same bottleneck as the spoke-hub model and would span trillions of unique files. A per-data-center index looked better on paper, but running a database cluster sized to hundreds of machines per site was effectively a second full-time project.
An index colocated on each machine resolved those objections: the proxy can reach it over UDS, avoiding network complexity on the hottest path; as a sidecar it adds no distributed-database machinery; and a machine's disk set changes seldom enough that the index size stays predictable. Fleet analysis of representative cache disks — assets per terabyte, average cache-tags per asset, and MISS, REVALIDATED and EXPIRED rates to estimate write throughput — confirmed feasibility, and benchmarking of relational and non-relational candidates led the team to RocksDB, a high-performance embedded key-value store that proved excellent at the required query shapes once tuned.
CacheDB and the instant-result trick
The result is CacheDB: a Rust service on RocksDB running beside the cache proxy, maintaining indices and buffering and executing purges. The proxy writes indices when a file is cached or updated and deletes them on eviction or purge. A background process drains the local purge queue, looks up matching files through the indices, removes them from disk, then clears the indices and the queue entry.
A second check hides the deletion lag. On a cache HIT, the proxy asks CacheDB to scan its queue for a pending purge matching the asset, and if one exists it fetches fresh instead. A purge therefore stops matching content from being served the moment it reaches a machine — even when a key maps to millions of files that take a while to delete.
The distribution pipeline was redesigned to broadcast flexible purges to the CacheDB service on every machine, not just to every data center. CacheDB handles last-mile fan-out within a data center and uses consul to track peer health, letting the Workers stay largely unchanged for purge-by-URL despite the different termination point.
What the redesign bought
Active deletion cut the lazy purge's long tail and saved 10x storage; the reclaimed disk raises cache retention for all users, which should lift cache HIT ratios and reduce origin egress. A purge now reaches any data center, distributes its keys, and clears the cache database in under 150ms at P50 for tags, hostnames and URL prefixes, across 330 cities in 120+ countries — a 90% improvement over May 2022.
The numbers come from production traffic rather than synthetic tests. A timestamp is stamped on the request as it enters the local data center and a second is added once every data center has executed the purge; each request's data lands in a database used for quantiles. August 2024 data was segmented by the region of the ingress data center.

Rate limits and availability ahead
Purge by cache-tag, hostname and prefix has historically been Enterprise-only. The rebuilt infrastructure scales beyond current capacity, so the company plans to revise rate limits and roll these methods out to Free, Pro and Business plans beginning in early 2025. Single-file purges currently sit at 234ms P50, with identified improvements expected to bring that below 200ms.



