Cache purge, end to end
In Part 1 of this series, we described the scale and complexity of cache invalidation at Cloudflare and sketched a high-level architecture for a new purge system. That system has now been in production for single-URL purges since July 2022, and this post explains how it actually works as a request travels from API client to edge cache.
The key detail that shaped the design is where purges are evaluated. In the old system, every purge request had to be shipped to a core data center to be authenticated, validated and dispatched. The new architecture moves that logic out to the edge, so most requests never touch a core facility at all.
Step one: validation and local purge
When a client sends a purge request, it is routed to the nearest Cloudflare data center and lands in an API Gateway worker. Most API endpoints are still handled by centralized services, in which case this worker simply proxies to the nearest core data center. But for purge-by-URL requests, the gateway worker performs authentication and route authorization itself, then forwards the request to a Purge Ingest worker running in the same data center.
The ingest worker validates the URLs in the request body and attempts to purge them from the local cache. This "local purging" step was introduced specifically for the coreless system and reuses logic that already exists in every data center. In particular, the worker performs the same zone-ownership checks that a data center applies when serving normal traffic for a zone, in order to determine whether the URLs are even cacheable.
That check is surprisingly valuable. Currently more than 50% of URLs submitted for purge cannot be cached by the requesting zone — either because the zone doesn't own the URL or because a caching rule (such as a "bypass" rule) prevents those URLs from being stored. These purges are superfluous, so filtering them out at the edge avoids needless broadcast traffic to every other data center.
Generating a cache key for a file is also not free. It requires loading zone configuration that can affect the key and applying various transformations. But the cache key for a file is identical across data centers, so the ingest worker now returns the generated key from the local cache and broadcasts that key elsewhere, rather than forcing each remote data center to recompute it.
Step two: durable queuing
Once the local purge is complete, the ingest worker forwards the request — along with the computed cache key — to a Purge Queue worker. This is a Durable Object that uses its persistent state to hold a rolling queue of purge requests and, for each data center in the network, a pointer indicating how far along that data center is in processing the queue. Keeping this state durably means that if a data center goes offline, or connectivity is lost, purges can be replayed to bring the data center up to date after it returns.
A single global Durable Object would have just relocated the centrality problem from a core data center to wherever that object was provisioned. Instead, the system uses dozens of Durable Objects per region. An ingest worker selects one from its region's load-balancing pool — typically one in the same data center — and forwards the request there. That Durable Object records the purge in its queue and loops through its list of data-center pointers, trying to push any outstanding purges to each.
Benchmarking revealed that purge traffic sits in a "goldilocks zone" with respect to Durable Object throughput. Startup costs are amortized only if a Durable Object has enough steady traffic to keep it active. But each object is single-threaded, so pushing it with too many simultaneous requests creates a bottleneck. Because regional traffic varies substantially with time of day, no static number of Durable Objects could keep the system in that efficiency band. The solution was to build load monitoring into the Durable Objects themselves, plus a Regional Autoscaler worker that aggregates the data and adjusts the load-balancing pools when they approach the upper or lower edges of the band.
Step three: fanout
Once a purge is queued, it must reach every data center. The local region's Durable Objects broadcast directly to data centers within their region, but for other regions they delegate distribution to one Purge Fanout worker per region. Fanout workers maintain their own queues and per-data-center pointers for their region, using much of the same logic as the Purge Queue workers, with a key difference: fanout workers are ordinary Worker scripts, not Durable Objects, so their queues are entirely in-memory.
Because fanout workers hold no durable state, they can be torn down and restarted on any machine in the data center at any time. All the state they track is reported back to the Durable Objects, which persist it reliably. This design has two benefits. First, each Durable Object only needs to make one request per remote region instead of one request to each of Cloudflare's hundreds of data centers, dramatically reducing its outbound request load and letting it process purges faster.
Second, retries get cheaper. When a delivery failure happens between data centers in the same region it goes unnoticed; when a Durable Object in Canada has to retry a request to a data center in rural South Africa, the entire distance is traversed again. Fanout workers are elected to run in the data centers with the most reliable connections to the rest of their region's network, so inter-regional retries are rare and the latency cost of a retry is kept within a region. This fanout layer reduced end-to-end purge latency by 50% and tripled throughput.
What coreless handles today
Purge by URL has been fully coreless since July 2022. Flexible purge requests — purge by tag, host, or prefix, and purge everything — share the same entrypoint workers for authentication and validation before being forwarded to core facilities for fulfillment, but remain centrally executed.
The reason is that flexible purges require a fundamentally different removal process. A single flexible request may need to invalidate many objects or even an entire zone, and that happens through a mechanism incompatible with a fully coreless design. Making flexible purge coreless would have required inventing a new multi-purge mechanism on top of the redesigned distribution layer. Starting with purge by URL let us focus on improving how purges are distributed without reworking how data centers remove objects from cache.
Flexible purges still benefit from the coreless entrypoint. Since the API accepts bundles mixing single-file and flexible purges in a single request, these requests are now authorized, authenticated and validated at the edge before they ever reach a core facility, which reduces load on central auth services. And because validation happens in the same data center where the request arrived, users with malformed requests get much faster feedback.
Ongoing work
Latency for purge-by-URL still has room to improve, and that work continues. In parallel, we're redesigning flexible purge to become fully coreless — it's an extremely popular API and deserves the full treatment. We intend to share our results on that front soon.
