From Argo to a self-tuning cache hierarchy

Tiered Cache, part of the Argo feature set, reduces the number of data centers that fetch assets from the origin: a request in South Africa is answered by a large nearby data center rather than going straight to an origin in North America. Which data centers act as proxies is defined by a configuration called the topology. Until now, a single generic topology served all customers. Smart Topology replaces it with a configuration that maximizes cache hit ratios by identifying, through Argo's own infrastructure, the best data center from which to contact the origin.

The limits of flat caching

In the standard scheme, each data center is a reverse proxy for the origin and is unaware of the others. Any miss results in an origin fetch even if the asset already sits in another data center, so a single asset could be requested from the origin as many times as there are data centers.

Driving misses to their theoretical minimum would mean sending every request to every data center, which is impractical. Nor is it possible to predict demand and pre-place assets in the nearest cache. The practical heuristic is to copy an asset to a known location after its first miss, so other data centers can find it — precisely what Tiered Cache does by making some data centers proxies for others. When the proxied data centers request the same asset, it is served from the proxy rather than the origin, and overall origin traffic falls.

Topology trade-offs

An optimal topology needs both customer-held knowledge (the preferred balance of latency against hit ratio) and Cloudflare-held knowledge (how best to use the Internet). Enterprise customers typically maintain it by hand with our solutions engineers. The generic topology exists for everyone else: it places multiple proxying data centers worldwide so a miss never takes a long detour, balancing the number of proxies against the hit ratio since proxies are oblivious to each other. Smaller data centers are proxied by larger ones in the same region. If a proxy goes offline, its proxied data centers use a fallback when available or otherwise behave as if Tiered Cache were disabled.

How Smart Topology picks a proxy

Smart Topology assumes the origin sits in one location and configures itself automatically once the customer flips a switch in the dashboard. That requires determining which data center has the lowest latency to the origin without being told where the origin is.

IP geolocation was rejected: even the best commercial database lacks the required coverage and accuracy, and physical distance is in any case a questionable proxy for Internet latency. Instead, each data center probes the origin, using the TCP handshake time as the measurement, and the minimum is taken. Apart from edge cases involving Anycast and TCP termination, latency to an IP address is assumed to equal latency to the origin server behind it.

Weighing stability, capacity and latency

The algorithm selects a single proxying data center to maximize the hit ratio, placed close to the origin so misses at proxied data centers are not much slower than with Tiered Cache off. Three concerns shape the choice: stability, because every change of proxy tends to produce fresh misses in the new one; capacity, because an offline data center can trigger many misses at once; and latency to the origin, for efficient network use.

Selection works like a leaderboard of the fastest data center per origin. Faster data centers knock others off an origin's list, with competition held hourly and ranked on 24-hour median latency; only data centers deemed large enough may compete. Over time records replace weaker ones, so an origin's latency can only decrease monotonically until a record is physically unbeatable. The leaderboard keeps both the lowest and second-lowest latency data centers, the latter as fallback during maintenance.

When probing lies: Anycast origins

Probing measures latency to the origin IP, which breaks down when a cloud provider other than Cloudflare uses Anycast to serve that address. Packets are then routed to the nearest machine sharing the IP, so the measured figure reflects the latency to the Anycast network's edge, not to the origin server. The apparent "best" data center could even sit on another continent. Selecting it would hurt latency and could be costly.

To catch this, Smart Topology exploits a physical bound: latency is limited by the speed of light through fiber, and while the distance from a data center to the origin is unknown, the distances between data centers are known. Treating the origin as a pitstop on the journey between two data centers yields the theoretical minimum observable pair of latencies to that origin. Comparing probe data against that bound exposes impossible observations — faster-than-light results — falsifying the assumption that the IP identifies a server in one location. The check runs across many data centers worldwide, and a single physically impossible observation is enough to fall back to the generic topology.

What happens when the switch is flipped

If a proxying data center has already been assigned to the CIDR block covering the origin IP, the preferred or fallback data center pulls assets from the origin. Otherwise the generic topology decides which proxies pull them. As the choice of proxy is updated over time, latency to it should only decrease.