Tunnel health checks: a shared problem

Magic Transit advertises customer IP prefixes from Cloudflare's edge network, scrubbing traffic with DDoS mitigation and firewall policies before delivering it over GRE tunnels. Because the public Internet paths between Cloudflare and a customer can degrade or fail, customers often run multiple tunnels over different routes and rely on Cloudflare to pick the healthiest one. Since Anycast GRE lets every server in every location send traffic to any customer tunnel, each server needs to know the state of every tunnel—and each location has its own distinct set of network paths.

The original health check design was deliberately simple: every server sent one check to every tunnel every minute. It worked, but it had two weaknesses.

Inconsistent verdicts

With only one probe per minute, a single failed check could make a server declare a tunnel degraded and start shifting traffic toward a fallback. Different servers could reach different conclusions about the same tunnel, and a server could overreact to a single blip. From the customer's perspective, it looked like Cloudflare had detected a real problem when in fact one server had simply gotten a flaky result.

Slow reaction to outages

The other problem was latency of detection. If a tunnel went down a second after a successful check, the server could keep sending traffic over the dead tunnel for almost a full minute until the next probe failed. More frequent checks would fix that, but having every server across the edge check every tunnel more often would quickly overwhelm customer networks.

Sharing observations within a location

The first refinement was to make servers in the same data center collaborate. Health checks from servers in one location traverse the same Internet path, so their results ought to agree. Rather than each server trusting only its own single probe, servers joined a multicast group and exchanged their individual check results. The verdict for a tunnel was then based on data from all servers in the location, not just one.

This made tunnel state more consistent and sped up detection, particularly in large data centers where peers received updates frequently. But it did not scale. As the customer base grew, the message volume exploded; in the biggest data centers, each server was receiving close to half a billion multicast messages per minute.

Dividing the work with consistent hashing

The third version flips the model: instead of every server infrequently checking every tunnel and broadcasting every observation, each tunnel is checked frequently by one designated server, which then shares only the summary forecast. The challenge is deciding which server is responsible for which tunnel without a central coordinator or shared database.

Every server periodically sends a multicast "heartbeat." By listening to heartbeats, each server builds the same list of live peer IPs (including its own), sorted by the hash of each address. To assign a tunnel, a server hashes the tunnel ID and finds the peer whose hash is the smallest value greater than the tunnel's hash—wrapping around to the first peer if needed. That peer is the tunnel's designated checker; when the peer address matches the server's own address, that server knows it owns the tunnel.

If a server stops sending heartbeats, it drops out of the peer list, and its tunnels are reassigned to the next peer on the next hash lookup. No explicit rebalancing or leader election is required. This scheme is simple, resilient to server failures, and consistent across all servers in a location.

Scale and stability gains

With the new design, the server responsible for a tunnel can probe it much more frequently and can build a precise picture of tunnel health. Since only summary reports are multicast, message volume drops sharply even in the largest data centers. The results:

  • CPU usage for Magic Transit dropped by over 70%.
  • Memory usage dropped by nearly 85%.

These reductions let the edge keep up with the Internet's changing conditions and leave room for further growth of the Magic Transit service.