How Orpheus automatically routes around bad Internet weather

Keeping origins reachable when the Internet misbehaves

Cloudflare’s Orpheus system is designed to solve a specific, frustrating problem: a customer’s origin server is up and working, but some end users still can’t reach it because of connectivity problems on the path between them. The system identifies these broken Internet paths in real time and reroutes traffic through alternative routes that still work. Since Orpheus was made freely available to all Cloudflare customers almost two years ago, it has saved 132 billion requests from failing and prevented more than 50 Internet incidents from affecting customers.

The root cause of these failures is often not the origin itself, but the unpredictable state of the Internet. A fiber cut, a misconfigured router, or an incorrectly advertised BGP route can make a destination appear reachable when it actually is not, or simply drop packets along the way. These conditions change daily, which is why Cloudflare describes the problem in terms of Internet "weather." When a 522 error occurs, it may be that the origin is offline for everyone, but it can also be the case that the origin is fine and only a specific network path between Cloudflare and the origin is broken.

Orpheus works from a simple goal: if any Cloudflare data center can reach an origin, then any user on Cloudflare’s network should be able to reach that origin too. In practice, this means detecting when a request to an origin fails and routing it through a different Cloudflare data center or network path that can make the connection. For example, if a user in Tampa gets a 522 error when trying to reach an origin in Chicago due to a damaged fiber line, Orpheus can route the request through Boston, which has a working path to Chicago, and save the request.

Where BGP falls short

BGP is the protocol that routes traffic across the Internet by selecting paths to a destination IP. It is designed to find alternate routes when a preferred one fails. The problem is that BGP does not always detect a failure, especially when an autonomous system continues to advertise a route but drops the packets it receives. Even when BGP does detect a problem, it takes time to converge on a new route. Orpheus takes a different approach: it constantly tests the health of routes between Cloudflare data centers and customer origins using synthetic probes, and it records which paths actually work. When a request fails on a preferred path, Orpheus can immediately select a failover path from its database that it knows to be valid, rather than waiting for BGP to update.

This becomes critical in scenarios involving BGP hijacking or misconfiguration. If a route is hijacked, requests sent down that path will be dropped. Orpheus can detect this by testing routes and examining the results, then route around the invalid path. The scale of Cloudflare’s network, which directly connects to 12,000 networks across 300 cities with 197 Tbps of capacity, gives it many alternative paths to choose from. For instance, a user in Mumbai trying to reach an origin in New York might encounter a failed route caused by a cable cut in Egypt. Orpheus can instead route the traffic through London, using a different set of physical connections, and complete the request.

Built on Argo Smart Routing’s architecture

Orpheus is not a new system; it is a specialized deployment of the technology behind Argo Smart Routing, Cloudflare’s paid service that optimizes traffic for speed. Argo’s architecture has a control plane that computes the fastest and most reliable routes between locations, and a data plane that sends traffic along those routes. Orpheus uses an alternate version of Argo’s control plane where the routing table contains only failover paths. When a request cannot reach an origin via the preferred route, the data plane selects a failover path from this table, and it prioritizes the routes predicted to be the most reliable.

The system also relies on the Internet monitoring infrastructure that was built for Argo Smart Routing. This system regularly tests routing paths between Cloudflare’s data centers and origin servers by opening and closing TCP connections, a method called synthetic probing. The results are stored in a database that records which routes succeed and which fail. This database feeds both Argo and Orpheus, allowing them to choose working paths at any given moment.

Scaling to handle every failed request

Cloudflare proxies millions of requests to origins every second, so Orpheus needed to be efficient about which requests it reroutes. The system processes only requests that would otherwise fail, and it uses an error cache in Cloudflare’s layer 7 HTTP stack to identify them. When a TCP request to an origin gets no response and produces a 522 error, Orpheus logs an entry in this cache with the origin address and the specific route that failed. The next time a request targets that origin over that same route, Orpheus checks the cache and, on a hit, sends the request over an alternate route.

The error cache is kept current in two ways. Routes that previously failed are retried with live traffic using exponential backoff on the 5th, 25th, 125th, 625th, and 3,125th request. If a retry succeeds, the failed route is removed from the cache and normal routing resumes. If it fails, the entry stays and the backoff counter is updated. There is also a 10-minute expiry for cache entries, which prevents the cache from retaining stale records for routes that rarely see traffic anymore.

This design has one inherent limitation: at least one request must fail before Orpheus can reroute subsequent requests. While the first request to a broken path still gets a 522 error, all following requests are saved. Cloudflare acknowledges that the ideal solution would anticipate failed routes before the first request, and the engineering team is working on eliminating this initial failure.

Recent improvements and next steps

Orpheus originally selected failover paths based entirely on their predicted reliability. This approach sometimes resulted in rerouted requests taking a path that was reachable but slow. The routing algorithm has since been updated to balance both latency and origin reachability, so a rerouted request is nearly as fast as a normal request on Cloudflare’s network.

The team is also working to make Orpheus respond faster to changes in connectivity. Currently the system relies primarily on synthetic probes to assess route health. Plans call for using real-time traffic data as an additional input, which would allow Orpheus to detect unreachable routes within minutes rather than hours. This faster detection would also help the system notice when a previously broken route becomes healthy again.

Another active area of development is extending Orpheus to protect WebSocket traffic. While HTTP connections are opened and closed per request, WebSockets maintain an open connection between client and server for the duration of a session, so a connection failure can disrupt many requests at once. Orpheus now applies the same rerouting logic to WebSocket connections, giving customers resiliency across HTTP, TCP, and WebSocket traffic.

Orpheus and Argo Smart Routing together

Orpheus shares its underlying technology with Argo Smart Routing, but the two serve different functions. Orpheus maximizes origin reachability by routing around failures, while Argo Smart Routing uses latency data to find the fastest working path, typically improving web asset performance by 30%. The two services complement each other: Orpheus runs automatically for all Cloudflare customers, while Argo Smart Routing is an optional addition for those who want the speed optimization.