Inside Cloudflare's private backbone: scale, routing, and resilience

Cloudflare operates data centers in 330 cities in more than 120 countries as of July 2024, each running company equipment and services. These sites are placed so that customers are present in all major regions and can meet local regulatory obligations; the network is programmable, sending traffic to the best available data center, which lets sensitive data stay regional under the Data Localization Suite. Tying these locations together — and exchanging data with customers, public clouds, partners and the wider Internet — is the network managed by Cloudflare's infrastructure engineering and network strategy teams.

How Cloudflare connects to other networks

Interconnection happens in two principal ways:

  • Peering. Voluntary interconnection between administratively separate networks, allowing traffic to be exchanged between their users. Cloudflare is among the most peered networks globally, with agreements in 330 cities and across all major Internet Exchanges (IXs). Parties can register to peer or attach directly via a private network interconnect (PNI).
  • IP transit. A paid service for crossing another operator's network, typically used by a smaller ISP to reach the wider Internet — effectively a toll for a private highway.

The backbone itself is a dedicated, high-capacity optical fiber network carrying traffic between Cloudflare data centers, where the interconnection methods above are used. Within a city and over long distances, Cloudflare manages its own dark fiber or leases wavelengths using Dense Wavelength Division Multiplexing (DWDM), which sends multiple data streams on different wavelengths of light through a single fiber. These services are bought and leased from carrier partners worldwide. Carrying data over the backbone is more reliable than sending it across the public Internet.

BLOG-2432-Hero
BLOG-2432-3

Why run a backbone at all

Operating a global backbone is hard work that many competitors avoid. Cloudflare does it for two reasons: control over routing and cost. With IP transit, traffic from Cloudflare to the destination network depends on transit partners; the backbone gives full control over internal and external routing, usually yielding the lowest-latency paths. At large traffic volumes, serving traffic over the backbone is also, on average, cheaper than IP transit, which is why capacity is being doubled down in Frankfurt, London, Amsterdam, Paris and Marseille — regions of sustained traffic growth with widely available, competitively priced connectivity.

Both internal and external traffic ride the backbone. Internal traffic covers customer traffic on security or performance products plus data shuttled between Cloudflare data centers by internal systems. Tiered caching, for example, arranges data centers into lower and upper tiers: a lower tier requests missing content from an upper tier, which in turn goes to the origin, reducing origin requests and improving cache efficiency. Moving that cached content over the backbone is often the most cost-effective option at Cloudflare's scale. Magic Transit attracts traffic via BGP anycast to the nearest data center for DDoS mitigation, after which the backbone carries the clean traffic to the customer's data center over a Cloudflare Network Interconnect (CNI).

External traffic includes origin responses from providers such as AWS, Oracle, Alibaba, Google Cloud Platform and Azure. Those responses travel through peering points and the backbone to the Cloudflare data center nearest the customer, giving more control over backhaul, better reliability and performance, and less dependence on the public Internet. This interconnection of public clouds, offices and the Internet, with a controlled layer of performance, security, programmability and visibility, is the Connectivity Cloud.

Expansion and the global ring

Backbone capacity in Tbps has grown more than 500% since 2021, and with sub-sea cable capacity added to Africa, the global backbone ring was completed in 2023. It now reaches six continents over terrestrial fiber and subsea cables.

A key recent strategy is building backbone capacity in regions with less developed Internet infrastructure than Central Europe or the US, in pursuit of a shared goal with regional ISP partners: keeping data flow local and close to the end user. Insufficient local peering and regional infrastructure often pushes traffic onto inefficient routes outside the region — traffic tromboning — via cheaper international routes and existing peering agreements. Regional backbone investments in countries such as India and Turkey reduce that need: traffic can move directly between in-country data centers, for instance Mumbai to New Delhi to Chennai, lowering latency and raising reliability to the level of more developed markets. The in-region backbone also keeps data local, supporting the Data Localization Suite (DLS) for regional data privacy compliance.

BLOG-2432-5
This map is a simplification of our current backbone network and does not show all paths

This map is a simplification of the current backbone network and does not show all paths.

Latency gains in practice

Expansion has measurably improved latency. Since the backbone between Lisbon and Johannesburg was deployed, users in Johannesburg have seen major performance gains — relevant, for example, to a financial institution running APIs for real-time trading, or to Magic WAN users relying on site-to-site connectivity between branch offices.

BLOG-2432-6

The table measures round-trip time (RTT) for an uncached origin fetch from an end user in Johannesburg to various origin locations, comparing the backbone with the public Internet. Carrying the origin request over the backbone instead of IP transit or peering makes content arrive up to 22% faster for local users there. Long-hauling traffic over the backbone puts the path, and thus performance, fully under Cloudflare's control. The gain varies by location but consistently shows the backbone's advantage.

BLOG-2432-7

Steering traffic: BGP plus Segment Routing

Picture a navigation system that picks the route with GPS and carries a toll pass valid to the destination, letting the car pass toll stations without stopping. The backbone works similarly, resting on two pillars: BGP, the Internet's routing protocol, and Segment Routing MPLS, which steers traffic along predefined forwarding paths. Segment Routing encapsulates traffic end to end from ingress to egress router; intermediate nodes perform no route lookup and instead forward along a label-switched path — a virtual circuit or tunnel — executed by the routers. Once traffic enters a label-switched path it cannot detour onto the public Internet and must follow the predetermined route. Running a "BGP Free Core" in this way is common practice, with route intelligence at the edge and intermediate nodes only forwarding.

Segment Routing Traffic Engineering (SR-TE) additionally lets the backbone automatically choose paths between data centers optimized for latency and performance, since the routing protocol's "shortest path" is not always the lowest-latency or best-performing one.

Argo and Orpheus on top of the backbone

Argo Smart Routing uses the combined portfolio of backbone, transit and peering connectivity to find the best path between the data center where a request lands and the origin server, potentially forwarding through another Cloudflare data center. Orpheus, its counterpart, routes around degraded paths for all customer origin requests at no charge, analyzing network conditions in real time and avoiding reachability failures: Argo optimizes requests from Cloudflare data centers to origins, while Orpheus provides self-healing error avoidance universally. Together, Segment Routing forms the underlay and Argo and Orpheus the connectivity overlay, moving customer traffic along the most optimized available paths.

Argo Transit Selection extends Argo Smart Routing by explicitly selecting the lowest-latency path between Cloudflare data center hops for origin requests — frequently the backbone, being more dedicated and private than third-party transit networks. Consider a multinational Dutch pharmaceutical company using Cloudflare's SASE solution to connect global offices, research centers and remote employees, with Asian branch offices securely reaching data in central data centers. If a cable is cut between regions, the network automatically looks for the best alternative route, limiting business impact.

Argo measures every potential combination of provider paths, including the backbone, to reach origins. Given the breadth of interconnection and the private backbone, it can identify the most performant path, and the backbone is consistently among the lowest-latency options. Resilience against fiber cuts and third-party transit issues is a further priority. During a disruption of the AAE-1 (Asia Africa Europe-1) submarine cable, Argo observed the following between Singapore and Amsterdam on some transit provider paths versus the backbone:

BLOG-2432-8
BLOG-2432-9

The large purple spike is a latency increase on one third-party IP transit path due to congestion, eventually resolved after likely traffic engineering within that provider's network. Other transit networks showed a smaller but noticeable increase (yellow). The backbone (green), backed by diverse connectivity between Asia and Europe, stayed essentially flat at around 200ms between Amsterdam and Singapore throughout the fiber cut, with no hiccup of the kind seen on transit paths — so Argo actively used the backbone.

BLOG-2432-10

Getting onto the network

Just as Argo improves performance inside the network, Cloudflare Network Interconnects (CNIs) improve how traffic gets onto it. Enterprise customers are encouraged to use the free CNIs as on-ramps wherever practical, gaining full leverage of the network, backbone included, and higher performance across every product in their Connectivity Cloud. The global network is the main product, and the backbone is critical to it. Cloudflare also invites network on-ramp partners to integrate directly and offer their customers secure, reliable connectivity; details are available on the on-ramp partnerships page.