How a hijack and a leak took down 1.1.1.1

On June 27, 2024, a small percentage of users worldwide saw 1.1.1.1 become unreachable or degraded. The cause was a combination of a BGP hijack and a route leak affecting the popular DNS resolver address.

Cloudflare was an early adopter of Resource Public Key Infrastructure (RPKI) for route origin validation (ROV). With RPKI, IP prefix owners can store and share ownership information securely, and other operators can validate BGP announcements by comparing received routes with what is stored in the form of Route Origin Authorizations (ROAs). When networks enforce ROV properly and prefixes are signed via ROA, the impact of a BGP hijack is greatly limited. Despite increased adoption of RPKI and the 1.1.1.0/24 prefix being a signed resource, during this incident 1.1.1.1/32 was originated by ELETRONET S.A. (AS267613) and accepted by multiple networks, including at least one Tier 1 provider that accepted it as a blackhole route.

This caused immediate unreachability for the DNS resolver from over 300 networks in 70 countries. The overall impact was low — less than 1% of users in the UK and Germany, for example — and some countries saw no impact at all.

Why 1.1.1.1 is vulnerable

Cloudflare launched 1.1.1.1 in 2018, and it has become one of the most popular free resolver IP addresses. That popularity, combined with historical use of 1.1.1.1 as a lab or testing address, creates operational challenges. Networks have long used 1.1.1.1 in test environments, leaving residual unexpected traffic or blackholed routing behavior.

The role of longest prefix matching

Hijacks of the individual address are possible because of Longest Prefix Matching (LPM). If a fictitious FooBar Networks assigns 1.1.1.1/32 to one of its routers and advertises that prefix internally, its customers will have difficulty routing to the DNS service. If the network advertises 1.1.1.1/32 externally, the impact grows.

Many prefixes in a route table could match the 1.1.1.1 address — 1.1.1.0/24, 1.1.1.0/29, and 1.1.1.1/32 all match. The /32 is the "most specific" route and wins the LPM algorithm because it has the highest number of identical bits and longest subnet mask. Instead of traffic toward 1.1.1.1 routing via anycast and landing on a Cloudflare server, it lands on a device within the hijacking network, and a legitimate response fails to be served. This can be done purposefully or accidentally by network operators.

How route leaks propagate

Route leaks are a second source of impact. A route leak occurs when a network becomes an upstream, in terms of BGP announcement, for a network it shouldn't be an upstream provider for. This is a type 1 leak as defined in RFC7908, where a customer forwards routes learned from one provider to another.

If enough networks in the Default-Free Zone accept a route leak, it may be widely used for forwarding traffic along the bad path. Often the leaking network overloads because it isn't prepared for the global traffic it now attracts. Cloudflare previously documented a wide-scale leak that knocked off a large portion of the Internet when a provider in Pennsylvania attracted traffic toward global destinations it typically never transited.

The acceptance of leaked routes comes down to BGP local-preference. A transit provider has customers who pay for transport, so it logically assigns higher local-preference to customer routes than to private or Internet Exchange (IX) peers, ensuring the paid connection is used. Networks also may prefer Private Network Interconnects (PNIs) over IX routes for reliability — a private connection has no third-party managed fabric — or for cost efficiency, since allocating a router port and purchasing a cross connect represents an investment worth utilizing.

BGP hijacks and route leaks can happen to any IP or prefix on the Internet. But 1.1.1.1 is so recognizable and historically misappropriated that it is more prone to accidental incidents than other IP resources. The June 27, 2024 incident involved both types of failure simultaneously.

Timeline and impact

All timestamps are in UTC.

  • 2024-06-27 18:51:00 AS267613 (Eletronet) begins announcing 1.1.1.1/32 to peers, providers, and customers with the AS267613 origin AS.
  • 2024-06-27 18:52:00 AS262504 (Nova) leaks 1.1.1.0/24, received from AS267613, upstream to AS1031 (PEER 1 Global Internet Exchange) with AS path "1031 262504 267613 13335".
  • 2024-06-27 18:52:00 AS1031, as upstream of Nova, propagates 1.1.1.0/24 to various Internet Exchange peers and route-servers, widening the leak's impact.
  • 2024-06-27 18:52:00 One Tier 1 provider receives the 1.1.1.1/32 announcement from AS267613 as a Remote Triggered Blackhole route, causing blackholed traffic for all that provider's customers.
  • 2024-06-27 20:03:00 Cloudflare raises an internal incident for 1.1.1.1 reachability issues from various countries.
  • 2024-06-27 20:08:00 Cloudflare disables a partner peering location with AS267613 that is receiving traffic toward 1.1.1.0/24 and engages the partner about the incident.
  • 2024-06-27 20:10:00 AS262504 leaks 1.1.1.0/24 with a new AS path, "262504 53072 7738 13335", also redistributed by AS1031. Traffic is delivered to Cloudflare along this path, but with high latency for affected clients.
  • 2024-06-27 20:17:00 Cloudflare engages AS262504 regarding the route leak to its upstream providers.
  • 2024-06-27 21:56:00 Cloudflare engineers disable a second peering point with AS267613 receiving traffic meant for 1.1.1.0/24 from multiple sources not in Brazil.
  • 2024-06-27 22:16:00 AS262504 leaks 1.1.1.0/24 again, attracting some traffic to a Cloudflare peering with AS267613 in São Paulo. Some 1.1.1.1 requests return with higher latency, but the 1.1.1.1/32 hijack and blackholing appears resolved.
  • 2024-06-28 02:28:00 AS262504 fully resolves the route leak of 1.1.1.0/24.

Customers experienced the impact in one of two ways: unable to reach 1.1.1.1 at all, or able to reach it only with high latency per request. Since AS267613 was hijacking the 1.1.1.1/32 address somewhere within its network, many requests failed at some device within that autonomous system. There were intermittent flaps during the incident when requests were successfully routed toward Cloudflare data centers, albeit with high latency.

Looking at Germany and the United States as source countries, the traffic pattern shows requests to 1.1.1.1 landing in Brazilian data centers — traffic that would normally never route to Brazil. The gaps between spikes are when requests were blackholed prior to or within AS267613, and the spikes themselves are when traffic was delivered to Cloudflare with high latency. The brief spikes likely reflect the 1.1.1.1/32 route flapping within AS267613's network, occasionally allowing traffic through instead of dropping it somewhere in the intermediate path.

Although the root cause was external to Cloudflare, the company says it will continue improving detection methods to yield quicker response times and will promote adoption of RPKI-based hijack and leak prevention such as Route Origin Validation and Autonomous Systems Provider Authorization (ASPA) objects for BGP.

Anatomy of the outage: hijack and leak in parallel

Under normal conditions, user traffic to 1.1.1.1 reaches the nearest Cloudflare data center via BGP anycast. During the incident, two separate announcements corrupted that path. AS267613 (Eletronet) advertised 1.1.1.1/32 to peers and upstreams, while AS262504 leaked 1.1.1.0/24 upstream. The combined effect redirected or blackholed traffic for numerous eyeball networks.

Public route collectors and the monocle tool reveal the rogue updates.

monocle search --start-ts 2024-06-27T18:51:00Z --end-ts 2024-06-27T18:55:00Z --prefix '1.1.1.1/32'

A|1719514377.130203|206.126.236.209|398465|1.1.1.1/32|398465 267613|IGP|206.126.236.209|0|0||false|||route-views.eqix
–
A|1719514377.681932|206.82.104.185|398465|1.1.1.1/32|398465 267613|IGP|206.82.104.185|0|0|13538:1|false|||route-views.ny
–
A|1719514388.996829|198.32.132.129|13760|1.1.1.1/32|13760 267613|IGP|198.32.132.129|0|0||false|||route-views.telxatl

The route-views collectors show AS398465 and AS13760 receiving 1.1.1.1/32 from AS267613 around the time impact began. The IPv4 default-free zone (DFZ) normally accepts nothing longer than a /24, yet multiple networks forwarded traffic using this /32 route from AS267613. Those packets were blackholed before reaching a Cloudflare POP (Point of Presence). The announcement is a BGP hijack: the origin AS is AS267613, while the Route Origin Authorization (ROA) for the prefix is signed only for origin AS13335 (Cloudflare) with a maximum length of /24. Cloudflare's own BMP (BGP Monitoring Protocol) data also captured the spurious 1.1.1.1/32 update from several route servers. Cloudflare rejects this route on import as both RPKI Invalid and DFZ Invalid, but the pre-policy BMP feed still showed where the update arrived over peering sessions.

The trouble deepened because a Tier-1 transit provider treated the 1.1.1.1/32 from AS267613 as a Remote-Triggered Blackhole (RTBH) route. RTBH is a blunt DDoS mitigation where a customer signals a provider to discard traffic for a destination before it reaches the customer's port. The Tier-1 provider's acceptance of an unauthorized blackhole for 1.1.1.1/32 meant any network routing to 1.1.1.1 through that provider saw total unreachability. Only AS13335 should be authorized to blackhole its own destinations, and Cloudflare would never do so for 1.1.1.1.

Meanwhile, the route collectors also logged networks accepting 1.1.1.0/24 from AS262504.

~> monocle search --start-ts 2024-06-27T20:10:00Z --end-ts 2024-06-27T20:13:00Z --prefix '1.1.1.0/24' --as-path ".* 267613 13335" --include-sub

.. some advertisements removed for brevity ..

A|1719519011.378028|187.16.217.158|1031|1.1.1.0/24|1031 262504 267613 13335|IGP|187.16.217.158|0|0|1031:1031 1031:4209 1031:6045 1031:7019 1031:8010|false|13335|162.158.177.1|route-views2.saopaulo
–
A|1719519011.629398|45.184.147.17|1031|1.1.1.0/24|1031 262504 267613 13335|IGP|45.184.147.17|0|0|1031:1031 1031:4209 1031:4259 1031:6045 1031:7019 1031:8010|false|13335|162.158.177.1|route-views.fortaleza
–
A|1719519036.943174|80.249.210.99|50763|1.1.1.0/24|50763 1031 262504 267613 13335|IGP|80.249.210.99|0|0|1031:1031 50763:400|false|13335|162.158.177.1|route-views.amsix
–
A|1719519037|80.249.210.99|50763|1.1.1.0/24|50763 1031 262504 267613 13335|IGP|80.249.210.99|0|0|1031:1031 50763:400|false|13335|162.158.177.1|rrc03
–
A|1719519087.4546|45.184.146.59|199524|1.1.1.0/24|199524 1031 262504 267613 13335|IGP|45.184.147.17|0|0||false|13335|162.158.177.1|route-views.fortaleza
A|1719519087.464375|45.184.147.74|264409|1.1.1.0/24|264409 1031 262504 267613 13335|IGP|45.184.147.74|0|0|65100:7010|false|13335|162.158.177.1|route-views.fortaleza
–
A|1719519096.059558|190.15.124.18|61568|1.1.1.0/24|61568 262504 267613 13335|IGP|190.15.124.18|0|0|1031:1031 1031:4209 1031:6045 1031:7019 1031:8010|false|13335|162.158.177.1|route-views3
–
A|1719519128.843415|190.15.124.18|61568|1.1.1.0/24|61568 262504 267613 13335|IGP|190.15.124.18|0|0|1031:1031 1031:4209 1031:6045 1031:7019 1031:8010|false|13335|162.158.177.1|route-views3

Here the AS path ends with AS13335 as origin, so the announcement is RPKI Valid. The problem is the path itself: this is a route leak, not a hijack.

Why it qualifies as a route leak

Consider the example path 199524 1031 262504 267613 13335. AS267613 is a functional peer of AS13335 and should only pass 1.1.1.0/24 to its customers (its AS Cone), not to peers or upstreams. AS262504 is a customer of AS267613, so the announcement between them is legitimate up to that point. The leak starts at AS262504, which forwards the prefix to its upstream AS1031. AS1031 then redistributes the announcement to peers at multiple global peering points.

BLOG-2469 Embedded Image - cXypjx

AS262504 is the original leaking network. AS1031 (Peer-1 Global Internet Exchange) accepted the customer's announcement and re-advertised it to the wider peer community. Cloudflare is not a customer of AS1031, so the prefix should never have been passed from AS1031 to its peers, route servers, or upstreams. The evidence points to AS1031 performing no extensive filtering on customer BGP sessions, matching only on adjacency and redistributing everything that meets that criterion. The leak's impact was severely amplified by AS1031 and others that accepted and propagated a path that should have died at the AS262504 boundary.

For most of the event, the leak funneled requests through AS267613, which was discarding 1.1.1.1/32 traffic. The leak therefore extended the reach of the blackhole. Cloudflare disabled peering in several locations with AS267613, which narrowed the problem but did not eliminate it; AS262504 continued leaking a stale path pointing to São Paulo. Traffic landing in São Paulo could still be served, with high round-trip times. Cloudflare is engaging the involved networks, plus the Tier-1 provider that accepted the unauthorized blackhole, about remediation and prevention.

Prevention and response measures

BGP hijack defense

RPKI origin validation. RPKI has passed 50% deployment for prefixes with signed ROAs. That still leaves gaps, and during this hijack, multiple networks accepted and forwarded an RPKI Invalid route. Large networks with many downstream autonomous systems carry particular responsibility for rejecting such announcements.

RPKI and RTBH. The unauthorized RTBH action by the Tier-1 provider is itself a hijack with outsized impact. RTBH filtering typically relies on loose AS-SET checks against Internet Routing Registries, not on direct authority to blackhole a destination. Relying on ROAs for RTBH filtering would be impractical at Cloudflare's scale; it would require thousands of ROAs, and allowing /32 entries invites impersonation attacks where a false AS13335 announcement for 1.1.1.1/32 becomes globally preferred. AS-SET objects do not certify authority to blackhole. The expired IETF draft draft-spaghetti-sidrops-rpki-doa-00 proposed a Discard Origin Authorization (DOA) object precisely to authorize specific origins for blackhole actions. Had such an object existed and been validated against RTBH requests, the Tier-1 provider would have rejected AS267613's blackhole attempt as invalid.

Basic BGP hygiene. Adherence to MANRS best practices — specifically the rejection of IPv4 prefixes longer than /24 in the DFZ — would have substantially curtailed the impact on 1.1.1.1. Invalid prefix-length filtering should be routine for every network.

Route leak mitigation

Route leaks cannot be fully avoided today because the Internet's interconnection model is trust-based. Cloudflare, however, is acting to reduce exposure. Its route leak detection system now ingests additional data sources for wider network coverage, with real-time feeds being integrated for faster reaction to similar events.

ASPA for BGP. Cloudflare continues to advocate for RPKI-based AS path validation through Autonomous System Provider Authorization (ASPA). ASPA objects sign an AS with the list of providers allowed to propagate its routes — a parallel to ROAs, but for AS-level authorization rather than prefix-origin. With ASPA in place, the leak path where AS262504 shared 1.1.1.0/24 upstream would be caught, provided AS267613 signed its authorized providers and AS1031 validated paths against that list. Full ASPA adoption will take time, especially for major providers to reject invalid AS paths.

Alternative mechanisms. RFC9234 defines BGP peer roles and the "Only-To-Customer" (OTC) attribute, which can stop the upstream propagation of prefixes like 1.1.1.0/24 that have crossed a role boundary. Its effectiveness depends on role assignment for each peering session and on vendor support maturing for production deployment. All route-leak solutions share a common dependency: cooperation and correct configuration across the Internet's operating networks.

Lessons and the path forward

The 1.1.1.1 outage came from a concurrent BGP hijack and route leak initiated by external networks. Cloudflare cannot control those networks, but it can sharpen its own detection and advocate for structural improvements. RPKI origin validation has established deployment across major networks; AS path validation via ASPA remains an IETF draft. Users can check whether their own ISP validates routes against RPKI at isbgpsafeyet.com via the Test Your ISP facility.