A Route Leak Behind the OVHcloud Outage
On October 30, 2024, OVHcloud (AS16276) experienced a brief outage on its backbone infrastructure. The incident ran from 13:23 UTC to 13:40 UTC, according to OVHcloud's incident report. For Cloudflare, the impact was visible through traffic patterns from OVHcloud, which acts as an origin for some customer sites when assets are not already cached.
Cloudflare observed traffic from OVHcloud starting to decline at 13:21 UTC, slightly before the official start time. By 13:28 UTC, traffic was roughly 95% lower than pre-incident levels. Recovery began around 13:31 UTC, and by the time OVHcloud declared the incident over at 13:40 UTC, traffic had recovered to about 50% of normal levels.
Routing Shifts and Peering Disruptions
Under normal conditions, Cloudflare and OVHcloud exchange most traffic over direct peering links. During the incident, peered traffic volumes dropped sharply, with a small amount of traffic briefly shifting to transit links due to sudden changes in which Cloudflare data centers received OVHcloud requests.
OVHcloud's routing to Cloudflare disappeared entirely for a few minutes, then reappeared through a single Internet Exchange port in Amsterdam before normalizing globally. Under normal operations, most OVHcloud traffic arrives at Cloudflare's Frankfurt and Paris data centers, reflecting OVHcloud's large regional presence. The routing shifts caused a spike in traffic through Cloudflare's Amsterdam data center, suggesting that BGP reconvergence or network recovery within AS16276 began near that Amsterdam peering point.
Root Cause: A BGP Route Leak
OVHcloud's postmortem attributed the incident to a network configuration error pushed by one of its peering partners, noting that routes were reconfigured to restore traffic. A likely mechanism is a BGP route leak: the peering partner may have sent a full Internet table, overwhelming OVHcloud's network or causing unexpected internal BGP route updates.
Cloudflare's investigation pointed to Worldstream (AS49981) in Amsterdam. A maximum prefix-limit threshold on Cloudflare's peering session with Worldstream was breached, automatically putting the BGP session into an idle state and preventing the route leak from affecting Cloudflare's network.
Oct 30 13:16:53 edge02.ams01 rpd[9669]: RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer 141.101.65.53 (External AS 49981) changed state from Established to Idle (event PrefixLimitExceeded) (instance master)Analysis of BGP Monitoring Protocol (BMP) data from AS49981 before the session shutdown confirmed that Worldstream was advertising routes with AS paths containing their upstream Tier 1 transit provider. During this period, over 500,000 BGP announcements were detected from AS49981, visible on Cloudflare Radar.
Worldstream later acknowledged the issue on its status page, stating:
"Due to a configuration error on one of the core routers, all routes were briefly announced to all our peers. As a result, we pulled in more traffic than expected, leading to congestion on some paths. To address this, we temporarily shut down these BGP sessions to locate the issue and stabilize the network. We are sorry for the inconvenience."
Evidence suggests Worldstream also leaked routes on an OVHcloud peering session in Amsterdam, directly causing the outage.
Preventing Route Leaks
Several methods exist to prevent BGP route leaks from affecting a network. Setting max prefix-limits per peer ensures a BGP session is automatically torn down when a peer sends more prefixes than expected. Forward-looking measures include Autonomous System Provider Authorization (ASPA), where Resource Public Key Infrastructure (RPKI) protects networks from accepting BGP routes with invalid AS paths, and RFC9234, which ties strict customer, peer, and provider relationships to BGP updates. For broader Internet resilience, network operators should follow the recommendations in MANRS for Network Operators.
