What happened

On September 17, 2024, Cloudflare inadvertently withdrew fifteen IPv4 prefixes from BGP during routine maintenance, cutting off IPv4 connectivity for roughly one hour for a subset of Business plan websites. Traffic to those addresses stopped reaching Cloudflare, and users attempting to reach affected sites received errors. The incident was not caused by an attack but by an internal software error.

The renumbering workflow

Cloudflare's Addressing team regularly performs “renumbering” exercises to move customer websites between sets of IPv4 addresses, freeing up increasingly scarce address space for other purposes like deploying infrastructure in new locations. A set of IP addresses with no websites associated to it can be retired, after which those addresses are free for reuse.

In July 2024, a batch of Business plan websites was moved from an original set of IPv4 addresses to a smaller set matched to forecast plan requirements. The September 17 maintenance step was intended to detach the original IPv4 prefixes from Cloudflare's network and withdraw them from the global routing table via BGP, making them available for redeployment elsewhere.

Root cause

The July migration used a one-time template that cycles through all websites associated with the old IP addresses and moves them to new ones, calling a function that updates the IP address-to-website mapping. A couple of months prior, that function had been updated as part of unrelated legacy SSL configuration work. The update replaced legacy code synchronizing two address pools with a call to an existing synchronization function.

That fix had an unintended side effect: it suppressed updates in cases where there was no linked reference to a legacy SSL certificate. Because not every website uses legacy certificates, 1,661 customer websites remained linked to addresses in the pools slated for withdrawal. This was not caught during the July renumbering, which proceeded on the assumption — never verified — that every website had been moved.

At 2024-09-17 17:51 UTC, the fifteen IPv4 prefixes were withdrawn via BGP. Cloudflare's global network spans hundreds of data centers, so withdrawal timing varied by region. Within ten minutes, network-wide traffic to the affected prefixes dropped by an aggregate 10 Gbps.

Timeline of the incident

All timestamps are UTC on 2024-09-17:

  • 17:41 — The Addressing team initiated the production release that began disabling prefixes.
  • 17:51 — BGP announcements started being withdrawn; traffic to the impacted prefixes began dropping.
  • 17:57 — SRE alerts triggered by the increase in unreachable IP space were investigated, but the situation appeared consistent with normal address removal, so the investigation quickly closed.
  • 18:36 — Two customer escalations arrived and an incident was declared. A limited deployment window was put in place.
  • 18:46 — Addressing engineers confirmed the renumbering release change had caused the incident and began rollback preparations.
  • 18:50 — The release was rolled back, the affected prefixes were re-announced in BGP, and traffic began flowing again.

There was no impact to IPv6 traffic. Only the 1,661 customer websites associated with addresses in the withdrawn IPv4 prefixes were affected.

Immediate fix and follow-up work

The immediate fix was rolling back the release — a pragmatic first step, since all approved changes have tested rollback procedures. Identifying the root cause was more involved. The code modified earlier in the year is part of a legacy system the Addressing team has been working to replace since the team's formation in 2019. Most engineering effort has gone into building the modern replacement rather than deep dives into legacy code.

The specific bug has been fixed, but Cloudflare acknowledges that relying on old, less-understood code is a general risk. The plan going forward addresses that in three areas:

  • Test: Adding checks that confirm no websites remain assigned to old IP addresses as part of future renumbering exercises — eliminating the incorrect assumption that all websites had been migrated.
  • Process: Improving prefix withdrawal procedures, regardless of context, to include automated and manual verification of traffic levels associated with addresses being withdrawn. Where more detailed logging is available, request logs will be checked for signs that purportedly unused addresses still carry active traffic.
  • Implementation: Reviewing every use of stored procedures and functions tied to legacy systems. Where confidence is low, functionality will be re-implemented with modern documentation and test coverage.