Making BYOIP prefixes work across multiple services

IPv4 address space has become an expensive asset. With the original pool exhausted, organizations increasingly turn to the secondary market, where prices regularly exceed $30–$50 per address and continue climbing. That scarcity makes efficient address management critical. Cloudflare has updated how Bring Your Own IP (BYOIP) prefixes are handled to give customers more flexibility in how they use their address space.

Previously, a BYOIP prefix had to be bound to a single service at onboarding — CDN, Spectrum, or Magic Transit — and the binding could not change without offboarding and re-onboarding the prefix. Customers wanting to use the same address block for a different service had to start over. Now, portions of any onboarded prefix can be used for additional services, allowing businesses to make better use of the addresses they already own.

The engineering behind dynamic service migration

Migrating prefixes between services at Cloudflare’s scale requires solving problems across IP address management, kernel-level bindings, and orchestration across thousands of servers.

Updating firewall rules dynamically

Each service has distinct traffic expectations. CDN, for example, opens TCP ports 80 and 443 for HTTP and HTTPS. Servers use Linux iptables and IP sets to manage which prefixes are allowed through for which service. Moving a prefix between services means reassigning it to different IP sets and firewall rules across the distributed fleet. Servers must update these dynamically to avoid routing loops or dropped connections.

Extending Tubular's bindings

Most web applications bind to a fixed list of IP addresses at startup. To let customers change bindings without restarting services, Cloudflare built Tubular, an eBPF-based program that allows services to listen on a single socket while dynamically updating the set of addresses routed to it.

The challenge was extending Tubular to handle CDN traffic. Cloudflare's CDN relies on an NGINX ingress proxy that terminates incoming connections. NGINX does not support systemd socket activation, which is the mechanism Tubular uses to discover sockets owned by other services. Without that integration path, Tubular could not detect and route traffic to the ingress proxy.

The solution was a shared Unix domain socket between Tubular and the ingress proxy on each server. Through this channel, the ingress proxy explicitly sends socket information to Tubular, allowing it to register the sockets in its datapath.

Deploying this integration required care. As of April 2025, Cloudflare handles an average of 71 million HTTP requests per second, peaking at 100 million. The configuration changes were staged across all servers without disrupting active connections, and bindings for CDN prefixes were gradually enabled in a controlled rollout.

The data plane in action

In the standard case, a binding ties together the Layer 4 protocol (TCP), a prefix like 192.0.2.0/24, and a port number — port 0 meaning any port. Incoming packets matching that combination are directed to the appropriate socket, for example the Spectrum service.

When an individual address is upgraded to CDN via the Service Bindings API, Tubular consumes that update and adds a new entry to its data plane. With Longest Prefix Match, all traffic in the /24 for any port continues to Spectrum, except the specific /32 address and port 443, which routes to CDN.

Orchestrating transitions

Currently, when a customer migrates a prefix between services, there is a 4–6 hour window where incoming packets may be dropped. Cloudflare is working on systems to reduce that transition time from hours to minutes, improving reliability and minimizing disruption during migrations.

Finer-grained service bindings

Service Bindings control whether traffic destined for a given IP address is routed to Magic Transit, CDN, or Spectrum. Previously only whole prefixes could be mapped. Now sub-ranges can be selectively assigned.

Consider a customer using BYOIP with a /24 bound to Spectrum for DDoS protection of TCP and UDP traffic, but only using a few addresses for actual applications. The same customer wants Static IPs for CDN so their own customers can allowlist a consistent set of addresses. Instead of acquiring another block, they can carve out unused sub-ranges of the /24:

  • 192.0.2.0/24 remains bound to Spectrum
    • 192.0.2.0/25 is bound to CDN
    • 192.0.2.200/32 is also bound to CDN

Both the /25 and the /32 are within the /24 prefix and will receive CDN traffic. All remaining addresses continue to use the default Spectrum binding. The setup is done through a few HTTP requests to the Cloudflare API, specifying which addresses or subnets should route to which service. The process is the same whether configuring a single address or splitting up larger subnets.

The flexibility extends beyond coarse service-level choices. A single prefix could simultaneously serve web traffic via CDN, protect infrastructure through Magic Transit, and support Spectrum workloads — with granularity down to the port level. The rigid, static model is giving way to programmable traffic orchestration across services. Existing BYOIP customers using Magic Transit, CDN, or Spectrum can find the configuration guide for more details.