Spectrum, Dogfooding, and the Problem With Cloudflare IPs as Origins
Cloudflare’s internal teams are encouraged to “dogfood” the company’s own products, and Spectrum — the layer 4 reverse proxy — is no exception. Many internal services sit behind Spectrum to take advantage of its DDoS protection. But for those teams, the onboarding experience was a far cry from what an external customer gets. A typical customer can spin up a Spectrum app in under a minute through the Cloudflare Dashboard. An internal team needed an engineer from the Spectrum team to manually update a configuration file managed through Salt, Cloudflare’s configuration management system. The process took roughly a week.
Why the disconnect? And could it be fixed? Those questions became the focus of an intern project on the Spectrum team.
Why the Manual Step Exists
Cloudflare operates many IP ranges, and customers can also authorize Cloudflare to announce their own prefixes (BYOIP). Collectively, these are referred to as managed addresses. Cloudflare prohibits managed addresses from being used as Spectrum origins — a policy designed to prevent a class of problems that includes proxying loops.
Consider two Spectrum apps, A and B, where the origin of app A is the edge IP of app B, and the origin of app B is the edge IP of app A. Each app would proxy incoming connections to the other’s edge, creating a cycle that never reaches an actual origin. This can crash the daemon or degrade performance. In practice, such a configuration is either useless or malicious. Even the more general case — where one Spectrum app points to another as its origin without creating a cycle — is almost never needed and therefore disallowed.
The system also blocks IP ranges that cannot be used on the public Internet, as defined in RFC 6890.
For external customers, checking against a simple deny list of managed addresses worked well. But internal services all use Cloudflare IPs. They have zero legitimate reason to point at a public origin that isn’t a Cloudflare address. To get around the deny list, those internal teams had to rely on a customized Spectrum configuration that had to be deployed to the edge through a pull request to the Salt repository. Changing that configuration demanded the same tedious, error-prone process.
The Fix: Move Validation to the Addressing API
The solution to this problem lay in Cloudflare’s Addressing API, an internal database and toolset that manages IP prefixes across the organization. It serves as a unified source of truth for how IP addresses are used, powering features like BYOIP and BGP On Demand. For every Cloudflare-managed prefix, the Addressing API tracks its owner and any delegations — explicit grants permissioning another party to use a prefix.
A user’s permission to use an IP is determined by two questions:
- Is the user the owner of the prefix containing the IP? If yes, they have permission.
- If not, has the user been delegated a prefix containing the IP? If yes, they have permission. If no, they do not.
With that information, the new validation logic becomes:
- Is the IP a Cloudflare-managed address (or in the RFC-denied list)? If no, allow it as an origin.
- If yes, does the customer have permission to use the IP? If yes, allow it. If no, deny it.
This approach lets internal customers specify Cloudflare IPs as origins, as long as those IPs are delegated to them in the Addressing API. There is, however, a corner case: BYOIP customers have permission to use their own ranges, so they could technically designate their own IP as an origin and create a proxying loop. To block that, the system also checks whether the IP is already in use as a Spectrum edge IP and denies it if so.
Since all the deny-network checks now live in the Addressing API, Spectrum’s own deny network database was removed entirely, cutting down on engineering maintenance.
In practice, a request to the Addressing API for an internal customer wanting to use 104.16.8.54/32 as an origin would pass because the address is managed, the customer is delegated permission, and the IP is not currently an edge IP.
Performance and Observability
Moving validation to the Addressing API meant adding another HTTP request to the critical path of every create-app request in the Spectrum configuration service. Basic performance tests showed an expected increase of about 100ms per request. That prompted the team to take a closer look at the impact of different HTTP requests on the critical path.
Using OpenTracing, the standard for distributed tracing across microservices, the Spectrum config service could track a request from start to finish: how long a SQL query takes, the duration of a function call, and the time spent at any given service. After instrumenting the service, the team found that the Addressing API accounted for only a very small fraction of the overall request time, and the added tracing helped surface potentially problematic request times in other services.
Lessons From the Project
The project was a reminder of two critical habits: reading documentation and writing it. A thorough understanding of both the Addressing API and the configuration service was essential for integrating an endpoint that fit the existing architecture. And when onboarding the internal Crossbow diagnostics tool to Spectrum using the new features, the written guide required feedback from the Crossbow team to smooth over unclear areas.
The bigger lesson was that even straightforward validation logic has hidden complexity. Implementing it properly required understanding how multiple microservices interact to validate a configuration, and how data moves from the Core to the Edge and gets processed there. Reaching that level of understanding proved just as valuable as the completed project itself.
