Why One Authority Wasn’t Enough
Cloudflare runs its internal infrastructure zone—the DNS zone holding hostnames for every machine at the edge and in core data centers—on its own authoritative DNS product. That setup delivers the performance and redundancy of the global anycast network, but it introduces a single point of failure: if the edge nameservers become unreachable during an incident, internal resolvers can’t reach the only configured authority for the zone. Engineers already busy debugging an outage would then face DNS resolution errors when trying to connect to affected infrastructure, adding confusion and delays to the response.
Hosting the zone with a single external provider wouldn’t solve the underlying problem—that provider could experience its own outage. The DNS community’s standard advice is to diversify network paths, routing, and software between primary and secondary servers, but in a traditional setup one authority still owns the zone. The fix is to remove the concept of a single authority entirely.
Multiple Primaries, No Zone Transfers
Cloudflare’s solution was to configure the infrastructure zone with multiple primary nameservers—a split-authority, or primary/primary, setup. Three nameservers from an additional provider were added at the registrar alongside Cloudflare’s own edge nameservers. Each provider operates independently; there is no DNS AXFR/IXFR replication between them. Instead, zone changes are pushed to both providers simultaneously using OctoDNS, GitHub’s DNS management tool, which Cloudflare now uses to keep the zone synchronized across both authorities.
Enforcing Query Routing with DNSDist
With two authorities, Cloudflare needed to control which provider handles each query—something that can’t be coordinated at the DNS protocol level. Recursive resolvers pick among multiple nameservers using their own criteria, often lowest round-trip time (RTT). That doesn’t guarantee traffic stays on the preferred path.
Cloudflare’s edge infrastructure servers use DNSDist as their primary system resolver, load-balancing queries to multiple local Unbound instances. This resolver setup is internal only and separate from the company’s public 1.1.1.1 service. DNSDist was reconfigured with two server pools for the infrastructure zone’s authoritative servers—one for Cloudflare, one for the external provider. Active health checks with weighted routes ensure queries are always directed to the Cloudflare pool when it’s available, keeping RTT low since the authoritative servers share data centers with the resolvers. If Cloudflare’s authoritative DNS fails the health check, DNSDist automatically reroutes all zone queries to the external provider.
OctoDNS-Based Zone Management
The move to a multi-primary setup required abandoning traditional DNS replication in favor of an external management workflow. Cloudflare had previously used a manual provisioning tool that queried its provisioning database and updated DNS records via the Cloudflare API. The tool had grown slow with the number of records and required a human review step in a chat room before changes were applied.
OctoDNS’s pluggable architecture let Cloudflare rewrite that old script as a custom source plugin, then use existing provider plugins to push changes to both the Cloudflare API and the external provider’s API. Adding another provider later would only require a configuration change. Because OctoDNS syncs the full zone state, any records created outside OctoDNS—manual changes, for example—are removed during operation, guaranteeing the zone never drifts from the provisioning database’s source of truth.
Cloudflare chose to run its OctoDNS workflow through its existing TeamCity CI/CD infrastructure rather than a standalone tool. TeamCity offered several operational benefits: it is already managed as a service by the DevTools team, has granular per-user permissions, retains audit logs, allows simple rollback of zone revisions, and integrates with chat ops via Google Chat webhooks.
The zone management workflow has three phases:
- Build — evaluate the sources and assemble the complete zone.
- Compare — diff the built zone against live records on both providers and notify SRE of any changes.
- Deploy — push approved changes to the providers.
Build
During the build phase, OctoDNS consumes data from two custom source modules:
- ProvAPI — queries the internal provisioning API for data center and node configuration.
- NetBox — queries the internal NetBox deployment for hardware metadata.
Static records are defined in a YAML file. Together these sources represent the full infrastructure zone state. The build process also maintains a per-execution staging area, generating a static YAML file containing the complete zone and an emergency hosts file. Both outputs are revision-controlled in CI, so prior versions can be redeployed if needed.
Compare
After a successful build, CI runs a dry-run octodns-sync against the staged YAML configuration, comparing zone records against the live records at both authoritative providers via their APIs. Any differences are parsed into a summary line that is posted to the SRE chat room for review, with links to the relevant CI build for deployment.
Deploy
The deployment build is access-controlled and scoped to the SRE group via single sign-on. Following approval and peer review, an SRE executes the deploy build, which consumes the staged YAML data and pushes changes to both authorities with octodns-sync --doit. The emergency hosts file generated during the build is also packaged and deployed so it is available in case of a complete DNS failure.
What’s Next
Cloudflare is looking to cut the approval step out of the process for record additions, which are common during machine provisioning and are generally harmless. Automating that path would remove one more manual step from the provisioning workflow. The introduction of OctoDNS and the external provider has already made the infrastructure zone more resilient and easier to maintain, and the pluggable source architecture means new record data sources can be added without reworking the management pipeline.
