Secondary DNS - Deep Dive

The Mechanics of Secondary DNS

Secondary DNS is a mechanism for replicating DNS zone data from a primary server to one or more secondary servers. While its traditional role was providing a synchronized backup when the primary was unavailable, modern implementations emphasize redundancy across multiple nameservers, often sharing the same anycasted IP address. The process involves a one-way transfer of zone data from the primary to secondaries, keeping all servers in sync as zone contents change.

Every change to a zone triggers an increment in its Start of Authority (SOA) serial number. This serial is the core synchronization mechanism—when a secondary observes a higher serial than what it has stored, it knows a zone transfer is required. The SOA record's fields govern this behavior:

1. Serial - Monotonically increasing value that changes with every zone modification.
2. Refresh - Maximum seconds a secondary waits before polling the primary for a serial change.
3. Retry - Maximum seconds before re-checking after a failed contact with the primary.
4. Expire - How long a secondary may serve stale data if the primary is unreachable.
5. Minimum TTL - Per RFC 2308, how long negative DNS responses are cached.

Secondaries detect serial changes through two mechanisms. The faster path is the NOTIFY message (RFC 1996): the primary sends an unsolicited message when a zone changes, prompting an immediate transfer. The fallback is polling based on the Refresh interval. Since NOTIFY is an open protocol, any Internet host could theoretically trigger zone transfers. Mitigations include transaction signatures (TSIG, RFC 2845), which authenticate notification messages, and IP-based access control lists, though the latter restricts server placement flexibility.

Zone Transfer Protocols

Zone transfers themselves follow two standardized protocols, neither of which provides confidentiality, authentication, or integrity natively. TSIG can be layered on top to supply authentication and integrity, but since DNS is inherently public, confidentiality is rarely a requirement.

Authoritative Zone Transfer (AXFR)

AXFR is the original transfer protocol from RFC 1034/1035 (clarified in RFC 5936). It runs over TCP to guarantee reliable delivery. An AXFR transfers the complete zone contents—all records—in a single connection, regardless of the current serial numbers. This makes it the appropriate choice for the initial transfer when a secondary has no data yet.

Incremental Zone Transfer (IXFR)

IXFR (RFC 1995) is more efficient for ongoing synchronization. The secondary sends its current serial, and the primary responds with only the changes since that version. IXFR responses follow a strict structure of SOA record pairs and delta sets:

1. Current latest SOA
2. Secondary's current SOA
3. Deleted records
4. Secondary's current SOA plus changes
5. Added records
6. Current latest SOA

Steps 2 through 6 may repeat to represent multiple incremental change sets. While IXFR technically works over UDP, TCP is commonly preferred to avoid packet loss.

Cloudflare's Secondary DNS Implementation

Cloudflare originally built its Secondary DNS service on Mesos Marathon, separating concerns into several independently scalable microservices running in core data centers:

• Zone Transferer - Attempts IXFR, falling back to AXFR on failure.
• Zone Transfer Scheduler - Periodically checks zone SOA serials for updates.
• Rest API - Handles registration of new zones and primary nameservers.
• Notify Listener - An external app that receives primary NOTIFY messages and triggers the Zone Transferer. This service sat outside the Marathon cluster.

These services communicate through Kafka messaging. Once a zone transfer completes, the data flows through a zone builder and gets pushed to Cloudflare's edge network at each of its data centers worldwide. This microservice approach worked initially but introduced vulnerabilities and scaling constraints as the product grew.

Cloudflare responded by migrating its core services to Kubernetes, moving all Marathon-based applications plus the Notify Listener into the new orchestration framework. The migration presented significant engineering challenges to achieve a seamless, zero-downtime transition, but ultimately modernized the architecture to better handle the demands of a widely adopted Secondary DNS product.

Scaling the Migrated System

Moving the Notify Listener, Zone Transferer, and supporting infrastructure into Kubernetes solved the orchestration and scaling questions, but created two fresh problem areas that needed dedicated attention before production traffic could flow.

Stable Egress IPs for Zone Transfers

Primary DNS servers commonly protect themselves by allowing zone transfers only from explicitly configured IP addresses. A Kubernetes service — with pods being created and destroyed continuously — lacks stable source IPs. We evaluated five options: open-source Kubernetes controllers for static egress IPs, NAT entry changes, running zone transfers outside Kubernetes, updating primary-server ACLs with the full Cloudflare IP range, and proxying egress traffic.

Proxying won. We settled on Shadowsocks-libev as a SOCKS5 proxy, selecting it for its speed, security, track record of scaling, and support for UDP/TCP and IPv4/IPv6. This approach gives us horizontal scaling, efficient load balancing, a one-time ACL update on primary servers, and a proxy that other Cloudflare services can reuse.

Protecting the Public-Facing Notify Listener

The Notify Listener waits on static IPs for NOTIFY messages from primary servers, and exposing a raw UDP/TCP service to the internet invites DNS floods and other abuse. Our DDoS protection is one of our core strengths, so we put Spectrum — our own reverse proxy — in front of the Notify Listener. Spectrum terminates inbound TCP/UDP, filters malicious traffic, makes optimal routing decisions from edge to core, and handles dual-stack (IPv4/IPv6) connections.

This setup has two specifics worth noting. First, Spectrum listens on both IPv4 and IPv6, but terminates each connection and opens a new, IPv4-only connection to our Kubernetes cluster because our load balancer does not yet support IPv6. Second, Spectrum routes based on the L4 protocol, which matters because Kubernetes services of type LoadBalancer support only one of TCP, UDP, or SCTP per service.

An L4 proxy introduces the classic problem of source-IP loss. Without the true source addresses, we cannot determine which primary server sent a NOTIFY message, which would expose us to attack. Spectrum’s proxy-protocol feature adds source IP and port headers to each TCP/UDP packet, and we used it extensively. The forwarding mechanism, however, required care because the standard DNS library we rely on (miekg/dns) would reject incoming DNS messages carrying the extra proxy headers.

A custom pair of read and write decorators solved this:

• Reader: Extracts source address data from inbound NOTIFY messages and inserts that information into the additional section of the DNS message as new records.
• Writer: Strips those additional records from outbound replies and reassembles them using proxy-protocol headers.

Spoofing those records is not possible because the DNS server permits only two extra records, one of which is reserved for TSIG; anything else is overwritten. Using these decorators keeps the proxying opaque to the Notify Listener itself.

Even with accurate source IPs, NOTIFY uses both UDP and TCP and therefore remains prone to IP spoofing. As a final safeguard against flooding primary servers with spurious zone-transfer requests, the Zone Transferer only initiates a transfer after confirming the SOA serial actually changed, and it never allows more than one active and one queued transfer per zone.

The Kafka Scheduling Problem in the Zone Transferer

Multiple producers — an HTTP API and the Notify Listener — can push zone-transfer requests to a Kafka topic. A zone that has just been transferred needs no immediate retransfer, so keeping a backlog of queued work per zone is wasted effort; at most one transfer in flight and one scheduled per zone is sufficient.

Constraining scheduled messages to one per zone requires deliberately not committing some Kafka offsets immediately. The normal message-commit contract does not make this straightforward. A Kafka consumer commits a message’s offset once it finishes processing; since consumers can process different messages from the same topic concurrently, offsets are not necessarily committed in order. That, combined with the ephemeral nature of Kubernetes pods, can lose work permanently:

1. Read offset 1. Start transferring zone 1.
2. Read offset 2. Start transferring zone 2.
3. Zone 2 finishes; commit offset 2, which also marks offset 1 done.
4. The pod restarts.
5. Read offset 3. Start transferring zone 3.

Zone 1 now never transfers. Unsynchronized zones would eventually serve stale data. The answer is a soft-commit in the consumer itself:

1. Track processed Kafka messages in a list sorted by offset.
2. Remove a message from the list once its zone transfer finishes.
3. Commit the offset only if the finished message is the oldest still in the list.

The scheme commits messages only when all earlier messages in the list have confirmed processing. It only works distributed if messages are keyed by zone ID so the same zone is always consumed by the same consumer.

Path of a Secondary DNS Request

Transfers run from our core data centers rather than from every edge location. The pipeline after a transfer reaches the edge in two more steps:

1. Zone Builder: Rebuilds the zone into the format the Cloudflare edge expects, then writes the result to Quicksilver, our distributed key-value store.
2. Authoritative DNS servers: Read the built zone from Quicksilver and respond to queries.

Measured Performance

Secondary DNS is part of our Authoritative DNS offering, which at publication ranks first globally for performance at dnsperf.com. Breaking down latency across the pipeline:

1. Primary to Notify Listener: instrumentation precision caps measurement at one second, though UDP/TCP round trips will take far less.
2. Notify Listener to Zone Transferer: negligible.
3. Zone Transferer to Primary (zone transfer): ~800ms median for 99% of transfers.

4. Zone Transferer to Zone Builder: roughly 10ms to rebuild a zone for 99% of requests.

5. Zone Builder to Quicksilver: under one second for 95% of propagation updates.

Synthesizing all components, end-to-end latency from a primary-server record change to global edge propagation runs under five seconds on average. External probes underestimate real-world timing since they are constrained by polling intervals, geographic spread, provider differences, and zone count.

A manual test tightened the picture:

• Primary server: NS1
• Record changes: one
• Timing started: at the NS1 record change
• Timing stopped: upon observing the change via dig at a Cloudflare edge
• Measured time: 6 seconds

Cloudflare handles 15.8 trillion DNS queries monthly, serving 99% of the internet-connected population within 100ms. The goal of Secondary DNS remains making that infrastructure available to customers running their own DNS or using another provider — and, with Secondary Override, extending Cloudflare’s proxying and security to those secondaries as well. Secondary DNS is an Enterprise plan feature; setup details are covered in our support documentation.