Why DNS propagation speed matters
Cloudflare's DNS infrastructure now handles 38.7 trillion queries per month across more than 270 cities, and the platform serves as the DNS provider for roughly 15% of all websites. While query response time gets most of the attention, there is a second performance metric that matters just as much for customers: DNS record propagation time — how long it takes for changes submitted via the API to show up in live DNS responses. Faster propagation means customers can adjust configuration quickly and keep their systems agile.
The propagation pipeline has several stages, and historically the biggest bottleneck sat in the Zone Builder, the component responsible for collecting zone contents and writing them to Quicksilver, Cloudflare's distributed KV store that feeds the global network. The impact was most pronounced for large zones, where a single build could take tens of seconds.
The original build model and its limits
The Zone Builder was designed when the platform received 5-10 DNS record changes per second. Today that number averages 250 per second — about 25x growth — and individual customer zones have scaled up along with the platform, with the largest zones containing millions of records.
The original approach was a full build: whenever a zone changed, the Zone Builder fetched all records for that zone from the database, compared them with what was stored in Quicksilver, and fixed any differences. Full builds had a useful property — since each build grabs the entire zone, multiple change events for the same zone could be batched and deduplicated. Ten record changes meant ten events, but only one build was needed after the final change landed.
The problem was cost. A full build for a zone with one million records could take up to 35 seconds, mostly due to database query latency. The Zone Builder also had to pull all records from Quicksilver for comparison, adding more time. And the load didn't affect only the customer making the change — it consumed database resources shared by other services and slowed down builds for other zones.
Moving to per-record builds
The obvious fix was to have the Zone Builder process only the record that actually changed, rather than rebuilding the entire zone. Getting there required changes across several layers of the pipeline:
- A new Kafka event pipeline for record changes that includes information about which record changed.
- A new scheduler interface that separates the Zone Builder's different responsibilities.
- A per-record scheduler that processes events sequentially in the correct order.
- A new Quicksilver interface to support single-record writes.
Splitting the Zone Builder introduced an important synchronization requirement. Per-record builds and full builds must be locked against each other; otherwise the full build scheduler could overwrite newer per-record changes with stale data.
This architecture also depends on Cloudflare's black lie approach to negative answers with DNSSEC. The conventional method for serving negative answers requires all records in a zone to be canonically sorted, so adding a single record would mean fetching the entire zone just to determine its insertion point. The black lie method removes that constraint entirely.
One bug worth noting
The transition wasn't entirely smooth. The full build model made cleanup trivial because the Zone Builder always knew exactly which records existed in both the database and Quicksilver. Per-record builds changed that calculus — creates and deletes were simple, but record updates only carried the new record data. When a customer renamed a record through the DNS Records API, the Zone Builder had no way to know which old record to remove from Quicksilver, leaving stale data behind.
The fix was to replace those update events with paired create and delete events so the Zone Builder has enough information to clean up properly.
Measured results
All DNS Records API changes now go through per-record builds. Full builds still exist but only for zone settings that require knowledge of the entire zone's contents, representing roughly 13% of total DNS builds.
On average, per-record builds are 150x faster than full builds when comparing both database query time and Quicksilver write time. The gap widens as zones grow. Testing with a one-million-record zone showed per-record build times no higher than 8ms across five runs, while full builds averaged 34 seconds — a reduction of 4250x.
These gains benefit all customers regardless of zone size, and the reduced load on database and Quicksilver resources means other systems on the platform operate with more headroom.
On the roadmap
There is room for further improvement. The plan is to eliminate full builds entirely in favor of dedicated zone setting builds that fetch only the settings and propagate them via Quicksilver, without pulling every record. The challenge is the complexity of zone settings and the number of actors that can modify them.
The team also intends to add a batching system that groups record changes together, cutting down the number of database and Quicksilver queries required.
