Why Cloudflare’s DNS database had to move

As of October 2024, Cloudflare acts as the authoritative DNS provider for 14.5% of all websites. That responsibility starts at the data plane, where DNS records for millions of domains are stored and served. Zone files—the DNS equivalent of a phone book—can grow to millions of records for a single domain, and Cloudflare’s largest zone holds roughly 4 million records. With the majority of zones containing fewer than 100 records, the sheer volume across all customers makes the storage layer a critical concern.

In 2022, DNS data accounted for about 40% of the storage in Cloudflare’s main database cluster, cfdb. This Postgres cluster, which serves most of Cloudflare’s APIs including the widely used DNS Records API, also handled requests from numerous other services. As traffic scaled, spikes in DNS-related database queries began to degrade performance for those unrelated services sharing the same cluster.

The DNS team concluded that system-level optimizations alone could not resolve the strain. In late 2022, they decided to decouple DNS records from cfdb entirely, migrating the data to a dedicated cluster called dnsdb. This effort involved collaboration with 25 other teams across the company.

Untangling zones from DNS records

Since roughly 2017, Cloudflare has progressively moved services off the central cfdb cluster into microservices. DNS presented a unique challenge because zone data and DNS record data are historically tightly coupled. The concept of a “zone” has expanded beyond DNS: zones now store non-DNS settings and link products to customer websites. Migrating zone data and DNS record data together no longer made sense, but separating them required significant architectural changes.

The migration also addressed a longstanding issue: the DNS team had little control over which systems accessed DNS data directly and at what rate. The solution was an internal DNS Records gRPC API, providing a tightly controlled interface to DNS data. This allowed the team to implement centralized rate limiting, caching, and auditing, and to make sweeping changes with a single API modification rather than coordinating across systems. Over 20 services written in 5 different programming languages—covering DNSSEC, TLS, Email, Tunnels, Workers, Spectrum, and R2 storage—were migrated from direct database access to this new API.

Another critical step was logically decoupling DNS database functions from zone data at the SQL level. For example, checking that a zone had not exceeded its maximum record count previously required a SQL query joining record and zone data. Similarly, audit functions needed access to both. By moving these features into separate microservices and up to the application layer, the DNS team eliminated the dependency on the zones database. The new dnsdb cluster still uses SQL functions, but they are fully independent of legacy systems and take advantage of a modern Postgres version.

Building the Change Data Capture and Transfer Service

With the DNS layer decoupled, the Database team took on the data migration of two tables—cf_rec and cf_archived_rec—from cfdb to dnsdb. The design requirements were strict:

  • Don’t lose data. The process had to be auditable, capable of proving that no records were lost.
  • Minimize downtime. The migration needed to complete with less than a minute of downtime, ideally only seconds.

These constraints pointed toward near-real-time change capture, either via logical replication or a custom method. The team initially investigated pgLogical for Postgres logical replication but had concerns about performance and auditability. Further requirements ruled it out entirely:

  • Movement must be bidirectional. The ability to switch back to cfdb without significant downtime was essential in case the new implementation failed.
  • Partition the cf_rec table. Access to cf_rec is mostly by zone_id, so the team partitioned the new table using mod(zone_id, num_partitions) as the key.
  • Provide emergency access from the original database. A foreign table in cfdb would point to dnsdb, allowing any missed processes to access data without a full rollback.
  • Allow writes in only one database at a time. Applications needed to know which cluster was primary and be blocked from writing to both simultaneously.

These requirements led to the development of the Change Data Capture and Transfer Service (CDCTS).

Scale of the data involved

The migration touched two highly active tables. cf_rec stores current DNS record information and is subject to frequent inserts, updates, and deletes. At migration time, it contained 1.7 billion records and, including indexes, consumed 1.5 TB of disk. Daily activity typically included 3–5 million inserts, 1 million updates, and 3–5 million deletes.

The second table, cf_archived_rec, holds obsolete copies of cf_rec records and is insert-only. It grew by roughly 3–5 million records per day, corresponding to deletes from cf_rec. This table held approximately 4.3 billion records at the time of migration.

Fortunately, neither table relied on database triggers or foreign keys, allowing the migration to insert, update, and delete records without side effects or dependency concerns.

BLOG-2780 Hero Image

A sample zone file illustrates the structure of DNS records:

example.com      59 IN A 198.51.100.0
blog.example.com 59 IN A 198.51.100.1
ask.example.com  59 IN A 198.51.100.2

Both tables serve as the source of truth for critical Cloudflare systems, making the migration one of the most data-intensive operations the DNS and Database teams had undertaken.

Change Capture and Transfer Design

The migration itself split into two distinct data movements. The first was the initial copy of all rows from cfdb to dnsdb. The second was the change copy: replaying every insert, update, and delete that occurred in cfdb after the initial copy began. Logical replication would have handled this, but it applies changes in a single-threaded, transaction-by-transaction order — too slow for our needs. A queue-based system was also rejected; any queue would typically replay one change at a time, and we wanted to apply large batches. For clarity, the following describes the process for cf_rec only, but an identical mechanism was used for cf_archived_rec.

Our solution was a simple change capture table. A database trigger loaded rows into this table in real time, and a transfer service migrated batches of thousands of changed records to dnsdb at once. Auditing logic was layered on top to verify that every record made the trip safely.

Capture Table Model

For cf_rec, we created a logging table log_cf_rec with the same columns as the source table, plus four new columns:

  • change_id: a sequence-generated unique identifier for the change record
  • action: a single character indicating whether the record is an [i]nsert, [u]pdate, or [d]elete
  • change_timestamp: when the change record was created
  • change_user: the database user who made the change

A trigger on cf_rec wrote new row values into log_cf_rec on every insert and update; deletes created a 'D' record containing only the primary key.

dns_records=# DELETE FROM  cf_rec WHERE rec_id = 13;

dns_records=# SELECT * from log_cf_rec;
Change_id | action | rec_id | zone_id | name
----------------------------------------------
1         | D      | 13     |         |   

dns_records=# INSERT INTO cf_rec VALUES(13,299,'cloudflare.example.com');  

dns_records=# UPDATE cf_rec SET name = 'test.example.com' WHERE rec_id = 13;

dns_records=# SELECT * from log_cf_rec;
Change_id | action | rec_id | zone_id | name
----------------------------------------------
1         | D      | 13     |         |  
2         | I      | 13     | 299     | cloudflare.example.com
3         | U      | 13     | 299     | test.example.com

Beyond log_cf_rec, we introduced additional bookkeeping tables in both databases:

cfdb

  1. transferred_log_cf_rec: audits batches transferred to dnsdb.
  2. log_change_action: stores transfer-size summaries for comparison with its counterpart in dnsdb.

dnsdb

  1. migrate_log_cf_rec: collects incoming change batches before they are applied to cf_rec.
  2. applied_migrate_log_cf_rec: audits batches successfully applied in dnsdb.
  3. log_change_action: summarizes transferred batch sizes for cross-database comparison.

Initial Copy Mechanics

With change logging active, the initial copy could begin. A full pg_dump or a single multi-hour copy was ruled out: the table structure was changing at the destination, network timeouts were a risk, and a long-running read could impact production. Instead, data was moved in small, resumable pieces — a psql COPY statement piped directly into another psql COPY statement, with no intermediate files:

psql_cfdb -c "COPY (SELECT * FROM cf_rec WHERE id BETWEEN n and n+1000000 TO STDOUT)" | psql_dnsdb -c "COPY cf_rec FROM STDIN"

Before each batch, the record count was written to cfdb. After the batch landed, a count was recorded in dnsdb and matched against the source count so that an interrupted network connection or other failure could not silently drop data. The shell script also checked for touch-files that could pause or abort the copy if production load became a concern.

#!/bin/bash
for i in "$@"; do
   # Allow user to control whether this is paused or not via pause_copy file
   while [ -f pause_copy ]; do
      sleep 1
   done
   # Allow user to end migration by creating end_copy file
   if [ ! -f end_copy ]; then
      # Copy a batch of records from cfdb to dnsdb
      # Get count of records from cfdb 
	# Get count of records from dnsdb
 	# Compare cfdb count with dnsdb count and alert if different 
   fi
done

Applying Changes

Once the initial copy finished, we synced changes that had accumulated during it. A function fn_log_change_transfer_log_cf_rec, given a batch_id and batch_size, performed five operations inside one database transaction:

  1. Selected batch_size records from log_cf_rec in cfdb.
  2. Copied them to transferred_log_cf_rec to mark them as transferred.
  3. Deleted the batch from log_cf_rec.
  4. Wrote a summary row to log_change_action.
  5. Returned the batch for the next step.

The batch was then piped to migrate_log_cf_rec in dnsdb using the same bash copy approach, with the function replacing the SELECT:

psql_cfdb -c "COPY (SELECT * FROM fn_log_change_transfer_log_cf_rec(<batch_id>,<batch_size>) TO STDOUT" | psql_dnsdb -c "COPY migrate_log_cf_rec FROM STDIN"

Applying the batch in dnsdb was handled by a second function, log_change_apply, also running within a single transaction. It:

  1. Moved a batch from migrate_log_cf_rec to a temporary table.
  2. Wrote counts for batch_id to log_change_action.
  3. Retained only the latest record per unique id from the temporary table — an insert followed by 30 updates collapses to just the final update, since intermediate versions need not be applied.
  4. Deleted any rows from cf_rec that had corresponding changes.
  5. Inserted [i]nsert and [u]pdate records into cf_rec.
  6. Copied the batch to applied_migrate_log_cf_rec for a complete audit trail.

The complete cycle comprised four separate transaction-bound steps: pulling a batch in cfdb, copying it to dnsdb, applying it there, and comparing log_change_action counts between the two databases.

image1

This loop ran every three seconds for several weeks prior to the migration, keeping dnsdb continuously synchronized with cfdb.

Coordinating the Live Cutover

The final pre-migration piece was orchestrating which database was live. A new table, cf_migration_manager, was polled periodically by the DNS Records API. It communicated two critical values:

  1. Which database was active, using a simple A or B naming convention.
  2. Whether the database was write-locked. When locked, the DNS Records API held HTTP requests until the lock was released.

Both values were controlled by a migration manager script. Because the 20+ internal services had already been moved from direct database access to the internal DNS Records gRPC API, all writes were forced through this manager — no external service could bypass the lock.

Executing the Migration

While we aimed for a cutover measured in seconds, we announced a maintenance window of a couple of hours as a precaution. With both databases roughly synchronized, the steps were:

  1. Reduce the copy interval from 3 seconds to 0.5 seconds.
  2. Lock cfdb for writes via cf_migration_manager, causing the DNS Records API to hold write connections.
  3. Set cfdb read-only and migrate the final logged changes to dnsdb.
  4. Enable writes on dnsdb.
  5. Instruct the DNS Records API, via cf_migration_manager, that dnsdb was now the primary and that writes could proceed.

Ensuring the last changes were copied before enabling writes took no more than two seconds total. During the cutover, API latency spiked as the migration manager locked writes and then processed a backlog of queued requests; normal latencies returned after several minutes.

image3

DNS at Cloudflare has wide-reaching effects, so this was not quite the end. Three lesser-used services had escaped our earlier scan of cfdb users. The foreign table setup made the fix fast — those services were pointed at the new table name, and residual issues disappeared.

Post-migration: Capacity and Headroom

Once the migration completed, the shift in traffic was immediately visible. Usage of cfdb dropped sharply, freeing up substantial resources for other services.

image6

cfdb usage dropped significantly after the migration period.

The headroom has since translated into real gains. Average requests per second to the DNS Records API have more than doubled since the switch, while CPU usage on both cfdb and dnsdb has settled below 10%. That leaves ample room for traffic spikes and future growth.

image2
image4

cfdb and dnsdb CPU usage now

With the added capacity, the frequency of database-related incidents has dropped considerably. Query latencies are also slightly lower on average, with fewer sustained spikes above 500ms. The most noticeable improvement, however, comes under load: the database now handles bursts without significant issues. These bursts typically stem from clients pulling large sets of DNS records or making rapid-fire zone changes—both common during onboarding of large customers.

The operational benefits extend beyond raw performance. The DNS team now has finer-grained control over dnsdb-specific settings, rather than tuning for the needs of all services on a shared cluster. For example, custom adjustments to replication lag limits let replica reads proceed with confidence that the data is consistent. This shift has reduced pressure on the primary, since nearly all read queries can now be served by replicas.

The migration was a success, but the work continues. As both Cloudflare and its customers scale up, the demand for even greater capacity persists. The team has further improvements planned and will share more details as they roll out.