A stale root zone breaks 1.1.1.1 for four hours

On 4 October 2023, Cloudflare's 1.1.1.1 resolver returned SERVFAIL errors for valid queries between 07:00 and 11:00 UTC. The outage affected WARP, Zero Trust, and third-party resolvers that depend on 1.1.1.1. Cloudflare attributes the cause to an internal software failure, not an attack: a new DNS record type in the root zone was not parseable by its software, leaving resolvers stuck on an expired version of the zone.

How the root zone feeds resolution

In DNS, every domain lives in a zone that is managed as a unit. The root zone, which sits above all top-level domains, tells resolvers how to reach TLDs, making it essential for resolving any name. Like other critical parts of DNS, the root zone is signed with DNSSEC, meaning it carries cryptographic signatures.

Rather than querying root servers for every request, Cloudflare's recursive infrastructure retrieves and keeps a local copy of the root zone. New versions are normally published twice a day. A WebAssembly app called static_zone, running on top of the main DNS logic in 1.1.1.1, is responsible for parsing and serving those updates.

The failure sequence

On 21 September, the root zone included a new resource record type — ZONEMD — for the first time, as part of a planned change in root zone management. ZONEMD is essentially a checksum of the zone's contents. After the change, the root zone continued to be retrieved and distributed normally through Cloudflare's core network, but the resolver systems parsing the zone data failed on the ZONEMD record.

Since a zone must be loaded and served in its entirety, the parse failure meant new root zone versions were ignored. Some resolver servers fell back to querying root servers directly per request; others kept serving the last known good version from memory — the copy pulled on 21 September, before ZONEMD was introduced.

At 07:00 UTC on 4 October, DNSSEC signatures in that stale 21 September zone expired. With no newer usable version, resolver systems that relied on the cached zone could no longer validate DNSSEC signatures and began returning SERVFAIL responses. The rate of SERVFAILs rose by 12% and peaked at 15% of total queries, up from the normal ~3% baseline. Impact concentrated in Cloudflare's larger data centers, particularly Ashburn, Frankfurt, and Singapore.

Key timeline events:

  • 21 September, 6:30 UTC: Last successful root zone pull.
  • 4 October, 7:00 UTC: DNSSEC signatures expire; SERVFAILs increase.
  • 7:57 UTC: First external SERVFAIL reports.
  • 8:03 UTC: Internal incident declared.
  • 8:50 UTC: First attempt to disable serving the stale zone via an override rule.
  • 10:30 UTC: 1.1.1.1 stops preloading the root zone file entirely.
  • 10:32 UTC: Responses return to normal.
  • 11:02 UTC: Incident closed.

Root causes

Why ZONEMD parsing failed

DNS uses a binary format for resource records, where the record type is a 16-bit integer that determines how the resource data (RDATA) is interpreted. Unknown types don't need special handling — the RDLENGTH field allows a parser to skip the RDATA as opaque data. However, Cloudflare distributed the root zone internally in its presentation format, not the binary format. In text form, record representation varies far more, and when the parsing library encountered the unfamiliar ZONEMD record, it returned a parser error instead of safely skipping it.

Why a stale version was served

The static_zone app keeps the latest parsed root zone in memory and only replaces it after a successful parse. Since parsing failed, the app never switched versions and served the 21 September copy indefinitely. If the service started fresh with no cached version and parsing failed, it fell back to querying root servers directly.

Why the first fix attempt failed

Cloudflare tried disabling static_zone via an override rule that tagged requests with rec_disable_static, instructing the app not to answer from the cached zone. However, requests are sometimes forwarded to other nodes to improve cache performance, and the tag wasn't included in forwarded queries. Those forwarded requests still hit static_zone's stale data until the app was fully disabled.

Why impact was partial

Rolling reboots of resolver servers between the ZONEMD change and signature expiration meant restarted instances failed to load the zone on startup and fell back to querying root servers. Additionally, 1.1.1.1 uses the RFC 8767 serve-stale technique to continue serving popular records from cache after TTL expiry. This prevented a full outage; the worst effects were in data centers with many servers that had not restarted in that two-week window.

Remediation

Cloudflare has outlined several immediate improvements:

  • Visibility: Add alerts to detect when static_zone is serving a stale root zone file — the condition went unnoticed for too long.
  • Resilience: Rework how the root zone is ingested and distributed internally so new resource record types are handled seamlessly and pipeline interruptions are invisible to end users.
  • Testing: Improve test coverage for scenarios where the root zone fails to parse; existing tests covered unreleased ZONEMD parsing changes but not the failure mode.
  • Architecture: Better manage the lifetime of cached root zone data per RFC 8806 directions, avoiding stale copies beyond an acceptable operational limit.

Cloudflare acknowledges the outage as a reminder that core internet systems and their inputs evolve, and that software chains must anticipate change. The company states it is working to ensure parsing failures and format changes in critical infrastructure like DNS and BGP are handled gracefully.