The chain of trust and why a TLD misconfiguration is total

DNSSEC adds cryptographic authentication on top of DNS without hiding anything: a zone's record sets are published alongside RRSIG records that a resolver can check, so the data remains visible but its authenticity can be proven. Because signatures travel with the records they cover, a cached copy is as verifiable as a freshly fetched one, no matter how many hops or caches it has passed through. This is integrity, not privacy — that is the job of DoT and DoH.

Validation follows a chain of trust. The root zone's trust anchor is hard-coded into resolvers, and each zone delegates trust downward through Delegation Signer (DS) records, which carry a cryptographic hash of a public key in the child zone. Validating example.de therefore means checking the whole path: root trusts .de, and .de trusts example.de. Any break in that path invalidates everything beneath it.

Zones generally use two keys: a Zone Signing Key (ZSK) that signs the zone's records, and a Key Signing Key (KSK) that signs the ZSK and is anchored by the parent's DS record. Replacing a ZSK is comparatively simple — generate, re-sign, wait for caches to expire. A KSK rollover is harder because the parent's DS record has to change too, which usually means coordinating with a registrar or registry. The dangerous interval is the overlap: if the zone publishes signatures made with a key that resolvers cannot match against its DNSKEY records — whether the signing step failed, the timing was off, or the new key had not fully propagated — resolvers must reject the answers and return SERVFAIL.

Timeline: what 1.1.1.1 observed

At approximately 19:30 UTC, DENIC began publishing invalid DNSSEC signatures for the .de zone. Every validating resolver that received them was required to reject the responses, and 1.1.1.1 was no exception. Because .de is one of the largest ccTLDs and ranks among the most queried TLDs worldwide, the blast radius was potentially enormous: millions of domains below it could become unreachable.

Right after the initial SERVFAIL spike at 19:30 UTC, the failure rate climbed for roughly three hours as cached entries aged out. Each time a domain's cached records expired and a resolver went back to DENIC for a fresh copy, it received broken signatures and failed. Query volume rose sharply at the same time, which is typical: clients retry failed lookups, often three or more times, so raw error counts overstate real user impact. Notably, the NOERROR rate held fairly steady — the effect of serve-stale behavior described below.

Serve stale as the first line of defense

Recursive resolvers keep records from authoritative servers for each record's TTL and answer from cache until it expires. During the outage, anything fetched fresh was DNSSEC-bogus and correctly rejected, but many .de records were already cached from before the incident. Instead of dropping those and returning errors, 1.1.1.1 kept serving them past their TTL under RFC 8767, which permits a resolver to answer with expired cached data when upstream resolution fails. Stripping out the stale-served answers makes the real picture visible: the NOERROR rate declines steadily from 19:30 onward, and those are the queries users only got good answers for because a cached copy still existed. Serve stale softened a large upstream failure and bought operators time to respond.

Mitigation: treating .de as insecure

Serve stale helped, but plenty of users still hit real failures, so we took direct action. RFC 7646 defines the Negative Trust Anchor (NTA): an explicit instruction to a validating resolver to treat a named zone as unsigned, skipping validation for everything beneath it. NTAs exist for exactly this scenario — when a parent zone publishes broken signatures, returning SERVFAIL for every child provides no security value, since the fault is known, public and being fixed. RFC 7646 names TLD misconfiguration as the primary use case.

Big Pineapple, our own resolver which also backs 1.1.1.1 for Families, Gateway DNS, DNS Firewall and more, does not yet implement a native NTA mechanism. We instead used an existing override rule to mark .de as insecure, causing its names to resolve as though DNSSEC were not enabled — functionally an NTA, though not defined by any RFC. The tradeoff is deliberate: without validation, .de domains are exposed to genuine attacks for the duration of the incident. We judged that acceptable because the signing failure was widespread, publicly confirmed, and affected every validating resolver on the Internet alike — as one colleague put it in the incident room, no 1.1.1.1 user resolving a .de name wanted a SERVFAIL over an unvalidated answer. Rollout completed at 22:17 UTC, which ended impact for 1.1.1.1, and we shared the change with other DNS operators through the DNS-OARC Mattermost.

A separate internal resolver handles origin resolution for our CDN platform, distinct from 1.1.1.1. Customers with .de origin hostnames were affected too, and applying a similar NTA for .de there restored origin connectivity.

A diagnostic bug: wrong Extended DNS Error code

Before the mitigation, cache misses from 1.1.1.1 returned SERVFAIL, each carrying an Extended DNS Error (EDE) code per RFC 8914. Some resolvers sent EDE 6 (DNSSEC Bogus) with a message pointing at the bad signature — the correct behavior. 1.1.1.1 instead returned EDE 22 (No Reachable Authority), which misleadingly suggests an upstream connectivity problem.

The root cause is in how DNSSEC EDE codes are propagated from our trust chain verifier. When the verifier detects a bogus signature it creates the DNSSEC Bogus code, but that code never makes it into the response; the outer resolver layer sees a recursive resolution failure with no error code and falls back to No Reachable Authority, masking the DNSSEC cause. We will be fixing these responses so the DNSSEC errors surface properly.

DNSSEC did not fail — the configuration did

It is tempting to read this outage as proof that DNSSEC does not work. It is not: any misconfigured technology will break the users who depend on it. The important role DNSSEC plays in letting resolvers trust DNS answers against tampering is unchanged.

DENIC's own blog post links the outage to a routine, scheduled key rollover: "During this process, non-validatable signatures were generated and distributed. As a precautionary measure, future rollovers have been suspended until the exact technical causes have been identified." More details will presumably follow once DENIC's analysis is complete.

What this incident shows about the DNS hierarchy

A failure at the registry tier does not stay local. When a TLD-level zone breaks, every domain beneath that TLD is hit at the same moment, no matter which host serves those domains or which resolver a client queries. DNSSEC is only one way to trigger this: if a TLD's nameservers become unreachable, the outcome is the same. The same hierarchy that lets the global DNS function is what carries a top-level failure downward.

Mitigation in the field

No single fix removes the dependency, so the practical goal is a fast, consistent response. During this incident, resolver operators across the Internet applied Negative Trust Anchors independently within an hour, which restored resolution while DENIC worked to repair the zone. Alongside this kind of peer action, operational practices, industry communication channels such as DNS-OARC, and features like serve stale all limit the blast radius — though none of them eliminates the underlying dependency.

Follow-up work

For our own systems, the incident produced concrete follow-ups, starting with our EDE errors, which we will rework to surface DNSSEC errors more clearly. We also await DENIC's post-incident report and appreciate the transparency they showed throughout.

Background on the protocol is available in How does DNSSEC work?, and live DNS trends and TLD data can be followed on Cloudflare Radar.