BLOG-3109 hero image

A memory optimization pushed to 1.1.1.1 on January 8, 2026 produced widespread DNS resolution failures for Internet users. Neither an attack nor an outage was responsible. The trigger was a change in the ordering of records inside 1.1.1.1's DNS responses: some client implementations rely on CNAME records arriving first, and once that ordering shifted, lookups began to fail. The code change arrived on December 2, 2025, moved to a testing environment on December 10, and entered deployment on January 7, 2026.

Time

Description

2025-12-02

The record reordering is introduced to the 1.1.1.1 codebase

2025-12-10

The change is released to our testing environment

2026-01-07 23:48

A global release containing the change starts

2026-01-08 17:40

The release reaches 90% of servers

2026-01-08 18:19

Incident is declared

2026-01-08 18:27

The release is reverted

2026-01-08 19:55

Revert is completed. Impact ends

Resolver caching and partial expiry

A query for a name such as www.example.com may return a CNAME record declaring that the name is an alias for another. Recursive resolvers like 1.1.1.1 follow the alias chain until they reach a final response:

www.example.com → cdn.example.com → server.cdn-provider.com → 198.51.100.1

Every intermediate record is cached separately with its own TTL, so the TTLs in a chain may differ:

www.example.com → cdn.example.com (TTL: 3600 seconds) # Still cached
cdn.example.com → 198.51.100.1 (TTL: 300 seconds) # Expired

When only part of a chain has expired, the resolver does not need to re-resolve the whole thing — just the stale portion. In this example it re-resolves the expired cdn.example.com A record, then merges the cached CNAME chain with the freshly resolved records into one response.

The allocation change that altered ordering

The reordering was introduced in the merge step. The original code created a new list, inserted the existing CNAME chain, and appended the new records:

impl PartialChain {
    /// Merges records to the cache entry to make the cached records complete.
    pub fn fill_cache(&self, entry: &mut CacheEntry) {
        let mut answer_rrs = Vec::with_capacity(entry.answer.len() + self.records.len());
        answer_rrs.extend_from_slice(&self.records); // CNAMEs first
        answer_rrs.extend_from_slice(&entry.answer); // Then A/AAAA records
        entry.answer = answer_rrs;
    }
}

To avoid the extra allocations and copies, the code was rewritten to append the CNAMEs directly to the existing answer list:

impl PartialChain {
    /// Merges records to the cache entry to make the cached records complete.
    pub fn fill_cache(&self, entry: &mut CacheEntry) {
        entry.answer.extend(self.records); // CNAMEs last
    }
}

Responses from 1.1.1.1 could now carry CNAME records at the end, after the final resolved answer.

Sequential parsers assume CNAME-first

Clients that receive a CNAME chain in the answer section must follow it themselves. One common approach tracks the expected name while iterating records in sequence, updating that name each time a CNAME is met:

;; QUESTION SECTION:
;; www.example.com.        IN    A

;; ANSWER SECTION:
www.example.com.    3600   IN    CNAME  cdn.example.com.
cdn.example.com.    300    IN    A      198.51.100.1
  1. Find records for www.example.com
  2. Encounter www.example.com. CNAME cdn.example.com
  3. Find records for cdn.example.com
  4. Encounter cdn.example.com. A 198.51.100.1

With the CNAME at the bottom, the same walk falls apart:

;; QUESTION SECTION:
;; www.example.com.	       IN    A

;; ANSWER SECTION:
cdn.example.com.    300    IN    A      198.51.100.1
www.example.com.    3600   IN    CNAME  cdn.example.com.
  1. Find records for www.example.com
  2. Ignore cdn.example.com. A 198.51.100.1 as it doesn't match the expected name
  3. Encounter www.example.com. CNAME cdn.example.com
  4. Find records for cdn.example.com
  5. No more records are present, so the response is considered empty

glibc's getaddrinfo, the standard Linux resolution path, was one affected implementation; its getanswer_r expects CNAME records to precede any answers:

for (; ancount > 0; --ancount)
  {
    // ... parsing DNS records ...
    
    if (rr.rtype == T_CNAME)
      {
        /* Record the CNAME target as the new expected name. */
        int n = __ns_name_unpack (c.begin, c.end, rr.rdata,
                                  name_buffer, sizeof (name_buffer));
        expected_name = name_buffer;  // Update what we're looking for
      }
    else if (rr.rtype == qtype
             && __ns_samebinaryname (rr.rname, expected_name)  // Must match!
             && rr.rdlength == rrtype_to_rdata_length (type:qtype))
      {
        /* Address record matches - store it */
        ptrlist_add (list:addresses, item:(char *) alloc_buffer_next (abuf, uint32_t));
        alloc_buffer_copy_bytes (buf:abuf, src:rr.rdata, size:rr.rdlength);
      }
  }

The ordering also broke the DNSC process on three Cisco ethernet switch models. When configured against 1.1.1.1, those switches entered reboot loops upon receiving a response with reordered CNAMEs; Cisco has published a service document covering the issue.

Implementations that cope with any order

Most DNS clients are unaffected. systemd-resolved, for example, first parses records into an ordered set:

typedef struct DnsAnswerItem {
        DnsResourceRecord *rr; // The actual record
        DnsAnswerFlags flags;  // Which section it came from
        // ... other metadata
} DnsAnswerItem;

typedef struct DnsAnswer {
        unsigned n_ref;
        OrderedSet *items;
} DnsAnswer;

It can therefore search the full answer set while following a chain, whether or not the CNAMEs sit at the top.

The protocol ambiguity behind the assumption

RFC 1034 (1987) Section 4.3.1 is the passage most often cited here:

If recursive service is requested and available, the recursive response to a query will be one of the following:

- The answer to the query, possibly preface by one or more CNAME RRs that specify aliases encountered on the way to an answer.

"Possibly preface" can be read as requiring CNAMEs first, but the text uses no normative keywords such as MUST or SHOULD. That is not a defect in RFC 1034 so much as a consequence of its age: RFC 2119, which standardized those keywords, appeared ten years later, in 1997. 1.1.1.1 originally did place CNAMEs first, but no test asserted that behavior, since the RFC language left it ambiguous.

RRsets versus records in a message section

RFC 1034 Section 3.6 is unambiguous about Resource Record Sets — records sharing a name, type and class:

The order of RRs in a set is not significant, and need not be preserved by name servers, resolvers, or other parts of the DNS.

The RFC never clearly maps message sections onto RRsets. It describes sections as holding individual RRs, whereas modern usage shows a section can hold several RRsets, as in DNSSEC responses carrying signatures. Ordering within an RRset is covered; ordering between RRsets in the same section is not, and that gap is where the ambiguity lives. The example in Section 6.2.1 does not resolve it:

The difference in ordering of the RRs in the answer section is not significant.

That example has two A records for one name in one RRset; it says nothing about mixing types such as CNAME and A.

Chain order is a separate problem

The issue extends beyond CNAME-versus-A placement. Even with CNAMEs first, a chain whose own links are out of order defeats sequential parsing:

;; QUESTION SECTION:
;; www.example.com.              IN    A

;; ANSWER SECTION:
cdn.example.com.           3600  IN    CNAME  server.cdn-provider.com.
www.example.com.           3600  IN    CNAME  cdn.example.com.
server.cdn-provider.com.   300   IN    A      198.51.100.1

The links belong to different RRsets — they have different owners — so the statement that RRset order is insignificant does not apply. RFC 1034 nonetheless imposes no order on chain links; nothing requires www.example.com. CNAME cdn.example.com. to precede cdn.example.com. CNAME server.cdn-provider.com.. Sequential parsing fails the same way:

  1. Find records for www.example.com
  2. Ignore cdn.example.com. CNAME server.cdn-provider.com. as it doesn't match the expected name
  3. Encounter www.example.com. CNAME cdn.example.com
  4. Find records for cdn.example.com
  5. Ignore server.cdn-provider.com. A 198.51.100.1 as it doesn't match the expected name

Resolver expectations, from recursive to stub

Section 5.2.2 describes how resolvers handle aliases:

In most cases a resolver simply restarts the query at the new name when it encounters a CNAME.

Read strictly, that says a resolver should restart upon a CNAME wherever it appears. But the two resolver roles differ:

  • Recursive resolvers such as 1.1.1.1 perform full resolution by querying authoritative nameservers.
  • Stub resolvers such as glibc's getaddrinfo forward queries to a recursive resolver and process the answers.

The RFC's resolver sections were written mainly for full recursive resolvers, not for stub resolvers. Some stubs, it turns out, skip spec logic such as CNAME restart.

Contrast with DNSSEC

RFC 4035 handles ordering more explicitly:

When placing a signed RRset in the Answer section, the name server MUST also place its RRSIG RRs in the Answer section. The RRSIG RRs have a higher priority for inclusion than any other RRsets that may have to be included.

"Higher priority for inclusion" governs whether RRSIG records are included at all, not where they sit. For unsigned zones the RFC 1034 ambiguity stands, and the word "preface" has guided implementations for nearly four decades without ever becoming a formal requirement.

Outcome and next steps

1.1.1.1's reading is that the RFCs mandate no particular CNAME order — yet widely deployed clients violate that reading and some of the systems running them are rarely, or never, updated. The resolver has reverted the reordering, does not plan to change it again, and takes the position that CNAME records should appear first, in order, ahead of all other records. To give that expectation a firm basis, an Internet-Draft has been submitted for discussion at the IETF; with consensus it would become an RFC defining CNAME handling in DNS responses. The draft is at https://datatracker.ietf.org/doc/draft-jabley-dnsop-ordered-answer-section, and feedback is requested through the IETF DNSOP working group.