The Case of the Recursive Resolvers

DNSSEC Fallout: What Broke Slack for 24 Hours

In late September 2021, Slack engineers flipped a switch that knocked out DNS resolution for a small slice of users for a full day. The move — enabling DNSSEC signing on slack.com — was the third attempt at a change that had already failed twice. The incident, which affected under 1% of the online user base, traces back to a mix of protocol edge cases, strict resolver behavior, and the inherent risk of per-domain security rollouts.

DNS translates human-friendly domain names into the IP addresses that clients actually connect to. By default, the protocol is unauthenticated; anyone between a client and an authoritative name server can tamper with a response and redirect traffic. DNSSEC cryptographically signs responses from authoritative servers, preventing that kind of tampering. It does not, however, protect the final hop between a client and its recursive resolver — that last mile remains vulnerable to man-in-the-middle attacks. Slack's motivation for the rollout was compliance: DNSSEC is required for FedRAMP Moderate, and the company states it remains committed to the security extension despite ongoing community debate about its utility.

Slack runs its public DNS on Amazon Route 53, with some subzones delegated to NS1 for advanced traffic management. All zones are managed by the Traffic Engineering team. For four months prior to the incident, the team worked on enabling DNSSEC signing across all Slack-owned domains. By the time of the outage, every critical domain except slack.com had been signed for at least a month without issue.

A key limitation drove the risk profile: DNSSEC can only be enabled on a whole domain, never on an individual subdomain or delegated subzone. That means signing slack.com forces signing of everything beneath it, including subzones like wss-backup.slack.com delegated to NS1. The team's pre-flight checks were extensive — external monitoring from multiple global locations across resolvers including Cloudflare, Google, OpenDNS, Quad9, and Verisign; zone validation through dnsviz.net and Verisign's DNSSEC debugger; dnschecker.org resolution tests; and hand-crafted dig queries. All passed for the earlier domain rollouts, but slack.com proved different.

Two Failed Attempts First

Attempt one came on September 7, 2021. Signing was enabled on slack.com, but before the DS record was published at the registrar, a major US ISP suffered its own outage. Customers began reporting access problems. Not yet knowing the ISP was the cause, the team rolled back the slack.com zone changes as a precaution.

Attempt two followed the next day, September 8. Signing was enabled again, again before publishing the DS record. This time Customer Experience flagged a small number of users with DNS resolution failures — disproportionately users behind VPN providers. The root cause turned out to be a class of resolvers that enforce DNS specifications more strictly. When signing is enabled at the authoritative name servers before the DS record is published at the root or TLD servers, those strict resolvers begin rejecting responses. Specifically, they refuse to accept a CNAME record at the apex of a zone, as proscribed by RFC-2181 — even at the apex of a sub-delegated subdomain. Many VPN providers use such resolvers, which explained the user distribution.

The fix was immediate: the team stopped signing the slack.com zone in both Route 53 and NS1, and resolution returned to normal for affected users. Afterward, the sub-delegated zones were updated from CNAME to ALIAS records, which do not trigger the same strict rejection behavior.

Rolling Back, and Making Things Worse

With monitoring green and customers quiet, the team considered the rollout complete. That calm lasted until Customer Experience began receiving reports of ERR_NAME_NOT_RESOLVED errors in Slack clients. NetLog captures from affected clients pointed to NODATA responses from recursive resolvers. The reports clustered around a private corporate resolver and Google Public DNS (8.8.8.8), a combination that seemed odd: broad Google DNS impact should have produced a larger incident footprint than the monitoring showed.

To "stop the bleeding," the team removed the slack.com DS record from the registrar. After nearly an hour, error rates showed no sign of improvement. The next step was a full rollback of DNSSEC signing on the authoritative and delegated zones, restoring the DNS configuration to its previous healthy state and ruling DNSSEC out as the cause.

That rollback made things dramatically worse. Within moments, the Traffic team was paged for resolution failures from multiple resolvers worldwide.

The team’s mental model had a critical flaw. They assumed DS records at the .com zone were never cached, so removing the DS record would immediately halt validation. In reality, the .com zone directs resolvers to cache the slack.com DS record for 24 hours by default, a TTL the zone owner cannot modify. Some resolvers cap this TTL lower—Google’s 8.8.8.8 uses six hours—but many do not.

Expected resolver behavior after removing the DS record, versus actual behavior with cached DS records.

Removing the DNSKEY from the authoritative zone is a high-risk operation unless the DS record is gone. The team believed the DS record was gone. Instead, resolvers had cached it for up to 24 hours. With the DNSKEY no longer published but the DS record still cached, every validating resolver that had recently fetched the DS record began returning SERVFAIL for slack.com.

The tooling compounded the problem. All DNS configuration is managed through Terraform, which masked a critical warning that the Route53 console displays when disabling DNSSEC signing.

Route53 console warning when disabling DNSSEC signing.

The team immediately contacted major ISP and public resolver operators to flush cached slack.com records, using public cache-flush pages where available, such as Google’s 8.8.8.8 and Cloudflare’s 1.1.1.1. As caches flushed, error rates slowly declined. Slack client telemetry continued to flow via slackb.com—a domain with DNSSEC already enabled—providing a window into the outage.

Restoring DNSSEC signing was considered during the incident. The team held back for two reasons. First, the root cause of the original NODATA responses was still unknown, and restoring signing would not have fixed those users. Second, the key material had been deleted during rollback, and the recovery path was uncertain.

The KSK in Route53 is backed by an AWS KMS key, which was recoverable and would have remained the same. The ZSK, however, is entirely managed by AWS with no customer visibility. Route53 later confirmed that re-enabling signing generates a new ZSK key pair. The service team is designing a mechanism for faster recovery in the future.

A further misunderstanding shaped the decision not to restore signing: the team believed the DS record TTL could be set by the slack.com zone and that re-signing would “reset the clock,” forcing a second round of cache flushes. In fact, the DS record TTL originates from the .com zone; re-enabling signing would have had no impact on it. Since the ZSK would have changed anyway, validating resolvers would have kept failing—the action would neither have helped nor hurt.

Resolution came as the 24-hour TTL on the cached DS records expired and public resolver caches were flushed. Error rates returned to baseline, leaving a long tail of smaller resolvers that had not flushed their caches as the final stragglers.

What Went Wrong: A Wildcard Gap in Signed NSEC Responses

After restoring service for Slack customers, the engineering team shifted focus to pinpointing the exact cause of the outage. DNSSEC had previously been enabled without incident on all other Slack public domains, so slack.com stood out as uniquely problematic. Replicating the rollout in a test environment with identical zone settings and known-problematic resolvers yielded no failures, forcing the team to dig into customer NetLogs for answers.

The logs revealed a telling discrepancy: slack.com resolved successfully, but app.slack.com returned ERR_NAME_NOT_RESOLVED. This pattern pointed squarely at the *.slack.com wildcard record — a feature absent from every other domain that had undergone DNSSEC enablement without issue. It was an oversight that wildcard records had never been tested during previous rollouts.

Engaging the Amazon Route 53 team with this evidence quickly exposed the root cause: a bug in how Route 53 generated NSEC responses for wildcard records. When a DNSSEC-enabled zone receives a query for a wildcard record type that doesn't exist (such as an AAAA lookup on a host that only has an A record), the authoritative server must return a signed NSEC response confirming the name exists but the queried type does not. Per RFC 7129, this response must include additional information designed to preserve wildcard behavior. Route 53 was omitting that data.

The consequence was that resolvers receiving the malformed NSEC response could misinterpret it as proof that nothing exists at that name level — not just that the specific record type was absent. Instead of falling back to querying for an A record, they concluded no A record could exist either. This behavior was observed in Google Public DNS (8.8.8.8) but not Cloudflare Public DNS (1.1.1.1).

The problem was compounded by caching. Once a resolver cached this incorrect NSEC response, it effectively became poison for subsequent queries. The scope was limited to individual regional cache instances, so a resolver backed by multiple regional caches would show intermittent failures: traffic hitting the poisoned cache returned errors while queries routed to other caches succeeded.

Resolver Behavior and the NSEC Ambiguity

Investigating why the two major public resolvers diverged led to a second question: which one was out of spec? The answer turned out to be neither. RFC 8198 describes Aggressive NSEC, an optional specification that Google implements. Under this scheme, a resolver can legitimately reuse a cached NSEC record to answer subsequent queries by inferring both name non-existence (the NSEC span) and record type non-existence (the NSEC type bitmap). Google's implementation performs both types of inference. Cloudflare's appears to do neither, or only the name-based inference.

The conclusion: both resolutions of the ambiguity are semantically valid under the RFC, and the only way to guarantee unambiguous behavior is for the authoritative server to return a complete NSEC type bitmap that includes all existing record types at the queried name. That was precisely the fix AWS implemented.

dig A qqq.slackexperts.com @8.8.8.8 +dnssec

; <<>> DiG 9.10.6 <<>> A qqq.slackexperts.com @8.8.8.8 +dnssec
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 48704
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;qqq.slackexperts.com.          IN      A

;; ANSWER SECTION:
qqq.slackexperts.com.   300     IN      A       54.211.89.16
qqq.slackexperts.com.   300     IN      RRSIG   A 13 3 300 20211105130720 20211105110220 28453 slackexperts.com. zksghNF9JTx4mRXjkoEGP6C5zCPb5JjX2MzeihENzx3JIjnZLZyokvdd /dnxcA5Qjl3sc3eC0bkryoGATb4ghw==

;; Query time: 66 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Wed Oct 13 15:40:32 EDT 2021
;; MSG SIZE  rcvd: 177

Lookups against the wildcard record demonstrate the issue in practice. After enabling DNSSEC on a test zone with a wildcard, a query for the A record of *.slackexperts.com succeeds. The problem emerges when checking for a nonexistent AAAA record at the same wildcard level: the signed NSEC response omits the A type from its bitmap, even though the A record genuinely exists.

dig AAAA qqq.slackexperts.com @8.8.8.8 +dnssec

; <<>> DiG 9.10.6 <<>> AAAA qqq.slackexperts.com @8.8.8.8 +dnssec
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 63101
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 4, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;qqq.slackexperts.com.          IN      AAAA

;; AUTHORITY SECTION:
slackexperts.com.       2370    IN      SOA     ns-869.awsdns-44.net. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400
slackexperts.com.       2370    IN      RRSIG   SOA 13 2 900 20211013205618 20211013184118 30624 slackexperts.com. iBn8JmsuVq+F5zD+cm6C1Ypr25TtpKAIduCUZ80+sbuCbMQh0beC7RFc 2MctSnC9tf1RuCQHUxa/sEVVyHw6xQ==
qqq.slackexperts.com.   2370    IN      NSEC    \000.qqq.slackexperts.com. RRSIG NSEC
qqq.slackexperts.com.   2370    IN      RRSIG   NSEC 13 3 86400 20211014204118 20211013184118 30624 slackexperts.com. yB7d/Fl15M0z6zEHuE4XDvdxBT91DPiV38vhvI/42piqA1YI9ibb178E X0L82AaIaxcvSXKsdrWD8bAxJGVVyw==

;; Query time: 40 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Wed Oct 13 15:41:47 EDT 2021
;; MSG SIZE  rcvd: 398

Once that defective NSEC response is cached, a subsequent query for the existing A record comes back empty — the resolver's cached NSEC record led it to conclude no A record exists, so it never queries the authoritative server.

dig A qqq.slackexperts.com @8.8.8.8 +dnssec

; <<>> DiG 9.10.6 <<>> A qqq.slackexperts.com @8.8.8.8 +dnssec
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 17738
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 4, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;qqq.slackexperts.com.          IN      A

;; AUTHORITY SECTION:
slackexperts.com.       2382    IN      SOA     ns-869.awsdns-44.net. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400
slackexperts.com.       2382    IN      RRSIG   SOA 13 2 900 20211013205618 20211013184118 30624 slackexperts.com. iBn8JmsuVq+F5zD+cm6C1Ypr25TtpKAIduCUZ80+sbuCbMQh0beC7RFc 2MctSnC9tf1RuCQHUxa/sEVVyHw6xQ==
qqq.slackexperts.com.   2382    IN      NSEC    \000.qqq.slackexperts.com. RRSIG NSEC
qqq.slackexperts.com.   2382    IN      RRSIG   NSEC 13 3 86400 20211014204118 20211013184118 30624 slackexperts.com. yB7d/Fl15M0z6zEHuE4XDvdxBT91DPiV38vhvI/42piqA1YI9ibb178E X0L82AaIaxcvSXKsdrWD8bAxJGVVyw==

;; Query time: 10 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Wed Oct 13 15:42:12 EDT 2021
;; MSG SIZE  rcvd: 398

The Route 53 team has since deployed a fix that corrects the NSEC type bitmap for wildcard records. The incident serves as a stark reminder of the edge cases that DNSSEC introduces, particularly when wildcard records and negative caching interact. Slack's engineering team conceded that the rollout process should have included test coverage for wildcard-enabled domains before attempting the same configuration on production infrastructure.

Acknowledgements

Credit goes to the Amazon Route 53 team for identifying the bug and rapidly implementing a fix, to the Salesforce DNS team for their expertise during the incident, and to the broader DNS community for their assistance in rapidly flushing poisoned caches. Slack's Customer Experience team was also instrumental in bridging the gap between affected users and the engineering team throughout the outage.