Verisign’s Stale .com and .net DNSSEC Signatures Spell Trouble for Asia Pacific Traffic

On July 9, 2023, around 06:00 UTC, Cloudflare detected a significant uptick in DNS resolution failures across its Asia Pacific data centers. The culprit: invalid DNSSEC signatures emanating from Verisign’s servers for the .com and .net top-level domains (TLDs). At the peak, this affected roughly 5-7% of all DNS queries handled by Cloudflare in the region, which translated to connection errors for users trying to access proxied websites.

The issue was traced back to Verisign’s local nameserver infrastructure in Asia Pacific, which was intermittently serving expired DNSSEC signatures. Traffic that hit these local instances received responses that failed cryptographic validation. As a temporary workaround, Cloudflare rerouted its upstream DNS queries destined for Verisign’s Asia Pacific nameservers to Verisign’s US west coast locations, which were serving valid signatures. That move immediately quelled the error rates.

How Cloudflare’s DNS Resolution Works

Understanding the incident requires separating two distinct DNS resolution paths that operate at the network edge. The first is external resolution: when a public resolver like 1.1.1.1 queries for a proxied domain (say, blog.cloudflare.com), Cloudflare’s authoritative nameservers respond with Anycast IP addresses so the user’s browser can connect.

$ dig blog.cloudflare.com. +short
104.18.28.7
104.18.29.7

Once a request arrives at a Cloudflare data center, a second, internal process kicks in if the origin server is specified via a CNAME. The data center must perform an upstream resolution to find the IP address behind that CNAME target—for example, origin.example. It is this second, upstream path, plus an edge case in external resolution known as CNAME flattening, that was vulnerable to the broken Verisign responses.

$ dig origin.example. +short
192.0.2.1

DNS is a hierarchical system, meaning a recursive resolver must traverse multiple nameserver layers—from the root to the TLD to the authoritative server—to assemble a final answer. The DNSSEC protocol adds cryptographic signatures (RRSIG records) to DNS responses, allowing resolvers to verify the data’s authenticity. If a signature has expired, the record becomes invalid and the resolver is forced to return an error to the requesting client.

Incident Timeline and Detection

The first signs of trouble appeared in the logs on Saturday, July 8, at 21:10 UTC, with a trickle of DNSSEC validation errors during upstream resolution at several Asia Pacific sites. Internal alerts weren’t triggered until roughly an hour later, when the error rate had stabilized at about 0.5% of upstream queries. The problem escalated slowly, crossing the 10-15% threshold for affected upstream queries by the early hours of July 9, which finally prompted an incident declaration at 02:58 UTC.

Initial analysis pointed fingers at a single upstream provider, though that theory fell apart by 04:52 UTC when logs revealed the issue spanned numerous .com and .net domains, always localized to Asia Pacific. Testing with external recursive resolvers like 8.8.8.8 and 1.1.1.1 showed no errors, but queries issued via local stub resolvers continued to fail. By 06:24 UTC, Cloudflare engineers had pinpointed the fresh signatures in responses from Verisign’s nameservers in Singapore, while other locations responded correctly.

Cloudflare contacted Verisign at 06:41 UTC and began implementing rerouting measures nine minutes later. Moving IPv4 traffic to US west IPs produced a sharp drop in error rates, and adding IPv6 traffic at 07:06 UTC drove failures down to zero.

The workaround remained in place for over a day. At 18:23 UTC on July 10, Verisign confirmed that a local site had been serving stale data, and the issue had been resolved. Verisign’s subsequent technical post-mortem revealed the root cause:

During a migration of one of our DNS resolution sites in Singapore, from one provider to another, we unexpectedly lost management access and the ability to deliver changes and DNS updates to the site. Following our standard procedure, we disabled all transit links to the affected site. Unfortunately, a peering router remained active, which was not immediately obvious to our teams due to the lack of connectivity there.

Over the weekend, this caused an issue that may have affected the ability of some internet users in the region to reach some .com and .net domains, as DNSSEC signatures on the site began expiring. The issue was resolved by powering off the site’s peering router, causing the anycast route announcement to be withdrawn and traffic to be directed to other sites.

The Technical Failure Mode

DNSSEC validation relies on each response carrying an RRSIG record containing the signature itself, message metadata, and, critically, an expiration timestamp.

As you can see, the main part of the payload is associated with the signature and its corresponding metadata. Each recursive resolver is responsible for not only checking the signature but also the expiration time of the signature. It is important to obey the expiration time in order to avoid returning responses for RRsets that have been signed by old keys, which could have potentially been compromised by that time.

A recursive resolver must check both the mathematical validity of the signature and the expiration time. Enforcing the expiration window prevents resolvers from accepting stale data that could have been signed by compromised keys. When Verisign’s Singapore site began serving expired signatures for the .com and .net zones, Cloudflare’s in-house recursive resolver correctly flagged those responses as invalid. The resulting SERVFAIL codes propagated upward, ultimately preventing Cloudflare’s servers from resolving CNAME targets and thereby blocking proxied requests to origin servers.

Remediation and Impact

The bypass strategy was surprisingly low-cost despite the geographic redirection. Because Cloudflare shares and pools recursive resolver caches across data centers, frequently accessed names only incurred the lengthy trans-Pacific round trip once, after which answers were cached locally.

The outage underscores the fragility of the DNS ecosystem and the outsized impact that a single operator’s regional misconfiguration can have, especially given DNSSEC’s lack of tolerance for stale signatures. While Cloudflare’s rerouting kept the impact contained to a period of roughly 13 hours, the incident highlights the need for robust communication channels between infrastructure providers to diagnose and resolve issues of this nature quickly.

Update: Verisign confirmed that the Singapore site is part of a wider network of over 200 sites and that the incident had no effect on global .com and .net resolution. The company stated it is updating its processes to prevent similar occurrences in the future.