Serving stale DNS answers from the Consul path
Service discovery in Consul exposes both HTTP and DNS interfaces, and at scale, most clients prefer DNS. For a Consul cluster spanning multiple data centers, that preference creates a latency problem: queries from regions far from the cluster’s authoritative servers have to traverse long network paths, and when the resolver is also speaking DNS over TLS, the cost per query grows further. Short record TTLs compound the issue because they push cache expirations into the hot path.
Conventional caching in a resolver like Unbound helps only until the TTL expires. After that, the next client request has to wait for a fresh network resolution. With records set to a 30-second TTL, that expiration happens constantly, and failover requirements keep the TTL low deliberately.
Two strategies exist to soften that blow: prefetching and serving stale cache entries.
Why prefetching falls short
Prefetching refreshes a record in the background before it expires. When Unbound serves a cached record and the remaining TTL drops below 10% of the record’s lifetime, it answers the client immediately and dispatches a job to re-resolve the name. If the refresh completes before expiry, no client ever waits on the network.
The mechanism works well for frequently queried names, but it breaks down for records that are accessed infrequently or that carry a short TTL. In those cases, a request rarely lands in the prefetch window, the cache expires without a refresh, and the next query pays the full network cost. With a 30-second TTL, requests often miss the window entirely, leaving the cache empty and the next client waiting. Prefetching alone did not solve the problem.
Stale cache: another option
Serving stale records flips the approach: instead of forcing the client to wait for a refresh, the resolver returns the expired entry from cache and refreshes it in the background. That behaviour helps in at least two situations:
- High baseline latency: even after the TTL expires, subsequent queries get a cached answer while the refresh happens out of band.
- Consul or network outages: if the upstream becomes unreachable, expired answers remain available for a configured window instead of causing resolution failures.
Serving stale data for a limited period after TTL expiry matches the guidance in RFC 8767, “Serving Stale Data to Improve DNS Resiliency,” published in March 2020.
Earlier problems with Unbound's stale cache
Stale caching in Unbound had been tried before, back in 2017, and two concerns emerged. First, Unbound could conceivably serve stale records indefinitely. That risk is now manageable with configuration options that did not exist then. Second, there were reports that Unbound could serve negative or incomplete information from expired cache entries—for instance, a CNAME with no corresponding A or AAAA record. The report was never confirmed, but it remained a plausible failure mode because parts of a response chain could be purged from the stale cache at different times.
Testing current Unbound behaviour
Testing used Unbound 1.13.0 with the following stale-cache configuration:
serve-expired: yes— enables the stale cache.serve-expired-ttl: 3600— limits serving expired records to one hour after expiry, assuming no upstream response arrives.serve-expired-client-timeout: 500— serves from stale cache only when the upstream takes longer than 500 ms to answer.
The serve-expired-ttl-reset option was considered but rejected. It would keep records alive indefinitely as long as they were queried regularly, even with the upstream down, which was deemed too risky because it applies globally and without a hard bound.
In the test zone service1.sd, normal resolution produces a CNAME chain plus an A record. After the TTLs expire, a cache dump no longer lists the entries, but with network access to the Consul resolver blocked, queries still return answers. Unbound waited the configured 500 ms before responding with the stale data. Notably, unbound-control dump_cache does not display expired entries even though they are still present in the cache.
Test: Can stale records be served indefinitely?
After 3600 seconds, repeated queries for the same name eventually go unanswered; dig times out. Unbound stops serving the stale record once the configured window passes, even if the record was requested recently. That is the expected behaviour of serve-expired-ttl: 3600 with serve-expired-ttl-reset set to “no” by default—options that were unavailable when the stale-cache feature first caused problems.
Test: Can incomplete chains be served from stale cache?
Because the record in question contains a CNAME and an A record with different TTLs, testing focused on whether Unbound could return a partial answer—for instance, the CNAME without its corresponding A record.
With both entries in cache and the upstream blocked, removing the A record from the cache and querying the name yielded no response at all. A further scenario lowered serve-expired-ttl to 40 to accelerate testing. Once the A entry was no longer eligible to be served stale, the CNAME also stopped being returned. The chain was not served partially.
Test: Is the cache refreshed in the background?
To verify that a stale answer did not prevent a background refresh, artificial latency was introduced with the Network Link Conditioner. Queries confirmed the added delay was in effect. Then the address in Consul was changed to 192.0.2.200 while stale data was still cached. The first query returned the old address after 500 ms—served from stale cache. The next query returned 192.0.2.200 without delay, proving the refresh had completed in the background.
Test notes
All scenarios completed successfully, though one upstream issue surfaced and has since been fixed. The prefetching logic could block the cache while it attempted a refresh, preventing stale entries from being served when the upstream was unresponsive. The fix is included in Unbound version 1.13.1.
Deployment results
The change has been deployed across all data centers. The original problem—high latency on cache expiry—was resolved, and several operational metrics improved as side effects.
Lower in-flight request counts
The Toronto data center received the change at approximately 14:00 UTC. Prior spikes in in-flight requests flattened noticeably. Those spikes were likely caused by slow or unresponsive upstreams, which now return a stale answer after a short wait instead of holding the request open.
A similar effect was observed in the Perth data center, which received the change on a later day.
Sharply reduced P99 latency
The Lisbon data center received the change at approximately 12:37 UTC. The long tail of request latencies dropped dramatically. Serving stale answers keeps clients from waiting on slow upstream resolutions that might time out entirely.
The effect was even stronger in Paris. The reduction in P99 latency tracks the start of stale serving, despite the absolute number of stale records being modest—the graph shows the per-second rate of stale answers for the same data center, using the last three minutes of datapoints.
Fewer general errors
The Perth data center applied the change on February 8 and saw the general error rate fall. Requests that previously failed because the upstream was slow or unresponsive now get served from the stale cache instead.
A similar decrease was visible in the Vientiane data center, which received the change on February 1.
Less jitter in load balancer query rates
A further metric from Perth, which changed on February 8, showed reduced variation in how frequently RRDNS queried load balancers. The likely explanation is that resolvers typically retry after a short delay when they receive no response. By returning a stale answer instead of leaving the client waiting, the surge of retry queries for expired and slow-to-resolve records is avoided.
The Doha data center, which changed on February 1, showed the same pattern.
Improvements to Unbound-specific metrics
The change also improved several Unbound-specific counters in Perth, the data center that received it on February 8. The before-and-after comparison was noticeable across all of them.
Future tuning
The current serve-expired-client-timeout of 500 ms was chosen conservatively while the feature was being validated through production. The value can likely be at least halved without adverse effects, further improving the client experience. There is also room to vary the value per data center—for instance, aligning it with each site’s P90 resolution latency—which is an option worth exploring.
In short: Unbound now answers from cache even after TTL expiry when an upstream takes longer than 500 ms to respond, refreshing the record in the background, until 3600 seconds have elapsed from the original TTL.
