Why DNS Keeps Breaking Your Systems

DNS looks simple on paper: records live on a server, clients look them up, done. In practice, it's a different story. The failure modes are numerous and often disguised as something else entirely. Here are the most common ways DNS breaks real systems, based on reports from engineers who've lived through them.

Slow Lookups and Timeouts

The most straightforward failure: requests are just slow. Your DNS resolver is overloaded, leaking memory, or otherwise misbehaving, so every query takes longer than it should. A restart of the resolver—often your router—can fix this.

Timeouts are the harsher cousin of slow requests. A query that times out can hang your application for 2, 30, or more seconds. This is a well-known pain point in Kubernetes environments, where DNS timeouts can cascade into significant service degradation.

The Kubernetes ndots:5 Problem

Kubernetes configures pods with ndots:5 in /etc/resolv.conf, which tells the resolver to try multiple search domains before giving up on a query. Consider a typical pod configuration:

nameserver 100.64.0.10
search namespace.svc.cluster.local svc.cluster.local cluster.local eu-west-1.compute.internal
options ndots:5

With this setup, a lookup for google.com triggers a sequence of probes through the C getaddrinfo function:

  1. google.com.namespace.svc.cluster.local.
  2. google.com.svc.cluster.local.
  3. google.com.cluster.local.
  4. google.com.eu-west-1.compute.internal.
  5. google.com.

Your application waits for four failed queries before the real one succeeds. Every lookup in a pod pays this tax, and it adds up quickly under load.

Knowing Which Resolver You're Actually Using

When DNS breaks, the first question is usually "which resolver is involved?" That's harder to answer than it should be. On Linux, most software consults /etc/resolv.conf, but browsers may ignore it in favor of DNS-over-HTTPS. If you're on UDP, sudo tcpdump port 53 shows where queries go—but that's useless if you're on DNS-over-HTTPS or DNS-over-TLS.

Wrong Error Codes: NXDOMAIN Instead of NOERROR

A subtle bug: a DNS server returns NXDOMAIN (domain doesn't exist) for an A record query when it should return NOERROR (domain exists, but no A record). The client, like nginx, treats NXDOMAIN as fatal and gives up, even if a subsequent AAAA query succeeds. The server fixed it; in the meantime, domains appeared dead when they weren't.

Negative Caching

If you query for a domain before its record exists, the absence of that record gets cached. The TTL for this negative entry comes from the domain's SOA record—often an hour or longer. So you create a DNS record, and for the next hour, clients still see "not found."

Clients That Cache DNS Forever

Some software resolves hostnames once at startup and never again. nginx is a classic example:

location / {
    proxy_pass https://some.domain.com;
}

With this config, some.domain.com is resolved exactly once. If the IP changes—say, an AWS load balancer gets a new address—the old IP keeps working for months, then breaks at 2 a.m. when the stale IP finally stops responding.

Java has the same problem. Depending on JVM configuration, DNS entries may never refresh until the JVM restarts. The AWS SDK documentation explicitly warns about this. Any software can have this flaw, but nginx and Java are the most common offenders reported in practice.

Hidden /etc/hosts Entries

Entries in /etc/hosts silently override your DNS configuration. This is especially confusing because dig ignores /etc/hosts, so your queries appear to work fine while your application is hitting a completely different IP.

Email Failures

Email routing and validation are built on DNS: MX records for delivery, SPF and DKIM records for authentication. A misconfigured or broken DNS entry can silently cause email to fail delivery or land in spam folders, with no obvious connection to DNS.

Internationalized Domain Names

Domains with non-ASCII characters or emoji (like 💩.la) are translated to punycode (xn--ls8h.la) for DNS. The standard is clear, but software support is spotty. Some applications mishandle these translations, causing lookups to fail entirely.

TCP DNS Blocked or Unsupported

Large DNS responses need TCP port 53, not just UDP. Some firewalls block TCP 53 while allowing UDP, causing intermittent failures for queries that exceed the usual 512-byte UDP limit.

Similarly, musl—the libc used in Alpine containers—doesn't support TCP DNS. When a DNS server sees a response too large for UDP, it returns an empty, truncated response expecting a retry over TCP. musl doesn't retry, so the query fails. The client gets nothing.

Round Robin DNS and getaddrinfo

Round robin DNS, where each query returns a different IP for load balancing, breaks silently when software switches from gethostbyname to getaddrinfo. The newer function sorts the IP responses, defeating the rotation. This is especially insidious when the switch happens inside a library or during an innocuous-looking upgrade, with no obvious connection to DNS behavior.

Race Conditions on Startup

A final issue: services that start simultaneously and try to resolve each other can fail because the DNS records haven't propagated yet. The failed lookup gets cached negatively, and the services keep failing even after the records exist. This is a known pattern in Kubernetes environments with multiple containers starting at once.