Prasanta Kumar Behera
Incident Handling — On-Call Playbook & Real Scenarios›14 · Cluster-wide DNS timeouts
Learning Hub / Incident Handling / Incident Handling — On-Call Playbook & Real Scenarios

Lesson 14 of 19 · Real-world incident scenarios

Cluster-wide DNS timeouts

Across the cluster, requests intermittently take five seconds or fail with 'no such host'. Separate CoreDNS overload, search-path amplification (ndots:5), conntrack issues and upstream DNS failures, then fix with scaling, node-local caching and saner resolver settings.

Advanced
Key wordsCoreDNSDNS timeoutsndotssearch domainsconntrackUDPNodeLocal DNSCacheautoscaling CoreDNSSERVFAILupstream DNS

The page

Friday 16:00 — error rates rise across many services. Traces show calls stuck for ~5 seconds before succeeding, and some fail with dial tcp: lookup payments-api: i/o timeout. A new batch system launched this week makes thousands of outbound calls per second to an external API.

First five minutes

  • Impact: broad, intermittent latency and errors → a shared dependency. DNS is a prime suspect when many services fail at once with lookup errors.
  • Test: resolve an internal name and an external name from a pod; time them.

DNS is the school phone book. If there's only one librarian answering "what's the number for…?" and a new class starts asking a thousand questions a minute (each asked five different ways), everyone waits in a long queue. The fixes: more librarians (scale CoreDNS), a little phone book in every classroom (node-local cache), and teaching kids to ask properly the first time (ndots / FQDNs).

Diagnose

$ kubectl run dns --rm -it --image=busybox:1.36 -- sh
/ # time nslookup kubernetes.default
/ # time nslookup api.partner.example
/ # cat /etc/resolv.conf
search shop.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:5
$ kubectl -n kube-system top pods -l k8s-app=kube-dns
$ kubectl -n kube-system logs -l k8s-app=kube-dns --tail=50 | grep -iE 'error|timeout'
Finding Points to
CoreDNS pods at CPU limit / few replicas Capacity
Internal fine, external slow/failing, upstream timeouts in logs Upstream DNS or network path
5-second stalls, random, worse under load UDP/conntrack issues on nodes
Huge query counts for names with suffixes like .svc.cluster.local appended to external names ndots/search amplification

Fix

  • Scale CoreDNS: more replicas spread across nodes; the cluster-proportional-autoscaler is a common way to scale with cluster size. Check its resource limits.
  • NodeLocal DNSCache: a caching agent on every node; pods query it locally, which cuts load and avoids many conntrack races.
  • Reduce amplification for heavy external callers:
spec:
  dnsConfig:
    options:
      - name: ndots
        value: "2"

or call external hosts with a trailing dot (api.partner.example.), and reuse connections (HTTP keep-alive) instead of resolving on every request. - Upstream: fix or add upstream resolvers; configure CoreDNS forward with multiple servers.

Verify

  • DNS latency back to milliseconds (coredns_dns_request_duration_seconds histogram).
  • Application error rates recover; 5-second stalls gone in traces.

Prevent

  • Alert on CoreDNS latency, errors (SERVFAIL) and saturation.
  • Load-test new high-volume clients, including their DNS behaviour.
  • Standardise NodeLocal DNSCache and CoreDNS autoscaling in the platform baseline (see Networking Deep Dive, lesson 07).

Try it: stress and fix DNS (lab)

  1. On a lab cluster, check CoreDNS replicas and resources.
  2. Run a pod that loops nslookup api.example.com in parallel (e.g. 50 background loops) and watch CoreDNS CPU and latency.
  3. Inspect /etc/resolv.conf and count the queries per lookup with CoreDNS log plugin enabled temporarily.
  4. Add ndots: 2 to the test pod and compare query counts.
  5. Install NodeLocal DNSCache (per the Kubernetes docs) and repeat the load test.

Going deeper: DNS at scale

  • Cache negative responses sensibly; missing records looked up in a hot loop can hammer upstreams.
  • Watch for DNS-based service discovery in apps that re-resolve on every request.
  • On EKS and other clouds, VPC DNS resolvers have per-interface packet limits; node-local caching helps there too.

Recap

  • Many services failing with lookup errors or 5-second stalls → suspect DNS.
  • Separate capacity, upstream, conntrack/UDP and ndots amplification.
  • Fix with scaling, NodeLocal DNSCache, lower ndots / FQDNs, connection reuse and better upstreams.
  • Alert on CoreDNS latency, errors and saturation.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.