DNS end to end · Prasanta Kumar Behera

Follow a DNS lookup from a pod to CoreDNS and beyond: resolv.conf, search domains and ndots, headless Services, forwarding to corporate DNS, and fixing slow or failing lookups at scale.

The lookup path

app: getaddrinfo("web")
  → /etc/resolv.conf (written by the kubelet): nameserver 10.96.0.10, search domains, ndots:5
  → CoreDNS (Service kube-dns in kube-system)
      cluster names (*.cluster.local) → answered from the Kubernetes API
      everything else               → forwarded upstream (node resolvers or configured servers)

Looking up a name is like asking the school receptionist for a phone number. If you just say "Sam", the receptionist first checks your own class (the search domains), then the whole school, and only then the city phone book. That's handy for classmates, but slow if you actually wanted "Sam's Pizza in town", because they check every class list first. ndots decides how many dots a name needs before the receptionist goes straight to the city phone book.

resolv.conf inside a pod

$ kubectl -n shop exec deploy/web -- cat /etc/resolv.conf
search shop.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:5

web → tries web.shop.svc.cluster.local first: found. ✅
api.example.com (2 dots < 5) → tries api.example.com.shop.svc.cluster.local, …svc.cluster.local, …cluster.local, then api.example.com. Up to four failed lookups first (often doubled for A and AAAA).

Fixes for external-heavy apps: use fully qualified names with a trailing dot (api.example.com.), or lower ndots for that pod:

spec:
  dnsConfig:
    options:
      - name: ndots
        value: "2"

Names CoreDNS serves

Name	Resolves to
`web.shop.svc.cluster.local`	The Service's ClusterIP
`db.shop.svc.cluster.local` (headless)	Each ready pod's IP
`db-0.db.shop.svc.cluster.local`	One StatefulSet pod's IP
`_http._tcp.web.shop.svc.cluster.local` (SRV)	Port and target for a named port

The Corefile

.:53 {
    errors
    health
    ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {
       pods insecure
       fallthrough in-addr.arpa ip6.arpa
    }
    prometheus :9153
    forward . /etc/resolv.conf
    cache 30
    loop
    reload
    loadbalance
}
corp.internal:53 {
    errors
    cache 30
    forward . 10.1.0.53 10.1.0.54     # company DNS servers for corp.internal
}

(A typical default, plus a stub domain for corp.internal.) Edit it with kubectl -n kube-system edit configmap coredns; the reload plugin picks up changes.

When DNS is slow or failing

Symptom	Likely cause	Fix
Intermittent 5-second delays	UDP conntrack races on parallel A/AAAA queries	NodeLocal DNSCache; `single-request-reopen` (glibc) in some cases
Timeouts under load	CoreDNS CPU-throttled or too few replicas	More replicas, proper requests/limits, autoscaling
Many NXDOMAIN answers in metrics	ndots/search expansion of external names	Trailing dots, lower ndots
Every lookup fails	CoreDNS down, or egress to port 53 blocked by NetworkPolicy	Check pods, endpoints, and allow DNS (see Kubernetes Security, lesson 09)
Only external names fail	Upstream resolvers unreachable	Check the `forward` targets from the nodes

NodeLocal DNSCache runs a caching DNS agent on every node, so pods query a local address. That means fewer conntrack entries, lower latency, and less load on CoreDNS.

Try it: trace and tune DNS

Print /etc/resolv.conf from a pod, and resolve web, web.shop and kubernetes.default with busybox's nslookup.
Create a headless Service for a 3-replica StatefulSet and resolve both the Service name (three IPs) and db-0.db.
Enable query logging (add log to the Corefile), resolve example.com from a pod, and count the queries in CoreDNS logs, search-domain attempts included.
Set ndots: 2 on a test pod and compare.
Add a stub domain pointing at a public resolver (e.g. forward . 1.1.1.1 for a test zone) and verify with dig from a pod that has it.

Going deeper: DNS at scale

Scale CoreDNS with the cluster-proportional autoscaler or HPA, and spread replicas across nodes and zones.
Watch CoreDNS metrics: request rate, latency, SERVFAIL/NXDOMAIN ratios, cache hit rate, forward errors.
dnsPolicy: ClusterFirst (default), Default (use the node's resolver), None (only your dnsConfig), and ClusterFirstWithHostNet for hostNetwork pods.
For multi-cluster service discovery, look at the Multi-Cluster Services API (clusterset.local) or mesh-based discovery.

Recap

Pod → resolv.conf (search domains, ndots:5) → CoreDNS → API (cluster names) or upstream (everything else).
ndots multiplies queries for external names: trailing dots or lower ndots fix it.
Headless Services return pod IPs; StatefulSets get per-pod names.
Stub domains in the Corefile; NodeLocal DNSCache for scale and to avoid conntrack races.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.