Troubleshooting: nodes, pods, networking

A repeatable method for broken clusters: narrow the scope, read the evidence, walk the layers. Applied to nodes, pods, networking and the control plane.

Method beats memorised commands

When something breaks, the temptation is to start typing commands you've seen before. A method is faster:

Scope: one pod, one app, one node, or everything?
Change: what changed recently? A deploy, an upgrade, a certificate, a config?
Evidence: events, logs, statuses. Read them fully.
Layers: walk from the symptom down (app → pod → Service → network → node → control plane) and stop at the first layer that's broken.
Test one hypothesis at a time, fix, then prevent the whole class of failure.

A doctor doesn't guess. First: where does it hurt, and since when? (scope and change). Then: let me see the X-ray (evidence). Then they check bone, then muscle, then nerve (layers). Only then do they treat, and tell you how to avoid it next time (prevention).

Scope first: three commands

$ kubectl get nodes
$ kubectl get pods -A -o wide | grep -v Running
$ kubectl get events -A --sort-by=.lastTimestamp | tail -30

What you see	Most likely layer
One pod broken, its siblings fine	That pod: config, app, image, probes
All pods of one app broken	The app's rollout, config, Secret, dependency
Everything on one node broken	That node: kubelet, runtime, disk, memory, network
Everything new stays Pending	Scheduler, capacity, quotas, admission
kubectl fails, apps still serve	Control plane: API server, etcd, certificates, LB

Pods: read the whole story

$ kubectl describe pod api-5f7d9c6b8-kx2lp
...
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
...
Events:
  Warning  BackOff  kubelet  Back-off restarting failed container api in pod api-5f7d9c6b8-kx2lp

Exit codes decoded:

Code	Meaning
0	Exited normally (a problem only if it should have kept running)
1, 2…	The application's own error. Read `logs --previous`
137	Killed by SIGKILL (128+9): OOMKilled, or killed after the grace period
143	SIGTERM (128+15): asked to stop, e.g. during a rollout or eviction

No shell in the image? Modern images are often minimal ("distroless"). Bring your own tools with an ephemeral debug container:

$ kubectl debug -it api-5f7d9c6b8-kx2lp --image=busybox:1.36 --target=api
Targeting container "api". If you don't see processes from this container it may be because the container runtime doesn't support this feature.
/ # ps
/ # wget -qO- http://localhost:8080/healthz

Nodes: conditions, kubelet, runtime

$ kubectl describe node w2 | sed -n '/Conditions:/,/Addresses:/p'
Conditions:
  Type             Status  Reason                       Message
  MemoryPressure   False   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     True    KubeletHasDiskPressure       kubelet has disk pressure
  PIDPressure      False   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    KubeletReady                 kubelet is posting ready status

DiskPressure True explains a lot: image garbage collection kicks in, and pods get evicted. On the node (SSH, or kubectl debug node/w2 -it --image=busybox:1.36, which mounts the host at /host):

$ df -h /var/lib/containerd /var/log
$ systemctl status kubelet containerd
$ journalctl -u kubelet --since "15 min ago" | tail -50
$ sudo crictl ps -a

NotReady node checklist: is the kubelet running? Can it reach the API server (network, firewall, load balancer)? Is its client certificate expired? Is the container runtime up? Is the CNI healthy on that node?

Networking: the ladder from lesson 04

Pod Ready? → 2. Service has endpoints? → 3. Pod IP reachable? (CNI) → 4. ClusterIP reachable? (kube-proxy) → 5. DNS resolves? (CoreDNS)

Stop at the first rung that fails. That's your layer.

Control plane: when kubectl itself fails

The control plane runs as static pods, so you can inspect it even when the API is down, straight from the node:

$ sudo crictl ps -a | grep -E 'kube-apiserver|etcd'
$ sudo crictl logs <container-id> 2>&1 | tail -30
$ sudo ls /etc/kubernetes/manifests
$ sudo kubeadm certs check-expiration

Typical culprits: an expired certificate, a typo in a static pod manifest after a manual edit, etcd out of disk or quorum, or the load balancer in front of the API servers.

Try it: five breakages, find each one blind

Ask a friend to break your kind lab while you look away, or do it yourself and come back tomorrow:

Give a Deployment an image tag that doesn't exist.
Set a container's memory limit to 8Mi.
Change a Service selector to a typo.
Taint both workers NoSchedule and scale a Deployment up.
docker exec lab-worker2 systemctl stop kubelet and wait a minute.

For each: write down the symptom, the scope, the one command that revealed the cause, and the fix. Restore everything afterwards (restart the kubelet with docker exec lab-worker2 systemctl start kubelet).

Going deeper: troubleshooting like an SRE

Keep a "what changed" timeline: deploy history, GitOps commits, upgrade logs. Most incidents follow a change.
Events expire (about an hour by default). Ship them to your logging stack if you want to investigate the past.
kubectl get --raw='/readyz?verbose' and /livez show the API server's own health checks, including etcd.
Write down what you ruled out and how. It turns a lucky guess into a repeatable RCA, and makes you credible in the postmortem (see SRE & Production Incident Response).

Recap

Scope → change → evidence → layers → one hypothesis at a time → prevent.
Pods: describe + logs --previous + exit codes; kubectl debug for minimal images.
Nodes: conditions, kubelet and runtime status, disk and memory; crictl works without the API.
kubectl fails but apps serve → look at the control plane: certificates, manifests, etcd, load balancer.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.