Where to start troubleshooting · Prasanta Kumar Behera

Where to start when everything looks broken: scope the problem first (one user, pod, node, zone or everything), check what changed, walk the request path outside-in, and test cheap hypotheses one at a time, writing down what you've ruled out.

Scope first, then walk the path outside-in.

Don't start with your favourite tool

Under pressure, people jump to what they know best: "let me check the database", "must be DNS". Sometimes that's right. Often it wastes the most important 20 minutes. A method beats instinct.

If the lights go out, you don't start by unscrewing one bulb. You look: is it just this room, the whole house, or the whole street? If the whole street is dark, it's not your bulb. Then you follow the path: street → fuse box → room switch → bulb.

Step 1: scope the blast radius

Question	If only part is broken…	…it points to
One user or everyone?	One user/customer	Client, account, data, a specific route
One pod or all replicas?	One pod	That pod's node, data, or a bad instance
One version or all?	New version only	The release
One node or many?	One node	Disk, network, kubelet, runtime on that node
One zone/site or all?	One zone	Infrastructure, network, a dependency in that zone
Everything	All of it	Shared dependencies: DNS, control plane, identity, network, a global config push

Step 2: what changed?

Line up the start of the symptom with recent changes (lesson 01): deploys, config, node upgrades, certificates, cron jobs, cloud events. A strong correlation is the fastest lead.

Step 3: walk the path outside-in

Follow the user's request and find the first hop that fails:

DNS, CDN/WAF, load balancer (is traffic arriving?)
Ingress/gateway (status codes, access logs)
Service → endpoints (are there ready pods?)
Pods (status, restarts, logs, events)
Nodes (pressure conditions, kubelet, runtime, kernel messages)
Control plane, network, storage, cloud provider

$ kubectl get events -A --sort-by=.lastTimestamp | tail -30
$ kubectl get pods -A | grep -vE 'Running|Completed'
$ kubectl get nodes -o wide
$ kubectl describe node <node> | sed -n '/Conditions/,/Addresses/p'

Step 4: hypotheses, one at a time

Write the hypothesis down: "The new image can't reach the database because the secret changed."
Pick the cheapest test: check the secret's timestamp; exec into a pod and connect.
Record the result, especially what's ruled out. It prevents going in circles and helps whoever joins later.
Change one thing at a time, and note each change in the timeline.

Step 5: compare good and bad

A healthy pod next to a broken one, a healthy node next to a failing one, yesterday's config next to today's: diff is one of the best troubleshooting tools there is.

$ kubectl get pod good-pod -o yaml > good.yaml && kubectl get pod bad-pod -o yaml > bad.yaml && diff good.yaml bad.yaml

Try it: find it by method

In a lab cluster, ask a colleague to break one thing silently (a NetworkPolicy, a taint, a wrong Service selector, a full disk on one node).
Before touching anything, write the scope table for the symptom.
Walk the path outside-in and note the first failing hop.
Write two hypotheses, test the cheaper one first, and record eliminations.
Afterwards, compare your time with a "just start poking" attempt on a different break.

Going deeper: better starting points

Build a service overview dashboard per critical journey: edge errors, latency, saturation, recent deploys (annotations).
Keep a dependency map so "everything is broken" quickly narrows to shared dependencies (see SRE & Production Incident Response, lesson 06).
Use traces and correlated logs to jump straight to the failing hop (see Observability with OpenTelemetry, lesson 05).

Recap

Scope first: user, pod, version, node, zone, everything.
Ask what changed, and correlate with the start time.
Walk the path outside-in and find the first failing hop.
One hypothesis at a time, cheapest test first; write down what's ruled out; compare good vs bad.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.