The shape of a strong RCA · Prasanta Kumar Behera

Write root-cause analyses that prevent the whole class of failure: the eight-part shape (situation, claim, pivot, elimination, root cause, fix, prevention, honesty), a worked generic example, blameless language, strong action items, and telling the story in two minutes.

Why the shape matters

An incident report can be a list of events, or it can change how the system and the team behave. A strong RCA explains not only what broke, but why the investigation took the path it did, and what makes the whole class of failure impossible or harmless next time.

A good RCA is like a detective story told honestly: what happened, who the first suspect was and why we suspected them, the clue that pointed elsewhere, who we ruled out, who really did it and why they could, how we stopped it, and what locks we're adding so it can't happen again, including the parts where the detectives got it wrong.

The eight parts

Situation: impact in user terms (what failed, for whom, how long), and a timeline with detection, response and recovery times.
Claim: the first hypothesis and the signal behind it.
Pivot: the piece of evidence that redirected the investigation.
Elimination: what was ruled out and how (commands, graphs, experiments).
Root cause: the mechanism, plus contributing factors (why it could happen, why it wasn't caught earlier).
Fix: what restored service, and any temporary measures still in place.
Prevention: changes that remove or contain the class of failure.
Honesty: what went badly in the response, open questions, luck involved.

A worked (generic) example

Situation. For three weeks, the public API's p99 latency spiked from 150 ms to 2–4 s for about 90 seconds at the start of every hour. Around 2% of requests in those windows timed out. Detected by the latency SLO burn alert after a new batch feature launched.

Claim. The database dashboard showed CPU jumping to 95% at the same moments, so the team assumed a slow query and started query tuning.

Pivot. Services that don't use the database also showed latency spikes at the same second. The common factor wasn't the database; it was something every pod shared.

Elimination. Node CPU and network were normal (node dashboards). Ingress showed no traffic surge (request rate flat). CoreDNS request rate, however, tripled at :00, with SERVFAIL/timeouts rising.

Root cause. Forty CronJobs across teams were scheduled at 0 * * * *. At :00, hundreds of new pods started simultaneously, each resolving many names (with the default ndots:5 search-path expansion multiplying queries), overloading CoreDNS. Slow DNS delayed connections for all services, and the batch jobs' simultaneous start also caused the database CPU spike: a symptom, not the cause. Contributing: no guidance on schedules; CoreDNS sized for steady load; no DNS latency alert.

Fix. Spread CronJob schedules with jitter; scaled CoreDNS; later enabled NodeLocal DNSCache.

Prevention. Admission policy warning on minute-0 schedules; NodeLocal DNSCache on all clusters; DNS latency SLO and alert; load tests include batch start storms.

Honesty. Three days were spent tuning queries based on the database graph. The latency SLO alert fired only after the batch feature raised the spike's size; the underlying pattern had existed for months.

Blameless and useful

Describe decisions in context: what people knew at the time, and why their actions made sense.
Ask "what made this possible?" and "what made it hard to detect/fix?", not "who did it?".
Action items: specific, owned, dated, verifiable. "Improve monitoring" is not an action item; "Add DNS p99 latency alert at 100 ms for 5 min, owner platform, due 10 Oct" is.
Track action items to completion; review overdue ones in a regular forum.

Tell it in two minutes

For reviews, handovers and interviews, compress the RCA:

Impact (one sentence): "Hourly 90-second latency spikes on the public API, ~2% timeouts, for three weeks."
The trap (one sentence): "The database CPU graph pointed us at queries."
The pivot (one sentence): "Non-database services spiked too, so it had to be shared infrastructure."
The cause (two sentences): "Forty CronJobs at minute 0 caused a DNS storm; slow DNS slowed everything."
The prevention (one sentence): "Jittered schedules enforced by policy, node-local DNS caching, and a DNS SLO."
The lesson (one sentence): "Correlation across unrelated services beats the loudest graph."

Try it: write and tell an RCA

Pick an incident you've seen (or invent one) and write it in the eight-part shape, with no names or confidential details.
Mark the misleading first signal and explain what dashboard or alert would have pointed the right way.
Rewrite every action item to be specific, owned, dated and verifiable.
Reproduce the example in a lab: 30 CronJobs at the same minute in a kind cluster, and watch CoreDNS metrics; then add jitter.
Tell the story aloud in under two minutes using the six-line structure; time yourself.

Going deeper: RCA culture

Hold postmortem reviews for near misses too; they're cheaper lessons.
Classify incidents by failure class (config push, capacity, dependency, expiry…) and look for patterns across quarters.
Share RCAs widely (sanitised if needed); other teams learn more from your mistakes than from your successes.

Recap

Eight parts: situation, claim, pivot, elimination, root cause, fix, prevention, honesty.
Record the misleading signal and the pivot evidence: they improve future investigations.
Blameless: ask what made the failure possible and hard to detect.
Action items: specific, owned, dated, verifiable, class-level.
Practise the two-minute story.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.