Writing the RCA

Why write it down?

An RCA (root-cause analysis, or postmortem) turns a painful night into lasting improvements. It's for future engineers and future you, not for blame. For the deeper eight-part method, see SRE & Production Incident Response, lesson 01; this lesson gives a practical template you can fill in within an hour of calm time.

The template

# RCA: <short title>  (INC-0927, SEV2)

## Summary
<What happened, who was affected, how it was fixed. 3 sentences.>

## Impact
- Duration: 02:01–02:52 UTC (51 min)
- Users: ~6% of checkout attempts failed (EU); ~1,900 orders delayed
- SLO: 38% of the monthly error budget used

## Timeline (UTC)
02:01 errors start · 02:06 alert fires · 02:07 acknowledged · 02:14 SEV2 declared ·
02:15 rollback (no effect) · 02:31 cause found · 02:44 fix applied · 02:52 recovered

## Root cause and contributing factors
<The mechanism. Then: why it was possible, why it wasn't caught earlier.>

## What went well / badly / where we got lucky

## Action items
| Action | Type | Owner | Due |

Write it within a few days while memories are fresh, share a draft with responders, then review it together.

Finding the root cause: 5 whys (with care)

Why did the API fail?          The API server certificate expired.
Why did it expire?             kubeadm leaf certificates last one year; the cluster wasn't upgraded this year.
Why wasn't it upgraded?        The upgrade was postponed twice; no one owned the date.
Why didn't we notice expiry?   No alert on certificate expiry existed.
Why no alert?                  Monitoring covered availability, not upcoming time-based risks.

Stop when you reach causes you can change. Real incidents usually have several contributing factors; list them all rather than forcing one "root".

A short worked example

Summary. The staging-edge cluster's API server certificate expired, making kubectl, controllers and GitOps fail for 3 hours. Running workloads kept serving. We renewed certificates with kubeadm certs renew all and restarted control-plane components.

Root cause. kubeadm's one-year leaf certificates expired because the cluster had not been upgraded (upgrades renew them).

Contributing factors. No certificate-expiry alert; upgrade ownership unclear; the runbook for expired certificates didn't exist.

Went well. Workloads unaffected; the fix was quick once found. Went badly. 40 minutes spent suspecting the network. Lucky. It was staging.

Actions. Expiry alert at 30/14 days on all clusters (platform, 10 Oct) · quarterly upgrade calendar with owners (platform lead, 1 Nov) · runbook "Expired kubeadm certificates" (on-call, 3 Oct) · check all clusters' expiry today (platform, done).

The review meeting

• 30–60 minutes, responders plus owners; a facilitator who wasn't the main responder.
• Walk the timeline; ask "what made this hard?", not "who did this?".
• Agree action items with owners and dates; track them like any other work.
• Share the RCA widely (sanitised if needed).

Recap

• Template: summary, impact, timeline, root cause + contributing factors, went well/badly/lucky, action items.
• Use 5 whys until you reach changeable causes; list several factors.
• Blameless review with owned, dated actions, shared widely.