The on-call playbook: from page to prevention · wrap-up
Cheat sheet & self-check
Every command from this section on one page.
First 5 minutes
Acknowledge the page | Means 'I own this now', not 'it's fixed' |
Read the alert + its runbook link | What fired, since when, which service |
Check user impact (SLO dashboard, edge errors, support) | Who is hurt, how many, how badly |
What changed? (deploys, config, infra, certificates, time) | Most incidents follow a change |
Decide severity; declare if SEV1/SEV2 | Get help early |
Post the first update | Impact · what we know · next update time |
Stabilise options
kubectl rollout undo deploy/<name> -n <ns> | Roll back a bad release |
git revert <sha> (GitOps) | Roll back through Git |
Disable the feature flag | Turn off new behaviour |
Scale up / fail over / shed load | Buy time while you diagnose |
Kubernetes first look
kubectl get events -A --sort-by=.lastTimestamp | tail -30 | What just happened, everywhere |
kubectl get pods -A | grep -vE 'Running|Completed' | Anything unhealthy |
kubectl get nodes -o wide / kubectl top nodes | Node status and load |
kubectl describe pod <p> -n <ns> | Events, restarts, last state |
kubectl logs <p> -n <ns> --previous | Logs of the crashed container |
Method
Scope: user → pod → node → zone → everything | Where does it stop being broken? |
Compare a good and a bad instance | The difference is the clue |
One hypothesis at a time, cheapest test first | Write down what's ruled out |
Escalate when
No mitigation after 15–30 minutes | Don't struggle alone |
Impact is growing or severe (SEV1/SEV2) | More hands, clear roles |
It needs access or knowledge you don't have | Service owners, DBAs, network, vendor |
You're tired or unsure | Fresh eyes are a feature |
A good ask
Context: service, impact, since when | One line |
What's done and ruled out | Saves them repeating it |
The specific ask + where to join | "Check the payments DB, bridge link…" |
Runbook sections
What the alert means + user impact | Why you were woken |
Quick checks (copy-paste commands) | Confirm and scope in minutes |
Mitigations (safest first) | Stop the bleeding |
Escalation (who, how) | When to call whom |
Links: dashboard, logs, past incidents | One click away |
Known-issue entry
Symptom (exact error text) | What people will search for |
Cause | Why it happens |
Action / fix / workaround | What to do |
Prevention status + links | Is it fixed for good? |
RCA template
Summary (3 sentences) | What happened, impact, fix |
Impact (numbers) | Users, duration, errors, SLO budget used |
Timeline (UTC) | Detection, response, mitigation, resolution |
Root cause + contributing factors | Mechanism, and why it was possible |
What went well / badly / lucky | Honest response review |
Action items (owner, date) | Specific and verifiable |
Useful tools
5 whys | Ask 'why' until you reach something you can change |
Time to detect / mitigate / resolve | Measure the response |
Four kinds of fixes (strongest first)
Eliminate | Design it away (automation, safer defaults, remove the dependency) |
Detect earlier | Alert on leading indicators (80% full, 30 days to expiry) |
Limit blast radius | Canaries, quotas, limits, staged rollouts, isolation |
Recover faster | Runbooks, tested rollback/restore, automation |
Follow-through
Track actions like features (owner, due date) | Not in a forgotten doc |
Verify: test, drill or chaos experiment | Prove the fix works |
Monthly review of repeats and overdue actions | Keep it honest |
