Lesson 06 of 19 · The on-call playbook: from page to prevention
Preventing the repeat
Turn RCA action items into real prevention: choose fixes that eliminate the failure class, detect it earlier with leading indicators, limit its blast radius, or make recovery fast and boring, then verify the fix works and watch for repeats.
The point of all this
Handling an incident well matters, but not having it again matters more. The difference between teams that improve and teams that repeat incidents is mostly follow-through.
If you trip over the same loose stair every week, putting a "careful!" sign helps a little. Fixing the stair is better. A light over the stair helps you see it (detect). A handrail stops you falling far (limit harm). And knowing where the first-aid kit is means you recover fast. The best families do all four, starting with fixing the stair.
Four kinds of fixes
| Kind | Examples | Strength |
|---|---|---|
| Eliminate | Automate certificate renewal; remove a single point of failure; safer defaults; fix the churning controller | Strongest |
| Detect earlier | Alert at 80% disk/quota, 30 days before expiry, on rising error ratios, on missing telemetry | Strong |
| Limit blast radius | ResourceQuotas and limits, canary rollouts, per-site staging, PodDisruptionBudgets | Strong |
| Recover faster | Runbooks, tested rollback and restore, one-click automation | Necessary |
Aim to have at least one eliminate or detect earlier item per significant incident.
From incident to prevention: an example
Incident: a node's disk filled with container images and logs; pods were evicted (scenario lesson 16).
- Eliminate: tune image garbage collection thresholds and container log rotation in the node image; move a chatty debug log to a lower level.
- Detect earlier: alert when node filesystem is predicted to fill within 24 hours.
- Limit blast radius: ephemeral-storage requests/limits on workloads so one pod can't fill a node.
- Recover faster: runbook with safe cleanup commands; automation to cordon and clean.
- Verify: fill a test node deliberately in staging and check alert, eviction behaviour and runbook.
Follow-through
- Track action items as real work (in the team's backlog), with owners and due dates.
- Review overdue actions and repeat incidents monthly.
- When an incident repeats, ask why the previous actions didn't prevent it; that's often the most valuable RCA.
- Use error budgets to balance feature work and reliability work (see Observability with OpenTelemetry, lesson 06).
Verify it
- Tests for code/config fixes; policy checks in CI for configuration classes (e.g. no workloads without limits).
- Drills for runbooks and restores.
- Chaos experiments for resilience fixes (see Cluster Design — Architect Track, lesson 11).
Try it: strengthen your action items
- Take the action items from your last RCA and label each: eliminate, detect earlier, limit blast radius, recover faster.
- If none are "eliminate" or "detect earlier", propose one of each.
- Choose one fix and write how you'll verify it (test, drill, chaos).
- Add a leading-indicator alert for one known risk (disk, quota, certificate expiry).
- Put a monthly "repeats and overdue actions" review in your team calendar.
Going deeper: organisational learning
- Look across incidents for failure classes (config pushes, capacity, expiry, dependencies) and fund fixes per class.
- Share prevention work visibly; it's invisible when it works, so tell the story.
- Feed patterns back into design reviews (see SRE & Production Incident Response, lesson 03).
Recap
- Four kinds of fixes: eliminate, detect earlier, limit blast radius, recover faster.
- Prefer eliminate and leading indicators; always improve recovery.
- Track actions like features, verify fixes, review repeats monthly.
This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.