SRE & Production Incident Response cheat sheet
51 commands from every lesson of SRE & Production Incident Response, on one page.
The eight parts
Situation | Impact, who/what/when, timeline |
Claim | The first hypothesis and the signal that suggested it |
Pivot | The evidence that changed direction |
Elimination | What was ruled out, and how |
Root cause | Mechanism + contributing factors (not 'human error') |
Fix | What stopped the bleeding |
Prevention | Changes that remove the class of failure |
Honesty | What went badly, what's still unknown |
Action items
Specific · owned · dated · verifiable | Good |
'Be more careful', 'improve monitoring' | Not action items |
Roles
Incident Commander (IC) | Coordinates, decides, delegates; doesn't debug |
Operations / technical lead | Leads investigation and mitigation |
Communications lead | Stakeholder and customer updates |
Scribe | Timeline of actions and findings |
Rhythm
Declare early; downgrade later | Cheap to cancel, expensive to be late |
Status update every 30 min (SEV1) / 60 min (SEV2) | Even when there's no news |
Update = impact · what we know · what we're doing · next update time | Fixed structure |
Antipattern → safer pattern
Liveness probe checks the database | Liveness = process health only; readiness may check dependencies |
imagePullPolicy: Always + single registry | Digests + IfNotPresent, registry mirrors/pull-through cache |
App calls the Kubernetes API per request | Cache/watch; data path independent of the control plane |
Config pushed to all clusters at once | Staged rollout with health gates |
Deep synchronous chains | Timeouts, async where possible, degrade gracefully |
Detect
count by (cluster, image) (kube_pod_container_info{container="api"}) | Which image versions run where |
kubectl rollout status deploy/api --timeout=10m | Did this rollout finish? |
GitOps per-cluster sync status + revision | Which clusters are behind |
kubectl get ds -A (DESIRED vs UP-TO-DATE vs AVAILABLE) | DaemonSets stuck on some nodes |
Design for mixed versions
Expand → migrate → contract | Schema/API changes in compatible steps |
N and N-1 must interoperate | Every change assumes mixed versions |
Feature flags decoupled from deploys | Turn on only when rollout is complete |
Guard rails
Never act on more than X% of the fleet per interval | Percentage/rate limit |
Dry run + diff reviewed before apply | See the blast radius first |
Canary scope first (1 cluster / 1 node / 1 namespace) | Small, then wider |
Stop if more than N targets fail or health drops | Automatic halt |
Kill switch (feature flag / pause annotation) | Stop everything quickly |
Built into Kubernetes
PodDisruptionBudget | Limits voluntary evictions (drains, autoscalers) |
kube-controller-manager --node-eviction-rate / --unhealthy-zone-threshold | Slows evictions when many nodes look unhealthy |
Deployment maxUnavailable / maxSurge | Rollout speed |
Karpenter NodePool disruption budgets | Limits node consolidation/replacement |
Common loops
Registry runs in the cluster that needs it to pull images | Images can't be pulled to start the registry |
Identity provider hosted on the platform you log into | Nobody can log in to fix it |
GitOps pulls from a Git server hosted in the same cluster | Nothing to sync from |
Secrets manager needs unsealing by a service that needs secrets | Deadlock |
Recovery tools
rabbitmqctl force_boot (on a stopped node) | Let a RabbitMQ node start without waiting for peers (after checking which node is most recent) |
podManagementPolicy: Parallel (StatefulSet) | Start all members together when they must find each other |
Break-glass accounts + local credentials | Access that doesn't depend on SSO |
Page on symptoms
SLO burn rate (errors, latency) at the user edge | Users are being hurt at a rate that matters |
Synthetic journey failures | Users can't complete a key flow |
Absence: no data / no traffic when there should be | Silent failures |
Ticket or dashboard for causes
High CPU, memory, disk growth, pod restarts | Investigate during work hours unless they threaten an SLO soon |
Capacity forecasts (disk full in < 3 days) | Predictive ticket, not a page |
Alert quality: % actionable, pages/week, time to acknowledge | Measure the alerts themselves |
