Prasanta Kumar Behera
Incident Handling — On-Call Playbook & Real Scenarios›01 · Answering the page
Learning Hub / Incident Handling / Incident Handling — On-Call Playbook & Real Scenarios

Lesson 01 of 19 · The on-call playbook: from page to prevention

Answering the page

What to do in the first minutes after a page: acknowledge so people know it's owned, assess real user impact, decide severity, stabilise before you diagnose, check what changed, and send the first update, while starting a timeline you'll thank yourself for later.

Practitioner
Key wordson-callacknowledgetriageimpactseveritystabilise firstwhat changedfirst updatetimeline
Page alert or report 1 Acknowledge "I own it" 2 Assess impact · severity 3 Stabilise rollback, failover 4 Diagnose evidence, not guesses 5 Fix + verify SLOs back to normal 6 RCA blameless 7 Prevent remove the class 8 communicate at every step: first update within minutes, then on a fixed rhythm
The life of an incident, from the page to prevention.

The page arrives

It's 02:07. Your phone says APIHighErrorRate — checkout 5xx > 5% for 5m. What you do in the next five minutes matters more than anything clever you do later.

When the fire alarm rings in a school, the teacher doesn't first investigate which wire caused it. She says "I've got this" (acknowledge), checks if anyone is in danger (impact), gets everyone out safely (stabilise), and tells the office (first update). Only then does someone look for the cause.

Step 1: acknowledge

Acknowledge in your paging tool right away. It stops escalation to the next person and tells the team the incident is owned. It does not mean you know what's wrong.

Step 2: assess impact, not just the alert

Alerts describe symptoms from one angle. Ask:

  • Who is affected? All users, one region, one customer, internal only?
  • How badly? Errors, slowness, data at risk?
  • Since when, and is it getting worse?

Look at the SLO/edge dashboard (ingress or load-balancer error rate and latency), synthetic checks, and support channels. A noisy alert with no user impact is still worth fixing, but it's not a 2 a.m. emergency.

Step 3: decide severity and get help early

Severity Typical meaning Response
SEV1 Core journey down, data at risk Declare, incident commander, frequent updates
SEV2 Major degradation, key feature down Declare, updates hourly
SEV3 Minor impact, workaround exists Fix in working hours

If it looks like SEV1/SEV2, declare it and pull in help now (lesson 03). Downgrading later is cheap.

Step 4: stabilise before you diagnose

Your first goal is to stop user impact, not to understand it:

  • Recent change? Roll back (kubectl rollout undo, git revert for GitOps, redeploy the previous image digest).
  • New feature? Turn off its feature flag.
  • Capacity? Scale up, or shed non-critical load.
  • One zone/site bad? Fail over or drain it.

Rollback is not admitting defeat; it buys calm time to find the real cause.

Step 5: what changed?

Most incidents follow a change. Check quickly:

  • Deploys and GitOps syncs in the last hours
  • Config changes, feature flags, infrastructure (Terraform applies, node upgrades)
  • Time-based changes: certificate expiry, cron jobs, token expiry, DNS TTLs
  • External dependencies: cloud provider status, third-party APIs

Step 6: first update and a running timeline

[02:14] SEV2: checkout errors ~6% since 02:01. Browsing OK.
Known: payments v2.3 deployed 01:58. Rolling back now.
Next update 02:40 or sooner.

Keep a timeline in the incident channel: time, what you did, what you saw. One line each is enough.

Try it: a timed first-five-minutes drill

  1. Pick a service you know and write its "first five minutes" checklist: dashboards, rollback command, owners, status page.
  2. Ask a colleague to break something in a lab (bad image, wrong config) and page you (a chat message is fine).
  3. Time yourself: acknowledge, impact assessment, stabilisation action, first update.
  4. Review: what slowed you down? Missing dashboard, unknown rollback path, no runbook link?
  5. Fix one gap and repeat next week.

Going deeper: on-call hygiene

  • Every paging alert should have a runbook link and a dashboard link in its annotations (lesson 04).
  • Keep rollback paths tested; a rollback you've never run is a risk in itself.
  • Protect on-call health: reasonable rotations, handover notes, and time off after heavy nights.

Recap

  • Acknowledge immediately; it means ownership, not a fix.
  • Assess real user impact, set severity, declare early for big ones.
  • Stabilise first (rollback, flag off, scale, fail over), then diagnose.
  • Ask what changed; post the first update; keep a timeline from minute one.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.