Prasanta Kumar Behera
SRE & Production Incident Response›02 · Incident command & communication
Learning Hub / Observability & Reliability / SRE & Production Incident Response

Lesson 02 of 7 · Method

Incident command & communication

Run incidents calmly: declaring early, severity levels, the incident commander and supporting roles, a steady rhythm of status updates, communicating with stakeholders and customers, clean handovers, and closing with a timeline and postmortem.

Practitioner → Advanced
Key wordsincident commandincident commanderrolesseverity levelsdeclaring incidentsstatus updatesstakeholder communicationhandoverwar roomtimelinepostmortem

Why structure helps

Under pressure, people do too many things at once, repeat each other's work, forget to tell anyone, and lose track of what was tried. Incident command (adapted from emergency services) gives a small, fixed structure so the team can focus.

When there's a fire, firefighters don't all grab hoses. One person is the chief who watches the whole scene and gives jobs. Some fight the fire, one talks to the neighbours and the news, and one writes everything down. The chief doesn't hold a hose, because then nobody would be watching the whole fire.

Severity levels (example)

Severity Definition Response
SEV1 Major user impact (core journey down or data at risk) Page IC + on-call; updates every 30 min; exec notification
SEV2 Significant degradation or a major feature down Page on-call + IC; updates every 60 min
SEV3 Minor impact, workaround exists Business-hours response; ticket

Define severities by user impact, not by which component is broken.

Roles

  • Incident Commander: declares, assigns roles, sets priorities, makes decisions (e.g. roll back vs keep investigating), keeps time, and decides when to escalate.
  • Operations/technical lead: runs the investigation, coordinates responders, proposes mitigations.
  • Communications lead: writes status updates, talks to support, customers and executives.
  • Scribe: keeps the timeline (times, actions, findings, decisions) in the incident channel or doc.

In small teams one person may hold two roles, but not IC + technical lead during a serious incident.

Running the incident

  1. Declare in a known channel ("Declaring SEV2: checkout errors ~5% since 14:05. I'm IC.").
  2. Assign roles and open a dedicated channel/bridge.
  3. Stabilise first: prefer quick mitigations (roll back, fail over, scale, disable a feature flag) over root-causing while users suffer.
  4. Update on schedule:
[14:40] SEV2 update #2: checkout errors
Impact: ~5% of checkout attempts fail with 502 since 14:05; browsing unaffected.
Known: errors started with the 14:02 payments deploy; rollback in progress.
Doing: rolling back payments to the previous version (ETA 14:50); watching error rate.
Next update: 15:10 or sooner if status changes.
  1. Resolve when user impact ends; keep monitoring; announce clearly.
  2. Postmortem within a few working days (lesson 01).

Handovers

Long incidents cross shifts. Hand over explicitly:

Handover (IC):  current impact · current hypothesis · what's been ruled out ·
                mitigations in place (and how to undo them) · next steps · open risks ·
                who's doing what · next update time

The new IC confirms "I have command" in the channel.

Communication hygiene

  • One incident channel; side discussions in threads; decisions stated explicitly ("Decision: fail over to site B at 15:00").
  • Keep customer-facing messages factual and free of speculation about causes.
  • Protect responders from interruptions; stakeholders get updates via the comms lead.

Try it: a tabletop incident

  1. With two or three colleagues (or alone, role-playing), pick a scenario: "API error rate at 8% after a deploy".
  2. Assign IC, technical lead, comms and scribe; run 30 minutes with injects (new symptoms every 5 minutes) from a facilitator.
  3. Write two status updates in the template above.
  4. Hand over IC halfway using the handover template.
  5. Afterwards, list three things that slowed you down and one change to your process.

Going deeper: incident programmes

  • Train ICs separately from engineers; rotate the role so many people can do it.
  • Tooling helps (incident bots that create channels, timelines and status pages), but roles and rhythm matter more.
  • Track time to declare, time to mitigate and update punctuality as process metrics.

Recap

  • Declare early, define severities by user impact.
  • Roles: IC (coordinates, doesn't debug), technical lead, comms, scribe.
  • Stabilise first; update on a fixed rhythm: impact, known, doing, next update.
  • Explicit handovers; close with a timeline and postmortem.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.