Runbooks & the knowledge base · Prasanta Kumar Behera

Build a knowledge base people actually use at 2 a.m.: runbooks linked from every alert, a searchable catalogue of known issues (symptom → cause → action), where to keep them, how to keep them current after every incident, and when to automate a runbook step.

Memory doesn't scale

Every incident teaches something. If that lesson only lives in one engineer's head (or a chat thread nobody will find), the next person pays for it again. A good knowledge base turns incidents into reusable answers.

A family first-aid box has a card inside: "Cut → clean, plaster. Burn → cool water, 10 minutes." Nobody has to remember in the panic; the card says what to do. Runbooks are those cards for systems, and every time something new happens, you add a card.

Two kinds of documents

	Runbook	Known-issue entry
Starts from	An alert	A symptom or error message
Purpose	What to do when this alert fires	Recognise a problem seen before and apply the known fix
Linked from	The alert's `runbook_url`	Search, runbooks, RCAs

A runbook template

# Runbook: EtcdDatabaseQuotaLowSpace

**Meaning:** etcd is close to its database quota. At the quota, writes fail cluster-wide.
**Impact:** new deployments, scaling and leader elections start failing.

## Quick checks
- `kubectl get --raw /healthz/etcd`
- On a control-plane node: `etcdctl endpoint status -w table` (DB size, leader)

## Mitigate
1. Take a snapshot first.
2. Compact + defragment one member at a time (see scenario: etcd out of space).
3. `etcdctl alarm list` / `alarm disarm` once space is back.

## Escalate
Platform on-call secondary; if data loss is possible, the platform lead.

## Links
Dashboard · Logs · Past incidents: INC-… · Known issue: KI-017

Link it from the alert rule:

annotations:
  summary: "etcd database is above 80% of its quota"
  runbook_url: "https://wiki.example.com/runbooks/etcd-quota"

A known-issue entry

# KI-023: Longhorn volume stuck attaching after node reboot

**Symptom:** pod events show `AttachVolume.Attach failed` / `MountVolume.MountDevice failed`;
Longhorn UI shows the volume "Attaching" or "Faulted".
**Cause (most common):** multipathd claims Longhorn's block devices; or iscsid not running after reboot.
**Action:** check `systemctl status iscsid`, `multipath -ll`; blacklist Longhorn devices in multipath.conf.
**Prevention:** node image has multipath blacklist + iscsid enabled (done 2026-09, image v8).
**Links:** INC-0419 RCA, runbook LonghornVolumeDegraded

Where to keep them

Docs as code (Markdown in Git, rendered by a docs site or a developer portal such as Backstage TechDocs): reviewed, versioned, near the code.
A wiki (Confluence or similar): easy to edit, but needs ownership to stay current.
Whatever you choose: searchable, linked from alerts, owned (each page has a team), and accessible during an outage (don't host the only copy on the platform that just failed; keep an export or mirror).

Keep it alive

Every RCA ends with "update runbook/known issues" as an action item.
Review runbooks when an alert fires: was it accurate? Fix it right after the incident.
Delete or archive outdated pages; stale instructions are dangerous.

Automate the proven steps

Frequent, safe, well-understood steps (collecting diagnostics, restarting a stuck component, cleaning old images) can become scripts, then automation, with humans still approving risky actions (see AI-Assisted Infrastructure Engineering, lesson 12).

Try it: start your knowledge base

Pick your three noisiest alerts and write a runbook for each using the template.
Add runbook_url annotations to those alert rules.
Write three known-issue entries from incidents you remember, with the exact error text.
Search your knowledge base for an error message you've seen recently; if nothing comes up, write the entry.
Make sure a copy of the runbooks is reachable if your main platform is down.

Going deeper: knowledge that scales

Track runbook coverage: percentage of paging alerts with a tested runbook.
Tag incidents and known issues by failure class to spot patterns (see SRE & Production Incident Response, lesson 01).
Use AI search over the knowledge base carefully: it helps find entries, but humans verify before acting.

Recap

Runbooks start from alerts; known-issue entries start from symptoms.
Link runbooks from alerts (runbook_url); put exact error text in known issues.
Keep them searchable, owned, versioned, and reachable during outages.
Update them after every incident; automate proven, safe steps.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.