Lesson 07 of 7 · Fleet-scale scenarios
Symptom-based monitoring
Alert on what users feel, not what hosts report: the difference between symptom and cause alerts, turning noisy host alerts into SLO-based pages and cause dashboards, measuring alert quality, and a practical review process to cut alert fatigue.
Symptoms vs causes
| Symptom | Cause | |
|---|---|---|
| Describes | What users experience | Why it might be happening |
| Examples | Error rate, latency, failed journeys, missing data | CPU, memory, disk, restarts, queue depth |
| Number of possible alerts | Few (per user journey) | Endless (per component) |
| Good for | Paging | Dashboards, tickets, investigation |
Paging on causes creates two problems: noise (causes that don't hurt users) and gaps (user failures with no matching cause alert, as in Observability with OpenTelemetry, lesson 05's scenario).
A parent doesn't phone the doctor every time a child's temperature rises a little after running around (a cause-like signal). They phone when the child is actually unwell: can't eat, can't sleep, says it hurts (symptoms). The thermometer is still useful; it just helps explain the illness, it doesn't decide when to call.
From noisy to useful
A generic before/after for one service:
| Before (pages) | After |
|---|---|
| CPU > 90% for 5 min | Dashboard + capacity ticket if sustained > 1 day |
| Pod restarted | Dashboard; ticket if restart loop > 30 min (not paging) |
| Disk > 85% | Ticket with forecast ("full in N days"); page only if < 4 h to full |
| p99 latency > 500 ms for 1 min | Replaced by latency SLO burn-rate page |
| Error rate > 1% for 1 min | Replaced by availability SLO burn-rate page |
| — | New: synthetic checkout journey failing from 2 locations (page) |
| — | New: no requests received for 10 min during business hours (page) |
(See Observability with OpenTelemetry, lesson 06 for burn-rate rules.)
Keep causes, but in the right place
- Dashboards organised by user journey, then by service, then by component, so a symptom page leads quickly to candidate causes.
- Tickets for slow-burning risks (capacity, certificate expiry, error trends), routed to owners.
- Predictive alerts (e.g.
predict_linearfor disk) as tickets with lead time, not pages.
Measure the alerts
- Pages per on-call shift (and at night).
- Actionability: % of pages where someone had to act.
- Time to acknowledge and time to mitigate.
- Missed incidents: user-visible problems that no page caught.
A monthly alert review: for each alert that fired, ask Was it urgent? Was it actionable? Did it have a runbook? Was it a duplicate? Then delete, downgrade to ticket, merge, or improve.
Try it: an alert review
- Export last month's alerts from your system (or list the rules in a lab kube-prometheus-stack).
- Classify each as symptom or cause, and page or ticket.
- Write burn-rate symptom alerts for one service and a synthetic check with the Blackbox Exporter.
- Convert two cause alerts into dashboard panels and one into a predictive ticket (
predict_linear(node_filesystem_avail_bytes[6h], 3*24*3600) < 0). - Define three alert-quality metrics your team will track.
Going deeper: sustainable on-call
- Aim for few, meaningful pages; many teams target at most a couple of pages per on-call shift on average.
- Give every page an owner, runbook and dashboard link (see Cluster Design — Architect Track, lesson 10).
- Treat recurring pages as reliability work items; fix the system, not the alert threshold.
Recap
- Page on symptoms (SLO burn, failed journeys, absence); investigate with causes.
- Move cause alerts to dashboards, tickets and predictions.
- Measure alert quality (pages, actionability, missed incidents) and review monthly.
This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.