Lesson 10 of 13 · Operations
Observability & SLOs
Design how ParcelPath will know the platform meets its NFRs: a monitoring architecture that survives the failures it reports on, SLOs derived from the NFR sheet, an alert catalogue with owners and runbooks, and the user-side measurements that internal health checks miss.
From NFRs to SLOs
Each measurable NFR (lesson 01) becomes an SLO with an SLI and a data source:
| NFR | SLI | Source | SLO |
|---|---|---|---|
| NFR-01 | Good responses / all (5xx + timeouts = bad) on tracking API | Ingress metrics | 99.95% / 30 d |
| NFR-02 | GET /track requests < 300 ms / all |
Ingress histogram | 99% / 30 d |
| — (new) | Synthetic "track a parcel" journey success from outside | External probes (2 regions) | 99.9% / 30 d |
| NFR-04 | Replication lag of orders DB < 5 min | CloudNativePG metrics | 99.9% of minutes |
Alert on these with burn-rate rules (see Observability with OpenTelemetry, lesson 06).
A hospital doesn't only check that the machines are switched on; it checks that patients are getting better. Machines on (health checks) and patients healthy (user SLOs) are different questions. And the hospital's alarm system must have its own backup power, or it goes quiet exactly when the lights go out.
Monitoring architecture
prod-a / prod-b / nonprod: Prometheus (HA pair) + log/trace collectors
│ remote write / OTLP (buffered)
▼
mgmt cluster (DC-B) + object storage: Mimir or Thanos, Loki, Tempo, Grafana, Alertmanager (3)
│
├── meta-monitoring Prometheus (separate) → watches the stack itself
└── Watchdog alert → external dead man's switch service
external synthetic probes (outside both DCs) → tracking journey, DNS, TLS expiry
Design choices to record:
- Where central storage lives (and what happens if DC-B fails: alerts from prod-a's local Prometheus still route to Alertmanager peers or a fallback).
- Retention per signal (see Scaling Prometheus to Production, lesson 06).
- Access: teams see their namespaces' data; platform sees everything.
The alert catalogue
| Alert | Signal | Severity | Owner | Runbook |
|---|---|---|---|---|
| TrackingAPIErrorBudgetFastBurn | NFR-01 burn 14.4× (1h/5m) | page | Tracking team | rb/tracking-errors |
| TrackingJourneySyntheticFailing | Synthetic failures from both regions | page | Platform | rb/synthetic |
| OrdersReplicationLagHigh | Lag > 2 min for 5 min | page | Platform (DB) | rb/pg-lag |
| EtcdHighFsyncLatency | p99 > 25 ms for 10 min | ticket | Platform | rb/etcd-disk |
| NodeNotReady | > 1 node for 10 min | ticket | Platform | rb/node |
| CertificateExpiringSoon | < 14 days | ticket | Platform | rb/certs |
| MonitoringWatchdogMissing | External switch fires | page | Platform | rb/monitoring-down |
Rules: page only for user impact or imminent risk; everything else is a ticket; every alert has an owner and runbook; review the catalogue quarterly.
Scenario: all health checks pass, users still fail
During a partner launch, customers report tracking pages failing. Every pod is Ready, the services' own error rates are near zero, and no alerts fire.
Which SLO was missing from the design?
The design measured services from inside. The failure was before the services: a WAF rule deployed for the launch blocked a new mobile app version's requests, which never reached the ingress, so internal error rates stayed perfect.
Missing: an SLI measured at the edge and from outside:
- Synthetic journeys from external locations with the real client paths (web and app user agents).
- Edge metrics (CDN/WAF block rates and status codes) in the same dashboards and alerts.
- Client-side telemetry (real user monitoring) where possible.
Add the synthetic SLO to the NFR/SLO sheet and an alert for sudden changes in WAF block rates.
Try it: design ParcelPath's monitoring
- Draw the monitoring architecture above for your environment and mark what still works if the mgmt cluster fails.
- Write burn-rate rules for NFR-01 using your ingress controller's metrics.
- Deploy the Prometheus Blackbox Exporter (or a synthetic tool) probing an external URL, and add an SLO for it.
- Build five rows of your own alert catalogue with owners and runbook outlines.
- Route the Watchdog alert to an external heartbeat service in a lab (many have free tiers) and stop Alertmanager to see it fire.
Going deeper: operable observability
- Put deployment markers and change events on dashboards; "what changed?" answers most incidents.
- Treat dashboards and alerts as code (reviewed, versioned, deployed via GitOps).
- Budget observability cost from day one: cardinality limits, sampling and retention per tier.
Recap
- Derive SLOs from the NFR sheet, plus user-side SLIs (synthetic, edge).
- Monitoring must survive the failures it reports: central storage elsewhere, meta-monitoring, dead man's switch.
- Keep an alert catalogue: signal, severity, owner, runbook; page only for user impact.
- Internal green ≠ users happy: measure at the edge and from outside.
This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.