Operations · wrap-up
Cheat sheet & self-check
Every command from this section on one page.
Architecture
Per-cluster collection (Prometheus/agents, log and trace collectors) | Close to the workloads |
Central storage/query in the mgmt cluster or another site | Survives a production cluster failure |
Meta-monitoring + external dead man's switch | Know when monitoring itself is down |
Synthetic checks from outside both DCs | The user's view |
Alert catalogue columns
Name · signal/SLO · condition · severity · owner · runbook · notification route | One row per alert |
FMEA
Component · failure mode · effect · S · O · D · RPN = S×O×D · mitigation · owner | One row per failure mode |
S, O, D scored 1–10 (10 = worst: severe, frequent, undetectable) | Scoring scale |
Re-score after mitigation | Show risk reduced |
Chaos experiments
Steady-state hypothesis → inject → observe → abort conditions | Experiment structure |
Chaos Mesh / LitmusChaos (PodChaos, NetworkChaos, IOChaos…) | Kubernetes-native fault injection |
Start in nonprod, small blast radius, business hours | Safety rules |
Patterns
Active-passive (warm standby) | Secondary runs the platform + replicas; takes traffic on failover |
Active-active | Both sites serve traffic; needs multi-site data design |
GSLB / DNS with health checks (low TTL) | Switch traffic between sites |
Failover essentials
Decide → fence old primary → promote → switch traffic → verify | Order matters |
Measure RPO (lag at failover) and RTO (time to healthy) | Compare with NFR-04 |
Failback is a second, planned failover | Resync, then switch back |
Document structure
1 Summary · 2 Requirements (NFR/SLO sheet) · 3 Architecture overview | Front |
4 Compute & sizing · 5 Network & IPAM · 6 Traffic & edge · 7 Storage & data | Core design |
8 Security & tenancy · 9 Observability · 10 Resilience (FMEA, chaos) · 11 DR | Operability |
12 Cost · 13 Risks & open questions · Appendix: ADR index, evidence pack | Close |
Traceability
NFR → design section → ADR → evidence (test/drill) → status | One row per NFR |
