Fleet-scale scenarios · wrap-up
Cheat sheet & self-check
Every command from this section on one page.
Detect
count by (cluster, image) (kube_pod_container_info{container="api"}) | Which image versions run where |
kubectl rollout status deploy/api --timeout=10m | Did this rollout finish? |
GitOps per-cluster sync status + revision | Which clusters are behind |
kubectl get ds -A (DESIRED vs UP-TO-DATE vs AVAILABLE) | DaemonSets stuck on some nodes |
Design for mixed versions
Expand → migrate → contract | Schema/API changes in compatible steps |
N and N-1 must interoperate | Every change assumes mixed versions |
Feature flags decoupled from deploys | Turn on only when rollout is complete |
Guard rails
Never act on more than X% of the fleet per interval | Percentage/rate limit |
Dry run + diff reviewed before apply | See the blast radius first |
Canary scope first (1 cluster / 1 node / 1 namespace) | Small, then wider |
Stop if more than N targets fail or health drops | Automatic halt |
Kill switch (feature flag / pause annotation) | Stop everything quickly |
Built into Kubernetes
PodDisruptionBudget | Limits voluntary evictions (drains, autoscalers) |
kube-controller-manager --node-eviction-rate / --unhealthy-zone-threshold | Slows evictions when many nodes look unhealthy |
Deployment maxUnavailable / maxSurge | Rollout speed |
Karpenter NodePool disruption budgets | Limits node consolidation/replacement |
Common loops
Registry runs in the cluster that needs it to pull images | Images can't be pulled to start the registry |
Identity provider hosted on the platform you log into | Nobody can log in to fix it |
GitOps pulls from a Git server hosted in the same cluster | Nothing to sync from |
Secrets manager needs unsealing by a service that needs secrets | Deadlock |
Recovery tools
rabbitmqctl force_boot (on a stopped node) | Let a RabbitMQ node start without waiting for peers (after checking which node is most recent) |
podManagementPolicy: Parallel (StatefulSet) | Start all members together when they must find each other |
Break-glass accounts + local credentials | Access that doesn't depend on SSO |
Page on symptoms
SLO burn rate (errors, latency) at the user edge | Users are being hurt at a rate that matters |
Synthetic journey failures | Users can't complete a key flow |
Absence: no data / no traffic when there should be | Silent failures |
Ticket or dashboard for causes
High CPU, memory, disk growth, pod restarts | Investigate during work hours unless they threaten an SLO soon |
Capacity forecasts (disk full in < 3 days) | Predictive ticket, not a page |
Alert quality: % actionable, pages/week, time to acknowledge | Measure the alerts themselves |
