Lesson 05 of 7 · Fleet-scale scenarios
Automation blast radius
Automation makes good changes fast and bad changes faster. Learn the guard rails that keep automation safe: scoping, rate and percentage limits, dry runs and diffs, canaries, kill switches, approval thresholds, and the blast-radius controls Kubernetes already has.
Fast in both directions
Automation removes toil and human error, and it executes mistakes at machine speed across the whole fleet. Generic examples:
- A cleanup script with a wrong label selector deletes workloads in production clusters.
- A node auto-repair system, fed by a monitoring false positive, cordons and drains half the nodes.
- An autoscaler misreads a metric and scales a critical service to its minimum.
- A config push with a bad value reaches every cluster in minutes.
A robot lawnmower is wonderful, until it misreads the map and heads for the flower beds. You don't throw the robot away; you put up a small fence, tell it to mow only one strip first, give it a big red stop button, and make it stop by itself if it has cut more than it expected.
Guard rails for any automation
| Guard rail | Example |
|---|---|
| Scope | Explicit target lists or selectors reviewed in Git; production excluded by default |
| Dry run + diff | Print exactly what will change; require review for first runs and large diffs |
| Canary | One cluster/node/namespace first, then waves (like application rollouts) |
| Rate/percentage limits | Max N actions per minute; never more than X% of the fleet per hour |
| Health-based halt | Stop if error rates rise or more than N targets fail |
| Kill switch | A flag or pause annotation that stops all instances of the automation |
| Approval thresholds | Human approval when a run exceeds a size limit |
| Audit trail | Every action logged with who/what triggered it |
Kubernetes' own blast-radius controls
- PodDisruptionBudgets: drains, upgrades and autoscaler scale-downs respect them.
- Node controller eviction rates: when many nodes become NotReady, the controller reduces evictions (
--node-eviction-rate,--secondary-node-eviction-rate,--unhealthy-zone-threshold), and in a zone that looks fully down it stops evicting, on the assumption that the problem is the signal (e.g. network), not every node. - Rollout parameters:
maxUnavailable/maxSurgefor Deployments and DaemonSets. - Node disruption budgets in tools like Karpenter limit how many nodes can be replaced at once.
Treat these as patterns for your own automation: when many things look broken at once, suspect the observer and slow down.
Designing safe automation
- Start read-only (report what you would do) for a period; compare with human judgement.
- Add limits before adding speed.
- Make "do nothing" the safe default when inputs are missing or strange (e.g. zero targets found → stop, don't delete everything else).
- Test automation like software: unit tests for selectors, staging runs, and chaos tests of its inputs (what if the monitoring data is empty?).
Try it: fence your robot
- Write a small script (Python or Bash) that deletes pods with a label in a kind cluster; add
--dry-runthat prints the targets. - Add a limit: refuse to act on more than 20% of matching pods in one run, and stop if zero or "all" pods match unexpectedly.
- Add a kill switch: the script exits if a ConfigMap
automation-paused=trueexists. - Create a PodDisruptionBudget for a Deployment and run
kubectl drainon a node; watch the eviction wait for the PDB. - Write down the guard rails your team's riskiest automation has today, and one it's missing.
Going deeper: automation governance
- Keep an inventory of automation that can change production, with owners and guard rails documented.
- Review automation-caused incidents as a class; they often share the same missing guard rail.
- Prefer declarative reconciliation (controllers, GitOps) with staged rollout over imperative scripts across the fleet.
Recap
- Automation executes mistakes at fleet speed; add guard rails before speed.
- Scope, dry run, canary, rate/percentage limits, health halt, kill switch, approval thresholds, audit.
- Kubernetes already models this: PDBs, node eviction rate limits, rollout parameters, disruption budgets.
- When many things look broken at once, suspect the signal and slow down.
This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.