Prasanta Kumar Behera
Kubernetes Administration — Level by Level›25 · Cluster lifecycle at scale
Learning Hub / Kubernetes & Platform / Kubernetes Administration — Level by Level

Lesson 25 of 32 · Level 4 — Production

Cluster lifecycle at scale

One cluster upgrade is a procedure; fifty is a programme. Inventory, upgrade waves, automated pre- and post-checks, maintenance windows and immutable nodes keep a fleet current without drama.

Advanced
Key wordsfleetupgrade wavescanary clustermaintenance windowpre-checksimmutable nodesversion inventory

From procedure to programme

Lesson 12 upgraded one cluster by hand. Real platforms run tens or hundreds of clusters: per environment, per region, per edge site. At that size the questions change:

  • Which versions are we running, where? → inventory
  • In what order do we upgrade? → waves
  • How do we know each step worked? → automated gates
  • When are we allowed to touch production? → maintenance windows
  • How do we undo? → immutable nodes and rebuilds

Updating the software on one phone is easy. Updating every phone in a school is different. You try it on the teachers' phones first (canary), then one class, then one year group, then everyone. You check each group works before moving on, you only do it after lessons (maintenance window), and you keep the old version ready in case the new one breaks the timetable app.

1. Know your fleet

You can't plan what you can't see. Collect, for every cluster: Kubernetes version, node OS image, add-on versions (CNI, CSI, ingress, cert-manager…), certificate expiry, and owner. The quick version:

$ for ctx in $(kubectl config get-contexts -o name); do
    printf '%-24s %s\n' "$ctx" "$(kubectl --context "$ctx" version -o json 2>/dev/null | jq -r .serverVersion.gitVersion)"
  done
kind-lab                 v1.31.0
edge-site-042            v1.30.4
prod-eu-1                v1.31.2

At scale this lives in a dashboard (metrics labels, or a fleet-management tool), not a laptop loop.

2. Upgrade in waves

Wave Clusters Wait before next wave
0 · Canary Internal/test clusters that mirror production 2–3 days
1 · Dev Development clusters 2–3 days
2 · Staging Pre-production 1 week
3 · Prod A A small slice of production (one region or a few sites) 3–7 days
4 · Prod B… The rest, in batches —

Each wave widens the blast radius only after the previous one has proved itself under real traffic.

3. Gates: automated pre- and post-checks

Every cluster, every time, runs the same checks. A cluster that fails pre-checks is skipped, and a wave whose post-checks fail stops the programme.

Pre-checks Post-checks
All nodes Ready All nodes Ready on the new version
No crash-looping system pods System pods healthy (CNI, DNS, CSI, ingress)
etcd snapshot taken and verified Synthetic test: deploy, expose, call, delete a probe app
No deprecated APIs in use Error rate and latency SLOs unchanged
PDBs won't block drains No new warning events
Certificates not near expiry Certificates renewed

4. Maintenance windows and change control

  • Agree windows per environment (for example, production edge sites only overnight local time).
  • Announce, execute, verify, close: the same steps each time, logged.
  • Make the automation respect the window: a job that starts late should wait for the next window, not run into business hours.

5. Immutable nodes and add-ons

  • Nodes: build a new OS image (OS + container runtime + kubelet) in CI, test it, then roll nodes by replacing them (surge a new node, drain the old one). Rolling back = rolling out the previous image. This is how Cluster API, managed node groups and EKS Anywhere work.
  • Add-ons: manage them as versioned releases (Helm charts, GitOps) with their own compatibility matrix against the Kubernetes version. Upgrade them before the control plane when the new Kubernetes version requires it.

Try it: a mini fleet

  1. Create three kind clusters on different versions using --image kindest/node:<version> (pick versions listed in the kind release notes): canary, dev, prod.
  2. Write a small script that runs the pre-checks above against each context and prints PASS/FAIL per cluster.
  3. Add a post-check that deploys a probe pod and Service, calls it, and deletes it.
  4. "Upgrade" by recreating canary on a newer image, run the post-checks, and only then do dev. You've just built the skeleton of an upgrade pipeline.

Going deeper: running a fleet for years

  • Budget for three minor upgrades a year. Falling two versions behind turns routine work into a risky project.
  • Use declarative cluster definitions (Cluster API, EKS Anywhere cluster specs, Rancher/RKE2 configs) stored in Git, so an upgrade is a reviewed pull request that changes a version field.
  • Keep edge sites independently upgradable and roll them in small batches: one bad image shouldn't strand hundreds of remote locations.
  • Track "time to patch" (from upstream release to 100% of the fleet) as a platform metric. It shows whether your process scales.

Recap

  • At scale: inventory → waves → automated gates → windows → immutable rebuilds.
  • Start with a canary wave; stop the programme when a gate fails.
  • Replace nodes from tested images instead of patching them in place.
  • Treat add-ons and clusters as versioned, declarative configuration in Git.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.