Upgrades & version skew

Why upgrades matter

Kubernetes releases a new minor version roughly three times a year, and each minor gets about a year of patch support. Fall behind and you lose security fixes. kubeadm clusters also renew their certificates during upgrades, so never upgrading eventually means expired certificates.

The version-skew rules

These rules are what make a rolling upgrade possible:

The rule to remember: nothing may be newer than the API server. So the order is always control plane first, then kubelets, one minor at a time.

Before you touch anything

1. Read the release notes for the target version, especially removed APIs and changed defaults.
2. Back up etcd (next lesson). An upgrade is exactly when you want a fresh snapshot.
3. Find deprecated API usage. The API server counts requests to deprecated APIs. Here something still calls an API that 1.32 removes:

$ kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis
apiserver_requested_deprecated_apis{group="flowcontrol.apiserver.k8s.io",removed_release="1.32",resource="flowschemas",subresource="",version="v1beta3"} 1

4. Check PodDisruptionBudgets that could block drains: kubectl get pdb -A.
5. Update the package repository. pkgs.k8s.io has one repository per minor version, so change v1.31 to v1.32 in /etc/apt/sources.list.d/kubernetes.list on every node before installing new packages.

Step 1: the first control-plane node

$ sudo apt-mark unhold kubeadm
$ sudo apt-get update && sudo apt-get install -y kubeadm='1.32.x-*'
$ sudo apt-mark hold kubeadm
$ sudo kubeadm upgrade plan
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   NODE   CURRENT    TARGET
kubelet     cp1    v1.31.x    v1.32.x
...
You can now apply the upgrade by executing the following command:
    kubeadm upgrade apply v1.32.x
$ sudo kubeadm upgrade apply v1.32.x
...
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.32.x". Enjoy!

Replace 1.32.x with the exact patch version you chose. upgrade apply rewrites the static pod manifests (API server, controller manager, scheduler, etcd), which the kubelet restarts one by one. It also renews the certificates.

Other control-plane nodes: upgrade the kubeadm package, then run sudo kubeadm upgrade node (not apply).

Step 2: each node's kubelet, one node at a time

Control-plane nodes first, then workers. For each node:

$ kubectl drain w1 --ignore-daemonsets --delete-emptydir-data
node/w1 cordoned
evicting pod default/web-6d4b9c8f7b-4hx9t
...
node/w1 drained

On the node itself (workers also run sudo kubeadm upgrade node first, to refresh the kubelet config):

$ sudo apt-mark unhold kubelet kubectl
$ sudo apt-get install -y kubelet='1.32.x-*' kubectl='1.32.x-*'
$ sudo apt-mark hold kubelet kubectl
$ sudo systemctl daemon-reload && sudo systemctl restart kubelet

Back on a machine with kubectl:

$ kubectl uncordon w1
$ kubectl get nodes
NAME   STATUS   ROLES           VERSION
cp1    Ready    control-plane   v1.32.x
w1     Ready    <none>          v1.32.x
w2     Ready    <none>          v1.31.x    ← next

PodDisruptionBudgets: your upgrade safety net

A PDB tells Kubernetes how many pods of an app must stay up during voluntary disruptions such as drains:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web
spec:
  minAvailable: 2          # never evict below 2 ready pods
  selector:
    matchLabels:
      app: web

With 3 replicas, drain evicts one pod, waits for its replacement to be Ready elsewhere, then continues. A PDB with minAvailable equal to the replica count (or maxUnavailable: 0) blocks the drain forever. That's a classic stuck upgrade.

Recap

• One minor version at a time, control plane first, then kubelets node by node.
• Nothing newer than the API server; kubelets may lag a few minors.
• Before starting: release notes, etcd backup, deprecated APIs, PDBs, package repo updated.
• drain → upgrade kubelet → restart → uncordon, and let PDBs protect availability.