kubeadm at scale: immutable images

From pets to cattle

In lesson 09 you prepared each machine by hand: swap off, modules, containerd, packages. That's fine for three machines. For 30, or 300, or for edge sites with no engineers, hand-built nodes are slow and drift: one has a different containerd version, another a leftover sysctl.

The alternative: build a node image once, test it, and create every node from it. Nodes become disposable: to change or upgrade one, you replace it.

Step 1: what goes in the image

Never bake secrets (tokens, keys) into the image: images get copied, cached and shared.

One image or two?

• One image for every role is simplest: the role is decided at boot by cloud-init. Most teams start here.
• Separate control-plane and worker images make sense when they really differ: pre-pulled control-plane images and etcd tools on control planes, or a smaller, more locked-down worker image. The cost: two images to build, test and track.

Either way, version images (e.g. k8s-1.31.4-img-7) and record which version each node runs.

Building images

• kubernetes-sigs/image-builder builds Kubernetes node images for many targets (QEMU/raw for bare metal and KVM, OVA for vSphere, cloud images) using Packer and Ansible. Cluster API providers and EKS Anywhere use it.
• Plain Packer with your own scripts works too.
• Build in CI: pinned inputs, a test boot, then publish with a version and checksum (see Edge Kubernetes & Zero-Touch Provisioning, lesson 06 for image formats).

Before turning a VM into a clonable template, remove its identity:

$ sudo cloud-init clean --logs
$ sudo truncate -s 0 /etc/machine-id
$ sudo rm -f /etc/ssh/ssh_host_*          # regenerated on first boot (distribution-dependent)

Step 2: boot machines and give them an identity

• VMs: clone the template (KVM/libvirt, vSphere, OpenStack, cloud) and attach user-data (cloud-init NoCloud ISO, config drive, or the platform's metadata service).
• Bare metal: write the image to disk with PXE/iPXE or BMC virtual media, using a provisioner such as Tinkerbell or Metal3 (see Edge Kubernetes & Zero-Touch Provisioning, lessons 04–05).

IP addresses, two common patterns:

1. DHCP reservations: the DHCP server maps each machine's MAC to a fixed IP and hostname. The image stays generic.
2. Static config in user-data: cloud-init writes the network config per machine.

# network-config (cloud-init NoCloud, netplan v2 format)
version: 2
ethernets:
  nic0:
    match: { macaddress: "52:54:00:12:34:17" }
    set-name: nic0
    addresses: [ 10.10.0.17/24 ]
    routes: [ { to: default, via: 10.10.0.1 } ]
    nameservers: { addresses: [ 10.10.0.1 ] }

Also create DNS records for nodes and, most importantly, the API endpoint (a VIP or load balancer name such as k8s-api.example.internal) before the first node boots.

Step 3: first boot, init or join by itself

Everything kubeadm needs can live in config files instead of long command lines. Prepare these values once and store them in a secret store, not in the image:

$ kubeadm token generate                     # e.g. 9a08jv.c0izixklcxtmnze7
$ kubeadm certs certificate-key              # key used to encrypt the uploaded control-plane certs

First control plane (init.yaml, delivered by user-data):

apiVersion: kubeadm.k8s.io/v1beta4          # v1beta4 is used from Kubernetes 1.31; older versions use v1beta3
kind: InitConfiguration
bootstrapTokens:
  - token: "9a08jv.c0izixklcxtmnze7"
    ttl: "24h0m0s"
certificateKey: "<certificate-key>"
nodeRegistration:
  name: cp-01
---
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
kubernetesVersion: v1.31.4
controlPlaneEndpoint: "k8s-api.example.internal:6443"
networking:
  podSubnet: 10.244.0.0/16

$ sudo kubeadm init --config /etc/kubeadm/init.yaml --upload-certs

Workers (user-data with a JoinConfiguration):

#cloud-config
hostname: w-017
write_files:
  - path: /etc/kubeadm/join.yaml
    permissions: "0600"
    content: |
      apiVersion: kubeadm.k8s.io/v1beta4
      kind: JoinConfiguration
      discovery:
        bootstrapToken:
          apiServerEndpoint: "k8s-api.example.internal:6443"
          token: "9a08jv.c0izixklcxtmnze7"
          caCertHashes: [ "sha256:<ca-cert-hash>" ]
      nodeRegistration:
        name: w-017
        kubeletExtraArgs:
          - name: node-labels
            value: "node.example.com/pool=general"
runcmd:
  - [ kubeadm, join, --config, /etc/kubeadm/join.yaml ]

Extra control planes use the same file plus a controlPlane: section with the certificateKey. Two expiry rules drive the automation design:

• Bootstrap tokens expire (24 hours by default): create new ones for later joins, or use short-lived tokens issued on demand.
• The control-plane certificates uploaded with --upload-certs are deleted after 2 hours. Joining a control plane later needs kubeadm init phase upload-certs --upload-certs (which prints a new key), or distributing the PKI securely from a secret store.

(A node can't give itself node-role.kubernetes.io/* labels through the kubelet; the NodeRestriction admission plugin blocks that. Use your own label prefix, as above, or label nodes from outside.)

Step 4: upgrades become replacements

With images, an upgrade or patch is:

1. Build and test a new image version.
2. For each node: drain → remove → boot a new node from the new image → wait for Ready (control planes one at a time, following the version-skew rules from lesson 12).
3. Watch workloads and SLOs between nodes; stop if anything degrades.

Doing this by hand works for small clusters. At scale, Cluster API (and products built on it such as EKS Anywhere) automates exactly this loop: machines from images, joined with generated configs, replaced during upgrades (lesson 28).

Recap

• Golden images hold everything identical; user-data holds identity (hostname, IP, role, credentials).
• One image for all roles is simplest; separate control-plane/worker images when they genuinely differ.
• IPs from DHCP reservations or static network-config; API VIP/DNS exists before the first boot.
• kubeadm config files + cloud-init = nodes that init or join by themselves; mind token (24 h) and uploaded-cert (2 h) expiry.
• Upgrades = new image, replace nodes, which Cluster API automates.