OS image creation · Prasanta Kumar Behera

Build the OS images your edge nodes boot: why immutable, versioned images beat configuring each node, image formats (ISO, RAW, QCOW2) and where each fits, building with image-builder/Packer, first-boot configuration with cloud-init or Ignition, and container-optimised OSes.

Images, not snowflakes

Configuring each node after install (packages, config, kubelet) makes every node slightly different. At the edge, you instead build a golden image: OS + container runtime + kubelet + agents, tested once, versioned, and written to every node. Changes ship as a new image version; nodes are reimaged or replaced (in-place for some OSes).

Instead of baking every cake from scratch at each lemonade stand (and getting slightly different cakes), headquarters bakes one perfect cake, photographs the recipe, and sends identical frozen cakes to every stand. Each stand only adds a name card on top (cloud-init): "Stand #42".

Formats

Format	What it is	Typical use
RAW	A byte-for-byte disk image	Stream to disk on bare metal (Tinkerbell `image2disk`)
QCOW2	Sparse, copy-on-write VM disk	VMs; Ironic/Metal3 can write it to disk
ISO	Bootable installer or live image	Virtual media or USB; appliance installs
Container/OCI images	OS delivered as OCI artifacts	Some immutable OSes and image-based updates

Compress RAW images (.gz, .xz) for transfer, and publish a checksum (and ideally a signature) next to each file.

Building images

kubernetes-sigs/image-builder builds Kubernetes node images (Ubuntu, RHEL and others) for many targets, including bare metal (raw) and VMs, using Packer and Ansible. EKS Anywhere documents image-builder for its bare-metal node images.
Packer on its own for custom images; Kiwi (SUSE), osbuild/Image Builder (Red Hat) and mkosi are other builders.
Build in CI: pinned base image, pinned package versions, a test boot (VM) with smoke tests, then publish with version, checksum and SBOM.

$ # Illustrative image-builder flow; follow the docs for your target
$ make deps-raw
$ make build-raw-ubuntu-2204-efi
$ sha256sum output/ubuntu-2204-efi-kube-*.gz > SHA256SUMS

(Make targets and supported OS versions change between image-builder releases; check its documentation.)

First boot: cloud-init and Ignition

One image, many nodes: per-node settings arrive at first boot.

#cloud-config
hostname: site042-node1
users:
  - name: ops
    groups: [ sudo ]
    ssh_authorized_keys: [ "ssh-ed25519 AAAA… ops@example" ]
write_files:
  - path: /etc/sysctl.d/90-edge.conf
    content: |
      vm.max_map_count=262144
runcmd:
  - [ systemctl, enable, --now, chronyd ]

NoCloud datasource: user-data/meta-data/network-config from a small ISO/partition or an HTTP metadata service (Tinkerbell's Tootles, Metal3's userData Secret).
Ignition (Flatcar, Fedora CoreOS) runs once in the initramfs and is declarative; Bottlerocket takes TOML settings as user data.

Container-optimised and immutable OSes

OS	Notes
Bottlerocket	Minimal, API-driven, image-based updates; supported by EKS Anywhere
Flatcar	Immutable, auto-updating container Linux; Ignition
Talos	API-only Kubernetes OS (no SSH)
SUSE Linux Micro (formerly SLE Micro)	Transactional updates, used with RKE2/K3s at the edge
Ubuntu / RHEL	General-purpose; immutability via image discipline

Immutable OSes reduce drift and attack surface; general-purpose OSes are familiar and flexible. Either works if images are versioned and nodes aren't hand-edited.

Scenario: same image, two sites, different behaviour

Image v1.8 runs perfectly at 40 sites. At two new sites, nodes randomly lose their NICs after a few hours.

What's different, if the image isn't?

The hardware batch: newer servers shipped with a different NIC firmware (or a different NIC revision) and BIOS settings (power-saving states, SR-IOV defaults). The image's driver behaves differently with that firmware.

Compare the firmware/BIOS inventory between a good and a bad site (Redfish inventory, ethtool -i, dmidecode).
Pin the firmware baseline per hardware model (lesson 04) and apply it during provisioning.
Add the new hardware revision to the compatibility test matrix before shipping it to sites.

Try it: build and personalise an image

Download an Ubuntu cloud image (QCOW2), convert it to RAW with qemu-img, compress it, and write a SHA256 checksum.
Create a NoCloud seed (cloud-localds seed.iso user-data meta-data) that sets a hostname, a user and a sysctl.
Boot the image with the seed in a VM and verify every setting.
Change one setting, rebuild the seed, boot a second VM: same image, different node.
(Stretch) Build a Kubernetes node image with image-builder for a QEMU target and boot it.

Going deeper: image pipelines

Version images semantically and keep at least the previous version available for rollback.
Generate an SBOM per image and scan it; rebuild on critical CVEs.
Sign images and verify the signature before writing to disk (a provisioning action can do this).
Keep the image-to-hardware matrix: which image versions are tested on which models and firmware.

Recap

Golden, immutable, versioned images, tested once and used everywhere; changes = new version.
Formats: RAW (stream to disk), QCOW2 (VMs/Ironic), ISO (virtual media/installers).
Build reproducibly (image-builder/Packer) in CI with checksums, SBOMs and signatures.
Personalise at first boot with cloud-init or Ignition; consider immutable OSes.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.