Prasanta Kumar Behera
Edge Kubernetes & Zero-Touch Provisioning›02 · Architecture for edge ZTP
Learning Hub / Bare Metal & Edge / Edge Kubernetes & Zero-Touch Provisioning

Lesson 02 of 12 · Foundations

Architecture for edge ZTP

The reference architecture for edge ZTP: a central management plane, what must run at or near each site because of layer-2 boot protocols, the networks every site needs, outbound-only connectivity, and the trust boundaries that keep one compromised site from reaching the others.

Practitioner → Advanced
Key wordsmanagement clusterworkload clustershub and spokeprovisioning networkBMC networkL2 constraintsDHCP relayoutbound-onlytrust boundariessite credentials
Central management plane Git site definitions Registry Harbor Lifecycle CAPI · EKS-A Observability metrics, logs edge site A provisioner DHCP · iPXE · HTTP cluster 1 or 3 nodes registry mirror · local DNS/NTP · BMC network edge site B provisioner DHCP · iPXE · HTTP cluster 1 or 3 nodes registry mirror · local DNS/NTP · BMC network sites dial OUT only; each has its own credentials; no site-to-site paths
A central management plane and small, self-sufficient edge sites.

The big picture

Headquarters (the management plane) keeps the master plans and the warehouse. Each lemonade stand has a small starter box (the provisioner) that knows how to unpack the kit locally, because some jobs can only be done by someone standing right there, like switching on the lights (DHCP and PXE are "local only"). Stands phone headquarters; headquarters never needs to walk in.

What must be local

Boot protocols are layer-2 local:

  • DHCP discovery is a broadcast; it doesn't cross routers unless a DHCP relay (ip helper) forwards it.
  • TFTP/HTTP image downloads over a slow WAN make boots slow and fragile.
  • BMC (Redfish/IPMI) access needs a route to the out-of-band network, which should not be internet-exposed.

So each site gets one of:

Option How Trade-off
Local provisioner A small seed node, admin machine or the first server runs DHCP/iPXE/HTTP (e.g. a Tinkerbell stack) Most robust; one more thing at the site
DHCP relay + central provisioner Site router relays DHCP to the centre Fewer components; depends on the WAN during boot
BMC virtual media Central tooling mounts an ISO via Redfish No PXE needed; needs BMC reachability and bandwidth

Site networks

  • BMC/OOB: isolated VLAN; only provisioning tools reach it.
  • Provisioning: DHCP/PXE; often the same as the node network, sometimes separate.
  • Node/cluster: node IPs and the API VIP (kube-vip or similar, lesson 11).
  • Workloads/uplink: service traffic and the outbound path to the centre.

Plan IP ranges per site from a template so every site looks the same except for a site number.

Trust boundaries

  • Per-site identity: each site has its own credentials (Git read token for its path only, registry pull credentials, telemetry push credentials).
  • No site-to-site paths: sites talk to the centre, never to each other.
  • Central secrets stay central: BMC passwords and cluster admin kubeconfigs are stored in a vault, delivered only when needed.
  • Hardware trust: record serials and MACs at day 0; where available, use TPM-based identity so a swapped device is detected. Standards such as FIDO Device Onboard (FDO) aim to automate secure device onboarding.

Management cluster responsibilities

  • Cluster lifecycle (create, upgrade, delete) through Cluster API-based tools (EKS Anywhere, Metal3/CAPM3, Rancher).
  • Fleet GitOps (lesson 10), registry and observability endpoints.
  • Protect it like tier-0 infrastructure: if the management plane is down, existing sites must keep running; you only lose the ability to change them.

Try it: draw your reference architecture

  1. Draw the central plane and two sites of different sizes, marking every network and the direction of every connection.
  2. Decide, for each site, which boot option (local provisioner, relay, virtual media) you'd use and why.
  3. List every credential a site holds, its scope, and how it's rotated.
  4. Walk through "the management cluster is down for a day": what still works at the sites?
  5. Walk through "one site is stolen": what could the thief access, and how do you revoke it?

Going deeper: architecture at scale

  • Some tools run the provisioning stack inside the management cluster on the same L2 as the hardware (common in data centres); edge designs usually bring a small provisioning footprint to each site instead.
  • Keep the site template versioned; a fleet often has several template generations alive at once.
  • Budget WAN bandwidth for image pulls and upgrades: local registry mirrors and pre-staged images matter (lesson 07).

Recap

  • Central management plane (Git, registry, lifecycle, fleet GitOps, observability) + per-site workload clusters.
  • Boot is L2-local: use a local provisioner, DHCP relay, or BMC virtual media.
  • Standard site networks from a template: BMC, provisioning, node, workload.
  • Outbound-only, per-site credentials, no site-to-site paths; sites keep running without the centre.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.