Edge networking, storage & security · Prasanta Kumar Behera

What changes at the edge: virtual IPs without cloud load balancers, living with small or intermittent uplinks, local DNS and time, telecom-style networking with Multus and SR-IOV, storage choices for one to three nodes, and security when anyone can walk up to the hardware.

Networking without a cloud

Virtual IPs. Cloud clusters get load balancers from the provider. At the edge:

kube-vip provides the control-plane VIP (used by EKS Anywhere and many RKE2/kubeadm setups) via ARP (L2) or BGP, and can also serve Service LoadBalancer IPs.
MetalLB assigns and announces IPs for type: LoadBalancer Services (L2 or BGP).
Reserve VIP ranges per site in the site template; ARP mode needs all nodes on one L2 segment.

An edge site is a cabin in the mountains. There's no city electricity company (no cloud load balancer), so you bring a generator (kube-vip/MetalLB). The road to town sometimes closes (the uplink), so you keep a clock and a phone book in the cabin (NTP and DNS). And because hikers could wander in, you lock the valuables in a safe that only opens in this cabin (TPM-bound disk encryption).

Limited or intermittent uplinks.

Assume low bandwidth, high latency and outages; nothing critical should depend on the centre in real time.
Keep images local (registry mirror, pre-loaded images), and sync telemetry with buffering (agents that store and forward).
Rate-limit big transfers (image pulls during upgrades) to protect business traffic.

Local DNS and time. Run a DNS forwarder/cache and an NTP server at the site (or use GPS/PTP where precise time matters, as in telecom). Wrong time breaks TLS certificates, etcd behaviour and log correlation.

Telecom-style networking. Network functions often need several interfaces and near line-rate performance:

Multus attaches additional networks to pods (via NetworkAttachmentDefinitions).
SR-IOV (device plugin + CNI) gives pods virtual functions of the physical NIC; DPDK for user-space packet processing.
Combine with CPU pinning, huge pages and NUMA alignment (see Linux — Level by Level, lesson 15).

Storage for one to three nodes

Site size	Options	Notes
1 node	local-path provisioner, TopoLVM (LVM-backed, capacity-aware)	No replication: back up to the centre
3 nodes	Longhorn (replicated), or local PVs + app-level replication	Ceph is usually too heavy for 3 small nodes
Any	App-level replication (databases with their own replicas)	Often the most robust choice

Remember: replicas cost capacity (3 replicas = 3× disk), and rebuilds after a node failure load the same small uplink-less site. Snapshots and off-site backups (lesson 12) matter more at the edge, not less.

Security with physical exposure

Disk encryption bound to hardware: LUKS with a TPM2 key (e.g. systemd-cryptenroll or Clevis), optionally combined with network-bound unlock (Tang) so a stolen disk or server doesn't unlock elsewhere.
Secure Boot and measured boot (TPM PCRs) so only signed boot chains run and tampering can be detected by attestation.
BMC hardening: isolated network, unique credentials, current firmware, no web UI exposure.
Per-site, revocable credentials (lesson 02); no shared keys across sites.
Minimal attack surface: immutable OS, no SSH by default (or keys only, from a bastion), CIS-hardened Kubernetes (e.g. RKE2 profile), network policies.
Tamper evidence: chassis intrusion sensors and alerts through the BMC where available.

Try it: edge building blocks

On a 3-node lab cluster, install kube-vip for the control-plane VIP (static pod or DaemonSet per its docs); stop the node holding the VIP and watch it move.
Install MetalLB in L2 mode with a small address pool, and expose a Service as LoadBalancer.
Install the local-path provisioner on a single-node cluster and Longhorn on a 3-node cluster; kill a node in the latter and watch replica rebuild.
In a VM with a virtual TPM (libvirt swtpm), encrypt a data disk with LUKS and enrol a TPM2 key with systemd-cryptenroll; reboot and confirm automatic unlock.
Block the lab's uplink and list what still works (DNS, time, image pulls, app traffic).

Going deeper: designing for the harsh edge

Environmental limits (heat, dust, power quality) shorten hardware life: monitor hardware health through the BMC and plan spares.
Use UPS signals for clean shutdown, and make sure the cluster comes back after power loss (lesson 12's scenario).
For regulated environments, keep evidence of encryption, Secure Boot state and patch levels per site (lesson 12).

Recap

VIPs with kube-vip/MetalLB replace cloud load balancers; plan ranges per site.
Design for small, intermittent uplinks: local images, buffered telemetry, local DNS and NTP.
Telecom sites: Multus, SR-IOV, CPU pinning and huge pages.
Storage: local PVs/TopoLVM for 1 node, Longhorn or app replication for 3, plus off-site backups.
Physical exposure: TPM-bound encryption, Secure Boot, BMC hardening, revocable per-site credentials.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.