Bare-metal HA lab: kube-vip, MetalLB, ingress, TLS

Build a production-shaped cluster on one Linux host: five KVM VMs, three control planes behind a kube-vip virtual IP, Cilium, MetalLB for real LoadBalancer IPs, an ingress controller and cert-manager for HTTPS, then prove it survives losing a control-plane node.

Five VMs on one KVM host: HA control plane behind a VIP, MetalLB IPs, ingress and TLS.

The brief

"No cloud. Build a highly available cluster on bare metal (VMs are fine), give Services real IPs, serve an app over HTTPS, and show me it survives losing a control-plane node."

Three head teachers share one office phone number (the VIP). Whoever answers the phone is in charge; if she's out, another picks up the same number. The school also gets its own street addresses for visitors (MetalLB) and a reception desk with ID checks (ingress + TLS).

Resources needed

Resource	Detail
Linux host	8+ cores, 24–32 GB RAM, 150 GB free disk, virtualisation enabled
libvirt/KVM	`qemu-kvm libvirt-daemon-system virtinst` (or Terraform libvirt provider)
Ubuntu 24.04 cloud image	For the VMs
5 VMs	cp1–cp3: 2 vCPU/3 GB; w1–w2: 2 vCPU/4 GB; 20 GB disks
IP plan (libvirt `default` network 192.168.122.0/24)	DHCP .2–.99 · nodes .11–.13, .21–.22 · VIP .100 · MetalLB .200–.220
Tools	kubectl, helm, cilium CLI

Step 1: network and VMs

Shrink the default network's DHCP range so static addresses are safe:

$ virsh net-edit default          # set <range start='192.168.122.2' end='192.168.122.99'/>
$ virsh net-destroy default && virsh net-start default

Create the five VMs from the cloud image with static IPs via cloud-init (the Terraform libvirt setup in Terraform & Infrastructure as Code, lesson 06, or virt-install --cloud-init). Then prepare every node as in Kubernetes Administration, lesson 09.

Step 2: kube-vip on the first control plane

Generate the kube-vip static pod on cp1 before kubeadm init (commands from the kube-vip documentation; check the current version and flags):

$ export VIP=192.168.122.100 INTERFACE=enp1s0 KVVERSION=v0.8.9      # use the current release
$ sudo ctr image pull ghcr.io/kube-vip/kube-vip:$KVVERSION
$ sudo ctr run --rm --net-host ghcr.io/kube-vip/kube-vip:$KVVERSION vip /kube-vip manifest pod \
    --interface $INTERFACE --address $VIP --controlplane --arp --leaderElection \
    | sudo tee /etc/kubernetes/manifests/kube-vip.yaml

On recent Kubernetes versions, kubeadm's admin.conf only gets its permissions after init completes, so the kube-vip docs describe pointing the first node's kube-vip at super-admin.conf during init (and switching back afterwards). Follow the current kube-vip guide for your Kubernetes version.

Step 3: initialise, then join control planes and workers

$ sudo kubeadm init --control-plane-endpoint 192.168.122.100:6443 --upload-certs \
    --pod-network-cidr 10.244.0.0/16

On cp2 and cp3: generate the same kube-vip manifest, then run the control-plane join command from the init output (--control-plane --certificate-key …; the key expires after 2 hours). On w1 and w2: the worker join command.

$ cilium install --set ipam.mode=kubernetes && cilium status --wait
$ kubectl get nodes -o wide

Step 4: MetalLB

$ helm repo add metallb https://metallb.github.io/metallb
$ helm install metallb metallb/metallb -n metallb-system --create-namespace

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata: { name: lab-pool, namespace: metallb-system }
spec:
  addresses: [ 192.168.122.200-192.168.122.220 ]
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata: { name: lab, namespace: metallb-system }
spec:
  ipAddressPools: [ lab-pool ]

(See Kubernetes Administration, lesson 11. Cilium also offers its own LB IPAM and L2 announcements if you prefer one tool.)

Step 5: ingress and TLS

$ helm upgrade --install ingress-nginx ingress-nginx --repo https://kubernetes.github.io/ingress-nginx \
    -n ingress-nginx --create-namespace
$ kubectl -n ingress-nginx get svc ingress-nginx-controller      # EXTERNAL-IP from the MetalLB pool
$ kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.16.2/cert-manager.yaml

Create the lab CA issuer from Kubernetes Administration, lesson 19, deploy an app (e.g. podinfo from lesson 17) with an Ingress for hello.lab annotated cert-manager.io/cluster-issuer: lab-ca, and add hello.lab → the ingress EXTERNAL-IP to the host's /etc/hosts. (The community ingress-nginx project is retired; any maintained controller works the same way here.)

Step 6: prove it

$ curl -k https://hello.lab/                        # HTTPS through MetalLB + ingress
$ virsh shutdown cp1
$ kubectl get nodes                                 # still answers via the VIP (cp1 NotReady after a while)
$ kubectl -n kube-system get pods -o wide | grep kube-vip
$ curl -k https://hello.lab/                        # app still up
$ virsh start cp1

Also try: stop the node that holds the MetalLB IP and watch the address move; delete a worker and watch pods reschedule.

Interview talking points

Endpoint first: VIP/LB decided before init; stable name in certificates.
Why three control planes (etcd quorum), and why not two.
IP planning: DHCP range vs static, VIP and LB pools.
kube-vip vs external load balancer vs keepalived+HAProxy.
MetalLB L2 vs BGP; ingress vs Gateway API; cert-manager for automated TLS.

Command summary

virsh net-edit default ; virsh net-destroy default ; virsh net-start default
# 5 VMs + node prep (Admin lesson 09)
ctr run … /kube-vip manifest pod --interface $INTERFACE --address $VIP --controlplane --arp --leaderElection > /etc/kubernetes/manifests/kube-vip.yaml
sudo kubeadm init --control-plane-endpoint $VIP:6443 --upload-certs --pod-network-cidr 10.244.0.0/16
sudo kubeadm join $VIP:6443 --token … --discovery-token-ca-cert-hash … --control-plane --certificate-key …   # cp2, cp3
sudo kubeadm join $VIP:6443 --token … --discovery-token-ca-cert-hash …                                      # w1, w2
cilium install --set ipam.mode=kubernetes ; cilium status --wait
helm install metallb metallb/metallb -n metallb-system --create-namespace ; kubectl apply -f metallb-pool.yaml
helm upgrade --install ingress-nginx ingress-nginx --repo https://kubernetes.github.io/ingress-nginx -n ingress-nginx --create-namespace
kubectl apply -f cert-manager.yaml ; kubectl apply -f lab-ca-issuer.yaml ; kubectl apply -f app-with-tls-ingress.yaml
curl -k https://hello.lab/ ; virsh shutdown cp1 ; kubectl get nodes

Recap

Five VMs, IP plan with DHCP shrunk, VIP and MetalLB pool reserved.
kube-vip gives the API a floating VIP; three control planes keep etcd quorum.
Cilium for networking, MetalLB for LoadBalancer IPs, ingress + cert-manager for HTTPS.
Prove HA by shutting down a control plane and checking the API and app.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.