HA control plane & load balancing · Prasanta Kumar Behera

Remove the control plane as a single point of failure: three control-plane nodes, a stable API endpoint behind a load balancer or VIP, and the etcd topology that fits your risk.

Three control planes behind one VIP, spread across failure domains.

What "HA control plane" means

In lesson 09 we built one control-plane node. If it dies, running pods continue, but nothing can be deployed, scaled, healed or even inspected. High availability means any single control-plane node can fail and the cluster keeps being manageable.

Three ingredients:

Several API servers (they're stateless, so this is easy).
A replicated etcd with quorum (3 or 5 members).
One stable endpoint (a load balancer or virtual IP) that all clients use.

A shop with one till closes when the cashier is sick. Give it three tills (API servers), one shared stock book kept in three copies that must agree (etcd), and one front door that sends customers to whichever till is open (the load balancer). One cashier can go home and the shop stays open.

Two etcd topologies

	Stacked etcd	External etcd
Where etcd runs	On each control-plane node	On its own hosts
Machines (minimum)	3	6 (3 control plane + 3 etcd)
One node failure loses	An API server and an etcd member	Just one of the two
Operations	Simpler; kubeadm's default	More moving parts, more isolation
Good for	Most clusters	Very large or strictly isolated environments

Start with stacked. Choose external when etcd needs its own disks, scaling or blast-radius isolation.

The stable endpoint

Clients (kubelets, kubectl, controllers) must reach some healthy API server. Common options:

Option	How it works	Where it fits
Cloud load balancer	Managed L4 LB in front of port 6443	Clouds, OpenStack (Octavia)
HAProxy + keepalived	HAProxy balances; keepalived floats a VIP between two LB hosts	Bare metal, VMs
kube-vip	A static pod on the control-plane nodes announces the VIP itself (ARP or BGP)	Bare metal/edge without extra LB machines
DNS round-robin	Several A records	Avoid: no health checks, slow failover

A minimal HAProxy config on dedicated LB hosts, health-checking each API server's /readyz:

frontend k8s-api
    bind *:6443
    mode tcp
    default_backend k8s-api

backend k8s-api
    mode tcp
    balance roundrobin
    option httpchk GET /readyz
    http-check expect status 200
    default-server inter 5s fall 3 rise 2 check check-ssl verify none
    server cp1 10.10.0.10:6443
    server cp2 10.10.0.11:6443
    server cp3 10.10.0.12:6443

And keepalived moving a VIP (10.10.0.100) between the two LB hosts:

vrrp_instance K8S_API {
    state MASTER            # BACKUP on the second host
    interface eth0
    virtual_router_id 51
    priority 100            # lower (e.g. 90) on the second host
    advert_int 1
    virtual_ipaddress {
        10.10.0.100/24
    }
}

Building it with kubeadm

Create the endpoint first (VIP or LB), for example with the name k8s-api resolving to 10.10.0.100.
On cp1: kubeadm init --control-plane-endpoint k8s-api:6443 --upload-certs … (as in lesson 09).
On cp2 and cp3, run the control-plane join command printed by init. It includes --control-plane --certificate-key <key>.
Join workers as usual. They talk to k8s-api:6443, never to a specific node.

$ kubectl get nodes -l node-role.kubernetes.io/control-plane
NAME   STATUS   ROLES           AGE   VERSION
cp1    Ready    control-plane   40m   v1.31.x
cp2    Ready    control-plane   12m   v1.31.x
cp3    Ready    control-plane   10m   v1.31.x

The certificate key expires

--upload-certs stores the control-plane certificates, encrypted, in the cluster, and deletes them after two hours. Joining later? Run sudo kubeadm init phase upload-certs --upload-certs on an existing control-plane node to get a new key.

Who is actually in charge?

API servers are all active. The scheduler and controller manager are active on one node at a time, chosen by leader election using Lease objects:

$ kubectl get lease -n kube-system kube-scheduler kube-controller-manager
NAME                      HOLDER                                   AGE
kube-scheduler            cp2_4c1b9f3e-…                           40m
kube-controller-manager   cp1_8a2d7e61-…                           40m

Stop the holder and, within seconds, another node takes the lease.

Try it: fail a control-plane node

With a 3-control-plane lab (VMs, or kind with three control-plane entries, which also sets up a load balancer container for you):

Note the lease holders and the etcd leader.
Stop one control-plane node entirely (multipass stop cp2, or docker stop lab-control-plane2 on kind).
kubectl get nodes still works (through the endpoint), and scaling a Deployment still works.
Check which node holds the leases now, and that etcd still has a leader (2 of 3 = quorum).
Stop a second control-plane node. Writes now fail: that's quorum loss. Start both again.

Going deeper: HA that survives real failures

Spread control-plane nodes across failure domains: racks, power feeds, availability zones. Three nodes on one hypervisor is not HA.
etcd across zones needs low, stable latency between members. Across distant sites, prefer a cluster per site.
Health-check /readyz, not just the TCP port. An API server can accept connections while its etcd is unhealthy.
Upgrades (lesson 12) run one control-plane node at a time; HA is what makes that zero-downtime.
Workers also cache and retry, so short API blips are survivable. Long ones stop scaling, healing and deployments. Set alerts on API availability from the client's point of view.

Recap

HA control plane = multiple API servers + quorum etcd + one stable endpoint.
Stacked etcd (3 nodes) for most clusters; external etcd for isolation at scale.
Use a health-checked LB or VIP (cloud LB, HAProxy + keepalived, kube-vip), never a single node's address.
Scheduler and controller manager use leader election; etcd needs a majority to keep writing.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.