Lesson 03 of 12 · Foundations
Cluster sizing for edge
Choose the right cluster shape for an edge site: single-node, 3-node compact (control plane and workloads together), or small HA with separate workers. Understand etcd quorum and the two-node problem, reserve resources for the system, and size for failure and upgrades.
The three common shapes
| Shape | Nodes | Survives | Good for |
|---|---|---|---|
| Single-node | 1 (control plane + workloads) | Nothing (node loss = site down) | Small sites, cost-sensitive, apps that tolerate downtime or are replicated elsewhere |
| 3-node compact | 3, each control plane + workloads | 1 node failure | The typical "HA at the edge" default |
| Small HA | 3 control plane + 2..N workers | 1 CP failure, worker failures | Bigger sites, heavier workloads, stricter isolation |
A single-node site is a one-person lemonade stand: cheap, but if that person is sick, the stand is closed. A 3-node compact site is three friends who can each do every job: if one is sick, the other two keep it open. To make decisions they need a majority: with three friends, two can still vote. With only two friends, if one is missing, nobody can outvote anyone and everything stops.
etcd quorum and the two-node problem
etcd (the control plane's database) needs a majority of members to accept writes:
| Members | Quorum | Failures tolerated |
|---|---|---|
| 1 | 1 | 0 |
| 2 | 2 | 0 |
| 3 | 2 | 1 |
| 5 | 3 | 2 |
Two members are worse than one: twice the hardware, still no fault tolerance, plus a split-brain risk. If budget allows only two servers, either run a single-node control plane (with the second node as a worker or standby) or use a platform that supports an external witness/arbiter.
Compact clusters in practice
- Control-plane nodes are tainted by default; remove the taint (or let your distribution's "compact" mode do it) so workloads schedule there.
- Protect the control plane: resource reservations, PriorityClasses for system components, and limits on workloads.
- Spread replicas with topology spread constraints or anti-affinity across the three nodes.
Reserve resources
# KubeletConfiguration (excerpt)
systemReserved:
cpu: "500m"
memory: "1Gi"
kubeReserved:
cpu: "500m"
memory: "1Gi"
evictionHard:
memory.available: "500Mi"
nodefs.available: "10%"
Measure on your hardware: the control plane (etcd, API server), CNI, CSI, monitoring agents and log collectors can easily use 2–4 GB of RAM on each node before any application runs.
Size for failure and upgrades
- N-1 capacity: with 3 nodes, the workloads must fit on 2.
- Upgrades: rolling replacement needs either spare capacity or a spare node (some platforms support in-place upgrades that avoid the extra node).
- Storage: replicated storage (e.g. 3 replicas) consumes 3× disk; a single-node site uses local storage and relies on backups (lesson 11).
- Headroom for growth: edge hardware is hard to add later; plan 2–3 years of growth.
Try it: quorum and capacity (kind)
- Create a kind cluster with 3 control-plane nodes (
kind: Clusterconfig with threerole: control-planeentries). - Stop one control-plane container (
docker stop) and confirmkubectlstill works. Stop a second and watch the API become unavailable. Start them again. - Remove the control-plane taint from all three nodes and deploy an app with 3 replicas and a topology spread constraint by hostname.
- Drain one node and check whether all replicas still have room.
- Write the sizing for a real site: workloads' requests, system overhead per node, N-1 check, 3-year growth.
Going deeper: shapes at fleet scale
- Offer two or three standard site sizes (S/M/L) rather than custom designs per site.
- For single-node sites, focus on fast rebuild (lesson 12) and app-level redundancy across sites.
- Consider distribution footprint: lightweight distributions (k3s, RKE2, MicroShift, EKS Anywhere on small hardware) differ in memory use; measure before choosing.
Recap
- Single-node (cheap, no HA), 3-node compact (edge default, tolerates 1 failure), small HA (separate workers).
- etcd needs a majority: avoid 2-node control planes (or use a witness).
- Reserve resources for the system on small nodes.
- Size for N-1, upgrades, storage replication and growth.
This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.