Control plane & node topology · Prasanta Kumar Behera

Decide the shape of ParcelPath's clusters: how many clusters and why, control-plane size and etcd layout, spreading across racks, node pools for different workloads, and recording each choice as an Architecture Decision Record.

How many clusters?

For ParcelPath:

Cluster	Where	Purpose
`prod-a`	DC-A (primary)	Production
`prod-b`	DC-B (secondary)	Production standby / failover (lesson 12)
`nonprod`	DC-B	Dev + staging (namespaces), load tests
`mgmt`	DC-B (small)	GitOps, cluster lifecycle, observability backends

Reasons to split: blast radius (a bad change or upgrade hits one cluster), different change rates (prod vs dev), and failure domains (sites). Reasons not to split further: every cluster costs operations effort. (See Kubernetes Administration — Level by Level, lesson 28 for fleet patterns.)

A town doesn't put the hospital, school and fire station in one building: if that building has a problem, everything stops. But it also doesn't build a separate building for every classroom. You split where a problem in one place mustn't spread, and keep things together where splitting only adds work.

Control plane and etcd

3 control-plane nodes, stacked etcd, one per rack in each production DC. Survives one node or one rack.
Dedicated control-plane nodes (tainted; no workloads) with fast NVMe for etcd (lesson 02's fsync test).
API access through a VIP or load balancer (lesson 07).
5 members only if there are ≥ 5 independent failure domains or a requirement to survive two simultaneous failures; external etcd only if you have a strong reason (isolation, very large clusters) and the skills.

Node pools

Pool	Nodes (prod-a, 18 months)	Taints / labels	For
`system`	3 (1 per rack)	`node-role=system:NoSchedule`	Ingress controllers, monitoring agents' heavier parts, CoreDNS
`general`	12 (4 per rack)	—	Stateless services
`stateful`	3 (1 per rack, local NVMe)	`workload=stateful:NoSchedule`	Kafka, (optionally) databases

(The 15 workers from lesson 04 become the system and general pools. The stateful pool adds local-NVMe nodes for Kafka, so Kafka's share of lesson 04's demand moves out of the general pool; re-run the sizing per pool.) Use topology spread constraints with topology.kubernetes.io/zone (set to the rack) so replicas spread across racks.

ADR: control-plane topology

ADR-001: Control-plane topology for production clusters
Status: Accepted (2026-09-27)

Context
  ParcelPath runs production in DC-A with 3 racks (independent PDUs, dual ToR per rack).
  NFR-01 requires 99.95%; NFR-07 requires upgrades without customer-visible downtime.
  The team has experience operating stacked etcd; no dedicated etcd expertise.

Decision
  3 dedicated control-plane nodes per production cluster, stacked etcd, one per rack,
  NVMe for etcd (fsync p99 < 10 ms verified), API behind a VIP.

Consequences
  + Survives loss of one node or one rack; simple to operate; matches distribution defaults.
  − Losing two racks at once stops the control plane (workloads keep running, no changes possible).
  − Control-plane and etcd share nodes; etcd performance must be monitored.

Alternatives considered
  5 stacked members: no 5th failure domain in DC-A; extra cost and write latency.
  External 3-node etcd: more machines and skills needed for little gain at this scale.

Try it: ParcelPath topology

Draw DC-A with 3 racks, placing control-plane nodes and the three node pools.
Label nodes in a kind cluster with topology.kubernetes.io/zone=rack-a/b/c and deploy a 6-replica app with a topology spread constraint; verify 2 per "rack".
Write ADR-002 "Number of clusters" with at least two alternatives.
List what happens if DC-A loses two racks: control plane, workloads, data.
Decide whether databases run in the stateful pool or outside Kubernetes (you'll revisit in lesson 09).

Going deeper: topology trade-offs

Stretched clusters across two sites look attractive but etcd needs low latency and a quorum-breaking third site; usually prefer one cluster per site plus application-level replication (lesson 12).
Keep control-plane nodes boring: no extra agents beyond monitoring; changes via the distribution's lifecycle tooling only.
Revisit ADRs when context changes (new DC, new team skills, new distribution features); mark old ones Superseded.

Recap

Split clusters for blast radius, change rate and failure domains; don't over-split.
3 stacked control-plane nodes, one per rack, fast etcd disks, API behind a VIP.
Node pools by workload shape with taints/labels; spread replicas across racks.
Record each decision as an ADR: context, decision, consequences, alternatives.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.