Lesson 04 of 13 · Requirements & Proof
Sizing & capacity math
Size ParcelPath's clusters with math you can defend: start from workload requests at peak, convert node capacity to usable allocatable, apply target utilisation, add headroom to survive a rack failure, plan growth, and check pod-density and control-plane limits.
Start from demand
ParcelPath's services (production, at holiday peak, after right-sizing requests from PoC measurements):
| Tier | Replicas × request (CPU, memory) | CPU | Memory |
|---|---|---|---|
| Tracking API | 30 × (1 vCPU, 1 GiB) | 30 | 30 GiB |
| Other 38 services | ≈ 150 pods × (0.4 vCPU, 0.75 GiB) | 60 | 112.5 GiB |
| Kafka (in-cluster) | 3 × (4 vCPU, 16 GiB) | 12 | 48 GiB |
| Platform add-ons (ingress, monitoring, logging) | — | 18 | 60 GiB |
| Total demand | 120 vCPU | ≈ 250 GiB |
Add growth: NFR-05 wants 2× traffic within 18 months. Stateless tiers scale roughly with traffic; platform add-ons less so. Say the 18-month demand is ≈ 210 vCPU and ≈ 420 GiB. (State such assumptions explicitly in the sizing sheet.)
Planning seats for a school trip: count the children (demand), check how many seats each bus really has after the driver and teacher sit down (allocatable), don't plan to fill every seat because siblings want to sit together (utilisation), bring one extra bus in case one breaks down (failure headroom), and remember next year's class is bigger (growth).
Node allocatable
Planned worker: 32 cores, 256 GiB RAM.
| Item | CPU | Memory |
|---|---|---|
| Capacity | 32 | 256 GiB |
| − system-reserved + kube-reserved | 1 | 6 GiB |
| − eviction threshold | — | 1 GiB |
| − DaemonSets (CNI, CSI, log/metrics agents) | 1 | 3 GiB |
| Available for workloads | 30 | 246 GiB |
| × 70% target utilisation | 21 | ≈ 172 GiB |
Nodes needed
| CPU | Memory | |
|---|---|---|
| Today: demand ÷ usable | 120 ÷ 21 = 5.7 → 6 | 250 ÷ 172 = 1.5 → 2 |
| 18 months | 210 ÷ 21 = 10 → 10 | 420 ÷ 172 = 2.4 → 3 |
CPU is the binding constraint, so memory-heavy nodes are over-provisioned for this workload. That's an input to the hardware choice (maybe 128 GiB nodes are enough, saving cost).
Failure-domain headroom
ParcelPath's production DC has 3 racks. To survive losing one rack, the remaining two must carry everything:
- Today: 6 × 3 ÷ 2 = 9 workers (3 per rack).
- 18 months: 10 × 3 ÷ 2 = 15 workers (5 per rack).
Add one more node per pool if rolling node replacement (upgrades) must happen while also tolerating a failure, or accept that upgrades pause when a rack is down.
Other limits
- Pod density: 15 nodes × 110 pods = 1,650 max; planned pods ≈ 250 + DaemonSets. Fine.
- Pod CIDR: a /24 per node gives 256 addresses (≥ 110 pods); the cluster CIDR must hold enough /24s for maximum nodes + growth (lesson 06).
- Control plane: 3 dedicated control-plane nodes (e.g. 8 cores, 32 GiB, fast NVMe for etcd) are plenty at this scale; the upstream-tested limits (5,000 nodes, 150,000 pods) are far away. Watch object counts and API request rates rather than node count.
The sizing sheet
Deliver a spreadsheet (or table) with: assumptions (traffic, growth, requests per service), node spec, allocatable calculation, utilisation target, failure-domain rule, results today and at 18 months, and triggers for re-sizing (e.g. "re-run when peak req/s exceeds 4,000 or CPU requests exceed 75% of usable").
Try it: size your own
- From a real or lab cluster, export requests per namespace:
kubectl get pods -A -o json | jq(or kube-state-metrics'kube_pod_container_resource_requests). - Compute node allocatable from
kubectl describe node(Capacity vs Allocatable) and subtract DaemonSet requests. - Apply a 70% target and your failure-domain rule; calculate node counts for CPU and memory.
- Recalculate with 128 GiB nodes and compare cost.
- Write three re-sizing triggers for your sheet.
Going deeper: capacity as a practice
- Right-size requests continuously (VPA recommendations, usage percentiles); over-requested pods waste real money.
- Separate node pools by shape (CPU-heavy, memory-heavy, stateful, ingress) when workloads differ a lot.
- Keep capacity dashboards: requests vs allocatable per pool, and alert well before thresholds.
- Plan hardware lead times: at edge or on-prem, capacity takes months to arrive.
Recap
- Demand from peak requests (+ growth, stated as assumptions).
- Allocatable = capacity − reserved − eviction − DaemonSets; then apply a utilisation target.
- Node count = max(CPU, memory); add failure-domain headroom (× R ÷ (R − 1)).
- Check pod density, IPAM and control-plane limits; publish a sizing sheet with re-sizing triggers.
This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.