Architecture · wrap-up
Cheat sheet & self-check
Every command from this section on one page.
Control plane options
3 stacked control-plane nodes (etcd on the same nodes) | Default for most clusters |
5 members | Survives 2 failures; more write latency and cost |
External etcd cluster | Isolates etcd; more machines to run |
Spread members across racks/rooms | One rack loss keeps quorum |
ADR template
Title · Status · Context · Decision · Consequences · Alternatives considered | One decision per record |
docs/adr/ADR-00N-<slug>.md in the platform repo | Versioned with the code |
Sizing ranges
/16 pod CIDR with /24 per node → 256 nodes × 256 addresses | kubeadm/controller-manager: --cluster-cidr, --node-cidr-mask-size |
Service CIDR /16 → 65,536 ClusterIPs | --service-cluster-ip-range (API server) |
Calico default IPAM block /26; Cilium cluster-pool default /24 per node | CNI-specific allocation |
Ranges must not overlap: clusters, DCs, VPN users, partners | Future routing and peering |
Routing
Overlay (VXLAN/Geneve) | Works anywhere; ~50 bytes overhead; pods not routable outside |
Native routing + BGP to ToR | Pods routable; no encapsulation; needs network team |
Layers
L4: MetalLB (BGP/L2), kube-vip, or hardware/virtual LB | Stable IPs for ingress and the API server |
L7: ingress controller or Gateway API implementation | Host/path routing, TLS, retries, rate limits |
externalTrafficPolicy: Local | Keep client source IP; only nodes with local endpoints receive traffic |
PROXY protocol | Pass client IP through an L4 LB that terminates TCP |
TLS options
Terminate at external LB/WAF | Central certs; LB sees plaintext |
Terminate at ingress (cert-manager) | Kubernetes-native certs; common default |
Re-encrypt to pods / mesh mTLS | Encrypted end to end inside the DC |
Passthrough (SNI routing) | App holds the key; no L7 features at ingress |
Edge layers
DNS/GSLB → CDN → WAF + DDoS → L4 VIP → ingress | Outside-in order |
Rate limits: per client/API key at the gateway or ingress | Protect backends from floods and abuse |
API gateway: auth, quotas, versioning for partners | Partner-facing APIs |
Isolation layers (namespace model)
RBAC + NetworkPolicy (default deny) + ResourceQuota/LimitRange | The minimum set |
Pod Security Admission (restricted) + policy engine | Workload hardening |
Dedicated node pools / taints for sensitive tenants | Kernel and noisy-neighbour isolation |
Choices per data class
Operator-managed DB on local NVMe (app-level replication) | Fast; the database replicates, not the storage |
Replicated block storage (Ceph RBD, Longhorn) | Storage-level replication; easier moves |
External database service / DBA-run cluster | Existing expertise, outside the cluster lifecycle |
Replication across sites
Synchronous | RPO ≈ 0; needs low round-trip latency (metro distances); writes slower |
Asynchronous | RPO = replication lag; any distance; possible data loss on failover |
Backups (PITR) off-site | Protect against corruption and mistakes, not just hardware loss |
