Lesson 32 of 32 · Level 5 — Architect
Capstone: platform design review
Bring all five levels together: design a Kubernetes platform for a realistic (fictional) company, write it up as a short design document, and defend it against failure modes, cost and operability questions.
The brief
Northwind Logistics (fictional) is moving its applications to Kubernetes. You're the platform architect. Here's what you know:
- Teams: about 30 development teams, around 120 services, mostly stateless HTTP APIs and queue workers; a handful of PostgreSQL databases.
- Environments: dev, staging and production.
- Regions: two data centres (or cloud regions) today; production must survive losing one.
- Edge: 40 warehouses, each needing a small local cluster for scanners and printers that must work when the WAN link is down.
- Compliance: payment services must be isolated from everything else.
- Targets: customer-facing APIs 99.9% available; payments RPO 5 minutes, RTO 30 minutes; everything else RPO 24 hours, RTO 4 hours.
- Team: a platform team of 6 engineers.
You've been asked to plan a new school campus: how many buildings, who shares which rooms, what happens in a fire drill, how you'll repaint every classroom each summer without closing the school, and what it will all cost. There's no single right answer, but there are well-argued answers.
Your task
Write a design document (4–8 pages) using the outline in the cheat sheet, plus at least three ADRs. Use everything from Levels 1–5:
| Level | Use it for |
|---|---|
| 1 · Foundations | Workload patterns, Services, Ingress, config, storage choices |
| 2 · Operator | Build method (images), bare-metal load balancing, upgrades, etcd backups, scheduling, RBAC, troubleshooting |
| 3 · Application delivery | App exposure path, namespaces and data, TLS, ingress/egress control |
| 4 · Production | HA, resources and quotas, autoscaling, fleet lifecycle, performance, DR |
| 5 · Architect | Fleet management, tenancy, platform APIs and golden paths, SLOs |
Failure modes to answer in the review
Your document should explain what happens, and what the users notice, when:
- One production control-plane node dies.
- A whole production region/data centre is lost.
- A warehouse loses its WAN link for 8 hours.
- A bad CNI upgrade breaks networking in the first wave.
- A team's runaway job floods the API server.
- The payments database volume is corrupted at 14:05.
- The management cluster is down for a day.
- A certificate authority expires unnoticed.
Scoring rubric
| Area | Strong answer shows |
|---|---|
| Requirements | Restated clearly; assumptions and unknowns listed |
| Topology | Cluster count and placement justified by blast radius, compliance, latency and ops cost |
| Tenancy | A model per tenant type; a concrete tenant kit; payments isolation explained |
| Lifecycle | Declarative provisioning, upgrade waves with gates, add-on management |
| Resilience | Failure domains, backups matched to RPO/RTO, drills scheduled |
| Observability | User-journey SLIs, SLOs, burn-rate alerting, fleet-wide visibility |
| Honesty | Trade-offs and risks named, with mitigations and owners |
One possible answer (sketch, read after you've tried)
Topology (ADR-001): per region, separate prod and non-prod clusters (4 clusters), plus a dedicated payments prod cluster in each region (2), plus 40 edge clusters, managed from one HA management cluster with Cluster API (or EKS Anywhere / Rancher at the edge). Why: blast radius, compliance scope, and independent upgrades; 46 clusters is only manageable because they're declarative.
Tenancy (ADR-002): namespace-per-team with the full tenant kit in shared clusters; payments gets its own clusters, separate credentials and stricter policy; virtual clusters for ephemeral dev/CI environments.
Edge (ADR-003): 3-node compact clusters (control plane + workload on the same nodes) for local HA; everything needed at the site is cached locally (registry mirror, images, config) so a WAN outage only pauses updates. GitOps pulls changes when the link returns.
Lifecycle: node images built in CI; upgrade waves non-prod → one edge batch → prod region A → prod region B → remaining edge batches, with automated pre- and post-checks and a synthetic probe gate.
Resilience: active-active stateless services across regions behind global load balancing; payments PostgreSQL with continuous archiving and a synchronous or near-synchronous standby for a 5-minute RPO; nightly Velero for everything else; quarterly timed DR drills.
SLOs: API availability and latency per cluster; probe deploy and reach SLIs; burn-rate paging; a fleet dashboard labelled by cluster and wave.
Risks: 6 engineers for 46 clusters requires strict standardisation (no snowflakes); edge hardware failures need a spares-and-reprovision process (zero-touch provisioning, see Edge Kubernetes & Zero-Touch Provisioning); management-cluster outage pauses changes, so it's HA and rebuildable from Git.
Going deeper: running a real design review
- Send the document before the meeting; spend the meeting on questions, not presenting.
- Invite people who will operate it and people who will use it, not just architects.
- Record decisions and follow-ups in the ADRs, and revisit them when the context changes (new regions, new compliance rules).
- A good review often ends with "yes, if…": approval with a short list of risks to address. That's success, not failure.
You've finished Kubernetes Administration
From your first kind create cluster to defending a 46-cluster platform design: that's the path from beginner to architect. Next, go deeper where your work needs it: Kubernetes Security & Hardening, Networking Deep Dive, Amazon EKS in Production, or Edge Kubernetes & Zero-Touch Provisioning.
This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.