Lesson 02 of 13 · Requirements & Proof
PoC from scratch
Run a proof of concept that decides something: shortlist platform options against weighted criteria, define success criteria up front, build a small but representative environment, and capture a measured baseline for conformance, control plane, network, storage and application performance.
What the PoC must decide
For ParcelPath the open questions are: which Kubernetes distribution on their bare metal, which CNI, and which storage approach for stateful services. A PoC answers them with evidence, in a fixed time.
Before buying a family car, you test-drive two or three. But you decide beforehand what matters, like "five seats, fits the pram, under this price", so you're not dazzled by a shiny dashboard. The PoC is the test drive with a checklist written in advance.
Shortlist with weighted criteria
| Criterion | Weight | Option A (kubeadm + tooling) | Option B (RKE2 + Rancher) | Option C (EKS Anywhere) |
|---|---|---|---|---|
| Lifecycle automation (upgrades, node replacement) | 25% | 2 | 4 | 4 |
| Security defaults / compliance | 20% | 3 | 5 | 4 |
| Team skills & learning curve | 15% | 4 | 3 | 3 |
| Support model | 15% | 2 | 4 | 4 |
| Bare-metal provisioning integration | 15% | 2 | 4 | 5 |
| Cost | 10% | 5 | 3 | 4 |
| Weighted total | 2.8 | 3.95 | 4.0 |
(Scores are illustrative for the exercise; yours must come from evidence. Weighted total = Σ weight × score, e.g. Option B: 0.25×4 + 0.20×5 + 0.15×3 + 0.15×4 + 0.15×4 + 0.10×3 = 3.95.)
Agree the weights with stakeholders before scoring; that's where priorities get argued, not after.
Success criteria (examples)
| ID | Criterion | Pass if |
|---|---|---|
| POC-1 | Conformance | Sonobuoy certified-conformance passes |
| POC-2 | Control plane | API p99 latency < 1 s under kube-burner churn of N objects |
| POC-3 | etcd disks | fsync p99 < 10 ms (fio etcd profile) |
| POC-4 | Network | Pod-to-pod ≥ 80% of NIC line rate (iperf3, with overlay) |
| POC-5 | Storage | Volume p99 write latency < X ms at Y IOPS (fio) |
| POC-6 | App | Tracking API p99 < 300 ms at 1,000 req/s on the PoC footprint (k6) |
| POC-7 | Operations | Minor upgrade with a running app, no failed requests |
| POC-8 | Recovery | Node loss: workloads rescheduled within 5 min |
Build it small but representative
- Same hardware class, NICs, disks and network design as production (at least one rack's worth), not laptops.
- The real CNI, storage and ingress candidates; a representative slice of ParcelPath services with realistic data sizes.
- Everything as code from day one, so the winning option's PoC becomes the seed of production automation.
Measure the baseline
$ sonobuoy run --mode=certified-conformance --wait && sonobuoy retrieve . && sonobuoy results *.tar.gz
$ fio --name=etcd --rw=write --ioengine=sync --fdatasync=1 --bs=2300 --size=22m --directory=/var/lib/etcd-test
$ iperf3 -s # in one pod; iperf3 -c <server-pod-ip> -P 4 -t 30 # in another, on a different node
$ k6 run --vus 200 --duration 10m load.js
Record hardware, versions and settings with every result: a number without context is useless six months later.
Try it: a mini PoC
- Write five success criteria for your own lab (conformance, etcd fsync, pod network, storage, one app).
- Run Sonobuoy in quick mode (
--mode quick) on a kind cluster, then certified-conformance on VMs if you have them. - Run the fio etcd profile on two different disks (e.g. SSD vs HDD, or local vs network) and compare p99 fdatasync latency.
- Run iperf3 between pods on different nodes with two different CNIs (or overlay vs native routing).
- Fill in a weighted scoring matrix for two options with your evidence.
Going deeper: PoC traps
- Vendor-run demos aren't evidence; run the tests yourselves, on your hardware.
- Scope creep turns a PoC into an unfunded pilot; time-box it and park new ideas.
- Test the boring parts: upgrades, certificate rotation, backup/restore, node replacement. They decide day-2 cost.
Recap
- A PoC answers specific questions with evidence in a fixed time.
- Weighted scoring: weights agreed first, scores from measurements.
- Success criteria written up front: conformance, control plane, etcd disks, network, storage, app, upgrades, recovery.
- Build representative and as code; record context with every number.
This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.