Proving stability

Prove the chosen platform is stable, not just fast: soak tests under realistic load, failure injection and upgrades while traffic flows, and package the results as an evidence pack plus a one-page summary that executives can decide on.

Fast isn't stable

The PoC (lesson 02) showed the platform can meet ParcelPath's targets. Now prove it keeps meeting them: over days, during failures, and during upgrades.

A new bridge is tested with one truck, then with many trucks all day and night for days, with some lanes closed for repairs, to be sure it doesn't slowly crack. Then the engineers write a thick report for other engineers and a one-page summary for the mayor: "safe to open, here are the three things to watch."

Test plan

Test	Load	Duration	Pass criteria
Baseline	1,000 req/s	1 h	p99 < 300 ms, errors < 0.05%
Peak	3,000 req/s (NFR-03)	2 h	p99 < 300 ms, errors < 0.05%
Burst	4,500 req/s	10 min	No crash; recovery to SLO within 2 min
Soak	Daily-shaped curve up to 2,000 req/s	72 h	Flat memory, stable latency, no restarts
Node loss	Peak	During soak	Error budget spent < agreed amount; recovery < 5 min
etcd member loss	Peak	During soak	API available; no leader thrash
Upgrade	1,000 req/s	Full minor upgrade	Zero failed requests (or within budget)

A k6 load profile with stages:

import http from 'k6/http';
import { check } from 'k6';

export const options = {
  stages: [
    { duration: '10m', target: 200 },   // ramp up (virtual users)
    { duration: '2h',  target: 600 },   // peak
    { duration: '10m', target: 900 },   // burst
    { duration: '10m', target: 200 },   // recover
  ],
  thresholds: {
    http_req_duration: ['p(99)<300'],
    http_req_failed: ['rate<0.0005'],
  },
};

export default function () {
  const res = http.get(`${__ENV.BASE_URL}/track/PP${Math.floor(Math.random() * 1e6)}`);
  check(res, { 'status 200': (r) => r.status === 200 });
}

(Virtual users map to request rate through think time and response time; calibrate the VU counts to reach your target req/s, or use k6's arrival-rate executors to set the rate directly.)

What to watch during the soak

Memory of every component (including CNI, CSI, ingress, monitoring agents): flat, not creeping.
etcd: DB size, leader changes, fsync and commit latencies.
Disk usage: logs, container images, etcd, PV growth.
Latency percentiles hour by hour: any drift?
Anything expiring: tokens, certificates, leases.

The evidence pack

Method: environment (hardware, versions, config as code commit), tools and load profiles.
Results: per test, pass/fail against criteria, with graphs and raw data locations.
Issues found and how they were fixed (this builds confidence, not doubt).
Residual risks in a risk register (risk, likelihood, impact, mitigation, owner).

Recommendation: adopt Option C (distribution X + CNI Y + storage Z) for ParcelPath's platform. Confidence: high for NFR-01..03 and 07 (tested at peak and during upgrade); medium for NFR-04 (DR drill scheduled in design phase). Top risks: (1) holiday peak > 1.5× forecast; (2) team skills for storage operations; (3) single data-centre power feed. Cost: N servers + support subscription; within budget. Decision needed: approve design phase and hardware order by .

Try it: a stability mini-campaign

Deploy a sample app with a database on your lab cluster and write the k6 script above for it.
Run a 4–8 hour soak (as long as your lab allows) and graph memory for every pod in kube-system and your app.
During the soak, drain a node and record errors and recovery time.
Run a Kubernetes patch or minor upgrade on the lab cluster under load; count failed requests.
Write the one-pager for your results, including two honest risks.

Going deeper: credible evidence

Keep all raw data (k6 JSON, Prometheus snapshots) linked from the pack; reviewers will ask.
Repeat key tests after fixes; a single run proves little.
Separate what was measured from what was extrapolated (e.g. PoC footprint → production size).

Recap

Stability = soak, failure injection and upgrades under realistic load, with criteria set in advance.
Watch for slow problems: leaks, growth, drift, expiries.
Deliver an evidence pack (method, results, issues, risks) and an executive one-pager (recommendation, confidence, risks, cost, decision).

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.

Fast isn't stable

Test plan

What to watch during the soak

The evidence pack

The one-pager

Recap