Lesson 07 of 14 · Modules
Simulated testing with AI
Use AI to widen what you test before QA: generate failure-scenario lists, upgrade-path matrices, chaos experiments and load scripts from your architecture and change, then execute them with real tools in disposable environments, so AI expands coverage while tools provide the evidence.
Why simulate before QA?
Human test plans cover the scenarios people think of. AI is good at breadth: listing many ways a change could fail, combinations of versions, and edge cases. Paired with disposable environments and real tools, that becomes a pre-QA safety net.
Before a real fire drill, the school asks a creative helper to imagine lots of "what ifs": the main door is blocked, it's raining, a class is on a trip. Then the teachers actually practise a few of them. The imagination is from the helper; the proof comes from really doing the drill.
Step 1: generate scenarios from the change
Context: We're upgrading ingress-nginx from chart 4.10 to 4.12 on 3 clusters (K8s 1.30),
behind MetalLB BGP, with cert-manager-issued certificates and ~200 Ingress objects,
some using the configuration-snippet annotation.
Task: List failure scenarios this upgrade could cause. For each: trigger, symptom,
how to detect it in a test cluster, and how to test it automatically. Mark items you're
unsure about as "verify in release notes".
Then filter: remove irrelevant items, check the "verify" items against the actual release notes and chart changelog.
Step 2: build a test matrix
| Scenario | Environment | Test | Pass criteria |
|---|---|---|---|
| Snippet annotations disabled by default in new version | kind + our Ingress samples | Apply Ingresses, curl paths | All routes 200; no admission rejections |
| Admission webhook blocks invalid Ingress during rollout | kind | Apply an invalid Ingress | Clear rejection message; valid ones unaffected |
| Rolling upgrade drops connections | kind + k6 at 200 RPS | Upgrade during load | Error rate < 0.1% |
| Certificate secret references | kind + cert-manager | Issue certs, upgrade | TLS handshakes succeed |
(Illustrative; your scenarios come from your change and the real release notes.)
Step 3: let AI draft the assets, then validate them
$ kubectl apply --dry-run=server -f generated/chaos-podkill.yaml # catches invented fields
$ kubeconform -strict generated/*.yaml
$ k6 inspect generated/load.js
Typical generated assets: kind cluster configs, Chaos Mesh experiments, k6 scripts, a shell script that applies samples and asserts responses. Review each; run a server-side dry run against a cluster with the CRDs installed.
Step 4: run in disposable environments
- kind/k3d clusters created per test run in CI; destroyed after.
- Same chart versions and representative config (sanitised copies of real Ingresses, values files).
- Assertions produce pass/fail; attach logs and metrics to the PR.
Upgrade-path matrices
For platform upgrades, ask for a matrix of component × current → target version × order × known constraints, then verify each constraint against official compatibility tables (Kubernetes version skew policy, CNI/CSI/operator support matrices). AI enumerates; documentation and test runs decide.
Try it: AI-widened tests for one change
- Pick a real or sample change (e.g. a Helm chart minor upgrade) and generate a scenario list with the prompt pattern above.
- Check every "verify" item against the release notes; delete wrong or irrelevant scenarios and note how many there were.
- Ask for a kind config, a k6 script and one Chaos Mesh experiment; validate them with dry runs and kubeconform.
- Run them in a kind cluster in CI (GitHub Actions has kind actions) with pass/fail assertions.
- Record which scenarios found real issues, and add the useful ones to your permanent test suite.
Going deeper: making it routine
- Keep a scenario library per component, grown from incidents and AI brainstorming, reviewed by humans.
- Attach simulated-test evidence to change requests for risky upgrades.
- Watch cost and flakiness: disposable clusters in CI need time budgets and retries for infrastructure hiccups (not for real failures).
Recap
- AI adds breadth: failure scenarios, upgrade matrices, test assets.
- Filter and verify against release notes and docs; validate manifests with dry runs.
- Execute in disposable environments with pass/fail assertions: tools produce the evidence.
- Keep what finds real issues in a permanent scenario library.
This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.