Resilience & failure modes · Prasanta Kumar Behera

Find weaknesses before production does: run an FMEA over ParcelPath's design (failure mode, effect, severity, occurrence, detection, risk priority), mitigate the top risks, and prove the mitigations with controlled chaos experiments and game days.

FMEA: think through failures systematically

Failure Mode and Effects Analysis walks through each component and asks: how can it fail, what happens, how bad, how likely, and would we notice?

Before a school trip, the teacher lists what could go wrong: the bus breaks down, a child gets lost, it rains. For each, she rates how bad, how likely, and how quickly she'd notice, then fixes the worst first (a spare bus number, name tags, raincoats). Then she practises: a pretend "lost child" drill to check the plan really works (chaos experiment).

ParcelPath FMEA (excerpt)

Component	Failure mode	Effect	S	O	D	RPN	Mitigation
Rack power (DC-A)	One rack loses power	1/3 of nodes, 1 etcd member, 1 DB replica down	7	3	2	42	Spread by rack (done); verify N-1 capacity
etcd disk	fsync latency spikes	API slow, leader elections	7	4	5	140	NVMe, fsync alert, dedicated disks
WAF	Bad rule blocks valid clients	Users fail; internal metrics green	8	4	8	256	Synthetic journeys, block-rate alerts, staged rule rollout
PostgreSQL primary	Node crash	Writes fail until failover	8	3	2	48	CloudNativePG failover (< 60 s measured)
Cert-manager / CA	Certificate not renewed	TLS failures at ingress	8	2	6	96	Expiry alerts at 14/7 days
DNS (CoreDNS)	Overloaded / misconfigured	Widespread timeouts	8	3	5	120	NodeLocal DNSCache, CoreDNS SLO
GitOps	Bad commit to platform config	All clusters affected	9	3	4	108	Staged rollout (nonprod → prod-b → prod-a), CI validation
Kafka broker	Disk full	Producers blocked	7	3	3	63	Retention limits, disk alerts at 70/80%

(Scores are illustrative. Agree the scale with the team, score together, and re-score after each mitigation.)

The top RPN here, the WAF row, isn't a Kubernetes failure at all: FMEA helps find risks outside your favourite component.

Prove it with chaos experiments

For each important mitigation, design an experiment:

Field	Example
Hypothesis	While one rack's worker nodes are unavailable, tracking API success stays ≥ 99.95% and p99 < 300 ms
Method	Cordon + drain (or power off in a lab) all nodes labelled `zone=rack-b` during a 1,000 req/s load test
Blast radius	`prod-b` (standby) first, business hours, team on call
Abort if	Success < 99% for 2 minutes, or data-plane alerts fire
Observe	SLO dashboards, reschedule times, DB failover events

Kubernetes-native tools such as Chaos Mesh and LitmusChaos inject pod kills, network latency/loss, IO faults and more:

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: orders-db-latency
  namespace: chaos-testing
spec:
  action: delay
  mode: all
  selector:
    namespaces: [ orders ]
    labelSelectors: { cnpg.io/cluster: orders-db }
  delay:
    latency: "100ms"
    jitter: "20ms"
  duration: "10m"

Record results

Experiment	Hypothesis held?	Findings	Actions
Rack loss (prod-b)	Yes (99.97%)	Kafka rebalancing took 4 min	Tune rebalance settings
DB latency +100 ms	No (p99 480 ms)	Tracking API calls DB synchronously for a non-critical field	Cache field; async write
CoreDNS pod kill	Yes	—	—

Failed hypotheses are successes of the process: you found the weakness before customers did.

Try it: FMEA and one experiment

Write ten FMEA rows for your lab or ParcelPath, score them, and sort by RPN.
Pick the top item and propose a mitigation; re-score.
Install Chaos Mesh (or LitmusChaos) in a lab cluster.
Write a steady-state hypothesis for a sample app under load, then run a PodChaos (kill) and a NetworkChaos (delay) experiment.
Record results in the table format above, including at least one action item.

Going deeper: resilience culture

Run game days regularly with the on-call team, including the humans: paging, communication, runbooks.
Automate safe experiments in CI/staging (continuous verification) once they're well understood.
Keep the FMEA alive: update it after every incident and architecture change.

Recap

FMEA: per component, failure mode → effect → S, O, D → RPN → mitigation → re-score.
Look beyond Kubernetes: edge, DNS, certificates, GitOps and people fail too.
Chaos experiments prove mitigations: hypothesis, controlled injection, blast radius, abort conditions.
Record results; failed hypotheses are valuable findings.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.