Lesson 09 of 13 · Architecture
Storage & the stateful tier
Design ParcelPath's stateful tier: classify data, decide which databases run in Kubernetes (with operators) and which stay outside, choose storage per class, set replication and backups to meet an RPO/RTO table, and record it as a storage ADR.
Classify the data
| Data | Store | RPO | RTO | Notes |
|---|---|---|---|---|
| Orders | PostgreSQL | 5 min | 60 min | Money-relevant, can't be recreated |
| Tracking events | Kafka | 15 min | 60 min | Partly replayable from partners |
| Tracking read model | PostgreSQL (read replicas) | 60 min | 60 min | Rebuildable from events |
| Sessions / cache | Redis | none | 15 min | Rebuild on restart |
| Documents (labels, PDFs) | Object storage | 1 h | 4 h | Versioned bucket |
Not everything in a house needs the same protection. Passports go in a fireproof safe with a copy at grandma's house (orders: replicated + off-site backups). Photos get backed up weekly (tracking read model). The shopping list on the fridge doesn't need a backup at all (cache). Protecting everything like passports is too expensive; protecting passports like shopping lists is a disaster.
In the cluster or outside?
| In-cluster with operators | External | |
|---|---|---|
| Examples | CloudNativePG (PostgreSQL), Strimzi (Kafka), Redis operators | DBA-run clusters, database appliances |
| Pros | Same GitOps/tooling, self-service, fast provisioning | Existing expertise, independent lifecycle |
| Cons | Team needs database-on-Kubernetes skills; upgrades touch data | Separate automation, slower provisioning |
ParcelPath decision (ADR-006): Kafka via Strimzi and PostgreSQL via CloudNativePG in the stateful pool on local NVMe, with application-level replication; Redis in-cluster without persistence; object storage external (the DC's S3-compatible service). See Kubernetes Storage & Data Protection, lessons 06–07.
Storage per class
- Local NVMe PVs (local-path/TopoLVM/static local PVs) for PostgreSQL and Kafka: lowest latency; the database replicates across nodes/racks.
- Replicated block storage (Ceph RBD or Longhorn) for apps that can't replicate themselves and need to move between nodes.
- Spread replicas across racks (topology spread / anti-affinity), so one rack loss keeps quorum for Kafka and a replica for PostgreSQL.
Replication across sites and backups
- PostgreSQL: primary in DC-A with a synchronous standby in DC-A (another rack) for local failover; an asynchronous replica cluster in DC-B for DR. Monitor replication lag; alert well below the 5-minute RPO.
- Kafka: replication factor 3 across racks in DC-A; MirrorMaker 2 (or similar) to DC-B for DR, with lag monitoring.
- Backups: continuous WAL archiving + daily base backups for PostgreSQL to object storage in DC-B (PITR); tested restores monthly.
- Synchronous cross-site replication is only realistic at metro distances (low single-digit ms round trip); ParcelPath's DCs are ~300 km apart, so cross-site is async.
The storage ADR
ADR-006: Stateful tier for production
Status: Accepted
Context: RPO/RTO table above; DCs ~300 km apart; team trained on CloudNativePG and Strimzi in the PoC.
Decision: Operator-managed PostgreSQL and Kafka on local NVMe in the stateful pool; sync standby in-DC,
async to DC-B; PITR backups to DC-B object storage; Redis without persistence.
Consequences: + low latency, GitOps-managed, meets RPO via async lag < 1 min (measured).
− on DC-A loss, up to replication lag of orders may be lost (within RPO);
database operations skills required in the platform team (training plan).
Alternatives: Ceph RBD for databases (extra latency, doubled replication); external DBA-run
PostgreSQL (slower provisioning, separate tooling).
Try it: a stateful tier in the lab
- Install CloudNativePG and create a 3-instance PostgreSQL cluster with anti-affinity across nodes; kill the primary and time failover.
- Configure backups to an S3-compatible bucket and perform a point-in-time restore into a new cluster.
- Install Strimzi and create a 3-broker Kafka cluster with replication factor 3; stop one broker and keep producing.
- Measure write latency for PostgreSQL on local-path storage vs replicated storage (e.g. Longhorn) with pgbench.
- Fill in the RPO/RTO table for your own lab services.
Going deeper: stateful design
- Test failover and restore as part of acceptance (lesson 12), not just configuration review.
- Plan database upgrades (major versions) as projects with rehearsals; operators help but don't remove risk.
- Keep an eye on Kubernetes node maintenance: PodDisruptionBudgets and operator-aware draining protect quorum.
Recap
- Classify data with an RPO/RTO table; protect each class appropriately.
- Decide in-cluster (operators) vs external per store, based on skills and requirements.
- Local NVMe + app-level replication for databases; replicated block storage for apps that can't replicate.
- Sync replication within a metro/DC, async across distance; PITR backups off-site, restores tested.
- Record it as a storage ADR.
This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.