Prasanta Kumar Behera
Kubernetes Storage & Data Protection›07 · Production patterns
Learning Hub / Kubernetes & Platform / Kubernetes Storage & Data Protection

Lesson 07 of 7 · Modules

Production patterns

Run storage as a reliable platform service: tiered storage classes, capacity metrics and alerts, safe reclaim policies, upgrading storage systems and nodes without data loss, and the runbooks that go with them.

Advanced
Key wordscapacity planningvolume metricsalertsstorage classesreclaim policyupgradesnode maintenancerunbooks

Storage as a product

Application teams should get storage the same way they get compute: pick a class, create a claim, and trust that it's fast enough, backed up, monitored and won't vanish. That takes a small set of deliberate platform decisions.

A school's lockers need rules too: different sizes for different needs, a caretaker who notices when lockers are nearly full, a rule that lockers aren't emptied just because a pupil changed classes, and a plan for repainting the corridor without losing anyone's things.

1. A small menu of storage classes

Class Backing For Reclaim
fast Local NVMe / high-IOPS volumes Databases with their own replication Retain
standard (default) Replicated block (Longhorn/Ceph/cloud) General stateful apps Delete (with backups) or Retain
shared CephFS / NFS / EFS RWX shared files Retain
scratch Cheap, non-replicated Caches, CI workspaces Delete

Keep the menu short, name classes by purpose, and document each class's performance, durability and backup expectations.

2. Capacity: measure, predict, alert

The kubelet exports per-PVC usage metrics. Alert on time to full, not just percentages:

# PVC predicted to fill within 24 hours (based on the last 6 hours)
predict_linear(kubelet_volume_stats_available_bytes[6h], 24 * 3600) < 0

Also watch inodes, the storage system's own pool capacity (Ceph/Longhorn), and growth trends per team for capacity planning. Enable allowVolumeExpansion so the fix is a patch, not a migration.

3. Protect data from deletion

  • Retain for important classes; Released PVs then wait for a human decision instead of vanishing.
  • Backups (lesson 04) for everything that matters, tested.
  • Be careful with namespace deletion: it deletes every PVC inside. Consider admission policies that block deleting namespaces labelled as stateful, or require a confirmation label.

4. Upgrades and node maintenance

Storage system upgrades (CSI drivers, Longhorn, Rook/Ceph):

  • Read the vendor's upgrade order and compatibility matrix (Kubernetes version ↔ driver version).
  • Upgrade in a staging cluster with realistic volumes first.
  • Upgrade one component at a time, waiting for full health between steps.

Node maintenance:

  1. Check storage health: no degraded volumes, Ceph HEALTH_OK.
  2. Drain with PDBs in place (lesson 06); let the storage system's drain policy handle replica movement.
  3. Wait for health to return before the next node.

5. Runbooks

Keep short, tested runbooks for:

Situation First steps
PVC stuck Pending describe pvc; provisioner logs; quotas and pool capacity
Pod stuck on Multi-Attach after node failure Confirm the old node is really down; check VolumeAttachments; follow the driver's force-detach guidance
Volume degraded (Longhorn) / HEALTH_WARN (Ceph) Identify failed disk/node; confirm rebuild progress; avoid further maintenance until healthy
PVC almost full Expand the PVC; find the growth source; fix retention
Accidental deletion Stop further changes; restore from backup into a new namespace; verify; switch over

Try it: build the storage dashboard

With kube-prometheus-stack (see Scaling Prometheus) or any Prometheus that scrapes the kubelet:

  1. Graph kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes per PVC.
  2. Write the predict_linear alert and trigger it with a pod that fills a small PVC steadily (dd if=/dev/zero of=/data/f bs=1M count=… in a loop).
  3. Change a StorageClass for new volumes to Retain, delete a test PVC, and find the Released PV. Recover its data by binding a new PVC to it (clear claimRef first).
  4. Write a one-page runbook for "PVC almost full" for your team.

Going deeper: storage platform maturity

  • Offer self-service with guard-rails: quotas on requests.storage per namespace, and allowed storage classes per tenant.
  • Report cost per namespace for storage, just like CPU and memory.
  • Test disaster recovery for the storage system itself (losing a Ceph pool, restoring a Longhorn backup target) at least once a year.
  • Keep stateful workloads on fewer, well-understood storage backends. Every additional system is another thing to upgrade, monitor and restore.

Recap

  • A short menu of classes named by purpose, with documented performance, durability and reclaim behaviour.
  • Alert on predicted time to full (and inodes); allow expansion.
  • Retain + backups + deletion guard-rails for important data.
  • Upgrade and drain one step at a time, healthy between steps; keep runbooks for the common failures.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.