Stateful patterns · Prasanta Kumar Behera

Decide when databases belong on Kubernetes, and run them well when they do: StatefulSets and their guarantees, operators that handle replication and failover, and the settings that protect data during node maintenance.

Should the database run on Kubernetes?

It can. Mature operators make it realistic. But it's a decision, not a default.

Favours Kubernetes	Favours managed or separate
On-prem or edge, no managed service available	Cloud with good managed offerings
Team has (or builds) database operations skills	Small team, no DBA expertise
Many small databases (dev/test, per-tenant)	A few critical, huge databases
Portability across environments matters	Need turnkey cross-region, PITR, compliance reports

Keeping a pet fish at school: possible, but someone must feed it, clean the tank and have a plan for holidays. If nobody knows how, it's kinder to let the aquarium shop look after it (a managed database). If you do keep it, buy the proper tank with an automatic feeder (an operator), not a bucket.

StatefulSets: identity and storage

apiVersion: v1
kind: Service
metadata:
  name: db
spec:
  clusterIP: None              # headless: stable DNS per pod (db-0.db, db-1.db)
  selector: { app: db }
  ports: [ { port: 5432 } ]
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: db
spec:
  serviceName: db
  replicas: 3
  selector:
    matchLabels: { app: db }
  template:
    metadata:
      labels: { app: db }
    spec:
      containers:
        - name: db
          image: registry.example.com/db/engine:1.0     # placeholder image
          volumeMounts:
            - { name: data, mountPath: /var/lib/data }
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: fast
        resources:
          requests:
            storage: 20Gi

Guarantees: pods are named db-0, db-1, db-2; each gets its own PVC (data-db-0…) that follows it if it's rescheduled; creation is in order, and updates go one pod at a time. PVCs are not deleted when you scale down (by default), which protects data.

What a StatefulSet does not know: which pod is the primary, how to set up replication, how to fail over, how to back up. That's the operator's job.

Operators: operational knowledge as code

Database	Operator examples
PostgreSQL	CloudNativePG, Crunchy PGO, Zalando postgres-operator
MySQL	Percona Operator, MySQL Operator (Oracle)
Kafka	Strimzi
Redis-compatible	Several community operators

A CloudNativePG cluster, for example, is one small resource:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: orders-db
  namespace: shop
spec:
  instances: 3
  storage:
    size: 20Gi
    storageClass: fast
  backup:
    barmanObjectStore:
      destinationPath: s3://db-backups/orders
      s3Credentials:
        accessKeyId: { name: backup-creds, key: ACCESS_KEY_ID }
        secretAccessKey: { name: backup-creds, key: SECRET_ACCESS_KEY }

The operator creates a primary and two streaming replicas, services for read-write and read-only traffic, automatic failover, and continuous WAL archiving for point-in-time recovery. (Field names follow the operator's documentation for your version; newer releases also offer plugin-based backup configuration.)

Protect data during maintenance

PodDisruptionBudget: maxUnavailable: 1, so drains never take down more than one replica.
Anti-affinity / topology spread across nodes and zones, so one failure never removes several replicas.
Storage replication vs app replication: if the database replicates itself, you often don't need 3 storage replicas underneath too (1 local replica + 3 database instances can be both faster and safe).
Resources: Guaranteed QoS (requests = limits) for database pods, to avoid eviction under pressure.

Try it: a StatefulSet, then an operator

Deploy a 3-replica StatefulSet (any simple image that writes its hostname to the volume) with a headless Service. Resolve db-0.db from another pod.
Delete db-1, and confirm it comes back with the same name and the same PVC (and its file still there).
Scale to 1, then back to 3: are data-db-1 and data-db-2 still there?
Install CloudNativePG, create a 3-instance cluster, find the primary, delete the primary pod, and watch failover (kubectl cnpg status with the kubectl plugin).
Add a PDB and drain the node running the primary. What happens?

Going deeper: stateful at scale

Test failover and restore regularly: time to promote a replica, and time to restore from WAL archives to a point in time.
Keep major version upgrades as rehearsed procedures; operators help, but data migrations still need testing.
Watch replication lag, connection counts, disk growth and backup age, not just pod health.
For many tenant databases, one operator + templates (golden path) gives self-service with guard-rails (see Kubernetes Administration, lesson 30).

Recap

Running databases on Kubernetes is a decision: skills, scale and managed alternatives matter.
StatefulSets give stable names, per-pod PVCs and ordered rollouts, but no database smarts.
Operators add replication, failover, backups and upgrades.
Protect data with PDBs, spreading, Guaranteed QoS and tested restores.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.