Lesson 04 of 7 · Modules
Snapshots & backups
Protect volume data: CSI VolumeSnapshots and restoring from them, Velero backups of namespaces with their volumes to object storage, and making backups application-consistent with hooks.
Three layers of protection
| Layer | Protects against | Speed | Lives |
|---|---|---|---|
| Replication (Longhorn, Ceph, cloud disks) | Disk or node failure | Instant | Same storage system |
| Snapshots | Mistakes: bad deploy, deleted files | Seconds | Usually the same storage system |
| Backups | Losing the storage system, the cluster or the site | Minutes to hours | Somewhere else (object storage, another region) |
Replication is writing in three notebooks at once, a snapshot is a quick photo of your notebook page, and a backup is a photocopy kept at your grandma's house. If the school burns down, only grandma's copy survives, so you need all three, for different disasters.
CSI VolumeSnapshots
Snapshots need the snapshot CRDs and controller (from the external-snapshotter project; many distributions and managed services include them) and a driver that supports them.
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
name: longhorn-snap
driver: driver.longhorn.io
deletionPolicy: Delete
---
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
name: data-before-upgrade
namespace: shop
spec:
volumeSnapshotClassName: longhorn-snap
source:
persistentVolumeClaimName: db-data
Restore by creating a new PVC from it:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: db-data-restored
namespace: shop
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: longhorn
resources:
requests:
storage: 10Gi
dataSource:
name: data-before-upgrade
kind: VolumeSnapshot
apiGroup: snapshot.storage.k8s.io
Velero: namespaces + volumes, off-cluster
Velero backs up Kubernetes objects and volume data to object storage. Volume data can come from:
- CSI snapshots, optionally moved to object storage (
--snapshot-move-data), so the backup survives the storage system. - File-system backup (Kopia): reads files from the pod's volumes. Works with any volume type, but is slower for large volumes.
$ velero backup create shop-2026-09-27 --include-namespaces shop --snapshot-move-data
$ velero backup describe shop-2026-09-27 --details
$ velero restore create --from-backup shop-2026-09-27 --namespace-mappings shop:shop-restore
Schedule it (velero schedule create), keep copies in another region or account, and test restores (see Kubernetes Administration, lesson 27).
Crash-consistent vs application-consistent
A snapshot of a running database is like pulling the power plug: most databases recover, but you're relying on crash recovery. Options, from good to best:
- Hooks: Velero pre/post hooks run commands in the pod (e.g.
fsfreeze, or a database flush/checkpoint) around the snapshot. Set via pod annotations such aspre.hook.backup.velero.io/command. - Database-native backups:
pg_dump,mysqldump, or better, continuous archiving (PostgreSQL WAL archiving via an operator like CloudNativePG) for point-in-time recovery. - Both: native backups for the data, Velero for the Kubernetes objects around it.
Try it: snapshot, break, restore
With Longhorn (or another snapshot-capable driver) and MinIO as object storage:
- Write a file to a PVC. Take a VolumeSnapshot. Delete the file.
- Create a new PVC from the snapshot, mount it in a pod, and find the file again.
- Install Velero with the AWS plugin pointing at MinIO. Back up the
shopnamespace with--snapshot-move-data(or file-system backup). - Restore into
shop-restorewith a namespace mapping and verify the data. - Add a pre-backup hook that runs
syncin the app container and check the hook ran (velero backup describe --details).
Going deeper: backup strategy
- Write down RPO/RTO per application first; backup frequency and method follow from it.
- Encrypt backups, use object lock (immutability) where possible, and separate backup credentials from cluster admins.
- Track backup success and age as metrics; alert when the newest good backup is older than the RPO.
- Restoring to a different storage class or cluster is common in DR. Velero supports storage-class mapping; test it.
Recap
- Replication ≠ snapshot ≠ backup: each protects against different failures.
- VolumeSnapshots (CRDs + controller + a capable driver); restore via a PVC
dataSource. - Velero backs up namespaces + volumes to object storage (CSI with data movement, or file-system backup).
- Aim for application-consistent backups (hooks or native tools), and test restores into a separate namespace.
This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.