Disaster recovery · Prasanta Kumar Behera

Plan for losing a namespace, a cluster or a whole site. Define RPO and RTO, combine etcd snapshots, Velero and GitOps, and prove the plan with timed restore drills.

Start with two numbers

Before tools, agree with the business on two numbers per service:

RPO (Recovery Point Objective): how much data may be lost, measured in time. "At most 15 minutes of orders."
RTO (Recovery Time Objective): how quickly service must be back. "Within 1 hour."

Everything else (backup frequency, standby clusters, cost) follows from them.

Imagine your homework folder falls in a puddle. RPO is how much work you'd have to redo: if you photocopy it every evening, at most one day. RTO is how long until you can hand it in again: finding the copy, re-printing, re-stapling. A spare folder already packed in your bag (warm standby) makes RTO tiny, but you have to keep it up to date.

What can be lost, and what brings it back

Disaster	Recovery tool
Someone deletes a namespace	Velero restore of that namespace, or GitOps re-sync plus a data restore
Bad change corrupts cluster objects	etcd snapshot restore (lesson 13), or GitOps revert
A volume's data is corrupted	Volume snapshots / Velero file-system backup
The whole cluster is lost	Rebuild from Git (cluster definition + GitOps apps) + restore data
The whole site or region is lost	Second site: standby cluster, data replicated or restored there

Three layers, used together:

Git (declarative configuration): the cluster definition, platform add-ons and applications. Rebuilds what should run.
Data backups: PersistentVolumes, databases, object storage. Restores what was stored.
etcd snapshots: the exact state of one specific cluster, fastest for "undo the last hour" on that cluster.

Velero: backups of namespaces and volumes

Velero stores Kubernetes objects (and optionally volume data) in object storage (S3, MinIO, Azure Blob, GCS…):

$ velero backup create shop-2026-09-27 --include-namespaces shop
Backup request "shop-2026-09-27" submitted successfully.
$ velero backup describe shop-2026-09-27 --details
Phase:  Completed
...
$ kubectl delete namespace shop          # the "disaster"
$ velero restore create --from-backup shop-2026-09-27
$ kubectl get all -n shop

Volume data is captured either by CSI volume snapshots (fast, storage-dependent) or by Velero's file-system backup (copies files from the pod's volumes, storage-independent). Choose per workload, and check describe --details shows the volumes you expect.

Schedule it:

$ velero schedule create daily-shop --schedule="0 2 * * *" --include-namespaces shop --ttl 720h

Databases deserve their own backups

For databases, prefer the database's own tools (for example, continuous WAL archiving for PostgreSQL through an operator). They give consistent, point-in-time recovery. A file-level copy of a running database volume may not be consistent.

DR strategies for a whole site

Strategy	Idea	RTO	Cost
Backup & restore	Rebuild everything elsewhere from Git and backups	Hours	$
Pilot light	A minimal cluster always running; scale up and restore data when needed	~1 hour	$$
Warm standby	A smaller, fully running copy with data replicated	Minutes	$$$
Active-active	Both sites serve traffic all the time	~0	$$$$

Most platforms use different strategies for different services: active-active for checkout, backup-and-restore for internal tools.

The runbook: order matters

When everything is gone, restore in dependency order:

Cluster(s) from the declarative definition (Terraform, Cluster API, EKS Anywhere spec…).
Platform add-ons (CNI, storage, ingress, cert-manager, secrets tooling) via GitOps.
Secrets and keys from your secret manager (without these, nothing else starts).
Data: databases and volumes, to the agreed RPO.
Applications via GitOps, then DNS and traffic cut-over.
Verify with synthetic checks and business checks, then declare recovery.

Try it: a timed restore drill (kind + MinIO)

Run MinIO in the cluster (or on your laptop) as an S3-compatible target, and install Velero with the AWS plugin pointing at it (s3Url and s3ForcePathStyle=true in the backup-location config).
Deploy the shop app from lesson 08 and add some orders.
Start a stopwatch. velero backup create, then kubectl delete namespace shop, then velero restore create.
Stop the stopwatch when the orders are readable again. That's your RTO for this service. Were the orders all there? That's your RPO check.
Write down every surprise. Those surprises are the real output of a drill.

Going deeper: DR that works on the day

Keep backups in a different account/region with immutability (object lock), so ransomware or a compromised admin can't delete them.
Test quarterly, rotate who runs the drill, and time it. The runbook should work for someone who has never done it.
Make DNS TTLs, certificate issuance and identity providers part of the plan. Recoveries often stall on "we can't log in" or "the certificate is for the old cluster".
Back up the backup system's configuration too: credentials, bucket names, encryption keys.

Recap

Agree RPO and RTO per service first; they drive every design choice.
Combine Git (what should run), data backups (what was stored) and etcd snapshots (this cluster's exact state).
Velero for namespaces and volumes; database-native backups for databases.
Restore in dependency order, and prove it all with timed drills.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.