DR & compliance at scale · Prasanta Kumar Behera

Recover and prove compliance across a fleet: rebuild a whole site from Git plus backups, etcd and application backups sent to the centre, recovering from power loss (boot order, quorum, time), and continuous compliance evidence: CIS checks, inventory, versions and drift per site.

Rebuild from Git

The goal: any site can be wiped and rebuilt remotely.

Layer	Recreated from
Hardware provisioning	Site inventory (CSV/Hardware/BMH) in Git + OS image
Cluster	Cluster spec in Git (EKS-A, CAPI, RKE2 config)
Platform + apps	GitOps repo (lesson 10) + registry/bundle
Data	Backups (application data, volumes, etcd if restoring)

Many teams recreate the cluster from its definition and restore only application data, rather than restoring etcd, because it's simpler and avoids resurrecting stale state. Restoring etcd is still needed for state that isn't in Git.

If a lemonade stand burns down, headquarters doesn't panic: it has the stand's plan, the recipe book, and a copy of the stand's sales notebook sent every night. It ships new parts, the stand is rebuilt from the plan, and the notebook copy is put back. Every few months, headquarters practises this on a test stand to be sure it really works.

Backups that leave the site

etcd: RKE2 takes automatic snapshots and can upload them to S3-compatible storage; kubeadm-based clusters use etcdctl snapshot save (see Kubernetes Administration, lesson 13).
Kubernetes objects and volumes: Velero (with CSI snapshots or file-level backup) to object storage at the centre.
Application-native backups for databases (dumps, WAL shipping) are often the most reliable.
Budget the uplink: incremental backups, compression, scheduling outside business hours; a single-node site's backup is its redundancy.
Test restores regularly on a lab site, and measure how long a full site rebuild takes.

Scenario: the site came back after a power loss; the cluster didn't

A storm cut power to a 3-node site for 6 hours. When power returned, all three servers booted, but the Kubernetes API stayed down and workloads didn't start.

Where do you look, and what prevents it?

Time: the servers' clocks drifted or reset; with no NTP reachable yet, certificates appeared not-yet-valid or expired and components refused TLS. Check timedatectl, chrony status, and errors like x509: certificate has expired or is not yet valid.
Boot order: servers came up before the switch or router; DHCP/static network wasn't ready when kubelet and etcd started, and they backed off. Check service logs from boot time.
etcd quorum: members couldn't reach each other in time; check etcdctl endpoint health and member logs (on RKE2, etcd runs as a static pod).
VIP: kube-vip didn't claim the VIP because the control plane wasn't healthy yet, so nodes couldn't reach the API.

Prevention: local NTP with a hardware clock source where possible, BIOS power-on delays so network gear starts first, services that retry rather than give up, and a full power-cycle test during site acceptance.

Compliance at fleet scale

Baseline: CIS Kubernetes Benchmark (run kube-bench or your distribution's scanner), OS hardening benchmarks, and your own policies (Kyverno/Gatekeeper).
Inventory per site: OS image version, Kubernetes version, platform bundle, firmware/BIOS versions, Secure Boot and disk-encryption state, last successful backup and restore test.
Drift detection: GitOps corrects configuration drift; scanners and inventory catch OS/firmware drift.
Evidence: export scan results and inventory centrally (with timestamps and signatures if required), so audits become a report rather than a project.
Change records: Git history + GitOps sync history per site answer "who changed what, when, where".

Try it: rebuild and prove it

On a lab cluster (kind or VMs), install Velero with an S3-compatible bucket (MinIO or Ceph RGW), deploy an app with a PVC, and back it up.
Delete the whole cluster. Recreate it from your cluster definition and GitOps repo, then restore the app data with Velero. Time the process.
Take an etcd snapshot on an RKE2 or kubeadm lab cluster and practise a restore following the official procedure.
Run kube-bench on a node and fix two findings.
Build a one-page site inventory report (versions, encryption, last backup) from commands or APIs you can automate.

Going deeper: resilience as a routine

Schedule game days: power-cycle a lab site, rebuild a canary site, restore from backups, with timings recorded.
Define RTO/RPO per site class (store vs factory vs cell site) and design backups accordingly.
Keep spares and runbooks for hardware replacement that a non-expert can follow on site, with the rest automated.

Recap

Rebuild from Git: inventory + cluster spec + GitOps + registry, plus backups for data.
Send etcd snapshots, Velero and app-native backups off-site; test restores and time rebuilds.
Power-loss recovery: time sync, boot order, etcd quorum, VIP; test full power cycles.
Compliance: automated CIS checks, per-site inventory, drift detection and exported evidence.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.