etcd backup & restore · Prasanta Kumar Behera

etcd holds every object in your cluster. Take snapshots, verify them, restore them on purpose, and understand quorum so a failed member never becomes a lost cluster.

Why etcd deserves its own lesson

Every Deployment, Secret, ConfigMap, RBAC rule and node record lives in etcd. Lose etcd without a backup and you've lost the cluster's memory. The containers may keep running for a while, but nothing can be managed and nothing can be rebuilt as it was.

etcd is the school's register: who's enrolled, which class they're in, who's allowed where. The classrooms keep going for a bit if the register burns, but nobody can enrol, move or check anything. A snapshot is a photocopy of the register kept in another building. And a photocopy is only useful if you've practised re-typing it into a new register.

Quorum in one table

etcd members vote. A write succeeds when a majority (quorum) agrees.

Members	Quorum	Failures tolerated
1	1	0
3	2	1
4	3	1 (no better than 3)
5	3	2

Use 3 or 5 members. Even numbers add cost and failure points without adding tolerance.

Take a snapshot

On a kubeadm control-plane node, the easiest way to get a matching etcdctl is the one inside the etcd pod. The certificates are already mounted there, and /var/lib/etcd is a host directory, so the file lands on the node:

$ kubectl -n kube-system exec etcd-cp1 -- etcdctl \
    --endpoints=https://127.0.0.1:2379 \
    --cacert=/etc/kubernetes/pki/etcd/ca.crt \
    --cert=/etc/kubernetes/pki/etcd/server.crt \
    --key=/etc/kubernetes/pki/etcd/server.key \
    snapshot save /var/lib/etcd/snapshot-$(date +%F).db
Snapshot saved at /var/lib/etcd/snapshot-2026-09-27.db

($(date +%F) is expanded on your machine, which is what you want.) Copy it off the node straight away. A backup on the machine that fails is not a backup.

Check it:

$ sudo etcdutl snapshot status /var/lib/etcd/snapshot-2026-09-27.db -w table
+----------+----------+------------+------------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| 7ef3c9a1 |   482913 |       1734 |     8.4 MB |
+----------+----------+------------+------------+

etcdctl vs etcdutl

etcdctl talks to a running etcd over the network (snapshot save, member list, endpoint status). etcdutl works on files offline (snapshot status, snapshot restore). Recent etcd versions moved the offline commands to etcdutl. Install both from the etcd release matching your server version if they aren't on the host.

Restore: the drill you must practise

This is the procedure for a single control-plane kubeadm cluster. Do it in your lab first.

Restoring rewinds the whole cluster

Everything created after the snapshot (new Deployments, rotated Secrets, new nodes) disappears from the API. Controllers then reconcile the running world back towards the restored state. That's intended, but know it before you press the button.

Stop the API server so nothing writes during the restore. Moving its static pod manifest out makes the kubelet stop it:

$ sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml /root/

Restore into a new data directory:

$ sudo etcdutl snapshot restore /var/lib/etcd/snapshot-2026-09-27.db --data-dir /var/lib/etcd-restored

Point etcd at it. In /etc/kubernetes/manifests/etcd.yaml, change the etcd-data volume's hostPath.path from /var/lib/etcd to /var/lib/etcd-restored. The kubelet notices the change and restarts etcd.

  volumes:
    - hostPath:
        path: /var/lib/etcd-restored     # was /var/lib/etcd
        type: DirectoryOrCreate
      name: etcd-data

Start the API server again and check:

$ sudo mv /root/kube-apiserver.yaml /etc/kubernetes/manifests/
$ kubectl get nodes
$ kubectl get deploy -A

Try it: prove your backup

Create a namespace before with a Deployment in it. Take a snapshot.
Create a namespace after. Then delete the before namespace.
Restore the snapshot using the steps above.
Expected result: before is back, and after is gone. You've just done a real point-in-time recovery. Note how long it took: that's your recovery time.

Going deeper: etcd in production

Schedule snapshots (a systemd timer or CronJob on control-plane nodes), encrypt them (they contain every Secret), ship them off-cluster, and alert when they stop.
Back up /etc/kubernetes/pki and your EncryptionConfiguration and keys alongside. A snapshot of encrypted Secrets is useless without the key.
Multi-member restore: restore the same snapshot on each member with its own --name, --initial-cluster and --initial-advertise-peer-urls, then start them together. Practise it: it's rarely done and easy to get wrong under pressure.
Health signals: fsync and backend-commit latency, leader changes, and DB size against the quota (default 2 GiB; mvcc: database space exceeded means writes stop). Compaction and defragmentation keep size in check.
Losing one member of three is routine: remove it with member remove, add a new one with member add. Losing quorum is the emergency you rehearse with snapshots.

Recap

etcd is the cluster's only source of truth. Run 3 or 5 members; quorum = majority.
etcdctl snapshot save to take, etcdutl snapshot status to verify, etcdutl snapshot restore into a new directory to recover.
Also back up the PKI and encryption keys, store copies off-cluster, and practise restores, because an untested backup is a hope, not a plan.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.