etcd out of space at 2 a.m.

First five minutes

• Impact: running workloads are fine; changes fail (deploys, scaling, leader elections that write leases, and eventually controllers). Treat it as SEV2 (SEV1 if autoscaling or failover depends on writes right now).
• Freeze changes: pause CI/CD and GitOps syncs; they'll only add writes and noise.
• Confirm it's etcd, not the API server: the error text names etcd.

Quota alarm or full disk?

$ export ETCDCTL_API=3
$ ETCD="--endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key"
$ etcdctl $ETCD endpoint status --cluster -w table
$ etcdctl $ETCD alarm list
memberID:8e9e05c52164694d alarm:NOSPACE
$ df -h /var/lib/etcd

(Paths are kubeadm defaults; etcdctl may need installing on the host, or run it inside the etcd pod with kubectl -n kube-system exec etcd-<node> -- etcdctl ….)

Recover (quota case)

$ etcdctl $ETCD snapshot save /var/backups/etcd-$(date +%F-%H%M).db     # 1. safety first
$ rev=$(etcdctl $ETCD endpoint status -w json | jq '.[0].Status.header.revision')
$ etcdctl $ETCD compact "$rev"                                            # 2. drop old revisions
$ etcdctl $ETCD defrag --endpoints=https://10.10.0.12:2379                # 3. one member at a time (followers first)
$ etcdctl $ETCD defrag --endpoints=https://10.10.0.13:2379
$ etcdctl $ETCD defrag --endpoints=https://10.10.0.11:2379                # leader last
$ etcdctl $ETCD endpoint status --cluster -w table                        # DB size should drop
$ etcdctl $ETCD alarm disarm                                              # 4. clear the alarm

(Adjust endpoint addresses to your members. Compaction removes history, not current data; the API server normally compacts every few minutes, so a big gap means something unusual is growing.)

Full disk case: free space on the etcd volume first (old snapshots, logs, unrelated files on the same disk); never delete files inside the etcd data directory by hand.

Verify

• kubectl create deploy smoke --image=registry.k8s.io/pause:3.10 succeeds; delete it.
• Controllers catch up (Deployments reconcile, events flow).
• Resume GitOps/CI.

Find what filled it

$ kubectl get events -A --no-headers | wc -l
$ kubectl get events -A --sort-by=.count | tail
$ etcdctl $ETCD get / --prefix --keys-only | cut -d/ -f3 | sort | uniq -c | sort -rn | head

Common causes: a controller or operator rewriting objects in a loop, event storms from a crash-looping workload, very large Secrets/ConfigMaps, Helm release history kept forever (each release is a Secret), CRDs storing big objects.

Prevent

• Alert on DB size (e.g. etcd_mvcc_db_total_size_in_bytes above 70–80% of the quota) and on the NOSPACE alarm.
• Fix the source (loop, history limits such as Helm --history-max, event rate).
• Right-size the quota deliberately (etcd documents a practical maximum; larger databases slow recovery), and defragment periodically in maintenance windows.
• Keep etcd on its own fast disk, separate from logs and images; keep regular snapshots off-node (see Kubernetes Administration, lesson 13).

Recap

• NOSPACE = quota reached; a full disk is different. Check both.
• Snapshot → compact → defrag one member at a time → disarm, then verify writes.
• Find the source of growth and alert on DB size before the quota.