Prasanta Kumar Behera
Incident Handling — On-Call Playbook & Real Scenarios›16 · Node disk full: images and logs
Learning Hub / Incident Handling / Incident Handling — On-Call Playbook & Real Scenarios

Lesson 16 of 19 · Real-world incident scenarios

Node disk full: images and logs

Pods are evicted with 'The node was low on resource: ephemeral-storage' and new pods won't schedule on a node marked DiskPressure. Find what filled the disk (images, container logs, emptyDir, journald, stray files), free space safely, and set limits and garbage collection so it doesn't happen again.

Practitioner
Key wordsDiskPressureevictionsnodefsimagefsimage garbage collectioncontainer log rotationephemeral-storagecrictl rmi --prunejournalddu

The page

03:40 — KubeNodeDiskPressure on two nodes, followed by pod evictions. A new version of a service started logging every request at debug level, and a report job writes temp files to emptyDir. The nodes have 50 GB root disks shared by images, logs and pod storage.

First five minutes

  • Impact: evicted pods (hopefully rescheduled elsewhere), nodes refusing new pods. If several nodes are affected, capacity may collapse.
  • Scope: which nodes, and what changed (a new chatty version, a new job)?
  • Stop the source if it's obvious: roll back the debug-logging version, pause the job.

The classroom bin is overflowing. The cleaner starts taking things off desks to make room (evictions). First you stop the kid who's shredding paper all day (the chatty log), then empty the bin properly (prune images, vacuum logs), and finally give everyone their own small bin with a size limit (ephemeral-storage limits).

Diagnose

$ kubectl describe node gen-04 | grep -A6 Conditions
  DiskPressure   True    KubeletHasDiskPressure
$ df -h / /var/lib/containerd /var/log
$ sudo du -xh --max-depth=2 /var | sort -h | tail -15
$ sudo crictl images | wc -l
$ sudo journalctl --disk-usage
$ sudo du -sh /var/log/pods/* | sort -h | tail
$ sudo du -sh /var/lib/kubelet/pods/*/volumes/kubernetes.io~empty-dir/* 2>/dev/null | sort -h | tail

Typical culprits: container logs (/var/log/pods), images (/var/lib/containerd), emptyDir volumes, journald, core dumps, or files someone left in /root or /tmp.

Free space safely

$ sudo crictl rmi --prune                                      # unused images only
$ sudo journalctl --vacuum-size=500M
$ kubectl get pods -A --field-selector=status.phase=Failed     # evicted pods remain as Failed records
$ kubectl delete pods -A --field-selector=status.phase=Failed  # clean up the records (optional)

Don't delete files inside /var/lib/containerd, or a running container's log file, by hand. If a container's log is huge, fix the log level and let log rotation handle it.

Verify

  • DiskPressure returns to False, the taint is removed, and pods schedule again.
  • The disk-usage trend is flat once the source is fixed.

Prevent

  • Container log rotation: kubelet containerLogMaxSize / containerLogMaxFiles, and sane application log levels.
  • Image GC: kubelet imageGCHighThresholdPercent / imageGCLowThresholdPercent; fewer, smaller images.
  • ephemeral-storage requests/limits and emptyDir.sizeLimit for pods that use local disk.
  • Separate disks for container runtime data where possible; bigger root disks for busy nodes.
  • Alert on predicted fill (predict_linear(node_filesystem_avail_bytes[6h], 24*3600) < 0) and on DiskPressure.
  • journald SystemMaxUse= in /etc/systemd/journald.conf.

Try it: fill a node (lab)

  1. On a lab node, run a pod that writes to emptyDir in a loop (dd if=/dev/zero of=/data/f bs=1M count=…) without limits, and watch DiskPressure and evictions.
  2. Add ephemeral-storage limits and an emptyDir.sizeLimit; watch the pod get evicted instead of the node filling.
  3. Pull several large images, then prune them with crictl.
  4. Check your kubelet's eviction and image-GC settings (/var/lib/kubelet/config.yaml).
  5. Add the predictive disk alert.

Going deeper: local storage as a resource

  • Nodes report allocatable ephemeral-storage, and the scheduler uses requests, so set them for heavy writers.
  • Ship logs off-node (see Centralized Logging with EFK) and keep local retention short.
  • For build/CI nodes, consider separate pools with larger disks and aggressive cleanup.

Recap

  • DiskPressure → no new pods + evictions; find the filesystem and the culprit with df, du, crictl, journalctl.
  • Free space with crictl rmi --prune, a journal vacuum, and fixing chatty logs; never hand-delete runtime data.
  • Prevent with log rotation, image GC, ephemeral-storage limits, separate disks and predictive alerts.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.