Prasanta Kumar Behera
Linux — Level by Level›08 · Disks & filesystems
Learning Hub / Linux & Scripting / Linux — Level by Level

Lesson 08 of 19 · Level 2 — Intermediate: troubleshooting

Disks & filesystems

Solve 'disk full' properly: df vs du, inode exhaustion, space held by deleted files, mounts and /etc/fstab, growing a filesystem with LVM, and keeping logs from filling disks again.

Practitioner
Key wordsdfduinodesdeleted open fileslsofmountfstabLVMresizelogrotate

"The disk is full" is four different problems

  1. Real files are taking up the space.
  2. Deleted files are still held open by a process.
  3. The filesystem ran out of inodes, not bytes.
  4. The data is on a different filesystem than you think.

A library shelf can be "full" in different ways. The books really fill it. Or books were "thrown away" but someone is still holding them in the reading room, so they can't leave the building. Or there's space on the shelf but the catalogue cards have run out (inodes), so no new book can be registered. Or you're looking at the wrong shelf entirely.

Start with df, then du

$ df -h
Filesystem                 Size  Used Avail Use% Mounted on
/dev/mapper/vg0-root        30G   29G  1.0G  97% /
/dev/mapper/vg0-data       200G   80G  120G  40% /var/lib/data
tmpfs                      3.9G     0  3.9G   0% /dev/shm

df = free space per filesystem. du = space used by files you can see:

$ sudo du -xh --max-depth=1 / | sort -rh | head
29G /
18G /var
6.1G    /usr
3.2G    /home
$ sudo du -xh --max-depth=1 /var | sort -rh | head -3
17G /var
15G /var/log
1.8G    /var/lib

-x stays on one filesystem, so it doesn't wander into /var/lib/data on another disk. Drill down level by level until you find the culprit.

Deleted but still open

$ sudo lsof +L1
COMMAND  PID     USER  FD  TYPE DEVICE    SIZE/OFF NLINK  NODE NAME
java    2211   appsvc  5w  REG  253,0  12884901888     0  1311 /var/log/app/app.log (deleted)

Someone deleted a 12 GB log while the Java process was still writing to it. The space returns when the process closes the file: restart it, or make it reopen its logs. As an emergency measure without a restart, you can truncate through the process's file descriptor: sudo truncate -s 0 /proc/2211/fd/5.

Don't rm a live log. Truncate or rotate it.

To empty a log that's in use: sudo truncate -s 0 /var/log/app/app.log (the process keeps writing to the same, now empty, file). Better: configure logrotate so it never grows out of control.

Out of inodes

$ df -i /
Filesystem              Inodes   IUsed  IFree IUse% Mounted on
/dev/mapper/vg0-root   1966080 1966080      0  100% /
$ sudo find / -xdev -type d -exec sh -c 'echo "$(ls -A "$1" | wc -l) $1"' _ {} \; 2>/dev/null | sort -rn | head -3
1843201 /var/spool/app/sessions

Millions of tiny files (sessions, cache entries, a mail queue) used every inode. Clean them up and fix what creates them.

Disks, partitions and mounts

$ lsblk -f
NAME            FSTYPE      FSVER    LABEL UUID                                 MOUNTPOINTS
sda
├─sda1          vfat        FAT32          7C1A-3F2E                            /boot/efi
└─sda2          LVM2_member LVM2 001       c3Xh…
  ├─vg0-root    ext4        1.0            1f2e…                                /
  └─vg0-data    xfs                        9a8b…                                /var/lib/data
sdb

Permanent mounts live in /etc/fstab. Refer to filesystems by UUID, not /dev/sdX (device names can change between boots):

UUID=9a8b…  /var/lib/data  xfs  defaults,nofail  0  2

Test right away with sudo mount -a. A broken fstab line can drop the next boot into emergency mode; nofail keeps non-essential mounts from blocking boot.

Growing space with LVM

LVM (Logical Volume Manager) pools disks into volume groups and carves out logical volumes you can grow online:

disks (PV: /dev/sdb) → volume group (VG: vg0) → logical volumes (LV: vg0/data) → filesystem (xfs) → mount

Add a new disk /dev/sdb and grow /var/lib/data by 50 GB:

$ sudo pvcreate /dev/sdb
$ sudo vgextend vg0 /dev/sdb
$ sudo lvextend -r -L +50G /dev/vg0/data     # -r also grows the filesystem
$ df -h /var/lib/data

(XFS can grow but never shrink; ext4 can shrink only while unmounted.)

Keep it from happening again

  • logrotate (/etc/logrotate.d/): rotate, compress and delete old logs on a schedule, with copytruncate or a post-rotate signal so apps reopen files.
  • Journal limits: SystemMaxUse= in journald.conf.
  • Separate filesystems for /var/log and /var/lib/<data> so a log flood can't fill /.
  • Alerts at 80% and 90% of space and inodes.
  • On Kubernetes nodes: the kubelet's image garbage collection and eviction thresholds (see Kubernetes Administration, lesson 26).

Try it: create and solve 'disk full'

On a disposable VM:

  1. Create a small test filesystem from a file: truncate -s 200M /tmp/disk.img && mkfs.ext4 -q /tmp/disk.img && sudo mkdir -p /mnt/test && sudo mount -o loop /tmp/disk.img /mnt/test.
  2. Start a process that holds a file open: sudo sh -c 'yes > /mnt/test/big.log' & and wait until df -h /mnt/test shows 100%. Stop it with sudo pkill yes.
  3. Start a holder: sudo sh -c 'exec 3>>/mnt/test/big.log; sleep 600' &, then sudo rm /mnt/test/big.log. Check df -h (still full!) and sudo lsof +L1.
  4. Kill the holder and watch df recover.
  5. Inodes: sudo sh -c 'for i in $(seq 1 60000); do : > /mnt/test/f$i; done' and check df -i /mnt/test. How many files fit before you hit "No space left on device"?
  6. Clean up: sudo umount /mnt/test && rm /tmp/disk.img.

Going deeper: storage for platforms

  • Container runtimes store images and writable layers under /var/lib/containerd (or /var/lib/docker). Put it on its own filesystem on Kubernetes nodes so image pulls can't fill /.
  • iostat -x 1 shows per-device utilisation and latency (lesson 10); a disk at 100% %util with high await is the bottleneck, whatever df says.
  • Filesystem choice: ext4 (versatile, can shrink offline), XFS (large files and parallel I/O, grows only). Both are fine defaults for most workloads.
  • For databases and etcd, what matters is fsync latency. Test with fio before trusting new storage.

Recap

  • df (per filesystem) and du -x (per directory) answer different questions. Use both.
  • df ≫ du → deleted-but-open files (lsof +L1); free space but "No space left" → inodes (df -i).
  • Mount by UUID, test fstab with mount -a, use nofail for optional mounts.
  • LVM + lvextend -r grows filesystems online; logrotate, journal limits and alerts prevent repeats.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.