Prasanta Kumar Behera
Linux — Level by Level›10 · Performance triage
Learning Hub / Linux & Scripting / Linux — Level by Level

Lesson 10 of 19 · Level 2 — Intermediate: troubleshooting

Performance triage

When a server is 'slow', find out why in minutes. The USE method (utilisation, saturation, errors) applied to CPU, memory, disk and network, with the classic 60-second checklist of commands.

Practitioner
Key wordsUSE methodload averagevmstatiostatfreepidstatsarOOM killeriowait

Don't guess. Check every resource the same way.

"The server is slow" could be CPU, memory, disk, network, or a lock inside the application. The USE method (Brendan Gregg) gives you a checklist. For every resource, check:

  • Utilisation: how busy is it?
  • Saturation: is work queuing for it?
  • Errors: is it failing?

A supermarket feels slow. Is every till busy (utilisation)? Is there a long queue at the tills (saturation)? Is a till broken (errors)? You check the tills, then the trolleys, then the car park, the same three questions each time, until you find the bottleneck.

Resource Utilisation Saturation Errors
CPU mpstat, top (%us, %sy) vmstat r > number of CPUs; load average rare
Memory free -h (available) swap activity (vmstat si/so), OOM kills dmesg OOM messages
Disk iostat -x %util iostat aqu-sz, await dmesg I/O errors
Network sar -n DEV throughput vs link speed drops, retransmits (sar -n ETCP, ss -ti) ip -s link errors

The 60-second checklist

$ uptime
 10:42:11 up 12 days,  3:04,  2 users,  load average: 7.92, 6.10, 3.40

Load averages over 1, 5 and 15 minutes. Compare them with the number of CPUs (nproc). Rising (1-min > 15-min) means it's getting worse now.

$ vmstat 1 5
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 9  0      0 181212  90312 5213440    0    0     2    40 3121 5902 88 10  2  0  0
 8  0      0 180980  90312 5213460    0    0     0    52 3050 5811 89  9  2  0  0
  • r (runnable) 8–9 on a 4-CPU box → CPU saturated.
  • us 88% → user code is burning CPU (find it with pidstat 1).
  • wa would show time waiting for I/O; si/so non-zero → swapping (memory pressure).
$ pidstat 1 1
UID       PID    %usr %system  %CPU   CPU  Command
1001     3321  350.00    8.00 358.00     2  java

One Java process using ~3.6 CPUs. Now it's an application question (a hot loop, garbage collection, traffic spike), and you have the evidence.

Disk:

$ iostat -xz 1 2
Device   r/s   w/s  rkB/s   wkB/s  r_await  w_await  aqu-sz  %util
nvme0n1  3.0 910.0   12.0 58240.0     0.40    38.20   34.80  99.60

%util near 100%, a queue (aqu-sz) of 35, and writes taking 38 ms (w_await): the disk is the bottleneck.

Memory:

$ free -h
               total        used        free      shared  buff/cache   available
Mem:           7.7Gi       1.3Gi       180Mi        12Mi       6.2Gi       6.1Gi
Swap:             0B          0B          0B

Only 180 MiB "free", but 6.1 GiB available. Linux fills spare RAM with cache and gives it back on demand. Worry when available is low, when swap is active, or when dmesg shows OOM kills.

History: what happened at 3 a.m.?

sar (from sysstat, with its collector enabled) records all of the above every 10 minutes by default:

$ sar -u -f /var/log/sysstat/sa27 -s 02:30:00 -e 03:30:00     # CPU on the 27th, 02:30–03:30
$ sar -r                                                       # memory history today

(The data path is /var/log/sa/ on the RHEL family.)

Try it: create load, then find it

Install sysstat and stress-ng on a VM, then for each scenario open a second terminal and run the checklist. Name the bottleneck before looking at what you started.

  1. CPU: stress-ng --cpu $(nproc) --timeout 120s
  2. Memory: stress-ng --vm 2 --vm-bytes 90% --timeout 120s. Watch free -h, vmstat si/so and dmesg.
  3. Disk: stress-ng --hdd 2 --timeout 120s. Watch iostat -xz 1 and the wa column.
  4. Write one line per scenario: utilisation / saturation / errors for the resource that was the bottleneck.

Going deeper: beyond the checklist

  • Steal time (st in vmstat/top) on VMs means the hypervisor gives your CPU time to someone else. A noisy neighbour, not your code.
  • CPU throttling in containers looks like slowness with idle CPUs: check cgroup cpu.stat (nr_throttled) or container_cpu_cfs_throttled_* metrics (see Kubernetes Administration, lesson 23).
  • Latency, not averages: the average can look fine while the p99 is terrible. Tools like biolatency and runqlat (lesson 16, eBPF) show distributions.
  • Always record before and after numbers for any tuning change. Tuning without measurement is superstition.

Recap

  • USE: utilisation, saturation, errors, for CPU, memory, disk and network.
  • Checklist: uptime, dmesg, vmstat, mpstat, pidstat, iostat -x, free -h, sar -n DEV, top.
  • High load + idle CPU → I/O wait; low "free" but high "available" → healthy cache.
  • sar answers "what happened earlier?"; dmesg catches OOM kills and hardware errors.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.