Storage performance · Prasanta Kumar Behera

Measure storage honestly: IOPS, throughput and latency percentiles with fio, how replication and networks cost performance, the fsync test that matters for etcd and databases, and taming noisy neighbours.

Three numbers, not one

Metric	Meaning	Matters for
IOPS	Operations per second (usually small, random)	Databases, many small files
Throughput	MB/s (usually large, sequential)	Backups, logs, analytics, media
Latency	Time per operation, including percentiles (p99)	Everything interactive; etcd and databases most of all

Throughput ≈ IOPS × block size, and latency caps both: with one request in flight, IOPS ≤ 1 / latency. Queue depth (requests in flight) is how systems get high IOPS despite latency.

Measuring a water pipe: how many cups per minute it can fill (IOPS), how many litres per minute gush out (throughput), and how long you wait before the first drop arrives after turning the tap (latency). A giant pipe with a long wait is useless for a quick drink, and databases are always thirsty for quick drinks.

Measure with fio

Run fio inside a pod that mounts the volume you care about, so you measure the real path (CSI, network, replicas):

apiVersion: v1
kind: Pod
metadata:
  name: fio
spec:
  restartPolicy: Never
  containers:
    - name: fio
      image: registry.example.com/tools/fio:3      # any image with fio installed
      command: ["sleep", "infinity"]
      volumeMounts:
        - { name: data, mountPath: /data }
  volumes:
    - name: data
      persistentVolumeClaim: { claimName: perf-test }

$ kubectl exec -it fio -- fio --name=randwrite --rw=randwrite --bs=4k --iodepth=32 \
    --size=2G --runtime=60 --time_based --direct=1 --filename=/data/fio.test --group_reporting
  write: IOPS=8412, BW=32.9MiB/s (34.5MB/s)
    clat percentiles (usec):
     |  50.00th=[ 3589],  90.00th=[ 5145],  99.00th=[ 9765],  99.90th=[18744]

(Illustrative numbers.) Always look at the percentiles, not just the average.

The fsync test (etcd and databases)

etcd and databases do small writes followed by a flush to stable storage. This test, widely used for etcd disk qualification, mimics that pattern:

$ fio --name=fsync --rw=write --ioengine=sync --fdatasync=1 --bs=2300 --size=22m --filename=/data/fio.test
  fsync/fdatasync/sync_file_range:
    sync (usec): min=512, max=14201, avg=1102.40
    sync percentiles (usec):
     |  99.00th=[ 3163]

A common guideline for etcd is a 99th percentile fdatasync under ~10 ms. Replicated network storage often struggles here, which is why etcd usually runs on local SSD/NVMe (see Kubernetes Administration, lesson 26).

Where performance goes

Factor	Effect
Replication (Longhorn, Ceph)	Each write crosses the network to every replica: higher latency, more network and CPU
Network	Bandwidth caps throughput; latency and jitter hit every operation
Encryption, compression	CPU cost per I/O
Filesystem & block size	Mismatched sizes cause extra work
Noisy neighbours	Other volumes on the same disks or nodes compete
Cloud volume limits	Provisioned IOPS/throughput caps, and burst credits that run out

Choosing storage by workload

Workload	Good fit
etcd, latency-critical databases with their own replication	Local NVMe (local PVs, TopoLVM), let the app replicate
General stateful apps	Replicated block (Longhorn, Ceph RBD, cloud disks)
Shared files	CephFS, NFS/EFS (RWX): expect higher latency
Backups, artifacts, logs at rest	Object storage (S3, RGW, MinIO)

Taming noisy neighbours

Separate storage classes (fast NVMe vs capacity HDD) and put critical workloads on dedicated disks or nodes.
Use QoS features where the backend offers them (cloud provisioned IOPS, Ceph QoS, Longhorn per-volume settings).
Watch per-volume latency; a sudden rise often means another tenant started a heavy job.

Try it: compare storage classes

Create two PVCs: one on kind's standard (local path) and one on Longhorn (or another replicated class).
Run the random-write and fsync tests in a pod on each; record IOPS and p99 latency.
Repeat the Longhorn test with 1 replica vs 3 replicas.
Run a second fio pod on the same node at the same time and measure the drop: that's a noisy neighbour.
Write a one-paragraph recommendation: which class would you give etcd, a PostgreSQL, and a shared uploads folder?

Going deeper: performance engineering

Benchmark with realistic block sizes and read/write mixes (look at your app's I/O pattern with iostat or eBPF tools like biolatency) instead of vendor-favourite numbers.
Run tests long enough to exhaust caches and cloud burst credits.
Always record the whole setup (node type, disk, replica count, network, fio command) with results. Numbers without context can't be compared later.
Monitor storage latency in production continuously, not only during benchmarks.

Recap

Measure IOPS, throughput and latency percentiles; use direct I/O and realistic patterns.
The fdatasync test predicts etcd and database commit performance; aim for low p99.
Replication, network, encryption and neighbours all cost performance. Pick storage per workload.
Separate classes and QoS tame noisy neighbours; monitor latency in production.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.