Longhorn in practice · Prasanta Kumar Behera

Run replicated block storage on your own nodes with Longhorn: prerequisites, installation, how replicas and self-healing work, RWX volumes, backups to S3, and the operational habits that keep it healthy.

What Longhorn is

Longhorn (a CNCF project, originally from Rancher) turns the disks of your Kubernetes nodes into replicated block storage. Each volume has an engine (the controller for that volume) and replicas on different nodes. Every write goes to all replicas. It's popular for bare metal and edge clusters, where there's no cloud disk service.

Imagine writing your homework in three identical notebooks kept in three different classrooms. Every sentence you write goes into all three at once. If one classroom floods, you still have two notebooks, and the teacher makes a new third copy from them. But if you write something wrong, all three notebooks get the mistake, which is why you still keep a photocopy at home (a backup).

Prerequisites and install

Every node needs open-iscsi (and an NFS client for RWX volumes):

$ sudo apt install -y open-iscsi nfs-common
$ sudo systemctl enable --now iscsid
$ helm repo add longhorn https://charts.longhorn.io && helm repo update
$ helm install longhorn longhorn/longhorn -n longhorn-system --create-namespace
$ kubectl get sc
NAME                 PROVISIONER          RECLAIMPOLICY   VOLUMEBINDINGMODE
longhorn (default)   driver.longhorn.io   Delete          Immediate

Longhorn provides an environment-check script in its documentation. Run it before installing on real nodes.

Replicas, engines and self-healing

$ kubectl -n longhorn-system get volumes.longhorn.io
NAME                                       STATE      ROBUSTNESS   SIZE          NODE
pvc-3f1c2a4e-9d8b-4c1e-a7f2-0b6e5d4c3b2a   attached   healthy      10737418240   w1
$ kubectl -n longhorn-system get replicas.longhorn.io -o wide | grep pvc-3f1c
pvc-3f1c…-r-5a2c   running   w1
pvc-3f1c…-r-8b1d   running   w2
pvc-3f1c…-r-c4e9   running   w3

Robustness	Meaning
healthy	All replicas in sync
degraded	Fewer replicas than configured; still serving; rebuild in progress or pending
faulted	No usable replica; data unavailable

Useful settings (per StorageClass or volume):

numberOfReplicas: 3 for most data; 2 in small clusters (with care).
Replica anti-affinity across nodes (default) or zones.
Data locality (best-effort): keep one replica on the pod's node for faster reads.

RWX volumes

Longhorn serves ReadWriteMany through a share-manager pod that exports the volume over NFS. It's convenient for shared files, but it's a single NFS server per volume. Use RWO where possible.

Backups and snapshots

Snapshots are point-in-time copies stored with the replicas: fast, but on the same nodes.
Backups copy snapshots to a backup target (S3-compatible storage or NFS) outside the cluster. Configure the target in Longhorn settings, and use recurring jobs (e.g. snapshot hourly, backup nightly, keep 14).
Longhorn also supports CSI VolumeSnapshots, so Velero can drive it (lesson 04).

Try it: break a node, keep your data (3-node kind or VMs)

kind nodes are containers, so Longhorn on kind needs open-iscsi available inside each node container. VMs or a kubeadm lab are easier.

Install Longhorn and create a 2 GiB PVC with the longhorn class; run a pod that writes a file every second.
Find the three replicas and their nodes.
Stop a node that holds a replica but not the pod (multipass stop w3). Watch the volume go degraded, and the pod keep writing.
Start the node again (or wait) and watch the replica rebuild to healthy.
Configure an S3 backup target (MinIO works), take a backup, delete the PVC, and restore it from the backup.

Going deeper: operating Longhorn

Disks: give Longhorn dedicated disks (not the OS disk), keep reserved space, and monitor usage; over-provisioning is allowed but needs watching.
Node drains: set the node drain policy so drains wait for replicas to be safely rebuilt elsewhere, and upgrade Longhorn following its documented order (manager, then engines).
Network: replication traffic is heavy; a dedicated storage network (Multus) helps on busy clusters.
Monitoring: Longhorn exposes Prometheus metrics for volume robustness, capacity and rebuilds. Alert on degraded or faulted volumes.

Recap

Longhorn = replicated block storage on node disks: an engine per volume, replicas on different nodes.
States: healthy / degraded / faulted; lost replicas are rebuilt automatically.
RWX via an NFS share-manager; snapshots local, backups to S3/NFS outside the cluster.
Prerequisites (open-iscsi, NFS client), dedicated disks, drain policies and monitoring keep it healthy.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.