Longhorn volumes won't attach after a reboot

After rebooting a node (or a whole small site), pods using Longhorn volumes are stuck in ContainerCreating with attach or mount errors. Check the Longhorn engine and replicas, iscsid on the node, multipathd grabbing Longhorn devices, and stale attachments to the old node, then fix the root cause in the node image.

The page

Sunday 07:30 — after planned maintenance rebooted all three nodes of a small edge cluster, the database and message-queue pods are stuck in ContainerCreating. Stateless apps are fine. The Longhorn UI shows two volumes "Attaching" and one "Degraded".

First five minutes

Impact: stateful services down; stateless may be working but without their data stores.
Don't delete volumes or replicas "to reset". That's how data gets lost.
Collect events for one stuck pod and the Longhorn state of its volume.

After a power cut, the library's book trolleys (volumes) have to be wheeled back to the right rooms. If the lift isn't working (iscsid off), trolleys can't go upstairs. If a cleaner parked their own cart in the doorway (multipathd grabbing the device), the trolley can't get in. And if the register still says a trolley is in a room that's locked (stale attachment), you must check the room is really empty before moving it.

Diagnose

$ kubectl describe pod postgres-0 -n shop-data | sed -n '/Events/,$p'
  Warning  FailedMount  MountVolume.MountDevice failed for volume "pvc-8f…" : … already mounted or mount point busy
$ kubectl -n longhorn-system get volumes.longhorn.io pvc-8f… -o wide
$ kubectl get volumeattachments | grep pvc-8f

On the node where the pod is scheduled:

$ systemctl status iscsid --no-pager
$ multipath -ll
$ lsblk

Common causes and fixes

Cause	Evidence	Fix
iscsid not running after reboot	`iscsid` inactive; attach errors	`systemctl enable --now iscsid`; bake into the node image
multipathd claims Longhorn devices	`multipath -ll` lists the Longhorn disk; mount "busy"	Blacklist Longhorn's devices in `/etc/multipath.conf` (per Longhorn's KB), restart multipathd, then retry the pod
Stale attachment to another node (RWO multi-attach)	`Multi-Attach error`, VolumeAttachment to an old node	Confirm that node is down; let the controller detach, or remove the stale attachment/force-delete the old pod
Degraded/faulted replicas after all nodes rebooted	Volume robustness "degraded"/"faulted"	Wait for rebuild on healthy nodes; for faulted volumes follow Longhorn's salvage procedure; check disk space on nodes
Node not ready yet for Longhorn	longhorn-manager/instance-manager pods not running	Wait/fix those pods first

Longhorn has a setting for pod deletion when a node is down (so StatefulSet pods on a dead node are cleaned up and their volumes can move); know how yours is configured.

Verify

Pod Running, volume healthy with the expected number of replicas.
Application checks: database accepts connections, data present.

Prevent

Node image: open-iscsi installed and enabled, multipath blacklist configured (or multipathd disabled if unused), Longhorn's environment check passing before joining.
Maintenance procedure: drain one node at a time and wait for volumes to be healthy before the next; avoid rebooting all nodes at once on small sites.
Alerts on Longhorn volume robustness (degraded/faulted) and node disk space.
Backups to off-site object storage (Longhorn backups or Velero), tested restores (see Kubernetes Storage & Data Protection, lesson 04).

Try it: break Longhorn safely (lab)

Install Longhorn on a 3-node lab and create a StatefulSet with a volume; write some data.
Stop and disable iscsid on one node, reboot it, and schedule the pod there (node affinity); read the events.
Re-enable iscsid and watch it recover.
Enable multipathd without a blacklist (on a lab node) and reproduce the "busy" mount; then add the blacklist.
Reboot one node while writing data and watch replica rebuild in the Longhorn UI.

Going deeper: stateful at the edge

Three replicas on three nodes is the minimum for real redundancy; single-node sites need backups, not replicas.
Keep Longhorn versions and node prerequisites in the site template; upgrades have their own procedures.
Consider app-level replication (database operators) on top of or instead of storage replication for critical data.

Recap

Stuck Longhorn volumes after reboot: check events, volume/replica state, iscsid, multipathd, stale attachments.
Fix the node prerequisites (iscsid enabled, multipath blacklist) in the image.
Never force a RWO volume onto a second node until the first is really gone.
Reboot one node at a time; alert on degraded volumes; keep tested backups.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.