Prasanta Kumar Behera
Incident Handling — On-Call Playbook & Real Scenarios›11 · Node unreachable after a network restart
Learning Hub / Incident Handling / Incident Handling — On-Call Playbook & Real Scenarios

Lesson 11 of 19 · Real-world incident scenarios

Node unreachable after a network restart

Someone 'just restarted the network' (or a cable was unplugged and replugged) on an edge node, and it never came back: SSH is gone, the node is NotReady. Learn what actually breaks (routes added by hand, a downed loopback, DHCP, VIPs), how to recover through out-of-band access, and how to make network changes safe on remote nodes.

Practitioner → Advanced
Key wordsnode unreachablenetwork restartlost routesloopback downifdown -apersistent configurationnetplanNetworkManagerout-of-band consoleremote hands

The page

14:20 — KubeNodeNotReady: edge-site-31-node1. A support engineer was fixing a DNS setting over SSH and ran a network restart. The SSH session froze and never came back. On a second node at another site, a customer's technician unplugged a cable to move the switch; after replugging, the node stayed unreachable. Both sites are hundreds of kilometres away.

First five minutes

  • Impact: a single-node site is down; on a 3-node site, the cluster runs degraded (and the node's pods are rescheduled after the eviction timeout).
  • Don't ask the customer to power-cycle straight away: you'll lose the evidence and it may not fix a config problem.
  • Get out-of-band access: the BMC's remote console (Redfish/IPMI serial-over-LAN or HTML console).

You're on a phone call with a friend and ask them to turn the phone off and on again to fix the sound. The moment they do, the call ends, and you can't tell them what to do next. For remote machines, you need a second phone line (the BMC console) before doing anything that might hang up the first.

What actually breaks

Cause What you see
Hand-added routes (ip route add …) not in persistent config Interface up, but no route to the management network or default gateway
Loopback down (ifdown -a with lo in /etc/network/interfaces, or a bad config) ip -br link shows lo DOWN; etcd, kubelet health checks and anything on 127.0.0.1 fail
DHCP lease not renewed / static config missing after the link flap No IPv4 address on the interface
Wrong interface name / VLAN in a new config Address on the wrong NIC or untagged
VIP / kube-vip not reclaimed API endpoint unreachable though the node is up
CNI routes/interfaces removed and not recreated yet Pod networking broken on that node until the CNI agent reconciles

Diagnose from the console

$ ip -br link
lo        DOWN   00:00:00:00:00:00
eno1      UP     3c:ec:ef:…
$ ip -br addr
eno1      UP     10.31.0.11/24
$ ip route
10.31.0.0/24 dev eno1 proto kernel scope link src 10.31.0.11      # no default route!
$ ip route show table local | head -3
$ journalctl -u systemd-networkd -u NetworkManager --since -1h | tail -30

Recover

$ sudo ip link set lo up                      # if the loopback was down
$ sudo netplan apply                          # re-apply persistent config (or: nmcli con up <name>)
$ ip route                                    # default route back?
$ ping -c1 10.31.0.1 && ping -c1 127.0.0.1
$ sudo systemctl restart containerd kubelet   # once networking is correct
$ kubectl get node edge-site-31-node1 -w      # from the central side

If the missing route was added by hand months ago, add it to the persistent config now (netplan routes:), not just with ip route add again.

If there's no BMC access, you'll need remote hands: give the on-site person a short, exact script (connect a monitor, log in with a break-glass account, run specific commands), or ask them to reboot only if a reboot will apply the correct persistent configuration.

Prevent

  • All network config persistent and in Git (netplan/NetworkManager files rendered from the site definition); no hand-made routes.
  • Out-of-band access for every remote node, tested regularly (see Edge Kubernetes & Zero-Touch Provisioning, lesson 04).
  • Safe apply: netplan try (automatic rollback if you don't confirm), or scheduled rollback jobs (at now + 5 minutes) before risky changes.
  • Don't restart networking on Kubernetes nodes as a fix; change the specific setting and reload that component.
  • Alert on node unreachable and on loopback/interface state via node-exporter.

Try it: lose and regain a VM (lab)

  1. On a lab VM with a console (virt-manager/VirtualBox), add a route by hand, then restart networking and confirm it's gone.
  2. Make it persistent in netplan and repeat.
  3. Take lo down (ip link set lo down) on a kubeadm lab node and watch what fails (etcd, kubelet health checks); bring it back.
  4. Practise netplan try with a deliberately broken config and let it roll back.
  5. Write a one-page "remote hands" script for a customer technician.

Going deeper: resilient remote nodes

  • Separate management and workload interfaces where possible, so workload network changes can't cut you off.
  • Keep a break-glass local account and console access documented per site.
  • Consider immutable, image-based nodes where network config comes from the provisioning data, not manual edits (see Kubernetes Administration, lesson 10).

Recap

  • "Restart the network" on a remote node can drop hand-added routes, the loopback, DHCP/VIPs and CNI routes.
  • Recover through the BMC console: check links, addresses, routes (including lo), re-apply persistent config, then restart kubelet.
  • Prevent with persistent config in Git, tested out-of-band access, netplan try/timed rollback, and alerts.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.