Incident Handling — On-Call Playbook & Real Scenarios›10 · Docker network collides with the site network

Lesson 10 of 19 · Real-world incident scenarios

Docker network collides with the site network

At a customer site, a newly installed edge server can reach its own subnet but not the customer's NTP, DNS or registry servers. The culprit: Docker's default bridge (172.17.0.0/16) or a pod/service CIDR overlapping the site's network. Find the overlapping route, move Docker and Kubernetes ranges, and plan per-site IP ranges.

Practitioner → Advanced
Key wordsdocker0bridge network172.17.0.0/16IP address overlaprouting conflictdaemon.jsonbipdefault-address-poolspod CIDRedge sites
edge server (at the customer site) eth0 172.17.20.15/24 customer LAN, gateway 172.17.20.1 docker0 172.17.0.1/16 Docker's default bridge routing table 172.17.0.0/16 dev docker0 172.17.20.0/24 dev eth0 Customer servers 172.17.5.40 (ERP, NTP, DNS…) outside eth0's /24 Result replies to 172.17.5.40 go to docker0, not the LAN: 'host unreachable', NTP/DNS/registry fail at the site fix: move Docker (bip, default-address-pools) and pod/service CIDRs off site ranges
Docker's default bridge claims 172.17.0.0/16, which the customer site already uses.

The page

Day one at a customer's factory. The edge server boots, gets its IP (172.17.20.15/24) and the cluster comes up, but pods can't pull from the customer's local registry, time sync fails, and the customer's DNS servers time out. From a laptop on the same LAN, everything works. The customer's services live in 172.17.5.0/24.

First five minutes

  • Impact: the site can't finish provisioning; workloads that need local services fail.
  • Scope: this site only, other sites fine → something site-specific (network ranges, firewall, VLANs).
  • Test from the host itself, not only from pods.

Two streets in town are both called "Station Road". The postman at the new house always delivers letters for "Station Road" to the little lane behind the house (docker0), because that's the Station Road he knows best. Letters for the real Station Road across town never arrive. The fix: rename the lane (move Docker's range) so there's no confusion.

Diagnose

$ ip -br addr
eth0      UP   172.17.20.15/24
docker0   DOWN 172.17.0.1/16
$ ip route
default via 172.17.20.1 dev eth0
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
172.17.20.0/24 dev eth0 proto kernel scope link src 172.17.20.15
$ ip route get 172.17.5.40
172.17.5.40 dev docker0 src 172.17.0.1                 # wrong interface!

The /16 on docker0 is more specific than the default route, so anything in 172.17.x.x outside eth0's /24 is sent to Docker's bridge. (Even with linkdown, the route can still capture the traffic, depending on kernel settings.)

The same class of problem appears with user-defined Docker networks (172.18.0.0/16, 172.19…, created by tools like kind or docker compose), and with Kubernetes pod or Service CIDRs that overlap the site network (e.g. site uses 10.244.x.x).

Fix

Move Docker off the site's ranges:

{
  "bip": "192.168.254.1/24",
  "default-address-pools": [
    { "base": "192.168.240.0/20", "size": 24 }
  ]
}
$ sudo cp /etc/docker/daemon.json /etc/docker/daemon.json.bak 2>/dev/null; sudoedit /etc/docker/daemon.json
$ sudo systemctl restart docker
$ ip route get 172.17.5.40
172.17.5.40 via 172.17.20.1 dev eth0 src 172.17.20.15     # correct now

Existing user-defined networks keep their old subnets until they're recreated (docker network rm, then recreate, or docker compose down && up).

If the pod or Service CIDR overlaps, the fix is harder: those ranges are fixed at cluster creation, so the cluster usually has to be rebuilt with new ranges. That's why planning matters.

Verify

  • ip route get for each customer service IP shows dev eth0.
  • chronyc sources, DNS lookups and registry pulls succeed from the host and from pods.

Prevent

  • Site intake: collect every range the customer uses (LAN, VPN, OT networks) before shipping.
  • Per-site templates that set Docker bip/pools, pod and Service CIDRs away from customer ranges, with an overlap check in CI (see Cluster Design — Architect Track, lesson 06).
  • Bake a safe daemon.json into the node image; on hosts that don't need Docker (containerd-only Kubernetes nodes), don't install it.
  • Provisioning tools that run temporary Docker/kind clusters (admin machines, bootstrap clusters) need the same care.

Try it: reproduce the collision

  1. On a Linux VM, add a secondary IP in 172.17.20.0/24 and a dummy "customer" host route (or use two VMs on a 172.17.x network).
  2. Install Docker with defaults; run ip route get to an address in 172.17.5.0/24 and see it choose docker0.
  3. Set bip and default-address-pools, restart Docker and re-test.
  4. Create a kind cluster and note which Docker network range it uses; move it too.
  5. Write a site intake checklist question list for network ranges.

Going deeper: address planning at the edge

  • Reserve a platform range that you document to customers ("we use 100.64.0.0/16 internally", for example the shared address space) and check it against their networks.
  • Remember VPN and management networks too; they collide as often as LANs.
  • Keep a site network record in Git with the ranges used, so support engineers can check quickly.

Recap

  • Symptom: host reaches its own subnet but not other customer ranges → check ip route for overlaps.
  • Docker's docker0 (172.17.0.0/16) and user networks often collide; pod/Service CIDRs can too.
  • Fix Docker with bip + default-address-pools; overlapping cluster CIDRs usually mean a rebuild.
  • Prevent with site intake, per-site templates and CI overlap checks.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.