Lesson 10 of 19 · Real-world incident scenarios
Docker network collides with the site network
At a customer site, a newly installed edge server can reach its own subnet but not the customer's NTP, DNS or registry servers. The culprit: Docker's default bridge (172.17.0.0/16) or a pod/service CIDR overlapping the site's network. Find the overlapping route, move Docker and Kubernetes ranges, and plan per-site IP ranges.
The page
Day one at a customer's factory. The edge server boots, gets its IP (172.17.20.15/24) and the cluster comes up, but pods can't pull from the customer's local registry, time sync fails, and the customer's DNS servers time out. From a laptop on the same LAN, everything works. The customer's services live in 172.17.5.0/24.
First five minutes
- Impact: the site can't finish provisioning; workloads that need local services fail.
- Scope: this site only, other sites fine → something site-specific (network ranges, firewall, VLANs).
- Test from the host itself, not only from pods.
Two streets in town are both called "Station Road". The postman at the new house always delivers letters for "Station Road" to the little lane behind the house (docker0), because that's the Station Road he knows best. Letters for the real Station Road across town never arrive. The fix: rename the lane (move Docker's range) so there's no confusion.
Diagnose
$ ip -br addr
eth0 UP 172.17.20.15/24
docker0 DOWN 172.17.0.1/16
$ ip route
default via 172.17.20.1 dev eth0
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
172.17.20.0/24 dev eth0 proto kernel scope link src 172.17.20.15
$ ip route get 172.17.5.40
172.17.5.40 dev docker0 src 172.17.0.1 # wrong interface!
The /16 on docker0 is more specific than the default route, so anything in 172.17.x.x outside eth0's /24 is sent to Docker's bridge. (Even with linkdown, the route can still capture the traffic, depending on kernel settings.)
The same class of problem appears with user-defined Docker networks (172.18.0.0/16, 172.19…, created by tools like kind or docker compose), and with Kubernetes pod or Service CIDRs that overlap the site network (e.g. site uses 10.244.x.x).
Fix
Move Docker off the site's ranges:
{
"bip": "192.168.254.1/24",
"default-address-pools": [
{ "base": "192.168.240.0/20", "size": 24 }
]
}
$ sudo cp /etc/docker/daemon.json /etc/docker/daemon.json.bak 2>/dev/null; sudoedit /etc/docker/daemon.json
$ sudo systemctl restart docker
$ ip route get 172.17.5.40
172.17.5.40 via 172.17.20.1 dev eth0 src 172.17.20.15 # correct now
Existing user-defined networks keep their old subnets until they're recreated (docker network rm, then recreate, or docker compose down && up).
If the pod or Service CIDR overlaps, the fix is harder: those ranges are fixed at cluster creation, so the cluster usually has to be rebuilt with new ranges. That's why planning matters.
Verify
ip route getfor each customer service IP showsdev eth0.chronyc sources, DNS lookups and registry pulls succeed from the host and from pods.
Prevent
- Site intake: collect every range the customer uses (LAN, VPN, OT networks) before shipping.
- Per-site templates that set Docker
bip/pools, pod and Service CIDRs away from customer ranges, with an overlap check in CI (see Cluster Design — Architect Track, lesson 06). - Bake a safe
daemon.jsoninto the node image; on hosts that don't need Docker (containerd-only Kubernetes nodes), don't install it. - Provisioning tools that run temporary Docker/kind clusters (admin machines, bootstrap clusters) need the same care.
Try it: reproduce the collision
- On a Linux VM, add a secondary IP in 172.17.20.0/24 and a dummy "customer" host route (or use two VMs on a 172.17.x network).
- Install Docker with defaults; run
ip route getto an address in 172.17.5.0/24 and see it choose docker0. - Set
bipanddefault-address-pools, restart Docker and re-test. - Create a kind cluster and note which Docker network range it uses; move it too.
- Write a site intake checklist question list for network ranges.
Going deeper: address planning at the edge
- Reserve a platform range that you document to customers ("we use 100.64.0.0/16 internally", for example the shared address space) and check it against their networks.
- Remember VPN and management networks too; they collide as often as LANs.
- Keep a site network record in Git with the ranges used, so support engineers can check quickly.
Recap
- Symptom: host reaches its own subnet but not other customer ranges → check ip route for overlaps.
- Docker's docker0 (172.17.0.0/16) and user networks often collide; pod/Service CIDRs can too.
- Fix Docker with bip + default-address-pools; overlapping cluster CIDRs usually mean a rebuild.
- Prevent with site intake, per-site templates and CI overlap checks.
This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.