IP routing, NAT & conntrack · Prasanta Kumar Behera

How Linux decides where packets go, how NAT rewrites addresses (the heart of Kubernetes Services and pod egress), and how connection tracking works, including the 'table full' failure that drops new connections under load.

Routing: where does this packet go?

For every packet, Linux looks up the destination in its routing table and picks the most specific matching route (longest prefix match).

$ ip route
default via 10.10.0.1 dev eth0 proto dhcp src 10.10.0.21 metric 100
10.10.0.0/24 dev eth0 proto kernel scope link src 10.10.0.21
10.244.1.0/24 via 10.10.0.22 dev eth0        # pods on node w2 (a CNI route)
10.244.2.0/24 via 10.10.0.23 dev eth0        # pods on node w3
$ ip route get 10.244.2.17
10.244.2.17 via 10.10.0.23 dev eth0 src 10.10.0.21

Many CNIs give each node a pod CIDR and install routes like these: "pods 10.244.2.0/24 live behind node 10.10.0.23" (via BGP, or directly in the kernel).

Routing is a postal sorting office. Each letter's address is checked against a list of rules: "anything for Elm Street goes to van 3", "anything for the whole town of Elmsbury goes to van 7". If both match, the more precise rule wins (Elm Street). Anything unknown goes to the main depot (the default route).

Policy routing (ip rule) lets Linux choose a different table based on source address, marks or interface. CNIs and VPNs use it for things like "pod traffic leaves via a specific interface".

NAT: rewriting addresses

Kind	Rewrites	Kubernetes example
DNAT	Destination	ClusterIP:port → a pod IP:port (Services)
SNAT / masquerade	Source	Pod IP → node IP when pods talk to the outside world

A request to a Service with iptables-based kube-proxy:

pod A 10.244.1.5 → 10.96.142.17:80      (ClusterIP)
   DNAT → 10.244.2.9:80                  (one of the backend pods)
reply 10.244.2.9 → 10.244.1.5
   reverse NAT → looks like it came from 10.96.142.17:80

The chains are readable (if long):

$ sudo iptables -t nat -S KUBE-SERVICES | grep web
-A KUBE-SERVICES -d 10.96.142.17/32 -p tcp -m comment --comment "default/web cluster IP" -m tcp --dport 80 -j KUBE-SVC-…
$ sudo iptables -t nat -S KUBE-SVC-… 
-A KUBE-SVC-… -m statistic --mode random --probability 0.33333 -j KUBE-SEP-…

Random-probability rules spread new connections across endpoints. Newer kube-proxy versions also offer an nftables mode, and eBPF dataplanes (Cilium) replace these chains entirely (lesson 06).

Connection tracking

NAT only works if the kernel remembers each connection's rewrite, so replies can be rewritten back. That memory is conntrack:

$ sudo conntrack -L -p tcp --dport 80 | head -2
tcp 6 86392 ESTABLISHED src=10.244.1.5 dst=10.96.142.17 sport=51722 dport=80 src=10.244.2.9 dst=10.244.1.5 sport=80 dport=51722 [ASSURED] mark=0 use=1
$ sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max
net.netfilter.nf_conntrack_count = 48213
net.netfilter.nf_conntrack_max = 262144

When count reaches max, the kernel logs nf_conntrack: table full, dropping packet and new connections fail: an outage that looks like random network flakiness under load.

Classic conntrack problems

Table full on busy nodes (lots of short connections, DNS over UDP): raise nf_conntrack_max (kube-proxy also sets it based on CPU count), and reduce needless connections (keep-alive, connection pooling).
UDP DNS races: parallel A/AAAA lookups over the same socket can hit conntrack insert races, causing 5-second DNS delays. NodeLocal DNSCache (lesson 07) avoids most of it.
Stale entries after endpoint changes can briefly send traffic to deleted pods, especially UDP.

Try it: follow a Service through NAT (kind)

Deploy web (3 replicas) and its Service; note the ClusterIP and the pod IPs.
On a node (docker exec -it lab-worker bash), find the Service's KUBE-SVC chain and its KUBE-SEP endpoint chains in iptables -t nat -S. Match each SEP to a pod IP. (If your kind version uses nftables mode for kube-proxy, look in nft list ruleset instead.)
From a pod, run wget -qO- http://web a few times, then list conntrack entries for the ClusterIP on that pod's node (install conntrack in the node container with apt-get install -y conntrack if missing).
Check nf_conntrack_count and nf_conntrack_max on the node.

Going deeper: routing and NAT in production

Source IP preservation: SNAT hides the client IP from pods. externalTrafficPolicy: Local on LoadBalancer/NodePort Services keeps it (at the cost of uneven load), and some dataplanes offer DSR (direct server return).
Egress IPs: partners often allow-list your IPs; CNIs offer egress gateways so specific pods leave via fixed IPs.
Watch conntrack -S counters (insert_failed, drop) and nf_conntrack_count in node monitoring.
BGP-based CNIs (Calico, Cilium) can advertise pod and Service routes to your physical routers, removing NAT and overlays entirely in data centres.

Recap

Routing uses longest prefix match; ip route get shows the decision; ip rule adds policy routing.
DNAT implements Services; SNAT/masquerade lets pods reach the outside.
conntrack remembers each translation; a full table drops new connections, so size and monitor it.
iptables → nftables → eBPF: same ideas, different machinery.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.