Ethernet, ARP & MTU · Prasanta Kumar Behera

The bottom of the stack: frames and MAC addresses, how ARP finds neighbours, VLANs and bonded links on servers, and MTU, the cause of the famous 'small requests work, big ones hang' problem in overlay networks.

Frames, MACs and switches

On a local network, data travels in Ethernet frames addressed by MAC address (a hardware address like 52:54:00:12:34:56). Switches learn which MAC lives on which port and forward frames accordingly. IP addresses matter only once traffic leaves the local segment.

Your IP address is like your home address, useful for letters from far away. Your MAC address is like your name on the classroom register. Inside the classroom, the teacher calls names, not addresses. ARP is shouting "Who is sitting at desk 20?" and someone answering "Me, I'm Sam."

ARP: from IP to MAC

$ ip neigh
10.10.0.1 dev eth0 lladdr 52:54:00:aa:bb:01 REACHABLE
10.10.0.30 dev eth0 lladdr 52:54:00:aa:bb:30 STALE
$ sudo tcpdump -eni eth0 arp
10:20:01 52:54:00:12:34:56 > ff:ff:ff:ff:ff:ff, ARP, Request who-has 10.10.0.30 tell 10.10.0.21
10:20:01 52:54:00:aa:bb:30 > 52:54:00:12:34:56, ARP, Reply 10.10.0.30 is-at 52:54:00:aa:bb:30

FAILED or INCOMPLETE neighbours mean nobody answered: wrong VLAN, host down, or a layer-2 problem. Load balancers such as MetalLB in L2 mode and kube-vip answer ARP for virtual IPs; that's how a floating IP "moves" between nodes (lesson 08).

VLANs and bonding on servers

VLANs (802.1Q) split one physical network into several isolated ones by tagging frames:

$ sudo ip link add link eth0 name eth0.100 type vlan id 100
$ sudo ip addr add 10.100.0.21/24 dev eth0.100 && sudo ip link set eth0.100 up

Bonding combines NICs for redundancy or throughput:

Mode	Name	Needs switch support	Behaviour
1	active-backup	No	One link active; the other takes over on failure
4	802.3ad (LACP)	Yes (LACP on the switch)	All links active; traffic hashed per flow

$ cat /proc/net/bonding/bond0 | grep -E 'Mode|Slave Interface|MII Status'
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
MII Status: up
Slave Interface: eno1
MII Status: up
Slave Interface: eno2
MII Status: up

On servers these are usually configured with netplan (Ubuntu) or NetworkManager (RHEL family), not by hand.

MTU: the silent killer

MTU (maximum transmission unit) is the largest packet an interface sends: usually 1500 bytes on Ethernet, often 9000 ("jumbo frames") in data centres.

Overlay networks wrap each pod packet in an extra header to cross the node network:

Encapsulation	Extra bytes (IPv4)	Pod MTU on a 1500 network
None (native routing, e.g. BGP)	0	1500
IP-in-IP	20	1480
VXLAN	50	1450
WireGuard encryption	~60–80	~1420

If the pod MTU isn't reduced accordingly, full-size packets become too big. Normally the sender gets an ICMP "fragmentation needed" message and shrinks them (path MTU discovery). If a firewall drops that ICMP, packets vanish: a black hole.

Classic symptom: TCP connects, small requests work, large responses (a big API reply, TLS handshakes with large certificate chains, image pulls) hang.

$ ping -M do -s 1472 10.10.0.30        # 1472 + 8 (ICMP) + 20 (IP) = 1500
PING 10.10.0.30 1472(1500) bytes of data.
ping: local error: message too long, mtu=1450
$ tracepath 10.10.0.30
 1?: [LOCALHOST]                      pmtu 1450

Where to set it

Most CNIs detect or let you set the MTU (Calico, Cilium and Flannel all have a setting). Set it once, correctly, for the whole cluster, and remember VPNs and cloud networks may already be below 1500.

Try it: see layer 2 and MTU

Run ip neigh, then sudo tcpdump -eni eth0 arp while you ping a neighbour whose entry you just deleted (sudo ip neigh flush dev eth0).
Find your default interface's MTU. Use ping -M do -s to find the largest payload that gets through to your gateway.
In your kind cluster, compare the MTU of a pod's eth0 (kubectl exec … -- cat /sys/class/net/eth0/mtu) with the node's.
Create a VLAN interface on a lab VM (it won't reach anything without a trunked switch port, but inspect it with ip -d link).

Going deeper: layer 2 in platforms

Jumbo frames help storage and east-west traffic, but every device on the path must agree. Mixed MTUs cause the same black holes.
LACP hashing is per flow: a single TCP connection uses one link, so bonding doesn't make one transfer faster.
L2-based VIPs (MetalLB L2, kube-vip ARP) fail over by sending gratuitous ARP; some networks rate-limit or ignore it, which slows failover. BGP modes avoid this.
SR-IOV gives pods direct access to NIC virtual functions for high-performance workloads (see Linux, lesson 15).

Recap

Frames use MAC addresses; ARP maps IP → MAC on the local segment (ip neigh).
VLANs isolate by tagging; bonding gives redundancy (active-backup) or aggregation (LACP).
MTU minus encapsulation overhead = pod MTU. Mismatches cause "small works, big hangs".
Test with ping -M do -s and tracepath; set the CNI MTU once, correctly.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.