Lesson 05 of 12 · Kubernetes Dataplane
CNI plugins compared
Compare the CNI plugins you'll meet (Flannel, Calico, Cilium, cloud-native CNIs and Multus) by how they move packets, enforce policy and manage IPs, and learn how to choose one for a new cluster.
What a CNI must decide
Every CNI plugin answers the same questions differently:
- How do packets cross nodes? Encapsulate (overlay) or route natively?
- How are IPs assigned? Per-node ranges, a cluster pool, or the cloud's network?
- How is policy enforced? iptables/nftables, eBPF, or not at all?
- What else? Encryption, observability, kube-proxy replacement, multi-cluster.
Different delivery companies get parcels between towns in different ways. Some put your parcel inside a bigger box addressed to the depot in the other town (overlay). Others make sure every road sign already knows your street (routed). Some also check every parcel against a list of what's allowed (policy), and some keep a video log of every delivery (observability).
The common choices
| CNI | Dataplane | Cross-node | NetworkPolicy | Notable |
|---|---|---|---|---|
| Flannel | Linux bridge + routes | VXLAN overlay (or host-gw) | ❌ on its own | Simplest; often paired with a policy engine (e.g. "Canal" with Calico) |
| Calico | iptables/nftables, or eBPF | BGP routed, or IPIP/VXLAN overlay | ✅ plus Calico policies (global, DNS) | BGP peering with physical routers; mature; runs on many platforms |
| Cilium | eBPF | Overlay (VXLAN/Geneve) or native routing | ✅ plus L7 and FQDN policies | kube-proxy replacement, Hubble flow logs, WireGuard/IPsec, cluster mesh |
| Cloud CNIs (e.g. AWS VPC CNI) | Cloud network | Pods get VPC IPs | Via add-ons or network policy agents | No overlay; IP capacity depends on subnets and instance types |
| Multus | Meta-plugin | — | — | Multiple interfaces per pod (telecom, storage, SR-IOV) |
| kindnet | Simple routes | Routed within kind | Basic, depending on version | For kind lab clusters only |
Overlay vs routed
| Overlay (VXLAN/IPIP/Geneve) | Routed (BGP, native) | |
|---|---|---|
| Works on any underlay | ✅ | Needs the network to carry pod routes (or the same L2 segment) |
| MTU overhead | Yes (lesson 01) | None |
| Pod IPs visible outside the cluster | No (NAT at the node) | Can be, which helps some integrations and troubleshooting |
| Typical use | Clouds, simple on-prem | Data centres with network team cooperation |
IP address management (IPAM)
- Per-node CIDR (e.g.
/24per node from10.244.0.0/16): simple; watch node count × range size. - IP pools/blocks (Calico, Cilium cluster-pool): allocate small blocks to nodes on demand.
- Cloud-assigned (AWS VPC CNI): pods consume real subnet IPs, so plan subnets generously or use prefix delegation and secondary CIDRs (see Amazon EKS in Production).
Plan pod and Service CIDRs so they don't overlap with your corporate networks, VPNs or other clusters you may connect later.
How to choose
| If you need… | Consider |
|---|---|
| A simple lab or small cluster | Flannel or the distribution's default |
| Solid policy and BGP to physical routers | Calico |
| eBPF performance, L7/FQDN policy, flow visibility | Cilium |
| Pods as first-class VPC citizens on AWS | AWS VPC CNI (plus a policy solution) |
| Extra interfaces for telecom or storage traffic | Multus + SR-IOV/macvlan alongside the primary CNI |
Changing CNI on a running cluster is a migration project (every pod must be re-networked). Choose deliberately, and record the decision in an ADR.
Try it: two CNIs, same test
- Create a kind cluster with Calico (
disableDefaultCNI: true, then Calico's manifest), deploy two pods on different nodes, and ping between them. Check a node's routes (ip route) for the other nodes' pod blocks. - Create a second kind cluster with Cilium (install the
ciliumCLI, thencilium install), and runcilium statusandcilium connectivity test. - On the Cilium cluster, apply a default-deny NetworkPolicy and watch denied flows with Hubble (
cilium hubble enable, thenhubble observe --verdict DROPPED, using the hubble CLI). - Compare the pod MTU in both clusters.
Going deeper: CNI operations
- CNI agents are node-critical DaemonSets: upgrade them carefully (one node at a time, following the vendor's guide), and alert when an agent isn't Ready. New pods on that node won't get networking.
- eBPF dataplanes depend on kernel features: check the CNI's kernel requirements against your node OS.
- Encrypting pod traffic (WireGuard/IPsec) costs CPU and MTU. Measure before enabling cluster-wide.
- For edge and air-gapped sites, pre-load the CNI images and pin versions in your cluster templates.
Recap
- CNIs differ in encapsulation (overlay vs routed), IPAM, policy enforcement and extras.
- Flannel = simple; Calico = policy + BGP; Cilium = eBPF + L7 policy + observability; cloud CNIs = VPC IPs; Multus = extra interfaces.
- Plan CIDRs and MTU up front; switching CNI later is a migration.
This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.