NUMA, huge pages & CPU pinning

When "a CPU is a CPU" stops being true

For most applications, the scheduler's choices are fine. But for latency-sensitive workloads (packet processing at millions of packets per second, trading systems, real-time audio, some databases) three things matter:

1. NUMA: which memory is close to which CPU.
2. Huge pages: how memory is mapped.
3. CPU pinning: whether the workload keeps "its" CPUs, undisturbed.

NUMA: Non-Uniform Memory Access

$ numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 16 17 18 19 20 21 22 23
node 0 size: 128000 MB
node 1 cpus: 8 9 10 11 12 13 14 15 24 25 26 27 28 29 30 31
node 1 size: 128000 MB
node distances:
node   0   1
  0:  10  21
  1:  21  10

Distance 21 vs 10: reaching the other node's memory costs roughly twice as much as local memory. Keep a latency-sensitive process's CPUs and memory on one node:

$ numactl --cpunodebind=0 --membind=0 ./packet-engine

Network cards and GPUs are attached to a NUMA node too (cat /sys/class/net/eth0/device/numa_node). Put the workload on the same node as its NIC.

Huge pages

Memory is managed in pages, normally 4 KiB. Every access needs a virtual-to-physical translation, cached in the CPU's TLB. With gigabytes of memory in 4 KiB pages, the TLB misses constantly. Huge pages (2 MiB, or 1 GiB on supporting CPUs) cover far more memory per entry.

$ sudo sysctl -w vm.nr_hugepages=512          # 512 × 2 MiB = 1 GiB, reserved now
$ grep -i huge /proc/meminfo
HugePages_Total:     512
HugePages_Free:      512
Hugepagesize:       2048 kB

For 1 GiB pages, or to reserve pages reliably before memory fragments, use boot parameters: default_hugepagesz=1G hugepagesz=1G hugepages=16.

Transparent Huge Pages (THP) is a different, automatic mechanism. Some databases recommend setting it to madvise or never because background compaction can cause latency spikes. Follow your database vendor's guidance:

$ cat /sys/kernel/mm/transparent_hugepage/enabled
always [madvise] never

CPU pinning and isolation

$ taskset -c 2,3 ./worker          # run only on CPUs 2 and 3
$ taskset -cp 4821                 # show the CPUs an existing process may use

Pinning alone doesn't stop other work (other processes, interrupts, kernel threads) from running on those CPUs. Isolation techniques include:

• Boot parameters such as isolcpus= or nohz_full= (remove CPUs from general scheduling or reduce timer ticks). Use them carefully; they change system behaviour significantly.
• IRQ affinity: keep interrupts off the isolated CPUs (/proc/irq/*/smp_affinity_list, irqbalance settings).
• Hyper-threading siblings: two logical CPUs sharing one core interfere with each other; latency-critical setups often allocate full cores.

How Kubernetes exposes this

The kubelet has managers for exactly these needs (configured in the KubeletConfiguration):

A pod that gets two exclusive CPUs and 1 GiB of 2 MiB huge pages:

apiVersion: v1
kind: Pod
metadata:
  name: packet-engine
spec:
  containers:
    - name: engine
      image: registry.example.com/packet-engine:1.0
      resources:
        requests: { cpu: "2", memory: 1Gi, hugepages-2Mi: 1Gi }
        limits:   { cpu: "2", memory: 1Gi, hugepages-2Mi: 1Gi }   # requests = limits → Guaranteed
      volumeMounts:
        - { name: hugepages, mountPath: /hugepages }
  volumes:
    - name: hugepages
      emptyDir: { medium: HugePages }

The node must have huge pages pre-allocated and, for exclusive CPUs, the CPU Manager static policy enabled. Changing the CPU Manager policy on an existing node requires draining it and resetting the kubelet's CPU manager state, so plan it as node configuration from day one.

Recap

• NUMA: keep CPUs, memory and devices on the same node for latency-sensitive work (numactl).
• Huge pages reduce TLB pressure; reserve them via sysctl or boot parameters; handle THP per vendor guidance.
• Pinning (taskset) plus isolation (boot parameters, IRQ affinity) protect critical CPUs.
• Kubernetes: CPU Manager static, Topology Manager, Memory Manager, and hugepages-* resources.