Kernel, modules & sysctl · Prasanta Kumar Behera

What the kernel actually does, how to load and inspect modules, and how to read and tune kernel parameters with sysctl safely and persistently, including the ones Kubernetes nodes depend on.

What the kernel does

The kernel is the core of Linux. Everything else (shells, databases, the kubelet) is a user-space program that asks the kernel for help through system calls.

Kernel job	Examples
Processes	Scheduling on CPUs, creating processes (`clone`), signals
Memory	Virtual memory, page cache, the OOM killer
Filesystems & disks	ext4, XFS, block devices, I/O scheduling
Networking	TCP/IP, routing, netfilter (iptables/nftables), bridges
Isolation	cgroups and namespaces (lesson 14), which containers are built on
Security	Permissions, capabilities, SELinux/AppArmor (lesson 17)

The kernel is the engine room of a ship. Passengers (programs) never touch the engine. They ring a bell and ask: "more speed", "open this door", "send this message" (system calls). The engineers have dials they can adjust while sailing (sysctl) and spare parts they can plug in without stopping the ship (modules).

You can watch a program's system calls:

$ strace -c -f curl -s https://example.com > /dev/null
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 31.02    0.000812          11        71           mmap
 12.45    0.000326          10        31           read
  ...

Kernel modules

Many drivers and features are modules, loaded on demand:

$ lsmod | head -4
Module                  Size  Used by
br_netfilter           32768  0
bridge                311296  1 br_netfilter
overlay               151552  12
$ modinfo overlay | head -3
filename:       /lib/modules/6.8.0-45-generic/kernel/fs/overlayfs/overlay.ko.zst
alias:          fs-overlay
license:        GPL
$ sudo modprobe br_netfilter                                   # load now
$ echo br_netfilter | sudo tee /etc/modules-load.d/k8s.conf    # load at boot

overlay provides the layered filesystem container images use; br_netfilter sends bridged traffic through netfilter. Both appear in every Kubernetes node setup (see Kubernetes Administration, lesson 09).

Kernel parameters with sysctl

The kernel exposes thousands of tunables under /proc/sys/. sysctl reads and writes them:

$ sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 0
$ sudo sysctl -w net.ipv4.ip_forward=1          # now, until reboot
$ cat /proc/sys/net/ipv4/ip_forward             # the same value, as a file
1

Make it permanent with a file in /etc/sysctl.d/ (processed in name order):

# /etc/sysctl.d/90-k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1

$ sudo sysctl --system

Parameters worth knowing

Parameter	What it controls	Typical note
`net.ipv4.ip_forward`	Route packets between interfaces	Required on Kubernetes nodes and routers
`net.bridge.bridge-nf-call-iptables`	Bridged traffic through iptables	Kubernetes nodes (needs br_netfilter)
`vm.swappiness`	Eagerness to swap	Lower on servers with enough RAM
`fs.file-max`, `fs.inotify.max_user_watches`	Open files; file watches	Raised for busy servers, IDEs, log agents
`net.core.somaxconn`	Listen backlog	Busy web servers
`net.netfilter.nf_conntrack_max`	Tracked connections	High-connection NAT hosts, Kubernetes nodes
`kernel.pid_max`	Maximum PID	Hosts running very many processes

Change one thing at a time, and measure

Tuning advice copied from the internet is often outdated or meant for a different workload. Record the current value, change one parameter, measure the effect, and keep a comment in the sysctl file explaining why.

Boot parameters

Some settings must be given to the kernel at boot, on its command line:

$ cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-6.8.0-45-generic root=/dev/mapper/vg0-root ro quiet splash

On GRUB systems, add parameters to GRUB_CMDLINE_LINUX in /etc/default/grub, then sudo update-grub (Debian/Ubuntu) or sudo grub2-mkconfig -o /boot/grub2/grub.cfg (RHEL family), and reboot. Examples: isolcpus= and hugepages= (lesson 15).

Try it: modules and parameters

lsmod | wc -l: how many modules are loaded? Find what uses the bridge module (the "Used by" column).
Check net.ipv4.ip_forward, turn it on with sysctl -w, confirm via /proc/sys, reboot a VM, and see it revert.
Make it persistent with a file in /etc/sysctl.d/, apply with sysctl --system, and reboot again.
Run strace -c ls / and find which system call it makes most.

Going deeper: kernels on platform nodes

Kernel version matters for containers: cgroup v2, eBPF features (used by Cilium), and filesystem fixes all depend on it. Keep node kernels in your version inventory (see Kubernetes Administration, lesson 25).
Kubernetes can set a small set of namespaced sysctls per pod (securityContext.sysctls); safe ones are allowed by default, unsafe ones must be allow-listed on the kubelet.
Livepatch services patch some kernel vulnerabilities without a reboot, but a reboot is still how most kernel updates take effect. Plan rolling reboots like rolling upgrades.
/sys exposes devices and drivers (e.g. /sys/block/sda/queue/scheduler for the I/O scheduler); /proc exposes processes and kernel state.

Recap

The kernel manages processes, memory, filesystems, networking, isolation and security; programs use it through system calls.
Modules: lsmod, modinfo, modprobe, and /etc/modules-load.d/ for boot.
sysctl: -w for now, /etc/sysctl.d/*.conf + sysctl --system for good.
Boot-time settings go on the kernel command line via GRUB. Tune one thing at a time and measure.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.