Performance tuning · Prasanta Kumar Behera

Find and fix the bottlenecks of a busy cluster: etcd disk latency, API server load and fairness, expensive clients, kubelet settings, and the documented limits of a single cluster.

Where clusters get slow

In a healthy small cluster, nothing here matters. As clusters grow (more nodes, more pods, more controllers, more CI jobs calling the API), four places run out of headroom first:

etcd disks: every write waits for an fsync.
The API server: request volume, and expensive LISTs.
Clients: controllers and scripts that poll instead of watching.
Nodes: kubelet settings, image pulls, pod density.

A library gets slow in predictable places. The librarian's notebook (etcd) is slow if they write with a blunt pencil (a slow disk). The front desk (API server) jams if one person keeps asking for every book in the building (a big LIST). A fair desk serves people in turns (Priority and Fairness), so one loud visitor can't block everyone.

1. etcd: disks first

etcd writes every change to its write-ahead log and fsyncs it before acknowledging. Slow storage = slow cluster.

Put etcd on fast SSD/NVMe, ideally a dedicated disk (not shared with container images and logs).
Watch etcd_disk_wal_fsync_duration_seconds and etcd_disk_backend_commit_duration_seconds. A p99 fsync consistently above ~10 ms is a warning sign.
Keep the database small: compaction (automatic in Kubernetes), periodic defragmentation, and don't store large blobs in ConfigMaps or custom resources.
Stay under the backend quota (2 GiB by default; raised with --quota-backend-bytes). Hitting it stops all writes.

2. API server: fairness and expensive calls

API Priority and Fairness (APF) is on by default. It sorts requests into priority levels (via FlowSchemas) and queues them fairly, so system components keep working even when a tenant's script floods the API:

$ kubectl get prioritylevelconfigurations
NAME              TYPE      NOMINALCONCURRENCYSHARES   QUEUES
catch-all         Limited   5                          <none>
exempt            Exempt    <none>                     <none>
global-default    Limited   20                         128
leader-election   Limited   10                         16
node-high         Limited   40                         64
system            Limited   30                         64
workload-high     Limited   40                         128
workload-low      Limited   100                        128

(Columns trimmed; defaults vary by version.) Rising apiserver_flowcontrol_rejected_requests_total means some clients are being throttled: find out which, and why.

The expensive calls are LISTs, especially cluster-wide, without label selectors, repeated often. One CI job running kubectl get pods -A -o json every few seconds across thousands of pods can visibly load a control plane.

3. Clients: watch, don't poll

Pattern	Cost
Informer (list once, then watch for changes)	✅ Cheap: how controllers should work
Periodic full LIST	❌ Expensive, grows with the cluster
GET per object in a loop	❌ Many requests, slow
Label/field selectors on LIST	✅ Less data to serialise and send

If you build tools or operators (see Python for Infrastructure), use the client library's informer/watch support.

4. Nodes: kubelet settings that matter

The kubelet's configuration lives in /var/lib/kubelet/config.yaml on kubeadm nodes (a KubeletConfiguration):

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
maxPods: 110                      # the default
kubeReserved: { cpu: 200m, memory: 500Mi }
systemReserved: { cpu: 200m, memory: 500Mi }
evictionHard:
  memory.available: "200Mi"
  nodefs.available: "10%"
imageGCHighThresholdPercent: 85   # start deleting unused images at 85% disk
imageGCLowThresholdPercent: 80    # …down to 80%
serializeImagePulls: true         # pull one image at a time (the default)

Reserve resources for the system and kubelet, so a busy node degrades gracefully instead of freezing.
Pod density: raising maxPods also needs enough pod IPs from your CNI and more memory per node. Measure before you raise it.
Image pulls: pre-pull or cache large images (a registry mirror) and consider serializeImagePulls: false on nodes that start many pods at once.

Know the documented limits

Upstream Kubernetes is tested up to roughly 5,000 nodes, 150,000 pods, and 110 pods per node in a single cluster, under specific conditions. Most organisations hit operational pain long before that. Several medium clusters are usually easier to run than one giant one (see Multi-cluster & fleet management, lesson 28).

Try it: see the API's view of load

kubectl get --raw /metrics | grep -E '^apiserver_request_total' | sort -t' ' -k2 -n | tail: which verbs and resources are busiest?
Run a tight loop while true; do kubectl get pods -A > /dev/null; done in one terminal for a minute, then check again. See how one client shows up.
kubectl get flowschemas: find which schema your user's requests fall into (kubectl get --raw /metrics | grep apiserver_flowcontrol_dispatched_requests_total).
On a kind node, docker exec lab-worker cat /var/lib/kubelet/config.yaml and find maxPods and the eviction settings (defaults may not be listed explicitly).

Going deeper: tuning in production

Measure first: dashboards for API latency by verb and resource, etcd fsync and commit latency, APF queueing and rejections, and kubelet PLEG/runtime latency.
Big clusters benefit from dedicated control-plane nodes sized for peak load, and from separating etcd events into their own etcd cluster (--etcd-servers-overrides) when events dominate writes.
Networking at scale: IPVS or eBPF dataplanes handle thousands of Services better than long iptables chains; NodeLocal DNSCache takes load off CoreDNS.
Before raising a flag, ask whether the load should exist at all. A misbehaving controller is fixed in the controller, not with a bigger API server.

Recap

etcd disk latency is the first thing to check, and the first thing to get right.
APF shares API capacity fairly; rejected requests point at noisy clients.
Watch, don't poll: frequent full LISTs are the classic self-inflicted load.
Tune the kubelet (reservations, eviction, image GC, maxPods) deliberately, and prefer several medium clusters over one giant one.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.