Prasanta Kumar Behera
Incident Handling — On-Call Playbook & Real Scenarios›08 · The noisy neighbour
Learning Hub / Incident Handling / Incident Handling — On-Call Playbook & Real Scenarios

Lesson 08 of 19 · Real-world incident scenarios

The noisy neighbour

Several services on the same nodes slow down or get evicted because one workload is eating CPU, memory, disk or network. Find the culprit fast, contain it, and use requests, limits, quotas and isolation so one tenant can't starve the rest.

Practitioner → Advanced
Key wordsnoisy neighbourrequests and limitsQoS classesCPU throttlingmemory pressureevictionsephemeral storageLimitRangeResourceQuotadedicated nodes

The page

Tuesday 10:40 — latency alerts for three unrelated services, all on nodes in the general pool. Two pods of the payments API were evicted with The node was low on resource: memory. A data team deployed a new analytics job an hour ago.

First five minutes

  • Impact: which user-facing services are slow or failing? Is it one node or many?
  • Scope: do the affected pods share nodes? (kubectl get pods -o wide for each service)
  • What changed: new or scaled workloads on those nodes in the last hours.

A shared flat with one fridge and one bathroom. Everyone gets along until one flatmate fills the whole fridge and takes two-hour showers. Nobody else can cook or wash. The fix is house rules (requests and limits), a shelf per person (quotas), and sometimes giving the big eater their own fridge (dedicated nodes).

Find the culprit

$ kubectl get pods -A -o wide | grep -E 'Evicted|OOMKilled'
$ kubectl top pods -A --sort-by=memory | head
NAMESPACE   NAME                        CPU(cores)   MEMORY(bytes)
analytics   nightly-agg-7d4f9-x8k2p     3100m        11800Mi
payments    api-5c7b8-2kq4m             180m         420Mi
$ kubectl get pod nightly-agg-7d4f9-x8k2p -n analytics -o jsonpath='{.spec.containers[*].resources}'
{}
$ kubectl describe node gen-07 | sed -n '/Conditions/,/Allocated resources/p'
  MemoryPressure   True    ...   KubeletHasInsufficientMemory

Empty resources means BestEffort (no requests, no limits): the scheduler reserved nothing, and the pod can use everything. Other contended resources to check: disk IO (iostat -x, pidstat -d), ephemeral storage (a pod filling the node disk, lesson 16), and network bandwidth.

Contain

  • Scale the hog down, or cordon the affected nodes and move it to a node where it can't hurt anyone.
  • If it must keep running, give it requests and limits now and redeploy.
  • Restart evicted critical pods if they haven't rescheduled.

Fix properly

resources:
  requests: { cpu: "2", memory: 8Gi }     # what the scheduler reserves
  limits:   { memory: 10Gi }              # hard cap (OOM kill beyond this); CPU limit optional
  • Every container gets requests; memory limits too.
  • LimitRange gives defaults to pods that forget; ResourceQuota caps each namespace's total.
  • QoS: critical services as Guaranteed or Burstable with realistic requests; batch as Burstable/BestEffort on separate nodes.
  • Isolation: taint a batch node pool; give heavy jobs tolerations and node affinity.
  • PriorityClasses so critical pods win when resources are short.

(Deep dive: Kubernetes Administration, lesson 23; tenancy models in lesson 29.)

Prevent

  • Admission policy: reject pods without requests/limits in shared namespaces (Kyverno/Gatekeeper).
  • Alerts on node pressure conditions, evictions, OOM kills and CPU throttling.
  • Capacity reviews: requests vs allocatable per node pool.

Try it: make and tame a noisy neighbour

  1. Deploy a small web app with requests on a two-node lab cluster.
  2. Deploy a stress-ng pod without resources on the same node (--vm 1 --vm-bytes 90%) and watch latency, pressure conditions and evictions.
  3. Find it with kubectl top and describe node.
  4. Add a LimitRange and ResourceQuota to its namespace and redeploy; what changes?
  5. Taint a node for batch work and move the stress pod there.

Going deeper: fairness at scale

  • CPU limits can cause throttling latency; many teams set CPU requests only for latency-sensitive services and rely on requests for fairness.
  • Memory is not compressible: memory limits and accurate requests matter most.
  • For IO-heavy tenants, separate disks or node pools; Kubernetes doesn't limit disk IO per pod by default.

Recap

  • Symptoms across unrelated services on the same nodes point to a noisy neighbour.
  • Find it with top, describe node, node-level tools; look for BestEffort pods.
  • Contain now (scale/cordon/move); fix with requests, limits, LimitRange, ResourceQuota, isolation, priorities.
  • Prevent with admission policies and pressure/eviction/throttling alerts.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.