Resource management & QoS · Prasanta Kumar Behera

Requests decide where pods fit; limits decide what they may use. Learn how CPU and memory behave differently, the three QoS classes, node eviction, and how LimitRanges and ResourceQuotas keep teams fair.

Requests and limits: two different promises

resources:
  requests:            # "reserve this for me": used by the SCHEDULER
    cpu: 250m
    memory: 256Mi
  limits:              # "never let me use more than this": enforced by the KERNEL
    cpu: "1"
    memory: 512Mi

Requests are a reservation. The scheduler only places a pod where the node's allocatable capacity minus other pods' requests still fits it.
Limits are a ceiling, enforced on the node through cgroups.

A request is booking a seat on a bus: the bus company won't sell more seats than it has. A limit is the size of your bag: you can bring anything up to that size, but not bigger. Most people don't fill their bag, and that's fine. But if the company sold seats based on what people usually carry, one day everyone brings a full bag and the bus can't close its doors.

CPU and memory behave differently

	CPU	Memory
Type	Compressible: can be slowed down	Incompressible: can't be taken back
Over the limit	Throttled (runs slower)	OOMKilled (exit code 137)
No limit set	May use idle CPU on the node	May grow until the node runs short, then risks eviction
Unit examples	`100m`, `0.5`, `2`	`128Mi`, `1Gi`

The three QoS classes

Kubernetes assigns each pod a Quality of Service class from its requests and limits:

QoS class	Rule	Under node memory pressure
Guaranteed	Every container: requests = limits, for CPU and memory	Evicted last
Burstable	Some requests or limits set, but not Guaranteed	Evicted when using more than requested
BestEffort	No requests or limits at all	Evicted first

$ kubectl get pod web-7f9c5d6b8d-4hx9t -o jsonpath='{.status.qosClass}'
Burstable

Node allocatable and eviction

A node doesn't give pods its whole capacity:

allocatable = capacity − kube-reserved − system-reserved − eviction threshold

When free memory or disk drops below the eviction threshold, the kubelet evicts pods: BestEffort first, then Burstable pods using the most above their requests. That's why a node with no requests set at all behaves unpredictably under pressure.

Seeing real usage

kubectl top needs metrics-server. On kind it also needs one extra flag, because kind's kubelet certificates are self-signed:

$ kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
$ kubectl -n kube-system patch deployment metrics-server --type=json \
    -p '[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]'
$ kubectl top pods --sort-by=memory
NAME                   CPU(cores)   MEMORY(bytes)
web-7f9c5d6b8d-4hx9t   1m           3Mi

(--kubelet-insecure-tls is for labs only. Real clusters should use properly signed kubelet serving certificates.)

Guard-rails per namespace

LimitRange: defaults and bounds for every container in a namespace, so forgotten resources: blocks still get sensible values:

apiVersion: v1
kind: LimitRange
metadata:
  name: defaults
  namespace: team-a
spec:
  limits:
    - type: Container
      defaultRequest: { cpu: 100m, memory: 128Mi }
      default: { memory: 256Mi }            # default limit
      max: { cpu: "2", memory: 2Gi }

ResourceQuota: a total budget for the namespace:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: budget
  namespace: team-a
spec:
  hard:
    requests.cpu: "8"
    requests.memory: 16Gi
    limits.memory: 24Gi
    pods: "50"

With a quota on CPU or memory, every new pod must declare those resources (a LimitRange default counts), otherwise the API server rejects it.

Try it: throttling, OOMKill and eviction order

Install metrics-server as above.
Apply the memory hog below (32Mi limit) and watch it die: kubectl get pod hog -w → OOMKilled, and kubectl describe pod hog shows exit code 137.
Create three pods (BestEffort, Burstable, Guaranteed) and check each .status.qosClass.
Add the ResourceQuota above to a namespace and try creating a pod without resources. Read the rejection message.

# hog.yaml: a string that doubles in size until the memory limit is hit
apiVersion: v1
kind: Pod
metadata:
  name: hog
spec:
  restartPolicy: Never
  containers:
    - name: hog
      image: busybox:1.36
      command: ["sh", "-c", "x=a; while true; do x=\"$x$x\"; done"]
      resources:
        limits:
          memory: 32Mi

Going deeper: sizing for real

Right-size from data: set requests near a high percentile of real usage (the VPA's recommender, lesson 24, does this) and review regularly. Idle requested capacity is the biggest hidden cost in most clusters.
Memory: set a limit, often equal to the request, so behaviour is predictable and the node isn't overcommitted.
CPU limits are a trade-off: they protect neighbours but cause throttling, which hurts latency-sensitive apps even when the node is idle. Many teams set CPU requests everywhere and CPU limits only where isolation matters. Measure throttling (container_cpu_cfs_throttled_periods_total) before deciding.
Reserve resources for the system (kubeReserved, systemReserved in the kubelet config). A node whose kubelet starves is worse than a full node.

Recap

Requests = scheduling reservation; limits = runtime ceiling.
CPU over the limit → throttled; memory over the limit → OOMKilled (137).
QoS: Guaranteed > Burstable > BestEffort, in eviction protection.
LimitRange for per-container defaults, ResourceQuota for namespace totals.
Size from measured usage, and watch CPU throttling.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.