Scheduling in depth

How the scheduler decides

For every pod without a node, the scheduler runs two phases:

1. Filter: remove nodes that can't run the pod: not enough CPU or memory for its requests, a taint it doesn't tolerate, a nodeSelector or required affinity that doesn't match, a port conflict, a volume in another zone…
2. Score: rank the nodes that are left, using spreading, preferred affinity and resource balance. The highest score wins, and the pod is bound to that node.

If the filter phase leaves zero nodes, the pod stays Pending and an event explains why.

Steering pods towards nodes

nodeSelector: the simple, hard rule.

spec:
  nodeSelector:
    disktype: ssd

Node affinity: more expressive, with required (hard) and preferred (soft) rules.

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: kubernetes.io/arch
                operator: In
                values: ["amd64"]
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 50
          preference:
            matchExpressions:
              - key: disktype
                operator: In
                values: ["ssd"]

"IgnoredDuringExecution" means that if a node's labels change later, running pods are not moved.

Keeping pods away from nodes: taints and tolerations

A taint on a node repels pods. A toleration on a pod lets it ignore that taint.

$ kubectl taint node w1 dedicated=gpu:NoSchedule
node/w1 tainted

spec:
  tolerations:
    - key: dedicated
      operator: Equal
      value: gpu
      effect: NoSchedule

Your control-plane nodes already carry node-role.kubernetes.io/control-plane:NoSchedule, which is why normal pods never land there.

Pods and other pods: affinity and anti-affinity

Keep replicas of the same app on different nodes, so one node failure doesn't take them all:

spec:
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          podAffinityTerm:
            topologyKey: kubernetes.io/hostname
            labelSelector:
              matchLabels:
                app: web

topologyKey defines what counts as "the same place": a hostname, a zone, a rack label. Pod affinity (the opposite) co-locates pods, such as a cache next to its app.

Spreading evenly: topology spread constraints

The modern, more precise way to spread replicas:

spec:
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: ScheduleAnyway     # or DoNotSchedule for a hard rule
      labelSelector:
        matchLabels:
          app: web

maxSkew: 1 means the busiest zone may have at most one more matching pod than the emptiest.

Priority and preemption

When the cluster is full, what matters most? A PriorityClass answers that:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: business-critical
value: 100000
globalDefault: false
description: "Customer-facing services"

Pods with priorityClassName: business-critical are scheduled first, and if they can't fit, the scheduler may preempt (evict) lower-priority pods. Kubernetes' own critical add-ons use the built-in system-cluster-critical and system-node-critical classes.

Reading a Pending pod

$ kubectl describe pod web-7c9d8b6f5-x2kqz
...
Events:
  Type     Reason            Message
  ----     ------            -------
  Warning  FailedScheduling  0/3 nodes are available: 1 node(s) had untolerated taint
           {node-role.kubernetes.io/control-plane: }, 2 Insufficient memory.

Read it as a list: every node was filtered out, and each clause explains why. Here the control-plane node is tainted, and both workers lack the requested memory. The fixes are smaller requests, more nodes, or evicting something less important. Not removing the control-plane taint.

Recap

• Scheduling = filter (can it run here?) then score (where is best?).
• nodeSelector / node affinity attract pods to nodes; taints / tolerations let nodes repel pods; use both for dedicated nodes.
• Anti-affinity and topology spread protect you from single-node and single-zone failures.
• PriorityClasses decide who wins when capacity runs out.
• Pending → read the FailedScheduling event; it lists every reason.