Architecture & the control plane · Prasanta Kumar Behera

Meet every component of a Kubernetes cluster, follow one request from kubectl to a running container, and see what breaks when each piece fails.

The control plane (brain) and worker nodes (muscle). Everything talks to the API server.

The big picture

A Kubernetes cluster has two halves:

The control plane: the "brain" that stores what you want and decides how to get there.
The worker nodes: the "muscle" that actually runs your containers.

You never tell Kubernetes how to do something. You tell it what you want ("3 copies of this web app"), and it keeps working until reality matches. This is called desired state, and it's the most important idea in Kubernetes.

Kubernetes is a restaurant

You are the customer. You say "3 pizzas, please". You don't tell the chef how to knead dough.
The API server is the waiter at the front desk. Every order goes through them, and they check you're allowed to order.
etcd is the order book. If it's not written there, it didn't happen.
The scheduler is the head chef who decides which cook makes each pizza, based on who's free.
The controllers are managers walking around checking: "3 pizzas ordered, only 2 on the counter. Make another!"
The kubelet is the cook at each station, actually making pizzas.
The container runtime is the oven.

If a pizza falls on the floor, nobody asks you to re-order. A manager notices and a cook makes a new one. That's Kubernetes self-healing.

Control-plane components

Component	Job	If it's down
kube-apiserver	The only front door. Authenticates, authorises, validates, and stores objects in etcd.	`kubectl` fails. Running pods keep running.
etcd	Key-value database holding all cluster state.	API can't read/write. Nothing new can happen.
kube-scheduler	Picks a node for each new pod.	New pods stay `Pending`. Existing pods are fine.
kube-controller-manager	Runs the reconcile loops (Deployments, ReplicaSets, Nodes, Jobs…).	No self-healing, no scaling, no rollouts.
cloud-controller-manager	Talks to a cloud (load balancers, node lifecycle). Not present on kind or bare metal.	Cloud load balancers aren't created.

Node components

Component	Job
kubelet	The agent on every node. Watches for pods assigned to it, starts them, reports their health.
container runtime	Actually runs containers. Usually containerd (sometimes CRI-O).
kube-proxy	Programs network rules so Services work (more in lesson 04).
CNI plugin	Gives each pod an IP and connects pods across nodes (kind uses kindnet).

See it in your cluster

$ kubectl get pods -n kube-system -o wide
NAME                                        READY   STATUS    NODE
coredns-7db6d8ff4d-5xk2p                    1/1     Running   lab-control-plane
coredns-7db6d8ff4d-q9wzn                    1/1     Running   lab-control-plane
etcd-lab-control-plane                      1/1     Running   lab-control-plane
kindnet-7mv4c                               1/1     Running   lab-worker
kube-apiserver-lab-control-plane            1/1     Running   lab-control-plane
kube-controller-manager-lab-control-plane   1/1     Running   lab-control-plane
kube-proxy-2hrxl                            1/1     Running   lab-worker
kube-scheduler-lab-control-plane            1/1     Running   lab-control-plane

(Output trimmed; you'll see one kindnet and one kube-proxy per node.)

Notice the control-plane pods have the node name in their pod name. They're static pods: the kubelet starts them straight from files on disk, not from the API server. That's how the API server can run as a pod before the API server exists.

$ docker exec lab-control-plane ls /etc/kubernetes/manifests
etcd.yaml
kube-apiserver.yaml
kube-controller-manager.yaml
kube-scheduler.yaml

Follow one request, end to end

What really happens when you type this?

$ kubectl create deployment web --image=nginx:1.27 --replicas=2
deployment.apps/web created

kubectl → API server. kubectl sends a POST with a Deployment object.
API server checks who you are (authentication), whether you're allowed (RBAC authorization), and whether the object is valid (admission). Then it writes it to etcd.
Deployment controller sees a new Deployment and creates a ReplicaSet.
ReplicaSet controller sees it wants 2 pods and has 0, so it creates 2 Pod objects. They have no node yet.
Scheduler sees unscheduled pods, scores the nodes, and binds each pod to a node.
kubelet on that node sees a pod assigned to it and tells containerd to pull nginx:1.27 and start it. The CNI plugin gives the pod an IP.
kubelet reports status back to the API server: Running.

Watch it happen live. In one terminal:

$ kubectl get events --watch

In another, run the create deployment command above. You'll see Scheduled, Pulling, Pulled, Created, Started scroll by: steps 5 and 6 in real time.

Everything talks to the API server

Components never talk to each other directly. The scheduler doesn't call the kubelet; it writes a binding to the API server, and the kubelet watches the API server. This "hub and spoke" design is why Kubernetes is so extensible: your own tools can watch and act exactly like built-in controllers.

The reconcile loop

Every controller runs the same simple loop, forever:

observe  →  compare with desired state  →  act to close the gap  →  repeat

Prove it by deleting a pod and watching it come back:

$ kubectl get pods -l app=web
NAME                   READY   STATUS    RESTARTS   AGE
web-6d4b9c8f7b-7kq2m   1/1     Running   0          2m
web-6d4b9c8f7b-xw8zn   1/1     Running   0          2m
$ kubectl delete pod web-6d4b9c8f7b-7kq2m
pod "web-6d4b9c8f7b-7kq2m" deleted
$ kubectl get pods -l app=web
NAME                   READY   STATUS    RESTARTS   AGE
web-6d4b9c8f7b-p5v9d   1/1     Running   0          3s
web-6d4b9c8f7b-xw8zn   1/1     Running   0          2m

A brand-new pod (p5v9d, 3 seconds old) replaced the one you deleted.

Try it: switch off the scheduler

Let's see what "the scheduler is down" looks like, safely.

Move the scheduler's static pod manifest away. The kubelet stops it within seconds: docker exec lab-control-plane mv /etc/kubernetes/manifests/kube-scheduler.yaml /tmp/
Scale up: kubectl scale deployment web --replicas=4
Run kubectl get pods -l app=web. The two new pods are stuck in Pending. kubectl describe pod <name> shows no Scheduled event.
Notice the two existing pods are still Running. Nothing else broke.
Put it back: docker exec lab-control-plane mv /tmp/kube-scheduler.yaml /etc/kubernetes/manifests/. Within seconds the Pending pods are scheduled.

You now recognise a real production symptom: "new pods are Pending, nothing in the events" → check the scheduler.

Going deeper: what a senior engineer knows about each failure

API server: in HA clusters several API servers sit behind a load balancer. They're stateless, so scale them out.
etcd: needs a majority (quorum): 3 members tolerate 1 failure, 5 tolerate 2. Even numbers add risk without adding tolerance. etcd is sensitive to disk latency (fsync), so give it fast SSDs. Back it up with etcdctl snapshot save (lesson 13).
Controller manager and scheduler: run one per control-plane node, but only one is active, chosen by leader election through a Lease object (kubectl get lease -n kube-system).
kubelet down: the node goes NotReady after the node-monitor grace period (about a minute). Pods on it are evicted after the default 5-minute toleration, so plan maintenance with kubectl drain, not by pulling the plug.

Recap

Control plane = API server + etcd + scheduler + controller manager. Nodes = kubelet + container runtime + kube-proxy + CNI.
You declare desired state; controllers reconcile reality towards it, forever.
Everything goes through the API server, and only the API server touches etcd.
Knowing what breaks when each part fails is what turns "it's broken" into a two-minute diagnosis.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.