Level 2 — Operator · wrap-up
Cheat sheet & self-check
Every command from this section on one page.
Node preparation
sudo swapoff -a | Disable swap now (also remove it from /etc/fstab) |
sudo modprobe overlay && sudo modprobe br_netfilter | Kernel modules containers and bridged traffic need |
containerd config default | sudo tee /etc/containerd/config.toml | Write a default containerd config (then set SystemdCgroup = true) |
sudo apt-mark hold kubelet kubeadm kubectl | Stop unattended upgrades changing versions |
Build the cluster
sudo kubeadm init --control-plane-endpoint <LB-or-DNS>:6443 --pod-network-cidr 10.244.0.0/16 --upload-certs | First control-plane node |
kubeadm token create --print-join-command | New worker join command (tokens expire after 24h) |
sudo kubeadm join <endpoint>:6443 --token … --discovery-token-ca-cert-hash sha256:… | Join a worker |
sudo kubeadm reset | Undo init/join on a node (then clean CNI config and iptables) |
Certificates
sudo kubeadm certs check-expiration | When each certificate expires |
sudo kubeadm certs renew all | Renew all kubeadm-managed certificates |
ls /etc/kubernetes/pki | Where the cluster PKI lives |
Image & boot
Base image: OS + containerd + kubelet/kubeadm/kubectl (pinned) + kernel settings | Everything that's the same on every node |
Role at boot: cloud-init user-data (init / join / join --control-plane) | One image, many roles |
DHCP reservation (MAC → IP) or static network-config | Predictable node IPs |
cloud-init clean && truncate -s0 /etc/machine-id | Make a template clonable (no duplicate identities) |
kubeadm automation
kubeadm token generate | Create a bootstrap token value in advance |
kubeadm init --config init.yaml --upload-certs | First control plane from a config file |
kubeadm init phase upload-certs --upload-certs | Re-upload certs for control-plane joins (the key expires after 2 h) |
kubeadm join --config join.yaml | Join as worker or control plane from a config file |
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | Compute the CA cert hash for discovery |
Install & configure
helm repo add metallb https://metallb.github.io/metallb && helm install metallb metallb/metallb -n metallb-system --create-namespace | Install MetalLB |
IPAddressPool (spec.addresses: [10.0.0.240-10.0.0.250]) | Which IPs MetalLB may hand out |
L2Advertisement (spec.ipAddressPools: [...]) | Announce those IPs with ARP/NDP |
BGPPeer + BGPAdvertisement | Announce them to routers with BGP |
Check
kubectl get svc -A | grep LoadBalancer | EXTERNAL-IP assigned? |
kubectl get ipaddresspools,l2advertisements,bgppeers -n metallb-system | MetalLB config |
kubectl logs -n metallb-system -l app.kubernetes.io/component=speaker | Which node announces what |
arping -I eth0 10.0.0.240 (from another host on the segment) | Who answers ARP for the VIP |
Before you start
kubectl version | Current client and server versions |
kubectl get nodes | Every node's kubelet version |
sudo kubeadm upgrade plan | What you can upgrade to, component by component |
kubectl get pdb -A | PodDisruptionBudgets that could block drains |
Control plane (first node)
sudo apt-mark unhold kubeadm && sudo apt-get install -y kubeadm='1.32.x-*' && sudo apt-mark hold kubeadm | Upgrade the kubeadm binary first |
sudo kubeadm upgrade apply v1.32.x | Upgrade control-plane components (first control-plane node only) |
sudo kubeadm upgrade node | Other control-plane nodes, and every worker |
Each node's kubelet
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data | Move workloads off safely |
sudo apt-get install -y kubelet='1.32.x-*' kubectl='1.32.x-*' | Upgrade kubelet and kubectl (unhold/hold around it) |
sudo systemctl daemon-reload && sudo systemctl restart kubelet | Restart with the new version |
kubectl uncordon <node> | Allow scheduling again |
Take and check a snapshot
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save /backup/etcd.db | Save a snapshot (kubeadm certificate paths) |
etcdutl snapshot status /backup/etcd.db -w table | Hash, revision, key count, size |
etcdctl … endpoint status -w table | Leader, DB size, raft term per member |
etcdctl … member list -w table | Members of the etcd cluster |
Restore (single control plane, kubeadm)
sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml /root/ | Stop the API server (no writes during restore) |
sudo etcdutl snapshot restore /backup/etcd.db --data-dir /var/lib/etcd-restored | Restore into a NEW data directory |
sudo vi /etc/kubernetes/manifests/etcd.yaml | Point the etcd-data hostPath at /var/lib/etcd-restored |
sudo mv /root/kube-apiserver.yaml /etc/kubernetes/manifests/ | Start the API server again |
Labels & taints
kubectl label node w1 disktype=ssd | Add a node label |
kubectl get nodes -L disktype,topology.kubernetes.io/zone | Show labels as columns |
kubectl taint node w1 dedicated=gpu:NoSchedule | Repel pods without a matching toleration |
kubectl taint node w1 dedicated=gpu:NoSchedule- | Remove that taint (trailing minus) |
kubectl describe node w1 | grep -A3 Taints | See a node's taints |
Why is it Pending?
kubectl describe pod <pod> | The FailedScheduling event lists each reason |
kubectl get events --field-selector reason=FailedScheduling | All scheduling failures |
kubectl describe node <node> | grep -A8 'Allocated resources' | How much CPU/memory is already requested |
Priority
kubectl get priorityclass | Available priorities (system-cluster-critical, …) |
Check permissions
kubectl auth can-i create deployments -n shop | Can I do this? |
kubectl auth can-i --list -n shop | Everything I can do in a namespace |
kubectl auth can-i get secrets -n shop --as=system:serviceaccount:shop:app | Test as a service account |
kubectl auth whoami | Who does the API server think I am? |
Create RBAC quickly
kubectl create role pod-reader --verb=get,list,watch --resource=pods -n shop | A namespaced Role |
kubectl create rolebinding read-pods --role=pod-reader --user=asha -n shop | Bind it to a user |
kubectl create clusterrolebinding ops-view --clusterrole=view --group=ops | Built-in read-only role, cluster-wide, for a group |
kubectl create serviceaccount app -n shop | An identity for a workload |
kubectl create token app -n shop --duration=1h | A short-lived token for that service account |
Inspect
kubectl get roles,rolebindings -n shop | Namespaced RBAC objects |
kubectl describe clusterrole edit | What a built-in role allows |
Scope it
kubectl get nodes | Is it one node or all of them? |
kubectl get pods -A -o wide | grep -v Running | Everything not Running, and where |
kubectl get events -A --sort-by=.lastTimestamp | tail -30 | What happened most recently |
Pods
kubectl describe pod <pod> | Events, state, last state, exit code |
kubectl logs <pod> --previous | Logs from the crashed container |
kubectl debug -it <pod> --image=busybox:1.36 --target=<container> | Ephemeral debug container sharing the pod's namespaces |
kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[*].lastState}' | Why the last run ended |
Nodes
kubectl describe node <node> | Conditions: Ready, MemoryPressure, DiskPressure, PIDPressure |
kubectl debug node/<node> -it --image=busybox:1.36 | Shell on the node (host filesystem at /host) |
systemctl status kubelet containerd | Are the node agents running? (on the node) |
journalctl -u kubelet --since '15 min ago' | Why the kubelet is unhappy (on the node) |
crictl ps -a | Containers the runtime knows about, even if the API is down |
