Level 4 — Production · wrap-up
Cheat sheet & self-check
Every command from this section on one page.
Join more control-plane nodes
sudo kubeadm init phase upload-certs --upload-certs | Re-upload control-plane certs; prints a new certificate key (valid 2 hours) |
kubeadm token create --print-join-command | Base join command (append --control-plane --certificate-key <key>) |
sudo kubeadm join <VIP>:6443 --token … --discovery-token-ca-cert-hash … --control-plane --certificate-key … | Join as a control-plane node |
Check HA health
kubectl get nodes -l node-role.kubernetes.io/control-plane | All control-plane nodes |
kubectl get lease -n kube-system kube-scheduler kube-controller-manager | Which node currently holds each leader lease |
etcdctl … endpoint status --cluster -w table | etcd members, leader and DB size |
curl -k https://<VIP>:6443/readyz | Is the API reachable through the load balancer? |
See usage vs requests
kubectl top nodes | Real CPU/memory usage per node (needs metrics-server) |
kubectl top pods -A --sort-by=memory | Hungriest pods |
kubectl describe node <node> | grep -A10 'Allocated resources' | Sum of requests and limits on a node |
kubectl get pod <pod> -o jsonpath='{.status.qosClass}' | A pod's QoS class |
Namespace guard-rails
kubectl get limitrange,resourcequota -n <ns> | Defaults and totals for a namespace |
kubectl describe resourcequota -n <ns> | Used vs hard limits |
Units
cpu: 500m | Half a CPU core (1000m = 1 core) |
memory: 256Mi | 256 mebibytes (Mi/Gi are powers of 2; M/G are powers of 10) |
HPA
kubectl autoscale deployment web --cpu-percent=60 --min=2 --max=10 | Quick CPU-based HPA |
kubectl get hpa -w | Watch current vs target and replica count |
kubectl describe hpa web | Conditions and scaling events (why it did or didn't scale) |
Which autoscaler?
HPA | More or fewer pods, from CPU/memory/custom metrics |
VPA | Bigger or smaller requests per pod, from observed usage |
Cluster Autoscaler | More or fewer nodes in existing node groups when pods are Pending |
Karpenter | Launches right-sized nodes directly for Pending pods; consolidates |
KEDA | Scales on events (queue length, Kafka lag, cron), including to zero |
Fleet inventory
kubectl config get-contexts -o name | Every cluster in your kubeconfig |
kubectl --context <ctx> version -o json | jq -r .serverVersion.gitVersion | One cluster's server version |
kubectl --context <ctx> get nodes -o custom-columns=NAME:.metadata.name,KUBELET:.status.nodeInfo.kubeletVersion | Kubelet versions per node |
Pre-flight checks (per cluster)
kubectl get nodes | grep -v ' Ready' | Anything not Ready? |
kubectl get pods -A | grep -vE 'Running|Completed' | Unhealthy pods before you start |
kubectl get pdb -A | Budgets that could block drains |
kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis | Deprecated API usage |
sudo kubeadm certs check-expiration | Certificate expiry (kubeadm clusters) |
Health and latency metrics
etcd_disk_wal_fsync_duration_seconds | etcd write-ahead-log fsync latency (keep p99 low, ~10ms) |
etcd_disk_backend_commit_duration_seconds | etcd commit latency |
apiserver_request_duration_seconds | API latency by verb and resource |
apiserver_flowcontrol_rejected_requests_total | Requests rejected by API Priority and Fairness |
Inspect
kubectl get flowschemas,prioritylevelconfigurations | How the API server prioritises clients |
kubectl get --raw /metrics | grep apiserver_request_total | Request counts by verb/resource/code |
sudo cat /var/lib/kubelet/config.yaml | The kubelet's configuration (kubeadm nodes) |
Velero
velero backup create shop-$(date +%F) --include-namespaces shop | Back up one namespace (objects + volumes as configured) |
velero backup describe <backup> --details | What was captured, warnings and errors |
velero restore create --from-backup <backup> | Restore it |
velero schedule create daily-shop --schedule='0 2 * * *' --include-namespaces shop --ttl 720h | Nightly backup kept for 30 days |
velero backup-location get | Is the object storage target available? |
Words to agree on
RPO | Recovery Point Objective: how much data you can afford to lose (time) |
RTO | Recovery Time Objective: how long until service is back |
