Modules · wrap-up
Cheat sheet & self-check
Every command from this section on one page.
CSI objects
kubectl get csidrivers | Installed CSI drivers |
kubectl get csinodes -o yaml | grep -A3 drivers | Which drivers run on which nodes (and volume limits) |
kubectl get volumeattachments | Which volumes are attached to which nodes |
kubectl get sc -o custom-columns=NAME:.metadata.name,PROVISIONER:.provisioner,EXPAND:.allowVolumeExpansion,BINDING:.volumeBindingMode | StorageClass summary |
Debug
kubectl describe pvc <pvc> | Provisioning events |
kubectl describe pod <pod> | sed -n '/Events/,$p' | Attach/mount events (FailedAttachVolume, FailedMount) |
kubectl logs -n <ns> <csi-controller-pod> -c csi-provisioner | Provisioner sidecar logs |
Install & check
sudo apt install -y open-iscsi nfs-common && sudo systemctl enable --now iscsid | Node prerequisites (Debian/Ubuntu) |
helm repo add longhorn https://charts.longhorn.io | Chart repository |
helm install longhorn longhorn/longhorn -n longhorn-system --create-namespace | Install |
kubectl -n longhorn-system get pods | Manager, driver, UI and engine pods |
Inspect
kubectl -n longhorn-system get volumes.longhorn.io | Longhorn volumes, state and robustness |
kubectl -n longhorn-system get replicas.longhorn.io -o wide | Where each replica lives |
kubectl -n longhorn-system port-forward svc/longhorn-frontend 8080:80 | Open the Longhorn UI |
Rook
kubectl -n rook-ceph get cephcluster | Cluster phase and health |
kubectl -n rook-ceph get pods -l app=rook-ceph-osd -o wide | OSD pods and their nodes |
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- bash | Open the Ceph toolbox |
Ceph (inside the toolbox)
ceph status | Overall health, MONs, OSDs, PGs, usage |
ceph health detail | What exactly is wrong |
ceph osd tree | OSDs by host/rack (the CRUSH hierarchy) |
ceph df | Capacity per pool |
ceph osd pool ls detail | Pools, replication size, rules |
CSI snapshots
kubectl get volumesnapshotclass | Snapshot classes (need the snapshot CRDs + controller) |
kubectl get volumesnapshot -n shop | Snapshots and readyToUse |
dataSource: {name: <snapshot>, kind: VolumeSnapshot, apiGroup: snapshot.storage.k8s.io} | In a new PVC: restore from a snapshot |
Velero
velero backup create shop-$(date +%F) --include-namespaces shop --snapshot-move-data | Back up a namespace, moving snapshot data to object storage |
velero backup create shop-fs --include-namespaces shop --default-volumes-to-fs-backup | Use file-system backup for volumes |
velero restore create --from-backup <backup> --namespace-mappings shop:shop-restore | Restore into a different namespace (safe test) |
velero backup describe <backup> --details | What was included, volume results |
fio recipes (run inside a pod on the volume under test)
fio --name=randread --rw=randread --bs=4k --iodepth=32 --size=2G --runtime=60 --time_based --direct=1 --filename=/data/fio.test | Random read IOPS |
fio --name=randwrite --rw=randwrite --bs=4k --iodepth=32 --size=2G --runtime=60 --time_based --direct=1 --filename=/data/fio.test | Random write IOPS |
fio --name=seqwrite --rw=write --bs=1M --iodepth=8 --size=4G --runtime=60 --time_based --direct=1 --filename=/data/fio.test | Sequential throughput |
fio --name=fsync --rw=write --ioengine=sync --fdatasync=1 --bs=2300 --size=22m --filename=/data/fio.test | Small writes each followed by fdatasync (etcd/DB style) |
Observe
iostat -xz 1 | Per-device utilisation, queue size and await |
kubectl top pods --containers | CPU of storage agents (replication costs CPU) |
StatefulSets
kubectl get sts,pods,pvc -l app=db | Pods db-0, db-1… and their PVCs data-db-0, data-db-1… |
kubectl rollout status sts/db | Rolling updates go one pod at a time, highest ordinal first |
kubectl scale sts db --replicas=5 | Scaling adds db-3, db-4 in order |
Operators (examples)
kubectl get clusters.postgresql.cnpg.io -A | CloudNativePG clusters |
kubectl cnpg status <cluster> -n <ns> | Primary, replicas, replication lag (cnpg plugin) |
Capacity metrics (kubelet)
kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes | PVC fill ratio |
kubelet_volume_stats_inodes_used / kubelet_volume_stats_inodes | PVC inode usage |
predict_linear(kubelet_volume_stats_available_bytes[6h], 24*3600) < 0 | Will this PVC fill within 24 h? |
Housekeeping
kubectl get pv | grep Released | Released volumes waiting for a decision (Retain) |
kubectl get pvc -A | grep -v Bound | Claims that aren't bound |
kubectl patch pv <pv> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}' | Protect an important PV from deletion |
