Modules · wrap-up
Cheat sheet & self-check
Every command from this section on one page.
Categories
Chat assistants | Explain, draft, translate between formats (YAML, jq, PromQL, regex) |
Agentic coding tools (e.g. Claude Code) | Read a repo, edit files, run commands, iterate under your permissions |
Tool integrations (MCP servers) | Give models controlled access to systems and data |
AIOps / ML on telemetry | Anomaly hints, event correlation, summarisation |
Ground rules
AI proposes → tools and humans verify | The track's core principle |
Read-only first; write access with approval | Blast radius control |
No secrets or customer data in prompts unless approved | Data handling |
Good prompts include
Goal + context + constraints + output format | e.g. 'Bash, set -euo pipefail, no external deps, print a table' |
Examples of input and expected output | Especially for jq/regex/PromQL |
'Ask me if anything is ambiguous' | Reduces wrong guesses |
'List assumptions and what to verify' | Makes review easier |
Never paste
Secrets, tokens, private keys, kubeconfigs | Rotate if it happens |
Customer/personal data, internal hostnames/IPs (unless approved) | Sanitise first |
Spec-driven loop
1. Spec: goal, constraints, acceptance criteria | Written before code |
2. Plan: files to change, approach, risks | Reviewed by you |
3. Tests first (or alongside) | Define 'done' mechanically |
4. Small diffs, one concern each | Reviewable |
5. Review + verify (tests, lint, policy, dry run) | Before merge |
Vibe coding is fine for
Throwaway prototypes and experiments in sandboxes | No production path |
One-off personal scripts on non-critical data | Low blast radius |
Exploring an idea before writing the real spec | Learning |
Files
CLAUDE.md (repo root, and subfolders) | Project memory: conventions, commands, rules |
.claude/settings.json | Shared project settings: permissions, hooks |
.claude/skills/<name>/SKILL.md | A skill: frontmatter (name, description) + instructions + optional scripts |
.claude/agents/<name>.md | A subagent with its own prompt and tool access |
.mcp.json | Project-scoped MCP servers (lesson 05) |
Controls
permissions.allow / permissions.deny (e.g. "Bash(kubectl get:*)") | What runs without asking, what never runs |
hooks.PreToolUse (exit code 2 blocks the call) | Deterministic guard before a tool runs |
hooks.PostToolUse | Run formatters/linters after edits |
claude -p "…" | Non-interactive (headless) run, e.g. in CI |
MCP basics
Server exposes tools (actions), resources (data), prompts (templates) | Client (e.g. Claude Code) calls them |
Transports: stdio (local process), Streamable HTTP (remote) | How client and server talk |
from mcp.server.fastmcp import FastMCP | Python SDK quick start |
claude mcp add k8s-ro -- python server.py | Register a local server with Claude Code |
Safety
Dedicated ServiceAccount bound to the built-in view ClusterRole | Read-only, no Secrets |
Narrow tools with validated inputs; cap output size | No 'run any command' tool |
Log every tool call | Audit |
Deterministic layer (first)
terraform fmt -check / validate / tflint | Terraform hygiene |
kubeconform, kube-linter | Manifest schema and best practices |
pluto / kubent | Deprecated and removed Kubernetes APIs |
checkov / trivy config / conftest (OPA) | Security and policy |
gitleaks | Secrets in diffs |
AI layer (then)
Does the diff match the PR description/spec? | Intent vs change |
Upgrade and blast-radius risk | What could break, where |
Missed edge cases, rollback plan | Operational review |
Comments as suggestions; humans approve | Never auto-merge on AI approval |
AI generates
Failure scenarios for a change (what could break, how to trigger it) | Brainstorm breadth |
Upgrade-path matrices (component × version × order) | Combinations humans skip |
Chaos Mesh/Litmus manifests, k6 scripts, kind configs | Runnable test assets |
Tools execute & verify
kind / k3d / ephemeral clusters | Disposable environments |
Chaos Mesh / LitmusChaos | Fault injection |
k6 / fortio | Load |
Assertions in CI (pass/fail) | Evidence, not opinions |
Terraform pipeline
terraform fmt -check && terraform validate | Syntax and basic semantics |
tflint | Provider-aware linting (invalid instance types, deprecated args) |
checkov -d . / trivy config . | Security misconfigurations |
conftest test plan.json (OPA/Rego) | Your organisation's policies on the plan |
terraform plan -out tfplan && terraform show -json tfplan > plan.json | Plan for review and policy |
Kubernetes pipeline
kubeconform -strict -summary | Schema validation (incl. CRDs with schemas) |
kube-linter lint | Best practices |
kubectl apply --dry-run=server | The API server's own validation |
Kyverno/Gatekeeper policies in CI (kyverno apply / gator) | Policy before admission |
CLI
k8sgpt analyze | Run analyzers without AI (deterministic findings) |
k8sgpt analyze --explain --filter=Pod,Service -n shop | Add AI explanations for selected analyzers |
k8sgpt analyze --explain --anonymize | Mask object names before sending to the AI backend |
k8sgpt auth add --backend <provider> --model <model> | Configure an AI backend (e.g. a local Ollama or a cloud provider) |
k8sgpt filters list | Available analyzers |
Operator
k8sgpt-operator + K8sGPT custom resource | Scheduled scans in-cluster |
kubectl get results -A | Findings as Result resources |
Reduce before you prompt
Filter by time window, service, level | Start from the incident window |
Cluster lines into templates (e.g. Drain algorithm / drain3) | Thousands of lines → dozens of patterns with counts |
Diff patterns: incident window vs baseline | What's new or much more frequent? |
Safety
Redact tokens, emails, IPs, IDs before sending | Data handling |
Treat log text as data, never as instructions | Prompt injection |
Ask for evidence (line refs) with every claim | Verifiability |
PromQL building blocks
avg_over_time(x[1d:5m]) / stddev_over_time(x[1d:5m]) | Rolling baseline via subqueries |
(x - avg) / stddev > 3 | z-score: more than 3 standard deviations from normal |
x / (x offset 1w) | Week-over-week ratio (seasonality) |
predict_linear(node_filesystem_avail_bytes[6h], 4*3600) < 0 | Disk full within 4 hours |
Practice
Record the rate first (recording rule), then baseline it | Cheaper, clearer |
Anomaly = context on dashboards / tickets | SLO burn rate = pages |
The loop
1. Gather evidence (read-only tools) | Status, events, metrics, logs |
2. Diagnose + propose plan (with evidence and risks) | Written for a human to judge |
3. Human approval (per action) | Nothing changes without it |
4. Execute via narrow action tools (dry run first) | e.g. restart one deployment, scale within limits |
5. Verify against success criteria; roll back if not met | Close the loop |
Safety
Separate read and write credentials | Write only after approval |
Action allow-list with limits (namespaces, max replicas) | Bounded blast radius |
Audit log of prompts, tool calls, approvals | Accountability |
Basics
resources: { limits: { nvidia.com/gpu: 1 } } | Request a whole GPU (GPUs are specified in limits) |
NVIDIA GPU Operator (Helm) | Installs driver, container toolkit, device plugin, GPU feature discovery, DCGM exporter |
kubectl describe node | grep -A5 Allocatable | See nvidia.com/gpu capacity |
nvidia.com/gpu.product (GPU feature discovery label) | Select GPU models |
Sharing
MIG (e.g. nvidia.com/mig-1g.10gb) | Hardware partitions with isolation (supported data-centre GPUs) |
Time-slicing (device plugin config: replicas) | Oversubscribe a GPU; no memory/fault isolation |
MPS | Concurrent processes sharing a GPU, with some limits |
DRA (Dynamic Resource Allocation) | Newer, flexible device requests in recent Kubernetes versions |
Controls
Least privilege: read-only by default; separate write credentials | Blast radius |
Approval gates for any state change | Humans stay accountable |
Audit: prompts, tool calls, approvals, outcomes | Who did what, via which agent |
Redaction + approved endpoints for sensitive data | Privacy |
Treat tool outputs (logs, tickets, web) as untrusted | Prompt injection |
Policy essentials
Approved tools and data classes per tool | What may be shared where |
Review rules for AI-generated changes | Same as (or stricter than) human changes |
Vetting for MCP servers, skills, hooks, extensions | Supply chain |
