Modules · wrap-up
Cheat sheet & self-check
Every command from this section on one page.
Where logs live
kubectl logs <pod> [-c container] [--previous] | Read logs via the API (the node's files) |
/var/log/pods/<ns>_<pod>_<uid>/<container>/0.log | Container log files on the node |
/var/log/containers/*.log | Symlinks with pod/namespace/container in the name (what collectors tail) |
kubelet: containerLogMaxSize / containerLogMaxFiles | Node-level log rotation |
The pipeline
Fluent Bit (DaemonSet) | Tail files, parse, add Kubernetes metadata, ship |
Elasticsearch | Store, index, search, aggregate |
Kibana | Discover, dashboards, alerting rules |
Cluster state
GET _cluster/health | green / yellow / red, unassigned shards |
GET _cat/nodes?v&h=name,node.role,heap.percent,disk.used_percent,cpu | Nodes at a glance |
GET _cat/indices?v&s=store.size:desc | Biggest indices |
GET _cat/shards?v&h=index,shard,prirep,state,node,store | Where shards live |
GET _cluster/allocation/explain | Why a shard is unassigned |
Data
GET logs-app-default/_mapping | Field types of an index or data stream |
GET _data_stream | Data streams and their backing indices |
PUT _index_template/logs-app {…} | Settings/mappings for new indices |
Install & inspect
helm install elastic-operator elastic/eck-operator -n elastic-system --create-namespace | Install ECK (Helm repo https://helm.elastic.co) |
kubectl get elasticsearch,kibana -A | Health and phase of managed clusters |
kubectl get secret logs-es-elastic-user -n logging -o go-template='{{.data.elastic | base64decode}}' | Password of the built-in elastic user |
kubectl port-forward svc/logs-es-http -n logging 9200 | Reach Elasticsearch (HTTPS, self-signed CA by default) |
kubectl logs -n elastic-system statefulset/elastic-operator | Operator logs |
Day 2
spec.version: 8.x → newer | Rolling upgrade performed by the operator |
nodeSets[].count | Scale a node group |
spec.secureSettings | Keystore entries (e.g. snapshot repository credentials) |
Fluent Bit pipeline
input → parser → filter → buffer → output | The data path |
tail: path /var/log/containers/*.log, multiline.parser cri, docker | Read container logs (CRI or Docker format) |
filter kubernetes: merge_log on | Add pod metadata; parse JSON log bodies into fields |
storage.type filesystem (+ service storage.path) | Buffer chunks on disk, not just memory |
output retry_limit no_limits (default is 1) | Keep retrying instead of dropping after one failure |
Debug
kubectl logs ds/fluent-bit -n logging | grep -iE 'error|warn|retry' | Delivery problems |
http_server on → curl :2020/api/v1/metrics/prometheus | Records in/out, retries, dropped |
outputs: - name: stdout, match: '*' | Temporarily print records to see what's parsed |
Parsing
merge_log on (kubernetes filter) | JSON log bodies become fields |
fluentbit.io/parser: nginx (pod annotation) | Pick a parser per pod (needs k8s-logging.parser on) |
fluentbit.io/exclude: "true" | Skip a pod's logs (needs k8s-logging.exclude on) |
filter multiline: multiline.parser java, python, go | Join stack traces |
Shaping & routing
filter grep: exclude log ^DEBUG | Drop matching records |
filter modify: remove / rename / add | Change fields |
filter rewrite_tag: $kubernetes['namespace_name'] ^(payments)$ audit.$TAG false | Re-tag records for different outputs |
outputs with different match + index | Different data streams per team or type |
KQL
kubernetes.namespace_name : "shop" | Field equals value |
log.level : (error or fatal) | Either value |
message : "payment declined" | Phrase in a text field |
kubernetes.pod_name : cart-* | Wildcard (keyword field) |
http.status >= 500 and not kubernetes.container_name : "istio-proxy" | Range + negation |
trace.id : * | Field exists |
ES|QL (recent versions)
FROM logs-* | WHERE log.level == "error" | STATS count = COUNT(*) BY service.name | SORT count DESC | Count errors per service |
Dashboard panels
Errors over time, split by service.name | Is something broken, and where? |
Top namespaces / pods by error count | Who's the noisiest right now? |
5xx rate from ingress/access logs | User impact |
Log volume per namespace | Spikes, and pipeline gaps (sudden zero) |
Saved search table: latest errors | Jump straight into the lines |
Alerting
Stack Management → Rules → Elasticsearch query | Alert when a query matches > N docs in a window |
Log threshold rule (Observability) | Count/ratio thresholds on log fields |
Connectors: Index, Server log (basic); Slack, email, webhook… (check subscription) | Where alerts go |
Lifecycle
PUT _ilm/policy/logs-30d {…} | Rollover in hot, delete after 30 days |
GET logs-k8s-default/_ilm/explain | Which phase/action each backing index is in |
GET _cat/allocation?v | Disk used per node |
cluster.routing.allocation.disk.watermark.low/high/flood_stage (85/90/95%) | Disk thresholds |
Snapshots
PUT _snapshot/s3-backups {"type":"s3","settings":{"bucket":"es-snapshots"}} | Register an S3 repository |
PUT _slm/policy/nightly {…} | Scheduled snapshots with retention |
POST _slm/policy/nightly/_execute | Run a policy now |
GET _snapshot/s3-backups/_all | List snapshots |
POST _snapshot/s3-backups/<snap>/_restore {…rename_pattern…} | Restore (to renamed indices) |
