Modules · wrap-up
Cheat sheet & self-check
Every command from this section on one page.
Install & inspect
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts | Chart repository |
helm install kps prometheus-community/kube-prometheus-stack -n monitoring --create-namespace | Install the stack |
kubectl get prometheus,alertmanager,servicemonitors,podmonitors,prometheusrules -A | Operator resources |
kubectl port-forward -n monitoring svc/kps-kube-prometheus-stack-prometheus 9090 | Prometheus UI (service name depends on release name) |
PromQL patterns
sum by (job) (rate(http_requests_total[5m])) | Requests/s per job |
sum(rate(http_requests_total{code=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) | Error ratio |
histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) | p95 latency |
up == 0 | Targets that can't be scraped |
Find the load
prometheus_tsdb_head_series | Active series in the head block |
topk(10, count by (__name__) ({__name__=~".+"})) | Metrics with the most series (expensive query) |
Status → TSDB Status (web UI) | Top metrics, labels and label values by series count |
promtool tsdb analyze /prometheus | Cardinality analysis of a data directory |
Stop it
metric_relabel_configs: action: labeldrop / drop | Drop a label or a whole metric at scrape time |
ServiceMonitor spec.sampleLimit | Fail a scrape that returns too many samples |
Prometheus CR spec.enforcedSampleLimit | A global per-scrape cap |
Bucket setup (mc client)
mc alias set lab https://s3.internal.example ACCESS_KEY SECRET_KEY | Configure a target |
mc mb lab/thanos | Create a bucket |
mc ls --recursive lab/thanos | head | See uploaded blocks |
mc admin info lab | Server/drive health (MinIO) |
Client configs
Thanos objstore: type: S3, config: { bucket, endpoint, access_key, secret_key, insecure } | Thanos bucket config |
Mimir: common.storage.backend: s3 (+ endpoint, bucket names per component) | Mimir storage config |
Components
thanos sidecar | Next to Prometheus: uploads 2h blocks, serves recent data via StoreAPI |
thanos store | Serves historical blocks from the bucket via StoreAPI |
thanos query --endpoint=<sidecar/store>:10901 --query.replica-label=prometheus_replica | Global PromQL with dedup |
thanos query-frontend | Splits, caches and retries queries in front of the querier |
thanos receive | Accepts remote-write instead of using sidecars |
Prometheus Operator
prometheusSpec.thanos.objectStorageConfig (Secret ref) | Enable the sidecar with upload |
prometheusSpec.externalLabels: { cluster: prod-eu-1 } | Identify the source of every series |
prometheusSpec.replicas: 2 | HA pair; operator adds prometheus_replica label |
Compactor
thanos compact --wait --objstore.config-file=objstore.yml --data-dir=/var/thanos/compact | Run continuously (one per bucket) |
--retention.resolution-raw=30d --retention.resolution-5m=180d --retention.resolution-1h=2y | Retention per resolution |
--deduplication.replica-label=prometheus_replica | Vertical compaction: dedup HA replicas in storage |
thanos_compact_halted == 1 | Alert: compactor stopped (e.g. overlapping blocks) |
Inspect
thanos tools bucket inspect --objstore.config-file=objstore.yml | List blocks, resolutions, time ranges |
thanos tools bucket web --objstore.config-file=objstore.yml | Visual block timeline |
Techniques
Recording rules → keep aggregates long, raw short | e.g. job:http_requests:rate5m kept for years |
metricRelabelings action: drop (by __name__) | Never store unused metrics |
Separate Prometheus/tenant per priority tier | Different retention per destination |
Mimir per-tenant compactor_blocks_retention_period | Retention override per tenant |
Find unused metrics
Grafana dashboards + rules → list of used metric names | What's actually queried |
mimirtool analyze grafana / prometheus / ruler | Mimir's usage analysis tooling |
Query logs (Prometheus query_log_file, query-frontend logs) | Ad-hoc usage |
Write path
remote_write url: http://<mimir-gateway>/api/v1/push | Prometheus → Mimir |
headers: { X-Scope-OrgID: team-a } | Tenant ID on every request (when multi-tenancy is on) |
distributor → ingesters (replication factor 3) → blocks to object storage | Where samples go |
Operate
helm install mimir grafana/mimir-distributed -n mimir --create-namespace -f values.yaml | Install (Helm repo https://grafana.github.io/helm-charts) |
-target=all (monolithic) vs per-component targets | Deployment modes |
runtime config overrides: ingestion_rate, max_global_series_per_user | Per-tenant limits |
mimirtool | Rules, alertmanager config, analysis and remote-read tools |
Quick contrast
Thanos: sidecar + querier fan-out (or Receive) | Add-on to existing Prometheus; incremental adoption |
Mimir: remote-write into a central cluster | Central, multi-tenant service |
Downsampling: Thanos yes; Mimir no | Long-range query speed vs raw fidelity |
Licence: Thanos Apache-2.0; Mimir AGPLv3 | Check your organisation's policy |
Federation
scrape job with metrics_path: /federate and params: { match[]: [ '{__name__=~"job:.*"}' ] } | Pull aggregated series from another Prometheus |
honor_labels: true | Keep the source's labels |
Remote write & agents
prometheus --agent (older: --enable-feature=agent) | Scrape + WAL + remote-write only; no local queries or rules |
kind: PrometheusAgent (Prometheus Operator) | Agent mode as a custom resource |
writeRelabelConfigs | Choose which series to send |
prometheus_remote_storage_samples_pending / _failed_total | Remote-write health |
Back-of-envelope formulas
samples/s = active series ÷ scrape interval (s) | 500k ÷ 30 s ≈ 16.7k samples/s |
bytes/day ≈ samples/s × 86,400 × ~1–2 bytes | ≈ 1.5–3 GB/day compressed (before replicas) |
memory: measure bytes per active series on your version | Plan headroom (2× for spikes and queries) |
Must-haves
HA Prometheus pairs (or agents) + dedup | No single scraper |
Alertmanager cluster (3 replicas) | Alerts survive a pod loss |
Meta-monitoring + Watchdog to an external receiver | Know when monitoring is down |
