kube-prometheus-stack baseline · Prasanta Kumar Behera

The standard starting point for Kubernetes metrics: what kube-prometheus-stack installs, how the Prometheus Operator's ServiceMonitors and PrometheusRules work (and the label-selector trap), Alertmanager routing, and the PromQL patterns you'll use every day.

What the stack installs

Component	Job
Prometheus Operator	Manages Prometheus/Alertmanager from custom resources
Prometheus	Scrapes and stores metrics, evaluates rules
Alertmanager	Groups, deduplicates, silences and routes alerts
Grafana	Dashboards (many included)
node-exporter (DaemonSet)	Node CPU, memory, disk, network
kube-state-metrics	Kubernetes object state
Default rules and dashboards	From the kubernetes-mixin and others

$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
$ helm install kps prometheus-community/kube-prometheus-stack -n monitoring --create-namespace

Pin the chart version and read the upgrade notes: major chart versions often require CRD updates that Helm doesn't apply automatically.

Prometheus is a nurse doing rounds: every 30 seconds she visits each patient (target), writes down temperature and heart rate (metrics) in a notebook (TSDB), and rings the bell (Alertmanager) when numbers look bad. The Operator is the ward manager who hands the nurse her list of patients (ServiceMonitors) and the rules for when to ring (PrometheusRules).

ServiceMonitors and PodMonitors

Instead of editing prometheus.yml, you declare what to scrape:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: cart
  namespace: shop
  labels:
    release: kps                   # matches the chart's default selector
spec:
  selector:
    matchLabels:
      app: cart                    # the Service's labels
  endpoints:
    - port: metrics                # the Service port NAME
      interval: 30s
      path: /metrics

The selector trap: the Prometheus resource only picks up ServiceMonitors that match its serviceMonitorSelector and serviceMonitorNamespaceSelector. The chart defaults to the release: <release-name> label. Either add that label, or set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues: false (and the same for PodMonitors/rules) to select everything. Check Status → Targets in the Prometheus UI.

Rules and Alertmanager

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: cart-rules
  namespace: shop
  labels:
    release: kps
spec:
  groups:
    - name: cart
      rules:
        - record: job:http_requests:rate5m
          expr: sum by (job) (rate(http_requests_total{job="cart"}[5m]))
        - alert: CartHighErrorRatio
          expr: |
            sum(rate(http_requests_total{job="cart", code=~"5.."}[5m]))
              / sum(rate(http_requests_total{job="cart"}[5m])) > 0.05
          for: 10m
          labels: { severity: warning }
          annotations:
            summary: "cart 5xx ratio above 5% for 10 minutes"

Alertmanager routes alerts by labels to receivers (Slack, PagerDuty, email, webhooks), groups related alerts, and supports silences and inhibition (e.g. suppress pod alerts when the whole node is down). Configure it via chart values or AlertmanagerConfig resources per namespace. (SLO-based alerting is covered in Observability with OpenTelemetry, lesson 06.)

PromQL you'll use daily

sum by (job) (rate(http_requests_total[5m]))                                   # throughput
sum(rate(http_requests_total{code=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))  # error ratio
histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))  # p95
kube_deployment_status_replicas_available / kube_deployment_spec_replicas < 1  # degraded deployments
increase(kube_pod_container_status_restarts_total[1h]) > 3                     # restart loops

Rules of thumb: rate before sum, keep le when aggregating histograms, and use a range at least 4× the scrape interval.

Try it: a baseline on kind

Install kube-prometheus-stack on kind and open Prometheus, Alertmanager and Grafana (port-forward).
Deploy an app exposing /metrics (e.g. podinfo), with a Service that has a named metrics/http port.
Create a ServiceMonitor without the release label and confirm it's ignored; add the label and see the target appear.
Add the recording and alert rules above (adapt metric names); lower the threshold to make the alert fire and see it in Alertmanager.
Explore the bundled dashboards: node, namespace and pod resource usage.

Going deeper: baseline hygiene

Run two Prometheus replicas (the Operator supports replicas: 2) for HA; both scrape everything, and you'll dedupe later with Thanos or Mimir.
Review the default rules: disable ones that don't apply (e.g. control-plane components you can't scrape on managed Kubernetes) so alerts stay meaningful.
Set retention and resources explicitly; the defaults are for trying things out.
The always-firing Watchdog alert is a dead man's switch: route it to an external service that alerts if it stops arriving.

Recap

kube-prometheus-stack = Operator + Prometheus + Alertmanager + Grafana + node-exporter + kube-state-metrics + default rules.
Declare scraping with ServiceMonitors/PodMonitors; mind the selector labels.
PrometheusRules hold recording and alerting rules; Alertmanager routes, groups, silences.
PromQL basics: rate then sum, histogram_quantile with le, ratios for errors.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.