Observability stack with SLO alerts · Prasanta Kumar Behera

Instrument and watch an application like a platform team would: install kube-prometheus-stack, scrape podinfo with a ServiceMonitor, generate traffic and errors, build a RED dashboard, and fire an SLO burn-rate alert through Alertmanager to a receiver you can see.

podinfo → ServiceMonitor → Prometheus rules → Alertmanager and Grafana.

The brief

"You've deployed an app. How do you know it's healthy? Set up monitoring, a dashboard and an alert that actually pages when users are hurt."

You fit the car with a speedometer and warning lights (metrics and dashboards), decide what "too many problems" means (the SLO), and wire a buzzer that goes off only when problems pile up fast enough to matter (burn-rate alert). Then you test the buzzer by causing a few problems on purpose.

Resources needed

Resource	Detail
Any cluster	kind is fine (4+ GB RAM for the stack), or the clusters from projects 1–4
helm, kubectl	Local tools
Load generator	`hey` or k6
A receiver	A small webhook echo service in the cluster (or Slack/email if you have them)

Step 1: install the stack and the app

$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
$ helm repo add podinfo https://stefanprodan.github.io/podinfo && helm repo update
$ helm install kps prometheus-community/kube-prometheus-stack -n monitoring --create-namespace \
    --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false \
    --set prometheus.prometheusSpec.ruleSelectorNilUsesHelmValues=false
$ helm install podinfo podinfo/podinfo -n demo --create-namespace --set serviceMonitor.enabled=true

In the Prometheus UI (Status → Targets), podinfo should be UP. Check which metric names it exposes at /metrics (request counts and duration histograms with a status label); adapt the queries below to the exact names.

Step 2: traffic, including errors

$ kubectl -n demo port-forward svc/podinfo 9898 &
$ hey -z 15m -q 20 -c 2 http://localhost:9898/ &              # good traffic
$ hey -z 15m -q 2  -c 1 http://localhost:9898/status/500 &    # errors (~5%)

Step 3: dashboard

In Grafana (kubectl -n monitoring port-forward svc/kps-grafana 3000:80; admin password in the kps-grafana Secret), build a RED panel set:

sum(rate(http_requests_total{namespace="demo"}[5m]))                                                   # rate
sum(rate(http_requests_total{namespace="demo",status=~"5.."}[5m])) / sum(rate(http_requests_total{namespace="demo"}[5m]))   # error ratio
histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket{namespace="demo"}[5m])))                   # p95

Step 4: SLO rules and a burn-rate alert

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata: { name: podinfo-slo, namespace: demo }
spec:
  groups:
    - name: podinfo-slo
      rules:
        - record: slo:podinfo_errors:ratio_rate5m
          expr: sum(rate(http_requests_total{namespace="demo",status=~"5.."}[5m])) / sum(rate(http_requests_total{namespace="demo"}[5m]))
        - record: slo:podinfo_errors:ratio_rate1h
          expr: sum(rate(http_requests_total{namespace="demo",status=~"5.."}[1h])) / sum(rate(http_requests_total{namespace="demo"}[1h]))
        - alert: PodinfoErrorBudgetFastBurn
          expr: slo:podinfo_errors:ratio_rate1h > (14.4 * 0.001) and slo:podinfo_errors:ratio_rate5m > (14.4 * 0.001)
          labels: { severity: page, team: demo }
          annotations:
            summary: "podinfo is burning its 99.9% error budget 14x too fast"
            runbook_url: "https://wiki.example.com/runbooks/podinfo-errors"

With ~5% errors, both windows exceed 1.44%, so the alert fires once the 1-hour window has enough data (lower the long window to 15m for a quicker lab).

Step 5: route the alert to a receiver

Deploy a webhook echo service (any small HTTP server that logs requests) in demo, then:

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata: { name: demo-routing, namespace: demo }
spec:
  route:
    receiver: demo-webhook
    matchers: [ { name: team, value: demo } ]
  receivers:
    - name: demo-webhook
      webhookConfigs: [ { url: "http://echo.demo.svc:8080/" } ]

(Alertmanager must be configured to select AlertmanagerConfig resources; check the chart's alertmanager.alertmanagerSpec.alertmanagerConfigSelector values.)

Step 6: check

Prometheus Alerts page: PodinfoErrorBudgetFastBurn pending → firing.
Alertmanager UI: the alert grouped under your route.
The echo service logs show the webhook payload.
Stop the error traffic and watch it resolve.

Interview talking points

RED/USE, and why you page on SLO burn rate, not CPU.
The ServiceMonitor selector gotcha.
Alert routing, grouping, silences, runbook links.
What's next: logs (Loki/EFK), traces (OpenTelemetry), long-term storage (Thanos/Mimir).

Command summary

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts ; helm repo add podinfo https://stefanprodan.github.io/podinfo
helm install kps prometheus-community/kube-prometheus-stack -n monitoring --create-namespace --set …NilUsesHelmValues=false
helm install podinfo podinfo/podinfo -n demo --create-namespace --set serviceMonitor.enabled=true
kubectl -n demo port-forward svc/podinfo 9898 ; hey -z 15m -q 20 http://localhost:9898/ ; hey … /status/500
kubectl apply -f podinfo-slo.yaml -f echo.yaml -f alertmanagerconfig.yaml
kubectl -n monitoring port-forward svc/kps-kube-prometheus-stack-prometheus 9090   # Alerts page
kubectl -n monitoring port-forward svc/kps-kube-prometheus-stack-alertmanager 9093

Recap

kube-prometheus-stack + ServiceMonitor = metrics in minutes (mind the selector).
RED dashboard, SLO recording rules, a fast-burn alert with a runbook link.
AlertmanagerConfig routes to a receiver; test the whole path by causing errors.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.