Anomaly detection on metrics · Prasanta Kumar Behera

Detect unusual metric behaviour with statistics before reaching for machine learning: z-scores against a rolling baseline, week-over-week seasonal comparisons, linear prediction for capacity, all in PromQL, and when anomaly signals should inform rather than page.

Unusual vs broken

An anomaly is behaviour that differs from normal. It may or may not hurt users. Anomaly detection is valuable for early warning and context, and dangerous as a paging source if it fires on every unusual-but-harmless change.

A teacher knows the class is noisy at break time and quiet during tests. If it's suddenly silent at break time, something unusual is happening, worth a look. But "unusual" isn't always "bad": maybe there's a magician in the playground. You look; you don't call the fire brigade.

Baselines with PromQL

Record the rate you care about first:

groups:
  - name: anomaly-baselines
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))

z-score against the last day (subqueries):

(
  job:http_requests:rate5m
  - avg_over_time(job:http_requests:rate5m[1d:5m])
)
/
stddev_over_time(job:http_requests:rate5m[1d:5m])

Values above +3 or below −3 are unusual relative to the recent past. Watch out for seasonality: the last day includes nights, so daytime peaks can look anomalous against a 24-hour average.

Seasonal comparison (same time last week):

job:http_requests:rate5m / (job:http_requests:rate5m offset 1w)

A ratio below 0.5 (half of last week's traffic at this hour) is a strong "something's off" signal for traffic that should be steady week to week. Handle holidays and releases as known exceptions.

Trend prediction for capacity:

predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 4 * 3600) < 0

"At the current trend, this disk is full within 4 hours." Good for tickets with lead time, or pages when lead time is very short.

(Prometheus also offers smoothing functions such as double_exponential_smoothing, formerly holt_winters, which newer versions place behind an experimental-functions feature flag.)

When to consider ML

Many metrics with complex, shifting patterns where hand-tuned baselines don't scale.
Correlating anomalies across many signals to suggest which change is related.
Tools range from vendor AIOps features to open-source libraries on exported data.

Require the same standards as any alert: explainability (why is this anomalous?), measured precision (how many were real?), and a clear action.

Put anomalies in the right place

Dashboards: shade anomalous periods; show week-over-week lines.
Tickets/Slack: "traffic 60% below last week for 30 min" for the owning team.
Incident context: attach anomaly summaries to SLO burn-rate pages to speed up diagnosis.
Pages: stay on user-impact signals (see SRE & Production Incident Response, lesson 07).

Try it: statistical anomaly detection

In a kube-prometheus-stack lab, generate traffic with a daily pattern (a k6 script with ramping stages on a schedule, or a loop varying rate by hour).
Add the recording rule, then graph the z-score and the week-over-week ratio (use offset 1h instead of 1w if your lab is new).
Inject an anomaly (stop traffic for 10 minutes, or double it) and see which method flags it and how quickly.
Write a predict_linear rule for a filling disk (fill a test volume with dd) and tune the lead time.
Decide for each signal: dashboard, ticket, or page? Justify in one sentence.

Going deeper: tuning and trust

Keep a record of anomaly signals and whether they mattered; precision decides whether people keep looking at them.
Use recording rules for baselines (avg/stddev over time) to keep queries cheap.
Mute expected anomalies during planned events (deploys, load tests, holidays) with annotations or silences.

Recap

Anomaly ≠ incident: use anomalies for early warning and context.
Start with statistics in PromQL: z-scores (subqueries), week-over-week seasonality, predict_linear for capacity.
Consider ML only for what statistics miss, with explainability and measured precision.
Page on user impact; put anomalies on dashboards, tickets and incident context.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.