SLOs, SLIs & burn-rate alerting · Prasanta Kumar Behera

Alerts people trust: define SLIs from the user's point of view, set SLOs and error budgets, and alert on burn rate with multiwindow, multi-burn-rate rules instead of static thresholds, with working PromQL and tools that generate the rules for you.

SLIs: measure what users feel

An SLI (service level indicator) is a ratio of good events to valid events:

SLI	Good event
Availability	Request didn't fail with a server error (5xx, timeout)
Latency	Request finished in under 300 ms
Freshness (pipelines)	Data processed within 10 minutes
Correctness	Job produced the right output (checked by probes)

Measure as close to the user as possible: at the load balancer/ingress, or with synthetic checks, not only inside the service (lesson 05's scenario).

Your school promises the bus will be on time 999 days out of 1,000 (the SLO). That means it's allowed to be late about once in 1,000 days (the error budget). If the bus is late three days in a row, the budget is disappearing fast, and someone should act now (a page). If it's a little late once a month, a note to fix it next week is enough (a ticket).

SLOs and error budgets

SLO: the target, e.g. 99.9% of requests good over 30 days.
Error budget: 1 − SLO = 0.1% of requests can fail. With 10 million requests a month, that's 10,000 failures.
An SLA is a contract with consequences (credits); SLOs should be stricter than any SLA.
The budget is a decision tool: budget left → ship features faster; budget burnt → focus on reliability.

Burn rate

burn rate = observed error ratio / (1 − SLO)

Burn rate 1: the budget lasts exactly the SLO window.
Burn rate 14.4 for 1 hour consumes 2% of a 30-day budget (14.4 × 1h / 720h).

Multiwindow, multi-burn-rate alerts (from the Google SRE Workbook):

Severity	Long window	Short window	Burn rate	Budget used
Page	1h	5m	14.4	2%
Page	6h	30m	6	5%
Ticket	3d	6h	1	10%

PromQL

Recording rules for the error ratio over each window keep alerts cheap:

groups:
  - name: slo-cart-availability
    rules:
      - record: slo:cart_errors:ratio_rate5m
        expr: |
          sum(rate(http_requests_total{job="cart", code=~"5.."}[5m]))
          /
          sum(rate(http_requests_total{job="cart"}[5m]))
      - record: slo:cart_errors:ratio_rate1h
        expr: |
          sum(rate(http_requests_total{job="cart", code=~"5.."}[1h]))
          /
          sum(rate(http_requests_total{job="cart"}[1h]))
      - alert: CartErrorBudgetFastBurn
        expr: |
          slo:cart_errors:ratio_rate1h > (14.4 * 0.001)
          and
          slo:cart_errors:ratio_rate5m > (14.4 * 0.001)
        labels:
          severity: page
        annotations:
          summary: "cart is burning its 30-day error budget 14x too fast"
          runbook_url: https://runbooks.example.com/cart/availability

Add the 6h/30m and 3d/6h pairs the same way (with their own recording rules). Metric names depend on your instrumentation (OTel HTTP metrics use names like http_server_request_duration_seconds with a status-code attribute once converted to Prometheus).

Let tools write the rules

Sloth and Pyrra generate recording and alerting rules (and dashboards) from a short SLO spec.
OpenSLO is a vendor-neutral SLO specification format.
Grafana and many vendors have SLO features built in.

Try it: an SLO end to end

Deploy an app that exposes http_requests_total with a code label (many demo apps do; or the OTel Demo with spanmetrics).
Write the 5m and 1h recording rules and the fast-burn alert above for a 99.9% SLO; load them into Prometheus.
Inject a 5% error rate and calculate by hand when the alert should fire; check it does.
Stop the errors and watch the short window clear the alert.
Generate the full rule set with Sloth or Pyrra from a spec and compare with yours.

Going deeper: SLOs as a practice

Start with one or two SLOs per user journey (checkout, login), not per microservice.
Agree an error budget policy with product owners: what happens when the budget is exhausted (feature freeze, reliability sprint).
Low-traffic services need care: a few failures swing the ratio; use longer windows or synthetic traffic.
Review SLOs quarterly: too loose and users suffer quietly; too tight and teams ignore them.

Recap

SLI = good / valid events, measured from the user's side; SLO = target; error budget = 1 − SLO.
Burn rate = error ratio / budget; alert with multiwindow, multi-burn-rate pairs (14.4× 1h/5m, 6× 6h/30m page; 1× 3d/6h ticket).
Use recording rules, or generate everything with Sloth/Pyrra.
Error budgets drive decisions between speed and reliability.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.