SLOs for the platform itself · Prasanta Kumar Behera

Define what 'the platform is healthy' means from your users' point of view. Choose SLIs for the cluster itself, set SLOs with error budgets, and alert on burn rate instead of every blip.

"Is the cluster healthy?" is the wrong question

A dashboard full of green component checks can coexist with developers unable to deploy. Health has to be defined from the user's side: what do the platform's users (application teams) need to work?

A school bus company can say "all buses have fuel and tyres" (component checks). But parents only care: did my child get to school on time? That's the SLI. "99% of children on time this term" is the SLO. The few late days allowed are the error budget. If a bus is late three days in a row, you act before the budget runs out.

Pick platform SLIs from user journeys

User journey	SLI (good ÷ total)	Where to measure
"I can use the API"	Non-5xx API requests ÷ all API requests	`apiserver_request_total`
"The API is fast"	Requests under a latency threshold ÷ all	`apiserver_request_duration_seconds` histogram
"I can deploy and my pod starts"	Probe deployments Ready within 2 min ÷ probe attempts	Synthetic probe
"My service is reachable"	Successful probe calls through Service/Ingress ÷ attempts	Synthetic probe
"Names resolve"	Successful in-cluster DNS lookups ÷ attempts	Probe or CoreDNS metrics

The synthetic probe is the most honest SLI: every few minutes, deploy a tiny app, expose it, call it and delete it, recording success and duration. It fails if any layer fails.

Set SLOs and error budgets

SLO	Window	Error budget
99.9% API availability	30 days	≈ 43 minutes
99% of probe deployments Ready < 2 min	30 days	1% of probes may be slow or fail
99.5% probe reachability	30 days	≈ 3.6 hours

Choose targets from what users need and what you can deliver, not from wishful nines. Each extra nine costs a lot more.

Use the budget: while budget remains, the platform team can ship changes (upgrades, new features). When it's exhausted, risky changes pause and reliability work comes first. That turns reliability from an argument into a shared rule.

Alert on burn rate

"Page when the error ratio is above zero" wakes people for blips. Burn rate asks how fast the budget is being spent:

burn rate = observed error ratio ÷ allowed error ratio
99.9% SLO → allowed 0.1%. An error ratio of 1.44% → burn rate 14.4.
At 14.4 for 1 hour you spend 14.4 × 1h ÷ 720h = 2% of the monthly budget.

A common pattern is multi-window alerts: page on a fast burn (e.g. 14.4× over 1h and 5m), open a ticket on a slow burn (e.g. 1× over 3 days). The two windows avoid both slow detection and flapping.

API error ratio in PromQL:

sum(rate(apiserver_request_total{code=~"5.."}[5m]))
  /
sum(rate(apiserver_request_total[5m]))

Try it: a minimal platform probe

Write a script (Bash or Python) that, every 5 minutes: creates a namespace probe-<timestamp>, deploys nginx:1.27, waits for Ready (kubectl wait --for=condition=available deploy/probe --timeout=120s), exposes it, calls it from a busybox pod, and deletes the namespace.
Record success (0/1) and the duration of each step.
After a day, compute your SLI: successful probes ÷ total. Which step was slowest?
Break something on purpose (taint all workers) and watch the SLI drop while component checks stay green.

Going deeper: SLOs as a platform practice

Publish platform SLOs to application teams. They set expectations, and give you a data-driven voice in prioritisation.
Exclude long-lived requests (WATCH, CONNECT, logs streaming) from API latency SLIs; they're long by design.
SLOs for dependencies (registry, DNS, identity provider, object storage) explain many platform incidents. Measure them too.
Tools such as Sloth or Pyrra generate recording rules and burn-rate alerts from a short SLO definition. More in Observability with OpenTelemetry.

Recap

Define health from user journeys: API usable and fast, deploys succeed, services reachable, DNS works.
SLI = good ÷ total; SLO = target over a window; error budget = allowed failure, used to steer change.
Prefer synthetic probes for end-to-end truth, and burn-rate alerts over "anything failed".

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.