SLO and error budget

They turn reliability from religion into a budget: “99.9% over 30 days = 43 minutes of downtime; budget exhausted → features stop, releases frozen” — a formalized trade-off between shipping and stability that both sides agreed to in advance.

The vocabulary comes from Google’s Site Reliability Engineering (2016), chapters “Service Level Objectives” and “Embracing Risk”. An SLI is a measured ratio of good events to valid events — successful /charge requests, or reads under 300 ms. An SLO is a target for that ratio over a window. The error budget is 1 − SLO: the unreliability you are allowed to spend on releases, experiments and maintenance. 100% is the wrong target, because users cannot tell 99.99% from 100% through their own networks and devices, and every extra nine costs more than the last. The SLA is a different object: a contract with penalties, set looser than the SLO so the internal alarm rings first.

Alerting follows the budget, not the raw error rate. The Site Reliability Workbook (2018, chapter “Alerting on SLOs”) defines burn rate as how fast the budget is consumed relative to the window: a burn rate of 1 spends exactly the 30-day budget in 30 days. Its recommended starting point pages when 2% of the budget burns in one hour (burn rate 14.4) or 5% in six hours (6), and opens a ticket for 10% in three days (1), each checked over a long and a short window so the alert also clears quickly. The Sloth config below generates those rules from one SLO definition.

slos:
  - name: charge-availability
    objective: 99.9
    sli:
      events:
        error_query: sum(rate(http_requests_total{path="/charge",code=~"5.."}[5m]))
        total_query: sum(rate(http_requests_total{path="/charge"}[5m]))
    alerting:
      page_alert: { labels: { severity: page } }   # multiwindow multi-burn-rate

Where it fails. An SLO on the wrong SLI is green while users suffer: server-side success misses requests that never arrived, and a mean latency hides the tail (26). Low-traffic services make ratios noisy — ten requests an hour cannot measure 99.9%. And a budget with no agreed consequence is only a dashboard; the freeze policy is what makes it a budget.