Error budget burn-rate alerts in Prometheus, explained

Most alerting starts with thresholds: page if the error rate is above 1 percent for five minutes. Those alerts are both too noisy and too quiet. They fire on short blips that barely matter, and they never fire on a steady 0.5 percent error rate that quietly spends a month's reliability budget in a week. Burn-rate alerts replace the arbitrary threshold with a question that matters: at the current rate, how soon will we break our SLO?

Error budgets in one paragraph

A service level objective (SLO) such as 99.9 percent of requests succeeding over 30 days leaves an error budget of 0.1 percent of requests. Over 30 days, that's equivalent to 43 minutes and 12 seconds of complete downtime. The burn rate compares how fast you're spending that budget with the pace the SLO allows. A burn rate of 1 uses up the budget exactly at the end of the window; a burn rate of 10 uses it up in a tenth of the window, three days for a 30-day SLO.

The maths

Burn rate is the observed error ratio divided by the error ratio the SLO allows. With a 99.9 percent SLO the allowed ratio is 0.001, so an error ratio of 0.0144 (1.44 percent) is a burn rate of 14.4. To alert when a given share of the budget is spent in a given time, use: burn rate = share of budget × window length ÷ alert window. Spending 2 percent of a 30-day budget in 1 hour is 0.02 × 720 ÷ 1 = 14.4.

Multi-window, multi-burn-rate alerts

The Google SRE Workbook recommends combining a fast and a slow condition, each checked over two windows. The long window shows the burn is significant; the short window shows it's still happening, so the alert stops soon after you fix the problem. A common set for a 30-day SLO is:

• Page: 2 percent of the budget in 1 hour (burn rate 14.4), confirmed over the last 5 minutes.
• Page: 5 percent of the budget in 6 hours (burn rate 6), confirmed over the last 30 minutes.
• Ticket: 10 percent of the budget in 3 days (burn rate 1), confirmed over the last 6 hours.

Recording rules first

Compute the error ratio over each window once, as recording rules, then write alerts against them. This keeps alert expressions short and cheap to evaluate. Replace the metric names with your own; the example uses an HTTP request counter with a status code label.

groups:
  - name: slo-recording
    rules:
      - record: slo:error_ratio:rate5m
        expr: sum(rate(http_requests_total{job="api",code=~"5.."}[5m])) / sum(rate(http_requests_total{job="api"}[5m]))
      - record: slo:error_ratio:rate1h
        expr: sum(rate(http_requests_total{job="api",code=~"5.."}[1h])) / sum(rate(http_requests_total{job="api"}[1h]))
      - record: slo:error_ratio:rate30m
        expr: sum(rate(http_requests_total{job="api",code=~"5.."}[30m])) / sum(rate(http_requests_total{job="api"}[30m]))
      - record: slo:error_ratio:rate6h
        expr: sum(rate(http_requests_total{job="api",code=~"5.."}[6h])) / sum(rate(http_requests_total{job="api"}[6h]))

The alerts

groups:
  - name: slo-alerts
    rules:
      - alert: ErrorBudgetBurnFast
        expr: slo:error_ratio:rate1h > (14.4 * 0.001) and slo:error_ratio:rate5m > (14.4 * 0.001)
        labels:
          severity: page
      - alert: ErrorBudgetBurnMedium
        expr: slo:error_ratio:rate6h > (6 * 0.001) and slo:error_ratio:rate30m > (6 * 0.001)
        labels:
          severity: page

Add the slow ticket alert the same way with 3-day and 6-hour windows. For long windows like 3 days, a recording rule over a 3-day range is expensive; many teams compute it from 1-hour recorded rates averaged over time instead.

Choosing factors for other windows

The factors depend on your SLO window. For a 28-day SLO the fast page is 0.02 × 672 ÷ 1 = 13.44; for a 7-day SLO it's 3.36. Low-traffic services need care: a handful of failed requests can produce a huge ratio. Add a minimum request rate to the alert, or use longer windows.

Making the alerts useful

• Put the SLO, the burn rate and the remaining budget in the alert annotation, so the person paged knows how urgent it is.
• Link a runbook and a dashboard that shows budget remaining over the window.
• Review every page: if one fired without a real problem, adjust the SLO or the measurement, not just the alert.
• Agree an error budget policy, for example releases pause when the budget is spent.

A worked example

An API has a 99.9 percent availability SLO over 30 days and handles about 2,000 requests a second. A bad deploy starts failing 2 percent of requests. The error ratio of 0.02 is a burn rate of 20, above the fast threshold of 14.4. Within a few minutes the 5-minute window crosses the threshold, and once the 1-hour window's average also crosses it, the page fires. If nothing were done, the whole month's budget would be gone in 36 hours. Compare a slow leak where 0.3 percent of requests fail all day: a burn rate of 3, below both page thresholds, so it raises a ticket through the slow 3-day alert instead of waking anyone, but it's still caught long before the budget runs out.

Common mistakes

• Measuring the SLO from inside the service (pod health, CPU) rather than from what users experience.
• Dividing by zero or near-zero traffic at night, which makes the ratio meaningless. Add a minimum request rate.
• Using burn-rate factors for a 30-day window on a 7-day SLO.
• Paging on the slow burn, which trains people to ignore pages.
• Counting client errors such as 404s as failures when they're the caller's fault.

How to verify

Test the rules with promtool test rules, using a series that injects a known error rate and checking when each alert fires. Replay a past incident through the same queries in Prometheus to see whether the alerts would have fired in time. After going live, review every page for a month: each should match a real problem users noticed.

Latency SLOs work the same way

Availability isn't the only promise worth measuring. A latency SLO such as "99 percent of search requests complete within 300 milliseconds" uses exactly the same burn-rate approach: count requests slower than the threshold as bad events. With a Prometheus histogram, the good requests are the bucket at or below 300 milliseconds, and the bad ratio is one minus good divided by total. Choose bucket boundaries that include your threshold exactly, or the measurement will be approximate. The budget for a 99 percent SLO is 1 percent of requests, so the same 14.4 and 6 burn-rate factors give the page thresholds, multiplied by 0.01 rather than 0.001.

Checklist

• An SLO measured from the user's side (requests at the load balancer or a synthetic journey).
• Recording rules for the error ratio over each window.
• Two page alerts (fast and medium burn) and one ticket alert (slow burn).
• Factors calculated for your SLO window.
• A minimum traffic condition for low-volume services.
• Runbooks, dashboards and an error budget policy.