Skip to main content

Burn-Rate Alerting (Multi-Window, Multi-Burn-Rate)

Level 7: Level 7: Reliability, Resilience & Operationshard30 minburn-ratealertingslo

Burn rate is multiples of sustainable spend (1x uses exactly the budget, 14.4x burns ~2% of a month in an hour); require a long window for significance and a short window for fast reset; page on fast burn and ticket on slow burn; and always alert on the SLO-burn symptom, never on causes like CPU.

When to wake a human

Once you have an SLO and an error budget, the question is when to wake a human. The naive answer, "alert whenever the error rate is above X," produces either a flood of pages during any tiny blip or a threshold so high you miss slow bleeds. Burn-rate alerting solves this by alerting on how fast you are spending the budget.

Burn rate is how many times faster than sustainable you are consuming the error budget. A burn rate of 1x means you are spending the budget exactly on pace: at 1x you will use precisely 100% of it by the end of the SLO window and no more. A burn rate of 2x means you will exhaust the whole window's budget in half the window. 14.4x means you burn a 30-day budget in about 2 days, or equivalently 2% of a 30-day budget in one hour. Concretely, burn rate = (observed error rate) / (1 - SLO). For a 99.9% SLO the budget is 0.1%, so a sustained 1.44% error rate is a 14.4x burn.

The canonical multi-window ladder

You pick a burn rate and a window so that tripping means you would consume a meaningful fraction of your total budget. The Google SRE canonical setup for a 99.9% SLO:

Alert type     Burn rate   Long window   Short window   Budget spent   Action
Fast burn      14.4x       1 hour        5 min          ~2% in 1h      Page now
Medium burn    6x          6 hours       30 min         ~5% in 6h      Page
Slow burn      1x          3 days        6 hours        ~10% in 3d     Ticket

Fast burn (14.4x over 1 hour) means something is badly wrong right now and you will blow the whole month's budget in a couple of days at this rate: that pages a human immediately. Medium burn (6x over 6 hours) is slower but still spends a twentieth of the month in an afternoon, so it also pages, just without the drop-everything urgency. Slow burn (1x over 3 days) is a chronic bleed that is not an emergency but must not be ignored: that files a ticket for business hours.

Why two windows

Each alert requires both a long window and a short window to be over threshold simultaneously. The long window (1 hour) gives significance so you do not page on a 30-second spike. The short window (5 minutes) makes the alert reset quickly once the problem is fixed, so you are not stuck with a firing page for an hour after recovery. Requiring both cuts false positives (a brief blip fails the long window) and flapping (a recovered incident clears the short window fast). This is the multi-window multi-burn-rate pattern.

Interview nuance: the single most important principle is alert on symptoms, not causes. Page on SLO burn (users are experiencing errors or slowness) not on CPU at 90% or memory pressure. High CPU might be fine; it is a cause that may or may not hurt users. A page must mean "a user is being hurt and a human must act now." Cause-based metrics belong on dashboards and in tickets for capacity planning, not on the pager. Alerting on causes is the number-one source of alert fatigue: engineers get paged for a high-CPU condition that auto-scaled away before they opened their laptop, learn to ignore pages, and then miss the real one.

The tuning tradeoff: shorter windows and lower burn-rate thresholds detect problems faster but page on smaller, sometimes self-healing events (more false positives, more budget-noise). Longer windows and higher thresholds page only on serious sustained problems but let more budget burn before you know. You trade detection time against budget spent and against page volume. Fast-burn catches acute outages quickly; slow-burn catches the chronic bleed that would otherwise silently drain you over a week.

Recap: burn rate is multiples of sustainable spend (1x uses exactly the budget, 14.4x burns ~2% of a month in an hour); require a long window for significance and a short window for fast reset; page on fast burn and ticket on slow burn; and always alert on the SLO-burn symptom, never on causes like CPU, to kill alert fatigue.

Apply

Your turn

The task this lesson builds to.

Design the alert rules for a 99.9% SLO service using multi-window multi-burn-rate alerts, giving thresholds for a fast-burn page vs a slow-burn ticket.

Think about

  1. What is a burn rate, and what does 1x mean?
  2. Why require both a short and a long window to trip?
  3. Why alert on SLO burn (symptom) not CPU (cause)?

Practice

Make it stick

A second problem on the same idea, so it survives past today.

Design burn-rate alerting for Amazon-style checkout with a 99.99% SLO and a strict latency SLO (p99 under 300 ms), where the business tolerates almost no false-negative on a real outage but on-call is already drowning in pages. Specify how you handle both the availability and latency SLOs and how you cut page volume without missing a genuine outage.

Think about

  1. Why does a four-nines budget need a sharper very-fast tier than a 99.9% service?
  2. Why does latency get its own burn-rate ladder separate from availability?
  3. What cuts page volume without lowering every threshold uniformly?