SLOs done right: how to define error budgets that your engineering team will actually respect

Why most SLOs fail before they start

We have seen this pattern hundreds of times. An engineering team reads the Google SRE book, spends two weeks debating what their availability target should be, settles on 99.9%, writes it in a wiki page, and then completely ignores it for the next six months. The on-call team still pages on every blip. Product still ships whenever it wants. The number is decorative.

The failure mode is almost never mathematical. It is cultural and definitional. Teams pick an SLO target before they know what they are actually measuring (the SLI), before they have agreed on consequences (the error budget policy), and before they have wired the number to their alerting. An SLO written in a wiki that is not connected to a real-time measurement system is fiction.

SLI, SLO, SLA: getting the definitions right

These three terms are consistently confused, even by experienced engineers. Here is the precise hierarchy:

• SLI — Service Level Indicator

  A quantitative measurement of a service behaviour. The raw number. Examples: request success rate, p99 latency, queue processing time.

• SLO — Service Level Objective

  A target range or threshold applied to an SLI over a rolling time window. Example: success rate ≥ 99.9% over a rolling 28-day window.

• SLA — Service Level Agreement

  A contractual commitment — usually external — with defined consequences for breach (credits, penalties, termination rights). SLAs are legal; SLOs are operational.

The practical implication: your SLO should be stricter than your SLA. If you promise customers 99.5% in your SLA, you should be operating to an internal 99.9% SLO — so that you detect problems and fix them before you breach your contractual obligation.

Choosing the right SLI for your service type

The SLI is where most SLO implementations go wrong. Teams either measure what is easy (CPU, memory, pod restarts) rather than what is meaningful, or they define the SLI so loosely that it is impossible to measure automatically.

For most services, the right SLI class depends on the service type:

• Request-serving (HTTP APIs, RPCs)

  Availability: proportion of requests that return a non-5xx response. Latency: proportion of requests served within threshold (e.g. p99 < 300ms).

• Data processing pipelines

  Freshness: age of the most recently processed record. Coverage: proportion of input records successfully processed within SLA window.

• Storage systems

  Durability: proportion of write operations that survive the retention period. Read availability: proportion of read requests that succeed.

• Batch jobs & scheduled tasks

  Completion rate: proportion of scheduled runs that complete successfully. Deadline compliance: proportion that complete within their window.

# ✅ Ratio-based SLI for HTTP availability
# Good events: 2xx and 3xx responses
sum(rate(http_requests_total{status=~"[23].."}[5m]))
/
sum(rate(http_requests_total[5m]))

# ✅ Latency SLI: proportion of requests < 300ms
sum(rate(http_request_duration_seconds_bucket{le="0.3"}[5m]))
/
sum(rate(http_request_duration_seconds_count[5m]))

Error budget arithmetic: what 99.9% really means

Before you pick a target, you need to understand what that target actually allows. A 99.9% SLO over a 28-day window means you have exactly 0.1% of 28 days of allowable bad minutes — approximately 40.3 minutes per month. That is your error budget.

The error budget is not just a measurement — it is a management tool. When you have budget remaining, your team has the freedom to ship faster, take more deployment risk, and run experiments. When the budget is running low, you slow down, focus on reliability work, and reduce deployment frequency. This is the fundamental mechanism that aligns product velocity with reliability.

One deployment causes a 15-minute outage? You just burned 37% of your monthly error budget. That is a concrete, quantified consequence that maps directly to a restriction on future deployment risk. This is far more meaningful than "our uptime this month was 99.9%".

Multi-window multi-burn-rate alerting

The single biggest mistake in SLO alerting: alerting on a single burn rate threshold. Teams set an alert for "error rate > 1%" and think they are done. This produces two failure modes simultaneously:

• False positives: a 60-second spike at 2% error rate triggers your alert but consumes 0.003% of your monthly budget — nothing to page about.
• Missed incidents: a slow 0.2% error rate sustained for 10 hours burns 30% of your monthly budget before any alert fires.

The solution from the Google SRE workbook is multi-window multi-burn-rate alerting. You monitor the burn rate (how fast you are consuming error budget) across multiple time windows simultaneously. A high burn rate in a short window indicates a fast, severe incident. A moderate burn rate in a long window indicates a slow leak.

# Multi-window multi-burn-rate SLO alert structure
#
# Burn rate = actual error rate / (1 - SLO target)
# Example: SLO = 99.9%, budget = 0.1%
# If error rate is 1.0%, burn rate = 1.0% / 0.1% = 10x
# At 10x burn rate you exhaust budget in 28d / 10 = 2.8 days

alerts:
  # Page immediately: fast burn, 2% of budget consumed in 1h
  - name: slo-critical-fast-burn
    condition: |
      burn_rate_1h > 14.4 AND burn_rate_5m > 14.4
    severity: page
    annotation: "Exhausts monthly budget in ~2 days if sustained"

  # Page: moderate burn, 5% of budget in 6h
  - name: slo-high-burn
    condition: |
      burn_rate_6h > 6 AND burn_rate_30m > 6
    severity: page
    annotation: "Exhausts monthly budget in ~5 days if sustained"

  # Ticket: slow leak, 10% of budget in 72h
  - name: slo-slow-burn
    condition: |
      burn_rate_3d > 1 AND burn_rate_6h > 1
    severity: ticket
    annotation: "Error budget is depleting — investigate during business hours"

The dual-window condition (both short and long window must exceed threshold) eliminates the false positive problem: a short spike does not fire the 6h window, and a slow leak does not fire the 5m window quickly enough to be a false alarm.

Configuring burn rate alerts in obseria.io

obseria.io's SLO engine computes burn rates continuously across all configured time windows. You define the SLI query once; the platform derives all burn rate windows and alert thresholds automatically.

# obseria.io SLO configuration (via API or YAML import)
slos:
  - name: checkout-api-availability
    description: "Checkout service HTTP availability"

    sli:
      ratio:
        good_events: |
          sum(rate(http_requests_total{
            service="checkout",
            status!~"5.."
          }[{{.window}}]))
        total_events: |
          sum(rate(http_requests_total{
            service="checkout"
          }[{{.window}}]))

    objectives:
      - target: 0.999          # 99.9%
        window: 28d

    alert_policy:
      page_burn_rate: 14.4     # fires on 1h + 5m windows
      ticket_burn_rate: 6.0    # fires on 6h + 30m windows
      slow_burn_rate: 1.0      # ticket on 3d + 6h windows

Error budget policies: when to freeze feature work

This is the component that most teams skip, and it is the most important one. An error budget policy is a written agreement that specifies exactly what changes in engineering behaviour when the error budget reaches certain thresholds.

• > 50% remaining

  Normal operations. Full deployment velocity. Feature work continues.

• 25–50% remaining

  Increased scrutiny on changes. Mandatory rollback plans for all deployments. No experimental features.

• 10–25% remaining

  Freeze non-critical deployments. Focus engineering capacity on reliability work. Daily budget review.

• < 10% remaining

  Full deployment freeze. All hands on reliability. No new features until budget recovers above 25%.

Write the policy in a document. Get sign-off from engineering leadership and product management before you launch the SLO. The conversation about what happens when budget runs out is far more valuable — and far less politically fraught — when it is hypothetical than when you are in the middle of a fire.

Common SLO mistakes and how to avoid them

• ✕ Setting the target too high too fast

  → Start by measuring your actual historical reliability for 30 days. Set your first SLO target at the 80th percentile of historical performance. You can always tighten it once you have the culture in place.

• ✕ SLIs that measure the service, not the user

  → Database CPU is not an SLI. Your payment API success rate is. Always trace the SLI back to a real user action or outcome. If users are not directly affected when this metric degrades, it is an operational metric — not an SLI.

• ✕ Alerting on budget consumed, not burn rate

  → Alerting on 'budget is 30% depleted' is a lagging indicator. You find out after significant damage. Burn rate tells you how fast you are consuming budget right now, so you can respond while you still have runway.

• ✕ Excluding maintenance windows from the SLO

  → Your users do not experience maintenance windows — they experience downtime. If you exclude planned maintenance, you are lying to yourself about reliability. Schedule changes to reduce blast radius instead.

• ✕ Having one SLO for the entire service

  → Different operations have different criticality. A checkout SLO should be stricter than a report-generation SLO. Define SLOs at the journey level, not the service level.