SLOs in Practice: From Error Budgets to Release Gates

Most teams that adopt SLOs get as far as a dashboard with a percentage on it. The number is green most of the time, red occasionally, and changes nothing about how the team works. SLOs in practice only pay off when the error budget drives decisions: which alerts page someone, when reliability work takes priority over features, and whether a release goes out today.

This post walks through the full path, from choosing an SLI to wiring the error budget into your delivery pipeline, with configuration examples you can adapt.

Step 1: choose SLIs that reflect user experience

A service level indicator (SLI) is a measurement of one aspect of service quality. The most robust form is a ratio: good events divided by valid events. That form works for availability, latency and correctness alike, and it maps directly onto error budgets.

SLI type	Good event	Valid event
Availability	Request answered without a server error (5xx)	All requests except health checks
Latency	Request completed in under 400 ms	All successful requests
Freshness	Data updated within the last 10 minutes	All reads of the dataset
Correctness	Job output passed validation	All completed jobs

Measure as close to the user as practical. An SLI computed at the load balancer or ingress includes failures that a service-internal metric misses, such as connection resets or a crashed pod. For user journeys like checkout or login, define the SLI on the entry span of the journey rather than on every internal service.

Exclude what users do not experience: health checks, synthetic probes (unless they are the SLI) and requests rejected because of client errors. A 4xx caused by an invalid request is not a reliability problem; a 429 caused by your own rate limiter may well be.

Step 2: set targets and windows deliberately

The target defines how much unreliability you accept. It should be based on what users tolerate and what the service has historically achieved, not on round numbers. Start slightly below current performance, so the SLO is achievable, and tighten it once you understand the budget dynamics.

Use a rolling window, typically 28 or 30 days. Calendar-month windows reset the budget on the first of the month, which encourages risky releases at the end of a month with budget left over.

The error budget is simply one minus the target. A 99.9% availability SLO over 30 days allows 0.1% of requests to fail. For a service with 10 million requests in that window, that is 10,000 failed requests. Expressed in time, 0.1% of 30 days is 43.2 minutes of complete outage. Moving to 99.95% halves both numbers, so every extra nine should be a deliberate investment.

Writing SLOs as code keeps them reviewable and portable. The OpenSLO specification is a vendor-neutral format for that:

apiVersion: openslo/v1
kind: SLO
metadata:
  name: checkout-availability
  displayName: Checkout availability
spec:
  service: checkout
  budgetingMethod: Occurrences
  timeWindow:
    - duration: 30d
      isRolling: true
  objectives:
    - displayName: 99.9% of checkout requests succeed
      target: 0.999
  indicator:
    metadata:
      name: checkout-good-requests
    spec:
      ratioMetric:
        counter: true
        good:
          metricSource:
            type: Obseria
            spec:
              query: 'fetch spans | filter service.name == "checkout" and span.kind == "server" and http.status_code < 500 | summarize count()'
        total:
          metricSource:
            type: Obseria
            spec:
              query: 'fetch spans | filter service.name == "checkout" and span.kind == "server" | summarize count()'

Step 3: measure the error budget continuously

The current state of an SLO comes down to two numbers: the SLI over the window and the share of budget that remains. In OQL, both can be computed directly from spans:

fetch spans, from: now() - 30d
| filter service.name == "checkout" and span.kind == "server"
| summarize total = count(), bad = count_if(http.status_code >= 500)
| fields_add sli = 1 - bad / total,
             budget_remaining = 1 - (bad / total) / 0.001

A budget_remaining of 0.4 means 60% of the allowed failures have been used and 40% are left. Negative values mean the SLO is violated for the current window. Showing this number per service on a team dashboard is far more actionable than a raw error rate, because it already encodes how much failure is acceptable.

Step 4: alert on burn rate, not on thresholds

Burn rate is the speed at which the budget is consumed relative to the SLO window. A burn rate of 1 uses exactly the whole budget over 30 days. A burn rate of 14.4 sustained for one hour consumes 2% of a 30-day budget, because 14.4 times one hour divided by 720 hours equals 0.02.

Multi-window, multi-burn-rate alerts combine a long window, which proves the problem is significant, with a short window, which proves it is still happening. A widely used set for a 30-day SLO:

Severity	Burn rate	Long window	Short window	Budget consumed
Page	14.4	1 hour	5 minutes	2%
Page	6	6 hours	30 minutes	5%
Ticket	1	3 days	6 hours	10%

For a 99.9% SLO, a burn rate of 14.4 corresponds to an error rate of 1.44%. The alert fires only if both the 1-hour and the 5-minute error rates exceed that value, so it triggers fast for real incidents and resolves quickly once the problem stops. This set replaces most static error rate alerts and drastically reduces pages that do not need a human.

Step 5: turn the error budget into a release gate

This is where SLOs start to change behavior. A release gate checks the error budget before a deployment proceeds and applies a policy that the team agreed on in advance:

• More than 25% budget remaining: deploy automatically.
• Between 0% and 25% remaining: deploy only with approval from the service owner, and prefer changes that improve reliability.
• Budget exhausted: freeze feature releases; only fixes, rollbacks and reliability work ship until the budget recovers.

The gate belongs in the delivery pipeline, not in a meeting. As a workflow defined in code, it can look like this:

name: release-gate-checkout
trigger:
  type: deployment.requested
  filter: service == "checkout"
steps:
  - id: budget
    action: slo.evaluate
    with:
      slo: checkout-availability
  - id: decide
    action: policy.decide
    with:
      rules:
        - when: steps.budget.budget_remaining > 0.25
          result: approve
        - when: steps.budget.budget_remaining > 0
          result: require_approval
          approvers: [team:checkout-owners]
        - otherwise: reject
          message: "Error budget exhausted: only fixes and rollbacks"

Add a second check after the deployment: compare the burn rate in the 15 minutes after rollout with the 15 minutes before. If the new version burns budget significantly faster, roll back automatically. Combined, the pre-deployment gate and the post-deployment check catch both "we should not ship right now" and "this release made things worse".

Agree on an error budget policy

Automation enforces a policy; it does not replace one. Write down, and have product and engineering leadership sign off on, what happens at each budget level. Good policies are short and specific:

1. Who owns the SLO and reviews it, and how often (monthly is a good default).
2. What happens when budget is exhausted, including who may grant exceptions.
3. How incidents caused by dependencies are treated: counted against the budget, but with a follow-up item for the dependency owner.
4. When targets are revisited, for example when user expectations or architecture change.

Review SLOs that are never at risk as critically as those that are always red. A target that is never threatened may be too loose to guide decisions; one that is always violated has stopped being taken seriously.

SLOs with Obseria

SLOs are included in every Obseria plan. You define them on any signal, including spans, logs, metrics and synthetic checks, with OQL as the SLI query. Burn rate alerts follow the multi-window model described above, and Obseria Automation runs release gates as workflows in code, triggered by deployment events from your CI/CD system. When budget burns, Obseria Intelligence starts a root cause investigation along the TopoGraph and links the affected SLO to the change that caused it.

Start with one user journey, one SLI and one burn rate alert. Once the team trusts that number, the release gate is a small step, and it is the one that turns SLOs from a report into a decision tool.