Site Reliability Engineering (SRE)

What is Site Reliability Engineering (SRE)?

Site Reliability Engineering (SRE) is an approach to running production systems that treats operations as a software problem. It was created at Google in the early 2000s; Ben Treynor Sloss, who founded the team, famously described SRE as "what happens when you ask a software engineer to design an operations team". The ideas became widely known through Google's books Site Reliability Engineering (2016) and The Site Reliability Workbook (2018), both freely available online.

SRE is both a job title and a set of practices. Many organizations adopt the practices without creating a dedicated SRE team, embedding them in product or platform engineering teams instead.

Core SRE principles

• Reliability is defined from the user's perspective. Service level indicators (SLIs) measure what users experience, and SLOs set targets for them, for example 99.9% of requests succeeding over 30 days.
• 100% is the wrong target. Users cannot tell 99.99% from 100%, but the cost of chasing it is huge. The gap between the SLO and 100% is the error budget.
• Error budgets align speed and stability. While budget remains, teams ship; when it is exhausted, reliability work takes priority. Burn-rate alerts show how fast the budget is being spent.
• Toil is capped. Google's SRE book recommends keeping operational work below 50% of an SRE's time, so the rest goes to engineering that removes toil permanently.
• Monitoring is symptom-based. Alerts should page a human only when users are affected or about to be, typically based on the four golden signals: latency, traffic, errors and saturation.
• Incidents are learning opportunities. Blameless postmortems focus on systems and processes, not on who made a mistake.
• Change is the main source of outages. Progressive rollouts, fast rollback and release automation reduce the risk of every deployment.

Error budget example

An illustrative API has an availability SLO of 99.9% over a rolling 30-day window. The error budget is 0.1% of requests. At 50 million requests a month that is 50,000 failed requests, or roughly 43 minutes of full outage. If a bad release consumes 30,000 of those failures in one afternoon, the team has used 60% of its monthly budget. Under a typical error budget policy, risky launches pause and the next sprint goes into fixing the cause, improving tests or adding a canary stage.

SRE versus DevOps

DevOps is a broad cultural movement about breaking down the wall between development and operations. SRE is one concrete way of implementing it, with prescriptive practices and numbers. Google summarizes the relationship as "class SRE implements interface DevOps". In practice the two overlap heavily: both value automation, shared ownership, small changes and measurement. DORA metrics describe delivery performance, while SLOs and error budgets describe the reliability that delivery must protect, so mature teams track both.

What SREs do day to day

• Define SLIs and SLOs together with product owners and maintain error budget policies.
• Build and tune alerting, dashboards and runbooks.
• Take part in on-call rotations and lead incident response.
• Automate deployments, capacity planning, failover and recurring operational tasks.
• Review architecture for reliability: redundancy, timeouts, retries, graceful degradation.
• Run postmortems and track their action items to completion.

Common mistakes when adopting SRE

• Renaming the operations team "SRE" without changing how work is prioritized.
• Setting SLOs that nobody enforces, so the error budget has no consequences.
• Alerting on every cause metric such as CPU instead of on user-facing symptoms.
• Letting SREs become the only people on call, which removes the feedback loop to developers.
• Copying Google's numbers without adapting them to your scale and risk.

SLIs, SLOs and SLAs

SRE separates three related terms. A service level indicator (SLI) is a measurement, typically a ratio of good events to all events, such as the share of HTTP requests that return a non-5xx status within 300 ms. A service level objective (SLO) is an internal target for that indicator over a window, for example 99.9% over 30 days. A service level agreement (SLA) is an external contract with consequences such as service credits if it is missed. SLOs are usually set stricter than SLAs, so that the team notices and reacts long before a contractual promise is at risk. Good SLIs are measured as close to the user as possible, for instance at the load balancer or in the browser, rather than deep inside a single service.

How Obseria helps

Obseria includes SLOs with error budgets, alerting, dashboards and incident management in every plan. Metrics, logs, traces and profiles live in one store, so an SLO breach can be followed down to the failing spans, and Obseria Intelligence can run root-cause investigations with a chain of evidence. Automation workflows as code help teams replace recurring manual steps.