Toil, as defined in Google's Site Reliability Engineering book, is the kind of work tied to running a production service that tends to be manual, repetitive, automatable, tactical, devoid of enduring value, and that scales linearly as the service grows. SRE teams measure toil and cap it, so most of their time goes into engineering that removes it.
What is toil?
Toil is a term popularized by Google's Site Reliability Engineering book, where a whole chapter is devoted to eliminating it. The book defines toil as "the kind of work tied to running a production service that tends to be manual, repetitive, automatable, tactical, devoid of enduring value, and that scales linearly as a service grows." Restarting a stuck worker by hand every few days, manually approving routine certificate renewals or copying log excerpts into tickets are classic examples.
Toil is not simply "work people dislike". The concept is useful precisely because it is specific: it names operational work that a machine could do and that keeps growing with your system unless you change something.
The six characteristics of toil
| Characteristic | Meaning | Example |
|---|---|---|
| Manual | A human runs the steps, even if a script is involved | Running a cleanup script by hand when disk usage alerts |
| Repetitive | The same task comes back again and again | Resetting the same service every week |
| Automatable | A machine could do it as well or better | Rotating credentials on a fixed schedule |
| Tactical | Interrupt-driven and reactive, not strategy-driven | Answering a page that always needs the same fix |
| No enduring value | The service is no better afterwards than before | Manually re-queueing failed jobs |
| Scales with growth | Effort grows linearly with traffic, hosts or customers | Hand-configuring each new tenant or cluster |
A task does not need all six properties to count as toil, but the more it has, the more clearly it qualifies.
What toil is not
The SRE book separates toil from overhead, such as meetings, planning, HR processes and travel, which is administrative work not directly tied to running a service. It also separates toil from engineering work, which is creative, requires judgment and produces lasting improvement: writing automation, improving monitoring, redesigning a fragile component. Some grungy work, such as a one-time migration or cleaning up a bad alert configuration, is not toil either, because it has enduring value.
Why toil matters
- It does not scale. If every new service or customer adds manual work, the operations team must grow at the same rate as the system.
- It crowds out improvement. Time spent on toil is time not spent on reliability work that would prevent the next incident.
- It hurts people. Large amounts of toil lead to boredom, mistakes, slower career growth and attrition. It also makes on-call harder, because toil-heavy services tend to page more.
- It introduces risk. Manual steps at 3 a.m. are error-prone; automation is consistent and auditable.
Google's SRE organization has an explicit goal of keeping operational work, including toil, below 50 percent of each SRE's time, so that at least half goes to engineering projects that reduce future toil or add features.
Measuring and reducing toil
Measure first
You cannot reduce what you do not see. Common approaches are tagging tickets and pages as toil, a short weekly survey of how much time went into toil, and tracking the recurrence of identical alerts and manual interventions. Postmortems are a rich source: repeated "fixed by restarting" or "manually scaled" entries point straight at automation candidates.
Then remove it
- Eliminate the cause. The best fix is often not automating a workaround but fixing the bug, leak or misconfiguration that makes the task necessary.
- Automate the response. Turn the steps of a runbook into a workflow that runs on an alert, with guardrails and an audit trail.
- Offer self-service. Replace "file a ticket and wait for ops" with a platform API or portal for routine requests.
- Delete noise. Alerts that are always acknowledged and ignored generate toil and alert fatigue at the same time.
Prioritize by frequency times effort: a five-minute task that happens twenty times a week costs more than a two-hour task that happens twice a year.
Common mistakes
- Automating a broken process instead of fixing its root cause.
- Building automation without ownership, tests or monitoring, which creates a new source of toil.
- Labeling all unpleasant work as toil, which dilutes the concept and the priority it deserves.
- Never measuring toil, so its growth goes unnoticed until the team is overloaded.
How Obseria helps
Obseria reduces typical observability toil in several places. Obseria Intelligence correlates alerts into problems and runs agentic root-cause investigations, so engineers no longer stitch together dashboards by hand. Obseria Automation lets you define remediation and ticketing workflows as code that trigger on problems, and incident management records timelines automatically. The Obseria Agent is installed once per host and discovers technologies without manual configuration per service. See also what is AIOps.