A runbook is a documented procedure that tells an on-call engineer or an automation system how to diagnose and resolve a specific operational problem, or how to perform a routine task such as a failover. Good runbooks are linked from alerts, list verifiable steps with expected results, and are gradually turned into automation.
What is a runbook?
A runbook is a written, repeatable procedure for a known situation: a disk filling up, a queue backing up, a certificate about to expire, a database failover. It captures the knowledge of the engineers who built and operate a service, so that whoever is on call at 3 a.m. can act quickly and consistently without having to reverse-engineer the system under pressure.
Runbooks reduce time to mitigate, a key driver of MTTR, and they spread operational knowledge across the team. They also expose toil: every runbook that is executed often and step-by-step identically is a candidate for automation.
Runbook vs. playbook
The terms are often used interchangeably, but many organizations distinguish them:
| Aspect | Runbook | Playbook |
|---|---|---|
| Scope | One specific technical task or failure mode | A broader response process, e.g. a major incident or security breach |
| Content | Concrete commands, queries, checks and expected results | Roles, communication, escalation paths and decision points |
| Typical trigger | A specific alert | An incident severity level or incident type |
A playbook often references several runbooks: the major-incident playbook defines who is incident commander, and the database runbook tells the responder how to promote a replica.
What a good runbook contains
- Title and trigger: which alert or symptom this runbook addresses.
- Impact and urgency: what users experience and how quickly action is needed.
- Diagnosis: links to the right dashboards and queries, and how to confirm the hypothesis.
- Mitigation steps: numbered, copy-pasteable actions, each with its expected result and how to verify it.
- Rollback: how to undo each step if it makes things worse.
- Escalation: when to stop and whom to call, including service owners and vendors.
- Metadata: owner, last reviewed date and links to related postmortems.
Example runbook
An illustrative runbook for a PostgreSQL connection saturation alert:
Alert: orders-db connections > 90% of max_connections for 5 min
Impact: checkout requests time out once the limit is reached
1. Confirm: open the "orders-db" dashboard; check active vs. idle
connections and which services hold them.
2. Identify long-running sessions:
SELECT pid, state, now() - xact_start AS age, query
FROM pg_stat_activity ORDER BY age DESC LIMIT 20;
3. If one service holds most idle-in-transaction sessions,
restart its pods one at a time and watch the connection count.
Expected: count drops below 70% within 2 minutes.
4. If a recent deployment changed pool settings, roll it back.
5. Escalate to the database team if saturation persists after
15 minutes or if replication lag exceeds 30 s.Each step states what to look at, what to do and what "good" looks like. The thresholds are examples; use the values that match your system.
From documentation to automation
Runbook automation turns the safe, deterministic parts of a procedure into code that runs on demand or in response to an alert: collecting diagnostics, scaling a deployment, rotating a credential, restarting a stuck consumer. A sensible progression is:
- Manual: written steps that a human follows.
- Assisted: scripts that gather context and propose the next step, with a human approving.
- Automated: the workflow runs on its own for well-understood cases, with guardrails, rate limits and an audit trail.
Keep the decision points explicit. Automation that restarts services blindly can hide a real problem or amplify it, for example by restarting every pod at once. Pair automated remediation with health checks that verify the fix worked.
Best practices and common mistakes
- Link every paging alert to a runbook; an alert without a runbook is a strong candidate for deletion and a source of alert fatigue.
- Store runbooks next to the code or in a searchable wiki with version history, and review them after every incident in which they were used.
- Test runbooks in game days; steps that only work on the author's laptop fail when they matter most.
- Avoid vague instructions such as "check the logs". Name the query, the field and the expected value.
- Do not let runbooks rot: outdated hostnames or commands are worse than none, because they cost time and trust.
How Obseria helps
In Obseria, alerts can carry a runbook link and the relevant dashboards, so responders land on the right context immediately. With Obseria Automation, the deterministic steps of a runbook can be implemented as workflows as code, triggered by alerts or run on demand, with every action recorded for the postmortem.
