A postmortem is a structured, written review created after an incident. It documents customer impact, a timeline of detection and response, the technical and organizational factors that contributed, and concrete action items with owners. Blameless postmortems focus on how the system allowed the failure rather than on who made a mistake, so teams learn instead of hiding errors.
What is a postmortem?
A postmortem, also called a post-incident review, is the document and meeting that close the loop on an incident. Once service is restored, the team reconstructs what happened, why it happened, how it was detected and resolved, and what will change so the same class of failure becomes less likely or less damaging. The practice is central to site reliability engineering and works hand in hand with incident management: incident response restores service, the postmortem improves the system.
Most teams require a postmortem when an incident crosses a defined threshold, for example any user-visible outage, a breach of an SLO or a significant share of the error budget consumed, data loss, or a manual intervention by on-call engineers.
Why postmortems must be blameless
Complex systems fail through combinations of latent conditions: a missing alert, an ambiguous dashboard, a config default, time pressure. If a postmortem ends with "engineer X ran the wrong command", the organization learns nothing about why the command was possible, why it looked safe and why nothing stopped it. A blameless postmortem assumes people acted reasonably given the information they had, and asks how tooling, process and design shaped those decisions. This is not about avoiding accountability; it is about getting honest, complete accounts, which people only give when they are not afraid of punishment.
A practical postmortem template
- Summary: two or three sentences on what broke, for how long, and for whom.
- Impact: affected users, requests, regions and revenue-relevant flows; SLO and error budget consumption.
- Timeline: timestamped events from the first trigger through detection, escalation, mitigation and full recovery, in one time zone (usually UTC).
- Detection: how the issue was noticed, whether by an alert, a customer or by chance, and how long it took.
- Contributing factors: technical triggers and the conditions that let them cause harm. Avoid forcing a single "root cause".
- What went well and what went poorly: including tools, runbooks and communication.
- Action items: specific, prioritized, each with an owner and a due date, tracked in the normal backlog.
Example: a timeline excerpt
An illustrative timeline for a failed release:
| Time (UTC) | Event |
|---|---|
| 14:02 | Deployment of checkout v2.41 starts, configuration change reduces database connection pool from 50 to 5. |
| 14:09 | p99 latency of POST /orders rises from 300 ms to 4 s; error rate reaches 6 percent. |
| 14:11 | Burn-rate alert pages on-call engineer. |
| 14:18 | Trace analysis shows requests waiting for pool connections; deployment correlated with onset. |
| 14:21 | Rollback started using the checkout runbook. |
| 14:27 | Latency and errors back to baseline; incident mitigated. |
From this timeline the team derives time to detect (9 minutes from deployment to alert) and time to mitigate (25 minutes), which feed into MTTR tracking. Action items might include validating pool size in CI, a canary stage with automatic rollback and a saturation alert on the pool.
Best practices
- Write the draft within a few days, while memories, chat logs and telemetry are fresh.
- Link evidence directly: dashboards, traces, log queries, deployment records, chat transcripts.
- Prefer action items that change the system (guardrails, automation, tests) over items that ask people to "be more careful".
- Review open action items regularly; an unfinished postmortem is a promise the next incident will collect on.
- Share postmortems widely and make them searchable so other teams can learn from recurring patterns.
Running the review meeting
The written draft comes first; the meeting is for questions, not for reading the document aloud. Invite the responders, the owners of affected services and someone who was not involved, because outsiders ask the questions insiders no longer notice. A neutral facilitator keeps the discussion on mechanisms rather than people and stops debates about who should have known better. End the meeting by agreeing on the prioritized action items and their owners. For severe incidents, publish a short customer-facing summary as well, ideally on the same status page you used during the incident.
Common mistakes
- Stopping at the first technical trigger instead of asking why safeguards did not catch it.
- Producing long lists of vague action items with no owners.
- Skipping postmortems for near misses, which are cheap lessons.
- Rewriting the timeline from memory rather than reconstructing it from telemetry.
How Obseria helps
Obseria keeps alerts, deployments, traces, logs and chat activity on one timeline, so the factual part of a postmortem can be assembled from evidence rather than recollection. Obseria Intelligence documents its root-cause investigation as a chain of evidence that can be pasted into the review, and incident data can be synced with ServiceNow for follow-up tracking.