The incident management process step by step: from alert to postmortem (with template)

An incident management process is the agreed sequence of steps, roles and communication rules your organisation follows from the moment something breaks until the lessons are written down and acted on. Without one, every outage is improvised: five engineers debug the same symptom, nobody tells customers what is going on, and two weeks later nobody remembers what actually happened.

This guide walks through the incident management process step by step, from the first alert to the postmortem. It is deliberately tool-neutral, so you can apply it whether you use a dedicated incident platform, a chat channel and a spreadsheet, or something in between. You will find an example severity matrix (SEV1 to SEV4) and a copyable postmortem template at the end that you can adapt to your own team.

What counts as an incident (and what does not)

Before you design a process, agree on a definition. A practical one: an incident is any unplanned event that degrades or threatens to degrade a service for users, and that needs a coordinated response. The second half matters. A single failed pod that Kubernetes restarts on its own is not an incident. A failing payment provider that blocks checkout for a fraction of users is.

Write down a few concrete triggers so that on-call engineers do not have to debate whether to declare:

• A customer-facing SLO is burning its error budget at a rate that will exhaust it within hours or days.
• A critical user journey (login, checkout, data export) fails in synthetic checks or real-user monitoring.
• Data loss, data corruption or a suspected security event, regardless of visible user impact.
• Any situation where the person on call needs help from another team to restore service.

The rule of thumb many teams adopt: declare early, downgrade freely. Declaring an incident that turns out to be minor costs a few minutes. Declaring too late costs hours of uncoordinated work.

Incident roles: who does what

The most important principle of incident response is separating coordination from hands-on debugging. The person fixing the database should not also be writing customer updates. For anything above minor severity, assign at least these roles:

Incident commander (IC)

The incident commander owns the incident, not the fix. The IC sets priorities, assigns tasks, decides when to escalate, makes the call on risky mitigations (rollback, failover, feature flag off) and declares the incident resolved. A good IC asks questions like "what do we know, what are we trying next, and who is doing it?" and avoids disappearing into the logs. The IC can hand over the role explicitly, for example at a shift change, and that handover should be noted in the timeline.

Communications lead

The communications lead owns all outbound information: internal stakeholder updates, status page entries and, where needed, coordination with customer support and account managers. They translate technical findings into plain language, keep to the agreed update cadence and shield the responders from "any news?" messages.

Operations and subject matter experts

Ops engineers and subject matter experts (SMEs) do the investigation and the mitigation. They are pulled in by the IC based on the affected components: database, network, a specific service team, a cloud provider liaison. They report findings back to the IC and propose actions, rather than making large changes silently.

Scribe (optional, recommended for SEV1)

For long or severe incidents, a scribe records the timeline: what was observed, what was decided, what was changed and when. In smaller teams the communications lead or IC often covers this, but a dedicated scribe makes the postmortem dramatically easier.

Severity levels: an example SEV1–SEV4 matrix

Incident severity determines how many people are paged, how fast they must respond and how often you communicate. The matrix below is an example to adapt, not a standard; the response times and cadences are illustrative and should reflect your own SLAs, team size and on-call setup.

Severity	Impact (example)	Response time (example)	Who is paged (example)	Update cadence (example)
SEV1	Core service down or data loss for many or all customers; security breach suspected	Acknowledge within 5 minutes, IC assigned within 15 minutes	Primary and secondary on-call, incident commander, communications lead, engineering manager; executives informed	Internal every 15 minutes, status page every 30 minutes
SEV2	Major feature degraded or unavailable for a significant group of customers; workaround may exist	Acknowledge within 15 minutes	Primary on-call, owning team, incident commander; communications lead on demand	Internal every 30 minutes, status page every 60 minutes
SEV3	Minor feature degraded, limited customer impact, or elevated error budget burn without visible outage	Acknowledge within 1 hour during business hours	Primary on-call of the owning team	Internal update at start and resolution; status page optional
SEV4	No current customer impact; risk that could become an incident (e.g. failing backups, expiring certificate)	Next business day	Ticket to the owning team, no page	None; tracked in the ticket

Two practical tips: tie severity to user impact, not to how hard the problem is technically, and allow the IC to change severity at any time. An incident that starts as SEV3 and becomes SEV1 after twenty minutes is normal.

The incident management process step by step

Every organisation names the phases a little differently, but the core sequence is consistent. Here is a seven-step version that works well in practice.

1. Detect. An alert fires, a synthetic check fails, or a customer reports a problem. The quality of this step is measured by MTTD (mean time to detect). Symptom-based alerts on SLOs and burn rates detect real user impact faster and with less noise than threshold alerts on individual hosts.
2. Triage and declare. The on-call engineer acknowledges the alert, checks impact and decides whether this is an incident. If yes, they declare it with an initial severity, open a dedicated channel (for example #inc-2026-10-17-checkout) and start the timeline.
3. Assemble. The IC is assigned (often the on-call engineer until someone takes over), and the IC pages the communications lead and the needed SMEs according to the severity matrix.
4. Mitigate. Focus on restoring service first, root cause second. Typical mitigations: roll back the last deployment, toggle a feature flag, fail over to another region, scale out, or rate-limit a noisy client. A linked runbook for known failure modes saves valuable minutes here.
5. Communicate. In parallel with mitigation, the communications lead posts the first internal update and, for customer-visible impact, the first status page entry. This continues at the agreed cadence until resolution.
6. Resolve. When user impact has ended and the system is stable, the IC declares the incident resolved, posts a final update and schedules the postmortem. Monitoring stays heightened for a while in case the issue returns.
7. Learn. Write a blameless postmortem, agree on follow-up actions with owners and due dates, and track them to completion.

Time to recovery across these steps is usually tracked as MTTR; our explainer on what MTTR is covers the different definitions and why it is worth measuring each phase separately.

Communication cadence and status page updates

Silence during an incident is interpreted as "nobody is working on it". A fixed cadence removes that doubt, even when there is nothing new to report. Each update, internal or external, should answer four questions: what is affected, what is the current impact, what are we doing, and when is the next update.

For status pages, keep it factual and free of internal jargon. A simple structure that works for most incidents:

• Investigating: "We are investigating elevated error rates for checkout. Some customers may be unable to complete payments. Next update in 30 minutes."
• Identified: "We have identified the cause as a failing dependency and are rolling out a fix. Payments may still fail intermittently."
• Monitoring: "A fix has been deployed and error rates have returned to normal. We are monitoring the situation."
• Resolved: "The incident is resolved. Payments have been working normally since 14:52 UTC. A summary will follow."

Avoid speculative root causes and estimated fix times you cannot keep. Use a single time zone (UTC is the safest choice for distributed teams) in all updates and in the timeline.

Timeline keeping during the incident

The timeline is the raw material for the postmortem, and it is almost impossible to reconstruct accurately afterwards. Record entries as they happen, with a timestamp and a short factual statement. Good timeline entries cover:

• Observations: "14:07 UTC, p99 latency of payment-service above 4 s, error rate 12 %."
• Decisions: "14:15 UTC, IC decides to roll back release 2026.10.17-3."
• Actions and their effect: "14:21 UTC, rollback complete; error rate unchanged."
• Role changes and severity changes: "14:30 UTC, severity raised to SEV1, new IC: on-call SRE."
• Communications: "14:32 UTC, status page updated (identified)."

Links to the dashboards, traces, deployments and log queries you looked at are worth adding, because they make the evidence reproducible later. Many teams pin a timeline document or use incident tooling that captures channel messages and alerts automatically.

The blameless postmortem

A postmortem is a written review of an incident that explains what happened, why it happened and what will change. "Blameless" means the review assumes that people acted reasonably given the information and tools they had at the time. The question is never "who made the mistake?" but "what made the mistake possible and not caught earlier?"

This is not about being soft. Blame produces worse data: people hide details, timelines get vague and the same failure mode returns. A blameless culture produces honest timelines and systemic fixes. Practical rules:

• Write the draft within a few working days, while memories are fresh.
• Describe actions by role ("the on-call engineer") rather than by name where possible.
• Look for contributing factors across the whole system: missing alerts, unclear runbooks, risky deployment paths, unclear ownership.
• Use a structured method such as the five whys or a fault tree; our guide to root cause analysis compares the common approaches.
• Review the postmortem in a meeting with everyone involved, then share it widely.

Copyable postmortem template

Use the structure below as a starting point. Copy it into your wiki or document tool and remove the sections you do not need.

POSTMORTEM: [short incident title]

1. Metadata
   - Incident ID / date:
   - Severity (initial / final):
   - Duration (start of impact to resolution, UTC):
   - Incident commander:
   - Communications lead:
   - Responders / SMEs:
   - Postmortem author and review date:

2. Summary (3-5 sentences)
   - What happened, who was affected, how it was resolved.

3. Impact
   - Affected services and user journeys:
   - Affected customers / regions (approximate):
   - SLO / error budget consumed:
   - Data loss or security implications (yes/no, details):

4. Timeline (UTC)
   - hh:mm  Detection: how was the issue noticed?
   - hh:mm  Incident declared, severity:
   - hh:mm  Key observations, decisions, actions:
   - hh:mm  Mitigation applied:
   - hh:mm  Resolved:

5. Root cause and contributing factors
   - Triggering event:
   - Underlying cause(s):
   - Contributing factors (detection, tooling, process, ownership):

6. What went well

7. What went poorly

8. Where we got lucky

9. Follow-up actions
   - Action | Type (prevent / detect / mitigate / process) | Owner | Due date | Ticket

10. Lessons learned and open questions

Follow-up actions that actually get done

A postmortem without completed actions is documentation of a future incident. Keep follow-ups few, specific and owned. A useful split is by type: actions that prevent recurrence (fix the bug, add a safeguard), actions that detect it faster (a new SLO alert, a synthetic check), actions that mitigate it faster (a runbook, an automated rollback) and process changes (ownership, on-call rotation, review steps).

Use this checklist when closing the postmortem:

• Every action has exactly one owner and a due date.
• Every action lives in the normal backlog of the owning team, with a link back to the postmortem.
• High-priority actions from SEV1 incidents are reviewed in a recurring meeting until done.
• Detection and runbook improvements are verified, for example by replaying the scenario in a game day.
• Recurring incident patterns are tracked across postmortems, not only within one.

How Obseria helps

Obseria is an OpenTelemetry-native observability platform that covers the detection, response and learning steps of the process in one place, with data stored in the EU. Alerts, traces, logs and topology share the same context, so responders can move from a burning SLO to the affected service and its recent deployments without switching tools.

• Incident management with roles, severity levels and an automatically captured timeline of alerts, changes and actions.
• Status pages that the communications lead can update directly from the incident.
• Obseria Intelligence proposes probable root causes with the supporting evidence, and Automation runs runbooks and rollbacks as workflows.
• SLOs with burn-rate alerts to detect user-facing impact early and with less noise.

If you want to see how this works with your own telemetry, start a free trial or book a demo.