Root cause analysis (RCA) is a structured method for finding the underlying causes of an incident, rather than only its symptoms, so that effective corrective actions can prevent it from recurring. In software operations, RCA combines telemetry, change history and team knowledge, and usually ends in a blameless postmortem.
Key takeaways
- RCA separates symptoms, triggers and underlying causes. Fixing only the trigger means the incident will return in a different form.
- Common methods are the 5 Whys, Ishikawa (fishbone) diagrams, fault tree analysis and change analysis; each suits different problems.
- Complex distributed systems rarely have a single root cause. Look for contributing factors and treat 'human error' as a starting point, not a conclusion.
- Fast RCA depends on correlated telemetry, topology and change data. AI can assemble the evidence, but conclusions must stay verifiable.
What is root cause analysis?
Root cause analysis (RCA) is the systematic investigation of why a problem happened, with the goal of preventing it from happening again. It originated in engineering disciplines such as manufacturing, aviation and healthcare, and is now a standard practice in IT operations and site reliability engineering. In software, an RCA typically follows a production incident: once service is restored, the team reconstructs what happened, identifies the causes and agrees on actions.
The key idea is the difference between symptoms and causes. Users seeing HTTP 503 errors is a symptom. A database connection pool being exhausted is a proximate cause. A code change that leaked connections on an error path, combined with a missing test and a missing alert on pool saturation, is closer to the underlying causes. Fixing only the symptom, for example by restarting the service, buys time but guarantees a repeat.
Symptom, trigger, root cause and contributing factors
| Term | Meaning | Example |
|---|---|---|
| Symptom | What users or monitors observe | Checkout error rate rises to 12% |
| Trigger | The event that started the incident | Deployment of payments service v4.2 |
| Proximate cause | The immediate technical mechanism | Connection pool exhausted, requests time out |
| Root cause | The underlying condition that allowed it | Connections not released on an error path |
| Contributing factors | Conditions that made it worse or harder to catch | No test for the error path, no pool saturation alert, retry storm from clients |
Modern reliability practice increasingly speaks of contributing factors rather than a single root cause. Richard Cook's widely cited paper "How Complex Systems Fail" argues that catastrophic failures in complex systems require multiple contributors, each necessary but only jointly sufficient. Hunting for exactly one root cause tends to stop the investigation too early.
The RCA process step by step
- Stabilise first. Mitigation comes before analysis. Roll back, fail over or scale, but preserve evidence such as logs, traces, heap dumps and configuration state.
- Build a timeline. Reconstruct events from telemetry: when the SLI started degrading, which alerts fired, which deployments, configuration changes, feature flag flips or infrastructure events happened around that time, and what responders did.
- Define the problem precisely. What was affected, for whom, from when to when, and how severe? A precise problem statement keeps the analysis focused.
- Ask what changed. Compare the failing state with the last known good state: code, configuration, traffic patterns, dependencies, data volume, infrastructure.
- Form and test hypotheses. Use traces to follow slow or failed requests to the component where time or errors originate, logs to confirm the mechanism, and profiles to see which code was running. Discard hypotheses the evidence contradicts.
- Identify causes and contributing factors. Apply a method such as the 5 Whys or a fishbone diagram to go beyond the proximate cause.
- Define corrective actions. Cover three categories: fix the defect, prevent the class of problem, and improve detection and response.
- Document and follow up. Write a blameless postmortem, assign owners and due dates to actions, and track them to completion.
What a good postmortem contains
- Summary and impact: what happened, which users and journeys were affected, for how long, and how much error budget was consumed.
- Timeline: timestamped events from first impact through detection, escalation, mitigation and resolution.
- Causes and contributing factors: the mechanism, the conditions that enabled it and the factors that delayed detection or recovery.
- What went well and what was lucky: luck is a risk that will not repeat reliably.
- Action items: each with an owner, a priority and a due date, linked to your issue tracker.
Keep postmortems searchable. Over time, recurring contributing factors across many incidents, such as missing timeouts or untested failover, point to systemic investments that pay off more than any single fix.
RCA methods compared
| Method | How it works | Strengths | Limitations |
|---|---|---|---|
| 5 Whys | Ask "why?" repeatedly, starting from the problem, until you reach an actionable cause | Fast, no tooling, easy to teach | Follows one causal chain; result depends on who asks |
| Ishikawa (fishbone) diagram | Group possible causes into categories such as code, configuration, infrastructure, dependencies, process and people | Broad view, surfaces multiple factors | Generates candidates, does not prove them |
| Fault tree analysis | Top-down tree with AND/OR logic from the failure to basic events | Rigorous, shows combinations of failures | Time-consuming; best for critical systems |
| Change analysis | Systematically compare the failing state with the last good state | Very effective, since many incidents follow a change | Misses latent issues triggered by load or time |
| Timeline analysis | Order all events and actions chronologically | Reveals detection and response gaps | Needs accurate, correlated timestamps |
In practice teams combine them: a timeline and change analysis to find the mechanism, a fishbone to widen the view, and 5 Whys to drill into each important branch.
Worked example: 5 Whys for a checkout outage
- Why did checkout fail? The payments service returned timeouts for 38 minutes.
- Why did it time out? All database connections in its pool were in use, so new requests waited until they timed out.
- Why was the pool exhausted? Release v4.2 did not return connections to the pool when the fraud check API returned an error.
- Why did that reach production? Tests covered only the success path of the fraud check, and the canary received too little traffic to hit the error path.
- Why did it take 38 minutes to recover? There was no alert on pool saturation, and the first responders investigated the database itself before checking recent deployments.
Notice that the chain branches: questions 4 and 5 lead to different, equally valid actions. Good corrective actions here would be to fix the leak, add error-path tests and fault injection to the pipeline, alert on connection pool saturation, and overlay deployments on service dashboards. None of these is "be more careful".
Telemetry, topology and AI in RCA
In a microservice architecture, the hardest part of RCA is not the method but the evidence. A single user request can touch dozens of services, queues and managed cloud components. Effective RCA needs:
- Distributed traces that show where a request spent its time and where errors originated.
- Logs and events correlated by trace ID and resource attributes, so you can move from a slow span to the exact log lines.
- Metrics and profiles to see resource saturation and which code was consuming CPU or memory.
- Change data: deployments, configuration changes, feature flags and infrastructure events on the same timeline.
- Topology: a current map of dependencies, so you can distinguish a cause upstream from an effect downstream.
AI-assisted RCA uses this context to speed up investigation. Correlation alone is risky, because during an incident many metrics move together. Approaches that reason over the dependency graph (causal analysis) can rank likely causes by following how failures propagate. Agentic approaches go further: they form hypotheses, query telemetry and change history, and present a chain of evidence. The important rule is that every conclusion must be traceable to data a human can verify.
Common RCA pitfalls
- Stopping at "human error". If one person could cause an outage with one command, the system allowed it. Ask what made the action possible and plausible at the time.
- Blame. People who fear consequences hide information. Blameless postmortems produce better data.
- Hindsight bias. Signals that look obvious afterwards were often buried in noise during the incident.
- Actions without owners. Postmortems whose action items are never completed teach nothing.
- Skipping small incidents. Near misses and minor incidents reveal the same weaknesses at lower cost.
How Obseria helps with root cause analysis
Obseria brings the evidence for RCA into one place. Traces, metrics, logs, profiles and events from OpenTelemetry and the Obseria Agent are stored in Obseria Lake with shared context, and Obseria TopoGraph keeps a live map of services, infrastructure and their dependencies. Deployment tracking adds changes to the same timeline. When an incident starts, Obseria Intelligence runs an agentic root-cause investigation across this data, traverses the topology to separate causes from downstream effects and presents its findings as a chain of evidence with links to the underlying traces, logs and changes, so engineers can verify the result and focus on corrective actions.