Alert fatigue is the desensitization that occurs when on-call engineers receive more alerts than they can meaningfully act on. When most pages are noisy, duplicated or non-actionable, people start to ignore, mute or batch-acknowledge them, so real incidents are noticed later and response times grow.
What alert fatigue is
Alert fatigue describes what happens to people and teams when the volume of alerts exceeds their capacity to act on them. The term originated in healthcare, where clinicians exposed to constant monitor alarms begin to tune them out. In software operations the mechanism is identical: if nine out of ten pages turn out to be harmless, engineers learn that the tenth is probably harmless too. They acknowledge without investigating, create mute rules, or simply stop looking at the alert channel.
The dangerous part is that alert fatigue rarely looks like a problem from the outside. Dashboards are green, alert rules exist for everything, and the team appears well covered. The cost only becomes visible during a real incident, when the critical signal was sent on time but nobody reacted to it, and the mean time to resolve stretches far beyond what the tooling would allow.
Typical causes
- Cause-based alerts instead of symptom-based alerts. Paging on high CPU, a full disk queue or a single failed pod produces many alerts that do not affect users at all.
- Static thresholds. A fixed limit such as "latency above 300 ms" fires every Monday morning peak and stays silent during a slow degradation at night. Comparing against a learned baseline is usually more precise.
- Duplicates and alert storms. One failing database triggers alerts in every dependent service, each on its own channel.
- Flapping. Metrics oscillating around a threshold fire and resolve repeatedly because there is no evaluation window or hysteresis.
- No owner and no action. Alerts that nobody owns, or that have no documented response, accumulate over years and are never cleaned up.
Why it matters
Alert fatigue hurts reliability and people at the same time. Missed or delayed pages extend outages and burn through error budgets. Interrupted nights and constant context switching lead to burnout and turnover in on-call rotations, which removes exactly the experienced engineers a team depends on during incidents.
A useful way to measure the problem is to track a few simple numbers per rotation: pages per shift, pages outside business hours, the share of pages that were acknowledged without any follow-up action, and the median time to acknowledge. If time to acknowledge rises while page volume rises, fatigue is very likely setting in. Noise also erodes trust: once engineers believe the alerting system is unreliable, they stop investing in it, and the signal-to-noise ratio gets even worse.
Example: from cause alerts to burn-rate alerts
Consider an illustrative checkout service with a 99.9% availability SLO over 30 days. The team originally had separate pages for CPU above 85%, error rate above 1% over one minute, and every pod restart. A common replacement, described in the Google SRE Workbook, is multi-window, multi-burn-rate alerting on the error budget:
| Severity | Burn rate | Long window | Short window |
|---|---|---|---|
| Page | 14.4x | 1 hour | 5 minutes |
| Page | 6x | 6 hours | 30 minutes |
| Ticket | 1x | 3 days | 6 hours |
A 14.4x burn rate sustained for one hour consumes about 2% of a 30-day budget. Both windows must exceed the threshold, so the alert fires quickly for real problems and resolves quickly once they stop. CPU and restarts become dashboard signals and diagnostic context rather than pages.
Best practices to reduce alert fatigue
- Page on symptoms users feel: errors, latency, saturation of user-facing capacity, ideally expressed as SLO burn rates.
- Make every page actionable. Each alert should link to a runbook and have a clear owner. If there is nothing to do, it should not page.
- Separate urgency levels. Pages for immediate action, tickets for work that can wait until business hours, dashboards for everything else.
- Group and deduplicate related alerts into one incident, using topology to suppress downstream noise.
- Review alerts regularly. Track pages per on-call shift and the share that led to action; delete or downgrade rules that never do.
Common mistakes
Teams often respond to a missed incident by adding another alert, which increases noise further. Others lower thresholds after every outage "to be safe". A third mistake is routing everything to a shared channel where no single person feels responsible. Each of these feels prudent in isolation and makes fatigue worse in aggregate.
How Obseria helps
Obseria supports SLOs and burn-rate alerting out of the box, and anomaly detection learns baselines instead of relying on static thresholds. Obseria TopoGraph knows which services depend on each other, so incident management can group related alerts into a single problem, and Obseria Intelligence attaches a chain of evidence for the likely root cause.
