How to reduce alert fatigue: a practical guide for SRE and on-call teams

If you want to reduce alert fatigue, start with an uncomfortable question: how many of last week's pages actually required a human to do something? In many teams the honest answer is "fewer than half". The rest were duplicates, flapping thresholds, warnings nobody owns or symptoms of an incident that was already being handled. Each of them teaches on-call engineers that alerts can be ignored, which is how the one important page gets missed.

Alert fatigue is not a personal resilience problem, it is a design problem in your alerting. This guide walks through the changes that remove noise in a predictable order: alert on symptoms instead of causes, move to SLO burn-rate alerts, deduplicate and group, route by ownership and severity, attach a runbook to every page and review the alert set on a fixed schedule. It ends with a copyable alert review checklist, a decision table for every alert type and two metrics that tell you whether it is working.

What causes alert fatigue

Alert fatigue builds up slowly. Each alert was added for a reason, usually after an incident, and almost none are ever removed. The typical root causes are:

• Cause-based thresholds: CPU above 80 percent, disk above 70 percent, one pod restarted. These fire whenever the system is busy, not when users are affected.
• One alert per instance: a database failover produces forty pages, one per dependent service and host, instead of one incident.
• No owner: alerts route to a shared channel or to whoever is on call, even when only one team can fix the problem.
• No defined action: the alert fires, the engineer looks at a dashboard, sees that it recovered and goes back to sleep. Nothing was learned and nothing changes.
• Everything is urgent: warnings and critical problems use the same channel, the same sound and the same escalation, so the severity carries no information.

The result is a team that acknowledges pages reflexively, investigates late and burns out. Being on call should be demanding during real incidents and quiet the rest of the time. If it is noisy every night, the alert set is broken, not the people.

Symptom-based vs cause-based alerting

The single most effective change is to page on symptoms and investigate causes. A symptom is something a user or a downstream consumer experiences: requests fail, checkout is slow, messages arrive late. A cause is something that might lead to a symptom: high CPU, a full queue, a restarted container.

Cause-based alert (investigate)	Symptom-based alert (page)
CPU on api-7 above 85 percent for 5 minutes	Error rate of the checkout API above its SLO budget
Kafka consumer lag above 10,000 messages	Orders confirmed later than 2 minutes after payment
Pod restarted 3 times	p95 latency of the login endpoint above 800 ms
Database connections above 90 percent of pool	Synthetic check for the main user journey failing from two regions

The thresholds in this table are examples, not recommendations. Cause-based signals are still valuable: they belong on dashboards, in tickets or as context attached to the symptom alert, so the responder sees "error rate high, and the connection pool on the primary is saturated" in one place. They should rarely wake anyone up on their own. The exceptions are causes that predict a symptom with certainty and leave little time to react, such as a disk that will be full within the next hour or a certificate that expires tomorrow.

SLO and burn-rate alerts

Symptom-based alerting becomes precise when you define it through service level objectives. You pick a service level indicator such as the share of successful requests, set a target such as 99.9 percent over 30 days and derive an error budget from it: 0.1 percent of requests may fail. If you are new to the concept, our introduction to what an SLO is covers the basics.

Instead of alerting on the error rate directly, you alert on the burn rate, which is how fast the budget is being consumed. A burn rate of 1 uses exactly the budget over the SLO window; a burn rate of 10 would exhaust a 30-day budget in three days. The Google SRE Workbook (chapter "Alerting on SLOs") describes a multi-window, multi-burn-rate approach that many teams use as a starting point:

• Page: burn rate above 14.4 over the last hour, confirmed by the last 5 minutes. That consumes about 2 percent of a 30-day budget in one hour.
• Page: burn rate above 6 over the last 6 hours, confirmed by the last 30 minutes, which consumes about 5 percent of the budget.
• Ticket: burn rate above 1 over the last 3 days, confirmed by the last 6 hours, which consumes about 10 percent of the budget.

The short confirmation window makes the alert resolve quickly once the problem is fixed, and the long window filters out brief spikes. One SLO with two or three burn-rate rules can replace dozens of threshold alerts on error rate and latency. For release gates and budget policies built on the same numbers, see our article on SLOs in practice.

Deduplication, grouping and suppression

Even good alerts become noise when one failure triggers them many times. Three mechanisms keep a single problem to a single notification:

• Deduplication: repeated firings of the same alert with the same labels update one open alert instead of sending a new notification. Define a stable fingerprint, for example alert name plus service plus environment, and exclude volatile labels such as pod name.
• Grouping: related alerts that fire within a short window are bundled into one incident, for example all alerts of the payment service in production within five minutes. Group by service and environment first; grouping by topology works even better when your platform knows which services depend on each other.
• Suppression and inhibition: if a higher-level alert is active, dependent alerts are held back. When the primary database is down, nobody needs separate pages for the forty services that cannot reach it. Also suppress alerts during planned maintenance windows and deployments that are known to cause short blips.

Add hysteresis to the remaining threshold alerts: require the condition to hold for several minutes before firing and to be clear for several minutes before resolving. This removes most flapping alerts that fire and resolve repeatedly around a threshold.

Routing, severity and escalation

A page should reach the person who can fix the problem, through a channel that matches the urgency. Three rules cover most setups:

1. Route by ownership: every alert carries a service or team label, and routing uses that label. Alerts without an owner are not allowed into the paging path.
2. Separate severities by channel: use clear incident severity levels. Only the top levels page by phone or push notification; lower levels create a ticket or a message in a team channel during working hours.
3. Escalate on silence, not on noise: if the primary on-call engineer does not acknowledge within a defined time, escalate to the secondary and then to the team lead. Do not escalate simply because an alert is still firing after it has been acknowledged.

Keep the routing rules as code next to the alert definitions, so "why did this page me?" is answered in one file.

A runbook for every page

An alert without a known next step forces the responder to start from zero at the worst possible time. Every alert that pages should link to a runbook with a short, fixed structure. A useful template:

Alert:        checkout-availability-burn-rate-fast
Owner:        team-payments
Severity:     SEV2 (pages primary on-call)
What it means: Checkout requests are failing fast enough to use
               about 2% of the monthly error budget per hour.
User impact:   Some customers cannot complete orders.
First checks:  1. Recent deploys of checkout or payment-gateway
               2. Error breakdown by endpoint and status code
               3. Health of payment provider (status page, latency)
Mitigation:    Roll back last deploy | switch provider | enable
               maintenance banner
Escalate to:   team-platform if the database or network is involved
Last reviewed: 2026-10-01

If you cannot fill in "What it means" and "First checks", that is a strong signal the alert should not page. Runbooks also make automation possible: steps that are always the same, such as collecting diagnostics or rolling back the last deployment, can be executed automatically and attached to the alert before a human even looks at it.

Alert hygiene: the alert review checklist

A one-time clean-up helps for a few months. Lasting results come from a recurring review, for example 30 minutes every month or after every on-call rotation, in which the team goes through every alert that fired. Copy this checklist into your review document:

ALERT REVIEW CHECKLIST (per alert that fired in the period)

[ ] Did a user or downstream system notice a problem?
[ ] Did the responder have to take an action? Which one?
[ ] Could the action have been automated?
[ ] Was the alert a duplicate of another alert or incident?
[ ] Did it fire outside working hours? Was that justified?
[ ] Is the severity correct for the actual impact?
[ ] Did it reach the right team on the first attempt?
[ ] Does it have an owner label and a working runbook link?
[ ] Did it flap (fire and resolve more than twice in an hour)?
[ ] Would an SLO burn-rate alert cover the same case?
[ ] Decision: keep | tune | convert to ticket | delete
[ ] Follow-up owner and due date recorded

Use this decision table to turn the answers into consistent outcomes:

Alert type	Decision	Reason
SLO burn-rate alert, fast window	Keep (page)	Direct user impact and urgent budget consumption
SLO burn-rate alert, slow window	Convert to ticket	Real problem, but it can wait until working hours
Synthetic check of a critical user journey failing from several locations	Keep (page)	Strong symptom signal with low false-positive rate
Disk or certificate predicted to run out within hours	Keep (page)	Certain future impact, little time to react
Disk or certificate predicted to run out within days	Convert to ticket	Impact is certain but not urgent
High CPU or memory on a single host	Delete or move to dashboard	Cause, not symptom; covered by latency and error SLOs
Single pod restart	Delete	Normal in Kubernetes; alert on crash loops or SLO impact instead
Anomaly in a business metric (orders, sign-ups)	Convert to ticket or low-severity notification	Useful context, but often needs confirmation before paging
Alert with no owner and no runbook	Delete after 30 days unless someone claims it	Nobody can act on it
Alert that has not fired in 12 months	Review and test	May be broken, obsolete or still important

Measuring alert quality

Without numbers, alert hygiene turns into a debate about feelings. Two metrics are simple to collect from your paging tool and easy to discuss in a review:

• Actionable-alert ratio: pages that required a human action divided by all pages in the period. Tag each page during the shift or in the review as actionable or not.
• Pages per on-call shift: the number of pages a responder receives per shift, split into working hours and night. The Google SRE book (chapter "Being On-Call") suggests that a shift should allow time to handle each incident properly, including follow-up, and names a maximum of two incidents per 12-hour shift as a guideline. Use that as a reference point, not a hard rule.

Useful secondary metrics are the share of alerts that auto-resolve without acknowledgement, the number of alerts per incident (a high value means grouping is weak), time to acknowledge at night versus during the day and the number of alerts without an owner or runbook. As a purely illustrative example: a team that receives 25 pages per week, of which 8 needed action, has an actionable-alert ratio of 32 percent and a clear list of 17 alerts to review first.

Detection speed matters too. If you remove noise but your incidents are now discovered by customers, you have gone too far. Watch your time to detect alongside the noise metrics, and confirm that every past incident would still have been caught by the new alert set.

How Obseria helps

Obseria is an OpenTelemetry-native observability platform, and its alerting is built around the ideas in this guide: SLOs with multi-window burn-rate alerts, incidents that group related signals and topology that knows which services depend on each other. Obseria Intelligence correlates alerts with deployments, traces and anomalies and shows the evidence behind a suggested root cause, so a responder starts from context instead of a bare threshold.

• SLOs with burn-rate alerting and error-budget tracking per service
• Incident management with grouping, ownership-based routing, escalation and status pages
• Anomaly detection as context for alerts or low-severity notifications instead of extra pages
• Automation that runs runbook steps such as diagnostics or rollbacks and attaches the results to the incident

To see how your current alerts would look as SLOs and grouped incidents, start a free trial with your own telemetry.