Alerting that pages people only when it matters

What is alerting in monitoring?

Alerting is the part of monitoring that turns telemetry into action. An alert rule evaluates a condition, for example an error ratio above a threshold or an SLO burning too fast, and notifies the people or systems that need to respond. A good alerting tool does more than send messages: it decides what is worth a notification, groups related signals, routes them to the owning team and provides enough context to start working immediately.

Monitoring alerts come in different urgencies. A page wakes someone up because users are affected right now. A ticket or chat message flags something that needs attention during working hours. Mixing the two is the most common reason why on-call engineers stop trusting their alerts.

Why good alerting matters

Alerting determines how quickly a team notices a problem and how much energy it has left to fix it. Too few alerts and outages are reported by customers. Too many and alert fatigue sets in: notifications are muted, acknowledged without investigation or simply ignored. Both outcomes increase mean time to detect and mean time to resolve.

In distributed systems, a single failure often triggers dozens of symptoms across services, hosts and queues. Without deduplication and grouping, one database problem produces a flood of notifications in several channels, and the actual cause is buried among them.

How alerting works in Obseria

Obseria evaluates alert rules on all signals in Obseria Lake: metrics, logs, traces, events and synthetic checks. Conditions are written in PromQL, OQL or SQL, so existing Prometheus rules can be reused. For SLOs, Obseria creates multi-window burn-rate alerts automatically, which page only when the error budget is being consumed fast enough to matter.

fetch logs
| filter service.name == "payments" and level == "error"
| summarize errors = count(), by:{bin(timestamp, 5m)}
| filter errors > 50

When alerts fire, Obseria groups them using TopoGraph. Alerts on services that depend on the same failing component are combined into one notification instead of many. Repeated firings of the same alert are deduplicated, and flapping conditions are dampened.

Routing and on-call integrations

Routing rules decide where an alert goes based on service ownership, severity, environment and time. Critical alerts can open a page in PagerDuty or Opsgenie, while warnings go to a Slack or Microsoft Teams channel. Webhooks connect any other system. Each notification includes the affected services, the triggering query, recent deployments and links to related logs and traces.

From an alert, responders can declare an incident, update a status page or trigger an automation workflow such as a runbook or rollback. Obseria Intelligence can start an investigation as soon as the alert fires and present a likely root cause with its chain of evidence. The Obseria AI Agent proposes remediation steps and only acts after approval or within the policies you define.

Alerting best practices

1. Page on symptoms. Alert on user-facing errors, latency and SLO burn rates; send cause-level signals like high CPU to dashboards or tickets.
2. Make every page actionable. If nobody needs to act, the alert should not wake anyone up.
3. Define owners. Every alert rule belongs to a team and routes to that team's on-call rotation.
4. Link a runbook. Describe the first diagnostic steps directly in the alert.
5. Review regularly. Remove alerts that are always acknowledged without action and tune thresholds that flap.
6. Manage rules as code. Versioned alert rules make changes reviewable and reproducible across environments.

Reducing alert fatigue

Alert fatigue is rarely solved by a single setting. It requires fewer but better rules, consistent severities and grouping that reflects how your system is built. Obseria supports this with alert analytics that show how often each rule fires, how often it leads to action and how long acknowledgement takes, so you can prune noise based on data rather than opinion.