An AI SRE agent that investigates with evidence

What is an AI SRE agent?

An AI SRE agent is software that takes on parts of the work a site reliability engineer does during an incident: reading alerts, collecting context, forming and testing hypotheses about the cause, and suggesting what to do next. Unlike a chatbot that answers questions, an agent runs a sequence of steps on its own, such as querying data, comparing time ranges and checking recent changes.

The goal is not to replace on-call engineers. It is to shorten the time between "an alert fired" and "we understand what is happening", and to take repetitive investigation and documentation work off the people who are woken up at night.

Why AI for incident response

Modern systems consist of many services, clusters and dependencies. When something breaks, the relevant evidence is spread across logs, metrics, traces, deployment history and configuration changes. Engineers spend a large part of an incident just gathering that context, often under time pressure and with incomplete knowledge of services owned by other teams.

An agent can run those first steps consistently every time. It does not get tired, it checks the same sources in the same order, and it can work through many signals in parallel. The value depends on two things: access to well-structured telemetry with topology, and transparency about how it reached its conclusions.

How the Obseria AI Agent works

The Obseria AI Agent is part of Obseria Intelligence. When an alert or SLO burn-rate alert fires, it starts an investigation:

1. Triage: it groups related alerts, identifies affected services and estimates user impact based on SLOs and request data.
2. Context: it uses TopoGraph to find upstream and downstream dependencies and checks recent deployments, configuration changes and automation runs.
3. Investigation: it queries logs, metrics and traces in Obseria Lake with OQL, PromQL or SQL, compares the incident window with normal behavior and narrows down likely causes.
4. Evidence: it presents its findings as a chain of evidence, with each step linked to the query and data behind it.
5. Proposal: it suggests remediation, for example a rollback of the last deployment or a runbook from Obseria Automation.

The investigation appears in the incident, so everyone on the call sees the same information. Engineers can ask follow-up questions, request additional checks or discard a hypothesis.

Guardrails and human approval

Letting software change production systems requires clear boundaries. The Obseria AI Agent does not execute actions on its own by default. You decide which actions it may propose, which require explicit approval from a person and which may run automatically within defined limits, for example restarting a single pod in a non-production namespace.

Permissions follow the role-based access control of your Obseria organization. Every proposal, approval and action is recorded in the incident timeline and audit log, so it is always clear who approved what and when. The agent can be wrong, and the evidence chain exists precisely so that people can check its reasoning before acting on it.

Incident summaries and postmortem drafts

After an incident, writing up what happened often takes longer than fixing it. The agent drafts an incident summary with timeline, affected services, impact, actions taken and the evidence collected during the investigation. For a postmortem, it prepares a draft structure with contributing factors and open questions that the team then reviews, corrects and completes. The draft is a starting point; conclusions and follow-up actions remain the responsibility of the team.

Getting good results from an AI SRE agent

• Instrument consistently. Use OpenTelemetry semantic conventions and consistent service names so the agent can connect signals.
• Track deployments. Send deployment and configuration events; many incidents start with a change.
• Define SLOs. They give the agent a clear notion of user impact and priority.
• Start with approval for everything. Widen automatic actions step by step once you trust specific workflows.
• Review its work. Treat findings as a well-prepared hypothesis and give feedback when it is wrong.