AI agent observability is the practice of making the behavior of LLM-based agents visible and explainable. It records every step of an agent run, including model calls, tool invocations, retrieval, memory access and hand-offs between agents, as traces, metrics and events, so teams can debug failures, control cost and latency, and judge output quality.
What is AI agent observability?
AI agent observability applies the principles of observability to software agents that use a large language model to plan and act. Unlike a single chat completion, an agent works in a loop: it reads a goal, asks a model what to do next, calls tools or APIs, retrieves documents, stores intermediate results and decides when it is finished. One user request can therefore trigger dozens of model calls and tool executions, some of them in parallel and some delegated to other agents.
Classic application monitoring sees this as a few slow HTTP requests to a model provider. Agent observability instead reconstructs the full reasoning path: which step was taken, why, with which inputs, at what cost and with what result. It extends LLM observability from individual prompts to multi-step, non-deterministic workflows.
Why agents are hard to observe
- Non-determinism. The same input can lead to a different sequence of steps on every run, so reproducing a failure from logs alone is rarely possible.
- Silent failures. An agent can return HTTP 200 and a fluent answer while having called the wrong tool, looped needlessly or ignored retrieved context.
- Cost and latency multiply. Every additional reasoning step adds model latency and token usage; runaway loops are an operational and a budget risk.
- External dependencies. Tools, vector stores and MCP servers each have their own failure modes, rate limits and timeouts.
- Sensitive content. Prompts and tool arguments can contain personal or confidential data that must not be stored unfiltered.
What to capture: traces, metrics and events
The natural data model for an agent run is a distributed trace. The root span represents the agent invocation, and child spans represent each model call, tool execution, retrieval and sub-agent call. The OpenTelemetry GenAI semantic conventions (status: development) define operation names for this, such as invoke_agent, chat, execute_tool and retrieval, plus attributes like gen_ai.agent.name, gen_ai.request.model, gen_ai.tool.name and gen_ai.usage.input_tokens.
On top of traces, useful metrics include end-to-end duration per agent, number of model and tool calls per run, token consumption per step, tool error rates and the share of runs that end in a fallback or a human hand-off. Prompt and response content is best captured only as opt-in and after masking, typically as span attributes or events with a shorter retention.
Multi-agent systems add another layer. When an orchestrator delegates work to specialized agents, or an agent calls tools exposed by a Model Context Protocol server, each hand-off should carry the trace context. Only then can you see which agent made which decision, where time was spent and which component returned the faulty data. Recording the agent version, the prompt template version and the model version on the root span makes it possible to compare runs before and after a change.
Example: debugging an agent run
An illustrative case: a support agent suddenly takes much longer per ticket. The trace of a slow run shows the following structure:
invoke_agent support-agent 38.2 s
chat gpt-model 2.1 s in 1,850 / out 120 tokens
execute_tool search_orders 0.3 s status=error (timeout)
chat gpt-model 2.4 s
execute_tool search_orders 0.3 s status=error (timeout)
... 9 more retries ...The model is not slow; a tool times out and the agent keeps retrying because its instructions do not limit attempts. Without per-step spans this would look like a generic latency problem with the model provider. The fix is a retry limit in the agent loop and an alert on tool error rates.
Best practices
- Instrument with OpenTelemetry and the GenAI conventions instead of a proprietary SDK, so traces stay portable across frameworks and backends.
- Propagate trace context through tools, sub-agents and MCP servers so one run forms one trace.
- Set hard limits for steps, tokens and wall-clock time per run, and record when a limit is hit.
- Combine runtime telemetry with LLM evaluation: traces show what happened, evals judge whether the result was good.
- Mask personal data before storage and keep full prompt content optional and short-lived.
- Tag runs with agent version, prompt version and model so regressions can be traced to a change.
How Obseria helps
Obseria ingests OpenTelemetry traces, metrics and events from agent frameworks via OTLP and shows each agent run as a trace with model calls, tool calls and token counts per step. The Obseria Pipeline can mask PII in prompts before storage. Agent telemetry sits next to the services and infrastructure the agent calls, so a failing tool can be correlated with its backend. See AI observability for details.
