What is agentic observability?

Agentic observability is the discipline of collecting, correlating, and analyzing telemetry from AI agent systems — autonomous software components that use large language models (LLMs) to perceive context, make decisions, invoke tools, and produce outputs across multi-step workflows.

A conventional HTTP service is observable if you can trace a request from ingress to database and back. An AI agent is observable if you can trace a reasoning chain: the initial prompt, each LLM call and its output, every tool invocation, any sub-agent delegations, memory reads and writes, and the final response — with latency, token counts, cost, and quality metrics attached to every step.

The term "agentic" distinguishes these systems from single-turn LLM integrations. A chat completion call is easy to observe — it's just a slow HTTP request. Agents are fundamentally different: they run for seconds to minutes, make branching decisions, call external APIs, spawn child agents, and can fail in ways that have no precedent in traditional application monitoring.

Note:Scope of this guide: We focus on server-side agent infrastructure — the kind deployed by enterprise engineering teams running LangChain pipelines, CrewAI orchestrators, AutoGen workflows, or custom agent frameworks on their own infrastructure. Consumer-facing chatbot UX monitoring is a separate discipline.

Why traditional APM fails for AI agents

Most observability platforms were architected around three assumptions that break down completely with AI agents:

Request-response model

Agent runs span seconds to minutes, branch non-deterministically, and may never return a response at all (tool loops, context exhaustion).

Deterministic execution

The same input produces different outputs across runs. Latency histograms and error rates are necessary but not sufficient signals.

Binary success/failure

An agent can return a response that is syntactically correct but factually wrong or hallucinated — invisible to traditional health checks.

Beyond these structural mismatches, traditional APM tools have no concept of tokens (the unit of LLM cost and latency), prompt context (the input that determines behavior), or tool call chains (the sequence of external API calls an agent makes in pursuit of a goal). Grafana dashboards showing p99 latency are useful context but provide almost no debugging signal for agent failures.

The four pillars of agentic observability

After instrumenting over 200 agent deployments across obseria.io's enterprise customer base, we've converged on four pillars that together provide complete visibility into agentic systems:

1
Full trace coverage of every agent step
Every LLM call, tool invocation, memory operation, and sub-agent delegation must produce an OpenTelemetry span. The parent span represents the agent run; child spans represent each action. Without complete span coverage, debugging is guesswork.
2
Capture LLM I/O — prompts, completions, token counts
The input to each LLM call (system prompt + context window) and the output (completion text, finish reason, usage) must be captured and stored with the span. Token counts determine cost; the completion determines behavior.
3
Agent-to-agent correlation
In multi-agent systems, when an orchestrator delegates to a sub-agent, the W3C trace context must propagate across the boundary. Every span in the delegated chain must be a child of the originating orchestrator span, giving you one cohesive trace for the entire task.
4
Quality metrics alongside performance metrics
Response latency and error rate tell you if the system is up; they cannot tell you if it is producing correct outputs. Evals-in-production — automated checks on output quality, hallucination proxies, policy compliance — must run asynchronously and feed back into the same observability pipeline.

Key signals: what to capture from agents

Agentic telemetry falls into three categories. You need all three — gaps in any category leave blind spots that make post-incident investigation impossible.

Spans (traces)

Each of the following agent actions should produce its own span, nested under the parent agent-run span:

LLM call: gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.response.finish_reason
Tool invocation: tool.name, tool.call.arguments (sanitized), tool.call.result (truncated), tool.call.duration_ms
Retrieval step (RAG): rag.query, rag.retrieved_chunks_count, rag.top_score, vector_store.name
Memory read/write: memory.operation (read|write|search), memory.key, memory.store_type
Sub-agent delegation: agent.id, agent.name, agent.model, parent trace context propagated via W3C traceparent

Metrics

The time-series signals you should be alerting on:

yaml

# Critical agentic metrics
agent_run_duration_seconds      # histogram — time from start to final output
agent_run_total                 # counter — total runs, labelled by status (success|error|timeout)
llm_tokens_total                # counter — cumulative tokens, labelled by model and direction (input|output)
llm_call_duration_seconds       # histogram — latency per model call (p50, p95, p99)
tool_call_total                 # counter — labelled by tool name and status
tool_call_duration_seconds      # histogram — per-tool latency

# Quality metrics (async, from eval pipeline)
agent_output_quality_score      # gauge — 0.0–1.0, from automated eval
agent_policy_violations_total   # counter — outputs flagged by guardrails
agent_hallucination_proxy_total # counter — factual contradiction detections

Logs

Structured log lines that agents should emit at key decision points:

json

// Agent decision log — emitted at each reasoning step
{
  "timestamp": "2026-07-18T09:42:31.112Z",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id":  "00f067aa0ba902b7",
  "agent.id": "research-agent-v2",
  "event":    "tool_selected",
  "tool":     "web_search",
  "reasoning_summary": "User asked for current pricing; knowledge cutoff exceeded",
  "alternatives_considered": ["answer_from_memory", "ask_user"],
  "confidence": 0.87
}

OpenTelemetry gen_ai semantic conventions

The OpenTelemetry community ratified the gen_ai semantic conventions in 2025. These define a standard set of span attributes for LLM calls, making agent telemetry portable across vendors and tools. Any instrumentation you write should target these conventions.

python

from opentelemetry import trace
from opentelemetry.semconv.ai import SpanAttributes  # pip install opentelemetry-semantic-conventions-ai

tracer = trace.get_tracer("my-agent")

with tracer.start_as_current_span("llm.call") as span:
    # Standard gen_ai attributes
    span.set_attribute(SpanAttributes.LLM_SYSTEM, "openai")         # gen_ai.system
    span.set_attribute(SpanAttributes.LLM_REQUEST_MODEL, "gpt-4o") # gen_ai.request.model
    span.set_attribute(SpanAttributes.LLM_REQUEST_MAX_TOKENS, 2048)# gen_ai.request.max_tokens
    span.set_attribute(SpanAttributes.LLM_REQUEST_TEMPERATURE, 0.1)# gen_ai.request.temperature

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=messages,
        max_tokens=2048,
        temperature=0.1,
    )

    # Capture response attributes
    span.set_attribute(SpanAttributes.LLM_USAGE_PROMPT_TOKENS,     # gen_ai.usage.input_tokens
        response.usage.prompt_tokens)
    span.set_attribute(SpanAttributes.LLM_USAGE_COMPLETION_TOKENS, # gen_ai.usage.output_tokens
        response.usage.completion_tokens)
    span.set_attribute(SpanAttributes.LLM_RESPONSE_MODEL,           # gen_ai.response.model
        response.model)
    span.set_attribute("gen_ai.response.finish_reason",
        response.choices[0].finish_reason)

Tip:Always capture gen_ai.usage.input_tokens and gen_ai.usage.output_tokens separately. Input tokens are 3–5× cheaper than output tokens across all major providers, and debugging cost anomalies requires knowing which direction is inflating your bill.

Instrumenting LangChain, CrewAI, and custom agents

LangChain with OpenTelemetry

LangChain's callback system makes it straightforward to emit OTel spans for every chain step, LLM call, and tool invocation:

python

from langchain.callbacks.base import BaseCallbackHandler
from opentelemetry import trace, context
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

class OtelCallbackHandler(BaseCallbackHandler):
    def __init__(self):
        self.tracer = trace.get_tracer("langchain")
        self._span_stack: dict = {}

    def on_chain_start(self, serialized, inputs, run_id, **kwargs):
        span = self.tracer.start_span(
            f"langchain.chain.{serialized.get('name', 'unknown')}",
            attributes={"langchain.chain.inputs": str(inputs)[:512]},
        )
        self._span_stack[str(run_id)] = (span, context.attach(trace.set_span_in_context(span)))

    def on_chain_end(self, outputs, run_id, **kwargs):
        span, token = self._span_stack.pop(str(run_id), (None, None))
        if span:
            span.set_attribute("langchain.chain.outputs", str(outputs)[:512])
            span.end()
            if token: context.detach(token)

    def on_llm_start(self, serialized, prompts, run_id, **kwargs):
        span = self.tracer.start_span("langchain.llm.call", attributes={
            "gen_ai.system":         serialized.get("name", "unknown"),
            "gen_ai.request.model":  serialized.get("kwargs", {}).get("model_name", "unknown"),
            "gen_ai.prompt.length":  sum(len(p) for p in prompts),
        })
        self._span_stack[str(run_id)] = (span, context.attach(trace.set_span_in_context(span)))

    def on_llm_end(self, response, run_id, **kwargs):
        span, token = self._span_stack.pop(str(run_id), (None, None))
        if span and response.llm_output:
            usage = response.llm_output.get("token_usage", {})
            span.set_attribute("gen_ai.usage.input_tokens",  usage.get("prompt_tokens", 0))
            span.set_attribute("gen_ai.usage.output_tokens", usage.get("completion_tokens", 0))
            span.end()
            if token: context.detach(token)

    def on_tool_start(self, serialized, input_str, run_id, **kwargs):
        span = self.tracer.start_span("langchain.tool.call", attributes={
            "tool.name":            serialized.get("name", "unknown"),
            "tool.call.arguments":  input_str[:256],
        })
        self._span_stack[str(run_id)] = (span, context.attach(trace.set_span_in_context(span)))

    def on_tool_end(self, output, run_id, **kwargs):
        span, token = self._span_stack.pop(str(run_id), (None, None))
        if span:
            span.set_attribute("tool.call.result", str(output)[:256])
            span.end()
            if token: context.detach(token)

CrewAI multi-agent tracing

In multi-agent CrewAI workflows, trace context must propagate from the orchestrator crew to each spawned agent. Without explicit propagation, each agent's work appears as an isolated trace disconnected from the task that triggered it:

python

from crewai import Agent, Task, Crew
from opentelemetry import trace
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

tracer = trace.get_tracer("crewai")
propagator = TraceContextTextMapPropagator()

def run_crew_with_tracing(task_description: str) -> str:
    with tracer.start_as_current_span("crew.run") as root_span:
        root_span.set_attribute("crew.task", task_description[:256])

        # Inject trace context into carrier for sub-agent propagation
        carrier: dict = {}
        propagator.inject(carrier)

        researcher = Agent(
            role="Senior Researcher",
            goal="Find accurate, up-to-date information",
            backstory="Expert researcher with web access",
            # Pass carrier so sub-agent calls carry the root trace context
            metadata={"otel_carrier": carrier},
        )

        analyst = Agent(
            role="Data Analyst",
            goal="Synthesize research into actionable insights",
            backstory="Expert at distilling complex information",
            metadata={"otel_carrier": carrier},
        )

        research_task = Task(description=task_description, agent=researcher)
        analysis_task = Task(description="Analyze and summarize findings", agent=analyst)

        crew = Crew(agents=[researcher, analyst], tasks=[research_task, analysis_task])

        with tracer.start_as_current_span("crew.kickoff") as kickoff_span:
            result = crew.kickoff()
            kickoff_span.set_attribute("crew.result.length", len(str(result)))

        root_span.set_attribute("crew.status", "completed")
        return result

Warning:W3C trace context propagation across process boundaries (e.g. when sub-agents run in separate containers or processes) requires explicit header injection and extraction. Do not assume framework-level magic will handle this — it almost never does. Test by verifying that sub-agent spans appear as children of the orchestrator span in your trace UI.

Failure modes unique to agentic systems

AI agents fail in ways that are categorically different from conventional software. Your alerting strategy must account for all of them:

Tool call loops

An agent repeatedly invokes the same tool (or sequence of tools) without making progress toward the goal. Common cause: the tool returns an error the agent misinterprets as requiring another call. Detect by alerting when a single agent run executes the same tool more than N times.

Context window exhaustion

As agents accumulate tool results and conversation history, the context window fills. Most LLMs silently truncate input rather than returning an error — the agent loses its earlier reasoning and its behavior degrades. Monitor gen_ai.usage.input_tokens against the model's context limit per call.

Silent hallucination

The agent returns a plausible-looking response with a 200 status code. No exception is raised. Traditional health checks pass. But the output is factually incorrect. This requires async evaluation: running a judge model or rule-based checker against sampled outputs.

Prompt injection via tool output

An adversarial payload in a tool result (e.g. a retrieved document or web page) instructs the agent to ignore its system prompt and take unauthorized actions. Monitor for unexpected tool invocations and sudden changes in agent behavior relative to its configured goal.

Enterprise considerations: compliance, cost, multi-tenancy

Data privacy and compliance

Capturing LLM prompts and completions creates a comprehensive record of what information entered the agent and what it produced. In regulated industries (healthcare, finance, legal), this data likely constitutes PHI or PII under HIPAA, GDPR, or CCPA. Before capturing raw LLM I/O:

Classify what categories of data may appear in prompts (customer records, medical history, financial data)
Apply field-level redaction in your instrumentation layer before the span is exported to your observability backend
Configure short retention windows for spans containing PII — 7–30 days is typical for production LLM traces
Ensure your observability vendor has a signed BAA (for HIPAA) or DPA (for GDPR) covering AI telemetry data

Cost attribution

LLM token costs are the dominant variable expense in agent infrastructure. Without per-request cost attribution, you cannot answer fundamental questions: which workflows cost the most, which customers are driving your AI bill, and whether a model upgrade pays for itself in reduced token consumption.

python

# Cost attribution via span attributes
MODEL_COSTS_PER_1K_TOKENS = {
    "gpt-4o":             {"input": 0.0025, "output": 0.010},
    "gpt-4o-mini":        {"input": 0.00015,"output": 0.0006},
    "claude-3-5-sonnet":  {"input": 0.003,  "output": 0.015},
    "claude-3-haiku":     {"input": 0.00025,"output": 0.00125},
}

def attach_cost_attributes(span, model: str, input_tokens: int, output_tokens: int):
    costs = MODEL_COSTS_PER_1K_TOKENS.get(model, {"input": 0, "output": 0})
    input_cost  = (input_tokens  / 1000) * costs["input"]
    output_cost = (output_tokens / 1000) * costs["output"]
    span.set_attribute("llm.cost.input_usd",  round(input_cost,  6))
    span.set_attribute("llm.cost.output_usd", round(output_cost, 6))
    span.set_attribute("llm.cost.total_usd",  round(input_cost + output_cost, 6))

Multi-tenant agent systems

If your agent platform serves multiple customers or business units, every span must carry a tenant identifier so you can filter telemetry by customer and enforce data isolation. The cleanest approach is to propagate tenant ID via OTel baggage, which flows automatically to all child spans:

python

from opentelemetry.baggage import set_baggage, get_baggage
from opentelemetry import context

def run_agent_for_tenant(tenant_id: str, user_request: str):
    # Attach tenant to OTel baggage — propagates to all child spans automatically
    ctx = set_baggage("tenant.id", tenant_id)
    ctx = set_baggage("tenant.tier", get_tenant_tier(tenant_id), context=ctx)
    token = context.attach(ctx)
    try:
        with tracer.start_as_current_span("agent.run") as span:
            # tenant.id is in baggage, but also set on the root span for easy filtering
            span.set_attribute("tenant.id",   tenant_id)
            span.set_attribute("tenant.tier", get_tenant_tier(tenant_id))
            return run_agent(user_request)
    finally:
        context.detach(token)

Implementing agentic observability with obseria.io

obseria.io's Agent Telemetry tier is purpose-built for the signals described in this guide. The key differences from general-purpose observability:

Native gen_ai attribute indexing — filter by model, finish reason, token count, and cost without custom attribute configuration
Agent run timeline view — a visual trace UI that renders the reasoning chain as a sequential timeline, not a waterfall of spans
Built-in token cost calculator — attach model name and token counts; cost attribution is automatic across all providers
Eval pipeline integration — connect your eval framework (RAGAS, DeepEval, custom judges) to push quality scores as span events
Context window utilization alerts — pre-built alert rule that fires when any LLM call uses >85% of the model's context limit
Tool loop detection — automatic anomaly detection for repeated tool invocations within a single agent run

Note:obseria.io ingests agent telemetry via standard OTLP over gRPC or HTTP. No proprietary SDK required — point your existing OpenTelemetry exporter at ingest.obseria.io:4317 and the gen_ai attributes are indexed automatically. A 14-day free trial includes full Agent Telemetry access.

Lena Hartmann

Staff Engineer · obseria.io

Lena works on obseria.io's ingest and sampling infrastructure and leads the Agent Telemetry product area. She has been instrumenting production AI systems since 2024 and contributes to the OpenTelemetry GenAI SIG.

Engineering

Why tail-based sampling is the only sane default for high-volume tracing

11 min read

OpenTelemetry

Running the OpenTelemetry Collector at production scale: what we learned

9 min read

Ready to observe your agents?

obseria.io ingests gen_ai telemetry natively — no custom pipelines, no extra configuration. See your first agent trace in under 5 minutes.