What is LLM observability?

What is LLM observability?

LLM observability extends classic observability to applications that use large language models: chat assistants, retrieval-augmented generation (RAG) systems, copilots and autonomous AI agents. It answers questions that normal APM cannot: which prompt produced this answer, which documents were retrieved, which tools the agent called and in what order, how many tokens each step consumed, what the request cost, and whether the response was accurate, grounded and safe.

Technically, LLM observability uses the same building blocks as any modern observability practice, namely traces, metrics and logs, and adds two things: GenAI-specific telemetry such as model names, token counts and finish reasons, and evaluations that score the quality of non-deterministic outputs.

Why LLM applications need different observability

• Success is not binary. A model call that returns HTTP 200 can still hallucinate, ignore instructions or leak data. Status codes and latency alone do not tell you whether the system works.
• Behaviour is non-deterministic. The same input can produce different outputs, and a provider-side model update can change behaviour without any deployment on your side.
• Cost is variable per request. Spend depends on prompt length, retrieved context, output length and the number of agent iterations. One runaway agent loop can cost more than thousands of normal requests.
• Agents are multi-step systems. An agent may plan, call a model, invoke tools, query a vector database and call the model again. Without a trace, debugging means guessing which step went wrong.
• External dependencies dominate. Model provider latency, rate limits and outages directly affect users, so they must be measured from your side.
• Prompts contain sensitive data. User inputs and retrieved documents can include personal or confidential information, which changes how telemetry must be collected and stored.

What to monitor: the five dimensions

Dimension	Key metrics	Why it matters
Performance	End-to-end latency, time to first token, time per output token, latency per agent step	Users judge streaming apps by time to first token; agents add latency per step
Cost	Input and output tokens per request, user, feature and model; cost per successful task; prompt cache hit rate	Token spend scales with usage and prompt design and is easy to lose control of
Reliability	Error rate, HTTP 429 rate limits, timeouts, retries, finish reasons such as length or content filter	Provider limits and truncated outputs cause silent failures
Quality	Evaluation scores (relevance, groundedness, correctness), user feedback, task success rate	The only way to see whether answers are actually good
Safety	Guardrail triggers, PII detections, prompt injection attempts, refused or blocked outputs	Protects users, data and compliance

For RAG systems, add retrieval-specific signals: retrieval latency, number of chunks returned, similarity scores and whether the retrieved context actually supported the answer. Many "hallucinations" turn out to be retrieval failures.

Tracing LLM apps and AI agents

Distributed tracing is the backbone of LLM observability. Each agent invocation becomes a trace, and each step becomes a span with timing, attributes and status. A typical agent trace looks like this:

invoke_agent support-assistant                 4.8 s
├── chat gpt-4o                (plan)          1.1 s   in 1,840 / out 120 tokens
├── execute_tool search_orders                 0.3 s
│   └── HTTP GET /orders?customer=…            0.2 s
├── embeddings text-embedding-3-small          0.1 s
├── vector search (top_k=5)                    0.2 s
└── chat gpt-4o                (answer)        3.0 s   in 3,950 / out 410 tokens

A trace like this immediately answers why a response was slow (the final generation), what it cost (sum of tokens across both model calls) and what context the model saw. Because the LLM spans live in the same trace as the HTTP, database and queue spans of your services, you can follow a user complaint from the frontend through the agent into the order system without switching tools.

Watch especially for agent loops: a trace with dozens of repeated tool calls is a classic sign of an agent that cannot reach its goal, and it is often the cause of sudden cost spikes. Alert on step counts and tokens per trace, not only on errors.

OpenTelemetry GenAI semantic conventions

OpenTelemetry defines semantic conventions for generative AI, so that instrumentation from different libraries and providers produces consistent telemetry. They are still marked as in development and evolve between releases, but they are already widely implemented. Key elements:

• Span names and operations: spans are named after the operation and model, such as chat gpt-4o, with gen_ai.operation.name values such as chat, embeddings, execute_tool, create_agent and invoke_agent.
• Request and response attributes: gen_ai.provider.name (which replaced the earlier gen_ai.system), gen_ai.request.model, gen_ai.response.model, gen_ai.request.temperature, gen_ai.request.max_tokens and gen_ai.response.finish_reasons.
• Usage attributes: gen_ai.usage.input_tokens and gen_ai.usage.output_tokens.
• Metrics: gen_ai.client.token.usage and gen_ai.client.operation.duration on the client side, plus server metrics such as time to first token for model-serving infrastructure.
• Content capture: prompts, completions and tool arguments are not recorded by default. Capturing them is opt-in because of their size and sensitivity.

Building on these conventions keeps your instrumentation portable: you can switch model providers, frameworks or observability backends without rewriting dashboards and alerts.

Measuring cost and quality

Token cost

Cost per call is input tokens × input price + output tokens × output price. With an illustrative price of $3 per million input tokens and $15 per million output tokens, a request with 2,000 input and 500 output tokens costs 0.006 + 0.0075 = $0.0135. At one million requests a month that is $13,500, and an agent that makes four such calls per task quadruples it. Attribute tokens to features, tenants and prompt versions so you can see which change moved the bill.

Quality evaluations

Quality needs explicit measurement. Common approaches include offline evaluations against curated test sets before each prompt or model change, online evaluations on a sample of production traffic using heuristics or an LLM-as-a-judge, and user feedback such as ratings or follow-up behaviour. Store evaluation scores as attributes or events on the trace they belong to, so a low score leads you directly to the prompt, context and tool calls that produced it.

Safety and guardrails

Guardrails check inputs and outputs against policies: detecting personal data, prompt injection and jailbreak attempts, toxic content, or answers that fall outside the allowed topic. Each guardrail decision should be recorded as an event on the trace, including which rule fired and whether the request was blocked, rewritten or allowed. Tracking the rate of guardrail triggers over time reveals both attacks and overly strict rules that frustrate legitimate users. For agents that can take actions, such as sending emails or changing records, also record which tools were invoked with which permissions, so every action can be audited afterwards.

Best practices and pitfalls

• Instrument at the framework level with OpenTelemetry-compatible libraries, and add custom spans for your own retrieval and business logic.
• Version prompts and record the version on every span, so quality and cost changes can be tied to a prompt change.
• Mask PII before storage. Prompts are logs with a higher risk profile. Redact at ingest and restrict access to raw content.
• Keep every error and expensive trace with tail-based sampling, and sample routine successful traces.
• Set budgets and alert on spend per feature and per tenant, as well as on tokens per trace.
• Define SLOs for AI features on latency, error rate and evaluation scores, not just uptime.
• Do not treat the model as a black box outside your traces. Provider latency and errors belong on the same timeline as your own services.

How Obseria helps with LLM observability

Obseria AI Observability traces LLM calls, agents and RAG pipelines end to end, based on OpenTelemetry, alongside the rest of your application telemetry. It tracks tokens, cost, latency and quality per model, prompt version, feature and tenant, and records guardrail events. Obseria Pipeline masks PII in prompts and completions at ingest, tail-based sampling keeps the traces that matter, and data can stay in the EU region or your own cloud account. When an AI feature degrades, Obseria Intelligence can investigate it like any other incident, following the trace from the user through the agent to the model provider or the tool that failed.