LLM observability in production: how to monitor AI applications, RAG pipelines and agents

LLM observability is the practice of making every model call in your application measurable and explainable: what went in, what came out, how long it took, how many tokens it used, what it cost and whether the answer was any good. Classic application monitoring tells you that an endpoint returned HTTP 200 in 1.8 seconds. It does not tell you that the answer was hallucinated, that a retrieval step returned the wrong documents or that an agent called the same tool eleven times before giving up.

This guide is a practical blueprint for teams that want to monitor LLM applications in production. It covers which signals to capture, how to trace LLM calls, RAG pipelines and agent tool calls with OpenTelemetry, how to add quality signals and guardrails, which dashboards and alerts are worth building and how to keep token costs predictable. It ends with a copyable checklist and a metric table you can use as a starting point for your own alerting. If you are new to the topic, our primer on what LLM observability is covers the basics first.

Why LLM applications need their own observability

An LLM feature behaves differently from a typical microservice in four ways, and each one changes what you need to observe:

• Non-deterministic output: the same input can produce different answers. A successful response is not automatically a correct one, so you need quality signals in addition to status codes.
• Usage-based cost per request: every call is billed by input and output tokens. A prompt change, a longer context window or a looping agent can multiply your bill without any error appearing.
• External dependencies with their own limits: most teams call hosted model APIs with rate limits, quotas, regional outages and silent model updates. Your reliability depends on a provider you do not control.
• Multi-step workflows: a single user question can trigger an embedding call, a vector search, a re-ranking step, several model calls and a chain of tool calls. Without tracing, you cannot see which step was slow, expensive or wrong.

The good news: you do not need a separate silo for AI. The most useful setup treats LLM calls as first-class spans and metrics in the same telemetry pipeline as the rest of your stack, so you can follow a slow chatbot answer from the browser through your API to the model provider and back.

What to capture from every LLM call

Start with a consistent record for each model call. These are the fields that matter in practice:

Prompts and completions, with privacy controls

Prompt and completion content is the most valuable and the most sensitive data you will collect. It is essential for debugging bad answers, but it often contains personal data, customer secrets or internal documents. Treat content capture as an explicit decision: capture it in development and staging, and in production either sample it, mask personal data before storage or store it in a separate store with shorter retention and stricter access. The OpenTelemetry GenAI conventions reflect this: attributes that carry message content are opt-in, not recorded by default.

Token usage and cost

Record input tokens, output tokens and, where the provider reports them, cached tokens per call. Derive cost from a price table that you version alongside your code, because prices change. Attach business context such as feature, tenant or team, so you can answer the question "which feature costs how much per day?" without a spreadsheet.

Latency, including time to first token

For streaming responses, total duration is only half the picture. Users perceive time to first token (TTFT) as responsiveness, while time per output token determines how fast the answer appears. Track both as histograms, together with the total duration of the operation, and look at percentiles rather than averages.

Errors, rate limits and retries

Distinguish error types: rate limit responses, quota exhaustion, timeouts, content filter refusals, invalid requests and provider-side server errors each need a different response. Record retries and fallbacks explicitly; a request that succeeded on the third attempt with a fallback model is not the same as a clean success.

Model and provider versions

Record the provider, the requested model and the model version the provider actually reports in the response. These are not always identical, and silent model updates are one of the most common reasons for quality changes that nobody can explain. Add your own prompt template version as well, so you can correlate regressions with prompt deployments.

Tracing LLM calls, RAG pipelines and agents with OpenTelemetry

OpenTelemetry is the natural foundation for AI observability, because it lets you instrument model calls with the same SDKs, context propagation and collectors you already use for your services. Each model call becomes a span inside the trace of the user request.

GenAI semantic conventions: current status

OpenTelemetry defines semantic conventions for generative AI, covering spans, metrics and events for model calls, agents and tools. As of October 2026, the opentelemetry.io documentation points to a dedicated GenAI semantic conventions repository, and the conventions there carry the status Development. That means names and attributes can still change. Use them, because they are the best shared vocabulary available, but pin the instrumentation library versions you depend on and expect to migrate attribute names during upgrades.

Attributes that are useful to standardise on include:

• gen_ai.operation.name for the operation type, for example chat, embeddings, retrieval, invoke_agent or execute_tool
• gen_ai.provider.name, gen_ai.request.model and gen_ai.response.model for provider and model
• gen_ai.usage.input_tokens and gen_ai.usage.output_tokens for token counts
• gen_ai.response.finish_reasons and error.type for outcomes and failures
• opt-in content attributes such as gen_ai.input.messages and gen_ai.output.messages, only where your privacy rules allow it

The conventions also define metrics such as gen_ai.client.operation.duration and, for model servers, gen_ai.server.time_to_first_token. Check the current specification before building dashboards on exact metric names.

Tracing a RAG pipeline

For retrieval-augmented generation, give every stage its own span: query embedding, vector search, re-ranking, prompt assembly and the final model call. Record the number of retrieved documents, the data source and the retrieval latency on the retrieval span, and document identifiers rather than full document text. When a user reports a wrong answer, this trace shows whether the problem was retrieval (wrong context) or generation (right context, wrong answer), which are fixed in completely different places.

Tracing agents and tool calls

Agents are loops: plan, call a tool, read the result, decide again. Model each agent run as a parent span and each model call and tool execution as child spans. Record the tool name, the outcome and the number of iterations. This makes runaway agents visible immediately: a trace with forty tool calls and a timeout tells a clearer story than any log line.

Sampling and volume

LLM traces are relatively low in volume but high in value and size, especially with content attributes. Decide sampling deliberately: keep all traces with errors, high cost or negative feedback, and sample the rest. Tail-based sampling in the OpenTelemetry Collector is well suited for this; our guide to tail-based sampling shows how to configure such policies.

Quality signals: evals, hallucination checks and user feedback

Operational health tells you the feature is up. Quality signals tell you whether it is useful. Combine three sources:

• Offline evals: a fixed test set of representative questions with expected answers or grading criteria, run on every prompt or model change before release. Treat a drop in eval scores like a failing test.
• Online evals: automated checks on a sample of production traffic, for example groundedness against the retrieved context, answer relevance, format validity or an LLM-as-judge score. Store the score as an attribute or event linked to the original trace, so you can jump from a low score to the exact request.
• User feedback: thumbs up or down, corrections, regenerations, copied answers and abandoned sessions. Implicit signals are often more honest than explicit ratings.

Hallucination checks deserve a word of caution: no automated check catches every fabricated answer. Groundedness scores against the retrieved context are a useful proxy for RAG applications, but they should trigger review, not replace it. Track the trend over time and per prompt version rather than relying on a single threshold.

Guardrails and PII: observe without creating a new data risk

Prompts are written by users, and users paste everything: email addresses, contract text, health information, access tokens. If you capture prompts naively, your observability backend becomes the least protected copy of your most sensitive data. Three practices help:

1. Mask before storage: apply PII detection and masking in the telemetry pipeline, before data reaches the backend, not afterwards in the UI.
2. Separate content from metadata: keep tokens, latency, model and error data broadly available, and restrict access to prompt and completion content to the people who need it for debugging.
3. Observe the guardrails themselves: log when input or output filters block, rewrite or flag a request, with the rule that fired. A sudden increase in blocked prompts can indicate abuse, a prompt injection attempt or a misconfigured filter.

Dashboards and alerts for AI features

Build one dashboard per AI feature, not one per model. A product owner wants to know whether the support assistant is healthy, not how a specific model endpoint behaves. A useful layout shows traffic, error rate by type, latency percentiles including TTFT, tokens and cost per request, eval scores and feedback rate. Break down by model version and prompt version, and keep an eye on cardinality: never use user IDs or prompt text as metric labels.

The table below is a starting point for alerting. Thresholds are examples and must be adapted to your traffic and baselines.

Metric	Why it matters	Typical alert (example)
Error rate by error type	Separates your bugs from provider outages and content filter refusals	Error rate above 5% for 10 minutes on a user-facing feature
Rate limit and quota responses	Early sign that traffic exceeds provider limits	Any sustained rate limit responses over 5 minutes
Time to first token (p95)	Drives perceived responsiveness of streaming answers	p95 TTFT above twice the 7-day baseline for 15 minutes
Total duration (p95, p99)	Catches slow retrieval, long outputs and agent loops	p99 above the request timeout budget
Tokens per request	Reveals prompt bloat, context growth and runaway outputs	Median input tokens up more than 30% after a deployment
Cost per hour and per feature	Protects the budget against loops and abuse	Hourly cost above forecast, or daily budget 80% consumed
Agent iterations per run	Detects loops and tools that never succeed	Runs hitting the iteration limit above 2% of runs
Retrieval result count and latency	Empty or slow retrieval produces poor answers	Share of retrievals with zero results above baseline
Online eval score	Quality regressions do not produce errors	Rolling groundedness score drops below the release threshold
Negative feedback rate	The most direct signal of user-visible quality	Negative feedback rate doubles week over week
Guardrail block rate	Signals abuse, prompt injection or filter misconfiguration	Block rate above three times baseline for 30 minutes

Where possible, wrap the most important signals into service level objectives, for example "99% of support assistant answers start streaming within 2 seconds", and alert on burn rate instead of static thresholds. This keeps on-call pages for AI features consistent with the rest of your SLO monitoring.

Cost control for LLM workloads

Token costs scale with usage, prompt length and output length, which makes them easy to underestimate. Observability turns cost from a monthly surprise into a metric you can act on:

• Attribute cost: tag every call with feature, team and tenant, and report cost per request and per successful outcome, not only in total.
• Watch prompt growth: system prompts, few-shot examples and retrieved context tend to grow over time. Track input tokens per request by prompt version.
• Use caching and smaller models deliberately: measure cache hit rates and compare quality and cost when routing simpler requests to smaller models. Without eval scores next to cost, you cannot tell whether a cheaper model is good enough.
• Cap agents: set iteration limits and token budgets per run, and alert when runs hit them.
• Control telemetry cost too: content attributes can make traces large. Filter, truncate and sample in the pipeline before storage.

LLM observability checklist

Copy this checklist into your rollout plan or architecture review:

LLM observability checklist

Instrumentation
[ ] Every model call is a span inside the user request trace
[ ] Provider, requested model and response model are recorded
[ ] Prompt template version is recorded on every call
[ ] Input, output and cached tokens are recorded per call
[ ] Cost is derived from a versioned price table
[ ] TTFT, time per output token and total duration are histograms
[ ] Error type, retries and fallbacks are recorded explicitly
[ ] RAG stages (embed, retrieve, rerank, generate) have own spans
[ ] Agent runs have parent spans; tool calls are child spans
[ ] GenAI semantic convention versions are pinned and documented

Privacy and guardrails
[ ] Content capture policy agreed with data protection
[ ] PII masked in the pipeline before storage
[ ] Content stored with shorter retention and restricted access
[ ] Guardrail decisions logged with the rule that fired

Quality
[ ] Offline eval set runs on every prompt or model change
[ ] Online evals score a sample of production traffic
[ ] Eval scores and user feedback are linked to trace IDs

Operations
[ ] One dashboard per AI feature, broken down by model and prompt version
[ ] Alerts on errors, rate limits, TTFT, cost and agent loops
[ ] At least one SLO for latency or availability of the feature
[ ] Daily cost budget with alert at 80%
[ ] Sampling keeps all errors, expensive and negatively rated traces
[ ] Runbook for provider outage and rate limiting (fallback model)

How Obseria helps

Obseria is an OpenTelemetry-native observability platform, so LLM spans and metrics from standard instrumentation libraries arrive in the same traces as your services, databases and frontends. AI observability in Obseria covers model calls, token usage, cost and latency per feature, while the rest of the platform provides SLOs, alerting and incident management for AI features in the same place as everything else.

• The Obseria Pipeline masks PII, truncates content and applies sampling before data is stored, with EU data residency.
• Obseria Intelligence and the AI SRE agent correlate LLM latency or error spikes with deployments, provider issues and infrastructure changes, with evidence.
• The MCP server lets your own AI assistants query telemetry, traces and incidents directly.

To see your own model calls traced end to end, start a free trial and connect your existing OpenTelemetry instrumentation.