Full visibility into
your AI systems.
Trace every prompt, track token cost, detect model drift, and catch regressions before your users do — across LLMs, ML pipelines, and AI agents. OpenTelemetry-native.
Prompt Tracing
See exactly what your model was given — and what it returned
Every LLM call is captured as an OpenTelemetry span. Prompt text, completion, model version, token counts, and latency are stored as structured attributes — searchable, filterable, and linkable to the rest of your infrastructure trace.
- Full prompt + completion stored per request
- Multi-step agent chains visualised as waterfall
- Filter by model, error, latency, or token cost
- Link AI spans to upstream HTTP and DB traces
Trace — chat request · 98 ms total
trace_id: a3f8c1…
Model
gpt-4o-mini
Input tkns
1,102
Output tkns
318
Cost
$0.0021
LLM response latency (ms)
Daily token usage (thousands)
Latency & Cost
Know exactly what every AI call costs — in time and money
Unoptimised LLM usage silently erodes margins and degrades UX. obseria.io gives you per-request cost breakdowns across every model and provider, latency percentiles, and spend trends — so you can identify which pipeline stages are slow or expensive before they hit production.
Cost per request
Down to the cent
Latency percentiles
p50 · p95 · p99
Spend forecasting
30-day projection
Per model breakdown
Multi-provider
Drift Detection
Catch model degradation before your users do
Model behaviour changes over time — inputs shift, upstream models get updated, training distribution drifts away from real-world data. obseria.io continuously computes a statistical similarity score between live outputs and your baseline window, alerting you the moment drift becomes significant.
- Statistical output distribution monitoring
- Configurable drift threshold per model
- Automatic re-baselining on new model versions
- Correlate drift events to deployment history
Output distribution drift score
Model: sentiment-classifier-v2 · 20-day window
Quality & Evals
Monitor quality, not just uptime
Infrastructure metrics tell you if your AI is running. obseria.io tells you if it's working.
Hallucination rate tracking
Connect your evaluation pipeline and track hallucination and factual error rates over time as models or prompts change.
Retrieval quality (RAG)
For retrieval-augmented pipelines, measure chunk relevance scores, retrieval latency, and context utilisation per query.
User feedback correlation
Attach thumbs-up/down signals to trace IDs and see which prompt patterns, models, or pipeline paths correlate with poor ratings.
A/B model comparison
Run two model versions side-by-side with traffic splitting, then compare latency, cost, and quality scores in a single dashboard.
Regression detection
Automated statistical tests on key quality metrics after every deployment — surfaces regressions before they reach all users.
Real-time eval pipeline
Plug in custom eval functions via webhook or SDK. Results are stored as span attributes and surface in the same trace view.
Integrations
Works with your entire AI stack
One SDK, every provider. No lock-in.
FAQ
Common questions
Your AI systems deserve the same observability as your infrastructure.
Start your 14-day free trial — no credit card required.