An exemplar is an individual sample measurement attached to an aggregated metric data point, typically together with the trace ID and span ID of the request that produced it. Exemplars let engineers jump from a latency spike or error count on a dashboard directly to a representative distributed trace, without adding high-cardinality labels to the metric.
What an exemplar is
Metrics are aggregates. A latency histogram tells you that 2% of requests in the last minute took longer than one second, but not which requests. Traces tell you exactly what happened inside individual requests, but there are far too many to read. An exemplar bridges the two: alongside an aggregated value, the metric stores a few concrete example measurements, each tagged with the trace_id (and often span_id) of the request it came from.
On a dashboard, exemplars typically appear as dots on top of a time-series graph. Hovering over a dot in the slow bucket shows the measured value, the timestamp and the trace ID; clicking it opens the corresponding trace. This turns "latency is high" into "here is a slow request, and here is where it spent its time" in a single step.
Why exemplars matter
Linking metrics to traces is one of the most frequent steps in an investigation. Without exemplars, engineers search for traces in the same time window and hope to find a slow one, which is tedious and often misleading. Exemplars give a direct, statistically relevant pointer.
They also solve a cardinality problem. Adding trace_id as a metric label would create a new time series for every request and overwhelm any time-series database. Exemplars are stored beside the series, not as part of its identity, so the metric's cardinality stays unchanged.
How exemplars work in OpenMetrics and Prometheus
The OpenMetrics exposition format defines exemplars as an optional suffix on counter and histogram bucket samples, introduced by #:
http_request_duration_seconds_bucket{le="0.5"} 1270 # {trace_id="4bf92f3577b34da6a3ce929d0e0e4736"} 0.43 1729000000.123
http_request_duration_seconds_bucket{le="1"} 1302 # {trace_id="00f067aa0ba902b74bf92f3577b34da6"} 0.91 1729000001.456Each exemplar has its own small label set, the observed value and an optional timestamp. OpenMetrics limits the combined length of exemplar label names and values to 128 UTF-8 characters. Prometheus stores exemplars when started with --enable-feature=exemplar-storage, keeps them in a fixed-size in-memory circular buffer, scrapes them only via the OpenMetrics format, and exposes them through the query_exemplars API. Remote write can forward them when send_exemplars is enabled.
Exemplars in OpenTelemetry
The OpenTelemetry metrics data model includes exemplars on histogram and sum data points, with trace ID, span ID, timestamp, value and filtered attributes. The SDK chooses which measurements become exemplars using an exemplar filter and an exemplar reservoir. The filter is configurable with OTEL_METRICS_EXEMPLAR_FILTER:
| Value | Behavior |
|---|---|
trace_based | Default. Only measurements recorded inside a sampled span can become exemplars. |
always_on | Any measurement can become an exemplar. |
always_off | No exemplars are collected. |
Because the default ties exemplars to sampled spans, the trace an exemplar points to is likely to exist in the backend. Exemplars are exported with OTLP alongside the metrics.
Example: from a p99 spike to the cause
In an illustrative scenario, the p99 latency of a checkout API rises from 450 ms to 2.1 seconds at 14:05 while the median stays flat. The dashboard shows exemplar dots clustered in the 2.5-second bucket. The on-call engineer opens one of them: the trace shows that the request spent 1.8 seconds waiting on a connection pool to the inventory database. Opening two more exemplars from the same bucket confirms the pattern. Within minutes, the investigation has moved from an aggregate symptom to a concrete, reproducible cause, without guessing time windows or filtering thousands of traces by hand.
Best practices
- Record metrics inside active spans so the SDK can attach the current trace context automatically.
- Align sampling. If traces are heavily sampled, ensure exemplars point to retained traces; trace-based filtering and tail-based sampling that keeps slow and failed requests help.
- Use histograms for latency. Exemplars on high buckets are the fastest route to the slowest requests behind a high percentile.
- Check the whole path. Exporters, collectors and backends must all support and forward exemplars, otherwise they are silently dropped.
Common mistakes
A frequent mistake is putting trace IDs into regular labels, which causes a cardinality explosion. Another is enabling exemplars in the application while the scrape uses the classic Prometheus text format, which has no exemplar support, so nothing arrives. Teams also forget that exemplars are samples: a single exemplar shows one representative request, not proof that all slow requests share the same cause.
How Obseria helps
Obseria accepts OpenTelemetry metrics with exemplars over OTLP and links them to traces stored in Obseria Lake. From any latency or error chart you can open a representative trace in distributed tracing and continue to related logs and profiles. See What is OpenTelemetry? and the Prometheus integration for setup details.
