A histogram is a metric type that records the distribution of measured values, such as request durations, by counting observations in buckets along with their total sum and count. Unlike a simple average, a histogram lets you estimate percentiles like p95 or p99 and aggregate distributions correctly across many instances and time windows.
What is a histogram metric?
A histogram in observability is a metric type that captures how values are distributed, not just their average. Each time a request finishes, its duration is "observed": the histogram increments the counter of the bucket the value falls into, adds the value to a running sum and increments a total count. From those counters you can later answer questions such as "how many requests took longer than 300 ms?" or "what was the p99 latency in the last five minutes?".
Histograms are the standard way to measure latency, payload sizes, queue wait times and similar values whose tail matters more than the mean. Because bucket counters are plain numbers, histograms from hundreds of pods can be summed and still yield a valid distribution, which is not true for pre-computed percentiles.
Classic Prometheus histograms and le buckets
A classic Prometheus histogram named http_request_duration_seconds exposes three kinds of series:
http_request_duration_seconds_bucket{le="0.25"}: cumulative counters, one per upper bound (lemeans "less than or equal"), always includingle="+Inf".http_request_duration_seconds_sum: the sum of all observed values.http_request_duration_seconds_count: the number of observations, equal to the+Infbucket.
Bucket boundaries are fixed at instrumentation time. The default in many Prometheus client libraries is 5 ms to 10 s (.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10). Every bucket is its own time series per label combination, so ten buckets times many labels multiplies cardinality quickly.
Calculating percentiles with histogram_quantile
In PromQL you estimate a percentile from bucket rates:
histogram_quantile(0.99,
sum by (le, service) (rate(http_request_duration_seconds_bucket[5m])))The function finds the bucket that contains the target rank and interpolates linearly inside it. The result is an estimate whose accuracy depends on bucket layout: if buckets are 250 ms and 500 ms, a reported p99 of 400 ms only means "somewhere between 250 and 500 ms". Always aggregate with sum by (le) before calling histogram_quantile, and keep the le label in the grouping.
Exponential and native histograms
Fixed buckets force a trade-off between precision and cost. Two newer formats solve this with exponentially growing buckets that need no manual configuration:
| Format | Buckets | Notes |
|---|---|---|
| Classic Prometheus histogram | Fixed, user-defined le bounds | One series per bucket; widely supported |
| OpenTelemetry explicit bucket histogram | Fixed boundaries, set per instrument or view | Equivalent to classic histograms |
| OpenTelemetry exponential histogram | Base 2^(2^-scale), scale adjusts automatically | Bounded number of buckets, relative error stays constant |
| Prometheus native histogram | Exponential schema, sparse buckets | Whole histogram in a single series; stable since Prometheus 3.8, scraping is opt-in |
With exponential buckets the relative error is the same at 2 ms and at 20 s, and the SDK lowers the scale when values spread too wide. Native histograms and OTel exponential histograms are designed to be convertible, and both drastically reduce series count compared with classic buckets.
Worked example: reading a histogram
An illustrative scrape window for one service shows these bucket rates per second: le="0.1" 900, le="0.25" 970, le="0.5" 995 and le="+Inf" 1000. That means 90 percent of requests finished within 100 ms, 97 percent within 250 ms and 99.5 percent within 500 ms. The p99 rank (990) falls into the 250–500 ms bucket, which holds 25 requests per second; linear interpolation places it at roughly 450 ms. The 5 requests per second above 500 ms are counted, but their exact duration is unknown, which is why the highest finite bucket should sit above any value you care about.
Histogram vs. summary
Prometheus also has a summary type that computes quantiles in the client. Summaries are precise for a single instance, but quantiles cannot be averaged or summed across instances. For services with more than one replica, histograms are almost always the better choice. Histograms can also carry exemplars, trace IDs attached to individual buckets, which link a slow bucket directly to a representative trace.
Best practices and common mistakes
- Put bucket bounds at your SLO thresholds. If the objective is "99 percent under 300 ms", a
le="0.3"bucket gives an exact ratio instead of an interpolated estimate. - Prefer exponential or native histograms where your stack supports them, to avoid guessing bucket layouts.
- Do not average percentiles. Aggregate bucket counts first, then compute the quantile.
- Watch label cardinality. A histogram with user IDs as labels multiplies every bucket.
- Use the sum and count too:
rate(_sum) / rate(_count)gives the mean, which is useful alongside the tail.
How Obseria helps
Obseria accepts OpenTelemetry histograms over OTLP, including exponential histograms, and supports PromQL alongside OQL and SQL, so existing histogram_quantile queries keep working. Metric data points are retained for 13 months, which makes long-term latency distributions and SLO reporting possible. Exemplars connect histogram buckets to traces in distributed tracing.
