A percentile is the value below which a given percentage of observations fall. In observability, p95 or p99 latency means 95 or 99 percent of requests completed at or below that duration. Percentiles reveal tail behavior that averages hide, which is why SLOs and latency alerts are usually defined on them.
What is a percentile?
A percentile ranks observations and reports the value at a given position. The 50th percentile (p50, the median) splits requests into a faster and a slower half. The 99th percentile (p99) is the duration that 99 percent of requests stay at or below; only the slowest 1 percent are slower. In monitoring, percentiles are almost always applied to latency, but they work for any distribution: payload sizes, queue wait times or page load times in real user monitoring.
The common shorthand is p50, p90, p95, p99 and p99.9. The further you go into the tail, the more data you need for a stable value: a p99.9 computed from 200 requests is just the slowest request.
How percentiles are calculated
For a set of n sorted values, the nearest-rank method takes the value at rank ceil(p / 100 × n). For 1,000 requests, p99 is the 990th fastest request and p50 is the 500th. Other methods interpolate between neighboring ranks, so tools can report slightly different numbers for the same data.
Storing every raw value does not scale, so monitoring systems estimate percentiles from histograms. A histogram counts observations per bucket (for example 0 to 100 ms, 100 to 250 ms, 250 to 500 ms). Prometheus histogram_quantile() finds the bucket that contains the target rank and interpolates linearly inside it, so the accuracy depends on bucket boundaries. OpenTelemetry exponential histograms use automatically scaled buckets and keep the relative error small without manual tuning.
Why averages mislead
Latency distributions are skewed: most requests are fast, a few are very slow. The mean mixes both groups into a number that describes almost nobody. Consider this illustrative set of 1,000 requests:
| Requests | Duration |
|---|---|
| 900 | 80 ms |
| 90 | 400 ms |
| 10 | 3,000 ms |
The mean is (900 × 80 + 90 × 400 + 10 × 3,000) / 1,000 = 138 ms, which looks healthy. The percentiles tell a different story: p50 = 80 ms, p95 = 400 ms, p99 = 400 ms and p99.9 = 3,000 ms. One request in ten is five times slower than the median, and one in a hundred takes three seconds. Those are the users who abandon a checkout.
Tail latency amplifies with fan-out
When a single page or API call depends on many backend requests, the slow tail becomes the norm. If a page makes 100 independent backend calls, each with a 1 percent chance of hitting p99 latency, the probability that at least one call is that slow is 1 − 0.99 to the power of 100, roughly 63 percent. The backend's p99 becomes the user's typical experience.
Aggregation rules you must know
- Never average percentiles. The mean of p99 values from ten pods is not the p99 of the service. A pod with little traffic counts as much as a busy one, and the math does not hold even with equal traffic.
- Aggregate histograms, then compute. Sum bucket counts across instances first, then derive the percentile. This is why histograms beat client-side summaries.
- Mind the window. A p99 over five minutes and a p99 over 30 days answer different questions. SLOs usually count good versus total events over the full window instead of averaging short-window percentiles.
Best practices
- Track at least p50, p95 and p99 per endpoint, plus the request rate so you know how many users a percentile represents.
- Define SLOs as threshold ratios, such as "99 percent of requests faster than 300 ms", which are easy to aggregate and explain.
- Place histogram bucket boundaries around your SLO thresholds, or use exponential histograms.
- Split percentiles by meaningful dimensions such as route, region or customer tier, but keep cardinality bounded.
- Use exemplars to jump from a slow bucket straight to a representative trace.
Common mistakes
- Alerting on average latency and missing regressions that only affect the tail.
- Reporting p99.9 from low-traffic endpoints, where one request decides the value.
- Comparing percentiles from tools that use different calculation or interpolation methods.
- Excluding errors or timeouts from latency data, which makes the tail look better than it is.
How Obseria helps
Obseria stores latency as histograms, including OpenTelemetry exponential histograms, so p50 to p99.9 can be computed and aggregated correctly across any dimension at query time. Percentile charts link to the underlying traces in distributed tracing, and Obseria Intelligence learns a baseline per endpoint to flag tail-latency regressions before they consume your error budget.
