Throughput is the rate at which a system completes work, measured per unit of time, for example requests per second, transactions per minute, messages consumed per second or bytes transferred per second. Together with latency, errors and saturation it describes service health, and it is the basis for capacity planning and anomaly detection on traffic.
What is throughput?
Throughput answers a simple question: how much work does the system get done per unit of time? For a web API that is usually requests per second (RPS); for a payment system, transactions per minute; for a Kafka consumer, messages per second; for a disk or network link, bytes per second. Throughput corresponds to the traffic signal of the golden signals and the rate in the RED method (rate, errors, duration).
Throughput is not the same as load. Load is what arrives; throughput is what is completed. When a system is healthy the two are equal. When it is overloaded, arrivals pile up in queues, throughput plateaus or even drops, and latency climbs.
Throughput, latency and Little's Law
Throughput and latency are linked. Little's Law from queueing theory states that, in a stable system, the average number of requests in flight equals the arrival rate multiplied by the average time each request spends in the system:
L = λ × W
concurrency = throughput × average latencyAn illustrative example: a service handles 200 requests per second with an average latency of 250 ms, so on average 50 requests are in flight. If latency doubles to 500 ms at the same throughput, concurrency doubles to 100. If the service has a pool of 64 worker threads or database connections, requests now start to queue, which increases latency further. This feedback loop explains why systems often fail suddenly rather than gradually.
As utilization approaches capacity, queueing makes latency rise steeply even though throughput hardly increases. The useful operating range is therefore well below theoretical maximum throughput, typically where tail latency is still flat.
How to measure throughput
- Request-driven services: count completed requests and take the rate. With OpenTelemetry, the
http.server.request.durationhistogram already contains a count, so throughput and latency come from the same metric. - Messaging: track produced and consumed messages per second plus consumer lag; a consumer whose throughput is below the producer rate will fall behind.
- Batch and data pipelines: measure records or bytes processed per second and time to completion.
- Infrastructure: disk read and write bytes per second, IOPS, and network bytes per second per interface.
In PromQL, throughput per service from an OpenTelemetry histogram exported to Prometheus looks like this:
sum by (job) (rate(http_server_request_duration_seconds_count[5m]))Prometheus' OTLP translation maps the resource attribute service.name to the job label, hence the grouping. Always split throughput by meaningful dimensions such as route, status class and region. Total RPS that looks normal can hide one endpoint that dropped to zero.
Why throughput matters
- Detecting outages: a sudden drop in throughput often signals a problem that error metrics miss, for example clients that can no longer reach the service at all.
- Context for other signals: an error rate of 5 percent means something different at 10 RPS than at 10,000 RPS. Ratios need the denominator.
- Capacity planning: throughput trends over months, combined with resource usage per request, show when you need to scale.
- Business view: orders, sign-ups or payments per minute are throughput metrics that executives understand directly.
Throughput across a call chain
In distributed systems one user request often triggers many internal calls. A checkout request might call inventory, pricing and payment, and each of them queries a database. Internal throughput can therefore be several times higher than edge throughput, and the ratio between the two, sometimes called fan-out, is worth watching. If internal calls per edge request suddenly double after a deployment, you may have introduced an N+1 query or a retry loop, even though user traffic has not changed at all.
Best practices
- Alert on unexpected changes, not fixed thresholds. Traffic follows daily and weekly patterns, so compare against a baseline or the same time last week.
- Measure throughput of successful work separately from total requests, so fast failures do not look like healthy traffic.
- Load test to find the knee. Increase load step by step and record where p99 latency starts to bend upward; that is your practical capacity.
- Watch saturation alongside throughput: thread pools, connection pools, CPU and queue depth tell you how close you are to the limit.
Common mistakes
- Reporting average throughput over a day, which hides the peaks that matter for capacity.
- Counting retries as throughput, which overstates useful work during incidents.
- Confusing bandwidth (theoretical capacity of a link) with throughput (what is actually transferred).
- Ignoring a throughput drop because latency and errors look fine.
How Obseria helps
Obseria derives throughput per service, endpoint and dependency automatically from traces and OpenTelemetry metrics, and shows it next to latency and errors in Application Observability. Obseria Intelligence learns seasonal traffic patterns and flags unexpected drops or spikes through anomaly detection, and Business Observability lets you track business throughput such as orders per minute alongside technical signals.
