Error rate is the proportion of requests or operations that fail within a given time window, usually expressed as a percentage of all requests. It is one of the four golden signals and the E in the RED method. What counts as a failure, for example HTTP 5xx responses or exceptions, must be defined explicitly per service.
What error rate means
Error rate measures how often a service fails to do what was asked of it. In the most common form it is a ratio:
error rate = failed requests / total requests (over a time window)If a service handled 120,000 requests in the last five minutes and 360 of them failed, the error rate is 0.3%. Sometimes "error rate" is also used for errors per second, an absolute count. Both are useful, but they answer different questions: the ratio tells you how bad the experience is for an average request, the absolute rate tells you how many users are affected. A ratio of 50% on a test endpoint with two calls matters less than 0.5% on checkout during peak traffic.
Error rate is one of the four golden signals, together with latency, traffic and saturation, and it is the basis of most availability SLOs.
What counts as an error
The formula is simple; the definition of "failed" is where teams disagree. Common rules:
| Protocol / signal | Usually an error | Usually not an error |
|---|---|---|
| HTTP server | 5xx responses, timeouts, dropped connections | 4xx responses caused by the client (404, 401, 422) |
| HTTP client | 4xx and 5xx responses, connection errors | Expected 404 checks, if handled |
| gRPC | INTERNAL, UNAVAILABLE, DEADLINE_EXCEEDED, UNKNOWN, DATA_LOSS | NOT_FOUND, INVALID_ARGUMENT for server-side metrics |
| Jobs and messages | Failed runs, messages moved to a dead-letter queue | Retried and later successful attempts, if tracked separately |
The OpenTelemetry semantic conventions follow the same logic. For HTTP server spans, a 5xx status sets the span status to Error, while 4xx responses leave it unset, because the server behaved correctly. For client spans, both 4xx and 5xx are errors from the caller's point of view. The error.type attribute records the status code or exception class, so metrics such as http.server.request.duration can be split by failure type.
Two special cases deserve attention. First, a 429 or 503 caused by your own rate limiting or overload protection is a real failure from the user's perspective. Second, a request that returns 200 with an error message in the body is invisible to status-code-based error rates. Business checks or synthetic tests catch these.
How to measure it
Error rate can be derived from three sources, which complement each other:
- Metrics. Request counters or histograms with a status label. Cheap, complete and ideal for alerting and SLOs.
- Traces. Spans with error status show where in the call chain the failure originated and which attributes the failing requests share.
- Logs and error events. Stack traces and messages explain what went wrong in the code.
An illustrative PromQL query for the five-minute server error ratio of a checkout service, using OpenTelemetry HTTP metrics exported to Prometheus:
sum(rate(http_server_request_duration_seconds_count{service_name="checkout",http_response_status_code=~"5.."}[5m]))
/
sum(rate(http_server_request_duration_seconds_count{service_name="checkout"}[5m]))Alerting on error rate
A static rule such as "page when errors exceed 1% for one minute" is easy to write and tends to be noisy. Short spikes trigger it, while a slow increase from 0.1% to 0.8% over a day goes unnoticed. Error budget burn rate alerts with multiple windows are more reliable, because they relate the current error rate to the SLO and react faster to severe problems than to mild ones.
For low-traffic services, guard against small denominators: one failed request out of three is a 33% error rate. Require a minimum number of requests in the window, aggregate over a longer period or add synthetic traffic.
Retries complicate the picture. If a client retries a failed call and the second attempt succeeds, the user sees success while the server records an error. Measure error rate both per attempt at the server and per logical operation at the edge; a rising gap between the two is an early warning that retries are masking a problem and adding load.
Best practices
- Write the error definition down per service and keep it consistent between dashboards, alerts and SLOs.
- Compute ratios from sums, not by averaging per-instance error rates, so busy instances carry their real weight.
- Break down by route, version and dependency. A global 0.2% can hide one endpoint failing at 30%.
- Correlate with deployments. Many error spikes start with a release or configuration change.
- Look at throughput at the same time. A falling error rate during a traffic drop may simply mean that clients gave up.
How Obseria helps
Obseria derives error rates from OpenTelemetry metrics and spans using the semantic conventions, so 4xx and 5xx are classified consistently across languages. Error tracking groups exceptions into issues with their first and last occurrence and affected releases, and distributed tracing shows which downstream call caused a failing request.
