The four golden signals are latency, traffic, errors and saturation: the four metrics Google's SRE book recommends monitoring for every user-facing system. They describe how fast a service responds, how much demand it serves, how often it fails and how close it is to its capacity limits.
Key takeaways
- Golden signals: latency (how long), traffic (how much), errors (how often it fails) and saturation (how full).
- The RED method (rate, errors, duration) applies the same idea to every request-driven service; the USE method (utilization, saturation, errors) applies it to every resource.
- Measure latency as percentiles from histograms and separate the latency of failed requests from successful ones.
- Alert on symptoms in the golden signals, ideally through SLOs, and use saturation and USE metrics to diagnose causes.
What are the four golden signals?
The four golden signals are a minimal set of metrics for monitoring any user-facing system: latency, traffic, errors and saturation. They come from the "Monitoring Distributed Systems" chapter of Google's Site Reliability Engineering book, which puts it simply: if you can only measure four metrics of your user-facing system, focus on these four. A service whose golden signals look healthy is, in almost all cases, serving its users well. When one of them changes, users are affected or soon will be.
The value of the golden signals lies in their universality. They work for a REST API, a gRPC microservice, a database, a message broker or a web frontend. They give every team the same starting vocabulary, and they make dashboards comparable across hundreds of services.
The four signals in detail
1. Latency
Latency is the time it takes to service a request. Two details matter. First, measure distributions, not averages: p50 tells you about the typical request, p95 and p99 about the tail that a meaningful share of users experience on every page load that fans out to many backend calls. Second, separate successful and failed requests. A database that rejects requests in 2 ms makes average latency look excellent while users see errors. A slow error is even worse than a fast one, so track both.
2. Traffic
Traffic measures demand on the system in a unit that makes sense for it: HTTP requests per second for a web service, messages per second for a queue consumer, transactions or reads per second for a database, concurrent sessions or bytes per second for a streaming service. Traffic gives context to every other signal: an error spike during a traffic spike tells a different story than one at 3 a.m.
3. Errors
Errors are the rate of requests that fail. The SRE book distinguishes three kinds: explicit failures (HTTP 500), implicit failures (HTTP 200 with wrong or empty content) and policy failures (for example, any response slower than one second counts as failed because you committed to that). Implicit errors are only visible through end-to-end checks, synthetic tests or business-level validation.
4. Saturation
Saturation describes how "full" the service is, with emphasis on the most constrained resource: CPU for compute-bound services, memory, I/O, connection pools, thread pools or queue depth. Many systems degrade well before 100% utilisation, so saturation needs a target that reflects where performance starts to suffer. Saturation is also about prediction: "the disk will be full in four hours" is a saturation signal. Rising latency is often the earliest indicator of saturation.
RED and USE: two related methods
Two other well-known frameworks apply the same thinking to different scopes.
The RED method, described by Tom Wilkie, focuses on request-driven services. For every service measure Rate (requests per second), Errors (failed requests per second) and Duration (distribution of request latency). RED is essentially the golden signals minus saturation, and it is a natural fit for microservices because the three metrics can be derived from a single latency histogram with an error label.
The USE method, created by Brendan Gregg for performance analysis, focuses on resources. For every resource (CPU, memory, disks, network interfaces, but also locks, thread pools and connection pools) check Utilization (share of time the resource is busy), Saturation (queued or waiting work) and Errors (error events). USE is designed as a checklist to quickly find bottlenecks.
| Method | Signals | Scope | Best for |
|---|---|---|---|
| Four golden signals | Latency, traffic, errors, saturation | User-facing systems | Service health, alerting, SLOs |
| RED | Rate, errors, duration | Every request-driven service | Microservice dashboards, consistent service views |
| USE | Utilization, saturation, errors | Every hardware or software resource | Bottleneck analysis, capacity planning |
The methods are complementary rather than competing. A practical rule: use RED or golden signals to detect that users are affected and where, then use USE on the underlying hosts, containers and pools to find out why.
Golden signals for different kinds of systems
The four signals are universal, but what you measure for each one depends on the type of system. The table shows typical choices:
| System | Latency | Traffic | Errors | Saturation |
|---|---|---|---|---|
| HTTP or gRPC API | Request duration p50/p99 | Requests per second | 5xx and gRPC error codes | CPU, thread and connection pools |
| Message queue consumer | End-to-end message age | Messages consumed per second | Failed and dead-lettered messages | Consumer lag, queue depth |
| Database | Query duration | Queries or transactions per second | Failed queries, deadlocks | Connections in use, disk and IOPS |
| Batch job | Job runtime vs. deadline | Records processed per run | Failed runs, rejected records | Worker utilisation, backlog |
| Web frontend | Core Web Vitals such as LCP and INP | Page views, sessions | JavaScript errors, failed API calls | Main-thread blocking, long tasks |
For asynchronous systems, latency should measure how long the work took from the user's point of view, for example the time from a message being produced until it was processed, not just the time a consumer spent on it. A consumer that processes each message in 5 ms is still failing users if the queue holds a two-hour backlog.
Measuring golden signals with OpenTelemetry
OpenTelemetry semantic conventions make the golden signals easy to derive in a vendor-neutral way. The stable HTTP metric http.server.request.duration is a histogram that carries attributes such as http.request.method, http.route, http.response.status_code and error.type. From this one instrument you get:
- Traffic / rate: the count of the histogram per second.
- Errors: the count where
error.typeis set or the status code is 5xx, divided by total count. - Latency / duration: percentiles computed from the histogram buckets.
Saturation comes from resource metrics: container CPU throttling and memory working set, JVM or runtime metrics such as garbage collection time, database client connection pool usage, and message queue lag. Distributed traces add the dimension metrics cannot: which downstream call made a slow request slow.
In PromQL, the classic RED queries over the exported histogram look like this:
# Rate
sum by (http_route) (rate(http_server_request_duration_seconds_count[5m]))
# Error ratio
sum(rate(http_server_request_duration_seconds_count{http_response_status_code=~"5.."}[5m]))
/ sum(rate(http_server_request_duration_seconds_count[5m]))
# p99 latency
histogram_quantile(0.99,
sum by (le) (rate(http_server_request_duration_seconds_bucket[5m])))From golden signals to alerts and SLOs
Golden signals are the right place to alert, because they describe symptoms that users feel. Alerting on causes such as "CPU above 80%" produces pages that often do not matter, and misses failures with unexpected causes. A mature setup looks like this:
- Turn latency and errors into SLIs and define SLOs for critical user journeys.
- Page on SLO burn rate rather than raw thresholds on individual signals.
- Use traffic anomalies (sudden drops are as important as spikes) as a secondary alert, because a silent outage often shows up only as missing traffic.
- Use saturation for capacity warnings and tickets, not pages, unless exhaustion is imminent.
- Keep USE metrics on dashboards for diagnosis, linked from each service view.
Common pitfalls
- Averaging latency. A mean of 120 ms can hide a p99 of four seconds. Use histograms and percentiles, and never average percentiles across instances.
- Ignoring traffic drops. Error rate stays at zero when no requests arrive. Watch for traffic falling below its expected pattern.
- Counting client errors as server errors. A wave of 404s from a broken client link is worth knowing about, but it should not burn your availability SLO.
- Measuring only at the service. The load balancer, CDN and browser see failures your service never logs. Add edge metrics, synthetic checks and real user monitoring.
- Cardinality explosions. Labelling metrics with user IDs or full URLs makes them expensive and slow. Use route templates and put high-cardinality detail in traces and logs.
- Treating saturation as CPU only. The real limit is often a connection pool, a thread pool, a rate-limited downstream API or a queue.
How Obseria helps
Obseria derives golden signals and RED metrics automatically for every service it discovers, from OpenTelemetry spans and metrics or from the Obseria Agent, and shows them on a consistent service view without manual dashboard work. Obseria TopoGraph places each service in its dependency graph, so a latency increase can be followed to the saturated database pool or container below it, where USE-style resource metrics are one click away. You can query the same data with OQL, PromQL or SQL, turn any golden signal into an SLO, and let anomaly detection and Obseria Intelligence flag unusual changes in traffic, errors or latency before they exhaust your error budget.