Saturation is the degree to which a resource or service is full, meaning demand reaches or exceeds its capacity so that additional work has to wait in a queue. It is one of the four golden signals and the S in the USE method, and is measured with signals such as run queue length, pool waits and pressure stalls.
What is saturation?
Saturation describes how full a resource or a service is. A resource is saturated when it has more work than it can serve right now, so extra work queues up: threads wait for a CPU, requests wait for a database connection, I/O requests wait at a disk. Saturation is a leading indicator: it usually rises before users notice errors, and its first visible effect is growing latency.
Saturation is one of the four golden signals from the Google SRE book, next to latency, traffic and errors. The SRE book describes it as how "full" a service is, with emphasis on the resources that are most constrained, and notes that many systems degrade well before they reach 100 percent utilization.
Utilization vs. saturation: the USE method
Brendan Gregg's USE method gives a checklist for every resource: look at Utilization, Saturation and Errors. The distinction between the first two matters:
- Utilization is the share of time or capacity a resource is busy, for example 85 percent CPU.
- Saturation is the amount of extra work the resource cannot serve yet, usually expressed as queue length or wait time.
A CPU at 100 percent utilization with an empty run queue is fully used but not saturated. A CPU at 70 percent average utilization can still be saturated in short bursts that averages hide. Saturation therefore tells you more about user impact than utilization alone.
Typical saturation metrics
| Resource | Saturation signal |
|---|---|
| CPU | Run queue length above the number of CPUs, load average relative to cores, CPU pressure (/proc/pressure/cpu), container CFS throttling |
| Memory | Swapping, major page faults, memory pressure stalls, OOM kills, frequent garbage collection pauses |
| Disk | Average I/O queue size (aqu-sz in iostat), I/O wait, I/O pressure |
| Network | Dropped packets, interface overruns, socket backlog overflows, TCP retransmits |
| Application pools | Threads waiting for a worker, connection pool wait time, pending requests |
| Queues and brokers | Queue depth, consumer lag, age of the oldest message |
On Linux, pressure stall information (PSI), available since kernel 4.20, is particularly useful because it reports directly what share of time tasks were stalled waiting for CPU, memory or I/O.
Why saturation matters
Queueing theory explains why saturation hurts so much: as utilization approaches capacity, waiting time grows sharply rather than linearly. A service that answers in 50 ms at moderate load can take seconds when a thread pool is exhausted, even though no component has failed. That is also why saturation is the right basis for capacity planning: if a disk fills at a steady rate, or a connection pool's wait time rises week over week, you can predict when it will run out and act before an incident.
Example: an exhausted connection pool
Illustrative case: an order service uses a database connection pool of 20 connections. Normal traffic needs about 12 at a time. A slow query introduced in a release raises the time each request holds a connection from 20 ms to 80 ms. By Little's Law, the number of connections in use rises roughly fourfold, the pool hits 20, and new requests start waiting for a connection. CPU and memory on the service look normal, but pool wait time jumps and p99 latency climbs to the pool timeout. The saturation signal (pool wait) points to the cause much faster than the utilization metrics.
Best practices
- Measure wait, not just busy. For every critical resource, track a queue length or wait time alongside utilization.
- Use high-resolution data. Saturation often occurs in bursts that one-minute averages flatten out.
- Instrument application-level pools. Thread pools, connection pools and in-process queues are frequent bottlenecks that host metrics do not show.
- Alert on symptoms, investigate with saturation. Page on latency and errors that affect users, then use saturation signals to find the constrained resource.
- Forecast. Trend saturation signals to plan capacity before limits are reached.
How Obseria helps
The Obseria Agent collects host and container saturation signals such as run queue, pressure stalls and throttling, while traces and runtime metrics show pool waits and garbage collection. Obseria TopoGraph links these to the services that depend on them, and anomaly detection flags rising saturation against learned baselines, so you can see which resource is constrained and who is affected.
