A noisy neighbor is a workload that monopolizes a shared resource such as CPU, memory bandwidth, disk I/O, network or a database connection pool, so that other tenants on the same host, Kubernetes node or multi-tenant service see higher latency or errors although their own load has not changed.
What is a noisy neighbor?
A noisy neighbor is a workload that consumes a disproportionate share of a resource it shares with others, and thereby degrades their performance. The classic case is a virtual machine on a busy hypervisor, but the pattern appears wherever resources are pooled: containers on a Kubernetes node, tenants in a shared database, partitions on a shared disk, services behind the same NAT gateway or a batch job sharing a connection pool with an API.
The defining symptom is that the victim gets slower without any change in its own traffic or code. Its latency rises, its tail percentiles get worse, timeouts appear, yet its request rate is flat. That is what makes noisy neighbors hard to debug: the cause sits outside the service that shows the symptom.
Which resources are affected?
| Shared resource | Typical noisy neighbor | Signal to look at |
|---|---|---|
| CPU on a hypervisor | Another VM on the same physical host | CPU steal time (%st in top/vmstat) |
| CPU on a Kubernetes node | A pod without limits or with low requests | CFS throttling, node CPU saturation, run queue length |
| Memory and caches | A memory-hungry pod or a cache-thrashing job | Memory pressure, OOM kills, evictions, page faults |
| Disk I/O | Log-heavy or backup workload on the same volume | I/O wait, disk queue length, IOPS and throughput limits |
| Network | Bulk transfer on the same NIC or gateway | Bandwidth, packet drops, retransmits |
| Shared database | One tenant running expensive queries | Lock waits, connection pool usage, slow query log |
How to detect a noisy neighbor
Detection is mostly about correlation across layers. You compare the victim's symptoms with resource signals of the shared layer underneath and with the behavior of the other tenants on it.
- Confirm the symptom is not self-inflicted. Check that request rate, payload sizes and deployed version of the slow service are unchanged. A baseline for latency and throughput makes deviations obvious.
- Find the shared layer. Which node, host, volume, database or gateway does the service run on? Topology data answers this quickly.
- Look at contention signals on that layer. Steal time, throttled CPU periods, Linux pressure stall information (
/proc/pressure/cpu,memory,io), disk queue length or connection pool waits. - Identify the top consumer. Rank co-located workloads by usage of the contended resource during the incident window. A spike that starts at the same minute as the victim's slowdown is a strong lead.
Example: a batch job on a shared node
An illustrative case: a checkout API runs on a Kubernetes node together with a nightly report job. At 02:00 the API's p99 latency jumps from 180 ms to 900 ms while its request rate stays constant. The node shows CPU utilization near 100 percent and a rising run queue, and the API's containers show a sharp increase in container_cpu_cfs_throttled_periods_total. The report pod has no CPU limit and a request of only 100m, so the scheduler placed it next to latency-sensitive pods. Setting realistic requests and a limit for the job, or moving it to a dedicated node pool, removes the interference.
How to prevent noisy neighbors
- Set resource requests and limits. In Kubernetes, requests drive scheduling and the QoS class (Guaranteed, Burstable, BestEffort). Latency-critical pods benefit from Guaranteed QoS.
- Isolate by workload type. Separate node pools, taints and tolerations, or dedicated hosts for batch, streaming and interactive workloads.
- Use quotas and fair scheduling. Namespace ResourceQuotas, I/O limits on volumes, per-tenant connection limits and rate limiting in shared services.
- Right-size burstable capacity. Burstable instance types and overcommitted nodes are cheap until several tenants burst at once.
- Alert on contention, not only on utilization. Steal time, throttling and pressure metrics show contention earlier than average CPU.
Common mistakes
- Scaling out the victim service, which adds cost but leaves the contention in place if new replicas land on the same node.
- Looking only at averages: noisy neighbors usually hurt p99 long before averages move.
- Setting very tight CPU limits on latency-sensitive services, which causes throttling that looks like a noisy neighbor but is self-inflicted.
- Ignoring non-CPU resources such as disk queues and connection pools.
How Obseria helps
The Obseria Agent collects host, container and process metrics together with traces, and Obseria TopoGraph maps which services share which nodes, hosts and databases. When latency rises, Obseria Intelligence can run an agentic root-cause investigation that follows this topology to the contended resource and the workload causing it, with a chain of evidence. See Kubernetes monitoring for node- and pod-level views.
