An OOM kill (OOMKilled) is the forced termination of a process by the Linux out-of-memory killer when the system or a cgroup runs out of memory. In Kubernetes it happens when a container exceeds its memory limit; the container is stopped with SIGKILL, reports exit code 137 and the reason OOMKilled.
What is an OOM kill?
An OOM kill is what happens when Linux cannot satisfy a memory request and has to free memory by force. The kernel's out-of-memory killer selects a victim process and sends it SIGKILL, which cannot be caught or handled. The process ends immediately, without running shutdown hooks, flushing buffers or finishing in-flight requests.
In containers, memory is constrained by cgroups. When a container's memory usage reaches its limit (memory.max in cgroup v2) and the kernel cannot reclaim enough, the OOM killer acts inside that cgroup only. Kubernetes then reports the container status as OOMKilled with exit code 137, which is 128 plus signal number 9.
How the OOM killer chooses a victim
For system-wide memory exhaustion, the kernel computes an oom_score for each process, mainly based on how much memory it uses, adjusted by oom_score_adj (from -1000 to 1000). The process with the highest score is killed. Kubernetes sets oom_score_adj according to the pod's quality-of-service class:
| QoS class | Condition | OOM priority |
|---|---|---|
| Guaranteed | Requests equal limits for CPU and memory in every container | Killed last |
| Burstable | At least one request or limit set, but not Guaranteed | In between, depending on requests |
| BestEffort | No requests or limits at all | Killed first |
A container OOM kill caused by its own limit is different from a kubelet eviction, where the node is under memory pressure and the kubelet removes pods proactively. Both end in a restart or rescheduling, but they show up with different reasons and need different fixes.
Common causes
- Limit set too low for the actual working set, often after traffic or data volume grew.
- A memory leak that slowly pushes usage towards the limit.
- Runtime not aware of the container limit. A JVM heap configured larger than the limit, or heap plus metaspace, thread stacks and direct buffers adding up to more than the limit.
- Load spikes that create many concurrent requests, each holding large buffers.
- Page cache and tmpfs usage that counts against the cgroup, for example files written to an in-memory volume.
What counts against the memory limit
Many OOM kills are surprising because teams compare the limit with the wrong number. The kernel charges a cgroup for anonymous memory (heap, stacks, native allocations), for page cache created by file reads and writes, and for kernel memory such as socket buffers. Reclaimable page cache is usually freed under pressure, so Kubernetes and the kubelet use the working set, roughly total usage minus inactive file cache, as the relevant figure. That is why container_memory_working_set_bytes is a better predictor of an OOM kill than container_memory_usage_bytes.
For managed runtimes, the heap is only part of the picture. A JVM also uses metaspace, code cache, thread stacks, garbage collector structures and direct buffers; a Node.js process holds buffers outside the V8 heap. A heap setting that looks safe can therefore still push the container over its limit. Watching both heap metrics and container working set side by side avoids this blind spot, and it helps to separate a slow leak from a sizing problem.
Example: diagnosing OOMKilled in Kubernetes
An illustrative check of a restarting pod:
kubectl describe pod checkout-7d9f
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
Limits: memory: 512MiA PromQL query such as max_over_time(container_memory_working_set_bytes{container="checkout"}[1h]) / on(pod) group_left kube_pod_container_resource_limits{resource="memory", container="checkout"} shows how close the container gets to its limit. If the working set rises steadily between restarts, look for a leak. If it jumps only during peaks, the limit or the concurrency is too tight. kube-state-metrics exposes the last termination reason, so you can count OOM kills per workload rather than finding them by accident.
How to prevent OOM kills
- Size memory limits from observed peak working set plus headroom, and set requests close to real usage.
- Make runtimes container-aware, for example with
-XX:MaxRAMPercentagefor the JVM orGOMEMLIMITfor Go, leaving room for non-heap memory. - Bound concurrency with pools and queues so peaks do not multiply memory use.
- Alert on memory approaching the limit and on OOMKilled events, not just on restarts.
- Treat repeated OOM kills as a reliability problem: they look like random failures to users and can cascade, which ties into saturation monitoring.
How Obseria helps
Obseria's Kubernetes monitoring shows container memory against requests and limits, termination reasons and restart counts per workload, linked to the traces and logs of the affected pods. Continuous profiling helps to see which code allocates memory before the kill. For a broader list of what to watch in clusters, see the Kubernetes observability checklist.
