The Kubernetes Observability Checklist for Production Clusters

Kubernetes makes deployment easy and debugging hard. A slow request can be caused by the application, a throttled container, a noisy neighbor on the same node, a DNS problem, a failing volume or a control plane under pressure. Kubernetes observability means you can move between those layers in minutes instead of hours.

This checklist is the one we use when reviewing production clusters. It is organized by layer, and each item is something you can verify today. Use it as an audit before your next on-call rotation, or as a target state when you set up a new cluster.

The four layers of Kubernetes observability

Every useful signal in a cluster belongs to one of four layers. Gaps usually appear where one layer is covered well and its neighbor is ignored:

Layer	Typical questions	Main sources
Infrastructure	Are nodes healthy? Is there disk, memory or network pressure?	Node metrics, kubelet, cloud provider APIs
Kubernetes objects	Are deployments at desired replicas? Are pods pending or restarting?	kube-state-metrics, Kubernetes events, API server
Workloads	Are containers throttled, OOMKilled or close to limits?	cAdvisor via kubelet, container logs
Applications	Are users affected? Which service causes the latency?	Traces, application metrics, structured logs

Checklist: nodes and cluster infrastructure

• Node conditions are collected and alerted on: MemoryPressure, DiskPressure, PIDPressure and Ready. A node that flaps between Ready and NotReady explains many "random" pod restarts.
• CPU, memory, filesystem and network per node are visible, including allocatable versus requested capacity. A node can be at 30% CPU usage and still be unable to schedule pods because requests are fully allocated.
• Ephemeral storage usage is tracked. Container logs and emptyDir volumes fill node disks and trigger evictions.
• Cloud provider context such as instance type, zone and node pool is attached to every node signal, so you can tell whether a problem is confined to one zone.
• Cluster autoscaler activity is visible: scale-up failures and pods pending because of insufficient capacity.

Checklist: workloads and Kubernetes objects

• Restarts and termination reasons are captured per container, especially OOMKilled, Error and CrashLoopBackOff.
• CPU throttling is measured, not only CPU usage. The ratio of throttled periods to total periods reveals limits that are too tight even when average usage looks low.
• Memory working set against limits is tracked per container. This is the value the kubelet uses for OOM decisions, not RSS or cache.
• Desired versus available replicas for Deployments, StatefulSets and DaemonSets are compared continuously.
• Pending pods and their reasons (unschedulable, image pull errors, missing volumes) are visible.
• Kubernetes events are stored, not only shown in kubectl describe. Events expire from the API server after a short time by default, so they are gone by the time you write the postmortem unless you collect them.
• PersistentVolumeClaims report capacity and usage, and failing volume attachments raise an alert.

With the OpenTelemetry Collector, most of these signals come from three receivers. The kubeletstats receiver runs in the DaemonSet; k8s_cluster and k8sobjects run in a single-replica Deployment so cluster-wide data is not collected once per node:

receivers:
  kubeletstats:
    collection_interval: 30s
    auth_type: serviceAccount
    endpoint: "https://${env:K8S_NODE_NAME}:10250"
    metric_groups: [node, pod, container, volume]
  k8s_cluster:
    collection_interval: 30s
    node_conditions_to_report: [Ready, MemoryPressure, DiskPressure, PIDPressure]
    allocatable_types_to_report: [cpu, memory, ephemeral-storage]
  k8sobjects:
    objects:
      - name: events
        mode: watch
        group: events.k8s.io

Checklist: control plane and cluster services

• API server latency and error rate per verb are tracked. Slow LIST calls from operators or controllers can degrade the whole cluster.
• etcd health (leader changes, database size, fsync latency) is monitored on self-managed clusters. On managed services, check which control plane metrics your provider exposes.
• CoreDNS request rate, error rate (SERVFAIL, NXDOMAIN spikes) and latency are visible. DNS issues often show up as timeouts in unrelated services.
• Ingress controllers and service meshes report request rate, errors and latency per route, giving you a user-facing view at the edge.
• Admission webhooks are monitored for latency and failures; a slow webhook blocks every deployment.
• Certificate expiry for ingress and internal certificates is tracked with enough lead time to renew.

Checklist: logs, traces and correlation

• Container logs are collected from every node, including system namespaces, with a defined retention.
• Logs are structured (JSON or logfmt) where you control the application, and parsed where you do not.
• Every signal carries the same resource attributes: k8s.cluster.name, k8s.namespace.name, k8s.deployment.name, k8s.pod.name, k8s.node.name and a team or owner label.
• Traces cover the request path through ingress, services and data stores, using OpenTelemetry SDKs or auto-instrumentation.
• Trace IDs appear in logs, so you can jump from a failing span to the exact log lines of that request.
• Deployments are recorded as events, with version and commit, so that changes can be correlated with regressions.

Collecting container logs with the filelog receiver and the container parser handles the different runtime formats automatically:

receivers:
  filelog:
    include: [/var/log/pods/*/*/*.log]
    exclude: [/var/log/pods/*/otel-collector/*.log]
    start_at: end
    include_file_path: true
    operators:
      - type: container
        id: container-parser

Combined with the k8sattributes processor, every log line is tagged with its namespace, workload and pod, which is the foundation for every query that follows.

Checklist: alerts that matter

A cluster can produce hundreds of alerts that are technically correct and practically useless. Keep paging alerts close to user impact and route everything else to tickets or dashboards:

1. Page on symptoms: SLO burn rate for user-facing services, error rate and latency at the ingress.
2. Ticket on causes: pods in CrashLoopBackOff for more than ten minutes, repeated OOMKills, sustained CPU throttling above 25%, PVCs above 85% usage.
3. Page on capacity only when it blocks recovery: pods pending for more than five minutes in critical namespaces, nodes NotReady in more than one zone.
4. Alert on missing data: a namespace that stops sending telemetry is a problem in itself.

A useful OQL query during reviews lists the workloads with the most OOM kills over the last week, which points directly at memory limits that need adjustment:

fetch events
| filter k8s.event.reason == "OOMKilling" or container.termination.reason == "OOMKilled"
| summarize kills = count(), by:{k8s.namespace.name, k8s.deployment.name}
| sort kills desc
| limit 15

Putting the checklist to work during an incident

The value of the checklist shows when something breaks. Take a typical page: the checkout SLO burns budget, and latency at the ingress has doubled. With all four layers covered, the investigation follows a predictable path instead of a series of guesses:

1. Application layer: the traces of slow checkout requests show that time is spent in the inventory service, not in checkout itself.
2. Workload layer: the inventory containers show CPU throttling above 40% since the last deployment, while average CPU usage looks unremarkable.
3. Kubernetes objects: the deployment event shows a new version with a reduced CPU limit, introduced by a change to a shared Helm chart.
4. Infrastructure: node metrics confirm there is no node-level pressure, which rules out a noisy neighbor or a failing node.

Each step takes a single query because the signals share the same resource attributes. Without stored events, the deployment would be invisible; without throttling metrics, the CPU graph would look healthy; without trace context, the search would start in the wrong service. Every gap in the checklist turns one of these steps into a long manual search, and that is where hours of MTTR are lost.

Checklist: cost and efficiency

• Requests versus actual usage per workload are compared regularly. Overprovisioned requests waste node capacity; underprovisioned ones cause throttling and evictions.
• Idle namespaces and orphaned resources (unused PVCs, load balancers without backends) are reported.
• Telemetry volume per namespace is known, so noisy workloads can be addressed before they drive observability costs.
• Health check traffic is filtered from traces and logs in the pipeline.

Kubernetes observability with Obseria

The Obseria Agent is installed once per node as a DaemonSet and collects node, pod and container metrics, logs, traces and profiles, with Kubernetes metadata attached automatically. Obseria TopoGraph maps the relationships between clusters, nodes, workloads and services, so an alert on a checkout service links straight to the throttled container and the node it runs on. Existing OpenTelemetry Collector deployments can send OTLP directly, and the receiver configurations above work unchanged.

Whichever tooling you use, run through this checklist once per quarter. Clusters change, new workloads arrive and the gaps tend to open where nobody is looking.