Health Check

What a health check is

A health check is an automated question asked over and over: is this instance OK? The asker is usually infrastructure, not a person: a container orchestrator, a load balancer, a service mesh or an external monitoring system. The answer drives automatic action. A failed instance is restarted, removed from rotation or replaced, often before any user notices.

Health checks come in several flavors. Process checks test whether something is running. Endpoint checks call an HTTP or gRPC endpoint such as /healthz and expect a success status. Synthetic checks run from outside the system and exercise a real user path. Each answers a slightly different question, and confusing them is a common source of outages.

Kubernetes liveness, readiness and startup probes

Kubernetes distinguishes three probe types, each with a different consequence when it fails:

Probe	Question	On failure
Liveness	Is the container stuck beyond recovery?	The kubelet kills and restarts the container.
Readiness	Can the container serve traffic right now?	The pod is removed from Service endpoints until it is ready again; it is not restarted.
Startup	Has the application finished starting?	Liveness and readiness checks are held back until it succeeds; if it never does, the container is restarted.

Each probe can use one of four mechanisms: httpGet (any status from 200 to 399 counts as success), tcpSocket, exec (exit code 0 is success) or grpc, which uses the standard gRPC health checking protocol. Timing is controlled by initialDelaySeconds, periodSeconds (default 10), timeoutSeconds (default 1), failureThreshold (default 3) and successThreshold (default 1).

Example configuration

containers:
- name: api
  image: example/api:1.4.2
  startupProbe:
    httpGet: { path: /livez, port: 8080 }
    periodSeconds: 5
    failureThreshold: 24      # allows up to 120 s to start
  livenessProbe:
    httpGet: { path: /livez, port: 8080 }
    periodSeconds: 10
    failureThreshold: 3
  readinessProbe:
    httpGet: { path: /readyz, port: 8080 }
    periodSeconds: 5
    failureThreshold: 2

In this illustrative setup, /livez only checks that the process can respond, while /readyz also verifies that caches are warm and the connection pool to the primary database is available. A slow start no longer triggers restarts, and a temporarily overloaded pod drops out of rotation without being killed.

Why health checks matter

Health checks are the basis of self-healing systems and safe rollouts. Rolling updates and canary deployments rely on readiness to decide when a new pod may receive traffic. Load balancers such as cloud application load balancers use their own health checks to spread traffic only across healthy targets. Badly designed checks, however, can amplify problems instead of containing them.

Health endpoint conventions

Kubernetes itself exposes /livez and /readyz on its API server, and many frameworks follow similar conventions; Spring Boot Actuator, for example, provides separate liveness and readiness health groups. A good health endpoint returns quickly, uses status codes rather than only a response body to signal state, and can optionally report details of individual checks for humans in a protected, verbose mode. Keep the contract stable, because platform teams, load balancers and monitoring tools all depend on it.

Best practices

• Keep liveness shallow. A liveness check should only fail when a restart will actually help, such as a deadlocked process. Do not check databases or downstream services in liveness.
• Make readiness meaningful. Readiness can include critical local dependencies, but beware of shared ones: if every pod reports not ready when one database blips, the whole service disappears at once.
• Use startup probes for slow starters instead of long initialDelaySeconds, which delay detection of real failures.
• Keep checks cheap and fast, well within timeoutSeconds, and exclude them from request metrics and logs to avoid noise.
• Do not rely on internal checks alone. Combine them with external synthetic monitoring that tests real user journeys and measures uptime.

Common mistakes

The most damaging mistake is a deep liveness check. When a shared database slows down, every pod's liveness probe times out, Kubernetes restarts all of them, and the restart storm turns a partial degradation into a full outage. Other mistakes include a default one-second timeout for an endpoint that sometimes needs two, health endpoints that always return 200 regardless of state, and health checks that are reachable from the public internet and leak internal details.

How Obseria helps

Obseria Kubernetes monitoring tracks probe failures, restarts and readiness changes per workload and correlates them with deployments and resource pressure. Synthetic checks from 50+ locations test endpoints and user journeys from the outside, and results can feed status pages. See also the Kubernetes observability checklist.