Heartbeat monitoring detects failures by expecting a periodic signal, the heartbeat, from a job, service or system and alerting when it does not arrive within the expected interval plus a grace period. It inverts classic monitoring: silence is the alarm. It is used for cron jobs, backups, batch pipelines and to verify that alerting itself works.
What heartbeat monitoring is
Most monitoring waits for something bad to happen: an error, a slow response, a failed health check. That does not work for tasks whose failure produces no signal at all. A nightly backup that never starts writes no error log. A cron job on a decommissioned host simply stops. Heartbeat monitoring covers this gap: the job sends a signal every time it runs successfully, and the monitor alerts when the signal does not arrive in time.
The same principle is known as a dead man's switch, after the device in trains that stops the vehicle when the operator stops pressing it. It applies wherever the absence of activity is the failure.
How it works
- Define the schedule. For each job, configure the expected interval, for example every 24 hours or a cron expression, and a grace period that tolerates normal variation in runtime.
- Send the heartbeat. At the end of a successful run the job calls a unique URL, pushes a metric or writes a timestamp. Optionally it also signals the start, so runtime can be measured, and an explicit failure.
- Evaluate silence. If no heartbeat arrives within interval plus grace, the monitor opens an alert. Many implementations also alert if a run started but never finished.
A minimal example for a backup script:
#!/bin/sh
set -e
pg_dump app > /backups/app.sql
curl -fsS --retry 3 https://heartbeat.example.com/ping/backup-nightlyBecause of set -e, the ping is only sent if the dump succeeded. The URL and service are illustrative; the pattern works with any tool that accepts check-ins.
Heartbeat versus health check and synthetic monitoring
| Approach | Direction | Detects |
|---|---|---|
| Health check | Pull: monitor asks the service | Service unreachable or unhealthy right now |
| Synthetic check | Pull: monitor simulates a user | Broken user journeys and endpoints from outside |
| Heartbeat | Push: job reports to the monitor | Jobs that did not run, ran too late or never finished |
Pull-based checks need a reachable endpoint, which batch jobs, workers behind NAT and short-lived containers do not have. Heartbeats work from anywhere that can make an outbound request.
Watching the watchers
A second important use is verifying the alerting path itself. If Prometheus, Alertmanager or the notification integration fails, no alerts fire and everything looks quiet. The kube-prometheus project therefore ships a Watchdog alert that always fires. Alertmanager forwards it continuously to an external dead man's switch, and if those notifications stop, the external service pages the on-call engineer. The same idea works for log and telemetry pipelines: an expected stream that falls silent is an incident.
Inside Prometheus you can express heartbeats as staleness checks, for example with kube-state-metrics:
time() - kube_cronjob_status_last_successful_time{cronjob="backup-nightly"} > 26 * 3600What to cover with heartbeats
- Scheduled jobs: cron jobs, Kubernetes CronJobs, scheduled serverless functions and CI pipelines that run on a timer.
- Data pipelines: ETL and ELT jobs, report generation, data exports to partners and search index rebuilds.
- Housekeeping: backups, certificate renewals, log rotation and cleanup tasks whose failure only shows weeks later.
- Long-running workers: queue consumers that should report regularly while they process messages.
- Edge devices and remote sites that can reach the internet but cannot be reached from it.
As a rule of thumb, anything that would hurt if it silently stopped for a day deserves a heartbeat.
Best practices
- Ping on success, not on start. A heartbeat sent before the work is done proves only that the scheduler ran.
- Choose grace periods deliberately. Too short creates noise and alert fatigue; too long delays detection past the point where it matters.
- Use one heartbeat per job and environment, so a missing check-in points to exactly one thing.
- Make the check-in resilient with short timeouts and retries, and never let a failed ping fail the job itself.
- Document the owner and runbook in the alert, since jobs are often forgotten until they fail.
Common mistakes
Typical mistakes are monitoring only whether a job started, forgetting to pause heartbeats during planned maintenance, and treating "ran successfully" as "did the right thing". A backup job that completes in two seconds with an empty file sends a perfect heartbeat. Add a sanity metric such as output size or record count where it matters.
How Obseria helps
Job metrics and logs sent to Obseria via OpenTelemetry or Prometheus can drive staleness and absence alerts, and incident management routes them to the right team. For outside-in coverage of endpoints and user journeys, combine heartbeats with synthetic monitoring from 50+ locations.
