A dead man's switch for cron jobs

Most monitoring waits for something to go wrong and report it. A job that never runs reports nothing. A dead man's switch flips that around: the job checks in on schedule, and silence is the alarm.

You need: a Lateping account and any scheduled job, about 5 minutes.

The problem it solves

Logs, error trackers and uptime checks all watch for something that happens: an exception, a failed request, a bad status code. The failures that hurt most with scheduled jobs are the ones where nothing happens.

In each case there is no error to catch. The only signal is that something that should have happened didn’t.

How a dead man’s switch works

The name comes from trains: the driver holds a handle, and if they let go, the train stops. For a job, “holding the handle” is checking in.

  1. The job sends a ping (an HTTP request to a URL) each time it finishes.
  2. The switch knows the schedule, so it knows when the next ping is due.
  3. If the ping hasn’t arrived by the due time plus a grace period, the switch alerts.

Because the alert is triggered by silence, it catches every way a job can fail to run, including ones you didn’t think of. It’s also called heartbeat monitoring or cron monitoring.

Set one up

  1. Create a check

    In Lateping, create a check with the job’s schedule (an interval like “every hour”, or a cron expression with a timezone) and a grace period a little longer than the job normally takes.

  2. Ping when the job succeeds

    Add one request to the end of the job, after the work, so it only runs when the work did:

    bash
    /usr/local/bin/nightly-report && curl -fsS -m 10 --retry 3 -o /dev/null https://lateping.com/p/<check-id>
    

    The && means the ping is sent only if the job exits 0. A failed run sends no ping, so it alerts too, after the grace period.

  3. Optional: report failures and run times

    Two more URLs make the switch faster and more useful. Pinging the job’s exit code alerts at once instead of waiting for the deadline, and shows the code in the alert. /start lets Lateping measure how long each run takes.

    bash
    curl -fsS -m 10 --retry 3 -o /dev/null https://lateping.com/p/<check-id>/start
    /usr/local/bin/nightly-report
    status=$?
    curl -fsS -m 10 --retry 3 -o /dev/null "https://lateping.com/p/<check-id>/$status"
    

    0 records a success and any other code, from 1 to 255, a failure. If you don’t have an exit code to send, /fail reports a failure without one.

Getting the details right

  • Ping last. A ping at the start of the job only proves it started. Ping after the work, or the switch stays happy while the job fails.
  • Set the grace period from real run times. Too short and a slow night is a false alarm; too long and you hear late. Start generous and tighten once you know the job’s normal duration.
  • Watch for runs that finish but do nothing. Once a check has 10 timed runs (runs that sent /start), Pro and above also get an alert when a run is much shorter or longer than the last 20. It is on by default and can be switched off in the check’s Settings tab. It does not catch a started job that never finishes.
  • Don’t let monitoring break the job. Keep the ping’s own failure from failing the job (|| true, or the && form above, which never changes the job’s exit code).
  • The switch must not cry wolf when it is the one down. If the monitoring service itself has an outage, it must not then alert on every check at once. Lateping moves every deadline that fell inside its own downtime back by the length of the outage, and holds alerts if a large share of checks go late together.
  • One check per job. Shared checks hide which job stopped.

Platform guides

The same idea, with copy-paste setup for each platform: GitHub Actions, Cloudflare Workers, crontab, Vercel Cron, pg_cron, Kubernetes CronJobs, systemd timers, Laravel and backup jobs.

Next steps

On this page