Cron job hangs and never finishes (kill it with timeout)

cron never stops a command that is still running. If a job hangs on a dead network connection or a locked file, it sits there until someone notices. timeout(1) kills it after a time you choose and exits with 124, which you can send as the ping.

You need: a Lateping account, shell access to a Linux server, about 5 minutes.

Why a job hangs forever

Check whether it is happening to you

Show how long each copy of the job has been running. A job that should take a minute and shows hours is hung.

bash
ps -eo pid,etime,cmd | grep '[b]ackup.sh'

The [b] stops grep from matching itself. etime is the time since the process started.

The fix: wrap the job in timeout

timeout runs a command and stops it if it is still running after a duration. The duration is a number with an optional suffix: s for seconds (the default), m, h or d. By default it sends TERM.

crontab
0 * * * * timeout 30m /usr/local/bin/backup.sh

Replace /usr/local/bin/backup.sh with your own command. Some programs ignore TERM. Add -k and a second duration to send KILL if the command is still running that long after TERM.

crontab
0 * * * * timeout -k 1m 30m /usr/local/bin/backup.sh

The exit status tells you what happened. According to timeout(1):

  • 124: the command timed out.
  • 125: timeout itself failed.
  • 126: the command was found but could not be run.
  • 127: the command was not found.
  • 137: the command, or timeout, was sent KILL.
  • Anything else is the command’s own exit status.

Find out when a run was killed

Send timeout’s exit status as the ping. 0 records a success. Any other code records a failure and shows in the check’s Pings list and in the alert, so 124 reads as “timed out”.

  1. Create a check

    Create a check in Lateping with the same cron expression as the job and the server’s timezone. Set the grace period longer than the job normally takes.

  2. Ping with the exit code

    crontab
    0 * * * * curl -fsS -m 10 --retry 5 -o /dev/null https://lateping.com/p/<check-id>/start; timeout -k 1m 30m /usr/local/bin/backup.sh; curl -fsS -m 10 --retry 5 -o /dev/null https://lateping.com/p/<check-id>/$?
    

    The ; runs each part whatever happened before it, and $? is the status of timeout. A code must be a plain number from 0 to 255; anything else returns 404.

  3. Test it

    Time out a command that sleeps longer than the limit. It should print 124.

    bash
    timeout 2 sleep 10; echo $?
    

    The same command with -k and a short second duration also ends at the limit.

    bash
    timeout -k 1 2 sleep 10; echo $?
    

Notes

  • Without a ping, a hang still alerts, but later. A job that hangs never pings, so the check goes late at its due time and you get one late alert. Lateping has no alert for a job that sent /start and never finished. With timeout and the exit code, the fail ping arrives when the limit is hit and the alert follows on the next scan, within about a minute.
  • A timeout is a fail, not an unusual run. A ping with code 124 is a fail, so it isn’t compared against the check’s recent run times and it can’t raise an unusual run time alert. Only success pings are timed.
  • Choose a limit above the normal run time. Set it well over your slowest healthy run, so a slow day is not killed.
  • --preserve-status hides the timeout. With -p, timeout exits with the command’s own status even when it times out, so you no longer get 124.
  • Children can survive. timeout signals the command it started. A child the command leaves behind may keep running. With -f (--foreground), children of the command are not timed out.
  • A duration of 0 turns the timeout off.
  • Combine with flock. A hung job that holds a lock makes every later run skip. Put both in the line: flock -n -E 200 /tmp/backup.lock timeout 30m /usr/local/bin/backup.sh. See the overlapping runs guide.
  • Alerts go by email. On Pro and above they can also go to Slack, Discord or a webhook.

Create a free check at https://app.lateping.com/login.

Next steps

On this page