Cron job hangs and never finishes (kill it with timeout)
cron never stops a command that is still running. If a job hangs on a dead network connection or a locked file, it sits there until someone notices. timeout(1) kills it after a time you choose and exits with 124, which you can send as the ping.
You need: a Lateping account, shell access to a Linux server, about 5 minutes.
Why a job hangs forever
Check whether it is happening to you
Show how long each copy of the job has been running. A job that should take a minute and shows hours is hung.
ps -eo pid,etime,cmd | grep '[b]ackup.sh'
The [b] stops grep from matching itself. etime is the time since the process started.
The fix: wrap the job in timeout
timeout runs a command and stops it if it is still running after a duration. The duration is a number with an optional suffix: s for seconds (the default), m, h or d. By default it sends TERM.
0 * * * * timeout 30m /usr/local/bin/backup.sh
Replace /usr/local/bin/backup.sh with your own command. Some programs ignore TERM. Add -k and a second duration to send KILL if the command is still running that long after TERM.
0 * * * * timeout -k 1m 30m /usr/local/bin/backup.sh
The exit status tells you what happened. According to timeout(1):
- 124: the command timed out.
- 125: timeout itself failed.
- 126: the command was found but could not be run.
- 127: the command was not found.
- 137: the command, or timeout, was sent KILL.
- Anything else is the command’s own exit status.
Find out when a run was killed
Send timeout’s exit status as the ping. 0 records a success. Any other code records a failure and shows in the check’s Pings list and in the alert, so 124 reads as “timed out”.
Create a check
Create a check in Lateping with the same cron expression as the job and the server’s timezone. Set the grace period longer than the job normally takes.
Ping with the exit code
0 * * * * curl -fsS -m 10 --retry 5 -o /dev/null https://lateping.com/p/<check-id>/start; timeout -k 1m 30m /usr/local/bin/backup.sh; curl -fsS -m 10 --retry 5 -o /dev/null https://lateping.com/p/<check-id>/$?The
;runs each part whatever happened before it, and$?is the status of timeout. A code must be a plain number from 0 to 255; anything else returns404.Test it
Time out a command that sleeps longer than the limit. It should print
124.timeout 2 sleep 10; echo $?The same command with
-kand a short second duration also ends at the limit.timeout -k 1 2 sleep 10; echo $?
Notes
- Without a ping, a hang still alerts, but later. A job that hangs never pings, so the check goes late at its due time and you get one late alert. Lateping has no alert for a job that sent
/startand never finished. Withtimeoutand the exit code, the fail ping arrives when the limit is hit and the alert follows on the next scan, within about a minute. - A timeout is a fail, not an unusual run. A ping with code 124 is a fail, so it isn’t compared against the check’s recent run times and it can’t raise an unusual run time alert. Only success pings are timed.
- Choose a limit above the normal run time. Set it well over your slowest healthy run, so a slow day is not killed.
--preserve-statushides the timeout. With-p, timeout exits with the command’s own status even when it times out, so you no longer get 124.- Children can survive. timeout signals the command it started. A child the command leaves behind may keep running. With
-f(--foreground), children of the command are not timed out. - A duration of 0 turns the timeout off.
- Combine with flock. A hung job that holds a lock makes every later run skip. Put both in the line:
flock -n -E 200 /tmp/backup.lock timeout 30m /usr/local/bin/backup.sh. See the overlapping runs guide. - Alerts go by email. On Pro and above they can also go to Slack, Discord or a webhook.
Create a free check at https://app.lateping.com/login.