How to retry a failed cron job
cron runs your command once per match and does not run it again if it fails. To retry, put a small loop in a wrapper script. Ping Lateping once at the end, so a flaky first attempt that then works sends no alert.
You need: a Lateping account, shell access to a Linux server, about 5 minutes.
Why a failed cron job is not retried
Source: crontab(5).
Check whether it is happening to you
Look for a job that fails now and then and works when you run it by hand. Run it twice and compare the exit codes.
/usr/local/bin/backup.sh; echo "exit code $?"
Replace /usr/local/bin/backup.sh with your own command. If the code is sometimes non-zero for a network or lock error, a retry may help.
The fix: a wrapper that retries
Save this as /usr/local/bin/backup-retry.sh and make it executable with chmod +x.
#!/bin/sh
URL=https://lateping.com/p/<check-id>
attempts=3
n=1
curl -fsS -m 10 --retry 5 -o /dev/null "$URL/start"
until /usr/local/bin/backup.sh; do
code=$?
if [ "$n" -ge "$attempts" ]; then
curl -fsS -m 10 --retry 5 -o /dev/null "$URL/$code"
exit "$code"
fi
n=$((n + 1))
sleep 60
done
curl -fsS -m 10 --retry 5 -o /dev/null "$URL"
until runs the command and loops while it exits non-zero. After the third failure the script pings with the last exit code and stops. If any attempt succeeds, it pings success. sleep 60 waits 60 seconds between attempts; sleep(1) also takes m, h and d suffixes.
Then point cron at the wrapper:
17 2 * * * /usr/local/bin/backup-retry.sh
Find out when every attempt failed
Create a check
Create a check in Lateping with the same cron expression as the job and the server’s timezone (see the timezone guide if unsure).
Set the grace period for the retries
A run now lasts up to the job’s time times the attempts, plus the sleeps. For three attempts of 5 minutes each and two 60-second waits, that is 17 minutes. Set the grace period above it.
Read the result
A final failure pings the exit code, which records a fail. The code shows in the Pings list and in the fail alert, and the alert goes out within a minute. If every attempt is skipped because the server is down, no ping arrives and the check alerts after the grace period.
Notes
- Retry only what is safe to repeat. A job that sends an email or charges a card may do it twice. Make the job idempotent first, or retry only the step that failed.
- Put a limit on each attempt. A hung attempt never exits, so the loop never retries. Wrap the command in
timeout, as in the timeout guide. - Keep retries shorter than the schedule. If the loop can run past the next match, two copies overlap. See the overlapping guide.
- One start per run. The wrapper sends
/startonce, so the run time includes retries and waits. On Pro and above the unusual run time alert compares against that total. - A failed attempt is not an alert. Only the final ping is sent, so a job that works on its second try stays quiet. If you want to see those, log each attempt with
logger. - Alerts go by email. On Pro and above they can also go to Slack, Discord or a webhook.
Create a free check at https://app.lateping.com/login.