Cron job running twice at the same time (overlapping runs)

crontab(5) describes no check for an earlier run that is still going. If a job takes longer than its interval, the next run starts anyway, and two copies write to the same files. flock lets the second copy give up. This guide shows the line, and how to make a skipped run visible instead of silent.

You need: a Lateping account, shell access to a Linux server, about 5 minutes.

Why runs overlap

Check whether it is happening to you

List the processes for the job. Two lines with different start times mean two copies are running.

bash
ps -eo pid,lstart,cmd | grep '[b]ackup.sh'

The [b] stops grep from matching itself.

The fix: take a lock with flock

flock takes a lock on a file and runs your command while it holds it. With -n, a second copy gives up at once instead of waiting. The -E option sets the exit status flock returns in that case. The default is 1, which a failing script can also return, so pick a number your script never uses, such as 200. Valid values run from 0 to 255.

crontab
0 * * * * flock -n -E 200 /tmp/backup.lock /usr/local/bin/backup.sh

Replace /usr/local/bin/backup.sh with your own command. The lock file can be any path the cron user can write to. It is created if it does not exist.

If you would rather the second copy wait for the first, use -w with a number of seconds. It fails with the same -E status when the time runs out.

crontab
0 * * * * flock -w 300 -E 200 /tmp/backup.lock /usr/local/bin/backup.sh

Find out when a run was skipped

A skipped run is not a failure of the job, but you still want to know it happened. Send flock’s exit status as the ping. 0 records a success. Any other code records a failure and shows in the check’s Pings list and in the alert, so 200 reads as “skipped, lock held”.

  1. Create a check

    Create a check in Lateping with the same cron expression as the job and the server’s timezone. Set the grace period longer than the job normally takes.

  2. Ping with the exit code

    crontab
    0 * * * * curl -fsS -m 10 --retry 5 -o /dev/null https://lateping.com/p/<check-id>/start; flock -n -E 200 /tmp/backup.lock /usr/local/bin/backup.sh; curl -fsS -m 10 --retry 5 -o /dev/null https://lateping.com/p/<check-id>/$?
    

    The ; runs each part whatever happened before it, and $? is the status of flock: the script’s own exit code when it ran, or 200 when the lock was held. A code must be a plain number from 0 to 255; anything else returns 404.

  3. Test it

    Hold the lock yourself in one terminal, then run the line from another. The second run should return 200 at once and the check should show a failure with that code.

    bash
    flock /tmp/backup.lock sleep 60
    
    bash
    flock -n -E 200 /tmp/backup.lock true; echo $?
    

    The second command prints 200 while the first is still running.

Notes

  • A skipped run that sends /start and then /200. The /200 is a fail ping, so the skip shows as a failed run: you get a fail alert on the next scan. It isn’t a success, so it can’t trigger an unusual run time alert and it isn’t added to the check’s run-time baseline.
  • Lock files in /tmp. Some systems clear /tmp on reboot, which is fine for a lock. Use a path such as /var/lock/backup.lock or $HOME/.backup.lock if the cron user cannot write where you want it.
  • Children keep the lock. If your command starts a background process, that process holds the lock too. Add -o to close the lock descriptor before the command runs if that is not what you want.
  • One lock file per job. Two different jobs sharing a lock file will block each other.
  • Skipping quietly with -E 0. With this option the skip is recorded as a success with a run time near zero. On Pro and above, once the check has 10 timed runs, that can trigger a “too short” alert (under half the shortest recent run and at least 30 seconds shorter than the median), and the skip joins the baseline. Either send /start after flock has the lock, so a skipped copy never sends it, or switch off unusual run time alerts in the check’s Settings tab.
  • A skip means two emails. After a skip’s /200 the check is down. The next successful run sends a “recovered” alert, so you get a fail email and then a recovered email.
  • The copy that overran isn’t timed. If the skip’s /200 arrives while the first copy is still running, that copy’s own run isn’t timed when it finishes, because no /start follows the fail. It doesn’t get a “too long” alert.
  • Overlap hides in the clock. On a daylight-saving change, crontab(5) says repeated times can make matching jobs run twice. Run the server in UTC to avoid it.
  • Alerts go by email. On Pro and above they can also go to Slack, Discord or a webhook.

Create a free check at https://app.lateping.com/login.

Next steps

On this page