How to know if a cron job failed

cron doesn't record whether a job succeeded. It starts the command, mails any output to a mailbox that usually doesn't exist, and moves on. You can find out after the fact from exit codes and logs. To hear about it the same day, including when the job never ran at all, have the job ping a URL and alert when the ping doesn't arrive.

You need: shell access to the server, and a Lateping account for the last step. About 10 minutes.

Why you don’t hear about it

Check the exit code

Every command exits with a status: 0 means success, anything else means failure. Run the job by hand and print its status:

bash
/usr/local/bin/backup.sh; echo "exit status: $?"

A job that works in your terminal can still fail under cron, because cron starts it with a minimal environment. To get closer to what cron sees, run it with an empty environment and cron’s default PATH:

bash
env -i HOME="$HOME" LOGNAME="$USER" SHELL=/bin/sh PATH=/usr/bin:/bin /bin/sh -c '/usr/local/bin/backup.sh; echo "exit status: $?"'

command not found here, and not in your own shell, means the script relies on your PATH. Use full paths in the script or set PATH at the top of the crontab.

Check cron’s own log

Look for the job’s CMD line around the time it should have run:

bash
grep CRON /var/log/syslog             # Debian and Ubuntu with rsyslog
journalctl -u cron --since yesterday  # Debian and Ubuntu with the systemd journal
journalctl -u crond --since yesterday # RHEL, Fedora, Rocky and Alma (cronie)

Debian 12 and later don’t install rsyslog by default, so /var/log/syslog may not exist; use journalctl there. On RHEL-family systems cronie also writes to /var/log/cron.

A CMD (...) line means cron started the command. It tells you nothing about how it ended. If there is no line at all for the expected time, the job didn’t run: check crontab -l for the line, and systemctl status cron for the daemon.

On Debian and Ubuntu, cron can also log jobs that exit non-zero. Its -L option takes the sum of 1 (log starts), 2 (log ends) and 4 (log failed jobs). Set it in /etc/default/cron, then run sudo systemctl restart cron:

/etc/default/cron
EXTRA_OPTS="-L 5"

That records failures, but only in a log on the same server. Someone still has to read it.

MAILTO, and why it’s usually broken

cron emails a job’s output (stdout and stderr) to MAILTO, or to the crontab’s owner if it isn’t set. In practice it rarely reaches anyone:

  • It needs a mail transfer agent. cron hands the message to a local sendmail. Without one, Debian’s cron logs No MTA installed, discarding output and the output is gone.
  • With one, mail often stays local. A default MTA delivers to /var/mail/<user> on the server, or sends from an address that other mail servers reject or file as spam.
  • No output, no mail. A job that fails quietly, or exits non-zero without printing, sends nothing. A job that prints on every successful run sends mail every night, which people learn to filter.

If you do have working outbound mail, MAILTO is worth keeping for the error text. It can’t tell you a job failed without saying so, or never ran.

Make the script fail loudly

A script can fail halfway and still exit 0. By default bash carries on after a failed command, and a pipeline’s exit status is its last command’s. Put this at the top of every job script:

bash
#!/usr/bin/env bash
set -euo pipefail
  • -e stops at the first command that fails.
  • -u treats an unset variable as an error, instead of quietly using an empty string.
  • -o pipefail makes pg_dump | gzip fail when pg_dump fails, not only when gzip does.

Now the script’s exit code reflects every step, which is what the logging and the ping below rely on.

Log each run with a timestamp

Send the script’s output to a file, with the time each run started and ended. One exec line near the top redirects everything after it:

bash
exec >> /var/log/nightly-report.log 2>&1
echo "$(date -Is) start"

The user the job runs as must be able to write the file. Rotate it with logrotate if the job is chatty. Like cron’s log, this answers “what happened?” after you already suspect something did.

What all of this misses

Exit codes, logs and mail all describe a run that happened. None of them can report a run that didn’t:

  • the server was off, or rebuilt without its crontab;
  • the cron daemon is stopped;
  • the line was edited away, or commented out while someone was testing;
  • the schedule never matches, such as 0 0 30 2 * (February 30th).

The only signal is silence. To catch it, something outside the server has to expect the job and notice when it doesn’t check in. That is a heartbeat, or dead man’s switch: the job pings a URL when it finishes, and you get an alert when the ping is late.

Add a heartbeat ping

  1. Create a check

    Create a check in Lateping with the job’s cron expression and the server’s timezone (timedatectl shows it). Set the grace period a little longer than the job normally takes.

  2. Wrap the job

    This script brings the pieces together. It stops on any failed step, logs each run with a timestamp and exit status, and pings Lateping: /start when the run begins, then the exit code when it ends. 0 records a success and any other code a failure. A failure alerts within a minute and shows the code; a run that never happens alerts once the grace period passes.

    /usr/local/bin/run-nightly-report.sh
    #!/usr/bin/env bash
    set -euo pipefail
    PING=https://lateping.com/p/<check-id>
    
    exec >> /var/log/nightly-report.log 2>&1
    log() { echo "$(date -Is) $*"; }
    
    # Runs on every exit, so a failed step, an unset variable or a plain `exit 1` all report.
    finish() {
      status=$?
      log "exit $status"
      curl -fsS -m 10 --retry 3 -o /dev/null "$PING/$status" || true
    }
    trap finish EXIT
    
    log start
    curl -fsS -m 10 --retry 3 -o /dev/null "$PING/start" || true
    
    /usr/local/bin/nightly-report
    

    Replace /usr/local/bin/nightly-report with your own command, and make the script executable with chmod +x. The exit status cron sees is still the job’s: the finish trap reports it without changing it.

  3. Point cron at the script

    crontab -e
    17 2 * * * /usr/local/bin/run-nightly-report.sh
    
  4. Run it once

    Run the script by hand, as the same user whose crontab runs it. The check should show a start ping and a success ping. Then make the job fail on purpose (replace its command with false) and run it again: you should get a failure email within a minute, showing exit code 1.

    bash
    /usr/local/bin/run-nightly-report.sh
    

Notes

  • || true on each ping means a network problem reaching Lateping never fails your job.
  • Ping last. The success ping must come after the work. A ping at the start only proves the job started.
  • The code must be a plain number from 0 to 255. A shell exit status always is. Anything else (a sign, a leading zero, a decimal point) returns 404.
  • Unusual run times. Once a check has 10 timed runs (runs that sent /start), Pro and above also get an alert when a run is much shorter or longer than the last 20. It is on by default and can be switched off in the check’s Settings tab. It does not catch a started job that never finishes.
  • % is special in a crontab line. cron turns it into a newline. Escape it as \%, or move the command into a script, where % is an ordinary character.
  • Timezones. cron uses the server’s timezone. If the check’s timezone differs, it expects the ping at the wrong hour.
  • One check per job. A shared check can’t tell you which job stopped.
  • Alerts go by email. On Pro and above they can also go to Slack, Discord or a webhook.

Next steps

On this page