Get alerted when a backup job stops running

A backup that stopped running looks exactly like one that works, until the day you need to restore. Ping a URL at the end of the backup script, only after every step succeeded, and we email you when one doesn't.

You need: a Lateping account, a backup script run by cron or a timer, about 10 minutes.

What can go wrong

Backups fail quietly in more ways than most jobs, and you usually find out when you try to restore.

Set it up

  1. Create a check

    Create a check in Lateping with the backup’s schedule and timezone. Set the grace period to the backup’s normal run time plus a margin: a backup that runs past it alerts as late, which is how you hear that it’s getting slow.

  2. Make every step fail loudly, then ping

    This script stops at the first failed command, including one in the middle of a pipeline, and refuses an empty dump. Only when the dump and the upload have both worked does it send the success ping. Any failure pings the failed command’s exit code, which alerts straight away and shows the code in the alert.

    /usr/local/bin/backup-db.sh
    #!/usr/bin/env bash
    # Nightly database backup: dump, check it isn't empty, upload, then tell Lateping.
    set -euo pipefail          # stop on any error, unset variable, or failed pipe stage
    PING=https://lateping.com/p/<check-id>
    trap 'curl -fsS -m 10 --retry 3 -o /dev/null "$PING/$?" || true' ERR
    
    curl -fsS -m 10 --retry 3 -o /dev/null "$PING/start" || true
    
    dump=$(mktemp --suffix=.sql.gz)
    trap 'rm -f "$dump"' EXIT
    pg_dump --no-owner "$DATABASE_URL" | gzip > "$dump"
    
    # A dump under 1 KB means pg_dump wrote nothing useful.
    size=$(stat -c %s "$dump")
    if [ "$size" -lt 1024 ]; then
      echo "dump is only $size bytes" >&2
      false
    fi
    
    restic backup --tag nightly --stdin --stdin-filename db.sql.gz < "$dump"
    
    curl -fsS -m 10 --retry 3 -o /dev/null "$PING" || true
    
    crontab -e
    17 2 * * * /usr/local/bin/backup-db.sh >> /var/log/backup-db.log 2>&1
    
    • set -o pipefail makes pg_dump | gzip fail when pg_dump fails, not just when gzip does.
    • false after the size check triggers the ERR trap like any other failed command, so an empty dump alerts.
    • restic exits non-zero if the snapshot wasn’t saved (and with 3 if some data couldn’t be read), so a failed upload alerts too.
    • In the ERR trap, $? is the exit status of the command that failed. The ping goes to /<exit code>, so a restic exit 3 shows as 3 in the check’s Pings list and in the alert. Any code except 0 records a failure.
    • || true on each ping means a network problem reaching Lateping never fails the backup itself.
  3. Run it once

    Run the script by hand. The check should show a start ping and a success ping. Then break something on purpose (a wrong DATABASE_URL, say) and run it again: you should get a failure email within a minute.

    bash
    sudo -u backup /usr/local/bin/backup-db.sh
    

Notes

  • Restore, too. A backup you’ve never restored is a hope. Schedule a restore drill (download the latest dump, load it into a scratch database, count the rows) and give it its own check.
  • Watch the run time. With /start and the success ping, Pro shows each backup’s duration, so a backup that’s creeping from 5 minutes to 50 is visible before it collides with the next run. Once a check has 10 timed runs, Pro and above also get an alert when a run is much shorter or longer than the last 20. A dump that suddenly finishes in seconds is often an empty one. It is on by default and can be switched off in the check’s Settings tab. It does not catch a backup that starts and never finishes.
  • Off-site matters more than the script. Alert on the upload step, as above, not only on the dump: a dump that never leaves the server dies with it.
  • The same pattern works for any backup tool: borg, rclone, aws s3 cp, mysqldump. Fail on any error, check the output looks real, and ping last.
  • Alerts go by email. On Pro and above they can also go to Slack, Discord or a webhook.

Next steps

On this page