Get alerted when a backup job stops running
A backup that stopped running looks exactly like one that works, until the day you need to restore. Ping a URL at the end of the backup script, only after every step succeeded, and we email you when one doesn't.
You need: a Lateping account, a backup script run by cron or a timer, about 10 minutes.
What can go wrong
Backups fail quietly in more ways than most jobs, and you usually find out when you try to restore.
Set it up
Create a check
Create a check in Lateping with the backup’s schedule and timezone. Set the grace period to the backup’s normal run time plus a margin: a backup that runs past it alerts as late, which is how you hear that it’s getting slow.
Make every step fail loudly, then ping
This script stops at the first failed command, including one in the middle of a pipeline, and refuses an empty dump. Only when the dump and the upload have both worked does it send the success ping. Any failure pings the failed command’s exit code, which alerts straight away and shows the code in the alert.
#!/usr/bin/env bash # Nightly database backup: dump, check it isn't empty, upload, then tell Lateping. set -euo pipefail # stop on any error, unset variable, or failed pipe stage PING=https://lateping.com/p/<check-id> trap 'curl -fsS -m 10 --retry 3 -o /dev/null "$PING/$?" || true' ERR curl -fsS -m 10 --retry 3 -o /dev/null "$PING/start" || true dump=$(mktemp --suffix=.sql.gz) trap 'rm -f "$dump"' EXIT pg_dump --no-owner "$DATABASE_URL" | gzip > "$dump" # A dump under 1 KB means pg_dump wrote nothing useful. size=$(stat -c %s "$dump") if [ "$size" -lt 1024 ]; then echo "dump is only $size bytes" >&2 false fi restic backup --tag nightly --stdin --stdin-filename db.sql.gz < "$dump" curl -fsS -m 10 --retry 3 -o /dev/null "$PING" || true17 2 * * * /usr/local/bin/backup-db.sh >> /var/log/backup-db.log 2>&1set -o pipefailmakespg_dump | gzipfail whenpg_dumpfails, not just when gzip does.falseafter the size check triggers theERRtrap like any other failed command, so an empty dump alerts.resticexits non-zero if the snapshot wasn’t saved (and with 3 if some data couldn’t be read), so a failed upload alerts too.- In the
ERRtrap,$?is the exit status of the command that failed. The ping goes to/<exit code>, so a restic exit 3 shows as 3 in the check’s Pings list and in the alert. Any code except 0 records a failure. || trueon each ping means a network problem reaching Lateping never fails the backup itself.
Run it once
Run the script by hand. The check should show a start ping and a success ping. Then break something on purpose (a wrong
DATABASE_URL, say) and run it again: you should get a failure email within a minute.sudo -u backup /usr/local/bin/backup-db.sh
Notes
- Restore, too. A backup you’ve never restored is a hope. Schedule a restore drill (download the latest dump, load it into a scratch database, count the rows) and give it its own check.
- Watch the run time. With
/startand the success ping, Pro shows each backup’s duration, so a backup that’s creeping from 5 minutes to 50 is visible before it collides with the next run. Once a check has 10 timed runs, Pro and above also get an alert when a run is much shorter or longer than the last 20. A dump that suddenly finishes in seconds is often an empty one. It is on by default and can be switched off in the check’s Settings tab. It does not catch a backup that starts and never finishes. - Off-site matters more than the script. Alert on the upload step, as above, not only on the dump: a dump that never leaves the server dies with it.
- The same pattern works for any backup tool:
borg,rclone,aws s3 cp,mysqldump. Fail on any error, check the output looks real, and ping last. - Alerts go by email. On Pro and above they can also go to Slack, Discord or a webhook.