Monitor systemd timers
A systemd timer that was never enabled, a service that fails into the "failed" state, a machine that was off at 02:00. None of it reaches you unless you go looking. Ping a URL from the service and we email you instead.
You need: a Lateping account, root on a Linux server with systemd, about 10 minutes.
What can go wrong
A systemd timer starts a service on a schedule. When the service fails, or never starts, systemd records it and does nothing else.
Set it up
Create a check
Create a check in Lateping with the same schedule as the timer’s
OnCalendar=, written as a cron expression (*-*-* 02:17:00is17 2 * * *), and the same timezone. Allow a few minutes of grace on top of the job’s normal run time: systemd starts a timer withinAccuracySec=(1 minute by default) of its time, later still withRandomizedDelaySec=.Add a ping script
One script sends every ping. Run it with
startbefore the job, and without it afterwards: then it reads$SERVICE_RESULT, which systemd sets forExecStopPost=commands. If the run failed it pings the job’s exit code when there is one ($EXIT_STATUS), and/failotherwise (a signal or a timeout, say).#!/bin/sh # Pings Lateping from a systemd service. $1 is the check's ping URL. # lateping-ping URL start before the job (ExecStartPre=) # lateping-ping URL after the job (ExecStopPost=): the exit code or /fail unless the run succeeded url="$1" if [ "$2" = start ]; then url="$url/start" elif [ "${SERVICE_RESULT:-success}" != success ]; then case "${EXIT_CODE:-}:${EXIT_STATUS:-}" in exited:[1-9]|exited:[1-9][0-9]|exited:1[0-9][0-9]|exited:2[0-4][0-9]|exited:25[0-5]) url="$url/$EXIT_STATUS" ;; *) url="$url/fail" ;; esac fi curl -fsS -m 10 --retry 3 -o /dev/null "$url" || truesudo chmod 755 /usr/local/bin/lateping-ping|| truestops a network problem reaching Lateping from failing a job that worked. A non-zero exit code shows in the check’s Pings list and in the fail alert.Call it from the service
[Unit] Description=Nightly backup [Service] Type=oneshot ExecStartPre=-/usr/local/bin/lateping-ping https://lateping.com/p/<check-id> start ExecStart=/usr/local/bin/backup.sh ExecStopPost=/usr/local/bin/lateping-ping https://lateping.com/p/<check-id>[Unit] Description=Run the nightly backup [Timer] OnCalendar=*-*-* 02:17:00 Persistent=true [Install] WantedBy=timers.targetExecStopPost=runs however the service ends: success, an error exit, a crash or a timeout. A failed run alerts straight away.- The
-beforeExecStartPre=means a failed/startping never stops the job. - If the timer doesn’t fire, or the machine is off, no ping arrives and the check alerts after the grace period.
Persistent=trueruns a missed run when the machine comes back. If that’s later than the grace period, you get a late alert first, then a recovery.
Enable the timer and test it
Enable the timer, not the service. Then start the service by hand once: the check should show a start ping and a success ping.
sudo systemctl daemon-reload sudo systemctl enable --now nightly-backup.timer sudo systemctl start nightly-backup.service systemctl list-timers nightly-backup.timer
Notes
- Check what a schedule means before you copy it into Lateping:
systemd-analyze calendar '*-*-* 02:17:00'prints the next run in your timezone. Add a timezone to pin it, for exampleOnCalendar=*-*-* 02:17:00 UTC. ExecStopPost=sees$SERVICE_RESULTvalues such asexit-code,signal,timeoutandoom-kill. The script treats anything other thansuccessas a failure.$EXIT_CODEand$EXIT_STATUSsay how the process ended:$EXIT_STATUSis a number only when$EXIT_CODEisexited.- Lateping accepts exit codes 0 to 255 as plain numbers. Anything else returns
404, which is why the script sends/failfor the other cases. - Because the script sends
/start, runs are timed. Once a check has 10 timed runs, Pro and above also get an alert when a run is unusually short or long compared with the last 20. It is on by default and can be switched off in the check’s Settings tab. It does not catch a started job that never finishes. - One check per timer. Several services can share the script, each with its own ping URL.
- Alerts go by email. On Pro and above they can also go to Slack, Discord or a webhook.