Monitor systemd timers

A systemd timer that was never enabled, a service that fails into the "failed" state, a machine that was off at 02:00. None of it reaches you unless you go looking. Ping a URL from the service and we email you instead.

You need: a Lateping account, root on a Linux server with systemd, about 10 minutes.

What can go wrong

A systemd timer starts a service on a schedule. When the service fails, or never starts, systemd records it and does nothing else.

Set it up

  1. Create a check

    Create a check in Lateping with the same schedule as the timer’s OnCalendar=, written as a cron expression (*-*-* 02:17:00 is 17 2 * * *), and the same timezone. Allow a few minutes of grace on top of the job’s normal run time: systemd starts a timer within AccuracySec= (1 minute by default) of its time, later still with RandomizedDelaySec=.

  2. Add a ping script

    One script sends every ping. Run it with start before the job, and without it afterwards: then it reads $SERVICE_RESULT, which systemd sets for ExecStopPost= commands. If the run failed it pings the job’s exit code when there is one ($EXIT_STATUS), and /fail otherwise (a signal or a timeout, say).

    /usr/local/bin/lateping-ping
    #!/bin/sh
    # Pings Lateping from a systemd service. $1 is the check's ping URL.
    #   lateping-ping URL start   before the job (ExecStartPre=)
    #   lateping-ping URL         after the job (ExecStopPost=): the exit code or /fail unless the run succeeded
    url="$1"
    if [ "$2" = start ]; then
      url="$url/start"
    elif [ "${SERVICE_RESULT:-success}" != success ]; then
      case "${EXIT_CODE:-}:${EXIT_STATUS:-}" in
        exited:[1-9]|exited:[1-9][0-9]|exited:1[0-9][0-9]|exited:2[0-4][0-9]|exited:25[0-5])
          url="$url/$EXIT_STATUS" ;;
        *) url="$url/fail" ;;
      esac
    fi
    curl -fsS -m 10 --retry 3 -o /dev/null "$url" || true
    
    bash
    sudo chmod 755 /usr/local/bin/lateping-ping
    

    || true stops a network problem reaching Lateping from failing a job that worked. A non-zero exit code shows in the check’s Pings list and in the fail alert.

  3. Call it from the service

    /etc/systemd/system/nightly-backup.service
    [Unit]
    Description=Nightly backup
    
    [Service]
    Type=oneshot
    ExecStartPre=-/usr/local/bin/lateping-ping https://lateping.com/p/<check-id> start
    ExecStart=/usr/local/bin/backup.sh
    ExecStopPost=/usr/local/bin/lateping-ping https://lateping.com/p/<check-id>
    
    /etc/systemd/system/nightly-backup.timer
    [Unit]
    Description=Run the nightly backup
    
    [Timer]
    OnCalendar=*-*-* 02:17:00
    Persistent=true
    
    [Install]
    WantedBy=timers.target
    
    • ExecStopPost= runs however the service ends: success, an error exit, a crash or a timeout. A failed run alerts straight away.
    • The - before ExecStartPre= means a failed /start ping never stops the job.
    • If the timer doesn’t fire, or the machine is off, no ping arrives and the check alerts after the grace period.
    • Persistent=true runs a missed run when the machine comes back. If that’s later than the grace period, you get a late alert first, then a recovery.
  4. Enable the timer and test it

    Enable the timer, not the service. Then start the service by hand once: the check should show a start ping and a success ping.

    bash
    sudo systemctl daemon-reload
    sudo systemctl enable --now nightly-backup.timer
    sudo systemctl start nightly-backup.service
    systemctl list-timers nightly-backup.timer
    

Notes

  • Check what a schedule means before you copy it into Lateping: systemd-analyze calendar '*-*-* 02:17:00' prints the next run in your timezone. Add a timezone to pin it, for example OnCalendar=*-*-* 02:17:00 UTC.
  • ExecStopPost= sees $SERVICE_RESULT values such as exit-code, signal, timeout and oom-kill. The script treats anything other than success as a failure. $EXIT_CODE and $EXIT_STATUS say how the process ended: $EXIT_STATUS is a number only when $EXIT_CODE is exited.
  • Lateping accepts exit codes 0 to 255 as plain numbers. Anything else returns 404, which is why the script sends /fail for the other cases.
  • Because the script sends /start, runs are timed. Once a check has 10 timed runs, Pro and above also get an alert when a run is unusually short or long compared with the last 20. It is on by default and can be switched off in the check’s Settings tab. It does not catch a started job that never finishes.
  • One check per timer. Several services can share the script, each with its own ping URL.
  • Alerts go by email. On Pro and above they can also go to Slack, Discord or a webhook.

Next steps

On this page