Monitor Kubernetes CronJobs

A CronJob that stops creating Jobs, or a Job that fails and is cleaned up, leaves almost nothing behind. Ping a URL from the container when the work finishes, and we email you when a run is late or fails.

You need: a Lateping account, a cluster with a CronJob, kubectl, about 10 minutes.

What can go wrong

Kubernetes creates a Job for each scheduled time and moves on. When a run fails or never starts, the evidence is easy to miss, and some of it is deleted for you.

Set it up

  1. Create a check

    Create a check in Lateping with the CronJob’s schedule, and the same timezone as its timeZone field (or UTC if it has none). Set the grace period longer than a normal run, plus a few minutes for pulling the image and scheduling the pod.

  2. Store the ping URL as a Secret

    The ping URL is a secret: anyone with it can ping your check.

    bash
    kubectl create secret generic lateping --from-literal=nightly-report=https://lateping.com/p/<check-id>
    
  3. Wrap the command with pings

    nightly-report.yaml
    apiVersion: batch/v1
    kind: CronJob
    metadata:
      name: nightly-report
    spec:
      schedule: "17 2 * * *"
      timeZone: "Etc/UTC"
      concurrencyPolicy: Forbid
      failedJobsHistoryLimit: 3
      jobTemplate:
        spec:
          backoffLimit: 2
          activeDeadlineSeconds: 3600
          template:
            spec:
              restartPolicy: Never
              containers:
                - name: report
                  image: registry.example.com/report:1.4.0
                  env:
                    - name: LATEPING_URL
                      valueFrom:
                        secretKeyRef:
                          name: lateping
                          key: nightly-report
                  command: ["/bin/sh", "-c"]
                  args:
                    - |
                      curl -fsS -m 10 --retry 3 -o /dev/null "$LATEPING_URL/start" || true
                      if /app/report; then
                        curl -fsS -m 10 --retry 3 -o /dev/null "$LATEPING_URL" || true
                      else
                        code=$?
                        curl -fsS -m 10 --retry 3 -o /dev/null "$LATEPING_URL/$code" || true
                        exit "$code"
                      fi
    
    • The success ping is sent only when /app/report exits 0.
    • If it fails, the ping carries its exit code (/$code). Lateping records any non-zero code as a failure, alerts straight away and shows the code in the alert and the Pings list. The pod then exits with the same code, so Kubernetes retries it (up to backoffLimit). A retry that succeeds sends the success ping, and you get a “recovered” email.
    • If no Job is created, the pod never starts, or activeDeadlineSeconds stops the run, no success ping arrives and the check alerts after the grace period.
    • || true stops a network problem reaching Lateping from failing a job that worked.
  4. Test it

    Create a Job from the CronJob straight away, then watch its logs. The check should show a start ping and a success ping.

    bash
    kubectl create job --from=cronjob/nightly-report nightly-report-test
    kubectl logs -f job/nightly-report-test
    

When Kubernetes skips a run

Two CronJob fields decide whether a scheduled run happens at all. Kubernetes’s CronJob documentation describes both.

  • concurrencyPolicy is Allow by default: Jobs may overlap. Forbid skips a new run if the previous Job hasn’t finished. Replace replaces the running Job with a new one. The policy applies only to Jobs from the same CronJob.
  • startingDeadlineSeconds is how late a Job may be created. If the controller finds the Job is later than that, it skips that run. Future runs are still scheduled. With no value, there is no deadline.

A skipped run starts no pod, so it sends no ping. Here is what that looks like in Lateping:

  • Forbid: the skipped run sends nothing. The long run that caused the skip sends its own pings when it finishes. If that happens before the next due time plus the grace period, you get no alert. If it doesn’t, the check goes late, which is correct: the job did not finish on schedule.
  • Replace: the replaced Job is stopped before it can send a success or fail ping. The new Job sends its own /start and success ping as usual.
  • startingDeadlineSeconds: a run that missed its deadline sends nothing, and the check alerts after the grace period.

To see which policy a CronJob uses, and when it last created a Job:

bash
kubectl get cronjob nightly-report -o jsonpath='{.spec.concurrencyPolicy}{"\n"}{.spec.startingDeadlineSeconds}{"\n"}{.status.lastScheduleTime}{"\n"}'

An empty second line means no deadline is set. If lastScheduleTime is older than your schedule says it should be, runs are being skipped or the CronJob is suspended.

To make a skip visible the same day, keep the grace period shorter than the gap you can tolerate and leave Forbid on. A skip then ends in a late alert instead of a silent gap. If a run must never be dropped for starting late, set startingDeadlineSeconds generously:

nightly-report.yaml (excerpt)
spec:
  schedule: "17 2 * * *"
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 3600

Notes

  • The image needs a shell and curl. If it has neither (a distroless image, say), add curl, or send the pings from your program’s own code at the end of the run: any HTTP client works.
  • The exit code must be a plain number from 0 to 255. A code over 255, or one with a sign, a leading zero or a decimal point, returns 404.
  • Because the pod sends /start, runs are timed. Once a check has 10 timed runs, Pro and above also get an alert when a run is unusually short or long compared with the last 20. It is on by default and can be switched off in the check’s Settings tab. It does not catch a started job that never finishes, so keep activeDeadlineSeconds set.
  • With Replace, a run that was replaced before it finished is not timed, because it never sends a success ping.
  • If a NetworkPolicy limits egress, allow HTTPS to lateping.com.
  • Kubernetes can occasionally create two Jobs for one scheduled time, or none. Make the job safe to run twice. The check catches the “none”.
  • One check per CronJob. Several CronJobs can share the lateping Secret with one key each.
  • Alerts go by email. On Pro and above they can also go to Slack, Discord or a webhook.

Create a free check at app.lateping.com.

Next steps

On this page