Monitor Kubernetes CronJobs
A CronJob that stops creating Jobs, or a Job that fails and is cleaned up, leaves almost nothing behind. Ping a URL from the container when the work finishes, and we email you when a run is late or fails.
You need: a Lateping account, a cluster with a CronJob, kubectl, about 10 minutes.
What can go wrong
Kubernetes creates a Job for each scheduled time and moves on. When a run fails or never starts, the evidence is easy to miss, and some of it is deleted for you.
Set it up
Create a check
Create a check in Lateping with the CronJob’s schedule, and the same timezone as its
timeZonefield (orUTCif it has none). Set the grace period longer than a normal run, plus a few minutes for pulling the image and scheduling the pod.Store the ping URL as a Secret
The ping URL is a secret: anyone with it can ping your check.
kubectl create secret generic lateping --from-literal=nightly-report=https://lateping.com/p/<check-id>Wrap the command with pings
apiVersion: batch/v1 kind: CronJob metadata: name: nightly-report spec: schedule: "17 2 * * *" timeZone: "Etc/UTC" concurrencyPolicy: Forbid failedJobsHistoryLimit: 3 jobTemplate: spec: backoffLimit: 2 activeDeadlineSeconds: 3600 template: spec: restartPolicy: Never containers: - name: report image: registry.example.com/report:1.4.0 env: - name: LATEPING_URL valueFrom: secretKeyRef: name: lateping key: nightly-report command: ["/bin/sh", "-c"] args: - | curl -fsS -m 10 --retry 3 -o /dev/null "$LATEPING_URL/start" || true if /app/report; then curl -fsS -m 10 --retry 3 -o /dev/null "$LATEPING_URL" || true else code=$? curl -fsS -m 10 --retry 3 -o /dev/null "$LATEPING_URL/$code" || true exit "$code" fi- The success ping is sent only when
/app/reportexits 0. - If it fails, the ping carries its exit code (
/$code). Lateping records any non-zero code as a failure, alerts straight away and shows the code in the alert and the Pings list. The pod then exits with the same code, so Kubernetes retries it (up tobackoffLimit). A retry that succeeds sends the success ping, and you get a “recovered” email. - If no Job is created, the pod never starts, or
activeDeadlineSecondsstops the run, no success ping arrives and the check alerts after the grace period. || truestops a network problem reaching Lateping from failing a job that worked.
- The success ping is sent only when
Test it
Create a Job from the CronJob straight away, then watch its logs. The check should show a start ping and a success ping.
kubectl create job --from=cronjob/nightly-report nightly-report-test kubectl logs -f job/nightly-report-test
When Kubernetes skips a run
Two CronJob fields decide whether a scheduled run happens at all. Kubernetes’s CronJob documentation describes both.
concurrencyPolicyisAllowby default: Jobs may overlap.Forbidskips a new run if the previous Job hasn’t finished.Replacereplaces the running Job with a new one. The policy applies only to Jobs from the same CronJob.startingDeadlineSecondsis how late a Job may be created. If the controller finds the Job is later than that, it skips that run. Future runs are still scheduled. With no value, there is no deadline.
A skipped run starts no pod, so it sends no ping. Here is what that looks like in Lateping:
Forbid: the skipped run sends nothing. The long run that caused the skip sends its own pings when it finishes. If that happens before the next due time plus the grace period, you get no alert. If it doesn’t, the check goes late, which is correct: the job did not finish on schedule.Replace: the replaced Job is stopped before it can send a success or fail ping. The new Job sends its own/startand success ping as usual.startingDeadlineSeconds: a run that missed its deadline sends nothing, and the check alerts after the grace period.
To see which policy a CronJob uses, and when it last created a Job:
kubectl get cronjob nightly-report -o jsonpath='{.spec.concurrencyPolicy}{"\n"}{.spec.startingDeadlineSeconds}{"\n"}{.status.lastScheduleTime}{"\n"}'
An empty second line means no deadline is set. If lastScheduleTime is older than your schedule says it should be, runs are being skipped or the CronJob is suspended.
To make a skip visible the same day, keep the grace period shorter than the gap you can tolerate and leave Forbid on. A skip then ends in a late alert instead of a silent gap. If a run must never be dropped for starting late, set startingDeadlineSeconds generously:
spec:
schedule: "17 2 * * *"
concurrencyPolicy: Forbid
startingDeadlineSeconds: 3600
Notes
- The image needs a shell and
curl. If it has neither (a distroless image, say), addcurl, or send the pings from your program’s own code at the end of the run: any HTTP client works. - The exit code must be a plain number from 0 to 255. A code over 255, or one with a sign, a leading zero or a decimal point, returns
404. - Because the pod sends
/start, runs are timed. Once a check has 10 timed runs, Pro and above also get an alert when a run is unusually short or long compared with the last 20. It is on by default and can be switched off in the check’s Settings tab. It does not catch a started job that never finishes, so keepactiveDeadlineSecondsset. - With
Replace, a run that was replaced before it finished is not timed, because it never sends a success ping. - If a NetworkPolicy limits egress, allow HTTPS to
lateping.com. - Kubernetes can occasionally create two Jobs for one scheduled time, or none. Make the job safe to run twice. The check catches the “none”.
- One check per CronJob. Several CronJobs can share the
latepingSecret with one key each. - Alerts go by email. On Pro and above they can also go to Slack, Discord or a webhook.
Create a free check at app.lateping.com.