Push notifications from a cron job

Cron doesn’t tell you when a job fails. Its only built-in alert is MAILTO, which emails you the output of every job that prints anything, and those emails are easy to stop reading. What you usually want is the opposite: nothing while the job works, and a push on your phone the first time it doesn’t.

This guide does that with the honk-me command. It takes five minutes and works with any job: a database backup, a sync script, an artisan command.

Before you start

• A Honk account and a project. Honk is invite-only for now; request access if you don’t have one.
• An ingestion key for the project: in the web app, open the project, then Keys, and create one. It looks like honk_… and is shown once.
• The honk-me CLI on the machine that runs cron. It’s a single binary: go install github.com/honk-me/honk-go/cmd/honk-me@latest, or download one from the releases page. No CLI? The cURL version is at the end.

1. Give cron the key

Cron runs jobs with an almost empty environment, so put the two variables at the top of the crontab. crontab -e edits a file only your user can read:

# crontab -e  (the file is readable by your user only)
HONK_URL=https://honk-me.app
HONK_KEY=honk_…

The CLI has no flag for the key on purpose, so it never shows up in ps or in your shell history.

2. Alert only when the job fails

Add || honk-me problem … after the job. The shell runs it only when the job exits with an error, and $? still holds the job’s exit code:

0 3 * * * pg_dump app > /backup/app.sql || honk-me problem --group-key db/backup --source "$(hostname)" --title "Backup failed" --message "pg_dump exited with $?"

The --group-key is what keeps your phone calm. Every failure of the backup lands in the same group, db/backup. The first one pushes right away; if the job keeps failing within the same episode, Honk sends a calm update at most every 5 minutes instead of a new alarm, and the inbox shows one row with a count.

A detail that bites everyone once: in a crontab, % stands for a newline. If your command contains a date +%F, write it as date +\%F.

3. Hear when it works again

A problem stays open until something closes it. Honk closes it when the same group key reports a recovery, and pushes that too, so you know you can stop worrying.

Don’t send a recovery after every successful run, though. A recovery with no open problem still opens a “recovered” episode and notifies, so you’d get a push every night. Send it only on the first success after a failure. This wrapper remembers the last result in a small state file:

#!/usr/bin/env bash
# honk-cron: run a command; honk when it fails, and once when it works again.
# usage: honk-cron <group-key> <command> [args…]
set -uo pipefail
key=$1; shift
state="${XDG_STATE_HOME:-$HOME/.local/state}/honk-cron/${key//\//_}.failed"
mkdir -p "$(dirname "$state")"
out=$(mktemp); trap 'rm -f "$out"' EXIT

"$@" >"$out" 2>&1
code=$?

if [ "$code" -ne 0 ]; then
  # the end of the output is usually the useful part; a message holds up to 8192 bytes
  tail -c 6000 "$out" | honk-me problem --group-key "$key" --source "$(hostname)" \
    --title "$key failed (exit $code)" --message - || true
  touch "$state"
elif [ -e "$state" ]; then
  honk-me recovery --group-key "$key" --source "$(hostname)" \
    --title "$key works again" --message "Exit code 0 after a failure." || true
  rm -f "$state"
fi
exit "$code"

Save it as /usr/local/bin/honk-cron, make it executable with chmod +x, and put it in front of any job, with the group key first:

0 3 * * *   /usr/local/bin/honk-cron db/backup pg_dump -f /backup/app.sql app
*/15 * * * * /usr/local/bin/honk-cron sync/invoices php /srv/app/artisan invoices:sync

It sends the last 6,000 bytes of the job’s output as the message, so the push and the inbox show the actual error, not just “failed.” It always exits with the job’s own exit code, and || true makes sure a problem sending the notification can never turn a good run into a failed one.

Without the CLI: cURL

curl is on almost every server. The same alert as one crontab line, with the JSON written out:

0 3 * * * pg_dump app > /backup/app.sql || curl -sS --max-time 10 --retry 3 "$HONK_URL/v1/messages" -H "Authorization: Bearer $HONK_KEY" -H "Content-Type: application/json" -d "{\"title\":\"Backup failed\",\"message\":\"pg_dump exited with $?\",\"severity\":\"long\",\"event_type\":\"problem\",\"group_key\":\"db/backup\"}"

The CLI adds what this line doesn’t: retries that reuse one idempotency key, a total deadline, and exit codes you can act on.

What the exit codes mean

honk-me exits 0 when Honk accepted the event (or already had it), 4 for a wrong or revoked key, 5 when a quota is used up, and 7 when the network or the server failed after every retry. The CLI page lists them all. In cron, || true after the command is usually what you want.

What Honk adds to cron

MAILTO sends one email per noisy run. Honk sends one push per new problem, folds the repeats into a group with their output, tells you when the job recovers, and keeps it all in one inbox next to your other jobs and servers. Quiet hours and the digest decide how loud your nights get.