Stop alert fatigue with fewer, better notifications

Alert fatigue doesn’t come from too many problems. It comes from too many notifications per problem. Redis times out for seven minutes, the billing API retries every ten seconds, and your phone buzzes 43 times. By the third buzz you get it. By the twentieth you’ve turned notifications off, and the next real problem arrives on a silent phone.

The fix is a handful of habits that work with any tool. Each section below covers one, then shows what it looks like in Honk.

1. Alert on the problem, not on each attempt

A retry loop isn’t 43 problems. It’s one problem that happened 43 times. Decide what “the same problem” means, give it a stable name, and send that name with every event.

In Honk, that name is the group_key. Events with the same project, environment, source, channel and group_key form one group, and only the first event of a new episode sends a push.

# Before: one push per failed attempt, 43 in seven minutes
curl … -d '{"title":"Redis timeout","message":"attempt 17 failed"}'

# After: one group, one episode, one push, then calm updates
honk-me problem --group-key billing/redis --source billing-api \
  --title "Redis connection failed" --message "Billing API could not connect to Redis (timeout after 5 s)"
# …and when it works again:
honk-me recovery --group-key billing/redis --source billing-api \
  --title "Redis is back" --message "Connected after 7 minutes"

2. Keep repeats quiet, and get loud only when it gets worse

While the problem lasts, you still want to know it’s ongoing, just not 43 times over. Honk sends at most one update per cooldown, 5 minutes by default, with the running count. If the level rises, say from a Loud honk to a Blast, that update is pushed right away, because worse is news.

3. Close the loop with a recovery

An alert that never ends keeps you guessing. When the thing works again, report a recovery for the same group_key: Honk pushes it, closes the episode, and the incident stops asking for your attention. Report recoveries only after a failure; a recovery after every healthy check is just more noise.

If nothing reports a recovery, the episode becomes inactive after a quiet period (24 hours by default). Inactive isn’t fixed: the next event opens a new episode and pushes again.

4. Send routine events to the digest

Not every event deserves a push. A finished import, a successful backup or a daily report is worth seeing, but not worth an interruption. Send them as Light honks or Beep-beeps at normal or low priority, and set your push threshold to High: everything below it is collected into a digest, every 15 minutes by default, and an empty digest is never sent.

A project rule does the same without touching code: for example, every event in the deployments category with severity success goes to the digest.

5. Keep the night for what can’t wait

Quiet hours hold pushes and send one summary when they end. Decide in advance what’s allowed to get through: in Honk, only events sent with urgent priority, from an ingestion key that’s allowed to send them, and only if your quiet hours allow urgent events. Give that permission to one key, for the one service that can wake you.

6. Choose the level honestly

The Honk scale has five levels, and they only help if each one means something. Use Blast for “something is down” and Long honk for “something failed.” If everything is a Blast, nothing is.

A checklist for group keys

• One key per thing you’d fix once. billing/redis, not billing/redis/attempt-17.
• No timestamps, request IDs or random values in the key, unless every event really is its own story, like a customer request (requests/4812).
• Put the place in the key when the same problem on two machines is two problems: disk/app-01/var.
• Use the same key for the problem and its recovery. That’s how Honk knows what recovered.
• Keep it short and readable. It shows up in the inbox and can be matched by rules.

How Honk fits

Honk was built around these habits. Grouping and episodes follow fixed rules on the server, cooldowns and digests are set per project, quiet hours and mutes are per person, and AI on the iPhone can summarize a group but can never hide or delay a notification. The how it works page has all the rules.