-
Deployment
Set alerting thresholds you'll actually respond to
Tune alerts so real incidents wake you up and noise doesn't train you to ignore them.
Free Prompt
Tune alerting thresholds for my app so I actually respond when they fire. Right now they're either untuned defaults (too noisy) or nonexistent (silent outages).
Set up alerts on these signals:
1. Availability: uptime monitor on the critical paths (already covered in the uptime monitoring prompt). Page me on two consecutive failures, not one.
2. Error rate: alert when server 5xx rate exceeds a threshold sustained for a few minutes (e.g., >1% for 5 minutes). Rate, not raw count; raw counts scale with traffic and produce false positives during promotion spikes.
3. Latency: alert when p95 latency exceeds a defined ceiling for a few minutes. Track p95 or p99, not average; average hides tail latency users notice.
4. Queue age: alert when the oldest job in critical queues exceeds a threshold. Not depth (deep-but-draining is fine), age (jobs older than N minutes = something is broken).
5. Database health: connection pool exhaustion, replication lag if you have replicas, disk usage >80%, and CPU pinned high for a sustained window.
6. Business-level signals: sign-ups per hour dropping to zero, checkout success rate crashing, email delivery failing. These catch outages that don't produce 500s but still break the app for users.
7. Certificate and cron: cert expiry within 14 days, and 'expected scheduled job didn't run' as a hard alert.
Routing: page-severity alerts (SEV1 territory) go to a pager. Warnings go to a channel. Info-level metrics go to a dashboard, not a notification. Do not send everything to the same channel or you'll mute all of them.
Don't alert on things you won't act on at 2am; those belong on a dashboard, not a pager. The deliverable is the alert rules configured, one intentional test-fire to confirm routing works, and a documented alert taxonomy (what fires where, at what threshold, and what the response is).
What This Does / How This Helps
This defines alerts around real signals (error rate, tail latency, queue age, business-level indicators) with thresholds you'll respond to, and routes them by severity so you don't mute the whole system. Bad alerting is worse than no alerting because it trains you to ignore the pager. If everything is urgent, nothing is. Tuning to real signals and using severity routing means when your phone actually goes off, you know it's worth looking at, and the important stuff isn't drowning in noise.
Want to skip doing this by hand?
Fortivibe audits your app for all of the areas these prompts cover (and more).