The question: how do you decide what deserves an alert on a small home or nonprofit network, and what thresholds have held up for you over the years? I would like to compare methods, because most monitoring guides show you how to collect everything and say very little about what should wake a person up.
My short answer is that an alert should mean a human has to do something. If nobody acts on it, it is a graph, not an alert.
The test I apply to every alert
- Does this need action today, or can it wait for the weekly review?
- Is there a clear first step when it fires? If I would just shrug and wait, I delete the rule.
- Does it describe a symptom a user would notice, or a cause that may or may not matter?
Alerts on symptoms hold up better than alerts on causes. A CPU spike on a firewall that nobody notices is not worth a message. A DNS server that stopped answering is, because every user will notice within minutes. It is like a house: you want the smoke detector, not a notice every time the toaster gets warm.
What I alert on, in rough order of importance
- Reachability of the things everything else depends on: the gateway, DNS, and the WAN link. Check from inside and, if you can, from outside.
- A DNS query that actually resolves a name, not just a ping to the server. A box can answer ping and still serve nothing.
- Disk space at 85 percent warning and 95 percent critical. Full disks are the most boring and most common cause of a dead service.
- Backup jobs that did not finish. A missing success is the alert, not a reported failure.
- Certificate expiry at 21 days and again at 7 days.
- Interface errors on uplinks. A counter that climbs steadily on one port usually means a bad cable or a duplex mismatch. Check the cable first.
- A VPN tunnel with a stale handshake, if you run one.
What I leave as a graph
CPU, memory, and bandwidth utilization. They are useful when you are diagnosing something, and noisy when you page on them. If you do alert on them, require the condition to hold for a stretch, say 10 to 15 minutes, before anything fires. A single bad sample is not a trend.
Two habits that cut the noise
- Set the check to fail three times in a row before it notifies. One lost poll over a busy link is not an outage.
- Add a dependency. If the gateway is down, do not also send me forty messages about everything behind it.
Here is a sketch of the idea, in plain pseudo-config, since every tool spells it differently:
check: dns-resolve internal.example
interval: 60s
fail_after: 3 consecutive
notify: once, then every 4h until cleared
depends_on: gateway-reachable
I write each rule in the binder with the reason it exists and the date I last changed it. Thresholds are not universal. Mine suit a handful of devices and a few users, and I am not certain they scale to a larger site.
Two questions for you. What is the one alert that has actually caught a real problem for you before a user noticed? And which alert did you finally delete because you had trained yourself to ignore it?
