Monitoring an early product: four signals and some quiet
The minimum set of uptime, errors, latency and background queue depth, plus rules that make alerts wake people only when it matters.
Monitoring tends to arrive in one of two forms. Either there’s none, and a customer reports the outage in chat. Or it was set up with enthusiasm, and a month later everyone has muted the alerts channel because it gets a hundred messages a day and none of them matter. The second is worse than it looks, because it feels like things are under control.
An early product needs four signals and a few rules that stop them turning into noise.
Uptime from the outside
An external check every minute, from a different network: does the site load and does the API respond. What you check matters. A home page served from a CDN keeps loading while the database is down. Add a /health endpoint that runs a trivial database query and returns an error if it fails.
Checking from several locations cuts down on false alarms. A network blip at the monitoring provider shouldn’t wake anyone up.
Errors
An error tracker that groups identical exceptions and shows how many people were affected. Settings worth having from day one:
- every error is tagged with the release version, so you can see which deploy introduced it;
- frontend errors are collected the same way as backend ones, since most of the breakage people actually see happens in the browser;
- you get alerted on a new error group or a sharp rise in an old one, not on every single event.
Latency
Average response time is close to useless. Nine fast requests and one that takes ten seconds produce a respectable average and one very unhappy person. Watch the 95th percentile instead: the time within which 95 out of 100 requests complete.
You don’t need to watch every endpoint. Pick three to five that the main flow depends on: login, the main page, saving, payment. A slow admin report that runs once a day isn’t worth an alert.
Background queue
If the product has background jobs, such as emails, file generation, webhook processing or AI calls, this is where the quietest failures live. The worker crashed, the site is fine, and confirmation emails haven’t gone out for three hours.
Two numbers worth seeing:
- how many jobs are waiting;
- how long the oldest one has been waiting.
The second is more useful. A long queue that drains quickly is normal; a single job stuck for an hour is not. Scheduled jobs need a reverse check: the job reports “I ran”, and if that report doesn’t arrive, you get an alert.
Not drowning in alerts
An alert is a request for a human to do something. If there’s nothing to do, it’s a log entry, not an alert. A few rules:
- Alert on symptoms users feel, not on causes. CPU at 85% isn’t a problem if requests are still fast.
- Thresholds with a duration: “errors above normal for five minutes straight”, not “one error”.
- Two levels. Urgent goes to a phone and needs a response now. Everything else lands in a digest someone reads in the morning.
- Every urgent alert comes with a short note: what to check first, and where.
A monthly cleanup
Once a month, go through the alert history and ask one question about each: did anyone do anything after it fired? If not, delete it, loosen it, or move it to the digest. If there was an outage that no alert caught, add one. After a few passes the urgent channel goes quiet, and that’s exactly why people start reading it again.
Who’s watching
On a team of three, “on-call rotation” sounds grand, but you do need to decide in advance who gets the alert on a Saturday. “Everyone” in practice means “no one”, because each person assumes someone else will look.
SUBXCAT can set up this minimum and spend the first couple of months pruning alerts with your team until they go quiet.