engineering writeup
Monitoring a cluster, including the monitoring
The problem
Dozens of services across several nodes, and the usual failure mode: something breaks quietly and I find out days later because a page I rarely visit is blank. I wanted metrics, logs, and alerts that reach me, and I wanted to know when the pipeline itself was lying.
The shape
node exporters ─┐
custom checks ─┼→ Prometheus → Alertmanager → push + email
log shippers ─┘ ↓
Grafana
The hard parts
The monitoring ran on the thing it monitored. The metrics stack lived on one VM. When that VM locked up hard, the guest kept answering cached HTTP with a 200 while its services were already dead — and the alerting that should have caught it was on the same box, equally frozen. It did not fire, because it could not. The lesson is not "add another check"; it is that a monitoring system inside the failure domain it watches is decorative for exactly the failures that matter most. The watcher has to sit outside.
A 200 is not proof of life. That same failure taught me that HTTP liveness checks are weak evidence. A cached response, a static error page, or a proxy answering on a dead upstream all look identical to a naive check. A useful probe asserts something only a working system can produce — a fresh timestamp, a real query result — not merely that something answered.
One malformed metric discards the whole file. The textfile collector rejects an entire file if a single metric repeats its HELP or TYPE line, so one careless append silently drops every unrelated metric alongside it. It fails quietly: the file is present, the scrape "succeeds", the metrics are simply gone. Watching the collector's own error metric is the only way to see it.
Alerts nobody reads are worse than no alerts. The first pass alerted on everything, which trained me to dismiss the notification channel within a week. Cutting it to conditions that genuinely require a human — and routing those to a channel I actually read — made the remaining alerts mean something.
What I'd do differently
The obvious one: move alerting outside the cluster entirely, or at minimum add an external dead-man's-switch that pages when the heartbeat stops rather than when a threshold trips. Absence of signal is the failure mode this design handles worst.
Log retention was chosen by guessing rather than by measuring what I actually query. I have never once needed logs from four weeks ago, and I keep paying to store them.
Dashboards were built per-service as each was added, so there is no single view answering "is anything wrong right now" without knowing which dashboard to open — which is precisely the question worth answering fastest.