Prometheus, Alertmanager and Grafana, with every rule written against a series
this system actually publishes. Nothing here is aimed at a metric that might
exist.
One scrape target for the whole fleet. Agents publish nothing of their own —
the control plane gathers what they report and republishes it — which is worth
writing down before somebody goes looking for an exporter to install on each
node.
It scrapes metrics_listen, not the admin listener. That one is loopback because
it also serves pprof, and pprof can dump the heap where API keys sit in
plaintext. Metrics are the one thing on it something else genuinely needs, so
they get a port of their own rather than becoming a reason to open the other.
The README shows how to confirm the new port carries nothing else; it answers
404 to both pprof and the admin API.
Nothing is sent anywhere. Alertmanager groups and holds and has no receiver:
where an alert should go is a decision with credentials attached, and a
monitoring system that quietly starts mailing people is worse than one that is
visibly not wired up. critical already routes separately so it can be pointed
somewhere else without touching the rest.
The rules encode what the metrics mean rather than round numbers. A saturation
of 1 is the healthy limit measured by ctl bench, not the point of failure, so
the alert waits five minutes above it. A thirty-second p99 first token is the
threshold because this is CPU inference and would be absurd on a GPU. A model
that is deliberately stopped publishes no series at all, so "nothing serving"
cannot fire for one nobody asked for — which is what lets an on-demand
catalogue sit quiet.
Inhibitions, because a node that is down explains every replica on it and
reporting both buries the cause.
The dashboard is provisioned from a file rather than clicked together, so it
survives a reinstall and can be read in a diff.
Checked by firing one: cordoning a node moved llmgw_node_up to 0, the rule went
pending and then firing after its two minutes, the alert arrived at Alertmanager
with the node in its labels, and uncordoning resolved it. An alerting system
nobody has seen fire is an assumption.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The metrics existed and nothing could record them. They shared a listener
with pprof, the management API and the dashboard, and pprof can dump the
process heap where API keys sit in plaintext — so a fleet could be monitored
or keep its heap private, and not both. That is not a choice anyone should
have to make, and it is why every previous answer to "how do I scrape this"
was a tunnel.
metrics_listen serves metrics and health and nothing else, so it can be
reached from wherever Prometheus lives. Metrics stay on the admin listener
too: adding a door is not the same as moving one, and anything already
scraping keeps working.
While there: admin_listen defaulted to :9090, every interface, with a
comment beside it saying not to expose it. A default that contradicts its
own documentation is a footgun, so it is loopback now and someone who wants
otherwise says so.
The handler is nil until the registry is built, and says so rather than
serving an empty page. Ordering has already caught this exact area once —
a collector registered after AdminHandler published nothing at all, silently
— and an empty scrape target looks like a working one.
deploy/monitoring/prometheus.yml has a starting configuration and the alerts
worth having: a node that has stopped reporting, a model that should be
serving and is not, a card kept out of placement by a fault, and a fleet
holding models nobody is asking for.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
