Ilker ManapandClaude Opus 5
b921fa0d6b
Prometheus, Alertmanager and Grafana, with every rule written against a series this system actually publishes. Nothing here is aimed at a metric that might exist. One scrape target for the whole fleet. Agents publish nothing of their own — the control plane gathers what they report and republishes it — which is worth writing down before somebody goes looking for an exporter to install on each node. It scrapes metrics_listen, not the admin listener. That one is loopback because it also serves pprof, and pprof can dump the heap where API keys sit in plaintext. Metrics are the one thing on it something else genuinely needs, so they get a port of their own rather than becoming a reason to open the other. The README shows how to confirm the new port carries nothing else; it answers 404 to both pprof and the admin API. Nothing is sent anywhere. Alertmanager groups and holds and has no receiver: where an alert should go is a decision with credentials attached, and a monitoring system that quietly starts mailing people is worse than one that is visibly not wired up. critical already routes separately so it can be pointed somewhere else without touching the rest. The rules encode what the metrics mean rather than round numbers. A saturation of 1 is the healthy limit measured by ctl bench, not the point of failure, so the alert waits five minutes above it. A thirty-second p99 first token is the threshold because this is CPU inference and would be absurd on a GPU. A model that is deliberately stopped publishes no series at all, so "nothing serving" cannot fire for one nobody asked for — which is what lets an on-demand catalogue sit quiet. Inhibitions, because a node that is down explains every replica on it and reporting both buries the cause. The dashboard is provisioned from a file rather than clicked together, so it survives a reinstall and can be read in a diff. Checked by firing one: cordoning a node moved llmgw_node_up to 0, the rule went pending and then firing after its two minutes, the alert arrived at Alertmanager with the node in its labels, and uncordoning resolved it. An alerting system nobody has seen fire is an assumption. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
4.8 KiB
Watching a cluster
Prometheus, Alertmanager and Grafana, configured for what llmgw actually publishes. Every alert here is written against a series this system emits; nothing is aimed at a metric that might exist.
What it is
One scrape target for the whole fleet.
Agents publish no metrics of their own. The control plane gathers what they report — node state, memory, disk, engine queue depth, KV occupancy — and republishes all of it, so a single endpoint describes every machine. That is worth knowing before going looking for an exporter on each node.
The port it scrapes
metrics_listen, not admin_listen.
The admin listener is bound to loopback for one reason: it also serves pprof, and pprof can dump the process heap, where API keys sit in plaintext. Metrics are the one thing on that listener something else genuinely needs to reach, so they get a port of their own instead of becoming an argument for opening the other one.
# /etc/llmgw/server.yaml
metrics_listen: ":9091"
Confirm it carries nothing else:
curl -s -o /dev/null -w '%{http_code}\n' http://<control-plane>:9091/debug/pprof/ # 404
curl -s -o /dev/null -w '%{http_code}\n' http://<control-plane>:9091/api/v1/nodes # 404
Installing
On Ubuntu 24.04 the first two come from the archive; Grafana needs its own repository.
apt-get install prometheus prometheus-alertmanager
install -m 0644 prometheus.yml /etc/prometheus/prometheus.yml
install -m 0644 llmgw-rules.yml /etc/prometheus/rules/llmgw.yml
install -m 0644 alertmanager.yml /etc/prometheus/alertmanager.yml
promtool check config /etc/prometheus/prometheus.yml
systemctl restart prometheus prometheus-alertmanager
install -m 0644 grafana/datasource.yaml /etc/grafana/provisioning/datasources/llmgw.yaml
install -m 0644 grafana/dashboards.yaml /etc/grafana/provisioning/dashboards/llmgw.yaml
install -m 0644 -o grafana -g grafana grafana/llmgw-overview.json \
/var/lib/grafana/dashboards/llmgw.json
systemctl restart grafana-server
The dashboard is provisioned from a file rather than clicked together, so it survives a reinstall and can be reviewed in a diff.
Where to run it
Not on the control plane. Monitoring that dies with the thing it monitors tells you nothing at the moment you need it, and the control plane is also the busiest machine in a small cluster.
Anywhere else on the management network will do. The mirror is a reasonable host: it has the disk, and serving weights is bursty rather than constant.
Nothing is sent anywhere
Alertmanager groups and holds alerts and has no receiver.
Where an alert should go — an address, a Slack workspace, a webhook — is a
decision with credentials attached, and a monitoring system that quietly starts
mailing people is worse than one that is visibly not wired up. Alerts are
visible at :9093 and in Prometheus; add a receiver to alertmanager.yml to
make them travel. critical already routes separately so it can be pointed
somewhere else without touching the rest.
Check that it fires
An alerting system nobody has seen fire is an assumption.
llmgw ctl cordon <node> # work already there keeps running
# ~2 minutes later
curl -s http://<host>:9093/api/v2/alerts | jq '.[].labels'
llmgw ctl uncordon <node> # the alert resolves on its own
Cordoning is the cheapest real test: it changes a metric the rules read, stops nothing, and reverses in one command.
What the alerts assume
A stopped model publishes no series. llmgw_model_replicas_ready exists
only for models with a replica placed, so == 0 cannot fire for one that is
deliberately off. That is what makes an on-demand catalogue quiet here.
Saturation of 1 is the healthy limit, not failure. It is the worse of queue
depth and KV occupancy, measured by ctl bench rather than guessed. The alert
allows five minutes above it before saying anything.
Latencies are in seconds. These thresholds are written for CPU inference. A p99 first token of thirty seconds is the alerting point here and would be absurd on a GPU; change it when the hardware changes.
Disk is only reported by nodes that hold weights. A node with no model directory reports nothing rather than zero, and the rule divides free by total so a node that reports neither never fires.
What is not here
No exporter on each node, so no host CPU, load average or network. The control
plane publishes memory, disk and liveness for every node, which is what
placement decides on; the rest is a node_exporter away if somebody wants it.
No probe of the mirror. Its /v1/health is JSON rather than Prometheus text,
and checking it properly means blackbox_exporter.
No recording rules. At this size every query in the dashboard runs fast enough raw, and a recording rule is a second definition of a number to keep in step with the first.