Ilker ManapandClaude Opus 5.5
c866b3a8ee
The busy figure is nvidia-smi's utilization.gpu: the share of time any kernel ran, which a kernel on one SM fills. On cuda01 decode read 92% busy while the SMs had work 69% of the time and issued on 18% of their cycles. Busy is now said to be that, and beside it: each card's draw against its power limit (efficiency table, fleet and node charts, the console's GPU work chart), and where the profiling counters can be read, SMs active, SM occupancy, tensor cores and DRAM (llmgw_gpu_*_active_percent), from NVML's GPM on Hopper and later or a dcgm-exporter on the node (discovery.dcgm_exporter). The GPU map colours by work by default and keeps busy as a mode. A MIG instance is never published with its card's profile. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Watching a cluster
Prometheus, Alertmanager and Grafana, configured for what llmgw actually publishes. Every alert here is written against a series this system emits; nothing is aimed at a metric that might exist.
What it is
One scrape target for the whole fleet.
Agents publish no metrics of their own. The control plane gathers what they report — node state, memory, disk, engine queue depth, KV occupancy — and republishes all of it, so a single endpoint describes every machine. That is worth knowing before going looking for an exporter on each node.
The port it scrapes
metrics_listen, not admin_listen.
The admin listener is bound to loopback for one reason: it also serves pprof, and pprof can dump the process heap, where API keys sit in plaintext. Metrics are the one thing on that listener something else genuinely needs to reach, so they get a port of their own instead of becoming an argument for opening the other one.
# /etc/llmgw/server.yaml
metrics_listen: ":9091"
Confirm it carries nothing else:
curl -s -o /dev/null -w '%{http_code}\n' http://<control-plane>:9091/debug/pprof/ # 404
curl -s -o /dev/null -w '%{http_code}\n' http://<control-plane>:9091/api/v1/nodes # 404
Installing
On Ubuntu 24.04 the first two come from the archive; Grafana needs its own repository.
apt-get install prometheus prometheus-alertmanager
install -m 0644 prometheus.yml /etc/prometheus/prometheus.yml
install -m 0644 llmgw-rules.yml /etc/prometheus/rules/llmgw.yml
install -m 0644 alertmanager.yml /etc/prometheus/alertmanager.yml
# Alertmanager answers on loopback only. Its API takes silences from anyone who
# can reach it, with no login, and silencing BackupTooOld or ModelParked is how
# something that should be noticed is not. Prometheus beside it is its only
# client; one Alertmanager needs no cluster port.
echo 'ARGS="--web.listen-address=127.0.0.1:9093 --cluster.listen-address="' \
> /etc/default/prometheus-alertmanager
# Prometheus itself is read-only as installed: its admin and lifecycle APIs
# stay off (no --web.enable-admin-api, no --web.enable-lifecycle). It still
# answers any query from anyone who reaches :9090, which says which models and
# nodes there are and how busy each is. Its readers are the control plane
# (monitoring.prometheus) and Grafana beside it; to keep it to those, allow
# the control plane and drop the rest, e.g. with nftables:
# nft add rule inet filter input tcp dport 9090 ip saddr != { 127.0.0.1, CONTROL_PLANE_IP } drop
promtool check config /etc/prometheus/prometheus.yml
systemctl restart prometheus prometheus-alertmanager
install -m 0644 grafana/datasource.yaml /etc/grafana/provisioning/datasources/llmgw.yaml
install -m 0644 grafana/dashboards.yaml /etc/grafana/provisioning/dashboards/llmgw.yaml
install -m 0644 -o grafana -g grafana grafana/llmgw-overview.json \
/var/lib/grafana/dashboards/llmgw.json
install -m 0644 -o grafana -g grafana grafana/llmgw-tenants.json \
/var/lib/grafana/dashboards/llmgw-tenants.json
systemctl restart grafana-server
The dashboard is provisioned from a file rather than clicked together, so it survives a reinstall and can be reviewed in a diff.
Where to run it
Not on the control plane. Monitoring that dies with the thing it monitors tells you nothing at the moment you need it, and the control plane is also the busiest machine in a small cluster.
Anywhere else on the management network will do. The mirror is a reasonable host: it has the disk, and serving weights is bursty rather than constant.
Nothing is sent anywhere
Alertmanager groups and holds alerts and has no receiver.
Where an alert should go — an address, a Slack workspace, a webhook — is a
decision with credentials attached, and a monitoring system that quietly starts
mailing people is worse than one that is visibly not wired up. Alerts are
visible in Prometheus, and at :9093 on the monitoring host itself (ssh -L 9093:127.0.0.1:9093 <host> to open its page); add a receiver to alertmanager.yml to
make them travel. critical already routes separately so it can be pointed
somewhere else without touching the rest.
Check that it fires
An alerting system nobody has seen fire is an assumption.
llmgw ctl cordon <node> # work already there keeps running
# ~2 minutes later
ssh <host> curl -s http://127.0.0.1:9093/api/v2/alerts | jq '.[].labels'
llmgw ctl uncordon <node> # the alert resolves on its own
Cordoning is the cheapest real test: it changes a metric the rules read, stops nothing, and reverses in one command.
What the alerts assume
A stopped model publishes no series. llmgw_model_replicas_ready exists
only for models with a replica placed, so == 0 cannot fire for one that is
deliberately off. That is what makes an on-demand catalogue quiet here.
Saturation of 1 is the healthy limit, not failure. It is the worse of queue
depth and KV occupancy, measured by ctl bench rather than guessed. The alert
allows five minutes above it before saying anything.
Latencies are in seconds. These thresholds are written for CPU inference. A p99 first token of thirty seconds is the alerting point here and would be absurd on a GPU; change it when the hardware changes.
Disk is only reported by nodes that hold weights. A node with no model directory reports nothing rather than zero, and the rule divides free by total so a node that reports neither never fires.
Where the metrics live
On the machine that scrapes them, in Prometheus's own store, and that is the right answer at this size. Measured on this cluster: 2,359 series at 41 samples a second, which compresses to roughly 7 MB a day — about 3 GB for a year.
# /etc/default/prometheus
ARGS="--storage.tsdb.retention.time=365d --storage.tsdb.retention.size=20GB"
A year because the questions worth asking of this data are seasonal: what did capacity look like last quarter, was it always this slow, how much did the fleet grow. Fifteen days answers none of them.
Both limits, not one. The time limit is the intent; the size cap is what stops Prometheus becoming the thing that fills the disk if the series count grows tenfold after somebody adds a fleet.
No remote write, no object store, no second tier. At two thousand series that is infrastructure for a problem this cluster does not have, and every one of those has its own failure modes to learn.
The part that is not about capacity
The disk is not the constraint — 900 GB free against 3 GB a year. What matters is what shares it.
On this cluster the model mirror and the metrics database are on the same filesystem. A mirror that fills it stops Prometheus writing, so monitoring goes quiet at exactly the moment the disk is the problem, and the alert that would have said so is the one that died.
Two ways out, in order of how much they cost:
- Watch it, which is now done.
prometheus-node-exporterarrives with the Prometheus package and was running, unscraped, on this machine — so the disk holding both was watched by nothing. It is a scrape target now, andMonitoringDiskFillingfires at fifteen percent rather than five, because the thing being protected is the ability to be told. - Separate them: a volume of its own for
/var/lib/prometheus, or monitoring on its own machine. Worth doing before the mirror holds a few hundred gigabytes; not worth doing today at seven percent used.
What is not here
No exporter on the llmgw nodes, so no host CPU, load average or network for
them. The control plane publishes memory, disk and liveness for every node,
which is what placement decides on; the rest is a node_exporter away if
somebody wants it. The monitoring host itself is scraped, for the reason above.
No probe of the mirror. Its /v1/health is JSON rather than Prometheus text,
and checking it properly means blackbox_exporter.
No recording rules. At this size every query in the dashboard runs fast enough raw, and a recording rule is a second definition of a number to keep in step with the first.