Give metrics a door that does not also open onto pprof
CI / test (push) Successful in 1m38s

The metrics existed and nothing could record them. They shared a listener
with pprof, the management API and the dashboard, and pprof can dump the
process heap where API keys sit in plaintext — so a fleet could be monitored
or keep its heap private, and not both. That is not a choice anyone should
have to make, and it is why every previous answer to "how do I scrape this"
was a tunnel.

metrics_listen serves metrics and health and nothing else, so it can be
reached from wherever Prometheus lives. Metrics stay on the admin listener
too: adding a door is not the same as moving one, and anything already
scraping keeps working.

While there: admin_listen defaulted to :9090, every interface, with a
comment beside it saying not to expose it. A default that contradicts its
own documentation is a footgun, so it is loopback now and someone who wants
otherwise says so.

The handler is nil until the registry is built, and says so rather than
serving an empty page. Ordering has already caught this exact area once —
a collector registered after AdminHandler published nothing at all, silently
— and an empty scrape target looks like a working one.

deploy/monitoring/prometheus.yml has a starting configuration and the alerts
worth having: a node that has stopped reporting, a model that should be
serving and is not, a card kept out of placement by a fault, and a fleet
holding models nobody is asking for.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-17 18:12:10 +02:00
1 parent ad37bee081
commit 1ea067d08e
8 files changed
+244 -10

No files matched your search

+28
View File
@@ -557,6 +557,34 @@ Or open `/chat` in a browser and use it without a key at all — the page carrie
your session, and the request still goes through admission, rate limits, budget
and accounting exactly as an API call does.
## Watching it
The control plane exports both halves on one endpoint: what the cluster is
serving — requests, latency, time to first token, tokens, admission, per
endpoint — and what it is made of, per node and per GPU, including memory,
temperature, ECC faults and utilisation.
Metrics get their own listener, because the admin one also carries pprof and
pprof can dump the process heap, where API keys sit in plaintext:
```yaml
admin_listen: "127.0.0.1:9090" # dashboard, management API, pprof — keep local
metrics_listen: "0.0.0.0:9091" # metrics only, safe to reach
```
One scrape target for the whole fleet. The node and GPU figures come from the
control plane, which already holds them, so there is nothing to install on the
nodes and no Prometheus configuration per machine.
A starting configuration, with the alerts worth having, is in
[deploy/monitoring/prometheus.yml](../deploy/monitoring/prometheus.yml).
Two figures repay knowing about. `llmgw_gpu_utilization_percent` is absent
rather than zero when the driver does not report it — a flat zero across a
fleet means the measurement is missing, not that the fleet is idle. And
`llmgw_gpu_nvlink_peers` at zero everywhere means the topology is unknown,
which is what a binary built without the nvml tag reports.
## When something does not work
**The node is `ready` but every replica fails.** Almost always a missing engine