Commit Graph
2 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5 1ea067d08e Give metrics a door that does not also open onto pprof
CI / test (push) Successful in 1m38s
The metrics existed and nothing could record them. They shared a listener
with pprof, the management API and the dashboard, and pprof can dump the
process heap where API keys sit in plaintext — so a fleet could be monitored
or keep its heap private, and not both. That is not a choice anyone should
have to make, and it is why every previous answer to "how do I scrape this"
was a tunnel.

metrics_listen serves metrics and health and nothing else, so it can be
reached from wherever Prometheus lives. Metrics stay on the admin listener
too: adding a door is not the same as moving one, and anything already
scraping keeps working.

While there: admin_listen defaulted to :9090, every interface, with a
comment beside it saying not to expose it. A default that contradicts its
own documentation is a footgun, so it is loopback now and someone who wants
otherwise says so.

The handler is nil until the registry is built, and says so rather than
serving an empty page. Ordering has already caught this exact area once —
a collector registered after AdminHandler published nothing at all, silently
— and an empty scrape target looks like a working one.

deploy/monitoring/prometheus.yml has a starting configuration and the alerts
worth having: a node that has stopped reporting, a model that should be
serving and is not, a card kept out of placement by a fault, and a fleet
holding models nobody is asking for.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 18:12:10 +02:00
Ilker ManapandClaude Opus 5 3a4cc0c719 Register the fleet collector before the registry is built
CI / test (push) Successful in 1m31s
42fd0aa deployed and published nothing. RegisterMetrics appends to a slice
that AdminHandler reads when it constructs the registry, and I called it
after — a comment on the function says "called before AdminHandler, because
the registry is built there", written by me, immediately above the code
that did not.

Nothing failed. The build was clean, the tests passed, the endpoint
returned 200 with fourteen kilobytes of metrics, and not one of them was a
node. Only grepping the live endpoint for what was supposed to be there
showed it.

So there is now a test that a registered collector reaches /metrics, which
is the assertion that was missing: everything around it was tested and the
wiring between them was not.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 15:27:39 +02:00