The metrics existed and nothing could record them. They shared a listener
with pprof, the management API and the dashboard, and pprof can dump the
process heap where API keys sit in plaintext — so a fleet could be monitored
or keep its heap private, and not both. That is not a choice anyone should
have to make, and it is why every previous answer to "how do I scrape this"
was a tunnel.
metrics_listen serves metrics and health and nothing else, so it can be
reached from wherever Prometheus lives. Metrics stay on the admin listener
too: adding a door is not the same as moving one, and anything already
scraping keeps working.
While there: admin_listen defaulted to :9090, every interface, with a
comment beside it saying not to expose it. A default that contradicts its
own documentation is a footgun, so it is loopback now and someone who wants
otherwise says so.
The handler is nil until the registry is built, and says so rather than
serving an empty page. Ordering has already caught this exact area once —
a collector registered after AdminHandler published nothing at all, silently
— and an empty scrape target looks like a working one.
deploy/monitoring/prometheus.yml has a starting configuration and the alerts
worth having: a node that has stopped reporting, a model that should be
serving and is not, a card kept out of placement by a fault, and a fleet
holding models nobody is asking for.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The feature the idle tracking was for. A cluster with more models than GPUs
can now hold every model in its catalog and only the wanted ones in its
cards: min 0 with a max above it means load on demand, and once nobody has
asked for idle_timeout the replicas stop and the cards go back.
No new field, because min 0 max 1 already reads correctly — keep none
running, allow up to one. Which made min 0 max 0 meaningful for the first
time: configured and deliberately not running. normalize() used to fill a
zero max in, on the reasonable ground that a model with no maximum cannot
run; that would now silently turn every switched-off model into an
on-demand one, including the ones already written that way in this
cluster's own configuration. A model with no scaling stanza stays off for
the same reason — defaulting it to on-demand would let any client request
start a 70B nobody intended to run.
The one real decision is what happens to the request that triggers a load.
Refusing makes the feature useless, since the first request for every model
fails; waiting without a bound turns a request into a job, and a client that
budgeted thirty seconds holds a connection for ten minutes. So it waits
briefly and then says what is happening — 503 with Retry-After and "the
model is loading", which is a different answer from "no endpoint serves
this" and produces different client behaviour. An unknown model still gets
the unknown-model error, or on-demand would turn every typo into "loading"
and the useful message would disappear.
Use keeps a model alive rather than only the request that started it,
without which the timer measures time since the load and a busy model is
evicted on schedule however hard it is working.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
42fd0aa deployed and published nothing. RegisterMetrics appends to a slice
that AdminHandler reads when it constructs the registry, and I called it
after — a comment on the function says "called before AdminHandler, because
the registry is built there", written by me, immediately above the code
that did not.
Nothing failed. The build was clean, the tests passed, the endpoint
returned 200 with fourteen kilobytes of metrics, and not one of them was a
node. Only grepping the live endpoint for what was supposed to be there
showed it.
So there is now a test that a registered collector reaches /metrics, which
is the assertion that was missing: everything around it was tested and the
wiring between them was not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The router published twenty-two metric families about requests, endpoints,
admission and scoring — everything about traffic and nothing about the
machines carrying it. There was no way to graph GPU usage, watch a node go
unhealthy, see a fabric fill up, or answer "was the cluster full at the
time", which is the first question anyone asks after an incident. The
agents exported nothing at all.
Exported by the control plane rather than by each agent. It already holds
every number: hardware arrives at registration and state on each heartbeat,
so this is a map read away. One scrape target for a fleet of hundreds, no
Prometheus configuration per node, no firewall rule per machine, and
nothing new running on the nodes. On the same /metrics as the traffic
figures, because correlating "the cluster was full" with "requests were
queueing" should not need two systems.
A collector rather than gauges kept current by a ticker: the numbers are
derived from state that already exists, so computing them when asked is
simpler than maintaining a copy that can drift, and nothing runs between
scrapes.
GPU utilisation is new, and is the number people actually watch. Memory
used says a model is loaded; this says whether it is doing anything, and a
fleet at ninety percent occupancy and two percent utilisation is a fleet
holding models nobody is asking for.
Adding it caught two things. utilization.gpu goes last in the nvidia-smi
query and is optional, because a driver too old to know it returns a
shorter row — requiring it made an llmgw upgrade silently stop discovering
GPUs on an older node, which is far worse than not knowing how busy a card
is. And I gave the new protobuf field number 14, which ecc_errors already
had; buf lint is why that was a moment rather than a corrupted wire format.
Absent is kept distinct from zero throughout, for the same reason as
everywhere else here: an unreported utilisation shown as a flat zero reads
as an idle fleet rather than a missing measurement, and a replica nothing
has reported would be the most idle thing in the cluster.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The last piece of not restarting a control plane to try a model:
llmgw ctl model add -f qwen-coder-32b.yaml
llmgw ctl model remove qwen-coder-32b
The file is the same YAML a model takes in server.yaml, read by the same
koanf parser through the same tags, so an entry moves between the two by
being copied and a field that works in one cannot silently mean nothing in
the other.
Wired only when there is a store. A control plane without one says so
rather than accepting a model that vanishes on the next restart, which is
the kind of success that wastes an afternoon. At startup the catalog is
rebuilt from configuration plus what was stored, so models added before a
restart are still there after it — without that the whole feature would be
a slower way to lose work.
Every failure adding a model is the definition being wrong — a collision, a
missing fallback, a profile that cannot be placed — so they are 400s. A 500
would send someone to the server's logs for a mistake in their own file.
The actor comes from the caller and is kept with the row, so a catalog that
has grown over months can say where each entry came from.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adding a model was an operator procedure with downtime: edit a file on the
control plane, restart it, wait out the settling window, watch replicas
churn while adoption catches up. I did that eight times in a day, and one
of the restarts cost two minutes of outage while stalled replicas timed
out. None of it was necessary. The reconciler already compares desired
against actual on every pass, so a new catalog is simply a different
desired.
A Catalog stays immutable — New is the only way to build one and it refuses
an invalid one — and what changes is which immutable catalog is current.
Readers take the pointer and finish their operation against it, so a swap
mid-request is invisible rather than a race. The router is told rather than
asked, the same shape as the endpoint list and for the same reason: a push
happens once per change, a pull would happen once per request.
The router still seeds from configuration as well as being pushed to. A
router with a static endpoint list and no control plane is a supported mode
and has nothing to push it a catalog; without seeding it would resolve no
model at all.
Which turned up a bug that was latent while catalogs were fixed.
reconcileModel only ever visits models that are in the catalog, so a
replica whose model had been removed was visited by nothing: it kept its
GPUs, kept counting, and kept running an engine for a name no client could
ask for. The endpoint publisher skipped such a replica with the comment
"the reconciler will tear it down", which was not true. It is now, and the
comment says what actually happens instead of what was assumed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two things a cluster with many nodes and many GPUs needs that were missing,
and they are the same thing seen from two distances.
Idle tracking. The control plane knew what was placed and whether it was
healthy, never whether anyone wanted it — so on a fleet with more models
than cards there was no way to answer the first question anyone asks, which
is what can I stop. The router is the only thing that knows, so it now
reports it: a timestamp written in Acquire, where a request already meets an
endpoint under a lock already held, and collected once a tick as a map. That
shape is the point. Per-request notification would be a firehose carrying
one bit — the newest timestamp wins — and this costs the same on four
replicas as on four hundred.
It travels through an interface in the direction opposite EndpointSink, for
the same reason that one exists: the control plane decides what should run,
the router knows whether anyone wants it, and neither should need to be in
the other's process to say so.
Unknown is kept distinct from idle throughout. A router that has just
restarted knows nothing, and a zero timestamp would be the oldest idle
replica in every comparison — so one restart would nominate an entire fleet
for eviction. Absent from the map, "-" in the table, negative in the JSON,
and IdleFor returns a second value rather than a duration anything could
act on by mistake.
ctl status. A per-node listing is right for four nodes and wrong for two
hundred: how full am I, what is broken, what is holding cards nobody is
using are aggregates, and reading them off a long table is a chore, which
means it gets skipped. Capacity per GPU class with usage bars, free whole
nodes per fabric — the number that decides whether a multi-node replica can
land, since free cards on the wrong fabric are not capacity for a model that
spans nodes — replica and model counts, models that cannot be placed, and
what has gone unasked-for. Derived from the same state the tables show, so
it cannot disagree with them.
Unhealthy GPUs count as neither used nor free, because counting them free is
how a cluster looks like it has room it does not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Clicking "metrics" on the web dashboard answered "no api key was provided".
The page is served twice and its navigation assumed one of them. Metrics,
pprof and the management API exist on the admin listener; account
administration, the profile and the chat exist on the client-facing port. The
nav offered the first set in both places, so on the public port those links
fell through to the API-key middleware — which is a far more confusing way to
say "that is not here" than a 404 would have been.
Each page now offers what is actually reachable from it, and the public one
says where the rest lives rather than pretending it does not exist, including
why that port is not exposed: pprof can dump the process heap, and the heap
holds every API key in plaintext.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reaching it meant an SSH tunnel, because the page lived on the admin listener
and that listener is bound to loopback. Opening the port was never the way to
fix that: it also serves pprof, and pprof can dump the process heap, where
every API key sits in plaintext. Exposing the port would have been exposing
the key store.
So the page moves rather than the port. The same dashboard is now mounted at
/admin on the client-facing port, behind the session and CSRF machinery that
already guards /profile, gated on a flag on the account. The admin listener
keeps pprof and metrics and stays on loopback.
One bit, not a role system: it grants looking. Everything that changes the
cluster or issues a credential stays on the CLI, where it needs a shell rather
than a session — which keeps the worst case of a stolen admin cookie to "saw
the cluster" rather than "drained your fleet".
Someone without the flag gets 404, not 403. A 403 confirms the page is there
and that admin accounts exist here, which is free reconnaissance; 404 is what
every other unknown path returns. The profile page does not show a link to a
door the visitor cannot open.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Pasting the gateway's address into a browser returned 404, which looks like a
broken endpoint rather than one that tells you how to use it.
Two pages, and the split between them is a security boundary rather than a
layout choice. The landing page is served without a key — it has to be,
because the person reading it is the one who does not have one yet — so it
contains no data at all: it says how to connect, not what is here. Model names
stay behind /v1/models. It reconstructs the address from the forwarded headers
so the copy-pasteable example names the address the client actually used
rather than the internal one; those headers are reflected into displayed text
and never used for a decision.
The dashboard shows nodes, models, replicas, placement failures and usage, and
lives on the admin listener only. That listener is loopback-bound and also
serves pprof, which can dump the process heap — where API keys sit in
plaintext — so reaching the page from elsewhere is an SSH tunnel's job. It is
read-only: anything that mints a credential stays in `llmgw keys`, where it
needs a shell rather than a browser session this project has no way to
authenticate.
Mounting the landing page uncovered a hole in the exemption list. An entry
ending in "/" was a prefix rule, and "/" ends in "/", so exempting the landing
page would have exempted every path there is — authentication still installed,
still logging, and letting the whole API through. "/" is now exact, with a
test enumerating what must and must not be exempt.
Server-rendered templates and one stylesheet, embedded. The single-binary goal
rules out a JavaScript toolchain and nothing on either page needs one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two bugs found while verifying mTLS against real machines.
session cancelled its own derived context and then asked that context
whether the agent was shutting down. The answer was always yes, so every
stream error read as a clean close: the log said err=<nil>, the backoff
reset, and an agent whose certificate the control plane refused hammered it
forever at the base interval with nothing to diagnose. It now keeps the
parent context to answer that question, and a refusal reaches the operator
naming both the certificate and the claim.
Cancelling the context also did not stop the agent. HTTP/2 watches the
context while the request body is being written, and a bidirectional stream
spends its life blocked on the read side instead, so the read never returned.
Measured on a deployed node: `systemctl stop llmgw-agent` took 90 seconds and
ended in SIGKILL, on every restart. Closing the response when the context
ends is what actually unblocks it.
TestAgentStopsWhenContextEnds hangs reliably without the second fix, which is
the only reason to trust that it tests it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A node's identity was whatever the agent said it was. With mutual TLS the
control plane compares the client certificate's common name against the
claimed node id and refuses the connection when they disagree, so identity
becomes something a node proves rather than something it picks.
A public authority cannot supply these certificates: Let's Encrypt and its
peers stopped issuing the clientAuth usage, and Go verifies client chains
against it. Hence internal/tlsconf/ca.go, a small authority whose key lives
on one machine and whose root is trusted everywhere, and `llmgw ctl cert`
over it so issuing is reproducible instead of remembered openssl.
The server certificate's hosts matter more than they look: hostname
verification reads subject alternative names and ignores the common name
entirely, so a certificate without the address agents dial — bare IP
included — is rejected no matter what its subject says. There is a test for
exactly that, because it is the failure everyone hits once.
requireIdentity stays wired to false until the configuration lands, so this
commit changes no running behaviour.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ctl talks to a management API on the admin listener, not the client-facing one.
The two have different audiences: one serves inference to whoever holds an API
key, the other lets a caller cordon nodes and read the cluster's shape.
- nodes surfaces unhealthy GPUs and why they are excluded. That is usually the
answer when a node looks idle but nothing will schedule onto it, and it was
invisible before.
- explain-placement prints every candidate considered and why it was rejected.
- replicas shows the shards a replica is made of, with per-shard errors.
- cordon, uncordon and drain are one endpoint with different targets, because
the difference between cordon and drain is exactly what an operator needs to
be sure of before running either.
Draining had to be fixed first to make ctl drain worth having. A drained replica
was stopped on the next pass, which cuts in-flight requests — the thing draining
exists to avoid. Now a node marked Draining has its replicas drained, the model
falls below its desired count, a replacement is placed elsewhere, and the
original is stopped only once the replacement is serving. A grace period bounds
it, because draining the only node that can host a model would otherwise never
finish and the node could never be taken out of service.
Verified against a two-node cluster of real binaries: draining moved a replica
to the surviving node and traffic kept returning 200 throughout.
ctl logs is deliberately absent. It needs a request/response pair the agent
protocol does not have, and the failure path already ships an engine log tail
with every status report — so the case that matters most is covered, and
reading a healthy shard's output on demand is not worth a protocol change yet.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One bidirectional connect-go stream per node, carrying registration, heartbeats
and shard status up, and start, stop and drain down. The agent dials out: GPU
nodes sit behind firewalls far more often than a control plane does, and an
outbound connection needs no inbound rule on every node in the fleet.
- registration carries the whole hardware inventory, so there is no separate
discovery call and a reconnecting agent refreshes what the scheduler knows.
The fabric labels travel with it; they cannot be inferred from the wire, so
losing them in transit would silently disable multi-node placement.
- agentsvc is both the stream handler and the reconciler's dispatcher, because
it holds the open streams and is therefore the only thing that can deliver a
command to a node.
- the agent validates a profile before launching anything, since an engine that
refuses a profile does so after minutes of loading, then supervises the
process and reports every transition with the engine's output tail on failure.
- reconnects forever with jittered backoff. A control plane restart must not
require touching every GPU node, and a fleet must not reconnect in lockstep
and knock it over again. Engines keep serving throughout, because the router
talks to them directly.
llmgw agent is real now, and llmgw server runs a control plane whenever any
model declares a profile. The static endpoint list stays for router-only runs.
Verified with real binaries as separate processes: the agent registers, a
replica is placed, an engine launches through the vLLM adapter's actual argument
construction, streaming traffic is served, and killing the agent moves the node
through unreachable to lost with placement explaining why nothing can run.
config.Model and catalog.Model were the same thing described twice, so the
router now reads catalog entries directly and the validation rules live in one
place.
Still missing: mTLS. The stream is plaintext HTTP/2 and must stay on a trusted
network until that lands.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
llmgw server is real now. It fronts a static set of engine replicas with an
OpenAI-compatible API, scoring endpoints through the pipeline added in the
previous commit.
- proxy: SSE relayed event by event with a flush after each, cancellation
propagated upstream, and separate deadlines for dial, headers, first token,
inter-token idle and total. The idle watchdog is what distinguishes a wedged
engine from a slow one.
- router: dispatch with retry and model fallback. Retry only happens while
nothing has reached the client; after that, splicing two responses together
would be worse than the short stream the client already sees.
- config: YAML via koanf overlaid on defaults, with LLMGW_ env overrides.
Validation rejects duplicate endpoint ids and aliases that shadow a model,
because both misroute silently at request time rather than failing loudly.
- metrics: requests, duration, TTFT, tokens, plus per-scorer values and
per-filter drops on a separate admin listener with health and pprof.
Phase 1 exit criterion, measured in test/e2e: 1000 concurrent streams across
3 sim-engines complete at ~6800 req/s; every client disconnect frees the KV
blocks on every engine; router overhead p99 is about 0.9 ms against the plan's
2 ms budget.
The handlers live in internal/router rather than an internal/router/openai
package as the plan's tree sketched: they need the dispatch loop, and splitting
them apart would have meant an interface with exactly one implementation. Plan
§4 updated to match.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ilkerandClaude Opus 5