Commit Graph
4 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 37a80a9b44 Park a model that fails every time it starts, and say so everywhere
Release / build (push) Successful in 1m3s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m34s
On the live cluster kev-0.5b — an architecture the nodes' llama.cpp did not
know — was started 155 times in eight hours, failing the same way each time,
and phi-4, which fitted on no node, logged "no placement" 250 times in two
days. Neither showed up anywhere an operator looked: the backoff never gave
up, no metric said a model was failing or unplaceable, and the one alert for
a model with nothing serving could not fire, because its series was published
only while a replica was ready.

After five failures in a row a model is parked: not placed again until its
definition changes — a new source or profile — or `llmgw ctl retry` is run.
Status, the model list and the admin API show it with its last error. The
placement warning is logged when its reason changes or every ten minutes.

A request for a parked or unplaceable model is answered at once with why,
rather than "loading; retry shortly", which a client retries forever. Each
request still clears the previous placement result and triggers a fresh
attempt, so a model that fits once room is made is not refused on a stale
answer; the router waits for that attempt, not for its full activation
window.

Metrics now cover model health (desired, ready at 0 for what should run,
consecutive start failures, parked, unplaceable), engine installs per node,
leadership, and certificate expiry read from disk at each scrape, with alerts
for each. `ctl status` counts a model against what it should have now, so an
on-demand model that was asked for and fitted nowhere no longer reads as
satisfied.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 01:21:57 +02:00
Ilker ManapandClaude Opus 5.5 93bd16f00e Watch InfiniBand ports while the agent runs, not only when it starts
A port's state was read once, at registration: one that went down later
left its node on the fabric, offered multi-node work that would hang on
its first collective, until somebody restarted the agent. The agent now
re-reads every IB and RoCE port on each heartbeat - state, physical
state, rate and the error and data counters - and when the fabrics it
can carry traffic on change, says so once; the control plane replaces
the node's fabrics, which is what placement reads.

The ports are exported as llmgw_ib_port_active (with state and
phys_state, so an unplugged cable reads differently from a missing
subnet manager), llmgw_ib_port_rate_gbps and llmgw_ib_port_counter_total,
and a node's page lists them with the fabric each carries and any error
counter that is not zero.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 00:13:54 +02:00
Ilker ManapandClaude Opus 5.5 daed835d88 Measure each node's CPU and memory, and give every node a page
The nodes are CPU machines, and nothing measured how busy one was: the
heartbeat carried GPU memory and disk, and llmgw_node_memory_bytes is the
total. The agent now reads /proc between heartbeats - CPU across all
cores, load averages, memory used and available, and each engine's
process group's cores and resident memory - and sends it with the
heartbeat. Nothing new runs on the nodes and Prometheus needs no new
target: the control plane exports it with the rest of the fleet, as
llmgw_node_cpu_utilization_percent, llmgw_node_cpus, llmgw_node_load,
llmgw_node_memory_used_bytes, llmgw_node_memory_available_bytes,
llmgw_shard_cpu_cores and llmgw_shard_memory_bytes. A measurement more
than a minute old is left out rather than repeated.

The cluster page's node table shows CPU and RAM as bars, and each node
links to a page of its own: current CPU, RAM, load and disk, the engines
on it with their cores and memory, and an hour of CPU, RAM, load,
per-model cores and memory and queue depth from Prometheus. It refreshes
in place.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 23:21:23 +02:00
Ilker ManapandClaude Opus 5 42fd0aa31e Export what the cluster is made of, not only what it is serving
CI / test (push) Successful in 1m28s
The router published twenty-two metric families about requests, endpoints,
admission and scoring — everything about traffic and nothing about the
machines carrying it. There was no way to graph GPU usage, watch a node go
unhealthy, see a fabric fill up, or answer "was the cluster full at the
time", which is the first question anyone asks after an incident. The
agents exported nothing at all.

Exported by the control plane rather than by each agent. It already holds
every number: hardware arrives at registration and state on each heartbeat,
so this is a map read away. One scrape target for a fleet of hundreds, no
Prometheus configuration per node, no firewall rule per machine, and
nothing new running on the nodes. On the same /metrics as the traffic
figures, because correlating "the cluster was full" with "requests were
queueing" should not need two systems.

A collector rather than gauges kept current by a ticker: the numbers are
derived from state that already exists, so computing them when asked is
simpler than maintaining a copy that can drift, and nothing runs between
scrapes.

GPU utilisation is new, and is the number people actually watch. Memory
used says a model is loaded; this says whether it is doing anything, and a
fleet at ninety percent occupancy and two percent utilisation is a fleet
holding models nobody is asking for.

Adding it caught two things. utilization.gpu goes last in the nvidia-smi
query and is optional, because a driver too old to know it returns a
shorter row — requiring it made an llmgw upgrade silently stop discovering
GPUs on an older node, which is far worse than not knowing how busy a card
is. And I gave the new protobuf field number 14, which ecc_errors already
had; buf lint is why that was a moment rather than a corrupted wire format.

Absent is kept distinct from zero throughout, for the same reason as
everywhere else here: an unreported utilisation shown as a flat zero reads
as an idle fleet rather than a missing measurement, and a replica nothing
has reported would be the most idle thing in the cluster.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 15:09:27 +02:00