Commit Graph
73 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 76746715ec A set's files checked four at once, the check said as it goes and not taken for a stuck download
CI / test (push) Failing after 6m51s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 14:58:44 +01:00
Ilker ManapandClaude Opus 5.5 12464ce0f6 Cluster upgrades: nodes told together with progress said, done when back on the version, the download kept past a broken stream
CI / test (push) Successful in 10m2s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 11:14:02 +01:00
Ilker ManapandClaude Opus 5.5 b8ed368e31 Weights from Hugging Face pinned to a commit and checked against the sha256 it published
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 07:33:15 +01:00
Ilker ManapandClaude Opus 5.5 1908bfb4b2 Strata on cuda01 end to end: its engine is the shard's, its metrics read
CI / test (push) Failing after 7m31s
Found running it through the cluster:

- Strata's server starts its engine in a process group of its own, so the
  agent saw a GPU replica on the CPU and its 22 GB as somebody else's
  memory. A process the shard started, at any remove, is now the shard's.
- An adopted shard's kept compute was sent at once and never again, so a
  wrong one read again on adoption never reached the control plane: what
  is sent is sent again when it changes, and a GPU shard kept as the CPU
  is read again on adoption.
- /metrics/prometheus is llama.cpp's names without the KV cache; vLLM's,
  with it, are /metrics?format=prometheus. The router now sees running 1,
  waiting 2 under load.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 15:48:15 +01:00
Ilker ManapandClaude Opus 5.5 b986ea180d Fixes from review: setup's memory check, MCP secrets, VS Code, bandwidth
CI / test (push) Successful in 9m47s
- server init run again on a machine already set up is not refused its
  memory (the docs fix a certificate with --host so), and judges count
  only when one will be installed (judgeComes: not without a mirror, nor
  from a bundle without it).
- A server's sealed secrets that it no longer reads are removed when it is
  put or deleted; replacing a credential replaces every header it was
  sent, rather than adding one beside the old.
- llmgw.gatewayUrl is a machine setting and restricted in untrusted
  workspaces: a cloned repository's settings cannot send a sign-in to
  another gateway.
- The bandwidth buffer is at least eight times the CPU's last cache and at
  most a quarter of free memory, else nothing is measured until there is
  room; a registration carries the figure measured so far.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 08:23:00 +01:00
Ilker ManapandClaude Opus 5.5 a583359ebf Agents started together measure their memory apart
CI / test (push) Successful in 9m12s
Seven VMs on one host restarted at once read a third less each than any
gets alone. An agent now waits 10 s to 2 min, at random, before measuring,
and measures again every hour, keeping the best.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 09:56:02 +01:00
Ilker ManapandClaude Opus 5.5 e70d6caa2f The memory is measured after the agent reaches its control plane
CI / test (push) Successful in 9m5s
Measuring it in the startup scan held every agent half a second or more
before registering, which on a shared CI runner was past the failover
tests' wait. Now it runs once a process, in the background, the first time
it is asked for, and Heartbeat.mem_bandwidth_gbs carries it when known;
the control plane keeps the last figure and a heartbeat without one leaves
it. The agent's tests do not measure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 15:31:21 +01:00
Ilker ManapandClaude Opus 5.5 84444e71c8 How fast a model would answer on a CPU, before it is fetched
CI / test (push) Failing after 6m37s
Each agent measures its node's memory read rate when it starts: every CPU
sums a share of a buffer past any cache, a tenth of free memory at most,
the best of four passes, in a fraction of a second; reported in Host
(mem_bandwidth_gbs) and NodeView. A token on a CPU reads every weight it
uses once, so one reply's speed is about that rate over those bytes:
modelplan.CPUTokensPerSecond, 0.68 of the measured rate plus 5 ms a token,
calibrated on the PoC against llama.cpp's predicted_per_second (node01
24.9 GB/s, qwen2.5-1.5b: 14.5 measured, 14.7 estimated; node05 41.9 GB/s,
qwen2.5-14b: 3.3 and 3.2). A model of experts named …-A3B reads its active
part (ActiveShare).

The explorer shows it as CPU tok/s, at Q4_K_M on the fastest node that
holds the model; a repository's page gives it for each GGUF by its size.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 00:09:47 +01:00
Ilker ManapandClaude Opus 5.5 4466d2d15a Weights that run code are refused
CI / test (push) Successful in 8m57s
Python pickles (.bin, .pt, .pth, .ckpt, .pkl) run whatever they hold when
loaded, as the engine's user, on the node; that is how the malicious models
found on Hugging Face work. The node refuses such a source, or a mirror set
holding one, before fetching anything. A model added with one, or given
one, is refused, while one already in the catalog still loads and is
refused by its node with the same words. vLLM and SGLang downloading a
model by its id get --load-format safetensors. The mirror page marks the
file. allow_pickle: true on a model lets it through on purpose, carried to
the node in EngineProfile.allow_pickle.

BACKLOG M lists the rest: checksums against Hugging Face on every path,
its own scan, trusted publishers, engines without the internet, an eval
before an untrusted model serves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 23:17:22 +01:00
Ilker ManapandClaude Opus 5.5 b09922b29b Leadership stays put while the store's primary is replaced
CI / test (push) Failing after 30m3s
Measured again after the store fixes. Cutting the store primary's power no
longer hung requests, but leadership moved anyway: the lease had lapsed for
everyone while Patroni promoted the standby, and a standby asked first. The
new leader then spent about 25 s without its nodes, and so without its
judges.

A standby that saw the store out of reach now leaves the lease alone for one
renew round after the store returns. The default lease is 10 s, and a node
detects a control plane that died silently in 8 s (5 s idle, 3 s ping)
rather than 15.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-15 23:17:58 +02:00
Ilker ManapandClaude Opus 5.5 aa531b1053 ComfyUI models of one environment share one replica
flux and sf3d on one card handed it to each other, each starting ComfyUI
and loading its weights from nothing (BACKLOG H3, placement side). An
environment of two or more models now gets a hidden catalog entry,
comfyui-env-<name>: on its first member's profiles with the most memory
any member needs, on demand for the longest idle timeout, carrying every
member's weights, repositories and the environment's custom nodes. The
members get no replicas of their own; asking for one wants the entry.

The agent fetches each member's weights; the gate links all of them,
refusing two weights files of one name, links each member's Hugging Face
repository, and answers /v1/models for every member. The control plane
offers the replica under the members' names with each one's workflow and
checkpoint, which the router sends by the model asked for. A member's
sibling never evicts the replica serving it, and the members' load counts
as the entry's.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-14 13:55:34 +02:00
Ilker ManapandClaude Opus 5.5 88cbf11570 Training finds the engines where the installer puts them when agent.yaml names no directory
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 20:55:50 +02:00
Ilker ManapandClaude Opus 5.5 79b3e9f887 LoRA adapters trained on the nodes: llmgw ctl train
A job names the catalog model the adapter is for, the safetensors weights to
train on, the node and the MiB of a card it needs, and question/answer pairs.
The control plane sends it when the node's agent is there and nothing else
trains on it; the agent trains with its installed vLLM's torch and a pinned
overlay of peft and accelerate (no second environment), as the engines'
user, on the card with that much free — beside an engine, if it leaves that
much — converts the adapter to GGUF with the installed llama.cpp's own
conversion code when the model is served by llama.cpp, and sends what it
made back in pieces, kept in the store to download. Progress, loss and peak
card memory are reported as it goes; a node that goes fails its job.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 20:25:57 +02:00
Ilker ManapandClaude Opus 5.5 dc243f8840 The nodes' agent settings kept with the cluster's, agent.yaml down to what an agent needs to start
agents.default and agents.nodes.<id> in the cluster's settings hold
everything agent.yaml can say but the agent's id, control planes, address and
certificate, which stay in the file and are refused there. The control plane
sends each node its settings on register and, a second apart, when they
change; an agent whose settings changed checks they load beside its file,
keeps them and starts again on them, engines serving throughout. An edited
agent.yaml that loads restarts the agent too. History, audit, ctl config, the
console and the portal's settings page cover them as the rest of the
settings.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 18:06:36 +02:00
Ilker ManapandClaude Opus 5.5 2d14918b99 An adopted engine that turns ready after registration is reported
The registration carries each shard's state as it is then. An adopted
engine is health-checked after that, and nothing reported the ready that
followed, so when the agent registered first the control plane kept the
replica starting while it served. After llmgw restart on the PoC two
replicas stayed so, one the only replica of qwen2.5-coder-14b. The
session's watcher now reports every shard the registration called not
ready once it is.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-07 19:43:13 +02:00
Ilker ManapandClaude Opus 5.5 7245dbe1dd llmgw stop, start and restart act on the whole system
Issue #3 asked for commands to stop and start the system, and 05aa21c gave
only one machine's services. Now:

- llmgw stop sets serving.stopped in the cluster's settings, live: the
  reconciler wants every model at no replicas, freeing every GPU, and the
  gateway answers every POST under /v1/ with 503 system_stopped. The control
  planes, agents and mirror keep running, and no model's definition changes.
- llmgw start clears it; every model is placed again as it asks.
- llmgw restart has every agent restart over its stream (RestartAgent), then
  the other control planes over ssh, standbys before a leader, then this
  one; engines keep serving.
- llmgw status is the cluster's summary, saying when it is stopped.

One machine's services moved to llmgw service status|start|stop|restart.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-07 18:01:45 +02:00
Ilker ManapandClaude Opus 5.5 b5d650af16 What a replica computes on comes from nvidia-smi; llama.cpp says its layers only at -lv 4
Checked against the installed llama.cpp (b10991): at its default verbosity
it prints neither the devices it uses nor the layers it offloads, so the
start-up output said nothing and every replica's ON column stayed empty.
The agent now takes compute from nvidia-smi - the GPUs the shard's processes
hold memory on, and how much - or CPU, for every engine, falling back to it
when llama.cpp's output says nothing; adopted engines get it too, and each
session reports it once per shard as a compute-only status. engine-check
runs its GPU token at -lv 4, where llama.cpp does say where its layers went.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-07 14:04:08 +02:00
Ilker ManapandClaude Opus 5.5 25b3e129fe What a shard computes on survives the agent's and the control plane's restarts
It was sent once, with READY, and kept only in the leader's memory: a
failover or an agent restart lost it. The agent now keeps it with the
shard's state on disk (ShardSpec.Compute, set by Supervisor.SetCompute) and
reports it with each running shard when it registers (RunningShard.compute),
which adoption carries onto the replica.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-07 13:51:40 +02:00
Ilker ManapandClaude Opus 5.5 0bd0d881c5 A replica says what its engine computes on, and llama.cpp's word on the CPU refuses a GPU placement
llama.cpp's start-up output is read as a shard becomes ready: the devices it
used, its backend, and the layers it offloaded. That goes up with READY in a
new ShardStatus.compute, is kept on the shard, and is shown by the admin
API, ctl replicas (ON) and the console (COMPUTE). A shard given GPUs whose
engine says it found no usable GPU or offloaded no layer is stopped and
failed, even where nvidia-smi cannot be asked.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-07 13:02:17 +02:00
Ilker ManapandClaude Opus 5.5 32c3942040 The firewall turns off and on with the cluster's settings; llmgw wipe
firewall.enabled (default true) is a cluster setting, applied live: the
leader sends a Firewall message to every agent when it registers and when
the setting or firewall_networks changes, and the agent (root) takes llmgw's
nft table out or installs it. A control plane writes the state to
/var/lib/llmgw/firewall.json, which llmgw-firewall-sync.path applies as root
with `llmgw firewall sync`; server init and server add install the units,
`llmgw firewall sync --install` on one set up before. node add and server
add put no firewall on a machine while the setting is off.

`llmgw wipe` removes llmgw from a machine after listing what goes and asking
for its name: units, engines' processes, the nft table, /etc/llmgw,
/var/lib/llmgw, the agent's state, the sudo grant, the CA from the trust
store, keepalived's configuration when llmgw wrote it, and the users.
--models, --store (Patroni member, etcd, or the local database) and --all
take more. The console has it under Do.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-06 08:00:06 +02:00
Ilker ManapandClaude Opus 5.5 1d1409faad A standby's admin API is the leader's
The console and ctl on a standby showed its own empty view: nodes and
replicas live in the leader's memory. A standby now passes /api/ to the
leader over the agent port, with the cluster's certificates both ways; the
leader answers only a certificate naming a control plane the store lists,
and only while it leads. Control plane certificates are issued for client
use as well, and server add reissues older ones.

An agent's reconnect backoff starts again after a session that worked, so
each failover it lives through no longer makes the next one slower.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 02:27:11 +02:00
Ilker ManapandClaude Opus 5.5 9e46ee83e4 Merge C24 KV-index routing from vLLM's cache events
Conflicts with the LoRA work: vLLM's arguments, the engine profile, router
metrics and the scheduler's request all keep both; agent.pb.go regenerated
from the merged proto; the two test fakes of vLLM given separate names.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 01:24:25 +02:00
Ilker ManapandClaude Opus 5.5 117f417ef5 Precise KV-index routing: vLLM KV cache events through the agent, routed by leading blocks held
Plan §7.5 tier 2, opt-in per profile with kv_events. The agent starts vLLM
publishing KV cache events to a loopback port it listens on (pure-Go zmq4,
msgpack in both of vLLM's encodings), with sha256_cbor hashing and a fixed
PYTHONHASHSEED; it mirrors each engine's blocks and sends net changes up the
existing stream, with a full copy on every new one and a reset on gaps,
restarts and clears. The control plane hands them to the router, for the
sending node's own replicas only. The router tokenizes through a replica's
/tokenize, hashes the tokens as vLLM does (checked against cbor2/hashlib
golden values), and scores replicas by the share of leading blocks held,
falling back to the approximate tree. New metrics beside
llmgw_prefix_decisions_total. Untested against a running vLLM.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 01:21:11 +02:00
Ilker ManapandClaude Opus 5.5 2a1e0e6c4c LoRA adapters on vLLM, loaded on demand with replica affinity: max_loaded_adapters
A vLLM model can carry PEFT adapters, registered at start, or with
max_loaded_adapters held at most N per replica: the router loads an adapter
through vLLM's runtime LoRA API when a request first wants it, sends the
adapter's requests to a replica that holds it while that one is not
saturated, loads each cold adapter once, and unloads the least recently used
idle one to make room. llama.cpp keeps loading every adapter at start.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 01:14:38 +02:00
Ilker ManapandClaude Opus 5.5 801e2d51c0 Merge C20 tracing beyond the router
Conflicts: config imports and docgen sources (both kept); agent.pb.go
regenerated with buf from the merged proto (MIG devices and trace context).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 00:05:29 +02:00
Ilker ManapandClaude Opus 5.5 e725adc2f0 Tracing beyond the router: reconcile passes, placement and a replica's start traced down to the node
A reconcile pass that does something is a trace, with placement (where it
went, or every candidate turned down), scale-up/down, drain, stop, eviction,
preemption, reaping and restarts beneath it. A replica's start is a span open
from placement until it serves; the start command carries its context in a
new ControlMessage.trace field, and the agent continues the trace through
the weights fetch, engine launch, rendezvous and readiness wait. Stops,
drains, adoption and engine installs are traced on the node, model jobs are
a trace with a span per step linked to the replica's start, and agent.yaml
takes the same tracing block as server.yaml. Off, every span is the no-op one
and commands carry nothing.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 00:03:17 +02:00
Ilker ManapandClaude Opus 5.5 0fd55a862b MIG: a card in MIG mode is placed as its instances, each a GPU of class <card>-mig-<profile>
Discovery reports a card in MIG mode as its instances instead of the whole
card, on both paths: nvidia-smi (-L names them, -q -x gives their memory,
ids and processes) and NVML (GetMigMode, GetMaxMigDeviceCount,
GetMigDeviceHandleByIndex, behind an interface so it is tested with a fake).
Each instance has its MIG UUID, its own memory, and a class from the card's
and the profile: a100-80g-mig-3g.40gb. Placement numbers instances
1000 + 100 x card + instance and every view shows them as card/migN.

A replica on MIG runs on exactly one instance: a process opens one MIG
device and instances have no NVLink between them, so tp, pp, gpus or nodes
above one on a MIG class are refused in catalog validation and again in
placement, and the planner offers one instance or none. The agent gives
the engine CUDA_VISIBLE_DEVICES=MIG-<uuid> (no Vulkan device variable on
MIG). Memory and processes are per instance; heat, power and wear the
card's; utilisation is left out rather than reported as zero. A card in MIG
mode with nothing carved stays out of placement with the reason, and
llmgw doctor reports each card's mode and instances and warns of the states
in which nothing can run. ctl nodes, ctl status, the console and the plan
count devices by class. docs/install.md has a MIG section.

Written against fakes of nvidia-smi and NVML output; not yet run on MIG
hardware.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 00:01:14 +02:00
Ilker ManapandClaude Opus 5.5 efb4b06fc5 Report a ready engine dying from every connection, adopted ones included
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 11:50:09 +02:00
Ilker ManapandClaude Opus 5.5 3b05ec0538 Have engines listen on the management address, and take SGLang's key off its command line
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 10:36:42 +02:00
Ilker ManapandClaude Opus 5.5 c19abe95e1 Refuse profile env and args that would run code or touch files on the nodes
env is an allowlist of tuning families less path, plugin, library, address
and credential names (NCCL_*_PLUGIN, CUDA_INJECTION64_PATH and the like
got through the old denylist); llama.cpp's LLAMA_ARG_* are checked as
flags. args refuse --trust-remote-code, --log-file, --slot-save-path,
--tool-parser-plugin, --middleware, --allowed-local-media-path and others
that read, write or load, and what the agent sets itself. Checked in the
catalog and again on the node before an engine starts.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 09:54:45 +02:00
Ilker ManapandClaude Opus 5.5 aa3efcebbe Add a standby control plane with server add, and draw the screens as windows
server add gives another machine this llmgw, a certificate, the
configuration and store address, lets it into the store through Patroni, and
starts it; --vip keeps a floating address on the leader with keepalived,
following a new /leader endpoint. Nodes learn every control plane from the
leader and notice a dead one by HTTP/2 ping instead of after the kernel's
fifteen minutes. The setup, console, node and init screens are drawn full
screen in one frame: title bar, sidebar, bordered panel, key bar, dialogs.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-03 22:29:12 +02:00
Ilker ManapandClaude Opus 5.5 477d0bf19d Judge a card shared with another process every heartbeat, not once
Release / build (push) Successful in 1m14s
Release / notes (push) Successful in 4s
CI / test (push) Successful in 3m50s
A card with something else on it when the agent started stayed out of
placement after that process let go of it, until the agent restarted. The
agent now sends what processes other than its own engines hold on each card,
and the control plane replaces the registration verdict with it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-03 18:46:23 +02:00
Ilker ManapandClaude Opus 5.5 82a63cfc6e Read what wears a GPU out: XIDs, remapped rows, PCIe and NVLink errors
Every minute the agent reads nvidia-smi's full report, its NVLink error
counters and the kernel's XID lines. They are published per card,
shown on the node page and the GPU map, charted in Grafana, and five
alerts watch them.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 16:02:07 +02:00
Ilker ManapandClaude Opus 5.5 bd36798b15 Sample GPU utilisation and memory every heartbeat, and chart them
Release / build (push) Successful in 1m12s
Release / notes (push) Successful in 4s
CI / test (push) Successful in 3m52s
They were read once, at agent start, and published unchanged since.
Each sample now carries utilisation, memory and bandwidth in use, the SM
clock and uncorrected ECC errors. The node page charts each card, the
cluster page the fleet by class, Grafana has a GPU row, and three GPU
alerts are added.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 15:42:17 +02:00
Ilker ManapandClaude Opus 5.5 43d7c4eac1 Let the upgrade repair what the field's catalog already holds
Release / build (push) Successful in 1m11s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m51s
tp on a llama.cpp GPU profile is read as gpus_per_node rather than
refused, and a Hugging Face repository page as a vLLM source as the
repository id, so models written that way are placed after the upgrade
instead of dropped from the catalog. Engines adopted after an upgrade
are checked for computing on their GPUs too.

Refs #2.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 14:20:11 +02:00
Ilker ManapandClaude Opus 5.5 e313f32696 Decide engines per node, and let a node join without one it cannot run
engine_overrides give nodes picked by GPU class, id or labels their own
engine versions; node add --disable-engine keeps one off a node; what
vLLM cannot do on a node is a warning rather than a refused node; the
CUDA requirement follows the version; a hand-named binary is no longer
replaced by an installed one.

Refs #2 (items 4 and 7).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 12:07:25 +02:00
Ilker ManapandClaude Opus 5.5 057be16a92 Do not call a GPU shard ready while it computes on the CPU
llama.cpp whose CUDA backend cannot load runs on the CPU without an
error. The CUDA build is now asked for its devices when installed, a
shard on GPUs must hold memory on each before it is ready, and
engine-check reports the devices. Shards get CUDA_DEVICE_ORDER=PCI_BUS_ID
so CUDA numbers GPUs as placement does.

Refs #2 (item 2).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 11:54:39 +02:00
Ilker ManapandClaude Opus 5.5 cf53904cc5 Serve LoRA adapters on llama.cpp models, asked for by name
Release / notes (push) Successful in 3s
Release / build (push) Successful in 1m11s
CI / test (push) Successful in 3m47s
A model's adapters are fetched by each node with its weights and loaded
unapplied at start; a request naming an adapter reaches the model's
replicas with llama.cpp's per-request lora field. Adapter names share the
model namespace, are listed, and follow allowlists on their own. The
router's config seed and pushed catalog now build the name map in one place.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 23:40:47 +02:00
Ilker ManapandClaude Opus 5.5 cd10e086c8 Read nodes' heat and power: GPUs, CPUs, board inlet, RAPL
Release / build (push) Successful in 1m10s
CI / test (push) Successful in 3m51s
Release / notes (push) Successful in 3s
Every agent samples, at most every ten seconds: nvidia-smi's temperature,
HBM temperature, power draw, enforced limit and throttle reasons per GPU;
hwmon's CPU package and board inlet temperatures; and CPU package power
from RAPL's energy counters. Sent with heartbeats, published as metrics
and shown on the node page; unmeasured sensors are absent, never zero.
Observe-only, step 0 of docs/facility-aware-placement.md, which now
describes the agent source and what these readings can and cannot say.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 19:49:08 +02:00
Ilker ManapandClaude Opus 5.5 990d82d801 Reserve CPU cores and require memory for CPU models
Release / build (push) Successful in 1m5s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m47s
A profile's cpu_cores is reserved out of its node's CPUs by placement, as
memory is, and becomes the engine's thread count. Agents report their CPU
count in registration and heartbeats. New CPU models must declare memory;
model views show reserved against measured memory with warnings, and
ctl model resources sets both for a stored model.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 17:43:13 +02:00
Ilker ManapandClaude Opus 5.5 62657f4c2a Release v0.10.4: CPU threads, tracing, and the rest of the review
Release / notes (push) Successful in 3s
CI / test (push) Successful in 4m57s
Release / build (push) Successful in 4m11s
CPU replicas start on half the machine's CPUs, not all of them: two busy
llama.cpp servers each spinning on all 8 vCPUs fell to 0.8 tok/s apiece,
where at 4 threads they keep 5.5. llamacpp.threads sets it per node.

Inference requests can be traced over OTLP/HTTP (tracing.endpoint), with
admission, schedule and upstream spans and traceparent sent to the engine.

Security: onboarding no longer puts the cluster root in the system trust
store and removes it where it was; the mirror's mutual-TLS API can be
limited to named certificates (--api-client); init.log no longer keeps the
administrator's password; a warning when auth is off on a public listen;
the recorded actor is sanitised and sudo-aware; Ray's dashboard is off, and
the remaining engine gaps are documented.

reconciler.go and agent.go are split by topic, the status mapping of each
failure is tested, and the plan's checklist is brought up to date.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 04:00:35 +02:00
Ilker ManapandClaude Opus 5.5 116e75f8cb Upgrade the cluster with one command, over the agents' own streams
Release / build (push) Successful in 1m5s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m42s
Every upgrade so far has been a script of mine copying a binary to each
machine over ssh, restarting it and polling until it came back — which is
the job, but not something the product could do, and which needed ssh into
every node.

`sudo llmgw ctl upgrade VERSION` on the control plane does it. This machine
first: the release for its platform, checked against the release's
checksums, the old binary kept beside it, llmgw-server restarted, and a wait
until every node that was ready has reconnected. Then each node in turn: the
control plane tells its agent the version over the stream it already holds;
the agent fetches and checks the release itself, swaps it in with a rename,
answers, and exits with status 75 so systemd starts the new binary. Engines
keep running, as for any agent restart, and are adopted again.

A node is done when it is back on the new version, ready, with as many ready
replicas as before; one that is not within the timeout stops the run before
the next node, so a bad release reaches one machine. Nodes already on the
version are skipped, so running it again resumes.

The fetch-and-verify is one package now, shared with onboarding's idea of
what a release version looks like.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 02:54:36 +02:00
Ilker ManapandClaude Opus 5.5 7bb360fc78 Measure two nodes' fabric rail by rail: llmgw ctl fabric-test
doctor checks a node alone, and every check can pass while two nodes still
cannot reach each other. ctl fabric-test A B asks B's agent for an
ib_write_bw server and A's for the client, adapter by adapter as NCCL pairs
them, one rail at a time, and grades each against its link's rate. It goes
through the agents' streams, so there is no SSH between nodes; the agent
installs perftest the first time, and names the RoCE v2 GID itself.

The result is read from perftest's table by its heading, not a column
number. Agents now list the requests they answer beyond the usual
(Register.features), and the control plane refuses a test on one too old to
answer instead of waiting minutes for it. The forwarding dispatcher carries
the new capability, and its test says so: a missing forward would have made
the command quietly unavailable in production.

Not yet run over real InfiniBand or RoCE; this cluster has neither.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 21:29:23 +02:00
Ilker ManapandClaude Opus 5.5 590e5b9f3e Ask a node only for engines it can install; doctor says when AVX2 is hidden
A control plane of this release asks every node for vLLM. An agent of the
last one does not know how to install it, and answered "unsupported", so a
cluster half way through its upgrade showed every node as failing. Agents
now say which engines they can install (Register.installable_engines), and
are asked for no others; an agent that says nothing is asked for llama.cpp
alone, as it always was.

doctor reports the CPU's vector extensions, and warns when AVX2 is missing,
with the fix for a VM: the test node for node add was one, on QEMU's default
CPU model, where llama.cpp runs several times slower and nothing else looks
wrong. node add shows it, since it runs doctor.

The changelog's unreleased section was two releases' worth of sections one
after the other, with two lists of fixes and the upgrade notes in the
middle. It is one entry now: largest changes first, the fixes together, and
every upgrade note at the end, with vLLM's default install and how to keep
your own.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 21:03:03 +02:00
Ilker ManapandClaude Opus 5.5 4c5a2fd6da Install vLLM on every GPU node that can run it, as llama.cpp is
vLLM was installed by hand and named in each agent's configuration. The
agent now installs it: a Python 3.12 environment built by uv, which is
itself a GitHub release checked against its checksum, from PyPI resolved as
it stood a day after that vLLM was published and installed with
--require-hashes. The resolved list is kept in the environment. The
environment is built relocatable under a scratch name and renamed into place
once it runs, like an archive.

The default is 0.30.0, whose flags were checked against the ones llmgw
builds. It is built for CUDA 13, so it needs driver 580 and a Turing GPU or
newer, and 25 GB free while it installs; a node lacking one says which, and a
node with no GPU skips it without being shown as a fault. gcc comes with it
because Triton compiles its launcher at the first model load. SGLang
installs the same way when the control plane names a version. llmgw doctor
reports whether vLLM can be installed on the node.

Checked by installing 0.30.0 this way on a test node: 7.7 GB, torch 2.13 for
CUDA 13.0, importing cleanly; the node has no GPU, so vLLM was not started.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 13:15:54 +02:00
Ilker ManapandClaude Opus 5.5 20479a2d5f Find multi-node NVLink domains, place across them, and show them as racks
A GB200 NVL72 rack or an NVLink Switch system is a scale-up island: its
GPUs reach each other over NVLink whatever node they are in. Each GPU
now reports its fabric from nvidia-smi -q - state, status, ClusterUUID,
CliqueId, bandwidth - at registration and once a minute after, since a
fabric manager restarting drops every card out at once; and each node
reports nvidia-imex and its channels, without which the domain's GPUs
cannot share memory across nodes. A single machine's NVSwitch, which
reports a cluster of zeros, is no domain.

A replica spanning nodes is placed inside one domain first, on nodes
whose every card is joined and whose IMEX is up, and its shards are
told NCCL_MNNVL_ENABLE=1; InfiniBand is the fallback.

/admin/nvlink draws each domain as a rack - the NVLink Switch band on
top, light running down a backplane, trays of GPU chips that glow when
joined, pulse red when not, turn amber on degraded bandwidth - and
lists what keeps a tray out. A node's page shows its GPU-by-GPU NVLink
matrix, each card's NUMA node and nearest adapter, and its domain and
IMEX. doctor checks joining, bandwidth, IMEX and its channel; metrics
llmgw_gpu_nvlink_joined and llmgw_imex_up; /api/v1/nvlink.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 09:07:59 +02:00
Ilker ManapandClaude Opus 5.5 5bbc619a31 Map every InfiniBand fabric, draw it, and place replicas by it
The control plane asks one node on each physical fabric to run
ibnetdiscover every five minutes and parses the text itself, so a
format the parser has not met is a control plane upgrade, not one on
every node. From the map it works out the switch tiers (leaf, spine,
core, by distance from the adapters), which adapter is which node's,
which cables trained below the rate the rest of their kind run at, and
whether the rails line up: every node's Nth adapter on its own leaf.

A page per fabric (/admin/fabrics) draws it: switches in their tiers,
adapters fanned out under their leaves, cables coloured by rate and
bundled, degraded ones marching red with a marker, and the cluster's
own adapters live - down ports glowing red, and particles running each
cable as fast as its adapter is sending and receiving, from the
counters. It zooms, pans, fades what a hovered switch or adapter does
not touch, opens a panel of detail on a click, colours by rail, finds a
node, and refreshes every fifteen seconds. The data is also at
/api/v1/fabrics/{id}/topology.

Placement reads the map: a replica spanning nodes takes nodes of one
leaf together when it can, and nodes whose cable trained short last.

Static assets are named by their content, as the stylesheet already
was.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 08:46:23 +02:00
Ilker ManapandClaude Opus 5.5 23648fa19a Watch each fabric's subnet managers, and keep a fabric one through a failover
A fabric's identity was its master subnet manager, which changes on a
failover: for the minute its ports took to agree, one fabric read as
two, and a name covering it as two separate fabrics. Each agent now
also lists every subnet manager on a fabric (saquery SMIR, through one
port per master, once a minute); managers listed together are one
fabric, named by the lowest GUID among them, so the identity survives
the master changing.

The same list says what a fabric can survive. The cluster page's new
subnet manager table, /api/v1/fabrics, doctor and metrics
(llmgw_fabric_master_sm, llmgw_fabric_subnet_managers,
llmgw_fabric_problems) show each fabric's master, priority and
standbys, and say when a fabric has no standby, its ports disagree on
the master, ports are up with no subnet manager (INIT), or the master
changed in the last hour. Every change of master a port reports is
kept, and a fabric's last day of them is listed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 07:47:03 +02:00
Ilker ManapandClaude Opus 5.5 77fe801892 Tell independent fabrics apart by their subnet manager, not their name
A fabric was what agent.yaml called it. With several independent
InfiniBand fabrics at a site, two given one name would have had a
replica spread across both, starting and then hanging on its first
collective. Each active InfiniBand port now reports the master subnet
manager it sees (sminfo, asked once a minute), which every port on one
fabric shares and ports on separate fabrics cannot; a RoCE port reports
the IP network of its RoCE v2 address. Ports also carry their node and
port GUIDs.

Placement divides a declared fabric by that physical identity and forms
a replica on one part only; a node whose adapters for one name are on
two fabrics is refused, and a node that cannot say is kept apart from
those that can. /api/v1/fabrics and the cluster page list each physical
fabric with its subnet manager, and say when one name covers two
fabrics, two names one fabric, an adapter is up but undeclared, or a
node cannot be checked. doctor reports each port's subnet manager, and
an agent on a node with adapters installs infiniband-diags when sminfo
is missing.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 07:29:16 +02:00
Ilker ManapandClaude Opus 5.5 51f1606b53 Resolve the management address from the servers list too
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 05:07:15 +02:00