Park a model that fails every time it starts, and say so everywhere

On the live cluster kev-0.5b — an architecture the nodes' llama.cpp did not
know — was started 155 times in eight hours, failing the same way each time,
and phi-4, which fitted on no node, logged "no placement" 250 times in two
days. Neither showed up anywhere an operator looked: the backoff never gave
up, no metric said a model was failing or unplaceable, and the one alert for
a model with nothing serving could not fire, because its series was published
only while a replica was ready.

After five failures in a row a model is parked: not placed again until its
definition changes — a new source or profile — or `llmgw ctl retry` is run.
Status, the model list and the admin API show it with its last error. The
placement warning is logged when its reason changes or every ten minutes.

A request for a parked or unplaceable model is answered at once with why,
rather than "loading; retry shortly", which a client retries forever. Each
request still clears the previous placement result and triggers a fresh
attempt, so a model that fits once room is made is not refused on a stale
answer; the router waits for that attempt, not for its full activation
window.

Metrics now cover model health (desired, ready at 0 for what should run,
consecutive start failures, parked, unplaceable), engine installs per node,
leadership, and certificate expiry read from disk at each scrape, with alerts
for each. `ctl status` counts a model against what it should have now, so an
on-demand model that was asked for and fitted nowhere no longer reads as
satisfied.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

CHANGELOG.md

@@ -14,11 +14,520 @@ is only released once its **exit criterion** is met, not when its code lands.
 | v0.7.0 | — | Conversations, agent memory, disk management | released |
 | v0.8.0 | — | PostgreSQL, leader election, agent failover | released |
 | v0.9.0 | — | Managing people and the tenancy tree | released |
+| v0.10.0 | — | InfiniBand fabrics and NVLink, doctor, node add, engines installed by the nodes | released |
 | v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
 
 Until v1.0.0 the minor version is the breaking one: configuration formats and Go
 APIs change between minors without a deprecation period.
 
+## Unreleased
+
+Changes since v0.10.3, published as release candidates (v0.10.4-rc.N) until
+there are enough for v0.10.4.
+
+### Restarting replicas
+
+`llmgw ctl restart REPLICA... | --model M | --all` replaces replicas with new
+ones. Each replacement is started beside the replica it replaces, which keeps
+serving until the new one is ready; replicas of one model go one at a time.
+Where no second copy fits anywhere, the old replica is stopped after the drain
+grace period and the new one starts in its place. This is how running engines
+pick up the keys introduced in v0.10.3, an edited profile, or a new engine
+version.
+
+### A model that keeps failing is parked, and everybody is told
+
+A model whose replicas failed at every start was retried forever: kev-0.5b,
+whose architecture the nodes' llama.cpp did not know, was started 155 times in
+eight hours. After five failures in a row a model is now parked — not placed
+again until its definition changes or `llmgw ctl retry MODEL` is run — and
+`ctl status`, `ctl models` and the admin API say so with the last error.
+
+A request for a model that will not come up — parked, or fitting on no node —
+is answered at once with the reason, instead of "the model is loading; retry
+shortly", which a client would retry forever. Each request still makes the
+control plane try placing it afresh, so a model that fits once room is made is
+not refused on an old answer. "no placement for model" is logged when its
+reason changes or every ten minutes, not on every pass: it was 250 identical
+lines in two days for one model.
+
+### Metrics and alerts for what goes wrong
+
+New series: `llmgw_model_replicas_desired`, `llmgw_model_start_failures`,
+`llmgw_model_parked`, `llmgw_model_unplaceable`, `llmgw_node_engine_install`,
+`llmgw_control_plane_leader` and `llmgw_certificate_expiry_timestamp_seconds`.
+`llmgw_model_replicas_ready` is now published at 0 for every model that should
+be running; it was published only while a replica was ready, so
+ModelHasNothingServing could never fire. New alerts: ModelParked,
+ModelFailingToStart, ModelCannotBePlaced, EngineInstallFailed,
+ControlPlaneLeadership, CertificateExpiresSoon and CertificateExpiresVerySoon.
+
+`ctl status` counts a model against the replicas it should have now rather
+than its minimum, so an on-demand model that was asked for and fitted nowhere
+no longer reads as satisfied.
+
+### Fixes
+
+- `ctl` reported a change the API had queued (202 Accepted) as an error, as
+  `ctl restart` did on its first real use.
+
+### Releases
+
+Small fixes are published as release candidates of the next version —
+v0.10.4-rc.1, rc.2 — which sort after v0.10.3 and before v0.10.4. Their release
+notes come from this section, and they are marked as pre-releases. Onboarding
+treats a release candidate as a release and downloads it.
+
+## v0.10.3 — 2026-09-30
+
+Security fixes from a review of v0.10.2. Three of them were confirmed on the
+live cluster before being fixed.
+
+### Engines demand a key only the router holds
+
+Engines listened on every address of their node with no key. A request from a
+neighbouring node reached the model directly, past the gateway's
+authentication, quotas and accounting. Each replica's engine is now started
+with a key derived from the replica id and a secret in the store. The router,
+its metrics scrape, the agent's readiness check and `ctl bench` send it.
+llama.cpp and vLLM receive it through their environment, not their command
+line. Engines started before this keep running without a key until their
+replica is next started.
+
+### The mirror's management API answers only who it was told to
+
+Without `--api-allow`, the management API answered anyone who could reach it,
+including every node. It now answers this machine only; the control plane is
+let in with `--api-allow` or mutual TLS. **Upgrading a mirror set up by hand:
+add `--api-allow <control plane address>` to its unit.** Downloads started
+through the API go to http and https only, and never to loopback, private or
+link-local addresses (`--allow-private-sources` to permit that). Model,
+version and file names can no longer be `..` or start with a dot.
+
+### The admin listener refuses what a browser page could send
+
+The admin API has no login and trusts whoever reaches it. A page open in an
+operator's browser, reaching it over an ssh tunnel, could post a model
+definition. The listener now requires Host to be an address or localhost,
+refuses cross-site requests, and takes changes only as `application/json`.
+A profile's `env` may no longer set variables that make an engine run other
+code (`LD_*`, `PATH`, `PYTHON*` start-up variables, `BASH_ENV` and similar).
+They are refused where a model is defined and dropped again before an engine
+starts.
+
+### Fixes
+
+- The agent installs only engine versions that look like one, whatever the
+  control plane sent; a version of `../x` could have installed outside
+  `/opt/llmgw/engines`.
+
+## v0.10.2 — 2026-09-29
+
+- After `server init`'s screen closes, only the administrator's password and
+  the next steps are printed, not the whole run again under the screen that
+  just showed it. The whole run is kept in `/var/lib/llmgw/init.log`, root's
+  only, and printed in full when init fails.
+- The screens' spinner and text field are llmgw's own rather than from
+  bubbles, one dependency fewer. Every llmgw command still asks the terminal
+  for its colours as it starts, which bubbletea v1 does on purpose; a terminal
+  that never answers waits for its timeout.
+
+## v0.10.1 — 2026-09-29
+
+### `llmgw ctl fabric-test`: two nodes, rail by rail
+
+`llmgw doctor` checks a node on its own, and every check can pass while two
+nodes still cannot reach each other: a cable into the wrong switch, a
+partition key, a RoCE network with no route. `llmgw ctl fabric-test A B` runs
+`ib_write_bw` from each of A's adapters on a fabric to the same adapter on B,
+one rail at a time, through the agents — no SSH between nodes, and perftest is
+installed the first time — and grades each against its link's rate: ok at 80%,
+slow at 50%, fail below or when nothing got through.
+
+Agents now say which requests they answer beyond the usual (`features`), so
+one too old for the test is refused at once instead of waited for.
+
+Written against perftest's real output and tested with it, but not yet run
+over real InfiniBand or RoCE: this cluster has neither.
+
+### The mirror on its own machine, in two commands
+
+`llmgw mirror init --control-plane <address>` sets a machine up as the model
+mirror: the user, `/srv/models`, and a service serving files to every node on
+:8081 and the management API on :8082. `server init --mirror
+http://<mirror>:8082` uses it instead of running one beside the control plane.
+It took a unit written by hand and an edit to `server.yaml` before.
+
+The management API can fetch and delete models; `mirror serve --api-allow`
+makes it answer only the addresses given and the machine itself, and `mirror
+init` gives it the control plane's. With `--no-node`, the starter model is
+fetched and catalogued anyway, unpinned, and starts on the first node added.
+
+Tried on three empty VMs: mirror, control plane and a node, and a request
+answered by the starter model on the node.
+
+### `server init` asks, and `node add` takes several nodes
+
+- **`sudo llmgw server init`** in a terminal, without `--admin`, asks for the
+  administrator's email, the other names nodes will reach the machine by
+  (showing the addresses it already found), SQLite or PostgreSQL, and whether
+  to run the mirror, a node and the starter model here. Then its steps fill
+  in as a list, downloads updating in place, and the administrator's password
+  stays in a box until the end — and is printed again after, with the whole
+  run, so it is in the terminal's history. The last screen says how to add a
+  node and to back up the authority's key. `--plain`, or no terminal, prints
+  lines as before.
+- **`llmgw node add a b c`** adds each in turn and ends with which were added.
+  A fabric named for one node's adapters is recognised on the next by what the
+  adapters report, and not asked again.
+
+Both tried on empty VMs: init from the form to a starter model answering, and
+two nodes added to it in one command in 40 seconds.
+
+## v0.10.0 — 2026-09-29
+
+Independent InfiniBand fabrics and NVLink domains, told apart, drawn and
+placed on; a node checked by `llmgw doctor` and added by `llmgw node add`; vLLM
+installed by the nodes themselves; people in many places at once; and a chat
+page, editor setup and a model explorer for the people using it.
+
+### Independent InfiniBand fabrics, told apart and drawn
+
+A fabric was whatever `agent.yaml` called it, and nothing checked. At a site
+with several independent fabrics, two given one name would have had a replica
+spread across both — started, then hung on its first collective. Now:
+
+- **Identity.** Each active InfiniBand port reports the master subnet manager
+  it sees (`sminfo`) and every subnet manager on its fabric (`saquery SMIR`),
+  asked once a minute through one port per fabric. Ports whose managers are
+  listed together are one physical fabric, so a failover does not split one in
+  two. A RoCE port's fabric is the IP network of its RoCE v2 address. The agent
+  installs `infiniband-diags` where there are adapters and no `sminfo`.
+- **Placement** divides a declared fabric by that identity and forms a replica
+  on one part only. A node whose adapters for one name are cabled to two
+  fabrics is refused; a node that cannot say is kept apart from those that can.
+- **Subnet managers.** The cluster page lists each fabric's master, its
+  priority, its standbys and its master changes over the last day, and warns
+  when a fabric has no standby, when its ports disagree on the master, when a
+  port is up with no subnet manager (INIT), and when one name covers two
+  fabrics or two names one fabric.
+- **Ports while running.** The agent re-reads every InfiniBand and RoCE port on
+  each heartbeat: state, physical state, rate, error and data counters. A port
+  going down takes its fabric off the node within seconds, and back when it
+  returns — it used to wait for the agent to restart.
+- **Maps.** Every five minutes one node on each fabric
+  runs `ibnetdiscover`, and the control plane parses it. `/admin/fabrics` draws
+  the fabric: switches in their tiers, adapters under their leaves, cables
+  coloured by rate, degraded ones — trained below the rest of their kind —
+  marked in red, rail alignment, and the cluster's own adapters live, with
+  their traffic running along the cables. It zooms, pans, highlights what a
+  switch or adapter touches, and opens detail on a click. Placement prefers
+  the nodes of one leaf and puts nodes with a degraded cable last.
+
+### NVLink, on every build and across nodes
+
+- **The release binary sees NVLink.** It is built without cgo, so without NVML,
+  which was the only source of NVLink peers: on every NVLink machine the
+  portable agent reported none. Discovery now reads `nvidia-smi topo -m`: NVLink
+  peers, each card's NUMA node, and how near each network adapter is.
+- **GPUs are chosen by it.** Every set of the right size is scored: NVLink pairs
+  within it, then — for a multi-node shard — nearness to the fabric's adapters,
+  then NUMA nodes spanned. It used to grow one card's peers, which took cards
+  joined to that one but not to each other.
+- **Multi-node NVLink domains** — an NVL72 rack, an NVLink Switch system — are
+  found from each GPU's fabric state, ClusterUUID and CliqueId, with
+  `nvidia-imex` and its channels. A replica spanning nodes is placed inside one
+  domain first, on nodes whose every card is joined and whose IMEX is up, and
+  its shards get `NCCL_MNNVL_ENABLE=1`; InfiniBand is the fallback.
+  `/admin/nvlink` draws each domain as a rack of trays and says what keeps a
+  tray out. A node's page shows its GPU-by-GPU NVLink matrix.
+
+### Multi-node launches fixed before they meet hardware
+
+- **vLLM launches multi-node its own way by default.** An empty
+  `engines.vllm.launch_mode` meant `ray`, never checked against a real cluster;
+  it now means `mp`, every node running `vllm serve` with `--nnodes`,
+  `--node-rank` and `--master-addr`, the workers `--headless`, as vLLM
+  documents. The `mp` path used to give workers no rank and the leader's
+  address as their own. Set `launch_mode: ray` to keep Ray.
+- **NCCL bootstraps over the node's own interface.** The agent's default of
+  `eth0` meant it never looked for the interface holding its management
+  address, and the control plane sent one name for every node. A node on
+  `ens3` or `bond0` had NCCL bind nothing.
+- **RoCE** shards get `NCCL_IB_GID_INDEX`, the RoCE v2 entry with an IPv4
+  address; NCCL's default is a v1 link-local one that does not route.
+- A Ray worker now waits for its head before joining.
+- The agent unit sets `LimitMEMLOCK=infinity`, which RDMA needs.
+
+### `llmgw doctor`
+
+Checks a node for what llmgw needs of its GPUs and fabric, and says what is
+wrong and how to fix it, most serious first: NVSwitch without the fabric
+manager, GPUs not joined to their NVLink domain, IMEX, `nvidia_peermem` and
+`ib_uverbs`, each port's link and subnet manager, a fabric with no standby,
+RoCE without a v2 GID, the agent's declared fabrics against the adapters, the
+interface NCCL will use, the unit's memlock limit, and the NCCL environment a
+shard on each fabric would be given. `--json` writes every fact it read, to
+send back; it exits non-zero when something fails.
+
+It also says when the CPU offers no AVX2 — a VM left on QEMU's default CPU
+model, where llama.cpp runs several times slower and nothing else looks wrong —
+and whether vLLM can be installed on the node, and why not.
+
+### vLLM installs itself on GPU nodes
+
+The agent installs vLLM as it does llama.cpp: on every node with an NVIDIA GPU
+that can run it, with nothing done by hand. A Python 3.12 environment is built
+by uv — itself a GitHub release checked against its checksum — from PyPI,
+resolved as PyPI stood a day after that vLLM was published and installed with
+`--require-hashes`, so every node gets the same files whenever it is built. The
+resolved list stays in the environment as `llmgw-requirements.txt`.
+
+- The default is vLLM 0.30.0, which takes every flag llmgw passes. Set
+  `engines.vllm` in `server.yaml` to change it, or to `""` to keep your own.
+- It is built for CUDA 13: it needs driver 580 and a Turing GPU or newer, and
+  25 GB free while it installs. A node that lacks one says which on the
+  cluster page; a node with no GPU skips vLLM without a word.
+- gcc is installed with it, since Triton compiles its kernel launcher on the
+  first model load.
+- SGLang installs the same way, on GPU nodes, when `engines.sglang` names a
+  version.
+
+Checked by installing vLLM 0.30.0 this way on a node of the test cluster: 7.7 GB,
+torch 2.13 for CUDA 13.0, importing cleanly. That node has no GPU, so vLLM
+itself was not started there.
+
+A node is asked only for engines its agent says it can install. An agent older
+than this release says nothing, and is asked for llama.cpp alone, so a cluster
+half way through its upgrade does not show every node as failing vLLM.
+
+### `llmgw node add`: a node in one command
+
+Adding a node was a bash script from the repository, not installed anywhere,
+that needed the node's name, the control plane's address and a dozen flags.
+It is now part of the binary:
+
+```
+sudo llmgw node add ubuntu@10.0.0.21
+```
+
+run on the control plane. The node id is the machine's hostname and the
+address its agent dials is this machine's on the route to it, checked against
+the server certificate. It installs llmgw on the node — the release for its
+architecture, fetched and checked there, which works on RHEL too — runs
+`llmgw doctor` there and stops on a problem, issues and installs the
+certificate, writes `agent.yaml` and the unit, starts the agent, and waits for
+the node to join and install llama.cpp.
+
+In a terminal it is a checklist that fills in, and asks what a script cannot:
+which fabric each group of active adapters is on — grouped by the subnet
+manager or RoCE network they report — and whether to go on past a problem.
+Elsewhere it prints one line per step. Running it again on a node is safe.
+
+`llmgw node remove` forgets a node that is gone for good, which otherwise
+stays listed as lost until the control plane restarts; only a lost node, or an
+unreachable one with nothing placed on it, is removed. `llmgw ctl nodes` no
+longer lists vLLM as skipped on every CPU node.
+
+Tried on an empty VM, from the terminal screen and plain: joined with
+llama.cpp ready in under a minute, and 17 seconds to add it again.
+
+### People belong to many places, or to none
+
+A person was in exactly one project, chosen at approval, and could not use the
+gateway before that. Now:
+
+- **Memberships.** A person can be a member of any number of organizations,
+  teams and projects, or of none. A member of a team or organization may use
+  every project in it. `llmgw users join/leave`, and the people page adds and
+  removes them from either side. Approving no longer requires a project.
+- **Personal keys.** A key can belong to a person with no project. On
+  `/profile` a person chooses what a new key spends against: one of their
+  projects, or Personal. `llmgw keys create --for <person>` does the same.
+  `auth.personal` sets default limits for personal use, or turns it off.
+- **Limits for people.** `--scope user` limits a person across every key,
+  project and the chat page. `--per-member` on an organization, team or
+  project gives each member their own share there. Both are on the quotas
+  page, and are enforced alongside the tree, each in its own pool.
+- **The chat page** asks what to spend against when a person has more than one
+  choice, and checks it on every message.
+- **Spend is recorded with who spent it**, so a person's budget survives a
+  restart, personal use and the chat page included.
+
+Leaving a project revokes the keys the person made there, unless another
+membership still reaches it. A personal key keeps no conversations or notes:
+those are kept per project.
+
+### An organization's own page
+
+A member of an organization can be made its **manager** (`users join --org …
+--role manager`, or the star on the people page). A manager gets `/org`: the
+organization's usage over time by team, project or person, its teams and
+projects with limits and spend, and its members — read-only, and only what was
+spent in its projects. Administrators can open every organization's page.
+
+### The chat page
+
+- A model that is not running shows a progress bar while it loads — waiting
+  for a node, downloading, starting — and the message is sent once it is up,
+  rather than failing after 45 seconds with "the model is loading".
+- A finished reply is rendered as markdown, by the server under the model
+  cards' rules, with its maths typeset by KaTeX: "$5 and $10" stays prose.
+- Stop, while a reply streams; its tokens, speed and time to first token under
+  it; a system prompt and a temperature; and the conversation kept in the
+  browser across a reload — the gateway still keeps none.
+
+### Connect an editor, and the Responses API
+
+- The account page makes a key for **Continue**, **Codex** or **Cline** and
+  shows what the tool needs, filled in: Continue's assistant file, Codex's
+  provider and profile, Cline's four settings. Continue and Codex get a one-line
+  command (`curl -fsSL <gateway>/setup/<tool>?model=… | sh`) whose script holds
+  no secret — the key comes from the environment — keeps a copy of whatever it
+  replaces, and leaves the rest of the file alone.
+- **`POST /v1/responses`** translates the Responses API, which Codex speaks, to
+  chat completions and back — instructions, messages, function and custom
+  tools with their calls and outputs, usage, streamed or not — down the same
+  path as any request.
+
+### Your own use
+
+The account page (was "Your keys") shows the last thirty days of a person's own
+use and, for each place they spend, a bar for every budget that applies — their
+own, or a project's shared with others, said as such.
+
+### The mirror page
+
+- **Explore models** opens on what is trending on Hugging Face, with views for
+  new and most downloaded, filters for task and size, and each repository's
+  parameters, architecture (✓ where the nodes' llama.cpp loads it), context,
+  base model and license. "Only what these nodes can run" is on by default.
+- A repository opens beneath its row with its model card, and a download
+  started there starts in place, its progress followed in the table above.
+- What the mirror holds is grouped by model family.
+- An importance matrix published beside a model's quantisations is no longer
+  offered as a model.
+
+**How a model runs.** Each model a mirrored file is serving now shows how it runs — always on, on
+demand, or stopped — and it can be changed there, without the CLI. The serve
+form asks the same: on demand loads the model when somebody asks for it and
+stops it after 15 minutes unused. A model declared in `server.yaml` shows its
+mode and says to change it there, because the file is read on every start.
+
+**Weights the nodes cannot load are not fetched.** `kev-0.5b` was a GGUF whose architecture, `ggmlc`, llama.cpp does not know. It
+was downloaded, catalogued and placed, and failed on every node it was tried on.
+The architecture was in the first kilobytes of the file all along. Now:
+
+- The mirror page's registry listing shows each file's architecture and, for
+  one the nodes' llama.cpp cannot load, says so instead of offering to fetch
+  it. A vision projector (`mmproj`) is marked as not a model on its own.
+- A fetch reads the file's header first — a range request, not a download —
+  and refuses one that cannot be served, unless "fetch anyway" is ticked.
+- Serving from the mirror and `llmgw ctl model add` refuse the same.
+
+The list of architectures is llama.cpp's own (`LLM_ARCH_NAMES`) at the build
+llmgw pins, b10991: 152 of them. A different build configured in `engines:` is
+not refused, only not checked.
+
+### Each node's CPU and memory, and a page per node
+
+The agent reads `/proc` between heartbeats: CPU across all cores, load, memory
+used and available, and each engine's cores and resident memory. The cluster
+page's node table shows CPU and RAM as bars; each node has a page with its
+current load, its engines, its fabric ports, its GPUs, and an hour of charts
+from Prometheus.
+
+### The admin pages
+
+- The cluster page leads with capacity, nodes, models and replicas, then usage;
+  it refreshes in place instead of reloading, and its usage table names people
+  and projects instead of ids.
+- The usage page's legend gives each line's total, not its last hour.
+- The people page keeps an account on one line and its rarer actions in a menu.
+- Every page fits a phone: a wide table scrolls in its own box.
+- Static files are named by their content, so a deploy's new styles and
+  scripts reach browsers at once instead of a day later.
+
+**One menu on every page.** Each page drew its own menu, and no two agreed: different links, in a different
+order, the page you were on sometimes listed and sometimes not. Every signed-in
+page now draws the same one — Cluster, People, Quotas, Usage, Mirror on the
+left for an administrator, Chat, Account and Sign out on the right for
+everyone — with the current page marked. The chat and key pages name the
+person's project by its path rather than its id.
+
+**The quotas page says what actually applies.** A branch that sets no limit of its own used to read "—" in every column while
+its organization held it to sixty requests a minute. Each limit now shows the
+value it is held to from above, muted and marked ↑, with where it comes from on
+hover; a larger value set below a stricter one says what caps it, for rate and
+token limits as well as budgets. Spend and budget share a column with a meter
+that turns red past 90%. The tree is indented like the people page, opening a
+form or saving one no longer reloads the page, and the explanations moved under
+"How limits combine". Projects are placed under their team by id rather than by
+the start of their path, which misplaced them when two organizations shared a
+name.
+
+### The project's website, in every gateway
+
+`/about` is llmgw's own page: what it does, InfiniBand and NVLink, installing,
+the reverse proxy configurations, and the release's files to download. It is
+served without a key like `/`, whose footer links to it, and says nothing about
+the installation. Its fonts are in the binary, so it makes no request elsewhere.
+The published website is rendered from the same template.
+
+### Metrics
+
+`llmgw_node_cpu_utilization_percent`, `llmgw_node_cpus`, `llmgw_node_load`,
+`llmgw_node_memory_used_bytes`, `llmgw_node_memory_available_bytes`,
+`llmgw_shard_cpu_cores`, `llmgw_shard_memory_bytes`, `llmgw_ib_port_active`,
+`llmgw_ib_port_rate_gbps`, `llmgw_ib_port_counter_total`,
+`llmgw_fabric_master_sm`, `llmgw_fabric_subnet_managers`,
+`llmgw_fabric_problems`, `llmgw_gpu_nvlink_joined`, `llmgw_imex_up`.
+
+### Fixes
+
+- PostgreSQL queries with an apostrophe in a comment failed: the placeholder
+  rewriting read it as the start of a string. Comments are now skipped.
+- Chart axes read "20.0BB" for sizes and 0% for a quiet CPU.
+- Usage by team, project or organization, and the spend on the quotas page,
+  left out every request made on the chat page: they were joined through the
+  key, and the chat page has none. They now count where a request was spent.
+- `/org` was answered with "no api key was provided" for a signed-in browser;
+  a test now drives every session page through the authentication middleware.
+
+### Upgrading
+
+Upgrade the control plane first, then the agents: an older agent is served as
+before and simply reports none of the new measurements. Existing agent units
+keep their old limits until `LimitMEMLOCK=infinity` is added to them. A node
+that set `mgmt_interface` keeps it; one that did not now uses the interface
+holding its management address rather than `eth0`. Multi-node vLLM replicas
+start with `mp` unless `launch_mode: ray` is set.
+
+The store migrates on its first start. Migration 0010 turns everyone's project into a membership and
+attributes past usage to people where a key had an owner. On SQLite it rebuilds
+the keys table; it was run against a copy of a production PostgreSQL store
+before release. `llmgw users move` is gone: use `join` and `leave`.
+
+vLLM is installed on every GPU node that can run it, once its agent is
+upgraded: about 8 GB each, from PyPI. Set `engines.vllm: ""` in `server.yaml`
+before upgrading to keep vLLM your own. `deploy/bootstrap/onboard-node.sh` is
+replaced by `llmgw node add`; it still works, for a machine holding the
+authority but not this llmgw.
+
+## v0.9.2 — 2026-09-28
+
+### A new control plane serves a model by itself
+
+`llmgw server init` now also runs a model mirror on the same machine, makes the
+machine a node of its own control plane, and fetches Qwen2.5 0.5B into the
+mirror and serves it there — on the machine's NVIDIA GPU when `nvidia-smi` finds
+one (the node installs the CUDA build of llama.cpp), on CPU otherwise, with a
+CPU profile as the fallback. An NVIDIA card on the PCI bus with no working
+driver is reported. `--no-mirror`, `--no-node` and `--no-model` leave these out,
+and every step is skipped on a rerun when it is already done.
+
+Not yet run end to end on a real machine.
+
 ## v0.9.1 — 2026-09-28
 
 ### The mirror page is one line per model

README.md

@@ -1,7 +1,7 @@
 # llmgw
 
-A single-binary LLM gateway for clusters with multiple InfiniBand fabrics and
-heterogeneous GPUs.
+A single-binary LLM gateway for clusters with multiple InfiniBand fabrics,
+NVLink domains and heterogeneous GPUs.
 
 It serves LLMs running on your own GPU servers (vLLM / SGLang) behind one
 OpenAI-compatible endpoint. Design document: [llmgw-plan.md](llmgw-plan.md).
@@ -49,15 +49,23 @@ a header would let any client promote itself into the top priority band.
 
 One thing is worth saying plainly:
 multi-node replicas are proven on a simulated cluster and on real hardware
-without a fabric. The InfiniBand, NVML and NCCL paths are written against the
-formats and failure modes those tools actually have, but they have not yet run
-on a machine that has them.
+without a fabric. The InfiniBand, NVLink and NCCL paths — telling fabrics apart
+by their subnet managers, mapping them, finding NVLink domains, launching across
+nodes — are written against the formats and failure modes those tools actually
+have and tested against them, but they have not yet run on a machine that has
+them. On such a machine, run `llmgw doctor --json` first and keep the report;
+[docs/fabrics.md](docs/fabrics.md) says what to look at after that.
 
 ## Documentation
 
-- [docs/install.md](docs/install.md) — installing from nothing: control plane,
-  nodes, engines, weights, first request
-- [docs/quickstart.md](docs/quickstart.md) — the short path
+- [docs/quick-install.md](docs/quick-install.md) — installing on servers, in
+  six steps: start here
+- [docs/install.md](docs/install.md) — the same from nothing, with every option
+  explained: control plane, nodes, engines, weights, first request
+- [docs/quickstart.md](docs/quickstart.md) — one workstation, from source
+- [docs/fabrics.md](docs/fabrics.md) — InfiniBand fabrics and NVLink: what
+  llmgw needs of them, checking a node with `llmgw doctor`, and what to look at
+  when a replica spanning nodes will not run
 - [docs/choosing-models.md](docs/choosing-models.md) — which models to run,
   how many of each, and where the weights should live
 - [docs/data.md](docs/data.md) — what llmgw stores, what it does not, and
@@ -91,6 +99,7 @@ llmgw ctl replicas --model llama     # replicas and their shards
 llmgw ctl explain-placement llama    # why a model has not been placed
 llmgw ctl drain gpu-a-01             # move work off a node, one replica at a time
 llmgw ctl cordon gpu-a-01            # stop new placement, leave what runs
+sudo llmgw doctor                    # on a node: what its GPUs, NVLink and fabric need
 ```
 
 `ctl` talks to the admin port (`:9090` by default), not the client-facing one.

api/proto/llmgw/v1/agent.proto

@@ -24,6 +24,8 @@ message AgentMessage {
     Heartbeat heartbeat = 2;
     ShardStatus shard = 3;
     LogsReply logs = 4;
+    FabricScanReply fabric_scan = 5;
+    FabricTestReply fabric_test = 6;
   }
 }
 
@@ -37,6 +39,8 @@ message ControlMessage {
     RemoveWeights remove_weights = 5;
     Provision provision = 6;
     LogsRequest logs = 7;
+    FabricScanRequest fabric_scan = 8;
+    FabricTestRequest fabric_test = 9;
   }
 }
 
@@ -114,6 +118,21 @@ message Register {
   // meant "an agent too old to say" and constrained nothing — so a freshly
   // built node with no engine installed yet was a candidate for every model.
   bool engines_probed = 14;
+  // imex is the service that lets a multi-node NVLink domain share memory.
+  IMEX imex = 15;
+
+  // installable_engines are the engines this agent knows how to install
+  // itself. The control plane asks it for no others. Empty is an agent older
+  // than the field, which could install llama.cpp and nothing else: asked for
+  // vLLM, it answered "unsupported", and every node of a cluster half way
+  // through an upgrade showed as failing.
+  repeated string installable_engines = 16;
+
+  // features are the requests this agent answers beyond the ones every agent
+  // does: "fabric-test". An agent that does not know a request ignores it, so
+  // without this the control plane would wait minutes for an answer that is
+  // never coming.
+  repeated string features = 17;
 }
 
 // Host describes the operating system and processor of a node.
@@ -207,6 +226,17 @@ message GPU {
   // Zero when the driver is too old to report it, which is not the same as
   // idle: the metric is simply absent rather than wrong.
   int32 utilization_percent = 15;
+  // nic_paths is how far each network adapter is from this card, by HCA
+  // name: PIX, PXB, PHB, NODE or SYS, from nvidia-smi topo.
+  map<string, string> nic_paths = 16;
+  // The card's multi-node NVLink membership, from nvidia-smi -q: state is
+  // Completed once the fabric manager has joined it; cluster_uuid and
+  // clique_id name the domain it joined.
+  string fabric_state = 17;
+  string fabric_status = 18;
+  string cluster_uuid = 19;
+  string clique_id = 20;
+  string fabric_bandwidth = 21;
 }
 
 message FabricAttachment {
@@ -231,6 +261,102 @@ message LocalWeights {
   bool shared = 3;
 }
 
+// NodeLoad is a node's use of its own CPU and memory, and each engine's share.
+//
+// Carried on the heartbeat rather than scraped from the node: the control
+// plane already exports the fleet from one place, and a node exporter per
+// machine would be a scrape target, a port and a firewall rule each.
+message NodeLoad {
+  // cpu_percent is the whole machine, 0 to 100 however many cores it has.
+  double cpu_percent = 1;
+  int32 cpu_count = 2;
+  // The kernel's run-queue averages over 1, 5 and 15 minutes.
+  double load1 = 3;
+  double load5 = 4;
+  double load15 = 5;
+  // memory_used_bytes is total less available: what the machine could not
+  // hand to a new process without evicting something.
+  int64 memory_used_bytes = 6;
+  int64 memory_available_bytes = 7;
+  repeated ShardLoad shards = 8;
+}
+
+// ShardLoad is one engine process group's use.
+message ShardLoad {
+  string shard_id = 1;
+  // cpu_cores is in cores, the way top counts a process: 2.5 is two and a
+  // half cores busy, which a percentage of the machine would hide on a large
+  // one.
+  double cpu_cores = 2;
+  int64 memory_rss_bytes = 3;
+}
+
+// IBPort is one InfiniBand or RoCE port's state and counters.
+message IBPort {
+  string hca = 1;
+  int32 port = 2;
+  // state is ACTIVE, INIT, DOWN...; phys_state tells a cable unplugged
+  // (Polling, Disabled) from a link up with no subnet manager (LinkUp and INIT).
+  string state = 3;
+  string phys_state = 4;
+  int32 rate_gbps = 5;
+  // link_layer is InfiniBand or Ethernet (RoCE).
+  string link_layer = 6;
+  // counters are cumulative, by sysfs name; data counters in bytes.
+  map<string, uint64> counters = 7;
+
+  // Identity: the adapter's node GUID and the port's own, as the fabric
+  // knows them.
+  string node_guid = 8;
+  string port_guid = 9;
+  // The subnet manager this port sees as master, on InfiniBand. Every port on
+  // one physical fabric sees the same one, which is what tells separate
+  // fabrics apart when their names are only what somebody typed.
+  string sm_guid = 10;
+  int32 sm_lid = 11;
+  int32 sm_priority = 12;
+  string sm_state = 13;
+  // sm_error says why sm_guid is empty on an active InfiniBand port:
+  // infiniband-diags missing, or no subnet manager answering.
+  string sm_error = 14;
+  // network is, on RoCE, the IP network of the RoCE v2 address: RoCE has no
+  // subnet manager, and the network is what its fabric is.
+  string network = 15;
+  string gid_prefix = 16;
+  // subnet_managers is every subnet manager the fabric's subnet
+  // administrator lists — master and standbys — asked through one port per
+  // fabric, so it is on that port only. It is what keeps a fabric one fabric
+  // through a failover: the master changes, the list does not.
+  repeated SubnetManager subnet_managers = 17;
+  string subnet_managers_error = 18;
+}
+
+// SubnetManager is one subnet manager on a fabric.
+message SubnetManager {
+  string guid = 1;
+  int32 lid = 2;
+  int32 priority = 3;
+  // state is master, standby, discovering or not active.
+  string state = 4;
+}
+
+// IMEX is the nvidia-imex service and its channels.
+message IMEX {
+  // state is systemd's word for it, empty where it is not installed.
+  string state = 1;
+  int32 channels = 2;
+}
+
+// GPUFabricState is one GPU's NVLink domain membership.
+message GPUFabricState {
+  int32 index = 1;
+  string state = 2;
+  string status = 3;
+  string cluster_uuid = 4;
+  string clique_id = 5;
+  string bandwidth = 6;
+}
+
 message Heartbeat {
   string node_id = 1;
   // gpu_memory_used_mb, indexed by GPU, catches a foreign process appearing on
@@ -271,6 +397,27 @@ message Heartbeat {
   // than "nothing to say", as for weights.
   bool provision_changed = 10;
   repeated ProvisionStatus provision = 11;
+
+  // load is how busy the machine is, sampled over the interval since the
+  // previous heartbeat. Absent on the first one, which has nothing to take a
+  // difference from, and from an agent on a system without /proc.
+  NodeLoad load = 12;
+
+  // ib_ports is every InfiniBand or RoCE port as it stands now, re-read on
+  // each heartbeat: a port that goes down after the agent started used to go
+  // unnoticed until the agent restarted.
+  repeated IBPort ib_ports = 13;
+  // fabrics is the fabrics this node can now carry traffic on, sent only when
+  // that changed since registration or the last heartbeat that said so — a
+  // port going down takes its fabric away, and one coming back returns it.
+  bool fabrics_changed = 14;
+  repeated FabricAttachment fabrics = 15;
+
+  // gpu_fabric is each GPU's NVLink domain membership as it stands now, sent
+  // once a minute: a fabric manager restarting drops every card out of its
+  // domain, and the node has to stop being offered work that spans it.
+  repeated GPUFabricState gpu_fabric = 16;
+  IMEX imex = 17;
 }
 
 // Provision is the software a node should have: which engines, at which
@@ -377,6 +524,10 @@ message StartShard {
   // profile_fp identifies the version of the profile, so the agent can say
   // which one a running shard was started from. The agent never interprets it.
   string profile_fp = 15;
+  // api_key is what the engine demands of every request, derived by the
+  // control plane from the replica id. The router sends it; nothing else
+  // that can reach the node's ports has it.
+  string api_key = 16;
 }
 
 message EngineProfile {
@@ -446,3 +597,55 @@ message LogsReply {
   // error says why the lines are absent: an unknown source, say.
   string error = 4;
 }
+
+// FabricScanRequest asks a node to map the fabric one of its ports is on,
+// with ibnetdiscover: every switch, every adapter and every link between
+// them. One node per fabric is asked, every few minutes; a fabric looks the
+// same from all of its ports.
+message FabricScanRequest {
+  string request_id = 1;
+  string hca = 2;
+  int32 port = 3;
+}
+
+// FabricScanReply is ibnetdiscover's output as it printed it. Parsed by the
+// control plane, so a format the parser does not yet know is a control plane
+// upgrade rather than one on every node.
+message FabricScanReply {
+  string request_id = 1;
+  string output = 2;
+  string error = 3;
+  int64 took_ms = 4;
+}
+
+// FabricTestRequest asks a node for one side of an RDMA bandwidth test
+// between two nodes: ib_write_bw, from perftest, over one adapter. The
+// server side is asked first and answers when the test is over; the client
+// dials it. What it proves is what a replica spanning nodes needs and nothing
+// on one node can show: that these two adapters reach each other through the
+// fabric, and how fast.
+message FabricTestRequest {
+  string request_id = 1;
+  // role is "server" or "client".
+  string role = 2;
+  string hca = 3;
+  int32 port = 4;
+  // peer is the server's management address, for the client. perftest
+  // exchanges its connection details over TCP before any RDMA.
+  string peer = 5;
+  // tcp_port is that exchange's port, the same on both sides.
+  int32 tcp_port = 6;
+  int32 seconds = 7;
+}
+
+// FabricTestReply is what one side measured.
+message FabricTestReply {
+  string request_id = 1;
+  // gbps is the average bandwidth, in gigabits a second.
+  double gbps = 2;
+  // output is the end of the tool's output, for when the number is not
+  // enough.
+  string output = 3;
+  string error = 4;
+  int64 took_ms = 5;
+}

deploy/bootstrap/onboard-node.sh

@@ -35,8 +35,9 @@ usage: $0 --node ID --addr HOST --server URL [options]
   --mgmt-if NAME    interface NCCL bootstraps over (default: the one holding
                     that address)
   --llamacpp-bin P  use this llama-server instead of the one the agent installs
-  --vllm-bin P      path to the vllm entry point (not yet installed by the agent)
-  --sglang-python P python interpreter of the SGLang virtualenv
+  --vllm-bin P      use this vllm instead of the one the agent installs on a
+                    GPU node (also set engines.vllm: "" in server.yaml)
+  --sglang-python P python interpreter of an SGLang virtualenv you installed
   --nvidia-smi PATH where nvidia-smi lives, when it is not on PATH. Containerized
                     driver installs routinely put it somewhere else.
   --ib-root DIR     the sysfs InfiniBand directory (default /sys/class/infiniband)
@@ -142,7 +143,7 @@ $SSH "set -e
 # that is not a release (a developer's, with commits past the tag) has no such
 # file, and is copied as before.
 TAG="$("$BINARY" --version 2>/dev/null | awk '{print $NF}')"
-if printf '%s' "$TAG" | grep -Eq '^v[0-9]+\.[0-9]+\.[0-9]+$'; then
+if printf '%s' "$TAG" | grep -Eq '^v[0-9]+\.[0-9]+\.[0-9]+(-rc\.[0-9]+)?$'; then
   say "installing llmgw $TAG from the release"
   $SSH "set -e
     arch=\$(dpkg --print-architecture)
@@ -226,6 +227,10 @@ ExecStart=/usr/local/bin/llmgw agent --config /etc/llmgw/agent.yaml
 KillMode=process
 Restart=on-failure
 RestartSec=2s
+# RDMA pins the buffers it registers, and NCCL registers a great deal: with
+# systemd's default 8 MB lock limit, an engine that drops the capability that
+# lets root ignore it fails its first collective over InfiniBand.
+LimitMEMLOCK=infinity
 StateDirectory=llmgw-agent
 
 [Install]

deploy/examples/agent.yaml

@@ -47,26 +47,24 @@ heartbeat_interval: 5s
 reconnect_base: 1s
 reconnect_max: 30s
 
-# Fabric labelling is explicit and there is no way around it. OpenSM's default
-# subnet prefix is fe80:: on every fabric, so a matching prefix proves nothing
-# and a wrong guess produces a replica that starts, passes health checks and
-# then hangs on its first all-reduce (plan §5.2).
+# Each fabric this node's adapters are on, by name. The name is checked against
+# what the adapters report — the subnet managers every port on one fabric sees —
+# so two independent fabrics given one name are placed as two and the cluster
+# page says so; give each its own. The rate is measured, not read from here.
 #
 # A node with no fabrics is its own single-node island: it can host single-node
-# replicas and will never be given a multi-node one.
+# replicas and will never be given a multi-node one. docs/fabrics.md has more.
 fabrics:
   - name: ib-a
     hcas: [mlx5_0, mlx5_1, mlx5_2, mlx5_3]
-    rate_gbps: 200
 
 discovery:
   # Worth setting where the driver lives outside the default paths, which
   # containerized driver installs often do.
   nvidia_smi: ""
   ib_root: /sys/class/infiniband
-  # NVML is the only source of NVLink topology, which is what keeps a
-  # tensor-parallel group off the PCIe path inside a node. It needs a binary
-  # built with `-tags nvml`; otherwise this silently falls back to nvidia-smi.
+  # NVLink topology comes from `nvidia-smi topo -m` on every build; a binary
+  # built with `-tags nvml` reads it from NVML instead, when this is set.
   prefer_nvml: true
 
 # Where this node keeps model weights besides /var/lib/llmgw-weights, which is
@@ -96,10 +94,11 @@ discovery:
 engines:
   vllm:
     bin: /opt/venvs/vllm/bin/vllm
-    # Which multi-node mechanism works depends on the vLLM version, and the
-    # flags have changed more than once. Check the version you deploy.
-    launch_mode: ray      # "ray" or "mp"
-    ray_bin: /opt/venvs/vllm/bin/ray
+    # How a replica spanning nodes is launched. mp, the default, is what vLLM
+    # documents today and needs nothing but vLLM; ray is for a vLLM too old
+    # for mp's multi-node flags, and needs Ray on every node.
+    launch_mode: mp       # "mp" or "ray"
+    # ray_bin: /opt/venvs/vllm/bin/ray
   sglang:
     # SGLang is launched as a module, which is its documented entry point.
     python: /opt/venvs/sglang/bin/python

deploy/monitoring/llmgw-rules.yml

@@ -42,10 +42,86 @@ groups:
         annotations:
           summary: "{{ $labels.model }} has no replica ready"
           description: >-
-            The model is placed and nothing is serving it. Ten minutes, because
-            a large model on CPU legitimately takes minutes to load. A model
-            that is deliberately stopped publishes no series at all, so this
-            cannot fire for one nobody asked for.
+            The model should be running and nothing is serving it. Ten minutes,
+            because a large model on CPU legitimately takes minutes to load. The
+            series is published at 0 for every model that should be running —
+            before v0.10.4 it was published only while a replica was ready, so
+            this could not fire at all. A model deliberately stopped, or an
+            on-demand one nobody is asking for, publishes nothing.
+
+      - alert: ModelParked
+        expr: llmgw_model_parked == 1
+        labels: { severity: critical }
+        annotations:
+          summary: "{{ $labels.model }} failed every time it started and is no longer retried"
+          description: >-
+            Five replicas in a row failed at start. Before this existed a model
+            whose weights its engine could not load was started 155 times over
+            eight hours. `llmgw ctl status` has the last error; once it is dealt
+            with, `llmgw ctl retry <model>`. Changing the model's definition
+            also retries it.
+
+      - alert: ModelFailingToStart
+        expr: llmgw_model_start_failures >= 2
+        for: 5m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.model }} has failed to start {{ $value }} times in a row"
+          description: >-
+            On its way to being parked. The engine's own words are in
+            `llmgw ctl logs <node> --source <shard>` for the next attempt, or in
+            the control plane's log for the last one.
+
+      - alert: ModelCannotBePlaced
+        expr: llmgw_model_unplaceable == 1
+        for: 10m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.model }} is wanted and fits on no node"
+          description: >-
+            Short of replicas, and the last attempt found nowhere with the
+            memory, cards or engine it needs. `llmgw ctl explain-placement
+            <model>` says what each node lacks. For an on-demand model, requests
+            for it are told this rather than asked to retry.
+
+      - alert: EngineInstallFailed
+        expr: llmgw_node_engine_install{state="failed"} == 1
+        for: 15m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.engine }} {{ $labels.version }} will not install on {{ $labels.node }}"
+          description: >-
+            The node takes no model needing it until it does. The agent retries
+            on its own; `llmgw ctl nodes` and the node's logs on the cluster
+            page say why it failed.
+
+      - alert: ControlPlaneLeadership
+        expr: sum(llmgw_control_plane_leader) != 1
+        for: 2m
+        labels: { severity: critical }
+        annotations:
+          summary: "{{ $value }} control planes believe they are placing work"
+          description: >-
+            None means nothing is placed or replaced; two means both are, each
+            into GPUs the other believes free. The lease is in the store: check
+            it is reachable from every control plane.
+
+      - alert: CertificateExpiresSoon
+        expr: (llmgw_certificate_expiry_timestamp_seconds - time()) < 30 * 86400
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.cert }} ({{ $labels.subject }}) expires in {{ $value | humanizeDuration }}"
+          description: >-
+            Certificates last a year and nothing renews them. When the agent
+            server's runs out every agent refuses to connect at once. Reissue
+            with `llmgw ctl cert` and restart the control plane.
+
+      - alert: CertificateExpiresVerySoon
+        expr: (llmgw_certificate_expiry_timestamp_seconds - time()) < 7 * 86400
+        labels: { severity: critical }
+        annotations:
+          summary: "{{ $labels.cert }} ({{ $labels.subject }}) expires in {{ $value | humanizeDuration }}"
+          description: See CertificateExpiresSoon; a week is not long.
 
       - alert: PoolSaturated
         expr: llmgw_pool_saturation > 1

deploy/proxy/README.md

@@ -10,6 +10,9 @@ put it behind a proxy that holds a publicly trusted certificate.
 
 Both are also attached to every release.
 
+The `/about` page shows both, from copies in `internal/web/proxy`; a test fails
+when those differ from these, so change them here and copy them over.
+
 Four things both files get right, and any other proxy has to as well:
 
 - **Streaming is not buffered.** Replies are server-sent events; a buffering

deploy/systemd/llmgw-agent.service

@@ -11,6 +11,10 @@ ExecStart=/usr/local/bin/llmgw agent --config /etc/llmgw/agent.yaml
 KillMode=process
 Restart=on-failure
 RestartSec=2s
+# RDMA pins the buffers it registers, and NCCL registers a great deal: with
+# systemd's default 8 MB lock limit, an engine that drops the capability that
+# lets root ignore it fails its first collective over InfiniBand.
+LimitMEMLOCK=infinity
 StateDirectory=llmgw-agent
 
 [Install]

docs/fabrics.md

@@ -0,0 +1,210 @@
+# InfiniBand fabrics and NVLink
+
+For a site whose GPU nodes are joined by InfiniBand or RoCE, by NVLink, or both
+— several independent fabrics, NVL72 racks, eight-GPU HGX boards. It says what
+llmgw does with them, what it needs from you before it can, how to check a node
+in one command, and where to look when a replica spanning nodes will not run.
+
+Read [install.md](install.md) first; this is the part of it that only matters
+once there is a fabric.
+
+> **Where this stands.** Every piece here was written against the real formats
+> of `sminfo`, `saquery`, `ibnetdiscover`, `nvidia-smi topo -m` and
+> `nvidia-smi -q`, and tested against them — but none of it has yet run on a
+> site with the hardware. Run `llmgw doctor --json` on two or three nodes first
+> and keep the reports: if a parser meets a format it does not know, that report
+> is what fixes it, and the fix is a control plane upgrade, not one per node.
+
+## What llmgw does with them
+
+**Two kinds of interconnect.** InfiniBand or RoCE joins nodes (scale-out).
+NVLink joins GPUs — inside one machine on an HGX board, or across the nodes of a
+rack on an NVL72 or an NVLink Switch system (scale-up). A replica spanning nodes
+is placed, in order of preference:
+
+1. inside one **multi-node NVLink domain**, if any has enough usable nodes:
+   tensor parallelism between its nodes then runs over NVLink;
+2. on one **physical InfiniBand or RoCE fabric**, preferring nodes under one
+   leaf switch and putting nodes with a degraded cable last.
+
+It is never split across two fabrics or two NVLink domains. Within a node, the
+GPUs a shard gets are the set with the most NVLink pairs, then the nearest to
+the fabric's adapters, then on one NUMA node.
+
+**A fabric has a name and an identity.** The name is what `agent.yaml` says; the
+identity is what the adapters report — on InfiniBand the subnet managers every
+port on one fabric sees, on RoCE the IP network of its RoCE v2 address. The name
+is how you address a fabric; the identity is how llmgw catches the name being
+wrong: two independent fabrics given one name are placed as two, and the
+cluster page says so.
+
+## Before a node joins
+
+On every GPU node:
+
+| What | Why | Check |
+|---|---|---|
+| NVIDIA driver | everything | `nvidia-smi` |
+| `nvidia-fabricmanager`, on an NVSwitch machine | without it CUDA does not start on those GPUs | `systemctl is-active nvidia-fabricmanager` |
+| `rdma-core`, `ib_uverbs` loaded | user-space RDMA at all | `ls /sys/class/infiniband`, `lsmod \| grep ib_uverbs` |
+| `nvidia_peermem` loaded | GPUDirect RDMA; without it NCCL copies every transfer through host memory | `lsmod \| grep nvidia_peermem` |
+| `infiniband-diags` | tells llmgw which fabric a port is on; the agent installs it if missing | `which sminfo` |
+
+On an NVL72 or NVLink Switch system, also:
+
+| What | Why | Check |
+|---|---|---|
+| `nvidia-imex` running, every node of the domain in `/etc/nvidia-imex/nodes_config.cfg` | GPUs in different nodes share memory through it | `systemctl is-active nvidia-imex` |
+| an IMEX channel (`options nvidia NVreg_CreateImexChannel0=1`) | an engine opens one | `ls /dev/nvidia-caps-imex-channels` |
+| every GPU joined to its domain | the fabric manager does this | `nvidia-smi -q \| grep -A6 Fabric` shows `State: Completed` |
+
+On each InfiniBand fabric:
+
+- **A subnet manager, and a standby.** Without one, ports come up to `INIT` and
+  stay there; without a standby, the fabric keeps its routes if the master
+  stops but cannot recover from any change after. llmgw watches this but does
+  not run one.
+- **Its own name** in `agent.yaml` on every node attached to it. Optionally, a
+  distinct `subnet_prefix` in each subnet manager's configuration, which makes
+  GIDs unique across fabrics; llmgw does not need it.
+
+## Declaring the fabrics
+
+In `/etc/llmgw/agent.yaml`, the adapters on each fabric, by name:
+
+```yaml
+fabrics:
+  - name: ib-compute
+    hcas: [mlx5_0, mlx5_1, mlx5_2, mlx5_3, mlx5_6, mlx5_7, mlx5_8, mlx5_9]
+  - name: ib-storage
+    hcas: [mlx5_4]
+```
+
+or `--fabric ib-compute:mlx5_0,mlx5_1,…` when onboarding ([node-onboarding.md](node-onboarding.md)).
+An adapter belongs to one fabric; a node may be on several. A replica on
+`ib-compute` is given only `ib-compute`'s adapters (`NCCL_IB_HCA`).
+
+The rate is measured, not taken from the file: a fabric's rate on a node is the
+slowest of its active adapters, because a collective runs at the speed of its
+slowest link.
+
+The agent checks the declaration against the adapters on every heartbeat. An
+adapter that is missing, or whose port is down, takes that fabric off the node
+within seconds — the node stops being offered multi-node work on it — and
+returns it when the port comes back.
+
+## Checking a node: `llmgw doctor`
+
+```
+sudo llmgw doctor                 # uses /etc/llmgw/agent.yaml when present
+sudo llmgw doctor --json > $(hostname)-doctor.json
+```
+
+It reads the node and says what is wrong and how to fix it, most serious first,
+and exits non-zero when anything fails:
+
+```
+FAIL  nvlink  this machine has NVSwitch but nvidia-fabricmanager is inactive: CUDA will not initialise on its GPUs
+              → systemctl enable --now nvidia-fabricmanager (its version must match the driver's)
+WARN  fabric  nvidia_peermem is not loaded: GPUDirect RDMA is off, and NCCL copies every transfer through host memory
+              → modprobe nvidia_peermem
+OK    gpu     8 GPUs: 8× NVIDIA H100 80GB HBM3
+OK    nvlink  NVLink groups: [0 1 2 3 4 5 6 7]
+OK    fabric  mlx5_0/1 ACTIVE at 400 Gb/s (InfiniBand), subnet manager 0x248a07030030f0c0 (LID 1, SMINFO_MASTER)
+OK    fabric  subnet managers of 0x248a07030030f0c0's fabric: 0x248a07030030f0c0 (master, priority 15), 0x248a07030030f1d4 (standby, priority 14)
+OK    agent   fabric ib-compute: mlx5_0,mlx5_1,… usable at 400 Gb/s
+OK    agent   NCCL bootstraps over ens3 (10.1.0.21)
+```
+
+followed by each GPU's NVLink peers and nearest adapter, each port's state,
+rate and errors, and the NCCL environment a shard on each fabric would be given.
+`--json` writes every fact it read as well as what it concluded — send that
+file when the report is not enough.
+
+## Checking two nodes: `llmgw ctl fabric-test`
+
+Every check on one node can pass while two nodes still cannot reach each other
+— a cable into the wrong switch, a partition key, a RoCE network with no route
+— and a replica spanning them then hangs on its first all-reduce. Before
+placing one, on the control plane:
+
+```
+llmgw ctl fabric-test gpu01 gpu02
+```
+
+```
+gpu01 → gpu02 on ib-compute, 41s
+
+RAIL              LINK      MEASURED    VERDICT  DETAIL
+mlx5_0 → mlx5_0   400 Gb/s  388.2 Gb/s  ok       97% of 400 Gb/s
+mlx5_1 → mlx5_1   400 Gb/s  241.0 Gb/s  slow     60% of 400 Gb/s: check the cable, and the card's PCIe width and NUMA node
+mlx5_2 → mlx5_2   400 Gb/s  0.0 Gb/s    fail     gpu01: ib_write_bw: Couldn't connect to 10.1.0.22:18517
+```
+
+It runs `ib_write_bw` (perftest, installed on a node the first time) from each
+of the first node's adapters on the fabric to the same adapter on the second —
+mlx5_0 to mlx5_0, the way NCCL pairs them — one rail at a time, through the
+agents, so no SSH between nodes is needed. Each rail is graded against its
+link's rate: **ok** at 80% or more, **slow** at 50%, **fail** below that or when
+nothing got through. `--fabric` picks one when the nodes share several,
+`--seconds` the length of each rail's run. The traffic is real: run it when the
+fabric is quiet.
+
+It measures RDMA between adapters, not GPUDirect or NCCL itself: a pass says
+the fabric carries traffic between these two nodes at its rate, which is what
+a first all-reduce needs and what nothing on one node can show.
+
+## Watching them
+
+- **Cluster page, Fabrics.** Each declared fabric and what it physically is;
+  a table of subnet managers per fabric — master, priority, standbys, the last
+  change of master — and every disagreement: one name on two fabrics, two names
+  on one, an adapter up but undeclared, ports up with no subnet manager, a
+  fabric with no standby, a recent failover.
+- **`/admin/fabrics`.** Each fabric's map, from `ibnetdiscover` run by one of
+  its nodes every five minutes: switches in their tiers, adapters under their
+  leaves, cables coloured by rate, degraded ones in red, rail alignment, and
+  the cluster's own adapters live with their traffic. Click anything for its
+  detail.
+- **`/admin/nvlink`.** Each multi-node NVLink domain as a rack of trays: which
+  GPUs are joined, which node's IMEX is down, which link runs degraded.
+- **A node's page.** Its ports with state, rate, errors and what they are
+  physically on; its GPU-by-GPU NVLink matrix; its load.
+- **Metrics**, for alerts:
+
+| Metric | Alert when |
+|---|---|
+| `llmgw_ib_port_active` | a declared port is 0 |
+| `llmgw_ib_port_counter_total{counter="symbol_error"}` and `link_downed` | rising |
+| `llmgw_ib_port_rate_gbps` | below its usual rate |
+| `llmgw_fabric_subnet_managers{state="standby"}` | absent or 0 |
+| `llmgw_fabric_master_sm` | its `sm_guid` changes (a failover) |
+| `llmgw_fabric_problems{level="fail"}` | above 0 |
+| `llmgw_gpu_nvlink_joined` | 0 |
+| `llmgw_imex_up` | 0 |
+
+## What an engine is given
+
+For a shard on an InfiniBand or RoCE fabric:
+
+| Variable | Value |
+|---|---|
+| `NCCL_IB_HCA` | that fabric's adapters on this node, and no others |
+| `NCCL_IB_DISABLE` | `0` (`1` for a single-node shard) |
+| `NCCL_SOCKET_IFNAME`, `GLOO_SOCKET_IFNAME` | the node's management interface: `mgmt_interface`, or the one holding its management address |
+| `NCCL_IB_GID_INDEX` | on RoCE only: the RoCE v2 entry with an IPv4 address |
+| `NCCL_MNNVL_ENABLE`, `NCCL_CUMEM_ENABLE` | `1`, for a replica inside a multi-node NVLink domain |
+
+A profile's `env` overrides any of them. vLLM launches a multi-node replica
+with its own multiprocessing (`engines.vllm.launch_mode: mp`, the default):
+every node runs `vllm serve` with its rank and the leader's address, the
+workers headless. `launch_mode: ray` is there for a vLLM too old for that.
+
+## When a replica spanning nodes will not run
+
+| It | Look at |
+|---|---|
+| is not placed | `llmgw ctl explain-placement <model>`: a fabric split by identity, a node whose IMEX is down, a node refused because its adapters are cabled to two fabrics |
+| starts and hangs on its first collective | `llmgw ctl fabric-test` between two of its nodes; the cluster page's fabric problems; `doctor` on each node — a port down, the wrong interface, RoCE without a v2 GID |
+| runs, slowly | `nvidia_peermem`; the map for a degraded cable; the node page's NVLink matrix for a group that crossed PCIe |
+| worked yesterday | the subnet manager table for a failover; `link_downed` on the node's ports |

docs/install.md

@@ -1,7 +1,8 @@
 # Installing llmgw from nothing
 
 Written for whoever is standing up the cluster: you have servers, you have root
-on them, and nothing is installed yet. By the end there is a control plane, a
+on them, and nothing is installed yet. For only the commands, read
+[quick-install.md](quick-install.md) first; this page explains them. By the end there is a control plane, a
 node in the fleet, and a model answering requests.
 
 It assumes Debian or Ubuntu and adjusts nothing for other distributions beyond
@@ -104,6 +105,17 @@ That is the whole step. In order, it:
    and promoted to administrator — the password is generated and **printed
    once**
 6. installs the systemd unit, starts it, and waits until `/health` answers
+7. runs a model mirror beside it (`/srv/models`, bytes on `:8081`, its
+   management API on loopback) and points the control plane at it
+8. makes this machine a node of its own control plane — on its GPUs when
+   `nvidia-smi` finds any, on CPU otherwise — and waits for it to install its
+   engine
+9. fetches Qwen2.5 0.5B into the mirror and serves it on this node, so the
+   first thing anyone opens `/chat` to find is a model that answers
+
+`--no-mirror`, `--no-node` and `--no-model` leave the last three out. An NVIDIA
+card with no working driver is reported rather than silently ignored; install
+the driver and run init again to serve on it.
 
 Add `--postgres` to keep accounts and keys in PostgreSQL rather than SQLite:
 init installs it and sets it up, and on a machine already running on SQLite it
@@ -307,32 +319,26 @@ hardware to it.
 
 ### NVLink, if your GPUs have it
 
-The published binary cannot see NVLink topology. NVML is the only source of it,
-its bindings need cgo, and the portable binary is built without cgo on purpose
-so it runs on a node whose libc is older than the builder's.
+The agent reads the topology from `nvidia-smi topo -m`: which cards NVLink
+joins, each card's NUMA node, and which network adapter is nearest each. A
+tensor-parallel group is then the set of cards with the most NVLink between
+them — on a PCIe box where bridges pair specific cards, a group straddling two
+bridges falls back to PCIe and runs several times slower, and nothing else
+would report it.
 
-Without that topology, a tensor-parallel group takes the lowest free GPU
-indices. On an SXM board with NVSwitch that is correct, because every card is
-equidistant. On a PCIe box where NVLink bridges pair specific cards, a group
-straddling two bridges falls back to PCIe and runs several times slower — and
-nothing reports it, because from the outside it is simply a replica that is
-slow.
-
-For those nodes, install the `-nvml` build published beside the portable one:
+On an NVSwitch machine, `nvidia-fabricmanager` has to be running, at the
+driver's version, or CUDA will not start on its GPUs. On an NVL72 rack or an
+NVLink Switch system, the GPUs of several nodes form one domain, and a replica
+can run tensor parallelism across its nodes — which needs `nvidia-imex` and an
+IMEX channel as well. [fabrics.md](fabrics.md) has the list, and `llmgw doctor`
+checks it:
 
 ```
-sudo install -m0755 llmgw-$VERSION-linux-amd64-nvml /usr/local/bin/llmgw
+sudo llmgw doctor
 ```
 
-It is dynamically linked, so it needs a libc no older than the builder's and the
-NVIDIA driver present. Check it took effect with:
-
-```
-llmgw ctl nodes --json | grep -c nvlink_peers    # 0 means the topology is unknown
-```
-
-`llmgw_gpu_nvlink_peers` in the metrics says the same thing across a fleet: a
-flat zero everywhere means the topology is unknown rather than absent.
+A `-nvml` build is still published beside the portable one; it reads NVLink
+from NVML directly, and needs a libc no older than the builder's.
 
 ### InfiniBand, if you have it
 
@@ -342,48 +348,60 @@ ibstat            # must list your HCAs with State: Active
 ls /sys/class/infiniband
 ```
 
-The agent checks what you declare against what the node has. An adapter that is
-not there, or that has no ACTIVE port — physically linked but with no subnet
-manager, which survives a reboot and looks like a software problem — takes that
-fabric out of service on this node, loudly, in the log. Without that check the
-replica starts, passes its health checks, and hangs on its first all-reduce.
+Each fabric needs a subnet manager, and should have a standby; `nvidia_peermem`
+should be loaded for GPUDirect RDMA. You name each fabric when you onboard the
+node, and llmgw checks the name against what the adapters report: every port on
+one fabric sees the same subnet managers, so two independent fabrics given one
+name are placed as two, and the cluster page says so. An adapter that is not
+there, or whose port goes down, takes that fabric off the node within seconds,
+and gives it back when the port returns.
 
-Fabric membership is configuration, not discovery, and there is no way around
-that: OpenSM's default subnet prefix is the same on every fabric, so a matching
-prefix proves nothing and a wrong guess produces a replica that starts, passes
-health checks, and then hangs on its first all-reduce. You name the fabric when
-you onboard the node, and two nodes are only in the same fabric if you said so.
+[fabrics.md](fabrics.md) covers declaring several fabrics, what the cluster and
+fabric pages show, and what to look at when a replica spanning nodes will not
+run. Run `llmgw doctor` on the node before it joins.
 
-### An engine
+### Engines
 
-Pick what you will actually serve. For a first cluster, llama.cpp is the least
-trouble and runs without a GPU.
+Nothing to do: the agent installs them once the node joins, under
+`/opt/llmgw/engines`, and checks each one runs before the node offers it.
 
-```
-# llama.cpp, from a release build
-sudo apt-get install -y libgomp1          # it will not start without this
-sudo mkdir -p /opt/llama.cpp
-# unpack a llama.cpp release into /opt/llama.cpp
-/opt/llama.cpp/llama-server --version
+| Engine | On | How |
+|---|---|---|
+| llama.cpp | every node | the release build for its processor, CUDA where there is a GPU, checked against GitHub's checksum |
+| vLLM | nodes with an NVIDIA GPU it can run on | a Python environment built by uv from PyPI, about 8 GB |
+| SGLang | only when you ask for it | the same way as vLLM |
+
+vLLM is resolved as PyPI stood a day after that vLLM was published, so every
+node gets the same dependencies whenever it is built, and installed with
+`--require-hashes` against the sha256 PyPI publishes. The resolved list stays in
+the environment as `llmgw-requirements.txt`. It needs:
+
+- **driver 580 or newer and a Turing GPU or newer** (compute capability 7.5):
+  vLLM is built for CUDA 13. A V100 cannot run it;
+- **about 25 GB free** under the engines directory while it installs;
+- **gcc**, which Triton builds its kernel launcher with. The agent installs it.
+
+A node with no GPU skips vLLM and says nothing about it. A node that has a GPU
+but cannot run it — an old driver, a V100, a small disk — says why on the
+cluster page. `llmgw doctor` checks the same before the node joins.
+
+The versions are the control plane's, in `server.yaml`:
+
+```yaml
+engines:
+  vllm: "0.30.0"      # the default; "" leaves vLLM to you
+  sglang: "0.5.20"    # SGLang only when named
 ```
 
-That `libgomp1` line is not padding. A `llama-server` binary missing it exists,
-is executable, and cannot start — and the node reports `ready` throughout.
+A changed version reaches every node when it next connects. To use an engine
+you installed yourself instead, set its version to `""` and name its path in
+the node's `agent.yaml` (`engines.vllm.bin`, `engines.sglang.python`).
 
-For vLLM, a virtualenv on the node:
+After changing a version, check it still takes the flags llmgw builds for it,
+which drift between releases:
 
 ```
-sudo apt-get install -y python3-venv
-sudo python3 -m venv /opt/venvs/vllm
-sudo /opt/venvs/vllm/bin/pip install vllm
-/opt/venvs/vllm/bin/vllm --version
-```
-
-Check the engine still takes the flags llmgw builds for it, which drift between
-releases:
-
-```
-llmgw ctl engine-check vllm --bin /opt/venvs/vllm/bin/vllm
+llmgw ctl engine-check vllm --bin /opt/llmgw/engines/vllm/0.30.0-cu13/bin/vllm
 ```
 
 ### Weights
@@ -470,40 +488,44 @@ bf16 is 141 GB and the same model at q4 is 43 GB.
 
 ## 5. Adding the node
 
-Two halves, because they need different access: building the machine needs the
-hypervisor, and issuing its identity needs the certificate authority. Where one
-person holds both — the common case — one command does it:
+On the control plane, which holds the certificate authority:
 
 ```
-llmgw-add-node \
-  --name llmgw-node01 --ip 10.0.0.141/24 \
-  --server https://10.0.0.140:7443 \
-  --cores 16 --memory 131072 --disk 200 \
-  --fabric ib-a:mlx5_0,mlx5_1 \
-  --llamacpp-bin /opt/llama.cpp/llama-server
+sudo llmgw node add ubuntu@10.0.0.141 --fabric ib-a:mlx5_0,mlx5_1
 ```
 
-**The VM name, the hostname, the node id and the certificate's common name are
-all the same string.** They do not have to be, and every time they are not,
-somebody spends an afternoon working out which of the four a message refers to.
+It logs in to the machine over SSH as that user, who needs passwordless sudo
+there, and:
 
-For a machine that already exists — bare metal, or a VM you built yourself — skip
-provisioning and issue the identity only:
+1. installs llmgw: the release for the machine's architecture, downloaded by
+   the machine and checked against the release's checksums, or this machine's
+   copy when it cannot download;
+2. runs `llmgw doctor` there and stops on any problem, printing the fix. Run it
+   again once fixed, or add `--ignore-checks`;
+3. issues the node's certificate and installs it, with the cluster's root,
+   trusted system-wide;
+4. writes `agent.yaml` — only what the node cannot find for itself — and the
+   systemd unit, and starts the agent;
+5. waits for the node to join and to install llama.cpp (`--wait` also waits
+   for vLLM, `--no-wait` for neither).
 
-```
-llmgw-onboard-node \
-  --node llmgw-node01 --addr 10.0.0.141 \
-  --server https://10.0.0.140:7443 \
-  --fabric ib-a:mlx5_0,mlx5_1 \
-  --llamacpp-bin /opt/llama.cpp/llama-server
-```
+In a terminal it draws this as a checklist and asks what a script cannot: which
+fabric each group of the node's active adapters is on, when `--fabric` was not
+given, and whether to go on past a problem doctor found.
 
-Onboarding needs SSH from the CA host to the node. It waits for SSH rather than
-failing on it, and learns the host key of a machine it has not seen — a new
-machine spends its first minute regenerating host keys and restarting sshd.
+**The node id is the machine's hostname**, and the address its agent dials is
+this machine's on the route to it; `--node` and `--server` override them. The
+VM name, the hostname, the node id and the certificate's common name being the
+same string saves an afternoon each time they would differ.
 
-It also checks that a configured engine actually runs, and says so loudly when
-it does not. That check exists because the failure is otherwise invisible.
+**SSH runs as root under sudo**, with root's keys. To use yours, log in to the
+control plane with `ssh -A` and run `sudo --preserve-env=SSH_AUTH_SOCK llmgw node
+add …`. It waits for SSH rather than failing on it, and learns the host key of
+a machine it has not seen — a new machine spends its first minute regenerating
+host keys and restarting sshd.
+
+Running it again on a node is safe: a fresh certificate and configuration, and
+the engines it runs are adopted by the restarted agent.
 
 Within a few seconds:
 
@@ -1037,6 +1059,8 @@ requests. On a slow model, set `timeouts.generation` higher.
   and what belongs on which port
 - `deploy/monitoring/README.md` — Prometheus, Grafana and the alerts, with what
   each one assumes about this system
+- `docs/fabrics.md` — InfiniBand fabrics and NVLink: what llmgw needs of them,
+  checking a node, and what to look at when a replica spanning nodes will not run
 - `docs/node-onboarding.md` — adding nodes in more detail, and what onboarding
   deliberately does not do
 - `docs/onboarding.md` — giving people accounts and keys
@@ -1052,14 +1076,16 @@ requests. On a slow model, set `timeouts.generation` higher.
   (step 2) or every request costs zero, and a budget is something the dashboard
   displays rather than something that stops anything. The quotas page says so
   where it can, which is not the same as being told before you rely on it.
-- **Engines are yours to install.** Onboarding verifies one runs; it installs
-  nothing.
+- **vLLM needs CUDA 13.** The version the agent installs needs driver 580
+  and a Turing GPU or newer. On older hardware, set `engines.vllm` to `""` and
+  install a vLLM built for your CUDA yourself.
 - **Two control planes have not been run for long.** The leader election and
   agent failover work and there is a pattern in `deploy/ha/`, but every cluster
   this has been run on in anger has had one. A single control plane is the
   well-trodden path.
-- **Multi-node replicas have not been run on real InfiniBand.** The scheduling,
-  the fabric invariant and the rendezvous are exercised against simulated
-  hardware only. Single-node placement on real GPUs is the well-trodden path;
-  if you are the first to run a TP group across real cards, expect to find
-  something.
+- **Multi-node replicas have not been run on real InfiniBand or NVLink.** The
+  scheduling, the fabric identity, the maps and the launch are written against
+  the real tools' formats and tested against them, not yet run on hardware that
+  has them. Single-node placement on real GPUs is the well-trodden path; if you
+  are the first to run a TP group across real cards, run `llmgw doctor --json`
+  on each node first and keep the reports.

docs/node-onboarding.md

@@ -4,24 +4,16 @@ Written for whoever runs the cluster.
 
 ## The short version
 
-Where one person holds both the hypervisor and the certificate authority —
-which is the common case — one command does the whole thing in about
-45 seconds:
+On the control plane:
 
 ```
-llmgw-add-node \
-  --name llmgw-node05 --ip 10.0.0.145/24 \
-  --server https://10.0.0.140:7443 \
-  --cores 8 --memory 32768 --disk 60 \
-  --llamacpp-bin /opt/llama.cpp/llama-server
+sudo llmgw node add ubuntu@10.0.0.145
 ```
 
-The VMID defaults to the last octet of the address, so `qm list` and
-`llmgw ctl nodes` stay in the same order. `--provision-only` and
-`--onboard-only` split it back into the two halves below when the two kinds
-of access belong to different people.
-
-The rest of this page is what that command does, and what it still does not.
+That is all a machine that already exists needs — bare metal, or a VM you
+built. [install.md](install.md#5-adding-the-node) says what it does. On
+Proxmox, the machine itself can be built first by the script below, on the
+hypervisor.
 
 ## The two halves
 
@@ -54,41 +46,31 @@ their working tree.
 
 ## 2. The identity
 
-On the host that holds the certificate authority — this is the part that cannot
-be delegated, because it issues the credential by which the control plane
-believes the node is who it says it is:
+On the control plane, which holds the certificate authority — this is the
+part that cannot be delegated, because it issues the credential by which the
+control plane believes the node is who it says it is:
 
 ```
-llmgw-onboard-node \
-  --node llmgw-node04 \
-  --addr 10.0.0.144 \
-  --server https://10.0.0.140:7443 \
-  --fabric ib-b
+sudo llmgw node add ilker@10.0.0.144 --fabric ib-b:mlx5_0,mlx5_1
 ```
 
-It issues a client certificate whose common name is the node id, installs it
-along with the CA root, trusts that root system-wide so other tooling on the
-machine stops treating llmgw's certificates as strangers, installs the binary
-and the unit, and starts the agent. The node appears in `llmgw ctl nodes` within
-a few seconds.
+It checks the machine with `llmgw doctor`, issues a client certificate whose
+common name is the node id (the hostname), installs it along with the CA root,
+trusts that root system-wide, installs the binary and the unit, starts the
+agent, and waits for the node to join.
 
-If it does not, the control plane's log says why. A certificate whose common
-name does not match the claimed node id is refused by design — that check is
-what makes a node's identity something it proves rather than something it picks.
+If it does not join, the agent's log on the node says why. A certificate whose
+common name does not match the claimed node id is refused by design — that
+check is what makes a node's identity something it proves rather than
+something it picks.
 
-**It needs ssh from the CA host to the node.** Provisioning copies that host's
-`authorized_keys` into the new machine, so a node built by the script above
-already satisfies it; an older machine may need the key added.
+**It needs SSH from the control plane to the node.** Provisioning copies the
+hypervisor's `authorized_keys` into the new machine; with `ssh -A` to the
+control plane and `sudo --preserve-env=SSH_AUTH_SOCK`, the same key that
+reaches the hypervisor reaches the node.
 
-It waits for ssh rather than failing on it, and learns the host key of a machine
-it has not seen (`accept-new`, which still refuses a *changed* key — the risk
-being guarded against is a substituted machine, not a new one). Both are needed
-to run the halves back to back: for the first minute of its life a new machine
-is still regenerating host keys and restarting sshd, so it refuses connections
-it accepted a moment earlier.
-
-It also checks that a configured engine actually runs, which is cheap here and
-expensive later — see below.
+`deploy/bootstrap/onboard-node.sh` is the script this replaced, for a machine
+holding an authority but not a current llmgw.
 
 ## The CPU model matters, and its default is wrong
 
@@ -116,32 +98,23 @@ The trade is live migration: a VM pinned to the host CPU cannot move to a
 machine with a different one. For a node whose job is arithmetic that is the
 right way round, and `--cpu` overrides it where it is not.
 
-## What onboarding does not do
+## Engines
 
-It gives a machine an identity and an agent. It does **not** install inference
-engines or model weights, and a node missing either looks healthy in
-`llmgw ctl nodes` right up until a replica is placed on it and the shard dies
-with "executable file not found".
+The script installs none, and does not need to: once the agent is connected,
+the control plane tells it which engines to have and it installs them itself
+— llama.cpp on every node, vLLM on every node with an NVIDIA GPU it can run
+on, SGLang where the control plane names a version. Each is checked to run
+before the node offers it, so a node never advertises an engine that cannot
+start. [install.md](install.md#engines) says what vLLM needs of the node.
 
-Onboarding does check that an engine you point it at can be executed, and says
-so loudly when it cannot. That check exists because the failure is invisible
-otherwise: `llama-server` on node05 was present, executable, and could not start
-for want of `libgomp1`, and the node reported `ready` in `llmgw ctl nodes` the
-whole time. Being told at onboarding beats finding out when a replica is placed
-and the reason is three logs away.
+`--llamacpp-bin`, `--vllm-bin` and `--sglang-python` are for an engine you
+installed yourself. For the agent to use yours rather than install its own,
+also set that engine's version to `""` under `engines` in `server.yaml`.
 
-So the node is not finished until:
-
-- the engines its catalog profiles name are installed (`vllm`, `sglang`,
-  `llama-server`), at the paths the agent configuration gives
-- the weights are on disk, at the `source` each model declares
-
-Both are deliberately outside this script. Engines are a platform decision with
-versions and drivers attached, and distributing weights is its own problem —
-hundreds of gigabytes that want a shared filesystem or a cache, not an scp from
-a laptop. llmgw will fetch weights for a model whose `source` is a URL, one
-copy per node, which covers the simple case; anything larger stays the
-operator's arrangement, and llmgw is content to find the files already there.
+Weights are the other thing a node needs. llmgw fetches them for a model whose
+`source` is a URL, one copy per node, which covers the simple case; anything
+larger — a shared filesystem, a cache — stays the operator's arrangement, and
+llmgw is content to find the files already there.
 
 The `sim` engine needs neither, which is what makes it useful for exercising
 everything else.
@@ -178,18 +151,24 @@ was cut short.
 
 ## Fabrics
 
-`--fabric` is how a node says which InfiniBand fabric it is attached to, and
-there is no way around stating it. OpenSM's default subnet prefix is `fe80::` on
-every fabric, so a matching prefix proves nothing and a wrong guess produces a
-replica that starts, passes health checks, and then hangs on its first
-all-reduce.
+`--fabric` is how a node says which InfiniBand fabric it is attached to. The
+name is checked against what the adapters report — every port on one fabric
+sees the same subnet managers — so two independent fabrics given one name are
+placed as two, and the cluster page says so; give each fabric its own name.
+The agent installs `infiniband-diags`, which that check needs, if it is missing.
+
+Before a node with a fabric joins, run `sudo llmgw doctor` on it: it checks the
+subnet manager, GPUDirect RDMA, the fabric manager and IMEX on NVLink machines,
+and the declared fabrics against the adapters. [fabrics.md](fabrics.md) has the
+rest.
 
 Omit it for a node with no fabric. Such a node is its own island: it can host
 single-node replicas and will never be given a multi-node one.
 
 Two adapters per fabric is the default; `--fabric ib-b:mlx5_2,mlx5_3` names them
 explicitly. An adapter must never appear under two fabrics — the agent refuses
-that configuration, because the consequence is the hang above.
+that configuration: a replica given adapters on two fabrics starts, passes its
+health checks, and hangs on its first all-reduce.
 
 ## Where the driver is
 

docs/onboarding.md

@@ -39,16 +39,40 @@ The landing page then links to it, and the rest is:
 
 ```
 llmgw users list --pending --store …
-ID                    EMAIL              NAME   STATE    PROJECT  SIGNED UP
-usr-k4mq…             ayse@example.com   Ayşe   pending  -        4m12s ago
+ID          EMAIL              STATE    ADMIN  MEMBER OF     SIGNED UP
+usr-k4mq…   ayse@example.com   pending  -      on their own  4m12s ago
 
-llmgw users approve usr-k4mq… --project default --by ilker --store …
+llmgw users approve ayse@example.com --by ilker --store …
+llmgw users approve ayse@example.com --project Acme/ml/training --store …   # or straight into a project
 ```
 
 That is the whole operator loop. From then on they sign in at `/login`, and
-`/profile` is where they create keys, see when each was last used, and revoke the
-ones they are done with. A key is shown once on the page that creates it and
-never again.
+their account page (`/profile`) is where they create keys, see when each was
+last used, revoke the ones they are done with, and see their own use over the
+last thirty days against every budget that applies to them. A key is shown once
+on the page that creates it and never again.
+
+### Connecting an editor
+
+The account page's **Connect your editor** makes a key for Continue, Codex or
+Cline and shows what the tool needs with that key, this gateway's address and a
+model already in it:
+
+- **Continue** — an assistant file of its own, `~/.continue/assistants/llmgw.yaml`,
+  so their `config.yaml` is not touched.
+- **Codex** — a provider and a profile in `~/.codex/config.toml`, run with
+  `codex --profile llmgw`; their default model stays as it was. Codex speaks the
+  Responses API, which the gateway translates at `/v1/responses`.
+- **Cline** — four settings to paste into its own screen, since it keeps them
+  in VS Code's secret storage where no file reaches.
+
+For Continue and Codex there is a one-line command,
+`curl -fsSL <gateway>/setup/<tool>?model=… | sh`. Its script is served without
+signing in and holds no secret — the key comes from the environment — and it
+keeps a copy of anything it replaces. Revoking the key disconnects the tool.
+
+How well an editor's agent mode works depends on the model: it needs one that
+calls tools well, which the coder models do and small general ones often do not.
 
 **This gateway sends no email.** There is no "your account was approved" message
 and no password reset, so tell people what to expect: they sign up, they tell
@@ -105,15 +129,69 @@ The tree is not fixed once made:
 llmgw org rename project Acme/platform/gateway api-gateway
 llmgw org move project Acme/platform/gateway --to Scania/ml
 llmgw org remove team Acme/old                 # only when nothing is in it
-llmgw users move ayse@example.com --project Acme/ml/research
 llmgw users delete ayse@example.com --yes      # signs out, revokes their keys
-llmgw keys list --all                          # every key, its project and owner
+llmgw keys list --all                          # every key, where it spends, and whose
 ```
 
-Moving a person takes the keys they made with them, so they stop spending the
-old project's budget; keys an operator issued in the project stay with it. A
-project that ever had a key cannot be removed, because usage history refers to
-it — rename it instead.
+A project that ever had a key cannot be removed, because usage history refers
+to it — rename it instead.
+
+### Where people belong
+
+A person can be a member of any number of organizations, teams and projects, or
+of none:
+
+```
+llmgw users join ayse@example.com --team Acme/ml          # every project in ml
+llmgw users join ayse@example.com --project Scania/research/vision
+llmgw users leave ayse@example.com --project Acme/ml/training
+```
+
+A member of a team or organization may use every project in it. Leaving revokes
+the keys the person made in projects they can no longer reach; a project they
+still reach through another membership keeps them.
+
+Someone who is a member of nothing still uses the gateway, on their own:
+personal keys, and the chat page, within their personal limits. When they make
+a key on `/profile` they choose what it spends against — one of their projects,
+or "Personal". An operator can do the same:
+
+```
+llmgw keys create laptop --for ayse@example.com                      # personal
+llmgw keys create ci --for ayse@example.com --project Acme/ml/training
+```
+
+Personal use can be limited by default, or turned off so people use the gateway
+only through their projects:
+
+```yaml
+auth:
+  personal:
+    rpm: 20
+    tpm: 50000
+    budget: 10        # whole units
+    period: month
+    # disabled: true
+```
+
+A personal key has no project, so it keeps no conversations or agent notes;
+those are kept per project.
+
+### Organization managers
+
+A member of an organization can be made its manager:
+
+```
+llmgw users join boss@acme.com --org Acme --role manager
+```
+
+or with the star on their organization membership on `/admin/users`. A manager
+gets an **Organization** link and a page at `/org`: usage over time by team,
+project or person, the teams and projects with their limits and spend, and every
+member. Only what was spent in that organization's projects is on it; what its
+people do elsewhere is not. The page is read-only — a manager is not an
+administrator of the cluster, and limits and memberships stay with those who
+are.
 
 `llmgw keys init` still works for provisioning scripts and is idempotent, but it
 will no longer move a team or project that already exists under another parent;
@@ -195,6 +273,20 @@ llmgw keys limits --store … --scope project --id Acme/platform/gateway \
   --budget 50 --period month              # 50 units a month
 ```
 
+Two more kinds of limit follow people rather than the tree:
+
+```
+# A person, everywhere: every key they hold, every project, the chat page
+llmgw keys limits --scope user --id ayse@example.com --budget 25
+
+# Each member of a project separately — a fair share, not one shared pool
+llmgw keys limits --scope project --id Acme/ml/training --per-member --rpm 20
+```
+
+A request is held to all of them at once: the key's own, its project's, team's
+and organization's, each per-member limit on those for its owner, and the
+owner's own. The strictest wins, and each is counted in its own pool.
+
 ## The landing page
 
 Someone who reaches the gateway without a key sees the landing page. Tell it who
@@ -314,9 +406,11 @@ page exists.
 ## Managing accounts from the browser
 
 `/admin/users` is the tree: organizations, teams and projects, each project
-opening onto its people and its keys. From there an admin approves, moves,
-suspends and removes people, revokes keys, and adds, renames, moves and removes
-organizations, teams and projects. A searchable list of every account is at the
+opening onto its members and its keys. From there an admin approves people
+(into a project or on their own), adds them to and takes them out of
+organizations, teams and projects, suspends and removes them, revokes keys, and
+adds, renames, moves and removes organizations, teams and projects. `/admin/quotas`
+sets limits on the tree, per member of any part of it, and per person. A searchable list of every account is at the
 bottom. The page works as plain forms; with scripts on, a change updates it in
 place and the projects you had open stay open.
 

docs/quick-install.md

@@ -0,0 +1,202 @@
+# Quick install
+
+From empty servers to a model answering requests, in six steps. This page
+has only the commands and what to check after each; [install.md](install.md)
+explains every choice.
+
+## What you need
+
+| | |
+|---|---|
+| **One control plane machine** | Linux, x86-64 or ARM64. Debian 12, Ubuntu 22.04+ or RHEL 10. No GPU needed; 2 GB RAM, 20 GB disk. |
+| **GPU nodes** (optional to start) | The same Linux, with the NVIDIA driver installed. vLLM needs driver 580 or newer and a Turing GPU or newer (not V100). |
+| **root** | on the control plane, and `sudo` without a password on every node. |
+| **SSH** | from the control plane to each node, as a user with passwordless `sudo` there. |
+| **Network** | nodes reach the control plane on 7443 and 8081; users reach it on 8080. The machines need internet access to download engines and models. |
+
+You do not install llama.cpp, vLLM, Python or CUDA. The nodes do that
+themselves once they join.
+
+## 1. Put llmgw on the control plane
+
+```
+VERSION=$(curl -fsSL https://gitea.mynodes.xyz/api/v1/repos/ilker/llmgw/releases/latest \
+  | grep -o '"tag_name":"[^"]*"' | cut -d'"' -f4)
+U=https://gitea.mynodes.xyz/ilker/llmgw/releases/download/$VERSION
+curl -fsSLO $U/llmgw-$VERSION-linux-amd64          # -linux-arm64 on ARM
+curl -fsSL $U/checksums.txt | sha256sum -c --ignore-missing
+sudo install -m0755 llmgw-$VERSION-linux-amd64 /usr/local/bin/llmgw
+llmgw --version
+```
+
+`sha256sum` must print `OK`. If it does not, the download is damaged; do not
+install it.
+
+## 2. Set up the control plane
+
+```
+sudo llmgw server init
+```
+
+It asks for your email and a few choices, then sets up everything on this
+machine, in a few minutes. In a script, give them as flags instead:
+`sudo llmgw server init --admin you@example.com`.
+
+- the service, its certificates and its configuration;
+- your administrator account. **The password is printed once. Copy it now.**
+- a model mirror, which downloads model files once for the whole cluster;
+- this machine as a first node, and a small model (Qwen2.5 0.5B) on it.
+
+If nodes will reach this machine by a name or address it does not know itself
+(a DNS name, a NAT address), add `--host` for each:
+
+```
+sudo llmgw server init --admin you@example.com --host llm.example.com
+```
+
+It is safe to run again. It keeps what exists and finishes what is missing.
+
+**The mirror on its own machine**, when it wants a larger disk than the
+control plane has: first on that machine, with llmgw installed as in step 1,
+
+```
+sudo llmgw mirror init --control-plane <control plane address>
+```
+
+then on the control plane `sudo llmgw server init --mirror http://<mirror
+address>:8082`. The mirror serves model files to every node, and takes
+orders only from the control plane. Untick "This machine as a node" in the
+form (or add `--no-node`) to keep models off the control plane; the starter
+model then starts on the first node you add.
+
+**Check:** open `http://<control-plane>:8080/chat`, sign in with the printed
+password, and ask something. The small model answers. Change the password on
+the account page.
+
+## 3. Add the GPU nodes
+
+On the control plane:
+
+```
+sudo llmgw node add ubuntu@10.0.0.21
+```
+
+Several at once: `sudo llmgw node add ubuntu@10.0.0.21 ubuntu@10.0.0.22 ubuntu@10.0.0.23`.
+
+Replace `ubuntu` with the user you log in to the node as, and `10.0.0.21` with
+its address. That user needs `sudo` without a password on the node.
+
+It logs in over SSH and, while you watch:
+
+1. installs llmgw on the node, the release for its processor, checksum checked;
+2. runs `llmgw doctor` there, and stops if the node has a problem. Each problem
+   comes with the command that fixes it. Fix it and run the same command again;
+3. gives the node its certificate and configuration, and starts its agent;
+4. waits for the node to join and install llama.cpp.
+
+The node's name in the cluster is its hostname. vLLM installs by itself after
+that, in a few more minutes on a GPU node (about 8 GB). The cluster page
+(`/admin`) shows its progress, or why the node cannot run it.
+
+**InfiniBand.** When the node's adapters are up, `node add` shows them grouped
+by the fabric they are on and asks for each fabric's name. Give every node on
+the same fabric the same name. In a script, give it instead:
+
+```
+sudo llmgw node add ubuntu@10.0.0.21 --fabric ib-a:mlx5_0,mlx5_1,mlx5_2,mlx5_3
+```
+
+Once two nodes on a fabric are in, check they reach each other at the link's
+rate before placing anything across them:
+
+```
+llmgw ctl fabric-test gpu01 gpu02
+```
+
+**If it cannot log in.** `sudo` runs SSH as root, with root's keys. To use your
+own keys instead, log in to the control plane with `ssh -A` and keep your
+agent through sudo:
+
+```
+sudo --preserve-env=SSH_AUTH_SOCK llmgw node add ubuntu@10.0.0.21
+```
+
+**Check:** `llmgw ctl nodes` lists the node. Running `node add` again on the
+same node is safe: it renews its certificate and configuration, and keeps what
+runs there.
+
+A machine retired for good is removed with `llmgw node remove <name>` once its
+agent is stopped.
+
+## 4. Add a model
+
+Open `/admin/mirror` and choose **Explore models**. It lists what is popular on
+Hugging Face, showing only what your nodes can run.
+
+1. Click a model to read its card.
+2. Pick one of its files, usually a `q4_k_m` for a first try, and click
+   **Fetch**. It is downloaded into the mirror once, for every node.
+3. When the download finishes, click **serve** on its row in the table above.
+4. In the form, choose how it runs: **always on**, or **on demand**, which
+   loads it on the first request and stops it after 15 minutes unused. Click
+   **Add to the catalog**.
+
+The model is placed on a node that fits it, and its status on `/admin` turns
+`ready`.
+
+**Check:** the model appears in the list on `/chat`. Ask it something.
+
+## 5. Give people access
+
+**A key for a program:** each person makes their own on the account page
+(`/profile`). The key is shown once.
+
+```
+curl http://<control-plane>:8080/v1/chat/completions \
+  -H "Authorization: Bearer $LLMGW_API_KEY" \
+  -H "Content-Type: application/json" \
+  -d '{"model": "<model-name>", "messages": [{"role": "user", "content": "hello"}]}'
+```
+
+Any OpenAI client works. Only the base URL changes:
+`http://<control-plane>:8080/v1`.
+
+**An editor:** the account page connects Continue, Codex or Cline. Pick the
+tool and a model, and it gives a one-line command that writes the
+configuration.
+
+**Other people's accounts:** they sign up at `/signup`, and you approve them on
+the people page (`/admin/users`).
+
+## 6. Put it behind HTTPS
+
+Port 8080 is plain HTTP. Before users outside a trusted network use it, put a
+reverse proxy in front with a real certificate. With Caddy, which gets the
+certificate by itself:
+
+```
+sudo apt-get install -y caddy          # Debian and Ubuntu
+curl -fsSL $U/Caddyfile | sed 's/llm.example.com/YOUR.DOMAIN/' | sudo tee /etc/caddy/Caddyfile
+sudo systemctl reload caddy
+```
+
+Keep 7443 (nodes) and 8081 (mirror) on the internal network. They never go
+through the proxy. There is an nginx configuration too:
+[deploy/proxy/](../deploy/proxy/).
+
+## When something does not work
+
+| What you see | What to do |
+|---|---|
+| `node add` stops at `join` | `sudo journalctl -u llmgw-agent -n 50` on the node. Usually it cannot reach port 7443 on the control plane. |
+| `node add` says the address is not in the certificate | Run `sudo llmgw server init --host <that address>`, then `node add` again. |
+| A node says it cannot run vLLM | The reason is on `/admin`: the driver is older than 580, the GPU is older than Turing, or there is less than 25 GB free. |
+| A model shows `not placed` | `llmgw ctl explain-placement <model>` names each node and why it was not used. |
+| A reply stops after 60 seconds behind nginx | Use the nginx configuration from the release. Its timeouts are set for slow models. |
+| Anything else about a node | `sudo llmgw doctor` on it. `--json` writes everything it read, to send to whoever helps you. |
+
+## Next
+
+- [install.md](install.md): the same steps with every option explained.
+- [fabrics.md](fabrics.md): InfiniBand and NVLink.
+- [onboarding.md](onboarding.md): accounts, projects, budgets and keys.

docs/quickstart.md

@@ -1,8 +1,8 @@
 # Quickstart: one machine, one model, ten minutes
 
-Written for someone who wants to see llmgw answer a request on a single
-workstation before reading anything else. No systemd, no `sudo`. For a real
-cluster, see [install.md](install.md).
+Written for a developer who wants to see llmgw answer a request on a single
+workstation, built from source, with no systemd and no `sudo`. **To install
+llmgw on servers, read [quick-install.md](quick-install.md) instead.**
 
 What you end up with: an OpenAI-compatible endpoint on `localhost:8080` in front
 of a real model served by llama.cpp.

docs/reference.md

@@ -29,6 +29,7 @@ rather than repeated.
 | `default_rpm` | `int` | [Auth](#auth) |
 | `default_tpm` | `int` | [Auth](#auth) |
 | `signup_open` | `bool` | [Auth](#auth) |
+| `personal` | `Personal` | [Auth](#auth) |
 | `pricing` | `map[string]Price` | [Auth](#auth) |
 | `default_price` | `Price` | [Auth](#auth) |
 | `flush_interval` | `time.Duration` | [Auth](#auth) |
@@ -70,6 +71,7 @@ rather than repeated.
 | `lease_ttl` | `time.Duration` | [Control](#control) |
 | `id` | `string` | [Endpoint](#endpoint) |
 | `url` | `string` | [Endpoint](#endpoint) |
+| `api_key` | `string` | [Endpoint](#endpoint) |
 | `models` | `[]string` | [Endpoint](#endpoint) |
 | `engine` | `string` | [Endpoint](#endpoint) |
 | `pool` | `string` | [Endpoint](#endpoint) |
@@ -102,6 +104,11 @@ rather than repeated.
 | `ib_root` | `string` | [Options](#options) |
 | `prefer_nvml` | `bool` | [Options](#options) |
 | `skip_gpus` | `bool` | [Options](#options) |
+| `disabled` | `bool` | [Personal](#personal) |
+| `rpm` | `int` | [Personal](#personal) |
+| `tpm` | `int` | [Personal](#personal) |
+| `budget` | `float64` | [Personal](#personal) |
+| `period` | `string` | [Personal](#personal) |
 | `enabled` | `bool` | [Prefix](#prefix) |
 | `max_nodes` | `int` | [Prefix](#prefix) |
 | `min_match` | `int` | [Prefix](#prefix) |
@@ -267,6 +274,15 @@ SignupOpen serves the signup form. Accounts it creates are pending until
 an operator approves them, so this is not a way in — but turning it off
 means accounts are made with `llmgw users create` instead.
 
+### `personal`
+
+`Personal`
+
+Personal is what a person may do on their own, outside any project: make
+personal keys and use the chat page personally, within these limits.
+
+Takes a [Personal](#personal) block.
+
 ### `pricing`
 
 `map[string]Price`
@@ -551,6 +567,13 @@ reading the alert. Zero takes the default.
 
 `string`
 
+### `api_key`
+
+`string`
+
+APIKey is sent to the engine as a bearer token, for an engine started
+with one.
+
 ### `models`
 
 `[]string`
@@ -778,8 +801,9 @@ IBRoot is the sysfs InfiniBand directory.
 
 `bool`
 
-PreferNVML uses the NVML backend when this binary has one. It is the only
-source of NVLink topology.
+PreferNVML uses the NVML backend when this binary has one. NVLink
+topology is read from nvidia-smi topo -m on every build; NVML, where it
+is used, is the more exact source of the peers.
 
 ### `skip_gpus`
 
@@ -788,6 +812,33 @@ source of NVLink topology.
 SkipGPUs omits GPU discovery. A node without nvidia-smi is already
 found to have none, so this is for one whose driver is broken.
 
+## Personal
+
+### `disabled`
+
+`bool`
+
+Disabled refuses personal keys and personal chat; people then use the
+gateway only through projects they are members of.
+
+### `rpm`
+
+`int`
+
+### `tpm`
+
+`int`
+
+### `budget`
+
+`float64`
+
+Budget is in whole units per Period ("day" or "month").
+
+### `period`
+
+`string`
+
 ## Prefix
 
 ### `enabled`
@@ -1309,11 +1360,13 @@ Bin is the vllm executable.
 
 `string`
 
-LaunchMode selects the multi-node mechanism: "ray" or "mp".
+LaunchMode selects the multi-node mechanism: "mp", the default, or
+"ray".
 
-Which one works depends on the vLLM version, and the flags have changed
-more than once (plan §14). Both paths exist so a cluster can pick the one
-its version supports instead of waiting for a code change.
+mp is what vLLM documents today — every node runs serve with its rank,
+the workers headless — and needs nothing but vLLM. ray is the older way,
+kept for a vLLM too old for mp's multi-node flags; it needs Ray on every
+node and has not been checked against a real cluster.
 
 ### `ray_bin`
 

go.mod

@@ -6,15 +6,19 @@ require (
 	connectrpc.com/connect v1.21.0
 	github.com/NVIDIA/go-nvml v0.13.4-0
 	github.com/cespare/xxhash/v2 v2.3.0
+	github.com/charmbracelet/bubbletea v1.3.10
+	github.com/charmbracelet/lipgloss v1.1.0
 	github.com/jackc/pgx/v5 v5.11.0
 	github.com/knadh/koanf/parsers/yaml v1.1.1
 	github.com/knadh/koanf/providers/env/v2 v2.0.1
 	github.com/knadh/koanf/providers/file v1.2.1
 	github.com/knadh/koanf/v2 v2.3.6
+	github.com/mattn/go-isatty v0.0.24
 	github.com/prometheus/client_golang v1.24.1
 	github.com/prometheus/client_model v0.6.3
 	github.com/prometheus/common v0.71.0
 	github.com/spf13/cobra v1.10.2
+	github.com/yuin/goldmark v1.8.6
 	golang.org/x/crypto v0.57.0
 	golang.org/x/net v0.59.0
 	golang.org/x/sync v0.23.0
@@ -23,8 +27,17 @@ require (
 )
 
 require (
+	github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
+	github.com/charmbracelet/colorprofile v0.4.1 // indirect
+	github.com/charmbracelet/x/ansi v0.11.6 // indirect
+	github.com/charmbracelet/x/cellbuf v0.0.15 // indirect
+	github.com/charmbracelet/x/term v0.2.2 // indirect
+	github.com/clipperhouse/displaywidth v0.9.0 // indirect
+	github.com/clipperhouse/stringish v0.1.1 // indirect
+	github.com/clipperhouse/uax29/v2 v2.5.0 // indirect
 	github.com/dustin/go-humanize v1.0.1 // indirect
+	github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f // indirect
 	github.com/fsnotify/fsnotify v1.9.0 // indirect
 	github.com/go-viper/mapstructure/v2 v2.4.0 // indirect
 	github.com/google/uuid v1.6.0 // indirect
@@ -33,14 +46,21 @@ require (
 	github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
 	github.com/jackc/puddle/v2 v2.2.2 // indirect
 	github.com/knadh/koanf/maps v0.1.2 // indirect
-	github.com/mattn/go-isatty v0.0.24 // indirect
+	github.com/lucasb-eyer/go-colorful v1.3.0 // indirect
+	github.com/mattn/go-localereader v0.0.1 // indirect
+	github.com/mattn/go-runewidth v0.0.19 // indirect
 	github.com/mitchellh/copystructure v1.2.0 // indirect
 	github.com/mitchellh/reflectwalk v1.0.2 // indirect
+	github.com/muesli/ansi v0.0.0-20230316100256-276c6243b2f6 // indirect
+	github.com/muesli/cancelreader v0.2.2 // indirect
+	github.com/muesli/termenv v0.16.0 // indirect
 	github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
 	github.com/ncruces/go-strftime v1.0.0 // indirect
 	github.com/prometheus/procfs v0.21.1 // indirect
 	github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
+	github.com/rivo/uniseg v0.4.7 // indirect
 	github.com/spf13/pflag v1.0.9 // indirect
+	github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e // indirect
 	go.yaml.in/yaml/v3 v3.0.5 // indirect
 	golang.org/x/sys v0.48.0 // indirect
 	golang.org/x/text v0.42.0 // indirect

go.sum

@@ -2,14 +2,36 @@ connectrpc.com/connect v1.21.0 h1:LhqSJt7jHf5NJBo9Jq/t/9FjcYAideif0mg+qe2jCUs=
 connectrpc.com/connect v1.21.0/go.mod h1:A2ygJrukXwWy32vkCAAHNVguZrqZ+jeZ9rGRnGR4dN4=
 github.com/NVIDIA/go-nvml v0.13.4-0 h1:o3jp9u2x1R9ShFE3v+Aesp55XOSIQFMJz/VGNUcJaNE=
 github.com/NVIDIA/go-nvml v0.13.4-0/go.mod h1:id63qwpoDWpFXwnwM6psDCSqW4BmNu6mWpr4YeQtPGo=
+github.com/aymanbagabas/go-osc52/v2 v2.0.1 h1:HwpRHbFMcZLEVr42D4p7XBqjyuxQH5SMiErDT4WkJ2k=
+github.com/aymanbagabas/go-osc52/v2 v2.0.1/go.mod h1:uYgXzlJ7ZpABp8OJ+exZzJJhRNQ2ASbcXHWsFqH8hp8=
 github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
 github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
 github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
 github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
+github.com/charmbracelet/bubbletea v1.3.10 h1:otUDHWMMzQSB0Pkc87rm691KZ3SWa4KUlvF9nRvCICw=
+github.com/charmbracelet/bubbletea v1.3.10/go.mod h1:ORQfo0fk8U+po9VaNvnV95UPWA1BitP1E0N6xJPlHr4=
+github.com/charmbracelet/colorprofile v0.4.1 h1:a1lO03qTrSIRaK8c3JRxJDZOvhvIeSco3ej+ngLk1kk=
+github.com/charmbracelet/colorprofile v0.4.1/go.mod h1:U1d9Dljmdf9DLegaJ0nGZNJvoXAhayhmidOdcBwAvKk=
+github.com/charmbracelet/lipgloss v1.1.0 h1:vYXsiLHVkK7fp74RkV7b2kq9+zDLoEU4MZoFqR/noCY=
+github.com/charmbracelet/lipgloss v1.1.0/go.mod h1:/6Q8FR2o+kj8rz4Dq0zQc3vYf7X+B0binUUBwA0aL30=
+github.com/charmbracelet/x/ansi v0.11.6 h1:GhV21SiDz/45W9AnV2R61xZMRri5NlLnl6CVF7ihZW8=
+github.com/charmbracelet/x/ansi v0.11.6/go.mod h1:2JNYLgQUsyqaiLovhU2Rv/pb8r6ydXKS3NIttu3VGZQ=
+github.com/charmbracelet/x/cellbuf v0.0.15 h1:ur3pZy0o6z/R7EylET877CBxaiE1Sp1GMxoFPAIztPI=
+github.com/charmbracelet/x/cellbuf v0.0.15/go.mod h1:J1YVbR7MUuEGIFPCaaZ96KDl5NoS0DAWkskup+mOY+Q=
+github.com/charmbracelet/x/term v0.2.2 h1:xVRT/S2ZcKdhhOuSP4t5cLi5o+JxklsoEObBSgfgZRk=
+github.com/charmbracelet/x/term v0.2.2/go.mod h1:kF8CY5RddLWrsgVwpw4kAa6TESp6EB5y3uxGLeCqzAI=
+github.com/clipperhouse/displaywidth v0.9.0 h1:Qb4KOhYwRiN3viMv1v/3cTBlz3AcAZX3+y9OLhMtAtA=
+github.com/clipperhouse/displaywidth v0.9.0/go.mod h1:aCAAqTlh4GIVkhQnJpbL0T/WfcrJXHcj8C0yjYcjOZA=
+github.com/clipperhouse/stringish v0.1.1 h1:+NSqMOr3GR6k1FdRhhnXrLfztGzuG+VuFDfatpWHKCs=
+github.com/clipperhouse/stringish v0.1.1/go.mod h1:v/WhFtE1q0ovMta2+m+UbpZ+2/HEXNWYXQgCt4hdOzA=
+github.com/clipperhouse/uax29/v2 v2.5.0 h1:x7T0T4eTHDONxFJsL94uKNKPHrclyFI0lm7+w94cO8U=
+github.com/clipperhouse/uax29/v2 v2.5.0/go.mod h1:Wn1g7MK6OoeDT0vL+Q0SQLDz/KpfsVRgg6W7ihQeh4g=
 github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
 github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
+github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f h1:Y/CXytFA4m6baUTXGLOoWe4PQhGxaX0KpnayAqC48p4=
+github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f/go.mod h1:vw97MGsxSvLiUE2X8qFplwetxpGLQrlU1Q9AUEIzCaM=
 github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
 github.com/fsnotify/fsnotify v1.9.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
 github.com/go-viper/mapstructure/v2 v2.4.0 h1:EBsztssimR/CONLSZZ04E8qAkxNYq4Qp9LvH92wZUgs=
@@ -46,12 +68,24 @@ github.com/knadh/koanf/v2 v2.3.6 h1:JoQPSJmvS4aP0xNc8xMDr5tcrkSEInL23/Il7pITAKo=
 github.com/knadh/koanf/v2 v2.3.6/go.mod h1:gRb40VRAbd4iJMYYD5IxZ6hfuopFcXBpc9bbQpZwo28=
 github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
 github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
+github.com/lucasb-eyer/go-colorful v1.3.0 h1:2/yBRLdWBZKrf7gB40FoiKfAWYQ0lqNcbuQwVHXptag=
+github.com/lucasb-eyer/go-colorful v1.3.0/go.mod h1:R4dSotOR9KMtayYi1e77YzuveK+i7ruzyGqttikkLy0=
 github.com/mattn/go-isatty v0.0.24 h1:tGZZoVgT/KiqK1c8ocVLeDS8BSWMRd47J3Lbz7vsReI=
 github.com/mattn/go-isatty v0.0.24/go.mod h1:nMCL3Zebbrt45jsMDgnfIwz6ydEQApk5oEI3HqDio6A=
+github.com/mattn/go-localereader v0.0.1 h1:ygSAOl7ZXTx4RdPYinUpg6W99U8jWvWi9Ye2JC/oIi4=
+github.com/mattn/go-localereader v0.0.1/go.mod h1:8fBrzywKY7BI3czFoHkuzRoWE9C+EiG4R1k4Cjx5p88=
+github.com/mattn/go-runewidth v0.0.19 h1:v++JhqYnZuu5jSKrk9RbgF5v4CGUjqRfBm05byFGLdw=
+github.com/mattn/go-runewidth v0.0.19/go.mod h1:XBkDxAl56ILZc9knddidhrOlY5R/pDhgLpndooCuJAs=
 github.com/mitchellh/copystructure v1.2.0 h1:vpKXTN4ewci03Vljg/q9QvCGUDttBOGBIa15WveJJGw=
 github.com/mitchellh/copystructure v1.2.0/go.mod h1:qLl+cE2AmVv+CoeAwDPye/v+N2HKCj9FbZEVFJRxO9s=
 github.com/mitchellh/reflectwalk v1.0.2 h1:G2LzWKi524PWgd3mLHV8Y5k7s6XUvT0Gef6zxSIeXaQ=
 github.com/mitchellh/reflectwalk v1.0.2/go.mod h1:mSTlrgnPZtwu0c4WaC2kGObEpuNDbx0jmZXqmk4esnw=
+github.com/muesli/ansi v0.0.0-20230316100256-276c6243b2f6 h1:ZK8zHtRHOkbHy6Mmr5D264iyp3TiX5OmNcI5cIARiQI=
+github.com/muesli/ansi v0.0.0-20230316100256-276c6243b2f6/go.mod h1:CJlz5H+gyd6CUWT45Oy4q24RdLyn7Md9Vj2/ldJBSIo=
+github.com/muesli/cancelreader v0.2.2 h1:3I4Kt4BQjOR54NavqnDogx/MIoWBFa0StPA8ELUXHmA=
+github.com/muesli/cancelreader v0.2.2/go.mod h1:3XuTXfFS2VjM+HTLZY9Ak0l6eUKfijIfMUZ4EgX0QYo=
+github.com/muesli/termenv v0.16.0 h1:S5AlUN9dENB57rsbnkPyfdGuWIlkmzJjbFf0Tf5FWUc=
+github.com/muesli/termenv v0.16.0/go.mod h1:ZRfOIKPFDYQoDFF4Olj7/QJbW60Ol/kL1pU3VfY/Cnk=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
 github.com/ncruces/go-strftime v1.0.0 h1:HMFp8mLCTPp341M/ZnA4qaf7ZlsbTc+miZjCLOFAw7w=
@@ -67,6 +101,8 @@ github.com/prometheus/procfs v0.21.1 h1:GljZCt+zSTS+NZq88cyQ1LjZ+RCHp3uVuabBWA5+
 github.com/prometheus/procfs v0.21.1/go.mod h1:aB55Cww9pdSJVHk0hUf0inxWyyjPogFIjmHKYgMKmtY=
 github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
 github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
+github.com/rivo/uniseg v0.4.7 h1:WUdvkW8uEhrYfLC4ZzdpI2ztxP1I582+49Oc5Mq64VQ=
+github.com/rivo/uniseg v0.4.7/go.mod h1:FN3SvrM+Zdj16jyLfmOkMNblXMcoc8DfTHruCPUcx88=
 github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
 github.com/spf13/cobra v1.10.2 h1:DMTTonx5m65Ic0GOoRY2c16WCbHxOOw6xxezuLaBpcU=
 github.com/spf13/cobra v1.10.2/go.mod h1:7C1pvHqHw5A4vrJfjNwvOdzYu0Gml16OCs2GRiTUUS4=
@@ -77,6 +113,10 @@ github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UV
 github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
 github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE=
 github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg=
+github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e h1:JVG44RsyaB9T2KIHavMF/ppJZNG9ZpyihvCd0w101no=
+github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e/go.mod h1:RbqR21r5mrJuqunuUZ/Dhy/avygyECGrLceyNeo4LiM=
+github.com/yuin/goldmark v1.8.6 h1:d0VcaP1sx9GkFVkoW+KtggpGi2KZ965i14b0+bDQST4=
+github.com/yuin/goldmark v1.8.6/go.mod h1:ip/1k0VRfGynBgxOz0yCqHrbZXhcjxyuS66Brc7iBKg=
 go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
 go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
 go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ=
@@ -86,12 +126,15 @@ go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
 go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
 golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
 golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
+golang.org/x/exp v0.0.0-20231006140011-7918f672742d h1:jtJma62tbqLibJ5sFQz8bKtEM8rJBtfilJ2qTU199MI=
+golang.org/x/exp v0.0.0-20231006140011-7918f672742d/go.mod h1:ldy0pHrwJyGW56pPQzzkH36rKxoZW1tw7ZJpeKx+hdo=
 golang.org/x/mod v0.41.0 h1:qJmnOUb4YB+FsEuM3HcWucdZASCPGhsX6uljO6pog0c=
 golang.org/x/mod v0.41.0/go.mod h1:Ek9pY8RKWXwsWvd3rQiHYtMqkjSUV+s1Rj7j4H5Ur6o=
 golang.org/x/net v0.59.0 h1:5zfYln+w5XCxwrnMMJPufRgNoXEaGxl0wo5GqPXyues=
 golang.org/x/net v0.59.0/go.mod h1:2DA/G1UfVbCpQPeWTmMPGY7Cs2PkBkwu743bVX5PIVg=
 golang.org/x/sync v0.23.0 h1:KameEIfc1IkluZyXWLn39Wd4tURc6GbCiISGiZm2bQk=
 golang.org/x/sync v0.23.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0=
+golang.org/x/sys v0.0.0-20210809222454-d867a43fc93e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
 golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
 golang.org/x/text v0.42.0 h1:JbOZXgfeCPU9gacVtYliJqOhD+zhrEqK4LfdpmlUZqI=

internal/account/account_test.go

@@ -99,17 +99,21 @@ func TestApprovalIsWhatMakesAnAccountWork(t *testing.T) {
 	if err != nil {
 		t.Fatal(err)
 	}
-	if got.ProjectID != "proj" {
-		t.Errorf("the approved account landed in project %q", got.ProjectID)
+	if ok, err := st.CanUseProject(ctx, got.ID, "proj"); err != nil || !ok {
+		t.Errorf("the approved account is not a member of the project it was approved into: %v", err)
 	}
 }
 
-func TestApprovalNeedsAProject(t *testing.T) {
-	// An approved account with nowhere to put its keys passes every check and
-	// then fails at the first key it tries to make.
-	_, st := newStoreAndUser(t)
-	if err := st.ApproveUser(context.Background(), "usr-1", "", "ilker", time.Now()); err == nil {
-		t.Error("an account was approved into no project")
+func TestApprovalNeedNotNameAProject(t *testing.T) {
+	// A person need not belong to anything to use the gateway: approved into
+	// nothing, they work on their own within their personal limits.
+	svc, st := newStoreAndUser(t)
+	ctx := context.Background()
+	if err := st.ApproveUser(ctx, "usr-1", "", "ilker", time.Now()); err != nil {
+		t.Fatalf("approving with no project: %v", err)
+	}
+	if _, _, err := svc.SignIn(ctx, "ayse@example.com", goodPassword); err != nil {
+		t.Errorf("an account approved into nothing cannot sign in: %v", err)
 	}
 }
 

internal/accounting/accounting.go

@@ -5,6 +5,7 @@ import (
 	"errors"
 	"fmt"
 	"log/slog"
+	"strings"
 	"time"
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/auth"
@@ -79,14 +80,27 @@ func (a *Accountant) RestoreBudgets(ctx context.Context, s store.Store, period s
 	// totals would bring the control plane back up believing the team had spent
 	// nothing — which is a budget that resets on every restart, for exactly the
 	// scopes where it matters most.
-	spend := make(map[string]int64, len(rows)*4)
+	// And every pool a person is in: their own, and each per-member limit of
+	// the project, team and organization they spent it in.
+	spend := make(map[string]int64, len(rows)*8)
 	for _, r := range rows {
-		for _, o := range []store.LimitOwner{
+		owners := []store.LimitOwner{
 			{Scope: store.ScopeKey, ID: r.KeyID},
 			{Scope: store.ScopeProject, ID: r.ProjectID},
 			{Scope: store.ScopeTeam, ID: r.TeamID},
 			{Scope: store.ScopeOrg, ID: r.OrgID},
-		} {
+			{Scope: store.ScopeUser, ID: r.UserID},
+		}
+		if r.UserID != "" && r.ProjectID != "" {
+			owners = append(owners,
+				store.MemberOwner(store.ScopeProjectMember, r.ProjectID, r.UserID),
+				store.MemberOwner(store.ScopeTeamMember, r.TeamID, r.UserID),
+				store.MemberOwner(store.ScopeOrgMember, r.OrgID, r.UserID))
+		}
+		for _, o := range owners {
+			if o.ID == "" || strings.HasPrefix(o.ID, "/") {
+				continue
+			}
 			if k := o.CounterKey(); k != "" {
 				spend[k] += r.Micros
 			}

internal/accounting/people_test.go

@@ -0,0 +1,126 @@
+package accounting
+
+import (
+	"context"
+	"errors"
+	"log/slog"
+	"path/filepath"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/auth"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// peopleStore has Acme/ml/training with two members, ayse and deniz.
+func peopleStore(t *testing.T) (*store.DB, string) {
+	t.Helper()
+	ctx := context.Background()
+	s, err := store.OpenSQLite(ctx, filepath.Join(t.TempDir(), "p.db"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	t.Cleanup(func() { _ = s.Close() })
+	org, _ := s.CreateOrg(ctx, "Acme")
+	team, _ := s.CreateTeam(ctx, org, "ml")
+	proj, _ := s.CreateProject(ctx, team, "training")
+	for _, id := range []string{"ayse", "deniz"} {
+		if err := s.CreateUser(ctx, store.User{ID: id, Email: id + "@x", Name: id,
+			PasswordHash: "h", State: store.UserPending, CreatedAt: time.Now()}); err != nil {
+			t.Fatal(err)
+		}
+		if err := s.ApproveUser(ctx, id, proj, "test", time.Now()); err != nil {
+			t.Fatal(err)
+		}
+	}
+	return s, proj
+}
+
+// keyFor makes a key for a person and resolves it the way authentication does.
+func keyFor(t *testing.T, s *store.DB, id, user, project string) auth.Principal {
+	t.Helper()
+	ctx := context.Background()
+	_, k, _ := store.NewKey(id, id, project)
+	if err := s.CreateUserKey(ctx, user, k); err != nil {
+		t.Fatal(err)
+	}
+	e, err := s.Resolve(ctx, id)
+	if err != nil {
+		t.Fatalf("resolving %s: %v", id, err)
+	}
+	return auth.Principal{KeyID: id, UserID: e.UserID, ProjectID: e.ProjectID, Limits: e}
+}
+
+func TestAPersonsLimitIsOnePoolAcrossTheirKeys(t *testing.T) {
+	s, proj := peopleStore(t)
+	if err := s.PutLimits(context.Background(),
+		store.Limits{Scope: store.ScopeUser, ScopeID: "ayse", RPM: intp(2)}); err != nil {
+		t.Fatal(err)
+	}
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	personal := keyFor(t, s, "k-personal", "ayse", "")
+	inProject := keyFor(t, s, "k-project", "ayse", proj)
+
+	// Two requests, one on each key, use up the person's two a minute.
+	for _, p := range []auth.Principal{personal, inProject} {
+		if _, err := a.Check(p, "free", 1); err != nil {
+			t.Fatalf("%s: %v", p.KeyID, err)
+		}
+	}
+	if _, err := a.Check(personal, "free", 1); !errors.Is(err, ErrRateLimited) {
+		t.Errorf("a third request across her keys was allowed: %v", err)
+	}
+	// Deniz is somebody else.
+	if _, err := a.Check(keyFor(t, s, "k-deniz", "deniz", proj), "free", 1); err != nil {
+		t.Errorf("another person was limited by hers: %v", err)
+	}
+}
+
+func TestAPerMemberLimitCountsEachMemberApart(t *testing.T) {
+	s, proj := peopleStore(t)
+	if err := s.PutLimits(context.Background(),
+		store.Limits{Scope: store.ScopeProjectMember, ScopeID: proj, RPM: intp(1)}); err != nil {
+		t.Fatal(err)
+	}
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	ayse := keyFor(t, s, "k-a", "ayse", proj)
+	deniz := keyFor(t, s, "k-d", "deniz", proj)
+
+	if _, err := a.Check(ayse, "free", 1); err != nil {
+		t.Fatal(err)
+	}
+	if _, err := a.Check(ayse, "free", 1); !errors.Is(err, ErrRateLimited) {
+		t.Errorf("her second request passed a one-a-minute per-member limit: %v", err)
+	}
+	// Her share is hers: Deniz still has his.
+	if _, err := a.Check(deniz, "free", 1); err != nil {
+		t.Errorf("one member's use counted against another: %v", err)
+	}
+	// And outside the project it does not apply at all.
+	if _, err := a.Check(keyFor(t, s, "k-a-own", "ayse", ""), "free", 1); err != nil {
+		t.Errorf("a project's per-member limit held her personal key: %v", err)
+	}
+}
+
+func TestAPersonsBudgetSurvivesARestart(t *testing.T) {
+	s, _ := peopleStore(t)
+	ctx := context.Background()
+	if err := s.PutLimits(ctx, store.Limits{Scope: store.ScopeUser, ScopeID: "ayse",
+		Budget: i64p(1000), BudgetPeriod: store.PeriodMonth}); err != nil {
+		t.Fatal(err)
+	}
+	// Spent before the restart, personally and on the chat page.
+	if err := s.WriteUsage(ctx, []store.Usage{
+		{At: time.Now(), KeyID: "k-old", UserID: "ayse", CostMicros: 600, Model: "paid", ServedModel: "paid"},
+		{At: time.Now(), KeyID: "session:ayse", UserID: "ayse", CostMicros: 500, Model: "paid", ServedModel: "paid"},
+	}); err != nil {
+		t.Fatal(err)
+	}
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	if err := a.RestoreBudgets(ctx, s, store.PeriodMonth); err != nil {
+		t.Fatal(err)
+	}
+	if _, err := a.Check(keyFor(t, s, "k-new", "ayse", ""), "paid", 1); !errors.Is(err, ErrBudgetSpent) {
+		t.Errorf("a restart forgot what she had spent: %v", err)
+	}
+}

internal/agent/agent.go

@@ -16,8 +16,10 @@ import (
 	"net/http"
 	"net/url"
 	"os"
+	"os/exec"
 	"path/filepath"
 	"slices"
+	"strconv"
 	"strings"
 	"sync"
 	"time"
@@ -118,8 +120,11 @@ type Config struct {
 
 // DefaultConfig returns sensible defaults.
 func DefaultConfig() Config {
+	// No MgmtInterface: normalize finds the one holding the management
+	// address, and falls back to eth0 only when it cannot. Defaulted here, it
+	// was never found — every node said eth0 unless told otherwise, and a node
+	// on ens3 or bond0 had NCCL bootstrap over an interface it does not have.
 	return Config{
-		MgmtInterface:     "eth0",
 		HeartbeatInterval: 5 * time.Second,
 		ReconnectBase:     time.Second,
 		ReconnectMax:      30 * time.Second,
@@ -159,7 +164,7 @@ func (c *Config) normalize() error {
 		c.MgmtInterface = interfaceOf(c.MgmtAddr)
 	}
 	if c.MgmtInterface == "" {
-		c.MgmtInterface = d.MgmtInterface
+		c.MgmtInterface = "eth0"
 	}
 	// The directory downloads land in is always one that is looked in. When
 	// they were two settings, a node could fill its disk with fetched weights
@@ -352,6 +357,22 @@ type Agent struct {
 	// failover the old stream's heartbeat loop can still be running when the
 	// new one's starts.
 	lastWeights string
+	// fabricsSent is what the control plane was last told about fabrics.
+	fabricsSent string
+	// sm caches which subnet manager each InfiniBand port sees, asked once
+	// a minute: a management datagram per port per heartbeat is more than
+	// an answer that changes on a failover is worth.
+	sm      map[string]*discovery.SMInfo
+	smErr   map[string]string
+	smAsked time.Time
+	// nvlAsked is when the GPUs' NVLink fabric state was last read.
+	nvlAsked time.Time
+	// sms is every subnet manager on a fabric, asked through one port per
+	// master, keyed by that port; smsErr says why a port could not ask.
+	sms    map[string][]discovery.SMRecord
+	smsErr map[string]string
+	// load measures the machine between heartbeats.
+	load *discovery.LoadSampler
 
 	// fetchWatchers are the shards waiting on each source, so progress on one
 	// download reaches every shard that asked for it. A fetch is single-flight
@@ -396,6 +417,7 @@ func New(cfg Config, httpClient connectHTTPClient, log *slog.Logger) (*Agent, er
 		provStatus:   map[string]provision.Status{},
 		logRing:      ring,
 		provKick:     make(chan struct{}, 1),
+		load:         discovery.NewLoadSampler(),
 	}
 	a.prov = &provision.Installer{Root: cfg.EnginesDir, Log: log, OnStatus: a.provisionStatus}
 	return a, nil
@@ -423,6 +445,7 @@ func (a *Agent) Run(ctx context.Context) error {
 		a.log.Warn("cannot create the weights directory", "dir", a.cfg.WeightsCache, "err", err)
 	}
 	go a.provisionLoop(ctx)
+	go a.ensureDiags(ctx)
 
 	attempt := 0
 	next := 0
@@ -571,21 +594,36 @@ func (a *Agent) settleAdopted(sh *supervisor.Shard, o supervisor.Orphan) {
 // machine where both halves are visible, rather than inferred later from a
 // replica that will not converge.
 func (a *Agent) usableFabrics(inv discovery.Inventory) []inventory.FabricAttachment {
+	out := a.checkFabrics(inv.HCAs, true)
+	a.mu.Lock()
+	a.fabricsSent = fabricsFingerprint(out)
+	a.mu.Unlock()
+	return out
+}
+
+// checkFabrics is the declared fabrics this node can carry traffic on, from
+// the adapters as they are now. Loud logs every problem; the heartbeat checks
+// quietly and speaks only when the answer changes.
+func (a *Agent) checkFabrics(hcas []discovery.HCA, loud bool) []inventory.FabricAttachment {
 	declared := map[string][]string{}
 	for _, f := range a.cfg.Fabrics {
 		declared[f.Fabric] = f.HCAs
 	}
 
 	var out []inventory.FabricAttachment
-	for _, c := range discovery.CheckFabrics(declared, inv.HCAs) {
-		for _, p := range c.Problems {
-			a.log.Error("a declared fabric adapter cannot carry traffic",
-				"fabric", c.Fabric, "problem", p)
+	for _, c := range discovery.CheckFabrics(declared, hcas) {
+		if loud {
+			for _, p := range c.Problems {
+				a.log.Error("a declared fabric adapter cannot carry traffic",
+					"fabric", c.Fabric, "problem", p)
+			}
 		}
 		if !c.OK() {
-			a.log.Error("fabric not advertised; no declared adapter on it can carry traffic",
-				"fabric", c.Fabric,
-				"consequence", "no multi-node replica will be placed here")
+			if loud {
+				a.log.Error("fabric not advertised; no declared adapter on it can carry traffic",
+					"fabric", c.Fabric,
+					"consequence", "no multi-node replica will be placed here")
+			}
 			continue
 		}
 		out = append(out, inventory.FabricAttachment{
@@ -595,6 +633,84 @@ func (a *Agent) usableFabrics(inv discovery.Inventory) []inventory.FabricAttachm
 	return out
 }
 
+// fabricsFingerprint is enough to notice a fabric, an adapter or a rate
+// changing.
+func fabricsFingerprint(fs []inventory.FabricAttachment) string {
+	var b strings.Builder
+	for _, f := range fs {
+		fmt.Fprintf(&b, "%s:%s:%d;", f.Fabric, strings.Join(f.HCAs, ","), f.RateGbps)
+	}
+	return b.String()
+}
+
+// sampleIB re-reads the adapters for the heartbeat: every port's state and
+// counters, and the fabrics again when they no longer match what the control
+// plane was told.
+//
+// Every heartbeat, because it is a few small files per port, and because a
+// port that went down used to be invisible until the agent restarted — the
+// node went on being offered multi-node work that would hang on its first
+// collective.
+func (a *Agent) sampleIB(hb *llmgwv1.Heartbeat) {
+	a.mu.Lock()
+	had := len(a.inv.HCAs) > 0
+	a.mu.Unlock()
+	if !had && len(a.cfg.Fabrics) == 0 {
+		return
+	}
+	hcas, err := discovery.DiscoverIB(a.cfg.Discovery.IBRoot)
+	if err != nil {
+		return
+	}
+	a.attachSM(hcas)
+	for _, h := range hcas {
+		for _, p := range h.Ports {
+			ip := &llmgwv1.IBPort{
+				Hca: h.Name, Port: int32(p.Num), State: p.State, PhysState: p.PhysState,
+				RateGbps: int32(p.RateGbps), LinkLayer: string(p.LinkLayer), Counters: p.Counters,
+				NodeGuid: h.NodeGUID, PortGuid: p.PortGUID, Network: p.Network, GidPrefix: p.GIDPrefix,
+			}
+			a.mu.Lock()
+			if p.SM != nil {
+				ip.SmGuid, ip.SmLid, ip.SmPriority, ip.SmState = p.SM.GUID, int32(p.SM.LID), int32(p.SM.Priority), p.SM.State
+			} else {
+				ip.SmError = a.smErr[smKey(h.Name, p.Num)]
+			}
+			for _, r := range a.sms[smKey(h.Name, p.Num)] {
+				ip.SubnetManagers = append(ip.SubnetManagers, &llmgwv1.SubnetManager{
+					Guid: r.GUID, Lid: int32(r.LID), Priority: int32(r.Priority), State: r.State})
+			}
+			ip.SubnetManagersError = a.smsErr[smKey(h.Name, p.Num)]
+			a.mu.Unlock()
+			hb.IbPorts = append(hb.IbPorts, ip)
+		}
+	}
+	a.mu.Lock()
+	a.inv.HCAs = hcas // a reconnect registers what is true now
+	a.mu.Unlock()
+	if len(a.cfg.Fabrics) == 0 {
+		return
+	}
+	now := a.checkFabrics(hcas, false)
+	fp := fabricsFingerprint(now)
+	a.mu.Lock()
+	changed := fp != a.fabricsSent
+	a.fabricsSent = fp
+	a.mu.Unlock()
+	if !changed {
+		return
+	}
+	// Said once, with the reasons, the moment it happens.
+	a.checkFabrics(hcas, true)
+	a.log.Warn("the fabrics this node can carry traffic on changed", "now", fp)
+	hb.FabricsChanged = true
+	for _, f := range now {
+		hb.Fabrics = append(hb.Fabrics, &llmgwv1.FabricAttachment{
+			Fabric: f.Fabric, Hcas: f.HCAs, RateGbps: int32(f.RateGbps),
+		})
+	}
+}
+
 // weights builds the fetcher lazily, so an agent that never starts a shard
 // never needs a cache directory.
 func (a *Agent) weights() *weights.Fetcher {
@@ -682,6 +798,7 @@ func (a *Agent) scan(ctx context.Context) error {
 	opts.ModelDirs = a.cfg.ModelDirs
 	inv, err := discovery.Discover(ctx, opts, a.log)
 	host := discovery.DiscoverHost()
+	host.MinComputeCapability = discovery.MinComputeCapability(inv.GPUs)
 	a.mu.Lock()
 	a.inv = inv
 	a.prov.Host = host
@@ -726,6 +843,8 @@ func (a *Agent) session(ctx context.Context, server string) error {
 	// vLLM accepted a vLLM shard and failed it minutes later.
 	reg.EngineVersions = a.engines(parent)
 	reg.EnginesProbed = true
+	reg.InstallableEngines = provision.Installable()
+	reg.Features = []string{FeatureFabricTest}
 	reg.RunningShards = a.runningShards()
 	a.mu.Lock()
 	h := a.prov.Host
@@ -883,9 +1002,41 @@ func (a *Agent) heartbeat() *llmgwv1.AgentMessage {
 			})
 		}
 	}
+	hb.Load = a.sampleLoad()
+	a.sampleIB(hb)
+	a.sampleNVLink(hb)
 	return &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_Heartbeat{Heartbeat: hb}}
 }
 
+// sampleLoad measures the machine and each running engine since the last
+// heartbeat. Nil on the first, which has nothing to compare with.
+func (a *Agent) sampleLoad() *llmgwv1.NodeLoad {
+	if a.load == nil {
+		return nil
+	}
+	pids := map[string]int{}
+	for _, st := range a.sup.Statuses() {
+		if st.PID > 0 {
+			pids[st.ID] = st.PID
+		}
+	}
+	l, ok := a.load.Sample(pids)
+	if !ok {
+		return nil
+	}
+	out := &llmgwv1.NodeLoad{
+		CpuPercent: l.CPUPercent, CpuCount: int32(l.CPUCount),
+		Load1: l.Load1, Load5: l.Load5, Load15: l.Load15,
+		MemoryUsedBytes: l.MemUsed, MemoryAvailableBytes: l.MemAvail,
+	}
+	for _, sh := range l.Shards {
+		out.Shards = append(out.Shards, &llmgwv1.ShardLoad{
+			ShardId: sh.ID, CpuCores: sh.CPUCores, MemoryRssBytes: sh.RSSBytes,
+		})
+	}
+	return out
+}
+
 // weightsFingerprint is enough to notice a model appearing, disappearing or
 // finishing its download.
 //
@@ -978,6 +1129,33 @@ func (a *Agent) handle(ctx context.Context, msg *llmgwv1.ControlMessage, out cha
 			case <-ctx.Done():
 			}
 		}()
+	case *llmgwv1.ControlMessage_FabricScan:
+		// A scan takes seconds to minutes: off the receive loop entirely.
+		go func(req *llmgwv1.FabricScanRequest) {
+			start := time.Now()
+			reply := &llmgwv1.FabricScanReply{RequestId: req.GetRequestId()}
+			text, err := scanFabric(ctx, req.GetHca(), int(req.GetPort()))
+			if err != nil {
+				reply.Error = err.Error()
+			} else {
+				reply.Output = text
+			}
+			reply.TookMs = time.Since(start).Milliseconds()
+			select {
+			case out <- &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_FabricScan{FabricScan: reply}}:
+			case <-ctx.Done():
+			}
+		}(m.FabricScan)
+	case *llmgwv1.ControlMessage_FabricTest:
+		// Seconds of traffic, and a package install the first time: off the
+		// receive loop too.
+		go func(req *llmgwv1.FabricTestRequest) {
+			reply := a.fabricTest(ctx, req)
+			select {
+			case out <- &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_FabricTest{FabricTest: reply}}:
+			case <-ctx.Done():
+			}
+		}(m.FabricTest)
 	case *llmgwv1.ControlMessage_Ack:
 		if !m.Ack.GetOk() {
 			a.log.Warn("control plane rejected a message",
@@ -1047,6 +1225,7 @@ func (a *Agent) startShard(ctx context.Context, m *llmgwv1.StartShard, out chan<
 	}
 
 	spec, timeout := agentsvc.ShardSpecFromStart(m)
+	a.localizeSpec(&spec)
 
 	eng, err := a.reg.Get(spec.Profile.Engine)
 	if err != nil {
@@ -1439,3 +1618,259 @@ func (a *Agent) logsReply(req *llmgwv1.LogsRequest) *llmgwv1.LogsReply {
 	}
 	return out
 }
+
+// localizeSpec fills in what only this node knows.
+//
+// The interface NCCL bootstraps over is this node's, not the cluster's: the
+// control plane sends one name for every node, and a node whose management
+// network is on ens3 rather than eth0 would have NCCL bind nothing. And on
+// RoCE, which GID entry routes is a property of these adapters.
+func (a *Agent) localizeSpec(spec *engines.ShardSpec) {
+	if a.cfg.MgmtInterface != "" {
+		spec.MgmtInterface = a.cfg.MgmtInterface
+	}
+	if len(spec.HCAs) == 0 {
+		return
+	}
+	a.mu.Lock()
+	hcas := a.inv.HCAs
+	a.mu.Unlock()
+	want := map[string]bool{}
+	for _, h := range spec.HCAs {
+		want[h] = true
+	}
+	for _, h := range hcas {
+		if !want[h.Name] {
+			continue
+		}
+		for _, p := range h.Ports {
+			if p.Active() && p.RoCEv2GID >= 0 {
+				idx := p.RoCEv2GID
+				spec.GIDIndex = &idx
+				return
+			}
+		}
+	}
+}
+
+// ResolveManagement is the address and interface this node's agent would use,
+// found the way normalize finds them, for a tool that reports them.
+func ResolveManagement(c Config) (addr, iface string) {
+	addr, iface = c.MgmtAddr, c.MgmtInterface
+	server := c.Server
+	if server == "" && len(c.Servers) > 0 {
+		server = c.Servers[0]
+	}
+	if addr == "" {
+		addr = addrToward(server)
+	}
+	if iface == "" {
+		iface = interfaceOf(addr)
+	}
+	return addr, iface
+}
+
+func smKey(hca string, port int) string { return fmt.Sprintf("%s/%d", hca, port) }
+
+// scanFabric is discovery.ScanFabric, replaced in tests.
+var scanFabric = discovery.ScanFabric
+
+// querySMs is discovery.QuerySMs, replaced in tests.
+var querySMs = func(hca string, port int) ([]discovery.SMRecord, error) {
+	return discovery.QuerySMs(context.Background(), hca, port)
+}
+
+func splitSMKey(k string) (string, int) {
+	i := strings.LastIndexByte(k, '/')
+	port, _ := strconv.Atoi(k[i+1:])
+	return k[:i], port
+}
+
+// querySM is discovery.QuerySM, replaced in tests.
+var querySM = func(hca string, port int) (*discovery.SMInfo, error) {
+	return discovery.QuerySM(context.Background(), hca, port)
+}
+
+// attachSM puts on each active InfiniBand port the subnet manager it sees,
+// asking again once a minute and otherwise reusing the last answer.
+func (a *Agent) attachSM(hcas []discovery.HCA) {
+	a.mu.Lock()
+	stale := time.Since(a.smAsked) > time.Minute || a.sm == nil
+	a.mu.Unlock()
+	if stale {
+		sm, errs := map[string]*discovery.SMInfo{}, map[string]string{}
+		_ = discovery.AttachSM(hcas, func(hca string, port int) (*discovery.SMInfo, error) {
+			got, err := querySM(hca, port)
+			if err != nil {
+				errs[smKey(hca, port)] = err.Error()
+				return nil, err
+			}
+			sm[smKey(hca, port)] = got
+			return got, nil
+		})
+		// Every subnet manager on each fabric, through one port per master:
+		// the same list through every port of one fabric is one query too
+		// many for each.
+		sms, smsErr := map[string][]discovery.SMRecord{}, map[string]string{}
+		asked := map[string]bool{}
+		for k, v := range sm {
+			if asked[v.GUID] {
+				continue
+			}
+			asked[v.GUID] = true
+			hca, port := splitSMKey(k)
+			if recs, err := querySMs(hca, port); err != nil {
+				smsErr[k] = err.Error()
+			} else {
+				sms[k] = recs
+			}
+		}
+		a.mu.Lock()
+		for k, v := range sm {
+			if old := a.sm[k]; old != nil && old.GUID != v.GUID {
+				a.log.Warn("the master subnet manager changed", "port", k, "from", old.GUID, "to", v.GUID)
+			}
+		}
+		a.sm, a.smErr, a.smAsked = sm, errs, time.Now()
+		a.sms, a.smsErr = sms, smsErr
+		a.mu.Unlock()
+		return
+	}
+	a.mu.Lock()
+	defer a.mu.Unlock()
+	for i := range hcas {
+		for j := range hcas[i].Ports {
+			if sm := a.sm[smKey(hcas[i].Name, hcas[i].Ports[j].Num)]; sm != nil && hcas[i].Ports[j].Active() {
+				hcas[i].Ports[j].SM = sm
+			}
+		}
+	}
+}
+
+// ensureDiags installs infiniband-diags on a node with adapters and without
+// it. sminfo is what tells which physical fabric a port is on; without it,
+// two separate fabrics given one name cannot be caught. Installed the way
+// engine dependencies are, from the machine's own package sources.
+func (a *Agent) ensureDiags(ctx context.Context) {
+	a.mu.Lock()
+	hasIB := len(a.inv.HCAs) > 0
+	a.mu.Unlock()
+	if !hasIB || a.prov == nil {
+		return
+	}
+	if _, err := exec.LookPath("sminfo"); err == nil {
+		return
+	}
+	a.log.Info("installing infiniband-diags, to tell which fabric each port is on")
+	pkgs := map[string][]string{"deb": {"infiniband-diags"}, "rpm": {"infiniband-diags"}}
+	if err := a.prov.InstallPackages(ctx, pkgs); err != nil {
+		a.log.Warn("could not install infiniband-diags; fabrics cannot be told apart by their subnet manager",
+			"err", err)
+	}
+}
+
+// FeatureFabricTest is the agent answering FabricTestRequest.
+const FeatureFabricTest = "fabric-test"
+
+// runWriteBW is discovery.RunWriteBW, and haveWriteBW whether it is
+// installed; both replaced in tests.
+var (
+	runWriteBW  = discovery.RunWriteBW
+	haveWriteBW = func() bool { _, err := exec.LookPath("ib_write_bw"); return err == nil }
+)
+
+// fabricTest runs this node's side of a bandwidth test with another.
+func (a *Agent) fabricTest(ctx context.Context, req *llmgwv1.FabricTestRequest) *llmgwv1.FabricTestReply {
+	start := time.Now()
+	reply := &llmgwv1.FabricTestReply{RequestId: req.GetRequestId()}
+	defer func() { reply.TookMs = time.Since(start).Milliseconds() }()
+
+	w := discovery.WriteBW{HCA: req.GetHca(), Port: int(req.GetPort()), GIDIndex: -1,
+		TCPPort: int(req.GetTcpPort()), Seconds: int(req.GetSeconds())}
+	if req.GetRole() == "client" {
+		w.Peer = req.GetPeer()
+	}
+	a.mu.Lock()
+	found := false
+	for _, h := range a.inv.HCAs {
+		for _, p := range h.Ports {
+			if h.Name == w.HCA && p.Num == w.Port {
+				found = true
+				// RoCE needs its v2 GID named, as NCCL does; InfiniBand uses the LID.
+				if p.LinkLayer == discovery.LinkEthernet && p.RoCEv2GID >= 0 {
+					w.GIDIndex = p.RoCEv2GID
+				}
+			}
+		}
+	}
+	a.mu.Unlock()
+	if !found {
+		reply.Error = fmt.Sprintf("this node has no port %s/%d", w.HCA, w.Port)
+		return reply
+	}
+	if !haveWriteBW() && a.prov != nil {
+		a.log.Info("installing perftest, for a bandwidth test with another node")
+		pkgs := map[string][]string{"deb": {"perftest"}, "rpm": {"perftest"}}
+		if err := a.prov.InstallPackages(ctx, pkgs); err != nil {
+			reply.Error = "installing perftest: " + err.Error()
+			return reply
+		}
+	}
+	gbps, out, err := runWriteBW(ctx, w)
+	reply.Gbps = gbps
+	reply.Output = tailLines(out, 12)
+	if err != nil {
+		reply.Error = err.Error()
+	}
+	return reply
+}
+
+func tailLines(s string, n int) string {
+	lines := strings.Split(strings.TrimRight(s, "\n"), "\n")
+	if len(lines) > n {
+		lines = lines[len(lines)-n:]
+	}
+	return strings.Join(lines, "\n")
+}
+
+// discoverGPUFabric and discoverIMEX are replaced in tests.
+var (
+	discoverGPUFabric = discovery.DiscoverGPUFabric
+	discoverIMEX      = discovery.DiscoverIMEX
+)
+
+// sampleNVLink reads, once a minute, which NVLink domain each GPU is joined
+// to and whether IMEX is up. nvidia-smi -q takes a second or two, too long
+// for every heartbeat, and a fabric that was joined an hour ago rarely
+// changes in five seconds — but when a fabric manager restarts it does, for
+// every card at once.
+func (a *Agent) sampleNVLink(hb *llmgwv1.Heartbeat) {
+	a.mu.Lock()
+	gpus := slices.Clone(a.inv.GPUs)
+	due := time.Since(a.nvlAsked) > time.Minute
+	if due {
+		a.nvlAsked = time.Now()
+	}
+	a.mu.Unlock()
+	if len(gpus) == 0 || !due {
+		return
+	}
+	ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
+	defer cancel()
+	fab, err := discoverGPUFabric(ctx, a.cfg.Discovery.NvidiaSMI)
+	if err != nil || len(fab) == 0 {
+		return
+	}
+	discovery.ApplyFabric(gpus, fab)
+	for _, g := range gpus {
+		if g.Fabric == nil {
+			continue
+		}
+		hb.GpuFabric = append(hb.GpuFabric, &llmgwv1.GPUFabricState{
+			Index: int32(g.Index), State: g.Fabric.State, Status: g.Fabric.Status,
+			ClusterUuid: g.Fabric.ClusterUUID, CliqueId: g.Fabric.CliqueID, Bandwidth: g.Fabric.Bandwidth,
+		})
+	}
+	m := discoverIMEX(ctx)
+	hb.Imex = &llmgwv1.IMEX{State: m.State, Channels: int32(m.Channels)}
+}

internal/agent/agent_test.go

@@ -4,6 +4,7 @@ import (
 	"context"
 	"crypto/tls"
 	"errors"
+	"fmt"
 	"log/slog"
 	"net"
 	"net/http"
@@ -22,6 +23,8 @@ import (
 	"golang.org/x/net/http2"
 	"golang.org/x/net/http2/h2c"
 
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/agentsvc"
@@ -756,3 +759,157 @@ func TestAFabricRateIsTheSlowestActiveLink(t *testing.T) {
 		}
 	}
 }
+
+func TestAPortGoingDownIsToldToTheControlPlaneOnce(t *testing.T) {
+	// Checked only at registration, a port going down went unnoticed until
+	// the agent restarted, and the node went on being offered multi-node
+	// work that would hang on its first collective.
+	root := ibTreeWithActivePorts(t, "mlx5_0")
+	h := start(t, func(c *Config) {
+		c.Discovery.IBRoot = root
+		c.Fabrics = []inventory.FabricAttachment{{Fabric: "ib-a", HCAs: []string{"mlx5_0"}}}
+	})
+	if n, _ := h.inv.Get("n1"); !n.HasFabric("ib-a") {
+		t.Fatalf("the fabric should be advertised at registration: %+v", n.Fabrics)
+	}
+
+	hb := h.ag.heartbeat().GetHeartbeat()
+	if hb.GetFabricsChanged() || len(hb.GetIbPorts()) != 1 || hb.GetIbPorts()[0].GetState() != "ACTIVE" {
+		t.Fatalf("a quiet heartbeat should carry the port and no change: %+v", hb)
+	}
+
+	if err := os.WriteFile(filepath.Join(root, "mlx5_0", "ports", "1", "state"), []byte("1: DOWN\n"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	hb = h.ag.heartbeat().GetHeartbeat()
+	if !hb.GetFabricsChanged() || len(hb.GetFabrics()) != 0 {
+		t.Errorf("the port going down should withdraw the fabric: changed=%v fabrics=%v",
+			hb.GetFabricsChanged(), hb.GetFabrics())
+	}
+	if hb = h.ag.heartbeat().GetHeartbeat(); hb.GetFabricsChanged() {
+		t.Error("the change should be sent once, not on every heartbeat")
+	}
+}
+
+func TestAShardUsesThisNodesInterfaceAndRoCEIndex(t *testing.T) {
+	// The control plane sends one interface name for every node; a node whose
+	// management network is on another had NCCL bind nothing. And which RoCE
+	// GID entry routes is a property of this node's adapters.
+	a := &Agent{cfg: Config{MgmtInterface: "ens3"}}
+	a.inv.HCAs = []discovery.HCA{{Name: "mlx5_0", Ports: []discovery.Port{
+		{Num: 1, State: "ACTIVE", LinkLayer: discovery.LinkEthernet, RoCEv2GID: 3}}}}
+	spec := engines.ShardSpec{MgmtInterface: "eth0", HCAs: []string{"mlx5_0"}}
+	a.localizeSpec(&spec)
+	if spec.MgmtInterface != "ens3" {
+		t.Errorf("interface = %q", spec.MgmtInterface)
+	}
+	if spec.GIDIndex == nil || *spec.GIDIndex != 3 {
+		t.Errorf("GID index = %v", spec.GIDIndex)
+	}
+
+	ib := engines.ShardSpec{HCAs: []string{"mlx5_1"}}
+	a.inv.HCAs = append(a.inv.HCAs, discovery.HCA{Name: "mlx5_1", Ports: []discovery.Port{
+		{Num: 1, State: "ACTIVE", LinkLayer: discovery.LinkInfiniBand, RoCEv2GID: -1}}})
+	a.localizeSpec(&ib)
+	if ib.GIDIndex != nil {
+		t.Error("an InfiniBand shard was given a GID index")
+	}
+}
+
+func TestTheManagementInterfaceIsFoundNotAssumed(t *testing.T) {
+	// It was defaulted to eth0 before it could be looked for, so it never
+	// was: a node on ens3 or bond0 had NCCL bootstrap over nothing.
+	cfg := DefaultConfig()
+	cfg.Server = "http://127.0.0.1:7443"
+	cfg.NodeID = "n1"
+	cfg.MgmtAddr = "127.0.0.1"
+	if err := cfg.normalize(); err != nil {
+		t.Fatal(err)
+	}
+	if cfg.MgmtInterface == "" || cfg.MgmtInterface == "eth0" {
+		t.Errorf("interface = %q, want the loopback interface holding 127.0.0.1", cfg.MgmtInterface)
+	}
+}
+
+func TestEachFabricsSubnetManagersAreAskedOnce(t *testing.T) {
+	// Two adapters on one fabric see one master. Its managers are listed
+	// through one of them — the same list twice is a query too many — and
+	// every port says which master it sees.
+	oldSM, oldSMs := querySM, querySMs
+	t.Cleanup(func() { querySM, querySMs = oldSM, oldSMs })
+	querySM = func(string, int) (*discovery.SMInfo, error) {
+		return &discovery.SMInfo{GUID: "0x01", LID: 1, State: "SMINFO_MASTER"}, nil
+	}
+	lists := 0
+	querySMs = func(string, int) ([]discovery.SMRecord, error) {
+		lists++
+		return []discovery.SMRecord{{GUID: "0x01", State: "master"}, {GUID: "0x02", State: "standby"}}, nil
+	}
+	h := start(t, func(c *Config) {
+		c.Discovery.IBRoot = ibTreeWithActivePorts(t, "mlx5_0", "mlx5_1")
+		c.Fabrics = []inventory.FabricAttachment{{Fabric: "ib-a", HCAs: []string{"mlx5_0", "mlx5_1"}}}
+	})
+	hb := h.ag.heartbeat().GetHeartbeat()
+	withList := 0
+	for _, p := range hb.GetIbPorts() {
+		if p.GetSmGuid() != "0x01" {
+			t.Errorf("%s sees master %q", p.GetHca(), p.GetSmGuid())
+		}
+		if len(p.GetSubnetManagers()) > 0 {
+			withList++
+		}
+	}
+	if lists != 1 || withList != 1 {
+		t.Errorf("listed %d times, carried on %d ports; want once, on one", lists, withList)
+	}
+}
+
+func TestAFabricScanIsAskedForAndAnswered(t *testing.T) {
+	// One node per fabric maps it; the control plane asks and parses.
+	old := scanFabric
+	t.Cleanup(func() { scanFabric = old })
+	var asked string
+	scanFabric = func(_ context.Context, hca string, port int) (string, error) {
+		asked = fmt.Sprintf("%s/%d", hca, port)
+		return "Switch\t36 \"S-00000000000000a1\"\t\t# \"MF0;leaf-a:QM8700/U1\" enhanced port 0 lid 1 lmc 0\n", nil
+	}
+	h := start(t)
+	r, err := h.svc.ScanFabric(t.Context(), "n1", "mlx5_0", 1)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if asked != "mlx5_0/1" || !strings.Contains(r.GetOutput(), "leaf-a") || r.GetError() != "" {
+		t.Errorf("asked %q, got %+v", asked, r)
+	}
+}
+
+func TestABandwidthTestIsRunOnTheAdapterAskedFor(t *testing.T) {
+	oldRun, oldHave := runWriteBW, haveWriteBW
+	t.Cleanup(func() { runWriteBW, haveWriteBW = oldRun, oldHave })
+	haveWriteBW = func() bool { return true }
+	var got discovery.WriteBW
+	runWriteBW = func(_ context.Context, w discovery.WriteBW) (float64, string, error) {
+		got = w
+		return 388.2, " #bytes  #iterations  BW peak[Gb/sec]  BW average[Gb/sec]\n", nil
+	}
+	h := start(t, func(c *Config) { c.Discovery.IBRoot = ibTreeWithActivePorts(t, "mlx5_0") })
+
+	r, err := h.svc.FabricTest(t.Context(), "n1", &llmgwv1.FabricTestRequest{
+		Role: "client", Hca: "mlx5_0", Port: 1, Peer: "10.0.0.2", TcpPort: 18515, Seconds: 3})
+	if err != nil {
+		t.Fatal(err)
+	}
+	if r.GetGbps() != 388.2 || r.GetError() != "" {
+		t.Errorf("reply %+v", r)
+	}
+	if got.HCA != "mlx5_0" || got.Peer != "10.0.0.2" || got.TCPPort != 18515 || got.GIDIndex != -1 {
+		t.Errorf("ran %+v", got)
+	}
+
+	// An adapter the node does not have is said so, without running anything.
+	got = discovery.WriteBW{}
+	r, _ = h.svc.FabricTest(t.Context(), "n1", &llmgwv1.FabricTestRequest{Role: "server", Hca: "mlx5_9", Port: 1, Seconds: 1})
+	if !strings.Contains(r.GetError(), "no port mlx5_9/1") || got.HCA != "" {
+		t.Errorf("reply %+v, ran %+v", r, got)
+	}
+}

internal/agent/discovery/discover.go

@@ -21,8 +21,9 @@ type Options struct {
 	NvidiaSMI string `koanf:"nvidia_smi"`
 	// IBRoot is the sysfs InfiniBand directory.
 	IBRoot string `koanf:"ib_root"`
-	// PreferNVML uses the NVML backend when this binary has one. It is the only
-	// source of NVLink topology.
+	// PreferNVML uses the NVML backend when this binary has one. NVLink
+	// topology is read from nvidia-smi topo -m on every build; NVML, where it
+	// is used, is the more exact source of the peers.
 	PreferNVML bool `koanf:"prefer_nvml"`
 	// SkipGPUs omits GPU discovery. A node without nvidia-smi is already
 	// found to have none, so this is for one whose driver is broken.
@@ -64,8 +65,7 @@ func Discover(ctx context.Context, opts Options, log *slog.Logger) (Inventory, e
 				inv.GPUs, inv.Source = gpus, "nvml"
 				break
 			}
-			log.Warn("NVML discovery failed, falling back to nvidia-smi; "+
-				"NVLink topology will be unavailable", "err", err)
+			log.Warn("NVML discovery failed, falling back to nvidia-smi", "err", err)
 			fallthrough
 		default:
 			gpus, err := DiscoverGPUsSMI(ctx, opts.NvidiaSMI)
@@ -74,6 +74,26 @@ func Discover(ctx context.Context, opts Options, log *slog.Logger) (Inventory, e
 			}
 			inv.GPUs, inv.Source = gpus, "nvidia-smi"
 		}
+		// NVLink and which adapter sits nearest each card, for either path:
+		// NVML knows NVLink but not the adapters, nvidia-smi's query knows
+		// neither. A failure costs the topology and nothing else — the cards
+		// are still cards.
+		if len(inv.GPUs) > 0 {
+			topo, err := DiscoverTopology(ctx, opts.NvidiaSMI)
+			if err != nil {
+				log.Warn("could not read the GPU topology; tensor-parallel groups "+
+					"will be chosen without NVLink", "err", err)
+			} else {
+				topo.Apply(inv.GPUs)
+			}
+			// Multi-node NVLink: which domain each card has joined.
+			if fab, err := DiscoverGPUFabric(ctx, opts.NvidiaSMI); err != nil {
+				log.Warn("could not read the GPUs' NVLink fabric state", "err", err)
+			} else {
+				ApplyFabric(inv.GPUs, fab)
+			}
+			inv.IMEX = DiscoverIMEX(ctx)
+		}
 	}
 
 	hcas, err := DiscoverIB(opts.IBRoot)

internal/agent/discovery/fabrics.go

@@ -10,6 +10,8 @@ import (
 // has.
 type FabricCheck struct {
 	Fabric string
+	// Declared are the adapters the configuration names for it.
+	Declared []string
 	// Usable are the declared adapters that exist and have an active port.
 	Usable []string
 	// RateGbps is the slowest active port among them, because a collective
@@ -49,7 +51,7 @@ func CheckFabrics(declared map[string][]string, discovered []HCA) []FabricCheck
 
 	out := make([]FabricCheck, 0, len(names))
 	for _, fabric := range names {
-		c := FabricCheck{Fabric: fabric}
+		c := FabricCheck{Fabric: fabric, Declared: declared[fabric]}
 		for _, want := range declared[fabric] {
 			hca, found := byName[want]
 			if !found {

internal/agent/discovery/gpu.go

@@ -38,6 +38,13 @@ type GPU struct {
 	// NVLinkPeers are the indices of GPUs reachable over NVLink. Within a node
 	// a TP group should stay inside one NVLink domain.
 	NVLinkPeers []int `json:"nvlink_peers,omitempty"`
+	// NICPaths is how far each network adapter is from this GPU, by HCA name:
+	// PIX, PXB, PHB, NODE or SYS, as nvidia-smi topo reports it. What decides
+	// which adapter a GPU's share of a multi-node job should go through.
+	NICPaths map[string]string `json:"nic_paths,omitempty"`
+	// Fabric is the GPU's multi-node NVLink membership, nil where the
+	// driver reports none.
+	Fabric *GPUFabric `json:"fabric,omitempty"`
 	// Health carries anything that should keep the GPU out of placement.
 	Health GPUHealth `json:"health"`
 }
@@ -117,6 +124,9 @@ type Inventory struct {
 	// Storage is the disk behind those weights, one entry per filesystem.
 	// Empty means the same thing it means for Weights: not asked.
 	Storage []Storage `json:"storage,omitempty"`
+	// IMEX is the service a multi-node NVLink domain needs, read where the
+	// node has GPUs.
+	IMEX IMEX `json:"imex"`
 }
 
 // Classes returns the distinct GPU classes present, sorted.

internal/agent/discovery/gpu_fabric.go

@@ -0,0 +1,160 @@
+package discovery
+
+import (
+	"context"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"strings"
+	"time"
+)
+
+// GPUFabric is a GPU's place in a multi-node NVLink domain, as the driver
+// reports it: the NVL72 rack, or the NVLink Switch system, it belongs to.
+type GPUFabric struct {
+	// State is Completed once the fabric manager has joined the GPU to its
+	// domain; In Progress, Not Started or Not Supported otherwise.
+	State string `json:"state"`
+	// Status is Success, or the reason the join failed.
+	Status string `json:"status,omitempty"`
+	// ClusterUUID and CliqueID name the domain: GPUs sharing both can reach
+	// each other over NVLink whatever node they are in.
+	ClusterUUID string `json:"cluster_uuid,omitempty"`
+	CliqueID    string `json:"clique_id,omitempty"`
+	// Bandwidth is Full, or how the domain's NVLink is degraded.
+	Bandwidth string `json:"bandwidth,omitempty"`
+}
+
+// Domain is the NVLink domain the GPU is joined to, empty when it is in none
+// that crosses nodes: a card with no fabric, one not joined, or one whose
+// cluster is all zeros, which is what a single machine's NVSwitch reports.
+func (f *GPUFabric) Domain() string {
+	if f == nil || !strings.EqualFold(f.State, "Completed") {
+		return ""
+	}
+	u := strings.ToLower(strings.TrimSpace(f.ClusterUUID))
+	if u == "" || strings.Trim(u, "0-") == "" {
+		return ""
+	}
+	return u + "/" + f.CliqueID
+}
+
+// ParseGPUFabric reads the Fabric sections of `nvidia-smi -q`, by the PCI
+// address each GPU's section starts with.
+func ParseGPUFabric(out string) map[string]GPUFabric {
+	res := map[string]GPUFabric{}
+	var bus string
+	fabricIndent := -1
+	var cur *GPUFabric
+	flush := func() {
+		if bus != "" && cur != nil && cur.State != "" {
+			res[bus] = *cur
+		}
+		cur, fabricIndent = nil, -1
+	}
+	for _, line := range strings.Split(out, "\n") {
+		trim := strings.TrimSpace(line)
+		indent := len(line) - len(strings.TrimLeft(line, " \t"))
+		if strings.HasPrefix(line, "GPU ") && indent == 0 {
+			flush()
+			bus = strings.ToLower(strings.TrimSpace(strings.TrimPrefix(line, "GPU ")))
+			continue
+		}
+		if trim == "Fabric" {
+			flush()
+			cur, fabricIndent = &GPUFabric{}, indent
+			continue
+		}
+		if cur == nil {
+			continue
+		}
+		if trim != "" && indent <= fabricIndent {
+			flush()
+			continue
+		}
+		k, v, ok := strings.Cut(trim, ":")
+		if !ok {
+			continue
+		}
+		k, v = strings.TrimSpace(k), strings.TrimSpace(v)
+		if strings.EqualFold(v, "N/A") {
+			v = ""
+		}
+		switch k {
+		case "State":
+			cur.State = v
+		case "Status":
+			cur.Status = v
+		case "ClusterUUID", "Cluster UUID":
+			cur.ClusterUUID = v
+		case "CliqueId", "Clique ID":
+			cur.CliqueID = v
+		case "Bandwidth":
+			cur.Bandwidth = v
+		}
+	}
+	flush()
+	return res
+}
+
+// DiscoverGPUFabric runs `nvidia-smi -q` for the Fabric sections. Nothing
+// is reported on a driver or a card without them.
+func DiscoverGPUFabric(ctx context.Context, binary string) (map[string]GPUFabric, error) {
+	if binary == "" {
+		if _, err := exec.LookPath("nvidia-smi"); err != nil {
+			return nil, nil
+		}
+		binary = "nvidia-smi"
+	}
+	ctx, cancel := context.WithTimeout(ctx, 20*time.Second)
+	defer cancel()
+	out, err := exec.CommandContext(ctx, binary, "-q").Output()
+	if err != nil {
+		return nil, err
+	}
+	return ParseGPUFabric(string(out)), nil
+}
+
+// ApplyFabric puts each GPU's fabric on it, matched by PCI address.
+func ApplyFabric(gpus []GPU, fab map[string]GPUFabric) {
+	for i := range gpus {
+		if f, ok := fab[strings.ToLower(gpus[i].PCIBusID)]; ok {
+			f := f
+			gpus[i].Fabric = &f
+		}
+	}
+}
+
+// IMEX is the service that lets GPUs in different nodes of one NVLink
+// domain share memory: without it, multi-node NVLink carries nothing.
+type IMEX struct {
+	// State is systemd's word for nvidia-imex: active, inactive, or empty
+	// where it is not installed.
+	State string `json:"state,omitempty"`
+	// Channels is how many IMEX channels exist; a process needs one.
+	Channels int `json:"channels"`
+}
+
+// DiscoverIMEX reads the IMEX service and its channels.
+func DiscoverIMEX(ctx context.Context) IMEX {
+	var m IMEX
+	if entries, err := os.ReadDir("/dev/nvidia-caps-imex-channels"); err == nil {
+		m.Channels = len(entries)
+	}
+	if _, err := exec.LookPath("systemctl"); err == nil {
+		ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
+		defer cancel()
+		out, _ := exec.CommandContext(ctx, "systemctl", "is-active", "nvidia-imex").Output()
+		m.State = strings.TrimSpace(string(out))
+		if m.State == "inactive" {
+			// "inactive" is also what systemd says of a unit that does not
+			// exist; the binary says which.
+			if _, err := exec.LookPath("nvidia-imex"); err != nil {
+				if _, err := os.Stat(filepath.Join("/usr/bin", "nvidia-imex")); err != nil {
+					m.State = ""
+				}
+			}
+		}
+	}
+	return m
+}

internal/agent/discovery/gpu_fabric_test.go

@@ -0,0 +1,77 @@
+package discovery
+
+import "testing"
+
+// smiQ is an excerpt of `nvidia-smi -q` on a GB200 tray: two GPUs joined to
+// one NVL72 domain, then a PCIe card that has no fabric at all.
+const smiQ = `
+==============NVSMI LOG==============
+
+Timestamp                                 : Mon Sep 29 10:00:00 2026
+Driver Version                            : 570.86.15
+Attached GPUs                             : 3
+GPU 00000008:01:00.0
+    Product Name                          : NVIDIA GB200
+    Minor Number                          : 0
+    Fabric
+        State                             : Completed
+        Status                            : Success
+        CliqueId                          : 32766
+        ClusterUUID                       : 76B1AD76-7c24-4b6b-9f6c-2a1e0d8c11aa
+        Health
+            Bandwidth                     : Full
+    Processes                             : None
+
+GPU 00000009:01:00.0
+    Product Name                          : NVIDIA GB200
+    Fabric
+        State                             : In Progress
+        Status                            : N/A
+        CliqueId                          : N/A
+        ClusterUUID                       : N/A
+    Processes                             : None
+
+GPU 0000000A:01:00.0
+    Product Name                          : NVIDIA L40S
+    Fabric
+        State                             : N/A
+        Status                            : N/A
+`
+
+func TestAGPUsNVLinkDomainIsRead(t *testing.T) {
+	fab := ParseGPUFabric(smiQ)
+	a, ok := fab["00000008:01:00.0"]
+	if !ok {
+		t.Fatalf("fabric = %+v", fab)
+	}
+	if a.State != "Completed" || a.CliqueID != "32766" || a.Bandwidth != "Full" {
+		t.Errorf("GB200 = %+v", a)
+	}
+	if got := a.Domain(); got != "76b1ad76-7c24-4b6b-9f6c-2a1e0d8c11aa/32766" {
+		t.Errorf("domain = %q", got)
+	}
+	// Still joining: no domain yet, and said as such.
+	b := fab["00000009:01:00.0"]
+	if b.State != "In Progress" || b.Domain() != "" {
+		t.Errorf("joining GPU = %+v", b)
+	}
+	// A card with no fabric reports nothing rather than an empty one.
+	if _, ok := fab["0000000a:01:00.0"]; ok {
+		t.Error("a card without a fabric was given one")
+	}
+
+	gpus := []GPU{{Index: 0, PCIBusID: "00000008:01:00.0"}, {Index: 1, PCIBusID: "0000000A:01:00.0"}}
+	ApplyFabric(gpus, fab)
+	if gpus[0].Fabric == nil || gpus[1].Fabric != nil {
+		t.Errorf("applied = %+v %+v", gpus[0].Fabric, gpus[1].Fabric)
+	}
+}
+
+func TestASingleMachinesNVSwitchIsNoDomain(t *testing.T) {
+	// An eight-GPU HGX reports its fabric joined, with a cluster of zeros: its
+	// NVLink ends at the chassis.
+	f := &GPUFabric{State: "Completed", ClusterUUID: "00000000-0000-0000-0000-000000000000", CliqueID: "0"}
+	if f.Domain() != "" {
+		t.Errorf("a single machine read as a domain: %q", f.Domain())
+	}
+}

internal/agent/discovery/gpu_smi.go

@@ -40,8 +40,8 @@ const mandatorySMIFields = 10
 //
 // This is the cgo-free path (plan §3.2): the NVML bindings need cgo, which
 // would force the whole binary — server included — to be built with a C
-// toolchain. nvidia-smi gives everything except NVLink topology, which the
-// NVML build adds.
+// toolchain. The query gives everything except topology, which Discover
+// reads from `nvidia-smi topo -m` afterwards.
 func DiscoverGPUsSMI(ctx context.Context, binary string) ([]GPU, error) {
 	if binary == "" {
 		// Not configured and not installed is a node without an NVIDIA card,

internal/agent/discovery/host.go

@@ -4,6 +4,7 @@ import (
 	"bufio"
 	"os"
 	"runtime"
+	"strconv"
 	"strings"
 )
 
@@ -26,6 +27,9 @@ type Host struct {
 	CPUFlags []string `json:"cpu_flags,omitempty"`
 	// NVIDIADriver is the loaded driver's version, empty when there is none.
 	NVIDIADriver string `json:"nvidia_driver,omitempty"`
+	// MinComputeCapability is the oldest GPU's, 0 when there are none or it
+	// is not known. Not read here but from the GPU inventory, by the agent.
+	MinComputeCapability float64 `json:"min_compute_capability,omitempty"`
 }
 
 // Family is the packaging family: "deb", "rpm", or "" when neither is known.
@@ -152,3 +156,19 @@ func nvidiaDriverFrom(path string) string {
 	}
 	return ""
 }
+
+// MinComputeCapability is the oldest GPU's compute capability, 0 when none
+// is known: an engine built for newer GPUs is refused by the oldest.
+func MinComputeCapability(gpus []GPU) float64 {
+	low := 0.0
+	for _, g := range gpus {
+		cc, err := strconv.ParseFloat(strings.TrimSpace(g.ComputeCapability), 64)
+		if err != nil || cc <= 0 {
+			continue
+		}
+		if low == 0 || cc < low {
+			low = cc
+		}
+	}
+	return low
+}

internal/agent/discovery/ib.go

@@ -5,6 +5,7 @@ package discovery
 import (
 	"errors"
 	"fmt"
+	"net"
 	"os"
 	"path/filepath"
 	"slices"
@@ -42,8 +43,66 @@ type Port struct {
 	GIDPrefix string `json:"gid_prefix"`
 	LID       int    `json:"lid"`
 	SMLID     int    `json:"sm_lid"`
+	// Counters are the port's error and traffic counters, by their sysfs
+	// names, as far as the driver has them. The data counters are converted
+	// to bytes; sysfs counts them in four-byte words.
+	Counters map[string]uint64 `json:"counters,omitempty"`
+	// RoCEv2GID is the GID table index NCCL should use on a RoCE port: the
+	// first RoCE v2 entry holding an IPv4 address, else the first RoCE v2
+	// entry. -1 on InfiniBand, and on a RoCE port with no v2 entry. NCCL's
+	// default of 0 is a RoCE v1 link-local entry on most setups, which does
+	// not route, and a job on it hangs.
+	RoCEv2GID int `json:"rocev2_gid"`
+	// PortGUID is the port's own identifier, the interface half of its GID.
+	PortGUID string `json:"port_guid,omitempty"`
+	// SM is the subnet manager this port sees as master, from sminfo. Nil on
+	// RoCE, which has none, and when infiniband-diags is not installed.
+	SM *SMInfo `json:"sm,omitempty"`
+	// Network is, on RoCE, the IP network of the RoCE v2 address: the only
+	// thing that says two RoCE ports are on the same fabric.
+	Network string `json:"network,omitempty"`
 }
 
+// SMInfo is a subnet manager as a port sees it.
+type SMInfo struct {
+	GUID     string `json:"guid"`
+	LID      int    `json:"lid"`
+	Priority int    `json:"priority"`
+	// State is SMINFO_MASTER, SMINFO_STANDBY, ...
+	State string `json:"state"`
+}
+
+// FabricKey is what identifies the physical fabric this port is on: the
+// master subnet manager on InfiniBand, since every port on one fabric sees
+// the same one and ports on separate fabrics cannot; the IP network on RoCE.
+// Empty when neither is known, and for a port that is not up.
+func (p Port) FabricKey() string {
+	if !p.Active() {
+		return ""
+	}
+	switch {
+	case p.LinkLayer == LinkEthernet && p.Network != "":
+		return "roce:" + p.Network
+	case p.SM != nil && p.SM.GUID != "":
+		return "sm:" + p.SM.GUID
+	}
+	return ""
+}
+
+// PortCounters are the counters worth watching, from each port's counters/
+// directory. The errors are the ones that say a cable, a transceiver or a
+// switch port is going bad before a job fails on it; the data counters say
+// whether the fabric is carrying anything at all.
+var PortCounters = []string{
+	"symbol_error", "link_error_recovery", "link_downed", "port_rcv_errors",
+	"port_rcv_remote_physical_errors", "port_xmit_discards",
+	"local_link_integrity_errors", "excessive_buffer_overrun_errors",
+	"port_xmit_data", "port_rcv_data",
+}
+
+// dataCounter reports whether a counter counts data in four-byte words.
+func dataCounter(name string) bool { return name == "port_xmit_data" || name == "port_rcv_data" }
+
 // Active reports whether the port can carry traffic.
 func (p Port) Active() bool { return strings.EqualFold(p.State, "ACTIVE") }
 
@@ -56,6 +115,9 @@ type HCA struct {
 	// FirmwareVersion is reported by the driver; worth capturing because
 	// NCCL behaviour differs between firmware revisions.
 	FirmwareVersion string `json:"firmware_version,omitempty"`
+	// NodeGUID identifies the adapter on its fabric: it is the name a
+	// switch's view of the fabric, and every other tool, knows it by.
+	NodeGUID string `json:"node_guid,omitempty"`
 }
 
 // ActivePorts returns the ports that can carry traffic.
@@ -102,6 +164,7 @@ func DiscoverIB(root string) ([]HCA, error) {
 func readHCA(dir, name string) (HCA, error) {
 	h := HCA{Name: name, NUMANode: -1}
 	h.FirmwareVersion = readTrimmed(filepath.Join(dir, "fw_ver"))
+	h.NodeGUID = normalizeGUID(readTrimmed(filepath.Join(dir, "node_guid")))
 
 	// device is a symlink into the PCI tree; its target's base name is the
 	// bus address, and numa_node lives beside it.
@@ -138,6 +201,30 @@ func readPort(dir string, num int) Port {
 		LID:       parseMaybeHex(readTrimmed(filepath.Join(dir, "lid"))),
 		SMLID:     parseMaybeHex(readTrimmed(filepath.Join(dir, "sm_lid"))),
 	}
+	p.RoCEv2GID = -1
+	p.PortGUID = portGUID(readTrimmed(filepath.Join(dir, "gids", "0")))
+	if p.LinkLayer == LinkEthernet {
+		p.RoCEv2GID = rocev2GID(dir)
+		if p.RoCEv2GID >= 0 {
+			idx := strconv.Itoa(p.RoCEv2GID)
+			p.Network = RoCENetwork(readTrimmed(filepath.Join(dir, "gid_attrs", "ndevs", idx)),
+				readTrimmed(filepath.Join(dir, "gids", idx)))
+		}
+	}
+	for _, name := range PortCounters {
+		raw := readTrimmed(filepath.Join(dir, "counters", name))
+		v, err := strconv.ParseUint(raw, 10, 64)
+		if err != nil {
+			continue // not every driver has every counter
+		}
+		if dataCounter(name) {
+			v *= 4
+		}
+		if p.Counters == nil {
+			p.Counters = map[string]uint64{}
+		}
+		p.Counters[name] = v
+	}
 	return p
 }
 
@@ -201,3 +288,91 @@ func readTrimmed(path string) string {
 	}
 	return strings.TrimSpace(string(b))
 }
+
+// rocev2GID finds the GID index a RoCE port's traffic should use.
+func rocev2GID(dir string) int {
+	first := -1
+	for i := 0; i < 64; i++ {
+		typ := readTrimmed(filepath.Join(dir, "gid_attrs", "types", strconv.Itoa(i)))
+		if !strings.EqualFold(typ, "RoCE v2") {
+			continue
+		}
+		gid := strings.ToLower(readTrimmed(filepath.Join(dir, "gids", strconv.Itoa(i))))
+		if strings.HasPrefix(gid, "0000:0000:0000:0000:0000:ffff:") {
+			return i
+		}
+		if first < 0 {
+			first = i
+		}
+	}
+	return first
+}
+
+// normalizeGUID writes a GUID one way: sysfs says "0c42:a103:0045:bd6c",
+// sminfo and ibnetdiscover say "0x0c42a1030045bd6c".
+func normalizeGUID(g string) string {
+	g = strings.ToLower(strings.TrimSpace(g))
+	g = strings.TrimPrefix(g, "0x")
+	g = strings.ReplaceAll(g, ":", "")
+	if g == "" || strings.Trim(g, "0") == "" {
+		return ""
+	}
+	for len(g) < 16 {
+		g = "0" + g
+	}
+	return "0x" + g
+}
+
+// NormalizeGUID is normalizeGUID for other packages.
+func NormalizeGUID(g string) string { return normalizeGUID(g) }
+
+// portGUID is the interface identifier of a GID: its last four groups.
+func portGUID(gid string) string {
+	parts := strings.Split(gid, ":")
+	if len(parts) != 8 {
+		return ""
+	}
+	return normalizeGUID(strings.Join(parts[4:], ""))
+}
+
+// RoCENetwork is the IP network an IPv4-mapped RoCE GID's address sits in,
+// read from the interface that carries it. Replaced in tests.
+var RoCENetwork = func(ndev, gid string) string {
+	ip := gidIPv4(gid)
+	if ip == nil || ndev == "" {
+		return ""
+	}
+	ifc, err := net.InterfaceByName(ndev)
+	if err != nil {
+		return ""
+	}
+	addrs, _ := ifc.Addrs()
+	for _, a := range addrs {
+		if n, ok := a.(*net.IPNet); ok && n.IP.Equal(ip) {
+			return (&net.IPNet{IP: ip.Mask(n.Mask), Mask: n.Mask}).String()
+		}
+	}
+	return ""
+}
+
+// gidIPv4 is the IPv4 address in an IPv4-mapped GID, nil otherwise.
+func gidIPv4(gid string) net.IP {
+	g := strings.ToLower(gid)
+	const mapped = "0000:0000:0000:0000:0000:ffff:"
+	if !strings.HasPrefix(g, mapped) {
+		return nil
+	}
+	hex := strings.ReplaceAll(strings.TrimPrefix(g, mapped), ":", "")
+	if len(hex) != 8 {
+		return nil
+	}
+	b := make(net.IP, 4)
+	for i := range 4 {
+		v, err := strconv.ParseUint(hex[i*2:i*2+2], 16, 8)
+		if err != nil {
+			return nil
+		}
+		b[i] = byte(v)
+	}
+	return b
+}

internal/agent/discovery/ib_test.go

@@ -31,6 +31,9 @@ func fakeSysfs(t *testing.T) string {
 	write("ib/mlx5_0/ports/1/gids/0", "FEC0:0000:0000:000A:0002:C903:00A0:1234")
 	write("ib/mlx5_0/ports/1/lid", "0x1a")
 	write("ib/mlx5_0/ports/1/sm_lid", "1")
+	write("ib/mlx5_0/ports/1/counters/symbol_error", "3")
+	write("ib/mlx5_0/ports/1/counters/link_downed", "1")
+	write("ib/mlx5_0/ports/1/counters/port_xmit_data", "1000")
 
 	// mlx5_1: two ports, the second one down.
 	write("ib/mlx5_1/ports/1/state", "4: ACTIVE")
@@ -46,7 +49,13 @@ func fakeSysfs(t *testing.T) string {
 	write("ib/mlx5_2/ports/1/state", "4: ACTIVE")
 	write("ib/mlx5_2/ports/1/rate", "100 Gb/sec (4X EDR)")
 	write("ib/mlx5_2/ports/1/link_layer", "Ethernet")
-	write("ib/mlx5_2/ports/1/gids/0", "0000:0000:0000:0000:0000:ffff:0a00:0001")
+	write("ib/mlx5_2/ports/1/gids/0", "fe80:0000:0000:0000:0000:0000:0000:0001")
+	write("ib/mlx5_2/ports/1/gid_attrs/types/0", "IB/RoCE v1")
+	write("ib/mlx5_2/ports/1/gids/1", "fe80:0000:0000:0000:0000:0000:0000:0001")
+	write("ib/mlx5_2/ports/1/gid_attrs/types/1", "RoCE v2")
+	write("ib/mlx5_2/ports/1/gids/3", "0000:0000:0000:0000:0000:ffff:0a00:0001")
+	write("ib/mlx5_2/ports/1/gid_attrs/types/3", "RoCE v2")
+	write("ib/mlx5_2/ports/1/gid_attrs/ndevs/3", "ens1f0")
 
 	// A PCI device with a NUMA node, linked the way the driver does it.
 	pci := filepath.Join(root, "pci", "0000:0b:00.0")
@@ -214,3 +223,131 @@ func TestParseHelpers(t *testing.T) {
 		}
 	})
 }
+
+func TestAPortsCountersAreRead(t *testing.T) {
+	hcas, err := DiscoverIB(fakeSysfs(t))
+	if err != nil {
+		t.Fatal(err)
+	}
+	c := hcas[0].Ports[0].Counters
+	// Data in bytes: sysfs counts four-byte words.
+	if c["symbol_error"] != 3 || c["link_downed"] != 1 || c["port_xmit_data"] != 4000 {
+		t.Errorf("counters = %v", c)
+	}
+	if _, ok := c["port_rcv_errors"]; ok {
+		t.Error("a counter the driver does not have should be absent, not zero")
+	}
+}
+
+func TestRoCEFindsItsRoutableGID(t *testing.T) {
+	// NCCL's default GID index is a v1 link-local entry on most hosts, which
+	// does not route: the job hangs. The v2 entry holding an IPv4 address is
+	// the one to use, over an earlier v2 link-local one.
+	hcas, err := DiscoverIB(fakeSysfs(t))
+	if err != nil {
+		t.Fatal(err)
+	}
+	for _, h := range hcas {
+		p := h.Ports[0]
+		switch h.Name {
+		case "mlx5_2":
+			if p.RoCEv2GID != 3 {
+				t.Errorf("RoCE GID index = %d, want 3", p.RoCEv2GID)
+			}
+		default:
+			if p.RoCEv2GID != -1 {
+				t.Errorf("%s is InfiniBand and has a GID index %d", h.Name, p.RoCEv2GID)
+			}
+		}
+	}
+}
+
+func TestTheSubnetManagerAPortSeesIsRead(t *testing.T) {
+	sm, err := ParseSMInfo("sminfo: sm lid 1 sm guid 0x98039B03009fcfd6, activity count 5452 priority 15 state 3 SMINFO_MASTER\n")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if sm.GUID != "0x98039b03009fcfd6" || sm.LID != 1 || sm.Priority != 15 || sm.State != "SMINFO_MASTER" {
+		t.Errorf("sm = %+v", sm)
+	}
+	if _, err := ParseSMInfo("ibwarn: [1234] mad_rpc_open_port: can't open UMAD port"); err == nil {
+		t.Error("an error from sminfo should not parse")
+	}
+}
+
+func TestAPortsFabricIsItsSubnetManagerOrItsNetwork(t *testing.T) {
+	// Two ports on one InfiniBand fabric see one master SM, and on separate
+	// fabrics cannot; RoCE has no SM, and is its IP network.
+	hcas, err := DiscoverIB(fakeSysfs(t))
+	if err != nil {
+		t.Fatal(err)
+	}
+	if hcas[0].NodeGUID != "" {
+		t.Errorf("no node_guid file, yet %q", hcas[0].NodeGUID)
+	}
+	err = AttachSM(hcas, func(hca string, port int) (*SMInfo, error) {
+		return &SMInfo{GUID: "0x00000000000000aa", LID: 1, State: "SMINFO_MASTER"}, nil
+	})
+	if err != nil {
+		t.Fatal(err)
+	}
+	if got := hcas[0].Ports[0].FabricKey(); got != "sm:0x00000000000000aa" {
+		t.Errorf("IB key = %q", got)
+	}
+	if got := hcas[1].Ports[1].FabricKey(); got != "" {
+		t.Errorf("a port that is down is on no fabric, got %q", got)
+	}
+	if hcas[2].Ports[0].SM != nil {
+		t.Error("RoCE was asked for a subnet manager")
+	}
+	if got := hcas[0].Ports[0].PortGUID; got != "0x0002c90300a01234" {
+		t.Errorf("port GUID = %q", got)
+	}
+}
+
+func TestARoCEPortsNetworkIsItsFabric(t *testing.T) {
+	old := RoCENetwork
+	t.Cleanup(func() { RoCENetwork = old })
+	RoCENetwork = func(ndev, gid string) string {
+		if ip := gidIPv4(gid); ip != nil && ip.String() == "10.0.0.1" {
+			return "10.0.0.0/24"
+		}
+		return ""
+	}
+	hcas, err := DiscoverIB(fakeSysfs(t))
+	if err != nil {
+		t.Fatal(err)
+	}
+	roce := hcas[2].Ports[0]
+	if roce.Network != "10.0.0.0/24" || roce.FabricKey() != "roce:10.0.0.0/24" {
+		t.Errorf("RoCE network %q key %q", roce.Network, roce.FabricKey())
+	}
+}
+
+// smirDump is `saquery SMIR` on a fabric with a master and a standby, in the
+// layout infiniband-diags prints.
+const smirDump = "SMInfoRecord dump:\n\t\tRID\n\t\tLID...................1\n\t\tSMInfo dump:\n" +
+	"\t\tGUID..................0x248a07030030f0c0\n\t\tSM_Key................0x0000000000000000\n" +
+	"\t\tActCount..............386773\n\t\tPriority..............15\n\t\tSMState...............3\n" +
+	"SMInfoRecord dump:\n\t\tRID\n\t\tLID...................12\n\t\tSMInfo dump:\n" +
+	"\t\tGUID..................0x248a07030030f1d4\n\t\tSM_Key................0x0000000000000000\n" +
+	"\t\tActCount..............12\n\t\tPriority..............14\n\t\tSMState...............2\n"
+
+func TestEverySubnetManagerOnAFabricIsListed(t *testing.T) {
+	recs, err := ParseSMIR(smirDump)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(recs) != 2 {
+		t.Fatalf("records = %+v", recs)
+	}
+	if recs[0].GUID != "0x248a07030030f0c0" || recs[0].State != "master" || recs[0].Priority != 15 || recs[0].LID != 1 {
+		t.Errorf("master = %+v", recs[0])
+	}
+	if recs[1].State != "standby" || recs[1].LID != 12 {
+		t.Errorf("standby = %+v", recs[1])
+	}
+	if _, err := ParseSMIR("ibwarn: can't open UMAD port"); err == nil {
+		t.Error("an error should not parse")
+	}
+}

internal/agent/discovery/load.go

@@ -0,0 +1,260 @@
+package discovery
+
+import (
+	"bufio"
+	"bytes"
+	"os"
+	"path/filepath"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+)
+
+// Load is how busy a node is: the machine as a whole, and each engine's share.
+type Load struct {
+	CPUPercent float64
+	CPUCount   int
+	Load1      float64
+	Load5      float64
+	Load15     float64
+	MemUsed    int64
+	MemAvail   int64
+	Shards     []ShardLoad
+}
+
+// ShardLoad is one engine process group's use.
+type ShardLoad struct {
+	ID       string
+	CPUCores float64
+	RSSBytes int64
+}
+
+// userHZ is the unit /proc counts CPU time in. Fixed at 100 by the kernel's
+// ABI on every architecture a node runs, whatever the internal tick rate.
+const userHZ = 100
+
+// LoadSampler measures load as the difference between two readings of /proc.
+//
+// CPU time is a counter, so a percentage needs two samples and the time
+// between them. The sampler keeps the previous one; the first call has
+// nothing to compare with and reports no load at all, rather than a zero that
+// would read as an idle machine.
+//
+// Linux only, like every node. Elsewhere the files are missing and it reports
+// nothing, as an older agent does.
+type LoadSampler struct {
+	root string
+	now  func() time.Time
+
+	mu      sync.Mutex
+	prevAt  time.Time
+	prevCPU cpuTimes
+	prevPG  map[int]groupUse // process group -> what it has used
+}
+
+// groupUse is a process group's CPU time so far and its resident memory now.
+type groupUse struct {
+	jiffies  uint64
+	rssPages int64
+}
+
+type cpuTimes struct{ total, idle uint64 }
+
+// NewLoadSampler reads the real /proc.
+func NewLoadSampler() *LoadSampler { return newLoadSampler("/proc", time.Now) }
+
+func newLoadSampler(root string, now func() time.Time) *LoadSampler {
+	return &LoadSampler{root: root, now: now}
+}
+
+// Sample reads the machine and the process groups led by the given shards'
+// PIDs, keyed by shard ID. False when there is nothing to report yet.
+func (s *LoadSampler) Sample(shards map[string]int) (Load, bool) {
+	cpu, ok := s.cpu()
+	if !ok {
+		return Load{}, false
+	}
+	groups := s.groups()
+	at := s.now()
+
+	s.mu.Lock()
+	prevAt, prevCPU, prevPG := s.prevAt, s.prevCPU, s.prevPG
+	s.prevAt, s.prevCPU, s.prevPG = at, cpu, groups
+	s.mu.Unlock()
+
+	if prevAt.IsZero() || cpu.total <= prevCPU.total {
+		return Load{}, false
+	}
+	l := Load{CPUCount: s.cpuCount()}
+	dTotal := float64(cpu.total - prevCPU.total)
+	dIdle := float64(cpu.idle - prevCPU.idle)
+	l.CPUPercent = clamp((dTotal-dIdle)/dTotal*100, 0, 100)
+	l.Load1, l.Load5, l.Load15 = s.loadavg()
+	total, avail := s.meminfo()
+	if total > 0 {
+		l.MemUsed, l.MemAvail = total-avail, avail
+	}
+
+	secs := at.Sub(prevAt).Seconds()
+	for id, pid := range shards {
+		if pid <= 0 {
+			continue
+		}
+		now := groups[pid]
+		sl := ShardLoad{ID: id, RSSBytes: now.rssPages * int64(os.Getpagesize())}
+		if before, ok := prevPG[pid]; ok && secs > 0 && now.jiffies >= before.jiffies {
+			sl.CPUCores = float64(now.jiffies-before.jiffies) / userHZ / secs
+		}
+		l.Shards = append(l.Shards, sl)
+	}
+	return l, true
+}
+
+func clamp(v, lo, hi float64) float64 { return max(lo, min(v, hi)) }
+
+// cpu reads the aggregate line of /proc/stat. Idle includes iowait: a core
+// waiting on the disk is not doing work.
+func (s *LoadSampler) cpu() (cpuTimes, bool) {
+	b, err := os.ReadFile(filepath.Join(s.root, "stat"))
+	if err != nil {
+		return cpuTimes{}, false
+	}
+	line, _, _ := bytes.Cut(b, []byte("\n"))
+	f := strings.Fields(string(line))
+	if len(f) < 5 || f[0] != "cpu" {
+		return cpuTimes{}, false
+	}
+	var t cpuTimes
+	for i, v := range f[1:] {
+		// guest and guest_nice are already counted in user and nice.
+		if i >= 8 {
+			break
+		}
+		n, _ := strconv.ParseUint(v, 10, 64)
+		t.total += n
+		if i == 3 || i == 4 {
+			t.idle += n
+		}
+	}
+	return t, true
+}
+
+func (s *LoadSampler) cpuCount() int {
+	b, err := os.ReadFile(filepath.Join(s.root, "stat"))
+	if err != nil {
+		return 0
+	}
+	n := 0
+	for _, line := range strings.Split(string(b), "\n") {
+		if strings.HasPrefix(line, "cpu") && len(line) > 3 && line[3] >= '0' && line[3] <= '9' {
+			n++
+		}
+	}
+	return n
+}
+
+func (s *LoadSampler) loadavg() (l1, l5, l15 float64) {
+	b, err := os.ReadFile(filepath.Join(s.root, "loadavg"))
+	if err != nil {
+		return 0, 0, 0
+	}
+	f := strings.Fields(string(b))
+	if len(f) < 3 {
+		return 0, 0, 0
+	}
+	l1, _ = strconv.ParseFloat(f[0], 64)
+	l5, _ = strconv.ParseFloat(f[1], 64)
+	l15, _ = strconv.ParseFloat(f[2], 64)
+	return l1, l5, l15
+}
+
+// meminfo returns MemTotal and MemAvailable in bytes.
+func (s *LoadSampler) meminfo() (total, avail int64) {
+	f, err := os.Open(filepath.Join(s.root, "meminfo"))
+	if err != nil {
+		return 0, 0
+	}
+	defer f.Close()
+	sc := bufio.NewScanner(f)
+	for sc.Scan() {
+		k, v, ok := strings.Cut(sc.Text(), ":")
+		if !ok {
+			continue
+		}
+		fields := strings.Fields(v)
+		if len(fields) == 0 {
+			continue
+		}
+		kb, _ := strconv.ParseInt(fields[0], 10, 64)
+		switch k {
+		case "MemTotal":
+			total = kb * 1024
+		case "MemAvailable":
+			avail = kb * 1024
+		}
+	}
+	return total, avail
+}
+
+// procStat is what one /proc/<pid>/stat says that load needs.
+type procStat struct {
+	pgrp     int
+	jiffies  uint64
+	rssPages int64
+}
+
+// readProcStat parses /proc/<pid>/stat. The command name is in parentheses
+// and may itself hold spaces and parentheses, so fields are counted from the
+// last closing one.
+func readProcStat(path string) (procStat, bool) {
+	b, err := os.ReadFile(path)
+	if err != nil {
+		return procStat{}, false
+	}
+	i := bytes.LastIndexByte(b, ')')
+	if i < 0 {
+		return procStat{}, false
+	}
+	f := strings.Fields(string(b[i+1:]))
+	// After the name: state(0) ppid(1) pgrp(2) ... utime(11) stime(12) ... rss(21)
+	if len(f) < 22 {
+		return procStat{}, false
+	}
+	var st procStat
+	st.pgrp, _ = strconv.Atoi(f[2])
+	ut, _ := strconv.ParseUint(f[11], 10, 64)
+	stt, _ := strconv.ParseUint(f[12], 10, 64)
+	st.jiffies = ut + stt
+	st.rssPages, _ = strconv.ParseInt(f[21], 10, 64)
+	return st, true
+}
+
+// groups totals CPU time and memory per process group, in one pass over
+// /proc. An engine is started as the leader of its own group, so its workers
+// — vLLM starts several — are counted with it.
+func (s *LoadSampler) groups() map[int]groupUse {
+	out := map[int]groupUse{}
+	s.eachProc(func(st procStat) {
+		g := out[st.pgrp]
+		g.jiffies += st.jiffies
+		g.rssPages += st.rssPages
+		out[st.pgrp] = g
+	})
+	return out
+}
+
+func (s *LoadSampler) eachProc(fn func(procStat)) {
+	entries, err := os.ReadDir(s.root)
+	if err != nil {
+		return
+	}
+	for _, e := range entries {
+		if _, err := strconv.Atoi(e.Name()); err != nil {
+			continue
+		}
+		if st, ok := readProcStat(filepath.Join(s.root, e.Name(), "stat")); ok {
+			fn(st)
+		}
+	}
+}

internal/agent/discovery/load_test.go

@@ -0,0 +1,96 @@
+package discovery
+
+import (
+	"fmt"
+	"os"
+	"path/filepath"
+	"testing"
+	"time"
+)
+
+// fakeProc is a /proc with two CPUs, a memory picture and processes.
+type fakeProc struct {
+	t    *testing.T
+	root string
+}
+
+func newFakeProc(t *testing.T) *fakeProc {
+	f := &fakeProc{t: t, root: t.TempDir()}
+	f.write("loadavg", "1.50 0.75 0.25 2/300 4242\n")
+	f.write("meminfo", "MemTotal:       8000000 kB\nMemFree:        1000000 kB\nMemAvailable:   3000000 kB\n")
+	return f
+}
+
+func (f *fakeProc) write(name, body string) {
+	f.t.Helper()
+	p := filepath.Join(f.root, name)
+	if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
+		f.t.Fatal(err)
+	}
+	if err := os.WriteFile(p, []byte(body), 0o644); err != nil {
+		f.t.Fatal(err)
+	}
+}
+
+// cpu sets the aggregate counters: user and idle, the rest zero.
+func (f *fakeProc) cpu(user, idle uint64) {
+	f.write("stat", fmt.Sprintf("cpu  %d 0 0 %d 0 0 0 0 0 0\ncpu0 0 0 0 0 0 0 0 0 0 0\ncpu1 0 0 0 0 0 0 0 0 0 0\nintr 1\n", user, idle))
+}
+
+// proc sets one process: its group, CPU jiffies (as utime) and RSS in pages.
+func (f *fakeProc) proc(pid, pgrp int, jiffies uint64, rss int64) {
+	// The name holds a space and a parenthesis, as real ones can.
+	f.write(fmt.Sprintf("%d/stat", pid), fmt.Sprintf(
+		"%d (llama-server (x)) S 1 %d %d 0 -1 0 0 0 0 0 %d 0 0 0 20 0 1 0 100 1000 %d 0\n",
+		pid, pgrp, pgrp, jiffies, rss))
+}
+
+func TestLoadIsTheDifferenceBetweenTwoReadings(t *testing.T) {
+	p := newFakeProc(t)
+	at := time.Unix(1000, 0)
+	s := newLoadSampler(p.root, func() time.Time { return at })
+
+	p.cpu(100, 900)
+	p.proc(42, 42, 1000, 10)
+	p.proc(43, 42, 500, 5) // a worker in the engine's group
+	p.proc(7, 7, 99999, 1) // something else entirely
+	if _, ok := s.Sample(map[string]int{"shard-a": 42}); ok {
+		t.Fatal("the first reading has nothing to compare with and should report nothing")
+	}
+
+	// Two seconds later: 300 of the 400 jiffies that passed were busy, and
+	// the engine's group used 400 of them.
+	at = at.Add(2 * time.Second)
+	p.cpu(400, 1000)
+	p.proc(42, 42, 1300, 10)
+	p.proc(43, 42, 600, 6)
+	l, ok := s.Sample(map[string]int{"shard-a": 42})
+	if !ok {
+		t.Fatal("the second reading should report")
+	}
+	if l.CPUPercent != 75 || l.CPUCount != 2 {
+		t.Errorf("cpu %.1f%% of %d, want 75%% of 2", l.CPUPercent, l.CPUCount)
+	}
+	if l.Load1 != 1.5 || l.Load15 != 0.25 {
+		t.Errorf("load %v %v", l.Load1, l.Load15)
+	}
+	if l.MemUsed != 5000000*1024 || l.MemAvail != 3000000*1024 {
+		t.Errorf("memory used %d available %d", l.MemUsed, l.MemAvail)
+	}
+	if len(l.Shards) != 1 {
+		t.Fatalf("shards = %+v", l.Shards)
+	}
+	sh := l.Shards[0]
+	// 400 jiffies over 2 seconds at 100 a second is two cores.
+	if sh.ID != "shard-a" || sh.CPUCores != 2 || sh.RSSBytes != 16*int64(os.Getpagesize()) {
+		t.Errorf("shard = %+v", sh)
+	}
+}
+
+func TestNoProcMeansNoLoad(t *testing.T) {
+	s := newLoadSampler(filepath.Join(t.TempDir(), "missing"), time.Now)
+	s.Sample(nil)
+	if _, ok := s.Sample(nil); ok {
+		t.Error("a system without /proc should report nothing, not zeros")
+	}
+}

internal/agent/discovery/perftest.go

@@ -0,0 +1,133 @@
+package discovery
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"os/exec"
+	"regexp"
+	"strconv"
+	"strings"
+	"time"
+)
+
+// WriteBW is one side of an ib_write_bw run, from perftest: an RDMA write
+// bandwidth test between two adapters, which is what a collective spanning
+// nodes rides on.
+type WriteBW struct {
+	HCA  string
+	Port int
+	// GIDIndex is the RoCE v2 GID to use; negative on InfiniBand, where the
+	// port's LID is used.
+	GIDIndex int
+	// Peer is the server's address, for the client; empty makes the server.
+	Peer    string
+	TCPPort int
+	Seconds int
+}
+
+// Args are the command line: a duration rather than a count, so the test
+// takes as long on a slow link as a fast one, and results in Gb/s.
+func (w WriteBW) Args() []string {
+	args := []string{"-d", w.HCA, "-i", strconv.Itoa(max(w.Port, 1)),
+		"-p", strconv.Itoa(w.TCPPort), "-D", strconv.Itoa(max(w.Seconds, 1)),
+		"-F", "--report_gbits"}
+	if w.GIDIndex >= 0 {
+		args = append(args, "-x", strconv.Itoa(w.GIDIndex))
+	}
+	if w.Peer != "" {
+		args = append(args, w.Peer)
+	}
+	return args
+}
+
+// RunWriteBW runs one side and returns the average bandwidth the output
+// reports, in Gb/s, and the output. The client retries its connection for a
+// while: the server is started first, but a few seconds may pass before it
+// listens.
+func RunWriteBW(ctx context.Context, w WriteBW) (float64, string, error) {
+	bin, err := exec.LookPath("ib_write_bw")
+	if err != nil {
+		return 0, "", errors.New("ib_write_bw is not installed (the perftest package)")
+	}
+	limit := time.Duration(w.Seconds+60) * time.Second
+	var out []byte
+	for attempt := 0; ; attempt++ {
+		cctx, cancel := context.WithTimeout(ctx, limit)
+		out, err = exec.CommandContext(cctx, bin, w.Args()...).CombinedOutput() //nolint:gosec // fixed tool
+		cancel()
+		if err == nil || w.Peer == "" || !refused(string(out)) || attempt >= 10 {
+			break
+		}
+		select {
+		case <-ctx.Done():
+			return 0, string(out), ctx.Err()
+		case <-time.After(time.Second):
+		}
+	}
+	text := string(out)
+	if err != nil {
+		return 0, text, fmt.Errorf("ib_write_bw: %v: %s", err, lastNonEmpty(text))
+	}
+	gbps, perr := ParseWriteBW(text)
+	if perr != nil && w.Peer != "" {
+		// Only the client is sure to print the result; a server that does not
+		// is not a failure.
+		return 0, text, perr
+	}
+	return gbps, text, nil
+}
+
+func refused(out string) bool {
+	for _, s := range []string{"Couldn't connect", "Unable to init the socket connection", "Connection refused"} {
+		if strings.Contains(out, s) {
+			return true
+		}
+	}
+	return false
+}
+
+var columnSplit = regexp.MustCompile(`\s{2,}`)
+
+// ParseWriteBW finds the average bandwidth in perftest's result table:
+//
+//	#bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
+//	65536      887036           0.00               96.93               0.184877
+//
+// by the heading's position rather than a fixed column, since versions add
+// columns and the peak reads 0.00 when the test runs for a duration.
+func ParseWriteBW(out string) (float64, error) {
+	lines := strings.Split(out, "\n")
+	for i, l := range lines {
+		if !strings.Contains(l, "BW average") {
+			continue
+		}
+		heads := columnSplit.Split(strings.TrimSpace(l), -1)
+		col := -1
+		for j, h := range heads {
+			if strings.HasPrefix(h, "BW average") {
+				col = j
+			}
+		}
+		for _, d := range lines[i+1:] {
+			f := strings.Fields(d)
+			if len(f) == 0 || strings.HasPrefix(f[0], "-") {
+				continue
+			}
+			if len(f) != len(heads) || col < 0 {
+				return 0, fmt.Errorf("perftest's result table has a shape this does not know: %q / %q", l, d)
+			}
+			v, err := strconv.ParseFloat(f[col], 64)
+			if err != nil {
+				return 0, fmt.Errorf("reading the average bandwidth %q: %w", f[col], err)
+			}
+			return v, nil
+		}
+	}
+	return 0, errors.New("perftest printed no result")
+}
+
+func lastNonEmpty(s string) string {
+	lines := strings.Split(strings.TrimSpace(s), "\n")
+	return strings.TrimSpace(lines[len(lines)-1])
+}

internal/agent/discovery/perftest_test.go

@@ -0,0 +1,49 @@
+package discovery
+
+import (
+	"slices"
+	"testing"
+)
+
+// The client's output of perftest 24.x, with a duration.
+const writeBWClient = `---------------------------------------------------------------------------------------
+                    RDMA_Write BW Test
+ Dual-port       : OFF		Device         : mlx5_0
+ Number of qps   : 1		Transport type : IB
+ Connection type : RC		Using SRQ      : OFF
+ TX depth        : 128
+ CQ Moderation   : 100
+ Mtu             : 4096[B]
+ Link type       : IB
+ Max inline data : 0[B]
+ rdma_cm QPs	 : OFF
+ Data ex. method : Ethernet
+---------------------------------------------------------------------------------------
+ local address: LID 0x05 QPN 0x0112 PSN 0x7b1f02 RKey 0x1fffbd VAddr 0x007f1d2a5f0000
+ remote address: LID 0x09 QPN 0x0113 PSN 0x2c1e44 RKey 0x1ffeaf VAddr 0x007fa4e1a60000
+---------------------------------------------------------------------------------------
+ #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
+ 65536      1851214          0.00               388.21		   0.740422
+---------------------------------------------------------------------------------------
+`
+
+func TestTheAverageBandwidthIsReadFromPerftest(t *testing.T) {
+	got, err := ParseWriteBW(writeBWClient)
+	if err != nil || got != 388.21 {
+		t.Errorf("got %v, %v", got, err)
+	}
+	if _, err := ParseWriteBW(" Couldn't connect to 10.0.0.2:18515\n"); err == nil {
+		t.Error("output with no result table gave a number")
+	}
+}
+
+func TestWriteBWArguments(t *testing.T) {
+	ib := WriteBW{HCA: "mlx5_0", Port: 1, GIDIndex: -1, TCPPort: 18515, Seconds: 3}.Args()
+	if slices.Contains(ib, "-x") {
+		t.Errorf("InfiniBand was given a GID index: %v", ib)
+	}
+	roce := WriteBW{HCA: "mlx5_0", Port: 1, GIDIndex: 3, Peer: "10.0.0.2", TCPPort: 18515, Seconds: 3}.Args()
+	if !slices.Contains(roce, "-x") || roce[len(roce)-1] != "10.0.0.2" {
+		t.Errorf("RoCE client: %v", roce)
+	}
+}

internal/agent/discovery/sminfo.go

@@ -0,0 +1,172 @@
+package discovery
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"os/exec"
+	"regexp"
+	"strconv"
+	"strings"
+	"time"
+)
+
+// ErrNoDiags means infiniband-diags is not installed, so the subnet manager
+// cannot be asked who it is.
+var ErrNoDiags = errors.New("sminfo not found: install infiniband-diags")
+
+// smLine is sminfo's one line of answer:
+//
+//	sminfo: sm lid 1 sm guid 0x98039b03009fcfd6, activity count 5452 priority 15 state 3 SMINFO_MASTER
+var smLine = regexp.MustCompile(`sm lid (\d+) sm guid (0x[0-9a-fA-F]+), activity count \d+ priority (\d+) state \d+ (\S+)`)
+
+// ParseSMInfo reads sminfo's output.
+func ParseSMInfo(out string) (*SMInfo, error) {
+	m := smLine.FindStringSubmatch(out)
+	if m == nil {
+		return nil, fmt.Errorf("unrecognised sminfo output: %q", strings.TrimSpace(out))
+	}
+	lid, _ := strconv.Atoi(m[1])
+	prio, _ := strconv.Atoi(m[3])
+	return &SMInfo{GUID: normalizeGUID(m[2]), LID: lid, Priority: prio, State: m[4]}, nil
+}
+
+// QuerySM asks, through one port, which subnet manager is master. It is a
+// management datagram to the SM, not a scan: cheap, but not free, which is
+// why the agent asks once a minute rather than every heartbeat.
+func QuerySM(ctx context.Context, hca string, port int) (*SMInfo, error) {
+	bin, err := exec.LookPath("sminfo")
+	if err != nil {
+		return nil, ErrNoDiags
+	}
+	ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
+	defer cancel()
+	out, err := exec.CommandContext(ctx, bin, "-C", hca, "-P", strconv.Itoa(port)).CombinedOutput()
+	if err != nil {
+		return nil, fmt.Errorf("sminfo on %s/%d: %w: %s", hca, port, err, strings.TrimSpace(string(out)))
+	}
+	return ParseSMInfo(string(out))
+}
+
+// AttachSM fills in the subnet manager each active InfiniBand port sees.
+// RoCE has none. The first error is returned for reporting; the ports it
+// could answer for keep their answer.
+func AttachSM(hcas []HCA, query func(hca string, port int) (*SMInfo, error)) error {
+	var first error
+	for i := range hcas {
+		for j := range hcas[i].Ports {
+			p := &hcas[i].Ports[j]
+			if !p.Active() || p.LinkLayer == LinkEthernet {
+				continue
+			}
+			sm, err := query(hcas[i].Name, p.Num)
+			if err != nil {
+				if first == nil {
+					first = err
+				}
+				continue
+			}
+			p.SM = sm
+		}
+	}
+	return first
+}
+
+// SMRecord is one subnet manager on a fabric, as the subnet administrator
+// lists them: master and standbys alike.
+type SMRecord struct {
+	GUID     string `json:"guid"`
+	LID      int    `json:"lid"`
+	Priority int    `json:"priority"`
+	// State is master, standby, discovering or not active.
+	State string `json:"state"`
+}
+
+// smStates are SMInfo's state numbers.
+var smStates = map[string]string{"0": "not active", "1": "discovering", "2": "standby", "3": "master"}
+
+// smirField is one line of saquery's record dump: "LID.........1".
+var smirField = regexp.MustCompile(`^\s*([A-Za-z_]+)\.{2,}\s*(\S+)`)
+
+// ParseSMIR reads `saquery SMIR`: every subnet manager the fabric's subnet
+// administrator knows of, one "SMInfoRecord dump:" block each.
+func ParseSMIR(out string) ([]SMRecord, error) {
+	var recs []SMRecord
+	var cur *SMRecord
+	for _, line := range strings.Split(out, "\n") {
+		if strings.Contains(line, "SMInfoRecord dump") {
+			recs = append(recs, SMRecord{})
+			cur = &recs[len(recs)-1]
+			continue
+		}
+		m := smirField.FindStringSubmatch(line)
+		if m == nil || cur == nil {
+			continue
+		}
+		switch strings.ToLower(m[1]) {
+		case "lid":
+			cur.LID, _ = strconv.Atoi(m[2])
+		case "guid":
+			cur.GUID = normalizeGUID(m[2])
+		case "priority":
+			cur.Priority, _ = strconv.Atoi(m[2])
+		case "smstate":
+			cur.State = smStates[m[2]]
+			if cur.State == "" {
+				cur.State = m[2]
+			}
+		}
+	}
+	out2 := recs[:0]
+	for _, r := range recs {
+		if r.GUID != "" {
+			out2 = append(out2, r)
+		}
+	}
+	if len(out2) == 0 {
+		return nil, fmt.Errorf("no subnet manager records in saquery output")
+	}
+	return out2, nil
+}
+
+// QuerySMs asks the subnet administrator, through one port, for every subnet
+// manager on the fabric.
+func QuerySMs(ctx context.Context, hca string, port int) ([]SMRecord, error) {
+	bin, err := exec.LookPath("saquery")
+	if err != nil {
+		return nil, ErrNoDiags
+	}
+	ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
+	defer cancel()
+	out, err := exec.CommandContext(ctx, bin, "-C", hca, "-P", strconv.Itoa(port), "SMIR").CombinedOutput()
+	if err != nil {
+		return nil, fmt.Errorf("saquery on %s/%d: %w: %s", hca, port, err, strings.TrimSpace(string(out)))
+	}
+	return ParseSMIR(string(out))
+}
+
+// ScanFabric maps the fabric a port is on with ibnetdiscover: every switch,
+// adapter and link. It can take a while on a large fabric, so it is bounded,
+// and its output is capped: a fabric of thousands of ports is a few
+// megabytes of text.
+func ScanFabric(ctx context.Context, hca string, port int) (string, error) {
+	bin, err := exec.LookPath("ibnetdiscover")
+	if err != nil {
+		return "", ErrNoDiags
+	}
+	ctx, cancel := context.WithTimeout(ctx, 2*time.Minute)
+	defer cancel()
+	out, err := exec.CommandContext(ctx, bin, "-C", hca, "-P", strconv.Itoa(port)).Output()
+	if err != nil {
+		var ee *exec.ExitError
+		if errors.As(err, &ee) {
+			return "", fmt.Errorf("ibnetdiscover on %s/%d: %w: %s", hca, port, err, strings.TrimSpace(string(ee.Stderr)))
+		}
+		return "", fmt.Errorf("ibnetdiscover on %s/%d: %w", hca, port, err)
+	}
+	const limit = 16 << 20
+	if len(out) > limit {
+		return "", fmt.Errorf("ibnetdiscover printed %d MB, more than the %d MB a scan may send", len(out)>>20, limit>>20)
+	}
+	return string(out), nil
+}

internal/agent/discovery/topo.go

@@ -0,0 +1,185 @@
+package discovery
+
+import (
+	"context"
+	"fmt"
+	"os/exec"
+	"regexp"
+	"sort"
+	"strconv"
+	"strings"
+)
+
+// Topology is what `nvidia-smi topo -m` says about how a node's GPUs and
+// network adapters are wired together.
+//
+// Read from nvidia-smi rather than NVML so the portable, cgo-free binary knows
+// it too. Without it that binary saw no NVLink at all: every GPU reported no
+// peers, and a tensor-parallel group was chosen by index on machines where
+// NVLink decides whether it runs at full speed.
+type Topology struct {
+	// NVLink is, per GPU, the GPUs it reaches over NVLink and how many links
+	// are bonded to each.
+	NVLink map[int]map[int]int
+	// NICPath is, per GPU, how far each network adapter is, by its HCA name:
+	// PIX, PXB, PHB, NODE or SYS, nearest first. A GPU should talk to the
+	// fabric through the adapter nearest it; one across the socket costs a
+	// trip over the processor interconnect on every transfer.
+	NICPath map[int]map[string]string
+	// NUMA is the NUMA node each GPU is attached to, where reported.
+	NUMA map[int]int
+}
+
+// PathRank orders the relations nvidia-smi reports, nearest first. Lower is
+// closer; an unknown relation ranks last.
+func PathRank(rel string) int {
+	switch strings.ToUpper(rel) {
+	case "PIX":
+		return 0
+	case "PXB":
+		return 1
+	case "PHB":
+		return 2
+	case "NODE":
+		return 3
+	case "SYS":
+		return 4
+	}
+	return 5
+}
+
+// DiscoverTopology runs `nvidia-smi topo -m`. A node without nvidia-smi has
+// no topology to report, which is not an error.
+func DiscoverTopology(ctx context.Context, binary string) (Topology, error) {
+	if binary == "" {
+		if _, err := exec.LookPath("nvidia-smi"); err != nil {
+			return Topology{}, nil
+		}
+		binary = "nvidia-smi"
+	}
+	out, err := exec.CommandContext(ctx, binary, "topo", "-m").Output()
+	if err != nil {
+		return Topology{}, fmt.Errorf("running %s topo -m: %w", binary, err)
+	}
+	return parseTopo(string(out))
+}
+
+// ParseTopology reads `nvidia-smi topo -m` output.
+func ParseTopology(out string) (Topology, error) { return parseTopo(out) }
+
+var (
+	ansi       = regexp.MustCompile(`\x1b\[[0-9;]*m`)
+	nvLinks    = regexp.MustCompile(`^NV(\d+)$`)
+	nicLegend  = regexp.MustCompile(`^\s*(NIC\d+)\s*:\s*(\S+)\s*$`)
+	gpuColumn  = regexp.MustCompile(`^GPU(\d+)$`)
+	numaColumn = "NUMA Affinity"
+)
+
+// parseTopo reads the matrix. Columns are tab-separated; the header names
+// each one — GPUn, then the adapters, as NICn with a legend below on newer
+// drivers or by HCA name on older ones — then CPU and NUMA affinity.
+func parseTopo(out string) (Topology, error) {
+	out = ansi.ReplaceAllString(out, "")
+	lines := strings.Split(out, "\n")
+
+	legend := map[string]string{}
+	for _, l := range lines {
+		if m := nicLegend.FindStringSubmatch(l); m != nil {
+			legend[m[1]] = m[2]
+		}
+	}
+
+	var header []string
+	start := -1
+	for i, l := range lines {
+		cells := splitTabs(l)
+		if len(cells) > 1 && cells[0] == "" && gpuColumn.MatchString(cells[1]) {
+			header, start = cells, i
+			break
+		}
+	}
+	if start < 0 {
+		return Topology{}, fmt.Errorf("no GPU matrix in nvidia-smi topo output")
+	}
+
+	t := Topology{NVLink: map[int]map[int]int{}, NICPath: map[int]map[string]string{}, NUMA: map[int]int{}}
+	for _, l := range lines[start+1:] {
+		cells := splitTabs(l)
+		if len(cells) < 2 {
+			// The matrix ends at the first blank line, before the legend.
+			if strings.TrimSpace(l) == "" {
+				break
+			}
+			continue
+		}
+		m := gpuColumn.FindStringSubmatch(cells[0])
+		if m == nil {
+			continue // an adapter's row: its view of the GPUs is the same matrix
+		}
+		gpu, _ := strconv.Atoi(m[1])
+		for c := 1; c < len(cells) && c < len(header); c++ {
+			col, v := header[c], strings.TrimSpace(cells[c])
+			switch {
+			case gpuColumn.MatchString(col):
+				peer, _ := strconv.Atoi(gpuColumn.FindStringSubmatch(col)[1])
+				if n := nvLinks.FindStringSubmatch(v); n != nil && peer != gpu {
+					links, _ := strconv.Atoi(n[1])
+					if t.NVLink[gpu] == nil {
+						t.NVLink[gpu] = map[int]int{}
+					}
+					t.NVLink[gpu][peer] = links
+				}
+			case col == numaColumn:
+				if n, err := strconv.Atoi(v); err == nil {
+					t.NUMA[gpu] = n
+				}
+			case col == "CPU Affinity" || col == "GPU NUMA ID" || col == "":
+			default:
+				// An adapter: NICn named by the legend, or an HCA by name.
+				hca := col
+				if name, ok := legend[col]; ok {
+					hca = name
+				}
+				if PathRank(v) < 5 {
+					if t.NICPath[gpu] == nil {
+						t.NICPath[gpu] = map[string]string{}
+					}
+					t.NICPath[gpu][hca] = strings.ToUpper(v)
+				}
+			}
+		}
+	}
+	if len(t.NVLink) == 0 && len(t.NICPath) == 0 && len(t.NUMA) == 0 {
+		return t, fmt.Errorf("the nvidia-smi topo matrix had no rows")
+	}
+	return t, nil
+}
+
+func splitTabs(l string) []string {
+	l = strings.TrimRight(l, "\r")
+	cells := strings.Split(l, "\t")
+	for i := range cells {
+		cells[i] = strings.TrimSpace(cells[i])
+	}
+	return cells
+}
+
+// Apply puts the topology on the GPUs it describes. What a GPU already knows
+// — NVLink peers from NVML, which is the more exact source — is kept.
+func (t Topology) Apply(gpus []GPU) {
+	for i := range gpus {
+		g := &gpus[i]
+		if len(g.NVLinkPeers) == 0 {
+			for peer := range t.NVLink[g.Index] {
+				g.NVLinkPeers = append(g.NVLinkPeers, peer)
+			}
+			sort.Ints(g.NVLinkPeers)
+		}
+		if n, ok := t.NUMA[g.Index]; ok {
+			g.NUMANode = n
+		}
+		if paths := t.NICPath[g.Index]; len(paths) > 0 {
+			g.NICPaths = paths
+		}
+	}
+}

internal/agent/discovery/topo_test.go

@@ -0,0 +1,113 @@
+package discovery
+
+import (
+	"reflect"
+	"strings"
+	"testing"
+)
+
+// row joins cells the way nvidia-smi prints them: tab-separated, each padded.
+func row(cells ...string) string { return strings.Join(cells, "\t") }
+
+// dgxH100 is `nvidia-smi topo -m` on an eight-GPU NVSwitch machine with two
+// adapters, as newer drivers print it: NICn columns and a legend naming them.
+var dgxH100 = strings.Join([]string{
+	row("\x1b[4m", "GPU0", "GPU1", "GPU2", "GPU3", "NIC0", "NIC1", "CPU Affinity", "NUMA Affinity", "GPU NUMA ID\x1b[0m"),
+	row("GPU0", " X ", "NV18", "NV18", "NV18", "PXB", "SYS", "0-55,112-167", "0", "N/A"),
+	row("GPU1", "NV18", " X ", "NV18", "NV18", "PXB", "SYS", "0-55,112-167", "0", "N/A"),
+	row("GPU2", "NV18", "NV18", " X ", "NV18", "SYS", "PXB", "56-111,168-223", "1", "N/A"),
+	row("GPU3", "NV18", "NV18", "NV18", " X ", "SYS", "PIX", "56-111,168-223", "1", "N/A"),
+	row("NIC0", "PXB", "PXB", "SYS", "SYS", " X ", "SYS"),
+	row("NIC1", "SYS", "SYS", "PXB", "PIX", "SYS", " X "),
+	"",
+	"Legend:",
+	"",
+	"  X    = Self",
+	"  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)",
+	"  NV#  = Connection traversing a bonded set of # NVLinks",
+	"",
+	"NIC Legend:",
+	"",
+	"  NIC0: mlx5_0",
+	"  NIC1: mlx5_1",
+	"",
+}, "\n")
+
+func TestNVLinkAndAdapterDistanceAreReadFromTopo(t *testing.T) {
+	topo, err := parseTopo(dgxH100)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if got := topo.NVLink[0]; !reflect.DeepEqual(got, map[int]int{1: 18, 2: 18, 3: 18}) {
+		t.Errorf("GPU0's NVLink peers = %v", got)
+	}
+	// The legend turns NIC1 into the adapter's own name.
+	if got := topo.NICPath[3]; !reflect.DeepEqual(got, map[string]string{"mlx5_0": "SYS", "mlx5_1": "PIX"}) {
+		t.Errorf("GPU3's adapters = %v", got)
+	}
+	if topo.NUMA[2] != 1 || topo.NUMA[0] != 0 {
+		t.Errorf("NUMA = %v", topo.NUMA)
+	}
+
+	gpus := []GPU{{Index: 0}, {Index: 3}}
+	topo.Apply(gpus)
+	if !reflect.DeepEqual(gpus[0].NVLinkPeers, []int{1, 2, 3}) || gpus[1].NUMANode != 1 ||
+		gpus[1].NICPaths["mlx5_1"] != "PIX" {
+		t.Errorf("applied = %+v", gpus)
+	}
+}
+
+func TestOlderDriversNameAdaptersInTheHeader(t *testing.T) {
+	// No legend: the columns are the adapters' own names, and NVLink between
+	// pairs only, as on an A100 PCIe box with bridges.
+	out := strings.Join([]string{
+		row("", "GPU0", "GPU1", "GPU2", "GPU3", "mlx5_0", "CPU Affinity", "NUMA Affinity"),
+		row("GPU0", " X ", "NV12", "SYS", "SYS", "NODE", "0-31", "0"),
+		row("GPU1", "NV12", " X ", "SYS", "SYS", "NODE", "0-31", "0"),
+		row("GPU2", "SYS", "SYS", " X ", "NV12", "SYS", "32-63", "1"),
+		row("GPU3", "SYS", "SYS", "NV12", " X ", "SYS", "32-63", "1"),
+		row("mlx5_0", "NODE", "NODE", "SYS", "SYS", " X "),
+		"",
+	}, "\n")
+	topo, err := parseTopo(out)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if !reflect.DeepEqual(topo.NVLink[2], map[int]int{3: 12}) || len(topo.NVLink[0]) != 1 {
+		t.Errorf("NVLink = %v", topo.NVLink)
+	}
+	if topo.NICPath[0]["mlx5_0"] != "NODE" {
+		t.Errorf("adapters = %v", topo.NICPath)
+	}
+}
+
+func TestAPCIeMachineHasNoNVLink(t *testing.T) {
+	out := strings.Join([]string{
+		row("", "GPU0", "GPU1", "CPU Affinity", "NUMA Affinity"),
+		row("GPU0", " X ", "PHB", "0-15", "0"),
+		row("GPU1", "PHB", " X ", "0-15", "0"),
+		"",
+	}, "\n")
+	topo, err := parseTopo(out)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(topo.NVLink) != 0 {
+		t.Errorf("NVLink on a PCIe machine: %v", topo.NVLink)
+	}
+}
+
+func TestNVMLPeersAreKeptOverTopo(t *testing.T) {
+	topo, _ := parseTopo(dgxH100)
+	gpus := []GPU{{Index: 0, NVLinkPeers: []int{1}}}
+	topo.Apply(gpus)
+	if !reflect.DeepEqual(gpus[0].NVLinkPeers, []int{1}) {
+		t.Errorf("NVML's answer was replaced: %v", gpus[0].NVLinkPeers)
+	}
+}
+
+func TestSomethingThatIsNotAMatrixIsAnError(t *testing.T) {
+	if _, err := parseTopo("NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver."); err == nil {
+		t.Error("expected an error")
+	}
+}

internal/agent/engines/engine.go

@@ -63,13 +63,24 @@ type ShardSpec struct {
 	HCAs []string
 	// MgmtInterface is the ethernet interface used for bootstrap traffic.
 	MgmtInterface string
+	// GIDIndex is the RoCE v2 GID index for the HCAs, set by the agent when
+	// they are RoCE; nil on InfiniBand, where NCCL needs none.
+	GIDIndex *int
 
 	HTTPPort int
 	Profile  Profile
 
+	// APIKey is what the engine demands of every request. Engines listen on
+	// all of a node's addresses; without a key anybody who can reach the port
+	// talks to the model directly, past the gateway's authentication, quotas
+	// and accounting. Given to the engine in its environment rather than on
+	// its command line, which every user on the node can read.
+	APIKey string `json:"api_key,omitempty"`
+
 	// Pool and Fabric are control plane concepts the agent cannot derive. They
-	// are carried so the agent can report them back after a restart, and are
-	// not used when building a command.
+	// are carried so the agent can report them back after a restart; Fabric
+	// also says, with an "nvl:" prefix, that the replica is inside a
+	// multi-node NVLink domain, which changes NCCL's environment.
 	Pool   string
 	Fabric string
 	// ProfileFP identifies the version of the profile this shard was started
@@ -193,6 +204,10 @@ type Config struct {
 
 // --- shared helpers ---
 
+// NCCLEnv is the environment a shard with this spec is given, for a tool that
+// shows an operator what an engine would see.
+func NCCLEnv(s ShardSpec) map[string]string { return baseEnv(s) }
+
 // baseEnv builds the environment every engine shares.
 //
 // This is the part of §5.3 the plan marks critical. The rules encoded here:
@@ -215,11 +230,24 @@ func baseEnv(s ShardSpec) map[string]string {
 	if len(s.HCAs) > 0 {
 		env["NCCL_IB_HCA"] = strings.Join(s.HCAs, ",")
 		env["NCCL_IB_DISABLE"] = "0"
+		// RoCE: the index of the routable RoCE v2 entry. NCCL's default is a
+		// v1 link-local one on most hosts, and a job on it hangs.
+		if s.GIDIndex != nil {
+			env["NCCL_IB_GID_INDEX"] = strconv.Itoa(*s.GIDIndex)
+		}
 	} else {
 		// Single node, or a node with no fabric. Either way IB is not the path.
 		env["NCCL_IB_DISABLE"] = "1"
 	}
 
+	// Placed inside a multi-node NVLink domain: NCCL is to carry the
+	// replica's collectives over NVLink between its nodes, which it only
+	// does when told to, and needs IMEX for.
+	if s.NNodes > 1 && strings.HasPrefix(s.Fabric, "nvl:") {
+		env["NCCL_MNNVL_ENABLE"] = "1"
+		env["NCCL_CUMEM_ENABLE"] = "1"
+	}
+
 	if s.MgmtInterface != "" {
 		env["NCCL_SOCKET_IFNAME"] = s.MgmtInterface
 		env["GLOO_SOCKET_IFNAME"] = s.MgmtInterface
@@ -228,6 +256,11 @@ func baseEnv(s ShardSpec) map[string]string {
 	// Profile overrides win: they are the escape hatch for the per-cluster
 	// tuning no adapter can guess.
 	for k, v := range s.Profile.Env {
+		// Checked again here, not only where the model was defined: this is
+		// the process about to run as root, whatever sent the profile.
+		if bad, _ := UnsafeEnv(k); bad {
+			continue
+		}
 		env[k] = v
 	}
 	return env

internal/agent/engines/engines_test.go

@@ -3,6 +3,7 @@ package engines
 import (
 	"fmt"
 	"slices"
+	"strconv"
 	"strings"
 	"testing"
 )
@@ -243,14 +244,41 @@ func TestVLLMRayWorkerRunsOnlyRay(t *testing.T) {
 }
 
 func TestVLLMMPMode(t *testing.T) {
+	// vLLM's multi-node launch: every node the same serve with its rank and
+	// the leader's address, workers headless. Workers used to be given the
+	// leader's address as their own VLLM_HOST_IP, and no rank at all.
 	v := &VLLM{cfg: VLLMConfig{LaunchMode: "mp"}}
-	cmd := buildOne(t, v, multiNodeSpec(0))
-	if argValue(cmd.Args, "--distributed-executor-backend") != "mp" {
-		t.Errorf("backend not set: %v", cmd.Args)
+	for _, rank := range []int{0, 1} {
+		cmd := buildOne(t, v, multiNodeSpec(rank))
+		for flag, want := range map[string]string{
+			"--distributed-executor-backend": "mp", "--nnodes": "2",
+			"--node-rank": strconv.Itoa(rank), "--master-addr": "10.0.0.10", "--master-port": "20001",
+		} {
+			if got := argValue(cmd.Args, flag); got != want {
+				t.Errorf("rank %d: %s = %q, want %q (%v)", rank, flag, got, want, cmd.Args)
+			}
+		}
+		if headless := slices.Contains(cmd.Args, "--headless"); headless != (rank > 0) {
+			t.Errorf("rank %d: headless = %v", rank, headless)
+		}
+		if ip, ok := envMap(cmd.Env)["VLLM_HOST_IP"]; ok && ip == "10.0.0.10" && rank > 0 {
+			t.Errorf("a worker was told the leader's address is its own")
+		}
 	}
-	env := envMap(cmd.Env)
-	if env["VLLM_HOST_IP"] != "10.0.0.10" || env["VLLM_PORT"] != "20001" {
-		t.Errorf("dist init not configured: %q %q", env["VLLM_HOST_IP"], env["VLLM_PORT"])
+}
+
+func TestRoCEGetsItsGIDIndex(t *testing.T) {
+	s := multiNodeSpec(0)
+	idx := 3
+	s.GIDIndex = &idx
+	env := envMap(buildOne(t, &VLLM{cfg: VLLMConfig{LaunchMode: "mp"}}, s).Env)
+	if env["NCCL_IB_GID_INDEX"] != "3" {
+		t.Errorf("NCCL_IB_GID_INDEX = %q", env["NCCL_IB_GID_INDEX"])
+	}
+	// InfiniBand needs none, and must not be given one.
+	env = envMap(buildOne(t, &VLLM{cfg: VLLMConfig{LaunchMode: "mp"}}, multiNodeSpec(0)).Env)
+	if _, ok := env["NCCL_IB_GID_INDEX"]; ok {
+		t.Error("an InfiniBand shard was given a GID index")
 	}
 }
 
@@ -423,3 +451,28 @@ func TestQuantizationOf(t *testing.T) {
 		t.Errorf("a dangling flag should yield nothing, got %q", got)
 	}
 }
+
+func TestAReplicaInAnNVLinkDomainIsToldToUseIt(t *testing.T) {
+	s := multiNodeSpec(0)
+	s.Fabric = "nvl:76b1ad76-7c24-4b6b-9f6c-2a1e0d8c11aa/7"
+	env := envMap(buildOne(t, &VLLM{cfg: VLLMConfig{LaunchMode: "mp"}}, s).Env)
+	if env["NCCL_MNNVL_ENABLE"] != "1" {
+		t.Errorf("NCCL_MNNVL_ENABLE = %q", env["NCCL_MNNVL_ENABLE"])
+	}
+	if env := envMap(buildOne(t, &VLLM{cfg: VLLMConfig{LaunchMode: "mp"}}, multiNodeSpec(0)).Env); env["NCCL_MNNVL_ENABLE"] != "" {
+		t.Error("a replica on InfiniBand was told to use multi-node NVLink")
+	}
+}
+
+func TestMultiNodeVLLMDefaultsToItsOwnLaunch(t *testing.T) {
+	// mp is what vLLM documents and needs nothing but vLLM; Ray was the
+	// default and was never checked against a real cluster.
+	cmds, err := (&VLLM{}).Commands(multiNodeSpec(1))
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(cmds) != 1 || cmds[0].Name != "vllm" || argValue(cmds[0].Args, "--distributed-executor-backend") != "mp" ||
+		!slices.Contains(cmds[0].Args, "--headless") {
+		t.Errorf("default worker = %+v", cmds)
+	}
+}

internal/agent/engines/llamacpp.go

@@ -120,8 +120,12 @@ func (l *LlamaCpp) Commands(s ShardSpec) ([]Command, error) {
 	}
 	args = append(args, p.Args...)
 
+	env := baseEnv(s)
+	if s.APIKey != "" {
+		env["LLAMA_API_KEY"] = s.APIKey
+	}
 	return []Command{{
-		Name: "llama-server", Path: bin, Args: args, Env: materializeEnv(baseEnv(s)),
+		Name: "llama-server", Path: bin, Args: args, Env: materializeEnv(env),
 	}}, nil
 }
 

internal/agent/engines/llamacpp_test.go

@@ -245,3 +245,19 @@ func TestAMissingPathIsLeftToTheEngine(t *testing.T) {
 		t.Errorf("expected it passed through, got %q %v", got, err)
 	}
 }
+
+func TestLlamaCppIsGivenItsKeyOutOfSightOfTheProcessList(t *testing.T) {
+	// The command line is readable by every user on the node; the
+	// environment of a root process is not.
+	cmds, err := (&LlamaCpp{}).Commands(ShardSpec{HTTPPort: 8080, APIKey: "k1",
+		Profile: Profile{ModelPath: "/m.gguf"}})
+	if err != nil {
+		t.Fatal(err)
+	}
+	if slices.Contains(cmds[0].Args, "k1") || slices.Contains(cmds[0].Args, "--api-key") {
+		t.Errorf("the key is on the command line: %v", cmds[0].Args)
+	}
+	if !slices.Contains(cmds[0].Env, "LLAMA_API_KEY=k1") {
+		t.Error("llama-server was not given its key")
+	}
+}

internal/agent/engines/probe.go

@@ -202,11 +202,7 @@ func (g *SGLang) Configured() bool { return g.cfg.Python != "" }
 
 // Probe reports whether vLLM is installed.
 func (v *VLLM) Probe(context.Context) (string, error) {
-	bin := v.cfg.Bin
-	if bin == "" {
-		bin = "vllm"
-	}
-	return installedBinary(bin)
+	return installedBinary(v.bin())
 }
 
 // Probe reports the SGLang version.
@@ -215,10 +211,6 @@ func (v *VLLM) Probe(context.Context) (string, error) {
 // asking the shell for "sglang" finds nothing on a machine that can run it
 // perfectly well.
 func (g *SGLang) Probe(ctx context.Context) (string, error) {
-	python := g.cfg.Python
-	if python == "" {
-		python = "python3"
-	}
-	return versionOfModule(ctx, python, "-c",
+	return versionOfModule(ctx, g.python(), "-c",
 		"import sglang, sys; sys.stdout.write(getattr(sglang, '__version__', ''))")
 }

internal/agent/engines/sglang.go

@@ -1,6 +1,9 @@
 package engines
 
-import "strconv"
+import (
+	"strconv"
+	"sync/atomic"
+)
 
 // SGLangConfig locates the SGLang installation.
 type SGLangConfig struct {
@@ -11,7 +14,25 @@ type SGLangConfig struct {
 }
 
 // SGLang builds SGLang launch commands.
-type SGLang struct{ cfg SGLangConfig }
+type SGLang struct {
+	cfg       SGLangConfig
+	installed atomic.Pointer[string]
+}
+
+// UseInstalled points the engine at the interpreter of the environment the
+// agent installed and checked, ahead of the configuration.
+func (g *SGLang) UseInstalled(path string) { g.installed.Store(&path) }
+
+// python is the interpreter to launch SGLang with.
+func (g *SGLang) python() string {
+	if p := g.installed.Load(); p != nil {
+		return *p
+	}
+	if g.cfg.Python != "" {
+		return g.cfg.Python
+	}
+	return "python3"
+}
 
 // Name identifies the engine in the catalog.
 func (*SGLang) Name() string { return "sglang" }
@@ -55,10 +76,7 @@ func (g *SGLang) Commands(s ShardSpec) ([]Command, error) {
 }
 
 func (g *SGLang) serverCommand(s ShardSpec) (Command, error) {
-	python := g.cfg.Python
-	if python == "" {
-		python = "python3"
-	}
+	python := g.python()
 	p := s.Profile
 
 	args := []string{"-m", "sglang.launch_server",
@@ -78,6 +96,11 @@ func (g *SGLang) serverCommand(s ShardSpec) (Command, error) {
 		)
 	}
 	args = append(args, p.Args...)
+	// SGLang reads its key only from the command line, so on a node with
+	// other users it is visible to them in the process list.
+	if s.APIKey != "" && s.Leader() {
+		args = append(args, "--api-key", s.APIKey)
+	}
 
 	return Command{Name: "sglang", Path: python, Args: args, Env: materializeEnv(baseEnv(s))}, nil
 }

internal/agent/engines/unsafeenv.go

@@ -0,0 +1,30 @@
+package engines
+
+import "strings"
+
+// UnsafeEnv reports whether a profile may not set an environment variable,
+// and why.
+//
+// A profile's env is the escape hatch for tuning no adapter can guess —
+// CUDA_*, NCCL_*, VLLM_*, HF_* — and it reaches an engine the agent starts as
+// root. Some variables are not tuning but a way to run other code in that
+// process: the dynamic loader's LD_PRELOAD, Python's PYTHONSTARTUP, a shell's
+// BASH_ENV. Set from a model definition, which an admin API or a config file
+// supplies, they would turn "add a model" into "run this as root on every
+// node". They are refused where a model is defined and again where an engine
+// starts.
+func UnsafeEnv(key string) (bool, string) {
+	k := strings.ToUpper(key)
+	switch {
+	case strings.HasPrefix(k, "LD_"), strings.HasPrefix(k, "DYLD_"):
+		return true, "it changes what the dynamic loader loads"
+	case k == "PATH":
+		return true, "it changes which binaries run"
+	case strings.HasPrefix(k, "PYTHON") && k != "PYTHONUNBUFFERED" && k != "PYTHONHASHSEED":
+		return true, "it changes what Python imports or runs at start"
+	case k == "BASH_ENV" || k == "ENV" || k == "PERL5OPT" || k == "PERL5LIB" ||
+		k == "NODE_OPTIONS" || k == "RUBYOPT" || k == "GCONV_PATH" || k == "OPENSSL_CONF":
+		return true, "it makes an interpreter or library load code"
+	}
+	return false, ""
+}

internal/agent/engines/vllm.go

@@ -3,24 +3,48 @@ package engines
 import (
 	"fmt"
 	"strconv"
+	"sync/atomic"
 )
 
 // VLLMConfig locates the vLLM installation and picks how it is launched.
 type VLLMConfig struct {
 	// Bin is the vllm executable.
 	Bin string `koanf:"bin"`
-	// LaunchMode selects the multi-node mechanism: "ray" or "mp".
+	// LaunchMode selects the multi-node mechanism: "mp", the default, or
+	// "ray".
 	//
-	// Which one works depends on the vLLM version, and the flags have changed
-	// more than once (plan §14). Both paths exist so a cluster can pick the one
-	// its version supports instead of waiting for a code change.
+	// mp is what vLLM documents today — every node runs serve with its rank,
+	// the workers headless — and needs nothing but vLLM. ray is the older way,
+	// kept for a vLLM too old for mp's multi-node flags; it needs Ray on every
+	// node and has not been checked against a real cluster.
 	LaunchMode string `koanf:"launch_mode"`
 	// RayBin is used when LaunchMode is "ray".
 	RayBin string `koanf:"ray_bin"`
 }
 
 // VLLM builds vLLM launch commands.
-type VLLM struct{ cfg VLLMConfig }
+type VLLM struct {
+	cfg       VLLMConfig
+	installed atomic.Pointer[string]
+}
+
+// UseInstalled points the engine at the environment the agent installed and
+// checked. It takes precedence over the configuration, as llama.cpp's does:
+// it is the one seen to run. An operator who wants their own sets the
+// control plane's engines.vllm to "".
+func (v *VLLM) UseInstalled(path string) { v.installed.Store(&path) }
+
+// bin is the vllm to run: the installed one, else the configured one, else
+// whatever PATH has.
+func (v *VLLM) bin() string {
+	if p := v.installed.Load(); p != nil {
+		return *p
+	}
+	if v.cfg.Bin != "" {
+		return v.cfg.Bin
+	}
+	return "vllm"
+}
 
 // Name identifies the engine in the catalog.
 func (*VLLM) Name() string { return "vllm" }
@@ -78,10 +102,7 @@ func (v *VLLM) Commands(s ShardSpec) ([]Command, error) {
 
 // serverCommand produces the vLLM server process.
 func (v *VLLM) serverCommand(s ShardSpec) (Command, error) {
-	bin := v.cfg.Bin
-	if bin == "" {
-		bin = "vllm"
-	}
+	bin := v.bin()
 	p := s.Profile
 
 	args := []string{"serve", p.ModelPath,
@@ -100,16 +121,27 @@ func (v *VLLM) serverCommand(s ShardSpec) (Command, error) {
 
 	if s.NNodes > 1 {
 		switch v.cfg.LaunchMode {
-		case "", "ray":
+		case "ray":
 			// Ray is started separately by the supervisor; vLLM then finds the
 			// cluster through the address in the environment.
 			env["VLLM_HOST_IP"] = s.LeaderAddr
 			env["RAY_ADDRESS"] = fmt.Sprintf("%s:%d", s.LeaderAddr, s.LeaderPort)
 			args = append(args, "--distributed-executor-backend", "ray")
-		case "mp":
-			env["VLLM_HOST_IP"] = s.LeaderAddr
-			env["VLLM_PORT"] = strconv.Itoa(s.LeaderPort)
-			args = append(args, "--distributed-executor-backend", "mp")
+		case "", "mp":
+			// vLLM's own multi-node launch: every node runs the same serve
+			// with its rank and the leader's address, and the workers run
+			// headless — no API server, only their GPUs. VLLM_HOST_IP is
+			// this node's address for vLLM to find itself, so it is left
+			// to vLLM; the leader's, which workers used to be given, made
+			// every worker announce itself as the leader.
+			args = append(args, "--distributed-executor-backend", "mp",
+				"--nnodes", strconv.Itoa(s.NNodes),
+				"--node-rank", strconv.Itoa(s.Rank),
+				"--master-addr", s.LeaderAddr,
+				"--master-port", strconv.Itoa(s.LeaderPort))
+			if !s.Leader() {
+				args = append(args, "--headless")
+			}
 		default:
 			return Command{}, fmt.Errorf("vllm: unknown launch_mode %q (want \"ray\" or \"mp\")", v.cfg.LaunchMode)
 		}
@@ -118,13 +150,16 @@ func (v *VLLM) serverCommand(s ShardSpec) (Command, error) {
 	// Profile arguments last, so an operator can override anything above.
 	args = append(args, p.Args...)
 
+	if s.APIKey != "" {
+		env["VLLM_API_KEY"] = s.APIKey
+	}
 	return Command{Name: "vllm", Path: bin, Args: args, Env: materializeEnv(env)}, nil
 }
 
 // rayCommand returns the command that starts or joins the Ray cluster, or
 // ok=false when the launch mode does not use Ray.
 func (v *VLLM) rayCommand(s ShardSpec) (Command, bool) {
-	if s.NNodes <= 1 || (v.cfg.LaunchMode != "" && v.cfg.LaunchMode != "ray") {
+	if s.NNodes <= 1 || v.cfg.LaunchMode != "ray" {
 		return Command{}, false
 	}
 	bin := v.cfg.RayBin

internal/agent/provision/install.go

@@ -16,6 +16,7 @@ import (
 	"os"
 	"os/exec"
 	"path/filepath"
+	"regexp"
 	"strings"
 	"time"
 
@@ -34,6 +35,9 @@ const (
 	Ready       = "ready"
 	Failed      = "failed"
 	Unsupported = "unsupported"
+	// Skipped is an engine this node is not meant to run, such as vLLM on a
+	// machine with no GPU. Nothing is wrong, and nothing is shown as wrong.
+	Skipped = "skipped"
 )
 
 // Status is where one engine's installation stands.
@@ -62,11 +66,22 @@ type Installer struct {
 
 	// Run executes a command; a seam for tests. Nil runs it for real.
 	Run func(ctx context.Context, env []string, name string, args ...string) ([]byte, error)
-	// APIBase and DownloadBase default to GitHub; tests point them at a fake.
+	// FreeSpace reports the gigabytes free under a directory; a seam for
+	// tests. Nil asks the filesystem.
+	FreeSpace func(dir string) (float64, error)
+	// RunStdin is Run with input; a seam for tests. Nil runs it for real.
+	RunStdin func(ctx context.Context, env []string, stdin, name string, args ...string) ([]byte, error)
+	// APIBase and DownloadBase default to GitHub, PyPIBase to PyPI; tests
+	// point them at a fake.
 	APIBase      string
 	DownloadBase string
+	PyPIBase     string
 }
 
+// validVersion is what a version may look like: "b10991", "0.29.0",
+// "0.11.0+cu128". No separators, and not starting with a dot.
+var validVersion = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._+-]{0,63}$`)
+
 // DefaultRoot is where engines are installed when nothing says otherwise.
 const DefaultRoot = "/opt/llmgw/engines"
 
@@ -75,12 +90,16 @@ const DefaultRoot = "/opt/llmgw/engines"
 const marker = ".llmgw-installed"
 
 type markerFile struct {
-	Engine    string            `json:"engine"`
-	Version   string            `json:"version"`
-	Variant   string            `json:"variant"`
-	Binary    string            `json:"binary"`
-	SHA256    map[string]string `json:"sha256"`
-	Installed time.Time         `json:"installed"`
+	Engine  string            `json:"engine"`
+	Version string            `json:"version"`
+	Variant string            `json:"variant"`
+	Binary  string            `json:"binary"`
+	SHA256  map[string]string `json:"sha256,omitempty"`
+	// For a Python engine: what was asked for, PyPI as of when, by which uv.
+	Requirements []string  `json:"requirements,omitempty"`
+	ExcludeNewer string    `json:"exclude_newer,omitempty"`
+	UV           string    `json:"uv,omitempty"`
+	Installed    time.Time `json:"installed"`
 }
 
 // Converge brings every wanted engine to ready, or says why it cannot.
@@ -100,6 +119,15 @@ func (in *Installer) Converge(ctx context.Context, wants []Want) []Status {
 
 func (in *Installer) one(ctx context.Context, w Want) Status {
 	st := Status{Engine: w.Engine, Version: w.Version}
+	// Checked here, on the node, whatever the control plane checked: the
+	// version names a directory under the install root and is passed to a
+	// package manager, and "../x" would put an install — and the RemoveAll
+	// that replaces one — outside it, as root.
+	if !validVersion.MatchString(w.Version) {
+		st.State = Unsupported
+		st.Detail = fmt.Sprintf("%q is not a version this agent will install", w.Version)
+		return st
+	}
 	recipe, ok := Recipes[w.Engine]
 	if !ok {
 		st.State = Unsupported
@@ -110,6 +138,10 @@ func (in *Installer) one(ctx context.Context, w Want) Status {
 	build, err := recipe.Choose(w.Version, in.Host)
 	if err != nil {
 		st.State, st.Detail = Unsupported, err.Error()
+		var nh NotHere
+		if errors.As(err, &nh) {
+			st.State = Skipped
+		}
 		return st
 	}
 	st.Variant = build.Variant
@@ -168,6 +200,9 @@ func (in *Installer) ready(st Status, b Build, bin, version string) Status {
 // Into a scratch directory renamed into place at the end, so a failure halfway
 // leaves nothing that looks installed.
 func (in *Installer) install(ctx context.Context, r Recipe, w Want, b Build, dir string, st *Status) error {
+	if b.Python != "" {
+		return in.installPython(ctx, r, w, b, dir, st)
+	}
 	if err := in.packages(ctx, b.Packages, st); err != nil {
 		return err
 	}
@@ -235,18 +270,32 @@ func (in *Installer) install(ctx context.Context, r Recipe, w Want, b Build, dir
 // running it is the only proof that the rest — the right instruction set, the
 // right driver — is in place.
 func (in *Installer) verify(ctx context.Context, dir string, b Build) (bin, version string, err error) {
-	bin, err = findFile(dir, b.Binary)
-	if err != nil {
-		return "", "", err
-	}
-	if out, err := in.run(ctx, nil, "ldd", bin); err == nil {
-		if missing := lddMissing(string(out)); len(missing) > 0 {
-			return "", "", fmt.Errorf("%s cannot run on this system: %s", b.Binary, strings.Join(missing, "; "))
+	timeout := 60 * time.Second
+	if b.Python != "" {
+		// A script in the environment, not a binary to ldd; and importing
+		// torch and the engine behind it takes a while on a cold disk.
+		bin, timeout = filepath.Join(dir, "bin", b.Binary), 3*time.Minute
+		if _, err := os.Stat(bin); err != nil {
+			return "", "", fmt.Errorf("%s is not in the environment: %w", b.Binary, err)
+		}
+	} else {
+		bin, err = findFile(dir, b.Binary)
+		if err != nil {
+			return "", "", err
+		}
+		if out, err := in.run(ctx, nil, "ldd", bin); err == nil {
+			if missing := lddMissing(string(out)); len(missing) > 0 {
+				return "", "", fmt.Errorf("%s cannot run on this system: %s", b.Binary, strings.Join(missing, "; "))
+			}
 		}
 	}
-	tctx, cancel := context.WithTimeout(ctx, 60*time.Second)
+	args := b.VersionArgs
+	if len(args) == 0 {
+		args = []string{"--version"}
+	}
+	tctx, cancel := context.WithTimeout(ctx, timeout)
 	defer cancel()
-	out, err := in.run(tctx, nil, bin, "--version")
+	out, err := in.run(tctx, nil, bin, args...)
 	if err != nil {
 		return "", "", fmt.Errorf("%s --version failed: %v: %s", b.Binary, err, lastLine(out))
 	}
@@ -607,6 +656,13 @@ func (in *Installer) report(st Status) {
 	}
 }
 
+func runWithStdin(ctx context.Context, env []string, stdin, name string, args ...string) ([]byte, error) {
+	cmd := exec.CommandContext(ctx, name, args...) //nolint:gosec // fixed commands and paths we installed
+	cmd.Env = append(os.Environ(), env...)
+	cmd.Stdin = strings.NewReader(stdin)
+	return cmd.CombinedOutput()
+}
+
 func (in *Installer) run(ctx context.Context, env []string, name string, args ...string) ([]byte, error) {
 	if in.Run != nil {
 		return in.Run(ctx, env, name, args...)
@@ -662,5 +718,11 @@ func firstVersionLine(b []byte) string {
 			return strings.TrimSpace(l)
 		}
 	}
+	// vllm --version says the number and nothing else.
+	for _, l := range strings.Split(string(b), "\n") {
+		if l = strings.TrimSpace(l); l != "" && l[0] >= '0' && l[0] <= '9' {
+			return l
+		}
+	}
 	return ""
 }

internal/agent/provision/install_test.go

@@ -265,8 +265,8 @@ func TestTheBuildFollowsTheHardware(t *testing.T) {
 
 func TestAnEngineTheAgentCannotInstallSaysSo(t *testing.T) {
 	in := &Installer{Root: t.TempDir(), Host: ubuntu}
-	got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.29.0"}})
-	if got[0].State != Unsupported {
+	got := in.Converge(t.Context(), []Want{{Engine: "tgi", Version: "3.0.0"}})
+	if got[0].State != Unsupported || !strings.Contains(got[0].Detail, "install it by hand") {
 		t.Errorf("got %+v", got[0])
 	}
 }
@@ -330,3 +330,15 @@ func TestAPackageManagerBusyWithItsOwnUpdatesIsWaitedFor(t *testing.T) {
 		t.Error("the wait was not reported")
 	}
 }
+
+func TestAVersionThatIsAPathIsRefused(t *testing.T) {
+	// The version names a directory under the install root; "../" would put
+	// an install, and the RemoveAll that replaces one, outside it, as root.
+	in := &Installer{Root: t.TempDir(), Host: ubuntu}
+	for _, v := range []string{"../../etc", "..", ".hidden", "b1/../../x", "", "b1 && rm -rf /"} {
+		got := in.Converge(t.Context(), []Want{{Engine: "llamacpp", Version: v}})
+		if got[0].State != Unsupported {
+			t.Errorf("version %q: %+v", v, got[0])
+		}
+	}
+}

internal/agent/provision/python.go

@@ -0,0 +1,205 @@
+package provision
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"net/http"
+	"os"
+	"path/filepath"
+	"strings"
+	"syscall"
+	"time"
+)
+
+// A Python engine is a virtual environment built by uv: an interpreter uv
+// fetches, and the engine with every dependency from PyPI.
+//
+// Two things make it as trustworthy as an archive checked against its
+// publisher's checksum. The dependencies are resolved as PyPI stood a day
+// after the engine was published (--exclude-newer), so a node built next
+// month gets what one built today got, not whatever a dependency released
+// since — the usual way a working vLLM stops working. And the resolution is
+// written out with every file's sha256 from PyPI and installed with
+// --require-hashes, so a file that does not match what PyPI published is
+// refused. The resolved list is kept in the environment, as the record of
+// exactly what is installed.
+
+// pythonInstallTimeout bounds one installation: several gigabytes, on a
+// network that may be slow, but not forever.
+const pythonInstallTimeout = time.Hour
+
+// lockName is the resolved requirements, kept in the environment.
+const lockName = "llmgw-requirements.txt"
+
+func (in *Installer) installPython(ctx context.Context, r Recipe, w Want, b Build, dir string, st *Status) error {
+	ctx, cancel := context.WithTimeout(ctx, pythonInstallTimeout)
+	defer cancel()
+
+	if err := in.packages(ctx, b.Packages, st); err != nil {
+		return err
+	}
+	if err := os.MkdirAll(in.root(), 0o755); err != nil {
+		return err
+	}
+	if b.FreeGB > 0 {
+		if free, err := in.free(in.root()); err == nil && free < float64(b.FreeGB) {
+			return fmt.Errorf("%s needs about %d GB free under %s and there are %.1f GB",
+				w.Engine, b.FreeGB, in.root(), free)
+		}
+	}
+
+	in.progress(st, "installing uv %s", UVVersion)
+	uv := in.one(ctx, Want{Engine: "uv", Version: UVVersion})
+	if uv.State != Ready {
+		return fmt.Errorf("installing uv, which installs %s: %s", w.Engine, uv.Detail)
+	}
+
+	in.progress(st, "asking PyPI when %s %s was published", r.PyPI, w.Version)
+	published, err := in.published(ctx, r.PyPI, w.Version)
+	if err != nil {
+		return err
+	}
+	asOf := published.Add(24 * time.Hour).UTC().Format(time.RFC3339)
+
+	cache := filepath.Join(in.root(), ".uv-cache")
+	env := []string{
+		"UV_CACHE_DIR=" + cache,
+		"UV_PYTHON_INSTALL_DIR=" + filepath.Join(in.root(), "python"),
+		// Only the interpreter uv fetched: the system's may be the wrong
+		// version, or change under the environment with the next upgrade.
+		"UV_PYTHON_PREFERENCE=only-managed",
+		// Nobody's uv.toml or pip.conf decides what is installed here.
+		"UV_NO_CONFIG=1",
+	}
+	tmp := dir + ".partial"
+	_ = os.RemoveAll(tmp)
+	defer os.RemoveAll(tmp) //nolint:errcheck // gone either way once renamed
+
+	in.progress(st, "creating a Python %s environment for %s", b.Python, w.Engine)
+	// Relocatable, because it is built under one name and renamed into place
+	// once it works: its scripts find their interpreter relative to
+	// themselves rather than by the path they were built at.
+	if out, err := in.run(ctx, env, uv.Binary, "venv", "--relocatable", "--python", b.Python, tmp); err != nil {
+		return fmt.Errorf("uv venv: %v: %s", err, tail(out))
+	}
+	python := filepath.Join(tmp, "bin", "python")
+
+	in.progress(st, "resolving %s against PyPI as of %s", strings.Join(b.Requirements, " "), asOf[:10])
+	lock := filepath.Join(tmp, lockName)
+	args := []string{"pip", "compile", "--python", python, "--exclude-newer", asOf,
+		"--generate-hashes", "--quiet", "--output-file", lock, "-"}
+	if out, err := in.runStdin(ctx, env, strings.Join(b.Requirements, "\n")+"\n", uv.Binary, args...); err != nil {
+		return fmt.Errorf("resolving %s: %v: %s", strings.Join(b.Requirements, " "), err, tail(out))
+	}
+
+	in.progress(st, "downloading and installing %s and its dependencies: several gigabytes, which takes minutes",
+		strings.Join(b.Requirements, " "))
+	if out, err := in.run(ctx, env, uv.Binary, "pip", "install", "--python", python,
+		"--require-hashes", "--no-deps", "-r", lock); err != nil {
+		return fmt.Errorf("installing %s: %v: %s", strings.Join(b.Requirements, " "), err, tail(out))
+	}
+
+	in.progress(st, "checking that %s runs", w.Engine)
+	if _, _, err := in.verify(ctx, tmp, b); err != nil {
+		return err
+	}
+	m, _ := json.MarshalIndent(markerFile{
+		Engine: w.Engine, Version: w.Version, Variant: b.Variant,
+		Binary: filepath.Join("bin", b.Binary), Requirements: b.Requirements,
+		ExcludeNewer: asOf, UV: UVVersion, Installed: time.Now().UTC(),
+	}, "", "  ")
+	if err := os.WriteFile(filepath.Join(tmp, marker), m, 0o644); err != nil { //nolint:gosec // not secret
+		return err
+	}
+	_ = os.RemoveAll(dir)
+	if err := os.Rename(tmp, dir); err != nil {
+		return err
+	}
+	// The cache is as large as the environment. Kept after a failure, so the
+	// next attempt does not download it all again; dropped after success.
+	_ = os.RemoveAll(cache)
+	in.log().Info("installed an engine", "engine", w.Engine, "version", w.Version,
+		"variant", b.Variant, "dir", dir, "as_of", asOf)
+	return nil
+}
+
+// published is when PyPI received a release's last file.
+func (in *Installer) published(ctx context.Context, project, version string) (time.Time, error) {
+	url := strings.TrimSuffix(in.pypiBase(), "/") + "/pypi/" + project + "/" + version + "/json"
+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
+	if err != nil {
+		return time.Time{}, err
+	}
+	resp, err := in.client().Do(req)
+	if err != nil {
+		return time.Time{}, fmt.Errorf("asking PyPI for %s %s: %w", project, version, err)
+	}
+	defer resp.Body.Close() //nolint:errcheck // read-only
+	if resp.StatusCode == http.StatusNotFound {
+		return time.Time{}, fmt.Errorf("PyPI has no %s %s", project, version)
+	}
+	if resp.StatusCode != http.StatusOK {
+		return time.Time{}, fmt.Errorf("asking PyPI for %s %s: %s", project, version, resp.Status)
+	}
+	var rel struct {
+		URLs []struct {
+			Uploaded time.Time `json:"upload_time_iso_8601"`
+		} `json:"urls"`
+	}
+	if err := json.NewDecoder(resp.Body).Decode(&rel); err != nil {
+		return time.Time{}, fmt.Errorf("reading PyPI's answer for %s %s: %w", project, version, err)
+	}
+	var last time.Time
+	for _, u := range rel.URLs {
+		if u.Uploaded.After(last) {
+			last = u.Uploaded
+		}
+	}
+	if last.IsZero() {
+		return time.Time{}, errors.New("PyPI lists no files for " + project + " " + version)
+	}
+	return last, nil
+}
+
+func (in *Installer) free(dir string) (float64, error) {
+	if in.FreeSpace != nil {
+		return in.FreeSpace(dir)
+	}
+	return FreeGB(dir)
+}
+
+func (in *Installer) pypiBase() string {
+	if in.PyPIBase != "" {
+		return in.PyPIBase
+	}
+	return "https://pypi.org"
+}
+
+// runStdin runs a command with input on its standard input.
+func (in *Installer) runStdin(ctx context.Context, env []string, stdin, name string, args ...string) ([]byte, error) {
+	if in.RunStdin != nil {
+		return in.RunStdin(ctx, env, stdin, name, args...)
+	}
+	return runWithStdin(ctx, env, stdin, name, args...)
+}
+
+// FreeGB is the space available to an unprivileged writer under dir.
+func FreeGB(dir string) (float64, error) {
+	var s syscall.Statfs_t
+	if err := syscall.Statfs(dir, &s); err != nil {
+		return 0, err
+	}
+	return float64(s.Bavail) * float64(s.Bsize) / (1 << 30), nil
+}
+
+// tail is the last few lines of a command's output: uv explains a failed
+// resolution over several, and the last alone is rarely the reason.
+func tail(b []byte) string {
+	lines := strings.Split(strings.TrimSpace(string(b)), "\n")
+	if len(lines) > 4 {
+		lines = lines[len(lines)-4:]
+	}
+	return strings.Join(lines, " / ")
+}

internal/agent/provision/python_test.go

@@ -0,0 +1,184 @@
+package provision
+
+import (
+	"context"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"slices"
+	"strings"
+	"sync"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+)
+
+var gpuNode = discovery.Host{OSID: "ubuntu", OSVersion: "24.04", Arch: "amd64",
+	NVIDIADriver: "580.65.06", MinComputeCapability: 8.0}
+
+// fakePython is uv and the environments it builds.
+type fakePython struct {
+	fakeSystem
+	pmu      sync.Mutex
+	commands []string
+	failPip  bool
+}
+
+func (f *fakePython) run(ctx context.Context, env []string, name string, args ...string) ([]byte, error) {
+	line := name + " " + strings.Join(args, " ")
+	f.pmu.Lock()
+	f.commands = append(f.commands, line)
+	f.pmu.Unlock()
+	switch {
+	case strings.HasSuffix(name, "/uv") && len(args) == 1 && args[0] == "--version":
+		return []byte("uv 0.12.20\n"), nil
+	case strings.HasSuffix(name, "/uv") && args[0] == "venv":
+		return nil, os.MkdirAll(filepath.Join(args[len(args)-1], "bin"), 0o755)
+	case strings.HasSuffix(name, "/uv") && args[0] == "pip" && args[1] == "install":
+		if f.failPip {
+			return []byte("error: Hash mismatch for torch\n"), errExit
+		}
+		venv := filepath.Dir(filepath.Dir(args[slices.Index(args, "--python")+1]))
+		return nil, os.WriteFile(filepath.Join(venv, "bin", "vllm"), []byte("#!/bin/sh\n"), 0o755)
+	case strings.HasSuffix(name, "/bin/vllm") && args[0] == "--version":
+		return []byte("0.30.0\n"), nil
+	}
+	return f.fakeSystem.run(ctx, env, name, args...)
+}
+
+func (f *fakePython) runStdin(_ context.Context, _ []string, stdin, name string, args ...string) ([]byte, error) {
+	f.pmu.Lock()
+	f.commands = append(f.commands, name+" "+strings.Join(args, " ")+" <<< "+strings.TrimSpace(stdin))
+	f.pmu.Unlock()
+	out := args[slices.Index(args, "--output-file")+1]
+	return nil, os.WriteFile(out, []byte("vllm==0.30.0 --hash=sha256:ab\n"), 0o644)
+}
+
+func pythonInstaller(t *testing.T, host discovery.Host, sys *fakePython) (*Installer, *release) {
+	uv := newRelease(map[string][]byte{
+		"uv-x86_64-unknown-linux-gnu.tar.gz": tarGz(t,
+			map[string]string{"uv-x86_64-unknown-linux-gnu/uv": "#!/bin/sh\n"}, nil),
+	})
+	gh := uv.serve(t)
+	mux := http.NewServeMux()
+	mux.HandleFunc("/pypi/vllm/0.30.0/json", func(w http.ResponseWriter, _ *http.Request) {
+		_, _ = w.Write([]byte(`{"urls":[{"upload_time_iso_8601":"2026-09-22T05:22:21.647189Z"},
+			{"upload_time_iso_8601":"2026-09-22T05:23:10.039243Z"}]}`))
+	})
+	pypi := httptest.NewServer(mux)
+	t.Cleanup(pypi.Close)
+	return &Installer{
+		Root: t.TempDir(), Host: host, Run: sys.run, RunStdin: sys.runStdin,
+		APIBase: gh.URL, DownloadBase: gh.URL, PyPIBase: pypi.URL, HTTP: gh.Client(),
+		FreeSpace: func(string) (float64, error) { return 100, nil },
+	}, uv
+}
+
+func TestVLLMIsInstalledOnAGPUNode(t *testing.T) {
+	sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
+	in, _ := pythonInstaller(t, gpuNode, sys)
+
+	got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})
+	if got[0].State != Ready {
+		t.Fatalf("not ready: %+v", got[0])
+	}
+	dir := filepath.Join(in.Root, "vllm", "0.30.0-cu13")
+	if got[0].Binary != filepath.Join(dir, "bin", "vllm") {
+		t.Errorf("binary %s, want the environment's vllm", got[0].Binary)
+	}
+	if !sys.installed["gcc"] {
+		t.Error("gcc, which Triton compiles its launcher with, was not installed")
+	}
+	all := strings.Join(sys.commands, "\n")
+	for _, want := range []string{
+		// PyPI as it stood a day after vLLM's last file, so the dependencies
+		// are the ones it was released against.
+		"--exclude-newer 2026-09-23T05:23:10Z",
+		"--generate-hashes",
+		"--require-hashes",
+		"venv --relocatable --python 3.12",
+	} {
+		if !strings.Contains(all, want) {
+			t.Errorf("no %q in:\n%s", want, all)
+		}
+	}
+	if _, err := os.Stat(filepath.Join(dir, lockName)); err != nil {
+		t.Errorf("the resolved requirements were not kept: %v", err)
+	}
+	if _, err := os.Stat(filepath.Join(in.Root, ".uv-cache")); err == nil {
+		t.Error("the download cache was left behind after a successful install")
+	}
+
+	// Checked again on the next pass, not installed again.
+	before := len(sys.commands)
+	if again := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}}); again[0].State != Ready {
+		t.Fatalf("second pass: %+v", again[0])
+	}
+	for _, c := range sys.commands[before:] {
+		if strings.Contains(c, "pip install") {
+			t.Errorf("a ready environment was installed again: %s", c)
+		}
+	}
+}
+
+func TestVLLMIsNotForEveryNode(t *testing.T) {
+	cases := []struct {
+		name  string
+		host  discovery.Host
+		state string
+		says  string
+	}{
+		{"no GPU", ubuntu, Skipped, "has none"},
+		{"old driver", discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15"}, Unsupported, "driver 580"},
+		{"V100", discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 7.0}, Unsupported, "7.5"},
+	}
+	for _, c := range cases {
+		t.Run(c.name, func(t *testing.T) {
+			sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
+			in, uv := pythonInstaller(t, c.host, sys)
+			got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})
+			if got[0].State != c.state || !strings.Contains(got[0].Detail, c.says) {
+				t.Errorf("got %s %q, want %s saying %q", got[0].State, got[0].Detail, c.state, c.says)
+			}
+			if uv.gets != 0 || len(sys.commands) != 0 {
+				t.Errorf("a node that cannot run it still started installing: %v", sys.commands)
+			}
+		})
+	}
+}
+
+func TestAPythonEngineIsRefusedWithoutTheDiskForIt(t *testing.T) {
+	sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{"gcc": true}}}
+	in, uv := pythonInstaller(t, gpuNode, sys)
+	in.FreeSpace = func(string) (float64, error) { return 12.5, nil }
+	got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})
+	if got[0].State != Failed || !strings.Contains(got[0].Detail, "12.5 GB") {
+		t.Errorf("got %s %q", got[0].State, got[0].Detail)
+	}
+	if uv.gets != 0 {
+		t.Error("it began downloading anyway")
+	}
+}
+
+func TestAFailedPythonInstallLeavesNothingThatLooksInstalled(t *testing.T) {
+	sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{"gcc": true}}, failPip: true}
+	in, _ := pythonInstaller(t, gpuNode, sys)
+	got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})
+	if got[0].State != Failed || !strings.Contains(got[0].Detail, "Hash mismatch") {
+		t.Fatalf("got %s %q", got[0].State, got[0].Detail)
+	}
+	entries, _ := os.ReadDir(filepath.Join(in.Root, "vllm"))
+	for _, e := range entries {
+		t.Errorf("left behind: %s", e.Name())
+	}
+}
+
+func TestAVersionPyPIDoesNotHaveIsSaidSo(t *testing.T) {
+	sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{"gcc": true}}}
+	in, _ := pythonInstaller(t, gpuNode, sys)
+	got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "9.9.9"}})
+	if got[0].State != Failed || !strings.Contains(got[0].Detail, "PyPI has no vllm 9.9.9") {
+		t.Errorf("got %s %q", got[0].State, got[0].Detail)
+	}
+}

internal/agent/provision/recipe.go

@@ -11,6 +11,7 @@ package provision
 
 import (
 	"fmt"
+	"slices"
 	"strings"
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
@@ -33,23 +34,163 @@ type Build struct {
 	// CPUFlags are extensions the build is slow without. Missing ones are
 	// reported, not refused: slow is still running.
 	CPUFlags []string
+
+	// Python, when set, makes the build a Python environment rather than an
+	// archive: that interpreter version, with Requirements installed from
+	// PyPI by uv. Binary is then looked for in the environment's bin.
+	Python       string
+	Requirements []string
+	// VersionArgs is what Binary is run with to prove the build works, when
+	// --version is not it.
+	VersionArgs []string
+	// FreeGB is the disk space the installation needs before it starts: a
+	// Python engine is several gigabytes, and one that fills the disk halfway
+	// leaves the node worse off than one that never began.
+	FreeGB int
 }
 
 // Recipe knows how to install one engine.
 type Recipe struct {
-	// Repo is the GitHub repository whose releases carry the builds.
+	// Repo is the GitHub repository whose releases carry the builds; empty
+	// for an engine installed from PyPI.
 	Repo string
+	// PyPI is the project an engine installed from PyPI is published as.
+	PyPI string
 	// Choose picks the build for a machine, or says why none fits.
 	Choose func(version string, h discovery.Host) (Build, error)
 }
 
 // Recipes are the engines this agent can install.
 //
-// vLLM and SGLang are not here yet: they are Python environments of several
-// gigabytes rather than an archive, and are installed by hand and named in the
-// agent's configuration until they are.
+// vLLM and SGLang are Python environments of several gigabytes, installed
+// from PyPI by uv, which is itself installed like llama.cpp: a GitHub release
+// checked against its published checksum.
 var Recipes = map[string]Recipe{
 	"llamacpp": {Repo: "ggml-org/llama.cpp", Choose: chooseLlamaCpp},
+	"vllm":     {PyPI: "vllm", Choose: chooseVLLM},
+	"sglang":   {PyPI: "sglang", Choose: chooseSGLang},
+	"uv":       {Repo: "astral-sh/uv", Choose: chooseUV},
+}
+
+// Installable are the engines this agent can install, as it tells the
+// control plane: every recipe but uv, which is a tool for the others.
+func Installable() []string {
+	var out []string
+	for name := range Recipes {
+		if name != "uv" {
+			out = append(out, name)
+		}
+	}
+	slices.Sort(out)
+	return out
+}
+
+// UVVersion is the uv that installs the Python engines.
+const UVVersion = "0.12.20"
+
+// pythonForEngines is the interpreter the Python engines run on. uv fetches
+// it, so the node's own Python — or its absence — does not matter.
+const pythonForEngines = "3.12"
+
+// NotHere is the error a recipe gives for a node the engine is not meant
+// for, as opposed to one it should run on and cannot: vLLM on a machine
+// with no GPU is not a fault, and is not reported as one.
+type NotHere struct{ Reason string }
+
+func (e NotHere) Error() string { return e.Reason }
+
+func chooseUV(version string, h discovery.Host) (Build, error) {
+	var arch string
+	switch h.Arch {
+	case "amd64":
+		arch = "x86_64"
+	case "arm64":
+		arch = "aarch64"
+	default:
+		return Build{}, fmt.Errorf("uv publishes no Linux build for %s", orUnknown(h.Arch))
+	}
+	return Build{
+		Variant: arch,
+		Assets:  []string{"uv-" + arch + "-unknown-linux-gnu.tar.gz"},
+		Binary:  "uv",
+	}, nil
+}
+
+// minDriverForCUDA13 is the oldest NVIDIA driver CUDA 13 runs on. Every vLLM
+// since 0.26 and SGLang 0.5 is built on torch for CUDA 13.
+const minDriverForCUDA13 = 580
+
+// minComputeForCUDA13 is the oldest GPU CUDA 13 supports: Turing. It dropped
+// Volta, so a V100 cannot run these builds at all.
+const minComputeForCUDA13 = 7.5
+
+// cudaGate says whether a node can run an engine built on CUDA 13.
+func cudaGate(engine string, h discovery.Host) error {
+	switch h.Arch {
+	case "amd64", "arm64":
+	default:
+		return fmt.Errorf("%s publishes no Linux build for %s", engine, orUnknown(h.Arch))
+	}
+	if h.NVIDIADriver == "" {
+		return NotHere{engine + " runs on NVIDIA GPUs and this node has none; it serves llama.cpp models"}
+	}
+	if major := leadingInt(h.NVIDIADriver); major > 0 && major < minDriverForCUDA13 {
+		return fmt.Errorf("%s is built for CUDA 13, which needs NVIDIA driver %d or newer; this node has %s",
+			engine, minDriverForCUDA13, h.NVIDIADriver)
+	}
+	if cc := h.MinComputeCapability; cc > 0 && cc < minComputeForCUDA13 {
+		return fmt.Errorf("%s is built for CUDA 13, which supports GPUs of compute capability %.1f and newer; this node has %.1f",
+			engine, minComputeForCUDA13, cc)
+	}
+	return nil
+}
+
+// Python engines need a C compiler at run time: Triton builds a small
+// launcher for its kernels the first time a model loads, and without gcc
+// that load fails long after the installation looked finished.
+var compilerPackages = map[string][]string{"deb": {"gcc"}, "rpm": {"gcc"}}
+
+func chooseVLLM(version string, h discovery.Host) (Build, error) {
+	if err := cudaGate("vLLM", h); err != nil {
+		return Build{}, err
+	}
+	return Build{
+		Variant:      "cu13",
+		Python:       pythonForEngines,
+		Requirements: []string{"vllm==" + version},
+		Binary:       "vllm",
+		Packages:     compilerPackages,
+		FreeGB:       25,
+	}, nil
+}
+
+func chooseSGLang(version string, h discovery.Host) (Build, error) {
+	if err := cudaGate("SGLang", h); err != nil {
+		return Build{}, err
+	}
+	// SGLang is launched as a module, so what the agent is given is the
+	// environment's interpreter, and the proof is importing it.
+	return Build{
+		Variant:      "cu13",
+		Python:       pythonForEngines,
+		Requirements: []string{"sglang==" + version},
+		Binary:       "python",
+		VersionArgs:  []string{"-c", "import sglang; print('sglang', sglang.__version__)"},
+		Packages:     compilerPackages,
+		FreeGB:       25,
+	}, nil
+}
+
+// leadingInt is the number a version string starts with: 580 of "580.65.06".
+func leadingInt(s string) int {
+	n := 0
+	for _, c := range strings.TrimSpace(s) {
+		if c < '0' || c > '9' {
+			break
+		}
+		n = n*10 + int(c-'0')
+	}
+	return n
 }
 
 // cudaForLlamaCpp is the CUDA build taken on an NVIDIA node. The release also

internal/agent/supervisor/health.go

@@ -53,8 +53,8 @@ func NewHTTPHealth(timeout time.Duration, opts ...HTTPHealthOption) *HTTPHealth
 }
 
 // Ready reports whether the engine at baseURL is serving model.
-func (h *HTTPHealth) Ready(ctx context.Context, baseURL, model string) error {
-	if err := h.get(ctx, baseURL+h.healthPath, nil); err != nil {
+func (h *HTTPHealth) Ready(ctx context.Context, baseURL, model, apiKey string) error {
+	if err := h.get(ctx, baseURL+h.healthPath, apiKey, nil); err != nil {
 		return fmt.Errorf("health: %w", err)
 	}
 	if !h.requireName || model == "" {
@@ -66,7 +66,7 @@ func (h *HTTPHealth) Ready(ctx context.Context, baseURL, model string) error {
 			ID string `json:"id"`
 		} `json:"data"`
 	}
-	if err := h.get(ctx, baseURL+h.modelsPath, &list); err != nil {
+	if err := h.get(ctx, baseURL+h.modelsPath, apiKey, &list); err != nil {
 		return fmt.Errorf("models: %w", err)
 	}
 	for _, d := range list.Data {
@@ -77,11 +77,14 @@ func (h *HTTPHealth) Ready(ctx context.Context, baseURL, model string) error {
 	return fmt.Errorf("engine is up but does not serve %q yet", model)
 }
 
-func (h *HTTPHealth) get(ctx context.Context, url string, out any) error {
+func (h *HTTPHealth) get(ctx context.Context, url, apiKey string, out any) error {
 	req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
 	if err != nil {
 		return err
 	}
+	if apiKey != "" {
+		req.Header.Set("Authorization", "Bearer "+apiKey)
+	}
 	resp, err := h.client.Do(req)
 	if err != nil {
 		return err

internal/agent/supervisor/health_test.go

@@ -39,7 +39,7 @@ func engineStub(t *testing.T, healthy bool, models ...string) *httptest.Server {
 
 func TestReadyWhenServingModel(t *testing.T) {
 	srv := engineStub(t, true, "llama-3.3-70b")
-	if err := NewHTTPHealth(time.Second).Ready(t.Context(), srv.URL, "llama-3.3-70b"); err != nil {
+	if err := NewHTTPHealth(time.Second).Ready(t.Context(), srv.URL, "llama-3.3-70b", ""); err != nil {
 		t.Errorf("a serving engine was reported unready: %v", err)
 	}
 }
@@ -48,7 +48,7 @@ func TestNotReadyWhileLoading(t *testing.T) {
 	// The case that matters: the process is up but the weights are not loaded.
 	// Routing on /health alone would send traffic into a loading model.
 	srv := engineStub(t, false, "llama-3.3-70b")
-	err := NewHTTPHealth(time.Second).Ready(t.Context(), srv.URL, "llama-3.3-70b")
+	err := NewHTTPHealth(time.Second).Ready(t.Context(), srv.URL, "llama-3.3-70b", "")
 	if err == nil {
 		t.Fatal("an unhealthy engine was reported ready")
 	}
@@ -61,7 +61,7 @@ func TestNotReadyWhenModelMissing(t *testing.T) {
 	// /health passes but the model is not listed: the engine is up and serving
 	// something else, or still registering.
 	srv := engineStub(t, true, "some-other-model")
-	err := NewHTTPHealth(time.Second).Ready(t.Context(), srv.URL, "llama-3.3-70b")
+	err := NewHTTPHealth(time.Second).Ready(t.Context(), srv.URL, "llama-3.3-70b", "")
 	if err == nil {
 		t.Fatal("a model that is not served was reported ready")
 	}
@@ -73,14 +73,33 @@ func TestNotReadyWhenModelMissing(t *testing.T) {
 func TestModelCheckCanBeSkipped(t *testing.T) {
 	srv := engineStub(t, true)
 	h := NewHTTPHealth(time.Second, WithoutModelCheck())
-	if err := h.Ready(t.Context(), srv.URL, "anything"); err != nil {
+	if err := h.Ready(t.Context(), srv.URL, "anything", ""); err != nil {
 		t.Errorf("the model check was not skipped: %v", err)
 	}
 }
 
 func TestUnreachableEngine(t *testing.T) {
 	h := NewHTTPHealth(200 * time.Millisecond)
-	if err := h.Ready(t.Context(), "http://127.0.0.1:1", "m"); err == nil {
+	if err := h.Ready(t.Context(), "http://127.0.0.1:1", "m", ""); err == nil {
 		t.Error("an unreachable engine was reported ready")
 	}
 }
+
+func TestTheHealthCheckSendsTheEnginesKey(t *testing.T) {
+	// An engine started with a key refuses /v1/models without it, and a
+	// shard that is serving would read as never ready.
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if r.Header.Get("Authorization") != "Bearer k1" {
+			w.WriteHeader(http.StatusUnauthorized)
+			return
+		}
+		_, _ = w.Write([]byte(`{"data":[{"id":"m"}]}`))
+	}))
+	defer srv.Close()
+	if err := NewHTTPHealth(time.Second).Ready(t.Context(), srv.URL, "m", "k1"); err != nil {
+		t.Errorf("with the key: %v", err)
+	}
+	if err := NewHTTPHealth(time.Second).Ready(t.Context(), srv.URL, "m", ""); err == nil {
+		t.Error("without the key the engine refused, and that read as ready")
+	}
+}

internal/agent/supervisor/supervisor.go

@@ -79,7 +79,7 @@ func (c *Config) normalize() {
 // supervisor can be tested without an HTTP engine.
 type HealthChecker interface {
 	// Ready reports whether the shard at baseURL serves model.
-	Ready(ctx context.Context, baseURL, model string) error
+	Ready(ctx context.Context, baseURL, model, apiKey string) error
 }
 
 // Shard is one supervised engine process.
@@ -243,6 +243,16 @@ func (s *Supervisor) launch(ctx context.Context, sh *Shard, cmds []engines.Comma
 	}
 
 	cmd := cmds[len(cmds)-1]
+	// The main process can need the rendezvous point too: a Ray worker's
+	// join is its only process, and started before the head listens it
+	// exits, failing the shard and with it the replica.
+	if cmd.WaitForLeaderPort && !sh.Spec.Leader() {
+		addr := fmt.Sprintf("%s:%d", sh.Spec.LeaderAddr, sh.Spec.LeaderPort)
+		if err := waitForPort(runCtx, addr, s.cfg.RendezvousTimeout); err != nil {
+			cancel()
+			return fmt.Errorf("waiting for %s (%s): %w", addr, cmd.Name, err)
+		}
+	}
 
 	c := exec.CommandContext(runCtx, cmd.Path, cmd.Args...)
 	c.Env = cmd.Env
@@ -401,7 +411,7 @@ func (s *Supervisor) WaitReady(ctx context.Context, sh *Shard, baseURL, model st
 			last := strings.Join(sh.Logs(10), "\n")
 			return fmt.Errorf("shard %s not ready within %v; last output:\n%s", sh.ID, timeout, last)
 		case <-t.C:
-			if err := s.health.Ready(ctx, baseURL, model); err != nil {
+			if err := s.health.Ready(ctx, baseURL, model, sh.Spec.APIKey); err != nil {
 				continue
 			}
 			sh.setState(Ready, nil)

internal/agent/supervisor/supervisor_test.go

@@ -21,7 +21,7 @@ import (
 // scriptedHealth reports ready once the flag is set.
 type scriptedHealth struct{ ready atomic.Bool }
 
-func (h *scriptedHealth) Ready(context.Context, string, string) error {
+func (h *scriptedHealth) Ready(context.Context, string, string, string) error {
 	if h.ready.Load() {
 		return nil
 	}
@@ -794,3 +794,36 @@ func TestAnAdoptedShardCanBecomeReady(t *testing.T) {
 	}
 	_ = s2.Stop(t.Context(), "a")
 }
+
+func TestAWorkersOnlyProcessWaitsForTheLeader(t *testing.T) {
+	// A Ray worker's join is its only process. The wait was honoured only for
+	// sidecars, so the join started at once, found no head, exited, and took
+	// the replica down with it.
+	s, _ := newSupervisor(t)
+	s.cfg.RendezvousTimeout = 5 * time.Second
+
+	ln, err := net.Listen("tcp", "127.0.0.1:0")
+	if err != nil {
+		t.Fatal(err)
+	}
+	port := ln.Addr().(*net.TCPAddr).Port
+	_ = ln.Close()
+	go func() {
+		time.Sleep(400 * time.Millisecond)
+		l, err := net.Listen("tcp", fmt.Sprintf("127.0.0.1:%d", port))
+		if err == nil {
+			t.Cleanup(func() { _ = l.Close() })
+		}
+	}()
+
+	spec := engines.ShardSpec{Rank: 1, LeaderAddr: "127.0.0.1", LeaderPort: port}
+	cmds := []engines.Command{{Name: "join", Path: "/bin/sh", Args: []string{"-c", "sleep 30"},
+		Env: os.Environ(), WaitForLeaderPort: true}}
+	start := time.Now()
+	if _, err := s.Start(t.Context(), "w", spec, cmds); err != nil {
+		t.Fatal(err)
+	}
+	if d := time.Since(start); d < 300*time.Millisecond {
+		t.Errorf("the worker started after %v, before the leader was listening", d)
+	}
+}

internal/auth/auth.go

@@ -21,6 +21,9 @@ import (
 type Principal struct {
 	KeyID     string
 	KeyPrefix string
+	// UserID is the person the key belongs to, empty for a key an operator
+	// issued in a project. A personal key has a person and no project.
+	UserID    string
 	ProjectID string
 	TeamID    string
 	OrgID     string
@@ -186,7 +189,7 @@ func (a *Authenticator) resolve(ctx context.Context, hash []byte) (Principal, er
 		return Principal{}, err
 	}
 	return Principal{
-		KeyID: k.ID, KeyPrefix: k.Prefix, ProjectID: eff.ProjectID,
+		KeyID: k.ID, KeyPrefix: k.Prefix, UserID: eff.UserID, ProjectID: eff.ProjectID,
 		TeamID: eff.TeamID, OrgID: eff.OrgID,
 		Band: eff.Band, RoutingProfile: eff.RoutingProfile,
 		Limits: eff,

internal/bench/bench.go

@@ -225,6 +225,7 @@ func runStep(ctx context.Context, client *http.Client, cfg Config, concurrency i
 	)
 
 	sampler := newSampler(cfg.MetricsURL, cfg.Engine)
+	sampler.apiKey = cfg.APIKey
 	sampler.start(ctx)
 
 	start := time.Now()

internal/bench/sampler.go

@@ -19,6 +19,7 @@ import (
 // Optional, because a run pointed at the gateway has no single engine to read.
 type sampler struct {
 	url    string
+	apiKey string
 	engine string
 	client *http.Client
 	cancel context.CancelFunc
@@ -65,6 +66,9 @@ func (s *sampler) sample(ctx context.Context) {
 	if err != nil {
 		return
 	}
+	if s.apiKey != "" {
+		req.Header.Set("Authorization", "Bearer "+s.apiKey)
+	}
 	resp, err := s.client.Do(req)
 	if err != nil {
 		return

internal/cli/agent.go

@@ -163,10 +163,10 @@ func printInventory(inv discovery.Inventory) {
 				g.Index, g.Class(), g.Name, g.MemoryTotalMB, g.ComputeCapability, health)
 		}
 		fmt.Fprintf(w, "  classes: %v\n", inv.Classes())
-		if !discovery.NVMLSupported() {
-			fmt.Fprintln(w, "  NVLink topology unavailable: rebuild with -tags nvml.")
-			fmt.Fprintln(w, "  Without it, placement cannot keep a tensor-parallel group inside")
-			fmt.Fprintln(w, "  one NVLink domain (plan §6.2).")
+		for _, g := range inv.GPUs {
+			if len(g.NVLinkPeers) > 0 {
+				fmt.Fprintf(w, "  GPU %d NVLink peers: %v\n", g.Index, g.NVLinkPeers)
+			}
 		}
 	}
 

internal/cli/bench.go

@@ -41,6 +41,11 @@ func newBenchCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 			cfg.MetricsURL = metrics
 			cfg.Engine = engine
 			cfg.APIKey = key
+			// A replica's engine demands its own key. Asked for rather than
+			// typed: the admin API knows which replica answers at the target.
+			if cfg.APIKey == "" {
+				cfg.APIKey = replicaKey(cmd, client, target)
+			}
 			if len(steps) > 0 {
 				cfg.Concurrency = steps
 			}
@@ -227,3 +232,18 @@ func ms(d time.Duration) string {
 		return d.Round(10 * time.Millisecond).String()
 	}
 }
+
+// replicaKey is the engine key of the replica at target, or "" when target is
+// not a replica (the gateway, say) or the admin API cannot be asked.
+func replicaKey(cmd *cobra.Command, client func() *ctlClient, target string) string {
+	var reps []adminapi.ReplicaView
+	if err := client().get(cmd.Context(), "/api/v1/replicas", &reps); err != nil {
+		return ""
+	}
+	for _, rep := range reps {
+		if sameEndpoint(rep.Endpoint, target) {
+			return rep.EngineKey
+		}
+	}
+	return ""
+}

internal/cli/ctl.go

@@ -67,7 +67,9 @@ func (c *ctlClient) do(ctx context.Context, method, path string, body, out any)
 	if err != nil {
 		return err
 	}
-	if body != nil {
+	// On every change, body or not: the admin listener refuses anything else,
+	// because a browser can send a bodiless POST across origins unasked.
+	if method != http.MethodGet {
 		req.Header.Set("Content-Type", "application/json")
 	}
 	// Who made the change, kept with a stored model so a catalog that grew
@@ -81,7 +83,10 @@ func (c *ctlClient) do(ctx context.Context, method, path string, body, out any)
 	}
 	defer resp.Body.Close()
 
-	if resp.StatusCode != http.StatusOK {
+	// Any 2xx. The API answers a change it has queued rather than made — a
+	// restart — with 202, and treating that as a failure printed
+	// "Error: 202 Accepted" over a request that had worked.
+	if resp.StatusCode < 200 || resp.StatusCode > 299 {
 		var e struct {
 			Error string `json:"error"`
 		}
@@ -120,6 +125,7 @@ func newCtlCmd() *cobra.Command {
 		newModelsCmd(client, &asJSON),
 		newReplicasCmd(client, &asJSON),
 		newExplainCmd(client, &asJSON),
+		newFabricTestCmd(client, &asJSON),
 		newNodeStateCmd(client, "cordon", "cordoned",
 			"Stop placing new work on a node; what runs there keeps running"),
 		newNodeStateCmd(client, "uncordon", "ready",
@@ -132,6 +138,8 @@ func newCtlCmd() *cobra.Command {
 		newStatusCmd(client, &asJSON),
 		newModelCmd(client),
 		newScaleCmd(client),
+		newRestartCmd(client),
+		newRetryCmd(client),
 		newEngineCheckCmd(),
 		newToolCheckCmd(),
 		newWeightsCmd(client, &asJSON),
@@ -199,7 +207,8 @@ func newNodesCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 					said = true
 				}
 				for _, p := range n.Provision {
-					if p.State == "ready" {
+					// Skipped is vLLM on a node with no GPU: as it should be.
+					if p.State == "ready" || p.State == "skipped" {
 						continue
 					}
 					fmt.Printf("\n%s: %s %s %s: %s", n.ID, p.Engine, p.Version, p.State, p.Detail)
@@ -240,6 +249,10 @@ func newNodesCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 // actually served, never what is merely catalogued.
 func modelStatus(m adminapi.ModelView) string {
 	switch {
+	case m.Parked:
+		return "parked"
+	case m.Unplaceable:
+		return "cannot place"
 	case m.Min == 0 && m.Total == 0:
 		return "stopped"
 	case m.Ready >= m.Min && m.Min > 0:
@@ -366,6 +379,9 @@ func newReplicasCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 				if r.Stale {
 					state += " (stale profile)"
 				}
+				if r.Restart != "" {
+					state += " (restart " + r.Restart + ")"
+				}
 				fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n",
 					r.ID, r.Model, state, r.Pool, join(nodes),
 					(time.Duration(r.AgeSec) * time.Second).String(),

internal/cli/ctl_fabrictest.go

@@ -0,0 +1,75 @@
+package cli
+
+import (
+	"encoding/json"
+	"fmt"
+	"text/tabwriter"
+	"time"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+)
+
+func newFabricTestCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
+	var (
+		fabric  string
+		seconds int
+	)
+	c := &cobra.Command{
+		Use:   "fabric-test NODE-A NODE-B",
+		Short: "Measure RDMA bandwidth between two nodes, rail by rail",
+		Long: `Runs ib_write_bw from each of NODE-A's adapters on a fabric to the same
+adapter on NODE-B — mlx5_0 to mlx5_0, and so on, the way NCCL pairs them —
+one rail at a time, and grades each against its link's rate: ok at 80% or
+more, slow at 50%, fail below or when nothing got through.
+
+Every check on one node can pass while two nodes still cannot reach each
+other: a cable into the wrong switch, a partition key, a RoCE network with no
+route. A replica spanning them then hangs on its first all-reduce. Run this
+before placing one.
+
+It goes through the agents, so it needs no ssh between nodes. perftest is
+installed on a node the first time it is asked. The traffic is real and
+takes a few seconds a rail; run it when the fabric is quiet.`,
+		Example: `  llmgw ctl fabric-test gpu01 gpu02
+  llmgw ctl fabric-test gpu01 gpu02 --fabric ib-compute --seconds 10`,
+		Args: cobra.ExactArgs(2),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			body, _ := json.Marshal(map[string]any{"a": args[0], "b": args[1], "fabric": fabric, "seconds": seconds})
+			cl := client()
+			// Rails one after another, each for its seconds and a settle; the
+			// client's usual timeout is for questions, not measurements.
+			cl.http.Timeout = 30 * time.Minute
+			var v adminapi.FabricTestView
+			if err := cl.post(cmd.Context(), "/api/v1/fabric-test", body, &v); err != nil {
+				return err
+			}
+			if *asJSON {
+				return json.NewEncoder(cmd.OutOrStdout()).Encode(v)
+			}
+			out := cmd.OutOrStdout()
+			fmt.Fprintf(out, "%s → %s on %s, %s\n\n", v.A, v.B, v.Fabric, v.Took)
+			w := tabwriter.NewWriter(out, 0, 0, 2, ' ', 0)
+			fmt.Fprintln(w, "RAIL\tLINK\tMEASURED\tVERDICT\tDETAIL")
+			for _, r := range v.Rails {
+				link := "-"
+				if r.RateGbps > 0 {
+					link = fmt.Sprintf("%d Gb/s", r.RateGbps)
+				}
+				fmt.Fprintf(w, "%s → %s\t%s\t%.1f Gb/s\t%s\t%s\n", r.A, r.B, link, r.Gbps, r.Verdict, r.Detail)
+			}
+			_ = w.Flush()
+			for _, n := range v.Notes {
+				fmt.Fprintf(out, "\nnote: %s\n", n)
+			}
+			if !v.OK {
+				return fmt.Errorf("not every rail passed")
+			}
+			return nil
+		},
+	}
+	c.Flags().StringVar(&fabric, "fabric", "", "the fabric to test (default: the one the two nodes share)")
+	c.Flags().IntVar(&seconds, "seconds", 3, "how long each rail runs, up to 30")
+	return c
+}

internal/cli/ctl_restart.go

@@ -0,0 +1,75 @@
+package cli
+
+import (
+	"encoding/json"
+	"errors"
+	"fmt"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+)
+
+func newRestartCmd(client func() *ctlClient) *cobra.Command {
+	var (
+		models []string
+		all    bool
+	)
+	c := &cobra.Command{
+		Use:   "restart [REPLICA...]",
+		Short: "Replace replicas with new ones, starting each replacement before stopping the old",
+		Long: "How a running engine picks up what only applies at start: an engine key,\n" +
+			"an edited profile, a new engine version.\n\n" +
+			"Each replacement is placed beside the replica it replaces, and the old one\n" +
+			"keeps serving until the new one is ready. Replicas of one model are\n" +
+			"replaced one at a time. Where nothing else has room for a second copy —\n" +
+			"the usual case on a small CPU fleet — the old one is stopped after the\n" +
+			"drain grace period (5 minutes by default) and the new one starts in its\n" +
+			"place; a model with only that replica is unserved while it loads.\n\n" +
+			"  llmgw ctl restart qwen2.5-7b-3f2a91c0\n" +
+			"  llmgw ctl restart --model qwen2.5-7b\n" +
+			"  llmgw ctl restart --all",
+		RunE: func(cmd *cobra.Command, args []string) error {
+			if len(args) == 0 && len(models) == 0 && !all {
+				return errors.New("name replicas, --model, or --all")
+			}
+			body, _ := json.Marshal(adminapi.RestartRequest{Replicas: args, Models: models, All: all})
+			var res adminapi.RestartResult
+			if err := client().post(cmd.Context(), "/api/v1/restart", body, &res); err != nil {
+				return err
+			}
+			fmt.Printf("%d replica(s) queued for restart; watch with: llmgw ctl replicas\n", len(res.Queued))
+			return nil
+		},
+	}
+	c.Flags().StringArrayVar(&models, "model", nil, "every replica of this model; repeatable")
+	c.Flags().BoolVar(&all, "all", false, "every ready replica in the cluster")
+	return c
+}
+
+func newRetryCmd(client func() *ctlClient) *cobra.Command {
+	return &cobra.Command{
+		Use:   "retry MODEL",
+		Short: "Try a parked model again, and clear its backoff",
+		Long: "A model whose replicas fail every time they start is parked after five\n" +
+			"failures in a row: it is not placed again until its definition changes\n" +
+			"or this is run. Run it once the cause — the weights, the engine, the\n" +
+			"profile — has been dealt with; `llmgw ctl status` lists parked models with\n" +
+			"their last error.",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			var res struct {
+				WasParked bool `json:"was_parked"`
+			}
+			if err := client().post(cmd.Context(), "/api/v1/models/"+args[0]+"/retry", []byte(`{}`), &res); err != nil {
+				return err
+			}
+			if res.WasParked {
+				fmt.Printf("%s unparked; it is placed on the next pass\n", args[0])
+			} else {
+				fmt.Printf("%s was not parked; its backoff is cleared\n", args[0])
+			}
+			return nil
+		},
+	}
+}

internal/cli/ctl_status_test.go

@@ -0,0 +1,24 @@
+package cli
+
+import (
+	"net/http"
+	"net/http/httptest"
+	"testing"
+)
+
+func TestAQueuedChangeIsNotReportedAsAnError(t *testing.T) {
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		w.WriteHeader(http.StatusAccepted)
+		_, _ = w.Write([]byte(`{"queued":["r1"]}`))
+	}))
+	defer srv.Close()
+	var out struct {
+		Queued []string `json:"queued"`
+	}
+	if err := newCtlClient(srv.URL).post(t.Context(), "/api/v1/restart", []byte(`{}`), &out); err != nil {
+		t.Fatalf("202 read as a failure: %v", err)
+	}
+	if len(out.Queued) != 1 {
+		t.Errorf("the body of a 202 was not read: %+v", out)
+	}
+}

internal/cli/dispatcherref_test.go

@@ -5,6 +5,7 @@ import (
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/agentsvc"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/fabrictest"
 )
 
 func TestTheForwardingDispatcherCarriesEveryCapability(t *testing.T) {
@@ -23,10 +24,17 @@ func TestTheForwardingDispatcherCarriesEveryCapability(t *testing.T) {
 		t.Error("dispatcherRef drops WeightRemover, so the control plane cannot reclaim disk")
 	}
 
+	if _, ok := ref.(fabrictest.Asker); !ok {
+		t.Error("dispatcherRef drops the bandwidth test, so ctl fabric-test cannot reach the agents")
+	}
+
 	// And the thing it forwards to must have it, or the forwarding has nothing
 	// to reach.
 	var svc any = &agentsvc.Service{}
 	if _, ok := svc.(control.WeightRemover); !ok {
 		t.Error("the agent service does not implement WeightRemover")
 	}
+	if _, ok := svc.(fabrictest.Asker); !ok {
+		t.Error("the agent service does not implement the bandwidth test")
+	}
 }

internal/cli/doctor.go

@@ -0,0 +1,185 @@
+package cli
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"io"
+	"os"
+	"sort"
+	"strings"
+	"text/tabwriter"
+	"time"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/doctor"
+)
+
+func newDoctorCmd() *cobra.Command {
+	var (
+		opts    doctor.Options
+		asJSON  bool
+		verbose bool
+	)
+	cmd := &cobra.Command{
+		Use:   "doctor",
+		Short: "Check this node's GPUs, NVLink, InfiniBand and engines for what llmgw needs",
+		Long: `Reads this node's GPUs, their NVLink and PCIe topology, its InfiniBand and
+RoCE ports, the kernel modules RDMA needs, the agent's configuration and
+whether vLLM can be installed here, and says what is wrong and how to fix
+it — most serious first.
+
+Run it on a node before it joins, and whenever a multi-node replica will not
+come up. --json writes everything it read as well as what it concluded; send
+that file when the report is not enough to go on.`,
+		Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			doctor.SetConfigLoader(config.LoadAgent)
+			if opts.VLLMVersion == "" {
+				opts.VLLMVersion = config.DefaultEngines["vllm"]
+			}
+			ctx, cancel := context.WithTimeout(cmd.Context(), 60*time.Second)
+			defer cancel()
+			facts := doctor.Collect(ctx, opts)
+			findings := doctor.Evaluate(facts)
+			if asJSON {
+				enc := json.NewEncoder(os.Stdout)
+				enc.SetIndent("", "  ")
+				return enc.Encode(struct {
+					Findings []doctor.Finding `json:"findings"`
+					Facts    doctor.Facts     `json:"facts"`
+				}{findings, facts})
+			}
+			printDoctor(os.Stdout, facts, findings, verbose)
+			for _, f := range findings {
+				if f.Level == doctor.Fail {
+					return fmt.Errorf("%d problem(s) need fixing", countLevel(findings, doctor.Fail))
+				}
+			}
+			return nil
+		},
+	}
+	f := cmd.Flags()
+	f.StringVar(&opts.ConfigPath, "config", "", "agent configuration (default /etc/llmgw/agent.yaml when present)")
+	f.StringVar(&opts.IBRoot, "ib-root", discovery.SysfsRoot, "sysfs InfiniBand directory")
+	f.StringVar(&opts.NvidiaSMI, "nvidia-smi", "", "path to nvidia-smi")
+	f.StringVar(&opts.VLLMVersion, "vllm", "", "the vLLM version to check for (default: the one this llmgw installs)")
+	f.BoolVar(&asJSON, "json", false, "write every fact and finding as JSON, to send back")
+	f.BoolVar(&verbose, "verbose", false, "also print the raw nvidia-smi topology")
+	return cmd
+}
+
+func countLevel(fs []doctor.Finding, l doctor.Level) int {
+	n := 0
+	for _, f := range fs {
+		if f.Level == l {
+			n++
+		}
+	}
+	return n
+}
+
+func printDoctor(out io.Writer, f doctor.Facts, findings []doctor.Finding, verbose bool) {
+	fmt.Fprintf(out, "%s · %s %s %s · driver %s\n\n", f.Hostname, f.Host.OSID, f.Host.OSVersion,
+		f.Host.Arch, orDash(f.Host.NVIDIADriver))
+
+	for _, x := range findings {
+		fmt.Fprintf(out, "%-4s  %-6s  %s\n", x.Level, x.Area, x.Message)
+		if x.Fix != "" && (x.Level == doctor.Fail || x.Level == doctor.Warn) {
+			fmt.Fprintf(out, "              → %s\n", x.Fix)
+		}
+	}
+
+	w := tabwriter.NewWriter(out, 0, 0, 2, ' ', 0)
+	if len(f.GPUs) > 0 {
+		fmt.Fprintln(w, "\nGPU\tNAME\tNUMA\tNVLINK PEERS\tNEAREST ADAPTER")
+		for _, g := range f.GPUs {
+			fmt.Fprintf(w, "%d\t%s\t%d\t%s\t%s\n", g.Index, g.Name, g.NUMANode,
+				orDash(ints(g.NVLinkPeers)), orDash(nearest(g.NICPaths)))
+		}
+	}
+	if len(f.HCAs) > 0 {
+		fmt.Fprintln(w, "\nADAPTER\tSTATE\tPHYS\tRATE\tLINK\tNUMA\tROCE GID\tERRORS")
+		for _, h := range f.HCAs {
+			for _, p := range h.Ports {
+				gid := "-"
+				if p.RoCEv2GID >= 0 {
+					gid = fmt.Sprint(p.RoCEv2GID)
+				}
+				fmt.Fprintf(w, "%s/%d\t%s\t%s\t%d Gb/s\t%s\t%d\t%s\t%s\n", h.Name, p.Num, p.State,
+					orDash(p.PhysState), p.RateGbps, p.LinkLayer, h.NUMANode, gid, orDash(errorsOf(p.Counters)))
+			}
+		}
+	}
+	_ = w.Flush()
+
+	for _, fabric := range sortedKeys(f.NCCL) {
+		fmt.Fprintf(out, "\nA shard on %s is given:\n", fabric)
+		env := f.NCCL[fabric]
+		for _, k := range sortedKeys(env) {
+			fmt.Fprintf(out, "  %s=%s\n", k, env[k])
+		}
+	}
+	if len(f.Engines) > 0 {
+		fmt.Fprintln(out, "\nEngines:")
+		for _, k := range sortedKeys(f.Engines) {
+			fmt.Fprintf(out, "  %s: %s\n", k, f.Engines[k])
+		}
+	}
+	if verbose && f.TopoRaw != "" {
+		fmt.Fprintf(out, "\nnvidia-smi topo -m:\n%s\n", f.TopoRaw)
+	}
+	fmt.Fprintf(out, "\n%d failed, %d warnings. `llmgw doctor --json > report.json` keeps everything read, to send back.\n",
+		countLevel(findings, doctor.Fail), countLevel(findings, doctor.Warn))
+}
+
+func nearest(paths map[string]string) string {
+	best, rel := "", ""
+	for h, r := range paths {
+		if best == "" || discovery.PathRank(r) < discovery.PathRank(rel) ||
+			(discovery.PathRank(r) == discovery.PathRank(rel) && h < best) {
+			best, rel = h, r
+		}
+	}
+	if best == "" {
+		return ""
+	}
+	return best + " (" + rel + ")"
+}
+
+func errorsOf(c map[string]uint64) string {
+	var out []string
+	for _, k := range sortedKeys(c) {
+		if c[k] > 0 && k != "port_xmit_data" && k != "port_rcv_data" {
+			out = append(out, fmt.Sprintf("%s=%d", k, c[k]))
+		}
+	}
+	return strings.Join(out, " ")
+}
+
+func ints(v []int) string {
+	s := make([]string, len(v))
+	for i, x := range v {
+		s[i] = fmt.Sprint(x)
+	}
+	return strings.Join(s, ",")
+}
+
+func orDash(s string) string {
+	if s == "" {
+		return "-"
+	}
+	return s
+}
+
+func sortedKeys[V any](m map[string]V) []string {
+	out := make([]string, 0, len(m))
+	for k := range m {
+		out = append(out, k)
+	}
+	sort.Strings(out)
+	return out
+}

internal/cli/keys.go

@@ -85,7 +85,7 @@ func newKeyInitCmd(open opener) *cobra.Command {
 }
 
 func newKeyCreateCmd(open opener) *cobra.Command {
-	var project, band, profile, baseURL string
+	var project, band, profile, baseURL, forUser string
 	var expires, claimTTL time.Duration
 	var claim bool
 
@@ -108,9 +108,22 @@ func newKeyCreateCmd(open opener) *cobra.Command {
 			}
 			defer s.Close()
 
-			p, err := resolveProject(cmd.Context(), s, project)
-			if err != nil {
-				return err
+			// Three kinds: an operator's key in a project, a person's key in a
+			// project they are a member of, and a person's personal key.
+			var owner store.User
+			if forUser != "" {
+				if owner, err = resolveUser(cmd.Context(), s, forUser); err != nil {
+					return err
+				}
+			}
+			var p store.ProjectPath
+			if project != "" || forUser == "" {
+				if p, err = resolveProject(cmd.Context(), s, project); err != nil {
+					return err
+				}
+			}
+			if p.ID == "" {
+				p.Path = "personal, " + owner.Email
 			}
 			secret, k, err := store.NewKey(randomID("key"), args[0], p.ID)
 			if err != nil {
@@ -121,7 +134,15 @@ func newKeyCreateCmd(open opener) *cobra.Command {
 			if expires > 0 {
 				k.ExpiresAt = k.CreatedAt.Add(expires)
 			}
-			if err := s.CreateKey(cmd.Context(), k); err != nil {
+			if owner.ID != "" {
+				err = s.CreateUserKey(cmd.Context(), owner.ID, k)
+			} else {
+				err = s.CreateKey(cmd.Context(), k)
+			}
+			if errors.Is(err, store.ErrNotMember) {
+				return fmt.Errorf("%s is not a member of %s; `llmgw users join` first", owner.Email, p.Path)
+			}
+			if err != nil {
 				return fmt.Errorf("creating the key in %s: %w", p.Path, err)
 			}
 
@@ -161,6 +182,8 @@ func newKeyCreateCmd(open opener) *cobra.Command {
 	}
 	cmd.Flags().StringVar(&project, "project", "",
 		"project the key belongs to: id, path (Acme/platform/gateway) or unique name")
+	cmd.Flags().StringVar(&forUser, "for", "",
+		"a person the key belongs to (id or email); without --project it is their personal key")
 	cmd.Flags().StringVar(&band, "band", "", "admission band; empty inherits")
 	cmd.Flags().StringVar(&profile, "profile", "", "routing profile; empty inherits")
 	cmd.Flags().DurationVar(&expires, "expires-in", 0, "lifetime; 0 never expires")
@@ -200,8 +223,12 @@ func newKeyListCmd(open opener) *cobra.Command {
 				fmt.Fprintln(w, "ID\tNAME\tKEY\tPROJECT\tOWNER\tSTATE\tLAST USED")
 				now := time.Now()
 				for _, r := range rows {
+					where := r.ProjectPath
+					if where == "" {
+						where = "personal"
+					}
 					fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\t%s\n", r.ID, r.Name, r.Prefix,
-						r.ProjectPath, dash(r.OwnerEmail), keyState(r.Key, now), ago(r.LastUsedAt, now))
+						where, dash(r.OwnerEmail), keyState(r.Key, now), ago(r.LastUsedAt, now))
 				}
 				return w.Flush()
 			}
@@ -260,6 +287,7 @@ func newKeyRevokeCmd(open opener) *cobra.Command {
 
 func newLimitsCmd(open opener) *cobra.Command {
 	var scope, id, period string
+	var perMember bool
 	var rpm, tpm int
 	var budget float64
 	var models []string
@@ -298,14 +326,27 @@ func newLimitsCmd(open opener) *cobra.Command {
 					return err
 				}
 				scopeID = p.ID
+			case store.ScopeUser:
+				u, err := resolveUser(cmd.Context(), s, id)
+				if err != nil {
+					return err
+				}
+				scopeID = u.ID
 			case store.ScopeKey:
 				if id == "" {
 					return errors.New("--scope key needs --id, the key's id")
 				}
 			default:
-				return fmt.Errorf("--scope is org, team, project or key, not %q", scope)
+				return fmt.Errorf("--scope is org, team, project, user or key, not %q", scope)
 			}
-			l := store.Limits{Scope: store.Scope(scope), ScopeID: scopeID}
+			level := store.Scope(scope)
+			if perMember {
+				// Written on the scope, applied to each member separately.
+				if level = store.MemberScope(level); level == "" {
+					return errors.New("--per-member is for --scope org, team or project")
+				}
+			}
+			l := store.Limits{Scope: level, ScopeID: scopeID}
 			if cmd.Flags().Changed("rpm") {
 				l.RPM = &rpm
 			}
@@ -330,11 +371,13 @@ func newLimitsCmd(open opener) *cobra.Command {
 			if err := s.PutLimits(cmd.Context(), l); err != nil {
 				return err
 			}
-			fmt.Fprintf(cmd.OutOrStdout(), "limits set on %s %s\n", scope, scopeID)
+			fmt.Fprintf(cmd.OutOrStdout(), "limits set on %s %s\n", level, scopeID)
 			return nil
 		},
 	}
-	cmd.Flags().StringVar(&scope, "scope", "project", "org | team | project | key")
+	cmd.Flags().StringVar(&scope, "scope", "project", "org | team | project | user | key")
+	cmd.Flags().BoolVar(&perMember, "per-member", false,
+		"with org, team or project: a limit for each member there, not one shared by all")
 	cmd.Flags().StringVar(&id, "id", "", "which one: id, path or unique name (a key's id for --scope key)")
 	cmd.Flags().IntVar(&rpm, "rpm", 0, "requests per minute")
 	cmd.Flags().IntVar(&tpm, "tpm", 0, "tokens per minute")

internal/cli/mirror.go

@@ -4,7 +4,9 @@ import (
 	"encoding/json"
 	"fmt"
 	"log/slog"
+	"net"
 	"net/http"
+	"net/netip"
 	"os"
 	"strings"
 	"time"
@@ -44,6 +46,7 @@ func newMirrorCmd() *cobra.Command {
 		newMirrorUnusedCmd(&dir),
 		newMirrorRemoveCmd(&dir),
 		newMirrorServeCmd(&dir),
+		newMirrorInitCmd(&dir),
 	)
 	return c
 }
@@ -229,6 +232,8 @@ func newMirrorRemoveCmd(dir *string) *cobra.Command {
 func newMirrorServeCmd(dir *string) *cobra.Command {
 	var listen, apiListen, base string
 	var tlsc tlsconf.Config
+	var allow []string
+	var allowPrivate bool
 
 	c := &cobra.Command{
 		Use:   "serve",
@@ -258,6 +263,9 @@ func newMirrorServeCmd(dir *string) *cobra.Command {
 			out := cmd.OutOrStdout()
 			log := slog.Default()
 			mgr := mirror.NewManager(*dir, log)
+			if allowPrivate {
+				mgr.AllowPrivateSources()
+			}
 			api := mirror.NewAPI(mgr, strings.TrimSuffix(base, "/"))
 
 			// http.Dir refuses paths that escape the root, which is the whole
@@ -268,9 +276,13 @@ func newMirrorServeCmd(dir *string) *cobra.Command {
 				Handler:           readOnly(withManifest(*dir, http.FileServer(http.Dir(*dir)))),
 				ReadHeaderTimeout: 10 * time.Second,
 			}
+			allowed, err := parseAllow(allow)
+			if err != nil {
+				return err
+			}
 			admin := &http.Server{
 				Addr:              apiListen,
-				Handler:           api.Handler(),
+				Handler:           apiGate(tlsc.Enabled(), allowed, api.Handler()),
 				ReadHeaderTimeout: 10 * time.Second,
 			}
 
@@ -279,14 +291,17 @@ func newMirrorServeCmd(dir *string) *cobra.Command {
 			fmt.Fprintf(out, "serving %s on %s\n", *dir, listen)
 
 			go func() { errs <- fmt.Errorf("serving the API: %w", serveAPI(admin, tlsc)) }()
-			if tlsc.Enabled() {
+			switch {
+			case tlsc.Enabled():
 				fmt.Fprintf(out, "management API on %s, mutual TLS against %s\n", apiListen, tlsc.CA)
-			} else {
-				// Said plainly rather than left to be discovered. An API that
-				// can delete models and start downloads, open to anyone who can
-				// reach the port, is a choice the operator should have made
-				// rather than inherited from a default.
-				fmt.Fprintf(out, "management API on %s, UNAUTHENTICATED — keep it on a trusted network\n", apiListen)
+			case len(allowed) > 0:
+				fmt.Fprintf(out, "management API on %s, for %s and this machine only\n", apiListen, strings.Join(allow, ", "))
+			default:
+				// This machine only, which was once "anyone": an API that can
+				// delete models and start downloads, open to every node that
+				// could reach the port, is a choice the operator should make
+				// with --api-allow rather than inherit.
+				fmt.Fprintf(out, "management API on %s, for this machine only (--api-allow to add the control plane)\n", apiListen)
 			}
 			fmt.Fprintf(out, "catalog sources should point at %s\n", base)
 
@@ -300,6 +315,10 @@ func newMirrorServeCmd(dir *string) *cobra.Command {
 	c.Flags().StringVar(&apiListen, "api-listen", ":8082", "address to serve the management API on")
 	c.Flags().StringVar(&base, "base", "http://localhost:8081",
 		"the address nodes use to reach this mirror, as it should appear in catalog sources")
+	c.Flags().StringArrayVar(&allow, "api-allow", nil,
+		"an address or network the management API answers, besides this machine; repeatable (default: this machine only)")
+	c.Flags().BoolVar(&allowPrivate, "allow-private-sources", false,
+		"let downloads reach loopback, private and link-local addresses")
 	c.Flags().StringVar(&tlsc.CA, "tls-ca", "", "CA bundle that management clients must present a certificate from")
 	c.Flags().StringVar(&tlsc.Cert, "tls-cert", "", "this mirror's certificate")
 	c.Flags().StringVar(&tlsc.Key, "tls-key", "", "this mirror's private key")
@@ -398,3 +417,61 @@ func versionPath(p string) (model, version string, ok bool) {
 	}
 	return parts[0], parts[1], true
 }
+
+// parseAllow reads addresses and networks: 10.0.0.140, 10.0.0.0/24.
+func parseAllow(list []string) ([]netip.Prefix, error) {
+	var out []netip.Prefix
+	for _, s := range list {
+		for _, one := range strings.Split(s, ",") {
+			one = strings.TrimSpace(one)
+			if one == "" {
+				continue
+			}
+			if p, err := netip.ParsePrefix(one); err == nil {
+				out = append(out, p.Masked())
+				continue
+			}
+			a, err := netip.ParseAddr(one)
+			if err != nil {
+				return nil, fmt.Errorf("--api-allow %q is not an address or a network", one)
+			}
+			out = append(out, netip.PrefixFrom(a.Unmap(), a.Unmap().BitLen()))
+		}
+	}
+	return out, nil
+}
+
+// apiGate decides who reaches the management API. With mutual TLS the
+// certificate decides, narrowed by --api-allow when given. Without it, only
+// this machine and the addresses allowed: nothing allowed used to mean anyone,
+// and every node that could reach the port could delete models.
+func apiGate(mtls bool, allowed []netip.Prefix, next http.Handler) http.Handler {
+	if mtls && len(allowed) == 0 {
+		return next
+	}
+	return onlyFrom(allowed, next)
+}
+
+// onlyFrom answers only this machine and the addresses allowed: the management
+// API can delete models and start downloads, and a mirror on its own machine
+// is reached over the network by one control plane and no one else.
+func onlyFrom(allowed []netip.Prefix, next http.Handler) http.Handler {
+	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		host, _, err := net.SplitHostPort(r.RemoteAddr)
+		a, perr := netip.ParseAddr(host)
+		if err != nil || perr != nil {
+			http.Error(w, "forbidden", http.StatusForbidden)
+			return
+		}
+		a = a.Unmap()
+		ok := a.IsLoopback()
+		for _, p := range allowed {
+			ok = ok || p.Contains(a)
+		}
+		if !ok {
+			http.Error(w, "this mirror is managed by its control plane only", http.StatusForbidden)
+			return
+		}
+		next.ServeHTTP(w, r)
+	})
+}

internal/cli/mirror_gate_test.go

@@ -0,0 +1,39 @@
+package cli
+
+import (
+	"net/http"
+	"net/http/httptest"
+	"net/netip"
+	"testing"
+)
+
+func TestTheMirrorAPIAnswersOnlyThisMachineUnlessTold(t *testing.T) {
+	ok := http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {})
+	call := func(h http.Handler, from string) int {
+		r := httptest.NewRequest("DELETE", "/v1/models/m/v", nil)
+		r.RemoteAddr = from + ":5000"
+		w := httptest.NewRecorder()
+		h.ServeHTTP(w, r)
+		return w.Code
+	}
+	cp := []netip.Prefix{netip.MustParsePrefix("10.0.0.140/32")}
+
+	// Nothing allowed and no TLS used to mean anyone: every node could delete
+	// models.
+	if got := call(apiGate(false, nil, ok), "10.0.0.141"); got != http.StatusForbidden {
+		t.Errorf("a node reached an unconfigured mirror API: %d", got)
+	}
+	if got := call(apiGate(false, nil, ok), "127.0.0.1"); got != http.StatusOK {
+		t.Errorf("the mirror's own machine was refused: %d", got)
+	}
+	if got := call(apiGate(false, cp, ok), "10.0.0.140"); got != http.StatusOK {
+		t.Errorf("the allowed control plane was refused: %d", got)
+	}
+	if got := call(apiGate(false, cp, ok), "10.0.0.141"); got != http.StatusForbidden {
+		t.Errorf("a node outside the allowlist got through: %d", got)
+	}
+	// With mutual TLS the certificate decides.
+	if got := call(apiGate(true, nil, ok), "10.0.0.141"); got != http.StatusOK {
+		t.Errorf("mTLS without an allowlist refused a client: %d", got)
+	}
+}

internal/cli/mirrorinit.go

@@ -0,0 +1,125 @@
+package cli
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"io"
+	"net"
+	"os"
+	"runtime"
+	"strings"
+	"time"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
+)
+
+// mirror init sets a machine up as the cluster's model mirror, on its own:
+// the bytes for every node on :8081, the management API on :8082 for the
+// control plane alone. server init does the same beside the control plane;
+// this is for when the mirror wants its own disk.
+
+func newMirrorInitCmd(dir *string) *cobra.Command {
+	var (
+		controlPlanes []string
+		base          string
+	)
+	cmd := &cobra.Command{
+		Use:   "init",
+		Short: "Set this machine up as the cluster's model mirror and start it",
+		Long: `Creates the llmgw user and the mirror's directory, and a systemd unit that
+serves the model files to every node on :8081 and the management API on
+:8082, and starts it.
+
+The management API can fetch and delete models, so it answers this machine
+and the control planes given with --control-plane, and nobody else. Then on
+the control plane:
+
+  sudo llmgw server init --mirror http://<this machine>:8082
+
+Safe to run again: it changes only what differs.`,
+		Example: `  sudo llmgw mirror init --control-plane 10.0.0.10`,
+		Args:    cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			if runtime.GOOS != "linux" {
+				return fmt.Errorf("mirror init sets up a Linux machine; this is %s", runtime.GOOS)
+			}
+			if os.Geteuid() != 0 {
+				return errors.New("mirror init creates a user, writes /etc and starts a service: run it as root")
+			}
+			if len(controlPlanes) == 0 {
+				return errors.New("--control-plane: which control plane manages this mirror (its address)")
+			}
+			if _, err := parseAllow(controlPlanes); err != nil {
+				return err
+			}
+			return mirrorInit(cmd.Context(), cmd.OutOrStdout(), *dir, base, controlPlanes)
+		},
+	}
+	cmd.Flags().StringArrayVar(&controlPlanes, "control-plane", nil,
+		"address of the control plane that manages this mirror; repeatable")
+	cmd.Flags().StringVar(&base, "base", "",
+		"the address nodes fetch from, http://host:8081 (default: this machine's first address)")
+	return cmd
+}
+
+func mirrorInit(ctx context.Context, out io.Writer, dir, base string, controlPlanes []string) error {
+	in := &serverInit{user: "llmgw", stateDir: "/var/lib/llmgw", out: out}
+	if err := in.ensureUser(ctx); err != nil {
+		return err
+	}
+	if err := os.MkdirAll(dir, 0o755); err != nil {
+		return err
+	}
+	if err := in.chown(dir, true); err != nil {
+		return err
+	}
+	addr := ""
+	if ips := localAddrs(); len(ips) > 0 {
+		addr = ips[0].String()
+	}
+	if base == "" {
+		if addr == "" {
+			return errors.New("this machine has no address to give nodes; give one with --base")
+		}
+		base = "http://" + net.JoinHostPort(addr, "8081")
+	}
+	unit := mirrorUnitFor(dir, ":8082", base, controlPlanes,
+		"'llmgw mirror init'. Bytes on :8081 for every node; the\n# management API on :8082, for the control plane only.")
+	changed, err := in.writeUnit(mirrorUnitPath, unit)
+	if err != nil {
+		return err
+	}
+	for _, args := range [][]string{{"daemon-reload"}, {"enable", "--now", "llmgw-mirror"}} {
+		if err := in.systemctl(ctx, args...); err != nil {
+			return err
+		}
+	}
+	if changed {
+		if err := in.systemctl(ctx, "restart", "llmgw-mirror"); err != nil {
+			return err
+		}
+		in.done("model mirror in %s: files on :8081 at %s, management on :8082 for %s", dir, base, strings.Join(controlPlanes, ", "))
+	} else {
+		in.kept("model mirror in %s", dir)
+	}
+	m, err := mirror.NewClient("http://127.0.0.1:8082", tlsconf.Config{}, 5*time.Second)
+	if err != nil {
+		return err
+	}
+	if err := waitFor(ctx, 30*time.Second, "the mirror to answer", func() (bool, error) {
+		return m.Health(ctx) == nil, nil
+	}); err != nil {
+		return err
+	}
+	in.kept("answering")
+	api := "http://<this machine>:8082"
+	if addr != "" {
+		api = "http://" + net.JoinHostPort(addr, "8082")
+	}
+	fmt.Fprintf(out, "\nThe mirror is up. On the control plane:\n\n  sudo llmgw server init --mirror %s\n\n", api)
+	return nil
+}

internal/cli/node.go

@@ -0,0 +1,440 @@
+package cli
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"io"
+	"os"
+	"slices"
+	"strings"
+	"time"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/version"
+)
+
+func newNodeCmd() *cobra.Command {
+	cmd := &cobra.Command{
+		Use:   "node",
+		Short: "Add machines to the cluster",
+	}
+	cmd.AddCommand(newNodeAddCmd(), newNodeRemoveCmd())
+	return cmd
+}
+
+func newNodeRemoveCmd() *cobra.Command {
+	var admin, serverCfg string
+	cmd := &cobra.Command{
+		Use:   "remove <node>",
+		Short: "Forget a node that is gone for good",
+		Long: `For a machine retired, or rebuilt under another name, which would otherwise
+stay listed as lost until the control plane restarts.
+
+Only a node that is not there is removed: stop its agent first, and wait for
+it to be lost, a minute later. Its certificate stays valid, since the
+authority keeps no revocation list: a machine that still has it can join
+again, which is how a node rebuilt under the same name comes back.`,
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			na := nodeAdd{admin: admin, serverCfg: serverCfg}
+			var out map[string]string
+			if err := na.adminAPI().delete(cmd.Context(), "/api/v1/nodes/"+args[0], &out); err != nil {
+				return err
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "%s removed\n", args[0])
+			return nil
+		},
+	}
+	cmd.Flags().StringVar(&admin, "admin", "", "the control plane's admin API (default: from server.yaml, else 127.0.0.1:9090)")
+	cmd.Flags().StringVar(&serverCfg, "config", "/etc/llmgw/server.yaml", "the control plane's configuration")
+	return cmd
+}
+
+// nodeAdd is what `llmgw node add` was asked to do.
+type nodeAdd struct {
+	opts      onboard.Options
+	fabrics   []string
+	servers   []string
+	admin     string
+	waitAll   bool
+	noWait    bool
+	plain     bool
+	serverCfg string
+	// named are the fabric names given in this run, by what the adapters
+	// report the fabric to be.
+	named map[string]string
+}
+
+func newNodeAddCmd() *cobra.Command {
+	var na nodeAdd
+	cmd := &cobra.Command{
+		Use:   "add [user@]address...",
+		Short: "Make a machine a node: check it, give it an identity, start its agent",
+		Long: `Run on the machine holding the certificate authority — the control plane,
+after 'llmgw server init' — as root. It logs in to the machine with ssh,
+as you would by hand, and:
+
+  1. installs llmgw there: the release for its architecture, checksum verified
+  2. runs 'llmgw doctor' there, and stops if it finds a problem
+  3. issues the node's certificate and installs it
+  4. writes agent.yaml and the systemd unit, and starts the agent
+  5. waits for the node to join and install llama.cpp
+
+The node's id is its hostname, and the address its agent dials is this
+machine's on the route to it, unless --node or --server say otherwise.
+Nothing else needs giving, except --fabric for InfiniBand or RoCE: which
+fabric an adapter is on has a name only you can choose, and it must be the
+same on every node attached to it. Doctor suggests one when it sees adapters.
+
+Several addresses add several nodes, one after another; a fabric named for
+one is recognised on the next and not asked again.
+
+The user needs sudo on the machine without a password. Run it again at any
+time: the node gets a fresh certificate and configuration, and its running
+engines are kept.`,
+		Example: `  sudo llmgw node add 10.0.0.21
+  sudo llmgw node add ubuntu@10.0.0.21 ubuntu@10.0.0.22 ubuntu@10.0.0.23
+  sudo llmgw node add ubuntu@gpu01 --fabric ib-a:mlx5_0,mlx5_1,mlx5_2,mlx5_3`,
+		Args: cobra.MinimumNArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			return na.run(cmd.Context(), args, cmd.OutOrStdout())
+		},
+	}
+	f := cmd.Flags()
+	f.StringVar(&na.opts.Node, "node", "", "node id (default: the machine's hostname)")
+	f.StringVar(&na.opts.User, "ssh-user", "", "user to log in as (default: the one before @ in the address, else whoever ran sudo, else you)")
+	f.StringArrayVar(&na.servers, "server", nil, "control plane address the agent dials, https://host:7443; repeatable for failover (default: this machine)")
+	f.StringArrayVar(&na.fabrics, "fabric", nil, "InfiniBand or RoCE fabric and its adapters, name:mlx5_0,mlx5_1; repeatable")
+	f.StringVar(&na.opts.MgmtAddr, "mgmt-addr", "", "address engines are reached at (default: found by the agent)")
+	f.StringVar(&na.opts.MgmtIface, "mgmt-if", "", "interface NCCL bootstraps over (default: the one holding that address)")
+	f.StringVar(&na.opts.LlamaCppBin, "llamacpp-bin", "", "a llama-server you installed, instead of the one the agent installs")
+	f.StringVar(&na.opts.VLLMBin, "vllm-bin", "", "a vllm you installed; also set engines.vllm: \"\" in server.yaml")
+	f.StringVar(&na.opts.SGLangPython, "sglang-python", "", "python of an SGLang virtualenv you installed")
+	f.StringVar(&na.opts.NvidiaSMI, "nvidia-smi", "", "where nvidia-smi is, when it is not on PATH")
+	f.StringVar(&na.opts.IBRoot, "ib-root", "", "the sysfs InfiniBand directory, when not /sys/class/infiniband")
+	f.StringVar(&na.opts.PKI, "pki", defaultPKIDir, "directory holding the certificate authority")
+	f.StringVar(&na.opts.Releases, "releases", "https://gitea.mynodes.xyz/ilker/llmgw/releases/download", "where release files are downloaded from")
+	f.BoolVar(&na.opts.IgnoreChecks, "ignore-checks", false, "add the node even when doctor finds a problem")
+	f.StringVar(&na.admin, "admin", "", "the control plane's admin API, to watch the node join (default: from server.yaml, else 127.0.0.1:9090)")
+	f.StringVar(&na.serverCfg, "config", "/etc/llmgw/server.yaml", "the control plane's configuration, for the admin API's address")
+	f.BoolVar(&na.waitAll, "wait", false, "wait for every engine, vLLM included, not only llama.cpp")
+	f.BoolVar(&na.noWait, "no-wait", false, "do not wait for the node to join")
+	f.BoolVar(&na.plain, "plain", false, "plain output, even in a terminal")
+	return cmd
+}
+
+// prepare works out what every node added in this run shares.
+func (na *nodeAdd) prepare() error {
+	o := &na.opts
+	for _, s := range na.fabrics {
+		fab, err := onboard.ParseFabric(s)
+		if err != nil {
+			return err
+		}
+		if len(fab.HCAs) == 0 {
+			return fmt.Errorf("--fabric %s names no adapters: give them as %s:mlx5_0,mlx5_1", s, fab.Name)
+		}
+		o.Fabrics = append(o.Fabrics, fab)
+	}
+	for _, s := range na.servers {
+		for _, one := range strings.Split(s, ",") {
+			if one = strings.TrimSpace(one); one != "" {
+				o.Servers = append(o.Servers, one)
+			}
+		}
+	}
+	o.Version = version.Version
+	if o.Binary == "" {
+		exe, err := os.Executable()
+		if err != nil {
+			return err
+		}
+		o.Binary = exe
+	}
+	onboard.AgentUnit = agentUnit
+	na.named = map[string]string{}
+	return nil
+}
+
+// target is the options for one address, [user@]host.
+func (na *nodeAdd) target(addr string) onboard.Options {
+	o := na.opts
+	o.Fabrics = slices.Clone(na.opts.Fabrics)
+	o.Addr = addr
+	if user, host, ok := strings.Cut(addr, "@"); ok {
+		o.User, o.Addr = user, host
+	}
+	if o.User == "" {
+		// Under sudo, the person who ran it — their account is usually the
+		// one that exists on the nodes too. root's own keys still do the
+		// logging in, since that is who is running ssh.
+		o.User = os.Getenv("SUDO_USER")
+	}
+	return o
+}
+
+// rememberFabrics wraps the question of which fabric a node's adapters are
+// on, so a name given for one node is used for the next whose adapters
+// report the same fabric, without asking again. ask nil is a script, which
+// is asked nothing and gets only what an earlier answer said.
+func (na *nodeAdd) rememberFabrics(ask func([]onboard.Fabric) []onboard.Fabric) func([]onboard.Fabric) []onboard.Fabric {
+	return func(s []onboard.Fabric) []onboard.Fabric {
+		var known []onboard.Fabric
+		all := true
+		for i, f := range s {
+			if name, ok := na.named[f.Key]; ok && f.Key != "" && f.Key != "unknown" {
+				f.Name = name
+				s[i].Name = name
+				known = append(known, f)
+			} else {
+				all = false
+			}
+		}
+		if all || ask == nil {
+			return known
+		}
+		got := ask(s)
+		for _, f := range got {
+			if f.Key != "" && f.Key != "unknown" {
+				na.named[f.Key] = f.Name
+			}
+		}
+		return got
+	}
+}
+
+func (na *nodeAdd) run(ctx context.Context, addrs []string, out io.Writer) error {
+	if len(addrs) > 1 && na.opts.Node != "" {
+		return errors.New("--node names one node; give one address with it")
+	}
+	if err := na.prepare(); err != nil {
+		return err
+	}
+	tui := !na.plain && isTerminal(out)
+	type outcome struct {
+		addr, node string
+		err        error
+	}
+	var results []outcome
+	for i, addr := range addrs {
+		o := na.target(addr)
+		title := "llmgw · adding " + o.Addr
+		if len(addrs) > 1 {
+			title += fmt.Sprintf(" (%d of %d)", i+1, len(addrs))
+		}
+		var node string
+		var err error
+		if tui {
+			node, err = na.runTUI(ctx, o, title)
+		} else {
+			node, err = na.runPlain(ctx, o, title, out)
+		}
+		results = append(results, outcome{addr, node, err})
+		if ctx.Err() != nil {
+			break
+		}
+	}
+	if len(addrs) == 1 {
+		return results[0].err
+	}
+	fmt.Fprintln(out)
+	failed := 0
+	for _, r := range results {
+		if r.err != nil {
+			failed++
+			fmt.Fprintf(out, "  FAIL  %-20s %s\n", r.addr, r.err)
+		} else {
+			fmt.Fprintf(out, "  ok    %-20s %s\n", r.addr, r.node)
+		}
+	}
+	if failed > 0 {
+		return fmt.Errorf("%d of %d not added; running the same command again finishes them and leaves the rest as they are", failed, len(addrs))
+	}
+	return nil
+}
+
+func (na *nodeAdd) runPlain(ctx context.Context, o onboard.Options, title string, out io.Writer) (string, error) {
+	p := &plainProgress{out: out}
+	fmt.Fprintf(out, "%s\n", title)
+	o.AskFabrics = na.rememberFabrics(nil)
+	res, err := onboard.New(o, p.event).Run(ctx)
+	if err != nil {
+		return "", err
+	}
+	if na.noWait {
+		fmt.Fprintf(out, "\n%s is added. 'llmgw ctl nodes' shows it once it joins.\n\n", res.Node)
+		return res.Node, nil
+	}
+	if err := na.watch(ctx, res.Node, p.event); err != nil {
+		return res.Node, err
+	}
+	fmt.Fprintf(out, "\n%s is in the cluster.\n\n", res.Node)
+	return res.Node, nil
+}
+
+// adminAPI is the control plane's admin API, from its configuration when
+// this machine has it.
+func (na *nodeAdd) adminAPI() *ctlClient {
+	addr := na.admin
+	if addr == "" {
+		addr = "127.0.0.1:9090"
+		if cfg, err := config.Load(na.serverCfg); err == nil && cfg.AdminListen != "" {
+			addr = cfg.AdminListen
+			if strings.HasPrefix(addr, ":") {
+				addr = "127.0.0.1" + addr
+			}
+		}
+	}
+	return newCtlClient(addr)
+}
+
+// errNoAdmin is the watch giving up because the admin API is not here, which
+// is not a failure of onboarding.
+var errNoAdmin = errors.New("no admin API")
+
+// watch follows the node from the control plane: joining, then installing
+// its engines. llama.cpp is waited for, since without it the node serves
+// nothing; vLLM takes minutes more and is waited for only with --wait.
+func (na *nodeAdd) watch(ctx context.Context, node string, emit func(onboard.Event)) error {
+	client := na.adminAPI()
+	var nodes []adminapi.NodeView
+	if err := client.get(ctx, "/api/v1/nodes", &nodes); err != nil {
+		emit(onboard.Event{Step: onboard.StepJoin, Level: onboard.Warn,
+			Detail: "cannot reach the control plane's admin API here, so the node's joining is not followed",
+			Fix:    "llmgw ctl nodes on the control plane shows it"})
+		return nil
+	}
+	emit(onboard.Event{Step: onboard.StepJoin, Level: onboard.Running, Detail: "waiting for " + node + " to join"})
+	var view adminapi.NodeView
+	started := time.Now()
+	err := waitFor(ctx, 2*time.Minute, node+" to join", func() (bool, error) {
+		if err := client.get(ctx, "/api/v1/nodes", &nodes); err != nil {
+			return false, nil
+		}
+		for _, n := range nodes {
+			if n.ID == node && heardSince(n, started) {
+				view = n
+				return true, nil
+			}
+		}
+		return false, nil
+	})
+	if err != nil {
+		return fmt.Errorf("%s did not join: its agent's log says why (journalctl -u llmgw-agent on the node); a certificate whose name differs from the node id is refused", node)
+	}
+	emit(onboard.Event{Step: onboard.StepJoin, Level: onboard.Done, Detail: node + " joined: " + describeNode(view)})
+
+	emit(onboard.Event{Step: onboard.StepEngines, Level: onboard.Running, Detail: "installing engines"})
+	seen := map[string]string{}
+	limit := 20 * time.Minute
+	if na.waitAll {
+		limit = time.Hour
+	}
+	err = waitFor(ctx, limit, "the engines on "+node, func() (bool, error) {
+		if err := client.get(ctx, "/api/v1/nodes", &nodes); err != nil {
+			return false, nil
+		}
+		for _, n := range nodes {
+			if n.ID != node {
+				continue
+			}
+			finished := true
+			for _, p := range n.Provision {
+				line := p.State + " " + p.Detail
+				if seen[p.Engine] != line {
+					seen[p.Engine] = line
+					emit(engineEvent(p.Engine, p.Version, p.State, p.Detail))
+				}
+				switch p.State {
+				case "installing":
+					if p.Engine == "llamacpp" || na.waitAll {
+						finished = false
+					}
+				case "failed":
+					if p.Engine == "llamacpp" {
+						return false, fmt.Errorf("llama.cpp did not install: %s", p.Detail)
+					}
+				}
+			}
+			return finished && len(n.Provision) > 0, nil
+		}
+		return false, nil
+	})
+	if err != nil {
+		return err
+	}
+	msg := "engines ready"
+	if v, ok := seen["vllm"]; ok && strings.HasPrefix(v, "installing") && !na.waitAll {
+		msg = "llama.cpp ready; vLLM is still installing, which takes a few minutes more (watch /admin, or add --wait)"
+	}
+	emit(onboard.Event{Step: onboard.StepEngines, Level: onboard.Done, Detail: msg})
+	return nil
+}
+
+// heardSince reports whether the node has been heard from since t, by the
+// agent just started. A node added again is already listed, from its agent
+// before the restart, and a cordoned one is not "ready": neither says whether
+// the new agent connected.
+func heardSince(n adminapi.NodeView, t time.Time) bool {
+	switch n.State {
+	case "unreachable", "lost", "unknown":
+		return false
+	}
+	ago, err := time.ParseDuration(n.LastSeenAgo)
+	if err != nil {
+		return false
+	}
+	// A second's margin: the age is rounded to one.
+	return ago+time.Second < time.Since(t)
+}
+
+func engineEvent(engine, version, state, detail string) onboard.Event {
+	level := onboard.Info
+	switch state {
+	case "failed", "unsupported":
+		level = onboard.Warn
+	}
+	return onboard.Event{Step: onboard.StepEngines, Level: level,
+		Detail: fmt.Sprintf("%s %s: %s%s", engine, version, state, prefixed(": ", detail))}
+}
+
+func prefixed(p, s string) string {
+	if s == "" {
+		return ""
+	}
+	return p + s
+}
+
+func describeNode(n adminapi.NodeView) string {
+	parts := []string{}
+	if n.GPUTotal > 0 {
+		parts = append(parts, fmt.Sprintf("%d GPU(s) %s", n.GPUTotal, strings.Join(n.Classes, ", ")))
+	} else {
+		parts = append(parts, "no GPU")
+	}
+	if len(n.Fabrics) > 0 {
+		parts = append(parts, "fabrics "+strings.Join(n.Fabrics, ", "))
+	}
+	return strings.Join(parts, ", ")
+}
+
+// plainProgress prints events as lines, for a log or a pipe.
+type plainProgress struct {
+	out io.Writer
+}
+
+func (p *plainProgress) event(e onboard.Event) {
+	mark := map[string]string{
+		onboard.Running: "  ..  ", onboard.Done: "  ok  ", onboard.Info: "      ",
+		onboard.Warn: "  !!  ", onboard.Fail: "  FAIL",
+	}[e.Level]
+	fmt.Fprintf(p.out, "%s %-13s %s\n", mark, e.Step, e.Detail)
+	if e.Fix != "" {
+		fmt.Fprintf(p.out, "       %-13s → %s\n", "", e.Fix)
+	}
+}

internal/cli/org.go

@@ -46,21 +46,40 @@ func newOrgTreeCmd(open dbOpener) *cobra.Command {
 			if err != nil {
 				return err
 			}
-			var byProject map[string][]store.User
+			var members map[string][]string
+			var alone []store.User
 			if people {
 				users, err := s.Users(cmd.Context(), "")
 				if err != nil {
 					return err
 				}
-				byProject = map[string][]store.User{}
+				all, err := s.AllMemberships(cmd.Context())
+				if err != nil {
+					return err
+				}
+				email := map[string]string{}
 				for _, u := range users {
-					byProject[u.ProjectID] = append(byProject[u.ProjectID], u)
+					email[u.ID] = u.Email + "  " + string(u.State)
+					if u.IsAdmin {
+						email[u.ID] += "  admin"
+					}
+				}
+				members = map[string][]string{}
+				in := map[string]bool{}
+				for _, m := range all {
+					members[string(m.Scope)+":"+m.ScopeID] = append(members[string(m.Scope)+":"+m.ScopeID], email[m.UserID])
+					in[m.UserID] = true
+				}
+				for _, u := range users {
+					if !in[u.ID] {
+						alone = append(alone, u)
+					}
 				}
 			}
-			printTree(cmd.OutOrStdout(), tree, byProject)
-			if people && len(byProject[""]) > 0 {
-				fmt.Fprintf(cmd.OutOrStdout(), "\nwaiting for approval\n")
-				for _, u := range byProject[""] {
+			printTree(cmd.OutOrStdout(), tree, members)
+			if people && len(alone) > 0 {
+				fmt.Fprintf(cmd.OutOrStdout(), "\non their own\n")
+				for _, u := range alone {
 					fmt.Fprintf(cmd.OutOrStdout(), "  %s  %s  %s\n", u.Email, u.State, u.ID)
 				}
 			}
@@ -71,36 +90,44 @@ func newOrgTreeCmd(open dbOpener) *cobra.Command {
 	return cmd
 }
 
-func printTree(w io.Writer, tree []store.TreeOrg, people map[string][]store.User) {
+func printTree(w io.Writer, tree []store.TreeOrg, members map[string][]string) {
 	if len(tree) == 0 {
 		fmt.Fprintln(w, "no organizations yet; `llmgw org add org <name>` starts one")
 		return
 	}
+	list := func(indent, key string) {
+		for _, m := range members[key] {
+			fmt.Fprintf(w, "%s%s\n", indent, m)
+		}
+	}
 	for _, o := range tree {
-		fmt.Fprintf(w, "%s  (%s)\n", o.Name, o.ID)
+		fmt.Fprintf(w, "%s  (%s)%s\n", o.Name, o.ID, membersNote(o.Members))
+		list("    · ", "org:"+o.ID)
 		if len(o.Teams) == 0 {
 			fmt.Fprintln(w, "    no teams")
 		}
 		for _, t := range o.Teams {
-			fmt.Fprintf(w, "  %s  (%s)\n", t.Name, t.ID)
+			fmt.Fprintf(w, "  %s  (%s)%s\n", t.Name, t.ID, membersNote(t.Members))
+			list("      · ", "team:"+t.ID)
 			if len(t.Projects) == 0 {
 				fmt.Fprintln(w, "      no projects")
 			}
 			for _, p := range t.Projects {
 				fmt.Fprintf(w, "    %s  (%s)  %s, %s\n", p.Name, p.ID,
-					countOf(p.Users, "account"), keysOf(p))
-				for _, u := range people[p.ID] {
-					admin := ""
-					if u.IsAdmin {
-						admin = "  admin"
-					}
-					fmt.Fprintf(w, "      %s  %s%s\n", u.Email, u.State, admin)
-				}
+					countOf(p.Users, "member"), keysOf(p))
+				list("        · ", "project:"+p.ID)
 			}
 		}
 	}
 }
 
+func membersNote(n int) string {
+	if n == 0 {
+		return ""
+	}
+	return "  " + countOf(n, "member")
+}
+
 func keysOf(p store.TreeProject) string {
 	s := countOf(p.ActiveKeys, "key")
 	if revoked := p.Keys - p.ActiveKeys; revoked > 0 {

internal/cli/org_test.go

@@ -51,15 +51,24 @@ func TestTheTreeIsManagedByNameNotId(t *testing.T) {
 	mustRun(t, db, "keys", "create", "laptop", "--project", "gateway")
 
 	out := mustRun(t, db, "org", "tree", "--people")
-	for _, want := range []string{"Acme", "gateway", "1 account, 1 key", "ayse@example.com"} {
+	for _, want := range []string{"Acme", "gateway", "1 member, 1 key", "ayse@example.com"} {
 		if !strings.Contains(out, want) {
 			t.Errorf("tree lacks %q:\n%s", want, out)
 		}
 	}
 
-	out = mustRun(t, db, "users", "move", "ayse@example.com", "--project", "research")
+	out = mustRun(t, db, "users", "join", "ayse@example.com", "--project", "research")
 	if !strings.Contains(out, "Scania / platform / research") {
-		t.Errorf("move said: %s", out)
+		t.Errorf("join said: %s", out)
+	}
+	mustRun(t, db, "users", "leave", "ayse@example.com", "--project", "Acme/platform/gateway")
+	// She now uses the gateway through research alone; a personal key is hers.
+	mustRun(t, db, "keys", "create", "mine", "--for", "ayse@example.com")
+	if out := mustRun(t, db, "keys", "list", "--all"); !strings.Contains(out, "personal") {
+		t.Errorf("a personal key is not listed as one:\n%s", out)
+	}
+	if _, err := run(t, db, "keys", "create", "x", "--for", "ayse@example.com", "--project", "gateway"); err == nil {
+		t.Error("a key was made for her in a project she left")
 	}
 
 	mustRun(t, db, "org", "rename", "project", "research", "lab")

internal/cli/root.go

@@ -45,6 +45,8 @@ func Root() *cobra.Command {
 		newOrgCmd(),
 		newMirrorCmd(),
 		newSimEngineCmd(),
+		newDoctorCmd(),
+		newNodeCmd(),
 	)
 	return root
 }

internal/cli/servable_test.go

@@ -0,0 +1,52 @@
+package cli
+
+import (
+	"bytes"
+	"context"
+	"encoding/binary"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+)
+
+func ggufAt(t *testing.T, arch string) string {
+	t.Helper()
+	var b bytes.Buffer
+	w := func(v any) { _ = binary.Write(&b, binary.LittleEndian, v) }
+	str := func(s string) { w(uint64(len(s))); b.WriteString(s) }
+	b.WriteString("GGUF")
+	w(uint32(3))
+	w(uint64(0))
+	w(uint64(1))
+	str("general.architecture")
+	w(uint32(8))
+	str(arch)
+	body := b.Bytes()
+	srv := httptest.NewServer(http.HandlerFunc(func(rw http.ResponseWriter, r *http.Request) {
+		http.ServeContent(rw, r, "m.gguf", time.Time{}, bytes.NewReader(body))
+	}))
+	t.Cleanup(srv.Close)
+	return srv.URL + "/m.gguf"
+}
+
+func TestAModelTheNodesCannotLoadIsNotCatalogued(t *testing.T) {
+	check := servableCheck("b10991")
+	llama := []catalog.Profile{{Name: "cpu", Engine: "llamacpp"}}
+
+	err := check(context.Background(), catalog.Model{Name: "kev", Source: ggufAt(t, "ggmlc"), Profiles: llama})
+	if err == nil || !strings.Contains(err.Error(), "ggmlc") {
+		t.Errorf("a ggmlc model was accepted: %v", err)
+	}
+	if err := check(context.Background(), catalog.Model{Name: "q", Source: ggufAt(t, "qwen2"), Profiles: llama}); err != nil {
+		t.Errorf("a qwen2 model was refused: %v", err)
+	}
+	// Not llama.cpp's to judge: vLLM reads a repository, not a GGUF header.
+	vllm := []catalog.Profile{{Name: "gpu", Engine: "vllm"}}
+	if err := check(context.Background(), catalog.Model{Name: "v", Source: ggufAt(t, "ggmlc"), Profiles: vllm}); err != nil {
+		t.Errorf("a vLLM profile was judged by llama.cpp's list: %v", err)
+	}
+}

internal/cli/server.go

@@ -5,6 +5,8 @@ import (
 	"crypto/tls"
 	"errors"
 	"fmt"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/sourcecheck"
 	"log/slog"
 	"net"
 	"net/http"
@@ -19,11 +21,13 @@ import (
 	"golang.org/x/net/http2/h2c"
 	"golang.org/x/sync/errgroup"
 
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/agentsvc"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/fabrictest"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/fleetmetrics"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelsvc"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
@@ -32,6 +36,7 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/promq"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/router"
 	storepkg "gitea.mynodes.xyz/ilker/llmgw/internal/store"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/version"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/web"
 )
@@ -127,6 +132,11 @@ func newServerCmd() *cobra.Command {
 				// On the same /metrics as the traffic figures: correlating
 				// "the cluster was full" with "requests were queueing" should
 				// not need two systems.
+				if err := rt.RegisterMetrics(tlsconf.ExpiryCollector(map[string]string{
+					"agent-server": cfg.AgentTLS.Cert, "agent-ca": cfg.AgentTLS.CA,
+				})); err != nil {
+					return err
+				}
 				if err := rt.RegisterMetrics(fleetmetrics.New(ctl)); err != nil {
 					return err
 				}
@@ -150,6 +160,7 @@ func newServerCmd() *cobra.Command {
 				// rather than accepting a model that vanishes on restart.
 				if st := rt.Store(); st != nil {
 					svc := modelsvc.New(cfg.Models, st, admin, log)
+					svc.SetCheck(servableCheck(cfg.WantedEngines()["llamacpp"]))
 					// Rebuild the catalog from configuration plus what was
 					// stored, and put that in force — otherwise models added
 					// before the last restart would be silently absent until
@@ -204,6 +215,11 @@ func newServerCmd() *cobra.Command {
 					web.NavExtras{History: history, Window: cfg.Monitoring.Window}, log))
 				adminMux.Handle("GET /_static/", web.Static())
 				adminMux.HandleFunc("GET /nodes/{id}/logs", web.NodeLogsPage(api, true, version.Version))
+				adminMux.HandleFunc("GET /nodes/{id}", web.NodePage(api, history, cfg.Monitoring.Window, true, version.Version, log))
+				adminMux.HandleFunc("GET /nvlink", web.NVLinkPage(api, true, version.Version))
+				adminMux.HandleFunc("GET /fabrics", web.FabricsPage(api, true, version.Version))
+				adminMux.HandleFunc("GET /fabrics/{id}", web.FabricsPage(api, true, version.Version))
+				adminMux.HandleFunc("GET /fabrics/{id}/topology.json", web.FabricTopologyJSON(api))
 
 				// The same page on the client-facing port at /admin, behind a
 				// session and an admin flag. Handing the router the data rather
@@ -236,7 +252,7 @@ func newServerCmd() *cobra.Command {
 				// enforces its own per-phase deadlines.
 			}
 			adminSrv := &http.Server{
-				Handler:           adminMux,
+				Handler:           router.AdminGuard(adminMux),
 				ReadHeaderTimeout: 10 * time.Second,
 			}
 
@@ -375,6 +391,20 @@ func startControlPlane(ctx context.Context, cfg config.Server, cat *catalog.Live
 		ccfg.Reconciler.PortRange = [2]int{cfg.Control.PortRange[0], cfg.Control.PortRange[1]}
 	}
 
+	// Each replica's engine demands a key only the router has. The secret it
+	// is derived from lives in the store, so every control plane against it —
+	// and this one after a restart — computes the same key for a replica.
+	if h, ok := sink.(interface{ Store() *storepkg.DB }); ok && h.Store() != nil {
+		secret, err := h.Store().ClusterSecret(ctx, "engine-key")
+		if err != nil {
+			return nil, nil, fmt.Errorf("reading the engine key secret: %w", err)
+		}
+		ccfg.Reconciler.EngineKey = control.EngineKeys([]byte(secret))
+	} else {
+		log.Warn("engines start without a key: there is no store to keep the secret " +
+			"they are derived from, so anyone who can reach a node's ports reaches its models")
+	}
+
 	// The service needs the inventory and the status sink, both of which the
 	// controller owns; the controller needs the service as its dispatcher. The
 	// cycle is broken by giving the controller a dispatcher that forwards to
@@ -413,6 +443,8 @@ func startControlPlane(ctx context.Context, cfg config.Server, cat *catalog.Live
 		svc.SetLeadership(lead)
 	}
 	fwd.set(svc)
+	// Every InfiniBand fabric mapped every five minutes, through one node.
+	go svc.ScanFabrics(ctx, 5*time.Minute)
 
 	ctl.Start(ctx)
 	return ctl, svc, nil
@@ -483,6 +515,18 @@ func (r *dispatcherRef) Logs(ctx context.Context, nodeID, source string, lines i
 	return lr.Logs(ctx, nodeID, source, lines)
 }
 
+func (r *dispatcherRef) FabricTest(ctx context.Context, nodeID string, req *llmgwv1.FabricTestRequest) (*llmgwv1.FabricTestReply, error) {
+	d, err := r.get()
+	if err != nil {
+		return nil, err
+	}
+	ft, ok := d.(fabrictest.Asker)
+	if !ok {
+		return nil, errors.New("this dispatcher cannot run a bandwidth test")
+	}
+	return ft.FabricTest(ctx, nodeID, req)
+}
+
 func (r *dispatcherRef) StartShard(ctx context.Context, cmd reconciler.ShardCommand) error {
 	d, err := r.get()
 	if err != nil {
@@ -535,3 +579,27 @@ func parseEndpointFlag(spec string) (config.Endpoint, error) {
 		Weight: 1,
 	}, nil
 }
+
+// servableCheck refuses a model whose llama.cpp weights the nodes' build
+// cannot load, read from the weights' own header before anything is placed.
+func servableCheck(llamaVersion string) func(context.Context, catalog.Model) error {
+	return func(ctx context.Context, m catalog.Model) error {
+		if llamaVersion == "" {
+			return nil
+		}
+		usesLlama := false
+		for _, p := range m.Profiles {
+			if p.Engine == "llamacpp" {
+				usesLlama = true
+			}
+		}
+		url, ok := sourcecheck.HTTPSource(m.Source)
+		if !usesLlama || !ok || !gguf.IsGGUF(url) {
+			return nil
+		}
+		if v := gguf.Check(ctx, url, llamaVersion); !v.Servable {
+			return fmt.Errorf("%s cannot be served: %s", m.Name, v.Reason)
+		}
+		return nil
+	}
+}

internal/cli/serverinit.go

@@ -69,6 +69,12 @@ type serverInit struct {
 	// postgres keeps the store in a PostgreSQL on this machine, installed if
 	// need be, moving whatever SQLite already holds.
 	postgres bool
+	// noMirror, noNode and noModel turn off the parts of a first install that
+	// make it serve something: a mirror, this machine as a node, and a small
+	// model on it.
+	noMirror, noNode, noModel bool
+	// mirrorURL is a mirror on another machine to use instead of running one.
+	mirrorURL string
 	// pgSetup prepares PostgreSQL and returns the store address; a seam for
 	// tests. Nil does it for real.
 	pgSetup func(context.Context) (string, error)
@@ -88,6 +94,7 @@ func newServerInitCmd() *cobra.Command {
 		unitPath: "/etc/systemd/system/llmgw-server.service",
 		user:     "llmgw",
 	}
+	var plain bool
 	cmd := &cobra.Command{
 		Use:   "init",
 		Short: "Set this machine up as a control plane and start it",
@@ -95,6 +102,11 @@ func newServerInitCmd() *cobra.Command {
 			"control plane's certificate for every address this machine has, a\n" +
 			"configuration, an administrator account, and the systemd unit — then\n" +
 			"starts it and waits until it answers.\n\n" +
+			"Then it makes the machine serve something: a model mirror beside the\n" +
+			"control plane, this machine as a node of it (on its GPUs when nvidia-smi\n" +
+			"finds any, on CPU otherwise), and " + starterName + " fetched into the\n" +
+			"mirror and served on this node. --no-mirror, --no-node and --no-model\n" +
+			"leave those out.\n\n" +
 			"With --postgres the store is a PostgreSQL database on this machine:\n" +
 			"installed if there is none, with its own role and a generated password\n" +
 			"kept in /etc/llmgw/secrets. An existing SQLite store is copied into it,\n" +
@@ -113,14 +125,23 @@ func newServerInitCmd() *cobra.Command {
 				return errors.New("init creates a user, writes /etc and starts a service: run it as root")
 			}
 			in.out = cmd.OutOrStdout()
+			if !plain && isTerminal(in.out) {
+				return in.runInitTUI(cmd.Context(), in.out)
+			}
 			return in.run(cmd.Context())
 		},
 	}
 	f := cmd.Flags()
+	f.BoolVar(&plain, "plain", false, "plain output, even in a terminal")
 	f.StringArrayVar(&in.hosts, "host", nil,
 		"an address agents dial besides the ones found on this machine; repeatable")
 	f.StringVar(&in.admin, "admin", "", "email of the first administrator; a password is generated and printed once")
 	f.StringVar(&in.pkiDir, "pki", in.pkiDir, "directory holding the certificate authority")
+	f.BoolVar(&in.noMirror, "no-mirror", false, "do not run a model mirror on this machine")
+	f.StringVar(&in.mirrorURL, "mirror", "",
+		"use the mirror on another machine, set up there by 'llmgw mirror init': its management API, http://host:8082")
+	f.BoolVar(&in.noNode, "no-node", false, "do not make this machine a node (and serve no model on it)")
+	f.BoolVar(&in.noModel, "no-model", false, "do not fetch and serve the starter model ("+starterName+")")
 	f.BoolVar(&in.postgres, "postgres", false,
 		"keep the store in PostgreSQL on this machine, installing it and moving an existing SQLite store into it")
 	return cmd
@@ -146,9 +167,38 @@ func (in *serverInit) run(ctx context.Context) error {
 	if in.unitPath == "" {
 		return nil
 	}
+	hasSystemd := true
+	if _, err := exec.LookPath("systemctl"); err != nil {
+		hasSystemd = false
+	}
+	// Before the service starts, because the mirror goes into server.yaml.
+	switch {
+	case in.mirrorURL != "":
+		if err := in.useMirror(ctx); err != nil {
+			return fmt.Errorf("mirror: %w", err)
+		}
+	case hasSystemd && !in.noMirror:
+		if err := in.ensureMirror(ctx); err != nil {
+			return fmt.Errorf("mirror: %w", err)
+		}
+	}
 	if err := in.ensureService(ctx); err != nil {
 		return fmt.Errorf("service: %w", err)
 	}
+	if hasSystemd && !in.noNode {
+		if _, err := in.ensureNode(ctx); err != nil {
+			return fmt.Errorf("node: %w", err)
+		}
+	}
+	switch {
+	case !hasSystemd || in.noModel:
+	case in.noMirror && in.mirrorURL == "":
+		fmt.Fprintln(in.out, "  skip  no starter model without the mirror")
+	default:
+		if err := in.ensureStarter(ctx); err != nil {
+			return fmt.Errorf("starter model: %w", err)
+		}
+	}
 	in.nextSteps()
 	return nil
 }
@@ -603,21 +653,17 @@ func (in *serverInit) waitHealthy(ctx context.Context) error {
 }
 
 func (in *serverInit) nextSteps() {
-	server := "https://<this machine>:7443"
-	if ips := localAddrs(); len(ips) > 0 {
-		server = "https://" + net.JoinHostPort(ips[0].String(), "7443")
-	}
-	fmt.Fprintf(in.out, `
+	fmt.Fprint(in.out, `
 The control plane is up. Next:
 
-  llmgw ctl status                      # no nodes, no models: correct for now
+  llmgw ctl status                      # the fleet in one answer
 
   # each node, from this machine (it holds the authority):
-  deploy/bootstrap/onboard-node.sh --node <name> --addr <node ip> --server %s
+  sudo llmgw node add <user>@<node address>
 
-  # then a model:
+  # then a model: /admin/mirror, "Explore models", or
   llmgw ctl model add -f model.yaml
 
 The web UI is on port 8080; /admin once you are signed in as the administrator.
-`, server)
+`)
 }

internal/cli/serverinit_cluster.go

@@ -0,0 +1,582 @@
+package cli
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"net"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"slices"
+	"strings"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/sourcecheck"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
+)
+
+// A control plane on its own serves nothing: it needs a node, weights, and a
+// model, and the install guide made each of those a separate afternoon. init
+// makes the machine all three — a mirror, a node, and a small model served on
+// it — so the first thing anyone sees is an answer. Each is skipped when it is
+// already there and can be turned off.
+
+const (
+	mirrorDir      = "/srv/models"
+	mirrorAPI      = "127.0.0.1:8082"
+	mirrorUnitPath = "/etc/systemd/system/llmgw-mirror.service"
+	agentUnitPath  = "/etc/systemd/system/llmgw-agent.service"
+
+	// The starter: small enough to download in a minute and run on any CPU,
+	// real enough to answer a question. Q4_K_M is about 400 MB.
+	starterName    = "qwen2.5-0.5b-instruct"
+	starterAlias   = "qwen-0.5b"
+	starterSource  = "hf:Qwen/Qwen2.5-0.5B-Instruct-GGUF/qwen2.5-0.5b-instruct-q4_k_m.gguf"
+	starterVersion = "starter"
+
+	// localLabel marks the control plane's own node, so the starter model is
+	// served where it was asked for and does not wander to the first GPU node
+	// that joins later.
+	localLabelKey   = "llmgw.control-plane"
+	localLabelValue = "true"
+)
+
+// agentUnit is deploy/systemd/llmgw-agent.service; a test holds them together.
+const agentUnit = `[Unit]
+Description=llmgw node agent
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+ExecStart=/usr/local/bin/llmgw agent --config /etc/llmgw/agent.yaml
+# Root, because the agent launches engines and needs device access. Engines
+# outlive a restart of this unit and are adopted again on the next start, so
+# KillMode must not take the whole process group with it.
+KillMode=process
+Restart=on-failure
+RestartSec=2s
+# RDMA pins the buffers it registers, and NCCL registers a great deal: with
+# systemd's default 8 MB lock limit, an engine that drops the capability that
+# lets root ignore it fails its first collective over InfiniBand.
+LimitMEMLOCK=infinity
+StateDirectory=llmgw-agent
+
+[Install]
+WantedBy=multi-user.target
+`
+
+// reachableAddr is the address other machines use for this one: the first
+// name given with --host, else the first address it has, else its hostname.
+// It is in the server certificate either way.
+func (in *serverInit) reachableAddr() string {
+	if len(in.hosts) > 0 {
+		return in.hosts[0]
+	}
+	if ips := localAddrs(); len(ips) > 0 {
+		return ips[0].String()
+	}
+	name, _ := os.Hostname()
+	return name
+}
+
+func (in *serverInit) systemctl(ctx context.Context, args ...string) error {
+	out, err := exec.CommandContext(ctx, "systemctl", args...).CombinedOutput()
+	if err != nil {
+		return fmt.Errorf("systemctl %s: %v: %s", strings.Join(args, " "), err, strings.TrimSpace(string(out)))
+	}
+	return nil
+}
+
+// writeUnit installs a unit when it differs from what is there, keeping
+// somebody's edited copy beside it, and says whether it changed.
+func (in *serverInit) writeUnit(path, body string) (bool, error) {
+	b, err := os.ReadFile(path)
+	if err == nil && string(b) == body {
+		return false, nil
+	}
+	if err == nil {
+		_ = os.WriteFile(path+".bak", b, 0o644)
+	}
+	return true, os.WriteFile(path, []byte(body), 0o644)
+}
+
+// --- the mirror ---
+
+func (in *serverInit) mirrorUnit() string {
+	base := "http://" + net.JoinHostPort(in.reachableAddr(), "8081")
+	return mirrorUnitFor(mirrorDir, mirrorAPI, base, nil,
+		"'llmgw server init'. Bytes on :8081 for every node; the\n# management API on loopback only, for the control plane beside it.")
+}
+
+// mirrorUnitFor is a mirror's systemd unit. allow is who, besides the
+// machine itself, may use the management API.
+func mirrorUnitFor(dir, apiListen, base string, allow []string, about string) string {
+	exec := "/usr/local/bin/llmgw mirror serve --dir " + dir + " --listen :8081 --api-listen " + apiListen + " --base " + base
+	for _, a := range allow {
+		exec += " --api-allow " + a
+	}
+	return `[Unit]
+Description=llmgw model mirror
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+# Written by ` + about + `
+ExecStart=` + exec + `
+User=llmgw
+Group=llmgw
+Restart=on-failure
+RestartSec=2s
+NoNewPrivileges=true
+ProtectSystem=strict
+ProtectHome=true
+PrivateTmp=true
+ReadWritePaths=` + dir + `
+
+[Install]
+WantedBy=multi-user.target
+`
+}
+
+// mirrorAPIURL is the mirror's management API: one on its own machine, or
+// the one beside this control plane.
+func (in *serverInit) mirrorAPIURL() string {
+	if in.mirrorURL != "" {
+		return in.mirrorURL
+	}
+	return "http://" + mirrorAPI
+}
+
+// useMirror points the control plane at a mirror on another machine, set up
+// there by 'llmgw mirror init'.
+func (in *serverInit) useMirror(ctx context.Context) error {
+	m, err := mirror.NewClient(in.mirrorURL, tlsconf.Config{}, 5*time.Second)
+	if err != nil {
+		return err
+	}
+	if err := m.Health(ctx); err != nil {
+		return fmt.Errorf("the mirror at %s does not answer: %v. On that machine: sudo llmgw mirror init --control-plane <this machine's address>", in.mirrorURL, err)
+	}
+	cfg, err := config.Load(in.configPath())
+	if err != nil {
+		return err
+	}
+	switch cfg.Mirror.API {
+	case in.mirrorURL:
+		in.kept("mirror at %s", in.mirrorURL)
+		return nil
+	case "":
+	default:
+		in.kept("mirror at %s, as server.yaml says; change it there to use %s", cfg.Mirror.API, in.mirrorURL)
+		return nil
+	}
+	if err := appendMirrorBlockFor(in.configPath(), in.mirrorURL,
+		"# The model mirror, on its own machine, added by 'llmgw server init'."); err != nil {
+		return err
+	}
+	in.done("server.yaml points at the mirror at %s", in.mirrorURL)
+	return nil
+}
+
+// ensureMirror runs a mirror on this machine and points the control plane at
+// it. Its management API is on loopback, which is why it needs no certificate:
+// nothing but this machine can reach it.
+func (in *serverInit) ensureMirror(ctx context.Context) error {
+	cfg, err := config.Load(in.configPath())
+	if err != nil {
+		return err
+	}
+	if cfg.Mirror.API != "" && cfg.Mirror.API != "http://"+mirrorAPI {
+		in.kept("mirror at %s, as server.yaml says", cfg.Mirror.API)
+		return nil
+	}
+
+	if err := os.MkdirAll(mirrorDir, 0o755); err != nil {
+		return err
+	}
+	if err := in.chown(mirrorDir, true); err != nil {
+		return err
+	}
+	changed, err := in.writeUnit(mirrorUnitPath, in.mirrorUnit())
+	if err != nil {
+		return err
+	}
+	if err := in.systemctl(ctx, "daemon-reload"); err != nil {
+		return err
+	}
+	if err := in.systemctl(ctx, "enable", "--now", "llmgw-mirror"); err != nil {
+		return err
+	}
+	if changed {
+		if err := in.systemctl(ctx, "restart", "llmgw-mirror"); err != nil {
+			return err
+		}
+		in.done("model mirror in %s, serving on :8081", mirrorDir)
+	} else {
+		in.kept("model mirror in %s", mirrorDir)
+	}
+
+	if cfg.Mirror.API == "" {
+		if err := appendMirrorBlockFor(in.configPath(), "http://"+mirrorAPI,
+			"# The model mirror on this machine, added by 'llmgw server init'. Its\n"+
+				"# management API is on loopback, so it needs no certificate."); err != nil {
+			return err
+		}
+		in.done("server.yaml points at the mirror")
+	}
+
+	m, err := mirror.NewClient("http://"+mirrorAPI, tlsconf.Config{}, 5*time.Second)
+	if err != nil {
+		return err
+	}
+	return waitFor(ctx, 30*time.Second, "the mirror to answer", func() (bool, error) {
+		return m.Health(ctx) == nil, nil
+	})
+}
+
+// appendMirrorBlockFor adds the mirror to server.yaml without touching the
+// rest, keeping the file as it was beside it.
+func appendMirrorBlockFor(path, api, comment string) error {
+	b, err := os.ReadFile(path)
+	if err != nil {
+		return err
+	}
+	backup := fmt.Sprintf("%s.bak-%s", path, time.Now().Format("20060102-150405"))
+	if err := os.WriteFile(backup, b, 0o640); err != nil {
+		return err
+	}
+	block := "\n" + comment + "\nmirror:\n  api: " + api + "\n"
+	info, err := os.Stat(path)
+	if err != nil {
+		return err
+	}
+	return os.WriteFile(path, append(b, []byte(block)...), info.Mode().Perm())
+}
+
+// --- this machine as a node ---
+
+// gpuClasses asks nvidia-smi what cards this machine has. None is an answer,
+// not an error: the node then serves on CPU.
+func gpuClasses(ctx context.Context) ([]string, int) {
+	gpus, err := discovery.DiscoverGPUsSMI(ctx, "")
+	if err != nil || len(gpus) == 0 {
+		return nil, 0
+	}
+	return discovery.Inventory{GPUs: gpus}.Classes(), len(gpus)
+}
+
+// ensureNode makes this machine an agent of its own control plane.
+func (in *serverInit) ensureNode(ctx context.Context) (string, error) {
+	nodeID, _ := os.Hostname()
+	if nodeID == "" {
+		return "", errors.New("this machine has no hostname to use as its node id")
+	}
+
+	if classes, n := gpuClasses(ctx); n > 0 {
+		in.kept("%d NVIDIA GPU(s) here: %s — the node will use them", n, strings.Join(classes, ", "))
+	} else if hasNVIDIADevice() {
+		fmt.Fprintf(in.out, "  !!    an NVIDIA card is on the PCI bus but nvidia-smi does not answer: "+
+			"install the driver and run init again to serve on it; until then this node is CPU only\n")
+	} else {
+		in.kept("no NVIDIA GPU here; the node serves on CPU")
+	}
+
+	certPath := filepath.Join(in.etcDir, "agent.pem")
+	if _, err := os.Stat(certPath); err == nil {
+		in.kept("agent certificate for %s", nodeID)
+	} else {
+		if _, err := os.Stat(filepath.Join(in.pkiDir, "ca-key.pem")); err != nil {
+			return "", fmt.Errorf("the authority is not on this machine, so init cannot give it a node " +
+				"identity; onboard it from where the authority is, or pass --no-node")
+		}
+		ca, err := loadCA(in.pkiDir)
+		if err != nil {
+			return "", err
+		}
+		leaf, err := ca.IssueClient(nodeID)
+		if err != nil {
+			return "", err
+		}
+		if err := leaf.Write(in.etcDir, "agent"); err != nil {
+			return "", err
+		}
+		in.done("agent certificate for %s", nodeID)
+	}
+
+	agentConf := filepath.Join(in.etcDir, "agent.yaml")
+	if _, err := os.Stat(agentConf); err == nil {
+		in.kept("%s (never overwritten)", agentConf)
+	} else {
+		server := "https://" + net.JoinHostPort(in.reachableAddr(), "7443")
+		body := fmt.Sprintf(`# This machine as a node of its own control plane, written by 'llmgw server init'.
+server: %q
+node_id: %q
+tls:
+  ca: %s/ca.pem
+  cert: %s/agent.pem
+  key: %s/agent-key.pem
+labels:
+  %s: %q
+`, server, nodeID, in.etcDir, in.etcDir, in.etcDir, localLabelKey, localLabelValue)
+		if err := os.WriteFile(agentConf, []byte(body), 0o644); err != nil {
+			return "", err
+		}
+		in.done("%s", agentConf)
+	}
+
+	changed, err := in.writeUnit(agentUnitPath, agentUnit)
+	if err != nil {
+		return "", err
+	}
+	if err := in.systemctl(ctx, "daemon-reload"); err != nil {
+		return "", err
+	}
+	if err := in.systemctl(ctx, "enable", "--now", "llmgw-agent"); err != nil {
+		return "", err
+	}
+	if changed {
+		_ = in.systemctl(ctx, "restart", "llmgw-agent")
+		in.done("llmgw-agent started")
+	}
+
+	// Registered is quick. Having an engine is not: the node downloads and
+	// checks its own llama.cpp, the CUDA build when it has a GPU, and a new
+	// machine is often busy with its own package updates first.
+	client := in.adminAPI()
+	last := ""
+	err = waitFor(ctx, 20*time.Minute, "this node to be ready with an engine", func() (bool, error) {
+		var nodes []adminapi.NodeView
+		if err := client.get(ctx, "/api/v1/nodes", &nodes); err != nil {
+			return false, nil
+		}
+		for _, n := range nodes {
+			if n.ID != nodeID {
+				continue
+			}
+			if n.State == "ready" && slices.Contains(n.Engines, "llamacpp") {
+				return true, nil
+			}
+			for _, p := range n.Provision {
+				if line := fmt.Sprintf("%s %s: %s %s", p.Engine, p.Version, p.State, p.Detail); line != last {
+					fmt.Fprintf(in.out, "  ..    %s\n", strings.TrimSpace(line))
+					last = line
+				}
+				// Only llama.cpp is waited for: vLLM installs for minutes after,
+				// and one this node cannot run is not a reason to stop here.
+				if p.Engine == "llamacpp" && (p.State == "failed" || p.State == "unsupported") {
+					return false, fmt.Errorf("installing %s on this node: %s", p.Engine, p.Detail)
+				}
+			}
+		}
+		return false, nil
+	})
+	if err != nil {
+		return "", err
+	}
+	in.kept("node %s is ready", nodeID)
+	return nodeID, nil
+}
+
+// hasNVIDIADevice reports an NVIDIA card on the PCI bus, driver or not, so a
+// machine whose GPU is invisible for want of a driver says so instead of
+// quietly serving on CPU.
+func hasNVIDIADevice() bool {
+	vendors, _ := filepath.Glob("/sys/bus/pci/devices/*/vendor")
+	for _, v := range vendors {
+		b, err := os.ReadFile(v)
+		if err == nil && strings.TrimSpace(string(b)) == "0x10de" {
+			class, _ := os.ReadFile(filepath.Join(filepath.Dir(v), "class"))
+			// 0x03xxxx is a display or 3D controller; NVIDIA also makes bridges.
+			if strings.HasPrefix(strings.TrimSpace(string(class)), "0x03") {
+				return true
+			}
+		}
+	}
+	return false
+}
+
+// --- the starter model ---
+
+// starterModel is the catalog entry: a GPU profile first when there is a card,
+// and a CPU profile always, both pinned to this machine.
+func starterModel(source string, gpuClasses []string, pinned bool) catalog.Model {
+	var pin map[string]string
+	if pinned {
+		pin = map[string]string{localLabelKey: localLabelValue}
+	}
+	var profiles []catalog.Profile
+	for _, class := range gpuClasses {
+		profiles = append(profiles, catalog.Profile{
+			Name: "gpu-" + class, Engine: "llamacpp", GPUClass: class,
+			Priority: 200, MinMemoryGB: 2, NodeSelector: pin,
+			Args: []string{"--ctx-size", "32768"},
+		})
+	}
+	profiles = append(profiles, catalog.Profile{
+		Name: "cpu", Engine: "llamacpp", Priority: 100, MinMemoryGB: 2, NodeSelector: pin,
+		Args: []string{"--ctx-size", "16384"},
+	})
+	return catalog.Model{
+		Name:           starterName,
+		Aliases:        []string{starterAlias},
+		Description:    "A small model init put here so there is something to try; fast, not clever.",
+		Source:         source,
+		RoutingProfile: "chat",
+		ContextLength:  4096,
+		MaxOutput:      1024,
+		Scaling:        catalog.Scaling{Min: 1, Max: 1},
+		Profiles:       profiles,
+	}
+}
+
+// ensureStarter fetches the starter model into the mirror and serves it here.
+func (in *serverInit) ensureStarter(ctx context.Context) error {
+	client := in.adminAPI()
+	var models []adminapi.ModelView
+	if err := client.get(ctx, "/api/v1/models", &models); err != nil {
+		return fmt.Errorf("reading the catalog: %w", err)
+	}
+	for _, m := range models {
+		if m.Name == starterName {
+			in.kept("%s in the catalog", starterName)
+			return in.waitForStarter(ctx)
+		}
+	}
+
+	upstream, ok := sourcecheck.HTTPSource(starterSource)
+	if !ok {
+		return fmt.Errorf("cannot turn %s into a download", starterSource)
+	}
+	m, err := mirror.NewClient(in.mirrorAPIURL(), tlsconf.Config{}, 15*time.Second)
+	if err != nil {
+		return err
+	}
+	st, err := m.Status(ctx, starterName, starterVersion)
+	var apiErr *mirror.APIError
+	switch {
+	case err == nil:
+	case errors.As(err, &apiErr) && apiErr.Status == 404:
+		if st, err = m.Fetch(ctx, mirror.FetchRequest{
+			Source: upstream, Model: starterName, Version: starterVersion,
+		}); err != nil {
+			return fmt.Errorf("asking the mirror for %s: %w", starterName, err)
+		}
+		fmt.Fprintf(in.out, "  ..    downloading %s into the mirror\n", starterName)
+	default:
+		return err
+	}
+
+	lastPct := -10
+	err = waitFor(ctx, 30*time.Minute, starterName+" to download", func() (bool, error) {
+		cur, err := m.Status(ctx, starterName, starterVersion)
+		if err != nil {
+			return false, nil
+		}
+		st = cur
+		switch cur.Status.State {
+		case mirror.Present:
+			return true, nil
+		case mirror.Failed:
+			return false, fmt.Errorf("the mirror could not download %s: %s", starterName, cur.Status.Err)
+		}
+		if pct, ok := cur.Status.Percent(); ok && pct >= lastPct+10 {
+			fmt.Fprintf(in.out, "  ..    %d%%\n", pct)
+			lastPct = pct
+		}
+		return false, nil
+	})
+	if err != nil {
+		return err
+	}
+	in.done("%s in the mirror", starterName)
+
+	// Served here when this machine is a node, pinned so it does not wander
+	// to the first GPU node that joins; otherwise on whichever node joins.
+	pinned := !in.noNode
+	var classes []string
+	if pinned {
+		classes, _ = gpuClasses(ctx)
+	}
+	body, err := json.Marshal(starterModel(st.URL, classes, pinned))
+	if err != nil {
+		return err
+	}
+	if err := client.post(ctx, "/api/v1/models", body, &map[string]string{}); err != nil {
+		return fmt.Errorf("adding %s to the catalog: %w", starterName, err)
+	}
+	if !pinned {
+		in.done("%s in the catalog; it starts on the first node that joins", starterName)
+		return nil
+	}
+	where := "on CPU"
+	if len(classes) > 0 {
+		where = "on " + strings.Join(classes, ", ") + ", or CPU if the card is busy"
+	}
+	in.done("%s in the catalog, served here %s", starterName, where)
+	return in.waitForStarter(ctx)
+}
+
+func (in *serverInit) waitForStarter(ctx context.Context) error {
+	client := in.adminAPI()
+	last := ""
+	return waitFor(ctx, 10*time.Minute, starterName+" to start", func() (bool, error) {
+		var reps []adminapi.ReplicaView
+		if err := client.get(ctx, "/api/v1/replicas?model="+starterName, &reps); err != nil {
+			return false, nil
+		}
+		for _, r := range reps {
+			if r.State == "ready" {
+				in.kept("%s is serving (profile %s)", starterName, r.Profile)
+				return true, nil
+			}
+		}
+		for _, r := range reps {
+			line := r.State
+			if r.Err != "" {
+				line += ": " + r.Err
+			}
+			if line != last {
+				fmt.Fprintf(in.out, "  ..    %s %s\n", starterName, line)
+				last = line
+			}
+		}
+		return false, nil
+	})
+}
+
+func (in *serverInit) adminAPI() *ctlClient {
+	addr := "127.0.0.1:9090"
+	if cfg, err := config.Load(in.configPath()); err == nil && cfg.AdminListen != "" {
+		addr = cfg.AdminListen
+	}
+	return newCtlClient(addr)
+}
+
+// waitFor polls until done says so, fails, or the time runs out.
+func waitFor(ctx context.Context, limit time.Duration, what string, done func() (bool, error)) error {
+	deadline := time.Now().Add(limit)
+	for {
+		ok, err := done()
+		if err != nil {
+			return err
+		}
+		if ok {
+			return nil
+		}
+		if time.Now().After(deadline) {
+			return fmt.Errorf("gave up waiting for %s after %s; run init again to keep waiting", what, limit)
+		}
+		select {
+		case <-ctx.Done():
+			return ctx.Err()
+		case <-time.After(2 * time.Second):
+		}
+	}
+}

internal/cli/serverinit_cluster_test.go

@@ -0,0 +1,123 @@
+package cli
+
+import (
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+)
+
+func TestTheInitAgentUnitIsTheShippedUnit(t *testing.T) {
+	b, err := os.ReadFile("../../deploy/systemd/llmgw-agent.service")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if string(b) != agentUnit {
+		t.Error("deploy/systemd/llmgw-agent.service and the unit 'server init' writes have drifted")
+	}
+}
+
+func TestTheStarterUsesTheGPUAndFallsBackToCPUOnThisMachine(t *testing.T) {
+	m := starterModel("http://10.0.0.5:8081/qwen2.5-0.5b-instruct/starter/q.gguf", []string{"4090-24g"}, true)
+	c, err := catalog.New([]catalog.Model{m})
+	if err != nil {
+		t.Fatalf("the starter is not a valid catalog entry: %v", err)
+	}
+	got, _ := c.Resolve(starterName)
+	if len(got.Profiles) != 2 {
+		t.Fatalf("profiles = %+v", got.Profiles)
+	}
+	gpu, cpu := got.Profiles[0], got.Profiles[1]
+	if gpu.GPUClass != "4090-24g" || gpu.Priority <= cpu.Priority {
+		t.Errorf("the GPU profile should come first: %+v", gpu)
+	}
+	if cpu.GPUClass != "" || cpu.Engine != "llamacpp" {
+		t.Errorf("the fallback should be llama.cpp on CPU: %+v", cpu)
+	}
+	for _, p := range got.Profiles {
+		if p.NodeSelector[localLabelKey] != localLabelValue {
+			t.Errorf("profile %s is not pinned to this machine", p.Name)
+		}
+	}
+	if got.Scaling.Min != 1 {
+		t.Error("the starter should be running, not waiting to be asked for")
+	}
+
+	if m := starterModel("http://x/q.gguf", nil, true); len(m.Profiles) != 1 || m.Profiles[0].Name != "cpu" {
+		t.Errorf("without a GPU only the CPU profile: %+v", m.Profiles)
+	}
+}
+
+func TestTheMirrorBlockIsAddedAndReads(t *testing.T) {
+	in, out := newTestInit(t)
+	if err := in.run(t.Context()); err != nil {
+		t.Fatalf("%v\n%s", err, out)
+	}
+	if err := appendMirrorBlockFor(in.configPath(), "http://"+mirrorAPI, "# the mirror"); err != nil {
+		t.Fatal(err)
+	}
+	cfg, err := config.Load(in.configPath())
+	if err != nil {
+		t.Fatal(err)
+	}
+	if cfg.Mirror.API != "http://"+mirrorAPI {
+		t.Errorf("mirror.api = %q", cfg.Mirror.API)
+	}
+	if err := cfg.Validate(); err != nil {
+		t.Error(err)
+	}
+	backups, _ := filepath.Glob(in.configPath() + ".bak-*")
+	if len(backups) != 1 {
+		t.Errorf("backups = %v", backups)
+	}
+}
+
+func TestTheMirrorUnitAdvertisesAnAddressNodesCanReach(t *testing.T) {
+	in := &serverInit{hosts: []string{"10.0.0.140"}}
+	u := in.mirrorUnit()
+	if !strings.Contains(u, "--base http://10.0.0.140:8081") {
+		t.Errorf("base: %s", u)
+	}
+	// The management port can delete models; nothing but this machine may
+	// reach it, which is what lets it go without a certificate.
+	if !strings.Contains(u, "--api-listen 127.0.0.1:8082") {
+		t.Errorf("the management API is not on loopback: %s", u)
+	}
+}
+
+func TestWithNoNodeHereTheStarterGoesToTheFirstNodeThatJoins(t *testing.T) {
+	m := starterModel("http://x/q.gguf", nil, false)
+	for _, p := range m.Profiles {
+		if len(p.NodeSelector) > 0 {
+			t.Errorf("profile %s is pinned to a machine that is not a node: %v", p.Name, p.NodeSelector)
+		}
+	}
+}
+
+func TestTheMirrorAPIOnlyAnswersItsControlPlane(t *testing.T) {
+	allowed, err := parseAllow([]string{"10.0.0.140", "10.1.0.0/24"})
+	if err != nil {
+		t.Fatal(err)
+	}
+	h := onlyFrom(allowed, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {}))
+	for addr, want := range map[string]int{
+		"10.0.0.140:5000": 200, "10.1.0.7:5000": 200, "127.0.0.1:5000": 200, "[::1]:5000": 200,
+		"10.0.0.141:5000": 403, "192.168.1.9:5000": 403,
+	} {
+		r := httptest.NewRequest("GET", "/api/v1/health", nil)
+		r.RemoteAddr = addr
+		w := httptest.NewRecorder()
+		h.ServeHTTP(w, r)
+		if w.Code != want {
+			t.Errorf("%s: %d, want %d", addr, w.Code, want)
+		}
+	}
+	if _, err := parseAllow([]string{"not-an-address"}); err == nil {
+		t.Error("nonsense was accepted")
+	}
+}

internal/cli/serverinit_test.go

@@ -59,7 +59,8 @@ func TestInitLeavesAControlPlaneThatStarts(t *testing.T) {
 	if err != nil {
 		t.Fatal(err)
 	}
-	if !u.IsAdmin || u.State != store.UserActive || u.ProjectID != "default" {
+	inDefault, _ := s.CanUseProject(context.Background(), u.ID, "default")
+	if !u.IsAdmin || u.State != store.UserActive || !inDefault {
 		t.Errorf("administrator not ready to use: %+v", u)
 	}
 	if !strings.Contains(out.String(), "password: ") {

internal/cli/status.go

@@ -94,6 +94,14 @@ func printSummary(cmd *cobra.Command, s adminapi.Summary) {
 	fmt.Fprintf(out, "\nMODELS    %d of %d have the replicas they asked for\n",
 		s.Models.Satisfied, s.Models.Total)
 
+	if len(s.Parked) > 0 {
+		fmt.Fprintln(out, "\nPARKED — failed every time they started; not retried until fixed")
+		for _, p := range s.Parked {
+			fmt.Fprintf(out, "  %s (%d failures): %s\n", p.Model, p.Failures, p.Reason)
+		}
+		fmt.Fprintln(out, "\n  Once fixed: llmgw ctl retry <model>")
+	}
+
 	if len(s.Unplaceable) > 0 {
 		fmt.Fprintln(out, "\nNOT PLACED")
 		for _, m := range s.Unplaceable {

internal/cli/tui_init.go

@@ -0,0 +1,421 @@
+package cli
+
+import (
+	"context"
+	"fmt"
+	"io"
+	"net/mail"
+	"os"
+	"path/filepath"
+	"strings"
+	"sync"
+
+	tea "github.com/charmbracelet/bubbletea"
+	"github.com/charmbracelet/lipgloss"
+)
+
+// The full-screen way through 'llmgw server init': a short form for what
+// init would otherwise take as flags, then init's own lines as a list that
+// fills in, with the administrator's password kept in view — and printed
+// again once the screen closes, so it is in the terminal's history.
+
+// initLineMsg is one line init wrote.
+type initLineMsg string
+
+// initDoneMsg is init finishing.
+type initDoneMsg struct{ err error }
+
+// lineWriter turns init's output into one message per line.
+type lineWriter struct {
+	mu   sync.Mutex
+	buf  []byte
+	send func(string)
+}
+
+func (w *lineWriter) Write(p []byte) (int, error) {
+	w.mu.Lock()
+	defer w.mu.Unlock()
+	w.buf = append(w.buf, p...)
+	for {
+		i := strings.IndexByte(string(w.buf), '\n')
+		if i < 0 {
+			return len(p), nil
+		}
+		w.send(string(w.buf[:i]))
+		w.buf = w.buf[i+1:]
+	}
+}
+
+// Form fields, in order.
+const (
+	fAdmin = iota
+	fHosts
+	fPostgres
+	fMirror
+	fNode
+	fModel
+	fStart
+	fCount
+)
+
+type initModel struct {
+	in    *serverInit
+	found []string // this machine's addresses
+
+	// The form, until it is submitted.
+	form      bool
+	focus     int
+	admin     textField
+	hosts     textField
+	toggles   [fCount]bool
+	formError string
+
+	// Then the run.
+	running  bool
+	lines    []string // init's lines, as it wrote them
+	password string
+	next     []string // the closing "Next:" text
+	spin     spinnerModel
+	width    int
+	done     bool
+	err      error
+	quit     bool
+	start    func()
+}
+
+func newInitModel(in *serverInit, found []string, askForm bool) initModel {
+	admin := newTextField()
+	admin.Placeholder = "you@example.com"
+	admin.SetValue(in.admin)
+	admin.CharLimit = 200
+	admin.Width = 40
+	hosts := newTextField()
+	hosts.Placeholder = "llm.example.com, 203.0.113.10"
+	hosts.SetValue(strings.Join(in.hosts, ", "))
+	hosts.CharLimit = 400
+	hosts.Width = 40
+	sp := newSpinner()
+	m := initModel{in: in, found: found, form: askForm, admin: admin, hosts: hosts, spin: sp, width: 100}
+	m.toggles[fPostgres] = in.postgres
+	m.toggles[fMirror] = !in.noMirror
+	m.toggles[fNode] = !in.noNode
+	m.toggles[fModel] = !in.noModel
+	if askForm {
+		m.admin.Focus()
+	}
+	return m
+}
+
+func (m initModel) Init() tea.Cmd {
+	if m.form {
+		return nil
+	}
+	return tea.Batch(m.spin.Tick, func() tea.Msg { return startInitMsg{} })
+}
+
+type startInitMsg struct{}
+
+func (m initModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
+	switch msg := msg.(type) {
+	case tea.WindowSizeMsg:
+		m.width = msg.Width
+	case spinTickMsg:
+		var cmd tea.Cmd
+		m.spin, cmd = m.spin.Update(msg)
+		return m, cmd
+	case startInitMsg:
+		m.running = true
+		if m.start != nil {
+			m.start()
+		}
+	case initLineMsg:
+		m.addLine(string(msg))
+	case initDoneMsg:
+		m.done, m.err = true, msg.err
+		return m, tea.Quit
+	case tea.KeyMsg:
+		if msg.Type == tea.KeyCtrlC {
+			m.quit = true
+			return m, tea.Quit
+		}
+		if m.form {
+			return m.formKey(msg)
+		}
+	}
+	return m, nil
+}
+
+// addLine files one of init's lines: the password aside, the closing text
+// aside, a progress line replacing the one before it.
+func (m *initModel) addLine(l string) {
+	t := strings.TrimSpace(l)
+	switch {
+	case t == "":
+		return
+	case strings.HasPrefix(t, "password: "):
+		m.password = strings.TrimPrefix(t, "password: ")
+		return
+	case strings.HasPrefix(t, "shown once"):
+		return
+	case !strings.HasPrefix(l, "  ") || len(m.next) > 0:
+		// Unindented lines are the closing text, and everything after its
+		// first line belongs to it.
+		m.next = append(m.next, l)
+		return
+	}
+	if strings.HasPrefix(t, "..") && len(m.lines) > 0 && strings.HasPrefix(strings.TrimSpace(m.lines[len(m.lines)-1]), "..") {
+		m.lines[len(m.lines)-1] = l
+		return
+	}
+	m.lines = append(m.lines, l)
+}
+
+func (m initModel) formKey(k tea.KeyMsg) (tea.Model, tea.Cmd) {
+	switch k.Type {
+	case tea.KeyTab, tea.KeyDown:
+		m.moveFocus(1)
+		return m, nil
+	case tea.KeyShiftTab, tea.KeyUp:
+		m.moveFocus(-1)
+		return m, nil
+	case tea.KeyEnter:
+		if m.focus != fStart {
+			m.moveFocus(1)
+			return m, nil
+		}
+		admin := strings.TrimSpace(m.admin.Value())
+		if admin != "" {
+			if _, err := mail.ParseAddress(admin); err != nil {
+				m.formError = "the administrator's email is not an address"
+				m.focus = fAdmin
+				m.admin.Focus()
+				return m, nil
+			}
+		}
+		m.in.admin = admin
+		m.in.hosts = nil
+		for _, h := range strings.Split(m.hosts.Value(), ",") {
+			if h = strings.TrimSpace(h); h != "" {
+				m.in.hosts = append(m.in.hosts, h)
+			}
+		}
+		m.in.postgres = m.toggles[fPostgres]
+		m.in.noMirror = !m.toggles[fMirror]
+		m.in.noNode = !m.toggles[fNode]
+		m.in.noModel = !m.toggles[fModel]
+		m.form = false
+		return m, tea.Batch(m.spin.Tick, func() tea.Msg { return startInitMsg{} })
+	}
+	switch m.focus {
+	case fAdmin:
+		var cmd tea.Cmd
+		m.admin, cmd = m.admin.Update(k)
+		return m, cmd
+	case fHosts:
+		var cmd tea.Cmd
+		m.hosts, cmd = m.hosts.Update(k)
+		return m, cmd
+	case fPostgres, fMirror, fNode, fModel:
+		if k.String() == " " || k.String() == "x" {
+			m.toggles[m.focus] = !m.toggles[m.focus]
+		}
+	}
+	return m, nil
+}
+
+func (m *initModel) moveFocus(d int) {
+	m.focus = (m.focus + d + fCount) % fCount
+	// With --mirror there is no mirror to run here, so nothing to tick.
+	if m.focus == fMirror && m.in.mirrorURL != "" {
+		m.focus = (m.focus + d + fCount) % fCount
+	}
+	m.admin.Blur()
+	m.hosts.Blur()
+	switch m.focus {
+	case fAdmin:
+		m.admin.Focus()
+	case fHosts:
+		m.hosts.Focus()
+	}
+}
+
+func (m initModel) View() string {
+	var b strings.Builder
+	b.WriteString(sTitle.Render("llmgw · setting up this machine as a control plane") + "\n\n")
+	if m.form {
+		return b.String() + m.formView()
+	}
+	for _, l := range m.visibleLines() {
+		b.WriteString(m.lineView(l) + "\n")
+	}
+	if m.password != "" {
+		box := sBox.BorderForeground(tuiWarn).Render(
+			fmt.Sprintf("Administrator %s\npassword  %s\n%s", m.in.admin,
+				lipgloss.NewStyle().Bold(true).Render(m.password),
+				sDim.Render("Shown once. Copy it now; change it on the account page after signing in.")))
+		b.WriteString("\n" + box + "\n")
+	}
+	switch {
+	case m.done && m.err != nil:
+		b.WriteString("\n" + sBad.Render("✗ ") + wrap(m.err.Error(), m.width-4, "  ") + "\n" +
+			sDim.Render("  It is safe to run init again once that is fixed: it finishes what is missing.") + "\n")
+	case m.done:
+		b.WriteString("\n" + sOK.Render("✓ ") + "The control plane is up. Add a node: " +
+			lipgloss.NewStyle().Bold(true).Render("sudo llmgw node add <user>@<address>") + "\n")
+		b.WriteString("  " + sDim.Render("Back up "+m.in.pkiDir+": its ca-key.pem signs every node's identity, and nothing recovers it.") + "\n")
+	case m.quit:
+		b.WriteString("\n" + sDim.Render("stopped; run init again to finish") + "\n")
+	case m.running:
+		b.WriteString("\n" + sDim.Render("ctrl+c to stop") + "\n")
+	}
+	return b.String()
+}
+
+// visibleLines are the last lines, enough to follow without the screen
+// outgrowing the terminal: the whole log is printed when it closes.
+func (m initModel) visibleLines() []string {
+	const keep = 16
+	if len(m.lines) > keep {
+		return m.lines[len(m.lines)-keep:]
+	}
+	return m.lines
+}
+
+func (m initModel) lineView(l string) string {
+	t := strings.TrimSpace(l)
+	word, rest, _ := strings.Cut(t, " ")
+	rest = truncate(strings.TrimSpace(rest), max(20, m.width-6))
+	switch word {
+	case "ok":
+		return "  " + sOK.Render("✓") + " " + rest
+	case "kept":
+		return "  " + sDim.Render("· "+rest)
+	case "skip":
+		return "  " + sDim.Render("- "+rest)
+	case "!!":
+		return "  " + sWarn.Render("! ") + rest
+	case "..":
+		if m.done {
+			return "  " + sDim.Render("· "+rest)
+		}
+		return "  " + m.spin.View() + " " + rest
+	}
+	return "    " + t
+}
+
+func (m initModel) formView() string {
+	var b strings.Builder
+	label := func(i int, s string) string {
+		if m.focus == i {
+			return sTitle.Render("› " + s)
+		}
+		return "  " + s
+	}
+	check := func(i int, s, hint string) string {
+		box := "[ ]"
+		if m.toggles[i] {
+			box = "[" + sOK.Render("x") + "]"
+		}
+		return fmt.Sprintf("%s %s %s\n", label(i, box), s, sDim.Render(hint))
+	}
+	fmt.Fprintf(&b, "%s\n    %s\n", label(fAdmin, "Administrator's email"), m.admin.View())
+	b.WriteString("    " + sDim.Render("A password is generated and shown once. Leave empty to create accounts later.") + "\n\n")
+	fmt.Fprintf(&b, "%s\n    %s\n", label(fHosts, "Other names nodes will reach this machine by"), m.hosts.View())
+	found := "none found"
+	if len(m.found) > 0 {
+		found = strings.Join(m.found, ", ")
+	}
+	b.WriteString("    " + sDim.Render("Already in the certificate: "+found+". Add a DNS name or NAT address here.") + "\n\n")
+	b.WriteString(check(fPostgres, "PostgreSQL", "rather than SQLite; installed here, for several control planes or a larger store"))
+	if m.in.mirrorURL != "" {
+		b.WriteString("  Model mirror " + sDim.Render("at "+m.in.mirrorURL+", from --mirror") + "\n")
+	} else {
+		b.WriteString(check(fMirror, "Model mirror", "downloads model files once for every node"))
+	}
+	b.WriteString(check(fNode, "This machine as a node", "serves models here too, on its GPUs or CPU"))
+	b.WriteString(check(fModel, "Starter model", "a small Qwen, so there is something to ask at once"))
+	b.WriteString("\n" + label(fStart, lipgloss.NewStyle().Bold(true).Render("Set up")) + "\n")
+	if m.formError != "" {
+		b.WriteString("\n" + sBad.Render(m.formError) + "\n")
+	}
+	b.WriteString("\n" + sDim.Render("tab or ↑↓ to move · space to tick · enter on Set up to start · ctrl+c to leave") + "\n")
+	return b.String()
+}
+
+// runInitTUI runs init with the screen above, asking the form first when
+// the flags left the administrator unset.
+func (in *serverInit) runInitTUI(ctx context.Context, out io.Writer) error {
+	var found []string
+	for _, ip := range localAddrs() {
+		found = append(found, ip.String())
+	}
+	m := newInitModel(in, found, in.admin == "")
+	var p *tea.Program
+
+	var (
+		logMu sync.Mutex
+		log   []string
+	)
+	w := &lineWriter{send: func(l string) {
+		logMu.Lock()
+		log = append(log, l)
+		logMu.Unlock()
+		p.Send(initLineMsg(l))
+	}}
+	ctx, cancel := context.WithCancel(ctx)
+	defer cancel()
+	started := make(chan struct{})
+	m.start = func() {
+		close(started)
+		go func() {
+			in.out = w
+			err := in.run(ctx)
+			p.Send(initDoneMsg{err})
+		}()
+	}
+	p = tea.NewProgram(m, tea.WithOutput(out))
+	final, err := p.Run()
+	if err != nil {
+		return err
+	}
+	fm := final.(initModel)
+	select {
+	case <-started:
+	default:
+		return fmt.Errorf("left before anything was done")
+	}
+	if fm.quit && !fm.done {
+		cancel()
+		return fmt.Errorf("stopped; run init again to finish what was started")
+	}
+
+	// The whole run is kept, root's only, since it holds the password. On a
+	// failure it is printed too, for the lines the screen had no room for;
+	// otherwise the screen's last frame says it, and only what must outlive
+	// the screen is printed again: the password, and what to do next.
+	logMu.Lock()
+	defer logMu.Unlock()
+	whole := strings.Join(log, "\n") + "\n"
+	saved := ""
+	if in.stateDir != "" {
+		p := filepath.Join(in.stateDir, "init.log")
+		if err := os.WriteFile(p, []byte(whole), 0o600); err == nil {
+			saved = p
+		}
+	}
+	fmt.Fprintln(out)
+	if fm.err != nil {
+		fmt.Fprint(out, whole)
+		return fm.err
+	}
+	if fm.password != "" {
+		fmt.Fprintf(out, "Administrator %s, password %s (shown once)\n\n", in.admin, fm.password)
+	}
+	for _, l := range fm.next {
+		fmt.Fprintln(out, l)
+	}
+	if saved != "" {
+		fmt.Fprintf(out, "\nThe whole run is in %s.\n", saved)
+	}
+	return fm.err
+}

internal/cli/tui_init_test.go

@@ -0,0 +1,103 @@
+package cli
+
+import (
+	"strings"
+	"testing"
+
+	tea "github.com/charmbracelet/bubbletea"
+)
+
+func typeIn(m tea.Model, s string) tea.Model {
+	return send(m, tea.KeyMsg{Type: tea.KeyRunes, Runes: []rune(s)})
+}
+
+func TestTheInitFormBecomesInitsSettings(t *testing.T) {
+	in := &serverInit{pkiDir: "/etc/llmgw/pki"}
+	var m tea.Model = newInitModel(in, []string{"10.0.0.140"}, true)
+	if v := m.View(); !strings.Contains(v, "Already in the certificate: 10.0.0.140") {
+		t.Fatalf("the addresses found are not shown:\n%s", v)
+	}
+	m = typeIn(m, "ops@example.com")
+	m = send(m, tea.KeyMsg{Type: tea.KeyTab})
+	m = typeIn(m, "llm.example.com, 203.0.113.10")
+	m = send(m, tea.KeyMsg{Type: tea.KeyTab}, tea.KeyMsg{Type: tea.KeyRunes, Runes: []rune(" ")}) // PostgreSQL on
+	m = send(m, tea.KeyMsg{Type: tea.KeyTab}, tea.KeyMsg{Type: tea.KeyTab}, tea.KeyMsg{Type: tea.KeyTab},
+		tea.KeyMsg{Type: tea.KeyRunes, Runes: []rune(" ")}) // starter model off
+	m = send(m, tea.KeyMsg{Type: tea.KeyTab}, tea.KeyMsg{Type: tea.KeyEnter})
+
+	if in.admin != "ops@example.com" || strings.Join(in.hosts, "|") != "llm.example.com|203.0.113.10" {
+		t.Errorf("admin %q hosts %q", in.admin, in.hosts)
+	}
+	if !in.postgres || in.noMirror || in.noNode || !in.noModel {
+		t.Errorf("toggles: postgres %v mirror %v node %v model %v", in.postgres, !in.noMirror, !in.noNode, !in.noModel)
+	}
+	if m.(initModel).form {
+		t.Error("the form stayed up after Set up")
+	}
+}
+
+func TestAnEmailThatIsNotOneIsSaidSo(t *testing.T) {
+	in := &serverInit{}
+	var m tea.Model = newInitModel(in, nil, true)
+	m = typeIn(m, "not an address")
+	for range fStart {
+		m = send(m, tea.KeyMsg{Type: tea.KeyTab})
+	}
+	m = send(m, tea.KeyMsg{Type: tea.KeyEnter})
+	if !m.(initModel).form || !strings.Contains(m.View(), "not an address") {
+		t.Errorf("a bad email was accepted:\n%s", m.View())
+	}
+}
+
+func TestInitsLinesAreFiledAndThePasswordKeptInView(t *testing.T) {
+	in := &serverInit{admin: "ops@example.com", pkiDir: "/etc/llmgw/pki"}
+	var m tea.Model = newInitModel(in, nil, false)
+	for _, l := range []string{
+		"  ok    system user llmgw",
+		"  kept  certificate authority in /etc/llmgw/pki",
+		"  ok    administrator ops@example.com (u1)",
+		"",
+		"        password: Tr0ub4dor-3",
+		"        shown once — sign in and change it from /profile",
+		"",
+		"  ..    downloading qwen2.5-0.5b-instruct into the mirror",
+		"  ..    40%",
+		"  ..    80%",
+		"  !!    an NVIDIA card is on the PCI bus but nvidia-smi does not answer",
+		"The control plane is up. Next:",
+		"  llmgw ctl status",
+	} {
+		m = send(m, initLineMsg(l))
+	}
+	m = send(m, initDoneMsg{})
+	im := m.(initModel)
+	if im.password != "Tr0ub4dor-3" {
+		t.Errorf("password %q", im.password)
+	}
+	v := m.View()
+	for _, want := range []string{"✓ system user llmgw", "Tr0ub4dor-3", "Shown once", "80%", "! an NVIDIA card", "Back up /etc/llmgw/pki", "node add"} {
+		if !strings.Contains(v, want) {
+			t.Errorf("screen lacks %q:\n%s", want, v)
+		}
+	}
+	if strings.Contains(v, "40%") {
+		t.Error("progress lines piled up instead of replacing each other")
+	}
+	if strings.Contains(v, "llmgw ctl status") {
+		t.Error("the closing text was listed as a step")
+	}
+}
+
+func TestWithAMirrorElsewhereTheFormDoesNotOfferOne(t *testing.T) {
+	in := &serverInit{mirrorURL: "http://10.0.0.150:8082"}
+	var m tea.Model = newInitModel(in, nil, true)
+	if v := m.View(); !strings.Contains(v, "at http://10.0.0.150:8082, from --mirror") {
+		t.Errorf("the mirror in use is not shown:\n%s", v)
+	}
+	// Tab from the addresses goes past the mirror line to PostgreSQL, then
+	// the node.
+	m = send(m, tea.KeyMsg{Type: tea.KeyTab}, tea.KeyMsg{Type: tea.KeyTab}, tea.KeyMsg{Type: tea.KeyTab})
+	if f := m.(initModel).focus; f != fNode {
+		t.Errorf("focus %d, want the node toggle %d", f, fNode)
+	}
+}

internal/cli/tui_node.go

@@ -0,0 +1,351 @@
+package cli
+
+import (
+	"context"
+	"fmt"
+	"io"
+	"os"
+	"strings"
+
+	tea "github.com/charmbracelet/bubbletea"
+	"github.com/charmbracelet/lipgloss"
+	"github.com/mattn/go-isatty"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
+)
+
+// The full-screen way through 'llmgw node add': the same steps as the plain
+// one, drawn as a checklist that fills in, with the questions a script
+// cannot ask — which fabric the adapters are on, whether to go on past a
+// problem — asked in place.
+
+func isTerminal(w io.Writer) bool {
+	f, ok := w.(*os.File)
+	return ok && isatty.IsTerminal(f.Fd()) && os.Getenv("TERM") != "dumb"
+}
+
+var (
+	tuiAccent = lipgloss.AdaptiveColor{Light: "#0B7F7D", Dark: "#3FC2BC"}
+	tuiOK     = lipgloss.AdaptiveColor{Light: "#1F8A4C", Dark: "#6FD3A6"}
+	tuiWarn   = lipgloss.AdaptiveColor{Light: "#A15C00", Dark: "#E9B35A"}
+	tuiBad    = lipgloss.AdaptiveColor{Light: "#B42318", Dark: "#F07C6C"}
+	tuiDim    = lipgloss.AdaptiveColor{Light: "#56696C", Dark: "#8FA5A4"}
+
+	sTitle = lipgloss.NewStyle().Bold(true).Foreground(tuiAccent)
+	sStep  = lipgloss.NewStyle().Width(14)
+	sDim   = lipgloss.NewStyle().Foreground(tuiDim)
+	sOK    = lipgloss.NewStyle().Foreground(tuiOK)
+	sWarn  = lipgloss.NewStyle().Foreground(tuiWarn)
+	sBad   = lipgloss.NewStyle().Foreground(tuiBad)
+	sBox   = lipgloss.NewStyle().Border(lipgloss.RoundedBorder()).BorderForeground(tuiAccent).Padding(0, 1)
+)
+
+// Messages from the onboarding goroutine to the screen.
+type (
+	evMsg     onboard.Event
+	finishMsg struct {
+		node, summary string
+		err           error
+	}
+	askFabricsMsg struct {
+		suggested []onboard.Fabric
+		reply     chan []onboard.Fabric
+	}
+	askContinueMsg struct {
+		fails int
+		reply chan bool
+	}
+)
+
+type stepState struct {
+	level, detail string
+}
+
+type nodeModel struct {
+	title   string
+	steps   map[string]stepState
+	notes   []onboard.Event // warnings and failures, with their fixes
+	spin    spinnerModel
+	cancel  context.CancelFunc
+	width   int
+	done    bool
+	err     error
+	node    string
+	summary string
+
+	// The question being asked, if any.
+	fabrics   *askFabricsMsg
+	inputs    []textField
+	focus     int
+	confirm   *askContinueMsg
+	quitAsked bool
+}
+
+func newNodeModel(title string, cancel context.CancelFunc) nodeModel {
+	sp := newSpinner()
+	return nodeModel{title: title, steps: map[string]stepState{}, spin: sp, cancel: cancel, width: 100}
+}
+
+func (m nodeModel) Init() tea.Cmd { return m.spin.Tick }
+
+func (m nodeModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
+	switch msg := msg.(type) {
+	case tea.WindowSizeMsg:
+		m.width = msg.Width
+	case spinTickMsg:
+		var cmd tea.Cmd
+		m.spin, cmd = m.spin.Update(msg)
+		return m, cmd
+	case evMsg:
+		e := onboard.Event(msg)
+		switch e.Level {
+		case onboard.Running, onboard.Done:
+			m.steps[e.Step] = stepState{e.Level, e.Detail}
+		case onboard.Fail:
+			m.steps[e.Step] = stepState{e.Level, e.Detail}
+			if e.Step == onboard.StepCheck && strings.Contains(e.Detail, ":") {
+				m.notes = append(m.notes, e)
+			}
+		case onboard.Warn:
+			m.notes = append(m.notes, e)
+		case onboard.Info:
+			if e.Step == onboard.StepEngines {
+				m.steps[e.Step] = stepState{onboard.Running, e.Detail}
+			}
+		}
+	case askFabricsMsg:
+		m.fabrics = &msg
+		m.inputs = nil
+		for i, f := range msg.suggested {
+			in := newTextField()
+			in.Prompt = ""
+			in.SetValue(f.Name)
+			in.CharLimit = 40
+			in.Width = 24
+			if i == 0 {
+				in.Focus()
+			}
+			m.inputs = append(m.inputs, in)
+		}
+		m.focus = 0
+	case askContinueMsg:
+		m.confirm = &msg
+	case finishMsg:
+		m.done, m.err, m.node, m.summary = true, msg.err, msg.node, msg.summary
+		return m, tea.Quit
+	case tea.KeyMsg:
+		return m.key(msg)
+	}
+	return m, nil
+}
+
+func (m nodeModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) {
+	if k.Type == tea.KeyCtrlC {
+		m.cancel()
+		m.quitAsked = true
+		return m, tea.Quit
+	}
+	switch {
+	case m.confirm != nil:
+		switch strings.ToLower(k.String()) {
+		case "y":
+			m.confirm.reply <- true
+			m.confirm = nil
+		case "n", "enter", "esc":
+			m.confirm.reply <- false
+			m.confirm = nil
+		}
+		return m, nil
+	case m.fabrics != nil:
+		switch k.Type {
+		case tea.KeyTab, tea.KeyDown:
+			m.move(1)
+			return m, nil
+		case tea.KeyShiftTab, tea.KeyUp:
+			m.move(-1)
+			return m, nil
+		case tea.KeyEsc:
+			m.fabrics.reply <- nil
+			m.fabrics = nil
+			return m, nil
+		case tea.KeyEnter:
+			var out []onboard.Fabric
+			for i, in := range m.inputs {
+				if name := strings.TrimSpace(in.Value()); name != "" {
+					out = append(out, onboard.Fabric{Name: name, HCAs: m.fabrics.suggested[i].HCAs, Key: m.fabrics.suggested[i].Key})
+				}
+			}
+			m.fabrics.reply <- out
+			m.fabrics = nil
+			return m, nil
+		}
+		var cmd tea.Cmd
+		m.inputs[m.focus], cmd = m.inputs[m.focus].Update(k)
+		return m, cmd
+	}
+	if k.String() == "q" {
+		m.cancel()
+		m.quitAsked = true
+		return m, tea.Quit
+	}
+	return m, nil
+}
+
+func (m *nodeModel) move(d int) {
+	m.inputs[m.focus].Blur()
+	m.focus = (m.focus + d + len(m.inputs)) % len(m.inputs)
+	m.inputs[m.focus].Focus()
+}
+
+func (m nodeModel) View() string {
+	var b strings.Builder
+	b.WriteString(sTitle.Render(m.title) + "\n\n")
+	detailWidth := max(20, m.width-22)
+	for _, step := range onboard.Steps {
+		st, ok := m.steps[step]
+		var mark string
+		switch {
+		case !ok:
+			mark = sDim.Render("·")
+		case st.level == onboard.Running && !m.done:
+			mark = m.spin.View()
+		case st.level == onboard.Fail:
+			mark = sBad.Render("✗")
+		case st.level == onboard.Running:
+			mark = sDim.Render("·")
+		default:
+			mark = sOK.Render("✓")
+		}
+		name := sStep.Render(step)
+		if !ok {
+			name = sDim.Render(sStep.Render(step))
+		}
+		fmt.Fprintf(&b, "  %s %s %s\n", mark, name, truncate(st.detail, detailWidth))
+	}
+
+	if len(m.notes) > 0 {
+		b.WriteString("\n")
+		for _, n := range m.notes {
+			mark := sWarn.Render("!")
+			if n.Level == onboard.Fail {
+				mark = sBad.Render("✗")
+			}
+			fmt.Fprintf(&b, "  %s %s\n", mark, wrap(n.Detail, m.width-6, "    "))
+			if n.Fix != "" {
+				fmt.Fprintf(&b, "    %s\n", sDim.Render("→ "+wrap(n.Fix, m.width-8, "      ")))
+			}
+		}
+	}
+
+	switch {
+	case m.fabrics != nil:
+		var q strings.Builder
+		q.WriteString("These adapters are up. Which fabric is each group on?\n")
+		q.WriteString(sDim.Render("Give each the name it has on every other node attached to it.\nLeave a name empty to leave that group out.") + "\n\n")
+		for i, f := range m.fabrics.suggested {
+			fmt.Fprintf(&q, "  %s  %s\n", m.inputs[i].View(), sDim.Render(strings.Join(f.HCAs, ", ")))
+		}
+		q.WriteString("\n" + sDim.Render("tab next · enter use these · esc none"))
+		b.WriteString("\n" + sBox.Render(q.String()) + "\n")
+	case m.confirm != nil:
+		q := fmt.Sprintf("doctor found %d problem(s) above.\nAdd the node anyway? %s", m.confirm.fails, sDim.Render("y / N"))
+		b.WriteString("\n" + sBox.BorderForeground(tuiWarn).Render(q) + "\n")
+	case m.done && m.err != nil:
+		b.WriteString("\n" + sBad.Render("✗ ") + wrap(m.err.Error(), m.width-4, "  ") + "\n")
+	case m.done:
+		b.WriteString("\n" + sOK.Render("✓ ") + m.summary + "\n")
+	case m.quitAsked:
+		b.WriteString("\n" + sDim.Render("stopped") + "\n")
+	default:
+		b.WriteString("\n" + sDim.Render("ctrl+c to stop") + "\n")
+	}
+	return b.String()
+}
+
+func truncate(s string, n int) string {
+	if n <= 1 || len([]rune(s)) <= n {
+		return s
+	}
+	r := []rune(s)
+	return string(r[:n-1]) + "…"
+}
+
+// wrap breaks s at spaces to fit width, indenting every line after the first.
+func wrap(s string, width int, indent string) string {
+	if width < 20 {
+		return s
+	}
+	var lines []string
+	line := ""
+	for _, w := range strings.Fields(s) {
+		if line != "" && len(line)+1+len(w) > width {
+			lines = append(lines, line)
+			line = w
+			continue
+		}
+		if line == "" {
+			line = w
+		} else {
+			line += " " + w
+		}
+	}
+	lines = append(lines, line)
+	return strings.Join(lines, "\n"+indent)
+}
+
+// runTUI adds one node with the screen above.
+func (na *nodeAdd) runTUI(ctx context.Context, o onboard.Options, title string) (string, error) {
+	ctx, cancel := context.WithCancel(ctx)
+	defer cancel()
+	m := newNodeModel(title, cancel)
+	p := tea.NewProgram(m)
+
+	emit := func(e onboard.Event) { p.Send(evMsg(e)) }
+	o.AskFabrics = na.rememberFabrics(func(s []onboard.Fabric) []onboard.Fabric {
+		reply := make(chan []onboard.Fabric)
+		p.Send(askFabricsMsg{s, reply})
+		select {
+		case f := <-reply:
+			return f
+		case <-ctx.Done():
+			return nil
+		}
+	})
+	o.AskContinue = func(fails int) bool {
+		reply := make(chan bool)
+		p.Send(askContinueMsg{fails, reply})
+		select {
+		case ok := <-reply:
+			return ok
+		case <-ctx.Done():
+			return false
+		}
+	}
+
+	go func() {
+		res, err := onboard.New(o, emit).Run(ctx)
+		node := ""
+		if res != nil {
+			node = res.Node
+		}
+		if err == nil && !na.noWait {
+			err = na.watch(ctx, node, emit)
+		}
+		summary := ""
+		if err == nil {
+			summary = node + " is in the cluster"
+		}
+		p.Send(finishMsg{node, summary, err})
+	}()
+
+	final, err := p.Run()
+	if err != nil {
+		return "", err
+	}
+	fm := final.(nodeModel)
+	if fm.quitAsked && !fm.done {
+		cancel()
+		return fm.node, fmt.Errorf("stopped; the node may be half set up, and running this again finishes it")
+	}
+	return fm.node, fm.err
+}

internal/cli/tui_node_test.go

@@ -0,0 +1,117 @@
+package cli
+
+import (
+	"strings"
+	"testing"
+
+	"time"
+
+	tea "github.com/charmbracelet/bubbletea"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
+)
+
+func send(m tea.Model, msgs ...tea.Msg) tea.Model {
+	for _, msg := range msgs {
+		m, _ = m.Update(msg)
+	}
+	return m
+}
+
+func TestTheNodeScreenFillsInAsStepsFinish(t *testing.T) {
+	var m tea.Model = newNodeModel("adding gpu01", func() {})
+	m = send(m,
+		evMsg{Step: onboard.StepConnect, Level: onboard.Done, Detail: "gpu01 (amd64)"},
+		evMsg{Step: onboard.StepCheck, Level: onboard.Warn, Detail: "fabric: nvidia_peermem is not loaded", Fix: "modprobe nvidia_peermem"},
+		evMsg{Step: onboard.StepCheck, Level: onboard.Running, Detail: "running llmgw doctor"},
+	)
+	v := m.View()
+	for _, want := range []string{"✓", "gpu01 (amd64)", "nvidia_peermem is not loaded", "→ modprobe nvidia_peermem", "running llmgw doctor", "engines"} {
+		if !strings.Contains(v, want) {
+			t.Errorf("screen lacks %q:\n%s", want, v)
+		}
+	}
+}
+
+func TestTheFabricQuestionAnswersWithWhatWasTyped(t *testing.T) {
+	var m tea.Model = newNodeModel("adding gpu01", func() {})
+	reply := make(chan []onboard.Fabric, 1)
+	m = send(m, askFabricsMsg{suggested: []onboard.Fabric{
+		{Name: "fabric1", HCAs: []string{"mlx5_0", "mlx5_1"}},
+		{Name: "fabric2", HCAs: []string{"mlx5_4"}},
+	}, reply: reply})
+	if v := m.View(); !strings.Contains(v, "Which fabric") || !strings.Contains(v, "mlx5_0, mlx5_1") {
+		t.Fatalf("no question on screen:\n%s", m.View())
+	}
+	// Rename the first to ib-a, empty the second to leave it out.
+	for range len("fabric1") {
+		m = send(m, tea.KeyMsg{Type: tea.KeyBackspace})
+	}
+	m = send(m, tea.KeyMsg{Type: tea.KeyRunes, Runes: []rune("ib-a")}, tea.KeyMsg{Type: tea.KeyTab})
+	for range len("fabric2") {
+		m = send(m, tea.KeyMsg{Type: tea.KeyBackspace})
+	}
+	m = send(m, tea.KeyMsg{Type: tea.KeyEnter})
+	got := <-reply
+	if len(got) != 1 || got[0].Name != "ib-a" || strings.Join(got[0].HCAs, ",") != "mlx5_0,mlx5_1" {
+		t.Errorf("got %+v", got)
+	}
+	if strings.Contains(m.View(), "Which fabric") {
+		t.Error("the question stayed on screen after it was answered")
+	}
+}
+
+func TestContinuingPastAProblemIsNoUnlessSaidYes(t *testing.T) {
+	var m tea.Model = newNodeModel("adding gpu01", func() {})
+	reply := make(chan bool, 1)
+	m = send(m, askContinueMsg{fails: 2, reply: reply}, tea.KeyMsg{Type: tea.KeyEnter})
+	if <-reply {
+		t.Error("enter went on past the problems")
+	}
+	m = send(m, askContinueMsg{fails: 2, reply: reply}, tea.KeyMsg{Type: tea.KeyRunes, Runes: []rune("y")})
+	if !<-reply {
+		t.Error("y did not go on")
+	}
+	_ = m
+}
+
+func TestANodeCountsAsJoinedOnlyWhenHeardSinceItsAgentStarted(t *testing.T) {
+	start := time.Now().Add(-10 * time.Second)
+	cases := []struct {
+		state, ago string
+		want       bool
+	}{
+		{"ready", "3s", true},
+		{"cordoned", "3s", true}, // added again while cordoned
+		{"ready", "30s", false},  // the agent before the restart
+		{"unreachable", "3s", false},
+		{"ready", "", false},
+	}
+	for _, c := range cases {
+		if got := heardSince(adminapi.NodeView{State: c.state, LastSeenAgo: c.ago}, start); got != c.want {
+			t.Errorf("%s seen %q ago: got %v", c.state, c.ago, got)
+		}
+	}
+}
+
+func TestAFabricNamedForOneNodeIsNotAskedAgainForTheNext(t *testing.T) {
+	na := &nodeAdd{named: map[string]string{}}
+	asked := 0
+	ask := func(s []onboard.Fabric) []onboard.Fabric {
+		asked++
+		return []onboard.Fabric{{Name: "ib-compute", HCAs: s[0].HCAs, Key: s[0].Key}}
+	}
+	first := na.rememberFabrics(ask)([]onboard.Fabric{{Name: "fabric", HCAs: []string{"mlx5_0"}, Key: "sm:0xa"}})
+	second := na.rememberFabrics(ask)([]onboard.Fabric{{Name: "fabric", HCAs: []string{"mlx5_0", "mlx5_1"}, Key: "sm:0xa"}})
+	if asked != 1 {
+		t.Errorf("asked %d times", asked)
+	}
+	if len(second) != 1 || second[0].Name != "ib-compute" || len(second[0].HCAs) != 2 || first[0].Name != "ib-compute" {
+		t.Errorf("first %+v second %+v", first, second)
+	}
+	// A fabric not seen before is asked about; in a script nothing is asked.
+	if got := na.rememberFabrics(nil)([]onboard.Fabric{{Name: "fabric", Key: "sm:0xb"}}); len(got) != 0 {
+		t.Errorf("a script was given a fabric nobody named: %+v", got)
+	}
+}

internal/cli/tui_widgets.go

@@ -0,0 +1,98 @@
+package cli
+
+import (
+	"strings"
+	"time"
+
+	tea "github.com/charmbracelet/bubbletea"
+	"github.com/charmbracelet/lipgloss"
+)
+
+// Two small widgets, written here rather than taken from bubbles: its
+// packages ask the terminal for its colours as they load, so every llmgw
+// command — llmgw --version included — sent a query and, in a terminal that
+// never answers one, waited five seconds for it. Only the screens need to
+// know the colours, and lipgloss asks when they first draw.
+
+// spinTickMsg advances a spinner.
+type spinTickMsg struct{}
+
+var spinFrames = []string{"⠋", "⠙", "⠹", "⠸", "⠼", "⠴", "⠦", "⠧", "⠇", "⠏"}
+
+type spinnerModel struct {
+	frame int
+	Style lipgloss.Style
+}
+
+func newSpinner() spinnerModel { return spinnerModel{Style: lipgloss.NewStyle().Foreground(tuiAccent)} }
+
+// Tick starts the spinner turning; it is a tea.Cmd.
+func (s spinnerModel) Tick() tea.Msg { return spinTickMsg{} }
+
+func (s spinnerModel) Update(msg tea.Msg) (spinnerModel, tea.Cmd) {
+	if _, ok := msg.(spinTickMsg); !ok {
+		return s, nil
+	}
+	s.frame = (s.frame + 1) % len(spinFrames)
+	return s, tea.Tick(100*time.Millisecond, func(time.Time) tea.Msg { return spinTickMsg{} })
+}
+
+func (s spinnerModel) View() string { return s.Style.Render(spinFrames[s.frame]) }
+
+// textField is one line of text typed at its end: all a form here needs.
+type textField struct {
+	value       []rune
+	focused     bool
+	Prompt      string
+	Placeholder string
+	CharLimit   int
+	Width       int
+}
+
+func newTextField() textField { return textField{Prompt: "> "} }
+
+func (f *textField) SetValue(s string) { f.value = []rune(s) }
+func (f textField) Value() string      { return string(f.value) }
+func (f *textField) Focus()            { f.focused = true }
+func (f *textField) Blur()             { f.focused = false }
+
+func (f textField) Update(msg tea.Msg) (textField, tea.Cmd) {
+	k, ok := msg.(tea.KeyMsg)
+	if !ok || !f.focused {
+		return f, nil
+	}
+	switch k.Type {
+	case tea.KeyRunes, tea.KeySpace:
+		r := k.Runes
+		if k.Type == tea.KeySpace {
+			r = []rune{' '}
+		}
+		for _, c := range r {
+			if f.CharLimit == 0 || len(f.value) < f.CharLimit {
+				f.value = append(f.value, c)
+			}
+		}
+	case tea.KeyBackspace:
+		if len(f.value) > 0 {
+			f.value = f.value[:len(f.value)-1]
+		}
+	case tea.KeyCtrlU:
+		f.value = nil
+	}
+	return f, nil
+}
+
+func (f textField) View() string {
+	text := string(f.value)
+	shown := text
+	if text == "" && f.Placeholder != "" {
+		shown = sDim.Render(f.Placeholder)
+	}
+	if f.focused {
+		shown += "█"
+	}
+	if pad := f.Width - len([]rune(text)); pad > 0 && text != "" {
+		shown += strings.Repeat(" ", pad)
+	}
+	return f.Prompt + shown
+}

internal/cli/users.go

@@ -1,7 +1,9 @@
 package cli
 
 import (
+	"errors"
 	"fmt"
+	"strings"
 	"text/tabwriter"
 	"time"
 
@@ -33,7 +35,7 @@ func newUsersCmd() *cobra.Command {
 		newUserStateCmd(open, "reinstate", store.UserActive,
 			"Let a suspended account work again"),
 		newUserCreateCmd(open), newUserResetPasswordCmd(open),
-		newUserMoveCmd(open), newUserDeleteCmd(open),
+		newUserJoinCmd(open), newUserLeaveCmd(open), newUserDeleteCmd(open),
 		newUserAdminCmd(open, "promote", true, "Let an account see the cluster dashboard at /admin"),
 		newUserAdminCmd(open, "demote", false, "Take away access to the cluster dashboard"))
 	return cmd
@@ -67,16 +69,28 @@ func newUserListCmd(open userOpener) *cobra.Command {
 				return nil
 			}
 
+			all, err := s.AllMemberships(cmd.Context())
+			if err != nil {
+				return err
+			}
+			where := map[string][]string{}
+			for _, m := range all {
+				where[m.UserID] = append(where[m.UserID], m.Path)
+			}
 			w := tabwriter.NewWriter(cmd.OutOrStdout(), 0, 0, 2, ' ', 0)
-			fmt.Fprintln(w, "ID\tEMAIL\tNAME\tSTATE\tADMIN\tPROJECT\tSIGNED UP\tLAST LOGIN")
+			fmt.Fprintln(w, "ID\tEMAIL\tSTATE\tADMIN\tMEMBER OF\tSIGNED UP\tLAST LOGIN")
 			now := time.Now()
 			for _, u := range users {
 				admin := "-"
 				if u.IsAdmin {
 					admin = "yes"
 				}
-				fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n",
-					u.ID, u.Email, u.Name, u.State, admin, dash(u.ProjectID),
+				member := "on their own"
+				if ms := where[u.ID]; len(ms) > 0 {
+					member = strings.Join(ms, "; ")
+				}
+				fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\t%s\n",
+					u.ID, u.Email, u.State, admin, member,
 					ago(u.CreatedAt, now), ago(u.LastLoginAt, now))
 			}
 			return w.Flush()
@@ -91,9 +105,9 @@ func newUserApproveCmd(open userOpener) *cobra.Command {
 	cmd := &cobra.Command{
 		Use:   "approve <user>",
 		Short: "Let an account sign in and issue its own keys",
-		Long: "The project is required: it decides which limits this person's keys\n" +
-			"inherit. An account approved into no project would pass every check and\n" +
-			"then fail at the first key it tried to make.",
+		Long: "A project is optional. Approved into nothing, a person uses the gateway on\n" +
+			"their own — personal keys and chat within their personal limits — and can\n" +
+			"be added to organizations, teams and projects with 'llmgw users join'.",
 		Args: cobra.ExactArgs(1),
 		RunE: func(cmd *cobra.Command, args []string) error {
 			s, err := open(cmd)
@@ -106,20 +120,26 @@ func newUserApproveCmd(open userOpener) *cobra.Command {
 			if err != nil {
 				return err
 			}
-			p, err := resolveProject(cmd.Context(), s, project)
-			if err != nil {
-				return err
+			var p store.ProjectPath
+			if project != "" {
+				if p, err = resolveProject(cmd.Context(), s, project); err != nil {
+					return err
+				}
 			}
 			if err := s.ApproveUser(cmd.Context(), u.ID, p.ID, by, time.Now()); err != nil {
 				return fmt.Errorf("approving %s: %w", u.Email, err)
 			}
-			fmt.Fprintf(cmd.OutOrStdout(), "%s approved into %s\n", u.Email, p.Path)
+			if p.ID == "" {
+				fmt.Fprintf(cmd.OutOrStdout(), "%s approved, on their own for now\n", u.Email)
+			} else {
+				fmt.Fprintf(cmd.OutOrStdout(), "%s approved into %s\n", u.Email, p.Path)
+			}
 			cmd.PrintErrln("They can sign in and create their own keys from /profile.")
 			return nil
 		},
 	}
 	cmd.Flags().StringVar(&project, "project", "",
-		"project the person's keys belong to: id, path (Acme/platform/gateway) or unique name")
+		"a project to put them in (optional): id, path (Acme/platform/gateway) or unique name")
 	cmd.Flags().StringVar(&by, "by", "", "who approved, for the record")
 	return cmd
 }
@@ -254,9 +274,11 @@ func newUserCreateCmd(open userOpener) *cobra.Command {
 			}
 			defer s.Close()
 
-			p, err := resolveProject(cmd.Context(), s, project)
-			if err != nil {
-				return err
+			var p store.ProjectPath
+			if project != "" {
+				if p, err = resolveProject(cmd.Context(), s, project); err != nil {
+					return err
+				}
 			}
 			hash, err := account.HashPassword(password)
 			if err != nil {
@@ -274,27 +296,64 @@ func newUserCreateCmd(open userOpener) *cobra.Command {
 				return err
 			}
 			fmt.Fprintf(cmd.OutOrStdout(), "%s\n", id)
-			cmd.PrintErrf("%s created and approved into %s\n", args[0], p.Path)
+			if p.ID == "" {
+				cmd.PrintErrf("%s created and approved, on their own for now\n", args[0])
+			} else {
+				cmd.PrintErrf("%s created and approved into %s\n", args[0], p.Path)
+			}
 			return nil
 		},
 	}
 	cmd.Flags().StringVar(&project, "project", "",
-		"project the person's keys belong to: id, path or unique name (optional when there is one)")
+		"a project to put them in (optional): id, path or unique name")
 	cmd.Flags().StringVar(&password, "password", "", "initial password")
 	return cmd
 }
 
-func newUserMoveCmd(open userOpener) *cobra.Command {
-	var project string
+// scopeFlags names one org, team or project by whichever flag is given.
+type scopeFlags struct{ org, team, project string }
+
+func (f *scopeFlags) bind(cmd *cobra.Command) {
+	cmd.Flags().StringVar(&f.org, "org", "", "an organization: id or name")
+	cmd.Flags().StringVar(&f.team, "team", "", "a team: id, path (Acme/ml) or unique name")
+	cmd.Flags().StringVar(&f.project, "project", "", "a project: id, path (Acme/ml/training) or unique name")
+}
+
+func (f *scopeFlags) resolve(cmd *cobra.Command, s *store.DB) (store.Scope, string, string, error) {
+	n := 0
+	for _, v := range []string{f.org, f.team, f.project} {
+		if v != "" {
+			n++
+		}
+	}
+	if n != 1 {
+		return "", "", "", fmt.Errorf("give exactly one of --org, --team or --project")
+	}
+	ctx := cmd.Context()
+	switch {
+	case f.org != "":
+		o, err := resolveOrg(ctx, s, f.org)
+		return store.ScopeOrg, o.ID, o.Name, err
+	case f.team != "":
+		t, err := resolveTeam(ctx, s, f.team)
+		return store.ScopeTeam, t.ID, t.OrgName + " / " + t.Name, err
+	default:
+		p, err := resolveProject(ctx, s, f.project)
+		return store.ScopeProject, p.ID, p.Path, err
+	}
+}
+
+func newUserJoinCmd(open userOpener) *cobra.Command {
+	var where scopeFlags
+	var role, by string
 	cmd := &cobra.Command{
-		Use:   "move <user> --project <project>",
-		Short: "Put an approved account in another project, with the keys it made",
-		Long: "The person's own keys go with them, so they stop spending the old project's\n" +
-			"budget. Keys an operator issued in the old project stay there. Usage already\n" +
-			"recorded is not rewritten. A running server picks the change up when its key\n" +
-			"cache expires (auth.cache_ttl).",
-		Example: "  llmgw users move ayse@example.com --project Acme/ml/research",
-		Args:    cobra.ExactArgs(1),
+		Use:   "join <user> --org|--team|--project <which>",
+		Short: "Make a person a member of an organization, team or project",
+		Long: "A person can be a member of any number of them. A member of a team or\n" +
+			"organization may use every project in it.",
+		Example: "  llmgw users join ayse@example.com --team Acme/ml\n" +
+			"  llmgw users join ayse@example.com --project Scania/research/vision",
+		Args: cobra.ExactArgs(1),
 		RunE: func(cmd *cobra.Command, args []string) error {
 			s, err := open(cmd)
 			if err != nil {
@@ -305,21 +364,59 @@ func newUserMoveCmd(open userOpener) *cobra.Command {
 			if err != nil {
 				return err
 			}
-			p, err := resolveProject(cmd.Context(), s, project)
+			scope, id, label, err := where.resolve(cmd, s)
 			if err != nil {
 				return err
 			}
-			moved, err := s.MoveUser(cmd.Context(), u.ID, p.ID)
-			if err != nil {
+			if err := s.AddMembership(cmd.Context(), u.ID, scope, id, role, by, time.Now()); err != nil {
 				return err
 			}
-			fmt.Fprintf(cmd.OutOrStdout(), "%s is now in %s (%s moved with them)\n",
-				u.Email, p.Path, countOf(moved, "key"))
+			fmt.Fprintf(cmd.OutOrStdout(), "%s is a %s of %s\n", u.Email, role, label)
 			return nil
 		},
 	}
-	cmd.Flags().StringVar(&project, "project", "", "where they go: id, path or unique name")
-	_ = cmd.MarkFlagRequired("project")
+	where.bind(cmd)
+	cmd.Flags().StringVar(&role, "role", store.RoleMember, "member, or manager: with --org, may see the organization's page")
+	cmd.Flags().StringVar(&by, "by", "", "who added them, for the record")
+	return cmd
+}
+
+func newUserLeaveCmd(open userOpener) *cobra.Command {
+	var where scopeFlags
+	cmd := &cobra.Command{
+		Use:   "leave <user> --org|--team|--project <which>",
+		Short: "Take a person out of an organization, team or project",
+		Long: "Keys they made in a project they can no longer use are revoked; a project\n" +
+			"they still reach through another membership keeps them. Their personal\n" +
+			"keys are not touched.",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			s, err := open(cmd)
+			if err != nil {
+				return err
+			}
+			defer s.Close()
+			u, err := resolveUser(cmd.Context(), s, args[0])
+			if err != nil {
+				return err
+			}
+			scope, id, label, err := where.resolve(cmd, s)
+			if err != nil {
+				return err
+			}
+			revoked, err := s.RemoveMembership(cmd.Context(), u.ID, scope, id, time.Now())
+			if errors.Is(err, store.ErrNotFound) {
+				return fmt.Errorf("%s is not a member of %s itself", u.Email, label)
+			}
+			if err != nil {
+				return err
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "%s is no longer in %s (%s revoked)\n",
+				u.Email, label, countOf(revoked, "key"))
+			return nil
+		},
+	}
+	where.bind(cmd)
 	return cmd
 }
 

internal/config/config.go

@@ -204,6 +204,10 @@ type Auth struct {
 	// means accounts are made with `llmgw users create` instead.
 	SignupOpen bool `koanf:"signup_open"`
 
+	// Personal is what a person may do on their own, outside any project: make
+	// personal keys and use the chat page personally, within these limits.
+	Personal Personal `koanf:"personal"`
+
 	// Pricing is what each model costs, for budgets. A model absent from the
 	// table costs whatever DefaultPrice says, which is nothing unless set.
 	Pricing      map[string]Price `koanf:"pricing"`
@@ -218,6 +222,22 @@ type Auth struct {
 	BufferSize int `koanf:"buffer_size"`
 }
 
+// Personal configures use that belongs to a person rather than a project.
+//
+// On by default: a person need not belong to anything to use the gateway. The
+// limits are each person's own pool across all their personal keys and their
+// personal chat, and apply only where nobody wrote a limit for that person.
+type Personal struct {
+	// Disabled refuses personal keys and personal chat; people then use the
+	// gateway only through projects they are members of.
+	Disabled bool `koanf:"disabled"`
+	RPM      int  `koanf:"rpm"`
+	TPM      int  `koanf:"tpm"`
+	// Budget is in whole units per Period ("day" or "month").
+	Budget float64 `koanf:"budget"`
+	Period string  `koanf:"period"`
+}
+
 // Price is a model's cost in micro-units per thousand tokens.
 //
 // What a unit means is the operator's choice — self-hosted models have no
@@ -384,8 +404,11 @@ type Scoring struct {
 
 // Endpoint is one engine replica. Static in phase 1.
 type Endpoint struct {
-	ID     string   `koanf:"id"`
-	URL    string   `koanf:"url"`
+	ID  string `koanf:"id"`
+	URL string `koanf:"url"`
+	// APIKey is sent to the engine as a bearer token, for an engine started
+	// with one.
+	APIKey string   `koanf:"api_key"`
 	Models []string `koanf:"models"`
 	// Engine names the inference engine, which decides what its metrics are
 	// called. Empty means vLLM's names.
@@ -559,9 +582,14 @@ func (s Server) ControlPlaneMode() bool {
 }
 
 // DefaultEngines are the engine versions this release was checked against:
-// `llmgw ctl engine-check` accepted every flag llmgw builds for them.
+// every flag llmgw builds for them is one they take.
+//
+// vLLM is installed on every node with an NVIDIA GPU it can run on and
+// skipped elsewhere. SGLang is not here: it is opt-in, by naming a version
+// under engines.
 var DefaultEngines = map[string]string{
 	"llamacpp": "b10991",
+	"vllm":     "0.30.0",
 }
 
 // WantedEngines is DefaultEngines with the configuration laid over it.

internal/config/example_test.go

@@ -36,9 +36,8 @@ func TestTheDocumentedAgentExampleLoads(t *testing.T) {
 	if len(f.HCAs) == 0 {
 		t.Error("the fabric's adapters did not bind")
 	}
-	if f.RateGbps == 0 {
-		t.Error("the fabric's rate did not bind")
-	}
+	// The rate is not documented: the agent measures it, as the slowest of
+	// the fabric's active ports, and a figure written here would be ignored.
 	// node_id is left to the hostname, so the example need not say it; where
 	// the control plane is, nothing can find out.
 	if cfg.Server == "" {

internal/control/adminapi/api.go

@@ -16,11 +16,15 @@ import (
 	"strings"
 	"time"
 
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/bench"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/fabrics"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/nvlink"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/topology"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
 )
 
@@ -48,6 +52,20 @@ func (a *API) Register(mux *http.ServeMux) {
 	mux.HandleFunc("GET /api/v1/capacity", a.listCapacity)
 	mux.HandleFunc("GET /api/v1/usage", a.usageByScope)
 	mux.HandleFunc("GET /api/v1/nodes", a.listNodes)
+	mux.HandleFunc("GET /api/v1/fabrics", func(w http.ResponseWriter, _ *http.Request) {
+		writeJSON(w, http.StatusOK, a.Fabrics())
+	})
+	mux.HandleFunc("GET /api/v1/nvlink", func(w http.ResponseWriter, _ *http.Request) {
+		writeJSON(w, http.StatusOK, a.NVLink())
+	})
+	mux.HandleFunc("GET /api/v1/fabrics/{id}/topology", func(w http.ResponseWriter, r *http.Request) {
+		t, ok := a.Topology(r.PathValue("id"))
+		if !ok {
+			http.NotFound(w, r)
+			return
+		}
+		writeJSON(w, http.StatusOK, t)
+	})
 	mux.HandleFunc("GET /api/v1/models", a.listModels)
 	mux.HandleFunc("GET /api/v1/replicas", a.listReplicas)
 	// Listing is a read and deleting is a separate verb, so a mistyped
@@ -56,7 +74,11 @@ func (a *API) Register(mux *http.ServeMux) {
 	mux.HandleFunc("POST /api/v1/weights/collect", a.collectWeights)
 	mux.HandleFunc("GET /api/v1/explain-placement", a.explainPlacement)
 	mux.HandleFunc("POST /api/v1/nodes/{id}/state", a.setNodeState)
+	mux.HandleFunc("DELETE /api/v1/nodes/{id}", a.removeNode)
+	mux.HandleFunc("POST /api/v1/fabric-test", a.fabricTest)
 	mux.HandleFunc("GET /api/v1/nodes/{id}/logs", a.nodeLogs)
+	mux.HandleFunc("POST /api/v1/restart", a.restart)
+	mux.HandleFunc("POST /api/v1/models/{name}/retry", a.retryModel)
 }
 
 // --- responses ---
@@ -114,7 +136,91 @@ type NodeView struct {
 
 	LastSeenAgo  string `json:"last_seen_ago"`
 	AgentVersion string `json:"agent_version,omitempty"`
-	Replicas     int    `json:"replicas"`
+	// Features are the requests the agent answers beyond the usual:
+	// "fabric-test". Absent is an agent too old to say.
+	Features []string `json:"features,omitempty"`
+	Replicas int      `json:"replicas"`
+
+	// Load is how busy the machine is, absent when its agent has not
+	// measured it lately — which is unknown, not idle.
+	Load *LoadView `json:"load,omitempty"`
+	// IBPorts are the node's InfiniBand and RoCE ports at the last heartbeat.
+	IBPorts []IBPortView `json:"ib_ports,omitempty"`
+	// GPUs are the cards, with how they are wired: NVLink peers, the
+	// adapter nearest each, and the NVLink domain each has joined.
+	GPUs []GPUView `json:"gpus,omitempty"`
+	// NVLinkDomain is the multi-node domain every card has joined, if any.
+	NVLinkDomain string `json:"nvlink_domain,omitempty"`
+	IMEX         string `json:"imex,omitempty"`
+	IMEXChannels int    `json:"imex_channels,omitempty"`
+}
+
+// GPUView is one card as a node's page shows it.
+type GPUView struct {
+	Index       int    `json:"index"`
+	Name        string `json:"name"`
+	NUMA        int    `json:"numa"`
+	NVLinkPeers []int  `json:"nvlink_peers,omitempty"`
+	// NearestNIC is the adapter closest to the card, and how close.
+	NearestNIC  string `json:"nearest_nic,omitempty"`
+	NICPath     string `json:"nic_path,omitempty"`
+	FabricState string `json:"fabric_state,omitempty"`
+	Domain      string `json:"domain,omitempty"`
+	Bandwidth   string `json:"bandwidth,omitempty"`
+}
+
+// IBPortView is one port as a page shows it.
+type IBPortView struct {
+	HCA       string `json:"hca"`
+	Port      int    `json:"port"`
+	State     string `json:"state"`
+	PhysState string `json:"phys_state,omitempty"`
+	RateGbps  int    `json:"rate_gbps"`
+	LinkLayer string `json:"link_layer,omitempty"`
+	Active    bool   `json:"active"`
+	// Fabric is the declared fabric this port's adapter carries, if any.
+	Fabric string `json:"fabric,omitempty"`
+	// Physical is the fabric the port is physically on: its subnet manager,
+	// or its RoCE network. SMError says why that is not known.
+	Physical string `json:"physical,omitempty"`
+	SMError  string `json:"sm_error,omitempty"`
+	// Errors are the error counters that are not zero, by name.
+	Errors  map[string]uint64 `json:"errors,omitempty"`
+	TxBytes uint64            `json:"tx_bytes,omitempty"`
+	RxBytes uint64            `json:"rx_bytes,omitempty"`
+}
+
+// LoadView is a node's CPU and memory in use, at its last heartbeat.
+type LoadView struct {
+	CPUPercent float64 `json:"cpu_percent"`
+	CPUs       int     `json:"cpus,omitempty"`
+	Load1      float64 `json:"load1"`
+	Load5      float64 `json:"load5"`
+	Load15     float64 `json:"load15"`
+	MemUsed    int64   `json:"memory_used_bytes"`
+	MemTotal   int64   `json:"memory_total_bytes"`
+	// MemPercent is used over used plus available.
+	MemPercent float64 `json:"memory_percent"`
+	AgeSec     int64   `json:"age_seconds"`
+}
+
+// loadFresh is how old a measurement may be and still be shown as current.
+const loadFresh = time.Minute
+
+func loadView(l *inventory.Load, now time.Time) *LoadView {
+	if l == nil || now.Sub(l.At) > loadFresh {
+		return nil
+	}
+	v := &LoadView{
+		CPUPercent: l.CPUPercent, CPUs: l.CPUCount,
+		Load1: l.Load1, Load5: l.Load5, Load15: l.Load15,
+		MemUsed: l.MemUsedBytes, MemTotal: l.MemUsedBytes + l.MemAvailBytes,
+		AgeSec: int64(now.Sub(l.At).Seconds()),
+	}
+	if v.MemTotal > 0 {
+		v.MemPercent = float64(v.MemUsed) * 100 / float64(v.MemTotal)
+	}
+	return v
 }
 
 // ModelView is a model as `ctl models` shows it.
@@ -128,13 +234,32 @@ type ModelView struct {
 	Max         int      `json:"max_replicas"`
 	Ready       int      `json:"ready_replicas"`
 	Total       int      `json:"total_replicas"`
+	// Desired is how many it should have now: Min, or for an on-demand
+	// model one while it is being asked for and none otherwise.
+	Desired int `json:"desired_replicas"`
+	// Failures counts replicas that failed to start in a row.
+	Failures int `json:"failures,omitempty"`
+	// Parked is set once Failures reached the limit; the model is not retried
+	// until its definition changes or `ctl retry` is run. Reason is the last
+	// error.
+	Parked bool   `json:"parked,omitempty"`
+	Reason string `json:"reason,omitempty"`
+	// Unplaceable is set when it is short and the last attempt found no node.
+	Unplaceable bool     `json:"unplaceable,omitempty"`
 	Fallbacks   []string `json:"fallbacks,omitempty"`
 	Profiles    []string `json:"profiles"`
 }
 
 // ReplicaView is a replica as `ctl replicas` shows it.
 type ReplicaView struct {
-	ID      string `json:"id"`
+	ID string `json:"id"`
+	// EngineKey is the key this replica's engine demands, for `ctl bench`
+	// and anyone else holding the admin listener who needs to reach an engine
+	// directly. The client-facing pages never render it.
+	EngineKey string `json:"engine_key,omitempty"`
+	// Restart is "queued" or "replacing" while an operator's restart of this
+	// replica is under way.
+	Restart string `json:"restart,omitempty"`
 	Model   string `json:"model"`
 	Profile string `json:"profile"`
 	// Stale is true when the profile has been edited since this replica was
@@ -170,6 +295,10 @@ type ShardView struct {
 	// waiting on one. It is what tells a replica pulling a hundred gigabytes
 	// apart from one that is wedged — which look identical without it.
 	Fetch *FetchView `json:"fetch,omitempty"`
+	// CPUCores and MemoryBytes are the engine's use of its node, from the
+	// node's last heartbeat; zero when not measured.
+	CPUCores    float64 `json:"cpu_cores,omitempty"`
+	MemoryBytes int64   `json:"memory_bytes,omitempty"`
 }
 
 // FetchView is a download in flight.
@@ -230,7 +359,7 @@ func (a *API) Nodes() []NodeView {
 			State: n.State.String(), Labels: n.Labels,
 			GPUTotal: len(n.GPUs), GPUFree: len(inv.FreeGPUs(n.ID)),
 			MemoryGB: n.MemoryTotalMB / 1024,
-			Fabrics:  n.FabricNames(), AgentVersion: n.AgentVersion,
+			Fabrics:  n.FabricNames(), AgentVersion: n.AgentVersion, Features: n.Features,
 			Replicas: replicasOn[n.ID],
 		}
 		if free, total := n.LocalDisk(); total > 0 {
@@ -266,6 +395,47 @@ func (a *API) Nodes() []NodeView {
 		if !n.LastSeen.IsZero() {
 			v.LastSeenAgo = time.Since(n.LastSeen).Round(time.Second).String()
 		}
+		v.Load = loadView(n.Load, time.Now())
+		v.NVLinkDomain, v.IMEX, v.IMEXChannels = n.NVLinkDomain(), n.IMEX.State, n.IMEX.Channels
+		for _, g := range n.GPUs {
+			gv := GPUView{Index: g.Index, Name: g.Name, NUMA: g.NUMANode, NVLinkPeers: g.NVLinkPeers}
+			for hca, rel := range g.NICPaths {
+				if gv.NearestNIC == "" || discovery.PathRank(rel) < discovery.PathRank(gv.NICPath) ||
+					(discovery.PathRank(rel) == discovery.PathRank(gv.NICPath) && hca < gv.NearestNIC) {
+					gv.NearestNIC, gv.NICPath = hca, rel
+				}
+			}
+			if f := g.Fabric; f != nil {
+				gv.FabricState, gv.Domain, gv.Bandwidth = f.State, f.Domain(), f.Bandwidth
+			}
+			v.GPUs = append(v.GPUs, gv)
+		}
+		fabricOf := map[string]string{}
+		for _, f := range n.Fabrics {
+			for _, h := range f.HCAs {
+				fabricOf[h] = f.Fabric
+			}
+		}
+		for _, p := range n.IBPorts {
+			pv := IBPortView{HCA: p.HCA, Port: p.Port, State: p.State, PhysState: p.PhysState,
+				RateGbps: p.RateGbps, LinkLayer: p.LinkLayer, Active: p.Active(), Fabric: fabricOf[p.HCA],
+				TxBytes: p.Counters["port_xmit_data"], RxBytes: p.Counters["port_rcv_data"], SMError: p.SMError}
+			switch {
+			case p.RoCE() && p.Network != "":
+				pv.Physical = "RoCE " + p.Network
+			case p.SMGUID != "":
+				pv.Physical = fmt.Sprintf("SM %s (LID %d)", p.SMGUID, p.SMLID)
+			}
+			for name, c := range p.Counters {
+				if c > 0 && name != "port_xmit_data" && name != "port_rcv_data" {
+					if pv.Errors == nil {
+						pv.Errors = map[string]uint64{}
+					}
+					pv.Errors[name] = c
+				}
+			}
+			v.IBPorts = append(v.IBPorts, pv)
+		}
 		out = append(out, v)
 	}
 	return out
@@ -285,13 +455,20 @@ func (a *API) Models() []ModelView {
 		}
 	}
 
+	health := map[string]reconciler.ModelHealth{}
+	for _, h := range a.ctl.Reconciler().ModelHealth() {
+		health[h.Model] = h
+	}
 	out := make([]ModelView, 0)
 	for _, m := range a.ctl.Catalog().Models() {
+		h := health[m.Name]
 		v := ModelView{
 			Name: m.Name, Description: m.Description, Aliases: m.Aliases, Source: m.Source,
 			Min: m.Scaling.Min, Max: m.Scaling.Max,
 			Ready: ready[m.Name], Total: total[m.Name],
 			Fallbacks: m.Fallbacks,
+			Desired:   h.Desired, Failures: h.Failures,
+			Parked: h.Parked, Reason: h.Reason, Unplaceable: h.Unplaceable,
 		}
 		for _, p := range m.Profiles {
 			v.Profiles = append(v.Profiles, p.Name)
@@ -308,6 +485,12 @@ func (a *API) listReplicas(w http.ResponseWriter, r *http.Request) {
 // Replicas builds the replica view, optionally filtered to one model.
 func (a *API) Replicas(model string) []ReplicaView {
 	out := make([]ReplicaView, 0)
+	loads := map[string]*inventory.Load{}
+	for _, n := range a.ctl.Inventory().Nodes() {
+		if n.Load != nil && time.Since(n.Load.At) <= loadFresh {
+			loads[n.ID] = n.Load
+		}
+	}
 	for _, rep := range a.ctl.Reconciler().Replicas() {
 		if model != "" && rep.Model != model {
 			continue
@@ -316,9 +499,11 @@ func (a *API) Replicas(model string) []ReplicaView {
 			ID: rep.ID, Model: rep.Model, Profile: rep.Profile,
 			State: rep.State.String(), Pool: rep.Pool, Fabric: rep.Fabric,
 			Endpoint: rep.Endpoint, Err: rep.Err,
-			Stale:   a.profileChanged(rep.Model, rep.Profile, rep.ProfileFP),
-			AgeSec:  int64(time.Since(rep.CreatedAt).Seconds()),
-			IdleSec: -1,
+			EngineKey: a.ctl.EngineKey(rep.ID),
+			Restart:   a.ctl.Reconciler().RestartState(rep.ID),
+			Stale:     a.profileChanged(rep.Model, rep.Profile, rep.ProfileFP),
+			AgeSec:    int64(time.Since(rep.CreatedAt).Seconds()),
+			IdleSec:   -1,
 		}
 		if rep.LastUsedAt.IsZero() {
 			v.IdleSec = -1
@@ -338,6 +523,11 @@ func (a *API) Replicas(model string) []ReplicaView {
 					sv.Fetch.Percent = &p
 				}
 			}
+			if l := loads[s.NodeID]; l != nil {
+				if use, ok := l.Shards[s.ID]; ok {
+					sv.CPUCores, sv.MemoryBytes = use.CPUCores, use.RSSBytes
+				}
+			}
 			v.Shards = append(v.Shards, sv)
 		}
 		out = append(out, v)
@@ -657,6 +847,35 @@ func (a *API) Explain(model string) ExplainView {
 // One endpoint rather than three verbs, because they are the same operation
 // with different targets and the difference matters: cordon stops new
 // placement, drain also moves existing work off.
+// removeNode forgets a node that has gone for good: a machine retired or
+// rebuilt under another name, which otherwise stays listed as lost until the
+// control plane restarts.
+//
+// Only one that is not there: lost, or unreachable with nothing placed on
+// it. A connected node removed here would register again on its next
+// heartbeat, and one with replicas would take them out of the reconciler's
+// sight while they may still be running.
+func (a *API) removeNode(w http.ResponseWriter, r *http.Request) {
+	id := r.PathValue("id")
+	inv := a.ctl.Inventory()
+	n, ok := inv.Get(id)
+	if !ok {
+		writeError(w, http.StatusNotFound, fmt.Sprintf("no node %q", id))
+		return
+	}
+	switch {
+	case n.State == inventory.Lost:
+	case n.State == inventory.Unreachable && len(inv.Allocations(id)) == 0:
+	default:
+		writeError(w, http.StatusConflict, fmt.Sprintf(
+			"%s is %s: stop its agent (systemctl stop llmgw-agent) and remove it once it is lost, a minute later", id, n.State))
+		return
+	}
+	inv.Remove(id)
+	a.ctl.Reconciler().Notify()
+	writeJSON(w, http.StatusOK, map[string]string{"id": id, "removed": "true"})
+}
+
 func (a *API) setNodeState(w http.ResponseWriter, r *http.Request) {
 	id := r.PathValue("id")
 
@@ -726,3 +945,173 @@ func (a *API) profileChanged(model, profile, was string) bool {
 	}
 	return false
 }
+
+// Fabrics is the physical fabrics the adapters are on, against the names
+// the nodes gave them.
+func (a *API) Fabrics() fabrics.Report {
+	inv := a.ctl.Inventory()
+	return fabrics.Analyze(inv.Nodes(), inv.SMChanges(), time.Now())
+}
+
+// TopologyView is one fabric's map, with the cluster's own adapters on it
+// live: their state, rate and traffic from the last heartbeat.
+type TopologyView struct {
+	ID        string          `json:"id"`
+	Key       string          `json:"key"`
+	Label     string          `json:"label"`
+	Declared  []string        `json:"declared"`
+	ScannedAt time.Time       `json:"scanned_at"`
+	ScannedBy string          `json:"scanned_by,omitempty"`
+	TookMs    int64           `json:"took_ms"`
+	Err       string          `json:"error,omitempty"`
+	Map       *topology.View  `json:"map,omitempty"`
+	Live      map[string]Live `json:"live"`
+	// Problems are the fabric's own, from the analysis of its identity and
+	// subnet managers.
+	Problems []fabrics.Problem `json:"problems,omitempty"`
+	SM       *fabrics.SM       `json:"sm,omitempty"`
+	SMs      []fabrics.SM      `json:"sms,omitempty"`
+}
+
+// Live is one of the cluster's adapters as its node last reported it.
+type Live struct {
+	Node   string            `json:"node"`
+	HCA    string            `json:"hca"`
+	State  string            `json:"state"`
+	Phys   string            `json:"phys_state"`
+	Gbps   int               `json:"gbps"`
+	TxBps  float64           `json:"tx_bps"`
+	RxBps  float64           `json:"rx_bps"`
+	Errors map[string]uint64 `json:"errors,omitempty"`
+}
+
+// FabricID is a physical fabric's key as an address can carry it.
+func FabricID(key string) string {
+	return strings.NewReplacer(":", "-", "/", "_").Replace(key)
+}
+
+// Topologies lists every physical fabric with what is known of its map.
+func (a *API) Topologies() []TopologyView {
+	rep := a.Fabrics()
+	scans := a.ctl.Inventory().Scans()
+	out := make([]TopologyView, 0, len(rep.Physical))
+	for _, ph := range rep.Physical {
+		t := TopologyView{ID: FabricID(ph.Key), Key: ph.Key, Label: ph.Label, Declared: ph.Declared}
+		if sc, ok := scans[ph.Key]; ok {
+			t.ScannedAt, t.ScannedBy, t.TookMs, t.Err = sc.At, sc.Node+" "+sc.Port, sc.Took.Milliseconds(), sc.Err
+		}
+		out = append(out, t)
+	}
+	return out
+}
+
+// Topology is one fabric's map, by the id Topologies gives it.
+func (a *API) Topology(id string) (TopologyView, bool) {
+	rep := a.Fabrics()
+	var ph *fabrics.Physical
+	for i := range rep.Physical {
+		if FabricID(rep.Physical[i].Key) == id {
+			ph = &rep.Physical[i]
+		}
+	}
+	if ph == nil {
+		return TopologyView{}, false
+	}
+	t := TopologyView{ID: id, Key: ph.Key, Label: ph.Label, Declared: ph.Declared, Live: map[string]Live{},
+		SM: ph.SM, SMs: ph.SMs}
+	for _, p := range rep.Problems {
+		for _, d := range ph.Declared {
+			if p.Fabric == d {
+				t.Problems = append(t.Problems, p)
+				break
+			}
+		}
+	}
+	hosts := map[string]topology.Host{}
+	for _, n := range a.ctl.Inventory().Nodes() {
+		for _, p := range n.IBPorts {
+			if p.NodeGUID == "" {
+				continue
+			}
+			hosts[p.NodeGUID] = topology.Host{Node: n.ID, HCA: p.HCA}
+			l := Live{Node: n.ID, HCA: p.HCA, State: p.State, Phys: p.PhysState, Gbps: p.RateGbps,
+				TxBps: p.TxBps, RxBps: p.RxBps}
+			for name, c := range p.Counters {
+				if c > 0 && name != "port_xmit_data" && name != "port_rcv_data" {
+					if l.Errors == nil {
+						l.Errors = map[string]uint64{}
+					}
+					l.Errors[name] = c
+				}
+			}
+			t.Live[p.NodeGUID] = l
+		}
+	}
+	sc, ok := a.ctl.Inventory().Scans()[ph.Key]
+	if !ok {
+		return t, true
+	}
+	t.ScannedAt, t.ScannedBy, t.TookMs, t.Err = sc.At, sc.Node+" "+sc.Port, sc.Took.Milliseconds(), sc.Err
+	if sc.Output != "" {
+		f, err := topology.Parse(sc.Output)
+		if err != nil {
+			t.Err = "the map could not be read: " + err.Error()
+			return t, true
+		}
+		v := topology.Build(f, hosts)
+		t.Map = &v
+	}
+	return t, true
+}
+
+// NVLink is the cluster's multi-node NVLink domains, and what keeps any of
+// their nodes out.
+func (a *API) NVLink() nvlink.Report {
+	return nvlink.Analyze(a.ctl.Inventory().Nodes())
+}
+
+// RestartRequest names what to restart: replicas by id, every replica of some
+// models, or — All — every ready replica in the cluster.
+type RestartRequest struct {
+	Replicas []string `json:"replicas,omitempty"`
+	Models   []string `json:"models,omitempty"`
+	All      bool     `json:"all,omitempty"`
+}
+
+// RestartResult lists the replicas queued.
+type RestartResult struct {
+	Queued []string `json:"queued"`
+}
+
+func (a *API) restart(w http.ResponseWriter, r *http.Request) {
+	var req RestartRequest
+	if err := json.NewDecoder(io.LimitReader(r.Body, 1<<16)).Decode(&req); err != nil {
+		writeError(w, http.StatusBadRequest, "the restart request is not valid JSON: "+err.Error())
+		return
+	}
+	ids := slices.Clone(req.Replicas)
+	for _, rep := range a.ctl.Reconciler().ReadyReplicas() {
+		if req.All || slices.Contains(req.Models, rep.Model) {
+			ids = append(ids, rep.ID)
+		}
+	}
+	if len(ids) == 0 {
+		writeError(w, http.StatusBadRequest, "nothing ready matches; name replicas, models, or all")
+		return
+	}
+	if err := a.ctl.Reconciler().Restart(ids...); err != nil {
+		writeError(w, http.StatusNotFound, err.Error())
+		return
+	}
+	writeJSON(w, http.StatusAccepted, RestartResult{Queued: ids})
+}
+
+func (a *API) retryModel(w http.ResponseWriter, r *http.Request) {
+	name := r.PathValue("name")
+	if _, ok := a.ctl.Catalog().Resolve(name); !ok {
+		writeError(w, http.StatusNotFound, "no model called "+name)
+		return
+	}
+	was := a.ctl.Reconciler().Retry(name)
+	writeJSON(w, http.StatusOK, map[string]bool{"was_parked": was})
+}

internal/control/adminapi/api_test.go

@@ -667,3 +667,24 @@ func TestAProfileThatIsGoneIsNotReportedAsStale(t *testing.T) {
 		t.Error("a missing model is not a changed profile")
 	}
 }
+
+func TestOnlyANodeThatIsGoneCanBeRemoved(t *testing.T) {
+	a, ctl := setup(t, testModels(), gpuNode("n1", 8))
+
+	// Connected, even cordoned: it would register again on its next
+	// heartbeat, so removing it would do nothing but confuse.
+	ctl.Inventory().SetState("n1", inventory.Cordoned)
+	if code := call(t, a, "DELETE", "/api/v1/nodes/n1", "", nil); code != 409 {
+		t.Errorf("a connected node gave %d, expected 409", code)
+	}
+	ctl.Inventory().SetState("n1", inventory.Lost)
+	if code := call(t, a, "DELETE", "/api/v1/nodes/n1", "", nil); code != 200 {
+		t.Fatalf("a lost node gave %d, expected 200", code)
+	}
+	if _, ok := ctl.Inventory().Get("n1"); ok {
+		t.Error("the node is still listed")
+	}
+	if code := call(t, a, "DELETE", "/api/v1/nodes/n1", "", nil); code != 404 {
+		t.Errorf("removing it again gave %d, expected 404", code)
+	}
+}

internal/control/adminapi/fabrictest.go

@@ -0,0 +1,71 @@
+package adminapi
+
+import (
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/fabrictest"
+)
+
+// FabricTestView is a bandwidth test between two nodes.
+type FabricTestView struct {
+	A      string     `json:"a"`
+	B      string     `json:"b"`
+	Fabric string     `json:"fabric"`
+	OK     bool       `json:"ok"`
+	Rails  []RailView `json:"rails"`
+	Notes  []string   `json:"notes,omitempty"`
+	Took   string     `json:"took"`
+}
+
+// RailView is one rail's result.
+type RailView struct {
+	A        string  `json:"a"`
+	B        string  `json:"b"`
+	RateGbps int     `json:"rate_gbps"`
+	Gbps     float64 `json:"gbps"`
+	Verdict  string  `json:"verdict"`
+	Detail   string  `json:"detail,omitempty"`
+}
+
+// fabricTest measures RDMA bandwidth between two nodes, rail by rail. It
+// sends traffic over the fabric for seconds a rail, so it is a POST and only
+// ever run when asked.
+func (a *API) fabricTest(w http.ResponseWriter, r *http.Request) {
+	var body struct {
+		A, B, Fabric string
+		Seconds      int
+	}
+	if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	}
+	ask := a.ctl.FabricTester()
+	if ask == nil {
+		writeError(w, http.StatusNotImplemented, "this control plane cannot reach its nodes' agents")
+		return
+	}
+	inv := a.ctl.Inventory()
+	na, okA := inv.Get(body.A)
+	nb, okB := inv.Get(body.B)
+	if !okA || !okB {
+		writeError(w, http.StatusNotFound, fmt.Sprintf("no node %q", map[bool]string{true: body.B, false: body.A}[okA]))
+		return
+	}
+	fabric, rails, notes, err := fabrictest.Plan(na, nb, body.Fabric)
+	if err != nil {
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	}
+	start := time.Now()
+	rep := fabrictest.Run(r.Context(), ask, na, nb, fabric, rails,
+		fabrictest.Options{Seconds: min(max(body.Seconds, 1), 30), Settle: 2 * time.Second})
+	v := FabricTestView{A: na.ID, B: nb.ID, Fabric: fabric, OK: rep.OK(), Notes: append(notes, rep.Notes...),
+		Took: time.Since(start).Round(time.Second).String()}
+	for _, x := range rep.Results {
+		v.Rails = append(v.Rails, RailView{x.A, x.B, x.RateGbps, x.Gbps, x.Verdict, x.Detail})
+	}
+	writeJSON(w, http.StatusOK, v)
+}

internal/control/adminapi/summary.go

@@ -25,6 +25,9 @@ type Summary struct {
 	// Unplaceable names models the scheduler currently cannot satisfy. This is
 	// the line that turns "something is wrong" into a place to look.
 	Unplaceable []string `json:"unplaceable,omitempty"`
+	// Parked are models that failed every time they started and are no
+	// longer retried, with the last error.
+	Parked []ParkedModel `json:"parked,omitempty"`
 	// IdleReplicas are serving replicas nothing has asked for in a while, which
 	// on a fleet with more models than GPUs is the first place to find room.
 	IdleReplicas []IdleReplica `json:"idle_replicas,omitempty"`
@@ -201,17 +204,30 @@ func (a *API) Summary() Summary {
 
 	for _, m := range a.Models() {
 		s.Models.Total++
-		if m.Ready >= m.Min {
+		if m.Parked {
+			s.Parked = append(s.Parked, ParkedModel{Model: m.Name, Failures: m.Failures, Reason: m.Reason})
+		}
+		// Against what it should have now, not its minimum: an on-demand
+		// model's minimum is 0, so one that was asked for and fitted nowhere
+		// used to count as satisfied.
+		if m.Ready >= m.Desired && !m.Parked {
 			s.Models.Satisfied++
 			continue
 		}
 		s.Models.Short++
 		// Short of replicas and unable to place more is the actionable case;
 		// short and still starting is just a cluster doing its job.
-		if m.Total <= m.Ready {
+		if m.Unplaceable || (m.Total <= m.Ready && !m.Parked) {
 			s.Unplaceable = append(s.Unplaceable, m.Name)
 		}
 	}
 	sort.Strings(s.Unplaceable)
 	return s
 }
+
+// ParkedModel is a model no longer retried, and why.
+type ParkedModel struct {
+	Model    string `json:"model"`
+	Failures int    `json:"failures"`
+	Reason   string `json:"reason"`
+}

internal/control/agentsvc/convert.go

@@ -3,6 +3,7 @@ package agentsvc
 
 import (
 	"fmt"
+	"maps"
 	"time"
 
 	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
@@ -22,6 +23,7 @@ func NodeFromRegister(r *llmgwv1.Register) inventory.Node {
 		Labels:         r.GetLabels(),
 		State:          inventory.Ready,
 		AgentVersion:   r.GetAgentVersion(),
+		Features:       r.GetFeatures(),
 		EngineVersions: r.GetEngineVersions(),
 		EnginesProbed:  r.GetEnginesProbed(),
 		MemoryTotalMB:  int(r.GetMemoryTotalMb()),
@@ -59,6 +61,9 @@ func NodeFromRegister(r *llmgwv1.Register) inventory.Node {
 			UtilizationPercent: int(g.GetUtilizationPercent()),
 			CUDAVersion:        g.GetCudaVersion(),
 			NVLinkPeers:        intSlice(g.GetNvlinkPeers()),
+			NICPaths:           maps.Clone(g.GetNicPaths()),
+			Fabric: fabricFrom(g.GetFabricState(), g.GetFabricStatus(), g.GetClusterUuid(),
+				g.GetCliqueId(), g.GetFabricBandwidth()),
 			Health: discovery.GPUHealth{
 				ECCErrors:    int(g.GetEccErrors()),
 				TemperatureC: int(g.GetTemperatureC()),
@@ -66,6 +71,7 @@ func NodeFromRegister(r *llmgwv1.Register) inventory.Node {
 			},
 		})
 	}
+	n.IMEX = discovery.IMEX{State: r.GetImex().GetState(), Channels: int(r.GetImex().GetChannels())}
 	for _, f := range r.GetFabrics() {
 		n.Fabrics = append(n.Fabrics, inventory.FabricAttachment{
 			Fabric:   f.GetFabric(),
@@ -112,11 +118,18 @@ func RegisterFromInventory(nodeID, hostname, mgmtAddr, agentVersion string,
 			UtilizationPercent: int32(g.UtilizationPercent),
 			CudaVersion:        g.CUDAVersion,
 			NvlinkPeers:        int32Slice(g.NVLinkPeers),
+			NicPaths:           maps.Clone(g.NICPaths),
 			Problems:           g.Health.Problems,
 			TemperatureC:       int32(g.Health.TemperatureC),
 			EccErrors:          int32(g.Health.ECCErrors),
 		})
+		if f := g.Fabric; f != nil {
+			last := r.Gpus[len(r.Gpus)-1]
+			last.FabricState, last.FabricStatus, last.ClusterUuid = f.State, f.Status, f.ClusterUUID
+			last.CliqueId, last.FabricBandwidth = f.CliqueID, f.Bandwidth
+		}
 	}
+	r.Imex = &llmgwv1.IMEX{State: inv.IMEX.State, Channels: int32(inv.IMEX.Channels)}
 	for _, f := range fabrics {
 		r.Fabrics = append(r.Fabrics, &llmgwv1.FabricAttachment{
 			Fabric: f.Fabric, Hcas: f.HCAs, RateGbps: int32(f.RateGbps),
@@ -178,6 +191,7 @@ func StartShardFromCommand(cmd reconciler.ShardCommand) *llmgwv1.StartShard {
 		Pool:             cmd.Pool,
 		Fabric:           cmd.Fabric,
 		ProfileFp:        cmd.ProfileFP,
+		ApiKey:           cmd.Spec.APIKey,
 	}
 }
 
@@ -198,6 +212,7 @@ func ShardSpecFromStart(m *llmgwv1.StartShard) (engines.ShardSpec, time.Duration
 		Pool:          m.GetPool(),
 		Fabric:        m.GetFabric(),
 		ProfileFP:     m.GetProfileFp(),
+		APIKey:        m.GetApiKey(),
 		Profile: engines.Profile{
 			Name: p.GetName(), Engine: p.GetEngine(), GPUClass: p.GetGpuClass(),
 			TP: int(p.GetTp()), PP: int(p.GetPp()),
@@ -263,3 +278,12 @@ func int32Slice(v []int) []int32 {
 	}
 	return out
 }
+
+// fabricFrom is a GPU's NVLink fabric as the wire carries it, nil when the
+// driver reported none.
+func fabricFrom(state, status, cluster, clique, bandwidth string) *discovery.GPUFabric {
+	if state == "" {
+		return nil
+	}
+	return &discovery.GPUFabric{State: state, Status: status, ClusterUUID: cluster, CliqueID: clique, Bandwidth: bandwidth}
+}

internal/control/agentsvc/heartbeat_test.go

@@ -188,3 +188,79 @@ func TestALogRequestIsAnsweredByTheNodeItWasSentTo(t *testing.T) {
 		t.Error("a node with no stream answered")
 	}
 }
+
+func TestAHeartbeatCarriesHowBusyTheNodeIs(t *testing.T) {
+	s := newTestService()
+	nodeWith(s, inventory.Node{ID: "n1", State: inventory.Ready})
+
+	s.applyHeartbeat("n1", &llmgwv1.Heartbeat{NodeId: "n1", Load: &llmgwv1.NodeLoad{
+		CpuPercent: 62.5, CpuCount: 8, Load1: 3.2, MemoryUsedBytes: 6 << 30, MemoryAvailableBytes: 2 << 30,
+		Shards: []*llmgwv1.ShardLoad{{ShardId: "r1-0", CpuCores: 4.5, MemoryRssBytes: 5 << 30}},
+	}})
+	n, _ := s.inv.Get("n1")
+	if n.Load == nil || n.Load.CPUPercent != 62.5 || n.Load.MemUsedBytes != 6<<30 ||
+		n.Load.Shards["r1-0"].CPUCores != 4.5 {
+		t.Fatalf("load = %+v", n.Load)
+	}
+
+	// A heartbeat that measured nothing — the agent's first — leaves the last
+	// measurement standing rather than blanking it.
+	s.applyHeartbeat("n1", &llmgwv1.Heartbeat{NodeId: "n1"})
+	if n, _ := s.inv.Get("n1"); n.Load == nil {
+		t.Error("a heartbeat without load erased it")
+	}
+}
+
+func TestAPortGoingDownTakesItsFabricAway(t *testing.T) {
+	// Checked only at registration, a port that went down left the node on
+	// its fabric until the agent restarted, still offered multi-node work.
+	s := newTestService()
+	nodeWith(s, inventory.Node{ID: "n1", State: inventory.Ready,
+		Fabrics: []inventory.FabricAttachment{{Fabric: "ib-a", HCAs: []string{"mlx5_0"}, RateGbps: 400}}})
+
+	s.applyHeartbeat("n1", &llmgwv1.Heartbeat{NodeId: "n1", FabricsChanged: true,
+		IbPorts: []*llmgwv1.IBPort{{Hca: "mlx5_0", Port: 1, State: "DOWN", PhysState: "Polling",
+			Counters: map[string]uint64{"link_downed": 4}}}})
+	n, _ := s.inv.Get("n1")
+	if len(n.Fabrics) != 0 {
+		t.Errorf("the fabric should be gone: %+v", n.Fabrics)
+	}
+	if len(n.IBPorts) != 1 || n.IBPorts[0].Active() || n.IBPorts[0].Counters["link_downed"] != 4 {
+		t.Errorf("ports = %+v", n.IBPorts)
+	}
+
+	// Back up: the fabric returns.
+	s.applyHeartbeat("n1", &llmgwv1.Heartbeat{NodeId: "n1", FabricsChanged: true,
+		Fabrics: []*llmgwv1.FabricAttachment{{Fabric: "ib-a", Hcas: []string{"mlx5_0"}, RateGbps: 400}}})
+	if n, _ := s.inv.Get("n1"); !n.HasFabric("ib-a") {
+		t.Error("the fabric should be back")
+	}
+
+	// A heartbeat that does not say the fabrics changed leaves them alone.
+	s.applyHeartbeat("n1", &llmgwv1.Heartbeat{NodeId: "n1", IbPorts: []*llmgwv1.IBPort{{Hca: "mlx5_0", Port: 1, State: "ACTIVE"}}})
+	if n, _ := s.inv.Get("n1"); !n.HasFabric("ib-a") {
+		t.Error("a heartbeat without a fabric change emptied the fabrics")
+	}
+}
+
+func TestAGPUFallingOutOfItsDomainTakesTheNodeOut(t *testing.T) {
+	// A fabric manager restarting drops cards out of the domain; the node has
+	// to stop being offered work that spans it until they are back.
+	s := newTestService()
+	fab := &discovery.GPUFabric{State: "Completed", ClusterUUID: "76b1ad76-7c24", CliqueID: "1"}
+	nodeWith(s, inventory.Node{ID: "t1", State: inventory.Ready, GPUs: []discovery.GPU{
+		{Index: 0, Fabric: fab}, {Index: 1, Fabric: fab}}})
+	if n, _ := s.inv.Get("t1"); n.NVLinkDomain() == "" {
+		t.Fatal("the node should start in its domain")
+	}
+	s.applyHeartbeat("t1", &llmgwv1.Heartbeat{NodeId: "t1",
+		GpuFabric: []*llmgwv1.GPUFabricState{{Index: 1, State: "In Progress"}},
+		Imex:      &llmgwv1.IMEX{State: "active", Channels: 1}})
+	n, _ := s.inv.Get("t1")
+	if n.NVLinkDomain() != "" {
+		t.Errorf("a node with a card out of the domain is still in it")
+	}
+	if n.IMEX.State != "active" || n.IMEX.Channels != 1 {
+		t.Errorf("imex = %+v", n.IMEX)
+	}
+}

internal/control/agentsvc/installable_test.go

@@ -0,0 +1,27 @@
+package agentsvc
+
+import (
+	"slices"
+	"testing"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
+)
+
+func TestANodeIsAskedOnlyForEnginesItCanInstall(t *testing.T) {
+	// An agent older than the list says nothing, and could install llama.cpp
+	// alone. Asked for vLLM, it showed as failing until it was upgraded.
+	if got := installable(&llmgwv1.Register{}); !slices.Equal(got, []string{"llamacpp"}) {
+		t.Errorf("an agent that said nothing: %v", got)
+	}
+	// This agent says what it can, and that is vLLM and SGLang besides.
+	now := &llmgwv1.Register{InstallableEngines: provision.Installable()}
+	for _, e := range []string{"llamacpp", "vllm", "sglang"} {
+		if !slices.Contains(installable(now), e) {
+			t.Errorf("this agent does not offer %s: %v", e, installable(now))
+		}
+	}
+	if slices.Contains(installable(now), "uv") {
+		t.Error("uv, a tool for the others, is offered as an engine")
+	}
+}

internal/control/agentsvc/service.go

@@ -9,6 +9,7 @@ import (
 	"maps"
 	"net/http"
 	"slices"
+	"sort"
 	"sync"
 	"time"
 
@@ -18,6 +19,7 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1/llmgwv1connect"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/fabrictest"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
 )
@@ -58,6 +60,9 @@ type Service struct {
 	logMu    sync.Mutex
 	logWaits map[string]chan *llmgwv1.LogsReply
 	logSeq   uint64
+	// scanWaits are the fabric scans awaiting a reply, by request id.
+	scanWaits map[string]chan *llmgwv1.FabricScanReply
+	testWaits map[string]chan *llmgwv1.FabricTestReply
 }
 
 // SetEngines sets the engines every node installs, by name and version.
@@ -174,7 +179,11 @@ func (s *Service) Connect(
 	// month, and a changed version reaches every node on its next connect.
 	if s.engines != nil {
 		p := &llmgwv1.Provision{}
+		can := installable(reg)
 		for _, e := range slices.Sorted(maps.Keys(s.engines)) {
+			if !slices.Contains(can, e) {
+				continue
+			}
 			p.Engines = append(p.Engines, &llmgwv1.EngineWant{Engine: e, Version: s.engines[e]})
 		}
 		if err := stream.Send(&llmgwv1.ControlMessage{
@@ -282,6 +291,24 @@ func (s *Service) handle(nodeID string, msg *llmgwv1.AgentMessage) {
 				"last_output", m.Shard.GetLastOutput())
 		}
 
+	case *llmgwv1.AgentMessage_FabricTest:
+		s.logMu.Lock()
+		ch, ok := s.testWaits[m.FabricTest.GetRequestId()]
+		delete(s.testWaits, m.FabricTest.GetRequestId())
+		s.logMu.Unlock()
+		if ok {
+			ch <- m.FabricTest
+		}
+
+	case *llmgwv1.AgentMessage_FabricScan:
+		s.logMu.Lock()
+		ch, ok := s.scanWaits[m.FabricScan.GetRequestId()]
+		delete(s.scanWaits, m.FabricScan.GetRequestId())
+		s.logMu.Unlock()
+		if ok {
+			ch <- m.FabricScan
+		}
+
 	case *llmgwv1.AgentMessage_Logs:
 		s.logMu.Lock()
 		ch, ok := s.logWaits[m.Logs.GetRequestId()]
@@ -309,7 +336,8 @@ func (s *Service) applyHeartbeat(nodeID string, hb *llmgwv1.Heartbeat) {
 	memMB := int(hb.GetMemoryTotalMb())
 	if len(used) == 0 && len(problems) == 0 && len(storage) == 0 &&
 		!hb.GetWeightsChanged() && memMB == 0 && !hb.GetEnginesChanged() &&
-		!hb.GetProvisionChanged() {
+		!hb.GetProvisionChanged() && hb.GetLoad() == nil && len(hb.GetIbPorts()) == 0 &&
+		!hb.GetFabricsChanged() && len(hb.GetGpuFabric()) == 0 && hb.GetImex() == nil {
 		return
 	}
 
@@ -400,9 +428,108 @@ func (s *Service) applyHeartbeat(nodeID string, hb *llmgwv1.Heartbeat) {
 			n.Provision = append(n.Provision, e)
 		}
 	}
+	if l := hb.GetLoad(); l != nil {
+		n.Load = loadFrom(l, time.Now())
+	}
+	if ports := hb.GetIbPorts(); len(ports) > 0 {
+		before := map[string]string{}
+		prev := map[string]inventory.IBPort{}
+		for _, p := range n.IBPorts {
+			before[fmt.Sprintf("%s/%d", p.HCA, p.Port)] = p.SMGUID
+			prev[fmt.Sprintf("%s/%d", p.HCA, p.Port)] = p
+		}
+		now := time.Now()
+		secs := now.Sub(n.IBAt).Seconds()
+		n.IBPorts = nil
+		for _, p := range ports {
+			// A different master than last time is a failover, or a port
+			// moved: the fabric's own event, kept for the page to show.
+			where := fmt.Sprintf("%s/%d", p.GetHca(), p.GetPort())
+			if old := before[where]; old != "" && p.GetSmGuid() != "" && old != p.GetSmGuid() {
+				s.log.Warn("a port sees a different master subnet manager", "node", nodeID,
+					"port", where, "from", old, "to", p.GetSmGuid())
+				s.inv.RecordSMChange(inventory.SMChange{At: time.Now(), Node: nodeID, Port: where,
+					From: old, To: p.GetSmGuid()})
+			}
+			var sms []inventory.SubnetManager
+			for _, r := range p.GetSubnetManagers() {
+				sms = append(sms, inventory.SubnetManager{GUID: r.GetGuid(), LID: int(r.GetLid()),
+					Priority: int(r.GetPriority()), State: r.GetState()})
+			}
+			n.IBPorts = append(n.IBPorts, inventory.IBPort{
+				SMs: sms, SMsError: p.GetSubnetManagersError(),
+				TxBps: rate(prev[where].Counters["port_xmit_data"], p.GetCounters()["port_xmit_data"], secs),
+				RxBps: rate(prev[where].Counters["port_rcv_data"], p.GetCounters()["port_rcv_data"], secs),
+				HCA:   p.GetHca(), Port: int(p.GetPort()), State: p.GetState(),
+				PhysState: p.GetPhysState(), RateGbps: int(p.GetRateGbps()),
+				LinkLayer: p.GetLinkLayer(), Counters: maps.Clone(p.GetCounters()),
+				NodeGUID: p.GetNodeGuid(), PortGUID: p.GetPortGuid(), GIDPrefix: p.GetGidPrefix(),
+				SMGUID: p.GetSmGuid(), SMLID: int(p.GetSmLid()), SMPriority: int(p.GetSmPriority()),
+				SMState: p.GetSmState(), SMError: p.GetSmError(), Network: p.GetNetwork(),
+			})
+		}
+	}
+	if len(hb.GetIbPorts()) > 0 {
+		n.IBAt = time.Now()
+	}
+	// NVLink domain membership, which a fabric manager restarting takes
+	// away from every card at once.
+	if fab := hb.GetGpuFabric(); len(fab) > 0 {
+		before := n.NVLinkDomain()
+		for _, f := range fab {
+			for i := range n.GPUs {
+				if n.GPUs[i].Index == int(f.GetIndex()) {
+					n.GPUs[i].Fabric = fabricFrom(f.GetState(), f.GetStatus(), f.GetClusterUuid(), f.GetCliqueId(), f.GetBandwidth())
+				}
+			}
+		}
+		if after := n.NVLinkDomain(); after != before {
+			s.log.Warn("a node's NVLink domain changed", "node", nodeID, "from", before, "to", after)
+		}
+	}
+	if m := hb.GetImex(); m != nil {
+		n.IMEX = discovery.IMEX{State: m.GetState(), Channels: int(m.GetChannels())}
+	}
+	// A port went down, or came back. Placement reads the fabrics, so this is
+	// what stops a node being offered multi-node work it can no longer carry.
+	if hb.GetFabricsChanged() {
+		var fabs []inventory.FabricAttachment
+		for _, f := range hb.GetFabrics() {
+			fabs = append(fabs, inventory.FabricAttachment{
+				Fabric: f.GetFabric(), HCAs: slices.Clone(f.GetHcas()), RateGbps: int(f.GetRateGbps()),
+			})
+		}
+		s.log.Warn("a node's fabrics changed", "node", nodeID,
+			"from", n.FabricNames(), "to", fabricNames(fabs))
+		n.Fabrics = fabs
+	}
 	s.inv.Upsert(n)
 }
 
+func fabricNames(fs []inventory.FabricAttachment) []string {
+	out := make([]string, len(fs))
+	for i, f := range fs {
+		out[i] = f.Fabric
+	}
+	return out
+}
+
+// loadFrom is a heartbeat's load as the inventory keeps it.
+func loadFrom(l *llmgwv1.NodeLoad, at time.Time) *inventory.Load {
+	out := &inventory.Load{
+		At: at, CPUPercent: l.GetCpuPercent(), CPUCount: int(l.GetCpuCount()),
+		Load1: l.GetLoad1(), Load5: l.GetLoad5(), Load15: l.GetLoad15(),
+		MemUsedBytes: l.GetMemoryUsedBytes(), MemAvailBytes: l.GetMemoryAvailableBytes(),
+		Shards: map[string]inventory.ShardLoad{},
+	}
+	for _, sh := range l.GetShards() {
+		out.Shards[sh.GetShardId()] = inventory.ShardLoad{
+			CPUCores: sh.GetCpuCores(), RSSBytes: sh.GetMemoryRssBytes(),
+		}
+	}
+	return out
+}
+
 func (s *Service) attach(ctx context.Context, nodeID string) *conn {
 	c := &conn{
 		nodeID: nodeID,
@@ -569,3 +696,149 @@ func (s *Service) Logs(ctx context.Context, nodeID, source string, lines int) ([
 		return nil, nil, fmt.Errorf("node %s did not answer; an agent older than v0.8.4 cannot", nodeID)
 	}
 }
+
+// ScanFabric asks a node to map the fabric one of its ports is on.
+func (s *Service) ScanFabric(ctx context.Context, nodeID, hca string, port int) (*llmgwv1.FabricScanReply, error) {
+	s.logMu.Lock()
+	if s.scanWaits == nil {
+		s.scanWaits = map[string]chan *llmgwv1.FabricScanReply{}
+	}
+	s.logSeq++
+	id := fmt.Sprintf("scan-%d", s.logSeq)
+	ch := make(chan *llmgwv1.FabricScanReply, 1)
+	s.scanWaits[id] = ch
+	s.logMu.Unlock()
+	defer func() {
+		s.logMu.Lock()
+		delete(s.scanWaits, id)
+		s.logMu.Unlock()
+	}()
+	if err := s.send(nodeID, &llmgwv1.ControlMessage{Msg: &llmgwv1.ControlMessage_FabricScan{
+		FabricScan: &llmgwv1.FabricScanRequest{RequestId: id, Hca: hca, Port: int32(port)}, //nolint:gosec // a port number
+	}}); err != nil {
+		return nil, err
+	}
+	ctx, cancel := context.WithTimeout(ctx, 3*time.Minute)
+	defer cancel()
+	select {
+	case r := <-ch:
+		return r, nil
+	case <-ctx.Done():
+		return nil, fmt.Errorf("node %s did not answer the fabric scan", nodeID)
+	}
+}
+
+// FabricTest asks a node for one side of a bandwidth test, and waits for
+// what it measured. The request's id is filled in here.
+func (s *Service) FabricTest(ctx context.Context, nodeID string, req *llmgwv1.FabricTestRequest) (*llmgwv1.FabricTestReply, error) {
+	s.logMu.Lock()
+	if s.testWaits == nil {
+		s.testWaits = map[string]chan *llmgwv1.FabricTestReply{}
+	}
+	s.logSeq++
+	id := fmt.Sprintf("test-%d", s.logSeq)
+	ch := make(chan *llmgwv1.FabricTestReply, 1)
+	s.testWaits[id] = ch
+	s.logMu.Unlock()
+	defer func() {
+		s.logMu.Lock()
+		delete(s.testWaits, id)
+		s.logMu.Unlock()
+	}()
+	req.RequestId = id
+	if err := s.send(nodeID, &llmgwv1.ControlMessage{Msg: &llmgwv1.ControlMessage_FabricTest{FabricTest: req}}); err != nil {
+		return nil, err
+	}
+	// The test's own length, and a package install the first time.
+	ctx, cancel := context.WithTimeout(ctx, time.Duration(req.GetSeconds())*time.Second+5*time.Minute)
+	defer cancel()
+	select {
+	case r := <-ch:
+		return r, nil
+	case <-ctx.Done():
+		return nil, fmt.Errorf("node %s did not answer the bandwidth test", nodeID)
+	}
+}
+
+// ScanFabrics maps every physical InfiniBand fabric every so often, asking
+// one node on each: the first, by name, whose port on it is up. A fabric
+// looks the same from all its ports, so one is enough, and the rest of its
+// nodes are left to their work.
+func (s *Service) ScanFabrics(ctx context.Context, every time.Duration) {
+	if every <= 0 {
+		every = 5 * time.Minute
+	}
+	// Soon after a start, once agents have sent a heartbeat or two.
+	timer := time.NewTimer(30 * time.Second)
+	defer timer.Stop()
+	for {
+		select {
+		case <-ctx.Done():
+			return
+		case <-timer.C:
+		}
+		s.scanOnce(ctx)
+		timer.Reset(every)
+	}
+}
+
+func (s *Service) scanOnce(ctx context.Context) {
+	nodes := s.inv.Nodes()
+	sort.Slice(nodes, func(i, j int) bool { return nodes[i].ID < nodes[j].ID })
+	canon := inventory.CanonicalSMs(nodes)
+	chosen := map[string]bool{}
+	for _, n := range nodes {
+		if n.State != inventory.Ready {
+			continue
+		}
+		for _, p := range n.IBPorts {
+			key := p.FabricKeyIn(canon)
+			if key == "" || p.RoCE() || chosen[key] {
+				continue
+			}
+			chosen[key] = true
+			where := fmt.Sprintf("%s/%d", p.HCA, p.Port)
+			r, err := s.ScanFabric(ctx, n.ID, p.HCA, p.Port)
+			sc := inventory.FabricScan{Key: key, At: time.Now(), Node: n.ID, Port: where}
+			switch {
+			case err != nil:
+				sc.Err = err.Error()
+			case r.GetError() != "":
+				sc.Err = r.GetError()
+			default:
+				sc.Output = r.GetOutput()
+			}
+			if r != nil {
+				sc.Took = time.Duration(r.GetTookMs()) * time.Millisecond
+			}
+			if sc.Err != "" {
+				s.log.Warn("could not map a fabric", "fabric", key, "node", n.ID, "port", where, "err", sc.Err)
+			}
+			s.inv.SetScan(sc)
+		}
+	}
+}
+
+// rate is a counter's increase a second, zero when there is no earlier
+// reading or the counter went backwards — an adapter reset starts it again.
+func rate(before, now uint64, secs float64) float64 {
+	if before == 0 || now < before || secs <= 0 || secs > 120 {
+		return 0
+	}
+	return float64(now-before) / secs
+}
+
+// installable are the engines a node can be asked to install. An agent older
+// than the list said nothing, and could install llama.cpp alone: asked for
+// vLLM too, it reported it unsupported, and a cluster half way through an
+// upgrade showed every such node as failing.
+func installable(reg *llmgwv1.Register) []string {
+	if can := reg.GetInstallableEngines(); len(can) > 0 {
+		return can
+	}
+	return []string{"llamacpp"}
+}
+
+// The admin API finds the bandwidth test by asserting this; a signature that
+// drifted would make it quietly nil.
+var _ fabrictest.Asker = (*Service)(nil)

internal/control/catalog/catalog.go

@@ -302,6 +302,13 @@ func (m Model) validate() error {
 		}
 		seen[p.Name] = struct{}{}
 
+		for k := range p.Env {
+			if bad, why := engines.UnsafeEnv(k); bad {
+				return fmt.Errorf("model %q profile %q: env %s is not allowed: %s",
+					m.Name, p.Name, k, why)
+			}
+		}
+
 		if p.GPUClass == "" && !cpuCapable(p.Engine) {
 			return fmt.Errorf("model %q profile %q: gpu_class is required for engine %q",
 				m.Name, p.Name, p.Engine)

internal/control/catalog/catalog_test.go

@@ -302,3 +302,22 @@ func TestSimulatedNeedsEveryProfileToBeSimulated(t *testing.T) {
 		})
 	}
 }
+
+func TestAProfileCannotSetWhatRunsCodeInTheEngine(t *testing.T) {
+	// A profile's env reaches an engine started as root. LD_PRELOAD from a
+	// model definition would make "add a model" into "run this everywhere".
+	for _, k := range []string{"LD_PRELOAD", "ld_library_path", "PYTHONSTARTUP", "PATH", "BASH_ENV"} {
+		m := Model{Name: "m", Source: "http://x/m.gguf", Profiles: []Profile{{
+			Name: "cpu", Engine: "llamacpp", Env: map[string]string{k: "/tmp/x"},
+		}}}
+		if _, err := New([]Model{m}); err == nil || !strings.Contains(err.Error(), "not allowed") {
+			t.Errorf("%s: %v", k, err)
+		}
+	}
+	ok := Model{Name: "m", Source: "http://x/m.gguf", Profiles: []Profile{{
+		Name: "cpu", Engine: "llamacpp", Env: map[string]string{"OMP_NUM_THREADS": "4", "PYTHONUNBUFFERED": "1"},
+	}}}
+	if _, err := New([]Model{ok}); err != nil {
+		t.Errorf("ordinary tuning refused: %v", err)
+	}
+}