Describe v0.10.0

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

.gitea/workflows/release.yml

@@ -55,6 +55,13 @@ jobs:
             (cd dist && sha256sum "${name}" >> checksums.txt)
           done
 
+          # The reverse proxy configurations, so a release page carries
+          # everything install.md asks for and the repository is not needed.
+          for f in deploy/proxy/Caddyfile deploy/proxy/nginx.conf; do
+            cp "$f" dist/
+            (cd dist && sha256sum "$(basename "$f")" >> checksums.txt)
+          done
+
           # An NVML build for nodes with NVLink, which the portable binary
           # cannot serve.
           #

CHANGELOG.md

@@ -13,11 +13,472 @@ is only released once its **exit criterion** is met, not when its code lands.
 | v0.6.0 | 4 | Multi-node replicas and fabric awareness | released |
 | v0.7.0 | — | Conversations, agent memory, disk management | released |
 | v0.8.0 | — | PostgreSQL, leader election, agent failover | released |
+| v0.9.0 | — | Managing people and the tenancy tree | released |
+| v0.10.0 | — | InfiniBand fabrics and NVLink, doctor, node add, engines installed by the nodes | released |
 | v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
 
 Until v1.0.0 the minor version is the breaking one: configuration formats and Go
 APIs change between minors without a deprecation period.
 
+## v0.10.0 — 2026-09-29
+
+Independent InfiniBand fabrics and NVLink domains, told apart, drawn and
+placed on; a node checked by `llmgw doctor` and added by `llmgw node add`; vLLM
+installed by the nodes themselves; people in many places at once; and a chat
+page, editor setup and a model explorer for the people using it.
+
+### Independent InfiniBand fabrics, told apart and drawn
+
+A fabric was whatever `agent.yaml` called it, and nothing checked. At a site
+with several independent fabrics, two given one name would have had a replica
+spread across both — started, then hung on its first collective. Now:
+
+- **Identity.** Each active InfiniBand port reports the master subnet manager
+  it sees (`sminfo`) and every subnet manager on its fabric (`saquery SMIR`),
+  asked once a minute through one port per fabric. Ports whose managers are
+  listed together are one physical fabric, so a failover does not split one in
+  two. A RoCE port's fabric is the IP network of its RoCE v2 address. The agent
+  installs `infiniband-diags` where there are adapters and no `sminfo`.
+- **Placement** divides a declared fabric by that identity and forms a replica
+  on one part only. A node whose adapters for one name are cabled to two
+  fabrics is refused; a node that cannot say is kept apart from those that can.
+- **Subnet managers.** The cluster page lists each fabric's master, its
+  priority, its standbys and its master changes over the last day, and warns
+  when a fabric has no standby, when its ports disagree on the master, when a
+  port is up with no subnet manager (INIT), and when one name covers two
+  fabrics or two names one fabric.
+- **Ports while running.** The agent re-reads every InfiniBand and RoCE port on
+  each heartbeat: state, physical state, rate, error and data counters. A port
+  going down takes its fabric off the node within seconds, and back when it
+  returns — it used to wait for the agent to restart.
+- **Maps.** Every five minutes one node on each fabric
+  runs `ibnetdiscover`, and the control plane parses it. `/admin/fabrics` draws
+  the fabric: switches in their tiers, adapters under their leaves, cables
+  coloured by rate, degraded ones — trained below the rest of their kind —
+  marked in red, rail alignment, and the cluster's own adapters live, with
+  their traffic running along the cables. It zooms, pans, highlights what a
+  switch or adapter touches, and opens detail on a click. Placement prefers
+  the nodes of one leaf and puts nodes with a degraded cable last.
+
+### NVLink, on every build and across nodes
+
+- **The release binary sees NVLink.** It is built without cgo, so without NVML,
+  which was the only source of NVLink peers: on every NVLink machine the
+  portable agent reported none. Discovery now reads `nvidia-smi topo -m`: NVLink
+  peers, each card's NUMA node, and how near each network adapter is.
+- **GPUs are chosen by it.** Every set of the right size is scored: NVLink pairs
+  within it, then — for a multi-node shard — nearness to the fabric's adapters,
+  then NUMA nodes spanned. It used to grow one card's peers, which took cards
+  joined to that one but not to each other.
+- **Multi-node NVLink domains** — an NVL72 rack, an NVLink Switch system — are
+  found from each GPU's fabric state, ClusterUUID and CliqueId, with
+  `nvidia-imex` and its channels. A replica spanning nodes is placed inside one
+  domain first, on nodes whose every card is joined and whose IMEX is up, and
+  its shards get `NCCL_MNNVL_ENABLE=1`; InfiniBand is the fallback.
+  `/admin/nvlink` draws each domain as a rack of trays and says what keeps a
+  tray out. A node's page shows its GPU-by-GPU NVLink matrix.
+
+### Multi-node launches fixed before they meet hardware
+
+- **vLLM launches multi-node its own way by default.** An empty
+  `engines.vllm.launch_mode` meant `ray`, never checked against a real cluster;
+  it now means `mp`, every node running `vllm serve` with `--nnodes`,
+  `--node-rank` and `--master-addr`, the workers `--headless`, as vLLM
+  documents. The `mp` path used to give workers no rank and the leader's
+  address as their own. Set `launch_mode: ray` to keep Ray.
+- **NCCL bootstraps over the node's own interface.** The agent's default of
+  `eth0` meant it never looked for the interface holding its management
+  address, and the control plane sent one name for every node. A node on
+  `ens3` or `bond0` had NCCL bind nothing.
+- **RoCE** shards get `NCCL_IB_GID_INDEX`, the RoCE v2 entry with an IPv4
+  address; NCCL's default is a v1 link-local one that does not route.
+- A Ray worker now waits for its head before joining.
+- The agent unit sets `LimitMEMLOCK=infinity`, which RDMA needs.
+
+### `llmgw doctor`
+
+Checks a node for what llmgw needs of its GPUs and fabric, and says what is
+wrong and how to fix it, most serious first: NVSwitch without the fabric
+manager, GPUs not joined to their NVLink domain, IMEX, `nvidia_peermem` and
+`ib_uverbs`, each port's link and subnet manager, a fabric with no standby,
+RoCE without a v2 GID, the agent's declared fabrics against the adapters, the
+interface NCCL will use, the unit's memlock limit, and the NCCL environment a
+shard on each fabric would be given. `--json` writes every fact it read, to
+send back; it exits non-zero when something fails.
+
+It also says when the CPU offers no AVX2 — a VM left on QEMU's default CPU
+model, where llama.cpp runs several times slower and nothing else looks wrong —
+and whether vLLM can be installed on the node, and why not.
+
+### vLLM installs itself on GPU nodes
+
+The agent installs vLLM as it does llama.cpp: on every node with an NVIDIA GPU
+that can run it, with nothing done by hand. A Python 3.12 environment is built
+by uv — itself a GitHub release checked against its checksum — from PyPI,
+resolved as PyPI stood a day after that vLLM was published and installed with
+`--require-hashes`, so every node gets the same files whenever it is built. The
+resolved list stays in the environment as `llmgw-requirements.txt`.
+
+- The default is vLLM 0.30.0, which takes every flag llmgw passes. Set
+  `engines.vllm` in `server.yaml` to change it, or to `""` to keep your own.
+- It is built for CUDA 13: it needs driver 580 and a Turing GPU or newer, and
+  25 GB free while it installs. A node that lacks one says which on the
+  cluster page; a node with no GPU skips vLLM without a word.
+- gcc is installed with it, since Triton compiles its kernel launcher on the
+  first model load.
+- SGLang installs the same way, on GPU nodes, when `engines.sglang` names a
+  version.
+
+Checked by installing vLLM 0.30.0 this way on a node of the test cluster: 7.7 GB,
+torch 2.13 for CUDA 13.0, importing cleanly. That node has no GPU, so vLLM
+itself was not started there.
+
+A node is asked only for engines its agent says it can install. An agent older
+than this release says nothing, and is asked for llama.cpp alone, so a cluster
+half way through its upgrade does not show every node as failing vLLM.
+
+### `llmgw node add`: a node in one command
+
+Adding a node was a bash script from the repository, not installed anywhere,
+that needed the node's name, the control plane's address and a dozen flags.
+It is now part of the binary:
+
+```
+sudo llmgw node add ubuntu@10.0.0.21
+```
+
+run on the control plane. The node id is the machine's hostname and the
+address its agent dials is this machine's on the route to it, checked against
+the server certificate. It installs llmgw on the node — the release for its
+architecture, fetched and checked there, which works on RHEL too — runs
+`llmgw doctor` there and stops on a problem, issues and installs the
+certificate, writes `agent.yaml` and the unit, starts the agent, and waits for
+the node to join and install llama.cpp.
+
+In a terminal it is a checklist that fills in, and asks what a script cannot:
+which fabric each group of active adapters is on — grouped by the subnet
+manager or RoCE network they report — and whether to go on past a problem.
+Elsewhere it prints one line per step. Running it again on a node is safe.
+
+`llmgw node remove` forgets a node that is gone for good, which otherwise
+stays listed as lost until the control plane restarts; only a lost node, or an
+unreachable one with nothing placed on it, is removed. `llmgw ctl nodes` no
+longer lists vLLM as skipped on every CPU node.
+
+Tried on an empty VM, from the terminal screen and plain: joined with
+llama.cpp ready in under a minute, and 17 seconds to add it again.
+
+### People belong to many places, or to none
+
+A person was in exactly one project, chosen at approval, and could not use the
+gateway before that. Now:
+
+- **Memberships.** A person can be a member of any number of organizations,
+  teams and projects, or of none. A member of a team or organization may use
+  every project in it. `llmgw users join/leave`, and the people page adds and
+  removes them from either side. Approving no longer requires a project.
+- **Personal keys.** A key can belong to a person with no project. On
+  `/profile` a person chooses what a new key spends against: one of their
+  projects, or Personal. `llmgw keys create --for <person>` does the same.
+  `auth.personal` sets default limits for personal use, or turns it off.
+- **Limits for people.** `--scope user` limits a person across every key,
+  project and the chat page. `--per-member` on an organization, team or
+  project gives each member their own share there. Both are on the quotas
+  page, and are enforced alongside the tree, each in its own pool.
+- **The chat page** asks what to spend against when a person has more than one
+  choice, and checks it on every message.
+- **Spend is recorded with who spent it**, so a person's budget survives a
+  restart, personal use and the chat page included.
+
+Leaving a project revokes the keys the person made there, unless another
+membership still reaches it. A personal key keeps no conversations or notes:
+those are kept per project.
+
+### An organization's own page
+
+A member of an organization can be made its **manager** (`users join --org …
+--role manager`, or the star on the people page). A manager gets `/org`: the
+organization's usage over time by team, project or person, its teams and
+projects with limits and spend, and its members — read-only, and only what was
+spent in its projects. Administrators can open every organization's page.
+
+### The chat page
+
+- A model that is not running shows a progress bar while it loads — waiting
+  for a node, downloading, starting — and the message is sent once it is up,
+  rather than failing after 45 seconds with "the model is loading".
+- A finished reply is rendered as markdown, by the server under the model
+  cards' rules, with its maths typeset by KaTeX: "$5 and $10" stays prose.
+- Stop, while a reply streams; its tokens, speed and time to first token under
+  it; a system prompt and a temperature; and the conversation kept in the
+  browser across a reload — the gateway still keeps none.
+
+### Connect an editor, and the Responses API
+
+- The account page makes a key for **Continue**, **Codex** or **Cline** and
+  shows what the tool needs, filled in: Continue's assistant file, Codex's
+  provider and profile, Cline's four settings. Continue and Codex get a one-line
+  command (`curl -fsSL <gateway>/setup/<tool>?model=… | sh`) whose script holds
+  no secret — the key comes from the environment — keeps a copy of whatever it
+  replaces, and leaves the rest of the file alone.
+- **`POST /v1/responses`** translates the Responses API, which Codex speaks, to
+  chat completions and back — instructions, messages, function and custom
+  tools with their calls and outputs, usage, streamed or not — down the same
+  path as any request.
+
+### Your own use
+
+The account page (was "Your keys") shows the last thirty days of a person's own
+use and, for each place they spend, a bar for every budget that applies — their
+own, or a project's shared with others, said as such.
+
+### The mirror page
+
+- **Explore models** opens on what is trending on Hugging Face, with views for
+  new and most downloaded, filters for task and size, and each repository's
+  parameters, architecture (✓ where the nodes' llama.cpp loads it), context,
+  base model and license. "Only what these nodes can run" is on by default.
+- A repository opens beneath its row with its model card, and a download
+  started there starts in place, its progress followed in the table above.
+- What the mirror holds is grouped by model family.
+- An importance matrix published beside a model's quantisations is no longer
+  offered as a model.
+
+**How a model runs.** Each model a mirrored file is serving now shows how it runs — always on, on
+demand, or stopped — and it can be changed there, without the CLI. The serve
+form asks the same: on demand loads the model when somebody asks for it and
+stops it after 15 minutes unused. A model declared in `server.yaml` shows its
+mode and says to change it there, because the file is read on every start.
+
+**Weights the nodes cannot load are not fetched.** `kev-0.5b` was a GGUF whose architecture, `ggmlc`, llama.cpp does not know. It
+was downloaded, catalogued and placed, and failed on every node it was tried on.
+The architecture was in the first kilobytes of the file all along. Now:
+
+- The mirror page's registry listing shows each file's architecture and, for
+  one the nodes' llama.cpp cannot load, says so instead of offering to fetch
+  it. A vision projector (`mmproj`) is marked as not a model on its own.
+- A fetch reads the file's header first — a range request, not a download —
+  and refuses one that cannot be served, unless "fetch anyway" is ticked.
+- Serving from the mirror and `llmgw ctl model add` refuse the same.
+
+The list of architectures is llama.cpp's own (`LLM_ARCH_NAMES`) at the build
+llmgw pins, b10991: 152 of them. A different build configured in `engines:` is
+not refused, only not checked.
+
+### Each node's CPU and memory, and a page per node
+
+The agent reads `/proc` between heartbeats: CPU across all cores, load, memory
+used and available, and each engine's cores and resident memory. The cluster
+page's node table shows CPU and RAM as bars; each node has a page with its
+current load, its engines, its fabric ports, its GPUs, and an hour of charts
+from Prometheus.
+
+### The admin pages
+
+- The cluster page leads with capacity, nodes, models and replicas, then usage;
+  it refreshes in place instead of reloading, and its usage table names people
+  and projects instead of ids.
+- The usage page's legend gives each line's total, not its last hour.
+- The people page keeps an account on one line and its rarer actions in a menu.
+- Every page fits a phone: a wide table scrolls in its own box.
+- Static files are named by their content, so a deploy's new styles and
+  scripts reach browsers at once instead of a day later.
+
+**One menu on every page.** Each page drew its own menu, and no two agreed: different links, in a different
+order, the page you were on sometimes listed and sometimes not. Every signed-in
+page now draws the same one — Cluster, People, Quotas, Usage, Mirror on the
+left for an administrator, Chat, Account and Sign out on the right for
+everyone — with the current page marked. The chat and key pages name the
+person's project by its path rather than its id.
+
+**The quotas page says what actually applies.** A branch that sets no limit of its own used to read "—" in every column while
+its organization held it to sixty requests a minute. Each limit now shows the
+value it is held to from above, muted and marked ↑, with where it comes from on
+hover; a larger value set below a stricter one says what caps it, for rate and
+token limits as well as budgets. Spend and budget share a column with a meter
+that turns red past 90%. The tree is indented like the people page, opening a
+form or saving one no longer reloads the page, and the explanations moved under
+"How limits combine". Projects are placed under their team by id rather than by
+the start of their path, which misplaced them when two organizations shared a
+name.
+
+### The project's website, in every gateway
+
+`/about` is llmgw's own page: what it does, InfiniBand and NVLink, installing,
+the reverse proxy configurations, and the release's files to download. It is
+served without a key like `/`, whose footer links to it, and says nothing about
+the installation. Its fonts are in the binary, so it makes no request elsewhere.
+The published website is rendered from the same template.
+
+### Metrics
+
+`llmgw_node_cpu_utilization_percent`, `llmgw_node_cpus`, `llmgw_node_load`,
+`llmgw_node_memory_used_bytes`, `llmgw_node_memory_available_bytes`,
+`llmgw_shard_cpu_cores`, `llmgw_shard_memory_bytes`, `llmgw_ib_port_active`,
+`llmgw_ib_port_rate_gbps`, `llmgw_ib_port_counter_total`,
+`llmgw_fabric_master_sm`, `llmgw_fabric_subnet_managers`,
+`llmgw_fabric_problems`, `llmgw_gpu_nvlink_joined`, `llmgw_imex_up`.
+
+### Fixes
+
+- PostgreSQL queries with an apostrophe in a comment failed: the placeholder
+  rewriting read it as the start of a string. Comments are now skipped.
+- Chart axes read "20.0BB" for sizes and 0% for a quiet CPU.
+- Usage by team, project or organization, and the spend on the quotas page,
+  left out every request made on the chat page: they were joined through the
+  key, and the chat page has none. They now count where a request was spent.
+- `/org` was answered with "no api key was provided" for a signed-in browser;
+  a test now drives every session page through the authentication middleware.
+
+### Upgrading
+
+Upgrade the control plane first, then the agents: an older agent is served as
+before and simply reports none of the new measurements. Existing agent units
+keep their old limits until `LimitMEMLOCK=infinity` is added to them. A node
+that set `mgmt_interface` keeps it; one that did not now uses the interface
+holding its management address rather than `eth0`. Multi-node vLLM replicas
+start with `mp` unless `launch_mode: ray` is set.
+
+The store migrates on its first start. Migration 0010 turns everyone's project into a membership and
+attributes past usage to people where a key had an owner. On SQLite it rebuilds
+the keys table; it was run against a copy of a production PostgreSQL store
+before release. `llmgw users move` is gone: use `join` and `leave`.
+
+vLLM is installed on every GPU node that can run it, once its agent is
+upgraded: about 8 GB each, from PyPI. Set `engines.vllm: ""` in `server.yaml`
+before upgrading to keep vLLM your own. `deploy/bootstrap/onboard-node.sh` is
+replaced by `llmgw node add`; it still works, for a machine holding the
+authority but not this llmgw.
+
+## v0.9.2 — 2026-09-28
+
+### A new control plane serves a model by itself
+
+`llmgw server init` now also runs a model mirror on the same machine, makes the
+machine a node of its own control plane, and fetches Qwen2.5 0.5B into the
+mirror and serves it there — on the machine's NVIDIA GPU when `nvidia-smi` finds
+one (the node installs the CUDA build of llama.cpp), on CPU otherwise, with a
+CPU profile as the fallback. An NVIDIA card on the PCI bus with no working
+driver is reported. `--no-mirror`, `--no-node` and `--no-model` leave these out,
+and every step is skipped on a rerun when it is already done.
+
+Not yet run end to end on a real machine.
+
+## v0.9.1 — 2026-09-28
+
+### The mirror page is one line per model
+
+What the mirror holds used to take two table rows per file — seven columns and
+the address under them — so a sharded model filled the screen. Now each version
+is one line: model and version, the file (or "4 files" for a set), total size,
+and which catalog models use it. The address moved into a "copy address" button;
+the checksum is in the file's tooltip. A version already in use is not offered
+for serving again, and removing one warns that the catalog uses it. The header
+counts versions rather than files.
+
+## v0.9.0 — 2026-09-28
+
+### Managing people and the tenancy tree
+
+Organizations, teams and projects can now be renamed, moved and removed, and an
+approved person can be moved to another project or removed — none of which
+could be done at all before, from the web or the CLI.
+
+- **`/admin/users`** shows the whole tree, each project opening onto its people
+  and keys, with approve, move, suspend, reinstate and remove for people,
+  revoke for keys, and add, rename, move and remove at every level. A search
+  over every account sits below it. Changes update the page in place (htmx,
+  embedded in the binary) and it still works with scripts off.
+- **`llmgw org`** — `tree [--people]`, `add`, `rename`, `move`, `remove`.
+- **`llmgw users move`** and **`llmgw users delete`**.
+- **`llmgw keys list --all`** lists every key with its project and owner.
+- Every command that takes a project, team or organization accepts an id, a
+  path such as `Acme/platform/gateway`, or a unique name, and `--project` may
+  be omitted when there is only one. Users are named by id or email.
+
+Moving a person takes the keys they made with them; keys an operator issued stay
+with the project. Only empty things are removed, and a project that ever had a
+key is never removed, because usage history refers to it.
+
+### Changes
+
+- **`--project` no longer defaults to `default`** in `keys create`, `keys list`,
+  `users approve`, `users create`, `keys threads` and `keys memory`: a tree
+  built from the web has no project of that name. With one project it is
+  chosen; with several the command lists them.
+- **`keys limits --budget` is in whole units**, as the quotas page takes it,
+  rather than micro-units. `--budget 50000000` now means fifty million units;
+  divide existing scripts' figures by 1 000 000.
+
+### Fixes
+
+- `keys init` could silently move an existing team or project to another
+  organization or team by redeclaring it. It now refuses.
+- An account rejected while pending and then reinstated became active with no
+  project, and failed at its first key. Reinstating one is refused; approving
+  gives it a project. Rejecting a signup now removes it.
+
+## v0.8.8 — 2026-09-28
+
+### Fixes
+
+- **`server init --postgres` failed on Rocky Linux** with "postgresql-setup:
+  executable file not found". The PostgreSQL project's own packages (PGDG)
+  provide `postgresql-server` too, so dnf can answer that name with
+  `postgresql16-server`, which names its unit `postgresql-16.service` and keeps
+  its tools in `/usr/pgsql-16/bin`. init now finds the server unit systemd has,
+  reads the data directory from it, and runs `initdb` itself when the directory
+  is empty. A PostgreSQL that was already installed is used as it is.
+
+### Known gaps
+
+Not yet run on a real machine. Three arrangements are not handled: a
+PostgreSQL initialised with `ident` for TCP logins (the connection check fails
+and says so), one listening on a port other than 5432, and several PGDG
+versions where the highest is not the one on 5432.
+
+## v0.8.7 — 2026-09-28
+
+### PostgreSQL in one command
+
+```
+sudo llmgw server init --postgres
+```
+
+Installs PostgreSQL with the machine's own package manager when there is none,
+creates the `llmgw` role and database with a generated password kept in
+`/etc/llmgw/secrets/store-dsn`, and checks that it connects. On a control plane
+already running on SQLite it stops the gateway, copies the store across so
+every account and key survives, points `auth.store` at PostgreSQL and starts
+the gateway again; if the move fails, the gateway is started again on SQLite.
+Given on the first run, there is never a SQLite store.
+
+`server.yaml` is edited in place with its comments intact, and the previous
+file is kept beside it. A PostgreSQL that init did not install is left sized as
+it was, since something else may depend on it.
+
+The PostgreSQL installation itself has not yet run on a real machine; the move,
+the configuration edit and the rerun behaviour are tested.
+
+### Fixes
+
+- **A new control plane with no models did not start.** A configuration with
+  no static endpoints and no models was refused as having nothing to serve, so
+  under systemd the unit restarted every two seconds and nothing listened on
+  8080. It now runs as a control plane with an empty catalog.
+
+## v0.8.6 — 2026-09-28
+
+### Reverse proxy configurations
+
+`deploy/proxy/` has a Caddyfile and an nginx configuration for the client
+port, and both are attached to every release beside the binaries. They do not
+buffer streamed replies, their read timeout outlasts a model's wait before its
+first token, their body limit matches the router's, and they set
+`X-Forwarded-Proto` and `X-Forwarded-Host` themselves — llmgw marks session
+cookies `Secure` from the first and shows the second as the base URL on its
+landing page.
+
 ## v0.8.5 — 2026-09-28
 
 ### A control plane in one command

README.md

@@ -1,7 +1,7 @@
 # llmgw
 
-A single-binary LLM gateway for clusters with multiple InfiniBand fabrics and
-heterogeneous GPUs.
+A single-binary LLM gateway for clusters with multiple InfiniBand fabrics,
+NVLink domains and heterogeneous GPUs.
 
 It serves LLMs running on your own GPU servers (vLLM / SGLang) behind one
 OpenAI-compatible endpoint. Design document: [llmgw-plan.md](llmgw-plan.md).
@@ -32,6 +32,7 @@ Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
 | v0.6.0 | 4 | Multi-node placement and fabric awareness | released |
 | v0.7.0 | — | Conversations, agent memory, disk management | released |
 | v0.8.0 | — | PostgreSQL, leader election, agent failover | released |
+| v0.9.0 | — | Managing people and the tenancy tree | released |
 | v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
 
 As of v0.6.0 a cluster runs unattended: agents report their hardware, the
@@ -48,15 +49,23 @@ a header would let any client promote itself into the top priority band.
 
 One thing is worth saying plainly:
 multi-node replicas are proven on a simulated cluster and on real hardware
-without a fabric. The InfiniBand, NVML and NCCL paths are written against the
-formats and failure modes those tools actually have, but they have not yet run
-on a machine that has them.
+without a fabric. The InfiniBand, NVLink and NCCL paths — telling fabrics apart
+by their subnet managers, mapping them, finding NVLink domains, launching across
+nodes — are written against the formats and failure modes those tools actually
+have and tested against them, but they have not yet run on a machine that has
+them. On such a machine, run `llmgw doctor --json` first and keep the report;
+[docs/fabrics.md](docs/fabrics.md) says what to look at after that.
 
 ## Documentation
 
-- [docs/install.md](docs/install.md) — installing from nothing: control plane,
-  nodes, engines, weights, first request
-- [docs/quickstart.md](docs/quickstart.md) — the short path
+- [docs/quick-install.md](docs/quick-install.md) — installing on servers, in
+  six steps: start here
+- [docs/install.md](docs/install.md) — the same from nothing, with every option
+  explained: control plane, nodes, engines, weights, first request
+- [docs/quickstart.md](docs/quickstart.md) — one workstation, from source
+- [docs/fabrics.md](docs/fabrics.md) — InfiniBand fabrics and NVLink: what
+  llmgw needs of them, checking a node with `llmgw doctor`, and what to look at
+  when a replica spanning nodes will not run
 - [docs/choosing-models.md](docs/choosing-models.md) — which models to run,
   how many of each, and where the weights should live
 - [docs/data.md](docs/data.md) — what llmgw stores, what it does not, and
@@ -66,6 +75,8 @@ on a machine that has them.
   does not exist or miss one that does
 - [deploy/mirror/](deploy/mirror/) — a model mirror, which is the cheapest
   and most useful thing to build around llmgw
+- [deploy/proxy/](deploy/proxy/) — Caddy and nginx in front of the client
+  port, with streaming and the forwarded headers right
 - [deploy/ha/](deploy/ha/) — running two control planes, and the one health
   check that has to be right
 - [docs/node-onboarding.md](docs/node-onboarding.md),
@@ -88,6 +99,7 @@ llmgw ctl replicas --model llama     # replicas and their shards
 llmgw ctl explain-placement llama    # why a model has not been placed
 llmgw ctl drain gpu-a-01             # move work off a node, one replica at a time
 llmgw ctl cordon gpu-a-01            # stop new placement, leave what runs
+sudo llmgw doctor                    # on a node: what its GPUs, NVLink and fabric need
 ```
 
 `ctl` talks to the admin port (`:9090` by default), not the client-facing one.

api/proto/llmgw/v1/agent.proto

@@ -24,6 +24,7 @@ message AgentMessage {
     Heartbeat heartbeat = 2;
     ShardStatus shard = 3;
     LogsReply logs = 4;
+    FabricScanReply fabric_scan = 5;
   }
 }
 
@@ -37,6 +38,7 @@ message ControlMessage {
     RemoveWeights remove_weights = 5;
     Provision provision = 6;
     LogsRequest logs = 7;
+    FabricScanRequest fabric_scan = 8;
   }
 }
 
@@ -114,6 +116,15 @@ message Register {
   // meant "an agent too old to say" and constrained nothing — so a freshly
   // built node with no engine installed yet was a candidate for every model.
   bool engines_probed = 14;
+  // imex is the service that lets a multi-node NVLink domain share memory.
+  IMEX imex = 15;
+
+  // installable_engines are the engines this agent knows how to install
+  // itself. The control plane asks it for no others. Empty is an agent older
+  // than the field, which could install llama.cpp and nothing else: asked for
+  // vLLM, it answered "unsupported", and every node of a cluster half way
+  // through an upgrade showed as failing.
+  repeated string installable_engines = 16;
 }
 
 // Host describes the operating system and processor of a node.
@@ -207,6 +218,17 @@ message GPU {
   // Zero when the driver is too old to report it, which is not the same as
   // idle: the metric is simply absent rather than wrong.
   int32 utilization_percent = 15;
+  // nic_paths is how far each network adapter is from this card, by HCA
+  // name: PIX, PXB, PHB, NODE or SYS, from nvidia-smi topo.
+  map<string, string> nic_paths = 16;
+  // The card's multi-node NVLink membership, from nvidia-smi -q: state is
+  // Completed once the fabric manager has joined it; cluster_uuid and
+  // clique_id name the domain it joined.
+  string fabric_state = 17;
+  string fabric_status = 18;
+  string cluster_uuid = 19;
+  string clique_id = 20;
+  string fabric_bandwidth = 21;
 }
 
 message FabricAttachment {
@@ -231,6 +253,102 @@ message LocalWeights {
   bool shared = 3;
 }
 
+// NodeLoad is a node's use of its own CPU and memory, and each engine's share.
+//
+// Carried on the heartbeat rather than scraped from the node: the control
+// plane already exports the fleet from one place, and a node exporter per
+// machine would be a scrape target, a port and a firewall rule each.
+message NodeLoad {
+  // cpu_percent is the whole machine, 0 to 100 however many cores it has.
+  double cpu_percent = 1;
+  int32 cpu_count = 2;
+  // The kernel's run-queue averages over 1, 5 and 15 minutes.
+  double load1 = 3;
+  double load5 = 4;
+  double load15 = 5;
+  // memory_used_bytes is total less available: what the machine could not
+  // hand to a new process without evicting something.
+  int64 memory_used_bytes = 6;
+  int64 memory_available_bytes = 7;
+  repeated ShardLoad shards = 8;
+}
+
+// ShardLoad is one engine process group's use.
+message ShardLoad {
+  string shard_id = 1;
+  // cpu_cores is in cores, the way top counts a process: 2.5 is two and a
+  // half cores busy, which a percentage of the machine would hide on a large
+  // one.
+  double cpu_cores = 2;
+  int64 memory_rss_bytes = 3;
+}
+
+// IBPort is one InfiniBand or RoCE port's state and counters.
+message IBPort {
+  string hca = 1;
+  int32 port = 2;
+  // state is ACTIVE, INIT, DOWN...; phys_state tells a cable unplugged
+  // (Polling, Disabled) from a link up with no subnet manager (LinkUp and INIT).
+  string state = 3;
+  string phys_state = 4;
+  int32 rate_gbps = 5;
+  // link_layer is InfiniBand or Ethernet (RoCE).
+  string link_layer = 6;
+  // counters are cumulative, by sysfs name; data counters in bytes.
+  map<string, uint64> counters = 7;
+
+  // Identity: the adapter's node GUID and the port's own, as the fabric
+  // knows them.
+  string node_guid = 8;
+  string port_guid = 9;
+  // The subnet manager this port sees as master, on InfiniBand. Every port on
+  // one physical fabric sees the same one, which is what tells separate
+  // fabrics apart when their names are only what somebody typed.
+  string sm_guid = 10;
+  int32 sm_lid = 11;
+  int32 sm_priority = 12;
+  string sm_state = 13;
+  // sm_error says why sm_guid is empty on an active InfiniBand port:
+  // infiniband-diags missing, or no subnet manager answering.
+  string sm_error = 14;
+  // network is, on RoCE, the IP network of the RoCE v2 address: RoCE has no
+  // subnet manager, and the network is what its fabric is.
+  string network = 15;
+  string gid_prefix = 16;
+  // subnet_managers is every subnet manager the fabric's subnet
+  // administrator lists — master and standbys — asked through one port per
+  // fabric, so it is on that port only. It is what keeps a fabric one fabric
+  // through a failover: the master changes, the list does not.
+  repeated SubnetManager subnet_managers = 17;
+  string subnet_managers_error = 18;
+}
+
+// SubnetManager is one subnet manager on a fabric.
+message SubnetManager {
+  string guid = 1;
+  int32 lid = 2;
+  int32 priority = 3;
+  // state is master, standby, discovering or not active.
+  string state = 4;
+}
+
+// IMEX is the nvidia-imex service and its channels.
+message IMEX {
+  // state is systemd's word for it, empty where it is not installed.
+  string state = 1;
+  int32 channels = 2;
+}
+
+// GPUFabricState is one GPU's NVLink domain membership.
+message GPUFabricState {
+  int32 index = 1;
+  string state = 2;
+  string status = 3;
+  string cluster_uuid = 4;
+  string clique_id = 5;
+  string bandwidth = 6;
+}
+
 message Heartbeat {
   string node_id = 1;
   // gpu_memory_used_mb, indexed by GPU, catches a foreign process appearing on
@@ -271,6 +389,27 @@ message Heartbeat {
   // than "nothing to say", as for weights.
   bool provision_changed = 10;
   repeated ProvisionStatus provision = 11;
+
+  // load is how busy the machine is, sampled over the interval since the
+  // previous heartbeat. Absent on the first one, which has nothing to take a
+  // difference from, and from an agent on a system without /proc.
+  NodeLoad load = 12;
+
+  // ib_ports is every InfiniBand or RoCE port as it stands now, re-read on
+  // each heartbeat: a port that goes down after the agent started used to go
+  // unnoticed until the agent restarted.
+  repeated IBPort ib_ports = 13;
+  // fabrics is the fabrics this node can now carry traffic on, sent only when
+  // that changed since registration or the last heartbeat that said so — a
+  // port going down takes its fabric away, and one coming back returns it.
+  bool fabrics_changed = 14;
+  repeated FabricAttachment fabrics = 15;
+
+  // gpu_fabric is each GPU's NVLink domain membership as it stands now, sent
+  // once a minute: a fabric manager restarting drops every card out of its
+  // domain, and the node has to stop being offered work that spans it.
+  repeated GPUFabricState gpu_fabric = 16;
+  IMEX imex = 17;
 }
 
 // Provision is the software a node should have: which engines, at which
@@ -446,3 +585,23 @@ message LogsReply {
   // error says why the lines are absent: an unknown source, say.
   string error = 4;
 }
+
+// FabricScanRequest asks a node to map the fabric one of its ports is on,
+// with ibnetdiscover: every switch, every adapter and every link between
+// them. One node per fabric is asked, every few minutes; a fabric looks the
+// same from all of its ports.
+message FabricScanRequest {
+  string request_id = 1;
+  string hca = 2;
+  int32 port = 3;
+}
+
+// FabricScanReply is ibnetdiscover's output as it printed it. Parsed by the
+// control plane, so a format the parser does not yet know is a control plane
+// upgrade rather than one on every node.
+message FabricScanReply {
+  string request_id = 1;
+  string output = 2;
+  string error = 3;
+  int64 took_ms = 4;
+}

deploy/bootstrap/onboard-node.sh

@@ -35,8 +35,9 @@ usage: $0 --node ID --addr HOST --server URL [options]
   --mgmt-if NAME    interface NCCL bootstraps over (default: the one holding
                     that address)
   --llamacpp-bin P  use this llama-server instead of the one the agent installs
-  --vllm-bin P      path to the vllm entry point (not yet installed by the agent)
-  --sglang-python P python interpreter of the SGLang virtualenv
+  --vllm-bin P      use this vllm instead of the one the agent installs on a
+                    GPU node (also set engines.vllm: "" in server.yaml)
+  --sglang-python P python interpreter of an SGLang virtualenv you installed
   --nvidia-smi PATH where nvidia-smi lives, when it is not on PATH. Containerized
                     driver installs routinely put it somewhere else.
   --ib-root DIR     the sysfs InfiniBand directory (default /sys/class/infiniband)
@@ -226,6 +227,10 @@ ExecStart=/usr/local/bin/llmgw agent --config /etc/llmgw/agent.yaml
 KillMode=process
 Restart=on-failure
 RestartSec=2s
+# RDMA pins the buffers it registers, and NCCL registers a great deal: with
+# systemd's default 8 MB lock limit, an engine that drops the capability that
+# lets root ignore it fails its first collective over InfiniBand.
+LimitMEMLOCK=infinity
 StateDirectory=llmgw-agent
 
 [Install]

deploy/examples/agent.yaml

@@ -47,26 +47,24 @@ heartbeat_interval: 5s
 reconnect_base: 1s
 reconnect_max: 30s
 
-# Fabric labelling is explicit and there is no way around it. OpenSM's default
-# subnet prefix is fe80:: on every fabric, so a matching prefix proves nothing
-# and a wrong guess produces a replica that starts, passes health checks and
-# then hangs on its first all-reduce (plan §5.2).
+# Each fabric this node's adapters are on, by name. The name is checked against
+# what the adapters report — the subnet managers every port on one fabric sees —
+# so two independent fabrics given one name are placed as two and the cluster
+# page says so; give each its own. The rate is measured, not read from here.
 #
 # A node with no fabrics is its own single-node island: it can host single-node
-# replicas and will never be given a multi-node one.
+# replicas and will never be given a multi-node one. docs/fabrics.md has more.
 fabrics:
   - name: ib-a
     hcas: [mlx5_0, mlx5_1, mlx5_2, mlx5_3]
-    rate_gbps: 200
 
 discovery:
   # Worth setting where the driver lives outside the default paths, which
   # containerized driver installs often do.
   nvidia_smi: ""
   ib_root: /sys/class/infiniband
-  # NVML is the only source of NVLink topology, which is what keeps a
-  # tensor-parallel group off the PCIe path inside a node. It needs a binary
-  # built with `-tags nvml`; otherwise this silently falls back to nvidia-smi.
+  # NVLink topology comes from `nvidia-smi topo -m` on every build; a binary
+  # built with `-tags nvml` reads it from NVML instead, when this is set.
   prefer_nvml: true
 
 # Where this node keeps model weights besides /var/lib/llmgw-weights, which is
@@ -96,10 +94,11 @@ discovery:
 engines:
   vllm:
     bin: /opt/venvs/vllm/bin/vllm
-    # Which multi-node mechanism works depends on the vLLM version, and the
-    # flags have changed more than once. Check the version you deploy.
-    launch_mode: ray      # "ray" or "mp"
-    ray_bin: /opt/venvs/vllm/bin/ray
+    # How a replica spanning nodes is launched. mp, the default, is what vLLM
+    # documents today and needs nothing but vLLM; ray is for a vLLM too old
+    # for mp's multi-node flags, and needs Ray on every node.
+    launch_mode: mp       # "mp" or "ray"
+    # ray_bin: /opt/venvs/vllm/bin/ray
   sglang:
     # SGLang is launched as a module, which is its documented entry point.
     python: /opt/venvs/sglang/bin/python

deploy/proxy/Caddyfile

@@ -0,0 +1,41 @@
+# llmgw behind Caddy: the client-facing port (8080) on a public name, with a
+# certificate Caddy obtains and renews by itself.
+#
+# Replace llm.example.com, point its DNS at this machine, open 80 and 443, and
+# run: caddy run --config /etc/caddy/Caddyfile
+#
+# Only 8080 goes through here. 7443 (agents) is mutual TLS with the cluster's
+# own authority and must reach llmgw directly; 9090 and 9091 stay private.
+
+llm.example.com {
+	# The router reads at most 32 MiB of body (router.max_body_bytes). Refusing
+	# more here gives the client a 413 before the upload, not after it.
+	request_body {
+		max_size 32MB
+	}
+
+	reverse_proxy 127.0.0.1:8080 {
+		# Tokens are written the moment they are generated. Without this a
+		# streamed reply can arrive in bursts, and a client measuring time to
+		# first token measures the proxy instead.
+		flush_interval -1
+
+		# Caddy sets X-Forwarded-Proto and X-Forwarded-Host itself and, with
+		# no trusted_proxies, discards whatever the client sent. llmgw relies
+		# on that: the proto decides whether session cookies are Secure, and
+		# the host is the base URL the landing page tells people to use.
+		# Behind a load balancer of your own, name it in trusted_proxies.
+
+		transport http {
+			# A long generation can be silent for a while before the first
+			# token. llmgw has its own limits (timeouts.*); this must not be
+			# the shorter one.
+			response_header_timeout 15m
+		}
+	}
+
+	# No `encode` on purpose: compressing a stream buffers it.
+
+	# Access log to stderr, which under systemd is the journal.
+	log
+}

deploy/proxy/README.md

@@ -0,0 +1,39 @@
+# A reverse proxy in front of llmgw
+
+The client-facing port (8080) is plain HTTP. On anything but a trusted network,
+put it behind a proxy that holds a publicly trusted certificate.
+
+| File | For |
+| --- | --- |
+| [Caddyfile](Caddyfile) | Caddy — obtains and renews its own certificate; the least work |
+| [nginx.conf](nginx.conf) | nginx, with a certificate from certbot |
+
+Both are also attached to every release.
+
+The `/about` page shows both, from copies in `internal/web/proxy`; a test fails
+when those differ from these, so change them here and copy them over.
+
+Four things both files get right, and any other proxy has to as well:
+
+- **Streaming is not buffered.** Replies are server-sent events; a buffering
+  proxy delivers them in one burst at the end.
+- **The read timeout is longer than a model's silence.** nginx's default of 60
+  seconds turns a slow first token into a 504.
+- **`X-Forwarded-Proto` and `X-Forwarded-Host` are set by the proxy**, replacing
+  whatever the client sent. llmgw uses the first to mark session cookies
+  `Secure` and the second for the base URL on its landing page.
+- **The body limit matches llmgw's** (32 MiB, `router.max_body_bytes`), so an
+  oversized request is refused before it is uploaded.
+
+Only port 8080 is proxied. Agents reach 7443 directly — it is mutual TLS with
+the cluster's own authority, and a proxy terminating it would take away the
+node identity it carries. 9090 (admin, pprof) stays on loopback; 9091
+(metrics) goes to your monitoring host only.
+
+With a proxy on the same machine, bind llmgw to loopback so nothing reaches it
+around the proxy:
+
+```yaml
+# /etc/llmgw/server.yaml
+listen: "127.0.0.1:8080"
+```

deploy/proxy/nginx.conf

@@ -0,0 +1,63 @@
+# llmgw behind nginx: the client-facing port (8080) on a public name.
+#
+# Put this in /etc/nginx/sites-available/llmgw (or conf.d/llmgw.conf), replace
+# llm.example.com, and get a certificate — certbot --nginx -d llm.example.com
+# fills in the ssl_ lines below. Then: nginx -t && systemctl reload nginx
+#
+# Only 8080 goes through here. 7443 (agents) is mutual TLS with the cluster's
+# own authority and must reach llmgw directly; 9090 and 9091 stay private.
+
+upstream llmgw {
+    server 127.0.0.1:8080;
+    keepalive 32;
+}
+
+server {
+    listen 80;
+    listen [::]:80;
+    server_name llm.example.com;
+    return 301 https://$host$request_uri;
+}
+
+server {
+    # `listen ... http2` rather than `http2 on;`: the latter needs nginx
+    # 1.25.1, and Ubuntu 22.04 and Debian 12 ship older ones. Newer nginx
+    # accepts this form with a deprecation warning.
+    listen 443 ssl http2;
+    listen [::]:443 ssl http2;
+    server_name llm.example.com;
+
+    ssl_certificate     /etc/letsencrypt/live/llm.example.com/fullchain.pem;
+    ssl_certificate_key /etc/letsencrypt/live/llm.example.com/privkey.pem;
+
+    # The router reads at most 32 MiB of body (router.max_body_bytes).
+    client_max_body_size 32m;
+
+    location / {
+        proxy_pass http://llmgw;
+        proxy_http_version 1.1;
+        proxy_set_header Connection "";
+
+        # Set, never appended: llmgw decides from X-Forwarded-Proto whether
+        # session cookies are Secure, and builds the base URL the landing page
+        # shows from X-Forwarded-Host. A value passed through from the client
+        # would be the client's choice.
+        proxy_set_header Host              $host;
+        proxy_set_header X-Forwarded-Host  $host;
+        proxy_set_header X-Forwarded-Proto $scheme;
+        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
+
+        # Streamed replies: tokens go out as they are generated. Buffering
+        # turns a stream into one late burst, and gzip buffers too.
+        proxy_buffering off;
+        proxy_cache off;
+        gzip off;
+
+        # nginx's default is 60 seconds between two reads from upstream. A
+        # large model can think longer than that before its first token, and
+        # the client then sees a 504 for a request that was about to succeed.
+        # llmgw has its own limits (timeouts.*); this must not be the shorter.
+        proxy_read_timeout 15m;
+        proxy_send_timeout 15m;
+    }
+}

deploy/systemd/llmgw-agent.service

@@ -11,6 +11,10 @@ ExecStart=/usr/local/bin/llmgw agent --config /etc/llmgw/agent.yaml
 KillMode=process
 Restart=on-failure
 RestartSec=2s
+# RDMA pins the buffers it registers, and NCCL registers a great deal: with
+# systemd's default 8 MB lock limit, an engine that drops the capability that
+# lets root ignore it fails its first collective over InfiniBand.
+LimitMEMLOCK=infinity
 StateDirectory=llmgw-agent
 
 [Install]

docs/choosing-models.md

@@ -141,7 +141,7 @@ not count the same free terabyte once per node.
 Set budgets and limits on day one, not later:
 
 ```
-llmgw keys limits --scope project --id <project> --rpm 60 --budget 50000000
+llmgw keys limits --scope project --id <project> --rpm 60 --budget 50
 ```
 
 The reason is not technical. A resource that appears free and unlimited becomes,

docs/fabrics.md

@@ -0,0 +1,177 @@
+# InfiniBand fabrics and NVLink
+
+For a site whose GPU nodes are joined by InfiniBand or RoCE, by NVLink, or both
+— several independent fabrics, NVL72 racks, eight-GPU HGX boards. It says what
+llmgw does with them, what it needs from you before it can, how to check a node
+in one command, and where to look when a replica spanning nodes will not run.
+
+Read [install.md](install.md) first; this is the part of it that only matters
+once there is a fabric.
+
+> **Where this stands.** Every piece here was written against the real formats
+> of `sminfo`, `saquery`, `ibnetdiscover`, `nvidia-smi topo -m` and
+> `nvidia-smi -q`, and tested against them — but none of it has yet run on a
+> site with the hardware. Run `llmgw doctor --json` on two or three nodes first
+> and keep the reports: if a parser meets a format it does not know, that report
+> is what fixes it, and the fix is a control plane upgrade, not one per node.
+
+## What llmgw does with them
+
+**Two kinds of interconnect.** InfiniBand or RoCE joins nodes (scale-out).
+NVLink joins GPUs — inside one machine on an HGX board, or across the nodes of a
+rack on an NVL72 or an NVLink Switch system (scale-up). A replica spanning nodes
+is placed, in order of preference:
+
+1. inside one **multi-node NVLink domain**, if any has enough usable nodes:
+   tensor parallelism between its nodes then runs over NVLink;
+2. on one **physical InfiniBand or RoCE fabric**, preferring nodes under one
+   leaf switch and putting nodes with a degraded cable last.
+
+It is never split across two fabrics or two NVLink domains. Within a node, the
+GPUs a shard gets are the set with the most NVLink pairs, then the nearest to
+the fabric's adapters, then on one NUMA node.
+
+**A fabric has a name and an identity.** The name is what `agent.yaml` says; the
+identity is what the adapters report — on InfiniBand the subnet managers every
+port on one fabric sees, on RoCE the IP network of its RoCE v2 address. The name
+is how you address a fabric; the identity is how llmgw catches the name being
+wrong: two independent fabrics given one name are placed as two, and the
+cluster page says so.
+
+## Before a node joins
+
+On every GPU node:
+
+| What | Why | Check |
+|---|---|---|
+| NVIDIA driver | everything | `nvidia-smi` |
+| `nvidia-fabricmanager`, on an NVSwitch machine | without it CUDA does not start on those GPUs | `systemctl is-active nvidia-fabricmanager` |
+| `rdma-core`, `ib_uverbs` loaded | user-space RDMA at all | `ls /sys/class/infiniband`, `lsmod \| grep ib_uverbs` |
+| `nvidia_peermem` loaded | GPUDirect RDMA; without it NCCL copies every transfer through host memory | `lsmod \| grep nvidia_peermem` |
+| `infiniband-diags` | tells llmgw which fabric a port is on; the agent installs it if missing | `which sminfo` |
+
+On an NVL72 or NVLink Switch system, also:
+
+| What | Why | Check |
+|---|---|---|
+| `nvidia-imex` running, every node of the domain in `/etc/nvidia-imex/nodes_config.cfg` | GPUs in different nodes share memory through it | `systemctl is-active nvidia-imex` |
+| an IMEX channel (`options nvidia NVreg_CreateImexChannel0=1`) | an engine opens one | `ls /dev/nvidia-caps-imex-channels` |
+| every GPU joined to its domain | the fabric manager does this | `nvidia-smi -q \| grep -A6 Fabric` shows `State: Completed` |
+
+On each InfiniBand fabric:
+
+- **A subnet manager, and a standby.** Without one, ports come up to `INIT` and
+  stay there; without a standby, the fabric keeps its routes if the master
+  stops but cannot recover from any change after. llmgw watches this but does
+  not run one.
+- **Its own name** in `agent.yaml` on every node attached to it. Optionally, a
+  distinct `subnet_prefix` in each subnet manager's configuration, which makes
+  GIDs unique across fabrics; llmgw does not need it.
+
+## Declaring the fabrics
+
+In `/etc/llmgw/agent.yaml`, the adapters on each fabric, by name:
+
+```yaml
+fabrics:
+  - name: ib-compute
+    hcas: [mlx5_0, mlx5_1, mlx5_2, mlx5_3, mlx5_6, mlx5_7, mlx5_8, mlx5_9]
+  - name: ib-storage
+    hcas: [mlx5_4]
+```
+
+or `--fabric ib-compute:mlx5_0,mlx5_1,…` when onboarding ([node-onboarding.md](node-onboarding.md)).
+An adapter belongs to one fabric; a node may be on several. A replica on
+`ib-compute` is given only `ib-compute`'s adapters (`NCCL_IB_HCA`).
+
+The rate is measured, not taken from the file: a fabric's rate on a node is the
+slowest of its active adapters, because a collective runs at the speed of its
+slowest link.
+
+The agent checks the declaration against the adapters on every heartbeat. An
+adapter that is missing, or whose port is down, takes that fabric off the node
+within seconds — the node stops being offered multi-node work on it — and
+returns it when the port comes back.
+
+## Checking a node: `llmgw doctor`
+
+```
+sudo llmgw doctor                 # uses /etc/llmgw/agent.yaml when present
+sudo llmgw doctor --json > $(hostname)-doctor.json
+```
+
+It reads the node and says what is wrong and how to fix it, most serious first,
+and exits non-zero when anything fails:
+
+```
+FAIL  nvlink  this machine has NVSwitch but nvidia-fabricmanager is inactive: CUDA will not initialise on its GPUs
+              → systemctl enable --now nvidia-fabricmanager (its version must match the driver's)
+WARN  fabric  nvidia_peermem is not loaded: GPUDirect RDMA is off, and NCCL copies every transfer through host memory
+              → modprobe nvidia_peermem
+OK    gpu     8 GPUs: 8× NVIDIA H100 80GB HBM3
+OK    nvlink  NVLink groups: [0 1 2 3 4 5 6 7]
+OK    fabric  mlx5_0/1 ACTIVE at 400 Gb/s (InfiniBand), subnet manager 0x248a07030030f0c0 (LID 1, SMINFO_MASTER)
+OK    fabric  subnet managers of 0x248a07030030f0c0's fabric: 0x248a07030030f0c0 (master, priority 15), 0x248a07030030f1d4 (standby, priority 14)
+OK    agent   fabric ib-compute: mlx5_0,mlx5_1,… usable at 400 Gb/s
+OK    agent   NCCL bootstraps over ens3 (10.1.0.21)
+```
+
+followed by each GPU's NVLink peers and nearest adapter, each port's state,
+rate and errors, and the NCCL environment a shard on each fabric would be given.
+`--json` writes every fact it read as well as what it concluded — send that
+file when the report is not enough.
+
+## Watching them
+
+- **Cluster page, Fabrics.** Each declared fabric and what it physically is;
+  a table of subnet managers per fabric — master, priority, standbys, the last
+  change of master — and every disagreement: one name on two fabrics, two names
+  on one, an adapter up but undeclared, ports up with no subnet manager, a
+  fabric with no standby, a recent failover.
+- **`/admin/fabrics`.** Each fabric's map, from `ibnetdiscover` run by one of
+  its nodes every five minutes: switches in their tiers, adapters under their
+  leaves, cables coloured by rate, degraded ones in red, rail alignment, and
+  the cluster's own adapters live with their traffic. Click anything for its
+  detail.
+- **`/admin/nvlink`.** Each multi-node NVLink domain as a rack of trays: which
+  GPUs are joined, which node's IMEX is down, which link runs degraded.
+- **A node's page.** Its ports with state, rate, errors and what they are
+  physically on; its GPU-by-GPU NVLink matrix; its load.
+- **Metrics**, for alerts:
+
+| Metric | Alert when |
+|---|---|
+| `llmgw_ib_port_active` | a declared port is 0 |
+| `llmgw_ib_port_counter_total{counter="symbol_error"}` and `link_downed` | rising |
+| `llmgw_ib_port_rate_gbps` | below its usual rate |
+| `llmgw_fabric_subnet_managers{state="standby"}` | absent or 0 |
+| `llmgw_fabric_master_sm` | its `sm_guid` changes (a failover) |
+| `llmgw_fabric_problems{level="fail"}` | above 0 |
+| `llmgw_gpu_nvlink_joined` | 0 |
+| `llmgw_imex_up` | 0 |
+
+## What an engine is given
+
+For a shard on an InfiniBand or RoCE fabric:
+
+| Variable | Value |
+|---|---|
+| `NCCL_IB_HCA` | that fabric's adapters on this node, and no others |
+| `NCCL_IB_DISABLE` | `0` (`1` for a single-node shard) |
+| `NCCL_SOCKET_IFNAME`, `GLOO_SOCKET_IFNAME` | the node's management interface: `mgmt_interface`, or the one holding its management address |
+| `NCCL_IB_GID_INDEX` | on RoCE only: the RoCE v2 entry with an IPv4 address |
+| `NCCL_MNNVL_ENABLE`, `NCCL_CUMEM_ENABLE` | `1`, for a replica inside a multi-node NVLink domain |
+
+A profile's `env` overrides any of them. vLLM launches a multi-node replica
+with its own multiprocessing (`engines.vllm.launch_mode: mp`, the default):
+every node runs `vllm serve` with its rank and the leader's address, the
+workers headless. `launch_mode: ray` is there for a vLLM too old for that.
+
+## When a replica spanning nodes will not run
+
+| It | Look at |
+|---|---|
+| is not placed | `llmgw ctl explain-placement <model>`: a fabric split by identity, a node whose IMEX is down, a node refused because its adapters are cabled to two fabrics |
+| starts and hangs on its first collective | the cluster page's fabric problems; `doctor` on each node — a port down, the wrong interface, RoCE without a v2 GID |
+| runs, slowly | `nvidia_peermem`; the map for a degraded cable; the node page's NVLink matrix for a group that crossed PCIe |
+| worked yesterday | the subnet manager table for a failover; `link_downed` on the node's ports |

docs/install.md

@@ -1,7 +1,8 @@
 # Installing llmgw from nothing
 
 Written for whoever is standing up the cluster: you have servers, you have root
-on them, and nothing is installed yet. By the end there is a control plane, a
+on them, and nothing is installed yet. For only the commands, read
+[quick-install.md](quick-install.md) first; this page explains them. By the end there is a control plane, a
 node in the fleet, and a model answering requests.
 
 It assumes Debian or Ubuntu and adjusts nothing for other distributions beyond
@@ -104,6 +105,21 @@ That is the whole step. In order, it:
    and promoted to administrator — the password is generated and **printed
    once**
 6. installs the systemd unit, starts it, and waits until `/health` answers
+7. runs a model mirror beside it (`/srv/models`, bytes on `:8081`, its
+   management API on loopback) and points the control plane at it
+8. makes this machine a node of its own control plane — on its GPUs when
+   `nvidia-smi` finds any, on CPU otherwise — and waits for it to install its
+   engine
+9. fetches Qwen2.5 0.5B into the mirror and serves it on this node, so the
+   first thing anyone opens `/chat` to find is a model that answers
+
+`--no-mirror`, `--no-node` and `--no-model` leave the last three out. An NVIDIA
+card with no working driver is reported rather than silently ignored; install
+the driver and run init again to serve on it.
+
+Add `--postgres` to keep accounts and keys in PostgreSQL rather than SQLite:
+init installs it and sets it up, and on a machine already running on SQLite it
+moves everything across. See [PostgreSQL](#postgresql-when-sqlite-is-no-longer-the-right-shape).
 
 Every step looks before it acts. Running it again finishes an install that
 stopped halfway and changes nothing on one that did not; it never overwrites
@@ -128,8 +144,10 @@ If this machine is exposed, move the directory elsewhere once the nodes are in
 and pass `--pki` to onboarding.
 
 This covers the fleet-facing port only. The client-facing API is public and
-belongs behind an ordinary publicly trusted certificate, usually terminated by a
-reverse proxy in front of port 8080.
+belongs behind an ordinary publicly trusted certificate, terminated by a
+reverse proxy in front of port 8080. [deploy/proxy/](../deploy/proxy/) has a
+Caddyfile and an nginx configuration that get streaming, timeouts and the
+forwarded headers right; both are attached to every release.
 
 ### Check it
 
@@ -301,32 +319,26 @@ hardware to it.
 
 ### NVLink, if your GPUs have it
 
-The published binary cannot see NVLink topology. NVML is the only source of it,
-its bindings need cgo, and the portable binary is built without cgo on purpose
-so it runs on a node whose libc is older than the builder's.
+The agent reads the topology from `nvidia-smi topo -m`: which cards NVLink
+joins, each card's NUMA node, and which network adapter is nearest each. A
+tensor-parallel group is then the set of cards with the most NVLink between
+them — on a PCIe box where bridges pair specific cards, a group straddling two
+bridges falls back to PCIe and runs several times slower, and nothing else
+would report it.
 
-Without that topology, a tensor-parallel group takes the lowest free GPU
-indices. On an SXM board with NVSwitch that is correct, because every card is
-equidistant. On a PCIe box where NVLink bridges pair specific cards, a group
-straddling two bridges falls back to PCIe and runs several times slower — and
-nothing reports it, because from the outside it is simply a replica that is
-slow.
-
-For those nodes, install the `-nvml` build published beside the portable one:
+On an NVSwitch machine, `nvidia-fabricmanager` has to be running, at the
+driver's version, or CUDA will not start on its GPUs. On an NVL72 rack or an
+NVLink Switch system, the GPUs of several nodes form one domain, and a replica
+can run tensor parallelism across its nodes — which needs `nvidia-imex` and an
+IMEX channel as well. [fabrics.md](fabrics.md) has the list, and `llmgw doctor`
+checks it:
 
 ```
-sudo install -m0755 llmgw-$VERSION-linux-amd64-nvml /usr/local/bin/llmgw
+sudo llmgw doctor
 ```
 
-It is dynamically linked, so it needs a libc no older than the builder's and the
-NVIDIA driver present. Check it took effect with:
-
-```
-llmgw ctl nodes --json | grep -c nvlink_peers    # 0 means the topology is unknown
-```
-
-`llmgw_gpu_nvlink_peers` in the metrics says the same thing across a fleet: a
-flat zero everywhere means the topology is unknown rather than absent.
+A `-nvml` build is still published beside the portable one; it reads NVLink
+from NVML directly, and needs a libc no older than the builder's.
 
 ### InfiniBand, if you have it
 
@@ -336,48 +348,60 @@ ibstat            # must list your HCAs with State: Active
 ls /sys/class/infiniband
 ```
 
-The agent checks what you declare against what the node has. An adapter that is
-not there, or that has no ACTIVE port — physically linked but with no subnet
-manager, which survives a reboot and looks like a software problem — takes that
-fabric out of service on this node, loudly, in the log. Without that check the
-replica starts, passes its health checks, and hangs on its first all-reduce.
+Each fabric needs a subnet manager, and should have a standby; `nvidia_peermem`
+should be loaded for GPUDirect RDMA. You name each fabric when you onboard the
+node, and llmgw checks the name against what the adapters report: every port on
+one fabric sees the same subnet managers, so two independent fabrics given one
+name are placed as two, and the cluster page says so. An adapter that is not
+there, or whose port goes down, takes that fabric off the node within seconds,
+and gives it back when the port returns.
 
-Fabric membership is configuration, not discovery, and there is no way around
-that: OpenSM's default subnet prefix is the same on every fabric, so a matching
-prefix proves nothing and a wrong guess produces a replica that starts, passes
-health checks, and then hangs on its first all-reduce. You name the fabric when
-you onboard the node, and two nodes are only in the same fabric if you said so.
+[fabrics.md](fabrics.md) covers declaring several fabrics, what the cluster and
+fabric pages show, and what to look at when a replica spanning nodes will not
+run. Run `llmgw doctor` on the node before it joins.
 
-### An engine
+### Engines
 
-Pick what you will actually serve. For a first cluster, llama.cpp is the least
-trouble and runs without a GPU.
+Nothing to do: the agent installs them once the node joins, under
+`/opt/llmgw/engines`, and checks each one runs before the node offers it.
 
-```
-# llama.cpp, from a release build
-sudo apt-get install -y libgomp1          # it will not start without this
-sudo mkdir -p /opt/llama.cpp
-# unpack a llama.cpp release into /opt/llama.cpp
-/opt/llama.cpp/llama-server --version
+| Engine | On | How |
+|---|---|---|
+| llama.cpp | every node | the release build for its processor, CUDA where there is a GPU, checked against GitHub's checksum |
+| vLLM | nodes with an NVIDIA GPU it can run on | a Python environment built by uv from PyPI, about 8 GB |
+| SGLang | only when you ask for it | the same way as vLLM |
+
+vLLM is resolved as PyPI stood a day after that vLLM was published, so every
+node gets the same dependencies whenever it is built, and installed with
+`--require-hashes` against the sha256 PyPI publishes. The resolved list stays in
+the environment as `llmgw-requirements.txt`. It needs:
+
+- **driver 580 or newer and a Turing GPU or newer** (compute capability 7.5):
+  vLLM is built for CUDA 13. A V100 cannot run it;
+- **about 25 GB free** under the engines directory while it installs;
+- **gcc**, which Triton builds its kernel launcher with. The agent installs it.
+
+A node with no GPU skips vLLM and says nothing about it. A node that has a GPU
+but cannot run it — an old driver, a V100, a small disk — says why on the
+cluster page. `llmgw doctor` checks the same before the node joins.
+
+The versions are the control plane's, in `server.yaml`:
+
+```yaml
+engines:
+  vllm: "0.30.0"      # the default; "" leaves vLLM to you
+  sglang: "0.5.20"    # SGLang only when named
 ```
 
-That `libgomp1` line is not padding. A `llama-server` binary missing it exists,
-is executable, and cannot start — and the node reports `ready` throughout.
+A changed version reaches every node when it next connects. To use an engine
+you installed yourself instead, set its version to `""` and name its path in
+the node's `agent.yaml` (`engines.vllm.bin`, `engines.sglang.python`).
 
-For vLLM, a virtualenv on the node:
+After changing a version, check it still takes the flags llmgw builds for it,
+which drift between releases:
 
 ```
-sudo apt-get install -y python3-venv
-sudo python3 -m venv /opt/venvs/vllm
-sudo /opt/venvs/vllm/bin/pip install vllm
-/opt/venvs/vllm/bin/vllm --version
-```
-
-Check the engine still takes the flags llmgw builds for it, which drift between
-releases:
-
-```
-llmgw ctl engine-check vllm --bin /opt/venvs/vllm/bin/vllm
+llmgw ctl engine-check vllm --bin /opt/llmgw/engines/vllm/0.30.0-cu13/bin/vllm
 ```
 
 ### Weights
@@ -464,40 +488,44 @@ bf16 is 141 GB and the same model at q4 is 43 GB.
 
 ## 5. Adding the node
 
-Two halves, because they need different access: building the machine needs the
-hypervisor, and issuing its identity needs the certificate authority. Where one
-person holds both — the common case — one command does it:
+On the control plane, which holds the certificate authority:
 
 ```
-llmgw-add-node \
-  --name llmgw-node01 --ip 10.0.0.141/24 \
-  --server https://10.0.0.140:7443 \
-  --cores 16 --memory 131072 --disk 200 \
-  --fabric ib-a:mlx5_0,mlx5_1 \
-  --llamacpp-bin /opt/llama.cpp/llama-server
+sudo llmgw node add ubuntu@10.0.0.141 --fabric ib-a:mlx5_0,mlx5_1
 ```
 
-**The VM name, the hostname, the node id and the certificate's common name are
-all the same string.** They do not have to be, and every time they are not,
-somebody spends an afternoon working out which of the four a message refers to.
+It logs in to the machine over SSH as that user, who needs passwordless sudo
+there, and:
 
-For a machine that already exists — bare metal, or a VM you built yourself — skip
-provisioning and issue the identity only:
+1. installs llmgw: the release for the machine's architecture, downloaded by
+   the machine and checked against the release's checksums, or this machine's
+   copy when it cannot download;
+2. runs `llmgw doctor` there and stops on any problem, printing the fix. Run it
+   again once fixed, or add `--ignore-checks`;
+3. issues the node's certificate and installs it, with the cluster's root,
+   trusted system-wide;
+4. writes `agent.yaml` — only what the node cannot find for itself — and the
+   systemd unit, and starts the agent;
+5. waits for the node to join and to install llama.cpp (`--wait` also waits
+   for vLLM, `--no-wait` for neither).
 
-```
-llmgw-onboard-node \
-  --node llmgw-node01 --addr 10.0.0.141 \
-  --server https://10.0.0.140:7443 \
-  --fabric ib-a:mlx5_0,mlx5_1 \
-  --llamacpp-bin /opt/llama.cpp/llama-server
-```
+In a terminal it draws this as a checklist and asks what a script cannot: which
+fabric each group of the node's active adapters is on, when `--fabric` was not
+given, and whether to go on past a problem doctor found.
 
-Onboarding needs SSH from the CA host to the node. It waits for SSH rather than
-failing on it, and learns the host key of a machine it has not seen — a new
-machine spends its first minute regenerating host keys and restarting sshd.
+**The node id is the machine's hostname**, and the address its agent dials is
+this machine's on the route to it; `--node` and `--server` override them. The
+VM name, the hostname, the node id and the certificate's common name being the
+same string saves an afternoon each time they would differ.
 
-It also checks that a configured engine actually runs, and says so loudly when
-it does not. That check exists because the failure is otherwise invisible.
+**SSH runs as root under sudo**, with root's keys. To use yours, log in to the
+control plane with `ssh -A` and run `sudo --preserve-env=SSH_AUTH_SOCK llmgw node
+add …`. It waits for SSH rather than failing on it, and learns the host key of
+a machine it has not seen — a new machine spends its first minute regenerating
+host keys and restarting sshd.
+
+Running it again on a node is safe: a fresh certificate and configuration, and
+the engines it runs are adopted by the restarted agent.
 
 Within a few seconds:
 
@@ -741,6 +769,20 @@ turn, which is the first per-request load this store has had; and SQLite takes
 one writer at a time, so those turns serialise. And a second control plane is
 impossible while the store is a file on one machine's disk.
 
+On the control plane, one command does all of it — installs PostgreSQL if the
+machine has none, creates its role and database with a generated password kept
+in `/etc/llmgw/secrets/store-dsn`, stops the gateway, copies the SQLite store
+into it so every account and key survives, points `server.yaml` at it, and
+starts the gateway again:
+
+```
+sudo llmgw server init --postgres
+```
+
+It can be given on the first run too, and then there is never a SQLite store.
+The rest of this section is what that command does, for a PostgreSQL somewhere
+else.
+
 The setting is the same one, and the address decides:
 
 ```yaml
@@ -1017,6 +1059,8 @@ requests. On a slow model, set `timeouts.generation` higher.
   and what belongs on which port
 - `deploy/monitoring/README.md` — Prometheus, Grafana and the alerts, with what
   each one assumes about this system
+- `docs/fabrics.md` — InfiniBand fabrics and NVLink: what llmgw needs of them,
+  checking a node, and what to look at when a replica spanning nodes will not run
 - `docs/node-onboarding.md` — adding nodes in more detail, and what onboarding
   deliberately does not do
 - `docs/onboarding.md` — giving people accounts and keys
@@ -1032,14 +1076,16 @@ requests. On a slow model, set `timeouts.generation` higher.
   (step 2) or every request costs zero, and a budget is something the dashboard
   displays rather than something that stops anything. The quotas page says so
   where it can, which is not the same as being told before you rely on it.
-- **Engines are yours to install.** Onboarding verifies one runs; it installs
-  nothing.
+- **vLLM needs CUDA 13.** The version the agent installs needs driver 580
+  and a Turing GPU or newer. On older hardware, set `engines.vllm` to `""` and
+  install a vLLM built for your CUDA yourself.
 - **Two control planes have not been run for long.** The leader election and
   agent failover work and there is a pattern in `deploy/ha/`, but every cluster
   this has been run on in anger has had one. A single control plane is the
   well-trodden path.
-- **Multi-node replicas have not been run on real InfiniBand.** The scheduling,
-  the fabric invariant and the rendezvous are exercised against simulated
-  hardware only. Single-node placement on real GPUs is the well-trodden path;
-  if you are the first to run a TP group across real cards, expect to find
-  something.
+- **Multi-node replicas have not been run on real InfiniBand or NVLink.** The
+  scheduling, the fabric identity, the maps and the launch are written against
+  the real tools' formats and tested against them, not yet run on hardware that
+  has them. Single-node placement on real GPUs is the well-trodden path; if you
+  are the first to run a TP group across real cards, run `llmgw doctor --json`
+  on each node first and keep the reports.

docs/node-onboarding.md

@@ -4,24 +4,16 @@ Written for whoever runs the cluster.
 
 ## The short version
 
-Where one person holds both the hypervisor and the certificate authority —
-which is the common case — one command does the whole thing in about
-45 seconds:
+On the control plane:
 
 ```
-llmgw-add-node \
-  --name llmgw-node05 --ip 10.0.0.145/24 \
-  --server https://10.0.0.140:7443 \
-  --cores 8 --memory 32768 --disk 60 \
-  --llamacpp-bin /opt/llama.cpp/llama-server
+sudo llmgw node add ubuntu@10.0.0.145
 ```
 
-The VMID defaults to the last octet of the address, so `qm list` and
-`llmgw ctl nodes` stay in the same order. `--provision-only` and
-`--onboard-only` split it back into the two halves below when the two kinds
-of access belong to different people.
-
-The rest of this page is what that command does, and what it still does not.
+That is all a machine that already exists needs — bare metal, or a VM you
+built. [install.md](install.md#5-adding-the-node) says what it does. On
+Proxmox, the machine itself can be built first by the script below, on the
+hypervisor.
 
 ## The two halves
 
@@ -54,41 +46,31 @@ their working tree.
 
 ## 2. The identity
 
-On the host that holds the certificate authority — this is the part that cannot
-be delegated, because it issues the credential by which the control plane
-believes the node is who it says it is:
+On the control plane, which holds the certificate authority — this is the
+part that cannot be delegated, because it issues the credential by which the
+control plane believes the node is who it says it is:
 
 ```
-llmgw-onboard-node \
-  --node llmgw-node04 \
-  --addr 10.0.0.144 \
-  --server https://10.0.0.140:7443 \
-  --fabric ib-b
+sudo llmgw node add ilker@10.0.0.144 --fabric ib-b:mlx5_0,mlx5_1
 ```
 
-It issues a client certificate whose common name is the node id, installs it
-along with the CA root, trusts that root system-wide so other tooling on the
-machine stops treating llmgw's certificates as strangers, installs the binary
-and the unit, and starts the agent. The node appears in `llmgw ctl nodes` within
-a few seconds.
+It checks the machine with `llmgw doctor`, issues a client certificate whose
+common name is the node id (the hostname), installs it along with the CA root,
+trusts that root system-wide, installs the binary and the unit, starts the
+agent, and waits for the node to join.
 
-If it does not, the control plane's log says why. A certificate whose common
-name does not match the claimed node id is refused by design — that check is
-what makes a node's identity something it proves rather than something it picks.
+If it does not join, the agent's log on the node says why. A certificate whose
+common name does not match the claimed node id is refused by design — that
+check is what makes a node's identity something it proves rather than
+something it picks.
 
-**It needs ssh from the CA host to the node.** Provisioning copies that host's
-`authorized_keys` into the new machine, so a node built by the script above
-already satisfies it; an older machine may need the key added.
+**It needs SSH from the control plane to the node.** Provisioning copies the
+hypervisor's `authorized_keys` into the new machine; with `ssh -A` to the
+control plane and `sudo --preserve-env=SSH_AUTH_SOCK`, the same key that
+reaches the hypervisor reaches the node.
 
-It waits for ssh rather than failing on it, and learns the host key of a machine
-it has not seen (`accept-new`, which still refuses a *changed* key — the risk
-being guarded against is a substituted machine, not a new one). Both are needed
-to run the halves back to back: for the first minute of its life a new machine
-is still regenerating host keys and restarting sshd, so it refuses connections
-it accepted a moment earlier.
-
-It also checks that a configured engine actually runs, which is cheap here and
-expensive later — see below.
+`deploy/bootstrap/onboard-node.sh` is the script this replaced, for a machine
+holding an authority but not a current llmgw.
 
 ## The CPU model matters, and its default is wrong
 
@@ -116,32 +98,23 @@ The trade is live migration: a VM pinned to the host CPU cannot move to a
 machine with a different one. For a node whose job is arithmetic that is the
 right way round, and `--cpu` overrides it where it is not.
 
-## What onboarding does not do
+## Engines
 
-It gives a machine an identity and an agent. It does **not** install inference
-engines or model weights, and a node missing either looks healthy in
-`llmgw ctl nodes` right up until a replica is placed on it and the shard dies
-with "executable file not found".
+The script installs none, and does not need to: once the agent is connected,
+the control plane tells it which engines to have and it installs them itself
+— llama.cpp on every node, vLLM on every node with an NVIDIA GPU it can run
+on, SGLang where the control plane names a version. Each is checked to run
+before the node offers it, so a node never advertises an engine that cannot
+start. [install.md](install.md#engines) says what vLLM needs of the node.
 
-Onboarding does check that an engine you point it at can be executed, and says
-so loudly when it cannot. That check exists because the failure is invisible
-otherwise: `llama-server` on node05 was present, executable, and could not start
-for want of `libgomp1`, and the node reported `ready` in `llmgw ctl nodes` the
-whole time. Being told at onboarding beats finding out when a replica is placed
-and the reason is three logs away.
+`--llamacpp-bin`, `--vllm-bin` and `--sglang-python` are for an engine you
+installed yourself. For the agent to use yours rather than install its own,
+also set that engine's version to `""` under `engines` in `server.yaml`.
 
-So the node is not finished until:
-
-- the engines its catalog profiles name are installed (`vllm`, `sglang`,
-  `llama-server`), at the paths the agent configuration gives
-- the weights are on disk, at the `source` each model declares
-
-Both are deliberately outside this script. Engines are a platform decision with
-versions and drivers attached, and distributing weights is its own problem —
-hundreds of gigabytes that want a shared filesystem or a cache, not an scp from
-a laptop. llmgw will fetch weights for a model whose `source` is a URL, one
-copy per node, which covers the simple case; anything larger stays the
-operator's arrangement, and llmgw is content to find the files already there.
+Weights are the other thing a node needs. llmgw fetches them for a model whose
+`source` is a URL, one copy per node, which covers the simple case; anything
+larger — a shared filesystem, a cache — stays the operator's arrangement, and
+llmgw is content to find the files already there.
 
 The `sim` engine needs neither, which is what makes it useful for exercising
 everything else.
@@ -178,18 +151,24 @@ was cut short.
 
 ## Fabrics
 
-`--fabric` is how a node says which InfiniBand fabric it is attached to, and
-there is no way around stating it. OpenSM's default subnet prefix is `fe80::` on
-every fabric, so a matching prefix proves nothing and a wrong guess produces a
-replica that starts, passes health checks, and then hangs on its first
-all-reduce.
+`--fabric` is how a node says which InfiniBand fabric it is attached to. The
+name is checked against what the adapters report — every port on one fabric
+sees the same subnet managers — so two independent fabrics given one name are
+placed as two, and the cluster page says so; give each fabric its own name.
+The agent installs `infiniband-diags`, which that check needs, if it is missing.
+
+Before a node with a fabric joins, run `sudo llmgw doctor` on it: it checks the
+subnet manager, GPUDirect RDMA, the fabric manager and IMEX on NVLink machines,
+and the declared fabrics against the adapters. [fabrics.md](fabrics.md) has the
+rest.
 
 Omit it for a node with no fabric. Such a node is its own island: it can host
 single-node replicas and will never be given a multi-node one.
 
 Two adapters per fabric is the default; `--fabric ib-b:mlx5_2,mlx5_3` names them
 explicitly. An adapter must never appear under two fabrics — the agent refuses
-that configuration, because the consequence is the hang above.
+that configuration: a replica given adapters on two fabrics starts, passes its
+health checks, and hangs on its first all-reduce.
 
 ## Where the driver is
 

docs/onboarding.md

@@ -39,16 +39,40 @@ The landing page then links to it, and the rest is:
 
 ```
 llmgw users list --pending --store …
-ID                    EMAIL              NAME   STATE    PROJECT  SIGNED UP
-usr-k4mq…             ayse@example.com   Ayşe   pending  -        4m12s ago
+ID          EMAIL              STATE    ADMIN  MEMBER OF     SIGNED UP
+usr-k4mq…   ayse@example.com   pending  -      on their own  4m12s ago
 
-llmgw users approve usr-k4mq… --project default --by ilker --store …
+llmgw users approve ayse@example.com --by ilker --store …
+llmgw users approve ayse@example.com --project Acme/ml/training --store …   # or straight into a project
 ```
 
 That is the whole operator loop. From then on they sign in at `/login`, and
-`/profile` is where they create keys, see when each was last used, and revoke the
-ones they are done with. A key is shown once on the page that creates it and
-never again.
+their account page (`/profile`) is where they create keys, see when each was
+last used, revoke the ones they are done with, and see their own use over the
+last thirty days against every budget that applies to them. A key is shown once
+on the page that creates it and never again.
+
+### Connecting an editor
+
+The account page's **Connect your editor** makes a key for Continue, Codex or
+Cline and shows what the tool needs with that key, this gateway's address and a
+model already in it:
+
+- **Continue** — an assistant file of its own, `~/.continue/assistants/llmgw.yaml`,
+  so their `config.yaml` is not touched.
+- **Codex** — a provider and a profile in `~/.codex/config.toml`, run with
+  `codex --profile llmgw`; their default model stays as it was. Codex speaks the
+  Responses API, which the gateway translates at `/v1/responses`.
+- **Cline** — four settings to paste into its own screen, since it keeps them
+  in VS Code's secret storage where no file reaches.
+
+For Continue and Codex there is a one-line command,
+`curl -fsSL <gateway>/setup/<tool>?model=… | sh`. Its script is served without
+signing in and holds no secret — the key comes from the environment — and it
+keeps a copy of anything it replaces. Revoking the key disconnects the tool.
+
+How well an editor's agent mode works depends on the model: it needs one that
+calls tools well, which the coder models do and small general ones often do not.
 
 **This gateway sends no email.** There is no "your account was approved" message
 and no password reset, so tell people what to expect: they sign up, they tell
@@ -88,11 +112,90 @@ means anything.
 If you have no structure yet, one of each is fine:
 
 ```
-llmgw keys init --store … --org mynodes --team default --project default
+llmgw org add org mynodes
+llmgw org add team default --in mynodes
+llmgw org add project default --in mynodes/default
+llmgw org tree --people
 ```
 
-It is idempotent: running it again renames rather than duplicating, so it can
-live in whatever provisions the machine.
+Everything that takes an organization, team or project — here, in `users` and
+in `keys` — accepts its id, its path (`mynodes/default/default`) or its name
+when the name is unique, so an id never has to be copied off a page. When there
+is only one project, `--project` can be left out.
+
+The tree is not fixed once made:
+
+```
+llmgw org rename project Acme/platform/gateway api-gateway
+llmgw org move project Acme/platform/gateway --to Scania/ml
+llmgw org remove team Acme/old                 # only when nothing is in it
+llmgw users delete ayse@example.com --yes      # signs out, revokes their keys
+llmgw keys list --all                          # every key, where it spends, and whose
+```
+
+A project that ever had a key cannot be removed, because usage history refers
+to it — rename it instead.
+
+### Where people belong
+
+A person can be a member of any number of organizations, teams and projects, or
+of none:
+
+```
+llmgw users join ayse@example.com --team Acme/ml          # every project in ml
+llmgw users join ayse@example.com --project Scania/research/vision
+llmgw users leave ayse@example.com --project Acme/ml/training
+```
+
+A member of a team or organization may use every project in it. Leaving revokes
+the keys the person made in projects they can no longer reach; a project they
+still reach through another membership keeps them.
+
+Someone who is a member of nothing still uses the gateway, on their own:
+personal keys, and the chat page, within their personal limits. When they make
+a key on `/profile` they choose what it spends against — one of their projects,
+or "Personal". An operator can do the same:
+
+```
+llmgw keys create laptop --for ayse@example.com                      # personal
+llmgw keys create ci --for ayse@example.com --project Acme/ml/training
+```
+
+Personal use can be limited by default, or turned off so people use the gateway
+only through their projects:
+
+```yaml
+auth:
+  personal:
+    rpm: 20
+    tpm: 50000
+    budget: 10        # whole units
+    period: month
+    # disabled: true
+```
+
+A personal key has no project, so it keeps no conversations or agent notes;
+those are kept per project.
+
+### Organization managers
+
+A member of an organization can be made its manager:
+
+```
+llmgw users join boss@acme.com --org Acme --role manager
+```
+
+or with the star on their organization membership on `/admin/users`. A manager
+gets an **Organization** link and a page at `/org`: usage over time by team,
+project or person, the teams and projects with their limits and spend, and every
+member. Only what was spent in that organization's projects is on it; what its
+people do elsewhere is not. The page is read-only — a manager is not an
+administrator of the cluster, and limits and memberships stay with those who
+are.
+
+`llmgw keys init` still works for provisioning scripts and is idempotent, but it
+will no longer move a team or project that already exists under another parent;
+it says so and stops.
 
 Split it when you want separate limits or separate reporting, not before.
 
@@ -156,18 +259,34 @@ Allowlists compose by intersection: a key cannot add a model its team withheld.
 A model's canonical name and any of its aliases name the same thing, so either
 form works.
 
-Budgets are counted in micro-units, and what a unit means is yours to decide.
-Self-hosted models have no per-token price, so it can be GPU-seconds, a notional
-rate, or 1 000 000 everywhere — which turns a budget into a plain token quota.
+Budgets are given in whole units, as the quotas page shows them, and what a unit
+means is yours to decide. Self-hosted models have no per-token price, so it can
+be GPU-seconds, a notional rate, or one unit per thousand tokens everywhere —
+a price of 1 000 000 in `auth.pricing`, which is in micro-units — and a budget
+becomes a plain quota of thousands of tokens.
 Models are priced in the server's `auth.pricing` section; a model priced at zero
 is exempt from budget checks, which is what lets a key with nothing left still be
 served a cheap fallback.
 
 ```
-llmgw keys limits --store … --scope project --id default \
-  --budget 50000000 --period month        # 50 units a month
+llmgw keys limits --store … --scope project --id Acme/platform/gateway \
+  --budget 50 --period month              # 50 units a month
 ```
 
+Two more kinds of limit follow people rather than the tree:
+
+```
+# A person, everywhere: every key they hold, every project, the chat page
+llmgw keys limits --scope user --id ayse@example.com --budget 25
+
+# Each member of a project separately — a fair share, not one shared pool
+llmgw keys limits --scope project --id Acme/ml/training --per-member --rpm 20
+```
+
+A request is held to all of them at once: the key's own, its project's, team's
+and organization's, each per-member limit on those for its owner, and the
+owner's own. The strictest wins, and each is counted in its own pool.
+
 ## The landing page
 
 Someone who reaches the gateway without a key sees the landing page. Tell it who
@@ -286,9 +405,14 @@ page exists.
 
 ## Managing accounts from the browser
 
-`/admin/users` lists who is waiting and who is active. Approving (into a project)
-and suspending are there, because that is the work an operator does constantly
-and it should not need a terminal.
+`/admin/users` is the tree: organizations, teams and projects, each project
+opening onto its members and its keys. From there an admin approves people
+(into a project or on their own), adds them to and takes them out of
+organizations, teams and projects, suspends and removes them, revokes keys, and
+adds, renames, moves and removes organizations, teams and projects. `/admin/quotas`
+sets limits on the tree, per member of any part of it, and per person. A searchable list of every account is at the
+bottom. The page works as plain forms; with scripts on, a change updates it in
+place and the projects you had open stay open.
 
 Three things are deliberately not there:
 

docs/quick-install.md

@@ -0,0 +1,178 @@
+# Quick install
+
+From empty servers to a model answering requests, in six steps. This page
+has only the commands and what to check after each; [install.md](install.md)
+explains every choice.
+
+## What you need
+
+| | |
+|---|---|
+| **One control plane machine** | Linux, x86-64 or ARM64. Debian 12, Ubuntu 22.04+ or RHEL 10. No GPU needed; 2 GB RAM, 20 GB disk. |
+| **GPU nodes** (optional to start) | The same Linux, with the NVIDIA driver installed. vLLM needs driver 580 or newer and a Turing GPU or newer (not V100). |
+| **root** | on the control plane, and `sudo` without a password on every node. |
+| **SSH** | from the control plane to each node, as a user with passwordless `sudo` there. |
+| **Network** | nodes reach the control plane on 7443 and 8081; users reach it on 8080. The machines need internet access to download engines and models. |
+
+You do not install llama.cpp, vLLM, Python or CUDA. The nodes do that
+themselves once they join.
+
+## 1. Put llmgw on the control plane
+
+```
+VERSION=$(curl -fsSL https://gitea.mynodes.xyz/api/v1/repos/ilker/llmgw/releases/latest \
+  | grep -o '"tag_name":"[^"]*"' | cut -d'"' -f4)
+U=https://gitea.mynodes.xyz/ilker/llmgw/releases/download/$VERSION
+curl -fsSLO $U/llmgw-$VERSION-linux-amd64          # -linux-arm64 on ARM
+curl -fsSL $U/checksums.txt | sha256sum -c --ignore-missing
+sudo install -m0755 llmgw-$VERSION-linux-amd64 /usr/local/bin/llmgw
+llmgw --version
+```
+
+`sha256sum` must print `OK`. If it does not, the download is damaged; do not
+install it.
+
+## 2. Set up the control plane
+
+```
+sudo llmgw server init --admin you@example.com
+```
+
+This one command sets up everything on this machine, and takes a few minutes:
+
+- the service, its certificates and its configuration;
+- your administrator account. **The password is printed once. Copy it now.**
+- a model mirror, which downloads model files once for the whole cluster;
+- this machine as a first node, and a small model (Qwen2.5 0.5B) on it.
+
+If nodes will reach this machine by a name or address it does not know itself
+(a DNS name, a NAT address), add `--host` for each:
+
+```
+sudo llmgw server init --admin you@example.com --host llm.example.com
+```
+
+It is safe to run again. It keeps what exists and finishes what is missing.
+
+**Check:** open `http://<control-plane>:8080/chat`, sign in with the printed
+password, and ask something. The small model answers. Change the password on
+the account page.
+
+## 3. Add the GPU nodes
+
+On the control plane, for each node:
+
+```
+sudo llmgw node add ubuntu@10.0.0.21
+```
+
+Replace `ubuntu` with the user you log in to the node as, and `10.0.0.21` with
+its address. That user needs `sudo` without a password on the node.
+
+It logs in over SSH and, while you watch:
+
+1. installs llmgw on the node, the release for its processor, checksum checked;
+2. runs `llmgw doctor` there, and stops if the node has a problem. Each problem
+   comes with the command that fixes it. Fix it and run the same command again;
+3. gives the node its certificate and configuration, and starts its agent;
+4. waits for the node to join and install llama.cpp.
+
+The node's name in the cluster is its hostname. vLLM installs by itself after
+that, in a few more minutes on a GPU node (about 8 GB). The cluster page
+(`/admin`) shows its progress, or why the node cannot run it.
+
+**InfiniBand.** When the node's adapters are up, `node add` shows them grouped
+by the fabric they are on and asks for each fabric's name. Give every node on
+the same fabric the same name. In a script, give it instead:
+
+```
+sudo llmgw node add ubuntu@10.0.0.21 --fabric ib-a:mlx5_0,mlx5_1,mlx5_2,mlx5_3
+```
+
+**If it cannot log in.** `sudo` runs SSH as root, with root's keys. To use your
+own keys instead, log in to the control plane with `ssh -A` and keep your
+agent through sudo:
+
+```
+sudo --preserve-env=SSH_AUTH_SOCK llmgw node add ubuntu@10.0.0.21
+```
+
+**Check:** `llmgw ctl nodes` lists the node. Running `node add` again on the
+same node is safe: it renews its certificate and configuration, and keeps what
+runs there.
+
+A machine retired for good is removed with `llmgw node remove <name>` once its
+agent is stopped.
+
+## 4. Add a model
+
+Open `/admin/mirror` and choose **Explore models**. It lists what is popular on
+Hugging Face, showing only what your nodes can run.
+
+1. Click a model to read its card.
+2. Pick one of its files, usually a `q4_k_m` for a first try, and click
+   **Fetch**. It is downloaded into the mirror once, for every node.
+3. When the download finishes, click **serve** on its row in the table above.
+4. In the form, choose how it runs: **always on**, or **on demand**, which
+   loads it on the first request and stops it after 15 minutes unused. Click
+   **Add to the catalog**.
+
+The model is placed on a node that fits it, and its status on `/admin` turns
+`ready`.
+
+**Check:** the model appears in the list on `/chat`. Ask it something.
+
+## 5. Give people access
+
+**A key for a program:** each person makes their own on the account page
+(`/profile`). The key is shown once.
+
+```
+curl http://<control-plane>:8080/v1/chat/completions \
+  -H "Authorization: Bearer $LLMGW_API_KEY" \
+  -H "Content-Type: application/json" \
+  -d '{"model": "<model-name>", "messages": [{"role": "user", "content": "hello"}]}'
+```
+
+Any OpenAI client works. Only the base URL changes:
+`http://<control-plane>:8080/v1`.
+
+**An editor:** the account page connects Continue, Codex or Cline. Pick the
+tool and a model, and it gives a one-line command that writes the
+configuration.
+
+**Other people's accounts:** they sign up at `/signup`, and you approve them on
+the people page (`/admin/users`).
+
+## 6. Put it behind HTTPS
+
+Port 8080 is plain HTTP. Before users outside a trusted network use it, put a
+reverse proxy in front with a real certificate. With Caddy, which gets the
+certificate by itself:
+
+```
+sudo apt-get install -y caddy          # Debian and Ubuntu
+curl -fsSL $U/Caddyfile | sed 's/llm.example.com/YOUR.DOMAIN/' | sudo tee /etc/caddy/Caddyfile
+sudo systemctl reload caddy
+```
+
+Keep 7443 (nodes) and 8081 (mirror) on the internal network. They never go
+through the proxy. There is an nginx configuration too:
+[deploy/proxy/](../deploy/proxy/).
+
+## When something does not work
+
+| What you see | What to do |
+|---|---|
+| `node add` stops at `join` | `sudo journalctl -u llmgw-agent -n 50` on the node. Usually it cannot reach port 7443 on the control plane. |
+| `node add` says the address is not in the certificate | Run `sudo llmgw server init --host <that address>`, then `node add` again. |
+| A node says it cannot run vLLM | The reason is on `/admin`: the driver is older than 580, the GPU is older than Turing, or there is less than 25 GB free. |
+| A model shows `not placed` | `llmgw ctl explain-placement <model>` names each node and why it was not used. |
+| A reply stops after 60 seconds behind nginx | Use the nginx configuration from the release. Its timeouts are set for slow models. |
+| Anything else about a node | `sudo llmgw doctor` on it. `--json` writes everything it read, to send to whoever helps you. |
+
+## Next
+
+- [install.md](install.md): the same steps with every option explained.
+- [fabrics.md](fabrics.md): InfiniBand and NVLink.
+- [onboarding.md](onboarding.md): accounts, projects, budgets and keys.

docs/quickstart.md

@@ -1,8 +1,8 @@
 # Quickstart: one machine, one model, ten minutes
 
-Written for someone who wants to see llmgw answer a request on a single
-workstation before reading anything else. No systemd, no `sudo`. For a real
-cluster, see [install.md](install.md).
+Written for a developer who wants to see llmgw answer a request on a single
+workstation, built from source, with no systemd and no `sudo`. **To install
+llmgw on servers, read [quick-install.md](quick-install.md) instead.**
 
 What you end up with: an OpenAI-compatible endpoint on `localhost:8080` in front
 of a real model served by llama.cpp.

docs/reference.md

@@ -29,6 +29,7 @@ rather than repeated.
 | `default_rpm` | `int` | [Auth](#auth) |
 | `default_tpm` | `int` | [Auth](#auth) |
 | `signup_open` | `bool` | [Auth](#auth) |
+| `personal` | `Personal` | [Auth](#auth) |
 | `pricing` | `map[string]Price` | [Auth](#auth) |
 | `default_price` | `Price` | [Auth](#auth) |
 | `flush_interval` | `time.Duration` | [Auth](#auth) |
@@ -102,6 +103,11 @@ rather than repeated.
 | `ib_root` | `string` | [Options](#options) |
 | `prefer_nvml` | `bool` | [Options](#options) |
 | `skip_gpus` | `bool` | [Options](#options) |
+| `disabled` | `bool` | [Personal](#personal) |
+| `rpm` | `int` | [Personal](#personal) |
+| `tpm` | `int` | [Personal](#personal) |
+| `budget` | `float64` | [Personal](#personal) |
+| `period` | `string` | [Personal](#personal) |
 | `enabled` | `bool` | [Prefix](#prefix) |
 | `max_nodes` | `int` | [Prefix](#prefix) |
 | `min_match` | `int` | [Prefix](#prefix) |
@@ -267,6 +273,15 @@ SignupOpen serves the signup form. Accounts it creates are pending until
 an operator approves them, so this is not a way in — but turning it off
 means accounts are made with `llmgw users create` instead.
 
+### `personal`
+
+`Personal`
+
+Personal is what a person may do on their own, outside any project: make
+personal keys and use the chat page personally, within these limits.
+
+Takes a [Personal](#personal) block.
+
 ### `pricing`
 
 `map[string]Price`
@@ -778,8 +793,9 @@ IBRoot is the sysfs InfiniBand directory.
 
 `bool`
 
-PreferNVML uses the NVML backend when this binary has one. It is the only
-source of NVLink topology.
+PreferNVML uses the NVML backend when this binary has one. NVLink
+topology is read from nvidia-smi topo -m on every build; NVML, where it
+is used, is the more exact source of the peers.
 
 ### `skip_gpus`
 
@@ -788,6 +804,33 @@ source of NVLink topology.
 SkipGPUs omits GPU discovery. A node without nvidia-smi is already
 found to have none, so this is for one whose driver is broken.
 
+## Personal
+
+### `disabled`
+
+`bool`
+
+Disabled refuses personal keys and personal chat; people then use the
+gateway only through projects they are members of.
+
+### `rpm`
+
+`int`
+
+### `tpm`
+
+`int`
+
+### `budget`
+
+`float64`
+
+Budget is in whole units per Period ("day" or "month").
+
+### `period`
+
+`string`
+
 ## Prefix
 
 ### `enabled`
@@ -1309,11 +1352,13 @@ Bin is the vllm executable.
 
 `string`
 
-LaunchMode selects the multi-node mechanism: "ray" or "mp".
+LaunchMode selects the multi-node mechanism: "mp", the default, or
+"ray".
 
-Which one works depends on the vLLM version, and the flags have changed
-more than once (plan §14). Both paths exist so a cluster can pick the one
-its version supports instead of waiting for a code change.
+mp is what vLLM documents today — every node runs serve with its rank,
+the workers headless — and needs nothing but vLLM. ray is the older way,
+kept for a vLLM too old for mp's multi-node flags; it needs Ray on every
+node and has not been checked against a real cluster.
 
 ### `ray_bin`
 

go.mod

@@ -6,15 +6,20 @@ require (
 	connectrpc.com/connect v1.21.0
 	github.com/NVIDIA/go-nvml v0.13.4-0
 	github.com/cespare/xxhash/v2 v2.3.0
+	github.com/charmbracelet/bubbles v1.0.0
+	github.com/charmbracelet/bubbletea v1.3.10
+	github.com/charmbracelet/lipgloss v1.1.0
 	github.com/jackc/pgx/v5 v5.11.0
 	github.com/knadh/koanf/parsers/yaml v1.1.1
 	github.com/knadh/koanf/providers/env/v2 v2.0.1
 	github.com/knadh/koanf/providers/file v1.2.1
 	github.com/knadh/koanf/v2 v2.3.6
+	github.com/mattn/go-isatty v0.0.24
 	github.com/prometheus/client_golang v1.24.1
 	github.com/prometheus/client_model v0.6.3
 	github.com/prometheus/common v0.71.0
 	github.com/spf13/cobra v1.10.2
+	github.com/yuin/goldmark v1.8.6
 	golang.org/x/crypto v0.57.0
 	golang.org/x/net v0.59.0
 	golang.org/x/sync v0.23.0
@@ -23,8 +28,18 @@ require (
 )
 
 require (
+	github.com/atotto/clipboard v0.1.4 // indirect
+	github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
+	github.com/charmbracelet/colorprofile v0.4.1 // indirect
+	github.com/charmbracelet/x/ansi v0.11.6 // indirect
+	github.com/charmbracelet/x/cellbuf v0.0.15 // indirect
+	github.com/charmbracelet/x/term v0.2.2 // indirect
+	github.com/clipperhouse/displaywidth v0.9.0 // indirect
+	github.com/clipperhouse/stringish v0.1.1 // indirect
+	github.com/clipperhouse/uax29/v2 v2.5.0 // indirect
 	github.com/dustin/go-humanize v1.0.1 // indirect
+	github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f // indirect
 	github.com/fsnotify/fsnotify v1.9.0 // indirect
 	github.com/go-viper/mapstructure/v2 v2.4.0 // indirect
 	github.com/google/uuid v1.6.0 // indirect
@@ -33,14 +48,21 @@ require (
 	github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
 	github.com/jackc/puddle/v2 v2.2.2 // indirect
 	github.com/knadh/koanf/maps v0.1.2 // indirect
-	github.com/mattn/go-isatty v0.0.24 // indirect
+	github.com/lucasb-eyer/go-colorful v1.3.0 // indirect
+	github.com/mattn/go-localereader v0.0.1 // indirect
+	github.com/mattn/go-runewidth v0.0.19 // indirect
 	github.com/mitchellh/copystructure v1.2.0 // indirect
 	github.com/mitchellh/reflectwalk v1.0.2 // indirect
+	github.com/muesli/ansi v0.0.0-20230316100256-276c6243b2f6 // indirect
+	github.com/muesli/cancelreader v0.2.2 // indirect
+	github.com/muesli/termenv v0.16.0 // indirect
 	github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
 	github.com/ncruces/go-strftime v1.0.0 // indirect
 	github.com/prometheus/procfs v0.21.1 // indirect
 	github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
+	github.com/rivo/uniseg v0.4.7 // indirect
 	github.com/spf13/pflag v1.0.9 // indirect
+	github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e // indirect
 	go.yaml.in/yaml/v3 v3.0.5 // indirect
 	golang.org/x/sys v0.48.0 // indirect
 	golang.org/x/text v0.42.0 // indirect

go.sum

@@ -2,14 +2,40 @@ connectrpc.com/connect v1.21.0 h1:LhqSJt7jHf5NJBo9Jq/t/9FjcYAideif0mg+qe2jCUs=
 connectrpc.com/connect v1.21.0/go.mod h1:A2ygJrukXwWy32vkCAAHNVguZrqZ+jeZ9rGRnGR4dN4=
 github.com/NVIDIA/go-nvml v0.13.4-0 h1:o3jp9u2x1R9ShFE3v+Aesp55XOSIQFMJz/VGNUcJaNE=
 github.com/NVIDIA/go-nvml v0.13.4-0/go.mod h1:id63qwpoDWpFXwnwM6psDCSqW4BmNu6mWpr4YeQtPGo=
+github.com/atotto/clipboard v0.1.4 h1:EH0zSVneZPSuFR11BlR9YppQTVDbh5+16AmcJi4g1z4=
+github.com/atotto/clipboard v0.1.4/go.mod h1:ZY9tmq7sm5xIbd9bOK4onWV4S6X0u6GY7Vn0Yu86PYI=
+github.com/aymanbagabas/go-osc52/v2 v2.0.1 h1:HwpRHbFMcZLEVr42D4p7XBqjyuxQH5SMiErDT4WkJ2k=
+github.com/aymanbagabas/go-osc52/v2 v2.0.1/go.mod h1:uYgXzlJ7ZpABp8OJ+exZzJJhRNQ2ASbcXHWsFqH8hp8=
 github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
 github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
 github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
 github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
+github.com/charmbracelet/bubbles v1.0.0 h1:12J8/ak/uCZEMQ6KU7pcfwceyjLlWsDLAxB5fXonfvc=
+github.com/charmbracelet/bubbles v1.0.0/go.mod h1:9d/Zd5GdnauMI5ivUIVisuEm3ave1XwXtD1ckyV6r3E=
+github.com/charmbracelet/bubbletea v1.3.10 h1:otUDHWMMzQSB0Pkc87rm691KZ3SWa4KUlvF9nRvCICw=
+github.com/charmbracelet/bubbletea v1.3.10/go.mod h1:ORQfo0fk8U+po9VaNvnV95UPWA1BitP1E0N6xJPlHr4=
+github.com/charmbracelet/colorprofile v0.4.1 h1:a1lO03qTrSIRaK8c3JRxJDZOvhvIeSco3ej+ngLk1kk=
+github.com/charmbracelet/colorprofile v0.4.1/go.mod h1:U1d9Dljmdf9DLegaJ0nGZNJvoXAhayhmidOdcBwAvKk=
+github.com/charmbracelet/lipgloss v1.1.0 h1:vYXsiLHVkK7fp74RkV7b2kq9+zDLoEU4MZoFqR/noCY=
+github.com/charmbracelet/lipgloss v1.1.0/go.mod h1:/6Q8FR2o+kj8rz4Dq0zQc3vYf7X+B0binUUBwA0aL30=
+github.com/charmbracelet/x/ansi v0.11.6 h1:GhV21SiDz/45W9AnV2R61xZMRri5NlLnl6CVF7ihZW8=
+github.com/charmbracelet/x/ansi v0.11.6/go.mod h1:2JNYLgQUsyqaiLovhU2Rv/pb8r6ydXKS3NIttu3VGZQ=
+github.com/charmbracelet/x/cellbuf v0.0.15 h1:ur3pZy0o6z/R7EylET877CBxaiE1Sp1GMxoFPAIztPI=
+github.com/charmbracelet/x/cellbuf v0.0.15/go.mod h1:J1YVbR7MUuEGIFPCaaZ96KDl5NoS0DAWkskup+mOY+Q=
+github.com/charmbracelet/x/term v0.2.2 h1:xVRT/S2ZcKdhhOuSP4t5cLi5o+JxklsoEObBSgfgZRk=
+github.com/charmbracelet/x/term v0.2.2/go.mod h1:kF8CY5RddLWrsgVwpw4kAa6TESp6EB5y3uxGLeCqzAI=
+github.com/clipperhouse/displaywidth v0.9.0 h1:Qb4KOhYwRiN3viMv1v/3cTBlz3AcAZX3+y9OLhMtAtA=
+github.com/clipperhouse/displaywidth v0.9.0/go.mod h1:aCAAqTlh4GIVkhQnJpbL0T/WfcrJXHcj8C0yjYcjOZA=
+github.com/clipperhouse/stringish v0.1.1 h1:+NSqMOr3GR6k1FdRhhnXrLfztGzuG+VuFDfatpWHKCs=
+github.com/clipperhouse/stringish v0.1.1/go.mod h1:v/WhFtE1q0ovMta2+m+UbpZ+2/HEXNWYXQgCt4hdOzA=
+github.com/clipperhouse/uax29/v2 v2.5.0 h1:x7T0T4eTHDONxFJsL94uKNKPHrclyFI0lm7+w94cO8U=
+github.com/clipperhouse/uax29/v2 v2.5.0/go.mod h1:Wn1g7MK6OoeDT0vL+Q0SQLDz/KpfsVRgg6W7ihQeh4g=
 github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
 github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
+github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f h1:Y/CXytFA4m6baUTXGLOoWe4PQhGxaX0KpnayAqC48p4=
+github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f/go.mod h1:vw97MGsxSvLiUE2X8qFplwetxpGLQrlU1Q9AUEIzCaM=
 github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
 github.com/fsnotify/fsnotify v1.9.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
 github.com/go-viper/mapstructure/v2 v2.4.0 h1:EBsztssimR/CONLSZZ04E8qAkxNYq4Qp9LvH92wZUgs=
@@ -46,12 +72,24 @@ github.com/knadh/koanf/v2 v2.3.6 h1:JoQPSJmvS4aP0xNc8xMDr5tcrkSEInL23/Il7pITAKo=
 github.com/knadh/koanf/v2 v2.3.6/go.mod h1:gRb40VRAbd4iJMYYD5IxZ6hfuopFcXBpc9bbQpZwo28=
 github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
 github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
+github.com/lucasb-eyer/go-colorful v1.3.0 h1:2/yBRLdWBZKrf7gB40FoiKfAWYQ0lqNcbuQwVHXptag=
+github.com/lucasb-eyer/go-colorful v1.3.0/go.mod h1:R4dSotOR9KMtayYi1e77YzuveK+i7ruzyGqttikkLy0=
 github.com/mattn/go-isatty v0.0.24 h1:tGZZoVgT/KiqK1c8ocVLeDS8BSWMRd47J3Lbz7vsReI=
 github.com/mattn/go-isatty v0.0.24/go.mod h1:nMCL3Zebbrt45jsMDgnfIwz6ydEQApk5oEI3HqDio6A=
+github.com/mattn/go-localereader v0.0.1 h1:ygSAOl7ZXTx4RdPYinUpg6W99U8jWvWi9Ye2JC/oIi4=
+github.com/mattn/go-localereader v0.0.1/go.mod h1:8fBrzywKY7BI3czFoHkuzRoWE9C+EiG4R1k4Cjx5p88=
+github.com/mattn/go-runewidth v0.0.19 h1:v++JhqYnZuu5jSKrk9RbgF5v4CGUjqRfBm05byFGLdw=
+github.com/mattn/go-runewidth v0.0.19/go.mod h1:XBkDxAl56ILZc9knddidhrOlY5R/pDhgLpndooCuJAs=
 github.com/mitchellh/copystructure v1.2.0 h1:vpKXTN4ewci03Vljg/q9QvCGUDttBOGBIa15WveJJGw=
 github.com/mitchellh/copystructure v1.2.0/go.mod h1:qLl+cE2AmVv+CoeAwDPye/v+N2HKCj9FbZEVFJRxO9s=
 github.com/mitchellh/reflectwalk v1.0.2 h1:G2LzWKi524PWgd3mLHV8Y5k7s6XUvT0Gef6zxSIeXaQ=
 github.com/mitchellh/reflectwalk v1.0.2/go.mod h1:mSTlrgnPZtwu0c4WaC2kGObEpuNDbx0jmZXqmk4esnw=
+github.com/muesli/ansi v0.0.0-20230316100256-276c6243b2f6 h1:ZK8zHtRHOkbHy6Mmr5D264iyp3TiX5OmNcI5cIARiQI=
+github.com/muesli/ansi v0.0.0-20230316100256-276c6243b2f6/go.mod h1:CJlz5H+gyd6CUWT45Oy4q24RdLyn7Md9Vj2/ldJBSIo=
+github.com/muesli/cancelreader v0.2.2 h1:3I4Kt4BQjOR54NavqnDogx/MIoWBFa0StPA8ELUXHmA=
+github.com/muesli/cancelreader v0.2.2/go.mod h1:3XuTXfFS2VjM+HTLZY9Ak0l6eUKfijIfMUZ4EgX0QYo=
+github.com/muesli/termenv v0.16.0 h1:S5AlUN9dENB57rsbnkPyfdGuWIlkmzJjbFf0Tf5FWUc=
+github.com/muesli/termenv v0.16.0/go.mod h1:ZRfOIKPFDYQoDFF4Olj7/QJbW60Ol/kL1pU3VfY/Cnk=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
 github.com/ncruces/go-strftime v1.0.0 h1:HMFp8mLCTPp341M/ZnA4qaf7ZlsbTc+miZjCLOFAw7w=
@@ -67,6 +105,8 @@ github.com/prometheus/procfs v0.21.1 h1:GljZCt+zSTS+NZq88cyQ1LjZ+RCHp3uVuabBWA5+
 github.com/prometheus/procfs v0.21.1/go.mod h1:aB55Cww9pdSJVHk0hUf0inxWyyjPogFIjmHKYgMKmtY=
 github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
 github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
+github.com/rivo/uniseg v0.4.7 h1:WUdvkW8uEhrYfLC4ZzdpI2ztxP1I582+49Oc5Mq64VQ=
+github.com/rivo/uniseg v0.4.7/go.mod h1:FN3SvrM+Zdj16jyLfmOkMNblXMcoc8DfTHruCPUcx88=
 github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
 github.com/spf13/cobra v1.10.2 h1:DMTTonx5m65Ic0GOoRY2c16WCbHxOOw6xxezuLaBpcU=
 github.com/spf13/cobra v1.10.2/go.mod h1:7C1pvHqHw5A4vrJfjNwvOdzYu0Gml16OCs2GRiTUUS4=
@@ -77,6 +117,10 @@ github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UV
 github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
 github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE=
 github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg=
+github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e h1:JVG44RsyaB9T2KIHavMF/ppJZNG9ZpyihvCd0w101no=
+github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e/go.mod h1:RbqR21r5mrJuqunuUZ/Dhy/avygyECGrLceyNeo4LiM=
+github.com/yuin/goldmark v1.8.6 h1:d0VcaP1sx9GkFVkoW+KtggpGi2KZ965i14b0+bDQST4=
+github.com/yuin/goldmark v1.8.6/go.mod h1:ip/1k0VRfGynBgxOz0yCqHrbZXhcjxyuS66Brc7iBKg=
 go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
 go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
 go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ=
@@ -86,12 +130,15 @@ go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
 go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
 golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
 golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
+golang.org/x/exp v0.0.0-20231006140011-7918f672742d h1:jtJma62tbqLibJ5sFQz8bKtEM8rJBtfilJ2qTU199MI=
+golang.org/x/exp v0.0.0-20231006140011-7918f672742d/go.mod h1:ldy0pHrwJyGW56pPQzzkH36rKxoZW1tw7ZJpeKx+hdo=
 golang.org/x/mod v0.41.0 h1:qJmnOUb4YB+FsEuM3HcWucdZASCPGhsX6uljO6pog0c=
 golang.org/x/mod v0.41.0/go.mod h1:Ek9pY8RKWXwsWvd3rQiHYtMqkjSUV+s1Rj7j4H5Ur6o=
 golang.org/x/net v0.59.0 h1:5zfYln+w5XCxwrnMMJPufRgNoXEaGxl0wo5GqPXyues=
 golang.org/x/net v0.59.0/go.mod h1:2DA/G1UfVbCpQPeWTmMPGY7Cs2PkBkwu743bVX5PIVg=
 golang.org/x/sync v0.23.0 h1:KameEIfc1IkluZyXWLn39Wd4tURc6GbCiISGiZm2bQk=
 golang.org/x/sync v0.23.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0=
+golang.org/x/sys v0.0.0-20210809222454-d867a43fc93e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
 golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
 golang.org/x/text v0.42.0 h1:JbOZXgfeCPU9gacVtYliJqOhD+zhrEqK4LfdpmlUZqI=

internal/account/account_test.go

@@ -99,17 +99,21 @@ func TestApprovalIsWhatMakesAnAccountWork(t *testing.T) {
 	if err != nil {
 		t.Fatal(err)
 	}
-	if got.ProjectID != "proj" {
-		t.Errorf("the approved account landed in project %q", got.ProjectID)
+	if ok, err := st.CanUseProject(ctx, got.ID, "proj"); err != nil || !ok {
+		t.Errorf("the approved account is not a member of the project it was approved into: %v", err)
 	}
 }
 
-func TestApprovalNeedsAProject(t *testing.T) {
-	// An approved account with nowhere to put its keys passes every check and
-	// then fails at the first key it tries to make.
-	_, st := newStoreAndUser(t)
-	if err := st.ApproveUser(context.Background(), "usr-1", "", "ilker", time.Now()); err == nil {
-		t.Error("an account was approved into no project")
+func TestApprovalNeedNotNameAProject(t *testing.T) {
+	// A person need not belong to anything to use the gateway: approved into
+	// nothing, they work on their own within their personal limits.
+	svc, st := newStoreAndUser(t)
+	ctx := context.Background()
+	if err := st.ApproveUser(ctx, "usr-1", "", "ilker", time.Now()); err != nil {
+		t.Fatalf("approving with no project: %v", err)
+	}
+	if _, _, err := svc.SignIn(ctx, "ayse@example.com", goodPassword); err != nil {
+		t.Errorf("an account approved into nothing cannot sign in: %v", err)
 	}
 }
 

internal/accounting/accounting.go

@@ -5,6 +5,7 @@ import (
 	"errors"
 	"fmt"
 	"log/slog"
+	"strings"
 	"time"
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/auth"
@@ -79,14 +80,27 @@ func (a *Accountant) RestoreBudgets(ctx context.Context, s store.Store, period s
 	// totals would bring the control plane back up believing the team had spent
 	// nothing — which is a budget that resets on every restart, for exactly the
 	// scopes where it matters most.
-	spend := make(map[string]int64, len(rows)*4)
+	// And every pool a person is in: their own, and each per-member limit of
+	// the project, team and organization they spent it in.
+	spend := make(map[string]int64, len(rows)*8)
 	for _, r := range rows {
-		for _, o := range []store.LimitOwner{
+		owners := []store.LimitOwner{
 			{Scope: store.ScopeKey, ID: r.KeyID},
 			{Scope: store.ScopeProject, ID: r.ProjectID},
 			{Scope: store.ScopeTeam, ID: r.TeamID},
 			{Scope: store.ScopeOrg, ID: r.OrgID},
-		} {
+			{Scope: store.ScopeUser, ID: r.UserID},
+		}
+		if r.UserID != "" && r.ProjectID != "" {
+			owners = append(owners,
+				store.MemberOwner(store.ScopeProjectMember, r.ProjectID, r.UserID),
+				store.MemberOwner(store.ScopeTeamMember, r.TeamID, r.UserID),
+				store.MemberOwner(store.ScopeOrgMember, r.OrgID, r.UserID))
+		}
+		for _, o := range owners {
+			if o.ID == "" || strings.HasPrefix(o.ID, "/") {
+				continue
+			}
 			if k := o.CounterKey(); k != "" {
 				spend[k] += r.Micros
 			}

internal/accounting/people_test.go

@@ -0,0 +1,126 @@
+package accounting
+
+import (
+	"context"
+	"errors"
+	"log/slog"
+	"path/filepath"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/auth"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// peopleStore has Acme/ml/training with two members, ayse and deniz.
+func peopleStore(t *testing.T) (*store.DB, string) {
+	t.Helper()
+	ctx := context.Background()
+	s, err := store.OpenSQLite(ctx, filepath.Join(t.TempDir(), "p.db"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	t.Cleanup(func() { _ = s.Close() })
+	org, _ := s.CreateOrg(ctx, "Acme")
+	team, _ := s.CreateTeam(ctx, org, "ml")
+	proj, _ := s.CreateProject(ctx, team, "training")
+	for _, id := range []string{"ayse", "deniz"} {
+		if err := s.CreateUser(ctx, store.User{ID: id, Email: id + "@x", Name: id,
+			PasswordHash: "h", State: store.UserPending, CreatedAt: time.Now()}); err != nil {
+			t.Fatal(err)
+		}
+		if err := s.ApproveUser(ctx, id, proj, "test", time.Now()); err != nil {
+			t.Fatal(err)
+		}
+	}
+	return s, proj
+}
+
+// keyFor makes a key for a person and resolves it the way authentication does.
+func keyFor(t *testing.T, s *store.DB, id, user, project string) auth.Principal {
+	t.Helper()
+	ctx := context.Background()
+	_, k, _ := store.NewKey(id, id, project)
+	if err := s.CreateUserKey(ctx, user, k); err != nil {
+		t.Fatal(err)
+	}
+	e, err := s.Resolve(ctx, id)
+	if err != nil {
+		t.Fatalf("resolving %s: %v", id, err)
+	}
+	return auth.Principal{KeyID: id, UserID: e.UserID, ProjectID: e.ProjectID, Limits: e}
+}
+
+func TestAPersonsLimitIsOnePoolAcrossTheirKeys(t *testing.T) {
+	s, proj := peopleStore(t)
+	if err := s.PutLimits(context.Background(),
+		store.Limits{Scope: store.ScopeUser, ScopeID: "ayse", RPM: intp(2)}); err != nil {
+		t.Fatal(err)
+	}
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	personal := keyFor(t, s, "k-personal", "ayse", "")
+	inProject := keyFor(t, s, "k-project", "ayse", proj)
+
+	// Two requests, one on each key, use up the person's two a minute.
+	for _, p := range []auth.Principal{personal, inProject} {
+		if _, err := a.Check(p, "free", 1); err != nil {
+			t.Fatalf("%s: %v", p.KeyID, err)
+		}
+	}
+	if _, err := a.Check(personal, "free", 1); !errors.Is(err, ErrRateLimited) {
+		t.Errorf("a third request across her keys was allowed: %v", err)
+	}
+	// Deniz is somebody else.
+	if _, err := a.Check(keyFor(t, s, "k-deniz", "deniz", proj), "free", 1); err != nil {
+		t.Errorf("another person was limited by hers: %v", err)
+	}
+}
+
+func TestAPerMemberLimitCountsEachMemberApart(t *testing.T) {
+	s, proj := peopleStore(t)
+	if err := s.PutLimits(context.Background(),
+		store.Limits{Scope: store.ScopeProjectMember, ScopeID: proj, RPM: intp(1)}); err != nil {
+		t.Fatal(err)
+	}
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	ayse := keyFor(t, s, "k-a", "ayse", proj)
+	deniz := keyFor(t, s, "k-d", "deniz", proj)
+
+	if _, err := a.Check(ayse, "free", 1); err != nil {
+		t.Fatal(err)
+	}
+	if _, err := a.Check(ayse, "free", 1); !errors.Is(err, ErrRateLimited) {
+		t.Errorf("her second request passed a one-a-minute per-member limit: %v", err)
+	}
+	// Her share is hers: Deniz still has his.
+	if _, err := a.Check(deniz, "free", 1); err != nil {
+		t.Errorf("one member's use counted against another: %v", err)
+	}
+	// And outside the project it does not apply at all.
+	if _, err := a.Check(keyFor(t, s, "k-a-own", "ayse", ""), "free", 1); err != nil {
+		t.Errorf("a project's per-member limit held her personal key: %v", err)
+	}
+}
+
+func TestAPersonsBudgetSurvivesARestart(t *testing.T) {
+	s, _ := peopleStore(t)
+	ctx := context.Background()
+	if err := s.PutLimits(ctx, store.Limits{Scope: store.ScopeUser, ScopeID: "ayse",
+		Budget: i64p(1000), BudgetPeriod: store.PeriodMonth}); err != nil {
+		t.Fatal(err)
+	}
+	// Spent before the restart, personally and on the chat page.
+	if err := s.WriteUsage(ctx, []store.Usage{
+		{At: time.Now(), KeyID: "k-old", UserID: "ayse", CostMicros: 600, Model: "paid", ServedModel: "paid"},
+		{At: time.Now(), KeyID: "session:ayse", UserID: "ayse", CostMicros: 500, Model: "paid", ServedModel: "paid"},
+	}); err != nil {
+		t.Fatal(err)
+	}
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	if err := a.RestoreBudgets(ctx, s, store.PeriodMonth); err != nil {
+		t.Fatal(err)
+	}
+	if _, err := a.Check(keyFor(t, s, "k-new", "ayse", ""), "paid", 1); !errors.Is(err, ErrBudgetSpent) {
+		t.Errorf("a restart forgot what she had spent: %v", err)
+	}
+}

internal/agent/agent.go

@@ -16,8 +16,10 @@ import (
 	"net/http"
 	"net/url"
 	"os"
+	"os/exec"
 	"path/filepath"
 	"slices"
+	"strconv"
 	"strings"
 	"sync"
 	"time"
@@ -118,8 +120,11 @@ type Config struct {
 
 // DefaultConfig returns sensible defaults.
 func DefaultConfig() Config {
+	// No MgmtInterface: normalize finds the one holding the management
+	// address, and falls back to eth0 only when it cannot. Defaulted here, it
+	// was never found — every node said eth0 unless told otherwise, and a node
+	// on ens3 or bond0 had NCCL bootstrap over an interface it does not have.
 	return Config{
-		MgmtInterface:     "eth0",
 		HeartbeatInterval: 5 * time.Second,
 		ReconnectBase:     time.Second,
 		ReconnectMax:      30 * time.Second,
@@ -159,7 +164,7 @@ func (c *Config) normalize() error {
 		c.MgmtInterface = interfaceOf(c.MgmtAddr)
 	}
 	if c.MgmtInterface == "" {
-		c.MgmtInterface = d.MgmtInterface
+		c.MgmtInterface = "eth0"
 	}
 	// The directory downloads land in is always one that is looked in. When
 	// they were two settings, a node could fill its disk with fetched weights
@@ -352,6 +357,22 @@ type Agent struct {
 	// failover the old stream's heartbeat loop can still be running when the
 	// new one's starts.
 	lastWeights string
+	// fabricsSent is what the control plane was last told about fabrics.
+	fabricsSent string
+	// sm caches which subnet manager each InfiniBand port sees, asked once
+	// a minute: a management datagram per port per heartbeat is more than
+	// an answer that changes on a failover is worth.
+	sm      map[string]*discovery.SMInfo
+	smErr   map[string]string
+	smAsked time.Time
+	// nvlAsked is when the GPUs' NVLink fabric state was last read.
+	nvlAsked time.Time
+	// sms is every subnet manager on a fabric, asked through one port per
+	// master, keyed by that port; smsErr says why a port could not ask.
+	sms    map[string][]discovery.SMRecord
+	smsErr map[string]string
+	// load measures the machine between heartbeats.
+	load *discovery.LoadSampler
 
 	// fetchWatchers are the shards waiting on each source, so progress on one
 	// download reaches every shard that asked for it. A fetch is single-flight
@@ -396,6 +417,7 @@ func New(cfg Config, httpClient connectHTTPClient, log *slog.Logger) (*Agent, er
 		provStatus:   map[string]provision.Status{},
 		logRing:      ring,
 		provKick:     make(chan struct{}, 1),
+		load:         discovery.NewLoadSampler(),
 	}
 	a.prov = &provision.Installer{Root: cfg.EnginesDir, Log: log, OnStatus: a.provisionStatus}
 	return a, nil
@@ -423,6 +445,7 @@ func (a *Agent) Run(ctx context.Context) error {
 		a.log.Warn("cannot create the weights directory", "dir", a.cfg.WeightsCache, "err", err)
 	}
 	go a.provisionLoop(ctx)
+	go a.ensureDiags(ctx)
 
 	attempt := 0
 	next := 0
@@ -571,21 +594,36 @@ func (a *Agent) settleAdopted(sh *supervisor.Shard, o supervisor.Orphan) {
 // machine where both halves are visible, rather than inferred later from a
 // replica that will not converge.
 func (a *Agent) usableFabrics(inv discovery.Inventory) []inventory.FabricAttachment {
+	out := a.checkFabrics(inv.HCAs, true)
+	a.mu.Lock()
+	a.fabricsSent = fabricsFingerprint(out)
+	a.mu.Unlock()
+	return out
+}
+
+// checkFabrics is the declared fabrics this node can carry traffic on, from
+// the adapters as they are now. Loud logs every problem; the heartbeat checks
+// quietly and speaks only when the answer changes.
+func (a *Agent) checkFabrics(hcas []discovery.HCA, loud bool) []inventory.FabricAttachment {
 	declared := map[string][]string{}
 	for _, f := range a.cfg.Fabrics {
 		declared[f.Fabric] = f.HCAs
 	}
 
 	var out []inventory.FabricAttachment
-	for _, c := range discovery.CheckFabrics(declared, inv.HCAs) {
-		for _, p := range c.Problems {
-			a.log.Error("a declared fabric adapter cannot carry traffic",
-				"fabric", c.Fabric, "problem", p)
+	for _, c := range discovery.CheckFabrics(declared, hcas) {
+		if loud {
+			for _, p := range c.Problems {
+				a.log.Error("a declared fabric adapter cannot carry traffic",
+					"fabric", c.Fabric, "problem", p)
+			}
 		}
 		if !c.OK() {
-			a.log.Error("fabric not advertised; no declared adapter on it can carry traffic",
-				"fabric", c.Fabric,
-				"consequence", "no multi-node replica will be placed here")
+			if loud {
+				a.log.Error("fabric not advertised; no declared adapter on it can carry traffic",
+					"fabric", c.Fabric,
+					"consequence", "no multi-node replica will be placed here")
+			}
 			continue
 		}
 		out = append(out, inventory.FabricAttachment{
@@ -595,6 +633,84 @@ func (a *Agent) usableFabrics(inv discovery.Inventory) []inventory.FabricAttachm
 	return out
 }
 
+// fabricsFingerprint is enough to notice a fabric, an adapter or a rate
+// changing.
+func fabricsFingerprint(fs []inventory.FabricAttachment) string {
+	var b strings.Builder
+	for _, f := range fs {
+		fmt.Fprintf(&b, "%s:%s:%d;", f.Fabric, strings.Join(f.HCAs, ","), f.RateGbps)
+	}
+	return b.String()
+}
+
+// sampleIB re-reads the adapters for the heartbeat: every port's state and
+// counters, and the fabrics again when they no longer match what the control
+// plane was told.
+//
+// Every heartbeat, because it is a few small files per port, and because a
+// port that went down used to be invisible until the agent restarted — the
+// node went on being offered multi-node work that would hang on its first
+// collective.
+func (a *Agent) sampleIB(hb *llmgwv1.Heartbeat) {
+	a.mu.Lock()
+	had := len(a.inv.HCAs) > 0
+	a.mu.Unlock()
+	if !had && len(a.cfg.Fabrics) == 0 {
+		return
+	}
+	hcas, err := discovery.DiscoverIB(a.cfg.Discovery.IBRoot)
+	if err != nil {
+		return
+	}
+	a.attachSM(hcas)
+	for _, h := range hcas {
+		for _, p := range h.Ports {
+			ip := &llmgwv1.IBPort{
+				Hca: h.Name, Port: int32(p.Num), State: p.State, PhysState: p.PhysState,
+				RateGbps: int32(p.RateGbps), LinkLayer: string(p.LinkLayer), Counters: p.Counters,
+				NodeGuid: h.NodeGUID, PortGuid: p.PortGUID, Network: p.Network, GidPrefix: p.GIDPrefix,
+			}
+			a.mu.Lock()
+			if p.SM != nil {
+				ip.SmGuid, ip.SmLid, ip.SmPriority, ip.SmState = p.SM.GUID, int32(p.SM.LID), int32(p.SM.Priority), p.SM.State
+			} else {
+				ip.SmError = a.smErr[smKey(h.Name, p.Num)]
+			}
+			for _, r := range a.sms[smKey(h.Name, p.Num)] {
+				ip.SubnetManagers = append(ip.SubnetManagers, &llmgwv1.SubnetManager{
+					Guid: r.GUID, Lid: int32(r.LID), Priority: int32(r.Priority), State: r.State})
+			}
+			ip.SubnetManagersError = a.smsErr[smKey(h.Name, p.Num)]
+			a.mu.Unlock()
+			hb.IbPorts = append(hb.IbPorts, ip)
+		}
+	}
+	a.mu.Lock()
+	a.inv.HCAs = hcas // a reconnect registers what is true now
+	a.mu.Unlock()
+	if len(a.cfg.Fabrics) == 0 {
+		return
+	}
+	now := a.checkFabrics(hcas, false)
+	fp := fabricsFingerprint(now)
+	a.mu.Lock()
+	changed := fp != a.fabricsSent
+	a.fabricsSent = fp
+	a.mu.Unlock()
+	if !changed {
+		return
+	}
+	// Said once, with the reasons, the moment it happens.
+	a.checkFabrics(hcas, true)
+	a.log.Warn("the fabrics this node can carry traffic on changed", "now", fp)
+	hb.FabricsChanged = true
+	for _, f := range now {
+		hb.Fabrics = append(hb.Fabrics, &llmgwv1.FabricAttachment{
+			Fabric: f.Fabric, Hcas: f.HCAs, RateGbps: int32(f.RateGbps),
+		})
+	}
+}
+
 // weights builds the fetcher lazily, so an agent that never starts a shard
 // never needs a cache directory.
 func (a *Agent) weights() *weights.Fetcher {
@@ -682,6 +798,7 @@ func (a *Agent) scan(ctx context.Context) error {
 	opts.ModelDirs = a.cfg.ModelDirs
 	inv, err := discovery.Discover(ctx, opts, a.log)
 	host := discovery.DiscoverHost()
+	host.MinComputeCapability = discovery.MinComputeCapability(inv.GPUs)
 	a.mu.Lock()
 	a.inv = inv
 	a.prov.Host = host
@@ -726,6 +843,7 @@ func (a *Agent) session(ctx context.Context, server string) error {
 	// vLLM accepted a vLLM shard and failed it minutes later.
 	reg.EngineVersions = a.engines(parent)
 	reg.EnginesProbed = true
+	reg.InstallableEngines = provision.Installable()
 	reg.RunningShards = a.runningShards()
 	a.mu.Lock()
 	h := a.prov.Host
@@ -883,9 +1001,41 @@ func (a *Agent) heartbeat() *llmgwv1.AgentMessage {
 			})
 		}
 	}
+	hb.Load = a.sampleLoad()
+	a.sampleIB(hb)
+	a.sampleNVLink(hb)
 	return &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_Heartbeat{Heartbeat: hb}}
 }
 
+// sampleLoad measures the machine and each running engine since the last
+// heartbeat. Nil on the first, which has nothing to compare with.
+func (a *Agent) sampleLoad() *llmgwv1.NodeLoad {
+	if a.load == nil {
+		return nil
+	}
+	pids := map[string]int{}
+	for _, st := range a.sup.Statuses() {
+		if st.PID > 0 {
+			pids[st.ID] = st.PID
+		}
+	}
+	l, ok := a.load.Sample(pids)
+	if !ok {
+		return nil
+	}
+	out := &llmgwv1.NodeLoad{
+		CpuPercent: l.CPUPercent, CpuCount: int32(l.CPUCount),
+		Load1: l.Load1, Load5: l.Load5, Load15: l.Load15,
+		MemoryUsedBytes: l.MemUsed, MemoryAvailableBytes: l.MemAvail,
+	}
+	for _, sh := range l.Shards {
+		out.Shards = append(out.Shards, &llmgwv1.ShardLoad{
+			ShardId: sh.ID, CpuCores: sh.CPUCores, MemoryRssBytes: sh.RSSBytes,
+		})
+	}
+	return out
+}
+
 // weightsFingerprint is enough to notice a model appearing, disappearing or
 // finishing its download.
 //
@@ -978,6 +1128,23 @@ func (a *Agent) handle(ctx context.Context, msg *llmgwv1.ControlMessage, out cha
 			case <-ctx.Done():
 			}
 		}()
+	case *llmgwv1.ControlMessage_FabricScan:
+		// A scan takes seconds to minutes: off the receive loop entirely.
+		go func(req *llmgwv1.FabricScanRequest) {
+			start := time.Now()
+			reply := &llmgwv1.FabricScanReply{RequestId: req.GetRequestId()}
+			text, err := scanFabric(ctx, req.GetHca(), int(req.GetPort()))
+			if err != nil {
+				reply.Error = err.Error()
+			} else {
+				reply.Output = text
+			}
+			reply.TookMs = time.Since(start).Milliseconds()
+			select {
+			case out <- &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_FabricScan{FabricScan: reply}}:
+			case <-ctx.Done():
+			}
+		}(m.FabricScan)
 	case *llmgwv1.ControlMessage_Ack:
 		if !m.Ack.GetOk() {
 			a.log.Warn("control plane rejected a message",
@@ -1047,6 +1214,7 @@ func (a *Agent) startShard(ctx context.Context, m *llmgwv1.StartShard, out chan<
 	}
 
 	spec, timeout := agentsvc.ShardSpecFromStart(m)
+	a.localizeSpec(&spec)
 
 	eng, err := a.reg.Get(spec.Profile.Engine)
 	if err != nil {
@@ -1439,3 +1607,195 @@ func (a *Agent) logsReply(req *llmgwv1.LogsRequest) *llmgwv1.LogsReply {
 	}
 	return out
 }
+
+// localizeSpec fills in what only this node knows.
+//
+// The interface NCCL bootstraps over is this node's, not the cluster's: the
+// control plane sends one name for every node, and a node whose management
+// network is on ens3 rather than eth0 would have NCCL bind nothing. And on
+// RoCE, which GID entry routes is a property of these adapters.
+func (a *Agent) localizeSpec(spec *engines.ShardSpec) {
+	if a.cfg.MgmtInterface != "" {
+		spec.MgmtInterface = a.cfg.MgmtInterface
+	}
+	if len(spec.HCAs) == 0 {
+		return
+	}
+	a.mu.Lock()
+	hcas := a.inv.HCAs
+	a.mu.Unlock()
+	want := map[string]bool{}
+	for _, h := range spec.HCAs {
+		want[h] = true
+	}
+	for _, h := range hcas {
+		if !want[h.Name] {
+			continue
+		}
+		for _, p := range h.Ports {
+			if p.Active() && p.RoCEv2GID >= 0 {
+				idx := p.RoCEv2GID
+				spec.GIDIndex = &idx
+				return
+			}
+		}
+	}
+}
+
+// ResolveManagement is the address and interface this node's agent would use,
+// found the way normalize finds them, for a tool that reports them.
+func ResolveManagement(c Config) (addr, iface string) {
+	addr, iface = c.MgmtAddr, c.MgmtInterface
+	server := c.Server
+	if server == "" && len(c.Servers) > 0 {
+		server = c.Servers[0]
+	}
+	if addr == "" {
+		addr = addrToward(server)
+	}
+	if iface == "" {
+		iface = interfaceOf(addr)
+	}
+	return addr, iface
+}
+
+func smKey(hca string, port int) string { return fmt.Sprintf("%s/%d", hca, port) }
+
+// scanFabric is discovery.ScanFabric, replaced in tests.
+var scanFabric = discovery.ScanFabric
+
+// querySMs is discovery.QuerySMs, replaced in tests.
+var querySMs = func(hca string, port int) ([]discovery.SMRecord, error) {
+	return discovery.QuerySMs(context.Background(), hca, port)
+}
+
+func splitSMKey(k string) (string, int) {
+	i := strings.LastIndexByte(k, '/')
+	port, _ := strconv.Atoi(k[i+1:])
+	return k[:i], port
+}
+
+// querySM is discovery.QuerySM, replaced in tests.
+var querySM = func(hca string, port int) (*discovery.SMInfo, error) {
+	return discovery.QuerySM(context.Background(), hca, port)
+}
+
+// attachSM puts on each active InfiniBand port the subnet manager it sees,
+// asking again once a minute and otherwise reusing the last answer.
+func (a *Agent) attachSM(hcas []discovery.HCA) {
+	a.mu.Lock()
+	stale := time.Since(a.smAsked) > time.Minute || a.sm == nil
+	a.mu.Unlock()
+	if stale {
+		sm, errs := map[string]*discovery.SMInfo{}, map[string]string{}
+		_ = discovery.AttachSM(hcas, func(hca string, port int) (*discovery.SMInfo, error) {
+			got, err := querySM(hca, port)
+			if err != nil {
+				errs[smKey(hca, port)] = err.Error()
+				return nil, err
+			}
+			sm[smKey(hca, port)] = got
+			return got, nil
+		})
+		// Every subnet manager on each fabric, through one port per master:
+		// the same list through every port of one fabric is one query too
+		// many for each.
+		sms, smsErr := map[string][]discovery.SMRecord{}, map[string]string{}
+		asked := map[string]bool{}
+		for k, v := range sm {
+			if asked[v.GUID] {
+				continue
+			}
+			asked[v.GUID] = true
+			hca, port := splitSMKey(k)
+			if recs, err := querySMs(hca, port); err != nil {
+				smsErr[k] = err.Error()
+			} else {
+				sms[k] = recs
+			}
+		}
+		a.mu.Lock()
+		for k, v := range sm {
+			if old := a.sm[k]; old != nil && old.GUID != v.GUID {
+				a.log.Warn("the master subnet manager changed", "port", k, "from", old.GUID, "to", v.GUID)
+			}
+		}
+		a.sm, a.smErr, a.smAsked = sm, errs, time.Now()
+		a.sms, a.smsErr = sms, smsErr
+		a.mu.Unlock()
+		return
+	}
+	a.mu.Lock()
+	defer a.mu.Unlock()
+	for i := range hcas {
+		for j := range hcas[i].Ports {
+			if sm := a.sm[smKey(hcas[i].Name, hcas[i].Ports[j].Num)]; sm != nil && hcas[i].Ports[j].Active() {
+				hcas[i].Ports[j].SM = sm
+			}
+		}
+	}
+}
+
+// ensureDiags installs infiniband-diags on a node with adapters and without
+// it. sminfo is what tells which physical fabric a port is on; without it,
+// two separate fabrics given one name cannot be caught. Installed the way
+// engine dependencies are, from the machine's own package sources.
+func (a *Agent) ensureDiags(ctx context.Context) {
+	a.mu.Lock()
+	hasIB := len(a.inv.HCAs) > 0
+	a.mu.Unlock()
+	if !hasIB || a.prov == nil {
+		return
+	}
+	if _, err := exec.LookPath("sminfo"); err == nil {
+		return
+	}
+	a.log.Info("installing infiniband-diags, to tell which fabric each port is on")
+	pkgs := map[string][]string{"deb": {"infiniband-diags"}, "rpm": {"infiniband-diags"}}
+	if err := a.prov.InstallPackages(ctx, pkgs); err != nil {
+		a.log.Warn("could not install infiniband-diags; fabrics cannot be told apart by their subnet manager",
+			"err", err)
+	}
+}
+
+// discoverGPUFabric and discoverIMEX are replaced in tests.
+var (
+	discoverGPUFabric = discovery.DiscoverGPUFabric
+	discoverIMEX      = discovery.DiscoverIMEX
+)
+
+// sampleNVLink reads, once a minute, which NVLink domain each GPU is joined
+// to and whether IMEX is up. nvidia-smi -q takes a second or two, too long
+// for every heartbeat, and a fabric that was joined an hour ago rarely
+// changes in five seconds — but when a fabric manager restarts it does, for
+// every card at once.
+func (a *Agent) sampleNVLink(hb *llmgwv1.Heartbeat) {
+	a.mu.Lock()
+	gpus := slices.Clone(a.inv.GPUs)
+	due := time.Since(a.nvlAsked) > time.Minute
+	if due {
+		a.nvlAsked = time.Now()
+	}
+	a.mu.Unlock()
+	if len(gpus) == 0 || !due {
+		return
+	}
+	ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
+	defer cancel()
+	fab, err := discoverGPUFabric(ctx, a.cfg.Discovery.NvidiaSMI)
+	if err != nil || len(fab) == 0 {
+		return
+	}
+	discovery.ApplyFabric(gpus, fab)
+	for _, g := range gpus {
+		if g.Fabric == nil {
+			continue
+		}
+		hb.GpuFabric = append(hb.GpuFabric, &llmgwv1.GPUFabricState{
+			Index: int32(g.Index), State: g.Fabric.State, Status: g.Fabric.Status,
+			ClusterUuid: g.Fabric.ClusterUUID, CliqueId: g.Fabric.CliqueID, Bandwidth: g.Fabric.Bandwidth,
+		})
+	}
+	m := discoverIMEX(ctx)
+	hb.Imex = &llmgwv1.IMEX{State: m.State, Channels: int32(m.Channels)}
+}

internal/agent/agent_test.go

@@ -4,6 +4,7 @@ import (
 	"context"
 	"crypto/tls"
 	"errors"
+	"fmt"
 	"log/slog"
 	"net"
 	"net/http"
@@ -22,6 +23,7 @@ import (
 	"golang.org/x/net/http2"
 	"golang.org/x/net/http2/h2c"
 
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/agentsvc"
@@ -756,3 +758,126 @@ func TestAFabricRateIsTheSlowestActiveLink(t *testing.T) {
 		}
 	}
 }
+
+func TestAPortGoingDownIsToldToTheControlPlaneOnce(t *testing.T) {
+	// Checked only at registration, a port going down went unnoticed until
+	// the agent restarted, and the node went on being offered multi-node
+	// work that would hang on its first collective.
+	root := ibTreeWithActivePorts(t, "mlx5_0")
+	h := start(t, func(c *Config) {
+		c.Discovery.IBRoot = root
+		c.Fabrics = []inventory.FabricAttachment{{Fabric: "ib-a", HCAs: []string{"mlx5_0"}}}
+	})
+	if n, _ := h.inv.Get("n1"); !n.HasFabric("ib-a") {
+		t.Fatalf("the fabric should be advertised at registration: %+v", n.Fabrics)
+	}
+
+	hb := h.ag.heartbeat().GetHeartbeat()
+	if hb.GetFabricsChanged() || len(hb.GetIbPorts()) != 1 || hb.GetIbPorts()[0].GetState() != "ACTIVE" {
+		t.Fatalf("a quiet heartbeat should carry the port and no change: %+v", hb)
+	}
+
+	if err := os.WriteFile(filepath.Join(root, "mlx5_0", "ports", "1", "state"), []byte("1: DOWN\n"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	hb = h.ag.heartbeat().GetHeartbeat()
+	if !hb.GetFabricsChanged() || len(hb.GetFabrics()) != 0 {
+		t.Errorf("the port going down should withdraw the fabric: changed=%v fabrics=%v",
+			hb.GetFabricsChanged(), hb.GetFabrics())
+	}
+	if hb = h.ag.heartbeat().GetHeartbeat(); hb.GetFabricsChanged() {
+		t.Error("the change should be sent once, not on every heartbeat")
+	}
+}
+
+func TestAShardUsesThisNodesInterfaceAndRoCEIndex(t *testing.T) {
+	// The control plane sends one interface name for every node; a node whose
+	// management network is on another had NCCL bind nothing. And which RoCE
+	// GID entry routes is a property of this node's adapters.
+	a := &Agent{cfg: Config{MgmtInterface: "ens3"}}
+	a.inv.HCAs = []discovery.HCA{{Name: "mlx5_0", Ports: []discovery.Port{
+		{Num: 1, State: "ACTIVE", LinkLayer: discovery.LinkEthernet, RoCEv2GID: 3}}}}
+	spec := engines.ShardSpec{MgmtInterface: "eth0", HCAs: []string{"mlx5_0"}}
+	a.localizeSpec(&spec)
+	if spec.MgmtInterface != "ens3" {
+		t.Errorf("interface = %q", spec.MgmtInterface)
+	}
+	if spec.GIDIndex == nil || *spec.GIDIndex != 3 {
+		t.Errorf("GID index = %v", spec.GIDIndex)
+	}
+
+	ib := engines.ShardSpec{HCAs: []string{"mlx5_1"}}
+	a.inv.HCAs = append(a.inv.HCAs, discovery.HCA{Name: "mlx5_1", Ports: []discovery.Port{
+		{Num: 1, State: "ACTIVE", LinkLayer: discovery.LinkInfiniBand, RoCEv2GID: -1}}})
+	a.localizeSpec(&ib)
+	if ib.GIDIndex != nil {
+		t.Error("an InfiniBand shard was given a GID index")
+	}
+}
+
+func TestTheManagementInterfaceIsFoundNotAssumed(t *testing.T) {
+	// It was defaulted to eth0 before it could be looked for, so it never
+	// was: a node on ens3 or bond0 had NCCL bootstrap over nothing.
+	cfg := DefaultConfig()
+	cfg.Server = "http://127.0.0.1:7443"
+	cfg.NodeID = "n1"
+	cfg.MgmtAddr = "127.0.0.1"
+	if err := cfg.normalize(); err != nil {
+		t.Fatal(err)
+	}
+	if cfg.MgmtInterface == "" || cfg.MgmtInterface == "eth0" {
+		t.Errorf("interface = %q, want the loopback interface holding 127.0.0.1", cfg.MgmtInterface)
+	}
+}
+
+func TestEachFabricsSubnetManagersAreAskedOnce(t *testing.T) {
+	// Two adapters on one fabric see one master. Its managers are listed
+	// through one of them — the same list twice is a query too many — and
+	// every port says which master it sees.
+	oldSM, oldSMs := querySM, querySMs
+	t.Cleanup(func() { querySM, querySMs = oldSM, oldSMs })
+	querySM = func(string, int) (*discovery.SMInfo, error) {
+		return &discovery.SMInfo{GUID: "0x01", LID: 1, State: "SMINFO_MASTER"}, nil
+	}
+	lists := 0
+	querySMs = func(string, int) ([]discovery.SMRecord, error) {
+		lists++
+		return []discovery.SMRecord{{GUID: "0x01", State: "master"}, {GUID: "0x02", State: "standby"}}, nil
+	}
+	h := start(t, func(c *Config) {
+		c.Discovery.IBRoot = ibTreeWithActivePorts(t, "mlx5_0", "mlx5_1")
+		c.Fabrics = []inventory.FabricAttachment{{Fabric: "ib-a", HCAs: []string{"mlx5_0", "mlx5_1"}}}
+	})
+	hb := h.ag.heartbeat().GetHeartbeat()
+	withList := 0
+	for _, p := range hb.GetIbPorts() {
+		if p.GetSmGuid() != "0x01" {
+			t.Errorf("%s sees master %q", p.GetHca(), p.GetSmGuid())
+		}
+		if len(p.GetSubnetManagers()) > 0 {
+			withList++
+		}
+	}
+	if lists != 1 || withList != 1 {
+		t.Errorf("listed %d times, carried on %d ports; want once, on one", lists, withList)
+	}
+}
+
+func TestAFabricScanIsAskedForAndAnswered(t *testing.T) {
+	// One node per fabric maps it; the control plane asks and parses.
+	old := scanFabric
+	t.Cleanup(func() { scanFabric = old })
+	var asked string
+	scanFabric = func(_ context.Context, hca string, port int) (string, error) {
+		asked = fmt.Sprintf("%s/%d", hca, port)
+		return "Switch\t36 \"S-00000000000000a1\"\t\t# \"MF0;leaf-a:QM8700/U1\" enhanced port 0 lid 1 lmc 0\n", nil
+	}
+	h := start(t)
+	r, err := h.svc.ScanFabric(t.Context(), "n1", "mlx5_0", 1)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if asked != "mlx5_0/1" || !strings.Contains(r.GetOutput(), "leaf-a") || r.GetError() != "" {
+		t.Errorf("asked %q, got %+v", asked, r)
+	}
+}

internal/agent/discovery/discover.go

@@ -21,8 +21,9 @@ type Options struct {
 	NvidiaSMI string `koanf:"nvidia_smi"`
 	// IBRoot is the sysfs InfiniBand directory.
 	IBRoot string `koanf:"ib_root"`
-	// PreferNVML uses the NVML backend when this binary has one. It is the only
-	// source of NVLink topology.
+	// PreferNVML uses the NVML backend when this binary has one. NVLink
+	// topology is read from nvidia-smi topo -m on every build; NVML, where it
+	// is used, is the more exact source of the peers.
 	PreferNVML bool `koanf:"prefer_nvml"`
 	// SkipGPUs omits GPU discovery. A node without nvidia-smi is already
 	// found to have none, so this is for one whose driver is broken.
@@ -64,8 +65,7 @@ func Discover(ctx context.Context, opts Options, log *slog.Logger) (Inventory, e
 				inv.GPUs, inv.Source = gpus, "nvml"
 				break
 			}
-			log.Warn("NVML discovery failed, falling back to nvidia-smi; "+
-				"NVLink topology will be unavailable", "err", err)
+			log.Warn("NVML discovery failed, falling back to nvidia-smi", "err", err)
 			fallthrough
 		default:
 			gpus, err := DiscoverGPUsSMI(ctx, opts.NvidiaSMI)
@@ -74,6 +74,26 @@ func Discover(ctx context.Context, opts Options, log *slog.Logger) (Inventory, e
 			}
 			inv.GPUs, inv.Source = gpus, "nvidia-smi"
 		}
+		// NVLink and which adapter sits nearest each card, for either path:
+		// NVML knows NVLink but not the adapters, nvidia-smi's query knows
+		// neither. A failure costs the topology and nothing else — the cards
+		// are still cards.
+		if len(inv.GPUs) > 0 {
+			topo, err := DiscoverTopology(ctx, opts.NvidiaSMI)
+			if err != nil {
+				log.Warn("could not read the GPU topology; tensor-parallel groups "+
+					"will be chosen without NVLink", "err", err)
+			} else {
+				topo.Apply(inv.GPUs)
+			}
+			// Multi-node NVLink: which domain each card has joined.
+			if fab, err := DiscoverGPUFabric(ctx, opts.NvidiaSMI); err != nil {
+				log.Warn("could not read the GPUs' NVLink fabric state", "err", err)
+			} else {
+				ApplyFabric(inv.GPUs, fab)
+			}
+			inv.IMEX = DiscoverIMEX(ctx)
+		}
 	}
 
 	hcas, err := DiscoverIB(opts.IBRoot)

internal/agent/discovery/fabrics.go

@@ -10,6 +10,8 @@ import (
 // has.
 type FabricCheck struct {
 	Fabric string
+	// Declared are the adapters the configuration names for it.
+	Declared []string
 	// Usable are the declared adapters that exist and have an active port.
 	Usable []string
 	// RateGbps is the slowest active port among them, because a collective
@@ -49,7 +51,7 @@ func CheckFabrics(declared map[string][]string, discovered []HCA) []FabricCheck
 
 	out := make([]FabricCheck, 0, len(names))
 	for _, fabric := range names {
-		c := FabricCheck{Fabric: fabric}
+		c := FabricCheck{Fabric: fabric, Declared: declared[fabric]}
 		for _, want := range declared[fabric] {
 			hca, found := byName[want]
 			if !found {

internal/agent/discovery/gpu.go

@@ -38,6 +38,13 @@ type GPU struct {
 	// NVLinkPeers are the indices of GPUs reachable over NVLink. Within a node
 	// a TP group should stay inside one NVLink domain.
 	NVLinkPeers []int `json:"nvlink_peers,omitempty"`
+	// NICPaths is how far each network adapter is from this GPU, by HCA name:
+	// PIX, PXB, PHB, NODE or SYS, as nvidia-smi topo reports it. What decides
+	// which adapter a GPU's share of a multi-node job should go through.
+	NICPaths map[string]string `json:"nic_paths,omitempty"`
+	// Fabric is the GPU's multi-node NVLink membership, nil where the
+	// driver reports none.
+	Fabric *GPUFabric `json:"fabric,omitempty"`
 	// Health carries anything that should keep the GPU out of placement.
 	Health GPUHealth `json:"health"`
 }
@@ -117,6 +124,9 @@ type Inventory struct {
 	// Storage is the disk behind those weights, one entry per filesystem.
 	// Empty means the same thing it means for Weights: not asked.
 	Storage []Storage `json:"storage,omitempty"`
+	// IMEX is the service a multi-node NVLink domain needs, read where the
+	// node has GPUs.
+	IMEX IMEX `json:"imex"`
 }
 
 // Classes returns the distinct GPU classes present, sorted.

internal/agent/discovery/gpu_fabric.go

@@ -0,0 +1,160 @@
+package discovery
+
+import (
+	"context"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"strings"
+	"time"
+)
+
+// GPUFabric is a GPU's place in a multi-node NVLink domain, as the driver
+// reports it: the NVL72 rack, or the NVLink Switch system, it belongs to.
+type GPUFabric struct {
+	// State is Completed once the fabric manager has joined the GPU to its
+	// domain; In Progress, Not Started or Not Supported otherwise.
+	State string `json:"state"`
+	// Status is Success, or the reason the join failed.
+	Status string `json:"status,omitempty"`
+	// ClusterUUID and CliqueID name the domain: GPUs sharing both can reach
+	// each other over NVLink whatever node they are in.
+	ClusterUUID string `json:"cluster_uuid,omitempty"`
+	CliqueID    string `json:"clique_id,omitempty"`
+	// Bandwidth is Full, or how the domain's NVLink is degraded.
+	Bandwidth string `json:"bandwidth,omitempty"`
+}
+
+// Domain is the NVLink domain the GPU is joined to, empty when it is in none
+// that crosses nodes: a card with no fabric, one not joined, or one whose
+// cluster is all zeros, which is what a single machine's NVSwitch reports.
+func (f *GPUFabric) Domain() string {
+	if f == nil || !strings.EqualFold(f.State, "Completed") {
+		return ""
+	}
+	u := strings.ToLower(strings.TrimSpace(f.ClusterUUID))
+	if u == "" || strings.Trim(u, "0-") == "" {
+		return ""
+	}
+	return u + "/" + f.CliqueID
+}
+
+// ParseGPUFabric reads the Fabric sections of `nvidia-smi -q`, by the PCI
+// address each GPU's section starts with.
+func ParseGPUFabric(out string) map[string]GPUFabric {
+	res := map[string]GPUFabric{}
+	var bus string
+	fabricIndent := -1
+	var cur *GPUFabric
+	flush := func() {
+		if bus != "" && cur != nil && cur.State != "" {
+			res[bus] = *cur
+		}
+		cur, fabricIndent = nil, -1
+	}
+	for _, line := range strings.Split(out, "\n") {
+		trim := strings.TrimSpace(line)
+		indent := len(line) - len(strings.TrimLeft(line, " \t"))
+		if strings.HasPrefix(line, "GPU ") && indent == 0 {
+			flush()
+			bus = strings.ToLower(strings.TrimSpace(strings.TrimPrefix(line, "GPU ")))
+			continue
+		}
+		if trim == "Fabric" {
+			flush()
+			cur, fabricIndent = &GPUFabric{}, indent
+			continue
+		}
+		if cur == nil {
+			continue
+		}
+		if trim != "" && indent <= fabricIndent {
+			flush()
+			continue
+		}
+		k, v, ok := strings.Cut(trim, ":")
+		if !ok {
+			continue
+		}
+		k, v = strings.TrimSpace(k), strings.TrimSpace(v)
+		if strings.EqualFold(v, "N/A") {
+			v = ""
+		}
+		switch k {
+		case "State":
+			cur.State = v
+		case "Status":
+			cur.Status = v
+		case "ClusterUUID", "Cluster UUID":
+			cur.ClusterUUID = v
+		case "CliqueId", "Clique ID":
+			cur.CliqueID = v
+		case "Bandwidth":
+			cur.Bandwidth = v
+		}
+	}
+	flush()
+	return res
+}
+
+// DiscoverGPUFabric runs `nvidia-smi -q` for the Fabric sections. Nothing
+// is reported on a driver or a card without them.
+func DiscoverGPUFabric(ctx context.Context, binary string) (map[string]GPUFabric, error) {
+	if binary == "" {
+		if _, err := exec.LookPath("nvidia-smi"); err != nil {
+			return nil, nil
+		}
+		binary = "nvidia-smi"
+	}
+	ctx, cancel := context.WithTimeout(ctx, 20*time.Second)
+	defer cancel()
+	out, err := exec.CommandContext(ctx, binary, "-q").Output()
+	if err != nil {
+		return nil, err
+	}
+	return ParseGPUFabric(string(out)), nil
+}
+
+// ApplyFabric puts each GPU's fabric on it, matched by PCI address.
+func ApplyFabric(gpus []GPU, fab map[string]GPUFabric) {
+	for i := range gpus {
+		if f, ok := fab[strings.ToLower(gpus[i].PCIBusID)]; ok {
+			f := f
+			gpus[i].Fabric = &f
+		}
+	}
+}
+
+// IMEX is the service that lets GPUs in different nodes of one NVLink
+// domain share memory: without it, multi-node NVLink carries nothing.
+type IMEX struct {
+	// State is systemd's word for nvidia-imex: active, inactive, or empty
+	// where it is not installed.
+	State string `json:"state,omitempty"`
+	// Channels is how many IMEX channels exist; a process needs one.
+	Channels int `json:"channels"`
+}
+
+// DiscoverIMEX reads the IMEX service and its channels.
+func DiscoverIMEX(ctx context.Context) IMEX {
+	var m IMEX
+	if entries, err := os.ReadDir("/dev/nvidia-caps-imex-channels"); err == nil {
+		m.Channels = len(entries)
+	}
+	if _, err := exec.LookPath("systemctl"); err == nil {
+		ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
+		defer cancel()
+		out, _ := exec.CommandContext(ctx, "systemctl", "is-active", "nvidia-imex").Output()
+		m.State = strings.TrimSpace(string(out))
+		if m.State == "inactive" {
+			// "inactive" is also what systemd says of a unit that does not
+			// exist; the binary says which.
+			if _, err := exec.LookPath("nvidia-imex"); err != nil {
+				if _, err := os.Stat(filepath.Join("/usr/bin", "nvidia-imex")); err != nil {
+					m.State = ""
+				}
+			}
+		}
+	}
+	return m
+}

internal/agent/discovery/gpu_fabric_test.go

@@ -0,0 +1,77 @@
+package discovery
+
+import "testing"
+
+// smiQ is an excerpt of `nvidia-smi -q` on a GB200 tray: two GPUs joined to
+// one NVL72 domain, then a PCIe card that has no fabric at all.
+const smiQ = `
+==============NVSMI LOG==============
+
+Timestamp                                 : Mon Sep 29 10:00:00 2026
+Driver Version                            : 570.86.15
+Attached GPUs                             : 3
+GPU 00000008:01:00.0
+    Product Name                          : NVIDIA GB200
+    Minor Number                          : 0
+    Fabric
+        State                             : Completed
+        Status                            : Success
+        CliqueId                          : 32766
+        ClusterUUID                       : 76B1AD76-7c24-4b6b-9f6c-2a1e0d8c11aa
+        Health
+            Bandwidth                     : Full
+    Processes                             : None
+
+GPU 00000009:01:00.0
+    Product Name                          : NVIDIA GB200
+    Fabric
+        State                             : In Progress
+        Status                            : N/A
+        CliqueId                          : N/A
+        ClusterUUID                       : N/A
+    Processes                             : None
+
+GPU 0000000A:01:00.0
+    Product Name                          : NVIDIA L40S
+    Fabric
+        State                             : N/A
+        Status                            : N/A
+`
+
+func TestAGPUsNVLinkDomainIsRead(t *testing.T) {
+	fab := ParseGPUFabric(smiQ)
+	a, ok := fab["00000008:01:00.0"]
+	if !ok {
+		t.Fatalf("fabric = %+v", fab)
+	}
+	if a.State != "Completed" || a.CliqueID != "32766" || a.Bandwidth != "Full" {
+		t.Errorf("GB200 = %+v", a)
+	}
+	if got := a.Domain(); got != "76b1ad76-7c24-4b6b-9f6c-2a1e0d8c11aa/32766" {
+		t.Errorf("domain = %q", got)
+	}
+	// Still joining: no domain yet, and said as such.
+	b := fab["00000009:01:00.0"]
+	if b.State != "In Progress" || b.Domain() != "" {
+		t.Errorf("joining GPU = %+v", b)
+	}
+	// A card with no fabric reports nothing rather than an empty one.
+	if _, ok := fab["0000000a:01:00.0"]; ok {
+		t.Error("a card without a fabric was given one")
+	}
+
+	gpus := []GPU{{Index: 0, PCIBusID: "00000008:01:00.0"}, {Index: 1, PCIBusID: "0000000A:01:00.0"}}
+	ApplyFabric(gpus, fab)
+	if gpus[0].Fabric == nil || gpus[1].Fabric != nil {
+		t.Errorf("applied = %+v %+v", gpus[0].Fabric, gpus[1].Fabric)
+	}
+}
+
+func TestASingleMachinesNVSwitchIsNoDomain(t *testing.T) {
+	// An eight-GPU HGX reports its fabric joined, with a cluster of zeros: its
+	// NVLink ends at the chassis.
+	f := &GPUFabric{State: "Completed", ClusterUUID: "00000000-0000-0000-0000-000000000000", CliqueID: "0"}
+	if f.Domain() != "" {
+		t.Errorf("a single machine read as a domain: %q", f.Domain())
+	}
+}

internal/agent/discovery/gpu_smi.go

@@ -40,8 +40,8 @@ const mandatorySMIFields = 10
 //
 // This is the cgo-free path (plan §3.2): the NVML bindings need cgo, which
 // would force the whole binary — server included — to be built with a C
-// toolchain. nvidia-smi gives everything except NVLink topology, which the
-// NVML build adds.
+// toolchain. The query gives everything except topology, which Discover
+// reads from `nvidia-smi topo -m` afterwards.
 func DiscoverGPUsSMI(ctx context.Context, binary string) ([]GPU, error) {
 	if binary == "" {
 		// Not configured and not installed is a node without an NVIDIA card,

internal/agent/discovery/host.go

@@ -4,6 +4,7 @@ import (
 	"bufio"
 	"os"
 	"runtime"
+	"strconv"
 	"strings"
 )
 
@@ -26,6 +27,9 @@ type Host struct {
 	CPUFlags []string `json:"cpu_flags,omitempty"`
 	// NVIDIADriver is the loaded driver's version, empty when there is none.
 	NVIDIADriver string `json:"nvidia_driver,omitempty"`
+	// MinComputeCapability is the oldest GPU's, 0 when there are none or it
+	// is not known. Not read here but from the GPU inventory, by the agent.
+	MinComputeCapability float64 `json:"min_compute_capability,omitempty"`
 }
 
 // Family is the packaging family: "deb", "rpm", or "" when neither is known.
@@ -152,3 +156,19 @@ func nvidiaDriverFrom(path string) string {
 	}
 	return ""
 }
+
+// MinComputeCapability is the oldest GPU's compute capability, 0 when none
+// is known: an engine built for newer GPUs is refused by the oldest.
+func MinComputeCapability(gpus []GPU) float64 {
+	low := 0.0
+	for _, g := range gpus {
+		cc, err := strconv.ParseFloat(strings.TrimSpace(g.ComputeCapability), 64)
+		if err != nil || cc <= 0 {
+			continue
+		}
+		if low == 0 || cc < low {
+			low = cc
+		}
+	}
+	return low
+}

internal/agent/discovery/ib.go

@@ -5,6 +5,7 @@ package discovery
 import (
 	"errors"
 	"fmt"
+	"net"
 	"os"
 	"path/filepath"
 	"slices"
@@ -42,8 +43,66 @@ type Port struct {
 	GIDPrefix string `json:"gid_prefix"`
 	LID       int    `json:"lid"`
 	SMLID     int    `json:"sm_lid"`
+	// Counters are the port's error and traffic counters, by their sysfs
+	// names, as far as the driver has them. The data counters are converted
+	// to bytes; sysfs counts them in four-byte words.
+	Counters map[string]uint64 `json:"counters,omitempty"`
+	// RoCEv2GID is the GID table index NCCL should use on a RoCE port: the
+	// first RoCE v2 entry holding an IPv4 address, else the first RoCE v2
+	// entry. -1 on InfiniBand, and on a RoCE port with no v2 entry. NCCL's
+	// default of 0 is a RoCE v1 link-local entry on most setups, which does
+	// not route, and a job on it hangs.
+	RoCEv2GID int `json:"rocev2_gid"`
+	// PortGUID is the port's own identifier, the interface half of its GID.
+	PortGUID string `json:"port_guid,omitempty"`
+	// SM is the subnet manager this port sees as master, from sminfo. Nil on
+	// RoCE, which has none, and when infiniband-diags is not installed.
+	SM *SMInfo `json:"sm,omitempty"`
+	// Network is, on RoCE, the IP network of the RoCE v2 address: the only
+	// thing that says two RoCE ports are on the same fabric.
+	Network string `json:"network,omitempty"`
 }
 
+// SMInfo is a subnet manager as a port sees it.
+type SMInfo struct {
+	GUID     string `json:"guid"`
+	LID      int    `json:"lid"`
+	Priority int    `json:"priority"`
+	// State is SMINFO_MASTER, SMINFO_STANDBY, ...
+	State string `json:"state"`
+}
+
+// FabricKey is what identifies the physical fabric this port is on: the
+// master subnet manager on InfiniBand, since every port on one fabric sees
+// the same one and ports on separate fabrics cannot; the IP network on RoCE.
+// Empty when neither is known, and for a port that is not up.
+func (p Port) FabricKey() string {
+	if !p.Active() {
+		return ""
+	}
+	switch {
+	case p.LinkLayer == LinkEthernet && p.Network != "":
+		return "roce:" + p.Network
+	case p.SM != nil && p.SM.GUID != "":
+		return "sm:" + p.SM.GUID
+	}
+	return ""
+}
+
+// PortCounters are the counters worth watching, from each port's counters/
+// directory. The errors are the ones that say a cable, a transceiver or a
+// switch port is going bad before a job fails on it; the data counters say
+// whether the fabric is carrying anything at all.
+var PortCounters = []string{
+	"symbol_error", "link_error_recovery", "link_downed", "port_rcv_errors",
+	"port_rcv_remote_physical_errors", "port_xmit_discards",
+	"local_link_integrity_errors", "excessive_buffer_overrun_errors",
+	"port_xmit_data", "port_rcv_data",
+}
+
+// dataCounter reports whether a counter counts data in four-byte words.
+func dataCounter(name string) bool { return name == "port_xmit_data" || name == "port_rcv_data" }
+
 // Active reports whether the port can carry traffic.
 func (p Port) Active() bool { return strings.EqualFold(p.State, "ACTIVE") }
 
@@ -56,6 +115,9 @@ type HCA struct {
 	// FirmwareVersion is reported by the driver; worth capturing because
 	// NCCL behaviour differs between firmware revisions.
 	FirmwareVersion string `json:"firmware_version,omitempty"`
+	// NodeGUID identifies the adapter on its fabric: it is the name a
+	// switch's view of the fabric, and every other tool, knows it by.
+	NodeGUID string `json:"node_guid,omitempty"`
 }
 
 // ActivePorts returns the ports that can carry traffic.
@@ -102,6 +164,7 @@ func DiscoverIB(root string) ([]HCA, error) {
 func readHCA(dir, name string) (HCA, error) {
 	h := HCA{Name: name, NUMANode: -1}
 	h.FirmwareVersion = readTrimmed(filepath.Join(dir, "fw_ver"))
+	h.NodeGUID = normalizeGUID(readTrimmed(filepath.Join(dir, "node_guid")))
 
 	// device is a symlink into the PCI tree; its target's base name is the
 	// bus address, and numa_node lives beside it.
@@ -138,6 +201,30 @@ func readPort(dir string, num int) Port {
 		LID:       parseMaybeHex(readTrimmed(filepath.Join(dir, "lid"))),
 		SMLID:     parseMaybeHex(readTrimmed(filepath.Join(dir, "sm_lid"))),
 	}
+	p.RoCEv2GID = -1
+	p.PortGUID = portGUID(readTrimmed(filepath.Join(dir, "gids", "0")))
+	if p.LinkLayer == LinkEthernet {
+		p.RoCEv2GID = rocev2GID(dir)
+		if p.RoCEv2GID >= 0 {
+			idx := strconv.Itoa(p.RoCEv2GID)
+			p.Network = RoCENetwork(readTrimmed(filepath.Join(dir, "gid_attrs", "ndevs", idx)),
+				readTrimmed(filepath.Join(dir, "gids", idx)))
+		}
+	}
+	for _, name := range PortCounters {
+		raw := readTrimmed(filepath.Join(dir, "counters", name))
+		v, err := strconv.ParseUint(raw, 10, 64)
+		if err != nil {
+			continue // not every driver has every counter
+		}
+		if dataCounter(name) {
+			v *= 4
+		}
+		if p.Counters == nil {
+			p.Counters = map[string]uint64{}
+		}
+		p.Counters[name] = v
+	}
 	return p
 }
 
@@ -201,3 +288,91 @@ func readTrimmed(path string) string {
 	}
 	return strings.TrimSpace(string(b))
 }
+
+// rocev2GID finds the GID index a RoCE port's traffic should use.
+func rocev2GID(dir string) int {
+	first := -1
+	for i := 0; i < 64; i++ {
+		typ := readTrimmed(filepath.Join(dir, "gid_attrs", "types", strconv.Itoa(i)))
+		if !strings.EqualFold(typ, "RoCE v2") {
+			continue
+		}
+		gid := strings.ToLower(readTrimmed(filepath.Join(dir, "gids", strconv.Itoa(i))))
+		if strings.HasPrefix(gid, "0000:0000:0000:0000:0000:ffff:") {
+			return i
+		}
+		if first < 0 {
+			first = i
+		}
+	}
+	return first
+}
+
+// normalizeGUID writes a GUID one way: sysfs says "0c42:a103:0045:bd6c",
+// sminfo and ibnetdiscover say "0x0c42a1030045bd6c".
+func normalizeGUID(g string) string {
+	g = strings.ToLower(strings.TrimSpace(g))
+	g = strings.TrimPrefix(g, "0x")
+	g = strings.ReplaceAll(g, ":", "")
+	if g == "" || strings.Trim(g, "0") == "" {
+		return ""
+	}
+	for len(g) < 16 {
+		g = "0" + g
+	}
+	return "0x" + g
+}
+
+// NormalizeGUID is normalizeGUID for other packages.
+func NormalizeGUID(g string) string { return normalizeGUID(g) }
+
+// portGUID is the interface identifier of a GID: its last four groups.
+func portGUID(gid string) string {
+	parts := strings.Split(gid, ":")
+	if len(parts) != 8 {
+		return ""
+	}
+	return normalizeGUID(strings.Join(parts[4:], ""))
+}
+
+// RoCENetwork is the IP network an IPv4-mapped RoCE GID's address sits in,
+// read from the interface that carries it. Replaced in tests.
+var RoCENetwork = func(ndev, gid string) string {
+	ip := gidIPv4(gid)
+	if ip == nil || ndev == "" {
+		return ""
+	}
+	ifc, err := net.InterfaceByName(ndev)
+	if err != nil {
+		return ""
+	}
+	addrs, _ := ifc.Addrs()
+	for _, a := range addrs {
+		if n, ok := a.(*net.IPNet); ok && n.IP.Equal(ip) {
+			return (&net.IPNet{IP: ip.Mask(n.Mask), Mask: n.Mask}).String()
+		}
+	}
+	return ""
+}
+
+// gidIPv4 is the IPv4 address in an IPv4-mapped GID, nil otherwise.
+func gidIPv4(gid string) net.IP {
+	g := strings.ToLower(gid)
+	const mapped = "0000:0000:0000:0000:0000:ffff:"
+	if !strings.HasPrefix(g, mapped) {
+		return nil
+	}
+	hex := strings.ReplaceAll(strings.TrimPrefix(g, mapped), ":", "")
+	if len(hex) != 8 {
+		return nil
+	}
+	b := make(net.IP, 4)
+	for i := range 4 {
+		v, err := strconv.ParseUint(hex[i*2:i*2+2], 16, 8)
+		if err != nil {
+			return nil
+		}
+		b[i] = byte(v)
+	}
+	return b
+}

internal/agent/discovery/ib_test.go

@@ -31,6 +31,9 @@ func fakeSysfs(t *testing.T) string {
 	write("ib/mlx5_0/ports/1/gids/0", "FEC0:0000:0000:000A:0002:C903:00A0:1234")
 	write("ib/mlx5_0/ports/1/lid", "0x1a")
 	write("ib/mlx5_0/ports/1/sm_lid", "1")
+	write("ib/mlx5_0/ports/1/counters/symbol_error", "3")
+	write("ib/mlx5_0/ports/1/counters/link_downed", "1")
+	write("ib/mlx5_0/ports/1/counters/port_xmit_data", "1000")
 
 	// mlx5_1: two ports, the second one down.
 	write("ib/mlx5_1/ports/1/state", "4: ACTIVE")
@@ -46,7 +49,13 @@ func fakeSysfs(t *testing.T) string {
 	write("ib/mlx5_2/ports/1/state", "4: ACTIVE")
 	write("ib/mlx5_2/ports/1/rate", "100 Gb/sec (4X EDR)")
 	write("ib/mlx5_2/ports/1/link_layer", "Ethernet")
-	write("ib/mlx5_2/ports/1/gids/0", "0000:0000:0000:0000:0000:ffff:0a00:0001")
+	write("ib/mlx5_2/ports/1/gids/0", "fe80:0000:0000:0000:0000:0000:0000:0001")
+	write("ib/mlx5_2/ports/1/gid_attrs/types/0", "IB/RoCE v1")
+	write("ib/mlx5_2/ports/1/gids/1", "fe80:0000:0000:0000:0000:0000:0000:0001")
+	write("ib/mlx5_2/ports/1/gid_attrs/types/1", "RoCE v2")
+	write("ib/mlx5_2/ports/1/gids/3", "0000:0000:0000:0000:0000:ffff:0a00:0001")
+	write("ib/mlx5_2/ports/1/gid_attrs/types/3", "RoCE v2")
+	write("ib/mlx5_2/ports/1/gid_attrs/ndevs/3", "ens1f0")
 
 	// A PCI device with a NUMA node, linked the way the driver does it.
 	pci := filepath.Join(root, "pci", "0000:0b:00.0")
@@ -214,3 +223,131 @@ func TestParseHelpers(t *testing.T) {
 		}
 	})
 }
+
+func TestAPortsCountersAreRead(t *testing.T) {
+	hcas, err := DiscoverIB(fakeSysfs(t))
+	if err != nil {
+		t.Fatal(err)
+	}
+	c := hcas[0].Ports[0].Counters
+	// Data in bytes: sysfs counts four-byte words.
+	if c["symbol_error"] != 3 || c["link_downed"] != 1 || c["port_xmit_data"] != 4000 {
+		t.Errorf("counters = %v", c)
+	}
+	if _, ok := c["port_rcv_errors"]; ok {
+		t.Error("a counter the driver does not have should be absent, not zero")
+	}
+}
+
+func TestRoCEFindsItsRoutableGID(t *testing.T) {
+	// NCCL's default GID index is a v1 link-local entry on most hosts, which
+	// does not route: the job hangs. The v2 entry holding an IPv4 address is
+	// the one to use, over an earlier v2 link-local one.
+	hcas, err := DiscoverIB(fakeSysfs(t))
+	if err != nil {
+		t.Fatal(err)
+	}
+	for _, h := range hcas {
+		p := h.Ports[0]
+		switch h.Name {
+		case "mlx5_2":
+			if p.RoCEv2GID != 3 {
+				t.Errorf("RoCE GID index = %d, want 3", p.RoCEv2GID)
+			}
+		default:
+			if p.RoCEv2GID != -1 {
+				t.Errorf("%s is InfiniBand and has a GID index %d", h.Name, p.RoCEv2GID)
+			}
+		}
+	}
+}
+
+func TestTheSubnetManagerAPortSeesIsRead(t *testing.T) {
+	sm, err := ParseSMInfo("sminfo: sm lid 1 sm guid 0x98039B03009fcfd6, activity count 5452 priority 15 state 3 SMINFO_MASTER\n")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if sm.GUID != "0x98039b03009fcfd6" || sm.LID != 1 || sm.Priority != 15 || sm.State != "SMINFO_MASTER" {
+		t.Errorf("sm = %+v", sm)
+	}
+	if _, err := ParseSMInfo("ibwarn: [1234] mad_rpc_open_port: can't open UMAD port"); err == nil {
+		t.Error("an error from sminfo should not parse")
+	}
+}
+
+func TestAPortsFabricIsItsSubnetManagerOrItsNetwork(t *testing.T) {
+	// Two ports on one InfiniBand fabric see one master SM, and on separate
+	// fabrics cannot; RoCE has no SM, and is its IP network.
+	hcas, err := DiscoverIB(fakeSysfs(t))
+	if err != nil {
+		t.Fatal(err)
+	}
+	if hcas[0].NodeGUID != "" {
+		t.Errorf("no node_guid file, yet %q", hcas[0].NodeGUID)
+	}
+	err = AttachSM(hcas, func(hca string, port int) (*SMInfo, error) {
+		return &SMInfo{GUID: "0x00000000000000aa", LID: 1, State: "SMINFO_MASTER"}, nil
+	})
+	if err != nil {
+		t.Fatal(err)
+	}
+	if got := hcas[0].Ports[0].FabricKey(); got != "sm:0x00000000000000aa" {
+		t.Errorf("IB key = %q", got)
+	}
+	if got := hcas[1].Ports[1].FabricKey(); got != "" {
+		t.Errorf("a port that is down is on no fabric, got %q", got)
+	}
+	if hcas[2].Ports[0].SM != nil {
+		t.Error("RoCE was asked for a subnet manager")
+	}
+	if got := hcas[0].Ports[0].PortGUID; got != "0x0002c90300a01234" {
+		t.Errorf("port GUID = %q", got)
+	}
+}
+
+func TestARoCEPortsNetworkIsItsFabric(t *testing.T) {
+	old := RoCENetwork
+	t.Cleanup(func() { RoCENetwork = old })
+	RoCENetwork = func(ndev, gid string) string {
+		if ip := gidIPv4(gid); ip != nil && ip.String() == "10.0.0.1" {
+			return "10.0.0.0/24"
+		}
+		return ""
+	}
+	hcas, err := DiscoverIB(fakeSysfs(t))
+	if err != nil {
+		t.Fatal(err)
+	}
+	roce := hcas[2].Ports[0]
+	if roce.Network != "10.0.0.0/24" || roce.FabricKey() != "roce:10.0.0.0/24" {
+		t.Errorf("RoCE network %q key %q", roce.Network, roce.FabricKey())
+	}
+}
+
+// smirDump is `saquery SMIR` on a fabric with a master and a standby, in the
+// layout infiniband-diags prints.
+const smirDump = "SMInfoRecord dump:\n\t\tRID\n\t\tLID...................1\n\t\tSMInfo dump:\n" +
+	"\t\tGUID..................0x248a07030030f0c0\n\t\tSM_Key................0x0000000000000000\n" +
+	"\t\tActCount..............386773\n\t\tPriority..............15\n\t\tSMState...............3\n" +
+	"SMInfoRecord dump:\n\t\tRID\n\t\tLID...................12\n\t\tSMInfo dump:\n" +
+	"\t\tGUID..................0x248a07030030f1d4\n\t\tSM_Key................0x0000000000000000\n" +
+	"\t\tActCount..............12\n\t\tPriority..............14\n\t\tSMState...............2\n"
+
+func TestEverySubnetManagerOnAFabricIsListed(t *testing.T) {
+	recs, err := ParseSMIR(smirDump)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(recs) != 2 {
+		t.Fatalf("records = %+v", recs)
+	}
+	if recs[0].GUID != "0x248a07030030f0c0" || recs[0].State != "master" || recs[0].Priority != 15 || recs[0].LID != 1 {
+		t.Errorf("master = %+v", recs[0])
+	}
+	if recs[1].State != "standby" || recs[1].LID != 12 {
+		t.Errorf("standby = %+v", recs[1])
+	}
+	if _, err := ParseSMIR("ibwarn: can't open UMAD port"); err == nil {
+		t.Error("an error should not parse")
+	}
+}

internal/agent/discovery/load.go

@@ -0,0 +1,260 @@
+package discovery
+
+import (
+	"bufio"
+	"bytes"
+	"os"
+	"path/filepath"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+)
+
+// Load is how busy a node is: the machine as a whole, and each engine's share.
+type Load struct {
+	CPUPercent float64
+	CPUCount   int
+	Load1      float64
+	Load5      float64
+	Load15     float64
+	MemUsed    int64
+	MemAvail   int64
+	Shards     []ShardLoad
+}
+
+// ShardLoad is one engine process group's use.
+type ShardLoad struct {
+	ID       string
+	CPUCores float64
+	RSSBytes int64
+}
+
+// userHZ is the unit /proc counts CPU time in. Fixed at 100 by the kernel's
+// ABI on every architecture a node runs, whatever the internal tick rate.
+const userHZ = 100
+
+// LoadSampler measures load as the difference between two readings of /proc.
+//
+// CPU time is a counter, so a percentage needs two samples and the time
+// between them. The sampler keeps the previous one; the first call has
+// nothing to compare with and reports no load at all, rather than a zero that
+// would read as an idle machine.
+//
+// Linux only, like every node. Elsewhere the files are missing and it reports
+// nothing, as an older agent does.
+type LoadSampler struct {
+	root string
+	now  func() time.Time
+
+	mu      sync.Mutex
+	prevAt  time.Time
+	prevCPU cpuTimes
+	prevPG  map[int]groupUse // process group -> what it has used
+}
+
+// groupUse is a process group's CPU time so far and its resident memory now.
+type groupUse struct {
+	jiffies  uint64
+	rssPages int64
+}
+
+type cpuTimes struct{ total, idle uint64 }
+
+// NewLoadSampler reads the real /proc.
+func NewLoadSampler() *LoadSampler { return newLoadSampler("/proc", time.Now) }
+
+func newLoadSampler(root string, now func() time.Time) *LoadSampler {
+	return &LoadSampler{root: root, now: now}
+}
+
+// Sample reads the machine and the process groups led by the given shards'
+// PIDs, keyed by shard ID. False when there is nothing to report yet.
+func (s *LoadSampler) Sample(shards map[string]int) (Load, bool) {
+	cpu, ok := s.cpu()
+	if !ok {
+		return Load{}, false
+	}
+	groups := s.groups()
+	at := s.now()
+
+	s.mu.Lock()
+	prevAt, prevCPU, prevPG := s.prevAt, s.prevCPU, s.prevPG
+	s.prevAt, s.prevCPU, s.prevPG = at, cpu, groups
+	s.mu.Unlock()
+
+	if prevAt.IsZero() || cpu.total <= prevCPU.total {
+		return Load{}, false
+	}
+	l := Load{CPUCount: s.cpuCount()}
+	dTotal := float64(cpu.total - prevCPU.total)
+	dIdle := float64(cpu.idle - prevCPU.idle)
+	l.CPUPercent = clamp((dTotal-dIdle)/dTotal*100, 0, 100)
+	l.Load1, l.Load5, l.Load15 = s.loadavg()
+	total, avail := s.meminfo()
+	if total > 0 {
+		l.MemUsed, l.MemAvail = total-avail, avail
+	}
+
+	secs := at.Sub(prevAt).Seconds()
+	for id, pid := range shards {
+		if pid <= 0 {
+			continue
+		}
+		now := groups[pid]
+		sl := ShardLoad{ID: id, RSSBytes: now.rssPages * int64(os.Getpagesize())}
+		if before, ok := prevPG[pid]; ok && secs > 0 && now.jiffies >= before.jiffies {
+			sl.CPUCores = float64(now.jiffies-before.jiffies) / userHZ / secs
+		}
+		l.Shards = append(l.Shards, sl)
+	}
+	return l, true
+}
+
+func clamp(v, lo, hi float64) float64 { return max(lo, min(v, hi)) }
+
+// cpu reads the aggregate line of /proc/stat. Idle includes iowait: a core
+// waiting on the disk is not doing work.
+func (s *LoadSampler) cpu() (cpuTimes, bool) {
+	b, err := os.ReadFile(filepath.Join(s.root, "stat"))
+	if err != nil {
+		return cpuTimes{}, false
+	}
+	line, _, _ := bytes.Cut(b, []byte("\n"))
+	f := strings.Fields(string(line))
+	if len(f) < 5 || f[0] != "cpu" {
+		return cpuTimes{}, false
+	}
+	var t cpuTimes
+	for i, v := range f[1:] {
+		// guest and guest_nice are already counted in user and nice.
+		if i >= 8 {
+			break
+		}
+		n, _ := strconv.ParseUint(v, 10, 64)
+		t.total += n
+		if i == 3 || i == 4 {
+			t.idle += n
+		}
+	}
+	return t, true
+}
+
+func (s *LoadSampler) cpuCount() int {
+	b, err := os.ReadFile(filepath.Join(s.root, "stat"))
+	if err != nil {
+		return 0
+	}
+	n := 0
+	for _, line := range strings.Split(string(b), "\n") {
+		if strings.HasPrefix(line, "cpu") && len(line) > 3 && line[3] >= '0' && line[3] <= '9' {
+			n++
+		}
+	}
+	return n
+}
+
+func (s *LoadSampler) loadavg() (l1, l5, l15 float64) {
+	b, err := os.ReadFile(filepath.Join(s.root, "loadavg"))
+	if err != nil {
+		return 0, 0, 0
+	}
+	f := strings.Fields(string(b))
+	if len(f) < 3 {
+		return 0, 0, 0
+	}
+	l1, _ = strconv.ParseFloat(f[0], 64)
+	l5, _ = strconv.ParseFloat(f[1], 64)
+	l15, _ = strconv.ParseFloat(f[2], 64)
+	return l1, l5, l15
+}
+
+// meminfo returns MemTotal and MemAvailable in bytes.
+func (s *LoadSampler) meminfo() (total, avail int64) {
+	f, err := os.Open(filepath.Join(s.root, "meminfo"))
+	if err != nil {
+		return 0, 0
+	}
+	defer f.Close()
+	sc := bufio.NewScanner(f)
+	for sc.Scan() {
+		k, v, ok := strings.Cut(sc.Text(), ":")
+		if !ok {
+			continue
+		}
+		fields := strings.Fields(v)
+		if len(fields) == 0 {
+			continue
+		}
+		kb, _ := strconv.ParseInt(fields[0], 10, 64)
+		switch k {
+		case "MemTotal":
+			total = kb * 1024
+		case "MemAvailable":
+			avail = kb * 1024
+		}
+	}
+	return total, avail
+}
+
+// procStat is what one /proc/<pid>/stat says that load needs.
+type procStat struct {
+	pgrp     int
+	jiffies  uint64
+	rssPages int64
+}
+
+// readProcStat parses /proc/<pid>/stat. The command name is in parentheses
+// and may itself hold spaces and parentheses, so fields are counted from the
+// last closing one.
+func readProcStat(path string) (procStat, bool) {
+	b, err := os.ReadFile(path)
+	if err != nil {
+		return procStat{}, false
+	}
+	i := bytes.LastIndexByte(b, ')')
+	if i < 0 {
+		return procStat{}, false
+	}
+	f := strings.Fields(string(b[i+1:]))
+	// After the name: state(0) ppid(1) pgrp(2) ... utime(11) stime(12) ... rss(21)
+	if len(f) < 22 {
+		return procStat{}, false
+	}
+	var st procStat
+	st.pgrp, _ = strconv.Atoi(f[2])
+	ut, _ := strconv.ParseUint(f[11], 10, 64)
+	stt, _ := strconv.ParseUint(f[12], 10, 64)
+	st.jiffies = ut + stt
+	st.rssPages, _ = strconv.ParseInt(f[21], 10, 64)
+	return st, true
+}
+
+// groups totals CPU time and memory per process group, in one pass over
+// /proc. An engine is started as the leader of its own group, so its workers
+// — vLLM starts several — are counted with it.
+func (s *LoadSampler) groups() map[int]groupUse {
+	out := map[int]groupUse{}
+	s.eachProc(func(st procStat) {
+		g := out[st.pgrp]
+		g.jiffies += st.jiffies
+		g.rssPages += st.rssPages
+		out[st.pgrp] = g
+	})
+	return out
+}
+
+func (s *LoadSampler) eachProc(fn func(procStat)) {
+	entries, err := os.ReadDir(s.root)
+	if err != nil {
+		return
+	}
+	for _, e := range entries {
+		if _, err := strconv.Atoi(e.Name()); err != nil {
+			continue
+		}
+		if st, ok := readProcStat(filepath.Join(s.root, e.Name(), "stat")); ok {
+			fn(st)
+		}
+	}
+}

internal/agent/discovery/load_test.go

@@ -0,0 +1,96 @@
+package discovery
+
+import (
+	"fmt"
+	"os"
+	"path/filepath"
+	"testing"
+	"time"
+)
+
+// fakeProc is a /proc with two CPUs, a memory picture and processes.
+type fakeProc struct {
+	t    *testing.T
+	root string
+}
+
+func newFakeProc(t *testing.T) *fakeProc {
+	f := &fakeProc{t: t, root: t.TempDir()}
+	f.write("loadavg", "1.50 0.75 0.25 2/300 4242\n")
+	f.write("meminfo", "MemTotal:       8000000 kB\nMemFree:        1000000 kB\nMemAvailable:   3000000 kB\n")
+	return f
+}
+
+func (f *fakeProc) write(name, body string) {
+	f.t.Helper()
+	p := filepath.Join(f.root, name)
+	if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
+		f.t.Fatal(err)
+	}
+	if err := os.WriteFile(p, []byte(body), 0o644); err != nil {
+		f.t.Fatal(err)
+	}
+}
+
+// cpu sets the aggregate counters: user and idle, the rest zero.
+func (f *fakeProc) cpu(user, idle uint64) {
+	f.write("stat", fmt.Sprintf("cpu  %d 0 0 %d 0 0 0 0 0 0\ncpu0 0 0 0 0 0 0 0 0 0 0\ncpu1 0 0 0 0 0 0 0 0 0 0\nintr 1\n", user, idle))
+}
+
+// proc sets one process: its group, CPU jiffies (as utime) and RSS in pages.
+func (f *fakeProc) proc(pid, pgrp int, jiffies uint64, rss int64) {
+	// The name holds a space and a parenthesis, as real ones can.
+	f.write(fmt.Sprintf("%d/stat", pid), fmt.Sprintf(
+		"%d (llama-server (x)) S 1 %d %d 0 -1 0 0 0 0 0 %d 0 0 0 20 0 1 0 100 1000 %d 0\n",
+		pid, pgrp, pgrp, jiffies, rss))
+}
+
+func TestLoadIsTheDifferenceBetweenTwoReadings(t *testing.T) {
+	p := newFakeProc(t)
+	at := time.Unix(1000, 0)
+	s := newLoadSampler(p.root, func() time.Time { return at })
+
+	p.cpu(100, 900)
+	p.proc(42, 42, 1000, 10)
+	p.proc(43, 42, 500, 5) // a worker in the engine's group
+	p.proc(7, 7, 99999, 1) // something else entirely
+	if _, ok := s.Sample(map[string]int{"shard-a": 42}); ok {
+		t.Fatal("the first reading has nothing to compare with and should report nothing")
+	}
+
+	// Two seconds later: 300 of the 400 jiffies that passed were busy, and
+	// the engine's group used 400 of them.
+	at = at.Add(2 * time.Second)
+	p.cpu(400, 1000)
+	p.proc(42, 42, 1300, 10)
+	p.proc(43, 42, 600, 6)
+	l, ok := s.Sample(map[string]int{"shard-a": 42})
+	if !ok {
+		t.Fatal("the second reading should report")
+	}
+	if l.CPUPercent != 75 || l.CPUCount != 2 {
+		t.Errorf("cpu %.1f%% of %d, want 75%% of 2", l.CPUPercent, l.CPUCount)
+	}
+	if l.Load1 != 1.5 || l.Load15 != 0.25 {
+		t.Errorf("load %v %v", l.Load1, l.Load15)
+	}
+	if l.MemUsed != 5000000*1024 || l.MemAvail != 3000000*1024 {
+		t.Errorf("memory used %d available %d", l.MemUsed, l.MemAvail)
+	}
+	if len(l.Shards) != 1 {
+		t.Fatalf("shards = %+v", l.Shards)
+	}
+	sh := l.Shards[0]
+	// 400 jiffies over 2 seconds at 100 a second is two cores.
+	if sh.ID != "shard-a" || sh.CPUCores != 2 || sh.RSSBytes != 16*int64(os.Getpagesize()) {
+		t.Errorf("shard = %+v", sh)
+	}
+}
+
+func TestNoProcMeansNoLoad(t *testing.T) {
+	s := newLoadSampler(filepath.Join(t.TempDir(), "missing"), time.Now)
+	s.Sample(nil)
+	if _, ok := s.Sample(nil); ok {
+		t.Error("a system without /proc should report nothing, not zeros")
+	}
+}

internal/agent/discovery/sminfo.go

@@ -0,0 +1,172 @@
+package discovery
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"os/exec"
+	"regexp"
+	"strconv"
+	"strings"
+	"time"
+)
+
+// ErrNoDiags means infiniband-diags is not installed, so the subnet manager
+// cannot be asked who it is.
+var ErrNoDiags = errors.New("sminfo not found: install infiniband-diags")
+
+// smLine is sminfo's one line of answer:
+//
+//	sminfo: sm lid 1 sm guid 0x98039b03009fcfd6, activity count 5452 priority 15 state 3 SMINFO_MASTER
+var smLine = regexp.MustCompile(`sm lid (\d+) sm guid (0x[0-9a-fA-F]+), activity count \d+ priority (\d+) state \d+ (\S+)`)
+
+// ParseSMInfo reads sminfo's output.
+func ParseSMInfo(out string) (*SMInfo, error) {
+	m := smLine.FindStringSubmatch(out)
+	if m == nil {
+		return nil, fmt.Errorf("unrecognised sminfo output: %q", strings.TrimSpace(out))
+	}
+	lid, _ := strconv.Atoi(m[1])
+	prio, _ := strconv.Atoi(m[3])
+	return &SMInfo{GUID: normalizeGUID(m[2]), LID: lid, Priority: prio, State: m[4]}, nil
+}
+
+// QuerySM asks, through one port, which subnet manager is master. It is a
+// management datagram to the SM, not a scan: cheap, but not free, which is
+// why the agent asks once a minute rather than every heartbeat.
+func QuerySM(ctx context.Context, hca string, port int) (*SMInfo, error) {
+	bin, err := exec.LookPath("sminfo")
+	if err != nil {
+		return nil, ErrNoDiags
+	}
+	ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
+	defer cancel()
+	out, err := exec.CommandContext(ctx, bin, "-C", hca, "-P", strconv.Itoa(port)).CombinedOutput()
+	if err != nil {
+		return nil, fmt.Errorf("sminfo on %s/%d: %w: %s", hca, port, err, strings.TrimSpace(string(out)))
+	}
+	return ParseSMInfo(string(out))
+}
+
+// AttachSM fills in the subnet manager each active InfiniBand port sees.
+// RoCE has none. The first error is returned for reporting; the ports it
+// could answer for keep their answer.
+func AttachSM(hcas []HCA, query func(hca string, port int) (*SMInfo, error)) error {
+	var first error
+	for i := range hcas {
+		for j := range hcas[i].Ports {
+			p := &hcas[i].Ports[j]
+			if !p.Active() || p.LinkLayer == LinkEthernet {
+				continue
+			}
+			sm, err := query(hcas[i].Name, p.Num)
+			if err != nil {
+				if first == nil {
+					first = err
+				}
+				continue
+			}
+			p.SM = sm
+		}
+	}
+	return first
+}
+
+// SMRecord is one subnet manager on a fabric, as the subnet administrator
+// lists them: master and standbys alike.
+type SMRecord struct {
+	GUID     string `json:"guid"`
+	LID      int    `json:"lid"`
+	Priority int    `json:"priority"`
+	// State is master, standby, discovering or not active.
+	State string `json:"state"`
+}
+
+// smStates are SMInfo's state numbers.
+var smStates = map[string]string{"0": "not active", "1": "discovering", "2": "standby", "3": "master"}
+
+// smirField is one line of saquery's record dump: "LID.........1".
+var smirField = regexp.MustCompile(`^\s*([A-Za-z_]+)\.{2,}\s*(\S+)`)
+
+// ParseSMIR reads `saquery SMIR`: every subnet manager the fabric's subnet
+// administrator knows of, one "SMInfoRecord dump:" block each.
+func ParseSMIR(out string) ([]SMRecord, error) {
+	var recs []SMRecord
+	var cur *SMRecord
+	for _, line := range strings.Split(out, "\n") {
+		if strings.Contains(line, "SMInfoRecord dump") {
+			recs = append(recs, SMRecord{})
+			cur = &recs[len(recs)-1]
+			continue
+		}
+		m := smirField.FindStringSubmatch(line)
+		if m == nil || cur == nil {
+			continue
+		}
+		switch strings.ToLower(m[1]) {
+		case "lid":
+			cur.LID, _ = strconv.Atoi(m[2])
+		case "guid":
+			cur.GUID = normalizeGUID(m[2])
+		case "priority":
+			cur.Priority, _ = strconv.Atoi(m[2])
+		case "smstate":
+			cur.State = smStates[m[2]]
+			if cur.State == "" {
+				cur.State = m[2]
+			}
+		}
+	}
+	out2 := recs[:0]
+	for _, r := range recs {
+		if r.GUID != "" {
+			out2 = append(out2, r)
+		}
+	}
+	if len(out2) == 0 {
+		return nil, fmt.Errorf("no subnet manager records in saquery output")
+	}
+	return out2, nil
+}
+
+// QuerySMs asks the subnet administrator, through one port, for every subnet
+// manager on the fabric.
+func QuerySMs(ctx context.Context, hca string, port int) ([]SMRecord, error) {
+	bin, err := exec.LookPath("saquery")
+	if err != nil {
+		return nil, ErrNoDiags
+	}
+	ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
+	defer cancel()
+	out, err := exec.CommandContext(ctx, bin, "-C", hca, "-P", strconv.Itoa(port), "SMIR").CombinedOutput()
+	if err != nil {
+		return nil, fmt.Errorf("saquery on %s/%d: %w: %s", hca, port, err, strings.TrimSpace(string(out)))
+	}
+	return ParseSMIR(string(out))
+}
+
+// ScanFabric maps the fabric a port is on with ibnetdiscover: every switch,
+// adapter and link. It can take a while on a large fabric, so it is bounded,
+// and its output is capped: a fabric of thousands of ports is a few
+// megabytes of text.
+func ScanFabric(ctx context.Context, hca string, port int) (string, error) {
+	bin, err := exec.LookPath("ibnetdiscover")
+	if err != nil {
+		return "", ErrNoDiags
+	}
+	ctx, cancel := context.WithTimeout(ctx, 2*time.Minute)
+	defer cancel()
+	out, err := exec.CommandContext(ctx, bin, "-C", hca, "-P", strconv.Itoa(port)).Output()
+	if err != nil {
+		var ee *exec.ExitError
+		if errors.As(err, &ee) {
+			return "", fmt.Errorf("ibnetdiscover on %s/%d: %w: %s", hca, port, err, strings.TrimSpace(string(ee.Stderr)))
+		}
+		return "", fmt.Errorf("ibnetdiscover on %s/%d: %w", hca, port, err)
+	}
+	const limit = 16 << 20
+	if len(out) > limit {
+		return "", fmt.Errorf("ibnetdiscover printed %d MB, more than the %d MB a scan may send", len(out)>>20, limit>>20)
+	}
+	return string(out), nil
+}

internal/agent/discovery/topo.go

@@ -0,0 +1,185 @@
+package discovery
+
+import (
+	"context"
+	"fmt"
+	"os/exec"
+	"regexp"
+	"sort"
+	"strconv"
+	"strings"
+)
+
+// Topology is what `nvidia-smi topo -m` says about how a node's GPUs and
+// network adapters are wired together.
+//
+// Read from nvidia-smi rather than NVML so the portable, cgo-free binary knows
+// it too. Without it that binary saw no NVLink at all: every GPU reported no
+// peers, and a tensor-parallel group was chosen by index on machines where
+// NVLink decides whether it runs at full speed.
+type Topology struct {
+	// NVLink is, per GPU, the GPUs it reaches over NVLink and how many links
+	// are bonded to each.
+	NVLink map[int]map[int]int
+	// NICPath is, per GPU, how far each network adapter is, by its HCA name:
+	// PIX, PXB, PHB, NODE or SYS, nearest first. A GPU should talk to the
+	// fabric through the adapter nearest it; one across the socket costs a
+	// trip over the processor interconnect on every transfer.
+	NICPath map[int]map[string]string
+	// NUMA is the NUMA node each GPU is attached to, where reported.
+	NUMA map[int]int
+}
+
+// PathRank orders the relations nvidia-smi reports, nearest first. Lower is
+// closer; an unknown relation ranks last.
+func PathRank(rel string) int {
+	switch strings.ToUpper(rel) {
+	case "PIX":
+		return 0
+	case "PXB":
+		return 1
+	case "PHB":
+		return 2
+	case "NODE":
+		return 3
+	case "SYS":
+		return 4
+	}
+	return 5
+}
+
+// DiscoverTopology runs `nvidia-smi topo -m`. A node without nvidia-smi has
+// no topology to report, which is not an error.
+func DiscoverTopology(ctx context.Context, binary string) (Topology, error) {
+	if binary == "" {
+		if _, err := exec.LookPath("nvidia-smi"); err != nil {
+			return Topology{}, nil
+		}
+		binary = "nvidia-smi"
+	}
+	out, err := exec.CommandContext(ctx, binary, "topo", "-m").Output()
+	if err != nil {
+		return Topology{}, fmt.Errorf("running %s topo -m: %w", binary, err)
+	}
+	return parseTopo(string(out))
+}
+
+// ParseTopology reads `nvidia-smi topo -m` output.
+func ParseTopology(out string) (Topology, error) { return parseTopo(out) }
+
+var (
+	ansi       = regexp.MustCompile(`\x1b\[[0-9;]*m`)
+	nvLinks    = regexp.MustCompile(`^NV(\d+)$`)
+	nicLegend  = regexp.MustCompile(`^\s*(NIC\d+)\s*:\s*(\S+)\s*$`)
+	gpuColumn  = regexp.MustCompile(`^GPU(\d+)$`)
+	numaColumn = "NUMA Affinity"
+)
+
+// parseTopo reads the matrix. Columns are tab-separated; the header names
+// each one — GPUn, then the adapters, as NICn with a legend below on newer
+// drivers or by HCA name on older ones — then CPU and NUMA affinity.
+func parseTopo(out string) (Topology, error) {
+	out = ansi.ReplaceAllString(out, "")
+	lines := strings.Split(out, "\n")
+
+	legend := map[string]string{}
+	for _, l := range lines {
+		if m := nicLegend.FindStringSubmatch(l); m != nil {
+			legend[m[1]] = m[2]
+		}
+	}
+
+	var header []string
+	start := -1
+	for i, l := range lines {
+		cells := splitTabs(l)
+		if len(cells) > 1 && cells[0] == "" && gpuColumn.MatchString(cells[1]) {
+			header, start = cells, i
+			break
+		}
+	}
+	if start < 0 {
+		return Topology{}, fmt.Errorf("no GPU matrix in nvidia-smi topo output")
+	}
+
+	t := Topology{NVLink: map[int]map[int]int{}, NICPath: map[int]map[string]string{}, NUMA: map[int]int{}}
+	for _, l := range lines[start+1:] {
+		cells := splitTabs(l)
+		if len(cells) < 2 {
+			// The matrix ends at the first blank line, before the legend.
+			if strings.TrimSpace(l) == "" {
+				break
+			}
+			continue
+		}
+		m := gpuColumn.FindStringSubmatch(cells[0])
+		if m == nil {
+			continue // an adapter's row: its view of the GPUs is the same matrix
+		}
+		gpu, _ := strconv.Atoi(m[1])
+		for c := 1; c < len(cells) && c < len(header); c++ {
+			col, v := header[c], strings.TrimSpace(cells[c])
+			switch {
+			case gpuColumn.MatchString(col):
+				peer, _ := strconv.Atoi(gpuColumn.FindStringSubmatch(col)[1])
+				if n := nvLinks.FindStringSubmatch(v); n != nil && peer != gpu {
+					links, _ := strconv.Atoi(n[1])
+					if t.NVLink[gpu] == nil {
+						t.NVLink[gpu] = map[int]int{}
+					}
+					t.NVLink[gpu][peer] = links
+				}
+			case col == numaColumn:
+				if n, err := strconv.Atoi(v); err == nil {
+					t.NUMA[gpu] = n
+				}
+			case col == "CPU Affinity" || col == "GPU NUMA ID" || col == "":
+			default:
+				// An adapter: NICn named by the legend, or an HCA by name.
+				hca := col
+				if name, ok := legend[col]; ok {
+					hca = name
+				}
+				if PathRank(v) < 5 {
+					if t.NICPath[gpu] == nil {
+						t.NICPath[gpu] = map[string]string{}
+					}
+					t.NICPath[gpu][hca] = strings.ToUpper(v)
+				}
+			}
+		}
+	}
+	if len(t.NVLink) == 0 && len(t.NICPath) == 0 && len(t.NUMA) == 0 {
+		return t, fmt.Errorf("the nvidia-smi topo matrix had no rows")
+	}
+	return t, nil
+}
+
+func splitTabs(l string) []string {
+	l = strings.TrimRight(l, "\r")
+	cells := strings.Split(l, "\t")
+	for i := range cells {
+		cells[i] = strings.TrimSpace(cells[i])
+	}
+	return cells
+}
+
+// Apply puts the topology on the GPUs it describes. What a GPU already knows
+// — NVLink peers from NVML, which is the more exact source — is kept.
+func (t Topology) Apply(gpus []GPU) {
+	for i := range gpus {
+		g := &gpus[i]
+		if len(g.NVLinkPeers) == 0 {
+			for peer := range t.NVLink[g.Index] {
+				g.NVLinkPeers = append(g.NVLinkPeers, peer)
+			}
+			sort.Ints(g.NVLinkPeers)
+		}
+		if n, ok := t.NUMA[g.Index]; ok {
+			g.NUMANode = n
+		}
+		if paths := t.NICPath[g.Index]; len(paths) > 0 {
+			g.NICPaths = paths
+		}
+	}
+}

internal/agent/discovery/topo_test.go

@@ -0,0 +1,113 @@
+package discovery
+
+import (
+	"reflect"
+	"strings"
+	"testing"
+)
+
+// row joins cells the way nvidia-smi prints them: tab-separated, each padded.
+func row(cells ...string) string { return strings.Join(cells, "\t") }
+
+// dgxH100 is `nvidia-smi topo -m` on an eight-GPU NVSwitch machine with two
+// adapters, as newer drivers print it: NICn columns and a legend naming them.
+var dgxH100 = strings.Join([]string{
+	row("\x1b[4m", "GPU0", "GPU1", "GPU2", "GPU3", "NIC0", "NIC1", "CPU Affinity", "NUMA Affinity", "GPU NUMA ID\x1b[0m"),
+	row("GPU0", " X ", "NV18", "NV18", "NV18", "PXB", "SYS", "0-55,112-167", "0", "N/A"),
+	row("GPU1", "NV18", " X ", "NV18", "NV18", "PXB", "SYS", "0-55,112-167", "0", "N/A"),
+	row("GPU2", "NV18", "NV18", " X ", "NV18", "SYS", "PXB", "56-111,168-223", "1", "N/A"),
+	row("GPU3", "NV18", "NV18", "NV18", " X ", "SYS", "PIX", "56-111,168-223", "1", "N/A"),
+	row("NIC0", "PXB", "PXB", "SYS", "SYS", " X ", "SYS"),
+	row("NIC1", "SYS", "SYS", "PXB", "PIX", "SYS", " X "),
+	"",
+	"Legend:",
+	"",
+	"  X    = Self",
+	"  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)",
+	"  NV#  = Connection traversing a bonded set of # NVLinks",
+	"",
+	"NIC Legend:",
+	"",
+	"  NIC0: mlx5_0",
+	"  NIC1: mlx5_1",
+	"",
+}, "\n")
+
+func TestNVLinkAndAdapterDistanceAreReadFromTopo(t *testing.T) {
+	topo, err := parseTopo(dgxH100)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if got := topo.NVLink[0]; !reflect.DeepEqual(got, map[int]int{1: 18, 2: 18, 3: 18}) {
+		t.Errorf("GPU0's NVLink peers = %v", got)
+	}
+	// The legend turns NIC1 into the adapter's own name.
+	if got := topo.NICPath[3]; !reflect.DeepEqual(got, map[string]string{"mlx5_0": "SYS", "mlx5_1": "PIX"}) {
+		t.Errorf("GPU3's adapters = %v", got)
+	}
+	if topo.NUMA[2] != 1 || topo.NUMA[0] != 0 {
+		t.Errorf("NUMA = %v", topo.NUMA)
+	}
+
+	gpus := []GPU{{Index: 0}, {Index: 3}}
+	topo.Apply(gpus)
+	if !reflect.DeepEqual(gpus[0].NVLinkPeers, []int{1, 2, 3}) || gpus[1].NUMANode != 1 ||
+		gpus[1].NICPaths["mlx5_1"] != "PIX" {
+		t.Errorf("applied = %+v", gpus)
+	}
+}
+
+func TestOlderDriversNameAdaptersInTheHeader(t *testing.T) {
+	// No legend: the columns are the adapters' own names, and NVLink between
+	// pairs only, as on an A100 PCIe box with bridges.
+	out := strings.Join([]string{
+		row("", "GPU0", "GPU1", "GPU2", "GPU3", "mlx5_0", "CPU Affinity", "NUMA Affinity"),
+		row("GPU0", " X ", "NV12", "SYS", "SYS", "NODE", "0-31", "0"),
+		row("GPU1", "NV12", " X ", "SYS", "SYS", "NODE", "0-31", "0"),
+		row("GPU2", "SYS", "SYS", " X ", "NV12", "SYS", "32-63", "1"),
+		row("GPU3", "SYS", "SYS", "NV12", " X ", "SYS", "32-63", "1"),
+		row("mlx5_0", "NODE", "NODE", "SYS", "SYS", " X "),
+		"",
+	}, "\n")
+	topo, err := parseTopo(out)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if !reflect.DeepEqual(topo.NVLink[2], map[int]int{3: 12}) || len(topo.NVLink[0]) != 1 {
+		t.Errorf("NVLink = %v", topo.NVLink)
+	}
+	if topo.NICPath[0]["mlx5_0"] != "NODE" {
+		t.Errorf("adapters = %v", topo.NICPath)
+	}
+}
+
+func TestAPCIeMachineHasNoNVLink(t *testing.T) {
+	out := strings.Join([]string{
+		row("", "GPU0", "GPU1", "CPU Affinity", "NUMA Affinity"),
+		row("GPU0", " X ", "PHB", "0-15", "0"),
+		row("GPU1", "PHB", " X ", "0-15", "0"),
+		"",
+	}, "\n")
+	topo, err := parseTopo(out)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(topo.NVLink) != 0 {
+		t.Errorf("NVLink on a PCIe machine: %v", topo.NVLink)
+	}
+}
+
+func TestNVMLPeersAreKeptOverTopo(t *testing.T) {
+	topo, _ := parseTopo(dgxH100)
+	gpus := []GPU{{Index: 0, NVLinkPeers: []int{1}}}
+	topo.Apply(gpus)
+	if !reflect.DeepEqual(gpus[0].NVLinkPeers, []int{1}) {
+		t.Errorf("NVML's answer was replaced: %v", gpus[0].NVLinkPeers)
+	}
+}
+
+func TestSomethingThatIsNotAMatrixIsAnError(t *testing.T) {
+	if _, err := parseTopo("NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver."); err == nil {
+		t.Error("expected an error")
+	}
+}

internal/agent/engines/engine.go

@@ -63,13 +63,17 @@ type ShardSpec struct {
 	HCAs []string
 	// MgmtInterface is the ethernet interface used for bootstrap traffic.
 	MgmtInterface string
+	// GIDIndex is the RoCE v2 GID index for the HCAs, set by the agent when
+	// they are RoCE; nil on InfiniBand, where NCCL needs none.
+	GIDIndex *int
 
 	HTTPPort int
 	Profile  Profile
 
 	// Pool and Fabric are control plane concepts the agent cannot derive. They
-	// are carried so the agent can report them back after a restart, and are
-	// not used when building a command.
+	// are carried so the agent can report them back after a restart; Fabric
+	// also says, with an "nvl:" prefix, that the replica is inside a
+	// multi-node NVLink domain, which changes NCCL's environment.
 	Pool   string
 	Fabric string
 	// ProfileFP identifies the version of the profile this shard was started
@@ -193,6 +197,10 @@ type Config struct {
 
 // --- shared helpers ---
 
+// NCCLEnv is the environment a shard with this spec is given, for a tool that
+// shows an operator what an engine would see.
+func NCCLEnv(s ShardSpec) map[string]string { return baseEnv(s) }
+
 // baseEnv builds the environment every engine shares.
 //
 // This is the part of §5.3 the plan marks critical. The rules encoded here:
@@ -215,11 +223,24 @@ func baseEnv(s ShardSpec) map[string]string {
 	if len(s.HCAs) > 0 {
 		env["NCCL_IB_HCA"] = strings.Join(s.HCAs, ",")
 		env["NCCL_IB_DISABLE"] = "0"
+		// RoCE: the index of the routable RoCE v2 entry. NCCL's default is a
+		// v1 link-local one on most hosts, and a job on it hangs.
+		if s.GIDIndex != nil {
+			env["NCCL_IB_GID_INDEX"] = strconv.Itoa(*s.GIDIndex)
+		}
 	} else {
 		// Single node, or a node with no fabric. Either way IB is not the path.
 		env["NCCL_IB_DISABLE"] = "1"
 	}
 
+	// Placed inside a multi-node NVLink domain: NCCL is to carry the
+	// replica's collectives over NVLink between its nodes, which it only
+	// does when told to, and needs IMEX for.
+	if s.NNodes > 1 && strings.HasPrefix(s.Fabric, "nvl:") {
+		env["NCCL_MNNVL_ENABLE"] = "1"
+		env["NCCL_CUMEM_ENABLE"] = "1"
+	}
+
 	if s.MgmtInterface != "" {
 		env["NCCL_SOCKET_IFNAME"] = s.MgmtInterface
 		env["GLOO_SOCKET_IFNAME"] = s.MgmtInterface

internal/agent/engines/engines_test.go

@@ -3,6 +3,7 @@ package engines
 import (
 	"fmt"
 	"slices"
+	"strconv"
 	"strings"
 	"testing"
 )
@@ -243,14 +244,41 @@ func TestVLLMRayWorkerRunsOnlyRay(t *testing.T) {
 }
 
 func TestVLLMMPMode(t *testing.T) {
+	// vLLM's multi-node launch: every node the same serve with its rank and
+	// the leader's address, workers headless. Workers used to be given the
+	// leader's address as their own VLLM_HOST_IP, and no rank at all.
 	v := &VLLM{cfg: VLLMConfig{LaunchMode: "mp"}}
-	cmd := buildOne(t, v, multiNodeSpec(0))
-	if argValue(cmd.Args, "--distributed-executor-backend") != "mp" {
-		t.Errorf("backend not set: %v", cmd.Args)
+	for _, rank := range []int{0, 1} {
+		cmd := buildOne(t, v, multiNodeSpec(rank))
+		for flag, want := range map[string]string{
+			"--distributed-executor-backend": "mp", "--nnodes": "2",
+			"--node-rank": strconv.Itoa(rank), "--master-addr": "10.0.0.10", "--master-port": "20001",
+		} {
+			if got := argValue(cmd.Args, flag); got != want {
+				t.Errorf("rank %d: %s = %q, want %q (%v)", rank, flag, got, want, cmd.Args)
+			}
+		}
+		if headless := slices.Contains(cmd.Args, "--headless"); headless != (rank > 0) {
+			t.Errorf("rank %d: headless = %v", rank, headless)
+		}
+		if ip, ok := envMap(cmd.Env)["VLLM_HOST_IP"]; ok && ip == "10.0.0.10" && rank > 0 {
+			t.Errorf("a worker was told the leader's address is its own")
+		}
 	}
-	env := envMap(cmd.Env)
-	if env["VLLM_HOST_IP"] != "10.0.0.10" || env["VLLM_PORT"] != "20001" {
-		t.Errorf("dist init not configured: %q %q", env["VLLM_HOST_IP"], env["VLLM_PORT"])
+}
+
+func TestRoCEGetsItsGIDIndex(t *testing.T) {
+	s := multiNodeSpec(0)
+	idx := 3
+	s.GIDIndex = &idx
+	env := envMap(buildOne(t, &VLLM{cfg: VLLMConfig{LaunchMode: "mp"}}, s).Env)
+	if env["NCCL_IB_GID_INDEX"] != "3" {
+		t.Errorf("NCCL_IB_GID_INDEX = %q", env["NCCL_IB_GID_INDEX"])
+	}
+	// InfiniBand needs none, and must not be given one.
+	env = envMap(buildOne(t, &VLLM{cfg: VLLMConfig{LaunchMode: "mp"}}, multiNodeSpec(0)).Env)
+	if _, ok := env["NCCL_IB_GID_INDEX"]; ok {
+		t.Error("an InfiniBand shard was given a GID index")
 	}
 }
 
@@ -423,3 +451,28 @@ func TestQuantizationOf(t *testing.T) {
 		t.Errorf("a dangling flag should yield nothing, got %q", got)
 	}
 }
+
+func TestAReplicaInAnNVLinkDomainIsToldToUseIt(t *testing.T) {
+	s := multiNodeSpec(0)
+	s.Fabric = "nvl:76b1ad76-7c24-4b6b-9f6c-2a1e0d8c11aa/7"
+	env := envMap(buildOne(t, &VLLM{cfg: VLLMConfig{LaunchMode: "mp"}}, s).Env)
+	if env["NCCL_MNNVL_ENABLE"] != "1" {
+		t.Errorf("NCCL_MNNVL_ENABLE = %q", env["NCCL_MNNVL_ENABLE"])
+	}
+	if env := envMap(buildOne(t, &VLLM{cfg: VLLMConfig{LaunchMode: "mp"}}, multiNodeSpec(0)).Env); env["NCCL_MNNVL_ENABLE"] != "" {
+		t.Error("a replica on InfiniBand was told to use multi-node NVLink")
+	}
+}
+
+func TestMultiNodeVLLMDefaultsToItsOwnLaunch(t *testing.T) {
+	// mp is what vLLM documents and needs nothing but vLLM; Ray was the
+	// default and was never checked against a real cluster.
+	cmds, err := (&VLLM{}).Commands(multiNodeSpec(1))
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(cmds) != 1 || cmds[0].Name != "vllm" || argValue(cmds[0].Args, "--distributed-executor-backend") != "mp" ||
+		!slices.Contains(cmds[0].Args, "--headless") {
+		t.Errorf("default worker = %+v", cmds)
+	}
+}

internal/agent/engines/probe.go

@@ -202,11 +202,7 @@ func (g *SGLang) Configured() bool { return g.cfg.Python != "" }
 
 // Probe reports whether vLLM is installed.
 func (v *VLLM) Probe(context.Context) (string, error) {
-	bin := v.cfg.Bin
-	if bin == "" {
-		bin = "vllm"
-	}
-	return installedBinary(bin)
+	return installedBinary(v.bin())
 }
 
 // Probe reports the SGLang version.
@@ -215,10 +211,6 @@ func (v *VLLM) Probe(context.Context) (string, error) {
 // asking the shell for "sglang" finds nothing on a machine that can run it
 // perfectly well.
 func (g *SGLang) Probe(ctx context.Context) (string, error) {
-	python := g.cfg.Python
-	if python == "" {
-		python = "python3"
-	}
-	return versionOfModule(ctx, python, "-c",
+	return versionOfModule(ctx, g.python(), "-c",
 		"import sglang, sys; sys.stdout.write(getattr(sglang, '__version__', ''))")
 }

internal/agent/engines/sglang.go

@@ -1,6 +1,9 @@
 package engines
 
-import "strconv"
+import (
+	"strconv"
+	"sync/atomic"
+)
 
 // SGLangConfig locates the SGLang installation.
 type SGLangConfig struct {
@@ -11,7 +14,25 @@ type SGLangConfig struct {
 }
 
 // SGLang builds SGLang launch commands.
-type SGLang struct{ cfg SGLangConfig }
+type SGLang struct {
+	cfg       SGLangConfig
+	installed atomic.Pointer[string]
+}
+
+// UseInstalled points the engine at the interpreter of the environment the
+// agent installed and checked, ahead of the configuration.
+func (g *SGLang) UseInstalled(path string) { g.installed.Store(&path) }
+
+// python is the interpreter to launch SGLang with.
+func (g *SGLang) python() string {
+	if p := g.installed.Load(); p != nil {
+		return *p
+	}
+	if g.cfg.Python != "" {
+		return g.cfg.Python
+	}
+	return "python3"
+}
 
 // Name identifies the engine in the catalog.
 func (*SGLang) Name() string { return "sglang" }
@@ -55,10 +76,7 @@ func (g *SGLang) Commands(s ShardSpec) ([]Command, error) {
 }
 
 func (g *SGLang) serverCommand(s ShardSpec) (Command, error) {
-	python := g.cfg.Python
-	if python == "" {
-		python = "python3"
-	}
+	python := g.python()
 	p := s.Profile
 
 	args := []string{"-m", "sglang.launch_server",

internal/agent/engines/vllm.go

@@ -3,24 +3,48 @@ package engines
 import (
 	"fmt"
 	"strconv"
+	"sync/atomic"
 )
 
 // VLLMConfig locates the vLLM installation and picks how it is launched.
 type VLLMConfig struct {
 	// Bin is the vllm executable.
 	Bin string `koanf:"bin"`
-	// LaunchMode selects the multi-node mechanism: "ray" or "mp".
+	// LaunchMode selects the multi-node mechanism: "mp", the default, or
+	// "ray".
 	//
-	// Which one works depends on the vLLM version, and the flags have changed
-	// more than once (plan §14). Both paths exist so a cluster can pick the one
-	// its version supports instead of waiting for a code change.
+	// mp is what vLLM documents today — every node runs serve with its rank,
+	// the workers headless — and needs nothing but vLLM. ray is the older way,
+	// kept for a vLLM too old for mp's multi-node flags; it needs Ray on every
+	// node and has not been checked against a real cluster.
 	LaunchMode string `koanf:"launch_mode"`
 	// RayBin is used when LaunchMode is "ray".
 	RayBin string `koanf:"ray_bin"`
 }
 
 // VLLM builds vLLM launch commands.
-type VLLM struct{ cfg VLLMConfig }
+type VLLM struct {
+	cfg       VLLMConfig
+	installed atomic.Pointer[string]
+}
+
+// UseInstalled points the engine at the environment the agent installed and
+// checked. It takes precedence over the configuration, as llama.cpp's does:
+// it is the one seen to run. An operator who wants their own sets the
+// control plane's engines.vllm to "".
+func (v *VLLM) UseInstalled(path string) { v.installed.Store(&path) }
+
+// bin is the vllm to run: the installed one, else the configured one, else
+// whatever PATH has.
+func (v *VLLM) bin() string {
+	if p := v.installed.Load(); p != nil {
+		return *p
+	}
+	if v.cfg.Bin != "" {
+		return v.cfg.Bin
+	}
+	return "vllm"
+}
 
 // Name identifies the engine in the catalog.
 func (*VLLM) Name() string { return "vllm" }
@@ -78,10 +102,7 @@ func (v *VLLM) Commands(s ShardSpec) ([]Command, error) {
 
 // serverCommand produces the vLLM server process.
 func (v *VLLM) serverCommand(s ShardSpec) (Command, error) {
-	bin := v.cfg.Bin
-	if bin == "" {
-		bin = "vllm"
-	}
+	bin := v.bin()
 	p := s.Profile
 
 	args := []string{"serve", p.ModelPath,
@@ -100,16 +121,27 @@ func (v *VLLM) serverCommand(s ShardSpec) (Command, error) {
 
 	if s.NNodes > 1 {
 		switch v.cfg.LaunchMode {
-		case "", "ray":
+		case "ray":
 			// Ray is started separately by the supervisor; vLLM then finds the
 			// cluster through the address in the environment.
 			env["VLLM_HOST_IP"] = s.LeaderAddr
 			env["RAY_ADDRESS"] = fmt.Sprintf("%s:%d", s.LeaderAddr, s.LeaderPort)
 			args = append(args, "--distributed-executor-backend", "ray")
-		case "mp":
-			env["VLLM_HOST_IP"] = s.LeaderAddr
-			env["VLLM_PORT"] = strconv.Itoa(s.LeaderPort)
-			args = append(args, "--distributed-executor-backend", "mp")
+		case "", "mp":
+			// vLLM's own multi-node launch: every node runs the same serve
+			// with its rank and the leader's address, and the workers run
+			// headless — no API server, only their GPUs. VLLM_HOST_IP is
+			// this node's address for vLLM to find itself, so it is left
+			// to vLLM; the leader's, which workers used to be given, made
+			// every worker announce itself as the leader.
+			args = append(args, "--distributed-executor-backend", "mp",
+				"--nnodes", strconv.Itoa(s.NNodes),
+				"--node-rank", strconv.Itoa(s.Rank),
+				"--master-addr", s.LeaderAddr,
+				"--master-port", strconv.Itoa(s.LeaderPort))
+			if !s.Leader() {
+				args = append(args, "--headless")
+			}
 		default:
 			return Command{}, fmt.Errorf("vllm: unknown launch_mode %q (want \"ray\" or \"mp\")", v.cfg.LaunchMode)
 		}
@@ -124,7 +156,7 @@ func (v *VLLM) serverCommand(s ShardSpec) (Command, error) {
 // rayCommand returns the command that starts or joins the Ray cluster, or
 // ok=false when the launch mode does not use Ray.
 func (v *VLLM) rayCommand(s ShardSpec) (Command, bool) {
-	if s.NNodes <= 1 || (v.cfg.LaunchMode != "" && v.cfg.LaunchMode != "ray") {
+	if s.NNodes <= 1 || v.cfg.LaunchMode != "ray" {
 		return Command{}, false
 	}
 	bin := v.cfg.RayBin

internal/agent/provision/install.go

@@ -34,6 +34,9 @@ const (
 	Ready       = "ready"
 	Failed      = "failed"
 	Unsupported = "unsupported"
+	// Skipped is an engine this node is not meant to run, such as vLLM on a
+	// machine with no GPU. Nothing is wrong, and nothing is shown as wrong.
+	Skipped = "skipped"
 )
 
 // Status is where one engine's installation stands.
@@ -62,9 +65,16 @@ type Installer struct {
 
 	// Run executes a command; a seam for tests. Nil runs it for real.
 	Run func(ctx context.Context, env []string, name string, args ...string) ([]byte, error)
-	// APIBase and DownloadBase default to GitHub; tests point them at a fake.
+	// FreeSpace reports the gigabytes free under a directory; a seam for
+	// tests. Nil asks the filesystem.
+	FreeSpace func(dir string) (float64, error)
+	// RunStdin is Run with input; a seam for tests. Nil runs it for real.
+	RunStdin func(ctx context.Context, env []string, stdin, name string, args ...string) ([]byte, error)
+	// APIBase and DownloadBase default to GitHub, PyPIBase to PyPI; tests
+	// point them at a fake.
 	APIBase      string
 	DownloadBase string
+	PyPIBase     string
 }
 
 // DefaultRoot is where engines are installed when nothing says otherwise.
@@ -75,12 +85,16 @@ const DefaultRoot = "/opt/llmgw/engines"
 const marker = ".llmgw-installed"
 
 type markerFile struct {
-	Engine    string            `json:"engine"`
-	Version   string            `json:"version"`
-	Variant   string            `json:"variant"`
-	Binary    string            `json:"binary"`
-	SHA256    map[string]string `json:"sha256"`
-	Installed time.Time         `json:"installed"`
+	Engine  string            `json:"engine"`
+	Version string            `json:"version"`
+	Variant string            `json:"variant"`
+	Binary  string            `json:"binary"`
+	SHA256  map[string]string `json:"sha256,omitempty"`
+	// For a Python engine: what was asked for, PyPI as of when, by which uv.
+	Requirements []string  `json:"requirements,omitempty"`
+	ExcludeNewer string    `json:"exclude_newer,omitempty"`
+	UV           string    `json:"uv,omitempty"`
+	Installed    time.Time `json:"installed"`
 }
 
 // Converge brings every wanted engine to ready, or says why it cannot.
@@ -110,6 +124,10 @@ func (in *Installer) one(ctx context.Context, w Want) Status {
 	build, err := recipe.Choose(w.Version, in.Host)
 	if err != nil {
 		st.State, st.Detail = Unsupported, err.Error()
+		var nh NotHere
+		if errors.As(err, &nh) {
+			st.State = Skipped
+		}
 		return st
 	}
 	st.Variant = build.Variant
@@ -168,6 +186,9 @@ func (in *Installer) ready(st Status, b Build, bin, version string) Status {
 // Into a scratch directory renamed into place at the end, so a failure halfway
 // leaves nothing that looks installed.
 func (in *Installer) install(ctx context.Context, r Recipe, w Want, b Build, dir string, st *Status) error {
+	if b.Python != "" {
+		return in.installPython(ctx, r, w, b, dir, st)
+	}
 	if err := in.packages(ctx, b.Packages, st); err != nil {
 		return err
 	}
@@ -235,18 +256,32 @@ func (in *Installer) install(ctx context.Context, r Recipe, w Want, b Build, dir
 // running it is the only proof that the rest — the right instruction set, the
 // right driver — is in place.
 func (in *Installer) verify(ctx context.Context, dir string, b Build) (bin, version string, err error) {
-	bin, err = findFile(dir, b.Binary)
-	if err != nil {
-		return "", "", err
-	}
-	if out, err := in.run(ctx, nil, "ldd", bin); err == nil {
-		if missing := lddMissing(string(out)); len(missing) > 0 {
-			return "", "", fmt.Errorf("%s cannot run on this system: %s", b.Binary, strings.Join(missing, "; "))
+	timeout := 60 * time.Second
+	if b.Python != "" {
+		// A script in the environment, not a binary to ldd; and importing
+		// torch and the engine behind it takes a while on a cold disk.
+		bin, timeout = filepath.Join(dir, "bin", b.Binary), 3*time.Minute
+		if _, err := os.Stat(bin); err != nil {
+			return "", "", fmt.Errorf("%s is not in the environment: %w", b.Binary, err)
+		}
+	} else {
+		bin, err = findFile(dir, b.Binary)
+		if err != nil {
+			return "", "", err
+		}
+		if out, err := in.run(ctx, nil, "ldd", bin); err == nil {
+			if missing := lddMissing(string(out)); len(missing) > 0 {
+				return "", "", fmt.Errorf("%s cannot run on this system: %s", b.Binary, strings.Join(missing, "; "))
+			}
 		}
 	}
-	tctx, cancel := context.WithTimeout(ctx, 60*time.Second)
+	args := b.VersionArgs
+	if len(args) == 0 {
+		args = []string{"--version"}
+	}
+	tctx, cancel := context.WithTimeout(ctx, timeout)
 	defer cancel()
-	out, err := in.run(tctx, nil, bin, "--version")
+	out, err := in.run(tctx, nil, bin, args...)
 	if err != nil {
 		return "", "", fmt.Errorf("%s --version failed: %v: %s", b.Binary, err, lastLine(out))
 	}
@@ -290,6 +325,14 @@ func (in *Installer) packages(ctx context.Context, byFamily map[string][]string,
 	return nil
 }
 
+// InstallPackages installs whichever of these packages the machine lacks,
+// from its own package sources, waiting out another package manager's lock.
+// byFamily is keyed by discovery.Host.Family: "deb" or "rpm".
+func (in *Installer) InstallPackages(ctx context.Context, byFamily map[string][]string) error {
+	var st Status
+	return in.packages(ctx, byFamily, &st)
+}
+
 // lockWait bounds how long a package command waits for another to finish.
 const lockWait = 10 * time.Minute
 
@@ -599,6 +642,13 @@ func (in *Installer) report(st Status) {
 	}
 }
 
+func runWithStdin(ctx context.Context, env []string, stdin, name string, args ...string) ([]byte, error) {
+	cmd := exec.CommandContext(ctx, name, args...) //nolint:gosec // fixed commands and paths we installed
+	cmd.Env = append(os.Environ(), env...)
+	cmd.Stdin = strings.NewReader(stdin)
+	return cmd.CombinedOutput()
+}
+
 func (in *Installer) run(ctx context.Context, env []string, name string, args ...string) ([]byte, error) {
 	if in.Run != nil {
 		return in.Run(ctx, env, name, args...)
@@ -654,5 +704,11 @@ func firstVersionLine(b []byte) string {
 			return strings.TrimSpace(l)
 		}
 	}
+	// vllm --version says the number and nothing else.
+	for _, l := range strings.Split(string(b), "\n") {
+		if l = strings.TrimSpace(l); l != "" && l[0] >= '0' && l[0] <= '9' {
+			return l
+		}
+	}
 	return ""
 }

internal/agent/provision/install_test.go

@@ -265,8 +265,8 @@ func TestTheBuildFollowsTheHardware(t *testing.T) {
 
 func TestAnEngineTheAgentCannotInstallSaysSo(t *testing.T) {
 	in := &Installer{Root: t.TempDir(), Host: ubuntu}
-	got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.29.0"}})
-	if got[0].State != Unsupported {
+	got := in.Converge(t.Context(), []Want{{Engine: "tgi", Version: "3.0.0"}})
+	if got[0].State != Unsupported || !strings.Contains(got[0].Detail, "install it by hand") {
 		t.Errorf("got %+v", got[0])
 	}
 }

internal/agent/provision/python.go

@@ -0,0 +1,205 @@
+package provision
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"net/http"
+	"os"
+	"path/filepath"
+	"strings"
+	"syscall"
+	"time"
+)
+
+// A Python engine is a virtual environment built by uv: an interpreter uv
+// fetches, and the engine with every dependency from PyPI.
+//
+// Two things make it as trustworthy as an archive checked against its
+// publisher's checksum. The dependencies are resolved as PyPI stood a day
+// after the engine was published (--exclude-newer), so a node built next
+// month gets what one built today got, not whatever a dependency released
+// since — the usual way a working vLLM stops working. And the resolution is
+// written out with every file's sha256 from PyPI and installed with
+// --require-hashes, so a file that does not match what PyPI published is
+// refused. The resolved list is kept in the environment, as the record of
+// exactly what is installed.
+
+// pythonInstallTimeout bounds one installation: several gigabytes, on a
+// network that may be slow, but not forever.
+const pythonInstallTimeout = time.Hour
+
+// lockName is the resolved requirements, kept in the environment.
+const lockName = "llmgw-requirements.txt"
+
+func (in *Installer) installPython(ctx context.Context, r Recipe, w Want, b Build, dir string, st *Status) error {
+	ctx, cancel := context.WithTimeout(ctx, pythonInstallTimeout)
+	defer cancel()
+
+	if err := in.packages(ctx, b.Packages, st); err != nil {
+		return err
+	}
+	if err := os.MkdirAll(in.root(), 0o755); err != nil {
+		return err
+	}
+	if b.FreeGB > 0 {
+		if free, err := in.free(in.root()); err == nil && free < float64(b.FreeGB) {
+			return fmt.Errorf("%s needs about %d GB free under %s and there are %.1f GB",
+				w.Engine, b.FreeGB, in.root(), free)
+		}
+	}
+
+	in.progress(st, "installing uv %s", UVVersion)
+	uv := in.one(ctx, Want{Engine: "uv", Version: UVVersion})
+	if uv.State != Ready {
+		return fmt.Errorf("installing uv, which installs %s: %s", w.Engine, uv.Detail)
+	}
+
+	in.progress(st, "asking PyPI when %s %s was published", r.PyPI, w.Version)
+	published, err := in.published(ctx, r.PyPI, w.Version)
+	if err != nil {
+		return err
+	}
+	asOf := published.Add(24 * time.Hour).UTC().Format(time.RFC3339)
+
+	cache := filepath.Join(in.root(), ".uv-cache")
+	env := []string{
+		"UV_CACHE_DIR=" + cache,
+		"UV_PYTHON_INSTALL_DIR=" + filepath.Join(in.root(), "python"),
+		// Only the interpreter uv fetched: the system's may be the wrong
+		// version, or change under the environment with the next upgrade.
+		"UV_PYTHON_PREFERENCE=only-managed",
+		// Nobody's uv.toml or pip.conf decides what is installed here.
+		"UV_NO_CONFIG=1",
+	}
+	tmp := dir + ".partial"
+	_ = os.RemoveAll(tmp)
+	defer os.RemoveAll(tmp) //nolint:errcheck // gone either way once renamed
+
+	in.progress(st, "creating a Python %s environment for %s", b.Python, w.Engine)
+	// Relocatable, because it is built under one name and renamed into place
+	// once it works: its scripts find their interpreter relative to
+	// themselves rather than by the path they were built at.
+	if out, err := in.run(ctx, env, uv.Binary, "venv", "--relocatable", "--python", b.Python, tmp); err != nil {
+		return fmt.Errorf("uv venv: %v: %s", err, tail(out))
+	}
+	python := filepath.Join(tmp, "bin", "python")
+
+	in.progress(st, "resolving %s against PyPI as of %s", strings.Join(b.Requirements, " "), asOf[:10])
+	lock := filepath.Join(tmp, lockName)
+	args := []string{"pip", "compile", "--python", python, "--exclude-newer", asOf,
+		"--generate-hashes", "--quiet", "--output-file", lock, "-"}
+	if out, err := in.runStdin(ctx, env, strings.Join(b.Requirements, "\n")+"\n", uv.Binary, args...); err != nil {
+		return fmt.Errorf("resolving %s: %v: %s", strings.Join(b.Requirements, " "), err, tail(out))
+	}
+
+	in.progress(st, "downloading and installing %s and its dependencies: several gigabytes, which takes minutes",
+		strings.Join(b.Requirements, " "))
+	if out, err := in.run(ctx, env, uv.Binary, "pip", "install", "--python", python,
+		"--require-hashes", "--no-deps", "-r", lock); err != nil {
+		return fmt.Errorf("installing %s: %v: %s", strings.Join(b.Requirements, " "), err, tail(out))
+	}
+
+	in.progress(st, "checking that %s runs", w.Engine)
+	if _, _, err := in.verify(ctx, tmp, b); err != nil {
+		return err
+	}
+	m, _ := json.MarshalIndent(markerFile{
+		Engine: w.Engine, Version: w.Version, Variant: b.Variant,
+		Binary: filepath.Join("bin", b.Binary), Requirements: b.Requirements,
+		ExcludeNewer: asOf, UV: UVVersion, Installed: time.Now().UTC(),
+	}, "", "  ")
+	if err := os.WriteFile(filepath.Join(tmp, marker), m, 0o644); err != nil { //nolint:gosec // not secret
+		return err
+	}
+	_ = os.RemoveAll(dir)
+	if err := os.Rename(tmp, dir); err != nil {
+		return err
+	}
+	// The cache is as large as the environment. Kept after a failure, so the
+	// next attempt does not download it all again; dropped after success.
+	_ = os.RemoveAll(cache)
+	in.log().Info("installed an engine", "engine", w.Engine, "version", w.Version,
+		"variant", b.Variant, "dir", dir, "as_of", asOf)
+	return nil
+}
+
+// published is when PyPI received a release's last file.
+func (in *Installer) published(ctx context.Context, project, version string) (time.Time, error) {
+	url := strings.TrimSuffix(in.pypiBase(), "/") + "/pypi/" + project + "/" + version + "/json"
+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
+	if err != nil {
+		return time.Time{}, err
+	}
+	resp, err := in.client().Do(req)
+	if err != nil {
+		return time.Time{}, fmt.Errorf("asking PyPI for %s %s: %w", project, version, err)
+	}
+	defer resp.Body.Close() //nolint:errcheck // read-only
+	if resp.StatusCode == http.StatusNotFound {
+		return time.Time{}, fmt.Errorf("PyPI has no %s %s", project, version)
+	}
+	if resp.StatusCode != http.StatusOK {
+		return time.Time{}, fmt.Errorf("asking PyPI for %s %s: %s", project, version, resp.Status)
+	}
+	var rel struct {
+		URLs []struct {
+			Uploaded time.Time `json:"upload_time_iso_8601"`
+		} `json:"urls"`
+	}
+	if err := json.NewDecoder(resp.Body).Decode(&rel); err != nil {
+		return time.Time{}, fmt.Errorf("reading PyPI's answer for %s %s: %w", project, version, err)
+	}
+	var last time.Time
+	for _, u := range rel.URLs {
+		if u.Uploaded.After(last) {
+			last = u.Uploaded
+		}
+	}
+	if last.IsZero() {
+		return time.Time{}, errors.New("PyPI lists no files for " + project + " " + version)
+	}
+	return last, nil
+}
+
+func (in *Installer) free(dir string) (float64, error) {
+	if in.FreeSpace != nil {
+		return in.FreeSpace(dir)
+	}
+	return FreeGB(dir)
+}
+
+func (in *Installer) pypiBase() string {
+	if in.PyPIBase != "" {
+		return in.PyPIBase
+	}
+	return "https://pypi.org"
+}
+
+// runStdin runs a command with input on its standard input.
+func (in *Installer) runStdin(ctx context.Context, env []string, stdin, name string, args ...string) ([]byte, error) {
+	if in.RunStdin != nil {
+		return in.RunStdin(ctx, env, stdin, name, args...)
+	}
+	return runWithStdin(ctx, env, stdin, name, args...)
+}
+
+// FreeGB is the space available to an unprivileged writer under dir.
+func FreeGB(dir string) (float64, error) {
+	var s syscall.Statfs_t
+	if err := syscall.Statfs(dir, &s); err != nil {
+		return 0, err
+	}
+	return float64(s.Bavail) * float64(s.Bsize) / (1 << 30), nil
+}
+
+// tail is the last few lines of a command's output: uv explains a failed
+// resolution over several, and the last alone is rarely the reason.
+func tail(b []byte) string {
+	lines := strings.Split(strings.TrimSpace(string(b)), "\n")
+	if len(lines) > 4 {
+		lines = lines[len(lines)-4:]
+	}
+	return strings.Join(lines, " / ")
+}

internal/agent/provision/python_test.go

@@ -0,0 +1,184 @@
+package provision
+
+import (
+	"context"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"slices"
+	"strings"
+	"sync"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+)
+
+var gpuNode = discovery.Host{OSID: "ubuntu", OSVersion: "24.04", Arch: "amd64",
+	NVIDIADriver: "580.65.06", MinComputeCapability: 8.0}
+
+// fakePython is uv and the environments it builds.
+type fakePython struct {
+	fakeSystem
+	pmu      sync.Mutex
+	commands []string
+	failPip  bool
+}
+
+func (f *fakePython) run(ctx context.Context, env []string, name string, args ...string) ([]byte, error) {
+	line := name + " " + strings.Join(args, " ")
+	f.pmu.Lock()
+	f.commands = append(f.commands, line)
+	f.pmu.Unlock()
+	switch {
+	case strings.HasSuffix(name, "/uv") && len(args) == 1 && args[0] == "--version":
+		return []byte("uv 0.12.20\n"), nil
+	case strings.HasSuffix(name, "/uv") && args[0] == "venv":
+		return nil, os.MkdirAll(filepath.Join(args[len(args)-1], "bin"), 0o755)
+	case strings.HasSuffix(name, "/uv") && args[0] == "pip" && args[1] == "install":
+		if f.failPip {
+			return []byte("error: Hash mismatch for torch\n"), errExit
+		}
+		venv := filepath.Dir(filepath.Dir(args[slices.Index(args, "--python")+1]))
+		return nil, os.WriteFile(filepath.Join(venv, "bin", "vllm"), []byte("#!/bin/sh\n"), 0o755)
+	case strings.HasSuffix(name, "/bin/vllm") && args[0] == "--version":
+		return []byte("0.30.0\n"), nil
+	}
+	return f.fakeSystem.run(ctx, env, name, args...)
+}
+
+func (f *fakePython) runStdin(_ context.Context, _ []string, stdin, name string, args ...string) ([]byte, error) {
+	f.pmu.Lock()
+	f.commands = append(f.commands, name+" "+strings.Join(args, " ")+" <<< "+strings.TrimSpace(stdin))
+	f.pmu.Unlock()
+	out := args[slices.Index(args, "--output-file")+1]
+	return nil, os.WriteFile(out, []byte("vllm==0.30.0 --hash=sha256:ab\n"), 0o644)
+}
+
+func pythonInstaller(t *testing.T, host discovery.Host, sys *fakePython) (*Installer, *release) {
+	uv := newRelease(map[string][]byte{
+		"uv-x86_64-unknown-linux-gnu.tar.gz": tarGz(t,
+			map[string]string{"uv-x86_64-unknown-linux-gnu/uv": "#!/bin/sh\n"}, nil),
+	})
+	gh := uv.serve(t)
+	mux := http.NewServeMux()
+	mux.HandleFunc("/pypi/vllm/0.30.0/json", func(w http.ResponseWriter, _ *http.Request) {
+		_, _ = w.Write([]byte(`{"urls":[{"upload_time_iso_8601":"2026-09-22T05:22:21.647189Z"},
+			{"upload_time_iso_8601":"2026-09-22T05:23:10.039243Z"}]}`))
+	})
+	pypi := httptest.NewServer(mux)
+	t.Cleanup(pypi.Close)
+	return &Installer{
+		Root: t.TempDir(), Host: host, Run: sys.run, RunStdin: sys.runStdin,
+		APIBase: gh.URL, DownloadBase: gh.URL, PyPIBase: pypi.URL, HTTP: gh.Client(),
+		FreeSpace: func(string) (float64, error) { return 100, nil },
+	}, uv
+}
+
+func TestVLLMIsInstalledOnAGPUNode(t *testing.T) {
+	sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
+	in, _ := pythonInstaller(t, gpuNode, sys)
+
+	got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})
+	if got[0].State != Ready {
+		t.Fatalf("not ready: %+v", got[0])
+	}
+	dir := filepath.Join(in.Root, "vllm", "0.30.0-cu13")
+	if got[0].Binary != filepath.Join(dir, "bin", "vllm") {
+		t.Errorf("binary %s, want the environment's vllm", got[0].Binary)
+	}
+	if !sys.installed["gcc"] {
+		t.Error("gcc, which Triton compiles its launcher with, was not installed")
+	}
+	all := strings.Join(sys.commands, "\n")
+	for _, want := range []string{
+		// PyPI as it stood a day after vLLM's last file, so the dependencies
+		// are the ones it was released against.
+		"--exclude-newer 2026-09-23T05:23:10Z",
+		"--generate-hashes",
+		"--require-hashes",
+		"venv --relocatable --python 3.12",
+	} {
+		if !strings.Contains(all, want) {
+			t.Errorf("no %q in:\n%s", want, all)
+		}
+	}
+	if _, err := os.Stat(filepath.Join(dir, lockName)); err != nil {
+		t.Errorf("the resolved requirements were not kept: %v", err)
+	}
+	if _, err := os.Stat(filepath.Join(in.Root, ".uv-cache")); err == nil {
+		t.Error("the download cache was left behind after a successful install")
+	}
+
+	// Checked again on the next pass, not installed again.
+	before := len(sys.commands)
+	if again := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}}); again[0].State != Ready {
+		t.Fatalf("second pass: %+v", again[0])
+	}
+	for _, c := range sys.commands[before:] {
+		if strings.Contains(c, "pip install") {
+			t.Errorf("a ready environment was installed again: %s", c)
+		}
+	}
+}
+
+func TestVLLMIsNotForEveryNode(t *testing.T) {
+	cases := []struct {
+		name  string
+		host  discovery.Host
+		state string
+		says  string
+	}{
+		{"no GPU", ubuntu, Skipped, "has none"},
+		{"old driver", discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15"}, Unsupported, "driver 580"},
+		{"V100", discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 7.0}, Unsupported, "7.5"},
+	}
+	for _, c := range cases {
+		t.Run(c.name, func(t *testing.T) {
+			sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
+			in, uv := pythonInstaller(t, c.host, sys)
+			got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})
+			if got[0].State != c.state || !strings.Contains(got[0].Detail, c.says) {
+				t.Errorf("got %s %q, want %s saying %q", got[0].State, got[0].Detail, c.state, c.says)
+			}
+			if uv.gets != 0 || len(sys.commands) != 0 {
+				t.Errorf("a node that cannot run it still started installing: %v", sys.commands)
+			}
+		})
+	}
+}
+
+func TestAPythonEngineIsRefusedWithoutTheDiskForIt(t *testing.T) {
+	sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{"gcc": true}}}
+	in, uv := pythonInstaller(t, gpuNode, sys)
+	in.FreeSpace = func(string) (float64, error) { return 12.5, nil }
+	got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})
+	if got[0].State != Failed || !strings.Contains(got[0].Detail, "12.5 GB") {
+		t.Errorf("got %s %q", got[0].State, got[0].Detail)
+	}
+	if uv.gets != 0 {
+		t.Error("it began downloading anyway")
+	}
+}
+
+func TestAFailedPythonInstallLeavesNothingThatLooksInstalled(t *testing.T) {
+	sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{"gcc": true}}, failPip: true}
+	in, _ := pythonInstaller(t, gpuNode, sys)
+	got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})
+	if got[0].State != Failed || !strings.Contains(got[0].Detail, "Hash mismatch") {
+		t.Fatalf("got %s %q", got[0].State, got[0].Detail)
+	}
+	entries, _ := os.ReadDir(filepath.Join(in.Root, "vllm"))
+	for _, e := range entries {
+		t.Errorf("left behind: %s", e.Name())
+	}
+}
+
+func TestAVersionPyPIDoesNotHaveIsSaidSo(t *testing.T) {
+	sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{"gcc": true}}}
+	in, _ := pythonInstaller(t, gpuNode, sys)
+	got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "9.9.9"}})
+	if got[0].State != Failed || !strings.Contains(got[0].Detail, "PyPI has no vllm 9.9.9") {
+		t.Errorf("got %s %q", got[0].State, got[0].Detail)
+	}
+}

internal/agent/provision/recipe.go

@@ -11,6 +11,7 @@ package provision
 
 import (
 	"fmt"
+	"slices"
 	"strings"
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
@@ -33,23 +34,163 @@ type Build struct {
 	// CPUFlags are extensions the build is slow without. Missing ones are
 	// reported, not refused: slow is still running.
 	CPUFlags []string
+
+	// Python, when set, makes the build a Python environment rather than an
+	// archive: that interpreter version, with Requirements installed from
+	// PyPI by uv. Binary is then looked for in the environment's bin.
+	Python       string
+	Requirements []string
+	// VersionArgs is what Binary is run with to prove the build works, when
+	// --version is not it.
+	VersionArgs []string
+	// FreeGB is the disk space the installation needs before it starts: a
+	// Python engine is several gigabytes, and one that fills the disk halfway
+	// leaves the node worse off than one that never began.
+	FreeGB int
 }
 
 // Recipe knows how to install one engine.
 type Recipe struct {
-	// Repo is the GitHub repository whose releases carry the builds.
+	// Repo is the GitHub repository whose releases carry the builds; empty
+	// for an engine installed from PyPI.
 	Repo string
+	// PyPI is the project an engine installed from PyPI is published as.
+	PyPI string
 	// Choose picks the build for a machine, or says why none fits.
 	Choose func(version string, h discovery.Host) (Build, error)
 }
 
 // Recipes are the engines this agent can install.
 //
-// vLLM and SGLang are not here yet: they are Python environments of several
-// gigabytes rather than an archive, and are installed by hand and named in the
-// agent's configuration until they are.
+// vLLM and SGLang are Python environments of several gigabytes, installed
+// from PyPI by uv, which is itself installed like llama.cpp: a GitHub release
+// checked against its published checksum.
 var Recipes = map[string]Recipe{
 	"llamacpp": {Repo: "ggml-org/llama.cpp", Choose: chooseLlamaCpp},
+	"vllm":     {PyPI: "vllm", Choose: chooseVLLM},
+	"sglang":   {PyPI: "sglang", Choose: chooseSGLang},
+	"uv":       {Repo: "astral-sh/uv", Choose: chooseUV},
+}
+
+// Installable are the engines this agent can install, as it tells the
+// control plane: every recipe but uv, which is a tool for the others.
+func Installable() []string {
+	var out []string
+	for name := range Recipes {
+		if name != "uv" {
+			out = append(out, name)
+		}
+	}
+	slices.Sort(out)
+	return out
+}
+
+// UVVersion is the uv that installs the Python engines.
+const UVVersion = "0.12.20"
+
+// pythonForEngines is the interpreter the Python engines run on. uv fetches
+// it, so the node's own Python — or its absence — does not matter.
+const pythonForEngines = "3.12"
+
+// NotHere is the error a recipe gives for a node the engine is not meant
+// for, as opposed to one it should run on and cannot: vLLM on a machine
+// with no GPU is not a fault, and is not reported as one.
+type NotHere struct{ Reason string }
+
+func (e NotHere) Error() string { return e.Reason }
+
+func chooseUV(version string, h discovery.Host) (Build, error) {
+	var arch string
+	switch h.Arch {
+	case "amd64":
+		arch = "x86_64"
+	case "arm64":
+		arch = "aarch64"
+	default:
+		return Build{}, fmt.Errorf("uv publishes no Linux build for %s", orUnknown(h.Arch))
+	}
+	return Build{
+		Variant: arch,
+		Assets:  []string{"uv-" + arch + "-unknown-linux-gnu.tar.gz"},
+		Binary:  "uv",
+	}, nil
+}
+
+// minDriverForCUDA13 is the oldest NVIDIA driver CUDA 13 runs on. Every vLLM
+// since 0.26 and SGLang 0.5 is built on torch for CUDA 13.
+const minDriverForCUDA13 = 580
+
+// minComputeForCUDA13 is the oldest GPU CUDA 13 supports: Turing. It dropped
+// Volta, so a V100 cannot run these builds at all.
+const minComputeForCUDA13 = 7.5
+
+// cudaGate says whether a node can run an engine built on CUDA 13.
+func cudaGate(engine string, h discovery.Host) error {
+	switch h.Arch {
+	case "amd64", "arm64":
+	default:
+		return fmt.Errorf("%s publishes no Linux build for %s", engine, orUnknown(h.Arch))
+	}
+	if h.NVIDIADriver == "" {
+		return NotHere{engine + " runs on NVIDIA GPUs and this node has none; it serves llama.cpp models"}
+	}
+	if major := leadingInt(h.NVIDIADriver); major > 0 && major < minDriverForCUDA13 {
+		return fmt.Errorf("%s is built for CUDA 13, which needs NVIDIA driver %d or newer; this node has %s",
+			engine, minDriverForCUDA13, h.NVIDIADriver)
+	}
+	if cc := h.MinComputeCapability; cc > 0 && cc < minComputeForCUDA13 {
+		return fmt.Errorf("%s is built for CUDA 13, which supports GPUs of compute capability %.1f and newer; this node has %.1f",
+			engine, minComputeForCUDA13, cc)
+	}
+	return nil
+}
+
+// Python engines need a C compiler at run time: Triton builds a small
+// launcher for its kernels the first time a model loads, and without gcc
+// that load fails long after the installation looked finished.
+var compilerPackages = map[string][]string{"deb": {"gcc"}, "rpm": {"gcc"}}
+
+func chooseVLLM(version string, h discovery.Host) (Build, error) {
+	if err := cudaGate("vLLM", h); err != nil {
+		return Build{}, err
+	}
+	return Build{
+		Variant:      "cu13",
+		Python:       pythonForEngines,
+		Requirements: []string{"vllm==" + version},
+		Binary:       "vllm",
+		Packages:     compilerPackages,
+		FreeGB:       25,
+	}, nil
+}
+
+func chooseSGLang(version string, h discovery.Host) (Build, error) {
+	if err := cudaGate("SGLang", h); err != nil {
+		return Build{}, err
+	}
+	// SGLang is launched as a module, so what the agent is given is the
+	// environment's interpreter, and the proof is importing it.
+	return Build{
+		Variant:      "cu13",
+		Python:       pythonForEngines,
+		Requirements: []string{"sglang==" + version},
+		Binary:       "python",
+		VersionArgs:  []string{"-c", "import sglang; print('sglang', sglang.__version__)"},
+		Packages:     compilerPackages,
+		FreeGB:       25,
+	}, nil
+}
+
+// leadingInt is the number a version string starts with: 580 of "580.65.06".
+func leadingInt(s string) int {
+	n := 0
+	for _, c := range strings.TrimSpace(s) {
+		if c < '0' || c > '9' {
+			break
+		}
+		n = n*10 + int(c-'0')
+	}
+	return n
 }
 
 // cudaForLlamaCpp is the CUDA build taken on an NVIDIA node. The release also

internal/agent/supervisor/supervisor.go

@@ -243,6 +243,16 @@ func (s *Supervisor) launch(ctx context.Context, sh *Shard, cmds []engines.Comma
 	}
 
 	cmd := cmds[len(cmds)-1]
+	// The main process can need the rendezvous point too: a Ray worker's
+	// join is its only process, and started before the head listens it
+	// exits, failing the shard and with it the replica.
+	if cmd.WaitForLeaderPort && !sh.Spec.Leader() {
+		addr := fmt.Sprintf("%s:%d", sh.Spec.LeaderAddr, sh.Spec.LeaderPort)
+		if err := waitForPort(runCtx, addr, s.cfg.RendezvousTimeout); err != nil {
+			cancel()
+			return fmt.Errorf("waiting for %s (%s): %w", addr, cmd.Name, err)
+		}
+	}
 
 	c := exec.CommandContext(runCtx, cmd.Path, cmd.Args...)
 	c.Env = cmd.Env

internal/agent/supervisor/supervisor_test.go

@@ -794,3 +794,36 @@ func TestAnAdoptedShardCanBecomeReady(t *testing.T) {
 	}
 	_ = s2.Stop(t.Context(), "a")
 }
+
+func TestAWorkersOnlyProcessWaitsForTheLeader(t *testing.T) {
+	// A Ray worker's join is its only process. The wait was honoured only for
+	// sidecars, so the join started at once, found no head, exited, and took
+	// the replica down with it.
+	s, _ := newSupervisor(t)
+	s.cfg.RendezvousTimeout = 5 * time.Second
+
+	ln, err := net.Listen("tcp", "127.0.0.1:0")
+	if err != nil {
+		t.Fatal(err)
+	}
+	port := ln.Addr().(*net.TCPAddr).Port
+	_ = ln.Close()
+	go func() {
+		time.Sleep(400 * time.Millisecond)
+		l, err := net.Listen("tcp", fmt.Sprintf("127.0.0.1:%d", port))
+		if err == nil {
+			t.Cleanup(func() { _ = l.Close() })
+		}
+	}()
+
+	spec := engines.ShardSpec{Rank: 1, LeaderAddr: "127.0.0.1", LeaderPort: port}
+	cmds := []engines.Command{{Name: "join", Path: "/bin/sh", Args: []string{"-c", "sleep 30"},
+		Env: os.Environ(), WaitForLeaderPort: true}}
+	start := time.Now()
+	if _, err := s.Start(t.Context(), "w", spec, cmds); err != nil {
+		t.Fatal(err)
+	}
+	if d := time.Since(start); d < 300*time.Millisecond {
+		t.Errorf("the worker started after %v, before the leader was listening", d)
+	}
+}

internal/auth/auth.go

@@ -21,6 +21,9 @@ import (
 type Principal struct {
 	KeyID     string
 	KeyPrefix string
+	// UserID is the person the key belongs to, empty for a key an operator
+	// issued in a project. A personal key has a person and no project.
+	UserID    string
 	ProjectID string
 	TeamID    string
 	OrgID     string
@@ -186,7 +189,7 @@ func (a *Authenticator) resolve(ctx context.Context, hash []byte) (Principal, er
 		return Principal{}, err
 	}
 	return Principal{
-		KeyID: k.ID, KeyPrefix: k.Prefix, ProjectID: eff.ProjectID,
+		KeyID: k.ID, KeyPrefix: k.Prefix, UserID: eff.UserID, ProjectID: eff.ProjectID,
 		TeamID: eff.TeamID, OrgID: eff.OrgID,
 		Band: eff.Band, RoutingProfile: eff.RoutingProfile,
 		Limits: eff,

internal/cli/agent.go

@@ -163,10 +163,10 @@ func printInventory(inv discovery.Inventory) {
 				g.Index, g.Class(), g.Name, g.MemoryTotalMB, g.ComputeCapability, health)
 		}
 		fmt.Fprintf(w, "  classes: %v\n", inv.Classes())
-		if !discovery.NVMLSupported() {
-			fmt.Fprintln(w, "  NVLink topology unavailable: rebuild with -tags nvml.")
-			fmt.Fprintln(w, "  Without it, placement cannot keep a tensor-parallel group inside")
-			fmt.Fprintln(w, "  one NVLink domain (plan §6.2).")
+		for _, g := range inv.GPUs {
+			if len(g.NVLinkPeers) > 0 {
+				fmt.Fprintf(w, "  GPU %d NVLink peers: %v\n", g.Index, g.NVLinkPeers)
+			}
 		}
 	}
 

internal/cli/ctl.go

@@ -199,7 +199,8 @@ func newNodesCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 					said = true
 				}
 				for _, p := range n.Provision {
-					if p.State == "ready" {
+					// Skipped is vLLM on a node with no GPU: as it should be.
+					if p.State == "ready" || p.State == "skipped" {
 						continue
 					}
 					fmt.Printf("\n%s: %s %s %s: %s", n.ID, p.Engine, p.Version, p.State, p.Detail)

internal/cli/doctor.go

@@ -0,0 +1,185 @@
+package cli
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"io"
+	"os"
+	"sort"
+	"strings"
+	"text/tabwriter"
+	"time"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/doctor"
+)
+
+func newDoctorCmd() *cobra.Command {
+	var (
+		opts    doctor.Options
+		asJSON  bool
+		verbose bool
+	)
+	cmd := &cobra.Command{
+		Use:   "doctor",
+		Short: "Check this node's GPUs, NVLink, InfiniBand and engines for what llmgw needs",
+		Long: `Reads this node's GPUs, their NVLink and PCIe topology, its InfiniBand and
+RoCE ports, the kernel modules RDMA needs, the agent's configuration and
+whether vLLM can be installed here, and says what is wrong and how to fix
+it — most serious first.
+
+Run it on a node before it joins, and whenever a multi-node replica will not
+come up. --json writes everything it read as well as what it concluded; send
+that file when the report is not enough to go on.`,
+		Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			doctor.SetConfigLoader(config.LoadAgent)
+			if opts.VLLMVersion == "" {
+				opts.VLLMVersion = config.DefaultEngines["vllm"]
+			}
+			ctx, cancel := context.WithTimeout(cmd.Context(), 60*time.Second)
+			defer cancel()
+			facts := doctor.Collect(ctx, opts)
+			findings := doctor.Evaluate(facts)
+			if asJSON {
+				enc := json.NewEncoder(os.Stdout)
+				enc.SetIndent("", "  ")
+				return enc.Encode(struct {
+					Findings []doctor.Finding `json:"findings"`
+					Facts    doctor.Facts     `json:"facts"`
+				}{findings, facts})
+			}
+			printDoctor(os.Stdout, facts, findings, verbose)
+			for _, f := range findings {
+				if f.Level == doctor.Fail {
+					return fmt.Errorf("%d problem(s) need fixing", countLevel(findings, doctor.Fail))
+				}
+			}
+			return nil
+		},
+	}
+	f := cmd.Flags()
+	f.StringVar(&opts.ConfigPath, "config", "", "agent configuration (default /etc/llmgw/agent.yaml when present)")
+	f.StringVar(&opts.IBRoot, "ib-root", discovery.SysfsRoot, "sysfs InfiniBand directory")
+	f.StringVar(&opts.NvidiaSMI, "nvidia-smi", "", "path to nvidia-smi")
+	f.StringVar(&opts.VLLMVersion, "vllm", "", "the vLLM version to check for (default: the one this llmgw installs)")
+	f.BoolVar(&asJSON, "json", false, "write every fact and finding as JSON, to send back")
+	f.BoolVar(&verbose, "verbose", false, "also print the raw nvidia-smi topology")
+	return cmd
+}
+
+func countLevel(fs []doctor.Finding, l doctor.Level) int {
+	n := 0
+	for _, f := range fs {
+		if f.Level == l {
+			n++
+		}
+	}
+	return n
+}
+
+func printDoctor(out io.Writer, f doctor.Facts, findings []doctor.Finding, verbose bool) {
+	fmt.Fprintf(out, "%s · %s %s %s · driver %s\n\n", f.Hostname, f.Host.OSID, f.Host.OSVersion,
+		f.Host.Arch, orDash(f.Host.NVIDIADriver))
+
+	for _, x := range findings {
+		fmt.Fprintf(out, "%-4s  %-6s  %s\n", x.Level, x.Area, x.Message)
+		if x.Fix != "" && (x.Level == doctor.Fail || x.Level == doctor.Warn) {
+			fmt.Fprintf(out, "              → %s\n", x.Fix)
+		}
+	}
+
+	w := tabwriter.NewWriter(out, 0, 0, 2, ' ', 0)
+	if len(f.GPUs) > 0 {
+		fmt.Fprintln(w, "\nGPU\tNAME\tNUMA\tNVLINK PEERS\tNEAREST ADAPTER")
+		for _, g := range f.GPUs {
+			fmt.Fprintf(w, "%d\t%s\t%d\t%s\t%s\n", g.Index, g.Name, g.NUMANode,
+				orDash(ints(g.NVLinkPeers)), orDash(nearest(g.NICPaths)))
+		}
+	}
+	if len(f.HCAs) > 0 {
+		fmt.Fprintln(w, "\nADAPTER\tSTATE\tPHYS\tRATE\tLINK\tNUMA\tROCE GID\tERRORS")
+		for _, h := range f.HCAs {
+			for _, p := range h.Ports {
+				gid := "-"
+				if p.RoCEv2GID >= 0 {
+					gid = fmt.Sprint(p.RoCEv2GID)
+				}
+				fmt.Fprintf(w, "%s/%d\t%s\t%s\t%d Gb/s\t%s\t%d\t%s\t%s\n", h.Name, p.Num, p.State,
+					orDash(p.PhysState), p.RateGbps, p.LinkLayer, h.NUMANode, gid, orDash(errorsOf(p.Counters)))
+			}
+		}
+	}
+	_ = w.Flush()
+
+	for _, fabric := range sortedKeys(f.NCCL) {
+		fmt.Fprintf(out, "\nA shard on %s is given:\n", fabric)
+		env := f.NCCL[fabric]
+		for _, k := range sortedKeys(env) {
+			fmt.Fprintf(out, "  %s=%s\n", k, env[k])
+		}
+	}
+	if len(f.Engines) > 0 {
+		fmt.Fprintln(out, "\nEngines:")
+		for _, k := range sortedKeys(f.Engines) {
+			fmt.Fprintf(out, "  %s: %s\n", k, f.Engines[k])
+		}
+	}
+	if verbose && f.TopoRaw != "" {
+		fmt.Fprintf(out, "\nnvidia-smi topo -m:\n%s\n", f.TopoRaw)
+	}
+	fmt.Fprintf(out, "\n%d failed, %d warnings. `llmgw doctor --json > report.json` keeps everything read, to send back.\n",
+		countLevel(findings, doctor.Fail), countLevel(findings, doctor.Warn))
+}
+
+func nearest(paths map[string]string) string {
+	best, rel := "", ""
+	for h, r := range paths {
+		if best == "" || discovery.PathRank(r) < discovery.PathRank(rel) ||
+			(discovery.PathRank(r) == discovery.PathRank(rel) && h < best) {
+			best, rel = h, r
+		}
+	}
+	if best == "" {
+		return ""
+	}
+	return best + " (" + rel + ")"
+}
+
+func errorsOf(c map[string]uint64) string {
+	var out []string
+	for _, k := range sortedKeys(c) {
+		if c[k] > 0 && k != "port_xmit_data" && k != "port_rcv_data" {
+			out = append(out, fmt.Sprintf("%s=%d", k, c[k]))
+		}
+	}
+	return strings.Join(out, " ")
+}
+
+func ints(v []int) string {
+	s := make([]string, len(v))
+	for i, x := range v {
+		s[i] = fmt.Sprint(x)
+	}
+	return strings.Join(s, ",")
+}
+
+func orDash(s string) string {
+	if s == "" {
+		return "-"
+	}
+	return s
+}
+
+func sortedKeys[V any](m map[string]V) []string {
+	out := make([]string, 0, len(m))
+	for k := range m {
+		out = append(out, k)
+	}
+	sort.Strings(out)
+	return out
+}

internal/cli/keys.go

@@ -6,6 +6,7 @@ import (
 	"encoding/base32"
 	"errors"
 	"fmt"
+	"math"
 	"strings"
 	"text/tabwriter"
 	"time"
@@ -84,7 +85,7 @@ func newKeyInitCmd(open opener) *cobra.Command {
 }
 
 func newKeyCreateCmd(open opener) *cobra.Command {
-	var project, band, profile, baseURL string
+	var project, band, profile, baseURL, forUser string
 	var expires, claimTTL time.Duration
 	var claim bool
 
@@ -107,7 +108,24 @@ func newKeyCreateCmd(open opener) *cobra.Command {
 			}
 			defer s.Close()
 
-			secret, k, err := store.NewKey(randomID("key"), args[0], project)
+			// Three kinds: an operator's key in a project, a person's key in a
+			// project they are a member of, and a person's personal key.
+			var owner store.User
+			if forUser != "" {
+				if owner, err = resolveUser(cmd.Context(), s, forUser); err != nil {
+					return err
+				}
+			}
+			var p store.ProjectPath
+			if project != "" || forUser == "" {
+				if p, err = resolveProject(cmd.Context(), s, project); err != nil {
+					return err
+				}
+			}
+			if p.ID == "" {
+				p.Path = "personal, " + owner.Email
+			}
+			secret, k, err := store.NewKey(randomID("key"), args[0], p.ID)
 			if err != nil {
 				return err
 			}
@@ -116,9 +134,16 @@ func newKeyCreateCmd(open opener) *cobra.Command {
 			if expires > 0 {
 				k.ExpiresAt = k.CreatedAt.Add(expires)
 			}
-			if err := s.CreateKey(cmd.Context(), k); err != nil {
-				return fmt.Errorf("creating the key (does project %q exist? run `llmgw keys init`): %w",
-					project, err)
+			if owner.ID != "" {
+				err = s.CreateUserKey(cmd.Context(), owner.ID, k)
+			} else {
+				err = s.CreateKey(cmd.Context(), k)
+			}
+			if errors.Is(err, store.ErrNotMember) {
+				return fmt.Errorf("%s is not a member of %s; `llmgw users join` first", owner.Email, p.Path)
+			}
+			if err != nil {
+				return fmt.Errorf("creating the key in %s: %w", p.Path, err)
 			}
 
 			// The one output that matters goes to stdout and everything else to
@@ -141,13 +166,13 @@ func newKeyCreateCmd(open opener) *cobra.Command {
 				link := strings.TrimRight(baseURL, "/") + "/claim/" + token
 				fmt.Fprintln(out, link)
 				cmd.PrintErrf("\nid       %s\nproject  %s\nexpires  %s\n",
-					k.ID, k.ProjectID, time.Now().Add(orDefault(claimTTL, store.ClaimTTL)).Format(time.RFC3339))
+					k.ID, p.Path, time.Now().Add(orDefault(claimTTL, store.ClaimTTL)).Format(time.RFC3339))
 				cmd.PrintErrln("\nSend the link. It works once, and the key is shown only to whoever opens it.")
 				return nil
 			}
 
 			fmt.Fprintln(out, secret)
-			cmd.PrintErrf("\nid       %s\nproject  %s\n", k.ID, k.ProjectID)
+			cmd.PrintErrf("\nid       %s\nproject  %s\n", k.ID, p.Path)
 			if !k.ExpiresAt.IsZero() {
 				cmd.PrintErrf("expires  %s\n", k.ExpiresAt.Format(time.RFC3339))
 			}
@@ -155,7 +180,10 @@ func newKeyCreateCmd(open opener) *cobra.Command {
 			return nil
 		},
 	}
-	cmd.Flags().StringVar(&project, "project", "default", "project the key belongs to")
+	cmd.Flags().StringVar(&project, "project", "",
+		"project the key belongs to: id, path (Acme/platform/gateway) or unique name")
+	cmd.Flags().StringVar(&forUser, "for", "",
+		"a person the key belongs to (id or email); without --project it is their personal key")
 	cmd.Flags().StringVar(&band, "band", "", "admission band; empty inherits")
 	cmd.Flags().StringVar(&profile, "profile", "", "routing profile; empty inherits")
 	cmd.Flags().DurationVar(&expires, "expires-in", 0, "lifetime; 0 never expires")
@@ -174,9 +202,10 @@ func orDefault(d, fallback time.Duration) time.Duration {
 
 func newKeyListCmd(open opener) *cobra.Command {
 	var project string
+	var all bool
 	cmd := &cobra.Command{
 		Use:   "list",
-		Short: "List a project's keys",
+		Short: "List a project's keys, or with --all every key and whose it is",
 		Args:  cobra.NoArgs,
 		RunE: func(cmd *cobra.Command, _ []string) error {
 			s, err := open(cmd.Context())
@@ -185,12 +214,35 @@ func newKeyListCmd(open opener) *cobra.Command {
 			}
 			defer s.Close()
 
-			keys, err := s.Keys(cmd.Context(), project)
+			if all {
+				rows, err := s.AllKeys(cmd.Context())
+				if err != nil {
+					return err
+				}
+				w := tabwriter.NewWriter(cmd.OutOrStdout(), 0, 0, 2, ' ', 0)
+				fmt.Fprintln(w, "ID\tNAME\tKEY\tPROJECT\tOWNER\tSTATE\tLAST USED")
+				now := time.Now()
+				for _, r := range rows {
+					where := r.ProjectPath
+					if where == "" {
+						where = "personal"
+					}
+					fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\t%s\n", r.ID, r.Name, r.Prefix,
+						where, dash(r.OwnerEmail), keyState(r.Key, now), ago(r.LastUsedAt, now))
+				}
+				return w.Flush()
+			}
+
+			p, err := resolveProject(cmd.Context(), s, project)
+			if err != nil {
+				return err
+			}
+			keys, err := s.Keys(cmd.Context(), p.ID)
 			if err != nil {
 				return err
 			}
 			if len(keys) == 0 {
-				fmt.Fprintf(cmd.OutOrStdout(), "no keys in project %q\n", project)
+				fmt.Fprintf(cmd.OutOrStdout(), "no keys in %s\n", p.Path)
 				return nil
 			}
 
@@ -204,7 +256,8 @@ func newKeyListCmd(open opener) *cobra.Command {
 			return w.Flush()
 		},
 	}
-	cmd.Flags().StringVar(&project, "project", "default", "project to list")
+	cmd.Flags().StringVar(&project, "project", "", "project to list: id, path or unique name")
+	cmd.Flags().BoolVar(&all, "all", false, "every key in every project, with its owner")
 	return cmd
 }
 
@@ -234,8 +287,9 @@ func newKeyRevokeCmd(open opener) *cobra.Command {
 
 func newLimitsCmd(open opener) *cobra.Command {
 	var scope, id, period string
+	var perMember bool
 	var rpm, tpm int
-	var budget int64
+	var budget float64
 	var models []string
 
 	cmd := &cobra.Command{
@@ -252,7 +306,47 @@ func newLimitsCmd(open opener) *cobra.Command {
 			}
 			defer s.Close()
 
-			l := store.Limits{Scope: store.Scope(scope), ScopeID: id}
+			scopeID := id
+			switch store.Scope(scope) {
+			case store.ScopeOrg:
+				o, err := resolveOrg(cmd.Context(), s, id)
+				if err != nil {
+					return err
+				}
+				scopeID = o.ID
+			case store.ScopeTeam:
+				t, err := resolveTeam(cmd.Context(), s, id)
+				if err != nil {
+					return err
+				}
+				scopeID = t.ID
+			case store.ScopeProject:
+				p, err := resolveProject(cmd.Context(), s, id)
+				if err != nil {
+					return err
+				}
+				scopeID = p.ID
+			case store.ScopeUser:
+				u, err := resolveUser(cmd.Context(), s, id)
+				if err != nil {
+					return err
+				}
+				scopeID = u.ID
+			case store.ScopeKey:
+				if id == "" {
+					return errors.New("--scope key needs --id, the key's id")
+				}
+			default:
+				return fmt.Errorf("--scope is org, team, project, user or key, not %q", scope)
+			}
+			level := store.Scope(scope)
+			if perMember {
+				// Written on the scope, applied to each member separately.
+				if level = store.MemberScope(level); level == "" {
+					return errors.New("--per-member is for --scope org, team or project")
+				}
+			}
+			l := store.Limits{Scope: level, ScopeID: scopeID}
 			if cmd.Flags().Changed("rpm") {
 				l.RPM = &rpm
 			}
@@ -260,7 +354,10 @@ func newLimitsCmd(open opener) *cobra.Command {
 				l.TPM = &tpm
 			}
 			if cmd.Flags().Changed("budget") {
-				l.Budget = &budget
+				// Whole units, as the quotas page takes them; stored as
+				// micro-units so repeated addition does not drift.
+				micros := int64(math.Round(budget * 1e6))
+				l.Budget = &micros
 				l.BudgetPeriod = store.Period(period)
 			}
 			if cmd.Flags().Changed("models") {
@@ -274,15 +371,17 @@ func newLimitsCmd(open opener) *cobra.Command {
 			if err := s.PutLimits(cmd.Context(), l); err != nil {
 				return err
 			}
-			fmt.Fprintf(cmd.OutOrStdout(), "limits set on %s %s\n", scope, id)
+			fmt.Fprintf(cmd.OutOrStdout(), "limits set on %s %s\n", level, scopeID)
 			return nil
 		},
 	}
-	cmd.Flags().StringVar(&scope, "scope", "project", "org | team | project | key")
-	cmd.Flags().StringVar(&id, "id", "default", "id at that scope")
+	cmd.Flags().StringVar(&scope, "scope", "project", "org | team | project | user | key")
+	cmd.Flags().BoolVar(&perMember, "per-member", false,
+		"with org, team or project: a limit for each member there, not one shared by all")
+	cmd.Flags().StringVar(&id, "id", "", "which one: id, path or unique name (a key's id for --scope key)")
 	cmd.Flags().IntVar(&rpm, "rpm", 0, "requests per minute")
 	cmd.Flags().IntVar(&tpm, "tpm", 0, "tokens per minute")
-	cmd.Flags().Int64Var(&budget, "budget", 0, "budget per period, in micro-units")
+	cmd.Flags().Float64Var(&budget, "budget", 0, "budget per period, in whole units as the quotas page shows them")
 	cmd.Flags().StringVar(&period, "period", "month", "day | month")
 	cmd.Flags().StringSliceVar(&models, "models", nil, "allowed models; empty allows none")
 	return cmd

internal/cli/memory.go

@@ -42,7 +42,12 @@ func newMemoryCmd(open opener) *cobra.Command {
 			if off {
 				keep = 0
 			}
-			if err := s.SetMemoryRetention(cmd.Context(), project, keep); err != nil {
+			p, err := resolveProject(cmd.Context(), s, project)
+			if err != nil {
+				return err
+			}
+			project = p.Path
+			if err := s.SetMemoryRetention(cmd.Context(), p.ID, keep); err != nil {
 				return err
 			}
 			if keep == 0 {
@@ -55,7 +60,7 @@ func newMemoryCmd(open opener) *cobra.Command {
 			return nil
 		},
 	}
-	cmd.Flags().StringVar(&project, "project", "default", "project id")
+	cmd.Flags().StringVar(&project, "project", "", "project: id, path or unique name")
 	cmd.Flags().IntVar(&hours, "retain-hours", 0, "how long a note is kept after it was last written")
 	cmd.Flags().BoolVar(&off, "off", false, "stop keeping notes for this project")
 	return cmd

internal/cli/node.go

@@ -0,0 +1,349 @@
+package cli
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"io"
+	"os"
+	"strings"
+	"time"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/version"
+)
+
+func newNodeCmd() *cobra.Command {
+	cmd := &cobra.Command{
+		Use:   "node",
+		Short: "Add machines to the cluster",
+	}
+	cmd.AddCommand(newNodeAddCmd(), newNodeRemoveCmd())
+	return cmd
+}
+
+func newNodeRemoveCmd() *cobra.Command {
+	var admin, serverCfg string
+	cmd := &cobra.Command{
+		Use:   "remove <node>",
+		Short: "Forget a node that is gone for good",
+		Long: `For a machine retired, or rebuilt under another name, which would otherwise
+stay listed as lost until the control plane restarts.
+
+Only a node that is not there is removed: stop its agent first, and wait for
+it to be lost, a minute later. Its certificate stays valid, since the
+authority keeps no revocation list: a machine that still has it can join
+again, which is how a node rebuilt under the same name comes back.`,
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			na := nodeAdd{admin: admin, serverCfg: serverCfg}
+			var out map[string]string
+			if err := na.adminAPI().delete(cmd.Context(), "/api/v1/nodes/"+args[0], &out); err != nil {
+				return err
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "%s removed\n", args[0])
+			return nil
+		},
+	}
+	cmd.Flags().StringVar(&admin, "admin", "", "the control plane's admin API (default: from server.yaml, else 127.0.0.1:9090)")
+	cmd.Flags().StringVar(&serverCfg, "config", "/etc/llmgw/server.yaml", "the control plane's configuration")
+	return cmd
+}
+
+// nodeAdd is what `llmgw node add` was asked to do.
+type nodeAdd struct {
+	opts      onboard.Options
+	fabrics   []string
+	servers   []string
+	admin     string
+	waitAll   bool
+	noWait    bool
+	plain     bool
+	serverCfg string
+}
+
+func newNodeAddCmd() *cobra.Command {
+	var na nodeAdd
+	cmd := &cobra.Command{
+		Use:   "add [user@]address",
+		Short: "Make a machine a node: check it, give it an identity, start its agent",
+		Long: `Run on the machine holding the certificate authority — the control plane,
+after 'llmgw server init' — as root. It logs in to the machine with ssh,
+as you would by hand, and:
+
+  1. installs llmgw there: the release for its architecture, checksum verified
+  2. runs 'llmgw doctor' there, and stops if it finds a problem
+  3. issues the node's certificate and installs it
+  4. writes agent.yaml and the systemd unit, and starts the agent
+  5. waits for the node to join and install llama.cpp
+
+The node's id is its hostname, and the address its agent dials is this
+machine's on the route to it, unless --node or --server say otherwise.
+Nothing else needs giving, except --fabric for InfiniBand or RoCE: which
+fabric an adapter is on has a name only you can choose, and it must be the
+same on every node attached to it. Doctor suggests one when it sees adapters.
+
+The user needs sudo on the machine without a password. Run it again at any
+time: the node gets a fresh certificate and configuration, and its running
+engines are kept.`,
+		Example: `  sudo llmgw node add 10.0.0.21
+  sudo llmgw node add ubuntu@gpu01 --fabric ib-a:mlx5_0,mlx5_1,mlx5_2,mlx5_3`,
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			return na.run(cmd.Context(), args[0], cmd.OutOrStdout())
+		},
+	}
+	f := cmd.Flags()
+	f.StringVar(&na.opts.Node, "node", "", "node id (default: the machine's hostname)")
+	f.StringVar(&na.opts.User, "ssh-user", "", "user to log in as (default: the one before @ in the address, else whoever ran sudo, else you)")
+	f.StringArrayVar(&na.servers, "server", nil, "control plane address the agent dials, https://host:7443; repeatable for failover (default: this machine)")
+	f.StringArrayVar(&na.fabrics, "fabric", nil, "InfiniBand or RoCE fabric and its adapters, name:mlx5_0,mlx5_1; repeatable")
+	f.StringVar(&na.opts.MgmtAddr, "mgmt-addr", "", "address engines are reached at (default: found by the agent)")
+	f.StringVar(&na.opts.MgmtIface, "mgmt-if", "", "interface NCCL bootstraps over (default: the one holding that address)")
+	f.StringVar(&na.opts.LlamaCppBin, "llamacpp-bin", "", "a llama-server you installed, instead of the one the agent installs")
+	f.StringVar(&na.opts.VLLMBin, "vllm-bin", "", "a vllm you installed; also set engines.vllm: \"\" in server.yaml")
+	f.StringVar(&na.opts.SGLangPython, "sglang-python", "", "python of an SGLang virtualenv you installed")
+	f.StringVar(&na.opts.NvidiaSMI, "nvidia-smi", "", "where nvidia-smi is, when it is not on PATH")
+	f.StringVar(&na.opts.IBRoot, "ib-root", "", "the sysfs InfiniBand directory, when not /sys/class/infiniband")
+	f.StringVar(&na.opts.PKI, "pki", defaultPKIDir, "directory holding the certificate authority")
+	f.StringVar(&na.opts.Releases, "releases", "https://gitea.mynodes.xyz/ilker/llmgw/releases/download", "where release files are downloaded from")
+	f.BoolVar(&na.opts.IgnoreChecks, "ignore-checks", false, "add the node even when doctor finds a problem")
+	f.StringVar(&na.admin, "admin", "", "the control plane's admin API, to watch the node join (default: from server.yaml, else 127.0.0.1:9090)")
+	f.StringVar(&na.serverCfg, "config", "/etc/llmgw/server.yaml", "the control plane's configuration, for the admin API's address")
+	f.BoolVar(&na.waitAll, "wait", false, "wait for every engine, vLLM included, not only llama.cpp")
+	f.BoolVar(&na.noWait, "no-wait", false, "do not wait for the node to join")
+	f.BoolVar(&na.plain, "plain", false, "plain output, even in a terminal")
+	return cmd
+}
+
+func (na *nodeAdd) prepare(addr string) error {
+	o := &na.opts
+	o.Addr = addr
+	if user, host, ok := strings.Cut(addr, "@"); ok {
+		o.User, o.Addr = user, host
+	}
+	if o.User == "" {
+		// Under sudo, the person who ran it — their account is usually the
+		// one that exists on the nodes too. root's own keys still do the
+		// logging in, since that is who is running ssh.
+		o.User = os.Getenv("SUDO_USER")
+	}
+	for _, s := range na.fabrics {
+		fab, err := onboard.ParseFabric(s)
+		if err != nil {
+			return err
+		}
+		if len(fab.HCAs) == 0 {
+			return fmt.Errorf("--fabric %s names no adapters: give them as %s:mlx5_0,mlx5_1", s, fab.Name)
+		}
+		o.Fabrics = append(o.Fabrics, fab)
+	}
+	for _, s := range na.servers {
+		for _, one := range strings.Split(s, ",") {
+			if one = strings.TrimSpace(one); one != "" {
+				o.Servers = append(o.Servers, one)
+			}
+		}
+	}
+	o.Version = version.Version
+	if o.Binary == "" {
+		exe, err := os.Executable()
+		if err != nil {
+			return err
+		}
+		o.Binary = exe
+	}
+	onboard.AgentUnit = agentUnit
+	return nil
+}
+
+func (na *nodeAdd) run(ctx context.Context, addr string, out io.Writer) error {
+	if err := na.prepare(addr); err != nil {
+		return err
+	}
+	if !na.plain && isTerminal(out) {
+		return na.runTUI(ctx)
+	}
+	p := &plainProgress{out: out}
+	fmt.Fprintf(out, "adding %s\n", na.opts.Addr)
+	r := onboard.New(na.opts, p.event)
+	res, err := r.Run(ctx)
+	if err != nil {
+		return err
+	}
+	if na.noWait {
+		fmt.Fprintf(out, "\n%s is added. 'llmgw ctl nodes' shows it once it joins.\n", res.Node)
+		return nil
+	}
+	if err := na.watch(ctx, res.Node, p.event); err != nil {
+		return err
+	}
+	fmt.Fprintf(out, "\n%s is in the cluster.\n", res.Node)
+	return nil
+}
+
+// adminAPI is the control plane's admin API, from its configuration when
+// this machine has it.
+func (na *nodeAdd) adminAPI() *ctlClient {
+	addr := na.admin
+	if addr == "" {
+		addr = "127.0.0.1:9090"
+		if cfg, err := config.Load(na.serverCfg); err == nil && cfg.AdminListen != "" {
+			addr = cfg.AdminListen
+			if strings.HasPrefix(addr, ":") {
+				addr = "127.0.0.1" + addr
+			}
+		}
+	}
+	return newCtlClient(addr)
+}
+
+// errNoAdmin is the watch giving up because the admin API is not here, which
+// is not a failure of onboarding.
+var errNoAdmin = errors.New("no admin API")
+
+// watch follows the node from the control plane: joining, then installing
+// its engines. llama.cpp is waited for, since without it the node serves
+// nothing; vLLM takes minutes more and is waited for only with --wait.
+func (na *nodeAdd) watch(ctx context.Context, node string, emit func(onboard.Event)) error {
+	client := na.adminAPI()
+	var nodes []adminapi.NodeView
+	if err := client.get(ctx, "/api/v1/nodes", &nodes); err != nil {
+		emit(onboard.Event{Step: onboard.StepJoin, Level: onboard.Warn,
+			Detail: "cannot reach the control plane's admin API here, so the node's joining is not followed",
+			Fix:    "llmgw ctl nodes on the control plane shows it"})
+		return nil
+	}
+	emit(onboard.Event{Step: onboard.StepJoin, Level: onboard.Running, Detail: "waiting for " + node + " to join"})
+	var view adminapi.NodeView
+	started := time.Now()
+	err := waitFor(ctx, 2*time.Minute, node+" to join", func() (bool, error) {
+		if err := client.get(ctx, "/api/v1/nodes", &nodes); err != nil {
+			return false, nil
+		}
+		for _, n := range nodes {
+			if n.ID == node && heardSince(n, started) {
+				view = n
+				return true, nil
+			}
+		}
+		return false, nil
+	})
+	if err != nil {
+		return fmt.Errorf("%s did not join: its agent's log says why (journalctl -u llmgw-agent on the node); a certificate whose name differs from the node id is refused", node)
+	}
+	emit(onboard.Event{Step: onboard.StepJoin, Level: onboard.Done, Detail: node + " joined: " + describeNode(view)})
+
+	emit(onboard.Event{Step: onboard.StepEngines, Level: onboard.Running, Detail: "installing engines"})
+	seen := map[string]string{}
+	limit := 20 * time.Minute
+	if na.waitAll {
+		limit = time.Hour
+	}
+	err = waitFor(ctx, limit, "the engines on "+node, func() (bool, error) {
+		if err := client.get(ctx, "/api/v1/nodes", &nodes); err != nil {
+			return false, nil
+		}
+		for _, n := range nodes {
+			if n.ID != node {
+				continue
+			}
+			finished := true
+			for _, p := range n.Provision {
+				line := p.State + " " + p.Detail
+				if seen[p.Engine] != line {
+					seen[p.Engine] = line
+					emit(engineEvent(p.Engine, p.Version, p.State, p.Detail))
+				}
+				switch p.State {
+				case "installing":
+					if p.Engine == "llamacpp" || na.waitAll {
+						finished = false
+					}
+				case "failed":
+					if p.Engine == "llamacpp" {
+						return false, fmt.Errorf("llama.cpp did not install: %s", p.Detail)
+					}
+				}
+			}
+			return finished && len(n.Provision) > 0, nil
+		}
+		return false, nil
+	})
+	if err != nil {
+		return err
+	}
+	msg := "engines ready"
+	if v, ok := seen["vllm"]; ok && strings.HasPrefix(v, "installing") && !na.waitAll {
+		msg = "llama.cpp ready; vLLM is still installing, which takes a few minutes more (watch /admin, or add --wait)"
+	}
+	emit(onboard.Event{Step: onboard.StepEngines, Level: onboard.Done, Detail: msg})
+	return nil
+}
+
+// heardSince reports whether the node has been heard from since t, by the
+// agent just started. A node added again is already listed, from its agent
+// before the restart, and a cordoned one is not "ready": neither says whether
+// the new agent connected.
+func heardSince(n adminapi.NodeView, t time.Time) bool {
+	switch n.State {
+	case "unreachable", "lost", "unknown":
+		return false
+	}
+	ago, err := time.ParseDuration(n.LastSeenAgo)
+	if err != nil {
+		return false
+	}
+	// A second's margin: the age is rounded to one.
+	return ago+time.Second < time.Since(t)
+}
+
+func engineEvent(engine, version, state, detail string) onboard.Event {
+	level := onboard.Info
+	switch state {
+	case "failed", "unsupported":
+		level = onboard.Warn
+	}
+	return onboard.Event{Step: onboard.StepEngines, Level: level,
+		Detail: fmt.Sprintf("%s %s: %s%s", engine, version, state, prefixed(": ", detail))}
+}
+
+func prefixed(p, s string) string {
+	if s == "" {
+		return ""
+	}
+	return p + s
+}
+
+func describeNode(n adminapi.NodeView) string {
+	parts := []string{}
+	if n.GPUTotal > 0 {
+		parts = append(parts, fmt.Sprintf("%d GPU(s) %s", n.GPUTotal, strings.Join(n.Classes, ", ")))
+	} else {
+		parts = append(parts, "no GPU")
+	}
+	if len(n.Fabrics) > 0 {
+		parts = append(parts, "fabrics "+strings.Join(n.Fabrics, ", "))
+	}
+	return strings.Join(parts, ", ")
+}
+
+// plainProgress prints events as lines, for a log or a pipe.
+type plainProgress struct {
+	out io.Writer
+}
+
+func (p *plainProgress) event(e onboard.Event) {
+	mark := map[string]string{
+		onboard.Running: "  ..  ", onboard.Done: "  ok  ", onboard.Info: "      ",
+		onboard.Warn: "  !!  ", onboard.Fail: "  FAIL",
+	}[e.Level]
+	fmt.Fprintf(p.out, "%s %-13s %s\n", mark, e.Step, e.Detail)
+	if e.Fix != "" {
+		fmt.Fprintf(p.out, "       %-13s → %s\n", "", e.Fix)
+	}
+}

internal/cli/org.go

@@ -0,0 +1,361 @@
+package cli
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"io"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+func newOrgCmd() *cobra.Command {
+	var sf storeFlags
+	cmd := &cobra.Command{
+		Use:   "org",
+		Short: "Organizations, teams and projects: see, add, rename, move, remove",
+		Long: "The tenancy tree an account lives in. A person is in a project, a project\n" +
+			"in a team, a team in an organization, and limits can be set on any of them.\n\n" +
+			"Anything that takes one of them accepts its id, its path\n" +
+			"(\"Acme/platform/gateway\") or its name when that name is unique.",
+	}
+	sf.bind(cmd)
+	open := func(cmd *cobra.Command) (*store.DB, error) { return sf.open(cmd) }
+	cmd.AddCommand(newOrgTreeCmd(open), newOrgAddCmd(open), newOrgRenameCmd(open),
+		newOrgMoveCmd(open), newOrgRemoveCmd(open))
+	return cmd
+}
+
+type dbOpener func(*cobra.Command) (*store.DB, error)
+
+func newOrgTreeCmd(open dbOpener) *cobra.Command {
+	var people bool
+	cmd := &cobra.Command{
+		Use:   "tree",
+		Short: "Show the whole tree, with how many accounts and keys each project has",
+		Args:  cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			s, err := open(cmd)
+			if err != nil {
+				return err
+			}
+			defer s.Close()
+			tree, err := s.Tree(cmd.Context())
+			if err != nil {
+				return err
+			}
+			var members map[string][]string
+			var alone []store.User
+			if people {
+				users, err := s.Users(cmd.Context(), "")
+				if err != nil {
+					return err
+				}
+				all, err := s.AllMemberships(cmd.Context())
+				if err != nil {
+					return err
+				}
+				email := map[string]string{}
+				for _, u := range users {
+					email[u.ID] = u.Email + "  " + string(u.State)
+					if u.IsAdmin {
+						email[u.ID] += "  admin"
+					}
+				}
+				members = map[string][]string{}
+				in := map[string]bool{}
+				for _, m := range all {
+					members[string(m.Scope)+":"+m.ScopeID] = append(members[string(m.Scope)+":"+m.ScopeID], email[m.UserID])
+					in[m.UserID] = true
+				}
+				for _, u := range users {
+					if !in[u.ID] {
+						alone = append(alone, u)
+					}
+				}
+			}
+			printTree(cmd.OutOrStdout(), tree, members)
+			if people && len(alone) > 0 {
+				fmt.Fprintf(cmd.OutOrStdout(), "\non their own\n")
+				for _, u := range alone {
+					fmt.Fprintf(cmd.OutOrStdout(), "  %s  %s  %s\n", u.Email, u.State, u.ID)
+				}
+			}
+			return nil
+		},
+	}
+	cmd.Flags().BoolVar(&people, "people", false, "list the accounts in each project")
+	return cmd
+}
+
+func printTree(w io.Writer, tree []store.TreeOrg, members map[string][]string) {
+	if len(tree) == 0 {
+		fmt.Fprintln(w, "no organizations yet; `llmgw org add org <name>` starts one")
+		return
+	}
+	list := func(indent, key string) {
+		for _, m := range members[key] {
+			fmt.Fprintf(w, "%s%s\n", indent, m)
+		}
+	}
+	for _, o := range tree {
+		fmt.Fprintf(w, "%s  (%s)%s\n", o.Name, o.ID, membersNote(o.Members))
+		list("    · ", "org:"+o.ID)
+		if len(o.Teams) == 0 {
+			fmt.Fprintln(w, "    no teams")
+		}
+		for _, t := range o.Teams {
+			fmt.Fprintf(w, "  %s  (%s)%s\n", t.Name, t.ID, membersNote(t.Members))
+			list("      · ", "team:"+t.ID)
+			if len(t.Projects) == 0 {
+				fmt.Fprintln(w, "      no projects")
+			}
+			for _, p := range t.Projects {
+				fmt.Fprintf(w, "    %s  (%s)  %s, %s\n", p.Name, p.ID,
+					countOf(p.Users, "member"), keysOf(p))
+				list("        · ", "project:"+p.ID)
+			}
+		}
+	}
+}
+
+func membersNote(n int) string {
+	if n == 0 {
+		return ""
+	}
+	return "  " + countOf(n, "member")
+}
+
+func keysOf(p store.TreeProject) string {
+	s := countOf(p.ActiveKeys, "key")
+	if revoked := p.Keys - p.ActiveKeys; revoked > 0 {
+		s += fmt.Sprintf(" (+%d revoked)", revoked)
+	}
+	return s
+}
+
+func countOf(n int, noun string) string {
+	if n == 1 {
+		return "1 " + noun
+	}
+	return fmt.Sprintf("%d %ss", n, noun)
+}
+
+func levelArg(arg string) (string, error) {
+	switch arg {
+	case "org", "organization":
+		return "org", nil
+	case "team", "project":
+		return arg, nil
+	}
+	return "", fmt.Errorf("%q is not org, team or project", arg)
+}
+
+func newOrgAddCmd(open dbOpener) *cobra.Command {
+	var in string
+	cmd := &cobra.Command{
+		Use:   "add <org|team|project> <name>",
+		Short: "Add an organization, a team to one, or a project to a team",
+		Example: "  llmgw org add org Acme\n" +
+			"  llmgw org add team platform --in Acme\n" +
+			"  llmgw org add project gateway --in Acme/platform",
+		Args: cobra.ExactArgs(2),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			level, err := levelArg(args[0])
+			if err != nil {
+				return err
+			}
+			s, err := open(cmd)
+			if err != nil {
+				return err
+			}
+			defer s.Close()
+			ctx := cmd.Context()
+			var id, where string
+			switch level {
+			case "org":
+				id, err = s.CreateOrg(ctx, args[1])
+			case "team":
+				var o store.Org
+				if o, err = resolveOrg(ctx, s, in); err == nil {
+					id, err = s.CreateTeam(ctx, o.ID, args[1])
+					where = " in " + o.Name
+				}
+			case "project":
+				var t store.TeamPath
+				if t, err = resolveTeam(ctx, s, in); err == nil {
+					id, err = s.CreateProject(ctx, t.ID, args[1])
+					where = " in " + t.OrgName + " / " + t.Name
+				}
+			}
+			if errors.Is(err, store.ErrDuplicateName) {
+				return fmt.Errorf("there is already a %s named %q there", level, args[1])
+			}
+			if err != nil {
+				return err
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "%s %s%s  (%s)\n", level, args[1], where, id)
+			return nil
+		},
+	}
+	cmd.Flags().StringVar(&in, "in", "", "the organization a team goes in, or the team a project goes in")
+	return cmd
+}
+
+func newOrgRenameCmd(open dbOpener) *cobra.Command {
+	return &cobra.Command{
+		Use:     "rename <org|team|project> <which> <new name>",
+		Short:   "Rename an organization, team or project; ids and everything under it stay",
+		Example: "  llmgw org rename project Acme/platform/gateway api-gateway",
+		Args:    cobra.ExactArgs(3),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			level, err := levelArg(args[0])
+			if err != nil {
+				return err
+			}
+			s, err := open(cmd)
+			if err != nil {
+				return err
+			}
+			defer s.Close()
+			id, err := resolveLevel(cmd.Context(), s, level, args[1])
+			if err != nil {
+				return err
+			}
+			switch level {
+			case "org":
+				err = s.RenameOrg(cmd.Context(), id, args[2])
+			case "team":
+				err = s.RenameTeam(cmd.Context(), id, args[2])
+			case "project":
+				err = s.RenameProject(cmd.Context(), id, args[2])
+			}
+			if errors.Is(err, store.ErrDuplicateName) {
+				return fmt.Errorf("there is already a %s named %q beside it", level, args[2])
+			}
+			if err != nil {
+				return err
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "%s %s is now %q\n", level, id, args[2])
+			return nil
+		},
+	}
+}
+
+func newOrgMoveCmd(open dbOpener) *cobra.Command {
+	var to string
+	cmd := &cobra.Command{
+		Use:   "move <team|project> <which> --to <parent>",
+		Short: "Put a team in another organization, or a project in another team",
+		Long: "Everything under it moves too: projects, accounts and keys. From then on it\n" +
+			"inherits the new parent's limits. Usage already recorded is not rewritten.",
+		Example: "  llmgw org move project Acme/platform/gateway --to Scania/ml\n" +
+			"  llmgw org move team platform --to Scania",
+		Args: cobra.ExactArgs(2),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			s, err := open(cmd)
+			if err != nil {
+				return err
+			}
+			defer s.Close()
+			ctx := cmd.Context()
+			switch args[0] {
+			case "team":
+				t, err := resolveTeam(ctx, s, args[1])
+				if err != nil {
+					return err
+				}
+				o, err := resolveOrg(ctx, s, to)
+				if err != nil {
+					return err
+				}
+				if err := s.MoveTeam(ctx, t.ID, o.ID); err != nil {
+					return dupAs(err, "team", t.Name)
+				}
+				fmt.Fprintf(cmd.OutOrStdout(), "team %s is now in %s\n", t.Name, o.Name)
+			case "project":
+				p, err := resolveProject(ctx, s, args[1])
+				if err != nil {
+					return err
+				}
+				t, err := resolveTeam(ctx, s, to)
+				if err != nil {
+					return err
+				}
+				if err := s.MoveProject(ctx, p.ID, t.ID); err != nil {
+					return dupAs(err, "project", p.Path)
+				}
+				fmt.Fprintf(cmd.OutOrStdout(), "%s is now in %s / %s\n", p.Path, t.OrgName, t.Name)
+			default:
+				return fmt.Errorf("only a team or a project moves; to move an account, `llmgw users move`")
+			}
+			return nil
+		},
+	}
+	cmd.Flags().StringVar(&to, "to", "", "the organization or team it goes to")
+	_ = cmd.MarkFlagRequired("to")
+	return cmd
+}
+
+func dupAs(err error, level, name string) error {
+	if errors.Is(err, store.ErrDuplicateName) {
+		return fmt.Errorf("the destination already has a %s named like %s", level, name)
+	}
+	return err
+}
+
+func newOrgRemoveCmd(open dbOpener) *cobra.Command {
+	return &cobra.Command{
+		Use:   "remove <org|team|project> <which>",
+		Short: "Remove an organization, team or project that nothing is in",
+		Long: "Only something empty is removed: an organization with no teams, a team with\n" +
+			"no projects, a project with no accounts and no keys. A project that ever had\n" +
+			"a key keeps it, revoked, because usage history refers to it; rename such a\n" +
+			"project instead. Its limits go with it.",
+		Args: cobra.ExactArgs(2),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			level, err := levelArg(args[0])
+			if err != nil {
+				return err
+			}
+			s, err := open(cmd)
+			if err != nil {
+				return err
+			}
+			defer s.Close()
+			id, err := resolveLevel(cmd.Context(), s, level, args[1])
+			if err != nil {
+				return err
+			}
+			switch level {
+			case "org":
+				err = s.DeleteOrg(cmd.Context(), id)
+			case "team":
+				err = s.DeleteTeam(cmd.Context(), id)
+			case "project":
+				err = s.DeleteProject(cmd.Context(), id)
+			}
+			if err != nil {
+				return fmt.Errorf("%s %s was not removed: %w", level, args[1], err)
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "%s %s removed\n", level, id)
+			return nil
+		},
+	}
+}
+
+func resolveLevel(ctx context.Context, s *store.DB, level, arg string) (string, error) {
+	switch level {
+	case "org":
+		o, err := resolveOrg(ctx, s, arg)
+		return o.ID, err
+	case "team":
+		t, err := resolveTeam(ctx, s, arg)
+		return t.ID, err
+	default:
+		p, err := resolveProject(ctx, s, arg)
+		return p.ID, err
+	}
+}

internal/cli/org_test.go

@@ -0,0 +1,133 @@
+package cli
+
+import (
+	"bytes"
+	"context"
+	"path/filepath"
+	"strings"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// run executes the CLI against one store file and returns stdout.
+func run(t *testing.T, db string, args ...string) (string, error) {
+	t.Helper()
+	cmd := Root()
+	var out, errOut bytes.Buffer
+	cmd.SetOut(&out)
+	cmd.SetErr(&errOut)
+	// --config pointed nowhere, so a server.yaml on the test machine is never read.
+	cmd.SetArgs(append(args, "--store", db, "--config", ""))
+	err := cmd.ExecuteContext(context.Background())
+	return out.String(), err
+}
+
+func mustRun(t *testing.T, db string, args ...string) string {
+	t.Helper()
+	out, err := run(t, db, args...)
+	if err != nil {
+		t.Fatalf("%v: %v", args, err)
+	}
+	return out
+}
+
+func TestTheTreeIsManagedByNameNotId(t *testing.T) {
+	db := filepath.Join(t.TempDir(), "s.db")
+	mustRun(t, db, "org", "add", "org", "Acme")
+	mustRun(t, db, "org", "add", "org", "Scania")
+	mustRun(t, db, "org", "add", "team", "platform", "--in", "Acme")
+	mustRun(t, db, "org", "add", "team", "platform", "--in", "Scania")
+
+	// "platform" is in both, so a project needs the path.
+	_, err := run(t, db, "org", "add", "project", "gateway", "--in", "platform")
+	if err == nil || !strings.Contains(err.Error(), "more than one team") {
+		t.Fatalf("an ambiguous name was accepted: %v", err)
+	}
+	mustRun(t, db, "org", "add", "project", "gateway", "--in", "acme/Platform")
+	mustRun(t, db, "org", "add", "project", "research", "--in", "Scania / platform")
+
+	mustRun(t, db, "users", "create", "ayse@example.com", "--password", "correct horse battery", "--project", "Acme/platform/gateway")
+	mustRun(t, db, "keys", "create", "laptop", "--project", "gateway")
+
+	out := mustRun(t, db, "org", "tree", "--people")
+	for _, want := range []string{"Acme", "gateway", "1 member, 1 key", "ayse@example.com"} {
+		if !strings.Contains(out, want) {
+			t.Errorf("tree lacks %q:\n%s", want, out)
+		}
+	}
+
+	out = mustRun(t, db, "users", "join", "ayse@example.com", "--project", "research")
+	if !strings.Contains(out, "Scania / platform / research") {
+		t.Errorf("join said: %s", out)
+	}
+	mustRun(t, db, "users", "leave", "ayse@example.com", "--project", "Acme/platform/gateway")
+	// She now uses the gateway through research alone; a personal key is hers.
+	mustRun(t, db, "keys", "create", "mine", "--for", "ayse@example.com")
+	if out := mustRun(t, db, "keys", "list", "--all"); !strings.Contains(out, "personal") {
+		t.Errorf("a personal key is not listed as one:\n%s", out)
+	}
+	if _, err := run(t, db, "keys", "create", "x", "--for", "ayse@example.com", "--project", "gateway"); err == nil {
+		t.Error("a key was made for her in a project she left")
+	}
+
+	mustRun(t, db, "org", "rename", "project", "research", "lab")
+	mustRun(t, db, "org", "move", "project", "Acme/platform/gateway", "--to", "Scania/platform")
+
+	// gateway still has the operator's key, so it stays.
+	if _, err := run(t, db, "org", "remove", "project", "gateway"); err == nil {
+		t.Error("a project with a key was removed")
+	}
+	mustRun(t, db, "org", "remove", "team", "Acme/platform")
+	mustRun(t, db, "org", "remove", "org", "Acme")
+
+	s, err := store.Open(context.Background(), db)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer s.Close()
+	paths, _ := s.AllProjects(context.Background())
+	var got []string
+	for _, p := range paths {
+		got = append(got, p.Path)
+	}
+	if strings.Join(got, "|") != "Scania / platform / gateway|Scania / platform / lab" {
+		t.Errorf("projects = %v", got)
+	}
+}
+
+func TestOneProjectNeedsNoFlag(t *testing.T) {
+	db := filepath.Join(t.TempDir(), "s.db")
+	mustRun(t, db, "org", "add", "org", "Acme")
+	mustRun(t, db, "org", "add", "team", "platform")
+	mustRun(t, db, "org", "add", "project", "gateway")
+	mustRun(t, db, "keys", "create", "laptop")
+	if out := mustRun(t, db, "keys", "list"); !strings.Contains(out, "laptop") {
+		t.Errorf("list: %s", out)
+	}
+	mustRun(t, db, "org", "add", "project", "search", "--in", "platform")
+	if _, err := run(t, db, "keys", "create", "other"); err == nil ||
+		!strings.Contains(err.Error(), "which project?") {
+		t.Errorf("two projects and no flag: %v", err)
+	}
+}
+
+func TestKeyBudgetsAreWholeUnits(t *testing.T) {
+	db := filepath.Join(t.TempDir(), "s.db")
+	mustRun(t, db, "org", "add", "org", "Acme")
+	mustRun(t, db, "org", "add", "team", "platform")
+	mustRun(t, db, "org", "add", "project", "gateway")
+	mustRun(t, db, "keys", "limits", "--scope", "project", "--id", "gateway", "--budget", "12.5")
+
+	s, err := store.Open(context.Background(), db)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer s.Close()
+	all, _ := s.AllLimits(context.Background())
+	for _, l := range all {
+		if l.Budget == nil || *l.Budget != 12_500_000 {
+			t.Errorf("budget stored as %v micro-units", l.Budget)
+		}
+	}
+}

internal/cli/resolve.go

@@ -0,0 +1,131 @@
+package cli
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"strings"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// The tree can be built from the web, which mints ids like proj-3kq7..., or
+// from `keys init`, which makes slugs. A command that wanted the id made the
+// operator copy one off a page, and every --project defaulted to "default",
+// which a tree built from the web does not have. These take what a person
+// would type instead: an id, a path, or a name that is unique.
+
+// normPath lowercases a path and spaces its separators the way the store joins
+// them, so "acme/platform/gateway" matches "Acme / platform / gateway".
+func normPath(p string) string {
+	parts := strings.Split(p, "/")
+	for i := range parts {
+		parts[i] = strings.ToLower(strings.TrimSpace(parts[i]))
+	}
+	return strings.Join(parts, " / ")
+}
+
+type candidate struct{ id, path string }
+
+// pick finds the one candidate an argument names.
+func pick(kind, arg string, all []candidate) (candidate, error) {
+	if arg == "" {
+		if len(all) == 1 {
+			return all[0], nil
+		}
+		if len(all) == 0 {
+			return candidate{}, fmt.Errorf("there is no %s yet; create one with `llmgw org add %s`", kind, kind)
+		}
+		return candidate{}, fmt.Errorf("which %s? give an id, a path or a name:\n%s", kind, listCandidates(all))
+	}
+	for _, c := range all {
+		if c.id == arg {
+			return c, nil
+		}
+	}
+	want := normPath(arg)
+	var matches []candidate
+	for _, c := range all {
+		full := normPath(c.path)
+		segs := strings.Split(full, " / ")
+		if full == want || segs[len(segs)-1] == want {
+			matches = append(matches, c)
+		}
+	}
+	switch len(matches) {
+	case 1:
+		return matches[0], nil
+	case 0:
+		return candidate{}, fmt.Errorf("no %s %q; there is:\n%s", kind, arg, listCandidates(all))
+	default:
+		return candidate{}, fmt.Errorf("%q names more than one %s; give its path:\n%s",
+			arg, kind, listCandidates(matches))
+	}
+}
+
+func listCandidates(cs []candidate) string {
+	var b strings.Builder
+	for _, c := range cs {
+		fmt.Fprintf(&b, "  %-40s %s\n", c.path, c.id)
+	}
+	return strings.TrimRight(b.String(), "\n")
+}
+
+func resolveProject(ctx context.Context, s *store.DB, arg string) (store.ProjectPath, error) {
+	all, err := s.AllProjects(ctx)
+	if err != nil {
+		return store.ProjectPath{}, err
+	}
+	cs := make([]candidate, len(all))
+	for i, p := range all {
+		cs[i] = candidate{p.ID, p.Path}
+	}
+	c, err := pick("project", arg, cs)
+	return store.ProjectPath{ID: c.id, Path: c.path}, err
+}
+
+func resolveTeam(ctx context.Context, s *store.DB, arg string) (store.TeamPath, error) {
+	all, err := s.AllTeams(ctx)
+	if err != nil {
+		return store.TeamPath{}, err
+	}
+	cs := make([]candidate, len(all))
+	for i, t := range all {
+		cs[i] = candidate{t.ID, t.OrgName + " / " + t.Name}
+	}
+	c, err := pick("team", arg, cs)
+	if err != nil {
+		return store.TeamPath{}, err
+	}
+	for _, t := range all {
+		if t.ID == c.id {
+			return t, nil
+		}
+	}
+	return store.TeamPath{}, store.ErrNotFound
+}
+
+func resolveOrg(ctx context.Context, s *store.DB, arg string) (store.Org, error) {
+	all, err := s.AllOrgs(ctx)
+	if err != nil {
+		return store.Org{}, err
+	}
+	cs := make([]candidate, len(all))
+	for i, o := range all {
+		cs[i] = candidate{o.ID, o.Name}
+	}
+	c, err := pick("org", arg, cs)
+	return store.Org{ID: c.id, Name: c.path}, err
+}
+
+// resolveUser takes an account's id or its email.
+func resolveUser(ctx context.Context, s *store.DB, arg string) (store.User, error) {
+	if u, err := s.UserByID(ctx, arg); err == nil {
+		return u, nil
+	}
+	u, err := s.UserByEmail(ctx, arg)
+	if errors.Is(err, store.ErrNotFound) {
+		return u, fmt.Errorf("no account with id or email %q; `llmgw users list` shows them", arg)
+	}
+	return u, err
+}

internal/cli/root.go

@@ -42,8 +42,11 @@ func Root() *cobra.Command {
 		newCtlCmd(),
 		newTenancyCmd(),
 		newUsersCmd(),
+		newOrgCmd(),
 		newMirrorCmd(),
 		newSimEngineCmd(),
+		newDoctorCmd(),
+		newNodeCmd(),
 	)
 	return root
 }

internal/cli/servable_test.go

@@ -0,0 +1,52 @@
+package cli
+
+import (
+	"bytes"
+	"context"
+	"encoding/binary"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+)
+
+func ggufAt(t *testing.T, arch string) string {
+	t.Helper()
+	var b bytes.Buffer
+	w := func(v any) { _ = binary.Write(&b, binary.LittleEndian, v) }
+	str := func(s string) { w(uint64(len(s))); b.WriteString(s) }
+	b.WriteString("GGUF")
+	w(uint32(3))
+	w(uint64(0))
+	w(uint64(1))
+	str("general.architecture")
+	w(uint32(8))
+	str(arch)
+	body := b.Bytes()
+	srv := httptest.NewServer(http.HandlerFunc(func(rw http.ResponseWriter, r *http.Request) {
+		http.ServeContent(rw, r, "m.gguf", time.Time{}, bytes.NewReader(body))
+	}))
+	t.Cleanup(srv.Close)
+	return srv.URL + "/m.gguf"
+}
+
+func TestAModelTheNodesCannotLoadIsNotCatalogued(t *testing.T) {
+	check := servableCheck("b10991")
+	llama := []catalog.Profile{{Name: "cpu", Engine: "llamacpp"}}
+
+	err := check(context.Background(), catalog.Model{Name: "kev", Source: ggufAt(t, "ggmlc"), Profiles: llama})
+	if err == nil || !strings.Contains(err.Error(), "ggmlc") {
+		t.Errorf("a ggmlc model was accepted: %v", err)
+	}
+	if err := check(context.Background(), catalog.Model{Name: "q", Source: ggufAt(t, "qwen2"), Profiles: llama}); err != nil {
+		t.Errorf("a qwen2 model was refused: %v", err)
+	}
+	// Not llama.cpp's to judge: vLLM reads a repository, not a GGUF header.
+	vllm := []catalog.Profile{{Name: "gpu", Engine: "vllm"}}
+	if err := check(context.Background(), catalog.Model{Name: "v", Source: ggufAt(t, "ggmlc"), Profiles: vllm}); err != nil {
+		t.Errorf("a vLLM profile was judged by llama.cpp's list: %v", err)
+	}
+}

internal/cli/server.go

@@ -5,6 +5,8 @@ import (
 	"crypto/tls"
 	"errors"
 	"fmt"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/sourcecheck"
 	"log/slog"
 	"net"
 	"net/http"
@@ -150,6 +152,7 @@ func newServerCmd() *cobra.Command {
 				// rather than accepting a model that vanishes on restart.
 				if st := rt.Store(); st != nil {
 					svc := modelsvc.New(cfg.Models, st, admin, log)
+					svc.SetCheck(servableCheck(cfg.WantedEngines()["llamacpp"]))
 					// Rebuild the catalog from configuration plus what was
 					// stored, and put that in force — otherwise models added
 					// before the last restart would be silently absent until
@@ -204,6 +207,11 @@ func newServerCmd() *cobra.Command {
 					web.NavExtras{History: history, Window: cfg.Monitoring.Window}, log))
 				adminMux.Handle("GET /_static/", web.Static())
 				adminMux.HandleFunc("GET /nodes/{id}/logs", web.NodeLogsPage(api, true, version.Version))
+				adminMux.HandleFunc("GET /nodes/{id}", web.NodePage(api, history, cfg.Monitoring.Window, true, version.Version, log))
+				adminMux.HandleFunc("GET /nvlink", web.NVLinkPage(api, true, version.Version))
+				adminMux.HandleFunc("GET /fabrics", web.FabricsPage(api, true, version.Version))
+				adminMux.HandleFunc("GET /fabrics/{id}", web.FabricsPage(api, true, version.Version))
+				adminMux.HandleFunc("GET /fabrics/{id}/topology.json", web.FabricTopologyJSON(api))
 
 				// The same page on the client-facing port at /admin, behind a
 				// session and an admin flag. Handing the router the data rather
@@ -413,6 +421,8 @@ func startControlPlane(ctx context.Context, cfg config.Server, cat *catalog.Live
 		svc.SetLeadership(lead)
 	}
 	fwd.set(svc)
+	// Every InfiniBand fabric mapped every five minutes, through one node.
+	go svc.ScanFabrics(ctx, 5*time.Minute)
 
 	ctl.Start(ctx)
 	return ctl, svc, nil
@@ -535,3 +545,27 @@ func parseEndpointFlag(spec string) (config.Endpoint, error) {
 		Weight: 1,
 	}, nil
 }
+
+// servableCheck refuses a model whose llama.cpp weights the nodes' build
+// cannot load, read from the weights' own header before anything is placed.
+func servableCheck(llamaVersion string) func(context.Context, catalog.Model) error {
+	return func(ctx context.Context, m catalog.Model) error {
+		if llamaVersion == "" {
+			return nil
+		}
+		usesLlama := false
+		for _, p := range m.Profiles {
+			if p.Engine == "llamacpp" {
+				usesLlama = true
+			}
+		}
+		url, ok := sourcecheck.HTTPSource(m.Source)
+		if !usesLlama || !ok || !gguf.IsGGUF(url) {
+			return nil
+		}
+		if v := gguf.Check(ctx, url, llamaVersion); !v.Servable {
+			return fmt.Errorf("%s cannot be served: %s", m.Name, v.Reason)
+		}
+		return nil
+	}
+}

internal/cli/serverinit.go

@@ -66,6 +66,17 @@ type serverInit struct {
 	user  string
 	hosts []string
 	admin string
+	// postgres keeps the store in a PostgreSQL on this machine, installed if
+	// need be, moving whatever SQLite already holds.
+	postgres bool
+	// noMirror, noNode and noModel turn off the parts of a first install that
+	// make it serve something: a mirror, this machine as a node, and a small
+	// model on it.
+	noMirror, noNode, noModel bool
+	// pgSetup prepares PostgreSQL and returns the store address; a seam for
+	// tests. Nil does it for real.
+	pgSetup func(context.Context) (string, error)
+	pg      pgServer
 
 	out io.Writer
 
@@ -88,6 +99,15 @@ func newServerInitCmd() *cobra.Command {
 			"control plane's certificate for every address this machine has, a\n" +
 			"configuration, an administrator account, and the systemd unit — then\n" +
 			"starts it and waits until it answers.\n\n" +
+			"Then it makes the machine serve something: a model mirror beside the\n" +
+			"control plane, this machine as a node of it (on its GPUs when nvidia-smi\n" +
+			"finds any, on CPU otherwise), and " + starterName + " fetched into the\n" +
+			"mirror and served on this node. --no-mirror, --no-node and --no-model\n" +
+			"leave those out.\n\n" +
+			"With --postgres the store is a PostgreSQL database on this machine:\n" +
+			"installed if there is none, with its own role and a generated password\n" +
+			"kept in /etc/llmgw/secrets. An existing SQLite store is copied into it,\n" +
+			"so accounts and keys survive, and server.yaml is pointed at it.\n\n" +
 			"Each step is skipped when its result already exists, so it is safe to run\n" +
 			"again: to finish an install that stopped, or with another --host to\n" +
 			"reissue the certificate. It never overwrites server.yaml or the authority.\n\n" +
@@ -110,6 +130,11 @@ func newServerInitCmd() *cobra.Command {
 		"an address agents dial besides the ones found on this machine; repeatable")
 	f.StringVar(&in.admin, "admin", "", "email of the first administrator; a password is generated and printed once")
 	f.StringVar(&in.pkiDir, "pki", in.pkiDir, "directory holding the certificate authority")
+	f.BoolVar(&in.noMirror, "no-mirror", false, "do not run a model mirror on this machine")
+	f.BoolVar(&in.noNode, "no-node", false, "do not make this machine a node (and serve no model on it)")
+	f.BoolVar(&in.noModel, "no-model", false, "do not fetch and serve the starter model ("+starterName+")")
+	f.BoolVar(&in.postgres, "postgres", false,
+		"keep the store in PostgreSQL on this machine, installing it and moving an existing SQLite store into it")
 	return cmd
 }
 
@@ -122,6 +147,7 @@ func (in *serverInit) run(ctx context.Context) error {
 		{"directories", in.ensureDirs},
 		{"certificates", in.ensureCerts},
 		{"configuration", in.ensureConfig},
+		{"database", in.ensureDatabase},
 		{"store", in.ensureStore},
 	}
 	for _, s := range steps {
@@ -132,9 +158,33 @@ func (in *serverInit) run(ctx context.Context) error {
 	if in.unitPath == "" {
 		return nil
 	}
+	hasSystemd := true
+	if _, err := exec.LookPath("systemctl"); err != nil {
+		hasSystemd = false
+	}
+	// Before the service starts, because the mirror goes into server.yaml.
+	if hasSystemd && !in.noMirror {
+		if err := in.ensureMirror(ctx); err != nil {
+			return fmt.Errorf("mirror: %w", err)
+		}
+	}
 	if err := in.ensureService(ctx); err != nil {
 		return fmt.Errorf("service: %w", err)
 	}
+	if hasSystemd && !in.noNode {
+		if _, err := in.ensureNode(ctx); err != nil {
+			return fmt.Errorf("node: %w", err)
+		}
+		switch {
+		case in.noModel:
+		case in.noMirror:
+			fmt.Fprintln(in.out, "  skip  no starter model without the mirror")
+		default:
+			if err := in.ensureStarter(ctx); err != nil {
+				return fmt.Errorf("starter model: %w", err)
+			}
+		}
+	}
 	in.nextSteps()
 	return nil
 }
@@ -351,7 +401,7 @@ func (in *serverInit) ensureConfig(context.Context) error {
 			contact = "platform@example.com"
 		}
 		body := fmt.Sprintf(serverConfigTemplate, in.etcDir, in.etcDir, in.etcDir,
-			filepath.Join(in.stateDir, "llmgw.db"), contact)
+			in.storeAddress(), contact)
 		if err := os.WriteFile(path, []byte(body), 0o640); err != nil {
 			return err
 		}
@@ -393,7 +443,7 @@ control:
 
 auth:
   enabled: true
-  store: %s
+  store: "%s"   # a SQLite path, or @file holding a PostgreSQL address
   signup_open: true
   # Applied to any key no scope gives a limit to. Without them, a project
   # created without limits is metered by nothing at all.
@@ -589,21 +639,17 @@ func (in *serverInit) waitHealthy(ctx context.Context) error {
 }
 
 func (in *serverInit) nextSteps() {
-	server := "https://<this machine>:7443"
-	if ips := localAddrs(); len(ips) > 0 {
-		server = "https://" + net.JoinHostPort(ips[0].String(), "7443")
-	}
-	fmt.Fprintf(in.out, `
+	fmt.Fprint(in.out, `
 The control plane is up. Next:
 
-  llmgw ctl status                      # no nodes, no models: correct for now
+  llmgw ctl status                      # the fleet in one answer
 
   # each node, from this machine (it holds the authority):
-  deploy/bootstrap/onboard-node.sh --node <name> --addr <node ip> --server %s
+  sudo llmgw node add <user>@<node address>
 
-  # then a model:
+  # then a model: /admin/mirror, "Explore models", or
   llmgw ctl model add -f model.yaml
 
 The web UI is on port 8080; /admin once you are signed in as the administrator.
-`, server)
+`)
 }

internal/cli/serverinit_cluster.go

@@ -0,0 +1,519 @@
+package cli
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"net"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"slices"
+	"strings"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/sourcecheck"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
+)
+
+// A control plane on its own serves nothing: it needs a node, weights, and a
+// model, and the install guide made each of those a separate afternoon. init
+// makes the machine all three — a mirror, a node, and a small model served on
+// it — so the first thing anyone sees is an answer. Each is skipped when it is
+// already there and can be turned off.
+
+const (
+	mirrorDir      = "/srv/models"
+	mirrorAPI      = "127.0.0.1:8082"
+	mirrorUnitPath = "/etc/systemd/system/llmgw-mirror.service"
+	agentUnitPath  = "/etc/systemd/system/llmgw-agent.service"
+
+	// The starter: small enough to download in a minute and run on any CPU,
+	// real enough to answer a question. Q4_K_M is about 400 MB.
+	starterName    = "qwen2.5-0.5b-instruct"
+	starterAlias   = "qwen-0.5b"
+	starterSource  = "hf:Qwen/Qwen2.5-0.5B-Instruct-GGUF/qwen2.5-0.5b-instruct-q4_k_m.gguf"
+	starterVersion = "starter"
+
+	// localLabel marks the control plane's own node, so the starter model is
+	// served where it was asked for and does not wander to the first GPU node
+	// that joins later.
+	localLabelKey   = "llmgw.control-plane"
+	localLabelValue = "true"
+)
+
+// agentUnit is deploy/systemd/llmgw-agent.service; a test holds them together.
+const agentUnit = `[Unit]
+Description=llmgw node agent
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+ExecStart=/usr/local/bin/llmgw agent --config /etc/llmgw/agent.yaml
+# Root, because the agent launches engines and needs device access. Engines
+# outlive a restart of this unit and are adopted again on the next start, so
+# KillMode must not take the whole process group with it.
+KillMode=process
+Restart=on-failure
+RestartSec=2s
+# RDMA pins the buffers it registers, and NCCL registers a great deal: with
+# systemd's default 8 MB lock limit, an engine that drops the capability that
+# lets root ignore it fails its first collective over InfiniBand.
+LimitMEMLOCK=infinity
+StateDirectory=llmgw-agent
+
+[Install]
+WantedBy=multi-user.target
+`
+
+// reachableAddr is the address other machines use for this one: the first
+// name given with --host, else the first address it has, else its hostname.
+// It is in the server certificate either way.
+func (in *serverInit) reachableAddr() string {
+	if len(in.hosts) > 0 {
+		return in.hosts[0]
+	}
+	if ips := localAddrs(); len(ips) > 0 {
+		return ips[0].String()
+	}
+	name, _ := os.Hostname()
+	return name
+}
+
+func (in *serverInit) systemctl(ctx context.Context, args ...string) error {
+	out, err := exec.CommandContext(ctx, "systemctl", args...).CombinedOutput()
+	if err != nil {
+		return fmt.Errorf("systemctl %s: %v: %s", strings.Join(args, " "), err, strings.TrimSpace(string(out)))
+	}
+	return nil
+}
+
+// writeUnit installs a unit when it differs from what is there, keeping
+// somebody's edited copy beside it, and says whether it changed.
+func (in *serverInit) writeUnit(path, body string) (bool, error) {
+	b, err := os.ReadFile(path)
+	if err == nil && string(b) == body {
+		return false, nil
+	}
+	if err == nil {
+		_ = os.WriteFile(path+".bak", b, 0o644)
+	}
+	return true, os.WriteFile(path, []byte(body), 0o644)
+}
+
+// --- the mirror ---
+
+func (in *serverInit) mirrorUnit() string {
+	base := "http://" + net.JoinHostPort(in.reachableAddr(), "8081")
+	return `[Unit]
+Description=llmgw model mirror
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+# Written by 'llmgw server init'. Bytes on :8081 for every node; the
+# management API on loopback only, for the control plane beside it.
+ExecStart=/usr/local/bin/llmgw mirror serve --dir ` + mirrorDir + ` --listen :8081 --api-listen ` + mirrorAPI + ` --base ` + base + `
+User=llmgw
+Group=llmgw
+Restart=on-failure
+RestartSec=2s
+NoNewPrivileges=true
+ProtectSystem=strict
+ProtectHome=true
+PrivateTmp=true
+ReadWritePaths=` + mirrorDir + `
+
+[Install]
+WantedBy=multi-user.target
+`
+}
+
+// ensureMirror runs a mirror on this machine and points the control plane at
+// it. Its management API is on loopback, which is why it needs no certificate:
+// nothing but this machine can reach it.
+func (in *serverInit) ensureMirror(ctx context.Context) error {
+	cfg, err := config.Load(in.configPath())
+	if err != nil {
+		return err
+	}
+	if cfg.Mirror.API != "" && cfg.Mirror.API != "http://"+mirrorAPI {
+		in.kept("mirror at %s, as server.yaml says", cfg.Mirror.API)
+		return nil
+	}
+
+	if err := os.MkdirAll(mirrorDir, 0o755); err != nil {
+		return err
+	}
+	if err := in.chown(mirrorDir, true); err != nil {
+		return err
+	}
+	changed, err := in.writeUnit(mirrorUnitPath, in.mirrorUnit())
+	if err != nil {
+		return err
+	}
+	if err := in.systemctl(ctx, "daemon-reload"); err != nil {
+		return err
+	}
+	if err := in.systemctl(ctx, "enable", "--now", "llmgw-mirror"); err != nil {
+		return err
+	}
+	if changed {
+		if err := in.systemctl(ctx, "restart", "llmgw-mirror"); err != nil {
+			return err
+		}
+		in.done("model mirror in %s, serving on :8081", mirrorDir)
+	} else {
+		in.kept("model mirror in %s", mirrorDir)
+	}
+
+	if cfg.Mirror.API == "" {
+		if err := appendMirrorBlock(in.configPath()); err != nil {
+			return err
+		}
+		in.done("server.yaml points at the mirror")
+	}
+
+	m, err := mirror.NewClient("http://"+mirrorAPI, tlsconf.Config{}, 5*time.Second)
+	if err != nil {
+		return err
+	}
+	return waitFor(ctx, 30*time.Second, "the mirror to answer", func() (bool, error) {
+		return m.Health(ctx) == nil, nil
+	})
+}
+
+// appendMirrorBlock adds the mirror to server.yaml without touching the rest,
+// keeping the file as it was beside it.
+func appendMirrorBlock(path string) error {
+	b, err := os.ReadFile(path)
+	if err != nil {
+		return err
+	}
+	backup := fmt.Sprintf("%s.bak-%s", path, time.Now().Format("20060102-150405"))
+	if err := os.WriteFile(backup, b, 0o640); err != nil {
+		return err
+	}
+	block := "\n# The model mirror on this machine, added by 'llmgw server init'. Its\n" +
+		"# management API is on loopback, so it needs no certificate.\n" +
+		"mirror:\n  api: http://" + mirrorAPI + "\n"
+	info, err := os.Stat(path)
+	if err != nil {
+		return err
+	}
+	return os.WriteFile(path, append(b, []byte(block)...), info.Mode().Perm())
+}
+
+// --- this machine as a node ---
+
+// gpuClasses asks nvidia-smi what cards this machine has. None is an answer,
+// not an error: the node then serves on CPU.
+func gpuClasses(ctx context.Context) ([]string, int) {
+	gpus, err := discovery.DiscoverGPUsSMI(ctx, "")
+	if err != nil || len(gpus) == 0 {
+		return nil, 0
+	}
+	return discovery.Inventory{GPUs: gpus}.Classes(), len(gpus)
+}
+
+// ensureNode makes this machine an agent of its own control plane.
+func (in *serverInit) ensureNode(ctx context.Context) (string, error) {
+	nodeID, _ := os.Hostname()
+	if nodeID == "" {
+		return "", errors.New("this machine has no hostname to use as its node id")
+	}
+
+	if classes, n := gpuClasses(ctx); n > 0 {
+		in.kept("%d NVIDIA GPU(s) here: %s — the node will use them", n, strings.Join(classes, ", "))
+	} else if hasNVIDIADevice() {
+		fmt.Fprintf(in.out, "  !!    an NVIDIA card is on the PCI bus but nvidia-smi does not answer: "+
+			"install the driver and run init again to serve on it; until then this node is CPU only\n")
+	} else {
+		in.kept("no NVIDIA GPU here; the node serves on CPU")
+	}
+
+	certPath := filepath.Join(in.etcDir, "agent.pem")
+	if _, err := os.Stat(certPath); err == nil {
+		in.kept("agent certificate for %s", nodeID)
+	} else {
+		if _, err := os.Stat(filepath.Join(in.pkiDir, "ca-key.pem")); err != nil {
+			return "", fmt.Errorf("the authority is not on this machine, so init cannot give it a node " +
+				"identity; onboard it from where the authority is, or pass --no-node")
+		}
+		ca, err := loadCA(in.pkiDir)
+		if err != nil {
+			return "", err
+		}
+		leaf, err := ca.IssueClient(nodeID)
+		if err != nil {
+			return "", err
+		}
+		if err := leaf.Write(in.etcDir, "agent"); err != nil {
+			return "", err
+		}
+		in.done("agent certificate for %s", nodeID)
+	}
+
+	agentConf := filepath.Join(in.etcDir, "agent.yaml")
+	if _, err := os.Stat(agentConf); err == nil {
+		in.kept("%s (never overwritten)", agentConf)
+	} else {
+		server := "https://" + net.JoinHostPort(in.reachableAddr(), "7443")
+		body := fmt.Sprintf(`# This machine as a node of its own control plane, written by 'llmgw server init'.
+server: %q
+node_id: %q
+tls:
+  ca: %s/ca.pem
+  cert: %s/agent.pem
+  key: %s/agent-key.pem
+labels:
+  %s: %q
+`, server, nodeID, in.etcDir, in.etcDir, in.etcDir, localLabelKey, localLabelValue)
+		if err := os.WriteFile(agentConf, []byte(body), 0o644); err != nil {
+			return "", err
+		}
+		in.done("%s", agentConf)
+	}
+
+	changed, err := in.writeUnit(agentUnitPath, agentUnit)
+	if err != nil {
+		return "", err
+	}
+	if err := in.systemctl(ctx, "daemon-reload"); err != nil {
+		return "", err
+	}
+	if err := in.systemctl(ctx, "enable", "--now", "llmgw-agent"); err != nil {
+		return "", err
+	}
+	if changed {
+		_ = in.systemctl(ctx, "restart", "llmgw-agent")
+		in.done("llmgw-agent started")
+	}
+
+	// Registered is quick. Having an engine is not: the node downloads and
+	// checks its own llama.cpp, the CUDA build when it has a GPU, and a new
+	// machine is often busy with its own package updates first.
+	client := in.adminAPI()
+	last := ""
+	err = waitFor(ctx, 20*time.Minute, "this node to be ready with an engine", func() (bool, error) {
+		var nodes []adminapi.NodeView
+		if err := client.get(ctx, "/api/v1/nodes", &nodes); err != nil {
+			return false, nil
+		}
+		for _, n := range nodes {
+			if n.ID != nodeID {
+				continue
+			}
+			if n.State == "ready" && slices.Contains(n.Engines, "llamacpp") {
+				return true, nil
+			}
+			for _, p := range n.Provision {
+				if line := fmt.Sprintf("%s %s: %s %s", p.Engine, p.Version, p.State, p.Detail); line != last {
+					fmt.Fprintf(in.out, "  ..    %s\n", strings.TrimSpace(line))
+					last = line
+				}
+				// Only llama.cpp is waited for: vLLM installs for minutes after,
+				// and one this node cannot run is not a reason to stop here.
+				if p.Engine == "llamacpp" && (p.State == "failed" || p.State == "unsupported") {
+					return false, fmt.Errorf("installing %s on this node: %s", p.Engine, p.Detail)
+				}
+			}
+		}
+		return false, nil
+	})
+	if err != nil {
+		return "", err
+	}
+	in.kept("node %s is ready", nodeID)
+	return nodeID, nil
+}
+
+// hasNVIDIADevice reports an NVIDIA card on the PCI bus, driver or not, so a
+// machine whose GPU is invisible for want of a driver says so instead of
+// quietly serving on CPU.
+func hasNVIDIADevice() bool {
+	vendors, _ := filepath.Glob("/sys/bus/pci/devices/*/vendor")
+	for _, v := range vendors {
+		b, err := os.ReadFile(v)
+		if err == nil && strings.TrimSpace(string(b)) == "0x10de" {
+			class, _ := os.ReadFile(filepath.Join(filepath.Dir(v), "class"))
+			// 0x03xxxx is a display or 3D controller; NVIDIA also makes bridges.
+			if strings.HasPrefix(strings.TrimSpace(string(class)), "0x03") {
+				return true
+			}
+		}
+	}
+	return false
+}
+
+// --- the starter model ---
+
+// starterModel is the catalog entry: a GPU profile first when there is a card,
+// and a CPU profile always, both pinned to this machine.
+func starterModel(source string, gpuClasses []string) catalog.Model {
+	pin := map[string]string{localLabelKey: localLabelValue}
+	var profiles []catalog.Profile
+	for _, class := range gpuClasses {
+		profiles = append(profiles, catalog.Profile{
+			Name: "gpu-" + class, Engine: "llamacpp", GPUClass: class,
+			Priority: 200, MinMemoryGB: 2, NodeSelector: pin,
+			Args: []string{"--ctx-size", "32768"},
+		})
+	}
+	profiles = append(profiles, catalog.Profile{
+		Name: "cpu", Engine: "llamacpp", Priority: 100, MinMemoryGB: 2, NodeSelector: pin,
+		Args: []string{"--ctx-size", "16384"},
+	})
+	return catalog.Model{
+		Name:           starterName,
+		Aliases:        []string{starterAlias},
+		Description:    "A small model init put here so there is something to try; fast, not clever.",
+		Source:         source,
+		RoutingProfile: "chat",
+		ContextLength:  4096,
+		MaxOutput:      1024,
+		Scaling:        catalog.Scaling{Min: 1, Max: 1},
+		Profiles:       profiles,
+	}
+}
+
+// ensureStarter fetches the starter model into the mirror and serves it here.
+func (in *serverInit) ensureStarter(ctx context.Context) error {
+	client := in.adminAPI()
+	var models []adminapi.ModelView
+	if err := client.get(ctx, "/api/v1/models", &models); err != nil {
+		return fmt.Errorf("reading the catalog: %w", err)
+	}
+	for _, m := range models {
+		if m.Name == starterName {
+			in.kept("%s in the catalog", starterName)
+			return in.waitForStarter(ctx)
+		}
+	}
+
+	upstream, ok := sourcecheck.HTTPSource(starterSource)
+	if !ok {
+		return fmt.Errorf("cannot turn %s into a download", starterSource)
+	}
+	m, err := mirror.NewClient("http://"+mirrorAPI, tlsconf.Config{}, 15*time.Second)
+	if err != nil {
+		return err
+	}
+	st, err := m.Status(ctx, starterName, starterVersion)
+	var apiErr *mirror.APIError
+	switch {
+	case err == nil:
+	case errors.As(err, &apiErr) && apiErr.Status == 404:
+		if st, err = m.Fetch(ctx, mirror.FetchRequest{
+			Source: upstream, Model: starterName, Version: starterVersion,
+		}); err != nil {
+			return fmt.Errorf("asking the mirror for %s: %w", starterName, err)
+		}
+		fmt.Fprintf(in.out, "  ..    downloading %s into the mirror\n", starterName)
+	default:
+		return err
+	}
+
+	lastPct := -10
+	err = waitFor(ctx, 30*time.Minute, starterName+" to download", func() (bool, error) {
+		cur, err := m.Status(ctx, starterName, starterVersion)
+		if err != nil {
+			return false, nil
+		}
+		st = cur
+		switch cur.Status.State {
+		case mirror.Present:
+			return true, nil
+		case mirror.Failed:
+			return false, fmt.Errorf("the mirror could not download %s: %s", starterName, cur.Status.Err)
+		}
+		if pct, ok := cur.Status.Percent(); ok && pct >= lastPct+10 {
+			fmt.Fprintf(in.out, "  ..    %d%%\n", pct)
+			lastPct = pct
+		}
+		return false, nil
+	})
+	if err != nil {
+		return err
+	}
+	in.done("%s in the mirror", starterName)
+
+	classes, _ := gpuClasses(ctx)
+	body, err := json.Marshal(starterModel(st.URL, classes))
+	if err != nil {
+		return err
+	}
+	if err := client.post(ctx, "/api/v1/models", body, &map[string]string{}); err != nil {
+		return fmt.Errorf("adding %s to the catalog: %w", starterName, err)
+	}
+	where := "on CPU"
+	if len(classes) > 0 {
+		where = "on " + strings.Join(classes, ", ") + ", or CPU if the card is busy"
+	}
+	in.done("%s in the catalog, served here %s", starterName, where)
+	return in.waitForStarter(ctx)
+}
+
+func (in *serverInit) waitForStarter(ctx context.Context) error {
+	client := in.adminAPI()
+	last := ""
+	return waitFor(ctx, 10*time.Minute, starterName+" to start", func() (bool, error) {
+		var reps []adminapi.ReplicaView
+		if err := client.get(ctx, "/api/v1/replicas?model="+starterName, &reps); err != nil {
+			return false, nil
+		}
+		for _, r := range reps {
+			if r.State == "ready" {
+				in.kept("%s is serving (profile %s)", starterName, r.Profile)
+				return true, nil
+			}
+		}
+		for _, r := range reps {
+			line := r.State
+			if r.Err != "" {
+				line += ": " + r.Err
+			}
+			if line != last {
+				fmt.Fprintf(in.out, "  ..    %s %s\n", starterName, line)
+				last = line
+			}
+		}
+		return false, nil
+	})
+}
+
+func (in *serverInit) adminAPI() *ctlClient {
+	addr := "127.0.0.1:9090"
+	if cfg, err := config.Load(in.configPath()); err == nil && cfg.AdminListen != "" {
+		addr = cfg.AdminListen
+	}
+	return newCtlClient(addr)
+}
+
+// waitFor polls until done says so, fails, or the time runs out.
+func waitFor(ctx context.Context, limit time.Duration, what string, done func() (bool, error)) error {
+	deadline := time.Now().Add(limit)
+	for {
+		ok, err := done()
+		if err != nil {
+			return err
+		}
+		if ok {
+			return nil
+		}
+		if time.Now().After(deadline) {
+			return fmt.Errorf("gave up waiting for %s after %s; run init again to keep waiting", what, limit)
+		}
+		select {
+		case <-ctx.Done():
+			return ctx.Err()
+		case <-time.After(2 * time.Second):
+		}
+	}
+}

internal/cli/serverinit_cluster_test.go

@@ -0,0 +1,89 @@
+package cli
+
+import (
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+)
+
+func TestTheInitAgentUnitIsTheShippedUnit(t *testing.T) {
+	b, err := os.ReadFile("../../deploy/systemd/llmgw-agent.service")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if string(b) != agentUnit {
+		t.Error("deploy/systemd/llmgw-agent.service and the unit 'server init' writes have drifted")
+	}
+}
+
+func TestTheStarterUsesTheGPUAndFallsBackToCPUOnThisMachine(t *testing.T) {
+	m := starterModel("http://10.0.0.5:8081/qwen2.5-0.5b-instruct/starter/q.gguf", []string{"4090-24g"})
+	c, err := catalog.New([]catalog.Model{m})
+	if err != nil {
+		t.Fatalf("the starter is not a valid catalog entry: %v", err)
+	}
+	got, _ := c.Resolve(starterName)
+	if len(got.Profiles) != 2 {
+		t.Fatalf("profiles = %+v", got.Profiles)
+	}
+	gpu, cpu := got.Profiles[0], got.Profiles[1]
+	if gpu.GPUClass != "4090-24g" || gpu.Priority <= cpu.Priority {
+		t.Errorf("the GPU profile should come first: %+v", gpu)
+	}
+	if cpu.GPUClass != "" || cpu.Engine != "llamacpp" {
+		t.Errorf("the fallback should be llama.cpp on CPU: %+v", cpu)
+	}
+	for _, p := range got.Profiles {
+		if p.NodeSelector[localLabelKey] != localLabelValue {
+			t.Errorf("profile %s is not pinned to this machine", p.Name)
+		}
+	}
+	if got.Scaling.Min != 1 {
+		t.Error("the starter should be running, not waiting to be asked for")
+	}
+
+	if m := starterModel("http://x/q.gguf", nil); len(m.Profiles) != 1 || m.Profiles[0].Name != "cpu" {
+		t.Errorf("without a GPU only the CPU profile: %+v", m.Profiles)
+	}
+}
+
+func TestTheMirrorBlockIsAddedAndReads(t *testing.T) {
+	in, out := newTestInit(t)
+	if err := in.run(t.Context()); err != nil {
+		t.Fatalf("%v\n%s", err, out)
+	}
+	if err := appendMirrorBlock(in.configPath()); err != nil {
+		t.Fatal(err)
+	}
+	cfg, err := config.Load(in.configPath())
+	if err != nil {
+		t.Fatal(err)
+	}
+	if cfg.Mirror.API != "http://"+mirrorAPI {
+		t.Errorf("mirror.api = %q", cfg.Mirror.API)
+	}
+	if err := cfg.Validate(); err != nil {
+		t.Error(err)
+	}
+	backups, _ := filepath.Glob(in.configPath() + ".bak-*")
+	if len(backups) != 1 {
+		t.Errorf("backups = %v", backups)
+	}
+}
+
+func TestTheMirrorUnitAdvertisesAnAddressNodesCanReach(t *testing.T) {
+	in := &serverInit{hosts: []string{"10.0.0.140"}}
+	u := in.mirrorUnit()
+	if !strings.Contains(u, "--base http://10.0.0.140:8081") {
+		t.Errorf("base: %s", u)
+	}
+	// The management port can delete models; nothing but this machine may
+	// reach it, which is what lets it go without a certificate.
+	if !strings.Contains(u, "--api-listen 127.0.0.1:8082") {
+		t.Errorf("the management API is not on loopback: %s", u)
+	}
+}

internal/cli/serverinit_postgres.go

@@ -0,0 +1,525 @@
+package cli
+
+import (
+	"context"
+	"crypto/rand"
+	"encoding/hex"
+	"fmt"
+	"log/slog"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"strconv"
+	"strings"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// pgRole and pgDatabase are what init creates in a PostgreSQL it manages.
+const (
+	pgRole     = "llmgw"
+	pgDatabase = "llmgw"
+)
+
+// dsnPath is where the PostgreSQL address is kept, out of server.yaml, so the
+// password in it is not in a file that gets copied, diffed and pasted.
+func (in *serverInit) dsnPath() string { return filepath.Join(in.etcDir, "secrets", "store-dsn") }
+
+// storeAddress is what auth.store says for the store init manages.
+func (in *serverInit) storeAddress() string {
+	if in.postgres {
+		return "@" + in.dsnPath()
+	}
+	return filepath.Join(in.stateDir, "llmgw.db")
+}
+
+// ensureDatabase puts the store in PostgreSQL when asked to, moving whatever
+// the SQLite store already holds.
+//
+// The move is the point. Keys are stored hashed, so starting PostgreSQL empty
+// means reissuing every key and being unable to say which was which; and the
+// administrator init created a minute ago would be gone too.
+func (in *serverInit) ensureDatabase(ctx context.Context) (err error) {
+	if !in.postgres {
+		return nil
+	}
+	setup := in.pgSetup
+	if setup == nil {
+		setup = in.setupPostgres
+	}
+	target, err := setup(ctx)
+	if err != nil {
+		return err
+	}
+
+	cfg, err := config.Load(in.configPath())
+	if err != nil {
+		return err
+	}
+	current := cfg.Auth.Store
+	if current == target {
+		in.kept("store in PostgreSQL (%s)", in.dsnPath())
+		return nil
+	}
+	if isPostgresAddress(current) {
+		// Two PostgreSQL stores, and a guess about which is meant would lose
+		// one of them.
+		return fmt.Errorf("server.yaml already names a PostgreSQL store (%s); "+
+			"init moves SQLite into PostgreSQL, not one PostgreSQL into another", current)
+	}
+
+	// Nothing may write the store while it is copied: a row landing after its
+	// table has been passed would be absent, and nothing would say so.
+	stopped, err := in.stopService(ctx)
+	if err != nil {
+		return err
+	}
+	defer func() {
+		// A move that failed leaves the configuration naming SQLite, which
+		// still holds everything: start the service on it again rather than
+		// leave the gateway down over a database that did not work out.
+		if err != nil && stopped {
+			_ = exec.Command("systemctl", "start", "llmgw-server").Run()
+			fmt.Fprintf(in.out, "  ..    llmgw-server started again on its SQLite store\n")
+		}
+	}()
+
+	if current != "" {
+		if _, err := os.Stat(current); err == nil {
+			if err := in.copyStore(ctx, current, target); err != nil {
+				return err
+			}
+		}
+	}
+
+	if err := setAuthStore(in.configPath(), target); err != nil {
+		return err
+	}
+	in.done("server.yaml now keeps the store in PostgreSQL")
+	return nil
+}
+
+func (in *serverInit) copyStore(ctx context.Context, from, to string) error {
+	src, err := store.Open(ctx, from)
+	if err != nil {
+		return fmt.Errorf("opening %s: %w", from, err)
+	}
+	defer src.Close()
+	dst, err := store.Open(ctx, to)
+	if err != nil {
+		return fmt.Errorf("opening PostgreSQL: %w", err)
+	}
+	defer dst.Close()
+
+	empty, err := dst.Empty(ctx)
+	if err != nil {
+		return err
+	}
+	if !empty {
+		// A previous run that copied and then stopped before changing the
+		// configuration. The service was stopped for that copy too, so what
+		// is there is what SQLite held.
+		in.kept("PostgreSQL already holds a store; %s is left as it is", from)
+		return nil
+	}
+	res, err := src.CopyTo(ctx, dst)
+	if err != nil {
+		return fmt.Errorf("copying %s into PostgreSQL: %w", from, err)
+	}
+	var rows int64
+	for _, r := range res {
+		rows += r.Rows
+	}
+	in.done("copied %d rows from %s into PostgreSQL (the SQLite file is kept)", rows, from)
+	return nil
+}
+
+func isPostgresAddress(a string) bool {
+	return strings.HasPrefix(a, "@") || strings.HasPrefix(a, "postgres://") ||
+		strings.HasPrefix(a, "postgresql://")
+}
+
+// setAuthStore changes auth.store in server.yaml and nothing else.
+//
+// Edited as text rather than unmarshalled and written back, because writing
+// it back would drop every comment in a file people annotate. The old file is
+// kept beside it.
+func setAuthStore(path, address string) error {
+	b, err := os.ReadFile(path)
+	if err != nil {
+		return err
+	}
+	lines := strings.Split(string(b), "\n")
+	value := fmt.Sprintf("%q", address)
+
+	authAt, storeAt, indent := -1, -1, "  "
+	for i, l := range lines {
+		trimmed := strings.TrimSpace(l)
+		top := l != "" && l[0] != ' ' && l[0] != '\t' && !strings.HasPrefix(trimmed, "#")
+		if top {
+			if authAt >= 0 && i > authAt {
+				break // past the auth block
+			}
+			if strings.HasPrefix(trimmed, "auth:") {
+				authAt = i
+			}
+			continue
+		}
+		if authAt >= 0 && strings.HasPrefix(trimmed, "store:") {
+			storeAt = i
+			indent = l[:len(l)-len(strings.TrimLeft(l, " \t"))]
+			break
+		}
+	}
+	switch {
+	case storeAt >= 0:
+		lines[storeAt] = indent + "store: " + value
+	case authAt >= 0:
+		lines = append(lines[:authAt+1], append([]string{indent + "store: " + value}, lines[authAt+1:]...)...)
+	default:
+		lines = append(lines, "auth:", "  enabled: true", "  store: "+value)
+	}
+
+	backup := fmt.Sprintf("%s.bak-%s", path, time.Now().Format("20060102-150405"))
+	if err := os.WriteFile(backup, b, 0o640); err != nil {
+		return err
+	}
+	info, err := os.Stat(path)
+	if err != nil {
+		return err
+	}
+	return os.WriteFile(path, []byte(strings.Join(lines, "\n")), info.Mode().Perm())
+}
+
+// pgServer is the PostgreSQL installation on this machine, as systemd and the
+// file system describe it.
+//
+// Found rather than assumed. A distribution's own package calls its unit
+// postgresql.service and puts initdb and psql on PATH; the PostgreSQL
+// project's packages (PGDG) call it postgresql-16.service and keep them in
+// /usr/pgsql-16/bin — and they provide "postgresql-server" too, so asking dnf
+// for that name can install or find one of them. Assuming the first found a
+// Rocky machine with the second unable to initialise the database it had.
+type pgServer struct {
+	unit   string // postgresql.service, postgresql-16.service
+	pgdata string // "" when the unit does not say, which is Debian's case
+	bin    string // directory holding initdb and psql; "" means PATH
+}
+
+func (p pgServer) tool(name string) string {
+	if p.bin != "" {
+		return filepath.Join(p.bin, name)
+	}
+	return name
+}
+
+// pickPostgresUnit chooses the server unit from systemctl's list of
+// postgresql* unit files: the plain one when there is one, else the highest
+// version. Templates (postgresql@.service) are not a server by themselves.
+func pickPostgresUnit(list string) string {
+	best, bestVersion := "", -1
+	for _, line := range strings.Split(list, "\n") {
+		f := strings.Fields(line)
+		if len(f) == 0 {
+			continue
+		}
+		unit := f[0]
+		if !strings.HasSuffix(unit, ".service") || strings.Contains(unit, "@") {
+			continue
+		}
+		name := strings.TrimSuffix(unit, ".service")
+		if name == "postgresql" {
+			return unit
+		}
+		v := strings.TrimPrefix(name, "postgresql-")
+		if n, err := strconv.Atoi(strings.SplitN(v, ".", 2)[0]); err == nil && n > bestVersion {
+			best, bestVersion = unit, n
+		}
+	}
+	return best
+}
+
+// pgdataFrom reads PGDATA out of `systemctl show -p Environment --value`.
+func pgdataFrom(env string) string {
+	for _, kv := range strings.Fields(env) {
+		if v, ok := strings.CutPrefix(kv, "PGDATA="); ok {
+			return v
+		}
+	}
+	return ""
+}
+
+func findPostgres(ctx context.Context) (pgServer, bool) {
+	out, _ := exec.CommandContext(ctx, "systemctl", "list-unit-files", "--type=service",
+		"--no-legend", "--plain", "postgresql*").Output()
+	unit := pickPostgresUnit(string(out))
+	if unit == "" {
+		return pgServer{}, false
+	}
+	p := pgServer{unit: unit}
+	env, _ := exec.CommandContext(ctx, "systemctl", "show", "-p", "Environment", "--value", unit).Output()
+	p.pgdata = pgdataFrom(string(env))
+
+	if v, ok := strings.CutPrefix(strings.TrimSuffix(unit, ".service"), "postgresql-"); ok {
+		if dir := "/usr/pgsql-" + v + "/bin"; isDir(dir) {
+			p.bin = dir
+		}
+		if p.pgdata == "" && isDir("/var/lib/pgsql/"+v) {
+			p.pgdata = "/var/lib/pgsql/" + v + "/data"
+		}
+	}
+	if p.bin == "" {
+		if _, err := exec.LookPath("psql"); err != nil {
+			// Debian keeps the server's tools per version and puts only
+			// wrappers on PATH; without even those, look where it keeps them.
+			if dirs, _ := filepath.Glob("/usr/lib/postgresql/*/bin"); len(dirs) > 0 {
+				p.bin = dirs[len(dirs)-1]
+			}
+		}
+	}
+	return p, true
+}
+
+func isDir(p string) bool {
+	st, err := os.Stat(p)
+	return err == nil && st.IsDir()
+}
+
+// setupPostgres makes sure a PostgreSQL on this machine holds a database for
+// llmgw, installing the server if there is none, and returns the store
+// address.
+func (in *serverInit) setupPostgres(ctx context.Context) (string, error) {
+	installed, err := in.installPostgres(ctx)
+	if err != nil {
+		return "", err
+	}
+	if err := in.waitForPostgres(ctx); err != nil {
+		return "", err
+	}
+
+	exists, err := in.psqlBool(ctx, "SELECT 1 FROM pg_roles WHERE rolname = '"+pgRole+"'")
+	if err != nil {
+		return "", err
+	}
+	_, statErr := os.Stat(in.dsnPath())
+	if exists && statErr == nil {
+		in.kept("PostgreSQL role %s", pgRole)
+	} else {
+		// A role with no address written down is one whose password nobody
+		// knows; giving it a new one is the only way to use it.
+		password := make([]byte, 24)
+		_, _ = rand.Read(password)
+		pass := hex.EncodeToString(password)
+		verb := "CREATE ROLE"
+		if exists {
+			verb = "ALTER ROLE"
+		}
+		if _, err := in.psql(ctx, verb+" "+pgRole+" WITH LOGIN PASSWORD '"+pass+"'"); err != nil {
+			return "", err
+		}
+		dsn := fmt.Sprintf("postgres://%s:%s@127.0.0.1:5432/%s?sslmode=disable", pgRole, pass, pgDatabase)
+		if err := in.writeDSN(dsn); err != nil {
+			return "", err
+		}
+		in.done("PostgreSQL role %s, its address in %s", pgRole, in.dsnPath())
+	}
+
+	exists, err = in.psqlBool(ctx, "SELECT 1 FROM pg_database WHERE datname = '"+pgDatabase+"'")
+	if err != nil {
+		return "", err
+	}
+	if exists {
+		in.kept("PostgreSQL database %s", pgDatabase)
+	} else {
+		if _, err := in.psql(ctx, "CREATE DATABASE "+pgDatabase+" OWNER "+pgRole); err != nil {
+			return "", err
+		}
+		in.done("PostgreSQL database %s", pgDatabase)
+	}
+
+	if installed {
+		// Its defaults assume it owns the machine. A gateway swapped out to
+		// give the database cache is worse than a smaller cache, and a store
+		// whose working set is megabytes needs little. Only a PostgreSQL init
+		// installed itself: one that was already here may serve something
+		// else that needs its connections.
+		for _, q := range []string{
+			"ALTER SYSTEM SET shared_buffers = '96MB'",
+			"ALTER SYSTEM SET max_connections = 40",
+		} {
+			if _, err := in.psql(ctx, q); err != nil {
+				return "", err
+			}
+		}
+		if out, err := exec.CommandContext(ctx, "systemctl", "restart", in.pg.unit).CombinedOutput(); err != nil {
+			return "", fmt.Errorf("restarting PostgreSQL: %v: %s", err, strings.TrimSpace(string(out)))
+		}
+		if err := in.waitForPostgres(ctx); err != nil {
+			return "", err
+		}
+		in.done("PostgreSQL sized for a machine it shares (shared_buffers 96MB, 40 connections)")
+	}
+
+	// Proof the address works, before anything is moved into it.
+	address := "@" + in.dsnPath()
+	s, err := store.Open(ctx, address)
+	if err != nil {
+		return "", fmt.Errorf("connecting with the address in %s: %w (PostgreSQL's pg_hba.conf "+
+			"must allow password logins from 127.0.0.1: scram-sha-256 or md5, not ident)", in.dsnPath(), err)
+	}
+	_ = s.Close()
+	return address, nil
+}
+
+func (in *serverInit) writeDSN(dsn string) error {
+	dir := filepath.Dir(in.dsnPath())
+	if err := os.MkdirAll(dir, 0o750); err != nil {
+		return err
+	}
+	if err := in.chown(dir, false); err != nil {
+		return err
+	}
+	if err := os.WriteFile(in.dsnPath(), []byte(dsn+"\n"), 0o600); err != nil {
+		return err
+	}
+	// Read by the service, which runs as llmgw; root reads it anyway.
+	return in.chown(in.dsnPath(), true)
+}
+
+// installPostgres installs the server when this machine has none, makes sure
+// its database is initialised and running, and says whether it installed it.
+func (in *serverInit) installPostgres(ctx context.Context) (bool, error) {
+	host := discovery.DiscoverHost()
+	pg, found := findPostgres(ctx)
+	installed := false
+	if found {
+		in.kept("PostgreSQL server (%s)", pg.unit)
+	} else {
+		if host.Family() == "" {
+			return false, fmt.Errorf("no PostgreSQL here and no package manager init knows (%s); "+
+				"install PostgreSQL and run init again", host.OSID)
+		}
+		fmt.Fprintf(in.out, "  ..    installing PostgreSQL (a minute or two)\n")
+		pi := &provision.Installer{Host: host, Log: slog.Default()}
+		if err := pi.InstallPackages(ctx, map[string][]string{
+			"deb": {"postgresql"},
+			"rpm": {"postgresql-server", "postgresql"},
+		}); err != nil {
+			return false, err
+		}
+		if pg, found = findPostgres(ctx); !found {
+			q := "rpm -qa 'postgresql*'"
+			if host.Family() == "deb" {
+				q = "dpkg -l 'postgresql*'"
+			}
+			have, _ := exec.CommandContext(ctx, "sh", "-c", q).CombinedOutput()
+			return false, fmt.Errorf("PostgreSQL was installed but no postgresql service unit "+
+				"appeared; what is installed:\n%s", strings.TrimSpace(string(have)))
+		}
+		installed = true
+		in.done("PostgreSQL installed (%s)", pg.unit)
+	}
+	in.pg = pg
+
+	// Debian's package creates and starts a cluster itself. The RHEL family's,
+	// and PGDG's, leave the data directory empty until someone runs initdb.
+	if pg.pgdata != "" {
+		if _, err := os.Stat(filepath.Join(pg.pgdata, "PG_VERSION")); err != nil {
+			// Directly rather than through postgresql-setup, which is named
+			// differently by every packager. Password logins over TCP are
+			// asked for: RHEL's default for them is ident, which no service
+			// account passes.
+			if err := os.MkdirAll(pg.pgdata, 0o700); err != nil {
+				return false, err
+			}
+			if out, err := exec.CommandContext(ctx, "chown", "postgres:postgres", pg.pgdata).CombinedOutput(); err != nil {
+				return false, fmt.Errorf("chown %s: %v: %s", pg.pgdata, err, strings.TrimSpace(string(out)))
+			}
+			cmd := exec.CommandContext(ctx, "runuser", "-u", "postgres", "--", pg.tool("initdb"),
+				"-D", pg.pgdata, "--auth-host=scram-sha-256", "--auth-local=peer")
+			cmd.Dir = "/"
+			if out, err := cmd.CombinedOutput(); err != nil {
+				return false, fmt.Errorf("initdb %s: %v: %s", pg.pgdata, err, lastLines(out, 5))
+			}
+			in.done("PostgreSQL database cluster initialised in %s", pg.pgdata)
+		}
+	}
+	if exec.CommandContext(ctx, "systemctl", "is-active", "--quiet", pg.unit).Run() != nil {
+		if out, err := exec.CommandContext(ctx, "systemctl", "enable", "--now", pg.unit).CombinedOutput(); err != nil {
+			return false, fmt.Errorf("starting %s: %v: %s", pg.unit, err, strings.TrimSpace(string(out)))
+		}
+		in.done("%s started", pg.unit)
+	}
+	return installed, nil
+}
+
+func lastLines(b []byte, n int) string {
+	lines := strings.Split(strings.TrimSpace(string(b)), "\n")
+	if len(lines) > n {
+		lines = lines[len(lines)-n:]
+	}
+	return strings.Join(lines, "\n")
+}
+
+// psql runs one statement as the postgres superuser, over the local socket.
+func (in *serverInit) psql(ctx context.Context, query string) (string, error) {
+	cmd := exec.CommandContext(ctx, "runuser", "-u", "postgres", "--",
+		in.pg.tool("psql"), "-v", "ON_ERROR_STOP=1", "-qAtX", "-c", query)
+	// postgres cannot enter root's home, and psql says so on every call.
+	cmd.Dir = "/"
+	out, err := cmd.CombinedOutput()
+	if err != nil {
+		// The statement can hold a password; the error names only its verb.
+		verb := strings.Join(strings.Fields(query)[:2], " ")
+		return "", fmt.Errorf("psql (%s): %v: %s", verb, err, strings.TrimSpace(string(out)))
+	}
+	return strings.TrimSpace(string(out)), nil
+}
+
+func (in *serverInit) psqlBool(ctx context.Context, query string) (bool, error) {
+	out, err := in.psql(ctx, query)
+	return out == "1", err
+}
+
+// waitForPostgres waits for a server that was just started or restarted to
+// take connections.
+func (in *serverInit) waitForPostgres(ctx context.Context) error {
+	deadline := time.Now().Add(60 * time.Second)
+	for {
+		_, err := in.psql(ctx, "SELECT 1")
+		if err == nil {
+			return nil
+		}
+		if time.Now().After(deadline) {
+			return fmt.Errorf("PostgreSQL is not taking connections after 60s: %w", err)
+		}
+		select {
+		case <-ctx.Done():
+			return ctx.Err()
+		case <-time.After(2 * time.Second):
+		}
+	}
+}
+
+// stopService stops the control plane if it runs, so the store is still.
+// It says whether it stopped it, so a failed move can start it again.
+func (in *serverInit) stopService(ctx context.Context) (bool, error) {
+	if in.unitPath == "" {
+		return false, nil
+	}
+	if _, err := exec.LookPath("systemctl"); err != nil {
+		return false, nil
+	}
+	if exec.CommandContext(ctx, "systemctl", "is-active", "--quiet", "llmgw-server").Run() != nil {
+		return false, nil
+	}
+	if out, err := exec.CommandContext(ctx, "systemctl", "stop", "llmgw-server").CombinedOutput(); err != nil {
+		return false, fmt.Errorf("stopping llmgw-server to move its store: %v: %s", err, strings.TrimSpace(string(out)))
+	}
+	in.done("llmgw-server stopped while its store moves")
+	return true, nil
+}

internal/cli/serverinit_postgres_test.go

@@ -0,0 +1,203 @@
+package cli
+
+import (
+	"context"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+func TestSetAuthStoreChangesOnlyAuthStore(t *testing.T) {
+	p := filepath.Join(t.TempDir(), "server.yaml")
+	body := `# my notes
+threads:
+  store: /var/lib/llmgw/threads.db   # not this one
+  retain_hours: 24
+auth:
+  enabled: true
+  # the store
+  store: /var/lib/llmgw/llmgw.db
+  signup_open: true
+landing:
+  contact: "x"
+`
+	if err := os.WriteFile(p, []byte(body), 0o640); err != nil {
+		t.Fatal(err)
+	}
+	if err := setAuthStore(p, "@/etc/llmgw/secrets/store-dsn"); err != nil {
+		t.Fatal(err)
+	}
+	got, _ := os.ReadFile(p)
+	want := strings.Replace(body, "  store: /var/lib/llmgw/llmgw.db",
+		`  store: "@/etc/llmgw/secrets/store-dsn"`, 1)
+	if string(got) != want {
+		t.Errorf("got:\n%s\nwant:\n%s", got, want)
+	}
+	cfg, err := config.Load(p)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if cfg.Auth.Store != "@/etc/llmgw/secrets/store-dsn" || cfg.Threads.Store != "/var/lib/llmgw/threads.db" {
+		t.Errorf("auth.store %q, threads.store %q", cfg.Auth.Store, cfg.Threads.Store)
+	}
+	backups, _ := filepath.Glob(p + ".bak-*")
+	if len(backups) != 1 {
+		t.Errorf("the old file should be kept beside it, found %v", backups)
+	}
+}
+
+func TestSetAuthStoreAddsTheLineWhenAuthHasNone(t *testing.T) {
+	p := filepath.Join(t.TempDir(), "server.yaml")
+	if err := os.WriteFile(p, []byte("auth:\n  enabled: true\nlanding:\n  contact: x\n"), 0o640); err != nil {
+		t.Fatal(err)
+	}
+	if err := setAuthStore(p, "@/x"); err != nil {
+		t.Fatal(err)
+	}
+	cfg, err := config.Load(p)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if cfg.Auth.Store != "@/x" || !cfg.Auth.Enabled {
+		t.Errorf("auth = %+v", cfg.Auth)
+	}
+}
+
+// fakePostgres stands in for setupPostgres the way it really answers: an
+// address file under /etc/llmgw/secrets, returned as "@file". What the file
+// holds is a SQLite path, so everything after the setup runs for real.
+func fakePostgres(t *testing.T, in *serverInit) (address, db string) {
+	t.Helper()
+	db = filepath.Join(t.TempDir(), "pg-standin.db")
+	in.pgSetup = func(context.Context) (string, error) {
+		if err := os.MkdirAll(filepath.Dir(in.dsnPath()), 0o750); err != nil {
+			return "", err
+		}
+		return "@" + in.dsnPath(), os.WriteFile(in.dsnPath(), []byte(db+"\n"), 0o600)
+	}
+	return "@" + in.dsnPath(), db
+}
+
+// Moving an install that began on SQLite: the administrator it created must
+// be in the new store, the configuration must name it, and the SQLite file
+// must still be there. The new store is SQLite too here — what PostgreSQL
+// adds is the setup, which needs a machine to run on.
+func TestInitMovesAnExistingStore(t *testing.T) {
+	in, out := newTestInit(t)
+	if err := in.run(context.Background()); err != nil {
+		t.Fatalf("%v\n%s", err, out)
+	}
+	old := filepath.Join(in.stateDir, "llmgw.db")
+
+	in.postgres = true
+	target, db := fakePostgres(t, in)
+	out.Reset()
+	if err := in.run(context.Background()); err != nil {
+		t.Fatalf("%v\n%s", err, out)
+	}
+	if !strings.Contains(out.String(), "copied") {
+		t.Errorf("nothing was copied:\n%s", out)
+	}
+
+	cfg, err := config.Load(in.configPath())
+	if err != nil {
+		t.Fatal(err)
+	}
+	if cfg.Auth.Store != target {
+		t.Fatalf("server.yaml still names %q", cfg.Auth.Store)
+	}
+	s, err := store.Open(context.Background(), db)
+	if err != nil {
+		t.Fatal(err)
+	}
+	u, err := s.UserByEmail(context.Background(), "admin@example.com")
+	s.Close()
+	if err != nil || !u.IsAdmin {
+		t.Fatalf("the administrator did not move: %v %+v", err, u)
+	}
+	if _, err := os.Stat(old); err != nil {
+		t.Error("the SQLite store should be kept")
+	}
+
+	// And once moved, again is nothing.
+	in.changed = false
+	out.Reset()
+	if err := in.run(context.Background()); err != nil {
+		t.Fatalf("%v\n%s", err, out)
+	}
+	if in.changed {
+		t.Errorf("a run after the move changed something:\n%s", out)
+	}
+}
+
+func TestInitWritesAPostgresConfigurationFromTheStart(t *testing.T) {
+	in, out := newTestInit(t)
+	in.postgres = true
+	target, _ := fakePostgres(t, in)
+	if err := in.run(context.Background()); err != nil {
+		t.Fatalf("%v\n%s", err, out)
+	}
+	cfg, _ := config.Load(in.configPath())
+	if cfg.Auth.Store != target {
+		t.Errorf("auth.store = %q", cfg.Auth.Store)
+	}
+	if _, err := os.Stat(filepath.Join(in.stateDir, "llmgw.db")); err == nil {
+		t.Error("a PostgreSQL install should not leave a SQLite store behind")
+	}
+}
+
+func TestInitWillNotMoveOnePostgresIntoAnother(t *testing.T) {
+	in, out := newTestInit(t)
+	if err := os.MkdirAll(in.etcDir, 0o755); err != nil {
+		t.Fatal(err)
+	}
+	cfg := "control:\n  enabled: true\nauth:\n  enabled: true\n  store: postgres://elsewhere/llmgw\n"
+	if err := os.WriteFile(in.configPath(), []byte(cfg), 0o640); err != nil {
+		t.Fatal(err)
+	}
+	in.postgres = true
+	in.pgSetup = func(context.Context) (string, error) { return "@/somewhere/else", nil }
+	err := in.run(context.Background())
+	if err == nil || !strings.Contains(err.Error(), "already names a PostgreSQL store") {
+		t.Fatalf("expected a refusal, got %v\n%s", err, out)
+	}
+}
+
+// The lists below are what `systemctl list-unit-files --type=service
+// --no-legend --plain 'postgresql*'` prints on each kind of machine.
+func TestPickPostgresUnit(t *testing.T) {
+	for _, c := range []struct{ name, list, want string }{
+		{"rocky, distribution package",
+			"postgresql.service disabled disabled\npostgresql@.service disabled disabled\n", "postgresql.service"},
+		{"rocky, PGDG package",
+			"postgresql-16.service disabled disabled\n", "postgresql-16.service"},
+		{"two PGDG versions",
+			"postgresql-13.service enabled disabled\npostgresql-16.service disabled disabled\n", "postgresql-16.service"},
+		{"debian",
+			"postgresql.service enabled enabled\npostgresql@.service indirect enabled\n", "postgresql.service"},
+		{"only the template", "postgresql@.service indirect enabled\n", ""},
+		{"none", "", ""},
+	} {
+		if got := pickPostgresUnit(c.list); got != c.want {
+			t.Errorf("%s: got %q, want %q", c.name, got, c.want)
+		}
+	}
+}
+
+// What `systemctl show -p Environment --value` prints for each unit, from
+// the unit files the packages ship.
+func TestPGDATAFromTheUnit(t *testing.T) {
+	for env, want := range map[string]string{
+		"PG_OOM_ADJUST_FILE=/proc/self/oom_score_adj PG_OOM_ADJUST_VALUE=0 PGDATA=/var/lib/pgsql/data": "/var/lib/pgsql/data",
+		"PGDATA=/var/lib/pgsql/16/data/ PG_OOM_ADJUST_FILE=/proc/self/oom_score_adj":                   "/var/lib/pgsql/16/data/",
+		"": "",
+	} {
+		if got := pgdataFrom(env); got != want {
+			t.Errorf("%q: got %q, want %q", env, got, want)
+		}
+	}
+}

internal/cli/serverinit_test.go

@@ -59,7 +59,8 @@ func TestInitLeavesAControlPlaneThatStarts(t *testing.T) {
 	if err != nil {
 		t.Fatal(err)
 	}
-	if !u.IsAdmin || u.State != store.UserActive || u.ProjectID != "default" {
+	inDefault, _ := s.CanUseProject(context.Background(), u.ID, "default")
+	if !u.IsAdmin || u.State != store.UserActive || !inDefault {
 		t.Errorf("administrator not ready to use: %+v", u)
 	}
 	if !strings.Contains(out.String(), "password: ") {

internal/cli/threads.go

@@ -50,7 +50,12 @@ func newThreadsCmd(open opener) *cobra.Command {
 			if off {
 				keep = 0
 			}
-			if err := s.SetThreadRetention(cmd.Context(), project, keep); err != nil {
+			p, err := resolveProject(cmd.Context(), s, project)
+			if err != nil {
+				return err
+			}
+			project = p.Path
+			if err := s.SetThreadRetention(cmd.Context(), p.ID, keep); err != nil {
 				return err
 			}
 			if keep == 0 {
@@ -63,7 +68,7 @@ func newThreadsCmd(open opener) *cobra.Command {
 			return nil
 		},
 	}
-	cmd.Flags().StringVar(&project, "project", "default", "project id")
+	cmd.Flags().StringVar(&project, "project", "", "project: id, path or unique name")
 	cmd.Flags().IntVar(&hours, "retain-hours", 0, "how long a conversation is kept after its last turn")
 	cmd.Flags().BoolVar(&off, "off", false, "stop keeping conversations for this project")
 	return cmd

internal/cli/tui_node.go

@@ -0,0 +1,354 @@
+package cli
+
+import (
+	"context"
+	"fmt"
+	"io"
+	"os"
+	"strings"
+
+	"github.com/charmbracelet/bubbles/spinner"
+	"github.com/charmbracelet/bubbles/textinput"
+	tea "github.com/charmbracelet/bubbletea"
+	"github.com/charmbracelet/lipgloss"
+	"github.com/mattn/go-isatty"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
+)
+
+// The full-screen way through 'llmgw node add': the same steps as the plain
+// one, drawn as a checklist that fills in, with the questions a script
+// cannot ask — which fabric the adapters are on, whether to go on past a
+// problem — asked in place.
+
+func isTerminal(w io.Writer) bool {
+	f, ok := w.(*os.File)
+	return ok && isatty.IsTerminal(f.Fd()) && os.Getenv("TERM") != "dumb"
+}
+
+var (
+	tuiAccent = lipgloss.AdaptiveColor{Light: "#0B7F7D", Dark: "#3FC2BC"}
+	tuiOK     = lipgloss.AdaptiveColor{Light: "#1F8A4C", Dark: "#6FD3A6"}
+	tuiWarn   = lipgloss.AdaptiveColor{Light: "#A15C00", Dark: "#E9B35A"}
+	tuiBad    = lipgloss.AdaptiveColor{Light: "#B42318", Dark: "#F07C6C"}
+	tuiDim    = lipgloss.AdaptiveColor{Light: "#56696C", Dark: "#8FA5A4"}
+
+	sTitle = lipgloss.NewStyle().Bold(true).Foreground(tuiAccent)
+	sStep  = lipgloss.NewStyle().Width(14)
+	sDim   = lipgloss.NewStyle().Foreground(tuiDim)
+	sOK    = lipgloss.NewStyle().Foreground(tuiOK)
+	sWarn  = lipgloss.NewStyle().Foreground(tuiWarn)
+	sBad   = lipgloss.NewStyle().Foreground(tuiBad)
+	sBox   = lipgloss.NewStyle().Border(lipgloss.RoundedBorder()).BorderForeground(tuiAccent).Padding(0, 1)
+)
+
+// Messages from the onboarding goroutine to the screen.
+type (
+	evMsg     onboard.Event
+	finishMsg struct {
+		node, summary string
+		err           error
+	}
+	askFabricsMsg struct {
+		suggested []onboard.Fabric
+		reply     chan []onboard.Fabric
+	}
+	askContinueMsg struct {
+		fails int
+		reply chan bool
+	}
+)
+
+type stepState struct {
+	level, detail string
+}
+
+type nodeModel struct {
+	title   string
+	steps   map[string]stepState
+	notes   []onboard.Event // warnings and failures, with their fixes
+	spin    spinner.Model
+	cancel  context.CancelFunc
+	width   int
+	done    bool
+	err     error
+	node    string
+	summary string
+
+	// The question being asked, if any.
+	fabrics   *askFabricsMsg
+	inputs    []textinput.Model
+	focus     int
+	confirm   *askContinueMsg
+	quitAsked bool
+}
+
+func newNodeModel(title string, cancel context.CancelFunc) nodeModel {
+	sp := spinner.New()
+	sp.Spinner = spinner.MiniDot
+	sp.Style = lipgloss.NewStyle().Foreground(tuiAccent)
+	return nodeModel{title: title, steps: map[string]stepState{}, spin: sp, cancel: cancel, width: 100}
+}
+
+func (m nodeModel) Init() tea.Cmd { return m.spin.Tick }
+
+func (m nodeModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
+	switch msg := msg.(type) {
+	case tea.WindowSizeMsg:
+		m.width = msg.Width
+	case spinner.TickMsg:
+		var cmd tea.Cmd
+		m.spin, cmd = m.spin.Update(msg)
+		return m, cmd
+	case evMsg:
+		e := onboard.Event(msg)
+		switch e.Level {
+		case onboard.Running, onboard.Done:
+			m.steps[e.Step] = stepState{e.Level, e.Detail}
+		case onboard.Fail:
+			m.steps[e.Step] = stepState{e.Level, e.Detail}
+			if e.Step == onboard.StepCheck && strings.Contains(e.Detail, ":") {
+				m.notes = append(m.notes, e)
+			}
+		case onboard.Warn:
+			m.notes = append(m.notes, e)
+		case onboard.Info:
+			if e.Step == onboard.StepEngines {
+				m.steps[e.Step] = stepState{onboard.Running, e.Detail}
+			}
+		}
+	case askFabricsMsg:
+		m.fabrics = &msg
+		m.inputs = nil
+		for i, f := range msg.suggested {
+			in := textinput.New()
+			in.Prompt = ""
+			in.SetValue(f.Name)
+			in.CharLimit = 40
+			in.Width = 24
+			if i == 0 {
+				in.Focus()
+			}
+			m.inputs = append(m.inputs, in)
+		}
+		m.focus = 0
+	case askContinueMsg:
+		m.confirm = &msg
+	case finishMsg:
+		m.done, m.err, m.node, m.summary = true, msg.err, msg.node, msg.summary
+		return m, tea.Quit
+	case tea.KeyMsg:
+		return m.key(msg)
+	}
+	return m, nil
+}
+
+func (m nodeModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) {
+	if k.Type == tea.KeyCtrlC {
+		m.cancel()
+		m.quitAsked = true
+		return m, tea.Quit
+	}
+	switch {
+	case m.confirm != nil:
+		switch strings.ToLower(k.String()) {
+		case "y":
+			m.confirm.reply <- true
+			m.confirm = nil
+		case "n", "enter", "esc":
+			m.confirm.reply <- false
+			m.confirm = nil
+		}
+		return m, nil
+	case m.fabrics != nil:
+		switch k.Type {
+		case tea.KeyTab, tea.KeyDown:
+			m.move(1)
+			return m, nil
+		case tea.KeyShiftTab, tea.KeyUp:
+			m.move(-1)
+			return m, nil
+		case tea.KeyEsc:
+			m.fabrics.reply <- nil
+			m.fabrics = nil
+			return m, nil
+		case tea.KeyEnter:
+			var out []onboard.Fabric
+			for i, in := range m.inputs {
+				if name := strings.TrimSpace(in.Value()); name != "" {
+					out = append(out, onboard.Fabric{Name: name, HCAs: m.fabrics.suggested[i].HCAs})
+				}
+			}
+			m.fabrics.reply <- out
+			m.fabrics = nil
+			return m, nil
+		}
+		var cmd tea.Cmd
+		m.inputs[m.focus], cmd = m.inputs[m.focus].Update(k)
+		return m, cmd
+	}
+	if k.String() == "q" {
+		m.cancel()
+		m.quitAsked = true
+		return m, tea.Quit
+	}
+	return m, nil
+}
+
+func (m *nodeModel) move(d int) {
+	m.inputs[m.focus].Blur()
+	m.focus = (m.focus + d + len(m.inputs)) % len(m.inputs)
+	m.inputs[m.focus].Focus()
+}
+
+func (m nodeModel) View() string {
+	var b strings.Builder
+	b.WriteString(sTitle.Render(m.title) + "\n\n")
+	detailWidth := max(20, m.width-22)
+	for _, step := range onboard.Steps {
+		st, ok := m.steps[step]
+		var mark string
+		switch {
+		case !ok:
+			mark = sDim.Render("·")
+		case st.level == onboard.Running && !m.done:
+			mark = m.spin.View()
+		case st.level == onboard.Fail:
+			mark = sBad.Render("✗")
+		case st.level == onboard.Running:
+			mark = sDim.Render("·")
+		default:
+			mark = sOK.Render("✓")
+		}
+		name := sStep.Render(step)
+		if !ok {
+			name = sDim.Render(sStep.Render(step))
+		}
+		fmt.Fprintf(&b, "  %s %s %s\n", mark, name, truncate(st.detail, detailWidth))
+	}
+
+	if len(m.notes) > 0 {
+		b.WriteString("\n")
+		for _, n := range m.notes {
+			mark := sWarn.Render("!")
+			if n.Level == onboard.Fail {
+				mark = sBad.Render("✗")
+			}
+			fmt.Fprintf(&b, "  %s %s\n", mark, wrap(n.Detail, m.width-6, "    "))
+			if n.Fix != "" {
+				fmt.Fprintf(&b, "    %s\n", sDim.Render("→ "+wrap(n.Fix, m.width-8, "      ")))
+			}
+		}
+	}
+
+	switch {
+	case m.fabrics != nil:
+		var q strings.Builder
+		q.WriteString("These adapters are up. Which fabric is each group on?\n")
+		q.WriteString(sDim.Render("Give each the name it has on every other node attached to it.\nLeave a name empty to leave that group out.") + "\n\n")
+		for i, f := range m.fabrics.suggested {
+			fmt.Fprintf(&q, "  %s  %s\n", m.inputs[i].View(), sDim.Render(strings.Join(f.HCAs, ", ")))
+		}
+		q.WriteString("\n" + sDim.Render("tab next · enter use these · esc none"))
+		b.WriteString("\n" + sBox.Render(q.String()) + "\n")
+	case m.confirm != nil:
+		q := fmt.Sprintf("doctor found %d problem(s) above.\nAdd the node anyway? %s", m.confirm.fails, sDim.Render("y / N"))
+		b.WriteString("\n" + sBox.BorderForeground(tuiWarn).Render(q) + "\n")
+	case m.done && m.err != nil:
+		b.WriteString("\n" + sBad.Render("✗ ") + wrap(m.err.Error(), m.width-4, "  ") + "\n")
+	case m.done:
+		b.WriteString("\n" + sOK.Render("✓ ") + m.summary + "\n")
+	case m.quitAsked:
+		b.WriteString("\n" + sDim.Render("stopped") + "\n")
+	default:
+		b.WriteString("\n" + sDim.Render("ctrl+c to stop") + "\n")
+	}
+	return b.String()
+}
+
+func truncate(s string, n int) string {
+	if n <= 1 || len([]rune(s)) <= n {
+		return s
+	}
+	r := []rune(s)
+	return string(r[:n-1]) + "…"
+}
+
+// wrap breaks s at spaces to fit width, indenting every line after the first.
+func wrap(s string, width int, indent string) string {
+	if width < 20 {
+		return s
+	}
+	var lines []string
+	line := ""
+	for _, w := range strings.Fields(s) {
+		if line != "" && len(line)+1+len(w) > width {
+			lines = append(lines, line)
+			line = w
+			continue
+		}
+		if line == "" {
+			line = w
+		} else {
+			line += " " + w
+		}
+	}
+	lines = append(lines, line)
+	return strings.Join(lines, "\n"+indent)
+}
+
+// runTUI adds the node with the screen above.
+func (na *nodeAdd) runTUI(ctx context.Context) error {
+	ctx, cancel := context.WithCancel(ctx)
+	defer cancel()
+	m := newNodeModel("llmgw · adding "+na.opts.Addr, cancel)
+	p := tea.NewProgram(m)
+
+	emit := func(e onboard.Event) { p.Send(evMsg(e)) }
+	na.opts.AskFabrics = func(s []onboard.Fabric) []onboard.Fabric {
+		reply := make(chan []onboard.Fabric)
+		p.Send(askFabricsMsg{s, reply})
+		select {
+		case f := <-reply:
+			return f
+		case <-ctx.Done():
+			return nil
+		}
+	}
+	na.opts.AskContinue = func(fails int) bool {
+		reply := make(chan bool)
+		p.Send(askContinueMsg{fails, reply})
+		select {
+		case ok := <-reply:
+			return ok
+		case <-ctx.Done():
+			return false
+		}
+	}
+
+	go func() {
+		res, err := onboard.New(na.opts, emit).Run(ctx)
+		node := ""
+		if res != nil {
+			node = res.Node
+		}
+		if err == nil && !na.noWait {
+			err = na.watch(ctx, node, emit)
+		}
+		summary := ""
+		if err == nil {
+			summary = node + " is in the cluster"
+		}
+		p.Send(finishMsg{node, summary, err})
+	}()
+
+	final, err := p.Run()
+	if err != nil {
+		return err
+	}
+	fm := final.(nodeModel)
+	if fm.quitAsked && !fm.done {
+		return fmt.Errorf("stopped; the node may be half set up, and running this again finishes it")
+	}
+	return fm.err
+}

internal/cli/tui_node_test.go

@@ -0,0 +1,96 @@
+package cli
+
+import (
+	"strings"
+	"testing"
+
+	"time"
+
+	tea "github.com/charmbracelet/bubbletea"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
+)
+
+func send(m tea.Model, msgs ...tea.Msg) tea.Model {
+	for _, msg := range msgs {
+		m, _ = m.Update(msg)
+	}
+	return m
+}
+
+func TestTheNodeScreenFillsInAsStepsFinish(t *testing.T) {
+	var m tea.Model = newNodeModel("adding gpu01", func() {})
+	m = send(m,
+		evMsg{Step: onboard.StepConnect, Level: onboard.Done, Detail: "gpu01 (amd64)"},
+		evMsg{Step: onboard.StepCheck, Level: onboard.Warn, Detail: "fabric: nvidia_peermem is not loaded", Fix: "modprobe nvidia_peermem"},
+		evMsg{Step: onboard.StepCheck, Level: onboard.Running, Detail: "running llmgw doctor"},
+	)
+	v := m.View()
+	for _, want := range []string{"✓", "gpu01 (amd64)", "nvidia_peermem is not loaded", "→ modprobe nvidia_peermem", "running llmgw doctor", "engines"} {
+		if !strings.Contains(v, want) {
+			t.Errorf("screen lacks %q:\n%s", want, v)
+		}
+	}
+}
+
+func TestTheFabricQuestionAnswersWithWhatWasTyped(t *testing.T) {
+	var m tea.Model = newNodeModel("adding gpu01", func() {})
+	reply := make(chan []onboard.Fabric, 1)
+	m = send(m, askFabricsMsg{suggested: []onboard.Fabric{
+		{Name: "fabric1", HCAs: []string{"mlx5_0", "mlx5_1"}},
+		{Name: "fabric2", HCAs: []string{"mlx5_4"}},
+	}, reply: reply})
+	if v := m.View(); !strings.Contains(v, "Which fabric") || !strings.Contains(v, "mlx5_0, mlx5_1") {
+		t.Fatalf("no question on screen:\n%s", m.View())
+	}
+	// Rename the first to ib-a, empty the second to leave it out.
+	for range len("fabric1") {
+		m = send(m, tea.KeyMsg{Type: tea.KeyBackspace})
+	}
+	m = send(m, tea.KeyMsg{Type: tea.KeyRunes, Runes: []rune("ib-a")}, tea.KeyMsg{Type: tea.KeyTab})
+	for range len("fabric2") {
+		m = send(m, tea.KeyMsg{Type: tea.KeyBackspace})
+	}
+	m = send(m, tea.KeyMsg{Type: tea.KeyEnter})
+	got := <-reply
+	if len(got) != 1 || got[0].Name != "ib-a" || strings.Join(got[0].HCAs, ",") != "mlx5_0,mlx5_1" {
+		t.Errorf("got %+v", got)
+	}
+	if strings.Contains(m.View(), "Which fabric") {
+		t.Error("the question stayed on screen after it was answered")
+	}
+}
+
+func TestContinuingPastAProblemIsNoUnlessSaidYes(t *testing.T) {
+	var m tea.Model = newNodeModel("adding gpu01", func() {})
+	reply := make(chan bool, 1)
+	m = send(m, askContinueMsg{fails: 2, reply: reply}, tea.KeyMsg{Type: tea.KeyEnter})
+	if <-reply {
+		t.Error("enter went on past the problems")
+	}
+	m = send(m, askContinueMsg{fails: 2, reply: reply}, tea.KeyMsg{Type: tea.KeyRunes, Runes: []rune("y")})
+	if !<-reply {
+		t.Error("y did not go on")
+	}
+	_ = m
+}
+
+func TestANodeCountsAsJoinedOnlyWhenHeardSinceItsAgentStarted(t *testing.T) {
+	start := time.Now().Add(-10 * time.Second)
+	cases := []struct {
+		state, ago string
+		want       bool
+	}{
+		{"ready", "3s", true},
+		{"cordoned", "3s", true}, // added again while cordoned
+		{"ready", "30s", false},  // the agent before the restart
+		{"unreachable", "3s", false},
+		{"ready", "", false},
+	}
+	for _, c := range cases {
+		if got := heardSince(adminapi.NodeView{State: c.state, LastSeenAgo: c.ago}, start); got != c.want {
+			t.Errorf("%s seen %q ago: got %v", c.state, c.ago, got)
+		}
+	}
+}

internal/cli/users.go

@@ -1,7 +1,9 @@
 package cli
 
 import (
+	"errors"
 	"fmt"
+	"strings"
 	"text/tabwriter"
 	"time"
 
@@ -33,6 +35,7 @@ func newUsersCmd() *cobra.Command {
 		newUserStateCmd(open, "reinstate", store.UserActive,
 			"Let a suspended account work again"),
 		newUserCreateCmd(open), newUserResetPasswordCmd(open),
+		newUserJoinCmd(open), newUserLeaveCmd(open), newUserDeleteCmd(open),
 		newUserAdminCmd(open, "promote", true, "Let an account see the cluster dashboard at /admin"),
 		newUserAdminCmd(open, "demote", false, "Take away access to the cluster dashboard"))
 	return cmd
@@ -66,16 +69,28 @@ func newUserListCmd(open userOpener) *cobra.Command {
 				return nil
 			}
 
+			all, err := s.AllMemberships(cmd.Context())
+			if err != nil {
+				return err
+			}
+			where := map[string][]string{}
+			for _, m := range all {
+				where[m.UserID] = append(where[m.UserID], m.Path)
+			}
 			w := tabwriter.NewWriter(cmd.OutOrStdout(), 0, 0, 2, ' ', 0)
-			fmt.Fprintln(w, "ID\tEMAIL\tNAME\tSTATE\tADMIN\tPROJECT\tSIGNED UP\tLAST LOGIN")
+			fmt.Fprintln(w, "ID\tEMAIL\tSTATE\tADMIN\tMEMBER OF\tSIGNED UP\tLAST LOGIN")
 			now := time.Now()
 			for _, u := range users {
 				admin := "-"
 				if u.IsAdmin {
 					admin = "yes"
 				}
-				fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n",
-					u.ID, u.Email, u.Name, u.State, admin, dash(u.ProjectID),
+				member := "on their own"
+				if ms := where[u.ID]; len(ms) > 0 {
+					member = strings.Join(ms, "; ")
+				}
+				fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\t%s\n",
+					u.ID, u.Email, u.State, admin, member,
 					ago(u.CreatedAt, now), ago(u.LastLoginAt, now))
 			}
 			return w.Flush()
@@ -88,11 +103,11 @@ func newUserListCmd(open userOpener) *cobra.Command {
 func newUserApproveCmd(open userOpener) *cobra.Command {
 	var project, by string
 	cmd := &cobra.Command{
-		Use:   "approve <user-id>",
+		Use:   "approve <user>",
 		Short: "Let an account sign in and issue its own keys",
-		Long: "The project is required: it decides which limits this person's keys\n" +
-			"inherit. An account approved into no project would pass every check and\n" +
-			"then fail at the first key it tried to make.",
+		Long: "A project is optional. Approved into nothing, a person uses the gateway on\n" +
+			"their own — personal keys and chat within their personal limits — and can\n" +
+			"be added to organizations, teams and projects with 'llmgw users join'.",
 		Args: cobra.ExactArgs(1),
 		RunE: func(cmd *cobra.Command, args []string) error {
 			s, err := open(cmd)
@@ -101,27 +116,37 @@ func newUserApproveCmd(open userOpener) *cobra.Command {
 			}
 			defer s.Close()
 
-			if err := s.ApproveUser(cmd.Context(), args[0], project, by, time.Now()); err != nil {
-				return fmt.Errorf("approving %s: %w", args[0], err)
-			}
-			u, err := s.UserByID(cmd.Context(), args[0])
+			u, err := resolveUser(cmd.Context(), s, args[0])
 			if err != nil {
 				return err
 			}
-			fmt.Fprintf(cmd.OutOrStdout(), "%s approved into project %s\n", u.Email, project)
+			var p store.ProjectPath
+			if project != "" {
+				if p, err = resolveProject(cmd.Context(), s, project); err != nil {
+					return err
+				}
+			}
+			if err := s.ApproveUser(cmd.Context(), u.ID, p.ID, by, time.Now()); err != nil {
+				return fmt.Errorf("approving %s: %w", u.Email, err)
+			}
+			if p.ID == "" {
+				fmt.Fprintf(cmd.OutOrStdout(), "%s approved, on their own for now\n", u.Email)
+			} else {
+				fmt.Fprintf(cmd.OutOrStdout(), "%s approved into %s\n", u.Email, p.Path)
+			}
 			cmd.PrintErrln("They can sign in and create their own keys from /profile.")
 			return nil
 		},
 	}
-	cmd.Flags().StringVar(&project, "project", "default", "project the person's keys belong to")
+	cmd.Flags().StringVar(&project, "project", "",
+		"a project to put them in (optional): id, path (Acme/platform/gateway) or unique name")
 	cmd.Flags().StringVar(&by, "by", "", "who approved, for the record")
-	_ = cmd.MarkFlagRequired("project")
 	return cmd
 }
 
 func newUserStateCmd(open userOpener, verb string, state store.UserState, short string) *cobra.Command {
 	return &cobra.Command{
-		Use:   verb + " <user-id>",
+		Use:   verb + " <user>",
 		Short: short,
 		Args:  cobra.ExactArgs(1),
 		RunE: func(cmd *cobra.Command, args []string) error {
@@ -131,18 +156,22 @@ func newUserStateCmd(open userOpener, verb string, state store.UserState, short
 			}
 			defer s.Close()
 
-			if err := s.SetUserState(cmd.Context(), args[0], state); err != nil {
+			u, err := resolveUser(cmd.Context(), s, args[0])
+			if err != nil {
+				return err
+			}
+			if err := s.SetUserState(cmd.Context(), u.ID, state); err != nil {
 				return err
 			}
 			if state == store.UserSuspended {
 				// Otherwise the person stays signed in and suspension is
 				// advisory until their session happens to expire.
-				if err := s.DeleteUserSessions(cmd.Context(), args[0]); err != nil {
+				if err := s.DeleteUserSessions(cmd.Context(), u.ID); err != nil {
 					return err
 				}
 				cmd.PrintErrln("Their sessions were closed. Their API keys still work — revoke those separately.")
 			}
-			fmt.Fprintf(cmd.OutOrStdout(), "%s is now %s\n", args[0], state)
+			fmt.Fprintf(cmd.OutOrStdout(), "%s is now %s\n", u.Email, state)
 			return nil
 		},
 	}
@@ -150,7 +179,7 @@ func newUserStateCmd(open userOpener, verb string, state store.UserState, short
 
 func newUserAdminCmd(open userOpener, verb string, admin bool, short string) *cobra.Command {
 	return &cobra.Command{
-		Use:   verb + " <user-id>",
+		Use:   verb + " <user>",
 		Short: short,
 		Long: "The dashboard is read-only, so this grants looking and nothing else.\n" +
 			"Everything that changes the cluster or issues a credential stays here on\n" +
@@ -164,11 +193,11 @@ func newUserAdminCmd(open userOpener, verb string, admin bool, short string) *co
 			}
 			defer s.Close()
 
-			if err := s.SetAdmin(cmd.Context(), args[0], admin); err != nil {
+			u, err := resolveUser(cmd.Context(), s, args[0])
+			if err != nil {
 				return err
 			}
-			u, err := s.UserByID(cmd.Context(), args[0])
-			if err != nil {
+			if err := s.SetAdmin(cmd.Context(), u.ID, admin); err != nil {
 				return err
 			}
 			if admin {
@@ -186,7 +215,7 @@ func newUserAdminCmd(open userOpener, verb string, admin bool, short string) *co
 func newUserResetPasswordCmd(open userOpener) *cobra.Command {
 	var password string
 	cmd := &cobra.Command{
-		Use:   "reset-password <user-id>",
+		Use:   "reset-password <user>",
 		Short: "Set a new password for someone who lost theirs",
 		Long: "There is no self-service reset, because this gateway sends no email and\n" +
 			"a reset link with nowhere to send it is not a reset. Set one here, tell\n" +
@@ -208,13 +237,17 @@ func newUserResetPasswordCmd(open userOpener) *cobra.Command {
 			if err != nil {
 				return err
 			}
-			if err := s.SetPassword(cmd.Context(), args[0], hash); err != nil {
+			u, err := resolveUser(cmd.Context(), s, args[0])
+			if err != nil {
 				return err
 			}
-			if err := s.DeleteUserSessions(cmd.Context(), args[0]); err != nil {
+			if err := s.SetPassword(cmd.Context(), u.ID, hash); err != nil {
 				return err
 			}
-			fmt.Fprintf(cmd.OutOrStdout(), "password set for %s\n", args[0])
+			if err := s.DeleteUserSessions(cmd.Context(), u.ID); err != nil {
+				return err
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "password set for %s\n", u.Email)
 			cmd.PrintErrln("Their sessions were closed. Their API keys still work.")
 			return nil
 		},
@@ -241,6 +274,12 @@ func newUserCreateCmd(open userOpener) *cobra.Command {
 			}
 			defer s.Close()
 
+			var p store.ProjectPath
+			if project != "" {
+				if p, err = resolveProject(cmd.Context(), s, project); err != nil {
+					return err
+				}
+			}
 			hash, err := account.HashPassword(password)
 			if err != nil {
 				return err
@@ -253,15 +292,163 @@ func newUserCreateCmd(open userOpener) *cobra.Command {
 			if err := s.CreateUser(cmd.Context(), u); err != nil {
 				return err
 			}
-			if err := s.ApproveUser(cmd.Context(), id, project, "cli", time.Now()); err != nil {
+			if err := s.ApproveUser(cmd.Context(), id, p.ID, "cli", time.Now()); err != nil {
 				return err
 			}
 			fmt.Fprintf(cmd.OutOrStdout(), "%s\n", id)
-			cmd.PrintErrf("%s created and approved into project %s\n", args[0], project)
+			if p.ID == "" {
+				cmd.PrintErrf("%s created and approved, on their own for now\n", args[0])
+			} else {
+				cmd.PrintErrf("%s created and approved into %s\n", args[0], p.Path)
+			}
 			return nil
 		},
 	}
-	cmd.Flags().StringVar(&project, "project", "default", "project the person's keys belong to")
+	cmd.Flags().StringVar(&project, "project", "",
+		"a project to put them in (optional): id, path or unique name")
 	cmd.Flags().StringVar(&password, "password", "", "initial password")
 	return cmd
 }
+
+// scopeFlags names one org, team or project by whichever flag is given.
+type scopeFlags struct{ org, team, project string }
+
+func (f *scopeFlags) bind(cmd *cobra.Command) {
+	cmd.Flags().StringVar(&f.org, "org", "", "an organization: id or name")
+	cmd.Flags().StringVar(&f.team, "team", "", "a team: id, path (Acme/ml) or unique name")
+	cmd.Flags().StringVar(&f.project, "project", "", "a project: id, path (Acme/ml/training) or unique name")
+}
+
+func (f *scopeFlags) resolve(cmd *cobra.Command, s *store.DB) (store.Scope, string, string, error) {
+	n := 0
+	for _, v := range []string{f.org, f.team, f.project} {
+		if v != "" {
+			n++
+		}
+	}
+	if n != 1 {
+		return "", "", "", fmt.Errorf("give exactly one of --org, --team or --project")
+	}
+	ctx := cmd.Context()
+	switch {
+	case f.org != "":
+		o, err := resolveOrg(ctx, s, f.org)
+		return store.ScopeOrg, o.ID, o.Name, err
+	case f.team != "":
+		t, err := resolveTeam(ctx, s, f.team)
+		return store.ScopeTeam, t.ID, t.OrgName + " / " + t.Name, err
+	default:
+		p, err := resolveProject(ctx, s, f.project)
+		return store.ScopeProject, p.ID, p.Path, err
+	}
+}
+
+func newUserJoinCmd(open userOpener) *cobra.Command {
+	var where scopeFlags
+	var role, by string
+	cmd := &cobra.Command{
+		Use:   "join <user> --org|--team|--project <which>",
+		Short: "Make a person a member of an organization, team or project",
+		Long: "A person can be a member of any number of them. A member of a team or\n" +
+			"organization may use every project in it.",
+		Example: "  llmgw users join ayse@example.com --team Acme/ml\n" +
+			"  llmgw users join ayse@example.com --project Scania/research/vision",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			s, err := open(cmd)
+			if err != nil {
+				return err
+			}
+			defer s.Close()
+			u, err := resolveUser(cmd.Context(), s, args[0])
+			if err != nil {
+				return err
+			}
+			scope, id, label, err := where.resolve(cmd, s)
+			if err != nil {
+				return err
+			}
+			if err := s.AddMembership(cmd.Context(), u.ID, scope, id, role, by, time.Now()); err != nil {
+				return err
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "%s is a %s of %s\n", u.Email, role, label)
+			return nil
+		},
+	}
+	where.bind(cmd)
+	cmd.Flags().StringVar(&role, "role", store.RoleMember, "member, or manager: with --org, may see the organization's page")
+	cmd.Flags().StringVar(&by, "by", "", "who added them, for the record")
+	return cmd
+}
+
+func newUserLeaveCmd(open userOpener) *cobra.Command {
+	var where scopeFlags
+	cmd := &cobra.Command{
+		Use:   "leave <user> --org|--team|--project <which>",
+		Short: "Take a person out of an organization, team or project",
+		Long: "Keys they made in a project they can no longer use are revoked; a project\n" +
+			"they still reach through another membership keeps them. Their personal\n" +
+			"keys are not touched.",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			s, err := open(cmd)
+			if err != nil {
+				return err
+			}
+			defer s.Close()
+			u, err := resolveUser(cmd.Context(), s, args[0])
+			if err != nil {
+				return err
+			}
+			scope, id, label, err := where.resolve(cmd, s)
+			if err != nil {
+				return err
+			}
+			revoked, err := s.RemoveMembership(cmd.Context(), u.ID, scope, id, time.Now())
+			if errors.Is(err, store.ErrNotFound) {
+				return fmt.Errorf("%s is not a member of %s itself", u.Email, label)
+			}
+			if err != nil {
+				return err
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "%s is no longer in %s (%s revoked)\n",
+				u.Email, label, countOf(revoked, "key"))
+			return nil
+		},
+	}
+	where.bind(cmd)
+	return cmd
+}
+
+func newUserDeleteCmd(open userOpener) *cobra.Command {
+	var yes bool
+	cmd := &cobra.Command{
+		Use:   "delete <user>",
+		Short: "Remove an account for good",
+		Long: "Signs them out, revokes every key they made, and removes the account. The\n" +
+			"keys are kept, revoked and ownerless, because usage history refers to them.\n" +
+			"To stop an account working and keep it, use suspend.",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			s, err := open(cmd)
+			if err != nil {
+				return err
+			}
+			defer s.Close()
+			u, err := resolveUser(cmd.Context(), s, args[0])
+			if err != nil {
+				return err
+			}
+			if !yes {
+				return fmt.Errorf("this removes %s and revokes their keys; run again with --yes", u.Email)
+			}
+			if err := s.DeleteUser(cmd.Context(), u.ID, time.Now()); err != nil {
+				return err
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "%s removed\n", u.Email)
+			return nil
+		},
+	}
+	cmd.Flags().BoolVar(&yes, "yes", false, "confirm")
+	return cmd
+}

internal/config/config.go

@@ -204,6 +204,10 @@ type Auth struct {
 	// means accounts are made with `llmgw users create` instead.
 	SignupOpen bool `koanf:"signup_open"`
 
+	// Personal is what a person may do on their own, outside any project: make
+	// personal keys and use the chat page personally, within these limits.
+	Personal Personal `koanf:"personal"`
+
 	// Pricing is what each model costs, for budgets. A model absent from the
 	// table costs whatever DefaultPrice says, which is nothing unless set.
 	Pricing      map[string]Price `koanf:"pricing"`
@@ -218,6 +222,22 @@ type Auth struct {
 	BufferSize int `koanf:"buffer_size"`
 }
 
+// Personal configures use that belongs to a person rather than a project.
+//
+// On by default: a person need not belong to anything to use the gateway. The
+// limits are each person's own pool across all their personal keys and their
+// personal chat, and apply only where nobody wrote a limit for that person.
+type Personal struct {
+	// Disabled refuses personal keys and personal chat; people then use the
+	// gateway only through projects they are members of.
+	Disabled bool `koanf:"disabled"`
+	RPM      int  `koanf:"rpm"`
+	TPM      int  `koanf:"tpm"`
+	// Budget is in whole units per Period ("day" or "month").
+	Budget float64 `koanf:"budget"`
+	Period string  `koanf:"period"`
+}
+
 // Price is a model's cost in micro-units per thousand tokens.
 //
 // What a unit means is the operator's choice — self-hosted models have no
@@ -540,8 +560,14 @@ func (s Server) Catalog() (*catalog.Catalog, error) {
 // A model with no profiles cannot be placed, so a configuration of bare names
 // plus a static endpoint list is the router-only mode used for testing against
 // sim-engines.
+//
+// With no static endpoints there is nothing a router-only process could serve,
+// so a configuration naming neither endpoints nor models is a control plane
+// with an empty catalog — which is what every new cluster is. It used to be
+// refused as having nothing to serve, and under systemd that is a unit
+// restarting every two seconds with nothing on 8080.
 func (s Server) ControlPlaneMode() bool {
-	if s.Control.Enabled {
+	if s.Control.Enabled || len(s.Endpoints) == 0 {
 		return true
 	}
 	for _, m := range s.Models {
@@ -553,9 +579,14 @@ func (s Server) ControlPlaneMode() bool {
 }
 
 // DefaultEngines are the engine versions this release was checked against:
-// `llmgw ctl engine-check` accepted every flag llmgw builds for them.
+// every flag llmgw builds for them is one they take.
+//
+// vLLM is installed on every node with an NVIDIA GPU it can run on and
+// skipped elsewhere. SGLang is not here: it is opt-in, by naming a version
+// under engines.
 var DefaultEngines = map[string]string{
 	"llamacpp": "b10991",
+	"vllm":     "0.30.0",
 }
 
 // WantedEngines is DefaultEngines with the configuration laid over it.

internal/config/config_test.go

@@ -155,19 +155,26 @@ func TestValidateAcceptsDistinctAliases(t *testing.T) {
 	}
 }
 
-func TestAControlPlaneCanBeAskedForWithNoModelsInTheFile(t *testing.T) {
-	// Without this, a deployment whose catalog lives entirely in the database
-	// cannot start: the mode is inferred from the file, an empty models block
-	// reads as "router only", and the process refuses to start for want of
-	// anything to serve — so the stored models it would have loaded are never
-	// read at all.
+func TestAnEmptyConfigurationIsAControlPlane(t *testing.T) {
+	// A new cluster has no models and no endpoints. Refusing it as having
+	// nothing to serve left a systemd unit restarting every two seconds and
+	// nothing listening on 8080 — on the first start anyone ever made.
 	var s Server
+	if !s.ControlPlaneMode() {
+		t.Fatal("a configuration with no endpoints and no models should run a control plane")
+	}
+}
+
+func TestStaticEndpointsAloneAreRouterOnly(t *testing.T) {
+	var s Server
+	s.Endpoints = []Endpoint{{ID: "e", URL: "http://127.0.0.1:1"}}
+	s.Models = []catalog.Model{{Name: "m"}}
 	if s.ControlPlaneMode() {
-		t.Fatal("an empty configuration is not a control plane")
+		t.Error("static endpoints and bare model names are the router-only mode")
 	}
 	s.Control.Enabled = true
 	if !s.ControlPlaneMode() {
-		t.Error("control.enabled should run the control plane with no models declared")
+		t.Error("control.enabled should run the control plane alongside static endpoints")
 	}
 }
 

internal/config/example_test.go

@@ -36,9 +36,8 @@ func TestTheDocumentedAgentExampleLoads(t *testing.T) {
 	if len(f.HCAs) == 0 {
 		t.Error("the fabric's adapters did not bind")
 	}
-	if f.RateGbps == 0 {
-		t.Error("the fabric's rate did not bind")
-	}
+	// The rate is not documented: the agent measures it, as the slowest of
+	// the fabric's active ports, and a figure written here would be ignored.
 	// node_id is left to the hostname, so the example need not say it; where
 	// the control plane is, nothing can find out.
 	if cfg.Server == "" {

internal/control/adminapi/api.go

@@ -16,11 +16,15 @@ import (
 	"strings"
 	"time"
 
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/bench"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/fabrics"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/nvlink"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/topology"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
 )
 
@@ -48,6 +52,20 @@ func (a *API) Register(mux *http.ServeMux) {
 	mux.HandleFunc("GET /api/v1/capacity", a.listCapacity)
 	mux.HandleFunc("GET /api/v1/usage", a.usageByScope)
 	mux.HandleFunc("GET /api/v1/nodes", a.listNodes)
+	mux.HandleFunc("GET /api/v1/fabrics", func(w http.ResponseWriter, _ *http.Request) {
+		writeJSON(w, http.StatusOK, a.Fabrics())
+	})
+	mux.HandleFunc("GET /api/v1/nvlink", func(w http.ResponseWriter, _ *http.Request) {
+		writeJSON(w, http.StatusOK, a.NVLink())
+	})
+	mux.HandleFunc("GET /api/v1/fabrics/{id}/topology", func(w http.ResponseWriter, r *http.Request) {
+		t, ok := a.Topology(r.PathValue("id"))
+		if !ok {
+			http.NotFound(w, r)
+			return
+		}
+		writeJSON(w, http.StatusOK, t)
+	})
 	mux.HandleFunc("GET /api/v1/models", a.listModels)
 	mux.HandleFunc("GET /api/v1/replicas", a.listReplicas)
 	// Listing is a read and deleting is a separate verb, so a mistyped
@@ -56,6 +74,7 @@ func (a *API) Register(mux *http.ServeMux) {
 	mux.HandleFunc("POST /api/v1/weights/collect", a.collectWeights)
 	mux.HandleFunc("GET /api/v1/explain-placement", a.explainPlacement)
 	mux.HandleFunc("POST /api/v1/nodes/{id}/state", a.setNodeState)
+	mux.HandleFunc("DELETE /api/v1/nodes/{id}", a.removeNode)
 	mux.HandleFunc("GET /api/v1/nodes/{id}/logs", a.nodeLogs)
 }
 
@@ -115,6 +134,87 @@ type NodeView struct {
 	LastSeenAgo  string `json:"last_seen_ago"`
 	AgentVersion string `json:"agent_version,omitempty"`
 	Replicas     int    `json:"replicas"`
+
+	// Load is how busy the machine is, absent when its agent has not
+	// measured it lately — which is unknown, not idle.
+	Load *LoadView `json:"load,omitempty"`
+	// IBPorts are the node's InfiniBand and RoCE ports at the last heartbeat.
+	IBPorts []IBPortView `json:"ib_ports,omitempty"`
+	// GPUs are the cards, with how they are wired: NVLink peers, the
+	// adapter nearest each, and the NVLink domain each has joined.
+	GPUs []GPUView `json:"gpus,omitempty"`
+	// NVLinkDomain is the multi-node domain every card has joined, if any.
+	NVLinkDomain string `json:"nvlink_domain,omitempty"`
+	IMEX         string `json:"imex,omitempty"`
+	IMEXChannels int    `json:"imex_channels,omitempty"`
+}
+
+// GPUView is one card as a node's page shows it.
+type GPUView struct {
+	Index       int    `json:"index"`
+	Name        string `json:"name"`
+	NUMA        int    `json:"numa"`
+	NVLinkPeers []int  `json:"nvlink_peers,omitempty"`
+	// NearestNIC is the adapter closest to the card, and how close.
+	NearestNIC  string `json:"nearest_nic,omitempty"`
+	NICPath     string `json:"nic_path,omitempty"`
+	FabricState string `json:"fabric_state,omitempty"`
+	Domain      string `json:"domain,omitempty"`
+	Bandwidth   string `json:"bandwidth,omitempty"`
+}
+
+// IBPortView is one port as a page shows it.
+type IBPortView struct {
+	HCA       string `json:"hca"`
+	Port      int    `json:"port"`
+	State     string `json:"state"`
+	PhysState string `json:"phys_state,omitempty"`
+	RateGbps  int    `json:"rate_gbps"`
+	LinkLayer string `json:"link_layer,omitempty"`
+	Active    bool   `json:"active"`
+	// Fabric is the declared fabric this port's adapter carries, if any.
+	Fabric string `json:"fabric,omitempty"`
+	// Physical is the fabric the port is physically on: its subnet manager,
+	// or its RoCE network. SMError says why that is not known.
+	Physical string `json:"physical,omitempty"`
+	SMError  string `json:"sm_error,omitempty"`
+	// Errors are the error counters that are not zero, by name.
+	Errors  map[string]uint64 `json:"errors,omitempty"`
+	TxBytes uint64            `json:"tx_bytes,omitempty"`
+	RxBytes uint64            `json:"rx_bytes,omitempty"`
+}
+
+// LoadView is a node's CPU and memory in use, at its last heartbeat.
+type LoadView struct {
+	CPUPercent float64 `json:"cpu_percent"`
+	CPUs       int     `json:"cpus,omitempty"`
+	Load1      float64 `json:"load1"`
+	Load5      float64 `json:"load5"`
+	Load15     float64 `json:"load15"`
+	MemUsed    int64   `json:"memory_used_bytes"`
+	MemTotal   int64   `json:"memory_total_bytes"`
+	// MemPercent is used over used plus available.
+	MemPercent float64 `json:"memory_percent"`
+	AgeSec     int64   `json:"age_seconds"`
+}
+
+// loadFresh is how old a measurement may be and still be shown as current.
+const loadFresh = time.Minute
+
+func loadView(l *inventory.Load, now time.Time) *LoadView {
+	if l == nil || now.Sub(l.At) > loadFresh {
+		return nil
+	}
+	v := &LoadView{
+		CPUPercent: l.CPUPercent, CPUs: l.CPUCount,
+		Load1: l.Load1, Load5: l.Load5, Load15: l.Load15,
+		MemUsed: l.MemUsedBytes, MemTotal: l.MemUsedBytes + l.MemAvailBytes,
+		AgeSec: int64(now.Sub(l.At).Seconds()),
+	}
+	if v.MemTotal > 0 {
+		v.MemPercent = float64(v.MemUsed) * 100 / float64(v.MemTotal)
+	}
+	return v
 }
 
 // ModelView is a model as `ctl models` shows it.
@@ -170,6 +270,10 @@ type ShardView struct {
 	// waiting on one. It is what tells a replica pulling a hundred gigabytes
 	// apart from one that is wedged — which look identical without it.
 	Fetch *FetchView `json:"fetch,omitempty"`
+	// CPUCores and MemoryBytes are the engine's use of its node, from the
+	// node's last heartbeat; zero when not measured.
+	CPUCores    float64 `json:"cpu_cores,omitempty"`
+	MemoryBytes int64   `json:"memory_bytes,omitempty"`
 }
 
 // FetchView is a download in flight.
@@ -266,6 +370,47 @@ func (a *API) Nodes() []NodeView {
 		if !n.LastSeen.IsZero() {
 			v.LastSeenAgo = time.Since(n.LastSeen).Round(time.Second).String()
 		}
+		v.Load = loadView(n.Load, time.Now())
+		v.NVLinkDomain, v.IMEX, v.IMEXChannels = n.NVLinkDomain(), n.IMEX.State, n.IMEX.Channels
+		for _, g := range n.GPUs {
+			gv := GPUView{Index: g.Index, Name: g.Name, NUMA: g.NUMANode, NVLinkPeers: g.NVLinkPeers}
+			for hca, rel := range g.NICPaths {
+				if gv.NearestNIC == "" || discovery.PathRank(rel) < discovery.PathRank(gv.NICPath) ||
+					(discovery.PathRank(rel) == discovery.PathRank(gv.NICPath) && hca < gv.NearestNIC) {
+					gv.NearestNIC, gv.NICPath = hca, rel
+				}
+			}
+			if f := g.Fabric; f != nil {
+				gv.FabricState, gv.Domain, gv.Bandwidth = f.State, f.Domain(), f.Bandwidth
+			}
+			v.GPUs = append(v.GPUs, gv)
+		}
+		fabricOf := map[string]string{}
+		for _, f := range n.Fabrics {
+			for _, h := range f.HCAs {
+				fabricOf[h] = f.Fabric
+			}
+		}
+		for _, p := range n.IBPorts {
+			pv := IBPortView{HCA: p.HCA, Port: p.Port, State: p.State, PhysState: p.PhysState,
+				RateGbps: p.RateGbps, LinkLayer: p.LinkLayer, Active: p.Active(), Fabric: fabricOf[p.HCA],
+				TxBytes: p.Counters["port_xmit_data"], RxBytes: p.Counters["port_rcv_data"], SMError: p.SMError}
+			switch {
+			case p.RoCE() && p.Network != "":
+				pv.Physical = "RoCE " + p.Network
+			case p.SMGUID != "":
+				pv.Physical = fmt.Sprintf("SM %s (LID %d)", p.SMGUID, p.SMLID)
+			}
+			for name, c := range p.Counters {
+				if c > 0 && name != "port_xmit_data" && name != "port_rcv_data" {
+					if pv.Errors == nil {
+						pv.Errors = map[string]uint64{}
+					}
+					pv.Errors[name] = c
+				}
+			}
+			v.IBPorts = append(v.IBPorts, pv)
+		}
 		out = append(out, v)
 	}
 	return out
@@ -308,6 +453,12 @@ func (a *API) listReplicas(w http.ResponseWriter, r *http.Request) {
 // Replicas builds the replica view, optionally filtered to one model.
 func (a *API) Replicas(model string) []ReplicaView {
 	out := make([]ReplicaView, 0)
+	loads := map[string]*inventory.Load{}
+	for _, n := range a.ctl.Inventory().Nodes() {
+		if n.Load != nil && time.Since(n.Load.At) <= loadFresh {
+			loads[n.ID] = n.Load
+		}
+	}
 	for _, rep := range a.ctl.Reconciler().Replicas() {
 		if model != "" && rep.Model != model {
 			continue
@@ -338,6 +489,11 @@ func (a *API) Replicas(model string) []ReplicaView {
 					sv.Fetch.Percent = &p
 				}
 			}
+			if l := loads[s.NodeID]; l != nil {
+				if use, ok := l.Shards[s.ID]; ok {
+					sv.CPUCores, sv.MemoryBytes = use.CPUCores, use.RSSBytes
+				}
+			}
 			v.Shards = append(v.Shards, sv)
 		}
 		out = append(out, v)
@@ -657,6 +813,35 @@ func (a *API) Explain(model string) ExplainView {
 // One endpoint rather than three verbs, because they are the same operation
 // with different targets and the difference matters: cordon stops new
 // placement, drain also moves existing work off.
+// removeNode forgets a node that has gone for good: a machine retired or
+// rebuilt under another name, which otherwise stays listed as lost until the
+// control plane restarts.
+//
+// Only one that is not there: lost, or unreachable with nothing placed on
+// it. A connected node removed here would register again on its next
+// heartbeat, and one with replicas would take them out of the reconciler's
+// sight while they may still be running.
+func (a *API) removeNode(w http.ResponseWriter, r *http.Request) {
+	id := r.PathValue("id")
+	inv := a.ctl.Inventory()
+	n, ok := inv.Get(id)
+	if !ok {
+		writeError(w, http.StatusNotFound, fmt.Sprintf("no node %q", id))
+		return
+	}
+	switch {
+	case n.State == inventory.Lost:
+	case n.State == inventory.Unreachable && len(inv.Allocations(id)) == 0:
+	default:
+		writeError(w, http.StatusConflict, fmt.Sprintf(
+			"%s is %s: stop its agent (systemctl stop llmgw-agent) and remove it once it is lost, a minute later", id, n.State))
+		return
+	}
+	inv.Remove(id)
+	a.ctl.Reconciler().Notify()
+	writeJSON(w, http.StatusOK, map[string]string{"id": id, "removed": "true"})
+}
+
 func (a *API) setNodeState(w http.ResponseWriter, r *http.Request) {
 	id := r.PathValue("id")
 
@@ -726,3 +911,127 @@ func (a *API) profileChanged(model, profile, was string) bool {
 	}
 	return false
 }
+
+// Fabrics is the physical fabrics the adapters are on, against the names
+// the nodes gave them.
+func (a *API) Fabrics() fabrics.Report {
+	inv := a.ctl.Inventory()
+	return fabrics.Analyze(inv.Nodes(), inv.SMChanges(), time.Now())
+}
+
+// TopologyView is one fabric's map, with the cluster's own adapters on it
+// live: their state, rate and traffic from the last heartbeat.
+type TopologyView struct {
+	ID        string          `json:"id"`
+	Key       string          `json:"key"`
+	Label     string          `json:"label"`
+	Declared  []string        `json:"declared"`
+	ScannedAt time.Time       `json:"scanned_at"`
+	ScannedBy string          `json:"scanned_by,omitempty"`
+	TookMs    int64           `json:"took_ms"`
+	Err       string          `json:"error,omitempty"`
+	Map       *topology.View  `json:"map,omitempty"`
+	Live      map[string]Live `json:"live"`
+	// Problems are the fabric's own, from the analysis of its identity and
+	// subnet managers.
+	Problems []fabrics.Problem `json:"problems,omitempty"`
+	SM       *fabrics.SM       `json:"sm,omitempty"`
+	SMs      []fabrics.SM      `json:"sms,omitempty"`
+}
+
+// Live is one of the cluster's adapters as its node last reported it.
+type Live struct {
+	Node   string            `json:"node"`
+	HCA    string            `json:"hca"`
+	State  string            `json:"state"`
+	Phys   string            `json:"phys_state"`
+	Gbps   int               `json:"gbps"`
+	TxBps  float64           `json:"tx_bps"`
+	RxBps  float64           `json:"rx_bps"`
+	Errors map[string]uint64 `json:"errors,omitempty"`
+}
+
+// FabricID is a physical fabric's key as an address can carry it.
+func FabricID(key string) string {
+	return strings.NewReplacer(":", "-", "/", "_").Replace(key)
+}
+
+// Topologies lists every physical fabric with what is known of its map.
+func (a *API) Topologies() []TopologyView {
+	rep := a.Fabrics()
+	scans := a.ctl.Inventory().Scans()
+	out := make([]TopologyView, 0, len(rep.Physical))
+	for _, ph := range rep.Physical {
+		t := TopologyView{ID: FabricID(ph.Key), Key: ph.Key, Label: ph.Label, Declared: ph.Declared}
+		if sc, ok := scans[ph.Key]; ok {
+			t.ScannedAt, t.ScannedBy, t.TookMs, t.Err = sc.At, sc.Node+" "+sc.Port, sc.Took.Milliseconds(), sc.Err
+		}
+		out = append(out, t)
+	}
+	return out
+}
+
+// Topology is one fabric's map, by the id Topologies gives it.
+func (a *API) Topology(id string) (TopologyView, bool) {
+	rep := a.Fabrics()
+	var ph *fabrics.Physical
+	for i := range rep.Physical {
+		if FabricID(rep.Physical[i].Key) == id {
+			ph = &rep.Physical[i]
+		}
+	}
+	if ph == nil {
+		return TopologyView{}, false
+	}
+	t := TopologyView{ID: id, Key: ph.Key, Label: ph.Label, Declared: ph.Declared, Live: map[string]Live{},
+		SM: ph.SM, SMs: ph.SMs}
+	for _, p := range rep.Problems {
+		for _, d := range ph.Declared {
+			if p.Fabric == d {
+				t.Problems = append(t.Problems, p)
+				break
+			}
+		}
+	}
+	hosts := map[string]topology.Host{}
+	for _, n := range a.ctl.Inventory().Nodes() {
+		for _, p := range n.IBPorts {
+			if p.NodeGUID == "" {
+				continue
+			}
+			hosts[p.NodeGUID] = topology.Host{Node: n.ID, HCA: p.HCA}
+			l := Live{Node: n.ID, HCA: p.HCA, State: p.State, Phys: p.PhysState, Gbps: p.RateGbps,
+				TxBps: p.TxBps, RxBps: p.RxBps}
+			for name, c := range p.Counters {
+				if c > 0 && name != "port_xmit_data" && name != "port_rcv_data" {
+					if l.Errors == nil {
+						l.Errors = map[string]uint64{}
+					}
+					l.Errors[name] = c
+				}
+			}
+			t.Live[p.NodeGUID] = l
+		}
+	}
+	sc, ok := a.ctl.Inventory().Scans()[ph.Key]
+	if !ok {
+		return t, true
+	}
+	t.ScannedAt, t.ScannedBy, t.TookMs, t.Err = sc.At, sc.Node+" "+sc.Port, sc.Took.Milliseconds(), sc.Err
+	if sc.Output != "" {
+		f, err := topology.Parse(sc.Output)
+		if err != nil {
+			t.Err = "the map could not be read: " + err.Error()
+			return t, true
+		}
+		v := topology.Build(f, hosts)
+		t.Map = &v
+	}
+	return t, true
+}
+
+// NVLink is the cluster's multi-node NVLink domains, and what keeps any of
+// their nodes out.
+func (a *API) NVLink() nvlink.Report {
+	return nvlink.Analyze(a.ctl.Inventory().Nodes())
+}

internal/control/adminapi/api_test.go

@@ -667,3 +667,24 @@ func TestAProfileThatIsGoneIsNotReportedAsStale(t *testing.T) {
 		t.Error("a missing model is not a changed profile")
 	}
 }
+
+func TestOnlyANodeThatIsGoneCanBeRemoved(t *testing.T) {
+	a, ctl := setup(t, testModels(), gpuNode("n1", 8))
+
+	// Connected, even cordoned: it would register again on its next
+	// heartbeat, so removing it would do nothing but confuse.
+	ctl.Inventory().SetState("n1", inventory.Cordoned)
+	if code := call(t, a, "DELETE", "/api/v1/nodes/n1", "", nil); code != 409 {
+		t.Errorf("a connected node gave %d, expected 409", code)
+	}
+	ctl.Inventory().SetState("n1", inventory.Lost)
+	if code := call(t, a, "DELETE", "/api/v1/nodes/n1", "", nil); code != 200 {
+		t.Fatalf("a lost node gave %d, expected 200", code)
+	}
+	if _, ok := ctl.Inventory().Get("n1"); ok {
+		t.Error("the node is still listed")
+	}
+	if code := call(t, a, "DELETE", "/api/v1/nodes/n1", "", nil); code != 404 {
+		t.Errorf("removing it again gave %d, expected 404", code)
+	}
+}

internal/control/agentsvc/convert.go

@@ -3,6 +3,7 @@ package agentsvc
 
 import (
 	"fmt"
+	"maps"
 	"time"
 
 	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
@@ -59,6 +60,9 @@ func NodeFromRegister(r *llmgwv1.Register) inventory.Node {
 			UtilizationPercent: int(g.GetUtilizationPercent()),
 			CUDAVersion:        g.GetCudaVersion(),
 			NVLinkPeers:        intSlice(g.GetNvlinkPeers()),
+			NICPaths:           maps.Clone(g.GetNicPaths()),
+			Fabric: fabricFrom(g.GetFabricState(), g.GetFabricStatus(), g.GetClusterUuid(),
+				g.GetCliqueId(), g.GetFabricBandwidth()),
 			Health: discovery.GPUHealth{
 				ECCErrors:    int(g.GetEccErrors()),
 				TemperatureC: int(g.GetTemperatureC()),
@@ -66,6 +70,7 @@ func NodeFromRegister(r *llmgwv1.Register) inventory.Node {
 			},
 		})
 	}
+	n.IMEX = discovery.IMEX{State: r.GetImex().GetState(), Channels: int(r.GetImex().GetChannels())}
 	for _, f := range r.GetFabrics() {
 		n.Fabrics = append(n.Fabrics, inventory.FabricAttachment{
 			Fabric:   f.GetFabric(),
@@ -112,11 +117,18 @@ func RegisterFromInventory(nodeID, hostname, mgmtAddr, agentVersion string,
 			UtilizationPercent: int32(g.UtilizationPercent),
 			CudaVersion:        g.CUDAVersion,
 			NvlinkPeers:        int32Slice(g.NVLinkPeers),
+			NicPaths:           maps.Clone(g.NICPaths),
 			Problems:           g.Health.Problems,
 			TemperatureC:       int32(g.Health.TemperatureC),
 			EccErrors:          int32(g.Health.ECCErrors),
 		})
+		if f := g.Fabric; f != nil {
+			last := r.Gpus[len(r.Gpus)-1]
+			last.FabricState, last.FabricStatus, last.ClusterUuid = f.State, f.Status, f.ClusterUUID
+			last.CliqueId, last.FabricBandwidth = f.CliqueID, f.Bandwidth
+		}
 	}
+	r.Imex = &llmgwv1.IMEX{State: inv.IMEX.State, Channels: int32(inv.IMEX.Channels)}
 	for _, f := range fabrics {
 		r.Fabrics = append(r.Fabrics, &llmgwv1.FabricAttachment{
 			Fabric: f.Fabric, Hcas: f.HCAs, RateGbps: int32(f.RateGbps),
@@ -263,3 +275,12 @@ func int32Slice(v []int) []int32 {
 	}
 	return out
 }
+
+// fabricFrom is a GPU's NVLink fabric as the wire carries it, nil when the
+// driver reported none.
+func fabricFrom(state, status, cluster, clique, bandwidth string) *discovery.GPUFabric {
+	if state == "" {
+		return nil
+	}
+	return &discovery.GPUFabric{State: state, Status: status, ClusterUUID: cluster, CliqueID: clique, Bandwidth: bandwidth}
+}

internal/control/agentsvc/heartbeat_test.go

@@ -188,3 +188,79 @@ func TestALogRequestIsAnsweredByTheNodeItWasSentTo(t *testing.T) {
 		t.Error("a node with no stream answered")
 	}
 }
+
+func TestAHeartbeatCarriesHowBusyTheNodeIs(t *testing.T) {
+	s := newTestService()
+	nodeWith(s, inventory.Node{ID: "n1", State: inventory.Ready})
+
+	s.applyHeartbeat("n1", &llmgwv1.Heartbeat{NodeId: "n1", Load: &llmgwv1.NodeLoad{
+		CpuPercent: 62.5, CpuCount: 8, Load1: 3.2, MemoryUsedBytes: 6 << 30, MemoryAvailableBytes: 2 << 30,
+		Shards: []*llmgwv1.ShardLoad{{ShardId: "r1-0", CpuCores: 4.5, MemoryRssBytes: 5 << 30}},
+	}})
+	n, _ := s.inv.Get("n1")
+	if n.Load == nil || n.Load.CPUPercent != 62.5 || n.Load.MemUsedBytes != 6<<30 ||
+		n.Load.Shards["r1-0"].CPUCores != 4.5 {
+		t.Fatalf("load = %+v", n.Load)
+	}
+
+	// A heartbeat that measured nothing — the agent's first — leaves the last
+	// measurement standing rather than blanking it.
+	s.applyHeartbeat("n1", &llmgwv1.Heartbeat{NodeId: "n1"})
+	if n, _ := s.inv.Get("n1"); n.Load == nil {
+		t.Error("a heartbeat without load erased it")
+	}
+}
+
+func TestAPortGoingDownTakesItsFabricAway(t *testing.T) {
+	// Checked only at registration, a port that went down left the node on
+	// its fabric until the agent restarted, still offered multi-node work.
+	s := newTestService()
+	nodeWith(s, inventory.Node{ID: "n1", State: inventory.Ready,
+		Fabrics: []inventory.FabricAttachment{{Fabric: "ib-a", HCAs: []string{"mlx5_0"}, RateGbps: 400}}})
+
+	s.applyHeartbeat("n1", &llmgwv1.Heartbeat{NodeId: "n1", FabricsChanged: true,
+		IbPorts: []*llmgwv1.IBPort{{Hca: "mlx5_0", Port: 1, State: "DOWN", PhysState: "Polling",
+			Counters: map[string]uint64{"link_downed": 4}}}})
+	n, _ := s.inv.Get("n1")
+	if len(n.Fabrics) != 0 {
+		t.Errorf("the fabric should be gone: %+v", n.Fabrics)
+	}
+	if len(n.IBPorts) != 1 || n.IBPorts[0].Active() || n.IBPorts[0].Counters["link_downed"] != 4 {
+		t.Errorf("ports = %+v", n.IBPorts)
+	}
+
+	// Back up: the fabric returns.
+	s.applyHeartbeat("n1", &llmgwv1.Heartbeat{NodeId: "n1", FabricsChanged: true,
+		Fabrics: []*llmgwv1.FabricAttachment{{Fabric: "ib-a", Hcas: []string{"mlx5_0"}, RateGbps: 400}}})
+	if n, _ := s.inv.Get("n1"); !n.HasFabric("ib-a") {
+		t.Error("the fabric should be back")
+	}
+
+	// A heartbeat that does not say the fabrics changed leaves them alone.
+	s.applyHeartbeat("n1", &llmgwv1.Heartbeat{NodeId: "n1", IbPorts: []*llmgwv1.IBPort{{Hca: "mlx5_0", Port: 1, State: "ACTIVE"}}})
+	if n, _ := s.inv.Get("n1"); !n.HasFabric("ib-a") {
+		t.Error("a heartbeat without a fabric change emptied the fabrics")
+	}
+}
+
+func TestAGPUFallingOutOfItsDomainTakesTheNodeOut(t *testing.T) {
+	// A fabric manager restarting drops cards out of the domain; the node has
+	// to stop being offered work that spans it until they are back.
+	s := newTestService()
+	fab := &discovery.GPUFabric{State: "Completed", ClusterUUID: "76b1ad76-7c24", CliqueID: "1"}
+	nodeWith(s, inventory.Node{ID: "t1", State: inventory.Ready, GPUs: []discovery.GPU{
+		{Index: 0, Fabric: fab}, {Index: 1, Fabric: fab}}})
+	if n, _ := s.inv.Get("t1"); n.NVLinkDomain() == "" {
+		t.Fatal("the node should start in its domain")
+	}
+	s.applyHeartbeat("t1", &llmgwv1.Heartbeat{NodeId: "t1",
+		GpuFabric: []*llmgwv1.GPUFabricState{{Index: 1, State: "In Progress"}},
+		Imex:      &llmgwv1.IMEX{State: "active", Channels: 1}})
+	n, _ := s.inv.Get("t1")
+	if n.NVLinkDomain() != "" {
+		t.Errorf("a node with a card out of the domain is still in it")
+	}
+	if n.IMEX.State != "active" || n.IMEX.Channels != 1 {
+		t.Errorf("imex = %+v", n.IMEX)
+	}
+}

internal/control/agentsvc/installable_test.go

@@ -0,0 +1,27 @@
+package agentsvc
+
+import (
+	"slices"
+	"testing"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
+)
+
+func TestANodeIsAskedOnlyForEnginesItCanInstall(t *testing.T) {
+	// An agent older than the list says nothing, and could install llama.cpp
+	// alone. Asked for vLLM, it showed as failing until it was upgraded.
+	if got := installable(&llmgwv1.Register{}); !slices.Equal(got, []string{"llamacpp"}) {
+		t.Errorf("an agent that said nothing: %v", got)
+	}
+	// This agent says what it can, and that is vLLM and SGLang besides.
+	now := &llmgwv1.Register{InstallableEngines: provision.Installable()}
+	for _, e := range []string{"llamacpp", "vllm", "sglang"} {
+		if !slices.Contains(installable(now), e) {
+			t.Errorf("this agent does not offer %s: %v", e, installable(now))
+		}
+	}
+	if slices.Contains(installable(now), "uv") {
+		t.Error("uv, a tool for the others, is offered as an engine")
+	}
+}

internal/control/agentsvc/service.go

@@ -9,6 +9,7 @@ import (
 	"maps"
 	"net/http"
 	"slices"
+	"sort"
 	"sync"
 	"time"
 
@@ -58,6 +59,8 @@ type Service struct {
 	logMu    sync.Mutex
 	logWaits map[string]chan *llmgwv1.LogsReply
 	logSeq   uint64
+	// scanWaits are the fabric scans awaiting a reply, by request id.
+	scanWaits map[string]chan *llmgwv1.FabricScanReply
 }
 
 // SetEngines sets the engines every node installs, by name and version.
@@ -174,7 +177,11 @@ func (s *Service) Connect(
 	// month, and a changed version reaches every node on its next connect.
 	if s.engines != nil {
 		p := &llmgwv1.Provision{}
+		can := installable(reg)
 		for _, e := range slices.Sorted(maps.Keys(s.engines)) {
+			if !slices.Contains(can, e) {
+				continue
+			}
 			p.Engines = append(p.Engines, &llmgwv1.EngineWant{Engine: e, Version: s.engines[e]})
 		}
 		if err := stream.Send(&llmgwv1.ControlMessage{
@@ -282,6 +289,15 @@ func (s *Service) handle(nodeID string, msg *llmgwv1.AgentMessage) {
 				"last_output", m.Shard.GetLastOutput())
 		}
 
+	case *llmgwv1.AgentMessage_FabricScan:
+		s.logMu.Lock()
+		ch, ok := s.scanWaits[m.FabricScan.GetRequestId()]
+		delete(s.scanWaits, m.FabricScan.GetRequestId())
+		s.logMu.Unlock()
+		if ok {
+			ch <- m.FabricScan
+		}
+
 	case *llmgwv1.AgentMessage_Logs:
 		s.logMu.Lock()
 		ch, ok := s.logWaits[m.Logs.GetRequestId()]
@@ -309,7 +325,8 @@ func (s *Service) applyHeartbeat(nodeID string, hb *llmgwv1.Heartbeat) {
 	memMB := int(hb.GetMemoryTotalMb())
 	if len(used) == 0 && len(problems) == 0 && len(storage) == 0 &&
 		!hb.GetWeightsChanged() && memMB == 0 && !hb.GetEnginesChanged() &&
-		!hb.GetProvisionChanged() {
+		!hb.GetProvisionChanged() && hb.GetLoad() == nil && len(hb.GetIbPorts()) == 0 &&
+		!hb.GetFabricsChanged() && len(hb.GetGpuFabric()) == 0 && hb.GetImex() == nil {
 		return
 	}
 
@@ -400,9 +417,108 @@ func (s *Service) applyHeartbeat(nodeID string, hb *llmgwv1.Heartbeat) {
 			n.Provision = append(n.Provision, e)
 		}
 	}
+	if l := hb.GetLoad(); l != nil {
+		n.Load = loadFrom(l, time.Now())
+	}
+	if ports := hb.GetIbPorts(); len(ports) > 0 {
+		before := map[string]string{}
+		prev := map[string]inventory.IBPort{}
+		for _, p := range n.IBPorts {
+			before[fmt.Sprintf("%s/%d", p.HCA, p.Port)] = p.SMGUID
+			prev[fmt.Sprintf("%s/%d", p.HCA, p.Port)] = p
+		}
+		now := time.Now()
+		secs := now.Sub(n.IBAt).Seconds()
+		n.IBPorts = nil
+		for _, p := range ports {
+			// A different master than last time is a failover, or a port
+			// moved: the fabric's own event, kept for the page to show.
+			where := fmt.Sprintf("%s/%d", p.GetHca(), p.GetPort())
+			if old := before[where]; old != "" && p.GetSmGuid() != "" && old != p.GetSmGuid() {
+				s.log.Warn("a port sees a different master subnet manager", "node", nodeID,
+					"port", where, "from", old, "to", p.GetSmGuid())
+				s.inv.RecordSMChange(inventory.SMChange{At: time.Now(), Node: nodeID, Port: where,
+					From: old, To: p.GetSmGuid()})
+			}
+			var sms []inventory.SubnetManager
+			for _, r := range p.GetSubnetManagers() {
+				sms = append(sms, inventory.SubnetManager{GUID: r.GetGuid(), LID: int(r.GetLid()),
+					Priority: int(r.GetPriority()), State: r.GetState()})
+			}
+			n.IBPorts = append(n.IBPorts, inventory.IBPort{
+				SMs: sms, SMsError: p.GetSubnetManagersError(),
+				TxBps: rate(prev[where].Counters["port_xmit_data"], p.GetCounters()["port_xmit_data"], secs),
+				RxBps: rate(prev[where].Counters["port_rcv_data"], p.GetCounters()["port_rcv_data"], secs),
+				HCA:   p.GetHca(), Port: int(p.GetPort()), State: p.GetState(),
+				PhysState: p.GetPhysState(), RateGbps: int(p.GetRateGbps()),
+				LinkLayer: p.GetLinkLayer(), Counters: maps.Clone(p.GetCounters()),
+				NodeGUID: p.GetNodeGuid(), PortGUID: p.GetPortGuid(), GIDPrefix: p.GetGidPrefix(),
+				SMGUID: p.GetSmGuid(), SMLID: int(p.GetSmLid()), SMPriority: int(p.GetSmPriority()),
+				SMState: p.GetSmState(), SMError: p.GetSmError(), Network: p.GetNetwork(),
+			})
+		}
+	}
+	if len(hb.GetIbPorts()) > 0 {
+		n.IBAt = time.Now()
+	}
+	// NVLink domain membership, which a fabric manager restarting takes
+	// away from every card at once.
+	if fab := hb.GetGpuFabric(); len(fab) > 0 {
+		before := n.NVLinkDomain()
+		for _, f := range fab {
+			for i := range n.GPUs {
+				if n.GPUs[i].Index == int(f.GetIndex()) {
+					n.GPUs[i].Fabric = fabricFrom(f.GetState(), f.GetStatus(), f.GetClusterUuid(), f.GetCliqueId(), f.GetBandwidth())
+				}
+			}
+		}
+		if after := n.NVLinkDomain(); after != before {
+			s.log.Warn("a node's NVLink domain changed", "node", nodeID, "from", before, "to", after)
+		}
+	}
+	if m := hb.GetImex(); m != nil {
+		n.IMEX = discovery.IMEX{State: m.GetState(), Channels: int(m.GetChannels())}
+	}
+	// A port went down, or came back. Placement reads the fabrics, so this is
+	// what stops a node being offered multi-node work it can no longer carry.
+	if hb.GetFabricsChanged() {
+		var fabs []inventory.FabricAttachment
+		for _, f := range hb.GetFabrics() {
+			fabs = append(fabs, inventory.FabricAttachment{
+				Fabric: f.GetFabric(), HCAs: slices.Clone(f.GetHcas()), RateGbps: int(f.GetRateGbps()),
+			})
+		}
+		s.log.Warn("a node's fabrics changed", "node", nodeID,
+			"from", n.FabricNames(), "to", fabricNames(fabs))
+		n.Fabrics = fabs
+	}
 	s.inv.Upsert(n)
 }
 
+func fabricNames(fs []inventory.FabricAttachment) []string {
+	out := make([]string, len(fs))
+	for i, f := range fs {
+		out[i] = f.Fabric
+	}
+	return out
+}
+
+// loadFrom is a heartbeat's load as the inventory keeps it.
+func loadFrom(l *llmgwv1.NodeLoad, at time.Time) *inventory.Load {
+	out := &inventory.Load{
+		At: at, CPUPercent: l.GetCpuPercent(), CPUCount: int(l.GetCpuCount()),
+		Load1: l.GetLoad1(), Load5: l.GetLoad5(), Load15: l.GetLoad15(),
+		MemUsedBytes: l.GetMemoryUsedBytes(), MemAvailBytes: l.GetMemoryAvailableBytes(),
+		Shards: map[string]inventory.ShardLoad{},
+	}
+	for _, sh := range l.GetShards() {
+		out.Shards[sh.GetShardId()] = inventory.ShardLoad{
+			CPUCores: sh.GetCpuCores(), RSSBytes: sh.GetMemoryRssBytes(),
+		}
+	}
+	return out
+}
+
 func (s *Service) attach(ctx context.Context, nodeID string) *conn {
 	c := &conn{
 		nodeID: nodeID,
@@ -569,3 +685,113 @@ func (s *Service) Logs(ctx context.Context, nodeID, source string, lines int) ([
 		return nil, nil, fmt.Errorf("node %s did not answer; an agent older than v0.8.4 cannot", nodeID)
 	}
 }
+
+// ScanFabric asks a node to map the fabric one of its ports is on.
+func (s *Service) ScanFabric(ctx context.Context, nodeID, hca string, port int) (*llmgwv1.FabricScanReply, error) {
+	s.logMu.Lock()
+	if s.scanWaits == nil {
+		s.scanWaits = map[string]chan *llmgwv1.FabricScanReply{}
+	}
+	s.logSeq++
+	id := fmt.Sprintf("scan-%d", s.logSeq)
+	ch := make(chan *llmgwv1.FabricScanReply, 1)
+	s.scanWaits[id] = ch
+	s.logMu.Unlock()
+	defer func() {
+		s.logMu.Lock()
+		delete(s.scanWaits, id)
+		s.logMu.Unlock()
+	}()
+	if err := s.send(nodeID, &llmgwv1.ControlMessage{Msg: &llmgwv1.ControlMessage_FabricScan{
+		FabricScan: &llmgwv1.FabricScanRequest{RequestId: id, Hca: hca, Port: int32(port)}, //nolint:gosec // a port number
+	}}); err != nil {
+		return nil, err
+	}
+	ctx, cancel := context.WithTimeout(ctx, 3*time.Minute)
+	defer cancel()
+	select {
+	case r := <-ch:
+		return r, nil
+	case <-ctx.Done():
+		return nil, fmt.Errorf("node %s did not answer the fabric scan", nodeID)
+	}
+}
+
+// ScanFabrics maps every physical InfiniBand fabric every so often, asking
+// one node on each: the first, by name, whose port on it is up. A fabric
+// looks the same from all its ports, so one is enough, and the rest of its
+// nodes are left to their work.
+func (s *Service) ScanFabrics(ctx context.Context, every time.Duration) {
+	if every <= 0 {
+		every = 5 * time.Minute
+	}
+	// Soon after a start, once agents have sent a heartbeat or two.
+	timer := time.NewTimer(30 * time.Second)
+	defer timer.Stop()
+	for {
+		select {
+		case <-ctx.Done():
+			return
+		case <-timer.C:
+		}
+		s.scanOnce(ctx)
+		timer.Reset(every)
+	}
+}
+
+func (s *Service) scanOnce(ctx context.Context) {
+	nodes := s.inv.Nodes()
+	sort.Slice(nodes, func(i, j int) bool { return nodes[i].ID < nodes[j].ID })
+	canon := inventory.CanonicalSMs(nodes)
+	chosen := map[string]bool{}
+	for _, n := range nodes {
+		if n.State != inventory.Ready {
+			continue
+		}
+		for _, p := range n.IBPorts {
+			key := p.FabricKeyIn(canon)
+			if key == "" || p.RoCE() || chosen[key] {
+				continue
+			}
+			chosen[key] = true
+			where := fmt.Sprintf("%s/%d", p.HCA, p.Port)
+			r, err := s.ScanFabric(ctx, n.ID, p.HCA, p.Port)
+			sc := inventory.FabricScan{Key: key, At: time.Now(), Node: n.ID, Port: where}
+			switch {
+			case err != nil:
+				sc.Err = err.Error()
+			case r.GetError() != "":
+				sc.Err = r.GetError()
+			default:
+				sc.Output = r.GetOutput()
+			}
+			if r != nil {
+				sc.Took = time.Duration(r.GetTookMs()) * time.Millisecond
+			}
+			if sc.Err != "" {
+				s.log.Warn("could not map a fabric", "fabric", key, "node", n.ID, "port", where, "err", sc.Err)
+			}
+			s.inv.SetScan(sc)
+		}
+	}
+}
+
+// rate is a counter's increase a second, zero when there is no earlier
+// reading or the counter went backwards — an adapter reset starts it again.
+func rate(before, now uint64, secs float64) float64 {
+	if before == 0 || now < before || secs <= 0 || secs > 120 {
+		return 0
+	}
+	return float64(now-before) / secs
+}
+
+// installable are the engines a node can be asked to install. An agent older
+// than the list said nothing, and could install llama.cpp alone: asked for
+// vLLM too, it reported it unsupported, and a cluster half way through an
+// upgrade showed every such node as failing.
+func installable(reg *llmgwv1.Register) []string {
+	if can := reg.GetInstallableEngines(); len(can) > 0 {
+		return can
+	}
+	return []string{"llamacpp"}
+}

internal/control/fabrics/fabrics.go

@@ -0,0 +1,379 @@
+// Package fabrics works out which physical fabrics the cluster's adapters
+// are on, and where that disagrees with what the nodes were told.
+//
+// A fabric's name is configuration. What it is physically is what the
+// adapters report: on InfiniBand the master subnet manager every port on one
+// fabric sees, on RoCE the IP network. Kept apart so a site with several
+// independent fabrics can see each, and see at once when two were given one
+// name, one was given two, or an adapter is cabled somewhere nobody declared.
+package fabrics
+
+import (
+	"fmt"
+	"slices"
+	"sort"
+	"strings"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+)
+
+// Level is how much a problem matters.
+type Level string
+
+// The levels.
+const (
+	Fail Level = "fail"
+	Warn Level = "warn"
+	Info Level = "info"
+)
+
+// Problem is one disagreement, in words.
+type Problem struct {
+	Level   Level  `json:"level"`
+	Fabric  string `json:"fabric,omitempty"`
+	Message string `json:"message"`
+}
+
+// Physical is one fabric as the adapters find it.
+type Physical struct {
+	Key string `json:"key"`
+	// Kind is infiniband or roce.
+	Kind string `json:"kind"`
+	// Label is the key as a person reads it.
+	Label string `json:"label"`
+	// SM is the master subnet manager, on InfiniBand: the one most of its
+	// ports see.
+	SM *SM `json:"sm,omitempty"`
+	// SMs is every subnet manager the fabric lists, master first; empty when
+	// no port could ask. Standbys counts those that could take over.
+	SMs      []SM `json:"sms,omitempty"`
+	Standbys int  `json:"standbys"`
+	// Masters is how many ports see each master, when they do not all agree.
+	Masters map[string]int `json:"masters,omitempty"`
+	// Changes are the master changes seen in the last day, newest first.
+	Changes []Change `json:"changes,omitempty"`
+	// Declared are the names nodes gave this fabric; one, when all is well.
+	Declared  []string `json:"declared"`
+	Members   []Member `json:"members"`
+	RateGbps  int      `json:"rate_gbps"`
+	GIDPrefix string   `json:"gid_prefix,omitempty"`
+}
+
+// SM is a subnet manager as the ports see it.
+type SM struct {
+	GUID     string `json:"guid"`
+	LID      int    `json:"lid"`
+	Priority int    `json:"priority"`
+	State    string `json:"state"`
+}
+
+// Change is a fabric's master subnet manager changing, as its ports saw it.
+type Change struct {
+	At    time.Time `json:"at"`
+	From  string    `json:"from"`
+	To    string    `json:"to"`
+	Ports int       `json:"ports"`
+}
+
+// Member is one node's adapters on a physical fabric.
+type Member struct {
+	Node string   `json:"node"`
+	HCAs []string `json:"hcas"`
+	// Declared is the name this node gives these adapters, empty when it
+	// declares them in no fabric.
+	Declared string `json:"declared,omitempty"`
+}
+
+// Report is the whole picture.
+type Report struct {
+	Physical []Physical `json:"physical"`
+	Problems []Problem  `json:"problems"`
+	// Unverified are declared fabrics some of whose nodes cannot say which
+	// physical fabric they are on, with why.
+	Unverified map[string][]string `json:"unverified,omitempty"`
+}
+
+// Analyze reads every node's ports, and the changes of master subnet manager
+// the control plane has recorded.
+func Analyze(nodes []inventory.Node, changes []inventory.SMChange, now time.Time) Report {
+	r := Report{Unverified: map[string][]string{}}
+	canon := inventory.CanonicalSMs(nodes)
+	masters := map[string]map[string]int{} // key -> master GUID -> ports
+	smLists := map[string]map[string]inventory.SubnetManager{}
+	listErr := map[string]string{}
+	noSM := map[string][]string{} // declared -> node/port up with no SM
+	byKey := map[string]*Physical{}
+	declaredKeys := map[string]map[string][]string{} // declared -> key -> nodes
+
+	for _, n := range nodes {
+		declOf := map[string]string{} // hca -> declared fabric
+		for _, f := range n.Fabrics {
+			for _, h := range f.HCAs {
+				declOf[h] = f.Fabric
+			}
+		}
+		// Which physical fabric each declared fabric's adapters are on here.
+		seenDecl := map[string]map[string]bool{}
+		members := map[string]*Member{} // key -> this node's membership
+
+		for _, p := range n.IBPorts {
+			key := p.FabricKeyIn(canon)
+			decl := declOf[p.HCA]
+			if decl != "" && !p.RoCE() && strings.EqualFold(p.State, "INIT") && strings.EqualFold(p.PhysState, "LinkUp") {
+				noSM[decl] = appendOnce(noSM[decl], fmt.Sprintf("%s %s/%d", n.ID, p.HCA, p.Port))
+			}
+			if key != "" && !p.RoCE() {
+				if masters[key] == nil {
+					masters[key] = map[string]int{}
+					smLists[key] = map[string]inventory.SubnetManager{}
+				}
+				masters[key][p.SMGUID]++
+				for _, sm := range p.SMs {
+					smLists[key][sm.GUID] = sm
+				}
+				if p.SMsError != "" {
+					listErr[key] = p.SMsError
+				}
+			}
+			if key == "" {
+				if p.Active() && decl != "" && !p.RoCE() && p.SMError != "" {
+					r.Unverified[decl] = appendOnce(r.Unverified[decl], n.ID+": "+p.SMError)
+				}
+				continue
+			}
+			ph := byKey[key]
+			if ph == nil {
+				ph = &Physical{Key: key, Kind: "infiniband", GIDPrefix: p.GIDPrefix}
+				if p.RoCE() {
+					ph.Kind, ph.Label = "roce", "RoCE "+p.Network
+				} else {
+					ph.SM = &SM{GUID: p.SMGUID, LID: p.SMLID, Priority: p.SMPriority, State: p.SMState}
+					ph.Label = "fabric of subnet manager " + strings.TrimPrefix(key, "sm:")
+				}
+				byKey[key] = ph
+			}
+			if p.RateGbps > 0 && (ph.RateGbps == 0 || p.RateGbps < ph.RateGbps) {
+				ph.RateGbps = p.RateGbps
+			}
+			m := members[key]
+			if m == nil {
+				m = &Member{Node: n.ID, Declared: decl}
+				members[key] = m
+			}
+			m.HCAs = appendOnce(m.HCAs, p.HCA)
+			if decl != "" {
+				if seenDecl[decl] == nil {
+					seenDecl[decl] = map[string]bool{}
+				}
+				seenDecl[decl][key] = true
+				ph.Declared = appendOnce(ph.Declared, decl)
+				if declaredKeys[decl] == nil {
+					declaredKeys[decl] = map[string][]string{}
+				}
+				declaredKeys[decl][key] = appendOnce(declaredKeys[decl][key], n.ID)
+			} else {
+				r.Problems = append(r.Problems, Problem{Level: Warn, Message: fmt.Sprintf(
+					"%s: %s is up on %s but declared in no fabric, so llmgw sends nothing over it",
+					n.ID, p.HCA, ph.Label)})
+			}
+		}
+		for decl, keys := range seenDecl {
+			if len(keys) > 1 {
+				r.Problems = append(r.Problems, Problem{Level: Fail, Fabric: decl, Message: fmt.Sprintf(
+					"%s: the adapters it declares as %s are cabled to %d different fabrics; it is left out of multi-node work on %s",
+					n.ID, decl, len(keys), decl)})
+			}
+		}
+		for key, m := range members {
+			byKey[key].Members = append(byKey[key].Members, *m)
+		}
+	}
+
+	for decl, keys := range declaredKeys {
+		if len(keys) < 2 {
+			continue
+		}
+		var parts []string
+		for _, k := range sortedKeys(keys) {
+			parts = append(parts, fmt.Sprintf("%s: %s", byKey[k].Label, strings.Join(keys[k], ", ")))
+		}
+		r.Problems = append(r.Problems, Problem{Level: Fail, Fabric: decl, Message: fmt.Sprintf(
+			"%s is %d separate fabrics — %s. A replica is kept to one of them; give each its own name",
+			decl, len(keys), strings.Join(parts, "; "))})
+	}
+	for key, ph := range byKey {
+		if ph.Kind == "infiniband" {
+			analyzeSMs(&r, ph, masters[key], smLists[key], listErr[key], nodes, changes, now)
+		}
+		sort.Strings(ph.Declared)
+		sort.Slice(ph.Members, func(i, j int) bool { return ph.Members[i].Node < ph.Members[j].Node })
+		if len(ph.Declared) > 1 {
+			r.Problems = append(r.Problems, Problem{Level: Warn, Fabric: ph.Declared[0], Message: fmt.Sprintf(
+				"%s are one fabric (%s): a replica could use all of them, but each name is placed on alone",
+				strings.Join(ph.Declared, " and "), ph.Label)})
+		}
+		r.Physical = append(r.Physical, *ph)
+	}
+	sort.Slice(r.Physical, func(i, j int) bool {
+		a, b := r.Physical[i], r.Physical[j]
+		if na, nb := first(a.Declared), first(b.Declared); na != nb {
+			return na < nb
+		}
+		return a.Key < b.Key
+	})
+
+	// Separate InfiniBand fabrics on one subnet prefix: harmless while they
+	// stay apart, and the reason their GIDs cannot tell them apart.
+	byPrefix := map[string][]string{}
+	for _, ph := range r.Physical {
+		if ph.Kind == "infiniband" && ph.GIDPrefix != "" {
+			byPrefix[ph.GIDPrefix] = append(byPrefix[ph.GIDPrefix], ph.Label)
+		}
+	}
+	for prefix, labels := range byPrefix {
+		if len(labels) > 1 {
+			r.Problems = append(r.Problems, Problem{Level: Info, Message: fmt.Sprintf(
+				"%d separate fabrics share the subnet prefix %s; that is the default, and fine while they stay apart — a distinct subnet_prefix per subnet manager makes their GIDs unique",
+				len(labels), prefix)})
+		}
+	}
+	for decl, ports := range noSM {
+		r.Problems = append(r.Problems, Problem{Level: Fail, Fabric: decl, Message: fmt.Sprintf(
+			"%s: %d port(s) have a link but no subnet manager has configured them (INIT): %s",
+			decl, len(ports), strings.Join(ports, ", "))})
+	}
+	for decl, why := range r.Unverified {
+		r.Problems = append(r.Problems, Problem{Level: Info, Fabric: decl, Message: fmt.Sprintf(
+			"%s cannot be checked on every node: %s", decl, strings.Join(why, "; "))})
+	}
+
+	sort.SliceStable(r.Problems, func(i, j int) bool {
+		if li, lj := rank(r.Problems[i].Level), rank(r.Problems[j].Level); li != lj {
+			return li < lj
+		}
+		return r.Problems[i].Message < r.Problems[j].Message
+	})
+	return r
+}
+
+// analyzeSMs fills in a fabric's subnet managers and what is wrong with them.
+func analyzeSMs(r *Report, ph *Physical, masters map[string]int, list map[string]inventory.SubnetManager,
+	listErr string, nodes []inventory.Node, changes []inventory.SMChange, now time.Time,
+) {
+	name := first(ph.Declared)
+	if name == "~" {
+		name = ph.Label
+	}
+	// The master is the one most ports see.
+	best, most := "", -1
+	for g, n := range masters {
+		if n > most || (n == most && g < best) {
+			best, most = g, n
+		}
+	}
+	if best != "" && ph.SM != nil && ph.SM.GUID != best {
+		ph.SM = &SM{GUID: best}
+	}
+	if len(masters) > 1 {
+		ph.Masters = masters
+		var parts []string
+		for _, g := range sortedKeys(masters) {
+			parts = append(parts, fmt.Sprintf("%s by %d", g, masters[g]))
+		}
+		r.Problems = append(r.Problems, Problem{Level: Warn, Fabric: name, Message: fmt.Sprintf(
+			"%s: its ports do not agree on the master subnet manager (%s) — a failover in progress, or a fabric split in two",
+			name, strings.Join(parts, ", "))})
+	}
+
+	for _, sm := range list {
+		v := SM{GUID: sm.GUID, LID: sm.LID, Priority: sm.Priority, State: sm.State}
+		ph.SMs = append(ph.SMs, v)
+		if sm.State == "standby" {
+			ph.Standbys++
+		}
+		if ph.SM != nil && sm.GUID == ph.SM.GUID {
+			ph.SM.LID, ph.SM.Priority, ph.SM.State = sm.LID, sm.Priority, sm.State
+		}
+	}
+	sort.Slice(ph.SMs, func(i, j int) bool {
+		if (ph.SMs[i].State == "master") != (ph.SMs[j].State == "master") {
+			return ph.SMs[i].State == "master"
+		}
+		return ph.SMs[i].Priority > ph.SMs[j].Priority
+	})
+	switch {
+	case len(list) == 0 && listErr != "":
+		r.Problems = append(r.Problems, Problem{Level: Info, Fabric: name, Message: fmt.Sprintf(
+			"%s: its standby subnet managers cannot be listed: %s", name, listErr)})
+	case len(list) > 0 && ph.Standbys == 0:
+		r.Problems = append(r.Problems, Problem{Level: Warn, Fabric: name, Message: fmt.Sprintf(
+			"%s has one subnet manager and no standby: if it stops, the fabric keeps its routes but cannot recover from any change — a cable, a switch, a node rebooting",
+			name)})
+	}
+
+	// Its recent failovers: the recorded changes of ports on this fabric,
+	// one per minute of changes rather than one per port.
+	onFabric := map[string]bool{}
+	for _, m := range ph.Members {
+		for _, h := range m.HCAs {
+			onFabric[m.Node+" "+h] = true
+		}
+	}
+	day := now.Add(-24 * time.Hour)
+	for _, c := range changes {
+		hca := c.Port
+		if i := strings.LastIndexByte(hca, '/'); i >= 0 {
+			hca = hca[:i]
+		}
+		if c.At.Before(day) || !onFabric[c.Node+" "+hca] {
+			continue
+		}
+		if n := len(ph.Changes); n > 0 && ph.Changes[n-1].To == c.To && c.At.Sub(ph.Changes[n-1].At) < 2*time.Minute {
+			ph.Changes[n-1].Ports++
+			continue
+		}
+		ph.Changes = append(ph.Changes, Change{At: c.At, From: c.From, To: c.To, Ports: 1})
+	}
+	slices.Reverse(ph.Changes)
+	if len(ph.Changes) > 0 && now.Sub(ph.Changes[0].At) < time.Hour {
+		c := ph.Changes[0]
+		r.Problems = append(r.Problems, Problem{Level: Warn, Fabric: name, Message: fmt.Sprintf(
+			"%s: the master subnet manager changed %s ago, from %s to %s",
+			name, now.Sub(c.At).Round(time.Minute), c.From, c.To)})
+	}
+	_ = nodes
+}
+
+func rank(l Level) int {
+	switch l {
+	case Fail:
+		return 0
+	case Warn:
+		return 1
+	}
+	return 2
+}
+
+func first(s []string) string {
+	if len(s) == 0 {
+		return "~" // undeclared last
+	}
+	return s[0]
+}
+
+func appendOnce(s []string, v string) []string {
+	if slices.Contains(s, v) {
+		return s
+	}
+	return append(s, v)
+}
+
+func sortedKeys[V any](m map[string]V) []string {
+	out := make([]string, 0, len(m))
+	for k := range m {
+		out = append(out, k)
+	}
+	sort.Strings(out)
+	return out
+}

internal/control/fabrics/fabrics_test.go

@@ -0,0 +1,150 @@
+package fabrics
+
+import (
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+)
+
+func ibNode(id, declared, sm string, hcas ...string) inventory.Node {
+	n := inventory.Node{ID: id}
+	if declared != "" {
+		n.Fabrics = []inventory.FabricAttachment{{Fabric: declared, HCAs: hcas}}
+	}
+	for _, h := range hcas {
+		n.IBPorts = append(n.IBPorts, inventory.IBPort{HCA: h, Port: 1, State: "ACTIVE", RateGbps: 400,
+			LinkLayer: "InfiniBand", SMGUID: sm, SMLID: 1, GIDPrefix: "fe80:0000:0000:0000"})
+	}
+	return n
+}
+
+func problem(t *testing.T, r Report, l Level, fragment string) {
+	t.Helper()
+	for _, p := range r.Problems {
+		if p.Level == l && strings.Contains(p.Message, fragment) {
+			return
+		}
+	}
+	t.Errorf("no %s problem containing %q in %+v", l, fragment, r.Problems)
+}
+
+func TestIndependentFabricsWithTheirOwnNamesAreFine(t *testing.T) {
+	r := analyze([]inventory.Node{
+		ibNode("a1", "ib-a", "0x01", "mlx5_0"), ibNode("a2", "ib-a", "0x01", "mlx5_0"),
+		ibNode("b1", "ib-b", "0x02", "mlx5_0"),
+	})
+	if len(r.Physical) != 2 {
+		t.Fatalf("physical = %+v", r.Physical)
+	}
+	if r.Physical[0].Declared[0] != "ib-a" || len(r.Physical[0].Members) != 2 || r.Physical[0].SM.GUID != "0x01" {
+		t.Errorf("ib-a = %+v", r.Physical[0])
+	}
+	for _, p := range r.Problems {
+		if p.Level != Info {
+			t.Errorf("unexpected problem: %+v", p)
+		}
+	}
+	// Both on the default prefix: said, quietly.
+	problem(t, r, Info, "share the subnet prefix")
+}
+
+func TestOneNameOnTwoFabricsIsAFailure(t *testing.T) {
+	r := analyze([]inventory.Node{
+		ibNode("a1", "ib-a", "0x01", "mlx5_0"), ibNode("b1", "ib-a", "0x02", "mlx5_0"),
+	})
+	problem(t, r, Fail, "ib-a is 2 separate fabrics")
+}
+
+func TestTwoNamesOnOneFabricAreSaid(t *testing.T) {
+	r := analyze([]inventory.Node{
+		ibNode("a1", "ib-a", "0x01", "mlx5_0"), ibNode("b1", "ib-b", "0x01", "mlx5_0"),
+	})
+	problem(t, r, Warn, "ib-a and ib-b are one fabric")
+}
+
+func TestAnAdapterUpButUndeclaredIsSaid(t *testing.T) {
+	n := ibNode("a1", "ib-a", "0x01", "mlx5_0")
+	n.IBPorts = append(n.IBPorts, inventory.IBPort{HCA: "mlx5_3", Port: 1, State: "ACTIVE", LinkLayer: "InfiniBand", SMGUID: "0x09"})
+	problem(t, analyze([]inventory.Node{n}), Warn, "mlx5_3 is up on fabric of subnet manager 0x09")
+}
+
+func TestOneNodesAdaptersOnTwoFabricsAreAFailure(t *testing.T) {
+	n := ibNode("x", "ib-a", "0x01", "mlx5_0", "mlx5_1")
+	n.IBPorts[1].SMGUID = "0x02"
+	problem(t, analyze([]inventory.Node{n}), Fail, "cabled to 2 different fabrics")
+}
+
+func TestANodeThatCannotSayIsReported(t *testing.T) {
+	n := ibNode("x", "ib-a", "", "mlx5_0")
+	n.IBPorts[0].SMError = "sminfo not found: install infiniband-diags"
+	problem(t, analyze([]inventory.Node{n}), Info, "install infiniband-diags")
+}
+
+func TestRoCEIsItsNetwork(t *testing.T) {
+	n := inventory.Node{ID: "r1", Fabrics: []inventory.FabricAttachment{{Fabric: "roce-a", HCAs: []string{"mlx5_0"}}},
+		IBPorts: []inventory.IBPort{{HCA: "mlx5_0", Port: 1, State: "ACTIVE", LinkLayer: "Ethernet", Network: "10.8.0.0/24"}}}
+	r := analyze([]inventory.Node{n})
+	if len(r.Physical) != 1 || r.Physical[0].Kind != "roce" || r.Physical[0].Label != "RoCE 10.8.0.0/24" {
+		t.Errorf("physical = %+v", r.Physical)
+	}
+}
+
+func analyze(nodes []inventory.Node) Report { return Analyze(nodes, nil, time.Now()) }
+
+func withSMs(n inventory.Node, sms ...inventory.SubnetManager) inventory.Node {
+	n.IBPorts[0].SMs = sms
+	return n
+}
+
+func TestAFailoverKeepsAFabricOneFabric(t *testing.T) {
+	// Mid-failover some ports see the old master, some the new. Keyed by the
+	// master alone, one fabric read as two for a minute — and a declared name
+	// covering it as "two separate fabrics". The list of managers is the same
+	// before and after, which is what keeps it one.
+	sms := []inventory.SubnetManager{{GUID: "0x01", State: "master", Priority: 15}, {GUID: "0x02", State: "standby", Priority: 14}}
+	r := analyze([]inventory.Node{
+		withSMs(ibNode("a1", "ib-a", "0x01", "mlx5_0"), sms...),
+		ibNode("a2", "ib-a", "0x02", "mlx5_0"),
+	})
+	if len(r.Physical) != 1 {
+		t.Fatalf("one fabric read as %d: %+v", len(r.Physical), r.Physical)
+	}
+	for _, p := range r.Problems {
+		if p.Level == Fail {
+			t.Errorf("a failover was reported as a failure: %s", p.Message)
+		}
+	}
+	problem(t, r, Warn, "do not agree on the master subnet manager")
+	if r.Physical[0].Standbys != 1 || len(r.Physical[0].SMs) != 2 || r.Physical[0].SMs[0].State != "master" {
+		t.Errorf("managers = %+v", r.Physical[0].SMs)
+	}
+}
+
+func TestAFabricWithNoStandbyIsSaid(t *testing.T) {
+	r := analyze([]inventory.Node{withSMs(ibNode("a1", "ib-a", "0x01", "mlx5_0"),
+		inventory.SubnetManager{GUID: "0x01", State: "master"})})
+	problem(t, r, Warn, "one subnet manager and no standby")
+}
+
+func TestPortsUpWithoutASubnetManagerAreAFailure(t *testing.T) {
+	n := ibNode("a1", "ib-a", "", "mlx5_0")
+	n.IBPorts[0].State, n.IBPorts[0].PhysState = "INIT", "LinkUp"
+	problem(t, analyze([]inventory.Node{n}), Fail, "no subnet manager has configured them")
+}
+
+func TestARecentFailoverIsShown(t *testing.T) {
+	now := time.Now()
+	nodes := []inventory.Node{withSMs(ibNode("a1", "ib-a", "0x02", "mlx5_0"),
+		inventory.SubnetManager{GUID: "0x01", State: "standby"}, inventory.SubnetManager{GUID: "0x02", State: "master"})}
+	changes := []inventory.SMChange{
+		{At: now.Add(-20 * time.Minute), Node: "a1", Port: "mlx5_0/1", From: "0x01", To: "0x02"},
+		{At: now.Add(-48 * time.Hour), Node: "a1", Port: "mlx5_0/1", From: "0x02", To: "0x01"},
+	}
+	r := Analyze(nodes, changes, now)
+	if got := r.Physical[0].Changes; len(got) != 1 || got[0].To != "0x02" {
+		t.Errorf("changes in the last day = %+v", got)
+	}
+	problem(t, r, Warn, "the master subnet manager changed 20m0s ago")
+}

internal/control/fleetmetrics/fleetmetrics.go

@@ -16,10 +16,12 @@ package fleetmetrics
 
 import (
 	"strconv"
+	"strings"
 	"time"
 
 	"github.com/prometheus/client_golang/prometheus"
 
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/fabrics"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
 )
@@ -59,8 +61,32 @@ type Collector struct {
 	replicaState *prometheus.Desc
 	replicaIdle  *prometheus.Desc
 	replicaReady *prometheus.Desc
+
+	nodeCPU      *prometheus.Desc
+	nodeCPUs     *prometheus.Desc
+	nodeLoad     *prometheus.Desc
+	nodeMemUsed  *prometheus.Desc
+	nodeMemAvail *prometheus.Desc
+	shardCPU     *prometheus.Desc
+	shardMemory  *prometheus.Desc
+
+	ibActive  *prometheus.Desc
+	ibRate    *prometheus.Desc
+	ibCounter *prometheus.Desc
+
+	gpuNVLJoined *prometheus.Desc
+	imexUp       *prometheus.Desc
+
+	fabricSMs      *prometheus.Desc
+	fabricMaster   *prometheus.Desc
+	fabricProblems *prometheus.Desc
 }
 
+// loadFresh is how old a node's load may be and still be reported. A node
+// that stopped sending it has an unknown load, and repeating its last figure
+// would draw a busy machine long after it went away.
+const loadFresh = time.Minute
+
 // New builds the collector.
 func New(src Source) *Collector {
 	node := []string{"node"}
@@ -115,6 +141,52 @@ func New(src Source) *Collector {
 		replicaIdle: prometheus.NewDesc("llmgw_replica_idle_seconds",
 			"Seconds since a replica was last sent a request. Absent when nothing has been reported, "+
 				"which is not the same as idle.", []string{"model", "replica"}, nil),
+
+		// A node's own use, from its heartbeat. Absent when the agent has
+		// not measured it lately, which is not the same as idle.
+		nodeCPU: prometheus.NewDesc("llmgw_node_cpu_utilization_percent",
+			"How busy a node's CPUs were between its last two heartbeats, 0 to 100 across all cores.", node, nil),
+		nodeCPUs: prometheus.NewDesc("llmgw_node_cpus",
+			"CPUs a node has, so a load average can be read against it.", node, nil),
+		nodeLoad: prometheus.NewDesc("llmgw_node_load",
+			"The kernel's load average over 1, 5 and 15 minutes.", append(node, "window"), nil),
+		nodeMemUsed: prometheus.NewDesc("llmgw_node_memory_used_bytes",
+			"RAM in use: total less what the kernel could hand to a new process.", node, nil),
+		nodeMemAvail: prometheus.NewDesc("llmgw_node_memory_available_bytes",
+			"RAM the kernel could hand to a new process without swapping.", node, nil),
+		shardCPU: prometheus.NewDesc("llmgw_shard_cpu_cores",
+			"Cores an engine process group kept busy, the way top counts: 2.5 is two and a half.",
+			[]string{"node", "model", "replica", "shard"}, nil),
+		shardMemory: prometheus.NewDesc("llmgw_shard_memory_bytes",
+			"Resident memory of an engine process group.",
+			[]string{"node", "model", "replica", "shard"}, nil),
+
+		// Every InfiniBand or RoCE port, re-read on each heartbeat.
+		ibActive: prometheus.NewDesc("llmgw_ib_port_active",
+			"1 when an InfiniBand or RoCE port can carry traffic. phys_state tells a cable "+
+				"unplugged (Polling, Disabled) from a link with no subnet manager (LinkUp).",
+			[]string{"node", "hca", "port", "link_layer", "state", "phys_state"}, nil),
+		ibRate: prometheus.NewDesc("llmgw_ib_port_rate_gbps",
+			"A port's negotiated rate. A link that trained below its usual speed is a "+
+				"cable or transceiver to look at.", []string{"node", "hca", "port"}, nil),
+		gpuNVLJoined: prometheus.NewDesc("llmgw_gpu_nvlink_joined",
+			"1 when a GPU is joined to its multi-node NVLink domain, 0 when it reports a fabric it has not "+
+				"joined. Absent on cards with no multi-node fabric.", []string{"node", "gpu", "domain", "state"}, nil),
+		imexUp: prometheus.NewDesc("llmgw_imex_up",
+			"1 when nvidia-imex is active with a channel, on nodes whose GPUs report a fabric.", []string{"node"}, nil),
+		fabricSMs: prometheus.NewDesc("llmgw_fabric_subnet_managers",
+			"Subnet managers a physical fabric lists, by state. No standby means the fabric "+
+				"cannot recover from a change if its master stops.", []string{"fabric", "physical", "state"}, nil),
+		fabricMaster: prometheus.NewDesc("llmgw_fabric_master_sm",
+			"1 for a physical fabric's master subnet manager, as most of its ports see it; a "+
+				"change of sm_guid is a failover.", []string{"fabric", "physical", "sm_guid"}, nil),
+		fabricProblems: prometheus.NewDesc("llmgw_fabric_problems",
+			"Disagreements about the fabrics, by level: fail means multi-node work will not "+
+				"place or will not run where it should.", []string{"level"}, nil),
+		ibCounter: prometheus.NewDesc("llmgw_ib_port_counter_total",
+			"A port's cumulative counters, by sysfs name: symbol_error, link_downed and the "+
+				"other errors say a link is going bad before a job fails on it; port_xmit_data "+
+				"and port_rcv_data are bytes.", []string{"node", "hca", "port", "counter"}, nil),
 	}
 }
 
@@ -123,6 +195,9 @@ func (c *Collector) Describe(ch chan<- *prometheus.Desc) {
 	for _, d := range []*prometheus.Desc{
 		c.nodeUp, c.nodeGPUs, c.nodeGPUsFree, c.nodeMemory, c.nodeSeen,
 		c.nodeDiskFree, c.nodeDiskSize,
+		c.nodeCPU, c.nodeCPUs, c.nodeLoad, c.nodeMemUsed, c.nodeMemAvail, c.shardCPU, c.shardMemory,
+		c.ibActive, c.ibRate, c.ibCounter, c.fabricSMs, c.fabricMaster, c.fabricProblems,
+		c.gpuNVLJoined, c.imexUp,
 		c.gpuMemoryTotal, c.gpuMemoryUsed, c.gpuUtil, c.gpuTemp, c.gpuHealthy,
 		c.gpuNVLink, c.gpuECC,
 		c.replicaState, c.replicaReady, c.replicaIdle,
@@ -135,6 +210,7 @@ func (c *Collector) Describe(ch chan<- *prometheus.Desc) {
 func (c *Collector) Collect(ch chan<- prometheus.Metric) {
 	now := time.Now()
 
+	loads := map[string]*inventory.Load{}
 	for _, n := range c.src.Nodes() {
 		fabric := ""
 		if len(n.Fabrics) > 0 {
@@ -165,6 +241,59 @@ func (c *Collector) Collect(ch chan<- prometheus.Metric) {
 				float64(total), n.ID)
 		}
 
+		if l := n.Load; l != nil && now.Sub(l.At) < loadFresh {
+			ch <- prometheus.MustNewConstMetric(c.nodeCPU, prometheus.GaugeValue, l.CPUPercent, n.ID)
+			if l.CPUCount > 0 {
+				ch <- prometheus.MustNewConstMetric(c.nodeCPUs, prometheus.GaugeValue, float64(l.CPUCount), n.ID)
+			}
+			for w, v := range map[string]float64{"1m": l.Load1, "5m": l.Load5, "15m": l.Load15} {
+				ch <- prometheus.MustNewConstMetric(c.nodeLoad, prometheus.GaugeValue, v, n.ID, w)
+			}
+			if l.MemUsedBytes > 0 || l.MemAvailBytes > 0 {
+				ch <- prometheus.MustNewConstMetric(c.nodeMemUsed, prometheus.GaugeValue, float64(l.MemUsedBytes), n.ID)
+				ch <- prometheus.MustNewConstMetric(c.nodeMemAvail, prometheus.GaugeValue, float64(l.MemAvailBytes), n.ID)
+			}
+			loads[n.ID] = l
+		}
+
+		for _, p := range n.IBPorts {
+			port := strconv.Itoa(p.Port)
+			active := 0.0
+			if p.Active() {
+				active = 1
+			}
+			ch <- prometheus.MustNewConstMetric(c.ibActive, prometheus.GaugeValue, active,
+				n.ID, p.HCA, port, p.LinkLayer, p.State, p.PhysState)
+			ch <- prometheus.MustNewConstMetric(c.ibRate, prometheus.GaugeValue, float64(p.RateGbps),
+				n.ID, p.HCA, port)
+			for name, v := range p.Counters {
+				ch <- prometheus.MustNewConstMetric(c.ibCounter, prometheus.CounterValue, float64(v),
+					n.ID, p.HCA, port, name)
+			}
+		}
+
+		fabricGPUs := 0
+		for _, g := range n.GPUs {
+			fb := g.Fabric
+			if fb == nil || strings.EqualFold(fb.State, "Not Supported") {
+				continue
+			}
+			fabricGPUs++
+			joined := 0.0
+			if fb.Domain() != "" {
+				joined = 1
+			}
+			ch <- prometheus.MustNewConstMetric(c.gpuNVLJoined, prometheus.GaugeValue, joined,
+				n.ID, strconv.Itoa(g.Index), fb.Domain(), fb.State)
+		}
+		if fabricGPUs > 0 {
+			up := 0.0
+			if n.IMEX.State == "active" && n.IMEX.Channels > 0 {
+				up = 1
+			}
+			ch <- prometheus.MustNewConstMetric(c.imexUp, prometheus.GaugeValue, up, n.ID)
+		}
+
 		byClass := map[string]int{}
 		for _, g := range n.GPUs {
 			class := g.Class()
@@ -202,11 +331,46 @@ func (c *Collector) Collect(ch chan<- prometheus.Metric) {
 		}
 	}
 
+	// The fabrics as the adapters find them: their subnet managers, and how
+	// many things disagree.
+	rep := fabrics.Analyze(c.src.Nodes(), nil, now)
+	for _, ph := range rep.Physical {
+		name := strings.Join(ph.Declared, ",")
+		if ph.SM != nil && ph.SM.GUID != "" {
+			ch <- prometheus.MustNewConstMetric(c.fabricMaster, prometheus.GaugeValue, 1, name, ph.Key, ph.SM.GUID)
+		}
+		byState := map[string]int{}
+		for _, sm := range ph.SMs {
+			byState[sm.State]++
+		}
+		for st, n := range byState {
+			ch <- prometheus.MustNewConstMetric(c.fabricSMs, prometheus.GaugeValue, float64(n), name, ph.Key, st)
+		}
+	}
+	levels := map[fabrics.Level]int{fabrics.Fail: 0, fabrics.Warn: 0, fabrics.Info: 0}
+	for _, p := range rep.Problems {
+		levels[p.Level]++
+	}
+	for l, n := range levels {
+		ch <- prometheus.MustNewConstMetric(c.fabricProblems, prometheus.GaugeValue, float64(n), string(l))
+	}
+
 	// Replicas by model and state, so a model stuck starting is visible as a
 	// shape on a graph rather than only in a log line nobody tails.
 	counts := map[[2]string]int{}
 	ready := map[string]int{}
 	for _, rep := range c.src.Replicas() {
+		// Each engine's share of its node, labelled with what it serves, so
+		// "which model is eating this machine" is one query.
+		for _, sh := range rep.Shards {
+			if l := loads[sh.NodeID]; l != nil {
+				if use, ok := l.Shards[sh.ID]; ok {
+					labels := []string{sh.NodeID, rep.Model, rep.ID, sh.ID}
+					ch <- prometheus.MustNewConstMetric(c.shardCPU, prometheus.GaugeValue, use.CPUCores, labels...)
+					ch <- prometheus.MustNewConstMetric(c.shardMemory, prometheus.GaugeValue, float64(use.RSSBytes), labels...)
+				}
+			}
+		}
 		counts[[2]string{rep.Model, rep.State.String()}]++
 		if rep.State == reconciler.Ready {
 			ready[rep.Model]++

internal/control/fleetmetrics/fleetmetrics_test.go

@@ -139,3 +139,60 @@ func TestAnEmptyFleetCollectsWithoutError(t *testing.T) {
 		t.Log("no metrics for an empty fleet, which is correct")
 	}
 }
+
+func TestANodesLoadAndEachEnginesShareAreReported(t *testing.T) {
+	// The nodes are CPU machines: without this, how busy one was had no
+	// answer anywhere.
+	n := inventory.Node{ID: "n1", State: inventory.Ready, LastSeen: time.Now(), Load: &inventory.Load{
+		At: time.Now(), CPUPercent: 62.5, CPUCount: 8, Load1: 3.5, Load5: 2, Load15: 1,
+		MemUsedBytes: 6 << 30, MemAvailBytes: 2 << 30,
+		Shards: map[string]inventory.ShardLoad{"r1-0": {CPUCores: 4.5, RSSBytes: 5 << 30}},
+	}}
+	src := fakeSource{nodes: []inventory.Node{n}, replicas: []reconciler.Replica{{
+		ID: "r1", Model: "qwen", State: reconciler.Ready,
+		Shards: []reconciler.Shard{{ID: "r1-0", NodeID: "n1"}},
+	}}}
+	out := gather(t, src)
+	for _, want := range []string{
+		"llmgw_node_cpu_utilization_percent node=n1 -> 62.5",
+		"llmgw_node_cpus node=n1 -> 8",
+		"llmgw_node_load node=n1 window=1m -> 3.5",
+		"llmgw_node_memory_used_bytes node=n1 -> 6442450944",
+		"llmgw_shard_cpu_cores model=qwen node=n1 replica=r1 shard=r1-0 -> 4.5",
+		"llmgw_shard_memory_bytes model=qwen node=n1 replica=r1 shard=r1-0 -> 5368709120",
+	} {
+		if !strings.Contains(out, want) {
+			t.Errorf("missing %q in:\n%s", want, out)
+		}
+	}
+}
+
+func TestAStaleLoadIsAbsentNotRepeated(t *testing.T) {
+	// A node that stopped reporting has an unknown load. Repeating its last
+	// figure would draw a busy machine long after it went away.
+	n := inventory.Node{ID: "n1", State: inventory.Ready, LastSeen: time.Now(),
+		Load: &inventory.Load{At: time.Now().Add(-5 * time.Minute), CPUPercent: 90}}
+	if out := gather(t, fakeSource{nodes: []inventory.Node{n}}); strings.Contains(out, "llmgw_node_cpu_utilization_percent") {
+		t.Errorf("a stale load was reported:\n%s", out)
+	}
+}
+
+func TestEveryIBPortIsReportedWithItsCounters(t *testing.T) {
+	n := gpuNode("n1", 0, "ib-a")
+	n.IBPorts = []inventory.IBPort{
+		{HCA: "mlx5_0", Port: 1, State: "ACTIVE", PhysState: "LinkUp", RateGbps: 400, LinkLayer: "InfiniBand",
+			Counters: map[string]uint64{"symbol_error": 12, "port_xmit_data": 4096}},
+		{HCA: "mlx5_1", Port: 1, State: "DOWN", PhysState: "Polling", LinkLayer: "InfiniBand"},
+	}
+	out := gather(t, fakeSource{nodes: []inventory.Node{n}})
+	for _, want := range []string{
+		"llmgw_ib_port_active hca=mlx5_0 link_layer=InfiniBand node=n1 phys_state=LinkUp port=1 state=ACTIVE -> 1",
+		"llmgw_ib_port_active hca=mlx5_1 link_layer=InfiniBand node=n1 phys_state=Polling port=1 state=DOWN -> 0",
+		"llmgw_ib_port_rate_gbps hca=mlx5_0 node=n1 port=1 -> 400",
+		"llmgw_ib_port_counter_total counter=symbol_error hca=mlx5_0 node=n1 port=1",
+	} {
+		if !strings.Contains(out, want) {
+			t.Errorf("missing %q in:\n%s", want, out)
+		}
+	}
+}

internal/control/inventory/inventory.go

@@ -110,6 +110,200 @@ type Node struct {
 	Host discovery.Host `json:"host"`
 	// Provision is where installing each wanted engine stands on this node.
 	Provision []EngineInstall `json:"provision,omitempty"`
+
+	// Load is how busy the machine was at the last heartbeat that measured
+	// it. Nil before the first one, and from an agent too old to say. Not
+	// persisted: it describes a moment, and a restored one would be stale.
+	Load *Load `json:"-"`
+	// IBPorts are the node's InfiniBand and RoCE ports at the last heartbeat,
+	// with their counters. Not persisted, for the same reason as Load.
+	IBPorts []IBPort `json:"-"`
+	// IBAt is when IBPorts was read, for the rates.
+	IBAt time.Time `json:"-"`
+	// IMEX is the service a multi-node NVLink domain needs on this node.
+	IMEX discovery.IMEX `json:"imex"`
+}
+
+// NVLinkDomain is the multi-node NVLink domain this node's GPUs are joined
+// to, when every one of them is joined to the same. A node half joined — a
+// card still coming up, or one that fell out — is in none: a replica that
+// counts on its NVLink would find a card that has none.
+func (n Node) NVLinkDomain() string {
+	d := ""
+	for _, g := range n.GPUs {
+		gd := g.Fabric.Domain()
+		if gd == "" || (d != "" && gd != d) {
+			return ""
+		}
+		d = gd
+	}
+	return d
+}
+
+// IBPort is one port's state, identity and cumulative counters.
+type IBPort struct {
+	HCA       string
+	Port      int
+	State     string
+	PhysState string
+	RateGbps  int
+	LinkLayer string
+	Counters  map[string]uint64
+
+	NodeGUID  string
+	PortGUID  string
+	GIDPrefix string
+	// SMGUID and the rest are the master subnet manager this port sees;
+	// SMError says why they are empty on an active InfiniBand port.
+	SMGUID     string
+	SMLID      int
+	SMPriority int
+	SMState    string
+	SMError    string
+	// Network is a RoCE port's IP network.
+	Network string
+	// SMs is every subnet manager on the port's fabric, master and standbys,
+	// on the one port per fabric that asked; SMsError says why it could not.
+	SMs      []SubnetManager
+	SMsError string
+	// TxBps and RxBps are bytes a second, from this heartbeat's data
+	// counters against the last one's; zero on the first.
+	TxBps, RxBps float64
+}
+
+// SubnetManager is one subnet manager on a fabric.
+type SubnetManager struct {
+	GUID     string
+	LID      int
+	Priority int
+	State    string
+}
+
+// CanonicalSMs maps every subnet manager GUID the cluster has heard of to
+// one name for its fabric: the lowest GUID among the managers listed
+// together. A fabric's master changes on a failover and its list does not,
+// so ports that see the old master and ports that already see the new one
+// stay one fabric through it.
+func CanonicalSMs(nodes []Node) map[string]string {
+	parent := map[string]string{}
+	var find func(string) string
+	find = func(x string) string {
+		p, ok := parent[x]
+		if !ok || p == x {
+			parent[x] = x
+			return x
+		}
+		root := find(p)
+		parent[x] = root
+		return root
+	}
+	union := func(a, b string) {
+		ra, rb := find(a), find(b)
+		if ra == rb {
+			return
+		}
+		if ra < rb {
+			parent[rb] = ra
+		} else {
+			parent[ra] = rb
+		}
+	}
+	for _, n := range nodes {
+		for _, p := range n.IBPorts {
+			if p.SMGUID != "" {
+				find(p.SMGUID)
+			}
+			for _, sm := range p.SMs {
+				union(sm.GUID, p.SMGUID)
+				if p.SMGUID == "" {
+					find(sm.GUID)
+				}
+			}
+		}
+	}
+	out := make(map[string]string, len(parent))
+	for g := range parent {
+		out[g] = find(g)
+	}
+	return out
+}
+
+// SMChange is a port seeing a different master subnet manager than before.
+type SMChange struct {
+	At       time.Time `json:"at"`
+	Node     string    `json:"node"`
+	Port     string    `json:"port"`
+	From, To string
+}
+
+// Active reports whether the port can carry traffic.
+func (p IBPort) Active() bool { return strings.EqualFold(p.State, "ACTIVE") }
+
+// RoCE reports whether the port is Ethernet carrying RDMA.
+func (p IBPort) RoCE() bool { return strings.EqualFold(p.LinkLayer, "Ethernet") }
+
+// FabricKey names the physical fabric the port is on: its master subnet
+// manager on InfiniBand, its IP network on RoCE. Empty when not known, and
+// for a port that is not up.
+func (p IBPort) FabricKey() string { return p.FabricKeyIn(nil) }
+
+// FabricKeyIn is FabricKey with the subnet manager named as canon has it,
+// so a fabric keeps one key through a failover.
+func (p IBPort) FabricKeyIn(canon map[string]string) string {
+	if !p.Active() {
+		return ""
+	}
+	switch {
+	case p.RoCE() && p.Network != "":
+		return "roce:" + p.Network
+	case !p.RoCE() && p.SMGUID != "":
+		if c, ok := canon[p.SMGUID]; ok {
+			return "sm:" + c
+		}
+		return "sm:" + p.SMGUID
+	}
+	return ""
+}
+
+// PhysicalFabric is the physical fabric this node's adapters for a declared
+// fabric are on. Known is false when none of them says; ok is false when
+// they disagree — two adapters declared as one fabric that are cabled to two.
+func (n Node) PhysicalFabric(fabric string, canon map[string]string) (key string, known, ok bool) {
+	hcas := n.HCAsFor(fabric)
+	for _, p := range n.IBPorts {
+		if !slices.Contains(hcas, p.HCA) {
+			continue
+		}
+		k := p.FabricKeyIn(canon)
+		if k == "" {
+			continue
+		}
+		if key != "" && k != key {
+			return "", true, false
+		}
+		key = k
+	}
+	return key, key != "", true
+}
+
+// Load is a node's use of its CPU and memory, and each engine's share.
+type Load struct {
+	At            time.Time
+	CPUPercent    float64
+	CPUCount      int
+	Load1         float64
+	Load5         float64
+	Load15        float64
+	MemUsedBytes  int64
+	MemAvailBytes int64
+	// Shards is each engine's use, by shard ID.
+	Shards map[string]ShardLoad
+}
+
+// ShardLoad is one engine's use: cores busy, and resident memory.
+type ShardLoad struct {
+	CPUCores float64
+	RSSBytes int64
 }
 
 // EngineInstall is where one engine's installation stands on a node.
@@ -184,6 +378,10 @@ type Store struct {
 	nodes map[string]*Node
 	// allocs maps node id to the shards placed there.
 	allocs map[string][]Allocation
+	// smChanges are ports' changes of master subnet manager, newest last.
+	smChanges []SMChange
+	// scans are the latest ibnetdiscover of each physical fabric, by key.
+	scans map[string]FabricScan
 }
 
 // NewStore returns an empty inventory.
@@ -193,6 +391,62 @@ func NewStore() *Store {
 
 // Upsert records a node's registration, preserving operator-set state such as
 // a cordon: an agent reconnecting must not silently undo an operator's decision.
+// RecordSMChange keeps a port's change of master subnet manager, the last
+// few hundred of them: a failover is the fabric's own event, and nothing
+// else would remember it once the ports agree again.
+func (s *Store) RecordSMChange(c SMChange) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	s.smChanges = append(s.smChanges, c)
+	if len(s.smChanges) > 500 {
+		s.smChanges = slices.Clone(s.smChanges[len(s.smChanges)-500:])
+	}
+}
+
+// FabricScan is one ibnetdiscover of a physical fabric: its text, and who
+// ran it when.
+type FabricScan struct {
+	Key    string
+	At     time.Time
+	Node   string
+	Port   string
+	Output string
+	Err    string
+	Took   time.Duration
+}
+
+// SetScan keeps a fabric's latest scan. A failed one does not replace a
+// good one's text: a map from ten minutes ago is worth more than none.
+func (s *Store) SetScan(sc FabricScan) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	if s.scans == nil {
+		s.scans = map[string]FabricScan{}
+	}
+	if sc.Err != "" {
+		if old, ok := s.scans[sc.Key]; ok && old.Output != "" {
+			old.Err = sc.Err
+			s.scans[sc.Key] = old
+			return
+		}
+	}
+	s.scans[sc.Key] = sc
+}
+
+// Scans returns the latest scan of every fabric.
+func (s *Store) Scans() map[string]FabricScan {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	return maps.Clone(s.scans)
+}
+
+// SMChanges returns the recorded changes, oldest first.
+func (s *Store) SMChanges() []SMChange {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	return slices.Clone(s.smChanges)
+}
+
 func (s *Store) Upsert(n Node) {
 	s.mu.Lock()
 	defer s.mu.Unlock()

internal/control/modelsvc/modelsvc.go

@@ -46,6 +46,10 @@ type Service struct {
 	apply    Applier
 	log      *slog.Logger
 
+	// check refuses a model that cannot run before it is added; see
+	// SetCheck.
+	check func(context.Context, catalog.Model) error
+
 	// mu serialises rebuilds. They are rare — a person adding a model — and two
 	// at once could apply catalogs in either order, leaving the one in force
 	// disagreeing with the database that outlives it.
@@ -70,6 +74,22 @@ func (s *Service) Declared(name string) bool {
 	return false
 }
 
+// Sources maps each weights source in the catalog to the models that name it,
+// so a page listing files can say which of them are in use and how they run.
+func (s *Service) Sources(ctx context.Context) (map[string][]catalog.Model, error) {
+	c, err := s.Load(ctx)
+	if err != nil {
+		return nil, err
+	}
+	out := map[string][]catalog.Model{}
+	for _, m := range c.Models() {
+		if m.Source != "" {
+			out[m.Source] = append(out[m.Source], m)
+		}
+	}
+	return out, nil
+}
+
 // Add puts a model in the catalog and keeps it across restarts.
 //
 // Validated by building the whole catalog before anything is written. A model
@@ -84,6 +104,11 @@ func (s *Service) Add(ctx context.Context, m catalog.Model, by string) error {
 	if s.Declared(m.Name) {
 		return fmt.Errorf("model %q is declared in configuration; edit that rather than adding a second definition", m.Name)
 	}
+	if s.check != nil {
+		if err := s.check(ctx, m); err != nil {
+			return err
+		}
+	}
 
 	stored, err := s.storedModels(ctx)
 	if err != nil {
@@ -283,3 +308,8 @@ func (s *Service) storedModels(ctx context.Context) ([]catalog.Model, error) {
 	}
 	return out, nil
 }
+
+// SetCheck runs before a model is added, and refuses it by returning an error.
+// Used to catch weights the engine cannot load, which otherwise are placed and
+// then fail on every node they are tried on.
+func (s *Service) SetCheck(check func(context.Context, catalog.Model) error) { s.check = check }

internal/control/nvlink/nvlink.go

@@ -0,0 +1,172 @@
+// Package nvlink works out the cluster's multi-node NVLink domains — an NVL72
+// rack, an NVLink Switch system — from what each GPU reports, and what keeps
+// one from being usable.
+//
+// A domain is a scale-up island: its GPUs reach each other over NVLink
+// whatever node they are in, so a tensor-parallel group can span its nodes
+// at NVLink speed. What it needs to be one is every card joined by the
+// fabric manager and IMEX running on every node; either missing, and a
+// replica placed across it fails or crawls.
+package nvlink
+
+import (
+	"fmt"
+	"sort"
+	"strings"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+)
+
+// Level is how much a problem matters.
+type Level string
+
+// The levels.
+const (
+	Fail Level = "fail"
+	Warn Level = "warn"
+)
+
+// Problem is one thing wrong with a domain.
+type Problem struct {
+	Level   Level  `json:"level"`
+	Domain  string `json:"domain"`
+	Message string `json:"message"`
+}
+
+// GPU is one card as its domain sees it.
+type GPU struct {
+	Index     int    `json:"index"`
+	Name      string `json:"name"`
+	State     string `json:"state"`
+	Status    string `json:"status,omitempty"`
+	Bandwidth string `json:"bandwidth,omitempty"`
+	Joined    bool   `json:"joined"`
+}
+
+// Member is one node of a domain.
+type Member struct {
+	Node string `json:"node"`
+	GPUs []GPU  `json:"gpus"`
+	// Usable is every card joined and IMEX up: what placement requires.
+	Usable       bool   `json:"usable"`
+	IMEX         string `json:"imex"`
+	IMEXChannels int    `json:"imex_channels"`
+}
+
+// Domain is one multi-node NVLink domain.
+type Domain struct {
+	Key         string   `json:"key"`
+	ClusterUUID string   `json:"cluster_uuid"`
+	CliqueID    string   `json:"clique_id"`
+	Members     []Member `json:"members"`
+	GPUs        int      `json:"gpus"`
+	Joined      int      `json:"joined"`
+	Usable      int      `json:"usable_nodes"`
+}
+
+// Report is every domain and what is wrong with them.
+type Report struct {
+	Domains  []Domain  `json:"domains"`
+	Problems []Problem `json:"problems"`
+}
+
+// Analyze reads every node's GPUs.
+func Analyze(nodes []inventory.Node) Report {
+	var r Report
+	by := map[string]*Domain{}
+	for _, n := range nodes {
+		// The domain a node is in is the one any of its cards joined; a card
+		// still joining has no domain of its own yet, and is counted against
+		// the node's.
+		key := ""
+		for _, g := range n.GPUs {
+			if d := g.Fabric.Domain(); d != "" {
+				key = d
+				break
+			}
+		}
+		if key == "" {
+			continue
+		}
+		d := by[key]
+		if d == nil {
+			uuid, clique, _ := strings.Cut(key, "/")
+			d = &Domain{Key: key, ClusterUUID: uuid, CliqueID: clique}
+			by[key] = d
+		}
+		m := Member{Node: n.ID, IMEX: n.IMEX.State, IMEXChannels: n.IMEX.Channels}
+		joined := 0
+		for _, g := range n.GPUs {
+			v := GPU{Index: g.Index, Name: g.Name}
+			if f := g.Fabric; f != nil {
+				v.State, v.Status, v.Bandwidth = f.State, f.Status, f.Bandwidth
+				v.Joined = f.Domain() == key
+			}
+			if v.Joined {
+				joined++
+			} else {
+				why := v.State
+				if why == "" {
+					why = "no fabric"
+				}
+				if v.Status != "" && !strings.EqualFold(v.Status, "Success") {
+					why += ": " + v.Status
+				}
+				r.Problems = append(r.Problems, Problem{Level: Fail, Domain: key, Message: fmt.Sprintf(
+					"%s GPU %d is not joined to its domain (%s)", n.ID, g.Index, why)})
+			}
+			if v.Bandwidth != "" && !strings.EqualFold(v.Bandwidth, "Full") {
+				r.Problems = append(r.Problems, Problem{Level: Warn, Domain: key, Message: fmt.Sprintf(
+					"%s GPU %d has %s NVLink bandwidth", n.ID, g.Index, strings.ToLower(v.Bandwidth))})
+			}
+			m.GPUs = append(m.GPUs, v)
+		}
+		imexOK := n.IMEX.State == "active" && n.IMEX.Channels > 0
+		switch {
+		case n.IMEX.State != "active":
+			state := n.IMEX.State
+			if state == "" {
+				state = "not installed"
+			}
+			r.Problems = append(r.Problems, Problem{Level: Fail, Domain: key, Message: fmt.Sprintf(
+				"%s: nvidia-imex is %s — without it the domain's GPUs cannot share memory across nodes",
+				n.ID, state)})
+		case n.IMEX.Channels == 0:
+			r.Problems = append(r.Problems, Problem{Level: Fail, Domain: key, Message: fmt.Sprintf(
+				"%s: nvidia-imex runs but there is no IMEX channel for an engine to open (/dev/nvidia-caps-imex-channels)", n.ID)})
+		}
+		m.Usable = joined == len(n.GPUs) && joined > 0 && imexOK
+		if joined > 0 && joined < len(n.GPUs) {
+			r.Problems = append(r.Problems, Problem{Level: Fail, Domain: key, Message: fmt.Sprintf(
+				"%s is half joined (%d of %d GPUs), so no replica spans the domain through it", n.ID, joined, len(n.GPUs))})
+		}
+		d.GPUs += len(n.GPUs)
+		d.Joined += joined
+		if m.Usable {
+			d.Usable++
+		}
+		d.Members = append(d.Members, m)
+	}
+	for _, d := range by {
+		sort.Slice(d.Members, func(i, j int) bool { return d.Members[i].Node < d.Members[j].Node })
+		r.Domains = append(r.Domains, *d)
+	}
+	sort.Slice(r.Domains, func(i, j int) bool { return r.Domains[i].Key < r.Domains[j].Key })
+	sort.SliceStable(r.Problems, func(i, j int) bool {
+		if r.Problems[i].Level != r.Problems[j].Level {
+			return r.Problems[i].Level == Fail
+		}
+		return r.Problems[i].Message < r.Problems[j].Message
+	})
+	return r
+}
+
+// Short is a domain's key as a person reads it: the cluster's first group
+// and the clique.
+func Short(key string) string {
+	uuid, clique, _ := strings.Cut(key, "/")
+	if i := strings.IndexByte(uuid, '-'); i > 0 {
+		uuid = uuid[:i]
+	}
+	return uuid + "·" + clique
+}

internal/control/nvlink/nvlink_test.go

@@ -0,0 +1,55 @@
+package nvlink
+
+import (
+	"strings"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+)
+
+func tray(id string, gpus int, state string) inventory.Node {
+	n := inventory.Node{ID: id, IMEX: discovery.IMEX{State: "active", Channels: 1}}
+	for i := range gpus {
+		n.GPUs = append(n.GPUs, discovery.GPU{Index: i, Name: "NVIDIA GB200", Fabric: &discovery.GPUFabric{
+			State: state, Status: "Success", ClusterUUID: "76b1ad76-7c24-4b6b-9f6c-2a1e0d8c11aa", CliqueID: "1", Bandwidth: "Full"}})
+	}
+	return n
+}
+
+func has(t *testing.T, r Report, l Level, s string) {
+	t.Helper()
+	for _, p := range r.Problems {
+		if p.Level == l && strings.Contains(p.Message, s) {
+			return
+		}
+	}
+	t.Errorf("no %s problem with %q in %+v", l, s, r.Problems)
+}
+
+func TestARackOfJoinedTraysIsOneUsableDomain(t *testing.T) {
+	r := Analyze([]inventory.Node{tray("t1", 4, "Completed"), tray("t2", 4, "Completed"), {ID: "cpu"}})
+	if len(r.Domains) != 1 || r.Domains[0].Usable != 2 || r.Domains[0].Joined != 8 || len(r.Problems) != 0 {
+		t.Errorf("report = %+v", r)
+	}
+	if Short(r.Domains[0].Key) != "76b1ad76·1" {
+		t.Errorf("short = %q", Short(r.Domains[0].Key))
+	}
+}
+
+func TestWhatKeepsATrayOutIsSaid(t *testing.T) {
+	half := tray("t2", 4, "Completed")
+	half.GPUs[3].Fabric = &discovery.GPUFabric{State: "In Progress"}
+	noIMEX := tray("t3", 4, "Completed")
+	noIMEX.IMEX = discovery.IMEX{State: "inactive"}
+	slow := tray("t4", 4, "Completed")
+	slow.GPUs[0].Fabric.Bandwidth = "Degraded"
+	r := Analyze([]inventory.Node{tray("t1", 4, "Completed"), half, noIMEX, slow})
+	has(t, r, Fail, "t2 GPU 3 is not joined to its domain (In Progress)")
+	has(t, r, Fail, "t2 is half joined")
+	has(t, r, Fail, "t3: nvidia-imex is inactive")
+	has(t, r, Warn, "t4 GPU 0 has degraded NVLink bandwidth")
+	if r.Domains[0].Usable != 2 {
+		t.Errorf("usable = %d, want t1 and t4", r.Domains[0].Usable)
+	}
+}

internal/control/placement/placement.go

@@ -213,7 +213,7 @@ func (s *Scheduler) placeSingleNode(p Profile, avoid []string) (*Placement, []Re
 	preferNodesHoldingWeights(p.Source, candidates)
 
 	best := candidates[0]
-	gpus := s.pickGPUs(best.node, best.free, p.GPUsPerNode)
+	gpus := s.pickGPUs(best.node, best.free, p.GPUsPerNode, nil)
 	if gpus == nil {
 		rejections = append(rejections, Rejection{
 			Profile: p.Name, Node: best.node.ID,
@@ -239,60 +239,192 @@ func (s *Scheduler) placeMultiNode(p Profile, avoid []string) (*Placement, []Rej
 	var rejections []Rejection
 
 	fabrics := s.inv.Fabrics()
-	if len(fabrics) == 0 {
+	if len(fabrics) == 0 && !s.anyNVLinkDomain() {
 		return nil, []Rejection{{
 			Profile: p.Name,
 			Reason:  fmt.Sprintf("needs %d nodes on one fabric, but no node reports a fabric", p.Nodes),
 		}}
 	}
 
+	// A multi-node NVLink domain first: a replica inside one runs tensor
+	// parallelism across its nodes at NVLink speed, which no InfiniBand
+	// fabric matches.
+	if pl := s.placeOnNVLink(p, avoid, &rejections); pl != nil {
+		return pl, rejections
+	}
+
+	canon := inventory.CanonicalSMs(s.inv.Nodes())
 	for _, fabric := range fabrics {
-		pool := fabric + "/" + p.classLabel()
-		candidates := s.eligibleNodes(p, fabric, avoid, &rejections)
-
-		if len(candidates) < p.Nodes {
-			rejections = append(rejections, Rejection{
-				Profile: p.Name, Pool: pool,
-				Reason: fmt.Sprintf("needs %d nodes, only %d qualify", p.Nodes, len(candidates)),
-			})
-			continue
-		}
-
-		// Prefer the emptiest nodes for multi-node work: a replica spanning
-		// nodes is the least movable thing in the cluster, so give it the
-		// cleanest machines rather than squeezing it into the gaps.
-		sort.SliceStable(candidates, func(i, j int) bool {
-			return len(candidates[i].free) > len(candidates[j].free)
-		})
-
-		assignments := make([]NodeAssignment, 0, p.Nodes)
-		for rank, c := range candidates[:p.Nodes] {
-			gpus := s.pickGPUs(c.node, c.free, p.GPUsPerNode)
-			if gpus == nil {
-				assignments = nil
-				rejections = append(rejections, Rejection{
-					Profile: p.Name, Pool: pool, Node: c.node.ID,
-					Reason: fmt.Sprintf("no usable set of %d GPUs", p.GPUsPerNode),
-				})
-				break
+		all := s.eligibleNodes(p, fabric, avoid, &rejections)
+		for _, group := range physicalGroups(fabric, all, p, canon, &rejections) {
+			pool := fabric + "/" + p.classLabel()
+			if group.key != "" && group.split {
+				pool = fabric + "@" + shortKey(group.key) + "/" + p.classLabel()
+			}
+			if pl := s.placeOnFabric(p, fabric, pool, group.nodes, &rejections); pl != nil {
+				return pl, rejections
 			}
-			assignments = append(assignments, NodeAssignment{
-				NodeID:     c.node.ID,
-				Rank:       rank,
-				GPUIndices: gpus,
-				// Only this fabric's adapters, never the node's others.
-				HCAs: c.node.HCAsFor(fabric),
-			})
 		}
-		if assignments == nil {
-			continue
-		}
-
-		return &Placement{Profile: p, Fabric: fabric, Pool: pool, Nodes: assignments}, rejections
 	}
 	return nil, rejections
 }
 
+// NVLinkFabric is how a placement names a multi-node NVLink domain in its
+// Fabric, so the shards on it are told to use it.
+const NVLinkFabric = "nvl:"
+
+func (s *Scheduler) anyNVLinkDomain() bool {
+	for _, n := range s.inv.Nodes() {
+		if n.NVLinkDomain() != "" {
+			return true
+		}
+	}
+	return false
+}
+
+// placeOnNVLink forms a replica inside one multi-node NVLink domain, from
+// nodes whose every card is joined to it and whose IMEX is up: a node
+// missing either would leave the replica a card, or its memory, short.
+func (s *Scheduler) placeOnNVLink(p Profile, avoid []string, rejections *[]Rejection) *Placement {
+	var none []Rejection
+	cands := s.eligibleNodes(p, "", avoid, &none)
+	by := map[string][]candidate{}
+	for _, c := range cands {
+		d := c.node.NVLinkDomain()
+		if d == "" {
+			continue
+		}
+		if c.node.IMEX.State != "active" || c.node.IMEX.Channels == 0 {
+			*rejections = append(*rejections, Rejection{Profile: p.Name, Pool: NVLinkFabric + shortDomain(d), Node: c.node.ID,
+				Reason: "in an NVLink domain, but nvidia-imex is not up"})
+			continue
+		}
+		by[d] = append(by[d], c)
+	}
+	keys := make([]string, 0, len(by))
+	for k := range by {
+		keys = append(keys, k)
+	}
+	sort.Slice(keys, func(i, j int) bool {
+		if len(by[keys[i]]) != len(by[keys[j]]) {
+			return len(by[keys[i]]) > len(by[keys[j]])
+		}
+		return keys[i] < keys[j]
+	})
+	for _, d := range keys {
+		pool := NVLinkFabric + shortDomain(d) + "/" + p.classLabel()
+		if pl := s.placeOnFabric(p, NVLinkFabric+d, pool, by[d], rejections); pl != nil {
+			return pl
+		}
+	}
+	return nil
+}
+
+// shortDomain is an NVLink domain as a pool name carries it.
+func shortDomain(d string) string {
+	uuid, clique, _ := strings.Cut(d, "/")
+	if i := strings.IndexByte(uuid, '-'); i > 0 {
+		uuid = uuid[:i]
+	}
+	return uuid + "." + clique
+}
+
+// fabricGroup is the candidates on one physical fabric.
+type fabricGroup struct {
+	key   string
+	nodes []candidate
+	// split is set when the declared fabric turned out to be several.
+	split bool
+}
+
+// physicalGroups divides a declared fabric's candidates by the physical
+// fabric their adapters are on.
+//
+// A fabric's name is what an operator typed, and two separate fabrics given
+// one name would have had a replica spread across both: it starts, and hangs
+// on its first collective, since nothing routes between them. Grouped by the
+// subnet manager each node's ports see (by IP network on RoCE), a replica
+// stays on one. Nodes that do not say — an agent without infiniband-diags —
+// are a group of their own rather than trusted to join one that does.
+func physicalGroups(fabric string, cands []candidate, p Profile, canon map[string]string, rejections *[]Rejection) []fabricGroup {
+	by := map[string][]candidate{}
+	for _, c := range cands {
+		key, _, ok := c.node.PhysicalFabric(fabric, canon)
+		if !ok {
+			*rejections = append(*rejections, Rejection{
+				Profile: p.Name, Pool: fabric + "/" + p.classLabel(), Node: c.node.ID,
+				Reason: "its adapters declared as " + fabric + " are cabled to different fabrics",
+			})
+			continue
+		}
+		by[key] = append(by[key], c)
+	}
+	out := make([]fabricGroup, 0, len(by))
+	for k, v := range by {
+		out = append(out, fabricGroup{key: k, nodes: v, split: len(by) > 1})
+	}
+	// Largest first, then by key, so the choice is the same every time.
+	sort.Slice(out, func(i, j int) bool {
+		if len(out[i].nodes) != len(out[j].nodes) {
+			return len(out[i].nodes) > len(out[j].nodes)
+		}
+		return out[i].key < out[j].key
+	})
+	return out
+}
+
+// shortKey is a physical fabric's key as a pool name carries it.
+func shortKey(k string) string {
+	if i := strings.IndexByte(k, ':'); i >= 0 {
+		k = k[i+1:]
+	}
+	if len(k) > 10 && strings.HasPrefix(k, "0x") {
+		return "sm-" + k[len(k)-6:]
+	}
+	return k
+}
+
+// placeOnFabric forms a replica from candidates that share one fabric.
+func (s *Scheduler) placeOnFabric(p Profile, fabric, pool string, candidates []candidate, rejections *[]Rejection) *Placement {
+	if len(candidates) < p.Nodes {
+		*rejections = append(*rejections, Rejection{
+			Profile: p.Name, Pool: pool,
+			Reason: fmt.Sprintf("needs %d nodes, only %d qualify", p.Nodes, len(candidates)),
+		})
+		return nil
+	}
+
+	// Prefer the emptiest nodes for multi-node work: a replica spanning
+	// nodes is the least movable thing in the cluster, so give it the
+	// cleanest machines rather than squeezing it into the gaps.
+	sort.SliceStable(candidates, func(i, j int) bool {
+		return len(candidates[i].free) > len(candidates[j].free)
+	})
+	// Then by the fabric's map, where there is one: one leaf's nodes
+	// together, a degraded cable last.
+	candidates = byLeaf(candidates, hints(s.inv))
+
+	assignments := make([]NodeAssignment, 0, p.Nodes)
+	for rank, c := range candidates[:p.Nodes] {
+		gpus := s.pickGPUs(c.node, c.free, p.GPUsPerNode, c.node.HCAsFor(fabric))
+		if gpus == nil {
+			*rejections = append(*rejections, Rejection{
+				Profile: p.Name, Pool: pool, Node: c.node.ID,
+				Reason: fmt.Sprintf("no usable set of %d GPUs", p.GPUsPerNode),
+			})
+			return nil
+		}
+		assignments = append(assignments, NodeAssignment{
+			NodeID:     c.node.ID,
+			Rank:       rank,
+			GPUIndices: gpus,
+			// Only this fabric's adapters, never the node's others.
+			HCAs: c.node.HCAsFor(fabric),
+		})
+	}
+	return &Placement{Profile: p, Fabric: fabric, Pool: pool, Nodes: assignments}
+}
+
 // candidate is a node that passed the hard constraints.
 type candidate struct {
 	node inventory.Node
@@ -475,48 +607,107 @@ func (s *Scheduler) eligibleNodes(p Profile, fabric string, avoid []string, reje
 
 // pickGPUs chooses which of a node's free GPUs to use.
 //
-// It prefers a set that shares an NVLink domain: within a node, a
-// tensor-parallel group that has to fall back to PCIe for its all-reduces runs
-// several times slower. NVLink topology is only available from the NVML build,
-// so without it this degrades to taking the lowest indices — which is usually
-// right, but only by accident of how vendors number slots.
-func (s *Scheduler) pickGPUs(n inventory.Node, free []discovery.GPU, want int) []int {
+// Every set of the right size is scored, which a node's handful of cards
+// makes cheap, in this order:
+//
+//   - NVLink: the most pairs in the set joined by it. A tensor-parallel group
+//     that falls back to PCIe for its all-reduces runs several times slower,
+//     and "seed and its peers", the rule before, took cards that were each
+//     joined to the seed but not to each other.
+//   - For a multi-node shard, the fabric: cards near one of its adapters (PIX
+//     or PXB rather than across the socket), since every transfer between
+//     nodes goes through one.
+//   - One NUMA node rather than two.
+//   - The lowest indices, so the answer is the same every time.
+//
+// hcas are the fabric's adapters for a multi-node shard, nil otherwise.
+// Without topology — an agent that reported none — every set scores the same
+// and it is the lowest indices, as it always was.
+func (s *Scheduler) pickGPUs(n inventory.Node, free []discovery.GPU, want int, hcas []string) []int {
 	if len(free) < want {
 		return nil
 	}
-	if want == 1 {
+	if want == 1 && len(hcas) == 0 {
 		return []int{free[0].Index}
 	}
+	sorted := slices.Clone(free)
+	slices.SortFunc(sorted, func(a, b discovery.GPU) int { return a.Index - b.Index })
 
-	freeSet := map[int]struct{}{}
-	for _, g := range free {
-		freeSet[g.Index] = struct{}{}
+	// Too many to score every set: take the lowest, as before topology.
+	if len(sorted) > 16 {
+		out := make([]int, 0, want)
+		for _, g := range sorted[:want] {
+			out = append(out, g.Index)
+		}
+		return out
 	}
 
-	// Try each free GPU as a seed and grow its NVLink-connected set.
-	for _, seed := range free {
-		group := []int{seed.Index}
-		for _, peer := range seed.NVLinkPeers {
-			if _, ok := freeSet[peer]; !ok || peer == seed.Index {
-				continue
-			}
-			group = append(group, peer)
-			if len(group) == want {
-				slices.Sort(group)
-				return group
+	var best []int
+	bestScore := [3]int{-1, 0, 0}
+	pick := make([]int, 0, want)
+	var walk func(from int)
+	walk = func(from int) {
+		if len(pick) == want {
+			sc := gpuSetScore(sorted, pick, hcas)
+			if best == nil || better(sc, bestScore) {
+				bestScore = sc
+				best = slices.Clone(pick)
 			}
+			return
+		}
+		for i := from; i <= len(sorted)-(want-len(pick)); i++ {
+			pick = append(pick, i)
+			walk(i + 1)
+			pick = pick[:len(pick)-1]
 		}
 	}
+	walk(0)
 
-	// No NVLink domain large enough, or no topology at all.
-	out := make([]int, 0, want)
-	for _, g := range free[:want] {
-		out = append(out, g.Index)
+	out := make([]int, len(best))
+	for i, k := range best {
+		out[i] = sorted[k].Index
 	}
-	slices.Sort(out)
 	return out
 }
 
+// gpuSetScore rates a set, as positions in gpus: NVLink pairs, nearness to
+// the fabric (negated, so higher is better) and NUMA nodes spanned (negated).
+func gpuSetScore(gpus []discovery.GPU, set []int, hcas []string) [3]int {
+	var sc [3]int
+	numa := map[int]bool{}
+	for a, i := range set {
+		g := gpus[i]
+		numa[g.NUMANode] = true
+		for _, j := range set[a+1:] {
+			if slices.Contains(g.NVLinkPeers, gpus[j].Index) {
+				sc[0]++
+			}
+		}
+		if len(hcas) > 0 && len(g.NICPaths) > 0 {
+			nearest := 5
+			for _, h := range hcas {
+				if rel, ok := g.NICPaths[h]; ok {
+					nearest = min(nearest, discovery.PathRank(rel))
+				}
+			}
+			sc[1] -= nearest
+		}
+	}
+	sc[2] = -len(numa)
+	return sc
+}
+
+// better reports whether a scores above b, most important first. Equal is
+// not better, so the first set found — the lowest indices — keeps a tie.
+func better(a, b [3]int) bool {
+	for i := range a {
+		if a[i] != b[i] {
+			return a[i] > b[i]
+		}
+	}
+	return false
+}
+
 // claimedMemory is what the shards already on a node asked for.
 //
 // Declared, not measured. A profile that says it needs 15 GB is treated as

internal/control/placement/placement_test.go

@@ -6,6 +6,7 @@ import (
 	"slices"
 	"strings"
 	"testing"
+	"time"
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
@@ -827,3 +828,246 @@ func TestANodeStillInstallingItsEngineIsNotACandidate(t *testing.T) {
 		t.Errorf("sim was refused:\n%s", res.Explain())
 	}
 }
+
+func TestAGroupIsJoinedToItselfNotOnlyToOneCard(t *testing.T) {
+	// Cards bridged in pairs, as on PCIe boxes with NVLink bridges. For two
+	// cards, a pair joined by NVLink wins over lower indices that are not
+	// joined at all.
+	n := node("n1", "h100-80g", 8)
+	link := func(a, b int) {
+		n.GPUs[a].NVLinkPeers = append(n.GPUs[a].NVLinkPeers, b)
+		n.GPUs[b].NVLinkPeers = append(n.GPUs[b].NVLinkPeers, a)
+	}
+	link(2, 3)
+	link(4, 5)
+	st := store(n)
+	// 0 and 1 are taken; 2-3 is the lowest pair that shares NVLink.
+	st.Allocate("n1", inventory.Allocation{ShardID: "s1", ReplicaID: "r1", GPUIndices: []int{0, 1}})
+	res := New(st).Place([]Profile{{Name: "tp2", GPUClass: "h100-80g", GPUsPerNode: 2, Nodes: 1}}, nil)
+	if res.Placement == nil {
+		t.Fatalf("no placement:\n%s", res.Explain())
+	}
+	if got := res.Placement.Nodes[0].GPUIndices; !slices.Equal(got, []int{2, 3}) {
+		t.Errorf("chose %v, want the NVLink pair 2-3", got)
+	}
+}
+
+func TestAMultiNodeShardTakesTheCardsNearestTheFabric(t *testing.T) {
+	// Every transfer between nodes goes through an adapter. Cards 4-7 sit
+	// under the same PCIe switch as the fabric's adapter; 0-3 are across the
+	// socket from it, and would cross the processor interconnect every time.
+	mk := func(id string) inventory.Node {
+		n := node(id, "h100-80g", 8, "ib-a")
+		n.Fabrics[0].HCAs = []string{"mlx5_1"}
+		for i := range n.GPUs {
+			rel := "SYS"
+			if i >= 4 {
+				rel = "PIX"
+			}
+			n.GPUs[i].NICPaths = map[string]string{"mlx5_1": rel}
+			n.GPUs[i].NUMANode = i / 4
+		}
+		return n
+	}
+	res := New(store(mk("a"), mk("b"))).Place(
+		[]Profile{{Name: "tp8pp", GPUClass: "h100-80g", GPUsPerNode: 4, Nodes: 2}}, nil)
+	if res.Placement == nil {
+		t.Fatalf("no placement:\n%s", res.Explain())
+	}
+	for _, a := range res.Placement.Nodes {
+		if !slices.Equal(a.GPUIndices, []int{4, 5, 6, 7}) {
+			t.Errorf("%s: chose %v, want the cards beside the adapter", a.NodeID, a.GPUIndices)
+		}
+	}
+}
+
+func TestWithoutTopologyItIsTheLowestIndices(t *testing.T) {
+	st := store(node("n1", "h100-80g", 8))
+	st.Allocate("n1", inventory.Allocation{ShardID: "s1", ReplicaID: "r1", GPUIndices: []int{1}})
+	res := New(st).Place([]Profile{{Name: "tp4", GPUClass: "h100-80g", GPUsPerNode: 4, Nodes: 1}}, nil)
+	if got := res.Placement.Nodes[0].GPUIndices; !slices.Equal(got, []int{0, 2, 3, 4}) {
+		t.Errorf("chose %v", got)
+	}
+}
+
+// onSM puts a node's declared adapters on the fabric a subnet manager runs.
+func onSM(n inventory.Node, sm string) inventory.Node {
+	for _, f := range n.Fabrics {
+		for _, h := range f.HCAs {
+			n.IBPorts = append(n.IBPorts, inventory.IBPort{HCA: h, Port: 1, State: "ACTIVE",
+				LinkLayer: "InfiniBand", SMGUID: sm})
+		}
+	}
+	return n
+}
+
+func TestTwoFabricsGivenOneNameAreNeverSpanned(t *testing.T) {
+	// Somebody named two separate fabrics ib-a. Each has two free nodes; the
+	// replica needs three. Spread across both it would start and hang on its
+	// first collective, since nothing routes between them.
+	st := store(
+		onSM(node("a1", "h100-80g", 8, "ib-a"), "0x01"), onSM(node("a2", "h100-80g", 8, "ib-a"), "0x01"),
+		onSM(node("b1", "h100-80g", 8, "ib-a"), "0x02"), onSM(node("b2", "h100-80g", 8, "ib-a"), "0x02"),
+	)
+	res := New(st).Place([]Profile{{Name: "big", GPUClass: "h100-80g", GPUsPerNode: 8, Nodes: 3}}, nil)
+	if res.Placement != nil {
+		t.Fatalf("placed across two physical fabrics: %+v", res.Placement.Nodes)
+	}
+	if !strings.Contains(res.Explain(), "ib-a@") {
+		t.Errorf("the explanation should name the physical fabrics:\n%s", res.Explain())
+	}
+
+	// Two nodes fit on either; they come from one.
+	res = New(st).Place([]Profile{{Name: "pair", GPUClass: "h100-80g", GPUsPerNode: 8, Nodes: 2}}, nil)
+	if res.Placement == nil {
+		t.Fatalf("no placement:\n%s", res.Explain())
+	}
+	first := res.Placement.Nodes[0].NodeID[0]
+	for _, a := range res.Placement.Nodes {
+		if a.NodeID[0] != first {
+			t.Errorf("a pair spans fabrics: %+v", res.Placement.Nodes)
+		}
+	}
+}
+
+func TestANodeThatDoesNotSayIsNotTrustedToJoin(t *testing.T) {
+	// Without infiniband-diags a node cannot say which fabric it is on. It is
+	// kept apart from nodes that can, rather than assumed to be with them.
+	st := store(
+		onSM(node("a1", "h100-80g", 8, "ib-a"), "0x01"),
+		node("u1", "h100-80g", 8, "ib-a"),
+	)
+	if res := New(st).Place([]Profile{{Name: "pair", GPUClass: "h100-80g", GPUsPerNode: 8, Nodes: 2}}, nil); res.Placement != nil {
+		t.Errorf("a node of unknown fabric was paired with one of known: %+v", res.Placement.Nodes)
+	}
+	// Where no node says, the declared name is all there is, as before.
+	st = store(node("u1", "h100-80g", 8, "ib-a"), node("u2", "h100-80g", 8, "ib-a"))
+	if res := New(st).Place([]Profile{{Name: "pair", GPUClass: "h100-80g", GPUsPerNode: 8, Nodes: 2}}, nil); res.Placement == nil {
+		t.Errorf("no placement:\n%s", res.Explain())
+	}
+}
+
+func TestAdaptersOfOneDeclaredFabricOnTwoAreRefused(t *testing.T) {
+	n := node("x", "h100-80g", 8, "ib-a")
+	n.IBPorts = []inventory.IBPort{
+		{HCA: "mlx5_0", Port: 1, State: "ACTIVE", LinkLayer: "InfiniBand", SMGUID: "0x01"},
+		{HCA: "mlx5_1", Port: 1, State: "ACTIVE", LinkLayer: "InfiniBand", SMGUID: "0x02"},
+	}
+	res := New(store(n, onSM(node("y", "h100-80g", 8, "ib-a"), "0x01"))).Place(
+		[]Profile{{Name: "pair", GPUClass: "h100-80g", GPUsPerNode: 8, Nodes: 2}}, nil)
+	if res.Placement != nil || !strings.Contains(res.Explain(), "cabled to different fabrics") {
+		t.Errorf("expected a refusal naming the cabling:\n%s", res.Explain())
+	}
+}
+
+func TestAFailoverDoesNotSplitAFabricForPlacement(t *testing.T) {
+	// One port already sees the new master, the other the old; the managers
+	// listed are the same, so the two nodes are still one fabric.
+	a := onSM(node("a1", "h100-80g", 8, "ib-a"), "0x01")
+	a.IBPorts[0].SMs = []inventory.SubnetManager{{GUID: "0x01", State: "master"}, {GUID: "0x02", State: "standby"}}
+	b := onSM(node("a2", "h100-80g", 8, "ib-a"), "0x02")
+	res := New(store(a, b)).Place([]Profile{{Name: "pair", GPUClass: "h100-80g", GPUsPerNode: 8, Nodes: 2}}, nil)
+	if res.Placement == nil {
+		t.Fatalf("a failover split the fabric:\n%s", res.Explain())
+	}
+}
+
+// scanOf is ibnetdiscover's output for two leaves: leaf-a holding the first
+// two nodes' adapters, leaf-b the other two. degraded names a node whose
+// cable trained at half width.
+func scanOf(ids []string, degraded string) string {
+	var b strings.Builder
+	for li, leaf := range []string{"00000000000000a1", "00000000000000a2"} {
+		fmt.Fprintf(&b, "Switch\t36 \"S-%s\"\t\t# \"MF0;leaf-%c:QM8700/U1\" enhanced port 0 lid %d lmc 0\n", leaf, 'a'+li, li+1)
+		for k := 0; k < 2; k++ {
+			id := ids[li*2+k]
+			rate := "4xHDR"
+			if id == degraded {
+				rate = "2xHDR"
+			}
+			fmt.Fprintf(&b, "[%d]\t\"H-%s\"[1](%s) \t\t# \"%s mlx5_0\" lid %d %s\n", k+1, guidOf(id), guidOf(id), id, 10+li*2+k, rate)
+		}
+		b.WriteString("\n")
+	}
+	return b.String()
+}
+
+func guidOf(id string) string { return fmt.Sprintf("%016x", []byte(id)[len(id)-1]) }
+
+func TestAReplicaStaysUnderOneLeafAndOffDegradedCables(t *testing.T) {
+	ids := []string{"n1", "n2", "n3", "n4"}
+	var nodes []inventory.Node
+	for _, id := range ids {
+		n := onSM(node(id, "h100-80g", 8, "ib-a"), "0x01")
+		n.IBPorts[0].NodeGUID = "0x" + guidOf(id)
+		nodes = append(nodes, n)
+	}
+	st := store(nodes...)
+	// n1 sits on leaf-a with n2, but its cable trained at half width: leaf-a
+	// has one good node, leaf-b two.
+	st.SetScan(inventory.FabricScan{Key: "sm:0x01", At: time.Now(), Output: scanOf(ids, "n1")})
+	res := New(st).Place([]Profile{{Name: "pair", GPUClass: "h100-80g", GPUsPerNode: 8, Nodes: 2}}, nil)
+	if res.Placement == nil {
+		t.Fatalf("no placement:\n%s", res.Explain())
+	}
+	var got []string
+	for _, a := range res.Placement.Nodes {
+		got = append(got, a.NodeID)
+	}
+	if !slices.Equal(got, []string{"n3", "n4"}) {
+		t.Errorf("placed on %v, want the two good nodes under leaf-b", got)
+	}
+}
+
+// inDomain joins every card of a node to one multi-node NVLink domain, with
+// IMEX up.
+func inDomain(n inventory.Node, clique string) inventory.Node {
+	for i := range n.GPUs {
+		n.GPUs[i].Fabric = &discovery.GPUFabric{State: "Completed", ClusterUUID: "76b1ad76-7c24-4b6b-9f6c-2a1e0d8c11aa", CliqueID: clique}
+	}
+	n.IMEX = discovery.IMEX{State: "active", Channels: 1}
+	return n
+}
+
+func TestAReplicaSpansNodesOverNVLinkBeforeInfiniBand(t *testing.T) {
+	// Four trays on one InfiniBand fabric, two of them in an NVLink domain:
+	// the pair is placed across the domain, where tensor parallelism between
+	// nodes runs at NVLink speed, and given no adapters to use.
+	st := store(
+		onSM(node("ib1", "h100-80g", 4, "ib-a"), "0x01"), onSM(node("ib2", "h100-80g", 4, "ib-a"), "0x01"),
+		inDomain(onSM(node("nv1", "h100-80g", 4, "ib-a"), "0x01"), "7"),
+		inDomain(onSM(node("nv2", "h100-80g", 4, "ib-a"), "0x01"), "7"),
+	)
+	res := New(st).Place([]Profile{{Name: "tp8", GPUClass: "h100-80g", GPUsPerNode: 4, Nodes: 2}}, nil)
+	if res.Placement == nil {
+		t.Fatalf("no placement:\n%s", res.Explain())
+	}
+	if !strings.HasPrefix(res.Placement.Fabric, NVLinkFabric) || !strings.HasPrefix(res.Placement.Pool, "nvl:76b1ad76.7/") {
+		t.Errorf("fabric %q pool %q", res.Placement.Fabric, res.Placement.Pool)
+	}
+	for _, a := range res.Placement.Nodes {
+		if !strings.HasPrefix(a.NodeID, "nv") || len(a.HCAs) != 0 {
+			t.Errorf("assignment %+v", a)
+		}
+	}
+}
+
+func TestANodeWithoutIMEXIsNotCountedInItsDomain(t *testing.T) {
+	a := inDomain(onSM(node("nv1", "h100-80g", 4, "ib-a"), "0x01"), "7")
+	b := inDomain(onSM(node("nv2", "h100-80g", 4, "ib-a"), "0x01"), "7")
+	b.IMEX.State = "inactive"
+	res := New(store(a, b)).Place([]Profile{{Name: "tp8", GPUClass: "h100-80g", GPUsPerNode: 4, Nodes: 2}}, nil)
+	// Not across the domain; InfiniBand still carries it.
+	if res.Placement == nil || strings.HasPrefix(res.Placement.Fabric, NVLinkFabric) {
+		t.Fatalf("placement %+v\n%s", res.Placement, res.Explain())
+	}
+
+	// With no InfiniBand to fall back on, the refusal says why.
+	a2 := inDomain(node("nv1", "h100-80g", 4), "7")
+	b2 := inDomain(node("nv2", "h100-80g", 4), "7")
+	b2.IMEX.State = "inactive"
+	res = New(store(a2, b2)).Place([]Profile{{Name: "tp8", GPUClass: "h100-80g", GPUsPerNode: 4, Nodes: 2}}, nil)
+	if res.Placement != nil || !strings.Contains(res.Explain(), "nvidia-imex is not up") {
+		t.Errorf("the explanation should say why the domain was passed over:\n%s", res.Explain())
+	}
+}

internal/control/placement/topology.go

@@ -0,0 +1,135 @@
+package placement
+
+import (
+	"slices"
+	"sync"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/topology"
+)
+
+// fabricHints is what a fabric's map says about placing on it: the leaf
+// switch each node's first adapter hangs off, and which nodes have a cable
+// that trained below the rest.
+type fabricHints struct {
+	leaf     map[string]string
+	degraded map[string]bool
+}
+
+// hintCache keeps a parsed map until its fabric is scanned again; parsing
+// thousands of lines on every placement would be the scheduler's main cost.
+var hintCache struct {
+	sync.Mutex
+	at    map[string]time.Time
+	hints map[string]fabricHints
+}
+
+// hints reads every scanned fabric. Nodes on no mapped fabric are absent,
+// which leaves them placed as before.
+func hints(inv *inventory.Store) fabricHints {
+	out := fabricHints{leaf: map[string]string{}, degraded: map[string]bool{}}
+	scans := inv.Scans()
+	if len(scans) == 0 {
+		return out
+	}
+	var hosts map[string]topology.Host
+	hintCache.Lock()
+	defer hintCache.Unlock()
+	if hintCache.at == nil {
+		hintCache.at, hintCache.hints = map[string]time.Time{}, map[string]fabricHints{}
+	}
+	for key, sc := range scans {
+		if sc.Output == "" {
+			continue
+		}
+		h, ok := hintCache.hints[key]
+		if !ok || !hintCache.at[key].Equal(sc.At) {
+			if hosts == nil {
+				hosts = map[string]topology.Host{}
+				for _, n := range inv.Nodes() {
+					for _, p := range n.IBPorts {
+						if p.NodeGUID != "" {
+							hosts[p.NodeGUID] = topology.Host{Node: n.ID, HCA: p.HCA}
+						}
+					}
+				}
+			}
+			f, err := topology.Parse(sc.Output)
+			if err != nil {
+				continue
+			}
+			v := topology.Build(f, hosts)
+			h = fabricHints{leaf: map[string]string{}, degraded: map[string]bool{}}
+			guidNode := map[string]string{}
+			for _, hv := range v.Hosts {
+				if hv.Node == "" {
+					continue
+				}
+				guidNode[hv.GUID] = hv.Node
+				if hv.Rail == 0 {
+					h.leaf[hv.Node] = hv.Leaf
+				}
+			}
+			for _, l := range v.Links {
+				if l.Host && l.Degraded {
+					if n := guidNode[l.To]; n != "" {
+						h.degraded[n] = true
+					}
+				}
+			}
+			hintCache.hints[key], hintCache.at[key] = h, sc.At
+		}
+		for n, l := range h.leaf {
+			out.leaf[n] = l
+		}
+		for n := range h.degraded {
+			out.degraded[n] = true
+		}
+	}
+	return out
+}
+
+// byLeaf orders candidates for a replica spanning nodes: those whose cables
+// all trained at their rate first, and among them the nodes of one leaf
+// together — the leaf with the most of them first — so a replica's traffic
+// stays under one switch when it can, and crosses the spines only when it
+// must. Within a leaf the order given (emptiest first) is kept.
+func byLeaf(cands []candidate, h fabricHints) []candidate {
+	if len(h.leaf) == 0 && len(h.degraded) == 0 {
+		return cands
+	}
+	count := map[string]int{}
+	for _, c := range cands {
+		if !h.degraded[c.node.ID] {
+			count[h.leaf[c.node.ID]]++
+		}
+	}
+	firstSeen := map[string]int{}
+	for i, c := range cands {
+		if _, ok := firstSeen[h.leaf[c.node.ID]]; !ok {
+			firstSeen[h.leaf[c.node.ID]] = i
+		}
+	}
+	out := make([]candidate, len(cands))
+	copy(out, cands)
+	rank := func(c candidate) (bad int, size int, first int) {
+		if h.degraded[c.node.ID] {
+			bad = 1
+		}
+		l := h.leaf[c.node.ID]
+		return bad, count[l], firstSeen[l]
+	}
+	slices.SortStableFunc(out, func(a, b candidate) int {
+		ba, sa, fa := rank(a)
+		bb, sb, fb := rank(b)
+		if ba != bb {
+			return ba - bb
+		}
+		if sa != sb {
+			return sb - sa
+		}
+		return fa - fb
+	})
+	return out
+}

internal/control/topology/topology.go

@@ -0,0 +1,390 @@
+// Package topology reads ibnetdiscover's map of a fabric and works out its
+// shape: which switches are leaves and which spines, which adapter belongs to
+// which node, which links trained below the rest, and whether the rails line
+// up.
+package topology
+
+import (
+	"fmt"
+	"regexp"
+	"sort"
+	"strconv"
+	"strings"
+)
+
+// Device is a switch, an adapter or a router on the fabric.
+type Device struct {
+	GUID  string `json:"guid"`
+	Kind  string `json:"kind"` // switch, ca, router
+	Desc  string `json:"desc"`
+	Ports int    `json:"ports"`
+	LID   int    `json:"lid,omitempty"`
+}
+
+// End is one side of a link.
+type End struct {
+	GUID string `json:"guid"`
+	Port int    `json:"port"`
+}
+
+// Link is one cable, once.
+type Link struct {
+	A     End    `json:"a"`
+	B     End    `json:"b"`
+	Width string `json:"width"` // 4x
+	Speed string `json:"speed"` // HDR
+	Gbps  int    `json:"gbps"`
+}
+
+// Fabric is a parsed ibnetdiscover.
+type Fabric struct {
+	Devices map[string]*Device `json:"devices"`
+	Links   []Link             `json:"links"`
+}
+
+var (
+	header = regexp.MustCompile(`^(Switch|Ca|Rt)\s+(\d+)\s+"([SHR])-([0-9a-fA-F]+)"\s*(?:#\s*"([^"]*)"(.*))?$`)
+	portLn = regexp.MustCompile(`^\[(\d+)\](?:\([0-9a-fA-F]+\))?\s+"([SHR])-([0-9a-fA-F]+)"\[(\d+)\](?:\([0-9a-fA-F]+\))?\s*(?:#\s*(.*))?$`)
+	lidRe  = regexp.MustCompile(`\blid (\d+)`)
+	rateRe = regexp.MustCompile(`\b(\d+)x([A-Z]+[0-9]*)\s*$`)
+)
+
+// laneGbps is the signalling rate of one lane at each speed, as the link
+// reports it.
+var laneGbps = map[string]float64{
+	"SDR": 2.5, "DDR": 5, "QDR": 10, "FDR10": 10, "FDR": 14, "EDR": 25, "HDR": 50, "NDR": 100, "XDR": 200,
+}
+
+// Parse reads ibnetdiscover's default output.
+func Parse(out string) (*Fabric, error) {
+	f := &Fabric{Devices: map[string]*Device{}}
+	seen := map[string]bool{}
+	var cur *Device
+	for _, raw := range strings.Split(out, "\n") {
+		line := strings.TrimSpace(raw)
+		if line == "" || strings.HasPrefix(line, "#") {
+			continue
+		}
+		if m := header.FindStringSubmatch(line); m != nil {
+			kind := map[string]string{"Switch": "switch", "Ca": "ca", "Rt": "router"}[m[1]]
+			n, _ := strconv.Atoi(m[2])
+			d := &Device{GUID: guid(m[4]), Kind: kind, Ports: n, Desc: m[5]}
+			if l := lidRe.FindStringSubmatch(m[6]); l != nil {
+				d.LID, _ = strconv.Atoi(l[1])
+			}
+			if old, ok := f.Devices[d.GUID]; ok && old.Desc != "" && d.Desc == "" {
+				d.Desc = old.Desc
+			}
+			f.Devices[d.GUID] = d
+			cur = d
+			continue
+		}
+		m := portLn.FindStringSubmatch(line)
+		if m == nil || cur == nil {
+			continue
+		}
+		port, _ := strconv.Atoi(m[1])
+		peer := guid(m[3])
+		peerPort, _ := strconv.Atoi(m[4])
+		comment := m[5]
+		// The peer's own block may come later; its description is in the
+		// comment, quoted, which is enough to name it until then.
+		if _, ok := f.Devices[peer]; !ok {
+			kind := map[string]string{"S": "switch", "H": "ca", "R": "router"}[m[2]]
+			d := &Device{GUID: peer, Kind: kind}
+			if q := strings.Index(comment, `"`); q >= 0 {
+				if e := strings.Index(comment[q+1:], `"`); e >= 0 {
+					d.Desc = comment[q+1 : q+1+e]
+				}
+			}
+			f.Devices[peer] = d
+		}
+		a, b := End{cur.GUID, port}, End{peer, peerPort}
+		key := linkKey(a, b)
+		if seen[key] {
+			continue
+		}
+		seen[key] = true
+		l := Link{A: a, B: b}
+		if r := rateRe.FindStringSubmatch(comment); r != nil {
+			l.Width, l.Speed = r[1]+"x", r[2]
+			lanes, _ := strconv.Atoi(r[1])
+			l.Gbps = int(float64(lanes) * laneGbps[r[2]])
+		}
+		f.Links = append(f.Links, l)
+	}
+	if len(f.Devices) == 0 {
+		return nil, fmt.Errorf("no devices in ibnetdiscover output")
+	}
+	return f, nil
+}
+
+func guid(hex string) string {
+	hex = strings.ToLower(hex)
+	for len(hex) < 16 {
+		hex = "0" + hex
+	}
+	return "0x" + hex
+}
+
+func linkKey(a, b End) string {
+	x, y := fmt.Sprintf("%s/%d", a.GUID, a.Port), fmt.Sprintf("%s/%d", b.GUID, b.Port)
+	if x > y {
+		x, y = y, x
+	}
+	return x + "-" + y
+}
+
+// Host is an adapter the cluster knows: which node, which card.
+type Host struct {
+	Node string
+	HCA  string
+}
+
+// View is a fabric's shape, ready to draw.
+type View struct {
+	Switches []SwitchView `json:"switches"`
+	Hosts    []HostView   `json:"hosts"`
+	Links    []LinkView   `json:"links"`
+	// Tiers is how many switch levels: 1 for a single switch, 2 for
+	// leaf and spine, 3 with a core.
+	Tiers int `json:"tiers"`
+	// Expected is the rate most links of each kind trained at; a link below
+	// it is degraded.
+	ExpectedHost   int `json:"expected_host_gbps"`
+	ExpectedSwitch int `json:"expected_switch_gbps"`
+	Degraded       int `json:"degraded"`
+	// Rails is, per rail — an adapter's position on its node — the leaves it
+	// reaches, and Aligned whether each rail has its own.
+	Rails   []Rail   `json:"rails"`
+	Aligned bool     `json:"rail_aligned"`
+	Notes   []string `json:"notes,omitempty"`
+}
+
+// SwitchView is a switch and its place in the tiers.
+type SwitchView struct {
+	GUID  string `json:"guid"`
+	Name  string `json:"name"`
+	Tier  int    `json:"tier"` // 1 leaf, 2 spine, 3 core
+	Ports int    `json:"ports"`
+	Used  int    `json:"used"`
+	LID   int    `json:"lid,omitempty"`
+	// Hosts counts adapters hanging off it.
+	Hosts int `json:"hosts"`
+}
+
+// HostView is an adapter, and the node it is in when the cluster knows it.
+type HostView struct {
+	GUID string `json:"guid"`
+	Desc string `json:"desc"`
+	Node string `json:"node,omitempty"`
+	HCA  string `json:"hca,omitempty"`
+	Rail int    `json:"rail"`
+	Leaf string `json:"leaf,omitempty"`
+}
+
+// LinkView is a cable, with what it trained at and whether that is short of
+// its kind.
+type LinkView struct {
+	From     string `json:"from"`
+	FromPort int    `json:"from_port"`
+	To       string `json:"to"`
+	ToPort   int    `json:"to_port"`
+	Rate     string `json:"rate"`
+	Gbps     int    `json:"gbps"`
+	Host     bool   `json:"host"`
+	Degraded bool   `json:"degraded"`
+}
+
+// Rail is the leaves one rail's adapters reach.
+type Rail struct {
+	Index  int      `json:"index"`
+	Leaves []string `json:"leaves"`
+}
+
+// Build works out the shape. hosts maps adapter GUIDs to the cluster's nodes.
+func Build(f *Fabric, hosts map[string]Host) View {
+	var v View
+	adj := map[string][]string{}
+	for _, l := range f.Links {
+		adj[l.A.GUID] = append(adj[l.A.GUID], l.B.GUID)
+		adj[l.B.GUID] = append(adj[l.B.GUID], l.A.GUID)
+	}
+	// A switch's tier is its distance from the nearest adapter: leaves hold
+	// adapters, spines hold leaves, cores hold spines.
+	tier := map[string]int{}
+	var frontier []string
+	for g, d := range f.Devices {
+		if d.Kind == "ca" {
+			tier[g] = 0
+			frontier = append(frontier, g)
+		}
+	}
+	for len(frontier) > 0 {
+		var next []string
+		for _, g := range frontier {
+			for _, p := range adj[g] {
+				if _, ok := tier[p]; ok || f.Devices[p] == nil || f.Devices[p].Kind == "ca" {
+					continue
+				}
+				tier[p] = tier[g] + 1
+				next = append(next, p)
+			}
+		}
+		frontier = next
+	}
+
+	name := func(g string) string {
+		d := f.Devices[g]
+		if d == nil || d.Desc == "" {
+			return g
+		}
+		desc := d.Desc
+		// Mellanox switches say "MF0;leaf-01:MSB7700/U1": the name is between.
+		if i := strings.Index(desc, ";"); i >= 0 {
+			desc = desc[i+1:]
+			if j := strings.Index(desc, ":"); j >= 0 {
+				desc = desc[:j]
+			}
+		}
+		return desc
+	}
+
+	// Expected rate per kind of link: the most common.
+	hostRates, swRates := map[int]int{}, map[int]int{}
+	isHost := func(l Link) bool {
+		return f.Devices[l.A.GUID].Kind == "ca" || f.Devices[l.B.GUID].Kind == "ca"
+	}
+	for _, l := range f.Links {
+		if isHost(l) {
+			hostRates[l.Gbps]++
+		} else {
+			swRates[l.Gbps]++
+		}
+	}
+	v.ExpectedHost, v.ExpectedSwitch = mode(hostRates), mode(swRates)
+
+	used := map[string]int{}
+	hostsOn := map[string]int{}
+	leafOf := map[string]string{}
+	for _, l := range f.Links {
+		used[l.A.GUID]++
+		used[l.B.GUID]++
+		lv := LinkView{From: l.A.GUID, FromPort: l.A.Port, To: l.B.GUID, ToPort: l.B.Port,
+			Rate: l.Width + l.Speed, Gbps: l.Gbps, Host: isHost(l)}
+		// Keep adapters on the To side, so a drawing can go down the tiers.
+		if f.Devices[lv.From].Kind == "ca" {
+			lv.From, lv.FromPort, lv.To, lv.ToPort = lv.To, lv.ToPort, lv.From, lv.FromPort
+		}
+		if lv.Host {
+			hostsOn[lv.From]++
+			leafOf[lv.To] = lv.From
+			lv.Degraded = l.Gbps > 0 && l.Gbps < v.ExpectedHost
+		} else {
+			lv.Degraded = l.Gbps > 0 && l.Gbps < v.ExpectedSwitch
+			if tier[lv.From] < tier[lv.To] {
+				lv.From, lv.FromPort, lv.To, lv.ToPort = lv.To, lv.ToPort, lv.From, lv.FromPort
+			}
+		}
+		if lv.Degraded {
+			v.Degraded++
+		}
+		v.Links = append(v.Links, lv)
+	}
+
+	for g, d := range f.Devices {
+		switch d.Kind {
+		case "switch":
+			t := tier[g]
+			if t == 0 {
+				t = 1 // a switch with nothing below it is drawn with the leaves
+			}
+			v.Switches = append(v.Switches, SwitchView{GUID: g, Name: name(g), Tier: t, Ports: d.Ports,
+				Used: used[g], LID: d.LID, Hosts: hostsOn[g]})
+			v.Tiers = max(v.Tiers, t)
+		case "ca":
+			h := HostView{GUID: g, Desc: d.Desc, Leaf: leafOf[g]}
+			if known, ok := hosts[g]; ok {
+				h.Node, h.HCA = known.Node, known.HCA
+			}
+			v.Hosts = append(v.Hosts, h)
+		}
+	}
+	sort.Slice(v.Switches, func(i, j int) bool {
+		a, b := v.Switches[i], v.Switches[j]
+		if a.Tier != b.Tier {
+			return a.Tier > b.Tier
+		}
+		return a.Name < b.Name
+	})
+
+	// Rails: an adapter's position among its node's adapters on this fabric.
+	byNode := map[string][]int{}
+	for i, h := range v.Hosts {
+		key := h.Node
+		if key == "" {
+			key = strings.Fields(h.Desc + " ?")[0] // "node01 mlx5_0" names its host
+		}
+		byNode[key] = append(byNode[key], i)
+	}
+	railLeaves := map[int]map[string]bool{}
+	for _, idx := range byNode {
+		sort.Slice(idx, func(a, b int) bool {
+			return hcaName(v.Hosts[idx[a]]) < hcaName(v.Hosts[idx[b]])
+		})
+		for r, i := range idx {
+			v.Hosts[i].Rail = r
+			if railLeaves[r] == nil {
+				railLeaves[r] = map[string]bool{}
+			}
+			if v.Hosts[i].Leaf != "" {
+				railLeaves[r][v.Hosts[i].Leaf] = true
+			}
+		}
+	}
+	sort.Slice(v.Hosts, func(i, j int) bool {
+		a, b := v.Hosts[i], v.Hosts[j]
+		if a.Leaf != b.Leaf {
+			return a.Leaf < b.Leaf
+		}
+		return a.Desc < b.Desc
+	})
+	owner := map[string]int{}
+	v.Aligned = len(railLeaves) > 1
+	for r := 0; r < len(railLeaves); r++ {
+		var leaves []string
+		for l := range railLeaves[r] {
+			leaves = append(leaves, name(l))
+			if prev, ok := owner[l]; ok && prev != r {
+				v.Aligned = false
+			}
+			owner[l] = r
+		}
+		sort.Strings(leaves)
+		v.Rails = append(v.Rails, Rail{Index: r, Leaves: leaves})
+	}
+	if v.Degraded > 0 {
+		v.Notes = append(v.Notes, fmt.Sprintf("%d link(s) trained below the rest of their kind", v.Degraded))
+	}
+	return v
+}
+
+func hcaName(h HostView) string {
+	if h.HCA != "" {
+		return h.HCA
+	}
+	return h.Desc
+}
+
+func mode(counts map[int]int) int {
+	best, n := 0, -1
+	for v, c := range counts {
+		if v == 0 {
+			continue
+		}
+		if c > n || (c == n && v > best) {
+			best, n = v, c
+		}
+	}
+	return best
+}

internal/control/topology/topology_test.go

@@ -0,0 +1,149 @@
+package topology
+
+import (
+	"fmt"
+	"strings"
+	"testing"
+)
+
+// fatTree writes ibnetdiscover's default output for two spines, two leaves
+// and four nodes of two adapters each, rail-aligned: every node's mlx5_0 on
+// leaf-01 and mlx5_1 on leaf-02. node03's mlx5_1 trained at 2x instead of 4x.
+func fatTree() string {
+	var b strings.Builder
+	w := func(format string, a ...any) { fmt.Fprintf(&b, format+"\n", a...) }
+	w("#\n# Topology file: generated on Mon Sep 29 10:00:00 2026\n#\n# Initiated from node 0000000000000a01 port 0000000000000a01\n")
+	spine := []string{"00000000000000c1", "00000000000000c2"}
+	leaf := []string{"00000000000000b1", "00000000000000b2"}
+	hca := func(n, r int) string { return fmt.Sprintf("000000000000%d%02d", r+10, n) }
+	for si, s := range spine {
+		w("vendid=0x2c9\ndevid=0xd2f0\nsysimgguid=0x%s\nswitchguid=0x%s(%s)", s, s, s)
+		w("Switch\t64 \"S-%s\"\t\t# \"MF0;spine-0%d:MQM9700/U1\" enhanced port 0 lid %d lmc 0", s, si+1, 10+si)
+		for li, l := range leaf {
+			for k := 0; k < 2; k++ {
+				w("[%d]\t\"S-%s\"[%d]\t\t# \"MF0;leaf-0%d:MQM9700/U1\" lid %d 4xNDR", li*2+k+1, l, 33+si*2+k, li+1, 20+li)
+			}
+		}
+		w("")
+	}
+	for li, l := range leaf {
+		w("vendid=0x2c9\ndevid=0xd2f0\nsysimgguid=0x%s\nswitchguid=0x%s(%s)", l, l, l)
+		w("Switch\t64 \"S-%s\"\t\t# \"MF0;leaf-0%d:MQM9700/U1\" enhanced port 0 lid %d lmc 0", l, li+1, 20+li)
+		for n := 1; n <= 4; n++ {
+			rate := "4xNDR"
+			if n == 3 && li == 1 {
+				rate = "2xNDR"
+			}
+			h := hca(n, li)
+			w("[%d]\t\"H-%s\"[1](%s) \t\t# \"node0%d mlx5_%d\" lid %d %s", n, h, h, n, li, 100+n*2+li, rate)
+		}
+		for si, s := range spine {
+			for k := 0; k < 2; k++ {
+				w("[%d]\t\"S-%s\"[%d]\t\t# \"MF0;spine-0%d:MQM9700/U1\" lid %d 4xNDR", 33+si*2+k, s, li*2+k+1, si+1, 10+si)
+			}
+		}
+		w("")
+	}
+	for n := 1; n <= 4; n++ {
+		for li, l := range leaf {
+			h := hca(n, li)
+			rate := "4xNDR"
+			if n == 3 && li == 1 {
+				rate = "2xNDR"
+			}
+			w("vendid=0x2c9\ndevid=0x1021\nsysimgguid=0x%s\ncaguid=0x%s", h, h)
+			w("Ca\t1 \"H-%s\"\t\t# \"node0%d mlx5_%d\"", h, n, li)
+			w("[1](%s) \t\"S-%s\"[%d]\t\t# lid %d lmc 0 \"MF0;leaf-0%d:MQM9700/U1\" lid %d %s", h, l, n, 100+n*2+li, li+1, 20+li, rate)
+			w("")
+		}
+	}
+	return b.String()
+}
+
+func TestAFatTreeIsReadIntoTiers(t *testing.T) {
+	f, err := Parse(fatTree())
+	if err != nil {
+		t.Fatal(err)
+	}
+	// 2 spines + 2 leaves + 8 adapters; 8 spine-leaf cables + 8 host cables,
+	// each once though both ends list it.
+	if len(f.Devices) != 12 || len(f.Links) != 16 {
+		t.Fatalf("%d devices, %d links", len(f.Devices), len(f.Links))
+	}
+	hosts := map[string]Host{}
+	for n := 1; n <= 4; n++ {
+		for r := 0; r < 2; r++ {
+			hosts[fmt.Sprintf("0x000000000000%d%02d", r+10, n)] = Host{Node: fmt.Sprintf("n%d", n), HCA: fmt.Sprintf("mlx5_%d", r)}
+		}
+	}
+	v := Build(f, hosts)
+	if v.Tiers != 2 {
+		t.Errorf("tiers = %d", v.Tiers)
+	}
+	tiers := map[string]int{}
+	for _, s := range v.Switches {
+		tiers[s.Name] = s.Tier
+	}
+	if tiers["spine-01"] != 2 || tiers["leaf-02"] != 1 {
+		t.Errorf("tiers = %v", tiers)
+	}
+	if v.ExpectedHost != 400 || v.ExpectedSwitch != 400 {
+		t.Errorf("expected rates %d / %d", v.ExpectedHost, v.ExpectedSwitch)
+	}
+	// node03's second adapter trained at half width.
+	if v.Degraded != 1 {
+		t.Errorf("degraded = %d", v.Degraded)
+	}
+	for _, l := range v.Links {
+		if l.Degraded && (l.Rate != "2xNDR" || !l.Host) {
+			t.Errorf("wrong link degraded: %+v", l)
+		}
+	}
+	// Every node's first adapter on leaf-01, its second on leaf-02: aligned.
+	if !v.Aligned || len(v.Rails) != 2 || v.Rails[0].Leaves[0] != "leaf-01" || v.Rails[1].Leaves[0] != "leaf-02" {
+		t.Errorf("rails = %+v aligned=%v", v.Rails, v.Aligned)
+	}
+	mapped := 0
+	for _, h := range v.Hosts {
+		if h.Node != "" {
+			mapped++
+		}
+	}
+	if mapped != 8 {
+		t.Errorf("%d adapters mapped to nodes", mapped)
+	}
+}
+
+func TestCrossedRailsAreNotAligned(t *testing.T) {
+	// node02's two cables swapped at the leaves: its mlx5_0 on leaf-02 and
+	// its mlx5_1 on leaf-01, so each leaf now carries both rails.
+	f, err := Parse(fatTree())
+	if err != nil {
+		t.Fatal(err)
+	}
+	swap := map[string]string{"0x0000000000001002": "0x00000000000000b2", "0x0000000000001102": "0x00000000000000b1"}
+	for i, l := range f.Links {
+		for _, host := range []*End{&f.Links[i].A, &f.Links[i].B} {
+			if leaf, ok := swap[host.GUID]; ok {
+				other := &f.Links[i].A
+				if host == other {
+					other = &f.Links[i].B
+				}
+				other.GUID = leaf
+			}
+		}
+		_ = l
+	}
+	v := Build(f, map[string]Host{
+		"0x0000000000001002": {Node: "n2", HCA: "mlx5_0"}, "0x0000000000001102": {Node: "n2", HCA: "mlx5_1"},
+	})
+	if v.Aligned {
+		t.Errorf("crossed rails read as aligned: %+v", v.Rails)
+	}
+}
+
+func TestNotAMapIsAnError(t *testing.T) {
+	if _, err := Parse("ibwarn: [1234] mad_rpc_open_port: can't open UMAD port ((null):0)"); err == nil {
+		t.Error("expected an error")
+	}
+}

internal/doctor/doctor.go

@@ -0,0 +1,659 @@
+// Package doctor checks a node for what llmgw needs of its GPUs and fabric,
+// and says what is wrong in words an operator can act on.
+//
+// Written for a machine nobody here can log into. The report is meant to be
+// run on the node, read there, and sent back when it is not enough: every
+// fact it concluded from is in it, so the conclusion can be checked.
+package doctor
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"net"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"slices"
+	"sort"
+	"strings"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
+)
+
+// Level is how much a finding matters.
+type Level string
+
+// The levels, most serious first.
+const (
+	Fail Level = "FAIL"
+	Warn Level = "WARN"
+	OK   Level = "OK"
+	Info Level = "INFO"
+)
+
+// Finding is one conclusion, and what to do about it.
+type Finding struct {
+	Level   Level  `json:"level"`
+	Area    string `json:"area"`
+	Message string `json:"message"`
+	Fix     string `json:"fix,omitempty"`
+}
+
+// Facts are what was read from the node. Every finding is drawn from these
+// and nothing else, so a report sent back can be re-read here.
+type Facts struct {
+	At       time.Time      `json:"at"`
+	Hostname string         `json:"hostname"`
+	Host     discovery.Host `json:"host"`
+
+	SMIFound bool            `json:"nvidia_smi_found"`
+	GPUs     []discovery.GPU `json:"gpus"`
+	GPUError string          `json:"gpu_error,omitempty"`
+	TopoRaw  string          `json:"topo_raw,omitempty"`
+	TopoErr  string          `json:"topo_error,omitempty"`
+	// NVSwitch is set when the machine has NVSwitch devices, which need the
+	// fabric manager running before CUDA will start.
+	NVSwitch bool `json:"nvswitch"`
+	// IMEX is the service multi-node NVLink needs.
+	IMEX          discovery.IMEX `json:"imex"`
+	FabricManager string         `json:"fabric_manager,omitempty"`
+
+	HCAs []discovery.HCA `json:"hcas"`
+	// SMErrors say, per port, why the subnet manager could not be asked.
+	SMErrors map[string]string `json:"sm_errors,omitempty"`
+	// SMLists are every subnet manager on each fabric, by master GUID;
+	// SMListErrors why a fabric's could not be listed.
+	SMLists      map[string][]discovery.SMRecord `json:"sm_lists,omitempty"`
+	SMListErrors map[string]string               `json:"sm_list_errors,omitempty"`
+	Modules      map[string]bool                 `json:"modules"`
+
+	// ConfigPath is the agent configuration read, empty when there was none.
+	ConfigPath  string                       `json:"config_path,omitempty"`
+	ConfigErr   string                       `json:"config_error,omitempty"`
+	Fabrics     []discovery.FabricCheck      `json:"fabrics,omitempty"`
+	MgmtAddr    string                       `json:"mgmt_addr,omitempty"`
+	MgmtIface   string                       `json:"mgmt_interface,omitempty"`
+	IfaceExists bool                         `json:"mgmt_interface_exists"`
+	LaunchMode  string                       `json:"vllm_launch_mode,omitempty"`
+	NCCL        map[string]map[string]string `json:"nccl_env,omitempty"`
+	UnitMemlock string                       `json:"agent_unit_memlock,omitempty"`
+	Engines     map[string]string            `json:"engines"`
+
+	// What installing vLLM here would meet: the version this llmgw installs,
+	// where, the space there, and whether it already has.
+	VLLMVersion   string  `json:"vllm_version,omitempty"`
+	EnginesDir    string  `json:"engines_dir,omitempty"`
+	EnginesFreeGB float64 `json:"engines_free_gb,omitempty"`
+	VLLMInstalled bool    `json:"vllm_installed"`
+}
+
+// Modules checked, and why each matters.
+var watchedModules = []string{"ib_core", "ib_uverbs", "mlx5_ib", "rdma_ucm", "nvidia_peermem", "nv_peer_mem"}
+
+// Options says where to look.
+type Options struct {
+	ConfigPath string
+	IBRoot     string
+	NvidiaSMI  string
+	// VLLMVersion is the vLLM this llmgw installs; its prerequisites are
+	// checked when set.
+	VLLMVersion string
+}
+
+// Collect reads the node.
+func Collect(ctx context.Context, o Options) Facts {
+	f := Facts{At: time.Now().UTC(), Modules: map[string]bool{}, Engines: map[string]string{}}
+	f.Hostname, _ = os.Hostname()
+	f.Host = discovery.DiscoverHost()
+
+	smi := o.NvidiaSMI
+	if smi == "" {
+		if p, err := exec.LookPath("nvidia-smi"); err == nil {
+			smi = p
+		}
+	}
+	if smi != "" {
+		f.SMIFound = true
+		gpus, err := discovery.DiscoverGPUsSMI(ctx, smi)
+		if err != nil {
+			f.GPUError = err.Error()
+		}
+		out, err := exec.CommandContext(ctx, smi, "topo", "-m").Output()
+		if err != nil {
+			f.TopoErr = err.Error()
+		} else {
+			f.TopoRaw = string(out)
+			if t, err := discovery.ParseTopology(f.TopoRaw); err == nil {
+				t.Apply(gpus)
+			} else {
+				f.TopoErr = err.Error()
+			}
+		}
+		if fab, err := discovery.DiscoverGPUFabric(ctx, smi); err == nil {
+			discovery.ApplyFabric(gpus, fab)
+		}
+		f.GPUs = gpus
+		f.IMEX = discovery.DiscoverIMEX(ctx)
+	}
+	if m, _ := filepath.Glob("/dev/nvidia-nvswitch*"); len(m) > 0 {
+		f.NVSwitch = true
+		f.FabricManager = systemctl(ctx, "is-active", "nvidia-fabricmanager")
+	}
+
+	root := o.IBRoot
+	if root == "" {
+		root = discovery.SysfsRoot
+	}
+	f.HCAs, _ = discovery.DiscoverIB(root)
+	f.SMErrors = map[string]string{}
+	_ = discovery.AttachSM(f.HCAs, func(hca string, port int) (*discovery.SMInfo, error) {
+		sm, err := discovery.QuerySM(ctx, hca, port)
+		if err != nil {
+			f.SMErrors[fmt.Sprintf("%s/%d", hca, port)] = err.Error()
+		}
+		return sm, err
+	})
+	f.SMLists, f.SMListErrors = map[string][]discovery.SMRecord{}, map[string]string{}
+	for _, h := range f.HCAs {
+		for _, p := range h.Ports {
+			if p.SM == nil {
+				continue
+			}
+			if _, done := f.SMLists[p.SM.GUID]; done {
+				continue
+			}
+			if _, done := f.SMListErrors[p.SM.GUID]; done {
+				continue
+			}
+			if recs, err := discovery.QuerySMs(ctx, h.Name, p.Num); err != nil {
+				f.SMListErrors[p.SM.GUID] = err.Error()
+			} else {
+				f.SMLists[p.SM.GUID] = recs
+			}
+		}
+	}
+	for _, m := range watchedModules {
+		_, err := os.Stat(filepath.Join("/sys/module", m))
+		f.Modules[m] = err == nil
+	}
+
+	f.readConfig(o)
+	f.Host.MinComputeCapability = discovery.MinComputeCapability(f.GPUs)
+	if f.VLLMVersion = o.VLLMVersion; f.VLLMVersion != "" {
+		if f.EnginesDir == "" {
+			f.EnginesDir = provision.DefaultRoot
+		}
+		// The directory may not exist yet: the space is the filesystem's.
+		for dir := f.EnginesDir; dir != "/" && dir != "."; dir = filepath.Dir(dir) {
+			if gb, err := provision.FreeGB(dir); err == nil {
+				f.EnginesFreeGB = gb
+				break
+			}
+		}
+		matches, _ := filepath.Glob(filepath.Join(f.EnginesDir, "vllm", f.VLLMVersion+"-*", ".llmgw-installed"))
+		f.VLLMInstalled = len(matches) > 0
+	}
+	f.UnitMemlock = strings.TrimPrefix(systemctl(ctx, "show", "llmgw-agent", "-p", "LimitMEMLOCK"), "LimitMEMLOCK=")
+
+	for name, bin := range map[string]string{"vllm": "vllm", "ray": "ray", "sglang": "python3"} {
+		if name == "sglang" {
+			continue // no binary of its own to ask
+		}
+		if p, err := exec.LookPath(bin); err == nil {
+			out, _ := exec.CommandContext(ctx, p, "--version").CombinedOutput()
+			f.Engines[name] = firstLine(string(out))
+		}
+	}
+	return f
+}
+
+func (f *Facts) readConfig(o Options) {
+	path := o.ConfigPath
+	if path == "" {
+		if _, err := os.Stat("/etc/llmgw/agent.yaml"); err != nil {
+			return
+		}
+		path = "/etc/llmgw/agent.yaml"
+	}
+	f.ConfigPath = path
+	cfg, err := loadAgent(path)
+	if err != nil {
+		f.ConfigErr = err.Error()
+		return
+	}
+	declared := map[string][]string{}
+	for _, fa := range cfg.Fabrics {
+		declared[fa.Fabric] = fa.HCAs
+	}
+	f.Fabrics = discovery.CheckFabrics(declared, f.HCAs)
+	f.MgmtAddr, f.MgmtIface = agent.ResolveManagement(cfg)
+	if f.MgmtIface == "" {
+		f.MgmtIface = "eth0"
+	}
+	_, err = net.InterfaceByName(f.MgmtIface)
+	f.IfaceExists = err == nil
+	f.EnginesDir = cfg.EnginesDir
+	f.LaunchMode = cfg.Engines.VLLM.LaunchMode
+	if f.LaunchMode == "" {
+		f.LaunchMode = "mp"
+	}
+
+	// What a two-node shard on each fabric would be given, the part of an
+	// engine's environment that decides whether NCCL finds its peers.
+	f.NCCL = map[string]map[string]string{}
+	for _, c := range f.Fabrics {
+		if !c.OK() {
+			continue
+		}
+		spec := engines.ShardSpec{NNodes: 2, HCAs: c.Usable, MgmtInterface: f.MgmtIface, GIDIndex: rocev2(f.HCAs, c.Usable)}
+		env := engines.NCCLEnv(spec)
+		delete(env, "CUDA_VISIBLE_DEVICES")
+		f.NCCL[c.Fabric] = env
+	}
+}
+
+// loadAgent is replaced in tests; the real one lives in config, which this
+// package cannot import without a cycle through the CLI.
+var loadAgent = func(string) (agent.Config, error) {
+	return agent.Config{}, fmt.Errorf("no agent configuration loader")
+}
+
+// SetConfigLoader gives Collect the configuration loader.
+func SetConfigLoader(fn func(string) (agent.Config, error)) { loadAgent = fn }
+
+func rocev2(hcas []discovery.HCA, usable []string) *int {
+	for _, h := range hcas {
+		if !slices.Contains(usable, h.Name) {
+			continue
+		}
+		for _, p := range h.Ports {
+			if p.Active() && p.RoCEv2GID >= 0 {
+				v := p.RoCEv2GID
+				return &v
+			}
+		}
+	}
+	return nil
+}
+
+func systemctl(ctx context.Context, args ...string) string {
+	out, err := exec.CommandContext(ctx, "systemctl", args...).Output()
+	if err != nil && len(out) == 0 {
+		return "unknown"
+	}
+	return strings.TrimSpace(string(out))
+}
+
+func firstLine(s string) string {
+	s = strings.TrimSpace(s)
+	if i := strings.IndexByte(s, '\n'); i >= 0 {
+		s = s[:i]
+	}
+	return s
+}
+
+// Evaluate draws the findings from the facts, most serious first.
+func Evaluate(f Facts) []Finding {
+	var out []Finding
+	add := func(l Level, area, msg, fix string) {
+		out = append(out, Finding{Level: l, Area: area, Message: msg, Fix: fix})
+	}
+
+	// --- CPU ---
+	// Read from /proc/cpuinfo, so only on a node whose OS was read too;
+	// elsewhere an empty list means unknown, not missing.
+	if f.Host.Arch == "amd64" && f.Host.OSID != "" {
+		if f.Host.HasCPUFlag("avx2") {
+			add(OK, "cpu", "vector extensions: "+strings.Join(f.Host.CPUFlags, ", "), "")
+		} else {
+			has := "none of AVX, AVX2 or FMA"
+			if len(f.Host.CPUFlags) > 0 {
+				has = "only " + strings.Join(f.Host.CPUFlags, ", ")
+			}
+			// Nothing else looks wrong on such a node: it registers, answers
+			// correctly, and is several times slower than the hardware.
+			add(Warn, "cpu", "this CPU offers no AVX2 ("+has+"): llama.cpp runs several times slower on it",
+				"on a VM, give it the host's CPU type (Proxmox: qm set <vmid> --cpu host, then restart the VM)")
+		}
+	}
+
+	// --- GPUs ---
+	switch {
+	case !f.SMIFound:
+		add(Info, "gpu", "no nvidia-smi: a CPU node, or the driver is not installed", "")
+	case f.GPUError != "":
+		add(Fail, "gpu", "nvidia-smi fails: "+f.GPUError, "check the driver: run nvidia-smi by hand")
+	case len(f.GPUs) == 0:
+		add(Warn, "gpu", "nvidia-smi is installed but reports no GPUs", "")
+	default:
+		add(OK, "gpu", fmt.Sprintf("%d GPUs: %s", len(f.GPUs), gpuNames(f.GPUs)), "")
+		for _, g := range f.GPUs {
+			if !g.Health.OK() {
+				add(Warn, "gpu", fmt.Sprintf("GPU %d is kept out of placement: %s", g.Index, strings.Join(g.Health.Problems, "; ")), "")
+			}
+		}
+	}
+	if f.NVSwitch && f.FabricManager != "active" {
+		add(Fail, "nvlink", "this machine has NVSwitch but nvidia-fabricmanager is "+f.FabricManager+
+			": CUDA will not initialise on its GPUs",
+			"systemctl enable --now nvidia-fabricmanager (its version must match the driver's)")
+	}
+	if len(f.GPUs) > 1 {
+		pairs, groups := nvlinkGroups(f.GPUs)
+		switch {
+		case f.TopoErr != "":
+			add(Warn, "nvlink", "could not read nvidia-smi topo -m: "+f.TopoErr+
+				"; tensor-parallel groups will be chosen without NVLink", "")
+		case pairs == 0 && f.NVSwitch:
+			add(Warn, "nvlink", "NVSwitch is present but no GPU reports an NVLink peer", "check nvidia-smi nvlink --status")
+		case pairs == 0:
+			add(Info, "nvlink", "no NVLink: GPUs talk over PCIe, and tensor parallelism across them is slow", "")
+		default:
+			add(OK, "nvlink", "NVLink groups: "+groups, "")
+		}
+	}
+
+	// --- multi-node NVLink ---
+	domain := ""
+	for _, g := range f.GPUs {
+		fb := g.Fabric
+		if fb == nil || strings.EqualFold(fb.State, "Not Supported") {
+			continue
+		}
+		if d := fb.Domain(); d != "" {
+			domain = d
+			if fb.Bandwidth != "" && !strings.EqualFold(fb.Bandwidth, "Full") {
+				add(Warn, "nvlink", fmt.Sprintf("GPU %d's NVLink bandwidth is %s", g.Index, strings.ToLower(fb.Bandwidth)),
+					"check its NVLink cables and trays: nvidia-smi -q -d FABRIC")
+			}
+			continue
+		}
+		if !strings.EqualFold(fb.State, "Completed") {
+			why := fb.State
+			if fb.Status != "" && !strings.EqualFold(fb.Status, "Success") {
+				why += ": " + fb.Status
+			}
+			add(Fail, "nvlink", fmt.Sprintf("GPU %d is not joined to its NVLink domain (%s)", g.Index, why),
+				"the fabric manager joins it: check nvidia-fabricmanager and the NVLink switch trays")
+		}
+	}
+	if domain != "" {
+		add(OK, "nvlink", "the GPUs are in multi-node NVLink domain "+domain, "")
+		switch {
+		case f.IMEX.State != "active":
+			state := f.IMEX.State
+			if state == "" {
+				state = "not installed"
+			}
+			add(Fail, "nvlink", "nvidia-imex is "+state+": without it the domain's GPUs cannot share memory across nodes",
+				"systemctl enable --now nvidia-imex, with every node of the domain in /etc/nvidia-imex/nodes_config.cfg")
+		case f.IMEX.Channels == 0:
+			add(Fail, "nvlink", "nvidia-imex runs but no IMEX channel exists for an engine to open",
+				"load the driver with options nvidia NVreg_CreateImexChannel0=1")
+		default:
+			add(OK, "nvlink", fmt.Sprintf("nvidia-imex is active with %d channel(s)", f.IMEX.Channels), "")
+		}
+	}
+
+	// --- InfiniBand and RoCE ---
+	if len(f.HCAs) == 0 {
+		add(Info, "fabric", "no InfiniBand or RoCE adapters: this node can only run single-node replicas", "")
+	}
+	declared := map[string]string{} // hca -> fabric
+	for _, c := range f.Fabrics {
+		for _, h := range c.Declared {
+			declared[h] = c.Fabric
+		}
+	}
+	for _, h := range f.HCAs {
+		for _, p := range h.Ports {
+			where := fmt.Sprintf("%s/%d", h.Name, p.Num)
+			level := Warn
+			if declared[h.Name] != "" {
+				level = Fail
+			}
+			switch {
+			case p.Active() && p.LinkLayer == discovery.LinkEthernet && p.RoCEv2GID < 0:
+				add(Fail, "fabric", where+" is RoCE with no RoCE v2 GID entry: NCCL would use a v1 entry and hang",
+					"give the interface an IPv4 address, or check the adapter's RoCE mode")
+			case p.Active():
+				on := ""
+				switch {
+				case p.SM != nil:
+					on = fmt.Sprintf(", subnet manager %s (LID %d, %s)", p.SM.GUID, p.SM.LID, p.SM.State)
+				case p.Network != "":
+					on = ", network " + p.Network
+				}
+				add(OK, "fabric", fmt.Sprintf("%s %s at %d Gb/s (%s)%s", where, p.State, p.RateGbps, p.LinkLayer, on), "")
+				if p.LinkLayer == discovery.LinkInfiniBand && p.SM == nil {
+					why := f.SMErrors[where]
+					fix := ""
+					if strings.Contains(why, "infiniband-diags") {
+						fix = "install infiniband-diags"
+					}
+					add(Warn, "fabric", where+": which fabric it is on cannot be told — "+orUnknown(why)+
+						"; without it, separate fabrics given one name cannot be caught", fix)
+				}
+				if p.LinkLayer == discovery.LinkInfiniBand && p.SMLID == 0 {
+					add(Warn, "fabric", where+" is active but reports no subnet manager LID", "")
+				}
+			case strings.EqualFold(p.PhysState, "LinkUp"):
+				add(level, "fabric", where+" has a link but is "+p.State+": no subnet manager has configured it",
+					"run a subnet manager on the fabric (opensm, or the switch's own)")
+			default:
+				add(level, "fabric", fmt.Sprintf("%s is %s (%s): no link", where, p.State, p.PhysState),
+					"check the cable and the switch port")
+			}
+			for _, name := range []string{"symbol_error", "link_downed", "link_error_recovery", "port_rcv_errors", "local_link_integrity_errors"} {
+				if v := p.Counters[name]; v > 0 {
+					add(Warn, "fabric", fmt.Sprintf("%s: %s = %d since the adapter came up", where, name, v),
+						"a count that keeps growing is a cable or transceiver going bad")
+				}
+			}
+		}
+	}
+	for _, master := range sortedKeys(f.SMLists) {
+		recs := f.SMLists[master]
+		var parts []string
+		standby := 0
+		for _, r := range recs {
+			parts = append(parts, fmt.Sprintf("%s (%s, priority %d)", r.GUID, r.State, r.Priority))
+			if r.State == "standby" {
+				standby++
+			}
+		}
+		if standby == 0 {
+			add(Warn, "fabric", "the fabric of subnet manager "+master+" has no standby: "+strings.Join(parts, ", "),
+				"run a second subnet manager, on a switch or a host, at a lower priority")
+		} else {
+			add(OK, "fabric", "subnet managers of "+master+"'s fabric: "+strings.Join(parts, ", "), "")
+		}
+	}
+	for _, master := range sortedKeys(f.SMListErrors) {
+		add(Info, "fabric", "the subnet managers of "+master+"'s fabric could not be listed: "+f.SMListErrors[master], "")
+	}
+	if len(f.HCAs) > 0 {
+		if !f.Modules["ib_uverbs"] {
+			add(Fail, "fabric", "ib_uverbs is not loaded: nothing in user space can use RDMA", "modprobe ib_uverbs")
+		}
+		if len(f.GPUs) > 0 && !f.Modules["nvidia_peermem"] && !f.Modules["nv_peer_mem"] {
+			add(Warn, "fabric", "nvidia_peermem is not loaded: GPUDirect RDMA is off, and NCCL copies every transfer through host memory",
+				"modprobe nvidia_peermem")
+		}
+	}
+
+	// --- the agent's view ---
+	switch {
+	case f.ConfigErr != "":
+		add(Fail, "agent", "the agent configuration "+f.ConfigPath+" does not load: "+f.ConfigErr, "")
+	case f.ConfigPath == "":
+		add(Info, "agent", "no agent configuration found; declared fabrics were not checked", "pass --config")
+	default:
+		for _, c := range f.Fabrics {
+			if c.OK() {
+				add(OK, "agent", fmt.Sprintf("fabric %s: %s usable at %d Gb/s", c.Fabric, strings.Join(c.Usable, ","), c.RateGbps), "")
+			}
+			for _, p := range c.Problems {
+				lvl := Warn
+				if !c.OK() {
+					lvl = Fail
+				}
+				add(lvl, "agent", "fabric "+c.Fabric+": "+p, "")
+			}
+			if !c.OK() {
+				add(Fail, "agent", "fabric "+c.Fabric+" will not be advertised: no multi-node replica will be placed here", "")
+			}
+		}
+		active := activeHCAs(f.HCAs)
+		if len(f.Fabrics) == 0 && len(active) > 0 {
+			add(Warn, "agent", "adapters are up ("+strings.Join(active, ", ")+") but agent.yaml declares no fabric: this node will never get multi-node work",
+				"add fabrics: [{name: <fabric>, hcas: ["+strings.Join(active, ", ")+"]}]")
+		}
+		if !f.IfaceExists {
+			add(Fail, "agent", "the management interface "+f.MgmtIface+" does not exist: NCCL and Gloo would bootstrap over nothing",
+				"set mgmt_interface in agent.yaml")
+		} else {
+			add(OK, "agent", "NCCL bootstraps over "+f.MgmtIface+" ("+f.MgmtAddr+")", "")
+		}
+		if len(f.Fabrics) > 0 && f.LaunchMode == "ray" && f.Engines["ray"] == "" && f.Engines["vllm"] != "" {
+			add(Fail, "agent", "vLLM's launch_mode is ray but ray is not installed", "install ray, or set engines.vllm.launch_mode: mp")
+		}
+	}
+	// --- engines ---
+	if f.VLLMVersion != "" {
+		vllmFindings(f, add)
+	}
+
+	if f.UnitMemlock != "" && f.UnitMemlock != "unknown" && f.UnitMemlock != "infinity" && len(f.HCAs) > 0 {
+		add(Warn, "agent", "the llmgw-agent unit limits locked memory to "+f.UnitMemlock+"; RDMA pins what it registers",
+			"LimitMEMLOCK=infinity in the unit")
+	}
+
+	sort.SliceStable(out, func(i, j int) bool { return rank(out[i].Level) < rank(out[j].Level) })
+	return out
+}
+
+// vllmFindings says whether the agent can install vLLM here, and if not why:
+// the same decision the agent makes, so the report and the node agree.
+func vllmFindings(f Facts, add func(Level, string, string, string)) {
+	v := f.VLLMVersion
+	b, err := provision.Recipes["vllm"].Choose(v, f.Host)
+	var nh provision.NotHere
+	switch {
+	case errors.As(err, &nh):
+		add(Info, "engines", "vLLM will not be installed: "+nh.Reason, "")
+		return
+	case err != nil:
+		add(Fail, "engines", "vLLM "+v+" cannot be installed: "+err.Error(),
+			"upgrade the NVIDIA driver, or set the control plane's engines.vllm to a version for this one")
+		return
+	case f.VLLMInstalled:
+		add(OK, "engines", "vLLM "+v+" is installed under "+f.EnginesDir, "")
+		return
+	}
+	if f.EnginesFreeGB > 0 && f.EnginesFreeGB < float64(b.FreeGB) {
+		add(Fail, "engines", fmt.Sprintf("vLLM %s needs about %d GB free under %s and there are %.1f GB",
+			v, b.FreeGB, f.EnginesDir, f.EnginesFreeGB),
+			"free space there, or point engines_dir in agent.yaml at a larger disk")
+		return
+	}
+	add(OK, "engines", fmt.Sprintf("vLLM %s (%s) can be installed here: driver %s, %.1f GB free under %s",
+		v, b.Variant, f.Host.NVIDIADriver, f.EnginesFreeGB, f.EnginesDir), "")
+}
+
+func rank(l Level) int {
+	switch l {
+	case Fail:
+		return 0
+	case Warn:
+		return 1
+	case OK:
+		return 2
+	}
+	return 3
+}
+
+func gpuNames(gpus []discovery.GPU) string {
+	count := map[string]int{}
+	var order []string
+	for _, g := range gpus {
+		if count[g.Name] == 0 {
+			order = append(order, g.Name)
+		}
+		count[g.Name]++
+	}
+	parts := make([]string, len(order))
+	for i, n := range order {
+		parts[i] = fmt.Sprintf("%d× %s", count[n], n)
+	}
+	return strings.Join(parts, ", ")
+}
+
+// nvlinkGroups counts NVLink pairs and names the connected groups.
+func nvlinkGroups(gpus []discovery.GPU) (int, string) {
+	parent := map[int]int{}
+	var find func(int) int
+	find = func(x int) int {
+		if p, ok := parent[x]; ok && p != x {
+			parent[x] = find(p)
+			return parent[x]
+		}
+		parent[x] = x
+		return x
+	}
+	pairs := 0
+	for _, g := range gpus {
+		find(g.Index)
+		for _, p := range g.NVLinkPeers {
+			if p > g.Index {
+				pairs++
+			}
+			parent[find(g.Index)] = find(p)
+		}
+	}
+	groups := map[int][]int{}
+	for _, g := range gpus {
+		r := find(g.Index)
+		groups[r] = append(groups[r], g.Index)
+	}
+	var names []string
+	for _, members := range groups {
+		sort.Ints(members)
+		if len(members) > 1 {
+			names = append(names, fmt.Sprint(members))
+		}
+	}
+	sort.Strings(names)
+	return pairs, strings.Join(names, " ")
+}
+
+func activeHCAs(hcas []discovery.HCA) []string {
+	var out []string
+	for _, h := range hcas {
+		if len(h.ActivePorts()) > 0 {
+			out = append(out, h.Name)
+		}
+	}
+	return out
+}
+
+func orUnknown(s string) string {
+	if s == "" {
+		return "no answer from the subnet manager"
+	}
+	return s
+}
+
+func sortedKeys[V any](m map[string]V) []string {
+	out := make([]string, 0, len(m))
+	for k := range m {
+		out = append(out, k)
+	}
+	sort.Strings(out)
+	return out
+}

internal/doctor/doctor_test.go

@@ -0,0 +1,202 @@
+package doctor
+
+import (
+	"strings"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+)
+
+// healthy is an eight-GPU NVSwitch node on one InfiniBand fabric, set up
+// right.
+func healthy() Facts {
+	f := Facts{SMIFound: true, NVSwitch: true, FabricManager: "active",
+		Modules:    map[string]bool{"ib_uverbs": true, "nvidia_peermem": true},
+		ConfigPath: "/etc/llmgw/agent.yaml", MgmtIface: "ens3", MgmtAddr: "10.1.0.5", IfaceExists: true,
+		LaunchMode: "mp", UnitMemlock: "infinity", Engines: map[string]string{"vllm": "0.11.0"}}
+	for i := range 8 {
+		g := discovery.GPU{Index: i, Name: "NVIDIA H100 80GB HBM3"}
+		for j := range 8 {
+			if j != i {
+				g.NVLinkPeers = append(g.NVLinkPeers, j)
+			}
+		}
+		f.GPUs = append(f.GPUs, g)
+	}
+	f.HCAs = []discovery.HCA{{Name: "mlx5_0", Ports: []discovery.Port{{Num: 1, State: "ACTIVE",
+		PhysState: "LinkUp", RateGbps: 400, LinkLayer: discovery.LinkInfiniBand, SMLID: 1, RoCEv2GID: -1,
+		SM: &discovery.SMInfo{GUID: "0x98039b03009fcfd6", LID: 1, State: "SMINFO_MASTER"}}}}}
+	f.Fabrics = discovery.CheckFabrics(map[string][]string{"ib-a": {"mlx5_0"}}, f.HCAs)
+	f.SMLists = map[string][]discovery.SMRecord{"0x98039b03009fcfd6": {
+		{GUID: "0x98039b03009fcfd6", State: "master", Priority: 15}, {GUID: "0x98039b03009fd000", State: "standby", Priority: 14}}}
+	return f
+}
+
+func levels(fs []Finding, l Level) []string {
+	var out []string
+	for _, f := range fs {
+		if f.Level == l {
+			out = append(out, f.Message)
+		}
+	}
+	return out
+}
+
+func has(t *testing.T, fs []Finding, l Level, fragment string) {
+	t.Helper()
+	for _, f := range fs {
+		if f.Level == l && strings.Contains(f.Message, fragment) {
+			return
+		}
+	}
+	t.Errorf("no %s finding containing %q in %+v", l, fragment, fs)
+}
+
+func TestAHealthyNodeHasNothingToFix(t *testing.T) {
+	fs := Evaluate(healthy())
+	if bad := append(levels(fs, Fail), levels(fs, Warn)...); len(bad) > 0 {
+		t.Errorf("a healthy node was faulted: %v", bad)
+	}
+	has(t, fs, OK, "NVLink groups: [0 1 2 3 4 5 6 7]")
+	has(t, fs, OK, "fabric ib-a: mlx5_0 usable at 400 Gb/s")
+	has(t, fs, OK, "NCCL bootstraps over ens3")
+	has(t, fs, OK, "subnet manager 0x98039b03009fcfd6")
+}
+
+func TestTheMistakesThatStopAFabricNodeAreNamed(t *testing.T) {
+	cases := map[string]struct {
+		change func(*Facts)
+		level  Level
+		says   string
+	}{
+		"fabric manager down": {func(f *Facts) { f.FabricManager = "inactive" }, Fail, "nvidia-fabricmanager is inactive"},
+		"no subnet manager": {func(f *Facts) {
+			f.HCAs[0].Ports[0].State = "INIT"
+			f.Fabrics = discovery.CheckFabrics(map[string][]string{"ib-a": {"mlx5_0"}}, f.HCAs)
+		}, Fail, "no subnet manager has configured it"},
+		"cable out": {func(f *Facts) {
+			f.HCAs[0].Ports[0].State, f.HCAs[0].Ports[0].PhysState = "DOWN", "Polling"
+			f.Fabrics = discovery.CheckFabrics(map[string][]string{"ib-a": {"mlx5_0"}}, f.HCAs)
+		}, Fail, "no link"},
+		"fabric not advertised": {func(f *Facts) {
+			f.HCAs[0].Ports[0].State = "DOWN"
+			f.Fabrics = discovery.CheckFabrics(map[string][]string{"ib-a": {"mlx5_0"}}, f.HCAs)
+		}, Fail, "will not be advertised"},
+		"roce without v2": {func(f *Facts) {
+			f.HCAs[0].Ports[0].LinkLayer = discovery.LinkEthernet
+		}, Fail, "no RoCE v2 GID"},
+		"no peermem":        {func(f *Facts) { f.Modules["nvidia_peermem"] = false }, Warn, "GPUDirect RDMA is off"},
+		"no uverbs":         {func(f *Facts) { f.Modules["ib_uverbs"] = false }, Fail, "ib_uverbs"},
+		"nothing declared":  {func(f *Facts) { f.Fabrics = nil }, Warn, "declares no fabric"},
+		"wrong interface":   {func(f *Facts) { f.IfaceExists = false }, Fail, "does not exist"},
+		"errors on a cable": {func(f *Facts) { f.HCAs[0].Ports[0].Counters = map[string]uint64{"symbol_error": 40} }, Warn, "symbol_error = 40"},
+		"small memlock":     {func(f *Facts) { f.UnitMemlock = "8388608" }, Warn, "limits locked memory"},
+		"ray missing":       {func(f *Facts) { f.LaunchMode = "ray" }, Fail, "ray is not installed"},
+		"no nvlink on nvswitch": {func(f *Facts) {
+			for i := range f.GPUs {
+				f.GPUs[i].NVLinkPeers = nil
+			}
+		}, Warn, "no GPU reports an NVLink peer"},
+	}
+	for name, c := range cases {
+		t.Run(name, func(t *testing.T) {
+			f := healthy()
+			c.change(&f)
+			fs := Evaluate(f)
+			has(t, fs, c.level, c.says)
+			// Most serious first, so the thing to fix is at the top.
+			if fs[0].Level != Fail && len(levels(fs, Fail)) > 0 {
+				t.Error("a failure is not first")
+			}
+		})
+	}
+}
+
+func TestANodeOutsideTheFabricIsAnAdapterLeftAlone(t *testing.T) {
+	// An adapter no declared fabric names being down is worth a warning, not
+	// a failure: nothing here was going to use it.
+	f := healthy()
+	f.HCAs = append(f.HCAs, discovery.HCA{Name: "mlx5_4", Ports: []discovery.Port{{Num: 1, State: "DOWN", PhysState: "Disabled", RoCEv2GID: -1}}})
+	fs := Evaluate(f)
+	has(t, fs, Warn, "mlx5_4/1 is DOWN")
+	if len(levels(fs, Fail)) > 0 {
+		t.Errorf("failures: %v", levels(fs, Fail))
+	}
+}
+
+func TestAnNVLinkDomainNeedsEveryGPUJoinedAndIMEX(t *testing.T) {
+	f := healthy()
+	for i := range f.GPUs {
+		f.GPUs[i].Fabric = &discovery.GPUFabric{State: "Completed", Status: "Success",
+			ClusterUUID: "76b1ad76-7c24-4b6b-9f6c-2a1e0d8c11aa", CliqueID: "1", Bandwidth: "Full"}
+	}
+	f.IMEX = discovery.IMEX{State: "active", Channels: 1}
+	fs := Evaluate(f)
+	has(t, fs, OK, "multi-node NVLink domain")
+	has(t, fs, OK, "nvidia-imex is active")
+
+	f.GPUs[5].Fabric = &discovery.GPUFabric{State: "In Progress"}
+	f.IMEX.Channels = 0
+	fs = Evaluate(f)
+	has(t, fs, Fail, "GPU 5 is not joined to its NVLink domain (In Progress)")
+	has(t, fs, Fail, "no IMEX channel")
+}
+
+func TestDoctorSaysWhetherVLLMCanBeInstalled(t *testing.T) {
+	gpu := discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 9.0}
+	cases := []struct {
+		name  string
+		f     Facts
+		level Level
+		says  string
+	}{
+		{"CPU node", Facts{Host: discovery.Host{Arch: "amd64"}, VLLMVersion: "0.30.0"}, Info, "has none"},
+		{"old driver", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15"}, VLLMVersion: "0.30.0"}, Fail, "driver 580"},
+		{"small disk", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 9}, Fail, "9.0 GB"},
+		{"ready to install", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "can be installed"},
+		{"installed", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", VLLMInstalled: true}, OK, "is installed"},
+	}
+	for _, c := range cases {
+		t.Run(c.name, func(t *testing.T) {
+			var found bool
+			for _, fd := range Evaluate(c.f) {
+				if fd.Area == "engines" {
+					found = true
+					if fd.Level != c.level || !strings.Contains(fd.Message, c.says) {
+						t.Errorf("got %s %q, want %s saying %q", fd.Level, fd.Message, c.level, c.says)
+					}
+				}
+			}
+			if !found {
+				t.Error("no finding about engines")
+			}
+		})
+	}
+}
+
+func TestDoctorSaysWhenTheCPUHidesAVX2(t *testing.T) {
+	area := func(fs []Finding, a string) *Finding {
+		for i := range fs {
+			if fs[i].Area == a {
+				return &fs[i]
+			}
+		}
+		return nil
+	}
+	// A VM with QEMU's default CPU model: nothing looks wrong but the speed.
+	f := area(Evaluate(Facts{Host: discovery.Host{OSID: "ubuntu", Arch: "amd64"}}), "cpu")
+	if f == nil || f.Level != Warn || !strings.Contains(f.Fix, "--cpu host") {
+		t.Errorf("no AVX2: %+v", f)
+	}
+	f = area(Evaluate(Facts{Host: discovery.Host{OSID: "ubuntu", Arch: "amd64", CPUFlags: []string{"avx", "avx2", "fma"}}}), "cpu")
+	if f == nil || f.Level != OK {
+		t.Errorf("with AVX2: %+v", f)
+	}
+	// Not read at all, or not x86: nothing to say.
+	if f := area(Evaluate(Facts{Host: discovery.Host{Arch: "amd64"}}), "cpu"); f != nil {
+		t.Errorf("unknown CPU: %+v", f)
+	}
+	if f := area(Evaluate(Facts{Host: discovery.Host{OSID: "ubuntu", Arch: "arm64"}}), "cpu"); f != nil {
+		t.Errorf("ARM: %+v", f)
+	}
+}

internal/gguf/gguf.go

@@ -0,0 +1,213 @@
+// Package gguf reads what a GGUF file says about itself, from its header.
+//
+// Enough to ask one question before anything is downloaded or served: can the
+// engine load this? A file downloaded, catalogued and placed, and then refused
+// by llama.cpp with "unknown model architecture", costs a download and a
+// replica failing on every node it is tried on — and the answer was in the
+// first few kilobytes all along.
+package gguf
+
+import (
+	"bufio"
+	"context"
+	"encoding/binary"
+	"errors"
+	"fmt"
+	"io"
+	"math"
+	"net/http"
+	"strings"
+	"time"
+)
+
+// Header is the part of a GGUF header this package reads.
+type Header struct {
+	Version      uint32
+	Architecture string
+	Name         string
+}
+
+// ErrNotGGUF means the bytes are not a GGUF file.
+var ErrNotGGUF = errors.New("not a GGUF file")
+
+// Value types, from the GGUF specification.
+const (
+	tUint8 = iota
+	tInt8
+	tUint16
+	tInt16
+	tUint32
+	tInt32
+	tFloat32
+	tBool
+	tString
+	tArray
+	tUint64
+	tInt64
+	tFloat64
+)
+
+// Read parses a header up to general.architecture and general.name.
+func Read(r io.Reader) (Header, error) {
+	br := bufio.NewReader(r)
+	var h Header
+	magic := make([]byte, 4)
+	if _, err := io.ReadFull(br, magic); err != nil || string(magic) != "GGUF" {
+		return h, ErrNotGGUF
+	}
+	if err := binary.Read(br, binary.LittleEndian, &h.Version); err != nil {
+		return h, err
+	}
+	if h.Version < 1 || h.Version > 3 {
+		return h, fmt.Errorf("GGUF version %d is not one this reads", h.Version)
+	}
+	wide := h.Version >= 2 // v1 used 32-bit counts and lengths
+	count := func() (uint64, error) {
+		if wide {
+			var n uint64
+			return n, binary.Read(br, binary.LittleEndian, &n)
+		}
+		var n uint32
+		err := binary.Read(br, binary.LittleEndian, &n)
+		return uint64(n), err
+	}
+	str := func() (string, error) {
+		n, err := count()
+		if err != nil {
+			return "", err
+		}
+		if n > 1<<20 {
+			return "", fmt.Errorf("a %d-byte string in a GGUF header", n)
+		}
+		b := make([]byte, n)
+		_, err = io.ReadFull(br, b)
+		return string(b), err
+	}
+	skip := func(n int64) error {
+		_, err := io.CopyN(io.Discard, br, n)
+		return err
+	}
+	var value func(t uint32) (string, error)
+	value = func(t uint32) (string, error) {
+		switch t {
+		case tUint8, tInt8, tBool:
+			return "", skip(1)
+		case tUint16, tInt16:
+			return "", skip(2)
+		case tUint32, tInt32, tFloat32:
+			return "", skip(4)
+		case tUint64, tInt64, tFloat64:
+			return "", skip(8)
+		case tString:
+			return str()
+		case tArray:
+			var et uint32
+			if err := binary.Read(br, binary.LittleEndian, &et); err != nil {
+				return "", err
+			}
+			n, err := count()
+			if err != nil {
+				return "", err
+			}
+			if n > math.MaxInt32 {
+				return "", fmt.Errorf("a %d-element array in a GGUF header", n)
+			}
+			for i := uint64(0); i < n; i++ {
+				if _, err := value(et); err != nil {
+					return "", err
+				}
+			}
+			return "", nil
+		}
+		return "", fmt.Errorf("GGUF value type %d is unknown", t)
+	}
+
+	if _, err := count(); err != nil { // tensors
+		return h, err
+	}
+	kvs, err := count()
+	if err != nil {
+		return h, err
+	}
+	for i := uint64(0); i < kvs; i++ {
+		key, err := str()
+		if err != nil {
+			return h, err
+		}
+		var t uint32
+		if err := binary.Read(br, binary.LittleEndian, &t); err != nil {
+			return h, err
+		}
+		v, err := value(t)
+		if err != nil {
+			return h, err
+		}
+		switch key {
+		case "general.architecture":
+			h.Architecture = v
+		case "general.name":
+			h.Name = v
+		}
+		if h.Architecture != "" && h.Name != "" {
+			break
+		}
+	}
+	if h.Architecture == "" {
+		return h, errors.New("the GGUF header names no architecture")
+	}
+	return h, nil
+}
+
+// headerBytes is how much of a file is fetched to read its header. The
+// architecture is among the first keys; the tokenizer's arrays, which can be
+// megabytes, come after it.
+const headerBytes = 8 << 20
+
+// FromURL reads a GGUF header over HTTP, fetching only the start of the file.
+func FromURL(ctx context.Context, client *http.Client, url string) (Header, error) {
+	if client == nil {
+		client = &http.Client{Timeout: 30 * time.Second}
+	}
+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
+	if err != nil {
+		return Header{}, err
+	}
+	req.Header.Set("Range", fmt.Sprintf("bytes=0-%d", headerBytes-1))
+	resp, err := client.Do(req)
+	if err != nil {
+		return Header{}, err
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusPartialContent {
+		return Header{}, fmt.Errorf("%s answered %s", url, resp.Status)
+	}
+	return Read(io.LimitReader(resp.Body, headerBytes))
+}
+
+// IsGGUF reports whether a source names a GGUF file, or a directory of them.
+func IsGGUF(source string) bool {
+	s := strings.ToLower(strings.TrimSuffix(source, "/"))
+	return strings.HasSuffix(s, ".gguf")
+}
+
+// Check reads a source's header and judges it against a llama.cpp build.
+//
+// Only a definite answer refuses: a file that is not GGUF, or whose
+// architecture the build cannot load. A header that cannot be fetched — the
+// network, a directory of shards — is not a reason to refuse, so the verdict
+// then says it was not checked.
+func Check(ctx context.Context, url, llamaVersion string) Verdict {
+	if strings.HasSuffix(url, "/") {
+		return Verdict{Servable: true, Reason: "a set of files; its header was not read"}
+	}
+	ctx, cancel := context.WithTimeout(ctx, 20*time.Second)
+	defer cancel()
+	h, err := FromURL(ctx, nil, url)
+	if errors.Is(err, ErrNotGGUF) {
+		return Verdict{Checked: true, Reason: "it is not a GGUF file"}
+	}
+	if err != nil {
+		return Verdict{Servable: true, Reason: "its header could not be read: " + err.Error()}
+	}
+	return LlamaCpp(llamaVersion, h.Architecture)
+}