Merge ha: replicated store, second control plane, join codes, setup and console screens

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

CHANGELOG.md

@@ -15,11 +15,518 @@ is only released once its **exit criterion** is met, not when its code lands.
 | v0.8.0 | — | PostgreSQL, leader election, agent failover | released |
 | v0.9.0 | — | Managing people and the tenancy tree | released |
 | v0.10.0 | — | InfiniBand fabrics and NVLink, doctor, node add, engines installed by the nodes | released |
+| v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released |
 | v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
 
 Until v1.0.0 the minor version is the breaking one: configuration formats and Go
 APIs change between minors without a deprecation period.
 
+## v0.11.0 — 2026-10-03
+
+A minor version: the store can be replicated and the control plane doubled,
+both set up by llmgw itself; `sudo llmgw` opens setup screens and a console;
+usage is read from sums kept as it is written. Released without candidates.
+
+**Upgrading.** Migrations 0015 (usage sums, filled from the request table
+already recorded — once, at the first start) and 0016 (control planes) run by
+themselves. An agent older than this ignores the list of control planes and
+keeps its configured one. Nothing in `server.yaml` or `agent.yaml` changes
+meaning; `auth.usage_retention` and `router.tenant_label` are new and optional.
+
+The GPU and fabric work below began from the field report in issue #2, after
+a deployment on GPU nodes.
+
+### GPU metrics that move, and charts of them
+
+A GPU's utilisation and memory in use were read once, when the agent
+started, and published from then on: a card busy since looked as idle as it
+was at registration, and one idle since as busy. They are now sampled with
+the temperatures and power every heartbeat (ten seconds at most), with the
+memory bandwidth in use, the SM clock and uncorrected ECC errors besides.
+An agent too old to send them still has its registration figures published,
+and a figure the driver does not give is left out rather than shown as zero.
+
+- **New series**: `llmgw_gpu_memory_utilization_percent`,
+  `llmgw_gpu_sm_clock_mhz`; `llmgw_gpu_utilization_percent`,
+  `llmgw_gpu_memory_used_bytes` and `llmgw_gpu_ecc_errors` now from the
+  sample.
+- **The node page** shows each card's utilisation, memory, clock and ECC
+  errors in its table, and charts of utilisation, memory, temperature and
+  power, one line per card.
+- **The cluster page** charts the fleet's GPUs by class: average utilisation
+  and memory in use.
+- **Grafana** (`deploy/monitoring/grafana/llmgw-overview.json`) has a GPU row:
+  utilisation, memory, memory bandwidth, temperature, power against the
+  limit, clock, and what holds cards back.
+- **Alerts**: `GPUThermalThrottling` (ten minutes slowed by heat),
+  `GPUUncorrectedECCErrors`, `GPUHot` (above 87 °C for ten minutes).
+- **The cluster page's GPU map**: every card a square, a row per node,
+  coloured by how busy, how full or how hot it is (`?gpumap=util|memory|temp`),
+  with what holds it and its figures under the pointer; free cards dashed.
+- **Cards held and doing nothing**: the replicas whose every card reads under
+  5% busy, with when they were last asked — the first place to find room.
+- **What each model gets from its cards**, on the cluster page and in
+  Grafana: output tokens a second per GPU and per watt-hour, with the cards'
+  draw and average utilisation, over the last quarter hour.
+- **What wears a card out**, read every minute: XID errors from the kernel's
+  log (counted from when the agent started), memory rows remapped or pages
+  retired and whether either waits for a reset, a card out of spare rows,
+  PCIe link retries and NVLink errors. Published as `llmgw_gpu_xid_errors`,
+  `llmgw_gpu_last_xid`, `llmgw_gpu_remapped_rows`, `llmgw_gpu_remap_pending`,
+  `llmgw_gpu_remap_failure`, `llmgw_gpu_retired_pages`,
+  `llmgw_gpu_retirement_pending`, `llmgw_gpu_pcie_replays`,
+  `llmgw_gpu_nvlink_errors`; said in words on the node page; a red outline
+  on the GPU map; a Grafana panel; and alerts — `GPUXidError`,
+  `GPUOutOfSpareRows`, `GPUNeedsReset`, `GPUPCIeRetriesClimbing`,
+  `GPUNVLinkErrorsClimbing`.
+- **GPU time by owner** on the usage page: the cards each owner's models
+  held over the window, charted and summed into GPU-hours, whether or not the
+  models were asked anything — the time a GPU quota is about. From the
+  metrics store, as far back as it keeps history.
+- **`llmgw_gpu_assigned{node,gpu,model,replica,owner}`**: which replica holds
+  each card and whose quota it counts against, for joining the GPU series to
+  models and tenants.
+
+### Usage is read from sums kept as it is written
+
+Every chart and total — the usage tab, quotas, the dashboard, a person's own
+page, and budgets restored at start — summed the request table again on each
+view: a row per request, scanned whole. Usage is now also summed as it is
+written, in the same transaction, into five-minute slices, hours and days per
+key, person, project and model (`usage_5m`, `usage_1h`, `usage_1d`; migration
+0015 fills them from what is already recorded).
+
+- **Charts** read the coarsest of the three their slices are made of: a day in
+  20-minute slices from the five-minute table, a week or a month from hours.
+- **Totals and budgets** read the rows up to the next five-minute slice, then
+  slices, hours and days — exact, not rounded to a slice. On three million
+  requests over thirty days, a month's total per person went from 705 ms to
+  18 ms on PostgreSQL.
+- **Latency and time to first token** are kept as histograms in the sums, so a
+  percentile per team or model can be read after the rows are gone.
+- **`auth.usage_retention`** (e.g. `2160h`) deletes request records older than
+  that, hourly. Totals, charts and budgets are unchanged by it — they read the
+  sums — but `ctl request` finds nothing older. Unset keeps every record, as
+  before.
+
+### Alerts per tenant and per budget
+
+- **`llmgw_tenant_requests_total{tenant,outcome}`**, `llmgw_tenant_tokens_total`
+  and `llmgw_tenant_ttft_seconds`: requests by outcome (ok, error, limited),
+  tokens and time to first token per tenant. The tenant is the level
+  `router.tenant_label` names — `team` by default, or `org`, `project`, or
+  `none` — named by its path, "acme/platform". One level rather than every
+  project, so a cluster of thousands of projects does not become a Prometheus
+  of millions of series.
+- **`llmgw_budget_spent_ratio{scope,id,name,period}`**: this period's spend
+  over the budget for every organization, team, project and person given one,
+  refreshed every minute.
+- **Alerts** in `deploy/monitoring/llmgw-rules.yml`: `BudgetMostlySpent` (80%),
+  `BudgetSpent`, `TenantHittingLimits` (a tenth of requests refused),
+  `TenantErrors`, `TenantTrafficSpike` (three times the weekly average) and
+  `TenantSlowFirstToken`.
+- **A Grafana dashboard per tenant**, `deploy/monitoring/grafana/llmgw-tenants.json`
+  (uid `llmgw-tenants`): requests and tokens by tenant, the share failing and
+  refused by a limit, first-token p95, every budget's share spent, and who used
+  most in the last day — with a tenant chooser at the top.
+
+### How each team's requests fared, not only what they cost
+
+- **Percentiles from the sums.** First token and latency, p50, p95 and p99,
+  per organization, team, project, person or model, read from the histograms
+  kept with the usage sums: on the usage tab's table, and in a branch's or
+  person's panel on the people page, beside the error rate and requests
+  refused by a limit.
+- **Refused requests are counted against their owner.** A request a rate,
+  token or budget limit refused was counted only in Prometheus, by model; a
+  team running into its limit looked like a team that had gone quiet. It is
+  now recorded as a 429 against its key, person and project — no tokens, no
+  cost — and shown as "Limited".
+- **Usage by model** is a grouping of the usage tab, by the model that
+  answered.
+### The console watches the cluster and holds every setting
+
+`sudo llmgw` on a control plane is a console in three groups down the left:
+
+- **Watch**, refreshed every three seconds: the overview; nodes, with CPU,
+  memory, GPUs free, disk and agent version, and a node's detail — load,
+  engines, each card's busy, memory, temperature and power; every GPU in the
+  fleet and which model holds it; models, with a model's profiles, warnings
+  and replicas; replicas and where they run; the last day's use by team,
+  project or organization; and a node's agent log.
+- **Settings**: every key of `server.yaml`, block by block, with what applies
+  when the file sets nothing and what each key is for, from its comment in
+  the code. Enter changes a key or opens a block; lists and maps of blocks
+  (endpoints, models, prices) take entries added and removed. A save is read
+  by the same loader the service uses first — nothing is written that would
+  not load — keeps the file's comments, leaves the old one beside it, and
+  offers to restart the service.
+- **Do**: add a node, replicate the store, leave.
+
+On a node, `sudo llmgw` opens the same settings for its `agent.yaml`.
+
+The keys come from the configuration structs themselves, through the
+generator that writes `docs/reference.md`, so a setting added to the code is on
+the screen without anyone listing it, and a test fails if the two drift.
+That generator told blocks apart by name alone, so `tlsconf.Config`,
+`engines.Config` and `supervisor.Config` were each taken for the agent's
+`Config` and dropped: the reference had no TLS, engine or supervisor settings
+at all. It now has 37 blocks, six more than before.
+
+### A second control plane with `llmgw server add`, and the screens as windows
+
+- **`sudo llmgw server add [user@]host [--vip 10.0.0.160/24]`**, run on a
+  control plane, makes another machine a standby: this llmgw copied over, a
+  certificate from the cluster's authority, the same configuration and store
+  address, the machine let into the store (for a store under Patroni, one
+  change through Patroni's API), and the service started. Every node is told
+  of it within half a minute and fails over to it untouched. `--vip` keeps a
+  floating address on whichever control plane leads, with keepalived on both.
+  Tried on new machines: switching the leader off, the standby led after 14
+  seconds and the floating address answered from it after 23.
+- **Nodes learn every control plane** from the leader (a new `ControlPlanes`
+  message, kept in `/var/lib/llmgw-agent/control-planes.json`), after the
+  ones in their configuration.
+- **A node notices a dead control plane within a quarter of a minute.** A
+  leader whose machine stopped dead left its nodes' streams open until the
+  kernel gave up retransmitting, about fifteen minutes; the agent now pings
+  the stream after ten quiet seconds, and bounds dialling too.
+- **`GET /leader`**: 200 on the control plane that leads, 503 on a standby,
+  open like `/ready` — what keepalived follows.
+- **The store's access rules live in Patroni's shared configuration** for a
+  cluster set up by `server init --patroni`, so another control plane is let
+  in without touching each member.
+- **The screens are windows**: full screen, a title bar, the steps or actions
+  down the left, the content in a bordered panel, the keys along the bottom,
+  questions and errors as dialogs over it — the setup, the console, adding a
+  node and one-machine init alike. Keyboard only.
+
+### Machines nothing may log into: `llmgw join`
+
+For sites that allow no ssh into a machine, the setup screens offer a fourth
+way to reach one: a join code. The setup listens on port 7444 while it waits
+and shows a code for each such machine, with the commands that fetch llmgw
+onto it; there, `sudo llmgw join <code>` dials out — nothing dials in —
+checks by the certificate fingerprint in the code that it reached the setup
+that made it, admits itself with the code's secret, and runs what the setup
+sends, printing each step, until the setup says it is done. A code is good
+for one machine and an hour, and is under 90 characters so it can be copied
+whole from one screen into another.
+
+To the setup a joined machine is one more way of running a script there:
+the checks, PostgreSQL, Patroni and etcd go over it unchanged. Tried with one
+machine joining by code and another logged into with a password, in one
+setup.
+
+### Setting up from nothing in screens: `sudo llmgw`
+
+Run with nothing after it in a terminal, as root, `llmgw` opens full-screen
+setup on a machine with nothing installed, and a console on a control plane.
+In a pipe or a script it prints its help, as before.
+
+- **Setup** asks one machine or three. For three — this one as the control
+  plane and etcd's third vote, two more for the store under Patroni — it asks
+  how to reach the other two: the ssh agent, a key file, or a password, and a
+  sudo password where sudo asks for one. Passwords stay in memory for the run;
+  a sudo password grants the user passwordless sudo until the install
+  finishes, and the grant is removed after, failed or not.
+- **Every machine is checked before anything is installed**: login, system
+  (Ubuntu 22.04/24.04, Debian 12), architecture, CPUs, memory and disk for its
+  role, clock against this one, NTP, the ports its role needs, internet, and
+  whether it reaches the other machines. Failures stop it, with what to do;
+  warnings let it go on.
+- **Then the plan, and the command line that does the same** from a script,
+  then the install with its progress, and the administrator's password.
+- **The console** shows `ctl status`, refreshed every five seconds, and offers
+  to add a node or, for a store on one machine, to replicate it.
+- Machines are reached with Go's ssh client rather than the system's, so a
+  password can be used; a host key is recorded on first sight and refused if it
+  changes.
+
+Tried end to end on three new machines whose user logs in and sudoes only with
+a password.
+
+### A store that survives losing its machine
+
+`deploy/ha/` now covers the store as well as the control planes: PostgreSQL
+under Patroni on two machines, its leader lock in a three-member etcd, and
+llmgw given every member with `target_session_attrs=read-write`. Measured: a
+planned switchover cost one failed lease renewal; switching off the primary's
+machine promoted the standby in 24 seconds with the gateway serving
+throughout. Example configurations are beside it.
+
+**`llmgw server init --patroni user@host --patroni user@host`** sets it up:
+PostgreSQL, Patroni and etcd on those machines over ssh — this one as etcd's
+third vote — the role and database, and the store already in use moved in.
+Every step looks first; run against a cluster that is up, it changes nothing.
+Tried from scratch on three new machines (two minutes, then a failover by
+switching the primary off) and against the reference cluster (nothing
+touched).
+
+**Watching it**: for a store of several hosts the control plane publishes
+`llmgw_store_member`, `llmgw_store_replica_lag_bytes` and
+`llmgw_store_patroni_up` from each member's Patroni, a member that stopped
+showing as `missing` rather than vanishing; `llmgw ctl status` has a STORE
+line; and four alerts — `StoreHasNoLeader`, `StorePatroniUnreachable`,
+`StoreReplicaNotStreaming`, `StoreReplicaBehind`.
+
+**Making an administrator on PostgreSQL** (`init --admin`, `users promote`)
+failed: the flag was sent as a bool to an integer column, which SQLite takes
+and PostgreSQL refuses. Every existing PostgreSQL store got its administrator
+by being copied from SQLite, which is why nobody had met it.
+
+`llmgw server backup` reads a store address of several hosts. It took the
+list for one host name and reached the database only by falling through to
+the second; it now hands libpq the list, and dumps from a standby when there
+is one.
+
+### People, limits and usage are one page
+
+Managing a team meant its members on the people page, its quota on the quotas
+page and its spend on the usage page, each linking to the others. They are now
+one page in three tabs — **Organizations & people**, **Limits**, **Usage** —
+each at the address it had before, so links and bookmarks still land. The menu
+has one entry for them, People.
+
+Choosing an organization, team, project or person in the tree, or a person in
+the account list, opens a panel beside it with everything about that one
+thing: its own limits and what it is held to from above, spend against its
+budget, its per-member limit, its capacity quota, and its requests, tokens
+and spend over the last week, drawn. Its limits can be set or changed there,
+and the answer comes back to the same panel.
+
+The administrator's menu no longer has an Organizations entry beside it: `/org`
+and `/org/<id>` send an administrator to that organization's panel here. The
+page stays as it was for an organization's managers, who cannot see this one.
+
+### The chat page says whether a GPU or a CPU answered
+
+A model can have replicas on a card and on CPUs at once, and the two answer
+at very different speeds; the chat page gave no way to tell which one a reply
+came from. The picker now says where each model runs — `GPU`, `CPU` or
+`GPU + CPU` — and the line under each reply names what answered it: `GPU
+6000-24g` or `CPU`, beside its tokens a second.
+
+Every answer through the gateway carries the same as `x-llmgw-hardware`:
+`cpu`, or the GPU class. An endpoint configured by hand, with no pool, sends
+none.
+
+### A card shared with something else is judged every heartbeat
+
+A card with another process on it when its agent started was kept out of
+placement, and stayed out — "5014 MiB already allocated by another process" —
+after that process let go of it, until the agent was restarted. Found on a
+node shared with ComfyUI. The other way round, a tenant arriving after
+registration was never seen, and a card the agent's own engine held when the
+agent restarted read as someone else's.
+
+The agent now reads every card's memory per process on each heartbeat,
+leaves out what its own engines hold, and sends the rest; the control plane
+keeps a card out of placement on that reading, and lets it back in when it
+falls under 512 MiB. `llmgw_gpu_memory_used_bytes` reads the same figure. An
+agent too old to send it keeps the reading from registration.
+
+### A GPU placement no longer runs on the CPU unnoticed
+
+llama.cpp placed on a GPU, with its CUDA backend unable to load (cuBLAS not
+where the loader looks), came up on the CPU, answered at CPU speed, and was
+reported ready while its GPU sat reserved and idle. Three checks now stop it:
+
+- **Installing** the CUDA build asks it which devices it sees
+  (`llama-server --list-devices`). None, on a node with GPUs, is a failed
+  install, with what `ldd` says `libggml-cuda.so` is missing. The binary's
+  own `ldd` could not say it: the backend is opened at run time.
+- **After the upgrade**, engines the agent finds already running are held to
+  the same check when it adopts them: one serving a GPU placement on the CPU
+  is stopped, and its replacement starts through the check. The installed
+  CUDA build is checked again within ten minutes, and installed afresh if it
+  sees no GPU.
+- **Starting** a shard on GPUs is ready only when its process holds memory on
+  each of them (`nvidia-smi --query-compute-apps`, its process group
+  included). One that does not is stopped and reported failed: "placed on GPU
+  0 but computing on the CPU", with the devices the engine lists. This covers
+  a hand-installed binary (`engines.llamacpp.bin`), which nothing else checks.
+- **`llmgw ctl engine-check llamacpp`**, run on a GPU node, now also fails
+  when the engine sees no GPU, and otherwise prints the devices it computes on.
+
+Shards are also given `CUDA_DEVICE_ORDER=PCI_BUS_ID`: placement numbers GPUs
+as nvidia-smi does, by PCI bus, and CUDA's own order is fastest first — on a
+node with two kinds of GPU, a shard could get a different card from the one it
+was given.
+
+### A model of many files, mirrored whole for vLLM
+
+vLLM loads a directory — shards, config.json, the tokenizer — and does not
+read GGUF. Mirroring one could go wrong several ways: the explorer offered the
+shards alone, a node could be handed a version still downloading or missing a
+shard, and a repository's page given as a model's source was fetched as one
+file of HTML.
+
+- **The explorer fetches a safetensors set with what the engine reads beside
+  it**: config.json, generation_config.json, the tokenizer files, the shard
+  index, the quantisation config. Never code: a `.py` would be the engine
+  running what the repository ships, which is to be chosen on purpose.
+- **A version is offered to nodes only whole.** The mirror refuses its
+  manifest (409, with what is missing) while a file is still downloading, or
+  while a shard its `model.safetensors.index.json` names, or a part of a
+  split (`-00002-of-00004`), is absent. A node reports "not complete on the
+  mirror yet" and is tried again.
+- **Checksums go with the files.** The mirror keeps each file's sha256 beside
+  it and the manifest carries it; a node checks every file of a set against
+  it before using the set, and fetches one that differs again.
+- **`llmgw mirror add` takes several URLs** into one version.
+- **A Hugging Face repository page given as a source** is read as the
+  repository id for a model vLLM or SGLang runs, which the engine downloads
+  itself (the node then needs to reach Hugging Face), and refused for one
+  llama.cpp runs, which needs a file.
+
+With a vLLM profile of `tp: 4`, placement reserves four GPUs of one node for
+such a model, as before.
+
+### Signing up with an address that has an account
+
+On PostgreSQL, signing up with an address already registered showed
+"something went wrong; try again": the store recognised a duplicate by the
+word "UNIQUE", which is SQLite's, and PostgreSQL says it in lower case under
+code 23505. It now knows both, and the address is answered as on SQLite —
+with the same page as a new sign-up, so the page cannot be used to find out
+who has an account. That page now also says what to do if the address had
+one: sign in, or ask an operator to reset the password.
+
+An error the page does not explain says the account was not created and
+gives a reference (`err-…`), logged beside the error, for an operator to find
+it; the database's own words stay in the log.
+
+### An adapter no fabric uses does not keep a node out
+
+`llmgw doctor` failed a node for a RoCE port with no RoCE v2 GID entry, or for
+`ib_uverbs` not being loaded, whether or not any fabric used them; on a node
+added for single-node inference, with no fabric given, that refused it for
+RDMA nobody had asked for.
+
+- Those are failures only for ports a declared fabric uses, and only when a
+  fabric is declared. Otherwise they are warnings saying nothing uses them yet.
+- Which fabric an undeclared InfiniBand port is on, and whether it has a
+  subnet manager LID, are notes rather than warnings; `node add` still warns
+  once that adapters are up with no fabric given.
+- `node add --fabric` hands its fabrics to doctor (`doctor --fabric
+  name:mlx5_0,…`), so they are checked as declared even though agent.yaml is
+  written afterwards.
+
+### A restart with no room beside it
+
+`ctl restart` places each replacement beside the replica it replaces. With
+no room for it — one GPU, every card that fits held by the replica itself —
+the old replica was stopped only after the drain grace period, five minutes
+of "restart replacing" with nothing saying why, which read as a restart that
+never happened; scaling to zero and back was the way out. And a replacement
+that fit nowhere even with the old replica gone was stopped for all the same,
+leaving the model with nothing. Now:
+
+- **The replica is the only thing in the way**, found by placing the
+  replacement on a copy of the inventory without it: stopped at once, the
+  replacement taking its place.
+- **The replacement fits nowhere even then**: the restart is given up and the
+  replica keeps serving. `ctl replicas` shows `restart refused:` and why.
+- **Room being made** — a lower class preempted, an idle on-demand replica
+  stopping — is waited for, as before.
+- **`ctl restart --in-place`** stops each replica first, whatever room there is.
+
+### Every GPU node gets engine builds it can run, by itself
+
+Which build of an engine a node installs follows its hardware — the CPU
+architecture, the driver, and the oldest GPU on it — from what each project
+publishes and how it was compiled. Each engine has a list of builds, best
+first; the node takes the first its hardware meets, and the first of those
+that installs and is seen to compute on the GPU. A build the table is wrong
+about costs its download, and the next one is tried.
+
+| | llama.cpp | vLLM |
+|---|---|---|
+| Driver 580+, Turing and newer | CUDA 13.3 | the version asked for, CUDA 13 (PyPI) |
+| Driver 525–579 | CUDA 12.8 (x64): its machine code for 8.6, 8.9, 12.0; the GPUs it ships as PTX need 570 | the version asked for, CUDA 12.9 (vLLM's GitHub release, PyTorch's cu129 torch) |
+| Volta (V100) | CUDA 12.8 on driver 570+, else Vulkan | **0.19.1**, the newest vLLM built for 7.0: no build of 0.20 or later has its kernels |
+| Pascal, Maxwell | CUDA 12.8 on 570+, else Vulkan | none: no vLLM is built for them |
+| arm64 GPU (GH200, Grace) | CUDA 13.3, else Vulkan — never the CPU build | CUDA 13 or 12.9, 8.0 and newer |
+| No NVIDIA GPU | the CPU build | none |
+
+Read from the projects' release pipelines: vLLM's x86 builds start at 7.5
+from 0.20.0, its PyPI build is CUDA 13 from 0.20, 0.19.1 was the last
+compiled for 7.0 (PyPI's torch 2.10 is too); llama.cpp's CUDA 12.8 build
+carries 5.0–9.0 as PTX and 8.6/8.9/12.0 as machine code, its CUDA 13.3 build
+starts at 7.5.
+
+- **A node running an older vLLM says so**: the cluster page's *vLLM build*
+  column shows the version installed, and the installation's detail says why
+  ("vLLM 0.19.1, the newest built for compute capability 7.0, instead of
+  0.30.0"). Profiles choose GPUs by class, so a model meant for the newer
+  vLLM is not placed on a V100 by accident.
+- **The install computes on the GPU before it counts**: a small tensor
+  operation for vLLM, the devices llama.cpp lists for its GPU builds.
+- **The build can still be chosen per node** on the cluster page (auto, CUDA
+  13, CUDA 12.9, none), with `llmgw ctl engine-build NODE vllm BUILD`, or
+  `POST /api/v1/nodes/{id}/engines`; kept in the store and sent to the node
+  at once.
+- **Whether an engine computes on its GPUs is read from every process
+  nvidia-smi lists**, not compute processes alone: the Vulkan build is a
+  graphics one. A llama.cpp shard is given `GGML_VK_VISIBLE_DEVICES` beside
+  `CUDA_VISIBLE_DEVICES`.
+
+### Engines per node, and nodes that do not run them all
+
+A cluster with older GPUs could not add them cleanly: `node add` refused a
+V100 node for the vLLM it could not run, though it was meant for llama.cpp,
+and `--ignore-checks` passed over every other problem with it.
+
+- **What vLLM cannot do is a warning**, in `llmgw doctor` and `node add`: the
+  node joins without it and serves what it does run. `node add` checks the
+  vLLM the control plane installs, not the binary's own default.
+- **`node add --disable-engine vllm`** keeps an engine off a node for good:
+  written as `engines.disabled` in its `agent.yaml`, never installed or
+  advertised there, and not checked by doctor.
+- **`engine_overrides`** in `server.yaml` give nodes picked by GPU class, id
+  or labels a different version of an engine, or none — for exceptions; the
+  build each GPU generation needs is chosen without it (above).
+- **The requirements follow the version**, as the table above sets out; the
+  threshold had been CUDA 13's for every version.
+- **An engine binary named in `agent.yaml` is used.** The agent had installed
+  its own beside it, which was then used first.
+- The node page shows each GPU's compute capability.
+
+### Serving from the mirror says where a model will run
+
+The serve form put every GGUF model on the CPU: it offered no choice, and its
+"Profile name" field, pre-filled "cpu", read as though renaming it would move
+the model to a GPU. Placement is decided by `gpu_class`, which the form never
+set. Now:
+
+- **Runs on** lists the fleet's GPU classes with how many cards are free, and
+  for llama.cpp the CPU. On a fleet with GPUs the class with the most free
+  cards is chosen.
+- **GPUs per replica** sets how many cards a replica takes on one node
+  (`gpus_per_node` for llama.cpp, `tp` for vLLM), and **parallel requests**
+  how many a llama.cpp replica serves at once.
+- The profile is named for where it runs (`t4`, `t4-x4`, `cpu`) rather than
+  typed.
+- A line under the form says what the model will run as — engine, CPU or how
+  many of which GPU, memory, context pool and how it is divided — and turns
+  red when the CPU is chosen with GPUs free. The message after adding repeats
+  it.
+- Choosing the CPU while GPUs are free is refused unless **run on the CPU
+  anyway** is ticked.
+
+### llama.cpp on several GPUs of one node
+
+A llama.cpp profile takes several of a node's GPUs with `gpus_per_node: N`:
+placement reserves N cards of its `gpu_class` on one node, the engine sees
+those N (`CUDA_VISIBLE_DEVICES`) and spreads the layers over them
+(`--split-mode layer`, in proportion to each card's free memory, which for
+whole cards is even). `tp` or `pp` on a llama.cpp profile was a width nothing
+placed — the replica stayed unplaced or ran on one card. It is now read as
+`gpus_per_node`, so a profile written that way is placed on that many cards
+after the upgrade, rather than dropped; only both given and disagreeing, or
+`nodes` above one, is refused.
+
 ## v0.10.6 — 2026-10-01
 
 Published first as release candidates v0.10.6-rc.1 to rc.8; rc.5 failed its

api/proto/llmgw/v1/agent.proto

@@ -43,9 +43,18 @@ message ControlMessage {
     FabricScanRequest fabric_scan = 8;
     FabricTestRequest fabric_test = 9;
     UpgradeAgent upgrade = 10;
+    ControlPlanes control_planes = 11;
   }
 }
 
+// ControlPlanes is every control plane an agent may dial, sent by the leader
+// when an agent registers and whenever the list changes. The agent keeps it
+// beside its configuration, so a second control plane added after the node
+// was onboarded is one it fails over to without anybody editing the node.
+message ControlPlanes {
+  repeated string servers = 1;
+}
+
 // Register is the first message on every stream. It carries the node's full
 // hardware inventory, so the control plane needs no separate discovery call and
 // a reconnecting agent automatically refreshes what the scheduler knows.
@@ -304,6 +313,35 @@ message GPUSensor {
   // throttle names what is holding the clocks down now: sw_power_cap,
   // hw_slowdown, sw_thermal, hw_thermal, power_brake.
   repeated string throttle = 6;
+  // How busy the card is, sampled with the rest, rather than read once at
+  // registration. -1 is not reported by the driver. memory_total_mb is set
+  // by every agent that sends these, so zero there is an older agent.
+  double utilization_percent = 7;
+  double memory_utilization_percent = 8;
+  double memory_used_mb = 9;
+  double memory_total_mb = 10;
+  double sm_clock_mhz = 11;
+  // ecc_uncorrected counts uncorrected ECC errors since the driver loaded.
+  double ecc_uncorrected = 12;
+  // What wears the card out, read every minute; has_wear says these are
+  // present. Counts of -1 are not reported for this card.
+  bool has_wear = 13;
+  int64 pcie_replays = 14;
+  int64 retired_pages_sbe = 15;
+  int64 retired_pages_dbe = 16;
+  bool retirement_pending = 17;
+  int64 remapped_rows_correctable = 18;
+  int64 remapped_rows_uncorrectable = 19;
+  bool remap_pending = 20;
+  bool remap_failure = 21;
+  int64 nvlink_crc_errors = 22;
+  int64 nvlink_replay_errors = 23;
+  int64 nvlink_recovery_errors = 24;
+  // xid_errors are the driver's error events since the agent started.
+  int64 xid_errors = 25;
+  int32 last_xid = 26;
+  string last_xid_text = 27;
+  int64 last_xid_unix = 28;
 }
 
 // ShardLoad is one engine process group's use.
@@ -445,6 +483,15 @@ message Heartbeat {
   IMEX imex = 17;
   // cpu_count as in Register, for a VM given more CPUs while it runs.
   int32 cpu_count = 18;
+
+  // gpu_foreign_memory_mb, indexed by GPU, is what processes that are not
+  // this agent's shards hold on each card, read afresh every heartbeat. What
+  // keeps a card out of placement for another tenant is decided from this
+  // rather than from registration: a card that had something else on it when
+  // the agent started is not taken forever, and one that gains a tenant later
+  // is not handed out. Empty from an agent that does not measure it, or from
+  // one that could not this time.
+  repeated int32 gpu_foreign_memory_mb = 19;
 }
 
 // Provision is the software a node should have: which engines, at which
@@ -465,6 +512,9 @@ message Provision {
 message EngineWant {
   string engine = 1;  // "llamacpp"
   string version = 2; // "b10991"
+  // variant is the build to install, when one is chosen for the node rather
+  // than left to its hardware: "cu13", "cu129". Empty is the agent's choice.
+  string variant = 3;
 }
 
 // ProvisionStatus is where one engine's installation stands on a node.

deploy/examples/server.yaml

@@ -92,6 +92,11 @@ agent_listen: ":7443"
 #   flush_interval: 200ms
 #   batch_size: 500
 #   buffer_size: 8192
+#
+#   # How long the record of each request is kept (what `ctl request` reads).
+#   # Charts, totals and budgets come from five-minute sums that are kept, so
+#   # this only bounds the table that grows by a row a request. 0 keeps all.
+#   usage_retention: 2160h   # 90 days
 
 router:
   scrape_interval: 2s

deploy/ha/README.md

@@ -85,3 +85,63 @@ auth:
 
 See [docs/install.md](../../docs/install.md) for moving an existing
 installation onto PostgreSQL without reissuing every key.
+
+## The store itself: PostgreSQL under Patroni
+
+Two control planes on one PostgreSQL have moved the single point of failure,
+not removed it. Patroni keeps PostgreSQL on two machines, one streaming to the
+other, and promotes the standby when the primary is lost. It keeps its leader
+lock in etcd, and etcd decides only with a majority — so three members, not
+two: two machines that lose sight of each other must not both decide they are
+the primary. The third member can be any machine that is up; it holds a vote,
+not data.
+
+On the reference cluster:
+
+| machine | etcd | PostgreSQL |
+|---|---|---|
+| llmgw-ctl (control plane) | member | — |
+| llmgw-pg01 | member | Patroni |
+| llmgw-ctl02 (second control plane) | member | Patroni |
+
+`llmgw server init --patroni user@host --patroni user@host` sets all of it up
+over ssh and moves the store in — see [install.md](../../docs/install.md).
+It looks before it acts at every step, so it is also how an existing cluster
+is checked: run against one that is up, it changes nothing. What it writes
+is in `etcd.default.example` (one member's `/etc/default/etcd`) and
+`patroni.yml.example` (one node's `/etc/patroni/config.yml`), for anyone
+doing it by hand; passwords are generated into `/root/llmgw-pg` on each
+database machine.
+
+**Give llmgw every member, and ask for the writable one:**
+
+```
+postgres://llmgw:…@10.0.0.149:5432,10.0.0.150:5432/llmgw?sslmode=disable&target_session_attrs=read-write
+```
+
+The driver connects to whichever member accepts writes, and reconnects to the
+new one after a failover. No floating address or HAProxy is needed for that.
+`llmgw server backup` reads the same address and dumps from a standby when
+there is one.
+
+Measured on the reference cluster, with the gateway serving throughout:
+
+- **Planned switchover** (`patronictl switchover`): the control plane logged
+  one failed lease renewal, reconnected to the new primary, and kept its
+  leadership; nothing else noticed.
+- **The primary's machine switched off**: the standby was promoted 24 seconds
+  later — the 30-second TTL in `patroni.yml` — and the control plane rode it
+  out without losing leadership or a replica. Started again, the old primary
+  rejoined as a standby by itself.
+
+**Watching it.** For a store of several hosts the control plane reads the
+cluster from each member's Patroni (port 8008) and publishes
+`llmgw_store_member{member,host,role,state}`, `llmgw_store_replica_lag_bytes`
+and `llmgw_store_patroni_up`; a host of the address that the cluster no longer
+lists is a member in state `missing`. `llmgw ctl status` prints it as one
+line, and `deploy/monitoring/llmgw-rules.yml` alerts on no leader, a member
+not streaming or missing, and a standby too far behind to be promoted.
+
+Not done here, and worth doing in production: a watchdog (`softdog`) so a
+primary that has lost the lock but not died cannot keep accepting writes, TLS
+on etcd and between PostgreSQL members, and more than one standby.

deploy/ha/etcd.default.example

@@ -0,0 +1,10 @@
+# llmgw: one member of the three that hold Patroni's leader lock (140, 149, 150).
+ETCD_NAME=llmgw-pg01
+ETCD_DATA_DIR=/var/lib/etcd/default
+ETCD_LISTEN_PEER_URLS=http://10.0.0.149:2380
+ETCD_LISTEN_CLIENT_URLS=http://10.0.0.149:2379,http://127.0.0.1:2379
+ETCD_INITIAL_ADVERTISE_PEER_URLS=http://10.0.0.149:2380
+ETCD_ADVERTISE_CLIENT_URLS=http://10.0.0.149:2379
+ETCD_INITIAL_CLUSTER=llmgw-ctl=http://10.0.0.140:2380,llmgw-pg01=http://10.0.0.149:2380,llmgw-ctl02=http://10.0.0.150:2380
+ETCD_INITIAL_CLUSTER_STATE=new
+ETCD_INITIAL_CLUSTER_TOKEN=llmgw-patroni

deploy/ha/patroni.yml.example

@@ -0,0 +1,68 @@
+# llmgw: the control planes' store, kept by Patroni on llmgw-pg01 and
+# llmgw-ctl02, with the leader lock in etcd on 140, 149 and 150.
+scope: llmgw
+namespace: /service/
+name: llmgw-pg01
+
+restapi:
+  listen: 10.0.0.149:8008
+  connect_address: 10.0.0.149:8008
+
+etcd3:
+  hosts: 10.0.0.140:2379,10.0.0.149:2379,10.0.0.150:2379
+
+bootstrap:
+  dcs:
+    ttl: 30
+    loop_wait: 10
+    retry_timeout: 10
+    maximum_lag_on_failover: 1048576
+    postgresql:
+      use_pg_rewind: true
+      use_slots: true
+      parameters:
+        wal_level: replica
+        hot_standby: "on"
+        max_wal_senders: 10
+        max_replication_slots: 10
+        wal_keep_size: 1GB
+        wal_log_hints: "on"
+        password_encryption: scram-sha-256
+  initdb:
+    - encoding: UTF8
+    - data-checksums
+
+postgresql:
+  listen: 127.0.0.1,10.0.0.149:5432
+  connect_address: 10.0.0.149:5432
+  data_dir: /var/lib/postgresql/16/llmgw
+  bin_dir: /usr/lib/postgresql/16/bin
+  pgpass: /var/lib/postgresql/.pgpass_patroni
+  authentication:
+    superuser:
+      username: postgres
+      password: "<from /root/llmgw-pg>"
+    replication:
+      username: replicator
+      password: "<from /root/llmgw-pg>"
+    rewind:
+      username: postgres
+      password: "<from /root/llmgw-pg>"
+  parameters:
+    unix_socket_directories: /var/run/postgresql
+  pg_hba:
+    - local all all peer
+    - host all all 127.0.0.1/32 scram-sha-256
+    - host replication replicator 10.0.0.149/32 scram-sha-256
+    - host replication replicator 10.0.0.150/32 scram-sha-256
+    - host all postgres 10.0.0.149/32 scram-sha-256
+    - host all postgres 10.0.0.150/32 scram-sha-256
+    - host llmgw llmgw 10.0.0.140/32 scram-sha-256
+    - host llmgw llmgw 10.0.0.149/32 scram-sha-256
+    - host llmgw llmgw 10.0.0.150/32 scram-sha-256
+
+tags:
+  nofailover: false
+  noloadbalance: false
+  clonefrom: false
+  nosync: false

deploy/monitoring/README.md

@@ -66,6 +66,8 @@ install -m 0644 grafana/datasource.yaml   /etc/grafana/provisioning/datasources/
 install -m 0644 grafana/dashboards.yaml   /etc/grafana/provisioning/dashboards/llmgw.yaml
 install -m 0644 -o grafana -g grafana grafana/llmgw-overview.json \
     /var/lib/grafana/dashboards/llmgw.json
+install -m 0644 -o grafana -g grafana grafana/llmgw-tenants.json \
+    /var/lib/grafana/dashboards/llmgw-tenants.json
 systemctl restart grafana-server
 ```
 

deploy/monitoring/grafana/llmgw-overview.json

@@ -487,6 +487,427 @@
         "defaults": {},
         "overrides": []
       }
+    },
+    {
+      "type": "row",
+      "title": "GPUs",
+      "id": 14,
+      "gridPos": {
+        "x": 0,
+        "y": 30,
+        "w": 24,
+        "h": 1
+      },
+      "collapsed": false,
+      "panels": []
+    },
+    {
+      "type": "timeseries",
+      "title": "GPU busy, by card",
+      "description": "Share of each sample period a kernel was running, sampled every heartbeat.",
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "gridPos": {
+        "x": 0,
+        "y": 31,
+        "w": 12,
+        "h": 7
+      },
+      "id": 15,
+      "targets": [
+        {
+          "expr": "llmgw_gpu_utilization_percent",
+          "legendFormat": "{{node}} gpu {{gpu}}",
+          "refId": "A"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percent",
+          "min": 0,
+          "max": 100
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi"
+        }
+      }
+    },
+    {
+      "type": "timeseries",
+      "title": "GPU memory in use, by card",
+      "description": "Framebuffer in use. Engines reserve most of it at start.",
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "gridPos": {
+        "x": 12,
+        "y": 31,
+        "w": 12,
+        "h": 7
+      },
+      "id": 16,
+      "targets": [
+        {
+          "expr": "llmgw_gpu_memory_used_bytes / llmgw_gpu_memory_total_bytes",
+          "legendFormat": "{{node}} gpu {{gpu}}",
+          "refId": "A"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percentunit",
+          "min": 0,
+          "max": 1
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi"
+        }
+      }
+    },
+    {
+      "type": "timeseries",
+      "title": "GPU memory bandwidth busy, by card",
+      "description": "Decode is bound by memory: high here with compute low is a card waiting on memory.",
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "gridPos": {
+        "x": 0,
+        "y": 38,
+        "w": 12,
+        "h": 7
+      },
+      "id": 17,
+      "targets": [
+        {
+          "expr": "llmgw_gpu_memory_utilization_percent",
+          "legendFormat": "{{node}} gpu {{gpu}}",
+          "refId": "A"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percent",
+          "min": 0,
+          "max": 100
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi"
+        }
+      }
+    },
+    {
+      "type": "timeseries",
+      "title": "GPU temperature",
+      "description": "Throttling usually starts in the high 80s.",
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "gridPos": {
+        "x": 12,
+        "y": 38,
+        "w": 12,
+        "h": 7
+      },
+      "id": 18,
+      "targets": [
+        {
+          "expr": "llmgw_gpu_temperature_celsius",
+          "legendFormat": "{{node}} gpu {{gpu}}",
+          "refId": "A"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "celsius"
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi"
+        }
+      }
+    },
+    {
+      "type": "timeseries",
+      "title": "GPU power against its limit",
+      "description": "1 is the enforced limit; a card held there is power-capped.",
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "gridPos": {
+        "x": 0,
+        "y": 45,
+        "w": 12,
+        "h": 7
+      },
+      "id": 19,
+      "targets": [
+        {
+          "expr": "llmgw_gpu_power_watts / llmgw_gpu_power_limit_watts",
+          "legendFormat": "{{node}} gpu {{gpu}}",
+          "refId": "A"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percentunit",
+          "min": 0,
+          "max": 1.1
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi"
+        }
+      }
+    },
+    {
+      "type": "timeseries",
+      "title": "GPU clock",
+      "description": "A clock held low under load is the card throttled; see the next panel for why.",
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "gridPos": {
+        "x": 12,
+        "y": 45,
+        "w": 12,
+        "h": 7
+      },
+      "id": 20,
+      "targets": [
+        {
+          "expr": "llmgw_gpu_sm_clock_mhz",
+          "legendFormat": "{{node}} gpu {{gpu}}",
+          "refId": "A"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "rotmhz"
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi"
+        }
+      }
+    },
+    {
+      "type": "timeseries",
+      "title": "What holds GPUs back",
+      "description": "Cards throttled now, by reason, and uncorrected ECC errors since the driver loaded.",
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "gridPos": {
+        "x": 0,
+        "y": 52,
+        "w": 24,
+        "h": 7
+      },
+      "id": 21,
+      "targets": [
+        {
+          "expr": "sum by (reason) (llmgw_gpu_throttled)",
+          "legendFormat": "throttled: {{reason}}",
+          "refId": "A"
+        },
+        {
+          "expr": "sum by (node, gpu) (llmgw_gpu_ecc_errors)",
+          "legendFormat": "ECC errors {{node}} gpu {{gpu}}",
+          "refId": "B"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi"
+        }
+      }
+    },
+    {
+      "type": "timeseries",
+      "title": "Output tokens/s per GPU, by model",
+      "description": "What each model gets out of each card it holds.",
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "gridPos": {
+        "x": 0,
+        "y": 59,
+        "w": 12,
+        "h": 7
+      },
+      "id": 22,
+      "targets": [
+        {
+          "expr": "sum by (model) (rate(llmgw_tokens_total{direction=\"completion\"}[5m])) / count by (model) (llmgw_gpu_assigned)",
+          "legendFormat": "{{model}}",
+          "refId": "A"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi"
+        }
+      }
+    },
+    {
+      "type": "timeseries",
+      "title": "Output tokens per watt-hour, by model",
+      "description": "Tokens for the energy the model's cards draw.",
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "gridPos": {
+        "x": 12,
+        "y": 59,
+        "w": 12,
+        "h": 7
+      },
+      "id": 23,
+      "targets": [
+        {
+          "expr": "sum by (model) (rate(llmgw_tokens_total{direction=\"completion\"}[5m])) * 3600 / sum by (model) (llmgw_gpu_power_watts * on (node, gpu) group_left (model) llmgw_gpu_assigned)",
+          "legendFormat": "{{model}}",
+          "refId": "A"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi"
+        }
+      }
+    },
+    {
+      "type": "timeseries",
+      "title": "GPU wear",
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "id": 24,
+      "description": "What wears cards out: XID errors in the last 15 minutes, memory rows remapped, PCIe retries an hour. Anything above zero here is worth a look on that card's node page.",
+      "gridPos": {
+        "x": 0,
+        "y": 66,
+        "w": 24,
+        "h": 7
+      },
+      "targets": [
+        {
+          "expr": "increase(llmgw_gpu_xid_errors[15m]) > 0",
+          "legendFormat": "XID {{node}} gpu {{gpu}}",
+          "refId": "A"
+        },
+        {
+          "expr": "sum by (node, gpu) (llmgw_gpu_remapped_rows) > 0",
+          "legendFormat": "rows remapped {{node}} gpu {{gpu}}",
+          "refId": "B"
+        },
+        {
+          "expr": "increase(llmgw_gpu_pcie_replays[1h]) > 0",
+          "legendFormat": "PCIe retries/h {{node}} gpu {{gpu}}",
+          "refId": "C"
+        },
+        {
+          "expr": "increase(llmgw_gpu_nvlink_errors[1h]) > 0",
+          "legendFormat": "NVLink {{kind}}/h {{node}} gpu {{gpu}}",
+          "refId": "D"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi"
+        }
+      }
     }
   ]
 }

deploy/monitoring/grafana/llmgw-tenants.json

@@ -0,0 +1,650 @@
+{
+  "uid": "llmgw-tenants",
+  "title": "llmgw \u2014 tenants",
+  "schemaVersion": 39,
+  "version": 1,
+  "editable": true,
+  "time": {
+    "from": "now-24h",
+    "to": "now"
+  },
+  "refresh": "1m",
+  "tags": [
+    "llmgw"
+  ],
+  "description": "Requests, tokens, failures, refusals and first-token time per tenant \u2014 the level router.tenant_label names \u2014 and every budget's share spent.",
+  "templating": {
+    "list": [
+      {
+        "name": "tenant",
+        "label": "Tenant",
+        "type": "query",
+        "datasource": {
+          "type": "prometheus",
+          "uid": "llmgw-prom"
+        },
+        "query": {
+          "query": "label_values(llmgw_tenant_requests_total, tenant)",
+          "refId": "tenant"
+        },
+        "definition": "label_values(llmgw_tenant_requests_total, tenant)",
+        "multi": true,
+        "includeAll": true,
+        "allValue": ".*",
+        "current": {
+          "text": "All",
+          "value": "$__all"
+        },
+        "refresh": 2,
+        "sort": 1
+      }
+    ]
+  },
+  "panels": [
+    {
+      "id": 1,
+      "title": "Requests a second",
+      "type": "stat",
+      "description": "Across the tenants chosen above.",
+      "gridPos": {
+        "x": 0,
+        "y": 0,
+        "w": 6,
+        "h": 4
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "sum(rate(llmgw_tenant_requests_total{tenant=~\"$tenant\"}[5m]))",
+          "legendFormat": "",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps"
+        },
+        "overrides": []
+      },
+      "options": {
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        },
+        "textMode": "auto",
+        "colorMode": "value"
+      }
+    },
+    {
+      "id": 2,
+      "title": "Failing",
+      "type": "stat",
+      "description": "Requests that failed, not counting refusals by a limit.",
+      "gridPos": {
+        "x": 6,
+        "y": 0,
+        "w": 6,
+        "h": 4
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "sum(rate(llmgw_tenant_requests_total{tenant=~\"$tenant\",outcome=\"error\"}[5m])) / sum(rate(llmgw_tenant_requests_total{tenant=~\"$tenant\"}[5m]))",
+          "legendFormat": "",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percentunit",
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "orange",
+                "value": 0.02
+              },
+              {
+                "color": "red",
+                "value": 0.05
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        },
+        "textMode": "auto",
+        "colorMode": "value"
+      }
+    },
+    {
+      "id": 3,
+      "title": "Refused by a limit",
+      "type": "stat",
+      "description": "Rate, token and budget limits; 429 to the caller.",
+      "gridPos": {
+        "x": 12,
+        "y": 0,
+        "w": 6,
+        "h": 4
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "sum(rate(llmgw_tenant_requests_total{tenant=~\"$tenant\",outcome=\"limited\"}[5m])) / sum(rate(llmgw_tenant_requests_total{tenant=~\"$tenant\"}[5m]))",
+          "legendFormat": "",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percentunit",
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "orange",
+                "value": 0.02
+              },
+              {
+                "color": "red",
+                "value": 0.1
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        },
+        "textMode": "auto",
+        "colorMode": "value"
+      }
+    },
+    {
+      "id": 4,
+      "title": "First token, p95",
+      "type": "stat",
+      "description": "What people wait for before anything appears.",
+      "gridPos": {
+        "x": 18,
+        "y": 0,
+        "w": 6,
+        "h": 4
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "histogram_quantile(0.95, sum by (le) (rate(llmgw_tenant_ttft_seconds_bucket{tenant=~\"$tenant\"}[5m])))",
+          "legendFormat": "",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "s",
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "orange",
+                "value": 3
+              },
+              {
+                "color": "red",
+                "value": 10
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        },
+        "textMode": "auto",
+        "colorMode": "value"
+      }
+    },
+    {
+      "id": 5,
+      "title": "Requests by tenant",
+      "type": "timeseries",
+      "description": "Stacked, so the top edge is the whole of it.",
+      "gridPos": {
+        "x": 0,
+        "y": 4,
+        "w": 12,
+        "h": 9
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "sum by (tenant) (rate(llmgw_tenant_requests_total{tenant=~\"$tenant\"}[5m]))",
+          "legendFormat": "{{tenant}}",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps",
+          "custom": {
+            "stacking": {
+              "mode": "normal"
+            },
+            "fillOpacity": 30
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "table",
+          "placement": "right",
+          "calcs": [
+            "mean",
+            "max"
+          ]
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      }
+    },
+    {
+      "id": 6,
+      "title": "Tokens by tenant",
+      "type": "timeseries",
+      "description": "Prompt and completion together: what was consumed.",
+      "gridPos": {
+        "x": 12,
+        "y": 4,
+        "w": 12,
+        "h": 9
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "sum by (tenant) (rate(llmgw_tenant_tokens_total{tenant=~\"$tenant\"}[5m]))",
+          "legendFormat": "{{tenant}}",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short",
+          "custom": {
+            "stacking": {
+              "mode": "normal"
+            },
+            "fillOpacity": 30
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "table",
+          "placement": "right",
+          "calcs": [
+            "mean",
+            "max"
+          ]
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      }
+    },
+    {
+      "id": 7,
+      "title": "Refused by a limit, by tenant",
+      "type": "timeseries",
+      "description": "The share of each tenant's requests a limit turned away.",
+      "gridPos": {
+        "x": 0,
+        "y": 13,
+        "w": 12,
+        "h": 8
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "sum by (tenant) (rate(llmgw_tenant_requests_total{tenant=~\"$tenant\",outcome=\"limited\"}[5m])) / sum by (tenant) (rate(llmgw_tenant_requests_total{tenant=~\"$tenant\"}[5m]))",
+          "legendFormat": "{{tenant}}",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "table",
+          "placement": "right",
+          "calcs": [
+            "mean",
+            "max"
+          ]
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      }
+    },
+    {
+      "id": 8,
+      "title": "Failing, by tenant",
+      "type": "timeseries",
+      "description": "One tenant alone failing is usually what it sends; all of them together is a model.",
+      "gridPos": {
+        "x": 12,
+        "y": 13,
+        "w": 12,
+        "h": 8
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "sum by (tenant) (rate(llmgw_tenant_requests_total{tenant=~\"$tenant\",outcome=\"error\"}[5m])) / sum by (tenant) (rate(llmgw_tenant_requests_total{tenant=~\"$tenant\"}[5m]))",
+          "legendFormat": "{{tenant}}",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "table",
+          "placement": "right",
+          "calcs": [
+            "mean",
+            "max"
+          ]
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      }
+    },
+    {
+      "id": 9,
+      "title": "First token p95, by tenant",
+      "type": "timeseries",
+      "description": "",
+      "gridPos": {
+        "x": 0,
+        "y": 21,
+        "w": 12,
+        "h": 8
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "histogram_quantile(0.95, sum by (tenant, le) (rate(llmgw_tenant_ttft_seconds_bucket{tenant=~\"$tenant\"}[10m])))",
+          "legendFormat": "{{tenant}}",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "s"
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "table",
+          "placement": "right",
+          "calcs": [
+            "mean",
+            "max"
+          ]
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      }
+    },
+    {
+      "id": 10,
+      "title": "Budgets spent this period",
+      "type": "bargauge",
+      "description": "Every organization, team, project and person with a budget. Requests are refused at 100%.",
+      "gridPos": {
+        "x": 12,
+        "y": 21,
+        "w": 12,
+        "h": 8
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "sort_desc(llmgw_budget_spent_ratio)",
+          "legendFormat": "{{scope}} {{name}} ({{period}})",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percentunit",
+          "min": 0,
+          "max": 1,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "orange",
+                "value": 0.8
+              },
+              {
+                "color": "red",
+                "value": 1
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "displayMode": "gradient",
+        "orientation": "horizontal",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        },
+        "showUnfilled": true
+      }
+    },
+    {
+      "id": 11,
+      "title": "Who used most, last 24 hours",
+      "type": "table",
+      "description": "Spend in money is on the people page's usage tab, which reads it from the store.",
+      "gridPos": {
+        "x": 0,
+        "y": 29,
+        "w": 24,
+        "h": 8
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "sort_desc(sum by (tenant) (increase(llmgw_tenant_requests_total{tenant=~\"$tenant\"}[24h])))",
+          "legendFormat": "requests",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          },
+          "format": "table",
+          "instant": true
+        },
+        {
+          "refId": "B",
+          "expr": "sum by (tenant) (increase(llmgw_tenant_tokens_total{tenant=~\"$tenant\"}[24h]))",
+          "legendFormat": "tokens",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          },
+          "format": "table",
+          "instant": true
+        },
+        {
+          "refId": "C",
+          "expr": "sum by (tenant) (increase(llmgw_tenant_requests_total{tenant=~\"$tenant\",outcome=\"limited\"}[24h]))",
+          "legendFormat": "limited",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          },
+          "format": "table",
+          "instant": true
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {},
+        "overrides": []
+      },
+      "options": {
+        "showHeader": true
+      },
+      "transformations": [
+        {
+          "id": "merge",
+          "options": {}
+        },
+        {
+          "id": "organize",
+          "options": {
+            "excludeByName": {
+              "Time": true
+            },
+            "renameByName": {
+              "Value #A": "Requests",
+              "Value #B": "Tokens",
+              "Value #C": "Refused by a limit"
+            }
+          }
+        }
+      ]
+    }
+  ]
+}

deploy/monitoring/llmgw-rules.yml

@@ -200,6 +200,94 @@ groups:
             is skipped by placement rather than failing, so this is quiet until
             somebody wonders why a model will not land there.
 
+  - name: llmgw-gpus
+    rules:
+      - alert: GPUThermalThrottling
+        expr: llmgw_gpu_throttled{reason=~"sw_thermal|hw_thermal|hw_slowdown"} == 1
+        for: 10m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.node }} GPU {{ $labels.gpu }} is slowed by heat ({{ $labels.reason }})"
+          description: >-
+            Ten minutes held back by temperature: every replica on the card is
+            slower than its measurements say. Usually airflow, a fan, or the
+            room — the node page shows the card's temperature over time.
+
+      - alert: GPUUncorrectedECCErrors
+        expr: llmgw_gpu_ecc_errors > 0
+        for: 1m
+        labels: { severity: critical }
+        annotations:
+          summary: "{{ $labels.node }} GPU {{ $labels.gpu }} has {{ $value }} uncorrected ECC errors"
+          description: >-
+            Uncorrected memory errors since the driver loaded: what runs on the
+            card may be computing on corrupted memory. Drain the node
+            (llmgw ctl drain) and have the card looked at.
+
+      - alert: GPUXidError
+        expr: increase(llmgw_gpu_xid_errors[15m]) > 0
+        for: 0m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.node }} GPU {{ $labels.gpu }} logged an XID error"
+          description: >-
+            The driver reported an error on the card (its code is
+            llmgw_gpu_last_xid, and the node page says it in words). 79 is a
+            card fallen off the bus, 48 and 63/64 memory errors, 74 NVLink,
+            94/95 contained and uncontained ECC errors: those need the card
+            looked at. Others are often an engine's own fault, and pass.
+
+      - alert: GPUOutOfSpareRows
+        expr: llmgw_gpu_remap_failure == 1
+        for: 1m
+        labels: { severity: critical }
+        annotations:
+          summary: "{{ $labels.node }} GPU {{ $labels.gpu }} has no spare memory rows left"
+          description: >-
+            The card cannot remap another failing row. Drain the node and
+            have the card replaced.
+
+      - alert: GPUNeedsReset
+        expr: llmgw_gpu_remap_pending == 1 or llmgw_gpu_retirement_pending == 1
+        for: 5m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.node }} GPU {{ $labels.gpu }} has memory repairs waiting for a reset"
+          description: >-
+            Rows remapped or pages retired take effect only after the card is
+            reset. Drain the node (llmgw ctl drain), reset or reboot, and bring
+            it back.
+
+      - alert: GPUPCIeRetriesClimbing
+        expr: increase(llmgw_gpu_pcie_replays[1h]) > 100
+        for: 0m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.node }} GPU {{ $labels.gpu }}: {{ $value }} PCIe retries in an hour"
+          description: >-
+            The card's link to the host is retrying. A riser, the slot or a
+            cable is the usual cause, and transfers to the card slow down.
+
+      - alert: GPUNVLinkErrorsClimbing
+        expr: increase(llmgw_gpu_nvlink_errors{kind="crc"}[1h]) > 10
+        for: 0m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.node }} GPU {{ $labels.gpu }}: NVLink CRC errors climbing"
+          description: >-
+            Errors on the card's NVLink links: tensor-parallel replicas on it
+            slow down, and a link that keeps failing takes a replica with it.
+
+      - alert: GPUHot
+        expr: llmgw_gpu_temperature_celsius > 87
+        for: 10m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.node }} GPU {{ $labels.gpu }} at {{ $value }} °C"
+          description: >-
+            Ten minutes above 87 °C, where most data-centre cards begin to
+            throttle.
+
   - name: monitoring-host
     rules:
       - alert: MonitoringDiskFilling
@@ -262,3 +350,130 @@ groups:
             Working as designed when the cluster is full: the lowest band is
             held first. Worth knowing about because the callers being refused
             do not know why.
+
+  # Per tenant: the level router.tenant_label names (a team by default), and
+  # every organization, team, project and person given a budget.
+  - name: llmgw-tenants
+    rules:
+      - alert: BudgetMostlySpent
+        expr: llmgw_budget_spent_ratio >= 0.8 and llmgw_budget_spent_ratio < 1
+        for: 5m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.scope }} {{ $labels.name }} has spent {{ $value | humanizePercentage }} of its {{ $labels.period }}ly budget"
+          description: >-
+            Its requests are refused once it reaches all of it, until the
+            period turns. Raise it on the people page, or tell them now
+            rather than when they are refused.
+
+      - alert: BudgetSpent
+        expr: llmgw_budget_spent_ratio >= 1
+        for: 1m
+        labels: { severity: critical }
+        annotations:
+          summary: "{{ $labels.scope }} {{ $labels.name }} has spent its {{ $labels.period }}ly budget"
+          description: >-
+            Every request under it is refused with 429 until the period
+            turns, unless a cheaper model is named as its fallback.
+
+      - alert: TenantHittingLimits
+        expr: >-
+          sum by (tenant) (rate(llmgw_tenant_requests_total{outcome="limited"}[10m]))
+          / sum by (tenant) (rate(llmgw_tenant_requests_total[10m])) > 0.1
+        for: 10m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $value | humanizePercentage }} of {{ $labels.tenant }}'s requests are refused by a limit"
+          description: >-
+            A rate, token or budget limit is turning their traffic away. Either
+            the limit is too low for what they now do, or something of theirs
+            is looping.
+
+      - alert: TenantErrors
+        expr: >-
+          sum by (tenant) (rate(llmgw_tenant_requests_total{outcome="error"}[10m]))
+          / sum by (tenant) (rate(llmgw_tenant_requests_total[10m])) > 0.05
+          and sum by (tenant) (rate(llmgw_tenant_requests_total[10m])) > 0.05
+        for: 10m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $value | humanizePercentage }} of {{ $labels.tenant }}'s requests fail"
+          description: >-
+            Not counting refusals by a limit. If every tenant fails alike it
+            is a model (ModelFailing); one alone is usually what they send —
+            prompts past the context, a model they may not use.
+
+      - alert: TenantTrafficSpike
+        expr: >-
+          sum by (tenant) (rate(llmgw_tenant_requests_total[30m]))
+          > 3 * sum by (tenant) (rate(llmgw_tenant_requests_total[7d]))
+          and sum by (tenant) (rate(llmgw_tenant_requests_total[30m])) > 0.2
+        for: 15m
+        labels: { severity: info }
+        annotations:
+          summary: "{{ $labels.tenant }} is sending three times its weekly average"
+          description: >-
+            A new workload, a batch job, or a client retrying in a loop. Only
+            for a tenant sending more than one request every five seconds, so
+            a quiet team starting work in the morning does not page anybody.
+
+      - alert: TenantSlowFirstToken
+        expr: >-
+          histogram_quantile(0.95,
+            sum by (tenant, le) (rate(llmgw_tenant_ttft_seconds_bucket[15m]))) > 10
+          and sum by (tenant) (rate(llmgw_tenant_ttft_seconds_count[15m])) > 0.02
+        for: 15m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.tenant }} waits {{ $value | humanizeDuration }} for a first token at p95"
+          description: >-
+            What their people feel. Compare with SlowFirstToken per model: if
+            only this tenant is slow, it is the models it uses or its queue
+            band, not the cluster.
+  # The store, when it is a replicated cluster under Patroni (llmgw server
+  # init --patroni). Published only for a store of several hosts.
+  - name: llmgw-store
+    rules:
+      - alert: StoreHasNoLeader
+        expr: llmgw_store_patroni_up == 1 and on() count(llmgw_store_member{role="leader"}) == 0
+        for: 1m
+        labels: { severity: critical }
+        annotations:
+          summary: "The store's cluster has no primary"
+          description: >-
+            Every write fails — keys, usage, budgets, leadership. Patroni
+            promotes a standby within its 30-second TTL; a minute without one
+            means it cannot: etcd has lost its majority, or no standby is fit.
+            `patronictl -c /etc/patroni/config.yml list` on a member says which.
+
+      - alert: StorePatroniUnreachable
+        expr: llmgw_store_patroni_up == 0
+        for: 2m
+        labels: { severity: warning }
+        annotations:
+          summary: "No member of the store's cluster answers on its Patroni port"
+          description: >-
+            The control plane reads the cluster's state from port 8008 on each
+            member. The database may still be serving; nothing here can tell.
+
+      - alert: StoreReplicaNotStreaming
+        expr: llmgw_store_member{role!="leader", state!="streaming"} == 1
+        for: 5m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.member }} is {{ $labels.state }}, not streaming from the primary"
+          description: >-
+            A standby that is not streaming is not a standby: if the primary
+            is lost now, there is nothing to promote. "missing" is a host of
+            the store address the cluster no longer lists — usually a machine
+            that is down.
+
+      - alert: StoreReplicaBehind
+        expr: llmgw_store_replica_lag_bytes > 64 * 1024 * 1024 or llmgw_store_replica_lag_bytes == -1
+        for: 10m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.member }} is {{ $value | humanize1024 }}B behind the primary"
+          description: >-
+            Patroni will not promote a standby more than 1 MB behind, so a
+            failover now would find none fit. -1 is a lag Patroni cannot read.

docs/install.md

@@ -87,7 +87,24 @@ CGO_ENABLED=0 go build -o llmgw ./cmd/llmgw
 
 ## 2. The control plane
 
-On the machine that will be the control plane, as root:
+**The short way is the screens.** On the machine that will be the control
+plane, run `sudo llmgw` with nothing after it. With nothing installed it asks
+whether this is one machine or three for high availability; for three, it asks
+for the other two — address, user, and an ssh key, the ssh agent, or a
+password, and the sudo password if sudo asks for one; or, for a machine
+nothing may log into, a join code to run there with `sudo llmgw join <code>`,
+which dials out to this one on port 7444 — checks all three before
+installing anything (system, room, clock, ports, internet, whether they reach
+one another), shows the plan, and installs. Passwords are held in memory for
+the run only; a sudo password lets the user sudo without one until the install
+finishes, and is then taken back. At the end it shows the administrator's
+password and the command that does the same from a script.
+
+Run again later, `sudo llmgw` is the console: the cluster's state, refreshed,
+and what to do next — add a node, replicate the store. A setup that stopped
+halfway opens the setup screens again, which finish it.
+
+The rest of this section is the same by command, as root:
 
 ```
 sudo llmgw server init --admin you@example.com
@@ -121,6 +138,20 @@ Add `--postgres` to keep accounts and keys in PostgreSQL rather than SQLite:
 init installs it and sets it up, and on a machine already running on SQLite it
 moves everything across. See [PostgreSQL](#postgresql-when-sqlite-is-no-longer-the-right-shape).
 
+Or `--patroni`, given two machines (or more), to keep it in PostgreSQL
+replicated across them, so losing one loses nothing:
+
+```
+sudo --preserve-env=SSH_AUTH_SOCK llmgw server init --patroni ubuntu@10.0.0.149 --patroni ubuntu@10.0.0.150
+```
+
+Over ssh with passwordless sudo, as `node add` works, it installs PostgreSQL,
+Patroni and etcd there — this machine is etcd's third vote — and moves the
+store already in use, SQLite or PostgreSQL, into the cluster. A standby is
+promoted within about thirty seconds of the primary being lost, and the
+control plane follows it by itself. `llmgw ctl status` says who leads; see
+[deploy/ha](../deploy/ha/README.md) for what it sets up.
+
 Every step looks before it acts. Running it again finishes an install that
 stopped halfway and changes nothing on one that did not; it never overwrites
 `server.yaml`, the authority, or an existing account's password.
@@ -367,7 +398,7 @@ Nothing to do: the agent installs them once the node joins, under
 
 | Engine | On | How |
 |---|---|---|
-| llama.cpp | every node | the release build for its processor, CUDA where there is a GPU, checked against GitHub's checksum |
+| llama.cpp | every node | the release build for its processor and GPU — CUDA 13.3, CUDA 12.8 (x64), or Vulkan for GPUs and drivers the CUDA builds do not cover; the CPU build only where there is no NVIDIA GPU — checked against GitHub's checksum and seen to list the GPU |
 | vLLM | nodes with an NVIDIA GPU it can run on | a Python environment built by uv from PyPI, about 8 GB |
 | SGLang | only when you ask for it | the same way as vLLM |
 
@@ -376,8 +407,17 @@ node gets the same dependencies whenever it is built, and installed with
 `--require-hashes` against the sha256 PyPI publishes. The resolved list stays in
 the environment as `llmgw-requirements.txt`. It needs:
 
-- **driver 580 or newer and a Turing GPU or newer** (compute capability 7.5):
-  vLLM is built for CUDA 13. A V100 cannot run it;
+- **the build its hardware runs**, chosen by the node. Driver 580 and newer:
+  the CUDA 13 build from PyPI. Driver 525 to 579: the same version built for
+  CUDA 12.9, from vLLM's GitHub release (checked against the digest GitHub
+  publishes) with PyTorch's CUDA 12.9 torch. Both have kernels for Turing
+  (7.5) and newer, 8.0 and newer on aarch64. **A V100** (Volta, 7.0) gets
+  **vLLM 0.19.1**, the newest built for it — no build from 0.20 on has its
+  kernels — and the cluster page shows it running that version; run `llmgw
+  ctl engine-check vllm` there once, since llmgw's flags are checked against
+  the current one. Pascal and older run no vLLM. Whatever is chosen, the
+  install ends by computing something on the GPU; a build that cannot is
+  passed over for the next;
 - **about 25 GB free** under the engines directory while it installs;
 - **gcc**, which Triton builds its kernel launcher with. The agent installs it.
 
@@ -394,8 +434,51 @@ engines:
 ```
 
 A changed version reaches every node when it next connects. To use an engine
-you installed yourself instead, set its version to `""` and name its path in
-the node's `agent.yaml` (`engines.vllm.bin`, `engines.sglang.python`).
+you installed yourself on a node, name its path in that node's `agent.yaml`
+(`engines.llamacpp.bin`, `engines.vllm.bin`, `engines.sglang.python`): the agent
+then installs none of its own beside it.
+
+#### Choosing a node's vLLM build
+
+Which build a GPU node installs is chosen on the cluster page, in the **vLLM
+build** column — *auto* (what its hardware runs, the default), *CUDA 13*,
+*CUDA 12.9*, or *none* — or from the command line:
+
+```
+llmgw ctl engine-build gpu07 vllm cu129
+```
+
+The choice is kept by the control plane and the node is told at once; it
+installs the build in the background, and the column shows how that stands.
+
+#### Different GPUs, different engines
+
+A cluster of two GPU generations may need two builds of an engine, or one on
+only some nodes. `engine_overrides` picks nodes — by a GPU class they have, by
+id, or by labels, every one given having to match — and lays engines over the
+cluster's for them, in order:
+
+```yaml
+engines:
+  vllm: "0.30.0"              # CUDA 13: Turing and newer
+engine_overrides:
+  - gpu_class: a10g           # pin a class to a version you have tested
+    engines: { vllm: "0.29.0" }
+  - labels: { role: gguf }    # nodes meant for llama.cpp alone
+    engines: { vllm: "" }     # none
+```
+
+A node told nothing it can run of an engine joins without it: `llmgw doctor`
+and `node add` warn, and the node serves what it does run. To keep an engine
+off a node for good, whatever the control plane says, add the node with
+`llmgw node add --disable-engine vllm`, which writes `engines.disabled: [vllm]`
+into its `agent.yaml` and leaves vLLM out of doctor's checks. That is narrower
+than `--ignore-checks`, which passes over every other problem as well.
+
+A llama.cpp CUDA build is checked for the GPU it is meant for: installing it
+asks it which devices it sees, and a replica placed on GPUs is not ready until
+its process holds memory on each of them. A build missing a CUDA library fails
+there, naming the library, rather than serving on the CPU.
 
 After changing a version, check it still takes the flags llmgw builds for it,
 which drift between releases:
@@ -1032,7 +1115,12 @@ A replica may come back marked **`ready (stale profile)`**. That means it is
 running a profile that has since been edited — the reconciler leaves a serving
 replica alone on purpose, because churning one costs more than the difference.
 `llmgw ctl restart <replica>` replaces it with one on the new profile, starting
-the new one before stopping the old.
+the new one before stopping the old. Where the old one is all that stands in
+the new one's way — a single GPU — it is stopped first, at once, and the model
+is unserved while the new one loads; where the new profile fits nowhere even
+then, the restart is refused and the old one keeps serving, which `ctl
+replicas` shows as `restart refused:` with the reason. `--in-place` asks for
+stop-then-start whatever room there is.
 
 ### Signed releases
 
@@ -1214,11 +1302,12 @@ requests. On a slow model, set `timeouts.generation` higher.
   can see each other, seeding between them is the obvious next thing to build.
 - **A budget without a price never binds.** Set `default_price` or `pricing`
   (step 2) or every request costs zero, and a budget is something the dashboard
-  displays rather than something that stops anything. The quotas page says so
+  displays rather than something that stops anything. The Limits tab says so
   where it can, which is not the same as being told before you rely on it.
-- **vLLM needs CUDA 13.** The version the agent installs needs driver 580
-  and a Turing GPU or newer. On older hardware, set `engines.vllm` to `""` and
-  install a vLLM built for your CUDA yourself.
+- **A V100 runs vLLM 0.19.1**, not the version the rest of the cluster runs:
+  no build from 0.20 on has Volta kernels, and the node picks the newest that
+  has. llmgw's flags are checked against the current version, so run `llmgw
+  ctl engine-check vllm` on a V100 node once.
 - **Two control planes have not been run for long.** The leader election and
   agent failover work and there is a pattern in `deploy/ha/`, but every cluster
   this has been run on in anger has had one. A single control plane is the

docs/onboarding.md

@@ -191,7 +191,8 @@ project or person, the teams and projects with their limits and spend, and every
 member. Only what was spent in that organization's projects is on it; what its
 people do elsewhere is not. The page is read-only — a manager is not an
 administrator of the cluster, and limits and memberships stay with those who
-are.
+are. An administrator following `/org` lands on the same organization's panel
+on the people page instead, where all of that can be changed.
 
 `llmgw keys init` still works for provisioning scripts and is idempotent, but it
 will no longer move a team or project that already exists under another parent;
@@ -259,7 +260,7 @@ Allowlists compose by intersection: a key cannot add a model its team withheld.
 A model's canonical name and any of its aliases name the same thing, so either
 form works.
 
-Budgets are given in whole units, as the quotas page shows them, and what a unit
+Budgets are given in whole units, as the Limits tab shows them, and what a unit
 means is yours to decide. Self-hosted models have no per-token price, so it can
 be GPU-seconds, a notional rate, or one unit per thousand tokens everywhere —
 a price of 1 000 000 in `auth.pricing`, which is in micro-units — and a budget
@@ -409,10 +410,19 @@ page exists.
 opening onto its members and its keys. From there an admin approves people
 (into a project or on their own), adds them to and takes them out of
 organizations, teams and projects, suspends and removes them, revokes keys, and
-adds, renames, moves and removes organizations, teams and projects. `/admin/quotas`
-sets limits on the tree, per member of any part of it, and per person. A searchable list of every account is at the
-bottom. The page works as plain forms; with scripts on, a change updates it in
-place and the projects you had open stay open.
+adds, renames, moves and removes organizations, teams and projects. A
+searchable list of every account is at the bottom. Choosing any organization,
+team, project or person opens a panel beside the tree with its limits, its
+capacity quota and its use over the last week, and its limits can be changed
+there.
+
+The same page has two more tabs, each at the address it had as a page of its
+own: **Limits** (`/admin/quotas`) is every limit in the tree at once — per
+branch, per member of any part of it, and per person — with the capacity quotas;
+**Usage** (`/admin/usage`) is the cluster's use over time by organization, team,
+project or person, and the GPU time each owner held. The page works as plain
+forms; with scripts on, a change updates it in place and the projects you had
+open stay open.
 
 Three things are deliberately not there:
 

docs/reference.md

@@ -39,31 +39,10 @@ rather than repeated.
 | `flush_interval` | `time.Duration` | [Auth](#auth) |
 | `batch_size` | `int` | [Auth](#auth) |
 | `buffer_size` | `int` | [Auth](#auth) |
-| `name` | `string` | [Band](#band) |
-| `priority` | `int` | [Band](#band) |
-| `saturation_ceiling` | `float64` | [Band](#band) |
+| `usage_retention` | `time.Duration` | [Auth](#auth) |
 | `tools` | `bool` | [Capabilities](#capabilities) |
 | `vision` | `bool` | [Capabilities](#capabilities) |
 | `embeddings` | `bool` | [Capabilities](#capabilities) |
-| `node_id` | `string` | [Config](#config) |
-| `server` | `string` | [Config](#config) |
-| `servers` | `[]string` | [Config](#config) |
-| `hostname` | `string` | [Config](#config) |
-| `mgmt_addr` | `string` | [Config](#config) |
-| `labels` | `map[string]string` | [Config](#config) |
-| `mgmt_interface` | `string` | [Config](#config) |
-| `fabrics` | `[]inventory.FabricAttachment` | [Config](#config) |
-| `heartbeat_interval` | `time.Duration` | [Config](#config) |
-| `reconnect_base` | `time.Duration` | [Config](#config) |
-| `reconnect_max` | `time.Duration` | [Config](#config) |
-| `adopt_ready_timeout` | `time.Duration` | [Config](#config) |
-| `tls` | `tlsconf.Config` | [Config](#config) |
-| `model_dirs` | `[]string` | [Config](#config) |
-| `engines_dir` | `string` | [Config](#config) |
-| `weights_cache` | `string` | [Config](#config) |
-| `discovery` | `discovery.Options` | [Config](#config) |
-| `engines` | `engines.Config` | [Config](#config) |
-| `supervisor` | `supervisor.Config` | [Config](#config) |
 | `interval` | `time.Duration` | [Control](#control) |
 | `base_backoff` | `time.Duration` | [Control](#control) |
 | `max_backoff` | `time.Duration` | [Control](#control) |
@@ -81,6 +60,10 @@ rather than repeated.
 | `pool` | `string` | [Endpoint](#endpoint) |
 | `weight` | `float64` | [Endpoint](#endpoint) |
 | `priority` | `int` | [Endpoint](#endpoint) |
+| `gpu_class` | `string` | [EngineOverride](#engineoverride) |
+| `nodes` | `[]string` | [EngineOverride](#engineoverride) |
+| `labels` | `map[string]string` | [EngineOverride](#engineoverride) |
+| `engines` | `map[string]string` | [EngineOverride](#engineoverride) |
 | `url` | `string` | [External](#external) |
 | `model` | `string` | [External](#external) |
 | `api_key_file` | `string` | [External](#external) |
@@ -128,8 +111,6 @@ rather than repeated.
 | `match_ratio` | `float64` | [Prefix](#prefix) |
 | `balance_abs` | `float64` | [Prefix](#prefix) |
 | `balance_rel` | `float64` | [Prefix](#prefix) |
-| `prompt_per_1k` | `int64` | [Price](#price) |
-| `completion_per_1k` | `int64` | [Price](#price) |
 | `name` | `string` | [Profile](#profile) |
 | `engine` | `string` | [Profile](#profile) |
 | `gpu_class` | `string` | [Profile](#profile) |
@@ -157,6 +138,7 @@ rather than repeated.
 | `prefix` | `Prefix` | [Router](#router) |
 | `session_header` | `string` | [Router](#router) |
 | `inject_usage` | `bool` | [Router](#router) |
+| `tenant_label` | `string` | [Router](#router) |
 | `max_body_bytes` | `int64` | [Router](#router) |
 | `timeouts` | `Timeouts` | [Router](#router) |
 | `scoring` | `Scoring` | [Router](#router) |
@@ -183,6 +165,7 @@ rather than repeated.
 | `tracing` | `Tracing` | [Server](#server) |
 | `mirror` | `Mirror` | [Server](#server) |
 | `engines` | `map[string]string` | [Server](#server) |
+| `engine_overrides` | `[]EngineOverride` | [Server](#server) |
 | `backup_dir` | `string` | [Server](#server) |
 | `releases` | `string` | [Server](#server) |
 | `router` | `Router` | [Server](#server) |
@@ -203,6 +186,56 @@ rather than repeated.
 | `bin` | `string` | [VLLMConfig](#vllmconfig) |
 | `launch_mode` | `string` | [VLLMConfig](#vllmconfig) |
 | `ray_bin` | `string` | [VLLMConfig](#vllmconfig) |
+| `prompt_per_1k` | `int64` | [accounting.Price](#accountingprice) |
+| `completion_per_1k` | `int64` | [accounting.Price](#accountingprice) |
+| `name` | `string` | [admission.Band](#admissionband) |
+| `priority` | `int` | [admission.Band](#admissionband) |
+| `saturation_ceiling` | `float64` | [admission.Band](#admissionband) |
+| `bands` | `[]Band` | [admission.Config](#admissionconfig) |
+| `default_band` | `string` | [admission.Config](#admissionconfig) |
+| `queue_ttl` | `time.Duration` | [admission.Config](#admissionconfig) |
+| `queue_capacity_per_band` | `int` | [admission.Config](#admissionconfig) |
+| `queue_capacity_total` | `int` | [admission.Config](#admissionconfig) |
+| `interval` | `time.Duration` | [admission.Config](#admissionconfig) |
+| `node_id` | `string` | [agent.Config](#agentconfig) |
+| `server` | `string` | [agent.Config](#agentconfig) |
+| `servers` | `[]string` | [agent.Config](#agentconfig) |
+| `hostname` | `string` | [agent.Config](#agentconfig) |
+| `mgmt_addr` | `string` | [agent.Config](#agentconfig) |
+| `labels` | `map[string]string` | [agent.Config](#agentconfig) |
+| `mgmt_interface` | `string` | [agent.Config](#agentconfig) |
+| `fabrics` | `[]inventory.FabricAttachment` | [agent.Config](#agentconfig) |
+| `heartbeat_interval` | `time.Duration` | [agent.Config](#agentconfig) |
+| `reconnect_base` | `time.Duration` | [agent.Config](#agentconfig) |
+| `reconnect_max` | `time.Duration` | [agent.Config](#agentconfig) |
+| `adopt_ready_timeout` | `time.Duration` | [agent.Config](#agentconfig) |
+| `tls` | `tlsconf.Config` | [agent.Config](#agentconfig) |
+| `model_dirs` | `[]string` | [agent.Config](#agentconfig) |
+| `engines_dir` | `string` | [agent.Config](#agentconfig) |
+| `weights_cache` | `string` | [agent.Config](#agentconfig) |
+| `discovery` | `discovery.Options` | [agent.Config](#agentconfig) |
+| `engines` | `engines.Config` | [agent.Config](#agentconfig) |
+| `supervisor` | `supervisor.Config` | [agent.Config](#agentconfig) |
+| `name` | `string` | [config.Band](#configband) |
+| `priority` | `int` | [config.Band](#configband) |
+| `saturation_ceiling` | `float64` | [config.Band](#configband) |
+| `prompt_per_1k` | `int64` | [config.Price](#configprice) |
+| `completion_per_1k` | `int64` | [config.Price](#configprice) |
+| `disabled` | `[]string` | [engines.Config](#enginesconfig) |
+| `vllm` | `VLLMConfig` | [engines.Config](#enginesconfig) |
+| `sglang` | `SGLangConfig` | [engines.Config](#enginesconfig) |
+| `llamacpp` | `LlamaCppConfig` | [engines.Config](#enginesconfig) |
+| `sim` | `SimConfig` | [engines.Config](#enginesconfig) |
+| `work_dir` | `string` | [supervisor.Config](#supervisorconfig) |
+| `log_lines` | `int` | [supervisor.Config](#supervisorconfig) |
+| `stop_timeout` | `time.Duration` | [supervisor.Config](#supervisorconfig) |
+| `health_interval` | `time.Duration` | [supervisor.Config](#supervisorconfig) |
+| `rendezvous_timeout` | `time.Duration` | [supervisor.Config](#supervisorconfig) |
+| `alive_grace` | `time.Duration` | [supervisor.Config](#supervisorconfig) |
+| `ca` | `string` | [tlsconf.Config](#tlsconfconfig) |
+| `cert` | `string` | [tlsconf.Config](#tlsconfconfig) |
+| `key` | `string` | [tlsconf.Config](#tlsconfconfig) |
+| `insecure` | `bool` | [tlsconf.Config](#tlsconfconfig) |
 
 ## Adapter
 
@@ -342,7 +375,7 @@ table costs whatever DefaultPrice says, which is nothing unless set.
 
 `Price`
 
-Takes a [Price](#price) block.
+Takes a [config.Price](#configprice) block.
 
 ### `flush_interval`
 
@@ -362,19 +395,16 @@ BufferSize bounds the queue. Beyond it records are dropped rather than
 blocking a request: losing accounting is bad, turning a stalled database
 into an outage is worse.
 
-## Band
+### `usage_retention`
 
-### `name`
+`time.Duration`
 
-`string`
-
-### `priority`
-
-`int`
-
-### `saturation_ceiling`
-
-`float64`
+UsageRetention is how long the record of each request is kept: who sent
+it, which replica answered, how long it queued — what `ctl request`
+reads. Zero keeps it for ever. Totals, charts and budgets are read from
+five-minute sums kept beside the records, which are not pruned, so this
+bounds the table that grows by a row a request without changing any
+figure a page shows. At least a day when set.
 
 ## Capabilities
 
@@ -390,152 +420,6 @@ into an outage is worse.
 
 `bool`
 
-## Config
-
-### `node_id`
-
-`string`
-
-NodeID defaults to the hostname. It must equal the common name of the
-agent's certificate, so set it only where the two differ.
-
-### `server`
-
-`string`
-
-Server is the control plane, and Servers is the same thing when there is
-more than one. Both are accepted: every configuration written before
-there could be a second control plane names one address, and none of
-them should have to change.
-
-The order is tried in order, and a refusal moves to the next. Only the
-leader accepts an agent, so a standby is skipped in the time it takes to
-be told so — which is what makes failover a reconnect rather than an
-edit to every node's configuration.
-
-### `servers`
-
-`[]string`
-
-### `hostname`
-
-`string`
-
-Hostname defaults to the machine's.
-
-### `mgmt_addr`
-
-`string`
-
-MgmtAddr is the address engines are reached on. It defaults to this
-machine's address on the route to the control plane, which is the one
-the control plane and the router can reach.
-
-### `labels`
-
-`map[string]string`
-
-Labels are free-form facts about a node for a profile's node_selector.
-Nothing needs them: memory, GPUs, engines and disk are measured, and a
-label restating one of those goes stale the day the machine changes.
-
-### `mgmt_interface`
-
-`string`
-
-MgmtInterface is the ethernet interface NCCL bootstraps over. It
-defaults to the interface holding MgmtAddr, then eth0.
-
-### `fabrics`
-
-`[]inventory.FabricAttachment`
-
-Fabrics labels this node's HCAs. It is explicit on purpose: OpenSM's
-default subnet prefix is the same on every fabric, so nothing can be
-inferred from the wire (plan §5.2).
-
-### `heartbeat_interval`
-
-`time.Duration`
-
-### `reconnect_base`
-
-`time.Duration`
-
-ReconnectBase is the first reconnect delay; it backs off from there.
-
-### `reconnect_max`
-
-`time.Duration`
-
-### `adopt_ready_timeout`
-
-`time.Duration`
-
-AdoptReadyTimeout bounds how long an adopted engine is given to answer a
-health check before the agent stops waiting on it.
-
-Shorter than a startup timeout on purpose: this engine is already
-running, so it is not loading a model, it is either serving or it is not.
-
-### `tls`
-
-`tlsconf.Config`
-
-TLS authenticates both ends of the control-plane stream. Leaving it unset
-keeps the plaintext behaviour, which only belongs on a trusted network.
-
-Takes a [Config](#config) block.
-
-### `model_dirs`
-
-`[]string`
-
-ModelDirs are the directories this node keeps model weights in, beyond
-WeightsCache, which is always looked in. Storage differs at every installation,
-so llmgw observes it rather than owning it: a Lustre mount, an image
-exported over the fabric, local NVMe and a copy made by hand all look
-identical from here, and placement only needs to know whether the file
-is present.
-
-### `engines_dir`
-
-`string`
-
-EnginesDir is where the agent installs engines, one directory per
-build. Defaults to /opt/llmgw/engines.
-
-### `weights_cache`
-
-`string`
-
-WeightsCache is where a remote source is downloaded to. Defaults to the
-first entry of ModelDirs, because a node that says where its models live
-has already answered this question, and otherwise to
-/var/lib/llmgw-weights.
-
-Fetching is the simple arrangement: every node keeps its own copy. The
-exotic ones need no code — weights on a shared filer or an image exported
-over the fabric are simply present, and present means nothing is fetched.
-
-### `discovery`
-
-`discovery.Options`
-
-Takes a [Options](#options) block.
-
-### `engines`
-
-`engines.Config`
-
-Takes a [Config](#config) block.
-
-### `supervisor`
-
-`supervisor.Config`
-
-Takes a [Config](#config) block.
-
 ## Control
 
 ### `interval`
@@ -645,6 +529,32 @@ called. Empty means vLLM's names.
 
 `int`
 
+## EngineOverride
+
+### `gpu_class`
+
+`string`
+
+GPUClass picks nodes with a GPU of this class ("v100-32g").
+
+### `nodes`
+
+`[]string`
+
+Nodes picks nodes by id.
+
+### `labels`
+
+`map[string]string`
+
+Labels picks nodes carrying all of these.
+
+### `engines`
+
+`map[string]string`
+
+Engines are laid over the cluster's: a version, or "" for none.
+
 ## External
 
 ### `url`
@@ -738,7 +648,7 @@ agents use. Empty leaves the connection in plain HTTP, which is only
 acceptable on a network where nothing untrusted can reach that port —
 the mirror says as much in its own log when it starts that way.
 
-Takes a [Config](#config) block.
+Takes a [tlsconf.Config](#tlsconfconfig) block.
 
 ### `registry`
 
@@ -1000,16 +910,6 @@ before affinity is abandoned.
 
 `float64`
 
-## Price
-
-### `prompt_per_1k`
-
-`int64`
-
-### `completion_per_1k`
-
-`int64`
-
 ## Profile
 
 ### `name`
@@ -1211,6 +1111,16 @@ SessionHeader carries the session id for affinity (plan §7.5 tier 0).
 InjectUsage asks engines for usage even when the client did not, so token
 accounting works regardless of client behaviour.
 
+### `tenant_label`
+
+`string`
+
+TenantLabel is the level of the tree the per-tenant request series are
+labelled by — llmgw_tenant_requests_total and its siblings: "org",
+"team" or "project", or "none" for no such series. A label per project
+on a cluster of thousands is more series than Prometheus wants, and a
+team is what an alert is usually about; "team" is the default.
+
 ### `max_body_bytes`
 
 `int64`
@@ -1371,7 +1281,7 @@ AgentTLS authenticates the fleet-facing port. It deliberately does not
 cover the client-facing one: that surface is fronted by a publicly
 trusted certificate, while this one is mutual and private.
 
-Takes a [Config](#config) block.
+Takes a [tlsconf.Config](#tlsconfconfig) block.
 
 ### `monitoring`
 
@@ -1412,6 +1322,16 @@ never trusted to already have what somebody once put there.
 Unset keeps DefaultEngines. A version replaces the default; an empty
 one leaves that engine to be installed by hand.
 
+### `engine_overrides`
+
+`[]EngineOverride`
+
+EngineOverrides give some nodes different engines from the rest: a
+vLLM built for CUDA 12 on the Volta cards CUDA 13 dropped, or none at
+all on nodes meant only for llama.cpp. Each picks nodes and lays its
+engines over the cluster's, in order, later ones last; an empty version
+removes that engine from the nodes it picks.
+
 ### `backup_dir`
 
 `string`
@@ -1553,3 +1473,345 @@ node and has not been checked against a real cluster.
 
 RayBin is used when LaunchMode is "ray".
 
+## accounting.Price
+
+### `prompt_per_1k`
+
+`int64`
+
+### `completion_per_1k`
+
+`int64`
+
+## admission.Band
+
+### `name`
+
+`string`
+
+### `priority`
+
+`int`
+
+Priority orders bands; higher is served first.
+
+### `saturation_ceiling`
+
+`float64`
+
+SaturationCeiling is the pool saturation above which this band waits.
+1.0 means "never hold this band back".
+
+## admission.Config
+
+### `bands`
+
+`[]Band`
+
+### `default_band`
+
+`string`
+
+### `queue_ttl`
+
+`time.Duration`
+
+QueueTTL drops a request that has waited this long. Sending the GPU work
+the client has already given up on helps nobody.
+
+### `queue_capacity_per_band`
+
+`int`
+
+QueueCapacityPerBand bounds each band's queue. Rejecting immediately
+beats accumulating silently.
+
+### `queue_capacity_total`
+
+`int`
+
+QueueCapacityTotal bounds everything queued, across bands. When it is
+reached, a request is not refused while a lower band has requests
+queued: the newest of the lowest band is turned away instead
+(displacement), since the oldest have waited longest and are nearest
+their TTL. Zero takes half of every band's capacity together.
+
+### `interval`
+
+`time.Duration`
+
+Interval is how often the dispatcher re-checks saturation. Releases also
+wake it, so this is only the backstop for saturation falling on its own.
+
+## agent.Config
+
+### `node_id`
+
+`string`
+
+NodeID defaults to the hostname. It must equal the common name of the
+agent's certificate, so set it only where the two differ.
+
+### `server`
+
+`string`
+
+Server is the control plane, and Servers is the same thing when there is
+more than one. Both are accepted: every configuration written before
+there could be a second control plane names one address, and none of
+them should have to change.
+
+The order is tried in order, and a refusal moves to the next. Only the
+leader accepts an agent, so a standby is skipped in the time it takes to
+be told so — which is what makes failover a reconnect rather than an
+edit to every node's configuration.
+
+### `servers`
+
+`[]string`
+
+### `hostname`
+
+`string`
+
+Hostname defaults to the machine's.
+
+### `mgmt_addr`
+
+`string`
+
+MgmtAddr is the address engines are reached on. It defaults to this
+machine's address on the route to the control plane, which is the one
+the control plane and the router can reach.
+
+### `labels`
+
+`map[string]string`
+
+Labels are free-form facts about a node for a profile's node_selector.
+Nothing needs them: memory, GPUs, engines and disk are measured, and a
+label restating one of those goes stale the day the machine changes.
+
+### `mgmt_interface`
+
+`string`
+
+MgmtInterface is the ethernet interface NCCL bootstraps over. It
+defaults to the interface holding MgmtAddr, then eth0.
+
+### `fabrics`
+
+`[]inventory.FabricAttachment`
+
+Fabrics labels this node's HCAs. It is explicit on purpose: OpenSM's
+default subnet prefix is the same on every fabric, so nothing can be
+inferred from the wire (plan §5.2).
+
+### `heartbeat_interval`
+
+`time.Duration`
+
+### `reconnect_base`
+
+`time.Duration`
+
+ReconnectBase is the first reconnect delay; it backs off from there.
+
+### `reconnect_max`
+
+`time.Duration`
+
+### `adopt_ready_timeout`
+
+`time.Duration`
+
+AdoptReadyTimeout bounds how long an adopted engine is given to answer a
+health check before the agent stops waiting on it.
+
+Shorter than a startup timeout on purpose: this engine is already
+running, so it is not loading a model, it is either serving or it is not.
+
+### `tls`
+
+`tlsconf.Config`
+
+TLS authenticates both ends of the control-plane stream. Leaving it unset
+keeps the plaintext behaviour, which only belongs on a trusted network.
+
+Takes a [tlsconf.Config](#tlsconfconfig) block.
+
+### `model_dirs`
+
+`[]string`
+
+ModelDirs are the directories this node keeps model weights in, beyond
+WeightsCache, which is always looked in. Storage differs at every installation,
+so llmgw observes it rather than owning it: a Lustre mount, an image
+exported over the fabric, local NVMe and a copy made by hand all look
+identical from here, and placement only needs to know whether the file
+is present.
+
+### `engines_dir`
+
+`string`
+
+EnginesDir is where the agent installs engines, one directory per
+build. Defaults to /opt/llmgw/engines.
+
+### `weights_cache`
+
+`string`
+
+WeightsCache is where a remote source is downloaded to. Defaults to the
+first entry of ModelDirs, because a node that says where its models live
+has already answered this question, and otherwise to
+/var/lib/llmgw-weights.
+
+Fetching is the simple arrangement: every node keeps its own copy. The
+exotic ones need no code — weights on a shared filer or an image exported
+over the fabric are simply present, and present means nothing is fetched.
+
+### `discovery`
+
+`discovery.Options`
+
+Takes a [Options](#options) block.
+
+### `engines`
+
+`engines.Config`
+
+Takes a [engines.Config](#enginesconfig) block.
+
+### `supervisor`
+
+`supervisor.Config`
+
+Takes a [supervisor.Config](#supervisorconfig) block.
+
+## config.Band
+
+### `name`
+
+`string`
+
+### `priority`
+
+`int`
+
+### `saturation_ceiling`
+
+`float64`
+
+## config.Price
+
+### `prompt_per_1k`
+
+`int64`
+
+### `completion_per_1k`
+
+`int64`
+
+## engines.Config
+
+### `disabled`
+
+`[]string`
+
+Disabled are engines this node never installs or advertises, whatever
+the control plane asks: vLLM on a node of cards no build of it runs on,
+that is meant for llama.cpp alone.
+
+### `vllm`
+
+`VLLMConfig`
+
+Takes a [VLLMConfig](#vllmconfig) block.
+
+### `sglang`
+
+`SGLangConfig`
+
+Takes a [SGLangConfig](#sglangconfig) block.
+
+### `llamacpp`
+
+`LlamaCppConfig`
+
+Takes a [LlamaCppConfig](#llamacppconfig) block.
+
+### `sim`
+
+`SimConfig`
+
+Takes a [SimConfig](#simconfig) block.
+
+## supervisor.Config
+
+### `work_dir`
+
+`string`
+
+WorkDir holds pid files and per-shard scratch space.
+
+### `log_lines`
+
+`int`
+
+LogLines is how many lines of engine output to retain per shard.
+
+### `stop_timeout`
+
+`time.Duration`
+
+StopTimeout is how long a process gets to exit after SIGTERM before it is
+killed. Engines need time to free GPU memory cleanly.
+
+### `health_interval`
+
+`time.Duration`
+
+HealthInterval is how often a starting shard is polled.
+
+### `rendezvous_timeout`
+
+`time.Duration`
+
+RendezvousTimeout bounds the wait for a multi-node rendezvous point to
+start listening.
+
+### `alive_grace`
+
+`time.Duration`
+
+AliveGrace is how long a non-serving rank must stay up before it counts
+as ready. There is no external signal for a worker, so staying alive is
+the only evidence available.
+
+## tlsconf.Config
+
+### `ca`
+
+`string`
+
+CA is the bundle used to verify the other side. In a real deployment this
+is whichever authority issued the fleet's certificates.
+
+### `cert`
+
+`string`
+
+Cert and Key are this component's own identity.
+
+### `key`
+
+`string`
+
+### `insecure`
+
+`bool`
+
+Insecure disables TLS entirely. It exists for local runs against
+sim-engines; anything using it must stay on a trusted network.
+

go.mod

@@ -23,6 +23,7 @@ require (
 	go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.46.0
 	go.opentelemetry.io/otel/sdk v1.46.0
 	go.opentelemetry.io/otel/trace v1.46.0
+	go.yaml.in/yaml/v3 v3.0.5
 	golang.org/x/crypto v0.57.0
 	golang.org/x/net v0.59.0
 	golang.org/x/sync v0.23.0
@@ -54,6 +55,7 @@ require (
 	github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
 	github.com/jackc/puddle/v2 v2.2.2 // indirect
 	github.com/knadh/koanf/maps v0.1.2 // indirect
+	github.com/kylelemons/godebug v1.1.0 // indirect
 	github.com/lucasb-eyer/go-colorful v1.3.0 // indirect
 	github.com/mattn/go-localereader v0.0.1 // indirect
 	github.com/mattn/go-runewidth v0.0.19 // indirect
@@ -73,7 +75,6 @@ require (
 	go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.46.0 // indirect
 	go.opentelemetry.io/otel/metric v1.46.0 // indirect
 	go.opentelemetry.io/proto/otlp v1.11.0 // indirect
-	go.yaml.in/yaml/v3 v3.0.5 // indirect
 	golang.org/x/sys v0.48.0 // indirect
 	golang.org/x/text v0.42.0 // indirect
 	google.golang.org/genproto/googleapis/api v0.0.0-20260819154853-08b0e4226688 // indirect

go.sum

@@ -166,6 +166,8 @@ golang.org/x/sync v0.23.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0=
 golang.org/x/sys v0.0.0-20210809222454-d867a43fc93e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
 golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
+golang.org/x/term v0.46.0 h1:3+OXuTbaKDgwk8jTi3aSLHRlmWqHEUDUtxnbFigO4YE=
+golang.org/x/term v0.46.0/go.mod h1:+K02xbkittuwc0Am4abfA3Fc+XRGXkvBXNO88NCXPoc=
 golang.org/x/text v0.42.0 h1:JbOZXgfeCPU9gacVtYliJqOhD+zhrEqK4LfdpmlUZqI=
 golang.org/x/text v0.42.0/go.mod h1:ojzP1Z+2QtioaF8DTtO8K5q7JWVVYwZKenzujK0Zd0E=
 golang.org/x/tools v0.49.0 h1:3NI7VXzL9+1WZD52Dx2ttoPwD5DWrFGpl9mFZDlmisI=

internal/accounting/accounting.go

@@ -316,6 +316,17 @@ func (a *Accountant) Record(res Reservation, u store.Usage) {
 	}
 }
 
+// RecordRefused queues the row of a request a limit refused: nothing is
+// charged — it used nothing — but it is counted, so a team running into its
+// limits shows as one rather than as a team that went quiet.
+func (a *Accountant) RecordRefused(u store.Usage) {
+	if u.KeyID == "" || a.recorder == nil {
+		return
+	}
+	u.PromptTokens, u.CompletionTokens, u.CostMicros = 0, 0, 0
+	a.recorder.Record(u)
+}
+
 // Status returns what a key has used, for headers and `ctl`.
 func (a *Accountant) Status(p auth.Principal) (spent int64, rpm, tpm float64) {
 	if p.KeyID == "" {

internal/agent/agent.go

@@ -73,8 +73,11 @@ type Agent struct {
 	lastEngineProbe time.Time
 	probingEngines  bool
 
-	mu  sync.Mutex
-	inv discovery.Inventory
+	mu sync.Mutex
+	// serverList is every control plane to dial: the configured ones, and
+	// any the leader has named since. Read through servers().
+	serverList []string
+	inv        discovery.Inventory
 	// shardReplica remembers which replica a shard belongs to, so a status
 	// report can name both. The control plane keys on the replica.
 	shardReplica map[string]string
@@ -104,6 +107,8 @@ type Agent struct {
 	// sensors reads heat and power: CPU and board from hwmon and RAPL, GPUs
 	// from nvidia-smi.
 	sensors *discovery.SensorSampler
+	// wear reads what wears a GPU out: remapped rows, PCIe retries, XIDs.
+	wear *discovery.HealthSampler
 
 	// fetchWatchers are the shards waiting on each source, so progress on one
 	// download reaches every shard that asked for it. A fetch is single-flight
@@ -150,6 +155,8 @@ func New(cfg Config, httpClient connectHTTPClient, log *slog.Logger) (*Agent, er
 		provKick:     make(chan struct{}, 1),
 		load:         discovery.NewLoadSampler(),
 		sensors:      discovery.NewSensorSampler(),
+		wear:         discovery.NewHealthSampler(),
+		serverList:   loadControlPlanes(cfg.Servers, cfg.TLS.Enabled()),
 	}
 	a.prov = &provision.Installer{Root: cfg.EnginesDir, Log: log, OnStatus: a.provisionStatus}
 	return a, nil
@@ -182,7 +189,9 @@ func (a *Agent) Run(ctx context.Context) error {
 	attempt := 0
 	next := 0
 	for ctx.Err() == nil {
-		server := a.cfg.Servers[next%len(a.cfg.Servers)]
+		// Read each time round: the leader may have named another since.
+		servers := a.servers()
+		server := servers[next%len(servers)]
 		err := a.session(ctx, server)
 		if ctx.Err() != nil {
 			break
@@ -197,12 +206,12 @@ func (a *Agent) Run(ctx context.Context) error {
 		// No wait when there is somewhere else to try: the delay exists to
 		// stop hammering one control plane, and the next address has not been
 		// hammered. With a single address this is the old behaviour exactly.
-		if len(a.cfg.Servers) > 1 && next%len(a.cfg.Servers) != 0 {
+		if len(servers) > 1 && next%len(servers) != 0 {
 			delay = 0
 		}
 		a.log.Warn("disconnected from the control plane, retrying",
 			"err", err, "server", server,
-			"next", a.cfg.Servers[next%len(a.cfg.Servers)],
+			"next", servers[next%len(servers)],
 			"in", delay.Round(time.Millisecond), "attempt", attempt)
 
 		select {
@@ -314,6 +323,17 @@ func (a *Agent) settleAdopted(sh *supervisor.Shard, o supervisor.Orphan) {
 			"shard", o.ShardID, "replica", o.ReplicaID, "err", err)
 		return
 	}
+	// An engine started by an agent older than the GPU check may be one
+	// serving a GPU placement on the CPU. Stopped, so its replacement starts
+	// through the check and fails there in words, if the build still sees
+	// no GPU — rather than going on unnoticed after the upgrade that could
+	// have caught it.
+	if err := a.onItsGPUs(ctx, eng, o.Spec, sh.Status().PID); err != nil {
+		a.log.Error("stopping an adopted engine that computes on the CPU though placed on GPUs",
+			"shard", o.ShardID, "replica", o.ReplicaID, "err", err)
+		_ = a.sup.Stop(ctx, o.ShardID)
+		return
+	}
 	a.log.Info("adopted engine is serving",
 		"shard", o.ShardID, "replica", o.ReplicaID)
 }
@@ -450,7 +470,7 @@ func (a *Agent) session(ctx context.Context, server string) error {
 	// vLLM accepted a vLLM shard and failed it minutes later.
 	reg.EngineVersions = a.engines(parent)
 	reg.EnginesProbed = true
-	reg.InstallableEngines = provision.Installable()
+	reg.InstallableEngines = slices.DeleteFunc(provision.Installable(), a.disabled)
 	reg.Features = []string{FeatureFabricTest}
 	reg.RunningShards = a.runningShards()
 	a.mu.Lock()
@@ -577,8 +597,24 @@ func (a *Agent) heartbeat() *llmgwv1.AgentMessage {
 	}
 	a.mu.Unlock()
 
-	for _, g := range inv.GPUs {
-		hb.GpuMemoryUsedMb = append(hb.GpuMemoryUsedMb, int32(g.MemoryUsedMB))
+	// Measured now, not carried from registration: a card is shared with
+	// whatever else runs on the machine, and that comes and goes while the
+	// agent runs. The registration figure stays only for a node where
+	// nvidia-smi cannot be asked.
+	if len(inv.GPUs) > 0 {
+		var shards []int
+		for _, st := range a.sup.Statuses() {
+			if st.PID > 0 {
+				shards = append(shards, st.PID)
+			}
+		}
+		hb.GpuForeignMemoryMb, hb.GpuMemoryUsedMb = a.foreignGPUMemory(context.Background(), shards)
+	}
+	if len(hb.GpuMemoryUsedMb) != len(inv.GPUs) {
+		hb.GpuForeignMemoryMb, hb.GpuMemoryUsedMb = nil, nil
+		for _, g := range inv.GPUs {
+			hb.GpuMemoryUsedMb = append(hb.GpuMemoryUsedMb, int32(g.MemoryUsedMB))
+		}
 	}
 
 	// Rescanned here rather than reused from registration. Disk is not a fact
@@ -645,11 +681,34 @@ func (a *Agent) sampleLoad() *llmgwv1.NodeLoad {
 	if a.sensors != nil {
 		se := a.sensors.Sample(context.Background())
 		out.CpuTempC, out.CpuPowerW, out.InletTempC = se.CPUTempC, se.CPUPowerW, se.InletTempC
+		var wear map[int]discovery.GPUWear
+		if a.wear != nil && len(se.GPUs) > 0 {
+			wear = map[int]discovery.GPUWear{}
+			for _, w := range a.wear.Sample(context.Background()) {
+				wear[w.Index] = w
+			}
+		}
 		for _, g := range se.GPUs {
-			out.Gpus = append(out.Gpus, &llmgwv1.GPUSensor{
+			gs := &llmgwv1.GPUSensor{
 				Index: int32(g.Index), TempC: g.TempC, MemoryTempC: g.MemoryTempC,
 				PowerW: g.PowerW, PowerLimitW: g.PowerLimitW, Throttle: g.Throttle,
-			})
+				UtilizationPercent: g.UtilPercent, MemoryUtilizationPercent: g.MemUtilPercent,
+				MemoryUsedMb: g.MemoryUsedMB, MemoryTotalMb: g.MemoryTotalMB,
+				SmClockMhz: g.SMClockMHz, EccUncorrected: g.ECCUncorrected,
+			}
+			if w, ok := wear[g.Index]; ok {
+				gs.HasWear = true
+				gs.PcieReplays, gs.RetiredPagesSbe, gs.RetiredPagesDbe, gs.RetirementPending =
+					w.PCIeReplays, w.RetiredSBE, w.RetiredDBE, w.RetirePending
+				gs.RemappedRowsCorrectable, gs.RemappedRowsUncorrectable, gs.RemapPending, gs.RemapFailure =
+					w.RemappedCorrectable, w.RemappedUncorrectable, w.RemapPending, w.RemapFailure
+				gs.NvlinkCrcErrors, gs.NvlinkReplayErrors, gs.NvlinkRecoveryErrors = w.NVLinkCRC, w.NVLinkReplay, w.NVLinkRecovery
+				gs.XidErrors, gs.LastXid, gs.LastXidText = w.XIDs, int32(w.LastXID), w.LastXIDText
+				if !w.LastXIDAt.IsZero() {
+					gs.LastXidUnix = w.LastXIDAt.Unix()
+				}
+			}
+			out.Gpus = append(out.Gpus, gs)
 		}
 	}
 	return out
@@ -739,6 +798,8 @@ func (a *Agent) handle(ctx context.Context, msg *llmgwv1.ControlMessage, out cha
 		a.setProvision(m.Provision)
 	case *llmgwv1.ControlMessage_Upgrade:
 		go a.upgrade(ctx, m.Upgrade, out)
+	case *llmgwv1.ControlMessage_ControlPlanes:
+		a.setControlPlanes(m.ControlPlanes)
 	case *llmgwv1.ControlMessage_Logs:
 		reply := a.logsReply(m.Logs)
 		// Off the receive loop: a full send queue must not stop commands being
@@ -931,6 +992,14 @@ func (a *Agent) startShard(ctx context.Context, m *llmgwv1.StartShard, out chan<
 	} else {
 		err = a.sup.WaitAlive(ctx, sh)
 	}
+	if err == nil {
+		err = a.onItsGPUs(ctx, eng, spec, sh.Status().PID)
+		if err != nil {
+			// Serving, but not where it was placed: stopped, so the GPU it
+			// holds in the inventory is not taken by a CPU process.
+			_ = a.sup.Stop(ctx, shardID)
+		}
+	}
 	if err != nil {
 		report(supervisor.Failed, err.Error(), sh.Logs(20))
 		return
@@ -1023,7 +1092,9 @@ func Client(cfg tlsconf.Config, timeout time.Duration) (*http.Client, error) {
 	if !cfg.Enabled() {
 		return &http.Client{
 			Transport: &http2.Transport{
-				AllowHTTP: true,
+				AllowHTTP:       true,
+				ReadIdleTimeout: streamIdle,
+				PingTimeout:     streamPing,
 				DialTLSContext: func(ctx context.Context, network, addr string, _ *tls.Config) (net.Conn, error) {
 					return (&net.Dialer{Timeout: timeout}).DialContext(ctx, network, addr)
 				},
@@ -1038,9 +1109,31 @@ func Client(cfg tlsconf.Config, timeout time.Duration) (*http.Client, error) {
 	if err != nil {
 		return nil, err
 	}
-	return &http.Client{Transport: &http2.Transport{TLSClientConfig: tc}}, nil
+	return &http.Client{Transport: &http2.Transport{
+		TLSClientConfig: tc,
+		ReadIdleTimeout: streamIdle,
+		PingTimeout:     streamPing,
+		// Bounded, as the plaintext branch's is: dialling a control plane
+		// whose machine is off waits out the kernel's connect retries,
+		// about two minutes, before the next address is tried.
+		DialTLSContext: func(ctx context.Context, network, addr string, c *tls.Config) (net.Conn, error) {
+			d := tls.Dialer{NetDialer: &net.Dialer{Timeout: timeout}, Config: c}
+			return d.DialContext(ctx, network, addr)
+		},
+	}}, nil
 }
 
+// A stream to a control plane whose machine stopped dead — powered off, cut
+// off — is never closed: nothing sends the FIN, and the agent's heartbeats sit
+// in the socket's buffer until the kernel gives up retransmitting, fifteen
+// minutes later. Found by switching off a leader: the standby took over in 14
+// seconds and the node stayed with the dead one. An HTTP/2 ping after ten
+// quiet seconds, given five to answer, finds it in a quarter of a minute.
+const (
+	streamIdle = 10 * time.Second
+	streamPing = 5 * time.Second
+)
+
 // localizeSpec fills in what only this node knows.
 //
 // The interface NCCL bootstraps over is this node's, not the cluster's: the

internal/agent/controlplanes.go

@@ -0,0 +1,77 @@
+package agent
+
+import (
+	"encoding/json"
+	"os"
+	"path/filepath"
+	"slices"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+)
+
+// controlPlanesFile keeps the control planes the leader last named, beside
+// the agent's configuration rather than in it: the configuration is the
+// operator's, and a list rewritten in it would be a file nobody wrote.
+var controlPlanesFile = "/var/lib/llmgw-agent/control-planes.json"
+
+// servers is every address to dial, in the order to try them: the
+// configuration's, then any the leader named that it lacks.
+func (a *Agent) servers() []string {
+	a.mu.Lock()
+	defer a.mu.Unlock()
+	if len(a.serverList) == 0 {
+		return a.cfg.Servers
+	}
+	return a.serverList
+}
+
+// mergeServers puts the leader's list after the configured one, without
+// repeats, and says whether that changed anything.
+func mergeServers(configured, told []string, secure bool) []string {
+	out := slices.Clone(configured)
+	for _, s := range told {
+		fixed, err := normalizeOne(s, secure)
+		if err != nil {
+			continue
+		}
+		if !slices.Contains(out, fixed) {
+			out = append(out, fixed)
+		}
+	}
+	return out
+}
+
+// setControlPlanes takes the leader's list: used from the next reconnect,
+// and kept for the next start.
+func (a *Agent) setControlPlanes(m *llmgwv1.ControlPlanes) {
+	told := m.GetServers()
+	merged := mergeServers(a.cfg.Servers, told, a.cfg.TLS.Enabled())
+	a.mu.Lock()
+	changed := !slices.Equal(merged, a.serverList)
+	a.serverList = merged
+	a.mu.Unlock()
+	if !changed {
+		return
+	}
+	a.log.Info("the control planes to dial", "servers", merged)
+	b, _ := json.Marshal(told)
+	if err := os.MkdirAll(filepath.Dir(controlPlanesFile), 0o755); err == nil {
+		tmp := controlPlanesFile + ".new"
+		if err := os.WriteFile(tmp, b, 0o644); err == nil {
+			_ = os.Rename(tmp, controlPlanesFile)
+		}
+	}
+}
+
+// loadControlPlanes is the list kept from the last time the leader named it.
+func loadControlPlanes(configured []string, secure bool) []string {
+	b, err := os.ReadFile(controlPlanesFile)
+	if err != nil {
+		return configured
+	}
+	var told []string
+	if json.Unmarshal(b, &told) != nil {
+		return configured
+	}
+	return mergeServers(configured, told, secure)
+}

internal/agent/controlplanes_test.go

@@ -0,0 +1,41 @@
+package agent
+
+import (
+	"log/slog"
+	"path/filepath"
+	"slices"
+	"testing"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
+)
+
+func TestAControlPlaneTheLeaderNamesIsDialledAndRemembered(t *testing.T) {
+	old := controlPlanesFile
+	controlPlanesFile = filepath.Join(t.TempDir(), "control-planes.json")
+	t.Cleanup(func() { controlPlanesFile = old })
+
+	configured := []string{"https://10.0.0.140:7443"}
+	// TLS on, as every onboarded node has it: an agent without it refuses
+	// https:// addresses, and the leader's are https://.
+	a := &Agent{cfg: Config{Servers: configured, TLS: tlsconf.Config{CA: "/etc/llmgw/ca.pem"}}, log: slog.New(slog.DiscardHandler)}
+	if got := a.servers(); !slices.Equal(got, configured) {
+		t.Fatalf("before anything is named: %v", got)
+	}
+	// The configured one stays first; the new one is added after it, once.
+	a.setControlPlanes(&llmgwv1.ControlPlanes{Servers: []string{"https://10.0.0.140:7443", "https://10.0.0.150:7443"}})
+	want := []string{"https://10.0.0.140:7443", "https://10.0.0.150:7443"}
+	if got := a.servers(); !slices.Equal(got, want) {
+		t.Errorf("after the leader named two: %v", got)
+	}
+	// And after a restart, before the leader says anything.
+	if got := loadControlPlanes(configured, true); !slices.Equal(got, want) {
+		t.Errorf("after a restart: %v", got)
+	}
+	// A list that drops the configured one does not drop it: the
+	// configuration is the operator's.
+	a.setControlPlanes(&llmgwv1.ControlPlanes{Servers: []string{"https://10.0.0.150:7443"}})
+	if got := a.servers(); got[0] != "https://10.0.0.140:7443" {
+		t.Errorf("the configured server was dropped: %v", got)
+	}
+}

internal/agent/disabled_test.go

@@ -0,0 +1,32 @@
+package agent
+
+import (
+	"testing"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
+)
+
+func TestAnEngineDisabledOnTheNodeIsNeitherInstalledNorAdvertised(t *testing.T) {
+	a := &Agent{}
+	a.cfg.Engines.Disabled = []string{"vllm"}
+	a.setProvision(&llmgwv1.Provision{Engines: []*llmgwv1.EngineWant{
+		{Engine: "llamacpp", Version: "b1"}, {Engine: "vllm", Version: "0.30.0"},
+	}})
+	if len(a.provWants) != 1 || a.provWants[0].Engine != "llamacpp" {
+		t.Errorf("wants %+v: vLLM is disabled here and must not be installed", a.provWants)
+	}
+	a.configuredVers = map[string]string{"vllm": "0.25.0"}
+	a.installedVers = map[string]string{"llamacpp": "b1"}
+	if got := a.engineReportLocked(); got["vllm"] != "" || got["llamacpp"] != "b1" {
+		t.Errorf("advertised %v: a disabled engine found on disk is still not offered", got)
+	}
+}
+
+func TestAnEngineNamedByHandIsNotInstalledOverIt(t *testing.T) {
+	a := &Agent{reg: engines.NewRegistry(engines.Config{LlamaCpp: engines.LlamaCppConfig{Bin: "/opt/own/llama-server"}})}
+	a.setProvision(&llmgwv1.Provision{Engines: []*llmgwv1.EngineWant{{Engine: "llamacpp", Version: "b1"}}})
+	if len(a.provWants) != 0 {
+		t.Errorf("wants %+v: an installed build would be used over the operator's own", a.provWants)
+	}
+}

internal/agent/discovery/gpu_smi.go

@@ -106,9 +106,27 @@ func parseSMI(out string) ([]GPU, error) {
 	return gpus, nil
 }
 
-// residentMemoryToleranceMB is how much allocated memory is written off as
+// ForeignMemoryToleranceMB is how much allocated memory is written off as
 // driver overhead rather than a foreign process.
-const residentMemoryToleranceMB = 512
+const ForeignMemoryToleranceMB = 512
+
+// foreignMemorySuffix ends the problem a card with another tenant on it is
+// kept out of placement for. One phrase, so that the control plane can find
+// it again and drop it when the tenant leaves.
+const foreignMemorySuffix = " MiB already allocated by another process"
+
+// ForeignMemoryProblem is the problem for mb of memory held by something
+// that is not one of this agent's engines, or "" when that is too little to
+// count.
+func ForeignMemoryProblem(mb int) string {
+	if mb <= ForeignMemoryToleranceMB {
+		return ""
+	}
+	return fmt.Sprintf("%d%s", mb, foreignMemorySuffix)
+}
+
+// IsForeignMemoryProblem reports whether a problem is ForeignMemoryProblem's.
+func IsForeignMemoryProblem(p string) bool { return strings.HasSuffix(p, foreignMemorySuffix) }
 
 // assessHealth flags conditions that should keep a GPU out of placement.
 func assessHealth(g *GPU) {
@@ -119,8 +137,12 @@ func assessHealth(g *GPU) {
 	// Memory already in use means something else is on the card. Starting an
 	// engine there would either fail to allocate or quietly evict the other
 	// process's throughput.
-	if g.MemoryUsedMB > residentMemoryToleranceMB {
-		g.Health.Problems = append(g.Health.Problems,
-			fmt.Sprintf("%d MiB already allocated by another process", g.MemoryUsedMB))
+	//
+	// Only a first reading: this runs before the agent knows which processes
+	// are its own, so an engine still running from before a restart reads as
+	// foreign here. Every heartbeat measures it again, without the agent's
+	// own engines, and the control plane replaces this with that.
+	if p := ForeignMemoryProblem(g.MemoryUsedMB); p != "" {
+		g.Health.Problems = append(g.Health.Problems, p)
 	}
 }

internal/agent/discovery/gpuhealth.go

@@ -0,0 +1,280 @@
+package discovery
+
+import (
+	"context"
+	"encoding/xml"
+	"os/exec"
+	"regexp"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+)
+
+// GPUWear is what wears a card out or takes it down, beyond temperature:
+// read every minute, from nvidia-smi and the kernel's log. The early signs of
+// a card failing are here — a growing count of rows its memory had to remap,
+// PCIe link retries, XID errors — long before a replica on it fails.
+//
+// A count of -1 is not reported: a card without row remapping (before
+// Ampere) has no such count, which is not the same as none.
+type GPUWear struct {
+	Index int
+	// PCIeReplays are link-level retries on the card's PCIe link: a few are
+	// noise, a count that keeps climbing is a riser, slot or cable.
+	PCIeReplays int64
+	// RetiredSBE and RetiredDBE are memory pages retired for single and
+	// double bit errors (cards before Ampere); RetirePending is a page
+	// waiting for a reset to be retired.
+	RetiredSBE, RetiredDBE int64
+	RetirePending          bool
+	// RemappedCorrectable and RemappedUncorrectable are memory rows remapped
+	// (Ampere and later). RemapPending waits for a reset; RemapFailure is a
+	// card out of spare rows — it needs replacing.
+	RemappedCorrectable, RemappedUncorrectable int64
+	RemapPending, RemapFailure                 bool
+	// NVLink errors summed over the card's links, by kind.
+	NVLinkCRC, NVLinkReplay, NVLinkRecovery int64
+	// XIDs are the driver's error events for this card since the agent
+	// started, and the last of them.
+	XIDs        int64
+	LastXID     int
+	LastXIDText string
+	LastXIDAt   time.Time
+}
+
+// healthEvery is how often health is read: it changes over hours, and the
+// kernel log and nvidia-smi's full report are the heavier reads.
+const healthEvery = time.Minute
+
+// HealthSampler reads GPUWear, keeping the XID counts between reads.
+type HealthSampler struct {
+	run func(ctx context.Context, name string, args ...string) ([]byte, error)
+	now func() time.Time
+
+	mu      sync.Mutex
+	at      time.Time
+	last    []GPUWear
+	since   time.Time
+	xids    map[int]*GPUWear
+	started bool
+}
+
+// NewHealthSampler reads the machine's own nvidia-smi and kernel log.
+func NewHealthSampler() *HealthSampler {
+	return &HealthSampler{now: time.Now, xids: map[int]*GPUWear{},
+		run: func(ctx context.Context, name string, args ...string) ([]byte, error) {
+			return exec.CommandContext(ctx, name, args...).Output() //nolint:gosec // fixed tools
+		}}
+}
+
+// Sample is the cards' health, read again once healthEvery has passed. Nil
+// on a machine without nvidia-smi.
+func (h *HealthSampler) Sample(ctx context.Context) []GPUWear {
+	h.mu.Lock()
+	defer h.mu.Unlock()
+	now := h.now()
+	if !h.at.IsZero() && now.Sub(h.at) < healthEvery {
+		return h.last
+	}
+	h.at = now
+	ctx, cancel := context.WithTimeout(ctx, 15*time.Second)
+	defer cancel()
+	report, err := h.run(ctx, "nvidia-smi", "-q", "-x")
+	if err != nil {
+		h.last = nil
+		return nil
+	}
+	cards, bus := parseHealthReport(report)
+	if out, err := h.run(ctx, "nvidia-smi", "nvlink", "-e"); err == nil {
+		for i, e := range parseNVLinkErrors(string(out)) {
+			if i < len(cards) {
+				cards[i].NVLinkCRC, cards[i].NVLinkReplay, cards[i].NVLinkRecovery = e[0], e[1], e[2]
+			}
+		}
+	}
+	h.readXIDs(ctx, now, bus)
+	for i := range cards {
+		if x := h.xids[i]; x != nil {
+			cards[i].XIDs, cards[i].LastXID, cards[i].LastXIDText, cards[i].LastXIDAt = x.XIDs, x.LastXID, x.LastXIDText, x.LastXIDAt
+		}
+	}
+	h.last = cards
+	return cards
+}
+
+// readXIDs adds the XID events logged since the last read. The first read
+// starts from now: what happened before the agent started was before, and
+// counting the whole log would raise every alert on every restart.
+func (h *HealthSampler) readXIDs(ctx context.Context, now time.Time, bus map[string]int) {
+	if !h.started {
+		h.started, h.since = true, now
+		return
+	}
+	since := h.since
+	h.since = now
+	out, err := h.run(ctx, "journalctl", "-k", "--no-pager", "-o", "short-unix",
+		"--since", "@"+strconv.FormatInt(since.Unix(), 10), "-g", "NVRM: Xid")
+	if err != nil {
+		// No journal, or nothing matched (journalctl exits 1 for that).
+		return
+	}
+	for _, e := range parseXIDs(string(out)) {
+		i, ok := bus[e.bus]
+		if !ok {
+			continue
+		}
+		x := h.xids[i]
+		if x == nil {
+			x = &GPUWear{Index: i}
+			h.xids[i] = x
+		}
+		x.XIDs++
+		x.LastXID, x.LastXIDText = e.code, e.text
+		x.LastXIDAt = now
+		if e.at > 0 {
+			x.LastXIDAt = time.Unix(int64(e.at), 0)
+		}
+	}
+}
+
+// healthLog is the part of nvidia-smi's XML report health is read from.
+type healthLog struct {
+	GPUs []struct {
+		ID  string `xml:"id,attr"`
+		PCI struct {
+			Replay string `xml:"replay_counter"`
+		} `xml:"pci"`
+		Retired struct {
+			SBE     string `xml:"multiple_single_bit_retirement>retired_count"`
+			DBE     string `xml:"double_bit_retirement>retired_count"`
+			Pending string `xml:"pending_retirement"`
+			// Older drivers name it pending_blacklist.
+			Blacklist string `xml:"pending_blacklist"`
+		} `xml:"retired_pages"`
+		Remapped struct {
+			Corr    string `xml:"remapped_row_corr"`
+			Unc     string `xml:"remapped_row_unc"`
+			Pending string `xml:"remapped_row_pending"`
+			Failure string `xml:"remapped_row_failure"`
+		} `xml:"remapped_rows"`
+	} `xml:"gpu"`
+}
+
+// parseHealthReport reads each card's health, in index order, and maps each
+// card's PCI bus ("3b:00", as the kernel names it) to its index.
+func parseHealthReport(report []byte) ([]GPUWear, map[string]int) {
+	var log healthLog
+	if err := xml.Unmarshal(report, &log); err != nil {
+		return nil, nil
+	}
+	bus := map[string]int{}
+	out := make([]GPUWear, 0, len(log.GPUs))
+	for i, g := range log.GPUs {
+		bus[busOf(g.ID)] = i
+		out = append(out, GPUWear{
+			Index:                 i,
+			PCIeReplays:           count(g.PCI.Replay),
+			RetiredSBE:            count(g.Retired.SBE),
+			RetiredDBE:            count(g.Retired.DBE),
+			RetirePending:         yes(g.Retired.Pending) || yes(g.Retired.Blacklist),
+			RemappedCorrectable:   count(g.Remapped.Corr),
+			RemappedUncorrectable: count(g.Remapped.Unc),
+			RemapPending:          yes(g.Remapped.Pending),
+			RemapFailure:          yes(g.Remapped.Failure),
+		})
+	}
+	return out, bus
+}
+
+// busOf is a PCI address as the kernel's XID lines carry it: nvidia-smi
+// says "00000000:3B:00.0", the kernel "0000:3b:00".
+func busOf(id string) string {
+	id = strings.ToLower(strings.TrimSpace(id))
+	if i := strings.LastIndex(id, "."); i >= 0 {
+		id = id[:i]
+	}
+	parts := strings.Split(id, ":")
+	if len(parts) >= 2 {
+		return parts[len(parts)-2] + ":" + parts[len(parts)-1]
+	}
+	return id
+}
+
+func count(s string) int64 {
+	n, err := strconv.ParseInt(strings.TrimSpace(s), 10, 64)
+	if err != nil {
+		return -1
+	}
+	return n
+}
+
+func yes(s string) bool { return strings.EqualFold(strings.TrimSpace(s), "yes") }
+
+// nvlinkError matches a line of `nvidia-smi nvlink -e`, whose wording varies
+// by driver: "Link 0: Replay Errors: 0", "Link 0: CRC Errors: 0",
+// "Link 3: Data CRC Errors: 2".
+var (
+	nvlinkGPU   = regexp.MustCompile(`^GPU (\d+):`)
+	nvlinkError = regexp.MustCompile(`Link \d+:\s*(.*?)\s*Errors?\s*:\s*(\d+)`)
+)
+
+// parseNVLinkErrors sums each card's link errors by kind: CRC, replay,
+// recovery.
+func parseNVLinkErrors(out string) map[int][3]int64 {
+	res := map[int][3]int64{}
+	gpu := -1
+	for _, line := range strings.Split(out, "\n") {
+		if m := nvlinkGPU.FindStringSubmatch(strings.TrimSpace(line)); m != nil {
+			gpu, _ = strconv.Atoi(m[1])
+			continue
+		}
+		m := nvlinkError.FindStringSubmatch(line)
+		if m == nil || gpu < 0 {
+			continue
+		}
+		n, _ := strconv.ParseInt(m[2], 10, 64)
+		e := res[gpu]
+		kind := strings.ToLower(m[1])
+		switch {
+		case strings.Contains(kind, "crc"):
+			e[0] += n
+		case strings.Contains(kind, "replay"):
+			e[1] += n
+		case strings.Contains(kind, "recovery"):
+			e[2] += n
+		}
+		res[gpu] = e
+	}
+	return res
+}
+
+type xidEvent struct {
+	bus  string
+	code int
+	text string
+	at   float64
+}
+
+// xidLine matches the driver's report of an XID:
+// "NVRM: Xid (PCI:0000:3b:00): 79, pid=..., GPU has fallen off the bus."
+var xidLine = regexp.MustCompile(`NVRM: Xid \(PCI:([0-9a-fA-F:.]+)\): (\d+),\s*(.*)$`)
+
+// parseXIDs reads journalctl's short-unix lines: a timestamp, then the
+// message.
+func parseXIDs(out string) []xidEvent {
+	var evs []xidEvent
+	for _, line := range strings.Split(out, "\n") {
+		m := xidLine.FindStringSubmatch(line)
+		if m == nil {
+			continue
+		}
+		code, _ := strconv.Atoi(m[2])
+		e := xidEvent{bus: busOf(m[1]), code: code, text: strings.TrimSpace(m[3])}
+		if f := strings.Fields(line); len(f) > 0 {
+			e.at, _ = strconv.ParseFloat(f[0], 64)
+		}
+		evs = append(evs, e)
+	}
+	return evs
+}

internal/agent/discovery/gpuhealth_test.go

@@ -0,0 +1,101 @@
+package discovery
+
+import (
+	"context"
+	"errors"
+	"strings"
+	"testing"
+	"time"
+)
+
+const healthXML = `<?xml version="1.0" ?>
+<nvidia_smi_log>
+  <gpu id="00000000:3B:00.0">
+    <pci><replay_counter>0</replay_counter></pci>
+    <retired_pages>
+      <multiple_single_bit_retirement><retired_count>N/A</retired_count></multiple_single_bit_retirement>
+      <double_bit_retirement><retired_count>N/A</retired_count></double_bit_retirement>
+      <pending_retirement>N/A</pending_retirement>
+    </retired_pages>
+    <remapped_rows>
+      <remapped_row_corr>2</remapped_row_corr>
+      <remapped_row_unc>1</remapped_row_unc>
+      <remapped_row_pending>Yes</remapped_row_pending>
+      <remapped_row_failure>No</remapped_row_failure>
+    </remapped_rows>
+  </gpu>
+  <gpu id="00000000:5E:00.0">
+    <pci><replay_counter>4021</replay_counter></pci>
+    <retired_pages>
+      <multiple_single_bit_retirement><retired_count>3</retired_count></multiple_single_bit_retirement>
+      <double_bit_retirement><retired_count>1</retired_count></double_bit_retirement>
+      <pending_blacklist>Yes</pending_blacklist>
+    </retired_pages>
+    <remapped_rows>N/A</remapped_rows>
+  </gpu>
+</nvidia_smi_log>`
+
+const nvlinkOut = `GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-aaa)
+	 Link 0: Replay Errors: 1
+	 Link 0: Recovery Errors: 0
+	 Link 0: CRC Errors: 5
+	 Link 1: Data CRC Errors: 2
+GPU 1: NVIDIA H100 80GB HBM3 (UUID: GPU-bbb)
+	 Link 0: Replay Errors: 0
+`
+
+func TestAGPUsWearAndFailuresAreReadAndCountedFromWhenTheAgentStarted(t *testing.T) {
+	now := time.Unix(1_800_000_000, 0)
+	journal := ""
+	h := &HealthSampler{now: func() time.Time { return now }, xids: map[int]*GPUWear{},
+		run: func(_ context.Context, name string, args ...string) ([]byte, error) {
+			switch {
+			case name == "nvidia-smi" && args[0] == "-q":
+				return []byte(healthXML), nil
+			case name == "nvidia-smi" && args[0] == "nvlink":
+				return []byte(nvlinkOut), nil
+			case name == "journalctl":
+				if journal == "" {
+					return nil, errors.New("exit status 1")
+				}
+				return []byte(journal), nil
+			}
+			return nil, errors.New("unexpected " + name)
+		}}
+	got := h.Sample(t.Context())
+	if len(got) != 2 {
+		t.Fatalf("%+v", got)
+	}
+	a, b := got[0], got[1]
+	if a.RemappedCorrectable != 2 || a.RemappedUncorrectable != 1 || !a.RemapPending || a.RemapFailure ||
+		a.RetiredSBE != -1 || a.PCIeReplays != 0 || a.NVLinkCRC != 7 || a.NVLinkReplay != 1 {
+		t.Errorf("card 0 %+v", a)
+	}
+	if b.PCIeReplays != 4021 || b.RetiredSBE != 3 || b.RetiredDBE != 1 || !b.RetirePending || b.RemappedCorrectable != -1 {
+		t.Errorf("card 1 %+v", b)
+	}
+
+	// An XID on card 1, logged after the agent started; read on the next
+	// sample a minute on.
+	journal = "1800000030.123456 gpu07 kernel: NVRM: Xid (PCI:0000:5e:00): 79, pid=1234, name=llama-server, GPU has fallen off the bus.\n"
+	now = now.Add(61 * time.Second)
+	got = h.Sample(t.Context())
+	if got[1].XIDs != 1 || got[1].LastXID != 79 || !strings.Contains(got[1].LastXIDText, "fallen off the bus") ||
+		got[1].LastXIDAt.Unix() != 1_800_000_030 || got[0].XIDs != 0 {
+		t.Errorf("after an XID: %+v / %+v", got[0], got[1])
+	}
+	// Within the minute, the last reading.
+	journal = ""
+	now = now.Add(10 * time.Second)
+	if again := h.Sample(t.Context()); again[1].XIDs != 1 {
+		t.Errorf("%+v", again[1])
+	}
+}
+
+func TestBusIDsAreReadAsTheKernelWritesThem(t *testing.T) {
+	for in, want := range map[string]string{"00000000:3B:00.0": "3b:00", "0000:3b:00": "3b:00", "0000:5e:00.0": "5e:00"} {
+		if got := busOf(in); got != want {
+			t.Errorf("%s: %s, want %s", in, got, want)
+		}
+	}
+}

internal/agent/discovery/gpuprocs.go

@@ -0,0 +1,120 @@
+package discovery
+
+import (
+	"context"
+	"encoding/xml"
+	"fmt"
+	"os/exec"
+	"strconv"
+	"strings"
+)
+
+// GPUProcesses reports which processes hold memory on each GPU, by the GPU's
+// nvidia-smi index: what a process actually computes on, as opposed to what it
+// was told to use. A llama.cpp started with --n-gpu-layers on a GPU whose
+// backend did not load runs on the CPU without a word, and this is where that
+// shows — the process is on no GPU's list.
+//
+// From the full query rather than --query-compute-apps, which lists compute
+// processes only: llama.cpp's Vulkan build is listed as a graphics one, and
+// would have read as not on its GPU at all.
+//
+// A node with no nvidia-smi returns nil and no error, as discovery does.
+func GPUProcesses(ctx context.Context, binary string) (map[int][]int, error) {
+	if binary == "" {
+		if _, err := exec.LookPath("nvidia-smi"); err != nil {
+			return nil, nil
+		}
+		binary = "nvidia-smi"
+	}
+	out, err := exec.CommandContext(ctx, binary, "-q", "-x").Output()
+	if err != nil {
+		return nil, fmt.Errorf("running %s: %w", binary, err)
+	}
+	return parseGPUProcesses(out)
+}
+
+// GPUMemory is one card's memory as nvidia-smi reports it at a moment: what
+// is allocated on it, and which processes hold how much of that.
+type GPUMemory struct {
+	UsedMB int
+	Procs  []GPUProcess
+}
+
+// GPUProcess is one process holding memory on a card.
+type GPUProcess struct {
+	PID    int
+	UsedMB int
+}
+
+// GPUMemoryByProcess reports each GPU's memory, by index, with the processes
+// holding it — from the same report as GPUProcesses, so the two never
+// disagree about who is on a card.
+//
+// A node with no nvidia-smi returns nil and no error, as discovery does.
+func GPUMemoryByProcess(ctx context.Context, binary string) ([]GPUMemory, error) {
+	if binary == "" {
+		if _, err := exec.LookPath("nvidia-smi"); err != nil {
+			return nil, nil
+		}
+		binary = "nvidia-smi"
+	}
+	out, err := exec.CommandContext(ctx, binary, "-q", "-x").Output()
+	if err != nil {
+		return nil, fmt.Errorf("running %s: %w", binary, err)
+	}
+	return parseGPUMemory(out)
+}
+
+// smiLog is the part of nvidia-smi's XML report this reads: each GPU, in
+// index order, its memory and the processes on it.
+type smiLog struct {
+	GPUs []struct {
+		Used      string `xml:"fb_memory_usage>used"`
+		Processes []struct {
+			PID    string `xml:"pid"`
+			Type   string `xml:"type"`
+			UsedMB string `xml:"used_memory"`
+		} `xml:"processes>process_info"`
+	} `xml:"gpu"`
+}
+
+// parseGPUProcesses reads the report. GPUs are listed in index order, which
+// is nvidia-smi's own numbering — the one placement assigns by.
+func parseGPUProcesses(report []byte) (map[int][]int, error) {
+	mem, err := parseGPUMemory(report)
+	if err != nil {
+		return nil, err
+	}
+	out := map[int][]int{}
+	for i, g := range mem {
+		for _, p := range g.Procs {
+			out[i] = append(out[i], p.PID)
+		}
+	}
+	return out, nil
+}
+
+// parseGPUMemory reads the report into each GPU's memory and its holders.
+func parseGPUMemory(report []byte) ([]GPUMemory, error) {
+	var log smiLog
+	if err := xml.Unmarshal(report, &log); err != nil {
+		return nil, fmt.Errorf("reading nvidia-smi's report: %w", err)
+	}
+	out := make([]GPUMemory, len(log.GPUs))
+	for i, g := range log.GPUs {
+		out[i].UsedMB = mib(g.Used)
+		for _, p := range g.Processes {
+			if pid, err := strconv.Atoi(strings.TrimSpace(p.PID)); err == nil {
+				out[i].Procs = append(out[i].Procs, GPUProcess{PID: pid, UsedMB: mib(p.UsedMB)})
+			}
+		}
+	}
+	return out, nil
+}
+
+// mib reads "4932 MiB"; anything else, "N/A" included, is zero.
+func mib(s string) int {
+	n, _ := strconv.Atoi(strings.TrimSpace(strings.TrimSuffix(strings.TrimSpace(s), "MiB")))
+	return n
+}

internal/agent/discovery/gpuprocs_test.go

@@ -0,0 +1,78 @@
+package discovery
+
+import (
+	"reflect"
+	"testing"
+)
+
+func TestGPUProcessesOfEveryKindAreListedByIndex(t *testing.T) {
+	report := `<?xml version="1.0" ?>
+<nvidia_smi_log>
+  <driver_version>580.65.06</driver_version>
+  <gpu id="00000000:17:00.0"><product_name>Tesla V100-SXM2-32GB</product_name>
+    <processes></processes>
+  </gpu>
+  <gpu id="00000000:65:00.0"><product_name>Tesla V100-SXM2-32GB</product_name>
+    <processes>
+      <process_info><pid>4100</pid><type>C</type><process_name>llama-server</process_name></process_info>
+      <process_info><pid>4300</pid><type>G</type><process_name>llama-server</process_name></process_info>
+    </processes>
+  </gpu>
+  <gpu id="00000000:B3:00.0"><processes>
+      <process_info><pid>4200</pid><type>C+G</type></process_info>
+  </processes></gpu>
+</nvidia_smi_log>`
+	got, err := parseGPUProcesses([]byte(report))
+	if err != nil {
+		t.Fatal(err)
+	}
+	// A Vulkan engine is a graphics process ("G") and is on its GPU all the
+	// same; a compute-only query would have missed it.
+	want := map[int][]int{1: {4100, 4300}, 2: {4200}}
+	if !reflect.DeepEqual(got, want) {
+		t.Errorf("got %v, want %v", got, want)
+	}
+	if _, err := parseGPUProcesses([]byte("not xml")); err == nil {
+		t.Error("a report that does not parse was read as no processes")
+	}
+}
+
+func TestGPUMemoryIsReadPerProcess(t *testing.T) {
+	report := `<?xml version="1.0" ?>
+<nvidia_smi_log>
+  <gpu id="00000000:01:00.0"><product_name>Quadro RTX 6000</product_name>
+    <fb_memory_usage><total>24576 MiB</total><used>14514 MiB</used><free>10062 MiB</free></fb_memory_usage>
+    <processes>
+      <process_info><pid>71315</pid><type>C</type><used_memory>4932 MiB</used_memory></process_info>
+      <process_info><pid>4100</pid><type>C</type><used_memory>9500 MiB</used_memory></process_info>
+    </processes>
+  </gpu>
+  <gpu id="00000000:02:00.0">
+    <fb_memory_usage><used>N/A</used></fb_memory_usage>
+    <processes></processes>
+  </gpu>
+</nvidia_smi_log>`
+	got, err := parseGPUMemory([]byte(report))
+	if err != nil {
+		t.Fatal(err)
+	}
+	want := []GPUMemory{
+		{UsedMB: 14514, Procs: []GPUProcess{{PID: 71315, UsedMB: 4932}, {PID: 4100, UsedMB: 9500}}},
+		{},
+	}
+	if !reflect.DeepEqual(got, want) {
+		t.Errorf("got %+v, want %+v", got, want)
+	}
+}
+
+func TestForeignMemoryBelowTheToleranceIsNoProblem(t *testing.T) {
+	// A CUDA context that has let go of its model still holds a few hundred
+	// MiB: ComfyUI after /free held 324.
+	if p := ForeignMemoryProblem(330); p != "" {
+		t.Errorf("330 MiB is a problem: %q", p)
+	}
+	p := ForeignMemoryProblem(5014)
+	if !IsForeignMemoryProblem(p) || IsForeignMemoryProblem("2 uncorrected ECC errors") {
+		t.Errorf("%q is not recognised as the foreign memory problem, or ECC is", p)
+	}
+}

internal/agent/discovery/sensors.go

@@ -41,6 +41,14 @@ type GPUSensor struct {
 	// is running free. Idle and application clock settings are left out;
 	// they are not the card being held back.
 	Throttle []string
+	// How busy it is, read with the rest every sample. Discovery reads the
+	// same once, at start, which made a busy card look as idle as it was
+	// when the agent came up. -1 is not reported by the driver; zero
+	// MemoryTotalMB is a sample without these at all.
+	UtilPercent, MemUtilPercent float64
+	MemoryUsedMB, MemoryTotalMB float64
+	SMClockMHz                  float64
+	ECCUncorrected              float64
 }
 
 // sensorEvery bounds how often nvidia-smi is run. Heartbeats come more often
@@ -187,13 +195,33 @@ var throttleBits = []struct {
 // gpus asks nvidia-smi for each card's heat and power. The memory temperature
 // is not known to every driver, and a field nvidia-smi does not know fails the
 // whole query, so it is asked for first and dropped on failure.
+// gpuFields are what each sample asks nvidia-smi for, in order. The busy
+// figures follow the throttle reasons; the memory temperature is last, being
+// the one field a driver may reject outright.
+const (
+	gpuBase  = "index,temperature.gpu,power.draw,enforced.power.limit,clocks_throttle_reasons.active"
+	gpuBusy  = ",utilization.gpu,utilization.memory,memory.used,memory.total,clocks.sm,ecc.errors.uncorrected.volatile.total"
+	gpuHBMTC = ",temperature.memory"
+)
+
 func (s *SensorSampler) gpus(ctx context.Context) []GPUSensor {
-	withMem := "index,temperature.gpu,power.draw,enforced.power.limit,clocks_throttle_reasons.active,temperature.memory"
-	out, err := s.smi(ctx, withMem)
-	if err != nil {
-		if out, err = s.smi(ctx, strings.TrimSuffix(withMem, ",temperature.memory")); err != nil {
-			return nil
+	// Most fields first, fewer on a driver that refuses one: a query naming
+	// a field the driver does not know fails as a whole.
+	var out []byte
+	var busy bool
+	for _, q := range []struct {
+		fields string
+		busy   bool
+	}{{gpuBase + gpuBusy + gpuHBMTC, true}, {gpuBase + gpuBusy, true}, {gpuBase + gpuHBMTC, false}, {gpuBase, false}} {
+		var err error
+		if out, err = s.smi(ctx, q.fields); err == nil {
+			busy = q.busy
+			break
 		}
+		out = nil
+	}
+	if out == nil {
+		return nil
 	}
 	var gs []GPUSensor
 	for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
@@ -216,14 +244,31 @@ func (s *SensorSampler) gpus(ctx context.Context) []GPUSensor {
 				}
 			}
 		}
-		if len(f) > 5 {
-			g.MemoryTempC = num(f[5])
+		rest := f[5:]
+		if busy && len(rest) >= 6 {
+			g.UtilPercent, g.MemUtilPercent = numOrUnknown(rest[0]), numOrUnknown(rest[1])
+			g.MemoryUsedMB, g.MemoryTotalMB = num(rest[2]), num(rest[3])
+			g.SMClockMHz, g.ECCUncorrected = num(rest[4]), numOrUnknown(rest[5])
+			rest = rest[6:]
+		}
+		if len(rest) > 0 {
+			g.MemoryTempC = num(rest[0])
 		}
 		gs = append(gs, g)
 	}
 	return gs
 }
 
+// numOrUnknown is a figure, or -1 where nvidia-smi says it has none: a card
+// without ECC has no error count, which is not the same as none.
+func numOrUnknown(s string) float64 {
+	v, err := strconv.ParseFloat(s, 64)
+	if err != nil || v < 0 {
+		return -1
+	}
+	return v
+}
+
 // num reads a figure nvidia-smi may give as "[N/A]" or "N/A".
 func num(s string) float64 {
 	v, err := strconv.ParseFloat(s, 64)

internal/agent/discovery/sensors_test.go

@@ -167,3 +167,44 @@ func TestADriverThatDoesNotKnowMemoryTemperatureStillReportsTheRest(t *testing.T
 		t.Errorf("%+v", got.GPUs)
 	}
 }
+
+func TestEachSampleSaysHowBusyEachGPUIs(t *testing.T) {
+	// What the card is doing now, every sample: discovery read it once, at
+	// start, and a card busy since looked as idle as it was then.
+	smi := func(_ context.Context, fields string) ([]byte, error) {
+		if !strings.Contains(fields, "utilization.gpu") {
+			return nil, errors.New("asked without how busy the cards are")
+		}
+		return []byte(
+			"0, 71, 612.45, 700.00, 0x0000000000000000, 97, 64, 71234, 81559, 1980, 0, 84\n" +
+				"1, 30, 70.10, 700.00, 0x0000000000000001, 0, 0, 512, 81559, 345, [N/A], [N/A]\n"), nil
+	}
+	now := time.Now()
+	got := sampler(t.TempDir(), smi, &now).Sample(t.Context()).GPUs
+	if len(got) != 2 {
+		t.Fatalf("%+v", got)
+	}
+	g0, g1 := got[0], got[1]
+	if g0.UtilPercent != 97 || g0.MemUtilPercent != 64 || g0.MemoryUsedMB != 71234 || g0.MemoryTotalMB != 81559 ||
+		g0.SMClockMHz != 1980 || g0.ECCUncorrected != 0 || g0.MemoryTempC != 84 {
+		t.Errorf("card 0 %+v", g0)
+	}
+	// Idle is zero; a count the card does not keep is unknown, not zero.
+	if g1.UtilPercent != 0 || g1.ECCUncorrected != -1 || g1.MemoryTempC != 0 {
+		t.Errorf("card 1 %+v", g1)
+	}
+}
+
+func TestADriverThatDoesNotKnowTheBusyFieldsStillReportsHeatAndPower(t *testing.T) {
+	smi := func(_ context.Context, fields string) ([]byte, error) {
+		if strings.Contains(fields, "utilization.gpu") {
+			return nil, errors.New(`Field "ecc.errors.uncorrected.volatile.total" is not a valid field to query.`)
+		}
+		return []byte("0, 66, 250.0, 300.0, 0x0000000000000000, 70\n"), nil
+	}
+	now := time.Now()
+	got := sampler(t.TempDir(), smi, &now).Sample(t.Context()).GPUs
+	if len(got) != 1 || got[0].TempC != 66 || got[0].MemoryTempC != 70 || got[0].MemoryTotalMB != 0 {
+		t.Errorf("%+v", got)
+	}
+}

internal/agent/engines/engine.go

@@ -9,6 +9,7 @@
 package engines
 
 import (
+	"context"
 	"fmt"
 	"os"
 	"slices"
@@ -146,6 +147,15 @@ type Engine interface {
 	MetricsPath() string
 }
 
+// CPUFallback is an engine that can come up on the CPU when it was placed on
+// GPUs, and keep serving there without an error. The agent checks such a shard
+// is on its GPUs before calling it ready.
+type CPUFallback interface {
+	FallsBackToCPU() bool
+	// Devices lists the GPUs the engine's build can see, for saying why.
+	Devices(ctx context.Context) ([]string, error)
+}
+
 // GPUInfo is the subset of GPU facts validation needs.
 type GPUInfo struct {
 	Index             int
@@ -203,6 +213,10 @@ func (r *Registry) Names() []string {
 
 // Config configures the engine adapters.
 type Config struct {
+	// Disabled are engines this node never installs or advertises, whatever
+	// the control plane asks: vLLM on a node of cards no build of it runs on,
+	// that is meant for llama.cpp alone.
+	Disabled []string       `koanf:"disabled"`
 	VLLM     VLLMConfig     `koanf:"vllm"`
 	SGLang   SGLangConfig   `koanf:"sglang"`
 	LlamaCpp LlamaCppConfig `koanf:"llamacpp"`
@@ -232,6 +246,11 @@ func NCCLEnv(s ShardSpec) map[string]string { return baseEnv(s) }
 func baseEnv(s ShardSpec) map[string]string {
 	env := map[string]string{
 		"CUDA_VISIBLE_DEVICES": joinInts(s.GPUIndices),
+		// The indices above are nvidia-smi's, which counts by PCI bus. CUDA
+		// counts fastest first unless told otherwise, and on a node with two
+		// kinds of GPU the shard would get a different card than placement
+		// gave it — one another shard may hold.
+		"CUDA_DEVICE_ORDER": "PCI_BUS_ID",
 	}
 
 	if len(s.HCAs) > 0 {

internal/agent/engines/llamacpp.go

@@ -1,8 +1,10 @@
 package engines
 
 import (
+	"context"
 	"fmt"
 	"os"
+	"os/exec"
 	"path/filepath"
 	"runtime"
 	"sort"
@@ -65,25 +67,25 @@ func (*LlamaCpp) ServesHTTP(ShardSpec) bool { return true }
 
 // Validate checks the profile against the GPUs it would run on.
 //
-// The two refusals here are about not pretending: llama.cpp has no equivalent
-// of a tensor-parallel group spanning machines, and its multi-GPU support is a
-// different mechanism (--split-mode, --tensor-split) that this adapter does not
-// drive. Accepting either would produce a replica that starts and then serves
-// from one machine while the catalog believes it has several.
+// llama.cpp spreads a model over a node's GPUs by layer (--split-mode layer),
+// so gpus_per_node is how many it gets. It has no tensor parallelism and no
+// group spanning machines, and accepting tp or nodes would produce a replica
+// that serves from fewer cards, or one machine, while the catalog believes
+// otherwise.
 func (*LlamaCpp) Validate(p Profile, gpus []GPUInfo) error {
-	if err := validateCommon(p, gpus); err != nil {
-		return err
-	}
 	if p.Nodes > 1 {
 		return fmt.Errorf("profile %q: llamacpp does not span nodes; use vllm or sglang for a %d-node replica",
 			p.Name, p.Nodes)
 	}
-	if p.TP > 1 {
-		return fmt.Errorf("profile %q: llamacpp does not do tensor parallelism; "+
-			"it splits a model across local GPUs with --tensor-split, which this adapter does not drive",
-			p.Name)
+	if p.TP > 1 || p.PP > 1 {
+		return fmt.Errorf("profile %q: llamacpp splits a model across a node's GPUs by layer, not by tp or pp; "+
+			"write gpus_per_node: %d instead", p.Name, max(p.TP, 1)*max(p.PP, 1))
 	}
-	return nil
+	// The common checks count a replica's GPUs as tp; for llama.cpp the
+	// count is gpus_per_node.
+	q := p
+	q.TP = max(p.GPUsPerNode, 1)
+	return validateCommon(q, gpus)
 }
 
 // Commands returns the processes for one shard, which is always one process.
@@ -132,6 +134,13 @@ func (l *LlamaCpp) Commands(s ShardSpec) ([]Command, error) {
 	if len(s.GPUIndices) > 0 && !hasFlag(p.Args, "--n-gpu-layers", "-ngl") {
 		args = append(args, "--n-gpu-layers", "999")
 	}
+	// Several GPUs: the layers spread across all of them, which is the only
+	// split llama.cpp does without a matched interconnect. In proportion to
+	// each card's free memory, which for the whole cards placement gives is
+	// an even split.
+	if len(s.GPUIndices) > 1 && !hasFlag(p.Args, "--split-mode", "-sm") {
+		args = append(args, "--split-mode", "layer")
+	}
 	if len(s.GPUIndices) == 0 && !hasFlag(p.Args, "--threads", "-t") {
 		// What placement reserved first: it is what the node's other
 		// replicas were placed around.
@@ -159,6 +168,14 @@ func (l *LlamaCpp) Commands(s ShardSpec) ([]Command, error) {
 	if s.APIKey != "" {
 		env["LLAMA_API_KEY"] = s.APIKey
 	}
+	// The Vulkan build — what a GPU older than the CUDA builds gets — does
+	// not read CUDA_VISIBLE_DEVICES; this is its own. Vulkan numbers devices
+	// as the driver lists them, which on NVIDIA follows the same bus order
+	// as nvidia-smi; were it not to, the shard would not hold memory on its
+	// own GPUs, and starting it checks exactly that.
+	if len(s.GPUIndices) > 0 {
+		env["GGML_VK_VISIBLE_DEVICES"] = joinInts(s.GPUIndices)
+	}
 	return []Command{{
 		Name: "llama-server", Path: bin, Args: args, Env: materializeEnv(env),
 	}}, nil
@@ -234,3 +251,38 @@ func (l *LlamaCpp) bin() string {
 	}
 	return "llama-server"
 }
+
+// FallsBackToCPU says that llama.cpp keeps running when its GPU backend does
+// not load: --n-gpu-layers asks for layers on a GPU it then has none of, and
+// the server comes up and answers on the CPU, at a tenth of the speed, with no
+// error anywhere. Its readiness on a GPU placement needs checking separately.
+func (*LlamaCpp) FallsBackToCPU() bool { return true }
+
+// Devices lists the GPUs this build can compute on, as llama-server's
+// --list-devices names them ("CUDA0: NVIDIA L4 (22994 MiB, 22713 MiB free)").
+// Empty, on a machine with GPUs, is a build whose GPU backend did not load:
+// the CPU build, or a CUDA build missing a library its backend links.
+func (l *LlamaCpp) Devices(ctx context.Context) ([]string, error) {
+	out, err := exec.CommandContext(ctx, l.bin(), "--list-devices").CombinedOutput() //nolint:gosec // the engine's own binary
+	if err != nil {
+		return nil, fmt.Errorf("%s --list-devices: %v: %s", l.bin(), err, strings.TrimSpace(string(out)))
+	}
+	return parseDevices(string(out)), nil
+}
+
+// parseDevices reads --list-devices: the lines after "Available devices:",
+// which says "(none)" when there are none. Log lines before it are skipped.
+func parseDevices(out string) []string {
+	var devs []string
+	listing := false
+	for _, line := range strings.Split(out, "\n") {
+		line = strings.TrimSpace(line)
+		switch {
+		case strings.HasPrefix(line, "Available devices"):
+			listing = true
+		case listing && line != "" && line != "(none)":
+			devs = append(devs, line)
+		}
+	}
+	return devs
+}

internal/agent/engines/llamacpp_test.go

@@ -323,3 +323,41 @@ func TestAdaptersAreLoadedAtStartAndAppliedToNoneByDefault(t *testing.T) {
 		t.Error("a model with no adapters was given lora flags")
 	}
 }
+
+func TestDevicesAreReadFromListDevices(t *testing.T) {
+	cuda := "ggml_cuda_init: found 2 CUDA devices:\nAvailable devices:\n  CUDA0: NVIDIA L4 (22994 MiB, 22713 MiB free)\n  CUDA1: NVIDIA L4 (22994 MiB, 22713 MiB free)\n"
+	if got := parseDevices(cuda); len(got) != 2 || !strings.HasPrefix(got[0], "CUDA0: NVIDIA L4") {
+		t.Errorf("CUDA build: %q", got)
+	}
+	// A CUDA build whose backend did not load says what the CPU build says.
+	if got := parseDevices("Available devices:\n  (none)\n"); len(got) != 0 {
+		t.Errorf("no devices: %q", got)
+	}
+}
+
+func TestLlamaCppSpreadsAReplicaOverItsNodesGPUsByLayer(t *testing.T) {
+	four := Profile{Name: "t4x4", Engine: "llamacpp", GPUClass: "t4", TP: 1, GPUsPerNode: 4, ModelPath: "/m"}
+	if err := (&LlamaCpp{}).Validate(four, h100(4)); err != nil {
+		t.Fatalf("four GPUs on a four-GPU node: %v", err)
+	}
+	if err := (&LlamaCpp{}).Validate(four, h100(2)); err == nil {
+		t.Error("four GPUs were accepted on a node with two")
+	}
+
+	s := llamaSpec([]int{0, 1, 2, 3})
+	s.Profile.GPUsPerNode = 4
+	cmds, err := (&LlamaCpp{}).Commands(s)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if v, _ := flagValue(cmds[0].Args, "--split-mode"); v != "layer" {
+		t.Errorf("--split-mode %q over four GPUs", v)
+	}
+	if !slices.Contains(cmds[0].Env, "CUDA_VISIBLE_DEVICES=0,1,2,3") {
+		t.Errorf("env %v: the replica must see all four of its GPUs", cmds[0].Env)
+	}
+	one, _ := (&LlamaCpp{}).Commands(llamaSpec([]int{2}))
+	if hasFlag(one[0].Args, "--split-mode") {
+		t.Error("a one-GPU replica was given a split mode")
+	}
+}

internal/agent/gpucheck.go

@@ -0,0 +1,155 @@
+package agent
+
+import (
+	"context"
+	"fmt"
+	"os"
+	"strconv"
+	"strings"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
+)
+
+// gpuProcesses and processGroup are variables so tests can stand in for
+// nvidia-smi and /proc.
+var (
+	gpuProcesses = discovery.GPUProcesses
+	gpuMemory    = discovery.GPUMemoryByProcess
+	processGroup = pgidOf
+	// gpuCheckWait is how long a shard has, once it answers, to show up on
+	// its GPUs: the weights are on them by then, but nvidia-smi can lag.
+	gpuCheckWait = 15 * time.Second
+)
+
+// onItsGPUs checks that a shard placed on GPUs computes on them.
+//
+// For an engine that falls back to the CPU when its GPU backend does not load,
+// "it answers" is not "it is ready": llama.cpp built for CUDA but missing
+// cuBLAS comes up on the CPU, answers at a tenth of the speed, and the GPU it
+// was given sits idle and reserved. What it computes on is what nvidia-smi
+// lists it on, so a shard none of whose processes hold memory on one of its
+// GPUs is refused, with what the engine itself can see.
+func (a *Agent) onItsGPUs(ctx context.Context, eng engines.Engine, spec engines.ShardSpec, pid int) error {
+	fb, ok := eng.(engines.CPUFallback)
+	if !ok || !fb.FallsBackToCPU() || len(spec.GPUIndices) == 0 || pid <= 0 {
+		return nil
+	}
+	group := processGroup(pid)
+	deadline := time.Now().Add(gpuCheckWait)
+	var missing []int
+	for {
+		procs, err := gpuProcesses(ctx, a.cfg.Discovery.NvidiaSMI)
+		if err != nil || procs == nil {
+			// No nvidia-smi to ask is no evidence either way, and a node that
+			// was given GPUs has one; say so and let the shard serve.
+			a.log.Warn("could not check that a GPU shard runs on its GPUs", "shard", spec.ShardID, "err", err)
+			return nil
+		}
+		missing = missing[:0]
+		for _, i := range spec.GPUIndices {
+			if !holds(procs[i], pid, group) {
+				missing = append(missing, i)
+			}
+		}
+		if len(missing) == 0 {
+			return nil
+		}
+		if time.Now().After(deadline) {
+			break
+		}
+		select {
+		case <-ctx.Done():
+			return ctx.Err()
+		case <-time.After(time.Second):
+		}
+	}
+	seen := "it lists no GPU at all"
+	if devs, err := fb.Devices(ctx); err != nil {
+		seen = "listing its devices failed: " + err.Error()
+	} else if len(devs) > 0 {
+		seen = "it lists " + strings.Join(devs, "; ")
+	}
+	return fmt.Errorf("placed on GPU %s but computing on the CPU: the engine holds no memory there. "+
+		"Its GPU backend did not load (%s). Check the build with `ldd` on its libggml-cuda.so, "+
+		"or run `llmgw doctor` on this node", joinIndices(missing), seen)
+}
+
+// foreignGPUMemory is what each GPU holds for anything other than this
+// agent's shards, by index, and each GPU's memory in use. Nil when nvidia-smi
+// cannot be asked, which is no reading rather than an empty card.
+//
+// Everything allocated less what the shards' processes hold, rather than the
+// sum of the other processes: a process in another container's PID namespace
+// holds memory nvidia-smi lists under no PID this node can see, and it is
+// still not memory placement can hand out.
+func (a *Agent) foreignGPUMemory(ctx context.Context, shards []int) (foreign, used []int32) {
+	mem, err := gpuMemory(ctx, a.cfg.Discovery.NvidiaSMI)
+	if err != nil || mem == nil {
+		return nil, nil
+	}
+	groups := make([]int, len(shards))
+	for i, pid := range shards {
+		groups[i] = processGroup(pid)
+	}
+	ours := func(p int) bool {
+		for i, pid := range shards {
+			if holds([]int{p}, pid, groups[i]) {
+				return true
+			}
+		}
+		return false
+	}
+	for _, g := range mem {
+		f := g.UsedMB
+		for _, p := range g.Procs {
+			if ours(p.PID) {
+				f -= p.UsedMB
+			}
+		}
+		foreign = append(foreign, int32(max(f, 0)))
+		used = append(used, int32(g.UsedMB))
+	}
+	return foreign, used
+}
+
+// holds reports whether any of pids is the shard's process or in its group:
+// an engine that computes in a child process is still the shard.
+func holds(pids []int, pid, group int) bool {
+	for _, p := range pids {
+		if p == pid || (group > 0 && processGroup(p) == group) {
+			return true
+		}
+	}
+	return false
+}
+
+// pgidOf reads a process's group from /proc; zero when it cannot.
+func pgidOf(pid int) int {
+	b, err := os.ReadFile("/proc/" + strconv.Itoa(pid) + "/stat")
+	if err != nil {
+		return 0
+	}
+	// The command name, in parentheses, may hold spaces; the fields after its
+	// closing parenthesis are state, ppid, pgrp.
+	s := string(b)
+	i := strings.LastIndexByte(s, ')')
+	if i < 0 {
+		return 0
+	}
+	f := strings.Fields(s[i+1:])
+	if len(f) < 3 {
+		return 0
+	}
+	n, _ := strconv.Atoi(f[2])
+	return n
+}
+
+func joinIndices(is []int) string {
+	s := make([]string, len(is))
+	for k, i := range is {
+		s[k] = strconv.Itoa(i)
+	}
+	return strings.Join(s, ", ")
+}

internal/agent/gpucheck_test.go

@@ -0,0 +1,124 @@
+package agent
+
+import (
+	"context"
+	"log/slog"
+	"slices"
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
+)
+
+// fallbackEngine is an engine that comes up on the CPU when its GPU backend
+// does not load, and lists the devices it can see.
+type fallbackEngine struct {
+	engines.Engine
+	devices []string
+}
+
+func (fallbackEngine) FallsBackToCPU() bool { return true }
+func (f fallbackEngine) Devices(context.Context) ([]string, error) {
+	return f.devices, nil
+}
+
+func TestAGPUShardComputingOnTheCPUIsNotReady(t *testing.T) {
+	procs := map[int][]int{}
+	oldProcs, oldGroup, oldWait := gpuProcesses, processGroup, gpuCheckWait
+	gpuProcesses = func(context.Context, string) (map[int][]int, error) { return procs, nil }
+	processGroup = func(pid int) int {
+		if pid >= 500 && pid < 600 {
+			return 500 // the shard's group: 500 and its children
+		}
+		return pid
+	}
+	gpuCheckWait = 0
+	t.Cleanup(func() { gpuProcesses, processGroup, gpuCheckWait = oldProcs, oldGroup, oldWait })
+
+	a := &Agent{log: slog.Default()}
+	spec := engines.ShardSpec{ShardID: "s", GPUIndices: []int{0, 1}}
+	eng := fallbackEngine{}
+
+	// On no GPU at all: refused, saying what the engine sees.
+	err := a.onItsGPUs(context.Background(), eng, spec, 500)
+	if err == nil || !strings.Contains(err.Error(), "computing on the CPU") || !strings.Contains(err.Error(), "lists no GPU") {
+		t.Fatalf("a shard on no GPU was let through: %v", err)
+	}
+	// On one of its two: refused, naming the one it is missing.
+	procs[0] = []int{500}
+	if err := a.onItsGPUs(context.Background(), eng, spec, 500); err == nil || !strings.Contains(err.Error(), "GPU 1 ") {
+		t.Fatalf("a shard on half its GPUs: %v", err)
+	}
+	// Another process on GPU 1 is not the shard.
+	procs[1] = []int{900}
+	if err := a.onItsGPUs(context.Background(), eng, spec, 500); err == nil {
+		t.Fatal("another process's memory was taken for the shard's")
+	}
+	// A child of the shard holding GPU 1 is the shard.
+	procs[1] = []int{900, 512}
+	if err := a.onItsGPUs(context.Background(), eng, spec, 500); err != nil {
+		t.Fatalf("a shard on both its GPUs: %v", err)
+	}
+
+	// An engine that fails rather than falling back is not checked, nor is a
+	// shard with no GPUs.
+	procs = map[int][]int{}
+	if err := a.onItsGPUs(context.Background(), struct{ engines.Engine }{}, spec, 500); err != nil {
+		t.Errorf("checked an engine that does not fall back: %v", err)
+	}
+	if err := a.onItsGPUs(context.Background(), eng, engines.ShardSpec{}, 500); err != nil {
+		t.Errorf("checked a CPU shard: %v", err)
+	}
+}
+
+func TestTheCheckWaitsForTheShardToShowUpOnItsGPU(t *testing.T) {
+	calls := 0
+	oldProcs, oldWait := gpuProcesses, gpuCheckWait
+	gpuProcesses = func(context.Context, string) (map[int][]int, error) {
+		calls++
+		if calls < 2 {
+			return map[int][]int{}, nil
+		}
+		return map[int][]int{0: {77}}, nil
+	}
+	gpuCheckWait = 5 * time.Second
+	t.Cleanup(func() { gpuProcesses, gpuCheckWait = oldProcs, oldWait })
+
+	a := &Agent{log: slog.Default()}
+	if err := a.onItsGPUs(context.Background(), fallbackEngine{}, engines.ShardSpec{GPUIndices: []int{0}}, 77); err != nil {
+		t.Fatalf("nvidia-smi lagging by a second failed the shard: %v", err)
+	}
+}
+
+func TestForeignGPUMemoryLeavesOutTheAgentsOwnShards(t *testing.T) {
+	// cuda01: ComfyUI and one of our engines on the same card. Only ComfyUI's
+	// share — and whatever no listed process accounts for — is foreign.
+	oldMem, oldGroup := gpuMemory, processGroup
+	gpuMemory = func(context.Context, string) ([]discovery.GPUMemory, error) {
+		return []discovery.GPUMemory{
+			{UsedMB: 15000, Procs: []discovery.GPUProcess{{PID: 71315, UsedMB: 4932}, {PID: 512, UsedMB: 9500}}},
+			{UsedMB: 300},
+		}, nil
+	}
+	processGroup = func(pid int) int {
+		if pid >= 500 && pid < 600 {
+			return 500
+		}
+		return pid
+	}
+	t.Cleanup(func() { gpuMemory, processGroup = oldMem, oldGroup })
+
+	a := &Agent{log: slog.Default()}
+	foreign, used := a.foreignGPUMemory(context.Background(), []int{500})
+	if !slices.Equal(foreign, []int32{5500, 300}) || !slices.Equal(used, []int32{15000, 300}) {
+		t.Errorf("foreign %v, used %v; want [5500 300] and [15000 300]", foreign, used)
+	}
+
+	// No nvidia-smi to ask is no reading, not an empty card.
+	gpuMemory = func(context.Context, string) ([]discovery.GPUMemory, error) { return nil, nil }
+	if foreign, _ := a.foreignGPUMemory(context.Background(), nil); foreign != nil {
+		t.Errorf("no reading came back as %v", foreign)
+	}
+}

internal/agent/provision.go

@@ -46,9 +46,30 @@ func (a *Agent) engineReportLocked() map[string]string {
 		out = map[string]string{}
 	}
 	maps.Copy(out, a.installedVers)
+	maps.DeleteFunc(out, func(e, _ string) bool { return a.disabled(e) })
 	return out
 }
 
+// byHand reports whether agent.yaml names this engine's binary, which opts
+// it out of being installed.
+func (a *Agent) byHand(engine string) bool {
+	if a.reg == nil {
+		return false
+	}
+	e, err := a.reg.Get(engine)
+	if err != nil {
+		return false
+	}
+	c, ok := e.(engines.Configured)
+	return ok && c.Configured()
+}
+
+// disabled reports whether agent.yaml says this node never runs an engine:
+// not installed when asked, and not advertised when found.
+func (a *Agent) disabled(engine string) bool {
+	return slices.Contains(a.cfg.Engines.Disabled, engine)
+}
+
 // provisionEvery is how often installed engines are checked again. A library
 // removed by an unrelated upgrade is found here rather than by the next
 // replica to start.
@@ -58,7 +79,16 @@ const provisionEvery = 10 * time.Minute
 func (a *Agent) setProvision(p *llmgwv1.Provision) {
 	wants := []provision.Want{}
 	for _, e := range p.GetEngines() {
-		wants = append(wants, provision.Want{Engine: e.GetEngine(), Version: e.GetVersion()})
+		if a.disabled(e.GetEngine()) {
+			continue // a control plane older than the node's say in it
+		}
+		if a.byHand(e.GetEngine()) {
+			// The operator named a binary for it: a local build, a CUDA
+			// runtime of their own. Installing one beside it would win —
+			// the installed build is used first — and quietly replace theirs.
+			continue
+		}
+		wants = append(wants, provision.Want{Engine: e.GetEngine(), Version: e.GetVersion(), Variant: e.GetVariant()})
 	}
 	a.mu.Lock()
 	a.provWants = wants

internal/agent/provision/install.go

@@ -27,6 +27,9 @@ import (
 type Want struct {
 	Engine  string
 	Version string
+	// Variant is a build chosen for this node ("cu129"); empty leaves it to
+	// the node's hardware.
+	Variant string
 }
 
 // The states an engine's installation can be in.
@@ -135,7 +138,12 @@ func (in *Installer) one(ctx context.Context, w Want) Status {
 			"; install it by hand and name it in the agent's configuration"
 		return st
 	}
-	build, err := recipe.Choose(w.Version, in.Host)
+	if w.Variant != "" && !validVersion.MatchString(w.Variant) {
+		st.State = Unsupported
+		st.Detail = fmt.Sprintf("%q is not a build this agent will install", w.Variant)
+		return st
+	}
+	builds, err := recipe.Candidates(w.Version, w.Variant, in.Host)
 	if err != nil {
 		st.State, st.Detail = Unsupported, err.Error()
 		var nh NotHere
@@ -144,14 +152,18 @@ func (in *Installer) one(ctx context.Context, w Want) Status {
 		}
 		return st
 	}
-	st.Variant = build.Variant
-	dir := filepath.Join(in.root(), w.Engine, w.Version+"-"+build.Variant)
 
-	// Installed before: check it still runs, and trust nothing else about it.
-	if _, err := os.Stat(filepath.Join(dir, marker)); err == nil {
+	// Installed before: whichever candidate is, best first, checked that it
+	// still runs and trusted for nothing else.
+	for _, build := range builds {
+		dir := in.dirFor(w, build)
+		if _, err := os.Stat(filepath.Join(dir, marker)); err != nil {
+			continue
+		}
+		st := st.as(w, build)
 		// The packages are checked too. Removing one is exactly the kind of
-		// change an unrelated upgrade makes, and ldd below would catch it only
-		// as a missing library with no hint of which package held it.
+		// change an unrelated upgrade makes, and ldd below would catch it
+		// only as a missing library with no hint of which package held it.
 		if err := in.packages(ctx, build.Packages, &st); err != nil {
 			st.State, st.Detail = Failed, err.Error()
 			return st
@@ -161,24 +173,64 @@ func (in *Installer) one(ctx context.Context, w Want) Status {
 			return in.ready(st, build, bin, version)
 		}
 		in.log().Warn("an installed engine no longer runs; installing it again",
-			"engine", w.Engine, "version", w.Version, "err", err)
+			"engine", w.Engine, "version", st.Version, "variant", build.Variant, "err", err)
+		break
 	}
 
-	if err := in.install(ctx, recipe, w, build, dir, &st); err != nil {
-		st.State, st.Detail, st.Binary = Failed, err.Error(), ""
-		return st
+	// Each candidate in turn until one installs and is seen to work: a build
+	// the table was wrong about — a driver too old for it after all, a GPU
+	// it has no kernels for — costs its download, and the next is tried.
+	var tried []string
+	for _, build := range builds {
+		st := st.as(w, build)
+		if err := in.install(ctx, recipe, w, build, in.dirFor(w, build), &st); err != nil {
+			tried = append(tried, build.Variant+": "+err.Error())
+			in.log().Warn("an engine build did not work here; trying the next",
+				"engine", w.Engine, "variant", build.Variant, "err", err)
+			continue
+		}
+		bin, version, err := in.verify(ctx, in.dirFor(w, build), build)
+		if err != nil {
+			tried = append(tried, build.Variant+": "+err.Error())
+			continue
+		}
+		return in.ready(st, build, bin, version)
 	}
-	bin, version, err := in.verify(ctx, dir, build)
-	if err != nil {
-		st.State, st.Detail = Failed, err.Error()
-		return st
+	st.State, st.Binary = Failed, ""
+	st.Detail = strings.Join(tried, "; then ")
+	if len(builds) == 1 {
+		_, st.Detail, _ = strings.Cut(st.Detail, ": ")
 	}
-	return in.ready(st, build, bin, version)
+	return st
+}
+
+// dirFor is where a build of an engine is installed: by the version it is
+// and its variant, so two builds of one version sit side by side.
+func (in *Installer) dirFor(w Want, b Build) string {
+	return filepath.Join(in.root(), w.Engine, b.versionOr(w.Version)+"-"+b.Variant)
+}
+
+// versionOr is the version a build is: its own, when it is not the one asked
+// for.
+func (b Build) versionOr(asked string) string {
+	if b.Version != "" {
+		return b.Version
+	}
+	return asked
+}
+
+// as is a status for one build of what was asked for.
+func (st Status) as(w Want, b Build) Status {
+	st.Variant, st.Version = b.Variant, b.versionOr(w.Version)
+	return st
 }
 
 func (in *Installer) ready(st Status, b Build, bin, version string) Status {
 	st.State, st.Binary = Ready, bin
 	st.Detail = strings.TrimSpace(filepath.Base(bin) + " " + version)
+	if b.Note != "" {
+		st.Detail += "; " + b.Note
+	}
 	var missing []string
 	for _, f := range b.CPUFlags {
 		if !in.Host.HasCPUFlag(f) {
@@ -299,9 +351,55 @@ func (in *Installer) verify(ctx context.Context, dir string, b Build) (bin, vers
 	if err != nil {
 		return "", "", fmt.Errorf("%s --version failed: %v: %s", b.Binary, err, lastLine(out))
 	}
+	if b.GPUBackend != "" {
+		if err := in.verifyGPUBackend(tctx, dir, bin, b); err != nil {
+			return "", "", err
+		}
+	}
 	return bin, firstVersionLine(out), nil
 }
 
+// verifyGPUBackend checks a GPU build can compute on this node's GPUs.
+//
+// The binary's own ldd says nothing about it: the GPU backend is a library
+// the server opens at run time, and when it cannot — cuBLAS missing, a driver
+// too old — the server carries on with its CPU backend. So the build is asked
+// which devices it sees, and when it sees none, ldd on the backend says why.
+func (in *Installer) verifyGPUBackend(ctx context.Context, dir, bin string, b Build) error {
+	out, err := in.run(ctx, nil, bin, "--list-devices")
+	if err != nil {
+		return fmt.Errorf("%s --list-devices failed: %v: %s", b.Binary, err, lastLine(out))
+	}
+	if listsDevice(string(out)) {
+		return nil
+	}
+	why := "its GPU backend did not load"
+	if lib, err := findFile(dir, b.GPUBackend); err != nil {
+		why = "the build has no " + b.GPUBackend
+	} else if out, err := in.run(ctx, nil, "ldd", lib); err == nil {
+		if missing := lddMissing(string(out)); len(missing) > 0 {
+			why = b.GPUBackend + " cannot load: " + strings.Join(missing, "; ")
+		}
+	}
+	return fmt.Errorf("the %s build of %s sees no GPU, and would run on the CPU: %s", b.Variant, b.Binary, why)
+}
+
+// listsDevice reads llama-server --list-devices: a device line after
+// "Available devices:", which holds "(none)" when there is none.
+func listsDevice(out string) bool {
+	listing := false
+	for _, line := range strings.Split(out, "\n") {
+		line = strings.TrimSpace(line)
+		switch {
+		case strings.HasPrefix(line, "Available devices"):
+			listing = true
+		case listing && line != "" && line != "(none)":
+			return true
+		}
+	}
+	return false
+}
+
 // packages installs the operating system packages a build needs.
 //
 // Only those missing, and from the node's own package sources: a package the

internal/agent/provision/install_test.go

@@ -89,7 +89,11 @@ type fakeSystem struct {
 	installed map[string]bool
 	tools     map[string]bool
 	lddOut    string
-	calls     []string
+	// devices is what llama-server --list-devices prints, and backendLdd
+	// what ldd says of its GPU backend.
+	devices    string
+	backendLdd string
+	calls      []string
 	// locked is how many more package commands find the lock held.
 	locked int
 }
@@ -127,8 +131,12 @@ func (f *fakeSystem) run(_ context.Context, _ []string, name string, args ...str
 			}
 		}
 		return nil, nil
+	case name == "ldd" && strings.HasSuffix(args[0], "libggml-cuda.so"):
+		return []byte(f.backendLdd), nil
 	case name == "ldd":
 		return []byte(f.lddOut), nil
+	case strings.HasSuffix(name, "llama-server") && args[0] == "--list-devices":
+		return []byte(f.devices), nil
 	case strings.HasSuffix(name, "llama-server"):
 		return []byte("version: 10991 (abc)\n"), nil
 	}
@@ -247,13 +255,38 @@ func TestAnArchiveCannotWriteOutsideItsDirectory(t *testing.T) {
 func TestTheBuildFollowsTheHardware(t *testing.T) {
 	gpu := ubuntu
 	gpu.NVIDIADriver = "570.86.15"
-	b, err := chooseLlamaCpp("b1", gpu)
-	if err != nil || !strings.HasPrefix(b.Variant, "cuda") || len(b.Assets) != 2 {
-		t.Errorf("an NVIDIA node got %+v, %v; expected the CUDA build and its runtime", b, err)
+	got, err := llamaCppCandidates("b1", "", gpu)
+	if err != nil || got[0].Variant != "cuda-12.8" || len(got[0].Assets) != 2 || got[len(got)-1].Variant != "vulkan" {
+		t.Errorf("an NVIDIA node on driver 570 got %+v, %v; expected CUDA 12.8 with its runtime, then Vulkan", got, err)
 	}
-	if _, err := chooseLlamaCpp("b1", discovery.Host{OSID: "ubuntu", Arch: "riscv64"}); err == nil {
+	if _, err := llamaCppCandidates("b1", "", discovery.Host{OSID: "ubuntu", Arch: "riscv64"}); err == nil {
 		t.Error("a build was chosen for an architecture llama.cpp does not publish for")
 	}
+	for _, c := range []struct {
+		name string
+		host discovery.Host
+		want []string
+	}{
+		// Driver 580 and Turing: CUDA 13.3 first.
+		{"current", discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 8.0}, []string{"cuda-13.3", "cuda-12.8", "vulkan"}},
+		// A V100 on driver 550: CUDA 12.8 ships 7.0 as PTX, which needs 570.
+		{"V100, 550", discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0}, []string{"vulkan"}},
+		// An RTX 3090 on 550: 8.6 is machine code in the CUDA 12.8 build.
+		{"3090, 550", discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 8.6}, []string{"cuda-12.8", "vulkan"}},
+		// A Pascal card: CUDA 12.8, not 13.3.
+		{"P40", discovery.Host{Arch: "amd64", NVIDIADriver: "575.51.03", MinComputeCapability: 6.1}, []string{"cuda-12.8", "vulkan"}},
+		// arm64 with a GPU: never the CPU build.
+		{"GH200", discovery.Host{Arch: "arm64", NVIDIADriver: "580.65.06", MinComputeCapability: 9.0}, []string{"cuda-13.3", "vulkan"}},
+	} {
+		got, err := llamaCppCandidates("b1", "", c.host)
+		var names []string
+		for _, b := range got {
+			names = append(names, b.Variant)
+		}
+		if err != nil || strings.Join(names, " ") != strings.Join(c.want, " ") {
+			t.Errorf("%s: %v, %v; want %v", c.name, names, err, c.want)
+		}
+	}
 
 	slow := ubuntu
 	slow.CPUFlags = nil
@@ -388,3 +421,32 @@ func TestALibrarysVersionLinkIsStillUnpacked(t *testing.T) {
 		t.Errorf("link %q %v", target, err)
 	}
 }
+
+func TestACUDABuildThatSeesNoGPUIsNotInstalled(t *testing.T) {
+	// Found in the field: libggml-cuda.so in the build, cuBLAS nowhere the
+	// loader looks, and llama-server serving a GPU placement on the CPU.
+	cuda := "bin-ubuntu-cuda-12.8-x64.tar.gz"
+	rel := newRelease(map[string][]byte{
+		"llama-b1-" + cuda: tarGz(t, map[string]string{
+			"llama-b1/llama-server": "#!/bin/sh\n", "llama-b1/libggml-cuda.so": "x"}, nil),
+		"cudart-llama-b1-" + cuda: tarGz(t, map[string]string{"libcudart.so.12": "x"}, nil),
+	})
+	gpu := ubuntu
+	gpu.NVIDIADriver = "570.86.15"
+	sys := &fakeSystem{installed: map[string]bool{"libgomp1": true},
+		devices:    "Available devices:\n  (none)\n",
+		backendLdd: "\tlibcublas.so.12 => not found\n\tlibcudart.so.12 => /x/libcudart.so.12\n"}
+	in, _ := installer(t, rel.serve(t), sys, gpu)
+
+	got := in.Converge(t.Context(), []Want{{Engine: "llamacpp", Version: "b1"}})
+	if got[0].State != Failed || !strings.Contains(got[0].Detail, "sees no GPU") ||
+		!strings.Contains(got[0].Detail, "libcublas.so.12 is not installed") {
+		t.Fatalf("expected a failure naming the missing library: %+v", got[0])
+	}
+
+	// The same build seeing its GPU is installed.
+	sys.devices = "Available devices:\n  CUDA0: NVIDIA L4 (22994 MiB, 22713 MiB free)\n"
+	if got := in.Converge(t.Context(), []Want{{Engine: "llamacpp", Version: "b1"}}); got[0].State != Ready {
+		t.Fatalf("a CUDA build that sees its GPU: %+v", got[0])
+	}
+}

internal/agent/provision/python.go

@@ -36,6 +36,10 @@ const lockName = "llmgw-requirements.txt"
 func (in *Installer) installPython(ctx context.Context, r Recipe, w Want, b Build, dir string, st *Status) error {
 	ctx, cancel := context.WithTimeout(ctx, pythonInstallTimeout)
 	defer cancel()
+	// The version this build is, which for a GPU older than every build of
+	// the one asked for is an older one: its publication date is the one
+	// the dependencies are resolved as of.
+	w.Version = b.versionOr(w.Version)
 
 	if err := in.packages(ctx, b.Packages, st); err != nil {
 		return err
@@ -86,28 +90,67 @@ func (in *Installer) installPython(ctx context.Context, r Recipe, w Want, b Buil
 	}
 	python := filepath.Join(tmp, "bin", "python")
 
-	in.progress(st, "resolving %s against PyPI as of %s", strings.Join(b.Requirements, " "), asOf[:10])
+	reqs := b.Requirements
+	if b.Wheel != nil {
+		// The engine from its GitHub release, pinned to the digest GitHub
+		// publishes for the file: the same trust as an archive, and uv
+		// refuses a download that does not match it.
+		in.progress(st, "asking GitHub for %s %s's %s…%s", b.Wheel.Repo, b.Wheel.Tag, b.Wheel.Prefix, b.Wheel.Suffix)
+		sums, err := in.checksums(ctx, b.Wheel.Repo, b.Wheel.Tag)
+		if err != nil {
+			return err
+		}
+		var asset, sum string
+		for name, s := range sums {
+			if strings.HasPrefix(name, b.Wheel.Prefix) && strings.HasSuffix(name, b.Wheel.Suffix) {
+				asset, sum = name, s
+			}
+		}
+		if asset == "" {
+			return fmt.Errorf("%s %s publishes no %s…%s", b.Wheel.Repo, b.Wheel.Tag, b.Wheel.Prefix, b.Wheel.Suffix)
+		}
+		url := strings.TrimSuffix(in.downloadBase(), "/") + "/" + b.Wheel.Repo + "/releases/download/" +
+			b.Wheel.Tag + "/" + asset
+		reqs = append([]string{b.Wheel.Project + " @ " + url + "#sha256=" + sum}, reqs...)
+	}
+	// A second index, for the torch the engine was built on. Every package
+	// is taken from whichever of the two has the version resolved — both are
+	// the publishers' own — and each file is still pinned by its hash.
+	var index []string
+	if b.ExtraIndex != "" {
+		index = []string{"--extra-index-url", b.ExtraIndex, "--index-strategy", "unsafe-best-match"}
+	}
+
+	in.progress(st, "resolving %s against PyPI as of %s", strings.Join(reqs, " "), asOf[:10])
 	lock := filepath.Join(tmp, lockName)
-	args := []string{"pip", "compile", "--python", python, "--exclude-newer", asOf,
-		"--generate-hashes", "--quiet", "--output-file", lock, "-"}
-	if out, err := in.runStdin(ctx, env, strings.Join(b.Requirements, "\n")+"\n", uv.Binary, args...); err != nil {
-		return fmt.Errorf("resolving %s: %v: %s", strings.Join(b.Requirements, " "), err, tail(out))
+	args := append([]string{"pip", "compile", "--python", python, "--exclude-newer", asOf,
+		"--generate-hashes", "--quiet", "--output-file", lock}, index...)
+	args = append(args, "-")
+	if out, err := in.runStdin(ctx, env, strings.Join(reqs, "\n")+"\n", uv.Binary, args...); err != nil {
+		return fmt.Errorf("resolving %s: %v: %s", strings.Join(reqs, " "), err, tail(out))
 	}
 
 	in.progress(st, "downloading and installing %s and its dependencies: several gigabytes, which takes minutes",
-		strings.Join(b.Requirements, " "))
-	if out, err := in.run(ctx, env, uv.Binary, "pip", "install", "--python", python,
-		"--require-hashes", "--no-deps", "-r", lock); err != nil {
-		return fmt.Errorf("installing %s: %v: %s", strings.Join(b.Requirements, " "), err, tail(out))
+		strings.Join(reqs, " "))
+	install := append([]string{"pip", "install", "--python", python,
+		"--require-hashes", "--no-deps", "-r", lock}, index...)
+	if out, err := in.run(ctx, env, uv.Binary, install...); err != nil {
+		return fmt.Errorf("installing %s: %v: %s", strings.Join(reqs, " "), err, tail(out))
 	}
 
 	in.progress(st, "checking that %s runs", w.Engine)
 	if _, _, err := in.verify(ctx, tmp, b); err != nil {
 		return err
 	}
+	if b.GPUCheck && in.Host.NVIDIADriver != "" {
+		in.progress(st, "checking that %s (%s) computes on this node's GPU", w.Engine, b.Variant)
+		if err := in.gpuCheck(ctx, python, w, b); err != nil {
+			return err
+		}
+	}
 	m, _ := json.MarshalIndent(markerFile{
 		Engine: w.Engine, Version: w.Version, Variant: b.Variant,
-		Binary: filepath.Join("bin", b.Binary), Requirements: b.Requirements,
+		Binary: filepath.Join("bin", b.Binary), Requirements: reqs,
 		ExcludeNewer: asOf, UV: UVVersion, Installed: time.Now().UTC(),
 	}, "", "  ")
 	if err := os.WriteFile(filepath.Join(tmp, marker), m, 0o644); err != nil { //nolint:gosec // not secret
@@ -203,3 +246,26 @@ func tail(b []byte) string {
 	}
 	return strings.Join(lines, " / ")
 }
+
+// gpuProbe is what the environment is asked to run on the GPU: one small
+// operation, which needs the torch to have kernels for the card.
+const gpuProbe = `import torch
+assert torch.cuda.is_available(), "torch sees no CUDA device"
+x = torch.ones(64, device="cuda")
+print(float((x * 2).sum()), torch.cuda.get_device_name(0), torch.cuda.get_device_capability(0))`
+
+// gpuCheck runs gpuProbe in a freshly installed environment.
+func (in *Installer) gpuCheck(ctx context.Context, python string, w Want, b Build) error {
+	ctx, cancel := context.WithTimeout(ctx, 3*time.Minute)
+	defer cancel()
+	out, err := in.run(ctx, nil, python, "-c", gpuProbe)
+	if err != nil {
+		hint := ""
+		if strings.Contains(string(out), "no kernel image") {
+			hint = "; this build has no kernels for this GPU, so try the other build for this node"
+		}
+		return fmt.Errorf("%s %s (%s) installed but cannot compute on this node's GPU: %s%s",
+			w.Engine, w.Version, b.Variant, tail(out), hint)
+	}
+	return nil
+}

internal/agent/provision/python_test.go

@@ -2,6 +2,7 @@ package provision
 
 import (
 	"context"
+	"errors"
 	"net/http"
 	"net/http/httptest"
 	"os"
@@ -23,6 +24,10 @@ type fakePython struct {
 	pmu      sync.Mutex
 	commands []string
 	failPip  bool
+	// noKernel makes the GPU check fail as a torch without kernels for the
+	// card does: in every build, or in noKernelIn only ("-cu13.").
+	noKernel   bool
+	noKernelIn string
 }
 
 func (f *fakePython) run(ctx context.Context, env []string, name string, args ...string) ([]byte, error) {
@@ -43,6 +48,11 @@ func (f *fakePython) run(ctx context.Context, env []string, name string, args ..
 		return nil, os.WriteFile(filepath.Join(venv, "bin", "vllm"), []byte("#!/bin/sh\n"), 0o755)
 	case strings.HasSuffix(name, "/bin/vllm") && args[0] == "--version":
 		return []byte("0.30.0\n"), nil
+	case strings.HasSuffix(name, "/bin/python") && args[0] == "-c":
+		if f.noKernel || (f.noKernelIn != "" && strings.Contains(name, f.noKernelIn)) {
+			return []byte("RuntimeError: CUDA error: no kernel image is available for execution on the device\n"), errExit
+		}
+		return []byte("128.0 Tesla V100-SXM2-32GB (7, 0)\n"), nil
 	}
 	return f.fakeSystem.run(ctx, env, name, args...)
 }
@@ -59,6 +69,8 @@ func pythonInstaller(t *testing.T, host discovery.Host, sys *fakePython) (*Insta
 	uv := newRelease(map[string][]byte{
 		"uv-x86_64-unknown-linux-gnu.tar.gz": tarGz(t,
 			map[string]string{"uv-x86_64-unknown-linux-gnu/uv": "#!/bin/sh\n"}, nil),
+		// The fake GitHub serves every repository's release from this one.
+		"vllm-0.30.0+cu129-cp38-abi3-manylinux_2_28_x86_64.whl": []byte("the cu129 wheel"),
 	})
 	gh := uv.serve(t)
 	mux := http.NewServeMux()
@@ -66,6 +78,9 @@ func pythonInstaller(t *testing.T, host discovery.Host, sys *fakePython) (*Insta
 		_, _ = w.Write([]byte(`{"urls":[{"upload_time_iso_8601":"2026-09-22T05:22:21.647189Z"},
 			{"upload_time_iso_8601":"2026-09-22T05:23:10.039243Z"}]}`))
 	})
+	mux.HandleFunc("/pypi/vllm/0.19.1/json", func(w http.ResponseWriter, _ *http.Request) {
+		_, _ = w.Write([]byte(`{"urls":[{"upload_time_iso_8601":"2026-04-02T10:00:00Z"}]}`))
+	})
 	pypi := httptest.NewServer(mux)
 	t.Cleanup(pypi.Close)
 	return &Installer{
@@ -130,8 +145,10 @@ func TestVLLMIsNotForEveryNode(t *testing.T) {
 		says  string
 	}{
 		{"no GPU", ubuntu, Skipped, "has none"},
-		{"old driver", discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15"}, Unsupported, "driver 580"},
-		{"V100", discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 7.0}, Unsupported, "7.5"},
+		// Too old for CUDA 13 and for its CUDA 12.9 build both.
+		{"a driver too old for either", discovery.Host{Arch: "amd64", NVIDIADriver: "470.256.02"}, Unsupported, "driver 525"},
+		// No vLLM was ever built for Pascal.
+		{"P100", discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 6.0}, Unsupported, "no vLLM is built for GPUs of compute capability 6.0"},
 	}
 	for _, c := range cases {
 		t.Run(c.name, func(t *testing.T) {
@@ -182,3 +199,114 @@ func TestAVersionPyPIDoesNotHaveIsSaidSo(t *testing.T) {
 		t.Errorf("got %s %q", got[0].State, got[0].Detail)
 	}
 }
+
+func TestAnOlderDriverGetsTheSameVLLMBuiltForCUDA12(t *testing.T) {
+	// A Turing card on a driver too old for CUDA 13: the same vLLM's CUDA
+	// 12.9 build, from the project's GitHub release, with the torch PyTorch
+	// publishes for 12.9.
+	volta := discovery.Host{OSID: "ubuntu", Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.5}
+	sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
+	in, gh := pythonInstaller(t, volta, sys)
+	got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})
+	if got[0].State != Ready || got[0].Variant != "cu129" {
+		t.Fatalf("a Turing node on driver 550: %+v", got[0])
+	}
+	if _, err := os.Stat(filepath.Join(in.Root, "vllm", "0.30.0-cu129", marker)); err != nil {
+		t.Errorf("not installed under its own build's name: %v", err)
+	}
+	all := strings.Join(sys.commands, "\n")
+	wheel := "vllm-0.30.0+cu129-cp38-abi3-manylinux_2_28_x86_64.whl"
+	for _, want := range []string{
+		// Pinned to the digest GitHub publishes for the wheel.
+		"vllm @ " + in.DownloadBase + "/vllm-project/vllm/releases/download/v0.30.0/" + wheel + "#sha256=" +
+			strings.TrimPrefix(gh.digests[wheel], "sha256:"),
+		"--extra-index-url https://download.pytorch.org/whl/cu129 --index-strategy unsafe-best-match -",
+		"--require-hashes --no-deps -r",
+		// Computing on the card, not only importing.
+		"/bin/python -c import torch",
+	} {
+		if !strings.Contains(all, want) {
+			t.Errorf("no %q in:\n%s", want, all)
+		}
+	}
+	for _, c := range sys.commands {
+		if strings.Contains(c, "pip install") && !strings.Contains(c, "download.pytorch.org/whl/cu129") {
+			t.Errorf("installed without the index its torch comes from: %s", c)
+		}
+	}
+}
+
+func TestABuildChosenForANodeIsTheOneInstalled(t *testing.T) {
+	sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
+	in, _ := pythonInstaller(t, gpuNode, sys)
+	if got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0", Variant: "cu129"}}); got[0].Variant != "cu129" || got[0].State != Ready {
+		t.Errorf("cu129 chosen for a node that runs cu13 too: %+v", got[0])
+	}
+	if _, err := vllmCandidates("0.30.0", "cu200", gpuNode); err == nil {
+		t.Error("a build that does not exist was accepted")
+	}
+}
+
+func TestABuildThatCannotComputeOnTheCardIsNotInstalled(t *testing.T) {
+	sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}, noKernel: true}
+	in, _ := pythonInstaller(t, gpuNode, sys)
+	got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})
+	if got[0].State != Failed || !strings.Contains(got[0].Detail, "no kernels for this GPU") {
+		t.Fatalf("a build without kernels for the card: %+v", got[0])
+	}
+	if _, err := os.Stat(filepath.Join(in.Root, "vllm", "0.30.0-cu13", marker)); err == nil {
+		t.Error("marked installed")
+	}
+}
+
+func TestOnlyACUDARequirementFallsBackToTheCUDA12Build(t *testing.T) {
+	// A machine vLLM is not published for is refused, not given the other
+	// build to fail with.
+	if _, err := vllmCandidates("0.30.0", "", discovery.Host{NVIDIADriver: "580.65.06"}); err == nil ||
+		!strings.Contains(err.Error(), "no Linux build") {
+		t.Errorf("an unknown architecture: %v", err)
+	}
+	if _, err := vllmCandidates("0.30.0", "cu129", ubuntu); !errors.As(err, new(NotHere)) {
+		t.Errorf("a node with no GPU chosen cu129: %v", err)
+	}
+	// Driver 580 and Turing: CUDA 13 first, the CUDA 12.9 build behind it.
+	got, err := vllmCandidates("0.30.0", "", gpuNode)
+	if err != nil || len(got) != 2 || got[0].Variant != "cu13" || got[1].Variant != "cu129" {
+		t.Errorf("candidates on a current node: %+v, %v", got, err)
+	}
+}
+
+func TestAV100GetsTheNewestVLLMBuiltForItByItself(t *testing.T) {
+	// 0.30.0 asked for; no build of it has Volta kernels. The node installs
+	// 0.19.1, the newest that has, and says so.
+	v100 := discovery.Host{OSID: "ubuntu", Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0}
+	sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
+	in, _ := pythonInstaller(t, v100, sys)
+	got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})[0]
+	if got.State != Ready || got.Version != "0.19.1" || got.Variant != "cu12" ||
+		!strings.Contains(got.Detail, "the newest built for compute capability 7.0, instead of 0.30.0") {
+		t.Fatalf("a V100: %+v", got)
+	}
+	if _, err := os.Stat(filepath.Join(in.Root, "vllm", "0.19.1-cu12", marker)); err != nil {
+		t.Errorf("not installed as what it is: %v", err)
+	}
+	all := strings.Join(sys.commands, "\n")
+	if !strings.Contains(all, "<<< vllm==0.19.1") || strings.Contains(all, "download.pytorch.org") {
+		t.Errorf("0.19.1 is PyPI's CUDA 12 build, with PyPI's torch:\n%s", all)
+	}
+	// Chosen cu13, which 0.19.1 has none of: refused, saying why.
+	if _, err := vllmCandidates("0.30.0", "cu13", v100); err == nil {
+		t.Error("a CUDA 13 build was offered to a V100")
+	}
+}
+
+func TestABuildThatFailsOnTheCardGivesWayToTheNext(t *testing.T) {
+	// The table says CUDA 13 runs here; the card says otherwise. The next
+	// build is installed rather than the node going without.
+	sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}, noKernelIn: "-cu13."}
+	in, _ := pythonInstaller(t, gpuNode, sys)
+	got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})[0]
+	if got.State != Ready || got.Variant != "cu129" {
+		t.Fatalf("after the CUDA 13 build failed on the card: %+v", got)
+	}
+}

internal/agent/provision/recipe.go

@@ -43,12 +43,41 @@ type Build struct {
 	// VersionArgs is what Binary is run with to prove the build works, when
 	// --version is not it.
 	VersionArgs []string
+	// GPUBackend, when set, is the shared library the build computes on GPUs
+	// with, loaded at run time rather than linked: "libggml-cuda.so". The
+	// build must list a device with it, or it is not installed — a llama.cpp
+	// whose CUDA backend cannot load runs on the CPU without complaint.
+	GPUBackend string
+	// Wheel, when set, is the engine itself as a wheel from a GitHub
+	// release rather than from PyPI, checked against the digest GitHub
+	// publishes for it. ExtraIndex is a package index besides PyPI that the
+	// rest resolves from too: the CUDA 12.9 torch the wheel was built on.
+	Wheel      *Wheel
+	ExtraIndex string
+	// GPUCheck, on a node with an NVIDIA GPU, has the installed environment
+	// compute something on one before the build counts as installed. A build
+	// with no kernels for the card — a GPU generation its torch dropped —
+	// imports and prints its version, and fails on its first model.
+	GPUCheck bool
+	// Version is the engine version this build is, when it is not the one
+	// asked for: the newest built for this machine's GPUs, where the one
+	// asked for has no kernels for them. Empty is the version asked for.
+	Version string
+	// Note says why this build, when it is not the plain one: shown beside
+	// it wherever the installation is.
+	Note string
 	// FreeGB is the disk space the installation needs before it starts: a
 	// Python engine is several gigabytes, and one that fills the disk halfway
 	// leaves the node worse off than one that never began.
 	FreeGB int
 }
 
+// Wheel is a package published as a GitHub release asset: the asset whose
+// name starts with Prefix and ends with Suffix.
+type Wheel struct {
+	Project, Repo, Tag, Prefix, Suffix string
+}
+
 // Recipe knows how to install one engine.
 type Recipe struct {
 	// Repo is the GitHub repository whose releases carry the builds; empty
@@ -56,8 +85,12 @@ type Recipe struct {
 	Repo string
 	// PyPI is the project an engine installed from PyPI is published as.
 	PyPI string
-	// Choose picks the build for a machine, or says why none fits.
-	Choose func(version string, h discovery.Host) (Build, error)
+	// Candidates are the builds that may run on a machine, best first, or
+	// why none can. The installer takes the first that installs and is seen
+	// to work — computing on the GPU, where there is one — so a build this
+	// table is wrong about costs a download, not a node. variant is a build
+	// chosen for the node; empty leaves the choice to the machine.
+	Candidates func(version, variant string, h discovery.Host) ([]Build, error)
 }
 
 // Recipes are the engines this agent can install.
@@ -66,10 +99,10 @@ type Recipe struct {
 // from PyPI by uv, which is itself installed like llama.cpp: a GitHub release
 // checked against its published checksum.
 var Recipes = map[string]Recipe{
-	"llamacpp": {Repo: "ggml-org/llama.cpp", Choose: chooseLlamaCpp},
-	"vllm":     {PyPI: "vllm", Choose: chooseVLLM},
-	"sglang":   {PyPI: "sglang", Choose: chooseSGLang},
-	"uv":       {Repo: "astral-sh/uv", Choose: chooseUV},
+	"llamacpp": {Repo: "ggml-org/llama.cpp", Candidates: llamaCppCandidates},
+	"vllm":     {PyPI: "vllm", Candidates: vllmCandidates},
+	"sglang":   {PyPI: "sglang", Candidates: sglangCandidates},
+	"uv":       {Repo: "astral-sh/uv", Candidates: uvCandidates},
 }
 
 // Installable are the engines this agent can install, as it tells the
@@ -99,7 +132,15 @@ type NotHere struct{ Reason string }
 
 func (e NotHere) Error() string { return e.Reason }
 
-func chooseUV(version string, h discovery.Host) (Build, error) {
+func uvCandidates(version, _ string, h discovery.Host) ([]Build, error) {
+	b, err := chooseUV(version, h)
+	if err != nil {
+		return nil, err
+	}
+	return []Build{b}, nil
+}
+
+func chooseUV(_ string, h discovery.Host) (Build, error) {
 	var arch string
 	switch h.Arch {
 	case "amd64":
@@ -116,33 +157,227 @@ func chooseUV(version string, h discovery.Host) (Build, error) {
 	}, nil
 }
 
-// minDriverForCUDA13 is the oldest NVIDIA driver CUDA 13 runs on. Every vLLM
-// since 0.26 and SGLang 0.5 is built on torch for CUDA 13.
-const minDriverForCUDA13 = 580
+// What builds need of the NVIDIA side of a machine, read from what each
+// project publishes and how it was built:
+//
+//   - CUDA 13 needs driver 580; CUDA 12 builds run on driver 525 and newer,
+//     by CUDA's minor version compatibility — except code shipped only as
+//     PTX, which the driver compiles and so needs a driver as new as the
+//     CUDA it was compiled with.
+//   - Which GPUs a build has kernels for is the build's own list, not what
+//     its CUDA supports.
+const (
+	minDriverForCUDA13 = 580
+	minDriverForCUDA12 = 525
+)
 
-// minComputeForCUDA13 is the oldest GPU CUDA 13 supports: Turing. It dropped
-// Volta, so a V100 cannot run these builds at all.
-const minComputeForCUDA13 = 7.5
+// gpuNeeds is what one build needs of a machine's GPUs and driver.
+type gpuNeeds struct {
+	driver int
+	cc     float64
+}
 
-// cudaGate says whether a node can run an engine built on CUDA 13.
-func cudaGate(engine string, h discovery.Host) error {
+// fits says why a machine cannot run a build, or nil.
+func (n gpuNeeds) fits(what string, h discovery.Host) error {
+	if major := leadingInt(h.NVIDIADriver); n.driver > 0 && major > 0 && major < n.driver {
+		return fmt.Errorf("%s needs NVIDIA driver %d or newer; this node has %s", what, n.driver, h.NVIDIADriver)
+	}
+	if cc := h.MinComputeCapability; n.cc > 0 && cc > 0 && cc < n.cc {
+		return fmt.Errorf("%s has kernels for compute capability %.1f and newer; this node has %.1f", what, n.cc, cc)
+	}
+	return nil
+}
+
+// candidate is a build and what it needs.
+type candidate struct {
+	b     Build
+	needs gpuNeeds
+	what  string
+}
+
+// pick keeps the candidates a machine can run, in order, and those a
+// variant chosen for the node names; with none left, it says why each went.
+func pick(engine, variant string, cands []candidate, h discovery.Host) ([]Build, error) {
+	var out []Build
+	var why []string
+	for _, c := range cands {
+		if variant != "" && c.b.Variant != variant {
+			continue
+		}
+		if err := c.needs.fits(c.what, h); err != nil {
+			why = append(why, err.Error())
+			continue
+		}
+		out = append(out, c.b)
+	}
+	switch {
+	case len(out) > 0:
+		return out, nil
+	case len(why) == 0 && variant != "":
+		var names []string
+		for _, c := range cands {
+			names = append(names, c.b.Variant)
+		}
+		return nil, fmt.Errorf("%s has no build %q here; it has %s", engine, variant, strings.Join(names, ", "))
+	}
+	return nil, fmt.Errorf("no build of %s runs on this node: %s", engine, strings.Join(why, "; "))
+}
+
+// --- vLLM ---
+
+// VLLMVariants are the builds of vLLM a node can be given by choice: CUDA
+// 13 from PyPI, and CUDA 12.9 from the project's GitHub release — the same
+// version, built for drivers CUDA 13 does not run on.
+var VLLMVariants = []string{"cu13", "cu129"}
+
+// pytorchCU129 is where the CUDA 12.9 build's torch is published. PyPI's
+// torch is built for CUDA 13; the cu129 vLLM wheel needs this one.
+const pytorchCU129 = "https://download.pytorch.org/whl/cu129"
+
+// vllmCUDA13Since is the first vLLM whose PyPI build is CUDA 13, read from
+// its releases: 0.19.1 publishes "+cu130" beside a CUDA 12 default, 0.20.0
+// "+cu129" beside a CUDA 13 one.
+const vllmCUDA13Since = "0.20"
+
+// vllmMinCompute is the oldest GPU a vLLM version's builds have kernels for,
+// from its release pipeline: on x86, 7.0 (Volta) up to 0.19.1 — the
+// Dockerfile's default "7.0 7.5 8.0 8.9 9.0 10.0 12.0" — and 7.5 from 0.20.0,
+// CUDA 12.9 build included; on arm64, built for Grace machines, 8.7 before
+// 0.20 and 8.0 since.
+func vllmMinCompute(version, arch string) float64 {
+	old := versionBefore(version, vllmCUDA13Since)
+	switch {
+	case arch == "arm64" && old:
+		return 8.7
+	case arch == "arm64":
+		return 8.0
+	case old:
+		return 7.0
+	}
+	return 7.5
+}
+
+// vllmNewestFor is the newest vLLM built for GPUs of compute capability cc
+// on an architecture, for GPUs older than every build of the version asked
+// for: a V100 gets 0.19.1, where 0.20 and later have nothing for it. Nothing
+// for a GPU no vLLM was built for.
+var vllmNewestFor = []struct {
+	arch    string
+	from    float64 // the oldest GPU it covers
+	version string
+}{
+	{"amd64", 7.0, "0.19.1"},
+}
+
+// vllmCandidates are the vLLM builds for a machine.
+//
+// The version asked for, in the build the machine runs: CUDA 13 where the
+// driver allows, the same version's CUDA 12.9 build where it does not. One
+// version across the cluster wherever the GPUs allow it, because the flags
+// llmgw builds are checked against it. Where they do not — GPUs older than
+// every build of that version — the newest version built for them, said so
+// beside the installation: vLLM on a V100 at all, rather than not.
+func vllmCandidates(version, variant string, h discovery.Host) ([]Build, error) {
+	arch, err := gpuArch("vLLM", h)
+	if err != nil {
+		return nil, err
+	}
+	v, note := version, ""
+	if cc := h.MinComputeCapability; cc > 0 && cc < vllmMinCompute(version, h.Arch) {
+		older := ""
+		for _, n := range vllmNewestFor {
+			if n.arch == h.Arch && cc >= n.from {
+				older = n.version
+			}
+		}
+		if older == "" {
+			return nil, fmt.Errorf("no vLLM is built for GPUs of compute capability %.1f on %s; "+
+				"the node serves llama.cpp models", cc, arch)
+		}
+		v, note = older, fmt.Sprintf("vLLM %s, the newest built for compute capability %.1f, instead of %s", older, cc, version)
+	}
+	pypi := func(variant string) Build {
+		b := Build{
+			Variant:      variant,
+			Python:       pythonForEngines,
+			Requirements: []string{"vllm==" + v},
+			Binary:       "vllm",
+			Packages:     compilerPackages,
+			FreeGB:       25,
+			GPUCheck:     true,
+			Note:         note,
+		}
+		if v != version {
+			b.Version = v
+		}
+		return b
+	}
+	minCC := vllmMinCompute(v, h.Arch)
+	if versionBefore(v, vllmCUDA13Since) {
+		// PyPI's own build is CUDA 12, and is the one there is; asked for
+		// cu129, this is it.
+		if variant == "cu129" {
+			variant = "cu12"
+		}
+		return pick("vLLM "+v, variant, []candidate{
+			{pypi("cu12"), gpuNeeds{minDriverForCUDA12, minCC}, "vLLM " + v},
+		}, h)
+	}
+	cu129 := pypi("cu129")
+	cu129.Requirements = nil
+	// Found in the release's assets rather than named: the manylinux tag
+	// in the file's name changes between versions (2_28, 2_31, 2_34).
+	cu129.Wheel = &Wheel{
+		Project: "vllm", Repo: "vllm-project/vllm", Tag: "v" + v,
+		Prefix: "vllm-" + v + "+cu129-", Suffix: "_" + map[string]string{"amd64": "x86_64", "arm64": "aarch64"}[h.Arch] + ".whl",
+	}
+	cu129.ExtraIndex = pytorchCU129
+	return pick("vLLM "+v, variant, []candidate{
+		{pypi("cu13"), gpuNeeds{minDriverForCUDA13, minCC}, "vLLM " + v + " built for CUDA 13"},
+		{cu129, gpuNeeds{minDriverForCUDA12, minCC}, "vLLM " + v + " built for CUDA 12.9"},
+	}, h)
+}
+
+// gpuArch says whether a Python GPU engine is published for this machine
+// and whether it has an NVIDIA GPU at all.
+func gpuArch(engine string, h discovery.Host) (string, error) {
 	switch h.Arch {
 	case "amd64", "arm64":
 	default:
-		return fmt.Errorf("%s publishes no Linux build for %s", engine, orUnknown(h.Arch))
+		return "", fmt.Errorf("%s publishes no Linux build for %s", engine, orUnknown(h.Arch))
 	}
 	if h.NVIDIADriver == "" {
-		return NotHere{engine + " runs on NVIDIA GPUs and this node has none; it serves llama.cpp models"}
+		return "", NotHere{engine + " runs on NVIDIA GPUs and this node has none; it serves llama.cpp models"}
 	}
-	if major := leadingInt(h.NVIDIADriver); major > 0 && major < minDriverForCUDA13 {
-		return fmt.Errorf("%s is built for CUDA 13, which needs NVIDIA driver %d or newer; this node has %s",
-			engine, minDriverForCUDA13, h.NVIDIADriver)
+	return h.Arch, nil
+}
+
+// --- SGLang ---
+
+// sglangCUDA13Since is the first SGLang built for CUDA 13.
+const sglangCUDA13Since = "0.5"
+
+func sglangCandidates(version, variant string, h discovery.Host) ([]Build, error) {
+	if _, err := gpuArch("SGLang", h); err != nil {
+		return nil, err
 	}
-	if cc := h.MinComputeCapability; cc > 0 && cc < minComputeForCUDA13 {
-		return fmt.Errorf("%s is built for CUDA 13, which supports GPUs of compute capability %.1f and newer; this node has %.1f",
-			engine, minComputeForCUDA13, cc)
+	b := Build{
+		Variant:      "cu13",
+		Python:       pythonForEngines,
+		Requirements: []string{"sglang==" + version},
+		// SGLang is launched as a module, so what the agent is given is the
+		// environment's interpreter, and the proof is importing it.
+		Binary:      "python",
+		VersionArgs: []string{"-c", "import sglang; print('sglang', sglang.__version__)"},
+		Packages:    compilerPackages,
+		FreeGB:      25,
+		GPUCheck:    true,
 	}
-	return nil
+	needs := gpuNeeds{minDriverForCUDA13, 7.5}
+	if versionBefore(version, sglangCUDA13Since) {
+		b.Variant, needs = "cu12", gpuNeeds{minDriverForCUDA12, 7.0}
+	}
+	return pick("SGLang "+version, variant, []candidate{{b, needs, "SGLang " + version}}, h)
 }
 
 // Python engines need a C compiler at run time: Triton builds a small
@@ -150,35 +385,27 @@ func cudaGate(engine string, h discovery.Host) error {
 // that load fails long after the installation looked finished.
 var compilerPackages = map[string][]string{"deb": {"gcc"}, "rpm": {"gcc"}}
 
-func chooseVLLM(version string, h discovery.Host) (Build, error) {
-	if err := cudaGate("vLLM", h); err != nil {
-		return Build{}, err
+// versionBefore reports whether version a is older than b, comparing the
+// dotted numbers in turn ("0.25.1" before "0.26"). A part that is not a
+// number counts as zero.
+func versionBefore(a, b string) bool {
+	if b == "" {
+		return false
 	}
-	return Build{
-		Variant:      "cu13",
-		Python:       pythonForEngines,
-		Requirements: []string{"vllm==" + version},
-		Binary:       "vllm",
-		Packages:     compilerPackages,
-		FreeGB:       25,
-	}, nil
-}
-
-func chooseSGLang(version string, h discovery.Host) (Build, error) {
-	if err := cudaGate("SGLang", h); err != nil {
-		return Build{}, err
+	pa, pb := strings.Split(strings.TrimPrefix(a, "v"), "."), strings.Split(b, ".")
+	for i := 0; i < max(len(pa), len(pb)); i++ {
+		x, y := 0, 0
+		if i < len(pa) {
+			x = leadingInt(pa[i])
+		}
+		if i < len(pb) {
+			y = leadingInt(pb[i])
+		}
+		if x != y {
+			return x < y
+		}
 	}
-	// SGLang is launched as a module, so what the agent is given is the
-	// environment's interpreter, and the proof is importing it.
-	return Build{
-		Variant:      "cu13",
-		Python:       pythonForEngines,
-		Requirements: []string{"sglang==" + version},
-		Binary:       "python",
-		VersionArgs:  []string{"-c", "import sglang; print('sglang', sglang.__version__)"},
-		Packages:     compilerPackages,
-		FreeGB:       25,
-	}, nil
+	return false
 }
 
 // leadingInt is the number a version string starts with: 580 of "580.65.06".
@@ -193,17 +420,30 @@ func leadingInt(s string) int {
 	return n
 }
 
-// cudaForLlamaCpp is the CUDA build taken on an NVIDIA node. The release also
-// carries a matching cudart archive, so the node needs a driver and nothing
-// else — no toolkit.
-const cudaForLlamaCpp = "12.8"
+// --- llama.cpp ---
 
-func chooseLlamaCpp(version string, h discovery.Host) (Build, error) {
-	// The published Linux builds are built on Ubuntu, but nothing about them
-	// is Ubuntu's: they need glibc 2.34 and the libstdc++ of GCC 12, which
-	// Debian 12 and RHEL 10 have and RHEL 9 does not. So the distribution's
-	// name decides nothing here. The build is tried everywhere, and ldd says
-	// whether this system can load it — in the words of the missing symbol.
+// llamaCUDA12Real are the GPUs llama.cpp's CUDA 12.8 build carries machine
+// code for. The rest of what it covers — 5.0, 6.1, 7.0, 7.5, 8.0, 9.0 — it
+// carries as PTX, which a driver older than the CUDA it came from cannot
+// compile.
+var llamaCUDA12Real = []float64{8.6, 8.9, 12.0}
+
+// llamaCppCandidates are the llama.cpp builds for a machine.
+//
+// On an NVIDIA node, the GPU builds best first: CUDA 13.3 (driver 580,
+// Turing and newer), CUDA 12.8 on x64 (Maxwell to Blackwell; driver 570 for
+// the GPUs it ships as PTX, 525 for those it ships as machine code), and
+// Vulkan, which runs on any card whose driver has Vulkan — the generations
+// CUDA builds no longer cover. Never the CPU build on an NVIDIA node: that is
+// a GPU placement served on the CPU, which is what installing checks for.
+// The CPU build where there is no NVIDIA GPU.
+//
+// The published Linux builds are built on Ubuntu, but nothing about them is
+// Ubuntu's: they need glibc 2.34 and the libstdc++ of GCC 12, which Debian 12
+// and RHEL 10 have and RHEL 9 does not. So the distribution's name decides
+// nothing here. The build is tried everywhere, and ldd says whether this
+// system can load it — in the words of the missing symbol.
+func llamaCppCandidates(version, variant string, h discovery.Host) ([]Build, error) {
 	var arch string
 	switch h.Arch {
 	case "amd64":
@@ -211,35 +451,53 @@ func chooseLlamaCpp(version string, h discovery.Host) (Build, error) {
 	case "arm64":
 		arch = "arm64"
 	default:
-		return Build{}, fmt.Errorf("llama.cpp publishes no Linux build for %s", orUnknown(h.Arch))
+		return nil, fmt.Errorf("llama.cpp publishes no Linux build for %s", orUnknown(h.Arch))
 	}
-
-	b := Build{
-		Variant: "cpu",
-		Assets:  []string{fmt.Sprintf("llama-%s-bin-ubuntu-%s.tar.gz", version, arch)},
-		// Everything else the build links is in the archive or in any current
-		// glibc system; OpenMP's runtime is the one a cloud image lacks, and
-		// the server does not start without it.
-		Packages: map[string][]string{"deb": {"libgomp1"}, "rpm": {"libgomp"}},
-		Binary:   "llama-server",
-	}
-	if arch == "x64" {
-		// The archive carries a backend per CPU generation and loads the best
-		// one, down to SSE4.2. Below AVX2 it runs, several times slower.
-		b.CPUFlags = []string{"avx2"}
-	}
-	// The CUDA build is published for x64 at this version; on arm64 it is a
-	// newer CUDA that the driver may not support, so an arm64 GPU node runs
-	// the CPU build until someone decides otherwise.
-	if h.NVIDIADriver != "" && arch == "x64" {
-		name := fmt.Sprintf("bin-ubuntu-cuda-%s-x64.tar.gz", cudaForLlamaCpp)
-		b.Variant = "cuda-" + cudaForLlamaCpp
-		b.Assets = []string{
-			fmt.Sprintf("llama-%s-%s", version, name),
-			fmt.Sprintf("cudart-llama-%s-%s", version, name),
+	base := func(variant string, assets ...string) Build {
+		b := Build{
+			Variant: variant,
+			Assets:  assets,
+			// Everything else the build links is in the archive or in any
+			// current glibc system; OpenMP's runtime is the one a cloud image
+			// lacks, and the server does not start without it.
+			Packages: map[string][]string{"deb": {"libgomp1"}, "rpm": {"libgomp"}},
+			Binary:   "llama-server",
 		}
+		if arch == "x64" {
+			// The archive carries a backend per CPU generation and loads the
+			// best one, down to SSE4.2. Below AVX2 it runs, several times
+			// slower.
+			b.CPUFlags = []string{"avx2"}
+		}
+		return b
 	}
-	return b, nil
+	if h.NVIDIADriver == "" {
+		return pick("llama.cpp", variant, []candidate{
+			{base("cpu", fmt.Sprintf("llama-%s-bin-ubuntu-%s.tar.gz", version, arch)), gpuNeeds{}, "llama.cpp"},
+		}, h)
+	}
+	cuda := func(cv string) Build {
+		name := fmt.Sprintf("bin-ubuntu-cuda-%s-%s.tar.gz", cv, arch)
+		// The release carries the CUDA runtime beside the build, so the node
+		// needs a driver and nothing else — no toolkit.
+		b := base("cuda-"+cv, fmt.Sprintf("llama-%s-%s", version, name), fmt.Sprintf("cudart-llama-%s-%s", version, name))
+		b.GPUBackend = "libggml-cuda.so"
+		return b
+	}
+	vulkan := base("vulkan", fmt.Sprintf("llama-%s-bin-ubuntu-vulkan-%s.tar.gz", version, arch))
+	vulkan.GPUBackend = "libggml-vulkan.so"
+	vulkan.Packages = map[string][]string{"deb": {"libgomp1", "libvulkan1"}, "rpm": {"libgomp", "vulkan-loader"}}
+
+	cuda12Driver := 570
+	if slices.Contains(llamaCUDA12Real, h.MinComputeCapability) {
+		cuda12Driver = minDriverForCUDA12
+	}
+	cands := []candidate{{cuda("13.3"), gpuNeeds{minDriverForCUDA13, 7.5}, "llama.cpp built for CUDA 13.3"}}
+	if arch == "x64" {
+		cands = append(cands, candidate{cuda("12.8"), gpuNeeds{cuda12Driver, 5.0}, "llama.cpp built for CUDA 12.8"})
+	}
+	cands = append(cands, candidate{vulkan, gpuNeeds{}, "llama.cpp built for Vulkan"})
+	return pick("llama.cpp", variant, cands, h)
 }
 
 func orUnknown(s string) string {

internal/agent/weights/set.go

@@ -2,6 +2,8 @@ package weights
 
 import (
 	"context"
+	"crypto/sha256"
+	"encoding/hex"
 	"encoding/json"
 	"fmt"
 	"io"
@@ -26,8 +28,9 @@ type manifest struct {
 	Model   string `json:"model"`
 	Version string `json:"version"`
 	Files   []struct {
-		Name  string `json:"name"`
-		Bytes int64  `json:"bytes"`
+		Name   string `json:"name"`
+		Bytes  int64  `json:"bytes"`
+		SHA256 string `json:"sha256"`
 	} `json:"files"`
 	Bytes int64 `json:"bytes"`
 }
@@ -87,6 +90,25 @@ func (f *Fetcher) ensureSet(ctx context.Context, source string) (string, error)
 		}
 	}
 
+	// Each file against the checksum the mirror took when it fetched it,
+	// before the set is called complete: a shard damaged on the way loads as
+	// a model that answers nonsense, not as an error.
+	for _, file := range m.Files {
+		if file.SHA256 == "" {
+			continue
+		}
+		target := filepath.Join(dir, file.Name)
+		got, err := fileSHA256(target)
+		if err != nil {
+			return "", err
+		}
+		if !strings.EqualFold(got, file.SHA256) {
+			_ = os.Remove(target)
+			return "", fmt.Errorf("%s%s: sha256 %s, the mirror has %s; removed, to be fetched again",
+				source, file.Name, got, file.SHA256)
+		}
+	}
+
 	if err := os.WriteFile(done, []byte(source+"\n"), 0o644); err != nil {
 		return "", err
 	}
@@ -108,6 +130,12 @@ func (f *Fetcher) manifestFor(ctx context.Context, source string) (manifest, err
 		return manifest{}, fmt.Errorf("asking %s what it holds: %w", source, err)
 	}
 	defer resp.Body.Close()
+	if resp.StatusCode == http.StatusConflict {
+		// Still downloading on the mirror, or a shard missing: said in the
+		// mirror's words, which name what is not there yet.
+		why, _ := io.ReadAll(io.LimitReader(resp.Body, 4<<10))
+		return manifest{}, fmt.Errorf("%s is not complete on the mirror yet: %s", source, strings.TrimSpace(string(why)))
+	}
 	if resp.StatusCode != http.StatusOK {
 		return manifest{}, fmt.Errorf("%s answered %d", source, resp.StatusCode)
 	}
@@ -127,3 +155,16 @@ func (f *Fetcher) manifestFor(ctx context.Context, source string) (manifest, err
 	}
 	return m, nil
 }
+
+func fileSHA256(path string) (string, error) {
+	f, err := os.Open(path) //nolint:gosec // a file this fetcher wrote
+	if err != nil {
+		return "", err
+	}
+	defer f.Close()
+	h := sha256.New()
+	if _, err := io.Copy(h, f); err != nil {
+		return "", err
+	}
+	return hex.EncodeToString(h.Sum(nil)), nil
+}

internal/agent/weights/set_test.go

@@ -1,6 +1,8 @@
 package weights
 
 import (
+	"crypto/sha256"
+	"encoding/hex"
 	"fmt"
 	"net/http"
 	"net/http/httptest"
@@ -165,3 +167,40 @@ func TestOnlyATrailingSlashMeansASet(t *testing.T) {
 		}
 	}
 }
+
+func TestASetFileThatDoesNotMatchTheMirrorsChecksumIsRefused(t *testing.T) {
+	good := "the shard as mirrored"
+	sum := sha256.Sum256([]byte(good))
+	served := "the shard damaged on the way"
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if strings.HasSuffix(r.URL.Path, "/") {
+			fmt.Fprintf(w, `{"model":"m","version":"v","files":[{"name":"s.safetensors","bytes":%d,"sha256":%q}]}`,
+				len(served), hex.EncodeToString(sum[:]))
+			return
+		}
+		_, _ = w.Write([]byte(served))
+	}))
+	t.Cleanup(srv.Close)
+	f := &Fetcher{CacheDir: t.TempDir()}
+	if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/"); err == nil || !strings.Contains(err.Error(), "sha256") {
+		t.Fatalf("a damaged shard was accepted: %v", err)
+	}
+	served = good
+	// The size differs, so it is fetched again; this time it matches.
+	if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/"); err != nil {
+		t.Fatalf("the shard as mirrored: %v", err)
+	}
+}
+
+func TestASetStillDownloadingOnTheMirrorSaysSo(t *testing.T) {
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		http.Error(w, "m/v: this version is not complete: still downloading model-00002-of-00002.safetensors", http.StatusConflict)
+	}))
+	t.Cleanup(srv.Close)
+	f := &Fetcher{CacheDir: t.TempDir()}
+	_, err := f.Ensure(t.Context(), srv.URL+"/m/v/")
+	if err == nil || !strings.Contains(err.Error(), "not complete on the mirror yet") ||
+		!strings.Contains(err.Error(), "model-00002-of-00002") {
+		t.Fatalf("got %v", err)
+	}
+}

internal/cli/backup.go

@@ -8,7 +8,6 @@ import (
 	"fmt"
 	"io"
 	"io/fs"
-	"net/url"
 	"os"
 	"os/exec"
 	"path/filepath"
@@ -178,30 +177,49 @@ func addStore(ctx context.Context, tw *tar.Writer, addr, scratch string) error {
 // command line: a DSN carries the password, and every user on the machine
 // can read another process's arguments.
 func pgDump(ctx context.Context, dsn, out string) error {
-	u, err := url.Parse(dsn)
+	vars, err := pgEnv(dsn)
 	if err != nil {
-		return fmt.Errorf("the store address is not a URL")
+		return err
 	}
-	env := os.Environ()
+	cmd := exec.CommandContext(ctx, "pg_dump", "--format=custom", "--no-owner", "--file="+out) //nolint:gosec // fixed binary, our own file
+	cmd.Env = append(os.Environ(), vars...)
+	if b, err := cmd.CombinedOutput(); err != nil {
+		return fmt.Errorf("pg_dump: %v: %s", err, strings.TrimSpace(string(b)))
+	}
+	return nil
+}
+
+// pgEnv turns a store address into libpq's environment. Several hosts — the
+// members of a replicated cluster — become libpq's own lists in PGHOST and
+// PGPORT; read as one URL host, the list was one name libpq could not
+// resolve, and a backup worked only by falling through to the second.
+func pgEnv(dsn string) ([]string, error) {
+	hosts, ports, u, err := store.SplitPostgres(dsn)
+	if err != nil {
+		return nil, err
+	}
+	var env []string
 	set := func(k, v string) {
 		if v != "" {
 			env = append(env, k+"="+v)
 		}
 	}
-	set("PGHOST", u.Hostname())
-	set("PGPORT", u.Port())
+	set("PGHOST", strings.Join(hosts, ","))
+	set("PGPORT", strings.Join(ports, ","))
 	set("PGUSER", u.User.Username())
 	if pw, ok := u.User.Password(); ok {
 		set("PGPASSWORD", pw)
 	}
 	set("PGDATABASE", strings.TrimPrefix(u.Path, "/"))
 	set("PGSSLMODE", u.Query().Get("sslmode"))
-	cmd := exec.CommandContext(ctx, "pg_dump", "--format=custom", "--no-owner", "--file="+out) //nolint:gosec // fixed binary, our own file
-	cmd.Env = env
-	if b, err := cmd.CombinedOutput(); err != nil {
-		return fmt.Errorf("pg_dump: %v: %s", err, strings.TrimSpace(string(b)))
+	// A standby if there is one, else whichever answers — not the address's
+	// read-write: a dump only reads, it need not load the primary, and one
+	// that insists on the primary fails exactly while the cluster is failing
+	// over.
+	if len(hosts) > 1 {
+		set("PGTARGETSESSIONATTRS", "prefer-standby")
 	}
-	return nil
+	return env, nil
 }
 
 func addTree(tw *tar.Writer, root, as string) error {

internal/cli/backup_test.go

@@ -139,3 +139,26 @@ func TestNoScratchCopyOfTheStoreIsLeftBehind(t *testing.T) {
 		}
 	}
 }
+
+func TestAStoreOfSeveralHostsIsGivenToPgDumpAsLibpqLists(t *testing.T) {
+	// The address of a Patroni cluster: url.Parse read its host as one name,
+	// and pg_dump reached the database only by falling through to the second.
+	env, err := pgEnv("postgres://llmgw:s3cret@10.0.0.149:5432,10.0.0.150/llmgw?sslmode=disable&target_session_attrs=read-write")
+	if err != nil {
+		t.Fatal(err)
+	}
+	got := strings.Join(env, " ")
+	for _, want := range []string{"PGHOST=10.0.0.149,10.0.0.150", "PGPORT=5432,5432", "PGUSER=llmgw",
+		"PGPASSWORD=s3cret", "PGDATABASE=llmgw", "PGSSLMODE=disable", "PGTARGETSESSIONATTRS=prefer-standby"} {
+		if !strings.Contains(got, want) {
+			t.Errorf("%s is not in %s", want, got)
+		}
+	}
+	one, err := pgEnv("postgres://u:p@db.internal:6432/llmgw")
+	if err != nil || !strings.Contains(strings.Join(one, " "), "PGHOST=db.internal PGPORT=6432") {
+		t.Errorf("one host: %v, %v", one, err)
+	}
+	if _, err := pgEnv("postgres://u:p@a:5432,:5432/x"); err == nil {
+		t.Error("an empty host in the list was accepted")
+	}
+}

internal/cli/ctl.go

@@ -143,6 +143,7 @@ func newCtlCmd() *cobra.Command {
 		newRetryCmd(client),
 		newUpgradeCmd(client),
 		newEngineCheckCmd(),
+		newEngineBuildCmd(client),
 		newToolCheckCmd(),
 		newWeightsCmd(client, &asJSON),
 		newCapacityCmd(client, &asJSON),

internal/cli/ctl_restart.go

@@ -12,8 +12,9 @@ import (
 
 func newRestartCmd(client func() *ctlClient) *cobra.Command {
 	var (
-		models []string
-		all    bool
+		models  []string
+		all     bool
+		inPlace bool
 	)
 	c := &cobra.Command{
 		Use:   "restart [REPLICA...]",
@@ -22,10 +23,14 @@ func newRestartCmd(client func() *ctlClient) *cobra.Command {
 			"an edited profile, a new engine version.\n\n" +
 			"Each replacement is placed beside the replica it replaces, and the old one\n" +
 			"keeps serving until the new one is ready. Replicas of one model are\n" +
-			"replaced one at a time. Where nothing else has room for a second copy —\n" +
-			"the usual case on a small CPU fleet — the old one is stopped after the\n" +
-			"drain grace period (5 minutes by default) and the new one starts in its\n" +
-			"place; a model with only that replica is unserved while it loads.\n\n" +
+			"replaced one at a time.\n\n" +
+			"Where there is no room for a second copy and the old replica is all that\n" +
+			"is in the way — one GPU, every card that fits held — it is stopped at once\n" +
+			"and the new one starts in its place; a model with only that replica is\n" +
+			"unserved while it loads. Where the new one would not fit even then — a\n" +
+			"profile edited to need more than any node has — the restart is given up\n" +
+			"and the old replica keeps serving: llmgw ctl replicas says so, and why.\n\n" +
+			"--in-place stops each replica first, whatever room there is.\n\n" +
 			"  llmgw ctl restart qwen2.5-7b-3f2a91c0\n" +
 			"  llmgw ctl restart --model qwen2.5-7b\n" +
 			"  llmgw ctl restart --all",
@@ -33,7 +38,7 @@ func newRestartCmd(client func() *ctlClient) *cobra.Command {
 			if len(args) == 0 && len(models) == 0 && !all {
 				return errors.New("name replicas, --model, or --all")
 			}
-			body, _ := json.Marshal(adminapi.RestartRequest{Replicas: args, Models: models, All: all})
+			body, _ := json.Marshal(adminapi.RestartRequest{Replicas: args, Models: models, All: all, InPlace: inPlace})
 			var res adminapi.RestartResult
 			if err := client().post(cmd.Context(), "/api/v1/restart", body, &res); err != nil {
 				return err
@@ -44,6 +49,7 @@ func newRestartCmd(client func() *ctlClient) *cobra.Command {
 	}
 	c.Flags().StringArrayVar(&models, "model", nil, "every replica of this model; repeatable")
 	c.Flags().BoolVar(&all, "all", false, "every ready replica in the cluster")
+	c.Flags().BoolVar(&inPlace, "in-place", false, "stop each replica before starting its replacement")
 	return c
 }
 

internal/cli/ctl_status_test.go

@@ -1,9 +1,14 @@
 package cli
 
 import (
+	"bytes"
 	"net/http"
 	"net/http/httptest"
+	"strings"
 	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"github.com/spf13/cobra"
 )
 
 func TestAQueuedChangeIsNotReportedAsAnError(t *testing.T) {
@@ -22,3 +27,27 @@ func TestAQueuedChangeIsNotReportedAsAnError(t *testing.T) {
 		t.Errorf("the body of a 202 was not read: %+v", out)
 	}
 }
+
+func TestStatusSaysWhoLeadsTheStoreAndWhetherTheRestKeepUp(t *testing.T) {
+	render := func(s adminapi.Summary) string {
+		cmd := &cobra.Command{}
+		var b bytes.Buffer
+		cmd.SetOut(&b)
+		printSummary(cmd, s)
+		return b.String()
+	}
+	out := render(adminapi.Summary{Store: []adminapi.StoreMember{
+		{Name: "llmgw-ctl02", Role: "replica", State: "streaming", Lag: 2048},
+		{Name: "llmgw-pg01", Role: "leader", State: "running"},
+	}})
+	if !strings.Contains(out, "STORE     llmgw-ctl02 streaming (2 KB behind), llmgw-pg01 leads") {
+		t.Errorf("status:\n%s", out)
+	}
+	out = render(adminapi.Summary{Store: []adminapi.StoreMember{{Name: "llmgw-pg01", Role: "replica", State: "stopped"}}})
+	if !strings.Contains(out, "NO LEADER") {
+		t.Errorf("a store with no leader was not called out:\n%s", out)
+	}
+	if out := render(adminapi.Summary{}); strings.Contains(out, "STORE") {
+		t.Error("a single database got a store line")
+	}
+}

internal/cli/doctor.go

@@ -23,6 +23,7 @@ func newDoctorCmd() *cobra.Command {
 		opts    doctor.Options
 		asJSON  bool
 		verbose bool
+		fabrics []string
 	)
 	cmd := &cobra.Command{
 		Use:   "doctor",
@@ -38,8 +39,21 @@ that file when the report is not enough to go on.`,
 		Args: cobra.NoArgs,
 		RunE: func(cmd *cobra.Command, _ []string) error {
 			doctor.SetConfigLoader(config.LoadAgent)
-			if opts.VLLMVersion == "" {
+			for _, spec := range fabrics {
+				name, hcas, _ := strings.Cut(spec, ":")
+				if name == "" || hcas == "" {
+					return fmt.Errorf("--fabric %q: give it as name:mlx5_0,mlx5_1", spec)
+				}
+				if opts.Fabrics == nil {
+					opts.Fabrics = map[string][]string{}
+				}
+				opts.Fabrics[name] = strings.Split(hcas, ",")
+			}
+			switch opts.VLLMVersion {
+			case "":
 				opts.VLLMVersion = config.DefaultEngines["vllm"]
+			case "none":
+				opts.VLLMVersion = ""
 			}
 			ctx, cancel := context.WithTimeout(cmd.Context(), 60*time.Second)
 			defer cancel()
@@ -66,7 +80,8 @@ that file when the report is not enough to go on.`,
 	f.StringVar(&opts.ConfigPath, "config", "", "agent configuration (default /etc/llmgw/agent.yaml when present)")
 	f.StringVar(&opts.IBRoot, "ib-root", discovery.SysfsRoot, "sysfs InfiniBand directory")
 	f.StringVar(&opts.NvidiaSMI, "nvidia-smi", "", "path to nvidia-smi")
-	f.StringVar(&opts.VLLMVersion, "vllm", "", "the vLLM version to check for (default: the one this llmgw installs)")
+	f.StringVar(&opts.VLLMVersion, "vllm", "", "the vLLM version to check for (default: the one this llmgw installs; none to skip)")
+	f.StringArrayVar(&fabrics, "fabric", nil, "a fabric about to be declared, name:mlx5_0,mlx5_1, checked as if it were; repeatable")
 	f.BoolVar(&asJSON, "json", false, "write every fact and finding as JSON, to send back")
 	f.BoolVar(&verbose, "verbose", false, "also print the raw nvidia-smi topology")
 	return cmd

internal/cli/enginebuild.go

@@ -0,0 +1,39 @@
+package cli
+
+import (
+	"encoding/json"
+	"fmt"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+)
+
+func newEngineBuildCmd(client func() *ctlClient) *cobra.Command {
+	return &cobra.Command{
+		Use:   "engine-build <node> <engine> <build>",
+		Short: "Choose which build of an engine a GPU node installs",
+		Long: "A node installs the build of vLLM its hardware runs: CUDA 13 where the\n" +
+			"driver and GPUs allow, else the same version built for CUDA 12.9. This\n" +
+			"chooses instead, as the cluster page's vLLM build column does:\n\n" +
+			"  auto    what the hardware runs (the default)\n" +
+			"  cu13    the CUDA 13 build, from PyPI\n" +
+			"  cu129   the CUDA 12.9 build, from vLLM's GitHub release\n" +
+			"  none    no vLLM on this node\n\n" +
+			"The node is told at once and installs it in the background; a build it\n" +
+			"cannot compute with on its GPU fails to install, saying why, and the\n" +
+			"cluster page shows it. A V100 runs no build of vLLM 0.20 or later, and\n" +
+			"gets 0.19.1, the newest built for it, by itself.\n\n" +
+			"  llmgw ctl engine-build gpu07 vllm cu129",
+		Args: cobra.ExactArgs(3),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			body, _ := json.Marshal(adminapi.NodeEngineRequest{Engine: args[1], Build: args[2]})
+			if err := client().post(cmd.Context(), "/api/v1/nodes/"+args[0]+"/engines", body, nil); err != nil {
+				return err
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "%s: %s build %s; the node installs it in the background (llmgw ctl nodes shows it)\n",
+				args[0], args[1], args[2])
+			return nil
+		},
+	}
+}

internal/cli/enginecheck.go

@@ -3,6 +3,7 @@ package cli
 import (
 	"context"
 	"fmt"
+	"io"
 	"os/exec"
 	"path/filepath"
 	"sort"
@@ -126,6 +127,14 @@ func newEngineCheckCmd() *cobra.Command {
 			}
 			sort.Strings(unknown)
 
+			// Flags accepted is not the same as running where it is placed: a
+			// GPU build whose backend does not load accepts every flag and
+			// computes on the CPU. Asked of the engine here, not of a help.
+			var deviceErr error
+			if fb, ok := eng.(engines.CPUFallback); ok && help == "" {
+				deviceErr = checkDevices(cmd.Context(), name, fb, out)
+			}
+
 			fmt.Fprintf(out, "%s: %d flags checked", name, checked)
 			if skipped > 0 {
 				fmt.Fprintf(out, ", %d commands skipped", skipped)
@@ -137,7 +146,7 @@ func newEngineCheckCmd() *cobra.Command {
 					return fmt.Errorf("nothing was checked; install the engine here or pass --help-output")
 				}
 				fmt.Fprintln(out, "every flag llmgw builds is accepted")
-				return nil
+				return deviceErr
 			}
 			fmt.Fprintf(out, "\n%d flags not mentioned by the program they are passed to:\n", len(unknown))
 			for _, f := range unknown {
@@ -249,3 +258,22 @@ func isFlagChar(b byte) bool {
 	return b == '-' || b == '_' || (b >= 'a' && b <= 'z') ||
 		(b >= 'A' && b <= 'Z') || (b >= '0' && b <= '9')
 }
+
+// checkDevices says which GPUs an engine that can fall back to the CPU sees on
+// this machine, and fails when the machine has NVIDIA GPUs and it sees none.
+func checkDevices(ctx context.Context, name string, fb engines.CPUFallback, out io.Writer) error {
+	if _, err := exec.LookPath("nvidia-smi"); err != nil {
+		fmt.Fprintln(out, "no NVIDIA GPU here: which devices it computes on was not checked")
+		return nil
+	}
+	devs, err := fb.Devices(ctx)
+	if err != nil {
+		return err
+	}
+	if len(devs) == 0 {
+		return fmt.Errorf("this machine has NVIDIA GPUs and %s sees none: it would serve a GPU placement on the CPU. "+
+			"Its GPU backend did not load; ldd on its libggml-cuda.so names the missing library", name)
+	}
+	fmt.Fprintf(out, "computes on: %s\n", strings.Join(devs, "; "))
+	return nil
+}

internal/cli/join.go

@@ -0,0 +1,61 @@
+package cli
+
+import (
+	"errors"
+	"fmt"
+	"os"
+	"runtime"
+
+	"github.com/spf13/cobra"
+
+	joinpkg "gitea.mynodes.xyz/ilker/llmgw/internal/join"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/release"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/version"
+)
+
+func newJoinCmd() *cobra.Command {
+	return &cobra.Command{
+		Use:   "join <code>",
+		Short: "Let a setup on another machine set this one up, with the code it gave",
+		Long: "For a machine nothing may log into: the setup screens (sudo llmgw on\n" +
+			"the control plane) show a code for each machine they expect. Run this\n" +
+			"with it, as root, on that machine. It dials out to the setup — nothing\n" +
+			"dials in — checks it is the setup the code was made by, and runs what\n" +
+			"the setup sends until it is done: the same checks and installs ssh\n" +
+			"would carry. Each step is printed here as it runs.\n\n" +
+			"A code is good for one machine, for an hour, and only while the setup\n" +
+			"that made it is waiting.",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			if runtime.GOOS != "linux" {
+				return fmt.Errorf("join sets up a Linux machine; this is %s", runtime.GOOS)
+			}
+			if os.Geteuid() != 0 {
+				return errors.New("run it as root — sudo llmgw join … — since the setup installs software here")
+			}
+			code, err := joinpkg.Parse(args[0])
+			if err != nil {
+				return err
+			}
+			out := cmd.OutOrStdout()
+			role := code.Role
+			if role == "" {
+				role = "part of a cluster"
+			}
+			fmt.Fprintf(out, "Joining the setup at %s, as %s. It runs as root here until it is done.\n", code.Addr, role)
+			return joinpkg.Serve(cmd.Context(), code, func(s string) { fmt.Fprintln(out, s) })
+		},
+	}
+}
+
+// joinInstall is what to run on a machine that has no llmgw yet, to fetch
+// this version for its architecture: short lines, since it is copied from a
+// screen that cuts long ones off.
+func joinInstall() []string {
+	v := version.Version
+	return []string{
+		`a=$(uname -m | sed 's/x86_64/amd64/;s/aarch64/arm64/')`,
+		fmt.Sprintf("sudo curl -fsSL -o /usr/local/bin/llmgw \\\n    %s/%s/llmgw-%s-linux-$a", release.DefaultBase, v, v),
+		"sudo chmod 755 /usr/local/bin/llmgw",
+	}
+}

internal/cli/mirror.go

@@ -2,6 +2,7 @@ package cli
 
 import (
 	"encoding/json"
+	"errors"
 	"fmt"
 	"log/slog"
 	"net"
@@ -56,52 +57,69 @@ func newMirrorAddCmd(dir *string) *cobra.Command {
 	var model, version, as string
 
 	c := &cobra.Command{
-		Use:   "add <url>",
+		Use:   "add <url>...",
 		Short: "Download a model into the mirror",
 		Long: "Downloads to a temporary name and renames when complete, so an\n" +
 			"interrupted download never leaves something a catalog could be pointed\n" +
 			"at. Reports the checksum: a mirror is the one place in this system where\n" +
 			"a corrupted file is copied to every node rather than caught once.\n\n" +
+			"Several URLs go into one version, as the files of one model: the shards\n" +
+			"of a safetensors set with its config.json and tokenizer, for vLLM. The\n" +
+			"version is offered to nodes only once it is whole — nothing still\n" +
+			"downloading, every shard its index names present — and its source is\n" +
+			"the version, ending in a slash.\n\n" +
 			"Refuses to replace a version that is already here. Add a new one beside\n" +
 			"it instead — that is what makes the old one still available when the new\n" +
 			"one turns out to be worse.",
-		Args: cobra.ExactArgs(1),
+		Args: cobra.MinimumNArgs(1),
 		RunE: func(cmd *cobra.Command, args []string) error {
-			source := args[0]
-			file := as
-			if file == "" {
-				file = mirror.LastPathElement(source)
-			}
 			if model == "" || version == "" {
 				return fmt.Errorf("--model and --version are required: they are the layout")
 			}
-
-			e := mirror.Entry{Model: model, Version: version, File: file}
-			out := cmd.OutOrStdout()
-			fmt.Fprintf(out, "fetching %s\n", source)
-
-			added, err := mirror.Add(cmd.Context(), *dir, source, e, func(done, total int64) {
-				if total > 0 {
-					fmt.Fprintf(out, "  %s of %s (%d%%)\n",
-						fileSize(done), fileSize(total), done*100/total)
-					return
-				}
-				fmt.Fprintf(out, "  %s so far\n", fileSize(done))
-			})
-			if err != nil {
-				return err
+			if as != "" && len(args) > 1 {
+				return fmt.Errorf("--as names one file; with %d URLs each keeps its own name", len(args))
 			}
-			fmt.Fprintf(out, "\n%s/%s/%s  %s\n  sha256 %s\n",
-				added.Entry.Model, added.Entry.Version, added.Entry.File,
-				fileSize(added.Entry.Bytes), added.SHA256)
-			fmt.Fprintf(out, "\nPoint a model at it with:\n  source: %s\n",
-				added.Entry.URL("http://<this-host>:8081"))
+			out := cmd.OutOrStdout()
+			var last mirror.Added
+			for _, source := range args {
+				file := as
+				if file == "" {
+					file = mirror.LastPathElement(source)
+				}
+				e := mirror.Entry{Model: model, Version: version, File: file}
+				fmt.Fprintf(out, "fetching %s\n", source)
+				added, err := mirror.Add(cmd.Context(), *dir, source, e, func(done, total int64) {
+					if total > 0 {
+						fmt.Fprintf(out, "  %s of %s (%d%%)\n",
+							fileSize(done), fileSize(total), done*100/total)
+						return
+					}
+					fmt.Fprintf(out, "  %s so far\n", fileSize(done))
+				})
+				if err != nil {
+					return err
+				}
+				fmt.Fprintf(out, "%s/%s/%s  %s\n  sha256 %s\n",
+					added.Entry.Model, added.Entry.Version, added.Entry.File,
+					fileSize(added.Entry.Bytes), added.SHA256)
+				last = added
+			}
+			src := last.Entry.URL("http://<this-host>:8081")
+			if len(args) > 1 {
+				src = strings.TrimSuffix(mirror.Entry{Model: model, Version: version}.URL("http://<this-host>:8081"), "/") + "/"
+				if m, err := mirror.ManifestFor(*dir, model, version); err != nil {
+					fmt.Fprintf(out, "\nNot yet offered to nodes: %v\n", err)
+				} else {
+					fmt.Fprintf(out, "\n%d files, %s, whole.\n", len(m.Files), fileSize(m.Bytes))
+				}
+			}
+			fmt.Fprintf(out, "\nPoint a model at it with:\n  source: %s\n", src)
 			return nil
 		},
 	}
 	c.Flags().StringVar(&model, "model", "", "the model's name, one directory level")
 	c.Flags().StringVar(&version, "version", "", "the version, one directory level; a date works well")
-	c.Flags().StringVar(&as, "as", "", "the filename to save as; taken from the URL when omitted")
+	c.Flags().StringVar(&as, "as", "", "the filename to save as, for one URL; taken from the URL when omitted")
 	return c
 }
 
@@ -405,7 +423,14 @@ func withManifest(root string, next http.Handler) http.Handler {
 			return
 		}
 		m, err := mirror.ManifestFor(root, model, version)
-		if err != nil {
+		switch {
+		case errors.Is(err, mirror.ErrIncomplete):
+			// Said, not hidden behind a 404: a node asking now is told why
+			// it cannot have the version yet, and tries again later.
+			w.Header().Set("Retry-After", "60")
+			http.Error(w, err.Error(), http.StatusConflict)
+			return
+		case err != nil:
 			http.NotFound(w, r)
 			return
 		}

internal/cli/node.go

@@ -12,6 +12,7 @@ import (
 
 	"github.com/spf13/cobra"
 
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
@@ -113,13 +114,14 @@ engines are kept.`,
 	f.StringVar(&na.opts.MgmtAddr, "mgmt-addr", "", "address engines are reached at (default: found by the agent)")
 	f.StringVar(&na.opts.MgmtIface, "mgmt-if", "", "interface NCCL bootstraps over (default: the one holding that address)")
 	f.StringVar(&na.opts.LlamaCppBin, "llamacpp-bin", "", "a llama-server you installed, instead of the one the agent installs")
-	f.StringVar(&na.opts.VLLMBin, "vllm-bin", "", "a vllm you installed; also set engines.vllm: \"\" in server.yaml")
+	f.StringVar(&na.opts.VLLMBin, "vllm-bin", "", "a vllm you installed, instead of the one the agent installs")
 	f.StringVar(&na.opts.SGLangPython, "sglang-python", "", "python of an SGLang virtualenv you installed")
 	f.StringVar(&na.opts.NvidiaSMI, "nvidia-smi", "", "where nvidia-smi is, when it is not on PATH")
 	f.StringVar(&na.opts.IBRoot, "ib-root", "", "the sysfs InfiniBand directory, when not /sys/class/infiniband")
 	f.StringVar(&na.opts.PKI, "pki", defaultPKIDir, "directory holding the certificate authority")
 	f.StringVar(&na.opts.Releases, "releases", "https://gitea.mynodes.xyz/ilker/llmgw/releases/download", "where release files are downloaded from")
 	f.BoolVar(&na.opts.IgnoreChecks, "ignore-checks", false, "add the node even when doctor finds a problem")
+	f.StringArrayVar(&na.opts.DisableEngines, "disable-engine", nil, "an engine this node never installs, e.g. vllm on GPUs it does not run on; repeatable")
 	f.StringVar(&na.admin, "admin", "", "the control plane's admin API, to watch the node join (default: from server.yaml, else 127.0.0.1:9090)")
 	f.StringVar(&na.serverCfg, "config", "/etc/llmgw/server.yaml", "the control plane's configuration, for the admin API's address")
 	f.BoolVar(&na.waitAll, "wait", false, "wait for every engine, vLLM included, not only llama.cpp")
@@ -148,6 +150,23 @@ func (na *nodeAdd) prepare() error {
 			}
 		}
 	}
+	for _, e := range o.DisableEngines {
+		if !slices.Contains(provision.Installable(), e) {
+			return fmt.Errorf("--disable-engine %s: the engines are %s", e, strings.Join(provision.Installable(), ", "))
+		}
+	}
+	// What the control plane will ask the node to install, so doctor checks
+	// that vLLM rather than its own default. Its overrides for a GPU class
+	// are not known until the node says what it has, and are not applied
+	// here; doctor's verdict on vLLM is a warning either way.
+	if o.VLLMVersion == "" {
+		if cfg, err := config.Load(na.serverCfg); err == nil {
+			o.VLLMVersion = cfg.WantedEngines()["vllm"]
+			if o.VLLMVersion == "" {
+				o.VLLMVersion = "none"
+			}
+		}
+	}
 	o.Version = version.Version
 	if o.Binary == "" {
 		exe, err := os.Executable()

internal/cli/root.go

@@ -30,6 +30,14 @@ func Root() *cobra.Command {
 		PersistentPreRun: func(_ *cobra.Command, _ []string) {
 			obs.Setup(obs.Options{Level: global.logLevel, Format: global.logFormat})
 		},
+		// Bare, in a terminal: the setup screens or the console. Otherwise
+		// the help, as before.
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			if interactive(cmd) {
+				return runHome(cmd)
+			}
+			return cmd.Help()
+		},
 	}
 
 	pf := root.PersistentFlags()
@@ -49,6 +57,7 @@ func Root() *cobra.Command {
 		newNodeCmd(),
 		newSelfUpgradeCmd(),
 		newReleaseCmd(),
+		newJoinCmd(),
 	)
 	return root
 }

internal/cli/server.go

@@ -5,19 +5,24 @@ import (
 	"crypto/tls"
 	"errors"
 	"fmt"
-	"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
-	"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
-	"gitea.mynodes.xyz/ilker/llmgw/internal/sourcecheck"
-	"github.com/prometheus/client_golang/prometheus"
 	"log/slog"
+	"maps"
 	"net"
 	"net/http"
 	"os/signal"
+	"slices"
 	"strings"
 	"sync"
 	"syscall"
 	"time"
 
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/pgha"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/sourcecheck"
+	"github.com/prometheus/client_golang/prometheus"
+
 	"github.com/spf13/cobra"
 	"golang.org/x/net/http2"
 	"golang.org/x/net/http2/h2c"
@@ -123,6 +128,9 @@ func newServerCmd() *cobra.Command {
 			var agentHandler http.Handler
 			var agentPath string
 			var admin *control.Controller
+			// The store's cluster, when it is one; read by the metrics and
+			// by `ctl status`.
+			var storeHA *pgha.Collector
 			if controlMode {
 				cat, err := cfg.Catalog()
 				if err != nil {
@@ -175,6 +183,17 @@ func newServerCmd() *cobra.Command {
 				if err := rt.RegisterMetrics(fleetmetrics.New(ctl)); err != nil {
 					return err
 				}
+				// A store of several hosts is a replicated cluster: what state
+				// it is in is published beside everything else, read from its
+				// Patroni. One host is a single PostgreSQL, which has none.
+				if addr, err := storepkg.ResolveAddress(cfg.Auth.Store); err == nil && storepkg.IsPostgres(addr) {
+					if hosts, _, _, err := storepkg.SplitPostgres(strings.TrimSpace(addr)); err == nil && len(hosts) > 1 {
+						storeHA = pgha.NewCollector(hosts)
+						if err := rt.RegisterMetrics(storeHA); err != nil {
+							return err
+						}
+					}
+				}
 				// So a request for a model that is configured but not running
 				// starts it, rather than being told nothing serves it.
 				rt.SetActivator(ctl.Reconciler())
@@ -197,6 +216,16 @@ func newServerCmd() *cobra.Command {
 			)
 			if admin != nil {
 				api := adminapi.New(admin)
+				if storeHA != nil {
+					api.SetStoreStatus(func() ([]adminapi.StoreMember, error) {
+						c, err := storeHA.Read()
+						var out []adminapi.StoreMember
+						for _, m := range c.Members {
+							out = append(out, adminapi.StoreMember{Name: m.Name, Host: m.Host, Role: m.Role, State: m.State, Lag: m.Lag})
+						}
+						return out, err
+					})
+				}
 				adminAPI = api
 				api.SetReleases(cfg.Releases)
 				// Catalog changes need somewhere to persist, so this is wired
@@ -422,7 +451,7 @@ func newServerCmd() *cobra.Command {
 	f.StringVar(&metricsListen, "metrics-listen", "", "serve metrics only on this address, safe to expose")
 	f.StringArrayVar(&endpoints, "endpoint", nil, "static endpoint: id=URL=model[,model] (repeatable)")
 
-	cmd.AddCommand(newServerInitCmd())
+	cmd.AddCommand(newServerInitCmd(), newServerAddCmd())
 	cmd.AddCommand(newBackupCmd())
 	return cmd
 }
@@ -484,6 +513,9 @@ func startControlPlane(ctx context.Context, cfg config.Server, cat *catalog.Live
 	if h, ok := sink.(interface{ Store() *storepkg.DB }); ok {
 		if st := h.Store(); st != nil {
 			lead = control.NewLeadership(st, cfg.Control.LeaseTTL, log)
+			if r, ok := sink.(interface{ SetLeader(func() bool) }); ok {
+				r.SetLeader(lead.IsLeader)
+			}
 			ctl.SetLeadership(lead)
 			// The tenancy tree's capacity quotas: what each part is
 			// guaranteed and may hold.
@@ -491,13 +523,21 @@ func startControlPlane(ctx context.Context, cfg config.Server, cat *catalog.Live
 			// And what each model was measured to serve, for scaling and for
 			// choosing between profiles.
 			ctl.Reconciler().SetMeasurements(control.Measurements(st, log))
+			// The engine builds chosen per node on the cluster page.
+			ctl.SetEngineChoices(control.NewEngineChoices(ctx, st, log))
 			log.Info("competing to be the control plane that places work",
 				"owner", lead.Owner())
 		}
 	}
 	svc := agentsvc.New(ctl.Inventory(), ctl.Reconciler(), requireIdentity, log)
-	svc.SetEngines(cfg.WantedEngines())
-	log.Info("nodes install these engines", "engines", cfg.WantedEngines())
+	svc.SetEnginesFor(func(n inventory.Node) map[string]string {
+		return cfg.EnginesFor(n.ID, slices.Sorted(maps.Keys(n.GPUsByClass())), n.Labels)
+	})
+	log.Info("nodes install these engines", "engines", cfg.WantedEngines(), "overrides", len(cfg.EngineOverrides))
+	if ch := ctl.EngineChoices(); ch != nil {
+		svc.SetEngineChoices(ch.For)
+		ch.OnChange(svc.Reprovision)
+	}
 	// A standby must refuse agents as well as refusing to place. Commands
 	// travel down the stream the agent opened, so a standby holding one is a
 	// node the leader cannot command — and the leader's placements would fail
@@ -505,6 +545,21 @@ func startControlPlane(ctx context.Context, cfg config.Server, cat *catalog.Live
 	if lead != nil {
 		svc.SetLeadership(lead)
 	}
+	// The control planes agents may dial, from the store every control plane
+	// shares: told to each agent as it registers, and to all of them when
+	// `server add` records another.
+	if h, ok := sink.(interface{ Store() *storepkg.DB }); ok && h.Store() != nil {
+		st := h.Store()
+		svc.SetControlPlanes(func(ctx context.Context) ([]string, error) {
+			cps, err := st.ControlPlanes(ctx)
+			urls := make([]string, len(cps))
+			for i, c := range cps {
+				urls[i] = c.URL
+			}
+			return urls, err
+		})
+		go svc.WatchControlPlanes(ctx, 30*time.Second)
+	}
 	fwd.set(svc)
 	// Every InfiniBand fabric mapped every five minutes, through one node.
 	go svc.ScanFabrics(ctx, 5*time.Minute)

internal/cli/serveradd.go

@@ -0,0 +1,459 @@
+package cli
+
+import (
+	"bytes"
+	"context"
+	"crypto/rand"
+	"crypto/x509"
+	"encoding/hex"
+	"encoding/pem"
+	"errors"
+	"fmt"
+	"io"
+	"net"
+	"os"
+	"path/filepath"
+	"slices"
+	"strings"
+	"time"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/pgha"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/version"
+)
+
+// serverAdd makes another machine a control plane of this cluster: a standby
+// that takes over when this one is lost. Run on a control plane that holds
+// the authority, as root, like `node add`.
+type serverAdd struct {
+	etcDir, pkiDir string
+	// hosts are more names to put in the new control plane's certificate.
+	hosts []string
+	// vip is a floating address, "10.0.0.160/24", to keep on whichever of the
+	// two leads; "" leaves client traffic to whatever is in front.
+	vip string
+
+	remote onboard.Remote
+	// addr is the machine's address as given; ip what the others reach it at.
+	addr, ip, name string
+	out            io.Writer
+}
+
+func newServerAddCmd() *cobra.Command {
+	sa := serverAdd{etcDir: "/etc/llmgw", pkiDir: defaultPKIDir}
+	cmd := &cobra.Command{
+		Use:   "add [user@]address",
+		Short: "Make another machine a control plane of this cluster: a standby that takes over",
+		Long: "Run on a control plane, as root. It logs in to the machine with ssh, as\n" +
+			"node add does, and gives it this llmgw, a certificate from this\n" +
+			"cluster's authority, this control plane's configuration and the store's\n" +
+			"address, lets it into the store, and starts it. It stands by while this\n" +
+			"one leads, and takes over within seconds when this one is lost; every\n" +
+			"node is told of it and fails over to it without being touched.\n\n" +
+			"The store must be one both can reach — replicated with 'server init\n" +
+			"--patroni' — not a database on this machine.\n\n" +
+			"--vip keeps a floating address on whichever control plane leads, with\n" +
+			"keepalived on both: the address clients use for the gateway.\n\n" +
+			"Run it again at any time: it changes only what is missing.",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			if os.Geteuid() != 0 {
+				return errors.New("run it as root: it reads the certificate authority and the store's address")
+			}
+			user, host, ok := strings.Cut(args[0], "@")
+			if !ok {
+				user, host = "", args[0]
+			}
+			sa.remote = &onboard.SSH{Addr: host, User: user}
+			sa.addr = host
+			sa.out = cmd.OutOrStdout()
+			return sa.run(cmd.Context())
+		},
+	}
+	f := cmd.Flags()
+	f.StringArrayVar(&sa.hosts, "host", nil, "another name for the new control plane's certificate; repeatable")
+	f.StringVar(&sa.vip, "vip", "", "a floating address for whichever control plane leads, as 10.0.0.160/24")
+	f.StringVar(&sa.pkiDir, "pki", sa.pkiDir, "directory holding the certificate authority")
+	return cmd
+}
+
+func (sa *serverAdd) done(format string, a ...any) {
+	fmt.Fprintf(sa.out, "  ok    "+format+"\n", a...)
+}
+
+func (sa *serverAdd) kept(format string, a ...any) {
+	fmt.Fprintf(sa.out, "  kept  "+format+"\n", a...)
+}
+
+func (sa *serverAdd) sh(ctx context.Context, script string, stdin []byte) (string, error) {
+	out, err := sa.remote.Run(ctx, script, stdin)
+	if err != nil {
+		return string(out), fmt.Errorf("on %s: %v: %s", sa.addr, err, strings.TrimSpace(string(out)))
+	}
+	return string(out), nil
+}
+
+// put writes a file there, through standard input, with its mode and owner.
+func (sa *serverAdd) put(ctx context.Context, path, mode, owner string, body []byte) error {
+	_, err := sa.sh(ctx, fmt.Sprintf(`set -e
+sudo install -d -m 0755 %[1]s
+sudo sh -c 'umask 077; cat > %[2]s.new'
+sudo chmod %[3]s %[2]s.new
+sudo chown %[4]s %[2]s.new
+sudo mv -f %[2]s.new %[2]s`, filepath.Dir(path), path, mode, owner), body)
+	return err
+}
+
+func (sa *serverAdd) run(ctx context.Context) error {
+	steps := []struct {
+		name string
+		fn   func(context.Context) error
+	}{
+		{"connect", sa.connect},
+		{"binary", sa.binary},
+		{"directories", sa.dirs},
+		{"certificate", sa.certificate},
+		{"configuration", sa.configuration},
+		{"store", sa.storeAccess},
+		{"service", sa.service},
+	}
+	if sa.vip != "" {
+		steps = append(steps, struct {
+			name string
+			fn   func(context.Context) error
+		}{"floating address", sa.floating})
+	}
+	for _, s := range steps {
+		if err := s.fn(ctx); err != nil {
+			return fmt.Errorf("%s: %w", s.name, err)
+		}
+	}
+	fmt.Fprintf(sa.out, "\n%s is a control plane of this cluster, standing by while this one leads.\n", sa.name)
+	return nil
+}
+
+func (sa *serverAdd) connect(ctx context.Context) error {
+	out, err := sa.sh(ctx, "hostname; sudo -n true && echo sudo-ok", nil)
+	if err != nil {
+		return fmt.Errorf("%w — it needs ssh and sudo without a password", err)
+	}
+	lines := strings.Split(strings.TrimSpace(out), "\n")
+	sa.name = strings.TrimSpace(lines[0])
+	if sa.ip == "" {
+		if sa.ip, err = ipv4Of(sa.addr); err != nil {
+			return err
+		}
+	}
+	sa.done("%s (%s)", sa.name, sa.ip)
+	return nil
+}
+
+// binary gives it this very llmgw, so the two control planes never differ.
+func (sa *serverAdd) binary(ctx context.Context) error {
+	have, _ := sa.sh(ctx, "llmgw --version 2>/dev/null || true", nil)
+	if strings.TrimSpace(have) == "llmgw version "+version.Version {
+		sa.kept("llmgw %s", version.Version)
+		return nil
+	}
+	exe, err := os.Executable()
+	if err != nil {
+		return err
+	}
+	b, err := os.ReadFile(exe)
+	if err != nil {
+		return err
+	}
+	if _, err := sa.sh(ctx, `set -e; t=$(mktemp); cat > "$t"; sudo install -m 0755 "$t" /usr/local/bin/llmgw; rm -f "$t"`, b); err != nil {
+		return err
+	}
+	sa.done("llmgw %s, copied from this machine", version.Version)
+	return nil
+}
+
+func (sa *serverAdd) dirs(ctx context.Context) error {
+	_, err := sa.sh(ctx, `set -e
+id llmgw >/dev/null 2>&1 || sudo useradd --system --home /var/lib/llmgw --shell /usr/sbin/nologin llmgw
+sudo install -d -m 0755 /etc/llmgw
+sudo install -d -o llmgw -g llmgw -m 0700 /etc/llmgw/secrets
+sudo install -d -o llmgw -g llmgw -m 0750 /var/lib/llmgw`, nil)
+	if err == nil {
+		sa.done("user llmgw, /etc/llmgw and /var/lib/llmgw")
+	}
+	return err
+}
+
+// certificate issues it a server certificate from this cluster's authority,
+// for its name and addresses, unless it holds one that already names them.
+func (sa *serverAdd) certificate(ctx context.Context) error {
+	ca, err := loadCA(sa.pkiDir)
+	if err != nil {
+		return fmt.Errorf("the authority's key is not in %s: run this where it is (%v)", sa.pkiDir, err)
+	}
+	ips, _ := sa.sh(ctx, "hostname -I", nil)
+	want := []string{sa.name, sa.ip}
+	for _, ip := range strings.Fields(ips) {
+		if net.ParseIP(ip).To4() != nil && !slices.Contains(want, ip) {
+			want = append(want, ip)
+		}
+	}
+	want = append(want, sa.hosts...)
+
+	if have, _ := sa.sh(ctx, "sudo cat /etc/llmgw/server.pem 2>/dev/null || true", nil); have != "" {
+		if blk, _ := pem.Decode([]byte(have)); blk != nil {
+			if cert, err := x509.ParseCertificate(blk.Bytes); err == nil && cert.CheckSignatureFrom(ca.Cert) == nil {
+				names := strings.Split(sanNames(cert), ", ")
+				missing := false
+				for _, w := range want {
+					missing = missing || !slices.Contains(names, w)
+				}
+				if !missing {
+					sa.kept("certificate for %s", strings.Join(names, ", "))
+					return sa.put(ctx, "/etc/llmgw/ca.pem", "0644", "root:root", ca.PEM)
+				}
+			}
+		}
+	}
+	leaf, err := ca.IssueServer(sa.name, want)
+	if err != nil {
+		return err
+	}
+	if err := sa.put(ctx, "/etc/llmgw/ca.pem", "0644", "root:root", ca.PEM); err != nil {
+		return err
+	}
+	if err := sa.put(ctx, "/etc/llmgw/server.pem", "0644", "root:root", leaf.CertPEM); err != nil {
+		return err
+	}
+	if err := sa.put(ctx, "/etc/llmgw/server-key.pem", "0640", "root:llmgw", leaf.KeyPEM); err != nil {
+		return err
+	}
+	sa.done("certificate for %s, from this cluster's authority", strings.Join(want, ", "))
+	return nil
+}
+
+// configuration copies this control plane's, and the secrets beside it: the
+// two must agree on everything, the store above all.
+func (sa *serverAdd) configuration(ctx context.Context) error {
+	if have, _ := sa.sh(ctx, "sudo test -f /etc/llmgw/server.yaml && echo yes || true", nil); strings.TrimSpace(have) == "yes" {
+		sa.kept("/etc/llmgw/server.yaml (never overwritten)")
+	} else {
+		b, err := os.ReadFile(filepath.Join(sa.etcDir, "server.yaml"))
+		if err != nil {
+			return err
+		}
+		if err := sa.put(ctx, "/etc/llmgw/server.yaml", "0644", "root:root", b); err != nil {
+			return err
+		}
+		sa.done("/etc/llmgw/server.yaml, as this control plane's")
+	}
+	secrets, err := os.ReadDir(filepath.Join(sa.etcDir, "secrets"))
+	if err != nil && !os.IsNotExist(err) {
+		return err
+	}
+	for _, e := range secrets {
+		if e.IsDir() || strings.Contains(e.Name(), ".pre-") || strings.HasSuffix(e.Name(), ".new") {
+			continue
+		}
+		path := "/etc/llmgw/secrets/" + e.Name()
+		mine, err := os.ReadFile(filepath.Join(sa.etcDir, "secrets", e.Name()))
+		if err != nil {
+			return err
+		}
+		theirs, _ := sa.sh(ctx, "sudo cat "+path+" 2>/dev/null || true", nil)
+		switch {
+		case theirs == string(mine):
+			sa.kept("%s", path)
+		case theirs != "":
+			// Two control planes on two stores are two clusters.
+			return fmt.Errorf("%s there differs from this one's; remove it there if that machine is to join this cluster", path)
+		default:
+			if err := sa.put(ctx, path, "0600", "llmgw:llmgw", mine); err != nil {
+				return err
+			}
+			sa.done("%s", path)
+		}
+	}
+	return nil
+}
+
+// storeAccess lets it into the store, and records both control planes for
+// the nodes to be told.
+func (sa *serverAdd) storeAccess(ctx context.Context) error {
+	cfg, err := config.Load(filepath.Join(sa.etcDir, "server.yaml"))
+	if err != nil {
+		return err
+	}
+	addr, err := store.ResolveAddress(cfg.Auth.Store)
+	if err != nil {
+		return err
+	}
+	addr = strings.TrimSpace(addr)
+	if !store.IsPostgres(addr) {
+		return errors.New("the store is SQLite on this machine; a second control plane needs one both reach — run 'llmgw server init --patroni' first")
+	}
+	hosts, _, _, err := store.SplitPostgres(addr)
+	if err != nil {
+		return err
+	}
+	for _, h := range hosts {
+		if ip := net.ParseIP(h); ip != nil && ip.IsLoopback() || h == "localhost" {
+			return fmt.Errorf("the store is at %s, on this machine only; replicate it first with 'llmgw server init --patroni'", h)
+		}
+	}
+	switch {
+	case slices.Contains(hosts, sa.ip):
+		sa.kept("store: %s is one of its members", sa.ip)
+	case len(hosts) > 1:
+		changed, err := pgha.AllowClient(ctx, hosts, sa.ip)
+		if err != nil {
+			return err
+		}
+		if changed {
+			sa.done("store: %s let in, through Patroni", sa.ip)
+		} else {
+			sa.kept("store: %s already let in", sa.ip)
+		}
+	default:
+		sa.kept("store at %s: make sure it lets %s in", hosts[0], sa.ip)
+	}
+
+	st, err := store.Open(ctx, addr)
+	if err != nil {
+		return err
+	}
+	defer st.Close()
+	_, port, _ := net.SplitHostPort(cfg.AgentListen)
+	if port == "" {
+		port = "7443"
+	}
+	self, err := sourceIPToward(sa.ip)
+	if err != nil {
+		return err
+	}
+	me, _ := os.Hostname()
+	for _, cp := range []store.ControlPlane{
+		{URL: "https://" + net.JoinHostPort(self, port), Name: me},
+		{URL: "https://" + net.JoinHostPort(sa.ip, port), Name: sa.name, AddedAt: time.Now().Add(time.Second)},
+	} {
+		if err := st.PutControlPlane(ctx, cp); err != nil {
+			return err
+		}
+	}
+	sa.done("nodes will be told of both control planes within half a minute")
+	return nil
+}
+
+// service installs the unit and starts it, then waits for it to answer.
+func (sa *serverAdd) service(ctx context.Context) error {
+	if err := sa.put(ctx, "/etc/systemd/system/llmgw-server.service", "0644", "root:root", []byte(serverUnit)); err != nil {
+		return err
+	}
+	if _, err := sa.sh(ctx, "sudo systemctl daemon-reload && sudo systemctl enable llmgw-server >/dev/null 2>&1 && sudo systemctl restart llmgw-server", nil); err != nil {
+		return err
+	}
+	deadline := time.Now().Add(90 * time.Second)
+	for {
+		code, _ := sa.sh(ctx, "curl -s -o /dev/null -m 2 -w '%{http_code}' http://127.0.0.1:8080/leader || true", nil)
+		switch strings.TrimSpace(code) {
+		case "200":
+			sa.done("llmgw-server running, and leading")
+			return nil
+		case "503":
+			sa.done("llmgw-server running, standing by")
+			return nil
+		}
+		if time.Now().After(deadline) {
+			logs, _ := sa.sh(ctx, "sudo journalctl -u llmgw-server -n 15 --no-pager", nil)
+			return fmt.Errorf("it did not answer within 90s:\n%s", logs)
+		}
+		time.Sleep(2 * time.Second)
+	}
+}
+
+// floating keeps an address on whichever of the two leads, with keepalived
+// on both, following /leader.
+func (sa *serverAdd) floating(ctx context.Context) error {
+	vip, nw, err := net.ParseCIDR(sa.vip)
+	if err != nil || vip.To4() == nil {
+		return fmt.Errorf("--vip %q is not an IPv4 address with its prefix, like 10.0.0.160/24", sa.vip)
+	}
+	prefix, _ := nw.Mask.Size()
+	pass, err := vrrpPassword(sa.etcDir)
+	if err != nil {
+		return err
+	}
+	here := pgha.Local{}
+	for _, m := range []struct {
+		r        onboard.Remote
+		name     string
+		priority int
+	}{{here, "this machine", 110}, {sa.remote, sa.name, 100}} {
+		out, err := m.r.Run(ctx, fmt.Sprintf(`set -e
+command -v keepalived >/dev/null 2>&1 || { sudo DEBIAN_FRONTEND=noninteractive apt-get -qq update >/dev/null; sudo DEBIAN_FRONTEND=noninteractive apt-get -qq install -y keepalived >/dev/null; }
+ip -o route get %s | awk '{for(i=1;i<=NF;i++) if($i=="dev") {print $(i+1); exit}}'`, vip), nil)
+		if err != nil {
+			return fmt.Errorf("on %s: %v: %s", m.name, err, strings.TrimSpace(string(out)))
+		}
+		iface := strings.TrimSpace(string(out))
+		conf := keepalivedConf(iface, vip.String(), prefix, m.priority, pass)
+		if out, err := m.r.Run(ctx, `set -e
+sudo install -d -m 0755 /etc/keepalived
+sudo sh -c 'umask 077; cat > /etc/keepalived/keepalived.conf'
+sudo systemctl enable keepalived >/dev/null 2>&1
+sudo systemctl restart keepalived`, []byte(conf)); err != nil {
+			return fmt.Errorf("on %s: %v: %s", m.name, err, strings.TrimSpace(string(out)))
+		}
+	}
+	sa.done("%s follows whichever control plane leads (keepalived on both)", sa.vip)
+	return nil
+}
+
+// vrrpPassword is the shared secret of the two keepalived, made once and
+// kept with the other secrets, so running this again does not split them.
+func vrrpPassword(etcDir string) (string, error) {
+	p := filepath.Join(etcDir, "secrets", "vrrp")
+	if b, err := os.ReadFile(p); err == nil && len(bytes.TrimSpace(b)) > 0 {
+		return string(bytes.TrimSpace(b)), nil
+	}
+	b := make([]byte, 4)
+	if _, err := rand.Read(b); err != nil {
+		return "", err
+	}
+	pass := hex.EncodeToString(b) // keepalived reads eight characters
+	if err := os.WriteFile(p, []byte(pass+"\n"), 0o600); err != nil {
+		return "", err
+	}
+	return pass, nil
+}
+
+func keepalivedConf(iface, vip string, prefix, priority int, pass string) string {
+	return fmt.Sprintf(`# llmgw: the gateway's floating address, on whichever control plane leads.
+vrrp_script llmgw_leader {
+  script "/usr/bin/curl -fsS -o /dev/null -m 2 http://127.0.0.1:8080/leader"
+  interval 2
+  fall 2
+  rise 2
+}
+
+vrrp_instance llmgw {
+  state BACKUP
+  interface %s
+  virtual_router_id 51
+  priority %d
+  advert_int 1
+  authentication {
+    auth_type PASS
+    auth_pass %s
+  }
+  virtual_ipaddress {
+    %s/%d
+  }
+  track_script {
+    llmgw_leader
+  }
+}
+`, iface, priority, pass, vip, prefix)
+}

internal/cli/serverinit.go

@@ -23,6 +23,7 @@ import (
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/account"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/pgha"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
 )
@@ -69,6 +70,13 @@ type serverInit struct {
 	// postgres keeps the store in a PostgreSQL on this machine, installed if
 	// need be, moving whatever SQLite already holds.
 	postgres bool
+	// patroni are user@host of the machines to keep the store on under
+	// Patroni, replicated; this machine is the third etcd vote for two.
+	patroni []string
+	// patroniNodes are the same machines already connected to, which the
+	// setup screen does with the password it was given; when set, patroni
+	// only names them.
+	patroniNodes []pgha.Node
 	// noMirror, noNode and noModel turn off the parts of a first install that
 	// make it serve something: a mirror, this machine as a node, and a small
 	// model on it.
@@ -86,14 +94,19 @@ type serverInit struct {
 	changed  bool
 }
 
-func newServerInitCmd() *cobra.Command {
-	in := serverInit{
+// newDefaultServerInit is init as it sets up a real machine.
+func newDefaultServerInit() serverInit {
+	return serverInit{
 		etcDir:   "/etc/llmgw",
 		stateDir: "/var/lib/llmgw",
 		pkiDir:   defaultPKIDir,
 		unitPath: "/etc/systemd/system/llmgw-server.service",
 		user:     "llmgw",
 	}
+}
+
+func newServerInitCmd() *cobra.Command {
+	in := newDefaultServerInit()
 	var plain bool
 	cmd := &cobra.Command{
 		Use:   "init",
@@ -111,6 +124,12 @@ func newServerInitCmd() *cobra.Command {
 			"installed if there is none, with its own role and a generated password\n" +
 			"kept in /etc/llmgw/secrets. An existing SQLite store is copied into it,\n" +
 			"so accounts and keys survive, and server.yaml is pointed at it.\n\n" +
+			"With --patroni, given two or more machines, the store is PostgreSQL\n" +
+			"replicated across them under Patroni, which promotes a standby when the\n" +
+			"primary is lost. etcd holds its leader lock on those machines — and on\n" +
+			"this one, when they are two, as the third vote. Installed over ssh with\n" +
+			"passwordless sudo, as 'llmgw node add' does; the store already in use,\n" +
+			"SQLite or PostgreSQL, is moved in. Run again, it changes nothing.\n\n" +
 			"Each step is skipped when its result already exists, so it is safe to run\n" +
 			"again: to finish an install that stopped, or with another --host to\n" +
 			"reissue the certificate. It never overwrites server.yaml or the authority.\n\n" +
@@ -144,6 +163,9 @@ func newServerInitCmd() *cobra.Command {
 	f.BoolVar(&in.noModel, "no-model", false, "do not fetch and serve the starter model ("+starterName+")")
 	f.BoolVar(&in.postgres, "postgres", false,
 		"keep the store in PostgreSQL on this machine, installing it and moving an existing SQLite store into it")
+	f.StringArrayVar(&in.patroni, "patroni", nil,
+		"user@host of a machine to keep the store on, replicated under Patroni; give two or more. "+
+			"Installs PostgreSQL, Patroni and etcd there over ssh, and moves the existing store in")
 	return cmd
 }
 

internal/cli/serverinit_patroni.go

@@ -0,0 +1,182 @@
+package cli
+
+import (
+	"context"
+	"fmt"
+	"net"
+	"os"
+	"os/exec"
+	"strings"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/pgha"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// ensurePatroni puts the store in PostgreSQL under Patroni on the machines
+// given with --patroni, and moves whatever store server.yaml names into it:
+// SQLite, or a PostgreSQL on one machine. This machine is the third etcd vote
+// when they are two.
+func (in *serverInit) ensurePatroni(ctx context.Context) (err error) {
+	setup := pgha.Setup{Emit: func(step, detail string, changed bool) {
+		if changed {
+			in.done("%s: %s", step, detail)
+		} else {
+			in.kept("%s: %s", step, detail)
+		}
+	}}
+	setup.DB = in.patroniNodes
+	for _, t := range in.patroni {
+		if len(in.patroniNodes) > 0 {
+			break
+		}
+		user, host, ok := strings.Cut(t, "@")
+		if !ok {
+			user, host = "", t
+		}
+		remote := &onboard.SSH{Addr: host, User: user}
+		out, err := remote.Run(ctx, "hostname && sudo -n true", nil)
+		if err != nil {
+			return fmt.Errorf("%s: cannot log in and use sudo without a password: %v: %s",
+				t, err, strings.TrimSpace(string(out)))
+		}
+		ip, err := ipv4Of(host)
+		if err != nil {
+			return err
+		}
+		setup.DB = append(setup.DB, pgha.Node{
+			Member: pgha.Member{Name: strings.TrimSpace(strings.SplitN(string(out), "\n", 2)[0]), IP: ip},
+			Remote: remote,
+		})
+	}
+	self, err := sourceIPToward(setup.DB[0].IP)
+	if err != nil {
+		return err
+	}
+	name, _ := os.Hostname()
+	setup.Witness = &pgha.Node{Member: pgha.Member{Name: name, IP: self}, Remote: pgha.Local{}}
+	setup.Clients = []string{self}
+
+	dsn, err := setup.Run(ctx)
+	if err != nil {
+		return err
+	}
+
+	cfg, err := config.Load(in.configPath())
+	if err != nil {
+		return err
+	}
+	current := cfg.Auth.Store
+	if current == "" {
+		current = in.storeAddress()
+	}
+	resolved, _ := store.ResolveAddress(current)
+	if strings.TrimSpace(resolved) == dsn {
+		in.kept("store in the Patroni cluster (%s)", in.dsnPath())
+		return nil
+	}
+
+	// Still while it moves, and started again on the old store if the move
+	// fails: a gateway down over a database that did not work out is worse
+	// than one left where it was.
+	stopped, err := in.stopService(ctx)
+	if err != nil {
+		return err
+	}
+	defer func() {
+		if err != nil && stopped {
+			_ = exec.Command("systemctl", "start", "llmgw-server").Run()
+			fmt.Fprintf(in.out, "  ..    llmgw-server started again on its old store\n")
+		}
+	}()
+	if err := in.moveInto(ctx, current, dsn); err != nil {
+		return err
+	}
+
+	// The old address is kept beside the new one: the way back, if the
+	// cluster is not what was wanted.
+	if b, err := os.ReadFile(in.dsnPath()); err == nil && strings.TrimSpace(string(b)) != dsn {
+		if err := os.WriteFile(in.dsnPath()+".pre-patroni", b, 0o600); err != nil {
+			return err
+		}
+		_ = in.chown(in.dsnPath()+".pre-patroni", true)
+	}
+	if err := in.writeDSN(dsn); err != nil {
+		return err
+	}
+	if current != "@"+in.dsnPath() {
+		if err := setAuthStore(in.configPath(), "@"+in.dsnPath()); err != nil {
+			return err
+		}
+	}
+	// Started again by the service step that follows, on the new address.
+	in.done("server.yaml now keeps the store in the Patroni cluster")
+	return nil
+}
+
+// moveInto copies a store into the cluster, unless the cluster already holds
+// one — from a run that copied and stopped before changing the address.
+func (in *serverInit) moveInto(ctx context.Context, from, to string) error {
+	src, err := store.Open(ctx, from)
+	if err != nil {
+		// Nothing there yet: a first install, with nothing to move.
+		in.kept("no store at %s to move", from)
+		return nil
+	}
+	defer src.Close()
+	dst, err := store.Open(ctx, to)
+	if err != nil {
+		return fmt.Errorf("opening the cluster: %w", err)
+	}
+	defer dst.Close()
+	empty, err := dst.Empty(ctx)
+	if err != nil {
+		return err
+	}
+	if !empty {
+		in.kept("the cluster already holds a store; %s is left as it is", from)
+		return nil
+	}
+	res, err := src.CopyTo(ctx, dst)
+	if err != nil {
+		return fmt.Errorf("copying the store into the cluster: %w", err)
+	}
+	var rows int64
+	for _, r := range res {
+		rows += r.Rows
+	}
+	in.done("copied %d rows into the cluster (the old store is kept as it was)", rows)
+	return nil
+}
+
+// sourceIPToward is this machine's address on the route to another: the one
+// the other machine sees it connect from, which is what etcd advertises and
+// what the database lets in. Not the first --host, which names the gateway
+// for its clients and may not resolve yet, or resolve to a floating address.
+func sourceIPToward(ip string) (string, error) {
+	c, err := net.Dial("udp", net.JoinHostPort(ip, "5432"))
+	if err != nil {
+		return "", fmt.Errorf("no route from this machine to %s: %w", ip, err)
+	}
+	defer c.Close()
+	return c.LocalAddr().(*net.UDPAddr).IP.String(), nil
+}
+
+// ipv4Of is the address a host name or address stands for, as the other
+// machines will be told to reach it.
+func ipv4Of(host string) (string, error) {
+	if ip := net.ParseIP(host); ip != nil {
+		return ip.String(), nil
+	}
+	ips, err := net.LookupIP(host)
+	if err != nil {
+		return "", fmt.Errorf("%s does not resolve: %w", host, err)
+	}
+	for _, ip := range ips {
+		if v4 := ip.To4(); v4 != nil {
+			return v4.String(), nil
+		}
+	}
+	return "", fmt.Errorf("%s has no IPv4 address", host)
+}

internal/cli/serverinit_postgres.go

@@ -44,6 +44,12 @@ func (in *serverInit) storeAddress() string {
 // means reissuing every key and being unable to say which was which; and the
 // administrator init created a minute ago would be gone too.
 func (in *serverInit) ensureDatabase(ctx context.Context) (err error) {
+	if len(in.patroni) > 0 || len(in.patroniNodes) > 0 {
+		if in.postgres {
+			return fmt.Errorf("--postgres keeps the store on this machine and --patroni on others: give one")
+		}
+		return in.ensurePatroni(ctx)
+	}
 	if !in.postgres {
 		return nil
 	}

internal/cli/status.go

@@ -2,6 +2,7 @@ package cli
 
 import (
 	"fmt"
+	"strings"
 	"time"
 
 	"github.com/spf13/cobra"
@@ -94,6 +95,31 @@ func printSummary(cmd *cobra.Command, s adminapi.Summary) {
 	fmt.Fprintf(out, "\nMODELS    %d of %d have the replicas they asked for\n",
 		s.Models.Satisfied, s.Models.Total)
 
+	// The store, when it is a replicated cluster: one line, saying who leads
+	// and whether every other member keeps up.
+	switch {
+	case s.StoreError != "":
+		fmt.Fprintf(out, "STORE     cannot read its cluster: %s\n", s.StoreError)
+	case len(s.Store) > 0:
+		var parts []string
+		leader := false
+		for _, m := range s.Store {
+			switch {
+			case m.Role == "leader":
+				leader = true
+				parts = append(parts, m.Name+" leads")
+			case m.State == "streaming" && m.Lag >= 0:
+				parts = append(parts, fmt.Sprintf("%s streaming (%s behind)", m.Name, bytesShort(m.Lag)))
+			default:
+				parts = append(parts, fmt.Sprintf("%s %s", m.Name, m.State))
+			}
+		}
+		if !leader {
+			parts = append([]string{"NO LEADER — writes fail"}, parts...)
+		}
+		fmt.Fprintf(out, "STORE     %s\n", strings.Join(parts, ", "))
+	}
+
 	if len(s.Parked) > 0 {
 		fmt.Fprintln(out, "\nPARKED — failed every time they started; not retried until fixed")
 		for _, p := range s.Parked {
@@ -169,3 +195,16 @@ func unhealthyCell(n int) string {
 	}
 	return fmt.Sprintf("%d", n)
 }
+
+// bytesShort is a byte count as a person reads it: 0 B, 12 KB, 3.4 MB.
+func bytesShort(n int64) string {
+	switch {
+	case n < 1024:
+		return fmt.Sprintf("%d B", n)
+	case n < 1<<20:
+		return fmt.Sprintf("%d KB", n>>10)
+	case n < 1<<30:
+		return fmt.Sprintf("%.1f MB", float64(n)/(1<<20))
+	}
+	return fmt.Sprintf("%.1f GB", float64(n)/(1<<30))
+}

internal/cli/tui_frame.go

@@ -0,0 +1,237 @@
+package cli
+
+import (
+	"strings"
+
+	"github.com/charmbracelet/lipgloss"
+	"github.com/charmbracelet/x/ansi"
+)
+
+// The frame every full-screen llmgw screen is drawn in: a title bar, a
+// sidebar of steps or actions on the left, the content in a bordered panel on
+// the right, the keys along the bottom — and a dialog over it all when one is
+// open. One frame, so the setup and the console read as one program.
+//
+// Keyboard only: these run on serial consoles and IPMI as often as in a
+// desktop terminal.
+
+// navItem is one line of the sidebar.
+type navItem struct {
+	Label string
+	// State is "done", "current", "todo" for steps; "" for menu entries,
+	// where Current marks the one focused.
+	State   string
+	Current bool
+	// Dim greys an entry that cannot be chosen now.
+	Dim bool
+	// Heading is a title between groups of entries, not an entry.
+	Heading bool
+}
+
+type frame struct {
+	Width, Height int
+	Title         string // left of the title bar
+	Where         string // right of the title bar: the machine
+	NavTitle      string
+	Nav           []navItem
+	PanelTitle    string
+	Body          string
+	// Scroll is how many of the body's lines are scrolled off the top.
+	Scroll int
+	Keys   string
+	// Dialog, when set, is drawn over the middle of the screen.
+	Dialog      string
+	DialogTitle string
+}
+
+var (
+	fBar      = lipgloss.NewStyle().Bold(true).Foreground(lipgloss.AdaptiveColor{Light: "#FFFFFF", Dark: "#0B1B1B"}).Background(tuiAccent)
+	fBorder   = lipgloss.NewStyle().Border(lipgloss.RoundedBorder()).BorderForeground(tuiDim)
+	fFocus    = lipgloss.NewStyle().Border(lipgloss.RoundedBorder()).BorderForeground(tuiAccent)
+	fKeys     = lipgloss.NewStyle().Foreground(tuiDim)
+	fSelected = lipgloss.NewStyle().Bold(true).Foreground(tuiAccent)
+	fDialog   = lipgloss.NewStyle().Border(lipgloss.DoubleBorder()).BorderForeground(tuiAccent).Padding(1, 2)
+)
+
+const navWidth = 24
+
+// render draws the whole screen, exactly Width by Height.
+func (f frame) render() string {
+	w, h := max(f.Width, 60), max(f.Height, 16)
+
+	title := " " + f.Title
+	right := f.Where + " "
+	gap := max(1, w-ansi.StringWidth(title)-ansi.StringWidth(right))
+	bar := fBar.Render(title + strings.Repeat(" ", gap) + right)
+
+	// Panels fill what the bars leave: one line each above and below.
+	ph := h - 2
+	inner := ph - 2 // borders
+
+	var nav strings.Builder
+	for _, it := range f.Nav {
+		mark := "  "
+		label := it.Label
+		if it.Heading {
+			nav.WriteString(sDim.Render(label) + "\n")
+			continue
+		}
+		switch {
+		case it.State == "done":
+			mark = sOK.Render("✓ ")
+		case it.State == "current":
+			mark = fSelected.Render("› ")
+			label = fSelected.Render(label)
+		case it.Current:
+			mark = fSelected.Render("› ")
+			label = fSelected.Render(label)
+		case it.Dim || it.State == "todo":
+			label = sDim.Render(label)
+		}
+		nav.WriteString(truncateWidth(mark+label, navWidth-2) + "\n")
+	}
+	left := box(fBorder, f.NavTitle, nav.String(), navWidth, inner)
+
+	pw := w - navWidth - 2 - 2
+	lines := strings.Split(strings.TrimRight(f.Body, "\n"), "\n")
+	scroll := min(max(f.Scroll, 0), max(0, len(lines)-inner))
+	visible := lines[scroll:]
+	if len(visible) > inner {
+		visible = visible[:inner]
+		// Said in the last line, where it is seen.
+		visible[inner-1] = sDim.Render("  ↓ more — PgDn")
+	}
+	for i := range visible {
+		visible[i] = truncateWidth(visible[i], pw)
+	}
+	panelTitle := f.PanelTitle
+	if scroll > 0 {
+		panelTitle += " ↑"
+	}
+	main := box(fFocus, panelTitle, strings.Join(visible, "\n"), pw, inner)
+
+	keys := fKeys.Render(" " + truncateWidth(f.Keys, w-2))
+	screen := bar + "\n" + lipgloss.JoinHorizontal(lipgloss.Top, left, main) + "\n" + keys
+
+	if f.Dialog != "" {
+		dw := min(w-8, max(50, maxLineWidth(f.Dialog)+6))
+		d := fDialog.Width(dw).Render(f.Dialog)
+		if f.DialogTitle != "" {
+			d = titled(d, f.DialogTitle)
+		}
+		screen = overlay(screen, d, w, h)
+	}
+	return screen
+}
+
+// box draws content in a border of the given inner size, with a title set
+// into the top edge.
+func box(style lipgloss.Style, title, content string, width, height int) string {
+	b := style.Width(width).Height(height).Render(content)
+	if title == "" {
+		return b
+	}
+	return titled(b, title)
+}
+
+// titled sets " title " into a box's top border, after its corner.
+func titled(b, title string) string {
+	lines := strings.Split(b, "\n")
+	if len(lines) == 0 {
+		return b
+	}
+	top := lines[0]
+	t := " " + title + " "
+	tw := ansi.StringWidth(t)
+	if ansi.StringWidth(top) < tw+4 {
+		return b
+	}
+	lines[0] = ansi.Truncate(top, 2, "") + sTitle.Render(t) + ansi.TruncateLeft(top, 2+tw, "")
+	return strings.Join(lines, "\n")
+}
+
+// overlay draws top over the middle of base, both w by h.
+func overlay(base, top string, w, h int) string {
+	bl := strings.Split(base, "\n")
+	tl := strings.Split(top, "\n")
+	tw := maxLineWidth(top)
+	x := max(0, (w-tw)/2)
+	y := max(0, (h-len(tl))/2)
+	for i, l := range tl {
+		row := y + i
+		if row >= len(bl) {
+			break
+		}
+		under := bl[row]
+		left := ansi.Truncate(under, x, "")
+		if pad := x - ansi.StringWidth(left); pad > 0 {
+			left += strings.Repeat(" ", pad)
+		}
+		right := ansi.TruncateLeft(under, x+ansi.StringWidth(l), "")
+		bl[row] = left + l + right
+	}
+	return strings.Join(bl, "\n")
+}
+
+func maxLineWidth(s string) int {
+	n := 0
+	for _, l := range strings.Split(s, "\n") {
+		n = max(n, ansi.StringWidth(l))
+	}
+	return n
+}
+
+// truncateWidth cuts a line, colours and all, to w cells.
+func truncateWidth(s string, w int) string {
+	if ansi.StringWidth(s) <= w {
+		return s
+	}
+	return ansi.Truncate(s, max(w-1, 0), "…")
+}
+
+// table draws rows under a header in columns as wide as they need, cut to
+// width; the selected row is shown reversed when the panel has the focus.
+func table(headers []string, rows [][]string, selected int, width int, focused bool) string {
+	widths := make([]int, len(headers))
+	for i, h := range headers {
+		widths[i] = ansi.StringWidth(h)
+	}
+	for _, r := range rows {
+		for i := range headers {
+			if i < len(r) {
+				widths[i] = min(max(widths[i], ansi.StringWidth(r[i])), 44)
+			}
+		}
+	}
+	line := func(cells []string) string {
+		var b strings.Builder
+		for i := range headers {
+			c := ""
+			if i < len(cells) {
+				c = truncateWidth(cells[i], widths[i])
+			}
+			b.WriteString(c)
+			if i < len(headers)-1 {
+				b.WriteString(strings.Repeat(" ", widths[i]-ansi.StringWidth(c)+2))
+			}
+		}
+		return truncateWidth(b.String(), width)
+	}
+	var b strings.Builder
+	b.WriteString("  " + lipgloss.NewStyle().Bold(true).Foreground(tuiDim).Render(line(headers)) + "\n")
+	for i, r := range rows {
+		l := line(r)
+		switch {
+		case i == selected && focused:
+			pad := max(0, width-ansi.StringWidth(ansi.Strip(l)))
+			l = fBar.Render(ansi.Strip(l) + strings.Repeat(" ", pad))
+		case i == selected:
+			l = lipgloss.NewStyle().Bold(true).Render(l)
+		}
+		b.WriteString("  " + l + "\n")
+	}
+	if len(rows) == 0 {
+		b.WriteString("  " + sDim.Render("nothing yet") + "\n")
+	}
+	return b.String()
+}

internal/cli/tui_home.go

@@ -0,0 +1,48 @@
+package cli
+
+import (
+	"errors"
+	"fmt"
+	"os"
+	"runtime"
+
+	"github.com/mattn/go-isatty"
+	"github.com/spf13/cobra"
+)
+
+// A bare `llmgw` in a terminal: the setup screens on a machine with nothing
+// installed, the console on a control plane. Anywhere else — a pipe, a
+// script, another operating system — it prints its help, as it always did.
+
+func interactive(cmd *cobra.Command) bool {
+	return isTerminal(cmd.OutOrStdout()) && isatty.IsTerminal(os.Stdin.Fd())
+}
+
+func runHome(cmd *cobra.Command) error {
+	out := cmd.OutOrStdout()
+	if runtime.GOOS != "linux" {
+		fmt.Fprintln(out, "The setup and console screens run on the Linux machine they manage.")
+		fmt.Fprintln(out)
+		return cmd.Help()
+	}
+	if os.Geteuid() != 0 {
+		return errors.New("run it as root — sudo llmgw — since it installs software and holds the certificate authority")
+	}
+	switch {
+	// Installed is the service in place, not only its configuration: a setup
+	// that stopped halfway has written server.yaml and nothing that runs, and
+	// the setup screens are what finish it.
+	case exists("/etc/llmgw/server.yaml") && exists("/etc/systemd/system/llmgw-server.service"):
+		return runConsole(cmd.Context(), out, "/etc/llmgw/server.yaml")
+	case exists("/etc/llmgw/agent.yaml"):
+		// A node: its own settings. The cluster is watched from the
+		// control plane.
+		return runConsole(cmd.Context(), out, "/etc/llmgw/agent.yaml")
+	}
+	return runSetupTUI(cmd.Context(), out)
+}
+
+func exists(p string) bool {
+	_, err := os.Stat(p)
+	return err == nil
+}

internal/cli/tui_init.go

@@ -77,10 +77,14 @@ type initModel struct {
 	next     []string // the closing "Next:" text
 	spin     spinnerModel
 	width    int
-	done     bool
-	err      error
-	quit     bool
-	start    func()
+	height   int
+	// framed draws it in the full-screen frame; off when another screen
+	// shows it in its own panel.
+	framed bool
+	done   bool
+	err    error
+	quit   bool
+	start  func()
 }
 
 func newInitModel(in *serverInit, found []string, askForm bool) initModel {
@@ -118,7 +122,10 @@ type startInitMsg struct{}
 func (m initModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
 	switch msg := msg.(type) {
 	case tea.WindowSizeMsg:
-		m.width = msg.Width
+		m.width, m.height = msg.Width, msg.Height
+		if m.framed {
+			m.width = msg.Width - navWidth - 6
+		}
 	case spinTickMsg:
 		var cmd tea.Cmd
 		m.spin, cmd = m.spin.Update(msg)
@@ -132,12 +139,18 @@ func (m initModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
 		m.addLine(string(msg))
 	case initDoneMsg:
 		m.done, m.err = true, msg.err
+		if m.framed {
+			return m, nil // left on screen until a key: it is gone once closed
+		}
 		return m, tea.Quit
 	case tea.KeyMsg:
 		if msg.Type == tea.KeyCtrlC {
 			m.quit = true
 			return m, tea.Quit
 		}
+		if m.done && m.framed && (msg.Type == tea.KeyEnter || msg.String() == "q") {
+			return m, tea.Quit
+		}
 		if m.form {
 			return m.formKey(msg)
 		}
@@ -240,6 +253,30 @@ func (m *initModel) moveFocus(d int) {
 }
 
 func (m initModel) View() string {
+	if !m.framed {
+		return m.plainView()
+	}
+	body := m.plainView()
+	if i := strings.Index(body, "\n\n"); i >= 0 {
+		body = body[i+1:]
+	}
+	steps := []navItem{{Label: "Start", State: "done"}, {Label: "Settings", State: "current"}, {Label: "Install", State: "todo"}}
+	keys := "Tab/↑↓ move · Space tick · Enter on Set up · Ctrl+C leave"
+	if !m.form {
+		steps[1].State, steps[2].State = "done", "current"
+		keys = "Ctrl+C stop"
+		if m.done {
+			steps[2].State = "done"
+			keys = "Enter close"
+		}
+	}
+	host, _ := os.Hostname()
+	return frame{Width: m.width + navWidth + 6, Height: m.height, Title: "llmgw setup", Where: host,
+		NavTitle: "Steps", Nav: steps, PanelTitle: "This machine as a control plane", Body: body, Keys: keys}.render()
+}
+
+// plainView is the screen as text, for a panel or a terminal that is not one.
+func (m initModel) plainView() string {
 	var b strings.Builder
 	b.WriteString(sTitle.Render("llmgw · setting up this machine as a control plane") + "\n\n")
 	if m.form {
@@ -338,7 +375,9 @@ func (m initModel) formView() string {
 	if m.formError != "" {
 		b.WriteString("\n" + sBad.Render(m.formError) + "\n")
 	}
-	b.WriteString("\n" + sDim.Render("tab or ↑↓ to move · space to tick · enter on Set up to start · ctrl+c to leave") + "\n")
+	if !m.framed { // the frame's key bar says it
+		b.WriteString("\n" + sDim.Render("tab or ↑↓ to move · space to tick · enter on Set up to start · ctrl+c to leave") + "\n")
+	}
 	return b.String()
 }
 
@@ -350,6 +389,7 @@ func (in *serverInit) runInitTUI(ctx context.Context, out io.Writer) error {
 		found = append(found, ip.String())
 	}
 	m := newInitModel(in, found, in.admin == "")
+	m.framed = true
 	var p *tea.Program
 
 	var (
@@ -373,7 +413,7 @@ func (in *serverInit) runInitTUI(ctx context.Context, out io.Writer) error {
 			p.Send(initDoneMsg{err})
 		}()
 	}
-	p = tea.NewProgram(m, tea.WithOutput(out))
+	p = tea.NewProgram(m, tea.WithOutput(out), tea.WithAltScreen())
 	final, err := p.Run()
 	if err != nil {
 		return err

internal/cli/tui_node.go

@@ -68,6 +68,7 @@ type nodeModel struct {
 	spin    spinnerModel
 	cancel  context.CancelFunc
 	width   int
+	height  int
 	done    bool
 	err     error
 	node    string
@@ -91,7 +92,7 @@ func (m nodeModel) Init() tea.Cmd { return m.spin.Tick }
 func (m nodeModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
 	switch msg := msg.(type) {
 	case tea.WindowSizeMsg:
-		m.width = msg.Width
+		m.width, m.height = msg.Width, msg.Height
 	case spinTickMsg:
 		var cmd tea.Cmd
 		m.spin, cmd = m.spin.Update(msg)
@@ -131,9 +132,13 @@ func (m nodeModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
 	case askContinueMsg:
 		m.confirm = &msg
 	case finishMsg:
+		// Left on the screen until a key: the frame is gone once it closes.
 		m.done, m.err, m.node, m.summary = true, msg.err, msg.node, msg.summary
-		return m, tea.Quit
+		return m, nil
 	case tea.KeyMsg:
+		if m.done && (msg.Type == tea.KeyEnter || msg.String() == "q" || msg.Type == tea.KeyEsc) {
+			return m, tea.Quit
+		}
 		return m.key(msg)
 	}
 	return m, nil
@@ -197,10 +202,52 @@ func (m *nodeModel) move(d int) {
 	m.inputs[m.focus].Focus()
 }
 
+// View draws the onboarding in the frame: its steps down the side, what each
+// did in the panel, a question as a dialog.
 func (m nodeModel) View() string {
+	var nav []navItem
+	current := false
+	for _, step := range onboard.Steps {
+		st, ok := m.steps[step]
+		state := "todo"
+		switch {
+		case ok && st.level == onboard.Running && !m.done && !current:
+			state, current = "current", true
+		case ok && st.level != onboard.Running:
+			state = "done"
+		}
+		nav = append(nav, navItem{Label: step, State: state})
+	}
+	keys := "Ctrl+C stop"
+	if m.done {
+		keys = "Enter close"
+	}
+	host, _ := os.Hostname()
+	f := frame{
+		Width: m.width, Height: m.height, Title: "llmgw · add a node", Where: host,
+		NavTitle: "Steps", Nav: nav, PanelTitle: m.title, Body: "\n" + m.content(), Keys: keys,
+	}
+	switch {
+	case m.fabrics != nil:
+		var q strings.Builder
+		q.WriteString("These adapters are up. Which fabric is each group on?\n")
+		q.WriteString(sDim.Render("Give each the name it has on every other node attached to it.\nLeave a name empty to leave that group out.") + "\n\n")
+		for i, fb := range m.fabrics.suggested {
+			fmt.Fprintf(&q, "  %s  %s\n", m.inputs[i].View(), sDim.Render(strings.Join(fb.HCAs, ", ")))
+		}
+		q.WriteString("\n" + sDim.Render("Tab next · Enter use these · Esc none"))
+		f.DialogTitle, f.Dialog = "Fabrics", q.String()
+	case m.confirm != nil:
+		f.DialogTitle = "Problems found"
+		f.Dialog = fmt.Sprintf("doctor found %d problem(s) on the machine.\nAdd the node anyway?\n\n%s", m.confirm.fails, sDim.Render("Y yes · N no"))
+	}
+	return f.render()
+}
+
+// content is the onboarding's steps and notes, as the panel shows them.
+func (m nodeModel) content() string {
 	var b strings.Builder
-	b.WriteString(sTitle.Render(m.title) + "\n\n")
-	detailWidth := max(20, m.width-22)
+	detailWidth := max(20, m.width-navWidth-26)
 	for _, step := range onboard.Steps {
 		st, ok := m.steps[step]
 		var mark string
@@ -238,26 +285,12 @@ func (m nodeModel) View() string {
 	}
 
 	switch {
-	case m.fabrics != nil:
-		var q strings.Builder
-		q.WriteString("These adapters are up. Which fabric is each group on?\n")
-		q.WriteString(sDim.Render("Give each the name it has on every other node attached to it.\nLeave a name empty to leave that group out.") + "\n\n")
-		for i, f := range m.fabrics.suggested {
-			fmt.Fprintf(&q, "  %s  %s\n", m.inputs[i].View(), sDim.Render(strings.Join(f.HCAs, ", ")))
-		}
-		q.WriteString("\n" + sDim.Render("tab next · enter use these · esc none"))
-		b.WriteString("\n" + sBox.Render(q.String()) + "\n")
-	case m.confirm != nil:
-		q := fmt.Sprintf("doctor found %d problem(s) above.\nAdd the node anyway? %s", m.confirm.fails, sDim.Render("y / N"))
-		b.WriteString("\n" + sBox.BorderForeground(tuiWarn).Render(q) + "\n")
 	case m.done && m.err != nil:
 		b.WriteString("\n" + sBad.Render("✗ ") + wrap(m.err.Error(), m.width-4, "  ") + "\n")
 	case m.done:
 		b.WriteString("\n" + sOK.Render("✓ ") + m.summary + "\n")
 	case m.quitAsked:
 		b.WriteString("\n" + sDim.Render("stopped") + "\n")
-	default:
-		b.WriteString("\n" + sDim.Render("ctrl+c to stop") + "\n")
 	}
 	return b.String()
 }
@@ -298,7 +331,7 @@ func (na *nodeAdd) runTUI(ctx context.Context, o onboard.Options, title string)
 	ctx, cancel := context.WithCancel(ctx)
 	defer cancel()
 	m := newNodeModel(title, cancel)
-	p := tea.NewProgram(m)
+	p := tea.NewProgram(m, tea.WithAltScreen())
 
 	emit := func(e onboard.Event) { p.Send(evMsg(e)) }
 	o.AskFabrics = na.rememberFabrics(func(s []onboard.Fabric) []onboard.Fabric {
@@ -343,6 +376,8 @@ func (na *nodeAdd) runTUI(ctx context.Context, o onboard.Options, title string)
 		return "", err
 	}
 	fm := final.(nodeModel)
+	// The screen is gone; what it ended with is said again where it stays.
+	fmt.Println(fm.content())
 	if fm.quitAsked && !fm.done {
 		cancel()
 		return fm.node, fmt.Errorf("stopped; the node may be half set up, and running this again finishes it")

internal/cli/tui_setup.go

@@ -0,0 +1,949 @@
+package cli
+
+import (
+	"context"
+	"fmt"
+	"io"
+	"net/mail"
+	"os"
+	"strings"
+	"sync"
+	"time"
+
+	tea "github.com/charmbracelet/bubbletea"
+	"github.com/charmbracelet/lipgloss"
+
+	joinpkg "gitea.mynodes.xyz/ilker/llmgw/internal/join"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/pgha"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/preflight"
+)
+
+// The setup screens: what a bare `llmgw` shows on a machine with nothing
+// installed. One machine, or three for high availability — this one as the
+// control plane and etcd's third vote, two more for the store under Patroni —
+// with every machine checked before anything is installed on any of them.
+//
+// The other machines are reached with Go's own ssh rather than the system's,
+// so a password can be typed here when there is no key; it is held in memory
+// for the run and written nowhere. A sudo password, likewise, is used once to
+// let the user sudo for the length of the install, and that is taken back
+// when it finishes.
+
+type setupPage int
+
+const (
+	pWelcome setupPage = iota
+	pAdmin
+	pMachines
+	// pWait shows the codes for machines joining rather than logged into,
+	// until they have all dialled in.
+	pWait
+	pCheck
+	pPlan
+	pRun
+)
+
+// How a database machine is logged into.
+const (
+	authAgent = iota
+	authKey
+	authPassword
+	// authJoin is no login at all: the machine runs `llmgw join` with a code
+	// shown here and dials out to this one.
+	authJoin
+	authCount
+)
+
+var authNames = [authCount]string{"ssh agent", "key file", "password", "join code"}
+
+// Fields of one database machine, in order.
+const (
+	mfAddr = iota
+	mfUser
+	mfAuth
+	mfSecret
+	mfSudo
+	mfCount
+)
+
+type dbMachine struct {
+	addr, user, secret, sudo textField
+	auth                     int
+	// For a machine joining: its code, and it once it has dialled in.
+	code   string
+	joined *joinpkg.Remote
+	arrive <-chan *joinpkg.Remote
+}
+
+// visible reports whether a field is shown, and so can be moved to: a joining
+// machine is asked nothing but how it joins.
+func (d dbMachine) visible(f int) bool {
+	switch {
+	case d.auth == authJoin:
+		return f == mfAuth
+	case f == mfSecret:
+		return d.auth != authAgent
+	}
+	return true
+}
+
+// dbConn is a database machine as it is reached, whichever way.
+type dbConn struct {
+	remote onboard.Remote
+	addr   string
+	ssh    *onboard.GoSSH
+	joined *joinpkg.Remote
+}
+
+func newDBMachine(n int) dbMachine {
+	f := func(ph string, mask bool) textField {
+		t := newTextField()
+		t.Placeholder, t.Mask, t.CharLimit, t.Width = ph, mask, 200, 34
+		return t
+	}
+	m := dbMachine{
+		addr:   f(fmt.Sprintf("10.0.0.%d or db%d.internal", 10+n, n), false),
+		user:   f("ubuntu", false),
+		secret: f("", true),
+		sudo:   f("only if sudo asks for one", true),
+	}
+	if os.Getenv("SSH_AUTH_SOCK") == "" {
+		m.auth = authKey
+	}
+	return m
+}
+
+// field is the text field at a machine's position, nil for the selector.
+func (d *dbMachine) field(i int) *textField {
+	switch i {
+	case mfAddr:
+		return &d.addr
+	case mfUser:
+		return &d.user
+	case mfSecret:
+		return &d.secret
+	case mfSudo:
+		return &d.sudo
+	}
+	return nil
+}
+
+// Messages from the work behind the screens.
+type (
+	checkDoneMsg struct{ rep preflight.Report }
+	joinedMsg    struct {
+		i int
+		r *joinpkg.Remote
+	}
+)
+
+// progRef lets work started from Update send to the program it runs in.
+type progRef struct{ p *tea.Program }
+
+type setupModel struct {
+	ref *progRef
+
+	page   setupPage
+	host   string
+	ips    []string
+	choice int // 0 one machine, 1 high availability
+
+	admin, names textField
+	afocus       int // admin, names, next
+
+	db     [2]dbMachine
+	mfocus int // over both machines' fields, then the check button
+
+	conns    []dbConn
+	listener *joinpkg.Listener
+	report   preflight.Report
+	checking bool
+
+	in       *serverInit
+	run      initModel
+	finished bool
+	log      *[]string
+	logMu    *sync.Mutex
+
+	spin    spinnerModel
+	formErr string
+	width   int
+	height  int
+	// scroll is how far the panel is scrolled, reset with each page.
+	scroll int
+	quit   bool
+	// single is the choice of one machine: the screen closes and init's own
+	// form takes over.
+	single bool
+}
+
+func newSetupModel(host string, ips []string) setupModel {
+	admin := newTextField()
+	admin.Placeholder, admin.CharLimit, admin.Width = "you@example.com", 200, 40
+	names := newTextField()
+	names.Placeholder, names.CharLimit, names.Width = "llm.example.com, 203.0.113.10", 400, 40
+	return setupModel{
+		ref: &progRef{}, host: host, ips: ips, admin: admin, names: names,
+		db: [2]dbMachine{newDBMachine(1), newDBMachine(2)}, spin: newSpinner(), width: 100, height: 30,
+		log: &[]string{}, logMu: &sync.Mutex{},
+	}
+}
+
+func (m setupModel) Init() tea.Cmd { return m.spin.Tick }
+
+func (m setupModel) thisAddr() string {
+	if len(m.ips) > 0 {
+		return m.ips[0]
+	}
+	return m.host
+}
+
+func (m setupModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
+	switch msg := msg.(type) {
+	case tea.WindowSizeMsg:
+		m.width, m.height = msg.Width, msg.Height
+		m.run.width = msg.Width - navWidth - 6
+	case spinTickMsg:
+		var cmd tea.Cmd
+		m.spin, cmd = m.spin.Update(msg)
+		m.run.spin = m.spin
+		return m, cmd
+	case checkDoneMsg:
+		m.checking, m.report = false, msg.rep
+	case joinedMsg:
+		m.db[msg.i].joined = msg.r
+		if m.page == pWait && m.allJoined() {
+			m.page = pCheck
+			cmd := m.startCheck()
+			return m, cmd
+		}
+	case initLineMsg:
+		m.run.addLine(string(msg))
+	case initDoneMsg:
+		m.run.done, m.run.err, m.finished = true, msg.err, true
+	case tea.KeyMsg:
+		if msg.Type == tea.KeyCtrlC {
+			m.quit = true
+			return m, tea.Quit
+		}
+		before := m.page
+		next, cmd := m.key(msg)
+		if nm := next.(setupModel); nm.page != before {
+			// A new page starts at its top.
+			nm.scroll = 0
+			return nm, cmd
+		}
+		return next, cmd
+	}
+	return m, nil
+}
+
+func (m setupModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) {
+	// A dialog open takes the next key, whatever it is, to close it.
+	if m.formErr != "" {
+		m.formErr = ""
+		return m, nil
+	}
+	switch k.Type {
+	case tea.KeyPgDown:
+		m.scroll += max(1, m.height-8)
+		return m, nil
+	case tea.KeyPgUp:
+		m.scroll = max(0, m.scroll-max(1, m.height-8))
+		return m, nil
+	}
+	if k.Type == tea.KeyEsc && m.page > pWelcome && m.page < pRun && !m.checking {
+		m.page--
+		if m.page == pWait {
+			m.page = pMachines
+		}
+		m.formErr = ""
+		return m, nil
+	}
+	switch m.page {
+	case pWelcome:
+		switch k.String() {
+		case "up", "down", "tab", "k", "j":
+			m.choice = 1 - m.choice
+		case "enter":
+			if m.choice == 0 {
+				m.single = true
+				return m, tea.Quit
+			}
+			m.page = pAdmin
+			m.afocus = 0
+			m.admin.Focus()
+		}
+	case pAdmin:
+		return m.adminKey(k)
+	case pMachines:
+		return m.machinesKey(k)
+	case pCheck:
+		switch k.String() {
+		case "r":
+			if !m.checking {
+				cmd := m.startCheck()
+				return m, cmd
+			}
+		case "enter":
+			if !m.checking && m.report.Worst() < preflight.Fail {
+				m.page = pPlan
+			}
+		}
+	case pPlan:
+		if k.String() == "enter" {
+			m.page = pRun
+			cmd := m.startRun()
+			return m, cmd
+		}
+	case pRun:
+		if m.finished && (k.String() == "q" || k.String() == "enter") {
+			return m, tea.Quit
+		}
+	}
+	return m, nil
+}
+
+func (m setupModel) adminKey(k tea.KeyMsg) (tea.Model, tea.Cmd) {
+	move := func(d int) {
+		m.afocus = (m.afocus + d + 3) % 3
+		m.admin.Blur()
+		m.names.Blur()
+		switch m.afocus {
+		case 0:
+			m.admin.Focus()
+		case 1:
+			m.names.Focus()
+		}
+	}
+	switch k.Type {
+	case tea.KeyTab, tea.KeyDown:
+		move(1)
+		return m, nil
+	case tea.KeyShiftTab, tea.KeyUp:
+		move(-1)
+		return m, nil
+	case tea.KeyEnter:
+		if m.afocus < 2 {
+			move(1)
+			return m, nil
+		}
+		if _, err := mail.ParseAddress(strings.TrimSpace(m.admin.Value())); err != nil {
+			m.formErr = "the administrator's email is needed, and that is not one"
+			m.afocus = 0
+			move(0)
+			return m, nil
+		}
+		m.formErr = ""
+		m.page, m.mfocus = pMachines, 0
+		m.db[0].addr.Focus()
+		return m, nil
+	}
+	var cmd tea.Cmd
+	switch m.afocus {
+	case 0:
+		m.admin, cmd = m.admin.Update(k)
+	case 1:
+		m.names, cmd = m.names.Update(k)
+	}
+	return m, cmd
+}
+
+func (m *setupModel) focusMachine() {
+	for i := range m.db {
+		for f := 0; f < mfCount; f++ {
+			if t := m.db[i].field(f); t != nil {
+				t.Blur()
+			}
+		}
+	}
+	if m.mfocus < 2*mfCount {
+		if t := m.db[m.mfocus/mfCount].field(m.mfocus % mfCount); t != nil {
+			t.Focus()
+		}
+	}
+}
+
+func (m setupModel) machinesKey(k tea.KeyMsg) (tea.Model, tea.Cmd) {
+	n := 2*mfCount + 1
+	switch k.Type {
+	case tea.KeyTab, tea.KeyDown:
+		m.step(1, n)
+		return m, nil
+	case tea.KeyShiftTab, tea.KeyUp:
+		m.step(-1, n)
+		return m, nil
+	case tea.KeyEnter:
+		if m.mfocus < 2*mfCount {
+			m.step(1, n)
+			return m, nil
+		}
+		if err := m.validateMachines(); err != "" {
+			m.formErr = err
+			return m, nil
+		}
+		m.formErr = ""
+		if cmd, waiting := m.startJoins(); waiting {
+			m.page = pWait
+			return m, cmd
+		}
+		m.page = pCheck
+		cmd := m.startCheck()
+		return m, cmd
+	}
+	if m.mfocus == 2*mfCount {
+		return m, nil
+	}
+	d := &m.db[m.mfocus/mfCount]
+	if m.mfocus%mfCount == mfAuth {
+		switch k.String() {
+		case "left", "h":
+			d.auth = (d.auth + authCount - 1) % authCount
+		case "right", "l", " ":
+			d.auth = (d.auth + 1) % authCount
+		}
+		if d.auth != authJoin && d.joined != nil {
+			_ = d.joined.Close()
+			d.joined, d.code, d.arrive = nil, "", nil
+		}
+		d.secret.SetValue("")
+		d.secret.Mask = d.auth == authPassword
+		d.secret.Placeholder = map[int]string{authKey: "/root/.ssh/id_ed25519", authPassword: "the user's ssh password"}[d.auth]
+		return m, nil
+	}
+	var cmd tea.Cmd
+	t := d.field(m.mfocus % mfCount)
+	*t, cmd = t.Update(k)
+	return m, cmd
+}
+
+// step moves the focus over the fields that are shown.
+func (m *setupModel) step(d, n int) {
+	for range n {
+		m.mfocus = (m.mfocus + d + n) % n
+		if m.mfocus == 2*mfCount || m.db[m.mfocus/mfCount].visible(m.mfocus%mfCount) {
+			break
+		}
+	}
+	m.focusMachine()
+}
+
+func (m setupModel) allJoined() bool {
+	for _, d := range m.db {
+		if d.auth == authJoin && d.joined == nil {
+			return false
+		}
+	}
+	return true
+}
+
+// startJoins listens for the machines that join with a code, makes their
+// codes, and waits for them; it says whether there is anything to wait for.
+func (m *setupModel) startJoins() (tea.Cmd, bool) {
+	var cmds []tea.Cmd
+	for i := range m.db {
+		d := &m.db[i]
+		if d.auth != authJoin || d.joined != nil {
+			continue
+		}
+		if m.listener == nil {
+			l, err := joinpkg.Listen(fmt.Sprintf(":%d", joinpkg.DefaultPort),
+				fmt.Sprintf("%s:%d", m.thisAddr(), joinpkg.DefaultPort), time.Hour)
+			if err != nil {
+				m.formErr = err.Error()
+				return nil, false
+			}
+			m.listener = l
+		}
+		if d.arrive == nil {
+			c, ch := m.listener.Code("database")
+			d.code, d.arrive = c.String(), ch
+		}
+		ch, i := d.arrive, i
+		cmds = append(cmds, func() tea.Msg { return joinedMsg{i, <-ch} })
+	}
+	return tea.Batch(cmds...), len(cmds) > 0
+}
+
+func (m setupModel) validateMachines() string {
+	seen := map[string]bool{m.thisAddr(): true, m.host: true}
+	for i, d := range m.db {
+		if d.auth == authJoin {
+			continue
+		}
+		addr := strings.TrimSpace(d.addr.Value())
+		switch {
+		case addr == "":
+			return fmt.Sprintf("database machine %d has no address", i+1)
+		case seen[addr]:
+			return fmt.Sprintf("%s is given twice, or is this machine: the three must be different machines", addr)
+		case strings.TrimSpace(d.user.Value()) == "":
+			return fmt.Sprintf("database machine %d has no user to log in as", i+1)
+		case d.auth != authAgent && d.secret.Value() == "":
+			return fmt.Sprintf("database machine %d needs its %s", i+1, authNames[d.auth])
+		}
+		seen[addr] = true
+	}
+	return ""
+}
+
+// startCheck connects to the database machines and runs the checks on all
+// three, this one included.
+func (m *setupModel) startCheck() tea.Cmd {
+	for _, c := range m.conns {
+		if c.ssh != nil {
+			_ = c.ssh.Close()
+		}
+	}
+	m.conns = nil
+	machines := []preflight.Machine{{
+		Name: m.host + " (this machine)", Addr: m.thisAddr(), Local: true, Remote: pgha.Local{},
+		Roles: []preflight.Role{preflight.RoleControl, preflight.RoleWitness},
+	}}
+	for _, d := range m.db {
+		if d.auth == authJoin {
+			m.conns = append(m.conns, dbConn{remote: d.joined, addr: d.joined.Addr, joined: d.joined})
+			machines = append(machines, preflight.Machine{Name: d.joined.Hostname + " (" + d.joined.Addr + ", joined)",
+				Addr: d.joined.Addr, Roles: []preflight.Role{preflight.RoleDatabase}, Remote: d.joined})
+			continue
+		}
+		g := &onboard.GoSSH{
+			Addr: strings.TrimSpace(d.addr.Value()), User: strings.TrimSpace(d.user.Value()),
+			UseAgent: d.auth == authAgent, SudoPassword: d.sudo.Value(),
+		}
+		switch d.auth {
+		case authKey:
+			g.KeyFile = strings.TrimSpace(d.secret.Value())
+		case authPassword:
+			g.Password = d.secret.Value()
+		}
+		m.conns = append(m.conns, dbConn{remote: g, addr: g.Addr, ssh: g})
+		machines = append(machines, preflight.Machine{Name: g.Addr, Addr: g.Addr,
+			Roles: []preflight.Role{preflight.RoleDatabase}, Remote: g})
+	}
+	m.checking = true
+	m.report = preflight.Report{}
+	conns := m.conns
+	return func() tea.Msg {
+		rep := preflight.Run(context.Background(), machines)
+		// A sudo that asks for a password is fine with one given: the
+		// install grants itself sudo with it.
+		for i := range rep.Findings {
+			f := &rep.Findings[i]
+			if f.Check == "sudo" && f.Level == preflight.Warn {
+				for _, c := range conns {
+					g := c.ssh
+					switch {
+					case g == nil || g.Addr != f.Machine:
+					case g.SudoPassword == "":
+						f.Level = preflight.Fail
+						f.Fix = "give this user's sudo password on the previous screen"
+					default:
+						f.Level = preflight.OK
+						f.Detail = "with the password given"
+						f.Fix = ""
+					}
+				}
+			}
+		}
+		return checkDoneMsg{rep}
+	}
+}
+
+// targets are the database machines as user@address, for the command line.
+func (m setupModel) targets() []string {
+	var out []string
+	for _, d := range m.db {
+		if d.joined != nil {
+			out = append(out, "root@"+d.joined.Addr)
+			continue
+		}
+		out = append(out, strings.TrimSpace(d.user.Value())+"@"+strings.TrimSpace(d.addr.Value()))
+	}
+	return out
+}
+
+func (m setupModel) namesList() []string {
+	var out []string
+	for _, h := range strings.Split(m.names.Value(), ",") {
+		if h = strings.TrimSpace(h); h != "" {
+			out = append(out, h)
+		}
+	}
+	return out
+}
+
+// command is what this setup is on the command line, for doing it again
+// from a script — with keys in place, since a password cannot be given there.
+func (m setupModel) command() string {
+	parts := []string{"sudo --preserve-env=SSH_AUTH_SOCK llmgw server init",
+		"--admin " + strings.TrimSpace(m.admin.Value())}
+	for _, h := range m.namesList() {
+		parts = append(parts, "--host "+h)
+	}
+	for _, t := range m.targets() {
+		parts = append(parts, "--patroni "+t)
+	}
+	parts = append(parts, "--no-node", "--no-model")
+	return strings.Join(parts, " \\\n    ")
+}
+
+// startRun installs: sudo granted where it asks for a password, the control
+// plane here, the store on the other two, then sudo taken back.
+func (m *setupModel) startRun() tea.Cmd {
+	in := newDefaultServerInit()
+	in.admin = strings.TrimSpace(m.admin.Value())
+	in.hosts = m.namesList()
+	in.patroni = m.targets()
+	// The control plane of a replicated setup is not where models run;
+	// nodes are added once it is up.
+	in.noNode, in.noModel = true, true
+	m.in = &in
+	m.run = newInitModel(&in, nil, false)
+	m.run.running, m.run.width, m.run.spin = true, m.width, m.spin
+	ref, conns, logp, mu, listener := m.ref, m.conns, m.log, m.logMu, m.listener
+	return func() tea.Msg {
+		send := func(l string) {
+			mu.Lock()
+			*logp = append(*logp, l)
+			mu.Unlock()
+			ref.p.Send(initLineMsg(l))
+		}
+		in.out = &lineWriter{send: send}
+		err := func() error {
+			ctx := context.Background()
+			for _, c := range conns {
+				if c.ssh != nil {
+					fmt.Fprintf(in.out, "  ..    sudo on %s\n", c.addr)
+					if err := c.ssh.PrepareSudo(ctx); err != nil {
+						return err
+					}
+				}
+				out, err := c.remote.Run(ctx, "hostname", nil)
+				if err != nil {
+					return fmt.Errorf("%s: %v", c.addr, err)
+				}
+				ip, err := ipv4Of(c.addr)
+				if err != nil {
+					return err
+				}
+				in.patroniNodes = append(in.patroniNodes, pgha.Node{
+					Member: pgha.Member{Name: strings.TrimSpace(string(out)), IP: ip}, Remote: c.remote})
+			}
+			return in.run(ctx)
+		}()
+		// Whatever happened: a sudo grant taken back, a joined machine's
+		// `llmgw join` told it is done, the port for joining closed.
+		for _, c := range conns {
+			if c.ssh != nil {
+				_ = c.ssh.Close()
+			}
+			if c.joined != nil {
+				_ = c.joined.Close()
+			}
+		}
+		if listener != nil {
+			_ = listener.Close()
+		}
+		return initDoneMsg{err}
+	}
+}
+
+// setupSteps are the sidebar's steps, by page.
+var setupSteps = []struct {
+	label string
+	page  setupPage
+}{
+	{"Start", pWelcome}, {"Administrator", pAdmin}, {"Machines", pMachines},
+	{"Checks", pCheck}, {"Plan", pPlan}, {"Install", pRun},
+}
+
+func (m setupModel) View() string {
+	at := m.page
+	if at == pWait {
+		at = pMachines
+	}
+	var nav []navItem
+	for _, st := range setupSteps {
+		if m.choice == 0 && st.page > pWelcome {
+			break
+		}
+		state := "todo"
+		switch {
+		case st.page < at || (st.page == pRun && m.finished && m.run.err == nil):
+			state = "done"
+		case st.page == at:
+			state = "current"
+		}
+		nav = append(nav, navItem{Label: st.label, State: state})
+	}
+	title, body, keys := m.pageView()
+	f := frame{
+		Width: m.width, Height: m.height,
+		Title: "llmgw setup", Where: m.host + " · " + m.thisAddr(),
+		NavTitle: "Steps", Nav: nav,
+		PanelTitle: title, Body: body, Scroll: m.scroll, Keys: keys,
+	}
+	if m.formErr != "" {
+		f.DialogTitle, f.Dialog = "Not yet", m.formErr+"\n\n"+sDim.Render("any key to go back")
+	}
+	return f.render()
+}
+
+// pageView is the panel for the page: its title, what is in it, and its keys.
+func (m setupModel) pageView() (string, string, string) {
+	switch m.page {
+	case pWelcome:
+		return "Start", m.welcomeView(), "↑↓ choose · Enter go on · Ctrl+C leave"
+	case pAdmin:
+		return "Administrator", m.adminView(), "Tab/↑↓ move · Enter next · Esc back · Ctrl+C leave"
+	case pMachines:
+		return "Database machines", m.machinesView(), "Tab/↑↓ move · ←→ how to log in · Enter next · Esc back · PgUp/PgDn scroll"
+	case pWait:
+		return "Waiting for machines to join", m.waitView(), "Esc back · PgUp/PgDn scroll · Ctrl+C leave"
+	case pCheck:
+		keys := "R check again · Esc back · PgUp/PgDn scroll"
+		if !m.checking && m.report.Worst() < preflight.Fail {
+			keys = "Enter see the plan · " + keys
+		}
+		return "Checks", m.checkView(), keys
+	case pPlan:
+		return "Plan", m.planView(), "Enter install · Esc back · PgUp/PgDn scroll · Ctrl+C leave"
+	}
+	v := m.run.View()
+	if i := strings.Index(v, "\n\n"); i >= 0 {
+		v = v[i+2:] // init's own title; the frame has one
+	}
+	keys := "Ctrl+C stop"
+	if m.finished {
+		keys = "Enter or Q leave · PgUp/PgDn scroll"
+		if m.run.err == nil {
+			v += "\n" + sDim.Render("The same, from a script, with ssh keys in place:") + "\n" + m.command() + "\n"
+		}
+	}
+	return "Install", v, keys
+}
+
+func (m setupModel) welcomeView() string {
+	var b strings.Builder
+	fmt.Fprintf(&b, "\n  Nothing of llmgw is installed on %s.\n\n", lipgloss.NewStyle().Bold(true).Render(m.host))
+	opt := func(i int, title, detail string) {
+		mark := "( )"
+		if m.choice == i {
+			mark = "(" + sOK.Render("•") + ")"
+			title = fSelected.Render(title)
+		}
+		fmt.Fprintf(&b, "  %s %s\n      %s\n\n", mark, title, sDim.Render(detail))
+	}
+	opt(0, "One machine", "This machine runs everything: control plane, model mirror,\n      and models if it has room.")
+	opt(1, "High availability — three machines",
+		"This one is the control plane; two more keep the store, replicated,\n      so losing any one machine loses nothing.")
+	return b.String()
+}
+
+func (m setupModel) adminView() string {
+	var b strings.Builder
+	label := func(i int, s string) string {
+		if m.afocus == i {
+			return fSelected.Render("› " + s)
+		}
+		return "  " + s
+	}
+	b.WriteString("\n")
+	fmt.Fprintf(&b, "%s\n    %s\n    %s\n\n", label(0, "Administrator's email"), m.admin.View(),
+		sDim.Render("A password is generated and shown once, at the end."))
+	fmt.Fprintf(&b, "%s\n    %s\n    %s\n\n", label(1, "Other names nodes will reach this machine by"), m.names.View(),
+		sDim.Render("Known: "+strings.Join(append([]string{m.host}, m.ips...), ", ")+". Add a DNS name or a floating address."))
+	b.WriteString(button(m.afocus == 2, "Next") + "\n")
+	return b.String()
+}
+
+// button is a [ Label ] that shows when it has the focus.
+func button(focused bool, label string) string {
+	if focused {
+		return "  " + fBar.Render(" "+label+" ")
+	}
+	return "  " + lipgloss.NewStyle().Bold(true).Render("[ "+label+" ]")
+}
+
+func (m setupModel) machinesView() string {
+	var b strings.Builder
+	b.WriteString("\n  The two machines that keep the store: Ubuntu 22.04/24.04 or Debian 12.\n" +
+		"  PostgreSQL, Patroni and etcd are installed on them.\n\n")
+	for i, d := range m.db {
+		base := i * mfCount
+		// Padded before it is coloured: the colour's escape codes count
+		// toward a width and threw the focused row out of line.
+		label := func(f int, s string) string {
+			s = fmt.Sprintf("%-15s", s)
+			if m.mfocus == base+f {
+				return fSelected.Render("› " + s)
+			}
+			return "  " + s
+		}
+		fmt.Fprintf(&b, "  %s\n", lipgloss.NewStyle().Bold(true).Render(fmt.Sprintf("Database machine %d", i+1)))
+		if d.visible(mfAddr) {
+			fmt.Fprintf(&b, "  %s %s\n", label(mfAddr, "Address"), d.addr.View())
+			fmt.Fprintf(&b, "  %s %s\n", label(mfUser, "User"), d.user.View())
+		}
+		var auth []string
+		for a := 0; a < authCount; a++ {
+			if a == d.auth {
+				auth = append(auth, "("+sOK.Render("•")+") "+authNames[a])
+			} else {
+				auth = append(auth, "( ) "+sDim.Render(authNames[a]))
+			}
+		}
+		fmt.Fprintf(&b, "  %s   %s\n", label(mfAuth, "Log in with"), strings.Join(auth, "  "))
+		if d.visible(mfSecret) {
+			fmt.Fprintf(&b, "  %s %s\n", label(mfSecret, strings.ToUpper(authNames[d.auth][:1])+authNames[d.auth][1:]), d.secret.View())
+		}
+		if d.visible(mfSudo) {
+			fmt.Fprintf(&b, "  %s %s\n", label(mfSudo, "Sudo password"), d.sudo.View())
+		}
+		if d.auth == authJoin {
+			b.WriteString("      " + sDim.Render("Nothing logs into it: it runs llmgw join with a code shown next.") + "\n")
+		}
+		b.WriteString("\n")
+	}
+	b.WriteString(button(m.mfocus == 2*mfCount, "Check the three machines") + "\n\n")
+	b.WriteString("  " + sDim.Render("Passwords are kept in memory only, for this setup.") + "\n")
+	return b.String()
+}
+
+func (m setupModel) waitView() string {
+	var b strings.Builder
+	fmt.Fprintf(&b, "\n  On each machine below, as root. It dials this machine on port %d;\n  nothing dials in. Each code is for one machine and good for an hour.\n\n", joinpkg.DefaultPort)
+	for i, d := range m.db {
+		if d.auth != authJoin {
+			continue
+		}
+		title := lipgloss.NewStyle().Bold(true).Render(fmt.Sprintf("Database machine %d", i+1))
+		if d.joined != nil {
+			fmt.Fprintf(&b, "  %s  %s joined as %s (%s)\n\n", title, sOK.Render("✓"), d.joined.Hostname, d.joined.Addr)
+			continue
+		}
+		fmt.Fprintf(&b, "  %s  %s waiting\n", title, m.spin.View())
+		b.WriteString("  " + sDim.Render("without llmgw yet:") + "\n")
+		for _, l := range joinInstall() {
+			b.WriteString("    " + strings.ReplaceAll(l, "\n", "\n    ") + "\n")
+		}
+		b.WriteString("  " + sDim.Render("then:") + "\n    sudo llmgw join " + d.code + "\n\n")
+	}
+	return b.String()
+}
+
+func (m setupModel) checkView() string {
+	var b strings.Builder
+	b.WriteString("\n")
+	if m.checking {
+		b.WriteString("  " + m.spin.View() + " checking the three machines…\n")
+		return b.String()
+	}
+	switch m.report.Worst() {
+	case preflight.Fail:
+		b.WriteString("  " + sBad.Render("✗ Fix what is marked ✗, then check again (R).") + "\n")
+	case preflight.Warn:
+		b.WriteString("  " + sWarn.Render("! It can go ahead; what is marked ! is worth a look.") + "\n")
+	default:
+		b.WriteString("  " + sOK.Render("✓ All three are ready.") + "\n")
+	}
+	machine := ""
+	for _, f := range m.report.Findings {
+		if f.Machine != machine {
+			machine = f.Machine
+			b.WriteString("\n  " + lipgloss.NewStyle().Bold(true).Render(machine) + "\n")
+		}
+		mark := sOK.Render("✓")
+		switch f.Level {
+		case preflight.Warn:
+			mark = sWarn.Render("!")
+		case preflight.Fail:
+			mark = sBad.Render("✗")
+		}
+		fmt.Fprintf(&b, "    %s %-13s %s\n", mark, f.Check, f.Detail)
+		if f.Level != preflight.OK && f.Fix != "" {
+			b.WriteString("                    " + sDim.Render(f.Fix) + "\n")
+		}
+	}
+	return b.String()
+}
+
+func (m setupModel) planView() string {
+	var b strings.Builder
+	b.WriteString("\n  What will be installed:\n\n")
+	fmt.Fprintf(&b, "  %s  %s\n      %s\n", lipgloss.NewStyle().Bold(true).Render(m.host), sDim.Render(m.thisAddr()),
+		"control plane and gateway, model mirror, etcd (the third vote)")
+	fmt.Fprintf(&b, "      administrator %s\n", strings.TrimSpace(m.admin.Value()))
+	granted := false
+	for _, d := range m.db {
+		addr := strings.TrimSpace(d.addr.Value())
+		if d.joined != nil {
+			addr = d.joined.Hostname + "  " + sDim.Render(d.joined.Addr+", joined")
+		}
+		fmt.Fprintf(&b, "  %s\n      PostgreSQL under Patroni, etcd\n", lipgloss.NewStyle().Bold(true).Render(addr))
+		granted = granted || d.sudo.Value() != ""
+	}
+	b.WriteString("\n  The store is replicated from the first to the second; if the primary is lost\n" +
+		"  the standby takes over in about thirty seconds, and the control plane follows.\n")
+	if granted {
+		b.WriteString("  " + sDim.Render("Where a sudo password was given, the user may sudo without it until the\n  install finishes, and not after.") + "\n")
+	}
+	b.WriteString("\n  " + sDim.Render("The same from a script, with ssh keys in place:") + "\n")
+	for _, l := range strings.Split(m.command(), "\n") {
+		b.WriteString("  " + l + "\n")
+	}
+	for _, d := range m.db {
+		if d.joined != nil {
+			b.WriteString("  " + sDim.Render("(a machine that joined with a code needs ssh for that)") + "\n")
+			break
+		}
+	}
+	b.WriteString("\n" + button(true, "Enter to install") + "\n")
+	return b.String()
+}
+
+// runSetupTUI is the setup screens, until they finish or hand over to init's
+// own form for one machine.
+func runSetupTUI(ctx context.Context, out io.Writer) error {
+	host, _ := os.Hostname()
+	var ips []string
+	for _, ip := range localAddrs() {
+		if ip.To4() != nil {
+			ips = append(ips, ip.String())
+		}
+	}
+	m := newSetupModel(host, ips)
+	p := tea.NewProgram(m, tea.WithOutput(out), tea.WithAltScreen())
+	m.ref.p = p
+	final, err := p.Run()
+	if err != nil {
+		return err
+	}
+	fm := final.(setupModel)
+	if fm.single {
+		in := newDefaultServerInit()
+		in.out = out
+		return in.runInitTUI(ctx, out)
+	}
+	if fm.page < pRun {
+		return nil
+	}
+	fmt.Fprintln(out)
+	if fm.run.err != nil {
+		fm.logMu.Lock()
+		fmt.Fprintln(out, strings.Join(*fm.log, "\n"))
+		fm.logMu.Unlock()
+		return fm.run.err
+	}
+	if !fm.finished {
+		return fmt.Errorf("left while installing; running llmgw again finishes what was started")
+	}
+	if fm.run.password != "" {
+		fmt.Fprintf(out, "Administrator %s, password %s (shown once)\n\n", fm.in.admin, fm.run.password)
+	}
+	fmt.Fprintln(out, "Next: add nodes from the console — run llmgw again — or with")
+	fmt.Fprintln(out, "  sudo llmgw node add <user>@<address>")
+	return nil
+}

internal/cli/tui_setup_test.go

@@ -0,0 +1,173 @@
+package cli
+
+import (
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+
+	tea "github.com/charmbracelet/bubbletea"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/preflight"
+)
+
+func press(m tea.Model, keys ...string) tea.Model {
+	for _, k := range keys {
+		var msg tea.KeyMsg
+		switch k {
+		case "enter":
+			msg = tea.KeyMsg{Type: tea.KeyEnter}
+		case "tab":
+			msg = tea.KeyMsg{Type: tea.KeyTab}
+		case "down":
+			msg = tea.KeyMsg{Type: tea.KeyDown}
+		case "esc":
+			msg = tea.KeyMsg{Type: tea.KeyEsc}
+		case "right":
+			msg = tea.KeyMsg{Type: tea.KeyRight}
+		default:
+			msg = tea.KeyMsg{Type: tea.KeyRunes, Runes: []rune(k)}
+		}
+		m, _ = m.Update(msg)
+	}
+	return m
+}
+
+func TestTheSetupAsksForWhatHighAvailabilityNeedsAndRefusesWhatWillNotDo(t *testing.T) {
+	m := tea.Model(newSetupModel("ctl", []string{"10.0.0.5"}))
+	m = press(m, "down", "enter")
+	if s := m.(setupModel); s.page != pAdmin {
+		t.Fatalf("high availability did not lead to the administrator: page %d", s.page)
+	}
+	// No email: refused, with a reason.
+	m = press(m, "tab", "tab", "enter")
+	if s := m.(setupModel); s.page != pAdmin || !strings.Contains(s.formErr, "email") {
+		t.Fatalf("went on without an administrator: %+v", s.formErr)
+	}
+	// The refusal is a dialog; a key closes it, and the cursor is back on
+	// the email.
+	if !strings.Contains(m.View(), "Not yet") {
+		t.Error("the refusal was not shown as a dialog")
+	}
+	m = press(m, "x", "a@example.com", "tab", "tab", "enter")
+	s := m.(setupModel)
+	if s.page != pMachines {
+		t.Fatalf("page %d: %s", s.page, s.formErr)
+	}
+	// This machine's own address is not a database machine.
+	s.db[0].addr.SetValue("10.0.0.5")
+	s.db[0].user.SetValue("ubuntu")
+	s.db[0].auth = authAgent
+	s.db[1].addr.SetValue("10.0.0.7")
+	s.db[1].user.SetValue("ubuntu")
+	s.db[1].auth = authPassword
+	if err := s.validateMachines(); !strings.Contains(err, "this machine") {
+		t.Errorf("this machine as a database machine: %q", err)
+	}
+	s.db[0].addr.SetValue("10.0.0.6")
+	if err := s.validateMachines(); !strings.Contains(err, "password") {
+		t.Errorf("a password login with no password: %q", err)
+	}
+	s.db[1].secret.SetValue("pw")
+	if err := s.validateMachines(); err != "" {
+		t.Errorf("a valid pair refused: %s", err)
+	}
+	cmd := s.command()
+	for _, want := range []string{"--admin a@example.com", "--patroni ubuntu@10.0.0.6", "--patroni ubuntu@10.0.0.7", "--no-node"} {
+		if !strings.Contains(cmd, want) {
+			t.Errorf("command lacks %q:\n%s", want, cmd)
+		}
+	}
+	if strings.Contains(cmd, "pw") {
+		t.Error("the password is in the command line")
+	}
+}
+
+func TestChoosingOneMachineHandsOverToInit(t *testing.T) {
+	m := press(tea.Model(newSetupModel("ctl", nil)), "enter").(setupModel)
+	if !m.single {
+		t.Error("one machine did not hand over")
+	}
+}
+
+func TestTheChecksHoldTheSetupBackOnlyForAFailure(t *testing.T) {
+	m := newSetupModel("ctl", nil)
+	m.choice, m.page = 1, pCheck
+	m.report = preflight.Report{Findings: []preflight.Finding{{Machine: "a", Check: "disk", Level: preflight.Fail, Fix: "more"}}}
+	if s := press(m, "enter").(setupModel); s.page != pCheck {
+		t.Error("went on past a failed check")
+	}
+	m.report.Findings[0].Level = preflight.Warn
+	if s := press(m, "enter").(setupModel); s.page != pPlan {
+		t.Error("a warning held it back")
+	}
+}
+
+func TestTheConsoleOffersToReplicateOnlyAStoreThatIsNot(t *testing.T) {
+	m := consoleModel{}
+	if !m.entryOK(cReplicate) {
+		t.Error("a single store is not offered replication")
+	}
+	m.data.sum.Store = []adminapi.StoreMember{{Name: "a", Role: "leader"}}
+	if m.entryOK(cReplicate) {
+		t.Error("a replicated store is offered replication")
+	}
+	// A node is asked for in a dialog, then the console closes with it to do.
+	var mm tea.Model = consoleModel{menu: cAddNode, loaded: true, width: 100, height: 30}
+	mm = press(mm, "enter", "ubuntu@10.0.0.9", "enter")
+	if a := mm.(consoleModel).action; a.kind != "node" || a.targets[0] != "ubuntu@10.0.0.9" {
+		t.Errorf("%+v", a)
+	}
+}
+
+func TestTheConsoleWatchesTheClusterAndEditsItsSettings(t *testing.T) {
+	dir := t.TempDir()
+	file := filepath.Join(dir, "server.yaml")
+	if err := os.WriteFile(file, []byte("# mine\nlisten: \":8080\"\n"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	m := newConsoleModel("127.0.0.1:1", file)
+	m.width, m.height, m.loaded = 120, 40, true
+	m.data.nodes = []adminapi.NodeView{{ID: "cuda01", State: "ready", GPUTotal: 1, GPUs: []adminapi.GPUView{{Index: 0, Name: "Quadro RTX 6000"}}}}
+	m.data.replicas = []adminapi.ReplicaView{{ID: "r1", Model: "coder-14b", State: "ready",
+		Shards: []adminapi.ShardView{{Node: "cuda01", GPUs: []int{0}}}}}
+
+	// Watching: the GPUs view says which model holds each card.
+	m.menu = cGPUs
+	if v := m.View(); !strings.Contains(v, "Quadro RTX 6000") || !strings.Contains(v, "coder-14b") {
+		t.Errorf("the GPUs view:\n%s", v)
+	}
+
+	// Settings: into router, change retries, save.
+	var mm tea.Model = m
+	mm.(consoleModel).View()
+	cm := mm.(consoleModel)
+	cm.menu, cm.inPanel = cSettings, true
+	for i, r := range cm.settingsRows() {
+		if r.label == "router" {
+			cm.row = i
+		}
+	}
+	mm = press(cm, "enter") // into router
+	cm = mm.(consoleModel)
+	for i, r := range cm.settingsRows() {
+		if r.label == "retries" {
+			cm.row = i
+		}
+	}
+	if v := cm.View(); !strings.Contains(v, "(default)") {
+		t.Errorf("an unset key does not show its default:\n%s", v)
+	}
+	mm = press(cm, "enter") // the edit dialog
+	cm = mm.(consoleModel)
+	cm.prompt.SetValue("5")
+	mm = press(cm, "enter", "s")
+	if d := mm.(consoleModel).dialog; d != "saved" {
+		t.Fatalf("after saving the dialog is %q: %s", d, mm.(consoleModel).dialogText)
+	}
+	b, _ := os.ReadFile(file)
+	if !strings.Contains(string(b), "retries: 5") || !strings.Contains(string(b), "# mine") {
+		t.Errorf("server.yaml:\n%s", b)
+	}
+}

internal/cli/tui_widgets.go

@@ -47,6 +47,8 @@ type textField struct {
 	Placeholder string
 	CharLimit   int
 	Width       int
+	// Mask shows a dot for every character: a password.
+	Mask bool
 }
 
 func newTextField() textField { return textField{Prompt: "> "} }
@@ -84,6 +86,9 @@ func (f textField) Update(msg tea.Msg) (textField, tea.Cmd) {
 
 func (f textField) View() string {
 	text := string(f.value)
+	if f.Mask {
+		text = strings.Repeat("•", len(f.value))
+	}
 	shown := text
 	if text == "" && f.Placeholder != "" {
 		shown = sDim.Render(f.Placeholder)

internal/config/config.go

@@ -8,6 +8,7 @@ package config
 import (
 	"fmt"
 	"os"
+	"slices"
 	"strings"
 	"time"
 
@@ -85,6 +86,13 @@ type Server struct {
 	// one leaves that engine to be installed by hand.
 	Engines map[string]string `koanf:"engines"`
 
+	// EngineOverrides give some nodes different engines from the rest: a
+	// vLLM built for CUDA 12 on the Volta cards CUDA 13 dropped, or none at
+	// all on nodes meant only for llama.cpp. Each picks nodes and lays its
+	// engines over the cluster's, in order, later ones last; an empty version
+	// removes that engine from the nodes it picks.
+	EngineOverrides []EngineOverride `koanf:"engine_overrides"`
+
 	// BackupDir is where `llmgw server backup` writes, and where the newest
 	// archive's age is read for llmgw_backup_last_success_timestamp_seconds.
 	// Empty means /var/backups/llmgw-server.
@@ -249,6 +257,14 @@ type Auth struct {
 	// blocking a request: losing accounting is bad, turning a stalled database
 	// into an outage is worse.
 	BufferSize int `koanf:"buffer_size"`
+
+	// UsageRetention is how long the record of each request is kept: who sent
+	// it, which replica answered, how long it queued — what `ctl request`
+	// reads. Zero keeps it for ever. Totals, charts and budgets are read from
+	// five-minute sums kept beside the records, which are not pruned, so this
+	// bounds the table that grows by a row a request without changing any
+	// figure a page shows. At least a day when set.
+	UsageRetention time.Duration `koanf:"usage_retention"`
 }
 
 // Personal configures use that belongs to a person rather than a project.
@@ -355,6 +371,13 @@ type Router struct {
 	// accounting works regardless of client behaviour.
 	InjectUsage bool `koanf:"inject_usage"`
 
+	// TenantLabel is the level of the tree the per-tenant request series are
+	// labelled by — llmgw_tenant_requests_total and its siblings: "org",
+	// "team" or "project", or "none" for no such series. A label per project
+	// on a cluster of thousands is more series than Prometheus wants, and a
+	// team is what an alert is usually about; "team" is the default.
+	TenantLabel string `koanf:"tenant_label"`
+
 	// MaxBodyBytes bounds the request body the router will read.
 	MaxBodyBytes int64 `koanf:"max_body_bytes"`
 
@@ -475,6 +498,7 @@ func Default() Server {
 			Cooldown:         15 * time.Second,
 			FailureThreshold: 3,
 			DefaultProfile:   "chat",
+			TenantLabel:      "team",
 			SessionHeader:    "x-session-id",
 			InjectUsage:      true,
 			MaxBodyBytes:     32 << 20,
@@ -548,6 +572,14 @@ func Load(path string) (Server, error) {
 // Validate rejects configurations that would otherwise fail confusingly at
 // request time.
 func (s Server) Validate() error {
+	for i, o := range s.EngineOverrides {
+		if o.GPUClass == "" && len(o.Nodes) == 0 && len(o.Labels) == 0 {
+			return fmt.Errorf("engine_overrides[%d] picks no nodes: give it a gpu_class, nodes or labels", i)
+		}
+		if len(o.Engines) == 0 {
+			return fmt.Errorf("engine_overrides[%d] changes no engine: give it engines, with \"\" for none", i)
+		}
+	}
 	if s.Listen == "" {
 		return fmt.Errorf("listen is empty")
 	}
@@ -557,6 +589,14 @@ func (s Server) Validate() error {
 	if s.Auth.Enabled && s.Auth.Store == "" {
 		return fmt.Errorf("auth.store is required when auth is enabled")
 	}
+	switch s.Router.TenantLabel {
+	case "", "none", "org", "team", "project":
+	default:
+		return fmt.Errorf("router.tenant_label is %q: org, team, project or none", s.Router.TenantLabel)
+	}
+	if s.Auth.UsageRetention != 0 && s.Auth.UsageRetention < 24*time.Hour {
+		return fmt.Errorf("auth.usage_retention is %s: at least 24h, or 0 to keep every request", s.Auth.UsageRetention)
+	}
 	if s.Threads.Store != "" && s.Threads.RetainHours <= 0 {
 		return fmt.Errorf("threads.retain_hours must be positive when threads.store is set")
 	}
@@ -626,6 +666,58 @@ var DefaultEngines = map[string]string{
 	"vllm":     "0.30.0",
 }
 
+// EngineOverride gives the nodes it picks different engines. Every selector
+// set must match; at least one must be set.
+type EngineOverride struct {
+	// GPUClass picks nodes with a GPU of this class ("v100-32g").
+	GPUClass string `koanf:"gpu_class"`
+	// Nodes picks nodes by id.
+	Nodes []string `koanf:"nodes"`
+	// Labels picks nodes carrying all of these.
+	Labels map[string]string `koanf:"labels"`
+	// Engines are laid over the cluster's: a version, or "" for none.
+	Engines map[string]string `koanf:"engines"`
+}
+
+// Picks reports whether the override applies to a node with this id, these
+// GPU classes and these labels.
+func (o EngineOverride) Picks(node string, classes []string, labels map[string]string) bool {
+	if o.GPUClass == "" && len(o.Nodes) == 0 && len(o.Labels) == 0 {
+		return false
+	}
+	if o.GPUClass != "" && !slices.Contains(classes, o.GPUClass) {
+		return false
+	}
+	if len(o.Nodes) > 0 && !slices.Contains(o.Nodes, node) {
+		return false
+	}
+	for k, v := range o.Labels {
+		if labels[k] != v {
+			return false
+		}
+	}
+	return true
+}
+
+// EnginesFor is what one node installs: WantedEngines with every override
+// that picks it laid over, in order.
+func (s Server) EnginesFor(node string, classes []string, labels map[string]string) map[string]string {
+	out := s.WantedEngines()
+	for _, o := range s.EngineOverrides {
+		if !o.Picks(node, classes, labels) {
+			continue
+		}
+		for k, v := range o.Engines {
+			if v == "" {
+				delete(out, k)
+				continue
+			}
+			out[k] = v
+		}
+	}
+	return out
+}
+
 // WantedEngines is DefaultEngines with the configuration laid over it.
 func (s Server) WantedEngines() map[string]string {
 	out := map[string]string{}

internal/config/config_test.go

@@ -206,3 +206,17 @@ func TestEnginesDefaultToTheCheckedVersions(t *testing.T) {
 		t.Error("an emptied engine is still wanted")
 	}
 }
+
+func TestAUsageRetentionUnderADayIsRefused(t *testing.T) {
+	// Shorter than a day is almost certainly a unit slip — 90m for 90 days —
+	// and would delete every request record within the hour.
+	s := Default()
+	s.Auth.UsageRetention = 90 * time.Minute
+	if err := s.Validate(); err == nil || !strings.Contains(err.Error(), "usage_retention") {
+		t.Errorf("a 90-minute retention was accepted: %v", err)
+	}
+	s.Auth.UsageRetention = 90 * 24 * time.Hour
+	if err := s.Validate(); err != nil {
+		t.Errorf("a 90-day retention was refused: %v", err)
+	}
+}

internal/config/engines_test.go

@@ -0,0 +1,38 @@
+package config
+
+import (
+	"strings"
+	"testing"
+)
+
+func TestEachNodeGetsTheEnginesItsOverridesSay(t *testing.T) {
+	s := Server{
+		Engines: map[string]string{"vllm": "0.30.0"},
+		EngineOverrides: []EngineOverride{
+			// Volta: CUDA 13 dropped it, CUDA 12 vLLM still runs.
+			{GPUClass: "v100-32g", Engines: map[string]string{"vllm": "0.25.1"}},
+			// Nodes meant for llama.cpp alone.
+			{Labels: map[string]string{"role": "gguf"}, Engines: map[string]string{"vllm": ""}},
+		},
+	}
+	if got := s.EnginesFor("h1", []string{"h100-80g"}, nil); got["vllm"] != "0.30.0" || got["llamacpp"] == "" {
+		t.Errorf("an H100 node: %v", got)
+	}
+	if got := s.EnginesFor("v1", []string{"v100-32g"}, nil); got["vllm"] != "0.25.1" {
+		t.Errorf("a V100 node: %v", got)
+	}
+	got := s.EnginesFor("v2", []string{"v100-32g"}, map[string]string{"role": "gguf"})
+	if _, has := got["vllm"]; has || got["llamacpp"] == "" {
+		t.Errorf("a V100 node for llama.cpp alone, the later override last: %v", got)
+	}
+
+	for _, bad := range []EngineOverride{
+		{Engines: map[string]string{"vllm": ""}},
+		{GPUClass: "v100-32g"},
+	} {
+		s := Server{Listen: ":8080", EngineOverrides: []EngineOverride{bad}}
+		if err := s.Validate(); err == nil || !strings.Contains(err.Error(), "engine_overrides[0]") {
+			t.Errorf("%+v: %v", bad, err)
+		}
+	}
+}

internal/control/adminapi/api.go

@@ -9,6 +9,7 @@ package adminapi
 import (
 	"context"
 	"encoding/json"
+	"errors"
 	"fmt"
 	"io"
 	"net/http"
@@ -39,6 +40,8 @@ import (
 type API struct {
 	// releases is where release binaries are fetched from, for upgrades.
 	releases string
+	// storeStatus reads a replicated store's members; nil for one database.
+	storeStatus func() ([]StoreMember, error)
 	// serve runs registry-to-replica jobs; nil without a mirror and a store.
 	serve *modelserve.Server
 
@@ -111,6 +114,7 @@ func (a *API) Register(mux *http.ServeMux) {
 	mux.HandleFunc("GET /api/v1/explain-placement", a.explainPlacement)
 	mux.HandleFunc("POST /api/v1/nodes/{id}/state", a.setNodeState)
 	mux.HandleFunc("POST /api/v1/nodes/{id}/upgrade", a.upgradeNode)
+	mux.HandleFunc("POST /api/v1/nodes/{id}/engines", a.setNodeEngine)
 	mux.HandleFunc("DELETE /api/v1/nodes/{id}", a.removeNode)
 	mux.HandleFunc("POST /api/v1/fabric-test", a.fabricTest)
 	mux.HandleFunc("GET /api/v1/nodes/{id}/logs", a.nodeLogs)
@@ -156,6 +160,10 @@ type NodeView struct {
 	// Provision is where installing each wanted engine stands. An engine is in
 	// Engines only once its installation here is ready.
 	Provision []inventory.EngineInstall `json:"provision,omitempty"`
+	// EngineBuilds are the builds chosen for this node, by engine: "auto"
+	// (its hardware decides), a build ("cu129"), or "none". Only on a node
+	// with GPUs, the only kind with a choice to make.
+	EngineBuilds map[string]string `json:"engine_builds,omitempty"`
 	// OS is the operating system and architecture, as the agent read them.
 	OS string `json:"os,omitempty"`
 	// Warnings are facts about the machine that make it slow or unusable
@@ -208,10 +216,13 @@ type NodeView struct {
 
 // GPUView is one card as a node's page shows it.
 type GPUView struct {
-	Index       int    `json:"index"`
-	Name        string `json:"name"`
-	NUMA        int    `json:"numa"`
-	NVLinkPeers []int  `json:"nvlink_peers,omitempty"`
+	Index int    `json:"index"`
+	Name  string `json:"name"`
+	// ComputeCapability is what decides which engine builds run on the card:
+	// CUDA 13 builds need 7.5, so a V100 (7.0) runs only older ones.
+	ComputeCapability string `json:"compute_capability,omitempty"`
+	NUMA              int    `json:"numa"`
+	NVLinkPeers       []int  `json:"nvlink_peers,omitempty"`
 	// NearestNIC is the adapter closest to the card, and how close.
 	NearestNIC  string `json:"nearest_nic,omitempty"`
 	NICPath     string `json:"nic_path,omitempty"`
@@ -470,6 +481,13 @@ func (a *API) Nodes() []NodeView {
 		}
 		slices.Sort(v.Engines)
 		v.Provision = n.Provision
+		if ch := a.ctl.EngineChoices(); ch != nil && len(n.GPUs) > 0 {
+			chosen := ch.For(n.ID)
+			v.EngineBuilds = map[string]string{"vllm": "auto"}
+			for e, b := range chosen {
+				v.EngineBuilds[e] = b
+			}
+		}
 		if h := n.Host; h.OSID != "" {
 			v.OS = strings.TrimSpace(h.OSID + " " + h.OSVersion + " " + h.Arch)
 			if h.Arch == "amd64" && !h.HasCPUFlag("avx2") {
@@ -497,7 +515,8 @@ func (a *API) Nodes() []NodeView {
 		v.Load = loadView(n.Load, time.Now())
 		v.NVLinkDomain, v.IMEX, v.IMEXChannels = n.NVLinkDomain(), n.IMEX.State, n.IMEX.Channels
 		for _, g := range n.GPUs {
-			gv := GPUView{Index: g.Index, Name: g.Name, NUMA: g.NUMANode, NVLinkPeers: g.NVLinkPeers}
+			gv := GPUView{Index: g.Index, Name: g.Name, ComputeCapability: g.ComputeCapability,
+				NUMA: g.NUMANode, NVLinkPeers: g.NVLinkPeers}
 			for hca, rel := range g.NICPaths {
 				if gv.NearestNIC == "" || discovery.PathRank(rel) < discovery.PathRank(gv.NICPath) ||
 					(discovery.PathRank(rel) == discovery.PathRank(gv.NICPath) && hca < gv.NearestNIC) {
@@ -681,6 +700,9 @@ type Usage interface {
 // SetUsage supplies the usage reporting source.
 func (a *API) SetUsage(u Usage) { a.usage = u }
 
+// SetStoreStatus supplies the state of a replicated store, for the summary.
+func (a *API) SetStoreStatus(f func() ([]StoreMember, error)) { a.storeStatus = f }
+
 // ScopeUsageView is one group's spend, with its quota when it has one.
 type ScopeUsageView struct {
 	Scope string `json:"scope"`
@@ -1238,6 +1260,9 @@ type RestartRequest struct {
 	Replicas []string `json:"replicas,omitempty"`
 	Models   []string `json:"models,omitempty"`
 	All      bool     `json:"all,omitempty"`
+	// InPlace stops each replica before starting its replacement, rather
+	// than starting the replacement beside it first.
+	InPlace bool `json:"in_place,omitempty"`
 }
 
 // RestartResult lists the replicas queued.
@@ -1261,7 +1286,11 @@ func (a *API) restart(w http.ResponseWriter, r *http.Request) {
 		writeError(w, http.StatusBadRequest, "nothing ready matches; name replicas, models, or all")
 		return
 	}
-	if err := a.ctl.Reconciler().Restart(ids...); err != nil {
+	restart := a.ctl.Reconciler().Restart
+	if req.InPlace {
+		restart = a.ctl.Reconciler().RestartInPlace
+	}
+	if err := restart(ids...); err != nil {
 		writeError(w, http.StatusNotFound, err.Error())
 		return
 	}
@@ -1521,3 +1550,47 @@ func (a *API) setAdapters(w http.ResponseWriter, r *http.Request) {
 	}
 	writeJSON(w, http.StatusOK, AdaptersRequest{Adapters: out.Adapters})
 }
+
+// NodeEngineRequest chooses one engine's build for a node.
+type NodeEngineRequest struct {
+	Engine string `json:"engine"`
+	// Build is "auto", a build such as "cu129", or "none".
+	Build string `json:"build"`
+}
+
+// setNodeEngine chooses an engine's build for a node, and has the node
+// install it.
+func (a *API) setNodeEngine(w http.ResponseWriter, r *http.Request) {
+	ch := a.ctl.EngineChoices()
+	if ch == nil {
+		writeError(w, http.StatusNotImplemented, "this control plane has no store to keep the choice in")
+		return
+	}
+	id := r.PathValue("id")
+	if _, ok := a.ctl.Inventory().Get(id); !ok {
+		writeError(w, http.StatusNotFound, "no node "+id)
+		return
+	}
+	var req NodeEngineRequest
+	if err := json.NewDecoder(io.LimitReader(r.Body, 1<<12)).Decode(&req); err != nil {
+		writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
+		return
+	}
+	if err := ch.Set(r.Context(), id, req.Engine, req.Build, actorOf(r)); err != nil {
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	}
+	writeJSON(w, http.StatusOK, req)
+}
+
+// SetNodeEngine chooses an engine's build for a node, for the cluster page.
+func (a *API) SetNodeEngine(ctx context.Context, node, engine, build, by string) error {
+	ch := a.ctl.EngineChoices()
+	if ch == nil {
+		return errors.New("this control plane has no store to keep the choice in")
+	}
+	if _, ok := a.ctl.Inventory().Get(node); !ok {
+		return errors.New("no node " + node)
+	}
+	return ch.Set(ctx, node, engine, build, by)
+}

internal/control/adminapi/nodeengine_test.go

@@ -0,0 +1,63 @@
+package adminapi
+
+import (
+	"context"
+	"log/slog"
+	"net/http"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+)
+
+// choiceStore keeps engine choices in memory.
+type choiceStore struct{ m map[string]map[string]string }
+
+func (c *choiceStore) NodeEngines(context.Context) (map[string]map[string]string, error) {
+	return c.m, nil
+}
+func (c *choiceStore) SetNodeEngine(_ context.Context, node, engine, variant, _ string) error {
+	if c.m[node] == nil {
+		c.m[node] = map[string]string{}
+	}
+	c.m[node][engine] = variant
+	return nil
+}
+
+func TestAnEngineBuildIsChosenForANodeAndTheNodeIsTold(t *testing.T) {
+	a, ctl := setup(t, testModels(), gpuNode("gpu07", 8), inventory.Node{ID: "cpu01", State: inventory.Ready})
+	ch := control.NewEngineChoices(t.Context(), &choiceStore{m: map[string]map[string]string{}}, slog.New(slog.DiscardHandler))
+	var told []string
+	ch.OnChange(func(node string) error { told = append(told, node); return nil })
+	ctl.SetEngineChoices(ch)
+
+	if code := call(t, a, http.MethodPost, "/api/v1/nodes/gpu07/engines", `{"engine":"vllm","build":"cu129"}`, nil); code != http.StatusOK {
+		t.Fatalf("status %d", code)
+	}
+	if len(told) != 1 || told[0] != "gpu07" {
+		t.Errorf("the node was not told: %v", told)
+	}
+	var nodes []NodeView
+	call(t, a, http.MethodGet, "/api/v1/nodes", "", &nodes)
+	for _, n := range nodes {
+		switch n.ID {
+		case "gpu07":
+			if n.EngineBuilds["vllm"] != "cu129" {
+				t.Errorf("gpu07 shows %v", n.EngineBuilds)
+			}
+		case "cpu01":
+			if n.EngineBuilds != nil {
+				t.Errorf("a node with no GPU has no build to choose: %v", n.EngineBuilds)
+			}
+		}
+	}
+	for _, bad := range []string{`{"engine":"vllm","build":"cu200"}`, `{"engine":"llamacpp","build":"cu129"}`} {
+		if code := call(t, a, http.MethodPost, "/api/v1/nodes/gpu07/engines", bad, nil); code != http.StatusBadRequest {
+			t.Errorf("%s: status %d", bad, code)
+		}
+	}
+	if code := call(t, a, http.MethodPost, "/api/v1/nodes/gpu07/engines", `{"engine":"vllm","build":"auto"}`, nil); code != http.StatusOK ||
+		ch.For("gpu07")["vllm"] != "" {
+		t.Errorf("auto should leave it to the node again: %d %v", code, ch.For("gpu07"))
+	}
+}

internal/control/adminapi/summary.go

@@ -31,6 +31,20 @@ type Summary struct {
 	// IdleReplicas are serving replicas nothing has asked for in a while, which
 	// on a fleet with more models than GPUs is the first place to find room.
 	IdleReplicas []IdleReplica `json:"idle_replicas,omitempty"`
+	// Store is the members of the store's cluster when it is replicated, and
+	// StoreError why it could not be read; both empty for a single database.
+	Store      []StoreMember `json:"store,omitempty"`
+	StoreError string        `json:"store_error,omitempty"`
+}
+
+// StoreMember is one member of a replicated store, as its Patroni reports it.
+type StoreMember struct {
+	Name  string `json:"name"`
+	Host  string `json:"host"`
+	Role  string `json:"role"`
+	State string `json:"state"`
+	// Lag is bytes behind the leader; -1 unknown, 0 for the leader.
+	Lag int64 `json:"lag"`
 }
 
 // NodeTotals counts nodes by the states that need different responses.
@@ -107,6 +121,13 @@ const idleThreshold = 10 * time.Minute
 // Summary builds the fleet summary.
 func (a *API) Summary() Summary {
 	var s Summary
+	if a.storeStatus != nil {
+		members, err := a.storeStatus()
+		s.Store = members
+		if err != nil {
+			s.StoreError = err.Error()
+		}
+	}
 
 	nodes := a.Nodes()
 	byClass := map[string]*ClassUse{}

internal/control/agentsvc/controlplanes.go

@@ -0,0 +1,65 @@
+package agentsvc
+
+import (
+	"context"
+	"slices"
+	"time"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+)
+
+// SetControlPlanes gives the service where the list of control planes comes
+// from. Every agent is told it when it registers, and again when it changes.
+func (s *Service) SetControlPlanes(list func(context.Context) ([]string, error)) {
+	s.mu.Lock()
+	s.controlPlanes = list
+	s.mu.Unlock()
+}
+
+// controlPlanesMsg is the list as an agent is sent it, or nil when there is
+// none to send: a cluster with no list recorded keeps what its agents were
+// configured with.
+func (s *Service) controlPlanesMsg(ctx context.Context) *llmgwv1.ControlMessage {
+	s.mu.RLock()
+	list := s.controlPlanes
+	s.mu.RUnlock()
+	if list == nil {
+		return nil
+	}
+	servers, err := list(ctx)
+	if err != nil || len(servers) == 0 {
+		return nil
+	}
+	return &llmgwv1.ControlMessage{Msg: &llmgwv1.ControlMessage_ControlPlanes{
+		ControlPlanes: &llmgwv1.ControlPlanes{Servers: servers}}}
+}
+
+// WatchControlPlanes tells every connected agent the list when it changes,
+// checking every so often. A control plane added by `llmgw server add` reaches
+// the nodes within that time, without their being touched.
+func (s *Service) WatchControlPlanes(ctx context.Context, every time.Duration) {
+	var last []string
+	t := time.NewTicker(every)
+	defer t.Stop()
+	for {
+		if msg := s.controlPlanesMsg(ctx); msg != nil {
+			now := msg.GetControlPlanes().GetServers()
+			if !slices.Equal(now, last) {
+				for _, node := range s.ConnectedNodes() {
+					if err := s.send(node, msg); err != nil {
+						s.log.Warn("could not tell a node the control planes", "node", node, "err", err)
+					}
+				}
+				if last != nil {
+					s.log.Info("told the nodes the control planes changed", "servers", now)
+				}
+				last = now
+			}
+		}
+		select {
+		case <-ctx.Done():
+			return
+		case <-t.C:
+		}
+	}
+}

internal/control/agentsvc/heartbeat_test.go

@@ -283,3 +283,36 @@ func TestAnUpgradeWaitsForTheAgentsAnswer(t *testing.T) {
 		t.Errorf("the agent's refusal was not passed on: %v", err)
 	}
 }
+
+func TestAnotherTenantOnACardIsJudgedEveryHeartbeat(t *testing.T) {
+	// Found on cuda01: ComfyUI held 5 GB when the agent started, so the card
+	// was kept out of placement. ComfyUI let go of it and the card stayed
+	// out, saying "5014 MiB already allocated by another process", until the
+	// agent was restarted.
+	s := newTestService()
+	nodeWith(s, inventory.Node{
+		ID: "n1", State: inventory.Ready,
+		GPUs: []discovery.GPU{{Index: 0, Health: discovery.GPUHealth{Problems: []string{
+			discovery.ForeignMemoryProblem(5014), "2 uncorrected ECC errors"}}}},
+	})
+
+	s.applyHeartbeat("n1", &llmgwv1.Heartbeat{NodeId: "n1", GpuForeignMemoryMb: []int32{330}})
+	n, _ := s.inv.Get("n1")
+	if got := n.GPUs[0].Health.Problems; len(got) != 1 || got[0] != "2 uncorrected ECC errors" {
+		t.Fatalf("problems are %q: the tenant left, and only the ECC errors should remain", got)
+	}
+
+	// And the other way: a tenant arriving after registration.
+	s.applyHeartbeat("n1", &llmgwv1.Heartbeat{NodeId: "n1", GpuForeignMemoryMb: []int32{12000}})
+	n, _ = s.inv.Get("n1")
+	if got := n.GPUs[0].Health.Problems; len(got) != 2 || got[1] != discovery.ForeignMemoryProblem(12000) {
+		t.Fatalf("problems are %q, want the new tenant's 12000 MiB", got)
+	}
+
+	// An agent that sends no reading changes nothing.
+	s.applyHeartbeat("n1", &llmgwv1.Heartbeat{NodeId: "n1", GpuMemoryUsedMb: []int32{100}})
+	n, _ = s.inv.Get("n1")
+	if len(n.GPUs[0].Health.Problems) != 2 {
+		t.Errorf("a heartbeat with no foreign reading cleared the card: %q", n.GPUs[0].Health.Problems)
+	}
+}

internal/control/agentsvc/installable_test.go

@@ -6,6 +6,7 @@ import (
 
 	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
 )
 
 func TestANodeIsAskedOnlyForEnginesItCanInstall(t *testing.T) {
@@ -25,3 +26,57 @@ func TestANodeIsAskedOnlyForEnginesItCanInstall(t *testing.T) {
 		t.Error("uv, a tool for the others, is offered as an engine")
 	}
 }
+
+func TestEachNodeIsToldTheEnginesDecidedForIt(t *testing.T) {
+	s := &Service{}
+	s.SetEnginesFor(func(n inventory.Node) map[string]string {
+		if n.ID == "volta" {
+			return map[string]string{"llamacpp": "b1", "vllm": "0.25.1"}
+		}
+		return map[string]string{"llamacpp": "b1", "vllm": "0.30.0"}
+	})
+	reg := &llmgwv1.Register{InstallableEngines: []string{"llamacpp", "vllm"}}
+	want := func(p *llmgwv1.Provision) map[string]string {
+		out := map[string]string{}
+		for _, e := range p.GetEngines() {
+			out[e.GetEngine()] = e.GetVersion()
+		}
+		return out
+	}
+	if got := want(s.provisionFor(inventory.Node{ID: "volta"}, installable(reg))); got["vllm"] != "0.25.1" {
+		t.Errorf("the Volta node: %v", got)
+	}
+	if got := want(s.provisionFor(inventory.Node{ID: "hopper"}, installable(reg))); got["vllm"] != "0.30.0" {
+		t.Errorf("the Hopper node: %v", got)
+	}
+	// A node that disabled vLLM does not offer it, and is not asked for it.
+	if got := want(s.provisionFor(inventory.Node{ID: "hopper"}, []string{"llamacpp"})); got["vllm"] != "" {
+		t.Errorf("asked a node for an engine it does not install: %v", got)
+	}
+}
+
+func TestABuildChosenForANodeIsWhatItIsTold(t *testing.T) {
+	s := &Service{}
+	s.SetEngines(map[string]string{"llamacpp": "b1", "vllm": "0.30.0"})
+	s.SetEngineChoices(func(node string) map[string]string {
+		return map[string]map[string]string{"gpu07": {"vllm": "cu129"}, "gpu08": {"vllm": "none"}}[node]
+	})
+	can := []string{"llamacpp", "vllm"}
+	variant := func(node string) (string, bool) {
+		for _, e := range s.provisionFor(inventory.Node{ID: node}, can).GetEngines() {
+			if e.GetEngine() == "vllm" {
+				return e.GetVariant(), true
+			}
+		}
+		return "", false
+	}
+	if v, ok := variant("gpu07"); !ok || v != "cu129" {
+		t.Errorf("gpu07: %q, %v", v, ok)
+	}
+	if _, ok := variant("gpu08"); ok {
+		t.Error("gpu08 was asked for a vLLM it was chosen to run none of")
+	}
+	if v, ok := variant("gpu09"); !ok || v != "" {
+		t.Errorf("gpu09, left to its hardware: %q, %v", v, ok)
+	}
+}

internal/control/agentsvc/service.go

@@ -53,8 +53,16 @@ type Service struct {
 	mu    sync.RWMutex
 	conns map[string]*conn
 
-	// engines is what every node is told to install, set once at start.
-	engines map[string]string
+	// enginesFor is what a node is told to install, set once at start: the
+	// cluster's engines, with any override that picks the node laid over.
+	enginesFor func(node inventory.Node) map[string]string
+	// choices are the engine builds chosen per node, and installableBy what
+	// each connected node said it can install, for telling it again.
+	choices       func(nodeID string) map[string]string
+	installableBy map[string][]string
+
+	// controlPlanes lists what agents may dial; see SetControlPlanes.
+	controlPlanes func(context.Context) ([]string, error)
 
 	// logWaits are the log requests awaiting a reply, by request id.
 	logMu    sync.Mutex
@@ -68,7 +76,37 @@ type Service struct {
 }
 
 // SetEngines sets the engines every node installs, by name and version.
-func (s *Service) SetEngines(e map[string]string) { s.engines = maps.Clone(e) }
+func (s *Service) SetEngines(e map[string]string) {
+	e = maps.Clone(e)
+	s.enginesFor = func(inventory.Node) map[string]string { return e }
+}
+
+// SetEnginesFor sets what each node installs, decided per node: a cluster
+// of two GPU generations needs two vLLM builds, or vLLM on only one.
+func (s *Service) SetEnginesFor(f func(node inventory.Node) map[string]string) { s.enginesFor = f }
+
+// SetEngineChoices sets where the builds chosen for each node come from: by
+// engine, a build ("cu129") or "none". An engine not named is left to the
+// node's hardware.
+func (s *Service) SetEngineChoices(f func(nodeID string) map[string]string) { s.choices = f }
+
+// Reprovision tells a connected node again what to install, after a choice
+// for it changed; one not connected is told when it next connects.
+func (s *Service) Reprovision(nodeID string) error {
+	if s.enginesFor == nil {
+		return nil
+	}
+	s.mu.RLock()
+	can, ok := s.installableBy[nodeID]
+	s.mu.RUnlock()
+	node, known := s.inv.Get(nodeID)
+	if !ok || !known {
+		return nil
+	}
+	return s.send(nodeID, &llmgwv1.ControlMessage{
+		Msg: &llmgwv1.ControlMessage_Provision{Provision: s.provisionFor(node, can)},
+	})
+}
 
 // conn is one agent's open stream.
 type conn struct {
@@ -179,21 +217,28 @@ func (s *Service) Connect(
 	// What this node should have installed. Sent on every registration, so a
 	// node rebuilt from nothing is told the same as one that has been here a
 	// month, and a changed version reaches every node on its next connect.
-	if s.engines != nil {
-		p := &llmgwv1.Provision{}
+	if s.enginesFor != nil {
 		can := installable(reg)
-		for _, e := range slices.Sorted(maps.Keys(s.engines)) {
-			if !slices.Contains(can, e) {
-				continue
-			}
-			p.Engines = append(p.Engines, &llmgwv1.EngineWant{Engine: e, Version: s.engines[e]})
+		s.mu.Lock()
+		if s.installableBy == nil {
+			s.installableBy = map[string][]string{}
 		}
+		s.installableBy[nodeID] = can
+		s.mu.Unlock()
+		p := s.provisionFor(node, can)
 		if err := stream.Send(&llmgwv1.ControlMessage{
 			Msg: &llmgwv1.ControlMessage_Provision{Provision: p},
 		}); err != nil {
 			return err
 		}
 	}
+	// Which control planes there are, so an agent configured with one learns
+	// of a second added since.
+	if msg := s.controlPlanesMsg(ctx); msg != nil {
+		if err := stream.Send(msg); err != nil {
+			return err
+		}
+	}
 
 	// Recover replica state from what this node is already running, before any
 	// reconciliation pass can conclude the cluster is empty and place
@@ -342,11 +387,12 @@ func (s *Service) handle(nodeID string, msg *llmgwv1.AgentMessage) {
 // a reconnect.
 func (s *Service) applyHeartbeat(nodeID string, hb *llmgwv1.Heartbeat) {
 	used := hb.GetGpuMemoryUsedMb()
+	foreign := hb.GetGpuForeignMemoryMb()
 	problems := hb.GetNewProblems()
 	storage := hb.GetStorage()
 	memMB := int(hb.GetMemoryTotalMb())
 	cpus := int(hb.GetCpuCount())
-	if len(used) == 0 && len(problems) == 0 && len(storage) == 0 &&
+	if len(used) == 0 && len(foreign) == 0 && len(problems) == 0 && len(storage) == 0 &&
 		!hb.GetWeightsChanged() && memMB == 0 && cpus == 0 && !hb.GetEnginesChanged() &&
 		!hb.GetProvisionChanged() && hb.GetLoad() == nil && len(hb.GetIbPorts()) == 0 &&
 		!hb.GetFabricsChanged() && len(hb.GetGpuFabric()) == 0 && hb.GetImex() == nil {
@@ -362,6 +408,20 @@ func (s *Service) applyHeartbeat(nodeID string, hb *llmgwv1.Heartbeat) {
 			n.GPUs[i].MemoryUsedMB = int(used[i])
 		}
 	}
+	// Another tenant on a card, judged afresh: the reading at registration is
+	// replaced, both ways. Without this a card that had something on it when
+	// its agent started stayed out of placement after it left, and said so,
+	// until the agent was restarted. Only a reading for every card replaces
+	// it; an agent that sends none, or could not measure, leaves it standing.
+	if len(foreign) == len(n.GPUs) {
+		for i := range n.GPUs {
+			ps := slices.DeleteFunc(slices.Clone(n.GPUs[i].Health.Problems), discovery.IsForeignMemoryProblem)
+			if p := discovery.ForeignMemoryProblem(int(foreign[i])); p != "" {
+				ps = append(ps, p)
+			}
+			n.GPUs[i].Health.Problems = ps
+		}
+	}
 	if len(problems) > 0 {
 		// Problems are reported per node, not per GPU, so attach them to the
 		// node's first GPU rather than guessing which one is affected. Any
@@ -550,6 +610,10 @@ func loadFrom(l *llmgwv1.NodeLoad, at time.Time) *inventory.Load {
 		out.GPUs = append(out.GPUs, inventory.GPUSensor{
 			Index: int(g.GetIndex()), TempC: g.GetTempC(), MemoryTempC: g.GetMemoryTempC(),
 			PowerW: g.GetPowerW(), PowerLimitW: g.GetPowerLimitW(), Throttle: g.GetThrottle(),
+			UtilPercent: g.GetUtilizationPercent(), MemUtilPercent: g.GetMemoryUtilizationPercent(),
+			MemoryUsedMB: g.GetMemoryUsedMb(), MemoryTotalMB: g.GetMemoryTotalMb(),
+			SMClockMHz: g.GetSmClockMhz(), ECCUncorrected: g.GetEccUncorrected(),
+			Wear: wearFrom(g),
 		})
 	}
 	return out
@@ -853,6 +917,24 @@ func rate(before, now uint64, secs float64) float64 {
 	return float64(now-before) / secs
 }
 
+// provisionFor is what a node is told to install: the engines decided for
+// it, those it can install.
+func (s *Service) provisionFor(node inventory.Node, can []string) *llmgwv1.Provision {
+	p := &llmgwv1.Provision{}
+	wants := s.enginesFor(node)
+	var chosen map[string]string
+	if s.choices != nil {
+		chosen = s.choices(node.ID)
+	}
+	for _, e := range slices.Sorted(maps.Keys(wants)) {
+		if !slices.Contains(can, e) || chosen[e] == "none" {
+			continue
+		}
+		p.Engines = append(p.Engines, &llmgwv1.EngineWant{Engine: e, Version: wants[e], Variant: chosen[e]})
+	}
+	return p
+}
+
 // installable are the engines a node can be asked to install. An agent older
 // than the list said nothing, and could install llama.cpp alone: asked for
 // vLLM too, it reported it unsupported, and a cluster half way through an
@@ -908,3 +990,22 @@ func (s *Service) Upgrade(ctx context.Context, nodeID, version, releases string)
 		return fmt.Errorf("node %s did not answer the upgrade; an agent older than v0.10.4 cannot upgrade itself", nodeID)
 	}
 }
+
+// wearFrom is a card's wear, when the agent read it.
+func wearFrom(g *llmgwv1.GPUSensor) *inventory.GPUWear {
+	if !g.GetHasWear() {
+		return nil
+	}
+	w := &inventory.GPUWear{
+		PCIeReplays: g.GetPcieReplays(), RetiredSBE: g.GetRetiredPagesSbe(), RetiredDBE: g.GetRetiredPagesDbe(),
+		RetirePending: g.GetRetirementPending(), RemappedCorrectable: g.GetRemappedRowsCorrectable(),
+		RemappedUncorrectable: g.GetRemappedRowsUncorrectable(), RemapPending: g.GetRemapPending(),
+		RemapFailure: g.GetRemapFailure(), NVLinkCRC: g.GetNvlinkCrcErrors(), NVLinkReplay: g.GetNvlinkReplayErrors(),
+		NVLinkRecovery: g.GetNvlinkRecoveryErrors(), XIDs: g.GetXidErrors(), LastXID: int(g.GetLastXid()),
+		LastXIDText: g.GetLastXidText(),
+	}
+	if u := g.GetLastXidUnix(); u > 0 {
+		w.LastXIDAt = time.Unix(u, 0)
+	}
+	return w
+}

internal/control/catalog/catalog.go

@@ -305,6 +305,16 @@ func (m Model) normalize() Model {
 		if p.Nodes <= 0 {
 			p.Nodes = 1
 		}
+		// tp on a llama.cpp GPU profile, written before gpus_per_node meant
+		// anything for it, is read as the width it was meant as: N cards of
+		// one node. Refusing it would have dropped a catalog's model on the
+		// upgrade that made the width work. Only both given, disagreeing, is
+		// refused below, as ambiguous.
+		if p.Engine == "llamacpp" && p.GPUClass != "" && (p.TP > 1 || p.PP > 1) && p.Nodes == 1 {
+			if n := p.TP * max(p.PP, 1); p.GPUsPerNode == 0 || p.GPUsPerNode == n {
+				p.GPUsPerNode, p.TP, p.PP = n, 1, 0
+			}
+		}
 		switch {
 		case p.CPUOnly():
 			// Nothing to divide up: this profile asks for no cards at all.
@@ -323,6 +333,13 @@ func (m Model) normalize() Model {
 			p.Weight = 1
 		}
 	}
+	// A Hugging Face repository page for a model vLLM or SGLang runs is the
+	// repository it names: the engine downloads a repository id itself,
+	// where the page was fetched as one file of HTML and failed at start.
+	if repo, ok := repositoryPage(m.Source); ok && len(m.Profiles) > 0 &&
+		!slices.ContainsFunc(m.Profiles, func(p Profile) bool { return p.Engine == "llamacpp" }) {
+		m.Source = repo
+	}
 	return m
 }
 
@@ -333,6 +350,15 @@ func (m Model) validate() error {
 	if m.Source == "" {
 		return fmt.Errorf("model %q: source is required", m.Name)
 	}
+	if repo, ok := repositoryPage(m.Source); ok {
+		// A page, not weights: a node fetches the HTML as one file and the
+		// engine fails at start, on a path that looks right. For vLLM it was
+		// read as the repository id when the model was made; only a model
+		// llama.cpp runs is left here, which needs a file.
+		return fmt.Errorf("model %q: source %s is a Hugging Face repository page, not a file; "+
+			"llama.cpp needs a .gguf: mirror one and use its source (repository %s)",
+			m.Name, m.Source, repo)
+	}
 	if err := validateAdapters(m); err != nil {
 		return err
 	}
@@ -381,6 +407,22 @@ func (m Model) validate() error {
 			}
 			continue
 		}
+		// llama.cpp on GPUs splits a model across one node's cards by layer,
+		// and how many is gpus_per_node. Not tp: llama.cpp has no tensor
+		// parallelism, and a tp it was given was a width nothing placed — the
+		// replica stayed unplaced, or ran on one card while four were meant.
+		if p.Engine == "llamacpp" {
+			switch {
+			case p.Nodes > 1:
+				return fmt.Errorf("model %q profile %q: llama.cpp does not span nodes; use vllm or sglang for a %d-node replica",
+					m.Name, p.Name, p.Nodes)
+			case p.TP > 1 || p.PP > 1:
+				return fmt.Errorf("model %q profile %q: llama.cpp splits a model across a node's GPUs by layer, and "+
+					"tp %d does not agree with gpus_per_node %d: give gpus_per_node alone", m.Name, p.Name,
+					max(p.TP, 1)*max(p.PP, 1), p.GPUsPerNode)
+			}
+			continue
+		}
 		// The arithmetic has to add up, or the engine is handed a shard spec it
 		// cannot satisfy and fails minutes later during model loading.
 		if want := p.GPUsPerNode * p.Nodes; want != p.TP*max(p.PP, 1) {
@@ -704,3 +746,21 @@ func (m Model) engineAdapters() []engines.Adapter {
 	}
 	return out
 }
+
+// repositoryPage reports whether source is a Hugging Face repository's page
+// (https://huggingface.co/org/repo, or one of its tree views) rather than a
+// file under /resolve/, and the repository's id.
+func repositoryPage(source string) (string, bool) {
+	for _, host := range []string{"https://huggingface.co/", "http://huggingface.co/", "https://hf.co/"} {
+		rest, ok := strings.CutPrefix(source, host)
+		if !ok {
+			continue
+		}
+		parts := strings.Split(strings.Trim(rest, "/"), "/")
+		if len(parts) < 2 || (len(parts) > 2 && parts[2] == "resolve") {
+			return "", false
+		}
+		return parts[0] + "/" + parts[1], true
+	}
+	return "", false
+}

internal/control/catalog/catalog_test.go

@@ -321,3 +321,67 @@ func TestAProfileCannotSetWhatRunsCodeInTheEngine(t *testing.T) {
 		t.Errorf("ordinary tuning refused: %v", err)
 	}
 }
+
+func TestALlamaCppReplicaOnSeveralGPUsSaysHowManyAsGPUsPerNode(t *testing.T) {
+	four := model("big", Profile{Name: "t4", Engine: "llamacpp", GPUClass: "t4", GPUsPerNode: 4})
+	c, err := New([]Model{four})
+	if err != nil {
+		t.Fatalf("gpus_per_node 4 on llama.cpp: %v", err)
+	}
+	m, _ := c.Resolve("big")
+	if pl := m.PlacementProfiles(); len(pl) != 1 || pl[0].GPUsPerNode != 4 {
+		t.Errorf("placement was not asked for four GPUs: %+v", pl)
+	}
+	// tp, written before gpus_per_node meant anything for llama.cpp, is read
+	// as the same width rather than refused: refusing dropped the model on
+	// the upgrade that made the width work.
+	old := model("old", Profile{Name: "t4", Engine: "llamacpp", GPUClass: "t4", TP: 4})
+	c, err = New([]Model{old})
+	if err != nil {
+		t.Fatalf("tp 4 on llama.cpp: %v", err)
+	}
+	got, _ := c.Resolve("old")
+	if p := got.PlacementProfiles()[0]; p.GPUsPerNode != 4 {
+		t.Errorf("tp 4 should place on four GPUs: %+v", p)
+	}
+	// Both given and disagreeing is ambiguous, and refused; so is more than
+	// one node.
+	for _, p := range []Profile{
+		{Name: "both", Engine: "llamacpp", GPUClass: "t4", TP: 4, GPUsPerNode: 2},
+		{Name: "nodes", Engine: "llamacpp", GPUClass: "t4", Nodes: 2, GPUsPerNode: 1},
+	} {
+		if _, err := New([]Model{model("x", p)}); err == nil {
+			t.Errorf("%s: accepted", p.Name)
+		}
+	}
+}
+
+func TestARepositoryPageIsReadAsTheRepositoryForVLLM(t *testing.T) {
+	// The field: an AWQ model's page given as a vLLM source, fetched as one
+	// file of HTML. vLLM downloads a repository id itself.
+	for _, src := range []string{
+		"https://huggingface.co/Qwen/Qwen2.5-32B-Instruct-AWQ",
+		"https://huggingface.co/Qwen/Qwen2.5-32B-Instruct-AWQ/tree/main",
+	} {
+		m := model("awq")
+		m.Source = src
+		c, err := New([]Model{m})
+		if err != nil {
+			t.Fatalf("%s: %v", src, err)
+		}
+		if got, _ := c.Resolve("awq"); got.Source != "Qwen/Qwen2.5-32B-Instruct-AWQ" {
+			t.Errorf("%s became %q", src, got.Source)
+		}
+	}
+	// llama.cpp needs a file, which a page is not.
+	g := model("gguf", Profile{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 4})
+	g.Source = "https://huggingface.co/Qwen/X-GGUF"
+	if _, err := New([]Model{g}); err == nil || !strings.Contains(err.Error(), "needs a .gguf") {
+		t.Errorf("a page for llama.cpp: %v", err)
+	}
+	f := model("file")
+	f.Source = "https://huggingface.co/Qwen/X-GGUF/resolve/abc123/x.gguf"
+	if _, err := New([]Model{f}); err != nil {
+		t.Errorf("a file under /resolve/ is a source: %v", err)
+	}
+}

internal/control/control.go

@@ -75,15 +75,18 @@ func DefaultConfig() Config {
 
 // Controller owns the control plane's state.
 type Controller struct {
-	cfg   Config
-	log   *slog.Logger
-	inv   *inventory.Store
-	cat   *catalog.Live
-	rec   *reconciler.Reconciler
-	lead  *Leadership
-	disp  reconciler.Dispatcher
-	sink  EndpointSink
-	usage UsageSource
+	// engineChoices are the engine builds chosen per node; nil without a
+	// store to keep them in.
+	engineChoices *EngineChoices
+	cfg           Config
+	log           *slog.Logger
+	inv           *inventory.Store
+	cat           *catalog.Live
+	rec           *reconciler.Reconciler
+	lead          *Leadership
+	disp          reconciler.Dispatcher
+	sink          EndpointSink
+	usage         UsageSource
 
 	mu   sync.Mutex
 	last string // fingerprint of the last published list
@@ -408,3 +411,23 @@ func EngineKeys(secret []byte) func(replicaID string) string {
 
 // QuotaUse is every capacity quota beside what its scope holds.
 func (c *Controller) QuotaUse() []reconciler.QuotaUse { return c.rec.QuotaUse() }
+
+// SetEngineChoices gives the controller the engine builds chosen per node.
+func (c *Controller) SetEngineChoices(e *EngineChoices) { c.engineChoices = e }
+
+// EngineChoices are the engine builds chosen per node, or nil.
+func (c *Controller) EngineChoices() *EngineChoices { return c.engineChoices }
+
+// ModelOwners maps each model to the part of the tenancy tree its capacity
+// counts against; a model owned by nobody is not in it.
+func (c *Controller) ModelOwners() map[string]string {
+	out := map[string]string{}
+	if cat := c.Catalog(); cat != nil {
+		for _, m := range cat.Models() {
+			if m.Owner != "" {
+				out[m.Name] = m.Owner
+			}
+		}
+	}
+	return out
+}

internal/control/enginechoice.go

@@ -0,0 +1,108 @@
+package control
+
+import (
+	"context"
+	"fmt"
+	"log/slog"
+	"maps"
+	"slices"
+	"strings"
+	"sync"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
+)
+
+// EngineChoiceStore is where the engine builds chosen per node are kept.
+type EngineChoiceStore interface {
+	NodeEngines(ctx context.Context) (map[string]map[string]string, error)
+	SetNodeEngine(ctx context.Context, node, engine, variant, by string) error
+}
+
+// EngineChoices are the engine builds chosen for nodes, rather than left to
+// their hardware: vLLM's CUDA 12.9 build on a node its CUDA 13 build would
+// also install on, or no vLLM at all. Kept in the store, held here for the
+// agent service to ask on every registration, and pushed to a node the
+// moment its choice changes.
+type EngineChoices struct {
+	st  EngineChoiceStore
+	log *slog.Logger
+
+	mu     sync.RWMutex
+	byNode map[string]map[string]string
+	notify func(node string) error
+}
+
+// NewEngineChoices reads what was chosen before.
+func NewEngineChoices(ctx context.Context, st EngineChoiceStore, log *slog.Logger) *EngineChoices {
+	c := &EngineChoices{st: st, log: log, byNode: map[string]map[string]string{}}
+	if got, err := st.NodeEngines(ctx); err != nil {
+		log.Error("could not read the engine builds chosen per node; every node installs what its hardware runs", "err", err)
+	} else {
+		c.byNode = got
+	}
+	return c
+}
+
+// OnChange is called with a node whose choice changed, to tell it.
+func (c *EngineChoices) OnChange(f func(node string) error) {
+	c.mu.Lock()
+	c.notify = f
+	c.mu.Unlock()
+}
+
+// For is what was chosen for one node, by engine: a build, or "none".
+func (c *EngineChoices) For(node string) map[string]string {
+	c.mu.RLock()
+	defer c.mu.RUnlock()
+	return maps.Clone(c.byNode[node])
+}
+
+// EngineBuilds are what can be chosen for each engine that has a choice:
+// "auto" is the node's hardware deciding, "none" no engine at all.
+func EngineBuilds(engine string) []string {
+	switch engine {
+	case "vllm":
+		return append(append([]string{"auto"}, provision.VLLMVariants...), "none")
+	case "sglang":
+		return []string{"auto", "none"}
+	}
+	return nil
+}
+
+// Set chooses an engine's build for a node, keeps it, and tells the node.
+func (c *EngineChoices) Set(ctx context.Context, node, engine, build, by string) error {
+	builds := EngineBuilds(engine)
+	if builds == nil {
+		return fmt.Errorf("%s has no builds to choose between", engine)
+	}
+	if !slices.Contains(builds, build) {
+		return fmt.Errorf("%s has no build %q; choose %s", engine, build, strings.Join(builds, ", "))
+	}
+	variant := build
+	if build == "auto" {
+		variant = ""
+	}
+	if err := c.st.SetNodeEngine(ctx, node, engine, variant, by); err != nil {
+		return err
+	}
+	c.mu.Lock()
+	if variant == "" {
+		delete(c.byNode[node], engine)
+	} else {
+		if c.byNode[node] == nil {
+			c.byNode[node] = map[string]string{}
+		}
+		c.byNode[node][engine] = variant
+	}
+	notify := c.notify
+	c.mu.Unlock()
+	c.log.Info("engine build chosen for a node", "node", node, "engine", engine, "build", build, "by", by)
+	if notify != nil {
+		if err := notify(node); err != nil {
+			// Kept and applied on its next connection; said, since a node that
+			// is not connected is the usual reason.
+			c.log.Warn("the node was not told yet; it is when it next connects", "node", node, "err", err)
+		}
+	}
+	return nil
+}

internal/control/fleetmetrics/fleetmetrics.go

@@ -73,6 +73,18 @@ type Collector struct {
 	gpuMemoryTotal *prometheus.Desc
 	gpuMemoryUsed  *prometheus.Desc
 	gpuUtil        *prometheus.Desc
+	gpuAssigned    *prometheus.Desc
+	gpuPCIeReplay  *prometheus.Desc
+	gpuRetired     *prometheus.Desc
+	gpuRetirePend  *prometheus.Desc
+	gpuRemapped    *prometheus.Desc
+	gpuRemapPend   *prometheus.Desc
+	gpuRemapFail   *prometheus.Desc
+	gpuNVLinkErr   *prometheus.Desc
+	gpuXID         *prometheus.Desc
+	gpuLastXID     *prometheus.Desc
+	gpuMemUtil     *prometheus.Desc
+	gpuSMClock     *prometheus.Desc
 	gpuTemp        *prometheus.Desc
 	gpuMemTemp     *prometheus.Desc
 	gpuPower       *prometheus.Desc
@@ -149,6 +161,33 @@ func New(src Source) *Collector {
 			"How busy a GPU was over the last sample period. Absent when the driver does not "+
 				"report it, which is not the same as idle. Memory used says a model is loaded; "+
 				"this says whether it is doing anything.", gpu, nil),
+		gpuPCIeReplay: prometheus.NewDesc("llmgw_gpu_pcie_replays",
+			"PCIe link retries on a GPU's link since the driver loaded; one that keeps climbing is a riser, slot or cable.", gpu, nil),
+		gpuRetired: prometheus.NewDesc("llmgw_gpu_retired_pages",
+			"Memory pages retired, by cause (sbe, dbe), on GPUs before Ampere.", append(gpu, "cause"), nil),
+		gpuRetirePend: prometheus.NewDesc("llmgw_gpu_retirement_pending",
+			"1 when retired pages wait for a GPU reset to take effect.", gpu, nil),
+		gpuRemapped: prometheus.NewDesc("llmgw_gpu_remapped_rows",
+			"Memory rows remapped, by kind (correctable, uncorrectable), on Ampere and later.", append(gpu, "kind"), nil),
+		gpuRemapPend: prometheus.NewDesc("llmgw_gpu_remap_pending",
+			"1 when remapped rows wait for a GPU reset to take effect.", gpu, nil),
+		gpuRemapFail: prometheus.NewDesc("llmgw_gpu_remap_failure",
+			"1 when a GPU has run out of spare memory rows: it needs replacing.", gpu, nil),
+		gpuNVLinkErr: prometheus.NewDesc("llmgw_gpu_nvlink_errors",
+			"NVLink errors summed over a GPU's links, by kind (crc, replay, recovery).", append(gpu, "kind"), nil),
+		gpuXID: prometheus.NewDesc("llmgw_gpu_xid_errors",
+			"The driver's XID error events for a GPU since its node's agent started.", gpu, nil),
+		gpuLastXID: prometheus.NewDesc("llmgw_gpu_last_xid",
+			"The code of a GPU's last XID error since its agent started; absent when there was none.", gpu, nil),
+		gpuAssigned: prometheus.NewDesc("llmgw_gpu_assigned",
+			"1 for each GPU a replica holds: which model, which replica, and whose quota it counts against. "+
+				"Joined on node and gpu with the other llmgw_gpu_ series, it says what each model's cards are doing.",
+			[]string{"node", "gpu", "model", "replica", "owner"}, nil),
+		gpuMemUtil: prometheus.NewDesc("llmgw_gpu_memory_utilization_percent",
+			"How much of the sample period a GPU's memory was being read or written: "+
+				"decode is bound by it, so a card busy here and idle above is waiting on memory.", gpu, nil),
+		gpuSMClock: prometheus.NewDesc("llmgw_gpu_sm_clock_mhz",
+			"A GPU's streaming multiprocessor clock now; one held low under load is the card being throttled.", gpu, nil),
 		gpuMemTemp: prometheus.NewDesc("llmgw_gpu_memory_temperature_celsius",
 			"A GPU's memory (HBM) temperature, where the driver reports it.", gpu, nil),
 		gpuPower: prometheus.NewDesc("llmgw_gpu_power_watts",
@@ -271,7 +310,9 @@ func (c *Collector) Describe(ch chan<- *prometheus.Desc) {
 		c.nodeCPU, c.nodeCPUs, c.nodeLoad, c.nodeMemUsed, c.nodeMemAvail, c.shardCPU, c.shardMemory,
 		c.ibActive, c.ibRate, c.ibCounter, c.fabricSMs, c.fabricMaster, c.fabricProblems,
 		c.gpuNVLJoined, c.imexUp,
-		c.gpuMemoryTotal, c.gpuMemoryUsed, c.gpuUtil, c.gpuTemp, c.gpuHealthy,
+		c.gpuMemoryTotal, c.gpuMemoryUsed, c.gpuUtil, c.gpuMemUtil, c.gpuSMClock, c.gpuAssigned, c.gpuTemp, c.gpuHealthy,
+		c.gpuPCIeReplay, c.gpuRetired, c.gpuRetirePend, c.gpuRemapped, c.gpuRemapPend, c.gpuRemapFail,
+		c.gpuNVLinkErr, c.gpuXID, c.gpuLastXID,
 		c.gpuMemTemp, c.gpuPower, c.gpuPowerLimit, c.gpuThrottled, c.nodeCPUTemp, c.nodeCPUPower, c.nodeInletTemp,
 		c.gpuNVLink, c.gpuECC,
 		c.replicaState, c.replicaReady, c.replicaIdle,
@@ -287,6 +328,7 @@ func (c *Collector) Describe(ch chan<- *prometheus.Desc) {
 func (c *Collector) Collect(ch chan<- prometheus.Metric) {
 	now := time.Now()
 	c.collectQuotas(ch)
+	c.collectAssignments(ch)
 
 	loads := map[string]*inventory.Load{}
 	for _, n := range c.src.Nodes() {
@@ -388,20 +430,28 @@ func (c *Collector) Collect(ch chan<- prometheus.Metric) {
 			byClass[class]++
 			labels := []string{n.ID, strconv.Itoa(g.Index), class}
 
-			ch <- prometheus.MustNewConstMetric(c.gpuMemoryTotal, prometheus.GaugeValue,
-				float64(g.MemoryTotalMB)*1024*1024, labels...)
-			ch <- prometheus.MustNewConstMetric(c.gpuMemoryUsed, prometheus.GaugeValue,
-				float64(g.MemoryUsedMB)*1024*1024, labels...)
-			if g.UtilizationPercent > 0 {
-				// Only when reported. A driver too old to give it would
-				// otherwise show every card at a flat zero, which reads as an
-				// idle fleet rather than a missing measurement.
-				ch <- prometheus.MustNewConstMetric(c.gpuUtil, prometheus.GaugeValue,
-					float64(g.UtilizationPercent), labels...)
+			s, fresh := sensorFor(n.Load, g.Index, now)
+			if fresh && s.Busy() {
+				// What the card is doing now, sampled every heartbeat.
+				c.gpuBusy(ch, s, labels)
+			} else {
+				// An agent too old to sample it: what it read at
+				// registration, which is all there is.
+				ch <- prometheus.MustNewConstMetric(c.gpuMemoryTotal, prometheus.GaugeValue,
+					float64(g.MemoryTotalMB)*1024*1024, labels...)
+				ch <- prometheus.MustNewConstMetric(c.gpuMemoryUsed, prometheus.GaugeValue,
+					float64(g.MemoryUsedMB)*1024*1024, labels...)
+				if g.UtilizationPercent > 0 {
+					// Only when reported. A driver too old to give it would
+					// otherwise show every card at a flat zero, which reads as
+					// an idle fleet rather than a missing measurement.
+					ch <- prometheus.MustNewConstMetric(c.gpuUtil, prometheus.GaugeValue,
+						float64(g.UtilizationPercent), labels...)
+				}
 			}
 			// The sensors' fresh reading when there is one; the temperature
 			// read at registration otherwise, which is all an older agent sends.
-			if s, ok := sensorFor(n.Load, g.Index, now); ok {
+			if fresh {
 				c.gpuSensor(ch, s, labels)
 			} else if g.Health.TemperatureC > 0 {
 				ch <- prometheus.MustNewConstMetric(c.gpuTemp, prometheus.GaugeValue,
@@ -414,8 +464,11 @@ func (c *Collector) Collect(ch chan<- prometheus.Metric) {
 			ch <- prometheus.MustNewConstMetric(c.gpuHealthy, prometheus.GaugeValue, healthy, labels...)
 			ch <- prometheus.MustNewConstMetric(c.gpuNVLink, prometheus.GaugeValue,
 				float64(len(g.NVLinkPeers)), labels...)
-			ch <- prometheus.MustNewConstMetric(c.gpuECC, prometheus.GaugeValue,
-				float64(g.Health.ECCErrors), labels...)
+			ecc := float64(g.Health.ECCErrors)
+			if fresh && s.Busy() && s.ECCUncorrected >= 0 {
+				ecc = s.ECCUncorrected
+			}
+			ch <- prometheus.MustNewConstMetric(c.gpuECC, prometheus.GaugeValue, ecc, labels...)
 		}
 		for class, n2 := range byClass {
 			ch <- prometheus.MustNewConstMetric(c.nodeGPUs, prometheus.GaugeValue,
@@ -521,6 +574,12 @@ func b2f(b bool) float64 {
 	return 0
 }
 
+// OwnerSource says whose quota each model counts against, for labelling
+// what its replicas hold. Asked for by type assertion.
+type OwnerSource interface {
+	ModelOwners() map[string]string
+}
+
 // QuotaSource adds the tenancy tree's capacity quotas. Asked for by type
 // assertion, like HealthSource.
 type QuotaSource interface {
@@ -569,6 +628,23 @@ func sensorFor(l *inventory.Load, index int, now time.Time) (inventory.GPUSensor
 	return inventory.GPUSensor{}, false
 }
 
+// gpuBusy is how busy a card is, from a sample that carries it.
+func (c *Collector) gpuBusy(ch chan<- prometheus.Metric, s inventory.GPUSensor, labels []string) {
+	ch <- prometheus.MustNewConstMetric(c.gpuMemoryTotal, prometheus.GaugeValue, s.MemoryTotalMB*1024*1024, labels...)
+	ch <- prometheus.MustNewConstMetric(c.gpuMemoryUsed, prometheus.GaugeValue, s.MemoryUsedMB*1024*1024, labels...)
+	// Zero is a real reading here — an idle card — and only -1, the driver
+	// not saying, leaves the series out.
+	if s.UtilPercent >= 0 {
+		ch <- prometheus.MustNewConstMetric(c.gpuUtil, prometheus.GaugeValue, s.UtilPercent, labels...)
+	}
+	if s.MemUtilPercent >= 0 {
+		ch <- prometheus.MustNewConstMetric(c.gpuMemUtil, prometheus.GaugeValue, s.MemUtilPercent, labels...)
+	}
+	if s.SMClockMHz > 0 {
+		ch <- prometheus.MustNewConstMetric(c.gpuSMClock, prometheus.GaugeValue, s.SMClockMHz, labels...)
+	}
+}
+
 func (c *Collector) gpuSensor(ch chan<- prometheus.Metric, s inventory.GPUSensor, labels []string) {
 	for _, v := range []struct {
 		d   *prometheus.Desc
@@ -581,4 +657,61 @@ func (c *Collector) gpuSensor(ch chan<- prometheus.Metric, s inventory.GPUSensor
 	for _, reason := range s.Throttle {
 		ch <- prometheus.MustNewConstMetric(c.gpuThrottled, prometheus.GaugeValue, 1, append(labels, reason)...)
 	}
+	if w := s.Wear; w != nil {
+		c.gpuWear(ch, w, labels)
+	}
+}
+
+// gpuWear is what wears a card out. A count the card does not keep (-1) is
+// left out, not published as zero.
+func (c *Collector) gpuWear(ch chan<- prometheus.Metric, w *inventory.GPUWear, labels []string) {
+	gauge := func(d *prometheus.Desc, v float64, extra ...string) {
+		ch <- prometheus.MustNewConstMetric(d, prometheus.GaugeValue, v, append(append([]string{}, labels...), extra...)...)
+	}
+	flag := func(b bool) float64 {
+		if b {
+			return 1
+		}
+		return 0
+	}
+	if w.PCIeReplays >= 0 {
+		gauge(c.gpuPCIeReplay, float64(w.PCIeReplays))
+	}
+	if w.RetiredSBE >= 0 || w.RetiredDBE >= 0 {
+		gauge(c.gpuRetired, float64(max(w.RetiredSBE, 0)), "sbe")
+		gauge(c.gpuRetired, float64(max(w.RetiredDBE, 0)), "dbe")
+		gauge(c.gpuRetirePend, flag(w.RetirePending))
+	}
+	if w.RemappedCorrectable >= 0 || w.RemappedUncorrectable >= 0 {
+		gauge(c.gpuRemapped, float64(max(w.RemappedCorrectable, 0)), "correctable")
+		gauge(c.gpuRemapped, float64(max(w.RemappedUncorrectable, 0)), "uncorrectable")
+		gauge(c.gpuRemapPend, flag(w.RemapPending))
+		gauge(c.gpuRemapFail, flag(w.RemapFailure))
+	}
+	gauge(c.gpuNVLinkErr, float64(w.NVLinkCRC), "crc")
+	gauge(c.gpuNVLinkErr, float64(w.NVLinkReplay), "replay")
+	gauge(c.gpuNVLinkErr, float64(w.NVLinkRecovery), "recovery")
+	gauge(c.gpuXID, float64(w.XIDs))
+	if w.XIDs > 0 {
+		gauge(c.gpuLastXID, float64(w.LastXID))
+	}
+}
+
+// collectAssignments says which replica holds each GPU.
+func (c *Collector) collectAssignments(ch chan<- prometheus.Metric) {
+	var owners map[string]string
+	if o, ok := c.src.(OwnerSource); ok {
+		owners = o.ModelOwners()
+	}
+	for _, r := range c.src.Replicas() {
+		if !r.State.Counts() {
+			continue
+		}
+		for _, sh := range r.Shards {
+			for _, g := range sh.GPUIndices {
+				ch <- prometheus.MustNewConstMetric(c.gpuAssigned, prometheus.GaugeValue, 1,
+					sh.NodeID, strconv.Itoa(g), r.Model, r.ID, owners[r.Model])
+			}
+		}
+	}
 }

internal/control/fleetmetrics/sensors_test.go

@@ -7,6 +7,7 @@ import (
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
 )
 
 func TestSensorsArePublishedFreshAndOnlyWhenMeasured(t *testing.T) {
@@ -39,3 +40,90 @@ func TestSensorsArePublishedFreshAndOnlyWhenMeasured(t *testing.T) {
 		}
 	}
 }
+
+func TestHowBusyAGPUIsComesFromTheLatestSampleNotRegistration(t *testing.T) {
+	// Discovery read 87% and 70 GB used when the agent came up; the card has
+	// been idle since. The sample says so, zero included.
+	reg := discovery.GPU{Index: 0, Name: "NVIDIA H100 80GB HBM3", MemoryTotalMB: 81559,
+		MemoryUsedMB: 70000, UtilizationPercent: 87}
+	n := inventory.Node{ID: "g1", State: inventory.Ready, GPUs: []discovery.GPU{reg},
+		Load: &inventory.Load{At: time.Now(), GPUs: []inventory.GPUSensor{{Index: 0, TempC: 35,
+			UtilPercent: 0, MemUtilPercent: 0, MemoryUsedMB: 512, MemoryTotalMB: 81559,
+			SMClockMHz: 345, ECCUncorrected: 2}}}}
+	out := gather(t, fakeSource{nodes: []inventory.Node{n}})
+	for _, want := range []string{
+		"llmgw_gpu_utilization_percent class=h100-80g gpu=0 node=g1 -> 0\n",
+		"llmgw_gpu_memory_utilization_percent class=h100-80g gpu=0 node=g1 -> 0\n",
+		"llmgw_gpu_memory_used_bytes class=h100-80g gpu=0 node=g1 -> 536870912\n",
+		"llmgw_gpu_sm_clock_mhz class=h100-80g gpu=0 node=g1 -> 345\n",
+		"llmgw_gpu_ecc_errors class=h100-80g gpu=0 node=g1 -> 2\n",
+	} {
+		if !strings.Contains(out, want) {
+			t.Errorf("missing %s in:\n%s", want, out)
+		}
+	}
+	if strings.Contains(out, "-> 87\n") {
+		t.Error("the utilization read at registration was published over the sample's")
+	}
+
+	// An agent too old to sample it: registration's figures, and no zero
+	// standing in for a utilization the driver did not give.
+	n.Load.GPUs = []inventory.GPUSensor{{Index: 0, TempC: 35}}
+	out = gather(t, fakeSource{nodes: []inventory.Node{n}})
+	if !strings.Contains(out, "llmgw_gpu_utilization_percent class=h100-80g gpu=0 node=g1 -> 87") {
+		t.Errorf("an older agent's registration figure was dropped:\n%s", out)
+	}
+	n.Load.GPUs = []inventory.GPUSensor{{Index: 0, TempC: 35, UtilPercent: -1, MemUtilPercent: -1, MemoryTotalMB: 81559}}
+	if out := gather(t, fakeSource{nodes: []inventory.Node{n}}); strings.Contains(out, "llmgw_gpu_utilization_percent ") {
+		t.Errorf("a utilization the driver did not report was published:\n%s", out)
+	}
+}
+
+// ownedSource adds owners to fakeSource.
+type ownedSource struct{ fakeSource }
+
+func (ownedSource) ModelOwners() map[string]string { return map[string]string{"chat": "acme/ml"} }
+
+func TestEachGPUAReplicaHoldsSaysWhoseItIs(t *testing.T) {
+	src := ownedSource{fakeSource{replicas: []reconciler.Replica{
+		{ID: "chat-1", Model: "chat", State: reconciler.Ready, Shards: []reconciler.Shard{{NodeID: "g1", GPUIndices: []int{0, 1}}}},
+		{ID: "old-1", Model: "old", State: reconciler.Stopped, Shards: []reconciler.Shard{{NodeID: "g1", GPUIndices: []int{2}}}},
+	}}}
+	out := gather(t, src)
+	for _, want := range []string{
+		"llmgw_gpu_assigned gpu=0 model=chat node=g1 owner=acme/ml replica=chat-1 -> 1",
+		"llmgw_gpu_assigned gpu=1 model=chat node=g1 owner=acme/ml replica=chat-1 -> 1",
+	} {
+		if !strings.Contains(out, want) {
+			t.Errorf("missing %s in:\n%s", want, out)
+		}
+	}
+	if strings.Contains(out, "replica=old-1") {
+		t.Error("a stopped replica's GPU was counted as held")
+	}
+}
+
+func TestAGPUsWearIsPublishedAndWhatItDoesNotKeepIsLeftOut(t *testing.T) {
+	n := inventory.Node{ID: "g1", State: inventory.Ready,
+		GPUs: []discovery.GPU{{Index: 0, Name: "NVIDIA H100 80GB HBM3", MemoryTotalMB: 81559}},
+		Load: &inventory.Load{At: time.Now(), GPUs: []inventory.GPUSensor{{Index: 0, TempC: 40, MemoryTotalMB: 81559,
+			Wear: &inventory.GPUWear{PCIeReplays: 12, RetiredSBE: -1, RetiredDBE: -1, RemappedCorrectable: 2,
+				RemappedUncorrectable: 1, RemapPending: true, NVLinkCRC: 7, XIDs: 3, LastXID: 79}}}}}
+	out := gather(t, fakeSource{nodes: []inventory.Node{n}})
+	for _, want := range []string{
+		"llmgw_gpu_pcie_replays class=h100-80g gpu=0 node=g1 -> 12\n",
+		"llmgw_gpu_remapped_rows class=h100-80g gpu=0 kind=uncorrectable node=g1 -> 1\n",
+		"llmgw_gpu_remap_pending class=h100-80g gpu=0 node=g1 -> 1\n",
+		"llmgw_gpu_nvlink_errors class=h100-80g gpu=0 kind=crc node=g1 -> 7\n",
+		"llmgw_gpu_xid_errors class=h100-80g gpu=0 node=g1 -> 3\n",
+		"llmgw_gpu_last_xid class=h100-80g gpu=0 node=g1 -> 79\n",
+	} {
+		if !strings.Contains(out, want) {
+			t.Errorf("missing %s", want)
+		}
+	}
+	// An H100 keeps no retired pages; none are published.
+	if strings.Contains(out, "llmgw_gpu_retired_pages") {
+		t.Errorf("a count the card does not keep was published:\n%s", out)
+	}
+}

internal/control/inventory/inventory.go

@@ -3,6 +3,7 @@
 package inventory
 
 import (
+	"fmt"
 	"maps"
 	"slices"
 	"strings"
@@ -319,8 +320,74 @@ type GPUSensor struct {
 	PowerW      float64  `json:"power_w,omitempty"`
 	PowerLimitW float64  `json:"power_limit_w,omitempty"`
 	Throttle    []string `json:"throttle,omitempty"`
+	// How busy the card is, as of this sample; -1 is not reported. Zero
+	// MemoryTotalMB is an agent too old to send these, whose figures are
+	// the ones it read at registration.
+	UtilPercent    float64 `json:"util_percent,omitempty"`
+	MemUtilPercent float64 `json:"mem_util_percent,omitempty"`
+	MemoryUsedMB   float64 `json:"memory_used_mb,omitempty"`
+	MemoryTotalMB  float64 `json:"memory_total_mb,omitempty"`
+	SMClockMHz     float64 `json:"sm_clock_mhz,omitempty"`
+	ECCUncorrected float64 `json:"ecc_uncorrected,omitempty"`
+	// Wear is what wears the card out, read every minute; nil from an agent
+	// that does not read it.
+	Wear *GPUWear `json:"wear,omitempty"`
 }
 
+// GPUWear is a card's wear and failures. Counts of -1 are not reported for
+// it: a card before Ampere has no remapped rows, one after no retired pages.
+type GPUWear struct {
+	PCIeReplays           int64     `json:"pcie_replays"`
+	RetiredSBE            int64     `json:"retired_pages_sbe"`
+	RetiredDBE            int64     `json:"retired_pages_dbe"`
+	RetirePending         bool      `json:"retire_pending,omitempty"`
+	RemappedCorrectable   int64     `json:"remapped_rows_correctable"`
+	RemappedUncorrectable int64     `json:"remapped_rows_uncorrectable"`
+	RemapPending          bool      `json:"remap_pending,omitempty"`
+	RemapFailure          bool      `json:"remap_failure,omitempty"`
+	NVLinkCRC             int64     `json:"nvlink_crc_errors"`
+	NVLinkReplay          int64     `json:"nvlink_replay_errors"`
+	NVLinkRecovery        int64     `json:"nvlink_recovery_errors"`
+	XIDs                  int64     `json:"xid_errors"`
+	LastXID               int       `json:"last_xid,omitempty"`
+	LastXIDText           string    `json:"last_xid_text,omitempty"`
+	LastXIDAt             time.Time `json:"last_xid_at,omitempty"`
+}
+
+// Problems says, in a few words each, what is wrong with the card; empty
+// when nothing is.
+func (w *GPUWear) Problems() []string {
+	if w == nil {
+		return nil
+	}
+	var out []string
+	if w.RemapFailure {
+		out = append(out, "out of spare memory rows: replace the card")
+	}
+	if w.RemapPending {
+		out = append(out, "memory rows remapped, waiting for a reset")
+	}
+	if w.RetirePending {
+		out = append(out, "memory pages waiting for a reset to be retired")
+	}
+	if w.RemappedUncorrectable > 0 {
+		out = append(out, fmt.Sprintf("%d rows remapped for uncorrectable errors", w.RemappedUncorrectable))
+	}
+	if w.RetiredDBE > 0 {
+		out = append(out, fmt.Sprintf("%d pages retired for double-bit errors", w.RetiredDBE))
+	}
+	if w.XIDs > 0 {
+		out = append(out, fmt.Sprintf("XID %d (%s)", w.LastXID, w.LastXIDText))
+	}
+	if w.NVLinkCRC > 0 || w.NVLinkReplay > 0 || w.NVLinkRecovery > 0 {
+		out = append(out, fmt.Sprintf("NVLink errors: %d CRC, %d replay, %d recovery", w.NVLinkCRC, w.NVLinkReplay, w.NVLinkRecovery))
+	}
+	return out
+}
+
+// Busy reports whether the sample carries how busy the card is.
+func (s GPUSensor) Busy() bool { return s.MemoryTotalMB > 0 }
+
 // ShardLoad is one engine's use: cores busy, and resident memory.
 type ShardLoad struct {
 	CPUCores float64

internal/control/reconciler/reconciler.go

@@ -255,6 +255,11 @@ type Reconciler struct {
 	// restarted and not yet begun. One is replaced at a time: replacing them
 	// all at once would need the memory of every replica twice over.
 	restartQueue map[string][]string
+	// restartInPlace are the replicas asked to restart by stopping first.
+	restartInPlace map[string]bool
+	// restartRefused says, by replica, why a restart was given up rather
+	// than leave the model without it.
+	restartRefused map[string]string
 	// parked holds models that failed ParkAfter times in a row.
 	parked map[string]parkRecord
 	// extra is how many replicas load has added above a model's base, and
@@ -304,6 +309,8 @@ func New(cfg Config, cat *catalog.Live, inv *inventory.Store, disp Dispatcher, l
 		reservedPorts:   map[int]struct{}{},
 		wanted:          map[string]time.Time{},
 		restartQueue:    map[string][]string{},
+		restartInPlace:  map[string]bool{},
+		restartRefused:  map[string]string{},
 		parked:          map[string]parkRecord{},
 		extra:           map[string]int{},
 		busySince:       map[string]time.Time{},
@@ -729,6 +736,7 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
 	r.mu.Unlock()
 
 	desired := r.desiredFor(m)
+	outcome := notTried
 	switch {
 	case len(liveIDs) < desired && isParked:
 		// Not placed. Its failure is on every status the operator reads.
@@ -742,7 +750,7 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
 			// the real error in noise.
 			return
 		}
-		r.scaleUp(ctx, m, avoid)
+		outcome = r.scaleUp(ctx, m, avoid)
 	case len(liveIDs) > desired:
 		r.scaleDown(ctx, liveIDs[desired:])
 	}
@@ -781,7 +789,7 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
 		}
 	}
 
-	r.retire(ctx, retiringIDs, readyLive, len(liveIDs), desired)
+	r.retire(ctx, m, retiringIDs, readyLive, len(liveIDs), desired, outcome, isParked)
 }
 
 // scaleUp places and starts one replica per pass.
@@ -789,7 +797,7 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
 // One at a time on purpose: placement decisions depend on what is already
 // allocated, and starting several at once from the same inventory snapshot
 // would hand them the same GPUs.
-func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []string) {
+func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []string) scaleOutcome {
 	profiles := r.orderedProfiles(m)
 	res := r.sched.Place(profiles, avoid)
 	if res.Placement == nil && len(avoid) > 0 {
@@ -813,13 +821,13 @@ func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []strin
 		// the pass that reaps it, and the next pass places this model — which
 		// keeps the stop-then-start ordering in one place rather than two.
 		if m.Scaling.OnDemand() && r.makeRoomFor(ctx, m) {
-			return
+			return makingRoom
 		}
 		if r.awaitingPreemption(m.Name) || r.preemptFor(ctx, m, avoid) {
-			return
+			return makingRoom
 		}
 		r.notePlacementFailure(m.Name, res.Explain())
-		return
+		return noRoom
 	}
 	if why := r.overLimit(m, res.Placement); why != "" {
 		// Recorded as the placement's answer, so the model reads as not
@@ -829,13 +837,26 @@ func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []strin
 			{Profile: res.Placement.Profile.Name, Reason: why}}}
 		r.mu.Unlock()
 		r.notePlacementFailure(m.Name, why)
-		return
+		return overLimit
 	}
 	if err := r.startReplica(ctx, m, res.Placement); err != nil {
 		r.log.Error("starting replica failed", "model", m.Name, "err", err)
+		return noRoom
 	}
+	return placed
 }
 
+// scaleOutcome is what one scaleUp did.
+type scaleOutcome int
+
+const (
+	notTried   scaleOutcome = iota // nothing was asked of placement this pass
+	placed                         // a replica was started
+	makingRoom                     // room is being made: a preemption, an idle replica stopping
+	noRoom                         // placement found nowhere, and nothing is making room
+	overLimit                      // there is room, but the owner's quota does not allow it
+)
+
 func (r *Reconciler) startReplica(ctx context.Context, m catalog.Model, pl *placement.Placement) error {
 	prof, ok := m.ProfileByName(pl.Profile.Name)
 	if !ok {

internal/control/reconciler/reconciler_test.go

@@ -114,6 +114,17 @@ type harness struct {
 	inv   *inventory.Store
 	disp  *fakeDispatcher
 	clock *clock
+	live  *catalog.Live
+}
+
+// setModels replaces the catalog the reconciler works from, as an edit does.
+func (h *harness) setModels(t *testing.T, models ...catalog.Model) {
+	t.Helper()
+	cat, err := catalog.New(models)
+	if err != nil {
+		t.Fatal(err)
+	}
+	h.live.Store(cat)
 }
 
 func newHarness(t *testing.T, m catalog.Model, nodes ...inventory.Node) *harness {
@@ -135,7 +146,8 @@ func newHarness(t *testing.T, m catalog.Model, nodes ...inventory.Node) *harness
 	cfg.UnreachableAfter = 15 * time.Second
 	cfg.LostAfter = 60 * time.Second
 
-	r := New(cfg, catalog.NewLive(cat), inv, disp, slog.New(slog.DiscardHandler))
+	live := catalog.NewLive(cat)
+	r := New(cfg, live, inv, disp, slog.New(slog.DiscardHandler))
 	r.SetClock(c.now)
 	// Nodes were just registered, so their LastSeen is wall-clock now, which is
 	// far from the fake clock's epoch. Re-stamp them onto the fake timeline.
@@ -150,7 +162,7 @@ func newHarness(t *testing.T, m catalog.Model, nodes ...inventory.Node) *harness
 	for _, n := range nodes {
 		touchAt(inv, n.ID, c.now())
 	}
-	return &harness{r: r, inv: inv, disp: disp, clock: c}
+	return &harness{r: r, inv: inv, disp: disp, clock: c, live: live}
 }
 
 // touchAt rewrites a node's LastSeen onto the fake clock's timeline.
@@ -1787,7 +1799,9 @@ func TestARestartedReplicaServesUntilItsReplacementIsReady(t *testing.T) {
 
 func TestARestartWithNoRoomBesideItHappensInPlace(t *testing.T) {
 	// One node, room for one replica: the replacement can only start once the
-	// old one has gone. Waiting forever would mean never restarting at all.
+	// old one has gone. It is the only thing in the way, so it goes now —
+	// waiting out a grace period first changed nothing but the wait, which
+	// in the field read as a restart that never happened.
 	h := newHarness(t, testModel(1), gpuNode("1", 4))
 	h.r.Reconcile(t.Context())
 	h.markReady()
@@ -1796,14 +1810,8 @@ func TestARestartWithNoRoomBesideItHappensInPlace(t *testing.T) {
 		t.Fatal(err)
 	}
 	h.r.Reconcile(t.Context())
-	if rep, _ := replicaByID(h, old); rep.State != Ready {
-		t.Fatalf("stopped before the grace period: %s", rep.State)
-	}
-
-	h.advance(h.r.cfg.DrainGrace + time.Second)
-	h.r.Reconcile(t.Context())
 	if rep, ok := replicaByID(h, old); ok && rep.State == Ready {
-		t.Fatal("still serving after the grace period with no replacement possible")
+		t.Fatal("still serving with no replacement possible beside it")
 	}
 	for i := 0; i < 3; i++ {
 		h.r.Reconcile(t.Context())
@@ -1815,6 +1823,52 @@ func TestARestartWithNoRoomBesideItHappensInPlace(t *testing.T) {
 	}
 }
 
+func TestARestartWhoseReplacementFitsNowhereIsGivenUp(t *testing.T) {
+	// The profile was edited to need more cards than the node has: stopping
+	// the old replica would leave the model with nothing at all.
+	m := testModel(1)
+	h := newHarness(t, m, gpuNode("1", 4))
+	h.r.Reconcile(t.Context())
+	h.markReady()
+	old := h.r.ReadyReplicas()[0].ID
+
+	m.Profiles[0].TP, m.Profiles[0].GPUsPerNode = 8, 8
+	h.setModels(t, m)
+	if err := h.r.Restart(old); err != nil {
+		t.Fatal(err)
+	}
+	for range 3 {
+		h.r.Reconcile(t.Context())
+		h.advance(h.r.cfg.DrainGrace + time.Second)
+	}
+	if rep, ok := replicaByID(h, old); !ok || rep.State != Ready {
+		t.Fatalf("the only replica was stopped for a replacement that fits nowhere: %+v", rep)
+	}
+	if st := h.r.RestartState(old); !strings.HasPrefix(st, "refused: ") || !strings.Contains(st, "fits on no node") {
+		t.Errorf("restart state %q should say it was given up, and why", st)
+	}
+	// Asked again, it is tried again.
+	if err := h.r.Restart(old); err != nil || h.r.RestartState(old) != "queued" {
+		t.Errorf("a new restart did not clear the refusal: %q, %v", h.r.RestartState(old), err)
+	}
+}
+
+func TestARestartAskedInPlaceStopsFirst(t *testing.T) {
+	// Room beside it, but in place was asked for: an engine that must not
+	// run twice, or memory somebody knows is short.
+	h := newHarness(t, testModel(1), gpuNode("1", 4), gpuNode("2", 4))
+	h.r.Reconcile(t.Context())
+	h.markReady()
+	old := h.r.ReadyReplicas()[0].ID
+	if err := h.r.RestartInPlace(old); err != nil {
+		t.Fatal(err)
+	}
+	h.r.Reconcile(t.Context())
+	if rep, ok := replicaByID(h, old); ok && rep.State == Ready {
+		t.Fatalf("asked in place, it kept serving beside its replacement: %+v", rep)
+	}
+}
+
 func TestReplicasOfAModelAreRestartedOneAtATime(t *testing.T) {
 	h := newHarness(t, testModel(2), gpuNode("1", 8), gpuNode("2", 8), gpuNode("3", 8))
 	for i := 0; i < 3; i++ {

internal/control/reconciler/restart.go

@@ -6,17 +6,34 @@ import (
 	"fmt"
 	"slices"
 	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
 )
 
 // retire finishes restarting replicas.
 //
 // A retiring replica keeps serving until its replacement is ready, and is then
-// drained like any other. When no replacement could be placed at all —
-// nowhere else has the memory, which on a small CPU fleet is the usual case —
-// it is stopped after DrainGrace so its replacement can take its place: a
-// restart in place, with the model unserved while it loads if it had no other
-// replica. Said in the log, since that is the one case with an outage.
-func (r *Reconciler) retire(ctx context.Context, ids []string, readyLive, live, desired int) {
+// drained like any other. When no replacement could be placed beside it, what
+// happens depends on what is in the way:
+//
+//   - the replica itself — one node, or every card that fits held: it is
+//     stopped now, and its replacement takes its place. Waiting would change
+//     nothing; it used to wait DrainGrace anyway, five minutes reading only
+//     "restart replacing", which looked like a restart that never would.
+//   - more than it: the replacement fits nowhere even with it stopped — its
+//     profile was edited to need more than any node has free. Stopping it
+//     would leave the model with nothing, so the restart is given up, and
+//     says why where the replica is shown.
+//   - something being moved out of the way (a preemption, an idle model
+//     stopping): waited for.
+//   - nothing tried this pass (a backoff): the old rule, stopped after
+//     DrainGrace with no replacement in sight.
+//
+// A restart asked for in place (ctl restart --in-place) stops first, always.
+// Each stop is said in the log, since it is the one case with an outage.
+func (r *Reconciler) retire(ctx context.Context, m catalog.Model, ids []string, readyLive, live, desired int,
+	outcome scaleOutcome, parked bool,
+) {
 	for _, id := range ids {
 		r.mu.Lock()
 		rep, ok := r.replicas[id]
@@ -24,6 +41,7 @@ func (r *Reconciler) retire(ctx context.Context, ids []string, readyLive, live,
 		if ok {
 			since = rep.RetiringAt
 		}
+		inPlace := r.restartInPlace[id]
 		r.mu.Unlock()
 		if !ok {
 			continue
@@ -31,15 +49,48 @@ func (r *Reconciler) retire(ctx context.Context, ids []string, readyLive, live,
 		switch {
 		case readyLive >= desired:
 			r.drainForRestart(ctx, id)
-		case live < desired && r.now().Sub(since) > r.cfg.DrainGrace:
-			r.log.Warn("restarting a replica in place: no room for its replacement beside it",
+		case inPlace:
+			r.log.Warn("restarting a replica in place, as asked: stopped before its replacement starts", "replica", id)
+			r.stopForRestart(ctx, id)
+		case live >= desired:
+			// A replacement is placed and loading; it is waited for.
+		case parked:
+			r.refuseRestart(id, "its replacements fail to start; the model is parked (llmgw ctl retry "+m.Name+")")
+		case outcome == noRoom && r.fitsWithout(m, nil, []victim{{ID: id, Model: m.Name}}):
+			r.log.Warn("restarting a replica in place: it is the only thing in its replacement's way",
+				"replica", id, "model", m.Name)
+			r.stopForRestart(ctx, id)
+		case outcome == noRoom || outcome == overLimit:
+			r.refuseRestart(id, "its replacement fits on no node even with this replica stopped "+
+				"(llmgw ctl explain-placement "+m.Name+")")
+		case outcome == notTried && r.now().Sub(since) > r.cfg.DrainGrace:
+			r.log.Warn("restarting a replica in place: no replacement beside it",
 				"replica", id, "after", r.cfg.DrainGrace)
-			r.drainForRestart(ctx, id)
-			r.stopReplica(ctx, id)
+			r.stopForRestart(ctx, id)
 		}
 	}
 }
 
+// stopForRestart drains a replica and stops it at once, for its replacement
+// to take its place.
+func (r *Reconciler) stopForRestart(ctx context.Context, id string) {
+	r.drainForRestart(ctx, id)
+	r.stopReplica(ctx, id)
+}
+
+// refuseRestart gives up a replica's restart, leaving it serving, and keeps
+// why for whoever looks at the replica.
+func (r *Reconciler) refuseRestart(id, why string) {
+	r.mu.Lock()
+	if rep, ok := r.replicas[id]; ok {
+		rep.RetiringAt = time.Time{}
+	}
+	delete(r.restartInPlace, id)
+	r.restartRefused[id] = why
+	r.mu.Unlock()
+	r.log.Warn("not restarting a replica: it keeps serving", "replica", id, "because", why)
+}
+
 func (r *Reconciler) drainForRestart(ctx context.Context, id string) {
 	r.mu.Lock()
 	rep, ok := r.replicas[id]
@@ -48,6 +99,7 @@ func (r *Reconciler) drainForRestart(ctx context.Context, id string) {
 		rep.State, rep.DrainingAt, rep.RetiringAt = Draining, r.now(), time.Time{}
 		shards = slices.Clone(rep.Shards)
 	}
+	delete(r.restartInPlace, id)
 	r.mu.Unlock()
 	for _, s := range shards {
 		if err := r.disp.DrainShard(ctx, s.NodeID, s.ID); err != nil {
@@ -78,7 +130,14 @@ var ErrNoSuchReplica = errors.New("no ready replica with that id")
 // Restart queues replicas to be replaced, each by a new one started beside it
 // before it is stopped — to pick up an edited profile, an engine key, a new
 // engine version. Replicas of one model are replaced one at a time.
-func (r *Reconciler) Restart(ids ...string) error {
+func (r *Reconciler) Restart(ids ...string) error { return r.restart(false, ids) }
+
+// RestartInPlace queues replicas to be replaced by stopping each first and
+// then starting its replacement: for when there is knowingly no room beside
+// it, and the outage while it loads is accepted.
+func (r *Reconciler) RestartInPlace(ids ...string) error { return r.restart(true, ids) }
+
+func (r *Reconciler) restart(inPlace bool, ids []string) error {
 	r.mu.Lock()
 	for _, id := range ids {
 		rep, ok := r.replicas[id]
@@ -86,6 +145,10 @@ func (r *Reconciler) Restart(ids ...string) error {
 			r.mu.Unlock()
 			return fmt.Errorf("%w: %s", ErrNoSuchReplica, id)
 		}
+		delete(r.restartRefused, id)
+		if inPlace {
+			r.restartInPlace[id] = true
+		}
 		if !slices.Contains(r.restartQueue[rep.Model], id) && rep.RetiringAt.IsZero() {
 			r.restartQueue[rep.Model] = append(r.restartQueue[rep.Model], id)
 		}
@@ -95,8 +158,8 @@ func (r *Reconciler) Restart(ids ...string) error {
 	return nil
 }
 
-// RestartState says where a replica's restart stands: "queued", "replacing",
-// or "" when none was asked for.
+// RestartState says where a replica's restart stands: "queued",
+// "replacing", "in place", "refused: why", or "" when none was asked for.
 func (r *Reconciler) RestartState(id string) string {
 	r.mu.Lock()
 	defer r.mu.Unlock()
@@ -104,10 +167,14 @@ func (r *Reconciler) RestartState(id string) string {
 	switch {
 	case !ok:
 		return ""
+	case !rep.RetiringAt.IsZero() && r.restartInPlace[id]:
+		return "in place"
 	case !rep.RetiringAt.IsZero():
 		return "replacing"
 	case slices.Contains(r.restartQueue[rep.Model], id):
 		return "queued"
+	case r.restartRefused[id] != "":
+		return "refused: " + r.restartRefused[id]
 	}
 	return ""
 }

internal/docgen/docgen.go

@@ -29,19 +29,30 @@ type Field struct {
 	Key  string
 	Type string
 	Doc  string
-	// Nested names the struct this key expands into, empty for a leaf. It is
-	// what lets the reference say "see Auth" rather than printing the same
-	// twenty keys under four parents.
+	// Nested names the struct this key expands into, by its ID
+	// (config.Auth), empty for a leaf. It is what lets the reference say
+	// "see Auth" rather than printing the same twenty keys under four
+	// parents.
 	Nested string
 }
 
 // Block is a struct that appears in configuration.
 type Block struct {
-	Name   string
+	// Package and Name are where the struct is declared: what tells
+	// tlsconf.Config from the agent's Config.
+	Package string
+	Name    string
+	// Title is what it is called in the reference and on the screens: the
+	// name alone where no other block shares it, the package's name with it
+	// where one does.
+	Title  string
 	Doc    string
 	Fields []Field
 }
 
+// ID is the block's package and name, unique where Name is not.
+func (b Block) ID() string { return b.Package + "." + b.Name }
+
 // Collect reads koanf-tagged structs out of Go source files.
 func Collect(files []string) ([]Block, error) {
 	fset := token.NewFileSet()
@@ -62,22 +73,36 @@ func Collect(files []string) ([]Block, error) {
 			if !ok {
 				return true
 			}
-			b, ok := blockOf(ts, st)
-			if !ok || seen[b.Name] {
+			b, ok := blockOf(f.Name.Name, ts, st)
+			if !ok || seen[b.ID()] {
 				return true
 			}
-			seen[b.Name] = true
+			seen[b.ID()] = true
 			blocks = append(blocks, b)
 			return true
 		})
 	}
-	sort.Slice(blocks, func(i, j int) bool { return blocks[i].Name < blocks[j].Name })
+	// Named by their own name unless another block has it too. Keyed by
+	// name alone, tlsconf.Config, engines.Config and supervisor.Config were
+	// each taken for the agent's Config and dropped: the reference had no
+	// TLS settings at all.
+	count := map[string]int{}
+	for _, b := range blocks {
+		count[b.Name]++
+	}
+	for i := range blocks {
+		blocks[i].Title = blocks[i].Name
+		if count[blocks[i].Name] > 1 {
+			blocks[i].Title = blocks[i].ID()
+		}
+	}
+	sort.Slice(blocks, func(i, j int) bool { return blocks[i].Title < blocks[j].Title })
 	return blocks, nil
 }
 
 // blockOf builds a Block, or reports that this struct carries no settings.
-func blockOf(ts *ast.TypeSpec, st *ast.StructType) (Block, bool) {
-	b := Block{Name: ts.Name.Name, Doc: docText(ts.Doc)}
+func blockOf(pkg string, ts *ast.TypeSpec, st *ast.StructType) (Block, bool) {
+	b := Block{Package: pkg, Name: ts.Name.Name, Doc: docText(ts.Doc)}
 	for _, f := range st.Fields.List {
 		if f.Tag == nil {
 			continue
@@ -90,7 +115,7 @@ func blockOf(ts *ast.TypeSpec, st *ast.StructType) (Block, bool) {
 		if !ok || key == "" || key == "-" {
 			continue
 		}
-		typ, nested := typeOf(f.Type)
+		typ, nested := typeOf(pkg, f.Type)
 		b.Fields = append(b.Fields, Field{
 			Key: key, Type: typ, Doc: docText(f.Doc), Nested: nested,
 		})
@@ -99,29 +124,34 @@ func blockOf(ts *ast.TypeSpec, st *ast.StructType) (Block, bool) {
 }
 
 // typeOf renders a field's type, and names the struct it expands into.
-func typeOf(e ast.Expr) (rendered, nested string) {
+func typeOf(pkg string, e ast.Expr) (rendered, nested string) {
 	switch t := e.(type) {
 	case *ast.Ident:
 		if isNamedStruct(t.Name) {
-			return t.Name, t.Name
+			return t.Name, pkg + "." + t.Name
 		}
 		return t.Name, ""
 	case *ast.SelectorExpr:
-		// A type from another package, such as tlsconf.Config.
+		// A type from another package, such as tlsconf.Config. Standard
+		// library types are values, not blocks: a time.Duration is a
+		// setting, written "30s".
 		name := t.Sel.Name
-		if pkg, ok := t.X.(*ast.Ident); ok {
-			return pkg.Name + "." + name, name
+		if x, ok := t.X.(*ast.Ident); ok {
+			if x.Name == "time" {
+				return x.Name + "." + name, ""
+			}
+			return x.Name + "." + name, x.Name + "." + name
 		}
-		return name, name
+		return name, ""
 	case *ast.ArrayType:
-		inner, _ := typeOf(t.Elt)
+		inner, _ := typeOf(pkg, t.Elt)
 		return "[]" + inner, ""
 	case *ast.MapType:
-		k, _ := typeOf(t.Key)
-		v, _ := typeOf(t.Value)
+		k, _ := typeOf(pkg, t.Key)
+		v, _ := typeOf(pkg, t.Value)
 		return "map[" + k + "]" + v, ""
 	case *ast.StarExpr:
-		inner, n := typeOf(t.X)
+		inner, n := typeOf(pkg, t.X)
 		return "*" + inner, n
 	default:
 		return "value", ""

internal/docgen/docgen_test.go

@@ -136,3 +136,18 @@ func TestAFieldMarkedNotConfigurableIsLeftOut(t *testing.T) {
 		}
 	}
 }
+
+func TestTheSettingsScreensDescribeTheCode(t *testing.T) {
+	// The screens list Reference; it must be what the structs say now.
+	blocks, err := Collect(sourcePaths(t))
+	if err != nil {
+		t.Fatal(err)
+	}
+	have, err := os.ReadFile(filepath.Join(repoRoot(t), "internal", "docgen", "reference_gen.go"))
+	if err != nil {
+		t.Fatalf("reference_gen.go is missing; run go generate ./...: %v", err)
+	}
+	if string(have) != RenderGo(blocks) {
+		t.Error("internal/docgen/reference_gen.go no longer matches the configuration structs; run go generate ./...")
+	}
+}

internal/docgen/gen/main.go

@@ -38,5 +38,9 @@ func main() {
 		fmt.Fprintln(os.Stderr, "error:", err)
 		os.Exit(1)
 	}
-	fmt.Printf("docs/reference.md: %d blocks\n", len(blocks))
+	if err := os.WriteFile(filepath.Join(root, "internal", "docgen", "reference_gen.go"), []byte(docgen.RenderGo(blocks)), 0o644); err != nil {
+		fmt.Fprintln(os.Stderr, "error:", err)
+		os.Exit(1)
+	}
+	fmt.Printf("docs/reference.md and internal/docgen/reference_gen.go: %d blocks\n", len(blocks))
 }

internal/docgen/reference_gen.go

@@ -0,0 +1,302 @@
+// Code generated by go run ./internal/docgen/gen; DO NOT EDIT.
+
+package docgen
+
+// Reference is every configuration block, as Collect reads them.
+var Reference = []Block{
+	{Package: "catalog", Name: "Adapter", Title: "Adapter", Doc: "", Fields: []Field{
+		{Key: "name", Type: "string", Doc: "Name is what clients ask for, like a model name.", Nested: ""},
+		{Key: "source", Type: "string", Doc: "Source is the adapter file (GGUF for llama.cpp), fetched by each node\nas the model's weights are.", Nested: ""},
+		{Key: "scale", Type: "float64", Doc: "Scale is how strongly it is applied; zero takes 1.", Nested: ""},
+	}},
+	{Package: "config", Name: "Admission", Title: "Admission", Doc: "", Fields: []Field{
+		{Key: "enabled", Type: "bool", Doc: "Enabled turns the layer on. Off, every request goes straight through,\nwhich is the v0.2.0 behaviour.", Nested: ""},
+		{Key: "bands", Type: "[]Band", Doc: "", Nested: ""},
+		{Key: "default_band", Type: "string", Doc: "", Nested: ""},
+		{Key: "band_header", Type: "string", Doc: "BandHeader lets a client declare its band. Until authentication lands in\nv1.0.0 this is trusted, so the port must not be exposed to callers who\nshould not be able to promote themselves.", Nested: ""},
+		{Key: "queue_ttl", Type: "time.Duration", Doc: "", Nested: ""},
+		{Key: "queue_capacity_per_band", Type: "int", Doc: "", Nested: ""},
+		{Key: "queue_capacity_total", Type: "int", Doc: "QueueCapacityTotal bounds everything queued across bands. Reached, a\nrequest is not refused while a lower band has any queued: the newest of\nthe lowest band is turned away instead, with a 503 and Retry-After.\nZero takes half of all the bands' capacities together.", Nested: ""},
+		{Key: "interval", Type: "time.Duration", Doc: "", Nested: ""},
+	}},
+	{Package: "config", Name: "Auth", Title: "Auth", Doc: "", Fields: []Field{
+		{Key: "enabled", Type: "bool", Doc: "", Nested: ""},
+		{Key: "store", Type: "string", Doc: "Store is the SQLite file holding keys, tenancy and usage.", Nested: ""},
+		{Key: "cache_ttl", Type: "time.Duration", Doc: "CacheTTL bounds how long a revoked key keeps working, traded against a\ndatabase read per key per TTL.", Nested: ""},
+		{Key: "negative_ttl", Type: "time.Duration", Doc: "NegativeTTL is how long a rejected key is remembered as rejected, so a\nclient looping with a bad key cannot become a database read per request.", Nested: ""},
+		{Key: "default_rpm", Type: "int", Doc: "DefaultRPM and DefaultTPM apply to a key that no scope gives a limit to.\n\nUnlimited used to be the default, and a project created without limits is\nthe easy path — the \"default\" project on this cluster still is one. The\nfirst sign of that is the bill or the outage. Zero keeps the old\nbehaviour for anyone who wants it, but it should be a choice rather than\nan omission. A fallback, not a ceiling: an explicit limit at any scope\nwins, above or below.", Nested: ""},
+		{Key: "default_tpm", Type: "int", Doc: "", Nested: ""},
+		{Key: "signup_open", Type: "bool", Doc: "SignupOpen serves the signup form. Accounts it creates are pending until\nan operator approves them, so this is not a way in — but turning it off\nmeans accounts are made with `llmgw users create` instead.", Nested: ""},
+		{Key: "personal", Type: "Personal", Doc: "Personal is what a person may do on their own, outside any project: make\npersonal keys and use the chat page personally, within these limits.", Nested: "config.Personal"},
+		{Key: "pricing", Type: "map[string]Price", Doc: "Pricing is what each model costs, for budgets. A model absent from the\ntable costs whatever DefaultPrice says, which is nothing unless set.", Nested: ""},
+		{Key: "default_price", Type: "Price", Doc: "", Nested: "config.Price"},
+		{Key: "flush_interval", Type: "time.Duration", Doc: "FlushInterval and BatchSize decide when usage records are written.", Nested: ""},
+		{Key: "batch_size", Type: "int", Doc: "", Nested: ""},
+		{Key: "buffer_size", Type: "int", Doc: "BufferSize bounds the queue. Beyond it records are dropped rather than\nblocking a request: losing accounting is bad, turning a stalled database\ninto an outage is worse.", Nested: ""},
+		{Key: "usage_retention", Type: "time.Duration", Doc: "UsageRetention is how long the record of each request is kept: who sent\nit, which replica answered, how long it queued — what `ctl request`\nreads. Zero keeps it for ever. Totals, charts and budgets are read from\nfive-minute sums kept beside the records, which are not pruned, so this\nbounds the table that grows by a row a request without changing any\nfigure a page shows. At least a day when set.", Nested: ""},
+	}},
+	{Package: "catalog", Name: "Capabilities", Title: "Capabilities", Doc: "", Fields: []Field{
+		{Key: "tools", Type: "bool", Doc: "", Nested: ""},
+		{Key: "vision", Type: "bool", Doc: "", Nested: ""},
+		{Key: "embeddings", Type: "bool", Doc: "", Nested: ""},
+	}},
+	{Package: "config", Name: "Control", Title: "Control", Doc: "", Fields: []Field{
+		{Key: "interval", Type: "time.Duration", Doc: "Interval is how often desired and actual state are compared.", Nested: ""},
+		{Key: "base_backoff", Type: "time.Duration", Doc: "BaseBackoff and MaxBackoff bound how fast a failing model is retried.", Nested: ""},
+		{Key: "max_backoff", Type: "time.Duration", Doc: "", Nested: ""},
+		{Key: "unreachable_after", Type: "time.Duration", Doc: "UnreachableAfter and LostAfter stage node expiry. They are deliberately\nfar apart: a network blip must not tear down replicas that still serve,\nand rebuilding a multi-node replica costs tens of minutes of loading.", Nested: ""},
+		{Key: "lost_after", Type: "time.Duration", Doc: "", Nested: ""},
+		{Key: "port_range", Type: "[]int", Doc: "PortRange is the per-node range for engine HTTP and dist-init ports.", Nested: ""},
+		{Key: "mgmt_interface", Type: "string", Doc: "MgmtInterface is the ethernet interface NCCL bootstraps over.", Nested: ""},
+		{Key: "enabled", Type: "bool", Doc: "Enabled runs the control plane even when configuration declares no\nmodels.\n\nWithout it, a deployment whose catalog lives entirely in the database\ncannot start: control plane mode is inferred from the config file, so an\nempty models block means \"router only\", the process refuses to start for\nwant of anything to serve, and the models it would have loaded are never\nread. That is a cluster that cannot be managed at runtime because it is\nmanaged at runtime. AgentListen cannot carry this signal — it has a\ndefault, so every deployment would become a control plane.", Nested: ""},
+		{Key: "lease_ttl", Type: "time.Duration", Doc: "LeaseTTL is how long this control plane's claim to be the one placing\nwork stands without being renewed.\n\nOnly meaningful with a shared store, where a second control plane can\nsee the claim. Long enough to survive a slow query or a brief network\nstall, short enough that a standby takes over before anyone has finished\nreading the alert. Zero takes the default.", Nested: ""},
+	}},
+	{Package: "config", Name: "Endpoint", Title: "Endpoint", Doc: "", Fields: []Field{
+		{Key: "id", Type: "string", Doc: "", Nested: ""},
+		{Key: "url", Type: "string", Doc: "", Nested: ""},
+		{Key: "api_key", Type: "string", Doc: "APIKey is sent to the engine as a bearer token, for an engine started\nwith one.", Nested: ""},
+		{Key: "models", Type: "[]string", Doc: "", Nested: ""},
+		{Key: "engine", Type: "string", Doc: "Engine names the inference engine, which decides what its metrics are\ncalled. Empty means vLLM's names.", Nested: ""},
+		{Key: "pool", Type: "string", Doc: "", Nested: ""},
+		{Key: "weight", Type: "float64", Doc: "", Nested: ""},
+		{Key: "priority", Type: "int", Doc: "", Nested: ""},
+	}},
+	{Package: "config", Name: "EngineOverride", Title: "EngineOverride", Doc: "", Fields: []Field{
+		{Key: "gpu_class", Type: "string", Doc: "GPUClass picks nodes with a GPU of this class (\"v100-32g\").", Nested: ""},
+		{Key: "nodes", Type: "[]string", Doc: "Nodes picks nodes by id.", Nested: ""},
+		{Key: "labels", Type: "map[string]string", Doc: "Labels picks nodes carrying all of these.", Nested: ""},
+		{Key: "engines", Type: "map[string]string", Doc: "Engines are laid over the cluster's: a version, or \"\" for none.", Nested: ""},
+	}},
+	{Package: "catalog", Name: "External", Title: "External", Doc: "", Fields: []Field{
+		{Key: "url", Type: "string", Doc: "URL is the provider's base address, without /v1: https://api.openai.com.", Nested: ""},
+		{Key: "model", Type: "string", Doc: "Model is what the provider calls it; requests are rewritten to it.", Nested: ""},
+		{Key: "api_key_file", Type: "string", Doc: "APIKeyFile holds the provider's key, on the control plane. A file\nrather than the key itself, so the key is in neither the catalog, the\nstore nor the audit log; empty for a provider that asks for none.", Nested: ""},
+	}},
+	{Package: "config", Name: "Landing", Title: "Landing", Doc: "", Fields: []Field{
+		{Key: "contact", Type: "string", Doc: "Contact is where someone without a key should go — an address, a URL, a\nchannel name. Rendered as a link when it is one, as text otherwise.", Nested: ""},
+		{Key: "notice", Type: "string", Doc: "Notice is an optional line of your own, for whatever the reader needs to\nknow that this page cannot guess.", Nested: ""},
+	}},
+	{Package: "engines", Name: "LlamaCppConfig", Title: "LlamaCppConfig", Doc: "", Fields: []Field{
+		{Key: "bin", Type: "string", Doc: "Bin is the llama-server binary. It is the server, not the CLI: the\ngateway talks OpenAI over HTTP and needs the one that serves it.\nEmpty means the agent installs llama.cpp itself, at the version the\ncontrol plane asks for. Setting it opts out of that.", Nested: ""},
+		{Key: "parallel", Type: "int", Doc: "Parallel is how many requests a server handles at once, when a profile\ndoes not say. llama.cpp defaults to one, which would make every second\nrequest wait behind the first no matter what the router decided.", Nested: ""},
+		{Key: "threads", Type: "int", Doc: "Threads is how many CPU threads a shard with no GPU computes on. Zero\ntakes half this machine's CPUs.\n\nNot all of them, which is llama.cpp's own default: two busy servers\nthat each spin on every CPU fight for them, and on an 8-vCPU node\nthat made 7B generation fall from 10 tokens a second to under 1 each.\nAt half, one server alone is about a third slower and two together\nkeep 5.5 each. A node that only ever runs one CPU replica does better\nwith all of them; say so here.", Nested: ""},
+	}},
+	{Package: "config", Name: "Mirror", Title: "Mirror", Doc: "", Fields: []Field{
+		{Key: "api", Type: "string", Doc: "API is the mirror's management address, scheme included, for example\nhttps://models.internal:8082. Empty means no mirror is configured.\n\nThe management port, not the one nodes fetch bytes from. Those are two\nlisteners on the mirror precisely so this one can be locked down without\nre-certifying every agent's fetcher.", Nested: ""},
+		{Key: "tls", Type: "tlsconf.Config", Doc: "TLS authenticates to the mirror's management API, with the same CA the\nagents use. Empty leaves the connection in plain HTTP, which is only\nacceptable on a network where nothing untrusted can reach that port —\nthe mirror says as much in its own log when it starts that way.", Nested: "tlsconf.Config"},
+		{Key: "registry", Type: "string", Doc: "Registry is the model registry the mirror page browses, empty for the\npublic Hugging Face. Set it to an internal mirror of the registry where\nthe control plane cannot reach the internet — or leave the whole mirror\nblock out, which turns the explorer off rather than giving it a search\nbox that can only time out.", Nested: ""},
+		{Key: "timeout", Type: "time.Duration", Doc: "Timeout bounds a management call. It does not bound a download: the\nmirror answers a fetch as soon as it has started one, and the bytes keep\narriving long after this deadline has passed.", Nested: ""},
+	}},
+	{Package: "catalog", Name: "Model", Title: "Model", Doc: "", Fields: []Field{
+		{Key: "name", Type: "string", Doc: "", Nested: ""},
+		{Key: "aliases", Type: "[]string", Doc: "", Nested: ""},
+		{Key: "description", Type: "string", Doc: "Description is what this model is for, in a line.\n\nWritten by whoever added it, because nobody else can say it. A registry\noffers a pipeline tag and a wall of README, and neither answers the\nquestion a person picking from a list is asking: which of these nine\nnames should I send this request to.\n\nIt reaches clients through /v1/models, which is the only view most of\nthem ever see. A gateway serving nine models by name alone makes that\nchoice by guesswork.", Nested: ""},
+		{Key: "source", Type: "string", Doc: "Source is the weights path or a HuggingFace id.", Nested: ""},
+		{Key: "profiles", Type: "[]Profile", Doc: "", Nested: ""},
+		{Key: "scaling", Type: "Scaling", Doc: "", Nested: "catalog.Scaling"},
+		{Key: "class", Type: "string", Doc: "Class is how much this model's capacity is worth against another's,\nwhen there is not enough for both: critical, standard or opportunistic.\nEmpty is standard, or opportunistic for a model loaded on demand. See\nEffectiveClass.", Nested: ""},
+		{Key: "owner", Type: "string", Doc: "Owner is the part of the tenancy tree whose capacity quota this model\ncounts against, written org:ID, team:ID or project:ID. Empty is shared:\nheld by the cluster, counted against nobody.", Nested: ""},
+		{Key: "external", Type: "*External", Doc: "External is a model served by someone else — an OpenAI-compatible API\noutside the cluster — rather than by replicas placed here. Such a model\nhas no profiles and nothing is placed for it; the router sends its\nrequests to the provider. Most useful named in another model's\nfallbacks: used only when the cluster has no capacity left. Its\nprompts leave the cluster, so which keys may name it is worth an\nallowlist.", Nested: "catalog.External"},
+		{Key: "adapters", Type: "[]Adapter", Doc: "Adapters are LoRA adapters served on this model's replicas, each asked\nfor by its own name as though it were a model. Every replica loads\nevery adapter, applying one only to a request that names it.", Nested: ""},
+		{Key: "fallbacks", Type: "[]string", Doc: "", Nested: ""},
+		{Key: "routing_profile", Type: "string", Doc: "RoutingProfile selects how the router scores endpoints for this model.", Nested: ""},
+		{Key: "slo", Type: "SLO", Doc: "", Nested: "catalog.SLO"},
+		{Key: "context_length", Type: "int", Doc: "ContextLength is how many tokens one request may use, prompt and\ncompletion together, and MaxOutput bounds the completion alone. Zero\nmeans unknown, which is what every model was until now.\n\nThe gateway knew neither, so an over-long prompt was not refused: it was\nadmitted, given a replica, and failed at the engine, and a client sizing\nits prompts from /v1/models had nothing to read. Worse, the number is not\nthe one an operator writes in the engine's arguments — llama.cpp divides\n--ctx-size across its parallel slots, so a pool of 8192 across four slots\nis 2048 a request, and nothing said so.", Nested: ""},
+		{Key: "max_output", Type: "int", Doc: "", Nested: ""},
+		{Key: "capabilities", Type: "Capabilities", Doc: "Capabilities is what this model can do beyond plain completion.", Nested: "catalog.Capabilities"},
+	}},
+	{Package: "config", Name: "Monitoring", Title: "Monitoring", Doc: "", Fields: []Field{
+		{Key: "prometheus", Type: "string", Doc: "Prometheus is the server's address, for example\nhttp://monitoring.internal:9090. Empty leaves the charts off the\ndashboard rather than showing an error where one should be.", Nested: ""},
+		{Key: "window", Type: "time.Duration", Doc: "Window is how far back the charts look. Zero takes an hour.", Nested: ""},
+		{Key: "timeout", Type: "time.Duration", Doc: "Timeout bounds one query. It runs while somebody waits for a page, so a\nmonitoring system that is slow today is skipped rather than waited for.", Nested: ""},
+	}},
+	{Package: "discovery", Name: "Options", Title: "Options", Doc: "", Fields: []Field{
+		{Key: "nvidia_smi", Type: "string", Doc: "NvidiaSMI is the path to nvidia-smi; empty means look it up on PATH.\nWorth configuring on nodes where the driver lives outside the default\npaths, which containerized driver installs often do.", Nested: ""},
+		{Key: "ib_root", Type: "string", Doc: "IBRoot is the sysfs InfiniBand directory.", Nested: ""},
+		{Key: "prefer_nvml", Type: "bool", Doc: "PreferNVML uses the NVML backend when this binary has one. NVLink\ntopology is read from nvidia-smi topo -m on every build; NVML, where it\nis used, is the more exact source of the peers.", Nested: ""},
+		{Key: "skip_gpus", Type: "bool", Doc: "SkipGPUs omits GPU discovery. A node without nvidia-smi is already\nfound to have none, so this is for one whose driver is broken.", Nested: ""},
+	}},
+	{Package: "config", Name: "Personal", Title: "Personal", Doc: "", Fields: []Field{
+		{Key: "disabled", Type: "bool", Doc: "Disabled refuses personal keys and personal chat; people then use the\ngateway only through projects they are members of.", Nested: ""},
+		{Key: "rpm", Type: "int", Doc: "", Nested: ""},
+		{Key: "tpm", Type: "int", Doc: "", Nested: ""},
+		{Key: "budget", Type: "float64", Doc: "Budget is in whole units per Period (\"day\" or \"month\").", Nested: ""},
+		{Key: "period", Type: "string", Doc: "", Nested: ""},
+	}},
+	{Package: "config", Name: "Prefix", Title: "Prefix", Doc: "", Fields: []Field{
+		{Key: "enabled", Type: "bool", Doc: "", Nested: ""},
+		{Key: "max_nodes", Type: "int", Doc: "MaxNodes bounds the tree, MinMatch is the shortest match worth acting on\nin bytes, and TTL is how long an entry is believed — an engine evicts on\nits own schedule, so an old record may describe a cache that is gone.", Nested: ""},
+		{Key: "min_match", Type: "int", Doc: "", Nested: ""},
+		{Key: "ttl", Type: "time.Duration", Doc: "", Nested: ""},
+		{Key: "match_ratio", Type: "float64", Doc: "MatchRatio is the share of the prompt that must match to be worth acting\non. BalanceAbs and BalanceRel are the imbalance guard, and both must trip\nbefore affinity is abandoned.", Nested: ""},
+		{Key: "balance_abs", Type: "float64", Doc: "", Nested: ""},
+		{Key: "balance_rel", Type: "float64", Doc: "", Nested: ""},
+	}},
+	{Package: "catalog", Name: "Profile", Title: "Profile", Doc: "", Fields: []Field{
+		{Key: "name", Type: "string", Doc: "", Nested: ""},
+		{Key: "engine", Type: "string", Doc: "", Nested: ""},
+		{Key: "gpu_class", Type: "string", Doc: "", Nested: ""},
+		{Key: "tp", Type: "int", Doc: "", Nested: ""},
+		{Key: "pp", Type: "int", Doc: "", Nested: ""},
+		{Key: "gpus_per_node", Type: "int", Doc: "GPUsPerNode defaults to TP for a single-node profile.", Nested: ""},
+		{Key: "nodes", Type: "int", Doc: "Nodes above 1 makes this a multi-node profile, which requires a fabric.", Nested: ""},
+		{Key: "args", Type: "[]string", Doc: "", Nested: ""},
+		{Key: "env", Type: "map[string]string", Doc: "", Nested: ""},
+		{Key: "min_memory_gb", Type: "int", Doc: "MinMemoryGB is the host RAM this profile needs to run.\n\nThe number that matters is the model's working set: the weights, plus the\nKV cache for the whole context pool, plus the engine's own overhead. For\na GPU profile that lives in the cards and this is small; for a CPU\nprofile it is the whole thing, and getting it wrong means an OOM on a\nnode that reported itself healthy.\n\nZero means unconstrained, which is every profile that existed before\nthis. A node that does not report its memory is not excluded either —\nan older agent must not have its nodes quietly emptied by an upgrade of\nthe control plane.", Nested: ""},
+		{Key: "cpu_cores", Type: "int", Doc: "CPUCores is the cores a replica of a profile with no GPU is given:\nreserved out of its node's CPUs by placement, as memory is, and what\nits engine computes on (llama.cpp's --threads).\n\nReserved because an engine takes every core it is told about, and two\non one node that each think they have all of them spin against each\nother: on an 8-vCPU node, two 7B replicas fell from 10 tokens a second\nto under one each. Zero reserves nothing and leaves the thread count to\nthe node, which is every profile from before this.", Nested: ""},
+		{Key: "min_disk_gb", Type: "int", Doc: "MinDiskGB is the free space a node needs before it can hold these\nweights.\n\nOnly checked when the node does not already have them: a node holding\nthe file needs no room for it, and requiring the space twice would keep\na model off the one node best placed to run it.\n\nDeclared rather than derived, because the size of a model that has never\nbeen fetched anywhere is not knowable from here — and a wrong guess\nfails late, halfway through a download, which is the failure this exists\nto prevent. Zero means unconstrained, and a node that reports no storage\nis not excluded: unknown is not full.", Nested: ""},
+		{Key: "node_selector", Type: "map[string]string", Doc: "NodeSelector restricts this profile to nodes carrying every one of these\nlabels, with these values.\n\nA CPU profile otherwise fits everywhere, because there is no card to be\nfree — which is right for a 0.5B model and wrong for a 14B one. Nothing\nin the inventory says how much memory a node has, so the operator says\nwhich nodes are meant for a model instead of the scheduler inferring it.\nAgents have reported labels since the beginning; this is what reads them.", Nested: ""},
+		{Key: "weight", Type: "float64", Doc: "Weight is the router's capacity weight. It should come from ctl bench\nrather than a guess (plan §6.2.1).", Nested: ""},
+		{Key: "priority", Type: "int", Doc: "Priority orders profiles during placement; highest that fits wins.", Nested: ""},
+		{Key: "startup_timeout", Type: "time.Duration", Doc: "StartupTimeout bounds model loading. A 70B model across two nodes takes\ntens of minutes, so this is per profile rather than global.", Nested: ""},
+	}},
+	{Package: "config", Name: "Router", Title: "Router", Doc: "", Fields: []Field{
+		{Key: "activation_wait", Type: "time.Duration", Doc: "ActivationWait is how long a request waits for an on-demand model to\nload before the client is told to come back. Zero takes the default.\n\nThe one real decision in on-demand loading. Refusing outright makes the\nfeature useless, since the first request for every model would fail;\nwaiting without a bound turns a request into a job, and a client that\nbudgeted thirty seconds ends up holding a connection for ten minutes.", Nested: ""},
+		{Key: "scrape_interval", Type: "time.Duration", Doc: "", Nested: ""},
+		{Key: "scrape_timeout", Type: "time.Duration", Doc: "", Nested: ""},
+		{Key: "stale_after", Type: "time.Duration", Doc: "StaleAfter is how long an endpoint may go unscraped before being scored\ndown (plan §7.6).", Nested: ""},
+		{Key: "retries", Type: "int", Doc: "Retries is how many *additional* endpoints may be tried after a failure,\nand only while nothing has been written to the client (plan §7.7).", Nested: ""},
+		{Key: "cooldown", Type: "time.Duration", Doc: "Cooldown is how long a failing endpoint leaves the candidate pool.", Nested: ""},
+		{Key: "failure_threshold", Type: "int", Doc: "FailureThreshold is the consecutive failure count that opens a breaker.\n1 turns the breaker into a pure cooldown.", Nested: ""},
+		{Key: "default_profile", Type: "string", Doc: "DefaultProfile names the routing profile used when a model picks none.", Nested: ""},
+		{Key: "prefix", Type: "Prefix", Doc: "Prefix turns on the approximate prefix tree (plan §7.5 tier 1). Off, the\nrouter uses session affinity alone, which is tier 0 and already most of\nthe win for multi-turn traffic.", Nested: "config.Prefix"},
+		{Key: "session_header", Type: "string", Doc: "SessionHeader carries the session id for affinity (plan §7.5 tier 0).", Nested: ""},
+		{Key: "inject_usage", Type: "bool", Doc: "InjectUsage asks engines for usage even when the client did not, so token\naccounting works regardless of client behaviour.", Nested: ""},
+		{Key: "tenant_label", Type: "string", Doc: "TenantLabel is the level of the tree the per-tenant request series are\nlabelled by — llmgw_tenant_requests_total and its siblings: \"org\",\n\"team\" or \"project\", or \"none\" for no such series. A label per project\non a cluster of thousands is more series than Prometheus wants, and a\nteam is what an alert is usually about; \"team\" is the default.", Nested: ""},
+		{Key: "max_body_bytes", Type: "int64", Doc: "MaxBodyBytes bounds the request body the router will read.", Nested: ""},
+		{Key: "timeouts", Type: "Timeouts", Doc: "", Nested: "config.Timeouts"},
+		{Key: "scoring", Type: "Scoring", Doc: "", Nested: "config.Scoring"},
+		{Key: "admission", Type: "Admission", Doc: "", Nested: "config.Admission"},
+	}},
+	{Package: "engines", Name: "SGLangConfig", Title: "SGLangConfig", Doc: "", Fields: []Field{
+		{Key: "python", Type: "string", Doc: "Python is the interpreter of the SGLang virtualenv. SGLang is launched as\na module rather than a console script, which is the documented entry\npoint and avoids depending on the script being on PATH.", Nested: ""},
+	}},
+	{Package: "catalog", Name: "SLO", Title: "SLO", Doc: "", Fields: []Field{
+		{Key: "ttft_p99", Type: "time.Duration", Doc: "", Nested: ""},
+		{Key: "tpot_p99", Type: "time.Duration", Doc: "", Nested: ""},
+	}},
+	{Package: "catalog", Name: "Scaling", Title: "Scaling", Doc: "", Fields: []Field{
+		{Key: "min", Type: "int", Doc: "", Nested: ""},
+		{Key: "max", Type: "int", Doc: "", Nested: ""},
+		{Key: "idle_timeout", Type: "time.Duration", Doc: "IdleTimeout is how long an on-demand model may go unasked-for before its\nreplicas are stopped. Zero takes the cluster default.\n\nMinutes rather than seconds, always. Stopping a replica throws away the\nKV cache prefix routing spent its effort filling, and a model that flaps\nin and out loses that entirely — so the saving has to be worth more than\nthe cache, which it only is over a real idle period.", Nested: ""},
+	}},
+	{Package: "config", Name: "Scoring", Title: "Scoring", Doc: "", Fields: []Field{
+		{Key: "target_waiting", Type: "float64", Doc: "", Nested: ""},
+		{Key: "target_inflight", Type: "float64", Doc: "", Nested: ""},
+		{Key: "kv_ceiling", Type: "float64", Doc: "", Nested: ""},
+		{Key: "session_load_factor", Type: "float64", Doc: "", Nested: ""},
+	}},
+	{Package: "config", Name: "Server", Title: "Server", Doc: "", Fields: []Field{
+		{Key: "listen", Type: "string", Doc: "Listen is the client-facing OpenAI API address.", Nested: ""},
+		{Key: "admin_listen", Type: "string", Doc: "AdminListen serves the dashboard, the management API, health and pprof.\n\nLoopback by default, and it should stay there: pprof can dump the process\nheap, and API keys are in that heap in plaintext. Every comment beside\nthis setting has always said not to expose it, while the default bound\nevery interface — a default that contradicts its own documentation is a\nfootgun, so the default now matches the advice.", Nested: ""},
+		{Key: "metrics_listen", Type: "string", Doc: "MetricsListen serves Prometheus metrics and nothing else.\n\nSeparate from AdminListen because the reason that one is loopback is\npprof, not metrics — and metrics are the one thing on it that something\nelse genuinely needs to reach. Sharing a listener meant a fleet could\neither be monitored or keep its heap private, which is not a choice\nanybody should have to make.\n\nEmpty means off, and metrics remain available on AdminListen either way,\nso nothing that scrapes it today has to change.", Nested: ""},
+		{Key: "agent_listen", Type: "string", Doc: "AgentListen is where node agents connect. Separate from the client port\nso the fleet-facing surface can be firewalled independently.", Nested: ""},
+		{Key: "auth", Type: "Auth", Doc: "Auth turns on API keys on the client-facing port.", Nested: "config.Auth"},
+		{Key: "threads", Type: "Threads", Doc: "Threads keeps conversations and agent memory without API keys, for a\ngateway on one trusted machine. With auth on they live in the auth store,\nper project, and this block is ignored.", Nested: "config.Threads"},
+		{Key: "landing", Type: "Landing", Doc: "Landing configures the page served at \"/\".", Nested: "config.Landing"},
+		{Key: "agent_tls", Type: "tlsconf.Config", Doc: "AgentTLS authenticates the fleet-facing port. It deliberately does not\ncover the client-facing one: that surface is fronted by a publicly\ntrusted certificate, while this one is mutual and private.", Nested: "tlsconf.Config"},
+		{Key: "monitoring", Type: "Monitoring", Doc: "Monitoring is where the dashboard reads its history from.", Nested: "config.Monitoring"},
+		{Key: "tracing", Type: "Tracing", Doc: "Tracing sends a trace of each inference request to an OpenTelemetry\ncollector. Off unless an endpoint is set here or in\nOTEL_EXPORTER_OTLP_ENDPOINT.", Nested: "config.Tracing"},
+		{Key: "mirror", Type: "Mirror", Doc: "Mirror points the control plane at a model mirror it can administer.\nEmpty means there is none, and /admin/mirror stays unmounted.", Nested: "config.Mirror"},
+		{Key: "engines", Type: "map[string]string", Doc: "Engines are the engines every node installs, and at which version. The\nagents install them themselves — the right build for their hardware,\nchecked against its publisher's checksum and seen to run — so a node is\nnever trusted to already have what somebody once put there.\n\nUnset keeps DefaultEngines. A version replaces the default; an empty\none leaves that engine to be installed by hand.", Nested: ""},
+		{Key: "engine_overrides", Type: "[]EngineOverride", Doc: "EngineOverrides give some nodes different engines from the rest: a\nvLLM built for CUDA 12 on the Volta cards CUDA 13 dropped, or none at\nall on nodes meant only for llama.cpp. Each picks nodes and lays its\nengines over the cluster's, in order, later ones last; an empty version\nremoves that engine from the nodes it picks.", Nested: ""},
+		{Key: "backup_dir", Type: "string", Doc: "BackupDir is where `llmgw server backup` writes, and where the newest\narchive's age is read for llmgw_backup_last_success_timestamp_seconds.\nEmpty means /var/backups/llmgw-server.", Nested: ""},
+		{Key: "releases", Type: "string", Doc: "Releases is where `llmgw ctl upgrade` fetches release binaries from,\nfor this machine and — through their agents — every node. Empty means\nthe project's own releases.", Nested: ""},
+		{Key: "router", Type: "Router", Doc: "", Nested: "config.Router"},
+		{Key: "control", Type: "Control", Doc: "", Nested: "config.Control"},
+		{Key: "models", Type: "[]catalog.Model", Doc: "", Nested: ""},
+		{Key: "endpoints", Type: "[]Endpoint", Doc: "Endpoints is the phase-1 static replica list. It is only consulted when\nno model declares a profile, which is what makes a router-only smoke test\nagainst sim-engines possible without a control plane.", Nested: ""},
+	}},
+	{Package: "engines", Name: "SimConfig", Title: "SimConfig", Doc: "", Fields: []Field{
+		{Key: "bin", Type: "string", Doc: "Bin is the llmgw binary itself; the engine is one of its subcommands.\nEmpty means whatever started this agent, which is right almost always.", Nested: ""},
+	}},
+	{Package: "config", Name: "Threads", Title: "Threads", Doc: "", Fields: []Field{
+		{Key: "store", Type: "string", Doc: "Store is the SQLite file. Empty means nothing is kept.", Nested: ""},
+		{Key: "retain_hours", Type: "int", Doc: "RetainHours is how long idle conversations and notes are kept. Required\nwith Store: a store with no expiry keeps everything anyone typed, by\nomission.", Nested: ""},
+	}},
+	{Package: "config", Name: "Timeouts", Title: "Timeouts", Doc: "", Fields: []Field{
+		{Key: "dial", Type: "time.Duration", Doc: "", Nested: ""},
+		{Key: "response_header", Type: "time.Duration", Doc: "ResponseHeader bounds the wait for headers on a streaming request. It is\na liveness check there, because headers arrive before generation starts.", Nested: ""},
+		{Key: "generation", Type: "time.Duration", Doc: "Generation bounds a non-streaming request, where the engine sends no\nheaders until the completion is done. Setting response_header alone used\nto cap both, so 60s read as a liveness guard and behaved as a limit of\nabout a hundred tokens on a slow model.", Nested: ""},
+		{Key: "first_token", Type: "time.Duration", Doc: "", Nested: ""},
+		{Key: "idle", Type: "time.Duration", Doc: "", Nested: ""},
+		{Key: "total", Type: "time.Duration", Doc: "", Nested: ""},
+	}},
+	{Package: "config", Name: "Tracing", Title: "Tracing", Doc: "", Fields: []Field{
+		{Key: "endpoint", Type: "string", Doc: "Endpoint is the collector, for example http://otel.internal:4318.\nEmpty falls back to OTEL_EXPORTER_OTLP_ENDPOINT, and to no tracing.", Nested: ""},
+		{Key: "sample_ratio", Type: "float64", Doc: "SampleRatio is the share of requests traced, 0 to 1, when the caller\ndid not decide it with a sampled traceparent of its own. Zero takes 1.", Nested: ""},
+	}},
+	{Package: "engines", Name: "VLLMConfig", Title: "VLLMConfig", Doc: "", Fields: []Field{
+		{Key: "bin", Type: "string", Doc: "Bin is the vllm executable.", Nested: ""},
+		{Key: "launch_mode", Type: "string", Doc: "LaunchMode selects the multi-node mechanism: \"mp\", the default, or\n\"ray\".\n\nmp is what vLLM documents today — every node runs serve with its rank,\nthe workers headless — and needs nothing but vLLM. ray is the older way,\nkept for a vLLM too old for mp's multi-node flags; it needs Ray on every\nnode and has not been checked against a real cluster.", Nested: ""},
+		{Key: "ray_bin", Type: "string", Doc: "RayBin is used when LaunchMode is \"ray\".", Nested: ""},
+	}},
+	{Package: "accounting", Name: "Price", Title: "accounting.Price", Doc: "", Fields: []Field{
+		{Key: "prompt_per_1k", Type: "int64", Doc: "", Nested: ""},
+		{Key: "completion_per_1k", Type: "int64", Doc: "", Nested: ""},
+	}},
+	{Package: "admission", Name: "Band", Title: "admission.Band", Doc: "", Fields: []Field{
+		{Key: "name", Type: "string", Doc: "", Nested: ""},
+		{Key: "priority", Type: "int", Doc: "Priority orders bands; higher is served first.", Nested: ""},
+		{Key: "saturation_ceiling", Type: "float64", Doc: "SaturationCeiling is the pool saturation above which this band waits.\n1.0 means \"never hold this band back\".", Nested: ""},
+	}},
+	{Package: "admission", Name: "Config", Title: "admission.Config", Doc: "", Fields: []Field{
+		{Key: "bands", Type: "[]Band", Doc: "", Nested: ""},
+		{Key: "default_band", Type: "string", Doc: "", Nested: ""},
+		{Key: "queue_ttl", Type: "time.Duration", Doc: "QueueTTL drops a request that has waited this long. Sending the GPU work\nthe client has already given up on helps nobody.", Nested: ""},
+		{Key: "queue_capacity_per_band", Type: "int", Doc: "QueueCapacityPerBand bounds each band's queue. Rejecting immediately\nbeats accumulating silently.", Nested: ""},
+		{Key: "queue_capacity_total", Type: "int", Doc: "QueueCapacityTotal bounds everything queued, across bands. When it is\nreached, a request is not refused while a lower band has requests\nqueued: the newest of the lowest band is turned away instead\n(displacement), since the oldest have waited longest and are nearest\ntheir TTL. Zero takes half of every band's capacity together.", Nested: ""},
+		{Key: "interval", Type: "time.Duration", Doc: "Interval is how often the dispatcher re-checks saturation. Releases also\nwake it, so this is only the backstop for saturation falling on its own.", Nested: ""},
+	}},
+	{Package: "agent", Name: "Config", Title: "agent.Config", Doc: "", Fields: []Field{
+		{Key: "node_id", Type: "string", Doc: "NodeID defaults to the hostname. It must equal the common name of the\nagent's certificate, so set it only where the two differ.", Nested: ""},
+		{Key: "server", Type: "string", Doc: "Server is the control plane, and Servers is the same thing when there is\nmore than one. Both are accepted: every configuration written before\nthere could be a second control plane names one address, and none of\nthem should have to change.\n\nThe order is tried in order, and a refusal moves to the next. Only the\nleader accepts an agent, so a standby is skipped in the time it takes to\nbe told so — which is what makes failover a reconnect rather than an\nedit to every node's configuration.", Nested: ""},
+		{Key: "servers", Type: "[]string", Doc: "", Nested: ""},
+		{Key: "hostname", Type: "string", Doc: "Hostname defaults to the machine's.", Nested: ""},
+		{Key: "mgmt_addr", Type: "string", Doc: "MgmtAddr is the address engines are reached on. It defaults to this\nmachine's address on the route to the control plane, which is the one\nthe control plane and the router can reach.", Nested: ""},
+		{Key: "labels", Type: "map[string]string", Doc: "Labels are free-form facts about a node for a profile's node_selector.\nNothing needs them: memory, GPUs, engines and disk are measured, and a\nlabel restating one of those goes stale the day the machine changes.", Nested: ""},
+		{Key: "mgmt_interface", Type: "string", Doc: "MgmtInterface is the ethernet interface NCCL bootstraps over. It\ndefaults to the interface holding MgmtAddr, then eth0.", Nested: ""},
+		{Key: "fabrics", Type: "[]inventory.FabricAttachment", Doc: "Fabrics labels this node's HCAs. It is explicit on purpose: OpenSM's\ndefault subnet prefix is the same on every fabric, so nothing can be\ninferred from the wire (plan §5.2).", Nested: ""},
+		{Key: "heartbeat_interval", Type: "time.Duration", Doc: "", Nested: ""},
+		{Key: "reconnect_base", Type: "time.Duration", Doc: "ReconnectBase is the first reconnect delay; it backs off from there.", Nested: ""},
+		{Key: "reconnect_max", Type: "time.Duration", Doc: "", Nested: ""},
+		{Key: "adopt_ready_timeout", Type: "time.Duration", Doc: "AdoptReadyTimeout bounds how long an adopted engine is given to answer a\nhealth check before the agent stops waiting on it.\n\nShorter than a startup timeout on purpose: this engine is already\nrunning, so it is not loading a model, it is either serving or it is not.", Nested: ""},
+		{Key: "tls", Type: "tlsconf.Config", Doc: "TLS authenticates both ends of the control-plane stream. Leaving it unset\nkeeps the plaintext behaviour, which only belongs on a trusted network.", Nested: "tlsconf.Config"},
+		{Key: "model_dirs", Type: "[]string", Doc: "ModelDirs are the directories this node keeps model weights in, beyond\nWeightsCache, which is always looked in. Storage differs at every installation,\nso llmgw observes it rather than owning it: a Lustre mount, an image\nexported over the fabric, local NVMe and a copy made by hand all look\nidentical from here, and placement only needs to know whether the file\nis present.", Nested: ""},
+		{Key: "engines_dir", Type: "string", Doc: "EnginesDir is where the agent installs engines, one directory per\nbuild. Defaults to /opt/llmgw/engines.", Nested: ""},
+		{Key: "weights_cache", Type: "string", Doc: "WeightsCache is where a remote source is downloaded to. Defaults to the\nfirst entry of ModelDirs, because a node that says where its models live\nhas already answered this question, and otherwise to\n/var/lib/llmgw-weights.\n\nFetching is the simple arrangement: every node keeps its own copy. The\nexotic ones need no code — weights on a shared filer or an image exported\nover the fabric are simply present, and present means nothing is fetched.", Nested: ""},
+		{Key: "discovery", Type: "discovery.Options", Doc: "", Nested: "discovery.Options"},
+		{Key: "engines", Type: "engines.Config", Doc: "", Nested: "engines.Config"},
+		{Key: "supervisor", Type: "supervisor.Config", Doc: "", Nested: "supervisor.Config"},
+	}},
+	{Package: "config", Name: "Band", Title: "config.Band", Doc: "", Fields: []Field{
+		{Key: "name", Type: "string", Doc: "", Nested: ""},
+		{Key: "priority", Type: "int", Doc: "", Nested: ""},
+		{Key: "saturation_ceiling", Type: "float64", Doc: "", Nested: ""},
+	}},
+	{Package: "config", Name: "Price", Title: "config.Price", Doc: "", Fields: []Field{
+		{Key: "prompt_per_1k", Type: "int64", Doc: "", Nested: ""},
+		{Key: "completion_per_1k", Type: "int64", Doc: "", Nested: ""},
+	}},
+	{Package: "engines", Name: "Config", Title: "engines.Config", Doc: "", Fields: []Field{
+		{Key: "disabled", Type: "[]string", Doc: "Disabled are engines this node never installs or advertises, whatever\nthe control plane asks: vLLM on a node of cards no build of it runs on,\nthat is meant for llama.cpp alone.", Nested: ""},
+		{Key: "vllm", Type: "VLLMConfig", Doc: "", Nested: "engines.VLLMConfig"},
+		{Key: "sglang", Type: "SGLangConfig", Doc: "", Nested: "engines.SGLangConfig"},
+		{Key: "llamacpp", Type: "LlamaCppConfig", Doc: "", Nested: "engines.LlamaCppConfig"},
+		{Key: "sim", Type: "SimConfig", Doc: "", Nested: "engines.SimConfig"},
+	}},
+	{Package: "supervisor", Name: "Config", Title: "supervisor.Config", Doc: "", Fields: []Field{
+		{Key: "work_dir", Type: "string", Doc: "WorkDir holds pid files and per-shard scratch space.", Nested: ""},
+		{Key: "log_lines", Type: "int", Doc: "LogLines is how many lines of engine output to retain per shard.", Nested: ""},
+		{Key: "stop_timeout", Type: "time.Duration", Doc: "StopTimeout is how long a process gets to exit after SIGTERM before it is\nkilled. Engines need time to free GPU memory cleanly.", Nested: ""},
+		{Key: "health_interval", Type: "time.Duration", Doc: "HealthInterval is how often a starting shard is polled.", Nested: ""},
+		{Key: "rendezvous_timeout", Type: "time.Duration", Doc: "RendezvousTimeout bounds the wait for a multi-node rendezvous point to\nstart listening.", Nested: ""},
+		{Key: "alive_grace", Type: "time.Duration", Doc: "AliveGrace is how long a non-serving rank must stay up before it counts\nas ready. There is no external signal for a worker, so staying alive is\nthe only evidence available.", Nested: ""},
+	}},
+	{Package: "tlsconf", Name: "Config", Title: "tlsconf.Config", Doc: "", Fields: []Field{
+		{Key: "ca", Type: "string", Doc: "CA is the bundle used to verify the other side. In a real deployment this\nis whichever authority issued the fleet's certificates.", Nested: ""},
+		{Key: "cert", Type: "string", Doc: "Cert and Key are this component's own identity.", Nested: ""},
+		{Key: "key", Type: "string", Doc: "", Nested: ""},
+		{Key: "insecure", Type: "bool", Doc: "Insecure disables TLS entirely. It exists for local runs against\nsim-engines; anything using it must stay on a trusted network.", Nested: ""},
+	}},
+}

internal/docgen/render.go

@@ -61,13 +61,13 @@ rather than repeated.
 	for _, bl := range blocks {
 		for _, f := range bl.Fields {
 			b.WriteString(fmt.Sprintf("| `%s` | %s | [%s](#%s) |\n",
-				f.Key, code(f.Type), bl.Name, anchor(bl.Name)))
+				f.Key, code(f.Type), bl.Title, anchor(bl.Title)))
 		}
 	}
 	b.WriteString("\n")
 
 	for _, bl := range blocks {
-		fmt.Fprintf(&b, "## %s\n\n", bl.Name)
+		fmt.Fprintf(&b, "## %s\n\n", bl.Title)
 		if bl.Doc != "" {
 			b.WriteString(bl.Doc + "\n\n")
 		}
@@ -76,24 +76,44 @@ rather than repeated.
 			if f.Doc != "" {
 				b.WriteString(f.Doc + "\n\n")
 			}
-			if f.Nested != "" && hasBlock(blocks, f.Nested) {
-				fmt.Fprintf(&b, "Takes a [%s](#%s) block.\n\n", f.Nested, anchor(f.Nested))
+			if nb, ok := Find(blocks, f.Nested); ok {
+				fmt.Fprintf(&b, "Takes a [%s](#%s) block.\n\n", nb.Title, anchor(nb.Title))
 			}
 		}
 	}
 	return b.String()
 }
 
-func hasBlock(blocks []Block, name string) bool {
+// Find is the block with an ID, as a Field's Nested names it.
+func Find(blocks []Block, id string) (Block, bool) {
 	for _, b := range blocks {
-		if b.Name == name {
-			return true
+		if id != "" && b.ID() == id {
+			return b, true
 		}
 	}
-	return false
+	return Block{}, false
 }
 
 func code(s string) string { return "`" + s + "`" }
 
-// anchor is how GitHub and Gitea slug a heading.
-func anchor(s string) string { return strings.ToLower(s) }
+// anchor is how GitHub and Gitea slug a heading: lower case, and punctuation
+// such as the dot in tlsconf.Config dropped.
+func anchor(s string) string { return strings.ToLower(strings.ReplaceAll(s, ".", "")) }
+
+// RenderGo writes the blocks as Go, for the settings screens: every key, its
+// type and what it does, from the same source as the reference, so a setting
+// added to the code is on the screens without anyone listing it.
+func RenderGo(blocks []Block) string {
+	var b strings.Builder
+	b.WriteString("// Code generated by go run ./internal/docgen/gen; DO NOT EDIT.\n\npackage docgen\n\n")
+	b.WriteString("// Reference is every configuration block, as Collect reads them.\nvar Reference = []Block{\n")
+	for _, bl := range blocks {
+		fmt.Fprintf(&b, "\t{Package: %q, Name: %q, Title: %q, Doc: %q, Fields: []Field{\n", bl.Package, bl.Name, bl.Title, bl.Doc)
+		for _, f := range bl.Fields {
+			fmt.Fprintf(&b, "\t\t{Key: %q, Type: %q, Doc: %q, Nested: %q},\n", f.Key, f.Type, f.Doc, f.Nested)
+		}
+		b.WriteString("\t}},\n")
+	}
+	b.WriteString("}\n")
+	return b.String()
+}

internal/doctor/doctor.go

@@ -10,6 +10,7 @@ import (
 	"context"
 	"errors"
 	"fmt"
+	"maps"
 	"net"
 	"os"
 	"os/exec"
@@ -86,10 +87,12 @@ type Facts struct {
 
 	// What installing vLLM here would meet: the version this llmgw installs,
 	// where, the space there, and whether it already has.
-	VLLMVersion   string  `json:"vllm_version,omitempty"`
-	EnginesDir    string  `json:"engines_dir,omitempty"`
-	EnginesFreeGB float64 `json:"engines_free_gb,omitempty"`
-	VLLMInstalled bool    `json:"vllm_installed"`
+	VLLMVersion string `json:"vllm_version,omitempty"`
+	// DisabledEngines are those agent.yaml says this node never runs.
+	DisabledEngines []string `json:"disabled_engines,omitempty"`
+	EnginesDir      string   `json:"engines_dir,omitempty"`
+	EnginesFreeGB   float64  `json:"engines_free_gb,omitempty"`
+	VLLMInstalled   bool     `json:"vllm_installed"`
 }
 
 // Modules checked, and why each matters.
@@ -103,6 +106,10 @@ type Options struct {
 	// VLLMVersion is the vLLM this llmgw installs; its prerequisites are
 	// checked when set.
 	VLLMVersion string
+	// Fabrics are fabrics about to be declared, by name, with their
+	// adapters: what `node add --fabric` will write into agent.yaml, which
+	// does not exist yet when doctor runs before it.
+	Fabrics map[string][]string
 }
 
 // Collect reads the node.
@@ -183,6 +190,14 @@ func Collect(ctx context.Context, o Options) Facts {
 	}
 
 	f.readConfig(o)
+	if len(o.Fabrics) > 0 {
+		declared := map[string][]string{}
+		for _, c := range f.Fabrics {
+			declared[c.Fabric] = c.Declared
+		}
+		maps.Copy(declared, o.Fabrics)
+		f.Fabrics = discovery.CheckFabrics(declared, f.HCAs)
+	}
 	f.Host.MinComputeCapability = discovery.MinComputeCapability(f.GPUs)
 	if f.VLLMVersion = o.VLLMVersion; f.VLLMVersion != "" {
 		if f.EnginesDir == "" {
@@ -238,6 +253,7 @@ func (f *Facts) readConfig(o Options) {
 	_, err = net.InterfaceByName(f.MgmtIface)
 	f.IfaceExists = err == nil
 	f.EnginesDir = cfg.EnginesDir
+	f.DisabledEngines = cfg.Engines.Disabled
 	f.LaunchMode = cfg.Engines.VLLM.LaunchMode
 	if f.LaunchMode == "" {
 		f.LaunchMode = "mp"
@@ -419,8 +435,15 @@ func Evaluate(f Facts) []Finding {
 			}
 			switch {
 			case p.Active() && p.LinkLayer == discovery.LinkEthernet && p.RoCEv2GID < 0:
-				add(Fail, "fabric", where+" is RoCE with no RoCE v2 GID entry: NCCL would use a v1 entry and hang",
-					"give the interface an IPv4 address, or check the adapter's RoCE mode")
+				// Fatal only for a port a declared fabric uses. An adapter
+				// nothing is configured to use — a RoCE NIC on a node added
+				// for single-node inference — kept nodes out for a fabric
+				// nobody asked for.
+				msg := where + " is RoCE with no RoCE v2 GID entry: NCCL would use a v1 entry and hang"
+				if declared[h.Name] == "" {
+					msg += "; no declared fabric uses it, so nothing will until one does"
+				}
+				add(level, "fabric", msg, "give the interface an IPv4 address, or check the adapter's RoCE mode")
 			case p.Active():
 				on := ""
 				switch {
@@ -430,17 +453,24 @@ func Evaluate(f Facts) []Finding {
 					on = ", network " + p.Network
 				}
 				add(OK, "fabric", fmt.Sprintf("%s %s at %d Gb/s (%s)%s", where, p.State, p.RateGbps, p.LinkLayer, on), "")
+				// Questions about which fabric a port is on matter to the
+				// fabrics that use it; for a port none does they are noted,
+				// not warned of.
+				quiet := Warn
+				if declared[h.Name] == "" {
+					quiet = Info
+				}
 				if p.LinkLayer == discovery.LinkInfiniBand && p.SM == nil {
 					why := f.SMErrors[where]
 					fix := ""
 					if strings.Contains(why, "infiniband-diags") {
 						fix = "install infiniband-diags"
 					}
-					add(Warn, "fabric", where+": which fabric it is on cannot be told — "+orUnknown(why)+
+					add(quiet, "fabric", where+": which fabric it is on cannot be told — "+orUnknown(why)+
 						"; without it, separate fabrics given one name cannot be caught", fix)
 				}
 				if p.LinkLayer == discovery.LinkInfiniBand && p.SMLID == 0 {
-					add(Warn, "fabric", where+" is active but reports no subnet manager LID", "")
+					add(quiet, "fabric", where+" is active but reports no subnet manager LID", "")
 				}
 			case strings.EqualFold(p.PhysState, "LinkUp"):
 				add(level, "fabric", where+" has a link but is "+p.State+": no subnet manager has configured it",
@@ -479,7 +509,14 @@ func Evaluate(f Facts) []Finding {
 	}
 	if len(f.HCAs) > 0 {
 		if !f.Modules["ib_uverbs"] {
-			add(Fail, "fabric", "ib_uverbs is not loaded: nothing in user space can use RDMA", "modprobe ib_uverbs")
+			// RDMA is for replicas spanning nodes; a node that declares no
+			// fabric runs none, and is a node without it.
+			if len(declared) > 0 {
+				add(Fail, "fabric", "ib_uverbs is not loaded: nothing in user space can use RDMA", "modprobe ib_uverbs")
+			} else {
+				add(Warn, "fabric", "ib_uverbs is not loaded: nothing in user space can use RDMA, which a fabric declared "+
+					"later would need; this node runs single-node replicas without it", "modprobe ib_uverbs")
+			}
 		}
 		if len(f.GPUs) > 0 && !f.Modules["nvidia_peermem"] && !f.Modules["nv_peer_mem"] {
 			add(Warn, "fabric", "nvidia_peermem is not loaded: GPUDirect RDMA is off, and NCCL copies every transfer through host memory",
@@ -525,7 +562,9 @@ func Evaluate(f Facts) []Finding {
 		}
 	}
 	// --- engines ---
-	if f.VLLMVersion != "" {
+	// An engine disabled on this node: whether it could be installed is not
+	// a question anybody is asking.
+	if f.VLLMVersion != "" && !slices.Contains(f.DisabledEngines, "vllm") {
 		vllmFindings(f, add)
 	}
 
@@ -542,28 +581,44 @@ func Evaluate(f Facts) []Finding {
 // the same decision the agent makes, so the report and the node agree.
 func vllmFindings(f Facts, add func(Level, string, string, string)) {
 	v := f.VLLMVersion
-	b, err := provision.Recipes["vllm"].Choose(v, f.Host)
+	builds, err := provision.Recipes["vllm"].Candidates(v, "", f.Host)
+	var b provision.Build
+	if err == nil {
+		// The first the agent would try; it moves to the next by itself if
+		// that one does not work here.
+		b = builds[0]
+		if b.Version != "" {
+			v = b.Version
+		}
+	}
 	var nh provision.NotHere
 	switch {
 	case errors.As(err, &nh):
 		add(Info, "engines", "vLLM will not be installed: "+nh.Reason, "")
 		return
 	case err != nil:
-		add(Fail, "engines", "vLLM "+v+" cannot be installed: "+err.Error(),
-			"upgrade the NVIDIA driver, or set the control plane's engines.vllm to a version for this one")
+		// A warning, not a failure: the node is still a node, and serves
+		// llama.cpp. Refusing it here kept older GPU generations out of a
+		// cluster for an engine nobody meant to run on them.
+		add(Warn, "engines", "vLLM "+v+" will not run here, and the node joins without it: "+err.Error(),
+			"add it with --disable-engine vllm if it is meant for llama.cpp alone")
 		return
 	case f.VLLMInstalled:
 		add(OK, "engines", "vLLM "+v+" is installed under "+f.EnginesDir, "")
 		return
 	}
 	if f.EnginesFreeGB > 0 && f.EnginesFreeGB < float64(b.FreeGB) {
-		add(Fail, "engines", fmt.Sprintf("vLLM %s needs about %d GB free under %s and there are %.1f GB",
+		add(Warn, "engines", fmt.Sprintf("vLLM %s needs about %d GB free under %s and there are %.1f GB; the node joins without it until there is",
 			v, b.FreeGB, f.EnginesDir, f.EnginesFreeGB),
 			"free space there, or point engines_dir in agent.yaml at a larger disk")
 		return
 	}
-	add(OK, "engines", fmt.Sprintf("vLLM %s (%s) can be installed here: driver %s, %.1f GB free under %s",
-		v, b.Variant, f.Host.NVIDIADriver, f.EnginesFreeGB, f.EnginesDir), "")
+	note := ""
+	if b.Note != "" {
+		note = "; " + b.Note
+	}
+	add(OK, "engines", fmt.Sprintf("vLLM %s (%s) can be installed here: driver %s, %.1f GB free under %s%s",
+		v, b.Variant, f.Host.NVIDIADriver, f.EnginesFreeGB, f.EnginesDir, note), "")
 }
 
 func rank(l Level) int {

internal/doctor/doctor_test.go

@@ -151,8 +151,21 @@ func TestDoctorSaysWhetherVLLMCanBeInstalled(t *testing.T) {
 		says  string
 	}{
 		{"CPU node", Facts{Host: discovery.Host{Arch: "amd64"}, VLLMVersion: "0.30.0"}, Info, "has none"},
-		{"old driver", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15"}, VLLMVersion: "0.30.0"}, Fail, "driver 580"},
-		{"small disk", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 9}, Fail, "9.0 GB"},
+		// What vLLM cannot do keeps vLLM off the node, not the node out of
+		// the cluster: it still serves llama.cpp.
+		{"a driver too old for either build", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "470.256.02"}, VLLMVersion: "0.30.0"}, Warn, "driver 525"},
+		// A driver too old for CUDA 13 runs the same version's CUDA 12.9 build.
+		{"turing, old driver", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.5},
+			VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "(cu129) can be installed"},
+		// A V100 has no build of vLLM 0.20 or later; it gets the newest
+		// that has Volta kernels, and doctor says which.
+		{"volta", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
+			VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "vLLM 0.19.1 (cu12) can be installed here"},
+		{"pascal", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 6.1},
+			VLLMVersion: "0.30.0"}, Warn, "no vLLM is built for GPUs of compute capability 6.1"},
+		{"volta, the last vLLM built for it", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
+			VLLMVersion: "0.19.1", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "(cu12) can be installed"},
+		{"small disk", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 9}, Warn, "9.0 GB"},
 		{"ready to install", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "can be installed"},
 		{"installed", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", VLLMInstalled: true}, OK, "is installed"},
 	}
@@ -200,3 +213,38 @@ func TestDoctorSaysWhenTheCPUHidesAVX2(t *testing.T) {
 		t.Errorf("ARM: %+v", f)
 	}
 }
+
+func TestAnEngineDisabledOnTheNodeIsNotChecked(t *testing.T) {
+	// The field: a V100 node meant for llama.cpp, refused for a vLLM it was
+	// never going to run.
+	f := Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 7.0},
+		VLLMVersion: "0.30.0", DisabledEngines: []string{"vllm"}}
+	for _, fd := range Evaluate(f) {
+		if fd.Area == "engines" {
+			t.Errorf("a disabled vLLM was reported on: %s %q", fd.Level, fd.Message)
+		}
+	}
+}
+
+func TestAnAdapterNoFabricUsesDoesNotKeepTheNodeOut(t *testing.T) {
+	// The field: a RoCE NIC with no v2 GID on a node added for single-node
+	// inference, no fabric given, and the node refused for it.
+	roce := discovery.HCA{Name: "mlx5_2", Ports: []discovery.Port{{Num: 1, State: "ACTIVE",
+		PhysState: "LinkUp", RateGbps: 100, LinkLayer: discovery.LinkEthernet, RoCEv2GID: -1}}}
+	f := healthy()
+	f.HCAs = append(f.HCAs, roce)
+	f.Modules["ib_uverbs"] = false
+	f.Fabrics = nil // nothing declared
+	fs := Evaluate(f)
+	if fails := levels(fs, Fail); len(fails) > 0 {
+		t.Errorf("an undeclared adapter failed the node: %v", fails)
+	}
+	has(t, fs, Warn, "no declared fabric uses it")
+	has(t, fs, Warn, "runs single-node replicas without it")
+
+	// Declared, the same adapter and module are what the fabric needs.
+	f.Fabrics = discovery.CheckFabrics(map[string][]string{"roce-a": {"mlx5_2"}}, f.HCAs)
+	fs = Evaluate(f)
+	has(t, fs, Fail, "mlx5_2/1 is RoCE with no RoCE v2 GID entry")
+	has(t, fs, Fail, "ib_uverbs is not loaded")
+}

internal/hf/hf.go

@@ -258,6 +258,41 @@ func (f File) IsWeights() bool {
 	return false
 }
 
+// companionNames are the files beside safetensors weights that an engine
+// loading the directory reads: the model's shape, its tokenizer, how it was
+// quantised, and which shard holds which tensor.
+var companionNames = map[string]bool{
+	"config.json": true, "generation_config.json": true,
+	"tokenizer.json": true, "tokenizer_config.json": true, "tokenizer.model": true,
+	"special_tokens_map.json": true, "added_tokens.json": true,
+	"vocab.json": true, "vocab.txt": true, "merges.txt": true,
+	"chat_template.jinja": true, "chat_template.json": true,
+	"preprocessor_config.json": true, "processor_config.json": true,
+	"quantize_config.json": true, "quant_config.json": true,
+}
+
+// Companions are the files of a repository an engine reads beside a
+// safetensors set: config.json and the tokenizer, the shard index. Only
+// those at the top: the directory an engine is given is one level deep.
+//
+// Never code (a .py): a model that needs its own Python to load asks the
+// engine to run what the repository ships, which is a decision for whoever
+// runs the engine, made on purpose, not a file swept in with the tokenizer.
+func Companions(files []File) []File {
+	var out []File
+	for _, f := range files {
+		name := f.Path
+		if strings.Contains(name, "/") {
+			continue
+		}
+		if companionNames[name] || strings.HasSuffix(name, ".safetensors.index.json") ||
+			strings.HasSuffix(name, ".tiktoken") {
+			out = append(out, f)
+		}
+	}
+	return out
+}
+
 // Repo is a repository's current revision and what is in it.
 type Repo struct {
 	ID string

internal/join/join.go

@@ -0,0 +1,302 @@
+// Package join lets a machine be set up without anything logging into it.
+//
+// The machine doing the setup — running as root, holding the authority —
+// listens for a while and hands out one code per machine it expects. On the
+// other machine `sudo llmgw join <code>` dials out to it, checks it is talking
+// to the listener the code names, proves itself with the code's secret, and
+// from then on runs what it is sent, as ssh would, until the setup closes
+// the connection. To the setup it is one more onboard.Remote: preflight,
+// Patroni and node onboarding run over it unchanged.
+//
+// For sites where nothing may log into a machine from outside: the
+// connection is outbound from the machine being set up, no password is
+// typed anywhere, and the code is good for one connection and a limited time.
+package join
+
+import (
+	"bufio"
+	"context"
+	"crypto/ecdsa"
+	"crypto/elliptic"
+	"crypto/rand"
+	"crypto/sha256"
+	"crypto/subtle"
+	"crypto/tls"
+	"crypto/x509"
+	"crypto/x509/pkix"
+	"encoding/base64"
+	"encoding/binary"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"math/big"
+	"net"
+	"strings"
+	"sync"
+	"time"
+)
+
+// DefaultPort is where a setup listens for machines joining it.
+const DefaultPort = 7444
+
+// codePrefix marks a join code, and its version.
+const codePrefix = "llmgw-join1-"
+
+// Code is what `llmgw join` is given: where to dial, how to know it reached
+// the right place, and the secret that admits it.
+//
+// Short, because somebody copies it from one screen into another: the first
+// version was JSON and over 250 characters, which a terminal cut off at its
+// edge. Packed it is under 90.
+type Code struct {
+	Addr string
+	// Fingerprint is the first 16 bytes of the SHA-256 of the listener's
+	// certificate: 128 bits, more than enough to find nothing else by.
+	Fingerprint [16]byte
+	Secret      [16]byte
+	Expires     int64
+	// Role says what the machine is being set up as, for its operator to
+	// read before it runs: "database", "node" or "control plane".
+	Role string
+}
+
+var roles = []string{"", "database", "node", "control plane"}
+
+// String is the code as typed: one word, safe in a shell.
+func (c Code) String() string {
+	b := make([]byte, 0, 40+len(c.Addr))
+	b = append(b, c.Fingerprint[:]...)
+	b = append(b, c.Secret[:]...)
+	b = binary.BigEndian.AppendUint32(b, uint32(c.Expires))
+	role := byte(0)
+	for i, r := range roles {
+		if r == c.Role {
+			role = byte(i)
+		}
+	}
+	b = append(b, role)
+	b = append(b, c.Addr...)
+	return codePrefix + base64.RawURLEncoding.EncodeToString(b)
+}
+
+// Parse reads a code back.
+func Parse(s string) (Code, error) {
+	raw, ok := strings.CutPrefix(strings.TrimSpace(s), codePrefix)
+	if !ok {
+		return Code{}, errors.New("not a join code: it starts " + codePrefix)
+	}
+	b, err := base64.RawURLEncoding.DecodeString(raw)
+	if err != nil || len(b) < 16+16+4+1+3 {
+		return Code{}, errors.New("the join code is damaged: copy it whole")
+	}
+	var c Code
+	copy(c.Fingerprint[:], b[:16])
+	copy(c.Secret[:], b[16:32])
+	c.Expires = int64(binary.BigEndian.Uint32(b[32:36]))
+	if int(b[36]) < len(roles) {
+		c.Role = roles[b[36]]
+	}
+	c.Addr = string(b[37:])
+	if _, _, err := net.SplitHostPort(c.Addr); err != nil {
+		return Code{}, errors.New("the join code is damaged: copy it whole")
+	}
+	if time.Now().Unix() > c.Expires {
+		return Code{}, fmt.Errorf("the join code expired at %s; ask for a new one",
+			time.Unix(c.Expires, 0).Format(time.RFC3339))
+	}
+	return c, nil
+}
+
+// The wire: one JSON object a line, each way.
+type hello struct {
+	Secret   []byte `json:"secret"`
+	Hostname string `json:"hostname"`
+	// Addr is the address the machine dialled out from: how the others
+	// reach it, when nothing else says.
+	Addr string `json:"addr"`
+}
+
+type request struct {
+	Script string `json:"script,omitempty"`
+	Stdin  []byte `json:"stdin,omitempty"`
+	Bye    bool   `json:"bye,omitempty"`
+}
+
+type response struct {
+	Stdout []byte `json:"stdout,omitempty"`
+	Stderr []byte `json:"stderr,omitempty"`
+	Exit   int    `json:"exit"`
+	Error  string `json:"error,omitempty"`
+}
+
+// Listener waits for machines to join.
+type Listener struct {
+	ln          net.Listener
+	addr        string
+	fingerprint [16]byte
+	ttl         time.Duration
+
+	mu      sync.Mutex
+	pending map[[16]byte]chan *Remote // by secret
+}
+
+// Listen starts listening on bind (":7444") and advertises addr ("10.0.0.5:7444")
+// in its codes. Codes are good for ttl.
+func Listen(bind, addr string, ttl time.Duration) (*Listener, error) {
+	cert, fp, err := sessionCert()
+	if err != nil {
+		return nil, err
+	}
+	ln, err := tls.Listen("tcp", bind, &tls.Config{Certificates: []tls.Certificate{cert}, MinVersion: tls.VersionTLS13})
+	if err != nil {
+		return nil, fmt.Errorf("listening for machines to join on %s: %w", bind, err)
+	}
+	l := &Listener{ln: ln, addr: addr, fingerprint: fp, ttl: ttl, pending: map[[16]byte]chan *Remote{}}
+	go l.accept()
+	return l, nil
+}
+
+// sessionCert is a certificate for this listener alone, trusted by the
+// fingerprint in its codes rather than by any authority: it lives as long as
+// the setup does.
+func sessionCert() (tls.Certificate, [16]byte, error) {
+	var fp [16]byte
+	key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
+	if err != nil {
+		return tls.Certificate{}, fp, err
+	}
+	tmpl := &x509.Certificate{
+		SerialNumber: big.NewInt(time.Now().UnixNano()),
+		Subject:      pkix.Name{CommonName: "llmgw setup"},
+		NotBefore:    time.Now().Add(-time.Minute),
+		NotAfter:     time.Now().Add(48 * time.Hour),
+		KeyUsage:     x509.KeyUsageDigitalSignature,
+		ExtKeyUsage:  []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
+	}
+	der, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, &key.PublicKey, key)
+	if err != nil {
+		return tls.Certificate{}, fp, err
+	}
+	sum := sha256.Sum256(der)
+	copy(fp[:], sum[:16])
+	return tls.Certificate{Certificate: [][]byte{der}, PrivateKey: key}, fp, nil
+}
+
+// Code makes a code for one machine, and the channel its connection arrives
+// on.
+func (l *Listener) Code(role string) (Code, <-chan *Remote) {
+	var secret [16]byte
+	_, _ = rand.Read(secret[:])
+	ch := make(chan *Remote, 1)
+	l.mu.Lock()
+	l.pending[secret] = ch
+	l.mu.Unlock()
+	return Code{Addr: l.addr, Fingerprint: l.fingerprint, Secret: secret,
+		Expires: time.Now().Add(l.ttl).Unix(), Role: role}, ch
+}
+
+// Close stops listening; machines already joined stay connected until their
+// Remote is closed.
+func (l *Listener) Close() error { return l.ln.Close() }
+
+func (l *Listener) accept() {
+	for {
+		c, err := l.ln.Accept()
+		if err != nil {
+			return
+		}
+		go l.admit(c)
+	}
+}
+
+func (l *Listener) admit(c net.Conn) {
+	_ = c.SetDeadline(time.Now().Add(30 * time.Second))
+	r := bufio.NewReader(c)
+	var h hello
+	line, err := r.ReadBytes('\n')
+	if err != nil || json.Unmarshal(line, &h) != nil {
+		_ = c.Close()
+		return
+	}
+	l.mu.Lock()
+	var ch chan *Remote
+	for secret, p := range l.pending {
+		if subtle.ConstantTimeCompare(secret[:], h.Secret) == 1 {
+			ch = p
+			// One machine a code: whoever comes second is turned away.
+			delete(l.pending, secret)
+		}
+	}
+	l.mu.Unlock()
+	if ch == nil {
+		_, _ = c.Write([]byte(`{"error":"this code is not one the setup is waiting for, or it was used already"}` + "\n"))
+		_ = c.Close()
+		return
+	}
+	_ = c.SetDeadline(time.Time{})
+	if _, err := c.Write([]byte("{}\n")); err != nil {
+		_ = c.Close()
+		return
+	}
+	addr := h.Addr
+	if addr == "" {
+		addr, _, _ = net.SplitHostPort(c.RemoteAddr().String())
+	}
+	ch <- &Remote{conn: c, r: r, Hostname: h.Hostname, Addr: addr}
+}
+
+// Remote is a joined machine, as the setup runs things on it.
+type Remote struct {
+	Hostname string
+	Addr     string
+
+	mu   sync.Mutex
+	conn net.Conn
+	r    *bufio.Reader
+}
+
+// Run runs script with sh on the machine, as root.
+func (m *Remote) Run(ctx context.Context, script string, stdin []byte) ([]byte, error) {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+	if m.conn == nil {
+		return nil, fmt.Errorf("%s is no longer connected", m.Hostname)
+	}
+	if dl, ok := ctx.Deadline(); ok {
+		_ = m.conn.SetDeadline(dl)
+		defer m.conn.SetDeadline(time.Time{})
+	}
+	b, _ := json.Marshal(request{Script: script, Stdin: stdin})
+	if _, err := m.conn.Write(append(b, '\n')); err != nil {
+		return nil, fmt.Errorf("%s: %w", m.Hostname, err)
+	}
+	line, err := m.r.ReadBytes('\n')
+	if err != nil {
+		return nil, fmt.Errorf("%s went away: %w", m.Hostname, err)
+	}
+	var resp response
+	if err := json.Unmarshal(line, &resp); err != nil {
+		return nil, fmt.Errorf("%s answered something unreadable", m.Hostname)
+	}
+	if resp.Error != "" {
+		return append(resp.Stdout, resp.Stderr...), errors.New(resp.Error)
+	}
+	if resp.Exit != 0 {
+		return append(resp.Stdout, resp.Stderr...), fmt.Errorf("exit status %d", resp.Exit)
+	}
+	return resp.Stdout, nil
+}
+
+// Close tells the machine the setup is done, which ends its `llmgw join`.
+func (m *Remote) Close() error {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+	if m.conn == nil {
+		return nil
+	}
+	b, _ := json.Marshal(request{Bye: true})
+	_, _ = m.conn.Write(append(b, '\n'))
+	err := m.conn.Close()
+	m.conn = nil
+	return err
+}

internal/join/join_test.go

@@ -0,0 +1,94 @@
+package join
+
+import (
+	"context"
+	"strings"
+	"testing"
+	"time"
+)
+
+func listen(t *testing.T) *Listener {
+	t.Helper()
+	l, err := Listen("127.0.0.1:0", "", time.Hour)
+	if err != nil {
+		t.Fatal(err)
+	}
+	l.addr = l.ln.Addr().String()
+	t.Cleanup(func() { _ = l.Close() })
+	return l
+}
+
+func TestAJoinedMachineRunsWhatTheSetupSendsUntilItIsDone(t *testing.T) {
+	l := listen(t)
+	code, arrived := l.Code("database")
+	parsed, err := Parse(code.String())
+	if err != nil || parsed.Role != "database" {
+		t.Fatalf("%v %+v", err, parsed)
+	}
+	var said []string
+	served := make(chan error, 1)
+	go func() { served <- Serve(context.Background(), parsed, func(s string) { said = append(said, s) }) }()
+
+	var m *Remote
+	select {
+	case m = <-arrived:
+	case err := <-served:
+		t.Fatalf("join ended before arriving: %v", err)
+	case <-time.After(5 * time.Second):
+		t.Fatal("the machine never arrived")
+	}
+	if m.Hostname == "" || m.Addr != "127.0.0.1" {
+		t.Errorf("arrived as %q from %q", m.Hostname, m.Addr)
+	}
+	out, err := m.Run(context.Background(), "set -e\ntr a-z A-Z", []byte("hello\n"))
+	if err != nil || string(out) != "HELLO\n" {
+		t.Fatalf("%q, %v", out, err)
+	}
+	if out, err := m.Run(context.Background(), "echo nope >&2; exit 3", nil); err == nil || !strings.Contains(string(out), "nope") {
+		t.Errorf("a failing script: %q, %v", out, err)
+	}
+	_ = m.Close()
+	if err := <-served; err != nil {
+		t.Errorf("join did not end cleanly when the setup was done: %v", err)
+	}
+	if len(said) < 2 || !strings.Contains(strings.Join(said, "\n"), "tr a-z A-Z") {
+		t.Errorf("the machine's operator was not told what ran: %q", said)
+	}
+
+	// The code was for one machine.
+	if err := Serve(context.Background(), parsed, func(string) {}); err == nil || !strings.Contains(err.Error(), "used already") {
+		t.Errorf("a code used twice: %v", err)
+	}
+}
+
+func TestAJoinRefusesAListenerThatIsNotTheOneInTheCode(t *testing.T) {
+	l := listen(t)
+	code, _ := l.Code("node")
+	other := listen(t)
+	code.Addr = other.addr // what the code names is elsewhere
+	err := Serve(context.Background(), code, func(string) {})
+	if err == nil || !strings.Contains(err.Error(), "not the setup") {
+		t.Errorf("joined a stranger: %v", err)
+	}
+}
+
+func TestAnExpiredOrDamagedCodeIsRefusedBeforeDialling(t *testing.T) {
+	c := Code{Addr: "10.0.0.5:7444", Role: "node", Expires: time.Now().Add(-time.Minute).Unix()}
+	if _, err := Parse(c.String()); err == nil || !strings.Contains(err.Error(), "expired") {
+		t.Errorf("expired: %v", err)
+	}
+	if _, err := Parse(c.String()[:30]); err == nil {
+		t.Error("a truncated code was read")
+	}
+	if _, err := Parse("ssh-rsa AAAA"); err == nil {
+		t.Error("something else was read as a code")
+	}
+	// Short enough to copy from a screen in one line.
+	c.Expires = time.Now().Add(time.Hour).Unix()
+	if n := len(c.String()); n > 100 {
+		t.Errorf("a code is %d characters", n)
+	}
+	if back, err := Parse(c.String()); err != nil || back.Addr != c.Addr || back.Role != "node" {
+		t.Errorf("%+v, %v", back, err)
+	}
+}

internal/join/serve.go

@@ -0,0 +1,126 @@
+package join
+
+import (
+	"bufio"
+	"bytes"
+	"context"
+	"crypto/sha256"
+	"crypto/subtle"
+	"crypto/tls"
+	"crypto/x509"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"net"
+	"os"
+	"os/exec"
+	"strings"
+	"time"
+)
+
+// Serve dials the setup a code names and runs what it sends until it says it
+// is done. Each script is reported to log by its first line, so whoever ran
+// `llmgw join` sees what is being done to their machine.
+func Serve(ctx context.Context, c Code, log func(string)) error {
+	d := tls.Dialer{
+		NetDialer: &net.Dialer{Timeout: 10 * time.Second},
+		Config: &tls.Config{
+			MinVersion: tls.VersionTLS13,
+			// Trusted by the fingerprint in the code, not by an authority:
+			// the listener's certificate is made for this setup alone.
+			InsecureSkipVerify: true, //nolint:gosec // verified below, by fingerprint
+		},
+	}
+	want := c.Fingerprint[:]
+	d.Config.VerifyPeerCertificate = func(raw [][]byte, _ [][]*x509.Certificate) error {
+		if len(raw) == 0 {
+			return errors.New("the setup sent no certificate")
+		}
+		sum := sha256.Sum256(raw[0])
+		if subtle.ConstantTimeCompare(sum[:16], want) != 1 {
+			return errors.New("the machine at that address is not the setup this code is for")
+		}
+		return nil
+	}
+	conn, err := d.DialContext(ctx, "tcp", c.Addr)
+	if err != nil {
+		return fmt.Errorf("cannot reach the setup at %s: %w; it listens there only while it waits for this machine", c.Addr, err)
+	}
+	defer conn.Close()
+
+	host, _ := os.Hostname()
+	local, _, _ := net.SplitHostPort(conn.LocalAddr().String())
+	b, _ := json.Marshal(hello{Secret: c.Secret[:], Hostname: host, Addr: local})
+	if _, err := conn.Write(append(b, '\n')); err != nil {
+		return err
+	}
+	r := bufio.NewReader(conn)
+	line, err := r.ReadBytes('\n')
+	if err != nil {
+		return fmt.Errorf("the setup closed the connection: %w", err)
+	}
+	var ack response
+	if err := json.Unmarshal(line, &ack); err != nil {
+		return errors.New("the setup answered something unreadable")
+	}
+	if ack.Error != "" {
+		return errors.New(ack.Error)
+	}
+	log(fmt.Sprintf("joined the setup at %s; it is setting this machine up", c.Addr))
+
+	for {
+		line, err := r.ReadBytes('\n')
+		if err != nil {
+			return fmt.Errorf("the setup went away before it said it was done: %w", err)
+		}
+		var req request
+		if err := json.Unmarshal(line, &req); err != nil {
+			return errors.New("the setup sent something unreadable")
+		}
+		if req.Bye {
+			log("the setup is done with this machine")
+			return nil
+		}
+		log(summary(req.Script))
+		resp := run(ctx, req)
+		out, _ := json.Marshal(resp)
+		if _, err := conn.Write(append(out, '\n')); err != nil {
+			return err
+		}
+	}
+}
+
+// run runs one script as the setup asked.
+func run(ctx context.Context, req request) response {
+	cmd := exec.CommandContext(ctx, "sh", "-c", req.Script)
+	if req.Stdin != nil {
+		cmd.Stdin = bytes.NewReader(req.Stdin)
+	}
+	var stdout, stderr bytes.Buffer
+	cmd.Stdout, cmd.Stderr = &stdout, &stderr
+	err := cmd.Run()
+	resp := response{Stdout: stdout.Bytes(), Stderr: stderr.Bytes()}
+	var exit *exec.ExitError
+	switch {
+	case errors.As(err, &exit):
+		resp.Exit = exit.ExitCode()
+	case err != nil:
+		resp.Error = err.Error()
+	}
+	return resp
+}
+
+// summary is a script's first line that says something.
+func summary(script string) string {
+	for _, l := range strings.Split(script, "\n") {
+		l = strings.TrimSpace(l)
+		if l == "" || l == "set -e" || strings.HasPrefix(l, "#") {
+			continue
+		}
+		if len(l) > 100 {
+			l = l[:100] + "…"
+		}
+		return "  " + l
+	}
+	return "  (an empty script)"
+}

internal/mirror/add.go

@@ -94,6 +94,13 @@ func addWith(ctx context.Context, client *http.Client, root, source string, e En
 		_ = os.Remove(part)
 		return Added{}, err
 	}
+	// The checksum beside the file, before the file is in place: a manifest
+	// listing the file then always has its checksum to hand on, and a node
+	// can tell a file damaged on the way from the one that was mirrored.
+	if err := writeChecksum(final, sum); err != nil {
+		_ = os.Remove(part)
+		return Added{}, err
+	}
 	if err := os.Rename(part, final); err != nil {
 		return Added{}, err
 	}
@@ -146,6 +153,9 @@ func Remove(root string, e Entry) error {
 	if err := os.Remove(path); err != nil {
 		return err
 	}
+	// Its checksum goes with it; left, it would keep the directory from
+	// being empty, and so from going.
+	_ = os.Remove(checksumPath(path))
 	versionDir := filepath.Dir(path)
 	if empty, _ := isEmpty(versionDir); empty {
 		_ = os.Remove(versionDir)

internal/mirror/mirror.go

@@ -16,12 +16,15 @@ package mirror
 import (
 	"crypto/sha256"
 	"encoding/hex"
+	"encoding/json"
 	"errors"
 	"fmt"
 	"io"
 	"os"
 	"path/filepath"
+	"regexp"
 	"sort"
+	"strconv"
 	"strings"
 )
 
@@ -91,7 +94,7 @@ func List(root string) ([]Entry, error) {
 				continue
 			}
 			for _, f := range files {
-				if f.IsDir() || strings.HasSuffix(f.Name(), partSuffix) {
+				if f.IsDir() || strings.HasSuffix(f.Name(), partSuffix) || strings.HasPrefix(f.Name(), ".") {
 					continue
 				}
 				info, err := f.Info()
@@ -169,6 +172,31 @@ func normalizeURL(u string) string {
 
 const partSuffix = ".part"
 
+// checksumPath is where a file's sha256 is kept: beside it, hidden, so no
+// listing takes it for a file of the model and no name a person gives a
+// file (which may not start with a dot) can be the same.
+func checksumPath(file string) string {
+	return filepath.Join(filepath.Dir(file), "."+filepath.Base(file)+".sha256")
+}
+
+func writeChecksum(file, sum string) error {
+	tmp := checksumPath(file) + partSuffix
+	if err := os.WriteFile(tmp, []byte(sum+"\n"), 0o644); err != nil { //nolint:gosec // a checksum, for anyone to read
+		return err
+	}
+	return os.Rename(tmp, checksumPath(file))
+}
+
+// readChecksum is a file's recorded sha256, or "" for a file put in place by
+// hand, which has none.
+func readChecksum(file string) string {
+	b, err := os.ReadFile(checksumPath(file)) //nolint:gosec // beside a file in the mirror
+	if err != nil {
+		return ""
+	}
+	return strings.TrimSpace(string(b))
+}
+
 // sha256File is the checksum of a file on disk.
 //
 // Reported after a download so an operator has something to compare against
@@ -205,8 +233,16 @@ type Manifest struct {
 type ManifestFile struct {
 	Name  string `json:"name"`
 	Bytes int64  `json:"bytes"`
+	// SHA256 is the file's checksum when the mirror fetched it, for a node
+	// to check what it downloaded; empty for a file put there by hand.
+	SHA256 string `json:"sha256,omitempty"`
 }
 
+// ErrIncomplete is a version not yet whole: a download still running, or a
+// shard its own index names missing. Its manifest is not given out, so no
+// node starts a model from part of it.
+var ErrIncomplete = errors.New("this version is not complete")
+
 // ManifestFor lists the files of one model version.
 //
 // Built from the directory rather than from a file written alongside it. A
@@ -219,15 +255,79 @@ func ManifestFor(root, model, version string) (Manifest, error) {
 		return Manifest{}, err
 	}
 	m := Manifest{Model: model, Version: version}
+	dir := filepath.Join(root, model, version)
 	for _, e := range entries {
 		if e.Model != model || e.Version != version {
 			continue
 		}
-		m.Files = append(m.Files, ManifestFile{Name: e.File, Bytes: e.Bytes})
+		m.Files = append(m.Files, ManifestFile{Name: e.File, Bytes: e.Bytes,
+			SHA256: readChecksum(filepath.Join(dir, e.File))})
 		m.Bytes += e.Bytes
 	}
 	if len(m.Files) == 0 {
 		return Manifest{}, fmt.Errorf("%s/%s: %w", model, version, os.ErrNotExist)
 	}
+	if err := complete(dir, m.Files); err != nil {
+		return Manifest{}, fmt.Errorf("%s/%s: %w: %v", model, version, ErrIncomplete, err)
+	}
 	return m, nil
 }
+
+// complete says whether a version's files are the whole of it: nothing still
+// downloading, every shard a safetensors index maps a tensor to, and every
+// part of a split ("-00002-of-00004").
+//
+// From what the files say of each other rather than from a list written when
+// the fetch began, so a version filled by hand is held to the same.
+func complete(dir string, files []ManifestFile) error {
+	if parts, _ := filepath.Glob(filepath.Join(dir, "*"+partSuffix)); len(parts) > 0 {
+		var names []string
+		for _, p := range parts {
+			if b := filepath.Base(p); !strings.HasPrefix(b, ".") {
+				names = append(names, strings.TrimSuffix(b, partSuffix))
+			}
+		}
+		if len(names) > 0 {
+			return fmt.Errorf("still downloading %s", strings.Join(names, ", "))
+		}
+	}
+	have := map[string]bool{}
+	for _, f := range files {
+		have[f.Name] = true
+	}
+	var missing []string
+	want := map[string]bool{}
+	for _, f := range files {
+		if strings.HasSuffix(f.Name, ".safetensors.index.json") {
+			var idx struct {
+				WeightMap map[string]string `json:"weight_map"`
+			}
+			b, err := os.ReadFile(filepath.Join(dir, f.Name)) //nolint:gosec // a file in the mirror
+			if err != nil || json.Unmarshal(b, &idx) != nil {
+				return fmt.Errorf("%s cannot be read", f.Name)
+			}
+			for _, shard := range idx.WeightMap {
+				want[shard] = true
+			}
+		}
+		if m := splitPart.FindStringSubmatch(f.Name); m != nil {
+			total, _ := strconv.Atoi(m[3])
+			for i := 1; i <= total; i++ {
+				want[fmt.Sprintf("%s-%0*d-of-%s%s", m[1], len(m[2]), i, m[3], m[4])] = true
+			}
+		}
+	}
+	for name := range want {
+		if !have[name] {
+			missing = append(missing, name)
+		}
+	}
+	if len(missing) > 0 {
+		sort.Strings(missing)
+		return fmt.Errorf("missing %s", strings.Join(missing, ", "))
+	}
+	return nil
+}
+
+// splitPart matches one part of a split: prefix, part, total, extension.
+var splitPart = regexp.MustCompile(`^(.+)-(\d{5})-of-(\d{5})(\.[A-Za-z0-9]+)$`)