Merge ha: replicated store, second control plane, join codes, setup and console screens

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

CHANGELOG.md

@@ -15,14 +15,316 @@ is only released once its **exit criterion** is met, not when its code lands.
 | v0.8.0 | — | PostgreSQL, leader election, agent failover | released |
 | v0.9.0 | — | Managing people and the tenancy tree | released |
 | v0.10.0 | — | InfiniBand fabrics and NVLink, doctor, node add, engines installed by the nodes | released |
+| v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released |
 | v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
 
 Until v1.0.0 the minor version is the breaking one: configuration formats and Go
 APIs change between minors without a deprecation period.
 
-## Unreleased
+## v0.11.0 — 2026-10-03
 
-From the field report in issue #2, after a deployment on GPU nodes.
+A minor version: the store can be replicated and the control plane doubled,
+both set up by llmgw itself; `sudo llmgw` opens setup screens and a console;
+usage is read from sums kept as it is written. Released without candidates.
+
+**Upgrading.** Migrations 0015 (usage sums, filled from the request table
+already recorded — once, at the first start) and 0016 (control planes) run by
+themselves. An agent older than this ignores the list of control planes and
+keeps its configured one. Nothing in `server.yaml` or `agent.yaml` changes
+meaning; `auth.usage_retention` and `router.tenant_label` are new and optional.
+
+The GPU and fabric work below began from the field report in issue #2, after
+a deployment on GPU nodes.
+
+### GPU metrics that move, and charts of them
+
+A GPU's utilisation and memory in use were read once, when the agent
+started, and published from then on: a card busy since looked as idle as it
+was at registration, and one idle since as busy. They are now sampled with
+the temperatures and power every heartbeat (ten seconds at most), with the
+memory bandwidth in use, the SM clock and uncorrected ECC errors besides.
+An agent too old to send them still has its registration figures published,
+and a figure the driver does not give is left out rather than shown as zero.
+
+- **New series**: `llmgw_gpu_memory_utilization_percent`,
+  `llmgw_gpu_sm_clock_mhz`; `llmgw_gpu_utilization_percent`,
+  `llmgw_gpu_memory_used_bytes` and `llmgw_gpu_ecc_errors` now from the
+  sample.
+- **The node page** shows each card's utilisation, memory, clock and ECC
+  errors in its table, and charts of utilisation, memory, temperature and
+  power, one line per card.
+- **The cluster page** charts the fleet's GPUs by class: average utilisation
+  and memory in use.
+- **Grafana** (`deploy/monitoring/grafana/llmgw-overview.json`) has a GPU row:
+  utilisation, memory, memory bandwidth, temperature, power against the
+  limit, clock, and what holds cards back.
+- **Alerts**: `GPUThermalThrottling` (ten minutes slowed by heat),
+  `GPUUncorrectedECCErrors`, `GPUHot` (above 87 °C for ten minutes).
+- **The cluster page's GPU map**: every card a square, a row per node,
+  coloured by how busy, how full or how hot it is (`?gpumap=util|memory|temp`),
+  with what holds it and its figures under the pointer; free cards dashed.
+- **Cards held and doing nothing**: the replicas whose every card reads under
+  5% busy, with when they were last asked — the first place to find room.
+- **What each model gets from its cards**, on the cluster page and in
+  Grafana: output tokens a second per GPU and per watt-hour, with the cards'
+  draw and average utilisation, over the last quarter hour.
+- **What wears a card out**, read every minute: XID errors from the kernel's
+  log (counted from when the agent started), memory rows remapped or pages
+  retired and whether either waits for a reset, a card out of spare rows,
+  PCIe link retries and NVLink errors. Published as `llmgw_gpu_xid_errors`,
+  `llmgw_gpu_last_xid`, `llmgw_gpu_remapped_rows`, `llmgw_gpu_remap_pending`,
+  `llmgw_gpu_remap_failure`, `llmgw_gpu_retired_pages`,
+  `llmgw_gpu_retirement_pending`, `llmgw_gpu_pcie_replays`,
+  `llmgw_gpu_nvlink_errors`; said in words on the node page; a red outline
+  on the GPU map; a Grafana panel; and alerts — `GPUXidError`,
+  `GPUOutOfSpareRows`, `GPUNeedsReset`, `GPUPCIeRetriesClimbing`,
+  `GPUNVLinkErrorsClimbing`.
+- **GPU time by owner** on the usage page: the cards each owner's models
+  held over the window, charted and summed into GPU-hours, whether or not the
+  models were asked anything — the time a GPU quota is about. From the
+  metrics store, as far back as it keeps history.
+- **`llmgw_gpu_assigned{node,gpu,model,replica,owner}`**: which replica holds
+  each card and whose quota it counts against, for joining the GPU series to
+  models and tenants.
+
+### Usage is read from sums kept as it is written
+
+Every chart and total — the usage tab, quotas, the dashboard, a person's own
+page, and budgets restored at start — summed the request table again on each
+view: a row per request, scanned whole. Usage is now also summed as it is
+written, in the same transaction, into five-minute slices, hours and days per
+key, person, project and model (`usage_5m`, `usage_1h`, `usage_1d`; migration
+0015 fills them from what is already recorded).
+
+- **Charts** read the coarsest of the three their slices are made of: a day in
+  20-minute slices from the five-minute table, a week or a month from hours.
+- **Totals and budgets** read the rows up to the next five-minute slice, then
+  slices, hours and days — exact, not rounded to a slice. On three million
+  requests over thirty days, a month's total per person went from 705 ms to
+  18 ms on PostgreSQL.
+- **Latency and time to first token** are kept as histograms in the sums, so a
+  percentile per team or model can be read after the rows are gone.
+- **`auth.usage_retention`** (e.g. `2160h`) deletes request records older than
+  that, hourly. Totals, charts and budgets are unchanged by it — they read the
+  sums — but `ctl request` finds nothing older. Unset keeps every record, as
+  before.
+
+### Alerts per tenant and per budget
+
+- **`llmgw_tenant_requests_total{tenant,outcome}`**, `llmgw_tenant_tokens_total`
+  and `llmgw_tenant_ttft_seconds`: requests by outcome (ok, error, limited),
+  tokens and time to first token per tenant. The tenant is the level
+  `router.tenant_label` names — `team` by default, or `org`, `project`, or
+  `none` — named by its path, "acme/platform". One level rather than every
+  project, so a cluster of thousands of projects does not become a Prometheus
+  of millions of series.
+- **`llmgw_budget_spent_ratio{scope,id,name,period}`**: this period's spend
+  over the budget for every organization, team, project and person given one,
+  refreshed every minute.
+- **Alerts** in `deploy/monitoring/llmgw-rules.yml`: `BudgetMostlySpent` (80%),
+  `BudgetSpent`, `TenantHittingLimits` (a tenth of requests refused),
+  `TenantErrors`, `TenantTrafficSpike` (three times the weekly average) and
+  `TenantSlowFirstToken`.
+- **A Grafana dashboard per tenant**, `deploy/monitoring/grafana/llmgw-tenants.json`
+  (uid `llmgw-tenants`): requests and tokens by tenant, the share failing and
+  refused by a limit, first-token p95, every budget's share spent, and who used
+  most in the last day — with a tenant chooser at the top.
+
+### How each team's requests fared, not only what they cost
+
+- **Percentiles from the sums.** First token and latency, p50, p95 and p99,
+  per organization, team, project, person or model, read from the histograms
+  kept with the usage sums: on the usage tab's table, and in a branch's or
+  person's panel on the people page, beside the error rate and requests
+  refused by a limit.
+- **Refused requests are counted against their owner.** A request a rate,
+  token or budget limit refused was counted only in Prometheus, by model; a
+  team running into its limit looked like a team that had gone quiet. It is
+  now recorded as a 429 against its key, person and project — no tokens, no
+  cost — and shown as "Limited".
+- **Usage by model** is a grouping of the usage tab, by the model that
+  answered.
+### The console watches the cluster and holds every setting
+
+`sudo llmgw` on a control plane is a console in three groups down the left:
+
+- **Watch**, refreshed every three seconds: the overview; nodes, with CPU,
+  memory, GPUs free, disk and agent version, and a node's detail — load,
+  engines, each card's busy, memory, temperature and power; every GPU in the
+  fleet and which model holds it; models, with a model's profiles, warnings
+  and replicas; replicas and where they run; the last day's use by team,
+  project or organization; and a node's agent log.
+- **Settings**: every key of `server.yaml`, block by block, with what applies
+  when the file sets nothing and what each key is for, from its comment in
+  the code. Enter changes a key or opens a block; lists and maps of blocks
+  (endpoints, models, prices) take entries added and removed. A save is read
+  by the same loader the service uses first — nothing is written that would
+  not load — keeps the file's comments, leaves the old one beside it, and
+  offers to restart the service.
+- **Do**: add a node, replicate the store, leave.
+
+On a node, `sudo llmgw` opens the same settings for its `agent.yaml`.
+
+The keys come from the configuration structs themselves, through the
+generator that writes `docs/reference.md`, so a setting added to the code is on
+the screen without anyone listing it, and a test fails if the two drift.
+That generator told blocks apart by name alone, so `tlsconf.Config`,
+`engines.Config` and `supervisor.Config` were each taken for the agent's
+`Config` and dropped: the reference had no TLS, engine or supervisor settings
+at all. It now has 37 blocks, six more than before.
+
+### A second control plane with `llmgw server add`, and the screens as windows
+
+- **`sudo llmgw server add [user@]host [--vip 10.0.0.160/24]`**, run on a
+  control plane, makes another machine a standby: this llmgw copied over, a
+  certificate from the cluster's authority, the same configuration and store
+  address, the machine let into the store (for a store under Patroni, one
+  change through Patroni's API), and the service started. Every node is told
+  of it within half a minute and fails over to it untouched. `--vip` keeps a
+  floating address on whichever control plane leads, with keepalived on both.
+  Tried on new machines: switching the leader off, the standby led after 14
+  seconds and the floating address answered from it after 23.
+- **Nodes learn every control plane** from the leader (a new `ControlPlanes`
+  message, kept in `/var/lib/llmgw-agent/control-planes.json`), after the
+  ones in their configuration.
+- **A node notices a dead control plane within a quarter of a minute.** A
+  leader whose machine stopped dead left its nodes' streams open until the
+  kernel gave up retransmitting, about fifteen minutes; the agent now pings
+  the stream after ten quiet seconds, and bounds dialling too.
+- **`GET /leader`**: 200 on the control plane that leads, 503 on a standby,
+  open like `/ready` — what keepalived follows.
+- **The store's access rules live in Patroni's shared configuration** for a
+  cluster set up by `server init --patroni`, so another control plane is let
+  in without touching each member.
+- **The screens are windows**: full screen, a title bar, the steps or actions
+  down the left, the content in a bordered panel, the keys along the bottom,
+  questions and errors as dialogs over it — the setup, the console, adding a
+  node and one-machine init alike. Keyboard only.
+
+### Machines nothing may log into: `llmgw join`
+
+For sites that allow no ssh into a machine, the setup screens offer a fourth
+way to reach one: a join code. The setup listens on port 7444 while it waits
+and shows a code for each such machine, with the commands that fetch llmgw
+onto it; there, `sudo llmgw join <code>` dials out — nothing dials in —
+checks by the certificate fingerprint in the code that it reached the setup
+that made it, admits itself with the code's secret, and runs what the setup
+sends, printing each step, until the setup says it is done. A code is good
+for one machine and an hour, and is under 90 characters so it can be copied
+whole from one screen into another.
+
+To the setup a joined machine is one more way of running a script there:
+the checks, PostgreSQL, Patroni and etcd go over it unchanged. Tried with one
+machine joining by code and another logged into with a password, in one
+setup.
+
+### Setting up from nothing in screens: `sudo llmgw`
+
+Run with nothing after it in a terminal, as root, `llmgw` opens full-screen
+setup on a machine with nothing installed, and a console on a control plane.
+In a pipe or a script it prints its help, as before.
+
+- **Setup** asks one machine or three. For three — this one as the control
+  plane and etcd's third vote, two more for the store under Patroni — it asks
+  how to reach the other two: the ssh agent, a key file, or a password, and a
+  sudo password where sudo asks for one. Passwords stay in memory for the run;
+  a sudo password grants the user passwordless sudo until the install
+  finishes, and the grant is removed after, failed or not.
+- **Every machine is checked before anything is installed**: login, system
+  (Ubuntu 22.04/24.04, Debian 12), architecture, CPUs, memory and disk for its
+  role, clock against this one, NTP, the ports its role needs, internet, and
+  whether it reaches the other machines. Failures stop it, with what to do;
+  warnings let it go on.
+- **Then the plan, and the command line that does the same** from a script,
+  then the install with its progress, and the administrator's password.
+- **The console** shows `ctl status`, refreshed every five seconds, and offers
+  to add a node or, for a store on one machine, to replicate it.
+- Machines are reached with Go's ssh client rather than the system's, so a
+  password can be used; a host key is recorded on first sight and refused if it
+  changes.
+
+Tried end to end on three new machines whose user logs in and sudoes only with
+a password.
+
+### A store that survives losing its machine
+
+`deploy/ha/` now covers the store as well as the control planes: PostgreSQL
+under Patroni on two machines, its leader lock in a three-member etcd, and
+llmgw given every member with `target_session_attrs=read-write`. Measured: a
+planned switchover cost one failed lease renewal; switching off the primary's
+machine promoted the standby in 24 seconds with the gateway serving
+throughout. Example configurations are beside it.
+
+**`llmgw server init --patroni user@host --patroni user@host`** sets it up:
+PostgreSQL, Patroni and etcd on those machines over ssh — this one as etcd's
+third vote — the role and database, and the store already in use moved in.
+Every step looks first; run against a cluster that is up, it changes nothing.
+Tried from scratch on three new machines (two minutes, then a failover by
+switching the primary off) and against the reference cluster (nothing
+touched).
+
+**Watching it**: for a store of several hosts the control plane publishes
+`llmgw_store_member`, `llmgw_store_replica_lag_bytes` and
+`llmgw_store_patroni_up` from each member's Patroni, a member that stopped
+showing as `missing` rather than vanishing; `llmgw ctl status` has a STORE
+line; and four alerts — `StoreHasNoLeader`, `StorePatroniUnreachable`,
+`StoreReplicaNotStreaming`, `StoreReplicaBehind`.
+
+**Making an administrator on PostgreSQL** (`init --admin`, `users promote`)
+failed: the flag was sent as a bool to an integer column, which SQLite takes
+and PostgreSQL refuses. Every existing PostgreSQL store got its administrator
+by being copied from SQLite, which is why nobody had met it.
+
+`llmgw server backup` reads a store address of several hosts. It took the
+list for one host name and reached the database only by falling through to
+the second; it now hands libpq the list, and dumps from a standby when there
+is one.
+
+### People, limits and usage are one page
+
+Managing a team meant its members on the people page, its quota on the quotas
+page and its spend on the usage page, each linking to the others. They are now
+one page in three tabs — **Organizations & people**, **Limits**, **Usage** —
+each at the address it had before, so links and bookmarks still land. The menu
+has one entry for them, People.
+
+Choosing an organization, team, project or person in the tree, or a person in
+the account list, opens a panel beside it with everything about that one
+thing: its own limits and what it is held to from above, spend against its
+budget, its per-member limit, its capacity quota, and its requests, tokens
+and spend over the last week, drawn. Its limits can be set or changed there,
+and the answer comes back to the same panel.
+
+The administrator's menu no longer has an Organizations entry beside it: `/org`
+and `/org/<id>` send an administrator to that organization's panel here. The
+page stays as it was for an organization's managers, who cannot see this one.
+
+### The chat page says whether a GPU or a CPU answered
+
+A model can have replicas on a card and on CPUs at once, and the two answer
+at very different speeds; the chat page gave no way to tell which one a reply
+came from. The picker now says where each model runs — `GPU`, `CPU` or
+`GPU + CPU` — and the line under each reply names what answered it: `GPU
+6000-24g` or `CPU`, beside its tokens a second.
+
+Every answer through the gateway carries the same as `x-llmgw-hardware`:
+`cpu`, or the GPU class. An endpoint configured by hand, with no pool, sends
+none.
+
+### A card shared with something else is judged every heartbeat
+
+A card with another process on it when its agent started was kept out of
+placement, and stayed out — "5014 MiB already allocated by another process" —
+after that process let go of it, until the agent was restarted. Found on a
+node shared with ComfyUI. The other way round, a tenant arriving after
+registration was never seen, and a card the agent's own engine held when the
+agent restarted read as someone else's.
+
+The agent now reads every card's memory per process on each heartbeat,
+leaves out what its own engines hold, and sends the rest; the control plane
+keeps a card out of placement on that reading, and lets it back in when it
+falls under 512 MiB. `llmgw_gpu_memory_used_bytes` reads the same figure. An
+agent too old to send it keeps the reading from registration.
 
 ### A GPU placement no longer runs on the CPU unnoticed
 
@@ -130,31 +432,45 @@ leaving the model with nothing. Now:
   stopping — is waited for, as before.
 - **`ctl restart --in-place`** stops each replica first, whatever room there is.
 
-### Every GPU node gets a vLLM it can run, chosen on the cluster page
+### Every GPU node gets engine builds it can run, by itself
 
-The vLLM a node installs is the build its hardware runs: CUDA 13 from PyPI
-where the driver (580+) allows, and otherwise the same version's CUDA 12.9
-build from vLLM's GitHub release, pinned to GitHub's digest, with PyTorch's
-CUDA 12.9 torch (driver 525+). One version across the cluster, so the flags
-llmgw builds — checked against that version — hold on every node.
+Which build of an engine a node installs follows its hardware — the CPU
+architecture, the driver, and the oldest GPU on it — from what each project
+publishes and how it was compiled. Each engine has a list of builds, best
+first; the node takes the first its hardware meets, and the first of those
+that installs and is seen to compute on the GPU. A build the table is wrong
+about costs its download, and the next one is tried.
 
-**Volta (V100) is not covered by either build.** Read from vLLM's release
-pipeline: every x86 build from 0.20.0 on — CUDA 13 and CUDA 12.9 alike — has
-kernels for compute capability 7.5 and up (aarch64: 8.0 and up). The last
-vLLM built for 7.0 is 0.19.1, whose PyPI wheel is CUDA 12 on PyPI's torch 2.10
-(also built for 7.0). A V100 node asked for a newer vLLM is refused, saying
-so; give such nodes 0.19.1 with `engine_overrides` by `gpu_class`. llmgw's
-launch flags are checked against 0.30.0, so run `llmgw ctl engine-check vllm`
-on such a node once it is installed.
+| | llama.cpp | vLLM |
+|---|---|---|
+| Driver 580+, Turing and newer | CUDA 13.3 | the version asked for, CUDA 13 (PyPI) |
+| Driver 525–579 | CUDA 12.8 (x64): its machine code for 8.6, 8.9, 12.0; the GPUs it ships as PTX need 570 | the version asked for, CUDA 12.9 (vLLM's GitHub release, PyTorch's cu129 torch) |
+| Volta (V100) | CUDA 12.8 on driver 570+, else Vulkan | **0.19.1**, the newest vLLM built for 7.0: no build of 0.20 or later has its kernels |
+| Pascal, Maxwell | CUDA 12.8 on 570+, else Vulkan | none: no vLLM is built for them |
+| arm64 GPU (GH200, Grace) | CUDA 13.3, else Vulkan — never the CPU build | CUDA 13 or 12.9, 8.0 and newer |
+| No NVIDIA GPU | the CPU build | none |
 
+Read from the projects' release pipelines: vLLM's x86 builds start at 7.5
+from 0.20.0, its PyPI build is CUDA 13 from 0.20, 0.19.1 was the last
+compiled for 7.0 (PyPI's torch 2.10 is too); llama.cpp's CUDA 12.8 build
+carries 5.0–9.0 as PTX and 8.6/8.9/12.0 as machine code, its CUDA 13.3 build
+starts at 7.5.
+
+- **A node running an older vLLM says so**: the cluster page's *vLLM build*
+  column shows the version installed, and the installation's detail says why
+  ("vLLM 0.19.1, the newest built for compute capability 7.0, instead of
+  0.30.0"). Profiles choose GPUs by class, so a model meant for the newer
+  vLLM is not placed on a V100 by accident.
 - **The install computes on the GPU before it counts**: a small tensor
-  operation in the new environment. A build without kernels for the card
-  imports and prints its version, and failed only on its first model; now it
-  fails to install, saying so.
-- **The build is chosen per node** on the cluster page (*vLLM build*: auto,
-  CUDA 13, CUDA 12.9, none), with `llmgw ctl engine-build NODE vllm BUILD`, or
-  `POST /api/v1/nodes/{id}/engines`. The choice is kept in the store and sent
-  to the node at once.
+  operation for vLLM, the devices llama.cpp lists for its GPU builds.
+- **The build can still be chosen per node** on the cluster page (auto, CUDA
+  13, CUDA 12.9, none), with `llmgw ctl engine-build NODE vllm BUILD`, or
+  `POST /api/v1/nodes/{id}/engines`; kept in the store and sent to the node
+  at once.
+- **Whether an engine computes on its GPUs is read from every process
+  nvidia-smi lists**, not compute processes alone: the Vulkan build is a
+  graphics one. A llama.cpp shard is given `GGML_VK_VISIBLE_DEVICES` beside
+  `CUDA_VISIBLE_DEVICES`.
 
 ### Engines per node, and nodes that do not run them all
 
@@ -169,12 +485,10 @@ and `--ignore-checks` passed over every other problem with it.
   written as `engines.disabled` in its `agent.yaml`, never installed or
   advertised there, and not checked by doctor.
 - **`engine_overrides`** in `server.yaml` give nodes picked by GPU class, id
-  or labels a different version of an engine, or none: vLLM 0.19.1, the last
-  built for Volta, on V100s beside the current one on newer cards.
-- **The requirements follow the version.** vLLM before 0.20 is CUDA 12 on
-  PyPI (driver 525+), and only up to 0.19.1 has Volta kernels; the threshold
-  had been CUDA 13's for every version, so the doctor's advice to set an older
-  version could not work. A refusal now says which version would.
+  or labels a different version of an engine, or none — for exceptions; the
+  build each GPU generation needs is chosen without it (above).
+- **The requirements follow the version**, as the table above sets out; the
+  threshold had been CUDA 13's for every version.
 - **An engine binary named in `agent.yaml` is used.** The agent had installed
   its own beside it, which was then used first.
 - The node page shows each GPU's compute capability.

api/proto/llmgw/v1/agent.proto

@@ -43,9 +43,18 @@ message ControlMessage {
     FabricScanRequest fabric_scan = 8;
     FabricTestRequest fabric_test = 9;
     UpgradeAgent upgrade = 10;
+    ControlPlanes control_planes = 11;
   }
 }
 
+// ControlPlanes is every control plane an agent may dial, sent by the leader
+// when an agent registers and whenever the list changes. The agent keeps it
+// beside its configuration, so a second control plane added after the node
+// was onboarded is one it fails over to without anybody editing the node.
+message ControlPlanes {
+  repeated string servers = 1;
+}
+
 // Register is the first message on every stream. It carries the node's full
 // hardware inventory, so the control plane needs no separate discovery call and
 // a reconnecting agent automatically refreshes what the scheduler knows.
@@ -304,6 +313,35 @@ message GPUSensor {
   // throttle names what is holding the clocks down now: sw_power_cap,
   // hw_slowdown, sw_thermal, hw_thermal, power_brake.
   repeated string throttle = 6;
+  // How busy the card is, sampled with the rest, rather than read once at
+  // registration. -1 is not reported by the driver. memory_total_mb is set
+  // by every agent that sends these, so zero there is an older agent.
+  double utilization_percent = 7;
+  double memory_utilization_percent = 8;
+  double memory_used_mb = 9;
+  double memory_total_mb = 10;
+  double sm_clock_mhz = 11;
+  // ecc_uncorrected counts uncorrected ECC errors since the driver loaded.
+  double ecc_uncorrected = 12;
+  // What wears the card out, read every minute; has_wear says these are
+  // present. Counts of -1 are not reported for this card.
+  bool has_wear = 13;
+  int64 pcie_replays = 14;
+  int64 retired_pages_sbe = 15;
+  int64 retired_pages_dbe = 16;
+  bool retirement_pending = 17;
+  int64 remapped_rows_correctable = 18;
+  int64 remapped_rows_uncorrectable = 19;
+  bool remap_pending = 20;
+  bool remap_failure = 21;
+  int64 nvlink_crc_errors = 22;
+  int64 nvlink_replay_errors = 23;
+  int64 nvlink_recovery_errors = 24;
+  // xid_errors are the driver's error events since the agent started.
+  int64 xid_errors = 25;
+  int32 last_xid = 26;
+  string last_xid_text = 27;
+  int64 last_xid_unix = 28;
 }
 
 // ShardLoad is one engine process group's use.
@@ -445,6 +483,15 @@ message Heartbeat {
   IMEX imex = 17;
   // cpu_count as in Register, for a VM given more CPUs while it runs.
   int32 cpu_count = 18;
+
+  // gpu_foreign_memory_mb, indexed by GPU, is what processes that are not
+  // this agent's shards hold on each card, read afresh every heartbeat. What
+  // keeps a card out of placement for another tenant is decided from this
+  // rather than from registration: a card that had something else on it when
+  // the agent started is not taken forever, and one that gains a tenant later
+  // is not handed out. Empty from an agent that does not measure it, or from
+  // one that could not this time.
+  repeated int32 gpu_foreign_memory_mb = 19;
 }
 
 // Provision is the software a node should have: which engines, at which

deploy/examples/server.yaml

@@ -92,6 +92,11 @@ agent_listen: ":7443"
 #   flush_interval: 200ms
 #   batch_size: 500
 #   buffer_size: 8192
+#
+#   # How long the record of each request is kept (what `ctl request` reads).
+#   # Charts, totals and budgets come from five-minute sums that are kept, so
+#   # this only bounds the table that grows by a row a request. 0 keeps all.
+#   usage_retention: 2160h   # 90 days
 
 router:
   scrape_interval: 2s

deploy/ha/README.md

@@ -85,3 +85,63 @@ auth:
 
 See [docs/install.md](../../docs/install.md) for moving an existing
 installation onto PostgreSQL without reissuing every key.
+
+## The store itself: PostgreSQL under Patroni
+
+Two control planes on one PostgreSQL have moved the single point of failure,
+not removed it. Patroni keeps PostgreSQL on two machines, one streaming to the
+other, and promotes the standby when the primary is lost. It keeps its leader
+lock in etcd, and etcd decides only with a majority — so three members, not
+two: two machines that lose sight of each other must not both decide they are
+the primary. The third member can be any machine that is up; it holds a vote,
+not data.
+
+On the reference cluster:
+
+| machine | etcd | PostgreSQL |
+|---|---|---|
+| llmgw-ctl (control plane) | member | — |
+| llmgw-pg01 | member | Patroni |
+| llmgw-ctl02 (second control plane) | member | Patroni |
+
+`llmgw server init --patroni user@host --patroni user@host` sets all of it up
+over ssh and moves the store in — see [install.md](../../docs/install.md).
+It looks before it acts at every step, so it is also how an existing cluster
+is checked: run against one that is up, it changes nothing. What it writes
+is in `etcd.default.example` (one member's `/etc/default/etcd`) and
+`patroni.yml.example` (one node's `/etc/patroni/config.yml`), for anyone
+doing it by hand; passwords are generated into `/root/llmgw-pg` on each
+database machine.
+
+**Give llmgw every member, and ask for the writable one:**
+
+```
+postgres://llmgw:…@10.0.0.149:5432,10.0.0.150:5432/llmgw?sslmode=disable&target_session_attrs=read-write
+```
+
+The driver connects to whichever member accepts writes, and reconnects to the
+new one after a failover. No floating address or HAProxy is needed for that.
+`llmgw server backup` reads the same address and dumps from a standby when
+there is one.
+
+Measured on the reference cluster, with the gateway serving throughout:
+
+- **Planned switchover** (`patronictl switchover`): the control plane logged
+  one failed lease renewal, reconnected to the new primary, and kept its
+  leadership; nothing else noticed.
+- **The primary's machine switched off**: the standby was promoted 24 seconds
+  later — the 30-second TTL in `patroni.yml` — and the control plane rode it
+  out without losing leadership or a replica. Started again, the old primary
+  rejoined as a standby by itself.
+
+**Watching it.** For a store of several hosts the control plane reads the
+cluster from each member's Patroni (port 8008) and publishes
+`llmgw_store_member{member,host,role,state}`, `llmgw_store_replica_lag_bytes`
+and `llmgw_store_patroni_up`; a host of the address that the cluster no longer
+lists is a member in state `missing`. `llmgw ctl status` prints it as one
+line, and `deploy/monitoring/llmgw-rules.yml` alerts on no leader, a member
+not streaming or missing, and a standby too far behind to be promoted.
+
+Not done here, and worth doing in production: a watchdog (`softdog`) so a
+primary that has lost the lock but not died cannot keep accepting writes, TLS
+on etcd and between PostgreSQL members, and more than one standby.

deploy/ha/etcd.default.example

@@ -0,0 +1,10 @@
+# llmgw: one member of the three that hold Patroni's leader lock (140, 149, 150).
+ETCD_NAME=llmgw-pg01
+ETCD_DATA_DIR=/var/lib/etcd/default
+ETCD_LISTEN_PEER_URLS=http://10.0.0.149:2380
+ETCD_LISTEN_CLIENT_URLS=http://10.0.0.149:2379,http://127.0.0.1:2379
+ETCD_INITIAL_ADVERTISE_PEER_URLS=http://10.0.0.149:2380
+ETCD_ADVERTISE_CLIENT_URLS=http://10.0.0.149:2379
+ETCD_INITIAL_CLUSTER=llmgw-ctl=http://10.0.0.140:2380,llmgw-pg01=http://10.0.0.149:2380,llmgw-ctl02=http://10.0.0.150:2380
+ETCD_INITIAL_CLUSTER_STATE=new
+ETCD_INITIAL_CLUSTER_TOKEN=llmgw-patroni

deploy/ha/patroni.yml.example

@@ -0,0 +1,68 @@
+# llmgw: the control planes' store, kept by Patroni on llmgw-pg01 and
+# llmgw-ctl02, with the leader lock in etcd on 140, 149 and 150.
+scope: llmgw
+namespace: /service/
+name: llmgw-pg01
+
+restapi:
+  listen: 10.0.0.149:8008
+  connect_address: 10.0.0.149:8008
+
+etcd3:
+  hosts: 10.0.0.140:2379,10.0.0.149:2379,10.0.0.150:2379
+
+bootstrap:
+  dcs:
+    ttl: 30
+    loop_wait: 10
+    retry_timeout: 10
+    maximum_lag_on_failover: 1048576
+    postgresql:
+      use_pg_rewind: true
+      use_slots: true
+      parameters:
+        wal_level: replica
+        hot_standby: "on"
+        max_wal_senders: 10
+        max_replication_slots: 10
+        wal_keep_size: 1GB
+        wal_log_hints: "on"
+        password_encryption: scram-sha-256
+  initdb:
+    - encoding: UTF8
+    - data-checksums
+
+postgresql:
+  listen: 127.0.0.1,10.0.0.149:5432
+  connect_address: 10.0.0.149:5432
+  data_dir: /var/lib/postgresql/16/llmgw
+  bin_dir: /usr/lib/postgresql/16/bin
+  pgpass: /var/lib/postgresql/.pgpass_patroni
+  authentication:
+    superuser:
+      username: postgres
+      password: "<from /root/llmgw-pg>"
+    replication:
+      username: replicator
+      password: "<from /root/llmgw-pg>"
+    rewind:
+      username: postgres
+      password: "<from /root/llmgw-pg>"
+  parameters:
+    unix_socket_directories: /var/run/postgresql
+  pg_hba:
+    - local all all peer
+    - host all all 127.0.0.1/32 scram-sha-256
+    - host replication replicator 10.0.0.149/32 scram-sha-256
+    - host replication replicator 10.0.0.150/32 scram-sha-256
+    - host all postgres 10.0.0.149/32 scram-sha-256
+    - host all postgres 10.0.0.150/32 scram-sha-256
+    - host llmgw llmgw 10.0.0.140/32 scram-sha-256
+    - host llmgw llmgw 10.0.0.149/32 scram-sha-256
+    - host llmgw llmgw 10.0.0.150/32 scram-sha-256
+
+tags:
+  nofailover: false
+  noloadbalance: false
+  clonefrom: false
+  nosync: false

deploy/monitoring/README.md

@@ -66,6 +66,8 @@ install -m 0644 grafana/datasource.yaml   /etc/grafana/provisioning/datasources/
 install -m 0644 grafana/dashboards.yaml   /etc/grafana/provisioning/dashboards/llmgw.yaml
 install -m 0644 -o grafana -g grafana grafana/llmgw-overview.json \
     /var/lib/grafana/dashboards/llmgw.json
+install -m 0644 -o grafana -g grafana grafana/llmgw-tenants.json \
+    /var/lib/grafana/dashboards/llmgw-tenants.json
 systemctl restart grafana-server
 ```
 

deploy/monitoring/grafana/llmgw-overview.json

@@ -487,6 +487,427 @@
         "defaults": {},
         "overrides": []
       }
+    },
+    {
+      "type": "row",
+      "title": "GPUs",
+      "id": 14,
+      "gridPos": {
+        "x": 0,
+        "y": 30,
+        "w": 24,
+        "h": 1
+      },
+      "collapsed": false,
+      "panels": []
+    },
+    {
+      "type": "timeseries",
+      "title": "GPU busy, by card",
+      "description": "Share of each sample period a kernel was running, sampled every heartbeat.",
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "gridPos": {
+        "x": 0,
+        "y": 31,
+        "w": 12,
+        "h": 7
+      },
+      "id": 15,
+      "targets": [
+        {
+          "expr": "llmgw_gpu_utilization_percent",
+          "legendFormat": "{{node}} gpu {{gpu}}",
+          "refId": "A"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percent",
+          "min": 0,
+          "max": 100
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi"
+        }
+      }
+    },
+    {
+      "type": "timeseries",
+      "title": "GPU memory in use, by card",
+      "description": "Framebuffer in use. Engines reserve most of it at start.",
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "gridPos": {
+        "x": 12,
+        "y": 31,
+        "w": 12,
+        "h": 7
+      },
+      "id": 16,
+      "targets": [
+        {
+          "expr": "llmgw_gpu_memory_used_bytes / llmgw_gpu_memory_total_bytes",
+          "legendFormat": "{{node}} gpu {{gpu}}",
+          "refId": "A"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percentunit",
+          "min": 0,
+          "max": 1
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi"
+        }
+      }
+    },
+    {
+      "type": "timeseries",
+      "title": "GPU memory bandwidth busy, by card",
+      "description": "Decode is bound by memory: high here with compute low is a card waiting on memory.",
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "gridPos": {
+        "x": 0,
+        "y": 38,
+        "w": 12,
+        "h": 7
+      },
+      "id": 17,
+      "targets": [
+        {
+          "expr": "llmgw_gpu_memory_utilization_percent",
+          "legendFormat": "{{node}} gpu {{gpu}}",
+          "refId": "A"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percent",
+          "min": 0,
+          "max": 100
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi"
+        }
+      }
+    },
+    {
+      "type": "timeseries",
+      "title": "GPU temperature",
+      "description": "Throttling usually starts in the high 80s.",
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "gridPos": {
+        "x": 12,
+        "y": 38,
+        "w": 12,
+        "h": 7
+      },
+      "id": 18,
+      "targets": [
+        {
+          "expr": "llmgw_gpu_temperature_celsius",
+          "legendFormat": "{{node}} gpu {{gpu}}",
+          "refId": "A"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "celsius"
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi"
+        }
+      }
+    },
+    {
+      "type": "timeseries",
+      "title": "GPU power against its limit",
+      "description": "1 is the enforced limit; a card held there is power-capped.",
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "gridPos": {
+        "x": 0,
+        "y": 45,
+        "w": 12,
+        "h": 7
+      },
+      "id": 19,
+      "targets": [
+        {
+          "expr": "llmgw_gpu_power_watts / llmgw_gpu_power_limit_watts",
+          "legendFormat": "{{node}} gpu {{gpu}}",
+          "refId": "A"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percentunit",
+          "min": 0,
+          "max": 1.1
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi"
+        }
+      }
+    },
+    {
+      "type": "timeseries",
+      "title": "GPU clock",
+      "description": "A clock held low under load is the card throttled; see the next panel for why.",
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "gridPos": {
+        "x": 12,
+        "y": 45,
+        "w": 12,
+        "h": 7
+      },
+      "id": 20,
+      "targets": [
+        {
+          "expr": "llmgw_gpu_sm_clock_mhz",
+          "legendFormat": "{{node}} gpu {{gpu}}",
+          "refId": "A"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "rotmhz"
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi"
+        }
+      }
+    },
+    {
+      "type": "timeseries",
+      "title": "What holds GPUs back",
+      "description": "Cards throttled now, by reason, and uncorrected ECC errors since the driver loaded.",
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "gridPos": {
+        "x": 0,
+        "y": 52,
+        "w": 24,
+        "h": 7
+      },
+      "id": 21,
+      "targets": [
+        {
+          "expr": "sum by (reason) (llmgw_gpu_throttled)",
+          "legendFormat": "throttled: {{reason}}",
+          "refId": "A"
+        },
+        {
+          "expr": "sum by (node, gpu) (llmgw_gpu_ecc_errors)",
+          "legendFormat": "ECC errors {{node}} gpu {{gpu}}",
+          "refId": "B"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi"
+        }
+      }
+    },
+    {
+      "type": "timeseries",
+      "title": "Output tokens/s per GPU, by model",
+      "description": "What each model gets out of each card it holds.",
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "gridPos": {
+        "x": 0,
+        "y": 59,
+        "w": 12,
+        "h": 7
+      },
+      "id": 22,
+      "targets": [
+        {
+          "expr": "sum by (model) (rate(llmgw_tokens_total{direction=\"completion\"}[5m])) / count by (model) (llmgw_gpu_assigned)",
+          "legendFormat": "{{model}}",
+          "refId": "A"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi"
+        }
+      }
+    },
+    {
+      "type": "timeseries",
+      "title": "Output tokens per watt-hour, by model",
+      "description": "Tokens for the energy the model's cards draw.",
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "gridPos": {
+        "x": 12,
+        "y": 59,
+        "w": 12,
+        "h": 7
+      },
+      "id": 23,
+      "targets": [
+        {
+          "expr": "sum by (model) (rate(llmgw_tokens_total{direction=\"completion\"}[5m])) * 3600 / sum by (model) (llmgw_gpu_power_watts * on (node, gpu) group_left (model) llmgw_gpu_assigned)",
+          "legendFormat": "{{model}}",
+          "refId": "A"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi"
+        }
+      }
+    },
+    {
+      "type": "timeseries",
+      "title": "GPU wear",
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "id": 24,
+      "description": "What wears cards out: XID errors in the last 15 minutes, memory rows remapped, PCIe retries an hour. Anything above zero here is worth a look on that card's node page.",
+      "gridPos": {
+        "x": 0,
+        "y": 66,
+        "w": 24,
+        "h": 7
+      },
+      "targets": [
+        {
+          "expr": "increase(llmgw_gpu_xid_errors[15m]) > 0",
+          "legendFormat": "XID {{node}} gpu {{gpu}}",
+          "refId": "A"
+        },
+        {
+          "expr": "sum by (node, gpu) (llmgw_gpu_remapped_rows) > 0",
+          "legendFormat": "rows remapped {{node}} gpu {{gpu}}",
+          "refId": "B"
+        },
+        {
+          "expr": "increase(llmgw_gpu_pcie_replays[1h]) > 0",
+          "legendFormat": "PCIe retries/h {{node}} gpu {{gpu}}",
+          "refId": "C"
+        },
+        {
+          "expr": "increase(llmgw_gpu_nvlink_errors[1h]) > 0",
+          "legendFormat": "NVLink {{kind}}/h {{node}} gpu {{gpu}}",
+          "refId": "D"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi"
+        }
+      }
     }
   ]
 }

deploy/monitoring/grafana/llmgw-tenants.json

@@ -0,0 +1,650 @@
+{
+  "uid": "llmgw-tenants",
+  "title": "llmgw \u2014 tenants",
+  "schemaVersion": 39,
+  "version": 1,
+  "editable": true,
+  "time": {
+    "from": "now-24h",
+    "to": "now"
+  },
+  "refresh": "1m",
+  "tags": [
+    "llmgw"
+  ],
+  "description": "Requests, tokens, failures, refusals and first-token time per tenant \u2014 the level router.tenant_label names \u2014 and every budget's share spent.",
+  "templating": {
+    "list": [
+      {
+        "name": "tenant",
+        "label": "Tenant",
+        "type": "query",
+        "datasource": {
+          "type": "prometheus",
+          "uid": "llmgw-prom"
+        },
+        "query": {
+          "query": "label_values(llmgw_tenant_requests_total, tenant)",
+          "refId": "tenant"
+        },
+        "definition": "label_values(llmgw_tenant_requests_total, tenant)",
+        "multi": true,
+        "includeAll": true,
+        "allValue": ".*",
+        "current": {
+          "text": "All",
+          "value": "$__all"
+        },
+        "refresh": 2,
+        "sort": 1
+      }
+    ]
+  },
+  "panels": [
+    {
+      "id": 1,
+      "title": "Requests a second",
+      "type": "stat",
+      "description": "Across the tenants chosen above.",
+      "gridPos": {
+        "x": 0,
+        "y": 0,
+        "w": 6,
+        "h": 4
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "sum(rate(llmgw_tenant_requests_total{tenant=~\"$tenant\"}[5m]))",
+          "legendFormat": "",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps"
+        },
+        "overrides": []
+      },
+      "options": {
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        },
+        "textMode": "auto",
+        "colorMode": "value"
+      }
+    },
+    {
+      "id": 2,
+      "title": "Failing",
+      "type": "stat",
+      "description": "Requests that failed, not counting refusals by a limit.",
+      "gridPos": {
+        "x": 6,
+        "y": 0,
+        "w": 6,
+        "h": 4
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "sum(rate(llmgw_tenant_requests_total{tenant=~\"$tenant\",outcome=\"error\"}[5m])) / sum(rate(llmgw_tenant_requests_total{tenant=~\"$tenant\"}[5m]))",
+          "legendFormat": "",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percentunit",
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "orange",
+                "value": 0.02
+              },
+              {
+                "color": "red",
+                "value": 0.05
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        },
+        "textMode": "auto",
+        "colorMode": "value"
+      }
+    },
+    {
+      "id": 3,
+      "title": "Refused by a limit",
+      "type": "stat",
+      "description": "Rate, token and budget limits; 429 to the caller.",
+      "gridPos": {
+        "x": 12,
+        "y": 0,
+        "w": 6,
+        "h": 4
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "sum(rate(llmgw_tenant_requests_total{tenant=~\"$tenant\",outcome=\"limited\"}[5m])) / sum(rate(llmgw_tenant_requests_total{tenant=~\"$tenant\"}[5m]))",
+          "legendFormat": "",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percentunit",
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "orange",
+                "value": 0.02
+              },
+              {
+                "color": "red",
+                "value": 0.1
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        },
+        "textMode": "auto",
+        "colorMode": "value"
+      }
+    },
+    {
+      "id": 4,
+      "title": "First token, p95",
+      "type": "stat",
+      "description": "What people wait for before anything appears.",
+      "gridPos": {
+        "x": 18,
+        "y": 0,
+        "w": 6,
+        "h": 4
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "histogram_quantile(0.95, sum by (le) (rate(llmgw_tenant_ttft_seconds_bucket{tenant=~\"$tenant\"}[5m])))",
+          "legendFormat": "",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "s",
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "orange",
+                "value": 3
+              },
+              {
+                "color": "red",
+                "value": 10
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        },
+        "textMode": "auto",
+        "colorMode": "value"
+      }
+    },
+    {
+      "id": 5,
+      "title": "Requests by tenant",
+      "type": "timeseries",
+      "description": "Stacked, so the top edge is the whole of it.",
+      "gridPos": {
+        "x": 0,
+        "y": 4,
+        "w": 12,
+        "h": 9
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "sum by (tenant) (rate(llmgw_tenant_requests_total{tenant=~\"$tenant\"}[5m]))",
+          "legendFormat": "{{tenant}}",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps",
+          "custom": {
+            "stacking": {
+              "mode": "normal"
+            },
+            "fillOpacity": 30
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "table",
+          "placement": "right",
+          "calcs": [
+            "mean",
+            "max"
+          ]
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      }
+    },
+    {
+      "id": 6,
+      "title": "Tokens by tenant",
+      "type": "timeseries",
+      "description": "Prompt and completion together: what was consumed.",
+      "gridPos": {
+        "x": 12,
+        "y": 4,
+        "w": 12,
+        "h": 9
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "sum by (tenant) (rate(llmgw_tenant_tokens_total{tenant=~\"$tenant\"}[5m]))",
+          "legendFormat": "{{tenant}}",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short",
+          "custom": {
+            "stacking": {
+              "mode": "normal"
+            },
+            "fillOpacity": 30
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "table",
+          "placement": "right",
+          "calcs": [
+            "mean",
+            "max"
+          ]
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      }
+    },
+    {
+      "id": 7,
+      "title": "Refused by a limit, by tenant",
+      "type": "timeseries",
+      "description": "The share of each tenant's requests a limit turned away.",
+      "gridPos": {
+        "x": 0,
+        "y": 13,
+        "w": 12,
+        "h": 8
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "sum by (tenant) (rate(llmgw_tenant_requests_total{tenant=~\"$tenant\",outcome=\"limited\"}[5m])) / sum by (tenant) (rate(llmgw_tenant_requests_total{tenant=~\"$tenant\"}[5m]))",
+          "legendFormat": "{{tenant}}",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "table",
+          "placement": "right",
+          "calcs": [
+            "mean",
+            "max"
+          ]
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      }
+    },
+    {
+      "id": 8,
+      "title": "Failing, by tenant",
+      "type": "timeseries",
+      "description": "One tenant alone failing is usually what it sends; all of them together is a model.",
+      "gridPos": {
+        "x": 12,
+        "y": 13,
+        "w": 12,
+        "h": 8
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "sum by (tenant) (rate(llmgw_tenant_requests_total{tenant=~\"$tenant\",outcome=\"error\"}[5m])) / sum by (tenant) (rate(llmgw_tenant_requests_total{tenant=~\"$tenant\"}[5m]))",
+          "legendFormat": "{{tenant}}",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "table",
+          "placement": "right",
+          "calcs": [
+            "mean",
+            "max"
+          ]
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      }
+    },
+    {
+      "id": 9,
+      "title": "First token p95, by tenant",
+      "type": "timeseries",
+      "description": "",
+      "gridPos": {
+        "x": 0,
+        "y": 21,
+        "w": 12,
+        "h": 8
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "histogram_quantile(0.95, sum by (tenant, le) (rate(llmgw_tenant_ttft_seconds_bucket{tenant=~\"$tenant\"}[10m])))",
+          "legendFormat": "{{tenant}}",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "s"
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "table",
+          "placement": "right",
+          "calcs": [
+            "mean",
+            "max"
+          ]
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      }
+    },
+    {
+      "id": 10,
+      "title": "Budgets spent this period",
+      "type": "bargauge",
+      "description": "Every organization, team, project and person with a budget. Requests are refused at 100%.",
+      "gridPos": {
+        "x": 12,
+        "y": 21,
+        "w": 12,
+        "h": 8
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "sort_desc(llmgw_budget_spent_ratio)",
+          "legendFormat": "{{scope}} {{name}} ({{period}})",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percentunit",
+          "min": 0,
+          "max": 1,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "orange",
+                "value": 0.8
+              },
+              {
+                "color": "red",
+                "value": 1
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "displayMode": "gradient",
+        "orientation": "horizontal",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        },
+        "showUnfilled": true
+      }
+    },
+    {
+      "id": 11,
+      "title": "Who used most, last 24 hours",
+      "type": "table",
+      "description": "Spend in money is on the people page's usage tab, which reads it from the store.",
+      "gridPos": {
+        "x": 0,
+        "y": 29,
+        "w": 24,
+        "h": 8
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "sort_desc(sum by (tenant) (increase(llmgw_tenant_requests_total{tenant=~\"$tenant\"}[24h])))",
+          "legendFormat": "requests",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          },
+          "format": "table",
+          "instant": true
+        },
+        {
+          "refId": "B",
+          "expr": "sum by (tenant) (increase(llmgw_tenant_tokens_total{tenant=~\"$tenant\"}[24h]))",
+          "legendFormat": "tokens",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          },
+          "format": "table",
+          "instant": true
+        },
+        {
+          "refId": "C",
+          "expr": "sum by (tenant) (increase(llmgw_tenant_requests_total{tenant=~\"$tenant\",outcome=\"limited\"}[24h]))",
+          "legendFormat": "limited",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          },
+          "format": "table",
+          "instant": true
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {},
+        "overrides": []
+      },
+      "options": {
+        "showHeader": true
+      },
+      "transformations": [
+        {
+          "id": "merge",
+          "options": {}
+        },
+        {
+          "id": "organize",
+          "options": {
+            "excludeByName": {
+              "Time": true
+            },
+            "renameByName": {
+              "Value #A": "Requests",
+              "Value #B": "Tokens",
+              "Value #C": "Refused by a limit"
+            }
+          }
+        }
+      ]
+    }
+  ]
+}

deploy/monitoring/llmgw-rules.yml

@@ -200,6 +200,94 @@ groups:
             is skipped by placement rather than failing, so this is quiet until
             somebody wonders why a model will not land there.
 
+  - name: llmgw-gpus
+    rules:
+      - alert: GPUThermalThrottling
+        expr: llmgw_gpu_throttled{reason=~"sw_thermal|hw_thermal|hw_slowdown"} == 1
+        for: 10m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.node }} GPU {{ $labels.gpu }} is slowed by heat ({{ $labels.reason }})"
+          description: >-
+            Ten minutes held back by temperature: every replica on the card is
+            slower than its measurements say. Usually airflow, a fan, or the
+            room — the node page shows the card's temperature over time.
+
+      - alert: GPUUncorrectedECCErrors
+        expr: llmgw_gpu_ecc_errors > 0
+        for: 1m
+        labels: { severity: critical }
+        annotations:
+          summary: "{{ $labels.node }} GPU {{ $labels.gpu }} has {{ $value }} uncorrected ECC errors"
+          description: >-
+            Uncorrected memory errors since the driver loaded: what runs on the
+            card may be computing on corrupted memory. Drain the node
+            (llmgw ctl drain) and have the card looked at.
+
+      - alert: GPUXidError
+        expr: increase(llmgw_gpu_xid_errors[15m]) > 0
+        for: 0m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.node }} GPU {{ $labels.gpu }} logged an XID error"
+          description: >-
+            The driver reported an error on the card (its code is
+            llmgw_gpu_last_xid, and the node page says it in words). 79 is a
+            card fallen off the bus, 48 and 63/64 memory errors, 74 NVLink,
+            94/95 contained and uncontained ECC errors: those need the card
+            looked at. Others are often an engine's own fault, and pass.
+
+      - alert: GPUOutOfSpareRows
+        expr: llmgw_gpu_remap_failure == 1
+        for: 1m
+        labels: { severity: critical }
+        annotations:
+          summary: "{{ $labels.node }} GPU {{ $labels.gpu }} has no spare memory rows left"
+          description: >-
+            The card cannot remap another failing row. Drain the node and
+            have the card replaced.
+
+      - alert: GPUNeedsReset
+        expr: llmgw_gpu_remap_pending == 1 or llmgw_gpu_retirement_pending == 1
+        for: 5m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.node }} GPU {{ $labels.gpu }} has memory repairs waiting for a reset"
+          description: >-
+            Rows remapped or pages retired take effect only after the card is
+            reset. Drain the node (llmgw ctl drain), reset or reboot, and bring
+            it back.
+
+      - alert: GPUPCIeRetriesClimbing
+        expr: increase(llmgw_gpu_pcie_replays[1h]) > 100
+        for: 0m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.node }} GPU {{ $labels.gpu }}: {{ $value }} PCIe retries in an hour"
+          description: >-
+            The card's link to the host is retrying. A riser, the slot or a
+            cable is the usual cause, and transfers to the card slow down.
+
+      - alert: GPUNVLinkErrorsClimbing
+        expr: increase(llmgw_gpu_nvlink_errors{kind="crc"}[1h]) > 10
+        for: 0m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.node }} GPU {{ $labels.gpu }}: NVLink CRC errors climbing"
+          description: >-
+            Errors on the card's NVLink links: tensor-parallel replicas on it
+            slow down, and a link that keeps failing takes a replica with it.
+
+      - alert: GPUHot
+        expr: llmgw_gpu_temperature_celsius > 87
+        for: 10m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.node }} GPU {{ $labels.gpu }} at {{ $value }} °C"
+          description: >-
+            Ten minutes above 87 °C, where most data-centre cards begin to
+            throttle.
+
   - name: monitoring-host
     rules:
       - alert: MonitoringDiskFilling
@@ -262,3 +350,130 @@ groups:
             Working as designed when the cluster is full: the lowest band is
             held first. Worth knowing about because the callers being refused
             do not know why.
+
+  # Per tenant: the level router.tenant_label names (a team by default), and
+  # every organization, team, project and person given a budget.
+  - name: llmgw-tenants
+    rules:
+      - alert: BudgetMostlySpent
+        expr: llmgw_budget_spent_ratio >= 0.8 and llmgw_budget_spent_ratio < 1
+        for: 5m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.scope }} {{ $labels.name }} has spent {{ $value | humanizePercentage }} of its {{ $labels.period }}ly budget"
+          description: >-
+            Its requests are refused once it reaches all of it, until the
+            period turns. Raise it on the people page, or tell them now
+            rather than when they are refused.
+
+      - alert: BudgetSpent
+        expr: llmgw_budget_spent_ratio >= 1
+        for: 1m
+        labels: { severity: critical }
+        annotations:
+          summary: "{{ $labels.scope }} {{ $labels.name }} has spent its {{ $labels.period }}ly budget"
+          description: >-
+            Every request under it is refused with 429 until the period
+            turns, unless a cheaper model is named as its fallback.
+
+      - alert: TenantHittingLimits
+        expr: >-
+          sum by (tenant) (rate(llmgw_tenant_requests_total{outcome="limited"}[10m]))
+          / sum by (tenant) (rate(llmgw_tenant_requests_total[10m])) > 0.1
+        for: 10m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $value | humanizePercentage }} of {{ $labels.tenant }}'s requests are refused by a limit"
+          description: >-
+            A rate, token or budget limit is turning their traffic away. Either
+            the limit is too low for what they now do, or something of theirs
+            is looping.
+
+      - alert: TenantErrors
+        expr: >-
+          sum by (tenant) (rate(llmgw_tenant_requests_total{outcome="error"}[10m]))
+          / sum by (tenant) (rate(llmgw_tenant_requests_total[10m])) > 0.05
+          and sum by (tenant) (rate(llmgw_tenant_requests_total[10m])) > 0.05
+        for: 10m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $value | humanizePercentage }} of {{ $labels.tenant }}'s requests fail"
+          description: >-
+            Not counting refusals by a limit. If every tenant fails alike it
+            is a model (ModelFailing); one alone is usually what they send —
+            prompts past the context, a model they may not use.
+
+      - alert: TenantTrafficSpike
+        expr: >-
+          sum by (tenant) (rate(llmgw_tenant_requests_total[30m]))
+          > 3 * sum by (tenant) (rate(llmgw_tenant_requests_total[7d]))
+          and sum by (tenant) (rate(llmgw_tenant_requests_total[30m])) > 0.2
+        for: 15m
+        labels: { severity: info }
+        annotations:
+          summary: "{{ $labels.tenant }} is sending three times its weekly average"
+          description: >-
+            A new workload, a batch job, or a client retrying in a loop. Only
+            for a tenant sending more than one request every five seconds, so
+            a quiet team starting work in the morning does not page anybody.
+
+      - alert: TenantSlowFirstToken
+        expr: >-
+          histogram_quantile(0.95,
+            sum by (tenant, le) (rate(llmgw_tenant_ttft_seconds_bucket[15m]))) > 10
+          and sum by (tenant) (rate(llmgw_tenant_ttft_seconds_count[15m])) > 0.02
+        for: 15m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.tenant }} waits {{ $value | humanizeDuration }} for a first token at p95"
+          description: >-
+            What their people feel. Compare with SlowFirstToken per model: if
+            only this tenant is slow, it is the models it uses or its queue
+            band, not the cluster.
+  # The store, when it is a replicated cluster under Patroni (llmgw server
+  # init --patroni). Published only for a store of several hosts.
+  - name: llmgw-store
+    rules:
+      - alert: StoreHasNoLeader
+        expr: llmgw_store_patroni_up == 1 and on() count(llmgw_store_member{role="leader"}) == 0
+        for: 1m
+        labels: { severity: critical }
+        annotations:
+          summary: "The store's cluster has no primary"
+          description: >-
+            Every write fails — keys, usage, budgets, leadership. Patroni
+            promotes a standby within its 30-second TTL; a minute without one
+            means it cannot: etcd has lost its majority, or no standby is fit.
+            `patronictl -c /etc/patroni/config.yml list` on a member says which.
+
+      - alert: StorePatroniUnreachable
+        expr: llmgw_store_patroni_up == 0
+        for: 2m
+        labels: { severity: warning }
+        annotations:
+          summary: "No member of the store's cluster answers on its Patroni port"
+          description: >-
+            The control plane reads the cluster's state from port 8008 on each
+            member. The database may still be serving; nothing here can tell.
+
+      - alert: StoreReplicaNotStreaming
+        expr: llmgw_store_member{role!="leader", state!="streaming"} == 1
+        for: 5m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.member }} is {{ $labels.state }}, not streaming from the primary"
+          description: >-
+            A standby that is not streaming is not a standby: if the primary
+            is lost now, there is nothing to promote. "missing" is a host of
+            the store address the cluster no longer lists — usually a machine
+            that is down.
+
+      - alert: StoreReplicaBehind
+        expr: llmgw_store_replica_lag_bytes > 64 * 1024 * 1024 or llmgw_store_replica_lag_bytes == -1
+        for: 10m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.member }} is {{ $value | humanize1024 }}B behind the primary"
+          description: >-
+            Patroni will not promote a standby more than 1 MB behind, so a
+            failover now would find none fit. -1 is a lag Patroni cannot read.

docs/install.md

@@ -87,7 +87,24 @@ CGO_ENABLED=0 go build -o llmgw ./cmd/llmgw
 
 ## 2. The control plane
 
-On the machine that will be the control plane, as root:
+**The short way is the screens.** On the machine that will be the control
+plane, run `sudo llmgw` with nothing after it. With nothing installed it asks
+whether this is one machine or three for high availability; for three, it asks
+for the other two — address, user, and an ssh key, the ssh agent, or a
+password, and the sudo password if sudo asks for one; or, for a machine
+nothing may log into, a join code to run there with `sudo llmgw join <code>`,
+which dials out to this one on port 7444 — checks all three before
+installing anything (system, room, clock, ports, internet, whether they reach
+one another), shows the plan, and installs. Passwords are held in memory for
+the run only; a sudo password lets the user sudo without one until the install
+finishes, and is then taken back. At the end it shows the administrator's
+password and the command that does the same from a script.
+
+Run again later, `sudo llmgw` is the console: the cluster's state, refreshed,
+and what to do next — add a node, replicate the store. A setup that stopped
+halfway opens the setup screens again, which finish it.
+
+The rest of this section is the same by command, as root:
 
 ```
 sudo llmgw server init --admin you@example.com
@@ -121,6 +138,20 @@ Add `--postgres` to keep accounts and keys in PostgreSQL rather than SQLite:
 init installs it and sets it up, and on a machine already running on SQLite it
 moves everything across. See [PostgreSQL](#postgresql-when-sqlite-is-no-longer-the-right-shape).
 
+Or `--patroni`, given two machines (or more), to keep it in PostgreSQL
+replicated across them, so losing one loses nothing:
+
+```
+sudo --preserve-env=SSH_AUTH_SOCK llmgw server init --patroni ubuntu@10.0.0.149 --patroni ubuntu@10.0.0.150
+```
+
+Over ssh with passwordless sudo, as `node add` works, it installs PostgreSQL,
+Patroni and etcd there — this machine is etcd's third vote — and moves the
+store already in use, SQLite or PostgreSQL, into the cluster. A standby is
+promoted within about thirty seconds of the primary being lost, and the
+control plane follows it by itself. `llmgw ctl status` says who leads; see
+[deploy/ha](../deploy/ha/README.md) for what it sets up.
+
 Every step looks before it acts. Running it again finishes an install that
 stopped halfway and changes nothing on one that did not; it never overwrites
 `server.yaml`, the authority, or an existing account's password.
@@ -367,7 +398,7 @@ Nothing to do: the agent installs them once the node joins, under
 
 | Engine | On | How |
 |---|---|---|
-| llama.cpp | every node | the release build for its processor, CUDA where there is a GPU, checked against GitHub's checksum |
+| llama.cpp | every node | the release build for its processor and GPU — CUDA 13.3, CUDA 12.8 (x64), or Vulkan for GPUs and drivers the CUDA builds do not cover; the CPU build only where there is no NVIDIA GPU — checked against GitHub's checksum and seen to list the GPU |
 | vLLM | nodes with an NVIDIA GPU it can run on | a Python environment built by uv from PyPI, about 8 GB |
 | SGLang | only when you ask for it | the same way as vLLM |
 
@@ -376,17 +407,17 @@ node gets the same dependencies whenever it is built, and installed with
 `--require-hashes` against the sha256 PyPI publishes. The resolved list stays in
 the environment as `llmgw-requirements.txt`. It needs:
 
-- **a Turing GPU or newer** (compute capability 7.5; 8.0 on aarch64), and
-  **driver 580 or newer** for the CUDA 13 build that PyPI carries from vLLM
-  0.20 on. A node with an older driver gets **the same version built for CUDA
-  12.9** instead, from vLLM's GitHub release (checked against the digest
-  GitHub publishes) with PyTorch's CUDA 12.9 torch: driver 525 or newer.
-  Either way the install ends by computing something on the node's GPU, and a
-  build that cannot fails to install, saying why;
-- **a V100 (Volta, 7.0) needs vLLM 0.19.1**, the last version built for it:
-  every build from 0.20 on starts at 7.5. Give V100 nodes that version with
-  `engine_overrides` (below), and run `llmgw ctl engine-check vllm` there once
-  it is installed, since llmgw's flags are checked against the current one;
+- **the build its hardware runs**, chosen by the node. Driver 580 and newer:
+  the CUDA 13 build from PyPI. Driver 525 to 579: the same version built for
+  CUDA 12.9, from vLLM's GitHub release (checked against the digest GitHub
+  publishes) with PyTorch's CUDA 12.9 torch. Both have kernels for Turing
+  (7.5) and newer, 8.0 and newer on aarch64. **A V100** (Volta, 7.0) gets
+  **vLLM 0.19.1**, the newest built for it — no build from 0.20 on has its
+  kernels — and the cluster page shows it running that version; run `llmgw
+  ctl engine-check vllm` there once, since llmgw's flags are checked against
+  the current one. Pascal and older run no vLLM. Whatever is chosen, the
+  install ends by computing something on the GPU; a build that cannot is
+  passed over for the next;
 - **about 25 GB free** under the engines directory while it installs;
 - **gcc**, which Triton builds its kernel launcher with. The agent installs it.
 
@@ -431,8 +462,8 @@ cluster's for them, in order:
 engines:
   vllm: "0.30.0"              # CUDA 13: Turing and newer
 engine_overrides:
-  - gpu_class: v100-32g       # Volta: no vLLM build from 0.20 on has its kernels
-    engines: { vllm: "0.19.1" }  # the last one that does
+  - gpu_class: a10g           # pin a class to a version you have tested
+    engines: { vllm: "0.29.0" }
   - labels: { role: gguf }    # nodes meant for llama.cpp alone
     engines: { vllm: "" }     # none
 ```
@@ -1271,12 +1302,12 @@ requests. On a slow model, set `timeouts.generation` higher.
   can see each other, seeding between them is the obvious next thing to build.
 - **A budget without a price never binds.** Set `default_price` or `pricing`
   (step 2) or every request costs zero, and a budget is something the dashboard
-  displays rather than something that stops anything. The quotas page says so
+  displays rather than something that stops anything. The Limits tab says so
   where it can, which is not the same as being told before you rely on it.
-- **No vLLM from 0.20 on runs on a V100.** Every build of it has kernels for
-  Turing (7.5) and newer; 0.19.1 is the last built for Volta. Give V100 nodes
-  that version with `engine_overrides` (step 4) — its flags are not the ones
-  llmgw was checked against, so run `llmgw ctl engine-check vllm` there.
+- **A V100 runs vLLM 0.19.1**, not the version the rest of the cluster runs:
+  no build from 0.20 on has Volta kernels, and the node picks the newest that
+  has. llmgw's flags are checked against the current version, so run `llmgw
+  ctl engine-check vllm` on a V100 node once.
 - **Two control planes have not been run for long.** The leader election and
   agent failover work and there is a pattern in `deploy/ha/`, but every cluster
   this has been run on in anger has had one. A single control plane is the

docs/onboarding.md

@@ -191,7 +191,8 @@ project or person, the teams and projects with their limits and spend, and every
 member. Only what was spent in that organization's projects is on it; what its
 people do elsewhere is not. The page is read-only — a manager is not an
 administrator of the cluster, and limits and memberships stay with those who
-are.
+are. An administrator following `/org` lands on the same organization's panel
+on the people page instead, where all of that can be changed.
 
 `llmgw keys init` still works for provisioning scripts and is idempotent, but it
 will no longer move a team or project that already exists under another parent;
@@ -259,7 +260,7 @@ Allowlists compose by intersection: a key cannot add a model its team withheld.
 A model's canonical name and any of its aliases name the same thing, so either
 form works.
 
-Budgets are given in whole units, as the quotas page shows them, and what a unit
+Budgets are given in whole units, as the Limits tab shows them, and what a unit
 means is yours to decide. Self-hosted models have no per-token price, so it can
 be GPU-seconds, a notional rate, or one unit per thousand tokens everywhere —
 a price of 1 000 000 in `auth.pricing`, which is in micro-units — and a budget
@@ -409,10 +410,19 @@ page exists.
 opening onto its members and its keys. From there an admin approves people
 (into a project or on their own), adds them to and takes them out of
 organizations, teams and projects, suspends and removes them, revokes keys, and
-adds, renames, moves and removes organizations, teams and projects. `/admin/quotas`
-sets limits on the tree, per member of any part of it, and per person. A searchable list of every account is at the
-bottom. The page works as plain forms; with scripts on, a change updates it in
-place and the projects you had open stay open.
+adds, renames, moves and removes organizations, teams and projects. A
+searchable list of every account is at the bottom. Choosing any organization,
+team, project or person opens a panel beside the tree with its limits, its
+capacity quota and its use over the last week, and its limits can be changed
+there.
+
+The same page has two more tabs, each at the address it had as a page of its
+own: **Limits** (`/admin/quotas`) is every limit in the tree at once — per
+branch, per member of any part of it, and per person — with the capacity quotas;
+**Usage** (`/admin/usage`) is the cluster's use over time by organization, team,
+project or person, and the GPU time each owner held. The page works as plain
+forms; with scripts on, a change updates it in place and the projects you had
+open stay open.
 
 Three things are deliberately not there:
 

docs/reference.md

@@ -39,31 +39,10 @@ rather than repeated.
 | `flush_interval` | `time.Duration` | [Auth](#auth) |
 | `batch_size` | `int` | [Auth](#auth) |
 | `buffer_size` | `int` | [Auth](#auth) |
-| `name` | `string` | [Band](#band) |
-| `priority` | `int` | [Band](#band) |
-| `saturation_ceiling` | `float64` | [Band](#band) |
+| `usage_retention` | `time.Duration` | [Auth](#auth) |
 | `tools` | `bool` | [Capabilities](#capabilities) |
 | `vision` | `bool` | [Capabilities](#capabilities) |
 | `embeddings` | `bool` | [Capabilities](#capabilities) |
-| `node_id` | `string` | [Config](#config) |
-| `server` | `string` | [Config](#config) |
-| `servers` | `[]string` | [Config](#config) |
-| `hostname` | `string` | [Config](#config) |
-| `mgmt_addr` | `string` | [Config](#config) |
-| `labels` | `map[string]string` | [Config](#config) |
-| `mgmt_interface` | `string` | [Config](#config) |
-| `fabrics` | `[]inventory.FabricAttachment` | [Config](#config) |
-| `heartbeat_interval` | `time.Duration` | [Config](#config) |
-| `reconnect_base` | `time.Duration` | [Config](#config) |
-| `reconnect_max` | `time.Duration` | [Config](#config) |
-| `adopt_ready_timeout` | `time.Duration` | [Config](#config) |
-| `tls` | `tlsconf.Config` | [Config](#config) |
-| `model_dirs` | `[]string` | [Config](#config) |
-| `engines_dir` | `string` | [Config](#config) |
-| `weights_cache` | `string` | [Config](#config) |
-| `discovery` | `discovery.Options` | [Config](#config) |
-| `engines` | `engines.Config` | [Config](#config) |
-| `supervisor` | `supervisor.Config` | [Config](#config) |
 | `interval` | `time.Duration` | [Control](#control) |
 | `base_backoff` | `time.Duration` | [Control](#control) |
 | `max_backoff` | `time.Duration` | [Control](#control) |
@@ -132,8 +111,6 @@ rather than repeated.
 | `match_ratio` | `float64` | [Prefix](#prefix) |
 | `balance_abs` | `float64` | [Prefix](#prefix) |
 | `balance_rel` | `float64` | [Prefix](#prefix) |
-| `prompt_per_1k` | `int64` | [Price](#price) |
-| `completion_per_1k` | `int64` | [Price](#price) |
 | `name` | `string` | [Profile](#profile) |
 | `engine` | `string` | [Profile](#profile) |
 | `gpu_class` | `string` | [Profile](#profile) |
@@ -161,6 +138,7 @@ rather than repeated.
 | `prefix` | `Prefix` | [Router](#router) |
 | `session_header` | `string` | [Router](#router) |
 | `inject_usage` | `bool` | [Router](#router) |
+| `tenant_label` | `string` | [Router](#router) |
 | `max_body_bytes` | `int64` | [Router](#router) |
 | `timeouts` | `Timeouts` | [Router](#router) |
 | `scoring` | `Scoring` | [Router](#router) |
@@ -208,6 +186,56 @@ rather than repeated.
 | `bin` | `string` | [VLLMConfig](#vllmconfig) |
 | `launch_mode` | `string` | [VLLMConfig](#vllmconfig) |
 | `ray_bin` | `string` | [VLLMConfig](#vllmconfig) |
+| `prompt_per_1k` | `int64` | [accounting.Price](#accountingprice) |
+| `completion_per_1k` | `int64` | [accounting.Price](#accountingprice) |
+| `name` | `string` | [admission.Band](#admissionband) |
+| `priority` | `int` | [admission.Band](#admissionband) |
+| `saturation_ceiling` | `float64` | [admission.Band](#admissionband) |
+| `bands` | `[]Band` | [admission.Config](#admissionconfig) |
+| `default_band` | `string` | [admission.Config](#admissionconfig) |
+| `queue_ttl` | `time.Duration` | [admission.Config](#admissionconfig) |
+| `queue_capacity_per_band` | `int` | [admission.Config](#admissionconfig) |
+| `queue_capacity_total` | `int` | [admission.Config](#admissionconfig) |
+| `interval` | `time.Duration` | [admission.Config](#admissionconfig) |
+| `node_id` | `string` | [agent.Config](#agentconfig) |
+| `server` | `string` | [agent.Config](#agentconfig) |
+| `servers` | `[]string` | [agent.Config](#agentconfig) |
+| `hostname` | `string` | [agent.Config](#agentconfig) |
+| `mgmt_addr` | `string` | [agent.Config](#agentconfig) |
+| `labels` | `map[string]string` | [agent.Config](#agentconfig) |
+| `mgmt_interface` | `string` | [agent.Config](#agentconfig) |
+| `fabrics` | `[]inventory.FabricAttachment` | [agent.Config](#agentconfig) |
+| `heartbeat_interval` | `time.Duration` | [agent.Config](#agentconfig) |
+| `reconnect_base` | `time.Duration` | [agent.Config](#agentconfig) |
+| `reconnect_max` | `time.Duration` | [agent.Config](#agentconfig) |
+| `adopt_ready_timeout` | `time.Duration` | [agent.Config](#agentconfig) |
+| `tls` | `tlsconf.Config` | [agent.Config](#agentconfig) |
+| `model_dirs` | `[]string` | [agent.Config](#agentconfig) |
+| `engines_dir` | `string` | [agent.Config](#agentconfig) |
+| `weights_cache` | `string` | [agent.Config](#agentconfig) |
+| `discovery` | `discovery.Options` | [agent.Config](#agentconfig) |
+| `engines` | `engines.Config` | [agent.Config](#agentconfig) |
+| `supervisor` | `supervisor.Config` | [agent.Config](#agentconfig) |
+| `name` | `string` | [config.Band](#configband) |
+| `priority` | `int` | [config.Band](#configband) |
+| `saturation_ceiling` | `float64` | [config.Band](#configband) |
+| `prompt_per_1k` | `int64` | [config.Price](#configprice) |
+| `completion_per_1k` | `int64` | [config.Price](#configprice) |
+| `disabled` | `[]string` | [engines.Config](#enginesconfig) |
+| `vllm` | `VLLMConfig` | [engines.Config](#enginesconfig) |
+| `sglang` | `SGLangConfig` | [engines.Config](#enginesconfig) |
+| `llamacpp` | `LlamaCppConfig` | [engines.Config](#enginesconfig) |
+| `sim` | `SimConfig` | [engines.Config](#enginesconfig) |
+| `work_dir` | `string` | [supervisor.Config](#supervisorconfig) |
+| `log_lines` | `int` | [supervisor.Config](#supervisorconfig) |
+| `stop_timeout` | `time.Duration` | [supervisor.Config](#supervisorconfig) |
+| `health_interval` | `time.Duration` | [supervisor.Config](#supervisorconfig) |
+| `rendezvous_timeout` | `time.Duration` | [supervisor.Config](#supervisorconfig) |
+| `alive_grace` | `time.Duration` | [supervisor.Config](#supervisorconfig) |
+| `ca` | `string` | [tlsconf.Config](#tlsconfconfig) |
+| `cert` | `string` | [tlsconf.Config](#tlsconfconfig) |
+| `key` | `string` | [tlsconf.Config](#tlsconfconfig) |
+| `insecure` | `bool` | [tlsconf.Config](#tlsconfconfig) |
 
 ## Adapter
 
@@ -347,7 +375,7 @@ table costs whatever DefaultPrice says, which is nothing unless set.
 
 `Price`
 
-Takes a [Price](#price) block.
+Takes a [config.Price](#configprice) block.
 
 ### `flush_interval`
 
@@ -367,19 +395,16 @@ BufferSize bounds the queue. Beyond it records are dropped rather than
 blocking a request: losing accounting is bad, turning a stalled database
 into an outage is worse.
 
-## Band
+### `usage_retention`
 
-### `name`
+`time.Duration`
 
-`string`
-
-### `priority`
-
-`int`
-
-### `saturation_ceiling`
-
-`float64`
+UsageRetention is how long the record of each request is kept: who sent
+it, which replica answered, how long it queued — what `ctl request`
+reads. Zero keeps it for ever. Totals, charts and budgets are read from
+five-minute sums kept beside the records, which are not pruned, so this
+bounds the table that grows by a row a request without changing any
+figure a page shows. At least a day when set.
 
 ## Capabilities
 
@@ -395,152 +420,6 @@ into an outage is worse.
 
 `bool`
 
-## Config
-
-### `node_id`
-
-`string`
-
-NodeID defaults to the hostname. It must equal the common name of the
-agent's certificate, so set it only where the two differ.
-
-### `server`
-
-`string`
-
-Server is the control plane, and Servers is the same thing when there is
-more than one. Both are accepted: every configuration written before
-there could be a second control plane names one address, and none of
-them should have to change.
-
-The order is tried in order, and a refusal moves to the next. Only the
-leader accepts an agent, so a standby is skipped in the time it takes to
-be told so — which is what makes failover a reconnect rather than an
-edit to every node's configuration.
-
-### `servers`
-
-`[]string`
-
-### `hostname`
-
-`string`
-
-Hostname defaults to the machine's.
-
-### `mgmt_addr`
-
-`string`
-
-MgmtAddr is the address engines are reached on. It defaults to this
-machine's address on the route to the control plane, which is the one
-the control plane and the router can reach.
-
-### `labels`
-
-`map[string]string`
-
-Labels are free-form facts about a node for a profile's node_selector.
-Nothing needs them: memory, GPUs, engines and disk are measured, and a
-label restating one of those goes stale the day the machine changes.
-
-### `mgmt_interface`
-
-`string`
-
-MgmtInterface is the ethernet interface NCCL bootstraps over. It
-defaults to the interface holding MgmtAddr, then eth0.
-
-### `fabrics`
-
-`[]inventory.FabricAttachment`
-
-Fabrics labels this node's HCAs. It is explicit on purpose: OpenSM's
-default subnet prefix is the same on every fabric, so nothing can be
-inferred from the wire (plan §5.2).
-
-### `heartbeat_interval`
-
-`time.Duration`
-
-### `reconnect_base`
-
-`time.Duration`
-
-ReconnectBase is the first reconnect delay; it backs off from there.
-
-### `reconnect_max`
-
-`time.Duration`
-
-### `adopt_ready_timeout`
-
-`time.Duration`
-
-AdoptReadyTimeout bounds how long an adopted engine is given to answer a
-health check before the agent stops waiting on it.
-
-Shorter than a startup timeout on purpose: this engine is already
-running, so it is not loading a model, it is either serving or it is not.
-
-### `tls`
-
-`tlsconf.Config`
-
-TLS authenticates both ends of the control-plane stream. Leaving it unset
-keeps the plaintext behaviour, which only belongs on a trusted network.
-
-Takes a [Config](#config) block.
-
-### `model_dirs`
-
-`[]string`
-
-ModelDirs are the directories this node keeps model weights in, beyond
-WeightsCache, which is always looked in. Storage differs at every installation,
-so llmgw observes it rather than owning it: a Lustre mount, an image
-exported over the fabric, local NVMe and a copy made by hand all look
-identical from here, and placement only needs to know whether the file
-is present.
-
-### `engines_dir`
-
-`string`
-
-EnginesDir is where the agent installs engines, one directory per
-build. Defaults to /opt/llmgw/engines.
-
-### `weights_cache`
-
-`string`
-
-WeightsCache is where a remote source is downloaded to. Defaults to the
-first entry of ModelDirs, because a node that says where its models live
-has already answered this question, and otherwise to
-/var/lib/llmgw-weights.
-
-Fetching is the simple arrangement: every node keeps its own copy. The
-exotic ones need no code — weights on a shared filer or an image exported
-over the fabric are simply present, and present means nothing is fetched.
-
-### `discovery`
-
-`discovery.Options`
-
-Takes a [Options](#options) block.
-
-### `engines`
-
-`engines.Config`
-
-Takes a [Config](#config) block.
-
-### `supervisor`
-
-`supervisor.Config`
-
-Takes a [Config](#config) block.
-
 ## Control
 
 ### `interval`
@@ -769,7 +648,7 @@ agents use. Empty leaves the connection in plain HTTP, which is only
 acceptable on a network where nothing untrusted can reach that port —
 the mirror says as much in its own log when it starts that way.
 
-Takes a [Config](#config) block.
+Takes a [tlsconf.Config](#tlsconfconfig) block.
 
 ### `registry`
 
@@ -1031,16 +910,6 @@ before affinity is abandoned.
 
 `float64`
 
-## Price
-
-### `prompt_per_1k`
-
-`int64`
-
-### `completion_per_1k`
-
-`int64`
-
 ## Profile
 
 ### `name`
@@ -1242,6 +1111,16 @@ SessionHeader carries the session id for affinity (plan §7.5 tier 0).
 InjectUsage asks engines for usage even when the client did not, so token
 accounting works regardless of client behaviour.
 
+### `tenant_label`
+
+`string`
+
+TenantLabel is the level of the tree the per-tenant request series are
+labelled by — llmgw_tenant_requests_total and its siblings: "org",
+"team" or "project", or "none" for no such series. A label per project
+on a cluster of thousands is more series than Prometheus wants, and a
+team is what an alert is usually about; "team" is the default.
+
 ### `max_body_bytes`
 
 `int64`
@@ -1402,7 +1281,7 @@ AgentTLS authenticates the fleet-facing port. It deliberately does not
 cover the client-facing one: that surface is fronted by a publicly
 trusted certificate, while this one is mutual and private.
 
-Takes a [Config](#config) block.
+Takes a [tlsconf.Config](#tlsconfconfig) block.
 
 ### `monitoring`
 
@@ -1594,3 +1473,345 @@ node and has not been checked against a real cluster.
 
 RayBin is used when LaunchMode is "ray".
 
+## accounting.Price
+
+### `prompt_per_1k`
+
+`int64`
+
+### `completion_per_1k`
+
+`int64`
+
+## admission.Band
+
+### `name`
+
+`string`
+
+### `priority`
+
+`int`
+
+Priority orders bands; higher is served first.
+
+### `saturation_ceiling`
+
+`float64`
+
+SaturationCeiling is the pool saturation above which this band waits.
+1.0 means "never hold this band back".
+
+## admission.Config
+
+### `bands`
+
+`[]Band`
+
+### `default_band`
+
+`string`
+
+### `queue_ttl`
+
+`time.Duration`
+
+QueueTTL drops a request that has waited this long. Sending the GPU work
+the client has already given up on helps nobody.
+
+### `queue_capacity_per_band`
+
+`int`
+
+QueueCapacityPerBand bounds each band's queue. Rejecting immediately
+beats accumulating silently.
+
+### `queue_capacity_total`
+
+`int`
+
+QueueCapacityTotal bounds everything queued, across bands. When it is
+reached, a request is not refused while a lower band has requests
+queued: the newest of the lowest band is turned away instead
+(displacement), since the oldest have waited longest and are nearest
+their TTL. Zero takes half of every band's capacity together.
+
+### `interval`
+
+`time.Duration`
+
+Interval is how often the dispatcher re-checks saturation. Releases also
+wake it, so this is only the backstop for saturation falling on its own.
+
+## agent.Config
+
+### `node_id`
+
+`string`
+
+NodeID defaults to the hostname. It must equal the common name of the
+agent's certificate, so set it only where the two differ.
+
+### `server`
+
+`string`
+
+Server is the control plane, and Servers is the same thing when there is
+more than one. Both are accepted: every configuration written before
+there could be a second control plane names one address, and none of
+them should have to change.
+
+The order is tried in order, and a refusal moves to the next. Only the
+leader accepts an agent, so a standby is skipped in the time it takes to
+be told so — which is what makes failover a reconnect rather than an
+edit to every node's configuration.
+
+### `servers`
+
+`[]string`
+
+### `hostname`
+
+`string`
+
+Hostname defaults to the machine's.
+
+### `mgmt_addr`
+
+`string`
+
+MgmtAddr is the address engines are reached on. It defaults to this
+machine's address on the route to the control plane, which is the one
+the control plane and the router can reach.
+
+### `labels`
+
+`map[string]string`
+
+Labels are free-form facts about a node for a profile's node_selector.
+Nothing needs them: memory, GPUs, engines and disk are measured, and a
+label restating one of those goes stale the day the machine changes.
+
+### `mgmt_interface`
+
+`string`
+
+MgmtInterface is the ethernet interface NCCL bootstraps over. It
+defaults to the interface holding MgmtAddr, then eth0.
+
+### `fabrics`
+
+`[]inventory.FabricAttachment`
+
+Fabrics labels this node's HCAs. It is explicit on purpose: OpenSM's
+default subnet prefix is the same on every fabric, so nothing can be
+inferred from the wire (plan §5.2).
+
+### `heartbeat_interval`
+
+`time.Duration`
+
+### `reconnect_base`
+
+`time.Duration`
+
+ReconnectBase is the first reconnect delay; it backs off from there.
+
+### `reconnect_max`
+
+`time.Duration`
+
+### `adopt_ready_timeout`
+
+`time.Duration`
+
+AdoptReadyTimeout bounds how long an adopted engine is given to answer a
+health check before the agent stops waiting on it.
+
+Shorter than a startup timeout on purpose: this engine is already
+running, so it is not loading a model, it is either serving or it is not.
+
+### `tls`
+
+`tlsconf.Config`
+
+TLS authenticates both ends of the control-plane stream. Leaving it unset
+keeps the plaintext behaviour, which only belongs on a trusted network.
+
+Takes a [tlsconf.Config](#tlsconfconfig) block.
+
+### `model_dirs`
+
+`[]string`
+
+ModelDirs are the directories this node keeps model weights in, beyond
+WeightsCache, which is always looked in. Storage differs at every installation,
+so llmgw observes it rather than owning it: a Lustre mount, an image
+exported over the fabric, local NVMe and a copy made by hand all look
+identical from here, and placement only needs to know whether the file
+is present.
+
+### `engines_dir`
+
+`string`
+
+EnginesDir is where the agent installs engines, one directory per
+build. Defaults to /opt/llmgw/engines.
+
+### `weights_cache`
+
+`string`
+
+WeightsCache is where a remote source is downloaded to. Defaults to the
+first entry of ModelDirs, because a node that says where its models live
+has already answered this question, and otherwise to
+/var/lib/llmgw-weights.
+
+Fetching is the simple arrangement: every node keeps its own copy. The
+exotic ones need no code — weights on a shared filer or an image exported
+over the fabric are simply present, and present means nothing is fetched.
+
+### `discovery`
+
+`discovery.Options`
+
+Takes a [Options](#options) block.
+
+### `engines`
+
+`engines.Config`
+
+Takes a [engines.Config](#enginesconfig) block.
+
+### `supervisor`
+
+`supervisor.Config`
+
+Takes a [supervisor.Config](#supervisorconfig) block.
+
+## config.Band
+
+### `name`
+
+`string`
+
+### `priority`
+
+`int`
+
+### `saturation_ceiling`
+
+`float64`
+
+## config.Price
+
+### `prompt_per_1k`
+
+`int64`
+
+### `completion_per_1k`
+
+`int64`
+
+## engines.Config
+
+### `disabled`
+
+`[]string`
+
+Disabled are engines this node never installs or advertises, whatever
+the control plane asks: vLLM on a node of cards no build of it runs on,
+that is meant for llama.cpp alone.
+
+### `vllm`
+
+`VLLMConfig`
+
+Takes a [VLLMConfig](#vllmconfig) block.
+
+### `sglang`
+
+`SGLangConfig`
+
+Takes a [SGLangConfig](#sglangconfig) block.
+
+### `llamacpp`
+
+`LlamaCppConfig`
+
+Takes a [LlamaCppConfig](#llamacppconfig) block.
+
+### `sim`
+
+`SimConfig`
+
+Takes a [SimConfig](#simconfig) block.
+
+## supervisor.Config
+
+### `work_dir`
+
+`string`
+
+WorkDir holds pid files and per-shard scratch space.
+
+### `log_lines`
+
+`int`
+
+LogLines is how many lines of engine output to retain per shard.
+
+### `stop_timeout`
+
+`time.Duration`
+
+StopTimeout is how long a process gets to exit after SIGTERM before it is
+killed. Engines need time to free GPU memory cleanly.
+
+### `health_interval`
+
+`time.Duration`
+
+HealthInterval is how often a starting shard is polled.
+
+### `rendezvous_timeout`
+
+`time.Duration`
+
+RendezvousTimeout bounds the wait for a multi-node rendezvous point to
+start listening.
+
+### `alive_grace`
+
+`time.Duration`
+
+AliveGrace is how long a non-serving rank must stay up before it counts
+as ready. There is no external signal for a worker, so staying alive is
+the only evidence available.
+
+## tlsconf.Config
+
+### `ca`
+
+`string`
+
+CA is the bundle used to verify the other side. In a real deployment this
+is whichever authority issued the fleet's certificates.
+
+### `cert`
+
+`string`
+
+Cert and Key are this component's own identity.
+
+### `key`
+
+`string`
+
+### `insecure`
+
+`bool`
+
+Insecure disables TLS entirely. It exists for local runs against
+sim-engines; anything using it must stay on a trusted network.
+

go.mod

@@ -23,6 +23,7 @@ require (
 	go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.46.0
 	go.opentelemetry.io/otel/sdk v1.46.0
 	go.opentelemetry.io/otel/trace v1.46.0
+	go.yaml.in/yaml/v3 v3.0.5
 	golang.org/x/crypto v0.57.0
 	golang.org/x/net v0.59.0
 	golang.org/x/sync v0.23.0
@@ -54,6 +55,7 @@ require (
 	github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
 	github.com/jackc/puddle/v2 v2.2.2 // indirect
 	github.com/knadh/koanf/maps v0.1.2 // indirect
+	github.com/kylelemons/godebug v1.1.0 // indirect
 	github.com/lucasb-eyer/go-colorful v1.3.0 // indirect
 	github.com/mattn/go-localereader v0.0.1 // indirect
 	github.com/mattn/go-runewidth v0.0.19 // indirect
@@ -73,7 +75,6 @@ require (
 	go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.46.0 // indirect
 	go.opentelemetry.io/otel/metric v1.46.0 // indirect
 	go.opentelemetry.io/proto/otlp v1.11.0 // indirect
-	go.yaml.in/yaml/v3 v3.0.5 // indirect
 	golang.org/x/sys v0.48.0 // indirect
 	golang.org/x/text v0.42.0 // indirect
 	google.golang.org/genproto/googleapis/api v0.0.0-20260819154853-08b0e4226688 // indirect

go.sum

@@ -166,6 +166,8 @@ golang.org/x/sync v0.23.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0=
 golang.org/x/sys v0.0.0-20210809222454-d867a43fc93e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
 golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
+golang.org/x/term v0.46.0 h1:3+OXuTbaKDgwk8jTi3aSLHRlmWqHEUDUtxnbFigO4YE=
+golang.org/x/term v0.46.0/go.mod h1:+K02xbkittuwc0Am4abfA3Fc+XRGXkvBXNO88NCXPoc=
 golang.org/x/text v0.42.0 h1:JbOZXgfeCPU9gacVtYliJqOhD+zhrEqK4LfdpmlUZqI=
 golang.org/x/text v0.42.0/go.mod h1:ojzP1Z+2QtioaF8DTtO8K5q7JWVVYwZKenzujK0Zd0E=
 golang.org/x/tools v0.49.0 h1:3NI7VXzL9+1WZD52Dx2ttoPwD5DWrFGpl9mFZDlmisI=

internal/accounting/accounting.go

@@ -316,6 +316,17 @@ func (a *Accountant) Record(res Reservation, u store.Usage) {
 	}
 }
 
+// RecordRefused queues the row of a request a limit refused: nothing is
+// charged — it used nothing — but it is counted, so a team running into its
+// limits shows as one rather than as a team that went quiet.
+func (a *Accountant) RecordRefused(u store.Usage) {
+	if u.KeyID == "" || a.recorder == nil {
+		return
+	}
+	u.PromptTokens, u.CompletionTokens, u.CostMicros = 0, 0, 0
+	a.recorder.Record(u)
+}
+
 // Status returns what a key has used, for headers and `ctl`.
 func (a *Accountant) Status(p auth.Principal) (spent int64, rpm, tpm float64) {
 	if p.KeyID == "" {

internal/agent/agent.go

@@ -73,8 +73,11 @@ type Agent struct {
 	lastEngineProbe time.Time
 	probingEngines  bool
 
-	mu  sync.Mutex
-	inv discovery.Inventory
+	mu sync.Mutex
+	// serverList is every control plane to dial: the configured ones, and
+	// any the leader has named since. Read through servers().
+	serverList []string
+	inv        discovery.Inventory
 	// shardReplica remembers which replica a shard belongs to, so a status
 	// report can name both. The control plane keys on the replica.
 	shardReplica map[string]string
@@ -104,6 +107,8 @@ type Agent struct {
 	// sensors reads heat and power: CPU and board from hwmon and RAPL, GPUs
 	// from nvidia-smi.
 	sensors *discovery.SensorSampler
+	// wear reads what wears a GPU out: remapped rows, PCIe retries, XIDs.
+	wear *discovery.HealthSampler
 
 	// fetchWatchers are the shards waiting on each source, so progress on one
 	// download reaches every shard that asked for it. A fetch is single-flight
@@ -150,6 +155,8 @@ func New(cfg Config, httpClient connectHTTPClient, log *slog.Logger) (*Agent, er
 		provKick:     make(chan struct{}, 1),
 		load:         discovery.NewLoadSampler(),
 		sensors:      discovery.NewSensorSampler(),
+		wear:         discovery.NewHealthSampler(),
+		serverList:   loadControlPlanes(cfg.Servers, cfg.TLS.Enabled()),
 	}
 	a.prov = &provision.Installer{Root: cfg.EnginesDir, Log: log, OnStatus: a.provisionStatus}
 	return a, nil
@@ -182,7 +189,9 @@ func (a *Agent) Run(ctx context.Context) error {
 	attempt := 0
 	next := 0
 	for ctx.Err() == nil {
-		server := a.cfg.Servers[next%len(a.cfg.Servers)]
+		// Read each time round: the leader may have named another since.
+		servers := a.servers()
+		server := servers[next%len(servers)]
 		err := a.session(ctx, server)
 		if ctx.Err() != nil {
 			break
@@ -197,12 +206,12 @@ func (a *Agent) Run(ctx context.Context) error {
 		// No wait when there is somewhere else to try: the delay exists to
 		// stop hammering one control plane, and the next address has not been
 		// hammered. With a single address this is the old behaviour exactly.
-		if len(a.cfg.Servers) > 1 && next%len(a.cfg.Servers) != 0 {
+		if len(servers) > 1 && next%len(servers) != 0 {
 			delay = 0
 		}
 		a.log.Warn("disconnected from the control plane, retrying",
 			"err", err, "server", server,
-			"next", a.cfg.Servers[next%len(a.cfg.Servers)],
+			"next", servers[next%len(servers)],
 			"in", delay.Round(time.Millisecond), "attempt", attempt)
 
 		select {
@@ -588,8 +597,24 @@ func (a *Agent) heartbeat() *llmgwv1.AgentMessage {
 	}
 	a.mu.Unlock()
 
-	for _, g := range inv.GPUs {
-		hb.GpuMemoryUsedMb = append(hb.GpuMemoryUsedMb, int32(g.MemoryUsedMB))
+	// Measured now, not carried from registration: a card is shared with
+	// whatever else runs on the machine, and that comes and goes while the
+	// agent runs. The registration figure stays only for a node where
+	// nvidia-smi cannot be asked.
+	if len(inv.GPUs) > 0 {
+		var shards []int
+		for _, st := range a.sup.Statuses() {
+			if st.PID > 0 {
+				shards = append(shards, st.PID)
+			}
+		}
+		hb.GpuForeignMemoryMb, hb.GpuMemoryUsedMb = a.foreignGPUMemory(context.Background(), shards)
+	}
+	if len(hb.GpuMemoryUsedMb) != len(inv.GPUs) {
+		hb.GpuForeignMemoryMb, hb.GpuMemoryUsedMb = nil, nil
+		for _, g := range inv.GPUs {
+			hb.GpuMemoryUsedMb = append(hb.GpuMemoryUsedMb, int32(g.MemoryUsedMB))
+		}
 	}
 
 	// Rescanned here rather than reused from registration. Disk is not a fact
@@ -656,11 +681,34 @@ func (a *Agent) sampleLoad() *llmgwv1.NodeLoad {
 	if a.sensors != nil {
 		se := a.sensors.Sample(context.Background())
 		out.CpuTempC, out.CpuPowerW, out.InletTempC = se.CPUTempC, se.CPUPowerW, se.InletTempC
+		var wear map[int]discovery.GPUWear
+		if a.wear != nil && len(se.GPUs) > 0 {
+			wear = map[int]discovery.GPUWear{}
+			for _, w := range a.wear.Sample(context.Background()) {
+				wear[w.Index] = w
+			}
+		}
 		for _, g := range se.GPUs {
-			out.Gpus = append(out.Gpus, &llmgwv1.GPUSensor{
+			gs := &llmgwv1.GPUSensor{
 				Index: int32(g.Index), TempC: g.TempC, MemoryTempC: g.MemoryTempC,
 				PowerW: g.PowerW, PowerLimitW: g.PowerLimitW, Throttle: g.Throttle,
-			})
+				UtilizationPercent: g.UtilPercent, MemoryUtilizationPercent: g.MemUtilPercent,
+				MemoryUsedMb: g.MemoryUsedMB, MemoryTotalMb: g.MemoryTotalMB,
+				SmClockMhz: g.SMClockMHz, EccUncorrected: g.ECCUncorrected,
+			}
+			if w, ok := wear[g.Index]; ok {
+				gs.HasWear = true
+				gs.PcieReplays, gs.RetiredPagesSbe, gs.RetiredPagesDbe, gs.RetirementPending =
+					w.PCIeReplays, w.RetiredSBE, w.RetiredDBE, w.RetirePending
+				gs.RemappedRowsCorrectable, gs.RemappedRowsUncorrectable, gs.RemapPending, gs.RemapFailure =
+					w.RemappedCorrectable, w.RemappedUncorrectable, w.RemapPending, w.RemapFailure
+				gs.NvlinkCrcErrors, gs.NvlinkReplayErrors, gs.NvlinkRecoveryErrors = w.NVLinkCRC, w.NVLinkReplay, w.NVLinkRecovery
+				gs.XidErrors, gs.LastXid, gs.LastXidText = w.XIDs, int32(w.LastXID), w.LastXIDText
+				if !w.LastXIDAt.IsZero() {
+					gs.LastXidUnix = w.LastXIDAt.Unix()
+				}
+			}
+			out.Gpus = append(out.Gpus, gs)
 		}
 	}
 	return out
@@ -750,6 +798,8 @@ func (a *Agent) handle(ctx context.Context, msg *llmgwv1.ControlMessage, out cha
 		a.setProvision(m.Provision)
 	case *llmgwv1.ControlMessage_Upgrade:
 		go a.upgrade(ctx, m.Upgrade, out)
+	case *llmgwv1.ControlMessage_ControlPlanes:
+		a.setControlPlanes(m.ControlPlanes)
 	case *llmgwv1.ControlMessage_Logs:
 		reply := a.logsReply(m.Logs)
 		// Off the receive loop: a full send queue must not stop commands being
@@ -1042,7 +1092,9 @@ func Client(cfg tlsconf.Config, timeout time.Duration) (*http.Client, error) {
 	if !cfg.Enabled() {
 		return &http.Client{
 			Transport: &http2.Transport{
-				AllowHTTP: true,
+				AllowHTTP:       true,
+				ReadIdleTimeout: streamIdle,
+				PingTimeout:     streamPing,
 				DialTLSContext: func(ctx context.Context, network, addr string, _ *tls.Config) (net.Conn, error) {
 					return (&net.Dialer{Timeout: timeout}).DialContext(ctx, network, addr)
 				},
@@ -1057,9 +1109,31 @@ func Client(cfg tlsconf.Config, timeout time.Duration) (*http.Client, error) {
 	if err != nil {
 		return nil, err
 	}
-	return &http.Client{Transport: &http2.Transport{TLSClientConfig: tc}}, nil
+	return &http.Client{Transport: &http2.Transport{
+		TLSClientConfig: tc,
+		ReadIdleTimeout: streamIdle,
+		PingTimeout:     streamPing,
+		// Bounded, as the plaintext branch's is: dialling a control plane
+		// whose machine is off waits out the kernel's connect retries,
+		// about two minutes, before the next address is tried.
+		DialTLSContext: func(ctx context.Context, network, addr string, c *tls.Config) (net.Conn, error) {
+			d := tls.Dialer{NetDialer: &net.Dialer{Timeout: timeout}, Config: c}
+			return d.DialContext(ctx, network, addr)
+		},
+	}}, nil
 }
 
+// A stream to a control plane whose machine stopped dead — powered off, cut
+// off — is never closed: nothing sends the FIN, and the agent's heartbeats sit
+// in the socket's buffer until the kernel gives up retransmitting, fifteen
+// minutes later. Found by switching off a leader: the standby took over in 14
+// seconds and the node stayed with the dead one. An HTTP/2 ping after ten
+// quiet seconds, given five to answer, finds it in a quarter of a minute.
+const (
+	streamIdle = 10 * time.Second
+	streamPing = 5 * time.Second
+)
+
 // localizeSpec fills in what only this node knows.
 //
 // The interface NCCL bootstraps over is this node's, not the cluster's: the

internal/agent/controlplanes.go

@@ -0,0 +1,77 @@
+package agent
+
+import (
+	"encoding/json"
+	"os"
+	"path/filepath"
+	"slices"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+)
+
+// controlPlanesFile keeps the control planes the leader last named, beside
+// the agent's configuration rather than in it: the configuration is the
+// operator's, and a list rewritten in it would be a file nobody wrote.
+var controlPlanesFile = "/var/lib/llmgw-agent/control-planes.json"
+
+// servers is every address to dial, in the order to try them: the
+// configuration's, then any the leader named that it lacks.
+func (a *Agent) servers() []string {
+	a.mu.Lock()
+	defer a.mu.Unlock()
+	if len(a.serverList) == 0 {
+		return a.cfg.Servers
+	}
+	return a.serverList
+}
+
+// mergeServers puts the leader's list after the configured one, without
+// repeats, and says whether that changed anything.
+func mergeServers(configured, told []string, secure bool) []string {
+	out := slices.Clone(configured)
+	for _, s := range told {
+		fixed, err := normalizeOne(s, secure)
+		if err != nil {
+			continue
+		}
+		if !slices.Contains(out, fixed) {
+			out = append(out, fixed)
+		}
+	}
+	return out
+}
+
+// setControlPlanes takes the leader's list: used from the next reconnect,
+// and kept for the next start.
+func (a *Agent) setControlPlanes(m *llmgwv1.ControlPlanes) {
+	told := m.GetServers()
+	merged := mergeServers(a.cfg.Servers, told, a.cfg.TLS.Enabled())
+	a.mu.Lock()
+	changed := !slices.Equal(merged, a.serverList)
+	a.serverList = merged
+	a.mu.Unlock()
+	if !changed {
+		return
+	}
+	a.log.Info("the control planes to dial", "servers", merged)
+	b, _ := json.Marshal(told)
+	if err := os.MkdirAll(filepath.Dir(controlPlanesFile), 0o755); err == nil {
+		tmp := controlPlanesFile + ".new"
+		if err := os.WriteFile(tmp, b, 0o644); err == nil {
+			_ = os.Rename(tmp, controlPlanesFile)
+		}
+	}
+}
+
+// loadControlPlanes is the list kept from the last time the leader named it.
+func loadControlPlanes(configured []string, secure bool) []string {
+	b, err := os.ReadFile(controlPlanesFile)
+	if err != nil {
+		return configured
+	}
+	var told []string
+	if json.Unmarshal(b, &told) != nil {
+		return configured
+	}
+	return mergeServers(configured, told, secure)
+}

internal/agent/controlplanes_test.go

@@ -0,0 +1,41 @@
+package agent
+
+import (
+	"log/slog"
+	"path/filepath"
+	"slices"
+	"testing"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
+)
+
+func TestAControlPlaneTheLeaderNamesIsDialledAndRemembered(t *testing.T) {
+	old := controlPlanesFile
+	controlPlanesFile = filepath.Join(t.TempDir(), "control-planes.json")
+	t.Cleanup(func() { controlPlanesFile = old })
+
+	configured := []string{"https://10.0.0.140:7443"}
+	// TLS on, as every onboarded node has it: an agent without it refuses
+	// https:// addresses, and the leader's are https://.
+	a := &Agent{cfg: Config{Servers: configured, TLS: tlsconf.Config{CA: "/etc/llmgw/ca.pem"}}, log: slog.New(slog.DiscardHandler)}
+	if got := a.servers(); !slices.Equal(got, configured) {
+		t.Fatalf("before anything is named: %v", got)
+	}
+	// The configured one stays first; the new one is added after it, once.
+	a.setControlPlanes(&llmgwv1.ControlPlanes{Servers: []string{"https://10.0.0.140:7443", "https://10.0.0.150:7443"}})
+	want := []string{"https://10.0.0.140:7443", "https://10.0.0.150:7443"}
+	if got := a.servers(); !slices.Equal(got, want) {
+		t.Errorf("after the leader named two: %v", got)
+	}
+	// And after a restart, before the leader says anything.
+	if got := loadControlPlanes(configured, true); !slices.Equal(got, want) {
+		t.Errorf("after a restart: %v", got)
+	}
+	// A list that drops the configured one does not drop it: the
+	// configuration is the operator's.
+	a.setControlPlanes(&llmgwv1.ControlPlanes{Servers: []string{"https://10.0.0.150:7443"}})
+	if got := a.servers(); got[0] != "https://10.0.0.140:7443" {
+		t.Errorf("the configured server was dropped: %v", got)
+	}
+}

internal/agent/discovery/gpu_smi.go

@@ -106,9 +106,27 @@ func parseSMI(out string) ([]GPU, error) {
 	return gpus, nil
 }
 
-// residentMemoryToleranceMB is how much allocated memory is written off as
+// ForeignMemoryToleranceMB is how much allocated memory is written off as
 // driver overhead rather than a foreign process.
-const residentMemoryToleranceMB = 512
+const ForeignMemoryToleranceMB = 512
+
+// foreignMemorySuffix ends the problem a card with another tenant on it is
+// kept out of placement for. One phrase, so that the control plane can find
+// it again and drop it when the tenant leaves.
+const foreignMemorySuffix = " MiB already allocated by another process"
+
+// ForeignMemoryProblem is the problem for mb of memory held by something
+// that is not one of this agent's engines, or "" when that is too little to
+// count.
+func ForeignMemoryProblem(mb int) string {
+	if mb <= ForeignMemoryToleranceMB {
+		return ""
+	}
+	return fmt.Sprintf("%d%s", mb, foreignMemorySuffix)
+}
+
+// IsForeignMemoryProblem reports whether a problem is ForeignMemoryProblem's.
+func IsForeignMemoryProblem(p string) bool { return strings.HasSuffix(p, foreignMemorySuffix) }
 
 // assessHealth flags conditions that should keep a GPU out of placement.
 func assessHealth(g *GPU) {
@@ -119,8 +137,12 @@ func assessHealth(g *GPU) {
 	// Memory already in use means something else is on the card. Starting an
 	// engine there would either fail to allocate or quietly evict the other
 	// process's throughput.
-	if g.MemoryUsedMB > residentMemoryToleranceMB {
-		g.Health.Problems = append(g.Health.Problems,
-			fmt.Sprintf("%d MiB already allocated by another process", g.MemoryUsedMB))
+	//
+	// Only a first reading: this runs before the agent knows which processes
+	// are its own, so an engine still running from before a restart reads as
+	// foreign here. Every heartbeat measures it again, without the agent's
+	// own engines, and the control plane replaces this with that.
+	if p := ForeignMemoryProblem(g.MemoryUsedMB); p != "" {
+		g.Health.Problems = append(g.Health.Problems, p)
 	}
 }

internal/agent/discovery/gpuhealth.go

@@ -0,0 +1,280 @@
+package discovery
+
+import (
+	"context"
+	"encoding/xml"
+	"os/exec"
+	"regexp"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+)
+
+// GPUWear is what wears a card out or takes it down, beyond temperature:
+// read every minute, from nvidia-smi and the kernel's log. The early signs of
+// a card failing are here — a growing count of rows its memory had to remap,
+// PCIe link retries, XID errors — long before a replica on it fails.
+//
+// A count of -1 is not reported: a card without row remapping (before
+// Ampere) has no such count, which is not the same as none.
+type GPUWear struct {
+	Index int
+	// PCIeReplays are link-level retries on the card's PCIe link: a few are
+	// noise, a count that keeps climbing is a riser, slot or cable.
+	PCIeReplays int64
+	// RetiredSBE and RetiredDBE are memory pages retired for single and
+	// double bit errors (cards before Ampere); RetirePending is a page
+	// waiting for a reset to be retired.
+	RetiredSBE, RetiredDBE int64
+	RetirePending          bool
+	// RemappedCorrectable and RemappedUncorrectable are memory rows remapped
+	// (Ampere and later). RemapPending waits for a reset; RemapFailure is a
+	// card out of spare rows — it needs replacing.
+	RemappedCorrectable, RemappedUncorrectable int64
+	RemapPending, RemapFailure                 bool
+	// NVLink errors summed over the card's links, by kind.
+	NVLinkCRC, NVLinkReplay, NVLinkRecovery int64
+	// XIDs are the driver's error events for this card since the agent
+	// started, and the last of them.
+	XIDs        int64
+	LastXID     int
+	LastXIDText string
+	LastXIDAt   time.Time
+}
+
+// healthEvery is how often health is read: it changes over hours, and the
+// kernel log and nvidia-smi's full report are the heavier reads.
+const healthEvery = time.Minute
+
+// HealthSampler reads GPUWear, keeping the XID counts between reads.
+type HealthSampler struct {
+	run func(ctx context.Context, name string, args ...string) ([]byte, error)
+	now func() time.Time
+
+	mu      sync.Mutex
+	at      time.Time
+	last    []GPUWear
+	since   time.Time
+	xids    map[int]*GPUWear
+	started bool
+}
+
+// NewHealthSampler reads the machine's own nvidia-smi and kernel log.
+func NewHealthSampler() *HealthSampler {
+	return &HealthSampler{now: time.Now, xids: map[int]*GPUWear{},
+		run: func(ctx context.Context, name string, args ...string) ([]byte, error) {
+			return exec.CommandContext(ctx, name, args...).Output() //nolint:gosec // fixed tools
+		}}
+}
+
+// Sample is the cards' health, read again once healthEvery has passed. Nil
+// on a machine without nvidia-smi.
+func (h *HealthSampler) Sample(ctx context.Context) []GPUWear {
+	h.mu.Lock()
+	defer h.mu.Unlock()
+	now := h.now()
+	if !h.at.IsZero() && now.Sub(h.at) < healthEvery {
+		return h.last
+	}
+	h.at = now
+	ctx, cancel := context.WithTimeout(ctx, 15*time.Second)
+	defer cancel()
+	report, err := h.run(ctx, "nvidia-smi", "-q", "-x")
+	if err != nil {
+		h.last = nil
+		return nil
+	}
+	cards, bus := parseHealthReport(report)
+	if out, err := h.run(ctx, "nvidia-smi", "nvlink", "-e"); err == nil {
+		for i, e := range parseNVLinkErrors(string(out)) {
+			if i < len(cards) {
+				cards[i].NVLinkCRC, cards[i].NVLinkReplay, cards[i].NVLinkRecovery = e[0], e[1], e[2]
+			}
+		}
+	}
+	h.readXIDs(ctx, now, bus)
+	for i := range cards {
+		if x := h.xids[i]; x != nil {
+			cards[i].XIDs, cards[i].LastXID, cards[i].LastXIDText, cards[i].LastXIDAt = x.XIDs, x.LastXID, x.LastXIDText, x.LastXIDAt
+		}
+	}
+	h.last = cards
+	return cards
+}
+
+// readXIDs adds the XID events logged since the last read. The first read
+// starts from now: what happened before the agent started was before, and
+// counting the whole log would raise every alert on every restart.
+func (h *HealthSampler) readXIDs(ctx context.Context, now time.Time, bus map[string]int) {
+	if !h.started {
+		h.started, h.since = true, now
+		return
+	}
+	since := h.since
+	h.since = now
+	out, err := h.run(ctx, "journalctl", "-k", "--no-pager", "-o", "short-unix",
+		"--since", "@"+strconv.FormatInt(since.Unix(), 10), "-g", "NVRM: Xid")
+	if err != nil {
+		// No journal, or nothing matched (journalctl exits 1 for that).
+		return
+	}
+	for _, e := range parseXIDs(string(out)) {
+		i, ok := bus[e.bus]
+		if !ok {
+			continue
+		}
+		x := h.xids[i]
+		if x == nil {
+			x = &GPUWear{Index: i}
+			h.xids[i] = x
+		}
+		x.XIDs++
+		x.LastXID, x.LastXIDText = e.code, e.text
+		x.LastXIDAt = now
+		if e.at > 0 {
+			x.LastXIDAt = time.Unix(int64(e.at), 0)
+		}
+	}
+}
+
+// healthLog is the part of nvidia-smi's XML report health is read from.
+type healthLog struct {
+	GPUs []struct {
+		ID  string `xml:"id,attr"`
+		PCI struct {
+			Replay string `xml:"replay_counter"`
+		} `xml:"pci"`
+		Retired struct {
+			SBE     string `xml:"multiple_single_bit_retirement>retired_count"`
+			DBE     string `xml:"double_bit_retirement>retired_count"`
+			Pending string `xml:"pending_retirement"`
+			// Older drivers name it pending_blacklist.
+			Blacklist string `xml:"pending_blacklist"`
+		} `xml:"retired_pages"`
+		Remapped struct {
+			Corr    string `xml:"remapped_row_corr"`
+			Unc     string `xml:"remapped_row_unc"`
+			Pending string `xml:"remapped_row_pending"`
+			Failure string `xml:"remapped_row_failure"`
+		} `xml:"remapped_rows"`
+	} `xml:"gpu"`
+}
+
+// parseHealthReport reads each card's health, in index order, and maps each
+// card's PCI bus ("3b:00", as the kernel names it) to its index.
+func parseHealthReport(report []byte) ([]GPUWear, map[string]int) {
+	var log healthLog
+	if err := xml.Unmarshal(report, &log); err != nil {
+		return nil, nil
+	}
+	bus := map[string]int{}
+	out := make([]GPUWear, 0, len(log.GPUs))
+	for i, g := range log.GPUs {
+		bus[busOf(g.ID)] = i
+		out = append(out, GPUWear{
+			Index:                 i,
+			PCIeReplays:           count(g.PCI.Replay),
+			RetiredSBE:            count(g.Retired.SBE),
+			RetiredDBE:            count(g.Retired.DBE),
+			RetirePending:         yes(g.Retired.Pending) || yes(g.Retired.Blacklist),
+			RemappedCorrectable:   count(g.Remapped.Corr),
+			RemappedUncorrectable: count(g.Remapped.Unc),
+			RemapPending:          yes(g.Remapped.Pending),
+			RemapFailure:          yes(g.Remapped.Failure),
+		})
+	}
+	return out, bus
+}
+
+// busOf is a PCI address as the kernel's XID lines carry it: nvidia-smi
+// says "00000000:3B:00.0", the kernel "0000:3b:00".
+func busOf(id string) string {
+	id = strings.ToLower(strings.TrimSpace(id))
+	if i := strings.LastIndex(id, "."); i >= 0 {
+		id = id[:i]
+	}
+	parts := strings.Split(id, ":")
+	if len(parts) >= 2 {
+		return parts[len(parts)-2] + ":" + parts[len(parts)-1]
+	}
+	return id
+}
+
+func count(s string) int64 {
+	n, err := strconv.ParseInt(strings.TrimSpace(s), 10, 64)
+	if err != nil {
+		return -1
+	}
+	return n
+}
+
+func yes(s string) bool { return strings.EqualFold(strings.TrimSpace(s), "yes") }
+
+// nvlinkError matches a line of `nvidia-smi nvlink -e`, whose wording varies
+// by driver: "Link 0: Replay Errors: 0", "Link 0: CRC Errors: 0",
+// "Link 3: Data CRC Errors: 2".
+var (
+	nvlinkGPU   = regexp.MustCompile(`^GPU (\d+):`)
+	nvlinkError = regexp.MustCompile(`Link \d+:\s*(.*?)\s*Errors?\s*:\s*(\d+)`)
+)
+
+// parseNVLinkErrors sums each card's link errors by kind: CRC, replay,
+// recovery.
+func parseNVLinkErrors(out string) map[int][3]int64 {
+	res := map[int][3]int64{}
+	gpu := -1
+	for _, line := range strings.Split(out, "\n") {
+		if m := nvlinkGPU.FindStringSubmatch(strings.TrimSpace(line)); m != nil {
+			gpu, _ = strconv.Atoi(m[1])
+			continue
+		}
+		m := nvlinkError.FindStringSubmatch(line)
+		if m == nil || gpu < 0 {
+			continue
+		}
+		n, _ := strconv.ParseInt(m[2], 10, 64)
+		e := res[gpu]
+		kind := strings.ToLower(m[1])
+		switch {
+		case strings.Contains(kind, "crc"):
+			e[0] += n
+		case strings.Contains(kind, "replay"):
+			e[1] += n
+		case strings.Contains(kind, "recovery"):
+			e[2] += n
+		}
+		res[gpu] = e
+	}
+	return res
+}
+
+type xidEvent struct {
+	bus  string
+	code int
+	text string
+	at   float64
+}
+
+// xidLine matches the driver's report of an XID:
+// "NVRM: Xid (PCI:0000:3b:00): 79, pid=..., GPU has fallen off the bus."
+var xidLine = regexp.MustCompile(`NVRM: Xid \(PCI:([0-9a-fA-F:.]+)\): (\d+),\s*(.*)$`)
+
+// parseXIDs reads journalctl's short-unix lines: a timestamp, then the
+// message.
+func parseXIDs(out string) []xidEvent {
+	var evs []xidEvent
+	for _, line := range strings.Split(out, "\n") {
+		m := xidLine.FindStringSubmatch(line)
+		if m == nil {
+			continue
+		}
+		code, _ := strconv.Atoi(m[2])
+		e := xidEvent{bus: busOf(m[1]), code: code, text: strings.TrimSpace(m[3])}
+		if f := strings.Fields(line); len(f) > 0 {
+			e.at, _ = strconv.ParseFloat(f[0], 64)
+		}
+		evs = append(evs, e)
+	}
+	return evs
+}

internal/agent/discovery/gpuhealth_test.go

@@ -0,0 +1,101 @@
+package discovery
+
+import (
+	"context"
+	"errors"
+	"strings"
+	"testing"
+	"time"
+)
+
+const healthXML = `<?xml version="1.0" ?>
+<nvidia_smi_log>
+  <gpu id="00000000:3B:00.0">
+    <pci><replay_counter>0</replay_counter></pci>
+    <retired_pages>
+      <multiple_single_bit_retirement><retired_count>N/A</retired_count></multiple_single_bit_retirement>
+      <double_bit_retirement><retired_count>N/A</retired_count></double_bit_retirement>
+      <pending_retirement>N/A</pending_retirement>
+    </retired_pages>
+    <remapped_rows>
+      <remapped_row_corr>2</remapped_row_corr>
+      <remapped_row_unc>1</remapped_row_unc>
+      <remapped_row_pending>Yes</remapped_row_pending>
+      <remapped_row_failure>No</remapped_row_failure>
+    </remapped_rows>
+  </gpu>
+  <gpu id="00000000:5E:00.0">
+    <pci><replay_counter>4021</replay_counter></pci>
+    <retired_pages>
+      <multiple_single_bit_retirement><retired_count>3</retired_count></multiple_single_bit_retirement>
+      <double_bit_retirement><retired_count>1</retired_count></double_bit_retirement>
+      <pending_blacklist>Yes</pending_blacklist>
+    </retired_pages>
+    <remapped_rows>N/A</remapped_rows>
+  </gpu>
+</nvidia_smi_log>`
+
+const nvlinkOut = `GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-aaa)
+	 Link 0: Replay Errors: 1
+	 Link 0: Recovery Errors: 0
+	 Link 0: CRC Errors: 5
+	 Link 1: Data CRC Errors: 2
+GPU 1: NVIDIA H100 80GB HBM3 (UUID: GPU-bbb)
+	 Link 0: Replay Errors: 0
+`
+
+func TestAGPUsWearAndFailuresAreReadAndCountedFromWhenTheAgentStarted(t *testing.T) {
+	now := time.Unix(1_800_000_000, 0)
+	journal := ""
+	h := &HealthSampler{now: func() time.Time { return now }, xids: map[int]*GPUWear{},
+		run: func(_ context.Context, name string, args ...string) ([]byte, error) {
+			switch {
+			case name == "nvidia-smi" && args[0] == "-q":
+				return []byte(healthXML), nil
+			case name == "nvidia-smi" && args[0] == "nvlink":
+				return []byte(nvlinkOut), nil
+			case name == "journalctl":
+				if journal == "" {
+					return nil, errors.New("exit status 1")
+				}
+				return []byte(journal), nil
+			}
+			return nil, errors.New("unexpected " + name)
+		}}
+	got := h.Sample(t.Context())
+	if len(got) != 2 {
+		t.Fatalf("%+v", got)
+	}
+	a, b := got[0], got[1]
+	if a.RemappedCorrectable != 2 || a.RemappedUncorrectable != 1 || !a.RemapPending || a.RemapFailure ||
+		a.RetiredSBE != -1 || a.PCIeReplays != 0 || a.NVLinkCRC != 7 || a.NVLinkReplay != 1 {
+		t.Errorf("card 0 %+v", a)
+	}
+	if b.PCIeReplays != 4021 || b.RetiredSBE != 3 || b.RetiredDBE != 1 || !b.RetirePending || b.RemappedCorrectable != -1 {
+		t.Errorf("card 1 %+v", b)
+	}
+
+	// An XID on card 1, logged after the agent started; read on the next
+	// sample a minute on.
+	journal = "1800000030.123456 gpu07 kernel: NVRM: Xid (PCI:0000:5e:00): 79, pid=1234, name=llama-server, GPU has fallen off the bus.\n"
+	now = now.Add(61 * time.Second)
+	got = h.Sample(t.Context())
+	if got[1].XIDs != 1 || got[1].LastXID != 79 || !strings.Contains(got[1].LastXIDText, "fallen off the bus") ||
+		got[1].LastXIDAt.Unix() != 1_800_000_030 || got[0].XIDs != 0 {
+		t.Errorf("after an XID: %+v / %+v", got[0], got[1])
+	}
+	// Within the minute, the last reading.
+	journal = ""
+	now = now.Add(10 * time.Second)
+	if again := h.Sample(t.Context()); again[1].XIDs != 1 {
+		t.Errorf("%+v", again[1])
+	}
+}
+
+func TestBusIDsAreReadAsTheKernelWritesThem(t *testing.T) {
+	for in, want := range map[string]string{"00000000:3B:00.0": "3b:00", "0000:3b:00": "3b:00", "0000:5e:00.0": "5e:00"} {
+		if got := busOf(in); got != want {
+			t.Errorf("%s: %s, want %s", in, got, want)
+		}
+	}
+}

internal/agent/discovery/gpuprocs.go

@@ -2,6 +2,7 @@ package discovery
 
 import (
 	"context"
+	"encoding/xml"
 	"fmt"
 	"os/exec"
 	"strconv"
@@ -10,10 +11,14 @@ import (
 
 // GPUProcesses reports which processes hold memory on each GPU, by the GPU's
 // nvidia-smi index: what a process actually computes on, as opposed to what it
-// was told to use. A llama.cpp started with --n-gpu-layers on a GPU whose CUDA
+// was told to use. A llama.cpp started with --n-gpu-layers on a GPU whose
 // backend did not load runs on the CPU without a word, and this is where that
 // shows — the process is on no GPU's list.
 //
+// From the full query rather than --query-compute-apps, which lists compute
+// processes only: llama.cpp's Vulkan build is listed as a graphics one, and
+// would have read as not on its GPU at all.
+//
 // A node with no nvidia-smi returns nil and no error, as discovery does.
 func GPUProcesses(ctx context.Context, binary string) (map[int][]int, error) {
 	if binary == "" {
@@ -22,44 +27,94 @@ func GPUProcesses(ctx context.Context, binary string) (map[int][]int, error) {
 		}
 		binary = "nvidia-smi"
 	}
-	gpus, err := exec.CommandContext(ctx, binary, "--query-gpu=index,uuid", "--format=csv,noheader").Output()
+	out, err := exec.CommandContext(ctx, binary, "-q", "-x").Output()
 	if err != nil {
 		return nil, fmt.Errorf("running %s: %w", binary, err)
 	}
-	apps, err := exec.CommandContext(ctx, binary, "--query-compute-apps=gpu_uuid,pid", "--format=csv,noheader").Output()
-	if err != nil {
-		return nil, fmt.Errorf("running %s: %w", binary, err)
-	}
-	return parseGPUProcesses(string(gpus), string(apps)), nil
+	return parseGPUProcesses(out)
 }
 
-// parseGPUProcesses joins nvidia-smi's GPU list (index, uuid) to its compute
-// apps (gpu_uuid, pid): the apps are listed by UUID, placement speaks index.
-func parseGPUProcesses(gpus, apps string) map[int][]int {
-	index := map[string]int{}
-	for _, line := range strings.Split(gpus, "\n") {
-		i, uuid, ok := strings.Cut(line, ",")
-		if !ok {
-			continue
+// GPUMemory is one card's memory as nvidia-smi reports it at a moment: what
+// is allocated on it, and which processes hold how much of that.
+type GPUMemory struct {
+	UsedMB int
+	Procs  []GPUProcess
+}
+
+// GPUProcess is one process holding memory on a card.
+type GPUProcess struct {
+	PID    int
+	UsedMB int
+}
+
+// GPUMemoryByProcess reports each GPU's memory, by index, with the processes
+// holding it — from the same report as GPUProcesses, so the two never
+// disagree about who is on a card.
+//
+// A node with no nvidia-smi returns nil and no error, as discovery does.
+func GPUMemoryByProcess(ctx context.Context, binary string) ([]GPUMemory, error) {
+	if binary == "" {
+		if _, err := exec.LookPath("nvidia-smi"); err != nil {
+			return nil, nil
 		}
-		n, err := strconv.Atoi(strings.TrimSpace(i))
-		if err != nil {
-			continue
-		}
-		index[strings.TrimSpace(uuid)] = n
+		binary = "nvidia-smi"
+	}
+	out, err := exec.CommandContext(ctx, binary, "-q", "-x").Output()
+	if err != nil {
+		return nil, fmt.Errorf("running %s: %w", binary, err)
+	}
+	return parseGPUMemory(out)
+}
+
+// smiLog is the part of nvidia-smi's XML report this reads: each GPU, in
+// index order, its memory and the processes on it.
+type smiLog struct {
+	GPUs []struct {
+		Used      string `xml:"fb_memory_usage>used"`
+		Processes []struct {
+			PID    string `xml:"pid"`
+			Type   string `xml:"type"`
+			UsedMB string `xml:"used_memory"`
+		} `xml:"processes>process_info"`
+	} `xml:"gpu"`
+}
+
+// parseGPUProcesses reads the report. GPUs are listed in index order, which
+// is nvidia-smi's own numbering — the one placement assigns by.
+func parseGPUProcesses(report []byte) (map[int][]int, error) {
+	mem, err := parseGPUMemory(report)
+	if err != nil {
+		return nil, err
 	}
 	out := map[int][]int{}
-	for _, line := range strings.Split(apps, "\n") {
-		uuid, p, ok := strings.Cut(line, ",")
-		if !ok {
-			continue
+	for i, g := range mem {
+		for _, p := range g.Procs {
+			out[i] = append(out[i], p.PID)
 		}
-		pid, err := strconv.Atoi(strings.TrimSpace(p))
-		i, known := index[strings.TrimSpace(uuid)]
-		if err != nil || !known {
-			continue
-		}
-		out[i] = append(out[i], pid)
 	}
-	return out
+	return out, nil
+}
+
+// parseGPUMemory reads the report into each GPU's memory and its holders.
+func parseGPUMemory(report []byte) ([]GPUMemory, error) {
+	var log smiLog
+	if err := xml.Unmarshal(report, &log); err != nil {
+		return nil, fmt.Errorf("reading nvidia-smi's report: %w", err)
+	}
+	out := make([]GPUMemory, len(log.GPUs))
+	for i, g := range log.GPUs {
+		out[i].UsedMB = mib(g.Used)
+		for _, p := range g.Processes {
+			if pid, err := strconv.Atoi(strings.TrimSpace(p.PID)); err == nil {
+				out[i].Procs = append(out[i].Procs, GPUProcess{PID: pid, UsedMB: mib(p.UsedMB)})
+			}
+		}
+	}
+	return out, nil
+}
+
+// mib reads "4932 MiB"; anything else, "N/A" included, is zero.
+func mib(s string) int {
+	n, _ := strconv.Atoi(strings.TrimSpace(strings.TrimSuffix(strings.TrimSpace(s), "MiB")))
+	return n
 }

internal/agent/discovery/gpuprocs_test.go

@@ -5,17 +5,74 @@ import (
 	"testing"
 )
 
-func TestGPUProcessesAreListedByIndexNotUUID(t *testing.T) {
-	gpus := "0, GPU-aaa\n1, GPU-bbb\n2, GPU-ccc\n"
-	// Two processes on GPU 1, one on GPU 2, and a GPU this list has never
-	// heard of, which is dropped rather than guessed at.
-	apps := "GPU-bbb, 4100\nGPU-ccc, 4200\nGPU-bbb, 4300\nGPU-zzz, 9999\n"
-	got := parseGPUProcesses(gpus, apps)
+func TestGPUProcessesOfEveryKindAreListedByIndex(t *testing.T) {
+	report := `<?xml version="1.0" ?>
+<nvidia_smi_log>
+  <driver_version>580.65.06</driver_version>
+  <gpu id="00000000:17:00.0"><product_name>Tesla V100-SXM2-32GB</product_name>
+    <processes></processes>
+  </gpu>
+  <gpu id="00000000:65:00.0"><product_name>Tesla V100-SXM2-32GB</product_name>
+    <processes>
+      <process_info><pid>4100</pid><type>C</type><process_name>llama-server</process_name></process_info>
+      <process_info><pid>4300</pid><type>G</type><process_name>llama-server</process_name></process_info>
+    </processes>
+  </gpu>
+  <gpu id="00000000:B3:00.0"><processes>
+      <process_info><pid>4200</pid><type>C+G</type></process_info>
+  </processes></gpu>
+</nvidia_smi_log>`
+	got, err := parseGPUProcesses([]byte(report))
+	if err != nil {
+		t.Fatal(err)
+	}
+	// A Vulkan engine is a graphics process ("G") and is on its GPU all the
+	// same; a compute-only query would have missed it.
 	want := map[int][]int{1: {4100, 4300}, 2: {4200}}
 	if !reflect.DeepEqual(got, want) {
 		t.Errorf("got %v, want %v", got, want)
 	}
-	if got := parseGPUProcesses(gpus, ""); len(got) != 0 {
-		t.Errorf("no compute apps should be no GPU in use: %v", got)
+	if _, err := parseGPUProcesses([]byte("not xml")); err == nil {
+		t.Error("a report that does not parse was read as no processes")
+	}
+}
+
+func TestGPUMemoryIsReadPerProcess(t *testing.T) {
+	report := `<?xml version="1.0" ?>
+<nvidia_smi_log>
+  <gpu id="00000000:01:00.0"><product_name>Quadro RTX 6000</product_name>
+    <fb_memory_usage><total>24576 MiB</total><used>14514 MiB</used><free>10062 MiB</free></fb_memory_usage>
+    <processes>
+      <process_info><pid>71315</pid><type>C</type><used_memory>4932 MiB</used_memory></process_info>
+      <process_info><pid>4100</pid><type>C</type><used_memory>9500 MiB</used_memory></process_info>
+    </processes>
+  </gpu>
+  <gpu id="00000000:02:00.0">
+    <fb_memory_usage><used>N/A</used></fb_memory_usage>
+    <processes></processes>
+  </gpu>
+</nvidia_smi_log>`
+	got, err := parseGPUMemory([]byte(report))
+	if err != nil {
+		t.Fatal(err)
+	}
+	want := []GPUMemory{
+		{UsedMB: 14514, Procs: []GPUProcess{{PID: 71315, UsedMB: 4932}, {PID: 4100, UsedMB: 9500}}},
+		{},
+	}
+	if !reflect.DeepEqual(got, want) {
+		t.Errorf("got %+v, want %+v", got, want)
+	}
+}
+
+func TestForeignMemoryBelowTheToleranceIsNoProblem(t *testing.T) {
+	// A CUDA context that has let go of its model still holds a few hundred
+	// MiB: ComfyUI after /free held 324.
+	if p := ForeignMemoryProblem(330); p != "" {
+		t.Errorf("330 MiB is a problem: %q", p)
+	}
+	p := ForeignMemoryProblem(5014)
+	if !IsForeignMemoryProblem(p) || IsForeignMemoryProblem("2 uncorrected ECC errors") {
+		t.Errorf("%q is not recognised as the foreign memory problem, or ECC is", p)
 	}
 }

internal/agent/discovery/sensors.go

@@ -41,6 +41,14 @@ type GPUSensor struct {
 	// is running free. Idle and application clock settings are left out;
 	// they are not the card being held back.
 	Throttle []string
+	// How busy it is, read with the rest every sample. Discovery reads the
+	// same once, at start, which made a busy card look as idle as it was
+	// when the agent came up. -1 is not reported by the driver; zero
+	// MemoryTotalMB is a sample without these at all.
+	UtilPercent, MemUtilPercent float64
+	MemoryUsedMB, MemoryTotalMB float64
+	SMClockMHz                  float64
+	ECCUncorrected              float64
 }
 
 // sensorEvery bounds how often nvidia-smi is run. Heartbeats come more often
@@ -187,13 +195,33 @@ var throttleBits = []struct {
 // gpus asks nvidia-smi for each card's heat and power. The memory temperature
 // is not known to every driver, and a field nvidia-smi does not know fails the
 // whole query, so it is asked for first and dropped on failure.
+// gpuFields are what each sample asks nvidia-smi for, in order. The busy
+// figures follow the throttle reasons; the memory temperature is last, being
+// the one field a driver may reject outright.
+const (
+	gpuBase  = "index,temperature.gpu,power.draw,enforced.power.limit,clocks_throttle_reasons.active"
+	gpuBusy  = ",utilization.gpu,utilization.memory,memory.used,memory.total,clocks.sm,ecc.errors.uncorrected.volatile.total"
+	gpuHBMTC = ",temperature.memory"
+)
+
 func (s *SensorSampler) gpus(ctx context.Context) []GPUSensor {
-	withMem := "index,temperature.gpu,power.draw,enforced.power.limit,clocks_throttle_reasons.active,temperature.memory"
-	out, err := s.smi(ctx, withMem)
-	if err != nil {
-		if out, err = s.smi(ctx, strings.TrimSuffix(withMem, ",temperature.memory")); err != nil {
-			return nil
+	// Most fields first, fewer on a driver that refuses one: a query naming
+	// a field the driver does not know fails as a whole.
+	var out []byte
+	var busy bool
+	for _, q := range []struct {
+		fields string
+		busy   bool
+	}{{gpuBase + gpuBusy + gpuHBMTC, true}, {gpuBase + gpuBusy, true}, {gpuBase + gpuHBMTC, false}, {gpuBase, false}} {
+		var err error
+		if out, err = s.smi(ctx, q.fields); err == nil {
+			busy = q.busy
+			break
 		}
+		out = nil
+	}
+	if out == nil {
+		return nil
 	}
 	var gs []GPUSensor
 	for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
@@ -216,14 +244,31 @@ func (s *SensorSampler) gpus(ctx context.Context) []GPUSensor {
 				}
 			}
 		}
-		if len(f) > 5 {
-			g.MemoryTempC = num(f[5])
+		rest := f[5:]
+		if busy && len(rest) >= 6 {
+			g.UtilPercent, g.MemUtilPercent = numOrUnknown(rest[0]), numOrUnknown(rest[1])
+			g.MemoryUsedMB, g.MemoryTotalMB = num(rest[2]), num(rest[3])
+			g.SMClockMHz, g.ECCUncorrected = num(rest[4]), numOrUnknown(rest[5])
+			rest = rest[6:]
+		}
+		if len(rest) > 0 {
+			g.MemoryTempC = num(rest[0])
 		}
 		gs = append(gs, g)
 	}
 	return gs
 }
 
+// numOrUnknown is a figure, or -1 where nvidia-smi says it has none: a card
+// without ECC has no error count, which is not the same as none.
+func numOrUnknown(s string) float64 {
+	v, err := strconv.ParseFloat(s, 64)
+	if err != nil || v < 0 {
+		return -1
+	}
+	return v
+}
+
 // num reads a figure nvidia-smi may give as "[N/A]" or "N/A".
 func num(s string) float64 {
 	v, err := strconv.ParseFloat(s, 64)

internal/agent/discovery/sensors_test.go

@@ -167,3 +167,44 @@ func TestADriverThatDoesNotKnowMemoryTemperatureStillReportsTheRest(t *testing.T
 		t.Errorf("%+v", got.GPUs)
 	}
 }
+
+func TestEachSampleSaysHowBusyEachGPUIs(t *testing.T) {
+	// What the card is doing now, every sample: discovery read it once, at
+	// start, and a card busy since looked as idle as it was then.
+	smi := func(_ context.Context, fields string) ([]byte, error) {
+		if !strings.Contains(fields, "utilization.gpu") {
+			return nil, errors.New("asked without how busy the cards are")
+		}
+		return []byte(
+			"0, 71, 612.45, 700.00, 0x0000000000000000, 97, 64, 71234, 81559, 1980, 0, 84\n" +
+				"1, 30, 70.10, 700.00, 0x0000000000000001, 0, 0, 512, 81559, 345, [N/A], [N/A]\n"), nil
+	}
+	now := time.Now()
+	got := sampler(t.TempDir(), smi, &now).Sample(t.Context()).GPUs
+	if len(got) != 2 {
+		t.Fatalf("%+v", got)
+	}
+	g0, g1 := got[0], got[1]
+	if g0.UtilPercent != 97 || g0.MemUtilPercent != 64 || g0.MemoryUsedMB != 71234 || g0.MemoryTotalMB != 81559 ||
+		g0.SMClockMHz != 1980 || g0.ECCUncorrected != 0 || g0.MemoryTempC != 84 {
+		t.Errorf("card 0 %+v", g0)
+	}
+	// Idle is zero; a count the card does not keep is unknown, not zero.
+	if g1.UtilPercent != 0 || g1.ECCUncorrected != -1 || g1.MemoryTempC != 0 {
+		t.Errorf("card 1 %+v", g1)
+	}
+}
+
+func TestADriverThatDoesNotKnowTheBusyFieldsStillReportsHeatAndPower(t *testing.T) {
+	smi := func(_ context.Context, fields string) ([]byte, error) {
+		if strings.Contains(fields, "utilization.gpu") {
+			return nil, errors.New(`Field "ecc.errors.uncorrected.volatile.total" is not a valid field to query.`)
+		}
+		return []byte("0, 66, 250.0, 300.0, 0x0000000000000000, 70\n"), nil
+	}
+	now := time.Now()
+	got := sampler(t.TempDir(), smi, &now).Sample(t.Context()).GPUs
+	if len(got) != 1 || got[0].TempC != 66 || got[0].MemoryTempC != 70 || got[0].MemoryTotalMB != 0 {
+		t.Errorf("%+v", got)
+	}
+}

internal/agent/engines/llamacpp.go

@@ -168,6 +168,14 @@ func (l *LlamaCpp) Commands(s ShardSpec) ([]Command, error) {
 	if s.APIKey != "" {
 		env["LLAMA_API_KEY"] = s.APIKey
 	}
+	// The Vulkan build — what a GPU older than the CUDA builds gets — does
+	// not read CUDA_VISIBLE_DEVICES; this is its own. Vulkan numbers devices
+	// as the driver lists them, which on NVIDIA follows the same bus order
+	// as nvidia-smi; were it not to, the shard would not hold memory on its
+	// own GPUs, and starting it checks exactly that.
+	if len(s.GPUIndices) > 0 {
+		env["GGML_VK_VISIBLE_DEVICES"] = joinInts(s.GPUIndices)
+	}
 	return []Command{{
 		Name: "llama-server", Path: bin, Args: args, Env: materializeEnv(env),
 	}}, nil

internal/agent/gpucheck.go

@@ -16,6 +16,7 @@ import (
 // nvidia-smi and /proc.
 var (
 	gpuProcesses = discovery.GPUProcesses
+	gpuMemory    = discovery.GPUMemoryByProcess
 	processGroup = pgidOf
 	// gpuCheckWait is how long a shard has, once it answers, to show up on
 	// its GPUs: the weights are on them by then, but nvidia-smi can lag.
@@ -75,6 +76,44 @@ func (a *Agent) onItsGPUs(ctx context.Context, eng engines.Engine, spec engines.
 		"or run `llmgw doctor` on this node", joinIndices(missing), seen)
 }
 
+// foreignGPUMemory is what each GPU holds for anything other than this
+// agent's shards, by index, and each GPU's memory in use. Nil when nvidia-smi
+// cannot be asked, which is no reading rather than an empty card.
+//
+// Everything allocated less what the shards' processes hold, rather than the
+// sum of the other processes: a process in another container's PID namespace
+// holds memory nvidia-smi lists under no PID this node can see, and it is
+// still not memory placement can hand out.
+func (a *Agent) foreignGPUMemory(ctx context.Context, shards []int) (foreign, used []int32) {
+	mem, err := gpuMemory(ctx, a.cfg.Discovery.NvidiaSMI)
+	if err != nil || mem == nil {
+		return nil, nil
+	}
+	groups := make([]int, len(shards))
+	for i, pid := range shards {
+		groups[i] = processGroup(pid)
+	}
+	ours := func(p int) bool {
+		for i, pid := range shards {
+			if holds([]int{p}, pid, groups[i]) {
+				return true
+			}
+		}
+		return false
+	}
+	for _, g := range mem {
+		f := g.UsedMB
+		for _, p := range g.Procs {
+			if ours(p.PID) {
+				f -= p.UsedMB
+			}
+		}
+		foreign = append(foreign, int32(max(f, 0)))
+		used = append(used, int32(g.UsedMB))
+	}
+	return foreign, used
+}
+
 // holds reports whether any of pids is the shard's process or in its group:
 // an engine that computes in a child process is still the shard.
 func holds(pids []int, pid, group int) bool {

internal/agent/gpucheck_test.go

@@ -3,10 +3,12 @@ package agent
 import (
 	"context"
 	"log/slog"
+	"slices"
 	"strings"
 	"testing"
 	"time"
 
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
 )
 
@@ -89,3 +91,34 @@ func TestTheCheckWaitsForTheShardToShowUpOnItsGPU(t *testing.T) {
 		t.Fatalf("nvidia-smi lagging by a second failed the shard: %v", err)
 	}
 }
+
+func TestForeignGPUMemoryLeavesOutTheAgentsOwnShards(t *testing.T) {
+	// cuda01: ComfyUI and one of our engines on the same card. Only ComfyUI's
+	// share — and whatever no listed process accounts for — is foreign.
+	oldMem, oldGroup := gpuMemory, processGroup
+	gpuMemory = func(context.Context, string) ([]discovery.GPUMemory, error) {
+		return []discovery.GPUMemory{
+			{UsedMB: 15000, Procs: []discovery.GPUProcess{{PID: 71315, UsedMB: 4932}, {PID: 512, UsedMB: 9500}}},
+			{UsedMB: 300},
+		}, nil
+	}
+	processGroup = func(pid int) int {
+		if pid >= 500 && pid < 600 {
+			return 500
+		}
+		return pid
+	}
+	t.Cleanup(func() { gpuMemory, processGroup = oldMem, oldGroup })
+
+	a := &Agent{log: slog.Default()}
+	foreign, used := a.foreignGPUMemory(context.Background(), []int{500})
+	if !slices.Equal(foreign, []int32{5500, 300}) || !slices.Equal(used, []int32{15000, 300}) {
+		t.Errorf("foreign %v, used %v; want [5500 300] and [15000 300]", foreign, used)
+	}
+
+	// No nvidia-smi to ask is no reading, not an empty card.
+	gpuMemory = func(context.Context, string) ([]discovery.GPUMemory, error) { return nil, nil }
+	if foreign, _ := a.foreignGPUMemory(context.Background(), nil); foreign != nil {
+		t.Errorf("no reading came back as %v", foreign)
+	}
+}

internal/agent/provision/install.go

@@ -143,7 +143,7 @@ func (in *Installer) one(ctx context.Context, w Want) Status {
 		st.Detail = fmt.Sprintf("%q is not a build this agent will install", w.Variant)
 		return st
 	}
-	build, err := recipe.Choose(w.Version, w.Variant, in.Host)
+	builds, err := recipe.Candidates(w.Version, w.Variant, in.Host)
 	if err != nil {
 		st.State, st.Detail = Unsupported, err.Error()
 		var nh NotHere
@@ -152,14 +152,18 @@ func (in *Installer) one(ctx context.Context, w Want) Status {
 		}
 		return st
 	}
-	st.Variant = build.Variant
-	dir := filepath.Join(in.root(), w.Engine, w.Version+"-"+build.Variant)
 
-	// Installed before: check it still runs, and trust nothing else about it.
-	if _, err := os.Stat(filepath.Join(dir, marker)); err == nil {
+	// Installed before: whichever candidate is, best first, checked that it
+	// still runs and trusted for nothing else.
+	for _, build := range builds {
+		dir := in.dirFor(w, build)
+		if _, err := os.Stat(filepath.Join(dir, marker)); err != nil {
+			continue
+		}
+		st := st.as(w, build)
 		// The packages are checked too. Removing one is exactly the kind of
-		// change an unrelated upgrade makes, and ldd below would catch it only
-		// as a missing library with no hint of which package held it.
+		// change an unrelated upgrade makes, and ldd below would catch it
+		// only as a missing library with no hint of which package held it.
 		if err := in.packages(ctx, build.Packages, &st); err != nil {
 			st.State, st.Detail = Failed, err.Error()
 			return st
@@ -169,24 +173,64 @@ func (in *Installer) one(ctx context.Context, w Want) Status {
 			return in.ready(st, build, bin, version)
 		}
 		in.log().Warn("an installed engine no longer runs; installing it again",
-			"engine", w.Engine, "version", w.Version, "err", err)
+			"engine", w.Engine, "version", st.Version, "variant", build.Variant, "err", err)
+		break
 	}
 
-	if err := in.install(ctx, recipe, w, build, dir, &st); err != nil {
-		st.State, st.Detail, st.Binary = Failed, err.Error(), ""
-		return st
+	// Each candidate in turn until one installs and is seen to work: a build
+	// the table was wrong about — a driver too old for it after all, a GPU
+	// it has no kernels for — costs its download, and the next is tried.
+	var tried []string
+	for _, build := range builds {
+		st := st.as(w, build)
+		if err := in.install(ctx, recipe, w, build, in.dirFor(w, build), &st); err != nil {
+			tried = append(tried, build.Variant+": "+err.Error())
+			in.log().Warn("an engine build did not work here; trying the next",
+				"engine", w.Engine, "variant", build.Variant, "err", err)
+			continue
+		}
+		bin, version, err := in.verify(ctx, in.dirFor(w, build), build)
+		if err != nil {
+			tried = append(tried, build.Variant+": "+err.Error())
+			continue
+		}
+		return in.ready(st, build, bin, version)
 	}
-	bin, version, err := in.verify(ctx, dir, build)
-	if err != nil {
-		st.State, st.Detail = Failed, err.Error()
-		return st
+	st.State, st.Binary = Failed, ""
+	st.Detail = strings.Join(tried, "; then ")
+	if len(builds) == 1 {
+		_, st.Detail, _ = strings.Cut(st.Detail, ": ")
 	}
-	return in.ready(st, build, bin, version)
+	return st
+}
+
+// dirFor is where a build of an engine is installed: by the version it is
+// and its variant, so two builds of one version sit side by side.
+func (in *Installer) dirFor(w Want, b Build) string {
+	return filepath.Join(in.root(), w.Engine, b.versionOr(w.Version)+"-"+b.Variant)
+}
+
+// versionOr is the version a build is: its own, when it is not the one asked
+// for.
+func (b Build) versionOr(asked string) string {
+	if b.Version != "" {
+		return b.Version
+	}
+	return asked
+}
+
+// as is a status for one build of what was asked for.
+func (st Status) as(w Want, b Build) Status {
+	st.Variant, st.Version = b.Variant, b.versionOr(w.Version)
+	return st
 }
 
 func (in *Installer) ready(st Status, b Build, bin, version string) Status {
 	st.State, st.Binary = Ready, bin
 	st.Detail = strings.TrimSpace(filepath.Base(bin) + " " + version)
+	if b.Note != "" {
+		st.Detail += "; " + b.Note
+	}
 	var missing []string
 	for _, f := range b.CPUFlags {
 		if !in.Host.HasCPUFlag(f) {

internal/agent/provision/install_test.go

@@ -255,13 +255,38 @@ func TestAnArchiveCannotWriteOutsideItsDirectory(t *testing.T) {
 func TestTheBuildFollowsTheHardware(t *testing.T) {
 	gpu := ubuntu
 	gpu.NVIDIADriver = "570.86.15"
-	b, err := chooseLlamaCpp("b1", "", gpu)
-	if err != nil || !strings.HasPrefix(b.Variant, "cuda") || len(b.Assets) != 2 {
-		t.Errorf("an NVIDIA node got %+v, %v; expected the CUDA build and its runtime", b, err)
+	got, err := llamaCppCandidates("b1", "", gpu)
+	if err != nil || got[0].Variant != "cuda-12.8" || len(got[0].Assets) != 2 || got[len(got)-1].Variant != "vulkan" {
+		t.Errorf("an NVIDIA node on driver 570 got %+v, %v; expected CUDA 12.8 with its runtime, then Vulkan", got, err)
 	}
-	if _, err := chooseLlamaCpp("b1", "", discovery.Host{OSID: "ubuntu", Arch: "riscv64"}); err == nil {
+	if _, err := llamaCppCandidates("b1", "", discovery.Host{OSID: "ubuntu", Arch: "riscv64"}); err == nil {
 		t.Error("a build was chosen for an architecture llama.cpp does not publish for")
 	}
+	for _, c := range []struct {
+		name string
+		host discovery.Host
+		want []string
+	}{
+		// Driver 580 and Turing: CUDA 13.3 first.
+		{"current", discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 8.0}, []string{"cuda-13.3", "cuda-12.8", "vulkan"}},
+		// A V100 on driver 550: CUDA 12.8 ships 7.0 as PTX, which needs 570.
+		{"V100, 550", discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0}, []string{"vulkan"}},
+		// An RTX 3090 on 550: 8.6 is machine code in the CUDA 12.8 build.
+		{"3090, 550", discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 8.6}, []string{"cuda-12.8", "vulkan"}},
+		// A Pascal card: CUDA 12.8, not 13.3.
+		{"P40", discovery.Host{Arch: "amd64", NVIDIADriver: "575.51.03", MinComputeCapability: 6.1}, []string{"cuda-12.8", "vulkan"}},
+		// arm64 with a GPU: never the CPU build.
+		{"GH200", discovery.Host{Arch: "arm64", NVIDIADriver: "580.65.06", MinComputeCapability: 9.0}, []string{"cuda-13.3", "vulkan"}},
+	} {
+		got, err := llamaCppCandidates("b1", "", c.host)
+		var names []string
+		for _, b := range got {
+			names = append(names, b.Variant)
+		}
+		if err != nil || strings.Join(names, " ") != strings.Join(c.want, " ") {
+			t.Errorf("%s: %v, %v; want %v", c.name, names, err, c.want)
+		}
+	}
 
 	slow := ubuntu
 	slow.CPUFlags = nil
@@ -400,7 +425,7 @@ func TestALibrarysVersionLinkIsStillUnpacked(t *testing.T) {
 func TestACUDABuildThatSeesNoGPUIsNotInstalled(t *testing.T) {
 	// Found in the field: libggml-cuda.so in the build, cuBLAS nowhere the
 	// loader looks, and llama-server serving a GPU placement on the CPU.
-	cuda := "bin-ubuntu-cuda-" + cudaForLlamaCpp + "-x64.tar.gz"
+	cuda := "bin-ubuntu-cuda-12.8-x64.tar.gz"
 	rel := newRelease(map[string][]byte{
 		"llama-b1-" + cuda: tarGz(t, map[string]string{
 			"llama-b1/llama-server": "#!/bin/sh\n", "llama-b1/libggml-cuda.so": "x"}, nil),

internal/agent/provision/python.go

@@ -36,6 +36,10 @@ const lockName = "llmgw-requirements.txt"
 func (in *Installer) installPython(ctx context.Context, r Recipe, w Want, b Build, dir string, st *Status) error {
 	ctx, cancel := context.WithTimeout(ctx, pythonInstallTimeout)
 	defer cancel()
+	// The version this build is, which for a GPU older than every build of
+	// the one asked for is an older one: its publication date is the one
+	// the dependencies are resolved as of.
+	w.Version = b.versionOr(w.Version)
 
 	if err := in.packages(ctx, b.Packages, st); err != nil {
 		return err

internal/agent/provision/python_test.go

@@ -25,8 +25,9 @@ type fakePython struct {
 	commands []string
 	failPip  bool
 	// noKernel makes the GPU check fail as a torch without kernels for the
-	// card does.
-	noKernel bool
+	// card does: in every build, or in noKernelIn only ("-cu13.").
+	noKernel   bool
+	noKernelIn string
 }
 
 func (f *fakePython) run(ctx context.Context, env []string, name string, args ...string) ([]byte, error) {
@@ -48,7 +49,7 @@ func (f *fakePython) run(ctx context.Context, env []string, name string, args ..
 	case strings.HasSuffix(name, "/bin/vllm") && args[0] == "--version":
 		return []byte("0.30.0\n"), nil
 	case strings.HasSuffix(name, "/bin/python") && args[0] == "-c":
-		if f.noKernel {
+		if f.noKernel || (f.noKernelIn != "" && strings.Contains(name, f.noKernelIn)) {
 			return []byte("RuntimeError: CUDA error: no kernel image is available for execution on the device\n"), errExit
 		}
 		return []byte("128.0 Tesla V100-SXM2-32GB (7, 0)\n"), nil
@@ -77,6 +78,9 @@ func pythonInstaller(t *testing.T, host discovery.Host, sys *fakePython) (*Insta
 		_, _ = w.Write([]byte(`{"urls":[{"upload_time_iso_8601":"2026-09-22T05:22:21.647189Z"},
 			{"upload_time_iso_8601":"2026-09-22T05:23:10.039243Z"}]}`))
 	})
+	mux.HandleFunc("/pypi/vllm/0.19.1/json", func(w http.ResponseWriter, _ *http.Request) {
+		_, _ = w.Write([]byte(`{"urls":[{"upload_time_iso_8601":"2026-04-02T10:00:00Z"}]}`))
+	})
 	pypi := httptest.NewServer(mux)
 	t.Cleanup(pypi.Close)
 	return &Installer{
@@ -143,10 +147,8 @@ func TestVLLMIsNotForEveryNode(t *testing.T) {
 		{"no GPU", ubuntu, Skipped, "has none"},
 		// Too old for CUDA 13 and for its CUDA 12.9 build both.
 		{"a driver too old for either", discovery.Host{Arch: "amd64", NVIDIADriver: "470.256.02"}, Unsupported, "driver 525"},
-		{"Pascal", discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 6.1}, Unsupported, "7.5 and newer"},
-		// vLLM 0.20 and later have no Volta kernels in any build; the
-		// refusal names the last version that has.
-		{"V100", discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0}, Unsupported, "the last vLLM built for it is 0.19.1"},
+		// No vLLM was ever built for Pascal.
+		{"P100", discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 6.0}, Unsupported, "no vLLM is built for GPUs of compute capability 6.0"},
 	}
 	for _, c := range cases {
 		t.Run(c.name, func(t *testing.T) {
@@ -240,11 +242,7 @@ func TestABuildChosenForANodeIsTheOneInstalled(t *testing.T) {
 	if got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0", Variant: "cu129"}}); got[0].Variant != "cu129" || got[0].State != Ready {
 		t.Errorf("cu129 chosen for a node that runs cu13 too: %+v", got[0])
 	}
-	volta := discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 7.0}
-	if _, err := chooseVLLM("0.30.0", "cu13", volta); err == nil || !strings.Contains(err.Error(), "7.5") {
-		t.Errorf("cu13 chosen for a V100 should be refused, saying why: %v", err)
-	}
-	if _, err := chooseVLLM("0.30.0", "cu200", gpuNode); err == nil {
+	if _, err := vllmCandidates("0.30.0", "cu200", gpuNode); err == nil {
 		t.Error("a build that does not exist was accepted")
 	}
 }
@@ -264,25 +262,51 @@ func TestABuildThatCannotComputeOnTheCardIsNotInstalled(t *testing.T) {
 func TestOnlyACUDARequirementFallsBackToTheCUDA12Build(t *testing.T) {
 	// A machine vLLM is not published for is refused, not given the other
 	// build to fail with.
-	if _, err := chooseVLLM("0.30.0", "", discovery.Host{NVIDIADriver: "580.65.06"}); err == nil ||
+	if _, err := vllmCandidates("0.30.0", "", discovery.Host{NVIDIADriver: "580.65.06"}); err == nil ||
 		!strings.Contains(err.Error(), "no Linux build") {
 		t.Errorf("an unknown architecture: %v", err)
 	}
-	if _, err := chooseVLLM("0.30.0", "cu129", ubuntu); !errors.As(err, new(NotHere)) {
+	if _, err := vllmCandidates("0.30.0", "cu129", ubuntu); !errors.As(err, new(NotHere)) {
 		t.Errorf("a node with no GPU chosen cu129: %v", err)
 	}
+	// Driver 580 and Turing: CUDA 13 first, the CUDA 12.9 build behind it.
+	got, err := vllmCandidates("0.30.0", "", gpuNode)
+	if err != nil || len(got) != 2 || got[0].Variant != "cu13" || got[1].Variant != "cu129" {
+		t.Errorf("candidates on a current node: %+v, %v", got, err)
+	}
 }
 
-func TestAV100GetsTheLastVLLMBuiltForIt(t *testing.T) {
-	v100 := discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0}
-	b, err := chooseVLLM(voltaUntil, "", v100)
-	if err != nil || b.Variant != "cu12" || b.Wheel != nil {
-		t.Fatalf("vLLM %s on a V100: %+v, %v; want PyPI's CUDA 12 build", voltaUntil, b, err)
+func TestAV100GetsTheNewestVLLMBuiltForItByItself(t *testing.T) {
+	// 0.30.0 asked for; no build of it has Volta kernels. The node installs
+	// 0.19.1, the newest that has, and says so.
+	v100 := discovery.Host{OSID: "ubuntu", Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0}
+	sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
+	in, _ := pythonInstaller(t, v100, sys)
+	got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})[0]
+	if got.State != Ready || got.Version != "0.19.1" || got.Variant != "cu12" ||
+		!strings.Contains(got.Detail, "the newest built for compute capability 7.0, instead of 0.30.0") {
+		t.Fatalf("a V100: %+v", got)
 	}
-	if _, err := chooseVLLM("0.20.0", "", v100); err == nil {
-		t.Error("vLLM 0.20, with no Volta kernels in any build, was offered to a V100")
+	if _, err := os.Stat(filepath.Join(in.Root, "vllm", "0.19.1-cu12", marker)); err != nil {
+		t.Errorf("not installed as what it is: %v", err)
 	}
-	if _, err := chooseVLLM("0.30.0", "cu129", v100); err == nil || !strings.Contains(err.Error(), "7.5") {
-		t.Errorf("cu129 chosen for a V100 should be refused: %v", err)
+	all := strings.Join(sys.commands, "\n")
+	if !strings.Contains(all, "<<< vllm==0.19.1") || strings.Contains(all, "download.pytorch.org") {
+		t.Errorf("0.19.1 is PyPI's CUDA 12 build, with PyPI's torch:\n%s", all)
+	}
+	// Chosen cu13, which 0.19.1 has none of: refused, saying why.
+	if _, err := vllmCandidates("0.30.0", "cu13", v100); err == nil {
+		t.Error("a CUDA 13 build was offered to a V100")
+	}
+}
+
+func TestABuildThatFailsOnTheCardGivesWayToTheNext(t *testing.T) {
+	// The table says CUDA 13 runs here; the card says otherwise. The next
+	// build is installed rather than the node going without.
+	sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}, noKernelIn: "-cu13."}
+	in, _ := pythonInstaller(t, gpuNode, sys)
+	got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})[0]
+	if got.State != Ready || got.Variant != "cu129" {
+		t.Fatalf("after the CUDA 13 build failed on the card: %+v", got)
 	}
 }

internal/agent/provision/recipe.go

@@ -10,7 +10,6 @@
 package provision
 
 import (
-	"errors"
 	"fmt"
 	"slices"
 	"strings"
@@ -60,6 +59,13 @@ type Build struct {
 	// with no kernels for the card — a GPU generation its torch dropped —
 	// imports and prints its version, and fails on its first model.
 	GPUCheck bool
+	// Version is the engine version this build is, when it is not the one
+	// asked for: the newest built for this machine's GPUs, where the one
+	// asked for has no kernels for them. Empty is the version asked for.
+	Version string
+	// Note says why this build, when it is not the plain one: shown beside
+	// it wherever the installation is.
+	Note string
 	// FreeGB is the disk space the installation needs before it starts: a
 	// Python engine is several gigabytes, and one that fills the disk halfway
 	// leaves the node worse off than one that never began.
@@ -79,10 +85,12 @@ type Recipe struct {
 	Repo string
 	// PyPI is the project an engine installed from PyPI is published as.
 	PyPI string
-	// Choose picks the build for a machine, or says why none fits. variant
-	// is a build chosen for the node ("cu129"); empty leaves it to the
-	// machine.
-	Choose func(version, variant string, h discovery.Host) (Build, error)
+	// Candidates are the builds that may run on a machine, best first, or
+	// why none can. The installer takes the first that installs and is seen
+	// to work — computing on the GPU, where there is one — so a build this
+	// table is wrong about costs a download, not a node. variant is a build
+	// chosen for the node; empty leaves the choice to the machine.
+	Candidates func(version, variant string, h discovery.Host) ([]Build, error)
 }
 
 // Recipes are the engines this agent can install.
@@ -91,10 +99,10 @@ type Recipe struct {
 // from PyPI by uv, which is itself installed like llama.cpp: a GitHub release
 // checked against its published checksum.
 var Recipes = map[string]Recipe{
-	"llamacpp": {Repo: "ggml-org/llama.cpp", Choose: chooseLlamaCpp},
-	"vllm":     {PyPI: "vllm", Choose: chooseVLLM},
-	"sglang":   {PyPI: "sglang", Choose: chooseSGLang},
-	"uv":       {Repo: "astral-sh/uv", Choose: chooseUV},
+	"llamacpp": {Repo: "ggml-org/llama.cpp", Candidates: llamaCppCandidates},
+	"vllm":     {PyPI: "vllm", Candidates: vllmCandidates},
+	"sglang":   {PyPI: "sglang", Candidates: sglangCandidates},
+	"uv":       {Repo: "astral-sh/uv", Candidates: uvCandidates},
 }
 
 // Installable are the engines this agent can install, as it tells the
@@ -124,7 +132,15 @@ type NotHere struct{ Reason string }
 
 func (e NotHere) Error() string { return e.Reason }
 
-func chooseUV(version, _ string, h discovery.Host) (Build, error) {
+func uvCandidates(version, _ string, h discovery.Host) ([]Build, error) {
+	b, err := chooseUV(version, h)
+	if err != nil {
+		return nil, err
+	}
+	return []Build{b}, nil
+}
+
+func chooseUV(_ string, h discovery.Host) (Build, error) {
 	var arch string
 	switch h.Arch {
 	case "amd64":
@@ -141,58 +157,190 @@ func chooseUV(version, _ string, h discovery.Host) (Build, error) {
 	}, nil
 }
 
-// minDriverForCUDA13 is the oldest NVIDIA driver CUDA 13 runs on. vLLM's
-// PyPI build is CUDA 13 from 0.20 (before it, CUDA 12, with CUDA 13 as the
-// "+cu130" wheel on GitHub), and SGLang's from 0.5.
-const minDriverForCUDA13 = 580
-
-// minComputeForCUDA13 is the oldest GPU CUDA 13 supports: Turing. It dropped
-// Volta, so a V100 cannot run these builds at all.
-const minComputeForCUDA13 = 7.5
-
-// The CUDA 12 builds run on any driver from 525, by CUDA's minor version
-// compatibility; the oldest GPU one has kernels for is minComputeFor's
-// answer, which is Volta only up to voltaUntil.
+// What builds need of the NVIDIA side of a machine, read from what each
+// project publishes and how it was built:
+//
+//   - CUDA 13 needs driver 580; CUDA 12 builds run on driver 525 and newer,
+//     by CUDA's minor version compatibility — except code shipped only as
+//     PTX, which the driver compiles and so needs a driver as new as the
+//     CUDA it was compiled with.
+//   - Which GPUs a build has kernels for is the build's own list, not what
+//     its CUDA supports.
 const (
-	minDriverForCUDA12  = 525
-	minComputeForCUDA12 = 7.0
+	minDriverForCUDA13 = 580
+	minDriverForCUDA12 = 525
 )
 
-// cuda13Since is the first version of each Python engine whose PyPI build is
-// CUDA 13, read from vLLM's releases: 0.19.1 publishes "+cu130" beside a
-// CUDA 12 default, 0.20.0 "+cu129" beside a CUDA 13 one.
-var cuda13Since = map[string]string{"vLLM": "0.20", "SGLang": "0.5"}
-
-// voltaUntil is the last vLLM built for Volta (7.0). Its release pipeline
-// built with the Dockerfile's default arch list, "7.0 7.5 8.0 8.9 9.0 10.0
-// 12.0"; from 0.20.0 every x86 build starts at 7.5, CUDA 12.9's included.
-const voltaUntil = "0.19.1"
-
-// minComputeFor is the oldest GPU a vLLM version's builds have kernels for
-// on an architecture: what the project compiled, not what CUDA supports.
-func minComputeFor(version, arch string) float64 {
-	if arch == "arm64" {
-		// Built for the Grace machines: 8.7 and up before 0.20, 8.0 and up
-		// since.
-		if versionBefore(version, "0.20") {
-			return 8.7
-		}
-		return 8.0
-	}
-	if versionBefore(version, "0.20") {
-		return minComputeForCUDA12
-	}
-	return minComputeForCUDA13
+// gpuNeeds is what one build needs of a machine's GPUs and driver.
+type gpuNeeds struct {
+	driver int
+	cc     float64
 }
 
-// cudaGate says whether a node can run this version of an engine, and which
-// CUDA it is built for.
+// fits says why a machine cannot run a build, or nil.
+func (n gpuNeeds) fits(what string, h discovery.Host) error {
+	if major := leadingInt(h.NVIDIADriver); n.driver > 0 && major > 0 && major < n.driver {
+		return fmt.Errorf("%s needs NVIDIA driver %d or newer; this node has %s", what, n.driver, h.NVIDIADriver)
+	}
+	if cc := h.MinComputeCapability; n.cc > 0 && cc > 0 && cc < n.cc {
+		return fmt.Errorf("%s has kernels for compute capability %.1f and newer; this node has %.1f", what, n.cc, cc)
+	}
+	return nil
+}
+
+// candidate is a build and what it needs.
+type candidate struct {
+	b     Build
+	needs gpuNeeds
+	what  string
+}
+
+// pick keeps the candidates a machine can run, in order, and those a
+// variant chosen for the node names; with none left, it says why each went.
+func pick(engine, variant string, cands []candidate, h discovery.Host) ([]Build, error) {
+	var out []Build
+	var why []string
+	for _, c := range cands {
+		if variant != "" && c.b.Variant != variant {
+			continue
+		}
+		if err := c.needs.fits(c.what, h); err != nil {
+			why = append(why, err.Error())
+			continue
+		}
+		out = append(out, c.b)
+	}
+	switch {
+	case len(out) > 0:
+		return out, nil
+	case len(why) == 0 && variant != "":
+		var names []string
+		for _, c := range cands {
+			names = append(names, c.b.Variant)
+		}
+		return nil, fmt.Errorf("%s has no build %q here; it has %s", engine, variant, strings.Join(names, ", "))
+	}
+	return nil, fmt.Errorf("no build of %s runs on this node: %s", engine, strings.Join(why, "; "))
+}
+
+// --- vLLM ---
+
+// VLLMVariants are the builds of vLLM a node can be given by choice: CUDA
+// 13 from PyPI, and CUDA 12.9 from the project's GitHub release — the same
+// version, built for drivers CUDA 13 does not run on.
+var VLLMVariants = []string{"cu13", "cu129"}
+
+// pytorchCU129 is where the CUDA 12.9 build's torch is published. PyPI's
+// torch is built for CUDA 13; the cu129 vLLM wheel needs this one.
+const pytorchCU129 = "https://download.pytorch.org/whl/cu129"
+
+// vllmCUDA13Since is the first vLLM whose PyPI build is CUDA 13, read from
+// its releases: 0.19.1 publishes "+cu130" beside a CUDA 12 default, 0.20.0
+// "+cu129" beside a CUDA 13 one.
+const vllmCUDA13Since = "0.20"
+
+// vllmMinCompute is the oldest GPU a vLLM version's builds have kernels for,
+// from its release pipeline: on x86, 7.0 (Volta) up to 0.19.1 — the
+// Dockerfile's default "7.0 7.5 8.0 8.9 9.0 10.0 12.0" — and 7.5 from 0.20.0,
+// CUDA 12.9 build included; on arm64, built for Grace machines, 8.7 before
+// 0.20 and 8.0 since.
+func vllmMinCompute(version, arch string) float64 {
+	old := versionBefore(version, vllmCUDA13Since)
+	switch {
+	case arch == "arm64" && old:
+		return 8.7
+	case arch == "arm64":
+		return 8.0
+	case old:
+		return 7.0
+	}
+	return 7.5
+}
+
+// vllmNewestFor is the newest vLLM built for GPUs of compute capability cc
+// on an architecture, for GPUs older than every build of the version asked
+// for: a V100 gets 0.19.1, where 0.20 and later have nothing for it. Nothing
+// for a GPU no vLLM was built for.
+var vllmNewestFor = []struct {
+	arch    string
+	from    float64 // the oldest GPU it covers
+	version string
+}{
+	{"amd64", 7.0, "0.19.1"},
+}
+
+// vllmCandidates are the vLLM builds for a machine.
 //
-// By version, because the line moved: a V100 cannot run vLLM 0.20 or later,
-// and runs 0.19.1 — so a cluster with Volta and newer cards gives the Volta
-// nodes the older version (engine_overrides in the server's configuration)
-// rather than going without vLLM, or without the Volta nodes.
-func cudaGate(engine, version string, h discovery.Host) (variant string, err error) {
+// The version asked for, in the build the machine runs: CUDA 13 where the
+// driver allows, the same version's CUDA 12.9 build where it does not. One
+// version across the cluster wherever the GPUs allow it, because the flags
+// llmgw builds are checked against it. Where they do not — GPUs older than
+// every build of that version — the newest version built for them, said so
+// beside the installation: vLLM on a V100 at all, rather than not.
+func vllmCandidates(version, variant string, h discovery.Host) ([]Build, error) {
+	arch, err := gpuArch("vLLM", h)
+	if err != nil {
+		return nil, err
+	}
+	v, note := version, ""
+	if cc := h.MinComputeCapability; cc > 0 && cc < vllmMinCompute(version, h.Arch) {
+		older := ""
+		for _, n := range vllmNewestFor {
+			if n.arch == h.Arch && cc >= n.from {
+				older = n.version
+			}
+		}
+		if older == "" {
+			return nil, fmt.Errorf("no vLLM is built for GPUs of compute capability %.1f on %s; "+
+				"the node serves llama.cpp models", cc, arch)
+		}
+		v, note = older, fmt.Sprintf("vLLM %s, the newest built for compute capability %.1f, instead of %s", older, cc, version)
+	}
+	pypi := func(variant string) Build {
+		b := Build{
+			Variant:      variant,
+			Python:       pythonForEngines,
+			Requirements: []string{"vllm==" + v},
+			Binary:       "vllm",
+			Packages:     compilerPackages,
+			FreeGB:       25,
+			GPUCheck:     true,
+			Note:         note,
+		}
+		if v != version {
+			b.Version = v
+		}
+		return b
+	}
+	minCC := vllmMinCompute(v, h.Arch)
+	if versionBefore(v, vllmCUDA13Since) {
+		// PyPI's own build is CUDA 12, and is the one there is; asked for
+		// cu129, this is it.
+		if variant == "cu129" {
+			variant = "cu12"
+		}
+		return pick("vLLM "+v, variant, []candidate{
+			{pypi("cu12"), gpuNeeds{minDriverForCUDA12, minCC}, "vLLM " + v},
+		}, h)
+	}
+	cu129 := pypi("cu129")
+	cu129.Requirements = nil
+	// Found in the release's assets rather than named: the manylinux tag
+	// in the file's name changes between versions (2_28, 2_31, 2_34).
+	cu129.Wheel = &Wheel{
+		Project: "vllm", Repo: "vllm-project/vllm", Tag: "v" + v,
+		Prefix: "vllm-" + v + "+cu129-", Suffix: "_" + map[string]string{"amd64": "x86_64", "arm64": "aarch64"}[h.Arch] + ".whl",
+	}
+	cu129.ExtraIndex = pytorchCU129
+	return pick("vLLM "+v, variant, []candidate{
+		{pypi("cu13"), gpuNeeds{minDriverForCUDA13, minCC}, "vLLM " + v + " built for CUDA 13"},
+		{cu129, gpuNeeds{minDriverForCUDA12, minCC}, "vLLM " + v + " built for CUDA 12.9"},
+	}, h)
+}
+
+// gpuArch says whether a Python GPU engine is published for this machine
+// and whether it has an NVIDIA GPU at all.
+func gpuArch(engine string, h discovery.Host) (string, error) {
 	switch h.Arch {
 	case "amd64", "arm64":
 	default:
@@ -201,33 +349,42 @@ func cudaGate(engine, version string, h discovery.Host) (variant string, err err
 	if h.NVIDIADriver == "" {
 		return "", NotHere{engine + " runs on NVIDIA GPUs and this node has none; it serves llama.cpp models"}
 	}
-	variant, cuda, minDriver, minCC := "cu13", "13", minDriverForCUDA13, minComputeForCUDA13
-	if versionBefore(version, cuda13Since[engine]) {
-		variant, cuda, minDriver, minCC = "cu12", "12", minDriverForCUDA12, minComputeForCUDA12
-	}
-	if engine == "vLLM" {
-		minCC = minComputeFor(version, h.Arch)
-	}
-	if major := leadingInt(h.NVIDIADriver); major > 0 && major < minDriver {
-		return "", fmt.Errorf("%s %s is built for CUDA %s, which needs NVIDIA driver %d or newer; this node has %s",
-			engine, version, cuda, minDriver, h.NVIDIADriver)
-	}
-	if cc := h.MinComputeCapability; cc > 0 && cc < minCC {
-		return "", fmt.Errorf("%s %s has kernels for GPUs of compute capability %.1f and newer; this node has %.1f%s",
-			engine, version, minCC, cc, olderHint(engine, cc))
-	}
-	return variant, nil
+	return h.Arch, nil
 }
 
-// olderHint says which version runs on a GPU too old for the one asked for.
-func olderHint(engine string, cc float64) string {
-	if engine == "vLLM" && cc >= minComputeForCUDA12 && cc < minComputeForCUDA13 {
-		return "; the last vLLM built for it is " + voltaUntil +
-			": give these nodes that version with engine_overrides in the server's configuration"
+// --- SGLang ---
+
+// sglangCUDA13Since is the first SGLang built for CUDA 13.
+const sglangCUDA13Since = "0.5"
+
+func sglangCandidates(version, variant string, h discovery.Host) ([]Build, error) {
+	if _, err := gpuArch("SGLang", h); err != nil {
+		return nil, err
 	}
-	return ""
+	b := Build{
+		Variant:      "cu13",
+		Python:       pythonForEngines,
+		Requirements: []string{"sglang==" + version},
+		// SGLang is launched as a module, so what the agent is given is the
+		// environment's interpreter, and the proof is importing it.
+		Binary:      "python",
+		VersionArgs: []string{"-c", "import sglang; print('sglang', sglang.__version__)"},
+		Packages:    compilerPackages,
+		FreeGB:      25,
+		GPUCheck:    true,
+	}
+	needs := gpuNeeds{minDriverForCUDA13, 7.5}
+	if versionBefore(version, sglangCUDA13Since) {
+		b.Variant, needs = "cu12", gpuNeeds{minDriverForCUDA12, 7.0}
+	}
+	return pick("SGLang "+version, variant, []candidate{{b, needs, "SGLang " + version}}, h)
 }
 
+// Python engines need a C compiler at run time: Triton builds a small
+// launcher for its kernels the first time a model loads, and without gcc
+// that load fails long after the installation looked finished.
+var compilerPackages = map[string][]string{"deb": {"gcc"}, "rpm": {"gcc"}}
+
 // versionBefore reports whether version a is older than b, comparing the
 // dotted numbers in turn ("0.25.1" before "0.26"). A part that is not a
 // number counts as zero.
@@ -251,146 +408,6 @@ func versionBefore(a, b string) bool {
 	return false
 }
 
-// Python engines need a C compiler at run time: Triton builds a small
-// launcher for its kernels the first time a model loads, and without gcc
-// that load fails long after the installation looked finished.
-var compilerPackages = map[string][]string{"deb": {"gcc"}, "rpm": {"gcc"}}
-
-// VLLMVariants are the builds of vLLM a node can be given by choice: CUDA
-// 13 from PyPI, and CUDA 12.9 from the project's GitHub release — the same
-// version, built for the drivers and GPUs CUDA 13 left behind.
-var VLLMVariants = []string{"cu13", "cu129"}
-
-// pytorchCU129 is where the CUDA 12.9 build's torch is published. PyPI's
-// torch is built for CUDA 13; the cu129 vLLM wheel needs this one.
-const pytorchCU129 = "https://download.pytorch.org/whl/cu129"
-
-// chooseVLLM picks the vLLM build for a machine.
-//
-// Left to the machine (variant ""), it is CUDA 13 where that runs, and the
-// same version's CUDA 12.9 build where only that does: one version across
-// the cluster, which matters because the flags llmgw builds are checked
-// against one version, and an older vLLM on the older cards would be a
-// second set of flags nobody checked. A variant chosen for the node is that
-// build, if the node can run it.
-func chooseVLLM(version, variant string, h discovery.Host) (Build, error) {
-	// What no build of it gets past: a machine it is not published for, or
-	// one with no NVIDIA GPU. Only a CUDA requirement falls back to the
-	// CUDA 12.9 build.
-	switch h.Arch {
-	case "amd64", "arm64":
-	default:
-		return Build{}, fmt.Errorf("vLLM publishes no Linux build for %s", orUnknown(h.Arch))
-	}
-	if h.NVIDIADriver == "" {
-		return Build{}, NotHere{"vLLM runs on NVIDIA GPUs and this node has none; it serves llama.cpp models"}
-	}
-	pypi := func(v string) Build {
-		return Build{
-			Variant:      v,
-			Python:       pythonForEngines,
-			Requirements: []string{"vllm==" + version},
-			Binary:       "vllm",
-			Packages:     compilerPackages,
-			FreeGB:       25,
-			GPUCheck:     true,
-		}
-	}
-	cu129 := func() (Build, error) {
-		if err := cudaFits("vLLM", version, "12.9", minDriverForCUDA12, minComputeFor(version, h.Arch), h); err != nil {
-			return Build{}, err
-		}
-		arch := map[string]string{"amd64": "x86_64", "arm64": "aarch64"}[h.Arch]
-		b := pypi("cu129")
-		b.Requirements = nil
-		// Found in the release's assets rather than named: the manylinux tag
-		// in the file's name changes between versions (2_28, 2_31, 2_34).
-		b.Wheel = &Wheel{
-			Project: "vllm", Repo: "vllm-project/vllm", Tag: "v" + version,
-			Prefix: "vllm-" + version + "+cu129-", Suffix: "_" + arch + ".whl",
-		}
-		b.ExtraIndex = pytorchCU129
-		return b, nil
-	}
-	// From 0.20 the PyPI build is CUDA 13 and GitHub carries "+cu129".
-	modern := !versionBefore(version, cuda13Since["vLLM"])
-	switch variant {
-	case "":
-		v, err := cudaGate("vLLM", version, h)
-		if err == nil {
-			return pypi(v), nil
-		}
-		var nh NotHere
-		if errors.As(err, &nh) || !modern {
-			return Build{}, err
-		}
-		b, err129 := cu129()
-		if err129 != nil {
-			return Build{}, fmt.Errorf("%v; nor can its CUDA 12.9 build: %v", err, err129)
-		}
-		return b, nil
-	case "cu13":
-		if !modern {
-			return Build{}, fmt.Errorf("vLLM %s is built for CUDA 12; there is no CUDA 13 build of it", version)
-		}
-		if _, err := cudaGate("vLLM", version, h); err != nil {
-			return Build{}, err
-		}
-		return pypi("cu13"), nil
-	case "cu129":
-		if !modern {
-			// Before 0.20 PyPI's own build is CUDA 12, and is the one given.
-			v, err := cudaGate("vLLM", version, h)
-			if err != nil {
-				return Build{}, err
-			}
-			return pypi(v), nil
-		}
-		if _, err := cudaGate("vLLM", version, h); err != nil {
-			var nh NotHere
-			if errors.As(err, &nh) {
-				return Build{}, err
-			}
-		}
-		return cu129()
-	}
-	return Build{}, fmt.Errorf("vLLM has no build %q; the builds are %s", variant, strings.Join(VLLMVariants, ", "))
-}
-
-// cudaFits checks a node against a CUDA's driver and GPU requirements.
-func cudaFits(engine, version, cuda string, minDriver int, minCC float64, h discovery.Host) error {
-	if major := leadingInt(h.NVIDIADriver); major > 0 && major < minDriver {
-		return fmt.Errorf("%s %s built for CUDA %s needs NVIDIA driver %d or newer; this node has %s",
-			engine, version, cuda, minDriver, h.NVIDIADriver)
-	}
-	if cc := h.MinComputeCapability; cc > 0 && cc < minCC {
-		return fmt.Errorf("%s %s built for CUDA %s needs GPUs of compute capability %.1f or newer; this node has %.1f%s",
-			engine, version, cuda, minCC, cc, olderHint(engine, cc))
-	}
-	return nil
-}
-
-func chooseSGLang(version, chosen string, h discovery.Host) (Build, error) {
-	if chosen != "" {
-		return Build{}, fmt.Errorf("SGLang is installed in one build only, not %q", chosen)
-	}
-	variant, err := cudaGate("SGLang", version, h)
-	if err != nil {
-		return Build{}, err
-	}
-	// SGLang is launched as a module, so what the agent is given is the
-	// environment's interpreter, and the proof is importing it.
-	return Build{
-		Variant:      variant,
-		Python:       pythonForEngines,
-		Requirements: []string{"sglang==" + version},
-		Binary:       "python",
-		VersionArgs:  []string{"-c", "import sglang; print('sglang', sglang.__version__)"},
-		Packages:     compilerPackages,
-		FreeGB:       25,
-	}, nil
-}
-
 // leadingInt is the number a version string starts with: 580 of "580.65.06".
 func leadingInt(s string) int {
 	n := 0
@@ -403,17 +420,30 @@ func leadingInt(s string) int {
 	return n
 }
 
-// cudaForLlamaCpp is the CUDA build taken on an NVIDIA node. The release also
-// carries a matching cudart archive, so the node needs a driver and nothing
-// else — no toolkit.
-const cudaForLlamaCpp = "12.8"
+// --- llama.cpp ---
 
-func chooseLlamaCpp(version, _ string, h discovery.Host) (Build, error) {
-	// The published Linux builds are built on Ubuntu, but nothing about them
-	// is Ubuntu's: they need glibc 2.34 and the libstdc++ of GCC 12, which
-	// Debian 12 and RHEL 10 have and RHEL 9 does not. So the distribution's
-	// name decides nothing here. The build is tried everywhere, and ldd says
-	// whether this system can load it — in the words of the missing symbol.
+// llamaCUDA12Real are the GPUs llama.cpp's CUDA 12.8 build carries machine
+// code for. The rest of what it covers — 5.0, 6.1, 7.0, 7.5, 8.0, 9.0 — it
+// carries as PTX, which a driver older than the CUDA it came from cannot
+// compile.
+var llamaCUDA12Real = []float64{8.6, 8.9, 12.0}
+
+// llamaCppCandidates are the llama.cpp builds for a machine.
+//
+// On an NVIDIA node, the GPU builds best first: CUDA 13.3 (driver 580,
+// Turing and newer), CUDA 12.8 on x64 (Maxwell to Blackwell; driver 570 for
+// the GPUs it ships as PTX, 525 for those it ships as machine code), and
+// Vulkan, which runs on any card whose driver has Vulkan — the generations
+// CUDA builds no longer cover. Never the CPU build on an NVIDIA node: that is
+// a GPU placement served on the CPU, which is what installing checks for.
+// The CPU build where there is no NVIDIA GPU.
+//
+// The published Linux builds are built on Ubuntu, but nothing about them is
+// Ubuntu's: they need glibc 2.34 and the libstdc++ of GCC 12, which Debian 12
+// and RHEL 10 have and RHEL 9 does not. So the distribution's name decides
+// nothing here. The build is tried everywhere, and ldd says whether this
+// system can load it — in the words of the missing symbol.
+func llamaCppCandidates(version, variant string, h discovery.Host) ([]Build, error) {
 	var arch string
 	switch h.Arch {
 	case "amd64":
@@ -421,36 +451,53 @@ func chooseLlamaCpp(version, _ string, h discovery.Host) (Build, error) {
 	case "arm64":
 		arch = "arm64"
 	default:
-		return Build{}, fmt.Errorf("llama.cpp publishes no Linux build for %s", orUnknown(h.Arch))
+		return nil, fmt.Errorf("llama.cpp publishes no Linux build for %s", orUnknown(h.Arch))
 	}
-
-	b := Build{
-		Variant: "cpu",
-		Assets:  []string{fmt.Sprintf("llama-%s-bin-ubuntu-%s.tar.gz", version, arch)},
-		// Everything else the build links is in the archive or in any current
-		// glibc system; OpenMP's runtime is the one a cloud image lacks, and
-		// the server does not start without it.
-		Packages: map[string][]string{"deb": {"libgomp1"}, "rpm": {"libgomp"}},
-		Binary:   "llama-server",
-	}
-	if arch == "x64" {
-		// The archive carries a backend per CPU generation and loads the best
-		// one, down to SSE4.2. Below AVX2 it runs, several times slower.
-		b.CPUFlags = []string{"avx2"}
-	}
-	// The CUDA build is published for x64 at this version; on arm64 it is a
-	// newer CUDA that the driver may not support, so an arm64 GPU node runs
-	// the CPU build until someone decides otherwise.
-	if h.NVIDIADriver != "" && arch == "x64" {
-		name := fmt.Sprintf("bin-ubuntu-cuda-%s-x64.tar.gz", cudaForLlamaCpp)
-		b.Variant = "cuda-" + cudaForLlamaCpp
-		b.GPUBackend = "libggml-cuda.so"
-		b.Assets = []string{
-			fmt.Sprintf("llama-%s-%s", version, name),
-			fmt.Sprintf("cudart-llama-%s-%s", version, name),
+	base := func(variant string, assets ...string) Build {
+		b := Build{
+			Variant: variant,
+			Assets:  assets,
+			// Everything else the build links is in the archive or in any
+			// current glibc system; OpenMP's runtime is the one a cloud image
+			// lacks, and the server does not start without it.
+			Packages: map[string][]string{"deb": {"libgomp1"}, "rpm": {"libgomp"}},
+			Binary:   "llama-server",
 		}
+		if arch == "x64" {
+			// The archive carries a backend per CPU generation and loads the
+			// best one, down to SSE4.2. Below AVX2 it runs, several times
+			// slower.
+			b.CPUFlags = []string{"avx2"}
+		}
+		return b
 	}
-	return b, nil
+	if h.NVIDIADriver == "" {
+		return pick("llama.cpp", variant, []candidate{
+			{base("cpu", fmt.Sprintf("llama-%s-bin-ubuntu-%s.tar.gz", version, arch)), gpuNeeds{}, "llama.cpp"},
+		}, h)
+	}
+	cuda := func(cv string) Build {
+		name := fmt.Sprintf("bin-ubuntu-cuda-%s-%s.tar.gz", cv, arch)
+		// The release carries the CUDA runtime beside the build, so the node
+		// needs a driver and nothing else — no toolkit.
+		b := base("cuda-"+cv, fmt.Sprintf("llama-%s-%s", version, name), fmt.Sprintf("cudart-llama-%s-%s", version, name))
+		b.GPUBackend = "libggml-cuda.so"
+		return b
+	}
+	vulkan := base("vulkan", fmt.Sprintf("llama-%s-bin-ubuntu-vulkan-%s.tar.gz", version, arch))
+	vulkan.GPUBackend = "libggml-vulkan.so"
+	vulkan.Packages = map[string][]string{"deb": {"libgomp1", "libvulkan1"}, "rpm": {"libgomp", "vulkan-loader"}}
+
+	cuda12Driver := 570
+	if slices.Contains(llamaCUDA12Real, h.MinComputeCapability) {
+		cuda12Driver = minDriverForCUDA12
+	}
+	cands := []candidate{{cuda("13.3"), gpuNeeds{minDriverForCUDA13, 7.5}, "llama.cpp built for CUDA 13.3"}}
+	if arch == "x64" {
+		cands = append(cands, candidate{cuda("12.8"), gpuNeeds{cuda12Driver, 5.0}, "llama.cpp built for CUDA 12.8"})
+	}
+	cands = append(cands, candidate{vulkan, gpuNeeds{}, "llama.cpp built for Vulkan"})
+	return pick("llama.cpp", variant, cands, h)
 }
 
 func orUnknown(s string) string {

internal/cli/backup.go

@@ -8,7 +8,6 @@ import (
 	"fmt"
 	"io"
 	"io/fs"
-	"net/url"
 	"os"
 	"os/exec"
 	"path/filepath"
@@ -178,30 +177,49 @@ func addStore(ctx context.Context, tw *tar.Writer, addr, scratch string) error {
 // command line: a DSN carries the password, and every user on the machine
 // can read another process's arguments.
 func pgDump(ctx context.Context, dsn, out string) error {
-	u, err := url.Parse(dsn)
+	vars, err := pgEnv(dsn)
 	if err != nil {
-		return fmt.Errorf("the store address is not a URL")
+		return err
 	}
-	env := os.Environ()
+	cmd := exec.CommandContext(ctx, "pg_dump", "--format=custom", "--no-owner", "--file="+out) //nolint:gosec // fixed binary, our own file
+	cmd.Env = append(os.Environ(), vars...)
+	if b, err := cmd.CombinedOutput(); err != nil {
+		return fmt.Errorf("pg_dump: %v: %s", err, strings.TrimSpace(string(b)))
+	}
+	return nil
+}
+
+// pgEnv turns a store address into libpq's environment. Several hosts — the
+// members of a replicated cluster — become libpq's own lists in PGHOST and
+// PGPORT; read as one URL host, the list was one name libpq could not
+// resolve, and a backup worked only by falling through to the second.
+func pgEnv(dsn string) ([]string, error) {
+	hosts, ports, u, err := store.SplitPostgres(dsn)
+	if err != nil {
+		return nil, err
+	}
+	var env []string
 	set := func(k, v string) {
 		if v != "" {
 			env = append(env, k+"="+v)
 		}
 	}
-	set("PGHOST", u.Hostname())
-	set("PGPORT", u.Port())
+	set("PGHOST", strings.Join(hosts, ","))
+	set("PGPORT", strings.Join(ports, ","))
 	set("PGUSER", u.User.Username())
 	if pw, ok := u.User.Password(); ok {
 		set("PGPASSWORD", pw)
 	}
 	set("PGDATABASE", strings.TrimPrefix(u.Path, "/"))
 	set("PGSSLMODE", u.Query().Get("sslmode"))
-	cmd := exec.CommandContext(ctx, "pg_dump", "--format=custom", "--no-owner", "--file="+out) //nolint:gosec // fixed binary, our own file
-	cmd.Env = env
-	if b, err := cmd.CombinedOutput(); err != nil {
-		return fmt.Errorf("pg_dump: %v: %s", err, strings.TrimSpace(string(b)))
+	// A standby if there is one, else whichever answers — not the address's
+	// read-write: a dump only reads, it need not load the primary, and one
+	// that insists on the primary fails exactly while the cluster is failing
+	// over.
+	if len(hosts) > 1 {
+		set("PGTARGETSESSIONATTRS", "prefer-standby")
 	}
-	return nil
+	return env, nil
 }
 
 func addTree(tw *tar.Writer, root, as string) error {

internal/cli/backup_test.go

@@ -139,3 +139,26 @@ func TestNoScratchCopyOfTheStoreIsLeftBehind(t *testing.T) {
 		}
 	}
 }
+
+func TestAStoreOfSeveralHostsIsGivenToPgDumpAsLibpqLists(t *testing.T) {
+	// The address of a Patroni cluster: url.Parse read its host as one name,
+	// and pg_dump reached the database only by falling through to the second.
+	env, err := pgEnv("postgres://llmgw:s3cret@10.0.0.149:5432,10.0.0.150/llmgw?sslmode=disable&target_session_attrs=read-write")
+	if err != nil {
+		t.Fatal(err)
+	}
+	got := strings.Join(env, " ")
+	for _, want := range []string{"PGHOST=10.0.0.149,10.0.0.150", "PGPORT=5432,5432", "PGUSER=llmgw",
+		"PGPASSWORD=s3cret", "PGDATABASE=llmgw", "PGSSLMODE=disable", "PGTARGETSESSIONATTRS=prefer-standby"} {
+		if !strings.Contains(got, want) {
+			t.Errorf("%s is not in %s", want, got)
+		}
+	}
+	one, err := pgEnv("postgres://u:p@db.internal:6432/llmgw")
+	if err != nil || !strings.Contains(strings.Join(one, " "), "PGHOST=db.internal PGPORT=6432") {
+		t.Errorf("one host: %v, %v", one, err)
+	}
+	if _, err := pgEnv("postgres://u:p@a:5432,:5432/x"); err == nil {
+		t.Error("an empty host in the list was accepted")
+	}
+}

internal/cli/ctl_status_test.go

@@ -1,9 +1,14 @@
 package cli
 
 import (
+	"bytes"
 	"net/http"
 	"net/http/httptest"
+	"strings"
 	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"github.com/spf13/cobra"
 )
 
 func TestAQueuedChangeIsNotReportedAsAnError(t *testing.T) {
@@ -22,3 +27,27 @@ func TestAQueuedChangeIsNotReportedAsAnError(t *testing.T) {
 		t.Errorf("the body of a 202 was not read: %+v", out)
 	}
 }
+
+func TestStatusSaysWhoLeadsTheStoreAndWhetherTheRestKeepUp(t *testing.T) {
+	render := func(s adminapi.Summary) string {
+		cmd := &cobra.Command{}
+		var b bytes.Buffer
+		cmd.SetOut(&b)
+		printSummary(cmd, s)
+		return b.String()
+	}
+	out := render(adminapi.Summary{Store: []adminapi.StoreMember{
+		{Name: "llmgw-ctl02", Role: "replica", State: "streaming", Lag: 2048},
+		{Name: "llmgw-pg01", Role: "leader", State: "running"},
+	}})
+	if !strings.Contains(out, "STORE     llmgw-ctl02 streaming (2 KB behind), llmgw-pg01 leads") {
+		t.Errorf("status:\n%s", out)
+	}
+	out = render(adminapi.Summary{Store: []adminapi.StoreMember{{Name: "llmgw-pg01", Role: "replica", State: "stopped"}}})
+	if !strings.Contains(out, "NO LEADER") {
+		t.Errorf("a store with no leader was not called out:\n%s", out)
+	}
+	if out := render(adminapi.Summary{}); strings.Contains(out, "STORE") {
+		t.Error("a single database got a store line")
+	}
+}

internal/cli/enginebuild.go

@@ -22,8 +22,8 @@ func newEngineBuildCmd(client func() *ctlClient) *cobra.Command {
 			"  none    no vLLM on this node\n\n" +
 			"The node is told at once and installs it in the background; a build it\n" +
 			"cannot compute with on its GPU fails to install, saying why, and the\n" +
-			"cluster page shows it. A V100 runs neither build of vLLM 0.20 or later;\n" +
-			"give it 0.19.1 with engine_overrides in the server's configuration.\n\n" +
+			"cluster page shows it. A V100 runs no build of vLLM 0.20 or later, and\n" +
+			"gets 0.19.1, the newest built for it, by itself.\n\n" +
 			"  llmgw ctl engine-build gpu07 vllm cu129",
 		Args: cobra.ExactArgs(3),
 		RunE: func(cmd *cobra.Command, args []string) error {

internal/cli/join.go

@@ -0,0 +1,61 @@
+package cli
+
+import (
+	"errors"
+	"fmt"
+	"os"
+	"runtime"
+
+	"github.com/spf13/cobra"
+
+	joinpkg "gitea.mynodes.xyz/ilker/llmgw/internal/join"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/release"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/version"
+)
+
+func newJoinCmd() *cobra.Command {
+	return &cobra.Command{
+		Use:   "join <code>",
+		Short: "Let a setup on another machine set this one up, with the code it gave",
+		Long: "For a machine nothing may log into: the setup screens (sudo llmgw on\n" +
+			"the control plane) show a code for each machine they expect. Run this\n" +
+			"with it, as root, on that machine. It dials out to the setup — nothing\n" +
+			"dials in — checks it is the setup the code was made by, and runs what\n" +
+			"the setup sends until it is done: the same checks and installs ssh\n" +
+			"would carry. Each step is printed here as it runs.\n\n" +
+			"A code is good for one machine, for an hour, and only while the setup\n" +
+			"that made it is waiting.",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			if runtime.GOOS != "linux" {
+				return fmt.Errorf("join sets up a Linux machine; this is %s", runtime.GOOS)
+			}
+			if os.Geteuid() != 0 {
+				return errors.New("run it as root — sudo llmgw join … — since the setup installs software here")
+			}
+			code, err := joinpkg.Parse(args[0])
+			if err != nil {
+				return err
+			}
+			out := cmd.OutOrStdout()
+			role := code.Role
+			if role == "" {
+				role = "part of a cluster"
+			}
+			fmt.Fprintf(out, "Joining the setup at %s, as %s. It runs as root here until it is done.\n", code.Addr, role)
+			return joinpkg.Serve(cmd.Context(), code, func(s string) { fmt.Fprintln(out, s) })
+		},
+	}
+}
+
+// joinInstall is what to run on a machine that has no llmgw yet, to fetch
+// this version for its architecture: short lines, since it is copied from a
+// screen that cuts long ones off.
+func joinInstall() []string {
+	v := version.Version
+	return []string{
+		`a=$(uname -m | sed 's/x86_64/amd64/;s/aarch64/arm64/')`,
+		fmt.Sprintf("sudo curl -fsSL -o /usr/local/bin/llmgw \\\n    %s/%s/llmgw-%s-linux-$a", release.DefaultBase, v, v),
+		"sudo chmod 755 /usr/local/bin/llmgw",
+	}
+}

internal/cli/root.go

@@ -30,6 +30,14 @@ func Root() *cobra.Command {
 		PersistentPreRun: func(_ *cobra.Command, _ []string) {
 			obs.Setup(obs.Options{Level: global.logLevel, Format: global.logFormat})
 		},
+		// Bare, in a terminal: the setup screens or the console. Otherwise
+		// the help, as before.
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			if interactive(cmd) {
+				return runHome(cmd)
+			}
+			return cmd.Help()
+		},
 	}
 
 	pf := root.PersistentFlags()
@@ -49,6 +57,7 @@ func Root() *cobra.Command {
 		newNodeCmd(),
 		newSelfUpgradeCmd(),
 		newReleaseCmd(),
+		newJoinCmd(),
 	)
 	return root
 }

internal/cli/server.go

@@ -19,6 +19,7 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/pgha"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/sourcecheck"
 	"github.com/prometheus/client_golang/prometheus"
 
@@ -127,6 +128,9 @@ func newServerCmd() *cobra.Command {
 			var agentHandler http.Handler
 			var agentPath string
 			var admin *control.Controller
+			// The store's cluster, when it is one; read by the metrics and
+			// by `ctl status`.
+			var storeHA *pgha.Collector
 			if controlMode {
 				cat, err := cfg.Catalog()
 				if err != nil {
@@ -179,6 +183,17 @@ func newServerCmd() *cobra.Command {
 				if err := rt.RegisterMetrics(fleetmetrics.New(ctl)); err != nil {
 					return err
 				}
+				// A store of several hosts is a replicated cluster: what state
+				// it is in is published beside everything else, read from its
+				// Patroni. One host is a single PostgreSQL, which has none.
+				if addr, err := storepkg.ResolveAddress(cfg.Auth.Store); err == nil && storepkg.IsPostgres(addr) {
+					if hosts, _, _, err := storepkg.SplitPostgres(strings.TrimSpace(addr)); err == nil && len(hosts) > 1 {
+						storeHA = pgha.NewCollector(hosts)
+						if err := rt.RegisterMetrics(storeHA); err != nil {
+							return err
+						}
+					}
+				}
 				// So a request for a model that is configured but not running
 				// starts it, rather than being told nothing serves it.
 				rt.SetActivator(ctl.Reconciler())
@@ -201,6 +216,16 @@ func newServerCmd() *cobra.Command {
 			)
 			if admin != nil {
 				api := adminapi.New(admin)
+				if storeHA != nil {
+					api.SetStoreStatus(func() ([]adminapi.StoreMember, error) {
+						c, err := storeHA.Read()
+						var out []adminapi.StoreMember
+						for _, m := range c.Members {
+							out = append(out, adminapi.StoreMember{Name: m.Name, Host: m.Host, Role: m.Role, State: m.State, Lag: m.Lag})
+						}
+						return out, err
+					})
+				}
 				adminAPI = api
 				api.SetReleases(cfg.Releases)
 				// Catalog changes need somewhere to persist, so this is wired
@@ -426,7 +451,7 @@ func newServerCmd() *cobra.Command {
 	f.StringVar(&metricsListen, "metrics-listen", "", "serve metrics only on this address, safe to expose")
 	f.StringArrayVar(&endpoints, "endpoint", nil, "static endpoint: id=URL=model[,model] (repeatable)")
 
-	cmd.AddCommand(newServerInitCmd())
+	cmd.AddCommand(newServerInitCmd(), newServerAddCmd())
 	cmd.AddCommand(newBackupCmd())
 	return cmd
 }
@@ -488,6 +513,9 @@ func startControlPlane(ctx context.Context, cfg config.Server, cat *catalog.Live
 	if h, ok := sink.(interface{ Store() *storepkg.DB }); ok {
 		if st := h.Store(); st != nil {
 			lead = control.NewLeadership(st, cfg.Control.LeaseTTL, log)
+			if r, ok := sink.(interface{ SetLeader(func() bool) }); ok {
+				r.SetLeader(lead.IsLeader)
+			}
 			ctl.SetLeadership(lead)
 			// The tenancy tree's capacity quotas: what each part is
 			// guaranteed and may hold.
@@ -517,6 +545,21 @@ func startControlPlane(ctx context.Context, cfg config.Server, cat *catalog.Live
 	if lead != nil {
 		svc.SetLeadership(lead)
 	}
+	// The control planes agents may dial, from the store every control plane
+	// shares: told to each agent as it registers, and to all of them when
+	// `server add` records another.
+	if h, ok := sink.(interface{ Store() *storepkg.DB }); ok && h.Store() != nil {
+		st := h.Store()
+		svc.SetControlPlanes(func(ctx context.Context) ([]string, error) {
+			cps, err := st.ControlPlanes(ctx)
+			urls := make([]string, len(cps))
+			for i, c := range cps {
+				urls[i] = c.URL
+			}
+			return urls, err
+		})
+		go svc.WatchControlPlanes(ctx, 30*time.Second)
+	}
 	fwd.set(svc)
 	// Every InfiniBand fabric mapped every five minutes, through one node.
 	go svc.ScanFabrics(ctx, 5*time.Minute)

internal/cli/serveradd.go

@@ -0,0 +1,459 @@
+package cli
+
+import (
+	"bytes"
+	"context"
+	"crypto/rand"
+	"crypto/x509"
+	"encoding/hex"
+	"encoding/pem"
+	"errors"
+	"fmt"
+	"io"
+	"net"
+	"os"
+	"path/filepath"
+	"slices"
+	"strings"
+	"time"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/pgha"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/version"
+)
+
+// serverAdd makes another machine a control plane of this cluster: a standby
+// that takes over when this one is lost. Run on a control plane that holds
+// the authority, as root, like `node add`.
+type serverAdd struct {
+	etcDir, pkiDir string
+	// hosts are more names to put in the new control plane's certificate.
+	hosts []string
+	// vip is a floating address, "10.0.0.160/24", to keep on whichever of the
+	// two leads; "" leaves client traffic to whatever is in front.
+	vip string
+
+	remote onboard.Remote
+	// addr is the machine's address as given; ip what the others reach it at.
+	addr, ip, name string
+	out            io.Writer
+}
+
+func newServerAddCmd() *cobra.Command {
+	sa := serverAdd{etcDir: "/etc/llmgw", pkiDir: defaultPKIDir}
+	cmd := &cobra.Command{
+		Use:   "add [user@]address",
+		Short: "Make another machine a control plane of this cluster: a standby that takes over",
+		Long: "Run on a control plane, as root. It logs in to the machine with ssh, as\n" +
+			"node add does, and gives it this llmgw, a certificate from this\n" +
+			"cluster's authority, this control plane's configuration and the store's\n" +
+			"address, lets it into the store, and starts it. It stands by while this\n" +
+			"one leads, and takes over within seconds when this one is lost; every\n" +
+			"node is told of it and fails over to it without being touched.\n\n" +
+			"The store must be one both can reach — replicated with 'server init\n" +
+			"--patroni' — not a database on this machine.\n\n" +
+			"--vip keeps a floating address on whichever control plane leads, with\n" +
+			"keepalived on both: the address clients use for the gateway.\n\n" +
+			"Run it again at any time: it changes only what is missing.",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			if os.Geteuid() != 0 {
+				return errors.New("run it as root: it reads the certificate authority and the store's address")
+			}
+			user, host, ok := strings.Cut(args[0], "@")
+			if !ok {
+				user, host = "", args[0]
+			}
+			sa.remote = &onboard.SSH{Addr: host, User: user}
+			sa.addr = host
+			sa.out = cmd.OutOrStdout()
+			return sa.run(cmd.Context())
+		},
+	}
+	f := cmd.Flags()
+	f.StringArrayVar(&sa.hosts, "host", nil, "another name for the new control plane's certificate; repeatable")
+	f.StringVar(&sa.vip, "vip", "", "a floating address for whichever control plane leads, as 10.0.0.160/24")
+	f.StringVar(&sa.pkiDir, "pki", sa.pkiDir, "directory holding the certificate authority")
+	return cmd
+}
+
+func (sa *serverAdd) done(format string, a ...any) {
+	fmt.Fprintf(sa.out, "  ok    "+format+"\n", a...)
+}
+
+func (sa *serverAdd) kept(format string, a ...any) {
+	fmt.Fprintf(sa.out, "  kept  "+format+"\n", a...)
+}
+
+func (sa *serverAdd) sh(ctx context.Context, script string, stdin []byte) (string, error) {
+	out, err := sa.remote.Run(ctx, script, stdin)
+	if err != nil {
+		return string(out), fmt.Errorf("on %s: %v: %s", sa.addr, err, strings.TrimSpace(string(out)))
+	}
+	return string(out), nil
+}
+
+// put writes a file there, through standard input, with its mode and owner.
+func (sa *serverAdd) put(ctx context.Context, path, mode, owner string, body []byte) error {
+	_, err := sa.sh(ctx, fmt.Sprintf(`set -e
+sudo install -d -m 0755 %[1]s
+sudo sh -c 'umask 077; cat > %[2]s.new'
+sudo chmod %[3]s %[2]s.new
+sudo chown %[4]s %[2]s.new
+sudo mv -f %[2]s.new %[2]s`, filepath.Dir(path), path, mode, owner), body)
+	return err
+}
+
+func (sa *serverAdd) run(ctx context.Context) error {
+	steps := []struct {
+		name string
+		fn   func(context.Context) error
+	}{
+		{"connect", sa.connect},
+		{"binary", sa.binary},
+		{"directories", sa.dirs},
+		{"certificate", sa.certificate},
+		{"configuration", sa.configuration},
+		{"store", sa.storeAccess},
+		{"service", sa.service},
+	}
+	if sa.vip != "" {
+		steps = append(steps, struct {
+			name string
+			fn   func(context.Context) error
+		}{"floating address", sa.floating})
+	}
+	for _, s := range steps {
+		if err := s.fn(ctx); err != nil {
+			return fmt.Errorf("%s: %w", s.name, err)
+		}
+	}
+	fmt.Fprintf(sa.out, "\n%s is a control plane of this cluster, standing by while this one leads.\n", sa.name)
+	return nil
+}
+
+func (sa *serverAdd) connect(ctx context.Context) error {
+	out, err := sa.sh(ctx, "hostname; sudo -n true && echo sudo-ok", nil)
+	if err != nil {
+		return fmt.Errorf("%w — it needs ssh and sudo without a password", err)
+	}
+	lines := strings.Split(strings.TrimSpace(out), "\n")
+	sa.name = strings.TrimSpace(lines[0])
+	if sa.ip == "" {
+		if sa.ip, err = ipv4Of(sa.addr); err != nil {
+			return err
+		}
+	}
+	sa.done("%s (%s)", sa.name, sa.ip)
+	return nil
+}
+
+// binary gives it this very llmgw, so the two control planes never differ.
+func (sa *serverAdd) binary(ctx context.Context) error {
+	have, _ := sa.sh(ctx, "llmgw --version 2>/dev/null || true", nil)
+	if strings.TrimSpace(have) == "llmgw version "+version.Version {
+		sa.kept("llmgw %s", version.Version)
+		return nil
+	}
+	exe, err := os.Executable()
+	if err != nil {
+		return err
+	}
+	b, err := os.ReadFile(exe)
+	if err != nil {
+		return err
+	}
+	if _, err := sa.sh(ctx, `set -e; t=$(mktemp); cat > "$t"; sudo install -m 0755 "$t" /usr/local/bin/llmgw; rm -f "$t"`, b); err != nil {
+		return err
+	}
+	sa.done("llmgw %s, copied from this machine", version.Version)
+	return nil
+}
+
+func (sa *serverAdd) dirs(ctx context.Context) error {
+	_, err := sa.sh(ctx, `set -e
+id llmgw >/dev/null 2>&1 || sudo useradd --system --home /var/lib/llmgw --shell /usr/sbin/nologin llmgw
+sudo install -d -m 0755 /etc/llmgw
+sudo install -d -o llmgw -g llmgw -m 0700 /etc/llmgw/secrets
+sudo install -d -o llmgw -g llmgw -m 0750 /var/lib/llmgw`, nil)
+	if err == nil {
+		sa.done("user llmgw, /etc/llmgw and /var/lib/llmgw")
+	}
+	return err
+}
+
+// certificate issues it a server certificate from this cluster's authority,
+// for its name and addresses, unless it holds one that already names them.
+func (sa *serverAdd) certificate(ctx context.Context) error {
+	ca, err := loadCA(sa.pkiDir)
+	if err != nil {
+		return fmt.Errorf("the authority's key is not in %s: run this where it is (%v)", sa.pkiDir, err)
+	}
+	ips, _ := sa.sh(ctx, "hostname -I", nil)
+	want := []string{sa.name, sa.ip}
+	for _, ip := range strings.Fields(ips) {
+		if net.ParseIP(ip).To4() != nil && !slices.Contains(want, ip) {
+			want = append(want, ip)
+		}
+	}
+	want = append(want, sa.hosts...)
+
+	if have, _ := sa.sh(ctx, "sudo cat /etc/llmgw/server.pem 2>/dev/null || true", nil); have != "" {
+		if blk, _ := pem.Decode([]byte(have)); blk != nil {
+			if cert, err := x509.ParseCertificate(blk.Bytes); err == nil && cert.CheckSignatureFrom(ca.Cert) == nil {
+				names := strings.Split(sanNames(cert), ", ")
+				missing := false
+				for _, w := range want {
+					missing = missing || !slices.Contains(names, w)
+				}
+				if !missing {
+					sa.kept("certificate for %s", strings.Join(names, ", "))
+					return sa.put(ctx, "/etc/llmgw/ca.pem", "0644", "root:root", ca.PEM)
+				}
+			}
+		}
+	}
+	leaf, err := ca.IssueServer(sa.name, want)
+	if err != nil {
+		return err
+	}
+	if err := sa.put(ctx, "/etc/llmgw/ca.pem", "0644", "root:root", ca.PEM); err != nil {
+		return err
+	}
+	if err := sa.put(ctx, "/etc/llmgw/server.pem", "0644", "root:root", leaf.CertPEM); err != nil {
+		return err
+	}
+	if err := sa.put(ctx, "/etc/llmgw/server-key.pem", "0640", "root:llmgw", leaf.KeyPEM); err != nil {
+		return err
+	}
+	sa.done("certificate for %s, from this cluster's authority", strings.Join(want, ", "))
+	return nil
+}
+
+// configuration copies this control plane's, and the secrets beside it: the
+// two must agree on everything, the store above all.
+func (sa *serverAdd) configuration(ctx context.Context) error {
+	if have, _ := sa.sh(ctx, "sudo test -f /etc/llmgw/server.yaml && echo yes || true", nil); strings.TrimSpace(have) == "yes" {
+		sa.kept("/etc/llmgw/server.yaml (never overwritten)")
+	} else {
+		b, err := os.ReadFile(filepath.Join(sa.etcDir, "server.yaml"))
+		if err != nil {
+			return err
+		}
+		if err := sa.put(ctx, "/etc/llmgw/server.yaml", "0644", "root:root", b); err != nil {
+			return err
+		}
+		sa.done("/etc/llmgw/server.yaml, as this control plane's")
+	}
+	secrets, err := os.ReadDir(filepath.Join(sa.etcDir, "secrets"))
+	if err != nil && !os.IsNotExist(err) {
+		return err
+	}
+	for _, e := range secrets {
+		if e.IsDir() || strings.Contains(e.Name(), ".pre-") || strings.HasSuffix(e.Name(), ".new") {
+			continue
+		}
+		path := "/etc/llmgw/secrets/" + e.Name()
+		mine, err := os.ReadFile(filepath.Join(sa.etcDir, "secrets", e.Name()))
+		if err != nil {
+			return err
+		}
+		theirs, _ := sa.sh(ctx, "sudo cat "+path+" 2>/dev/null || true", nil)
+		switch {
+		case theirs == string(mine):
+			sa.kept("%s", path)
+		case theirs != "":
+			// Two control planes on two stores are two clusters.
+			return fmt.Errorf("%s there differs from this one's; remove it there if that machine is to join this cluster", path)
+		default:
+			if err := sa.put(ctx, path, "0600", "llmgw:llmgw", mine); err != nil {
+				return err
+			}
+			sa.done("%s", path)
+		}
+	}
+	return nil
+}
+
+// storeAccess lets it into the store, and records both control planes for
+// the nodes to be told.
+func (sa *serverAdd) storeAccess(ctx context.Context) error {
+	cfg, err := config.Load(filepath.Join(sa.etcDir, "server.yaml"))
+	if err != nil {
+		return err
+	}
+	addr, err := store.ResolveAddress(cfg.Auth.Store)
+	if err != nil {
+		return err
+	}
+	addr = strings.TrimSpace(addr)
+	if !store.IsPostgres(addr) {
+		return errors.New("the store is SQLite on this machine; a second control plane needs one both reach — run 'llmgw server init --patroni' first")
+	}
+	hosts, _, _, err := store.SplitPostgres(addr)
+	if err != nil {
+		return err
+	}
+	for _, h := range hosts {
+		if ip := net.ParseIP(h); ip != nil && ip.IsLoopback() || h == "localhost" {
+			return fmt.Errorf("the store is at %s, on this machine only; replicate it first with 'llmgw server init --patroni'", h)
+		}
+	}
+	switch {
+	case slices.Contains(hosts, sa.ip):
+		sa.kept("store: %s is one of its members", sa.ip)
+	case len(hosts) > 1:
+		changed, err := pgha.AllowClient(ctx, hosts, sa.ip)
+		if err != nil {
+			return err
+		}
+		if changed {
+			sa.done("store: %s let in, through Patroni", sa.ip)
+		} else {
+			sa.kept("store: %s already let in", sa.ip)
+		}
+	default:
+		sa.kept("store at %s: make sure it lets %s in", hosts[0], sa.ip)
+	}
+
+	st, err := store.Open(ctx, addr)
+	if err != nil {
+		return err
+	}
+	defer st.Close()
+	_, port, _ := net.SplitHostPort(cfg.AgentListen)
+	if port == "" {
+		port = "7443"
+	}
+	self, err := sourceIPToward(sa.ip)
+	if err != nil {
+		return err
+	}
+	me, _ := os.Hostname()
+	for _, cp := range []store.ControlPlane{
+		{URL: "https://" + net.JoinHostPort(self, port), Name: me},
+		{URL: "https://" + net.JoinHostPort(sa.ip, port), Name: sa.name, AddedAt: time.Now().Add(time.Second)},
+	} {
+		if err := st.PutControlPlane(ctx, cp); err != nil {
+			return err
+		}
+	}
+	sa.done("nodes will be told of both control planes within half a minute")
+	return nil
+}
+
+// service installs the unit and starts it, then waits for it to answer.
+func (sa *serverAdd) service(ctx context.Context) error {
+	if err := sa.put(ctx, "/etc/systemd/system/llmgw-server.service", "0644", "root:root", []byte(serverUnit)); err != nil {
+		return err
+	}
+	if _, err := sa.sh(ctx, "sudo systemctl daemon-reload && sudo systemctl enable llmgw-server >/dev/null 2>&1 && sudo systemctl restart llmgw-server", nil); err != nil {
+		return err
+	}
+	deadline := time.Now().Add(90 * time.Second)
+	for {
+		code, _ := sa.sh(ctx, "curl -s -o /dev/null -m 2 -w '%{http_code}' http://127.0.0.1:8080/leader || true", nil)
+		switch strings.TrimSpace(code) {
+		case "200":
+			sa.done("llmgw-server running, and leading")
+			return nil
+		case "503":
+			sa.done("llmgw-server running, standing by")
+			return nil
+		}
+		if time.Now().After(deadline) {
+			logs, _ := sa.sh(ctx, "sudo journalctl -u llmgw-server -n 15 --no-pager", nil)
+			return fmt.Errorf("it did not answer within 90s:\n%s", logs)
+		}
+		time.Sleep(2 * time.Second)
+	}
+}
+
+// floating keeps an address on whichever of the two leads, with keepalived
+// on both, following /leader.
+func (sa *serverAdd) floating(ctx context.Context) error {
+	vip, nw, err := net.ParseCIDR(sa.vip)
+	if err != nil || vip.To4() == nil {
+		return fmt.Errorf("--vip %q is not an IPv4 address with its prefix, like 10.0.0.160/24", sa.vip)
+	}
+	prefix, _ := nw.Mask.Size()
+	pass, err := vrrpPassword(sa.etcDir)
+	if err != nil {
+		return err
+	}
+	here := pgha.Local{}
+	for _, m := range []struct {
+		r        onboard.Remote
+		name     string
+		priority int
+	}{{here, "this machine", 110}, {sa.remote, sa.name, 100}} {
+		out, err := m.r.Run(ctx, fmt.Sprintf(`set -e
+command -v keepalived >/dev/null 2>&1 || { sudo DEBIAN_FRONTEND=noninteractive apt-get -qq update >/dev/null; sudo DEBIAN_FRONTEND=noninteractive apt-get -qq install -y keepalived >/dev/null; }
+ip -o route get %s | awk '{for(i=1;i<=NF;i++) if($i=="dev") {print $(i+1); exit}}'`, vip), nil)
+		if err != nil {
+			return fmt.Errorf("on %s: %v: %s", m.name, err, strings.TrimSpace(string(out)))
+		}
+		iface := strings.TrimSpace(string(out))
+		conf := keepalivedConf(iface, vip.String(), prefix, m.priority, pass)
+		if out, err := m.r.Run(ctx, `set -e
+sudo install -d -m 0755 /etc/keepalived
+sudo sh -c 'umask 077; cat > /etc/keepalived/keepalived.conf'
+sudo systemctl enable keepalived >/dev/null 2>&1
+sudo systemctl restart keepalived`, []byte(conf)); err != nil {
+			return fmt.Errorf("on %s: %v: %s", m.name, err, strings.TrimSpace(string(out)))
+		}
+	}
+	sa.done("%s follows whichever control plane leads (keepalived on both)", sa.vip)
+	return nil
+}
+
+// vrrpPassword is the shared secret of the two keepalived, made once and
+// kept with the other secrets, so running this again does not split them.
+func vrrpPassword(etcDir string) (string, error) {
+	p := filepath.Join(etcDir, "secrets", "vrrp")
+	if b, err := os.ReadFile(p); err == nil && len(bytes.TrimSpace(b)) > 0 {
+		return string(bytes.TrimSpace(b)), nil
+	}
+	b := make([]byte, 4)
+	if _, err := rand.Read(b); err != nil {
+		return "", err
+	}
+	pass := hex.EncodeToString(b) // keepalived reads eight characters
+	if err := os.WriteFile(p, []byte(pass+"\n"), 0o600); err != nil {
+		return "", err
+	}
+	return pass, nil
+}
+
+func keepalivedConf(iface, vip string, prefix, priority int, pass string) string {
+	return fmt.Sprintf(`# llmgw: the gateway's floating address, on whichever control plane leads.
+vrrp_script llmgw_leader {
+  script "/usr/bin/curl -fsS -o /dev/null -m 2 http://127.0.0.1:8080/leader"
+  interval 2
+  fall 2
+  rise 2
+}
+
+vrrp_instance llmgw {
+  state BACKUP
+  interface %s
+  virtual_router_id 51
+  priority %d
+  advert_int 1
+  authentication {
+    auth_type PASS
+    auth_pass %s
+  }
+  virtual_ipaddress {
+    %s/%d
+  }
+  track_script {
+    llmgw_leader
+  }
+}
+`, iface, priority, pass, vip, prefix)
+}

internal/cli/serverinit.go

@@ -23,6 +23,7 @@ import (
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/account"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/pgha"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
 )
@@ -69,6 +70,13 @@ type serverInit struct {
 	// postgres keeps the store in a PostgreSQL on this machine, installed if
 	// need be, moving whatever SQLite already holds.
 	postgres bool
+	// patroni are user@host of the machines to keep the store on under
+	// Patroni, replicated; this machine is the third etcd vote for two.
+	patroni []string
+	// patroniNodes are the same machines already connected to, which the
+	// setup screen does with the password it was given; when set, patroni
+	// only names them.
+	patroniNodes []pgha.Node
 	// noMirror, noNode and noModel turn off the parts of a first install that
 	// make it serve something: a mirror, this machine as a node, and a small
 	// model on it.
@@ -86,14 +94,19 @@ type serverInit struct {
 	changed  bool
 }
 
-func newServerInitCmd() *cobra.Command {
-	in := serverInit{
+// newDefaultServerInit is init as it sets up a real machine.
+func newDefaultServerInit() serverInit {
+	return serverInit{
 		etcDir:   "/etc/llmgw",
 		stateDir: "/var/lib/llmgw",
 		pkiDir:   defaultPKIDir,
 		unitPath: "/etc/systemd/system/llmgw-server.service",
 		user:     "llmgw",
 	}
+}
+
+func newServerInitCmd() *cobra.Command {
+	in := newDefaultServerInit()
 	var plain bool
 	cmd := &cobra.Command{
 		Use:   "init",
@@ -111,6 +124,12 @@ func newServerInitCmd() *cobra.Command {
 			"installed if there is none, with its own role and a generated password\n" +
 			"kept in /etc/llmgw/secrets. An existing SQLite store is copied into it,\n" +
 			"so accounts and keys survive, and server.yaml is pointed at it.\n\n" +
+			"With --patroni, given two or more machines, the store is PostgreSQL\n" +
+			"replicated across them under Patroni, which promotes a standby when the\n" +
+			"primary is lost. etcd holds its leader lock on those machines — and on\n" +
+			"this one, when they are two, as the third vote. Installed over ssh with\n" +
+			"passwordless sudo, as 'llmgw node add' does; the store already in use,\n" +
+			"SQLite or PostgreSQL, is moved in. Run again, it changes nothing.\n\n" +
 			"Each step is skipped when its result already exists, so it is safe to run\n" +
 			"again: to finish an install that stopped, or with another --host to\n" +
 			"reissue the certificate. It never overwrites server.yaml or the authority.\n\n" +
@@ -144,6 +163,9 @@ func newServerInitCmd() *cobra.Command {
 	f.BoolVar(&in.noModel, "no-model", false, "do not fetch and serve the starter model ("+starterName+")")
 	f.BoolVar(&in.postgres, "postgres", false,
 		"keep the store in PostgreSQL on this machine, installing it and moving an existing SQLite store into it")
+	f.StringArrayVar(&in.patroni, "patroni", nil,
+		"user@host of a machine to keep the store on, replicated under Patroni; give two or more. "+
+			"Installs PostgreSQL, Patroni and etcd there over ssh, and moves the existing store in")
 	return cmd
 }
 

internal/cli/serverinit_patroni.go

@@ -0,0 +1,182 @@
+package cli
+
+import (
+	"context"
+	"fmt"
+	"net"
+	"os"
+	"os/exec"
+	"strings"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/pgha"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// ensurePatroni puts the store in PostgreSQL under Patroni on the machines
+// given with --patroni, and moves whatever store server.yaml names into it:
+// SQLite, or a PostgreSQL on one machine. This machine is the third etcd vote
+// when they are two.
+func (in *serverInit) ensurePatroni(ctx context.Context) (err error) {
+	setup := pgha.Setup{Emit: func(step, detail string, changed bool) {
+		if changed {
+			in.done("%s: %s", step, detail)
+		} else {
+			in.kept("%s: %s", step, detail)
+		}
+	}}
+	setup.DB = in.patroniNodes
+	for _, t := range in.patroni {
+		if len(in.patroniNodes) > 0 {
+			break
+		}
+		user, host, ok := strings.Cut(t, "@")
+		if !ok {
+			user, host = "", t
+		}
+		remote := &onboard.SSH{Addr: host, User: user}
+		out, err := remote.Run(ctx, "hostname && sudo -n true", nil)
+		if err != nil {
+			return fmt.Errorf("%s: cannot log in and use sudo without a password: %v: %s",
+				t, err, strings.TrimSpace(string(out)))
+		}
+		ip, err := ipv4Of(host)
+		if err != nil {
+			return err
+		}
+		setup.DB = append(setup.DB, pgha.Node{
+			Member: pgha.Member{Name: strings.TrimSpace(strings.SplitN(string(out), "\n", 2)[0]), IP: ip},
+			Remote: remote,
+		})
+	}
+	self, err := sourceIPToward(setup.DB[0].IP)
+	if err != nil {
+		return err
+	}
+	name, _ := os.Hostname()
+	setup.Witness = &pgha.Node{Member: pgha.Member{Name: name, IP: self}, Remote: pgha.Local{}}
+	setup.Clients = []string{self}
+
+	dsn, err := setup.Run(ctx)
+	if err != nil {
+		return err
+	}
+
+	cfg, err := config.Load(in.configPath())
+	if err != nil {
+		return err
+	}
+	current := cfg.Auth.Store
+	if current == "" {
+		current = in.storeAddress()
+	}
+	resolved, _ := store.ResolveAddress(current)
+	if strings.TrimSpace(resolved) == dsn {
+		in.kept("store in the Patroni cluster (%s)", in.dsnPath())
+		return nil
+	}
+
+	// Still while it moves, and started again on the old store if the move
+	// fails: a gateway down over a database that did not work out is worse
+	// than one left where it was.
+	stopped, err := in.stopService(ctx)
+	if err != nil {
+		return err
+	}
+	defer func() {
+		if err != nil && stopped {
+			_ = exec.Command("systemctl", "start", "llmgw-server").Run()
+			fmt.Fprintf(in.out, "  ..    llmgw-server started again on its old store\n")
+		}
+	}()
+	if err := in.moveInto(ctx, current, dsn); err != nil {
+		return err
+	}
+
+	// The old address is kept beside the new one: the way back, if the
+	// cluster is not what was wanted.
+	if b, err := os.ReadFile(in.dsnPath()); err == nil && strings.TrimSpace(string(b)) != dsn {
+		if err := os.WriteFile(in.dsnPath()+".pre-patroni", b, 0o600); err != nil {
+			return err
+		}
+		_ = in.chown(in.dsnPath()+".pre-patroni", true)
+	}
+	if err := in.writeDSN(dsn); err != nil {
+		return err
+	}
+	if current != "@"+in.dsnPath() {
+		if err := setAuthStore(in.configPath(), "@"+in.dsnPath()); err != nil {
+			return err
+		}
+	}
+	// Started again by the service step that follows, on the new address.
+	in.done("server.yaml now keeps the store in the Patroni cluster")
+	return nil
+}
+
+// moveInto copies a store into the cluster, unless the cluster already holds
+// one — from a run that copied and stopped before changing the address.
+func (in *serverInit) moveInto(ctx context.Context, from, to string) error {
+	src, err := store.Open(ctx, from)
+	if err != nil {
+		// Nothing there yet: a first install, with nothing to move.
+		in.kept("no store at %s to move", from)
+		return nil
+	}
+	defer src.Close()
+	dst, err := store.Open(ctx, to)
+	if err != nil {
+		return fmt.Errorf("opening the cluster: %w", err)
+	}
+	defer dst.Close()
+	empty, err := dst.Empty(ctx)
+	if err != nil {
+		return err
+	}
+	if !empty {
+		in.kept("the cluster already holds a store; %s is left as it is", from)
+		return nil
+	}
+	res, err := src.CopyTo(ctx, dst)
+	if err != nil {
+		return fmt.Errorf("copying the store into the cluster: %w", err)
+	}
+	var rows int64
+	for _, r := range res {
+		rows += r.Rows
+	}
+	in.done("copied %d rows into the cluster (the old store is kept as it was)", rows)
+	return nil
+}
+
+// sourceIPToward is this machine's address on the route to another: the one
+// the other machine sees it connect from, which is what etcd advertises and
+// what the database lets in. Not the first --host, which names the gateway
+// for its clients and may not resolve yet, or resolve to a floating address.
+func sourceIPToward(ip string) (string, error) {
+	c, err := net.Dial("udp", net.JoinHostPort(ip, "5432"))
+	if err != nil {
+		return "", fmt.Errorf("no route from this machine to %s: %w", ip, err)
+	}
+	defer c.Close()
+	return c.LocalAddr().(*net.UDPAddr).IP.String(), nil
+}
+
+// ipv4Of is the address a host name or address stands for, as the other
+// machines will be told to reach it.
+func ipv4Of(host string) (string, error) {
+	if ip := net.ParseIP(host); ip != nil {
+		return ip.String(), nil
+	}
+	ips, err := net.LookupIP(host)
+	if err != nil {
+		return "", fmt.Errorf("%s does not resolve: %w", host, err)
+	}
+	for _, ip := range ips {
+		if v4 := ip.To4(); v4 != nil {
+			return v4.String(), nil
+		}
+	}
+	return "", fmt.Errorf("%s has no IPv4 address", host)
+}

internal/cli/serverinit_postgres.go

@@ -44,6 +44,12 @@ func (in *serverInit) storeAddress() string {
 // means reissuing every key and being unable to say which was which; and the
 // administrator init created a minute ago would be gone too.
 func (in *serverInit) ensureDatabase(ctx context.Context) (err error) {
+	if len(in.patroni) > 0 || len(in.patroniNodes) > 0 {
+		if in.postgres {
+			return fmt.Errorf("--postgres keeps the store on this machine and --patroni on others: give one")
+		}
+		return in.ensurePatroni(ctx)
+	}
 	if !in.postgres {
 		return nil
 	}

internal/cli/status.go

@@ -2,6 +2,7 @@ package cli
 
 import (
 	"fmt"
+	"strings"
 	"time"
 
 	"github.com/spf13/cobra"
@@ -94,6 +95,31 @@ func printSummary(cmd *cobra.Command, s adminapi.Summary) {
 	fmt.Fprintf(out, "\nMODELS    %d of %d have the replicas they asked for\n",
 		s.Models.Satisfied, s.Models.Total)
 
+	// The store, when it is a replicated cluster: one line, saying who leads
+	// and whether every other member keeps up.
+	switch {
+	case s.StoreError != "":
+		fmt.Fprintf(out, "STORE     cannot read its cluster: %s\n", s.StoreError)
+	case len(s.Store) > 0:
+		var parts []string
+		leader := false
+		for _, m := range s.Store {
+			switch {
+			case m.Role == "leader":
+				leader = true
+				parts = append(parts, m.Name+" leads")
+			case m.State == "streaming" && m.Lag >= 0:
+				parts = append(parts, fmt.Sprintf("%s streaming (%s behind)", m.Name, bytesShort(m.Lag)))
+			default:
+				parts = append(parts, fmt.Sprintf("%s %s", m.Name, m.State))
+			}
+		}
+		if !leader {
+			parts = append([]string{"NO LEADER — writes fail"}, parts...)
+		}
+		fmt.Fprintf(out, "STORE     %s\n", strings.Join(parts, ", "))
+	}
+
 	if len(s.Parked) > 0 {
 		fmt.Fprintln(out, "\nPARKED — failed every time they started; not retried until fixed")
 		for _, p := range s.Parked {
@@ -169,3 +195,16 @@ func unhealthyCell(n int) string {
 	}
 	return fmt.Sprintf("%d", n)
 }
+
+// bytesShort is a byte count as a person reads it: 0 B, 12 KB, 3.4 MB.
+func bytesShort(n int64) string {
+	switch {
+	case n < 1024:
+		return fmt.Sprintf("%d B", n)
+	case n < 1<<20:
+		return fmt.Sprintf("%d KB", n>>10)
+	case n < 1<<30:
+		return fmt.Sprintf("%.1f MB", float64(n)/(1<<20))
+	}
+	return fmt.Sprintf("%.1f GB", float64(n)/(1<<30))
+}

internal/cli/tui_frame.go

@@ -0,0 +1,237 @@
+package cli
+
+import (
+	"strings"
+
+	"github.com/charmbracelet/lipgloss"
+	"github.com/charmbracelet/x/ansi"
+)
+
+// The frame every full-screen llmgw screen is drawn in: a title bar, a
+// sidebar of steps or actions on the left, the content in a bordered panel on
+// the right, the keys along the bottom — and a dialog over it all when one is
+// open. One frame, so the setup and the console read as one program.
+//
+// Keyboard only: these run on serial consoles and IPMI as often as in a
+// desktop terminal.
+
+// navItem is one line of the sidebar.
+type navItem struct {
+	Label string
+	// State is "done", "current", "todo" for steps; "" for menu entries,
+	// where Current marks the one focused.
+	State   string
+	Current bool
+	// Dim greys an entry that cannot be chosen now.
+	Dim bool
+	// Heading is a title between groups of entries, not an entry.
+	Heading bool
+}
+
+type frame struct {
+	Width, Height int
+	Title         string // left of the title bar
+	Where         string // right of the title bar: the machine
+	NavTitle      string
+	Nav           []navItem
+	PanelTitle    string
+	Body          string
+	// Scroll is how many of the body's lines are scrolled off the top.
+	Scroll int
+	Keys   string
+	// Dialog, when set, is drawn over the middle of the screen.
+	Dialog      string
+	DialogTitle string
+}
+
+var (
+	fBar      = lipgloss.NewStyle().Bold(true).Foreground(lipgloss.AdaptiveColor{Light: "#FFFFFF", Dark: "#0B1B1B"}).Background(tuiAccent)
+	fBorder   = lipgloss.NewStyle().Border(lipgloss.RoundedBorder()).BorderForeground(tuiDim)
+	fFocus    = lipgloss.NewStyle().Border(lipgloss.RoundedBorder()).BorderForeground(tuiAccent)
+	fKeys     = lipgloss.NewStyle().Foreground(tuiDim)
+	fSelected = lipgloss.NewStyle().Bold(true).Foreground(tuiAccent)
+	fDialog   = lipgloss.NewStyle().Border(lipgloss.DoubleBorder()).BorderForeground(tuiAccent).Padding(1, 2)
+)
+
+const navWidth = 24
+
+// render draws the whole screen, exactly Width by Height.
+func (f frame) render() string {
+	w, h := max(f.Width, 60), max(f.Height, 16)
+
+	title := " " + f.Title
+	right := f.Where + " "
+	gap := max(1, w-ansi.StringWidth(title)-ansi.StringWidth(right))
+	bar := fBar.Render(title + strings.Repeat(" ", gap) + right)
+
+	// Panels fill what the bars leave: one line each above and below.
+	ph := h - 2
+	inner := ph - 2 // borders
+
+	var nav strings.Builder
+	for _, it := range f.Nav {
+		mark := "  "
+		label := it.Label
+		if it.Heading {
+			nav.WriteString(sDim.Render(label) + "\n")
+			continue
+		}
+		switch {
+		case it.State == "done":
+			mark = sOK.Render("✓ ")
+		case it.State == "current":
+			mark = fSelected.Render("› ")
+			label = fSelected.Render(label)
+		case it.Current:
+			mark = fSelected.Render("› ")
+			label = fSelected.Render(label)
+		case it.Dim || it.State == "todo":
+			label = sDim.Render(label)
+		}
+		nav.WriteString(truncateWidth(mark+label, navWidth-2) + "\n")
+	}
+	left := box(fBorder, f.NavTitle, nav.String(), navWidth, inner)
+
+	pw := w - navWidth - 2 - 2
+	lines := strings.Split(strings.TrimRight(f.Body, "\n"), "\n")
+	scroll := min(max(f.Scroll, 0), max(0, len(lines)-inner))
+	visible := lines[scroll:]
+	if len(visible) > inner {
+		visible = visible[:inner]
+		// Said in the last line, where it is seen.
+		visible[inner-1] = sDim.Render("  ↓ more — PgDn")
+	}
+	for i := range visible {
+		visible[i] = truncateWidth(visible[i], pw)
+	}
+	panelTitle := f.PanelTitle
+	if scroll > 0 {
+		panelTitle += " ↑"
+	}
+	main := box(fFocus, panelTitle, strings.Join(visible, "\n"), pw, inner)
+
+	keys := fKeys.Render(" " + truncateWidth(f.Keys, w-2))
+	screen := bar + "\n" + lipgloss.JoinHorizontal(lipgloss.Top, left, main) + "\n" + keys
+
+	if f.Dialog != "" {
+		dw := min(w-8, max(50, maxLineWidth(f.Dialog)+6))
+		d := fDialog.Width(dw).Render(f.Dialog)
+		if f.DialogTitle != "" {
+			d = titled(d, f.DialogTitle)
+		}
+		screen = overlay(screen, d, w, h)
+	}
+	return screen
+}
+
+// box draws content in a border of the given inner size, with a title set
+// into the top edge.
+func box(style lipgloss.Style, title, content string, width, height int) string {
+	b := style.Width(width).Height(height).Render(content)
+	if title == "" {
+		return b
+	}
+	return titled(b, title)
+}
+
+// titled sets " title " into a box's top border, after its corner.
+func titled(b, title string) string {
+	lines := strings.Split(b, "\n")
+	if len(lines) == 0 {
+		return b
+	}
+	top := lines[0]
+	t := " " + title + " "
+	tw := ansi.StringWidth(t)
+	if ansi.StringWidth(top) < tw+4 {
+		return b
+	}
+	lines[0] = ansi.Truncate(top, 2, "") + sTitle.Render(t) + ansi.TruncateLeft(top, 2+tw, "")
+	return strings.Join(lines, "\n")
+}
+
+// overlay draws top over the middle of base, both w by h.
+func overlay(base, top string, w, h int) string {
+	bl := strings.Split(base, "\n")
+	tl := strings.Split(top, "\n")
+	tw := maxLineWidth(top)
+	x := max(0, (w-tw)/2)
+	y := max(0, (h-len(tl))/2)
+	for i, l := range tl {
+		row := y + i
+		if row >= len(bl) {
+			break
+		}
+		under := bl[row]
+		left := ansi.Truncate(under, x, "")
+		if pad := x - ansi.StringWidth(left); pad > 0 {
+			left += strings.Repeat(" ", pad)
+		}
+		right := ansi.TruncateLeft(under, x+ansi.StringWidth(l), "")
+		bl[row] = left + l + right
+	}
+	return strings.Join(bl, "\n")
+}
+
+func maxLineWidth(s string) int {
+	n := 0
+	for _, l := range strings.Split(s, "\n") {
+		n = max(n, ansi.StringWidth(l))
+	}
+	return n
+}
+
+// truncateWidth cuts a line, colours and all, to w cells.
+func truncateWidth(s string, w int) string {
+	if ansi.StringWidth(s) <= w {
+		return s
+	}
+	return ansi.Truncate(s, max(w-1, 0), "…")
+}
+
+// table draws rows under a header in columns as wide as they need, cut to
+// width; the selected row is shown reversed when the panel has the focus.
+func table(headers []string, rows [][]string, selected int, width int, focused bool) string {
+	widths := make([]int, len(headers))
+	for i, h := range headers {
+		widths[i] = ansi.StringWidth(h)
+	}
+	for _, r := range rows {
+		for i := range headers {
+			if i < len(r) {
+				widths[i] = min(max(widths[i], ansi.StringWidth(r[i])), 44)
+			}
+		}
+	}
+	line := func(cells []string) string {
+		var b strings.Builder
+		for i := range headers {
+			c := ""
+			if i < len(cells) {
+				c = truncateWidth(cells[i], widths[i])
+			}
+			b.WriteString(c)
+			if i < len(headers)-1 {
+				b.WriteString(strings.Repeat(" ", widths[i]-ansi.StringWidth(c)+2))
+			}
+		}
+		return truncateWidth(b.String(), width)
+	}
+	var b strings.Builder
+	b.WriteString("  " + lipgloss.NewStyle().Bold(true).Foreground(tuiDim).Render(line(headers)) + "\n")
+	for i, r := range rows {
+		l := line(r)
+		switch {
+		case i == selected && focused:
+			pad := max(0, width-ansi.StringWidth(ansi.Strip(l)))
+			l = fBar.Render(ansi.Strip(l) + strings.Repeat(" ", pad))
+		case i == selected:
+			l = lipgloss.NewStyle().Bold(true).Render(l)
+		}
+		b.WriteString("  " + l + "\n")
+	}
+	if len(rows) == 0 {
+		b.WriteString("  " + sDim.Render("nothing yet") + "\n")
+	}
+	return b.String()
+}

internal/cli/tui_home.go

@@ -0,0 +1,48 @@
+package cli
+
+import (
+	"errors"
+	"fmt"
+	"os"
+	"runtime"
+
+	"github.com/mattn/go-isatty"
+	"github.com/spf13/cobra"
+)
+
+// A bare `llmgw` in a terminal: the setup screens on a machine with nothing
+// installed, the console on a control plane. Anywhere else — a pipe, a
+// script, another operating system — it prints its help, as it always did.
+
+func interactive(cmd *cobra.Command) bool {
+	return isTerminal(cmd.OutOrStdout()) && isatty.IsTerminal(os.Stdin.Fd())
+}
+
+func runHome(cmd *cobra.Command) error {
+	out := cmd.OutOrStdout()
+	if runtime.GOOS != "linux" {
+		fmt.Fprintln(out, "The setup and console screens run on the Linux machine they manage.")
+		fmt.Fprintln(out)
+		return cmd.Help()
+	}
+	if os.Geteuid() != 0 {
+		return errors.New("run it as root — sudo llmgw — since it installs software and holds the certificate authority")
+	}
+	switch {
+	// Installed is the service in place, not only its configuration: a setup
+	// that stopped halfway has written server.yaml and nothing that runs, and
+	// the setup screens are what finish it.
+	case exists("/etc/llmgw/server.yaml") && exists("/etc/systemd/system/llmgw-server.service"):
+		return runConsole(cmd.Context(), out, "/etc/llmgw/server.yaml")
+	case exists("/etc/llmgw/agent.yaml"):
+		// A node: its own settings. The cluster is watched from the
+		// control plane.
+		return runConsole(cmd.Context(), out, "/etc/llmgw/agent.yaml")
+	}
+	return runSetupTUI(cmd.Context(), out)
+}
+
+func exists(p string) bool {
+	_, err := os.Stat(p)
+	return err == nil
+}

internal/cli/tui_init.go

@@ -77,10 +77,14 @@ type initModel struct {
 	next     []string // the closing "Next:" text
 	spin     spinnerModel
 	width    int
-	done     bool
-	err      error
-	quit     bool
-	start    func()
+	height   int
+	// framed draws it in the full-screen frame; off when another screen
+	// shows it in its own panel.
+	framed bool
+	done   bool
+	err    error
+	quit   bool
+	start  func()
 }
 
 func newInitModel(in *serverInit, found []string, askForm bool) initModel {
@@ -118,7 +122,10 @@ type startInitMsg struct{}
 func (m initModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
 	switch msg := msg.(type) {
 	case tea.WindowSizeMsg:
-		m.width = msg.Width
+		m.width, m.height = msg.Width, msg.Height
+		if m.framed {
+			m.width = msg.Width - navWidth - 6
+		}
 	case spinTickMsg:
 		var cmd tea.Cmd
 		m.spin, cmd = m.spin.Update(msg)
@@ -132,12 +139,18 @@ func (m initModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
 		m.addLine(string(msg))
 	case initDoneMsg:
 		m.done, m.err = true, msg.err
+		if m.framed {
+			return m, nil // left on screen until a key: it is gone once closed
+		}
 		return m, tea.Quit
 	case tea.KeyMsg:
 		if msg.Type == tea.KeyCtrlC {
 			m.quit = true
 			return m, tea.Quit
 		}
+		if m.done && m.framed && (msg.Type == tea.KeyEnter || msg.String() == "q") {
+			return m, tea.Quit
+		}
 		if m.form {
 			return m.formKey(msg)
 		}
@@ -240,6 +253,30 @@ func (m *initModel) moveFocus(d int) {
 }
 
 func (m initModel) View() string {
+	if !m.framed {
+		return m.plainView()
+	}
+	body := m.plainView()
+	if i := strings.Index(body, "\n\n"); i >= 0 {
+		body = body[i+1:]
+	}
+	steps := []navItem{{Label: "Start", State: "done"}, {Label: "Settings", State: "current"}, {Label: "Install", State: "todo"}}
+	keys := "Tab/↑↓ move · Space tick · Enter on Set up · Ctrl+C leave"
+	if !m.form {
+		steps[1].State, steps[2].State = "done", "current"
+		keys = "Ctrl+C stop"
+		if m.done {
+			steps[2].State = "done"
+			keys = "Enter close"
+		}
+	}
+	host, _ := os.Hostname()
+	return frame{Width: m.width + navWidth + 6, Height: m.height, Title: "llmgw setup", Where: host,
+		NavTitle: "Steps", Nav: steps, PanelTitle: "This machine as a control plane", Body: body, Keys: keys}.render()
+}
+
+// plainView is the screen as text, for a panel or a terminal that is not one.
+func (m initModel) plainView() string {
 	var b strings.Builder
 	b.WriteString(sTitle.Render("llmgw · setting up this machine as a control plane") + "\n\n")
 	if m.form {
@@ -338,7 +375,9 @@ func (m initModel) formView() string {
 	if m.formError != "" {
 		b.WriteString("\n" + sBad.Render(m.formError) + "\n")
 	}
-	b.WriteString("\n" + sDim.Render("tab or ↑↓ to move · space to tick · enter on Set up to start · ctrl+c to leave") + "\n")
+	if !m.framed { // the frame's key bar says it
+		b.WriteString("\n" + sDim.Render("tab or ↑↓ to move · space to tick · enter on Set up to start · ctrl+c to leave") + "\n")
+	}
 	return b.String()
 }
 
@@ -350,6 +389,7 @@ func (in *serverInit) runInitTUI(ctx context.Context, out io.Writer) error {
 		found = append(found, ip.String())
 	}
 	m := newInitModel(in, found, in.admin == "")
+	m.framed = true
 	var p *tea.Program
 
 	var (
@@ -373,7 +413,7 @@ func (in *serverInit) runInitTUI(ctx context.Context, out io.Writer) error {
 			p.Send(initDoneMsg{err})
 		}()
 	}
-	p = tea.NewProgram(m, tea.WithOutput(out))
+	p = tea.NewProgram(m, tea.WithOutput(out), tea.WithAltScreen())
 	final, err := p.Run()
 	if err != nil {
 		return err

internal/cli/tui_node.go

@@ -68,6 +68,7 @@ type nodeModel struct {
 	spin    spinnerModel
 	cancel  context.CancelFunc
 	width   int
+	height  int
 	done    bool
 	err     error
 	node    string
@@ -91,7 +92,7 @@ func (m nodeModel) Init() tea.Cmd { return m.spin.Tick }
 func (m nodeModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
 	switch msg := msg.(type) {
 	case tea.WindowSizeMsg:
-		m.width = msg.Width
+		m.width, m.height = msg.Width, msg.Height
 	case spinTickMsg:
 		var cmd tea.Cmd
 		m.spin, cmd = m.spin.Update(msg)
@@ -131,9 +132,13 @@ func (m nodeModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
 	case askContinueMsg:
 		m.confirm = &msg
 	case finishMsg:
+		// Left on the screen until a key: the frame is gone once it closes.
 		m.done, m.err, m.node, m.summary = true, msg.err, msg.node, msg.summary
-		return m, tea.Quit
+		return m, nil
 	case tea.KeyMsg:
+		if m.done && (msg.Type == tea.KeyEnter || msg.String() == "q" || msg.Type == tea.KeyEsc) {
+			return m, tea.Quit
+		}
 		return m.key(msg)
 	}
 	return m, nil
@@ -197,10 +202,52 @@ func (m *nodeModel) move(d int) {
 	m.inputs[m.focus].Focus()
 }
 
+// View draws the onboarding in the frame: its steps down the side, what each
+// did in the panel, a question as a dialog.
 func (m nodeModel) View() string {
+	var nav []navItem
+	current := false
+	for _, step := range onboard.Steps {
+		st, ok := m.steps[step]
+		state := "todo"
+		switch {
+		case ok && st.level == onboard.Running && !m.done && !current:
+			state, current = "current", true
+		case ok && st.level != onboard.Running:
+			state = "done"
+		}
+		nav = append(nav, navItem{Label: step, State: state})
+	}
+	keys := "Ctrl+C stop"
+	if m.done {
+		keys = "Enter close"
+	}
+	host, _ := os.Hostname()
+	f := frame{
+		Width: m.width, Height: m.height, Title: "llmgw · add a node", Where: host,
+		NavTitle: "Steps", Nav: nav, PanelTitle: m.title, Body: "\n" + m.content(), Keys: keys,
+	}
+	switch {
+	case m.fabrics != nil:
+		var q strings.Builder
+		q.WriteString("These adapters are up. Which fabric is each group on?\n")
+		q.WriteString(sDim.Render("Give each the name it has on every other node attached to it.\nLeave a name empty to leave that group out.") + "\n\n")
+		for i, fb := range m.fabrics.suggested {
+			fmt.Fprintf(&q, "  %s  %s\n", m.inputs[i].View(), sDim.Render(strings.Join(fb.HCAs, ", ")))
+		}
+		q.WriteString("\n" + sDim.Render("Tab next · Enter use these · Esc none"))
+		f.DialogTitle, f.Dialog = "Fabrics", q.String()
+	case m.confirm != nil:
+		f.DialogTitle = "Problems found"
+		f.Dialog = fmt.Sprintf("doctor found %d problem(s) on the machine.\nAdd the node anyway?\n\n%s", m.confirm.fails, sDim.Render("Y yes · N no"))
+	}
+	return f.render()
+}
+
+// content is the onboarding's steps and notes, as the panel shows them.
+func (m nodeModel) content() string {
 	var b strings.Builder
-	b.WriteString(sTitle.Render(m.title) + "\n\n")
-	detailWidth := max(20, m.width-22)
+	detailWidth := max(20, m.width-navWidth-26)
 	for _, step := range onboard.Steps {
 		st, ok := m.steps[step]
 		var mark string
@@ -238,26 +285,12 @@ func (m nodeModel) View() string {
 	}
 
 	switch {
-	case m.fabrics != nil:
-		var q strings.Builder
-		q.WriteString("These adapters are up. Which fabric is each group on?\n")
-		q.WriteString(sDim.Render("Give each the name it has on every other node attached to it.\nLeave a name empty to leave that group out.") + "\n\n")
-		for i, f := range m.fabrics.suggested {
-			fmt.Fprintf(&q, "  %s  %s\n", m.inputs[i].View(), sDim.Render(strings.Join(f.HCAs, ", ")))
-		}
-		q.WriteString("\n" + sDim.Render("tab next · enter use these · esc none"))
-		b.WriteString("\n" + sBox.Render(q.String()) + "\n")
-	case m.confirm != nil:
-		q := fmt.Sprintf("doctor found %d problem(s) above.\nAdd the node anyway? %s", m.confirm.fails, sDim.Render("y / N"))
-		b.WriteString("\n" + sBox.BorderForeground(tuiWarn).Render(q) + "\n")
 	case m.done && m.err != nil:
 		b.WriteString("\n" + sBad.Render("✗ ") + wrap(m.err.Error(), m.width-4, "  ") + "\n")
 	case m.done:
 		b.WriteString("\n" + sOK.Render("✓ ") + m.summary + "\n")
 	case m.quitAsked:
 		b.WriteString("\n" + sDim.Render("stopped") + "\n")
-	default:
-		b.WriteString("\n" + sDim.Render("ctrl+c to stop") + "\n")
 	}
 	return b.String()
 }
@@ -298,7 +331,7 @@ func (na *nodeAdd) runTUI(ctx context.Context, o onboard.Options, title string)
 	ctx, cancel := context.WithCancel(ctx)
 	defer cancel()
 	m := newNodeModel(title, cancel)
-	p := tea.NewProgram(m)
+	p := tea.NewProgram(m, tea.WithAltScreen())
 
 	emit := func(e onboard.Event) { p.Send(evMsg(e)) }
 	o.AskFabrics = na.rememberFabrics(func(s []onboard.Fabric) []onboard.Fabric {
@@ -343,6 +376,8 @@ func (na *nodeAdd) runTUI(ctx context.Context, o onboard.Options, title string)
 		return "", err
 	}
 	fm := final.(nodeModel)
+	// The screen is gone; what it ended with is said again where it stays.
+	fmt.Println(fm.content())
 	if fm.quitAsked && !fm.done {
 		cancel()
 		return fm.node, fmt.Errorf("stopped; the node may be half set up, and running this again finishes it")

internal/cli/tui_setup.go

@@ -0,0 +1,949 @@
+package cli
+
+import (
+	"context"
+	"fmt"
+	"io"
+	"net/mail"
+	"os"
+	"strings"
+	"sync"
+	"time"
+
+	tea "github.com/charmbracelet/bubbletea"
+	"github.com/charmbracelet/lipgloss"
+
+	joinpkg "gitea.mynodes.xyz/ilker/llmgw/internal/join"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/pgha"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/preflight"
+)
+
+// The setup screens: what a bare `llmgw` shows on a machine with nothing
+// installed. One machine, or three for high availability — this one as the
+// control plane and etcd's third vote, two more for the store under Patroni —
+// with every machine checked before anything is installed on any of them.
+//
+// The other machines are reached with Go's own ssh rather than the system's,
+// so a password can be typed here when there is no key; it is held in memory
+// for the run and written nowhere. A sudo password, likewise, is used once to
+// let the user sudo for the length of the install, and that is taken back
+// when it finishes.
+
+type setupPage int
+
+const (
+	pWelcome setupPage = iota
+	pAdmin
+	pMachines
+	// pWait shows the codes for machines joining rather than logged into,
+	// until they have all dialled in.
+	pWait
+	pCheck
+	pPlan
+	pRun
+)
+
+// How a database machine is logged into.
+const (
+	authAgent = iota
+	authKey
+	authPassword
+	// authJoin is no login at all: the machine runs `llmgw join` with a code
+	// shown here and dials out to this one.
+	authJoin
+	authCount
+)
+
+var authNames = [authCount]string{"ssh agent", "key file", "password", "join code"}
+
+// Fields of one database machine, in order.
+const (
+	mfAddr = iota
+	mfUser
+	mfAuth
+	mfSecret
+	mfSudo
+	mfCount
+)
+
+type dbMachine struct {
+	addr, user, secret, sudo textField
+	auth                     int
+	// For a machine joining: its code, and it once it has dialled in.
+	code   string
+	joined *joinpkg.Remote
+	arrive <-chan *joinpkg.Remote
+}
+
+// visible reports whether a field is shown, and so can be moved to: a joining
+// machine is asked nothing but how it joins.
+func (d dbMachine) visible(f int) bool {
+	switch {
+	case d.auth == authJoin:
+		return f == mfAuth
+	case f == mfSecret:
+		return d.auth != authAgent
+	}
+	return true
+}
+
+// dbConn is a database machine as it is reached, whichever way.
+type dbConn struct {
+	remote onboard.Remote
+	addr   string
+	ssh    *onboard.GoSSH
+	joined *joinpkg.Remote
+}
+
+func newDBMachine(n int) dbMachine {
+	f := func(ph string, mask bool) textField {
+		t := newTextField()
+		t.Placeholder, t.Mask, t.CharLimit, t.Width = ph, mask, 200, 34
+		return t
+	}
+	m := dbMachine{
+		addr:   f(fmt.Sprintf("10.0.0.%d or db%d.internal", 10+n, n), false),
+		user:   f("ubuntu", false),
+		secret: f("", true),
+		sudo:   f("only if sudo asks for one", true),
+	}
+	if os.Getenv("SSH_AUTH_SOCK") == "" {
+		m.auth = authKey
+	}
+	return m
+}
+
+// field is the text field at a machine's position, nil for the selector.
+func (d *dbMachine) field(i int) *textField {
+	switch i {
+	case mfAddr:
+		return &d.addr
+	case mfUser:
+		return &d.user
+	case mfSecret:
+		return &d.secret
+	case mfSudo:
+		return &d.sudo
+	}
+	return nil
+}
+
+// Messages from the work behind the screens.
+type (
+	checkDoneMsg struct{ rep preflight.Report }
+	joinedMsg    struct {
+		i int
+		r *joinpkg.Remote
+	}
+)
+
+// progRef lets work started from Update send to the program it runs in.
+type progRef struct{ p *tea.Program }
+
+type setupModel struct {
+	ref *progRef
+
+	page   setupPage
+	host   string
+	ips    []string
+	choice int // 0 one machine, 1 high availability
+
+	admin, names textField
+	afocus       int // admin, names, next
+
+	db     [2]dbMachine
+	mfocus int // over both machines' fields, then the check button
+
+	conns    []dbConn
+	listener *joinpkg.Listener
+	report   preflight.Report
+	checking bool
+
+	in       *serverInit
+	run      initModel
+	finished bool
+	log      *[]string
+	logMu    *sync.Mutex
+
+	spin    spinnerModel
+	formErr string
+	width   int
+	height  int
+	// scroll is how far the panel is scrolled, reset with each page.
+	scroll int
+	quit   bool
+	// single is the choice of one machine: the screen closes and init's own
+	// form takes over.
+	single bool
+}
+
+func newSetupModel(host string, ips []string) setupModel {
+	admin := newTextField()
+	admin.Placeholder, admin.CharLimit, admin.Width = "you@example.com", 200, 40
+	names := newTextField()
+	names.Placeholder, names.CharLimit, names.Width = "llm.example.com, 203.0.113.10", 400, 40
+	return setupModel{
+		ref: &progRef{}, host: host, ips: ips, admin: admin, names: names,
+		db: [2]dbMachine{newDBMachine(1), newDBMachine(2)}, spin: newSpinner(), width: 100, height: 30,
+		log: &[]string{}, logMu: &sync.Mutex{},
+	}
+}
+
+func (m setupModel) Init() tea.Cmd { return m.spin.Tick }
+
+func (m setupModel) thisAddr() string {
+	if len(m.ips) > 0 {
+		return m.ips[0]
+	}
+	return m.host
+}
+
+func (m setupModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
+	switch msg := msg.(type) {
+	case tea.WindowSizeMsg:
+		m.width, m.height = msg.Width, msg.Height
+		m.run.width = msg.Width - navWidth - 6
+	case spinTickMsg:
+		var cmd tea.Cmd
+		m.spin, cmd = m.spin.Update(msg)
+		m.run.spin = m.spin
+		return m, cmd
+	case checkDoneMsg:
+		m.checking, m.report = false, msg.rep
+	case joinedMsg:
+		m.db[msg.i].joined = msg.r
+		if m.page == pWait && m.allJoined() {
+			m.page = pCheck
+			cmd := m.startCheck()
+			return m, cmd
+		}
+	case initLineMsg:
+		m.run.addLine(string(msg))
+	case initDoneMsg:
+		m.run.done, m.run.err, m.finished = true, msg.err, true
+	case tea.KeyMsg:
+		if msg.Type == tea.KeyCtrlC {
+			m.quit = true
+			return m, tea.Quit
+		}
+		before := m.page
+		next, cmd := m.key(msg)
+		if nm := next.(setupModel); nm.page != before {
+			// A new page starts at its top.
+			nm.scroll = 0
+			return nm, cmd
+		}
+		return next, cmd
+	}
+	return m, nil
+}
+
+func (m setupModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) {
+	// A dialog open takes the next key, whatever it is, to close it.
+	if m.formErr != "" {
+		m.formErr = ""
+		return m, nil
+	}
+	switch k.Type {
+	case tea.KeyPgDown:
+		m.scroll += max(1, m.height-8)
+		return m, nil
+	case tea.KeyPgUp:
+		m.scroll = max(0, m.scroll-max(1, m.height-8))
+		return m, nil
+	}
+	if k.Type == tea.KeyEsc && m.page > pWelcome && m.page < pRun && !m.checking {
+		m.page--
+		if m.page == pWait {
+			m.page = pMachines
+		}
+		m.formErr = ""
+		return m, nil
+	}
+	switch m.page {
+	case pWelcome:
+		switch k.String() {
+		case "up", "down", "tab", "k", "j":
+			m.choice = 1 - m.choice
+		case "enter":
+			if m.choice == 0 {
+				m.single = true
+				return m, tea.Quit
+			}
+			m.page = pAdmin
+			m.afocus = 0
+			m.admin.Focus()
+		}
+	case pAdmin:
+		return m.adminKey(k)
+	case pMachines:
+		return m.machinesKey(k)
+	case pCheck:
+		switch k.String() {
+		case "r":
+			if !m.checking {
+				cmd := m.startCheck()
+				return m, cmd
+			}
+		case "enter":
+			if !m.checking && m.report.Worst() < preflight.Fail {
+				m.page = pPlan
+			}
+		}
+	case pPlan:
+		if k.String() == "enter" {
+			m.page = pRun
+			cmd := m.startRun()
+			return m, cmd
+		}
+	case pRun:
+		if m.finished && (k.String() == "q" || k.String() == "enter") {
+			return m, tea.Quit
+		}
+	}
+	return m, nil
+}
+
+func (m setupModel) adminKey(k tea.KeyMsg) (tea.Model, tea.Cmd) {
+	move := func(d int) {
+		m.afocus = (m.afocus + d + 3) % 3
+		m.admin.Blur()
+		m.names.Blur()
+		switch m.afocus {
+		case 0:
+			m.admin.Focus()
+		case 1:
+			m.names.Focus()
+		}
+	}
+	switch k.Type {
+	case tea.KeyTab, tea.KeyDown:
+		move(1)
+		return m, nil
+	case tea.KeyShiftTab, tea.KeyUp:
+		move(-1)
+		return m, nil
+	case tea.KeyEnter:
+		if m.afocus < 2 {
+			move(1)
+			return m, nil
+		}
+		if _, err := mail.ParseAddress(strings.TrimSpace(m.admin.Value())); err != nil {
+			m.formErr = "the administrator's email is needed, and that is not one"
+			m.afocus = 0
+			move(0)
+			return m, nil
+		}
+		m.formErr = ""
+		m.page, m.mfocus = pMachines, 0
+		m.db[0].addr.Focus()
+		return m, nil
+	}
+	var cmd tea.Cmd
+	switch m.afocus {
+	case 0:
+		m.admin, cmd = m.admin.Update(k)
+	case 1:
+		m.names, cmd = m.names.Update(k)
+	}
+	return m, cmd
+}
+
+func (m *setupModel) focusMachine() {
+	for i := range m.db {
+		for f := 0; f < mfCount; f++ {
+			if t := m.db[i].field(f); t != nil {
+				t.Blur()
+			}
+		}
+	}
+	if m.mfocus < 2*mfCount {
+		if t := m.db[m.mfocus/mfCount].field(m.mfocus % mfCount); t != nil {
+			t.Focus()
+		}
+	}
+}
+
+func (m setupModel) machinesKey(k tea.KeyMsg) (tea.Model, tea.Cmd) {
+	n := 2*mfCount + 1
+	switch k.Type {
+	case tea.KeyTab, tea.KeyDown:
+		m.step(1, n)
+		return m, nil
+	case tea.KeyShiftTab, tea.KeyUp:
+		m.step(-1, n)
+		return m, nil
+	case tea.KeyEnter:
+		if m.mfocus < 2*mfCount {
+			m.step(1, n)
+			return m, nil
+		}
+		if err := m.validateMachines(); err != "" {
+			m.formErr = err
+			return m, nil
+		}
+		m.formErr = ""
+		if cmd, waiting := m.startJoins(); waiting {
+			m.page = pWait
+			return m, cmd
+		}
+		m.page = pCheck
+		cmd := m.startCheck()
+		return m, cmd
+	}
+	if m.mfocus == 2*mfCount {
+		return m, nil
+	}
+	d := &m.db[m.mfocus/mfCount]
+	if m.mfocus%mfCount == mfAuth {
+		switch k.String() {
+		case "left", "h":
+			d.auth = (d.auth + authCount - 1) % authCount
+		case "right", "l", " ":
+			d.auth = (d.auth + 1) % authCount
+		}
+		if d.auth != authJoin && d.joined != nil {
+			_ = d.joined.Close()
+			d.joined, d.code, d.arrive = nil, "", nil
+		}
+		d.secret.SetValue("")
+		d.secret.Mask = d.auth == authPassword
+		d.secret.Placeholder = map[int]string{authKey: "/root/.ssh/id_ed25519", authPassword: "the user's ssh password"}[d.auth]
+		return m, nil
+	}
+	var cmd tea.Cmd
+	t := d.field(m.mfocus % mfCount)
+	*t, cmd = t.Update(k)
+	return m, cmd
+}
+
+// step moves the focus over the fields that are shown.
+func (m *setupModel) step(d, n int) {
+	for range n {
+		m.mfocus = (m.mfocus + d + n) % n
+		if m.mfocus == 2*mfCount || m.db[m.mfocus/mfCount].visible(m.mfocus%mfCount) {
+			break
+		}
+	}
+	m.focusMachine()
+}
+
+func (m setupModel) allJoined() bool {
+	for _, d := range m.db {
+		if d.auth == authJoin && d.joined == nil {
+			return false
+		}
+	}
+	return true
+}
+
+// startJoins listens for the machines that join with a code, makes their
+// codes, and waits for them; it says whether there is anything to wait for.
+func (m *setupModel) startJoins() (tea.Cmd, bool) {
+	var cmds []tea.Cmd
+	for i := range m.db {
+		d := &m.db[i]
+		if d.auth != authJoin || d.joined != nil {
+			continue
+		}
+		if m.listener == nil {
+			l, err := joinpkg.Listen(fmt.Sprintf(":%d", joinpkg.DefaultPort),
+				fmt.Sprintf("%s:%d", m.thisAddr(), joinpkg.DefaultPort), time.Hour)
+			if err != nil {
+				m.formErr = err.Error()
+				return nil, false
+			}
+			m.listener = l
+		}
+		if d.arrive == nil {
+			c, ch := m.listener.Code("database")
+			d.code, d.arrive = c.String(), ch
+		}
+		ch, i := d.arrive, i
+		cmds = append(cmds, func() tea.Msg { return joinedMsg{i, <-ch} })
+	}
+	return tea.Batch(cmds...), len(cmds) > 0
+}
+
+func (m setupModel) validateMachines() string {
+	seen := map[string]bool{m.thisAddr(): true, m.host: true}
+	for i, d := range m.db {
+		if d.auth == authJoin {
+			continue
+		}
+		addr := strings.TrimSpace(d.addr.Value())
+		switch {
+		case addr == "":
+			return fmt.Sprintf("database machine %d has no address", i+1)
+		case seen[addr]:
+			return fmt.Sprintf("%s is given twice, or is this machine: the three must be different machines", addr)
+		case strings.TrimSpace(d.user.Value()) == "":
+			return fmt.Sprintf("database machine %d has no user to log in as", i+1)
+		case d.auth != authAgent && d.secret.Value() == "":
+			return fmt.Sprintf("database machine %d needs its %s", i+1, authNames[d.auth])
+		}
+		seen[addr] = true
+	}
+	return ""
+}
+
+// startCheck connects to the database machines and runs the checks on all
+// three, this one included.
+func (m *setupModel) startCheck() tea.Cmd {
+	for _, c := range m.conns {
+		if c.ssh != nil {
+			_ = c.ssh.Close()
+		}
+	}
+	m.conns = nil
+	machines := []preflight.Machine{{
+		Name: m.host + " (this machine)", Addr: m.thisAddr(), Local: true, Remote: pgha.Local{},
+		Roles: []preflight.Role{preflight.RoleControl, preflight.RoleWitness},
+	}}
+	for _, d := range m.db {
+		if d.auth == authJoin {
+			m.conns = append(m.conns, dbConn{remote: d.joined, addr: d.joined.Addr, joined: d.joined})
+			machines = append(machines, preflight.Machine{Name: d.joined.Hostname + " (" + d.joined.Addr + ", joined)",
+				Addr: d.joined.Addr, Roles: []preflight.Role{preflight.RoleDatabase}, Remote: d.joined})
+			continue
+		}
+		g := &onboard.GoSSH{
+			Addr: strings.TrimSpace(d.addr.Value()), User: strings.TrimSpace(d.user.Value()),
+			UseAgent: d.auth == authAgent, SudoPassword: d.sudo.Value(),
+		}
+		switch d.auth {
+		case authKey:
+			g.KeyFile = strings.TrimSpace(d.secret.Value())
+		case authPassword:
+			g.Password = d.secret.Value()
+		}
+		m.conns = append(m.conns, dbConn{remote: g, addr: g.Addr, ssh: g})
+		machines = append(machines, preflight.Machine{Name: g.Addr, Addr: g.Addr,
+			Roles: []preflight.Role{preflight.RoleDatabase}, Remote: g})
+	}
+	m.checking = true
+	m.report = preflight.Report{}
+	conns := m.conns
+	return func() tea.Msg {
+		rep := preflight.Run(context.Background(), machines)
+		// A sudo that asks for a password is fine with one given: the
+		// install grants itself sudo with it.
+		for i := range rep.Findings {
+			f := &rep.Findings[i]
+			if f.Check == "sudo" && f.Level == preflight.Warn {
+				for _, c := range conns {
+					g := c.ssh
+					switch {
+					case g == nil || g.Addr != f.Machine:
+					case g.SudoPassword == "":
+						f.Level = preflight.Fail
+						f.Fix = "give this user's sudo password on the previous screen"
+					default:
+						f.Level = preflight.OK
+						f.Detail = "with the password given"
+						f.Fix = ""
+					}
+				}
+			}
+		}
+		return checkDoneMsg{rep}
+	}
+}
+
+// targets are the database machines as user@address, for the command line.
+func (m setupModel) targets() []string {
+	var out []string
+	for _, d := range m.db {
+		if d.joined != nil {
+			out = append(out, "root@"+d.joined.Addr)
+			continue
+		}
+		out = append(out, strings.TrimSpace(d.user.Value())+"@"+strings.TrimSpace(d.addr.Value()))
+	}
+	return out
+}
+
+func (m setupModel) namesList() []string {
+	var out []string
+	for _, h := range strings.Split(m.names.Value(), ",") {
+		if h = strings.TrimSpace(h); h != "" {
+			out = append(out, h)
+		}
+	}
+	return out
+}
+
+// command is what this setup is on the command line, for doing it again
+// from a script — with keys in place, since a password cannot be given there.
+func (m setupModel) command() string {
+	parts := []string{"sudo --preserve-env=SSH_AUTH_SOCK llmgw server init",
+		"--admin " + strings.TrimSpace(m.admin.Value())}
+	for _, h := range m.namesList() {
+		parts = append(parts, "--host "+h)
+	}
+	for _, t := range m.targets() {
+		parts = append(parts, "--patroni "+t)
+	}
+	parts = append(parts, "--no-node", "--no-model")
+	return strings.Join(parts, " \\\n    ")
+}
+
+// startRun installs: sudo granted where it asks for a password, the control
+// plane here, the store on the other two, then sudo taken back.
+func (m *setupModel) startRun() tea.Cmd {
+	in := newDefaultServerInit()
+	in.admin = strings.TrimSpace(m.admin.Value())
+	in.hosts = m.namesList()
+	in.patroni = m.targets()
+	// The control plane of a replicated setup is not where models run;
+	// nodes are added once it is up.
+	in.noNode, in.noModel = true, true
+	m.in = &in
+	m.run = newInitModel(&in, nil, false)
+	m.run.running, m.run.width, m.run.spin = true, m.width, m.spin
+	ref, conns, logp, mu, listener := m.ref, m.conns, m.log, m.logMu, m.listener
+	return func() tea.Msg {
+		send := func(l string) {
+			mu.Lock()
+			*logp = append(*logp, l)
+			mu.Unlock()
+			ref.p.Send(initLineMsg(l))
+		}
+		in.out = &lineWriter{send: send}
+		err := func() error {
+			ctx := context.Background()
+			for _, c := range conns {
+				if c.ssh != nil {
+					fmt.Fprintf(in.out, "  ..    sudo on %s\n", c.addr)
+					if err := c.ssh.PrepareSudo(ctx); err != nil {
+						return err
+					}
+				}
+				out, err := c.remote.Run(ctx, "hostname", nil)
+				if err != nil {
+					return fmt.Errorf("%s: %v", c.addr, err)
+				}
+				ip, err := ipv4Of(c.addr)
+				if err != nil {
+					return err
+				}
+				in.patroniNodes = append(in.patroniNodes, pgha.Node{
+					Member: pgha.Member{Name: strings.TrimSpace(string(out)), IP: ip}, Remote: c.remote})
+			}
+			return in.run(ctx)
+		}()
+		// Whatever happened: a sudo grant taken back, a joined machine's
+		// `llmgw join` told it is done, the port for joining closed.
+		for _, c := range conns {
+			if c.ssh != nil {
+				_ = c.ssh.Close()
+			}
+			if c.joined != nil {
+				_ = c.joined.Close()
+			}
+		}
+		if listener != nil {
+			_ = listener.Close()
+		}
+		return initDoneMsg{err}
+	}
+}
+
+// setupSteps are the sidebar's steps, by page.
+var setupSteps = []struct {
+	label string
+	page  setupPage
+}{
+	{"Start", pWelcome}, {"Administrator", pAdmin}, {"Machines", pMachines},
+	{"Checks", pCheck}, {"Plan", pPlan}, {"Install", pRun},
+}
+
+func (m setupModel) View() string {
+	at := m.page
+	if at == pWait {
+		at = pMachines
+	}
+	var nav []navItem
+	for _, st := range setupSteps {
+		if m.choice == 0 && st.page > pWelcome {
+			break
+		}
+		state := "todo"
+		switch {
+		case st.page < at || (st.page == pRun && m.finished && m.run.err == nil):
+			state = "done"
+		case st.page == at:
+			state = "current"
+		}
+		nav = append(nav, navItem{Label: st.label, State: state})
+	}
+	title, body, keys := m.pageView()
+	f := frame{
+		Width: m.width, Height: m.height,
+		Title: "llmgw setup", Where: m.host + " · " + m.thisAddr(),
+		NavTitle: "Steps", Nav: nav,
+		PanelTitle: title, Body: body, Scroll: m.scroll, Keys: keys,
+	}
+	if m.formErr != "" {
+		f.DialogTitle, f.Dialog = "Not yet", m.formErr+"\n\n"+sDim.Render("any key to go back")
+	}
+	return f.render()
+}
+
+// pageView is the panel for the page: its title, what is in it, and its keys.
+func (m setupModel) pageView() (string, string, string) {
+	switch m.page {
+	case pWelcome:
+		return "Start", m.welcomeView(), "↑↓ choose · Enter go on · Ctrl+C leave"
+	case pAdmin:
+		return "Administrator", m.adminView(), "Tab/↑↓ move · Enter next · Esc back · Ctrl+C leave"
+	case pMachines:
+		return "Database machines", m.machinesView(), "Tab/↑↓ move · ←→ how to log in · Enter next · Esc back · PgUp/PgDn scroll"
+	case pWait:
+		return "Waiting for machines to join", m.waitView(), "Esc back · PgUp/PgDn scroll · Ctrl+C leave"
+	case pCheck:
+		keys := "R check again · Esc back · PgUp/PgDn scroll"
+		if !m.checking && m.report.Worst() < preflight.Fail {
+			keys = "Enter see the plan · " + keys
+		}
+		return "Checks", m.checkView(), keys
+	case pPlan:
+		return "Plan", m.planView(), "Enter install · Esc back · PgUp/PgDn scroll · Ctrl+C leave"
+	}
+	v := m.run.View()
+	if i := strings.Index(v, "\n\n"); i >= 0 {
+		v = v[i+2:] // init's own title; the frame has one
+	}
+	keys := "Ctrl+C stop"
+	if m.finished {
+		keys = "Enter or Q leave · PgUp/PgDn scroll"
+		if m.run.err == nil {
+			v += "\n" + sDim.Render("The same, from a script, with ssh keys in place:") + "\n" + m.command() + "\n"
+		}
+	}
+	return "Install", v, keys
+}
+
+func (m setupModel) welcomeView() string {
+	var b strings.Builder
+	fmt.Fprintf(&b, "\n  Nothing of llmgw is installed on %s.\n\n", lipgloss.NewStyle().Bold(true).Render(m.host))
+	opt := func(i int, title, detail string) {
+		mark := "( )"
+		if m.choice == i {
+			mark = "(" + sOK.Render("•") + ")"
+			title = fSelected.Render(title)
+		}
+		fmt.Fprintf(&b, "  %s %s\n      %s\n\n", mark, title, sDim.Render(detail))
+	}
+	opt(0, "One machine", "This machine runs everything: control plane, model mirror,\n      and models if it has room.")
+	opt(1, "High availability — three machines",
+		"This one is the control plane; two more keep the store, replicated,\n      so losing any one machine loses nothing.")
+	return b.String()
+}
+
+func (m setupModel) adminView() string {
+	var b strings.Builder
+	label := func(i int, s string) string {
+		if m.afocus == i {
+			return fSelected.Render("› " + s)
+		}
+		return "  " + s
+	}
+	b.WriteString("\n")
+	fmt.Fprintf(&b, "%s\n    %s\n    %s\n\n", label(0, "Administrator's email"), m.admin.View(),
+		sDim.Render("A password is generated and shown once, at the end."))
+	fmt.Fprintf(&b, "%s\n    %s\n    %s\n\n", label(1, "Other names nodes will reach this machine by"), m.names.View(),
+		sDim.Render("Known: "+strings.Join(append([]string{m.host}, m.ips...), ", ")+". Add a DNS name or a floating address."))
+	b.WriteString(button(m.afocus == 2, "Next") + "\n")
+	return b.String()
+}
+
+// button is a [ Label ] that shows when it has the focus.
+func button(focused bool, label string) string {
+	if focused {
+		return "  " + fBar.Render(" "+label+" ")
+	}
+	return "  " + lipgloss.NewStyle().Bold(true).Render("[ "+label+" ]")
+}
+
+func (m setupModel) machinesView() string {
+	var b strings.Builder
+	b.WriteString("\n  The two machines that keep the store: Ubuntu 22.04/24.04 or Debian 12.\n" +
+		"  PostgreSQL, Patroni and etcd are installed on them.\n\n")
+	for i, d := range m.db {
+		base := i * mfCount
+		// Padded before it is coloured: the colour's escape codes count
+		// toward a width and threw the focused row out of line.
+		label := func(f int, s string) string {
+			s = fmt.Sprintf("%-15s", s)
+			if m.mfocus == base+f {
+				return fSelected.Render("› " + s)
+			}
+			return "  " + s
+		}
+		fmt.Fprintf(&b, "  %s\n", lipgloss.NewStyle().Bold(true).Render(fmt.Sprintf("Database machine %d", i+1)))
+		if d.visible(mfAddr) {
+			fmt.Fprintf(&b, "  %s %s\n", label(mfAddr, "Address"), d.addr.View())
+			fmt.Fprintf(&b, "  %s %s\n", label(mfUser, "User"), d.user.View())
+		}
+		var auth []string
+		for a := 0; a < authCount; a++ {
+			if a == d.auth {
+				auth = append(auth, "("+sOK.Render("•")+") "+authNames[a])
+			} else {
+				auth = append(auth, "( ) "+sDim.Render(authNames[a]))
+			}
+		}
+		fmt.Fprintf(&b, "  %s   %s\n", label(mfAuth, "Log in with"), strings.Join(auth, "  "))
+		if d.visible(mfSecret) {
+			fmt.Fprintf(&b, "  %s %s\n", label(mfSecret, strings.ToUpper(authNames[d.auth][:1])+authNames[d.auth][1:]), d.secret.View())
+		}
+		if d.visible(mfSudo) {
+			fmt.Fprintf(&b, "  %s %s\n", label(mfSudo, "Sudo password"), d.sudo.View())
+		}
+		if d.auth == authJoin {
+			b.WriteString("      " + sDim.Render("Nothing logs into it: it runs llmgw join with a code shown next.") + "\n")
+		}
+		b.WriteString("\n")
+	}
+	b.WriteString(button(m.mfocus == 2*mfCount, "Check the three machines") + "\n\n")
+	b.WriteString("  " + sDim.Render("Passwords are kept in memory only, for this setup.") + "\n")
+	return b.String()
+}
+
+func (m setupModel) waitView() string {
+	var b strings.Builder
+	fmt.Fprintf(&b, "\n  On each machine below, as root. It dials this machine on port %d;\n  nothing dials in. Each code is for one machine and good for an hour.\n\n", joinpkg.DefaultPort)
+	for i, d := range m.db {
+		if d.auth != authJoin {
+			continue
+		}
+		title := lipgloss.NewStyle().Bold(true).Render(fmt.Sprintf("Database machine %d", i+1))
+		if d.joined != nil {
+			fmt.Fprintf(&b, "  %s  %s joined as %s (%s)\n\n", title, sOK.Render("✓"), d.joined.Hostname, d.joined.Addr)
+			continue
+		}
+		fmt.Fprintf(&b, "  %s  %s waiting\n", title, m.spin.View())
+		b.WriteString("  " + sDim.Render("without llmgw yet:") + "\n")
+		for _, l := range joinInstall() {
+			b.WriteString("    " + strings.ReplaceAll(l, "\n", "\n    ") + "\n")
+		}
+		b.WriteString("  " + sDim.Render("then:") + "\n    sudo llmgw join " + d.code + "\n\n")
+	}
+	return b.String()
+}
+
+func (m setupModel) checkView() string {
+	var b strings.Builder
+	b.WriteString("\n")
+	if m.checking {
+		b.WriteString("  " + m.spin.View() + " checking the three machines…\n")
+		return b.String()
+	}
+	switch m.report.Worst() {
+	case preflight.Fail:
+		b.WriteString("  " + sBad.Render("✗ Fix what is marked ✗, then check again (R).") + "\n")
+	case preflight.Warn:
+		b.WriteString("  " + sWarn.Render("! It can go ahead; what is marked ! is worth a look.") + "\n")
+	default:
+		b.WriteString("  " + sOK.Render("✓ All three are ready.") + "\n")
+	}
+	machine := ""
+	for _, f := range m.report.Findings {
+		if f.Machine != machine {
+			machine = f.Machine
+			b.WriteString("\n  " + lipgloss.NewStyle().Bold(true).Render(machine) + "\n")
+		}
+		mark := sOK.Render("✓")
+		switch f.Level {
+		case preflight.Warn:
+			mark = sWarn.Render("!")
+		case preflight.Fail:
+			mark = sBad.Render("✗")
+		}
+		fmt.Fprintf(&b, "    %s %-13s %s\n", mark, f.Check, f.Detail)
+		if f.Level != preflight.OK && f.Fix != "" {
+			b.WriteString("                    " + sDim.Render(f.Fix) + "\n")
+		}
+	}
+	return b.String()
+}
+
+func (m setupModel) planView() string {
+	var b strings.Builder
+	b.WriteString("\n  What will be installed:\n\n")
+	fmt.Fprintf(&b, "  %s  %s\n      %s\n", lipgloss.NewStyle().Bold(true).Render(m.host), sDim.Render(m.thisAddr()),
+		"control plane and gateway, model mirror, etcd (the third vote)")
+	fmt.Fprintf(&b, "      administrator %s\n", strings.TrimSpace(m.admin.Value()))
+	granted := false
+	for _, d := range m.db {
+		addr := strings.TrimSpace(d.addr.Value())
+		if d.joined != nil {
+			addr = d.joined.Hostname + "  " + sDim.Render(d.joined.Addr+", joined")
+		}
+		fmt.Fprintf(&b, "  %s\n      PostgreSQL under Patroni, etcd\n", lipgloss.NewStyle().Bold(true).Render(addr))
+		granted = granted || d.sudo.Value() != ""
+	}
+	b.WriteString("\n  The store is replicated from the first to the second; if the primary is lost\n" +
+		"  the standby takes over in about thirty seconds, and the control plane follows.\n")
+	if granted {
+		b.WriteString("  " + sDim.Render("Where a sudo password was given, the user may sudo without it until the\n  install finishes, and not after.") + "\n")
+	}
+	b.WriteString("\n  " + sDim.Render("The same from a script, with ssh keys in place:") + "\n")
+	for _, l := range strings.Split(m.command(), "\n") {
+		b.WriteString("  " + l + "\n")
+	}
+	for _, d := range m.db {
+		if d.joined != nil {
+			b.WriteString("  " + sDim.Render("(a machine that joined with a code needs ssh for that)") + "\n")
+			break
+		}
+	}
+	b.WriteString("\n" + button(true, "Enter to install") + "\n")
+	return b.String()
+}
+
+// runSetupTUI is the setup screens, until they finish or hand over to init's
+// own form for one machine.
+func runSetupTUI(ctx context.Context, out io.Writer) error {
+	host, _ := os.Hostname()
+	var ips []string
+	for _, ip := range localAddrs() {
+		if ip.To4() != nil {
+			ips = append(ips, ip.String())
+		}
+	}
+	m := newSetupModel(host, ips)
+	p := tea.NewProgram(m, tea.WithOutput(out), tea.WithAltScreen())
+	m.ref.p = p
+	final, err := p.Run()
+	if err != nil {
+		return err
+	}
+	fm := final.(setupModel)
+	if fm.single {
+		in := newDefaultServerInit()
+		in.out = out
+		return in.runInitTUI(ctx, out)
+	}
+	if fm.page < pRun {
+		return nil
+	}
+	fmt.Fprintln(out)
+	if fm.run.err != nil {
+		fm.logMu.Lock()
+		fmt.Fprintln(out, strings.Join(*fm.log, "\n"))
+		fm.logMu.Unlock()
+		return fm.run.err
+	}
+	if !fm.finished {
+		return fmt.Errorf("left while installing; running llmgw again finishes what was started")
+	}
+	if fm.run.password != "" {
+		fmt.Fprintf(out, "Administrator %s, password %s (shown once)\n\n", fm.in.admin, fm.run.password)
+	}
+	fmt.Fprintln(out, "Next: add nodes from the console — run llmgw again — or with")
+	fmt.Fprintln(out, "  sudo llmgw node add <user>@<address>")
+	return nil
+}

internal/cli/tui_setup_test.go

@@ -0,0 +1,173 @@
+package cli
+
+import (
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+
+	tea "github.com/charmbracelet/bubbletea"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/preflight"
+)
+
+func press(m tea.Model, keys ...string) tea.Model {
+	for _, k := range keys {
+		var msg tea.KeyMsg
+		switch k {
+		case "enter":
+			msg = tea.KeyMsg{Type: tea.KeyEnter}
+		case "tab":
+			msg = tea.KeyMsg{Type: tea.KeyTab}
+		case "down":
+			msg = tea.KeyMsg{Type: tea.KeyDown}
+		case "esc":
+			msg = tea.KeyMsg{Type: tea.KeyEsc}
+		case "right":
+			msg = tea.KeyMsg{Type: tea.KeyRight}
+		default:
+			msg = tea.KeyMsg{Type: tea.KeyRunes, Runes: []rune(k)}
+		}
+		m, _ = m.Update(msg)
+	}
+	return m
+}
+
+func TestTheSetupAsksForWhatHighAvailabilityNeedsAndRefusesWhatWillNotDo(t *testing.T) {
+	m := tea.Model(newSetupModel("ctl", []string{"10.0.0.5"}))
+	m = press(m, "down", "enter")
+	if s := m.(setupModel); s.page != pAdmin {
+		t.Fatalf("high availability did not lead to the administrator: page %d", s.page)
+	}
+	// No email: refused, with a reason.
+	m = press(m, "tab", "tab", "enter")
+	if s := m.(setupModel); s.page != pAdmin || !strings.Contains(s.formErr, "email") {
+		t.Fatalf("went on without an administrator: %+v", s.formErr)
+	}
+	// The refusal is a dialog; a key closes it, and the cursor is back on
+	// the email.
+	if !strings.Contains(m.View(), "Not yet") {
+		t.Error("the refusal was not shown as a dialog")
+	}
+	m = press(m, "x", "a@example.com", "tab", "tab", "enter")
+	s := m.(setupModel)
+	if s.page != pMachines {
+		t.Fatalf("page %d: %s", s.page, s.formErr)
+	}
+	// This machine's own address is not a database machine.
+	s.db[0].addr.SetValue("10.0.0.5")
+	s.db[0].user.SetValue("ubuntu")
+	s.db[0].auth = authAgent
+	s.db[1].addr.SetValue("10.0.0.7")
+	s.db[1].user.SetValue("ubuntu")
+	s.db[1].auth = authPassword
+	if err := s.validateMachines(); !strings.Contains(err, "this machine") {
+		t.Errorf("this machine as a database machine: %q", err)
+	}
+	s.db[0].addr.SetValue("10.0.0.6")
+	if err := s.validateMachines(); !strings.Contains(err, "password") {
+		t.Errorf("a password login with no password: %q", err)
+	}
+	s.db[1].secret.SetValue("pw")
+	if err := s.validateMachines(); err != "" {
+		t.Errorf("a valid pair refused: %s", err)
+	}
+	cmd := s.command()
+	for _, want := range []string{"--admin a@example.com", "--patroni ubuntu@10.0.0.6", "--patroni ubuntu@10.0.0.7", "--no-node"} {
+		if !strings.Contains(cmd, want) {
+			t.Errorf("command lacks %q:\n%s", want, cmd)
+		}
+	}
+	if strings.Contains(cmd, "pw") {
+		t.Error("the password is in the command line")
+	}
+}
+
+func TestChoosingOneMachineHandsOverToInit(t *testing.T) {
+	m := press(tea.Model(newSetupModel("ctl", nil)), "enter").(setupModel)
+	if !m.single {
+		t.Error("one machine did not hand over")
+	}
+}
+
+func TestTheChecksHoldTheSetupBackOnlyForAFailure(t *testing.T) {
+	m := newSetupModel("ctl", nil)
+	m.choice, m.page = 1, pCheck
+	m.report = preflight.Report{Findings: []preflight.Finding{{Machine: "a", Check: "disk", Level: preflight.Fail, Fix: "more"}}}
+	if s := press(m, "enter").(setupModel); s.page != pCheck {
+		t.Error("went on past a failed check")
+	}
+	m.report.Findings[0].Level = preflight.Warn
+	if s := press(m, "enter").(setupModel); s.page != pPlan {
+		t.Error("a warning held it back")
+	}
+}
+
+func TestTheConsoleOffersToReplicateOnlyAStoreThatIsNot(t *testing.T) {
+	m := consoleModel{}
+	if !m.entryOK(cReplicate) {
+		t.Error("a single store is not offered replication")
+	}
+	m.data.sum.Store = []adminapi.StoreMember{{Name: "a", Role: "leader"}}
+	if m.entryOK(cReplicate) {
+		t.Error("a replicated store is offered replication")
+	}
+	// A node is asked for in a dialog, then the console closes with it to do.
+	var mm tea.Model = consoleModel{menu: cAddNode, loaded: true, width: 100, height: 30}
+	mm = press(mm, "enter", "ubuntu@10.0.0.9", "enter")
+	if a := mm.(consoleModel).action; a.kind != "node" || a.targets[0] != "ubuntu@10.0.0.9" {
+		t.Errorf("%+v", a)
+	}
+}
+
+func TestTheConsoleWatchesTheClusterAndEditsItsSettings(t *testing.T) {
+	dir := t.TempDir()
+	file := filepath.Join(dir, "server.yaml")
+	if err := os.WriteFile(file, []byte("# mine\nlisten: \":8080\"\n"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	m := newConsoleModel("127.0.0.1:1", file)
+	m.width, m.height, m.loaded = 120, 40, true
+	m.data.nodes = []adminapi.NodeView{{ID: "cuda01", State: "ready", GPUTotal: 1, GPUs: []adminapi.GPUView{{Index: 0, Name: "Quadro RTX 6000"}}}}
+	m.data.replicas = []adminapi.ReplicaView{{ID: "r1", Model: "coder-14b", State: "ready",
+		Shards: []adminapi.ShardView{{Node: "cuda01", GPUs: []int{0}}}}}
+
+	// Watching: the GPUs view says which model holds each card.
+	m.menu = cGPUs
+	if v := m.View(); !strings.Contains(v, "Quadro RTX 6000") || !strings.Contains(v, "coder-14b") {
+		t.Errorf("the GPUs view:\n%s", v)
+	}
+
+	// Settings: into router, change retries, save.
+	var mm tea.Model = m
+	mm.(consoleModel).View()
+	cm := mm.(consoleModel)
+	cm.menu, cm.inPanel = cSettings, true
+	for i, r := range cm.settingsRows() {
+		if r.label == "router" {
+			cm.row = i
+		}
+	}
+	mm = press(cm, "enter") // into router
+	cm = mm.(consoleModel)
+	for i, r := range cm.settingsRows() {
+		if r.label == "retries" {
+			cm.row = i
+		}
+	}
+	if v := cm.View(); !strings.Contains(v, "(default)") {
+		t.Errorf("an unset key does not show its default:\n%s", v)
+	}
+	mm = press(cm, "enter") // the edit dialog
+	cm = mm.(consoleModel)
+	cm.prompt.SetValue("5")
+	mm = press(cm, "enter", "s")
+	if d := mm.(consoleModel).dialog; d != "saved" {
+		t.Fatalf("after saving the dialog is %q: %s", d, mm.(consoleModel).dialogText)
+	}
+	b, _ := os.ReadFile(file)
+	if !strings.Contains(string(b), "retries: 5") || !strings.Contains(string(b), "# mine") {
+		t.Errorf("server.yaml:\n%s", b)
+	}
+}

internal/cli/tui_widgets.go

@@ -47,6 +47,8 @@ type textField struct {
 	Placeholder string
 	CharLimit   int
 	Width       int
+	// Mask shows a dot for every character: a password.
+	Mask bool
 }
 
 func newTextField() textField { return textField{Prompt: "> "} }
@@ -84,6 +86,9 @@ func (f textField) Update(msg tea.Msg) (textField, tea.Cmd) {
 
 func (f textField) View() string {
 	text := string(f.value)
+	if f.Mask {
+		text = strings.Repeat("•", len(f.value))
+	}
 	shown := text
 	if text == "" && f.Placeholder != "" {
 		shown = sDim.Render(f.Placeholder)

internal/config/config.go

@@ -257,6 +257,14 @@ type Auth struct {
 	// blocking a request: losing accounting is bad, turning a stalled database
 	// into an outage is worse.
 	BufferSize int `koanf:"buffer_size"`
+
+	// UsageRetention is how long the record of each request is kept: who sent
+	// it, which replica answered, how long it queued — what `ctl request`
+	// reads. Zero keeps it for ever. Totals, charts and budgets are read from
+	// five-minute sums kept beside the records, which are not pruned, so this
+	// bounds the table that grows by a row a request without changing any
+	// figure a page shows. At least a day when set.
+	UsageRetention time.Duration `koanf:"usage_retention"`
 }
 
 // Personal configures use that belongs to a person rather than a project.
@@ -363,6 +371,13 @@ type Router struct {
 	// accounting works regardless of client behaviour.
 	InjectUsage bool `koanf:"inject_usage"`
 
+	// TenantLabel is the level of the tree the per-tenant request series are
+	// labelled by — llmgw_tenant_requests_total and its siblings: "org",
+	// "team" or "project", or "none" for no such series. A label per project
+	// on a cluster of thousands is more series than Prometheus wants, and a
+	// team is what an alert is usually about; "team" is the default.
+	TenantLabel string `koanf:"tenant_label"`
+
 	// MaxBodyBytes bounds the request body the router will read.
 	MaxBodyBytes int64 `koanf:"max_body_bytes"`
 
@@ -483,6 +498,7 @@ func Default() Server {
 			Cooldown:         15 * time.Second,
 			FailureThreshold: 3,
 			DefaultProfile:   "chat",
+			TenantLabel:      "team",
 			SessionHeader:    "x-session-id",
 			InjectUsage:      true,
 			MaxBodyBytes:     32 << 20,
@@ -573,6 +589,14 @@ func (s Server) Validate() error {
 	if s.Auth.Enabled && s.Auth.Store == "" {
 		return fmt.Errorf("auth.store is required when auth is enabled")
 	}
+	switch s.Router.TenantLabel {
+	case "", "none", "org", "team", "project":
+	default:
+		return fmt.Errorf("router.tenant_label is %q: org, team, project or none", s.Router.TenantLabel)
+	}
+	if s.Auth.UsageRetention != 0 && s.Auth.UsageRetention < 24*time.Hour {
+		return fmt.Errorf("auth.usage_retention is %s: at least 24h, or 0 to keep every request", s.Auth.UsageRetention)
+	}
 	if s.Threads.Store != "" && s.Threads.RetainHours <= 0 {
 		return fmt.Errorf("threads.retain_hours must be positive when threads.store is set")
 	}

internal/config/config_test.go

@@ -206,3 +206,17 @@ func TestEnginesDefaultToTheCheckedVersions(t *testing.T) {
 		t.Error("an emptied engine is still wanted")
 	}
 }
+
+func TestAUsageRetentionUnderADayIsRefused(t *testing.T) {
+	// Shorter than a day is almost certainly a unit slip — 90m for 90 days —
+	// and would delete every request record within the hour.
+	s := Default()
+	s.Auth.UsageRetention = 90 * time.Minute
+	if err := s.Validate(); err == nil || !strings.Contains(err.Error(), "usage_retention") {
+		t.Errorf("a 90-minute retention was accepted: %v", err)
+	}
+	s.Auth.UsageRetention = 90 * 24 * time.Hour
+	if err := s.Validate(); err != nil {
+		t.Errorf("a 90-day retention was refused: %v", err)
+	}
+}

internal/control/adminapi/api.go

@@ -40,6 +40,8 @@ import (
 type API struct {
 	// releases is where release binaries are fetched from, for upgrades.
 	releases string
+	// storeStatus reads a replicated store's members; nil for one database.
+	storeStatus func() ([]StoreMember, error)
 	// serve runs registry-to-replica jobs; nil without a mirror and a store.
 	serve *modelserve.Server
 
@@ -698,6 +700,9 @@ type Usage interface {
 // SetUsage supplies the usage reporting source.
 func (a *API) SetUsage(u Usage) { a.usage = u }
 
+// SetStoreStatus supplies the state of a replicated store, for the summary.
+func (a *API) SetStoreStatus(f func() ([]StoreMember, error)) { a.storeStatus = f }
+
 // ScopeUsageView is one group's spend, with its quota when it has one.
 type ScopeUsageView struct {
 	Scope string `json:"scope"`

internal/control/adminapi/summary.go

@@ -31,6 +31,20 @@ type Summary struct {
 	// IdleReplicas are serving replicas nothing has asked for in a while, which
 	// on a fleet with more models than GPUs is the first place to find room.
 	IdleReplicas []IdleReplica `json:"idle_replicas,omitempty"`
+	// Store is the members of the store's cluster when it is replicated, and
+	// StoreError why it could not be read; both empty for a single database.
+	Store      []StoreMember `json:"store,omitempty"`
+	StoreError string        `json:"store_error,omitempty"`
+}
+
+// StoreMember is one member of a replicated store, as its Patroni reports it.
+type StoreMember struct {
+	Name  string `json:"name"`
+	Host  string `json:"host"`
+	Role  string `json:"role"`
+	State string `json:"state"`
+	// Lag is bytes behind the leader; -1 unknown, 0 for the leader.
+	Lag int64 `json:"lag"`
 }
 
 // NodeTotals counts nodes by the states that need different responses.
@@ -107,6 +121,13 @@ const idleThreshold = 10 * time.Minute
 // Summary builds the fleet summary.
 func (a *API) Summary() Summary {
 	var s Summary
+	if a.storeStatus != nil {
+		members, err := a.storeStatus()
+		s.Store = members
+		if err != nil {
+			s.StoreError = err.Error()
+		}
+	}
 
 	nodes := a.Nodes()
 	byClass := map[string]*ClassUse{}

internal/control/agentsvc/controlplanes.go

@@ -0,0 +1,65 @@
+package agentsvc
+
+import (
+	"context"
+	"slices"
+	"time"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+)
+
+// SetControlPlanes gives the service where the list of control planes comes
+// from. Every agent is told it when it registers, and again when it changes.
+func (s *Service) SetControlPlanes(list func(context.Context) ([]string, error)) {
+	s.mu.Lock()
+	s.controlPlanes = list
+	s.mu.Unlock()
+}
+
+// controlPlanesMsg is the list as an agent is sent it, or nil when there is
+// none to send: a cluster with no list recorded keeps what its agents were
+// configured with.
+func (s *Service) controlPlanesMsg(ctx context.Context) *llmgwv1.ControlMessage {
+	s.mu.RLock()
+	list := s.controlPlanes
+	s.mu.RUnlock()
+	if list == nil {
+		return nil
+	}
+	servers, err := list(ctx)
+	if err != nil || len(servers) == 0 {
+		return nil
+	}
+	return &llmgwv1.ControlMessage{Msg: &llmgwv1.ControlMessage_ControlPlanes{
+		ControlPlanes: &llmgwv1.ControlPlanes{Servers: servers}}}
+}
+
+// WatchControlPlanes tells every connected agent the list when it changes,
+// checking every so often. A control plane added by `llmgw server add` reaches
+// the nodes within that time, without their being touched.
+func (s *Service) WatchControlPlanes(ctx context.Context, every time.Duration) {
+	var last []string
+	t := time.NewTicker(every)
+	defer t.Stop()
+	for {
+		if msg := s.controlPlanesMsg(ctx); msg != nil {
+			now := msg.GetControlPlanes().GetServers()
+			if !slices.Equal(now, last) {
+				for _, node := range s.ConnectedNodes() {
+					if err := s.send(node, msg); err != nil {
+						s.log.Warn("could not tell a node the control planes", "node", node, "err", err)
+					}
+				}
+				if last != nil {
+					s.log.Info("told the nodes the control planes changed", "servers", now)
+				}
+				last = now
+			}
+		}
+		select {
+		case <-ctx.Done():
+			return
+		case <-t.C:
+		}
+	}
+}

internal/control/agentsvc/heartbeat_test.go

@@ -283,3 +283,36 @@ func TestAnUpgradeWaitsForTheAgentsAnswer(t *testing.T) {
 		t.Errorf("the agent's refusal was not passed on: %v", err)
 	}
 }
+
+func TestAnotherTenantOnACardIsJudgedEveryHeartbeat(t *testing.T) {
+	// Found on cuda01: ComfyUI held 5 GB when the agent started, so the card
+	// was kept out of placement. ComfyUI let go of it and the card stayed
+	// out, saying "5014 MiB already allocated by another process", until the
+	// agent was restarted.
+	s := newTestService()
+	nodeWith(s, inventory.Node{
+		ID: "n1", State: inventory.Ready,
+		GPUs: []discovery.GPU{{Index: 0, Health: discovery.GPUHealth{Problems: []string{
+			discovery.ForeignMemoryProblem(5014), "2 uncorrected ECC errors"}}}},
+	})
+
+	s.applyHeartbeat("n1", &llmgwv1.Heartbeat{NodeId: "n1", GpuForeignMemoryMb: []int32{330}})
+	n, _ := s.inv.Get("n1")
+	if got := n.GPUs[0].Health.Problems; len(got) != 1 || got[0] != "2 uncorrected ECC errors" {
+		t.Fatalf("problems are %q: the tenant left, and only the ECC errors should remain", got)
+	}
+
+	// And the other way: a tenant arriving after registration.
+	s.applyHeartbeat("n1", &llmgwv1.Heartbeat{NodeId: "n1", GpuForeignMemoryMb: []int32{12000}})
+	n, _ = s.inv.Get("n1")
+	if got := n.GPUs[0].Health.Problems; len(got) != 2 || got[1] != discovery.ForeignMemoryProblem(12000) {
+		t.Fatalf("problems are %q, want the new tenant's 12000 MiB", got)
+	}
+
+	// An agent that sends no reading changes nothing.
+	s.applyHeartbeat("n1", &llmgwv1.Heartbeat{NodeId: "n1", GpuMemoryUsedMb: []int32{100}})
+	n, _ = s.inv.Get("n1")
+	if len(n.GPUs[0].Health.Problems) != 2 {
+		t.Errorf("a heartbeat with no foreign reading cleared the card: %q", n.GPUs[0].Health.Problems)
+	}
+}

internal/control/agentsvc/service.go

@@ -61,6 +61,9 @@ type Service struct {
 	choices       func(nodeID string) map[string]string
 	installableBy map[string][]string
 
+	// controlPlanes lists what agents may dial; see SetControlPlanes.
+	controlPlanes func(context.Context) ([]string, error)
+
 	// logWaits are the log requests awaiting a reply, by request id.
 	logMu    sync.Mutex
 	logWaits map[string]chan *llmgwv1.LogsReply
@@ -229,6 +232,13 @@ func (s *Service) Connect(
 			return err
 		}
 	}
+	// Which control planes there are, so an agent configured with one learns
+	// of a second added since.
+	if msg := s.controlPlanesMsg(ctx); msg != nil {
+		if err := stream.Send(msg); err != nil {
+			return err
+		}
+	}
 
 	// Recover replica state from what this node is already running, before any
 	// reconciliation pass can conclude the cluster is empty and place
@@ -377,11 +387,12 @@ func (s *Service) handle(nodeID string, msg *llmgwv1.AgentMessage) {
 // a reconnect.
 func (s *Service) applyHeartbeat(nodeID string, hb *llmgwv1.Heartbeat) {
 	used := hb.GetGpuMemoryUsedMb()
+	foreign := hb.GetGpuForeignMemoryMb()
 	problems := hb.GetNewProblems()
 	storage := hb.GetStorage()
 	memMB := int(hb.GetMemoryTotalMb())
 	cpus := int(hb.GetCpuCount())
-	if len(used) == 0 && len(problems) == 0 && len(storage) == 0 &&
+	if len(used) == 0 && len(foreign) == 0 && len(problems) == 0 && len(storage) == 0 &&
 		!hb.GetWeightsChanged() && memMB == 0 && cpus == 0 && !hb.GetEnginesChanged() &&
 		!hb.GetProvisionChanged() && hb.GetLoad() == nil && len(hb.GetIbPorts()) == 0 &&
 		!hb.GetFabricsChanged() && len(hb.GetGpuFabric()) == 0 && hb.GetImex() == nil {
@@ -397,6 +408,20 @@ func (s *Service) applyHeartbeat(nodeID string, hb *llmgwv1.Heartbeat) {
 			n.GPUs[i].MemoryUsedMB = int(used[i])
 		}
 	}
+	// Another tenant on a card, judged afresh: the reading at registration is
+	// replaced, both ways. Without this a card that had something on it when
+	// its agent started stayed out of placement after it left, and said so,
+	// until the agent was restarted. Only a reading for every card replaces
+	// it; an agent that sends none, or could not measure, leaves it standing.
+	if len(foreign) == len(n.GPUs) {
+		for i := range n.GPUs {
+			ps := slices.DeleteFunc(slices.Clone(n.GPUs[i].Health.Problems), discovery.IsForeignMemoryProblem)
+			if p := discovery.ForeignMemoryProblem(int(foreign[i])); p != "" {
+				ps = append(ps, p)
+			}
+			n.GPUs[i].Health.Problems = ps
+		}
+	}
 	if len(problems) > 0 {
 		// Problems are reported per node, not per GPU, so attach them to the
 		// node's first GPU rather than guessing which one is affected. Any
@@ -585,6 +610,10 @@ func loadFrom(l *llmgwv1.NodeLoad, at time.Time) *inventory.Load {
 		out.GPUs = append(out.GPUs, inventory.GPUSensor{
 			Index: int(g.GetIndex()), TempC: g.GetTempC(), MemoryTempC: g.GetMemoryTempC(),
 			PowerW: g.GetPowerW(), PowerLimitW: g.GetPowerLimitW(), Throttle: g.GetThrottle(),
+			UtilPercent: g.GetUtilizationPercent(), MemUtilPercent: g.GetMemoryUtilizationPercent(),
+			MemoryUsedMB: g.GetMemoryUsedMb(), MemoryTotalMB: g.GetMemoryTotalMb(),
+			SMClockMHz: g.GetSmClockMhz(), ECCUncorrected: g.GetEccUncorrected(),
+			Wear: wearFrom(g),
 		})
 	}
 	return out
@@ -961,3 +990,22 @@ func (s *Service) Upgrade(ctx context.Context, nodeID, version, releases string)
 		return fmt.Errorf("node %s did not answer the upgrade; an agent older than v0.10.4 cannot upgrade itself", nodeID)
 	}
 }
+
+// wearFrom is a card's wear, when the agent read it.
+func wearFrom(g *llmgwv1.GPUSensor) *inventory.GPUWear {
+	if !g.GetHasWear() {
+		return nil
+	}
+	w := &inventory.GPUWear{
+		PCIeReplays: g.GetPcieReplays(), RetiredSBE: g.GetRetiredPagesSbe(), RetiredDBE: g.GetRetiredPagesDbe(),
+		RetirePending: g.GetRetirementPending(), RemappedCorrectable: g.GetRemappedRowsCorrectable(),
+		RemappedUncorrectable: g.GetRemappedRowsUncorrectable(), RemapPending: g.GetRemapPending(),
+		RemapFailure: g.GetRemapFailure(), NVLinkCRC: g.GetNvlinkCrcErrors(), NVLinkReplay: g.GetNvlinkReplayErrors(),
+		NVLinkRecovery: g.GetNvlinkRecoveryErrors(), XIDs: g.GetXidErrors(), LastXID: int(g.GetLastXid()),
+		LastXIDText: g.GetLastXidText(),
+	}
+	if u := g.GetLastXidUnix(); u > 0 {
+		w.LastXIDAt = time.Unix(u, 0)
+	}
+	return w
+}

internal/control/control.go

@@ -417,3 +417,17 @@ func (c *Controller) SetEngineChoices(e *EngineChoices) { c.engineChoices = e }
 
 // EngineChoices are the engine builds chosen per node, or nil.
 func (c *Controller) EngineChoices() *EngineChoices { return c.engineChoices }
+
+// ModelOwners maps each model to the part of the tenancy tree its capacity
+// counts against; a model owned by nobody is not in it.
+func (c *Controller) ModelOwners() map[string]string {
+	out := map[string]string{}
+	if cat := c.Catalog(); cat != nil {
+		for _, m := range cat.Models() {
+			if m.Owner != "" {
+				out[m.Name] = m.Owner
+			}
+		}
+	}
+	return out
+}

internal/control/fleetmetrics/fleetmetrics.go

@@ -73,6 +73,18 @@ type Collector struct {
 	gpuMemoryTotal *prometheus.Desc
 	gpuMemoryUsed  *prometheus.Desc
 	gpuUtil        *prometheus.Desc
+	gpuAssigned    *prometheus.Desc
+	gpuPCIeReplay  *prometheus.Desc
+	gpuRetired     *prometheus.Desc
+	gpuRetirePend  *prometheus.Desc
+	gpuRemapped    *prometheus.Desc
+	gpuRemapPend   *prometheus.Desc
+	gpuRemapFail   *prometheus.Desc
+	gpuNVLinkErr   *prometheus.Desc
+	gpuXID         *prometheus.Desc
+	gpuLastXID     *prometheus.Desc
+	gpuMemUtil     *prometheus.Desc
+	gpuSMClock     *prometheus.Desc
 	gpuTemp        *prometheus.Desc
 	gpuMemTemp     *prometheus.Desc
 	gpuPower       *prometheus.Desc
@@ -149,6 +161,33 @@ func New(src Source) *Collector {
 			"How busy a GPU was over the last sample period. Absent when the driver does not "+
 				"report it, which is not the same as idle. Memory used says a model is loaded; "+
 				"this says whether it is doing anything.", gpu, nil),
+		gpuPCIeReplay: prometheus.NewDesc("llmgw_gpu_pcie_replays",
+			"PCIe link retries on a GPU's link since the driver loaded; one that keeps climbing is a riser, slot or cable.", gpu, nil),
+		gpuRetired: prometheus.NewDesc("llmgw_gpu_retired_pages",
+			"Memory pages retired, by cause (sbe, dbe), on GPUs before Ampere.", append(gpu, "cause"), nil),
+		gpuRetirePend: prometheus.NewDesc("llmgw_gpu_retirement_pending",
+			"1 when retired pages wait for a GPU reset to take effect.", gpu, nil),
+		gpuRemapped: prometheus.NewDesc("llmgw_gpu_remapped_rows",
+			"Memory rows remapped, by kind (correctable, uncorrectable), on Ampere and later.", append(gpu, "kind"), nil),
+		gpuRemapPend: prometheus.NewDesc("llmgw_gpu_remap_pending",
+			"1 when remapped rows wait for a GPU reset to take effect.", gpu, nil),
+		gpuRemapFail: prometheus.NewDesc("llmgw_gpu_remap_failure",
+			"1 when a GPU has run out of spare memory rows: it needs replacing.", gpu, nil),
+		gpuNVLinkErr: prometheus.NewDesc("llmgw_gpu_nvlink_errors",
+			"NVLink errors summed over a GPU's links, by kind (crc, replay, recovery).", append(gpu, "kind"), nil),
+		gpuXID: prometheus.NewDesc("llmgw_gpu_xid_errors",
+			"The driver's XID error events for a GPU since its node's agent started.", gpu, nil),
+		gpuLastXID: prometheus.NewDesc("llmgw_gpu_last_xid",
+			"The code of a GPU's last XID error since its agent started; absent when there was none.", gpu, nil),
+		gpuAssigned: prometheus.NewDesc("llmgw_gpu_assigned",
+			"1 for each GPU a replica holds: which model, which replica, and whose quota it counts against. "+
+				"Joined on node and gpu with the other llmgw_gpu_ series, it says what each model's cards are doing.",
+			[]string{"node", "gpu", "model", "replica", "owner"}, nil),
+		gpuMemUtil: prometheus.NewDesc("llmgw_gpu_memory_utilization_percent",
+			"How much of the sample period a GPU's memory was being read or written: "+
+				"decode is bound by it, so a card busy here and idle above is waiting on memory.", gpu, nil),
+		gpuSMClock: prometheus.NewDesc("llmgw_gpu_sm_clock_mhz",
+			"A GPU's streaming multiprocessor clock now; one held low under load is the card being throttled.", gpu, nil),
 		gpuMemTemp: prometheus.NewDesc("llmgw_gpu_memory_temperature_celsius",
 			"A GPU's memory (HBM) temperature, where the driver reports it.", gpu, nil),
 		gpuPower: prometheus.NewDesc("llmgw_gpu_power_watts",
@@ -271,7 +310,9 @@ func (c *Collector) Describe(ch chan<- *prometheus.Desc) {
 		c.nodeCPU, c.nodeCPUs, c.nodeLoad, c.nodeMemUsed, c.nodeMemAvail, c.shardCPU, c.shardMemory,
 		c.ibActive, c.ibRate, c.ibCounter, c.fabricSMs, c.fabricMaster, c.fabricProblems,
 		c.gpuNVLJoined, c.imexUp,
-		c.gpuMemoryTotal, c.gpuMemoryUsed, c.gpuUtil, c.gpuTemp, c.gpuHealthy,
+		c.gpuMemoryTotal, c.gpuMemoryUsed, c.gpuUtil, c.gpuMemUtil, c.gpuSMClock, c.gpuAssigned, c.gpuTemp, c.gpuHealthy,
+		c.gpuPCIeReplay, c.gpuRetired, c.gpuRetirePend, c.gpuRemapped, c.gpuRemapPend, c.gpuRemapFail,
+		c.gpuNVLinkErr, c.gpuXID, c.gpuLastXID,
 		c.gpuMemTemp, c.gpuPower, c.gpuPowerLimit, c.gpuThrottled, c.nodeCPUTemp, c.nodeCPUPower, c.nodeInletTemp,
 		c.gpuNVLink, c.gpuECC,
 		c.replicaState, c.replicaReady, c.replicaIdle,
@@ -287,6 +328,7 @@ func (c *Collector) Describe(ch chan<- *prometheus.Desc) {
 func (c *Collector) Collect(ch chan<- prometheus.Metric) {
 	now := time.Now()
 	c.collectQuotas(ch)
+	c.collectAssignments(ch)
 
 	loads := map[string]*inventory.Load{}
 	for _, n := range c.src.Nodes() {
@@ -388,20 +430,28 @@ func (c *Collector) Collect(ch chan<- prometheus.Metric) {
 			byClass[class]++
 			labels := []string{n.ID, strconv.Itoa(g.Index), class}
 
-			ch <- prometheus.MustNewConstMetric(c.gpuMemoryTotal, prometheus.GaugeValue,
-				float64(g.MemoryTotalMB)*1024*1024, labels...)
-			ch <- prometheus.MustNewConstMetric(c.gpuMemoryUsed, prometheus.GaugeValue,
-				float64(g.MemoryUsedMB)*1024*1024, labels...)
-			if g.UtilizationPercent > 0 {
-				// Only when reported. A driver too old to give it would
-				// otherwise show every card at a flat zero, which reads as an
-				// idle fleet rather than a missing measurement.
-				ch <- prometheus.MustNewConstMetric(c.gpuUtil, prometheus.GaugeValue,
-					float64(g.UtilizationPercent), labels...)
+			s, fresh := sensorFor(n.Load, g.Index, now)
+			if fresh && s.Busy() {
+				// What the card is doing now, sampled every heartbeat.
+				c.gpuBusy(ch, s, labels)
+			} else {
+				// An agent too old to sample it: what it read at
+				// registration, which is all there is.
+				ch <- prometheus.MustNewConstMetric(c.gpuMemoryTotal, prometheus.GaugeValue,
+					float64(g.MemoryTotalMB)*1024*1024, labels...)
+				ch <- prometheus.MustNewConstMetric(c.gpuMemoryUsed, prometheus.GaugeValue,
+					float64(g.MemoryUsedMB)*1024*1024, labels...)
+				if g.UtilizationPercent > 0 {
+					// Only when reported. A driver too old to give it would
+					// otherwise show every card at a flat zero, which reads as
+					// an idle fleet rather than a missing measurement.
+					ch <- prometheus.MustNewConstMetric(c.gpuUtil, prometheus.GaugeValue,
+						float64(g.UtilizationPercent), labels...)
+				}
 			}
 			// The sensors' fresh reading when there is one; the temperature
 			// read at registration otherwise, which is all an older agent sends.
-			if s, ok := sensorFor(n.Load, g.Index, now); ok {
+			if fresh {
 				c.gpuSensor(ch, s, labels)
 			} else if g.Health.TemperatureC > 0 {
 				ch <- prometheus.MustNewConstMetric(c.gpuTemp, prometheus.GaugeValue,
@@ -414,8 +464,11 @@ func (c *Collector) Collect(ch chan<- prometheus.Metric) {
 			ch <- prometheus.MustNewConstMetric(c.gpuHealthy, prometheus.GaugeValue, healthy, labels...)
 			ch <- prometheus.MustNewConstMetric(c.gpuNVLink, prometheus.GaugeValue,
 				float64(len(g.NVLinkPeers)), labels...)
-			ch <- prometheus.MustNewConstMetric(c.gpuECC, prometheus.GaugeValue,
-				float64(g.Health.ECCErrors), labels...)
+			ecc := float64(g.Health.ECCErrors)
+			if fresh && s.Busy() && s.ECCUncorrected >= 0 {
+				ecc = s.ECCUncorrected
+			}
+			ch <- prometheus.MustNewConstMetric(c.gpuECC, prometheus.GaugeValue, ecc, labels...)
 		}
 		for class, n2 := range byClass {
 			ch <- prometheus.MustNewConstMetric(c.nodeGPUs, prometheus.GaugeValue,
@@ -521,6 +574,12 @@ func b2f(b bool) float64 {
 	return 0
 }
 
+// OwnerSource says whose quota each model counts against, for labelling
+// what its replicas hold. Asked for by type assertion.
+type OwnerSource interface {
+	ModelOwners() map[string]string
+}
+
 // QuotaSource adds the tenancy tree's capacity quotas. Asked for by type
 // assertion, like HealthSource.
 type QuotaSource interface {
@@ -569,6 +628,23 @@ func sensorFor(l *inventory.Load, index int, now time.Time) (inventory.GPUSensor
 	return inventory.GPUSensor{}, false
 }
 
+// gpuBusy is how busy a card is, from a sample that carries it.
+func (c *Collector) gpuBusy(ch chan<- prometheus.Metric, s inventory.GPUSensor, labels []string) {
+	ch <- prometheus.MustNewConstMetric(c.gpuMemoryTotal, prometheus.GaugeValue, s.MemoryTotalMB*1024*1024, labels...)
+	ch <- prometheus.MustNewConstMetric(c.gpuMemoryUsed, prometheus.GaugeValue, s.MemoryUsedMB*1024*1024, labels...)
+	// Zero is a real reading here — an idle card — and only -1, the driver
+	// not saying, leaves the series out.
+	if s.UtilPercent >= 0 {
+		ch <- prometheus.MustNewConstMetric(c.gpuUtil, prometheus.GaugeValue, s.UtilPercent, labels...)
+	}
+	if s.MemUtilPercent >= 0 {
+		ch <- prometheus.MustNewConstMetric(c.gpuMemUtil, prometheus.GaugeValue, s.MemUtilPercent, labels...)
+	}
+	if s.SMClockMHz > 0 {
+		ch <- prometheus.MustNewConstMetric(c.gpuSMClock, prometheus.GaugeValue, s.SMClockMHz, labels...)
+	}
+}
+
 func (c *Collector) gpuSensor(ch chan<- prometheus.Metric, s inventory.GPUSensor, labels []string) {
 	for _, v := range []struct {
 		d   *prometheus.Desc
@@ -581,4 +657,61 @@ func (c *Collector) gpuSensor(ch chan<- prometheus.Metric, s inventory.GPUSensor
 	for _, reason := range s.Throttle {
 		ch <- prometheus.MustNewConstMetric(c.gpuThrottled, prometheus.GaugeValue, 1, append(labels, reason)...)
 	}
+	if w := s.Wear; w != nil {
+		c.gpuWear(ch, w, labels)
+	}
+}
+
+// gpuWear is what wears a card out. A count the card does not keep (-1) is
+// left out, not published as zero.
+func (c *Collector) gpuWear(ch chan<- prometheus.Metric, w *inventory.GPUWear, labels []string) {
+	gauge := func(d *prometheus.Desc, v float64, extra ...string) {
+		ch <- prometheus.MustNewConstMetric(d, prometheus.GaugeValue, v, append(append([]string{}, labels...), extra...)...)
+	}
+	flag := func(b bool) float64 {
+		if b {
+			return 1
+		}
+		return 0
+	}
+	if w.PCIeReplays >= 0 {
+		gauge(c.gpuPCIeReplay, float64(w.PCIeReplays))
+	}
+	if w.RetiredSBE >= 0 || w.RetiredDBE >= 0 {
+		gauge(c.gpuRetired, float64(max(w.RetiredSBE, 0)), "sbe")
+		gauge(c.gpuRetired, float64(max(w.RetiredDBE, 0)), "dbe")
+		gauge(c.gpuRetirePend, flag(w.RetirePending))
+	}
+	if w.RemappedCorrectable >= 0 || w.RemappedUncorrectable >= 0 {
+		gauge(c.gpuRemapped, float64(max(w.RemappedCorrectable, 0)), "correctable")
+		gauge(c.gpuRemapped, float64(max(w.RemappedUncorrectable, 0)), "uncorrectable")
+		gauge(c.gpuRemapPend, flag(w.RemapPending))
+		gauge(c.gpuRemapFail, flag(w.RemapFailure))
+	}
+	gauge(c.gpuNVLinkErr, float64(w.NVLinkCRC), "crc")
+	gauge(c.gpuNVLinkErr, float64(w.NVLinkReplay), "replay")
+	gauge(c.gpuNVLinkErr, float64(w.NVLinkRecovery), "recovery")
+	gauge(c.gpuXID, float64(w.XIDs))
+	if w.XIDs > 0 {
+		gauge(c.gpuLastXID, float64(w.LastXID))
+	}
+}
+
+// collectAssignments says which replica holds each GPU.
+func (c *Collector) collectAssignments(ch chan<- prometheus.Metric) {
+	var owners map[string]string
+	if o, ok := c.src.(OwnerSource); ok {
+		owners = o.ModelOwners()
+	}
+	for _, r := range c.src.Replicas() {
+		if !r.State.Counts() {
+			continue
+		}
+		for _, sh := range r.Shards {
+			for _, g := range sh.GPUIndices {
+				ch <- prometheus.MustNewConstMetric(c.gpuAssigned, prometheus.GaugeValue, 1,
+					sh.NodeID, strconv.Itoa(g), r.Model, r.ID, owners[r.Model])
+			}
+		}
+	}
 }

internal/control/fleetmetrics/sensors_test.go

@@ -7,6 +7,7 @@ import (
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
 )
 
 func TestSensorsArePublishedFreshAndOnlyWhenMeasured(t *testing.T) {
@@ -39,3 +40,90 @@ func TestSensorsArePublishedFreshAndOnlyWhenMeasured(t *testing.T) {
 		}
 	}
 }
+
+func TestHowBusyAGPUIsComesFromTheLatestSampleNotRegistration(t *testing.T) {
+	// Discovery read 87% and 70 GB used when the agent came up; the card has
+	// been idle since. The sample says so, zero included.
+	reg := discovery.GPU{Index: 0, Name: "NVIDIA H100 80GB HBM3", MemoryTotalMB: 81559,
+		MemoryUsedMB: 70000, UtilizationPercent: 87}
+	n := inventory.Node{ID: "g1", State: inventory.Ready, GPUs: []discovery.GPU{reg},
+		Load: &inventory.Load{At: time.Now(), GPUs: []inventory.GPUSensor{{Index: 0, TempC: 35,
+			UtilPercent: 0, MemUtilPercent: 0, MemoryUsedMB: 512, MemoryTotalMB: 81559,
+			SMClockMHz: 345, ECCUncorrected: 2}}}}
+	out := gather(t, fakeSource{nodes: []inventory.Node{n}})
+	for _, want := range []string{
+		"llmgw_gpu_utilization_percent class=h100-80g gpu=0 node=g1 -> 0\n",
+		"llmgw_gpu_memory_utilization_percent class=h100-80g gpu=0 node=g1 -> 0\n",
+		"llmgw_gpu_memory_used_bytes class=h100-80g gpu=0 node=g1 -> 536870912\n",
+		"llmgw_gpu_sm_clock_mhz class=h100-80g gpu=0 node=g1 -> 345\n",
+		"llmgw_gpu_ecc_errors class=h100-80g gpu=0 node=g1 -> 2\n",
+	} {
+		if !strings.Contains(out, want) {
+			t.Errorf("missing %s in:\n%s", want, out)
+		}
+	}
+	if strings.Contains(out, "-> 87\n") {
+		t.Error("the utilization read at registration was published over the sample's")
+	}
+
+	// An agent too old to sample it: registration's figures, and no zero
+	// standing in for a utilization the driver did not give.
+	n.Load.GPUs = []inventory.GPUSensor{{Index: 0, TempC: 35}}
+	out = gather(t, fakeSource{nodes: []inventory.Node{n}})
+	if !strings.Contains(out, "llmgw_gpu_utilization_percent class=h100-80g gpu=0 node=g1 -> 87") {
+		t.Errorf("an older agent's registration figure was dropped:\n%s", out)
+	}
+	n.Load.GPUs = []inventory.GPUSensor{{Index: 0, TempC: 35, UtilPercent: -1, MemUtilPercent: -1, MemoryTotalMB: 81559}}
+	if out := gather(t, fakeSource{nodes: []inventory.Node{n}}); strings.Contains(out, "llmgw_gpu_utilization_percent ") {
+		t.Errorf("a utilization the driver did not report was published:\n%s", out)
+	}
+}
+
+// ownedSource adds owners to fakeSource.
+type ownedSource struct{ fakeSource }
+
+func (ownedSource) ModelOwners() map[string]string { return map[string]string{"chat": "acme/ml"} }
+
+func TestEachGPUAReplicaHoldsSaysWhoseItIs(t *testing.T) {
+	src := ownedSource{fakeSource{replicas: []reconciler.Replica{
+		{ID: "chat-1", Model: "chat", State: reconciler.Ready, Shards: []reconciler.Shard{{NodeID: "g1", GPUIndices: []int{0, 1}}}},
+		{ID: "old-1", Model: "old", State: reconciler.Stopped, Shards: []reconciler.Shard{{NodeID: "g1", GPUIndices: []int{2}}}},
+	}}}
+	out := gather(t, src)
+	for _, want := range []string{
+		"llmgw_gpu_assigned gpu=0 model=chat node=g1 owner=acme/ml replica=chat-1 -> 1",
+		"llmgw_gpu_assigned gpu=1 model=chat node=g1 owner=acme/ml replica=chat-1 -> 1",
+	} {
+		if !strings.Contains(out, want) {
+			t.Errorf("missing %s in:\n%s", want, out)
+		}
+	}
+	if strings.Contains(out, "replica=old-1") {
+		t.Error("a stopped replica's GPU was counted as held")
+	}
+}
+
+func TestAGPUsWearIsPublishedAndWhatItDoesNotKeepIsLeftOut(t *testing.T) {
+	n := inventory.Node{ID: "g1", State: inventory.Ready,
+		GPUs: []discovery.GPU{{Index: 0, Name: "NVIDIA H100 80GB HBM3", MemoryTotalMB: 81559}},
+		Load: &inventory.Load{At: time.Now(), GPUs: []inventory.GPUSensor{{Index: 0, TempC: 40, MemoryTotalMB: 81559,
+			Wear: &inventory.GPUWear{PCIeReplays: 12, RetiredSBE: -1, RetiredDBE: -1, RemappedCorrectable: 2,
+				RemappedUncorrectable: 1, RemapPending: true, NVLinkCRC: 7, XIDs: 3, LastXID: 79}}}}}
+	out := gather(t, fakeSource{nodes: []inventory.Node{n}})
+	for _, want := range []string{
+		"llmgw_gpu_pcie_replays class=h100-80g gpu=0 node=g1 -> 12\n",
+		"llmgw_gpu_remapped_rows class=h100-80g gpu=0 kind=uncorrectable node=g1 -> 1\n",
+		"llmgw_gpu_remap_pending class=h100-80g gpu=0 node=g1 -> 1\n",
+		"llmgw_gpu_nvlink_errors class=h100-80g gpu=0 kind=crc node=g1 -> 7\n",
+		"llmgw_gpu_xid_errors class=h100-80g gpu=0 node=g1 -> 3\n",
+		"llmgw_gpu_last_xid class=h100-80g gpu=0 node=g1 -> 79\n",
+	} {
+		if !strings.Contains(out, want) {
+			t.Errorf("missing %s", want)
+		}
+	}
+	// An H100 keeps no retired pages; none are published.
+	if strings.Contains(out, "llmgw_gpu_retired_pages") {
+		t.Errorf("a count the card does not keep was published:\n%s", out)
+	}
+}

internal/control/inventory/inventory.go

@@ -3,6 +3,7 @@
 package inventory
 
 import (
+	"fmt"
 	"maps"
 	"slices"
 	"strings"
@@ -319,8 +320,74 @@ type GPUSensor struct {
 	PowerW      float64  `json:"power_w,omitempty"`
 	PowerLimitW float64  `json:"power_limit_w,omitempty"`
 	Throttle    []string `json:"throttle,omitempty"`
+	// How busy the card is, as of this sample; -1 is not reported. Zero
+	// MemoryTotalMB is an agent too old to send these, whose figures are
+	// the ones it read at registration.
+	UtilPercent    float64 `json:"util_percent,omitempty"`
+	MemUtilPercent float64 `json:"mem_util_percent,omitempty"`
+	MemoryUsedMB   float64 `json:"memory_used_mb,omitempty"`
+	MemoryTotalMB  float64 `json:"memory_total_mb,omitempty"`
+	SMClockMHz     float64 `json:"sm_clock_mhz,omitempty"`
+	ECCUncorrected float64 `json:"ecc_uncorrected,omitempty"`
+	// Wear is what wears the card out, read every minute; nil from an agent
+	// that does not read it.
+	Wear *GPUWear `json:"wear,omitempty"`
 }
 
+// GPUWear is a card's wear and failures. Counts of -1 are not reported for
+// it: a card before Ampere has no remapped rows, one after no retired pages.
+type GPUWear struct {
+	PCIeReplays           int64     `json:"pcie_replays"`
+	RetiredSBE            int64     `json:"retired_pages_sbe"`
+	RetiredDBE            int64     `json:"retired_pages_dbe"`
+	RetirePending         bool      `json:"retire_pending,omitempty"`
+	RemappedCorrectable   int64     `json:"remapped_rows_correctable"`
+	RemappedUncorrectable int64     `json:"remapped_rows_uncorrectable"`
+	RemapPending          bool      `json:"remap_pending,omitempty"`
+	RemapFailure          bool      `json:"remap_failure,omitempty"`
+	NVLinkCRC             int64     `json:"nvlink_crc_errors"`
+	NVLinkReplay          int64     `json:"nvlink_replay_errors"`
+	NVLinkRecovery        int64     `json:"nvlink_recovery_errors"`
+	XIDs                  int64     `json:"xid_errors"`
+	LastXID               int       `json:"last_xid,omitempty"`
+	LastXIDText           string    `json:"last_xid_text,omitempty"`
+	LastXIDAt             time.Time `json:"last_xid_at,omitempty"`
+}
+
+// Problems says, in a few words each, what is wrong with the card; empty
+// when nothing is.
+func (w *GPUWear) Problems() []string {
+	if w == nil {
+		return nil
+	}
+	var out []string
+	if w.RemapFailure {
+		out = append(out, "out of spare memory rows: replace the card")
+	}
+	if w.RemapPending {
+		out = append(out, "memory rows remapped, waiting for a reset")
+	}
+	if w.RetirePending {
+		out = append(out, "memory pages waiting for a reset to be retired")
+	}
+	if w.RemappedUncorrectable > 0 {
+		out = append(out, fmt.Sprintf("%d rows remapped for uncorrectable errors", w.RemappedUncorrectable))
+	}
+	if w.RetiredDBE > 0 {
+		out = append(out, fmt.Sprintf("%d pages retired for double-bit errors", w.RetiredDBE))
+	}
+	if w.XIDs > 0 {
+		out = append(out, fmt.Sprintf("XID %d (%s)", w.LastXID, w.LastXIDText))
+	}
+	if w.NVLinkCRC > 0 || w.NVLinkReplay > 0 || w.NVLinkRecovery > 0 {
+		out = append(out, fmt.Sprintf("NVLink errors: %d CRC, %d replay, %d recovery", w.NVLinkCRC, w.NVLinkReplay, w.NVLinkRecovery))
+	}
+	return out
+}
+
+// Busy reports whether the sample carries how busy the card is.
+func (s GPUSensor) Busy() bool { return s.MemoryTotalMB > 0 }
+
 // ShardLoad is one engine's use: cores busy, and resident memory.
 type ShardLoad struct {
 	CPUCores float64

internal/docgen/docgen.go

@@ -29,19 +29,30 @@ type Field struct {
 	Key  string
 	Type string
 	Doc  string
-	// Nested names the struct this key expands into, empty for a leaf. It is
-	// what lets the reference say "see Auth" rather than printing the same
-	// twenty keys under four parents.
+	// Nested names the struct this key expands into, by its ID
+	// (config.Auth), empty for a leaf. It is what lets the reference say
+	// "see Auth" rather than printing the same twenty keys under four
+	// parents.
 	Nested string
 }
 
 // Block is a struct that appears in configuration.
 type Block struct {
-	Name   string
+	// Package and Name are where the struct is declared: what tells
+	// tlsconf.Config from the agent's Config.
+	Package string
+	Name    string
+	// Title is what it is called in the reference and on the screens: the
+	// name alone where no other block shares it, the package's name with it
+	// where one does.
+	Title  string
 	Doc    string
 	Fields []Field
 }
 
+// ID is the block's package and name, unique where Name is not.
+func (b Block) ID() string { return b.Package + "." + b.Name }
+
 // Collect reads koanf-tagged structs out of Go source files.
 func Collect(files []string) ([]Block, error) {
 	fset := token.NewFileSet()
@@ -62,22 +73,36 @@ func Collect(files []string) ([]Block, error) {
 			if !ok {
 				return true
 			}
-			b, ok := blockOf(ts, st)
-			if !ok || seen[b.Name] {
+			b, ok := blockOf(f.Name.Name, ts, st)
+			if !ok || seen[b.ID()] {
 				return true
 			}
-			seen[b.Name] = true
+			seen[b.ID()] = true
 			blocks = append(blocks, b)
 			return true
 		})
 	}
-	sort.Slice(blocks, func(i, j int) bool { return blocks[i].Name < blocks[j].Name })
+	// Named by their own name unless another block has it too. Keyed by
+	// name alone, tlsconf.Config, engines.Config and supervisor.Config were
+	// each taken for the agent's Config and dropped: the reference had no
+	// TLS settings at all.
+	count := map[string]int{}
+	for _, b := range blocks {
+		count[b.Name]++
+	}
+	for i := range blocks {
+		blocks[i].Title = blocks[i].Name
+		if count[blocks[i].Name] > 1 {
+			blocks[i].Title = blocks[i].ID()
+		}
+	}
+	sort.Slice(blocks, func(i, j int) bool { return blocks[i].Title < blocks[j].Title })
 	return blocks, nil
 }
 
 // blockOf builds a Block, or reports that this struct carries no settings.
-func blockOf(ts *ast.TypeSpec, st *ast.StructType) (Block, bool) {
-	b := Block{Name: ts.Name.Name, Doc: docText(ts.Doc)}
+func blockOf(pkg string, ts *ast.TypeSpec, st *ast.StructType) (Block, bool) {
+	b := Block{Package: pkg, Name: ts.Name.Name, Doc: docText(ts.Doc)}
 	for _, f := range st.Fields.List {
 		if f.Tag == nil {
 			continue
@@ -90,7 +115,7 @@ func blockOf(ts *ast.TypeSpec, st *ast.StructType) (Block, bool) {
 		if !ok || key == "" || key == "-" {
 			continue
 		}
-		typ, nested := typeOf(f.Type)
+		typ, nested := typeOf(pkg, f.Type)
 		b.Fields = append(b.Fields, Field{
 			Key: key, Type: typ, Doc: docText(f.Doc), Nested: nested,
 		})
@@ -99,29 +124,34 @@ func blockOf(ts *ast.TypeSpec, st *ast.StructType) (Block, bool) {
 }
 
 // typeOf renders a field's type, and names the struct it expands into.
-func typeOf(e ast.Expr) (rendered, nested string) {
+func typeOf(pkg string, e ast.Expr) (rendered, nested string) {
 	switch t := e.(type) {
 	case *ast.Ident:
 		if isNamedStruct(t.Name) {
-			return t.Name, t.Name
+			return t.Name, pkg + "." + t.Name
 		}
 		return t.Name, ""
 	case *ast.SelectorExpr:
-		// A type from another package, such as tlsconf.Config.
+		// A type from another package, such as tlsconf.Config. Standard
+		// library types are values, not blocks: a time.Duration is a
+		// setting, written "30s".
 		name := t.Sel.Name
-		if pkg, ok := t.X.(*ast.Ident); ok {
-			return pkg.Name + "." + name, name
+		if x, ok := t.X.(*ast.Ident); ok {
+			if x.Name == "time" {
+				return x.Name + "." + name, ""
+			}
+			return x.Name + "." + name, x.Name + "." + name
 		}
-		return name, name
+		return name, ""
 	case *ast.ArrayType:
-		inner, _ := typeOf(t.Elt)
+		inner, _ := typeOf(pkg, t.Elt)
 		return "[]" + inner, ""
 	case *ast.MapType:
-		k, _ := typeOf(t.Key)
-		v, _ := typeOf(t.Value)
+		k, _ := typeOf(pkg, t.Key)
+		v, _ := typeOf(pkg, t.Value)
 		return "map[" + k + "]" + v, ""
 	case *ast.StarExpr:
-		inner, n := typeOf(t.X)
+		inner, n := typeOf(pkg, t.X)
 		return "*" + inner, n
 	default:
 		return "value", ""

internal/docgen/docgen_test.go

@@ -136,3 +136,18 @@ func TestAFieldMarkedNotConfigurableIsLeftOut(t *testing.T) {
 		}
 	}
 }
+
+func TestTheSettingsScreensDescribeTheCode(t *testing.T) {
+	// The screens list Reference; it must be what the structs say now.
+	blocks, err := Collect(sourcePaths(t))
+	if err != nil {
+		t.Fatal(err)
+	}
+	have, err := os.ReadFile(filepath.Join(repoRoot(t), "internal", "docgen", "reference_gen.go"))
+	if err != nil {
+		t.Fatalf("reference_gen.go is missing; run go generate ./...: %v", err)
+	}
+	if string(have) != RenderGo(blocks) {
+		t.Error("internal/docgen/reference_gen.go no longer matches the configuration structs; run go generate ./...")
+	}
+}

internal/docgen/gen/main.go

@@ -38,5 +38,9 @@ func main() {
 		fmt.Fprintln(os.Stderr, "error:", err)
 		os.Exit(1)
 	}
-	fmt.Printf("docs/reference.md: %d blocks\n", len(blocks))
+	if err := os.WriteFile(filepath.Join(root, "internal", "docgen", "reference_gen.go"), []byte(docgen.RenderGo(blocks)), 0o644); err != nil {
+		fmt.Fprintln(os.Stderr, "error:", err)
+		os.Exit(1)
+	}
+	fmt.Printf("docs/reference.md and internal/docgen/reference_gen.go: %d blocks\n", len(blocks))
 }

internal/docgen/reference_gen.go

@@ -0,0 +1,302 @@
+// Code generated by go run ./internal/docgen/gen; DO NOT EDIT.
+
+package docgen
+
+// Reference is every configuration block, as Collect reads them.
+var Reference = []Block{
+	{Package: "catalog", Name: "Adapter", Title: "Adapter", Doc: "", Fields: []Field{
+		{Key: "name", Type: "string", Doc: "Name is what clients ask for, like a model name.", Nested: ""},
+		{Key: "source", Type: "string", Doc: "Source is the adapter file (GGUF for llama.cpp), fetched by each node\nas the model's weights are.", Nested: ""},
+		{Key: "scale", Type: "float64", Doc: "Scale is how strongly it is applied; zero takes 1.", Nested: ""},
+	}},
+	{Package: "config", Name: "Admission", Title: "Admission", Doc: "", Fields: []Field{
+		{Key: "enabled", Type: "bool", Doc: "Enabled turns the layer on. Off, every request goes straight through,\nwhich is the v0.2.0 behaviour.", Nested: ""},
+		{Key: "bands", Type: "[]Band", Doc: "", Nested: ""},
+		{Key: "default_band", Type: "string", Doc: "", Nested: ""},
+		{Key: "band_header", Type: "string", Doc: "BandHeader lets a client declare its band. Until authentication lands in\nv1.0.0 this is trusted, so the port must not be exposed to callers who\nshould not be able to promote themselves.", Nested: ""},
+		{Key: "queue_ttl", Type: "time.Duration", Doc: "", Nested: ""},
+		{Key: "queue_capacity_per_band", Type: "int", Doc: "", Nested: ""},
+		{Key: "queue_capacity_total", Type: "int", Doc: "QueueCapacityTotal bounds everything queued across bands. Reached, a\nrequest is not refused while a lower band has any queued: the newest of\nthe lowest band is turned away instead, with a 503 and Retry-After.\nZero takes half of all the bands' capacities together.", Nested: ""},
+		{Key: "interval", Type: "time.Duration", Doc: "", Nested: ""},
+	}},
+	{Package: "config", Name: "Auth", Title: "Auth", Doc: "", Fields: []Field{
+		{Key: "enabled", Type: "bool", Doc: "", Nested: ""},
+		{Key: "store", Type: "string", Doc: "Store is the SQLite file holding keys, tenancy and usage.", Nested: ""},
+		{Key: "cache_ttl", Type: "time.Duration", Doc: "CacheTTL bounds how long a revoked key keeps working, traded against a\ndatabase read per key per TTL.", Nested: ""},
+		{Key: "negative_ttl", Type: "time.Duration", Doc: "NegativeTTL is how long a rejected key is remembered as rejected, so a\nclient looping with a bad key cannot become a database read per request.", Nested: ""},
+		{Key: "default_rpm", Type: "int", Doc: "DefaultRPM and DefaultTPM apply to a key that no scope gives a limit to.\n\nUnlimited used to be the default, and a project created without limits is\nthe easy path — the \"default\" project on this cluster still is one. The\nfirst sign of that is the bill or the outage. Zero keeps the old\nbehaviour for anyone who wants it, but it should be a choice rather than\nan omission. A fallback, not a ceiling: an explicit limit at any scope\nwins, above or below.", Nested: ""},
+		{Key: "default_tpm", Type: "int", Doc: "", Nested: ""},
+		{Key: "signup_open", Type: "bool", Doc: "SignupOpen serves the signup form. Accounts it creates are pending until\nan operator approves them, so this is not a way in — but turning it off\nmeans accounts are made with `llmgw users create` instead.", Nested: ""},
+		{Key: "personal", Type: "Personal", Doc: "Personal is what a person may do on their own, outside any project: make\npersonal keys and use the chat page personally, within these limits.", Nested: "config.Personal"},
+		{Key: "pricing", Type: "map[string]Price", Doc: "Pricing is what each model costs, for budgets. A model absent from the\ntable costs whatever DefaultPrice says, which is nothing unless set.", Nested: ""},
+		{Key: "default_price", Type: "Price", Doc: "", Nested: "config.Price"},
+		{Key: "flush_interval", Type: "time.Duration", Doc: "FlushInterval and BatchSize decide when usage records are written.", Nested: ""},
+		{Key: "batch_size", Type: "int", Doc: "", Nested: ""},
+		{Key: "buffer_size", Type: "int", Doc: "BufferSize bounds the queue. Beyond it records are dropped rather than\nblocking a request: losing accounting is bad, turning a stalled database\ninto an outage is worse.", Nested: ""},
+		{Key: "usage_retention", Type: "time.Duration", Doc: "UsageRetention is how long the record of each request is kept: who sent\nit, which replica answered, how long it queued — what `ctl request`\nreads. Zero keeps it for ever. Totals, charts and budgets are read from\nfive-minute sums kept beside the records, which are not pruned, so this\nbounds the table that grows by a row a request without changing any\nfigure a page shows. At least a day when set.", Nested: ""},
+	}},
+	{Package: "catalog", Name: "Capabilities", Title: "Capabilities", Doc: "", Fields: []Field{
+		{Key: "tools", Type: "bool", Doc: "", Nested: ""},
+		{Key: "vision", Type: "bool", Doc: "", Nested: ""},
+		{Key: "embeddings", Type: "bool", Doc: "", Nested: ""},
+	}},
+	{Package: "config", Name: "Control", Title: "Control", Doc: "", Fields: []Field{
+		{Key: "interval", Type: "time.Duration", Doc: "Interval is how often desired and actual state are compared.", Nested: ""},
+		{Key: "base_backoff", Type: "time.Duration", Doc: "BaseBackoff and MaxBackoff bound how fast a failing model is retried.", Nested: ""},
+		{Key: "max_backoff", Type: "time.Duration", Doc: "", Nested: ""},
+		{Key: "unreachable_after", Type: "time.Duration", Doc: "UnreachableAfter and LostAfter stage node expiry. They are deliberately\nfar apart: a network blip must not tear down replicas that still serve,\nand rebuilding a multi-node replica costs tens of minutes of loading.", Nested: ""},
+		{Key: "lost_after", Type: "time.Duration", Doc: "", Nested: ""},
+		{Key: "port_range", Type: "[]int", Doc: "PortRange is the per-node range for engine HTTP and dist-init ports.", Nested: ""},
+		{Key: "mgmt_interface", Type: "string", Doc: "MgmtInterface is the ethernet interface NCCL bootstraps over.", Nested: ""},
+		{Key: "enabled", Type: "bool", Doc: "Enabled runs the control plane even when configuration declares no\nmodels.\n\nWithout it, a deployment whose catalog lives entirely in the database\ncannot start: control plane mode is inferred from the config file, so an\nempty models block means \"router only\", the process refuses to start for\nwant of anything to serve, and the models it would have loaded are never\nread. That is a cluster that cannot be managed at runtime because it is\nmanaged at runtime. AgentListen cannot carry this signal — it has a\ndefault, so every deployment would become a control plane.", Nested: ""},
+		{Key: "lease_ttl", Type: "time.Duration", Doc: "LeaseTTL is how long this control plane's claim to be the one placing\nwork stands without being renewed.\n\nOnly meaningful with a shared store, where a second control plane can\nsee the claim. Long enough to survive a slow query or a brief network\nstall, short enough that a standby takes over before anyone has finished\nreading the alert. Zero takes the default.", Nested: ""},
+	}},
+	{Package: "config", Name: "Endpoint", Title: "Endpoint", Doc: "", Fields: []Field{
+		{Key: "id", Type: "string", Doc: "", Nested: ""},
+		{Key: "url", Type: "string", Doc: "", Nested: ""},
+		{Key: "api_key", Type: "string", Doc: "APIKey is sent to the engine as a bearer token, for an engine started\nwith one.", Nested: ""},
+		{Key: "models", Type: "[]string", Doc: "", Nested: ""},
+		{Key: "engine", Type: "string", Doc: "Engine names the inference engine, which decides what its metrics are\ncalled. Empty means vLLM's names.", Nested: ""},
+		{Key: "pool", Type: "string", Doc: "", Nested: ""},
+		{Key: "weight", Type: "float64", Doc: "", Nested: ""},
+		{Key: "priority", Type: "int", Doc: "", Nested: ""},
+	}},
+	{Package: "config", Name: "EngineOverride", Title: "EngineOverride", Doc: "", Fields: []Field{
+		{Key: "gpu_class", Type: "string", Doc: "GPUClass picks nodes with a GPU of this class (\"v100-32g\").", Nested: ""},
+		{Key: "nodes", Type: "[]string", Doc: "Nodes picks nodes by id.", Nested: ""},
+		{Key: "labels", Type: "map[string]string", Doc: "Labels picks nodes carrying all of these.", Nested: ""},
+		{Key: "engines", Type: "map[string]string", Doc: "Engines are laid over the cluster's: a version, or \"\" for none.", Nested: ""},
+	}},
+	{Package: "catalog", Name: "External", Title: "External", Doc: "", Fields: []Field{
+		{Key: "url", Type: "string", Doc: "URL is the provider's base address, without /v1: https://api.openai.com.", Nested: ""},
+		{Key: "model", Type: "string", Doc: "Model is what the provider calls it; requests are rewritten to it.", Nested: ""},
+		{Key: "api_key_file", Type: "string", Doc: "APIKeyFile holds the provider's key, on the control plane. A file\nrather than the key itself, so the key is in neither the catalog, the\nstore nor the audit log; empty for a provider that asks for none.", Nested: ""},
+	}},
+	{Package: "config", Name: "Landing", Title: "Landing", Doc: "", Fields: []Field{
+		{Key: "contact", Type: "string", Doc: "Contact is where someone without a key should go — an address, a URL, a\nchannel name. Rendered as a link when it is one, as text otherwise.", Nested: ""},
+		{Key: "notice", Type: "string", Doc: "Notice is an optional line of your own, for whatever the reader needs to\nknow that this page cannot guess.", Nested: ""},
+	}},
+	{Package: "engines", Name: "LlamaCppConfig", Title: "LlamaCppConfig", Doc: "", Fields: []Field{
+		{Key: "bin", Type: "string", Doc: "Bin is the llama-server binary. It is the server, not the CLI: the\ngateway talks OpenAI over HTTP and needs the one that serves it.\nEmpty means the agent installs llama.cpp itself, at the version the\ncontrol plane asks for. Setting it opts out of that.", Nested: ""},
+		{Key: "parallel", Type: "int", Doc: "Parallel is how many requests a server handles at once, when a profile\ndoes not say. llama.cpp defaults to one, which would make every second\nrequest wait behind the first no matter what the router decided.", Nested: ""},
+		{Key: "threads", Type: "int", Doc: "Threads is how many CPU threads a shard with no GPU computes on. Zero\ntakes half this machine's CPUs.\n\nNot all of them, which is llama.cpp's own default: two busy servers\nthat each spin on every CPU fight for them, and on an 8-vCPU node\nthat made 7B generation fall from 10 tokens a second to under 1 each.\nAt half, one server alone is about a third slower and two together\nkeep 5.5 each. A node that only ever runs one CPU replica does better\nwith all of them; say so here.", Nested: ""},
+	}},
+	{Package: "config", Name: "Mirror", Title: "Mirror", Doc: "", Fields: []Field{
+		{Key: "api", Type: "string", Doc: "API is the mirror's management address, scheme included, for example\nhttps://models.internal:8082. Empty means no mirror is configured.\n\nThe management port, not the one nodes fetch bytes from. Those are two\nlisteners on the mirror precisely so this one can be locked down without\nre-certifying every agent's fetcher.", Nested: ""},
+		{Key: "tls", Type: "tlsconf.Config", Doc: "TLS authenticates to the mirror's management API, with the same CA the\nagents use. Empty leaves the connection in plain HTTP, which is only\nacceptable on a network where nothing untrusted can reach that port —\nthe mirror says as much in its own log when it starts that way.", Nested: "tlsconf.Config"},
+		{Key: "registry", Type: "string", Doc: "Registry is the model registry the mirror page browses, empty for the\npublic Hugging Face. Set it to an internal mirror of the registry where\nthe control plane cannot reach the internet — or leave the whole mirror\nblock out, which turns the explorer off rather than giving it a search\nbox that can only time out.", Nested: ""},
+		{Key: "timeout", Type: "time.Duration", Doc: "Timeout bounds a management call. It does not bound a download: the\nmirror answers a fetch as soon as it has started one, and the bytes keep\narriving long after this deadline has passed.", Nested: ""},
+	}},
+	{Package: "catalog", Name: "Model", Title: "Model", Doc: "", Fields: []Field{
+		{Key: "name", Type: "string", Doc: "", Nested: ""},
+		{Key: "aliases", Type: "[]string", Doc: "", Nested: ""},
+		{Key: "description", Type: "string", Doc: "Description is what this model is for, in a line.\n\nWritten by whoever added it, because nobody else can say it. A registry\noffers a pipeline tag and a wall of README, and neither answers the\nquestion a person picking from a list is asking: which of these nine\nnames should I send this request to.\n\nIt reaches clients through /v1/models, which is the only view most of\nthem ever see. A gateway serving nine models by name alone makes that\nchoice by guesswork.", Nested: ""},
+		{Key: "source", Type: "string", Doc: "Source is the weights path or a HuggingFace id.", Nested: ""},
+		{Key: "profiles", Type: "[]Profile", Doc: "", Nested: ""},
+		{Key: "scaling", Type: "Scaling", Doc: "", Nested: "catalog.Scaling"},
+		{Key: "class", Type: "string", Doc: "Class is how much this model's capacity is worth against another's,\nwhen there is not enough for both: critical, standard or opportunistic.\nEmpty is standard, or opportunistic for a model loaded on demand. See\nEffectiveClass.", Nested: ""},
+		{Key: "owner", Type: "string", Doc: "Owner is the part of the tenancy tree whose capacity quota this model\ncounts against, written org:ID, team:ID or project:ID. Empty is shared:\nheld by the cluster, counted against nobody.", Nested: ""},
+		{Key: "external", Type: "*External", Doc: "External is a model served by someone else — an OpenAI-compatible API\noutside the cluster — rather than by replicas placed here. Such a model\nhas no profiles and nothing is placed for it; the router sends its\nrequests to the provider. Most useful named in another model's\nfallbacks: used only when the cluster has no capacity left. Its\nprompts leave the cluster, so which keys may name it is worth an\nallowlist.", Nested: "catalog.External"},
+		{Key: "adapters", Type: "[]Adapter", Doc: "Adapters are LoRA adapters served on this model's replicas, each asked\nfor by its own name as though it were a model. Every replica loads\nevery adapter, applying one only to a request that names it.", Nested: ""},
+		{Key: "fallbacks", Type: "[]string", Doc: "", Nested: ""},
+		{Key: "routing_profile", Type: "string", Doc: "RoutingProfile selects how the router scores endpoints for this model.", Nested: ""},
+		{Key: "slo", Type: "SLO", Doc: "", Nested: "catalog.SLO"},
+		{Key: "context_length", Type: "int", Doc: "ContextLength is how many tokens one request may use, prompt and\ncompletion together, and MaxOutput bounds the completion alone. Zero\nmeans unknown, which is what every model was until now.\n\nThe gateway knew neither, so an over-long prompt was not refused: it was\nadmitted, given a replica, and failed at the engine, and a client sizing\nits prompts from /v1/models had nothing to read. Worse, the number is not\nthe one an operator writes in the engine's arguments — llama.cpp divides\n--ctx-size across its parallel slots, so a pool of 8192 across four slots\nis 2048 a request, and nothing said so.", Nested: ""},
+		{Key: "max_output", Type: "int", Doc: "", Nested: ""},
+		{Key: "capabilities", Type: "Capabilities", Doc: "Capabilities is what this model can do beyond plain completion.", Nested: "catalog.Capabilities"},
+	}},
+	{Package: "config", Name: "Monitoring", Title: "Monitoring", Doc: "", Fields: []Field{
+		{Key: "prometheus", Type: "string", Doc: "Prometheus is the server's address, for example\nhttp://monitoring.internal:9090. Empty leaves the charts off the\ndashboard rather than showing an error where one should be.", Nested: ""},
+		{Key: "window", Type: "time.Duration", Doc: "Window is how far back the charts look. Zero takes an hour.", Nested: ""},
+		{Key: "timeout", Type: "time.Duration", Doc: "Timeout bounds one query. It runs while somebody waits for a page, so a\nmonitoring system that is slow today is skipped rather than waited for.", Nested: ""},
+	}},
+	{Package: "discovery", Name: "Options", Title: "Options", Doc: "", Fields: []Field{
+		{Key: "nvidia_smi", Type: "string", Doc: "NvidiaSMI is the path to nvidia-smi; empty means look it up on PATH.\nWorth configuring on nodes where the driver lives outside the default\npaths, which containerized driver installs often do.", Nested: ""},
+		{Key: "ib_root", Type: "string", Doc: "IBRoot is the sysfs InfiniBand directory.", Nested: ""},
+		{Key: "prefer_nvml", Type: "bool", Doc: "PreferNVML uses the NVML backend when this binary has one. NVLink\ntopology is read from nvidia-smi topo -m on every build; NVML, where it\nis used, is the more exact source of the peers.", Nested: ""},
+		{Key: "skip_gpus", Type: "bool", Doc: "SkipGPUs omits GPU discovery. A node without nvidia-smi is already\nfound to have none, so this is for one whose driver is broken.", Nested: ""},
+	}},
+	{Package: "config", Name: "Personal", Title: "Personal", Doc: "", Fields: []Field{
+		{Key: "disabled", Type: "bool", Doc: "Disabled refuses personal keys and personal chat; people then use the\ngateway only through projects they are members of.", Nested: ""},
+		{Key: "rpm", Type: "int", Doc: "", Nested: ""},
+		{Key: "tpm", Type: "int", Doc: "", Nested: ""},
+		{Key: "budget", Type: "float64", Doc: "Budget is in whole units per Period (\"day\" or \"month\").", Nested: ""},
+		{Key: "period", Type: "string", Doc: "", Nested: ""},
+	}},
+	{Package: "config", Name: "Prefix", Title: "Prefix", Doc: "", Fields: []Field{
+		{Key: "enabled", Type: "bool", Doc: "", Nested: ""},
+		{Key: "max_nodes", Type: "int", Doc: "MaxNodes bounds the tree, MinMatch is the shortest match worth acting on\nin bytes, and TTL is how long an entry is believed — an engine evicts on\nits own schedule, so an old record may describe a cache that is gone.", Nested: ""},
+		{Key: "min_match", Type: "int", Doc: "", Nested: ""},
+		{Key: "ttl", Type: "time.Duration", Doc: "", Nested: ""},
+		{Key: "match_ratio", Type: "float64", Doc: "MatchRatio is the share of the prompt that must match to be worth acting\non. BalanceAbs and BalanceRel are the imbalance guard, and both must trip\nbefore affinity is abandoned.", Nested: ""},
+		{Key: "balance_abs", Type: "float64", Doc: "", Nested: ""},
+		{Key: "balance_rel", Type: "float64", Doc: "", Nested: ""},
+	}},
+	{Package: "catalog", Name: "Profile", Title: "Profile", Doc: "", Fields: []Field{
+		{Key: "name", Type: "string", Doc: "", Nested: ""},
+		{Key: "engine", Type: "string", Doc: "", Nested: ""},
+		{Key: "gpu_class", Type: "string", Doc: "", Nested: ""},
+		{Key: "tp", Type: "int", Doc: "", Nested: ""},
+		{Key: "pp", Type: "int", Doc: "", Nested: ""},
+		{Key: "gpus_per_node", Type: "int", Doc: "GPUsPerNode defaults to TP for a single-node profile.", Nested: ""},
+		{Key: "nodes", Type: "int", Doc: "Nodes above 1 makes this a multi-node profile, which requires a fabric.", Nested: ""},
+		{Key: "args", Type: "[]string", Doc: "", Nested: ""},
+		{Key: "env", Type: "map[string]string", Doc: "", Nested: ""},
+		{Key: "min_memory_gb", Type: "int", Doc: "MinMemoryGB is the host RAM this profile needs to run.\n\nThe number that matters is the model's working set: the weights, plus the\nKV cache for the whole context pool, plus the engine's own overhead. For\na GPU profile that lives in the cards and this is small; for a CPU\nprofile it is the whole thing, and getting it wrong means an OOM on a\nnode that reported itself healthy.\n\nZero means unconstrained, which is every profile that existed before\nthis. A node that does not report its memory is not excluded either —\nan older agent must not have its nodes quietly emptied by an upgrade of\nthe control plane.", Nested: ""},
+		{Key: "cpu_cores", Type: "int", Doc: "CPUCores is the cores a replica of a profile with no GPU is given:\nreserved out of its node's CPUs by placement, as memory is, and what\nits engine computes on (llama.cpp's --threads).\n\nReserved because an engine takes every core it is told about, and two\non one node that each think they have all of them spin against each\nother: on an 8-vCPU node, two 7B replicas fell from 10 tokens a second\nto under one each. Zero reserves nothing and leaves the thread count to\nthe node, which is every profile from before this.", Nested: ""},
+		{Key: "min_disk_gb", Type: "int", Doc: "MinDiskGB is the free space a node needs before it can hold these\nweights.\n\nOnly checked when the node does not already have them: a node holding\nthe file needs no room for it, and requiring the space twice would keep\na model off the one node best placed to run it.\n\nDeclared rather than derived, because the size of a model that has never\nbeen fetched anywhere is not knowable from here — and a wrong guess\nfails late, halfway through a download, which is the failure this exists\nto prevent. Zero means unconstrained, and a node that reports no storage\nis not excluded: unknown is not full.", Nested: ""},
+		{Key: "node_selector", Type: "map[string]string", Doc: "NodeSelector restricts this profile to nodes carrying every one of these\nlabels, with these values.\n\nA CPU profile otherwise fits everywhere, because there is no card to be\nfree — which is right for a 0.5B model and wrong for a 14B one. Nothing\nin the inventory says how much memory a node has, so the operator says\nwhich nodes are meant for a model instead of the scheduler inferring it.\nAgents have reported labels since the beginning; this is what reads them.", Nested: ""},
+		{Key: "weight", Type: "float64", Doc: "Weight is the router's capacity weight. It should come from ctl bench\nrather than a guess (plan §6.2.1).", Nested: ""},
+		{Key: "priority", Type: "int", Doc: "Priority orders profiles during placement; highest that fits wins.", Nested: ""},
+		{Key: "startup_timeout", Type: "time.Duration", Doc: "StartupTimeout bounds model loading. A 70B model across two nodes takes\ntens of minutes, so this is per profile rather than global.", Nested: ""},
+	}},
+	{Package: "config", Name: "Router", Title: "Router", Doc: "", Fields: []Field{
+		{Key: "activation_wait", Type: "time.Duration", Doc: "ActivationWait is how long a request waits for an on-demand model to\nload before the client is told to come back. Zero takes the default.\n\nThe one real decision in on-demand loading. Refusing outright makes the\nfeature useless, since the first request for every model would fail;\nwaiting without a bound turns a request into a job, and a client that\nbudgeted thirty seconds ends up holding a connection for ten minutes.", Nested: ""},
+		{Key: "scrape_interval", Type: "time.Duration", Doc: "", Nested: ""},
+		{Key: "scrape_timeout", Type: "time.Duration", Doc: "", Nested: ""},
+		{Key: "stale_after", Type: "time.Duration", Doc: "StaleAfter is how long an endpoint may go unscraped before being scored\ndown (plan §7.6).", Nested: ""},
+		{Key: "retries", Type: "int", Doc: "Retries is how many *additional* endpoints may be tried after a failure,\nand only while nothing has been written to the client (plan §7.7).", Nested: ""},
+		{Key: "cooldown", Type: "time.Duration", Doc: "Cooldown is how long a failing endpoint leaves the candidate pool.", Nested: ""},
+		{Key: "failure_threshold", Type: "int", Doc: "FailureThreshold is the consecutive failure count that opens a breaker.\n1 turns the breaker into a pure cooldown.", Nested: ""},
+		{Key: "default_profile", Type: "string", Doc: "DefaultProfile names the routing profile used when a model picks none.", Nested: ""},
+		{Key: "prefix", Type: "Prefix", Doc: "Prefix turns on the approximate prefix tree (plan §7.5 tier 1). Off, the\nrouter uses session affinity alone, which is tier 0 and already most of\nthe win for multi-turn traffic.", Nested: "config.Prefix"},
+		{Key: "session_header", Type: "string", Doc: "SessionHeader carries the session id for affinity (plan §7.5 tier 0).", Nested: ""},
+		{Key: "inject_usage", Type: "bool", Doc: "InjectUsage asks engines for usage even when the client did not, so token\naccounting works regardless of client behaviour.", Nested: ""},
+		{Key: "tenant_label", Type: "string", Doc: "TenantLabel is the level of the tree the per-tenant request series are\nlabelled by — llmgw_tenant_requests_total and its siblings: \"org\",\n\"team\" or \"project\", or \"none\" for no such series. A label per project\non a cluster of thousands is more series than Prometheus wants, and a\nteam is what an alert is usually about; \"team\" is the default.", Nested: ""},
+		{Key: "max_body_bytes", Type: "int64", Doc: "MaxBodyBytes bounds the request body the router will read.", Nested: ""},
+		{Key: "timeouts", Type: "Timeouts", Doc: "", Nested: "config.Timeouts"},
+		{Key: "scoring", Type: "Scoring", Doc: "", Nested: "config.Scoring"},
+		{Key: "admission", Type: "Admission", Doc: "", Nested: "config.Admission"},
+	}},
+	{Package: "engines", Name: "SGLangConfig", Title: "SGLangConfig", Doc: "", Fields: []Field{
+		{Key: "python", Type: "string", Doc: "Python is the interpreter of the SGLang virtualenv. SGLang is launched as\na module rather than a console script, which is the documented entry\npoint and avoids depending on the script being on PATH.", Nested: ""},
+	}},
+	{Package: "catalog", Name: "SLO", Title: "SLO", Doc: "", Fields: []Field{
+		{Key: "ttft_p99", Type: "time.Duration", Doc: "", Nested: ""},
+		{Key: "tpot_p99", Type: "time.Duration", Doc: "", Nested: ""},
+	}},
+	{Package: "catalog", Name: "Scaling", Title: "Scaling", Doc: "", Fields: []Field{
+		{Key: "min", Type: "int", Doc: "", Nested: ""},
+		{Key: "max", Type: "int", Doc: "", Nested: ""},
+		{Key: "idle_timeout", Type: "time.Duration", Doc: "IdleTimeout is how long an on-demand model may go unasked-for before its\nreplicas are stopped. Zero takes the cluster default.\n\nMinutes rather than seconds, always. Stopping a replica throws away the\nKV cache prefix routing spent its effort filling, and a model that flaps\nin and out loses that entirely — so the saving has to be worth more than\nthe cache, which it only is over a real idle period.", Nested: ""},
+	}},
+	{Package: "config", Name: "Scoring", Title: "Scoring", Doc: "", Fields: []Field{
+		{Key: "target_waiting", Type: "float64", Doc: "", Nested: ""},
+		{Key: "target_inflight", Type: "float64", Doc: "", Nested: ""},
+		{Key: "kv_ceiling", Type: "float64", Doc: "", Nested: ""},
+		{Key: "session_load_factor", Type: "float64", Doc: "", Nested: ""},
+	}},
+	{Package: "config", Name: "Server", Title: "Server", Doc: "", Fields: []Field{
+		{Key: "listen", Type: "string", Doc: "Listen is the client-facing OpenAI API address.", Nested: ""},
+		{Key: "admin_listen", Type: "string", Doc: "AdminListen serves the dashboard, the management API, health and pprof.\n\nLoopback by default, and it should stay there: pprof can dump the process\nheap, and API keys are in that heap in plaintext. Every comment beside\nthis setting has always said not to expose it, while the default bound\nevery interface — a default that contradicts its own documentation is a\nfootgun, so the default now matches the advice.", Nested: ""},
+		{Key: "metrics_listen", Type: "string", Doc: "MetricsListen serves Prometheus metrics and nothing else.\n\nSeparate from AdminListen because the reason that one is loopback is\npprof, not metrics — and metrics are the one thing on it that something\nelse genuinely needs to reach. Sharing a listener meant a fleet could\neither be monitored or keep its heap private, which is not a choice\nanybody should have to make.\n\nEmpty means off, and metrics remain available on AdminListen either way,\nso nothing that scrapes it today has to change.", Nested: ""},
+		{Key: "agent_listen", Type: "string", Doc: "AgentListen is where node agents connect. Separate from the client port\nso the fleet-facing surface can be firewalled independently.", Nested: ""},
+		{Key: "auth", Type: "Auth", Doc: "Auth turns on API keys on the client-facing port.", Nested: "config.Auth"},
+		{Key: "threads", Type: "Threads", Doc: "Threads keeps conversations and agent memory without API keys, for a\ngateway on one trusted machine. With auth on they live in the auth store,\nper project, and this block is ignored.", Nested: "config.Threads"},
+		{Key: "landing", Type: "Landing", Doc: "Landing configures the page served at \"/\".", Nested: "config.Landing"},
+		{Key: "agent_tls", Type: "tlsconf.Config", Doc: "AgentTLS authenticates the fleet-facing port. It deliberately does not\ncover the client-facing one: that surface is fronted by a publicly\ntrusted certificate, while this one is mutual and private.", Nested: "tlsconf.Config"},
+		{Key: "monitoring", Type: "Monitoring", Doc: "Monitoring is where the dashboard reads its history from.", Nested: "config.Monitoring"},
+		{Key: "tracing", Type: "Tracing", Doc: "Tracing sends a trace of each inference request to an OpenTelemetry\ncollector. Off unless an endpoint is set here or in\nOTEL_EXPORTER_OTLP_ENDPOINT.", Nested: "config.Tracing"},
+		{Key: "mirror", Type: "Mirror", Doc: "Mirror points the control plane at a model mirror it can administer.\nEmpty means there is none, and /admin/mirror stays unmounted.", Nested: "config.Mirror"},
+		{Key: "engines", Type: "map[string]string", Doc: "Engines are the engines every node installs, and at which version. The\nagents install them themselves — the right build for their hardware,\nchecked against its publisher's checksum and seen to run — so a node is\nnever trusted to already have what somebody once put there.\n\nUnset keeps DefaultEngines. A version replaces the default; an empty\none leaves that engine to be installed by hand.", Nested: ""},
+		{Key: "engine_overrides", Type: "[]EngineOverride", Doc: "EngineOverrides give some nodes different engines from the rest: a\nvLLM built for CUDA 12 on the Volta cards CUDA 13 dropped, or none at\nall on nodes meant only for llama.cpp. Each picks nodes and lays its\nengines over the cluster's, in order, later ones last; an empty version\nremoves that engine from the nodes it picks.", Nested: ""},
+		{Key: "backup_dir", Type: "string", Doc: "BackupDir is where `llmgw server backup` writes, and where the newest\narchive's age is read for llmgw_backup_last_success_timestamp_seconds.\nEmpty means /var/backups/llmgw-server.", Nested: ""},
+		{Key: "releases", Type: "string", Doc: "Releases is where `llmgw ctl upgrade` fetches release binaries from,\nfor this machine and — through their agents — every node. Empty means\nthe project's own releases.", Nested: ""},
+		{Key: "router", Type: "Router", Doc: "", Nested: "config.Router"},
+		{Key: "control", Type: "Control", Doc: "", Nested: "config.Control"},
+		{Key: "models", Type: "[]catalog.Model", Doc: "", Nested: ""},
+		{Key: "endpoints", Type: "[]Endpoint", Doc: "Endpoints is the phase-1 static replica list. It is only consulted when\nno model declares a profile, which is what makes a router-only smoke test\nagainst sim-engines possible without a control plane.", Nested: ""},
+	}},
+	{Package: "engines", Name: "SimConfig", Title: "SimConfig", Doc: "", Fields: []Field{
+		{Key: "bin", Type: "string", Doc: "Bin is the llmgw binary itself; the engine is one of its subcommands.\nEmpty means whatever started this agent, which is right almost always.", Nested: ""},
+	}},
+	{Package: "config", Name: "Threads", Title: "Threads", Doc: "", Fields: []Field{
+		{Key: "store", Type: "string", Doc: "Store is the SQLite file. Empty means nothing is kept.", Nested: ""},
+		{Key: "retain_hours", Type: "int", Doc: "RetainHours is how long idle conversations and notes are kept. Required\nwith Store: a store with no expiry keeps everything anyone typed, by\nomission.", Nested: ""},
+	}},
+	{Package: "config", Name: "Timeouts", Title: "Timeouts", Doc: "", Fields: []Field{
+		{Key: "dial", Type: "time.Duration", Doc: "", Nested: ""},
+		{Key: "response_header", Type: "time.Duration", Doc: "ResponseHeader bounds the wait for headers on a streaming request. It is\na liveness check there, because headers arrive before generation starts.", Nested: ""},
+		{Key: "generation", Type: "time.Duration", Doc: "Generation bounds a non-streaming request, where the engine sends no\nheaders until the completion is done. Setting response_header alone used\nto cap both, so 60s read as a liveness guard and behaved as a limit of\nabout a hundred tokens on a slow model.", Nested: ""},
+		{Key: "first_token", Type: "time.Duration", Doc: "", Nested: ""},
+		{Key: "idle", Type: "time.Duration", Doc: "", Nested: ""},
+		{Key: "total", Type: "time.Duration", Doc: "", Nested: ""},
+	}},
+	{Package: "config", Name: "Tracing", Title: "Tracing", Doc: "", Fields: []Field{
+		{Key: "endpoint", Type: "string", Doc: "Endpoint is the collector, for example http://otel.internal:4318.\nEmpty falls back to OTEL_EXPORTER_OTLP_ENDPOINT, and to no tracing.", Nested: ""},
+		{Key: "sample_ratio", Type: "float64", Doc: "SampleRatio is the share of requests traced, 0 to 1, when the caller\ndid not decide it with a sampled traceparent of its own. Zero takes 1.", Nested: ""},
+	}},
+	{Package: "engines", Name: "VLLMConfig", Title: "VLLMConfig", Doc: "", Fields: []Field{
+		{Key: "bin", Type: "string", Doc: "Bin is the vllm executable.", Nested: ""},
+		{Key: "launch_mode", Type: "string", Doc: "LaunchMode selects the multi-node mechanism: \"mp\", the default, or\n\"ray\".\n\nmp is what vLLM documents today — every node runs serve with its rank,\nthe workers headless — and needs nothing but vLLM. ray is the older way,\nkept for a vLLM too old for mp's multi-node flags; it needs Ray on every\nnode and has not been checked against a real cluster.", Nested: ""},
+		{Key: "ray_bin", Type: "string", Doc: "RayBin is used when LaunchMode is \"ray\".", Nested: ""},
+	}},
+	{Package: "accounting", Name: "Price", Title: "accounting.Price", Doc: "", Fields: []Field{
+		{Key: "prompt_per_1k", Type: "int64", Doc: "", Nested: ""},
+		{Key: "completion_per_1k", Type: "int64", Doc: "", Nested: ""},
+	}},
+	{Package: "admission", Name: "Band", Title: "admission.Band", Doc: "", Fields: []Field{
+		{Key: "name", Type: "string", Doc: "", Nested: ""},
+		{Key: "priority", Type: "int", Doc: "Priority orders bands; higher is served first.", Nested: ""},
+		{Key: "saturation_ceiling", Type: "float64", Doc: "SaturationCeiling is the pool saturation above which this band waits.\n1.0 means \"never hold this band back\".", Nested: ""},
+	}},
+	{Package: "admission", Name: "Config", Title: "admission.Config", Doc: "", Fields: []Field{
+		{Key: "bands", Type: "[]Band", Doc: "", Nested: ""},
+		{Key: "default_band", Type: "string", Doc: "", Nested: ""},
+		{Key: "queue_ttl", Type: "time.Duration", Doc: "QueueTTL drops a request that has waited this long. Sending the GPU work\nthe client has already given up on helps nobody.", Nested: ""},
+		{Key: "queue_capacity_per_band", Type: "int", Doc: "QueueCapacityPerBand bounds each band's queue. Rejecting immediately\nbeats accumulating silently.", Nested: ""},
+		{Key: "queue_capacity_total", Type: "int", Doc: "QueueCapacityTotal bounds everything queued, across bands. When it is\nreached, a request is not refused while a lower band has requests\nqueued: the newest of the lowest band is turned away instead\n(displacement), since the oldest have waited longest and are nearest\ntheir TTL. Zero takes half of every band's capacity together.", Nested: ""},
+		{Key: "interval", Type: "time.Duration", Doc: "Interval is how often the dispatcher re-checks saturation. Releases also\nwake it, so this is only the backstop for saturation falling on its own.", Nested: ""},
+	}},
+	{Package: "agent", Name: "Config", Title: "agent.Config", Doc: "", Fields: []Field{
+		{Key: "node_id", Type: "string", Doc: "NodeID defaults to the hostname. It must equal the common name of the\nagent's certificate, so set it only where the two differ.", Nested: ""},
+		{Key: "server", Type: "string", Doc: "Server is the control plane, and Servers is the same thing when there is\nmore than one. Both are accepted: every configuration written before\nthere could be a second control plane names one address, and none of\nthem should have to change.\n\nThe order is tried in order, and a refusal moves to the next. Only the\nleader accepts an agent, so a standby is skipped in the time it takes to\nbe told so — which is what makes failover a reconnect rather than an\nedit to every node's configuration.", Nested: ""},
+		{Key: "servers", Type: "[]string", Doc: "", Nested: ""},
+		{Key: "hostname", Type: "string", Doc: "Hostname defaults to the machine's.", Nested: ""},
+		{Key: "mgmt_addr", Type: "string", Doc: "MgmtAddr is the address engines are reached on. It defaults to this\nmachine's address on the route to the control plane, which is the one\nthe control plane and the router can reach.", Nested: ""},
+		{Key: "labels", Type: "map[string]string", Doc: "Labels are free-form facts about a node for a profile's node_selector.\nNothing needs them: memory, GPUs, engines and disk are measured, and a\nlabel restating one of those goes stale the day the machine changes.", Nested: ""},
+		{Key: "mgmt_interface", Type: "string", Doc: "MgmtInterface is the ethernet interface NCCL bootstraps over. It\ndefaults to the interface holding MgmtAddr, then eth0.", Nested: ""},
+		{Key: "fabrics", Type: "[]inventory.FabricAttachment", Doc: "Fabrics labels this node's HCAs. It is explicit on purpose: OpenSM's\ndefault subnet prefix is the same on every fabric, so nothing can be\ninferred from the wire (plan §5.2).", Nested: ""},
+		{Key: "heartbeat_interval", Type: "time.Duration", Doc: "", Nested: ""},
+		{Key: "reconnect_base", Type: "time.Duration", Doc: "ReconnectBase is the first reconnect delay; it backs off from there.", Nested: ""},
+		{Key: "reconnect_max", Type: "time.Duration", Doc: "", Nested: ""},
+		{Key: "adopt_ready_timeout", Type: "time.Duration", Doc: "AdoptReadyTimeout bounds how long an adopted engine is given to answer a\nhealth check before the agent stops waiting on it.\n\nShorter than a startup timeout on purpose: this engine is already\nrunning, so it is not loading a model, it is either serving or it is not.", Nested: ""},
+		{Key: "tls", Type: "tlsconf.Config", Doc: "TLS authenticates both ends of the control-plane stream. Leaving it unset\nkeeps the plaintext behaviour, which only belongs on a trusted network.", Nested: "tlsconf.Config"},
+		{Key: "model_dirs", Type: "[]string", Doc: "ModelDirs are the directories this node keeps model weights in, beyond\nWeightsCache, which is always looked in. Storage differs at every installation,\nso llmgw observes it rather than owning it: a Lustre mount, an image\nexported over the fabric, local NVMe and a copy made by hand all look\nidentical from here, and placement only needs to know whether the file\nis present.", Nested: ""},
+		{Key: "engines_dir", Type: "string", Doc: "EnginesDir is where the agent installs engines, one directory per\nbuild. Defaults to /opt/llmgw/engines.", Nested: ""},
+		{Key: "weights_cache", Type: "string", Doc: "WeightsCache is where a remote source is downloaded to. Defaults to the\nfirst entry of ModelDirs, because a node that says where its models live\nhas already answered this question, and otherwise to\n/var/lib/llmgw-weights.\n\nFetching is the simple arrangement: every node keeps its own copy. The\nexotic ones need no code — weights on a shared filer or an image exported\nover the fabric are simply present, and present means nothing is fetched.", Nested: ""},
+		{Key: "discovery", Type: "discovery.Options", Doc: "", Nested: "discovery.Options"},
+		{Key: "engines", Type: "engines.Config", Doc: "", Nested: "engines.Config"},
+		{Key: "supervisor", Type: "supervisor.Config", Doc: "", Nested: "supervisor.Config"},
+	}},
+	{Package: "config", Name: "Band", Title: "config.Band", Doc: "", Fields: []Field{
+		{Key: "name", Type: "string", Doc: "", Nested: ""},
+		{Key: "priority", Type: "int", Doc: "", Nested: ""},
+		{Key: "saturation_ceiling", Type: "float64", Doc: "", Nested: ""},
+	}},
+	{Package: "config", Name: "Price", Title: "config.Price", Doc: "", Fields: []Field{
+		{Key: "prompt_per_1k", Type: "int64", Doc: "", Nested: ""},
+		{Key: "completion_per_1k", Type: "int64", Doc: "", Nested: ""},
+	}},
+	{Package: "engines", Name: "Config", Title: "engines.Config", Doc: "", Fields: []Field{
+		{Key: "disabled", Type: "[]string", Doc: "Disabled are engines this node never installs or advertises, whatever\nthe control plane asks: vLLM on a node of cards no build of it runs on,\nthat is meant for llama.cpp alone.", Nested: ""},
+		{Key: "vllm", Type: "VLLMConfig", Doc: "", Nested: "engines.VLLMConfig"},
+		{Key: "sglang", Type: "SGLangConfig", Doc: "", Nested: "engines.SGLangConfig"},
+		{Key: "llamacpp", Type: "LlamaCppConfig", Doc: "", Nested: "engines.LlamaCppConfig"},
+		{Key: "sim", Type: "SimConfig", Doc: "", Nested: "engines.SimConfig"},
+	}},
+	{Package: "supervisor", Name: "Config", Title: "supervisor.Config", Doc: "", Fields: []Field{
+		{Key: "work_dir", Type: "string", Doc: "WorkDir holds pid files and per-shard scratch space.", Nested: ""},
+		{Key: "log_lines", Type: "int", Doc: "LogLines is how many lines of engine output to retain per shard.", Nested: ""},
+		{Key: "stop_timeout", Type: "time.Duration", Doc: "StopTimeout is how long a process gets to exit after SIGTERM before it is\nkilled. Engines need time to free GPU memory cleanly.", Nested: ""},
+		{Key: "health_interval", Type: "time.Duration", Doc: "HealthInterval is how often a starting shard is polled.", Nested: ""},
+		{Key: "rendezvous_timeout", Type: "time.Duration", Doc: "RendezvousTimeout bounds the wait for a multi-node rendezvous point to\nstart listening.", Nested: ""},
+		{Key: "alive_grace", Type: "time.Duration", Doc: "AliveGrace is how long a non-serving rank must stay up before it counts\nas ready. There is no external signal for a worker, so staying alive is\nthe only evidence available.", Nested: ""},
+	}},
+	{Package: "tlsconf", Name: "Config", Title: "tlsconf.Config", Doc: "", Fields: []Field{
+		{Key: "ca", Type: "string", Doc: "CA is the bundle used to verify the other side. In a real deployment this\nis whichever authority issued the fleet's certificates.", Nested: ""},
+		{Key: "cert", Type: "string", Doc: "Cert and Key are this component's own identity.", Nested: ""},
+		{Key: "key", Type: "string", Doc: "", Nested: ""},
+		{Key: "insecure", Type: "bool", Doc: "Insecure disables TLS entirely. It exists for local runs against\nsim-engines; anything using it must stay on a trusted network.", Nested: ""},
+	}},
+}

internal/docgen/render.go

@@ -61,13 +61,13 @@ rather than repeated.
 	for _, bl := range blocks {
 		for _, f := range bl.Fields {
 			b.WriteString(fmt.Sprintf("| `%s` | %s | [%s](#%s) |\n",
-				f.Key, code(f.Type), bl.Name, anchor(bl.Name)))
+				f.Key, code(f.Type), bl.Title, anchor(bl.Title)))
 		}
 	}
 	b.WriteString("\n")
 
 	for _, bl := range blocks {
-		fmt.Fprintf(&b, "## %s\n\n", bl.Name)
+		fmt.Fprintf(&b, "## %s\n\n", bl.Title)
 		if bl.Doc != "" {
 			b.WriteString(bl.Doc + "\n\n")
 		}
@@ -76,24 +76,44 @@ rather than repeated.
 			if f.Doc != "" {
 				b.WriteString(f.Doc + "\n\n")
 			}
-			if f.Nested != "" && hasBlock(blocks, f.Nested) {
-				fmt.Fprintf(&b, "Takes a [%s](#%s) block.\n\n", f.Nested, anchor(f.Nested))
+			if nb, ok := Find(blocks, f.Nested); ok {
+				fmt.Fprintf(&b, "Takes a [%s](#%s) block.\n\n", nb.Title, anchor(nb.Title))
 			}
 		}
 	}
 	return b.String()
 }
 
-func hasBlock(blocks []Block, name string) bool {
+// Find is the block with an ID, as a Field's Nested names it.
+func Find(blocks []Block, id string) (Block, bool) {
 	for _, b := range blocks {
-		if b.Name == name {
-			return true
+		if id != "" && b.ID() == id {
+			return b, true
 		}
 	}
-	return false
+	return Block{}, false
 }
 
 func code(s string) string { return "`" + s + "`" }
 
-// anchor is how GitHub and Gitea slug a heading.
-func anchor(s string) string { return strings.ToLower(s) }
+// anchor is how GitHub and Gitea slug a heading: lower case, and punctuation
+// such as the dot in tlsconf.Config dropped.
+func anchor(s string) string { return strings.ToLower(strings.ReplaceAll(s, ".", "")) }
+
+// RenderGo writes the blocks as Go, for the settings screens: every key, its
+// type and what it does, from the same source as the reference, so a setting
+// added to the code is on the screens without anyone listing it.
+func RenderGo(blocks []Block) string {
+	var b strings.Builder
+	b.WriteString("// Code generated by go run ./internal/docgen/gen; DO NOT EDIT.\n\npackage docgen\n\n")
+	b.WriteString("// Reference is every configuration block, as Collect reads them.\nvar Reference = []Block{\n")
+	for _, bl := range blocks {
+		fmt.Fprintf(&b, "\t{Package: %q, Name: %q, Title: %q, Doc: %q, Fields: []Field{\n", bl.Package, bl.Name, bl.Title, bl.Doc)
+		for _, f := range bl.Fields {
+			fmt.Fprintf(&b, "\t\t{Key: %q, Type: %q, Doc: %q, Nested: %q},\n", f.Key, f.Type, f.Doc, f.Nested)
+		}
+		b.WriteString("\t}},\n")
+	}
+	b.WriteString("}\n")
+	return b.String()
+}

internal/doctor/doctor.go

@@ -581,7 +581,16 @@ func Evaluate(f Facts) []Finding {
 // the same decision the agent makes, so the report and the node agree.
 func vllmFindings(f Facts, add func(Level, string, string, string)) {
 	v := f.VLLMVersion
-	b, err := provision.Recipes["vllm"].Choose(v, "", f.Host)
+	builds, err := provision.Recipes["vllm"].Candidates(v, "", f.Host)
+	var b provision.Build
+	if err == nil {
+		// The first the agent would try; it moves to the next by itself if
+		// that one does not work here.
+		b = builds[0]
+		if b.Version != "" {
+			v = b.Version
+		}
+	}
 	var nh provision.NotHere
 	switch {
 	case errors.As(err, &nh):
@@ -592,8 +601,7 @@ func vllmFindings(f Facts, add func(Level, string, string, string)) {
 		// llama.cpp. Refusing it here kept older GPU generations out of a
 		// cluster for an engine nobody meant to run on them.
 		add(Warn, "engines", "vLLM "+v+" will not run here, and the node joins without it: "+err.Error(),
-			"give this node a vLLM it can run with engine_overrides in the server's configuration, "+
-				"or add it with --disable-engine vllm if it is meant for llama.cpp alone")
+			"add it with --disable-engine vllm if it is meant for llama.cpp alone")
 		return
 	case f.VLLMInstalled:
 		add(OK, "engines", "vLLM "+v+" is installed under "+f.EnginesDir, "")
@@ -605,8 +613,12 @@ func vllmFindings(f Facts, add func(Level, string, string, string)) {
 			"free space there, or point engines_dir in agent.yaml at a larger disk")
 		return
 	}
-	add(OK, "engines", fmt.Sprintf("vLLM %s (%s) can be installed here: driver %s, %.1f GB free under %s",
-		v, b.Variant, f.Host.NVIDIADriver, f.EnginesFreeGB, f.EnginesDir), "")
+	note := ""
+	if b.Note != "" {
+		note = "; " + b.Note
+	}
+	add(OK, "engines", fmt.Sprintf("vLLM %s (%s) can be installed here: driver %s, %.1f GB free under %s%s",
+		v, b.Variant, f.Host.NVIDIADriver, f.EnginesFreeGB, f.EnginesDir, note), "")
 }
 
 func rank(l Level) int {

internal/doctor/doctor_test.go

@@ -157,9 +157,12 @@ func TestDoctorSaysWhetherVLLMCanBeInstalled(t *testing.T) {
 		// A driver too old for CUDA 13 runs the same version's CUDA 12.9 build.
 		{"turing, old driver", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.5},
 			VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "(cu129) can be installed"},
-		// A V100 has no build of vLLM 0.20 or later, and is told which has.
+		// A V100 has no build of vLLM 0.20 or later; it gets the newest
+		// that has Volta kernels, and doctor says which.
 		{"volta", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
-			VLLMVersion: "0.30.0"}, Warn, "the last vLLM built for it is 0.19.1"},
+			VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "vLLM 0.19.1 (cu12) can be installed here"},
+		{"pascal", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 6.1},
+			VLLMVersion: "0.30.0"}, Warn, "no vLLM is built for GPUs of compute capability 6.1"},
 		{"volta, the last vLLM built for it", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
 			VLLMVersion: "0.19.1", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "(cu12) can be installed"},
 		{"small disk", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 9}, Warn, "9.0 GB"},

internal/join/join.go

@@ -0,0 +1,302 @@
+// Package join lets a machine be set up without anything logging into it.
+//
+// The machine doing the setup — running as root, holding the authority —
+// listens for a while and hands out one code per machine it expects. On the
+// other machine `sudo llmgw join <code>` dials out to it, checks it is talking
+// to the listener the code names, proves itself with the code's secret, and
+// from then on runs what it is sent, as ssh would, until the setup closes
+// the connection. To the setup it is one more onboard.Remote: preflight,
+// Patroni and node onboarding run over it unchanged.
+//
+// For sites where nothing may log into a machine from outside: the
+// connection is outbound from the machine being set up, no password is
+// typed anywhere, and the code is good for one connection and a limited time.
+package join
+
+import (
+	"bufio"
+	"context"
+	"crypto/ecdsa"
+	"crypto/elliptic"
+	"crypto/rand"
+	"crypto/sha256"
+	"crypto/subtle"
+	"crypto/tls"
+	"crypto/x509"
+	"crypto/x509/pkix"
+	"encoding/base64"
+	"encoding/binary"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"math/big"
+	"net"
+	"strings"
+	"sync"
+	"time"
+)
+
+// DefaultPort is where a setup listens for machines joining it.
+const DefaultPort = 7444
+
+// codePrefix marks a join code, and its version.
+const codePrefix = "llmgw-join1-"
+
+// Code is what `llmgw join` is given: where to dial, how to know it reached
+// the right place, and the secret that admits it.
+//
+// Short, because somebody copies it from one screen into another: the first
+// version was JSON and over 250 characters, which a terminal cut off at its
+// edge. Packed it is under 90.
+type Code struct {
+	Addr string
+	// Fingerprint is the first 16 bytes of the SHA-256 of the listener's
+	// certificate: 128 bits, more than enough to find nothing else by.
+	Fingerprint [16]byte
+	Secret      [16]byte
+	Expires     int64
+	// Role says what the machine is being set up as, for its operator to
+	// read before it runs: "database", "node" or "control plane".
+	Role string
+}
+
+var roles = []string{"", "database", "node", "control plane"}
+
+// String is the code as typed: one word, safe in a shell.
+func (c Code) String() string {
+	b := make([]byte, 0, 40+len(c.Addr))
+	b = append(b, c.Fingerprint[:]...)
+	b = append(b, c.Secret[:]...)
+	b = binary.BigEndian.AppendUint32(b, uint32(c.Expires))
+	role := byte(0)
+	for i, r := range roles {
+		if r == c.Role {
+			role = byte(i)
+		}
+	}
+	b = append(b, role)
+	b = append(b, c.Addr...)
+	return codePrefix + base64.RawURLEncoding.EncodeToString(b)
+}
+
+// Parse reads a code back.
+func Parse(s string) (Code, error) {
+	raw, ok := strings.CutPrefix(strings.TrimSpace(s), codePrefix)
+	if !ok {
+		return Code{}, errors.New("not a join code: it starts " + codePrefix)
+	}
+	b, err := base64.RawURLEncoding.DecodeString(raw)
+	if err != nil || len(b) < 16+16+4+1+3 {
+		return Code{}, errors.New("the join code is damaged: copy it whole")
+	}
+	var c Code
+	copy(c.Fingerprint[:], b[:16])
+	copy(c.Secret[:], b[16:32])
+	c.Expires = int64(binary.BigEndian.Uint32(b[32:36]))
+	if int(b[36]) < len(roles) {
+		c.Role = roles[b[36]]
+	}
+	c.Addr = string(b[37:])
+	if _, _, err := net.SplitHostPort(c.Addr); err != nil {
+		return Code{}, errors.New("the join code is damaged: copy it whole")
+	}
+	if time.Now().Unix() > c.Expires {
+		return Code{}, fmt.Errorf("the join code expired at %s; ask for a new one",
+			time.Unix(c.Expires, 0).Format(time.RFC3339))
+	}
+	return c, nil
+}
+
+// The wire: one JSON object a line, each way.
+type hello struct {
+	Secret   []byte `json:"secret"`
+	Hostname string `json:"hostname"`
+	// Addr is the address the machine dialled out from: how the others
+	// reach it, when nothing else says.
+	Addr string `json:"addr"`
+}
+
+type request struct {
+	Script string `json:"script,omitempty"`
+	Stdin  []byte `json:"stdin,omitempty"`
+	Bye    bool   `json:"bye,omitempty"`
+}
+
+type response struct {
+	Stdout []byte `json:"stdout,omitempty"`
+	Stderr []byte `json:"stderr,omitempty"`
+	Exit   int    `json:"exit"`
+	Error  string `json:"error,omitempty"`
+}
+
+// Listener waits for machines to join.
+type Listener struct {
+	ln          net.Listener
+	addr        string
+	fingerprint [16]byte
+	ttl         time.Duration
+
+	mu      sync.Mutex
+	pending map[[16]byte]chan *Remote // by secret
+}
+
+// Listen starts listening on bind (":7444") and advertises addr ("10.0.0.5:7444")
+// in its codes. Codes are good for ttl.
+func Listen(bind, addr string, ttl time.Duration) (*Listener, error) {
+	cert, fp, err := sessionCert()
+	if err != nil {
+		return nil, err
+	}
+	ln, err := tls.Listen("tcp", bind, &tls.Config{Certificates: []tls.Certificate{cert}, MinVersion: tls.VersionTLS13})
+	if err != nil {
+		return nil, fmt.Errorf("listening for machines to join on %s: %w", bind, err)
+	}
+	l := &Listener{ln: ln, addr: addr, fingerprint: fp, ttl: ttl, pending: map[[16]byte]chan *Remote{}}
+	go l.accept()
+	return l, nil
+}
+
+// sessionCert is a certificate for this listener alone, trusted by the
+// fingerprint in its codes rather than by any authority: it lives as long as
+// the setup does.
+func sessionCert() (tls.Certificate, [16]byte, error) {
+	var fp [16]byte
+	key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
+	if err != nil {
+		return tls.Certificate{}, fp, err
+	}
+	tmpl := &x509.Certificate{
+		SerialNumber: big.NewInt(time.Now().UnixNano()),
+		Subject:      pkix.Name{CommonName: "llmgw setup"},
+		NotBefore:    time.Now().Add(-time.Minute),
+		NotAfter:     time.Now().Add(48 * time.Hour),
+		KeyUsage:     x509.KeyUsageDigitalSignature,
+		ExtKeyUsage:  []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
+	}
+	der, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, &key.PublicKey, key)
+	if err != nil {
+		return tls.Certificate{}, fp, err
+	}
+	sum := sha256.Sum256(der)
+	copy(fp[:], sum[:16])
+	return tls.Certificate{Certificate: [][]byte{der}, PrivateKey: key}, fp, nil
+}
+
+// Code makes a code for one machine, and the channel its connection arrives
+// on.
+func (l *Listener) Code(role string) (Code, <-chan *Remote) {
+	var secret [16]byte
+	_, _ = rand.Read(secret[:])
+	ch := make(chan *Remote, 1)
+	l.mu.Lock()
+	l.pending[secret] = ch
+	l.mu.Unlock()
+	return Code{Addr: l.addr, Fingerprint: l.fingerprint, Secret: secret,
+		Expires: time.Now().Add(l.ttl).Unix(), Role: role}, ch
+}
+
+// Close stops listening; machines already joined stay connected until their
+// Remote is closed.
+func (l *Listener) Close() error { return l.ln.Close() }
+
+func (l *Listener) accept() {
+	for {
+		c, err := l.ln.Accept()
+		if err != nil {
+			return
+		}
+		go l.admit(c)
+	}
+}
+
+func (l *Listener) admit(c net.Conn) {
+	_ = c.SetDeadline(time.Now().Add(30 * time.Second))
+	r := bufio.NewReader(c)
+	var h hello
+	line, err := r.ReadBytes('\n')
+	if err != nil || json.Unmarshal(line, &h) != nil {
+		_ = c.Close()
+		return
+	}
+	l.mu.Lock()
+	var ch chan *Remote
+	for secret, p := range l.pending {
+		if subtle.ConstantTimeCompare(secret[:], h.Secret) == 1 {
+			ch = p
+			// One machine a code: whoever comes second is turned away.
+			delete(l.pending, secret)
+		}
+	}
+	l.mu.Unlock()
+	if ch == nil {
+		_, _ = c.Write([]byte(`{"error":"this code is not one the setup is waiting for, or it was used already"}` + "\n"))
+		_ = c.Close()
+		return
+	}
+	_ = c.SetDeadline(time.Time{})
+	if _, err := c.Write([]byte("{}\n")); err != nil {
+		_ = c.Close()
+		return
+	}
+	addr := h.Addr
+	if addr == "" {
+		addr, _, _ = net.SplitHostPort(c.RemoteAddr().String())
+	}
+	ch <- &Remote{conn: c, r: r, Hostname: h.Hostname, Addr: addr}
+}
+
+// Remote is a joined machine, as the setup runs things on it.
+type Remote struct {
+	Hostname string
+	Addr     string
+
+	mu   sync.Mutex
+	conn net.Conn
+	r    *bufio.Reader
+}
+
+// Run runs script with sh on the machine, as root.
+func (m *Remote) Run(ctx context.Context, script string, stdin []byte) ([]byte, error) {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+	if m.conn == nil {
+		return nil, fmt.Errorf("%s is no longer connected", m.Hostname)
+	}
+	if dl, ok := ctx.Deadline(); ok {
+		_ = m.conn.SetDeadline(dl)
+		defer m.conn.SetDeadline(time.Time{})
+	}
+	b, _ := json.Marshal(request{Script: script, Stdin: stdin})
+	if _, err := m.conn.Write(append(b, '\n')); err != nil {
+		return nil, fmt.Errorf("%s: %w", m.Hostname, err)
+	}
+	line, err := m.r.ReadBytes('\n')
+	if err != nil {
+		return nil, fmt.Errorf("%s went away: %w", m.Hostname, err)
+	}
+	var resp response
+	if err := json.Unmarshal(line, &resp); err != nil {
+		return nil, fmt.Errorf("%s answered something unreadable", m.Hostname)
+	}
+	if resp.Error != "" {
+		return append(resp.Stdout, resp.Stderr...), errors.New(resp.Error)
+	}
+	if resp.Exit != 0 {
+		return append(resp.Stdout, resp.Stderr...), fmt.Errorf("exit status %d", resp.Exit)
+	}
+	return resp.Stdout, nil
+}
+
+// Close tells the machine the setup is done, which ends its `llmgw join`.
+func (m *Remote) Close() error {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+	if m.conn == nil {
+		return nil
+	}
+	b, _ := json.Marshal(request{Bye: true})
+	_, _ = m.conn.Write(append(b, '\n'))
+	err := m.conn.Close()
+	m.conn = nil
+	return err
+}

internal/join/join_test.go

@@ -0,0 +1,94 @@
+package join
+
+import (
+	"context"
+	"strings"
+	"testing"
+	"time"
+)
+
+func listen(t *testing.T) *Listener {
+	t.Helper()
+	l, err := Listen("127.0.0.1:0", "", time.Hour)
+	if err != nil {
+		t.Fatal(err)
+	}
+	l.addr = l.ln.Addr().String()
+	t.Cleanup(func() { _ = l.Close() })
+	return l
+}
+
+func TestAJoinedMachineRunsWhatTheSetupSendsUntilItIsDone(t *testing.T) {
+	l := listen(t)
+	code, arrived := l.Code("database")
+	parsed, err := Parse(code.String())
+	if err != nil || parsed.Role != "database" {
+		t.Fatalf("%v %+v", err, parsed)
+	}
+	var said []string
+	served := make(chan error, 1)
+	go func() { served <- Serve(context.Background(), parsed, func(s string) { said = append(said, s) }) }()
+
+	var m *Remote
+	select {
+	case m = <-arrived:
+	case err := <-served:
+		t.Fatalf("join ended before arriving: %v", err)
+	case <-time.After(5 * time.Second):
+		t.Fatal("the machine never arrived")
+	}
+	if m.Hostname == "" || m.Addr != "127.0.0.1" {
+		t.Errorf("arrived as %q from %q", m.Hostname, m.Addr)
+	}
+	out, err := m.Run(context.Background(), "set -e\ntr a-z A-Z", []byte("hello\n"))
+	if err != nil || string(out) != "HELLO\n" {
+		t.Fatalf("%q, %v", out, err)
+	}
+	if out, err := m.Run(context.Background(), "echo nope >&2; exit 3", nil); err == nil || !strings.Contains(string(out), "nope") {
+		t.Errorf("a failing script: %q, %v", out, err)
+	}
+	_ = m.Close()
+	if err := <-served; err != nil {
+		t.Errorf("join did not end cleanly when the setup was done: %v", err)
+	}
+	if len(said) < 2 || !strings.Contains(strings.Join(said, "\n"), "tr a-z A-Z") {
+		t.Errorf("the machine's operator was not told what ran: %q", said)
+	}
+
+	// The code was for one machine.
+	if err := Serve(context.Background(), parsed, func(string) {}); err == nil || !strings.Contains(err.Error(), "used already") {
+		t.Errorf("a code used twice: %v", err)
+	}
+}
+
+func TestAJoinRefusesAListenerThatIsNotTheOneInTheCode(t *testing.T) {
+	l := listen(t)
+	code, _ := l.Code("node")
+	other := listen(t)
+	code.Addr = other.addr // what the code names is elsewhere
+	err := Serve(context.Background(), code, func(string) {})
+	if err == nil || !strings.Contains(err.Error(), "not the setup") {
+		t.Errorf("joined a stranger: %v", err)
+	}
+}
+
+func TestAnExpiredOrDamagedCodeIsRefusedBeforeDialling(t *testing.T) {
+	c := Code{Addr: "10.0.0.5:7444", Role: "node", Expires: time.Now().Add(-time.Minute).Unix()}
+	if _, err := Parse(c.String()); err == nil || !strings.Contains(err.Error(), "expired") {
+		t.Errorf("expired: %v", err)
+	}
+	if _, err := Parse(c.String()[:30]); err == nil {
+		t.Error("a truncated code was read")
+	}
+	if _, err := Parse("ssh-rsa AAAA"); err == nil {
+		t.Error("something else was read as a code")
+	}
+	// Short enough to copy from a screen in one line.
+	c.Expires = time.Now().Add(time.Hour).Unix()
+	if n := len(c.String()); n > 100 {
+		t.Errorf("a code is %d characters", n)
+	}
+	if back, err := Parse(c.String()); err != nil || back.Addr != c.Addr || back.Role != "node" {
+		t.Errorf("%+v, %v", back, err)
+	}
+}

internal/join/serve.go

@@ -0,0 +1,126 @@
+package join
+
+import (
+	"bufio"
+	"bytes"
+	"context"
+	"crypto/sha256"
+	"crypto/subtle"
+	"crypto/tls"
+	"crypto/x509"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"net"
+	"os"
+	"os/exec"
+	"strings"
+	"time"
+)
+
+// Serve dials the setup a code names and runs what it sends until it says it
+// is done. Each script is reported to log by its first line, so whoever ran
+// `llmgw join` sees what is being done to their machine.
+func Serve(ctx context.Context, c Code, log func(string)) error {
+	d := tls.Dialer{
+		NetDialer: &net.Dialer{Timeout: 10 * time.Second},
+		Config: &tls.Config{
+			MinVersion: tls.VersionTLS13,
+			// Trusted by the fingerprint in the code, not by an authority:
+			// the listener's certificate is made for this setup alone.
+			InsecureSkipVerify: true, //nolint:gosec // verified below, by fingerprint
+		},
+	}
+	want := c.Fingerprint[:]
+	d.Config.VerifyPeerCertificate = func(raw [][]byte, _ [][]*x509.Certificate) error {
+		if len(raw) == 0 {
+			return errors.New("the setup sent no certificate")
+		}
+		sum := sha256.Sum256(raw[0])
+		if subtle.ConstantTimeCompare(sum[:16], want) != 1 {
+			return errors.New("the machine at that address is not the setup this code is for")
+		}
+		return nil
+	}
+	conn, err := d.DialContext(ctx, "tcp", c.Addr)
+	if err != nil {
+		return fmt.Errorf("cannot reach the setup at %s: %w; it listens there only while it waits for this machine", c.Addr, err)
+	}
+	defer conn.Close()
+
+	host, _ := os.Hostname()
+	local, _, _ := net.SplitHostPort(conn.LocalAddr().String())
+	b, _ := json.Marshal(hello{Secret: c.Secret[:], Hostname: host, Addr: local})
+	if _, err := conn.Write(append(b, '\n')); err != nil {
+		return err
+	}
+	r := bufio.NewReader(conn)
+	line, err := r.ReadBytes('\n')
+	if err != nil {
+		return fmt.Errorf("the setup closed the connection: %w", err)
+	}
+	var ack response
+	if err := json.Unmarshal(line, &ack); err != nil {
+		return errors.New("the setup answered something unreadable")
+	}
+	if ack.Error != "" {
+		return errors.New(ack.Error)
+	}
+	log(fmt.Sprintf("joined the setup at %s; it is setting this machine up", c.Addr))
+
+	for {
+		line, err := r.ReadBytes('\n')
+		if err != nil {
+			return fmt.Errorf("the setup went away before it said it was done: %w", err)
+		}
+		var req request
+		if err := json.Unmarshal(line, &req); err != nil {
+			return errors.New("the setup sent something unreadable")
+		}
+		if req.Bye {
+			log("the setup is done with this machine")
+			return nil
+		}
+		log(summary(req.Script))
+		resp := run(ctx, req)
+		out, _ := json.Marshal(resp)
+		if _, err := conn.Write(append(out, '\n')); err != nil {
+			return err
+		}
+	}
+}
+
+// run runs one script as the setup asked.
+func run(ctx context.Context, req request) response {
+	cmd := exec.CommandContext(ctx, "sh", "-c", req.Script)
+	if req.Stdin != nil {
+		cmd.Stdin = bytes.NewReader(req.Stdin)
+	}
+	var stdout, stderr bytes.Buffer
+	cmd.Stdout, cmd.Stderr = &stdout, &stderr
+	err := cmd.Run()
+	resp := response{Stdout: stdout.Bytes(), Stderr: stderr.Bytes()}
+	var exit *exec.ExitError
+	switch {
+	case errors.As(err, &exit):
+		resp.Exit = exit.ExitCode()
+	case err != nil:
+		resp.Error = err.Error()
+	}
+	return resp
+}
+
+// summary is a script's first line that says something.
+func summary(script string) string {
+	for _, l := range strings.Split(script, "\n") {
+		l = strings.TrimSpace(l)
+		if l == "" || l == "set -e" || strings.HasPrefix(l, "#") {
+			continue
+		}
+		if len(l) > 100 {
+			l = l[:100] + "…"
+		}
+		return "  " + l
+	}
+	return "  (an empty script)"
+}

internal/onboard/gossh.go

@@ -0,0 +1,278 @@
+package onboard
+
+import (
+	"bytes"
+	"context"
+	"errors"
+	"fmt"
+	"net"
+	"os"
+	"path/filepath"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+
+	"golang.org/x/crypto/ssh"
+	"golang.org/x/crypto/ssh/agent"
+	"golang.org/x/crypto/ssh/knownhosts"
+)
+
+// GoSSH is Remote through Go's own ssh client rather than the system's: what
+// the setup screens use, because they ask for a password when there is no key,
+// which the system ssh in batch mode cannot take.
+//
+// Authentication is whichever is given, tried in order: the running ssh
+// agent, a key file, a password. The password is held in memory for the
+// session and never written anywhere.
+type GoSSH struct {
+	Addr string
+	Port int
+	User string
+
+	UseAgent bool
+	KeyFile  string
+	// KeyPassphrase opens an encrypted KeyFile.
+	KeyPassphrase string
+	Password      string
+
+	// SudoPassword is asked for when the user cannot sudo without one. With
+	// it, PrepareSudo lets the user sudo without a password until
+	// RevokeSudo, so every script can use plain sudo as on any other node.
+	SudoPassword string
+
+	// KnownHosts is where host keys are kept; "" is ~/.ssh/known_hosts.
+	KnownHosts string
+
+	mu        sync.Mutex
+	client    *ssh.Client
+	sudoGrant bool
+}
+
+// Run runs script with sh on the machine.
+func (g *GoSSH) Run(ctx context.Context, script string, stdin []byte) ([]byte, error) {
+	c, err := g.dial(ctx)
+	if err != nil {
+		return nil, err
+	}
+	sess, err := c.NewSession()
+	if err != nil {
+		// A connection that died since: dial again once.
+		g.drop()
+		if c, err = g.dial(ctx); err != nil {
+			return nil, err
+		}
+		if sess, err = c.NewSession(); err != nil {
+			return nil, err
+		}
+	}
+	defer sess.Close()
+	if stdin != nil {
+		sess.Stdin = bytes.NewReader(stdin)
+	}
+	var stdout, stderr bytes.Buffer
+	sess.Stdout, sess.Stderr = &stdout, &stderr
+	done := make(chan error, 1)
+	go func() { done <- sess.Run("sh -c " + shellQuote(script)) }()
+	select {
+	case <-ctx.Done():
+		_ = sess.Signal(ssh.SIGKILL)
+		return stdout.Bytes(), ctx.Err()
+	case err := <-done:
+		if err != nil {
+			return append(stdout.Bytes(), stderr.Bytes()...), err
+		}
+		return stdout.Bytes(), nil
+	}
+}
+
+// Close ends the connection, taking back a sudo grant first.
+func (g *GoSSH) Close() error {
+	_ = g.RevokeSudo(context.Background())
+	g.drop()
+	return nil
+}
+
+func (g *GoSSH) drop() {
+	g.mu.Lock()
+	defer g.mu.Unlock()
+	if g.client != nil {
+		_ = g.client.Close()
+		g.client = nil
+	}
+}
+
+func (g *GoSSH) target() string {
+	port := g.Port
+	if port == 0 {
+		port = 22
+	}
+	return net.JoinHostPort(g.Addr, strconv.Itoa(port))
+}
+
+func (g *GoSSH) dial(ctx context.Context) (*ssh.Client, error) {
+	g.mu.Lock()
+	defer g.mu.Unlock()
+	if g.client != nil {
+		return g.client, nil
+	}
+	var methods []ssh.AuthMethod
+	if g.UseAgent {
+		if sock := os.Getenv("SSH_AUTH_SOCK"); sock != "" {
+			if conn, err := net.Dial("unix", sock); err == nil {
+				methods = append(methods, ssh.PublicKeysCallback(agent.NewClient(conn).Signers))
+			}
+		}
+	}
+	if g.KeyFile != "" {
+		signer, err := loadKey(g.KeyFile, g.KeyPassphrase)
+		if err != nil {
+			return nil, err
+		}
+		methods = append(methods, ssh.PublicKeys(signer))
+	}
+	if g.Password != "" {
+		pw := g.Password
+		methods = append(methods, ssh.Password(pw),
+			ssh.KeyboardInteractive(func(_, _ string, qs []string, _ []bool) ([]string, error) {
+				out := make([]string, len(qs))
+				for i := range qs {
+					out[i] = pw
+				}
+				return out, nil
+			}))
+	}
+	if len(methods) == 0 {
+		return nil, fmt.Errorf("no way to log in to %s: no ssh agent, key or password", g.Addr)
+	}
+	hostKey, err := g.hostKeyCallback()
+	if err != nil {
+		return nil, err
+	}
+	cfg := &ssh.ClientConfig{User: g.User, Auth: methods, HostKeyCallback: hostKey, Timeout: 10 * time.Second}
+	d := net.Dialer{Timeout: 10 * time.Second}
+	conn, err := d.DialContext(ctx, "tcp", g.target())
+	if err != nil {
+		return nil, fmt.Errorf("cannot reach %s: %w", g.target(), err)
+	}
+	cc, chans, reqs, err := ssh.NewClientConn(conn, g.target(), cfg)
+	if err != nil {
+		_ = conn.Close()
+		return nil, fmt.Errorf("cannot log in to %s as %s: %w", g.Addr, g.User, err)
+	}
+	g.client = ssh.NewClient(cc, chans, reqs)
+	return g.client, nil
+}
+
+func loadKey(path, passphrase string) (ssh.Signer, error) {
+	b, err := os.ReadFile(path)
+	if err != nil {
+		return nil, fmt.Errorf("reading the key %s: %w", path, err)
+	}
+	if passphrase != "" {
+		return ssh.ParsePrivateKeyWithPassphrase(b, []byte(passphrase))
+	}
+	s, err := ssh.ParsePrivateKey(b)
+	var missing *ssh.PassphraseMissingError
+	if errors.As(err, &missing) {
+		return nil, fmt.Errorf("the key %s needs its passphrase", path)
+	}
+	return s, err
+}
+
+// hostKeyCallback trusts a machine on first sight and records its key, and
+// refuses one whose key has changed since — accept-new, as onboarding's
+// system ssh does: a machine provisioned a minute ago has a key nobody has
+// seen, and the risk guarded against is a machine substituted for one that
+// was known.
+func (g *GoSSH) hostKeyCallback() (ssh.HostKeyCallback, error) {
+	path := g.KnownHosts
+	if path == "" {
+		home, err := os.UserHomeDir()
+		if err != nil {
+			return nil, err
+		}
+		path = filepath.Join(home, ".ssh", "known_hosts")
+	}
+	if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil {
+		return nil, err
+	}
+	if f, err := os.OpenFile(path, os.O_CREATE|os.O_RDONLY, 0o600); err == nil {
+		_ = f.Close()
+	}
+	known, err := knownhosts.New(path)
+	if err != nil {
+		return nil, fmt.Errorf("reading %s: %w", path, err)
+	}
+	return func(host string, remote net.Addr, key ssh.PublicKey) error {
+		err := known(host, remote, key)
+		var ke *knownhosts.KeyError
+		if errors.As(err, &ke) && len(ke.Want) == 0 {
+			f, ferr := os.OpenFile(path, os.O_APPEND|os.O_WRONLY, 0o600)
+			if ferr != nil {
+				return ferr
+			}
+			defer f.Close()
+			_, ferr = fmt.Fprintln(f, knownhosts.Line([]string{knownhosts.Normalize(host)}, key))
+			return ferr
+		}
+		if errors.As(err, &ke) {
+			return fmt.Errorf("%s's host key is not the one recorded in %s: "+
+				"if the machine was rebuilt, remove its line there; otherwise something is in the way", host, path)
+		}
+		return err
+	}, nil
+}
+
+// sudoersDrop is the grant PrepareSudo writes and RevokeSudo removes.
+const sudoersDrop = "/etc/sudoers.d/llmgw-setup"
+
+// CanSudo reports whether the user can sudo without a password now.
+func (g *GoSSH) CanSudo(ctx context.Context) bool {
+	_, err := g.Run(ctx, "sudo -n true", nil)
+	return err == nil
+}
+
+// PrepareSudo makes plain sudo work for the rest of the session: nothing to do
+// when it already does, otherwise a sudoers grant written with SudoPassword
+// and taken back by RevokeSudo. A grant rather than the password piped into
+// every sudo, because the scripts feed their own standard input to the
+// commands they run under it.
+func (g *GoSSH) PrepareSudo(ctx context.Context) error {
+	if g.CanSudo(ctx) {
+		return nil
+	}
+	if g.SudoPassword == "" {
+		return fmt.Errorf("%s cannot use sudo without a password, and none was given", g.User)
+	}
+	line := fmt.Sprintf("%s ALL=(ALL) NOPASSWD:ALL", g.User)
+	// Checked by visudo before it is put in place: a broken file in
+	// sudoers.d breaks sudo for everyone on the machine.
+	inner := fmt.Sprintf("umask 337; printf '%%s\\n' %s %s > %[3]s.new && visudo -cf %[3]s.new >/dev/null && mv -f %[3]s.new %[3]s",
+		shellQuote("# llmgw setup: removed when it finishes"), shellQuote(line), sudoersDrop)
+	script := "sudo -S -p '' sh -c " + shellQuote(inner)
+	if out, err := g.Run(ctx, script, []byte(g.SudoPassword+"\n")); err != nil {
+		return fmt.Errorf("sudo with the password given failed on %s: %v: %s", g.Addr, err, strings.TrimSpace(string(out)))
+	}
+	g.mu.Lock()
+	g.sudoGrant = true
+	g.mu.Unlock()
+	return nil
+}
+
+// RevokeSudo removes the grant PrepareSudo wrote, if it wrote one.
+func (g *GoSSH) RevokeSudo(ctx context.Context) error {
+	g.mu.Lock()
+	granted := g.sudoGrant
+	g.mu.Unlock()
+	if !granted {
+		return nil
+	}
+	if _, err := g.Run(ctx, "sudo rm -f "+sudoersDrop, nil); err != nil {
+		return err
+	}
+	g.mu.Lock()
+	g.sudoGrant = false
+	g.mu.Unlock()
+	return nil
+}

internal/pgha/config.go

@@ -0,0 +1,178 @@
+// Package pgha sets up the store as PostgreSQL under Patroni: two or more
+// machines, one primary streaming to the others, and a standby promoted when
+// the primary is lost. Patroni keeps its leader lock in etcd, which decides
+// only with a majority, so etcd runs on three machines — the database
+// machines and, when they are two, the control plane as the third vote.
+//
+// Everything is done over the same ssh the rest of onboarding uses, and every
+// step looks before it acts: run against a cluster that is already up, it
+// changes nothing.
+package pgha
+
+import (
+	"fmt"
+	"net/url"
+	"strings"
+)
+
+// Scope is the Patroni cluster's name, and the etcd cluster token.
+const Scope = "llmgw"
+
+// PatroniPort is where Patroni's REST API listens on every member: what the
+// control plane reads the cluster's state from.
+const PatroniPort = 8008
+
+// restPort is where this package asks Patroni; a test moves it.
+var restPort = PatroniPort
+
+// Role and Database are what the control plane connects as.
+const (
+	Role     = "llmgw"
+	Database = "llmgw"
+)
+
+// Member is one machine of the cluster, by the address the others reach it on.
+type Member struct {
+	Name string
+	IP   string
+}
+
+// Secrets are the cluster's three passwords.
+type Secrets struct {
+	Superuser, Replicator, App string
+}
+
+// Etcd is a member's /etc/default/etcd, for Ubuntu's etcd-server.
+func Etcd(m Member, all []Member) string {
+	var peers []string
+	for _, p := range all {
+		peers = append(peers, fmt.Sprintf("%s=http://%s:2380", p.Name, p.IP))
+	}
+	return fmt.Sprintf(`# llmgw: one member of the etcd that holds Patroni's leader lock.
+ETCD_NAME=%[1]s
+ETCD_DATA_DIR=/var/lib/etcd/default
+ETCD_LISTEN_PEER_URLS=http://%[2]s:2380
+ETCD_LISTEN_CLIENT_URLS=http://%[2]s:2379,http://127.0.0.1:2379
+ETCD_INITIAL_ADVERTISE_PEER_URLS=http://%[2]s:2380
+ETCD_ADVERTISE_CLIENT_URLS=http://%[2]s:2379
+ETCD_INITIAL_CLUSTER=%[3]s
+ETCD_INITIAL_CLUSTER_STATE=new
+ETCD_INITIAL_CLUSTER_TOKEN=%[4]s
+`, m.Name, m.IP, strings.Join(peers, ","), Scope+"-patroni")
+}
+
+// Patroni is a database member's /etc/patroni/config.yml.
+//
+// clients are the addresses the control planes connect from; the members
+// themselves are always allowed, since a second control plane usually runs on
+// one of them.
+func Patroni(m Member, db, etcd []Member, clients []string, version string, s Secrets) string {
+	var hosts []string
+	for _, e := range etcd {
+		hosts = append(hosts, e.IP+":2379")
+	}
+	var hba []string
+	hba = append(hba, "local all all peer", "host all all 127.0.0.1/32 scram-sha-256")
+	for _, d := range db {
+		hba = append(hba,
+			"host replication replicator "+d.IP+"/32 scram-sha-256",
+			"host all postgres "+d.IP+"/32 scram-sha-256")
+	}
+	seen := map[string]bool{}
+	for _, c := range append(append([]string{}, clients...), ips(db)...) {
+		if c == "" || seen[c] {
+			continue
+		}
+		seen[c] = true
+		hba = append(hba, fmt.Sprintf("host %s %s %s/32 scram-sha-256", Database, Role, c))
+	}
+	var hbaLines string
+	for _, h := range hba {
+		hbaLines += "        - " + h + "\n"
+	}
+	var b strings.Builder
+	fmt.Fprintf(&b, `# llmgw: the control planes' store, kept by Patroni, its leader lock in etcd.
+scope: %[1]s
+namespace: /service/
+name: %[2]s
+
+restapi:
+  listen: %[3]s:%[4]d
+  connect_address: %[3]s:%[4]d
+
+etcd3:
+  hosts: %[5]s
+
+bootstrap:
+  dcs:
+    ttl: 30
+    loop_wait: 10
+    retry_timeout: 10
+    maximum_lag_on_failover: 1048576
+    postgresql:
+      use_pg_rewind: true
+      use_slots: true
+      parameters:
+        wal_level: replica
+        hot_standby: "on"
+        max_wal_senders: 10
+        max_replication_slots: 10
+        wal_keep_size: 1GB
+        wal_log_hints: "on"
+        password_encryption: scram-sha-256
+      # In the cluster's shared configuration rather than each member's, so a
+      # control plane added later is let in by one change through Patroni's
+      # API (llmgw server add) instead of an edit on every member.
+      pg_hba:
+%[9]s  initdb:
+    - encoding: UTF8
+    - data-checksums
+
+postgresql:
+  listen: 127.0.0.1,%[3]s:5432
+  connect_address: %[3]s:5432
+  data_dir: /var/lib/postgresql/%[6]s/llmgw
+  bin_dir: /usr/lib/postgresql/%[6]s/bin
+  pgpass: /var/lib/postgresql/.pgpass_patroni
+  authentication:
+    superuser:
+      username: postgres
+      password: "%[7]s"
+    replication:
+      username: replicator
+      password: "%[8]s"
+    rewind:
+      username: postgres
+      password: "%[7]s"
+  parameters:
+    unix_socket_directories: /var/run/postgresql
+`, Scope, m.Name, m.IP, PatroniPort, strings.Join(hosts, ","), version, s.Superuser, s.Replicator, hbaLines)
+	b.WriteString(`
+tags:
+  nofailover: false
+  noloadbalance: false
+  clonefrom: false
+  nosync: false
+`)
+	return b.String()
+}
+
+// DSN is the store address the control planes are given: every member, and
+// the driver told to use whichever accepts writes, so a failover is followed
+// without anything in front of the database.
+func DSN(db []Member, password string) string {
+	var hosts []string
+	for _, d := range db {
+		hosts = append(hosts, d.IP+":5432")
+	}
+	return fmt.Sprintf("postgres://%s:%s@%s/%s?sslmode=disable&target_session_attrs=read-write",
+		Role, url.QueryEscape(password), strings.Join(hosts, ","), Database)
+}
+
+func ips(ms []Member) []string {
+	out := make([]string, len(ms))
+	for i, m := range ms {
+		out[i] = m.IP
+	}
+	return out
+}

internal/pgha/hba.go

@@ -0,0 +1,91 @@
+package pgha
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"fmt"
+	"net"
+	"net/http"
+	"slices"
+	"strconv"
+	"time"
+)
+
+// ClientLine is the access rule that lets a control plane at ip in.
+func ClientLine(ip string) string {
+	return fmt.Sprintf("host %s %s %s/32 scram-sha-256", Database, Role, ip)
+}
+
+// AllowClient lets a control plane at ip into the cluster's database: one
+// change to the cluster's shared configuration through Patroni's API, which
+// every member applies. It says whether anything changed.
+//
+// A cluster set up before the rules moved into the shared configuration
+// keeps them in each member's own file, which this cannot reach; that is
+// said, with what to add, rather than half done.
+func AllowClient(ctx context.Context, hosts []string, ip string) (bool, error) {
+	client := &http.Client{Timeout: 5 * time.Second}
+	var lastErr error
+	for _, h := range hosts {
+		base := "http://" + net.JoinHostPort(h, strconv.Itoa(restPort))
+		cfg, err := getConfig(ctx, client, base)
+		if err != nil {
+			lastErr = err
+			continue
+		}
+		var hba []string
+		if pg, ok := cfg["postgresql"].(map[string]any); ok {
+			if list, ok := pg["pg_hba"].([]any); ok {
+				for _, l := range list {
+					if s, ok := l.(string); ok {
+						hba = append(hba, s)
+					}
+				}
+			}
+		}
+		if len(hba) == 0 {
+			return false, fmt.Errorf("this cluster keeps its access rules in each member's "+
+				"/etc/patroni/config.yml, not in its shared configuration: add %q to pg_hba there "+
+				"on every member, and run `patronictl reload llmgw`", ClientLine(ip))
+		}
+		line := ClientLine(ip)
+		if slices.Contains(hba, line) {
+			return false, nil
+		}
+		body, _ := json.Marshal(map[string]any{"postgresql": map[string]any{"pg_hba": append(hba, line)}})
+		req, err := http.NewRequestWithContext(ctx, http.MethodPatch, base+"/config", bytes.NewReader(body))
+		if err != nil {
+			return false, err
+		}
+		req.Header.Set("Content-Type", "application/json")
+		resp, err := client.Do(req)
+		if err != nil {
+			lastErr = err
+			continue
+		}
+		resp.Body.Close()
+		if resp.StatusCode != http.StatusOK {
+			return false, fmt.Errorf("Patroni on %s refused the change: %s", h, resp.Status)
+		}
+		return true, nil
+	}
+	return false, fmt.Errorf("no member's Patroni answered: %w", lastErr)
+}
+
+func getConfig(ctx context.Context, client *http.Client, base string) (map[string]any, error) {
+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, base+"/config", nil)
+	if err != nil {
+		return nil, err
+	}
+	resp, err := client.Do(req)
+	if err != nil {
+		return nil, err
+	}
+	defer resp.Body.Close()
+	var cfg map[string]any
+	if err := json.NewDecoder(resp.Body).Decode(&cfg); err != nil {
+		return nil, fmt.Errorf("%s/config: %w", base, err)
+	}
+	return cfg, nil
+}

internal/pgha/metrics.go

@@ -0,0 +1,100 @@
+package pgha
+
+import (
+	"context"
+	"sync"
+	"time"
+
+	"github.com/prometheus/client_golang/prometheus"
+)
+
+// Collector publishes the store cluster's state as the control plane sees it
+// through Patroni's API: who leads, who streams, how far behind. Read at most
+// every fifteen seconds, however often it is scraped.
+type Collector struct {
+	hosts  []string
+	status func(ctx context.Context, hosts []string) (Cluster, error)
+
+	mu   sync.Mutex
+	at   time.Time
+	last Cluster
+	err  error
+
+	up, member, lag *prometheus.Desc
+}
+
+// NewCollector reads the cluster from the Patroni of each host in turn.
+func NewCollector(hosts []string) *Collector {
+	return &Collector{
+		hosts:  hosts,
+		status: HTTPStatus,
+		up: prometheus.NewDesc("llmgw_store_patroni_up",
+			"1 when a member's Patroni answered for the store's cluster.", nil, nil),
+		member: prometheus.NewDesc("llmgw_store_member",
+			"Each member of the store's cluster, by role (leader, replica) and state (running, streaming, …); always 1.",
+			[]string{"member", "host", "role", "state"}, nil),
+		lag: prometheus.NewDesc("llmgw_store_replica_lag_bytes",
+			"How far each replica is behind the leader; -1 when Patroni does not know.",
+			[]string{"member"}, nil),
+	}
+}
+
+// Describe implements prometheus.Collector.
+func (c *Collector) Describe(ch chan<- *prometheus.Desc) {
+	ch <- c.up
+	ch <- c.member
+	ch <- c.lag
+}
+
+// Collect implements prometheus.Collector.
+func (c *Collector) Collect(ch chan<- prometheus.Metric) {
+	cl, err := c.read()
+	if err != nil {
+		ch <- prometheus.MustNewConstMetric(c.up, prometheus.GaugeValue, 0)
+		return
+	}
+	ch <- prometheus.MustNewConstMetric(c.up, prometheus.GaugeValue, 1)
+	for _, m := range cl.Members {
+		ch <- prometheus.MustNewConstMetric(c.member, prometheus.GaugeValue, 1, m.Name, m.Host, m.Role, m.State)
+		if m.Role != "leader" {
+			ch <- prometheus.MustNewConstMetric(c.lag, prometheus.GaugeValue, float64(m.Lag), m.Name)
+		}
+	}
+}
+
+// Read is the cluster's state, at most fifteen seconds old.
+func (c *Collector) Read() (Cluster, error) { return c.read() }
+
+func (c *Collector) read() (Cluster, error) {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	if time.Since(c.at) < 15*time.Second {
+		return c.last, c.err
+	}
+	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+	defer cancel()
+	c.last, c.err = c.status(ctx, c.hosts)
+	if c.err == nil {
+		c.last = withMissing(c.last, c.hosts)
+	}
+	c.at = time.Now()
+	return c.last, c.err
+}
+
+// withMissing adds every host of the store address that the cluster does not
+// list, as a member in state "missing". A member that stops is dropped from
+// Patroni's list once its key expires — so without this a cluster down to
+// one machine reads as a healthy cluster of one, and nothing says a standby
+// is gone.
+func withMissing(c Cluster, hosts []string) Cluster {
+	seen := map[string]bool{}
+	for _, m := range c.Members {
+		seen[m.Host] = true
+	}
+	for _, h := range hosts {
+		if !seen[h] {
+			c.Members = append(c.Members, MemberState{Name: h, Host: h, Role: "replica", State: "missing", Lag: -1})
+		}
+	}
+	return c
+}

internal/pgha/pgha_test.go

@@ -0,0 +1,282 @@
+package pgha
+
+import (
+	"context"
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"net/url"
+	"strconv"
+	"strings"
+	"sync"
+	"testing"
+	"time"
+
+	"github.com/prometheus/client_golang/prometheus/testutil"
+	"github.com/prometheus/common/expfmt"
+	"go.yaml.in/yaml/v3"
+)
+
+var (
+	pg01  = Member{"llmgw-pg01", "10.0.0.149"}
+	ctl02 = Member{"llmgw-ctl02", "10.0.0.150"}
+	ctl   = Member{"llmgw-ctl", "10.0.0.140"}
+)
+
+func TestTheConfigurationsNameEveryMemberAndOnlyTheClientsGiven(t *testing.T) {
+	sec := Secrets{"su", "rep", "app"}
+	cfg := Patroni(pg01, []Member{pg01, ctl02}, []Member{pg01, ctl02, ctl}, []string{"10.0.0.140"}, "16", sec)
+	for _, want := range []string{
+		"name: llmgw-pg01", "listen: 10.0.0.149:8008",
+		"hosts: 10.0.0.149:2379,10.0.0.150:2379,10.0.0.140:2379",
+		"data_dir: /var/lib/postgresql/16/llmgw", `password: "rep"`,
+		"- host replication replicator 10.0.0.150/32 scram-sha-256",
+		"- host llmgw llmgw 10.0.0.140/32 scram-sha-256",
+		// A member can run a control plane too.
+		"- host llmgw llmgw 10.0.0.150/32 scram-sha-256",
+	} {
+		if !strings.Contains(cfg, want) {
+			t.Errorf("patroni config lacks %q", want)
+		}
+	}
+	if strings.Count(cfg, "host llmgw llmgw 10.0.0.149/32") != 1 {
+		t.Error("a member's address is allowed twice")
+	}
+	e := Etcd(ctl, []Member{pg01, ctl02, ctl})
+	if !strings.Contains(e, "ETCD_NAME=llmgw-ctl\n") ||
+		!strings.Contains(e, "llmgw-pg01=http://10.0.0.149:2380,llmgw-ctl02=http://10.0.0.150:2380,llmgw-ctl=http://10.0.0.140:2380") {
+		t.Errorf("etcd defaults:\n%s", e)
+	}
+	if got := DSN([]Member{pg01, ctl02}, "pw"); got !=
+		"postgres://llmgw:pw@10.0.0.149:5432,10.0.0.150:5432/llmgw?sslmode=disable&target_session_attrs=read-write" {
+		t.Errorf("dsn %s", got)
+	}
+}
+
+func TestAClusterIsCompleteWithALeaderAndEveryOtherMemberStreaming(t *testing.T) {
+	var c Cluster
+	if err := json.Unmarshal([]byte(`{"members":[
+		{"name":"llmgw-ctl02","role":"replica","state":"streaming","host":"10.0.0.150","lag":0},
+		{"name":"llmgw-pg01","role":"leader","state":"running","host":"10.0.0.149"}]}`), &c); err != nil {
+		t.Fatal(err)
+	}
+	if c.Leader() != "llmgw-pg01" || !c.Complete(2) || c.Complete(3) {
+		t.Errorf("%s: leader %q", c, c.Leader())
+	}
+	c.Members[0].State = "starting"
+	if c.Complete(2) {
+		t.Error("a replica still starting counted as streaming")
+	}
+	if lagOf(json.RawMessage(`"unknown"`)) != -1 || lagOf(json.RawMessage(`1024`)) != 1024 {
+		t.Error("lag not read")
+	}
+}
+
+// machine is a fake host: it records what it was asked to run and answers
+// from a table of script fragments.
+type machine struct {
+	name    string
+	mu      sync.Mutex
+	ran     []string
+	answers map[string]string
+}
+
+func (m *machine) Run(_ context.Context, script string, _ []byte) ([]byte, error) {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+	m.ran = append(m.ran, script)
+	for frag, out := range m.answers {
+		if strings.Contains(script, frag) {
+			return []byte(out), nil
+		}
+	}
+	return nil, nil
+}
+
+func (m *machine) ranAny(frag string) bool {
+	for _, s := range m.ran {
+		if strings.Contains(s, frag) {
+			return true
+		}
+	}
+	return false
+}
+
+const pws = "superuser=a\nreplicator=b\nllmgw=c\n"
+
+func healthy(n int) string {
+	return strings.Repeat("http://x:2379 is healthy: successfully committed proposal\n", n)
+}
+
+func TestARunningClusterIsLeftAsItIs(t *testing.T) {
+	// What init does when run again, or against the reference cluster: find
+	// it up, change nothing, and give the address.
+	a := &machine{name: "pg01", answers: map[string]string{"openssl rand": pws, "endpoint health": healthy(3)}}
+	b := &machine{name: "ctl02", answers: map[string]string{"sudo cat /root/llmgw-pg": pws}}
+	w := &machine{name: "ctl"}
+	up := Cluster{Members: []MemberState{{Name: "llmgw-pg01", Role: "leader", State: "running"},
+		{Name: "llmgw-ctl02", Role: "replica", State: "streaming"}}}
+	s := Setup{
+		DB:      []Node{{pg01, a}, {ctl02, b}},
+		Witness: &Node{ctl, w},
+		Status:  func(context.Context, []string) (Cluster, error) { return up, nil },
+	}
+	dsn, err := s.Run(context.Background())
+	if err != nil {
+		t.Fatal(err)
+	}
+	if !strings.Contains(dsn, "llmgw:c@10.0.0.149:5432,10.0.0.150:5432") {
+		t.Errorf("dsn %s", dsn)
+	}
+	for _, m := range []*machine{a, b, w} {
+		for _, frag := range []string{"apt-get", "systemctl restart", "/etc/default/etcd", "/etc/patroni/config.yml"} {
+			if m.ranAny(frag) {
+				t.Errorf("%s was changed (%q) though the cluster is up", m.name, frag)
+			}
+		}
+	}
+	if !a.ranAny("CREATE ROLE") {
+		t.Error("the role was not made sure of on the leader")
+	}
+}
+
+func TestTwoDatabaseMachinesWithoutAThirdVoteAreRefused(t *testing.T) {
+	s := Setup{DB: []Node{{pg01, &machine{}}, {ctl02, &machine{}}}}
+	if _, err := s.Run(context.Background()); err == nil || !strings.Contains(err.Error(), "third etcd vote") {
+		t.Errorf("got %v", err)
+	}
+}
+
+func TestAMachineHoldingOtherPasswordsIsNotOverwritten(t *testing.T) {
+	a := &machine{answers: map[string]string{"openssl rand": pws}}
+	b := &machine{answers: map[string]string{"sudo cat /root/llmgw-pg": "superuser=x\nreplicator=y\nllmgw=z\n"}}
+	s := Setup{DB: []Node{{pg01, a}, {ctl02, b}}, Witness: &Node{ctl, &machine{}}}
+	_, err := s.Run(context.Background())
+	if err == nil || !strings.Contains(err.Error(), "differ") {
+		t.Fatalf("got %v", err)
+	}
+	if b.ranAny("cat > /root/llmgw-pg") {
+		t.Error("the other cluster's passwords were replaced")
+	}
+}
+
+func TestAPostgreSQLWithDatabasesInItIsNotReplaced(t *testing.T) {
+	a := &machine{answers: map[string]string{"openssl rand": pws, "endpoint health": healthy(3),
+		"pg_dropcluster": "holds-data\n"}}
+	b := &machine{answers: map[string]string{"sudo cat /root/llmgw-pg": pws}}
+	s := Setup{DB: []Node{{pg01, a}, {ctl02, b}}, Witness: &Node{ctl, &machine{}},
+		Status: func(context.Context, []string) (Cluster, error) { return Cluster{}, nil },
+		Wait:   time.Second}
+	_, err := s.Run(context.Background())
+	if err == nil || !strings.Contains(err.Error(), "databases in it") {
+		t.Fatalf("got %v", err)
+	}
+	if a.ranAny("systemctl restart patroni") {
+		t.Error("patroni was started over somebody's data")
+	}
+}
+
+func TestTheCollectorSaysWhoLeadsAndHowFarBehindTheRestAre(t *testing.T) {
+	c := NewCollector([]string{"10.0.0.149"})
+	c.status = func(context.Context, []string) (Cluster, error) {
+		return Cluster{Members: []MemberState{{Name: "llmgw-pg01", Host: "10.0.0.149", Role: "leader", State: "running"},
+			{Name: "llmgw-ctl02", Host: "10.0.0.150", Role: "replica", State: "streaming", Lag: 2048}}}, nil
+	}
+	got, err := testutil.CollectAndFormat(c, expfmt.TypeTextPlain, "llmgw_store_patroni_up", "llmgw_store_member", "llmgw_store_replica_lag_bytes")
+	if err != nil {
+		t.Fatal(err)
+	}
+	for _, want := range []string{
+		"llmgw_store_patroni_up 1",
+		`llmgw_store_member{host="10.0.0.149",member="llmgw-pg01",role="leader",state="running"} 1`,
+		`llmgw_store_replica_lag_bytes{member="llmgw-ctl02"} 2048`,
+	} {
+		if !strings.Contains(string(got), want) {
+			t.Errorf("missing %s in\n%s", want, got)
+		}
+	}
+	if strings.Contains(string(got), `lag_bytes{member="llmgw-pg01"}`) {
+		t.Error("the leader was given a lag")
+	}
+}
+
+func TestAMemberThatStoppedIsReportedMissingNotForgotten(t *testing.T) {
+	c := NewCollector([]string{"10.0.0.153", "10.0.0.154"})
+	c.status = func(context.Context, []string) (Cluster, error) {
+		return Cluster{Members: []MemberState{{Name: "hatest-pg2", Host: "10.0.0.154", Role: "leader", State: "running"}}}, nil
+	}
+	cl, err := c.Read()
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(cl.Members) != 2 || cl.Members[1].Host != "10.0.0.153" || cl.Members[1].State != "missing" {
+		t.Errorf("%s", cl)
+	}
+}
+
+func TestTheAccessRulesAreInTheClustersSharedConfiguration(t *testing.T) {
+	// Where `server add` can change them for every member at once.
+	cfg := Patroni(pg01, []Member{pg01, ctl02}, []Member{pg01, ctl02, ctl}, []string{"10.0.0.140"}, "16", Secrets{"a", "b", "c"})
+	var doc struct {
+		Bootstrap struct {
+			DCS struct {
+				PostgreSQL struct {
+					PgHBA []string `yaml:"pg_hba"`
+				} `yaml:"postgresql"`
+			} `yaml:"dcs"`
+		} `yaml:"bootstrap"`
+		PostgreSQL map[string]any `yaml:"postgresql"`
+	}
+	if err := yaml.Unmarshal([]byte(cfg), &doc); err != nil {
+		t.Fatalf("not YAML: %v\n%s", err, cfg)
+	}
+	hba := doc.Bootstrap.DCS.PostgreSQL.PgHBA
+	if len(hba) < 5 || hba[0] != "local all all peer" {
+		t.Errorf("shared pg_hba: %q", hba)
+	}
+	if _, local := doc.PostgreSQL["pg_hba"]; local {
+		t.Error("pg_hba is also in the member's own configuration, where it would win over the shared one")
+	}
+}
+
+func TestAControlPlaneIsLetInThroughTheSharedConfiguration(t *testing.T) {
+	var patched []string
+	hba := []any{"local all all peer", ClientLine("10.0.0.140")}
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		switch r.Method {
+		case http.MethodGet:
+			_ = json.NewEncoder(w).Encode(map[string]any{"postgresql": map[string]any{"pg_hba": hba}})
+		case http.MethodPatch:
+			var body struct {
+				PostgreSQL struct {
+					PgHBA []string `json:"pg_hba"`
+				} `json:"postgresql"`
+			}
+			_ = json.NewDecoder(r.Body).Decode(&body)
+			patched = body.PostgreSQL.PgHBA
+		}
+	}))
+	defer srv.Close()
+	u, _ := url.Parse(srv.URL)
+	old := restPort
+	restPort, _ = strconv.Atoi(u.Port())
+	defer func() { restPort = old }()
+
+	changed, err := AllowClient(context.Background(), []string{"127.0.0.1"}, "10.0.0.150")
+	if err != nil || !changed {
+		t.Fatalf("%v %v", changed, err)
+	}
+	// Everything that was there is kept, and the new line added once.
+	if len(patched) != 3 || patched[2] != ClientLine("10.0.0.150") || patched[1] != ClientLine("10.0.0.140") {
+		t.Errorf("patched %q", patched)
+	}
+	if changed, err := AllowClient(context.Background(), []string{"127.0.0.1"}, "10.0.0.140"); err != nil || changed {
+		t.Errorf("one already let in was changed again: %v %v", changed, err)
+	}
+
+	hba = nil // the rules are in each member's file
+	if _, err := AllowClient(context.Background(), []string{"127.0.0.1"}, "10.0.0.151"); err == nil ||
+		!strings.Contains(err.Error(), "each member") {
+		t.Errorf("rules kept per member: %v", err)
+	}
+}

internal/pgha/setup.go

@@ -0,0 +1,417 @@
+package pgha
+
+import (
+	"bytes"
+	"context"
+	"fmt"
+	"os/exec"
+	"strings"
+	"sync"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
+)
+
+// Node is a machine the setup reaches, and how.
+type Node struct {
+	Member
+	Remote onboard.Remote
+}
+
+// Setup builds, or finds, a Patroni cluster.
+type Setup struct {
+	// DB are the database machines; the first is where the cluster starts.
+	DB []Node
+	// Witness is the third etcd vote when there are only two database
+	// machines: the control plane, run locally. Ignored with three or more.
+	Witness *Node
+	// Clients are the addresses control planes connect from.
+	Clients []string
+	// Status reads the cluster from Patroni's API; nil asks over HTTP.
+	Status func(ctx context.Context, hosts []string) (Cluster, error)
+	// Emit reports progress: a step, what happened, and whether anything
+	// was changed or found as it should be.
+	Emit func(step, detail string, changed bool)
+	// Wait bounds each wait for the cluster to come up.
+	Wait time.Duration
+}
+
+// Local is a Remote for the machine this runs on, which init runs as root.
+type Local struct{}
+
+// Run runs script with sh here.
+func (Local) Run(ctx context.Context, script string, stdin []byte) ([]byte, error) {
+	cmd := exec.CommandContext(ctx, "sh", "-c", script)
+	if stdin != nil {
+		cmd.Stdin = bytes.NewReader(stdin)
+	}
+	var out, errb bytes.Buffer
+	cmd.Stdout, cmd.Stderr = &out, &errb
+	if err := cmd.Run(); err != nil {
+		return append(out.Bytes(), errb.Bytes()...), err
+	}
+	return out.Bytes(), nil
+}
+
+func (s *Setup) emit(step string, changed bool, format string, a ...any) {
+	if s.Emit != nil {
+		s.Emit(step, fmt.Sprintf(format, a...), changed)
+	}
+}
+
+// etcdMembers are the machines that vote: the database machines, and the
+// witness when they are only two.
+func (s *Setup) etcdMembers() []Node {
+	out := append([]Node{}, s.DB...)
+	if len(s.DB) == 2 && s.Witness != nil {
+		out = append(out, *s.Witness)
+	}
+	return out
+}
+
+// Run sets the cluster up, or confirms it is, and returns the store address.
+func (s *Setup) Run(ctx context.Context) (string, error) {
+	if len(s.DB) < 2 {
+		return "", fmt.Errorf("a replicated store needs at least two database machines, not %d", len(s.DB))
+	}
+	if len(s.DB) == 2 && s.Witness == nil {
+		return "", fmt.Errorf("two database machines need a third etcd vote, or losing either stops failover")
+	}
+	if s.Wait == 0 {
+		s.Wait = 3 * time.Minute
+	}
+	if s.Status == nil {
+		s.Status = HTTPStatus
+	}
+	sec, err := s.secrets(ctx)
+	if err != nil {
+		return "", fmt.Errorf("passwords: %w", err)
+	}
+	if err := s.etcd(ctx); err != nil {
+		return "", fmt.Errorf("etcd: %w", err)
+	}
+	if err := s.patroni(ctx, sec); err != nil {
+		return "", fmt.Errorf("patroni: %w", err)
+	}
+	if err := s.database(ctx); err != nil {
+		return "", fmt.Errorf("database: %w", err)
+	}
+	members := make([]Member, len(s.DB))
+	for i, d := range s.DB {
+		members[i] = d.Member
+	}
+	return DSN(members, sec.App), nil
+}
+
+// secretDir holds the passwords on every database machine, readable by root
+// alone. Generated on the first, carried to the others.
+const secretDir = "/root/llmgw-pg"
+
+var secretNames = []string{"superuser", "replicator", "llmgw"}
+
+func (s *Setup) secrets(ctx context.Context) (Secrets, error) {
+	script := `set -e
+sudo install -d -m 700 ` + secretDir + `
+for f in superuser replicator llmgw; do
+  sudo test -s ` + secretDir + `/$f.pw || sudo sh -c "umask 077; openssl rand -hex 24 > ` + secretDir + `/$f.pw"
+done
+for f in superuser replicator llmgw; do printf '%s=' $f; sudo cat ` + secretDir + `/$f.pw; done`
+	out, err := s.DB[0].Remote.Run(ctx, script, nil)
+	if err != nil {
+		return Secrets{}, fmt.Errorf("on %s: %v: %s", s.DB[0].Name, err, strings.TrimSpace(string(out)))
+	}
+	got := parseSecrets(out)
+	if got.Superuser == "" || got.Replicator == "" || got.App == "" {
+		return Secrets{}, fmt.Errorf("could not read them back from %s", s.DB[0].Name)
+	}
+	for _, d := range s.DB[1:] {
+		have, err := d.Remote.Run(ctx, `for f in superuser replicator llmgw; do
+  if sudo test -s `+secretDir+`/$f.pw; then printf '%s=' $f; sudo cat `+secretDir+`/$f.pw; fi
+done`, nil)
+		if err != nil {
+			return Secrets{}, fmt.Errorf("on %s: %v", d.Name, err)
+		}
+		if h := parseSecrets(have); h != (Secrets{}) {
+			if h != got {
+				// Not overwritten: a member holding other passwords belongs
+				// to another cluster, or to an earlier one somebody meant to
+				// keep, and replacing them would lock it out of itself.
+				return Secrets{}, fmt.Errorf("%s holds passwords in %s that differ from %s's; "+
+					"remove them there if that machine is meant to join this cluster", d.Name, secretDir, s.DB[0].Name)
+			}
+			continue
+		}
+		body := fmt.Sprintf("%s\n%s\n%s\n", got.Superuser, got.Replicator, got.App)
+		if out, err := d.Remote.Run(ctx, `set -e
+sudo install -d -m 700 `+secretDir+`
+read su; read rep; read app
+for kv in "superuser $su" "replicator $rep" "llmgw $app"; do
+  set -- $kv
+  printf '%s\n' "$2" | sudo sh -c "umask 077; cat > `+secretDir+`/$1.pw"
+done`, []byte(body)); err != nil {
+			return Secrets{}, fmt.Errorf("on %s: %v: %s", d.Name, err, strings.TrimSpace(string(out)))
+		}
+		s.emit("passwords", true, "carried to %s", d.Name)
+	}
+	s.emit("passwords", false, "kept in %s on every database machine", secretDir)
+	return got, nil
+}
+
+func parseSecrets(out []byte) Secrets {
+	var s Secrets
+	for _, line := range strings.Split(string(out), "\n") {
+		k, v, ok := strings.Cut(strings.TrimSpace(line), "=")
+		if !ok {
+			continue
+		}
+		switch k {
+		case "superuser":
+			s.Superuser = v
+		case "replicator":
+			s.Replicator = v
+		case "llmgw":
+			s.App = v
+		}
+	}
+	return s
+}
+
+// etcd brings up the voting members, unless every one of them is already
+// healthy in one cluster — in which case nothing is touched, since rewriting
+// a member's configuration means emptying its data.
+func (s *Setup) etcd(ctx context.Context) error {
+	nodes := s.etcdMembers()
+	var members []Member
+	var endpoints []string
+	for _, n := range nodes {
+		members = append(members, n.Member)
+		endpoints = append(endpoints, "http://"+n.IP+":2379")
+	}
+	if s.etcdHealthy(ctx, nodes[0], endpoints, len(nodes)) {
+		s.emit("etcd", false, "%d members healthy; left as they are", len(nodes))
+		return nil
+	}
+
+	changed := make([]bool, len(nodes))
+	for i, n := range nodes {
+		out, err := n.Remote.Run(ctx, `set -e
+if ! command -v etcd >/dev/null 2>&1; then
+  sudo DEBIAN_FRONTEND=noninteractive apt-get -qq update >/dev/null
+  sudo DEBIAN_FRONTEND=noninteractive apt-get -qq install -y etcd-server etcd-client >/dev/null
+fi
+new=$(cat)
+if [ "$(sudo cat /etc/default/etcd 2>/dev/null)" != "$new" ]; then
+  sudo systemctl stop etcd >/dev/null 2>&1 || true
+  sudo rm -rf /var/lib/etcd/default
+  printf '%s\n' "$new" | sudo tee /etc/default/etcd >/dev/null
+  echo changed
+fi
+sudo systemctl enable etcd >/dev/null 2>&1`, []byte(Etcd(n.Member, members)))
+		if err != nil {
+			return fmt.Errorf("on %s: %v: %s", n.Name, err, strings.TrimSpace(string(out)))
+		}
+		changed[i] = strings.Contains(string(out), "changed")
+	}
+	// Started together: a new member waits for the others before it says it
+	// is up, so starting them one after another would wait for ever on the
+	// first.
+	var wg sync.WaitGroup
+	errs := make([]error, len(nodes))
+	for i, n := range nodes {
+		wg.Add(1)
+		go func(i int, n Node) {
+			defer wg.Done()
+			verb := "start"
+			if changed[i] {
+				verb = "restart"
+			}
+			if out, err := n.Remote.Run(ctx, "sudo systemctl "+verb+" --no-block etcd", nil); err != nil {
+				errs[i] = fmt.Errorf("on %s: %v: %s", n.Name, err, strings.TrimSpace(string(out)))
+			}
+		}(i, n)
+	}
+	wg.Wait()
+	for _, err := range errs {
+		if err != nil {
+			return err
+		}
+	}
+	deadline := time.Now().Add(s.Wait)
+	for !s.etcdHealthy(ctx, nodes[0], endpoints, len(nodes)) {
+		if time.Now().After(deadline) {
+			return fmt.Errorf("the %d members did not become healthy within %s; `journalctl -u etcd` on each says why", len(nodes), s.Wait)
+		}
+		if err := sleep(ctx, 2*time.Second); err != nil {
+			return err
+		}
+	}
+	s.emit("etcd", true, "%d members up", len(nodes))
+	return nil
+}
+
+func (s *Setup) etcdHealthy(ctx context.Context, n Node, endpoints []string, want int) bool {
+	out, err := n.Remote.Run(ctx, "ETCDCTL_API=3 etcdctl --dial-timeout=3s --endpoints="+
+		strings.Join(endpoints, ",")+" endpoint health 2>&1", nil)
+	return err == nil && strings.Count(string(out), "is healthy") == want
+}
+
+// patroni installs PostgreSQL and Patroni on every database machine, unless
+// the cluster already has a leader and every member streaming.
+func (s *Setup) patroni(ctx context.Context, sec Secrets) error {
+	hosts := make([]string, len(s.DB))
+	for i, d := range s.DB {
+		hosts[i] = d.IP
+	}
+	if c, err := s.Status(ctx, hosts); err == nil && c.Complete(len(s.DB)) {
+		s.emit("patroni", false, "%s leads, %d streaming; left as it is", c.Leader(), len(s.DB)-1)
+		return nil
+	}
+
+	var version string
+	for _, d := range s.DB {
+		v, err := s.preparePostgres(ctx, d)
+		if err != nil {
+			return err
+		}
+		if version != "" && v != version {
+			// Streaming replication is between one major version.
+			return fmt.Errorf("%s has PostgreSQL %s and %s has %s; they must match", s.DB[0].Name, version, d.Name, v)
+		}
+		version = v
+	}
+
+	var etcd, db []Member
+	for _, n := range s.etcdMembers() {
+		etcd = append(etcd, n.Member)
+	}
+	for _, d := range s.DB {
+		db = append(db, d.Member)
+	}
+	for i, d := range s.DB {
+		cfg := Patroni(d.Member, db, etcd, s.Clients, version, sec)
+		if out, err := d.Remote.Run(ctx, `set -e
+sudo install -d -m 755 /etc/patroni
+sudo install -d -o postgres -g postgres -m 700 /var/lib/postgresql/`+version+`/llmgw
+sudo sh -c 'umask 027; cat > /etc/patroni/config.yml.new'
+sudo chown root:postgres /etc/patroni/config.yml.new
+sudo chmod 640 /etc/patroni/config.yml.new
+sudo mv -f /etc/patroni/config.yml.new /etc/patroni/config.yml
+sudo systemctl enable patroni >/dev/null 2>&1
+sudo systemctl restart patroni`, []byte(cfg)); err != nil {
+			return fmt.Errorf("on %s: %v: %s", d.Name, err, strings.TrimSpace(string(out)))
+		}
+		// The first starts the cluster; the rest clone it, which they can
+		// only do once it has a leader.
+		want := func(c Cluster) bool { return c.Leader() != "" }
+		if i > 0 {
+			want = func(c Cluster) bool { return c.Complete(i + 1) }
+		}
+		if err := s.waitFor(ctx, hosts, want); err != nil {
+			return fmt.Errorf("after starting %s: %w", d.Name, err)
+		}
+		s.emit("patroni", true, "%s up", d.Name)
+	}
+	return nil
+}
+
+// preparePostgres installs the packages and clears the distribution's own
+// cluster, which would hold the port Patroni's needs — unless it holds a
+// database somebody made, which is not this setup's to delete.
+func (s *Setup) preparePostgres(ctx context.Context, d Node) (string, error) {
+	out, err := d.Remote.Run(ctx, `set -e
+need=""
+command -v patroni >/dev/null 2>&1 || need="$need patroni"
+ls /usr/lib/postgresql/*/bin/postgres >/dev/null 2>&1 || need="$need postgresql"
+if [ -n "$need" ]; then
+  sudo DEBIAN_FRONTEND=noninteractive apt-get -qq update >/dev/null
+  sudo DEBIAN_FRONTEND=noninteractive apt-get -qq install -y $need >/dev/null
+fi
+v=$(ls /usr/lib/postgresql | sort -n | tail -1)
+if [ -d /var/lib/postgresql/$v/main ]; then
+  if sudo systemctl is-active --quiet postgresql@$v-main; then
+    n=$(sudo -u postgres psql -Atc "select count(*) from pg_database where datname not in ('postgres','template0','template1')")
+    if [ "$n" != "0" ]; then echo "holds-data"; exit 0; fi
+  fi
+  sudo systemctl disable --now postgresql@$v-main postgresql >/dev/null 2>&1 || true
+  sudo pg_dropcluster --stop $v main >/dev/null 2>&1 || sudo rm -rf /var/lib/postgresql/$v/main
+fi
+sudo systemctl disable --now postgresql >/dev/null 2>&1 || true
+echo "version=$v"`, nil)
+	if err != nil {
+		return "", fmt.Errorf("on %s: %v: %s", d.Name, err, strings.TrimSpace(string(out)))
+	}
+	if strings.Contains(string(out), "holds-data") {
+		return "", fmt.Errorf("%s runs a PostgreSQL with databases in it; move or drop them first — "+
+			"this setup replaces that cluster", d.Name)
+	}
+	for _, line := range strings.Split(string(out), "\n") {
+		if v, ok := strings.CutPrefix(strings.TrimSpace(line), "version="); ok && v != "" {
+			return v, nil
+		}
+	}
+	return "", fmt.Errorf("no PostgreSQL found on %s after installing it", d.Name)
+}
+
+// database makes the role and database on whichever member leads.
+func (s *Setup) database(ctx context.Context) error {
+	hosts := make([]string, len(s.DB))
+	for i, d := range s.DB {
+		hosts[i] = d.IP
+	}
+	c, err := s.Status(ctx, hosts)
+	if err != nil {
+		return err
+	}
+	var leader *Node
+	for i := range s.DB {
+		if s.DB[i].Name == c.Leader() {
+			leader = &s.DB[i]
+		}
+	}
+	if leader == nil {
+		return fmt.Errorf("no member leads")
+	}
+	out, err := leader.Remote.Run(ctx, `set -e
+pw=$(sudo cat `+secretDir+`/llmgw.pw)
+sudo -u postgres psql -q -v ON_ERROR_STOP=1 -v pw="$pw" <<'SQL'
+SELECT format('CREATE ROLE %I LOGIN', 'llmgw') WHERE NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'llmgw') \gexec
+ALTER ROLE llmgw PASSWORD :'pw';
+SELECT format('CREATE DATABASE %I OWNER %I', 'llmgw', 'llmgw') WHERE NOT EXISTS (SELECT 1 FROM pg_database WHERE datname = 'llmgw') \gexec
+SQL`, nil)
+	if err != nil {
+		return fmt.Errorf("on %s: %v: %s", leader.Name, err, strings.TrimSpace(string(out)))
+	}
+	s.emit("database", true, "role and database %s on %s", Database, leader.Name)
+	return nil
+}
+
+func (s *Setup) waitFor(ctx context.Context, hosts []string, ok func(Cluster) bool) error {
+	deadline := time.Now().Add(s.Wait)
+	var last error
+	for {
+		c, err := s.Status(ctx, hosts)
+		if err == nil && ok(c) {
+			return nil
+		}
+		if err != nil {
+			last = err
+		} else {
+			last = fmt.Errorf("the cluster is %s", c)
+		}
+		if time.Now().After(deadline) {
+			return fmt.Errorf("not up within %s: %v; `journalctl -u patroni` says why", s.Wait, last)
+		}
+		if err := sleep(ctx, 3*time.Second); err != nil {
+			return err
+		}
+	}
+}
+
+func sleep(ctx context.Context, d time.Duration) error {
+	select {
+	case <-ctx.Done():
+		return ctx.Err()
+	case <-time.After(d):
+		return nil
+	}
+}

internal/pgha/status.go

@@ -0,0 +1,114 @@
+package pgha
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"net"
+	"net/http"
+	"strconv"
+	"strings"
+	"time"
+)
+
+// MemberState is one member as Patroni reports it.
+type MemberState struct {
+	Name string `json:"name"`
+	// Role is "leader", "replica" or "sync_standby".
+	Role string `json:"role"`
+	// State is "running" for a leader, "streaming" for a replica keeping up,
+	// and anything else for one that is not.
+	State    string `json:"state"`
+	Host     string `json:"host"`
+	Timeline int    `json:"timeline"`
+	// Lag is how far behind the leader a replica is, in bytes; -1 when
+	// Patroni does not know, which is itself worth an alert.
+	Lag    int64           `json:"-"`
+	RawLag json.RawMessage `json:"lag"`
+}
+
+// Cluster is the cluster's state, as any member's API reports it.
+type Cluster struct {
+	Members []MemberState `json:"members"`
+}
+
+// Leader is the leading member's name, or "".
+func (c Cluster) Leader() string {
+	for _, m := range c.Members {
+		if m.Role == "leader" || m.Role == "standby_leader" {
+			return m.Name
+		}
+	}
+	return ""
+}
+
+// Complete reports whether n members are up: one leading, the rest streaming.
+func (c Cluster) Complete(n int) bool {
+	if c.Leader() == "" || len(c.Members) < n {
+		return false
+	}
+	streaming := 0
+	for _, m := range c.Members {
+		if m.Role != "leader" && m.State == "streaming" {
+			streaming++
+		}
+	}
+	return streaming >= n-1
+}
+
+func (c Cluster) String() string {
+	if len(c.Members) == 0 {
+		return "empty"
+	}
+	var parts []string
+	for _, m := range c.Members {
+		parts = append(parts, m.Name+" "+m.Role+"/"+m.State)
+	}
+	return strings.Join(parts, ", ")
+}
+
+// HTTPStatus asks each host's Patroni in turn and returns the first answer:
+// any member describes the whole cluster.
+func HTTPStatus(ctx context.Context, hosts []string) (Cluster, error) {
+	client := &http.Client{Timeout: 3 * time.Second}
+	var last error
+	for _, h := range hosts {
+		u := "http://" + net.JoinHostPort(h, strconv.Itoa(restPort)) + "/cluster"
+		req, err := http.NewRequestWithContext(ctx, http.MethodGet, u, nil)
+		if err != nil {
+			return Cluster{}, err
+		}
+		resp, err := client.Do(req)
+		if err != nil {
+			last = err
+			continue
+		}
+		var c Cluster
+		err = json.NewDecoder(resp.Body).Decode(&c)
+		resp.Body.Close()
+		if err != nil {
+			last = fmt.Errorf("%s: %w", u, err)
+			continue
+		}
+		for i := range c.Members {
+			c.Members[i].Lag = lagOf(c.Members[i].RawLag)
+		}
+		return c, nil
+	}
+	if last == nil {
+		last = fmt.Errorf("no hosts to ask")
+	}
+	return Cluster{}, fmt.Errorf("no member's Patroni answered: %w", last)
+}
+
+// lagOf reads Patroni's lag: a number of bytes, or "unknown".
+func lagOf(raw json.RawMessage) int64 {
+	if len(raw) == 0 {
+		return 0
+	}
+	var n int64
+	if json.Unmarshal(raw, &n) == nil {
+		return n
+	}
+	return -1
+}

internal/preflight/preflight.go

@@ -0,0 +1,360 @@
+// Package preflight checks machines before anything is installed on them:
+// that they can be logged into and administered, run a supported system, have
+// room for their role, keep time, have the ports free, and can reach the
+// internet and each other. Everything wrong is found at once, with what to
+// do about it, rather than one failure at a time halfway through an install.
+package preflight
+
+import (
+	"context"
+	"fmt"
+	"sort"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
+)
+
+// Role is what a machine is going to be.
+type Role string
+
+// The roles a setup gives machines.
+const (
+	RoleControl  Role = "control plane"
+	RoleDatabase Role = "database"
+	RoleWitness  Role = "etcd witness"
+	RoleNode     Role = "node"
+)
+
+// Need is the least a role asks of a machine.
+type Need struct {
+	CPUs     int
+	MemoryMB int
+	DiskGB   int
+	Ports    []int
+}
+
+// Needs are what each role asks for; a machine with several roles needs the
+// largest of each and the ports of all.
+var Needs = map[Role]Need{
+	RoleControl:  {CPUs: 2, MemoryMB: 3500, DiskGB: 15, Ports: []int{7443, 8080}},
+	RoleDatabase: {CPUs: 2, MemoryMB: 3500, DiskGB: 30, Ports: []int{5432, 8008, 2379, 2380}},
+	RoleWitness:  {CPUs: 1, MemoryMB: 900, DiskGB: 5, Ports: []int{2379, 2380}},
+	RoleNode:     {CPUs: 2, MemoryMB: 3500, DiskGB: 20},
+}
+
+// Machine is one machine to check.
+type Machine struct {
+	Name   string // how the screen calls it
+	Addr   string // the address the others will use
+	Roles  []Role
+	Remote onboard.Remote
+	// Local is this machine, where sudo is not needed (setup runs as root).
+	Local bool
+}
+
+// Level is how bad a finding is.
+type Level int
+
+// Levels, mildest first.
+const (
+	OK Level = iota
+	Warn
+	Fail
+)
+
+func (l Level) String() string { return [...]string{"ok", "warn", "fail"}[l] }
+
+// Finding is one thing checked on one machine.
+type Finding struct {
+	Machine string
+	Check   string
+	Level   Level
+	Detail  string
+	// Fix says what to do, for anything that is not OK.
+	Fix string
+}
+
+// Report is everything found.
+type Report struct {
+	Findings []Finding
+}
+
+// Worst is the most serious level found.
+func (r Report) Worst() Level {
+	w := OK
+	for _, f := range r.Findings {
+		if f.Level > w {
+			w = f.Level
+		}
+	}
+	return w
+}
+
+// For is one machine's findings, in check order.
+func (r Report) For(machine string) []Finding {
+	var out []Finding
+	for _, f := range r.Findings {
+		if f.Machine == machine {
+			out = append(out, f)
+		}
+	}
+	return out
+}
+
+// probeScript gathers everything in one round trip, as key=value lines, so
+// the checks are decided here and can be tested without a machine.
+const probeScript = `
+. /etc/os-release 2>/dev/null; echo "os=$ID"; echo "version=$VERSION_ID"; echo "pretty=$PRETTY_NAME"
+echo "arch=$(uname -m)"
+echo "cpus=$(nproc)"
+echo "mem_mb=$(awk '/MemTotal/{print int($2/1024)}' /proc/meminfo)"
+echo "disk_gb=$(df -BG --output=avail / | tail -1 | tr -dc 0-9)"
+echo "now=$(date +%s.%N)"
+echo "ntp=$(timedatectl show -p NTPSynchronized --value 2>/dev/null)"
+echo "listening=$(ss -Hltn 2>/dev/null | awk '{n=split($4,a,":"); print a[n]}' | sort -un | tr '\n' ' ')"
+if sudo -n true 2>/dev/null; then echo "sudo=yes"; else echo "sudo=no"; fi
+echo "apt=$(command -v apt-get >/dev/null && echo yes || echo no)"
+echo "internet=$(timeout 6 bash -c 'exec 3<>/dev/tcp/deb.debian.org/80' 2>/dev/null && echo yes || (command -v curl >/dev/null && curl -s -m 6 -o /dev/null http://deb.debian.org && echo yes) || echo no)"
+`
+
+// reachScript tries one TCP port on each address: whether the machines can
+// reach one another at all. Port 22 because it is open everywhere before
+// anything is installed; the ports llmgw will use have nothing listening yet.
+func reachScript(addrs []string) string {
+	var b strings.Builder
+	for _, a := range addrs {
+		fmt.Fprintf(&b, "if timeout 4 bash -c 'exec 3<>/dev/tcp/%s/22' 2>/dev/null; then echo \"reach %s yes\"; else echo \"reach %s no\"; fi\n", a, a, a)
+	}
+	return b.String()
+}
+
+// Run checks every machine, concurrently.
+func Run(ctx context.Context, machines []Machine) Report {
+	var (
+		mu  sync.Mutex
+		rep Report
+		wg  sync.WaitGroup
+	)
+	add := func(fs ...Finding) {
+		mu.Lock()
+		rep.Findings = append(rep.Findings, fs...)
+		mu.Unlock()
+	}
+	var addrs []string
+	for _, m := range machines {
+		addrs = append(addrs, m.Addr)
+	}
+	order := map[string]int{}
+	for i, m := range machines {
+		order[m.Name] = i
+		wg.Add(1)
+		go func(m Machine) {
+			defer wg.Done()
+			add(check(ctx, m, addrs)...)
+		}(m)
+	}
+	wg.Wait()
+	// Machines in the order given, checks in the order made.
+	sort.SliceStable(rep.Findings, func(i, j int) bool {
+		return order[rep.Findings[i].Machine] < order[rep.Findings[j].Machine]
+	})
+	return rep
+}
+
+func check(ctx context.Context, m Machine, all []string) []Finding {
+	sent := time.Now()
+	out, err := m.Remote.Run(ctx, probeScript, nil)
+	took := time.Since(sent)
+	if err != nil {
+		return []Finding{{Machine: m.Name, Check: "login", Level: Fail,
+			Detail: strings.TrimSpace(firstLine(err.Error() + " " + string(out))),
+			Fix:    "check the address, the user and the key or password given for it"}}
+	}
+	facts := parse(out)
+	fs := Evaluate(m, facts, sent.Add(took/2))
+	var others []string
+	for _, a := range all {
+		if a != m.Addr {
+			others = append(others, a)
+		}
+	}
+	if len(others) > 0 {
+		rout, _ := m.Remote.Run(ctx, reachScript(others), nil)
+		fs = append(fs, evaluateReach(m, string(rout))...)
+	}
+	return fs
+}
+
+func firstLine(s string) string {
+	s = strings.TrimSpace(s)
+	if i := strings.IndexByte(s, '\n'); i >= 0 {
+		return s[:i]
+	}
+	return s
+}
+
+func parse(out []byte) map[string]string {
+	facts := map[string]string{}
+	for _, line := range strings.Split(string(out), "\n") {
+		if k, v, ok := strings.Cut(strings.TrimSpace(line), "="); ok {
+			facts[k] = strings.TrimSpace(v)
+		}
+	}
+	return facts
+}
+
+// supported are the systems the setup installs packages on with apt.
+var supported = map[string][]string{
+	"ubuntu": {"22.04", "24.04"},
+	"debian": {"12"},
+}
+
+// Evaluate turns one machine's facts into findings. at is when the machine's
+// clock was read, by this machine's clock.
+func Evaluate(m Machine, facts map[string]string, at time.Time) []Finding {
+	var fs []Finding
+	f := func(check string, l Level, detail, fix string) {
+		fs = append(fs, Finding{Machine: m.Name, Check: check, Level: l, Detail: detail, Fix: fix})
+	}
+	f("login", OK, "logged in", "")
+
+	// The system.
+	os, ver := facts["os"], facts["version"]
+	switch vers, ok := supported[os]; {
+	case !ok:
+		f("system", Fail, facts["pretty"], "use Ubuntu 22.04 or 24.04, or Debian 12")
+	case !contains(vers, ver):
+		f("system", Warn, facts["pretty"]+" (tried on "+strings.Join(vers, ", ")+")", "it may work; Ubuntu 24.04 is what this is tested on")
+	default:
+		f("system", OK, facts["pretty"], "")
+	}
+	switch facts["arch"] {
+	case "x86_64", "aarch64":
+		f("architecture", OK, facts["arch"], "")
+	default:
+		f("architecture", Fail, facts["arch"], "llmgw is built for amd64 and arm64")
+	}
+
+	// Administering it.
+	if !m.Local {
+		if facts["sudo"] == "yes" {
+			f("sudo", OK, "without a password", "")
+		} else {
+			// Not a failure: the setup asks for the password and grants
+			// itself sudo for the length of the install.
+			f("sudo", Warn, "asks for a password", "give the sudo password on the access screen")
+		}
+	}
+	if facts["apt"] != "yes" {
+		f("packages", Fail, "no apt-get", "the setup installs PostgreSQL, Patroni and etcd with apt")
+	}
+
+	// Room for its roles.
+	need := needOf(m.Roles)
+	cpus, _ := strconv.Atoi(facts["cpus"])
+	mem, _ := strconv.Atoi(facts["mem_mb"])
+	disk, _ := strconv.Atoi(facts["disk_gb"])
+	room := func(check string, have, want int, unit string) {
+		d := fmt.Sprintf("%d %s (needs %d)", have, unit, want)
+		switch {
+		case have >= want:
+			f(check, OK, d, "")
+		case have*2 >= want:
+			f(check, Warn, d, "it will run, short of what the role is sized for")
+		default:
+			f(check, Fail, d, "give the machine more")
+		}
+	}
+	room("cpus", cpus, need.CPUs, "CPUs")
+	room("memory", mem, need.MemoryMB, "MB")
+	room("disk", disk, need.DiskGB, "GB free")
+
+	// Time. Leases and etcd's elections go wrong on clocks that disagree.
+	if now, err := strconv.ParseFloat(facts["now"], 64); err == nil {
+		skew := time.Duration((now - float64(at.UnixNano())/1e9) * float64(time.Second))
+		if skew < 0 {
+			skew = -skew
+		}
+		switch {
+		case skew > 5*time.Second:
+			f("clock", Fail, fmt.Sprintf("%s off from this machine", skew.Round(time.Second)), "turn on NTP (timedatectl set-ntp true) on both")
+		case skew > time.Second:
+			f("clock", Warn, fmt.Sprintf("%s off from this machine", skew.Round(100*time.Millisecond)), "turn on NTP on both")
+		default:
+			f("clock", OK, "within a second of this machine", "")
+		}
+	}
+	if facts["ntp"] != "yes" {
+		f("time sync", Warn, "NTP is not synchronised", "timedatectl set-ntp true")
+	}
+
+	// Ports its roles will listen on.
+	listening := map[string]bool{}
+	for _, p := range strings.Fields(facts["listening"]) {
+		listening[p] = true
+	}
+	var busy []string
+	for _, p := range need.Ports {
+		if listening[strconv.Itoa(p)] {
+			busy = append(busy, strconv.Itoa(p))
+		}
+	}
+	if len(busy) > 0 {
+		// A warning: a re-run finds its own services listening.
+		f("ports", Warn, "already listening: "+strings.Join(busy, ", "),
+			"fine if it is llmgw's own from an earlier run; otherwise stop what holds them")
+	} else if len(need.Ports) > 0 {
+		f("ports", OK, "free", "")
+	}
+
+	if facts["internet"] == "yes" {
+		f("internet", OK, "reaches the package mirrors", "")
+	} else {
+		f("internet", Fail, "cannot reach deb.debian.org", "packages and releases are downloaded; allow outbound HTTP/HTTPS or set a proxy")
+	}
+	return fs
+}
+
+func evaluateReach(m Machine, out string) []Finding {
+	var no []string
+	for _, line := range strings.Split(out, "\n") {
+		parts := strings.Fields(line)
+		if len(parts) == 3 && parts[0] == "reach" && parts[2] == "no" {
+			no = append(no, parts[1])
+		}
+	}
+	if len(no) > 0 {
+		return []Finding{{Machine: m.Name, Check: "network", Level: Fail,
+			Detail: "cannot reach " + strings.Join(no, ", "),
+			Fix:    "the machines must reach one another; check routes and firewalls"}}
+	}
+	return []Finding{{Machine: m.Name, Check: "network", Level: OK, Detail: "reaches every other machine"}}
+}
+
+func needOf(roles []Role) Need {
+	var n Need
+	seen := map[int]bool{}
+	for _, r := range roles {
+		x := Needs[r]
+		n.CPUs, n.MemoryMB, n.DiskGB = max(n.CPUs, x.CPUs), max(n.MemoryMB, x.MemoryMB), max(n.DiskGB, x.DiskGB)
+		for _, p := range x.Ports {
+			if !seen[p] {
+				seen[p] = true
+				n.Ports = append(n.Ports, p)
+			}
+		}
+	}
+	return n
+}
+
+func contains(xs []string, x string) bool {
+	for _, y := range xs {
+		if y == x {
+			return true
+		}
+	}
+	return false
+}

internal/preflight/preflight_test.go

@@ -0,0 +1,109 @@
+package preflight
+
+import (
+	"context"
+	"os"
+	"strconv"
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
+)
+
+func facts(over map[string]string) map[string]string {
+	f := map[string]string{
+		"os": "ubuntu", "version": "24.04", "pretty": "Ubuntu 24.04.3 LTS", "arch": "x86_64",
+		"cpus": "2", "mem_mb": "3900", "disk_gb": "35", "ntp": "yes", "listening": "22 53",
+		"sudo": "yes", "apt": "yes", "internet": "yes",
+		"now": strconv.FormatFloat(float64(time.Now().UnixNano())/1e9, 'f', 3, 64),
+	}
+	for k, v := range over {
+		f[k] = v
+	}
+	return f
+}
+
+func levelOf(fs []Finding, check string) (Level, string) {
+	for _, f := range fs {
+		if f.Check == check {
+			return f.Level, f.Detail
+		}
+	}
+	return -1, ""
+}
+
+func TestAMachineFitForItsRoleHasNothingWrong(t *testing.T) {
+	fs := Evaluate(Machine{Name: "pg01", Roles: []Role{RoleDatabase}}, facts(nil), time.Now())
+	if r := (Report{Findings: fs}); r.Worst() != OK {
+		t.Errorf("%+v", fs)
+	}
+}
+
+func TestEachProblemIsFoundAndSaysWhatToDo(t *testing.T) {
+	m := Machine{Name: "x", Roles: []Role{RoleDatabase, RoleControl}}
+	cases := []struct {
+		over  map[string]string
+		check string
+		want  Level
+	}{
+		{map[string]string{"os": "centos", "version": "9"}, "system", Fail},
+		{map[string]string{"version": "20.04"}, "system", Warn},
+		{map[string]string{"arch": "riscv64"}, "architecture", Fail},
+		{map[string]string{"sudo": "no"}, "sudo", Warn},
+		{map[string]string{"mem_mb": "1000"}, "memory", Fail},
+		{map[string]string{"disk_gb": "20"}, "disk", Warn},
+		{map[string]string{"ntp": "no"}, "time sync", Warn},
+		{map[string]string{"listening": "22 5432"}, "ports", Warn},
+		{map[string]string{"internet": "no"}, "internet", Fail},
+		{map[string]string{"now": strconv.FormatFloat(float64(time.Now().Add(-30*time.Second).UnixNano())/1e9, 'f', 3, 64)}, "clock", Fail},
+	}
+	for _, c := range cases {
+		fs := Evaluate(m, facts(c.over), time.Now())
+		got, detail := levelOf(fs, c.check)
+		if got != c.want {
+			t.Errorf("%v: %s is %v (%s), want %v", c.over, c.check, got, detail, c.want)
+		}
+		for _, f := range fs {
+			if f.Level != OK && f.Fix == "" {
+				t.Errorf("%s: %s says nothing about what to do", c.check, f.Check)
+			}
+		}
+	}
+	// This machine is run as root; sudo is not its question.
+	if l, _ := levelOf(Evaluate(Machine{Name: "here", Local: true}, facts(map[string]string{"sudo": "no"}), time.Now()), "sudo"); l != -1 {
+		t.Error("the local machine was asked about sudo")
+	}
+}
+
+func TestAMachineThatCannotReachAnotherIsCalledOut(t *testing.T) {
+	fs := evaluateReach(Machine{Name: "a"}, "reach 10.0.0.2 yes\nreach 10.0.0.3 no\n")
+	if fs[0].Level != Fail || !strings.Contains(fs[0].Detail, "10.0.0.3") {
+		t.Errorf("%+v", fs)
+	}
+}
+
+// TestAgainstRealMachines runs the probe over ssh when LLMGW_TEST_PREFLIGHT
+// names user@host,user@host: the scripts are shell, and only a real machine
+// says whether they parse.
+func TestAgainstRealMachines(t *testing.T) {
+	targets := os.Getenv("LLMGW_TEST_PREFLIGHT")
+	if targets == "" {
+		t.Skip("set LLMGW_TEST_PREFLIGHT=user@host,user@host to run against machines")
+	}
+	var ms []Machine
+	for _, tgt := range strings.Split(targets, ",") {
+		user, host, _ := strings.Cut(tgt, "@")
+		ms = append(ms, Machine{Name: host, Addr: host, Roles: []Role{RoleDatabase},
+			Remote: &onboard.GoSSH{Addr: host, User: user, UseAgent: true}})
+	}
+	rep := Run(context.Background(), ms)
+	for _, f := range rep.Findings {
+		t.Logf("%-12s %-12s %-5s %s", f.Machine, f.Check, f.Level, f.Detail)
+	}
+	for _, f := range rep.Findings {
+		if f.Check == "login" && f.Level != OK {
+			t.Fatalf("could not log in: %s", f.Detail)
+		}
+	}
+}

internal/router/chat.go

@@ -255,6 +255,18 @@ func (r *Router) modelsFor(req *http.Request, user store.User) []web.ChatModel {
 	}
 	seen := map[string]struct{}{}
 	var out []web.ChatModel
+	// What each model's replicas compute on, across all of them: one model
+	// can run on a card and on CPUs at once.
+	hardware := map[string][]string{}
+	for _, s := range r.reg.Snapshots("") {
+		hw := hardwareOf(s.Pool)
+		for _, m := range s.Models {
+			name, _ := r.resolve(m)
+			if hw != "" && !slices.Contains(hardware[name], hw) {
+				hardware[name] = append(hardware[name], hw)
+			}
+		}
+	}
 	for _, s := range r.reg.Snapshots("") {
 		for _, m := range s.Models {
 			// Resolved rather than taken as sent: an alias and the name it
@@ -265,7 +277,7 @@ func (r *Router) modelsFor(req *http.Request, user store.User) []web.ChatModel {
 			}
 			seen[name] = struct{}{}
 			if r.permits(principal, name) {
-				out = append(out, web.ChatModel{Name: name, Simulated: entry.Simulated()})
+				out = append(out, web.ChatModel{Name: name, Simulated: entry.Simulated(), Hardware: hardware[name]})
 			}
 		}
 	}

internal/router/handlers.go

@@ -123,6 +123,19 @@ func (r *Router) Handler() http.Handler {
 		w.WriteHeader(http.StatusOK)
 	})
 
+	// Leadership: 200 on the control plane that places work, 503 on a
+	// standby. What a floating address follows (llmgw server add --vip):
+	// unlike /ready it does not wait for a replica, so a cluster with no
+	// node yet still has its address on one machine. A process that is not
+	// a control plane, or one with no rival, is always the leader.
+	mux.HandleFunc("GET /leader", func(w http.ResponseWriter, _ *http.Request) {
+		if f := r.isLeader.Load(); f != nil && !(*f)() {
+			http.Error(w, "this control plane is a standby", http.StatusServiceUnavailable)
+			return
+		}
+		w.WriteHeader(http.StatusOK)
+	})
+
 	// The landing page says how to connect and nothing about what is here, so
 	// it is safe to serve without a key — and it has to be, because the person
 	// reading it is the one who does not have one yet.
@@ -203,6 +216,9 @@ func (r *Router) Handler() http.Handler {
 			}
 			pages.RegisterAdminQuotas(mux, r.store, r.acct)
 			// And the report of what that tree actually used.
+			if r.history != nil {
+				pages.SetHistory(r.history)
+			}
 			pages.RegisterAdminUsage(mux, r.store, r.acct)
 		}
 
@@ -224,7 +240,7 @@ func (r *Router) Handler() http.Handler {
 	// route must not be able to add an unauthenticated one by omission. Health
 	// stays open because a load balancer has no key.
 	return auth.Middleware(r.auth, r.log,
-		"/health", "/ready", "/", "/about", "/_static/", "/claim/",
+		"/health", "/ready", "/leader", "/", "/about", "/_static/", "/claim/",
 		"/signup", "/login", "/logout", "/profile", "/profile/",
 		"/admin", "/admin/", "/chat", "/chat/", "/org", "/org/", "/setup/")(mux)
 }
@@ -258,6 +274,7 @@ type parsedRequest struct {
 	KeyID           string
 	UserID          string
 	ProjectID       string
+	TeamID, OrgID   string
 	Tenant          string
 	ProfileOverride string
 	PromptPrefix    string
@@ -516,6 +533,7 @@ func (r *Router) account(req *parsedRequest, res dispatchResult, status int, too
 		// the truth than charging nothing.
 		u.PromptTokens = int64(req.PromptTokens)
 	}
+	r.m.tenant.observe(req, status, u.PromptTokens, u.CompletionTokens, res.outcome.TTFT)
 	r.acct.Record(req.reservation, u)
 }
 
@@ -534,6 +552,12 @@ func (r *Router) rejectLimit(w http.ResponseWriter, req *parsedRequest, err erro
 	// which is absent when waiting does not help.
 	writeError(w, http.StatusTooManyRequests, "rate_limit_exceeded", le.Error())
 	r.m.observeRequest(r.metricModel(req.Model), "", http.StatusTooManyRequests, time.Since(start), nil, 0)
+	r.m.tenant.observe(req, http.StatusTooManyRequests, 0, 0, 0)
+	r.acct.RecordRefused(store.Usage{
+		At: time.Now(), KeyID: req.KeyID, UserID: req.UserID, ProjectID: req.ProjectID,
+		Model: req.Model, ServedModel: req.Model, Status: http.StatusTooManyRequests,
+		LatencyMs: time.Since(start).Milliseconds(), RequestID: req.RequestID,
+	})
 	r.accessLog(req, dispatchResult{err: err}, http.StatusTooManyRequests, time.Since(start))
 }
 
@@ -576,6 +600,7 @@ func (r *Router) parse(req *http.Request, path string) (*parsedRequest, error) {
 		p.KeyID = principal.KeyID
 		p.UserID = principal.UserID
 		p.ProjectID = principal.ProjectID
+		p.TeamID, p.OrgID = principal.TeamID, principal.OrgID
 		// Tenant fairness within a band is per key rather than per session, so
 		// one key cannot take a whole band by opening many sessions.
 		p.Tenant = principal.KeyID

internal/router/hardware_leader_test.go

@@ -0,0 +1,43 @@
+package router
+
+import (
+	"net/http"
+	"net/http/httptest"
+	"path/filepath"
+	"testing"
+)
+
+func TestTheLeaderEndpointFollowsLeadership(t *testing.T) {
+	rt := newTestRouter(t, baseConfig())
+	get := func() int {
+		rec := httptest.NewRecorder()
+		rt.Handler().ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/leader", nil))
+		return rec.Code
+	}
+	// No rival, nothing to follow: the one process leads.
+	if c := get(); c != http.StatusOK {
+		t.Errorf("with no leadership: %d", c)
+	}
+	leading := false
+	rt.SetLeader(func() bool { return leading })
+	if c := get(); c != http.StatusServiceUnavailable {
+		t.Errorf("a standby: %d", c)
+	}
+	leading = true
+	if c := get(); c != http.StatusOK {
+		t.Errorf("the leader: %d", c)
+	}
+}
+
+func TestTheLeaderEndpointNeedsNoKey(t *testing.T) {
+	// keepalived and load balancers ask it, and they have no key.
+	cfg := baseConfig()
+	cfg.Auth.Enabled = true
+	cfg.Auth.Store = filepath.Join(t.TempDir(), "store.db")
+	rt := newTestRouter(t, cfg)
+	rec := httptest.NewRecorder()
+	rt.Handler().ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/leader", nil))
+	if rec.Code != http.StatusOK {
+		t.Errorf("/leader without a key: %d", rec.Code)
+	}
+}

internal/router/hardware_test.go

@@ -0,0 +1,50 @@
+package router
+
+import (
+	"net/http"
+	"net/http/httptest"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
+)
+
+func TestAReplySaysWhatHardwareAnsweredIt(t *testing.T) {
+	// qwen2.5-coder-14b had one replica on cuda01's card and one on a CPU
+	// node, and the chat page could not tell which had answered: the same
+	// model, ten times faster or slower, for no visible reason.
+	for pool, want := range map[string]string{
+		"none/6000-24g": "6000-24g",
+		"none/cpu":      "cpu",
+		"":              "",
+	} {
+		e := newFakeEngine(t, "m")
+		rt := newTestRouter(t, baseConfig())
+		rt.SetEndpoints(t.Context(), []endpoint.Info{
+			{ID: "r", URL: e.srv.URL, Models: []string{"m"}, Pool: pool, Weight: 1},
+		})
+		waitForEndpoint(t, rt)
+
+		rec := post(t, rt.Handler(), map[string]any{"model": "m", "messages": []any{}}, nil)
+		if rec.Code != http.StatusOK {
+			t.Fatalf("%q: status %d: %s", pool, rec.Code, rec.Body.String())
+		}
+		if got := rec.Header().Get(hardwareHeader); got != want {
+			t.Errorf("pool %q: hardware %q, want %q", pool, got, want)
+		}
+	}
+}
+
+func TestThePickerSaysWhereEachModelRuns(t *testing.T) {
+	gpu, cpu := newFakeEngine(t, "qwen2.5-coder-7b-instruct"), newFakeEngine(t, "qwen2.5-coder-7b-instruct")
+	rt, user := chatRouter(t, gpu.srv.URL)
+	rt.SetEndpoints(t.Context(), []endpoint.Info{
+		{ID: "g", URL: gpu.srv.URL, Models: []string{"qwen2.5-coder-7b-instruct"}, Pool: "none/6000-24g", Weight: 1},
+		{ID: "c", URL: cpu.srv.URL, Models: []string{"coder"}, Pool: "none/cpu", Weight: 1},
+	})
+	waitForEndpoint(t, rt)
+
+	got := rt.modelsFor(httptest.NewRequest("GET", "/chat", nil), user)
+	if len(got) != 1 || got[0].Runs() != "GPU + CPU" {
+		t.Fatalf("picker %+v; want one model running on GPU + CPU, its alias's replica included", got)
+	}
+}

internal/router/metrics.go

@@ -29,6 +29,8 @@ type metrics struct {
 	admissionRejected *prometheus.CounterVec
 
 	prefixDecisions *prometheus.CounterVec
+
+	tenant *tenantMetrics
 }
 
 func newMetrics() *metrics {
@@ -86,16 +88,17 @@ func newMetrics() *metrics {
 			Name: "llmgw_admission_rejected_total",
 			Help: "Requests refused by flow control, by reason.",
 		}, []string{"band", "reason"}),
+		tenant: newTenantMetrics(),
 	}
 }
 
 // collectors returns everything to register, so registration lives in one place.
 func (m *metrics) collectors() []prometheus.Collector {
-	return []prometheus.Collector{
+	return append([]prometheus.Collector{
 		m.requests, m.duration, m.ttft, m.tokens,
 		m.scorerValue, m.filtered, m.considered,
 		m.admissionWait, m.admissionRejected, m.prefixDecisions,
-	}
+	}, m.tenant.collectors()...)
 }
 
 // observePrefix records which branch a prefix decision took.

internal/router/pricing_test.go

@@ -7,6 +7,8 @@ import (
 	"testing"
 	"time"
 
+	"github.com/prometheus/client_golang/prometheus/testutil"
+
 	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
@@ -188,3 +190,75 @@ func TestAnIDThatIsNotOneIsReplaced(t *testing.T) {
 		t.Error("a plain id was not kept")
 	}
 }
+
+func TestARequestALimitRefusesIsCountedAgainstItsOwner(t *testing.T) {
+	// Refused requests were counted only in Prometheus, by model: a team
+	// running into its limit looked, on every usage page, like a team that
+	// had gone quiet.
+	e := newFakeEngine(t, "qwen2.5-14b-instruct")
+	rt, secret := pricedRouter(t, e.srv.URL)
+	rpm := 1
+	if err := rt.store.PutLimits(t.Context(), store.Limits{Scope: store.ScopeKey, ScopeID: "key-1", RPM: &rpm}); err != nil {
+		t.Fatal(err)
+	}
+	body := map[string]any{"model": "q14", "messages": []any{map[string]any{"role": "user", "content": "hi"}}}
+	auth := map[string]string{"Authorization": "Bearer " + secret}
+	codes := []int{post(t, rt.Handler(), body, auth).Code, post(t, rt.Handler(), body, auth).Code}
+	if codes[1] != 429 {
+		t.Fatalf("the second request in the minute answered %v, want 429", codes)
+	}
+	deadline := time.Now().Add(2 * time.Second)
+	for time.Now().Before(deadline) {
+		stats, err := rt.store.StatsBy(t.Context(), store.GroupByProject, time.Now().Add(-time.Hour))
+		if err != nil {
+			t.Fatal(err)
+		}
+		for _, s := range stats {
+			if s.Limited == 1 && s.Requests == 2 {
+				spend, _ := rt.store.SpendSince(t.Context(), time.Now().Add(-time.Hour))
+				if spend["key-1"] == 0 {
+					t.Error("the request that was served cost nothing")
+				}
+				return
+			}
+		}
+		time.Sleep(20 * time.Millisecond)
+	}
+	t.Fatal("the refused request was never counted against its project")
+}
+
+func TestTenantsAreCountedAndTheirBudgetsGauged(t *testing.T) {
+	e := newFakeEngine(t, "qwen2.5-14b-instruct")
+	rt, secret := pricedRouter(t, e.srv.URL)
+	ctx := t.Context()
+	orgs, _ := rt.store.AllOrgs(ctx)
+	teams, _ := rt.store.AllTeams(ctx)
+	budget := int64(1_000_000)
+	if err := rt.store.PutLimits(ctx, store.Limits{Scope: store.ScopeTeam, ScopeID: teams[0].ID,
+		Budget: &budget, BudgetPeriod: store.PeriodMonth}); err != nil {
+		t.Fatal(err)
+	}
+	if err := rt.m.tenant.refresh(ctx, rt.store, time.Now()); err != nil {
+		t.Fatal(err)
+	}
+	rec := post(t, rt.Handler(), map[string]any{"model": "q14", "messages": []any{
+		map[string]any{"role": "user", "content": "hello"}}},
+		map[string]string{"Authorization": "Bearer " + secret})
+	if rec.Code != 200 {
+		t.Fatalf("status %d", rec.Code)
+	}
+	tenant := orgs[0].Name + "/" + teams[0].Name
+	if got := testutil.ToFloat64(rt.m.tenant.requests.WithLabelValues(tenant, "ok")); got != 1 {
+		t.Errorf("%s has %v ok requests, want 1", tenant, got)
+	}
+
+	u := waitForUsage(t, rt)
+	if err := rt.m.tenant.refresh(ctx, rt.store, time.Now()); err != nil {
+		t.Fatal(err)
+	}
+	want := float64(u.CostMicros) / float64(budget)
+	got := testutil.ToFloat64(rt.m.tenant.budget.WithLabelValues("team", teams[0].ID, tenant, "month"))
+	if want == 0 || got != want {
+		t.Errorf("the team's budget gauge reads %v, want %v", got, want)
+	}
+}

internal/router/router.go

@@ -83,6 +83,10 @@ type Router struct {
 	auth     *auth.Authenticator
 	accounts *account.Service
 	store    *storepkg.DB
+	// usageRetention is how long request records are kept; zero is for ever.
+	usageRetention time.Duration
+	// isLeader says whether this process leads; nil is always.
+	isLeader atomic.Pointer[func() bool]
 	acct     *accounting.Accountant
 
 	// threads holds conversations and memory notes. It is store when auth is on,
@@ -189,6 +193,7 @@ func New(cfg config.Server, log *slog.Logger) (*Router, error) {
 	// the decision metric is the whole difference between tuning this and
 	// guessing at it.
 	metrics := newMetrics()
+	metrics.tenant.level = cfg.Router.TenantLabel
 	var prefixScorer *scheduler.PrefixScorer
 	if cfg.Router.Prefix.Enabled {
 		prefixScorer = scheduler.NewPrefixScorer(
@@ -248,6 +253,7 @@ func New(cfg config.Server, log *slog.Logger) (*Router, error) {
 			return nil, fmt.Errorf("auth store: %w", err)
 		}
 		r.store = st
+		r.usageRetention = cfg.Auth.UsageRetention
 		// Applied where no scope set one, so a project created without limits
 		// is still metered. Zero leaves it unlimited, which was the old
 		// behaviour and is now a choice rather than an omission.
@@ -412,6 +418,60 @@ func (r *Router) Start(ctx context.Context) {
 	if r.threads != nil {
 		go r.pruneThreads(ctx)
 	}
+	if r.store != nil && r.usageRetention > 0 {
+		go r.pruneUsage(ctx, r.usageRetention)
+	}
+	if r.store != nil {
+		go r.refreshTenants(ctx)
+	}
+}
+
+// refreshTenants keeps the tenant names the request series are labelled with,
+// and the budget gauges, a minute fresh.
+func (r *Router) refreshTenants(ctx context.Context) {
+	refresh := func() {
+		if err := r.m.tenant.refresh(ctx, r.store, time.Now()); err != nil && ctx.Err() == nil {
+			r.log.Warn("could not refresh tenant names and budget gauges", "err", err)
+		}
+	}
+	refresh()
+	t := time.NewTicker(time.Minute)
+	defer t.Stop()
+	for {
+		select {
+		case <-ctx.Done():
+			return
+		case <-t.C:
+			refresh()
+		}
+	}
+}
+
+// pruneUsage deletes request records older than the retention, hourly and
+// once at startup, as pruneThreads does. The five-minute sums stay, so no
+// total or chart moves; what goes is the trail of each old request.
+func (r *Router) pruneUsage(ctx context.Context, keep time.Duration) {
+	prune := func() {
+		n, err := r.store.PruneUsage(ctx, time.Now().Add(-keep))
+		if err != nil {
+			r.log.Warn("could not prune old request records", "err", err)
+			return
+		}
+		if n > 0 {
+			r.log.Info("removed request records past their retention", "rows", n, "retention", keep)
+		}
+	}
+	prune()
+	t := time.NewTicker(time.Hour)
+	defer t.Stop()
+	for {
+		select {
+		case <-ctx.Done():
+			return
+		case <-t.C:
+			prune()
+		}
+	}
 }
 
 // pruneThreads enforces each project's retention window.
@@ -474,6 +534,9 @@ func (r *Router) Stop() {
 	}
 }
 
+// SetLeader tells /leader how to know whether this process leads.
+func (r *Router) SetLeader(f func() bool) { r.isLeader.Store(&f) }
+
 // Registry exposes the endpoint registry, for tests and admin endpoints.
 func (r *Router) Registry() *endpoint.Registry { return r.reg }
 
@@ -749,6 +812,16 @@ func (r *Router) dispatchOne(ctx context.Context, w http.ResponseWriter, req *pa
 		if a, ok := r.adapter(req.Model); ok && a.base == model {
 			body = withAdapter(body, model, a.index, a.scale)
 		}
+		// What computes the answer, said before the proxy writes the headers.
+		// A model can have replicas on cards and on CPUs at once, and the two
+		// differ tenfold in speed: a reply that came slowly is otherwise
+		// indistinguishable from a model that is slow everywhere. Set again on
+		// a retry, which only happens before anything was written.
+		if hw := hardwareOf(dec.Endpoint.Pool); hw != "" {
+			w.Header().Set(hardwareHeader, hw)
+		} else {
+			w.Header().Del(hardwareHeader)
+		}
 		out, perr := r.px.Do(uctx, w, proxy.Request{
 			URL:                dec.Endpoint.URL + req.Path,
 			APIKey:             dec.Endpoint.APIKey,
@@ -791,6 +864,21 @@ func (r *Router) dispatchOne(ctx context.Context, w http.ResponseWriter, req *pa
 	return last
 }
 
+// hardwareHeader says what the replica that answered computes on: "cpu", or
+// the GPU class, such as "6000-24g".
+const hardwareHeader = "x-llmgw-hardware"
+
+// hardwareOf reads it from a pool, which placement names fabric/class —
+// "none/cpu", "none/6000-24g", "ib-a/h100-80g". Empty for an endpoint
+// configured by hand, which has no pool and says nothing about its hardware.
+func hardwareOf(pool string) string {
+	i := strings.LastIndexByte(pool, '/')
+	if i < 0 || i == len(pool)-1 {
+		return ""
+	}
+	return pool[i+1:]
+}
+
 func without(eps []endpoint.Snapshot, id string) []endpoint.Snapshot {
 	return slices.DeleteFunc(slices.Clone(eps), func(e endpoint.Snapshot) bool { return e.ID == id })
 }

internal/router/tenantmetrics.go

@@ -0,0 +1,182 @@
+package router
+
+import (
+	"context"
+	"strings"
+	"sync"
+	"time"
+
+	"github.com/prometheus/client_golang/prometheus"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/accounting"
+	storepkg "gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// tenantMetrics are the router's series per tenant: requests by how they
+// ended, tokens, time to first token, and how much of each budget is spent.
+//
+// The request series carry one level of the tree, chosen by
+// router.tenant_label, rather than every key or project: a series per project
+// on a cluster of thousands is a Prometheus that falls over, and alerting on a
+// team is what an operator does. Budgets are a series per limit that has one,
+// which is as many as somebody wrote.
+type tenantMetrics struct {
+	level string
+
+	requests *prometheus.CounterVec
+	tokens   *prometheus.CounterVec
+	ttft     *prometheus.HistogramVec
+	budget   *prometheus.GaugeVec
+
+	mu    sync.RWMutex
+	names map[string]string // "team:<id>" -> "acme/platform"
+}
+
+func newTenantMetrics() *tenantMetrics {
+	return &tenantMetrics{
+		level: "team",
+		requests: prometheus.NewCounterVec(prometheus.CounterOpts{
+			Name: "llmgw_tenant_requests_total",
+			Help: "Requests by tenant (the level router.tenant_label names) and outcome: ok, error, or limited — refused by a rate, token or budget limit.",
+		}, []string{"tenant", "outcome"}),
+		tokens: prometheus.NewCounterVec(prometheus.CounterOpts{
+			Name: "llmgw_tenant_tokens_total",
+			Help: "Tokens by tenant and direction.",
+		}, []string{"tenant", "direction"}),
+		ttft: prometheus.NewHistogramVec(prometheus.HistogramOpts{
+			Name:    "llmgw_tenant_ttft_seconds",
+			Help:    "Time to first token by tenant, for streamed requests.",
+			Buckets: []float64{0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60},
+		}, []string{"tenant"}),
+		budget: prometheus.NewGaugeVec(prometheus.GaugeOpts{
+			Name: "llmgw_budget_spent_ratio",
+			Help: "Spend this period over the budget, for every organization, team, project and person given one. 1 is spent.",
+		}, []string{"scope", "id", "name", "period"}),
+		names: map[string]string{},
+	}
+}
+
+func (t *tenantMetrics) collectors() []prometheus.Collector {
+	return []prometheus.Collector{t.requests, t.tokens, t.ttft, t.budget}
+}
+
+// tenantOf is the label a request is counted under: the name of its
+// organization, team or project at the configured level, "personal" for use
+// outside any project, or "" when tenants are not labelled at all.
+func (t *tenantMetrics) tenantOf(req *parsedRequest) string {
+	var scope, id string
+	switch t.level {
+	case "org":
+		scope, id = "org", req.OrgID
+	case "team":
+		scope, id = "team", req.TeamID
+	case "project":
+		scope, id = "project", req.ProjectID
+	default:
+		return ""
+	}
+	if id == "" {
+		return "personal"
+	}
+	t.mu.RLock()
+	name, ok := t.names[scope+":"+id]
+	t.mu.RUnlock()
+	if !ok {
+		// Not yet in the cache — made since the last refresh. The id is
+		// stable, which is what a label has to be; the name follows.
+		return id
+	}
+	return name
+}
+
+// observe counts one finished or refused request.
+func (t *tenantMetrics) observe(req *parsedRequest, status int, prompt, completion int64, ttft time.Duration) {
+	tenant := t.tenantOf(req)
+	if tenant == "" {
+		return
+	}
+	outcome := "ok"
+	switch {
+	case status == 429:
+		outcome = "limited"
+	case status >= 400:
+		outcome = "error"
+	}
+	t.requests.WithLabelValues(tenant, outcome).Inc()
+	if prompt > 0 {
+		t.tokens.WithLabelValues(tenant, "prompt").Add(float64(prompt))
+	}
+	if completion > 0 {
+		t.tokens.WithLabelValues(tenant, "completion").Add(float64(completion))
+	}
+	if ttft > 0 {
+		t.ttft.WithLabelValues(tenant).Observe(ttft.Seconds())
+	}
+}
+
+// tenantStore is what the refresh reads.
+type tenantStore interface {
+	Tree(ctx context.Context) ([]storepkg.TreeOrg, error)
+	AllLimits(ctx context.Context) (map[string]storepkg.Limits, error)
+	SpendOfPool(ctx context.Context, l storepkg.Limits, since time.Time) (int64, error)
+	Users(ctx context.Context, state storepkg.UserState) ([]storepkg.User, error)
+}
+
+// refresh reads the tree's names and every budget's spend. A minute old at
+// most, which is plenty for an alert that waits five before firing.
+func (t *tenantMetrics) refresh(ctx context.Context, s tenantStore, now time.Time) error {
+	tree, err := s.Tree(ctx)
+	if err != nil {
+		return err
+	}
+	names := map[string]string{}
+	for _, o := range tree {
+		names["org:"+o.ID] = o.Name
+		for _, tm := range o.Teams {
+			names["team:"+tm.ID] = o.Name + "/" + tm.Name
+			for _, p := range tm.Projects {
+				names["project:"+p.ID] = o.Name + "/" + tm.Name + "/" + p.Name
+			}
+		}
+	}
+	t.mu.Lock()
+	t.names = names
+	t.mu.Unlock()
+
+	limits, err := s.AllLimits(ctx)
+	if err != nil {
+		return err
+	}
+	emails := map[string]string{}
+	if users, err := s.Users(ctx, ""); err == nil {
+		for _, u := range users {
+			emails[u.ID] = u.Email
+		}
+	}
+	// Rebuilt whole, so a budget removed since the last refresh stops being
+	// published rather than staying at its last value for ever.
+	t.budget.Reset()
+	for _, l := range limits {
+		if l.Budget == nil || *l.Budget <= 0 {
+			continue
+		}
+		name := ""
+		switch l.Scope {
+		case storepkg.ScopeOrg, storepkg.ScopeTeam, storepkg.ScopeProject:
+			name = names[string(l.Scope)+":"+l.ScopeID]
+		case storepkg.ScopeUser:
+			name = emails[l.ScopeID]
+		default:
+			// Per key and per member are too many series and too little
+			// to act on; their pools are bounded by the levels above.
+			continue
+		}
+		spent, err := s.SpendOfPool(ctx, l, accounting.PeriodStart(l.BudgetPeriod, now))
+		if err != nil {
+			return err
+		}
+		t.budget.WithLabelValues(string(l.Scope), l.ScopeID, strings.TrimSpace(name), string(l.BudgetPeriod)).
+			Set(float64(spent) / float64(*l.Budget))
+	}
+	return nil
+}

internal/settings/settings.go

@@ -0,0 +1,513 @@
+// Package settings edits llmgw's configuration files key by key, for the
+// settings screens: what each key is comes from the generated reference, so
+// every setting the code reads is there without anyone listing it, and the
+// file is edited as YAML nodes, so its comments and order survive.
+package settings
+
+import (
+	"bytes"
+	"fmt"
+	"os"
+	"reflect"
+	"strconv"
+	"strings"
+	"time"
+
+	"go.yaml.in/yaml/v3"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/docgen"
+)
+
+// Kind is how a key is shown and edited.
+type Kind int
+
+// The kinds of key.
+const (
+	Text Kind = iota
+	Int
+	Float
+	Bool
+	Duration
+	TextList  // []string, []int: one value a line
+	TextMap   // map[string]string and the like: key = value lines
+	Block     // a nested block: a submenu
+	BlockList // a list of blocks: a submenu of entries
+	BlockMap  // a map of blocks: a submenu of named entries
+)
+
+// Key is one key of a block, as the screens need it.
+type Key struct {
+	docgen.Field
+	Kind Kind
+	// Block is the block it opens, for Block, BlockList and BlockMap.
+	Block *docgen.Block
+}
+
+// Blocks finds a block by ID in the generated reference.
+func Find(id string) (*docgen.Block, bool) {
+	for i := range docgen.Reference {
+		if docgen.Reference[i].ID() == id {
+			return &docgen.Reference[i], true
+		}
+	}
+	return nil, false
+}
+
+// Keys are a block's keys with their kinds.
+func Keys(b *docgen.Block) []Key {
+	out := make([]Key, 0, len(b.Fields))
+	for _, f := range b.Fields {
+		k := Key{Field: f}
+		k.Kind, k.Block = kindOf(b.Package, f)
+		out = append(out, k)
+	}
+	return out
+}
+
+func kindOf(pkg string, f docgen.Field) (Kind, *docgen.Block) {
+	if nb, ok := Find(f.Nested); ok {
+		return Block, nb
+	}
+	t := strings.TrimPrefix(f.Type, "*")
+	switch t {
+	case "string":
+		return Text, nil
+	case "int", "int64", "uint", "uint32":
+		return Int, nil
+	case "float64", "float32":
+		return Float, nil
+	case "bool":
+		return Bool, nil
+	case "time.Duration":
+		return Duration, nil
+	}
+	if elem, ok := strings.CutPrefix(t, "[]"); ok {
+		if b, ok := Find(qualify(pkg, elem)); ok {
+			return BlockList, b
+		}
+		return TextList, nil
+	}
+	if strings.HasPrefix(t, "map[") {
+		elem := t[strings.Index(t, "]")+1:]
+		if b, ok := Find(qualify(pkg, elem)); ok {
+			return BlockMap, b
+		}
+		return TextMap, nil
+	}
+	return Text, nil
+}
+
+// qualify is a type as a block ID: Endpoint in config is config.Endpoint.
+func qualify(pkg, typ string) string {
+	if strings.Contains(typ, ".") {
+		return typ
+	}
+	return pkg + "." + typ
+}
+
+// Root is the block a file is: server.yaml is config.Server, agent.yaml the
+// agent's Config.
+func Root(file string) (*docgen.Block, error) {
+	id := "config.Server"
+	if strings.Contains(file, "agent") {
+		id = "agent.Config"
+	}
+	b, ok := Find(id)
+	if !ok {
+		return nil, fmt.Errorf("no block %s in the reference", id)
+	}
+	return b, nil
+}
+
+// Doc is a configuration file, open for editing.
+type Doc struct {
+	Path string
+	root *yaml.Node // the document's top mapping
+	doc  yaml.Node
+	// Changed counts edits not yet saved.
+	Changed int
+}
+
+// Open reads a file; a missing one is an empty document.
+func Open(path string) (*Doc, error) {
+	d := &Doc{Path: path}
+	b, err := os.ReadFile(path)
+	if err != nil && !os.IsNotExist(err) {
+		return nil, err
+	}
+	if len(bytes.TrimSpace(b)) > 0 {
+		if err := yaml.Unmarshal(b, &d.doc); err != nil {
+			return nil, fmt.Errorf("%s: %w", path, err)
+		}
+	}
+	if d.doc.Kind == 0 {
+		d.doc = yaml.Node{Kind: yaml.DocumentNode, Content: []*yaml.Node{{Kind: yaml.MappingNode, Tag: "!!map"}}}
+	}
+	d.root = d.doc.Content[0]
+	if d.root.Kind != yaml.MappingNode {
+		return nil, fmt.Errorf("%s is not a mapping at the top", path)
+	}
+	return d, nil
+}
+
+// Bytes is the document as it would be written.
+func (d *Doc) Bytes() ([]byte, error) {
+	var b bytes.Buffer
+	enc := yaml.NewEncoder(&b)
+	enc.SetIndent(2)
+	if err := enc.Encode(&d.doc); err != nil {
+		return nil, err
+	}
+	return b.Bytes(), enc.Close()
+}
+
+// Node is the node at a path of keys and list indexes, or nil.
+func (d *Doc) Node(path []string) *yaml.Node {
+	n := d.root
+	for _, p := range path {
+		n = child(n, p)
+		if n == nil {
+			return nil
+		}
+	}
+	return n
+}
+
+func child(n *yaml.Node, key string) *yaml.Node {
+	switch n.Kind {
+	case yaml.MappingNode:
+		for i := 0; i+1 < len(n.Content); i += 2 {
+			if n.Content[i].Value == key {
+				return n.Content[i+1]
+			}
+		}
+	case yaml.SequenceNode:
+		if i, err := strconv.Atoi(key); err == nil && i >= 0 && i < len(n.Content) {
+			return n.Content[i]
+		}
+	}
+	return nil
+}
+
+// ensure is the node at a path, made as mappings where there is none.
+func (d *Doc) ensure(path []string) *yaml.Node {
+	n := d.root
+	for _, p := range path {
+		c := child(n, p)
+		if c == nil {
+			if n.Kind != yaml.MappingNode {
+				return nil
+			}
+			c = &yaml.Node{Kind: yaml.MappingNode, Tag: "!!map"}
+			n.Content = append(n.Content, &yaml.Node{Kind: yaml.ScalarNode, Tag: "!!str", Value: p}, c)
+		}
+		n = c
+	}
+	return n
+}
+
+// Set writes one key's value: a scalar, a list of lines or a map of
+// "key = value" lines, by its kind. It checks the value first.
+func (d *Doc) Set(path []string, k Kind, value string) error {
+	if len(path) == 0 {
+		return fmt.Errorf("no key")
+	}
+	n, err := encode(k, value)
+	if err != nil {
+		return err
+	}
+	parent := d.ensure(path[:len(path)-1])
+	if parent == nil {
+		return fmt.Errorf("%s is not a block", strings.Join(path[:len(path)-1], "."))
+	}
+	key := path[len(path)-1]
+	if parent.Kind == yaml.SequenceNode {
+		i, _ := strconv.Atoi(key)
+		parent.Content[i] = n
+	} else if c := child(parent, key); c != nil {
+		// Kept in place, with whatever comment it had.
+		n.HeadComment, n.LineComment, n.FootComment = c.HeadComment, c.LineComment, c.FootComment
+		*c = *n
+	} else {
+		parent.Content = append(parent.Content, &yaml.Node{Kind: yaml.ScalarNode, Tag: "!!str", Value: key}, n)
+	}
+	d.Changed++
+	return nil
+}
+
+// Unset removes a key, so its default applies.
+func (d *Doc) Unset(path []string) {
+	if len(path) == 0 {
+		return
+	}
+	parent := d.Node(path[:len(path)-1])
+	if parent == nil {
+		return
+	}
+	key := path[len(path)-1]
+	switch parent.Kind {
+	case yaml.MappingNode:
+		for i := 0; i+1 < len(parent.Content); i += 2 {
+			if parent.Content[i].Value == key {
+				parent.Content = append(parent.Content[:i], parent.Content[i+2:]...)
+				d.Changed++
+				return
+			}
+		}
+	case yaml.SequenceNode:
+		if i, err := strconv.Atoi(key); err == nil && i < len(parent.Content) {
+			parent.Content = append(parent.Content[:i], parent.Content[i+1:]...)
+			d.Changed++
+		}
+	}
+}
+
+// Append adds an empty block to the list at path, and returns its index.
+func (d *Doc) Append(path []string) int {
+	parent := d.ensure(path[:len(path)-1])
+	key := path[len(path)-1]
+	seq := child(parent, key)
+	if seq == nil || seq.Kind != yaml.SequenceNode {
+		seq = &yaml.Node{Kind: yaml.SequenceNode, Tag: "!!seq"}
+		if c := child(parent, key); c != nil {
+			*c = *seq
+			seq = c
+		} else {
+			parent.Content = append(parent.Content, &yaml.Node{Kind: yaml.ScalarNode, Tag: "!!str", Value: key}, seq)
+		}
+	}
+	seq.Content = append(seq.Content, &yaml.Node{Kind: yaml.MappingNode, Tag: "!!map"})
+	d.Changed++
+	return len(seq.Content) - 1
+}
+
+// AddEntry adds an empty block under a name to the map at path.
+func (d *Doc) AddEntry(path []string, name string) error {
+	m := d.ensure(path)
+	if m == nil || m.Kind != yaml.MappingNode {
+		return fmt.Errorf("%s is not a map", strings.Join(path, "."))
+	}
+	if child(m, name) != nil {
+		return fmt.Errorf("%s is there already", name)
+	}
+	m.Content = append(m.Content, &yaml.Node{Kind: yaml.ScalarNode, Tag: "!!str", Value: name},
+		&yaml.Node{Kind: yaml.MappingNode, Tag: "!!map"})
+	d.Changed++
+	return nil
+}
+
+// Len is how many entries a list or map at path has.
+func (d *Doc) Len(path []string) int {
+	n := d.Node(path)
+	switch {
+	case n == nil:
+		return 0
+	case n.Kind == yaml.SequenceNode:
+		return len(n.Content)
+	case n.Kind == yaml.MappingNode:
+		return len(n.Content) / 2
+	}
+	return 0
+}
+
+// MapKeys are a map's keys, in the file's order.
+func (d *Doc) MapKeys(path []string) []string {
+	n := d.Node(path)
+	if n == nil || n.Kind != yaml.MappingNode {
+		return nil
+	}
+	var out []string
+	for i := 0; i+1 < len(n.Content); i += 2 {
+		out = append(out, n.Content[i].Value)
+	}
+	return out
+}
+
+// Text is a key's value as it is edited: a scalar as written, a list one
+// item a line, a map "key = value" a line. ok is false for a key the file
+// does not set.
+func (d *Doc) Text(path []string) (string, bool) {
+	n := d.Node(path)
+	if n == nil {
+		return "", false
+	}
+	switch n.Kind {
+	case yaml.ScalarNode:
+		return n.Value, true
+	case yaml.SequenceNode:
+		var l []string
+		for _, c := range n.Content {
+			l = append(l, c.Value)
+		}
+		return strings.Join(l, "\n"), true
+	case yaml.MappingNode:
+		var l []string
+		for i := 0; i+1 < len(n.Content); i += 2 {
+			l = append(l, n.Content[i].Value+" = "+n.Content[i+1].Value)
+		}
+		return strings.Join(l, "\n"), true
+	}
+	return "", true
+}
+
+func encode(k Kind, value string) (*yaml.Node, error) {
+	v := strings.TrimSpace(value)
+	scalar := func(tag string) *yaml.Node { return &yaml.Node{Kind: yaml.ScalarNode, Tag: tag, Value: v} }
+	switch k {
+	case Int:
+		if _, err := strconv.ParseInt(v, 10, 64); err != nil {
+			return nil, fmt.Errorf("%q is not a whole number", v)
+		}
+		return scalar("!!int"), nil
+	case Float:
+		if _, err := strconv.ParseFloat(v, 64); err != nil {
+			return nil, fmt.Errorf("%q is not a number", v)
+		}
+		return scalar("!!float"), nil
+	case Bool:
+		b, err := strconv.ParseBool(v)
+		if err != nil {
+			return nil, fmt.Errorf("%q is not true or false", v)
+		}
+		return &yaml.Node{Kind: yaml.ScalarNode, Tag: "!!bool", Value: strconv.FormatBool(b)}, nil
+	case Duration:
+		if _, err := time.ParseDuration(v); err != nil {
+			return nil, fmt.Errorf("%q is not a duration, like 30s, 5m or 2h", v)
+		}
+		return scalar("!!str"), nil
+	case TextList:
+		seq := &yaml.Node{Kind: yaml.SequenceNode, Tag: "!!seq"}
+		for _, l := range splitLines(value) {
+			seq.Content = append(seq.Content, &yaml.Node{Kind: yaml.ScalarNode, Tag: "!!str", Value: l})
+		}
+		return seq, nil
+	case TextMap:
+		m := &yaml.Node{Kind: yaml.MappingNode, Tag: "!!map"}
+		for _, l := range splitLines(value) {
+			k, val, ok := strings.Cut(l, "=")
+			if !ok {
+				return nil, fmt.Errorf("%q is not key = value", l)
+			}
+			m.Content = append(m.Content,
+				&yaml.Node{Kind: yaml.ScalarNode, Tag: "!!str", Value: strings.TrimSpace(k)},
+				&yaml.Node{Kind: yaml.ScalarNode, Tag: "!!str", Value: strings.TrimSpace(val)})
+		}
+		return m, nil
+	case Block, BlockList, BlockMap:
+		return nil, fmt.Errorf("a block is edited key by key")
+	}
+	return scalar("!!str"), nil
+}
+
+func splitLines(s string) []string {
+	var out []string
+	for _, l := range strings.FieldsFunc(s, func(r rune) bool { return r == '\n' || r == ',' }) {
+		if l = strings.TrimSpace(l); l != "" {
+			out = append(out, l)
+		}
+	}
+	return out
+}
+
+// Save checks the document with check — a loader that reads and validates
+// a file — and writes it, the old file kept beside it. Nothing is written
+// when the check fails.
+func (d *Doc) Save(check func(path string) error) (string, error) {
+	b, err := d.Bytes()
+	if err != nil {
+		return "", err
+	}
+	tmp := d.Path + ".check"
+	if err := os.WriteFile(tmp, b, 0o600); err != nil {
+		return "", err
+	}
+	defer os.Remove(tmp)
+	if check != nil {
+		if err := check(tmp); err != nil {
+			return "", err
+		}
+	}
+	mode := os.FileMode(0o644)
+	backup := ""
+	if info, err := os.Stat(d.Path); err == nil {
+		mode = info.Mode().Perm()
+		old, err := os.ReadFile(d.Path)
+		if err != nil {
+			return "", err
+		}
+		backup = fmt.Sprintf("%s.bak-%s", d.Path, time.Now().Format("20060102-150405"))
+		if err := os.WriteFile(backup, old, mode); err != nil {
+			return "", err
+		}
+	}
+	if err := os.WriteFile(d.Path+".new", b, mode); err != nil {
+		return "", err
+	}
+	if err := os.Rename(d.Path+".new", d.Path); err != nil {
+		return "", err
+	}
+	d.Changed = 0
+	return backup, nil
+}
+
+// Effective is the value a loaded configuration has at a path of koanf
+// keys — what applies, default included — as text.
+func Effective(cfg any, path []string) (string, bool) {
+	v := reflect.ValueOf(cfg)
+	for _, p := range path {
+		for v.Kind() == reflect.Pointer {
+			if v.IsNil() {
+				return "", false
+			}
+			v = v.Elem()
+		}
+		switch v.Kind() {
+		case reflect.Struct:
+			found := false
+			t := v.Type()
+			for i := 0; i < t.NumField(); i++ {
+				if tag, _, _ := strings.Cut(t.Field(i).Tag.Get("koanf"), ","); tag == p {
+					v, found = v.Field(i), true
+					break
+				}
+			}
+			if !found {
+				return "", false
+			}
+		case reflect.Slice:
+			i, err := strconv.Atoi(p)
+			if err != nil || i >= v.Len() {
+				return "", false
+			}
+			v = v.Index(i)
+		case reflect.Map:
+			mv := v.MapIndex(reflect.ValueOf(p))
+			if !mv.IsValid() {
+				return "", false
+			}
+			v = mv
+		default:
+			return "", false
+		}
+	}
+	for v.Kind() == reflect.Pointer {
+		if v.IsNil() {
+			return "", true
+		}
+		v = v.Elem()
+	}
+	switch v.Kind() {
+	case reflect.Slice:
+		var l []string
+		for i := 0; i < v.Len(); i++ {
+			l = append(l, fmt.Sprint(v.Index(i).Interface()))
+		}
+		return strings.Join(l, ", "), true
+	case reflect.Struct, reflect.Map:
+		return "", true
+	}
+	if d, ok := v.Interface().(time.Duration); ok {
+		return d.String(), true
+	}
+	return fmt.Sprint(v.Interface()), true
+}

internal/settings/settings_test.go

@@ -0,0 +1,122 @@
+package settings
+
+import (
+	"os"
+	"path/filepath"
+	"strconv"
+	"strings"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/docgen"
+)
+
+func TestEveryKeyOfBothFilesCanBeReached(t *testing.T) {
+	// Walked from each file's root: every block a key opens is found, so no
+	// setting is a dead end on the screens.
+	for _, file := range []string{"/etc/llmgw/server.yaml", "/etc/llmgw/agent.yaml"} {
+		root, err := Root(file)
+		if err != nil {
+			t.Fatal(err)
+		}
+		seen := map[string]bool{}
+		var walk func(b *docgen.Block, path string)
+		count := 0
+		walk = func(b *docgen.Block, path string) {
+			if seen[b.ID()] {
+				return
+			}
+			seen[b.ID()] = true
+			for _, k := range Keys(b) {
+				count++
+				if k.Nested != "" && k.Block == nil {
+					t.Errorf("%s: %s.%s opens %s, which is not in the reference", file, path, k.Key, k.Nested)
+				}
+				if k.Block != nil {
+					walk(k.Block, path+"."+k.Key)
+				}
+			}
+		}
+		walk(root, "")
+		if count < 20 {
+			t.Errorf("%s: only %d keys reached", file, count)
+		}
+	}
+	// The TLS settings, which the reference used to lose to a name clash.
+	root, _ := Root("server.yaml")
+	for _, k := range Keys(root) {
+		if k.Key == "agent_tls" && (k.Kind != Block || k.Block.ID() != "tlsconf.Config") {
+			t.Errorf("agent_tls is %v %v", k.Kind, k.Block)
+		}
+		if k.Key == "endpoints" && k.Kind != BlockList {
+			t.Errorf("endpoints is %v", k.Kind)
+		}
+	}
+}
+
+const sample = `# The control plane.
+listen: ":8080"
+router:
+  # How many other endpoints to try.
+  retries: 2
+`
+
+func TestAnEditKeepsTheFilesCommentsAndIsCheckedBeforeItIsWritten(t *testing.T) {
+	dir := t.TempDir()
+	path := filepath.Join(dir, "server.yaml")
+	if err := os.WriteFile(path, []byte(sample), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	d, err := Open(path)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if err := d.Set([]string{"router", "retries"}, Int, "three"); err == nil {
+		t.Error("a word was taken for a number")
+	}
+	if err := d.Set([]string{"router", "retries"}, Int, "3"); err != nil {
+		t.Fatal(err)
+	}
+	if err := d.Set([]string{"router", "cooldown"}, Duration, "45s"); err != nil {
+		t.Fatal(err)
+	}
+	check := func(p string) error { _, err := config.Load(p); return err }
+	if _, err := d.Save(check); err != nil {
+		t.Fatalf("a good edit was refused: %v", err)
+	}
+	b, _ := os.ReadFile(path)
+	for _, want := range []string{"# The control plane.", "# How many other endpoints to try.", "retries: 3", "cooldown: 45s"} {
+		if !strings.Contains(string(b), want) {
+			t.Errorf("the file lacks %q:\n%s", want, b)
+		}
+	}
+
+	// Two endpoints with one id: the loader refuses, and nothing is written.
+	for range 2 {
+		i := d.Append([]string{"endpoints"})
+		for _, kv := range [][2]string{{"id", "a"}, {"url", "http://x"}} {
+			if err := d.Set([]string{"endpoints", itoa(i), kv[0]}, Text, kv[1]); err != nil {
+				t.Fatal(err)
+			}
+		}
+		_ = d.Set([]string{"endpoints", itoa(i), "models"}, TextList, "m")
+	}
+	if _, err := d.Save(check); err == nil {
+		t.Error("an invalid configuration was saved")
+	}
+	if b2, _ := os.ReadFile(path); string(b2) != string(b) {
+		t.Error("a refused save changed the file")
+	}
+}
+
+func TestTheValueThatAppliesIsShownWhenTheFileSetsNone(t *testing.T) {
+	cfg := config.Default()
+	if v, ok := Effective(cfg, []string{"router", "retries"}); !ok || v != "2" {
+		t.Errorf("router.retries: %q %v", v, ok)
+	}
+	if v, ok := Effective(cfg, []string{"router", "cooldown"}); !ok || v != "15s" {
+		t.Errorf("router.cooldown: %q", v)
+	}
+}
+
+func itoa(i int) string { return strconv.Itoa(i) }

internal/store/controlplanes.go

@@ -0,0 +1,44 @@
+package store
+
+import (
+	"context"
+	"database/sql"
+	"time"
+)
+
+// ControlPlane is one control plane agents may dial.
+type ControlPlane struct {
+	// URL is what an agent dials: https://10.0.0.150:7443.
+	URL     string
+	Name    string
+	AddedAt time.Time
+}
+
+// ControlPlanes lists them, oldest first: the first is the one agents try
+// first.
+func (s *DB) ControlPlanes(ctx context.Context) ([]ControlPlane, error) {
+	return queryAll(ctx, s.db, `SELECT url, name, added_at FROM control_planes ORDER BY added_at, url`, nil,
+		func(rows *sql.Rows) (ControlPlane, error) {
+			var c ControlPlane
+			var at int64
+			err := rows.Scan(&c.URL, &c.Name, &at)
+			c.AddedAt = fromMillis(at)
+			return c, err
+		})
+}
+
+// PutControlPlane records one, keeping when it was first added.
+func (s *DB) PutControlPlane(ctx context.Context, c ControlPlane) error {
+	if c.AddedAt.IsZero() {
+		c.AddedAt = time.Now()
+	}
+	_, err := s.db.ExecContext(ctx, `INSERT INTO control_planes (url, name, added_at) VALUES (?, ?, ?)
+		ON CONFLICT (url) DO UPDATE SET name = excluded.name`, c.URL, c.Name, millis(c.AddedAt))
+	return err
+}
+
+// RemoveControlPlane forgets one.
+func (s *DB) RemoveControlPlane(ctx context.Context, url string) error {
+	_, err := s.db.ExecContext(ctx, `DELETE FROM control_planes WHERE url = ?`, url)
+	return err
+}

internal/store/copy.go

@@ -33,6 +33,11 @@ var copyOrder = []string{
 	// quota on a scope that is later removed is inert, not an error.
 	"capacity_quotas",
 	"usage",
+	// Copied rather than rebuilt from the rows: once request rows have a
+	// retention, the sums are the only record of what is older.
+	"usage_5m",
+	"usage_1h",
+	"usage_1d",
 	"models",
 	// No foreign key to models, deliberately: a measurement outlives the
 	// catalog entry it was taken for, and losing it when somebody removes a
@@ -46,6 +51,7 @@ var copyOrder = []string{
 	// of them at the moment of the move.
 	"cluster_secrets",
 	"node_engines",
+	"control_planes",
 }
 
 // notCopied are tables deliberately left behind.

internal/store/latency.go

@@ -0,0 +1,132 @@
+package store
+
+import (
+	"context"
+	"fmt"
+	"strings"
+	"time"
+)
+
+// Quantiles are a duration's percentiles, in milliseconds, read from the
+// histogram kept with the usage sums. Within a bucket the value is
+// interpolated; above the last bucket's edge it is that edge, and Over says
+// so. N is how many durations they are read from — zero is no reading.
+type Quantiles struct {
+	P50, P95, P99 int64
+	N             int64
+	Over          bool
+}
+
+// GroupStats is one group's requests, how they ended, and how long they took.
+type GroupStats struct {
+	ID       string
+	Requests int64
+	Errors   int64
+	// Limited is requests refused by a rate, token or budget limit.
+	Limited int64
+	Latency Quantiles
+	// TTFT is time to first token, for requests that streamed one.
+	TTFT Quantiles
+}
+
+// ErrorRate is the share of requests that failed, limits included, in percent.
+func (g GroupStats) ErrorRate() float64 {
+	if g.Requests == 0 {
+		return 0
+	}
+	return 100 * float64(g.Errors) / float64(g.Requests)
+}
+
+// StatsBy is each group's request outcomes and latency percentiles since a
+// moment, keyed by the group's id.
+//
+// Read from the hourly sums for a window of two days or more and from the
+// five-minute ones otherwise, starting at the slice the moment falls in:
+// percentiles are a shape, and a window starting up to one slice early does
+// not change one. Budgets, which must be exact, are not read this way.
+func (s *DB) StatsBy(ctx context.Context, g Grouping, since time.Time) (map[string]GroupStats, error) {
+	c, ok := groupColumns[g]
+	if !ok {
+		return nil, fmt.Errorf("usage cannot be grouped by %q", g)
+	}
+	t := tier5m
+	if time.Since(since) >= 48*time.Hour {
+		t = tier1h
+	}
+	n := len(LatencyBounds) + 1
+	var sums []string
+	for i := range n {
+		sums = append(sums, fmt.Sprintf("SUM(u.lat_%d)", i))
+	}
+	for i := range n {
+		sums = append(sums, fmt.Sprintf("SUM(u.ttft_%d)", i))
+	}
+	rows, err := s.db.QueryContext(ctx, `
+		SELECT `+c.id+`, SUM(u.requests), SUM(u.errors), SUM(u.limited), `+strings.Join(sums, ", ")+`
+		FROM `+t.table+` u`+c.joins+`
+		WHERE u.bucket >= ?
+		GROUP BY 1`, t.floor(since))
+	if err != nil {
+		return nil, err
+	}
+	defer rows.Close()
+
+	out := map[string]GroupStats{}
+	for rows.Next() {
+		var gs GroupStats
+		lat, ttft := make([]int64, n), make([]int64, n)
+		dst := []any{&gs.ID, &gs.Requests, &gs.Errors, &gs.Limited}
+		for i := range lat {
+			dst = append(dst, &lat[i])
+		}
+		for i := range ttft {
+			dst = append(dst, &ttft[i])
+		}
+		if err := rows.Scan(dst...); err != nil {
+			return nil, err
+		}
+		gs.Latency, gs.TTFT = quantiles(lat), quantiles(ttft)
+		out[gs.ID] = gs
+	}
+	return out, rows.Err()
+}
+
+// quantiles reads p50, p95 and p99 from counts per LatencyBounds bucket.
+func quantiles(counts []int64) Quantiles {
+	var q Quantiles
+	for _, c := range counts {
+		q.N += c
+	}
+	if q.N == 0 {
+		return q
+	}
+	at := func(p float64) (int64, bool) {
+		rank := p * float64(q.N)
+		var seen float64
+		for i, c := range counts {
+			if c == 0 {
+				continue
+			}
+			if seen+float64(c) >= rank {
+				if i >= len(LatencyBounds) {
+					return LatencyBounds[len(LatencyBounds)-1], true
+				}
+				lo := int64(0)
+				if i > 0 {
+					lo = LatencyBounds[i-1]
+				}
+				hi := LatencyBounds[i]
+				frac := (rank - seen) / float64(c)
+				return lo + int64(frac*float64(hi-lo)), false
+			}
+			seen += float64(c)
+		}
+		return LatencyBounds[len(LatencyBounds)-1], true
+	}
+	var over bool
+	q.P50, _ = at(0.50)
+	q.P95, _ = at(0.95)
+	q.P99, over = at(0.99)
+	q.Over = over
+	return q
+}

internal/store/latency_test.go

@@ -0,0 +1,61 @@
+package store
+
+import (
+	"context"
+	"testing"
+	"time"
+)
+
+func TestQuantilesAreReadFromTheHistogram(t *testing.T) {
+	// 100 durations: 90 at <=100 ms, 8 in (1000, 2500], 2 above two minutes.
+	counts := make([]int64, len(LatencyBounds)+1)
+	counts[0], counts[4], counts[len(counts)-1] = 90, 8, 2
+	q := quantiles(counts)
+	if q.N != 100 || q.P50 > 100 || q.P95 <= 1000 || q.P95 > 2500 || !q.Over {
+		t.Errorf("%+v: want p50 within 100 ms, p95 in (1000, 2500], p99 past the last edge", q)
+	}
+	if (quantiles(make([]int64, len(LatencyBounds)+1)) != Quantiles{}) {
+		t.Error("no durations read as some")
+	}
+}
+
+func TestStatsSayHowEachGroupFaredAndHowLongItTook(t *testing.T) {
+	onBoth(t, func(t *testing.T, s *DB) {
+		ctx := context.Background()
+		at := slice()
+		var batch []Usage
+		for i := range 20 {
+			batch = append(batch, Usage{At: at, KeyID: "k", UserID: "ayse", ProjectID: "p", Model: "big",
+				ServedModel: "big", Status: 200, LatencyMs: 4000, TTFTMs: int64(200 + i)})
+		}
+		batch = append(batch,
+			Usage{At: at, KeyID: "k", UserID: "ayse", ProjectID: "p", Model: "big", ServedModel: "big", Status: 502, LatencyMs: 50},
+			Usage{At: at, KeyID: "k2", UserID: "bora", ProjectID: "p", Model: "small", ServedModel: "small", Status: 429},
+		)
+		if err := s.WriteUsage(ctx, batch); err != nil {
+			t.Fatal(err)
+		}
+		byUser, err := s.StatsBy(ctx, GroupByUser, at.Add(-time.Hour))
+		if err != nil {
+			t.Fatal(err)
+		}
+		a, b := byUser["ayse"], byUser["bora"]
+		if a.Requests != 21 || a.Errors != 1 || a.Limited != 0 || b.Limited != 1 || b.Errors != 1 {
+			t.Errorf("outcomes: ayse %+v, bora %+v", a, b)
+		}
+		if a.TTFT.N != 20 || a.TTFT.P95 <= 100 || a.TTFT.P95 > 250 || a.Latency.P95 <= 2500 || a.Latency.P95 > 5000 {
+			t.Errorf("ayse's first token %+v and latency %+v", a.TTFT, a.Latency)
+		}
+		byModel, err := s.StatsBy(ctx, GroupByModel, at.Add(-time.Hour))
+		if err != nil {
+			t.Fatal(err)
+		}
+		if byModel["big"].Requests != 21 || byModel["small"].Requests != 1 {
+			t.Errorf("by model: %+v", byModel)
+		}
+		series, err := s.UsageBucketsBy(ctx, GroupByModel, at.Add(-time.Hour), time.Hour)
+		if err != nil || len(series) != 2 {
+			t.Fatalf("usage over time by model: %v, %+v", err, series)
+		}
+	})
+}

internal/store/migrations/postgres/0015_usage_rollup.sql

@@ -0,0 +1,208 @@
+-- Ported from the SQLite migration of the same number.
+-- Usage summed as it is written, per key, person, project and model: into
+-- five-minute slices (usage_5m), hours (usage_1h) and days (usage_1d).
+--
+-- Every chart and total read the raw table, one row per request, and summed it
+-- again on every page view: fine for thousands of requests a day, and a full
+-- scan of tens of millions on a cluster in real use. The sums are kept in the
+-- transaction that writes the rows, so they never disagree with them; the raw
+-- table stays for following one request, and can be given a retention of its
+-- own (auth.usage_retention).
+--
+-- Three widths because how well a slice compresses depends on traffic: many
+-- keys each sending a little leave a five-minute slice with nearly a row per
+-- request, which an hour or a day folds much further. A chart reads the
+-- coarsest width its slices are made of; a total since a moment reads the
+-- rows up to the next five-minute slice, those slices up to the hour, hours
+-- up to the day and days from there — exact, and a month in about thirty
+-- rows per key and model.
+--
+-- Latency and time to first token are kept as histograms, a count per fixed
+-- bucket (ms: <=100, 250, 500, 1000, 2500, 5000, 10000, 30000, 60000, 120000,
+-- and above), which is what a percentile per team or model is read from after
+-- the rows are gone; a mean cannot do that. ttft_* counts only requests whose
+-- first token was measured. limited counts requests refused with 429 — a rate,
+-- token or budget limit — which are recorded with no tokens and no cost.
+
+CREATE TABLE usage_5m (
+  bucket            BIGINT NOT NULL,
+  key_id            TEXT NOT NULL,
+  user_id           TEXT NOT NULL,
+  project_id        TEXT NOT NULL,
+  model             TEXT NOT NULL,
+  served_model      TEXT NOT NULL,
+  requests          BIGINT NOT NULL DEFAULT 0,
+  errors            BIGINT NOT NULL DEFAULT 0,
+  limited           BIGINT NOT NULL DEFAULT 0,
+  prompt_tokens     BIGINT NOT NULL DEFAULT 0,
+  completion_tokens BIGINT NOT NULL DEFAULT 0,
+  cost_micros       BIGINT NOT NULL DEFAULT 0,
+  latency_ms_sum    BIGINT NOT NULL DEFAULT 0,
+  queue_ms_sum      BIGINT NOT NULL DEFAULT 0,
+  ttft_ms_sum       BIGINT NOT NULL DEFAULT 0,
+  ttft_n            BIGINT NOT NULL DEFAULT 0,
+  last_ts           BIGINT NOT NULL DEFAULT 0,
+  lat_0 BIGINT NOT NULL DEFAULT 0,
+  lat_1 BIGINT NOT NULL DEFAULT 0,
+  lat_2 BIGINT NOT NULL DEFAULT 0,
+  lat_3 BIGINT NOT NULL DEFAULT 0,
+  lat_4 BIGINT NOT NULL DEFAULT 0,
+  lat_5 BIGINT NOT NULL DEFAULT 0,
+  lat_6 BIGINT NOT NULL DEFAULT 0,
+  lat_7 BIGINT NOT NULL DEFAULT 0,
+  lat_8 BIGINT NOT NULL DEFAULT 0,
+  lat_9 BIGINT NOT NULL DEFAULT 0,
+  lat_10 BIGINT NOT NULL DEFAULT 0,
+  ttft_0 BIGINT NOT NULL DEFAULT 0,
+  ttft_1 BIGINT NOT NULL DEFAULT 0,
+  ttft_2 BIGINT NOT NULL DEFAULT 0,
+  ttft_3 BIGINT NOT NULL DEFAULT 0,
+  ttft_4 BIGINT NOT NULL DEFAULT 0,
+  ttft_5 BIGINT NOT NULL DEFAULT 0,
+  ttft_6 BIGINT NOT NULL DEFAULT 0,
+  ttft_7 BIGINT NOT NULL DEFAULT 0,
+  ttft_8 BIGINT NOT NULL DEFAULT 0,
+  ttft_9 BIGINT NOT NULL DEFAULT 0,
+  ttft_10 BIGINT NOT NULL DEFAULT 0,
+  PRIMARY KEY (bucket, key_id, user_id, project_id, model, served_model)
+);
+CREATE INDEX usage_5m_by_bucket ON usage_5m(bucket);
+CREATE INDEX usage_5m_by_project ON usage_5m(project_id, bucket);
+CREATE INDEX usage_5m_by_user ON usage_5m(user_id, bucket);
+CREATE INDEX usage_5m_by_key ON usage_5m(key_id, bucket);
+
+INSERT INTO usage_5m (bucket, key_id, user_id, project_id, model, served_model,
+  requests, errors, limited, prompt_tokens, completion_tokens, cost_micros,
+  latency_ms_sum, queue_ms_sum, ttft_ms_sum, ttft_n, last_ts,
+  lat_0, lat_1, lat_2, lat_3, lat_4, lat_5, lat_6, lat_7, lat_8, lat_9, lat_10,
+  ttft_0, ttft_1, ttft_2, ttft_3, ttft_4, ttft_5, ttft_6, ttft_7, ttft_8, ttft_9, ttft_10)
+SELECT (ts / 300000) * 300000, key_id, COALESCE(user_id, ''), project_id, model, served_model,
+  COUNT(*), SUM(CASE WHEN status >= 400 THEN 1 ELSE 0 END), SUM(CASE WHEN status = 429 THEN 1 ELSE 0 END),
+  SUM(prompt_tokens), SUM(completion_tokens), SUM(cost_micros),
+  SUM(latency_ms), SUM(queue_ms), SUM(ttft_ms), SUM(CASE WHEN ttft_ms > 0 THEN 1 ELSE 0 END), MAX(ts),
+  SUM(CASE WHEN latency_ms <= 100 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 100 AND latency_ms <= 250 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 250 AND latency_ms <= 500 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 500 AND latency_ms <= 1000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 1000 AND latency_ms <= 2500 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 2500 AND latency_ms <= 5000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 5000 AND latency_ms <= 10000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 10000 AND latency_ms <= 30000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 30000 AND latency_ms <= 60000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 60000 AND latency_ms <= 120000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 120000 THEN 1 ELSE 0 END),
+  SUM(CASE WHEN ttft_ms > 0 AND ttft_ms <= 100 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 100 AND ttft_ms <= 250 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 250 AND ttft_ms <= 500 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 500 AND ttft_ms <= 1000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 1000 AND ttft_ms <= 2500 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 2500 AND ttft_ms <= 5000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 5000 AND ttft_ms <= 10000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 10000 AND ttft_ms <= 30000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 30000 AND ttft_ms <= 60000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 60000 AND ttft_ms <= 120000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 120000 THEN 1 ELSE 0 END)
+FROM usage
+GROUP BY 1, 2, 3, 4, 5, 6;
+
+CREATE TABLE usage_1h (
+  bucket            BIGINT NOT NULL,
+  key_id            TEXT NOT NULL,
+  user_id           TEXT NOT NULL,
+  project_id        TEXT NOT NULL,
+  model             TEXT NOT NULL,
+  served_model      TEXT NOT NULL,
+  requests          BIGINT NOT NULL DEFAULT 0,
+  errors            BIGINT NOT NULL DEFAULT 0,
+  limited           BIGINT NOT NULL DEFAULT 0,
+  prompt_tokens     BIGINT NOT NULL DEFAULT 0,
+  completion_tokens BIGINT NOT NULL DEFAULT 0,
+  cost_micros       BIGINT NOT NULL DEFAULT 0,
+  latency_ms_sum    BIGINT NOT NULL DEFAULT 0,
+  queue_ms_sum      BIGINT NOT NULL DEFAULT 0,
+  ttft_ms_sum       BIGINT NOT NULL DEFAULT 0,
+  ttft_n            BIGINT NOT NULL DEFAULT 0,
+  last_ts           BIGINT NOT NULL DEFAULT 0,
+  lat_0 BIGINT NOT NULL DEFAULT 0,
+  lat_1 BIGINT NOT NULL DEFAULT 0,
+  lat_2 BIGINT NOT NULL DEFAULT 0,
+  lat_3 BIGINT NOT NULL DEFAULT 0,
+  lat_4 BIGINT NOT NULL DEFAULT 0,
+  lat_5 BIGINT NOT NULL DEFAULT 0,
+  lat_6 BIGINT NOT NULL DEFAULT 0,
+  lat_7 BIGINT NOT NULL DEFAULT 0,
+  lat_8 BIGINT NOT NULL DEFAULT 0,
+  lat_9 BIGINT NOT NULL DEFAULT 0,
+  lat_10 BIGINT NOT NULL DEFAULT 0,
+  ttft_0 BIGINT NOT NULL DEFAULT 0,
+  ttft_1 BIGINT NOT NULL DEFAULT 0,
+  ttft_2 BIGINT NOT NULL DEFAULT 0,
+  ttft_3 BIGINT NOT NULL DEFAULT 0,
+  ttft_4 BIGINT NOT NULL DEFAULT 0,
+  ttft_5 BIGINT NOT NULL DEFAULT 0,
+  ttft_6 BIGINT NOT NULL DEFAULT 0,
+  ttft_7 BIGINT NOT NULL DEFAULT 0,
+  ttft_8 BIGINT NOT NULL DEFAULT 0,
+  ttft_9 BIGINT NOT NULL DEFAULT 0,
+  ttft_10 BIGINT NOT NULL DEFAULT 0,
+  PRIMARY KEY (bucket, key_id, user_id, project_id, model, served_model)
+);
+CREATE INDEX usage_1h_by_bucket ON usage_1h(bucket);
+CREATE INDEX usage_1h_by_project ON usage_1h(project_id, bucket);
+CREATE INDEX usage_1h_by_user ON usage_1h(user_id, bucket);
+CREATE INDEX usage_1h_by_key ON usage_1h(key_id, bucket);
+
+INSERT INTO usage_1h (bucket, key_id, user_id, project_id, model, served_model,
+  requests, errors, limited, prompt_tokens, completion_tokens, cost_micros,
+  latency_ms_sum, queue_ms_sum, ttft_ms_sum, ttft_n, last_ts,
+  lat_0, lat_1, lat_2, lat_3, lat_4, lat_5, lat_6, lat_7, lat_8, lat_9, lat_10,
+  ttft_0, ttft_1, ttft_2, ttft_3, ttft_4, ttft_5, ttft_6, ttft_7, ttft_8, ttft_9, ttft_10)
+SELECT (ts / 3600000) * 3600000, key_id, COALESCE(user_id, ''), project_id, model, served_model,
+  COUNT(*), SUM(CASE WHEN status >= 400 THEN 1 ELSE 0 END), SUM(CASE WHEN status = 429 THEN 1 ELSE 0 END),
+  SUM(prompt_tokens), SUM(completion_tokens), SUM(cost_micros),
+  SUM(latency_ms), SUM(queue_ms), SUM(ttft_ms), SUM(CASE WHEN ttft_ms > 0 THEN 1 ELSE 0 END), MAX(ts),
+  SUM(CASE WHEN latency_ms <= 100 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 100 AND latency_ms <= 250 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 250 AND latency_ms <= 500 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 500 AND latency_ms <= 1000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 1000 AND latency_ms <= 2500 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 2500 AND latency_ms <= 5000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 5000 AND latency_ms <= 10000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 10000 AND latency_ms <= 30000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 30000 AND latency_ms <= 60000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 60000 AND latency_ms <= 120000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 120000 THEN 1 ELSE 0 END),
+  SUM(CASE WHEN ttft_ms > 0 AND ttft_ms <= 100 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 100 AND ttft_ms <= 250 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 250 AND ttft_ms <= 500 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 500 AND ttft_ms <= 1000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 1000 AND ttft_ms <= 2500 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 2500 AND ttft_ms <= 5000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 5000 AND ttft_ms <= 10000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 10000 AND ttft_ms <= 30000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 30000 AND ttft_ms <= 60000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 60000 AND ttft_ms <= 120000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 120000 THEN 1 ELSE 0 END)
+FROM usage
+GROUP BY 1, 2, 3, 4, 5, 6;
+
+CREATE TABLE usage_1d (
+  bucket            BIGINT NOT NULL,
+  key_id            TEXT NOT NULL,
+  user_id           TEXT NOT NULL,
+  project_id        TEXT NOT NULL,
+  model             TEXT NOT NULL,
+  served_model      TEXT NOT NULL,
+  requests          BIGINT NOT NULL DEFAULT 0,
+  errors            BIGINT NOT NULL DEFAULT 0,
+  limited           BIGINT NOT NULL DEFAULT 0,
+  prompt_tokens     BIGINT NOT NULL DEFAULT 0,
+  completion_tokens BIGINT NOT NULL DEFAULT 0,
+  cost_micros       BIGINT NOT NULL DEFAULT 0,
+  latency_ms_sum    BIGINT NOT NULL DEFAULT 0,
+  queue_ms_sum      BIGINT NOT NULL DEFAULT 0,
+  ttft_ms_sum       BIGINT NOT NULL DEFAULT 0,
+  ttft_n            BIGINT NOT NULL DEFAULT 0,
+  last_ts           BIGINT NOT NULL DEFAULT 0,
+  lat_0 BIGINT NOT NULL DEFAULT 0,
+  lat_1 BIGINT NOT NULL DEFAULT 0,
+  lat_2 BIGINT NOT NULL DEFAULT 0,
+  lat_3 BIGINT NOT NULL DEFAULT 0,
+  lat_4 BIGINT NOT NULL DEFAULT 0,
+  lat_5 BIGINT NOT NULL DEFAULT 0,
+  lat_6 BIGINT NOT NULL DEFAULT 0,
+  lat_7 BIGINT NOT NULL DEFAULT 0,
+  lat_8 BIGINT NOT NULL DEFAULT 0,
+  lat_9 BIGINT NOT NULL DEFAULT 0,
+  lat_10 BIGINT NOT NULL DEFAULT 0,
+  ttft_0 BIGINT NOT NULL DEFAULT 0,
+  ttft_1 BIGINT NOT NULL DEFAULT 0,
+  ttft_2 BIGINT NOT NULL DEFAULT 0,
+  ttft_3 BIGINT NOT NULL DEFAULT 0,
+  ttft_4 BIGINT NOT NULL DEFAULT 0,
+  ttft_5 BIGINT NOT NULL DEFAULT 0,
+  ttft_6 BIGINT NOT NULL DEFAULT 0,
+  ttft_7 BIGINT NOT NULL DEFAULT 0,
+  ttft_8 BIGINT NOT NULL DEFAULT 0,
+  ttft_9 BIGINT NOT NULL DEFAULT 0,
+  ttft_10 BIGINT NOT NULL DEFAULT 0,
+  PRIMARY KEY (bucket, key_id, user_id, project_id, model, served_model)
+);
+CREATE INDEX usage_1d_by_bucket ON usage_1d(bucket);
+CREATE INDEX usage_1d_by_project ON usage_1d(project_id, bucket);
+CREATE INDEX usage_1d_by_user ON usage_1d(user_id, bucket);
+CREATE INDEX usage_1d_by_key ON usage_1d(key_id, bucket);
+
+INSERT INTO usage_1d (bucket, key_id, user_id, project_id, model, served_model,
+  requests, errors, limited, prompt_tokens, completion_tokens, cost_micros,
+  latency_ms_sum, queue_ms_sum, ttft_ms_sum, ttft_n, last_ts,
+  lat_0, lat_1, lat_2, lat_3, lat_4, lat_5, lat_6, lat_7, lat_8, lat_9, lat_10,
+  ttft_0, ttft_1, ttft_2, ttft_3, ttft_4, ttft_5, ttft_6, ttft_7, ttft_8, ttft_9, ttft_10)
+SELECT (ts / 86400000) * 86400000, key_id, COALESCE(user_id, ''), project_id, model, served_model,
+  COUNT(*), SUM(CASE WHEN status >= 400 THEN 1 ELSE 0 END), SUM(CASE WHEN status = 429 THEN 1 ELSE 0 END),
+  SUM(prompt_tokens), SUM(completion_tokens), SUM(cost_micros),
+  SUM(latency_ms), SUM(queue_ms), SUM(ttft_ms), SUM(CASE WHEN ttft_ms > 0 THEN 1 ELSE 0 END), MAX(ts),
+  SUM(CASE WHEN latency_ms <= 100 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 100 AND latency_ms <= 250 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 250 AND latency_ms <= 500 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 500 AND latency_ms <= 1000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 1000 AND latency_ms <= 2500 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 2500 AND latency_ms <= 5000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 5000 AND latency_ms <= 10000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 10000 AND latency_ms <= 30000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 30000 AND latency_ms <= 60000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 60000 AND latency_ms <= 120000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 120000 THEN 1 ELSE 0 END),
+  SUM(CASE WHEN ttft_ms > 0 AND ttft_ms <= 100 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 100 AND ttft_ms <= 250 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 250 AND ttft_ms <= 500 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 500 AND ttft_ms <= 1000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 1000 AND ttft_ms <= 2500 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 2500 AND ttft_ms <= 5000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 5000 AND ttft_ms <= 10000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 10000 AND ttft_ms <= 30000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 30000 AND ttft_ms <= 60000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 60000 AND ttft_ms <= 120000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 120000 THEN 1 ELSE 0 END)
+FROM usage
+GROUP BY 1, 2, 3, 4, 5, 6;