Merge ha: replicated store, second control plane, join codes, setup and console screens

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

CHANGELOG.md

@@ -15,14 +15,26 @@ is only released once its **exit criterion** is met, not when its code lands.
 | v0.8.0 | — | PostgreSQL, leader election, agent failover | released |
 | v0.9.0 | — | Managing people and the tenancy tree | released |
 | v0.10.0 | — | InfiniBand fabrics and NVLink, doctor, node add, engines installed by the nodes | released |
+| v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released |
 | v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
 
 Until v1.0.0 the minor version is the breaking one: configuration formats and Go
 APIs change between minors without a deprecation period.
 
-## Unreleased
+## v0.11.0 — 2026-10-03
 
-From the field report in issue #2, after a deployment on GPU nodes.
+A minor version: the store can be replicated and the control plane doubled,
+both set up by llmgw itself; `sudo llmgw` opens setup screens and a console;
+usage is read from sums kept as it is written. Released without candidates.
+
+**Upgrading.** Migrations 0015 (usage sums, filled from the request table
+already recorded — once, at the first start) and 0016 (control planes) run by
+themselves. An agent older than this ignores the list of control planes and
+keeps its configured one. Nothing in `server.yaml` or `agent.yaml` changes
+meaning; `auth.usage_retention` and `router.tenant_label` are new and optional.
+
+The GPU and fabric work below began from the field report in issue #2, after
+a deployment on GPU nodes.
 
 ### GPU metrics that move, and charts of them
 
@@ -75,6 +87,230 @@ and a figure the driver does not give is left out rather than shown as zero.
   each card and whose quota it counts against, for joining the GPU series to
   models and tenants.
 
+### Usage is read from sums kept as it is written
+
+Every chart and total — the usage tab, quotas, the dashboard, a person's own
+page, and budgets restored at start — summed the request table again on each
+view: a row per request, scanned whole. Usage is now also summed as it is
+written, in the same transaction, into five-minute slices, hours and days per
+key, person, project and model (`usage_5m`, `usage_1h`, `usage_1d`; migration
+0015 fills them from what is already recorded).
+
+- **Charts** read the coarsest of the three their slices are made of: a day in
+  20-minute slices from the five-minute table, a week or a month from hours.
+- **Totals and budgets** read the rows up to the next five-minute slice, then
+  slices, hours and days — exact, not rounded to a slice. On three million
+  requests over thirty days, a month's total per person went from 705 ms to
+  18 ms on PostgreSQL.
+- **Latency and time to first token** are kept as histograms in the sums, so a
+  percentile per team or model can be read after the rows are gone.
+- **`auth.usage_retention`** (e.g. `2160h`) deletes request records older than
+  that, hourly. Totals, charts and budgets are unchanged by it — they read the
+  sums — but `ctl request` finds nothing older. Unset keeps every record, as
+  before.
+
+### Alerts per tenant and per budget
+
+- **`llmgw_tenant_requests_total{tenant,outcome}`**, `llmgw_tenant_tokens_total`
+  and `llmgw_tenant_ttft_seconds`: requests by outcome (ok, error, limited),
+  tokens and time to first token per tenant. The tenant is the level
+  `router.tenant_label` names — `team` by default, or `org`, `project`, or
+  `none` — named by its path, "acme/platform". One level rather than every
+  project, so a cluster of thousands of projects does not become a Prometheus
+  of millions of series.
+- **`llmgw_budget_spent_ratio{scope,id,name,period}`**: this period's spend
+  over the budget for every organization, team, project and person given one,
+  refreshed every minute.
+- **Alerts** in `deploy/monitoring/llmgw-rules.yml`: `BudgetMostlySpent` (80%),
+  `BudgetSpent`, `TenantHittingLimits` (a tenth of requests refused),
+  `TenantErrors`, `TenantTrafficSpike` (three times the weekly average) and
+  `TenantSlowFirstToken`.
+- **A Grafana dashboard per tenant**, `deploy/monitoring/grafana/llmgw-tenants.json`
+  (uid `llmgw-tenants`): requests and tokens by tenant, the share failing and
+  refused by a limit, first-token p95, every budget's share spent, and who used
+  most in the last day — with a tenant chooser at the top.
+
+### How each team's requests fared, not only what they cost
+
+- **Percentiles from the sums.** First token and latency, p50, p95 and p99,
+  per organization, team, project, person or model, read from the histograms
+  kept with the usage sums: on the usage tab's table, and in a branch's or
+  person's panel on the people page, beside the error rate and requests
+  refused by a limit.
+- **Refused requests are counted against their owner.** A request a rate,
+  token or budget limit refused was counted only in Prometheus, by model; a
+  team running into its limit looked like a team that had gone quiet. It is
+  now recorded as a 429 against its key, person and project — no tokens, no
+  cost — and shown as "Limited".
+- **Usage by model** is a grouping of the usage tab, by the model that
+  answered.
+### The console watches the cluster and holds every setting
+
+`sudo llmgw` on a control plane is a console in three groups down the left:
+
+- **Watch**, refreshed every three seconds: the overview; nodes, with CPU,
+  memory, GPUs free, disk and agent version, and a node's detail — load,
+  engines, each card's busy, memory, temperature and power; every GPU in the
+  fleet and which model holds it; models, with a model's profiles, warnings
+  and replicas; replicas and where they run; the last day's use by team,
+  project or organization; and a node's agent log.
+- **Settings**: every key of `server.yaml`, block by block, with what applies
+  when the file sets nothing and what each key is for, from its comment in
+  the code. Enter changes a key or opens a block; lists and maps of blocks
+  (endpoints, models, prices) take entries added and removed. A save is read
+  by the same loader the service uses first — nothing is written that would
+  not load — keeps the file's comments, leaves the old one beside it, and
+  offers to restart the service.
+- **Do**: add a node, replicate the store, leave.
+
+On a node, `sudo llmgw` opens the same settings for its `agent.yaml`.
+
+The keys come from the configuration structs themselves, through the
+generator that writes `docs/reference.md`, so a setting added to the code is on
+the screen without anyone listing it, and a test fails if the two drift.
+That generator told blocks apart by name alone, so `tlsconf.Config`,
+`engines.Config` and `supervisor.Config` were each taken for the agent's
+`Config` and dropped: the reference had no TLS, engine or supervisor settings
+at all. It now has 37 blocks, six more than before.
+
+### A second control plane with `llmgw server add`, and the screens as windows
+
+- **`sudo llmgw server add [user@]host [--vip 10.0.0.160/24]`**, run on a
+  control plane, makes another machine a standby: this llmgw copied over, a
+  certificate from the cluster's authority, the same configuration and store
+  address, the machine let into the store (for a store under Patroni, one
+  change through Patroni's API), and the service started. Every node is told
+  of it within half a minute and fails over to it untouched. `--vip` keeps a
+  floating address on whichever control plane leads, with keepalived on both.
+  Tried on new machines: switching the leader off, the standby led after 14
+  seconds and the floating address answered from it after 23.
+- **Nodes learn every control plane** from the leader (a new `ControlPlanes`
+  message, kept in `/var/lib/llmgw-agent/control-planes.json`), after the
+  ones in their configuration.
+- **A node notices a dead control plane within a quarter of a minute.** A
+  leader whose machine stopped dead left its nodes' streams open until the
+  kernel gave up retransmitting, about fifteen minutes; the agent now pings
+  the stream after ten quiet seconds, and bounds dialling too.
+- **`GET /leader`**: 200 on the control plane that leads, 503 on a standby,
+  open like `/ready` — what keepalived follows.
+- **The store's access rules live in Patroni's shared configuration** for a
+  cluster set up by `server init --patroni`, so another control plane is let
+  in without touching each member.
+- **The screens are windows**: full screen, a title bar, the steps or actions
+  down the left, the content in a bordered panel, the keys along the bottom,
+  questions and errors as dialogs over it — the setup, the console, adding a
+  node and one-machine init alike. Keyboard only.
+
+### Machines nothing may log into: `llmgw join`
+
+For sites that allow no ssh into a machine, the setup screens offer a fourth
+way to reach one: a join code. The setup listens on port 7444 while it waits
+and shows a code for each such machine, with the commands that fetch llmgw
+onto it; there, `sudo llmgw join <code>` dials out — nothing dials in —
+checks by the certificate fingerprint in the code that it reached the setup
+that made it, admits itself with the code's secret, and runs what the setup
+sends, printing each step, until the setup says it is done. A code is good
+for one machine and an hour, and is under 90 characters so it can be copied
+whole from one screen into another.
+
+To the setup a joined machine is one more way of running a script there:
+the checks, PostgreSQL, Patroni and etcd go over it unchanged. Tried with one
+machine joining by code and another logged into with a password, in one
+setup.
+
+### Setting up from nothing in screens: `sudo llmgw`
+
+Run with nothing after it in a terminal, as root, `llmgw` opens full-screen
+setup on a machine with nothing installed, and a console on a control plane.
+In a pipe or a script it prints its help, as before.
+
+- **Setup** asks one machine or three. For three — this one as the control
+  plane and etcd's third vote, two more for the store under Patroni — it asks
+  how to reach the other two: the ssh agent, a key file, or a password, and a
+  sudo password where sudo asks for one. Passwords stay in memory for the run;
+  a sudo password grants the user passwordless sudo until the install
+  finishes, and the grant is removed after, failed or not.
+- **Every machine is checked before anything is installed**: login, system
+  (Ubuntu 22.04/24.04, Debian 12), architecture, CPUs, memory and disk for its
+  role, clock against this one, NTP, the ports its role needs, internet, and
+  whether it reaches the other machines. Failures stop it, with what to do;
+  warnings let it go on.
+- **Then the plan, and the command line that does the same** from a script,
+  then the install with its progress, and the administrator's password.
+- **The console** shows `ctl status`, refreshed every five seconds, and offers
+  to add a node or, for a store on one machine, to replicate it.
+- Machines are reached with Go's ssh client rather than the system's, so a
+  password can be used; a host key is recorded on first sight and refused if it
+  changes.
+
+Tried end to end on three new machines whose user logs in and sudoes only with
+a password.
+
+### A store that survives losing its machine
+
+`deploy/ha/` now covers the store as well as the control planes: PostgreSQL
+under Patroni on two machines, its leader lock in a three-member etcd, and
+llmgw given every member with `target_session_attrs=read-write`. Measured: a
+planned switchover cost one failed lease renewal; switching off the primary's
+machine promoted the standby in 24 seconds with the gateway serving
+throughout. Example configurations are beside it.
+
+**`llmgw server init --patroni user@host --patroni user@host`** sets it up:
+PostgreSQL, Patroni and etcd on those machines over ssh — this one as etcd's
+third vote — the role and database, and the store already in use moved in.
+Every step looks first; run against a cluster that is up, it changes nothing.
+Tried from scratch on three new machines (two minutes, then a failover by
+switching the primary off) and against the reference cluster (nothing
+touched).
+
+**Watching it**: for a store of several hosts the control plane publishes
+`llmgw_store_member`, `llmgw_store_replica_lag_bytes` and
+`llmgw_store_patroni_up` from each member's Patroni, a member that stopped
+showing as `missing` rather than vanishing; `llmgw ctl status` has a STORE
+line; and four alerts — `StoreHasNoLeader`, `StorePatroniUnreachable`,
+`StoreReplicaNotStreaming`, `StoreReplicaBehind`.
+
+**Making an administrator on PostgreSQL** (`init --admin`, `users promote`)
+failed: the flag was sent as a bool to an integer column, which SQLite takes
+and PostgreSQL refuses. Every existing PostgreSQL store got its administrator
+by being copied from SQLite, which is why nobody had met it.
+
+`llmgw server backup` reads a store address of several hosts. It took the
+list for one host name and reached the database only by falling through to
+the second; it now hands libpq the list, and dumps from a standby when there
+is one.
+
+### People, limits and usage are one page
+
+Managing a team meant its members on the people page, its quota on the quotas
+page and its spend on the usage page, each linking to the others. They are now
+one page in three tabs — **Organizations & people**, **Limits**, **Usage** —
+each at the address it had before, so links and bookmarks still land. The menu
+has one entry for them, People.
+
+Choosing an organization, team, project or person in the tree, or a person in
+the account list, opens a panel beside it with everything about that one
+thing: its own limits and what it is held to from above, spend against its
+budget, its per-member limit, its capacity quota, and its requests, tokens
+and spend over the last week, drawn. Its limits can be set or changed there,
+and the answer comes back to the same panel.
+
+The administrator's menu no longer has an Organizations entry beside it: `/org`
+and `/org/<id>` send an administrator to that organization's panel here. The
+page stays as it was for an organization's managers, who cannot see this one.
+
+### The chat page says whether a GPU or a CPU answered
+
+A model can have replicas on a card and on CPUs at once, and the two answer
+at very different speeds; the chat page gave no way to tell which one a reply
+came from. The picker now says where each model runs — `GPU`, `CPU` or
+`GPU + CPU` — and the line under each reply names what answered it: `GPU
+6000-24g` or `CPU`, beside its tokens a second.
+
+Every answer through the gateway carries the same as `x-llmgw-hardware`:
+`cpu`, or the GPU class. An endpoint configured by hand, with no pool, sends
+none.
+
 ### A card shared with something else is judged every heartbeat
 
 A card with another process on it when its agent started was kept out of

api/proto/llmgw/v1/agent.proto

@@ -43,9 +43,18 @@ message ControlMessage {
     FabricScanRequest fabric_scan = 8;
     FabricTestRequest fabric_test = 9;
     UpgradeAgent upgrade = 10;
+    ControlPlanes control_planes = 11;
   }
 }
 
+// ControlPlanes is every control plane an agent may dial, sent by the leader
+// when an agent registers and whenever the list changes. The agent keeps it
+// beside its configuration, so a second control plane added after the node
+// was onboarded is one it fails over to without anybody editing the node.
+message ControlPlanes {
+  repeated string servers = 1;
+}
+
 // Register is the first message on every stream. It carries the node's full
 // hardware inventory, so the control plane needs no separate discovery call and
 // a reconnecting agent automatically refreshes what the scheduler knows.

deploy/examples/server.yaml

@@ -92,6 +92,11 @@ agent_listen: ":7443"
 #   flush_interval: 200ms
 #   batch_size: 500
 #   buffer_size: 8192
+#
+#   # How long the record of each request is kept (what `ctl request` reads).
+#   # Charts, totals and budgets come from five-minute sums that are kept, so
+#   # this only bounds the table that grows by a row a request. 0 keeps all.
+#   usage_retention: 2160h   # 90 days
 
 router:
   scrape_interval: 2s

deploy/ha/README.md

@@ -85,3 +85,63 @@ auth:
 
 See [docs/install.md](../../docs/install.md) for moving an existing
 installation onto PostgreSQL without reissuing every key.
+
+## The store itself: PostgreSQL under Patroni
+
+Two control planes on one PostgreSQL have moved the single point of failure,
+not removed it. Patroni keeps PostgreSQL on two machines, one streaming to the
+other, and promotes the standby when the primary is lost. It keeps its leader
+lock in etcd, and etcd decides only with a majority — so three members, not
+two: two machines that lose sight of each other must not both decide they are
+the primary. The third member can be any machine that is up; it holds a vote,
+not data.
+
+On the reference cluster:
+
+| machine | etcd | PostgreSQL |
+|---|---|---|
+| llmgw-ctl (control plane) | member | — |
+| llmgw-pg01 | member | Patroni |
+| llmgw-ctl02 (second control plane) | member | Patroni |
+
+`llmgw server init --patroni user@host --patroni user@host` sets all of it up
+over ssh and moves the store in — see [install.md](../../docs/install.md).
+It looks before it acts at every step, so it is also how an existing cluster
+is checked: run against one that is up, it changes nothing. What it writes
+is in `etcd.default.example` (one member's `/etc/default/etcd`) and
+`patroni.yml.example` (one node's `/etc/patroni/config.yml`), for anyone
+doing it by hand; passwords are generated into `/root/llmgw-pg` on each
+database machine.
+
+**Give llmgw every member, and ask for the writable one:**
+
+```
+postgres://llmgw:…@10.0.0.149:5432,10.0.0.150:5432/llmgw?sslmode=disable&target_session_attrs=read-write
+```
+
+The driver connects to whichever member accepts writes, and reconnects to the
+new one after a failover. No floating address or HAProxy is needed for that.
+`llmgw server backup` reads the same address and dumps from a standby when
+there is one.
+
+Measured on the reference cluster, with the gateway serving throughout:
+
+- **Planned switchover** (`patronictl switchover`): the control plane logged
+  one failed lease renewal, reconnected to the new primary, and kept its
+  leadership; nothing else noticed.
+- **The primary's machine switched off**: the standby was promoted 24 seconds
+  later — the 30-second TTL in `patroni.yml` — and the control plane rode it
+  out without losing leadership or a replica. Started again, the old primary
+  rejoined as a standby by itself.
+
+**Watching it.** For a store of several hosts the control plane reads the
+cluster from each member's Patroni (port 8008) and publishes
+`llmgw_store_member{member,host,role,state}`, `llmgw_store_replica_lag_bytes`
+and `llmgw_store_patroni_up`; a host of the address that the cluster no longer
+lists is a member in state `missing`. `llmgw ctl status` prints it as one
+line, and `deploy/monitoring/llmgw-rules.yml` alerts on no leader, a member
+not streaming or missing, and a standby too far behind to be promoted.
+
+Not done here, and worth doing in production: a watchdog (`softdog`) so a
+primary that has lost the lock but not died cannot keep accepting writes, TLS
+on etcd and between PostgreSQL members, and more than one standby.

deploy/ha/etcd.default.example

@@ -0,0 +1,10 @@
+# llmgw: one member of the three that hold Patroni's leader lock (140, 149, 150).
+ETCD_NAME=llmgw-pg01
+ETCD_DATA_DIR=/var/lib/etcd/default
+ETCD_LISTEN_PEER_URLS=http://10.0.0.149:2380
+ETCD_LISTEN_CLIENT_URLS=http://10.0.0.149:2379,http://127.0.0.1:2379
+ETCD_INITIAL_ADVERTISE_PEER_URLS=http://10.0.0.149:2380
+ETCD_ADVERTISE_CLIENT_URLS=http://10.0.0.149:2379
+ETCD_INITIAL_CLUSTER=llmgw-ctl=http://10.0.0.140:2380,llmgw-pg01=http://10.0.0.149:2380,llmgw-ctl02=http://10.0.0.150:2380
+ETCD_INITIAL_CLUSTER_STATE=new
+ETCD_INITIAL_CLUSTER_TOKEN=llmgw-patroni

deploy/ha/patroni.yml.example

@@ -0,0 +1,68 @@
+# llmgw: the control planes' store, kept by Patroni on llmgw-pg01 and
+# llmgw-ctl02, with the leader lock in etcd on 140, 149 and 150.
+scope: llmgw
+namespace: /service/
+name: llmgw-pg01
+
+restapi:
+  listen: 10.0.0.149:8008
+  connect_address: 10.0.0.149:8008
+
+etcd3:
+  hosts: 10.0.0.140:2379,10.0.0.149:2379,10.0.0.150:2379
+
+bootstrap:
+  dcs:
+    ttl: 30
+    loop_wait: 10
+    retry_timeout: 10
+    maximum_lag_on_failover: 1048576
+    postgresql:
+      use_pg_rewind: true
+      use_slots: true
+      parameters:
+        wal_level: replica
+        hot_standby: "on"
+        max_wal_senders: 10
+        max_replication_slots: 10
+        wal_keep_size: 1GB
+        wal_log_hints: "on"
+        password_encryption: scram-sha-256
+  initdb:
+    - encoding: UTF8
+    - data-checksums
+
+postgresql:
+  listen: 127.0.0.1,10.0.0.149:5432
+  connect_address: 10.0.0.149:5432
+  data_dir: /var/lib/postgresql/16/llmgw
+  bin_dir: /usr/lib/postgresql/16/bin
+  pgpass: /var/lib/postgresql/.pgpass_patroni
+  authentication:
+    superuser:
+      username: postgres
+      password: "<from /root/llmgw-pg>"
+    replication:
+      username: replicator
+      password: "<from /root/llmgw-pg>"
+    rewind:
+      username: postgres
+      password: "<from /root/llmgw-pg>"
+  parameters:
+    unix_socket_directories: /var/run/postgresql
+  pg_hba:
+    - local all all peer
+    - host all all 127.0.0.1/32 scram-sha-256
+    - host replication replicator 10.0.0.149/32 scram-sha-256
+    - host replication replicator 10.0.0.150/32 scram-sha-256
+    - host all postgres 10.0.0.149/32 scram-sha-256
+    - host all postgres 10.0.0.150/32 scram-sha-256
+    - host llmgw llmgw 10.0.0.140/32 scram-sha-256
+    - host llmgw llmgw 10.0.0.149/32 scram-sha-256
+    - host llmgw llmgw 10.0.0.150/32 scram-sha-256
+
+tags:
+  nofailover: false
+  noloadbalance: false
+  clonefrom: false
+  nosync: false

deploy/monitoring/README.md

@@ -66,6 +66,8 @@ install -m 0644 grafana/datasource.yaml   /etc/grafana/provisioning/datasources/
 install -m 0644 grafana/dashboards.yaml   /etc/grafana/provisioning/dashboards/llmgw.yaml
 install -m 0644 -o grafana -g grafana grafana/llmgw-overview.json \
     /var/lib/grafana/dashboards/llmgw.json
+install -m 0644 -o grafana -g grafana grafana/llmgw-tenants.json \
+    /var/lib/grafana/dashboards/llmgw-tenants.json
 systemctl restart grafana-server
 ```
 

deploy/monitoring/grafana/llmgw-tenants.json

@@ -0,0 +1,650 @@
+{
+  "uid": "llmgw-tenants",
+  "title": "llmgw \u2014 tenants",
+  "schemaVersion": 39,
+  "version": 1,
+  "editable": true,
+  "time": {
+    "from": "now-24h",
+    "to": "now"
+  },
+  "refresh": "1m",
+  "tags": [
+    "llmgw"
+  ],
+  "description": "Requests, tokens, failures, refusals and first-token time per tenant \u2014 the level router.tenant_label names \u2014 and every budget's share spent.",
+  "templating": {
+    "list": [
+      {
+        "name": "tenant",
+        "label": "Tenant",
+        "type": "query",
+        "datasource": {
+          "type": "prometheus",
+          "uid": "llmgw-prom"
+        },
+        "query": {
+          "query": "label_values(llmgw_tenant_requests_total, tenant)",
+          "refId": "tenant"
+        },
+        "definition": "label_values(llmgw_tenant_requests_total, tenant)",
+        "multi": true,
+        "includeAll": true,
+        "allValue": ".*",
+        "current": {
+          "text": "All",
+          "value": "$__all"
+        },
+        "refresh": 2,
+        "sort": 1
+      }
+    ]
+  },
+  "panels": [
+    {
+      "id": 1,
+      "title": "Requests a second",
+      "type": "stat",
+      "description": "Across the tenants chosen above.",
+      "gridPos": {
+        "x": 0,
+        "y": 0,
+        "w": 6,
+        "h": 4
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "sum(rate(llmgw_tenant_requests_total{tenant=~\"$tenant\"}[5m]))",
+          "legendFormat": "",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps"
+        },
+        "overrides": []
+      },
+      "options": {
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        },
+        "textMode": "auto",
+        "colorMode": "value"
+      }
+    },
+    {
+      "id": 2,
+      "title": "Failing",
+      "type": "stat",
+      "description": "Requests that failed, not counting refusals by a limit.",
+      "gridPos": {
+        "x": 6,
+        "y": 0,
+        "w": 6,
+        "h": 4
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "sum(rate(llmgw_tenant_requests_total{tenant=~\"$tenant\",outcome=\"error\"}[5m])) / sum(rate(llmgw_tenant_requests_total{tenant=~\"$tenant\"}[5m]))",
+          "legendFormat": "",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percentunit",
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "orange",
+                "value": 0.02
+              },
+              {
+                "color": "red",
+                "value": 0.05
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        },
+        "textMode": "auto",
+        "colorMode": "value"
+      }
+    },
+    {
+      "id": 3,
+      "title": "Refused by a limit",
+      "type": "stat",
+      "description": "Rate, token and budget limits; 429 to the caller.",
+      "gridPos": {
+        "x": 12,
+        "y": 0,
+        "w": 6,
+        "h": 4
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "sum(rate(llmgw_tenant_requests_total{tenant=~\"$tenant\",outcome=\"limited\"}[5m])) / sum(rate(llmgw_tenant_requests_total{tenant=~\"$tenant\"}[5m]))",
+          "legendFormat": "",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percentunit",
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "orange",
+                "value": 0.02
+              },
+              {
+                "color": "red",
+                "value": 0.1
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        },
+        "textMode": "auto",
+        "colorMode": "value"
+      }
+    },
+    {
+      "id": 4,
+      "title": "First token, p95",
+      "type": "stat",
+      "description": "What people wait for before anything appears.",
+      "gridPos": {
+        "x": 18,
+        "y": 0,
+        "w": 6,
+        "h": 4
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "histogram_quantile(0.95, sum by (le) (rate(llmgw_tenant_ttft_seconds_bucket{tenant=~\"$tenant\"}[5m])))",
+          "legendFormat": "",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "s",
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "orange",
+                "value": 3
+              },
+              {
+                "color": "red",
+                "value": 10
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        },
+        "textMode": "auto",
+        "colorMode": "value"
+      }
+    },
+    {
+      "id": 5,
+      "title": "Requests by tenant",
+      "type": "timeseries",
+      "description": "Stacked, so the top edge is the whole of it.",
+      "gridPos": {
+        "x": 0,
+        "y": 4,
+        "w": 12,
+        "h": 9
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "sum by (tenant) (rate(llmgw_tenant_requests_total{tenant=~\"$tenant\"}[5m]))",
+          "legendFormat": "{{tenant}}",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps",
+          "custom": {
+            "stacking": {
+              "mode": "normal"
+            },
+            "fillOpacity": 30
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "table",
+          "placement": "right",
+          "calcs": [
+            "mean",
+            "max"
+          ]
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      }
+    },
+    {
+      "id": 6,
+      "title": "Tokens by tenant",
+      "type": "timeseries",
+      "description": "Prompt and completion together: what was consumed.",
+      "gridPos": {
+        "x": 12,
+        "y": 4,
+        "w": 12,
+        "h": 9
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "sum by (tenant) (rate(llmgw_tenant_tokens_total{tenant=~\"$tenant\"}[5m]))",
+          "legendFormat": "{{tenant}}",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short",
+          "custom": {
+            "stacking": {
+              "mode": "normal"
+            },
+            "fillOpacity": 30
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "table",
+          "placement": "right",
+          "calcs": [
+            "mean",
+            "max"
+          ]
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      }
+    },
+    {
+      "id": 7,
+      "title": "Refused by a limit, by tenant",
+      "type": "timeseries",
+      "description": "The share of each tenant's requests a limit turned away.",
+      "gridPos": {
+        "x": 0,
+        "y": 13,
+        "w": 12,
+        "h": 8
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "sum by (tenant) (rate(llmgw_tenant_requests_total{tenant=~\"$tenant\",outcome=\"limited\"}[5m])) / sum by (tenant) (rate(llmgw_tenant_requests_total{tenant=~\"$tenant\"}[5m]))",
+          "legendFormat": "{{tenant}}",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "table",
+          "placement": "right",
+          "calcs": [
+            "mean",
+            "max"
+          ]
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      }
+    },
+    {
+      "id": 8,
+      "title": "Failing, by tenant",
+      "type": "timeseries",
+      "description": "One tenant alone failing is usually what it sends; all of them together is a model.",
+      "gridPos": {
+        "x": 12,
+        "y": 13,
+        "w": 12,
+        "h": 8
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "sum by (tenant) (rate(llmgw_tenant_requests_total{tenant=~\"$tenant\",outcome=\"error\"}[5m])) / sum by (tenant) (rate(llmgw_tenant_requests_total{tenant=~\"$tenant\"}[5m]))",
+          "legendFormat": "{{tenant}}",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "table",
+          "placement": "right",
+          "calcs": [
+            "mean",
+            "max"
+          ]
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      }
+    },
+    {
+      "id": 9,
+      "title": "First token p95, by tenant",
+      "type": "timeseries",
+      "description": "",
+      "gridPos": {
+        "x": 0,
+        "y": 21,
+        "w": 12,
+        "h": 8
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "histogram_quantile(0.95, sum by (tenant, le) (rate(llmgw_tenant_ttft_seconds_bucket{tenant=~\"$tenant\"}[10m])))",
+          "legendFormat": "{{tenant}}",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "s"
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "table",
+          "placement": "right",
+          "calcs": [
+            "mean",
+            "max"
+          ]
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      }
+    },
+    {
+      "id": 10,
+      "title": "Budgets spent this period",
+      "type": "bargauge",
+      "description": "Every organization, team, project and person with a budget. Requests are refused at 100%.",
+      "gridPos": {
+        "x": 12,
+        "y": 21,
+        "w": 12,
+        "h": 8
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "sort_desc(llmgw_budget_spent_ratio)",
+          "legendFormat": "{{scope}} {{name}} ({{period}})",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percentunit",
+          "min": 0,
+          "max": 1,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "orange",
+                "value": 0.8
+              },
+              {
+                "color": "red",
+                "value": 1
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "displayMode": "gradient",
+        "orientation": "horizontal",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        },
+        "showUnfilled": true
+      }
+    },
+    {
+      "id": 11,
+      "title": "Who used most, last 24 hours",
+      "type": "table",
+      "description": "Spend in money is on the people page's usage tab, which reads it from the store.",
+      "gridPos": {
+        "x": 0,
+        "y": 29,
+        "w": 24,
+        "h": 8
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "sort_desc(sum by (tenant) (increase(llmgw_tenant_requests_total{tenant=~\"$tenant\"}[24h])))",
+          "legendFormat": "requests",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          },
+          "format": "table",
+          "instant": true
+        },
+        {
+          "refId": "B",
+          "expr": "sum by (tenant) (increase(llmgw_tenant_tokens_total{tenant=~\"$tenant\"}[24h]))",
+          "legendFormat": "tokens",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          },
+          "format": "table",
+          "instant": true
+        },
+        {
+          "refId": "C",
+          "expr": "sum by (tenant) (increase(llmgw_tenant_requests_total{tenant=~\"$tenant\",outcome=\"limited\"}[24h]))",
+          "legendFormat": "limited",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          },
+          "format": "table",
+          "instant": true
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {},
+        "overrides": []
+      },
+      "options": {
+        "showHeader": true
+      },
+      "transformations": [
+        {
+          "id": "merge",
+          "options": {}
+        },
+        {
+          "id": "organize",
+          "options": {
+            "excludeByName": {
+              "Time": true
+            },
+            "renameByName": {
+              "Value #A": "Requests",
+              "Value #B": "Tokens",
+              "Value #C": "Refused by a limit"
+            }
+          }
+        }
+      ]
+    }
+  ]
+}

deploy/monitoring/llmgw-rules.yml

@@ -350,3 +350,130 @@ groups:
             Working as designed when the cluster is full: the lowest band is
             held first. Worth knowing about because the callers being refused
             do not know why.
+
+  # Per tenant: the level router.tenant_label names (a team by default), and
+  # every organization, team, project and person given a budget.
+  - name: llmgw-tenants
+    rules:
+      - alert: BudgetMostlySpent
+        expr: llmgw_budget_spent_ratio >= 0.8 and llmgw_budget_spent_ratio < 1
+        for: 5m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.scope }} {{ $labels.name }} has spent {{ $value | humanizePercentage }} of its {{ $labels.period }}ly budget"
+          description: >-
+            Its requests are refused once it reaches all of it, until the
+            period turns. Raise it on the people page, or tell them now
+            rather than when they are refused.
+
+      - alert: BudgetSpent
+        expr: llmgw_budget_spent_ratio >= 1
+        for: 1m
+        labels: { severity: critical }
+        annotations:
+          summary: "{{ $labels.scope }} {{ $labels.name }} has spent its {{ $labels.period }}ly budget"
+          description: >-
+            Every request under it is refused with 429 until the period
+            turns, unless a cheaper model is named as its fallback.
+
+      - alert: TenantHittingLimits
+        expr: >-
+          sum by (tenant) (rate(llmgw_tenant_requests_total{outcome="limited"}[10m]))
+          / sum by (tenant) (rate(llmgw_tenant_requests_total[10m])) > 0.1
+        for: 10m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $value | humanizePercentage }} of {{ $labels.tenant }}'s requests are refused by a limit"
+          description: >-
+            A rate, token or budget limit is turning their traffic away. Either
+            the limit is too low for what they now do, or something of theirs
+            is looping.
+
+      - alert: TenantErrors
+        expr: >-
+          sum by (tenant) (rate(llmgw_tenant_requests_total{outcome="error"}[10m]))
+          / sum by (tenant) (rate(llmgw_tenant_requests_total[10m])) > 0.05
+          and sum by (tenant) (rate(llmgw_tenant_requests_total[10m])) > 0.05
+        for: 10m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $value | humanizePercentage }} of {{ $labels.tenant }}'s requests fail"
+          description: >-
+            Not counting refusals by a limit. If every tenant fails alike it
+            is a model (ModelFailing); one alone is usually what they send —
+            prompts past the context, a model they may not use.
+
+      - alert: TenantTrafficSpike
+        expr: >-
+          sum by (tenant) (rate(llmgw_tenant_requests_total[30m]))
+          > 3 * sum by (tenant) (rate(llmgw_tenant_requests_total[7d]))
+          and sum by (tenant) (rate(llmgw_tenant_requests_total[30m])) > 0.2
+        for: 15m
+        labels: { severity: info }
+        annotations:
+          summary: "{{ $labels.tenant }} is sending three times its weekly average"
+          description: >-
+            A new workload, a batch job, or a client retrying in a loop. Only
+            for a tenant sending more than one request every five seconds, so
+            a quiet team starting work in the morning does not page anybody.
+
+      - alert: TenantSlowFirstToken
+        expr: >-
+          histogram_quantile(0.95,
+            sum by (tenant, le) (rate(llmgw_tenant_ttft_seconds_bucket[15m]))) > 10
+          and sum by (tenant) (rate(llmgw_tenant_ttft_seconds_count[15m])) > 0.02
+        for: 15m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.tenant }} waits {{ $value | humanizeDuration }} for a first token at p95"
+          description: >-
+            What their people feel. Compare with SlowFirstToken per model: if
+            only this tenant is slow, it is the models it uses or its queue
+            band, not the cluster.
+  # The store, when it is a replicated cluster under Patroni (llmgw server
+  # init --patroni). Published only for a store of several hosts.
+  - name: llmgw-store
+    rules:
+      - alert: StoreHasNoLeader
+        expr: llmgw_store_patroni_up == 1 and on() count(llmgw_store_member{role="leader"}) == 0
+        for: 1m
+        labels: { severity: critical }
+        annotations:
+          summary: "The store's cluster has no primary"
+          description: >-
+            Every write fails — keys, usage, budgets, leadership. Patroni
+            promotes a standby within its 30-second TTL; a minute without one
+            means it cannot: etcd has lost its majority, or no standby is fit.
+            `patronictl -c /etc/patroni/config.yml list` on a member says which.
+
+      - alert: StorePatroniUnreachable
+        expr: llmgw_store_patroni_up == 0
+        for: 2m
+        labels: { severity: warning }
+        annotations:
+          summary: "No member of the store's cluster answers on its Patroni port"
+          description: >-
+            The control plane reads the cluster's state from port 8008 on each
+            member. The database may still be serving; nothing here can tell.
+
+      - alert: StoreReplicaNotStreaming
+        expr: llmgw_store_member{role!="leader", state!="streaming"} == 1
+        for: 5m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.member }} is {{ $labels.state }}, not streaming from the primary"
+          description: >-
+            A standby that is not streaming is not a standby: if the primary
+            is lost now, there is nothing to promote. "missing" is a host of
+            the store address the cluster no longer lists — usually a machine
+            that is down.
+
+      - alert: StoreReplicaBehind
+        expr: llmgw_store_replica_lag_bytes > 64 * 1024 * 1024 or llmgw_store_replica_lag_bytes == -1
+        for: 10m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.member }} is {{ $value | humanize1024 }}B behind the primary"
+          description: >-
+            Patroni will not promote a standby more than 1 MB behind, so a
+            failover now would find none fit. -1 is a lag Patroni cannot read.

docs/install.md

@@ -87,7 +87,24 @@ CGO_ENABLED=0 go build -o llmgw ./cmd/llmgw
 
 ## 2. The control plane
 
-On the machine that will be the control plane, as root:
+**The short way is the screens.** On the machine that will be the control
+plane, run `sudo llmgw` with nothing after it. With nothing installed it asks
+whether this is one machine or three for high availability; for three, it asks
+for the other two — address, user, and an ssh key, the ssh agent, or a
+password, and the sudo password if sudo asks for one; or, for a machine
+nothing may log into, a join code to run there with `sudo llmgw join <code>`,
+which dials out to this one on port 7444 — checks all three before
+installing anything (system, room, clock, ports, internet, whether they reach
+one another), shows the plan, and installs. Passwords are held in memory for
+the run only; a sudo password lets the user sudo without one until the install
+finishes, and is then taken back. At the end it shows the administrator's
+password and the command that does the same from a script.
+
+Run again later, `sudo llmgw` is the console: the cluster's state, refreshed,
+and what to do next — add a node, replicate the store. A setup that stopped
+halfway opens the setup screens again, which finish it.
+
+The rest of this section is the same by command, as root:
 
 ```
 sudo llmgw server init --admin you@example.com
@@ -121,6 +138,20 @@ Add `--postgres` to keep accounts and keys in PostgreSQL rather than SQLite:
 init installs it and sets it up, and on a machine already running on SQLite it
 moves everything across. See [PostgreSQL](#postgresql-when-sqlite-is-no-longer-the-right-shape).
 
+Or `--patroni`, given two machines (or more), to keep it in PostgreSQL
+replicated across them, so losing one loses nothing:
+
+```
+sudo --preserve-env=SSH_AUTH_SOCK llmgw server init --patroni ubuntu@10.0.0.149 --patroni ubuntu@10.0.0.150
+```
+
+Over ssh with passwordless sudo, as `node add` works, it installs PostgreSQL,
+Patroni and etcd there — this machine is etcd's third vote — and moves the
+store already in use, SQLite or PostgreSQL, into the cluster. A standby is
+promoted within about thirty seconds of the primary being lost, and the
+control plane follows it by itself. `llmgw ctl status` says who leads; see
+[deploy/ha](../deploy/ha/README.md) for what it sets up.
+
 Every step looks before it acts. Running it again finishes an install that
 stopped halfway and changes nothing on one that did not; it never overwrites
 `server.yaml`, the authority, or an existing account's password.
@@ -1271,7 +1302,7 @@ requests. On a slow model, set `timeouts.generation` higher.
   can see each other, seeding between them is the obvious next thing to build.
 - **A budget without a price never binds.** Set `default_price` or `pricing`
   (step 2) or every request costs zero, and a budget is something the dashboard
-  displays rather than something that stops anything. The quotas page says so
+  displays rather than something that stops anything. The Limits tab says so
   where it can, which is not the same as being told before you rely on it.
 - **A V100 runs vLLM 0.19.1**, not the version the rest of the cluster runs:
   no build from 0.20 on has Volta kernels, and the node picks the newest that

docs/onboarding.md

@@ -191,7 +191,8 @@ project or person, the teams and projects with their limits and spend, and every
 member. Only what was spent in that organization's projects is on it; what its
 people do elsewhere is not. The page is read-only — a manager is not an
 administrator of the cluster, and limits and memberships stay with those who
-are.
+are. An administrator following `/org` lands on the same organization's panel
+on the people page instead, where all of that can be changed.
 
 `llmgw keys init` still works for provisioning scripts and is idempotent, but it
 will no longer move a team or project that already exists under another parent;
@@ -259,7 +260,7 @@ Allowlists compose by intersection: a key cannot add a model its team withheld.
 A model's canonical name and any of its aliases name the same thing, so either
 form works.
 
-Budgets are given in whole units, as the quotas page shows them, and what a unit
+Budgets are given in whole units, as the Limits tab shows them, and what a unit
 means is yours to decide. Self-hosted models have no per-token price, so it can
 be GPU-seconds, a notional rate, or one unit per thousand tokens everywhere —
 a price of 1 000 000 in `auth.pricing`, which is in micro-units — and a budget
@@ -409,10 +410,19 @@ page exists.
 opening onto its members and its keys. From there an admin approves people
 (into a project or on their own), adds them to and takes them out of
 organizations, teams and projects, suspends and removes them, revokes keys, and
-adds, renames, moves and removes organizations, teams and projects. `/admin/quotas`
-sets limits on the tree, per member of any part of it, and per person. A searchable list of every account is at the
-bottom. The page works as plain forms; with scripts on, a change updates it in
-place and the projects you had open stay open.
+adds, renames, moves and removes organizations, teams and projects. A
+searchable list of every account is at the bottom. Choosing any organization,
+team, project or person opens a panel beside the tree with its limits, its
+capacity quota and its use over the last week, and its limits can be changed
+there.
+
+The same page has two more tabs, each at the address it had as a page of its
+own: **Limits** (`/admin/quotas`) is every limit in the tree at once — per
+branch, per member of any part of it, and per person — with the capacity quotas;
+**Usage** (`/admin/usage`) is the cluster's use over time by organization, team,
+project or person, and the GPU time each owner held. The page works as plain
+forms; with scripts on, a change updates it in place and the projects you had
+open stay open.
 
 Three things are deliberately not there:
 

docs/reference.md

@@ -39,31 +39,10 @@ rather than repeated.
 | `flush_interval` | `time.Duration` | [Auth](#auth) |
 | `batch_size` | `int` | [Auth](#auth) |
 | `buffer_size` | `int` | [Auth](#auth) |
-| `name` | `string` | [Band](#band) |
-| `priority` | `int` | [Band](#band) |
-| `saturation_ceiling` | `float64` | [Band](#band) |
+| `usage_retention` | `time.Duration` | [Auth](#auth) |
 | `tools` | `bool` | [Capabilities](#capabilities) |
 | `vision` | `bool` | [Capabilities](#capabilities) |
 | `embeddings` | `bool` | [Capabilities](#capabilities) |
-| `node_id` | `string` | [Config](#config) |
-| `server` | `string` | [Config](#config) |
-| `servers` | `[]string` | [Config](#config) |
-| `hostname` | `string` | [Config](#config) |
-| `mgmt_addr` | `string` | [Config](#config) |
-| `labels` | `map[string]string` | [Config](#config) |
-| `mgmt_interface` | `string` | [Config](#config) |
-| `fabrics` | `[]inventory.FabricAttachment` | [Config](#config) |
-| `heartbeat_interval` | `time.Duration` | [Config](#config) |
-| `reconnect_base` | `time.Duration` | [Config](#config) |
-| `reconnect_max` | `time.Duration` | [Config](#config) |
-| `adopt_ready_timeout` | `time.Duration` | [Config](#config) |
-| `tls` | `tlsconf.Config` | [Config](#config) |
-| `model_dirs` | `[]string` | [Config](#config) |
-| `engines_dir` | `string` | [Config](#config) |
-| `weights_cache` | `string` | [Config](#config) |
-| `discovery` | `discovery.Options` | [Config](#config) |
-| `engines` | `engines.Config` | [Config](#config) |
-| `supervisor` | `supervisor.Config` | [Config](#config) |
 | `interval` | `time.Duration` | [Control](#control) |
 | `base_backoff` | `time.Duration` | [Control](#control) |
 | `max_backoff` | `time.Duration` | [Control](#control) |
@@ -132,8 +111,6 @@ rather than repeated.
 | `match_ratio` | `float64` | [Prefix](#prefix) |
 | `balance_abs` | `float64` | [Prefix](#prefix) |
 | `balance_rel` | `float64` | [Prefix](#prefix) |
-| `prompt_per_1k` | `int64` | [Price](#price) |
-| `completion_per_1k` | `int64` | [Price](#price) |
 | `name` | `string` | [Profile](#profile) |
 | `engine` | `string` | [Profile](#profile) |
 | `gpu_class` | `string` | [Profile](#profile) |
@@ -161,6 +138,7 @@ rather than repeated.
 | `prefix` | `Prefix` | [Router](#router) |
 | `session_header` | `string` | [Router](#router) |
 | `inject_usage` | `bool` | [Router](#router) |
+| `tenant_label` | `string` | [Router](#router) |
 | `max_body_bytes` | `int64` | [Router](#router) |
 | `timeouts` | `Timeouts` | [Router](#router) |
 | `scoring` | `Scoring` | [Router](#router) |
@@ -208,6 +186,56 @@ rather than repeated.
 | `bin` | `string` | [VLLMConfig](#vllmconfig) |
 | `launch_mode` | `string` | [VLLMConfig](#vllmconfig) |
 | `ray_bin` | `string` | [VLLMConfig](#vllmconfig) |
+| `prompt_per_1k` | `int64` | [accounting.Price](#accountingprice) |
+| `completion_per_1k` | `int64` | [accounting.Price](#accountingprice) |
+| `name` | `string` | [admission.Band](#admissionband) |
+| `priority` | `int` | [admission.Band](#admissionband) |
+| `saturation_ceiling` | `float64` | [admission.Band](#admissionband) |
+| `bands` | `[]Band` | [admission.Config](#admissionconfig) |
+| `default_band` | `string` | [admission.Config](#admissionconfig) |
+| `queue_ttl` | `time.Duration` | [admission.Config](#admissionconfig) |
+| `queue_capacity_per_band` | `int` | [admission.Config](#admissionconfig) |
+| `queue_capacity_total` | `int` | [admission.Config](#admissionconfig) |
+| `interval` | `time.Duration` | [admission.Config](#admissionconfig) |
+| `node_id` | `string` | [agent.Config](#agentconfig) |
+| `server` | `string` | [agent.Config](#agentconfig) |
+| `servers` | `[]string` | [agent.Config](#agentconfig) |
+| `hostname` | `string` | [agent.Config](#agentconfig) |
+| `mgmt_addr` | `string` | [agent.Config](#agentconfig) |
+| `labels` | `map[string]string` | [agent.Config](#agentconfig) |
+| `mgmt_interface` | `string` | [agent.Config](#agentconfig) |
+| `fabrics` | `[]inventory.FabricAttachment` | [agent.Config](#agentconfig) |
+| `heartbeat_interval` | `time.Duration` | [agent.Config](#agentconfig) |
+| `reconnect_base` | `time.Duration` | [agent.Config](#agentconfig) |
+| `reconnect_max` | `time.Duration` | [agent.Config](#agentconfig) |
+| `adopt_ready_timeout` | `time.Duration` | [agent.Config](#agentconfig) |
+| `tls` | `tlsconf.Config` | [agent.Config](#agentconfig) |
+| `model_dirs` | `[]string` | [agent.Config](#agentconfig) |
+| `engines_dir` | `string` | [agent.Config](#agentconfig) |
+| `weights_cache` | `string` | [agent.Config](#agentconfig) |
+| `discovery` | `discovery.Options` | [agent.Config](#agentconfig) |
+| `engines` | `engines.Config` | [agent.Config](#agentconfig) |
+| `supervisor` | `supervisor.Config` | [agent.Config](#agentconfig) |
+| `name` | `string` | [config.Band](#configband) |
+| `priority` | `int` | [config.Band](#configband) |
+| `saturation_ceiling` | `float64` | [config.Band](#configband) |
+| `prompt_per_1k` | `int64` | [config.Price](#configprice) |
+| `completion_per_1k` | `int64` | [config.Price](#configprice) |
+| `disabled` | `[]string` | [engines.Config](#enginesconfig) |
+| `vllm` | `VLLMConfig` | [engines.Config](#enginesconfig) |
+| `sglang` | `SGLangConfig` | [engines.Config](#enginesconfig) |
+| `llamacpp` | `LlamaCppConfig` | [engines.Config](#enginesconfig) |
+| `sim` | `SimConfig` | [engines.Config](#enginesconfig) |
+| `work_dir` | `string` | [supervisor.Config](#supervisorconfig) |
+| `log_lines` | `int` | [supervisor.Config](#supervisorconfig) |
+| `stop_timeout` | `time.Duration` | [supervisor.Config](#supervisorconfig) |
+| `health_interval` | `time.Duration` | [supervisor.Config](#supervisorconfig) |
+| `rendezvous_timeout` | `time.Duration` | [supervisor.Config](#supervisorconfig) |
+| `alive_grace` | `time.Duration` | [supervisor.Config](#supervisorconfig) |
+| `ca` | `string` | [tlsconf.Config](#tlsconfconfig) |
+| `cert` | `string` | [tlsconf.Config](#tlsconfconfig) |
+| `key` | `string` | [tlsconf.Config](#tlsconfconfig) |
+| `insecure` | `bool` | [tlsconf.Config](#tlsconfconfig) |
 
 ## Adapter
 
@@ -347,7 +375,7 @@ table costs whatever DefaultPrice says, which is nothing unless set.
 
 `Price`
 
-Takes a [Price](#price) block.
+Takes a [config.Price](#configprice) block.
 
 ### `flush_interval`
 
@@ -367,19 +395,16 @@ BufferSize bounds the queue. Beyond it records are dropped rather than
 blocking a request: losing accounting is bad, turning a stalled database
 into an outage is worse.
 
-## Band
+### `usage_retention`
 
-### `name`
+`time.Duration`
 
-`string`
-
-### `priority`
-
-`int`
-
-### `saturation_ceiling`
-
-`float64`
+UsageRetention is how long the record of each request is kept: who sent
+it, which replica answered, how long it queued — what `ctl request`
+reads. Zero keeps it for ever. Totals, charts and budgets are read from
+five-minute sums kept beside the records, which are not pruned, so this
+bounds the table that grows by a row a request without changing any
+figure a page shows. At least a day when set.
 
 ## Capabilities
 
@@ -395,152 +420,6 @@ into an outage is worse.
 
 `bool`
 
-## Config
-
-### `node_id`
-
-`string`
-
-NodeID defaults to the hostname. It must equal the common name of the
-agent's certificate, so set it only where the two differ.
-
-### `server`
-
-`string`
-
-Server is the control plane, and Servers is the same thing when there is
-more than one. Both are accepted: every configuration written before
-there could be a second control plane names one address, and none of
-them should have to change.
-
-The order is tried in order, and a refusal moves to the next. Only the
-leader accepts an agent, so a standby is skipped in the time it takes to
-be told so — which is what makes failover a reconnect rather than an
-edit to every node's configuration.
-
-### `servers`
-
-`[]string`
-
-### `hostname`
-
-`string`
-
-Hostname defaults to the machine's.
-
-### `mgmt_addr`
-
-`string`
-
-MgmtAddr is the address engines are reached on. It defaults to this
-machine's address on the route to the control plane, which is the one
-the control plane and the router can reach.
-
-### `labels`
-
-`map[string]string`
-
-Labels are free-form facts about a node for a profile's node_selector.
-Nothing needs them: memory, GPUs, engines and disk are measured, and a
-label restating one of those goes stale the day the machine changes.
-
-### `mgmt_interface`
-
-`string`
-
-MgmtInterface is the ethernet interface NCCL bootstraps over. It
-defaults to the interface holding MgmtAddr, then eth0.
-
-### `fabrics`
-
-`[]inventory.FabricAttachment`
-
-Fabrics labels this node's HCAs. It is explicit on purpose: OpenSM's
-default subnet prefix is the same on every fabric, so nothing can be
-inferred from the wire (plan §5.2).
-
-### `heartbeat_interval`
-
-`time.Duration`
-
-### `reconnect_base`
-
-`time.Duration`
-
-ReconnectBase is the first reconnect delay; it backs off from there.
-
-### `reconnect_max`
-
-`time.Duration`
-
-### `adopt_ready_timeout`
-
-`time.Duration`
-
-AdoptReadyTimeout bounds how long an adopted engine is given to answer a
-health check before the agent stops waiting on it.
-
-Shorter than a startup timeout on purpose: this engine is already
-running, so it is not loading a model, it is either serving or it is not.
-
-### `tls`
-
-`tlsconf.Config`
-
-TLS authenticates both ends of the control-plane stream. Leaving it unset
-keeps the plaintext behaviour, which only belongs on a trusted network.
-
-Takes a [Config](#config) block.
-
-### `model_dirs`
-
-`[]string`
-
-ModelDirs are the directories this node keeps model weights in, beyond
-WeightsCache, which is always looked in. Storage differs at every installation,
-so llmgw observes it rather than owning it: a Lustre mount, an image
-exported over the fabric, local NVMe and a copy made by hand all look
-identical from here, and placement only needs to know whether the file
-is present.
-
-### `engines_dir`
-
-`string`
-
-EnginesDir is where the agent installs engines, one directory per
-build. Defaults to /opt/llmgw/engines.
-
-### `weights_cache`
-
-`string`
-
-WeightsCache is where a remote source is downloaded to. Defaults to the
-first entry of ModelDirs, because a node that says where its models live
-has already answered this question, and otherwise to
-/var/lib/llmgw-weights.
-
-Fetching is the simple arrangement: every node keeps its own copy. The
-exotic ones need no code — weights on a shared filer or an image exported
-over the fabric are simply present, and present means nothing is fetched.
-
-### `discovery`
-
-`discovery.Options`
-
-Takes a [Options](#options) block.
-
-### `engines`
-
-`engines.Config`
-
-Takes a [Config](#config) block.
-
-### `supervisor`
-
-`supervisor.Config`
-
-Takes a [Config](#config) block.
-
 ## Control
 
 ### `interval`
@@ -769,7 +648,7 @@ agents use. Empty leaves the connection in plain HTTP, which is only
 acceptable on a network where nothing untrusted can reach that port —
 the mirror says as much in its own log when it starts that way.
 
-Takes a [Config](#config) block.
+Takes a [tlsconf.Config](#tlsconfconfig) block.
 
 ### `registry`
 
@@ -1031,16 +910,6 @@ before affinity is abandoned.
 
 `float64`
 
-## Price
-
-### `prompt_per_1k`
-
-`int64`
-
-### `completion_per_1k`
-
-`int64`
-
 ## Profile
 
 ### `name`
@@ -1242,6 +1111,16 @@ SessionHeader carries the session id for affinity (plan §7.5 tier 0).
 InjectUsage asks engines for usage even when the client did not, so token
 accounting works regardless of client behaviour.
 
+### `tenant_label`
+
+`string`
+
+TenantLabel is the level of the tree the per-tenant request series are
+labelled by — llmgw_tenant_requests_total and its siblings: "org",
+"team" or "project", or "none" for no such series. A label per project
+on a cluster of thousands is more series than Prometheus wants, and a
+team is what an alert is usually about; "team" is the default.
+
 ### `max_body_bytes`
 
 `int64`
@@ -1402,7 +1281,7 @@ AgentTLS authenticates the fleet-facing port. It deliberately does not
 cover the client-facing one: that surface is fronted by a publicly
 trusted certificate, while this one is mutual and private.
 
-Takes a [Config](#config) block.
+Takes a [tlsconf.Config](#tlsconfconfig) block.
 
 ### `monitoring`
 
@@ -1594,3 +1473,345 @@ node and has not been checked against a real cluster.
 
 RayBin is used when LaunchMode is "ray".
 
+## accounting.Price
+
+### `prompt_per_1k`
+
+`int64`
+
+### `completion_per_1k`
+
+`int64`
+
+## admission.Band
+
+### `name`
+
+`string`
+
+### `priority`
+
+`int`
+
+Priority orders bands; higher is served first.
+
+### `saturation_ceiling`
+
+`float64`
+
+SaturationCeiling is the pool saturation above which this band waits.
+1.0 means "never hold this band back".
+
+## admission.Config
+
+### `bands`
+
+`[]Band`
+
+### `default_band`
+
+`string`
+
+### `queue_ttl`
+
+`time.Duration`
+
+QueueTTL drops a request that has waited this long. Sending the GPU work
+the client has already given up on helps nobody.
+
+### `queue_capacity_per_band`
+
+`int`
+
+QueueCapacityPerBand bounds each band's queue. Rejecting immediately
+beats accumulating silently.
+
+### `queue_capacity_total`
+
+`int`
+
+QueueCapacityTotal bounds everything queued, across bands. When it is
+reached, a request is not refused while a lower band has requests
+queued: the newest of the lowest band is turned away instead
+(displacement), since the oldest have waited longest and are nearest
+their TTL. Zero takes half of every band's capacity together.
+
+### `interval`
+
+`time.Duration`
+
+Interval is how often the dispatcher re-checks saturation. Releases also
+wake it, so this is only the backstop for saturation falling on its own.
+
+## agent.Config
+
+### `node_id`
+
+`string`
+
+NodeID defaults to the hostname. It must equal the common name of the
+agent's certificate, so set it only where the two differ.
+
+### `server`
+
+`string`
+
+Server is the control plane, and Servers is the same thing when there is
+more than one. Both are accepted: every configuration written before
+there could be a second control plane names one address, and none of
+them should have to change.
+
+The order is tried in order, and a refusal moves to the next. Only the
+leader accepts an agent, so a standby is skipped in the time it takes to
+be told so — which is what makes failover a reconnect rather than an
+edit to every node's configuration.
+
+### `servers`
+
+`[]string`
+
+### `hostname`
+
+`string`
+
+Hostname defaults to the machine's.
+
+### `mgmt_addr`
+
+`string`
+
+MgmtAddr is the address engines are reached on. It defaults to this
+machine's address on the route to the control plane, which is the one
+the control plane and the router can reach.
+
+### `labels`
+
+`map[string]string`
+
+Labels are free-form facts about a node for a profile's node_selector.
+Nothing needs them: memory, GPUs, engines and disk are measured, and a
+label restating one of those goes stale the day the machine changes.
+
+### `mgmt_interface`
+
+`string`
+
+MgmtInterface is the ethernet interface NCCL bootstraps over. It
+defaults to the interface holding MgmtAddr, then eth0.
+
+### `fabrics`
+
+`[]inventory.FabricAttachment`
+
+Fabrics labels this node's HCAs. It is explicit on purpose: OpenSM's
+default subnet prefix is the same on every fabric, so nothing can be
+inferred from the wire (plan §5.2).
+
+### `heartbeat_interval`
+
+`time.Duration`
+
+### `reconnect_base`
+
+`time.Duration`
+
+ReconnectBase is the first reconnect delay; it backs off from there.
+
+### `reconnect_max`
+
+`time.Duration`
+
+### `adopt_ready_timeout`
+
+`time.Duration`
+
+AdoptReadyTimeout bounds how long an adopted engine is given to answer a
+health check before the agent stops waiting on it.
+
+Shorter than a startup timeout on purpose: this engine is already
+running, so it is not loading a model, it is either serving or it is not.
+
+### `tls`
+
+`tlsconf.Config`
+
+TLS authenticates both ends of the control-plane stream. Leaving it unset
+keeps the plaintext behaviour, which only belongs on a trusted network.
+
+Takes a [tlsconf.Config](#tlsconfconfig) block.
+
+### `model_dirs`
+
+`[]string`
+
+ModelDirs are the directories this node keeps model weights in, beyond
+WeightsCache, which is always looked in. Storage differs at every installation,
+so llmgw observes it rather than owning it: a Lustre mount, an image
+exported over the fabric, local NVMe and a copy made by hand all look
+identical from here, and placement only needs to know whether the file
+is present.
+
+### `engines_dir`
+
+`string`
+
+EnginesDir is where the agent installs engines, one directory per
+build. Defaults to /opt/llmgw/engines.
+
+### `weights_cache`
+
+`string`
+
+WeightsCache is where a remote source is downloaded to. Defaults to the
+first entry of ModelDirs, because a node that says where its models live
+has already answered this question, and otherwise to
+/var/lib/llmgw-weights.
+
+Fetching is the simple arrangement: every node keeps its own copy. The
+exotic ones need no code — weights on a shared filer or an image exported
+over the fabric are simply present, and present means nothing is fetched.
+
+### `discovery`
+
+`discovery.Options`
+
+Takes a [Options](#options) block.
+
+### `engines`
+
+`engines.Config`
+
+Takes a [engines.Config](#enginesconfig) block.
+
+### `supervisor`
+
+`supervisor.Config`
+
+Takes a [supervisor.Config](#supervisorconfig) block.
+
+## config.Band
+
+### `name`
+
+`string`
+
+### `priority`
+
+`int`
+
+### `saturation_ceiling`
+
+`float64`
+
+## config.Price
+
+### `prompt_per_1k`
+
+`int64`
+
+### `completion_per_1k`
+
+`int64`
+
+## engines.Config
+
+### `disabled`
+
+`[]string`
+
+Disabled are engines this node never installs or advertises, whatever
+the control plane asks: vLLM on a node of cards no build of it runs on,
+that is meant for llama.cpp alone.
+
+### `vllm`
+
+`VLLMConfig`
+
+Takes a [VLLMConfig](#vllmconfig) block.
+
+### `sglang`
+
+`SGLangConfig`
+
+Takes a [SGLangConfig](#sglangconfig) block.
+
+### `llamacpp`
+
+`LlamaCppConfig`
+
+Takes a [LlamaCppConfig](#llamacppconfig) block.
+
+### `sim`
+
+`SimConfig`
+
+Takes a [SimConfig](#simconfig) block.
+
+## supervisor.Config
+
+### `work_dir`
+
+`string`
+
+WorkDir holds pid files and per-shard scratch space.
+
+### `log_lines`
+
+`int`
+
+LogLines is how many lines of engine output to retain per shard.
+
+### `stop_timeout`
+
+`time.Duration`
+
+StopTimeout is how long a process gets to exit after SIGTERM before it is
+killed. Engines need time to free GPU memory cleanly.
+
+### `health_interval`
+
+`time.Duration`
+
+HealthInterval is how often a starting shard is polled.
+
+### `rendezvous_timeout`
+
+`time.Duration`
+
+RendezvousTimeout bounds the wait for a multi-node rendezvous point to
+start listening.
+
+### `alive_grace`
+
+`time.Duration`
+
+AliveGrace is how long a non-serving rank must stay up before it counts
+as ready. There is no external signal for a worker, so staying alive is
+the only evidence available.
+
+## tlsconf.Config
+
+### `ca`
+
+`string`
+
+CA is the bundle used to verify the other side. In a real deployment this
+is whichever authority issued the fleet's certificates.
+
+### `cert`
+
+`string`
+
+Cert and Key are this component's own identity.
+
+### `key`
+
+`string`
+
+### `insecure`
+
+`bool`
+
+Insecure disables TLS entirely. It exists for local runs against
+sim-engines; anything using it must stay on a trusted network.
+

go.mod

@@ -23,6 +23,7 @@ require (
 	go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.46.0
 	go.opentelemetry.io/otel/sdk v1.46.0
 	go.opentelemetry.io/otel/trace v1.46.0
+	go.yaml.in/yaml/v3 v3.0.5
 	golang.org/x/crypto v0.57.0
 	golang.org/x/net v0.59.0
 	golang.org/x/sync v0.23.0
@@ -54,6 +55,7 @@ require (
 	github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
 	github.com/jackc/puddle/v2 v2.2.2 // indirect
 	github.com/knadh/koanf/maps v0.1.2 // indirect
+	github.com/kylelemons/godebug v1.1.0 // indirect
 	github.com/lucasb-eyer/go-colorful v1.3.0 // indirect
 	github.com/mattn/go-localereader v0.0.1 // indirect
 	github.com/mattn/go-runewidth v0.0.19 // indirect
@@ -73,7 +75,6 @@ require (
 	go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.46.0 // indirect
 	go.opentelemetry.io/otel/metric v1.46.0 // indirect
 	go.opentelemetry.io/proto/otlp v1.11.0 // indirect
-	go.yaml.in/yaml/v3 v3.0.5 // indirect
 	golang.org/x/sys v0.48.0 // indirect
 	golang.org/x/text v0.42.0 // indirect
 	google.golang.org/genproto/googleapis/api v0.0.0-20260819154853-08b0e4226688 // indirect

go.sum

@@ -166,6 +166,8 @@ golang.org/x/sync v0.23.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0=
 golang.org/x/sys v0.0.0-20210809222454-d867a43fc93e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
 golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
+golang.org/x/term v0.46.0 h1:3+OXuTbaKDgwk8jTi3aSLHRlmWqHEUDUtxnbFigO4YE=
+golang.org/x/term v0.46.0/go.mod h1:+K02xbkittuwc0Am4abfA3Fc+XRGXkvBXNO88NCXPoc=
 golang.org/x/text v0.42.0 h1:JbOZXgfeCPU9gacVtYliJqOhD+zhrEqK4LfdpmlUZqI=
 golang.org/x/text v0.42.0/go.mod h1:ojzP1Z+2QtioaF8DTtO8K5q7JWVVYwZKenzujK0Zd0E=
 golang.org/x/tools v0.49.0 h1:3NI7VXzL9+1WZD52Dx2ttoPwD5DWrFGpl9mFZDlmisI=

internal/accounting/accounting.go

@@ -316,6 +316,17 @@ func (a *Accountant) Record(res Reservation, u store.Usage) {
 	}
 }
 
+// RecordRefused queues the row of a request a limit refused: nothing is
+// charged — it used nothing — but it is counted, so a team running into its
+// limits shows as one rather than as a team that went quiet.
+func (a *Accountant) RecordRefused(u store.Usage) {
+	if u.KeyID == "" || a.recorder == nil {
+		return
+	}
+	u.PromptTokens, u.CompletionTokens, u.CostMicros = 0, 0, 0
+	a.recorder.Record(u)
+}
+
 // Status returns what a key has used, for headers and `ctl`.
 func (a *Accountant) Status(p auth.Principal) (spent int64, rpm, tpm float64) {
 	if p.KeyID == "" {

internal/agent/agent.go

@@ -73,8 +73,11 @@ type Agent struct {
 	lastEngineProbe time.Time
 	probingEngines  bool
 
-	mu  sync.Mutex
-	inv discovery.Inventory
+	mu sync.Mutex
+	// serverList is every control plane to dial: the configured ones, and
+	// any the leader has named since. Read through servers().
+	serverList []string
+	inv        discovery.Inventory
 	// shardReplica remembers which replica a shard belongs to, so a status
 	// report can name both. The control plane keys on the replica.
 	shardReplica map[string]string
@@ -153,6 +156,7 @@ func New(cfg Config, httpClient connectHTTPClient, log *slog.Logger) (*Agent, er
 		load:         discovery.NewLoadSampler(),
 		sensors:      discovery.NewSensorSampler(),
 		wear:         discovery.NewHealthSampler(),
+		serverList:   loadControlPlanes(cfg.Servers, cfg.TLS.Enabled()),
 	}
 	a.prov = &provision.Installer{Root: cfg.EnginesDir, Log: log, OnStatus: a.provisionStatus}
 	return a, nil
@@ -185,7 +189,9 @@ func (a *Agent) Run(ctx context.Context) error {
 	attempt := 0
 	next := 0
 	for ctx.Err() == nil {
-		server := a.cfg.Servers[next%len(a.cfg.Servers)]
+		// Read each time round: the leader may have named another since.
+		servers := a.servers()
+		server := servers[next%len(servers)]
 		err := a.session(ctx, server)
 		if ctx.Err() != nil {
 			break
@@ -200,12 +206,12 @@ func (a *Agent) Run(ctx context.Context) error {
 		// No wait when there is somewhere else to try: the delay exists to
 		// stop hammering one control plane, and the next address has not been
 		// hammered. With a single address this is the old behaviour exactly.
-		if len(a.cfg.Servers) > 1 && next%len(a.cfg.Servers) != 0 {
+		if len(servers) > 1 && next%len(servers) != 0 {
 			delay = 0
 		}
 		a.log.Warn("disconnected from the control plane, retrying",
 			"err", err, "server", server,
-			"next", a.cfg.Servers[next%len(a.cfg.Servers)],
+			"next", servers[next%len(servers)],
 			"in", delay.Round(time.Millisecond), "attempt", attempt)
 
 		select {
@@ -792,6 +798,8 @@ func (a *Agent) handle(ctx context.Context, msg *llmgwv1.ControlMessage, out cha
 		a.setProvision(m.Provision)
 	case *llmgwv1.ControlMessage_Upgrade:
 		go a.upgrade(ctx, m.Upgrade, out)
+	case *llmgwv1.ControlMessage_ControlPlanes:
+		a.setControlPlanes(m.ControlPlanes)
 	case *llmgwv1.ControlMessage_Logs:
 		reply := a.logsReply(m.Logs)
 		// Off the receive loop: a full send queue must not stop commands being
@@ -1084,7 +1092,9 @@ func Client(cfg tlsconf.Config, timeout time.Duration) (*http.Client, error) {
 	if !cfg.Enabled() {
 		return &http.Client{
 			Transport: &http2.Transport{
-				AllowHTTP: true,
+				AllowHTTP:       true,
+				ReadIdleTimeout: streamIdle,
+				PingTimeout:     streamPing,
 				DialTLSContext: func(ctx context.Context, network, addr string, _ *tls.Config) (net.Conn, error) {
 					return (&net.Dialer{Timeout: timeout}).DialContext(ctx, network, addr)
 				},
@@ -1099,9 +1109,31 @@ func Client(cfg tlsconf.Config, timeout time.Duration) (*http.Client, error) {
 	if err != nil {
 		return nil, err
 	}
-	return &http.Client{Transport: &http2.Transport{TLSClientConfig: tc}}, nil
+	return &http.Client{Transport: &http2.Transport{
+		TLSClientConfig: tc,
+		ReadIdleTimeout: streamIdle,
+		PingTimeout:     streamPing,
+		// Bounded, as the plaintext branch's is: dialling a control plane
+		// whose machine is off waits out the kernel's connect retries,
+		// about two minutes, before the next address is tried.
+		DialTLSContext: func(ctx context.Context, network, addr string, c *tls.Config) (net.Conn, error) {
+			d := tls.Dialer{NetDialer: &net.Dialer{Timeout: timeout}, Config: c}
+			return d.DialContext(ctx, network, addr)
+		},
+	}}, nil
 }
 
+// A stream to a control plane whose machine stopped dead — powered off, cut
+// off — is never closed: nothing sends the FIN, and the agent's heartbeats sit
+// in the socket's buffer until the kernel gives up retransmitting, fifteen
+// minutes later. Found by switching off a leader: the standby took over in 14
+// seconds and the node stayed with the dead one. An HTTP/2 ping after ten
+// quiet seconds, given five to answer, finds it in a quarter of a minute.
+const (
+	streamIdle = 10 * time.Second
+	streamPing = 5 * time.Second
+)
+
 // localizeSpec fills in what only this node knows.
 //
 // The interface NCCL bootstraps over is this node's, not the cluster's: the

internal/agent/controlplanes.go

@@ -0,0 +1,77 @@
+package agent
+
+import (
+	"encoding/json"
+	"os"
+	"path/filepath"
+	"slices"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+)
+
+// controlPlanesFile keeps the control planes the leader last named, beside
+// the agent's configuration rather than in it: the configuration is the
+// operator's, and a list rewritten in it would be a file nobody wrote.
+var controlPlanesFile = "/var/lib/llmgw-agent/control-planes.json"
+
+// servers is every address to dial, in the order to try them: the
+// configuration's, then any the leader named that it lacks.
+func (a *Agent) servers() []string {
+	a.mu.Lock()
+	defer a.mu.Unlock()
+	if len(a.serverList) == 0 {
+		return a.cfg.Servers
+	}
+	return a.serverList
+}
+
+// mergeServers puts the leader's list after the configured one, without
+// repeats, and says whether that changed anything.
+func mergeServers(configured, told []string, secure bool) []string {
+	out := slices.Clone(configured)
+	for _, s := range told {
+		fixed, err := normalizeOne(s, secure)
+		if err != nil {
+			continue
+		}
+		if !slices.Contains(out, fixed) {
+			out = append(out, fixed)
+		}
+	}
+	return out
+}
+
+// setControlPlanes takes the leader's list: used from the next reconnect,
+// and kept for the next start.
+func (a *Agent) setControlPlanes(m *llmgwv1.ControlPlanes) {
+	told := m.GetServers()
+	merged := mergeServers(a.cfg.Servers, told, a.cfg.TLS.Enabled())
+	a.mu.Lock()
+	changed := !slices.Equal(merged, a.serverList)
+	a.serverList = merged
+	a.mu.Unlock()
+	if !changed {
+		return
+	}
+	a.log.Info("the control planes to dial", "servers", merged)
+	b, _ := json.Marshal(told)
+	if err := os.MkdirAll(filepath.Dir(controlPlanesFile), 0o755); err == nil {
+		tmp := controlPlanesFile + ".new"
+		if err := os.WriteFile(tmp, b, 0o644); err == nil {
+			_ = os.Rename(tmp, controlPlanesFile)
+		}
+	}
+}
+
+// loadControlPlanes is the list kept from the last time the leader named it.
+func loadControlPlanes(configured []string, secure bool) []string {
+	b, err := os.ReadFile(controlPlanesFile)
+	if err != nil {
+		return configured
+	}
+	var told []string
+	if json.Unmarshal(b, &told) != nil {
+		return configured
+	}
+	return mergeServers(configured, told, secure)
+}

internal/agent/controlplanes_test.go

@@ -0,0 +1,41 @@
+package agent
+
+import (
+	"log/slog"
+	"path/filepath"
+	"slices"
+	"testing"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
+)
+
+func TestAControlPlaneTheLeaderNamesIsDialledAndRemembered(t *testing.T) {
+	old := controlPlanesFile
+	controlPlanesFile = filepath.Join(t.TempDir(), "control-planes.json")
+	t.Cleanup(func() { controlPlanesFile = old })
+
+	configured := []string{"https://10.0.0.140:7443"}
+	// TLS on, as every onboarded node has it: an agent without it refuses
+	// https:// addresses, and the leader's are https://.
+	a := &Agent{cfg: Config{Servers: configured, TLS: tlsconf.Config{CA: "/etc/llmgw/ca.pem"}}, log: slog.New(slog.DiscardHandler)}
+	if got := a.servers(); !slices.Equal(got, configured) {
+		t.Fatalf("before anything is named: %v", got)
+	}
+	// The configured one stays first; the new one is added after it, once.
+	a.setControlPlanes(&llmgwv1.ControlPlanes{Servers: []string{"https://10.0.0.140:7443", "https://10.0.0.150:7443"}})
+	want := []string{"https://10.0.0.140:7443", "https://10.0.0.150:7443"}
+	if got := a.servers(); !slices.Equal(got, want) {
+		t.Errorf("after the leader named two: %v", got)
+	}
+	// And after a restart, before the leader says anything.
+	if got := loadControlPlanes(configured, true); !slices.Equal(got, want) {
+		t.Errorf("after a restart: %v", got)
+	}
+	// A list that drops the configured one does not drop it: the
+	// configuration is the operator's.
+	a.setControlPlanes(&llmgwv1.ControlPlanes{Servers: []string{"https://10.0.0.150:7443"}})
+	if got := a.servers(); got[0] != "https://10.0.0.140:7443" {
+		t.Errorf("the configured server was dropped: %v", got)
+	}
+}

internal/cli/backup.go

@@ -8,7 +8,6 @@ import (
 	"fmt"
 	"io"
 	"io/fs"
-	"net/url"
 	"os"
 	"os/exec"
 	"path/filepath"
@@ -178,30 +177,49 @@ func addStore(ctx context.Context, tw *tar.Writer, addr, scratch string) error {
 // command line: a DSN carries the password, and every user on the machine
 // can read another process's arguments.
 func pgDump(ctx context.Context, dsn, out string) error {
-	u, err := url.Parse(dsn)
+	vars, err := pgEnv(dsn)
 	if err != nil {
-		return fmt.Errorf("the store address is not a URL")
+		return err
 	}
-	env := os.Environ()
+	cmd := exec.CommandContext(ctx, "pg_dump", "--format=custom", "--no-owner", "--file="+out) //nolint:gosec // fixed binary, our own file
+	cmd.Env = append(os.Environ(), vars...)
+	if b, err := cmd.CombinedOutput(); err != nil {
+		return fmt.Errorf("pg_dump: %v: %s", err, strings.TrimSpace(string(b)))
+	}
+	return nil
+}
+
+// pgEnv turns a store address into libpq's environment. Several hosts — the
+// members of a replicated cluster — become libpq's own lists in PGHOST and
+// PGPORT; read as one URL host, the list was one name libpq could not
+// resolve, and a backup worked only by falling through to the second.
+func pgEnv(dsn string) ([]string, error) {
+	hosts, ports, u, err := store.SplitPostgres(dsn)
+	if err != nil {
+		return nil, err
+	}
+	var env []string
 	set := func(k, v string) {
 		if v != "" {
 			env = append(env, k+"="+v)
 		}
 	}
-	set("PGHOST", u.Hostname())
-	set("PGPORT", u.Port())
+	set("PGHOST", strings.Join(hosts, ","))
+	set("PGPORT", strings.Join(ports, ","))
 	set("PGUSER", u.User.Username())
 	if pw, ok := u.User.Password(); ok {
 		set("PGPASSWORD", pw)
 	}
 	set("PGDATABASE", strings.TrimPrefix(u.Path, "/"))
 	set("PGSSLMODE", u.Query().Get("sslmode"))
-	cmd := exec.CommandContext(ctx, "pg_dump", "--format=custom", "--no-owner", "--file="+out) //nolint:gosec // fixed binary, our own file
-	cmd.Env = env
-	if b, err := cmd.CombinedOutput(); err != nil {
-		return fmt.Errorf("pg_dump: %v: %s", err, strings.TrimSpace(string(b)))
+	// A standby if there is one, else whichever answers — not the address's
+	// read-write: a dump only reads, it need not load the primary, and one
+	// that insists on the primary fails exactly while the cluster is failing
+	// over.
+	if len(hosts) > 1 {
+		set("PGTARGETSESSIONATTRS", "prefer-standby")
 	}
-	return nil
+	return env, nil
 }
 
 func addTree(tw *tar.Writer, root, as string) error {

internal/cli/backup_test.go

@@ -139,3 +139,26 @@ func TestNoScratchCopyOfTheStoreIsLeftBehind(t *testing.T) {
 		}
 	}
 }
+
+func TestAStoreOfSeveralHostsIsGivenToPgDumpAsLibpqLists(t *testing.T) {
+	// The address of a Patroni cluster: url.Parse read its host as one name,
+	// and pg_dump reached the database only by falling through to the second.
+	env, err := pgEnv("postgres://llmgw:s3cret@10.0.0.149:5432,10.0.0.150/llmgw?sslmode=disable&target_session_attrs=read-write")
+	if err != nil {
+		t.Fatal(err)
+	}
+	got := strings.Join(env, " ")
+	for _, want := range []string{"PGHOST=10.0.0.149,10.0.0.150", "PGPORT=5432,5432", "PGUSER=llmgw",
+		"PGPASSWORD=s3cret", "PGDATABASE=llmgw", "PGSSLMODE=disable", "PGTARGETSESSIONATTRS=prefer-standby"} {
+		if !strings.Contains(got, want) {
+			t.Errorf("%s is not in %s", want, got)
+		}
+	}
+	one, err := pgEnv("postgres://u:p@db.internal:6432/llmgw")
+	if err != nil || !strings.Contains(strings.Join(one, " "), "PGHOST=db.internal PGPORT=6432") {
+		t.Errorf("one host: %v, %v", one, err)
+	}
+	if _, err := pgEnv("postgres://u:p@a:5432,:5432/x"); err == nil {
+		t.Error("an empty host in the list was accepted")
+	}
+}

internal/cli/ctl_status_test.go

@@ -1,9 +1,14 @@
 package cli
 
 import (
+	"bytes"
 	"net/http"
 	"net/http/httptest"
+	"strings"
 	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"github.com/spf13/cobra"
 )
 
 func TestAQueuedChangeIsNotReportedAsAnError(t *testing.T) {
@@ -22,3 +27,27 @@ func TestAQueuedChangeIsNotReportedAsAnError(t *testing.T) {
 		t.Errorf("the body of a 202 was not read: %+v", out)
 	}
 }
+
+func TestStatusSaysWhoLeadsTheStoreAndWhetherTheRestKeepUp(t *testing.T) {
+	render := func(s adminapi.Summary) string {
+		cmd := &cobra.Command{}
+		var b bytes.Buffer
+		cmd.SetOut(&b)
+		printSummary(cmd, s)
+		return b.String()
+	}
+	out := render(adminapi.Summary{Store: []adminapi.StoreMember{
+		{Name: "llmgw-ctl02", Role: "replica", State: "streaming", Lag: 2048},
+		{Name: "llmgw-pg01", Role: "leader", State: "running"},
+	}})
+	if !strings.Contains(out, "STORE     llmgw-ctl02 streaming (2 KB behind), llmgw-pg01 leads") {
+		t.Errorf("status:\n%s", out)
+	}
+	out = render(adminapi.Summary{Store: []adminapi.StoreMember{{Name: "llmgw-pg01", Role: "replica", State: "stopped"}}})
+	if !strings.Contains(out, "NO LEADER") {
+		t.Errorf("a store with no leader was not called out:\n%s", out)
+	}
+	if out := render(adminapi.Summary{}); strings.Contains(out, "STORE") {
+		t.Error("a single database got a store line")
+	}
+}

internal/cli/join.go

@@ -0,0 +1,61 @@
+package cli
+
+import (
+	"errors"
+	"fmt"
+	"os"
+	"runtime"
+
+	"github.com/spf13/cobra"
+
+	joinpkg "gitea.mynodes.xyz/ilker/llmgw/internal/join"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/release"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/version"
+)
+
+func newJoinCmd() *cobra.Command {
+	return &cobra.Command{
+		Use:   "join <code>",
+		Short: "Let a setup on another machine set this one up, with the code it gave",
+		Long: "For a machine nothing may log into: the setup screens (sudo llmgw on\n" +
+			"the control plane) show a code for each machine they expect. Run this\n" +
+			"with it, as root, on that machine. It dials out to the setup — nothing\n" +
+			"dials in — checks it is the setup the code was made by, and runs what\n" +
+			"the setup sends until it is done: the same checks and installs ssh\n" +
+			"would carry. Each step is printed here as it runs.\n\n" +
+			"A code is good for one machine, for an hour, and only while the setup\n" +
+			"that made it is waiting.",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			if runtime.GOOS != "linux" {
+				return fmt.Errorf("join sets up a Linux machine; this is %s", runtime.GOOS)
+			}
+			if os.Geteuid() != 0 {
+				return errors.New("run it as root — sudo llmgw join … — since the setup installs software here")
+			}
+			code, err := joinpkg.Parse(args[0])
+			if err != nil {
+				return err
+			}
+			out := cmd.OutOrStdout()
+			role := code.Role
+			if role == "" {
+				role = "part of a cluster"
+			}
+			fmt.Fprintf(out, "Joining the setup at %s, as %s. It runs as root here until it is done.\n", code.Addr, role)
+			return joinpkg.Serve(cmd.Context(), code, func(s string) { fmt.Fprintln(out, s) })
+		},
+	}
+}
+
+// joinInstall is what to run on a machine that has no llmgw yet, to fetch
+// this version for its architecture: short lines, since it is copied from a
+// screen that cuts long ones off.
+func joinInstall() []string {
+	v := version.Version
+	return []string{
+		`a=$(uname -m | sed 's/x86_64/amd64/;s/aarch64/arm64/')`,
+		fmt.Sprintf("sudo curl -fsSL -o /usr/local/bin/llmgw \\\n    %s/%s/llmgw-%s-linux-$a", release.DefaultBase, v, v),
+		"sudo chmod 755 /usr/local/bin/llmgw",
+	}
+}

internal/cli/root.go

@@ -30,6 +30,14 @@ func Root() *cobra.Command {
 		PersistentPreRun: func(_ *cobra.Command, _ []string) {
 			obs.Setup(obs.Options{Level: global.logLevel, Format: global.logFormat})
 		},
+		// Bare, in a terminal: the setup screens or the console. Otherwise
+		// the help, as before.
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			if interactive(cmd) {
+				return runHome(cmd)
+			}
+			return cmd.Help()
+		},
 	}
 
 	pf := root.PersistentFlags()
@@ -49,6 +57,7 @@ func Root() *cobra.Command {
 		newNodeCmd(),
 		newSelfUpgradeCmd(),
 		newReleaseCmd(),
+		newJoinCmd(),
 	)
 	return root
 }

internal/cli/server.go

@@ -19,6 +19,7 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/pgha"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/sourcecheck"
 	"github.com/prometheus/client_golang/prometheus"
 
@@ -127,6 +128,9 @@ func newServerCmd() *cobra.Command {
 			var agentHandler http.Handler
 			var agentPath string
 			var admin *control.Controller
+			// The store's cluster, when it is one; read by the metrics and
+			// by `ctl status`.
+			var storeHA *pgha.Collector
 			if controlMode {
 				cat, err := cfg.Catalog()
 				if err != nil {
@@ -179,6 +183,17 @@ func newServerCmd() *cobra.Command {
 				if err := rt.RegisterMetrics(fleetmetrics.New(ctl)); err != nil {
 					return err
 				}
+				// A store of several hosts is a replicated cluster: what state
+				// it is in is published beside everything else, read from its
+				// Patroni. One host is a single PostgreSQL, which has none.
+				if addr, err := storepkg.ResolveAddress(cfg.Auth.Store); err == nil && storepkg.IsPostgres(addr) {
+					if hosts, _, _, err := storepkg.SplitPostgres(strings.TrimSpace(addr)); err == nil && len(hosts) > 1 {
+						storeHA = pgha.NewCollector(hosts)
+						if err := rt.RegisterMetrics(storeHA); err != nil {
+							return err
+						}
+					}
+				}
 				// So a request for a model that is configured but not running
 				// starts it, rather than being told nothing serves it.
 				rt.SetActivator(ctl.Reconciler())
@@ -201,6 +216,16 @@ func newServerCmd() *cobra.Command {
 			)
 			if admin != nil {
 				api := adminapi.New(admin)
+				if storeHA != nil {
+					api.SetStoreStatus(func() ([]adminapi.StoreMember, error) {
+						c, err := storeHA.Read()
+						var out []adminapi.StoreMember
+						for _, m := range c.Members {
+							out = append(out, adminapi.StoreMember{Name: m.Name, Host: m.Host, Role: m.Role, State: m.State, Lag: m.Lag})
+						}
+						return out, err
+					})
+				}
 				adminAPI = api
 				api.SetReleases(cfg.Releases)
 				// Catalog changes need somewhere to persist, so this is wired
@@ -426,7 +451,7 @@ func newServerCmd() *cobra.Command {
 	f.StringVar(&metricsListen, "metrics-listen", "", "serve metrics only on this address, safe to expose")
 	f.StringArrayVar(&endpoints, "endpoint", nil, "static endpoint: id=URL=model[,model] (repeatable)")
 
-	cmd.AddCommand(newServerInitCmd())
+	cmd.AddCommand(newServerInitCmd(), newServerAddCmd())
 	cmd.AddCommand(newBackupCmd())
 	return cmd
 }
@@ -488,6 +513,9 @@ func startControlPlane(ctx context.Context, cfg config.Server, cat *catalog.Live
 	if h, ok := sink.(interface{ Store() *storepkg.DB }); ok {
 		if st := h.Store(); st != nil {
 			lead = control.NewLeadership(st, cfg.Control.LeaseTTL, log)
+			if r, ok := sink.(interface{ SetLeader(func() bool) }); ok {
+				r.SetLeader(lead.IsLeader)
+			}
 			ctl.SetLeadership(lead)
 			// The tenancy tree's capacity quotas: what each part is
 			// guaranteed and may hold.
@@ -517,6 +545,21 @@ func startControlPlane(ctx context.Context, cfg config.Server, cat *catalog.Live
 	if lead != nil {
 		svc.SetLeadership(lead)
 	}
+	// The control planes agents may dial, from the store every control plane
+	// shares: told to each agent as it registers, and to all of them when
+	// `server add` records another.
+	if h, ok := sink.(interface{ Store() *storepkg.DB }); ok && h.Store() != nil {
+		st := h.Store()
+		svc.SetControlPlanes(func(ctx context.Context) ([]string, error) {
+			cps, err := st.ControlPlanes(ctx)
+			urls := make([]string, len(cps))
+			for i, c := range cps {
+				urls[i] = c.URL
+			}
+			return urls, err
+		})
+		go svc.WatchControlPlanes(ctx, 30*time.Second)
+	}
 	fwd.set(svc)
 	// Every InfiniBand fabric mapped every five minutes, through one node.
 	go svc.ScanFabrics(ctx, 5*time.Minute)

internal/cli/serveradd.go

@@ -0,0 +1,459 @@
+package cli
+
+import (
+	"bytes"
+	"context"
+	"crypto/rand"
+	"crypto/x509"
+	"encoding/hex"
+	"encoding/pem"
+	"errors"
+	"fmt"
+	"io"
+	"net"
+	"os"
+	"path/filepath"
+	"slices"
+	"strings"
+	"time"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/pgha"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/version"
+)
+
+// serverAdd makes another machine a control plane of this cluster: a standby
+// that takes over when this one is lost. Run on a control plane that holds
+// the authority, as root, like `node add`.
+type serverAdd struct {
+	etcDir, pkiDir string
+	// hosts are more names to put in the new control plane's certificate.
+	hosts []string
+	// vip is a floating address, "10.0.0.160/24", to keep on whichever of the
+	// two leads; "" leaves client traffic to whatever is in front.
+	vip string
+
+	remote onboard.Remote
+	// addr is the machine's address as given; ip what the others reach it at.
+	addr, ip, name string
+	out            io.Writer
+}
+
+func newServerAddCmd() *cobra.Command {
+	sa := serverAdd{etcDir: "/etc/llmgw", pkiDir: defaultPKIDir}
+	cmd := &cobra.Command{
+		Use:   "add [user@]address",
+		Short: "Make another machine a control plane of this cluster: a standby that takes over",
+		Long: "Run on a control plane, as root. It logs in to the machine with ssh, as\n" +
+			"node add does, and gives it this llmgw, a certificate from this\n" +
+			"cluster's authority, this control plane's configuration and the store's\n" +
+			"address, lets it into the store, and starts it. It stands by while this\n" +
+			"one leads, and takes over within seconds when this one is lost; every\n" +
+			"node is told of it and fails over to it without being touched.\n\n" +
+			"The store must be one both can reach — replicated with 'server init\n" +
+			"--patroni' — not a database on this machine.\n\n" +
+			"--vip keeps a floating address on whichever control plane leads, with\n" +
+			"keepalived on both: the address clients use for the gateway.\n\n" +
+			"Run it again at any time: it changes only what is missing.",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			if os.Geteuid() != 0 {
+				return errors.New("run it as root: it reads the certificate authority and the store's address")
+			}
+			user, host, ok := strings.Cut(args[0], "@")
+			if !ok {
+				user, host = "", args[0]
+			}
+			sa.remote = &onboard.SSH{Addr: host, User: user}
+			sa.addr = host
+			sa.out = cmd.OutOrStdout()
+			return sa.run(cmd.Context())
+		},
+	}
+	f := cmd.Flags()
+	f.StringArrayVar(&sa.hosts, "host", nil, "another name for the new control plane's certificate; repeatable")
+	f.StringVar(&sa.vip, "vip", "", "a floating address for whichever control plane leads, as 10.0.0.160/24")
+	f.StringVar(&sa.pkiDir, "pki", sa.pkiDir, "directory holding the certificate authority")
+	return cmd
+}
+
+func (sa *serverAdd) done(format string, a ...any) {
+	fmt.Fprintf(sa.out, "  ok    "+format+"\n", a...)
+}
+
+func (sa *serverAdd) kept(format string, a ...any) {
+	fmt.Fprintf(sa.out, "  kept  "+format+"\n", a...)
+}
+
+func (sa *serverAdd) sh(ctx context.Context, script string, stdin []byte) (string, error) {
+	out, err := sa.remote.Run(ctx, script, stdin)
+	if err != nil {
+		return string(out), fmt.Errorf("on %s: %v: %s", sa.addr, err, strings.TrimSpace(string(out)))
+	}
+	return string(out), nil
+}
+
+// put writes a file there, through standard input, with its mode and owner.
+func (sa *serverAdd) put(ctx context.Context, path, mode, owner string, body []byte) error {
+	_, err := sa.sh(ctx, fmt.Sprintf(`set -e
+sudo install -d -m 0755 %[1]s
+sudo sh -c 'umask 077; cat > %[2]s.new'
+sudo chmod %[3]s %[2]s.new
+sudo chown %[4]s %[2]s.new
+sudo mv -f %[2]s.new %[2]s`, filepath.Dir(path), path, mode, owner), body)
+	return err
+}
+
+func (sa *serverAdd) run(ctx context.Context) error {
+	steps := []struct {
+		name string
+		fn   func(context.Context) error
+	}{
+		{"connect", sa.connect},
+		{"binary", sa.binary},
+		{"directories", sa.dirs},
+		{"certificate", sa.certificate},
+		{"configuration", sa.configuration},
+		{"store", sa.storeAccess},
+		{"service", sa.service},
+	}
+	if sa.vip != "" {
+		steps = append(steps, struct {
+			name string
+			fn   func(context.Context) error
+		}{"floating address", sa.floating})
+	}
+	for _, s := range steps {
+		if err := s.fn(ctx); err != nil {
+			return fmt.Errorf("%s: %w", s.name, err)
+		}
+	}
+	fmt.Fprintf(sa.out, "\n%s is a control plane of this cluster, standing by while this one leads.\n", sa.name)
+	return nil
+}
+
+func (sa *serverAdd) connect(ctx context.Context) error {
+	out, err := sa.sh(ctx, "hostname; sudo -n true && echo sudo-ok", nil)
+	if err != nil {
+		return fmt.Errorf("%w — it needs ssh and sudo without a password", err)
+	}
+	lines := strings.Split(strings.TrimSpace(out), "\n")
+	sa.name = strings.TrimSpace(lines[0])
+	if sa.ip == "" {
+		if sa.ip, err = ipv4Of(sa.addr); err != nil {
+			return err
+		}
+	}
+	sa.done("%s (%s)", sa.name, sa.ip)
+	return nil
+}
+
+// binary gives it this very llmgw, so the two control planes never differ.
+func (sa *serverAdd) binary(ctx context.Context) error {
+	have, _ := sa.sh(ctx, "llmgw --version 2>/dev/null || true", nil)
+	if strings.TrimSpace(have) == "llmgw version "+version.Version {
+		sa.kept("llmgw %s", version.Version)
+		return nil
+	}
+	exe, err := os.Executable()
+	if err != nil {
+		return err
+	}
+	b, err := os.ReadFile(exe)
+	if err != nil {
+		return err
+	}
+	if _, err := sa.sh(ctx, `set -e; t=$(mktemp); cat > "$t"; sudo install -m 0755 "$t" /usr/local/bin/llmgw; rm -f "$t"`, b); err != nil {
+		return err
+	}
+	sa.done("llmgw %s, copied from this machine", version.Version)
+	return nil
+}
+
+func (sa *serverAdd) dirs(ctx context.Context) error {
+	_, err := sa.sh(ctx, `set -e
+id llmgw >/dev/null 2>&1 || sudo useradd --system --home /var/lib/llmgw --shell /usr/sbin/nologin llmgw
+sudo install -d -m 0755 /etc/llmgw
+sudo install -d -o llmgw -g llmgw -m 0700 /etc/llmgw/secrets
+sudo install -d -o llmgw -g llmgw -m 0750 /var/lib/llmgw`, nil)
+	if err == nil {
+		sa.done("user llmgw, /etc/llmgw and /var/lib/llmgw")
+	}
+	return err
+}
+
+// certificate issues it a server certificate from this cluster's authority,
+// for its name and addresses, unless it holds one that already names them.
+func (sa *serverAdd) certificate(ctx context.Context) error {
+	ca, err := loadCA(sa.pkiDir)
+	if err != nil {
+		return fmt.Errorf("the authority's key is not in %s: run this where it is (%v)", sa.pkiDir, err)
+	}
+	ips, _ := sa.sh(ctx, "hostname -I", nil)
+	want := []string{sa.name, sa.ip}
+	for _, ip := range strings.Fields(ips) {
+		if net.ParseIP(ip).To4() != nil && !slices.Contains(want, ip) {
+			want = append(want, ip)
+		}
+	}
+	want = append(want, sa.hosts...)
+
+	if have, _ := sa.sh(ctx, "sudo cat /etc/llmgw/server.pem 2>/dev/null || true", nil); have != "" {
+		if blk, _ := pem.Decode([]byte(have)); blk != nil {
+			if cert, err := x509.ParseCertificate(blk.Bytes); err == nil && cert.CheckSignatureFrom(ca.Cert) == nil {
+				names := strings.Split(sanNames(cert), ", ")
+				missing := false
+				for _, w := range want {
+					missing = missing || !slices.Contains(names, w)
+				}
+				if !missing {
+					sa.kept("certificate for %s", strings.Join(names, ", "))
+					return sa.put(ctx, "/etc/llmgw/ca.pem", "0644", "root:root", ca.PEM)
+				}
+			}
+		}
+	}
+	leaf, err := ca.IssueServer(sa.name, want)
+	if err != nil {
+		return err
+	}
+	if err := sa.put(ctx, "/etc/llmgw/ca.pem", "0644", "root:root", ca.PEM); err != nil {
+		return err
+	}
+	if err := sa.put(ctx, "/etc/llmgw/server.pem", "0644", "root:root", leaf.CertPEM); err != nil {
+		return err
+	}
+	if err := sa.put(ctx, "/etc/llmgw/server-key.pem", "0640", "root:llmgw", leaf.KeyPEM); err != nil {
+		return err
+	}
+	sa.done("certificate for %s, from this cluster's authority", strings.Join(want, ", "))
+	return nil
+}
+
+// configuration copies this control plane's, and the secrets beside it: the
+// two must agree on everything, the store above all.
+func (sa *serverAdd) configuration(ctx context.Context) error {
+	if have, _ := sa.sh(ctx, "sudo test -f /etc/llmgw/server.yaml && echo yes || true", nil); strings.TrimSpace(have) == "yes" {
+		sa.kept("/etc/llmgw/server.yaml (never overwritten)")
+	} else {
+		b, err := os.ReadFile(filepath.Join(sa.etcDir, "server.yaml"))
+		if err != nil {
+			return err
+		}
+		if err := sa.put(ctx, "/etc/llmgw/server.yaml", "0644", "root:root", b); err != nil {
+			return err
+		}
+		sa.done("/etc/llmgw/server.yaml, as this control plane's")
+	}
+	secrets, err := os.ReadDir(filepath.Join(sa.etcDir, "secrets"))
+	if err != nil && !os.IsNotExist(err) {
+		return err
+	}
+	for _, e := range secrets {
+		if e.IsDir() || strings.Contains(e.Name(), ".pre-") || strings.HasSuffix(e.Name(), ".new") {
+			continue
+		}
+		path := "/etc/llmgw/secrets/" + e.Name()
+		mine, err := os.ReadFile(filepath.Join(sa.etcDir, "secrets", e.Name()))
+		if err != nil {
+			return err
+		}
+		theirs, _ := sa.sh(ctx, "sudo cat "+path+" 2>/dev/null || true", nil)
+		switch {
+		case theirs == string(mine):
+			sa.kept("%s", path)
+		case theirs != "":
+			// Two control planes on two stores are two clusters.
+			return fmt.Errorf("%s there differs from this one's; remove it there if that machine is to join this cluster", path)
+		default:
+			if err := sa.put(ctx, path, "0600", "llmgw:llmgw", mine); err != nil {
+				return err
+			}
+			sa.done("%s", path)
+		}
+	}
+	return nil
+}
+
+// storeAccess lets it into the store, and records both control planes for
+// the nodes to be told.
+func (sa *serverAdd) storeAccess(ctx context.Context) error {
+	cfg, err := config.Load(filepath.Join(sa.etcDir, "server.yaml"))
+	if err != nil {
+		return err
+	}
+	addr, err := store.ResolveAddress(cfg.Auth.Store)
+	if err != nil {
+		return err
+	}
+	addr = strings.TrimSpace(addr)
+	if !store.IsPostgres(addr) {
+		return errors.New("the store is SQLite on this machine; a second control plane needs one both reach — run 'llmgw server init --patroni' first")
+	}
+	hosts, _, _, err := store.SplitPostgres(addr)
+	if err != nil {
+		return err
+	}
+	for _, h := range hosts {
+		if ip := net.ParseIP(h); ip != nil && ip.IsLoopback() || h == "localhost" {
+			return fmt.Errorf("the store is at %s, on this machine only; replicate it first with 'llmgw server init --patroni'", h)
+		}
+	}
+	switch {
+	case slices.Contains(hosts, sa.ip):
+		sa.kept("store: %s is one of its members", sa.ip)
+	case len(hosts) > 1:
+		changed, err := pgha.AllowClient(ctx, hosts, sa.ip)
+		if err != nil {
+			return err
+		}
+		if changed {
+			sa.done("store: %s let in, through Patroni", sa.ip)
+		} else {
+			sa.kept("store: %s already let in", sa.ip)
+		}
+	default:
+		sa.kept("store at %s: make sure it lets %s in", hosts[0], sa.ip)
+	}
+
+	st, err := store.Open(ctx, addr)
+	if err != nil {
+		return err
+	}
+	defer st.Close()
+	_, port, _ := net.SplitHostPort(cfg.AgentListen)
+	if port == "" {
+		port = "7443"
+	}
+	self, err := sourceIPToward(sa.ip)
+	if err != nil {
+		return err
+	}
+	me, _ := os.Hostname()
+	for _, cp := range []store.ControlPlane{
+		{URL: "https://" + net.JoinHostPort(self, port), Name: me},
+		{URL: "https://" + net.JoinHostPort(sa.ip, port), Name: sa.name, AddedAt: time.Now().Add(time.Second)},
+	} {
+		if err := st.PutControlPlane(ctx, cp); err != nil {
+			return err
+		}
+	}
+	sa.done("nodes will be told of both control planes within half a minute")
+	return nil
+}
+
+// service installs the unit and starts it, then waits for it to answer.
+func (sa *serverAdd) service(ctx context.Context) error {
+	if err := sa.put(ctx, "/etc/systemd/system/llmgw-server.service", "0644", "root:root", []byte(serverUnit)); err != nil {
+		return err
+	}
+	if _, err := sa.sh(ctx, "sudo systemctl daemon-reload && sudo systemctl enable llmgw-server >/dev/null 2>&1 && sudo systemctl restart llmgw-server", nil); err != nil {
+		return err
+	}
+	deadline := time.Now().Add(90 * time.Second)
+	for {
+		code, _ := sa.sh(ctx, "curl -s -o /dev/null -m 2 -w '%{http_code}' http://127.0.0.1:8080/leader || true", nil)
+		switch strings.TrimSpace(code) {
+		case "200":
+			sa.done("llmgw-server running, and leading")
+			return nil
+		case "503":
+			sa.done("llmgw-server running, standing by")
+			return nil
+		}
+		if time.Now().After(deadline) {
+			logs, _ := sa.sh(ctx, "sudo journalctl -u llmgw-server -n 15 --no-pager", nil)
+			return fmt.Errorf("it did not answer within 90s:\n%s", logs)
+		}
+		time.Sleep(2 * time.Second)
+	}
+}
+
+// floating keeps an address on whichever of the two leads, with keepalived
+// on both, following /leader.
+func (sa *serverAdd) floating(ctx context.Context) error {
+	vip, nw, err := net.ParseCIDR(sa.vip)
+	if err != nil || vip.To4() == nil {
+		return fmt.Errorf("--vip %q is not an IPv4 address with its prefix, like 10.0.0.160/24", sa.vip)
+	}
+	prefix, _ := nw.Mask.Size()
+	pass, err := vrrpPassword(sa.etcDir)
+	if err != nil {
+		return err
+	}
+	here := pgha.Local{}
+	for _, m := range []struct {
+		r        onboard.Remote
+		name     string
+		priority int
+	}{{here, "this machine", 110}, {sa.remote, sa.name, 100}} {
+		out, err := m.r.Run(ctx, fmt.Sprintf(`set -e
+command -v keepalived >/dev/null 2>&1 || { sudo DEBIAN_FRONTEND=noninteractive apt-get -qq update >/dev/null; sudo DEBIAN_FRONTEND=noninteractive apt-get -qq install -y keepalived >/dev/null; }
+ip -o route get %s | awk '{for(i=1;i<=NF;i++) if($i=="dev") {print $(i+1); exit}}'`, vip), nil)
+		if err != nil {
+			return fmt.Errorf("on %s: %v: %s", m.name, err, strings.TrimSpace(string(out)))
+		}
+		iface := strings.TrimSpace(string(out))
+		conf := keepalivedConf(iface, vip.String(), prefix, m.priority, pass)
+		if out, err := m.r.Run(ctx, `set -e
+sudo install -d -m 0755 /etc/keepalived
+sudo sh -c 'umask 077; cat > /etc/keepalived/keepalived.conf'
+sudo systemctl enable keepalived >/dev/null 2>&1
+sudo systemctl restart keepalived`, []byte(conf)); err != nil {
+			return fmt.Errorf("on %s: %v: %s", m.name, err, strings.TrimSpace(string(out)))
+		}
+	}
+	sa.done("%s follows whichever control plane leads (keepalived on both)", sa.vip)
+	return nil
+}
+
+// vrrpPassword is the shared secret of the two keepalived, made once and
+// kept with the other secrets, so running this again does not split them.
+func vrrpPassword(etcDir string) (string, error) {
+	p := filepath.Join(etcDir, "secrets", "vrrp")
+	if b, err := os.ReadFile(p); err == nil && len(bytes.TrimSpace(b)) > 0 {
+		return string(bytes.TrimSpace(b)), nil
+	}
+	b := make([]byte, 4)
+	if _, err := rand.Read(b); err != nil {
+		return "", err
+	}
+	pass := hex.EncodeToString(b) // keepalived reads eight characters
+	if err := os.WriteFile(p, []byte(pass+"\n"), 0o600); err != nil {
+		return "", err
+	}
+	return pass, nil
+}
+
+func keepalivedConf(iface, vip string, prefix, priority int, pass string) string {
+	return fmt.Sprintf(`# llmgw: the gateway's floating address, on whichever control plane leads.
+vrrp_script llmgw_leader {
+  script "/usr/bin/curl -fsS -o /dev/null -m 2 http://127.0.0.1:8080/leader"
+  interval 2
+  fall 2
+  rise 2
+}
+
+vrrp_instance llmgw {
+  state BACKUP
+  interface %s
+  virtual_router_id 51
+  priority %d
+  advert_int 1
+  authentication {
+    auth_type PASS
+    auth_pass %s
+  }
+  virtual_ipaddress {
+    %s/%d
+  }
+  track_script {
+    llmgw_leader
+  }
+}
+`, iface, priority, pass, vip, prefix)
+}

internal/cli/serverinit.go

@@ -23,6 +23,7 @@ import (
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/account"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/pgha"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
 )
@@ -69,6 +70,13 @@ type serverInit struct {
 	// postgres keeps the store in a PostgreSQL on this machine, installed if
 	// need be, moving whatever SQLite already holds.
 	postgres bool
+	// patroni are user@host of the machines to keep the store on under
+	// Patroni, replicated; this machine is the third etcd vote for two.
+	patroni []string
+	// patroniNodes are the same machines already connected to, which the
+	// setup screen does with the password it was given; when set, patroni
+	// only names them.
+	patroniNodes []pgha.Node
 	// noMirror, noNode and noModel turn off the parts of a first install that
 	// make it serve something: a mirror, this machine as a node, and a small
 	// model on it.
@@ -86,14 +94,19 @@ type serverInit struct {
 	changed  bool
 }
 
-func newServerInitCmd() *cobra.Command {
-	in := serverInit{
+// newDefaultServerInit is init as it sets up a real machine.
+func newDefaultServerInit() serverInit {
+	return serverInit{
 		etcDir:   "/etc/llmgw",
 		stateDir: "/var/lib/llmgw",
 		pkiDir:   defaultPKIDir,
 		unitPath: "/etc/systemd/system/llmgw-server.service",
 		user:     "llmgw",
 	}
+}
+
+func newServerInitCmd() *cobra.Command {
+	in := newDefaultServerInit()
 	var plain bool
 	cmd := &cobra.Command{
 		Use:   "init",
@@ -111,6 +124,12 @@ func newServerInitCmd() *cobra.Command {
 			"installed if there is none, with its own role and a generated password\n" +
 			"kept in /etc/llmgw/secrets. An existing SQLite store is copied into it,\n" +
 			"so accounts and keys survive, and server.yaml is pointed at it.\n\n" +
+			"With --patroni, given two or more machines, the store is PostgreSQL\n" +
+			"replicated across them under Patroni, which promotes a standby when the\n" +
+			"primary is lost. etcd holds its leader lock on those machines — and on\n" +
+			"this one, when they are two, as the third vote. Installed over ssh with\n" +
+			"passwordless sudo, as 'llmgw node add' does; the store already in use,\n" +
+			"SQLite or PostgreSQL, is moved in. Run again, it changes nothing.\n\n" +
 			"Each step is skipped when its result already exists, so it is safe to run\n" +
 			"again: to finish an install that stopped, or with another --host to\n" +
 			"reissue the certificate. It never overwrites server.yaml or the authority.\n\n" +
@@ -144,6 +163,9 @@ func newServerInitCmd() *cobra.Command {
 	f.BoolVar(&in.noModel, "no-model", false, "do not fetch and serve the starter model ("+starterName+")")
 	f.BoolVar(&in.postgres, "postgres", false,
 		"keep the store in PostgreSQL on this machine, installing it and moving an existing SQLite store into it")
+	f.StringArrayVar(&in.patroni, "patroni", nil,
+		"user@host of a machine to keep the store on, replicated under Patroni; give two or more. "+
+			"Installs PostgreSQL, Patroni and etcd there over ssh, and moves the existing store in")
 	return cmd
 }
 

internal/cli/serverinit_patroni.go

@@ -0,0 +1,182 @@
+package cli
+
+import (
+	"context"
+	"fmt"
+	"net"
+	"os"
+	"os/exec"
+	"strings"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/pgha"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// ensurePatroni puts the store in PostgreSQL under Patroni on the machines
+// given with --patroni, and moves whatever store server.yaml names into it:
+// SQLite, or a PostgreSQL on one machine. This machine is the third etcd vote
+// when they are two.
+func (in *serverInit) ensurePatroni(ctx context.Context) (err error) {
+	setup := pgha.Setup{Emit: func(step, detail string, changed bool) {
+		if changed {
+			in.done("%s: %s", step, detail)
+		} else {
+			in.kept("%s: %s", step, detail)
+		}
+	}}
+	setup.DB = in.patroniNodes
+	for _, t := range in.patroni {
+		if len(in.patroniNodes) > 0 {
+			break
+		}
+		user, host, ok := strings.Cut(t, "@")
+		if !ok {
+			user, host = "", t
+		}
+		remote := &onboard.SSH{Addr: host, User: user}
+		out, err := remote.Run(ctx, "hostname && sudo -n true", nil)
+		if err != nil {
+			return fmt.Errorf("%s: cannot log in and use sudo without a password: %v: %s",
+				t, err, strings.TrimSpace(string(out)))
+		}
+		ip, err := ipv4Of(host)
+		if err != nil {
+			return err
+		}
+		setup.DB = append(setup.DB, pgha.Node{
+			Member: pgha.Member{Name: strings.TrimSpace(strings.SplitN(string(out), "\n", 2)[0]), IP: ip},
+			Remote: remote,
+		})
+	}
+	self, err := sourceIPToward(setup.DB[0].IP)
+	if err != nil {
+		return err
+	}
+	name, _ := os.Hostname()
+	setup.Witness = &pgha.Node{Member: pgha.Member{Name: name, IP: self}, Remote: pgha.Local{}}
+	setup.Clients = []string{self}
+
+	dsn, err := setup.Run(ctx)
+	if err != nil {
+		return err
+	}
+
+	cfg, err := config.Load(in.configPath())
+	if err != nil {
+		return err
+	}
+	current := cfg.Auth.Store
+	if current == "" {
+		current = in.storeAddress()
+	}
+	resolved, _ := store.ResolveAddress(current)
+	if strings.TrimSpace(resolved) == dsn {
+		in.kept("store in the Patroni cluster (%s)", in.dsnPath())
+		return nil
+	}
+
+	// Still while it moves, and started again on the old store if the move
+	// fails: a gateway down over a database that did not work out is worse
+	// than one left where it was.
+	stopped, err := in.stopService(ctx)
+	if err != nil {
+		return err
+	}
+	defer func() {
+		if err != nil && stopped {
+			_ = exec.Command("systemctl", "start", "llmgw-server").Run()
+			fmt.Fprintf(in.out, "  ..    llmgw-server started again on its old store\n")
+		}
+	}()
+	if err := in.moveInto(ctx, current, dsn); err != nil {
+		return err
+	}
+
+	// The old address is kept beside the new one: the way back, if the
+	// cluster is not what was wanted.
+	if b, err := os.ReadFile(in.dsnPath()); err == nil && strings.TrimSpace(string(b)) != dsn {
+		if err := os.WriteFile(in.dsnPath()+".pre-patroni", b, 0o600); err != nil {
+			return err
+		}
+		_ = in.chown(in.dsnPath()+".pre-patroni", true)
+	}
+	if err := in.writeDSN(dsn); err != nil {
+		return err
+	}
+	if current != "@"+in.dsnPath() {
+		if err := setAuthStore(in.configPath(), "@"+in.dsnPath()); err != nil {
+			return err
+		}
+	}
+	// Started again by the service step that follows, on the new address.
+	in.done("server.yaml now keeps the store in the Patroni cluster")
+	return nil
+}
+
+// moveInto copies a store into the cluster, unless the cluster already holds
+// one — from a run that copied and stopped before changing the address.
+func (in *serverInit) moveInto(ctx context.Context, from, to string) error {
+	src, err := store.Open(ctx, from)
+	if err != nil {
+		// Nothing there yet: a first install, with nothing to move.
+		in.kept("no store at %s to move", from)
+		return nil
+	}
+	defer src.Close()
+	dst, err := store.Open(ctx, to)
+	if err != nil {
+		return fmt.Errorf("opening the cluster: %w", err)
+	}
+	defer dst.Close()
+	empty, err := dst.Empty(ctx)
+	if err != nil {
+		return err
+	}
+	if !empty {
+		in.kept("the cluster already holds a store; %s is left as it is", from)
+		return nil
+	}
+	res, err := src.CopyTo(ctx, dst)
+	if err != nil {
+		return fmt.Errorf("copying the store into the cluster: %w", err)
+	}
+	var rows int64
+	for _, r := range res {
+		rows += r.Rows
+	}
+	in.done("copied %d rows into the cluster (the old store is kept as it was)", rows)
+	return nil
+}
+
+// sourceIPToward is this machine's address on the route to another: the one
+// the other machine sees it connect from, which is what etcd advertises and
+// what the database lets in. Not the first --host, which names the gateway
+// for its clients and may not resolve yet, or resolve to a floating address.
+func sourceIPToward(ip string) (string, error) {
+	c, err := net.Dial("udp", net.JoinHostPort(ip, "5432"))
+	if err != nil {
+		return "", fmt.Errorf("no route from this machine to %s: %w", ip, err)
+	}
+	defer c.Close()
+	return c.LocalAddr().(*net.UDPAddr).IP.String(), nil
+}
+
+// ipv4Of is the address a host name or address stands for, as the other
+// machines will be told to reach it.
+func ipv4Of(host string) (string, error) {
+	if ip := net.ParseIP(host); ip != nil {
+		return ip.String(), nil
+	}
+	ips, err := net.LookupIP(host)
+	if err != nil {
+		return "", fmt.Errorf("%s does not resolve: %w", host, err)
+	}
+	for _, ip := range ips {
+		if v4 := ip.To4(); v4 != nil {
+			return v4.String(), nil
+		}
+	}
+	return "", fmt.Errorf("%s has no IPv4 address", host)
+}

internal/cli/serverinit_postgres.go

@@ -44,6 +44,12 @@ func (in *serverInit) storeAddress() string {
 // means reissuing every key and being unable to say which was which; and the
 // administrator init created a minute ago would be gone too.
 func (in *serverInit) ensureDatabase(ctx context.Context) (err error) {
+	if len(in.patroni) > 0 || len(in.patroniNodes) > 0 {
+		if in.postgres {
+			return fmt.Errorf("--postgres keeps the store on this machine and --patroni on others: give one")
+		}
+		return in.ensurePatroni(ctx)
+	}
 	if !in.postgres {
 		return nil
 	}

internal/cli/status.go

@@ -2,6 +2,7 @@ package cli
 
 import (
 	"fmt"
+	"strings"
 	"time"
 
 	"github.com/spf13/cobra"
@@ -94,6 +95,31 @@ func printSummary(cmd *cobra.Command, s adminapi.Summary) {
 	fmt.Fprintf(out, "\nMODELS    %d of %d have the replicas they asked for\n",
 		s.Models.Satisfied, s.Models.Total)
 
+	// The store, when it is a replicated cluster: one line, saying who leads
+	// and whether every other member keeps up.
+	switch {
+	case s.StoreError != "":
+		fmt.Fprintf(out, "STORE     cannot read its cluster: %s\n", s.StoreError)
+	case len(s.Store) > 0:
+		var parts []string
+		leader := false
+		for _, m := range s.Store {
+			switch {
+			case m.Role == "leader":
+				leader = true
+				parts = append(parts, m.Name+" leads")
+			case m.State == "streaming" && m.Lag >= 0:
+				parts = append(parts, fmt.Sprintf("%s streaming (%s behind)", m.Name, bytesShort(m.Lag)))
+			default:
+				parts = append(parts, fmt.Sprintf("%s %s", m.Name, m.State))
+			}
+		}
+		if !leader {
+			parts = append([]string{"NO LEADER — writes fail"}, parts...)
+		}
+		fmt.Fprintf(out, "STORE     %s\n", strings.Join(parts, ", "))
+	}
+
 	if len(s.Parked) > 0 {
 		fmt.Fprintln(out, "\nPARKED — failed every time they started; not retried until fixed")
 		for _, p := range s.Parked {
@@ -169,3 +195,16 @@ func unhealthyCell(n int) string {
 	}
 	return fmt.Sprintf("%d", n)
 }
+
+// bytesShort is a byte count as a person reads it: 0 B, 12 KB, 3.4 MB.
+func bytesShort(n int64) string {
+	switch {
+	case n < 1024:
+		return fmt.Sprintf("%d B", n)
+	case n < 1<<20:
+		return fmt.Sprintf("%d KB", n>>10)
+	case n < 1<<30:
+		return fmt.Sprintf("%.1f MB", float64(n)/(1<<20))
+	}
+	return fmt.Sprintf("%.1f GB", float64(n)/(1<<30))
+}

internal/cli/tui_frame.go

@@ -0,0 +1,237 @@
+package cli
+
+import (
+	"strings"
+
+	"github.com/charmbracelet/lipgloss"
+	"github.com/charmbracelet/x/ansi"
+)
+
+// The frame every full-screen llmgw screen is drawn in: a title bar, a
+// sidebar of steps or actions on the left, the content in a bordered panel on
+// the right, the keys along the bottom — and a dialog over it all when one is
+// open. One frame, so the setup and the console read as one program.
+//
+// Keyboard only: these run on serial consoles and IPMI as often as in a
+// desktop terminal.
+
+// navItem is one line of the sidebar.
+type navItem struct {
+	Label string
+	// State is "done", "current", "todo" for steps; "" for menu entries,
+	// where Current marks the one focused.
+	State   string
+	Current bool
+	// Dim greys an entry that cannot be chosen now.
+	Dim bool
+	// Heading is a title between groups of entries, not an entry.
+	Heading bool
+}
+
+type frame struct {
+	Width, Height int
+	Title         string // left of the title bar
+	Where         string // right of the title bar: the machine
+	NavTitle      string
+	Nav           []navItem
+	PanelTitle    string
+	Body          string
+	// Scroll is how many of the body's lines are scrolled off the top.
+	Scroll int
+	Keys   string
+	// Dialog, when set, is drawn over the middle of the screen.
+	Dialog      string
+	DialogTitle string
+}
+
+var (
+	fBar      = lipgloss.NewStyle().Bold(true).Foreground(lipgloss.AdaptiveColor{Light: "#FFFFFF", Dark: "#0B1B1B"}).Background(tuiAccent)
+	fBorder   = lipgloss.NewStyle().Border(lipgloss.RoundedBorder()).BorderForeground(tuiDim)
+	fFocus    = lipgloss.NewStyle().Border(lipgloss.RoundedBorder()).BorderForeground(tuiAccent)
+	fKeys     = lipgloss.NewStyle().Foreground(tuiDim)
+	fSelected = lipgloss.NewStyle().Bold(true).Foreground(tuiAccent)
+	fDialog   = lipgloss.NewStyle().Border(lipgloss.DoubleBorder()).BorderForeground(tuiAccent).Padding(1, 2)
+)
+
+const navWidth = 24
+
+// render draws the whole screen, exactly Width by Height.
+func (f frame) render() string {
+	w, h := max(f.Width, 60), max(f.Height, 16)
+
+	title := " " + f.Title
+	right := f.Where + " "
+	gap := max(1, w-ansi.StringWidth(title)-ansi.StringWidth(right))
+	bar := fBar.Render(title + strings.Repeat(" ", gap) + right)
+
+	// Panels fill what the bars leave: one line each above and below.
+	ph := h - 2
+	inner := ph - 2 // borders
+
+	var nav strings.Builder
+	for _, it := range f.Nav {
+		mark := "  "
+		label := it.Label
+		if it.Heading {
+			nav.WriteString(sDim.Render(label) + "\n")
+			continue
+		}
+		switch {
+		case it.State == "done":
+			mark = sOK.Render("✓ ")
+		case it.State == "current":
+			mark = fSelected.Render("› ")
+			label = fSelected.Render(label)
+		case it.Current:
+			mark = fSelected.Render("› ")
+			label = fSelected.Render(label)
+		case it.Dim || it.State == "todo":
+			label = sDim.Render(label)
+		}
+		nav.WriteString(truncateWidth(mark+label, navWidth-2) + "\n")
+	}
+	left := box(fBorder, f.NavTitle, nav.String(), navWidth, inner)
+
+	pw := w - navWidth - 2 - 2
+	lines := strings.Split(strings.TrimRight(f.Body, "\n"), "\n")
+	scroll := min(max(f.Scroll, 0), max(0, len(lines)-inner))
+	visible := lines[scroll:]
+	if len(visible) > inner {
+		visible = visible[:inner]
+		// Said in the last line, where it is seen.
+		visible[inner-1] = sDim.Render("  ↓ more — PgDn")
+	}
+	for i := range visible {
+		visible[i] = truncateWidth(visible[i], pw)
+	}
+	panelTitle := f.PanelTitle
+	if scroll > 0 {
+		panelTitle += " ↑"
+	}
+	main := box(fFocus, panelTitle, strings.Join(visible, "\n"), pw, inner)
+
+	keys := fKeys.Render(" " + truncateWidth(f.Keys, w-2))
+	screen := bar + "\n" + lipgloss.JoinHorizontal(lipgloss.Top, left, main) + "\n" + keys
+
+	if f.Dialog != "" {
+		dw := min(w-8, max(50, maxLineWidth(f.Dialog)+6))
+		d := fDialog.Width(dw).Render(f.Dialog)
+		if f.DialogTitle != "" {
+			d = titled(d, f.DialogTitle)
+		}
+		screen = overlay(screen, d, w, h)
+	}
+	return screen
+}
+
+// box draws content in a border of the given inner size, with a title set
+// into the top edge.
+func box(style lipgloss.Style, title, content string, width, height int) string {
+	b := style.Width(width).Height(height).Render(content)
+	if title == "" {
+		return b
+	}
+	return titled(b, title)
+}
+
+// titled sets " title " into a box's top border, after its corner.
+func titled(b, title string) string {
+	lines := strings.Split(b, "\n")
+	if len(lines) == 0 {
+		return b
+	}
+	top := lines[0]
+	t := " " + title + " "
+	tw := ansi.StringWidth(t)
+	if ansi.StringWidth(top) < tw+4 {
+		return b
+	}
+	lines[0] = ansi.Truncate(top, 2, "") + sTitle.Render(t) + ansi.TruncateLeft(top, 2+tw, "")
+	return strings.Join(lines, "\n")
+}
+
+// overlay draws top over the middle of base, both w by h.
+func overlay(base, top string, w, h int) string {
+	bl := strings.Split(base, "\n")
+	tl := strings.Split(top, "\n")
+	tw := maxLineWidth(top)
+	x := max(0, (w-tw)/2)
+	y := max(0, (h-len(tl))/2)
+	for i, l := range tl {
+		row := y + i
+		if row >= len(bl) {
+			break
+		}
+		under := bl[row]
+		left := ansi.Truncate(under, x, "")
+		if pad := x - ansi.StringWidth(left); pad > 0 {
+			left += strings.Repeat(" ", pad)
+		}
+		right := ansi.TruncateLeft(under, x+ansi.StringWidth(l), "")
+		bl[row] = left + l + right
+	}
+	return strings.Join(bl, "\n")
+}
+
+func maxLineWidth(s string) int {
+	n := 0
+	for _, l := range strings.Split(s, "\n") {
+		n = max(n, ansi.StringWidth(l))
+	}
+	return n
+}
+
+// truncateWidth cuts a line, colours and all, to w cells.
+func truncateWidth(s string, w int) string {
+	if ansi.StringWidth(s) <= w {
+		return s
+	}
+	return ansi.Truncate(s, max(w-1, 0), "…")
+}
+
+// table draws rows under a header in columns as wide as they need, cut to
+// width; the selected row is shown reversed when the panel has the focus.
+func table(headers []string, rows [][]string, selected int, width int, focused bool) string {
+	widths := make([]int, len(headers))
+	for i, h := range headers {
+		widths[i] = ansi.StringWidth(h)
+	}
+	for _, r := range rows {
+		for i := range headers {
+			if i < len(r) {
+				widths[i] = min(max(widths[i], ansi.StringWidth(r[i])), 44)
+			}
+		}
+	}
+	line := func(cells []string) string {
+		var b strings.Builder
+		for i := range headers {
+			c := ""
+			if i < len(cells) {
+				c = truncateWidth(cells[i], widths[i])
+			}
+			b.WriteString(c)
+			if i < len(headers)-1 {
+				b.WriteString(strings.Repeat(" ", widths[i]-ansi.StringWidth(c)+2))
+			}
+		}
+		return truncateWidth(b.String(), width)
+	}
+	var b strings.Builder
+	b.WriteString("  " + lipgloss.NewStyle().Bold(true).Foreground(tuiDim).Render(line(headers)) + "\n")
+	for i, r := range rows {
+		l := line(r)
+		switch {
+		case i == selected && focused:
+			pad := max(0, width-ansi.StringWidth(ansi.Strip(l)))
+			l = fBar.Render(ansi.Strip(l) + strings.Repeat(" ", pad))
+		case i == selected:
+			l = lipgloss.NewStyle().Bold(true).Render(l)
+		}
+		b.WriteString("  " + l + "\n")
+	}
+	if len(rows) == 0 {
+		b.WriteString("  " + sDim.Render("nothing yet") + "\n")
+	}
+	return b.String()
+}

internal/cli/tui_home.go

@@ -0,0 +1,48 @@
+package cli
+
+import (
+	"errors"
+	"fmt"
+	"os"
+	"runtime"
+
+	"github.com/mattn/go-isatty"
+	"github.com/spf13/cobra"
+)
+
+// A bare `llmgw` in a terminal: the setup screens on a machine with nothing
+// installed, the console on a control plane. Anywhere else — a pipe, a
+// script, another operating system — it prints its help, as it always did.
+
+func interactive(cmd *cobra.Command) bool {
+	return isTerminal(cmd.OutOrStdout()) && isatty.IsTerminal(os.Stdin.Fd())
+}
+
+func runHome(cmd *cobra.Command) error {
+	out := cmd.OutOrStdout()
+	if runtime.GOOS != "linux" {
+		fmt.Fprintln(out, "The setup and console screens run on the Linux machine they manage.")
+		fmt.Fprintln(out)
+		return cmd.Help()
+	}
+	if os.Geteuid() != 0 {
+		return errors.New("run it as root — sudo llmgw — since it installs software and holds the certificate authority")
+	}
+	switch {
+	// Installed is the service in place, not only its configuration: a setup
+	// that stopped halfway has written server.yaml and nothing that runs, and
+	// the setup screens are what finish it.
+	case exists("/etc/llmgw/server.yaml") && exists("/etc/systemd/system/llmgw-server.service"):
+		return runConsole(cmd.Context(), out, "/etc/llmgw/server.yaml")
+	case exists("/etc/llmgw/agent.yaml"):
+		// A node: its own settings. The cluster is watched from the
+		// control plane.
+		return runConsole(cmd.Context(), out, "/etc/llmgw/agent.yaml")
+	}
+	return runSetupTUI(cmd.Context(), out)
+}
+
+func exists(p string) bool {
+	_, err := os.Stat(p)
+	return err == nil
+}

internal/cli/tui_init.go

@@ -77,10 +77,14 @@ type initModel struct {
 	next     []string // the closing "Next:" text
 	spin     spinnerModel
 	width    int
-	done     bool
-	err      error
-	quit     bool
-	start    func()
+	height   int
+	// framed draws it in the full-screen frame; off when another screen
+	// shows it in its own panel.
+	framed bool
+	done   bool
+	err    error
+	quit   bool
+	start  func()
 }
 
 func newInitModel(in *serverInit, found []string, askForm bool) initModel {
@@ -118,7 +122,10 @@ type startInitMsg struct{}
 func (m initModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
 	switch msg := msg.(type) {
 	case tea.WindowSizeMsg:
-		m.width = msg.Width
+		m.width, m.height = msg.Width, msg.Height
+		if m.framed {
+			m.width = msg.Width - navWidth - 6
+		}
 	case spinTickMsg:
 		var cmd tea.Cmd
 		m.spin, cmd = m.spin.Update(msg)
@@ -132,12 +139,18 @@ func (m initModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
 		m.addLine(string(msg))
 	case initDoneMsg:
 		m.done, m.err = true, msg.err
+		if m.framed {
+			return m, nil // left on screen until a key: it is gone once closed
+		}
 		return m, tea.Quit
 	case tea.KeyMsg:
 		if msg.Type == tea.KeyCtrlC {
 			m.quit = true
 			return m, tea.Quit
 		}
+		if m.done && m.framed && (msg.Type == tea.KeyEnter || msg.String() == "q") {
+			return m, tea.Quit
+		}
 		if m.form {
 			return m.formKey(msg)
 		}
@@ -240,6 +253,30 @@ func (m *initModel) moveFocus(d int) {
 }
 
 func (m initModel) View() string {
+	if !m.framed {
+		return m.plainView()
+	}
+	body := m.plainView()
+	if i := strings.Index(body, "\n\n"); i >= 0 {
+		body = body[i+1:]
+	}
+	steps := []navItem{{Label: "Start", State: "done"}, {Label: "Settings", State: "current"}, {Label: "Install", State: "todo"}}
+	keys := "Tab/↑↓ move · Space tick · Enter on Set up · Ctrl+C leave"
+	if !m.form {
+		steps[1].State, steps[2].State = "done", "current"
+		keys = "Ctrl+C stop"
+		if m.done {
+			steps[2].State = "done"
+			keys = "Enter close"
+		}
+	}
+	host, _ := os.Hostname()
+	return frame{Width: m.width + navWidth + 6, Height: m.height, Title: "llmgw setup", Where: host,
+		NavTitle: "Steps", Nav: steps, PanelTitle: "This machine as a control plane", Body: body, Keys: keys}.render()
+}
+
+// plainView is the screen as text, for a panel or a terminal that is not one.
+func (m initModel) plainView() string {
 	var b strings.Builder
 	b.WriteString(sTitle.Render("llmgw · setting up this machine as a control plane") + "\n\n")
 	if m.form {
@@ -338,7 +375,9 @@ func (m initModel) formView() string {
 	if m.formError != "" {
 		b.WriteString("\n" + sBad.Render(m.formError) + "\n")
 	}
-	b.WriteString("\n" + sDim.Render("tab or ↑↓ to move · space to tick · enter on Set up to start · ctrl+c to leave") + "\n")
+	if !m.framed { // the frame's key bar says it
+		b.WriteString("\n" + sDim.Render("tab or ↑↓ to move · space to tick · enter on Set up to start · ctrl+c to leave") + "\n")
+	}
 	return b.String()
 }
 
@@ -350,6 +389,7 @@ func (in *serverInit) runInitTUI(ctx context.Context, out io.Writer) error {
 		found = append(found, ip.String())
 	}
 	m := newInitModel(in, found, in.admin == "")
+	m.framed = true
 	var p *tea.Program
 
 	var (
@@ -373,7 +413,7 @@ func (in *serverInit) runInitTUI(ctx context.Context, out io.Writer) error {
 			p.Send(initDoneMsg{err})
 		}()
 	}
-	p = tea.NewProgram(m, tea.WithOutput(out))
+	p = tea.NewProgram(m, tea.WithOutput(out), tea.WithAltScreen())
 	final, err := p.Run()
 	if err != nil {
 		return err

internal/cli/tui_node.go

@@ -68,6 +68,7 @@ type nodeModel struct {
 	spin    spinnerModel
 	cancel  context.CancelFunc
 	width   int
+	height  int
 	done    bool
 	err     error
 	node    string
@@ -91,7 +92,7 @@ func (m nodeModel) Init() tea.Cmd { return m.spin.Tick }
 func (m nodeModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
 	switch msg := msg.(type) {
 	case tea.WindowSizeMsg:
-		m.width = msg.Width
+		m.width, m.height = msg.Width, msg.Height
 	case spinTickMsg:
 		var cmd tea.Cmd
 		m.spin, cmd = m.spin.Update(msg)
@@ -131,9 +132,13 @@ func (m nodeModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
 	case askContinueMsg:
 		m.confirm = &msg
 	case finishMsg:
+		// Left on the screen until a key: the frame is gone once it closes.
 		m.done, m.err, m.node, m.summary = true, msg.err, msg.node, msg.summary
-		return m, tea.Quit
+		return m, nil
 	case tea.KeyMsg:
+		if m.done && (msg.Type == tea.KeyEnter || msg.String() == "q" || msg.Type == tea.KeyEsc) {
+			return m, tea.Quit
+		}
 		return m.key(msg)
 	}
 	return m, nil
@@ -197,10 +202,52 @@ func (m *nodeModel) move(d int) {
 	m.inputs[m.focus].Focus()
 }
 
+// View draws the onboarding in the frame: its steps down the side, what each
+// did in the panel, a question as a dialog.
 func (m nodeModel) View() string {
+	var nav []navItem
+	current := false
+	for _, step := range onboard.Steps {
+		st, ok := m.steps[step]
+		state := "todo"
+		switch {
+		case ok && st.level == onboard.Running && !m.done && !current:
+			state, current = "current", true
+		case ok && st.level != onboard.Running:
+			state = "done"
+		}
+		nav = append(nav, navItem{Label: step, State: state})
+	}
+	keys := "Ctrl+C stop"
+	if m.done {
+		keys = "Enter close"
+	}
+	host, _ := os.Hostname()
+	f := frame{
+		Width: m.width, Height: m.height, Title: "llmgw · add a node", Where: host,
+		NavTitle: "Steps", Nav: nav, PanelTitle: m.title, Body: "\n" + m.content(), Keys: keys,
+	}
+	switch {
+	case m.fabrics != nil:
+		var q strings.Builder
+		q.WriteString("These adapters are up. Which fabric is each group on?\n")
+		q.WriteString(sDim.Render("Give each the name it has on every other node attached to it.\nLeave a name empty to leave that group out.") + "\n\n")
+		for i, fb := range m.fabrics.suggested {
+			fmt.Fprintf(&q, "  %s  %s\n", m.inputs[i].View(), sDim.Render(strings.Join(fb.HCAs, ", ")))
+		}
+		q.WriteString("\n" + sDim.Render("Tab next · Enter use these · Esc none"))
+		f.DialogTitle, f.Dialog = "Fabrics", q.String()
+	case m.confirm != nil:
+		f.DialogTitle = "Problems found"
+		f.Dialog = fmt.Sprintf("doctor found %d problem(s) on the machine.\nAdd the node anyway?\n\n%s", m.confirm.fails, sDim.Render("Y yes · N no"))
+	}
+	return f.render()
+}
+
+// content is the onboarding's steps and notes, as the panel shows them.
+func (m nodeModel) content() string {
 	var b strings.Builder
-	b.WriteString(sTitle.Render(m.title) + "\n\n")
-	detailWidth := max(20, m.width-22)
+	detailWidth := max(20, m.width-navWidth-26)
 	for _, step := range onboard.Steps {
 		st, ok := m.steps[step]
 		var mark string
@@ -238,26 +285,12 @@ func (m nodeModel) View() string {
 	}
 
 	switch {
-	case m.fabrics != nil:
-		var q strings.Builder
-		q.WriteString("These adapters are up. Which fabric is each group on?\n")
-		q.WriteString(sDim.Render("Give each the name it has on every other node attached to it.\nLeave a name empty to leave that group out.") + "\n\n")
-		for i, f := range m.fabrics.suggested {
-			fmt.Fprintf(&q, "  %s  %s\n", m.inputs[i].View(), sDim.Render(strings.Join(f.HCAs, ", ")))
-		}
-		q.WriteString("\n" + sDim.Render("tab next · enter use these · esc none"))
-		b.WriteString("\n" + sBox.Render(q.String()) + "\n")
-	case m.confirm != nil:
-		q := fmt.Sprintf("doctor found %d problem(s) above.\nAdd the node anyway? %s", m.confirm.fails, sDim.Render("y / N"))
-		b.WriteString("\n" + sBox.BorderForeground(tuiWarn).Render(q) + "\n")
 	case m.done && m.err != nil:
 		b.WriteString("\n" + sBad.Render("✗ ") + wrap(m.err.Error(), m.width-4, "  ") + "\n")
 	case m.done:
 		b.WriteString("\n" + sOK.Render("✓ ") + m.summary + "\n")
 	case m.quitAsked:
 		b.WriteString("\n" + sDim.Render("stopped") + "\n")
-	default:
-		b.WriteString("\n" + sDim.Render("ctrl+c to stop") + "\n")
 	}
 	return b.String()
 }
@@ -298,7 +331,7 @@ func (na *nodeAdd) runTUI(ctx context.Context, o onboard.Options, title string)
 	ctx, cancel := context.WithCancel(ctx)
 	defer cancel()
 	m := newNodeModel(title, cancel)
-	p := tea.NewProgram(m)
+	p := tea.NewProgram(m, tea.WithAltScreen())
 
 	emit := func(e onboard.Event) { p.Send(evMsg(e)) }
 	o.AskFabrics = na.rememberFabrics(func(s []onboard.Fabric) []onboard.Fabric {
@@ -343,6 +376,8 @@ func (na *nodeAdd) runTUI(ctx context.Context, o onboard.Options, title string)
 		return "", err
 	}
 	fm := final.(nodeModel)
+	// The screen is gone; what it ended with is said again where it stays.
+	fmt.Println(fm.content())
 	if fm.quitAsked && !fm.done {
 		cancel()
 		return fm.node, fmt.Errorf("stopped; the node may be half set up, and running this again finishes it")

internal/cli/tui_setup.go

@@ -0,0 +1,949 @@
+package cli
+
+import (
+	"context"
+	"fmt"
+	"io"
+	"net/mail"
+	"os"
+	"strings"
+	"sync"
+	"time"
+
+	tea "github.com/charmbracelet/bubbletea"
+	"github.com/charmbracelet/lipgloss"
+
+	joinpkg "gitea.mynodes.xyz/ilker/llmgw/internal/join"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/pgha"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/preflight"
+)
+
+// The setup screens: what a bare `llmgw` shows on a machine with nothing
+// installed. One machine, or three for high availability — this one as the
+// control plane and etcd's third vote, two more for the store under Patroni —
+// with every machine checked before anything is installed on any of them.
+//
+// The other machines are reached with Go's own ssh rather than the system's,
+// so a password can be typed here when there is no key; it is held in memory
+// for the run and written nowhere. A sudo password, likewise, is used once to
+// let the user sudo for the length of the install, and that is taken back
+// when it finishes.
+
+type setupPage int
+
+const (
+	pWelcome setupPage = iota
+	pAdmin
+	pMachines
+	// pWait shows the codes for machines joining rather than logged into,
+	// until they have all dialled in.
+	pWait
+	pCheck
+	pPlan
+	pRun
+)
+
+// How a database machine is logged into.
+const (
+	authAgent = iota
+	authKey
+	authPassword
+	// authJoin is no login at all: the machine runs `llmgw join` with a code
+	// shown here and dials out to this one.
+	authJoin
+	authCount
+)
+
+var authNames = [authCount]string{"ssh agent", "key file", "password", "join code"}
+
+// Fields of one database machine, in order.
+const (
+	mfAddr = iota
+	mfUser
+	mfAuth
+	mfSecret
+	mfSudo
+	mfCount
+)
+
+type dbMachine struct {
+	addr, user, secret, sudo textField
+	auth                     int
+	// For a machine joining: its code, and it once it has dialled in.
+	code   string
+	joined *joinpkg.Remote
+	arrive <-chan *joinpkg.Remote
+}
+
+// visible reports whether a field is shown, and so can be moved to: a joining
+// machine is asked nothing but how it joins.
+func (d dbMachine) visible(f int) bool {
+	switch {
+	case d.auth == authJoin:
+		return f == mfAuth
+	case f == mfSecret:
+		return d.auth != authAgent
+	}
+	return true
+}
+
+// dbConn is a database machine as it is reached, whichever way.
+type dbConn struct {
+	remote onboard.Remote
+	addr   string
+	ssh    *onboard.GoSSH
+	joined *joinpkg.Remote
+}
+
+func newDBMachine(n int) dbMachine {
+	f := func(ph string, mask bool) textField {
+		t := newTextField()
+		t.Placeholder, t.Mask, t.CharLimit, t.Width = ph, mask, 200, 34
+		return t
+	}
+	m := dbMachine{
+		addr:   f(fmt.Sprintf("10.0.0.%d or db%d.internal", 10+n, n), false),
+		user:   f("ubuntu", false),
+		secret: f("", true),
+		sudo:   f("only if sudo asks for one", true),
+	}
+	if os.Getenv("SSH_AUTH_SOCK") == "" {
+		m.auth = authKey
+	}
+	return m
+}
+
+// field is the text field at a machine's position, nil for the selector.
+func (d *dbMachine) field(i int) *textField {
+	switch i {
+	case mfAddr:
+		return &d.addr
+	case mfUser:
+		return &d.user
+	case mfSecret:
+		return &d.secret
+	case mfSudo:
+		return &d.sudo
+	}
+	return nil
+}
+
+// Messages from the work behind the screens.
+type (
+	checkDoneMsg struct{ rep preflight.Report }
+	joinedMsg    struct {
+		i int
+		r *joinpkg.Remote
+	}
+)
+
+// progRef lets work started from Update send to the program it runs in.
+type progRef struct{ p *tea.Program }
+
+type setupModel struct {
+	ref *progRef
+
+	page   setupPage
+	host   string
+	ips    []string
+	choice int // 0 one machine, 1 high availability
+
+	admin, names textField
+	afocus       int // admin, names, next
+
+	db     [2]dbMachine
+	mfocus int // over both machines' fields, then the check button
+
+	conns    []dbConn
+	listener *joinpkg.Listener
+	report   preflight.Report
+	checking bool
+
+	in       *serverInit
+	run      initModel
+	finished bool
+	log      *[]string
+	logMu    *sync.Mutex
+
+	spin    spinnerModel
+	formErr string
+	width   int
+	height  int
+	// scroll is how far the panel is scrolled, reset with each page.
+	scroll int
+	quit   bool
+	// single is the choice of one machine: the screen closes and init's own
+	// form takes over.
+	single bool
+}
+
+func newSetupModel(host string, ips []string) setupModel {
+	admin := newTextField()
+	admin.Placeholder, admin.CharLimit, admin.Width = "you@example.com", 200, 40
+	names := newTextField()
+	names.Placeholder, names.CharLimit, names.Width = "llm.example.com, 203.0.113.10", 400, 40
+	return setupModel{
+		ref: &progRef{}, host: host, ips: ips, admin: admin, names: names,
+		db: [2]dbMachine{newDBMachine(1), newDBMachine(2)}, spin: newSpinner(), width: 100, height: 30,
+		log: &[]string{}, logMu: &sync.Mutex{},
+	}
+}
+
+func (m setupModel) Init() tea.Cmd { return m.spin.Tick }
+
+func (m setupModel) thisAddr() string {
+	if len(m.ips) > 0 {
+		return m.ips[0]
+	}
+	return m.host
+}
+
+func (m setupModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
+	switch msg := msg.(type) {
+	case tea.WindowSizeMsg:
+		m.width, m.height = msg.Width, msg.Height
+		m.run.width = msg.Width - navWidth - 6
+	case spinTickMsg:
+		var cmd tea.Cmd
+		m.spin, cmd = m.spin.Update(msg)
+		m.run.spin = m.spin
+		return m, cmd
+	case checkDoneMsg:
+		m.checking, m.report = false, msg.rep
+	case joinedMsg:
+		m.db[msg.i].joined = msg.r
+		if m.page == pWait && m.allJoined() {
+			m.page = pCheck
+			cmd := m.startCheck()
+			return m, cmd
+		}
+	case initLineMsg:
+		m.run.addLine(string(msg))
+	case initDoneMsg:
+		m.run.done, m.run.err, m.finished = true, msg.err, true
+	case tea.KeyMsg:
+		if msg.Type == tea.KeyCtrlC {
+			m.quit = true
+			return m, tea.Quit
+		}
+		before := m.page
+		next, cmd := m.key(msg)
+		if nm := next.(setupModel); nm.page != before {
+			// A new page starts at its top.
+			nm.scroll = 0
+			return nm, cmd
+		}
+		return next, cmd
+	}
+	return m, nil
+}
+
+func (m setupModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) {
+	// A dialog open takes the next key, whatever it is, to close it.
+	if m.formErr != "" {
+		m.formErr = ""
+		return m, nil
+	}
+	switch k.Type {
+	case tea.KeyPgDown:
+		m.scroll += max(1, m.height-8)
+		return m, nil
+	case tea.KeyPgUp:
+		m.scroll = max(0, m.scroll-max(1, m.height-8))
+		return m, nil
+	}
+	if k.Type == tea.KeyEsc && m.page > pWelcome && m.page < pRun && !m.checking {
+		m.page--
+		if m.page == pWait {
+			m.page = pMachines
+		}
+		m.formErr = ""
+		return m, nil
+	}
+	switch m.page {
+	case pWelcome:
+		switch k.String() {
+		case "up", "down", "tab", "k", "j":
+			m.choice = 1 - m.choice
+		case "enter":
+			if m.choice == 0 {
+				m.single = true
+				return m, tea.Quit
+			}
+			m.page = pAdmin
+			m.afocus = 0
+			m.admin.Focus()
+		}
+	case pAdmin:
+		return m.adminKey(k)
+	case pMachines:
+		return m.machinesKey(k)
+	case pCheck:
+		switch k.String() {
+		case "r":
+			if !m.checking {
+				cmd := m.startCheck()
+				return m, cmd
+			}
+		case "enter":
+			if !m.checking && m.report.Worst() < preflight.Fail {
+				m.page = pPlan
+			}
+		}
+	case pPlan:
+		if k.String() == "enter" {
+			m.page = pRun
+			cmd := m.startRun()
+			return m, cmd
+		}
+	case pRun:
+		if m.finished && (k.String() == "q" || k.String() == "enter") {
+			return m, tea.Quit
+		}
+	}
+	return m, nil
+}
+
+func (m setupModel) adminKey(k tea.KeyMsg) (tea.Model, tea.Cmd) {
+	move := func(d int) {
+		m.afocus = (m.afocus + d + 3) % 3
+		m.admin.Blur()
+		m.names.Blur()
+		switch m.afocus {
+		case 0:
+			m.admin.Focus()
+		case 1:
+			m.names.Focus()
+		}
+	}
+	switch k.Type {
+	case tea.KeyTab, tea.KeyDown:
+		move(1)
+		return m, nil
+	case tea.KeyShiftTab, tea.KeyUp:
+		move(-1)
+		return m, nil
+	case tea.KeyEnter:
+		if m.afocus < 2 {
+			move(1)
+			return m, nil
+		}
+		if _, err := mail.ParseAddress(strings.TrimSpace(m.admin.Value())); err != nil {
+			m.formErr = "the administrator's email is needed, and that is not one"
+			m.afocus = 0
+			move(0)
+			return m, nil
+		}
+		m.formErr = ""
+		m.page, m.mfocus = pMachines, 0
+		m.db[0].addr.Focus()
+		return m, nil
+	}
+	var cmd tea.Cmd
+	switch m.afocus {
+	case 0:
+		m.admin, cmd = m.admin.Update(k)
+	case 1:
+		m.names, cmd = m.names.Update(k)
+	}
+	return m, cmd
+}
+
+func (m *setupModel) focusMachine() {
+	for i := range m.db {
+		for f := 0; f < mfCount; f++ {
+			if t := m.db[i].field(f); t != nil {
+				t.Blur()
+			}
+		}
+	}
+	if m.mfocus < 2*mfCount {
+		if t := m.db[m.mfocus/mfCount].field(m.mfocus % mfCount); t != nil {
+			t.Focus()
+		}
+	}
+}
+
+func (m setupModel) machinesKey(k tea.KeyMsg) (tea.Model, tea.Cmd) {
+	n := 2*mfCount + 1
+	switch k.Type {
+	case tea.KeyTab, tea.KeyDown:
+		m.step(1, n)
+		return m, nil
+	case tea.KeyShiftTab, tea.KeyUp:
+		m.step(-1, n)
+		return m, nil
+	case tea.KeyEnter:
+		if m.mfocus < 2*mfCount {
+			m.step(1, n)
+			return m, nil
+		}
+		if err := m.validateMachines(); err != "" {
+			m.formErr = err
+			return m, nil
+		}
+		m.formErr = ""
+		if cmd, waiting := m.startJoins(); waiting {
+			m.page = pWait
+			return m, cmd
+		}
+		m.page = pCheck
+		cmd := m.startCheck()
+		return m, cmd
+	}
+	if m.mfocus == 2*mfCount {
+		return m, nil
+	}
+	d := &m.db[m.mfocus/mfCount]
+	if m.mfocus%mfCount == mfAuth {
+		switch k.String() {
+		case "left", "h":
+			d.auth = (d.auth + authCount - 1) % authCount
+		case "right", "l", " ":
+			d.auth = (d.auth + 1) % authCount
+		}
+		if d.auth != authJoin && d.joined != nil {
+			_ = d.joined.Close()
+			d.joined, d.code, d.arrive = nil, "", nil
+		}
+		d.secret.SetValue("")
+		d.secret.Mask = d.auth == authPassword
+		d.secret.Placeholder = map[int]string{authKey: "/root/.ssh/id_ed25519", authPassword: "the user's ssh password"}[d.auth]
+		return m, nil
+	}
+	var cmd tea.Cmd
+	t := d.field(m.mfocus % mfCount)
+	*t, cmd = t.Update(k)
+	return m, cmd
+}
+
+// step moves the focus over the fields that are shown.
+func (m *setupModel) step(d, n int) {
+	for range n {
+		m.mfocus = (m.mfocus + d + n) % n
+		if m.mfocus == 2*mfCount || m.db[m.mfocus/mfCount].visible(m.mfocus%mfCount) {
+			break
+		}
+	}
+	m.focusMachine()
+}
+
+func (m setupModel) allJoined() bool {
+	for _, d := range m.db {
+		if d.auth == authJoin && d.joined == nil {
+			return false
+		}
+	}
+	return true
+}
+
+// startJoins listens for the machines that join with a code, makes their
+// codes, and waits for them; it says whether there is anything to wait for.
+func (m *setupModel) startJoins() (tea.Cmd, bool) {
+	var cmds []tea.Cmd
+	for i := range m.db {
+		d := &m.db[i]
+		if d.auth != authJoin || d.joined != nil {
+			continue
+		}
+		if m.listener == nil {
+			l, err := joinpkg.Listen(fmt.Sprintf(":%d", joinpkg.DefaultPort),
+				fmt.Sprintf("%s:%d", m.thisAddr(), joinpkg.DefaultPort), time.Hour)
+			if err != nil {
+				m.formErr = err.Error()
+				return nil, false
+			}
+			m.listener = l
+		}
+		if d.arrive == nil {
+			c, ch := m.listener.Code("database")
+			d.code, d.arrive = c.String(), ch
+		}
+		ch, i := d.arrive, i
+		cmds = append(cmds, func() tea.Msg { return joinedMsg{i, <-ch} })
+	}
+	return tea.Batch(cmds...), len(cmds) > 0
+}
+
+func (m setupModel) validateMachines() string {
+	seen := map[string]bool{m.thisAddr(): true, m.host: true}
+	for i, d := range m.db {
+		if d.auth == authJoin {
+			continue
+		}
+		addr := strings.TrimSpace(d.addr.Value())
+		switch {
+		case addr == "":
+			return fmt.Sprintf("database machine %d has no address", i+1)
+		case seen[addr]:
+			return fmt.Sprintf("%s is given twice, or is this machine: the three must be different machines", addr)
+		case strings.TrimSpace(d.user.Value()) == "":
+			return fmt.Sprintf("database machine %d has no user to log in as", i+1)
+		case d.auth != authAgent && d.secret.Value() == "":
+			return fmt.Sprintf("database machine %d needs its %s", i+1, authNames[d.auth])
+		}
+		seen[addr] = true
+	}
+	return ""
+}
+
+// startCheck connects to the database machines and runs the checks on all
+// three, this one included.
+func (m *setupModel) startCheck() tea.Cmd {
+	for _, c := range m.conns {
+		if c.ssh != nil {
+			_ = c.ssh.Close()
+		}
+	}
+	m.conns = nil
+	machines := []preflight.Machine{{
+		Name: m.host + " (this machine)", Addr: m.thisAddr(), Local: true, Remote: pgha.Local{},
+		Roles: []preflight.Role{preflight.RoleControl, preflight.RoleWitness},
+	}}
+	for _, d := range m.db {
+		if d.auth == authJoin {
+			m.conns = append(m.conns, dbConn{remote: d.joined, addr: d.joined.Addr, joined: d.joined})
+			machines = append(machines, preflight.Machine{Name: d.joined.Hostname + " (" + d.joined.Addr + ", joined)",
+				Addr: d.joined.Addr, Roles: []preflight.Role{preflight.RoleDatabase}, Remote: d.joined})
+			continue
+		}
+		g := &onboard.GoSSH{
+			Addr: strings.TrimSpace(d.addr.Value()), User: strings.TrimSpace(d.user.Value()),
+			UseAgent: d.auth == authAgent, SudoPassword: d.sudo.Value(),
+		}
+		switch d.auth {
+		case authKey:
+			g.KeyFile = strings.TrimSpace(d.secret.Value())
+		case authPassword:
+			g.Password = d.secret.Value()
+		}
+		m.conns = append(m.conns, dbConn{remote: g, addr: g.Addr, ssh: g})
+		machines = append(machines, preflight.Machine{Name: g.Addr, Addr: g.Addr,
+			Roles: []preflight.Role{preflight.RoleDatabase}, Remote: g})
+	}
+	m.checking = true
+	m.report = preflight.Report{}
+	conns := m.conns
+	return func() tea.Msg {
+		rep := preflight.Run(context.Background(), machines)
+		// A sudo that asks for a password is fine with one given: the
+		// install grants itself sudo with it.
+		for i := range rep.Findings {
+			f := &rep.Findings[i]
+			if f.Check == "sudo" && f.Level == preflight.Warn {
+				for _, c := range conns {
+					g := c.ssh
+					switch {
+					case g == nil || g.Addr != f.Machine:
+					case g.SudoPassword == "":
+						f.Level = preflight.Fail
+						f.Fix = "give this user's sudo password on the previous screen"
+					default:
+						f.Level = preflight.OK
+						f.Detail = "with the password given"
+						f.Fix = ""
+					}
+				}
+			}
+		}
+		return checkDoneMsg{rep}
+	}
+}
+
+// targets are the database machines as user@address, for the command line.
+func (m setupModel) targets() []string {
+	var out []string
+	for _, d := range m.db {
+		if d.joined != nil {
+			out = append(out, "root@"+d.joined.Addr)
+			continue
+		}
+		out = append(out, strings.TrimSpace(d.user.Value())+"@"+strings.TrimSpace(d.addr.Value()))
+	}
+	return out
+}
+
+func (m setupModel) namesList() []string {
+	var out []string
+	for _, h := range strings.Split(m.names.Value(), ",") {
+		if h = strings.TrimSpace(h); h != "" {
+			out = append(out, h)
+		}
+	}
+	return out
+}
+
+// command is what this setup is on the command line, for doing it again
+// from a script — with keys in place, since a password cannot be given there.
+func (m setupModel) command() string {
+	parts := []string{"sudo --preserve-env=SSH_AUTH_SOCK llmgw server init",
+		"--admin " + strings.TrimSpace(m.admin.Value())}
+	for _, h := range m.namesList() {
+		parts = append(parts, "--host "+h)
+	}
+	for _, t := range m.targets() {
+		parts = append(parts, "--patroni "+t)
+	}
+	parts = append(parts, "--no-node", "--no-model")
+	return strings.Join(parts, " \\\n    ")
+}
+
+// startRun installs: sudo granted where it asks for a password, the control
+// plane here, the store on the other two, then sudo taken back.
+func (m *setupModel) startRun() tea.Cmd {
+	in := newDefaultServerInit()
+	in.admin = strings.TrimSpace(m.admin.Value())
+	in.hosts = m.namesList()
+	in.patroni = m.targets()
+	// The control plane of a replicated setup is not where models run;
+	// nodes are added once it is up.
+	in.noNode, in.noModel = true, true
+	m.in = &in
+	m.run = newInitModel(&in, nil, false)
+	m.run.running, m.run.width, m.run.spin = true, m.width, m.spin
+	ref, conns, logp, mu, listener := m.ref, m.conns, m.log, m.logMu, m.listener
+	return func() tea.Msg {
+		send := func(l string) {
+			mu.Lock()
+			*logp = append(*logp, l)
+			mu.Unlock()
+			ref.p.Send(initLineMsg(l))
+		}
+		in.out = &lineWriter{send: send}
+		err := func() error {
+			ctx := context.Background()
+			for _, c := range conns {
+				if c.ssh != nil {
+					fmt.Fprintf(in.out, "  ..    sudo on %s\n", c.addr)
+					if err := c.ssh.PrepareSudo(ctx); err != nil {
+						return err
+					}
+				}
+				out, err := c.remote.Run(ctx, "hostname", nil)
+				if err != nil {
+					return fmt.Errorf("%s: %v", c.addr, err)
+				}
+				ip, err := ipv4Of(c.addr)
+				if err != nil {
+					return err
+				}
+				in.patroniNodes = append(in.patroniNodes, pgha.Node{
+					Member: pgha.Member{Name: strings.TrimSpace(string(out)), IP: ip}, Remote: c.remote})
+			}
+			return in.run(ctx)
+		}()
+		// Whatever happened: a sudo grant taken back, a joined machine's
+		// `llmgw join` told it is done, the port for joining closed.
+		for _, c := range conns {
+			if c.ssh != nil {
+				_ = c.ssh.Close()
+			}
+			if c.joined != nil {
+				_ = c.joined.Close()
+			}
+		}
+		if listener != nil {
+			_ = listener.Close()
+		}
+		return initDoneMsg{err}
+	}
+}
+
+// setupSteps are the sidebar's steps, by page.
+var setupSteps = []struct {
+	label string
+	page  setupPage
+}{
+	{"Start", pWelcome}, {"Administrator", pAdmin}, {"Machines", pMachines},
+	{"Checks", pCheck}, {"Plan", pPlan}, {"Install", pRun},
+}
+
+func (m setupModel) View() string {
+	at := m.page
+	if at == pWait {
+		at = pMachines
+	}
+	var nav []navItem
+	for _, st := range setupSteps {
+		if m.choice == 0 && st.page > pWelcome {
+			break
+		}
+		state := "todo"
+		switch {
+		case st.page < at || (st.page == pRun && m.finished && m.run.err == nil):
+			state = "done"
+		case st.page == at:
+			state = "current"
+		}
+		nav = append(nav, navItem{Label: st.label, State: state})
+	}
+	title, body, keys := m.pageView()
+	f := frame{
+		Width: m.width, Height: m.height,
+		Title: "llmgw setup", Where: m.host + " · " + m.thisAddr(),
+		NavTitle: "Steps", Nav: nav,
+		PanelTitle: title, Body: body, Scroll: m.scroll, Keys: keys,
+	}
+	if m.formErr != "" {
+		f.DialogTitle, f.Dialog = "Not yet", m.formErr+"\n\n"+sDim.Render("any key to go back")
+	}
+	return f.render()
+}
+
+// pageView is the panel for the page: its title, what is in it, and its keys.
+func (m setupModel) pageView() (string, string, string) {
+	switch m.page {
+	case pWelcome:
+		return "Start", m.welcomeView(), "↑↓ choose · Enter go on · Ctrl+C leave"
+	case pAdmin:
+		return "Administrator", m.adminView(), "Tab/↑↓ move · Enter next · Esc back · Ctrl+C leave"
+	case pMachines:
+		return "Database machines", m.machinesView(), "Tab/↑↓ move · ←→ how to log in · Enter next · Esc back · PgUp/PgDn scroll"
+	case pWait:
+		return "Waiting for machines to join", m.waitView(), "Esc back · PgUp/PgDn scroll · Ctrl+C leave"
+	case pCheck:
+		keys := "R check again · Esc back · PgUp/PgDn scroll"
+		if !m.checking && m.report.Worst() < preflight.Fail {
+			keys = "Enter see the plan · " + keys
+		}
+		return "Checks", m.checkView(), keys
+	case pPlan:
+		return "Plan", m.planView(), "Enter install · Esc back · PgUp/PgDn scroll · Ctrl+C leave"
+	}
+	v := m.run.View()
+	if i := strings.Index(v, "\n\n"); i >= 0 {
+		v = v[i+2:] // init's own title; the frame has one
+	}
+	keys := "Ctrl+C stop"
+	if m.finished {
+		keys = "Enter or Q leave · PgUp/PgDn scroll"
+		if m.run.err == nil {
+			v += "\n" + sDim.Render("The same, from a script, with ssh keys in place:") + "\n" + m.command() + "\n"
+		}
+	}
+	return "Install", v, keys
+}
+
+func (m setupModel) welcomeView() string {
+	var b strings.Builder
+	fmt.Fprintf(&b, "\n  Nothing of llmgw is installed on %s.\n\n", lipgloss.NewStyle().Bold(true).Render(m.host))
+	opt := func(i int, title, detail string) {
+		mark := "( )"
+		if m.choice == i {
+			mark = "(" + sOK.Render("•") + ")"
+			title = fSelected.Render(title)
+		}
+		fmt.Fprintf(&b, "  %s %s\n      %s\n\n", mark, title, sDim.Render(detail))
+	}
+	opt(0, "One machine", "This machine runs everything: control plane, model mirror,\n      and models if it has room.")
+	opt(1, "High availability — three machines",
+		"This one is the control plane; two more keep the store, replicated,\n      so losing any one machine loses nothing.")
+	return b.String()
+}
+
+func (m setupModel) adminView() string {
+	var b strings.Builder
+	label := func(i int, s string) string {
+		if m.afocus == i {
+			return fSelected.Render("› " + s)
+		}
+		return "  " + s
+	}
+	b.WriteString("\n")
+	fmt.Fprintf(&b, "%s\n    %s\n    %s\n\n", label(0, "Administrator's email"), m.admin.View(),
+		sDim.Render("A password is generated and shown once, at the end."))
+	fmt.Fprintf(&b, "%s\n    %s\n    %s\n\n", label(1, "Other names nodes will reach this machine by"), m.names.View(),
+		sDim.Render("Known: "+strings.Join(append([]string{m.host}, m.ips...), ", ")+". Add a DNS name or a floating address."))
+	b.WriteString(button(m.afocus == 2, "Next") + "\n")
+	return b.String()
+}
+
+// button is a [ Label ] that shows when it has the focus.
+func button(focused bool, label string) string {
+	if focused {
+		return "  " + fBar.Render(" "+label+" ")
+	}
+	return "  " + lipgloss.NewStyle().Bold(true).Render("[ "+label+" ]")
+}
+
+func (m setupModel) machinesView() string {
+	var b strings.Builder
+	b.WriteString("\n  The two machines that keep the store: Ubuntu 22.04/24.04 or Debian 12.\n" +
+		"  PostgreSQL, Patroni and etcd are installed on them.\n\n")
+	for i, d := range m.db {
+		base := i * mfCount
+		// Padded before it is coloured: the colour's escape codes count
+		// toward a width and threw the focused row out of line.
+		label := func(f int, s string) string {
+			s = fmt.Sprintf("%-15s", s)
+			if m.mfocus == base+f {
+				return fSelected.Render("› " + s)
+			}
+			return "  " + s
+		}
+		fmt.Fprintf(&b, "  %s\n", lipgloss.NewStyle().Bold(true).Render(fmt.Sprintf("Database machine %d", i+1)))
+		if d.visible(mfAddr) {
+			fmt.Fprintf(&b, "  %s %s\n", label(mfAddr, "Address"), d.addr.View())
+			fmt.Fprintf(&b, "  %s %s\n", label(mfUser, "User"), d.user.View())
+		}
+		var auth []string
+		for a := 0; a < authCount; a++ {
+			if a == d.auth {
+				auth = append(auth, "("+sOK.Render("•")+") "+authNames[a])
+			} else {
+				auth = append(auth, "( ) "+sDim.Render(authNames[a]))
+			}
+		}
+		fmt.Fprintf(&b, "  %s   %s\n", label(mfAuth, "Log in with"), strings.Join(auth, "  "))
+		if d.visible(mfSecret) {
+			fmt.Fprintf(&b, "  %s %s\n", label(mfSecret, strings.ToUpper(authNames[d.auth][:1])+authNames[d.auth][1:]), d.secret.View())
+		}
+		if d.visible(mfSudo) {
+			fmt.Fprintf(&b, "  %s %s\n", label(mfSudo, "Sudo password"), d.sudo.View())
+		}
+		if d.auth == authJoin {
+			b.WriteString("      " + sDim.Render("Nothing logs into it: it runs llmgw join with a code shown next.") + "\n")
+		}
+		b.WriteString("\n")
+	}
+	b.WriteString(button(m.mfocus == 2*mfCount, "Check the three machines") + "\n\n")
+	b.WriteString("  " + sDim.Render("Passwords are kept in memory only, for this setup.") + "\n")
+	return b.String()
+}
+
+func (m setupModel) waitView() string {
+	var b strings.Builder
+	fmt.Fprintf(&b, "\n  On each machine below, as root. It dials this machine on port %d;\n  nothing dials in. Each code is for one machine and good for an hour.\n\n", joinpkg.DefaultPort)
+	for i, d := range m.db {
+		if d.auth != authJoin {
+			continue
+		}
+		title := lipgloss.NewStyle().Bold(true).Render(fmt.Sprintf("Database machine %d", i+1))
+		if d.joined != nil {
+			fmt.Fprintf(&b, "  %s  %s joined as %s (%s)\n\n", title, sOK.Render("✓"), d.joined.Hostname, d.joined.Addr)
+			continue
+		}
+		fmt.Fprintf(&b, "  %s  %s waiting\n", title, m.spin.View())
+		b.WriteString("  " + sDim.Render("without llmgw yet:") + "\n")
+		for _, l := range joinInstall() {
+			b.WriteString("    " + strings.ReplaceAll(l, "\n", "\n    ") + "\n")
+		}
+		b.WriteString("  " + sDim.Render("then:") + "\n    sudo llmgw join " + d.code + "\n\n")
+	}
+	return b.String()
+}
+
+func (m setupModel) checkView() string {
+	var b strings.Builder
+	b.WriteString("\n")
+	if m.checking {
+		b.WriteString("  " + m.spin.View() + " checking the three machines…\n")
+		return b.String()
+	}
+	switch m.report.Worst() {
+	case preflight.Fail:
+		b.WriteString("  " + sBad.Render("✗ Fix what is marked ✗, then check again (R).") + "\n")
+	case preflight.Warn:
+		b.WriteString("  " + sWarn.Render("! It can go ahead; what is marked ! is worth a look.") + "\n")
+	default:
+		b.WriteString("  " + sOK.Render("✓ All three are ready.") + "\n")
+	}
+	machine := ""
+	for _, f := range m.report.Findings {
+		if f.Machine != machine {
+			machine = f.Machine
+			b.WriteString("\n  " + lipgloss.NewStyle().Bold(true).Render(machine) + "\n")
+		}
+		mark := sOK.Render("✓")
+		switch f.Level {
+		case preflight.Warn:
+			mark = sWarn.Render("!")
+		case preflight.Fail:
+			mark = sBad.Render("✗")
+		}
+		fmt.Fprintf(&b, "    %s %-13s %s\n", mark, f.Check, f.Detail)
+		if f.Level != preflight.OK && f.Fix != "" {
+			b.WriteString("                    " + sDim.Render(f.Fix) + "\n")
+		}
+	}
+	return b.String()
+}
+
+func (m setupModel) planView() string {
+	var b strings.Builder
+	b.WriteString("\n  What will be installed:\n\n")
+	fmt.Fprintf(&b, "  %s  %s\n      %s\n", lipgloss.NewStyle().Bold(true).Render(m.host), sDim.Render(m.thisAddr()),
+		"control plane and gateway, model mirror, etcd (the third vote)")
+	fmt.Fprintf(&b, "      administrator %s\n", strings.TrimSpace(m.admin.Value()))
+	granted := false
+	for _, d := range m.db {
+		addr := strings.TrimSpace(d.addr.Value())
+		if d.joined != nil {
+			addr = d.joined.Hostname + "  " + sDim.Render(d.joined.Addr+", joined")
+		}
+		fmt.Fprintf(&b, "  %s\n      PostgreSQL under Patroni, etcd\n", lipgloss.NewStyle().Bold(true).Render(addr))
+		granted = granted || d.sudo.Value() != ""
+	}
+	b.WriteString("\n  The store is replicated from the first to the second; if the primary is lost\n" +
+		"  the standby takes over in about thirty seconds, and the control plane follows.\n")
+	if granted {
+		b.WriteString("  " + sDim.Render("Where a sudo password was given, the user may sudo without it until the\n  install finishes, and not after.") + "\n")
+	}
+	b.WriteString("\n  " + sDim.Render("The same from a script, with ssh keys in place:") + "\n")
+	for _, l := range strings.Split(m.command(), "\n") {
+		b.WriteString("  " + l + "\n")
+	}
+	for _, d := range m.db {
+		if d.joined != nil {
+			b.WriteString("  " + sDim.Render("(a machine that joined with a code needs ssh for that)") + "\n")
+			break
+		}
+	}
+	b.WriteString("\n" + button(true, "Enter to install") + "\n")
+	return b.String()
+}
+
+// runSetupTUI is the setup screens, until they finish or hand over to init's
+// own form for one machine.
+func runSetupTUI(ctx context.Context, out io.Writer) error {
+	host, _ := os.Hostname()
+	var ips []string
+	for _, ip := range localAddrs() {
+		if ip.To4() != nil {
+			ips = append(ips, ip.String())
+		}
+	}
+	m := newSetupModel(host, ips)
+	p := tea.NewProgram(m, tea.WithOutput(out), tea.WithAltScreen())
+	m.ref.p = p
+	final, err := p.Run()
+	if err != nil {
+		return err
+	}
+	fm := final.(setupModel)
+	if fm.single {
+		in := newDefaultServerInit()
+		in.out = out
+		return in.runInitTUI(ctx, out)
+	}
+	if fm.page < pRun {
+		return nil
+	}
+	fmt.Fprintln(out)
+	if fm.run.err != nil {
+		fm.logMu.Lock()
+		fmt.Fprintln(out, strings.Join(*fm.log, "\n"))
+		fm.logMu.Unlock()
+		return fm.run.err
+	}
+	if !fm.finished {
+		return fmt.Errorf("left while installing; running llmgw again finishes what was started")
+	}
+	if fm.run.password != "" {
+		fmt.Fprintf(out, "Administrator %s, password %s (shown once)\n\n", fm.in.admin, fm.run.password)
+	}
+	fmt.Fprintln(out, "Next: add nodes from the console — run llmgw again — or with")
+	fmt.Fprintln(out, "  sudo llmgw node add <user>@<address>")
+	return nil
+}

internal/cli/tui_setup_test.go

@@ -0,0 +1,173 @@
+package cli
+
+import (
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+
+	tea "github.com/charmbracelet/bubbletea"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/preflight"
+)
+
+func press(m tea.Model, keys ...string) tea.Model {
+	for _, k := range keys {
+		var msg tea.KeyMsg
+		switch k {
+		case "enter":
+			msg = tea.KeyMsg{Type: tea.KeyEnter}
+		case "tab":
+			msg = tea.KeyMsg{Type: tea.KeyTab}
+		case "down":
+			msg = tea.KeyMsg{Type: tea.KeyDown}
+		case "esc":
+			msg = tea.KeyMsg{Type: tea.KeyEsc}
+		case "right":
+			msg = tea.KeyMsg{Type: tea.KeyRight}
+		default:
+			msg = tea.KeyMsg{Type: tea.KeyRunes, Runes: []rune(k)}
+		}
+		m, _ = m.Update(msg)
+	}
+	return m
+}
+
+func TestTheSetupAsksForWhatHighAvailabilityNeedsAndRefusesWhatWillNotDo(t *testing.T) {
+	m := tea.Model(newSetupModel("ctl", []string{"10.0.0.5"}))
+	m = press(m, "down", "enter")
+	if s := m.(setupModel); s.page != pAdmin {
+		t.Fatalf("high availability did not lead to the administrator: page %d", s.page)
+	}
+	// No email: refused, with a reason.
+	m = press(m, "tab", "tab", "enter")
+	if s := m.(setupModel); s.page != pAdmin || !strings.Contains(s.formErr, "email") {
+		t.Fatalf("went on without an administrator: %+v", s.formErr)
+	}
+	// The refusal is a dialog; a key closes it, and the cursor is back on
+	// the email.
+	if !strings.Contains(m.View(), "Not yet") {
+		t.Error("the refusal was not shown as a dialog")
+	}
+	m = press(m, "x", "a@example.com", "tab", "tab", "enter")
+	s := m.(setupModel)
+	if s.page != pMachines {
+		t.Fatalf("page %d: %s", s.page, s.formErr)
+	}
+	// This machine's own address is not a database machine.
+	s.db[0].addr.SetValue("10.0.0.5")
+	s.db[0].user.SetValue("ubuntu")
+	s.db[0].auth = authAgent
+	s.db[1].addr.SetValue("10.0.0.7")
+	s.db[1].user.SetValue("ubuntu")
+	s.db[1].auth = authPassword
+	if err := s.validateMachines(); !strings.Contains(err, "this machine") {
+		t.Errorf("this machine as a database machine: %q", err)
+	}
+	s.db[0].addr.SetValue("10.0.0.6")
+	if err := s.validateMachines(); !strings.Contains(err, "password") {
+		t.Errorf("a password login with no password: %q", err)
+	}
+	s.db[1].secret.SetValue("pw")
+	if err := s.validateMachines(); err != "" {
+		t.Errorf("a valid pair refused: %s", err)
+	}
+	cmd := s.command()
+	for _, want := range []string{"--admin a@example.com", "--patroni ubuntu@10.0.0.6", "--patroni ubuntu@10.0.0.7", "--no-node"} {
+		if !strings.Contains(cmd, want) {
+			t.Errorf("command lacks %q:\n%s", want, cmd)
+		}
+	}
+	if strings.Contains(cmd, "pw") {
+		t.Error("the password is in the command line")
+	}
+}
+
+func TestChoosingOneMachineHandsOverToInit(t *testing.T) {
+	m := press(tea.Model(newSetupModel("ctl", nil)), "enter").(setupModel)
+	if !m.single {
+		t.Error("one machine did not hand over")
+	}
+}
+
+func TestTheChecksHoldTheSetupBackOnlyForAFailure(t *testing.T) {
+	m := newSetupModel("ctl", nil)
+	m.choice, m.page = 1, pCheck
+	m.report = preflight.Report{Findings: []preflight.Finding{{Machine: "a", Check: "disk", Level: preflight.Fail, Fix: "more"}}}
+	if s := press(m, "enter").(setupModel); s.page != pCheck {
+		t.Error("went on past a failed check")
+	}
+	m.report.Findings[0].Level = preflight.Warn
+	if s := press(m, "enter").(setupModel); s.page != pPlan {
+		t.Error("a warning held it back")
+	}
+}
+
+func TestTheConsoleOffersToReplicateOnlyAStoreThatIsNot(t *testing.T) {
+	m := consoleModel{}
+	if !m.entryOK(cReplicate) {
+		t.Error("a single store is not offered replication")
+	}
+	m.data.sum.Store = []adminapi.StoreMember{{Name: "a", Role: "leader"}}
+	if m.entryOK(cReplicate) {
+		t.Error("a replicated store is offered replication")
+	}
+	// A node is asked for in a dialog, then the console closes with it to do.
+	var mm tea.Model = consoleModel{menu: cAddNode, loaded: true, width: 100, height: 30}
+	mm = press(mm, "enter", "ubuntu@10.0.0.9", "enter")
+	if a := mm.(consoleModel).action; a.kind != "node" || a.targets[0] != "ubuntu@10.0.0.9" {
+		t.Errorf("%+v", a)
+	}
+}
+
+func TestTheConsoleWatchesTheClusterAndEditsItsSettings(t *testing.T) {
+	dir := t.TempDir()
+	file := filepath.Join(dir, "server.yaml")
+	if err := os.WriteFile(file, []byte("# mine\nlisten: \":8080\"\n"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	m := newConsoleModel("127.0.0.1:1", file)
+	m.width, m.height, m.loaded = 120, 40, true
+	m.data.nodes = []adminapi.NodeView{{ID: "cuda01", State: "ready", GPUTotal: 1, GPUs: []adminapi.GPUView{{Index: 0, Name: "Quadro RTX 6000"}}}}
+	m.data.replicas = []adminapi.ReplicaView{{ID: "r1", Model: "coder-14b", State: "ready",
+		Shards: []adminapi.ShardView{{Node: "cuda01", GPUs: []int{0}}}}}
+
+	// Watching: the GPUs view says which model holds each card.
+	m.menu = cGPUs
+	if v := m.View(); !strings.Contains(v, "Quadro RTX 6000") || !strings.Contains(v, "coder-14b") {
+		t.Errorf("the GPUs view:\n%s", v)
+	}
+
+	// Settings: into router, change retries, save.
+	var mm tea.Model = m
+	mm.(consoleModel).View()
+	cm := mm.(consoleModel)
+	cm.menu, cm.inPanel = cSettings, true
+	for i, r := range cm.settingsRows() {
+		if r.label == "router" {
+			cm.row = i
+		}
+	}
+	mm = press(cm, "enter") // into router
+	cm = mm.(consoleModel)
+	for i, r := range cm.settingsRows() {
+		if r.label == "retries" {
+			cm.row = i
+		}
+	}
+	if v := cm.View(); !strings.Contains(v, "(default)") {
+		t.Errorf("an unset key does not show its default:\n%s", v)
+	}
+	mm = press(cm, "enter") // the edit dialog
+	cm = mm.(consoleModel)
+	cm.prompt.SetValue("5")
+	mm = press(cm, "enter", "s")
+	if d := mm.(consoleModel).dialog; d != "saved" {
+		t.Fatalf("after saving the dialog is %q: %s", d, mm.(consoleModel).dialogText)
+	}
+	b, _ := os.ReadFile(file)
+	if !strings.Contains(string(b), "retries: 5") || !strings.Contains(string(b), "# mine") {
+		t.Errorf("server.yaml:\n%s", b)
+	}
+}

internal/cli/tui_widgets.go

@@ -47,6 +47,8 @@ type textField struct {
 	Placeholder string
 	CharLimit   int
 	Width       int
+	// Mask shows a dot for every character: a password.
+	Mask bool
 }
 
 func newTextField() textField { return textField{Prompt: "> "} }
@@ -84,6 +86,9 @@ func (f textField) Update(msg tea.Msg) (textField, tea.Cmd) {
 
 func (f textField) View() string {
 	text := string(f.value)
+	if f.Mask {
+		text = strings.Repeat("•", len(f.value))
+	}
 	shown := text
 	if text == "" && f.Placeholder != "" {
 		shown = sDim.Render(f.Placeholder)

internal/config/config.go

@@ -257,6 +257,14 @@ type Auth struct {
 	// blocking a request: losing accounting is bad, turning a stalled database
 	// into an outage is worse.
 	BufferSize int `koanf:"buffer_size"`
+
+	// UsageRetention is how long the record of each request is kept: who sent
+	// it, which replica answered, how long it queued — what `ctl request`
+	// reads. Zero keeps it for ever. Totals, charts and budgets are read from
+	// five-minute sums kept beside the records, which are not pruned, so this
+	// bounds the table that grows by a row a request without changing any
+	// figure a page shows. At least a day when set.
+	UsageRetention time.Duration `koanf:"usage_retention"`
 }
 
 // Personal configures use that belongs to a person rather than a project.
@@ -363,6 +371,13 @@ type Router struct {
 	// accounting works regardless of client behaviour.
 	InjectUsage bool `koanf:"inject_usage"`
 
+	// TenantLabel is the level of the tree the per-tenant request series are
+	// labelled by — llmgw_tenant_requests_total and its siblings: "org",
+	// "team" or "project", or "none" for no such series. A label per project
+	// on a cluster of thousands is more series than Prometheus wants, and a
+	// team is what an alert is usually about; "team" is the default.
+	TenantLabel string `koanf:"tenant_label"`
+
 	// MaxBodyBytes bounds the request body the router will read.
 	MaxBodyBytes int64 `koanf:"max_body_bytes"`
 
@@ -483,6 +498,7 @@ func Default() Server {
 			Cooldown:         15 * time.Second,
 			FailureThreshold: 3,
 			DefaultProfile:   "chat",
+			TenantLabel:      "team",
 			SessionHeader:    "x-session-id",
 			InjectUsage:      true,
 			MaxBodyBytes:     32 << 20,
@@ -573,6 +589,14 @@ func (s Server) Validate() error {
 	if s.Auth.Enabled && s.Auth.Store == "" {
 		return fmt.Errorf("auth.store is required when auth is enabled")
 	}
+	switch s.Router.TenantLabel {
+	case "", "none", "org", "team", "project":
+	default:
+		return fmt.Errorf("router.tenant_label is %q: org, team, project or none", s.Router.TenantLabel)
+	}
+	if s.Auth.UsageRetention != 0 && s.Auth.UsageRetention < 24*time.Hour {
+		return fmt.Errorf("auth.usage_retention is %s: at least 24h, or 0 to keep every request", s.Auth.UsageRetention)
+	}
 	if s.Threads.Store != "" && s.Threads.RetainHours <= 0 {
 		return fmt.Errorf("threads.retain_hours must be positive when threads.store is set")
 	}

internal/config/config_test.go

@@ -206,3 +206,17 @@ func TestEnginesDefaultToTheCheckedVersions(t *testing.T) {
 		t.Error("an emptied engine is still wanted")
 	}
 }
+
+func TestAUsageRetentionUnderADayIsRefused(t *testing.T) {
+	// Shorter than a day is almost certainly a unit slip — 90m for 90 days —
+	// and would delete every request record within the hour.
+	s := Default()
+	s.Auth.UsageRetention = 90 * time.Minute
+	if err := s.Validate(); err == nil || !strings.Contains(err.Error(), "usage_retention") {
+		t.Errorf("a 90-minute retention was accepted: %v", err)
+	}
+	s.Auth.UsageRetention = 90 * 24 * time.Hour
+	if err := s.Validate(); err != nil {
+		t.Errorf("a 90-day retention was refused: %v", err)
+	}
+}

internal/control/adminapi/api.go

@@ -40,6 +40,8 @@ import (
 type API struct {
 	// releases is where release binaries are fetched from, for upgrades.
 	releases string
+	// storeStatus reads a replicated store's members; nil for one database.
+	storeStatus func() ([]StoreMember, error)
 	// serve runs registry-to-replica jobs; nil without a mirror and a store.
 	serve *modelserve.Server
 
@@ -698,6 +700,9 @@ type Usage interface {
 // SetUsage supplies the usage reporting source.
 func (a *API) SetUsage(u Usage) { a.usage = u }
 
+// SetStoreStatus supplies the state of a replicated store, for the summary.
+func (a *API) SetStoreStatus(f func() ([]StoreMember, error)) { a.storeStatus = f }
+
 // ScopeUsageView is one group's spend, with its quota when it has one.
 type ScopeUsageView struct {
 	Scope string `json:"scope"`

internal/control/adminapi/summary.go

@@ -31,6 +31,20 @@ type Summary struct {
 	// IdleReplicas are serving replicas nothing has asked for in a while, which
 	// on a fleet with more models than GPUs is the first place to find room.
 	IdleReplicas []IdleReplica `json:"idle_replicas,omitempty"`
+	// Store is the members of the store's cluster when it is replicated, and
+	// StoreError why it could not be read; both empty for a single database.
+	Store      []StoreMember `json:"store,omitempty"`
+	StoreError string        `json:"store_error,omitempty"`
+}
+
+// StoreMember is one member of a replicated store, as its Patroni reports it.
+type StoreMember struct {
+	Name  string `json:"name"`
+	Host  string `json:"host"`
+	Role  string `json:"role"`
+	State string `json:"state"`
+	// Lag is bytes behind the leader; -1 unknown, 0 for the leader.
+	Lag int64 `json:"lag"`
 }
 
 // NodeTotals counts nodes by the states that need different responses.
@@ -107,6 +121,13 @@ const idleThreshold = 10 * time.Minute
 // Summary builds the fleet summary.
 func (a *API) Summary() Summary {
 	var s Summary
+	if a.storeStatus != nil {
+		members, err := a.storeStatus()
+		s.Store = members
+		if err != nil {
+			s.StoreError = err.Error()
+		}
+	}
 
 	nodes := a.Nodes()
 	byClass := map[string]*ClassUse{}

internal/control/agentsvc/controlplanes.go

@@ -0,0 +1,65 @@
+package agentsvc
+
+import (
+	"context"
+	"slices"
+	"time"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+)
+
+// SetControlPlanes gives the service where the list of control planes comes
+// from. Every agent is told it when it registers, and again when it changes.
+func (s *Service) SetControlPlanes(list func(context.Context) ([]string, error)) {
+	s.mu.Lock()
+	s.controlPlanes = list
+	s.mu.Unlock()
+}
+
+// controlPlanesMsg is the list as an agent is sent it, or nil when there is
+// none to send: a cluster with no list recorded keeps what its agents were
+// configured with.
+func (s *Service) controlPlanesMsg(ctx context.Context) *llmgwv1.ControlMessage {
+	s.mu.RLock()
+	list := s.controlPlanes
+	s.mu.RUnlock()
+	if list == nil {
+		return nil
+	}
+	servers, err := list(ctx)
+	if err != nil || len(servers) == 0 {
+		return nil
+	}
+	return &llmgwv1.ControlMessage{Msg: &llmgwv1.ControlMessage_ControlPlanes{
+		ControlPlanes: &llmgwv1.ControlPlanes{Servers: servers}}}
+}
+
+// WatchControlPlanes tells every connected agent the list when it changes,
+// checking every so often. A control plane added by `llmgw server add` reaches
+// the nodes within that time, without their being touched.
+func (s *Service) WatchControlPlanes(ctx context.Context, every time.Duration) {
+	var last []string
+	t := time.NewTicker(every)
+	defer t.Stop()
+	for {
+		if msg := s.controlPlanesMsg(ctx); msg != nil {
+			now := msg.GetControlPlanes().GetServers()
+			if !slices.Equal(now, last) {
+				for _, node := range s.ConnectedNodes() {
+					if err := s.send(node, msg); err != nil {
+						s.log.Warn("could not tell a node the control planes", "node", node, "err", err)
+					}
+				}
+				if last != nil {
+					s.log.Info("told the nodes the control planes changed", "servers", now)
+				}
+				last = now
+			}
+		}
+		select {
+		case <-ctx.Done():
+			return
+		case <-t.C:
+		}
+	}
+}

internal/control/agentsvc/service.go

@@ -61,6 +61,9 @@ type Service struct {
 	choices       func(nodeID string) map[string]string
 	installableBy map[string][]string
 
+	// controlPlanes lists what agents may dial; see SetControlPlanes.
+	controlPlanes func(context.Context) ([]string, error)
+
 	// logWaits are the log requests awaiting a reply, by request id.
 	logMu    sync.Mutex
 	logWaits map[string]chan *llmgwv1.LogsReply
@@ -229,6 +232,13 @@ func (s *Service) Connect(
 			return err
 		}
 	}
+	// Which control planes there are, so an agent configured with one learns
+	// of a second added since.
+	if msg := s.controlPlanesMsg(ctx); msg != nil {
+		if err := stream.Send(msg); err != nil {
+			return err
+		}
+	}
 
 	// Recover replica state from what this node is already running, before any
 	// reconciliation pass can conclude the cluster is empty and place

internal/docgen/docgen.go

@@ -29,19 +29,30 @@ type Field struct {
 	Key  string
 	Type string
 	Doc  string
-	// Nested names the struct this key expands into, empty for a leaf. It is
-	// what lets the reference say "see Auth" rather than printing the same
-	// twenty keys under four parents.
+	// Nested names the struct this key expands into, by its ID
+	// (config.Auth), empty for a leaf. It is what lets the reference say
+	// "see Auth" rather than printing the same twenty keys under four
+	// parents.
 	Nested string
 }
 
 // Block is a struct that appears in configuration.
 type Block struct {
-	Name   string
+	// Package and Name are where the struct is declared: what tells
+	// tlsconf.Config from the agent's Config.
+	Package string
+	Name    string
+	// Title is what it is called in the reference and on the screens: the
+	// name alone where no other block shares it, the package's name with it
+	// where one does.
+	Title  string
 	Doc    string
 	Fields []Field
 }
 
+// ID is the block's package and name, unique where Name is not.
+func (b Block) ID() string { return b.Package + "." + b.Name }
+
 // Collect reads koanf-tagged structs out of Go source files.
 func Collect(files []string) ([]Block, error) {
 	fset := token.NewFileSet()
@@ -62,22 +73,36 @@ func Collect(files []string) ([]Block, error) {
 			if !ok {
 				return true
 			}
-			b, ok := blockOf(ts, st)
-			if !ok || seen[b.Name] {
+			b, ok := blockOf(f.Name.Name, ts, st)
+			if !ok || seen[b.ID()] {
 				return true
 			}
-			seen[b.Name] = true
+			seen[b.ID()] = true
 			blocks = append(blocks, b)
 			return true
 		})
 	}
-	sort.Slice(blocks, func(i, j int) bool { return blocks[i].Name < blocks[j].Name })
+	// Named by their own name unless another block has it too. Keyed by
+	// name alone, tlsconf.Config, engines.Config and supervisor.Config were
+	// each taken for the agent's Config and dropped: the reference had no
+	// TLS settings at all.
+	count := map[string]int{}
+	for _, b := range blocks {
+		count[b.Name]++
+	}
+	for i := range blocks {
+		blocks[i].Title = blocks[i].Name
+		if count[blocks[i].Name] > 1 {
+			blocks[i].Title = blocks[i].ID()
+		}
+	}
+	sort.Slice(blocks, func(i, j int) bool { return blocks[i].Title < blocks[j].Title })
 	return blocks, nil
 }
 
 // blockOf builds a Block, or reports that this struct carries no settings.
-func blockOf(ts *ast.TypeSpec, st *ast.StructType) (Block, bool) {
-	b := Block{Name: ts.Name.Name, Doc: docText(ts.Doc)}
+func blockOf(pkg string, ts *ast.TypeSpec, st *ast.StructType) (Block, bool) {
+	b := Block{Package: pkg, Name: ts.Name.Name, Doc: docText(ts.Doc)}
 	for _, f := range st.Fields.List {
 		if f.Tag == nil {
 			continue
@@ -90,7 +115,7 @@ func blockOf(ts *ast.TypeSpec, st *ast.StructType) (Block, bool) {
 		if !ok || key == "" || key == "-" {
 			continue
 		}
-		typ, nested := typeOf(f.Type)
+		typ, nested := typeOf(pkg, f.Type)
 		b.Fields = append(b.Fields, Field{
 			Key: key, Type: typ, Doc: docText(f.Doc), Nested: nested,
 		})
@@ -99,29 +124,34 @@ func blockOf(ts *ast.TypeSpec, st *ast.StructType) (Block, bool) {
 }
 
 // typeOf renders a field's type, and names the struct it expands into.
-func typeOf(e ast.Expr) (rendered, nested string) {
+func typeOf(pkg string, e ast.Expr) (rendered, nested string) {
 	switch t := e.(type) {
 	case *ast.Ident:
 		if isNamedStruct(t.Name) {
-			return t.Name, t.Name
+			return t.Name, pkg + "." + t.Name
 		}
 		return t.Name, ""
 	case *ast.SelectorExpr:
-		// A type from another package, such as tlsconf.Config.
+		// A type from another package, such as tlsconf.Config. Standard
+		// library types are values, not blocks: a time.Duration is a
+		// setting, written "30s".
 		name := t.Sel.Name
-		if pkg, ok := t.X.(*ast.Ident); ok {
-			return pkg.Name + "." + name, name
+		if x, ok := t.X.(*ast.Ident); ok {
+			if x.Name == "time" {
+				return x.Name + "." + name, ""
+			}
+			return x.Name + "." + name, x.Name + "." + name
 		}
-		return name, name
+		return name, ""
 	case *ast.ArrayType:
-		inner, _ := typeOf(t.Elt)
+		inner, _ := typeOf(pkg, t.Elt)
 		return "[]" + inner, ""
 	case *ast.MapType:
-		k, _ := typeOf(t.Key)
-		v, _ := typeOf(t.Value)
+		k, _ := typeOf(pkg, t.Key)
+		v, _ := typeOf(pkg, t.Value)
 		return "map[" + k + "]" + v, ""
 	case *ast.StarExpr:
-		inner, n := typeOf(t.X)
+		inner, n := typeOf(pkg, t.X)
 		return "*" + inner, n
 	default:
 		return "value", ""

internal/docgen/docgen_test.go

@@ -136,3 +136,18 @@ func TestAFieldMarkedNotConfigurableIsLeftOut(t *testing.T) {
 		}
 	}
 }
+
+func TestTheSettingsScreensDescribeTheCode(t *testing.T) {
+	// The screens list Reference; it must be what the structs say now.
+	blocks, err := Collect(sourcePaths(t))
+	if err != nil {
+		t.Fatal(err)
+	}
+	have, err := os.ReadFile(filepath.Join(repoRoot(t), "internal", "docgen", "reference_gen.go"))
+	if err != nil {
+		t.Fatalf("reference_gen.go is missing; run go generate ./...: %v", err)
+	}
+	if string(have) != RenderGo(blocks) {
+		t.Error("internal/docgen/reference_gen.go no longer matches the configuration structs; run go generate ./...")
+	}
+}

internal/docgen/gen/main.go

@@ -38,5 +38,9 @@ func main() {
 		fmt.Fprintln(os.Stderr, "error:", err)
 		os.Exit(1)
 	}
-	fmt.Printf("docs/reference.md: %d blocks\n", len(blocks))
+	if err := os.WriteFile(filepath.Join(root, "internal", "docgen", "reference_gen.go"), []byte(docgen.RenderGo(blocks)), 0o644); err != nil {
+		fmt.Fprintln(os.Stderr, "error:", err)
+		os.Exit(1)
+	}
+	fmt.Printf("docs/reference.md and internal/docgen/reference_gen.go: %d blocks\n", len(blocks))
 }

internal/docgen/reference_gen.go

@@ -0,0 +1,302 @@
+// Code generated by go run ./internal/docgen/gen; DO NOT EDIT.
+
+package docgen
+
+// Reference is every configuration block, as Collect reads them.
+var Reference = []Block{
+	{Package: "catalog", Name: "Adapter", Title: "Adapter", Doc: "", Fields: []Field{
+		{Key: "name", Type: "string", Doc: "Name is what clients ask for, like a model name.", Nested: ""},
+		{Key: "source", Type: "string", Doc: "Source is the adapter file (GGUF for llama.cpp), fetched by each node\nas the model's weights are.", Nested: ""},
+		{Key: "scale", Type: "float64", Doc: "Scale is how strongly it is applied; zero takes 1.", Nested: ""},
+	}},
+	{Package: "config", Name: "Admission", Title: "Admission", Doc: "", Fields: []Field{
+		{Key: "enabled", Type: "bool", Doc: "Enabled turns the layer on. Off, every request goes straight through,\nwhich is the v0.2.0 behaviour.", Nested: ""},
+		{Key: "bands", Type: "[]Band", Doc: "", Nested: ""},
+		{Key: "default_band", Type: "string", Doc: "", Nested: ""},
+		{Key: "band_header", Type: "string", Doc: "BandHeader lets a client declare its band. Until authentication lands in\nv1.0.0 this is trusted, so the port must not be exposed to callers who\nshould not be able to promote themselves.", Nested: ""},
+		{Key: "queue_ttl", Type: "time.Duration", Doc: "", Nested: ""},
+		{Key: "queue_capacity_per_band", Type: "int", Doc: "", Nested: ""},
+		{Key: "queue_capacity_total", Type: "int", Doc: "QueueCapacityTotal bounds everything queued across bands. Reached, a\nrequest is not refused while a lower band has any queued: the newest of\nthe lowest band is turned away instead, with a 503 and Retry-After.\nZero takes half of all the bands' capacities together.", Nested: ""},
+		{Key: "interval", Type: "time.Duration", Doc: "", Nested: ""},
+	}},
+	{Package: "config", Name: "Auth", Title: "Auth", Doc: "", Fields: []Field{
+		{Key: "enabled", Type: "bool", Doc: "", Nested: ""},
+		{Key: "store", Type: "string", Doc: "Store is the SQLite file holding keys, tenancy and usage.", Nested: ""},
+		{Key: "cache_ttl", Type: "time.Duration", Doc: "CacheTTL bounds how long a revoked key keeps working, traded against a\ndatabase read per key per TTL.", Nested: ""},
+		{Key: "negative_ttl", Type: "time.Duration", Doc: "NegativeTTL is how long a rejected key is remembered as rejected, so a\nclient looping with a bad key cannot become a database read per request.", Nested: ""},
+		{Key: "default_rpm", Type: "int", Doc: "DefaultRPM and DefaultTPM apply to a key that no scope gives a limit to.\n\nUnlimited used to be the default, and a project created without limits is\nthe easy path — the \"default\" project on this cluster still is one. The\nfirst sign of that is the bill or the outage. Zero keeps the old\nbehaviour for anyone who wants it, but it should be a choice rather than\nan omission. A fallback, not a ceiling: an explicit limit at any scope\nwins, above or below.", Nested: ""},
+		{Key: "default_tpm", Type: "int", Doc: "", Nested: ""},
+		{Key: "signup_open", Type: "bool", Doc: "SignupOpen serves the signup form. Accounts it creates are pending until\nan operator approves them, so this is not a way in — but turning it off\nmeans accounts are made with `llmgw users create` instead.", Nested: ""},
+		{Key: "personal", Type: "Personal", Doc: "Personal is what a person may do on their own, outside any project: make\npersonal keys and use the chat page personally, within these limits.", Nested: "config.Personal"},
+		{Key: "pricing", Type: "map[string]Price", Doc: "Pricing is what each model costs, for budgets. A model absent from the\ntable costs whatever DefaultPrice says, which is nothing unless set.", Nested: ""},
+		{Key: "default_price", Type: "Price", Doc: "", Nested: "config.Price"},
+		{Key: "flush_interval", Type: "time.Duration", Doc: "FlushInterval and BatchSize decide when usage records are written.", Nested: ""},
+		{Key: "batch_size", Type: "int", Doc: "", Nested: ""},
+		{Key: "buffer_size", Type: "int", Doc: "BufferSize bounds the queue. Beyond it records are dropped rather than\nblocking a request: losing accounting is bad, turning a stalled database\ninto an outage is worse.", Nested: ""},
+		{Key: "usage_retention", Type: "time.Duration", Doc: "UsageRetention is how long the record of each request is kept: who sent\nit, which replica answered, how long it queued — what `ctl request`\nreads. Zero keeps it for ever. Totals, charts and budgets are read from\nfive-minute sums kept beside the records, which are not pruned, so this\nbounds the table that grows by a row a request without changing any\nfigure a page shows. At least a day when set.", Nested: ""},
+	}},
+	{Package: "catalog", Name: "Capabilities", Title: "Capabilities", Doc: "", Fields: []Field{
+		{Key: "tools", Type: "bool", Doc: "", Nested: ""},
+		{Key: "vision", Type: "bool", Doc: "", Nested: ""},
+		{Key: "embeddings", Type: "bool", Doc: "", Nested: ""},
+	}},
+	{Package: "config", Name: "Control", Title: "Control", Doc: "", Fields: []Field{
+		{Key: "interval", Type: "time.Duration", Doc: "Interval is how often desired and actual state are compared.", Nested: ""},
+		{Key: "base_backoff", Type: "time.Duration", Doc: "BaseBackoff and MaxBackoff bound how fast a failing model is retried.", Nested: ""},
+		{Key: "max_backoff", Type: "time.Duration", Doc: "", Nested: ""},
+		{Key: "unreachable_after", Type: "time.Duration", Doc: "UnreachableAfter and LostAfter stage node expiry. They are deliberately\nfar apart: a network blip must not tear down replicas that still serve,\nand rebuilding a multi-node replica costs tens of minutes of loading.", Nested: ""},
+		{Key: "lost_after", Type: "time.Duration", Doc: "", Nested: ""},
+		{Key: "port_range", Type: "[]int", Doc: "PortRange is the per-node range for engine HTTP and dist-init ports.", Nested: ""},
+		{Key: "mgmt_interface", Type: "string", Doc: "MgmtInterface is the ethernet interface NCCL bootstraps over.", Nested: ""},
+		{Key: "enabled", Type: "bool", Doc: "Enabled runs the control plane even when configuration declares no\nmodels.\n\nWithout it, a deployment whose catalog lives entirely in the database\ncannot start: control plane mode is inferred from the config file, so an\nempty models block means \"router only\", the process refuses to start for\nwant of anything to serve, and the models it would have loaded are never\nread. That is a cluster that cannot be managed at runtime because it is\nmanaged at runtime. AgentListen cannot carry this signal — it has a\ndefault, so every deployment would become a control plane.", Nested: ""},
+		{Key: "lease_ttl", Type: "time.Duration", Doc: "LeaseTTL is how long this control plane's claim to be the one placing\nwork stands without being renewed.\n\nOnly meaningful with a shared store, where a second control plane can\nsee the claim. Long enough to survive a slow query or a brief network\nstall, short enough that a standby takes over before anyone has finished\nreading the alert. Zero takes the default.", Nested: ""},
+	}},
+	{Package: "config", Name: "Endpoint", Title: "Endpoint", Doc: "", Fields: []Field{
+		{Key: "id", Type: "string", Doc: "", Nested: ""},
+		{Key: "url", Type: "string", Doc: "", Nested: ""},
+		{Key: "api_key", Type: "string", Doc: "APIKey is sent to the engine as a bearer token, for an engine started\nwith one.", Nested: ""},
+		{Key: "models", Type: "[]string", Doc: "", Nested: ""},
+		{Key: "engine", Type: "string", Doc: "Engine names the inference engine, which decides what its metrics are\ncalled. Empty means vLLM's names.", Nested: ""},
+		{Key: "pool", Type: "string", Doc: "", Nested: ""},
+		{Key: "weight", Type: "float64", Doc: "", Nested: ""},
+		{Key: "priority", Type: "int", Doc: "", Nested: ""},
+	}},
+	{Package: "config", Name: "EngineOverride", Title: "EngineOverride", Doc: "", Fields: []Field{
+		{Key: "gpu_class", Type: "string", Doc: "GPUClass picks nodes with a GPU of this class (\"v100-32g\").", Nested: ""},
+		{Key: "nodes", Type: "[]string", Doc: "Nodes picks nodes by id.", Nested: ""},
+		{Key: "labels", Type: "map[string]string", Doc: "Labels picks nodes carrying all of these.", Nested: ""},
+		{Key: "engines", Type: "map[string]string", Doc: "Engines are laid over the cluster's: a version, or \"\" for none.", Nested: ""},
+	}},
+	{Package: "catalog", Name: "External", Title: "External", Doc: "", Fields: []Field{
+		{Key: "url", Type: "string", Doc: "URL is the provider's base address, without /v1: https://api.openai.com.", Nested: ""},
+		{Key: "model", Type: "string", Doc: "Model is what the provider calls it; requests are rewritten to it.", Nested: ""},
+		{Key: "api_key_file", Type: "string", Doc: "APIKeyFile holds the provider's key, on the control plane. A file\nrather than the key itself, so the key is in neither the catalog, the\nstore nor the audit log; empty for a provider that asks for none.", Nested: ""},
+	}},
+	{Package: "config", Name: "Landing", Title: "Landing", Doc: "", Fields: []Field{
+		{Key: "contact", Type: "string", Doc: "Contact is where someone without a key should go — an address, a URL, a\nchannel name. Rendered as a link when it is one, as text otherwise.", Nested: ""},
+		{Key: "notice", Type: "string", Doc: "Notice is an optional line of your own, for whatever the reader needs to\nknow that this page cannot guess.", Nested: ""},
+	}},
+	{Package: "engines", Name: "LlamaCppConfig", Title: "LlamaCppConfig", Doc: "", Fields: []Field{
+		{Key: "bin", Type: "string", Doc: "Bin is the llama-server binary. It is the server, not the CLI: the\ngateway talks OpenAI over HTTP and needs the one that serves it.\nEmpty means the agent installs llama.cpp itself, at the version the\ncontrol plane asks for. Setting it opts out of that.", Nested: ""},
+		{Key: "parallel", Type: "int", Doc: "Parallel is how many requests a server handles at once, when a profile\ndoes not say. llama.cpp defaults to one, which would make every second\nrequest wait behind the first no matter what the router decided.", Nested: ""},
+		{Key: "threads", Type: "int", Doc: "Threads is how many CPU threads a shard with no GPU computes on. Zero\ntakes half this machine's CPUs.\n\nNot all of them, which is llama.cpp's own default: two busy servers\nthat each spin on every CPU fight for them, and on an 8-vCPU node\nthat made 7B generation fall from 10 tokens a second to under 1 each.\nAt half, one server alone is about a third slower and two together\nkeep 5.5 each. A node that only ever runs one CPU replica does better\nwith all of them; say so here.", Nested: ""},
+	}},
+	{Package: "config", Name: "Mirror", Title: "Mirror", Doc: "", Fields: []Field{
+		{Key: "api", Type: "string", Doc: "API is the mirror's management address, scheme included, for example\nhttps://models.internal:8082. Empty means no mirror is configured.\n\nThe management port, not the one nodes fetch bytes from. Those are two\nlisteners on the mirror precisely so this one can be locked down without\nre-certifying every agent's fetcher.", Nested: ""},
+		{Key: "tls", Type: "tlsconf.Config", Doc: "TLS authenticates to the mirror's management API, with the same CA the\nagents use. Empty leaves the connection in plain HTTP, which is only\nacceptable on a network where nothing untrusted can reach that port —\nthe mirror says as much in its own log when it starts that way.", Nested: "tlsconf.Config"},
+		{Key: "registry", Type: "string", Doc: "Registry is the model registry the mirror page browses, empty for the\npublic Hugging Face. Set it to an internal mirror of the registry where\nthe control plane cannot reach the internet — or leave the whole mirror\nblock out, which turns the explorer off rather than giving it a search\nbox that can only time out.", Nested: ""},
+		{Key: "timeout", Type: "time.Duration", Doc: "Timeout bounds a management call. It does not bound a download: the\nmirror answers a fetch as soon as it has started one, and the bytes keep\narriving long after this deadline has passed.", Nested: ""},
+	}},
+	{Package: "catalog", Name: "Model", Title: "Model", Doc: "", Fields: []Field{
+		{Key: "name", Type: "string", Doc: "", Nested: ""},
+		{Key: "aliases", Type: "[]string", Doc: "", Nested: ""},
+		{Key: "description", Type: "string", Doc: "Description is what this model is for, in a line.\n\nWritten by whoever added it, because nobody else can say it. A registry\noffers a pipeline tag and a wall of README, and neither answers the\nquestion a person picking from a list is asking: which of these nine\nnames should I send this request to.\n\nIt reaches clients through /v1/models, which is the only view most of\nthem ever see. A gateway serving nine models by name alone makes that\nchoice by guesswork.", Nested: ""},
+		{Key: "source", Type: "string", Doc: "Source is the weights path or a HuggingFace id.", Nested: ""},
+		{Key: "profiles", Type: "[]Profile", Doc: "", Nested: ""},
+		{Key: "scaling", Type: "Scaling", Doc: "", Nested: "catalog.Scaling"},
+		{Key: "class", Type: "string", Doc: "Class is how much this model's capacity is worth against another's,\nwhen there is not enough for both: critical, standard or opportunistic.\nEmpty is standard, or opportunistic for a model loaded on demand. See\nEffectiveClass.", Nested: ""},
+		{Key: "owner", Type: "string", Doc: "Owner is the part of the tenancy tree whose capacity quota this model\ncounts against, written org:ID, team:ID or project:ID. Empty is shared:\nheld by the cluster, counted against nobody.", Nested: ""},
+		{Key: "external", Type: "*External", Doc: "External is a model served by someone else — an OpenAI-compatible API\noutside the cluster — rather than by replicas placed here. Such a model\nhas no profiles and nothing is placed for it; the router sends its\nrequests to the provider. Most useful named in another model's\nfallbacks: used only when the cluster has no capacity left. Its\nprompts leave the cluster, so which keys may name it is worth an\nallowlist.", Nested: "catalog.External"},
+		{Key: "adapters", Type: "[]Adapter", Doc: "Adapters are LoRA adapters served on this model's replicas, each asked\nfor by its own name as though it were a model. Every replica loads\nevery adapter, applying one only to a request that names it.", Nested: ""},
+		{Key: "fallbacks", Type: "[]string", Doc: "", Nested: ""},
+		{Key: "routing_profile", Type: "string", Doc: "RoutingProfile selects how the router scores endpoints for this model.", Nested: ""},
+		{Key: "slo", Type: "SLO", Doc: "", Nested: "catalog.SLO"},
+		{Key: "context_length", Type: "int", Doc: "ContextLength is how many tokens one request may use, prompt and\ncompletion together, and MaxOutput bounds the completion alone. Zero\nmeans unknown, which is what every model was until now.\n\nThe gateway knew neither, so an over-long prompt was not refused: it was\nadmitted, given a replica, and failed at the engine, and a client sizing\nits prompts from /v1/models had nothing to read. Worse, the number is not\nthe one an operator writes in the engine's arguments — llama.cpp divides\n--ctx-size across its parallel slots, so a pool of 8192 across four slots\nis 2048 a request, and nothing said so.", Nested: ""},
+		{Key: "max_output", Type: "int", Doc: "", Nested: ""},
+		{Key: "capabilities", Type: "Capabilities", Doc: "Capabilities is what this model can do beyond plain completion.", Nested: "catalog.Capabilities"},
+	}},
+	{Package: "config", Name: "Monitoring", Title: "Monitoring", Doc: "", Fields: []Field{
+		{Key: "prometheus", Type: "string", Doc: "Prometheus is the server's address, for example\nhttp://monitoring.internal:9090. Empty leaves the charts off the\ndashboard rather than showing an error where one should be.", Nested: ""},
+		{Key: "window", Type: "time.Duration", Doc: "Window is how far back the charts look. Zero takes an hour.", Nested: ""},
+		{Key: "timeout", Type: "time.Duration", Doc: "Timeout bounds one query. It runs while somebody waits for a page, so a\nmonitoring system that is slow today is skipped rather than waited for.", Nested: ""},
+	}},
+	{Package: "discovery", Name: "Options", Title: "Options", Doc: "", Fields: []Field{
+		{Key: "nvidia_smi", Type: "string", Doc: "NvidiaSMI is the path to nvidia-smi; empty means look it up on PATH.\nWorth configuring on nodes where the driver lives outside the default\npaths, which containerized driver installs often do.", Nested: ""},
+		{Key: "ib_root", Type: "string", Doc: "IBRoot is the sysfs InfiniBand directory.", Nested: ""},
+		{Key: "prefer_nvml", Type: "bool", Doc: "PreferNVML uses the NVML backend when this binary has one. NVLink\ntopology is read from nvidia-smi topo -m on every build; NVML, where it\nis used, is the more exact source of the peers.", Nested: ""},
+		{Key: "skip_gpus", Type: "bool", Doc: "SkipGPUs omits GPU discovery. A node without nvidia-smi is already\nfound to have none, so this is for one whose driver is broken.", Nested: ""},
+	}},
+	{Package: "config", Name: "Personal", Title: "Personal", Doc: "", Fields: []Field{
+		{Key: "disabled", Type: "bool", Doc: "Disabled refuses personal keys and personal chat; people then use the\ngateway only through projects they are members of.", Nested: ""},
+		{Key: "rpm", Type: "int", Doc: "", Nested: ""},
+		{Key: "tpm", Type: "int", Doc: "", Nested: ""},
+		{Key: "budget", Type: "float64", Doc: "Budget is in whole units per Period (\"day\" or \"month\").", Nested: ""},
+		{Key: "period", Type: "string", Doc: "", Nested: ""},
+	}},
+	{Package: "config", Name: "Prefix", Title: "Prefix", Doc: "", Fields: []Field{
+		{Key: "enabled", Type: "bool", Doc: "", Nested: ""},
+		{Key: "max_nodes", Type: "int", Doc: "MaxNodes bounds the tree, MinMatch is the shortest match worth acting on\nin bytes, and TTL is how long an entry is believed — an engine evicts on\nits own schedule, so an old record may describe a cache that is gone.", Nested: ""},
+		{Key: "min_match", Type: "int", Doc: "", Nested: ""},
+		{Key: "ttl", Type: "time.Duration", Doc: "", Nested: ""},
+		{Key: "match_ratio", Type: "float64", Doc: "MatchRatio is the share of the prompt that must match to be worth acting\non. BalanceAbs and BalanceRel are the imbalance guard, and both must trip\nbefore affinity is abandoned.", Nested: ""},
+		{Key: "balance_abs", Type: "float64", Doc: "", Nested: ""},
+		{Key: "balance_rel", Type: "float64", Doc: "", Nested: ""},
+	}},
+	{Package: "catalog", Name: "Profile", Title: "Profile", Doc: "", Fields: []Field{
+		{Key: "name", Type: "string", Doc: "", Nested: ""},
+		{Key: "engine", Type: "string", Doc: "", Nested: ""},
+		{Key: "gpu_class", Type: "string", Doc: "", Nested: ""},
+		{Key: "tp", Type: "int", Doc: "", Nested: ""},
+		{Key: "pp", Type: "int", Doc: "", Nested: ""},
+		{Key: "gpus_per_node", Type: "int", Doc: "GPUsPerNode defaults to TP for a single-node profile.", Nested: ""},
+		{Key: "nodes", Type: "int", Doc: "Nodes above 1 makes this a multi-node profile, which requires a fabric.", Nested: ""},
+		{Key: "args", Type: "[]string", Doc: "", Nested: ""},
+		{Key: "env", Type: "map[string]string", Doc: "", Nested: ""},
+		{Key: "min_memory_gb", Type: "int", Doc: "MinMemoryGB is the host RAM this profile needs to run.\n\nThe number that matters is the model's working set: the weights, plus the\nKV cache for the whole context pool, plus the engine's own overhead. For\na GPU profile that lives in the cards and this is small; for a CPU\nprofile it is the whole thing, and getting it wrong means an OOM on a\nnode that reported itself healthy.\n\nZero means unconstrained, which is every profile that existed before\nthis. A node that does not report its memory is not excluded either —\nan older agent must not have its nodes quietly emptied by an upgrade of\nthe control plane.", Nested: ""},
+		{Key: "cpu_cores", Type: "int", Doc: "CPUCores is the cores a replica of a profile with no GPU is given:\nreserved out of its node's CPUs by placement, as memory is, and what\nits engine computes on (llama.cpp's --threads).\n\nReserved because an engine takes every core it is told about, and two\non one node that each think they have all of them spin against each\nother: on an 8-vCPU node, two 7B replicas fell from 10 tokens a second\nto under one each. Zero reserves nothing and leaves the thread count to\nthe node, which is every profile from before this.", Nested: ""},
+		{Key: "min_disk_gb", Type: "int", Doc: "MinDiskGB is the free space a node needs before it can hold these\nweights.\n\nOnly checked when the node does not already have them: a node holding\nthe file needs no room for it, and requiring the space twice would keep\na model off the one node best placed to run it.\n\nDeclared rather than derived, because the size of a model that has never\nbeen fetched anywhere is not knowable from here — and a wrong guess\nfails late, halfway through a download, which is the failure this exists\nto prevent. Zero means unconstrained, and a node that reports no storage\nis not excluded: unknown is not full.", Nested: ""},
+		{Key: "node_selector", Type: "map[string]string", Doc: "NodeSelector restricts this profile to nodes carrying every one of these\nlabels, with these values.\n\nA CPU profile otherwise fits everywhere, because there is no card to be\nfree — which is right for a 0.5B model and wrong for a 14B one. Nothing\nin the inventory says how much memory a node has, so the operator says\nwhich nodes are meant for a model instead of the scheduler inferring it.\nAgents have reported labels since the beginning; this is what reads them.", Nested: ""},
+		{Key: "weight", Type: "float64", Doc: "Weight is the router's capacity weight. It should come from ctl bench\nrather than a guess (plan §6.2.1).", Nested: ""},
+		{Key: "priority", Type: "int", Doc: "Priority orders profiles during placement; highest that fits wins.", Nested: ""},
+		{Key: "startup_timeout", Type: "time.Duration", Doc: "StartupTimeout bounds model loading. A 70B model across two nodes takes\ntens of minutes, so this is per profile rather than global.", Nested: ""},
+	}},
+	{Package: "config", Name: "Router", Title: "Router", Doc: "", Fields: []Field{
+		{Key: "activation_wait", Type: "time.Duration", Doc: "ActivationWait is how long a request waits for an on-demand model to\nload before the client is told to come back. Zero takes the default.\n\nThe one real decision in on-demand loading. Refusing outright makes the\nfeature useless, since the first request for every model would fail;\nwaiting without a bound turns a request into a job, and a client that\nbudgeted thirty seconds ends up holding a connection for ten minutes.", Nested: ""},
+		{Key: "scrape_interval", Type: "time.Duration", Doc: "", Nested: ""},
+		{Key: "scrape_timeout", Type: "time.Duration", Doc: "", Nested: ""},
+		{Key: "stale_after", Type: "time.Duration", Doc: "StaleAfter is how long an endpoint may go unscraped before being scored\ndown (plan §7.6).", Nested: ""},
+		{Key: "retries", Type: "int", Doc: "Retries is how many *additional* endpoints may be tried after a failure,\nand only while nothing has been written to the client (plan §7.7).", Nested: ""},
+		{Key: "cooldown", Type: "time.Duration", Doc: "Cooldown is how long a failing endpoint leaves the candidate pool.", Nested: ""},
+		{Key: "failure_threshold", Type: "int", Doc: "FailureThreshold is the consecutive failure count that opens a breaker.\n1 turns the breaker into a pure cooldown.", Nested: ""},
+		{Key: "default_profile", Type: "string", Doc: "DefaultProfile names the routing profile used when a model picks none.", Nested: ""},
+		{Key: "prefix", Type: "Prefix", Doc: "Prefix turns on the approximate prefix tree (plan §7.5 tier 1). Off, the\nrouter uses session affinity alone, which is tier 0 and already most of\nthe win for multi-turn traffic.", Nested: "config.Prefix"},
+		{Key: "session_header", Type: "string", Doc: "SessionHeader carries the session id for affinity (plan §7.5 tier 0).", Nested: ""},
+		{Key: "inject_usage", Type: "bool", Doc: "InjectUsage asks engines for usage even when the client did not, so token\naccounting works regardless of client behaviour.", Nested: ""},
+		{Key: "tenant_label", Type: "string", Doc: "TenantLabel is the level of the tree the per-tenant request series are\nlabelled by — llmgw_tenant_requests_total and its siblings: \"org\",\n\"team\" or \"project\", or \"none\" for no such series. A label per project\non a cluster of thousands is more series than Prometheus wants, and a\nteam is what an alert is usually about; \"team\" is the default.", Nested: ""},
+		{Key: "max_body_bytes", Type: "int64", Doc: "MaxBodyBytes bounds the request body the router will read.", Nested: ""},
+		{Key: "timeouts", Type: "Timeouts", Doc: "", Nested: "config.Timeouts"},
+		{Key: "scoring", Type: "Scoring", Doc: "", Nested: "config.Scoring"},
+		{Key: "admission", Type: "Admission", Doc: "", Nested: "config.Admission"},
+	}},
+	{Package: "engines", Name: "SGLangConfig", Title: "SGLangConfig", Doc: "", Fields: []Field{
+		{Key: "python", Type: "string", Doc: "Python is the interpreter of the SGLang virtualenv. SGLang is launched as\na module rather than a console script, which is the documented entry\npoint and avoids depending on the script being on PATH.", Nested: ""},
+	}},
+	{Package: "catalog", Name: "SLO", Title: "SLO", Doc: "", Fields: []Field{
+		{Key: "ttft_p99", Type: "time.Duration", Doc: "", Nested: ""},
+		{Key: "tpot_p99", Type: "time.Duration", Doc: "", Nested: ""},
+	}},
+	{Package: "catalog", Name: "Scaling", Title: "Scaling", Doc: "", Fields: []Field{
+		{Key: "min", Type: "int", Doc: "", Nested: ""},
+		{Key: "max", Type: "int", Doc: "", Nested: ""},
+		{Key: "idle_timeout", Type: "time.Duration", Doc: "IdleTimeout is how long an on-demand model may go unasked-for before its\nreplicas are stopped. Zero takes the cluster default.\n\nMinutes rather than seconds, always. Stopping a replica throws away the\nKV cache prefix routing spent its effort filling, and a model that flaps\nin and out loses that entirely — so the saving has to be worth more than\nthe cache, which it only is over a real idle period.", Nested: ""},
+	}},
+	{Package: "config", Name: "Scoring", Title: "Scoring", Doc: "", Fields: []Field{
+		{Key: "target_waiting", Type: "float64", Doc: "", Nested: ""},
+		{Key: "target_inflight", Type: "float64", Doc: "", Nested: ""},
+		{Key: "kv_ceiling", Type: "float64", Doc: "", Nested: ""},
+		{Key: "session_load_factor", Type: "float64", Doc: "", Nested: ""},
+	}},
+	{Package: "config", Name: "Server", Title: "Server", Doc: "", Fields: []Field{
+		{Key: "listen", Type: "string", Doc: "Listen is the client-facing OpenAI API address.", Nested: ""},
+		{Key: "admin_listen", Type: "string", Doc: "AdminListen serves the dashboard, the management API, health and pprof.\n\nLoopback by default, and it should stay there: pprof can dump the process\nheap, and API keys are in that heap in plaintext. Every comment beside\nthis setting has always said not to expose it, while the default bound\nevery interface — a default that contradicts its own documentation is a\nfootgun, so the default now matches the advice.", Nested: ""},
+		{Key: "metrics_listen", Type: "string", Doc: "MetricsListen serves Prometheus metrics and nothing else.\n\nSeparate from AdminListen because the reason that one is loopback is\npprof, not metrics — and metrics are the one thing on it that something\nelse genuinely needs to reach. Sharing a listener meant a fleet could\neither be monitored or keep its heap private, which is not a choice\nanybody should have to make.\n\nEmpty means off, and metrics remain available on AdminListen either way,\nso nothing that scrapes it today has to change.", Nested: ""},
+		{Key: "agent_listen", Type: "string", Doc: "AgentListen is where node agents connect. Separate from the client port\nso the fleet-facing surface can be firewalled independently.", Nested: ""},
+		{Key: "auth", Type: "Auth", Doc: "Auth turns on API keys on the client-facing port.", Nested: "config.Auth"},
+		{Key: "threads", Type: "Threads", Doc: "Threads keeps conversations and agent memory without API keys, for a\ngateway on one trusted machine. With auth on they live in the auth store,\nper project, and this block is ignored.", Nested: "config.Threads"},
+		{Key: "landing", Type: "Landing", Doc: "Landing configures the page served at \"/\".", Nested: "config.Landing"},
+		{Key: "agent_tls", Type: "tlsconf.Config", Doc: "AgentTLS authenticates the fleet-facing port. It deliberately does not\ncover the client-facing one: that surface is fronted by a publicly\ntrusted certificate, while this one is mutual and private.", Nested: "tlsconf.Config"},
+		{Key: "monitoring", Type: "Monitoring", Doc: "Monitoring is where the dashboard reads its history from.", Nested: "config.Monitoring"},
+		{Key: "tracing", Type: "Tracing", Doc: "Tracing sends a trace of each inference request to an OpenTelemetry\ncollector. Off unless an endpoint is set here or in\nOTEL_EXPORTER_OTLP_ENDPOINT.", Nested: "config.Tracing"},
+		{Key: "mirror", Type: "Mirror", Doc: "Mirror points the control plane at a model mirror it can administer.\nEmpty means there is none, and /admin/mirror stays unmounted.", Nested: "config.Mirror"},
+		{Key: "engines", Type: "map[string]string", Doc: "Engines are the engines every node installs, and at which version. The\nagents install them themselves — the right build for their hardware,\nchecked against its publisher's checksum and seen to run — so a node is\nnever trusted to already have what somebody once put there.\n\nUnset keeps DefaultEngines. A version replaces the default; an empty\none leaves that engine to be installed by hand.", Nested: ""},
+		{Key: "engine_overrides", Type: "[]EngineOverride", Doc: "EngineOverrides give some nodes different engines from the rest: a\nvLLM built for CUDA 12 on the Volta cards CUDA 13 dropped, or none at\nall on nodes meant only for llama.cpp. Each picks nodes and lays its\nengines over the cluster's, in order, later ones last; an empty version\nremoves that engine from the nodes it picks.", Nested: ""},
+		{Key: "backup_dir", Type: "string", Doc: "BackupDir is where `llmgw server backup` writes, and where the newest\narchive's age is read for llmgw_backup_last_success_timestamp_seconds.\nEmpty means /var/backups/llmgw-server.", Nested: ""},
+		{Key: "releases", Type: "string", Doc: "Releases is where `llmgw ctl upgrade` fetches release binaries from,\nfor this machine and — through their agents — every node. Empty means\nthe project's own releases.", Nested: ""},
+		{Key: "router", Type: "Router", Doc: "", Nested: "config.Router"},
+		{Key: "control", Type: "Control", Doc: "", Nested: "config.Control"},
+		{Key: "models", Type: "[]catalog.Model", Doc: "", Nested: ""},
+		{Key: "endpoints", Type: "[]Endpoint", Doc: "Endpoints is the phase-1 static replica list. It is only consulted when\nno model declares a profile, which is what makes a router-only smoke test\nagainst sim-engines possible without a control plane.", Nested: ""},
+	}},
+	{Package: "engines", Name: "SimConfig", Title: "SimConfig", Doc: "", Fields: []Field{
+		{Key: "bin", Type: "string", Doc: "Bin is the llmgw binary itself; the engine is one of its subcommands.\nEmpty means whatever started this agent, which is right almost always.", Nested: ""},
+	}},
+	{Package: "config", Name: "Threads", Title: "Threads", Doc: "", Fields: []Field{
+		{Key: "store", Type: "string", Doc: "Store is the SQLite file. Empty means nothing is kept.", Nested: ""},
+		{Key: "retain_hours", Type: "int", Doc: "RetainHours is how long idle conversations and notes are kept. Required\nwith Store: a store with no expiry keeps everything anyone typed, by\nomission.", Nested: ""},
+	}},
+	{Package: "config", Name: "Timeouts", Title: "Timeouts", Doc: "", Fields: []Field{
+		{Key: "dial", Type: "time.Duration", Doc: "", Nested: ""},
+		{Key: "response_header", Type: "time.Duration", Doc: "ResponseHeader bounds the wait for headers on a streaming request. It is\na liveness check there, because headers arrive before generation starts.", Nested: ""},
+		{Key: "generation", Type: "time.Duration", Doc: "Generation bounds a non-streaming request, where the engine sends no\nheaders until the completion is done. Setting response_header alone used\nto cap both, so 60s read as a liveness guard and behaved as a limit of\nabout a hundred tokens on a slow model.", Nested: ""},
+		{Key: "first_token", Type: "time.Duration", Doc: "", Nested: ""},
+		{Key: "idle", Type: "time.Duration", Doc: "", Nested: ""},
+		{Key: "total", Type: "time.Duration", Doc: "", Nested: ""},
+	}},
+	{Package: "config", Name: "Tracing", Title: "Tracing", Doc: "", Fields: []Field{
+		{Key: "endpoint", Type: "string", Doc: "Endpoint is the collector, for example http://otel.internal:4318.\nEmpty falls back to OTEL_EXPORTER_OTLP_ENDPOINT, and to no tracing.", Nested: ""},
+		{Key: "sample_ratio", Type: "float64", Doc: "SampleRatio is the share of requests traced, 0 to 1, when the caller\ndid not decide it with a sampled traceparent of its own. Zero takes 1.", Nested: ""},
+	}},
+	{Package: "engines", Name: "VLLMConfig", Title: "VLLMConfig", Doc: "", Fields: []Field{
+		{Key: "bin", Type: "string", Doc: "Bin is the vllm executable.", Nested: ""},
+		{Key: "launch_mode", Type: "string", Doc: "LaunchMode selects the multi-node mechanism: \"mp\", the default, or\n\"ray\".\n\nmp is what vLLM documents today — every node runs serve with its rank,\nthe workers headless — and needs nothing but vLLM. ray is the older way,\nkept for a vLLM too old for mp's multi-node flags; it needs Ray on every\nnode and has not been checked against a real cluster.", Nested: ""},
+		{Key: "ray_bin", Type: "string", Doc: "RayBin is used when LaunchMode is \"ray\".", Nested: ""},
+	}},
+	{Package: "accounting", Name: "Price", Title: "accounting.Price", Doc: "", Fields: []Field{
+		{Key: "prompt_per_1k", Type: "int64", Doc: "", Nested: ""},
+		{Key: "completion_per_1k", Type: "int64", Doc: "", Nested: ""},
+	}},
+	{Package: "admission", Name: "Band", Title: "admission.Band", Doc: "", Fields: []Field{
+		{Key: "name", Type: "string", Doc: "", Nested: ""},
+		{Key: "priority", Type: "int", Doc: "Priority orders bands; higher is served first.", Nested: ""},
+		{Key: "saturation_ceiling", Type: "float64", Doc: "SaturationCeiling is the pool saturation above which this band waits.\n1.0 means \"never hold this band back\".", Nested: ""},
+	}},
+	{Package: "admission", Name: "Config", Title: "admission.Config", Doc: "", Fields: []Field{
+		{Key: "bands", Type: "[]Band", Doc: "", Nested: ""},
+		{Key: "default_band", Type: "string", Doc: "", Nested: ""},
+		{Key: "queue_ttl", Type: "time.Duration", Doc: "QueueTTL drops a request that has waited this long. Sending the GPU work\nthe client has already given up on helps nobody.", Nested: ""},
+		{Key: "queue_capacity_per_band", Type: "int", Doc: "QueueCapacityPerBand bounds each band's queue. Rejecting immediately\nbeats accumulating silently.", Nested: ""},
+		{Key: "queue_capacity_total", Type: "int", Doc: "QueueCapacityTotal bounds everything queued, across bands. When it is\nreached, a request is not refused while a lower band has requests\nqueued: the newest of the lowest band is turned away instead\n(displacement), since the oldest have waited longest and are nearest\ntheir TTL. Zero takes half of every band's capacity together.", Nested: ""},
+		{Key: "interval", Type: "time.Duration", Doc: "Interval is how often the dispatcher re-checks saturation. Releases also\nwake it, so this is only the backstop for saturation falling on its own.", Nested: ""},
+	}},
+	{Package: "agent", Name: "Config", Title: "agent.Config", Doc: "", Fields: []Field{
+		{Key: "node_id", Type: "string", Doc: "NodeID defaults to the hostname. It must equal the common name of the\nagent's certificate, so set it only where the two differ.", Nested: ""},
+		{Key: "server", Type: "string", Doc: "Server is the control plane, and Servers is the same thing when there is\nmore than one. Both are accepted: every configuration written before\nthere could be a second control plane names one address, and none of\nthem should have to change.\n\nThe order is tried in order, and a refusal moves to the next. Only the\nleader accepts an agent, so a standby is skipped in the time it takes to\nbe told so — which is what makes failover a reconnect rather than an\nedit to every node's configuration.", Nested: ""},
+		{Key: "servers", Type: "[]string", Doc: "", Nested: ""},
+		{Key: "hostname", Type: "string", Doc: "Hostname defaults to the machine's.", Nested: ""},
+		{Key: "mgmt_addr", Type: "string", Doc: "MgmtAddr is the address engines are reached on. It defaults to this\nmachine's address on the route to the control plane, which is the one\nthe control plane and the router can reach.", Nested: ""},
+		{Key: "labels", Type: "map[string]string", Doc: "Labels are free-form facts about a node for a profile's node_selector.\nNothing needs them: memory, GPUs, engines and disk are measured, and a\nlabel restating one of those goes stale the day the machine changes.", Nested: ""},
+		{Key: "mgmt_interface", Type: "string", Doc: "MgmtInterface is the ethernet interface NCCL bootstraps over. It\ndefaults to the interface holding MgmtAddr, then eth0.", Nested: ""},
+		{Key: "fabrics", Type: "[]inventory.FabricAttachment", Doc: "Fabrics labels this node's HCAs. It is explicit on purpose: OpenSM's\ndefault subnet prefix is the same on every fabric, so nothing can be\ninferred from the wire (plan §5.2).", Nested: ""},
+		{Key: "heartbeat_interval", Type: "time.Duration", Doc: "", Nested: ""},
+		{Key: "reconnect_base", Type: "time.Duration", Doc: "ReconnectBase is the first reconnect delay; it backs off from there.", Nested: ""},
+		{Key: "reconnect_max", Type: "time.Duration", Doc: "", Nested: ""},
+		{Key: "adopt_ready_timeout", Type: "time.Duration", Doc: "AdoptReadyTimeout bounds how long an adopted engine is given to answer a\nhealth check before the agent stops waiting on it.\n\nShorter than a startup timeout on purpose: this engine is already\nrunning, so it is not loading a model, it is either serving or it is not.", Nested: ""},
+		{Key: "tls", Type: "tlsconf.Config", Doc: "TLS authenticates both ends of the control-plane stream. Leaving it unset\nkeeps the plaintext behaviour, which only belongs on a trusted network.", Nested: "tlsconf.Config"},
+		{Key: "model_dirs", Type: "[]string", Doc: "ModelDirs are the directories this node keeps model weights in, beyond\nWeightsCache, which is always looked in. Storage differs at every installation,\nso llmgw observes it rather than owning it: a Lustre mount, an image\nexported over the fabric, local NVMe and a copy made by hand all look\nidentical from here, and placement only needs to know whether the file\nis present.", Nested: ""},
+		{Key: "engines_dir", Type: "string", Doc: "EnginesDir is where the agent installs engines, one directory per\nbuild. Defaults to /opt/llmgw/engines.", Nested: ""},
+		{Key: "weights_cache", Type: "string", Doc: "WeightsCache is where a remote source is downloaded to. Defaults to the\nfirst entry of ModelDirs, because a node that says where its models live\nhas already answered this question, and otherwise to\n/var/lib/llmgw-weights.\n\nFetching is the simple arrangement: every node keeps its own copy. The\nexotic ones need no code — weights on a shared filer or an image exported\nover the fabric are simply present, and present means nothing is fetched.", Nested: ""},
+		{Key: "discovery", Type: "discovery.Options", Doc: "", Nested: "discovery.Options"},
+		{Key: "engines", Type: "engines.Config", Doc: "", Nested: "engines.Config"},
+		{Key: "supervisor", Type: "supervisor.Config", Doc: "", Nested: "supervisor.Config"},
+	}},
+	{Package: "config", Name: "Band", Title: "config.Band", Doc: "", Fields: []Field{
+		{Key: "name", Type: "string", Doc: "", Nested: ""},
+		{Key: "priority", Type: "int", Doc: "", Nested: ""},
+		{Key: "saturation_ceiling", Type: "float64", Doc: "", Nested: ""},
+	}},
+	{Package: "config", Name: "Price", Title: "config.Price", Doc: "", Fields: []Field{
+		{Key: "prompt_per_1k", Type: "int64", Doc: "", Nested: ""},
+		{Key: "completion_per_1k", Type: "int64", Doc: "", Nested: ""},
+	}},
+	{Package: "engines", Name: "Config", Title: "engines.Config", Doc: "", Fields: []Field{
+		{Key: "disabled", Type: "[]string", Doc: "Disabled are engines this node never installs or advertises, whatever\nthe control plane asks: vLLM on a node of cards no build of it runs on,\nthat is meant for llama.cpp alone.", Nested: ""},
+		{Key: "vllm", Type: "VLLMConfig", Doc: "", Nested: "engines.VLLMConfig"},
+		{Key: "sglang", Type: "SGLangConfig", Doc: "", Nested: "engines.SGLangConfig"},
+		{Key: "llamacpp", Type: "LlamaCppConfig", Doc: "", Nested: "engines.LlamaCppConfig"},
+		{Key: "sim", Type: "SimConfig", Doc: "", Nested: "engines.SimConfig"},
+	}},
+	{Package: "supervisor", Name: "Config", Title: "supervisor.Config", Doc: "", Fields: []Field{
+		{Key: "work_dir", Type: "string", Doc: "WorkDir holds pid files and per-shard scratch space.", Nested: ""},
+		{Key: "log_lines", Type: "int", Doc: "LogLines is how many lines of engine output to retain per shard.", Nested: ""},
+		{Key: "stop_timeout", Type: "time.Duration", Doc: "StopTimeout is how long a process gets to exit after SIGTERM before it is\nkilled. Engines need time to free GPU memory cleanly.", Nested: ""},
+		{Key: "health_interval", Type: "time.Duration", Doc: "HealthInterval is how often a starting shard is polled.", Nested: ""},
+		{Key: "rendezvous_timeout", Type: "time.Duration", Doc: "RendezvousTimeout bounds the wait for a multi-node rendezvous point to\nstart listening.", Nested: ""},
+		{Key: "alive_grace", Type: "time.Duration", Doc: "AliveGrace is how long a non-serving rank must stay up before it counts\nas ready. There is no external signal for a worker, so staying alive is\nthe only evidence available.", Nested: ""},
+	}},
+	{Package: "tlsconf", Name: "Config", Title: "tlsconf.Config", Doc: "", Fields: []Field{
+		{Key: "ca", Type: "string", Doc: "CA is the bundle used to verify the other side. In a real deployment this\nis whichever authority issued the fleet's certificates.", Nested: ""},
+		{Key: "cert", Type: "string", Doc: "Cert and Key are this component's own identity.", Nested: ""},
+		{Key: "key", Type: "string", Doc: "", Nested: ""},
+		{Key: "insecure", Type: "bool", Doc: "Insecure disables TLS entirely. It exists for local runs against\nsim-engines; anything using it must stay on a trusted network.", Nested: ""},
+	}},
+}

internal/docgen/render.go

@@ -61,13 +61,13 @@ rather than repeated.
 	for _, bl := range blocks {
 		for _, f := range bl.Fields {
 			b.WriteString(fmt.Sprintf("| `%s` | %s | [%s](#%s) |\n",
-				f.Key, code(f.Type), bl.Name, anchor(bl.Name)))
+				f.Key, code(f.Type), bl.Title, anchor(bl.Title)))
 		}
 	}
 	b.WriteString("\n")
 
 	for _, bl := range blocks {
-		fmt.Fprintf(&b, "## %s\n\n", bl.Name)
+		fmt.Fprintf(&b, "## %s\n\n", bl.Title)
 		if bl.Doc != "" {
 			b.WriteString(bl.Doc + "\n\n")
 		}
@@ -76,24 +76,44 @@ rather than repeated.
 			if f.Doc != "" {
 				b.WriteString(f.Doc + "\n\n")
 			}
-			if f.Nested != "" && hasBlock(blocks, f.Nested) {
-				fmt.Fprintf(&b, "Takes a [%s](#%s) block.\n\n", f.Nested, anchor(f.Nested))
+			if nb, ok := Find(blocks, f.Nested); ok {
+				fmt.Fprintf(&b, "Takes a [%s](#%s) block.\n\n", nb.Title, anchor(nb.Title))
 			}
 		}
 	}
 	return b.String()
 }
 
-func hasBlock(blocks []Block, name string) bool {
+// Find is the block with an ID, as a Field's Nested names it.
+func Find(blocks []Block, id string) (Block, bool) {
 	for _, b := range blocks {
-		if b.Name == name {
-			return true
+		if id != "" && b.ID() == id {
+			return b, true
 		}
 	}
-	return false
+	return Block{}, false
 }
 
 func code(s string) string { return "`" + s + "`" }
 
-// anchor is how GitHub and Gitea slug a heading.
-func anchor(s string) string { return strings.ToLower(s) }
+// anchor is how GitHub and Gitea slug a heading: lower case, and punctuation
+// such as the dot in tlsconf.Config dropped.
+func anchor(s string) string { return strings.ToLower(strings.ReplaceAll(s, ".", "")) }
+
+// RenderGo writes the blocks as Go, for the settings screens: every key, its
+// type and what it does, from the same source as the reference, so a setting
+// added to the code is on the screens without anyone listing it.
+func RenderGo(blocks []Block) string {
+	var b strings.Builder
+	b.WriteString("// Code generated by go run ./internal/docgen/gen; DO NOT EDIT.\n\npackage docgen\n\n")
+	b.WriteString("// Reference is every configuration block, as Collect reads them.\nvar Reference = []Block{\n")
+	for _, bl := range blocks {
+		fmt.Fprintf(&b, "\t{Package: %q, Name: %q, Title: %q, Doc: %q, Fields: []Field{\n", bl.Package, bl.Name, bl.Title, bl.Doc)
+		for _, f := range bl.Fields {
+			fmt.Fprintf(&b, "\t\t{Key: %q, Type: %q, Doc: %q, Nested: %q},\n", f.Key, f.Type, f.Doc, f.Nested)
+		}
+		b.WriteString("\t}},\n")
+	}
+	b.WriteString("}\n")
+	return b.String()
+}

internal/join/join.go

@@ -0,0 +1,302 @@
+// Package join lets a machine be set up without anything logging into it.
+//
+// The machine doing the setup — running as root, holding the authority —
+// listens for a while and hands out one code per machine it expects. On the
+// other machine `sudo llmgw join <code>` dials out to it, checks it is talking
+// to the listener the code names, proves itself with the code's secret, and
+// from then on runs what it is sent, as ssh would, until the setup closes
+// the connection. To the setup it is one more onboard.Remote: preflight,
+// Patroni and node onboarding run over it unchanged.
+//
+// For sites where nothing may log into a machine from outside: the
+// connection is outbound from the machine being set up, no password is
+// typed anywhere, and the code is good for one connection and a limited time.
+package join
+
+import (
+	"bufio"
+	"context"
+	"crypto/ecdsa"
+	"crypto/elliptic"
+	"crypto/rand"
+	"crypto/sha256"
+	"crypto/subtle"
+	"crypto/tls"
+	"crypto/x509"
+	"crypto/x509/pkix"
+	"encoding/base64"
+	"encoding/binary"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"math/big"
+	"net"
+	"strings"
+	"sync"
+	"time"
+)
+
+// DefaultPort is where a setup listens for machines joining it.
+const DefaultPort = 7444
+
+// codePrefix marks a join code, and its version.
+const codePrefix = "llmgw-join1-"
+
+// Code is what `llmgw join` is given: where to dial, how to know it reached
+// the right place, and the secret that admits it.
+//
+// Short, because somebody copies it from one screen into another: the first
+// version was JSON and over 250 characters, which a terminal cut off at its
+// edge. Packed it is under 90.
+type Code struct {
+	Addr string
+	// Fingerprint is the first 16 bytes of the SHA-256 of the listener's
+	// certificate: 128 bits, more than enough to find nothing else by.
+	Fingerprint [16]byte
+	Secret      [16]byte
+	Expires     int64
+	// Role says what the machine is being set up as, for its operator to
+	// read before it runs: "database", "node" or "control plane".
+	Role string
+}
+
+var roles = []string{"", "database", "node", "control plane"}
+
+// String is the code as typed: one word, safe in a shell.
+func (c Code) String() string {
+	b := make([]byte, 0, 40+len(c.Addr))
+	b = append(b, c.Fingerprint[:]...)
+	b = append(b, c.Secret[:]...)
+	b = binary.BigEndian.AppendUint32(b, uint32(c.Expires))
+	role := byte(0)
+	for i, r := range roles {
+		if r == c.Role {
+			role = byte(i)
+		}
+	}
+	b = append(b, role)
+	b = append(b, c.Addr...)
+	return codePrefix + base64.RawURLEncoding.EncodeToString(b)
+}
+
+// Parse reads a code back.
+func Parse(s string) (Code, error) {
+	raw, ok := strings.CutPrefix(strings.TrimSpace(s), codePrefix)
+	if !ok {
+		return Code{}, errors.New("not a join code: it starts " + codePrefix)
+	}
+	b, err := base64.RawURLEncoding.DecodeString(raw)
+	if err != nil || len(b) < 16+16+4+1+3 {
+		return Code{}, errors.New("the join code is damaged: copy it whole")
+	}
+	var c Code
+	copy(c.Fingerprint[:], b[:16])
+	copy(c.Secret[:], b[16:32])
+	c.Expires = int64(binary.BigEndian.Uint32(b[32:36]))
+	if int(b[36]) < len(roles) {
+		c.Role = roles[b[36]]
+	}
+	c.Addr = string(b[37:])
+	if _, _, err := net.SplitHostPort(c.Addr); err != nil {
+		return Code{}, errors.New("the join code is damaged: copy it whole")
+	}
+	if time.Now().Unix() > c.Expires {
+		return Code{}, fmt.Errorf("the join code expired at %s; ask for a new one",
+			time.Unix(c.Expires, 0).Format(time.RFC3339))
+	}
+	return c, nil
+}
+
+// The wire: one JSON object a line, each way.
+type hello struct {
+	Secret   []byte `json:"secret"`
+	Hostname string `json:"hostname"`
+	// Addr is the address the machine dialled out from: how the others
+	// reach it, when nothing else says.
+	Addr string `json:"addr"`
+}
+
+type request struct {
+	Script string `json:"script,omitempty"`
+	Stdin  []byte `json:"stdin,omitempty"`
+	Bye    bool   `json:"bye,omitempty"`
+}
+
+type response struct {
+	Stdout []byte `json:"stdout,omitempty"`
+	Stderr []byte `json:"stderr,omitempty"`
+	Exit   int    `json:"exit"`
+	Error  string `json:"error,omitempty"`
+}
+
+// Listener waits for machines to join.
+type Listener struct {
+	ln          net.Listener
+	addr        string
+	fingerprint [16]byte
+	ttl         time.Duration
+
+	mu      sync.Mutex
+	pending map[[16]byte]chan *Remote // by secret
+}
+
+// Listen starts listening on bind (":7444") and advertises addr ("10.0.0.5:7444")
+// in its codes. Codes are good for ttl.
+func Listen(bind, addr string, ttl time.Duration) (*Listener, error) {
+	cert, fp, err := sessionCert()
+	if err != nil {
+		return nil, err
+	}
+	ln, err := tls.Listen("tcp", bind, &tls.Config{Certificates: []tls.Certificate{cert}, MinVersion: tls.VersionTLS13})
+	if err != nil {
+		return nil, fmt.Errorf("listening for machines to join on %s: %w", bind, err)
+	}
+	l := &Listener{ln: ln, addr: addr, fingerprint: fp, ttl: ttl, pending: map[[16]byte]chan *Remote{}}
+	go l.accept()
+	return l, nil
+}
+
+// sessionCert is a certificate for this listener alone, trusted by the
+// fingerprint in its codes rather than by any authority: it lives as long as
+// the setup does.
+func sessionCert() (tls.Certificate, [16]byte, error) {
+	var fp [16]byte
+	key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
+	if err != nil {
+		return tls.Certificate{}, fp, err
+	}
+	tmpl := &x509.Certificate{
+		SerialNumber: big.NewInt(time.Now().UnixNano()),
+		Subject:      pkix.Name{CommonName: "llmgw setup"},
+		NotBefore:    time.Now().Add(-time.Minute),
+		NotAfter:     time.Now().Add(48 * time.Hour),
+		KeyUsage:     x509.KeyUsageDigitalSignature,
+		ExtKeyUsage:  []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
+	}
+	der, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, &key.PublicKey, key)
+	if err != nil {
+		return tls.Certificate{}, fp, err
+	}
+	sum := sha256.Sum256(der)
+	copy(fp[:], sum[:16])
+	return tls.Certificate{Certificate: [][]byte{der}, PrivateKey: key}, fp, nil
+}
+
+// Code makes a code for one machine, and the channel its connection arrives
+// on.
+func (l *Listener) Code(role string) (Code, <-chan *Remote) {
+	var secret [16]byte
+	_, _ = rand.Read(secret[:])
+	ch := make(chan *Remote, 1)
+	l.mu.Lock()
+	l.pending[secret] = ch
+	l.mu.Unlock()
+	return Code{Addr: l.addr, Fingerprint: l.fingerprint, Secret: secret,
+		Expires: time.Now().Add(l.ttl).Unix(), Role: role}, ch
+}
+
+// Close stops listening; machines already joined stay connected until their
+// Remote is closed.
+func (l *Listener) Close() error { return l.ln.Close() }
+
+func (l *Listener) accept() {
+	for {
+		c, err := l.ln.Accept()
+		if err != nil {
+			return
+		}
+		go l.admit(c)
+	}
+}
+
+func (l *Listener) admit(c net.Conn) {
+	_ = c.SetDeadline(time.Now().Add(30 * time.Second))
+	r := bufio.NewReader(c)
+	var h hello
+	line, err := r.ReadBytes('\n')
+	if err != nil || json.Unmarshal(line, &h) != nil {
+		_ = c.Close()
+		return
+	}
+	l.mu.Lock()
+	var ch chan *Remote
+	for secret, p := range l.pending {
+		if subtle.ConstantTimeCompare(secret[:], h.Secret) == 1 {
+			ch = p
+			// One machine a code: whoever comes second is turned away.
+			delete(l.pending, secret)
+		}
+	}
+	l.mu.Unlock()
+	if ch == nil {
+		_, _ = c.Write([]byte(`{"error":"this code is not one the setup is waiting for, or it was used already"}` + "\n"))
+		_ = c.Close()
+		return
+	}
+	_ = c.SetDeadline(time.Time{})
+	if _, err := c.Write([]byte("{}\n")); err != nil {
+		_ = c.Close()
+		return
+	}
+	addr := h.Addr
+	if addr == "" {
+		addr, _, _ = net.SplitHostPort(c.RemoteAddr().String())
+	}
+	ch <- &Remote{conn: c, r: r, Hostname: h.Hostname, Addr: addr}
+}
+
+// Remote is a joined machine, as the setup runs things on it.
+type Remote struct {
+	Hostname string
+	Addr     string
+
+	mu   sync.Mutex
+	conn net.Conn
+	r    *bufio.Reader
+}
+
+// Run runs script with sh on the machine, as root.
+func (m *Remote) Run(ctx context.Context, script string, stdin []byte) ([]byte, error) {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+	if m.conn == nil {
+		return nil, fmt.Errorf("%s is no longer connected", m.Hostname)
+	}
+	if dl, ok := ctx.Deadline(); ok {
+		_ = m.conn.SetDeadline(dl)
+		defer m.conn.SetDeadline(time.Time{})
+	}
+	b, _ := json.Marshal(request{Script: script, Stdin: stdin})
+	if _, err := m.conn.Write(append(b, '\n')); err != nil {
+		return nil, fmt.Errorf("%s: %w", m.Hostname, err)
+	}
+	line, err := m.r.ReadBytes('\n')
+	if err != nil {
+		return nil, fmt.Errorf("%s went away: %w", m.Hostname, err)
+	}
+	var resp response
+	if err := json.Unmarshal(line, &resp); err != nil {
+		return nil, fmt.Errorf("%s answered something unreadable", m.Hostname)
+	}
+	if resp.Error != "" {
+		return append(resp.Stdout, resp.Stderr...), errors.New(resp.Error)
+	}
+	if resp.Exit != 0 {
+		return append(resp.Stdout, resp.Stderr...), fmt.Errorf("exit status %d", resp.Exit)
+	}
+	return resp.Stdout, nil
+}
+
+// Close tells the machine the setup is done, which ends its `llmgw join`.
+func (m *Remote) Close() error {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+	if m.conn == nil {
+		return nil
+	}
+	b, _ := json.Marshal(request{Bye: true})
+	_, _ = m.conn.Write(append(b, '\n'))
+	err := m.conn.Close()
+	m.conn = nil
+	return err
+}

internal/join/join_test.go

@@ -0,0 +1,94 @@
+package join
+
+import (
+	"context"
+	"strings"
+	"testing"
+	"time"
+)
+
+func listen(t *testing.T) *Listener {
+	t.Helper()
+	l, err := Listen("127.0.0.1:0", "", time.Hour)
+	if err != nil {
+		t.Fatal(err)
+	}
+	l.addr = l.ln.Addr().String()
+	t.Cleanup(func() { _ = l.Close() })
+	return l
+}
+
+func TestAJoinedMachineRunsWhatTheSetupSendsUntilItIsDone(t *testing.T) {
+	l := listen(t)
+	code, arrived := l.Code("database")
+	parsed, err := Parse(code.String())
+	if err != nil || parsed.Role != "database" {
+		t.Fatalf("%v %+v", err, parsed)
+	}
+	var said []string
+	served := make(chan error, 1)
+	go func() { served <- Serve(context.Background(), parsed, func(s string) { said = append(said, s) }) }()
+
+	var m *Remote
+	select {
+	case m = <-arrived:
+	case err := <-served:
+		t.Fatalf("join ended before arriving: %v", err)
+	case <-time.After(5 * time.Second):
+		t.Fatal("the machine never arrived")
+	}
+	if m.Hostname == "" || m.Addr != "127.0.0.1" {
+		t.Errorf("arrived as %q from %q", m.Hostname, m.Addr)
+	}
+	out, err := m.Run(context.Background(), "set -e\ntr a-z A-Z", []byte("hello\n"))
+	if err != nil || string(out) != "HELLO\n" {
+		t.Fatalf("%q, %v", out, err)
+	}
+	if out, err := m.Run(context.Background(), "echo nope >&2; exit 3", nil); err == nil || !strings.Contains(string(out), "nope") {
+		t.Errorf("a failing script: %q, %v", out, err)
+	}
+	_ = m.Close()
+	if err := <-served; err != nil {
+		t.Errorf("join did not end cleanly when the setup was done: %v", err)
+	}
+	if len(said) < 2 || !strings.Contains(strings.Join(said, "\n"), "tr a-z A-Z") {
+		t.Errorf("the machine's operator was not told what ran: %q", said)
+	}
+
+	// The code was for one machine.
+	if err := Serve(context.Background(), parsed, func(string) {}); err == nil || !strings.Contains(err.Error(), "used already") {
+		t.Errorf("a code used twice: %v", err)
+	}
+}
+
+func TestAJoinRefusesAListenerThatIsNotTheOneInTheCode(t *testing.T) {
+	l := listen(t)
+	code, _ := l.Code("node")
+	other := listen(t)
+	code.Addr = other.addr // what the code names is elsewhere
+	err := Serve(context.Background(), code, func(string) {})
+	if err == nil || !strings.Contains(err.Error(), "not the setup") {
+		t.Errorf("joined a stranger: %v", err)
+	}
+}
+
+func TestAnExpiredOrDamagedCodeIsRefusedBeforeDialling(t *testing.T) {
+	c := Code{Addr: "10.0.0.5:7444", Role: "node", Expires: time.Now().Add(-time.Minute).Unix()}
+	if _, err := Parse(c.String()); err == nil || !strings.Contains(err.Error(), "expired") {
+		t.Errorf("expired: %v", err)
+	}
+	if _, err := Parse(c.String()[:30]); err == nil {
+		t.Error("a truncated code was read")
+	}
+	if _, err := Parse("ssh-rsa AAAA"); err == nil {
+		t.Error("something else was read as a code")
+	}
+	// Short enough to copy from a screen in one line.
+	c.Expires = time.Now().Add(time.Hour).Unix()
+	if n := len(c.String()); n > 100 {
+		t.Errorf("a code is %d characters", n)
+	}
+	if back, err := Parse(c.String()); err != nil || back.Addr != c.Addr || back.Role != "node" {
+		t.Errorf("%+v, %v", back, err)
+	}
+}

internal/join/serve.go

@@ -0,0 +1,126 @@
+package join
+
+import (
+	"bufio"
+	"bytes"
+	"context"
+	"crypto/sha256"
+	"crypto/subtle"
+	"crypto/tls"
+	"crypto/x509"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"net"
+	"os"
+	"os/exec"
+	"strings"
+	"time"
+)
+
+// Serve dials the setup a code names and runs what it sends until it says it
+// is done. Each script is reported to log by its first line, so whoever ran
+// `llmgw join` sees what is being done to their machine.
+func Serve(ctx context.Context, c Code, log func(string)) error {
+	d := tls.Dialer{
+		NetDialer: &net.Dialer{Timeout: 10 * time.Second},
+		Config: &tls.Config{
+			MinVersion: tls.VersionTLS13,
+			// Trusted by the fingerprint in the code, not by an authority:
+			// the listener's certificate is made for this setup alone.
+			InsecureSkipVerify: true, //nolint:gosec // verified below, by fingerprint
+		},
+	}
+	want := c.Fingerprint[:]
+	d.Config.VerifyPeerCertificate = func(raw [][]byte, _ [][]*x509.Certificate) error {
+		if len(raw) == 0 {
+			return errors.New("the setup sent no certificate")
+		}
+		sum := sha256.Sum256(raw[0])
+		if subtle.ConstantTimeCompare(sum[:16], want) != 1 {
+			return errors.New("the machine at that address is not the setup this code is for")
+		}
+		return nil
+	}
+	conn, err := d.DialContext(ctx, "tcp", c.Addr)
+	if err != nil {
+		return fmt.Errorf("cannot reach the setup at %s: %w; it listens there only while it waits for this machine", c.Addr, err)
+	}
+	defer conn.Close()
+
+	host, _ := os.Hostname()
+	local, _, _ := net.SplitHostPort(conn.LocalAddr().String())
+	b, _ := json.Marshal(hello{Secret: c.Secret[:], Hostname: host, Addr: local})
+	if _, err := conn.Write(append(b, '\n')); err != nil {
+		return err
+	}
+	r := bufio.NewReader(conn)
+	line, err := r.ReadBytes('\n')
+	if err != nil {
+		return fmt.Errorf("the setup closed the connection: %w", err)
+	}
+	var ack response
+	if err := json.Unmarshal(line, &ack); err != nil {
+		return errors.New("the setup answered something unreadable")
+	}
+	if ack.Error != "" {
+		return errors.New(ack.Error)
+	}
+	log(fmt.Sprintf("joined the setup at %s; it is setting this machine up", c.Addr))
+
+	for {
+		line, err := r.ReadBytes('\n')
+		if err != nil {
+			return fmt.Errorf("the setup went away before it said it was done: %w", err)
+		}
+		var req request
+		if err := json.Unmarshal(line, &req); err != nil {
+			return errors.New("the setup sent something unreadable")
+		}
+		if req.Bye {
+			log("the setup is done with this machine")
+			return nil
+		}
+		log(summary(req.Script))
+		resp := run(ctx, req)
+		out, _ := json.Marshal(resp)
+		if _, err := conn.Write(append(out, '\n')); err != nil {
+			return err
+		}
+	}
+}
+
+// run runs one script as the setup asked.
+func run(ctx context.Context, req request) response {
+	cmd := exec.CommandContext(ctx, "sh", "-c", req.Script)
+	if req.Stdin != nil {
+		cmd.Stdin = bytes.NewReader(req.Stdin)
+	}
+	var stdout, stderr bytes.Buffer
+	cmd.Stdout, cmd.Stderr = &stdout, &stderr
+	err := cmd.Run()
+	resp := response{Stdout: stdout.Bytes(), Stderr: stderr.Bytes()}
+	var exit *exec.ExitError
+	switch {
+	case errors.As(err, &exit):
+		resp.Exit = exit.ExitCode()
+	case err != nil:
+		resp.Error = err.Error()
+	}
+	return resp
+}
+
+// summary is a script's first line that says something.
+func summary(script string) string {
+	for _, l := range strings.Split(script, "\n") {
+		l = strings.TrimSpace(l)
+		if l == "" || l == "set -e" || strings.HasPrefix(l, "#") {
+			continue
+		}
+		if len(l) > 100 {
+			l = l[:100] + "…"
+		}
+		return "  " + l
+	}
+	return "  (an empty script)"
+}

internal/onboard/gossh.go

@@ -0,0 +1,278 @@
+package onboard
+
+import (
+	"bytes"
+	"context"
+	"errors"
+	"fmt"
+	"net"
+	"os"
+	"path/filepath"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+
+	"golang.org/x/crypto/ssh"
+	"golang.org/x/crypto/ssh/agent"
+	"golang.org/x/crypto/ssh/knownhosts"
+)
+
+// GoSSH is Remote through Go's own ssh client rather than the system's: what
+// the setup screens use, because they ask for a password when there is no key,
+// which the system ssh in batch mode cannot take.
+//
+// Authentication is whichever is given, tried in order: the running ssh
+// agent, a key file, a password. The password is held in memory for the
+// session and never written anywhere.
+type GoSSH struct {
+	Addr string
+	Port int
+	User string
+
+	UseAgent bool
+	KeyFile  string
+	// KeyPassphrase opens an encrypted KeyFile.
+	KeyPassphrase string
+	Password      string
+
+	// SudoPassword is asked for when the user cannot sudo without one. With
+	// it, PrepareSudo lets the user sudo without a password until
+	// RevokeSudo, so every script can use plain sudo as on any other node.
+	SudoPassword string
+
+	// KnownHosts is where host keys are kept; "" is ~/.ssh/known_hosts.
+	KnownHosts string
+
+	mu        sync.Mutex
+	client    *ssh.Client
+	sudoGrant bool
+}
+
+// Run runs script with sh on the machine.
+func (g *GoSSH) Run(ctx context.Context, script string, stdin []byte) ([]byte, error) {
+	c, err := g.dial(ctx)
+	if err != nil {
+		return nil, err
+	}
+	sess, err := c.NewSession()
+	if err != nil {
+		// A connection that died since: dial again once.
+		g.drop()
+		if c, err = g.dial(ctx); err != nil {
+			return nil, err
+		}
+		if sess, err = c.NewSession(); err != nil {
+			return nil, err
+		}
+	}
+	defer sess.Close()
+	if stdin != nil {
+		sess.Stdin = bytes.NewReader(stdin)
+	}
+	var stdout, stderr bytes.Buffer
+	sess.Stdout, sess.Stderr = &stdout, &stderr
+	done := make(chan error, 1)
+	go func() { done <- sess.Run("sh -c " + shellQuote(script)) }()
+	select {
+	case <-ctx.Done():
+		_ = sess.Signal(ssh.SIGKILL)
+		return stdout.Bytes(), ctx.Err()
+	case err := <-done:
+		if err != nil {
+			return append(stdout.Bytes(), stderr.Bytes()...), err
+		}
+		return stdout.Bytes(), nil
+	}
+}
+
+// Close ends the connection, taking back a sudo grant first.
+func (g *GoSSH) Close() error {
+	_ = g.RevokeSudo(context.Background())
+	g.drop()
+	return nil
+}
+
+func (g *GoSSH) drop() {
+	g.mu.Lock()
+	defer g.mu.Unlock()
+	if g.client != nil {
+		_ = g.client.Close()
+		g.client = nil
+	}
+}
+
+func (g *GoSSH) target() string {
+	port := g.Port
+	if port == 0 {
+		port = 22
+	}
+	return net.JoinHostPort(g.Addr, strconv.Itoa(port))
+}
+
+func (g *GoSSH) dial(ctx context.Context) (*ssh.Client, error) {
+	g.mu.Lock()
+	defer g.mu.Unlock()
+	if g.client != nil {
+		return g.client, nil
+	}
+	var methods []ssh.AuthMethod
+	if g.UseAgent {
+		if sock := os.Getenv("SSH_AUTH_SOCK"); sock != "" {
+			if conn, err := net.Dial("unix", sock); err == nil {
+				methods = append(methods, ssh.PublicKeysCallback(agent.NewClient(conn).Signers))
+			}
+		}
+	}
+	if g.KeyFile != "" {
+		signer, err := loadKey(g.KeyFile, g.KeyPassphrase)
+		if err != nil {
+			return nil, err
+		}
+		methods = append(methods, ssh.PublicKeys(signer))
+	}
+	if g.Password != "" {
+		pw := g.Password
+		methods = append(methods, ssh.Password(pw),
+			ssh.KeyboardInteractive(func(_, _ string, qs []string, _ []bool) ([]string, error) {
+				out := make([]string, len(qs))
+				for i := range qs {
+					out[i] = pw
+				}
+				return out, nil
+			}))
+	}
+	if len(methods) == 0 {
+		return nil, fmt.Errorf("no way to log in to %s: no ssh agent, key or password", g.Addr)
+	}
+	hostKey, err := g.hostKeyCallback()
+	if err != nil {
+		return nil, err
+	}
+	cfg := &ssh.ClientConfig{User: g.User, Auth: methods, HostKeyCallback: hostKey, Timeout: 10 * time.Second}
+	d := net.Dialer{Timeout: 10 * time.Second}
+	conn, err := d.DialContext(ctx, "tcp", g.target())
+	if err != nil {
+		return nil, fmt.Errorf("cannot reach %s: %w", g.target(), err)
+	}
+	cc, chans, reqs, err := ssh.NewClientConn(conn, g.target(), cfg)
+	if err != nil {
+		_ = conn.Close()
+		return nil, fmt.Errorf("cannot log in to %s as %s: %w", g.Addr, g.User, err)
+	}
+	g.client = ssh.NewClient(cc, chans, reqs)
+	return g.client, nil
+}
+
+func loadKey(path, passphrase string) (ssh.Signer, error) {
+	b, err := os.ReadFile(path)
+	if err != nil {
+		return nil, fmt.Errorf("reading the key %s: %w", path, err)
+	}
+	if passphrase != "" {
+		return ssh.ParsePrivateKeyWithPassphrase(b, []byte(passphrase))
+	}
+	s, err := ssh.ParsePrivateKey(b)
+	var missing *ssh.PassphraseMissingError
+	if errors.As(err, &missing) {
+		return nil, fmt.Errorf("the key %s needs its passphrase", path)
+	}
+	return s, err
+}
+
+// hostKeyCallback trusts a machine on first sight and records its key, and
+// refuses one whose key has changed since — accept-new, as onboarding's
+// system ssh does: a machine provisioned a minute ago has a key nobody has
+// seen, and the risk guarded against is a machine substituted for one that
+// was known.
+func (g *GoSSH) hostKeyCallback() (ssh.HostKeyCallback, error) {
+	path := g.KnownHosts
+	if path == "" {
+		home, err := os.UserHomeDir()
+		if err != nil {
+			return nil, err
+		}
+		path = filepath.Join(home, ".ssh", "known_hosts")
+	}
+	if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil {
+		return nil, err
+	}
+	if f, err := os.OpenFile(path, os.O_CREATE|os.O_RDONLY, 0o600); err == nil {
+		_ = f.Close()
+	}
+	known, err := knownhosts.New(path)
+	if err != nil {
+		return nil, fmt.Errorf("reading %s: %w", path, err)
+	}
+	return func(host string, remote net.Addr, key ssh.PublicKey) error {
+		err := known(host, remote, key)
+		var ke *knownhosts.KeyError
+		if errors.As(err, &ke) && len(ke.Want) == 0 {
+			f, ferr := os.OpenFile(path, os.O_APPEND|os.O_WRONLY, 0o600)
+			if ferr != nil {
+				return ferr
+			}
+			defer f.Close()
+			_, ferr = fmt.Fprintln(f, knownhosts.Line([]string{knownhosts.Normalize(host)}, key))
+			return ferr
+		}
+		if errors.As(err, &ke) {
+			return fmt.Errorf("%s's host key is not the one recorded in %s: "+
+				"if the machine was rebuilt, remove its line there; otherwise something is in the way", host, path)
+		}
+		return err
+	}, nil
+}
+
+// sudoersDrop is the grant PrepareSudo writes and RevokeSudo removes.
+const sudoersDrop = "/etc/sudoers.d/llmgw-setup"
+
+// CanSudo reports whether the user can sudo without a password now.
+func (g *GoSSH) CanSudo(ctx context.Context) bool {
+	_, err := g.Run(ctx, "sudo -n true", nil)
+	return err == nil
+}
+
+// PrepareSudo makes plain sudo work for the rest of the session: nothing to do
+// when it already does, otherwise a sudoers grant written with SudoPassword
+// and taken back by RevokeSudo. A grant rather than the password piped into
+// every sudo, because the scripts feed their own standard input to the
+// commands they run under it.
+func (g *GoSSH) PrepareSudo(ctx context.Context) error {
+	if g.CanSudo(ctx) {
+		return nil
+	}
+	if g.SudoPassword == "" {
+		return fmt.Errorf("%s cannot use sudo without a password, and none was given", g.User)
+	}
+	line := fmt.Sprintf("%s ALL=(ALL) NOPASSWD:ALL", g.User)
+	// Checked by visudo before it is put in place: a broken file in
+	// sudoers.d breaks sudo for everyone on the machine.
+	inner := fmt.Sprintf("umask 337; printf '%%s\\n' %s %s > %[3]s.new && visudo -cf %[3]s.new >/dev/null && mv -f %[3]s.new %[3]s",
+		shellQuote("# llmgw setup: removed when it finishes"), shellQuote(line), sudoersDrop)
+	script := "sudo -S -p '' sh -c " + shellQuote(inner)
+	if out, err := g.Run(ctx, script, []byte(g.SudoPassword+"\n")); err != nil {
+		return fmt.Errorf("sudo with the password given failed on %s: %v: %s", g.Addr, err, strings.TrimSpace(string(out)))
+	}
+	g.mu.Lock()
+	g.sudoGrant = true
+	g.mu.Unlock()
+	return nil
+}
+
+// RevokeSudo removes the grant PrepareSudo wrote, if it wrote one.
+func (g *GoSSH) RevokeSudo(ctx context.Context) error {
+	g.mu.Lock()
+	granted := g.sudoGrant
+	g.mu.Unlock()
+	if !granted {
+		return nil
+	}
+	if _, err := g.Run(ctx, "sudo rm -f "+sudoersDrop, nil); err != nil {
+		return err
+	}
+	g.mu.Lock()
+	g.sudoGrant = false
+	g.mu.Unlock()
+	return nil
+}

internal/pgha/config.go

@@ -0,0 +1,178 @@
+// Package pgha sets up the store as PostgreSQL under Patroni: two or more
+// machines, one primary streaming to the others, and a standby promoted when
+// the primary is lost. Patroni keeps its leader lock in etcd, which decides
+// only with a majority, so etcd runs on three machines — the database
+// machines and, when they are two, the control plane as the third vote.
+//
+// Everything is done over the same ssh the rest of onboarding uses, and every
+// step looks before it acts: run against a cluster that is already up, it
+// changes nothing.
+package pgha
+
+import (
+	"fmt"
+	"net/url"
+	"strings"
+)
+
+// Scope is the Patroni cluster's name, and the etcd cluster token.
+const Scope = "llmgw"
+
+// PatroniPort is where Patroni's REST API listens on every member: what the
+// control plane reads the cluster's state from.
+const PatroniPort = 8008
+
+// restPort is where this package asks Patroni; a test moves it.
+var restPort = PatroniPort
+
+// Role and Database are what the control plane connects as.
+const (
+	Role     = "llmgw"
+	Database = "llmgw"
+)
+
+// Member is one machine of the cluster, by the address the others reach it on.
+type Member struct {
+	Name string
+	IP   string
+}
+
+// Secrets are the cluster's three passwords.
+type Secrets struct {
+	Superuser, Replicator, App string
+}
+
+// Etcd is a member's /etc/default/etcd, for Ubuntu's etcd-server.
+func Etcd(m Member, all []Member) string {
+	var peers []string
+	for _, p := range all {
+		peers = append(peers, fmt.Sprintf("%s=http://%s:2380", p.Name, p.IP))
+	}
+	return fmt.Sprintf(`# llmgw: one member of the etcd that holds Patroni's leader lock.
+ETCD_NAME=%[1]s
+ETCD_DATA_DIR=/var/lib/etcd/default
+ETCD_LISTEN_PEER_URLS=http://%[2]s:2380
+ETCD_LISTEN_CLIENT_URLS=http://%[2]s:2379,http://127.0.0.1:2379
+ETCD_INITIAL_ADVERTISE_PEER_URLS=http://%[2]s:2380
+ETCD_ADVERTISE_CLIENT_URLS=http://%[2]s:2379
+ETCD_INITIAL_CLUSTER=%[3]s
+ETCD_INITIAL_CLUSTER_STATE=new
+ETCD_INITIAL_CLUSTER_TOKEN=%[4]s
+`, m.Name, m.IP, strings.Join(peers, ","), Scope+"-patroni")
+}
+
+// Patroni is a database member's /etc/patroni/config.yml.
+//
+// clients are the addresses the control planes connect from; the members
+// themselves are always allowed, since a second control plane usually runs on
+// one of them.
+func Patroni(m Member, db, etcd []Member, clients []string, version string, s Secrets) string {
+	var hosts []string
+	for _, e := range etcd {
+		hosts = append(hosts, e.IP+":2379")
+	}
+	var hba []string
+	hba = append(hba, "local all all peer", "host all all 127.0.0.1/32 scram-sha-256")
+	for _, d := range db {
+		hba = append(hba,
+			"host replication replicator "+d.IP+"/32 scram-sha-256",
+			"host all postgres "+d.IP+"/32 scram-sha-256")
+	}
+	seen := map[string]bool{}
+	for _, c := range append(append([]string{}, clients...), ips(db)...) {
+		if c == "" || seen[c] {
+			continue
+		}
+		seen[c] = true
+		hba = append(hba, fmt.Sprintf("host %s %s %s/32 scram-sha-256", Database, Role, c))
+	}
+	var hbaLines string
+	for _, h := range hba {
+		hbaLines += "        - " + h + "\n"
+	}
+	var b strings.Builder
+	fmt.Fprintf(&b, `# llmgw: the control planes' store, kept by Patroni, its leader lock in etcd.
+scope: %[1]s
+namespace: /service/
+name: %[2]s
+
+restapi:
+  listen: %[3]s:%[4]d
+  connect_address: %[3]s:%[4]d
+
+etcd3:
+  hosts: %[5]s
+
+bootstrap:
+  dcs:
+    ttl: 30
+    loop_wait: 10
+    retry_timeout: 10
+    maximum_lag_on_failover: 1048576
+    postgresql:
+      use_pg_rewind: true
+      use_slots: true
+      parameters:
+        wal_level: replica
+        hot_standby: "on"
+        max_wal_senders: 10
+        max_replication_slots: 10
+        wal_keep_size: 1GB
+        wal_log_hints: "on"
+        password_encryption: scram-sha-256
+      # In the cluster's shared configuration rather than each member's, so a
+      # control plane added later is let in by one change through Patroni's
+      # API (llmgw server add) instead of an edit on every member.
+      pg_hba:
+%[9]s  initdb:
+    - encoding: UTF8
+    - data-checksums
+
+postgresql:
+  listen: 127.0.0.1,%[3]s:5432
+  connect_address: %[3]s:5432
+  data_dir: /var/lib/postgresql/%[6]s/llmgw
+  bin_dir: /usr/lib/postgresql/%[6]s/bin
+  pgpass: /var/lib/postgresql/.pgpass_patroni
+  authentication:
+    superuser:
+      username: postgres
+      password: "%[7]s"
+    replication:
+      username: replicator
+      password: "%[8]s"
+    rewind:
+      username: postgres
+      password: "%[7]s"
+  parameters:
+    unix_socket_directories: /var/run/postgresql
+`, Scope, m.Name, m.IP, PatroniPort, strings.Join(hosts, ","), version, s.Superuser, s.Replicator, hbaLines)
+	b.WriteString(`
+tags:
+  nofailover: false
+  noloadbalance: false
+  clonefrom: false
+  nosync: false
+`)
+	return b.String()
+}
+
+// DSN is the store address the control planes are given: every member, and
+// the driver told to use whichever accepts writes, so a failover is followed
+// without anything in front of the database.
+func DSN(db []Member, password string) string {
+	var hosts []string
+	for _, d := range db {
+		hosts = append(hosts, d.IP+":5432")
+	}
+	return fmt.Sprintf("postgres://%s:%s@%s/%s?sslmode=disable&target_session_attrs=read-write",
+		Role, url.QueryEscape(password), strings.Join(hosts, ","), Database)
+}
+
+func ips(ms []Member) []string {
+	out := make([]string, len(ms))
+	for i, m := range ms {
+		out[i] = m.IP
+	}
+	return out
+}

internal/pgha/hba.go

@@ -0,0 +1,91 @@
+package pgha
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"fmt"
+	"net"
+	"net/http"
+	"slices"
+	"strconv"
+	"time"
+)
+
+// ClientLine is the access rule that lets a control plane at ip in.
+func ClientLine(ip string) string {
+	return fmt.Sprintf("host %s %s %s/32 scram-sha-256", Database, Role, ip)
+}
+
+// AllowClient lets a control plane at ip into the cluster's database: one
+// change to the cluster's shared configuration through Patroni's API, which
+// every member applies. It says whether anything changed.
+//
+// A cluster set up before the rules moved into the shared configuration
+// keeps them in each member's own file, which this cannot reach; that is
+// said, with what to add, rather than half done.
+func AllowClient(ctx context.Context, hosts []string, ip string) (bool, error) {
+	client := &http.Client{Timeout: 5 * time.Second}
+	var lastErr error
+	for _, h := range hosts {
+		base := "http://" + net.JoinHostPort(h, strconv.Itoa(restPort))
+		cfg, err := getConfig(ctx, client, base)
+		if err != nil {
+			lastErr = err
+			continue
+		}
+		var hba []string
+		if pg, ok := cfg["postgresql"].(map[string]any); ok {
+			if list, ok := pg["pg_hba"].([]any); ok {
+				for _, l := range list {
+					if s, ok := l.(string); ok {
+						hba = append(hba, s)
+					}
+				}
+			}
+		}
+		if len(hba) == 0 {
+			return false, fmt.Errorf("this cluster keeps its access rules in each member's "+
+				"/etc/patroni/config.yml, not in its shared configuration: add %q to pg_hba there "+
+				"on every member, and run `patronictl reload llmgw`", ClientLine(ip))
+		}
+		line := ClientLine(ip)
+		if slices.Contains(hba, line) {
+			return false, nil
+		}
+		body, _ := json.Marshal(map[string]any{"postgresql": map[string]any{"pg_hba": append(hba, line)}})
+		req, err := http.NewRequestWithContext(ctx, http.MethodPatch, base+"/config", bytes.NewReader(body))
+		if err != nil {
+			return false, err
+		}
+		req.Header.Set("Content-Type", "application/json")
+		resp, err := client.Do(req)
+		if err != nil {
+			lastErr = err
+			continue
+		}
+		resp.Body.Close()
+		if resp.StatusCode != http.StatusOK {
+			return false, fmt.Errorf("Patroni on %s refused the change: %s", h, resp.Status)
+		}
+		return true, nil
+	}
+	return false, fmt.Errorf("no member's Patroni answered: %w", lastErr)
+}
+
+func getConfig(ctx context.Context, client *http.Client, base string) (map[string]any, error) {
+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, base+"/config", nil)
+	if err != nil {
+		return nil, err
+	}
+	resp, err := client.Do(req)
+	if err != nil {
+		return nil, err
+	}
+	defer resp.Body.Close()
+	var cfg map[string]any
+	if err := json.NewDecoder(resp.Body).Decode(&cfg); err != nil {
+		return nil, fmt.Errorf("%s/config: %w", base, err)
+	}
+	return cfg, nil
+}

internal/pgha/metrics.go

@@ -0,0 +1,100 @@
+package pgha
+
+import (
+	"context"
+	"sync"
+	"time"
+
+	"github.com/prometheus/client_golang/prometheus"
+)
+
+// Collector publishes the store cluster's state as the control plane sees it
+// through Patroni's API: who leads, who streams, how far behind. Read at most
+// every fifteen seconds, however often it is scraped.
+type Collector struct {
+	hosts  []string
+	status func(ctx context.Context, hosts []string) (Cluster, error)
+
+	mu   sync.Mutex
+	at   time.Time
+	last Cluster
+	err  error
+
+	up, member, lag *prometheus.Desc
+}
+
+// NewCollector reads the cluster from the Patroni of each host in turn.
+func NewCollector(hosts []string) *Collector {
+	return &Collector{
+		hosts:  hosts,
+		status: HTTPStatus,
+		up: prometheus.NewDesc("llmgw_store_patroni_up",
+			"1 when a member's Patroni answered for the store's cluster.", nil, nil),
+		member: prometheus.NewDesc("llmgw_store_member",
+			"Each member of the store's cluster, by role (leader, replica) and state (running, streaming, …); always 1.",
+			[]string{"member", "host", "role", "state"}, nil),
+		lag: prometheus.NewDesc("llmgw_store_replica_lag_bytes",
+			"How far each replica is behind the leader; -1 when Patroni does not know.",
+			[]string{"member"}, nil),
+	}
+}
+
+// Describe implements prometheus.Collector.
+func (c *Collector) Describe(ch chan<- *prometheus.Desc) {
+	ch <- c.up
+	ch <- c.member
+	ch <- c.lag
+}
+
+// Collect implements prometheus.Collector.
+func (c *Collector) Collect(ch chan<- prometheus.Metric) {
+	cl, err := c.read()
+	if err != nil {
+		ch <- prometheus.MustNewConstMetric(c.up, prometheus.GaugeValue, 0)
+		return
+	}
+	ch <- prometheus.MustNewConstMetric(c.up, prometheus.GaugeValue, 1)
+	for _, m := range cl.Members {
+		ch <- prometheus.MustNewConstMetric(c.member, prometheus.GaugeValue, 1, m.Name, m.Host, m.Role, m.State)
+		if m.Role != "leader" {
+			ch <- prometheus.MustNewConstMetric(c.lag, prometheus.GaugeValue, float64(m.Lag), m.Name)
+		}
+	}
+}
+
+// Read is the cluster's state, at most fifteen seconds old.
+func (c *Collector) Read() (Cluster, error) { return c.read() }
+
+func (c *Collector) read() (Cluster, error) {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	if time.Since(c.at) < 15*time.Second {
+		return c.last, c.err
+	}
+	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+	defer cancel()
+	c.last, c.err = c.status(ctx, c.hosts)
+	if c.err == nil {
+		c.last = withMissing(c.last, c.hosts)
+	}
+	c.at = time.Now()
+	return c.last, c.err
+}
+
+// withMissing adds every host of the store address that the cluster does not
+// list, as a member in state "missing". A member that stops is dropped from
+// Patroni's list once its key expires — so without this a cluster down to
+// one machine reads as a healthy cluster of one, and nothing says a standby
+// is gone.
+func withMissing(c Cluster, hosts []string) Cluster {
+	seen := map[string]bool{}
+	for _, m := range c.Members {
+		seen[m.Host] = true
+	}
+	for _, h := range hosts {
+		if !seen[h] {
+			c.Members = append(c.Members, MemberState{Name: h, Host: h, Role: "replica", State: "missing", Lag: -1})
+		}
+	}
+	return c
+}

internal/pgha/pgha_test.go

@@ -0,0 +1,282 @@
+package pgha
+
+import (
+	"context"
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"net/url"
+	"strconv"
+	"strings"
+	"sync"
+	"testing"
+	"time"
+
+	"github.com/prometheus/client_golang/prometheus/testutil"
+	"github.com/prometheus/common/expfmt"
+	"go.yaml.in/yaml/v3"
+)
+
+var (
+	pg01  = Member{"llmgw-pg01", "10.0.0.149"}
+	ctl02 = Member{"llmgw-ctl02", "10.0.0.150"}
+	ctl   = Member{"llmgw-ctl", "10.0.0.140"}
+)
+
+func TestTheConfigurationsNameEveryMemberAndOnlyTheClientsGiven(t *testing.T) {
+	sec := Secrets{"su", "rep", "app"}
+	cfg := Patroni(pg01, []Member{pg01, ctl02}, []Member{pg01, ctl02, ctl}, []string{"10.0.0.140"}, "16", sec)
+	for _, want := range []string{
+		"name: llmgw-pg01", "listen: 10.0.0.149:8008",
+		"hosts: 10.0.0.149:2379,10.0.0.150:2379,10.0.0.140:2379",
+		"data_dir: /var/lib/postgresql/16/llmgw", `password: "rep"`,
+		"- host replication replicator 10.0.0.150/32 scram-sha-256",
+		"- host llmgw llmgw 10.0.0.140/32 scram-sha-256",
+		// A member can run a control plane too.
+		"- host llmgw llmgw 10.0.0.150/32 scram-sha-256",
+	} {
+		if !strings.Contains(cfg, want) {
+			t.Errorf("patroni config lacks %q", want)
+		}
+	}
+	if strings.Count(cfg, "host llmgw llmgw 10.0.0.149/32") != 1 {
+		t.Error("a member's address is allowed twice")
+	}
+	e := Etcd(ctl, []Member{pg01, ctl02, ctl})
+	if !strings.Contains(e, "ETCD_NAME=llmgw-ctl\n") ||
+		!strings.Contains(e, "llmgw-pg01=http://10.0.0.149:2380,llmgw-ctl02=http://10.0.0.150:2380,llmgw-ctl=http://10.0.0.140:2380") {
+		t.Errorf("etcd defaults:\n%s", e)
+	}
+	if got := DSN([]Member{pg01, ctl02}, "pw"); got !=
+		"postgres://llmgw:pw@10.0.0.149:5432,10.0.0.150:5432/llmgw?sslmode=disable&target_session_attrs=read-write" {
+		t.Errorf("dsn %s", got)
+	}
+}
+
+func TestAClusterIsCompleteWithALeaderAndEveryOtherMemberStreaming(t *testing.T) {
+	var c Cluster
+	if err := json.Unmarshal([]byte(`{"members":[
+		{"name":"llmgw-ctl02","role":"replica","state":"streaming","host":"10.0.0.150","lag":0},
+		{"name":"llmgw-pg01","role":"leader","state":"running","host":"10.0.0.149"}]}`), &c); err != nil {
+		t.Fatal(err)
+	}
+	if c.Leader() != "llmgw-pg01" || !c.Complete(2) || c.Complete(3) {
+		t.Errorf("%s: leader %q", c, c.Leader())
+	}
+	c.Members[0].State = "starting"
+	if c.Complete(2) {
+		t.Error("a replica still starting counted as streaming")
+	}
+	if lagOf(json.RawMessage(`"unknown"`)) != -1 || lagOf(json.RawMessage(`1024`)) != 1024 {
+		t.Error("lag not read")
+	}
+}
+
+// machine is a fake host: it records what it was asked to run and answers
+// from a table of script fragments.
+type machine struct {
+	name    string
+	mu      sync.Mutex
+	ran     []string
+	answers map[string]string
+}
+
+func (m *machine) Run(_ context.Context, script string, _ []byte) ([]byte, error) {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+	m.ran = append(m.ran, script)
+	for frag, out := range m.answers {
+		if strings.Contains(script, frag) {
+			return []byte(out), nil
+		}
+	}
+	return nil, nil
+}
+
+func (m *machine) ranAny(frag string) bool {
+	for _, s := range m.ran {
+		if strings.Contains(s, frag) {
+			return true
+		}
+	}
+	return false
+}
+
+const pws = "superuser=a\nreplicator=b\nllmgw=c\n"
+
+func healthy(n int) string {
+	return strings.Repeat("http://x:2379 is healthy: successfully committed proposal\n", n)
+}
+
+func TestARunningClusterIsLeftAsItIs(t *testing.T) {
+	// What init does when run again, or against the reference cluster: find
+	// it up, change nothing, and give the address.
+	a := &machine{name: "pg01", answers: map[string]string{"openssl rand": pws, "endpoint health": healthy(3)}}
+	b := &machine{name: "ctl02", answers: map[string]string{"sudo cat /root/llmgw-pg": pws}}
+	w := &machine{name: "ctl"}
+	up := Cluster{Members: []MemberState{{Name: "llmgw-pg01", Role: "leader", State: "running"},
+		{Name: "llmgw-ctl02", Role: "replica", State: "streaming"}}}
+	s := Setup{
+		DB:      []Node{{pg01, a}, {ctl02, b}},
+		Witness: &Node{ctl, w},
+		Status:  func(context.Context, []string) (Cluster, error) { return up, nil },
+	}
+	dsn, err := s.Run(context.Background())
+	if err != nil {
+		t.Fatal(err)
+	}
+	if !strings.Contains(dsn, "llmgw:c@10.0.0.149:5432,10.0.0.150:5432") {
+		t.Errorf("dsn %s", dsn)
+	}
+	for _, m := range []*machine{a, b, w} {
+		for _, frag := range []string{"apt-get", "systemctl restart", "/etc/default/etcd", "/etc/patroni/config.yml"} {
+			if m.ranAny(frag) {
+				t.Errorf("%s was changed (%q) though the cluster is up", m.name, frag)
+			}
+		}
+	}
+	if !a.ranAny("CREATE ROLE") {
+		t.Error("the role was not made sure of on the leader")
+	}
+}
+
+func TestTwoDatabaseMachinesWithoutAThirdVoteAreRefused(t *testing.T) {
+	s := Setup{DB: []Node{{pg01, &machine{}}, {ctl02, &machine{}}}}
+	if _, err := s.Run(context.Background()); err == nil || !strings.Contains(err.Error(), "third etcd vote") {
+		t.Errorf("got %v", err)
+	}
+}
+
+func TestAMachineHoldingOtherPasswordsIsNotOverwritten(t *testing.T) {
+	a := &machine{answers: map[string]string{"openssl rand": pws}}
+	b := &machine{answers: map[string]string{"sudo cat /root/llmgw-pg": "superuser=x\nreplicator=y\nllmgw=z\n"}}
+	s := Setup{DB: []Node{{pg01, a}, {ctl02, b}}, Witness: &Node{ctl, &machine{}}}
+	_, err := s.Run(context.Background())
+	if err == nil || !strings.Contains(err.Error(), "differ") {
+		t.Fatalf("got %v", err)
+	}
+	if b.ranAny("cat > /root/llmgw-pg") {
+		t.Error("the other cluster's passwords were replaced")
+	}
+}
+
+func TestAPostgreSQLWithDatabasesInItIsNotReplaced(t *testing.T) {
+	a := &machine{answers: map[string]string{"openssl rand": pws, "endpoint health": healthy(3),
+		"pg_dropcluster": "holds-data\n"}}
+	b := &machine{answers: map[string]string{"sudo cat /root/llmgw-pg": pws}}
+	s := Setup{DB: []Node{{pg01, a}, {ctl02, b}}, Witness: &Node{ctl, &machine{}},
+		Status: func(context.Context, []string) (Cluster, error) { return Cluster{}, nil },
+		Wait:   time.Second}
+	_, err := s.Run(context.Background())
+	if err == nil || !strings.Contains(err.Error(), "databases in it") {
+		t.Fatalf("got %v", err)
+	}
+	if a.ranAny("systemctl restart patroni") {
+		t.Error("patroni was started over somebody's data")
+	}
+}
+
+func TestTheCollectorSaysWhoLeadsAndHowFarBehindTheRestAre(t *testing.T) {
+	c := NewCollector([]string{"10.0.0.149"})
+	c.status = func(context.Context, []string) (Cluster, error) {
+		return Cluster{Members: []MemberState{{Name: "llmgw-pg01", Host: "10.0.0.149", Role: "leader", State: "running"},
+			{Name: "llmgw-ctl02", Host: "10.0.0.150", Role: "replica", State: "streaming", Lag: 2048}}}, nil
+	}
+	got, err := testutil.CollectAndFormat(c, expfmt.TypeTextPlain, "llmgw_store_patroni_up", "llmgw_store_member", "llmgw_store_replica_lag_bytes")
+	if err != nil {
+		t.Fatal(err)
+	}
+	for _, want := range []string{
+		"llmgw_store_patroni_up 1",
+		`llmgw_store_member{host="10.0.0.149",member="llmgw-pg01",role="leader",state="running"} 1`,
+		`llmgw_store_replica_lag_bytes{member="llmgw-ctl02"} 2048`,
+	} {
+		if !strings.Contains(string(got), want) {
+			t.Errorf("missing %s in\n%s", want, got)
+		}
+	}
+	if strings.Contains(string(got), `lag_bytes{member="llmgw-pg01"}`) {
+		t.Error("the leader was given a lag")
+	}
+}
+
+func TestAMemberThatStoppedIsReportedMissingNotForgotten(t *testing.T) {
+	c := NewCollector([]string{"10.0.0.153", "10.0.0.154"})
+	c.status = func(context.Context, []string) (Cluster, error) {
+		return Cluster{Members: []MemberState{{Name: "hatest-pg2", Host: "10.0.0.154", Role: "leader", State: "running"}}}, nil
+	}
+	cl, err := c.Read()
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(cl.Members) != 2 || cl.Members[1].Host != "10.0.0.153" || cl.Members[1].State != "missing" {
+		t.Errorf("%s", cl)
+	}
+}
+
+func TestTheAccessRulesAreInTheClustersSharedConfiguration(t *testing.T) {
+	// Where `server add` can change them for every member at once.
+	cfg := Patroni(pg01, []Member{pg01, ctl02}, []Member{pg01, ctl02, ctl}, []string{"10.0.0.140"}, "16", Secrets{"a", "b", "c"})
+	var doc struct {
+		Bootstrap struct {
+			DCS struct {
+				PostgreSQL struct {
+					PgHBA []string `yaml:"pg_hba"`
+				} `yaml:"postgresql"`
+			} `yaml:"dcs"`
+		} `yaml:"bootstrap"`
+		PostgreSQL map[string]any `yaml:"postgresql"`
+	}
+	if err := yaml.Unmarshal([]byte(cfg), &doc); err != nil {
+		t.Fatalf("not YAML: %v\n%s", err, cfg)
+	}
+	hba := doc.Bootstrap.DCS.PostgreSQL.PgHBA
+	if len(hba) < 5 || hba[0] != "local all all peer" {
+		t.Errorf("shared pg_hba: %q", hba)
+	}
+	if _, local := doc.PostgreSQL["pg_hba"]; local {
+		t.Error("pg_hba is also in the member's own configuration, where it would win over the shared one")
+	}
+}
+
+func TestAControlPlaneIsLetInThroughTheSharedConfiguration(t *testing.T) {
+	var patched []string
+	hba := []any{"local all all peer", ClientLine("10.0.0.140")}
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		switch r.Method {
+		case http.MethodGet:
+			_ = json.NewEncoder(w).Encode(map[string]any{"postgresql": map[string]any{"pg_hba": hba}})
+		case http.MethodPatch:
+			var body struct {
+				PostgreSQL struct {
+					PgHBA []string `json:"pg_hba"`
+				} `json:"postgresql"`
+			}
+			_ = json.NewDecoder(r.Body).Decode(&body)
+			patched = body.PostgreSQL.PgHBA
+		}
+	}))
+	defer srv.Close()
+	u, _ := url.Parse(srv.URL)
+	old := restPort
+	restPort, _ = strconv.Atoi(u.Port())
+	defer func() { restPort = old }()
+
+	changed, err := AllowClient(context.Background(), []string{"127.0.0.1"}, "10.0.0.150")
+	if err != nil || !changed {
+		t.Fatalf("%v %v", changed, err)
+	}
+	// Everything that was there is kept, and the new line added once.
+	if len(patched) != 3 || patched[2] != ClientLine("10.0.0.150") || patched[1] != ClientLine("10.0.0.140") {
+		t.Errorf("patched %q", patched)
+	}
+	if changed, err := AllowClient(context.Background(), []string{"127.0.0.1"}, "10.0.0.140"); err != nil || changed {
+		t.Errorf("one already let in was changed again: %v %v", changed, err)
+	}
+
+	hba = nil // the rules are in each member's file
+	if _, err := AllowClient(context.Background(), []string{"127.0.0.1"}, "10.0.0.151"); err == nil ||
+		!strings.Contains(err.Error(), "each member") {
+		t.Errorf("rules kept per member: %v", err)
+	}
+}

internal/pgha/setup.go

@@ -0,0 +1,417 @@
+package pgha
+
+import (
+	"bytes"
+	"context"
+	"fmt"
+	"os/exec"
+	"strings"
+	"sync"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
+)
+
+// Node is a machine the setup reaches, and how.
+type Node struct {
+	Member
+	Remote onboard.Remote
+}
+
+// Setup builds, or finds, a Patroni cluster.
+type Setup struct {
+	// DB are the database machines; the first is where the cluster starts.
+	DB []Node
+	// Witness is the third etcd vote when there are only two database
+	// machines: the control plane, run locally. Ignored with three or more.
+	Witness *Node
+	// Clients are the addresses control planes connect from.
+	Clients []string
+	// Status reads the cluster from Patroni's API; nil asks over HTTP.
+	Status func(ctx context.Context, hosts []string) (Cluster, error)
+	// Emit reports progress: a step, what happened, and whether anything
+	// was changed or found as it should be.
+	Emit func(step, detail string, changed bool)
+	// Wait bounds each wait for the cluster to come up.
+	Wait time.Duration
+}
+
+// Local is a Remote for the machine this runs on, which init runs as root.
+type Local struct{}
+
+// Run runs script with sh here.
+func (Local) Run(ctx context.Context, script string, stdin []byte) ([]byte, error) {
+	cmd := exec.CommandContext(ctx, "sh", "-c", script)
+	if stdin != nil {
+		cmd.Stdin = bytes.NewReader(stdin)
+	}
+	var out, errb bytes.Buffer
+	cmd.Stdout, cmd.Stderr = &out, &errb
+	if err := cmd.Run(); err != nil {
+		return append(out.Bytes(), errb.Bytes()...), err
+	}
+	return out.Bytes(), nil
+}
+
+func (s *Setup) emit(step string, changed bool, format string, a ...any) {
+	if s.Emit != nil {
+		s.Emit(step, fmt.Sprintf(format, a...), changed)
+	}
+}
+
+// etcdMembers are the machines that vote: the database machines, and the
+// witness when they are only two.
+func (s *Setup) etcdMembers() []Node {
+	out := append([]Node{}, s.DB...)
+	if len(s.DB) == 2 && s.Witness != nil {
+		out = append(out, *s.Witness)
+	}
+	return out
+}
+
+// Run sets the cluster up, or confirms it is, and returns the store address.
+func (s *Setup) Run(ctx context.Context) (string, error) {
+	if len(s.DB) < 2 {
+		return "", fmt.Errorf("a replicated store needs at least two database machines, not %d", len(s.DB))
+	}
+	if len(s.DB) == 2 && s.Witness == nil {
+		return "", fmt.Errorf("two database machines need a third etcd vote, or losing either stops failover")
+	}
+	if s.Wait == 0 {
+		s.Wait = 3 * time.Minute
+	}
+	if s.Status == nil {
+		s.Status = HTTPStatus
+	}
+	sec, err := s.secrets(ctx)
+	if err != nil {
+		return "", fmt.Errorf("passwords: %w", err)
+	}
+	if err := s.etcd(ctx); err != nil {
+		return "", fmt.Errorf("etcd: %w", err)
+	}
+	if err := s.patroni(ctx, sec); err != nil {
+		return "", fmt.Errorf("patroni: %w", err)
+	}
+	if err := s.database(ctx); err != nil {
+		return "", fmt.Errorf("database: %w", err)
+	}
+	members := make([]Member, len(s.DB))
+	for i, d := range s.DB {
+		members[i] = d.Member
+	}
+	return DSN(members, sec.App), nil
+}
+
+// secretDir holds the passwords on every database machine, readable by root
+// alone. Generated on the first, carried to the others.
+const secretDir = "/root/llmgw-pg"
+
+var secretNames = []string{"superuser", "replicator", "llmgw"}
+
+func (s *Setup) secrets(ctx context.Context) (Secrets, error) {
+	script := `set -e
+sudo install -d -m 700 ` + secretDir + `
+for f in superuser replicator llmgw; do
+  sudo test -s ` + secretDir + `/$f.pw || sudo sh -c "umask 077; openssl rand -hex 24 > ` + secretDir + `/$f.pw"
+done
+for f in superuser replicator llmgw; do printf '%s=' $f; sudo cat ` + secretDir + `/$f.pw; done`
+	out, err := s.DB[0].Remote.Run(ctx, script, nil)
+	if err != nil {
+		return Secrets{}, fmt.Errorf("on %s: %v: %s", s.DB[0].Name, err, strings.TrimSpace(string(out)))
+	}
+	got := parseSecrets(out)
+	if got.Superuser == "" || got.Replicator == "" || got.App == "" {
+		return Secrets{}, fmt.Errorf("could not read them back from %s", s.DB[0].Name)
+	}
+	for _, d := range s.DB[1:] {
+		have, err := d.Remote.Run(ctx, `for f in superuser replicator llmgw; do
+  if sudo test -s `+secretDir+`/$f.pw; then printf '%s=' $f; sudo cat `+secretDir+`/$f.pw; fi
+done`, nil)
+		if err != nil {
+			return Secrets{}, fmt.Errorf("on %s: %v", d.Name, err)
+		}
+		if h := parseSecrets(have); h != (Secrets{}) {
+			if h != got {
+				// Not overwritten: a member holding other passwords belongs
+				// to another cluster, or to an earlier one somebody meant to
+				// keep, and replacing them would lock it out of itself.
+				return Secrets{}, fmt.Errorf("%s holds passwords in %s that differ from %s's; "+
+					"remove them there if that machine is meant to join this cluster", d.Name, secretDir, s.DB[0].Name)
+			}
+			continue
+		}
+		body := fmt.Sprintf("%s\n%s\n%s\n", got.Superuser, got.Replicator, got.App)
+		if out, err := d.Remote.Run(ctx, `set -e
+sudo install -d -m 700 `+secretDir+`
+read su; read rep; read app
+for kv in "superuser $su" "replicator $rep" "llmgw $app"; do
+  set -- $kv
+  printf '%s\n' "$2" | sudo sh -c "umask 077; cat > `+secretDir+`/$1.pw"
+done`, []byte(body)); err != nil {
+			return Secrets{}, fmt.Errorf("on %s: %v: %s", d.Name, err, strings.TrimSpace(string(out)))
+		}
+		s.emit("passwords", true, "carried to %s", d.Name)
+	}
+	s.emit("passwords", false, "kept in %s on every database machine", secretDir)
+	return got, nil
+}
+
+func parseSecrets(out []byte) Secrets {
+	var s Secrets
+	for _, line := range strings.Split(string(out), "\n") {
+		k, v, ok := strings.Cut(strings.TrimSpace(line), "=")
+		if !ok {
+			continue
+		}
+		switch k {
+		case "superuser":
+			s.Superuser = v
+		case "replicator":
+			s.Replicator = v
+		case "llmgw":
+			s.App = v
+		}
+	}
+	return s
+}
+
+// etcd brings up the voting members, unless every one of them is already
+// healthy in one cluster — in which case nothing is touched, since rewriting
+// a member's configuration means emptying its data.
+func (s *Setup) etcd(ctx context.Context) error {
+	nodes := s.etcdMembers()
+	var members []Member
+	var endpoints []string
+	for _, n := range nodes {
+		members = append(members, n.Member)
+		endpoints = append(endpoints, "http://"+n.IP+":2379")
+	}
+	if s.etcdHealthy(ctx, nodes[0], endpoints, len(nodes)) {
+		s.emit("etcd", false, "%d members healthy; left as they are", len(nodes))
+		return nil
+	}
+
+	changed := make([]bool, len(nodes))
+	for i, n := range nodes {
+		out, err := n.Remote.Run(ctx, `set -e
+if ! command -v etcd >/dev/null 2>&1; then
+  sudo DEBIAN_FRONTEND=noninteractive apt-get -qq update >/dev/null
+  sudo DEBIAN_FRONTEND=noninteractive apt-get -qq install -y etcd-server etcd-client >/dev/null
+fi
+new=$(cat)
+if [ "$(sudo cat /etc/default/etcd 2>/dev/null)" != "$new" ]; then
+  sudo systemctl stop etcd >/dev/null 2>&1 || true
+  sudo rm -rf /var/lib/etcd/default
+  printf '%s\n' "$new" | sudo tee /etc/default/etcd >/dev/null
+  echo changed
+fi
+sudo systemctl enable etcd >/dev/null 2>&1`, []byte(Etcd(n.Member, members)))
+		if err != nil {
+			return fmt.Errorf("on %s: %v: %s", n.Name, err, strings.TrimSpace(string(out)))
+		}
+		changed[i] = strings.Contains(string(out), "changed")
+	}
+	// Started together: a new member waits for the others before it says it
+	// is up, so starting them one after another would wait for ever on the
+	// first.
+	var wg sync.WaitGroup
+	errs := make([]error, len(nodes))
+	for i, n := range nodes {
+		wg.Add(1)
+		go func(i int, n Node) {
+			defer wg.Done()
+			verb := "start"
+			if changed[i] {
+				verb = "restart"
+			}
+			if out, err := n.Remote.Run(ctx, "sudo systemctl "+verb+" --no-block etcd", nil); err != nil {
+				errs[i] = fmt.Errorf("on %s: %v: %s", n.Name, err, strings.TrimSpace(string(out)))
+			}
+		}(i, n)
+	}
+	wg.Wait()
+	for _, err := range errs {
+		if err != nil {
+			return err
+		}
+	}
+	deadline := time.Now().Add(s.Wait)
+	for !s.etcdHealthy(ctx, nodes[0], endpoints, len(nodes)) {
+		if time.Now().After(deadline) {
+			return fmt.Errorf("the %d members did not become healthy within %s; `journalctl -u etcd` on each says why", len(nodes), s.Wait)
+		}
+		if err := sleep(ctx, 2*time.Second); err != nil {
+			return err
+		}
+	}
+	s.emit("etcd", true, "%d members up", len(nodes))
+	return nil
+}
+
+func (s *Setup) etcdHealthy(ctx context.Context, n Node, endpoints []string, want int) bool {
+	out, err := n.Remote.Run(ctx, "ETCDCTL_API=3 etcdctl --dial-timeout=3s --endpoints="+
+		strings.Join(endpoints, ",")+" endpoint health 2>&1", nil)
+	return err == nil && strings.Count(string(out), "is healthy") == want
+}
+
+// patroni installs PostgreSQL and Patroni on every database machine, unless
+// the cluster already has a leader and every member streaming.
+func (s *Setup) patroni(ctx context.Context, sec Secrets) error {
+	hosts := make([]string, len(s.DB))
+	for i, d := range s.DB {
+		hosts[i] = d.IP
+	}
+	if c, err := s.Status(ctx, hosts); err == nil && c.Complete(len(s.DB)) {
+		s.emit("patroni", false, "%s leads, %d streaming; left as it is", c.Leader(), len(s.DB)-1)
+		return nil
+	}
+
+	var version string
+	for _, d := range s.DB {
+		v, err := s.preparePostgres(ctx, d)
+		if err != nil {
+			return err
+		}
+		if version != "" && v != version {
+			// Streaming replication is between one major version.
+			return fmt.Errorf("%s has PostgreSQL %s and %s has %s; they must match", s.DB[0].Name, version, d.Name, v)
+		}
+		version = v
+	}
+
+	var etcd, db []Member
+	for _, n := range s.etcdMembers() {
+		etcd = append(etcd, n.Member)
+	}
+	for _, d := range s.DB {
+		db = append(db, d.Member)
+	}
+	for i, d := range s.DB {
+		cfg := Patroni(d.Member, db, etcd, s.Clients, version, sec)
+		if out, err := d.Remote.Run(ctx, `set -e
+sudo install -d -m 755 /etc/patroni
+sudo install -d -o postgres -g postgres -m 700 /var/lib/postgresql/`+version+`/llmgw
+sudo sh -c 'umask 027; cat > /etc/patroni/config.yml.new'
+sudo chown root:postgres /etc/patroni/config.yml.new
+sudo chmod 640 /etc/patroni/config.yml.new
+sudo mv -f /etc/patroni/config.yml.new /etc/patroni/config.yml
+sudo systemctl enable patroni >/dev/null 2>&1
+sudo systemctl restart patroni`, []byte(cfg)); err != nil {
+			return fmt.Errorf("on %s: %v: %s", d.Name, err, strings.TrimSpace(string(out)))
+		}
+		// The first starts the cluster; the rest clone it, which they can
+		// only do once it has a leader.
+		want := func(c Cluster) bool { return c.Leader() != "" }
+		if i > 0 {
+			want = func(c Cluster) bool { return c.Complete(i + 1) }
+		}
+		if err := s.waitFor(ctx, hosts, want); err != nil {
+			return fmt.Errorf("after starting %s: %w", d.Name, err)
+		}
+		s.emit("patroni", true, "%s up", d.Name)
+	}
+	return nil
+}
+
+// preparePostgres installs the packages and clears the distribution's own
+// cluster, which would hold the port Patroni's needs — unless it holds a
+// database somebody made, which is not this setup's to delete.
+func (s *Setup) preparePostgres(ctx context.Context, d Node) (string, error) {
+	out, err := d.Remote.Run(ctx, `set -e
+need=""
+command -v patroni >/dev/null 2>&1 || need="$need patroni"
+ls /usr/lib/postgresql/*/bin/postgres >/dev/null 2>&1 || need="$need postgresql"
+if [ -n "$need" ]; then
+  sudo DEBIAN_FRONTEND=noninteractive apt-get -qq update >/dev/null
+  sudo DEBIAN_FRONTEND=noninteractive apt-get -qq install -y $need >/dev/null
+fi
+v=$(ls /usr/lib/postgresql | sort -n | tail -1)
+if [ -d /var/lib/postgresql/$v/main ]; then
+  if sudo systemctl is-active --quiet postgresql@$v-main; then
+    n=$(sudo -u postgres psql -Atc "select count(*) from pg_database where datname not in ('postgres','template0','template1')")
+    if [ "$n" != "0" ]; then echo "holds-data"; exit 0; fi
+  fi
+  sudo systemctl disable --now postgresql@$v-main postgresql >/dev/null 2>&1 || true
+  sudo pg_dropcluster --stop $v main >/dev/null 2>&1 || sudo rm -rf /var/lib/postgresql/$v/main
+fi
+sudo systemctl disable --now postgresql >/dev/null 2>&1 || true
+echo "version=$v"`, nil)
+	if err != nil {
+		return "", fmt.Errorf("on %s: %v: %s", d.Name, err, strings.TrimSpace(string(out)))
+	}
+	if strings.Contains(string(out), "holds-data") {
+		return "", fmt.Errorf("%s runs a PostgreSQL with databases in it; move or drop them first — "+
+			"this setup replaces that cluster", d.Name)
+	}
+	for _, line := range strings.Split(string(out), "\n") {
+		if v, ok := strings.CutPrefix(strings.TrimSpace(line), "version="); ok && v != "" {
+			return v, nil
+		}
+	}
+	return "", fmt.Errorf("no PostgreSQL found on %s after installing it", d.Name)
+}
+
+// database makes the role and database on whichever member leads.
+func (s *Setup) database(ctx context.Context) error {
+	hosts := make([]string, len(s.DB))
+	for i, d := range s.DB {
+		hosts[i] = d.IP
+	}
+	c, err := s.Status(ctx, hosts)
+	if err != nil {
+		return err
+	}
+	var leader *Node
+	for i := range s.DB {
+		if s.DB[i].Name == c.Leader() {
+			leader = &s.DB[i]
+		}
+	}
+	if leader == nil {
+		return fmt.Errorf("no member leads")
+	}
+	out, err := leader.Remote.Run(ctx, `set -e
+pw=$(sudo cat `+secretDir+`/llmgw.pw)
+sudo -u postgres psql -q -v ON_ERROR_STOP=1 -v pw="$pw" <<'SQL'
+SELECT format('CREATE ROLE %I LOGIN', 'llmgw') WHERE NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'llmgw') \gexec
+ALTER ROLE llmgw PASSWORD :'pw';
+SELECT format('CREATE DATABASE %I OWNER %I', 'llmgw', 'llmgw') WHERE NOT EXISTS (SELECT 1 FROM pg_database WHERE datname = 'llmgw') \gexec
+SQL`, nil)
+	if err != nil {
+		return fmt.Errorf("on %s: %v: %s", leader.Name, err, strings.TrimSpace(string(out)))
+	}
+	s.emit("database", true, "role and database %s on %s", Database, leader.Name)
+	return nil
+}
+
+func (s *Setup) waitFor(ctx context.Context, hosts []string, ok func(Cluster) bool) error {
+	deadline := time.Now().Add(s.Wait)
+	var last error
+	for {
+		c, err := s.Status(ctx, hosts)
+		if err == nil && ok(c) {
+			return nil
+		}
+		if err != nil {
+			last = err
+		} else {
+			last = fmt.Errorf("the cluster is %s", c)
+		}
+		if time.Now().After(deadline) {
+			return fmt.Errorf("not up within %s: %v; `journalctl -u patroni` says why", s.Wait, last)
+		}
+		if err := sleep(ctx, 3*time.Second); err != nil {
+			return err
+		}
+	}
+}
+
+func sleep(ctx context.Context, d time.Duration) error {
+	select {
+	case <-ctx.Done():
+		return ctx.Err()
+	case <-time.After(d):
+		return nil
+	}
+}

internal/pgha/status.go

@@ -0,0 +1,114 @@
+package pgha
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"net"
+	"net/http"
+	"strconv"
+	"strings"
+	"time"
+)
+
+// MemberState is one member as Patroni reports it.
+type MemberState struct {
+	Name string `json:"name"`
+	// Role is "leader", "replica" or "sync_standby".
+	Role string `json:"role"`
+	// State is "running" for a leader, "streaming" for a replica keeping up,
+	// and anything else for one that is not.
+	State    string `json:"state"`
+	Host     string `json:"host"`
+	Timeline int    `json:"timeline"`
+	// Lag is how far behind the leader a replica is, in bytes; -1 when
+	// Patroni does not know, which is itself worth an alert.
+	Lag    int64           `json:"-"`
+	RawLag json.RawMessage `json:"lag"`
+}
+
+// Cluster is the cluster's state, as any member's API reports it.
+type Cluster struct {
+	Members []MemberState `json:"members"`
+}
+
+// Leader is the leading member's name, or "".
+func (c Cluster) Leader() string {
+	for _, m := range c.Members {
+		if m.Role == "leader" || m.Role == "standby_leader" {
+			return m.Name
+		}
+	}
+	return ""
+}
+
+// Complete reports whether n members are up: one leading, the rest streaming.
+func (c Cluster) Complete(n int) bool {
+	if c.Leader() == "" || len(c.Members) < n {
+		return false
+	}
+	streaming := 0
+	for _, m := range c.Members {
+		if m.Role != "leader" && m.State == "streaming" {
+			streaming++
+		}
+	}
+	return streaming >= n-1
+}
+
+func (c Cluster) String() string {
+	if len(c.Members) == 0 {
+		return "empty"
+	}
+	var parts []string
+	for _, m := range c.Members {
+		parts = append(parts, m.Name+" "+m.Role+"/"+m.State)
+	}
+	return strings.Join(parts, ", ")
+}
+
+// HTTPStatus asks each host's Patroni in turn and returns the first answer:
+// any member describes the whole cluster.
+func HTTPStatus(ctx context.Context, hosts []string) (Cluster, error) {
+	client := &http.Client{Timeout: 3 * time.Second}
+	var last error
+	for _, h := range hosts {
+		u := "http://" + net.JoinHostPort(h, strconv.Itoa(restPort)) + "/cluster"
+		req, err := http.NewRequestWithContext(ctx, http.MethodGet, u, nil)
+		if err != nil {
+			return Cluster{}, err
+		}
+		resp, err := client.Do(req)
+		if err != nil {
+			last = err
+			continue
+		}
+		var c Cluster
+		err = json.NewDecoder(resp.Body).Decode(&c)
+		resp.Body.Close()
+		if err != nil {
+			last = fmt.Errorf("%s: %w", u, err)
+			continue
+		}
+		for i := range c.Members {
+			c.Members[i].Lag = lagOf(c.Members[i].RawLag)
+		}
+		return c, nil
+	}
+	if last == nil {
+		last = fmt.Errorf("no hosts to ask")
+	}
+	return Cluster{}, fmt.Errorf("no member's Patroni answered: %w", last)
+}
+
+// lagOf reads Patroni's lag: a number of bytes, or "unknown".
+func lagOf(raw json.RawMessage) int64 {
+	if len(raw) == 0 {
+		return 0
+	}
+	var n int64
+	if json.Unmarshal(raw, &n) == nil {
+		return n
+	}
+	return -1
+}

internal/preflight/preflight.go

@@ -0,0 +1,360 @@
+// Package preflight checks machines before anything is installed on them:
+// that they can be logged into and administered, run a supported system, have
+// room for their role, keep time, have the ports free, and can reach the
+// internet and each other. Everything wrong is found at once, with what to
+// do about it, rather than one failure at a time halfway through an install.
+package preflight
+
+import (
+	"context"
+	"fmt"
+	"sort"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
+)
+
+// Role is what a machine is going to be.
+type Role string
+
+// The roles a setup gives machines.
+const (
+	RoleControl  Role = "control plane"
+	RoleDatabase Role = "database"
+	RoleWitness  Role = "etcd witness"
+	RoleNode     Role = "node"
+)
+
+// Need is the least a role asks of a machine.
+type Need struct {
+	CPUs     int
+	MemoryMB int
+	DiskGB   int
+	Ports    []int
+}
+
+// Needs are what each role asks for; a machine with several roles needs the
+// largest of each and the ports of all.
+var Needs = map[Role]Need{
+	RoleControl:  {CPUs: 2, MemoryMB: 3500, DiskGB: 15, Ports: []int{7443, 8080}},
+	RoleDatabase: {CPUs: 2, MemoryMB: 3500, DiskGB: 30, Ports: []int{5432, 8008, 2379, 2380}},
+	RoleWitness:  {CPUs: 1, MemoryMB: 900, DiskGB: 5, Ports: []int{2379, 2380}},
+	RoleNode:     {CPUs: 2, MemoryMB: 3500, DiskGB: 20},
+}
+
+// Machine is one machine to check.
+type Machine struct {
+	Name   string // how the screen calls it
+	Addr   string // the address the others will use
+	Roles  []Role
+	Remote onboard.Remote
+	// Local is this machine, where sudo is not needed (setup runs as root).
+	Local bool
+}
+
+// Level is how bad a finding is.
+type Level int
+
+// Levels, mildest first.
+const (
+	OK Level = iota
+	Warn
+	Fail
+)
+
+func (l Level) String() string { return [...]string{"ok", "warn", "fail"}[l] }
+
+// Finding is one thing checked on one machine.
+type Finding struct {
+	Machine string
+	Check   string
+	Level   Level
+	Detail  string
+	// Fix says what to do, for anything that is not OK.
+	Fix string
+}
+
+// Report is everything found.
+type Report struct {
+	Findings []Finding
+}
+
+// Worst is the most serious level found.
+func (r Report) Worst() Level {
+	w := OK
+	for _, f := range r.Findings {
+		if f.Level > w {
+			w = f.Level
+		}
+	}
+	return w
+}
+
+// For is one machine's findings, in check order.
+func (r Report) For(machine string) []Finding {
+	var out []Finding
+	for _, f := range r.Findings {
+		if f.Machine == machine {
+			out = append(out, f)
+		}
+	}
+	return out
+}
+
+// probeScript gathers everything in one round trip, as key=value lines, so
+// the checks are decided here and can be tested without a machine.
+const probeScript = `
+. /etc/os-release 2>/dev/null; echo "os=$ID"; echo "version=$VERSION_ID"; echo "pretty=$PRETTY_NAME"
+echo "arch=$(uname -m)"
+echo "cpus=$(nproc)"
+echo "mem_mb=$(awk '/MemTotal/{print int($2/1024)}' /proc/meminfo)"
+echo "disk_gb=$(df -BG --output=avail / | tail -1 | tr -dc 0-9)"
+echo "now=$(date +%s.%N)"
+echo "ntp=$(timedatectl show -p NTPSynchronized --value 2>/dev/null)"
+echo "listening=$(ss -Hltn 2>/dev/null | awk '{n=split($4,a,":"); print a[n]}' | sort -un | tr '\n' ' ')"
+if sudo -n true 2>/dev/null; then echo "sudo=yes"; else echo "sudo=no"; fi
+echo "apt=$(command -v apt-get >/dev/null && echo yes || echo no)"
+echo "internet=$(timeout 6 bash -c 'exec 3<>/dev/tcp/deb.debian.org/80' 2>/dev/null && echo yes || (command -v curl >/dev/null && curl -s -m 6 -o /dev/null http://deb.debian.org && echo yes) || echo no)"
+`
+
+// reachScript tries one TCP port on each address: whether the machines can
+// reach one another at all. Port 22 because it is open everywhere before
+// anything is installed; the ports llmgw will use have nothing listening yet.
+func reachScript(addrs []string) string {
+	var b strings.Builder
+	for _, a := range addrs {
+		fmt.Fprintf(&b, "if timeout 4 bash -c 'exec 3<>/dev/tcp/%s/22' 2>/dev/null; then echo \"reach %s yes\"; else echo \"reach %s no\"; fi\n", a, a, a)
+	}
+	return b.String()
+}
+
+// Run checks every machine, concurrently.
+func Run(ctx context.Context, machines []Machine) Report {
+	var (
+		mu  sync.Mutex
+		rep Report
+		wg  sync.WaitGroup
+	)
+	add := func(fs ...Finding) {
+		mu.Lock()
+		rep.Findings = append(rep.Findings, fs...)
+		mu.Unlock()
+	}
+	var addrs []string
+	for _, m := range machines {
+		addrs = append(addrs, m.Addr)
+	}
+	order := map[string]int{}
+	for i, m := range machines {
+		order[m.Name] = i
+		wg.Add(1)
+		go func(m Machine) {
+			defer wg.Done()
+			add(check(ctx, m, addrs)...)
+		}(m)
+	}
+	wg.Wait()
+	// Machines in the order given, checks in the order made.
+	sort.SliceStable(rep.Findings, func(i, j int) bool {
+		return order[rep.Findings[i].Machine] < order[rep.Findings[j].Machine]
+	})
+	return rep
+}
+
+func check(ctx context.Context, m Machine, all []string) []Finding {
+	sent := time.Now()
+	out, err := m.Remote.Run(ctx, probeScript, nil)
+	took := time.Since(sent)
+	if err != nil {
+		return []Finding{{Machine: m.Name, Check: "login", Level: Fail,
+			Detail: strings.TrimSpace(firstLine(err.Error() + " " + string(out))),
+			Fix:    "check the address, the user and the key or password given for it"}}
+	}
+	facts := parse(out)
+	fs := Evaluate(m, facts, sent.Add(took/2))
+	var others []string
+	for _, a := range all {
+		if a != m.Addr {
+			others = append(others, a)
+		}
+	}
+	if len(others) > 0 {
+		rout, _ := m.Remote.Run(ctx, reachScript(others), nil)
+		fs = append(fs, evaluateReach(m, string(rout))...)
+	}
+	return fs
+}
+
+func firstLine(s string) string {
+	s = strings.TrimSpace(s)
+	if i := strings.IndexByte(s, '\n'); i >= 0 {
+		return s[:i]
+	}
+	return s
+}
+
+func parse(out []byte) map[string]string {
+	facts := map[string]string{}
+	for _, line := range strings.Split(string(out), "\n") {
+		if k, v, ok := strings.Cut(strings.TrimSpace(line), "="); ok {
+			facts[k] = strings.TrimSpace(v)
+		}
+	}
+	return facts
+}
+
+// supported are the systems the setup installs packages on with apt.
+var supported = map[string][]string{
+	"ubuntu": {"22.04", "24.04"},
+	"debian": {"12"},
+}
+
+// Evaluate turns one machine's facts into findings. at is when the machine's
+// clock was read, by this machine's clock.
+func Evaluate(m Machine, facts map[string]string, at time.Time) []Finding {
+	var fs []Finding
+	f := func(check string, l Level, detail, fix string) {
+		fs = append(fs, Finding{Machine: m.Name, Check: check, Level: l, Detail: detail, Fix: fix})
+	}
+	f("login", OK, "logged in", "")
+
+	// The system.
+	os, ver := facts["os"], facts["version"]
+	switch vers, ok := supported[os]; {
+	case !ok:
+		f("system", Fail, facts["pretty"], "use Ubuntu 22.04 or 24.04, or Debian 12")
+	case !contains(vers, ver):
+		f("system", Warn, facts["pretty"]+" (tried on "+strings.Join(vers, ", ")+")", "it may work; Ubuntu 24.04 is what this is tested on")
+	default:
+		f("system", OK, facts["pretty"], "")
+	}
+	switch facts["arch"] {
+	case "x86_64", "aarch64":
+		f("architecture", OK, facts["arch"], "")
+	default:
+		f("architecture", Fail, facts["arch"], "llmgw is built for amd64 and arm64")
+	}
+
+	// Administering it.
+	if !m.Local {
+		if facts["sudo"] == "yes" {
+			f("sudo", OK, "without a password", "")
+		} else {
+			// Not a failure: the setup asks for the password and grants
+			// itself sudo for the length of the install.
+			f("sudo", Warn, "asks for a password", "give the sudo password on the access screen")
+		}
+	}
+	if facts["apt"] != "yes" {
+		f("packages", Fail, "no apt-get", "the setup installs PostgreSQL, Patroni and etcd with apt")
+	}
+
+	// Room for its roles.
+	need := needOf(m.Roles)
+	cpus, _ := strconv.Atoi(facts["cpus"])
+	mem, _ := strconv.Atoi(facts["mem_mb"])
+	disk, _ := strconv.Atoi(facts["disk_gb"])
+	room := func(check string, have, want int, unit string) {
+		d := fmt.Sprintf("%d %s (needs %d)", have, unit, want)
+		switch {
+		case have >= want:
+			f(check, OK, d, "")
+		case have*2 >= want:
+			f(check, Warn, d, "it will run, short of what the role is sized for")
+		default:
+			f(check, Fail, d, "give the machine more")
+		}
+	}
+	room("cpus", cpus, need.CPUs, "CPUs")
+	room("memory", mem, need.MemoryMB, "MB")
+	room("disk", disk, need.DiskGB, "GB free")
+
+	// Time. Leases and etcd's elections go wrong on clocks that disagree.
+	if now, err := strconv.ParseFloat(facts["now"], 64); err == nil {
+		skew := time.Duration((now - float64(at.UnixNano())/1e9) * float64(time.Second))
+		if skew < 0 {
+			skew = -skew
+		}
+		switch {
+		case skew > 5*time.Second:
+			f("clock", Fail, fmt.Sprintf("%s off from this machine", skew.Round(time.Second)), "turn on NTP (timedatectl set-ntp true) on both")
+		case skew > time.Second:
+			f("clock", Warn, fmt.Sprintf("%s off from this machine", skew.Round(100*time.Millisecond)), "turn on NTP on both")
+		default:
+			f("clock", OK, "within a second of this machine", "")
+		}
+	}
+	if facts["ntp"] != "yes" {
+		f("time sync", Warn, "NTP is not synchronised", "timedatectl set-ntp true")
+	}
+
+	// Ports its roles will listen on.
+	listening := map[string]bool{}
+	for _, p := range strings.Fields(facts["listening"]) {
+		listening[p] = true
+	}
+	var busy []string
+	for _, p := range need.Ports {
+		if listening[strconv.Itoa(p)] {
+			busy = append(busy, strconv.Itoa(p))
+		}
+	}
+	if len(busy) > 0 {
+		// A warning: a re-run finds its own services listening.
+		f("ports", Warn, "already listening: "+strings.Join(busy, ", "),
+			"fine if it is llmgw's own from an earlier run; otherwise stop what holds them")
+	} else if len(need.Ports) > 0 {
+		f("ports", OK, "free", "")
+	}
+
+	if facts["internet"] == "yes" {
+		f("internet", OK, "reaches the package mirrors", "")
+	} else {
+		f("internet", Fail, "cannot reach deb.debian.org", "packages and releases are downloaded; allow outbound HTTP/HTTPS or set a proxy")
+	}
+	return fs
+}
+
+func evaluateReach(m Machine, out string) []Finding {
+	var no []string
+	for _, line := range strings.Split(out, "\n") {
+		parts := strings.Fields(line)
+		if len(parts) == 3 && parts[0] == "reach" && parts[2] == "no" {
+			no = append(no, parts[1])
+		}
+	}
+	if len(no) > 0 {
+		return []Finding{{Machine: m.Name, Check: "network", Level: Fail,
+			Detail: "cannot reach " + strings.Join(no, ", "),
+			Fix:    "the machines must reach one another; check routes and firewalls"}}
+	}
+	return []Finding{{Machine: m.Name, Check: "network", Level: OK, Detail: "reaches every other machine"}}
+}
+
+func needOf(roles []Role) Need {
+	var n Need
+	seen := map[int]bool{}
+	for _, r := range roles {
+		x := Needs[r]
+		n.CPUs, n.MemoryMB, n.DiskGB = max(n.CPUs, x.CPUs), max(n.MemoryMB, x.MemoryMB), max(n.DiskGB, x.DiskGB)
+		for _, p := range x.Ports {
+			if !seen[p] {
+				seen[p] = true
+				n.Ports = append(n.Ports, p)
+			}
+		}
+	}
+	return n
+}
+
+func contains(xs []string, x string) bool {
+	for _, y := range xs {
+		if y == x {
+			return true
+		}
+	}
+	return false
+}

internal/preflight/preflight_test.go

@@ -0,0 +1,109 @@
+package preflight
+
+import (
+	"context"
+	"os"
+	"strconv"
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
+)
+
+func facts(over map[string]string) map[string]string {
+	f := map[string]string{
+		"os": "ubuntu", "version": "24.04", "pretty": "Ubuntu 24.04.3 LTS", "arch": "x86_64",
+		"cpus": "2", "mem_mb": "3900", "disk_gb": "35", "ntp": "yes", "listening": "22 53",
+		"sudo": "yes", "apt": "yes", "internet": "yes",
+		"now": strconv.FormatFloat(float64(time.Now().UnixNano())/1e9, 'f', 3, 64),
+	}
+	for k, v := range over {
+		f[k] = v
+	}
+	return f
+}
+
+func levelOf(fs []Finding, check string) (Level, string) {
+	for _, f := range fs {
+		if f.Check == check {
+			return f.Level, f.Detail
+		}
+	}
+	return -1, ""
+}
+
+func TestAMachineFitForItsRoleHasNothingWrong(t *testing.T) {
+	fs := Evaluate(Machine{Name: "pg01", Roles: []Role{RoleDatabase}}, facts(nil), time.Now())
+	if r := (Report{Findings: fs}); r.Worst() != OK {
+		t.Errorf("%+v", fs)
+	}
+}
+
+func TestEachProblemIsFoundAndSaysWhatToDo(t *testing.T) {
+	m := Machine{Name: "x", Roles: []Role{RoleDatabase, RoleControl}}
+	cases := []struct {
+		over  map[string]string
+		check string
+		want  Level
+	}{
+		{map[string]string{"os": "centos", "version": "9"}, "system", Fail},
+		{map[string]string{"version": "20.04"}, "system", Warn},
+		{map[string]string{"arch": "riscv64"}, "architecture", Fail},
+		{map[string]string{"sudo": "no"}, "sudo", Warn},
+		{map[string]string{"mem_mb": "1000"}, "memory", Fail},
+		{map[string]string{"disk_gb": "20"}, "disk", Warn},
+		{map[string]string{"ntp": "no"}, "time sync", Warn},
+		{map[string]string{"listening": "22 5432"}, "ports", Warn},
+		{map[string]string{"internet": "no"}, "internet", Fail},
+		{map[string]string{"now": strconv.FormatFloat(float64(time.Now().Add(-30*time.Second).UnixNano())/1e9, 'f', 3, 64)}, "clock", Fail},
+	}
+	for _, c := range cases {
+		fs := Evaluate(m, facts(c.over), time.Now())
+		got, detail := levelOf(fs, c.check)
+		if got != c.want {
+			t.Errorf("%v: %s is %v (%s), want %v", c.over, c.check, got, detail, c.want)
+		}
+		for _, f := range fs {
+			if f.Level != OK && f.Fix == "" {
+				t.Errorf("%s: %s says nothing about what to do", c.check, f.Check)
+			}
+		}
+	}
+	// This machine is run as root; sudo is not its question.
+	if l, _ := levelOf(Evaluate(Machine{Name: "here", Local: true}, facts(map[string]string{"sudo": "no"}), time.Now()), "sudo"); l != -1 {
+		t.Error("the local machine was asked about sudo")
+	}
+}
+
+func TestAMachineThatCannotReachAnotherIsCalledOut(t *testing.T) {
+	fs := evaluateReach(Machine{Name: "a"}, "reach 10.0.0.2 yes\nreach 10.0.0.3 no\n")
+	if fs[0].Level != Fail || !strings.Contains(fs[0].Detail, "10.0.0.3") {
+		t.Errorf("%+v", fs)
+	}
+}
+
+// TestAgainstRealMachines runs the probe over ssh when LLMGW_TEST_PREFLIGHT
+// names user@host,user@host: the scripts are shell, and only a real machine
+// says whether they parse.
+func TestAgainstRealMachines(t *testing.T) {
+	targets := os.Getenv("LLMGW_TEST_PREFLIGHT")
+	if targets == "" {
+		t.Skip("set LLMGW_TEST_PREFLIGHT=user@host,user@host to run against machines")
+	}
+	var ms []Machine
+	for _, tgt := range strings.Split(targets, ",") {
+		user, host, _ := strings.Cut(tgt, "@")
+		ms = append(ms, Machine{Name: host, Addr: host, Roles: []Role{RoleDatabase},
+			Remote: &onboard.GoSSH{Addr: host, User: user, UseAgent: true}})
+	}
+	rep := Run(context.Background(), ms)
+	for _, f := range rep.Findings {
+		t.Logf("%-12s %-12s %-5s %s", f.Machine, f.Check, f.Level, f.Detail)
+	}
+	for _, f := range rep.Findings {
+		if f.Check == "login" && f.Level != OK {
+			t.Fatalf("could not log in: %s", f.Detail)
+		}
+	}
+}

internal/router/chat.go

@@ -255,6 +255,18 @@ func (r *Router) modelsFor(req *http.Request, user store.User) []web.ChatModel {
 	}
 	seen := map[string]struct{}{}
 	var out []web.ChatModel
+	// What each model's replicas compute on, across all of them: one model
+	// can run on a card and on CPUs at once.
+	hardware := map[string][]string{}
+	for _, s := range r.reg.Snapshots("") {
+		hw := hardwareOf(s.Pool)
+		for _, m := range s.Models {
+			name, _ := r.resolve(m)
+			if hw != "" && !slices.Contains(hardware[name], hw) {
+				hardware[name] = append(hardware[name], hw)
+			}
+		}
+	}
 	for _, s := range r.reg.Snapshots("") {
 		for _, m := range s.Models {
 			// Resolved rather than taken as sent: an alias and the name it
@@ -265,7 +277,7 @@ func (r *Router) modelsFor(req *http.Request, user store.User) []web.ChatModel {
 			}
 			seen[name] = struct{}{}
 			if r.permits(principal, name) {
-				out = append(out, web.ChatModel{Name: name, Simulated: entry.Simulated()})
+				out = append(out, web.ChatModel{Name: name, Simulated: entry.Simulated(), Hardware: hardware[name]})
 			}
 		}
 	}

internal/router/handlers.go

@@ -123,6 +123,19 @@ func (r *Router) Handler() http.Handler {
 		w.WriteHeader(http.StatusOK)
 	})
 
+	// Leadership: 200 on the control plane that places work, 503 on a
+	// standby. What a floating address follows (llmgw server add --vip):
+	// unlike /ready it does not wait for a replica, so a cluster with no
+	// node yet still has its address on one machine. A process that is not
+	// a control plane, or one with no rival, is always the leader.
+	mux.HandleFunc("GET /leader", func(w http.ResponseWriter, _ *http.Request) {
+		if f := r.isLeader.Load(); f != nil && !(*f)() {
+			http.Error(w, "this control plane is a standby", http.StatusServiceUnavailable)
+			return
+		}
+		w.WriteHeader(http.StatusOK)
+	})
+
 	// The landing page says how to connect and nothing about what is here, so
 	// it is safe to serve without a key — and it has to be, because the person
 	// reading it is the one who does not have one yet.
@@ -227,7 +240,7 @@ func (r *Router) Handler() http.Handler {
 	// route must not be able to add an unauthenticated one by omission. Health
 	// stays open because a load balancer has no key.
 	return auth.Middleware(r.auth, r.log,
-		"/health", "/ready", "/", "/about", "/_static/", "/claim/",
+		"/health", "/ready", "/leader", "/", "/about", "/_static/", "/claim/",
 		"/signup", "/login", "/logout", "/profile", "/profile/",
 		"/admin", "/admin/", "/chat", "/chat/", "/org", "/org/", "/setup/")(mux)
 }
@@ -261,6 +274,7 @@ type parsedRequest struct {
 	KeyID           string
 	UserID          string
 	ProjectID       string
+	TeamID, OrgID   string
 	Tenant          string
 	ProfileOverride string
 	PromptPrefix    string
@@ -519,6 +533,7 @@ func (r *Router) account(req *parsedRequest, res dispatchResult, status int, too
 		// the truth than charging nothing.
 		u.PromptTokens = int64(req.PromptTokens)
 	}
+	r.m.tenant.observe(req, status, u.PromptTokens, u.CompletionTokens, res.outcome.TTFT)
 	r.acct.Record(req.reservation, u)
 }
 
@@ -537,6 +552,12 @@ func (r *Router) rejectLimit(w http.ResponseWriter, req *parsedRequest, err erro
 	// which is absent when waiting does not help.
 	writeError(w, http.StatusTooManyRequests, "rate_limit_exceeded", le.Error())
 	r.m.observeRequest(r.metricModel(req.Model), "", http.StatusTooManyRequests, time.Since(start), nil, 0)
+	r.m.tenant.observe(req, http.StatusTooManyRequests, 0, 0, 0)
+	r.acct.RecordRefused(store.Usage{
+		At: time.Now(), KeyID: req.KeyID, UserID: req.UserID, ProjectID: req.ProjectID,
+		Model: req.Model, ServedModel: req.Model, Status: http.StatusTooManyRequests,
+		LatencyMs: time.Since(start).Milliseconds(), RequestID: req.RequestID,
+	})
 	r.accessLog(req, dispatchResult{err: err}, http.StatusTooManyRequests, time.Since(start))
 }
 
@@ -579,6 +600,7 @@ func (r *Router) parse(req *http.Request, path string) (*parsedRequest, error) {
 		p.KeyID = principal.KeyID
 		p.UserID = principal.UserID
 		p.ProjectID = principal.ProjectID
+		p.TeamID, p.OrgID = principal.TeamID, principal.OrgID
 		// Tenant fairness within a band is per key rather than per session, so
 		// one key cannot take a whole band by opening many sessions.
 		p.Tenant = principal.KeyID

internal/router/hardware_leader_test.go

@@ -0,0 +1,43 @@
+package router
+
+import (
+	"net/http"
+	"net/http/httptest"
+	"path/filepath"
+	"testing"
+)
+
+func TestTheLeaderEndpointFollowsLeadership(t *testing.T) {
+	rt := newTestRouter(t, baseConfig())
+	get := func() int {
+		rec := httptest.NewRecorder()
+		rt.Handler().ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/leader", nil))
+		return rec.Code
+	}
+	// No rival, nothing to follow: the one process leads.
+	if c := get(); c != http.StatusOK {
+		t.Errorf("with no leadership: %d", c)
+	}
+	leading := false
+	rt.SetLeader(func() bool { return leading })
+	if c := get(); c != http.StatusServiceUnavailable {
+		t.Errorf("a standby: %d", c)
+	}
+	leading = true
+	if c := get(); c != http.StatusOK {
+		t.Errorf("the leader: %d", c)
+	}
+}
+
+func TestTheLeaderEndpointNeedsNoKey(t *testing.T) {
+	// keepalived and load balancers ask it, and they have no key.
+	cfg := baseConfig()
+	cfg.Auth.Enabled = true
+	cfg.Auth.Store = filepath.Join(t.TempDir(), "store.db")
+	rt := newTestRouter(t, cfg)
+	rec := httptest.NewRecorder()
+	rt.Handler().ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/leader", nil))
+	if rec.Code != http.StatusOK {
+		t.Errorf("/leader without a key: %d", rec.Code)
+	}
+}

internal/router/hardware_test.go

@@ -0,0 +1,50 @@
+package router
+
+import (
+	"net/http"
+	"net/http/httptest"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
+)
+
+func TestAReplySaysWhatHardwareAnsweredIt(t *testing.T) {
+	// qwen2.5-coder-14b had one replica on cuda01's card and one on a CPU
+	// node, and the chat page could not tell which had answered: the same
+	// model, ten times faster or slower, for no visible reason.
+	for pool, want := range map[string]string{
+		"none/6000-24g": "6000-24g",
+		"none/cpu":      "cpu",
+		"":              "",
+	} {
+		e := newFakeEngine(t, "m")
+		rt := newTestRouter(t, baseConfig())
+		rt.SetEndpoints(t.Context(), []endpoint.Info{
+			{ID: "r", URL: e.srv.URL, Models: []string{"m"}, Pool: pool, Weight: 1},
+		})
+		waitForEndpoint(t, rt)
+
+		rec := post(t, rt.Handler(), map[string]any{"model": "m", "messages": []any{}}, nil)
+		if rec.Code != http.StatusOK {
+			t.Fatalf("%q: status %d: %s", pool, rec.Code, rec.Body.String())
+		}
+		if got := rec.Header().Get(hardwareHeader); got != want {
+			t.Errorf("pool %q: hardware %q, want %q", pool, got, want)
+		}
+	}
+}
+
+func TestThePickerSaysWhereEachModelRuns(t *testing.T) {
+	gpu, cpu := newFakeEngine(t, "qwen2.5-coder-7b-instruct"), newFakeEngine(t, "qwen2.5-coder-7b-instruct")
+	rt, user := chatRouter(t, gpu.srv.URL)
+	rt.SetEndpoints(t.Context(), []endpoint.Info{
+		{ID: "g", URL: gpu.srv.URL, Models: []string{"qwen2.5-coder-7b-instruct"}, Pool: "none/6000-24g", Weight: 1},
+		{ID: "c", URL: cpu.srv.URL, Models: []string{"coder"}, Pool: "none/cpu", Weight: 1},
+	})
+	waitForEndpoint(t, rt)
+
+	got := rt.modelsFor(httptest.NewRequest("GET", "/chat", nil), user)
+	if len(got) != 1 || got[0].Runs() != "GPU + CPU" {
+		t.Fatalf("picker %+v; want one model running on GPU + CPU, its alias's replica included", got)
+	}
+}

internal/router/metrics.go

@@ -29,6 +29,8 @@ type metrics struct {
 	admissionRejected *prometheus.CounterVec
 
 	prefixDecisions *prometheus.CounterVec
+
+	tenant *tenantMetrics
 }
 
 func newMetrics() *metrics {
@@ -86,16 +88,17 @@ func newMetrics() *metrics {
 			Name: "llmgw_admission_rejected_total",
 			Help: "Requests refused by flow control, by reason.",
 		}, []string{"band", "reason"}),
+		tenant: newTenantMetrics(),
 	}
 }
 
 // collectors returns everything to register, so registration lives in one place.
 func (m *metrics) collectors() []prometheus.Collector {
-	return []prometheus.Collector{
+	return append([]prometheus.Collector{
 		m.requests, m.duration, m.ttft, m.tokens,
 		m.scorerValue, m.filtered, m.considered,
 		m.admissionWait, m.admissionRejected, m.prefixDecisions,
-	}
+	}, m.tenant.collectors()...)
 }
 
 // observePrefix records which branch a prefix decision took.

internal/router/pricing_test.go

@@ -7,6 +7,8 @@ import (
 	"testing"
 	"time"
 
+	"github.com/prometheus/client_golang/prometheus/testutil"
+
 	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
@@ -188,3 +190,75 @@ func TestAnIDThatIsNotOneIsReplaced(t *testing.T) {
 		t.Error("a plain id was not kept")
 	}
 }
+
+func TestARequestALimitRefusesIsCountedAgainstItsOwner(t *testing.T) {
+	// Refused requests were counted only in Prometheus, by model: a team
+	// running into its limit looked, on every usage page, like a team that
+	// had gone quiet.
+	e := newFakeEngine(t, "qwen2.5-14b-instruct")
+	rt, secret := pricedRouter(t, e.srv.URL)
+	rpm := 1
+	if err := rt.store.PutLimits(t.Context(), store.Limits{Scope: store.ScopeKey, ScopeID: "key-1", RPM: &rpm}); err != nil {
+		t.Fatal(err)
+	}
+	body := map[string]any{"model": "q14", "messages": []any{map[string]any{"role": "user", "content": "hi"}}}
+	auth := map[string]string{"Authorization": "Bearer " + secret}
+	codes := []int{post(t, rt.Handler(), body, auth).Code, post(t, rt.Handler(), body, auth).Code}
+	if codes[1] != 429 {
+		t.Fatalf("the second request in the minute answered %v, want 429", codes)
+	}
+	deadline := time.Now().Add(2 * time.Second)
+	for time.Now().Before(deadline) {
+		stats, err := rt.store.StatsBy(t.Context(), store.GroupByProject, time.Now().Add(-time.Hour))
+		if err != nil {
+			t.Fatal(err)
+		}
+		for _, s := range stats {
+			if s.Limited == 1 && s.Requests == 2 {
+				spend, _ := rt.store.SpendSince(t.Context(), time.Now().Add(-time.Hour))
+				if spend["key-1"] == 0 {
+					t.Error("the request that was served cost nothing")
+				}
+				return
+			}
+		}
+		time.Sleep(20 * time.Millisecond)
+	}
+	t.Fatal("the refused request was never counted against its project")
+}
+
+func TestTenantsAreCountedAndTheirBudgetsGauged(t *testing.T) {
+	e := newFakeEngine(t, "qwen2.5-14b-instruct")
+	rt, secret := pricedRouter(t, e.srv.URL)
+	ctx := t.Context()
+	orgs, _ := rt.store.AllOrgs(ctx)
+	teams, _ := rt.store.AllTeams(ctx)
+	budget := int64(1_000_000)
+	if err := rt.store.PutLimits(ctx, store.Limits{Scope: store.ScopeTeam, ScopeID: teams[0].ID,
+		Budget: &budget, BudgetPeriod: store.PeriodMonth}); err != nil {
+		t.Fatal(err)
+	}
+	if err := rt.m.tenant.refresh(ctx, rt.store, time.Now()); err != nil {
+		t.Fatal(err)
+	}
+	rec := post(t, rt.Handler(), map[string]any{"model": "q14", "messages": []any{
+		map[string]any{"role": "user", "content": "hello"}}},
+		map[string]string{"Authorization": "Bearer " + secret})
+	if rec.Code != 200 {
+		t.Fatalf("status %d", rec.Code)
+	}
+	tenant := orgs[0].Name + "/" + teams[0].Name
+	if got := testutil.ToFloat64(rt.m.tenant.requests.WithLabelValues(tenant, "ok")); got != 1 {
+		t.Errorf("%s has %v ok requests, want 1", tenant, got)
+	}
+
+	u := waitForUsage(t, rt)
+	if err := rt.m.tenant.refresh(ctx, rt.store, time.Now()); err != nil {
+		t.Fatal(err)
+	}
+	want := float64(u.CostMicros) / float64(budget)
+	got := testutil.ToFloat64(rt.m.tenant.budget.WithLabelValues("team", teams[0].ID, tenant, "month"))
+	if want == 0 || got != want {
+		t.Errorf("the team's budget gauge reads %v, want %v", got, want)
+	}
+}

internal/router/router.go

@@ -83,6 +83,10 @@ type Router struct {
 	auth     *auth.Authenticator
 	accounts *account.Service
 	store    *storepkg.DB
+	// usageRetention is how long request records are kept; zero is for ever.
+	usageRetention time.Duration
+	// isLeader says whether this process leads; nil is always.
+	isLeader atomic.Pointer[func() bool]
 	acct     *accounting.Accountant
 
 	// threads holds conversations and memory notes. It is store when auth is on,
@@ -189,6 +193,7 @@ func New(cfg config.Server, log *slog.Logger) (*Router, error) {
 	// the decision metric is the whole difference between tuning this and
 	// guessing at it.
 	metrics := newMetrics()
+	metrics.tenant.level = cfg.Router.TenantLabel
 	var prefixScorer *scheduler.PrefixScorer
 	if cfg.Router.Prefix.Enabled {
 		prefixScorer = scheduler.NewPrefixScorer(
@@ -248,6 +253,7 @@ func New(cfg config.Server, log *slog.Logger) (*Router, error) {
 			return nil, fmt.Errorf("auth store: %w", err)
 		}
 		r.store = st
+		r.usageRetention = cfg.Auth.UsageRetention
 		// Applied where no scope set one, so a project created without limits
 		// is still metered. Zero leaves it unlimited, which was the old
 		// behaviour and is now a choice rather than an omission.
@@ -412,6 +418,60 @@ func (r *Router) Start(ctx context.Context) {
 	if r.threads != nil {
 		go r.pruneThreads(ctx)
 	}
+	if r.store != nil && r.usageRetention > 0 {
+		go r.pruneUsage(ctx, r.usageRetention)
+	}
+	if r.store != nil {
+		go r.refreshTenants(ctx)
+	}
+}
+
+// refreshTenants keeps the tenant names the request series are labelled with,
+// and the budget gauges, a minute fresh.
+func (r *Router) refreshTenants(ctx context.Context) {
+	refresh := func() {
+		if err := r.m.tenant.refresh(ctx, r.store, time.Now()); err != nil && ctx.Err() == nil {
+			r.log.Warn("could not refresh tenant names and budget gauges", "err", err)
+		}
+	}
+	refresh()
+	t := time.NewTicker(time.Minute)
+	defer t.Stop()
+	for {
+		select {
+		case <-ctx.Done():
+			return
+		case <-t.C:
+			refresh()
+		}
+	}
+}
+
+// pruneUsage deletes request records older than the retention, hourly and
+// once at startup, as pruneThreads does. The five-minute sums stay, so no
+// total or chart moves; what goes is the trail of each old request.
+func (r *Router) pruneUsage(ctx context.Context, keep time.Duration) {
+	prune := func() {
+		n, err := r.store.PruneUsage(ctx, time.Now().Add(-keep))
+		if err != nil {
+			r.log.Warn("could not prune old request records", "err", err)
+			return
+		}
+		if n > 0 {
+			r.log.Info("removed request records past their retention", "rows", n, "retention", keep)
+		}
+	}
+	prune()
+	t := time.NewTicker(time.Hour)
+	defer t.Stop()
+	for {
+		select {
+		case <-ctx.Done():
+			return
+		case <-t.C:
+			prune()
+		}
+	}
 }
 
 // pruneThreads enforces each project's retention window.
@@ -474,6 +534,9 @@ func (r *Router) Stop() {
 	}
 }
 
+// SetLeader tells /leader how to know whether this process leads.
+func (r *Router) SetLeader(f func() bool) { r.isLeader.Store(&f) }
+
 // Registry exposes the endpoint registry, for tests and admin endpoints.
 func (r *Router) Registry() *endpoint.Registry { return r.reg }
 
@@ -749,6 +812,16 @@ func (r *Router) dispatchOne(ctx context.Context, w http.ResponseWriter, req *pa
 		if a, ok := r.adapter(req.Model); ok && a.base == model {
 			body = withAdapter(body, model, a.index, a.scale)
 		}
+		// What computes the answer, said before the proxy writes the headers.
+		// A model can have replicas on cards and on CPUs at once, and the two
+		// differ tenfold in speed: a reply that came slowly is otherwise
+		// indistinguishable from a model that is slow everywhere. Set again on
+		// a retry, which only happens before anything was written.
+		if hw := hardwareOf(dec.Endpoint.Pool); hw != "" {
+			w.Header().Set(hardwareHeader, hw)
+		} else {
+			w.Header().Del(hardwareHeader)
+		}
 		out, perr := r.px.Do(uctx, w, proxy.Request{
 			URL:                dec.Endpoint.URL + req.Path,
 			APIKey:             dec.Endpoint.APIKey,
@@ -791,6 +864,21 @@ func (r *Router) dispatchOne(ctx context.Context, w http.ResponseWriter, req *pa
 	return last
 }
 
+// hardwareHeader says what the replica that answered computes on: "cpu", or
+// the GPU class, such as "6000-24g".
+const hardwareHeader = "x-llmgw-hardware"
+
+// hardwareOf reads it from a pool, which placement names fabric/class —
+// "none/cpu", "none/6000-24g", "ib-a/h100-80g". Empty for an endpoint
+// configured by hand, which has no pool and says nothing about its hardware.
+func hardwareOf(pool string) string {
+	i := strings.LastIndexByte(pool, '/')
+	if i < 0 || i == len(pool)-1 {
+		return ""
+	}
+	return pool[i+1:]
+}
+
 func without(eps []endpoint.Snapshot, id string) []endpoint.Snapshot {
 	return slices.DeleteFunc(slices.Clone(eps), func(e endpoint.Snapshot) bool { return e.ID == id })
 }

internal/router/tenantmetrics.go

@@ -0,0 +1,182 @@
+package router
+
+import (
+	"context"
+	"strings"
+	"sync"
+	"time"
+
+	"github.com/prometheus/client_golang/prometheus"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/accounting"
+	storepkg "gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// tenantMetrics are the router's series per tenant: requests by how they
+// ended, tokens, time to first token, and how much of each budget is spent.
+//
+// The request series carry one level of the tree, chosen by
+// router.tenant_label, rather than every key or project: a series per project
+// on a cluster of thousands is a Prometheus that falls over, and alerting on a
+// team is what an operator does. Budgets are a series per limit that has one,
+// which is as many as somebody wrote.
+type tenantMetrics struct {
+	level string
+
+	requests *prometheus.CounterVec
+	tokens   *prometheus.CounterVec
+	ttft     *prometheus.HistogramVec
+	budget   *prometheus.GaugeVec
+
+	mu    sync.RWMutex
+	names map[string]string // "team:<id>" -> "acme/platform"
+}
+
+func newTenantMetrics() *tenantMetrics {
+	return &tenantMetrics{
+		level: "team",
+		requests: prometheus.NewCounterVec(prometheus.CounterOpts{
+			Name: "llmgw_tenant_requests_total",
+			Help: "Requests by tenant (the level router.tenant_label names) and outcome: ok, error, or limited — refused by a rate, token or budget limit.",
+		}, []string{"tenant", "outcome"}),
+		tokens: prometheus.NewCounterVec(prometheus.CounterOpts{
+			Name: "llmgw_tenant_tokens_total",
+			Help: "Tokens by tenant and direction.",
+		}, []string{"tenant", "direction"}),
+		ttft: prometheus.NewHistogramVec(prometheus.HistogramOpts{
+			Name:    "llmgw_tenant_ttft_seconds",
+			Help:    "Time to first token by tenant, for streamed requests.",
+			Buckets: []float64{0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60},
+		}, []string{"tenant"}),
+		budget: prometheus.NewGaugeVec(prometheus.GaugeOpts{
+			Name: "llmgw_budget_spent_ratio",
+			Help: "Spend this period over the budget, for every organization, team, project and person given one. 1 is spent.",
+		}, []string{"scope", "id", "name", "period"}),
+		names: map[string]string{},
+	}
+}
+
+func (t *tenantMetrics) collectors() []prometheus.Collector {
+	return []prometheus.Collector{t.requests, t.tokens, t.ttft, t.budget}
+}
+
+// tenantOf is the label a request is counted under: the name of its
+// organization, team or project at the configured level, "personal" for use
+// outside any project, or "" when tenants are not labelled at all.
+func (t *tenantMetrics) tenantOf(req *parsedRequest) string {
+	var scope, id string
+	switch t.level {
+	case "org":
+		scope, id = "org", req.OrgID
+	case "team":
+		scope, id = "team", req.TeamID
+	case "project":
+		scope, id = "project", req.ProjectID
+	default:
+		return ""
+	}
+	if id == "" {
+		return "personal"
+	}
+	t.mu.RLock()
+	name, ok := t.names[scope+":"+id]
+	t.mu.RUnlock()
+	if !ok {
+		// Not yet in the cache — made since the last refresh. The id is
+		// stable, which is what a label has to be; the name follows.
+		return id
+	}
+	return name
+}
+
+// observe counts one finished or refused request.
+func (t *tenantMetrics) observe(req *parsedRequest, status int, prompt, completion int64, ttft time.Duration) {
+	tenant := t.tenantOf(req)
+	if tenant == "" {
+		return
+	}
+	outcome := "ok"
+	switch {
+	case status == 429:
+		outcome = "limited"
+	case status >= 400:
+		outcome = "error"
+	}
+	t.requests.WithLabelValues(tenant, outcome).Inc()
+	if prompt > 0 {
+		t.tokens.WithLabelValues(tenant, "prompt").Add(float64(prompt))
+	}
+	if completion > 0 {
+		t.tokens.WithLabelValues(tenant, "completion").Add(float64(completion))
+	}
+	if ttft > 0 {
+		t.ttft.WithLabelValues(tenant).Observe(ttft.Seconds())
+	}
+}
+
+// tenantStore is what the refresh reads.
+type tenantStore interface {
+	Tree(ctx context.Context) ([]storepkg.TreeOrg, error)
+	AllLimits(ctx context.Context) (map[string]storepkg.Limits, error)
+	SpendOfPool(ctx context.Context, l storepkg.Limits, since time.Time) (int64, error)
+	Users(ctx context.Context, state storepkg.UserState) ([]storepkg.User, error)
+}
+
+// refresh reads the tree's names and every budget's spend. A minute old at
+// most, which is plenty for an alert that waits five before firing.
+func (t *tenantMetrics) refresh(ctx context.Context, s tenantStore, now time.Time) error {
+	tree, err := s.Tree(ctx)
+	if err != nil {
+		return err
+	}
+	names := map[string]string{}
+	for _, o := range tree {
+		names["org:"+o.ID] = o.Name
+		for _, tm := range o.Teams {
+			names["team:"+tm.ID] = o.Name + "/" + tm.Name
+			for _, p := range tm.Projects {
+				names["project:"+p.ID] = o.Name + "/" + tm.Name + "/" + p.Name
+			}
+		}
+	}
+	t.mu.Lock()
+	t.names = names
+	t.mu.Unlock()
+
+	limits, err := s.AllLimits(ctx)
+	if err != nil {
+		return err
+	}
+	emails := map[string]string{}
+	if users, err := s.Users(ctx, ""); err == nil {
+		for _, u := range users {
+			emails[u.ID] = u.Email
+		}
+	}
+	// Rebuilt whole, so a budget removed since the last refresh stops being
+	// published rather than staying at its last value for ever.
+	t.budget.Reset()
+	for _, l := range limits {
+		if l.Budget == nil || *l.Budget <= 0 {
+			continue
+		}
+		name := ""
+		switch l.Scope {
+		case storepkg.ScopeOrg, storepkg.ScopeTeam, storepkg.ScopeProject:
+			name = names[string(l.Scope)+":"+l.ScopeID]
+		case storepkg.ScopeUser:
+			name = emails[l.ScopeID]
+		default:
+			// Per key and per member are too many series and too little
+			// to act on; their pools are bounded by the levels above.
+			continue
+		}
+		spent, err := s.SpendOfPool(ctx, l, accounting.PeriodStart(l.BudgetPeriod, now))
+		if err != nil {
+			return err
+		}
+		t.budget.WithLabelValues(string(l.Scope), l.ScopeID, strings.TrimSpace(name), string(l.BudgetPeriod)).
+			Set(float64(spent) / float64(*l.Budget))
+	}
+	return nil
+}

internal/settings/settings.go

@@ -0,0 +1,513 @@
+// Package settings edits llmgw's configuration files key by key, for the
+// settings screens: what each key is comes from the generated reference, so
+// every setting the code reads is there without anyone listing it, and the
+// file is edited as YAML nodes, so its comments and order survive.
+package settings
+
+import (
+	"bytes"
+	"fmt"
+	"os"
+	"reflect"
+	"strconv"
+	"strings"
+	"time"
+
+	"go.yaml.in/yaml/v3"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/docgen"
+)
+
+// Kind is how a key is shown and edited.
+type Kind int
+
+// The kinds of key.
+const (
+	Text Kind = iota
+	Int
+	Float
+	Bool
+	Duration
+	TextList  // []string, []int: one value a line
+	TextMap   // map[string]string and the like: key = value lines
+	Block     // a nested block: a submenu
+	BlockList // a list of blocks: a submenu of entries
+	BlockMap  // a map of blocks: a submenu of named entries
+)
+
+// Key is one key of a block, as the screens need it.
+type Key struct {
+	docgen.Field
+	Kind Kind
+	// Block is the block it opens, for Block, BlockList and BlockMap.
+	Block *docgen.Block
+}
+
+// Blocks finds a block by ID in the generated reference.
+func Find(id string) (*docgen.Block, bool) {
+	for i := range docgen.Reference {
+		if docgen.Reference[i].ID() == id {
+			return &docgen.Reference[i], true
+		}
+	}
+	return nil, false
+}
+
+// Keys are a block's keys with their kinds.
+func Keys(b *docgen.Block) []Key {
+	out := make([]Key, 0, len(b.Fields))
+	for _, f := range b.Fields {
+		k := Key{Field: f}
+		k.Kind, k.Block = kindOf(b.Package, f)
+		out = append(out, k)
+	}
+	return out
+}
+
+func kindOf(pkg string, f docgen.Field) (Kind, *docgen.Block) {
+	if nb, ok := Find(f.Nested); ok {
+		return Block, nb
+	}
+	t := strings.TrimPrefix(f.Type, "*")
+	switch t {
+	case "string":
+		return Text, nil
+	case "int", "int64", "uint", "uint32":
+		return Int, nil
+	case "float64", "float32":
+		return Float, nil
+	case "bool":
+		return Bool, nil
+	case "time.Duration":
+		return Duration, nil
+	}
+	if elem, ok := strings.CutPrefix(t, "[]"); ok {
+		if b, ok := Find(qualify(pkg, elem)); ok {
+			return BlockList, b
+		}
+		return TextList, nil
+	}
+	if strings.HasPrefix(t, "map[") {
+		elem := t[strings.Index(t, "]")+1:]
+		if b, ok := Find(qualify(pkg, elem)); ok {
+			return BlockMap, b
+		}
+		return TextMap, nil
+	}
+	return Text, nil
+}
+
+// qualify is a type as a block ID: Endpoint in config is config.Endpoint.
+func qualify(pkg, typ string) string {
+	if strings.Contains(typ, ".") {
+		return typ
+	}
+	return pkg + "." + typ
+}
+
+// Root is the block a file is: server.yaml is config.Server, agent.yaml the
+// agent's Config.
+func Root(file string) (*docgen.Block, error) {
+	id := "config.Server"
+	if strings.Contains(file, "agent") {
+		id = "agent.Config"
+	}
+	b, ok := Find(id)
+	if !ok {
+		return nil, fmt.Errorf("no block %s in the reference", id)
+	}
+	return b, nil
+}
+
+// Doc is a configuration file, open for editing.
+type Doc struct {
+	Path string
+	root *yaml.Node // the document's top mapping
+	doc  yaml.Node
+	// Changed counts edits not yet saved.
+	Changed int
+}
+
+// Open reads a file; a missing one is an empty document.
+func Open(path string) (*Doc, error) {
+	d := &Doc{Path: path}
+	b, err := os.ReadFile(path)
+	if err != nil && !os.IsNotExist(err) {
+		return nil, err
+	}
+	if len(bytes.TrimSpace(b)) > 0 {
+		if err := yaml.Unmarshal(b, &d.doc); err != nil {
+			return nil, fmt.Errorf("%s: %w", path, err)
+		}
+	}
+	if d.doc.Kind == 0 {
+		d.doc = yaml.Node{Kind: yaml.DocumentNode, Content: []*yaml.Node{{Kind: yaml.MappingNode, Tag: "!!map"}}}
+	}
+	d.root = d.doc.Content[0]
+	if d.root.Kind != yaml.MappingNode {
+		return nil, fmt.Errorf("%s is not a mapping at the top", path)
+	}
+	return d, nil
+}
+
+// Bytes is the document as it would be written.
+func (d *Doc) Bytes() ([]byte, error) {
+	var b bytes.Buffer
+	enc := yaml.NewEncoder(&b)
+	enc.SetIndent(2)
+	if err := enc.Encode(&d.doc); err != nil {
+		return nil, err
+	}
+	return b.Bytes(), enc.Close()
+}
+
+// Node is the node at a path of keys and list indexes, or nil.
+func (d *Doc) Node(path []string) *yaml.Node {
+	n := d.root
+	for _, p := range path {
+		n = child(n, p)
+		if n == nil {
+			return nil
+		}
+	}
+	return n
+}
+
+func child(n *yaml.Node, key string) *yaml.Node {
+	switch n.Kind {
+	case yaml.MappingNode:
+		for i := 0; i+1 < len(n.Content); i += 2 {
+			if n.Content[i].Value == key {
+				return n.Content[i+1]
+			}
+		}
+	case yaml.SequenceNode:
+		if i, err := strconv.Atoi(key); err == nil && i >= 0 && i < len(n.Content) {
+			return n.Content[i]
+		}
+	}
+	return nil
+}
+
+// ensure is the node at a path, made as mappings where there is none.
+func (d *Doc) ensure(path []string) *yaml.Node {
+	n := d.root
+	for _, p := range path {
+		c := child(n, p)
+		if c == nil {
+			if n.Kind != yaml.MappingNode {
+				return nil
+			}
+			c = &yaml.Node{Kind: yaml.MappingNode, Tag: "!!map"}
+			n.Content = append(n.Content, &yaml.Node{Kind: yaml.ScalarNode, Tag: "!!str", Value: p}, c)
+		}
+		n = c
+	}
+	return n
+}
+
+// Set writes one key's value: a scalar, a list of lines or a map of
+// "key = value" lines, by its kind. It checks the value first.
+func (d *Doc) Set(path []string, k Kind, value string) error {
+	if len(path) == 0 {
+		return fmt.Errorf("no key")
+	}
+	n, err := encode(k, value)
+	if err != nil {
+		return err
+	}
+	parent := d.ensure(path[:len(path)-1])
+	if parent == nil {
+		return fmt.Errorf("%s is not a block", strings.Join(path[:len(path)-1], "."))
+	}
+	key := path[len(path)-1]
+	if parent.Kind == yaml.SequenceNode {
+		i, _ := strconv.Atoi(key)
+		parent.Content[i] = n
+	} else if c := child(parent, key); c != nil {
+		// Kept in place, with whatever comment it had.
+		n.HeadComment, n.LineComment, n.FootComment = c.HeadComment, c.LineComment, c.FootComment
+		*c = *n
+	} else {
+		parent.Content = append(parent.Content, &yaml.Node{Kind: yaml.ScalarNode, Tag: "!!str", Value: key}, n)
+	}
+	d.Changed++
+	return nil
+}
+
+// Unset removes a key, so its default applies.
+func (d *Doc) Unset(path []string) {
+	if len(path) == 0 {
+		return
+	}
+	parent := d.Node(path[:len(path)-1])
+	if parent == nil {
+		return
+	}
+	key := path[len(path)-1]
+	switch parent.Kind {
+	case yaml.MappingNode:
+		for i := 0; i+1 < len(parent.Content); i += 2 {
+			if parent.Content[i].Value == key {
+				parent.Content = append(parent.Content[:i], parent.Content[i+2:]...)
+				d.Changed++
+				return
+			}
+		}
+	case yaml.SequenceNode:
+		if i, err := strconv.Atoi(key); err == nil && i < len(parent.Content) {
+			parent.Content = append(parent.Content[:i], parent.Content[i+1:]...)
+			d.Changed++
+		}
+	}
+}
+
+// Append adds an empty block to the list at path, and returns its index.
+func (d *Doc) Append(path []string) int {
+	parent := d.ensure(path[:len(path)-1])
+	key := path[len(path)-1]
+	seq := child(parent, key)
+	if seq == nil || seq.Kind != yaml.SequenceNode {
+		seq = &yaml.Node{Kind: yaml.SequenceNode, Tag: "!!seq"}
+		if c := child(parent, key); c != nil {
+			*c = *seq
+			seq = c
+		} else {
+			parent.Content = append(parent.Content, &yaml.Node{Kind: yaml.ScalarNode, Tag: "!!str", Value: key}, seq)
+		}
+	}
+	seq.Content = append(seq.Content, &yaml.Node{Kind: yaml.MappingNode, Tag: "!!map"})
+	d.Changed++
+	return len(seq.Content) - 1
+}
+
+// AddEntry adds an empty block under a name to the map at path.
+func (d *Doc) AddEntry(path []string, name string) error {
+	m := d.ensure(path)
+	if m == nil || m.Kind != yaml.MappingNode {
+		return fmt.Errorf("%s is not a map", strings.Join(path, "."))
+	}
+	if child(m, name) != nil {
+		return fmt.Errorf("%s is there already", name)
+	}
+	m.Content = append(m.Content, &yaml.Node{Kind: yaml.ScalarNode, Tag: "!!str", Value: name},
+		&yaml.Node{Kind: yaml.MappingNode, Tag: "!!map"})
+	d.Changed++
+	return nil
+}
+
+// Len is how many entries a list or map at path has.
+func (d *Doc) Len(path []string) int {
+	n := d.Node(path)
+	switch {
+	case n == nil:
+		return 0
+	case n.Kind == yaml.SequenceNode:
+		return len(n.Content)
+	case n.Kind == yaml.MappingNode:
+		return len(n.Content) / 2
+	}
+	return 0
+}
+
+// MapKeys are a map's keys, in the file's order.
+func (d *Doc) MapKeys(path []string) []string {
+	n := d.Node(path)
+	if n == nil || n.Kind != yaml.MappingNode {
+		return nil
+	}
+	var out []string
+	for i := 0; i+1 < len(n.Content); i += 2 {
+		out = append(out, n.Content[i].Value)
+	}
+	return out
+}
+
+// Text is a key's value as it is edited: a scalar as written, a list one
+// item a line, a map "key = value" a line. ok is false for a key the file
+// does not set.
+func (d *Doc) Text(path []string) (string, bool) {
+	n := d.Node(path)
+	if n == nil {
+		return "", false
+	}
+	switch n.Kind {
+	case yaml.ScalarNode:
+		return n.Value, true
+	case yaml.SequenceNode:
+		var l []string
+		for _, c := range n.Content {
+			l = append(l, c.Value)
+		}
+		return strings.Join(l, "\n"), true
+	case yaml.MappingNode:
+		var l []string
+		for i := 0; i+1 < len(n.Content); i += 2 {
+			l = append(l, n.Content[i].Value+" = "+n.Content[i+1].Value)
+		}
+		return strings.Join(l, "\n"), true
+	}
+	return "", true
+}
+
+func encode(k Kind, value string) (*yaml.Node, error) {
+	v := strings.TrimSpace(value)
+	scalar := func(tag string) *yaml.Node { return &yaml.Node{Kind: yaml.ScalarNode, Tag: tag, Value: v} }
+	switch k {
+	case Int:
+		if _, err := strconv.ParseInt(v, 10, 64); err != nil {
+			return nil, fmt.Errorf("%q is not a whole number", v)
+		}
+		return scalar("!!int"), nil
+	case Float:
+		if _, err := strconv.ParseFloat(v, 64); err != nil {
+			return nil, fmt.Errorf("%q is not a number", v)
+		}
+		return scalar("!!float"), nil
+	case Bool:
+		b, err := strconv.ParseBool(v)
+		if err != nil {
+			return nil, fmt.Errorf("%q is not true or false", v)
+		}
+		return &yaml.Node{Kind: yaml.ScalarNode, Tag: "!!bool", Value: strconv.FormatBool(b)}, nil
+	case Duration:
+		if _, err := time.ParseDuration(v); err != nil {
+			return nil, fmt.Errorf("%q is not a duration, like 30s, 5m or 2h", v)
+		}
+		return scalar("!!str"), nil
+	case TextList:
+		seq := &yaml.Node{Kind: yaml.SequenceNode, Tag: "!!seq"}
+		for _, l := range splitLines(value) {
+			seq.Content = append(seq.Content, &yaml.Node{Kind: yaml.ScalarNode, Tag: "!!str", Value: l})
+		}
+		return seq, nil
+	case TextMap:
+		m := &yaml.Node{Kind: yaml.MappingNode, Tag: "!!map"}
+		for _, l := range splitLines(value) {
+			k, val, ok := strings.Cut(l, "=")
+			if !ok {
+				return nil, fmt.Errorf("%q is not key = value", l)
+			}
+			m.Content = append(m.Content,
+				&yaml.Node{Kind: yaml.ScalarNode, Tag: "!!str", Value: strings.TrimSpace(k)},
+				&yaml.Node{Kind: yaml.ScalarNode, Tag: "!!str", Value: strings.TrimSpace(val)})
+		}
+		return m, nil
+	case Block, BlockList, BlockMap:
+		return nil, fmt.Errorf("a block is edited key by key")
+	}
+	return scalar("!!str"), nil
+}
+
+func splitLines(s string) []string {
+	var out []string
+	for _, l := range strings.FieldsFunc(s, func(r rune) bool { return r == '\n' || r == ',' }) {
+		if l = strings.TrimSpace(l); l != "" {
+			out = append(out, l)
+		}
+	}
+	return out
+}
+
+// Save checks the document with check — a loader that reads and validates
+// a file — and writes it, the old file kept beside it. Nothing is written
+// when the check fails.
+func (d *Doc) Save(check func(path string) error) (string, error) {
+	b, err := d.Bytes()
+	if err != nil {
+		return "", err
+	}
+	tmp := d.Path + ".check"
+	if err := os.WriteFile(tmp, b, 0o600); err != nil {
+		return "", err
+	}
+	defer os.Remove(tmp)
+	if check != nil {
+		if err := check(tmp); err != nil {
+			return "", err
+		}
+	}
+	mode := os.FileMode(0o644)
+	backup := ""
+	if info, err := os.Stat(d.Path); err == nil {
+		mode = info.Mode().Perm()
+		old, err := os.ReadFile(d.Path)
+		if err != nil {
+			return "", err
+		}
+		backup = fmt.Sprintf("%s.bak-%s", d.Path, time.Now().Format("20060102-150405"))
+		if err := os.WriteFile(backup, old, mode); err != nil {
+			return "", err
+		}
+	}
+	if err := os.WriteFile(d.Path+".new", b, mode); err != nil {
+		return "", err
+	}
+	if err := os.Rename(d.Path+".new", d.Path); err != nil {
+		return "", err
+	}
+	d.Changed = 0
+	return backup, nil
+}
+
+// Effective is the value a loaded configuration has at a path of koanf
+// keys — what applies, default included — as text.
+func Effective(cfg any, path []string) (string, bool) {
+	v := reflect.ValueOf(cfg)
+	for _, p := range path {
+		for v.Kind() == reflect.Pointer {
+			if v.IsNil() {
+				return "", false
+			}
+			v = v.Elem()
+		}
+		switch v.Kind() {
+		case reflect.Struct:
+			found := false
+			t := v.Type()
+			for i := 0; i < t.NumField(); i++ {
+				if tag, _, _ := strings.Cut(t.Field(i).Tag.Get("koanf"), ","); tag == p {
+					v, found = v.Field(i), true
+					break
+				}
+			}
+			if !found {
+				return "", false
+			}
+		case reflect.Slice:
+			i, err := strconv.Atoi(p)
+			if err != nil || i >= v.Len() {
+				return "", false
+			}
+			v = v.Index(i)
+		case reflect.Map:
+			mv := v.MapIndex(reflect.ValueOf(p))
+			if !mv.IsValid() {
+				return "", false
+			}
+			v = mv
+		default:
+			return "", false
+		}
+	}
+	for v.Kind() == reflect.Pointer {
+		if v.IsNil() {
+			return "", true
+		}
+		v = v.Elem()
+	}
+	switch v.Kind() {
+	case reflect.Slice:
+		var l []string
+		for i := 0; i < v.Len(); i++ {
+			l = append(l, fmt.Sprint(v.Index(i).Interface()))
+		}
+		return strings.Join(l, ", "), true
+	case reflect.Struct, reflect.Map:
+		return "", true
+	}
+	if d, ok := v.Interface().(time.Duration); ok {
+		return d.String(), true
+	}
+	return fmt.Sprint(v.Interface()), true
+}

internal/settings/settings_test.go

@@ -0,0 +1,122 @@
+package settings
+
+import (
+	"os"
+	"path/filepath"
+	"strconv"
+	"strings"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/docgen"
+)
+
+func TestEveryKeyOfBothFilesCanBeReached(t *testing.T) {
+	// Walked from each file's root: every block a key opens is found, so no
+	// setting is a dead end on the screens.
+	for _, file := range []string{"/etc/llmgw/server.yaml", "/etc/llmgw/agent.yaml"} {
+		root, err := Root(file)
+		if err != nil {
+			t.Fatal(err)
+		}
+		seen := map[string]bool{}
+		var walk func(b *docgen.Block, path string)
+		count := 0
+		walk = func(b *docgen.Block, path string) {
+			if seen[b.ID()] {
+				return
+			}
+			seen[b.ID()] = true
+			for _, k := range Keys(b) {
+				count++
+				if k.Nested != "" && k.Block == nil {
+					t.Errorf("%s: %s.%s opens %s, which is not in the reference", file, path, k.Key, k.Nested)
+				}
+				if k.Block != nil {
+					walk(k.Block, path+"."+k.Key)
+				}
+			}
+		}
+		walk(root, "")
+		if count < 20 {
+			t.Errorf("%s: only %d keys reached", file, count)
+		}
+	}
+	// The TLS settings, which the reference used to lose to a name clash.
+	root, _ := Root("server.yaml")
+	for _, k := range Keys(root) {
+		if k.Key == "agent_tls" && (k.Kind != Block || k.Block.ID() != "tlsconf.Config") {
+			t.Errorf("agent_tls is %v %v", k.Kind, k.Block)
+		}
+		if k.Key == "endpoints" && k.Kind != BlockList {
+			t.Errorf("endpoints is %v", k.Kind)
+		}
+	}
+}
+
+const sample = `# The control plane.
+listen: ":8080"
+router:
+  # How many other endpoints to try.
+  retries: 2
+`
+
+func TestAnEditKeepsTheFilesCommentsAndIsCheckedBeforeItIsWritten(t *testing.T) {
+	dir := t.TempDir()
+	path := filepath.Join(dir, "server.yaml")
+	if err := os.WriteFile(path, []byte(sample), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	d, err := Open(path)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if err := d.Set([]string{"router", "retries"}, Int, "three"); err == nil {
+		t.Error("a word was taken for a number")
+	}
+	if err := d.Set([]string{"router", "retries"}, Int, "3"); err != nil {
+		t.Fatal(err)
+	}
+	if err := d.Set([]string{"router", "cooldown"}, Duration, "45s"); err != nil {
+		t.Fatal(err)
+	}
+	check := func(p string) error { _, err := config.Load(p); return err }
+	if _, err := d.Save(check); err != nil {
+		t.Fatalf("a good edit was refused: %v", err)
+	}
+	b, _ := os.ReadFile(path)
+	for _, want := range []string{"# The control plane.", "# How many other endpoints to try.", "retries: 3", "cooldown: 45s"} {
+		if !strings.Contains(string(b), want) {
+			t.Errorf("the file lacks %q:\n%s", want, b)
+		}
+	}
+
+	// Two endpoints with one id: the loader refuses, and nothing is written.
+	for range 2 {
+		i := d.Append([]string{"endpoints"})
+		for _, kv := range [][2]string{{"id", "a"}, {"url", "http://x"}} {
+			if err := d.Set([]string{"endpoints", itoa(i), kv[0]}, Text, kv[1]); err != nil {
+				t.Fatal(err)
+			}
+		}
+		_ = d.Set([]string{"endpoints", itoa(i), "models"}, TextList, "m")
+	}
+	if _, err := d.Save(check); err == nil {
+		t.Error("an invalid configuration was saved")
+	}
+	if b2, _ := os.ReadFile(path); string(b2) != string(b) {
+		t.Error("a refused save changed the file")
+	}
+}
+
+func TestTheValueThatAppliesIsShownWhenTheFileSetsNone(t *testing.T) {
+	cfg := config.Default()
+	if v, ok := Effective(cfg, []string{"router", "retries"}); !ok || v != "2" {
+		t.Errorf("router.retries: %q %v", v, ok)
+	}
+	if v, ok := Effective(cfg, []string{"router", "cooldown"}); !ok || v != "15s" {
+		t.Errorf("router.cooldown: %q", v)
+	}
+}
+
+func itoa(i int) string { return strconv.Itoa(i) }

internal/store/controlplanes.go

@@ -0,0 +1,44 @@
+package store
+
+import (
+	"context"
+	"database/sql"
+	"time"
+)
+
+// ControlPlane is one control plane agents may dial.
+type ControlPlane struct {
+	// URL is what an agent dials: https://10.0.0.150:7443.
+	URL     string
+	Name    string
+	AddedAt time.Time
+}
+
+// ControlPlanes lists them, oldest first: the first is the one agents try
+// first.
+func (s *DB) ControlPlanes(ctx context.Context) ([]ControlPlane, error) {
+	return queryAll(ctx, s.db, `SELECT url, name, added_at FROM control_planes ORDER BY added_at, url`, nil,
+		func(rows *sql.Rows) (ControlPlane, error) {
+			var c ControlPlane
+			var at int64
+			err := rows.Scan(&c.URL, &c.Name, &at)
+			c.AddedAt = fromMillis(at)
+			return c, err
+		})
+}
+
+// PutControlPlane records one, keeping when it was first added.
+func (s *DB) PutControlPlane(ctx context.Context, c ControlPlane) error {
+	if c.AddedAt.IsZero() {
+		c.AddedAt = time.Now()
+	}
+	_, err := s.db.ExecContext(ctx, `INSERT INTO control_planes (url, name, added_at) VALUES (?, ?, ?)
+		ON CONFLICT (url) DO UPDATE SET name = excluded.name`, c.URL, c.Name, millis(c.AddedAt))
+	return err
+}
+
+// RemoveControlPlane forgets one.
+func (s *DB) RemoveControlPlane(ctx context.Context, url string) error {
+	_, err := s.db.ExecContext(ctx, `DELETE FROM control_planes WHERE url = ?`, url)
+	return err
+}

internal/store/copy.go

@@ -33,6 +33,11 @@ var copyOrder = []string{
 	// quota on a scope that is later removed is inert, not an error.
 	"capacity_quotas",
 	"usage",
+	// Copied rather than rebuilt from the rows: once request rows have a
+	// retention, the sums are the only record of what is older.
+	"usage_5m",
+	"usage_1h",
+	"usage_1d",
 	"models",
 	// No foreign key to models, deliberately: a measurement outlives the
 	// catalog entry it was taken for, and losing it when somebody removes a
@@ -46,6 +51,7 @@ var copyOrder = []string{
 	// of them at the moment of the move.
 	"cluster_secrets",
 	"node_engines",
+	"control_planes",
 }
 
 // notCopied are tables deliberately left behind.

internal/store/latency.go

@@ -0,0 +1,132 @@
+package store
+
+import (
+	"context"
+	"fmt"
+	"strings"
+	"time"
+)
+
+// Quantiles are a duration's percentiles, in milliseconds, read from the
+// histogram kept with the usage sums. Within a bucket the value is
+// interpolated; above the last bucket's edge it is that edge, and Over says
+// so. N is how many durations they are read from — zero is no reading.
+type Quantiles struct {
+	P50, P95, P99 int64
+	N             int64
+	Over          bool
+}
+
+// GroupStats is one group's requests, how they ended, and how long they took.
+type GroupStats struct {
+	ID       string
+	Requests int64
+	Errors   int64
+	// Limited is requests refused by a rate, token or budget limit.
+	Limited int64
+	Latency Quantiles
+	// TTFT is time to first token, for requests that streamed one.
+	TTFT Quantiles
+}
+
+// ErrorRate is the share of requests that failed, limits included, in percent.
+func (g GroupStats) ErrorRate() float64 {
+	if g.Requests == 0 {
+		return 0
+	}
+	return 100 * float64(g.Errors) / float64(g.Requests)
+}
+
+// StatsBy is each group's request outcomes and latency percentiles since a
+// moment, keyed by the group's id.
+//
+// Read from the hourly sums for a window of two days or more and from the
+// five-minute ones otherwise, starting at the slice the moment falls in:
+// percentiles are a shape, and a window starting up to one slice early does
+// not change one. Budgets, which must be exact, are not read this way.
+func (s *DB) StatsBy(ctx context.Context, g Grouping, since time.Time) (map[string]GroupStats, error) {
+	c, ok := groupColumns[g]
+	if !ok {
+		return nil, fmt.Errorf("usage cannot be grouped by %q", g)
+	}
+	t := tier5m
+	if time.Since(since) >= 48*time.Hour {
+		t = tier1h
+	}
+	n := len(LatencyBounds) + 1
+	var sums []string
+	for i := range n {
+		sums = append(sums, fmt.Sprintf("SUM(u.lat_%d)", i))
+	}
+	for i := range n {
+		sums = append(sums, fmt.Sprintf("SUM(u.ttft_%d)", i))
+	}
+	rows, err := s.db.QueryContext(ctx, `
+		SELECT `+c.id+`, SUM(u.requests), SUM(u.errors), SUM(u.limited), `+strings.Join(sums, ", ")+`
+		FROM `+t.table+` u`+c.joins+`
+		WHERE u.bucket >= ?
+		GROUP BY 1`, t.floor(since))
+	if err != nil {
+		return nil, err
+	}
+	defer rows.Close()
+
+	out := map[string]GroupStats{}
+	for rows.Next() {
+		var gs GroupStats
+		lat, ttft := make([]int64, n), make([]int64, n)
+		dst := []any{&gs.ID, &gs.Requests, &gs.Errors, &gs.Limited}
+		for i := range lat {
+			dst = append(dst, &lat[i])
+		}
+		for i := range ttft {
+			dst = append(dst, &ttft[i])
+		}
+		if err := rows.Scan(dst...); err != nil {
+			return nil, err
+		}
+		gs.Latency, gs.TTFT = quantiles(lat), quantiles(ttft)
+		out[gs.ID] = gs
+	}
+	return out, rows.Err()
+}
+
+// quantiles reads p50, p95 and p99 from counts per LatencyBounds bucket.
+func quantiles(counts []int64) Quantiles {
+	var q Quantiles
+	for _, c := range counts {
+		q.N += c
+	}
+	if q.N == 0 {
+		return q
+	}
+	at := func(p float64) (int64, bool) {
+		rank := p * float64(q.N)
+		var seen float64
+		for i, c := range counts {
+			if c == 0 {
+				continue
+			}
+			if seen+float64(c) >= rank {
+				if i >= len(LatencyBounds) {
+					return LatencyBounds[len(LatencyBounds)-1], true
+				}
+				lo := int64(0)
+				if i > 0 {
+					lo = LatencyBounds[i-1]
+				}
+				hi := LatencyBounds[i]
+				frac := (rank - seen) / float64(c)
+				return lo + int64(frac*float64(hi-lo)), false
+			}
+			seen += float64(c)
+		}
+		return LatencyBounds[len(LatencyBounds)-1], true
+	}
+	var over bool
+	q.P50, _ = at(0.50)
+	q.P95, _ = at(0.95)
+	q.P99, over = at(0.99)
+	q.Over = over
+	return q
+}

internal/store/latency_test.go

@@ -0,0 +1,61 @@
+package store
+
+import (
+	"context"
+	"testing"
+	"time"
+)
+
+func TestQuantilesAreReadFromTheHistogram(t *testing.T) {
+	// 100 durations: 90 at <=100 ms, 8 in (1000, 2500], 2 above two minutes.
+	counts := make([]int64, len(LatencyBounds)+1)
+	counts[0], counts[4], counts[len(counts)-1] = 90, 8, 2
+	q := quantiles(counts)
+	if q.N != 100 || q.P50 > 100 || q.P95 <= 1000 || q.P95 > 2500 || !q.Over {
+		t.Errorf("%+v: want p50 within 100 ms, p95 in (1000, 2500], p99 past the last edge", q)
+	}
+	if (quantiles(make([]int64, len(LatencyBounds)+1)) != Quantiles{}) {
+		t.Error("no durations read as some")
+	}
+}
+
+func TestStatsSayHowEachGroupFaredAndHowLongItTook(t *testing.T) {
+	onBoth(t, func(t *testing.T, s *DB) {
+		ctx := context.Background()
+		at := slice()
+		var batch []Usage
+		for i := range 20 {
+			batch = append(batch, Usage{At: at, KeyID: "k", UserID: "ayse", ProjectID: "p", Model: "big",
+				ServedModel: "big", Status: 200, LatencyMs: 4000, TTFTMs: int64(200 + i)})
+		}
+		batch = append(batch,
+			Usage{At: at, KeyID: "k", UserID: "ayse", ProjectID: "p", Model: "big", ServedModel: "big", Status: 502, LatencyMs: 50},
+			Usage{At: at, KeyID: "k2", UserID: "bora", ProjectID: "p", Model: "small", ServedModel: "small", Status: 429},
+		)
+		if err := s.WriteUsage(ctx, batch); err != nil {
+			t.Fatal(err)
+		}
+		byUser, err := s.StatsBy(ctx, GroupByUser, at.Add(-time.Hour))
+		if err != nil {
+			t.Fatal(err)
+		}
+		a, b := byUser["ayse"], byUser["bora"]
+		if a.Requests != 21 || a.Errors != 1 || a.Limited != 0 || b.Limited != 1 || b.Errors != 1 {
+			t.Errorf("outcomes: ayse %+v, bora %+v", a, b)
+		}
+		if a.TTFT.N != 20 || a.TTFT.P95 <= 100 || a.TTFT.P95 > 250 || a.Latency.P95 <= 2500 || a.Latency.P95 > 5000 {
+			t.Errorf("ayse's first token %+v and latency %+v", a.TTFT, a.Latency)
+		}
+		byModel, err := s.StatsBy(ctx, GroupByModel, at.Add(-time.Hour))
+		if err != nil {
+			t.Fatal(err)
+		}
+		if byModel["big"].Requests != 21 || byModel["small"].Requests != 1 {
+			t.Errorf("by model: %+v", byModel)
+		}
+		series, err := s.UsageBucketsBy(ctx, GroupByModel, at.Add(-time.Hour), time.Hour)
+		if err != nil || len(series) != 2 {
+			t.Fatalf("usage over time by model: %v, %+v", err, series)
+		}
+	})
+}

internal/store/migrations/postgres/0015_usage_rollup.sql

@@ -0,0 +1,208 @@
+-- Ported from the SQLite migration of the same number.
+-- Usage summed as it is written, per key, person, project and model: into
+-- five-minute slices (usage_5m), hours (usage_1h) and days (usage_1d).
+--
+-- Every chart and total read the raw table, one row per request, and summed it
+-- again on every page view: fine for thousands of requests a day, and a full
+-- scan of tens of millions on a cluster in real use. The sums are kept in the
+-- transaction that writes the rows, so they never disagree with them; the raw
+-- table stays for following one request, and can be given a retention of its
+-- own (auth.usage_retention).
+--
+-- Three widths because how well a slice compresses depends on traffic: many
+-- keys each sending a little leave a five-minute slice with nearly a row per
+-- request, which an hour or a day folds much further. A chart reads the
+-- coarsest width its slices are made of; a total since a moment reads the
+-- rows up to the next five-minute slice, those slices up to the hour, hours
+-- up to the day and days from there — exact, and a month in about thirty
+-- rows per key and model.
+--
+-- Latency and time to first token are kept as histograms, a count per fixed
+-- bucket (ms: <=100, 250, 500, 1000, 2500, 5000, 10000, 30000, 60000, 120000,
+-- and above), which is what a percentile per team or model is read from after
+-- the rows are gone; a mean cannot do that. ttft_* counts only requests whose
+-- first token was measured. limited counts requests refused with 429 — a rate,
+-- token or budget limit — which are recorded with no tokens and no cost.
+
+CREATE TABLE usage_5m (
+  bucket            BIGINT NOT NULL,
+  key_id            TEXT NOT NULL,
+  user_id           TEXT NOT NULL,
+  project_id        TEXT NOT NULL,
+  model             TEXT NOT NULL,
+  served_model      TEXT NOT NULL,
+  requests          BIGINT NOT NULL DEFAULT 0,
+  errors            BIGINT NOT NULL DEFAULT 0,
+  limited           BIGINT NOT NULL DEFAULT 0,
+  prompt_tokens     BIGINT NOT NULL DEFAULT 0,
+  completion_tokens BIGINT NOT NULL DEFAULT 0,
+  cost_micros       BIGINT NOT NULL DEFAULT 0,
+  latency_ms_sum    BIGINT NOT NULL DEFAULT 0,
+  queue_ms_sum      BIGINT NOT NULL DEFAULT 0,
+  ttft_ms_sum       BIGINT NOT NULL DEFAULT 0,
+  ttft_n            BIGINT NOT NULL DEFAULT 0,
+  last_ts           BIGINT NOT NULL DEFAULT 0,
+  lat_0 BIGINT NOT NULL DEFAULT 0,
+  lat_1 BIGINT NOT NULL DEFAULT 0,
+  lat_2 BIGINT NOT NULL DEFAULT 0,
+  lat_3 BIGINT NOT NULL DEFAULT 0,
+  lat_4 BIGINT NOT NULL DEFAULT 0,
+  lat_5 BIGINT NOT NULL DEFAULT 0,
+  lat_6 BIGINT NOT NULL DEFAULT 0,
+  lat_7 BIGINT NOT NULL DEFAULT 0,
+  lat_8 BIGINT NOT NULL DEFAULT 0,
+  lat_9 BIGINT NOT NULL DEFAULT 0,
+  lat_10 BIGINT NOT NULL DEFAULT 0,
+  ttft_0 BIGINT NOT NULL DEFAULT 0,
+  ttft_1 BIGINT NOT NULL DEFAULT 0,
+  ttft_2 BIGINT NOT NULL DEFAULT 0,
+  ttft_3 BIGINT NOT NULL DEFAULT 0,
+  ttft_4 BIGINT NOT NULL DEFAULT 0,
+  ttft_5 BIGINT NOT NULL DEFAULT 0,
+  ttft_6 BIGINT NOT NULL DEFAULT 0,
+  ttft_7 BIGINT NOT NULL DEFAULT 0,
+  ttft_8 BIGINT NOT NULL DEFAULT 0,
+  ttft_9 BIGINT NOT NULL DEFAULT 0,
+  ttft_10 BIGINT NOT NULL DEFAULT 0,
+  PRIMARY KEY (bucket, key_id, user_id, project_id, model, served_model)
+);
+CREATE INDEX usage_5m_by_bucket ON usage_5m(bucket);
+CREATE INDEX usage_5m_by_project ON usage_5m(project_id, bucket);
+CREATE INDEX usage_5m_by_user ON usage_5m(user_id, bucket);
+CREATE INDEX usage_5m_by_key ON usage_5m(key_id, bucket);
+
+INSERT INTO usage_5m (bucket, key_id, user_id, project_id, model, served_model,
+  requests, errors, limited, prompt_tokens, completion_tokens, cost_micros,
+  latency_ms_sum, queue_ms_sum, ttft_ms_sum, ttft_n, last_ts,
+  lat_0, lat_1, lat_2, lat_3, lat_4, lat_5, lat_6, lat_7, lat_8, lat_9, lat_10,
+  ttft_0, ttft_1, ttft_2, ttft_3, ttft_4, ttft_5, ttft_6, ttft_7, ttft_8, ttft_9, ttft_10)
+SELECT (ts / 300000) * 300000, key_id, COALESCE(user_id, ''), project_id, model, served_model,
+  COUNT(*), SUM(CASE WHEN status >= 400 THEN 1 ELSE 0 END), SUM(CASE WHEN status = 429 THEN 1 ELSE 0 END),
+  SUM(prompt_tokens), SUM(completion_tokens), SUM(cost_micros),
+  SUM(latency_ms), SUM(queue_ms), SUM(ttft_ms), SUM(CASE WHEN ttft_ms > 0 THEN 1 ELSE 0 END), MAX(ts),
+  SUM(CASE WHEN latency_ms <= 100 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 100 AND latency_ms <= 250 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 250 AND latency_ms <= 500 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 500 AND latency_ms <= 1000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 1000 AND latency_ms <= 2500 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 2500 AND latency_ms <= 5000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 5000 AND latency_ms <= 10000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 10000 AND latency_ms <= 30000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 30000 AND latency_ms <= 60000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 60000 AND latency_ms <= 120000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 120000 THEN 1 ELSE 0 END),
+  SUM(CASE WHEN ttft_ms > 0 AND ttft_ms <= 100 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 100 AND ttft_ms <= 250 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 250 AND ttft_ms <= 500 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 500 AND ttft_ms <= 1000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 1000 AND ttft_ms <= 2500 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 2500 AND ttft_ms <= 5000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 5000 AND ttft_ms <= 10000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 10000 AND ttft_ms <= 30000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 30000 AND ttft_ms <= 60000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 60000 AND ttft_ms <= 120000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 120000 THEN 1 ELSE 0 END)
+FROM usage
+GROUP BY 1, 2, 3, 4, 5, 6;
+
+CREATE TABLE usage_1h (
+  bucket            BIGINT NOT NULL,
+  key_id            TEXT NOT NULL,
+  user_id           TEXT NOT NULL,
+  project_id        TEXT NOT NULL,
+  model             TEXT NOT NULL,
+  served_model      TEXT NOT NULL,
+  requests          BIGINT NOT NULL DEFAULT 0,
+  errors            BIGINT NOT NULL DEFAULT 0,
+  limited           BIGINT NOT NULL DEFAULT 0,
+  prompt_tokens     BIGINT NOT NULL DEFAULT 0,
+  completion_tokens BIGINT NOT NULL DEFAULT 0,
+  cost_micros       BIGINT NOT NULL DEFAULT 0,
+  latency_ms_sum    BIGINT NOT NULL DEFAULT 0,
+  queue_ms_sum      BIGINT NOT NULL DEFAULT 0,
+  ttft_ms_sum       BIGINT NOT NULL DEFAULT 0,
+  ttft_n            BIGINT NOT NULL DEFAULT 0,
+  last_ts           BIGINT NOT NULL DEFAULT 0,
+  lat_0 BIGINT NOT NULL DEFAULT 0,
+  lat_1 BIGINT NOT NULL DEFAULT 0,
+  lat_2 BIGINT NOT NULL DEFAULT 0,
+  lat_3 BIGINT NOT NULL DEFAULT 0,
+  lat_4 BIGINT NOT NULL DEFAULT 0,
+  lat_5 BIGINT NOT NULL DEFAULT 0,
+  lat_6 BIGINT NOT NULL DEFAULT 0,
+  lat_7 BIGINT NOT NULL DEFAULT 0,
+  lat_8 BIGINT NOT NULL DEFAULT 0,
+  lat_9 BIGINT NOT NULL DEFAULT 0,
+  lat_10 BIGINT NOT NULL DEFAULT 0,
+  ttft_0 BIGINT NOT NULL DEFAULT 0,
+  ttft_1 BIGINT NOT NULL DEFAULT 0,
+  ttft_2 BIGINT NOT NULL DEFAULT 0,
+  ttft_3 BIGINT NOT NULL DEFAULT 0,
+  ttft_4 BIGINT NOT NULL DEFAULT 0,
+  ttft_5 BIGINT NOT NULL DEFAULT 0,
+  ttft_6 BIGINT NOT NULL DEFAULT 0,
+  ttft_7 BIGINT NOT NULL DEFAULT 0,
+  ttft_8 BIGINT NOT NULL DEFAULT 0,
+  ttft_9 BIGINT NOT NULL DEFAULT 0,
+  ttft_10 BIGINT NOT NULL DEFAULT 0,
+  PRIMARY KEY (bucket, key_id, user_id, project_id, model, served_model)
+);
+CREATE INDEX usage_1h_by_bucket ON usage_1h(bucket);
+CREATE INDEX usage_1h_by_project ON usage_1h(project_id, bucket);
+CREATE INDEX usage_1h_by_user ON usage_1h(user_id, bucket);
+CREATE INDEX usage_1h_by_key ON usage_1h(key_id, bucket);
+
+INSERT INTO usage_1h (bucket, key_id, user_id, project_id, model, served_model,
+  requests, errors, limited, prompt_tokens, completion_tokens, cost_micros,
+  latency_ms_sum, queue_ms_sum, ttft_ms_sum, ttft_n, last_ts,
+  lat_0, lat_1, lat_2, lat_3, lat_4, lat_5, lat_6, lat_7, lat_8, lat_9, lat_10,
+  ttft_0, ttft_1, ttft_2, ttft_3, ttft_4, ttft_5, ttft_6, ttft_7, ttft_8, ttft_9, ttft_10)
+SELECT (ts / 3600000) * 3600000, key_id, COALESCE(user_id, ''), project_id, model, served_model,
+  COUNT(*), SUM(CASE WHEN status >= 400 THEN 1 ELSE 0 END), SUM(CASE WHEN status = 429 THEN 1 ELSE 0 END),
+  SUM(prompt_tokens), SUM(completion_tokens), SUM(cost_micros),
+  SUM(latency_ms), SUM(queue_ms), SUM(ttft_ms), SUM(CASE WHEN ttft_ms > 0 THEN 1 ELSE 0 END), MAX(ts),
+  SUM(CASE WHEN latency_ms <= 100 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 100 AND latency_ms <= 250 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 250 AND latency_ms <= 500 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 500 AND latency_ms <= 1000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 1000 AND latency_ms <= 2500 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 2500 AND latency_ms <= 5000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 5000 AND latency_ms <= 10000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 10000 AND latency_ms <= 30000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 30000 AND latency_ms <= 60000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 60000 AND latency_ms <= 120000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 120000 THEN 1 ELSE 0 END),
+  SUM(CASE WHEN ttft_ms > 0 AND ttft_ms <= 100 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 100 AND ttft_ms <= 250 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 250 AND ttft_ms <= 500 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 500 AND ttft_ms <= 1000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 1000 AND ttft_ms <= 2500 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 2500 AND ttft_ms <= 5000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 5000 AND ttft_ms <= 10000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 10000 AND ttft_ms <= 30000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 30000 AND ttft_ms <= 60000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 60000 AND ttft_ms <= 120000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 120000 THEN 1 ELSE 0 END)
+FROM usage
+GROUP BY 1, 2, 3, 4, 5, 6;
+
+CREATE TABLE usage_1d (
+  bucket            BIGINT NOT NULL,
+  key_id            TEXT NOT NULL,
+  user_id           TEXT NOT NULL,
+  project_id        TEXT NOT NULL,
+  model             TEXT NOT NULL,
+  served_model      TEXT NOT NULL,
+  requests          BIGINT NOT NULL DEFAULT 0,
+  errors            BIGINT NOT NULL DEFAULT 0,
+  limited           BIGINT NOT NULL DEFAULT 0,
+  prompt_tokens     BIGINT NOT NULL DEFAULT 0,
+  completion_tokens BIGINT NOT NULL DEFAULT 0,
+  cost_micros       BIGINT NOT NULL DEFAULT 0,
+  latency_ms_sum    BIGINT NOT NULL DEFAULT 0,
+  queue_ms_sum      BIGINT NOT NULL DEFAULT 0,
+  ttft_ms_sum       BIGINT NOT NULL DEFAULT 0,
+  ttft_n            BIGINT NOT NULL DEFAULT 0,
+  last_ts           BIGINT NOT NULL DEFAULT 0,
+  lat_0 BIGINT NOT NULL DEFAULT 0,
+  lat_1 BIGINT NOT NULL DEFAULT 0,
+  lat_2 BIGINT NOT NULL DEFAULT 0,
+  lat_3 BIGINT NOT NULL DEFAULT 0,
+  lat_4 BIGINT NOT NULL DEFAULT 0,
+  lat_5 BIGINT NOT NULL DEFAULT 0,
+  lat_6 BIGINT NOT NULL DEFAULT 0,
+  lat_7 BIGINT NOT NULL DEFAULT 0,
+  lat_8 BIGINT NOT NULL DEFAULT 0,
+  lat_9 BIGINT NOT NULL DEFAULT 0,
+  lat_10 BIGINT NOT NULL DEFAULT 0,
+  ttft_0 BIGINT NOT NULL DEFAULT 0,
+  ttft_1 BIGINT NOT NULL DEFAULT 0,
+  ttft_2 BIGINT NOT NULL DEFAULT 0,
+  ttft_3 BIGINT NOT NULL DEFAULT 0,
+  ttft_4 BIGINT NOT NULL DEFAULT 0,
+  ttft_5 BIGINT NOT NULL DEFAULT 0,
+  ttft_6 BIGINT NOT NULL DEFAULT 0,
+  ttft_7 BIGINT NOT NULL DEFAULT 0,
+  ttft_8 BIGINT NOT NULL DEFAULT 0,
+  ttft_9 BIGINT NOT NULL DEFAULT 0,
+  ttft_10 BIGINT NOT NULL DEFAULT 0,
+  PRIMARY KEY (bucket, key_id, user_id, project_id, model, served_model)
+);
+CREATE INDEX usage_1d_by_bucket ON usage_1d(bucket);
+CREATE INDEX usage_1d_by_project ON usage_1d(project_id, bucket);
+CREATE INDEX usage_1d_by_user ON usage_1d(user_id, bucket);
+CREATE INDEX usage_1d_by_key ON usage_1d(key_id, bucket);
+
+INSERT INTO usage_1d (bucket, key_id, user_id, project_id, model, served_model,
+  requests, errors, limited, prompt_tokens, completion_tokens, cost_micros,
+  latency_ms_sum, queue_ms_sum, ttft_ms_sum, ttft_n, last_ts,
+  lat_0, lat_1, lat_2, lat_3, lat_4, lat_5, lat_6, lat_7, lat_8, lat_9, lat_10,
+  ttft_0, ttft_1, ttft_2, ttft_3, ttft_4, ttft_5, ttft_6, ttft_7, ttft_8, ttft_9, ttft_10)
+SELECT (ts / 86400000) * 86400000, key_id, COALESCE(user_id, ''), project_id, model, served_model,
+  COUNT(*), SUM(CASE WHEN status >= 400 THEN 1 ELSE 0 END), SUM(CASE WHEN status = 429 THEN 1 ELSE 0 END),
+  SUM(prompt_tokens), SUM(completion_tokens), SUM(cost_micros),
+  SUM(latency_ms), SUM(queue_ms), SUM(ttft_ms), SUM(CASE WHEN ttft_ms > 0 THEN 1 ELSE 0 END), MAX(ts),
+  SUM(CASE WHEN latency_ms <= 100 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 100 AND latency_ms <= 250 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 250 AND latency_ms <= 500 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 500 AND latency_ms <= 1000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 1000 AND latency_ms <= 2500 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 2500 AND latency_ms <= 5000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 5000 AND latency_ms <= 10000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 10000 AND latency_ms <= 30000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 30000 AND latency_ms <= 60000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 60000 AND latency_ms <= 120000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 120000 THEN 1 ELSE 0 END),
+  SUM(CASE WHEN ttft_ms > 0 AND ttft_ms <= 100 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 100 AND ttft_ms <= 250 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 250 AND ttft_ms <= 500 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 500 AND ttft_ms <= 1000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 1000 AND ttft_ms <= 2500 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 2500 AND ttft_ms <= 5000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 5000 AND ttft_ms <= 10000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 10000 AND ttft_ms <= 30000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 30000 AND ttft_ms <= 60000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 60000 AND ttft_ms <= 120000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 120000 THEN 1 ELSE 0 END)
+FROM usage
+GROUP BY 1, 2, 3, 4, 5, 6;

internal/store/migrations/postgres/0016_control_planes.sql

@@ -0,0 +1,10 @@
+-- Ported from the SQLite migration of the same number.
+-- The control planes agents may dial, by the address they dial: what the
+-- leader tells every agent, so a control plane added after the nodes were
+-- onboarded is one they fail over to without their configuration being
+-- edited. Written by `llmgw server add` and by init for the first one.
+CREATE TABLE control_planes (
+  url      TEXT PRIMARY KEY,
+  name     TEXT NOT NULL,
+  added_at BIGINT NOT NULL
+);

internal/store/migrations/sqlite/0015_usage_rollup.sql

@@ -0,0 +1,207 @@
+-- Usage summed as it is written, per key, person, project and model: into
+-- five-minute slices (usage_5m), hours (usage_1h) and days (usage_1d).
+--
+-- Every chart and total read the raw table, one row per request, and summed it
+-- again on every page view: fine for thousands of requests a day, and a full
+-- scan of tens of millions on a cluster in real use. The sums are kept in the
+-- transaction that writes the rows, so they never disagree with them; the raw
+-- table stays for following one request, and can be given a retention of its
+-- own (auth.usage_retention).
+--
+-- Three widths because how well a slice compresses depends on traffic: many
+-- keys each sending a little leave a five-minute slice with nearly a row per
+-- request, which an hour or a day folds much further. A chart reads the
+-- coarsest width its slices are made of; a total since a moment reads the
+-- rows up to the next five-minute slice, those slices up to the hour, hours
+-- up to the day and days from there — exact, and a month in about thirty
+-- rows per key and model.
+--
+-- Latency and time to first token are kept as histograms, a count per fixed
+-- bucket (ms: <=100, 250, 500, 1000, 2500, 5000, 10000, 30000, 60000, 120000,
+-- and above), which is what a percentile per team or model is read from after
+-- the rows are gone; a mean cannot do that. ttft_* counts only requests whose
+-- first token was measured. limited counts requests refused with 429 — a rate,
+-- token or budget limit — which are recorded with no tokens and no cost.
+
+CREATE TABLE usage_5m (
+  bucket            INTEGER NOT NULL,
+  key_id            TEXT NOT NULL,
+  user_id           TEXT NOT NULL,
+  project_id        TEXT NOT NULL,
+  model             TEXT NOT NULL,
+  served_model      TEXT NOT NULL,
+  requests          INTEGER NOT NULL DEFAULT 0,
+  errors            INTEGER NOT NULL DEFAULT 0,
+  limited           INTEGER NOT NULL DEFAULT 0,
+  prompt_tokens     INTEGER NOT NULL DEFAULT 0,
+  completion_tokens INTEGER NOT NULL DEFAULT 0,
+  cost_micros       INTEGER NOT NULL DEFAULT 0,
+  latency_ms_sum    INTEGER NOT NULL DEFAULT 0,
+  queue_ms_sum      INTEGER NOT NULL DEFAULT 0,
+  ttft_ms_sum       INTEGER NOT NULL DEFAULT 0,
+  ttft_n            INTEGER NOT NULL DEFAULT 0,
+  last_ts           INTEGER NOT NULL DEFAULT 0,
+  lat_0 INTEGER NOT NULL DEFAULT 0,
+  lat_1 INTEGER NOT NULL DEFAULT 0,
+  lat_2 INTEGER NOT NULL DEFAULT 0,
+  lat_3 INTEGER NOT NULL DEFAULT 0,
+  lat_4 INTEGER NOT NULL DEFAULT 0,
+  lat_5 INTEGER NOT NULL DEFAULT 0,
+  lat_6 INTEGER NOT NULL DEFAULT 0,
+  lat_7 INTEGER NOT NULL DEFAULT 0,
+  lat_8 INTEGER NOT NULL DEFAULT 0,
+  lat_9 INTEGER NOT NULL DEFAULT 0,
+  lat_10 INTEGER NOT NULL DEFAULT 0,
+  ttft_0 INTEGER NOT NULL DEFAULT 0,
+  ttft_1 INTEGER NOT NULL DEFAULT 0,
+  ttft_2 INTEGER NOT NULL DEFAULT 0,
+  ttft_3 INTEGER NOT NULL DEFAULT 0,
+  ttft_4 INTEGER NOT NULL DEFAULT 0,
+  ttft_5 INTEGER NOT NULL DEFAULT 0,
+  ttft_6 INTEGER NOT NULL DEFAULT 0,
+  ttft_7 INTEGER NOT NULL DEFAULT 0,
+  ttft_8 INTEGER NOT NULL DEFAULT 0,
+  ttft_9 INTEGER NOT NULL DEFAULT 0,
+  ttft_10 INTEGER NOT NULL DEFAULT 0,
+  PRIMARY KEY (bucket, key_id, user_id, project_id, model, served_model)
+);
+CREATE INDEX usage_5m_by_bucket ON usage_5m(bucket);
+CREATE INDEX usage_5m_by_project ON usage_5m(project_id, bucket);
+CREATE INDEX usage_5m_by_user ON usage_5m(user_id, bucket);
+CREATE INDEX usage_5m_by_key ON usage_5m(key_id, bucket);
+
+INSERT INTO usage_5m (bucket, key_id, user_id, project_id, model, served_model,
+  requests, errors, limited, prompt_tokens, completion_tokens, cost_micros,
+  latency_ms_sum, queue_ms_sum, ttft_ms_sum, ttft_n, last_ts,
+  lat_0, lat_1, lat_2, lat_3, lat_4, lat_5, lat_6, lat_7, lat_8, lat_9, lat_10,
+  ttft_0, ttft_1, ttft_2, ttft_3, ttft_4, ttft_5, ttft_6, ttft_7, ttft_8, ttft_9, ttft_10)
+SELECT (ts / 300000) * 300000, key_id, COALESCE(user_id, ''), project_id, model, served_model,
+  COUNT(*), SUM(CASE WHEN status >= 400 THEN 1 ELSE 0 END), SUM(CASE WHEN status = 429 THEN 1 ELSE 0 END),
+  SUM(prompt_tokens), SUM(completion_tokens), SUM(cost_micros),
+  SUM(latency_ms), SUM(queue_ms), SUM(ttft_ms), SUM(CASE WHEN ttft_ms > 0 THEN 1 ELSE 0 END), MAX(ts),
+  SUM(CASE WHEN latency_ms <= 100 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 100 AND latency_ms <= 250 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 250 AND latency_ms <= 500 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 500 AND latency_ms <= 1000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 1000 AND latency_ms <= 2500 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 2500 AND latency_ms <= 5000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 5000 AND latency_ms <= 10000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 10000 AND latency_ms <= 30000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 30000 AND latency_ms <= 60000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 60000 AND latency_ms <= 120000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 120000 THEN 1 ELSE 0 END),
+  SUM(CASE WHEN ttft_ms > 0 AND ttft_ms <= 100 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 100 AND ttft_ms <= 250 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 250 AND ttft_ms <= 500 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 500 AND ttft_ms <= 1000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 1000 AND ttft_ms <= 2500 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 2500 AND ttft_ms <= 5000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 5000 AND ttft_ms <= 10000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 10000 AND ttft_ms <= 30000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 30000 AND ttft_ms <= 60000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 60000 AND ttft_ms <= 120000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 120000 THEN 1 ELSE 0 END)
+FROM usage
+GROUP BY 1, 2, 3, 4, 5, 6;
+
+CREATE TABLE usage_1h (
+  bucket            INTEGER NOT NULL,
+  key_id            TEXT NOT NULL,
+  user_id           TEXT NOT NULL,
+  project_id        TEXT NOT NULL,
+  model             TEXT NOT NULL,
+  served_model      TEXT NOT NULL,
+  requests          INTEGER NOT NULL DEFAULT 0,
+  errors            INTEGER NOT NULL DEFAULT 0,
+  limited           INTEGER NOT NULL DEFAULT 0,
+  prompt_tokens     INTEGER NOT NULL DEFAULT 0,
+  completion_tokens INTEGER NOT NULL DEFAULT 0,
+  cost_micros       INTEGER NOT NULL DEFAULT 0,
+  latency_ms_sum    INTEGER NOT NULL DEFAULT 0,
+  queue_ms_sum      INTEGER NOT NULL DEFAULT 0,
+  ttft_ms_sum       INTEGER NOT NULL DEFAULT 0,
+  ttft_n            INTEGER NOT NULL DEFAULT 0,
+  last_ts           INTEGER NOT NULL DEFAULT 0,
+  lat_0 INTEGER NOT NULL DEFAULT 0,
+  lat_1 INTEGER NOT NULL DEFAULT 0,
+  lat_2 INTEGER NOT NULL DEFAULT 0,
+  lat_3 INTEGER NOT NULL DEFAULT 0,
+  lat_4 INTEGER NOT NULL DEFAULT 0,
+  lat_5 INTEGER NOT NULL DEFAULT 0,
+  lat_6 INTEGER NOT NULL DEFAULT 0,
+  lat_7 INTEGER NOT NULL DEFAULT 0,
+  lat_8 INTEGER NOT NULL DEFAULT 0,
+  lat_9 INTEGER NOT NULL DEFAULT 0,
+  lat_10 INTEGER NOT NULL DEFAULT 0,
+  ttft_0 INTEGER NOT NULL DEFAULT 0,
+  ttft_1 INTEGER NOT NULL DEFAULT 0,
+  ttft_2 INTEGER NOT NULL DEFAULT 0,
+  ttft_3 INTEGER NOT NULL DEFAULT 0,
+  ttft_4 INTEGER NOT NULL DEFAULT 0,
+  ttft_5 INTEGER NOT NULL DEFAULT 0,
+  ttft_6 INTEGER NOT NULL DEFAULT 0,
+  ttft_7 INTEGER NOT NULL DEFAULT 0,
+  ttft_8 INTEGER NOT NULL DEFAULT 0,
+  ttft_9 INTEGER NOT NULL DEFAULT 0,
+  ttft_10 INTEGER NOT NULL DEFAULT 0,
+  PRIMARY KEY (bucket, key_id, user_id, project_id, model, served_model)
+);
+CREATE INDEX usage_1h_by_bucket ON usage_1h(bucket);
+CREATE INDEX usage_1h_by_project ON usage_1h(project_id, bucket);
+CREATE INDEX usage_1h_by_user ON usage_1h(user_id, bucket);
+CREATE INDEX usage_1h_by_key ON usage_1h(key_id, bucket);
+
+INSERT INTO usage_1h (bucket, key_id, user_id, project_id, model, served_model,
+  requests, errors, limited, prompt_tokens, completion_tokens, cost_micros,
+  latency_ms_sum, queue_ms_sum, ttft_ms_sum, ttft_n, last_ts,
+  lat_0, lat_1, lat_2, lat_3, lat_4, lat_5, lat_6, lat_7, lat_8, lat_9, lat_10,
+  ttft_0, ttft_1, ttft_2, ttft_3, ttft_4, ttft_5, ttft_6, ttft_7, ttft_8, ttft_9, ttft_10)
+SELECT (ts / 3600000) * 3600000, key_id, COALESCE(user_id, ''), project_id, model, served_model,
+  COUNT(*), SUM(CASE WHEN status >= 400 THEN 1 ELSE 0 END), SUM(CASE WHEN status = 429 THEN 1 ELSE 0 END),
+  SUM(prompt_tokens), SUM(completion_tokens), SUM(cost_micros),
+  SUM(latency_ms), SUM(queue_ms), SUM(ttft_ms), SUM(CASE WHEN ttft_ms > 0 THEN 1 ELSE 0 END), MAX(ts),
+  SUM(CASE WHEN latency_ms <= 100 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 100 AND latency_ms <= 250 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 250 AND latency_ms <= 500 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 500 AND latency_ms <= 1000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 1000 AND latency_ms <= 2500 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 2500 AND latency_ms <= 5000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 5000 AND latency_ms <= 10000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 10000 AND latency_ms <= 30000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 30000 AND latency_ms <= 60000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 60000 AND latency_ms <= 120000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 120000 THEN 1 ELSE 0 END),
+  SUM(CASE WHEN ttft_ms > 0 AND ttft_ms <= 100 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 100 AND ttft_ms <= 250 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 250 AND ttft_ms <= 500 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 500 AND ttft_ms <= 1000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 1000 AND ttft_ms <= 2500 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 2500 AND ttft_ms <= 5000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 5000 AND ttft_ms <= 10000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 10000 AND ttft_ms <= 30000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 30000 AND ttft_ms <= 60000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 60000 AND ttft_ms <= 120000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 120000 THEN 1 ELSE 0 END)
+FROM usage
+GROUP BY 1, 2, 3, 4, 5, 6;
+
+CREATE TABLE usage_1d (
+  bucket            INTEGER NOT NULL,
+  key_id            TEXT NOT NULL,
+  user_id           TEXT NOT NULL,
+  project_id        TEXT NOT NULL,
+  model             TEXT NOT NULL,
+  served_model      TEXT NOT NULL,
+  requests          INTEGER NOT NULL DEFAULT 0,
+  errors            INTEGER NOT NULL DEFAULT 0,
+  limited           INTEGER NOT NULL DEFAULT 0,
+  prompt_tokens     INTEGER NOT NULL DEFAULT 0,
+  completion_tokens INTEGER NOT NULL DEFAULT 0,
+  cost_micros       INTEGER NOT NULL DEFAULT 0,
+  latency_ms_sum    INTEGER NOT NULL DEFAULT 0,
+  queue_ms_sum      INTEGER NOT NULL DEFAULT 0,
+  ttft_ms_sum       INTEGER NOT NULL DEFAULT 0,
+  ttft_n            INTEGER NOT NULL DEFAULT 0,
+  last_ts           INTEGER NOT NULL DEFAULT 0,
+  lat_0 INTEGER NOT NULL DEFAULT 0,
+  lat_1 INTEGER NOT NULL DEFAULT 0,
+  lat_2 INTEGER NOT NULL DEFAULT 0,
+  lat_3 INTEGER NOT NULL DEFAULT 0,
+  lat_4 INTEGER NOT NULL DEFAULT 0,
+  lat_5 INTEGER NOT NULL DEFAULT 0,
+  lat_6 INTEGER NOT NULL DEFAULT 0,
+  lat_7 INTEGER NOT NULL DEFAULT 0,
+  lat_8 INTEGER NOT NULL DEFAULT 0,
+  lat_9 INTEGER NOT NULL DEFAULT 0,
+  lat_10 INTEGER NOT NULL DEFAULT 0,
+  ttft_0 INTEGER NOT NULL DEFAULT 0,
+  ttft_1 INTEGER NOT NULL DEFAULT 0,
+  ttft_2 INTEGER NOT NULL DEFAULT 0,
+  ttft_3 INTEGER NOT NULL DEFAULT 0,
+  ttft_4 INTEGER NOT NULL DEFAULT 0,
+  ttft_5 INTEGER NOT NULL DEFAULT 0,
+  ttft_6 INTEGER NOT NULL DEFAULT 0,
+  ttft_7 INTEGER NOT NULL DEFAULT 0,
+  ttft_8 INTEGER NOT NULL DEFAULT 0,
+  ttft_9 INTEGER NOT NULL DEFAULT 0,
+  ttft_10 INTEGER NOT NULL DEFAULT 0,
+  PRIMARY KEY (bucket, key_id, user_id, project_id, model, served_model)
+);
+CREATE INDEX usage_1d_by_bucket ON usage_1d(bucket);
+CREATE INDEX usage_1d_by_project ON usage_1d(project_id, bucket);
+CREATE INDEX usage_1d_by_user ON usage_1d(user_id, bucket);
+CREATE INDEX usage_1d_by_key ON usage_1d(key_id, bucket);
+
+INSERT INTO usage_1d (bucket, key_id, user_id, project_id, model, served_model,
+  requests, errors, limited, prompt_tokens, completion_tokens, cost_micros,
+  latency_ms_sum, queue_ms_sum, ttft_ms_sum, ttft_n, last_ts,
+  lat_0, lat_1, lat_2, lat_3, lat_4, lat_5, lat_6, lat_7, lat_8, lat_9, lat_10,
+  ttft_0, ttft_1, ttft_2, ttft_3, ttft_4, ttft_5, ttft_6, ttft_7, ttft_8, ttft_9, ttft_10)
+SELECT (ts / 86400000) * 86400000, key_id, COALESCE(user_id, ''), project_id, model, served_model,
+  COUNT(*), SUM(CASE WHEN status >= 400 THEN 1 ELSE 0 END), SUM(CASE WHEN status = 429 THEN 1 ELSE 0 END),
+  SUM(prompt_tokens), SUM(completion_tokens), SUM(cost_micros),
+  SUM(latency_ms), SUM(queue_ms), SUM(ttft_ms), SUM(CASE WHEN ttft_ms > 0 THEN 1 ELSE 0 END), MAX(ts),
+  SUM(CASE WHEN latency_ms <= 100 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 100 AND latency_ms <= 250 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 250 AND latency_ms <= 500 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 500 AND latency_ms <= 1000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 1000 AND latency_ms <= 2500 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 2500 AND latency_ms <= 5000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 5000 AND latency_ms <= 10000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 10000 AND latency_ms <= 30000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 30000 AND latency_ms <= 60000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 60000 AND latency_ms <= 120000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 120000 THEN 1 ELSE 0 END),
+  SUM(CASE WHEN ttft_ms > 0 AND ttft_ms <= 100 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 100 AND ttft_ms <= 250 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 250 AND ttft_ms <= 500 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 500 AND ttft_ms <= 1000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 1000 AND ttft_ms <= 2500 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 2500 AND ttft_ms <= 5000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 5000 AND ttft_ms <= 10000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 10000 AND ttft_ms <= 30000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 30000 AND ttft_ms <= 60000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 60000 AND ttft_ms <= 120000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 120000 THEN 1 ELSE 0 END)
+FROM usage
+GROUP BY 1, 2, 3, 4, 5, 6;

internal/store/migrations/sqlite/0016_control_planes.sql

@@ -0,0 +1,9 @@
+-- The control planes agents may dial, by the address they dial: what the
+-- leader tells every agent, so a control plane added after the nodes were
+-- onboarded is one they fail over to without their configuration being
+-- edited. Written by `llmgw server add` and by init for the first one.
+CREATE TABLE control_planes (
+  url      TEXT PRIMARY KEY,
+  name     TEXT NOT NULL,
+  added_at INTEGER NOT NULL
+);

internal/store/pghosts.go

@@ -0,0 +1,44 @@
+package store
+
+import (
+	"fmt"
+	"net/url"
+	"strings"
+)
+
+// SplitPostgres reads a PostgreSQL address that may name several hosts —
+// "postgres://u:p@h1:5432,h2:5432/db?…", the members of a replicated cluster,
+// as pgx and libpq both accept — into its hosts, their ports, and the rest of
+// the address with one placeholder host, which url.Parse can read. url.Parse
+// cannot read the list itself: it took everything before the last colon for
+// one host name.
+func SplitPostgres(dsn string) (hosts, ports []string, rest *url.URL, err error) {
+	scheme, after, ok := strings.Cut(dsn, "://")
+	if !ok {
+		return nil, nil, nil, fmt.Errorf("the store address is not a URL")
+	}
+	authority, tail := after, ""
+	if i := strings.IndexAny(after, "/?"); i >= 0 {
+		authority, tail = after[:i], after[i:]
+	}
+	userinfo, hostlist := "", authority
+	if i := strings.LastIndexByte(authority, '@'); i >= 0 {
+		userinfo, hostlist = authority[:i+1], authority[i+1:]
+	}
+	for _, hp := range strings.Split(hostlist, ",") {
+		u, err := url.Parse("x://" + hp)
+		if err != nil || u.Hostname() == "" {
+			return nil, nil, nil, fmt.Errorf("the store address has a host that is not one: %q", hp)
+		}
+		port := u.Port()
+		if port == "" {
+			port = "5432"
+		}
+		hosts, ports = append(hosts, u.Hostname()), append(ports, port)
+	}
+	rest, err = url.Parse(scheme + "://" + userinfo + "host" + tail)
+	if err != nil {
+		return nil, nil, nil, fmt.Errorf("the store address is not a URL")
+	}
+	return hosts, ports, rest, nil
+}

internal/store/postgres_test.go

@@ -41,9 +41,9 @@ func openPG(t *testing.T) *DB {
 		// inherits the previous one's lease is refused for reasons that have
 		// nothing to do with what it is testing. It passed on SQLite, where
 		// every subtest gets a fresh file, and failed only here.
-		`TRUNCATE thread_messages, threads, memory_notes, usage, api_keys, memberships,
+		`TRUNCATE thread_messages, threads, memory_notes, usage, usage_5m, usage_1h, usage_1d, api_keys, memberships,
 		         key_claims, limits, models, sessions, users, projects, teams,
-		         organizations, leadership RESTART IDENTITY CASCADE`)
+		         organizations, leadership, control_planes RESTART IDENTITY CASCADE`)
 	if err != nil {
 		t.Fatalf("clearing the test database: %v", err)
 	}
@@ -252,3 +252,27 @@ func TestPostgresBucketsUsageByEveryGrouping(t *testing.T) {
 		}
 	}
 }
+
+func TestPostgresMakesAnAdministrator(t *testing.T) {
+	// init --postgres and --patroni create the first administrator straight
+	// in PostgreSQL, which refused the bool SetAdmin sent.
+	ctx := context.Background()
+	s := openPG(t)
+	if err := s.CreateUser(ctx, User{ID: "u1", Email: "a@x", Name: "a", PasswordHash: "h",
+		State: UserActive, CreatedAt: time.Now()}); err != nil {
+		t.Fatal(err)
+	}
+	if err := s.SetAdmin(ctx, "u1", true); err != nil {
+		t.Fatalf("making an administrator: %v", err)
+	}
+	u, err := s.UserByEmail(ctx, "a@x")
+	if err != nil || !u.IsAdmin {
+		t.Fatalf("%+v, %v", u, err)
+	}
+	if err := s.SetAdmin(ctx, "u1", false); err != nil {
+		t.Fatal(err)
+	}
+	if u, _ := s.UserByEmail(ctx, "a@x"); u.IsAdmin {
+		t.Error("still an administrator")
+	}
+}

internal/store/rollup.go

@@ -0,0 +1,211 @@
+package store
+
+import (
+	"context"
+	"fmt"
+	"strings"
+	"time"
+)
+
+// tier is one table usage is summed into as it is written: slices of a fixed
+// width, per key, person, project and model.
+type tier struct {
+	table string
+	ms    int64
+}
+
+var (
+	tier5m = tier{"usage_5m", int64(5 * time.Minute / time.Millisecond)}
+	tier1h = tier{"usage_1h", int64(time.Hour / time.Millisecond)}
+	tier1d = tier{"usage_1d", int64(24 * time.Hour / time.Millisecond)}
+	// tiers, finest first.
+	tiers = []tier{tier5m, tier1h, tier1d}
+)
+
+// floor is the start of the slice a moment falls in. A chart read from a
+// tier starts its first slice there, as a chart read from the rows always did.
+func (t tier) floor(at time.Time) int64 { return (millis(at) / t.ms) * t.ms }
+
+// ceil is the start of the first slice wholly after a moment.
+func (t tier) ceil(at time.Time) int64 {
+	return ((millis(at) + t.ms - 1) / t.ms) * t.ms
+}
+
+// tierForSlices is the coarsest tier whose slices make up a chart slice of
+// width exactly, and false when none does — a slice finer than five minutes,
+// or not a whole number of them, is read from the rows.
+func tierForSlices(width time.Duration) (tier, bool) {
+	ms := width.Milliseconds()
+	for i := len(tiers) - 1; i >= 0; i-- {
+		if ms >= tiers[i].ms && ms%tiers[i].ms == 0 {
+			return tiers[i], true
+		}
+	}
+	return tier{}, false
+}
+
+// LatencyBounds are the upper edges, in milliseconds, of the histogram buckets
+// latency and time to first token are counted in; one more bucket holds
+// everything above the last. Fixed, because buckets are only summable across
+// slices, keys and teams when every row has the same ones.
+var LatencyBounds = []int64{100, 250, 500, 1000, 2500, 5000, 10000, 30000, 60000, 120000}
+
+// latencyBucket is the histogram bucket a duration is counted in.
+func latencyBucket(ms int64) int {
+	for i, b := range LatencyBounds {
+		if ms <= b {
+			return i
+		}
+	}
+	return len(LatencyBounds)
+}
+
+// rollupKey is one row of a tier: a slice, and who spent what on what.
+type rollupKey struct {
+	bucket                                       int64
+	keyID, userID, projectID, model, servedModel string
+}
+
+// rollupRow is what a batch adds to one row.
+type rollupRow struct {
+	requests, errors, limited, prompt int64
+	completion, cost                  int64
+	latency, queue, ttft, ttftN, last int64
+	lat, ttftH                        []int64
+}
+
+// rollupBatch sums a batch of records into the rows of one tier they add to.
+func rollupBatch(t tier, records []Usage) (map[rollupKey]*rollupRow, []rollupKey) {
+	rows := map[rollupKey]*rollupRow{}
+	var order []rollupKey
+	n := len(LatencyBounds) + 1
+	for _, r := range records {
+		ts := millis(r.At)
+		k := rollupKey{(ts / t.ms) * t.ms, r.KeyID, r.UserID, r.ProjectID, r.Model, r.ServedModel}
+		row, ok := rows[k]
+		if !ok {
+			row = &rollupRow{lat: make([]int64, n), ttftH: make([]int64, n)}
+			rows[k] = row
+			order = append(order, k)
+		}
+		row.requests++
+		if r.Status >= 400 {
+			row.errors++
+		}
+		if r.Status == 429 {
+			row.limited++
+		}
+		row.prompt += r.PromptTokens
+		row.completion += r.CompletionTokens
+		row.cost += r.CostMicros
+		row.latency += r.LatencyMs
+		row.queue += r.QueueMs
+		row.lat[latencyBucket(r.LatencyMs)]++
+		if r.TTFTMs > 0 {
+			row.ttft += r.TTFTMs
+			row.ttftN++
+			row.ttftH[latencyBucket(r.TTFTMs)]++
+		}
+		row.last = max(row.last, ts)
+	}
+	return rows, order
+}
+
+// rollupColumns are a tier's columns, in the order rollupUpsert takes them.
+var rollupColumns = func() []string {
+	n := len(LatencyBounds) + 1
+	cols := []string{"bucket", "key_id", "user_id", "project_id", "model", "served_model",
+		"requests", "errors", "limited", "prompt_tokens", "completion_tokens", "cost_micros",
+		"latency_ms_sum", "queue_ms_sum", "ttft_ms_sum", "ttft_n", "last_ts"}
+	for i := range n {
+		cols = append(cols, fmt.Sprintf("lat_%d", i))
+	}
+	for i := range n {
+		cols = append(cols, fmt.Sprintf("ttft_%d", i))
+	}
+	return cols
+}()
+
+// rollupUpsert is the statement that adds a batch's sums to a tier.
+func rollupUpsert(table string) string {
+	var sets []string
+	for _, c := range rollupColumns[6:] {
+		if c == "last_ts" {
+			sets = append(sets, "last_ts = CASE WHEN excluded.last_ts > "+table+".last_ts "+
+				"THEN excluded.last_ts ELSE "+table+".last_ts END")
+			continue
+		}
+		sets = append(sets, c+" = "+table+"."+c+" + excluded."+c)
+	}
+	return `INSERT INTO ` + table + ` (` + strings.Join(rollupColumns, ", ") + `)
+		VALUES (` + strings.TrimSuffix(strings.Repeat("?, ", len(rollupColumns)), ", ") + `)
+		ON CONFLICT (bucket, key_id, user_id, project_id, model, served_model)
+		DO UPDATE SET ` + strings.Join(sets, ", ")
+}
+
+// writeRollup adds a batch to every tier, inside the transaction that writes
+// its rows, so the sums never disagree with them.
+func writeRollup(ctx context.Context, tx conn, records []Usage) error {
+	for _, t := range tiers {
+		rows, order := rollupBatch(t, records)
+		stmt, err := tx.PrepareContext(ctx, rollupUpsert(t.table))
+		if err != nil {
+			return err
+		}
+		for _, k := range order {
+			r := rows[k]
+			args := []any{k.bucket, k.keyID, k.userID, k.projectID, k.model, k.servedModel,
+				r.requests, r.errors, r.limited, r.prompt, r.completion, r.cost,
+				r.latency, r.queue, r.ttft, r.ttftN, r.last}
+			for _, v := range r.lat {
+				args = append(args, v)
+			}
+			for _, v := range r.ttftH {
+				args = append(args, v)
+			}
+			if _, err := stmt.ExecContext(ctx, args...); err != nil {
+				_ = stmt.Close()
+				return fmt.Errorf("adding to %s: %w", t.table, err)
+			}
+		}
+		if err := stmt.Close(); err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+// usageSince is every request since a moment, exactly, as a table with the
+// columns a tier has (but the histograms): the rows up to the next
+// five-minute slice, those slices up to the next hour, hours up to the next
+// day, and days from there. Totals are read from this — a budget against it
+// must be exact, and so is everything else, at the cost of a few rows read
+// from each of four tables rather than a window rounded to a slice.
+//
+// Its placeholders come first in any query it is put into.
+func usageSince(since time.Time) (string, []any) {
+	from := millis(since)
+	to5m, to1h, to1d := tier5m.ceil(since), tier1h.ceil(since), tier1d.ceil(since)
+	const cols = `key_id, user_id, project_id, model, served_model, requests, errors, limited,
+		prompt_tokens, completion_tokens, cost_micros, latency_ms_sum, last_ts`
+	return `(SELECT key_id, user_id, project_id, model, served_model, 1 AS requests,
+			CASE WHEN status >= 400 THEN 1 ELSE 0 END AS errors,
+			CASE WHEN status = 429 THEN 1 ELSE 0 END AS limited,
+			prompt_tokens, completion_tokens, cost_micros, latency_ms AS latency_ms_sum, ts AS last_ts
+		FROM usage WHERE ts >= ? AND ts < ?
+		UNION ALL SELECT ` + cols + ` FROM usage_5m WHERE bucket >= ? AND bucket < ?
+		UNION ALL SELECT ` + cols + ` FROM usage_1h WHERE bucket >= ? AND bucket < ?
+		UNION ALL SELECT ` + cols + ` FROM usage_1d WHERE bucket >= ?)`,
+		[]any{from, to5m, to5m, to1h, to1h, to1d, to1d}
+}
+
+// PruneUsage deletes request rows older than a moment. The tiers keep their
+// sums, so charts, totals and budgets are unchanged; what goes is the record
+// of each request — `ctl request` on one that old finds nothing.
+func (s *DB) PruneUsage(ctx context.Context, before time.Time) (int64, error) {
+	res, err := s.db.ExecContext(ctx, `DELETE FROM usage WHERE ts < ?`, millis(before))
+	if err != nil {
+		return 0, err
+	}
+	return res.RowsAffected()
+}

internal/store/rollup_test.go

@@ -0,0 +1,219 @@
+package store
+
+import (
+	"context"
+	"database/sql"
+	"path/filepath"
+	"testing"
+	"time"
+)
+
+// onBoth runs a test on SQLite, and on PostgreSQL when LLMGW_TEST_POSTGRES
+// names one: the rollup is written with an upsert, and the two dialects are
+// where an upsert goes wrong.
+func onBoth(t *testing.T, f func(t *testing.T, s *DB)) {
+	t.Run("sqlite", func(t *testing.T) { f(t, openDB(t)) })
+	t.Run("postgres", func(t *testing.T) { f(t, openPG(t)) })
+}
+
+// slice is a moment a minute into an hour, a few hours back: inside a
+// five-minute slice and an hour alike.
+func slice() time.Time {
+	return time.UnixMilli(tier1h.floor(time.Now().Add(-3 * time.Hour))).Add(time.Minute)
+}
+
+func TestTheRollupAddsUpToTheRows(t *testing.T) {
+	onBoth(t, func(t *testing.T, s *DB) {
+		ctx := context.Background()
+		at := slice()
+		// Two batches into the same slice, so the second has to add to the
+		// row the first made rather than replace it.
+		for _, batch := range [][]Usage{
+			{
+				{At: at, KeyID: "k", UserID: "u", ProjectID: "p", Model: "m", ServedModel: "m",
+					PromptTokens: 10, CompletionTokens: 5, CostMicros: 100, Status: 200, LatencyMs: 80, TTFTMs: 40},
+				{At: at.Add(time.Second), KeyID: "k", UserID: "u", ProjectID: "p", Model: "m", ServedModel: "m",
+					PromptTokens: 1, CompletionTokens: 1, CostMicros: 7, Status: 500, LatencyMs: 3000},
+			},
+			{
+				{At: at.Add(2 * time.Second), KeyID: "k", UserID: "u", ProjectID: "p", Model: "m", ServedModel: "m",
+					PromptTokens: 2, CompletionTokens: 3, CostMicros: 50, Status: 200, LatencyMs: 400, TTFTMs: 200},
+			},
+		} {
+			if err := s.WriteUsage(ctx, batch); err != nil {
+				t.Fatal(err)
+			}
+		}
+
+		for _, tr := range tiers {
+			var req, errs, prompt, cost, lat, ttftN, last, lat0, lat2, lat5, ttft0, ttft1 int64
+			err := s.db.QueryRowContext(ctx, `SELECT requests, errors, prompt_tokens, cost_micros,
+			latency_ms_sum, ttft_n, last_ts, lat_0, lat_2, lat_5, ttft_0, ttft_1 FROM `+tr.table).
+				Scan(&req, &errs, &prompt, &cost, &lat, &ttftN, &last, &lat0, &lat2, &lat5, &ttft0, &ttft1)
+			if err != nil {
+				t.Fatalf("%s: %v", tr.table, err)
+			}
+			if req != 3 || errs != 1 || prompt != 13 || cost != 157 || lat != 3480 {
+				t.Errorf("sums: %d requests, %d errors, %d prompt, %d cost, %d ms; want 3, 1, 13, 157, 3480",
+					req, errs, prompt, cost, lat)
+			}
+			// 80 ms in <=100, 400 in <=500, 3000 in <=5000; first tokens 40 and
+			// 200, and the failed request had none to count.
+			if lat0 != 1 || lat2 != 1 || lat5 != 1 || ttftN != 2 || ttft0 != 1 || ttft1 != 1 {
+				t.Errorf("histograms: lat %d/%d/%d, ttft n=%d %d/%d", lat0, lat2, lat5, ttftN, ttft0, ttft1)
+			}
+			if last != millis(at.Add(2*time.Second)) {
+				t.Errorf("last seen %d, want the latest request", last)
+			}
+		}
+
+		b, err := s.UsageBuckets(ctx, at.Add(-time.Hour), time.Hour)
+		if err != nil {
+			t.Fatal(err)
+		}
+		var total Bucket
+		for _, x := range b {
+			total.Requests += x.Requests
+			total.CostMicros += x.CostMicros
+		}
+		if total.Requests != 3 || total.CostMicros != 157 {
+			t.Errorf("the chart counts %d requests and %d spent, want 3 and 157", total.Requests, total.CostMicros)
+		}
+	})
+}
+
+func TestABudgetIsCountedFromTheMomentNotTheSlice(t *testing.T) {
+	// Charts start at the slice a window falls in; a budget must not. A
+	// request a minute before the period began is not the period's.
+	onBoth(t, func(t *testing.T, s *DB) {
+		ctx := context.Background()
+		at := slice()
+		// Before the moment; after it in its own slice; in a later slice of
+		// the same hour; and in the next hour — one from each part a budget
+		// is read from.
+		if err := s.WriteUsage(ctx, []Usage{
+			{At: at, KeyID: "k", ProjectID: "p", Model: "m", CostMicros: 100, Status: 200},
+			{At: at.Add(2 * time.Minute), KeyID: "k", ProjectID: "p", Model: "m", CostMicros: 10, Status: 200},
+			{At: at.Add(10 * time.Minute), KeyID: "k", ProjectID: "p", Model: "m", CostMicros: 1, Status: 200},
+			{At: at.Add(70 * time.Minute), KeyID: "k", ProjectID: "p", Model: "m", CostMicros: 1000, Status: 200},
+		}); err != nil {
+			t.Fatal(err)
+		}
+		spend, err := s.SpendSince(ctx, at.Add(time.Minute))
+		if err != nil {
+			t.Fatal(err)
+		}
+		if spend["k"] != 1011 {
+			t.Errorf("spent %d since mid-slice, want 1011", spend["k"])
+		}
+	})
+}
+
+func TestPruningRequestsKeepsEveryTotal(t *testing.T) {
+	onBoth(t, func(t *testing.T, s *DB) {
+		ctx := context.Background()
+		at := slice()
+		if err := s.WriteUsage(ctx, []Usage{
+			{At: at, KeyID: "k", UserID: "u", ProjectID: "p", Model: "m", ServedModel: "m",
+				CostMicros: 100, Status: 200, RequestID: "req-1"},
+		}); err != nil {
+			t.Fatal(err)
+		}
+		n, err := s.PruneUsage(ctx, time.Now())
+		if err != nil || n != 1 {
+			t.Fatalf("pruned %d, %v", n, err)
+		}
+		if _, ok, _ := s.UsageByRequest(ctx, "req-1"); ok {
+			t.Error("a pruned request can still be looked up")
+		}
+		spend, _ := s.SpendSince(ctx, time.UnixMilli(tier5m.floor(at)))
+		users, _ := s.UsageByUser(ctx, at.Add(-time.Hour))
+		if spend["k"] != 100 || len(users) != 1 || users[0].Requests != 1 {
+			t.Errorf("after pruning: spend %v, people %+v; the totals should not move", spend, users)
+		}
+	})
+}
+
+func TestTheMigrationRollsUpWhatIsAlreadyRecorded(t *testing.T) {
+	ctx := context.Background()
+	db, err := sql.Open("sqlite", filepath.Join(t.TempDir(), "old.db")+"?_pragma=foreign_keys(1)")
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer db.Close()
+	db.SetMaxOpenConns(1)
+	if _, err := db.ExecContext(ctx,
+		`CREATE TABLE schema_migrations (version TEXT PRIMARY KEY, applied_at INTEGER NOT NULL)`); err != nil {
+		t.Fatal(err)
+	}
+	names, _ := migrationNames("sqlite")
+	for _, name := range names {
+		if name >= "0015" {
+			break
+		}
+		body, _ := migrationFS.ReadFile("migrations/sqlite/" + name)
+		if err := applyOne(ctx, db, "sqlite", name, string(body)); err != nil {
+			t.Fatalf("%s: %v", name, err)
+		}
+	}
+	for _, ts := range []int64{600_001, 600_002, 900_001} {
+		if _, err := db.ExecContext(ctx, `INSERT INTO usage (ts, key_id, project_id, model, served_model,
+			prompt_tokens, completion_tokens, cost_micros, status, latency_ms, user_id, ttft_ms)
+			VALUES (?, 'k', 'p', 'm', 'm', 1, 1, 5, 200, 700, 'u', 0)`, ts); err != nil {
+			t.Fatal(err)
+		}
+	}
+	if err := migrate(ctx, db, "sqlite"); err != nil {
+		t.Fatal(err)
+	}
+	// 600,001 and 600,002 share a five-minute slice and 900,001 starts the
+	// next; all three are in the first hour, and the first day.
+	for table, want := range map[string]int64{"usage_5m": 2, "usage_1h": 1, "usage_1d": 1} {
+		var slices, reqs, cost, lat3, ttftN int64
+		if err := db.QueryRowContext(ctx, `SELECT COUNT(*), SUM(requests), SUM(cost_micros), SUM(lat_3), SUM(ttft_n)
+			FROM `+table).Scan(&slices, &reqs, &cost, &lat3, &ttftN); err != nil {
+			t.Fatal(err)
+		}
+		if slices != want || reqs != 3 || cost != 15 || lat3 != 3 || ttftN != 0 {
+			t.Errorf("%s: %d slices, %d requests, %d spent, %d in <=1000ms, %d first tokens; want %d, 3, 15, 3, 0",
+				table, slices, reqs, cost, lat3, ttftN, want)
+		}
+	}
+}
+
+func TestATotalReadsEachTierForItsOwnStretchOnly(t *testing.T) {
+	// A total since a moment two days back, a few minutes into a slice, is
+	// read from the rows, then five-minute slices, then hours, then days.
+	// One request lands in each stretch, and one just before the moment;
+	// counting any of them twice, or the early one at all, shows here.
+	onBoth(t, func(t *testing.T, s *DB) {
+		ctx := context.Background()
+		day := time.UnixMilli(tier1d.floor(time.Now().Add(-48 * time.Hour)))
+		since := day.Add(22*time.Hour + 52*time.Minute + 30*time.Second)
+		var batch []Usage
+		for i, at := range []time.Time{
+			since.Add(-time.Second),     // before: not counted
+			since.Add(time.Minute),      // the rows
+			since.Add(5 * time.Minute),  // five-minute slices
+			since.Add(20 * time.Minute), // hours
+			since.Add(2 * time.Hour),    // days
+		} {
+			batch = append(batch, Usage{At: at, KeyID: "k", UserID: "u", ProjectID: "p", Model: "m",
+				ServedModel: "m", CostMicros: int64(1) << i, Status: 200})
+		}
+		if err := s.WriteUsage(ctx, batch); err != nil {
+			t.Fatal(err)
+		}
+		users, err := s.UsageByUser(ctx, since)
+		if err != nil {
+			t.Fatal(err)
+		}
+		spend, err := s.SpendSince(ctx, since)
+		if err != nil {
+			t.Fatal(err)
+		}
+		if len(users) != 1 || users[0].Requests != 4 || users[0].CostMicros != 2+4+8+16 || spend["k"] != 30 {
+			t.Errorf("people %+v, spend %d; want 4 requests and 30 spent", users, spend["k"])
+		}
+	})
+}

internal/store/selfusage.go

@@ -28,17 +28,18 @@ func (s *DB) UsageOfUser(
 	if userID == "" {
 		return nil, nil, fmt.Errorf("whose usage?")
 	}
+	src, args := usageSince(since)
 	places, err := queryAll(ctx, s.db, `
 		SELECT COALESCE(NULLIF(u.project_id, ''), ''),
 		       COALESCE(o.name || ' / ' || t.name || ' / ' || p.name, ''),
-		       COUNT(*), SUM(u.prompt_tokens + u.completion_tokens), SUM(u.cost_micros)
-		FROM usage u
+		       SUM(u.requests), SUM(u.prompt_tokens + u.completion_tokens), SUM(u.cost_micros)
+		FROM `+src+` u
 		LEFT JOIN projects p ON p.id = NULLIF(u.project_id, '')
 		LEFT JOIN teams t ON t.id = p.team_id
 		LEFT JOIN organizations o ON o.id = t.org_id
-		WHERE u.user_id = ? AND u.ts >= ?
+		WHERE u.user_id = ?
 		GROUP BY 1, 2
-		ORDER BY 4 DESC`, []any{userID, millis(since)},
+		ORDER BY 4 DESC`, append(args, userID),
 		func(rows *sql.Rows) (PlaceUsage, error) {
 			var pu PlaceUsage
 			return pu, rows.Scan(&pu.ProjectID, &pu.Path, &pu.Requests, &pu.Tokens, &pu.CostMicros)
@@ -51,12 +52,20 @@ func (s *DB) UsageOfUser(
 		width = 24 * time.Hour
 	}
 	ms := width.Milliseconds()
-	got, err := queryAll(ctx, s.db, `
+	q, from := `
 		SELECT (ts / ?) * ?, COUNT(*),
 		       SUM(CASE WHEN status >= 400 THEN 1 ELSE 0 END),
 		       SUM(prompt_tokens), SUM(completion_tokens), SUM(cost_micros)
 		FROM usage WHERE user_id = ? AND ts >= ?
-		GROUP BY 1 ORDER BY 1`, []any{ms, ms, userID, millis(since)},
+		GROUP BY 1 ORDER BY 1`, millis(since)
+	if t, ok := tierForSlices(width); ok {
+		q, from = `
+		SELECT (bucket / ?) * ?, SUM(requests), SUM(errors),
+		       SUM(prompt_tokens), SUM(completion_tokens), SUM(cost_micros)
+		FROM `+t.table+` WHERE user_id = ? AND bucket >= ?
+		GROUP BY 1 ORDER BY 1`, t.floor(since)
+	}
+	got, err := queryAll(ctx, s.db, q, []any{ms, ms, userID, from},
 		func(rows *sql.Rows) (Bucket, error) {
 			var b Bucket
 			var start int64
@@ -131,8 +140,9 @@ func (s *DB) SpendOfPool(ctx context.Context, l Limits, since time.Time) (int64,
 		return 0, nil
 	}
 	var total sql.NullInt64
+	src, srcArgs := usageSince(since)
 	err := s.db.QueryRowContext(ctx,
-		`SELECT SUM(u.cost_micros) FROM usage u WHERE `+where+` AND u.ts >= ?`,
-		append(args, millis(since))...).Scan(&total)
+		`SELECT SUM(u.cost_micros) FROM `+src+` u WHERE `+where,
+		append(srcArgs, args...)...).Scan(&total)
 	return total.Int64, err
 }

internal/store/sqlite.go

@@ -348,14 +348,14 @@ func (s *DB) applyDefaults(e *Effective) {
 // the tree. Personal keys and the chat page's requests have no project and are
 // kept: dropping them is what would reset a person's budget on every restart.
 func (s *DB) SpendSinceByScope(ctx context.Context, since time.Time) ([]ScopeSpend, error) {
+	src, args := usageSince(since)
 	rows, err := s.db.QueryContext(ctx,
 		`SELECT u.key_id, u.user_id, COALESCE(p.id, ''), COALESCE(t.id, ''), COALESCE(t.org_id, ''),
 		        SUM(u.cost_micros)
-		 FROM usage u
+		 FROM `+src+` u
 		 LEFT JOIN projects p ON p.id = NULLIF(u.project_id, '')
 		 LEFT JOIN teams t    ON t.id = p.team_id
-		 WHERE u.ts >= ?
-		 GROUP BY u.key_id, u.user_id, p.id, t.id, t.org_id`, millis(since))
+		 GROUP BY u.key_id, u.user_id, p.id, t.id, t.org_id`, args...)
 	if err != nil {
 		return nil, err
 	}
@@ -439,14 +439,15 @@ func (s *DB) WriteUsage(ctx context.Context, records []Usage) error {
 				return err
 			}
 		}
-		return nil
+		return writeRollup(ctx, tx, records)
 	})
 }
 
 // SpendSince totals cost per key since a moment.
 func (s *DB) SpendSince(ctx context.Context, since time.Time) (map[string]int64, error) {
+	src, args := usageSince(since)
 	rows, err := s.db.QueryContext(ctx,
-		`SELECT key_id, SUM(cost_micros) FROM usage WHERE ts >= ? GROUP BY key_id`, millis(since))
+		`SELECT u.key_id, SUM(u.cost_micros) FROM `+src+` u GROUP BY u.key_id`, args...)
 	if err != nil {
 		return nil, err
 	}
@@ -620,16 +621,15 @@ var _ Store = (*DB)(nil)
 // to show: a row identified only by an opaque id tells an operator nothing.
 // Revoked keys still appear, because their usage still happened.
 func (s *DB) UsageSince(ctx context.Context, since time.Time) ([]KeyUsage, error) {
+	src, args := usageSince(since)
 	return queryAll(ctx, s.db, `
 		SELECT u.key_id,
 		       COALESCE(k.name, ''), COALESCE(k.key_prefix, ''), u.project_id,
 		       COALESCE(us.email, ''),
 		       COALESCE(o.name || ' / ' || t.name || ' / ' || p.name, ''),
-		       COUNT(*), SUM(u.prompt_tokens), SUM(u.completion_tokens),
-		       SUM(u.cost_micros),
-		       SUM(CASE WHEN u.status >= 400 THEN 1 ELSE 0 END),
-		       MAX(u.ts)
-		FROM usage u
+		       SUM(u.requests), SUM(u.prompt_tokens), SUM(u.completion_tokens),
+		       SUM(u.cost_micros), SUM(u.errors), MAX(u.last_ts)
+		FROM `+src+` u
 		LEFT JOIN api_keys k ON k.id = u.key_id
 		-- Who, as recorded with the spend; the owner of the key for rows
 		-- written before the person was.
@@ -637,12 +637,11 @@ func (s *DB) UsageSince(ctx context.Context, since time.Time) ([]KeyUsage, error
 		LEFT JOIN projects p ON p.id = NULLIF(u.project_id, '')
 		LEFT JOIN teams t ON t.id = p.team_id
 		LEFT JOIN organizations o ON o.id = t.org_id
-		WHERE u.ts >= ?
 		-- Everything selected beside the sums is grouped, which PostgreSQL
 		-- wants; each is determined by the key and project already there.
 		GROUP BY u.key_id, u.project_id, k.name, k.key_prefix, us.email, o.name, t.name, p.name
-		ORDER BY SUM(u.cost_micros) DESC, COUNT(*) DESC`,
-		[]any{millis(since)}, func(rows *sql.Rows) (KeyUsage, error) {
+		ORDER BY SUM(u.cost_micros) DESC, SUM(u.requests) DESC`,
+		args, func(rows *sql.Rows) (KeyUsage, error) {
 			var k KeyUsage
 			var last int64
 			err := rows.Scan(&k.KeyID, &k.KeyName, &k.KeyPrefix, &k.ProjectID,
@@ -664,7 +663,9 @@ func (s *DB) UsageBuckets(ctx context.Context, since time.Time, width time.Durat
 	}
 	ms := width.Milliseconds()
 
-	rows, err := s.db.QueryContext(ctx, `
+	// From the rollup when the slices are whole rollup slices, which every
+	// page asks for; from the rows otherwise.
+	q, from := `
 		SELECT (ts / ?) * ?, COUNT(*),
 		       SUM(CASE WHEN status >= 400 THEN 1 ELSE 0 END),
 		       SUM(prompt_tokens), SUM(completion_tokens), SUM(cost_micros),
@@ -674,7 +675,16 @@ func (s *DB) UsageBuckets(ctx context.Context, since time.Time, width time.Durat
 		-- selected one syntactically: "ts / ?" and "(ts / ?) * ?" are not the
 		-- same text, so it saw a bare ts and refused. The ordinal is the same
 		-- expression by construction, and both dialects accept it.
-		GROUP BY 1 ORDER BY 1`, ms, ms, millis(since))
+		GROUP BY 1 ORDER BY 1`, millis(since)
+	if t, ok := tierForSlices(width); ok {
+		q, from = `
+		SELECT (bucket / ?) * ?, SUM(requests), SUM(errors),
+		       SUM(prompt_tokens), SUM(completion_tokens), SUM(cost_micros),
+		       1.0 * SUM(latency_ms_sum) / SUM(requests)
+		FROM `+t.table+` WHERE bucket >= ?
+		GROUP BY 1 ORDER BY 1`, t.floor(since)
+	}
+	rows, err := s.db.QueryContext(ctx, q, ms, ms, from)
 	if err != nil {
 		return nil, err
 	}
@@ -684,13 +694,13 @@ func (s *DB) UsageBuckets(ctx context.Context, since time.Time, width time.Durat
 	for rows.Next() {
 		var start int64
 		var b Bucket
-		var latency float64
+		var latency sql.NullFloat64
 		if err := rows.Scan(&start, &b.Requests, &b.Errors,
 			&b.Prompt, &b.Completion, &b.CostMicros, &latency); err != nil {
 			return nil, err
 		}
 		b.Start = fromMillis(start)
-		b.LatencyMs = int64(latency)
+		b.LatencyMs = int64(latency.Float64)
 		found[start] = b
 	}
 	if err := rows.Err(); err != nil {
@@ -736,23 +746,21 @@ func (u UserUsage) Tokens() int64 { return u.Prompt + u.Completion }
 // gets skipped. Keys with no owner — issued by an operator before accounts
 // existed — are grouped under an empty user, which is honest rather than tidy.
 func (s *DB) UsageByUser(ctx context.Context, since time.Time) ([]UserUsage, error) {
+	src, args := usageSince(since)
 	rows, err := s.db.QueryContext(ctx, `
 		SELECT u.user_id, COALESCE(us.email, ''), COALESCE(us.name, ''),
 		       COUNT(DISTINCT u.key_id),
-		       COUNT(*), SUM(u.prompt_tokens), SUM(u.completion_tokens),
-		       SUM(u.cost_micros),
-		       SUM(CASE WHEN u.status >= 400 THEN 1 ELSE 0 END),
-		       MAX(u.ts)
-		FROM usage u
+		       SUM(u.requests), SUM(u.prompt_tokens), SUM(u.completion_tokens),
+		       SUM(u.cost_micros), SUM(u.errors), MAX(u.last_ts)
+		FROM `+src+` u
 		LEFT JOIN users us ON us.id = NULLIF(u.user_id, '')
-		WHERE u.ts >= ?
 		-- Every selected column that is not an aggregate has to be grouped.
 		-- SQLite picks an arbitrary row for the rest and carries on; PostgreSQL
 		-- refuses, so this query worked in tests and returned nothing but an
 		-- error on the cluster. The email and name are functionally dependent
 		-- on the user id, so grouping by them changes no result.
 		GROUP BY u.user_id, COALESCE(us.email, ''), COALESCE(us.name, '')
-		ORDER BY SUM(u.cost_micros) DESC, COUNT(*) DESC`, millis(since))
+		ORDER BY SUM(u.cost_micros) DESC, SUM(u.requests) DESC`, args...)
 	if err != nil {
 		return nil, err
 	}
@@ -793,14 +801,14 @@ func (m ModelUsage) Tokens() int64 { return m.Prompt + m.Completion }
 // cluster. The requested name is kept in the same table for anyone who needs to
 // ask the other question.
 func (s *DB) UsageByModel(ctx context.Context, since time.Time) ([]ModelUsage, error) {
+	src, args := usageSince(since)
 	rows, err := s.db.QueryContext(ctx, `
 		SELECT CASE WHEN served_model != '' THEN served_model ELSE model END,
-		       COUNT(*), SUM(prompt_tokens), SUM(completion_tokens),
-		       SUM(cost_micros),
-		       SUM(CASE WHEN status >= 400 THEN 1 ELSE 0 END),
-		       AVG(latency_ms)
-		FROM usage WHERE ts >= ?
-		GROUP BY 1 ORDER BY COUNT(*) DESC`, millis(since))
+		       SUM(requests), SUM(prompt_tokens), SUM(completion_tokens),
+		       SUM(cost_micros), SUM(errors),
+		       1.0 * SUM(latency_ms_sum) / SUM(requests)
+		FROM `+src+` u
+		GROUP BY 1 ORDER BY SUM(requests) DESC`, args...)
 	if err != nil {
 		return nil, err
 	}

internal/store/store_test.go

@@ -493,3 +493,28 @@ func TestNoDefaultsLeavesLimitsUnset(t *testing.T) {
 		t.Errorf("limits appeared with no defaults set: rpm=%v tpm=%v", e.RPM, e.TPM)
 	}
 }
+
+func TestControlPlanesAreKeptInTheOrderTheyWereAdded(t *testing.T) {
+	ctx := context.Background()
+	s := openDB(t)
+	at := time.Now()
+	for i, u := range []string{"https://10.0.0.140:7443", "https://10.0.0.150:7443"} {
+		if err := s.PutControlPlane(ctx, ControlPlane{URL: u, Name: "c", AddedAt: at.Add(time.Duration(i) * time.Second)}); err != nil {
+			t.Fatal(err)
+		}
+	}
+	// Recorded again, as init and server add do: not moved, not doubled.
+	if err := s.PutControlPlane(ctx, ControlPlane{URL: "https://10.0.0.140:7443", Name: "ctl"}); err != nil {
+		t.Fatal(err)
+	}
+	cps, err := s.ControlPlanes(ctx)
+	if err != nil || len(cps) != 2 || cps[0].URL != "https://10.0.0.140:7443" || cps[0].Name != "ctl" {
+		t.Fatalf("%+v %v", cps, err)
+	}
+	if err := s.RemoveControlPlane(ctx, cps[1].URL); err != nil {
+		t.Fatal(err)
+	}
+	if cps, _ := s.ControlPlanes(ctx); len(cps) != 1 {
+		t.Errorf("%+v", cps)
+	}
+}

internal/store/usagescope.go

@@ -80,21 +80,19 @@ func (s *DB) UsageByScope(ctx context.Context, scope Scope, since time.Time) ([]
 	// same expression it is grouped on. PostgreSQL requires both; SQLite
 	// silently picks a row for whatever is missing, which is how three of these
 	// queries shipped working in tests and failing on the cluster.
+	src, args := usageSince(since)
 	q := `
 		SELECT ` + c.id + `, ` + c.name + `,
 		       COUNT(DISTINCT u.key_id),
-		       COUNT(*), SUM(u.prompt_tokens), SUM(u.completion_tokens),
-		       SUM(u.cost_micros),
-		       SUM(CASE WHEN u.status >= 400 THEN 1 ELSE 0 END),
-		       MAX(u.ts),
+		       SUM(u.requests), SUM(u.prompt_tokens), SUM(u.completion_tokens),
+		       SUM(u.cost_micros), SUM(u.errors), MAX(u.last_ts),
 		       MAX(l.budget_micros), MAX(COALESCE(l.budget_period, ''))
-		FROM usage u` + c.joins + `
+		FROM ` + src + ` u` + c.joins + `
 		LEFT JOIN limits l ON l.scope_type = ? AND l.scope_id = ` + c.id + `
-		WHERE u.ts >= ?
 		GROUP BY ` + c.id + `, ` + c.name + `
-		ORDER BY SUM(u.cost_micros) DESC, COUNT(*) DESC`
+		ORDER BY SUM(u.cost_micros) DESC, SUM(u.requests) DESC`
 
-	rows, err := s.db.QueryContext(ctx, q, string(scope), millis(since))
+	rows, err := s.db.QueryContext(ctx, q, append(args, string(scope))...)
 	if err != nil {
 		return nil, err
 	}

internal/store/usageseries.go

@@ -21,6 +21,9 @@ const (
 	GroupByTeam    Grouping = "team"
 	GroupByProject Grouping = "project"
 	GroupByUser    Grouping = "user"
+	// GroupByModel is by the model that answered, which differs from the one
+	// asked for when a fallback served it.
+	GroupByModel Grouping = "model"
 )
 
 // Valid reports whether this is a grouping usage can be read by.
@@ -32,7 +35,7 @@ func (g Grouping) Valid() bool {
 // Groupings lists them in the order a page should offer them: widest first,
 // then the one that is not a level at all.
 func Groupings() []Grouping {
-	return []Grouping{GroupByOrg, GroupByTeam, GroupByProject, GroupByUser}
+	return []Grouping{GroupByOrg, GroupByTeam, GroupByProject, GroupByUser, GroupByModel}
 }
 
 // UsageSeries is one group's usage over time.
@@ -87,8 +90,13 @@ var groupColumns = map[Grouping]struct{ id, name, joins string }{
 	// rather than hides.
 	GroupByUser: {"u.user_id", "COALESCE(us.email, '')", `
 		LEFT JOIN users us ON us.id = NULLIF(u.user_id, '')`},
+	GroupByModel: {servedModel, servedModel, ""},
 }
 
+// servedModel is what answered: the served model, or the one asked for on a
+// row from before the two were told apart.
+const servedModel = "CASE WHEN u.served_model != '' THEN u.served_model ELSE u.model END"
+
 // UsageBucketsBy returns each group's usage over time.
 //
 // One query for every group rather than one per group: the page shows them
@@ -145,8 +153,22 @@ func (s *DB) usageBuckets(
 		WHERE u.ts >= ?` + inOrg + `
 		GROUP BY 1, 2, 3
 		ORDER BY 1, 3`
+	from := millis(since)
+	// From the rollup for slices that are whole rollup slices — every
+	// window the pages offer — and from the rows for anything finer.
+	if t, ok := tierForSlices(width); ok {
+		q = `
+		SELECT ` + c.id + `, ` + c.name + `, (u.bucket / ?) * ?,
+		       SUM(u.requests), SUM(u.prompt_tokens), SUM(u.completion_tokens),
+		       SUM(u.cost_micros), SUM(u.errors)
+		FROM ` + t.table + ` u` + c.joins + `
+		WHERE u.bucket >= ?` + inOrg + `
+		GROUP BY 1, 2, 3
+		ORDER BY 1, 3`
+		from = t.floor(since)
+	}
 
-	args := []any{ms, ms, millis(since)}
+	args := []any{ms, ms, from}
 	if orgID != "" {
 		args = append(args, orgID)
 	}

internal/store/users.go

@@ -175,7 +175,14 @@ func (s *DB) SetUserState(ctx context.Context, id string, state UserState) error
 // account is what closes sessions, and that is the lever for someone who should
 // not be here at all.
 func (s *DB) SetAdmin(ctx context.Context, id string, admin bool) error {
-	res, err := s.db.ExecContext(ctx, `UPDATE users SET is_admin = ? WHERE id = ?`, admin, id)
+	// An integer, not a bool: the column is an integer on both dialects,
+	// and PostgreSQL refuses a bool for it where SQLite took one — so making
+	// an administrator worked in every test and failed on a fresh install.
+	flag := 0
+	if admin {
+		flag = 1
+	}
+	res, err := s.db.ExecContext(ctx, `UPDATE users SET is_admin = ? WHERE id = ?`, flag, id)
 	if err != nil {
 		return err
 	}

internal/web/account.go

@@ -88,6 +88,13 @@ type AccountPages struct {
 	// could run it. Both nil where the mirror page is read-only.
 	catalog CatalogAdmin
 	fleet   Source
+	// The stores behind the people page's three tabs — the tree and its
+	// people, the limits, the usage — each set when its part is mounted, so
+	// the page draws the parts that exist.
+	peopleStore AccountAdmin
+	quotaStore  QuotaAdmin
+	usageStore  UsageAdmin
+	usagePriced Priced
 }
 
 // ServedModels supplies the names the profile page's example should use.

internal/web/admin_people.go

@@ -0,0 +1,268 @@
+package web
+
+import (
+	"context"
+	"net/http"
+	"strings"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// The people page's tabs. One page, because people, the limits they are held
+// to and what they used are three views of one tree: managing a team meant its
+// members on one page, its quota on another and its spend on a third, each
+// linking to the others. Each tab keeps the address it had as a page of its
+// own, so links and bookmarks to /admin/quotas and /admin/usage still land.
+const (
+	tabPeople = "people"
+	tabLimits = "limits"
+	tabUsage  = "usage"
+)
+
+// peopleView is what a request asks the people page for.
+type peopleView struct {
+	Tab      string
+	Msg, Err string
+	// Open is the project shown expanded in the tree.
+	Open string
+}
+
+type peopleData struct {
+	Nav Nav
+	Tab string
+	// The tabs this server has, each built only when it is the one shown.
+	HasPeople, HasLimits, HasUsage bool
+	Users                          *adminUsersData
+	Quotas                         *adminQuotasData
+	Usage                          *adminUsageData
+	// Detail is the branch or person chosen in the tree, nil when none is.
+	Detail *peopleDetail
+	// Sel is what was chosen, "team:<id>" or "user:<id>", for links to keep.
+	Sel     string
+	Pending int
+	Version string
+	Now     string
+}
+
+// peopleDetail is one organization, team, project or person, with everything
+// the three tabs say about it in one place.
+type peopleDetail struct {
+	Kind store.Scope
+	ID   string
+	// Name is what it is called, Path where it sits.
+	Name, Path string
+
+	// Limits is its own quota row, with what it is held to from above; nil
+	// for a person, or where limits are not mounted.
+	Limits *quotaRow
+	// Person is a person's own limit.
+	Person *personRow
+	// Memberships are where a person is.
+	Memberships []store.Membership
+	// Capacity is its capacity quota, nil when it has none.
+	Capacity    *capacityRow
+	HasCapacity bool
+	BudgetsBind bool
+
+	// Its use over the last week, and the totals beside it.
+	HasUsage                bool
+	Requests, Tokens, Spend TimeSeries
+	Totals                  usageRow
+	Priced                  bool
+	// Stats is how its requests ended and how long they took, over the
+	// same week; HasStats is false where the store does not say.
+	Stats                    store.GroupStats
+	HasStats                 bool
+	UsageWindow, QuotaWindow string
+
+	CSRF string
+}
+
+// renderPeople draws the people page on the tab asked for.
+func (p *AccountPages) renderPeople(w http.ResponseWriter, r *http.Request, admin store.User, v peopleView) {
+	d := peopleData{
+		Nav: navFrom(r, "people"), Tab: v.Tab,
+		HasPeople: p.peopleStore != nil, HasLimits: p.quotaStore != nil, HasUsage: p.usageStore != nil,
+		Sel:     r.FormValue("sel"),
+		Version: p.version, Now: time.Now().UTC().Format(time.RFC3339),
+	}
+	switch {
+	case d.Tab == tabPeople && d.HasPeople:
+		u := p.buildUsers(r, p.peopleStore, admin, v.Msg, v.Err, v.Open)
+		d.Users, d.Pending = &u, len(u.Pending)
+		if d.Sel != "" {
+			d.Detail = p.detail(r, d.Sel, &u)
+		}
+	case d.Tab == tabLimits && d.HasLimits:
+		q := p.buildQuotas(r, p.quotaStore, v.Msg, v.Err)
+		d.Quotas = &q
+	case d.Tab == tabUsage && d.HasUsage:
+		u := p.buildUsage(r, p.usageStore, p.usagePriced)
+		d.Usage = &u
+	default:
+		// The tab asked for is not mounted here: the first one that is.
+		switch {
+		case d.HasPeople:
+			v.Tab = tabPeople
+		case d.HasLimits:
+			v.Tab = tabLimits
+		case d.HasUsage:
+			v.Tab = tabUsage
+		default:
+			http.NotFound(w, r)
+			return
+		}
+		p.renderPeople(w, r, admin, v)
+		return
+	}
+	render(w, "admin_people", d)
+}
+
+// usageGrouping is the grouping a branch's or a person's use is read under.
+func usageGrouping(kind store.Scope) (store.Grouping, bool) {
+	switch kind {
+	case store.ScopeOrg:
+		return store.GroupByOrg, true
+	case store.ScopeTeam:
+		return store.GroupByTeam, true
+	case store.ScopeProject:
+		return store.GroupByProject, true
+	case store.ScopeUser:
+		return store.GroupByUser, true
+	}
+	return "", false
+}
+
+// detail gathers what the page knows about one branch or person.
+func (p *AccountPages) detail(r *http.Request, sel string, u *adminUsersData) *peopleDetail {
+	kind, id, ok := strings.Cut(sel, ":")
+	if !ok || id == "" {
+		return nil
+	}
+	ctx := r.Context()
+	d := &peopleDetail{Kind: store.Scope(kind), ID: id, CSRF: csrfFor(r),
+		UsageWindow: "last 7 days", QuotaWindow: "30 days"}
+	if !p.nameDetail(d, u) {
+		return nil
+	}
+
+	if p.quotaStore != nil {
+		d.BudgetsBind = p.priced != nil && p.priced.AnythingPriced()
+		p.limitsDetail(ctx, d)
+	}
+	if p.capacity != nil && d.Kind != store.ScopeUser {
+		d.HasCapacity = true
+		if views, err := p.capacity.QuotaViews(ctx); err == nil {
+			for _, v := range views {
+				if v.Scope == sel {
+					c := capacityRow{QuotaView: v,
+						GPUs:   capCell(v.Holds.GPUs, v.Guarantee.GPUs, v.Limit.GPUs),
+						Cores:  capCell(v.Holds.CPUCores, v.Guarantee.CPUCores, v.Limit.CPUCores),
+						Memory: capCell(v.Holds.MemoryGB, v.Guarantee.MemoryGB, v.Limit.MemoryGB)}
+					d.Capacity = &c
+				}
+			}
+		}
+	}
+	if p.usageStore != nil {
+		p.usageDetail(ctx, d)
+	}
+	return d
+}
+
+// nameDetail finds what was chosen in the tree or among the accounts, and
+// says whether it is there at all.
+func (p *AccountPages) nameDetail(d *peopleDetail, u *adminUsersData) bool {
+	if d.Kind == store.ScopeUser {
+		for _, a := range u.Accounts {
+			if a.ID == d.ID {
+				d.Name, d.Path, d.Memberships = a.Email, a.Name, a.Memberships
+				return true
+			}
+		}
+		return false
+	}
+	for _, o := range u.Tree {
+		if d.Kind == store.ScopeOrg && o.ID == d.ID {
+			d.Name, d.Path = o.Name, o.Name
+			return true
+		}
+		for _, t := range o.Teams {
+			if d.Kind == store.ScopeTeam && t.ID == d.ID {
+				d.Name, d.Path = t.Name, o.Name+" / "+t.Name
+				return true
+			}
+			for _, pr := range t.Projects {
+				if d.Kind == store.ScopeProject && pr.ID == d.ID {
+					d.Name, d.Path = pr.Name, o.Name+" / "+t.Name+" / "+pr.Name
+					return true
+				}
+			}
+		}
+	}
+	return false
+}
+
+// limitsDetail is the branch's own quota row, or the person's own limit.
+func (p *AccountPages) limitsDetail(ctx context.Context, d *peopleDetail) {
+	s := p.quotaStore
+	limits, err := s.AllLimits(ctx)
+	if err != nil {
+		p.log.Warn("could not read the limits for a detail panel", "err", err)
+		return
+	}
+	since := time.Now().Add(-quotaWindow)
+	if d.Kind == store.ScopeUser {
+		for _, row := range p.personRows(ctx, s, limits, since, "") {
+			if row.ID == d.ID {
+				row := row
+				d.Person = &row
+			}
+		}
+		return
+	}
+	tree, err := s.Tree(ctx)
+	if err != nil {
+		return
+	}
+	for _, row := range quotaRows(tree, limits, p.spendByScope(ctx, s, since)) {
+		if row.Scope == d.Kind && row.ID == d.ID {
+			ms := store.MemberScope(row.Scope)
+			if l, ok := limits[store.LimitOwner{Scope: ms, ID: row.ID}.CounterKey()]; ok {
+				row.MemberRaw, row.Member = fieldsOf(l), describeLimits(l)
+			}
+			d.Limits = &row
+			return
+		}
+	}
+}
+
+// usageDetail is the branch's or person's use over the last week, drawn.
+func (p *AccountPages) usageDetail(ctx context.Context, d *peopleDetail) {
+	g, ok := usageGrouping(d.Kind)
+	if !ok {
+		return
+	}
+	win := windowFor("7d")
+	rows, err := p.usageStore.UsageBucketsBy(ctx, g, time.Now().Add(-win.Span), win.Width)
+	if err != nil {
+		p.log.Warn("could not read usage for a detail panel", "sel", string(d.Kind)+":"+d.ID, "err", err)
+		return
+	}
+	d.HasUsage = true
+	d.Priced = p.usagePriced != nil && p.usagePriced.AnythingPriced()
+	if sr, ok := p.usageStore.(StatsReader); ok {
+		if stats, err := sr.StatsBy(ctx, g, time.Now().Add(-win.Span)); err == nil {
+			d.Stats, d.HasStats = stats[d.ID], true
+		}
+	}
+	for _, s := range rows {
+		if s.ID == d.ID {
+			var table []usageRow
+			d.Requests, d.Tokens, d.Spend, table = usageCharts([]store.UsageSeries{s})
+			d.Totals = table[0]
+			return
+		}
+	}
+}

internal/web/admin_people_test.go

@@ -0,0 +1,111 @@
+package web
+
+import (
+	"io"
+	"log/slog"
+	"net/http"
+	"net/url"
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// peoplePage mounts all three tabs, as the server does.
+func peoplePage(t *testing.T, q *fakeQuotas, u *fakeScopeSeries) *http.ServeMux {
+	t.Helper()
+	a := fakeAccounts{user: store.User{ID: "u1", Email: "a@example.com", IsAdmin: true}}
+	p := NewAccountPages(a, nil, nil, "v0", slog.New(slog.NewTextHandler(io.Discard, nil)))
+	mux := http.NewServeMux()
+	p.RegisterAdminUsers(mux, &fakeAdmin{orgs: q.orgs})
+	p.RegisterAdminQuotas(mux, q, pricedYes{})
+	p.RegisterAdminUsage(mux, u, pricedYes{})
+	return mux
+}
+
+func TestABranchChosenInTheTreeShowsItsLimitsAndUseTogether(t *testing.T) {
+	// People, their limits and their use were three pages: a team's members
+	// on one, its quota on another, its spend on a third.
+	q := oneTeam()
+	rpm := 60
+	q.limits["org:org-1"] = store.Limits{Scope: store.ScopeOrg, ScopeID: "org-1", RPM: &rpm}
+	u := &fakeScopeSeries{byGroup: map[store.Grouping][]store.UsageSeries{
+		store.GroupByOrg: {{Grouping: store.GroupByOrg, ID: "org-1", Name: "Scania", Requests: 1234,
+			Buckets: []store.Bucket{{Start: time.Now().Add(-time.Hour), Requests: 1234}}}},
+	}}
+	body := strings.Join(strings.Fields(
+		mirrorGet(t, peoplePage(t, q, u), "/admin/users?sel=org:org-1").Body.String()), " ")
+
+	for _, want := range []string{
+		`class="people-detail"`, "<h2>Scania</h2>",
+		"<dt>Requests/min</dt><dd>60</dd>", // its own limit
+		"1.2k requests",                    // its use over the week
+		`name="sel" value="org:org-1"`,     // changing it comes back here
+		`aria-current="page">Organizations &amp; people`,
+	} {
+		if !strings.Contains(body, want) {
+			t.Errorf("the panel lacks %q", want)
+		}
+	}
+	if u.asked[len(u.asked)-1] != store.GroupByOrg {
+		t.Errorf("use was read by %v, not by organization", u.asked)
+	}
+}
+
+func TestAChangeMadeInAPanelComesBackToThePanel(t *testing.T) {
+	q := oneTeam()
+	body := mirrorPost(t, peoplePage(t, q, &fakeScopeSeries{}), "/admin/quotas", url.Values{
+		"csrf": {"tok"}, "scope": {"org"}, "id": {"org-1"}, "rpm": {"30"},
+		"tab": {"people"}, "sel": {"org:org-1"},
+	}).Body.String()
+	if len(q.put) != 1 || *q.put[0].RPM != 30 {
+		t.Fatalf("the limit was not saved: %+v", q.put)
+	}
+	if !strings.Contains(body, `class="people-detail"`) || !strings.Contains(body, "rpm 30") {
+		t.Error("saving from the panel did not come back to it with the result")
+	}
+}
+
+func TestEachOldAddressOpensItsTab(t *testing.T) {
+	mux := peoplePage(t, oneTeam(), &fakeScopeSeries{})
+	for path, tab := range map[string]string{
+		"/admin/users": "Organizations &amp; people", "/admin/quotas": "Limits", "/admin/usage": "Usage",
+	} {
+		body := mirrorGet(t, mux, path).Body.String()
+		if !strings.Contains(body, `aria-current="page">`+tab) {
+			t.Errorf("%s does not open the %s tab", path, tab)
+		}
+	}
+}
+
+func TestSomethingNotInTheTreeOpensNoPanel(t *testing.T) {
+	body := mirrorGet(t, peoplePage(t, oneTeam(), &fakeScopeSeries{}), "/admin/users?sel=org:gone").Body.String()
+	if strings.Contains(body, `class="people-detail"`) {
+		t.Error("a panel opened for an organization that does not exist")
+	}
+}
+
+func TestHowRequestsFaredIsShownBesideWhatTheyCost(t *testing.T) {
+	stats := map[string]store.GroupStats{"org-1": {ID: "org-1", Requests: 200, Errors: 30, Limited: 25,
+		TTFT:    store.Quantiles{P50: 180, P95: 900, P99: 2400, N: 150},
+		Latency: store.Quantiles{P50: 3000, P95: 8000, P99: 120000, N: 200, Over: true}}}
+	u := &fakeScopeSeries{stats: stats, byGroup: map[store.Grouping][]store.UsageSeries{
+		store.GroupByOrg: {{Grouping: store.GroupByOrg, ID: "org-1", Name: "Scania", Requests: 200,
+			Buckets: []store.Bucket{{Start: time.Now().Add(-time.Hour), Requests: 200}}}},
+	}}
+	mux := peoplePage(t, oneTeam(), u)
+
+	table := strings.Join(strings.Fields(mirrorGet(t, mux, "/admin/usage?by=org").Body.String()), " ")
+	for _, want := range []string{"15.0%", ">25<", "180 ms / 900 ms", "8.0 s"} {
+		if !strings.Contains(table, want) {
+			t.Errorf("the usage table lacks %q", want)
+		}
+	}
+	panel := strings.Join(strings.Fields(mirrorGet(t, mux, "/admin/users?sel=org:org-1").Body.String()), " ")
+	for _, want := range []string{"<dt>Refused by a limit</dt>", "p95 900 ms", "p99 &gt; 120 s"} {
+		if !strings.Contains(panel, want) {
+			t.Errorf("the panel lacks %q", want)
+		}
+	}
+}

internal/web/admin_quotas.go

@@ -159,10 +159,11 @@ type adminQuotasData struct {
 func (p *AccountPages) RegisterAdminQuotas(mux *http.ServeMux, s QuotaAdmin, priced Priced) {
 	p.hasQuotas = true
 	p.priced = priced
+	p.quotaStore = s
 
 	mux.HandleFunc("GET /admin/quotas", p.adminOnly(
-		func(w http.ResponseWriter, r *http.Request, _ store.User) {
-			p.renderQuotas(w, r, s, "", "")
+		func(w http.ResponseWriter, r *http.Request, admin store.User) {
+			p.renderPeople(w, r, admin, peopleView{Tab: tabLimits})
 		}))
 
 	mux.HandleFunc("POST /admin/quotas", p.adminOnly(
@@ -177,10 +178,10 @@ func (p *AccountPages) RegisterAdminQuotas(mux *http.ServeMux, s QuotaAdmin, pri
 				p.log.Warn("a quota change was refused",
 					"admin", admin.Email, "scope", r.FormValue("scope"),
 					"id", r.FormValue("id"), "err", err)
-				p.renderQuotas(w, r, s, "", err.Error())
+				p.renderQuotas(w, r, admin, "", err.Error())
 				return
 			}
-			p.renderQuotas(w, r, s, msg, "")
+			p.renderQuotas(w, r, admin, msg, "")
 		}))
 
 	mux.HandleFunc("POST /admin/quotas/capacity", p.adminOnly(
@@ -194,11 +195,11 @@ func (p *AccountPages) RegisterAdminQuotas(mux *http.ServeMux, s QuotaAdmin, pri
 			if err != nil {
 				p.log.Warn("a capacity quota change was refused",
 					"admin", admin.Email, "scope", r.FormValue("scope"), "err", err)
-				p.renderQuotas(w, r, s, "", err.Error())
+				p.renderQuotas(w, r, admin, "", err.Error())
 				return
 			}
 			p.log.Info("capacity quota changed", "admin", admin.Email, "what", msg)
-			p.renderQuotas(w, r, s, msg, "")
+			p.renderQuotas(w, r, admin, msg, "")
 		}))
 }
 
@@ -270,10 +271,24 @@ func (p *AccountPages) applyQuota(r *http.Request, s QuotaAdmin, admin store.Use
 	return string(scope) + " " + id + ": " + now, nil
 }
 
+// renderQuotas answers a change to a limit: on the limits tab, or on the
+// people tab when it was made from a branch's own panel there.
 func (p *AccountPages) renderQuotas(w http.ResponseWriter, r *http.Request,
-	s QuotaAdmin, msg, errMsg string,
+	admin store.User, msg, errMsg string,
 ) {
-	const window = 30 * 24 * time.Hour
+	tab := tabLimits
+	if r.FormValue("tab") == tabPeople {
+		tab = tabPeople
+	}
+	p.renderPeople(w, r, admin, peopleView{Tab: tab, Msg: msg, Err: errMsg})
+}
+
+// quotaWindow is how far back spend against a budget is counted.
+const quotaWindow = 30 * 24 * time.Hour
+
+// buildQuotas is every limit, as the limits tab shows them.
+func (p *AccountPages) buildQuotas(r *http.Request, s QuotaAdmin, msg, errMsg string) adminQuotasData {
+	const window = quotaWindow
 	d := adminQuotasData{
 		Nav:   navFrom(r, "quotas"),
 		Since: "30 days", Message: msg, Error: errMsg,
@@ -289,8 +304,7 @@ func (p *AccountPages) renderQuotas(w http.ResponseWriter, r *http.Request,
 	if err != nil {
 		p.log.Error("could not read the limits", "err", err)
 		d.Error = "could not read the limits"
-		render(w, "admin_quotas", d)
-		return
+		return d
 	}
 	spend := p.spendByScope(r.Context(), s, time.Now().Add(-window))
 
@@ -314,7 +328,7 @@ func (p *AccountPages) renderQuotas(w http.ResponseWriter, r *http.Request,
 		d.Rows[i].Editing = d.Editing ==
 			string(d.Rows[i].Scope)+":"+d.Rows[i].ID
 	}
-	render(w, "admin_quotas", d)
+	return d
 }
 
 // spendByScope is what each group has used in the window, keyed like a limit.

internal/web/admin_usage.go

@@ -2,6 +2,7 @@ package web
 
 import (
 	"context"
+	"fmt"
 	"net/http"
 	"sort"
 	"time"
@@ -16,6 +17,35 @@ type UsageAdmin interface {
 		since time.Time, width time.Duration) ([]store.UsageSeries, error)
 }
 
+// StatsReader is how each group fared and how long its requests took; the
+// store implements it. Optional: without it the usage table has no such
+// columns.
+type StatsReader interface {
+	StatsBy(ctx context.Context, g store.Grouping, since time.Time) (map[string]store.GroupStats, error)
+}
+
+// pctile is a percentile from the usage histograms as a person reads it:
+// "180 ms", "4.2 s", "> 120 s" past the last bucket, "—" with nothing to read.
+func pctile(q store.Quantiles, which int) string {
+	if q.N == 0 {
+		return "—"
+	}
+	v := q.P50
+	switch which {
+	case 95:
+		v = q.P95
+	case 99:
+		v = q.P99
+		if q.Over {
+			return fmt.Sprintf("> %d s", v/1000)
+		}
+	}
+	if v < 1000 {
+		return fmt.Sprintf("%d ms", v)
+	}
+	return fmt.Sprintf("%.1f s", float64(v)/1000)
+}
+
 // usageWindow is one length of history the page offers, and the slice it is cut
 // into.
 //
@@ -54,11 +84,16 @@ type usageGroup struct {
 
 // usageRow is one group's totals, shown under the charts.
 type usageRow struct {
+	ID       string
 	Name     string
 	Requests int64
 	Tokens   int64
 	Spend    string
 	Errors   int64
+	// Stats is how its requests ended and how long they took; HasStats is
+	// false where the store does not say.
+	Stats    store.GroupStats
+	HasStats bool
 }
 
 type adminUsageData struct {
@@ -86,6 +121,8 @@ type adminUsageData struct {
 	// model on a GPU.
 	GPUHeld  TimeSeries
 	GPUHours []gpuHoursRow
+	// HasStats adds outcomes and latency percentiles to the table.
+	HasStats bool
 
 	Error   string
 	Version string
@@ -99,15 +136,15 @@ type adminUsageData struct {
 // answers "who is using the cluster, and when" — including per person, which is
 // not a level quotas can be written on at all.
 func (p *AccountPages) RegisterAdminUsage(mux *http.ServeMux, s UsageAdmin, priced Priced) {
-	mux.HandleFunc("GET /admin/usage", p.AdminPage(
-		func(w http.ResponseWriter, r *http.Request) {
-			p.renderUsage(w, r, s, priced)
+	p.usageStore, p.usagePriced = s, priced
+	mux.HandleFunc("GET /admin/usage", p.adminOnly(
+		func(w http.ResponseWriter, r *http.Request, admin store.User) {
+			p.renderPeople(w, r, admin, peopleView{Tab: tabUsage})
 		}))
 }
 
-func (p *AccountPages) renderUsage(w http.ResponseWriter, r *http.Request,
-	s UsageAdmin, priced Priced,
-) {
+// buildUsage is the cluster's use over time, as the usage tab shows it.
+func (p *AccountPages) buildUsage(r *http.Request, s UsageAdmin, priced Priced) adminUsageData {
 	by := store.Grouping(r.URL.Query().Get("by"))
 	if !by.Valid() {
 		by = store.GroupByTeam
@@ -137,13 +174,22 @@ func (p *AccountPages) renderUsage(w http.ResponseWriter, r *http.Request,
 	if err != nil {
 		p.log.Error("could not read usage over time", "by", by, "err", err)
 		d.Error = "could not read usage"
-		render(w, "admin_usage", d)
-		return
+		return d
 	}
 
 	d.Requests, d.Tokens, d.Spend, d.Rows = usageCharts(rows)
+	if sr, ok := s.(StatsReader); ok {
+		stats, err := sr.StatsBy(r.Context(), by, time.Now().Add(-win.Span))
+		if err != nil {
+			p.log.Warn("could not read latency and outcomes for the usage tab", "by", by, "err", err)
+		}
+		for i := range d.Rows {
+			d.Rows[i].Stats, d.Rows[i].HasStats = stats[d.Rows[i].ID], stats != nil
+		}
+		d.HasStats = stats != nil
+	}
 	d.GPUHeld, d.GPUHours = p.gpuHours(r.Context(), win)
-	render(w, "admin_usage", d)
+	return d
 }
 
 // gpuHoursRow is the GPU time one owner's models held over the window.
@@ -227,6 +273,8 @@ func groupLabel(g store.Grouping) string {
 		return "project"
 	case store.GroupByUser:
 		return "person"
+	case store.GroupByModel:
+		return "model"
 	default:
 		return string(g)
 	}
@@ -247,6 +295,7 @@ func usageCharts(rows []store.UsageSeries) (req, tok, spend TimeSeries, table []
 	}), "", 4)
 	for _, u := range rows {
 		table = append(table, usageRow{
+			ID:       u.ID,
 			Name:     displayName(u),
 			Requests: u.Requests,
 			Tokens:   u.Tokens(),

internal/web/admin_usage_test.go

@@ -15,9 +15,11 @@ import (
 
 type fakeScopeSeries struct {
 	byGroup map[store.Grouping][]store.UsageSeries
-	asked   []store.Grouping
-	width   time.Duration
-	since   time.Time
+	// stats, when set, is what StatsBy answers for every grouping.
+	stats map[string]store.GroupStats
+	asked []store.Grouping
+	width time.Duration
+	since time.Time
 }
 
 func (f *fakeScopeSeries) UsageBucketsBy(_ context.Context, g store.Grouping,
@@ -28,6 +30,10 @@ func (f *fakeScopeSeries) UsageBucketsBy(_ context.Context, g store.Grouping,
 	return f.byGroup[g], nil
 }
 
+func (f *fakeScopeSeries) StatsBy(context.Context, store.Grouping, time.Time) (map[string]store.GroupStats, error) {
+	return f.stats, nil
+}
+
 func usagePage(t *testing.T, s *fakeScopeSeries, priced Priced) *http.ServeMux {
 	t.Helper()
 	a := fakeAccounts{user: store.User{ID: "u1", Email: "a@example.com", IsAdmin: true}}

internal/web/admin_users.go

@@ -87,6 +87,9 @@ type adminUsersData struct {
 	Projects []store.ProjectPath
 	Orgs     []store.Org
 	Teams    []store.TeamPath
+	// Sel is the branch or person whose panel is open, "team:<id>", so the
+	// tree can mark it.
+	Sel string
 	// Open is the project whose section is shown expanded: the one just acted
 	// on, so the result of a change is in view without the reader hunting for
 	// it. Scripted pages keep every open section open; this is for the rest.
@@ -115,6 +118,7 @@ type adminUsersData struct {
 // shell is no hardship. Revoking a key is here, because it only ever takes
 // access away.
 func (p *AccountPages) RegisterAdminUsers(mux *http.ServeMux, s AccountAdmin) {
+	p.peopleStore = s
 	mux.HandleFunc("GET /admin/users", p.adminOnly(func(w http.ResponseWriter, r *http.Request, admin store.User) {
 		p.renderUsers(w, r, s, admin, "", "", r.URL.Query().Get("open"))
 	}))
@@ -480,12 +484,17 @@ func (p *AccountPages) adminAction(s AccountAdmin,
 }
 
 func (p *AccountPages) renderUsers(w http.ResponseWriter, r *http.Request,
-	s AccountAdmin, admin store.User, msg, errMsg, open string,
+	_ AccountAdmin, admin store.User, msg, errMsg, open string,
 ) {
+	p.renderPeople(w, r, admin, peopleView{Tab: tabPeople, Msg: msg, Err: errMsg, Open: open})
+}
+
+// buildUsers is the tree and its people, as the first tab shows them.
+func (p *AccountPages) buildUsers(r *http.Request, s AccountAdmin, admin store.User, msg, errMsg, open string) adminUsersData {
 	ctx := r.Context()
 	d := adminUsersData{
 		Nav:     navFrom(r, "people"),
-		Message: msg, Error: errMsg, Open: open, AdminID: admin.ID,
+		Message: msg, Error: errMsg, Open: open, AdminID: admin.ID, Sel: r.FormValue("sel"),
 		CSRF:      csrfFor(r),
 		Version:   p.version,
 		HasMirror: p.hasMirror,
@@ -558,7 +567,7 @@ func (p *AccountPages) renderUsers(w http.ResponseWriter, r *http.Request,
 		d.Scopes = append(d.Scopes, scopeOption{"project:" + pp.ID, pp.Path})
 	}
 	sort.Slice(d.Scopes, func(i, j int) bool { return d.Scopes[i].Label < d.Scopes[j].Label })
-	render(w, "admin_users", d)
+	return d
 }
 
 func keyState(k store.Key, now time.Time) string {

internal/web/chat.go

@@ -18,6 +18,33 @@ type ChatModel struct {
 	// offered anyway: the first message is what asks, so hiding it until it
 	// runs meant it could never be started from here.
 	OnDemand bool
+	// Hardware is what the model's running replicas compute on: "cpu", or a
+	// GPU class. Several when it runs on both, and empty when nothing runs
+	// or nothing said.
+	Hardware []string
+}
+
+// Runs names the hardware for the picker: "GPU", "CPU", or "GPU + CPU" for a
+// model with replicas on both, where which one answers is the router's choice
+// and each reply says which it was.
+func (m ChatModel) Runs() string {
+	var gpu, cpu bool
+	for _, h := range m.Hardware {
+		if h == "cpu" {
+			cpu = true
+		} else {
+			gpu = true
+		}
+	}
+	switch {
+	case gpu && cpu:
+		return "GPU + CPU"
+	case gpu:
+		return "GPU"
+	case cpu:
+		return "CPU"
+	}
+	return ""
 }
 
 // ChatContext is one thing the chat page can spend against: a project, or the

internal/web/chat_test.go

@@ -185,3 +185,19 @@ func TestTheTruncationNoticeIsNotPutInsideTheReply(t *testing.T) {
 		t.Errorf("the reply is written as HTML other than from /chat/render (%d assignments)", n)
 	}
 }
+
+func TestChatModelRunsNamesTheHardware(t *testing.T) {
+	for _, c := range []struct {
+		hw   []string
+		want string
+	}{
+		{nil, ""},
+		{[]string{"cpu"}, "CPU"},
+		{[]string{"6000-24g"}, "GPU"},
+		{[]string{"cpu", "6000-24g"}, "GPU + CPU"},
+	} {
+		if got := (ChatModel{Hardware: c.hw}).Runs(); got != c.want {
+			t.Errorf("%v: %q, want %q", c.hw, got, c.want)
+		}
+	}
+}

internal/web/nav_test.go

@@ -25,7 +25,8 @@ func TestEveryPageDrawsTheSameMenu(t *testing.T) {
 	p.RegisterAdminMirror(mux, MirrorDeps{Mirror: &fakeMirror{models: []mirror.Status{}}})
 
 	pages := map[string]string{
-		"/admin/users": "People", "/admin/quotas": "Quotas", "/admin/mirror": "Mirror",
+		// Limits and usage are tabs of the people page, which they mark.
+		"/admin/users": "People", "/admin/quotas": "People", "/admin/mirror": "Mirror",
 	}
 	var first string
 	for path, current := range pages {
@@ -44,7 +45,7 @@ func TestEveryPageDrawsTheSameMenu(t *testing.T) {
 			t.Errorf("%s draws a different menu:\n%s\nfrom:\n%s", path, plain, first)
 		}
 	}
-	for _, link := range []string{"/admin", "/admin/users", "/admin/quotas", "/admin/usage",
+	for _, link := range []string{"/admin", "/admin/users",
 		"/admin/mirror", "/chat", "/profile", "/logout"} {
 		if !strings.Contains(first, `"`+link+`"`) {
 			t.Errorf("the menu lacks %s", link)

internal/web/org_page.go

@@ -3,6 +3,7 @@ package web
 import (
 	"context"
 	"net/http"
+	"net/url"
 	"sort"
 	"time"
 
@@ -70,6 +71,9 @@ type orgPageData struct {
 func (p *AccountPages) RegisterOrgPages(mux *http.ServeMux, s OrgViewer, priced Priced) {
 	p.orgs = s
 	mux.HandleFunc("GET /org", p.orgGuard(func(w http.ResponseWriter, r *http.Request, user store.User) {
+		if p.toPeople(w, r, user, "") {
+			return
+		}
 		orgs, err := p.viewableOrgs(r.Context(), s, user)
 		if err == nil && len(orgs) == 0 && user.IsAdmin {
 			// Said, not a 404: an administrator following the link on a
@@ -89,6 +93,9 @@ func (p *AccountPages) RegisterOrgPages(mux *http.ServeMux, s OrgViewer, priced
 	}))
 	mux.HandleFunc("GET /org/{id}", p.orgGuard(func(w http.ResponseWriter, r *http.Request, user store.User) {
 		id := r.PathValue("id")
+		if p.toPeople(w, r, user, id) {
+			return
+		}
 		if !user.IsAdmin {
 			ok, err := s.ManagesOrg(r.Context(), user.ID, id)
 			if err != nil || !ok {
@@ -102,6 +109,22 @@ func (p *AccountPages) RegisterOrgPages(mux *http.ServeMux, s OrgViewer, priced
 	}))
 }
 
+// toPeople sends an administrator to the people page, which shows an
+// organization's members, limits and use beside the whole tree and can change
+// them; this page stays for the managers who may not see that one. It says
+// whether it did.
+func (p *AccountPages) toPeople(w http.ResponseWriter, r *http.Request, user store.User, orgID string) bool {
+	if !user.IsAdmin || p.peopleStore == nil {
+		return false
+	}
+	to := "/admin/users"
+	if orgID != "" {
+		to += "?sel=org:" + url.QueryEscape(orgID)
+	}
+	http.Redirect(w, r, to, http.StatusSeeOther)
+	return true
+}
+
 // viewableOrgs is every organization an administrator may look at, or the
 // ones a person manages.
 func (p *AccountPages) viewableOrgs(ctx context.Context, s OrgViewer, user store.User) ([]store.Org, error) {