v0.14.9: changelog and README

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

.gitea/workflows/ci.yml

@@ -34,7 +34,11 @@ jobs:
         run: go vet ./...
 
       - name: Test with the race detector
-        run: go test -race ./...
+        # Under the job's limit, so a package that hangs panics with the
+        # test it was in and where every goroutine waited, rather than the
+        # job ending at its deadline with nothing said: from 2026-10-15 every
+        # run did, printing nothing after internal/release.
+        run: go test -race -timeout 12m ./...
 
       - name: Build
         run: go build ./...

BACKLOG.md

@@ -15,6 +15,41 @@ Conventions this codebase keeps, which every item here must keep too:
 - `go test ./...`, `go vet ./...` and `govulncheck ./...` pass. No release is
   cut and nothing is deployed to the PoC cluster without being told to.
 
+## Open, in order (2026-10-16, after v0.14.7)
+
+What is left, most needed first. The detail is in the item each names.
+
+1. ~~**J1**~~ — done 2026-10-16: a restart no longer refused for room
+   another restart was giving back.
+2. ~~**J5**~~ — done 2026-10-16: an upgrade refits an earlier install's
+   judge by itself.
+3. ~~**I3**~~ — done 2026-10-16: `ctl bench overhead`, figures in the
+   README.
+4. ~~**I4**~~ — done 2026-10-17: several keys for one outside provider.
+5. ~~**Active Directory and Entra ID**~~ — done 2026-10-18: `auth.ldap`
+   sign-in and sync, Entra's 200-group overflow kept from taking roles.
+   Still to do: try it against a real domain controller (a Samba AD DC
+   on Proxmox): LDAPS, nested groups, a disabled user suspended by the
+   sync.
+6. **J2** — a judge the leader adds by itself is sized as init's is. (S)
+7. **J3** — HA setup puts the judges on nodes of their own. (S)
+8. **J8** — a judge budget that fits the machine, measured at install. (S)
+9. **F5** — large imports limited by the embedding engine's slots. (M)
+10. **I1** — answers to similar questions from the cache. (M)
+11. **J4** — the model mirror on more than one machine. (M)
+12. **I2** — the customer's MCP servers behind llmgw. (L)
+13. Smaller leftovers: audit before/after values, a tenant-admin role and
+    forms hidden from viewers (C1); Parquet and a scheduled push for usage
+    export (C4); English errors on Turkish pages (C24); H1 tried on the PoC;
+    licences in video and audio (H5); nginx in front of several control
+    planes (J7).
+
+Not code, the operator's to decide or do: the PoC on v0.14.7 (it runs a
+v0.14.5 dev build; no hurry); where Alertmanager sends (its receiver is
+"none"). Settled 2026-10-16: Apertus 8B is on demand (min 0); the HA test
+VMs 157–159, 165, 166 and the e2e VMs 151, 153–156 (a v0.11 HA test from
+2026-10-03, idle since, holding 10.0.0.160 with VRRP id 51) were removed.
+
 ---
 
 ## A. Security — do first
@@ -1429,6 +1464,14 @@ console show and add them.
 - Done 2026-10-15: llmgw's `/mcp` with picture, 3D, VLM and workflow tools,
   so a client needs no server of its own (asked by the game team); the
   shared ComfyUI replica (H3's placement side).
+- Done 2026-10-16 (v0.14.6, v0.14.7): setup lists every part with what it
+  is and its settings; three machines each run the control plane, with a
+  floating address (VRRP id from the address); a lost store primary costs no
+  request (3 s connect per member, keepalive and TCP_USER_TIMEOUT, Patroni
+  ttl 20, a standby leaves the lease alone a round after the store returns);
+  a new leader waits for its judge within the budget; a judge is asked again
+  on another machine at half its budget; init sizes the judge (2 GB, no
+  prompt cache, two cores, one judge on up to four cores on one machine).
 
 ## I. What other gateways do that llmgw does not (written 2026-10-16)
 
@@ -1485,7 +1528,16 @@ in every client with its own credentials and nobody seeing what was called.
   through it, a key without the grant does not see the server's tools, and
   every call is in `ctl audit`.
 
-### I3. What llmgw adds to a request, measured (S)
+### I3. What llmgw adds to a request, measured (S) — done 2026-10-16
+`llmgw ctl bench overhead` (`internal/cli/bench_overhead.go`): an instant
+engine and the gateway in one process, four modes (direct, bare, auth, auth
+and guard patterns) at rising rates. On an Apple M5 the gateway added
+0.05–0.42 ms at p50 and at most 0.49 ms at p99, in the README.
+`TestTheGatewaysOverheadIsMeasured` runs a second of it in CI with no bound
+asserted. Left: the CPU figure is rough with everything in one process; a
+gateway in a process of its own would measure it properly. The original
+item follows.
+
 Bifrost's headline is "11 µs at 5k RPS" (its own figure, and its own pages
 disagree with it). llmgw has none, so any question about its overhead is
 answered with a guess.
@@ -1500,7 +1552,13 @@ answered with a guess.
 - Done when: the README states the overhead, with how it was measured, and
   CI runs the short version so a change that doubles it is seen.
 
-### I4. Several keys for one outside provider (S)
+### I4. Several keys for one outside provider (S) — done 2026-10-17
+`external.api_key_files` (`catalog.ExternalKey`: file, weight), one endpoint
+each in `control.externalEndpoints`, 401 retried on another key and
+`external_key_refused` notified (`router.keyRefused`). With one key a 401 is
+still the client's. `TestAKeyAtItsLimitOrRefusedIsPassedOverForAnother`. The
+original item follows.
+
 An external model has one `api_key_file` (`internal/control/control.go`,
 where external endpoints are made). A provider's limit is per key; one key
 answering 429 makes the whole fallback answer 429.
@@ -1513,9 +1571,101 @@ answering 429 makes the whole fallback answer 429.
 - Done when: with two keys and one returning 429, every request is answered
   by the other, and the console shows the one breaker open.
 
+## J. Found testing HA and single-machine installs (written 2026-10-16)
+
+Three VMs installed as HA from the setup screens, a fourth as a judge node,
+a fifth as a single 4-core, 8 GB machine; machines powered off under load.
+
+### J1. A restart refused for room another restart was giving back (S) — done 2026-10-16
+Not a wait for ever, as first written: on the 4-core VM the judge's restart
+was *refused* ("fits on no node even with this replica stopped") because the
+starter, restarted at the same time to reserve fewer cores, still held its
+old ones. Once refused, nothing tried again, so the judges kept their old
+figures. Two fixes in `internal/control/reconciler`:
+- a restart whose replacement fits once other restarts under way have gone
+  (queued, retiring or draining) waits for them, up to DrainGrace, instead
+  of being refused (`fitsAfterRestarts`, restart.go);
+- a drained replica counts a retiring one that still answers as serving, so
+  one restart after another no longer leaves the first's drained replica,
+  and the room it holds, waiting out its grace (reconciler.go).
+`TestARestartWaitsForRoomAnotherRestartWillGiveBack` is the 4-core case.
+
+### J2. A judge the leader adds is sized as init's is (S)
+When a policy names a judge the catalog lacks, the leader serves it itself
+(`internal/cli/judgekeeper.go` via `knownJudge.serveRequest` in
+`judges_known.go`, through modelserve). That judge reserves 3 GB (the
+weights, an 8192-token KV cache and a fixed gigabyte), no cores, and runs
+with llama.cpp's prompt cache, which grew one to 3.2 GB. init's judge
+(`guardModel` in `internal/cli/serverinit_cluster.go`) reserves 2 GB and
+its cores and runs with `--cache-ram 0`.
+- `modelserve` gives a guard model `--cache-ram 0`, and the keeper's
+  request the same memory and cores rule as `cpuPlan`.
+- Done when: a judge removed from the catalog comes back with the same
+  figures init would have given it.
+
+### J3. HA setup puts the judges on nodes of their own (S)
+Under load on a 4-core control plane that was also a node, 38 of 60
+requests were refused for want of a judgment. With the judges on a node of
+their own (4 cores, 32 GB, two cores each) the same test refused 2.
+- The HA setup offers a judge node or two: machines given like the store's,
+  added as nodes with a label (`llmgw.role: judge`), and the judge's
+  profile selects that label.
+- Done when: the setup installs the judges on the machines given for them,
+  and the load test above refuses no more than it did by hand.
+
+### J4. The model mirror on more than one machine (M)
+In an HA install the mirror stays on the first machine. While it is down,
+models the nodes already have keep serving and new ones wait for it.
+- A second mirror kept in step (each machine of the store, or one more), the
+  control planes and nodes given both, the first that answers used.
+- Done when: with the first machine powered off, a model added then is
+  fetched and served.
+
+### J5. An upgrade corrects an earlier install's judge (S) — done 2026-10-16
+Built in the judge keeper (`internal/cli/judgekeeper.go`, `refit`), on the
+leader: a judge whose entry is exactly what init before v0.14.7 wrote is
+given 2 GB, `--cache-ram 0` and cores by init's rule (`refitCores`: up to
+four for one judge, two each for more), through `modelsvc.SetProfiles`,
+and its ready replicas are restarted (after J1). Its replica count is left
+as it is. One that differs is left alone and said once in the log. The
+original item follows.
+
+v0.14.6 and earlier installed two judges reserving no cores or one, with
+2 or 4 GB and the prompt cache on. On fewer than eight cores every judgment
+missed its 5 s budget and every request was refused. v0.14.7's notes say
+how to fix it by hand (`ctl model resources`, then `ctl restart`).
+- On upgrade, or on `server init` run again, a judge whose profile is still
+  exactly what an earlier init wrote is moved to today's figures and
+  restarted (after J1). One changed by hand is left alone and named.
+- Done when: a v0.14.6 single-machine install upgraded to it judges within
+  budget without anything typed.
+
+### J6. Rate limits per process (note)
+Rate limits are counted in each control plane's memory. A failover starts
+every key's minute afresh. Documented in install.md; worth a shared count in
+the store only if a customer's limits must hold across a failover.
+
+### J7. nginx in front of several control planes (S)
+`deploy/proxy/Caddyfile` sends to whichever control plane answers `/leader`.
+nginx's open-source build has no active health checks, so its `upstream`
+block would send to standbys and to a lost leader alike.
+- nginx.conf: the floating address as its one upstream, and why; or the
+  `max_fails` passive approach with what it costs.
+
+### J8. A judge budget that fits the machine (S)
+A judgment takes 2.3 s on four cores and 4.2 on two, against the 5 s
+default: two cores are close to it, and a busy machine passes it.
+- At install, after the judge is up, time three judgments. If the slowest is
+  more than half the budget, say so in the plan's closing text with the
+  choices (more cores, a GPU judge, a longer `latency_budget`, `fail:
+  open`). The budget itself is not changed without being asked.
+- Done when: an install on a machine too slow for its judge says so before
+  its first refused request.
+
 ## D. Facts worth knowing before starting
 
-- Admin API: `127.0.0.1:9090`, no auth, `ctlClient` in `internal/cli/ctl.go`.
+- Admin API: `127.0.0.1:9090`, behind the token in
+  `/var/lib/llmgw/admin-token` (A5), `ctlClient` in `internal/cli/ctl.go`.
   Metrics on `:9091` with `/prometheus/targets` beside them.
 - Store: `internal/store`, migrations under `internal/store/migrations/{sqlite,postgres}`;
   the PoC runs PostgreSQL under Patroni.

CHANGELOG.md

@@ -18,12 +18,151 @@ is only released once its **exit criterion** is met, not when its code lands.
 | v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released |
 | v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released |
 | v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
-| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines) |
+| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines; v0.14.7: the judge sized for its machine; v0.14.8: sign-in with Active Directory, several keys for one outside provider; v0.14.9: sign-in set up from the console, a node's console that shows the machine) |
 | v1.0.0 | 5 | What phase 5 has left: precise KV routing proven on vLLM, the block cost picker, defragmentation across NVLink domains, prefill/decode disaggregation | planned |
 
 Until v1.0.0 the minor version is the breaking one: configuration formats and Go
 APIs change between minors without a deprecation period.
 
+## v0.14.9 — 2026-10-19
+
+Sign-in set up from the console, with Active Directory tried before it is
+kept. A node's console now shows what the machine is doing, and a node's web
+page shows what it runs. Also fixes found trying Active Directory against a
+Samba domain controller on the test cluster. Sign-in through it, nested
+groups and a disabled account suspended by the sync all worked there.
+
+**Upgrading.**
+- No migration.
+- `auth.ldap` and `auth.oidc` are in the cluster's settings, as before. A
+  block written into `server.yaml` on a cluster with a store was never
+  read; the log said so. Put it in with `llmgw ctl config edit`, or set it up
+  again from the console's Sign-in screen.
+- A Samba DC's own autogenerated certificate is refused (a negative serial
+  and no SAN). Give it one from a CA.
+
+### Added
+
+- **Sign-in set up from the console.** Under People → Sign-in, `sudo llmgw` shows how people get in: single sign-on,
+  Active Directory and SCIM, each on or off. It also shows the directory's last sync. **E** sets up Entra ID from the
+  tenant and client IDs. **D** sets up Active Directory, and the directory is tried before it is kept: each server
+  reached and bound, how many people may sign in, how many are in each group. Nothing is kept if a step fails. The
+  password, client secret and SCIM token are kept in the store as `store:auth/…`, sealed by the master key. The
+  domain's CA goes into the settings (`auth.ldap.ca`), so no file has to be put on each control plane. The admin API
+  has the same: `GET /api/v1/auth`, `POST /api/v1/auth/ldap/test` and `PUT /api/v1/auth/secrets/{name}`.
+- **A node's page says what it runs.** It now shows the agent's version, marked when the control plane runs another,
+  and the engines with their versions. Each GPU row now names the models loaded on that card, and each engine row its
+  GPUs.
+- **`sudo llmgw` on a node opens on what the machine is doing.** A new Overview screen shows the agent's state and
+  version, each control plane it dials and whether it answers, every GPU with the models on it, the engines running
+  here, the free space for weights and engines, and the agent's warnings of the last hour. It is read on the node
+  itself, every three seconds, so it needs no admin token and works while the control plane is out of reach.
+
+### Fixed
+
+- **A broken `auth.ldap` no longer stops every control plane.** It is the cluster's setting. A CA file that one
+  machine could not use made `llmgw server` exit on every control plane, and no `ctl config` could then put it right.
+  Directory sign-in is now turned off on that control plane, with an error in its log. Local accounts still sign in.
+- **A second `llmgw server` on a machine stops before it competes to lead.** `llmgw server check` (no such command)
+  ran a whole server. Its ports were taken, yet it went on to hold the leader's lease for three hours while the real
+  control planes stood by. `server` now refuses words it does not know and takes its ports before anything else.
+- **The single sign-on and Active Directory guides said `server.yaml`.** On a cluster with a store, `auth` is in the
+  cluster's settings and the file's block is not read. They now say `llmgw ctl config edit`, and what certificate a
+  domain controller needs.
+
+## v0.14.8 — 2026-10-18
+
+Sign-in with an on-premises Active Directory, and Entra ID made safe for
+people in many groups. Also several keys for one outside provider, and a
+measure of what the gateway adds to a request.
+
+**Upgrading.**
+- Migration 0048 adds the `user_directory` table, applied when the control
+  plane starts.
+- Nothing changes until `auth.ldap` is set; docs/install.md, "Active
+  Directory", has an example. It has been tested against a fake directory,
+  not yet against a real domain controller.
+- A judge installed by an earlier `server init` is refitted by the leader
+  by itself (below); the steps given for v0.14.7 are no longer needed.
+
+### Added
+
+- **Sign-in with Active Directory** (`auth.ldap`). People sign in on the gateway's form with their AD username or mail
+  and password, checked against the domain controllers over LDAPS or StartTLS. Groups, nested ones included, give
+  staff roles and places in the tree, as with OIDC. Accounts are tied to the `objectGUID` and made at first sign-in
+  when `create_accounts` is on. Every 15 minutes the leader reads the directory. People disabled or deleted there are
+  suspended and reinstated when they come back; roles and memberships follow their groups. A sync that would suspend
+  more than half the accounts is refused. See docs/install.md, "Active Directory".
+
+- **Several keys for one outside provider.** An external model takes `api_key_files`, each with an optional weight, in
+  place of `api_key_file`. Each key is an endpoint of its own (`external:MODEL#1`, `#2`), so a key answering 429 is
+  passed over while its breaker is open and the request goes to another. A key answering 401 is passed over too, and
+  `external_key_refused` is sent, naming the endpoint, never the key. Usage records which key answered by that name.
+
+- **`llmgw ctl bench overhead`**: what the gateway adds to a request, measured against an engine that answers at
+  once. Gateway and engine run in one process on loopback, so no cluster is needed. The load is sent straight to the
+  engine and through the gateway: bare, with authentication and accounting, and with the guardrails' pattern checks.
+  It reports p50 and p99 latency, what the gateway added, and a rough CPU per thousand requests. On an Apple M5 the
+  gateway added 0.05–0.42 ms at p50 and at most 0.49 ms at p99 (README).
+
+### Fixed
+
+- **Entra ID's groups past 200 no longer take a role away.** For someone in more than 200 groups Entra sends none in
+  the token, only a pointer to Graph. That was read as no groups, so an administrator lost their role at the next
+  sign-in. Their role and memberships are now left as they were, and an error is logged saying what to change in the app
+  registration. docs/install.md has an Entra ID section.
+- **An account made by SCIM or OIDC could not be reinstated.** Setting it active was refused as "never approved".
+- **An upgrade refits the judge an earlier `server init` installed.** Upgrading to v0.14.7 left such a judge as it was:
+  no cores or one, 2 or 4 GB, the prompt cache on. On fewer than eight cores every request was refused until it was
+  fixed by hand. Now the leader finds a judge whose entry is exactly what an earlier init wrote and gives it 2 GB, no
+  prompt cache and cores by today's rule (up to four for one judge, two each for more). It then restarts the judge's
+  replicas onto that. The number of replicas is left as it is. A judge whose entry was changed by hand is left alone,
+  and the log says so once.
+- **A restart is no longer refused for room another restart is giving back.** With every core of a node reserved, two
+  models restarted together, one to need more and one less (as after `ctl model resources` on a judge and a starter),
+  had the first refused: "fits on no node even with this replica stopped", though the second was about to give back
+  its cores. A restart now waits for other restarts under way, up to the drain grace, before it is refused. A drained
+  replica also counts a replica being restarted that still answers as serving, so replicas restarted one after
+  another no longer wait out the five-minute grace each.
+
+## v0.14.7 — 2026-10-16
+
+The judge installed by `server init` sized for what it does. On a machine
+of fewer than eight cores v0.14.6 left every judgment past its budget, so
+`fail: closed` refused every request. A single-machine install of v0.14.7
+on a 4-core VM passed 20 of 20, and on 8 cores 20 of 20 under load.
+
+**Upgrading.**
+- No migration.
+- **Upgrading does not change a judge already in the catalog.** v0.14.6
+  and earlier installed two judges, reserving no cores or one, and 2 or
+  4 GB. To give them what they need:
+  `llmgw ctl model resources qwen3guard-0.6b --memory 2 --cores 2`, then
+  `llmgw ctl restart --model qwen3guard-0.6b`. The two then need four cores
+  on a single machine; below that, a longer `latency_budget` or
+  `fail: open`.
+- On a node whose cores are all reserved, `ctl restart --model` cannot
+  start the new replica beside the old one and waits. Stop the old engine
+  process, and the replica comes back with the new figures.
+
+### Fixed
+
+- **Both judges fit an 8 GB machine again.** v0.14.6 had `server init` reserve 4 GB a judge replica. That figure came
+  from one replica at 3.2 GB after a day of traffic. The growth was llama.cpp's prompt cache, up to 8 GB by default,
+  which a judge never reuses: each text it reads is different. On a 4-core, 8 GB machine the second judge then fitted
+  nowhere ("has 7 GB of memory with 1 free, needs 4"). The judge now runs with `--cache-ram 0`, uses about 1.6 GB and
+  stays there, and reserves 2.
+- **One machine runs one judge, on up to four cores.** A second replica on the same machine shares its cores and is
+  lost with it, so it bought nothing but half the speed of each. `server init` on a single machine now serves one judge
+  on up to four cores (2.3 s a judgment where two on two cores took 4.2). Where there will be other nodes, it serves two
+  on two cores each, as in HA setups. `--judge-replicas` and the setup's Copies still choose.
+- **A judge gets the two cores it needs.** v0.14.6 gave each judge replica one core on machines under eight. A judgment
+  of about 280 tokens takes 8.0 s on one core, 4.2 s on two and 2.3 s on four, against the 5 s budget, so on a 4-core
+  machine every judgment timed out and every request was refused. Where there are two judges, each now reserves two
+  cores (one only below four CPUs), and the one judge of a single machine up to four; the starter has what is left, or
+  shares when nothing is. A judge on two cores stays close to its budget: give it more cores or a GPU, or raise
+  `latency_budget`, where requests must not wait.
+
 ## v0.14.6 — 2026-10-16
 
 Setup that says what each part of an install is and lets every one be

README.md

@@ -40,7 +40,7 @@ Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
 | v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released |
 | v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released |
 | v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
-| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines) |
+| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines; v0.14.7: the judge sized for its machine; v0.14.8: sign-in with Active Directory, several keys for one outside provider; v0.14.9: sign-in set up from the console, a node's console that shows the machine) |
 | v1.0.0 | 5 | What phase 5 has left: precise KV routing proven on vLLM, the block cost picker, defragmentation across NVLink domains, prefill/decode disaggregation | planned |
 
 As of v0.6.0 a cluster runs unattended: agents report their hardware, the
@@ -257,6 +257,33 @@ Add `--json` to any command for machine-readable output.
 says nothing about which of six constraints bit, so it prints every candidate
 considered and the reason it was rejected.
 
+## What llmgw adds to a request
+
+Measured with `llmgw ctl bench overhead`. It starts an engine that answers
+at once and a gateway in front of it, in one process on loopback. It sends
+the same load straight to the engine and through the gateway, and reports
+what the gateway added. Run on an Apple M5 (10 CPUs, load about 2) on
+2026-10-16, 10 s a step, 72,000 requests with none failed:
+
+| Through the gateway | added at p50 | added at p99 |
+|---|---|---|
+| bare | 0.06–0.24 ms | 0.10–0.33 ms |
+| with authentication and usage accounting | 0.05–0.32 ms | 0.09–0.37 ms |
+| with the guardrails' pattern checks as well | 0.10–0.42 ms | 0.15–0.49 ms |
+
+The range is across 100 to 2,000 requests a second. The overhead was
+*lowest* at 2,000, most likely because a busier processor keeps its cores
+awake. The guardrails' judge model is not in these figures: a judgment
+takes seconds on CPUs (see [install.md](docs/install.md)) and is a model's
+time, not the gateway's.
+
+The command also prints the gateway's CPU per thousand requests. Gateway,
+engine and load share one process there, so that figure is rough (0.04 to
+0.65 s per thousand in the run above); the latency is the one to quote.
+Figures from a VM on a busy host move with the host: run it where you will
+deploy, and keep the load it prints beside the result. CI runs a one-second
+version on every change (`TestTheGatewaysOverheadIsMeasured`).
+
 ## Running the router
 
 ```sh

docs/admin-api.openapi.json

@@ -23,6 +23,9 @@
     {
       "name": "audit"
     },
+    {
+      "name": "auth"
+    },
     {
       "name": "canaries"
     },
@@ -226,6 +229,129 @@
         }
       }
     },
+    "/api/v1/auth": {
+      "get": {
+        "operationId": "getAuth",
+        "summary": "How people sign in: OIDC, a directory, SCIM, and the directory's last sync",
+        "tags": [
+          "auth"
+        ],
+        "responses": {
+          "200": {
+            "description": "OK",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/AuthView"
+                }
+              }
+            }
+          },
+          "401": {
+            "$ref": "#/components/responses/Unauthorized"
+          },
+          "default": {
+            "$ref": "#/components/responses/Error"
+          }
+        }
+      }
+    },
+    "/api/v1/auth/ldap/test": {
+      "post": {
+        "operationId": "postAuthLdapTest",
+        "summary": "Try an auth.ldap block before it is kept: servers, the service account, people and groups",
+        "tags": [
+          "auth"
+        ],
+        "parameters": [
+          {
+            "$ref": "#/components/parameters/Actor"
+          }
+        ],
+        "requestBody": {
+          "required": true,
+          "content": {
+            "application/json": {
+              "schema": {
+                "$ref": "#/components/schemas/LDAPTestRequest"
+              }
+            }
+          }
+        },
+        "responses": {
+          "200": {
+            "description": "OK",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/LDAPTestView"
+                }
+              }
+            }
+          },
+          "401": {
+            "$ref": "#/components/responses/Unauthorized"
+          },
+          "default": {
+            "$ref": "#/components/responses/Error"
+          }
+        }
+      }
+    },
+    "/api/v1/auth/secrets/{name}": {
+      "put": {
+        "operationId": "putAuthSecretsName",
+        "summary": "Keep ldap-bind, oidc-client-secret or scim-token in the store, for settings to name as store:auth/NAME",
+        "tags": [
+          "auth"
+        ],
+        "parameters": [
+          {
+            "name": "name",
+            "in": "path",
+            "required": true,
+            "description": "ldap-bind, oidc-client-secret or scim-token.",
+            "schema": {
+              "type": "string"
+            }
+          },
+          {
+            "$ref": "#/components/parameters/Actor"
+          }
+        ],
+        "requestBody": {
+          "required": true,
+          "content": {
+            "application/json": {
+              "schema": {
+                "$ref": "#/components/schemas/SecretRequest"
+              }
+            }
+          }
+        },
+        "responses": {
+          "200": {
+            "description": "OK",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "type": "object",
+                  "additionalProperties": {
+                    "type": "string"
+                  }
+                }
+              }
+            }
+          },
+          "401": {
+            "$ref": "#/components/responses/Unauthorized"
+          },
+          "default": {
+            "$ref": "#/components/responses/Error"
+          }
+        }
+      }
+    },
     "/api/v1/canaries": {
       "get": {
         "operationId": "getCanaries",
@@ -5351,6 +5477,31 @@
           }
         }
       },
+      "AuthView": {
+        "type": "object",
+        "description": "AuthView is how people sign in on this gateway.",
+        "properties": {
+          "signup_open": {
+            "type": "boolean"
+          },
+          "oidc": {
+            "$ref": "#/components/schemas/OIDCView",
+            "description": "OIDC is single sign-on through an identity provider: Entra ID, Keycloak, Okta."
+          },
+          "ldap": {
+            "$ref": "#/components/schemas/LDAPView",
+            "description": "LDAP is sign-in checked by a directory: Active Directory."
+          },
+          "scim": {
+            "type": "boolean",
+            "description": "SCIM is whether a provider may keep accounts in step."
+          },
+          "base_url": {
+            "type": "string",
+            "description": "BaseURL is the portal's address, when the settings say it: what a provider's redirect address is made from."
+          }
+        }
+      },
       "CanaryPercentRequest": {
         "type": "object",
         "description": "CanaryPercentRequest changes the share of a model's requests its canary takes.",
@@ -5421,6 +5572,21 @@
           }
         }
       },
+      "CheckView": {
+        "type": "object",
+        "description": "CheckView is one step of trying a directory.",
+        "properties": {
+          "step": {
+            "type": "string"
+          },
+          "ok": {
+            "type": "boolean"
+          },
+          "detail": {
+            "type": "string"
+          }
+        }
+      },
       "ClassGPUs": {
         "type": "object",
         "description": "ClassGPUs is how many of a node's devices are of one class, and how many of those are free or kept out of placement.",
@@ -5573,6 +5739,34 @@
           }
         }
       },
+      "DirectorySyncView": {
+        "type": "object",
+        "description": "DirectorySyncView is the last time the leader read the directory.",
+        "properties": {
+          "at": {
+            "type": "string",
+            "format": "date-time"
+          },
+          "people": {
+            "type": "integer"
+          },
+          "accounts": {
+            "type": "integer"
+          },
+          "suspended": {
+            "type": "integer"
+          },
+          "reinstated": {
+            "type": "integer"
+          },
+          "roles_changed": {
+            "type": "integer"
+          },
+          "error": {
+            "type": "string"
+          }
+        }
+      },
       "DomainView": {
         "type": "object",
         "description": "DomainView is one domain: its nodes, its power and whether it is held.",
@@ -6826,6 +7020,61 @@
           }
         }
       },
+      "LDAPTestRequest": {
+        "type": "object",
+        "description": "LDAPTestRequest is an auth.ldap block to try, as YAML, and the service account's password when it is not kept yet.",
+        "properties": {
+          "settings": {
+            "type": "string"
+          },
+          "password": {
+            "type": "string"
+          }
+        }
+      },
+      "LDAPTestView": {
+        "type": "object",
+        "description": "LDAPTestView is how trying the directory went.",
+        "properties": {
+          "ok": {
+            "type": "boolean"
+          },
+          "checks": {
+            "type": "array",
+            "items": {
+              "$ref": "#/components/schemas/CheckView"
+            }
+          }
+        }
+      },
+      "LDAPView": {
+        "type": "object",
+        "description": "LDAPView is auth.ldap as it is in force, and how its last sync went.",
+        "properties": {
+          "urls": {
+            "type": "array",
+            "items": {
+              "type": "string"
+            }
+          },
+          "base_dn": {
+            "type": "string"
+          },
+          "bind_dn": {
+            "type": "string"
+          },
+          "label": {
+            "type": "string"
+          },
+          "off": {
+            "type": "string",
+            "description": "Off is why this control plane could not use the settings, when it could not; local accounts still sign in."
+          },
+          "last_sync": {
+            "$ref": "#/components/schemas/DirectorySyncView"
+          }
+        }
+      },
       "Live": {
         "type": "object",
         "description": "Live is one of the cluster's adapters as its node last reported it.",
@@ -7492,6 +7741,27 @@
           }
         }
       },
+      "OIDCView": {
+        "type": "object",
+        "description": "OIDCView is auth.oidc as it is in force.",
+        "properties": {
+          "issuer": {
+            "type": "string"
+          },
+          "client_id": {
+            "type": "string"
+          },
+          "label": {
+            "type": "string"
+          },
+          "create_accounts": {
+            "type": "boolean"
+          },
+          "no_passwords": {
+            "type": "boolean"
+          }
+        }
+      },
       "OrgView": {
         "type": "object",
         "description": "OrgView is an organization with what is in it.",
@@ -8182,6 +8452,15 @@
           }
         }
       },
+      "SecretRequest": {
+        "type": "object",
+        "description": "SecretRequest is a secret for the store to keep.",
+        "properties": {
+          "value": {
+            "type": "string"
+          }
+        }
+      },
       "SettingsRequest": {
         "type": "object",
         "description": "SettingsRequest is a new version of the cluster's settings, made from the version Base, which must still be the newest.",
@@ -9054,6 +9333,13 @@
             "type": "string",
             "description": "APIKeyFile holds the provider's key, on the control plane. A file rather than the key itself, so the key is in neither the catalog, the store nor the audit log; empty for a provider that asks for none."
           },
+          "api_key_files": {
+            "type": "array",
+            "items": {
+              "$ref": "#/components/schemas/catalog.ExternalKey"
+            },
+            "description": "APIKeyFiles are several keys for the provider, in place of APIKeyFile: one endpoint each (external:MODEL#1, #2…), so a key that answers 429 or 401 is passed over while its breaker is open and the request goes to another. A provider's limits are per key."
+          },
           "peer": {
             "type": "string",
             "description": "Peer names the federated llmgw cluster this model is offered by, for a model that federation added (`llmgw ctl federation add`) rather than one written by hand. Such a model is named PEER/MODEL, comes and goes with the peer's own list, and is sent only while the peer is up."
@@ -9072,6 +9358,20 @@
           }
         }
       },
+      "catalog.ExternalKey": {
+        "type": "object",
+        "description": "ExternalKey is one of a provider's keys.",
+        "properties": {
+          "file": {
+            "type": "string",
+            "description": "File holds the key, on the control plane, as APIKeyFile does."
+          },
+          "weight": {
+            "type": "number",
+            "description": "Weight is its share of the requests against the others'; zero is 1."
+          }
+        }
+      },
       "catalog.GPUNeed": {
         "type": "object",
         "description": "GPUNeed is what card a profile needs.",

docs/console.md

@@ -2,8 +2,8 @@
 
 `sudo llmgw`, with nothing after it, on a machine llmgw runs on. On a machine
 with nothing installed it opens the setup screens; on a control plane, the
-console described here; on a node, the node's own console, which edits
-`agent.yaml`. Everything on these screens can also be done by command — each
+console described here; on a node, the node's own console, which shows
+what runs on that machine and edits `agent.yaml`. Everything on these screens can also be done by command — each
 screen says which — so a script never needs them and a person never needs
 to remember the commands.
 
@@ -246,6 +246,28 @@ long a key made there may live, and is the expiry of one made without any
 Limits are requests and tokens a minute, a budget per day or month, and the
 models allowed. Each level only narrows what is above it.
 
+**Sign-in** — how people get in besides a password kept here: single
+sign-on, Active Directory and provisioning, each on or off, with the
+directory's last sync (how many people, accounts kept in step, suspended,
+reinstated).
+- **E** sets up Entra ID. Give the tenant ID, the client ID and secret,
+  and the groups for administrators and operators. A provider other than
+  Entra takes its issuer URL in place of the tenant. The form shows the
+  redirect address to register.
+- **D** sets up Active Directory: the servers, where people are, the
+  service account and its password, the domain's CA certificate as a PEM
+  file on this machine, the group whose members may sign in, and groups
+  for roles and teams.
+- Before it is kept, the directory is tried from the control plane: each
+  server reached and bound, how many people may sign in, how many are in
+  each group. If any step fails, nothing is kept and the steps are shown.
+  **T** tries the settings in force.
+- On the provisioning row, Enter makes a SCIM token, shown once.
+- The password, the client secret and the token are kept in the store,
+  sealed by the master key when one is set. The settings name them as
+  `store:auth/…`, so each control plane has them with no file put on it.
+  The CA is written into the settings themselves.
+
 **Guardrails…** on a row lists the guardrail policies server.yaml defines.
 Each policy is marked in one of three ways:
 
@@ -335,5 +357,18 @@ to restart it.
 
 ## The node's console
 
-On a node, `sudo llmgw` opens `agent.yaml` in the same settings view, and the
-manual. The cluster is watched and changed from a control plane.
+On a node, `sudo llmgw` opens on **Overview**: what this machine is doing, read
+on the machine itself, so it works without an admin token and while the control
+plane is out of reach. Refreshed every three seconds, it shows
+
+- llmgw-agent's state and since when, and the version of llmgw;
+- each control plane the agent dials, from `agent.yaml` and the list the
+  leader last sent, and whether it can be reached;
+- each GPU: how busy, its memory, its heat, and the models on it;
+- the engines running here, from the agent's own records: replica, model,
+  engine, GPUs, port, whether the process is still there, and for how long;
+- the free space where weights and engines are kept;
+- the agent's warnings and errors of the last hour.
+
+Below it, `agent.yaml` in the same settings view, and the manual. The cluster
+is watched and changed from a control plane.

docs/install.md

@@ -152,9 +152,9 @@ That is the whole step. In order, it:
    `nvidia-smi` finds any, on CPU otherwise — and waits for it to install its
    engine
 9. closes llmgw's internal ports to everything but the cluster's networks
-10. fetches Qwen3Guard-Gen 0.6B, the guardrails' judge, and runs two
-    replicas on the CPUs of nodes with room — this one, or the first that
-    join ([guardrails.md](guardrails.md#a-judge-model) has one on a GPU too)
+10. fetches Qwen3Guard-Gen 0.6B, the guardrails' judge, and runs it on
+    CPUs: one replica on up to four cores when this machine is the only
+    node, two on two cores each on the nodes of a larger cluster ([guardrails.md](guardrails.md#a-judge-model) has one on a GPU too)
 11. fetches Qwen2.5 0.5B into the mirror and serves it on this node, so the
     first thing anyone opens `/chat` to find is a model that answers. A
     machine without room for it is told so within two minutes, and why.
@@ -187,15 +187,19 @@ demonstration one:
 | Requests | every key limited (60 a minute, 200 000 tokens a minute) and priced, so budgets bind |
 | Guardrails | the `baseline` policy on every request: secrets in prompts and answers refused, card numbers masked, injection attempts recorded, and the judge refusing violence, crime, weapons, child abuse, privacy violations, hate, self-harm and sexual content, in prompts and answers |
 
-The judge runs as two replicas, each on a CPU — about half a second for a
-short text on any CPU with AVX2 — and reads a streamed answer every 600
-characters. The policy fails **closed**: a request the judge cannot read
-within 5 seconds is refused with 503 rather than served unjudged, and
-`judge_unavailable` goes to the notification webhooks. Two replicas are what
-make that bearable; on one small machine, or a VM without AVX2 (see
+The judge runs on CPUs. A judgment of about 280 tokens took 2.3 s on four
+cores, 4.2 s on two and 8.0 s on one (a 4-core VM). It also reads a
+streamed answer every 600 characters. The policy fails **closed**: a request
+the judge cannot read within 5 seconds is refused with 503 rather than
+served unjudged, and `judge_unavailable` goes to the notification webhooks.
+So a judge gets cores of its own: on one machine one replica on up to four,
+and on a larger cluster two replicas of two cores each, spread over its nodes
+so that losing one leaves the other. On a machine of fewer than four cores,
+a busy one, or a VM without AVX2 (see
 [step 6](#6-setting-the-nodes-cpu-model-if-it-is-a-vm)), the judge is the
-first thing to fail, so `fail: open` — serve it, recorded as flagged — or
-`--no-guard-model` is the choice there. Defamation, specialized advice,
+first thing to fail. The choices there are a longer `latency_budget`,
+`fail: open` (serve the request, recorded as flagged), the judge on a GPU,
+or `--no-guard-model`. Defamation, specialized advice,
 intellectual property and elections are left out of what it refuses: they
 catch too much ordinary work. [guardrails.md](guardrails.md) has the rest.
 
@@ -1319,6 +1323,26 @@ catalog, and a key rotated in the file is used within seconds. A model whose key
 file cannot be read is not offered. Everything else applies as to any model:
 prices and budgets, rate limits, usage records, and key allowlists.
 
+A provider's limits are per key. With several keys, one answering 429 need not
+hold up the model: give them all, each with an optional weight (its share
+against the others'), in place of `api_key_file`:
+
+```yaml
+  external:
+    url: https://api.openai.com
+    model: gpt-4o-mini
+    api_key_files:
+      - {file: /etc/llmgw/secrets/openai-1.key, weight: 2}
+      - {file: /etc/llmgw/secrets/openai-2.key}
+```
+
+Each key is an endpoint of its own, `external:cloud-mini#1`, `#2`. A key that
+answers 429 is passed over while its breaker is open, and the request goes to
+another. One that answers 401 (revoked, expired, mistyped) is passed over too,
+and `external_key_refused` goes to the notification webhooks, naming the
+endpoint and never the key. The usage records say which key answered by the
+same name. A key whose file cannot be read is left out, and the others stay.
+
 **Its prompts leave the cluster.** Decide who may send them: a key or project
 whose allowlist leaves the external model out cannot reach it, as a fallback or
 by name. The cluster page and `ctl models` mark it `external`.
@@ -2467,8 +2491,10 @@ Google, Okta, Authentik, anything that speaks OpenID Connect.
 
 1. Register a client there with the redirect address
    `https://<gateway>/login/oidc/callback`.
-2. Put its secret in a file only root reads.
-3. Add to `server.yaml`:
+2. Put its secret in a file the llmgw user reads, on every control plane.
+3. Add this to the cluster's settings with `llmgw ctl config edit`. On a
+   cluster with a store, `auth` is the cluster's: the same block in
+   `server.yaml` is not read, and the log says so.
 
 ```yaml
 auth:
@@ -2517,6 +2543,99 @@ and point it at `https://<gateway>/scim/v2` with that token as the bearer.
   join and leave them. Membership in a group is removed when someone leaves
   it.
 
+### Entra ID
+
+The console sets this up: `sudo llmgw`, People → Sign-in, **E**. What
+follows is what it writes, and what to do in Entra.
+
+Entra ID is an OpenID Connect provider like the others, with two things to
+set in the app registration:
+
+1. **Redirect URI:** under Authentication, add a Web platform with
+   `https://<gateway>/login/oidc/callback`. The issuer is
+   `https://login.microsoftonline.com/<tenant-id>/v2.0`.
+2. **Groups in the token:** under Token configuration, add a groups claim.
+   Choose **Groups assigned to the application**, and assign the groups you
+   map in `roles` and `memberships` to the enterprise application. Entra
+   sends groups by object id unless you choose to emit names
+   (`sAMAccountName` for groups synced from AD), so write them the same way
+   in the mapping.
+
+Entra leaves the groups out of the token for someone in more than 200 of
+them and sends a pointer to Graph instead. llmgw then keeps that person's
+role and memberships as they were, rather than reading it as no groups, and
+logs an error saying so. "Groups assigned to the application" avoids this.
+So does SCIM below, which Entra also speaks: in the enterprise application
+under Provisioning, set the tenant URL to `https://<gateway>/scim/v2` and
+the secret token to the contents of `auth.scim.token_file`.
+
+## Active Directory
+
+With an on-premises Active Directory, people sign in on the gateway's own
+form with their AD username (`ada`, `CORP\ada`, `ada@corp.example`) and
+password. The gateway checks them with the domain controllers over LDAPS.
+
+The console sets this up and tries it first: `sudo llmgw`, People →
+Sign-in, **D**. It keeps the service account's password in the store
+(`bind_password_file: store:auth/ldap-bind`) and writes the CA into the
+settings (`ca:`), so nothing has to be copied to each control plane.
+By hand it is this block in the cluster's settings (`llmgw ctl config
+edit`), where files it names must then be on every control plane:
+
+```yaml
+auth:
+  enabled: true
+  ldap:
+    urls: [ldaps://dc1.corp.example, ldaps://dc2.corp.example]   # tried in order
+    ca_file: /etc/llmgw/corp-ca.pem         # the domain's CA, when not in the system's
+    bind_dn: CN=svc-llmgw,OU=Service,DC=corp,DC=example
+    bind_password_file: /etc/llmgw/secrets/ldap-bind   # root-only
+    base_dn: OU=Staff,DC=corp,DC=example    # where people are
+    user_filter: (memberOf:1.2.840.113556.1.4.1941:=CN=llmgw-users,OU=Groups,DC=corp,DC=example)
+    label: Acme                             # "Sign in with your Acme account"
+    create_accounts: true
+    roles:                                  # by group CN or full DN; nested groups count
+      admin: [llmgw-admins]
+      operator: [llmgw-ops]
+    memberships:
+      ml-engineers: "team:Acme/ml"
+    sync_interval: 15m                      # negative turns the sync off
+```
+
+- **Service account:** an ordinary domain user is enough: it only reads.
+  It finds the person and reads their groups. The person's password is
+  checked by binding as them, and is never stored.
+- **Encryption:** `ldaps://`, or `ldap://` with `start_tls: true`.
+  Plaintext LDAP is refused, since passwords would cross the network.
+- **Certificate:** the domain controller's certificate must name it in
+  its subject alternative names: a name in the CN only is refused, as are
+  certificates with a negative serial number. Samba's own autogenerated
+  certificate is both, so a Samba DC needs one from a CA (`tls certfile`
+  in smb.conf). A CA file a control plane cannot read turns directory
+  sign-in off on that control plane, said in its log. Local accounts still
+  sign in.
+- **Who may sign in:** enabled user objects under `base_dn` that match
+  `user_filter`. The example admits members of `llmgw-users`, nested groups
+  included.
+- **Accounts:** an account is tied to the person's `objectGUID`, so a changed
+  mail or name is still the same account. An account that already exists
+  with the person's mail is taken over. Without `create_accounts`, only
+  people who already have an account can sign in. Local accounts, such as
+  the administrator `server init` made, keep signing in with their own
+  password.
+- **Sync:** every `sync_interval` the leader reads the directory.
+  - Someone disabled or deleted in AD is suspended and their sessions
+    end.
+  - When they are enabled again they are reinstated. An account an
+    administrator suspended by hand stays suspended.
+  - Roles follow groups, and memberships the directory gave are taken away
+    when the person leaves the group.
+  - A sync that would suspend more than half of the directory's accounts is
+    refused and logged, rather than locking everyone out over a wrong
+    `base_dn` or filter.
+- **Passwords:** a person's password is changed in AD. The gateway sends no
+  reset link for these accounts.
+
 ## Secrets in Vault
 
 Every setting that names a file holding a secret — the store's `@address`,

docs/reference.md

@@ -37,6 +37,7 @@ rather than repeated.
 | `default_tpm` | `int` | [Auth](#auth) |
 | `signup_open` | `bool` | [Auth](#auth) |
 | `oidc` | `OIDC` | [Auth](#auth) |
+| `ldap` | `LDAP` | [Auth](#auth) |
 | `scim` | `SCIM` | [Auth](#auth) |
 | `max_key_ttl` | `time.Duration` | [Auth](#auth) |
 | `personal` | `Personal` | [Auth](#auth) |
@@ -105,10 +106,13 @@ rather than repeated.
 | `url` | `string` | [External](#external) |
 | `model` | `string` | [External](#external) |
 | `api_key_file` | `string` | [External](#external) |
+| `api_key_files` | `[]ExternalKey` | [External](#external) |
 | `peer` | `string` | [External](#external) |
 | `api` | `string` | [External](#external) |
 | `workflow` | `string` | [External](#external) |
 | `output` | `string` | [External](#external) |
+| `file` | `string` | [ExternalKey](#externalkey) |
+| `weight` | `float64` | [ExternalKey](#externalkey) |
 | `budgets` | `[]FacilityBudget` | [Facility](#facility) |
 | `margin` | `float64` | [Facility](#facility) |
 | `gpu_watts` | `float64` | [Facility](#facility) |
@@ -144,6 +148,21 @@ rather than repeated.
 | `tokenize_timeout` | `time.Duration` | [KVIndex](#kvindex) |
 | `min_blocks` | `int` | [KVIndex](#kvindex) |
 | `max_blocks` | `int` | [KVIndex](#kvindex) |
+| `urls` | `[]string` | [LDAP](#ldap) |
+| `start_tls` | `bool` | [LDAP](#ldap) |
+| `ca_file` | `string` | [LDAP](#ldap) |
+| `ca` | `string` | [LDAP](#ldap) |
+| `bind_dn` | `string` | [LDAP](#ldap) |
+| `bind_password_file` | `string` | [LDAP](#ldap) |
+| `base_dn` | `string` | [LDAP](#ldap) |
+| `group_base_dn` | `string` | [LDAP](#ldap) |
+| `user_filter` | `string` | [LDAP](#ldap) |
+| `label` | `string` | [LDAP](#ldap) |
+| `create_accounts` | `bool` | [LDAP](#ldap) |
+| `roles` | `map[string][]string` | [LDAP](#ldap) |
+| `memberships` | `map[string]string` | [LDAP](#ldap) |
+| `sync_interval` | `time.Duration` | [LDAP](#ldap) |
+| `timeout` | `time.Duration` | [LDAP](#ldap) |
 | `contact` | `string` | [Landing](#landing) |
 | `notice` | `string` | [Landing](#landing) |
 | `language` | `string` | [Landing](#landing) |
@@ -616,6 +635,15 @@ OIDC is sign-in through an identity provider.
 
 Takes a [OIDC](#oidc) block.
 
+### `ldap`
+
+`LDAP`
+
+LDAP is sign-in against a directory, Active Directory above all, and
+the accounts kept in step with it.
+
+Takes a [LDAP](#ldap) block.
+
 ### `scim`
 
 `SCIM`
@@ -1103,6 +1131,15 @@ APIKeyFile holds the provider's key, on the control plane. A file
 rather than the key itself, so the key is in neither the catalog, the
 store nor the audit log; empty for a provider that asks for none.
 
+### `api_key_files`
+
+`[]ExternalKey`
+
+APIKeyFiles are several keys for the provider, in place of
+APIKeyFile: one endpoint each (external:MODEL#1, #2…), so a key that
+answers 429 or 401 is passed over while its breaker is open and the
+request goes to another. A provider's limits are per key.
+
 ### `peer`
 
 `string`
@@ -1135,6 +1172,20 @@ plane.
 
 Output is what it makes: pictures, or meshes (see ComfyUI.Output).
 
+## ExternalKey
+
+### `file`
+
+`string`
+
+File holds the key, on the control plane, as APIKeyFile does.
+
+### `weight`
+
+`float64`
+
+Weight is its share of the requests against the others'; zero is 1.
+
 ## Facility
 
 ### `budgets`
@@ -1401,6 +1452,114 @@ MaxBlocks bounds how many of a prompt's blocks are hashed and looked
 up. Beyond it a longer match still helps, but the difference no longer
 decides anything. Zero takes 4096.
 
+## LDAP
+
+### `urls`
+
+`[]string`
+
+URLs are the directory's servers, tried in turn: ldaps:// (636), or
+ldap:// (389) with StartTLS.
+
+### `start_tls`
+
+`bool`
+
+StartTLS upgrades an ldap:// connection before anything is sent. A
+password never crosses a connection without TLS: ldap:// without it
+is refused.
+
+### `ca_file`
+
+`string`
+
+CAFile is the authority the servers' certificates are checked against,
+when it is not the system's: an AD Certificate Services root.
+
+### `ca`
+
+`string`
+
+CA is the same authority written out, PEM: kept in the cluster's
+settings, so no file has to be put on every control plane. Used in
+place of CAFile when both are set.
+
+### `bind_dn`
+
+`string`
+
+BindDN and the file holding its password: a service account that may
+read users and groups. Nothing is written to the directory.
+
+### `bind_password_file`
+
+`string`
+
+### `base_dn`
+
+`string`
+
+BaseDN is where people are searched for, subtree included.
+
+### `group_base_dn`
+
+`string`
+
+GroupBaseDN is where their groups are; empty is BaseDN's domain root.
+
+### `user_filter`
+
+`string`
+
+UserFilter narrows who may sign in, ANDed with the login match: a
+group (memberOf=...) or an OU's people only. Empty is every enabled
+user object.
+
+### `label`
+
+`string`
+
+Label is what the form says to sign in with: "Sign in with your
+<label> account". Empty is "Active Directory".
+
+### `create_accounts`
+
+`bool`
+
+CreateAccounts makes an account, active, for a person the directory
+vouches for and the gateway does not know. Off, only existing accounts
+sign in this way.
+
+### `roles`
+
+`map[string][]string`
+
+Roles maps a staff role (viewer, operator, billing, admin) to the
+groups that give it, by name (CN) or whole DN. Set at every sign-in
+and sync: someone taken out of the group loses the role.
+
+### `memberships`
+
+`map[string]string`
+
+Memberships maps a group, by CN or DN, to a place in the tree —
+org:NAME, team:ORG/TEAM, project:ORG/TEAM/PROJECT — joined at sign-in
+and kept by the sync, which also takes away a place whose group the
+person has left.
+
+### `sync_interval`
+
+`time.Duration`
+
+SyncInterval is how often the leader reads the directory. Zero is
+15m; negative turns the sync off, leaving only sign-in.
+
+### `timeout`
+
+`time.Duration`
+
+Timeout bounds each round trip to a server. Zero is 10s.
+
 ## Landing
 
 ### `contact`

go.mod

@@ -12,6 +12,7 @@ require (
 	github.com/charmbracelet/x/ansi v0.11.6
 	github.com/coreos/go-oidc/v3 v3.21.0
 	github.com/go-jose/go-jose/v4 v4.1.4
+	github.com/go-ldap/ldap/v3 v3.4.15
 	github.com/go-zeromq/zmq4 v0.17.0
 	github.com/jackc/pgx/v5 v5.11.0
 	github.com/knadh/koanf/parsers/yaml v1.1.1
@@ -45,6 +46,7 @@ require (
 require (
 	filippo.io/edwards25519 v1.2.0 // indirect
 	filippo.io/hpke v0.4.0 // indirect
+	github.com/Azure/go-ntlmssp v0.1.1 // indirect
 	github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
 	github.com/cenkalti/backoff/v5 v5.0.3 // indirect
@@ -57,6 +59,7 @@ require (
 	github.com/dustin/go-humanize v1.0.1 // indirect
 	github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f // indirect
 	github.com/fsnotify/fsnotify v1.9.0 // indirect
+	github.com/go-asn1-ber/asn1-ber v1.5.8 // indirect
 	github.com/go-logr/logr v1.4.4 // indirect
 	github.com/go-logr/stdr v1.2.2 // indirect
 	github.com/go-viper/mapstructure/v2 v2.5.0 // indirect

go.sum

@@ -8,8 +8,12 @@ filippo.io/edwards25519 v1.2.0 h1:crnVqOiS4jqYleHd9vaKZ+HKtHfllngJIiOpNpoJsjo=
 filippo.io/edwards25519 v1.2.0/go.mod h1:xzAOLCNug/yB62zG1bQ8uziwrIqIuxhctzJT18Q77mc=
 filippo.io/hpke v0.4.0 h1:p575VVQ6ted4pL+it6M00V/f2qTZITO0zgmdKCkd5+A=
 filippo.io/hpke v0.4.0/go.mod h1:EmAN849/P3qdeK+PCMkDpDm83vRHM5cDipBJ8xbQLVY=
+github.com/Azure/go-ntlmssp v0.1.1 h1:l+FM/EEMb0U9QZE7mKNEDw5Mu3mFiaa2GKOoTSsNDPw=
+github.com/Azure/go-ntlmssp v0.1.1/go.mod h1:NYqdhxd/8aAct/s4qSYZEerdPuH1liG2/X9DiVTbhpk=
 github.com/NVIDIA/go-nvml v0.13.4-0 h1:o3jp9u2x1R9ShFE3v+Aesp55XOSIQFMJz/VGNUcJaNE=
 github.com/NVIDIA/go-nvml v0.13.4-0/go.mod h1:id63qwpoDWpFXwnwM6psDCSqW4BmNu6mWpr4YeQtPGo=
+github.com/alexbrainman/sspi v0.0.0-20250919150558-7d374ff0d59e h1:4dAU9FXIyQktpoUAgOJK3OTFc/xug0PCXYCqU0FgDKI=
+github.com/alexbrainman/sspi v0.0.0-20250919150558-7d374ff0d59e/go.mod h1:cEWa1LVoE5KvSD9ONXsZrj0z6KqySlCCNKHlLzbqAt4=
 github.com/aymanbagabas/go-osc52/v2 v2.0.1 h1:HwpRHbFMcZLEVr42D4p7XBqjyuxQH5SMiErDT4WkJ2k=
 github.com/aymanbagabas/go-osc52/v2 v2.0.1/go.mod h1:uYgXzlJ7ZpABp8OJ+exZzJJhRNQ2ASbcXHWsFqH8hp8=
 github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
@@ -46,8 +50,12 @@ github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f h1:Y/CXytFA4m6
 github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f/go.mod h1:vw97MGsxSvLiUE2X8qFplwetxpGLQrlU1Q9AUEIzCaM=
 github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
 github.com/fsnotify/fsnotify v1.9.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
+github.com/go-asn1-ber/asn1-ber v1.5.8 h1:H9AZkK22UOmfX8J84ubyaZxKJZ3FMHVwn8swoMML7iQ=
+github.com/go-asn1-ber/asn1-ber v1.5.8/go.mod h1:hEBeB/ic+5LoWskz+yKT7vGhhPYkProFKoKdwZRWMe0=
 github.com/go-jose/go-jose/v4 v4.1.4 h1:moDMcTHmvE6Groj34emNPLs/qtYXRVcd6S7NHbHz3kA=
 github.com/go-jose/go-jose/v4 v4.1.4/go.mod h1:x4oUasVrzR7071A4TnHLGSPpNOm2a21K9Kf04k1rs08=
+github.com/go-ldap/ldap/v3 v3.4.15 h1:eRB4f4Ecn1POL1mO86+uUDlecmyWsdiJwh60bAUS4UY=
+github.com/go-ldap/ldap/v3 v3.4.15/go.mod h1:jeD0c98EdZQCKKKKe4kTWs5aCf6y+j8KuYsYLznZSQk=
 github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
 github.com/go-logr/logr v1.4.4 h1:tG4xh9yMsRCAiodLVTxyrkzSZ9+o0L1Kg/+cPVcbP/8=
 github.com/go-logr/logr v1.4.4/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
@@ -69,6 +77,8 @@ github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
 github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
 github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0 h1:/Tnpcb2E0Pz/tN9s3bfEY2Q8ePCEX9iuS+cneUwncnw=
 github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0/go.mod h1:zOBXOsUaBSjKgmH4OGzV1esUpR3oUSCPYVd2cUBjKYY=
+github.com/hashicorp/go-uuid v1.0.3 h1:2gKiV6YVmrJ1i2CKKa9obLvRieoRGviZFL26PcT/Co8=
+github.com/hashicorp/go-uuid v1.0.3/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/bN7x4byOro=
 github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
 github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
 github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
@@ -81,6 +91,18 @@ github.com/jackc/pgx/v5 v5.11.0 h1:IzBBtyK9AHqf98cctWFifYSci2hgQR/cd56wB4p+ogg=
 github.com/jackc/pgx/v5 v5.11.0/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4=
 github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
 github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
+github.com/jcmturner/aescts/v2 v2.0.0 h1:9YKLH6ey7H4eDBXW8khjYslgyqG2xZikXP0EQFKrle8=
+github.com/jcmturner/aescts/v2 v2.0.0/go.mod h1:AiaICIRyfYg35RUkr8yESTqvSy7csK90qZ5xfvvsoNs=
+github.com/jcmturner/dnsutils/v2 v2.0.0 h1:lltnkeZGL0wILNvrNiVCR6Ro5PGU/SeBvVO/8c/iPbo=
+github.com/jcmturner/dnsutils/v2 v2.0.0/go.mod h1:b0TnjGOvI/n42bZa+hmXL+kFJZsFT7G4t3HTlQ184QM=
+github.com/jcmturner/gofork v1.7.6 h1:QH0l3hzAU1tfT3rZCnW5zXl+orbkNMMRGJfdJjHVETg=
+github.com/jcmturner/gofork v1.7.6/go.mod h1:1622LH6i/EZqLloHfE7IeZ0uEJwMSUyQ/nDd82IeqRo=
+github.com/jcmturner/goidentity/v6 v6.0.1 h1:VKnZd2oEIMorCTsFBnJWbExfNN7yZr3EhJAxwOkZg6o=
+github.com/jcmturner/goidentity/v6 v6.0.1/go.mod h1:X1YW3bgtvwAXju7V3LCIMpY0Gbxyjn/mY9zx4tFonSg=
+github.com/jcmturner/gokrb5/v8 v8.4.4 h1:x1Sv4HaTpepFkXbt2IkL29DXRf8sOfZXo8eRKh687T8=
+github.com/jcmturner/gokrb5/v8 v8.4.4/go.mod h1:1btQEpgT6k+unzCwX1KdWMEwPPkkgBtP+F6aCACiMrs=
+github.com/jcmturner/rpc/v2 v2.0.3 h1:7FXXj8Ti1IaVFpSAziCZWNzbNuZmnvw/i6CqLNdWfZY=
+github.com/jcmturner/rpc/v2 v2.0.3/go.mod h1:VUJYCIDm3PVOEHw8sgt091/20OJjskO/YJki3ELg/Hc=
 github.com/klauspost/compress v1.19.1 h1:VsB4HPswih7mmZ8WleSFQ75c/Ui1M4trX5oAsJnhSlk=
 github.com/klauspost/compress v1.19.1/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
 github.com/knadh/koanf/maps v0.1.2 h1:RBfmAW5CnZT+PJ1CVc1QSJKf4Xu9kxfQgYVQSu8hpbo=

internal/account/directory_test.go

@@ -0,0 +1,63 @@
+package account
+
+import (
+	"context"
+	"errors"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+type fakeDirectory struct {
+	asked []string
+	u     store.User
+	err   error
+}
+
+func (f *fakeDirectory) SignIn(_ context.Context, login, _ string) (store.User, error) {
+	f.asked = append(f.asked, login)
+	return f.u, f.err
+}
+
+// With a directory, a name the store has no password for is the
+// directory's to check; an account with a password here — the
+// administrator init made — is still checked here, so a directory that is
+// down locks nobody out.
+func TestPasswordsKeptElsewhereAreTheDirectorysToCheck(t *testing.T) {
+	svc, st := newService(t)
+	ctx := t.Context()
+	hash, _ := HashPassword("local-secret")
+	admin := store.User{ID: "usr-admin", Email: "admin@example.com", Name: "a", PasswordHash: hash,
+		State: store.UserActive, CreatedAt: time.Now()}
+	ada := store.User{ID: "usr-ada", Email: "ada@corp.example", Name: "Ada", PasswordHash: "!ldap",
+		State: store.UserActive, CreatedAt: time.Now()}
+	for _, u := range []store.User{admin, ada} {
+		if err := st.CreateUser(ctx, u); err != nil {
+			t.Fatal(err)
+		}
+	}
+	dir := &fakeDirectory{u: ada}
+	svc.SetDirectory(dir)
+
+	if _, _, err := svc.SignIn(ctx, "admin@example.com", "local-secret"); err != nil || len(dir.asked) != 0 {
+		t.Fatalf("the local administrator: %v, directory asked %v", err, dir.asked)
+	}
+	// A name, not an address, and an account kept by the directory.
+	for _, login := range []string{`CORP\ada`, "ada@corp.example"} {
+		if tok, _, err := svc.SignIn(ctx, login, "ad-secret"); err != nil || tok == "" {
+			t.Errorf("%s: %v", login, err)
+		}
+	}
+	if len(dir.asked) != 2 || dir.asked[0] != `CORP\ada` {
+		t.Errorf("the directory was asked %v", dir.asked)
+	}
+	dir.err = ErrBadCredentials
+	if _, _, err := svc.SignIn(ctx, "ada", "wrong"); !errors.Is(err, ErrBadCredentials) {
+		t.Errorf("a password the directory refused: %v", err)
+	}
+	// A reset link would give a directory's account a password here.
+	if _, _, err := svc.RequestReset(ctx, st, "ada@corp.example"); !errors.Is(err, ErrNoReset) {
+		t.Errorf("a directory's account was sent a reset link: %v", err)
+	}
+}

internal/account/service.go

@@ -80,8 +80,26 @@ type Service struct {
 
 	mu       sync.Mutex
 	attempts map[string]*attemptWindow
+
+	// dir checks the passwords of accounts this store keeps none for.
+	dir Directory
 }
 
+// Directory checks a password against an outside directory — Active
+// Directory over LDAP — and answers the account to sign in: made or kept in
+// step with the directory. Its errors are this package's ErrBadCredentials
+// and ErrSuspended for the person's own, and anything else for the
+// directory's.
+type Directory interface {
+	SignIn(ctx context.Context, login, password string) (store.User, error)
+}
+
+// SetDirectory has passwords for accounts with none here, and names that are
+// not addresses, checked by the directory. An account with a password here
+// — the administrator init made — is still checked here, so a directory that
+// is down locks nobody out.
+func (s *Service) SetDirectory(d Directory) { s.dir = d }
+
 type attemptWindow struct {
 	count int
 	since time.Time
@@ -137,6 +155,7 @@ func (s *Service) SignUp(ctx context.Context, email, name, password string) erro
 // SignIn checks a password and opens a session, returning the session token and
 // the CSRF token that must accompany the forms in it.
 func (s *Service) SignIn(ctx context.Context, email, password string) (token, csrf string, err error) {
+	login := strings.TrimSpace(email)
 	email = store.NormalizeEmail(email)
 	// Counted per address and the client trying it, so somebody guessing
 	// from elsewhere cannot lock the owner out of their own account; and per
@@ -151,6 +170,15 @@ func (s *Service) SignIn(ctx context.Context, email, password string) (token, cs
 	}
 
 	u, err := s.store.UserByEmail(ctx, email)
+	// No password kept here — the directory's, or nobody's — and a
+	// directory to ask: it decides.
+	if s.dir != nil && (errors.Is(err, store.ErrNotFound) || (err == nil && strings.HasPrefix(u.PasswordHash, "!"))) {
+		if u, err = s.dir.SignIn(ctx, login, password); err != nil {
+			return "", "", err
+		}
+		s.clearAttempts(byClient)
+		return s.open(ctx, u)
+	}
 	if errors.Is(err, store.ErrNotFound) {
 		// Verify anyway, against a hash that cannot match. Returning early here
 		// would let a stopwatch enumerate every account on the system.
@@ -180,7 +208,11 @@ func (s *Service) SignIn(ctx context.Context, email, password string) (token, cs
 	case store.UserSuspended:
 		return "", "", ErrSuspended
 	}
+	return s.open(ctx, u)
+}
 
+// open makes a session for a person whose password was right.
+func (s *Service) open(ctx context.Context, u store.User) (token, csrf string, err error) {
 	if token, err = store.NewSessionToken(); err != nil {
 		return "", "", err
 	}
@@ -402,7 +434,10 @@ func (s *Service) RequestReset(ctx context.Context, rs ResetStore, email string)
 		return "", store.User{}, ErrTooManyTries
 	}
 	u, err := rs.UserByEmail(ctx, email)
-	if err != nil || u.PasswordHash == "" || u.State != store.UserActive {
+	// An account whose password is kept elsewhere — "!oidc", "!scim",
+	// "!ldap" — gets no link: setting one here would give it a second way
+	// in that its directory knows nothing of, and outlives its leaving.
+	if err != nil || u.PasswordHash == "" || strings.HasPrefix(u.PasswordHash, "!") || u.State != store.UserActive {
 		return "", store.User{}, ErrNoReset
 	}
 	token, err := rs.CreatePasswordReset(ctx, u.ID, ResetLinkTTL)

internal/agent/discovery/sensors.go

@@ -74,6 +74,13 @@ func NewSensorSampler() *SensorSampler {
 	return &SensorSampler{sys: "/sys", smi: runSMI, now: time.Now}
 }
 
+// SampleGPUs reads every card's heat, power and how busy it is now, without
+// a sampler's ten seconds of memory: for a screen that asks every few seconds
+// and wants what it asked for. Nil on a machine without nvidia-smi or a card.
+func SampleGPUs(ctx context.Context) []GPUSensor {
+	return (&SensorSampler{smi: runSMI}).gpus(ctx)
+}
+
 func runSMI(ctx context.Context, fields string) ([]byte, error) {
 	ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
 	defer cancel()

internal/cli/bench.go

@@ -150,6 +150,7 @@ func newBenchCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 	f.DurationVar(&cfg.SLO.TPOTp99, "tpot-p99", cfg.SLO.TPOTp99, "per-output-token target")
 	f.BoolVar(&save, "save", false,
 		"record the capacity against the model and profile, so the number outlives this terminal")
+	cmd.AddCommand(newBenchOverheadCmd(asJSON))
 	return cmd
 }
 

internal/cli/bench_overhead.go

@@ -0,0 +1,421 @@
+package cli
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"fmt"
+	"io"
+	"log/slog"
+	"net"
+	"net/http"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"runtime"
+	"slices"
+	"sort"
+	"strings"
+	"sync"
+	"syscall"
+	"text/tabwriter"
+	"time"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/router"
+	storepkg "gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// What llmgw adds to a request, measured: the same load straight at an
+// engine that answers at once, and through the gateway in front of it —
+// bare, with authentication and usage accounting, and with the guardrails'
+// pattern checks besides. The engine's own time is the same on both paths,
+// so what differs is the gateway's.
+//
+// Everything runs in this process, on loopback: no cluster, no GPU, and the
+// numbers are this machine's. On a machine shared with other work (a VM on
+// an oversubscribed host) they move with it; compare modes run together,
+// not figures from different runs.
+
+// overheadMode is one way a request reaches the engine.
+type overheadMode struct {
+	name        string
+	gateway     bool
+	auth, guard bool
+}
+
+var overheadModes = []overheadMode{
+	{name: "direct"},
+	{name: "gateway", gateway: true},
+	{name: "gateway+auth", gateway: true, auth: true},
+	{name: "gateway+auth+guard", gateway: true, auth: true, guard: true},
+}
+
+// overheadStep is one mode at one request rate.
+type overheadStep struct {
+	Mode     string        `json:"mode"`
+	Rate     int           `json:"rate"`
+	Sent     int           `json:"sent"`
+	Failed   int           `json:"failed"`
+	P50      time.Duration `json:"p50"`
+	P99      time.Duration `json:"p99"`
+	AddedP50 time.Duration `json:"added_p50"`
+	AddedP99 time.Duration `json:"added_p99"`
+	// CPUPer1k is the process's CPU time per thousand requests beyond what
+	// the direct mode spent at the same rate: the gateway's.
+	CPUPer1k time.Duration `json:"cpu_per_1k"`
+	cpu      time.Duration
+}
+
+func newBenchOverheadCmd(asJSON *bool) *cobra.Command {
+	var (
+		rates []int
+		each  time.Duration
+	)
+	cmd := &cobra.Command{
+		Use:   "overhead",
+		Short: "Measure what the gateway adds to a request, against an engine that answers at once",
+		Long: "Runs an engine that answers at once and a gateway in front of it, in this\n" +
+			"process on loopback, and sends the same load at each rate straight to the\n" +
+			"engine and through the gateway: bare, with authentication and usage\n" +
+			"accounting, and with the guardrails' pattern checks too. Reports each\n" +
+			"request's latency at p50 and p99, what the gateway added to it, and the\n" +
+			"gateway's CPU time per thousand requests.\n\n" +
+			"Needs no cluster. The figures are this machine's: on one shared with other\n" +
+			"work they move with it, so compare the modes of one run, and note the\n" +
+			"machine's load beside any figure you keep.",
+		Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			steps, err := runOverhead(cmd.Context(), rates, each)
+			if err != nil {
+				return err
+			}
+			if *asJSON {
+				return json.NewEncoder(cmd.OutOrStdout()).Encode(steps)
+			}
+			printOverhead(cmd.OutOrStdout(), steps)
+			return nil
+		},
+	}
+	f := cmd.Flags()
+	f.IntSliceVar(&rates, "rates", []int{100, 500, 1000, 2000}, "requests a second to send, one step each")
+	f.DurationVar(&each, "each", 3*time.Second, "how long each step sends for")
+	return cmd
+}
+
+func printOverhead(w io.Writer, steps []overheadStep) {
+	tw := tabwriter.NewWriter(w, 0, 0, 2, ' ', 0)
+	fmt.Fprintln(tw, "MODE\tRATE/S\tSENT\tFAILED\tP50\tP99\t+P50\t+P99\tCPU/1K REQ")
+	for _, s := range steps {
+		added := func(d time.Duration) string {
+			if s.Mode == "direct" {
+				return "-"
+			}
+			return msOf(d)
+		}
+		cpu := "-"
+		if s.Mode != "direct" {
+			cpu = msOf(s.CPUPer1k)
+		}
+		fmt.Fprintf(tw, "%s\t%d\t%d\t%d\t%s\t%s\t%s\t%s\t%s\n", s.Mode, s.Rate, s.Sent, s.Failed,
+			msOf(s.P50), msOf(s.P99), added(s.AddedP50), added(s.AddedP99), cpu)
+	}
+	_ = tw.Flush()
+	fmt.Fprintf(w, "\n%d CPUs, %s. Loopback, in one process: the figures are this machine's.\n",
+		runtime.NumCPU(), loadAverage())
+}
+
+func msOf(d time.Duration) string { return fmt.Sprintf("%.2fms", float64(d)/float64(time.Millisecond)) }
+
+// loadAverage is the machine's, where it can be read, to keep beside the
+// figures.
+func loadAverage() string {
+	if b, err := os.ReadFile("/proc/loadavg"); err == nil {
+		if f := strings.Fields(string(b)); len(f) >= 3 {
+			return "load " + strings.Join(f[:3], " ")
+		}
+	}
+	// macOS: "{ 1.23 1.10 0.98 }".
+	if b, err := exec.Command("sysctl", "-n", "vm.loadavg").Output(); err == nil {
+		if f := strings.Fields(strings.Trim(strings.TrimSpace(string(b)), "{}")); len(f) >= 3 {
+			return "load " + strings.Join(f[:3], " ")
+		}
+	}
+	return "load not read"
+}
+
+// runOverhead measures every mode at every rate.
+func runOverhead(ctx context.Context, rates []int, each time.Duration) ([]overheadStep, error) {
+	engine, err := instantEngine()
+	if err != nil {
+		return nil, err
+	}
+	defer engine.Close()
+	dir, err := os.MkdirTemp("", "llmgw-overhead-")
+	if err != nil {
+		return nil, err
+	}
+	defer os.RemoveAll(dir)
+
+	type target struct {
+		url, key string
+		stop     func()
+	}
+	targets := map[string]target{}
+	defer func() {
+		for _, t := range targets {
+			if t.stop != nil {
+				t.stop()
+			}
+		}
+	}()
+	for _, m := range overheadModes {
+		if !m.gateway {
+			targets[m.name] = target{url: engine.URL}
+			continue
+		}
+		url, key, stop, err := overheadGateway(ctx, engine.URL, filepath.Join(dir, m.name+".db"), m)
+		if err != nil {
+			return nil, fmt.Errorf("%s: %w", m.name, err)
+		}
+		targets[m.name] = target{url: url, key: key, stop: stop}
+	}
+
+	client := &http.Client{Transport: &http.Transport{MaxIdleConns: 1024, MaxIdleConnsPerHost: 1024}, Timeout: 30 * time.Second}
+	var out []overheadStep
+	for _, rate := range rates {
+		var direct overheadStep
+		for _, m := range overheadModes {
+			t := targets[m.name]
+			s := loadStep(ctx, client, t.url, t.key, rate, each)
+			s.Mode = m.name
+			if m.name == "direct" {
+				direct = s
+			} else {
+				s.AddedP50, s.AddedP99 = max(0, s.P50-direct.P50), max(0, s.P99-direct.P99)
+				if s.Sent > 0 {
+					s.CPUPer1k = max(0, s.cpu-direct.cpu) * 1000 / time.Duration(s.Sent)
+				}
+			}
+			out = append(out, s)
+		}
+	}
+	return out, nil
+}
+
+// loadStep sends rate requests a second for d, each in its own goroutine,
+// and measures each one's latency and the process's CPU meanwhile.
+func loadStep(ctx context.Context, client *http.Client, url, key string, rate int, d time.Duration) overheadStep {
+	s := overheadStep{Rate: rate}
+	body := []byte(`{"model":"bench","max_tokens":8,"messages":[{"role":"user","content":"What is the capital of France?"}]}`)
+	var (
+		mu    sync.Mutex
+		lat   []time.Duration
+		wg    sync.WaitGroup
+		total = int(float64(rate) * d.Seconds())
+	)
+	interval := time.Second / time.Duration(max(rate, 1))
+	cpu0 := processCPU()
+	start := time.Now()
+	for i := range total {
+		if ctx.Err() != nil {
+			break
+		}
+		if wait := time.Until(start.Add(time.Duration(i) * interval)); wait > 0 {
+			time.Sleep(wait)
+		}
+		wg.Add(1)
+		go func() {
+			defer wg.Done()
+			req, _ := http.NewRequestWithContext(ctx, http.MethodPost, url+"/v1/chat/completions", bytes.NewReader(body))
+			req.Header.Set("Content-Type", "application/json")
+			if key != "" {
+				req.Header.Set("Authorization", "Bearer "+key)
+			}
+			t := time.Now()
+			resp, err := client.Do(req)
+			ok := err == nil && resp.StatusCode == http.StatusOK
+			if resp != nil {
+				_, _ = io.Copy(io.Discard, resp.Body)
+				resp.Body.Close()
+			}
+			el := time.Since(t)
+			mu.Lock()
+			s.Sent++
+			if ok {
+				lat = append(lat, el)
+			} else {
+				s.Failed++
+			}
+			mu.Unlock()
+		}()
+	}
+	wg.Wait()
+	s.cpu = processCPU() - cpu0
+	s.P50, s.P99 = percentile(lat, 0.50), percentile(lat, 0.99)
+	return s
+}
+
+func percentile(ds []time.Duration, p float64) time.Duration {
+	if len(ds) == 0 {
+		return 0
+	}
+	ds = slices.Clone(ds)
+	sort.Slice(ds, func(i, j int) bool { return ds[i] < ds[j] })
+	i := int(p * float64(len(ds)-1))
+	return ds[i]
+}
+
+// processCPU is the CPU time this process has used, user and system.
+func processCPU() time.Duration {
+	var ru syscall.Rusage
+	if err := syscall.Getrusage(syscall.RUSAGE_SELF, &ru); err != nil {
+		return 0
+	}
+	return time.Duration(ru.Utime.Nano() + ru.Stime.Nano())
+}
+
+// instantEngine is an OpenAI-compatible engine that answers at once, with
+// the metrics the router scrapes.
+func instantEngine() (*httpServer, error) {
+	mux := http.NewServeMux()
+	mux.HandleFunc("GET /metrics", func(w http.ResponseWriter, _ *http.Request) {
+		fmt.Fprint(w, "# TYPE vllm:num_requests_running gauge\nvllm:num_requests_running 0\n"+
+			"# TYPE vllm:num_requests_waiting gauge\nvllm:num_requests_waiting 0\n"+
+			"# TYPE vllm:kv_cache_usage_perc gauge\nvllm:kv_cache_usage_perc 0.1\n")
+	})
+	mux.HandleFunc("GET /health", func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(http.StatusOK) })
+	mux.HandleFunc("POST /v1/chat/completions", func(w http.ResponseWriter, r *http.Request) {
+		_, _ = io.Copy(io.Discard, r.Body)
+		w.Header().Set("Content-Type", "application/json")
+		fmt.Fprint(w, `{"id":"b","object":"chat.completion","model":"bench","choices":[{"index":0,"message":{"role":"assistant","content":"Paris."},"finish_reason":"stop"}],"usage":{"prompt_tokens":14,"completion_tokens":2,"total_tokens":16}}`)
+	})
+	return serveLoopback(mux)
+}
+
+// httpServer is a handler served on a loopback port.
+type httpServer struct {
+	URL string
+	srv *http.Server
+}
+
+func (s *httpServer) Close() { _ = s.srv.Close() }
+
+func serveLoopback(h http.Handler) (*httpServer, error) {
+	ln, err := net.Listen("tcp", "127.0.0.1:0")
+	if err != nil {
+		return nil, err
+	}
+	srv := &http.Server{Handler: h, ReadHeaderTimeout: 10 * time.Second}
+	go func() { _ = srv.Serve(ln) }()
+	return &httpServer{URL: "http://" + ln.Addr().String(), srv: srv}, nil
+}
+
+// overheadGateway starts a gateway in front of the engine as the mode asks,
+// and returns its address and, with auth, a key.
+func overheadGateway(ctx context.Context, engineURL, storePath string, m overheadMode) (string, string, func(), error) {
+	cfg := config.Default()
+	cfg.Listen = "127.0.0.1:0"
+	cfg.Router.ScrapeInterval = 200 * time.Millisecond
+	cfg.Endpoints = []config.Endpoint{{ID: "bench", URL: engineURL, Models: []string{"bench"}}}
+	key := ""
+	if m.auth {
+		var err error
+		if key, err = overheadKey(ctx, storePath); err != nil {
+			return "", "", nil, err
+		}
+		cfg.Auth.Enabled, cfg.Auth.Store = true, storePath
+		// Limits would measure themselves refusing: off.
+		cfg.Auth.DefaultRPM, cfg.Auth.DefaultTPM = 0, 0
+	}
+	rt, err := router.New(cfg, slog.New(slog.DiscardHandler))
+	if err != nil {
+		return "", "", nil, err
+	}
+	if m.guard {
+		g, err := baselineGuard()
+		if err != nil {
+			return "", "", nil, err
+		}
+		rt.SetGuard(g)
+	}
+	rctx, cancel := context.WithCancel(ctx)
+	rt.Start(rctx)
+	srv, err := serveLoopback(rt.Handler())
+	if err != nil {
+		cancel()
+		return "", "", nil, err
+	}
+	stop := func() { srv.Close(); cancel(); rt.Stop() }
+	// Ready once a request goes through: the endpoint is scraped first.
+	deadline := time.Now().Add(10 * time.Second)
+	for {
+		body := `{"model":"bench","messages":[{"role":"user","content":"hi"}]}`
+		req, _ := http.NewRequestWithContext(ctx, http.MethodPost, srv.URL+"/v1/chat/completions", strings.NewReader(body))
+		req.Header.Set("Content-Type", "application/json")
+		if key != "" {
+			req.Header.Set("Authorization", "Bearer "+key)
+		}
+		resp, err := http.DefaultClient.Do(req)
+		if err == nil {
+			_, _ = io.Copy(io.Discard, resp.Body)
+			resp.Body.Close()
+			if resp.StatusCode == http.StatusOK {
+				return srv.URL, key, stop, nil
+			}
+		}
+		if time.Now().After(deadline) {
+			stop()
+			status := "no answer"
+			if err == nil {
+				status = resp.Status
+			}
+			return "", "", nil, fmt.Errorf("the gateway did not serve within 10s (%s)", status)
+		}
+		time.Sleep(100 * time.Millisecond)
+	}
+}
+
+// overheadKey makes a store with one project and a key in it.
+func overheadKey(ctx context.Context, path string) (string, error) {
+	st, err := storepkg.Open(ctx, path)
+	if err != nil {
+		return "", err
+	}
+	defer st.Close()
+	org, err := st.CreateOrg(ctx, "bench")
+	if err != nil {
+		return "", err
+	}
+	team, err := st.CreateTeam(ctx, org, "bench")
+	if err != nil {
+		return "", err
+	}
+	project, err := st.CreateProject(ctx, team, "bench")
+	if err != nil {
+		return "", err
+	}
+	secret, k, err := storepkg.NewKey("key-bench", "bench", project)
+	if err != nil {
+		return "", err
+	}
+	k.CreatedAt = time.Now()
+	return secret, st.CreateKey(ctx, k)
+}
+
+// baselineGuard is the policy init installs, without its judge: the pattern
+// checks every request passes.
+func baselineGuard() (guard.Guard, error) {
+	m, err := config.Parse([]byte(guardSection(false)))
+	if err != nil {
+		return nil, err
+	}
+	cfg, err := config.FromMap(m)
+	if err != nil {
+		return nil, err
+	}
+	return guard.NewGate(cfg.Guard, nil, slog.New(slog.DiscardHandler))
+}

internal/cli/bench_overhead_test.go

@@ -0,0 +1,33 @@
+package cli
+
+import (
+	"bytes"
+	"strings"
+	"testing"
+	"time"
+)
+
+// The short version, so CI sees the gateway's overhead on every change: every
+// mode answers every request, and what the gateway adds is measured, not
+// guessed. No bound is asserted on it, as CI's machines vary; the table is
+// in the test's log.
+func TestTheGatewaysOverheadIsMeasured(t *testing.T) {
+	steps, err := runOverhead(t.Context(), []int{50}, time.Second)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(steps) != len(overheadModes) {
+		t.Fatalf("%d steps for %d modes", len(steps), len(overheadModes))
+	}
+	for _, s := range steps {
+		if s.Sent == 0 || s.Failed > 0 || s.P50 <= 0 || s.P99 < s.P50 {
+			t.Errorf("%s: sent %d failed %d p50 %s p99 %s", s.Mode, s.Sent, s.Failed, s.P50, s.P99)
+		}
+	}
+	var b bytes.Buffer
+	printOverhead(&b, steps)
+	if !strings.Contains(b.String(), "gateway+auth+guard") {
+		t.Errorf("table:\n%s", b.String())
+	}
+	t.Logf("\n%s", b.String())
+}

internal/cli/judgekeeper.go

@@ -32,6 +32,19 @@ type judgeKeeper struct {
 	// tried is when each judge was last asked for, so one that cannot be
 	// fetched is not asked for again on every look.
 	tried map[string]time.Time
+
+	// What refitting an earlier install's judge needs: the catalog to change
+	// its profile in, the replicas to restart, and the nodes' CPUs. Any nil
+	// leaves judges as they are.
+	profiles interface {
+		SetProfiles(ctx context.Context, name string, profiles []catalog.Profile, by string) (catalog.Model, error)
+	}
+	restart  func(ids ...string) error
+	replicas func(model string) []string
+	nodeCPUs func() []int
+	// looked is each judge already refitted or found changed by hand, so
+	// each is said once.
+	looked map[string]bool
 }
 
 // judgeRetry is how long a judge whose serving failed waits before it is
@@ -69,6 +82,7 @@ func (k *judgeKeeper) check() {
 	have := map[string]bool{}
 	for _, m := range k.models() {
 		have[m.Name] = true
+		k.refit(m)
 	}
 	busy := map[string]bool{}
 	for _, j := range k.serve.Jobs() {
@@ -114,3 +128,88 @@ func (k *judgeKeeper) check() {
 			"judge", name, "repo", kj.Repo, "file", kj.File, "job", job.ID)
 	}
 }
+
+// refit moves a judge an earlier `server init` installed to what init
+// installs now, and restarts its replicas onto it. Those judges reserved no
+// cores or one, 2 or 4 GB, and ran llama.cpp's prompt cache: on fewer than
+// eight cores every judgment missed its 5 s budget and fail: closed refused
+// every request, until fixed by hand. A judge whose profile differs in any
+// other way was set by somebody, and is left as it is, said once.
+func (k *judgeKeeper) refit(m catalog.Model) {
+	if k.profiles == nil || k.restart == nil || k.replicas == nil || m.Name != defaultJudge || !m.IsGuard() {
+		return
+	}
+	if k.looked == nil {
+		k.looked = map[string]bool{}
+	}
+	if k.looked[m.Name] {
+		return
+	}
+	k.looked[m.Name] = true
+	if !earlierInitJudge(m) {
+		if len(m.Profiles) != 1 || !slices.Contains(m.Profiles[0].Args, "--cache-ram") {
+			k.log.Info("the judge's profile is not one init wrote; left as it is",
+				"judge", m.Name, "see", "llmgw ctl model resources "+m.Name)
+		}
+		return
+	}
+	var cpus []int
+	if k.nodeCPUs != nil {
+		cpus = k.nodeCPUs()
+	}
+	p := m.Profiles[0]
+	p.MinMemoryGB = judgeMemoryGB
+	p.CPUCores = refitCores(m.Scaling.Min, cpus)
+	p.Args = []string{"--ctx-size", "8192", "--cache-ram", "0"}
+	ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
+	defer cancel()
+	if _, err := k.profiles.SetProfiles(ctx, m.Name, []catalog.Profile{p}, "llmgw (the judge an earlier init installed)"); err != nil {
+		k.log.Warn("could not refit the judge an earlier init installed", "judge", m.Name, "err", err)
+		k.looked[m.Name] = false
+		return
+	}
+	ids := k.replicas(m.Name)
+	if len(ids) > 0 {
+		if err := k.restart(ids...); err != nil {
+			k.log.Warn("refitted the judge, but could not restart it", "judge", m.Name, "err", err)
+		}
+	}
+	k.log.Info("refitted the judge an earlier init installed", "judge", m.Name,
+		"memory_gb", p.MinMemoryGB, "cpu_cores", p.CPUCores, "replicas_restarted", len(ids))
+}
+
+// earlierInitJudge is whether a judge's entry is exactly what `server init`
+// before v0.14.7 wrote: one llama.cpp profile on CPUs, an 8192 context and
+// no other args, 2 or 4 GB, none to two cores.
+func earlierInitJudge(m catalog.Model) bool {
+	if len(m.Profiles) != 1 {
+		return false
+	}
+	p := m.Profiles[0]
+	return p.Name == "cpu" && p.Engine == "llamacpp" && p.GPUClass == "" && p.GPU == nil &&
+		len(p.NodeSelector) == 0 && slices.Equal(p.Args, []string{"--ctx-size", "8192"}) &&
+		(p.MinMemoryGB == 2 || p.MinMemoryGB == 4) && p.CPUCores >= 0 && p.CPUCores <= 2
+}
+
+// refitCores is a judge replica's cores by init's rule: up to four for the
+// one judge of a cluster, two each where there are more, fewer where the
+// smallest node cannot hold them all.
+func refitCores(replicas int, cpus []int) int {
+	smallest := 0
+	for _, c := range cpus {
+		if c > 0 && (smallest == 0 || c < smallest) {
+			smallest = c
+		}
+	}
+	if replicas <= 1 {
+		if smallest == 0 {
+			return judgeCores
+		}
+		return min(smallest, judgeCoresAlone)
+	}
+	cores := judgeCores
+	if len(cpus) == 1 && smallest > 0 && replicas*cores > smallest {
+		cores = max(1, smallest/replicas)
+	}
+	return cores
+}

internal/cli/judgekeeper_test.go

@@ -1,6 +1,7 @@
 package cli
 
 import (
+	"context"
 	"errors"
 	"log/slog"
 	"slices"
@@ -124,3 +125,81 @@ func TestCategoriesFollowTheJudges(t *testing.T) {
 		t.Errorf("no categories (every one counts) became %v", got)
 	}
 }
+
+type fakeProfiles struct {
+	set map[string][]catalog.Profile
+}
+
+func (f *fakeProfiles) SetProfiles(_ context.Context, name string, ps []catalog.Profile, _ string) (catalog.Model, error) {
+	if f.set == nil {
+		f.set = map[string][]catalog.Profile{}
+	}
+	f.set[name] = ps
+	return catalog.Model{Name: name, Profiles: ps}, nil
+}
+
+// A judge that v0.14.6 or earlier's init installed is moved to today's
+// figures, once, and its replicas restarted onto them; one somebody changed
+// is left alone.
+func TestAnEarlierInstallsJudgeIsRefittedOnce(t *testing.T) {
+	lib := guard.Config{Policies: map[string]guard.Policy{
+		"baseline": {Judge: &guard.JudgePolicy{Model: "qwen3guard-0.6b", Input: "block"}},
+	}}
+	earlier := guardModel("http://10.0.0.5:8081/qwen3guard-0.6b/guard/g.gguf", 2, 1)
+	earlier.Profiles[0].MinMemoryGB, earlier.Profiles[0].Args = 4, []string{"--ctx-size", "8192"}
+	inCatalog := []catalog.Model{earlier}
+	fp := &fakeProfiles{}
+	var restarted []string
+	k := &judgeKeeper{
+		library:  func() guard.Config { return lib },
+		models:   func() []catalog.Model { return inCatalog },
+		serve:    &fakeServe{},
+		log:      slog.New(slog.DiscardHandler),
+		profiles: fp,
+		restart:  func(ids ...string) error { restarted = append(restarted, ids...); return nil },
+		replicas: func(string) []string { return []string{"j-1", "j-2"} },
+		nodeCPUs: func() []int { return []int{4} },
+	}
+	k.check()
+	got := fp.set["qwen3guard-0.6b"]
+	if len(got) != 1 || got[0].MinMemoryGB != 2 || got[0].CPUCores != 2 || !slices.Contains(got[0].Args, "--cache-ram") {
+		t.Fatalf("refitted to %+v", got)
+	}
+	if !slices.Equal(restarted, []string{"j-1", "j-2"}) {
+		t.Errorf("restarted %v", restarted)
+	}
+	k.check()
+	if len(restarted) != 2 {
+		t.Error("refitted twice")
+	}
+
+	// Changed by somebody: an extra arg. Left as it is.
+	mine := guardModel("http://x/g.gguf", 2, 3)
+	mine.Profiles[0].Args = []string{"--ctx-size", "16384"}
+	inCatalog = []catalog.Model{mine}
+	fp.set, k.looked = nil, nil
+	k.check()
+	if fp.set != nil {
+		t.Errorf("a judge set by hand was changed: %+v", fp.set)
+	}
+}
+
+func TestAJudgeIsRefittedToTheCoresInitWouldGive(t *testing.T) {
+	for _, c := range []struct {
+		replicas int
+		cpus     []int
+		want     int
+	}{
+		{1, []int{4}, 4},
+		{1, []int{2}, 2},
+		{1, []int{16}, 4},
+		{2, []int{4}, 2},
+		{2, []int{2}, 1},
+		{2, []int{4, 32}, 2},
+		{2, nil, 2},
+	} {
+		if got := refitCores(c.replicas, c.cpus); got != c.want {
+			t.Errorf("%d judges on %v: %d cores, want %d", c.replicas, c.cpus, got, c.want)
+		}
+	}
+}

internal/cli/server.go

@@ -76,6 +76,7 @@ func newServerCmd() *cobra.Command {
 
 	cmd := &cobra.Command{
 		Use:   "server",
+		Args:  cobra.NoArgs,
 		Short: "Run the router (phase 1; the control plane arrives in phase 3)",
 		Long: "Serves the OpenAI API in front of a set of engine replicas.\n\n" +
 			"In this phase the replica list is static: give it in the config file, or\n" +
@@ -138,6 +139,26 @@ func newServerCmd() *cobra.Command {
 				_ = shutdownTracing(sctx)
 			}()
 
+			// The ports before anything else: a second server on this machine,
+			// whose ports are taken, must stop here, not go on to compete for
+			// the lease and lead a cluster no request can reach.
+			apiLn, err := net.Listen("tcp", cfg.Listen)
+			if err != nil {
+				return fmt.Errorf("listening on %s: %w", cfg.Listen, err)
+			}
+			if cfg.MaxConnections > 0 {
+				// Past this, new connections wait in the kernel's backlog
+				// rather than each taking a goroutine and its buffers.
+				apiLn = netutil.LimitListener(apiLn, cfg.MaxConnections)
+			}
+			adminLn, err := net.Listen("tcp", cfg.AdminListen)
+			if err != nil {
+				_ = apiLn.Close()
+				return fmt.Errorf("listening on %s: %w", cfg.AdminListen, err)
+			}
+			defer apiLn.Close()
+			defer adminLn.Close()
+
 			rt, err := router.New(cfg, log)
 			if err != nil {
 				return err
@@ -418,6 +439,7 @@ func newServerCmd() *cobra.Command {
 					}
 					// The cluster's settings, each change checked against this
 					// machine's own keys before it is kept.
+					api.SetAuth(&adminapi.AuthAdmin{Status: rt.AuthStatus, TestLDAP: rt.TestDirectory, Secrets: st, Log: log})
 					api.SetSettings(&adminapi.Settings{Store: st, LocalKeys: config.LocalKeys(),
 						Check: func(body []byte) error { return checkClusterSettings(configPath, body) }})
 					// And the same document on the portal, for administrators.
@@ -593,6 +615,25 @@ func newServerCmd() *cobra.Command {
 							},
 							gpus: func() []adminapi.ClassUse { return adminAPI.Summary().GPUs },
 							log:  log,
+							// An earlier install's judge, moved to today's figures.
+							profiles: catalogSvc,
+							restart:  admin.Reconciler().Restart,
+							replicas: func(model string) []string {
+								var ids []string
+								for _, r := range admin.Reconciler().ReadyReplicas() {
+									if r.Model == model {
+										ids = append(ids, r.ID)
+									}
+								}
+								return ids
+							},
+							nodeCPUs: func() []int {
+								var out []int
+								for _, n := range admin.Nodes() {
+									out = append(out, n.CPUCount)
+								}
+								return out
+							},
 						}
 						go keeper.run(ctx, 30*time.Second)
 					}
@@ -642,21 +683,6 @@ func newServerCmd() *cobra.Command {
 				ReadTimeout: time.Minute,
 			})
 
-			apiLn, err := net.Listen("tcp", cfg.Listen)
-			if err != nil {
-				return fmt.Errorf("listening on %s: %w", cfg.Listen, err)
-			}
-			if cfg.MaxConnections > 0 {
-				// Past this, new connections wait in the kernel's backlog
-				// rather than each taking a goroutine and its buffers.
-				apiLn = netutil.LimitListener(apiLn, cfg.MaxConnections)
-			}
-			adminLn, err := net.Listen("tcp", cfg.AdminListen)
-			if err != nil {
-				_ = apiLn.Close()
-				return fmt.Errorf("listening on %s: %w", cfg.AdminListen, err)
-			}
-
 			// A metrics-only listener, when one is configured. Started after
 			// AdminHandler, which is where the registry it serves is built.
 			var metricsSrv *http.Server

internal/cli/server_args_test.go

@@ -0,0 +1,39 @@
+package cli
+
+import (
+	"context"
+	"net"
+	"strings"
+	"testing"
+	"time"
+)
+
+// `llmgw server check` once ran a whole second server: words after server
+// that name no subcommand are refused.
+func TestServerTakesNoWordsItDoesNotKnow(t *testing.T) {
+	cmd := newServerCmd()
+	cmd.SetArgs([]string{"check", "--config", "/nonexistent"})
+	cmd.SilenceUsage, cmd.SilenceErrors = true, true
+	if err := cmd.Execute(); err == nil || !strings.Contains(err.Error(), "unknown command") {
+		t.Errorf("err %v", err)
+	}
+}
+
+// A second server whose port is taken stops at once, before the router or
+// any control plane is built.
+func TestAServerWhosePortIsTakenStopsFirst(t *testing.T) {
+	taken, err := net.Listen("tcp", "127.0.0.1:0")
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer taken.Close()
+	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+	defer cancel()
+	cmd := newServerCmd()
+	cmd.SetArgs([]string{"--listen", taken.Addr().String(), "--admin-listen", "127.0.0.1:0",
+		"--endpoint", "a=http://127.0.0.1:1=m"})
+	cmd.SilenceUsage, cmd.SilenceErrors = true, true
+	if err := cmd.ExecuteContext(ctx); err == nil || !strings.Contains(err.Error(), "listening on") {
+		t.Errorf("err %v", err)
+	}
+}

internal/cli/serverinit_choices.go

@@ -194,6 +194,9 @@ func (in *serverInit) judgeFor() judgeSpec {
 	}
 	if j.Replicas <= 0 {
 		j.Replicas = 2
+		if in.singleMachine() {
+			j.Replicas = 1
+		}
 	}
 	if j.Budget <= 0 {
 		j.Budget = 5 * time.Second
@@ -201,6 +204,14 @@ func (in *serverInit) judgeFor() judgeSpec {
 	return j
 }
 
+// singleMachine is an install where this machine is the only node there
+// will be, as far as init knows: a second judge replica there shares the
+// same cores and is lost with the same machine, so it buys nothing but
+// half the speed of each.
+func (in *serverInit) singleMachine() bool {
+	return !in.noNode && !in.standbys && len(in.patroni) == 0 && len(in.patroniNodes) == 0
+}
+
 // The configuration's limits, as init writes them unless told otherwise.
 const (
 	defaultRPM       = 60
@@ -346,7 +357,10 @@ func (in *serverInit) initFlags() []string {
 	if in.noGuardModel {
 		add("--no-guard-model")
 	} else {
-		j, d := in.judgeFor(), (&serverInit{}).judgeFor()
+		// Against the defaults for an install of the same shape: one judge
+		// on one machine, two otherwise.
+		shape := serverInit{noNode: in.noNode, standbys: in.standbys, patroni: in.patroni, patroniNodes: in.patroniNodes}
+		j, d := in.judgeFor(), shape.judgeFor()
 		if j.Fail != d.Fail {
 			add("--judge-fail", j.Fail)
 		}

internal/cli/serverinit_choices_test.go

@@ -183,12 +183,17 @@ func TestTheJudgesAndTheStarterShareTheCPUsRatherThanEachTakingAll(t *testing.T)
 		in             serverInit
 		judge, starter int
 	}{
-		{serverInit{cpus: 4}, 1, 2},
-		{serverInit{cpus: 8}, 2, 4},
-		{serverInit{cpus: 32}, 2, 8},
-		{serverInit{cpus: 2}, 1, 0}, // nothing left: the starter shares rather than fits nowhere
+		// One machine: one judge, on up to four cores; the starter has the rest.
+		{serverInit{cpus: 4}, 4, 0},
+		{serverInit{cpus: 8}, 4, 4},
+		{serverInit{cpus: 32}, 4, 8},
+		{serverInit{cpus: 2}, 2, 0},
 		{serverInit{cpus: 1}, 0, 0},
-		{serverInit{cpus: 4, noGuardModel: true}, 1, 4},
+		{serverInit{cpus: 4, noGuardModel: true}, judgeCores, 4}, // no judge: all for the starter
+		// Two asked for on one machine: two cores each, one each where two
+		// would not fit both.
+		{serverInit{cpus: 8, judge: judgeSpec{Replicas: 2}}, 2, 4},
+		{serverInit{cpus: 2, judge: judgeSpec{Replicas: 2}}, 1, 0},
 		{serverInit{cpus: 8, judge: judgeSpec{Replicas: 3}}, 2, 2},
 		{serverInit{cpus: 4, noNode: true}, 2, 0}, // on nodes not known yet
 	} {
@@ -198,11 +203,39 @@ func TestTheJudgesAndTheStarterShareTheCPUsRatherThanEachTakingAll(t *testing.T)
 		}
 	}
 	m := guardModel("http://x/g.gguf", 2, 1)
-	if p := m.Profiles[0]; p.CPUCores != 1 || p.MinMemoryGB != 4 {
-		t.Errorf("judge profile: %d cores, %d GB", p.CPUCores, p.MinMemoryGB)
+	if p := m.Profiles[0]; p.CPUCores != 1 || p.MinMemoryGB != 2 || !slices.Contains(p.Args, "--cache-ram") {
+		t.Errorf("judge profile: %d cores, %d GB, args %v", p.CPUCores, p.MinMemoryGB, p.Args)
 	}
 	s := starterModel(starterChoices[0], defaultContext, 2, "http://x/q.gguf", []string{"6000-24g"}, true)
 	if cpu := s.Profiles[len(s.Profiles)-1]; cpu.Name != "cpu" || cpu.CPUCores != 2 || s.Profiles[0].CPUCores != 0 {
 		t.Errorf("starter: GPU profile %d cores, CPU %+v", s.Profiles[0].CPUCores, cpu)
 	}
 }
+
+// A second judge on the same machine shares its cores and is lost with it:
+// one machine has one, more machines two.
+func TestOneMachineHasOneJudgeAndMoreHaveTwo(t *testing.T) {
+	for _, c := range []struct {
+		in   serverInit
+		want int
+	}{
+		{serverInit{}, 1},
+		{serverInit{noNode: true}, 2},
+		{serverInit{patroni: []string{"a@db1", "b@db2"}}, 2},
+		{serverInit{standbys: true}, 2},
+		{serverInit{judge: judgeSpec{Replicas: 3}}, 3},
+	} {
+		if got := c.in.judgeFor().Replicas; got != c.want {
+			t.Errorf("%+v: %d judges, want %d", c.in, got, c.want)
+		}
+	}
+	// The defaults are left out of the command; a choice against them is in.
+	ha := serverInit{noNode: true, patroni: []string{"a@db1", "b@db2"}, judge: judgeSpec{Replicas: 2}}
+	if f := strings.Join(ha.initFlags(), " "); strings.Contains(f, "--judge-replicas") {
+		t.Errorf("two judges on three machines named: %s", f)
+	}
+	one := serverInit{judge: judgeSpec{Replicas: 2}}
+	if f := strings.Join(one.initFlags(), " "); !strings.Contains(f, "--judge-replicas 2") {
+		t.Errorf("two judges on one machine not named: %s", f)
+	}
+}

internal/cli/serverinit_cluster.go

@@ -496,10 +496,13 @@ func guardModel(source string, replicas, cores int) catalog.Model {
 		// judge that is not there refuses everything.
 		Scaling: catalog.Scaling{Min: replicas, Max: replicas},
 		Profiles: []catalog.Profile{{
-			// It uses about 3.2 GB with its context: reserving 2 filled nodes
-			// past their memory.
-			Name: "cpu", Engine: "llamacpp", Priority: 100, MinMemoryGB: 4, CPUCores: cores,
-			Args: []string{"--ctx-size", "8192"},
+			// About 1.6 GB with its context, which stays so with no prompt
+			// cache: llama.cpp keeps up to 8 GB of prompts by default, and a
+			// judge's, each a different text, are not asked again. With the
+			// cache one grew to 3.2 GB over a day of traffic; reserving 4 for
+			// that left a 7 GB machine room for only one of the two.
+			Name: "cpu", Engine: "llamacpp", Priority: 100, MinMemoryGB: judgeMemoryGB, CPUCores: cores,
+			Args: []string{"--ctx-size", "8192", "--cache-ram", "0"},
 		}},
 	}
 }
@@ -716,14 +719,16 @@ type cpuSplit struct{ judge, starter int }
 // the starter on one 4-core control plane all ran on all four: under load
 // most judgments missed their budget and the requests were refused.
 //
-// When this machine is the node, by its own CPUs: a core a judge under 8,
-// two from 8, and the starter what is left, up to 8, or no reservation at
-// all when nothing is, rather than a starter that fits nowhere. Otherwise
-// two a judge, on whichever node takes it, and the starter, on a node not
-// known yet, none.
+// A judge needs cores: a judgment of about 280 tokens took 8.0 s on one,
+// 4.2 s on two and 2.3 s on four, against a 5 s budget. On one machine its
+// one replica gets up to four; where there are two, two each (one each only
+// where two would not fit both). The starter has what is left, up to 8, or
+// no reservation at all when nothing is, rather than a starter that fits
+// nowhere. A machine that is not the node reserves two a judge on whichever
+// node takes it, and the starter, on a node not known yet, none.
 func (in *serverInit) cpuPlan() cpuSplit {
 	if in.noNode {
-		return cpuSplit{judge: 2}
+		return cpuSplit{judge: judgeCores}
 	}
 	n := in.cpus
 	if n <= 0 {
@@ -736,13 +741,23 @@ func (in *serverInit) cpuPlan() cpuSplit {
 	if !in.noGuardModel {
 		judges = in.judgeFor().Replicas
 	}
-	judge := 1
-	if n >= 8 {
-		judge = 2
-	}
-	left := n - judges*judge
-	if left < 1 {
-		left = 0
+	judge := judgeCores
+	switch {
+	case judges == 1:
+		judge = min(n, judgeCoresAlone)
+	case judges > 0 && judges*judge > n:
+		judge = max(1, n/judges)
 	}
+	left := max(n-judges*judge, 0)
 	return cpuSplit{judge: judge, starter: min(left, 8)}
 }
+
+// judgeCores and judgeMemoryGB are what a judge replica needs: two cores
+// for a judgment within its budget, and 2 GB for its 1.6 with no prompt
+// cache. judgeCoresAlone is what the one judge of a single machine takes,
+// past which a model this small hardly gets faster.
+const (
+	judgeCores      = 2
+	judgeCoresAlone = 4
+	judgeMemoryGB   = 2
+)

internal/cli/tui_choices.go

@@ -282,7 +282,7 @@ func newInstallForm(in *serverInit, host string, found []string, ha bool) *insta
 		"The request is served unjudged and marked so in the logs. Nothing is refused for the judge's sake.",
 	}
 	f.judgeCopies = textSetupField("Copies", strconv.Itoa(j.Replicas), "2",
-		"Each takes about 1 GB and two CPU threads on whichever node has room.")
+		"On one machine one copy, on up to four cores; with more machines two, two cores each, on different nodes. Each reserves 2 GB.")
 	f.judgeBudget = textSetupField("Waits at most", durationYAML(j.Budget), "5s",
 		"A short text is judged in about two seconds on CPUs.")
 	judgeFields := []*setupField{f.judgeFail, f.judgeCopies, f.judgeBudget, headingField("Refuses")}

internal/cli/tui_console.go

@@ -53,6 +53,7 @@ const (
 	cAccounts
 	cKeys
 	cOrgs
+	cSignIn
 	cAudit
 	cManual
 	cSettings
@@ -69,7 +70,7 @@ const (
 	cCount
 )
 
-var consoleLabels = [cCount]string{"Overview", "Monitor", "Nodes", "GPUs", "Models", "Replicas", "Usage", "Quotas", "Guardrails", "Knowledge", "Federation", "Mirror", "Jobs", "Logs", "Accounts", "API keys", "Organizations", "Audit", "Manual",
+var consoleLabels = [cCount]string{"Overview", "Monitor", "Nodes", "GPUs", "Models", "Replicas", "Usage", "Quotas", "Guardrails", "Knowledge", "Federation", "Mirror", "Jobs", "Logs", "Accounts", "API keys", "Organizations", "Sign-in", "Audit", "Manual",
 	"server.yaml", "Add a node", "Add a control plane", "Replicate the store", "Serve a model", "Serve the guardrails' judge", "Upgrade the cluster",
 	"Restart all replicas", "Back up now", "Wipe this machine", "Leave"}
 
@@ -140,8 +141,8 @@ type consoleModel struct {
 	detail bool
 	scroll int
 
-	// node is the console of a node: its agent.yaml and nothing to watch,
-	// since the cluster is watched from the control plane.
+	// node is the console of a node: what runs on the machine and its
+	// agent.yaml, since the cluster is watched from the control plane.
 	node bool
 
 	// The settings.
@@ -181,6 +182,7 @@ type consoleModel struct {
 	quotas    []adminapi.QuotaView
 	quotasErr error
 	guard     guardState
+	signIn    signInState
 	kb        kbState
 
 	st     *storeHolder
@@ -190,6 +192,11 @@ type consoleModel struct {
 	docs   docViewer
 	mon    monitorState
 	det    monitorState
+
+	// nodeOv is a node's overview as last read; nodeBusy says a reading is
+	// under way.
+	nodeOv   nodeOverview
+	nodeBusy bool
 }
 
 var usageGroupings = []string{"team", "project", "org"}
@@ -198,9 +205,6 @@ func newConsoleModel(addr, file string) consoleModel {
 	m := consoleModel{client: newCtlClient(addr), spin: newSpinner(), width: 100, height: 30, file: file,
 		st: &storeHolder{file: file}, docs: newDocViewer()}
 	m.node = strings.HasSuffix(file, "agent.yaml")
-	if m.node {
-		m.menu = cSettings
-	}
 	m.loadSettings()
 	return m
 }
@@ -257,7 +261,7 @@ func (m *consoleModel) loadSettings() {
 
 func (m consoleModel) Init() tea.Cmd {
 	if m.node {
-		return m.spin.Tick
+		return tea.Batch(m.fetchNodeOverview(), m.spin.Tick, tea.Tick(3*time.Second, func(time.Time) tea.Msg { return consoleTickMsg{} }))
 	}
 	return tea.Batch(m.fetch(), m.spin.Tick, tea.Tick(3*time.Second, func(time.Time) tea.Msg { return consoleTickMsg{} }))
 }
@@ -320,7 +324,7 @@ func (m consoleModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
 		return m, cmd
 	case consoleTickMsg:
 		if m.node {
-			return m, nil
+			return m.nodeTick()
 		}
 		cmds := []tea.Cmd{m.fetch(), tea.Tick(3*time.Second, func(time.Time) tea.Msg { return consoleTickMsg{} })}
 		if m.menu == cUsage {
@@ -332,6 +336,9 @@ func (m consoleModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
 		if m.menu == cGuard {
 			cmds = append(cmds, m.fetchGuard())
 		}
+		if m.menu == cSignIn {
+			cmds = append(cmds, m.fetchSignIn())
+		}
 		if m.menu == cKB {
 			cmds = append(cmds, m.fetchKB())
 		}
@@ -351,6 +358,8 @@ func (m consoleModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
 			cmds = append(cmds, m.fetchLogs(m.logs.node))
 		}
 		return m, tea.Batch(cmds...)
+	case nodeOverviewMsg:
+		m.nodeOv, m.nodeBusy = nodeOverview(msg), false
 	case consoleDataMsg:
 		m.loaded = true
 		if msg.err != nil {
@@ -395,6 +404,8 @@ func (m consoleModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
 		m.quotas, m.quotasErr = msg.rows, msg.err
 	case consoleKBMsg:
 		m.kb = kbState{view: msg.view, err: msg.err, loaded: true}
+	case consoleSignInMsg:
+		m.signIn = signInState{view: msg.view, err: msg.err, loaded: true, ldap: msg.ldap, oidc: msg.oidc}
 	case consoleGuardMsg:
 		window := m.guard.window
 		m.guard = guardState{view: msg.view, events: msg.events, err: msg.err, window: window}
@@ -423,7 +434,7 @@ func (m consoleModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
 
 func (m consoleModel) entryOK(i int) bool {
 	if m.node {
-		return i == cSettings || i == cManual || i == cWipe || i == cLeave
+		return i == cOverview || i == cSettings || i == cManual || i == cWipe || i == cLeave
 	}
 	return i != cReplicate || !m.storeReplicated()
 }
@@ -553,6 +564,8 @@ func (m consoleModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) {
 			m.menuD = m.quotaMenu(m.quotas[m.row])
 		case cGuard:
 			m.menuD = m.guardAttachmentMenu(m.guard.view.Attachments[m.row])
+		case cSignIn:
+			m.menuD = m.signInMenu(m.signInRowKey())
 		case cKB:
 			m.menuD = m.kbMenu(m.kb.view.Collections[m.row])
 		case cMirror:
@@ -649,6 +662,19 @@ func (m consoleModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) {
 			m.usageExportForm()
 			return m, nil
 		}
+		if m.menu == cSignIn && m.signIn.loaded {
+			m.entraForm()
+			return m, nil
+		}
+	case "d":
+		if m.menu == cSignIn && m.signIn.loaded {
+			m.directoryForm()
+			return m, nil
+		}
+	case "t":
+		if m.menu == cSignIn && m.signIn.view.LDAP != nil {
+			return m, m.tryDirectory()
+		}
 	case "right", "l":
 		if m.menu == cUsage {
 			m.usageBy = (m.usageBy + 1) % len(usageGroupings)
@@ -675,6 +701,8 @@ func (m consoleModel) onEnter() tea.Cmd {
 		return m.fetchQuotas()
 	case cGuard:
 		return m.fetchGuard()
+	case cSignIn:
+		return m.fetchSignIn()
 	case cKB:
 		return m.fetchKB()
 	case cMirror:
@@ -1053,6 +1081,8 @@ func (m consoleModel) rowCount() int {
 		return len(m.quotas)
 	case cGuard:
 		return len(m.guard.view.Attachments)
+	case cSignIn:
+		return len(signInRows)
 	case cKB:
 		return len(m.kb.view.Collections)
 	case cMirror:
@@ -1080,9 +1110,6 @@ func (m consoleModel) rowCount() int {
 func (m consoleModel) View() string {
 	host, _ := os.Hostname()
 	nav := []navItem{{Label: "Watch", Heading: true}}
-	if m.node {
-		nav = nil
-	}
 	for i := 0; i < cCount; i++ {
 		if m.node && !m.entryOK(i) {
 			continue
@@ -1177,6 +1204,9 @@ func (m consoleModel) panel() (title, body, keys string) {
 	}
 	switch m.menu {
 	case cOverview:
+		if m.node {
+			return m.nodeOverviewPanel(pw)
+		}
 		var b bytes.Buffer
 		c := &cobra.Command{}
 		c.SetOut(&b)
@@ -1292,6 +1322,8 @@ func (m consoleModel) panel() (title, body, keys string) {
 		return m.quotasPanel(pw)
 	case cGuard:
 		return m.guardPanel(pw)
+	case cSignIn:
+		return m.signInPanel(pw)
 	case cKB:
 		return m.kbPanel(pw)
 	case cAccounts, cKeys, cOrgs:
@@ -1648,9 +1680,6 @@ func runConsole(ctx context.Context, out io.Writer, file string) error {
 		}
 	}
 	menu := cOverview
-	if strings.HasSuffix(file, "agent.yaml") {
-		menu = cSettings
-	}
 	for {
 		m := newConsoleModel(addr, file)
 		m.menu = menu

internal/cli/tui_node_overview.go

@@ -0,0 +1,715 @@
+package cli
+
+import (
+	"bufio"
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"net"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"slices"
+	"sort"
+	"strconv"
+	"strings"
+	"sync"
+	"syscall"
+	"time"
+
+	tea "github.com/charmbracelet/bubbletea"
+	"github.com/charmbracelet/lipgloss"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/version"
+)
+
+// A node's overview: what a bare `sudo llmgw` opens on a machine that runs
+// only llmgw-agent. Everything on it is read here — the agent's own files,
+// systemd, the cards and the disks — because a node has no admin token, and
+// the control plane may be the very thing it cannot reach.
+
+// nodeCmdTimeout bounds every command and check the overview runs: a screen
+// refreshed every three seconds cannot wait on a tool that hangs.
+const nodeCmdTimeout = 2 * time.Second
+
+// nodeControlPlanesFile is where the agent keeps the control planes the
+// leader last named (controlPlanesFile in the agent package).
+var nodeControlPlanesFile = "/var/lib/llmgw-agent/control-planes.json"
+
+// nodeProblemLines is how many of the agent's warnings the overview shows.
+const nodeProblemLines = 6
+
+type nodeOverviewMsg nodeOverview
+
+// nodeOverview is everything the overview shows, read in one go.
+type nodeOverview struct {
+	at      time.Time
+	version string
+	agent   agentService
+	planes  []planeCheck
+	gpus    []nodeGPU
+	// gpuNote says why there are no GPU rows: no card, or no answer.
+	gpuNote string
+	// names is the cards' names by index, which do not change and are asked
+	// for once rather than every refresh.
+	names       map[int]string
+	engines     []engineRun
+	enginesNote string
+	disks       []diskUse
+	problems    []string
+	// problemsNote replaces the problems when the journal cannot be read.
+	problemsNote string
+}
+
+// agentService is what systemd says of llmgw-agent.
+type agentService struct {
+	state, sub string
+	since      time.Time
+	// note replaces the rest when systemd cannot say.
+	note string
+}
+
+// planeCheck is one control plane the agent dials, and whether it answers.
+type planeCheck struct {
+	addr string
+	// learned is an address the leader named rather than one in agent.yaml.
+	learned bool
+	err     string
+}
+
+type nodeGPU struct {
+	index int
+	name  string
+	// util is -1 where the driver does not report it.
+	util                  float64
+	memUsedMB, memTotalMB float64
+	tempC                 float64
+	models                []string
+}
+
+// engineRun is one engine the supervisor started here, from its state file.
+type engineRun struct {
+	replica, model, engine string
+	gpus                   []int
+	port, pid              int
+	alive                  bool
+	started                time.Time
+}
+
+type diskUse struct {
+	what, path  string
+	free, total uint64
+	err         string
+}
+
+// nodeProbe reads this machine. Every path and tool is a field, so a test
+// reads a directory of its own instead.
+type nodeProbe struct {
+	configured []string
+	planesFile string
+	workDir    string
+	weightsDir string
+	enginesDir string
+	root       bool
+	names      map[int]string
+
+	run    func(ctx context.Context, name string, args ...string) ([]byte, error)
+	dial   func(ctx context.Context, addr string) error
+	gpus   func(ctx context.Context) []discovery.GPUSensor
+	cards  func(ctx context.Context) ([]discovery.GPU, error)
+	alive  func(pid int) bool
+	statfs func(path string) (free, total uint64, err error)
+	now    func() time.Time
+}
+
+// newNodeProbe reads the machine agent.yaml describes, as the agent would.
+func newNodeProbe(cfg any, names map[int]string) nodeProbe {
+	c, ok := cfg.(agent.Config)
+	if !ok {
+		c = agent.DefaultConfig()
+	}
+	var configured []string
+	if c.Server != "" {
+		configured = append(configured, c.Server)
+	}
+	configured = append(configured, c.Servers...)
+	// The same defaults the agent applies as it starts.
+	weights := c.WeightsCache
+	if weights == "" && len(c.ModelDirs) > 0 {
+		weights = c.ModelDirs[0]
+	}
+	if weights == "" {
+		weights = agent.DefaultWeightsCache
+	}
+	enginesDir := c.EnginesDir
+	if enginesDir == "" {
+		enginesDir = provision.DefaultRoot
+	}
+	work := c.Superv.WorkDir
+	if work == "" {
+		work = "/var/lib/llmgw-agent"
+	}
+	return nodeProbe{
+		configured: configured, planesFile: nodeControlPlanesFile, workDir: work,
+		weightsDir: weights, enginesDir: enginesDir, root: os.Geteuid() == 0, names: names,
+		run: runLocal, dial: dialTCP, gpus: discovery.SampleGPUs,
+		cards: func(ctx context.Context) ([]discovery.GPU, error) { return discovery.DiscoverGPUsSMI(ctx, "") },
+		alive: pidAlive, statfs: statfs, now: time.Now,
+	}
+}
+
+// fetchNodeOverview reads the machine in the background.
+func (m consoleModel) fetchNodeOverview() tea.Cmd {
+	p := newNodeProbe(m.cfg, m.nodeOv.names)
+	return func() tea.Msg { return nodeOverviewMsg(p.gather(context.Background())) }
+}
+
+// nodeTick refreshes the overview while it is on screen, one reading at a
+// time; the other screens of a node watch nothing.
+func (m consoleModel) nodeTick() (consoleModel, tea.Cmd) {
+	tick := tea.Tick(3*time.Second, func(time.Time) tea.Msg { return consoleTickMsg{} })
+	if m.menu != cOverview || m.nodeBusy {
+		return m, tick
+	}
+	m.nodeBusy = true
+	return m, tea.Batch(m.fetchNodeOverview(), tick)
+}
+
+// gather reads everything at once, each part in its own goroutine, so the
+// slowest part bounds the whole.
+func (p nodeProbe) gather(ctx context.Context) nodeOverview {
+	ov := nodeOverview{version: version.Version}
+	var wg sync.WaitGroup
+	do := func(f func()) {
+		wg.Add(1)
+		go func() { defer wg.Done(); f() }()
+	}
+	var engines []engineRun
+	do(func() { ov.agent = p.agentState(ctx) })
+	do(func() { ov.planes = p.checkPlanes(ctx) })
+	do(func() { ov.gpus, ov.names, ov.gpuNote = p.readGPUs(ctx) })
+	do(func() { engines, ov.enginesNote = readShardStates(p.workDir, p.alive) })
+	do(func() { ov.disks = p.readDisks() })
+	do(func() { ov.problems, ov.problemsNote = p.readProblems(ctx) })
+	wg.Wait()
+	ov.engines = engines
+	ov.gpus = modelsOnGPUs(ov.gpus, engines)
+	ov.at = p.now()
+	return ov
+}
+
+// --- the agent ---
+
+func (p nodeProbe) agentState(ctx context.Context) agentService {
+	out, err := p.run(ctx, "systemctl", "show", "llmgw-agent",
+		"-p", "LoadState", "-p", "ActiveState", "-p", "SubState", "-p", "ActiveEnterTimestamp")
+	if err != nil && len(out) == 0 {
+		return agentService{note: "systemctl is not available: " + err.Error()}
+	}
+	return parseSystemctlShow(string(out))
+}
+
+// parseSystemctlShow reads `systemctl show -p …`: one key=value a line.
+func parseSystemctlShow(out string) agentService {
+	kv := map[string]string{}
+	for _, line := range strings.Split(out, "\n") {
+		if k, v, ok := strings.Cut(strings.TrimSpace(line), "="); ok {
+			kv[k] = v
+		}
+	}
+	if kv["LoadState"] == "not-found" {
+		return agentService{note: "llmgw-agent is not installed as a service here"}
+	}
+	s := agentService{state: kv["ActiveState"], sub: kv["SubState"]}
+	if s.state == "" {
+		s.note = "systemd does not say"
+	}
+	s.since = parseSystemdTime(kv["ActiveEnterTimestamp"])
+	return s
+}
+
+// parseSystemdTime reads systemd's "Mon 2026-10-19 08:12:03 CEST", in this
+// machine's zone; zero when it is empty or n/a.
+func parseSystemdTime(v string) time.Time {
+	v = strings.TrimSpace(v)
+	if v == "" || v == "n/a" {
+		return time.Time{}
+	}
+	for _, layout := range []string{"Mon 2006-01-02 15:04:05 MST", "Mon 2006-01-02 15:04:05 -0700", "Mon 2006-01-02 15:04:05"} {
+		if t, err := time.ParseInLocation(layout, v, time.Local); err == nil {
+			return t
+		}
+	}
+	return time.Time{}
+}
+
+// --- the control planes ---
+
+// readControlPlanes is every address the agent dials, in its order: the
+// configured ones, then those the leader named that agent.yaml lacks.
+func readControlPlanes(file string, configured []string) []planeCheck {
+	var out []planeCheck
+	seen := map[string]bool{}
+	add := func(s string, learned bool) {
+		hp := hostPort(s)
+		if hp == "" || seen[hp] {
+			return
+		}
+		seen[hp] = true
+		out = append(out, planeCheck{addr: hp, learned: learned})
+	}
+	for _, s := range configured {
+		add(s, false)
+	}
+	if b, err := os.ReadFile(file); err == nil {
+		var told []string
+		if json.Unmarshal(b, &told) == nil {
+			for _, s := range told {
+				add(s, true)
+			}
+		}
+	}
+	return out
+}
+
+// hostPort is the host:port a control plane's address dials, whether it was
+// written as a URL or not.
+func hostPort(s string) string {
+	s = strings.TrimSpace(s)
+	port := ""
+	switch {
+	case strings.HasPrefix(s, "https://"):
+		s, port = strings.TrimPrefix(s, "https://"), "443"
+	case strings.HasPrefix(s, "http://"):
+		s, port = strings.TrimPrefix(s, "http://"), "80"
+	}
+	s, _, _ = strings.Cut(s, "/")
+	if _, _, err := net.SplitHostPort(s); err != nil && port != "" && s != "" {
+		s = net.JoinHostPort(strings.Trim(s, "[]"), port)
+	}
+	return s
+}
+
+func (p nodeProbe) checkPlanes(ctx context.Context) []planeCheck {
+	planes := readControlPlanes(p.planesFile, p.configured)
+	var wg sync.WaitGroup
+	for i := range planes {
+		wg.Add(1)
+		go func(c *planeCheck) {
+			defer wg.Done()
+			if err := p.dial(ctx, c.addr); err != nil {
+				c.err = err.Error()
+			}
+		}(&planes[i])
+	}
+	wg.Wait()
+	return planes
+}
+
+func dialTCP(ctx context.Context, addr string) error {
+	d := net.Dialer{Timeout: nodeCmdTimeout}
+	c, err := d.DialContext(ctx, "tcp", addr)
+	if err != nil {
+		return err
+	}
+	return c.Close()
+}
+
+// --- the GPUs ---
+
+// readGPUs samples the cards, naming them from discovery the first time.
+func (p nodeProbe) readGPUs(ctx context.Context) ([]nodeGPU, map[int]string, string) {
+	ctx, cancel := context.WithTimeout(ctx, nodeCmdTimeout)
+	defer cancel()
+	sensors := p.gpus(ctx)
+	if len(sensors) == 0 {
+		// Without nvidia-smi discovery says nothing and no error: a machine
+		// without a card. With it and no answer, a driver in trouble.
+		if _, err := p.cards(ctx); err != nil {
+			return nil, p.names, "The GPUs do not answer: " + err.Error()
+		}
+		return nil, p.names, "No GPU on this machine."
+	}
+	names := p.names
+	for _, s := range sensors {
+		if names[s.Index] == "" {
+			names = map[int]string{}
+			if cards, err := p.cards(ctx); err == nil {
+				for _, c := range cards {
+					if c.MIG == nil {
+						names[c.Index] = c.Name
+					}
+				}
+			}
+			break
+		}
+	}
+	var out []nodeGPU
+	for _, s := range sensors {
+		g := nodeGPU{index: s.Index, name: names[s.Index], util: s.UtilPercent, tempC: s.TempC,
+			memUsedMB: s.MemoryUsedMB, memTotalMB: s.MemoryTotalMB}
+		if s.MemoryTotalMB == 0 {
+			g.util = -1
+		}
+		out = append(out, g)
+	}
+	return out, names, ""
+}
+
+// cardOf is the card a device number is on: itself, or a MIG instance's card.
+func cardOf(i int) int {
+	if discovery.IsMIGIndex(i) {
+		return (i - discovery.MIGIndex(0, 0)) / 100
+	}
+	return i
+}
+
+// modelsOnGPUs names, on each card, the models whose engines hold it.
+func modelsOnGPUs(gpus []nodeGPU, runs []engineRun) []nodeGPU {
+	for i := range gpus {
+		gpus[i].models = nil
+		for _, r := range runs {
+			for _, d := range r.gpus {
+				if cardOf(d) == gpus[i].index && !slices.Contains(gpus[i].models, r.model) {
+					gpus[i].models = append(gpus[i].models, r.model)
+				}
+			}
+		}
+	}
+	return gpus
+}
+
+// --- the engines ---
+
+// shardStateFile is the part of the supervisor's shard-*.json the overview
+// reads (shardState in the supervisor package).
+type shardStateFile struct {
+	ShardID   string            `json:"shard_id"`
+	ReplicaID string            `json:"replica_id"`
+	PID       int               `json:"pid"`
+	StartedAt time.Time         `json:"started_at"`
+	Spec      engines.ShardSpec `json:"spec"`
+}
+
+// readShardStates is the engines the supervisor keeps state files for in
+// its work directory, with whether each process is still there.
+func readShardStates(dir string, alive func(int) bool) ([]engineRun, string) {
+	entries, err := os.ReadDir(dir)
+	switch {
+	case errors.Is(err, os.ErrPermission):
+		return nil, "The engines' state is root's: run with sudo."
+	case errors.Is(err, os.ErrNotExist):
+		return nil, ""
+	case err != nil:
+		return nil, err.Error()
+	}
+	var runs []engineRun
+	note := ""
+	for _, e := range entries {
+		name := e.Name()
+		if !strings.HasPrefix(name, "shard-") || !strings.HasSuffix(name, ".json") {
+			continue
+		}
+		b, err := os.ReadFile(filepath.Join(dir, name))
+		if errors.Is(err, os.ErrPermission) {
+			note = "The engines' state is root's: run with sudo."
+			continue
+		}
+		var st shardStateFile
+		if err != nil || json.Unmarshal(b, &st) != nil {
+			continue
+		}
+		replica := st.ReplicaID
+		if replica == "" {
+			replica = st.Spec.ReplicaID
+		}
+		model := st.Spec.Profile.ServedModelName
+		if model == "" {
+			model = modelOfReplica(replica)
+		}
+		runs = append(runs, engineRun{replica: replica, model: model, engine: st.Spec.Profile.Engine,
+			gpus: st.Spec.GPUIndices, port: st.Spec.HTTPPort, pid: st.PID, alive: st.PID > 0 && alive(st.PID),
+			started: st.StartedAt})
+	}
+	sort.Slice(runs, func(i, j int) bool { return runs[i].replica < runs[j].replica })
+	return runs, note
+}
+
+// modelOfReplica is the model a replica id names: the control plane calls a
+// replica the model's name and a suffix.
+func modelOfReplica(id string) string {
+	if i := strings.LastIndexByte(id, '-'); i > 0 {
+		return id[:i]
+	}
+	return id
+}
+
+func pidAlive(pid int) bool {
+	err := syscall.Kill(pid, 0)
+	return err == nil || errors.Is(err, syscall.EPERM)
+}
+
+// --- the disks ---
+
+func (p nodeProbe) readDisks() []diskUse {
+	disks := []diskUse{{what: "weights", path: p.weightsDir}, {what: "engines", path: p.enginesDir}}
+	for i := range disks {
+		// In its own goroutine: statfs on a network mount that has gone away
+		// does not return.
+		type res struct {
+			free, total uint64
+			err         error
+		}
+		ch := make(chan res, 1)
+		go func(path string) {
+			f, t, err := p.statfs(path)
+			ch <- res{f, t, err}
+		}(disks[i].path)
+		select {
+		case r := <-ch:
+			disks[i].free, disks[i].total = r.free, r.total
+			if r.err != nil {
+				disks[i].err = r.err.Error()
+				if errors.Is(r.err, os.ErrNotExist) {
+					disks[i].err = "not there yet"
+				}
+			}
+		case <-time.After(nodeCmdTimeout):
+			disks[i].err = "does not answer"
+		}
+	}
+	return disks
+}
+
+func statfs(path string) (free, total uint64, err error) {
+	var st syscall.Statfs_t
+	if err := syscall.Statfs(path, &st); err != nil {
+		return 0, 0, &os.PathError{Op: "statfs", Path: path, Err: err}
+	}
+	//nolint:unconvert // the field types differ between systems
+	return uint64(st.Bavail) * uint64(st.Bsize), uint64(st.Blocks) * uint64(st.Bsize), nil
+}
+
+// --- the problems ---
+
+func (p nodeProbe) readProblems(ctx context.Context) ([]string, string) {
+	out, err := p.run(ctx, "journalctl", "-u", "llmgw-agent", "-p", "warning", "--since", "-1h",
+		"--no-pager", "-o", "cat", "-n", strconv.Itoa(nodeProblemLines))
+	if err != nil && len(out) == 0 {
+		return nil, "The journal cannot be read: " + err.Error()
+	}
+	lines := parseJournal(string(out), nodeProblemLines)
+	if len(lines) == 0 && !p.root {
+		return nil, "None that this user can see: run with sudo to read the agent's journal."
+	}
+	return lines, ""
+}
+
+// parseJournal keeps the last n lines journalctl printed, without its own
+// remarks.
+func parseJournal(out string, n int) []string {
+	var lines []string
+	sc := bufio.NewScanner(strings.NewReader(out))
+	sc.Buffer(make([]byte, 64*1024), 1024*1024)
+	for sc.Scan() {
+		l := strings.TrimSpace(sc.Text())
+		if l == "" || strings.HasPrefix(l, "-- ") || strings.HasPrefix(l, "Hint: ") {
+			continue
+		}
+		lines = append(lines, l)
+	}
+	if len(lines) > n {
+		lines = lines[len(lines)-n:]
+	}
+	return lines
+}
+
+// runLocal runs a tool on this machine, bounded by nodeCmdTimeout. A tool
+// that is not installed is said so, not waited for.
+func runLocal(ctx context.Context, name string, args ...string) ([]byte, error) {
+	if _, err := exec.LookPath(name); err != nil {
+		return nil, fmt.Errorf("%s is not installed", name)
+	}
+	ctx, cancel := context.WithTimeout(ctx, nodeCmdTimeout)
+	defer cancel()
+	//nolint:gosec // fixed tools and arguments, from this file
+	return exec.CommandContext(ctx, name, args...).Output()
+}
+
+// --- the panel ---
+
+func (m consoleModel) nodeOverviewPanel(pw int) (string, string, string) {
+	keys := "↑↓ choose · PgUp/PgDn scroll · Q leave · refreshed every three seconds"
+	if m.nodeOv.at.IsZero() {
+		return "Overview", "\n  " + m.spin.View() + " reading this machine…", keys
+	}
+	return "Overview", renderNodeOverview(m.nodeOv, pw, time.Now()), keys
+}
+
+// renderNodeOverview draws the overview from what was read.
+func renderNodeOverview(ov nodeOverview, pw int, now time.Time) string {
+	var b strings.Builder
+	head := func(s string) { b.WriteString("\n  " + lipgloss.NewStyle().Bold(true).Render(s) + "\n") }
+	kv := func(k, v string) { fmt.Fprintf(&b, "    %-14s %s\n", sDim.Render(k), v) }
+	line := func(s string) { b.WriteString("    " + s + "\n") }
+
+	head("Agent")
+	a := ov.agent
+	switch {
+	case a.note != "":
+		kv("llmgw-agent", sWarn.Render(a.note))
+	default:
+		state := a.state
+		if a.sub != "" {
+			state += " (" + a.sub + ")"
+		}
+		if a.state == "active" {
+			state = sOK.Render(state)
+		} else {
+			state = sBad.Render(state)
+		}
+		if !a.since.IsZero() {
+			state += sDim.Render(" since " + a.since.Format("2006-01-02 15:04") + ", " + uptime(now.Sub(a.since)) + " ago")
+		}
+		kv("llmgw-agent", state)
+	}
+	kv("llmgw", ov.version)
+
+	head("Control planes")
+	if len(ov.planes) == 0 {
+		line(sWarn.Render("None configured: agent.yaml names no server."))
+	}
+	for _, c := range ov.planes {
+		state := sOK.Render("reachable")
+		if c.err != "" {
+			state = sBad.Render("✗ ") + truncate(c.err, max(20, pw-30))
+		}
+		if c.learned {
+			state += sDim.Render("  (named by the leader)")
+		}
+		fmt.Fprintf(&b, "    %-22s %s\n", c.addr, state)
+	}
+
+	head("GPUs")
+	if len(ov.gpus) == 0 {
+		line(sDim.Render(ov.gpuNote))
+	} else {
+		var rows [][]string
+		for _, g := range ov.gpus {
+			busy, mem, temp := "—", "—", "—"
+			if g.util >= 0 {
+				busy = fmt.Sprintf("%.0f%%", g.util)
+			}
+			if g.memTotalMB > 0 {
+				mem = fmt.Sprintf("%.1f/%.0f GB", g.memUsedMB/1024, g.memTotalMB/1024)
+			}
+			if g.tempC > 0 {
+				temp = fmt.Sprintf("%.0f°C", g.tempC)
+			}
+			models := strings.Join(g.models, ", ")
+			if models == "" {
+				models = sDim.Render("free")
+			}
+			rows = append(rows, []string{strconv.Itoa(g.index), g.name, busy, mem, temp, models})
+		}
+		b.WriteString(indent(table([]string{"GPU", "CARD", "BUSY", "MEMORY", "TEMP", "MODELS"}, rows, -1, pw-2, false), "  ") + "\n")
+	}
+
+	head("Engines running here")
+	switch {
+	case len(ov.engines) > 0:
+		var rows [][]string
+		for _, r := range ov.engines {
+			// The process by its pid, marked when it is no longer there.
+			proc := sOK.Render(strconv.Itoa(r.pid))
+			if !r.alive {
+				proc = sBad.Render(strconv.Itoa(r.pid) + " gone")
+			}
+			gpus := strings.Join(discovery.DeviceLabels(r.gpus), ",")
+			if gpus == "" {
+				gpus = sDim.Render("CPU")
+			}
+			up := "—"
+			if !r.started.IsZero() {
+				up = uptime(now.Sub(r.started))
+			}
+			port := "—"
+			if r.port > 0 {
+				port = strconv.Itoa(r.port)
+			}
+			rows = append(rows, []string{r.replica, r.model, r.engine, gpus, port, proc, up})
+		}
+		b.WriteString(indent(table([]string{"REPLICA", "MODEL", "ENGINE", "GPUS", "PORT", "PID", "UP"}, rows, -1, pw-2, false), "  ") + "\n")
+		if ov.enginesNote != "" {
+			line(sWarn.Render(ov.enginesNote))
+		}
+	case ov.enginesNote != "":
+		line(sWarn.Render(ov.enginesNote))
+	default:
+		line(sDim.Render("None: the control plane has placed nothing here."))
+	}
+
+	head("Disk")
+	for _, d := range ov.disks {
+		use := sWarn.Render(d.err)
+		if d.err == "" {
+			use = fmt.Sprintf("%s free of %s", diskSize(d.free), diskSize(d.total))
+			if d.total > 0 && d.free*10 < d.total {
+				use = sWarn.Render(use)
+			}
+		}
+		fmt.Fprintf(&b, "    %-14s %-28s %s\n", sDim.Render(d.what), d.path, use)
+	}
+
+	head("Recent problems")
+	switch {
+	case ov.problemsNote != "":
+		line(sDim.Render(ov.problemsNote))
+	case len(ov.problems) == 0:
+		line(sOK.Render("None in the last hour."))
+	default:
+		for _, l := range ov.problems {
+			l = truncateWidth(l, max(20, pw-4))
+			if strings.Contains(l, "level=ERROR") {
+				l = sBad.Render(l)
+			} else {
+				l = sWarn.Render(l)
+			}
+			line(l)
+		}
+	}
+	b.WriteString("\n  " + sDim.Render("read at "+ov.at.Format("15:04:05")) + "\n")
+	return b.String()
+}
+
+// uptime is a duration as people say it: 3d4h, 2h15m, 40s.
+func uptime(d time.Duration) string {
+	d = d.Round(time.Second)
+	switch {
+	case d < 0:
+		return "0s"
+	case d >= 24*time.Hour:
+		return fmt.Sprintf("%dd%dh", d/(24*time.Hour), (d%(24*time.Hour))/time.Hour)
+	case d >= time.Hour:
+		return fmt.Sprintf("%dh%dm", d/time.Hour, (d%time.Hour)/time.Minute)
+	case d >= time.Minute:
+		return fmt.Sprintf("%dm%ds", d/time.Minute, (d%time.Minute)/time.Second)
+	}
+	return d.String()
+}
+
+// diskSize is a byte count in the unit disks are sold in.
+func diskSize(n uint64) string {
+	switch {
+	case n >= 1<<40:
+		return fmt.Sprintf("%.1f TB", float64(n)/(1<<40))
+	case n >= 1<<30:
+		return fmt.Sprintf("%.0f GB", float64(n)/(1<<30))
+	}
+	return fmt.Sprintf("%.0f MB", float64(n)/(1<<20))
+}

internal/cli/tui_node_overview_test.go

@@ -0,0 +1,206 @@
+package cli
+
+import (
+	"context"
+	"errors"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/charmbracelet/x/ansi"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+)
+
+func writeNodeFile(t *testing.T, path, body string) {
+	t.Helper()
+	if err := os.WriteFile(path, []byte(body), 0o600); err != nil {
+		t.Fatal(err)
+	}
+}
+
+func TestANodesOverviewDialsTheConfiguredControlPlanesThenTheLeadersOwn(t *testing.T) {
+	dir := t.TempDir()
+	file := filepath.Join(dir, "control-planes.json")
+	writeNodeFile(t, file, `["https://10.0.0.150:7443","10.0.0.140:7443","https://cp3.example/agent"]`)
+	got := readControlPlanes(file, []string{"10.0.0.140:7443", "http://10.0.0.141"})
+	want := []planeCheck{{addr: "10.0.0.140:7443"}, {addr: "10.0.0.141:80"},
+		{addr: "10.0.0.150:7443", learned: true}, {addr: "cp3.example:443", learned: true}}
+	if len(got) != len(want) {
+		t.Fatalf("got %+v, want %+v", got, want)
+	}
+	for i := range want {
+		if got[i] != want[i] {
+			t.Errorf("%d: got %+v, want %+v", i, got[i], want[i])
+		}
+	}
+	// Without the file, what agent.yaml says.
+	if got := readControlPlanes(filepath.Join(dir, "none.json"), []string{"cp:7443"}); len(got) != 1 || got[0].addr != "cp:7443" {
+		t.Errorf("without the file: %+v", got)
+	}
+}
+
+func TestANodesOverviewReadsTheEnginesFromTheSupervisorsStateFiles(t *testing.T) {
+	dir := t.TempDir()
+	writeNodeFile(t, filepath.Join(dir, "shard-qwen3-8b-ab12-0.json"), `{"shard_id":"qwen3-8b-ab12-0","replica_id":"qwen3-8b-ab12",
+		"pid":4242,"started_at":"2026-10-19T08:00:00Z","spec":{"ReplicaID":"qwen3-8b-ab12","GPUIndices":[0,1],"HTTPPort":8101,
+		"Profile":{"Name":"vllm-tp2","Engine":"vllm","ServedModelName":"qwen3-8b"}}}`)
+	writeNodeFile(t, filepath.Join(dir, "shard-bge-m3-7-0.json"), `{"replica_id":"bge-m3-7","pid":99,
+		"spec":{"GPUIndices":[1002],"HTTPPort":8102,"Profile":{"Engine":"llamacpp"}}}`)
+	writeNodeFile(t, filepath.Join(dir, "shard-x.json.tmp"), `{"replica_id":"half-written"}`)
+	writeNodeFile(t, filepath.Join(dir, "control-planes.json"), `[]`)
+	runs, note := readShardStates(dir, func(pid int) bool { return pid == 4242 })
+	if note != "" {
+		t.Fatalf("note %q", note)
+	}
+	if len(runs) != 2 {
+		t.Fatalf("got %+v", runs)
+	}
+	b, q := runs[0], runs[1]
+	if q.replica != "qwen3-8b-ab12" || q.model != "qwen3-8b" || q.engine != "vllm" || q.port != 8101 || !q.alive ||
+		len(q.gpus) != 2 || !q.started.Equal(time.Date(2026, 10, 19, 8, 0, 0, 0, time.UTC)) {
+		t.Errorf("qwen: %+v", q)
+	}
+	// No served name: the model from the replica's id.
+	if b.model != "bge-m3" || b.alive || b.engine != "llamacpp" {
+		t.Errorf("bge: %+v", b)
+	}
+	if runs, note := readShardStates(filepath.Join(dir, "absent"), pidAlive); runs != nil || note != "" {
+		t.Errorf("no work dir: %+v %q", runs, note)
+	}
+}
+
+func TestANodesOverviewNamesTheModelOnEachGPU(t *testing.T) {
+	gpus := []nodeGPU{{index: 0}, {index: 1}, {index: 2}}
+	runs := []engineRun{
+		{model: "qwen3-8b", gpus: []int{0, 1}},
+		{model: "bge-m3", gpus: []int{discovery.MIGIndex(1, 2)}},
+		{model: "qwen3-8b", gpus: []int{1}},
+	}
+	got := modelsOnGPUs(gpus, runs)
+	if strings.Join(got[0].models, ",") != "qwen3-8b" || strings.Join(got[1].models, ",") != "qwen3-8b,bge-m3" ||
+		len(got[2].models) != 0 {
+		t.Errorf("got %+v", got)
+	}
+}
+
+func TestANodesOverviewReadsWhatSystemdSaysOfTheAgent(t *testing.T) {
+	s := parseSystemctlShow("LoadState=loaded\nActiveState=active\nSubState=running\nActiveEnterTimestamp=Mon 2026-10-19 08:12:03 UTC\n")
+	if s.state != "active" || s.sub != "running" || s.note != "" || s.since.IsZero() || s.since.Minute() != 12 {
+		t.Errorf("got %+v", s)
+	}
+	if s := parseSystemctlShow("LoadState=not-found\nActiveState=inactive\nActiveEnterTimestamp=\n"); s.note == "" {
+		t.Errorf("a missing service: %+v", s)
+	}
+	if s := parseSystemctlShow("LoadState=loaded\nActiveState=failed\nSubState=failed\nActiveEnterTimestamp=n/a\n"); s.state != "failed" ||
+		!s.since.IsZero() {
+		t.Errorf("a failed service: %+v", s)
+	}
+}
+
+func TestANodesOverviewKeepsTheLastWarningsOfTheJournal(t *testing.T) {
+	var b strings.Builder
+	for i := 1; i <= 9; i++ {
+		b.WriteString("level=WARN msg=w" + string(rune('0'+i)) + "\n")
+	}
+	got := parseJournal(b.String(), 6)
+	if len(got) != 6 || got[0] != "level=WARN msg=w4" || got[5] != "level=WARN msg=w9" {
+		t.Errorf("got %q", got)
+	}
+	if got := parseJournal("-- No entries --\n", 6); len(got) != 0 {
+		t.Errorf("no entries: %q", got)
+	}
+}
+
+// fakeProbe is a node with two cards, one engine and a broken second control
+// plane, and no tool of the machine running the test touched.
+func fakeProbe(t *testing.T) nodeProbe {
+	dir := t.TempDir()
+	writeNodeFile(t, filepath.Join(dir, "shard-qwen3-8b-ab12-0.json"), `{"replica_id":"qwen3-8b-ab12","pid":4242,
+		"started_at":"2026-10-19T08:00:00Z","spec":{"GPUIndices":[0],"HTTPPort":8101,"Profile":{"Engine":"vllm","ServedModelName":"qwen3-8b"}}}`)
+	return nodeProbe{
+		configured: []string{"10.0.0.140:7443", "10.0.0.150:7443"},
+		planesFile: filepath.Join(dir, "control-planes.json"),
+		workDir:    dir, weightsDir: "/var/lib/llmgw-weights", enginesDir: "/opt/llmgw/engines", root: true,
+		run: func(_ context.Context, name string, _ ...string) ([]byte, error) {
+			switch name {
+			case "systemctl":
+				return []byte("LoadState=loaded\nActiveState=active\nSubState=running\nActiveEnterTimestamp=Mon 2026-10-19 08:00:00 UTC\n"), nil
+			case "journalctl":
+				return []byte("level=WARN msg=\"the control plane does not answer\" server=10.0.0.150:7443\n"), nil
+			}
+			return nil, errors.New(name + " is not installed")
+		},
+		dial: func(_ context.Context, addr string) error {
+			if addr == "10.0.0.150:7443" {
+				return errors.New("dial tcp 10.0.0.150:7443: connect: connection refused")
+			}
+			return nil
+		},
+		gpus: func(context.Context) []discovery.GPUSensor {
+			return []discovery.GPUSensor{
+				{Index: 0, TempC: 61, UtilPercent: 87, MemoryUsedMB: 20480, MemoryTotalMB: 24576},
+				{Index: 1, TempC: 35, UtilPercent: 0, MemoryUsedMB: 300, MemoryTotalMB: 24576},
+			}
+		},
+		cards: func(context.Context) ([]discovery.GPU, error) {
+			return []discovery.GPU{{Index: 0, Name: "NVIDIA RTX A5000"}, {Index: 1, Name: "NVIDIA RTX A5000"}}, nil
+		},
+		alive:  func(int) bool { return true },
+		statfs: func(string) (uint64, uint64, error) { return 120 << 30, 500 << 30, nil },
+		now:    func() time.Time { return time.Date(2026, 10, 19, 11, 12, 0, 0, time.UTC) },
+	}
+}
+
+func TestANodesOverviewShowsWhatRunsOnTheMachine(t *testing.T) {
+	ov := fakeProbe(t).gather(context.Background())
+	out := ansi.Strip(renderNodeOverview(ov, 100, ov.at))
+	t.Log("\n" + out)
+	for _, want := range []string{
+		"llmgw-agent", "active (running)", "3h12m ago",
+		"10.0.0.140:7443", "reachable", "connection refused",
+		"NVIDIA RTX A5000", "87%", "20.0/24 GB", "61°C", "qwen3-8b", "free",
+		"qwen3-8b-ab12", "vllm", "8101", "4242", "3h12m",
+		"/var/lib/llmgw-weights", "120 GB free of 500 GB", "/opt/llmgw/engines",
+		"the control plane does not answer",
+	} {
+		if !strings.Contains(out, want) {
+			t.Errorf("the overview lacks %q", want)
+		}
+	}
+	if ov.names[0] != "NVIDIA RTX A5000" {
+		t.Errorf("the names are not kept for the next reading: %v", ov.names)
+	}
+}
+
+func TestANodeWithoutAGPUSaysSoInOneLine(t *testing.T) {
+	p := fakeProbe(t)
+	p.gpus = func(context.Context) []discovery.GPUSensor { return nil }
+	p.cards = func(context.Context) ([]discovery.GPU, error) { return nil, nil }
+	p.run = func(context.Context, string, ...string) ([]byte, error) { return nil, errors.New("not installed") }
+	ov := p.gather(context.Background())
+	out := ansi.Strip(renderNodeOverview(ov, 100, ov.at))
+	if !strings.Contains(out, "No GPU on this machine.") {
+		t.Errorf("no line saying there is no GPU:\n%s", out)
+	}
+	if strings.Contains(out, "BUSY") {
+		t.Errorf("an empty GPU table:\n%s", out)
+	}
+	// No systemctl or journalctl is said, not a crash.
+	if !strings.Contains(out, "systemctl is not available") || !strings.Contains(out, "The journal cannot be read") {
+		t.Errorf("missing tools are not said:\n%s", out)
+	}
+}
+
+func TestANodesConsoleOpensOnItsOverview(t *testing.T) {
+	m := newConsoleModel("127.0.0.1:1", filepath.Join(t.TempDir(), "agent.yaml"))
+	if m.menu != cOverview || !m.entryOK(cOverview) {
+		t.Fatalf("menu %d", m.menu)
+	}
+	_, body, _ := m.panel()
+	if !strings.Contains(body, "reading this machine") {
+		t.Errorf("before the first reading: %q", body)
+	}
+}

internal/cli/tui_setup.go

@@ -188,6 +188,8 @@ func newSetupModel(host string, ips []string) setupModel {
 	// nodes are added once it is up. Either can still be chosen.
 	in.noNode, in.noModel = true, true
 	in.standbys = true
+	// Two judges, on two of the three machines' nodes.
+	in.judge.Replicas = 2
 	return setupModel{
 		ref: &progRef{}, host: host, ips: ips, in: &in, choices: newInstallForm(&in, host, ips, true),
 		db: [2]dbMachine{newDBMachine(1), newDBMachine(2)}, spin: newSpinner(), width: 100, height: 30,

internal/cli/tui_signin.go

@@ -0,0 +1,664 @@
+package cli
+
+import (
+	"context"
+	"crypto/rand"
+	"encoding/hex"
+	"errors"
+	"fmt"
+	"net/http"
+	"os"
+	"strings"
+	"time"
+
+	tea "github.com/charmbracelet/bubbletea"
+	"go.yaml.in/yaml/v3"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
+)
+
+// How people sign in, as the console shows and sets it: single sign-on
+// through Entra ID or another OpenID provider, an Active Directory checked
+// over LDAPS, and provisioning by SCIM. The secrets these need are kept in
+// the store, so every control plane has them without a file put on each,
+// and a directory is tried before its settings are kept.
+
+// inChainRule is AD's matching rule for nested group membership.
+const inChainRule = "1.2.840.113556.1.4.1941"
+
+type signInState struct {
+	view   adminapi.AuthView
+	err    error
+	loaded bool
+	// ldap and oidc are the blocks the cluster's settings hold now, for the
+	// forms to start from.
+	ldap config.LDAP
+	oidc config.OIDC
+}
+
+type consoleSignInMsg struct {
+	view adminapi.AuthView
+	ldap config.LDAP
+	oidc config.OIDC
+	err  error
+}
+
+// signInRows are the panel's rows, one per way in.
+var signInRows = []string{"oidc", "ldap", "scim"}
+
+func (m consoleModel) fetchSignIn() tea.Cmd {
+	c := m.client
+	return func() tea.Msg {
+		ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+		defer cancel()
+		var msg consoleSignInMsg
+		if msg.err = c.get(ctx, "/api/v1/auth", &msg.view); msg.err != nil {
+			return msg
+		}
+		var cur adminapi.SettingsView
+		if err := c.get(ctx, "/api/v1/settings", &cur); err == nil {
+			msg.ldap, msg.oidc = authBlocks(cur.Body)
+		}
+		return msg
+	}
+}
+
+// authBlocks are auth.ldap and auth.oidc as a settings document has them.
+func authBlocks(body string) (config.LDAP, config.OIDC) {
+	var doc struct {
+		Auth struct {
+			LDAP yaml.Node `yaml:"ldap"`
+			OIDC yaml.Node `yaml:"oidc"`
+		} `yaml:"auth"`
+	}
+	var l config.LDAP
+	var o config.OIDC
+	if yaml.Unmarshal([]byte(body), &doc) != nil {
+		return l, o
+	}
+	if doc.Auth.LDAP.Kind != 0 {
+		if b, err := yaml.Marshal(&doc.Auth.LDAP); err == nil {
+			// Read as kept, whether or not it would validate now.
+			l, _ = config.ParseLDAP(b)
+		}
+	}
+	if doc.Auth.OIDC.Kind != 0 {
+		if m, err := yaml.Marshal(&doc.Auth.OIDC); err == nil {
+			if cfg, err := config.FromMap(map[string]any{"auth": map[string]any{"oidc": yamlMap(m)}}); err == nil {
+				o = cfg.Auth.OIDC
+			}
+		}
+	}
+	return l, o
+}
+
+func yamlMap(b []byte) map[string]any {
+	var m map[string]any
+	_ = yaml.Unmarshal(b, &m)
+	return m
+}
+
+func (m consoleModel) signInPanel(pw int) (string, string, string) {
+	keys := "→ or Enter open · ↑↓ choose · Q leave"
+	if m.inPanel {
+		keys = "↑↓ row · Enter actions · E Entra ID · D Active Directory · T try the directory · ← back · Q leave"
+	}
+	s := m.signIn
+	var b strings.Builder
+	b.WriteString("\n  " + sDim.Render("How people sign in besides a password kept here. E for Entra ID, D for Active Directory.") + "\n\n")
+	if s.err != nil {
+		b.WriteString("  " + sBad.Render(s.err.Error()))
+		return "Sign-in", b.String(), keys
+	}
+	if !s.loaded {
+		b.WriteString("  " + sDim.Render("asking the control plane…"))
+		return "Sign-in", b.String(), keys
+	}
+	v := s.view
+	local := "passwords kept here; accounts made by an administrator"
+	if v.SignupOpen {
+		local = "passwords kept here; anyone may sign up, then wait for approval"
+	}
+	b.WriteString("  " + sTitle.Render("LOCAL") + "  " + sDim.Render(local) + "\n\n")
+
+	var rows [][]string
+	for _, r := range signInRows {
+		switch r {
+		case "oidc":
+			if v.OIDC == nil {
+				rows = append(rows, []string{"Single sign-on (Entra ID, OIDC)", "off", "E to set up"})
+				continue
+			}
+			state := "on"
+			if v.OIDC.NoPasswords {
+				state = "on, the only way in"
+			}
+			rows = append(rows, []string{"Single sign-on (Entra ID, OIDC)", state, or(v.OIDC.Label, v.OIDC.Issuer)})
+		case "ldap":
+			if v.LDAP == nil {
+				rows = append(rows, []string{"Active Directory (LDAP)", "off", "D to set up"})
+				continue
+			}
+			state := "on"
+			if v.LDAP.Off != "" {
+				state = "broken here"
+			}
+			rows = append(rows, []string{"Active Directory (LDAP)", state, strings.Join(v.LDAP.URLs, " ")})
+		case "scim":
+			state, detail := "off", "Enter to make a token"
+			if v.SCIM {
+				state, detail = "on", "at /scim/v2"
+			}
+			rows = append(rows, []string{"Provisioning (SCIM)", state, detail})
+		}
+	}
+	b.WriteString(table([]string{"WAY IN", "STATE", ""}, rows, m.row, pw, m.inPanel) + "\n")
+
+	if v.OIDC != nil {
+		b.WriteString("\n  " + sTitle.Render("SINGLE SIGN-ON") + "\n")
+		b.WriteString("  issuer     " + v.OIDC.Issuer + "\n")
+		b.WriteString("  redirect   " + redirectURL(v.BaseURL) + "\n")
+		b.WriteString("  " + sDim.Render(pickText(v.OIDC.CreateAccounts, "accounts made at first sign-in", "only people with an account sign in")) + "\n")
+	}
+	if l := v.LDAP; l != nil {
+		b.WriteString("\n  " + sTitle.Render("ACTIVE DIRECTORY") + "  " + sDim.Render(l.BaseDN) + "\n")
+		if l.Off != "" {
+			b.WriteString("  " + sBad.Render("off on this control plane: "+l.Off) + "\n")
+		}
+		switch ls := l.LastSync; {
+		case ls == nil:
+			b.WriteString("  " + sDim.Render("not read yet: the leader reads it every sync_interval") + "\n")
+		case ls.Error != "":
+			b.WriteString("  " + sBad.Render("last read "+agoText(ls.At)+": "+ls.Error) + "\n")
+		default:
+			b.WriteString(fmt.Sprintf("  last read %s: %d people there, %d accounts kept in step, %d suspended, %d reinstated\n",
+				agoText(ls.At), ls.People, ls.Accounts, ls.Suspended, ls.Reinstated))
+		}
+	}
+	return "Sign-in", b.String(), keys
+}
+
+func agoText(t time.Time) string { return time.Since(t).Round(time.Second).String() + " ago" }
+
+func pickText(b bool, yes, no string) string {
+	if b {
+		return yes
+	}
+	return no
+}
+
+// redirectURL is what an identity provider sends people back to.
+func redirectURL(base string) string {
+	if base == "" {
+		return "https://<this gateway>/login/oidc/callback"
+	}
+	return strings.TrimRight(base, "/") + "/login/oidc/callback"
+}
+
+// signInMenu is what can be done to one way in.
+func (m consoleModel) signInMenu(row string) *menuDialog {
+	v := m.signIn.view
+	switch row {
+	case "oidc":
+		d := &menuDialog{title: "Single sign-on", items: []menuItem{
+			{"Set up Entra ID…", func(m *consoleModel) tea.Cmd { m.entraForm(); return nil }},
+		}}
+		if v.OIDC != nil {
+			d.items = append(d.items, menuItem{"Turn off…", func(m *consoleModel) tea.Cmd {
+				m.signInOffConfirm("oidc", "Single sign-on", "People sign in with a password kept here again; accounts made through the provider have none.")
+				return nil
+			}})
+		}
+		return d
+	case "ldap":
+		d := &menuDialog{title: "Active Directory", items: []menuItem{
+			{"Set up…", func(m *consoleModel) tea.Cmd { m.directoryForm(); return nil }},
+		}}
+		if v.LDAP != nil {
+			d.items = append(d.items,
+				menuItem{"Try it", func(m *consoleModel) tea.Cmd { return m.tryDirectory() }},
+				menuItem{"Turn off…", func(m *consoleModel) tea.Cmd {
+					m.signInOffConfirm("ldap", "Active Directory", "Its people can no longer sign in; their accounts stay, and are no longer kept in step.")
+					return nil
+				}})
+		}
+		return d
+	default:
+		d := &menuDialog{title: "Provisioning (SCIM)", items: []menuItem{
+			{"Make a new token", func(m *consoleModel) tea.Cmd { return m.scimToken() }},
+		}}
+		if v.SCIM {
+			d.items = append(d.items, menuItem{"Turn off…", func(m *consoleModel) tea.Cmd {
+				m.signInOffConfirm("scim", "Provisioning", "The provider can no longer make, change or suspend accounts here.")
+				return nil
+			}})
+		}
+		return d
+	}
+}
+
+// signInRowKey is the way in the cursor is on.
+func (m consoleModel) signInRowKey() string {
+	if m.row >= 0 && m.row < len(signInRows) {
+		return signInRows[m.row]
+	}
+	return ""
+}
+
+// directoryForm sets up auth.ldap, starting from what is kept.
+func (m *consoleModel) directoryForm() {
+	l := m.signIn.ldap
+	allowed := ""
+	if f := l.UserFilter; strings.HasPrefix(f, "(memberOf:"+inChainRule+":=") {
+		allowed = strings.TrimSuffix(strings.TrimPrefix(f, "(memberOf:"+inChainRule+":="), ")")
+	} else {
+		allowed = f
+	}
+	pwHint := "the service account's password"
+	if l.BindPasswordFile != "" {
+		pwHint = "empty keeps the one kept"
+	}
+	caHint := "a PEM file on this machine; empty uses the system's"
+	if l.CA != "" || l.CAFile != "" {
+		caHint = "empty keeps the one kept; - for the system's"
+	}
+	create := "yes"
+	if l.Enabled() && !l.CreateAccounts {
+		create = "no"
+	}
+	var teams []string
+	for g, place := range l.Memberships {
+		teams = append(teams, g+"="+place)
+	}
+	pw := textF("password", "", pwHint)
+	pw.text.Mask = true
+	m.openForm(&formDialog{title: "Active Directory",
+		note: "People sign in with their AD name and password, checked over LDAPS. Tried before it is kept.",
+		fields: []formField{
+			textF("servers", strings.Join(l.URLs, " "), "ldaps://dc1.corp.example ldaps://dc2.corp.example"),
+			textF("people under", l.BaseDN, "OU=Staff,DC=corp,DC=example"),
+			textF("service account", l.BindDN, "CN=svc-llmgw,OU=Service,DC=corp,DC=example"),
+			pw,
+			textF("CA certificate", "", caHint),
+			textF("who may sign in", allowed, "a group's DN, nested groups count; empty is everyone under it"),
+			textF("administrators", strings.Join(l.Roles["admin"], ", "), "group names or DNs"),
+			textF("operators", strings.Join(l.Roles["operator"], ", "), "group names or DNs"),
+			textF("teams", strings.Join(teams, "; "), "ml-team=team:Acme/ml; sales=org:Sales"),
+			textF("label", l.Label, "the form says: Sign in with your LABEL account"),
+			choiceF("make accounts", []string{"yes", "no"}, create),
+		},
+		submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
+			next, err := directoryFromForm(l, v)
+			if err != nil {
+				return nil, err
+			}
+			return m.saveDirectory(next, v[3]), nil
+		}})
+}
+
+// directoryFromForm is the auth.ldap block the form's values make, over
+// what was kept.
+func directoryFromForm(kept config.LDAP, v []string) (config.LDAP, error) {
+	l := kept
+	l.URLs = strings.Fields(strings.ReplaceAll(v[0], ",", " "))
+	l.BaseDN, l.BindDN = v[1], v[2]
+	if v[3] != "" || l.BindPasswordFile == "" {
+		l.BindPasswordFile = adminapi.AuthSecretRef("ldap-bind")
+	}
+	switch ca := v[4]; ca {
+	case "":
+	case "-":
+		l.CA, l.CAFile = "", ""
+	default:
+		b, err := os.ReadFile(ca) //nolint:gosec // a file the operator names
+		if err != nil {
+			return l, fmt.Errorf("CA certificate: %w", err)
+		}
+		l.CA, l.CAFile = string(b), ""
+	}
+	switch g := v[5]; {
+	case g == "":
+		l.UserFilter = ""
+	case strings.HasPrefix(g, "("):
+		l.UserFilter = g
+	case strings.Contains(g, "="):
+		l.UserFilter = "(memberOf:" + inChainRule + ":=" + g + ")"
+	default:
+		return l, errors.New("who may sign in: a group's DN (CN=…,OU=…,DC=…), or an LDAP filter in parentheses")
+	}
+	l.Roles = map[string][]string{}
+	for role, i := range map[string]int{"admin": 6, "operator": 7} {
+		for _, g := range strings.Split(v[i], ",") {
+			if g = strings.TrimSpace(g); g != "" {
+				l.Roles[role] = append(l.Roles[role], g)
+			}
+		}
+	}
+	if len(l.Roles) == 0 {
+		l.Roles = nil
+	}
+	l.Memberships = nil
+	for _, pair := range strings.Split(v[8], ";") {
+		if pair = strings.TrimSpace(pair); pair == "" {
+			continue
+		}
+		g, place, ok := strings.Cut(pair, "=")
+		if !ok {
+			return l, fmt.Errorf("teams: %q is not GROUP=team:ORG/TEAM", pair)
+		}
+		if l.Memberships == nil {
+			l.Memberships = map[string]string{}
+		}
+		l.Memberships[strings.TrimSpace(g)] = strings.TrimSpace(place)
+	}
+	l.Label, l.CreateAccounts = v[9], v[10] == "yes"
+	return l, l.Validate()
+}
+
+// ldapYAML is a block as the cluster's settings write it.
+func ldapYAML(l config.LDAP) string {
+	m := map[string]any{"urls": l.URLs, "bind_dn": l.BindDN, "bind_password_file": l.BindPasswordFile,
+		"base_dn": l.BaseDN, "create_accounts": l.CreateAccounts}
+	put := func(k, v string) {
+		if v != "" {
+			m[k] = v
+		}
+	}
+	put("ca", l.CA)
+	put("ca_file", l.CAFile)
+	put("user_filter", l.UserFilter)
+	put("group_base_dn", l.GroupBaseDN)
+	put("label", l.Label)
+	if l.StartTLS {
+		m["start_tls"] = true
+	}
+	if len(l.Roles) > 0 {
+		m["roles"] = l.Roles
+	}
+	if len(l.Memberships) > 0 {
+		m["memberships"] = l.Memberships
+	}
+	if l.SyncInterval != 0 {
+		m["sync_interval"] = l.SyncInterval.String()
+	}
+	if l.Timeout != 0 {
+		m["timeout"] = l.Timeout.String()
+	}
+	b, _ := yaml.Marshal(m)
+	return string(b)
+}
+
+// saveDirectory tries the block, and only if every step passes keeps the
+// password and puts the block in the cluster's settings.
+func (m *consoleModel) saveDirectory(l config.LDAP, password string) tea.Cmd {
+	m.busy = "Active Directory"
+	c, block := m.client, ldapYAML(l)
+	return func() tea.Msg {
+		ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)
+		defer cancel()
+		var tv adminapi.LDAPTestView
+		if err := c.do(ctx, http.MethodPost, "/api/v1/auth/ldap/test",
+			adminapi.LDAPTestRequest{Settings: block, Password: password}, &tv); err != nil {
+			return actionDoneMsg{title: "Active Directory", err: err}
+		}
+		report := checksText(tv.Checks)
+		if !tv.OK {
+			return actionDoneMsg{title: "Active Directory", err: errors.New("not kept: the directory did not pass\n\n" + report)}
+		}
+		if password != "" {
+			if err := c.do(ctx, http.MethodPut, "/api/v1/auth/secrets/ldap-bind", adminapi.SecretRequest{Value: password}, nil); err != nil {
+				return actionDoneMsg{title: "Active Directory", err: err}
+			}
+		}
+		said, err := changeClusterSettings(ctx, c, "auth.ldap from the console", func(doc *settings.Doc) error {
+			return doc.SetYAML([]string{"auth", "ldap"}, block)
+		})
+		if err != nil {
+			return actionDoneMsg{title: "Active Directory", err: err}
+		}
+		return actionDoneMsg{title: "Active Directory", text: report + "\n\n" + said}
+	}
+}
+
+// tryDirectory tries the block in force.
+func (m *consoleModel) tryDirectory() tea.Cmd {
+	m.busy = "Trying the directory"
+	c, block := m.client, ldapYAML(m.signIn.ldap)
+	return func() tea.Msg {
+		ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)
+		defer cancel()
+		var tv adminapi.LDAPTestView
+		if err := c.do(ctx, http.MethodPost, "/api/v1/auth/ldap/test", adminapi.LDAPTestRequest{Settings: block}, &tv); err != nil {
+			return actionDoneMsg{title: "The directory", err: err}
+		}
+		if !tv.OK {
+			return actionDoneMsg{title: "The directory", err: errors.New("it did not pass\n\n" + checksText(tv.Checks))}
+		}
+		return actionDoneMsg{title: "The directory", text: checksText(tv.Checks)}
+	}
+}
+
+func checksText(checks []adminapi.CheckView) string {
+	var b strings.Builder
+	for i, c := range checks {
+		if i > 0 {
+			b.WriteString("\n")
+		}
+		mark := sOK.Render("✓")
+		if !c.OK {
+			mark = sBad.Render("✗")
+		}
+		fmt.Fprintf(&b, "%s %s: %s", mark, c.Step, c.Detail)
+	}
+	return b.String()
+}
+
+// entraForm sets up single sign-on through Entra ID, or another provider
+// when an issuer address is given in place of the tenant.
+func (m *consoleModel) entraForm() {
+	o := m.signIn.oidc
+	tenant := strings.TrimSuffix(strings.TrimPrefix(o.Issuer, "https://login.microsoftonline.com/"), "/v2.0")
+	secretHint := "the app registration's client secret"
+	if o.ClientSecretFile != "" {
+		secretHint = "empty keeps the one kept"
+	}
+	create, only := "yes", "no"
+	if o.Enabled() && !o.CreateAccounts {
+		create = "no"
+	}
+	if o.NoPasswords {
+		only = "yes"
+	}
+	secret := textF("client secret", "", secretHint)
+	secret.text.Mask = true
+	m.openForm(&formDialog{title: "Single sign-on: Entra ID",
+		note: "Register an app in Entra with the redirect address " + redirectURL(m.signIn.view.BaseURL) +
+			", and a groups claim of the groups assigned to it.",
+		fields: []formField{
+			textF("tenant", tenant, "the directory (tenant) ID; or another provider's issuer URL"),
+			textF("client ID", o.ClientID, "the application (client) ID"),
+			secret,
+			textF("administrators", strings.Join(o.Roles["admin"], ", "), "group object IDs, as Entra sends them"),
+			textF("operators", strings.Join(o.Roles["operator"], ", "), "group object IDs"),
+			textF("label", or(o.Label, "Microsoft"), "the button says: Sign in with LABEL"),
+			choiceF("make accounts", []string{"yes", "no"}, create),
+			choiceF("only this way", []string{"no", "yes"}, only),
+		},
+		submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
+			next, err := oidcFromForm(o, v)
+			if err != nil {
+				return nil, err
+			}
+			return m.saveOIDC(next, v[2]), nil
+		}})
+}
+
+func oidcFromForm(kept config.OIDC, v []string) (config.OIDC, error) {
+	o := kept
+	switch t := strings.TrimSpace(v[0]); {
+	case t == "":
+		return o, errors.New("tenant: the directory (tenant) ID, or an issuer URL")
+	case strings.Contains(t, "://"):
+		o.Issuer = t
+	default:
+		o.Issuer = "https://login.microsoftonline.com/" + t + "/v2.0"
+	}
+	if o.ClientID = v[1]; o.ClientID == "" {
+		return o, errors.New("client ID: the application (client) ID")
+	}
+	if v[2] != "" || o.ClientSecretFile == "" {
+		if v[2] == "" {
+			return o, errors.New("client secret: the app registration's secret")
+		}
+		o.ClientSecretFile = adminapi.AuthSecretRef("oidc-client-secret")
+	}
+	o.Roles = map[string][]string{}
+	for role, i := range map[string]int{"admin": 3, "operator": 4} {
+		for _, g := range strings.Split(v[i], ",") {
+			if g = strings.TrimSpace(g); g != "" {
+				o.Roles[role] = append(o.Roles[role], g)
+			}
+		}
+	}
+	if len(o.Roles) == 0 {
+		o.Roles = nil
+	}
+	o.Label, o.CreateAccounts, o.NoPasswords = v[5], v[6] == "yes", v[7] == "yes"
+	return o, o.Validate()
+}
+
+func oidcYAML(o config.OIDC) string {
+	m := map[string]any{"issuer": o.Issuer, "client_id": o.ClientID, "client_secret_file": o.ClientSecretFile,
+		"create_accounts": o.CreateAccounts}
+	if o.Label != "" {
+		m["label"] = o.Label
+	}
+	if o.NoPasswords {
+		m["no_passwords"] = true
+	}
+	if o.RedirectURL != "" {
+		m["redirect_url"] = o.RedirectURL
+	}
+	if len(o.Scopes) > 0 {
+		m["scopes"] = o.Scopes
+	}
+	if o.GroupsClaim != "" {
+		m["groups_claim"] = o.GroupsClaim
+	}
+	if len(o.Roles) > 0 {
+		m["roles"] = o.Roles
+	}
+	if len(o.Memberships) > 0 {
+		m["memberships"] = o.Memberships
+	}
+	b, _ := yaml.Marshal(m)
+	return string(b)
+}
+
+func (m *consoleModel) saveOIDC(o config.OIDC, secret string) tea.Cmd {
+	m.busy = "Single sign-on"
+	c, block, redirect := m.client, oidcYAML(o), redirectURL(m.signIn.view.BaseURL)
+	return func() tea.Msg {
+		ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
+		defer cancel()
+		if secret != "" {
+			if err := c.do(ctx, http.MethodPut, "/api/v1/auth/secrets/oidc-client-secret", adminapi.SecretRequest{Value: secret}, nil); err != nil {
+				return actionDoneMsg{title: "Single sign-on", err: err}
+			}
+		}
+		said, err := changeClusterSettings(ctx, c, "auth.oidc from the console", func(doc *settings.Doc) error {
+			return doc.SetYAML([]string{"auth", "oidc"}, block)
+		})
+		if err != nil {
+			return actionDoneMsg{title: "Single sign-on", err: err}
+		}
+		return actionDoneMsg{title: "Single sign-on", text: said + "\n\nIn the app registration the redirect address is " + redirect +
+			". For people in more than 200 groups, emit the groups assigned to the application."}
+	}
+}
+
+// scimToken makes a new provisioning token, keeps it, and shows it once.
+func (m *consoleModel) scimToken() tea.Cmd {
+	m.busy = "Provisioning"
+	c, base := m.client, m.signIn.view.BaseURL
+	return func() tea.Msg {
+		ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
+		defer cancel()
+		b := make([]byte, 32)
+		if _, err := rand.Read(b); err != nil {
+			return actionDoneMsg{title: "Provisioning", err: err}
+		}
+		token := hex.EncodeToString(b)
+		if err := c.do(ctx, http.MethodPut, "/api/v1/auth/secrets/scim-token", adminapi.SecretRequest{Value: token}, nil); err != nil {
+			return actionDoneMsg{title: "Provisioning", err: err}
+		}
+		said, err := changeClusterSettings(ctx, c, "auth.scim from the console", func(doc *settings.Doc) error {
+			return doc.SetYAML([]string{"auth", "scim", "token_file"}, adminapi.AuthSecretRef("scim-token"))
+		})
+		if err != nil {
+			return actionDoneMsg{title: "Provisioning", err: err}
+		}
+		url := "https://<this gateway>/scim/v2"
+		if base != "" {
+			url = strings.TrimRight(base, "/") + "/scim/v2"
+		}
+		return actionDoneMsg{title: "Provisioning", text: "Give the provider the tenant URL " + url + " and this token, shown only now:\n\n  " +
+			token + "\n\n" + said}
+	}
+}
+
+// signInOffConfirm turns one way in off, after asking.
+func (m *consoleModel) signInOffConfirm(key, title, text string) {
+	m.confirmD = &confirmDialog{title: "Turn off " + strings.ToLower(title), text: text,
+		yes: func(m *consoleModel) tea.Cmd {
+			m.busy = title
+			c := m.client
+			return func() tea.Msg {
+				ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
+				defer cancel()
+				said, err := changeClusterSettings(ctx, c, "auth."+key+" off from the console", func(doc *settings.Doc) error {
+					doc.Unset([]string{"auth", key})
+					return nil
+				})
+				return actionDoneMsg{title: title, text: said, err: err}
+			}
+		}}
+}
+
+// changeClusterSettings is changeSettings for the console: the newest
+// settings read, changed, and put in force, said in a sentence.
+func changeClusterSettings(ctx context.Context, c *ctlClient, note string, change func(doc *settings.Doc) error) (string, error) {
+	var cur adminapi.SettingsView
+	if err := c.get(ctx, "/api/v1/settings", &cur); err != nil {
+		return "", err
+	}
+	doc, err := settings.FromBytes("settings", []byte(cur.Body))
+	if err != nil {
+		return "", err
+	}
+	if err := change(doc); err != nil {
+		return "", err
+	}
+	body, err := doc.Bytes()
+	if err != nil {
+		return "", err
+	}
+	before, _ := config.Parse([]byte(cur.Body))
+	after, err := config.Parse(body)
+	if err != nil {
+		return "", err
+	}
+	changed := config.Changed(before, after)
+	if len(changed) == 0 {
+		return "Nothing changed.", nil
+	}
+	var v adminapi.SettingsView
+	if err := c.do(ctx, http.MethodPut, "/api/v1/settings",
+		adminapi.SettingsRequest{Base: cur.Version, Body: string(body), Note: note}, &v); err != nil {
+		return "", err
+	}
+	return fmt.Sprintf("Settings version %d. Every control plane starts again on it within seconds, standbys first; "+
+		"engines keep serving.", v.Version), nil
+}

internal/cli/tui_signin_test.go

@@ -0,0 +1,108 @@
+package cli
+
+import (
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+)
+
+// adForm is the directory form's values, in its order.
+func adForm(password, ca, allowed, teams string) []string {
+	return []string{"ldaps://dc1.corp.example, ldaps://dc2.corp.example", "OU=Staff,DC=corp,DC=example",
+		"CN=svc,OU=Service,DC=corp,DC=example", password, ca, allowed, "llmgw-admins", "", teams, "Acme", "yes"}
+}
+
+func TestTheDirectoryFormMakesTheBlockTheSettingsKeep(t *testing.T) {
+	pem := "-----BEGIN CERTIFICATE-----\nMIIB\n-----END CERTIFICATE-----\n"
+	caFile := filepath.Join(t.TempDir(), "ca.pem")
+	if err := os.WriteFile(caFile, []byte(pem), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	l, err := directoryFromForm(config.LDAP{}, adForm("pw", caFile, "CN=llmgw-users,OU=Groups,DC=corp,DC=example",
+		"ml-team=team:Acme/ml; sales=org:Sales"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	// The group, nested ones counting; the CA written into the settings;
+	// the password named in the store, not a file on one machine.
+	if l.UserFilter != "(memberOf:1.2.840.113556.1.4.1941:=CN=llmgw-users,OU=Groups,DC=corp,DC=example)" ||
+		l.CA != pem || l.BindPasswordFile != "store:auth/ldap-bind" || len(l.URLs) != 2 ||
+		l.Memberships["sales"] != "org:Sales" || l.Roles["admin"][0] != "llmgw-admins" || l.Roles["operator"] != nil {
+		t.Errorf("%+v", l)
+	}
+	// What the settings keep reads back the same.
+	back, err := config.ParseLDAP([]byte(ldapYAML(l)))
+	if err != nil || back.UserFilter != l.UserFilter || back.CA != pem || back.Memberships["ml-team"] != "team:Acme/ml" {
+		t.Errorf("read back %+v %v", back, err)
+	}
+
+	// Changed again with nothing typed for the password or the CA: both kept.
+	again, err := directoryFromForm(l, adForm("", "", "(objectClass=user)", ""))
+	if err != nil || again.CA != pem || again.BindPasswordFile != l.BindPasswordFile || again.UserFilter != "(objectClass=user)" ||
+		again.Memberships != nil {
+		t.Errorf("kept: %+v %v", again, err)
+	}
+	// A group by its bare name cannot be a filter, and a plaintext server is refused.
+	if _, err := directoryFromForm(config.LDAP{}, adForm("pw", "", "llmgw-users", "")); err == nil {
+		t.Error("a bare group name was taken")
+	}
+	plain := adForm("pw", "", "", "")
+	plain[0] = "ldap://dc1"
+	if _, err := directoryFromForm(config.LDAP{}, plain); err == nil || !strings.Contains(err.Error(), "plaintext") {
+		t.Errorf("plaintext: %v", err)
+	}
+}
+
+func TestTheEntraFormMakesTheIssuerFromTheTenant(t *testing.T) {
+	o, err := oidcFromForm(config.OIDC{}, []string{"0f2c-tenant", "app-id", "s3cret", "g-admins", "", "Microsoft", "yes", "no"})
+	if err != nil {
+		t.Fatal(err)
+	}
+	if o.Issuer != "https://login.microsoftonline.com/0f2c-tenant/v2.0" || o.ClientSecretFile != "store:auth/oidc-client-secret" ||
+		o.Roles["admin"][0] != "g-admins" || !o.CreateAccounts || o.NoPasswords {
+		t.Errorf("%+v", o)
+	}
+	// Another provider by its issuer; the secret kept when none is typed.
+	k, err := oidcFromForm(o, []string{"https://sso.example/realms/acme", "llmgw", "", "", "", "Acme SSO", "no", "yes"})
+	if err != nil || k.Issuer != "https://sso.example/realms/acme" || k.ClientSecretFile != o.ClientSecretFile || !k.NoPasswords {
+		t.Errorf("%+v %v", k, err)
+	}
+	if _, err := oidcFromForm(config.OIDC{}, []string{"t", "app", "", "", "", "", "yes", "no"}); err == nil {
+		t.Error("a first setup without a secret was taken")
+	}
+}
+
+func TestTheSignInPanelSaysWhatIsOnAndHowTheDirectoryLastWent(t *testing.T) {
+	m := consoleModel{menu: cSignIn, loaded: true, width: 120, height: 40}
+	m.signIn = signInState{loaded: true, view: adminapi.AuthView{
+		BaseURL: "https://llm.example",
+		OIDC:    &adminapi.OIDCView{Issuer: "https://login.microsoftonline.com/t/v2.0", Label: "Microsoft", CreateAccounts: true},
+		LDAP: &adminapi.LDAPView{URLs: []string{"ldaps://dc1"}, BaseDN: "OU=Staff,DC=corp",
+			LastSync: &adminapi.DirectorySyncView{At: time.Now().Add(-time.Minute), People: 40, Accounts: 12, Suspended: 1}},
+	}}
+	_, body, _ := m.signInPanel(110)
+	for _, want := range []string{"Single sign-on (Entra ID, OIDC)", "Microsoft", "https://llm.example/login/oidc/callback",
+		"Active Directory (LDAP)", "ldaps://dc1", "40 people there, 12 accounts kept in step, 1 suspended", "Provisioning (SCIM)"} {
+		if !strings.Contains(body, want) {
+			t.Errorf("the panel lacks %q:\n%s", want, body)
+		}
+	}
+	m.signIn.view.LDAP.Off = "auth.ldap.ca: holds no certificate"
+	if _, body, _ := m.signInPanel(110); !strings.Contains(body, "broken here") || !strings.Contains(body, "holds no certificate") {
+		t.Errorf("a directory off here is not said:\n%s", body)
+	}
+}
+
+func TestAuthBlocksAreReadFromTheSettingsDocument(t *testing.T) {
+	body := "auth:\n  ldap:\n    urls: [ldaps://dc1]\n    base_dn: DC=corp\n    bind_dn: CN=svc\n    bind_password_file: store:auth/ldap-bind\n" +
+		"    sync_interval: 1m\n  oidc:\n    issuer: https://sso.example\n    client_id: llmgw\n"
+	l, o := authBlocks(body)
+	if l.BaseDN != "DC=corp" || l.SyncInterval != time.Minute || o.ClientID != "llmgw" {
+		t.Errorf("%+v %+v", l, o)
+	}
+}

internal/config/config.go

@@ -442,6 +442,9 @@ type Auth struct {
 
 	// OIDC is sign-in through an identity provider.
 	OIDC OIDC `koanf:"oidc"`
+	// LDAP is sign-in against a directory, Active Directory above all, and
+	// the accounts kept in step with it.
+	LDAP LDAP `koanf:"ldap"`
 	// SCIM is account provisioning by the identity provider.
 	SCIM SCIM `koanf:"scim"`
 
@@ -968,6 +971,12 @@ func (s Server) Validate() error {
 	if err := s.Auth.OIDC.Validate(); err != nil {
 		return err
 	}
+	if err := s.Auth.LDAP.Validate(); err != nil {
+		return err
+	}
+	if s.Auth.LDAP.Enabled() && !s.Auth.Enabled {
+		return fmt.Errorf("auth.ldap needs auth.enabled: it signs people in to the accounts the store keeps")
+	}
 	if err := s.AgentTLS.Validate(); err != nil {
 		return err
 	}

internal/config/ldap.go

@@ -0,0 +1,160 @@
+package config
+
+import (
+	"errors"
+	"fmt"
+	"net/url"
+	"strings"
+	"time"
+
+	"github.com/knadh/koanf/parsers/yaml"
+	"github.com/knadh/koanf/v2"
+)
+
+// LDAP is sign-in against a directory — Active Directory above all — with
+// the directory's own passwords, and the accounts kept in step with it.
+//
+//	auth:
+//	  ldap:
+//	    urls: [ldaps://dc1.corp.example.com, ldaps://dc2.corp.example.com]
+//	    bind_dn: CN=llmgw,OU=Service Accounts,DC=corp,DC=example,DC=com
+//	    bind_password_file: /etc/llmgw/secrets/ldap-bind
+//	    base_dn: OU=Staff,DC=corp,DC=example,DC=com
+//	    create_accounts: true
+//	    roles: {admin: [llmgw-admins]}
+//	    memberships: {ml-team: "team:Acme/ml"}
+//
+// A person signs in on the portal's form with their sAMAccountName, their
+// userPrincipalName or their mail, and their directory password. The
+// gateway finds them with the bind account, checks the password by binding
+// as them, and reads their groups, nested ones included. Every sync_interval
+// the leader reads the directory again: an account disabled or gone there
+// is suspended here, its sessions ended and its API keys refused.
+type LDAP struct {
+	// URLs are the directory's servers, tried in turn: ldaps:// (636), or
+	// ldap:// (389) with StartTLS.
+	URLs []string `koanf:"urls"`
+	// StartTLS upgrades an ldap:// connection before anything is sent. A
+	// password never crosses a connection without TLS: ldap:// without it
+	// is refused.
+	StartTLS bool `koanf:"start_tls"`
+	// CAFile is the authority the servers' certificates are checked against,
+	// when it is not the system's: an AD Certificate Services root.
+	CAFile string `koanf:"ca_file"`
+	// CA is the same authority written out, PEM: kept in the cluster's
+	// settings, so no file has to be put on every control plane. Used in
+	// place of CAFile when both are set.
+	CA string `koanf:"ca"`
+	// BindDN and the file holding its password: a service account that may
+	// read users and groups. Nothing is written to the directory.
+	BindDN           string `koanf:"bind_dn"`
+	BindPasswordFile string `koanf:"bind_password_file"`
+	// BaseDN is where people are searched for, subtree included.
+	BaseDN string `koanf:"base_dn"`
+	// GroupBaseDN is where their groups are; empty is BaseDN's domain root.
+	GroupBaseDN string `koanf:"group_base_dn"`
+	// UserFilter narrows who may sign in, ANDed with the login match: a
+	// group (memberOf=...) or an OU's people only. Empty is every enabled
+	// user object.
+	UserFilter string `koanf:"user_filter"`
+	// Label is what the form says to sign in with: "Sign in with your
+	// <label> account". Empty is "Active Directory".
+	Label string `koanf:"label"`
+	// CreateAccounts makes an account, active, for a person the directory
+	// vouches for and the gateway does not know. Off, only existing accounts
+	// sign in this way.
+	CreateAccounts bool `koanf:"create_accounts"`
+	// Roles maps a staff role (viewer, operator, billing, admin) to the
+	// groups that give it, by name (CN) or whole DN. Set at every sign-in
+	// and sync: someone taken out of the group loses the role.
+	Roles map[string][]string `koanf:"roles"`
+	// Memberships maps a group, by CN or DN, to a place in the tree —
+	// org:NAME, team:ORG/TEAM, project:ORG/TEAM/PROJECT — joined at sign-in
+	// and kept by the sync, which also takes away a place whose group the
+	// person has left.
+	Memberships map[string]string `koanf:"memberships"`
+	// SyncInterval is how often the leader reads the directory. Zero is
+	// 15m; negative turns the sync off, leaving only sign-in.
+	SyncInterval time.Duration `koanf:"sync_interval"`
+	// Timeout bounds each round trip to a server. Zero is 10s.
+	Timeout time.Duration `koanf:"timeout"`
+}
+
+// Enabled reports whether directory sign-in is configured.
+func (l LDAP) Enabled() bool { return len(l.URLs) > 0 }
+
+// Validate says what in the configuration cannot work.
+func (l LDAP) Validate() error {
+	if !l.Enabled() {
+		return nil
+	}
+	for _, raw := range l.URLs {
+		u, err := url.Parse(raw)
+		if err != nil || u.Host == "" {
+			return fmt.Errorf("auth.ldap.urls: %q is not an ldap:// or ldaps:// address", raw)
+		}
+		switch u.Scheme {
+		case "ldaps":
+		case "ldap":
+			if !l.StartTLS {
+				return fmt.Errorf("auth.ldap.urls: %q is plaintext: use ldaps://, or set start_tls; passwords would cross it", raw)
+			}
+		default:
+			return fmt.Errorf("auth.ldap.urls: %q is not an ldap:// or ldaps:// address", raw)
+		}
+	}
+	switch {
+	case strings.TrimSpace(l.BindDN) == "" || l.BindPasswordFile == "":
+		return errors.New("auth.ldap: bind_dn and bind_password_file are needed, for the account that finds people")
+	case l.CA != "" && !strings.Contains(l.CA, "-----BEGIN CERTIFICATE-----"):
+		return errors.New("auth.ldap.ca: not a PEM certificate")
+	case strings.TrimSpace(l.BaseDN) == "":
+		return errors.New("auth.ldap: base_dn is needed, where people are searched for")
+	case l.UserFilter != "" && !(strings.HasPrefix(l.UserFilter, "(") && strings.HasSuffix(l.UserFilter, ")")):
+		return fmt.Errorf("auth.ldap.user_filter: %q is not an LDAP filter in parentheses", l.UserFilter)
+	}
+	for role := range l.Roles {
+		if !validStaffRole(role) {
+			return fmt.Errorf("auth.ldap.roles: %q is not viewer, operator, billing or admin", role)
+		}
+	}
+	for g, place := range l.Memberships {
+		if !validPlace(place) {
+			return fmt.Errorf("auth.ldap.memberships[%s]: %q is not org:NAME, team:ORG/TEAM or project:ORG/TEAM/PROJECT", g, place)
+		}
+	}
+	return nil
+}
+
+// Interval is SyncInterval with its default; zero means off.
+func (l LDAP) Interval() time.Duration {
+	switch {
+	case l.SyncInterval < 0:
+		return 0
+	case l.SyncInterval == 0:
+		return 15 * time.Minute
+	}
+	return l.SyncInterval
+}
+
+// RoundTrip is Timeout with its default.
+func (l LDAP) RoundTrip() time.Duration {
+	if l.Timeout <= 0 {
+		return 10 * time.Second
+	}
+	return l.Timeout
+}
+
+// ParseLDAP is an auth.ldap block, as YAML, read on its own as the cluster's
+// settings would read it, and checked.
+func ParseLDAP(block []byte) (LDAP, error) {
+	var l LDAP
+	k := koanf.New(".")
+	if err := k.Load(bytesProvider(block), yaml.Parser()); err != nil {
+		return l, fmt.Errorf("auth.ldap: %w", err)
+	}
+	if err := k.UnmarshalWithConf("", &l, koanf.UnmarshalConf{Tag: "koanf"}); err != nil {
+		return l, fmt.Errorf("auth.ldap: %w", err)
+	}
+	return l, l.Validate()
+}

internal/config/oidc.go

@@ -72,21 +72,34 @@ func (o OIDC) Validate() error {
 		return errors.New("auth.oidc.client_id is required")
 	}
 	for role := range o.Roles {
-		switch role {
-		case "viewer", "operator", "billing", "admin":
-		default:
+		if !validStaffRole(role) {
 			return fmt.Errorf("auth.oidc.roles: %q is not viewer, operator, billing or admin", role)
 		}
 	}
 	for g, where := range o.Memberships {
-		kind, path, ok := strings.Cut(where, ":")
-		if !ok || path == "" || (kind != "org" && kind != "team" && kind != "project") {
+		if !validPlace(where) {
 			return fmt.Errorf("auth.oidc.memberships.%s is %q: org:NAME, team:ORG/TEAM or project:ORG/TEAM/PROJECT", g, where)
 		}
 	}
 	return nil
 }
 
+// validStaffRole is a role groups may give: viewer, operator, billing, admin.
+func validStaffRole(role string) bool {
+	switch role {
+	case "viewer", "operator", "billing", "admin":
+		return true
+	}
+	return false
+}
+
+// validPlace is a place in the tree a group may give: org:NAME,
+// team:ORG/TEAM or project:ORG/TEAM/PROJECT.
+func validPlace(where string) bool {
+	kind, path, ok := strings.Cut(where, ":")
+	return ok && path != "" && (kind == "org" || kind == "team" || kind == "project")
+}
+
 // SCIM lets an identity provider provision accounts (/scim/v2): made,
 // suspended and reinstated as they are there, with groups mapped by
 // auth.oidc's roles and memberships.

internal/control/adminapi/api.go

@@ -76,6 +76,7 @@ type API struct {
 	// train is adapter training; nil without a store and agents.
 	train    *TrainAdmin
 	settings *Settings
+	auth     *AuthAdmin
 	// restartAgents tells every connected agent to start again.
 	restartAgents func(reason string) []string
 	// revoke records a removed node, so its certificate is refused.
@@ -129,6 +130,7 @@ func (a *API) Register(mux Mux) {
 	a.registerTrain(mux)
 	a.registerGuard(mux)
 	a.registerSettings(mux)
+	a.registerAuth(mux)
 	mux.HandleFunc("POST /api/v1/agents/restart", a.restartAgentsHandler)
 	mux.HandleFunc("GET /api/v1/audit", a.auditEvents)
 	mux.HandleFunc("GET /api/v1/usage/export", a.usageExport)

internal/control/adminapi/auth.go

@@ -0,0 +1,170 @@
+package adminapi
+
+import (
+	"context"
+	"encoding/json"
+	"log/slog"
+	"net/http"
+	"strings"
+	"time"
+)
+
+// AuthAdmin is what the sign-in endpoints need: how people sign in now, a
+// directory's settings tried before they are kept, and where the secrets
+// those settings name are kept for every control plane.
+type AuthAdmin struct {
+	Status func() AuthView
+	// TestLDAP tries an auth.ldap block, as YAML, binding with password
+	// when one is given rather than the one the block names.
+	TestLDAP func(ctx context.Context, settings, password string) ([]CheckView, error)
+	Secrets  interface {
+		SetClusterSecret(ctx context.Context, name, value string) error
+	}
+	Log *slog.Logger
+}
+
+// SetAuth enables the sign-in endpoints.
+func (a *API) SetAuth(x *AuthAdmin) { a.auth = x }
+
+// AuthView is how people sign in on this gateway.
+type AuthView struct {
+	SignupOpen bool `json:"signup_open"`
+	// OIDC is single sign-on through an identity provider: Entra ID,
+	// Keycloak, Okta.
+	OIDC *OIDCView `json:"oidc,omitempty"`
+	// LDAP is sign-in checked by a directory: Active Directory.
+	LDAP *LDAPView `json:"ldap,omitempty"`
+	// SCIM is whether a provider may keep accounts in step.
+	SCIM bool `json:"scim"`
+	// BaseURL is the portal's address, when the settings say it: what a
+	// provider's redirect address is made from.
+	BaseURL string `json:"base_url,omitempty"`
+}
+
+// OIDCView is auth.oidc as it is in force.
+type OIDCView struct {
+	Issuer         string `json:"issuer"`
+	ClientID       string `json:"client_id"`
+	Label          string `json:"label,omitempty"`
+	CreateAccounts bool   `json:"create_accounts"`
+	NoPasswords    bool   `json:"no_passwords"`
+}
+
+// LDAPView is auth.ldap as it is in force, and how its last sync went.
+type LDAPView struct {
+	URLs   []string `json:"urls"`
+	BaseDN string   `json:"base_dn"`
+	BindDN string   `json:"bind_dn"`
+	Label  string   `json:"label,omitempty"`
+	// Off is why this control plane could not use the settings, when it
+	// could not; local accounts still sign in.
+	Off      string             `json:"off,omitempty"`
+	LastSync *DirectorySyncView `json:"last_sync,omitempty"`
+}
+
+// DirectorySyncView is the last time the leader read the directory.
+type DirectorySyncView struct {
+	At           time.Time `json:"at"`
+	People       int       `json:"people"`
+	Accounts     int       `json:"accounts"`
+	Suspended    int       `json:"suspended"`
+	Reinstated   int       `json:"reinstated"`
+	RolesChanged int       `json:"roles_changed"`
+	Error        string    `json:"error,omitempty"`
+}
+
+// CheckView is one step of trying a directory.
+type CheckView struct {
+	Step   string `json:"step"`
+	OK     bool   `json:"ok"`
+	Detail string `json:"detail"`
+}
+
+// LDAPTestRequest is an auth.ldap block to try, as YAML, and the service
+// account's password when it is not kept yet.
+type LDAPTestRequest struct {
+	Settings string `json:"settings"`
+	Password string `json:"password,omitempty"`
+}
+
+// LDAPTestView is how trying the directory went.
+type LDAPTestView struct {
+	OK     bool        `json:"ok"`
+	Checks []CheckView `json:"checks"`
+}
+
+// SecretRequest is a secret for the store to keep.
+type SecretRequest struct {
+	Value string `json:"value"`
+}
+
+// authSecrets are the secrets an operator may give through the API, and
+// the reference the settings then name each by.
+var authSecrets = map[string]bool{"ldap-bind": true, "oidc-client-secret": true, "scim-token": true}
+
+// AuthSecretRef is the reference auth's settings name a kept secret by.
+func AuthSecretRef(name string) string { return "store:auth/" + name }
+
+func (a *API) registerAuth(mux Mux) {
+	need := func(w http.ResponseWriter) bool {
+		if a.auth == nil {
+			writeError(w, http.StatusNotImplemented, "this control plane has no accounts")
+			return false
+		}
+		return true
+	}
+	mux.HandleFunc("GET /api/v1/auth", func(w http.ResponseWriter, r *http.Request) {
+		if !need(w) {
+			return
+		}
+		writeJSON(w, http.StatusOK, a.auth.Status())
+	})
+	mux.HandleFunc("POST /api/v1/auth/ldap/test", func(w http.ResponseWriter, r *http.Request) {
+		if !need(w) {
+			return
+		}
+		var req LDAPTestRequest
+		if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
+			writeError(w, http.StatusBadRequest, err.Error())
+			return
+		}
+		checks, err := a.auth.TestLDAP(r.Context(), req.Settings, req.Password)
+		if err != nil {
+			writeError(w, http.StatusBadRequest, err.Error())
+			return
+		}
+		v := LDAPTestView{OK: len(checks) > 0, Checks: checks}
+		for _, c := range checks {
+			v.OK = v.OK && c.OK
+		}
+		writeJSON(w, http.StatusOK, v)
+	})
+	mux.HandleFunc("PUT /api/v1/auth/secrets/{name}", func(w http.ResponseWriter, r *http.Request) {
+		if !need(w) {
+			return
+		}
+		name := r.PathValue("name")
+		if !authSecrets[name] {
+			writeError(w, http.StatusNotFound, "no such secret: ldap-bind, oidc-client-secret or scim-token")
+			return
+		}
+		var req SecretRequest
+		if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
+			writeError(w, http.StatusBadRequest, err.Error())
+			return
+		}
+		if strings.TrimSpace(req.Value) == "" {
+			writeError(w, http.StatusBadRequest, "an empty secret")
+			return
+		}
+		if err := a.auth.Secrets.SetClusterSecret(r.Context(), "auth/"+name, req.Value); err != nil {
+			writeError(w, http.StatusInternalServerError, err.Error())
+			return
+		}
+		if a.auth.Log != nil {
+			// What changed and who changed it; never the value.
+			a.auth.Log.Info("a sign-in secret was set", "secret", name, "by", actorOf(r))
+		}
+		writeJSON(w, http.StatusOK, map[string]string{"ref": AuthSecretRef(name)})
+	})
+}

internal/control/adminapi/auth_test.go

@@ -0,0 +1,53 @@
+package adminapi
+
+import (
+	"context"
+	"strings"
+	"testing"
+)
+
+type memSecrets map[string]string
+
+func (m memSecrets) SetClusterSecret(_ context.Context, name, value string) error {
+	m[name] = value
+	return nil
+}
+
+func TestSignInIsShownTriedAndItsSecretsKept(t *testing.T) {
+	a, _ := setup(t, testModels())
+	kept := memSecrets{}
+	var tried string
+	a.SetAuth(&AuthAdmin{
+		Status: func() AuthView { return AuthView{LDAP: &LDAPView{URLs: []string{"ldaps://dc1"}}} },
+		TestLDAP: func(_ context.Context, settings, password string) ([]CheckView, error) {
+			tried = settings + "|" + password
+			return []CheckView{{Step: "ldaps://dc1", OK: true}, {Step: "people", OK: false, Detail: "no one"}}, nil
+		},
+		Secrets: kept,
+	})
+	var v AuthView
+	if code := call(t, a, "GET", "/api/v1/auth", "", &v); code != 200 || v.LDAP == nil {
+		t.Fatalf("status: %d %+v", code, v)
+	}
+	// One step failing fails the whole.
+	var tv LDAPTestView
+	if code := call(t, a, "POST", "/api/v1/auth/ldap/test", `{"settings":"urls: [ldaps://dc1]","password":"pw"}`, &tv); code != 200 ||
+		tv.OK || len(tv.Checks) != 2 || tried != "urls: [ldaps://dc1]|pw" {
+		t.Errorf("test: %d %+v %q", code, tv, tried)
+	}
+	var ref map[string]string
+	if code := call(t, a, "PUT", "/api/v1/auth/secrets/ldap-bind", `{"value":"s3cret"}`, &ref); code != 200 ||
+		kept["auth/ldap-bind"] != "s3cret" || ref["ref"] != "store:auth/ldap-bind" {
+		t.Errorf("secret: %d %v %v", code, kept, ref)
+	}
+	// Only the sign-in secrets, and never an empty one.
+	if code := call(t, a, "PUT", "/api/v1/auth/secrets/engine-key", `{"value":"x"}`, nil); code != 404 {
+		t.Errorf("another secret: %d", code)
+	}
+	if code := call(t, a, "PUT", "/api/v1/auth/secrets/ldap-bind", `{"value":"  "}`, nil); code != 400 {
+		t.Errorf("an empty secret: %d", code)
+	}
+	if strings.Contains(kept["auth/ldap-bind"], " ") {
+		t.Error("an empty secret replaced the kept one")
+	}
+}

internal/control/adminapi/routes.go

@@ -67,6 +67,7 @@ var PathParams = map[string]string{
 	"models/{name}":     "The model's name in the catalog.",
 	"presets/{name}":    "The preset's name, as `llmgw ctl model presets` lists it.",
 	"nodes/{id}":        "The node's ID, as `llmgw ctl nodes` lists it.",
+	"secrets/{name}":    "ldap-bind, oidc-client-secret or scim-token.",
 	"jobs/{id}":         "The job's ID, as starting it answered.",
 	"runs/{id}":         "The quality run's ID, as starting it answered.",
 	"requests/{id}":     "The request's ID: the x-request-id a client was answered with.",
@@ -396,6 +397,15 @@ var Routes = []Route{
 	{Method: "POST", Path: "/api/v1/settings/rollback", Summary: "Put an earlier version of the cluster's settings back in force",
 		Request: SettingsRollbackRequest{}, Response: SettingsView{}},
 
+	// How people sign in: single sign-on, a directory, and the secrets
+	// they need kept for every control plane.
+	{Method: "GET", Path: "/api/v1/auth", Summary: "How people sign in: OIDC, a directory, SCIM, and the directory's last sync",
+		Response: AuthView{}},
+	{Method: "POST", Path: "/api/v1/auth/ldap/test", Summary: "Try an auth.ldap block before it is kept: servers, the service account, people and groups",
+		Request: LDAPTestRequest{}, Response: LDAPTestView{}},
+	{Method: "PUT", Path: "/api/v1/auth/secrets/{name}", Summary: "Keep ldap-bind, oidc-client-secret or scim-token in the store, for settings to name as store:auth/NAME",
+		Request: SecretRequest{}, Response: map[string]string{}},
+
 	// Usage, requests and the record of changes.
 	{Method: "GET", Path: "/api/v1/usage", Summary: "What each group of the tenancy tree spent",
 		Query: []Param{

internal/control/catalog/catalog.go

@@ -1533,6 +1533,11 @@ type External struct {
 	// rather than the key itself, so the key is in neither the catalog, the
 	// store nor the audit log; empty for a provider that asks for none.
 	APIKeyFile string `json:"api_key_file,omitempty" koanf:"api_key_file"`
+	// APIKeyFiles are several keys for the provider, in place of
+	// APIKeyFile: one endpoint each (external:MODEL#1, #2…), so a key that
+	// answers 429 or 401 is passed over while its breaker is open and the
+	// request goes to another. A provider's limits are per key.
+	APIKeyFiles []ExternalKey `json:"api_key_files,omitempty" koanf:"api_key_files"`
 	// Peer names the federated llmgw cluster this model is offered by, for
 	// a model that federation added (`llmgw ctl federation add`) rather
 	// than one written by hand. Such a model is named PEER/MODEL, comes and
@@ -1551,6 +1556,14 @@ type External struct {
 	Output string `json:"output,omitempty" koanf:"output"`
 }
 
+// ExternalKey is one of a provider's keys.
+type ExternalKey struct {
+	// File holds the key, on the control plane, as APIKeyFile does.
+	File string `json:"file" koanf:"file"`
+	// Weight is its share of the requests against the others'; zero is 1.
+	Weight float64 `json:"weight,omitempty" koanf:"weight"`
+}
+
 // APIComfyUI is External.API for a ComfyUI server.
 const APIComfyUI = "comfyui"
 
@@ -1568,6 +1581,20 @@ func (e *External) validate(model string, profiles int) error {
 	if strings.TrimSpace(e.Model) == "" {
 		return fmt.Errorf("model %q: external model needs the name the provider knows it by", model)
 	}
+	if e.APIKeyFile != "" && len(e.APIKeyFiles) > 0 {
+		return fmt.Errorf("model %q: api_key_file or api_key_files, not both", model)
+	}
+	for i, k := range e.APIKeyFiles {
+		if strings.TrimSpace(k.File) == "" {
+			return fmt.Errorf("model %q: api_key_files[%d] names no file", model, i)
+		}
+		if k.Weight < 0 {
+			return fmt.Errorf("model %q: api_key_files[%d] has a negative weight", model, i)
+		}
+	}
+	if len(e.APIKeyFiles) > 0 && e.Peer != "" {
+		return fmt.Errorf("model %q: a federated model is reached with its peer's key, not api_key_files", model)
+	}
 	switch e.API {
 	case "", "openai":
 		if e.Workflow != "" || e.Output != "" {

internal/control/catalog/catalog_test.go

@@ -429,3 +429,23 @@ func TestAGuardModelSaysHowItIsReadAndAlwaysRuns(t *testing.T) {
 		t.Error("a judge is not critical by default")
 	}
 }
+
+func TestAProvidersKeysAreOneListOrOneFile(t *testing.T) {
+	ext := func(e External) Model {
+		e.URL, e.Model = "https://api.example.com", "m"
+		return Model{Name: "cloud", Source: "external", External: &e}
+	}
+	for _, bad := range []External{
+		{APIKeyFile: "/k", APIKeyFiles: []ExternalKey{{File: "/k2"}}},
+		{APIKeyFiles: []ExternalKey{{File: " "}}},
+		{APIKeyFiles: []ExternalKey{{File: "/k", Weight: -1}}},
+		{Peer: "p", APIKeyFiles: []ExternalKey{{File: "/k"}}},
+	} {
+		if _, err := New([]Model{ext(bad)}); err == nil {
+			t.Errorf("taken: %+v", bad)
+		}
+	}
+	if _, err := New([]Model{ext(External{APIKeyFiles: []ExternalKey{{File: "/a", Weight: 2}, {File: "/b"}}})}); err != nil {
+		t.Error(err)
+	}
+}

internal/control/control.go

@@ -438,15 +438,6 @@ func (c *Controller) externalEndpoints() []endpoint.Info {
 				continue
 			}
 		}
-		key := ""
-		if f := m.External.APIKeyFile; f != "" {
-			b, err := secrets.Read(context.Background(), f)
-			if err != nil {
-				c.log.Warn("an external model's key could not be read; not offered", "model", m.Name, "err", err)
-				continue
-			}
-			key = strings.TrimSpace(string(b))
-		}
 		// A peer's endpoint says so in its id, which is what usage records
 		// as the replica that answered.
 		id := "external:" + m.Name
@@ -462,14 +453,39 @@ func (c *Controller) externalEndpoints() []endpoint.Info {
 			}
 			engine, workflow = endpoint.EngineComfyUI, w
 		}
-		out = append(out, endpoint.Info{
+		base := endpoint.Info{
 			ID: id, URL: strings.TrimRight(m.External.URL, "/"),
-			Models: append([]string{m.Name}, m.Aliases...), APIKey: key,
+			Models: append([]string{m.Name}, m.Aliases...),
 			Engine: engine, Pool: "external", Weight: 1,
 			UpstreamModel: m.External.Model,
 			Peer:          m.External.Peer, PeerCluster: cluster, PeerTags: tags,
 			Workflow: workflow,
-		})
+		}
+		keys := m.External.APIKeyFiles
+		if len(keys) == 0 {
+			keys = []catalog.ExternalKey{{File: m.External.APIKeyFile}}
+		}
+		for i, k := range keys {
+			e := base
+			// One endpoint a key, numbered, so usage says which answered
+			// and a breaker stands for one key — never the key itself.
+			if len(m.External.APIKeyFiles) > 0 {
+				e.ID = fmt.Sprintf("%s#%d", id, i+1)
+				if k.Weight > 0 {
+					e.Weight = k.Weight
+				}
+			}
+			if k.File != "" {
+				b, err := secrets.Read(context.Background(), k.File)
+				if err != nil {
+					c.log.Warn("an external model's key could not be read; not offered", "model", m.Name,
+						"endpoint", e.ID, "err", err)
+					continue
+				}
+				e.APIKey = strings.TrimSpace(string(b))
+			}
+			out = append(out, e)
+		}
 	}
 	return out
 }

internal/control/external_test.go

@@ -73,3 +73,28 @@ func TestAnExternalModelIsPublishedWithItsKeyAndNothingIsPlacedForIt(t *testing.
 		}
 	}
 }
+
+// Several keys for one provider: an endpoint each, numbered, with its
+// weight; one whose file cannot be read is left out and the others stay.
+func TestEachOfAProvidersKeysIsAnEndpointOfItsOwn(t *testing.T) {
+	dir := t.TempDir()
+	a, c := filepath.Join(dir, "a.key"), filepath.Join(dir, "c.key")
+	_ = os.WriteFile(a, []byte("sk-a\n"), 0o600)
+	_ = os.WriteFile(c, []byte("sk-c"), 0o600)
+	cat, err := catalog.New([]catalog.Model{{Name: "cloud", Source: "external",
+		External: &catalog.External{URL: "https://api.example.com", Model: "gpt-4o-mini",
+			APIKeyFiles: []catalog.ExternalKey{{File: a, Weight: 3}, {File: filepath.Join(dir, "gone")}, {File: c}}}}})
+	if err != nil {
+		t.Fatal(err)
+	}
+	sink := &recordSink{}
+	New(DefaultConfig(), catalog.NewLive(cat), noDispatch{}, sink, slog.New(slog.DiscardHandler)).Publish(t.Context())
+	got := map[string]endpoint.Info{}
+	for _, e := range sink.infos {
+		got[e.ID] = e
+	}
+	if len(got) != 2 || got["external:cloud#1"].APIKey != "sk-a" || got["external:cloud#1"].Weight != 3 ||
+		got["external:cloud#3"].APIKey != "sk-c" || got["external:cloud#3"].Weight != 1 {
+		t.Errorf("published %+v", sink.infos)
+	}
+}

internal/control/modelserve/modelserve.go

@@ -504,7 +504,11 @@ func (s *Server) run(j *Job) {
 			req = fitted
 		}
 	}
+	// Under the lock: a job is read by Get while it runs, and the race
+	// detector caught this write beside it.
+	s.mu.Lock()
 	j.Request = req
+	s.mu.Unlock()
 	memGB := int((header.MemoryBytes(set.Bytes, req.CtxPool) + 1<<30 - 1) >> 30)
 	basis := fmt.Sprintf("weights %.1f GB + KV cache %.1f GB for %d tokens + engine",
 		gb(set.Bytes), gb(int64(req.CtxPool)*header.KVBytesPerToken()), req.CtxPool)

internal/control/modelsvc/modelsvc.go

@@ -21,6 +21,7 @@ import (
 	"errors"
 	"fmt"
 	"log/slog"
+	"slices"
 	"sync"
 	"time"
 
@@ -248,6 +249,19 @@ func (s *Service) Resources(ctx context.Context, name, profile string, memoryGB,
 	return final, err
 }
 
+// SetProfiles replaces a model's profiles, as a model added with them
+// would have: for a model whose engines are to run otherwise, its args among
+// it. Running replicas keep what they were started with until restarted.
+func (s *Service) SetProfiles(ctx context.Context, name string, profiles []catalog.Profile, by string) (catalog.Model, error) {
+	if len(profiles) == 0 {
+		return catalog.Model{}, errors.New("a model needs a profile")
+	}
+	return s.update(ctx, name, by, "replace its profiles", func(m *catalog.Model) error {
+		m.Profiles = slices.Clone(profiles)
+		return nil
+	})
+}
+
 // SetClass sets a model's capacity class: critical, standard or
 // opportunistic, or empty to go back to what its scaling implies.
 func (s *Service) SetClass(ctx context.Context, name, class, by string) (catalog.Model, error) {

internal/control/reconciler/reconciler.go

@@ -915,7 +915,20 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
 	// A drained replica is stopped once its replacement is serving, or once the
 	// grace period expires. Stopping it the moment it was drained would cut
 	// in-flight requests, which is the thing draining exists to avoid.
-	replaced := len(liveIDs) >= desired
+	//
+	// Serving counts a replica being restarted that still answers: with
+	// replicas restarted one after another, the second's retiring kept the
+	// first's drained one waiting out its grace — and with it the second's
+	// replacement, on a node with no room until the first had gone.
+	serving := len(liveIDs)
+	r.mu.Lock()
+	for _, id := range retiringIDs {
+		if rep, ok := r.replicas[id]; ok && rep.State == Ready {
+			serving++
+		}
+	}
+	r.mu.Unlock()
+	replaced := serving >= desired
 	for _, id := range drainingIDs {
 		r.mu.Lock()
 		rep, ok := r.replicas[id]

internal/control/reconciler/reconciler_test.go

@@ -2436,3 +2436,51 @@ func TestTheRoomAnInPlaceRestartFreesIsItsReplacements(t *testing.T) {
 		}
 	}
 }
+
+// Two models restarted together on a node whose cores are all reserved,
+// one to need more and one less: the judge fits only once the starter
+// has given back its cores. The judge's restart waits for that, rather
+// than being refused as if nothing would ever make room. Met on a 4-core
+// single-machine install after `ctl model resources` on both.
+func TestARestartWaitsForRoomAnotherRestartWillGiveBack(t *testing.T) {
+	cpu := inventory.Node{ID: "a", MgmtAddr: "10.0.0.1", State: inventory.Ready, MemoryTotalMB: 16 * 1024, CPUCount: 4}
+	judge := catalog.Model{Name: "judge", Source: "/m", Scaling: catalog.Scaling{Min: 2, Max: 2},
+		Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 2, CPUCores: 1}}}
+	starter := catalog.Model{Name: "starter", Source: "/m", Scaling: catalog.Scaling{Min: 1, Max: 1},
+		Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 2, CPUCores: 2}}}
+	h := newFleetHarness(t, []catalog.Model{judge, starter}, cpu)
+	h.settle(t, 3)
+	var old []string
+	for _, rep := range h.r.ReadyReplicas() {
+		old = append(old, rep.ID)
+	}
+	if len(old) != 3 {
+		t.Fatalf("%d replicas ready, want 3", len(old))
+	}
+
+	judge.Profiles[0].CPUCores, starter.Profiles[0].CPUCores = 2, 0
+	cat, err := catalog.New([]catalog.Model{judge, starter})
+	if err != nil {
+		t.Fatal(err)
+	}
+	h.r.cat.Store(cat)
+	if err := h.r.Restart(old...); err != nil {
+		t.Fatal(err)
+	}
+	for range 12 {
+		h.r.Reconcile(t.Context())
+		h.markReady()
+		h.advance(10 * time.Second)
+	}
+	for _, id := range old {
+		if st := h.r.RestartState(id); strings.HasPrefix(st, "refused") {
+			t.Errorf("%s: %s", id, st)
+		}
+		if rep, ok := replicaByID(h, id); ok && rep.State == Ready {
+			t.Errorf("%s still serves on its old figures", id)
+		}
+	}
+	if j, s := replicasOf(h, "judge"), replicasOf(h, "starter"); j != 2 || s != 1 {
+		t.Errorf("judge %d, starter %d after the restarts", j, s)
+	}
+}

internal/control/reconciler/restart.go

@@ -22,6 +22,10 @@ import (
 //     stopped now, and its replacement takes its place. Waiting would change
 //     nothing; it used to wait DrainGrace anyway, five minutes reading only
 //     "restart replacing", which looked like a restart that never would.
+//   - other replicas whose own restarts are under way: the replacement fits
+//     once they have restarted, as a starter edited to reserve fewer cores
+//     gives back what a judge edited to reserve more needs. Waited for, up
+//     to DrainGrace, so that two restarts each waiting on the other end.
 //   - more than it: the replacement fits nowhere even with it stopped — its
 //     profile was edited to need more than any node has free. Stopping it
 //     would leave the model with nothing, so the restart is given up, and
@@ -62,6 +66,8 @@ func (r *Reconciler) retire(ctx context.Context, m catalog.Model, ids []string,
 			r.log.Warn("restarting a replica in place: it is the only thing in its replacement's way",
 				"replica", id, "model", m.Name)
 			r.stopForRestart(ctx, id, "restarted in place: it was the only thing in its replacement's way")
+		case outcome == noRoom && r.now().Sub(since) <= r.cfg.DrainGrace && r.fitsAfterRestarts(m, id):
+			// Another restart under way gives the room back; waited for.
 		case outcome == noRoom || outcome == overLimit:
 			r.refuseRestart(ctx, id, "its replacement fits on no node even with this replica stopped "+
 				"(llmgw ctl explain-placement "+m.Name+")")
@@ -73,6 +79,28 @@ func (r *Reconciler) retire(ctx context.Context, m catalog.Model, ids []string,
 	}
 }
 
+// fitsAfterRestarts is whether a model's replacement for a replica fits with
+// that replica gone and every other on its way out: queued to restart, being
+// replaced, or draining. Their replacements take room too, so this is
+// whether waiting could help, not a promise that it will.
+func (r *Reconciler) fitsAfterRestarts(m catalog.Model, id string) bool {
+	gone := []victim{{ID: id, Model: m.Name}}
+	r.mu.Lock()
+	for oid, o := range r.replicas {
+		if oid == id {
+			continue
+		}
+		if o.State == Draining || !o.RetiringAt.IsZero() || slices.Contains(r.restartQueue[o.Model], oid) {
+			gone = append(gone, victim{ID: oid, Model: o.Model})
+		}
+	}
+	r.mu.Unlock()
+	if len(gone) == 1 {
+		return false
+	}
+	return r.fitsWithout(m, nil, gone)
+}
+
 // stopForRestart drains a replica and stops it at once, and places its
 // replacement in what it freed.
 //

internal/directory/directory.go

@@ -0,0 +1,395 @@
+// Package directory keeps accounts in step with an outside directory —
+// Active Directory over LDAP — at sign-in and by a periodic sync: made for
+// people the directory vouches for, given the staff role and the places in
+// the tree their groups map to, and suspended when the directory disables or
+// loses them.
+package directory
+
+import (
+	"context"
+	"crypto/rand"
+	"encoding/hex"
+	"errors"
+	"fmt"
+	"slices"
+	"strings"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// Identity is a person as the directory has them.
+type Identity struct {
+	// ID never changes for the person: an AD objectGUID.
+	ID    string
+	DN    string
+	Email string
+	Name  string
+	// Groups are the person's groups, nested ones included, each by its CN
+	// and by its DN, so a mapping may name either.
+	Groups []string
+	// Disabled is an account the directory has turned off.
+	Disabled bool
+}
+
+// Mapping is what a person's groups give them: a staff role and places in
+// the tree. The same shape as auth.oidc's.
+type Mapping struct {
+	Roles       map[string][]string
+	Memberships map[string]string
+}
+
+// Role is the highest staff role the groups give, "" when none.
+func (m Mapping) Role(groups []string) string {
+	for _, r := range []string{store.RoleAdmin, store.RoleOperator, store.RoleBilling, store.RoleViewer} {
+		if slices.ContainsFunc(m.Roles[r], func(g string) bool { return hasGroup(groups, g) }) {
+			return r
+		}
+	}
+	return ""
+}
+
+// Places are the places in the tree the groups give, each once.
+func (m Mapping) Places(groups []string) []string {
+	var out []string
+	for g, where := range m.Memberships {
+		if hasGroup(groups, g) && !slices.Contains(out, where) {
+			out = append(out, where)
+		}
+	}
+	slices.Sort(out)
+	return out
+}
+
+// Groups are every group the mapping names.
+func (m Mapping) Groups() []string {
+	var out []string
+	add := func(g string) {
+		if !slices.ContainsFunc(out, func(o string) bool { return strings.EqualFold(o, g) }) {
+			out = append(out, g)
+		}
+	}
+	for _, gs := range m.Roles {
+		for _, g := range gs {
+			add(g)
+		}
+	}
+	for g := range m.Memberships {
+		add(g)
+	}
+	slices.Sort(out)
+	return out
+}
+
+// hasGroup matches a group's CN or DN as AD does: case aside.
+func hasGroup(groups []string, g string) bool {
+	return slices.ContainsFunc(groups, func(have string) bool { return strings.EqualFold(have, g) })
+}
+
+// Store is what keeping accounts needs of the store.
+type Store interface {
+	UserByEmail(ctx context.Context, email string) (store.User, error)
+	UserByID(ctx context.Context, id string) (store.User, error)
+	CreateUser(ctx context.Context, u store.User) error
+	SetStaffRole(ctx context.Context, id, role string) error
+	SetUserState(ctx context.Context, id string, state store.UserState) error
+	DeleteUserSessions(ctx context.Context, userID string) error
+	AddMembership(ctx context.Context, user string, scope store.Scope, id, role, by string, at time.Time) error
+	RemoveMembership(ctx context.Context, userID string, scope store.Scope, scopeID string, at time.Time) (int, error)
+	MembershipsOf(ctx context.Context, userID string) ([]store.Membership, error)
+	AllOrgs(ctx context.Context) ([]store.Org, error)
+	AllTeams(ctx context.Context) ([]store.TeamPath, error)
+	AllProjects(ctx context.Context) ([]store.ProjectPath, error)
+	DirectoryUser(ctx context.Context, directory, externalID string) (store.DirectoryLink, error)
+	LinkDirectory(ctx context.Context, userID, directory, externalID string, at time.Time) error
+	DirectoryLinks(ctx context.Context, directory string) ([]store.DirectoryLink, error)
+	SetSyncSuspended(ctx context.Context, userID string, suspended bool, at time.Time) error
+}
+
+// Keeper keeps one directory's accounts.
+type Keeper struct {
+	St Store
+	// Name is the directory's, as links and memberships record it: "ldap".
+	Name           string
+	Mapping        Mapping
+	CreateAccounts bool
+	Now            func() time.Time
+}
+
+// The ways a person the directory vouches for is still refused.
+var (
+	ErrNoAccount = errors.New("no account here, and accounts are not made for the directory's people")
+	ErrSuspended = errors.New("the account is suspended here")
+	ErrNotActive = errors.New("the account is not active here")
+	ErrNoEmail   = errors.New("the directory has no mail for this person")
+	// ErrImplausible is a sync that would suspend too many at once: done
+	// nothing, and said.
+	ErrImplausible = errors.New("the directory's answer is implausible; nobody was suspended")
+)
+
+func (k *Keeper) now() time.Time {
+	if k.Now != nil {
+		return k.Now()
+	}
+	return time.Now()
+}
+
+// SignIn is the account for a person the directory has just vouched for:
+// found by their directory id, else their mail, else made; given what their
+// groups map to; refused when it may not sign in.
+func (k *Keeper) SignIn(ctx context.Context, id Identity) (store.User, error) {
+	u, link, err := k.account(ctx, id)
+	if err != nil {
+		return store.User{}, err
+	}
+	// The directory has them enabled, or the bind would have failed: an
+	// account the sync suspended for being disabled there is theirs again.
+	if u.State == store.UserSuspended && link.SuspendedBySync {
+		if u, err = k.reinstate(ctx, u); err != nil {
+			return store.User{}, err
+		}
+	}
+	switch {
+	case u.State == store.UserSuspended:
+		return store.User{}, ErrSuspended
+	case !u.CanSignIn():
+		return store.User{}, fmt.Errorf("%w: %s", ErrNotActive, u.State)
+	}
+	if err := k.apply(ctx, &u, id, false); err != nil {
+		return store.User{}, err
+	}
+	return u, nil
+}
+
+// account finds or makes the account and links it to the directory.
+func (k *Keeper) account(ctx context.Context, id Identity) (store.User, store.DirectoryLink, error) {
+	link, err := k.St.DirectoryUser(ctx, k.Name, id.ID)
+	switch {
+	case err == nil:
+		u, err := k.St.UserByID(ctx, link.UserID)
+		return u, link, err
+	case !errors.Is(err, store.ErrNotFound):
+		return store.User{}, link, err
+	}
+	email := store.NormalizeEmail(id.Email)
+	if email == "" {
+		return store.User{}, link, ErrNoEmail
+	}
+	u, err := k.St.UserByEmail(ctx, email)
+	switch {
+	case errors.Is(err, store.ErrNotFound) && k.CreateAccounts:
+		name := id.Name
+		if name == "" {
+			name = email
+		}
+		u = store.User{ID: "usr-" + randomID(), Email: email, Name: name,
+			// No password works here: the directory's is checked.
+			PasswordHash: "!" + k.Name, State: store.UserActive, CreatedAt: k.now()}
+		if err := k.St.CreateUser(ctx, u); err != nil {
+			return store.User{}, link, err
+		}
+	case errors.Is(err, store.ErrNotFound):
+		return store.User{}, link, fmt.Errorf("%w: %s", ErrNoAccount, email)
+	case err != nil:
+		return store.User{}, link, err
+	}
+	if err := k.St.LinkDirectory(ctx, u.ID, k.Name, id.ID, k.now()); err != nil {
+		return store.User{}, link, err
+	}
+	return u, store.DirectoryLink{UserID: u.ID, Directory: k.Name, ExternalID: id.ID}, nil
+}
+
+// apply gives the account the role and places its groups map to. With
+// remove, a place this directory gave that the groups no longer do is taken
+// away; one given otherwise is never touched.
+func (k *Keeper) apply(ctx context.Context, u *store.User, id Identity, remove bool) error {
+	if len(k.Mapping.Roles) > 0 {
+		if role := k.Mapping.Role(id.Groups); role != u.StaffRole() {
+			if err := k.St.SetStaffRole(ctx, u.ID, role); err != nil {
+				return err
+			}
+			u.IsAdmin, u.Role = role == store.RoleAdmin, role
+			if role == store.RoleAdmin {
+				u.Role = ""
+			}
+		}
+	}
+	want := map[string]store.Scope{}
+	for _, where := range k.Mapping.Places(id.Groups) {
+		scope, sid, err := k.place(ctx, where)
+		if err != nil {
+			continue // a mapping to a place the tree lacks: said by Sync
+		}
+		want[string(scope)+"/"+sid] = scope
+		if err := k.St.AddMembership(ctx, u.ID, scope, sid, store.RoleMember, k.Name, k.now()); err != nil {
+			return err
+		}
+	}
+	if !remove {
+		return nil
+	}
+	have, err := k.St.MembershipsOf(ctx, u.ID)
+	if err != nil {
+		return err
+	}
+	for _, m := range have {
+		if m.AddedBy != k.Name {
+			continue
+		}
+		if _, ok := want[string(m.Scope)+"/"+m.ScopeID]; ok {
+			continue
+		}
+		if _, err := k.St.RemoveMembership(ctx, u.ID, m.Scope, m.ScopeID, k.now()); err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+func (k *Keeper) reinstate(ctx context.Context, u store.User) (store.User, error) {
+	if err := k.St.SetUserState(ctx, u.ID, store.UserActive); err != nil {
+		return u, err
+	}
+	if err := k.St.SetSyncSuspended(ctx, u.ID, false, k.now()); err != nil {
+		return u, err
+	}
+	u.State = store.UserActive
+	return u, nil
+}
+
+// SyncReport is what a sync did.
+type SyncReport struct {
+	Seen, Updated, Suspended, Reinstated int
+	// Unmapped are mappings to places the tree does not have.
+	Unmapped []string
+}
+
+// Sync brings every account the directory keeps in step with what it says
+// now: people disabled or gone there are suspended here (their sessions
+// ended; their keys refuse, as a suspended owner's do), those enabled again
+// that the sync had suspended are reinstated, and everyone's role and places
+// follow their groups. People the directory has that never signed in here
+// are left until they do.
+func (k *Keeper) Sync(ctx context.Context, people []Identity) (SyncReport, error) {
+	var rep SyncReport
+	rep.Unmapped = k.unmapped(ctx)
+	byID := make(map[string]Identity, len(people))
+	for _, p := range people {
+		byID[p.ID] = p
+	}
+	links, err := k.St.DirectoryLinks(ctx, k.Name)
+	if err != nil {
+		return rep, err
+	}
+	// A directory that answers with nobody, or with half the people missing,
+	// is far likelier a wrong base, a filter or a bind account's lost rights
+	// than half the company leaving at once. Nothing is suspended for it.
+	gone := 0
+	for _, l := range links {
+		if p, there := byID[l.ExternalID]; !there || p.Disabled {
+			gone++
+		}
+	}
+	if len(links) > 0 && (len(people) == 0 || (gone >= 3 && gone*2 > len(links))) {
+		return rep, fmt.Errorf("%w: %d of the %d accounts it keeps would be suspended", ErrImplausible, gone, len(links))
+	}
+	for _, l := range links {
+		rep.Seen++
+		u, err := k.St.UserByID(ctx, l.UserID)
+		if err != nil {
+			continue
+		}
+		p, there := byID[l.ExternalID]
+		switch {
+		case !there || p.Disabled:
+			if u.State == store.UserActive {
+				if err := k.St.SetUserState(ctx, u.ID, store.UserSuspended); err != nil {
+					return rep, err
+				}
+				if err := k.St.DeleteUserSessions(ctx, u.ID); err != nil {
+					return rep, err
+				}
+				if err := k.St.SetSyncSuspended(ctx, u.ID, true, k.now()); err != nil {
+					return rep, err
+				}
+				rep.Suspended++
+			}
+		default:
+			if u.State == store.UserSuspended && l.SuspendedBySync {
+				if u, err = k.reinstate(ctx, u); err != nil {
+					return rep, err
+				}
+				rep.Reinstated++
+			}
+			if u.State != store.UserActive {
+				continue
+			}
+			before := u.StaffRole()
+			if err := k.apply(ctx, &u, p, true); err != nil {
+				return rep, err
+			}
+			if u.StaffRole() != before {
+				rep.Updated++
+			}
+		}
+	}
+	return rep, nil
+}
+
+// unmapped are the mapping's places the tree does not have.
+func (k *Keeper) unmapped(ctx context.Context) []string {
+	var out []string
+	for _, where := range k.Mapping.Memberships {
+		if _, _, err := k.place(ctx, where); err != nil && !slices.Contains(out, where) {
+			out = append(out, where)
+		}
+	}
+	slices.Sort(out)
+	return out
+}
+
+// place finds org:NAME, team:ORG/TEAM, project:ORG/TEAM/PROJECT, as
+// auth.oidc's mapping does.
+func (k *Keeper) place(ctx context.Context, where string) (store.Scope, string, error) {
+	kind, path, _ := strings.Cut(where, ":")
+	norm := func(s string) string {
+		parts := strings.Split(s, "/")
+		for i := range parts {
+			parts[i] = strings.TrimSpace(parts[i])
+		}
+		return strings.Join(parts, "/")
+	}
+	same := func(a string) bool { return strings.EqualFold(norm(a), norm(path)) }
+	switch kind {
+	case "org":
+		orgs, err := k.St.AllOrgs(ctx)
+		for _, o := range orgs {
+			if same(o.Name) {
+				return store.ScopeOrg, o.ID, nil
+			}
+		}
+		return "", "", errors.Join(err, errors.New("no such organization"))
+	case "team":
+		teams, err := k.St.AllTeams(ctx)
+		for _, t := range teams {
+			if same(t.OrgName + "/" + t.Name) {
+				return store.ScopeTeam, t.ID, nil
+			}
+		}
+		return "", "", errors.Join(err, errors.New("no such team"))
+	default:
+		projects, err := k.St.AllProjects(ctx)
+		for _, p := range projects {
+			if same(p.Path) {
+				return store.ScopeProject, p.ID, nil
+			}
+		}
+		return "", "", errors.Join(err, errors.New("no such project"))
+	}
+}
+
+func randomID() string {
+	var b [8]byte
+	_, _ = rand.Read(b[:])
+	return hex.EncodeToString(b[:])
+}

internal/directory/directory_test.go

@@ -0,0 +1,142 @@
+package directory
+
+import (
+	"errors"
+	"path/filepath"
+	"slices"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+func newKeeper(t *testing.T) (*Keeper, *store.DB, string) {
+	t.Helper()
+	st, err := store.Open(t.Context(), filepath.Join(t.TempDir(), "s.db"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	t.Cleanup(func() { st.Close() })
+	org, _ := st.CreateOrg(t.Context(), "Acme")
+	team, _ := st.CreateTeam(t.Context(), org, "ml")
+	k := &Keeper{St: st, Name: "ldap", CreateAccounts: true, Mapping: Mapping{
+		Roles:       map[string][]string{"admin": {"llmgw-admins"}, "viewer": {"CN=Staff,OU=Groups,DC=corp,DC=example"}},
+		Memberships: map[string]string{"ml-team": "team:Acme/ml", "ghosts": "team:Acme/nowhere"},
+	}}
+	return k, st, team
+}
+
+var ada = Identity{ID: "guid-ada", DN: "CN=Ada,OU=Staff,DC=corp,DC=example", Email: "Ada@Corp.example", Name: "Ada L",
+	Groups: []string{"ml-team", "CN=ml-team,OU=Groups,DC=corp,DC=example", "Staff", "CN=Staff,OU=Groups,DC=corp,DC=example"}}
+
+func TestAPersonTheDirectoryVouchesForIsMadeAnAccountWithWhatTheirGroupsGive(t *testing.T) {
+	k, st, team := newKeeper(t)
+	u, err := k.SignIn(t.Context(), ada)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if u.Email != "ada@corp.example" || u.Name != "Ada L" || u.PasswordHash != "!ldap" || u.StaffRole() != "viewer" {
+		t.Errorf("account %+v role %q", u, u.StaffRole())
+	}
+	ms, _ := st.MembershipsOf(t.Context(), u.ID)
+	if len(ms) != 1 || ms[0].ScopeID != team || ms[0].AddedBy != "ldap" {
+		t.Errorf("memberships %+v", ms)
+	}
+	// Their mail changes there: still the same account, by its directory id.
+	moved := ada
+	moved.Email = "ada.lovelace@corp.example"
+	again, err := k.SignIn(t.Context(), moved)
+	if err != nil || again.ID != u.ID {
+		t.Errorf("a new mail made a new account: %v %+v", err, again)
+	}
+	// The higher role of two wins.
+	boss := ada
+	boss.Groups = append(slices.Clone(ada.Groups), "llmgw-admins")
+	if u, _ := k.SignIn(t.Context(), boss); u.StaffRole() != "admin" {
+		t.Errorf("role %q", u.StaffRole())
+	}
+}
+
+func TestWithoutAccountsMadeOnlyThoseThatExistSignIn(t *testing.T) {
+	k, _, _ := newKeeper(t)
+	k.CreateAccounts = false
+	if _, err := k.SignIn(t.Context(), ada); !errors.Is(err, ErrNoAccount) {
+		t.Errorf("err %v", err)
+	}
+}
+
+func TestTheSyncSuspendsWhomTheDirectoryDisabledAndReinstatesThemAfter(t *testing.T) {
+	k, st, _ := newKeeper(t)
+	ctx := t.Context()
+	people := []Identity{ada}
+	for _, n := range []string{"b", "c", "d"} {
+		people = append(people, Identity{ID: "guid-" + n, Email: n + "@corp.example"})
+	}
+	ids := map[string]string{}
+	for _, p := range people {
+		u, err := k.SignIn(ctx, p)
+		if err != nil {
+			t.Fatal(err)
+		}
+		ids[p.ID] = u.ID
+	}
+	if err := st.CreateSession(ctx, ids["guid-ada"], "tok", "csrf", time.Hour); err != nil {
+		t.Fatal(err)
+	}
+	// An administrator suspends c here, by hand.
+	_ = st.SetUserState(ctx, ids["guid-c"], store.UserSuspended)
+
+	// Ada is disabled there, and has left ml-team.
+	off := ada
+	off.Disabled = true
+	rep, err := k.Sync(ctx, []Identity{off, people[1], people[2], people[3]})
+	if err != nil {
+		t.Fatal(err)
+	}
+	u, _ := st.UserByID(ctx, ids["guid-ada"])
+	if u.State != store.UserSuspended || rep.Suspended != 1 {
+		t.Errorf("ada %s, report %+v", u.State, rep)
+	}
+	if _, _, err := st.SessionByToken(ctx, "tok"); err == nil {
+		t.Error("a suspended account's session still works")
+	}
+	if !slices.Contains(rep.Unmapped, "team:Acme/nowhere") {
+		t.Errorf("an unmapped place was not said: %+v", rep)
+	}
+
+	// Enabled again, and out of ml-team: reinstated, and its place taken
+	// away. c, suspended by hand, stays so.
+	back := ada
+	back.Groups = []string{"Staff", "CN=Staff,OU=Groups,DC=corp,DC=example"}
+	rep, err = k.Sync(ctx, []Identity{back, people[1], people[2], people[3]})
+	if err != nil {
+		t.Fatal(err)
+	}
+	u, _ = st.UserByID(ctx, ids["guid-ada"])
+	ms, _ := st.MembershipsOf(ctx, u.ID)
+	if u.State != store.UserActive || rep.Reinstated != 1 || len(ms) != 0 {
+		t.Errorf("ada %s, memberships %+v, report %+v", u.State, ms, rep)
+	}
+	if c, _ := st.UserByID(ctx, ids["guid-c"]); c.State != store.UserSuspended {
+		t.Errorf("an account suspended by hand was reinstated: %s", c.State)
+	}
+}
+
+func TestASyncThatWouldSuspendMostIsRefused(t *testing.T) {
+	k, st, _ := newKeeper(t)
+	ctx := t.Context()
+	for _, n := range []string{"a", "b", "c", "d"} {
+		if _, err := k.SignIn(ctx, Identity{ID: "guid-" + n, Email: n + "@corp.example"}); err != nil {
+			t.Fatal(err)
+		}
+	}
+	for _, people := range [][]Identity{nil, {{ID: "guid-a", Email: "a@corp.example"}}} {
+		if _, err := k.Sync(ctx, people); !errors.Is(err, ErrImplausible) {
+			t.Errorf("%d people: %v", len(people), err)
+		}
+	}
+	users, _ := st.Users(ctx, store.UserSuspended)
+	if len(users) != 0 {
+		t.Errorf("suspended %d", len(users))
+	}
+}

internal/directory/ldap.go

@@ -0,0 +1,413 @@
+package directory
+
+import (
+	"context"
+	"crypto/tls"
+	"crypto/x509"
+	"encoding/binary"
+	"encoding/pem"
+	"errors"
+	"fmt"
+	"os"
+	"strconv"
+	"strings"
+
+	"github.com/go-ldap/ldap/v3"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/secrets"
+)
+
+// LDAP is a directory reached over LDAP, written for Active Directory: its
+// attribute names, its disabled flag, and its matching rule for nested
+// groups (LDAP_MATCHING_RULE_IN_CHAIN).
+type LDAP struct {
+	cfg  config.LDAP
+	tls  *tls.Config
+	dial func(url string) (ldapConn, error)
+	// password, when set, is the service account's in place of the one
+	// bind_password_file names: settings tried before they are kept.
+	password string
+}
+
+// WithPassword has the service account bind with pw rather than read it.
+func (d *LDAP) WithPassword(pw string) *LDAP {
+	d.password = pw
+	return d
+}
+
+func (d *LDAP) servicePassword(ctx context.Context) (string, error) {
+	if d.password != "" {
+		return d.password, nil
+	}
+	pw, err := secrets.Read(ctx, d.cfg.BindPasswordFile)
+	if err != nil {
+		return "", fmt.Errorf("auth.ldap.bind_password_file: %w", err)
+	}
+	return strings.TrimSpace(string(pw)), nil
+}
+
+// ldapConn is the part of a connection used here, so tests need no server.
+type ldapConn interface {
+	Bind(user, password string) error
+	Search(*ldap.SearchRequest) (*ldap.SearchResult, error)
+	SearchWithPaging(*ldap.SearchRequest, uint32) (*ldap.SearchResult, error)
+	Close() error
+}
+
+// inChain is AD's matching rule that follows nested group membership.
+const inChain = "1.2.840.113556.1.4.1941"
+
+// uacDisabled is userAccountControl's ACCOUNTDISABLE bit.
+const uacDisabled = 0x2
+
+var personAttrs = []string{"objectGUID", "sAMAccountName", "userPrincipalName", "mail", "displayName", "userAccountControl"}
+
+// ErrBadCredentials is a person the directory does not know, or a password
+// it does not take: said alike, so a form cannot tell which.
+var ErrBadCredentials = errors.New("the directory does not take that name and password")
+
+// NewLDAP prepares a directory from its configuration.
+func NewLDAP(cfg config.LDAP) (*LDAP, error) {
+	tc := &tls.Config{MinVersion: tls.VersionTLS12}
+	ca, where := []byte(cfg.CA), "auth.ldap.ca"
+	if cfg.CA == "" && cfg.CAFile != "" {
+		var err error
+		if ca, err = os.ReadFile(cfg.CAFile); err != nil {
+			return nil, fmt.Errorf("auth.ldap.ca_file: %w", err)
+		}
+		where = "auth.ldap.ca_file " + cfg.CAFile
+	}
+	if len(ca) > 0 {
+		pool, err := certPool(ca)
+		if err != nil {
+			return nil, fmt.Errorf("%s: %w", where, err)
+		}
+		tc.RootCAs = pool
+	}
+	d := &LDAP{cfg: cfg, tls: tc}
+	d.dial = d.dialURL
+	return d, nil
+}
+
+// certPool is the certificates in pem, each of which must parse: a
+// certificate Go refuses (a negative serial, as Samba's own has) is said,
+// not skipped into an empty pool.
+func certPool(data []byte) (*x509.CertPool, error) {
+	pool := x509.NewCertPool()
+	n := 0
+	for rest := data; ; {
+		var b *pem.Block
+		if b, rest = pem.Decode(rest); b == nil {
+			break
+		}
+		if b.Type != "CERTIFICATE" {
+			continue
+		}
+		c, err := x509.ParseCertificate(b.Bytes)
+		if err != nil {
+			return nil, err
+		}
+		pool.AddCert(c)
+		n++
+	}
+	if n == 0 {
+		return nil, errors.New("holds no certificate")
+	}
+	return pool, nil
+}
+
+func (d *LDAP) dialURL(u string) (ldapConn, error) {
+	c, err := ldap.DialURL(u, ldap.DialWithTLSConfig(d.tls))
+	if err != nil {
+		return nil, err
+	}
+	c.SetTimeout(d.cfg.RoundTrip())
+	if strings.HasPrefix(u, "ldap://") {
+		// Validate refuses ldap:// without start_tls: nothing is sent first.
+		if err := c.StartTLS(d.tls); err != nil {
+			c.Close()
+			return nil, fmt.Errorf("StartTLS: %w", err)
+		}
+	}
+	return c, nil
+}
+
+// service is a connection bound as the account that finds people, to the
+// first server that answers.
+func (d *LDAP) service(ctx context.Context) (ldapConn, error) {
+	pw, err := d.servicePassword(ctx)
+	if err != nil {
+		return nil, err
+	}
+	var errs []error
+	for _, u := range d.cfg.URLs {
+		c, err := d.dial(u)
+		if err != nil {
+			errs = append(errs, fmt.Errorf("%s: %w", u, err))
+			continue
+		}
+		if err := c.Bind(d.cfg.BindDN, pw); err != nil {
+			c.Close()
+			// A server that refuses the service account refuses it
+			// everywhere: not tried on the next.
+			return nil, fmt.Errorf("binding as %s on %s: %w", d.cfg.BindDN, u, err)
+		}
+		return c, nil
+	}
+	return nil, fmt.Errorf("no directory server answered: %w", errors.Join(errs...))
+}
+
+// Authenticate checks a person's name and password against the directory
+// and says who they are there, groups and all.
+func (d *LDAP) Authenticate(ctx context.Context, login, password string) (Identity, error) {
+	login = strings.TrimSpace(login)
+	// An empty password is an unauthenticated bind, which succeeds: never
+	// one for a person.
+	if login == "" || password == "" {
+		return Identity{}, ErrBadCredentials
+	}
+	// DOMAIN\user, as Windows writes it.
+	if i := strings.LastIndexByte(login, '\\'); i >= 0 {
+		login = login[i+1:]
+	}
+	c, err := d.service(ctx)
+	if err != nil {
+		return Identity{}, err
+	}
+	defer c.Close()
+	q := ldap.EscapeFilter(login)
+	filter := "(&" + d.people(false) + "(|(sAMAccountName=" + q + ")(userPrincipalName=" + q + ")(mail=" + q + ")))"
+	res, err := c.Search(ldap.NewSearchRequest(d.cfg.BaseDN, ldap.ScopeWholeSubtree, ldap.NeverDerefAliases,
+		2, 0, false, filter, personAttrs, nil))
+	if err != nil {
+		return Identity{}, fmt.Errorf("finding %s: %w", login, err)
+	}
+	if len(res.Entries) != 1 {
+		// None, or two people one name matches: neither is let in.
+		return Identity{}, ErrBadCredentials
+	}
+	id := identity(res.Entries[0])
+	if err := c.Bind(id.DN, password); err != nil {
+		var le *ldap.Error
+		if errors.As(err, &le) && le.ResultCode == ldap.LDAPResultInvalidCredentials {
+			return Identity{}, ErrBadCredentials
+		}
+		return Identity{}, fmt.Errorf("checking %s's password: %w", login, err)
+	}
+	// Back to the service account to read groups, which a person may not.
+	pw, err := d.servicePassword(ctx)
+	if err != nil {
+		return Identity{}, err
+	}
+	if err := c.Bind(d.cfg.BindDN, pw); err != nil {
+		return Identity{}, err
+	}
+	if id.Groups, err = d.groupsOf(c, id.DN); err != nil {
+		return Identity{}, err
+	}
+	return id, nil
+}
+
+// groupsOf are a person's groups, nested ones included, by CN and DN.
+func (d *LDAP) groupsOf(c ldapConn, dn string) ([]string, error) {
+	filter := "(&(objectClass=group)(member:" + inChain + ":=" + ldap.EscapeFilter(dn) + "))"
+	res, err := c.SearchWithPaging(ldap.NewSearchRequest(d.groupBase(), ldap.ScopeWholeSubtree, ldap.NeverDerefAliases,
+		0, 0, false, filter, []string{"cn"}, nil), 500)
+	if err != nil {
+		return nil, fmt.Errorf("reading the groups of %s: %w", dn, err)
+	}
+	var out []string
+	for _, e := range res.Entries {
+		out = append(out, e.GetAttributeValue("cn"), e.DN)
+	}
+	return out, nil
+}
+
+// People are everyone the directory has under the base, enabled or not,
+// each with the mapped groups they are in: what a sync compares against.
+func (d *LDAP) People(ctx context.Context, m Mapping) ([]Identity, error) {
+	c, err := d.service(ctx)
+	if err != nil {
+		return nil, err
+	}
+	defer c.Close()
+	res, err := c.SearchWithPaging(ldap.NewSearchRequest(d.cfg.BaseDN, ldap.ScopeWholeSubtree, ldap.NeverDerefAliases,
+		0, 0, false, d.people(true), personAttrs, nil), 500)
+	if err != nil {
+		return nil, fmt.Errorf("listing people: %w", err)
+	}
+	people := make([]Identity, 0, len(res.Entries))
+	at := map[string]int{}
+	for _, e := range res.Entries {
+		id := identity(e)
+		at[id.ID] = len(people)
+		people = append(people, id)
+	}
+	// Only the groups the mapping names matter: a query each, for its
+	// members nested ones included, rather than one per person.
+	for _, g := range m.Groups() {
+		dns, err := d.groupDNs(c, g)
+		if err != nil {
+			return nil, err
+		}
+		for _, gdn := range dns {
+			filter := "(&" + d.people(true) + "(memberOf:" + inChain + ":=" + ldap.EscapeFilter(gdn) + "))"
+			r, err := c.SearchWithPaging(ldap.NewSearchRequest(d.cfg.BaseDN, ldap.ScopeWholeSubtree,
+				ldap.NeverDerefAliases, 0, 0, false, filter, []string{"objectGUID"}, nil), 500)
+			if err != nil {
+				return nil, fmt.Errorf("reading the members of %s: %w", gdn, err)
+			}
+			for _, e := range r.Entries {
+				if i, ok := at[guid(e.GetRawAttributeValue("objectGUID"))]; ok {
+					people[i].Groups = append(people[i].Groups, g, gdn)
+				}
+			}
+		}
+	}
+	return people, nil
+}
+
+// Check is one step of trying a directory.
+type Check struct {
+	Step   string `json:"step"`
+	OK     bool   `json:"ok"`
+	Detail string `json:"detail"`
+}
+
+// Check tries the directory as configured, before its settings are kept:
+// each server reached and the service account bound on it, how many people
+// may sign in, and how many are in each group the mapping names.
+func (d *LDAP) Check(ctx context.Context, m Mapping) []Check {
+	pw, err := d.servicePassword(ctx)
+	if err != nil {
+		return []Check{{Step: "password", Detail: err.Error()}}
+	}
+	var out []Check
+	var c ldapConn
+	for _, u := range d.cfg.URLs {
+		conn, err := d.dial(u)
+		if err == nil {
+			if err = conn.Bind(d.cfg.BindDN, pw); err != nil {
+				conn.Close()
+			}
+		}
+		if err != nil {
+			out = append(out, Check{Step: u, Detail: err.Error()})
+			continue
+		}
+		out = append(out, Check{Step: u, OK: true, Detail: "reached, bound as " + d.cfg.BindDN})
+		if c == nil {
+			c = conn
+		} else {
+			conn.Close()
+		}
+	}
+	if c == nil {
+		return out
+	}
+	defer c.Close()
+	res, err := c.SearchWithPaging(ldap.NewSearchRequest(d.cfg.BaseDN, ldap.ScopeWholeSubtree, ldap.NeverDerefAliases,
+		0, 0, false, d.people(false), []string{"objectGUID"}, nil), 500)
+	switch {
+	case err != nil:
+		out = append(out, Check{Step: "people", Detail: err.Error()})
+	case len(res.Entries) == 0:
+		out = append(out, Check{Step: "people", Detail: "no enabled user under base_dn matches user_filter: no one could sign in"})
+	default:
+		out = append(out, Check{Step: "people", OK: true,
+			Detail: fmt.Sprintf("%d enabled people under base_dn may sign in", len(res.Entries))})
+	}
+	for _, g := range m.Groups() {
+		dns, err := d.groupDNs(c, g)
+		if err != nil || len(dns) == 0 {
+			detail := "no group of that name under " + d.groupBase()
+			if err != nil {
+				detail = err.Error()
+			}
+			out = append(out, Check{Step: "group " + g, Detail: detail})
+			continue
+		}
+		n := 0
+		for _, gdn := range dns {
+			filter := "(&" + d.people(false) + "(memberOf:" + inChain + ":=" + ldap.EscapeFilter(gdn) + "))"
+			r, err := c.SearchWithPaging(ldap.NewSearchRequest(d.cfg.BaseDN, ldap.ScopeWholeSubtree,
+				ldap.NeverDerefAliases, 0, 0, false, filter, []string{"objectGUID"}, nil), 500)
+			if err == nil {
+				n += len(r.Entries)
+			}
+		}
+		out = append(out, Check{Step: "group " + g, OK: true,
+			Detail: fmt.Sprintf("%d of the people who may sign in are in it, nested groups included", n)})
+	}
+	return out
+}
+
+// groupDNs are the groups a mapping's name means: a DN as it is, a CN
+// looked up.
+func (d *LDAP) groupDNs(c ldapConn, g string) ([]string, error) {
+	if strings.Contains(g, "=") {
+		return []string{g}, nil
+	}
+	res, err := c.Search(ldap.NewSearchRequest(d.groupBase(), ldap.ScopeWholeSubtree, ldap.NeverDerefAliases,
+		0, 0, false, "(&(objectClass=group)(cn="+ldap.EscapeFilter(g)+"))", []string{"cn"}, nil))
+	if err != nil {
+		return nil, fmt.Errorf("finding the group %s: %w", g, err)
+	}
+	var out []string
+	for _, e := range res.Entries {
+		out = append(out, e.DN)
+	}
+	return out, nil
+}
+
+// people is the filter for person accounts under the user filter; enabled
+// ones only, unless all.
+func (d *LDAP) people(all bool) string {
+	f := "(objectCategory=person)(objectClass=user)"
+	if !all {
+		f += "(!(userAccountControl:1.2.840.113556.1.4.803:=" + strconv.Itoa(uacDisabled) + "))"
+	}
+	return "(&" + f + d.cfg.UserFilter + ")"
+}
+
+// groupBase is where groups are searched: GroupBaseDN, else the domain root
+// of BaseDN (its DC= parts).
+func (d *LDAP) groupBase() string {
+	if d.cfg.GroupBaseDN != "" {
+		return d.cfg.GroupBaseDN
+	}
+	var dc []string
+	for _, rdn := range strings.Split(d.cfg.BaseDN, ",") {
+		if strings.HasPrefix(strings.ToUpper(strings.TrimSpace(rdn)), "DC=") {
+			dc = append(dc, strings.TrimSpace(rdn))
+		}
+	}
+	if len(dc) == 0 {
+		return d.cfg.BaseDN
+	}
+	return strings.Join(dc, ",")
+}
+
+func identity(e *ldap.Entry) Identity {
+	email := e.GetAttributeValue("mail")
+	if email == "" {
+		email = e.GetAttributeValue("userPrincipalName")
+	}
+	uac, _ := strconv.Atoi(e.GetAttributeValue("userAccountControl"))
+	return Identity{
+		ID: guid(e.GetRawAttributeValue("objectGUID")), DN: e.DN,
+		Email: email, Name: e.GetAttributeValue("displayName"),
+		Disabled: uac&uacDisabled != 0,
+	}
+}
+
+// guid is an objectGUID as Windows writes it: its first three parts
+// little-endian. Another directory's id is kept as hex.
+func guid(b []byte) string {
+	if len(b) != 16 {
+		return fmt.Sprintf("%x", b)
+	}
+	return fmt.Sprintf("%08x-%04x-%04x-%x-%x", binary.LittleEndian.Uint32(b[0:4]),
+		binary.LittleEndian.Uint16(b[4:6]), binary.LittleEndian.Uint16(b[6:8]), b[8:10], b[10:16])
+}

internal/directory/ldap_test.go

@@ -0,0 +1,162 @@
+package directory
+
+import (
+	"errors"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+
+	"github.com/go-ldap/ldap/v3"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+)
+
+// fakeConn is a directory with one person in it.
+type fakeConn struct {
+	binds    []string
+	filters  []string
+	password string // the person's
+	people   int    // how many the person search finds
+	refused  bool   // the service account's password is wrong
+}
+
+func (f *fakeConn) Bind(user, password string) error {
+	f.binds = append(f.binds, user)
+	if f.refused && strings.HasPrefix(user, "CN=llmgw") {
+		return ldap.NewError(ldap.LDAPResultInvalidCredentials, errors.New("invalid credentials"))
+	}
+	if strings.HasPrefix(user, "CN=Ada") && password != f.password {
+		return ldap.NewError(ldap.LDAPResultInvalidCredentials, errors.New("invalid credentials"))
+	}
+	return nil
+}
+
+func (f *fakeConn) Search(r *ldap.SearchRequest) (*ldap.SearchResult, error) {
+	f.filters = append(f.filters, r.Filter)
+	var out []*ldap.Entry
+	for i := range f.people {
+		e := ldap.NewEntry("CN=Ada,OU=Staff,DC=corp,DC=example", map[string][]string{
+			"mail": {"ada@corp.example"}, "displayName": {"Ada"}, "userAccountControl": {"512"},
+		})
+		e.Attributes = append(e.Attributes, &ldap.EntryAttribute{Name: "objectGUID", ByteValues: [][]byte{guidBytes(byte(i))}})
+		out = append(out, e)
+	}
+	return &ldap.SearchResult{Entries: out}, nil
+}
+
+func (f *fakeConn) SearchWithPaging(r *ldap.SearchRequest, _ uint32) (*ldap.SearchResult, error) {
+	f.filters = append(f.filters, r.Filter)
+	return &ldap.SearchResult{Entries: []*ldap.Entry{
+		ldap.NewEntry("CN=ml-team,OU=Groups,DC=corp,DC=example", map[string][]string{"cn": {"ml-team"}}),
+	}}, nil
+}
+
+func (f *fakeConn) Close() error { return nil }
+
+func guidBytes(last byte) []byte {
+	return []byte{0x78, 0x56, 0x34, 0x12, 0x34, 0x12, 0x78, 0x56, 0x9a, 0xbc, 0xde, 0xf0, 0x12, 0x34, 0x56, last}
+}
+
+func fakeLDAP(t *testing.T, c *fakeConn) *LDAP {
+	t.Helper()
+	pw := filepath.Join(t.TempDir(), "bind")
+	_ = os.WriteFile(pw, []byte("svc-secret\n"), 0o600)
+	d, err := NewLDAP(config.LDAP{URLs: []string{"ldaps://dc1"}, BindDN: "CN=llmgw,OU=Svc,DC=corp,DC=example",
+		BindPasswordFile: pw, BaseDN: "OU=Staff,DC=corp,DC=example", UserFilter: "(memberOf=CN=llm,DC=corp,DC=example)"})
+	if err != nil {
+		t.Fatal(err)
+	}
+	d.dial = func(string) (ldapConn, error) { return c, nil }
+	return d
+}
+
+func TestAPersonIsCheckedByBindingAsThemAndTheirGroupsReadNested(t *testing.T) {
+	c := &fakeConn{password: "ad-secret", people: 1}
+	d := fakeLDAP(t, c)
+	id, err := d.Authenticate(t.Context(), `CORP\ada*)(x=`, "ad-secret")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if id.Email != "ada@corp.example" || id.ID != "12345678-1234-5678-9abc-def012345600" ||
+		!slicesHas(id.Groups, "ml-team") || !slicesHas(id.Groups, "CN=ml-team,OU=Groups,DC=corp,DC=example") {
+		t.Errorf("identity %+v", id)
+	}
+	// The name is escaped into the filter, the domain part dropped; the
+	// user filter is ANDed in; disabled accounts are not found; groups are
+	// read with the in-chain rule.
+	find, groups := c.filters[0], c.filters[1]
+	if !strings.Contains(find, `(sAMAccountName=ada\2a\29\28x=)`) || !strings.Contains(find, "(memberOf=CN=llm,DC=corp,DC=example)") ||
+		!strings.Contains(find, "1.2.840.113556.1.4.803:=2") || !strings.Contains(groups, inChain) {
+		t.Errorf("filters %q", c.filters)
+	}
+	// Service account, then the person, then the service account again.
+	if len(c.binds) != 3 || c.binds[1] != "CN=Ada,OU=Staff,DC=corp,DC=example" {
+		t.Errorf("binds %v", c.binds)
+	}
+}
+
+func TestWrongPasswordsEmptyOnesAndAmbiguousNamesAreRefusedAlike(t *testing.T) {
+	for _, tc := range []struct {
+		people   int
+		password string
+	}{{1, "wrong"}, {1, ""}, {0, "ad-secret"}, {2, "ad-secret"}} {
+		d := fakeLDAP(t, &fakeConn{password: "ad-secret", people: tc.people})
+		if _, err := d.Authenticate(t.Context(), "ada", tc.password); !errors.Is(err, ErrBadCredentials) {
+			t.Errorf("%d found, password %q: %v", tc.people, tc.password, err)
+		}
+	}
+}
+
+func TestPlaintextLDAPIsRefusedWithoutStartTLS(t *testing.T) {
+	base := config.LDAP{BindDN: "CN=x", BindPasswordFile: "/f", BaseDN: "DC=corp"}
+	for _, c := range []struct {
+		urls  []string
+		start bool
+		ok    bool
+	}{{[]string{"ldaps://dc1"}, false, true}, {[]string{"ldap://dc1"}, false, false},
+		{[]string{"ldap://dc1"}, true, true}, {[]string{"http://dc1"}, true, false}} {
+		l := base
+		l.URLs, l.StartTLS = c.urls, c.start
+		if err := l.Validate(); (err == nil) != c.ok {
+			t.Errorf("%v start_tls=%v: %v", c.urls, c.start, err)
+		}
+	}
+}
+
+func slicesHas(xs []string, x string) bool {
+	for _, v := range xs {
+		if v == x {
+			return true
+		}
+	}
+	return false
+}
+
+func TestADirectoryIsTriedStepByStepBeforeItsSettingsAreKept(t *testing.T) {
+	m := Mapping{Roles: map[string][]string{"admin": {"llmgw-admins"}}}
+	checks := fakeLDAP(t, &fakeConn{people: 1}).WithPassword("typed-in").Check(t.Context(), m)
+	if len(checks) != 3 || !checks[0].OK || checks[0].Step != "ldaps://dc1" || !checks[1].OK ||
+		checks[2].Step != "group llmgw-admins" || !checks[2].OK {
+		t.Errorf("checks %+v", checks)
+	}
+	// A refused service account stops at the server, and says why.
+	checks = fakeLDAP(t, &fakeConn{refused: true}).Check(t.Context(), m)
+	if len(checks) != 1 || checks[0].OK || !strings.Contains(checks[0].Detail, "Invalid Credentials") {
+		t.Errorf("refused: %+v", checks)
+	}
+}
+
+func TestACertificateGoCannotReadIsSaidNotSkipped(t *testing.T) {
+	if _, err := certPool([]byte("not a certificate")); err == nil {
+		t.Error("an empty pool was taken")
+	}
+	// A CERTIFICATE block Go cannot parse, as it cannot Samba's own (its
+	// serial is negative): an error, not an empty pool.
+	neg := "-----BEGIN CERTIFICATE-----\nMIIBNTCB3KADAgECAgT/////MAoGCCqGSM49BAMCMA8xDTALBgNVBAMTBHRlc3Qw\n" +
+		"HhcNMjYwMTAxMDAwMDAwWhcNMjgwMTAxMDAwMDAwWjAPMQ0wCwYDVQQDEwR0ZXN0\n" +
+		"MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE\n-----END CERTIFICATE-----\n"
+	if _, err := certPool([]byte(neg)); err == nil {
+		t.Error("a broken certificate made a pool")
+	}
+}

internal/docgen/reference_gen.go

@@ -33,6 +33,7 @@ var Reference = []Block{
 		{Key: "default_tpm", Type: "int", Doc: "", Nested: ""},
 		{Key: "signup_open", Type: "bool", Doc: "SignupOpen serves the signup form. Accounts it creates are pending until\nan operator approves them, so this is not a way in — but turning it off\nmeans accounts are made with `llmgw users create` instead.", Nested: ""},
 		{Key: "oidc", Type: "OIDC", Doc: "OIDC is sign-in through an identity provider.", Nested: "config.OIDC"},
+		{Key: "ldap", Type: "LDAP", Doc: "LDAP is sign-in against a directory, Active Directory above all, and\nthe accounts kept in step with it.", Nested: "config.LDAP"},
 		{Key: "scim", Type: "SCIM", Doc: "SCIM is account provisioning by the identity provider.", Nested: "config.SCIM"},
 		{Key: "max_key_ttl", Type: "time.Duration", Doc: "MaxKeyTTL is how long any API key may live; a key asked for with no\nexpiry is given this. Organizations, teams and projects can set\nshorter ones (llmgw keys ttl). Zero is no limit.", Nested: ""},
 		{Key: "personal", Type: "Personal", Doc: "Personal is what a person may do on their own, outside any project: make\npersonal keys and use the chat page personally, within these limits.", Nested: "config.Personal"},
@@ -125,11 +126,16 @@ var Reference = []Block{
 		{Key: "url", Type: "string", Doc: "URL is the provider's base address, without /v1: https://api.openai.com.", Nested: ""},
 		{Key: "model", Type: "string", Doc: "Model is what the provider calls it; requests are rewritten to it.", Nested: ""},
 		{Key: "api_key_file", Type: "string", Doc: "APIKeyFile holds the provider's key, on the control plane. A file\nrather than the key itself, so the key is in neither the catalog, the\nstore nor the audit log; empty for a provider that asks for none.", Nested: ""},
+		{Key: "api_key_files", Type: "[]ExternalKey", Doc: "APIKeyFiles are several keys for the provider, in place of\nAPIKeyFile: one endpoint each (external:MODEL#1, #2…), so a key that\nanswers 429 or 401 is passed over while its breaker is open and the\nrequest goes to another. A provider's limits are per key.", Nested: ""},
 		{Key: "peer", Type: "string", Doc: "Peer names the federated llmgw cluster this model is offered by, for\na model that federation added (`llmgw ctl federation add`) rather\nthan one written by hand. Such a model is named PEER/MODEL, comes and\ngoes with the peer's own list, and is sent only while the peer is up.", Nested: ""},
 		{Key: "api", Type: "string", Doc: "API is what the provider speaks: OpenAI's, unless this says comfyui —\na ComfyUI server, asked for pictures at /v1/images/generations and\nsent the workflow below with the request's prompt in it. Model is then\nthe checkpoint file the workflow loads, as ComfyUI lists it.", Nested: ""},
 		{Key: "workflow", Type: "string", Doc: "Workflow is the ComfyUI workflow, in its API format: a built-in's name\n(flux-schnell, the default, or flux-dev) or a JSON file on the control\nplane.", Nested: ""},
 		{Key: "output", Type: "string", Doc: "Output is what it makes: pictures, or meshes (see ComfyUI.Output).", Nested: ""},
 	}},
+	{Package: "catalog", Name: "ExternalKey", Title: "ExternalKey", Doc: "", Fields: []Field{
+		{Key: "file", Type: "string", Doc: "File holds the key, on the control plane, as APIKeyFile does.", Nested: ""},
+		{Key: "weight", Type: "float64", Doc: "Weight is its share of the requests against the others'; zero is 1.", Nested: ""},
+	}},
 	{Package: "config", Name: "Facility", Title: "Facility", Doc: "", Fields: []Field{
 		{Key: "budgets", Type: "[]FacilityBudget", Doc: "Budgets are the domains with a power cap.", Nested: ""},
 		{Key: "margin", Type: "float64", Doc: "Margin is the share of every cap kept free, 0.1 for ten percent. Zero\nkeeps none, for caps already given derated.", Nested: ""},
@@ -183,6 +189,23 @@ var Reference = []Block{
 		{Key: "min_blocks", Type: "int", Doc: "MinBlocks is the fewest leading blocks a replica must hold to be\npreferred for them; fewer is coincidence — a chat template's opening\ntokens — rather than a prefix worth following. Zero takes 4, which is\n64 tokens at vLLM's usual block size of 16.", Nested: ""},
 		{Key: "max_blocks", Type: "int", Doc: "MaxBlocks bounds how many of a prompt's blocks are hashed and looked\nup. Beyond it a longer match still helps, but the difference no longer\ndecides anything. Zero takes 4096.", Nested: ""},
 	}},
+	{Package: "config", Name: "LDAP", Title: "LDAP", Doc: "", Fields: []Field{
+		{Key: "urls", Type: "[]string", Doc: "URLs are the directory's servers, tried in turn: ldaps:// (636), or\nldap:// (389) with StartTLS.", Nested: ""},
+		{Key: "start_tls", Type: "bool", Doc: "StartTLS upgrades an ldap:// connection before anything is sent. A\npassword never crosses a connection without TLS: ldap:// without it\nis refused.", Nested: ""},
+		{Key: "ca_file", Type: "string", Doc: "CAFile is the authority the servers' certificates are checked against,\nwhen it is not the system's: an AD Certificate Services root.", Nested: ""},
+		{Key: "ca", Type: "string", Doc: "CA is the same authority written out, PEM: kept in the cluster's\nsettings, so no file has to be put on every control plane. Used in\nplace of CAFile when both are set.", Nested: ""},
+		{Key: "bind_dn", Type: "string", Doc: "BindDN and the file holding its password: a service account that may\nread users and groups. Nothing is written to the directory.", Nested: ""},
+		{Key: "bind_password_file", Type: "string", Doc: "", Nested: ""},
+		{Key: "base_dn", Type: "string", Doc: "BaseDN is where people are searched for, subtree included.", Nested: ""},
+		{Key: "group_base_dn", Type: "string", Doc: "GroupBaseDN is where their groups are; empty is BaseDN's domain root.", Nested: ""},
+		{Key: "user_filter", Type: "string", Doc: "UserFilter narrows who may sign in, ANDed with the login match: a\ngroup (memberOf=...) or an OU's people only. Empty is every enabled\nuser object.", Nested: ""},
+		{Key: "label", Type: "string", Doc: "Label is what the form says to sign in with: \"Sign in with your\n<label> account\". Empty is \"Active Directory\".", Nested: ""},
+		{Key: "create_accounts", Type: "bool", Doc: "CreateAccounts makes an account, active, for a person the directory\nvouches for and the gateway does not know. Off, only existing accounts\nsign in this way.", Nested: ""},
+		{Key: "roles", Type: "map[string][]string", Doc: "Roles maps a staff role (viewer, operator, billing, admin) to the\ngroups that give it, by name (CN) or whole DN. Set at every sign-in\nand sync: someone taken out of the group loses the role.", Nested: ""},
+		{Key: "memberships", Type: "map[string]string", Doc: "Memberships maps a group, by CN or DN, to a place in the tree —\norg:NAME, team:ORG/TEAM, project:ORG/TEAM/PROJECT — joined at sign-in\nand kept by the sync, which also takes away a place whose group the\nperson has left.", Nested: ""},
+		{Key: "sync_interval", Type: "time.Duration", Doc: "SyncInterval is how often the leader reads the directory. Zero is\n15m; negative turns the sync off, leaving only sign-in.", Nested: ""},
+		{Key: "timeout", Type: "time.Duration", Doc: "Timeout bounds each round trip to a server. Zero is 10s.", Nested: ""},
+	}},
 	{Package: "config", Name: "Landing", Title: "Landing", Doc: "", Fields: []Field{
 		{Key: "contact", Type: "string", Doc: "Contact is where someone without a key should go — an address, a URL, a\nchannel name. Rendered as a link when it is one, as text otherwise.", Nested: ""},
 		{Key: "notice", Type: "string", Doc: "Notice is an optional line of your own, for whatever the reader needs to\nknow that this page cannot guess.", Nested: ""},

internal/docgen/render.go

@@ -33,6 +33,7 @@ var Sources = []string{
 	"internal/guard/policy.go",
 	"internal/guard/judge.go",
 	"internal/config/oidc.go",
+	"internal/config/ldap.go",
 	"internal/config/firewall.go",
 	"internal/config/serving.go",
 	"internal/config/presets.go",

internal/notify/notify.go

@@ -43,10 +43,15 @@ const (
 	// CanaryRolledBack is a model's canary taken out because it served
 	// worse than the model it was a candidate to replace.
 	CanaryRolledBack = "canary_rolled_back"
+	// ExternalKeyRefused is an outside provider answering 401 to one of a
+	// model's keys: revoked, expired or mistyped. Requests go to its other
+	// keys meanwhile, where it has any.
+	ExternalKeyRefused = "external_key_refused"
 )
 
 // Kinds are every event there is.
-var Kinds = []string{ModelParked, NodeLost, BudgetSpent, UpdateAvailable, JudgeUnavailable, CanaryRolledBack}
+var Kinds = []string{ModelParked, NodeLost, BudgetSpent, UpdateAvailable, JudgeUnavailable, CanaryRolledBack,
+	ExternalKeyRefused}
 
 // Event is one thing that happened.
 type Event struct {

internal/router/activate.go

@@ -131,7 +131,18 @@ const defaultActivationWait = 45 * time.Second
 // SetNotifier tells the accountant where to say a budget is spent. Nothing
 // without a store, where there are no budgets.
 func (r *Router) SetNotifier(n *notify.Notifier) {
+	r.notifier = n
 	if r.acct != nil {
 		r.acct.SetNotifier(n)
 	}
 }
+
+// keyRefused says that an outside provider answered 401 to one of a model's
+// keys, by its endpoint, which numbers it: never the key.
+func (r *Router) keyRefused(model, endpointID string) {
+	r.log.Warn("an outside provider refused a key", "model", model, "endpoint", endpointID)
+	r.notifier.Notify(notify.ExternalKeyRefused, endpointID,
+		"the provider of "+model+" answered 401 to the key of "+endpointID+
+			": revoked, expired or mistyped; replace its file on the control plane",
+		map[string]any{"model": model, "endpoint": endpointID})
+}

internal/router/directory.go

@@ -0,0 +1,178 @@
+package router
+
+import (
+	"cmp"
+	"context"
+	"errors"
+	"fmt"
+	"log/slog"
+	"sync"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/account"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/directory"
+	storepkg "gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// Sign-in against a directory (auth.ldap), and the accounts kept in step
+// with it by the leader.
+
+// ldapDirectory is auth.ldap as the gateway uses it.
+type ldapDirectory struct {
+	dir    *directory.LDAP
+	keeper *directory.Keeper
+	every  time.Duration
+	label  string
+	log    *slog.Logger
+
+	mu   sync.Mutex
+	last *adminapi.DirectorySyncView
+}
+
+func newLDAPDirectory(cfg config.LDAP, st *storepkg.DB, log *slog.Logger) (*ldapDirectory, error) {
+	dir, err := directory.NewLDAP(cfg)
+	if err != nil {
+		return nil, err
+	}
+	return &ldapDirectory{
+		dir: dir,
+		keeper: &directory.Keeper{St: st, Name: "ldap", CreateAccounts: cfg.CreateAccounts,
+			Mapping: directory.Mapping{Roles: cfg.Roles, Memberships: cfg.Memberships}},
+		every: cfg.Interval(),
+		label: cmp.Or(cfg.Label, "Active Directory"),
+		log:   log,
+	}, nil
+}
+
+// SignIn is account.Directory: the person's name and password checked by
+// the directory, then their account made or kept in step.
+func (d *ldapDirectory) SignIn(ctx context.Context, login, password string) (storepkg.User, error) {
+	id, err := d.dir.Authenticate(ctx, login, password)
+	switch {
+	case errors.Is(err, directory.ErrBadCredentials):
+		return storepkg.User{}, account.ErrBadCredentials
+	case err != nil:
+		d.log.Warn("the directory could not be asked about a sign-in", "err", err)
+		return storepkg.User{}, fmt.Errorf("the directory could not be asked: %w", err)
+	}
+	u, err := d.keeper.SignIn(ctx, id)
+	switch {
+	case errors.Is(err, directory.ErrSuspended):
+		return storepkg.User{}, account.ErrSuspended
+	case errors.Is(err, directory.ErrNoAccount), errors.Is(err, directory.ErrNoEmail):
+		// Said to the log, not the form: the form says what a wrong
+		// password says.
+		d.log.Info("the directory vouched for a person who has no account here", "dn", id.DN, "err", err)
+		return storepkg.User{}, account.ErrBadCredentials
+	case errors.Is(err, directory.ErrNotActive):
+		return storepkg.User{}, account.ErrNotApproved
+	case err != nil:
+		return storepkg.User{}, err
+	}
+	return u, nil
+}
+
+// sync reads the directory every interval, on the leader, and brings the
+// accounts it keeps in step.
+func (d *ldapDirectory) sync(ctx context.Context, leader func() bool) {
+	if d.every <= 0 {
+		return
+	}
+	t := time.NewTicker(d.every)
+	defer t.Stop()
+	for {
+		if leader() {
+			d.syncOnce(ctx)
+		}
+		select {
+		case <-ctx.Done():
+			return
+		case <-t.C:
+		}
+	}
+}
+
+func (d *ldapDirectory) syncOnce(ctx context.Context) {
+	ctx, cancel := context.WithTimeout(ctx, 5*time.Minute)
+	defer cancel()
+	last := &adminapi.DirectorySyncView{At: time.Now()}
+	defer func() {
+		d.mu.Lock()
+		d.last = last
+		d.mu.Unlock()
+	}()
+	people, err := d.dir.People(ctx, d.keeper.Mapping)
+	if err != nil {
+		d.log.Warn("could not read the directory; accounts left as they are", "err", err)
+		last.Error = "could not read the directory: " + err.Error()
+		return
+	}
+	last.People = len(people)
+	rep, err := d.keeper.Sync(ctx, people)
+	if err != nil {
+		d.log.Error("the directory sync stopped", "err", err, "people", len(people))
+		last.Error = err.Error()
+		return
+	}
+	last.Accounts, last.Suspended, last.Reinstated, last.RolesChanged = rep.Seen, rep.Suspended, rep.Reinstated, rep.Updated
+	for _, where := range rep.Unmapped {
+		d.log.Warn("auth.ldap.memberships maps a group to a place the tree does not have", "place", where)
+	}
+	if rep.Suspended+rep.Reinstated+rep.Updated > 0 {
+		d.log.Info("accounts kept in step with the directory", "people", len(people), "accounts", rep.Seen,
+			"suspended", rep.Suspended, "reinstated", rep.Reinstated, "roles_changed", rep.Updated)
+	}
+}
+
+// lastSync is how the leader's last read of the directory went; nil before
+// the first, and on a standby.
+func (d *ldapDirectory) lastSync() *adminapi.DirectorySyncView {
+	d.mu.Lock()
+	defer d.mu.Unlock()
+	return d.last
+}
+
+// AuthStatus is how people sign in on this gateway, for the admin API.
+func (r *Router) AuthStatus() adminapi.AuthView {
+	v := adminapi.AuthView{SCIM: r.scim.TokenFile != "", BaseURL: r.mail.BaseURL}
+	if r.accounts != nil {
+		v.SignupOpen = r.accounts.SignupOpen()
+	}
+	if o := r.oidc; o.Enabled() {
+		v.OIDC = &adminapi.OIDCView{Issuer: o.Issuer, ClientID: o.ClientID, Label: o.Label,
+			CreateAccounts: o.CreateAccounts, NoPasswords: o.NoPasswords}
+	}
+	if l := r.ldap; l.Enabled() {
+		v.LDAP = &adminapi.LDAPView{URLs: l.URLs, BaseDN: l.BaseDN, BindDN: l.BindDN, Label: l.Label, Off: r.ldapOff}
+		if r.directory != nil {
+			v.LDAP.LastSync = r.directory.lastSync()
+		}
+	}
+	return v
+}
+
+// TestDirectory tries an auth.ldap block, given as YAML, before it is kept:
+// checked as the settings would be, then each step against the directory.
+func (r *Router) TestDirectory(ctx context.Context, settings, password string) ([]adminapi.CheckView, error) {
+	l, err := config.ParseLDAP([]byte(settings))
+	if err != nil {
+		return nil, err
+	}
+	dir, err := directory.NewLDAP(l)
+	if err != nil {
+		return []adminapi.CheckView{{Step: "certificate", Detail: err.Error()}}, nil
+	}
+	if password != "" {
+		dir.WithPassword(password)
+	}
+	ctx, cancel := context.WithTimeout(ctx, time.Minute)
+	defer cancel()
+	checks := dir.Check(ctx, directory.Mapping{Roles: l.Roles, Memberships: l.Memberships})
+	out := make([]adminapi.CheckView, len(checks))
+	for i, c := range checks {
+		out[i] = adminapi.CheckView(c)
+	}
+	return out, nil
+}

internal/router/directory_test.go

@@ -0,0 +1,45 @@
+package router
+
+import (
+	"net/http"
+	"net/http/httptest"
+	"path/filepath"
+	"strings"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+)
+
+// auth.ldap is the cluster's: a CA file one machine lacks must not keep its
+// control plane from starting, where nothing could put the setting right.
+func TestADirectoryThatCannotBeUsedHereLeavesTheGatewayUp(t *testing.T) {
+	cfg := baseConfig(config.Endpoint{ID: "a", URL: "http://127.0.0.1:1", Models: []string{"m"}})
+	cfg.Auth.Enabled = true
+	cfg.Auth.Store = filepath.Join(t.TempDir(), "store.db")
+	cfg.Auth.LDAP = config.LDAP{URLs: []string{"ldaps://dc1"}, CAFile: filepath.Join(t.TempDir(), "missing.pem"),
+		BindDN: "CN=svc", BindPasswordFile: "/nonexistent", BaseDN: "DC=corp"}
+	r := newTestRouter(t, cfg)
+	if r.directory != nil {
+		t.Fatal("a directory was set up from a CA file that is not there")
+	}
+	w := httptest.NewRecorder()
+	r.Handler().ServeHTTP(w, httptest.NewRequest(http.MethodGet, "/login", nil))
+	if w.Code != http.StatusOK || !strings.Contains(w.Body.String(), `type="email"`) {
+		t.Errorf("/login: %d", w.Code)
+	}
+}
+
+// Settings tried are checked as kept ones are: a plaintext server is
+// refused before anything is sent to it.
+func TestDirectorySettingsTriedAreCheckedFirst(t *testing.T) {
+	r := &Router{}
+	if _, err := r.TestDirectory(t.Context(), "urls: [ldap://dc1]\nbind_dn: CN=svc\nbind_password_file: store:auth/ldap-bind\nbase_dn: DC=corp\n", "pw"); err == nil ||
+		!strings.Contains(err.Error(), "plaintext") {
+		t.Errorf("err %v", err)
+	}
+	checks, err := r.TestDirectory(t.Context(), "urls: [ldaps://dc1]\nca: \"-----BEGIN CERTIFICATE-----\\nAAAA\\n-----END CERTIFICATE-----\"\n"+
+		"bind_dn: CN=svc\nbind_password_file: store:auth/ldap-bind\nbase_dn: DC=corp\n", "pw")
+	if err != nil || len(checks) != 1 || checks[0].Step != "certificate" || checks[0].OK {
+		t.Errorf("a broken CA: %+v %v", checks, err)
+	}
+}

internal/router/external_test.go

@@ -6,11 +6,15 @@ import (
 	"io"
 	"net/http"
 	"net/http/httptest"
+	"strings"
 	"sync/atomic"
 	"testing"
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/notify"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/notify/notifytest"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/router/breaker"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
 )
 
@@ -87,3 +91,68 @@ func TestAnExternalModelIsNotAModelWithProfiles(t *testing.T) {
 		}
 	}
 }
+
+// Two keys for one provider, one of them at its limit, or refused: every
+// request is answered by the other key, the failing one's breaker opens, and
+// a refused key is said, by its endpoint and never its value.
+func TestAKeyAtItsLimitOrRefusedIsPassedOverForAnother(t *testing.T) {
+	for _, status := range []int{http.StatusTooManyRequests, http.StatusUnauthorized} {
+		provider := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			if r.Header.Get("Authorization") == "Bearer key-a" {
+				http.Error(w, "no", status)
+				return
+			}
+			w.Header().Set("Content-Type", "application/json")
+			fmt.Fprint(w, `{"model":"m","choices":[{"message":{"content":"ok"}}],"usage":{"prompt_tokens":1,"completion_tokens":1}}`)
+		}))
+		t.Cleanup(provider.Close)
+		cfg := baseConfig()
+		cfg.Models = []catalog.Model{{Name: "cloud", Source: "external",
+			External: &catalog.External{URL: provider.URL, Model: "m"}}}
+		rt := newTestRouter(t, cfg)
+		key := func(id, k string) endpoint.Info {
+			return endpoint.Info{ID: id, URL: provider.URL, Models: []string{"cloud"}, APIKey: k,
+				Engine: endpoint.EngineExternal, Pool: "external", Weight: 1, UpstreamModel: "m"}
+		}
+		rt.SetEndpoints(t.Context(), []endpoint.Info{key("external:cloud#1", "key-a"), key("external:cloud#2", "key-b")})
+		n, events := notifytest.Catch(t)
+		rt.SetNotifier(n)
+
+		// Until the failing key's breaker opens: how many requests the
+		// scheduler sends it before then is its own choice.
+		open := func() bool {
+			for _, s := range rt.reg.Snapshots("cloud") {
+				if s.ID == "external:cloud#1" && s.Breaker != breaker.Closed {
+					return true
+				}
+			}
+			return false
+		}
+		for i := 0; i < 100 && !open(); i++ {
+			if rec := post(t, rt.Handler(), map[string]any{"model": "cloud", "messages": []any{}}, nil); rec.Code != http.StatusOK {
+				t.Fatalf("key-a answering %d: request %d got %d: %s", status, i, rec.Code, rec.Body.String())
+			}
+		}
+		for _, s := range rt.reg.Snapshots("cloud") {
+			if s.ID == "external:cloud#1" && s.Breaker == breaker.Closed {
+				t.Errorf("%d: the failing key's breaker is still closed", status)
+			}
+			if s.ID == "external:cloud#2" && s.Breaker != breaker.Closed {
+				t.Errorf("%d: the good key's breaker opened", status)
+			}
+		}
+		var refused []notify.Event
+		for _, e := range events() {
+			if e.Kind == notify.ExternalKeyRefused {
+				refused = append(refused, e)
+			}
+		}
+		switch {
+		case status == http.StatusTooManyRequests && len(refused) != 0:
+			t.Errorf("a key at its limit was said to be refused: %+v", refused)
+		case status == http.StatusUnauthorized && (len(refused) != 1 || refused[0].Subject != "external:cloud#1" ||
+			strings.Contains(refused[0].Message, "key-a")):
+			t.Errorf("a refused key was said as %+v", refused)
+		}
+	}
+}

internal/router/handlers.go

@@ -243,6 +243,9 @@ func (r *Router) Handler() http.Handler {
 			return web.ModelFacts{ContextLength: m.ContextLength, MaxOutput: m.MaxOutput, Tools: m.Capabilities.Tools}
 		})
 		pages.SetPersonalOff(r.personalOff)
+		if r.directory != nil {
+			pages.SetDirectory(r.directory.label)
+		}
 		pages.SetLlamaCppVersion(r.llamaVersion)
 		pages.SetPersonalDefaults(r.personalDefaults)
 		if r.store != nil {

internal/router/router.go

@@ -34,6 +34,7 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/notify"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/router/admission"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/router/breaker"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
@@ -42,6 +43,7 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/router/proxy"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/router/scheduler"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/router/scrape"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/secrets"
 	storepkg "gitea.mynodes.xyz/ilker/llmgw/internal/store"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/web"
 )
@@ -116,6 +118,14 @@ type Router struct {
 	// federation is which cluster this is, to its peers; nil until said.
 	federation atomic.Pointer[federationID]
 	acct       *accounting.Accountant
+	// notifier says what the operator must act on; nil says nothing.
+	notifier *notify.Notifier
+	// directory is auth.ldap's: sign-in and the accounts kept in step.
+	// ldap is the setting, and ldapOff why this control plane could not
+	// use it.
+	directory *ldapDirectory
+	ldap      config.LDAP
+	ldapOff   string
 	// guard judges requests and answers; nil judges nothing. guardLibrary
 	// is the policies it was built from, for the pages that show them.
 	guard        guard.Guard
@@ -353,6 +363,9 @@ func New(cfg config.Server, log *slog.Logger) (*Router, error) {
 			return nil, fmt.Errorf("auth store: %w", err)
 		}
 		r.store = st
+		// Secrets the settings name as store:… are the same on every
+		// control plane: read from here.
+		secrets.SetStore(st.StoredSecret)
 		r.usageRetention = cfg.Auth.UsageRetention
 		// Applied where no scope set one, so a project created without limits
 		// is still metered. Zero leaves it unlimited, which was the old
@@ -392,6 +405,20 @@ func New(cfg config.Server, log *slog.Logger) (*Router, error) {
 		r.accounts = account.New(st, account.Config{
 			SignupOpen: cfg.Auth.SignupOpen,
 		}, log)
+		if cfg.Auth.LDAP.Enabled() {
+			// A setting every control plane shares: one that cannot be used
+			// here (a CA file missing on this machine) must not keep them all
+			// down, where nothing could be used to put it right. Local
+			// accounts still sign in.
+			r.ldap = cfg.Auth.LDAP
+			if d, err := newLDAPDirectory(cfg.Auth.LDAP, st, log); err != nil {
+				log.Error("sign-in with the directory is off on this control plane", "err", err)
+				r.ldapOff = err.Error()
+			} else {
+				r.directory = d
+				r.accounts.SetDirectory(d)
+			}
+		}
 
 		r.acct = accounting.New(
 			accounting.NewPricing(prices, accounting.Price{
@@ -532,6 +559,9 @@ func (r *Router) Start(ctx context.Context) {
 	if r.threads != nil {
 		go r.pruneThreads(ctx)
 	}
+	if r.directory != nil {
+		go r.directory.sync(ctx, r.leading)
+	}
 	if r.store != nil && r.usageRetention > 0 {
 		go r.pruneUsage(ctx, r.usageRetention)
 	}
@@ -1193,6 +1223,12 @@ func (r *Router) dispatchOne(ctx context.Context, w http.ResponseWriter, req *pa
 			// restart. Not relayed; loaded again and tried again.
 			preq.RetryStatus = http.StatusNotFound
 		}
+		// One of an outside provider's several keys refused: the request
+		// is the provider's to answer with another of them, and the key's
+		// breaker counts it. With one key the 401 is the client's to see.
+		if dec.Endpoint.Engine == endpoint.EngineExternal && strings.Contains(dec.Endpoint.ID, "#") {
+			preq.RetryStatus = http.StatusUnauthorized
+		}
 		if dec.Endpoint.Peer != "" {
 			preq.Extra = r.peerHeaders(ctx, dec.Endpoint, req.Header)
 		}
@@ -1209,6 +1245,9 @@ func (r *Router) dispatchOne(ctx context.Context, w http.ResponseWriter, req *pa
 		} else {
 			out, perr = r.px.Do(uctx, w, preq)
 		}
+		if dec.Endpoint.Engine == endpoint.EngineExternal && out.Status == http.StatusUnauthorized {
+			r.keyRefused(model, dec.Endpoint.ID)
+		}
 		releaseAdapter()
 		uspan.SetAttributes(attribute.Int("http.response.status_code", out.Status))
 		endSpan(uspan, perr)

internal/secrets/secrets.go

@@ -8,6 +8,8 @@
 //	/etc/llmgw/secrets/store-dsn         a file, trimmed
 //	env:LLMGW_STORE_DSN                  an environment variable
 //	vault:secret/llmgw/store#dsn         a field of a KV secret in Vault
+//	store:auth/ldap-bind                 a secret the cluster's store keeps,
+//	                                     on a control plane only
 //
 // A Vault path's first segment is its KV v2 mount; secret/data/… is taken
 // as written. KV v1 mounts work too.
@@ -85,7 +87,21 @@ func Configure(v Vault) error {
 
 // IsReference says whether s names a secret somewhere other than a file.
 func IsReference(s string) bool {
-	return strings.HasPrefix(s, "env:") || strings.HasPrefix(s, "vault:")
+	return strings.HasPrefix(s, "env:") || strings.HasPrefix(s, "vault:") || strings.HasPrefix(s, "store:")
+}
+
+var (
+	storeMu   sync.RWMutex
+	storeRead func(ctx context.Context, name string) (string, error)
+)
+
+// SetStore is how store: references are read: a control plane sets it once
+// its store is open. A secret kept there is the same for every control
+// plane, where a file would have to be put on each.
+func SetStore(read func(ctx context.Context, name string) (string, error)) {
+	storeMu.Lock()
+	defer storeMu.Unlock()
+	storeRead = read
 }
 
 // Read reads a secret by reference, trimmed of surrounding space.
@@ -100,6 +116,18 @@ func Read(ctx context.Context, ref string) ([]byte, error) {
 		return []byte(strings.TrimSpace(v)), nil
 	case strings.HasPrefix(ref, "vault:"):
 		return readVault(ctx, strings.TrimPrefix(ref, "vault:"))
+	case strings.HasPrefix(ref, "store:"):
+		storeMu.RLock()
+		read := storeRead
+		storeMu.RUnlock()
+		if read == nil {
+			return nil, fmt.Errorf("%s: a store: secret is read on a control plane, which has the store", ref)
+		}
+		v, err := read(ctx, strings.TrimPrefix(ref, "store:"))
+		if err != nil {
+			return nil, fmt.Errorf("%s: %w", ref, err)
+		}
+		return []byte(strings.TrimSpace(v)), nil
 	}
 	b, err := os.ReadFile(ref) //nolint:gosec // a path the operator configured
 	if err != nil {

internal/secrets/secrets_test.go

@@ -1,6 +1,7 @@
 package secrets
 
 import (
+	"context"
 	"encoding/base64"
 	"encoding/json"
 	"net/http"
@@ -140,3 +141,15 @@ func TestSealersSealAndOpen(t *testing.T) {
 		t.Errorf("transit open: %q %v", got, err)
 	}
 }
+
+func TestAStoreReferenceIsReadFromTheStoreOnAControlPlaneOnly(t *testing.T) {
+	SetStore(nil)
+	if _, err := Read(t.Context(), "store:auth/ldap-bind"); err == nil || !strings.Contains(err.Error(), "control plane") {
+		t.Errorf("without a store: %v", err)
+	}
+	SetStore(func(_ context.Context, name string) (string, error) { return " kept:" + name + "\n", nil })
+	t.Cleanup(func() { SetStore(nil) })
+	if b, err := Read(t.Context(), "store:auth/ldap-bind"); err != nil || string(b) != "kept:auth/ldap-bind" {
+		t.Errorf("%q %v", b, err)
+	}
+}

internal/store/copy.go

@@ -27,6 +27,8 @@ var copyOrder = []string{
 	"memberships",
 	// After users, which it points at.
 	"user_excluded_licences",
+	// After users, which it points at.
+	"user_directory",
 	"preset_state",
 	"api_keys",
 	"key_claims",

internal/store/directory.go

@@ -0,0 +1,92 @@
+package store
+
+import (
+	"context"
+	"database/sql"
+	"errors"
+	"time"
+)
+
+// DirectoryLink is an account kept by an outside directory: which one, the
+// account's id there, and whether the directory's sync is what suspended it.
+type DirectoryLink struct {
+	UserID     string
+	Directory  string // "ldap"
+	ExternalID string // an AD objectGUID
+	// SuspendedBySync is an account suspended because the directory
+	// disabled or lost it, and so reinstated when the directory has it
+	// enabled again. One an administrator suspended is not.
+	SuspendedBySync bool
+	SyncedAt        time.Time
+}
+
+// DirectoryUser is the account a directory knows by that id.
+func (s *DB) DirectoryUser(ctx context.Context, directory, externalID string) (DirectoryLink, error) {
+	var l DirectoryLink
+	var suspended int
+	var synced int64
+	err := s.db.QueryRowContext(ctx,
+		`SELECT user_id, directory, external_id, suspended_by_sync, synced_at FROM user_directory
+		 WHERE directory = ? AND external_id = ?`, directory, externalID).
+		Scan(&l.UserID, &l.Directory, &l.ExternalID, &suspended, &synced)
+	if errors.Is(err, sql.ErrNoRows) {
+		return DirectoryLink{}, ErrNotFound
+	}
+	if err != nil {
+		return DirectoryLink{}, err
+	}
+	l.SuspendedBySync, l.SyncedAt = suspended != 0, directoryTime(synced)
+	return l, nil
+}
+
+// LinkDirectory records that a directory keeps an account, by its id there.
+func (s *DB) LinkDirectory(ctx context.Context, userID, directory, externalID string, at time.Time) error {
+	_, err := s.db.ExecContext(ctx,
+		`INSERT INTO user_directory (user_id, directory, external_id, synced_at) VALUES (?, ?, ?, ?)
+		 ON CONFLICT (user_id) DO UPDATE SET directory = excluded.directory, external_id = excluded.external_id,
+		   synced_at = excluded.synced_at`,
+		userID, directory, externalID, at.UnixMilli())
+	return err
+}
+
+// DirectoryLinks are every account a directory keeps.
+func (s *DB) DirectoryLinks(ctx context.Context, directory string) ([]DirectoryLink, error) {
+	rows, err := s.db.QueryContext(ctx,
+		`SELECT user_id, directory, external_id, suspended_by_sync, synced_at FROM user_directory WHERE directory = ?`,
+		directory)
+	if err != nil {
+		return nil, err
+	}
+	defer func() { _ = rows.Close() }()
+	var out []DirectoryLink
+	for rows.Next() {
+		var l DirectoryLink
+		var suspended int
+		var synced int64
+		if err := rows.Scan(&l.UserID, &l.Directory, &l.ExternalID, &suspended, &synced); err != nil {
+			return nil, err
+		}
+		l.SuspendedBySync, l.SyncedAt = suspended != 0, directoryTime(synced)
+		out = append(out, l)
+	}
+	return out, rows.Err()
+}
+
+// SetSyncSuspended says whether the directory's sync is what has an account
+// suspended, and when it last looked.
+func (s *DB) SetSyncSuspended(ctx context.Context, userID string, suspended bool, at time.Time) error {
+	v := 0
+	if suspended {
+		v = 1
+	}
+	_, err := s.db.ExecContext(ctx,
+		`UPDATE user_directory SET suspended_by_sync = ?, synced_at = ? WHERE user_id = ?`, v, at.UnixMilli(), userID)
+	return err
+}
+
+func directoryTime(ms int64) time.Time {
+	if ms == 0 {
+		return time.Time{}
+	}
+	return time.UnixMilli(ms)
+}

internal/store/migrations/postgres/0048_user_directory.sql

@@ -0,0 +1,14 @@
+-- Ported from the SQLite migration of the same number.
+-- Which directory keeps an account, and its id there: an Active Directory
+-- objectGUID, so a person whose mail changes there is still the same account
+-- here. suspended_by_sync is an account the directory sync suspended,
+-- because it was disabled or gone there; one an administrator suspended
+-- here is not, and is never reinstated by the sync.
+CREATE TABLE user_directory (
+  user_id           TEXT PRIMARY KEY REFERENCES users(id) ON DELETE CASCADE,
+  directory         TEXT NOT NULL,
+  external_id       TEXT NOT NULL,
+  suspended_by_sync INTEGER NOT NULL DEFAULT 0,
+  synced_at         BIGINT NOT NULL DEFAULT 0,
+  UNIQUE (directory, external_id)
+);

internal/store/migrations/sqlite/0048_user_directory.sql

@@ -0,0 +1,13 @@
+-- Which directory keeps an account, and its id there: an Active Directory
+-- objectGUID, so a person whose mail changes there is still the same account
+-- here. suspended_by_sync is an account the directory sync suspended,
+-- because it was disabled or gone there; one an administrator suspended
+-- here is not, and is never reinstated by the sync.
+CREATE TABLE user_directory (
+  user_id           TEXT PRIMARY KEY REFERENCES users(id) ON DELETE CASCADE,
+  directory         TEXT NOT NULL,
+  external_id       TEXT NOT NULL,
+  suspended_by_sync INTEGER NOT NULL DEFAULT 0,
+  synced_at         INTEGER NOT NULL DEFAULT 0,
+  UNIQUE (directory, external_id)
+);

internal/store/secrets.go

@@ -3,10 +3,13 @@ package store
 import (
 	"context"
 	"crypto/rand"
+	"database/sql"
 	"encoding/hex"
+	"errors"
 	"fmt"
-	"gitea.mynodes.xyz/ilker/llmgw/internal/secrets"
 	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/secrets"
 )
 
 // ClusterSecret returns the named secret, creating a random one the first time
@@ -39,6 +42,43 @@ func (s *DB) ClusterSecret(ctx context.Context, name string) (string, error) {
 		`SELECT value FROM cluster_secrets WHERE name = ?`, name).Scan(&v); err != nil {
 		return "", err
 	}
+	return s.openSecret(ctx, name, v)
+}
+
+// SetClusterSecret keeps value under name, sealed when a master key is set,
+// in place of what was there: a secret an operator gives, such as the
+// password the directory is read with, the same for every control plane.
+func (s *DB) SetClusterSecret(ctx context.Context, name, value string) error {
+	stored := value
+	if s.sealer != nil {
+		var err error
+		if stored, err = s.sealer.Seal(ctx, []byte(value)); err != nil {
+			return err
+		}
+	}
+	_, err := s.db.ExecContext(ctx,
+		`INSERT INTO cluster_secrets (name, value, created_at) VALUES (?, ?, ?)
+		 ON CONFLICT (name) DO UPDATE SET value = excluded.value, created_at = excluded.created_at`,
+		name, stored, time.Now().UnixMilli())
+	return err
+}
+
+// StoredSecret is the named secret, ErrNotFound when none was kept; unlike
+// ClusterSecret it never makes one.
+func (s *DB) StoredSecret(ctx context.Context, name string) (string, error) {
+	var v string
+	err := s.db.QueryRowContext(ctx, `SELECT value FROM cluster_secrets WHERE name = ?`, name).Scan(&v)
+	if errors.Is(err, sql.ErrNoRows) {
+		return "", ErrNotFound
+	}
+	if err != nil {
+		return "", err
+	}
+	return s.openSecret(ctx, name, v)
+}
+
+// openSecret is a kept value in the clear.
+func (s *DB) openSecret(ctx context.Context, name, v string) (string, error) {
 	if secrets.IsSealed(v) {
 		if s.sealer == nil {
 			return "", fmt.Errorf("the store's secret %q is sealed, and secrets.master_key is not set to open it", name)

internal/store/secrets_test.go

@@ -1,6 +1,7 @@
 package store
 
 import (
+	"errors"
 	"path/filepath"
 	"testing"
 )
@@ -23,3 +24,22 @@ func TestAClusterSecretIsMadeOnceAndKept(t *testing.T) {
 		t.Error("two names share a secret")
 	}
 }
+
+func TestASecretGivenIsKeptReplacedAndNeverMade(t *testing.T) {
+	db, err := Open(t.Context(), filepath.Join(t.TempDir(), "s.db"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer db.Close()
+	if _, err := db.StoredSecret(t.Context(), "auth/ldap-bind"); !errors.Is(err, ErrNotFound) {
+		t.Errorf("a secret never given: %v", err)
+	}
+	for _, v := range []string{"first", "second"} {
+		if err := db.SetClusterSecret(t.Context(), "auth/ldap-bind", v); err != nil {
+			t.Fatal(err)
+		}
+		if got, err := db.StoredSecret(t.Context(), "auth/ldap-bind"); err != nil || got != v {
+			t.Errorf("read %q %v, want %q", got, err, v)
+		}
+	}
+}

internal/store/users.go

@@ -179,10 +179,17 @@ func (s *DB) ApproveUser(ctx context.Context, id, projectID, by string, at time.
 // signup suspended while pending and then reinstated would become active
 // without anybody approving it.
 func (s *DB) SetUserState(ctx context.Context, id string, state UserState) error {
+	// An account nobody approved is approved, not set active: that is what
+	// records who let it in. One an identity provider, SCIM or a directory
+	// made — active from the start, its password kept there ("!oidc",
+	// "!scim", "!ldap"), with no approval to record — is set active again
+	// when its provider reinstates it; refusing that left such accounts
+	// suspended for good.
 	if state == UserActive {
 		var approved sql.NullInt64
-		if err := s.db.QueryRowContext(ctx, `SELECT approved_at FROM users WHERE id = ?`, id).
-			Scan(&approved); err == nil && !approved.Valid {
+		var hash string
+		if err := s.db.QueryRowContext(ctx, `SELECT approved_at, password_hash FROM users WHERE id = ?`, id).
+			Scan(&approved, &hash); err == nil && !approved.Valid && !strings.HasPrefix(hash, "!") {
 			return errors.New("this account was never approved; approve it instead")
 		}
 	}

internal/web/account.go

@@ -86,6 +86,9 @@ type AccountPages struct {
 	hasQuotas bool
 	// oidc is sign-in through an identity provider; nil when there is none.
 	oidc *oidcLogin
+	// directory is the label of the directory that checks passwords
+	// (auth.ldap); empty when there is none.
+	directory string
 	// audit records every change made through these pages; nil records none.
 	audit audit.Sink
 	// hasGuard is set when /admin/guard is registered, and guardLibrary
@@ -250,6 +253,10 @@ func (p *AccountPages) signup(w http.ResponseWriter, r *http.Request) {
 // --- sign in and out ---
 
 // ssoLabel is what the sign-in button names, or "" without one.
+// SetDirectory says passwords are also checked by a directory, under this
+// label, so the sign-in form takes a username as well as a mail.
+func (p *AccountPages) SetDirectory(label string) { p.directory = label }
+
 func (p *AccountPages) ssoLabel(l *Lang) string {
 	if p.oidc == nil {
 		return ""
@@ -265,13 +272,22 @@ type loginData struct {
 	Version string
 	// SSO is the identity provider's label, when sign-in through one is on.
 	SSO string
+	// Directory is the directory's label when passwords are checked by one
+	// (auth.ldap): the field then takes a username too, not only a mail.
+	Directory string
 	// CSRF is the form's token, matched by a cookie.
 	CSRF string
 }
 
+// loginPage is what every rendering of the sign-in form carries.
+func (p *AccountPages) loginPage(w http.ResponseWriter, r *http.Request, l *Lang, email string) loginData {
+	return loginData{SSO: p.ssoLabel(l), Directory: p.directory, Email: email, SignupOpen: p.accounts.SignupOpen(),
+		Forgot: p.resets != nil, Version: p.version, CSRF: setLoginToken(w, r)}
+}
+
 func (p *AccountPages) loginForm(w http.ResponseWriter, r *http.Request) {
 	noStore(w)
-	render(w, r, "login", loginData{SSO: p.ssoLabel(langOf(r)), SignupOpen: p.accounts.SignupOpen(), Forgot: p.resets != nil, Version: p.version, CSRF: setLoginToken(w, r)})
+	render(w, r, "login", p.loginPage(w, r, langOf(r), ""))
 }
 
 func (p *AccountPages) login(w http.ResponseWriter, r *http.Request) {
@@ -281,15 +297,16 @@ func (p *AccountPages) login(w http.ResponseWriter, r *http.Request) {
 	if !loginTokenOK(r) {
 		// Not this site's form: another one posting a visitor into an
 		// account of its choosing, or a form left open past its hour.
-		renderStatus(w, r, http.StatusBadRequest, "login", loginData{SSO: p.ssoLabel(l), Email: email, SignupOpen: p.accounts.SignupOpen(), Version: p.version,
-			Error: l.T("login.err.form_expired"), CSRF: setLoginToken(w, r)})
+		d := p.loginPage(w, r, l, email)
+		d.Error = l.T("login.err.form_expired")
+		renderStatus(w, r, http.StatusBadRequest, "login", d)
 		return
 	}
 
 	ctx := account.WithClient(r.Context(), clientAddr(r))
 	token, csrf, err := p.accounts.SignIn(ctx, email, r.FormValue("password"))
 	if err != nil {
-		d := loginData{SSO: p.ssoLabel(l), Email: email, SignupOpen: p.accounts.SignupOpen(), Forgot: p.resets != nil, Version: p.version, CSRF: setLoginToken(w, r)}
+		d := p.loginPage(w, r, l, email)
 		switch {
 		case errors.Is(err, account.ErrBadCredentials),
 			errors.Is(err, account.ErrNotApproved),
@@ -309,9 +326,9 @@ func (p *AccountPages) login(w http.ResponseWriter, r *http.Request) {
 	if p.oidc != nil && p.oidc.cfg.NoPasswords {
 		if _, u, err := p.accounts.Resolve(ctx, token); err != nil || !u.IsAdmin {
 			_ = p.accounts.SignOut(ctx, token)
-			renderStatus(w, r, http.StatusForbidden, "login", loginData{SSO: p.ssoLabel(l), Email: email,
-				SignupOpen: p.accounts.SignupOpen(), Version: p.version, CSRF: setLoginToken(w, r),
-				Error: l.T("login.err.passwords_off", p.ssoLabel(l))})
+			d := p.loginPage(w, r, l, email)
+			d.Error = l.T("login.err.passwords_off", p.ssoLabel(l))
+			renderStatus(w, r, http.StatusForbidden, "login", d)
 			return
 		}
 	}

internal/web/locales/en/account.json

@@ -19,6 +19,8 @@
   "login.title": "Sign in · llmgw",
   "login.heading": "Sign in",
   "login.email": "Email",
+  "login.username": "Username or email",
+  "login.with_directory": "Sign in with your %s account.",
   "login.password": "Password",
   "login.submit": "Sign in",
   "login.with_sso": "Sign in with %s",

internal/web/locales/en/node.json

@@ -62,6 +62,8 @@
   "node.memory": "Memory",
   "node.cores_f": "%s cores",
   "node.no_engines": "No engines are placed on this node.",
+  "node.version_differs": "the control plane runs %s",
+  "node.gpu.models": "Models",
   "node.engines": "Engines: %s",
   "node.engines_none": "Engines: none installed",
   "node.model_files": "%d model file(s) on disk",

internal/web/locales/tr/account.json

@@ -19,6 +19,8 @@
   "login.title": "Giriş yap · llmgw",
   "login.heading": "Giriş yapın",
   "login.email": "E-posta",
+  "login.username": "Kullanıcı adı veya e-posta",
+  "login.with_directory": "%s hesabınızla giriş yapın.",
   "login.password": "Parola",
   "login.submit": "Giriş yap",
   "login.with_sso": "%s ile giriş yap",

internal/web/locales/tr/node.json

@@ -61,6 +61,8 @@
   "node.memory": "Bellek",
   "node.cores_f": "%s çekirdek",
   "node.no_engines": "Bu düğüme yerleştirilmiş motor yok.",
+  "node.version_differs": "control plane %s çalıştırıyor",
+  "node.gpu.models": "Modeller",
   "node.engines": "Motorlar: %s",
   "node.engines_none": "Motorlar: hiçbiri kurulu değil",
   "node.model_files": "diskte %d model dosyası",

internal/web/node_page.go

@@ -6,6 +6,7 @@ import (
 	"html/template"
 	"log/slog"
 	"net/http"
+	"slices"
 	"sort"
 	"strings"
 	"time"
@@ -20,9 +21,13 @@ type nodePageData struct {
 	OnAdminListener bool
 	Node            adminapi.NodeView
 	Engines         []nodeEngine
-	Charts          *nodeCharts
-	HasHistory      bool
-	Back, Logs      string
+	// GPUModels are the models on each of the node's GPUs, by index.
+	GPUModels map[int][]string
+	// EngineVersions are the installed engines and their versions, sorted.
+	EngineVersions []string
+	Charts         *nodeCharts
+	HasHistory     bool
+	Back, Logs     string
 	// Self is this page's address, for the in-place refresh.
 	Self         string
 	Version, Now string
@@ -34,6 +39,7 @@ type nodePageData struct {
 // nodeEngine is one engine running on the node, and its share of it.
 type nodeEngine struct {
 	Replica, Model, Shard, State string
+	GPUs                         []int
 	CPUCores                     float64
 	MemoryBytes                  int64
 	// CPUShare is its cores as a percentage of the machine, for the bar.
@@ -128,7 +134,15 @@ func NodePage(src Source, history History, window time.Duration, onAdminListener
 					continue
 				}
 				e := nodeEngine{Replica: rep.ID, Model: rep.Model, Shard: sh.ID, State: sh.State,
-					CPUCores: sh.CPUCores, MemoryBytes: sh.MemoryBytes}
+					GPUs: sh.GPUs, CPUCores: sh.CPUCores, MemoryBytes: sh.MemoryBytes}
+				for _, g := range sh.GPUs {
+					if d.GPUModels == nil {
+						d.GPUModels = map[int][]string{}
+					}
+					if !slices.Contains(d.GPUModels[g], rep.Model) {
+						d.GPUModels[g] = append(d.GPUModels[g], rep.Model)
+					}
+				}
 				if l := d.Node.Load; l != nil && l.CPUs > 0 {
 					e.CPUShare = sh.CPUCores * 100 / float64(l.CPUs)
 				}
@@ -137,6 +151,10 @@ func NodePage(src Source, history History, window time.Duration, onAdminListener
 			}
 		}
 		sort.Slice(d.Engines, func(i, j int) bool { return d.Engines[i].CPUCores > d.Engines[j].CPUCores })
+		for e, v := range d.Node.EngineVersions {
+			d.EngineVersions = append(d.EngineVersions, e+" "+v)
+		}
+		sort.Strings(d.EngineVersions)
 
 		if history != nil {
 			d.Charts = loadNodeCharts(r.Context(), l, history, id, replicas, d.Node.GPUTotal, window, log)

internal/web/node_page_test.go

@@ -270,3 +270,48 @@ func TestTheClusterPageDrawsItsGPUsByClass(t *testing.T) {
 		}
 	}
 }
+
+// versionSource is hotSource with an agent a release behind, a model on its
+// second GPU, and engines with their versions.
+type versionSource struct{ hotSource }
+
+func (versionSource) Nodes() []adminapi.NodeView {
+	n := hotSource{}.Nodes()[0]
+	n.AgentVersion = "v0"
+	n.EngineVersions = map[string]string{"llamacpp": "b10991", "comfyui": "0.39.2"}
+	return []adminapi.NodeView{n}
+}
+
+func (versionSource) Replicas(string) []adminapi.ReplicaView {
+	r := fakeSource{}.Replicas("")
+	r[0].Shards[0].GPUs = []int{1}
+	return r
+}
+
+func TestANodesPageSaysWhatItRunsAndWhichModelIsOnEachGPU(t *testing.T) {
+	_, body := get(t, nodeMux(versionSource{}, nil), "http://x/admin/nodes/llmgw-node01", nil)
+	model := versionSource{}.Replicas("")[0].Model
+	for _, want := range []string{
+		// The page's control plane runs v1: an agent on v0 is marked.
+		`class="bad" title="the control plane runs v1">llmgw v0</span>`,
+		"Engines: comfyui 0.39.2, llamacpp b10991",
+	} {
+		if !strings.Contains(body, want) {
+			t.Errorf("the page lacks %q", want)
+		}
+	}
+	// GPU 1 holds the model, GPU 0 nothing.
+	rows := strings.Split(body, "<tr>")
+	var gpu0, gpu1 string
+	for _, row := range rows {
+		switch {
+		case strings.HasPrefix(row, "\n  <td class=\"num\">0</td>"):
+			gpu0 = row
+		case strings.HasPrefix(row, "\n  <td class=\"num\">1</td>"):
+			gpu1 = row
+		}
+	}
+	if !strings.Contains(gpu1, "<td>"+model+"</td>") || strings.Contains(gpu0, model) {
+		t.Errorf("GPU rows:\n0: %.300s\n1: %.300s", gpu0, gpu1)
+	}
+}

internal/web/oidc.go

@@ -148,8 +148,9 @@ func (p *AccountPages) oidcStart(w http.ResponseWriter, r *http.Request) {
 	pr, err := o.providerFor(r.Context())
 	if err != nil {
 		p.log.Warn("the identity provider did not answer", "issuer", o.cfg.Issuer, "err", err)
-		renderStatus(w, r, http.StatusBadGateway, "login", loginData{SignupOpen: p.accounts.SignupOpen(), Version: p.version,
-			CSRF: setLoginToken(w, r), SSO: o.cfg.Label, Error: langOf(r).T("sso.err.no_answer")})
+		d := p.loginPage(w, r, langOf(r), "")
+		d.Error = langOf(r).T("sso.err.no_answer")
+		renderStatus(w, r, http.StatusBadGateway, "login", d)
 		return
 	}
 	state, nonce, verifier := randomToken(), randomToken(), oauth2.GenerateVerifier()
@@ -172,8 +173,9 @@ func (p *AccountPages) oidcCallback(w http.ResponseWriter, r *http.Request) {
 		if errors.As(err, &ae) {
 			msg = ae.in(l)
 		}
-		renderStatus(w, r, http.StatusForbidden, "login", loginData{SignupOpen: p.accounts.SignupOpen(),
-			Version: p.version, CSRF: setLoginToken(w, r), SSO: o.cfg.Label, Error: msg})
+		d := p.loginPage(w, r, l, "")
+		d.Error = msg
+		renderStatus(w, r, http.StatusForbidden, "login", d)
 	}
 	c, err := r.Cookie(oidcCookie)
 	if err != nil {
@@ -297,7 +299,17 @@ func (p *AccountPages) oidcUser(ctx context.Context, claims map[string]any) (sto
 		return store.User{}, accountError{"sso.err.state", []any{email, string(u.State)}}
 	}
 
-	groups := claimStrings(claims[or(o.cfg.GroupsClaim, "groups")])
+	claim := or(o.cfg.GroupsClaim, "groups")
+	if groupsOverflowed(claims, claim) {
+		// Read as no groups, this would take an administrator's role away
+		// at their next sign-in: kept as they are instead, and said.
+		p.log.Error("the identity provider left this person's groups out of the token, as Entra ID does past 200 groups; "+
+			"their role and memberships are left as they were. In the app registration's token configuration, "+
+			"emit only the groups assigned to the application, or provision through SCIM",
+			"email", email, "claim", claim)
+		return u, nil
+	}
+	groups := claimStrings(claims[claim])
 	if len(o.cfg.Roles) > 0 {
 		role := ""
 		for _, r := range []string{store.RoleAdmin, store.RoleOperator, store.RoleBilling, store.RoleViewer} {
@@ -333,6 +345,22 @@ func (p *AccountPages) oidcUser(ctx context.Context, claims map[string]any) (sto
 	return u, nil
 }
 
+// groupsOverflowed is a token whose groups claim was too long to send:
+// Entra ID then names a source to fetch it from (_claim_names), or, in a
+// token from the implicit flow, says only hasgroups.
+func groupsOverflowed(claims map[string]any, claim string) bool {
+	if _, ok := claims[claim]; ok {
+		return false
+	}
+	if names, ok := claims["_claim_names"].(map[string]any); ok {
+		if _, ok := names[claim]; ok {
+			return true
+		}
+	}
+	has, _ := claims["hasgroups"].(bool)
+	return has
+}
+
 // oidcPlace finds org:NAME, team:ORG/TEAM, project:ORG/TEAM/PROJECT.
 func (p *AccountPages) oidcPlace(ctx context.Context, where string) (store.Scope, string, error) {
 	kind, path, _ := strings.Cut(where, ":")

internal/web/oidc_test.go

@@ -165,6 +165,15 @@ func TestSigningInThroughAProvider(t *testing.T) {
 		t.Error("no session cookie")
 	}
 
+	// Past 200 groups Entra ID sends none, only where to fetch them: the
+	// role stays rather than being taken away as if she had no groups.
+	idp.claims = map[string]any{"email": "ann@example.com", "email_verified": true,
+		"_claim_names":   map[string]any{"groups": "src1"},
+		"_claim_sources": map[string]any{"src1": map[string]any{"endpoint": "https://graph.windows.net/t/users/u/getMemberObjects"}}}
+	if rec := signIn(t, mux, idp, false); rec.Code != http.StatusSeeOther || st.roles[u.ID] != "operator" {
+		t.Errorf("an overflowed groups claim: %d, role %q", rec.Code, st.roles[u.ID])
+	}
+
 	// Refused: a state that is not this browser's, an unverified address,
 	// and nobody the gateway knows when it does not make accounts.
 	if rec := signIn(t, mux, idp, true); rec.Code != http.StatusForbidden {

internal/web/templates/account.html

@@ -38,7 +38,9 @@
 {{if .Error}}<p class="error">{{.Error}}</p>{{end}}
 <form method="post" autocomplete="on">
   <input type="hidden" name="csrf" value="{{.CSRF}}">
-  <label>{{t "login.email"}}<input type="email" name="email" value="{{.Email}}" required autocomplete="email"></label>
+  {{if .Directory}}<p class="sub">{{t "login.with_directory" .Directory}}</p>
+  <label>{{t "login.username"}}<input type="text" name="email" value="{{.Email}}" required autocomplete="username" autocapitalize="none" spellcheck="false"></label>
+  {{else}}<label>{{t "login.email"}}<input type="email" name="email" value="{{.Email}}" required autocomplete="email"></label>{{end}}
   <label>{{t "login.password"}}<input type="password" name="password" required autocomplete="current-password"></label>
   <button type="submit">{{t "login.submit"}}</button>
 </form>

internal/web/templates/node.html

@@ -11,6 +11,7 @@
   <h1>{{.Node.ID}}</h1>
   <p class="sub"><span class="state state-{{.Node.State}}">{{t (print "dash.state." .Node.State)}}</span>
     {{with .Node.OS}} · {{.}}{{end}}{{with .Node.MgmtAddr}} · {{.}}{{end}}
+    {{with .Node.AgentVersion}} · <span{{if ne . $.Version}} class="bad" title="{{t "node.version_differs" $.Version}}"{{end}}>llmgw {{.}}</span>{{end}}
     · {{if .Node.LastSeenAgo}}{{t "node.last_seen" .Node.LastSeenAgo}}{{else}}{{t "node.never_seen"}}{{end}} · <a href="{{.Logs}}">{{t "dash.nodes.logs"}}</a></p>
   {{if .OnAdminListener}}<nav><a href="{{.Back}}">{{t "node.nav.cluster"}}</a></nav>{{else}}{{template "nav" .Nav}}{{end}}
 </header>
@@ -39,7 +40,7 @@
 {{if .Unhealthy}}<p class="note bad">{{join .Unhealthy}}</p>{{end}}
 {{with .Load}}{{if .GPUs}}
 <h2>{{t "node.gpus"}}</h2>
-<table class="rows"><tr><th class="num">GPU</th><th>{{t "node.gpu.busy"}}</th><th>{{t "node.gpu.memory_used"}}</th><th class="num">{{t "node.gpu.temp"}}</th><th class="num">{{t "node.gpu.memory_temp"}}</th><th>{{t "node.gpu.power"}}</th><th class="num">{{t "node.gpu.clock"}}</th><th>{{t "node.gpu.held_back"}}</th></tr>
+<table class="rows"><tr><th class="num">GPU</th><th>{{t "node.gpu.busy"}}</th><th>{{t "node.gpu.memory_used"}}</th><th class="num">{{t "node.gpu.temp"}}</th><th class="num">{{t "node.gpu.memory_temp"}}</th><th>{{t "node.gpu.power"}}</th><th class="num">{{t "node.gpu.clock"}}</th><th>{{t "node.gpu.models"}}</th><th>{{t "node.gpu.held_back"}}</th></tr>
 {{range .GPUs}}<tr>
   <td class="num">{{.Index}}</td>
   {{/* From the last sample; a dash where the driver or an older agent did
@@ -52,6 +53,7 @@
   <td class="num">{{if .SMClockMHz}}{{num .SMClockMHz 0}} MHz{{else}}—{{end}}</td>
   {{/* The driver's reasons and the wear are named as the driver and the
        inventory name them, in English. */}}
+  <td>{{with index $.GPUModels .Index}}{{join .}}{{else}}<span class="note">—</span>{{end}}</td>
   <td>{{if .Throttle}}<span class="bad">{{join .Throttle}}</span>{{else}}<span class="note">{{t "node.gpu.nothing"}}</span>{{end}}{{if gt .ECCUncorrected 0.0}} <span class="bad">{{t "node.gpu.ecc" (num .ECCUncorrected 0)}}</span>{{end}}{{with .Wear}}{{with .Problems}}<br><span class="bad">{{join .}}</span>{{end}}{{if gt .PCIeReplays 0}} <span class="note">{{t "node.gpu.pcie" (count .PCIeReplays)}}</span>{{end}}{{end}}</td>
 </tr>{{end}}</table>
 {{end}}{{end}}
@@ -106,16 +108,17 @@
 
 <h2>{{t "node.runs"}}</h2>
 {{if .Engines}}
-<table class="rows"><tr><th>{{t "dash.col.model"}}</th><th>{{t "dash.col.replica"}}</th><th>{{t "dash.col.state"}}</th><th>CPU</th><th class="num">{{t "node.memory"}}</th></tr>
+<table class="rows"><tr><th>{{t "dash.col.model"}}</th><th>{{t "dash.col.replica"}}</th><th>{{t "dash.col.state"}}</th><th>GPU</th><th>CPU</th><th class="num">{{t "node.memory"}}</th></tr>
 {{range .Engines}}<tr>
   <td>{{.Model}}</td>
   <td><code>{{.Replica}}</code></td>
   <td class="state state-{{.State}}">{{t (print "dash.state." .State)}}</td>
+  <td>{{if .GPUs}}{{range $i, $g := .GPUs}}{{if $i}}, {{end}}{{$g}}{{end}}{{else}}—{{end}}</td>
   <td class="load">{{template "minibar" pctbar .CPUShare}}<span class="load-v">{{t "node.cores_f" (num .CPUCores 1)}}</span></td>
   <td class="num">{{if .MemoryBytes}}{{bytes .MemoryBytes}}{{else}}—{{end}}</td>
 </tr>{{end}}</table>
 {{else}}<p class="empty">{{t "node.no_engines"}}</p>{{end}}
-<p class="note">{{if .Node.Engines}}{{t "node.engines" (join .Node.Engines)}}{{else}}{{t "node.engines_none"}}{{end}}{{if .Node.Models}} · {{t "node.model_files" (len .Node.Models)}}{{end}}.</p>
+<p class="note">{{if .EngineVersions}}{{t "node.engines" (join .EngineVersions)}}{{else if .Node.Engines}}{{t "node.engines" (join .Node.Engines)}}{{else}}{{t "node.engines_none"}}{{end}}{{if .Node.Models}} · {{t "node.model_files" (len .Node.Models)}}{{end}}.</p>
 </div>
 
 {{if .HasHistory}}{{with .Charts}}