v0.15.0: changelog and README

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

.gitea/workflows/ci.yml

@@ -34,7 +34,11 @@ jobs:
         run: go vet ./...
 
       - name: Test with the race detector
-        run: go test -race ./...
+        # Under the job's limit, so a package that hangs panics with the
+        # test it was in and where every goroutine waited, rather than the
+        # job ending at its deadline with nothing said: from 2026-10-15 every
+        # run did, printing nothing after internal/release.
+        run: go test -race -timeout 12m ./...
 
       - name: Build
         run: go build ./...

.gitignore

@@ -9,3 +9,5 @@ coverage.out
 /terraform-provider-llmgw/terraform-provider-llmgw
 /terraform-provider-llmgw/examples/.terraform*
 /terraform-provider-llmgw/examples/terraform.tfstate*
+# What a development server run from the checkout writes beside it.
+/secrets/

BACKLOG.md

@@ -15,6 +15,52 @@ Conventions this codebase keeps, which every item here must keep too:
 - `go test ./...`, `go vet ./...` and `govulncheck ./...` pass. No release is
   cut and nothing is deployed to the PoC cluster without being told to.
 
+## Open, in order (2026-10-20)
+
+What is left, most needed first. The detail is in the item each names.
+
+**Now: what goes wrong where it is installed.**
+1. ~~**J8**~~: done 2026-10-20. `ctl guard judge check`, and init times the judge. Left of **J3**: its load test
+   against judges on the control plane, which was stopped half way. Judges on a 4-core node of their own refused
+   0/60 at 2 at once, 13/60 at 4 and 42/60 at 10.
+2. ~~**I2 P2**~~: done 2026-10-20: the guardrails both ways, a rate per caller and server, a breaker. A portal page
+   for the servers is left, with P3 (stdio, per-person OAuth).
+3. **K**: fixes from the fabric test the operator runs with docs/large-models.md.
+
+**Soon: the customer's experience, and HA.**
+4. ~~**C24**~~: done 2026-10-21. The store's, catalog's and mirror's errors, percent signs, footer times and stale
+   forms are in the page's language. Left: the /docs documents, and a language kept per person (cookie only today).
+5. **J4**: the model mirror on more than one machine, the last single point of an HA install. (M)
+6. **F5**: large imports limited by the embedding engine's slots. (M)
+
+**From Kong** (section L): L1 done; L9, then L2 and L3, then L4/L5. L6 and L7 are llmgw-80's.
+
+**Governance.**
+7. **C1**: a tenant-admin role, and forms hidden from viewers.
+8. **Audit before and after**: what a change changed, and changes made with the CLI recorded too.
+
+**When there is room.**
+9. **I1**: answers to similar questions from the cache. (M)
+10. **H1**: tried on the PoC.
+11. **C4**: Parquet, and a scheduled push for usage export.
+
+**When needed.**
+12. **H5**: licences in video and audio. 13. **H3**: replicas shared, the catalog side. 14. **J7**: nginx in front
+of several control planes. 15. **J6**: rate limits per process (a note).
+
+**Done since 2026-10-16:**
+- J1, J5, I3, I4, J2.
+- Active Directory and Entra ID, tried against a Samba DC, with the console's Sign-in screen.
+- J3's code.
+- K's code and docs/large-models.md.
+- I2 P1 and the cluster's own read-only MCP server (llmgw:admin).
+- The node console's Overview; a node's page showing its version and the models on each GPU.
+
+**Not code, the operator's to decide or do:**
+- Where Alertmanager sends (its receiver is "none"): with a judge down, nobody hears.
+- The fabric test for K.
+- The PoC's control planes run a dev build of main (v0.14.9-11-g4eb49d0); its nodes run v0.14.8.
+
 ---
 
 ## A. Security — do first
@@ -1429,6 +1475,14 @@ console show and add them.
 - Done 2026-10-15: llmgw's `/mcp` with picture, 3D, VLM and workflow tools,
   so a client needs no server of its own (asked by the game team); the
   shared ComfyUI replica (H3's placement side).
+- Done 2026-10-16 (v0.14.6, v0.14.7): setup lists every part with what it
+  is and its settings; three machines each run the control plane, with a
+  floating address (VRRP id from the address); a lost store primary costs no
+  request (3 s connect per member, keepalive and TCP_USER_TIMEOUT, Patroni
+  ttl 20, a standby leaves the lease alone a round after the store returns);
+  a new leader waits for its judge within the budget; a judge is asked again
+  on another machine at half its budget; init sizes the judge (2 GB, no
+  prompt cache, two cores, one judge on up to four cores on one machine).
 
 ## I. What other gateways do that llmgw does not (written 2026-10-16)
 
@@ -1442,7 +1496,13 @@ Azure/Anthropic/Bedrock/Vertex adapters, request hooks a customer plugs in,
 routing by what a request holds. Not taken: a realtime (WebSocket audio)
 API, prompt management, an `npx` install.
 
-### I1. Answers to similar questions, not only the same one (M)
+### I1. Answers to similar questions, not only the same one (M) — done 2026-10-25
+`keys cache --similar`, `router.cache.embedding_model`, `internal/router/similar.go`. Changed
+from the plan: the vectors are in the gateway's memory beside the exact answers, not a pgvector
+table. The exact cache keeps answer text out of the store on purpose (data.md), and a table of
+answers would undo that; it also works on SQLite. The usage row marks a similar hit as a hit
+(no column of its own). Open: the 40-rewordings and eval-suite runs on the PoC.
+The original item follows.
 The answer cache (`internal/router/cache.go`, C17) answers a request whose
 body is the same byte for byte. A help desk asked "how do I reset my
 password" in forty wordings asks the model forty times.
@@ -1466,6 +1526,14 @@ password" in forty wordings asks the model forty times.
   point of the run with it off.
 
 ### I2. MCP gateway: the customer's tool servers behind llmgw (L)
+P1 done 2026-10-20: HTTP servers registered (`ctl mcp`, `/api/v1/mcp/servers`,
+store `mcp_servers`), tools as `server.tool` in `/mcp`, `allowed_tools`
+grants in the tree (deny by default), credentials in the store or by
+reference, the caller named in `X-Llmgw-*` headers, time and size bounds,
+progress passed on, audit (Source `mcp`) and metrics. Left: P2, the
+guardrails both ways, a per-tool rate limit, a breaker, and a portal page;
+P3, stdio servers and per-person OAuth. The "done when" with Claude Code
+against a real server is still to try.
 `/mcp` serves llmgw's own tools. An agent inside a company also wants Jira,
 the file server, a database — each an MCP server, each today configured
 in every client with its own credentials and nobody seeing what was called.
@@ -1485,7 +1553,16 @@ in every client with its own credentials and nobody seeing what was called.
   through it, a key without the grant does not see the server's tools, and
   every call is in `ctl audit`.
 
-### I3. What llmgw adds to a request, measured (S)
+### I3. What llmgw adds to a request, measured (S) — done 2026-10-16
+`llmgw ctl bench overhead` (`internal/cli/bench_overhead.go`): an instant
+engine and the gateway in one process, four modes (direct, bare, auth, auth
+and guard patterns) at rising rates. On an Apple M5 the gateway added
+0.05–0.42 ms at p50 and at most 0.49 ms at p99, in the README.
+`TestTheGatewaysOverheadIsMeasured` runs a second of it in CI with no bound
+asserted. Left: the CPU figure is rough with everything in one process; a
+gateway in a process of its own would measure it properly. The original
+item follows.
+
 Bifrost's headline is "11 µs at 5k RPS" (its own figure, and its own pages
 disagree with it). llmgw has none, so any question about its overhead is
 answered with a guess.
@@ -1500,7 +1577,13 @@ answered with a guess.
 - Done when: the README states the overhead, with how it was measured, and
   CI runs the short version so a change that doubles it is seen.
 
-### I4. Several keys for one outside provider (S)
+### I4. Several keys for one outside provider (S) — done 2026-10-17
+`external.api_key_files` (`catalog.ExternalKey`: file, weight), one endpoint
+each in `control.externalEndpoints`, 401 retried on another key and
+`external_key_refused` notified (`router.keyRefused`). With one key a 401 is
+still the client's. `TestAKeyAtItsLimitOrRefusedIsPassedOverForAnother`. The
+original item follows.
+
 An external model has one `api_key_file` (`internal/control/control.go`,
 where external endpoints are made). A provider's limit is per key; one key
 answering 429 makes the whole fallback answer 429.
@@ -1513,9 +1596,198 @@ answering 429 makes the whole fallback answer 429.
 - Done when: with two keys and one returning 429, every request is answered
   by the other, and the console shows the one breaker open.
 
+## J. Found testing HA and single-machine installs (written 2026-10-16)
+
+Three VMs installed as HA from the setup screens, a fourth as a judge node,
+a fifth as a single 4-core, 8 GB machine; machines powered off under load.
+
+### J1. A restart refused for room another restart was giving back (S) — done 2026-10-16
+Not a wait for ever, as first written: on the 4-core VM the judge's restart
+was *refused* ("fits on no node even with this replica stopped") because the
+starter, restarted at the same time to reserve fewer cores, still held its
+old ones. Once refused, nothing tried again, so the judges kept their old
+figures. Two fixes in `internal/control/reconciler`:
+- a restart whose replacement fits once other restarts under way have gone
+  (queued, retiring or draining) waits for them, up to DrainGrace, instead
+  of being refused (`fitsAfterRestarts`, restart.go);
+- a drained replica counts a retiring one that still answers as serving, so
+  one restart after another no longer leaves the first's drained replica,
+  and the room it holds, waiting out its grace (reconciler.go).
+`TestARestartWaitsForRoomAnotherRestartWillGiveBack` is the 4-core case.
+
+### J2. A judge the leader adds is sized as init's is (S)
+When a policy names a judge the catalog lacks, the leader serves it itself
+(`internal/cli/judgekeeper.go` via `knownJudge.serveRequest` in
+`judges_known.go`, through modelserve). That judge reserves 3 GB (the
+weights, an 8192-token KV cache and a fixed gigabyte), no cores, and runs
+with llama.cpp's prompt cache, which grew one to 3.2 GB. init's judge
+(`guardModel` in `internal/cli/serverinit_cluster.go`) reserves 2 GB and
+its cores and runs with `--cache-ram 0`.
+- `modelserve` gives a guard model `--cache-ram 0`, and the keeper's
+  request the same memory and cores rule as `cpuPlan`.
+- Done when: a judge removed from the catalog comes back with the same
+  figures init would have given it.
+
+### J3. HA setup puts the judges on nodes of their own (S)
+Under load on a 4-core control plane that was also a node, 38 of 60
+requests were refused for want of a judgment. With the judges on a node of
+their own (4 cores, 32 GB, two cores each) the same test refused 2.
+- The HA setup offers a judge node or two: machines given like the store's,
+  added as nodes with a label (`llmgw.role: judge`), and the judge's
+  profile selects that label.
+- Done when: the setup installs the judges on the machines given for them,
+  and the load test above refuses no more than it did by hand.
+
+### J4. The model mirror on more than one machine (M)
+In an HA install the mirror stays on the first machine. While it is down,
+models the nodes already have keep serving and new ones wait for it.
+- A second mirror kept in step (each machine of the store, or one more), the
+  control planes and nodes given both, the first that answers used.
+- Done when: with the first machine powered off, a model added then is
+  fetched and served.
+
+### J5. An upgrade corrects an earlier install's judge (S) — done 2026-10-16
+Built in the judge keeper (`internal/cli/judgekeeper.go`, `refit`), on the
+leader: a judge whose entry is exactly what init before v0.14.7 wrote is
+given 2 GB, `--cache-ram 0` and cores by init's rule (`refitCores`: up to
+four for one judge, two each for more), through `modelsvc.SetProfiles`,
+and its ready replicas are restarted (after J1). Its replica count is left
+as it is. One that differs is left alone and said once in the log. The
+original item follows.
+
+v0.14.6 and earlier installed two judges reserving no cores or one, with
+2 or 4 GB and the prompt cache on. On fewer than eight cores every judgment
+missed its 5 s budget and every request was refused. v0.14.7's notes say
+how to fix it by hand (`ctl model resources`, then `ctl restart`).
+- On upgrade, or on `server init` run again, a judge whose profile is still
+  exactly what an earlier init wrote is moved to today's figures and
+  restarted (after J1). One changed by hand is left alone and named.
+- Done when: a v0.14.6 single-machine install upgraded to it judges within
+  budget without anything typed.
+
+### J6. Rate limits per process (note)
+Rate limits are counted in each control plane's memory. A failover starts
+every key's minute afresh. Documented in install.md; worth a shared count in
+the store only if a customer's limits must hold across a failover.
+
+### J7. nginx in front of several control planes (S)
+`deploy/proxy/Caddyfile` sends to whichever control plane answers `/leader`.
+nginx's open-source build has no active health checks, so its `upstream`
+block would send to standbys and to a lost leader alike.
+- nginx.conf: the floating address as its one upstream, and why; or the
+  `max_fails` passive approach with what it costs.
+
+### J8. A judge budget that fits the machine (S)
+A judgment takes 2.3 s on four cores and 4.2 on two, against the 5 s
+default: two cores are close to it, and a busy machine passes it.
+- At install, after the judge is up, time three judgments. If the slowest is
+  more than half the budget, say so in the plan's closing text with the
+  choices (more cores, a GPU judge, a longer `latency_budget`, `fail:
+  open`). The budget itself is not changed without being asked.
+- Done when: an install on a machine too slow for its judge says so before
+  its first refused request.
+
+## L. Taken from Kong's AI Gateway (written 2026-10-25)
+
+From a look at Kong AI Gateway 3.9–3.16 and 2.x. Kong's open source stops at
+3.9, and only six of its AI plugins are open (proxy, prompt template,
+decorator, regex prompt guard, request and response transformer); the rest
+is Enterprise or Konnect. So nothing of Kong is embedded: the two meet over
+plain OpenAI/Anthropic HTTP, and these are the ideas worth taking. Not taken:
+prompt compression (Kong ships it as a private LLMLingua sidecar, its benefit
+unshown), prompt templates and decorators (no one has asked), Konnect's
+entity model. MCP items are llmgw-80's (I2); they are listed for it.
+
+### L1. Traces in the OpenTelemetry GenAI conventions (S) — done 2026-10-25
+`gen_ai.*` on the `inference` and `upstream` spans (`internal/router/trace.go`):
+operation, request and response model, max_tokens, temperature, top_p, the
+session as conversation id, input/output/cached tokens, the engine as
+provider. Never prompt or answer text (`TestTracesSpeakTheGenAIConventionsAndHoldNoText`).
+Prometheus already counted tokens by direction, cached apart. Left: the MCP
+calls' spans as `gen_ai.tool.*`, once /mcp is traced at all.
+
+### L2. Redacted personal data put back in the answer (S) — done 2026-10-25
+`restore: true` (`guard.Rules.Restore`), `Verdict.Restore`, `internal/router/restore.go`
+ahead of the output filter. The original item follows.
+The guardrails rewrite personal data as `[KIND_n]` (`internal/guard/gate.go`,
+`placeholders`), the same value always the same placeholder. A model that
+answers "Dear [PERSON_1]" leaves it so. Kong's sanitizer restores.
+- Per policy, `restore: true`: the placeholders of one request kept in memory
+  for its life, and swapped back in the answer, streamed too (a placeholder
+  split across two chunks held until it is whole).
+- Only into the answer to the request that held them; never stored, never in
+  usage rows or traces.
+- Done when: "write to Ayşe Yılmaz at ayse@example.com" through an enforce
+  policy reaches the model as placeholders and the client reads the real name
+  and address back.
+
+### L3. A guardrail that is someone else's service (S) — done 2026-10-25
+`guard.Service` (`internal/guard/service.go`), policy `services`, asked
+before the judge on input and once on the whole answer. The original item follows.
+Customers may be required to use one: Azure Content Safety, Lakera, NeMo
+Guardrails, an in-house classifier. Kong's AI Custom Guardrail is a
+templated HTTP call with an expression for "block".
+- A detector kind `http`: URL, headers from secret files, a body template
+  with the text, a JSON path that says blocked and one that says why;
+  timeout and `fail` as the judge's. Checked on input, output or both.
+- Done when: a policy with an `http` detector against a test service blocks
+  what it says to, and a service that does not answer is treated as `fail`
+  says.
+
+### L4. Routing by latency and by cost, and failover by priority (M) — latency done 2026-10-25
+`endpoint.Registry.ObserveTTFT`, `scheduler.Latency()` in `chat`. Failover in tiers is
+what `fallbacks` already does (local, then an outside model when no room). Left:
+routing by cost between several outside providers. The original item follows.
+The scorers (`internal/router/scheduler/scorers.go`) know sessions, queues
+and KV cache. Kong also balances by measured latency, by tokens or cost
+used, and in priority tiers.
+- A `latency` scorer: a moving average of time to first token per endpoint.
+- Fallbacks in tiers: local replicas first, then an outside provider, with
+  the tier said in the usage row.
+- Done when: with two replicas of one model, one on a slower card, `latency`
+  sends most of the traffic to the faster and the slower still gets some.
+
+### L5. Prices by kind and by cache (S–M) — done 2026-10-25
+`cached_per_1k`, charged on the answer's `cached_tokens`. A budget in money was already
+there (`keys limits --budget`). Not done: cache-write and audio/video token prices —
+no engine llmgw runs reports them, and no provider it reaches is billed by them yet.
+The original item follows.
+`catalog` prices are per million input/output tokens and per picture/mesh.
+Kong prices cached reads and cache writes apart, and audio, image and video
+tokens apart, and limits by cost.
+- Price fields for cached input and by modality; the usage row's cost from
+  them; a budget limit in money as well as tokens (`keys limits --cost`).
+- Done when: a request with 90% cached prompt costs what the cached rate says.
+
+### L6. REST APIs as MCP tools (M) — for llmgw-80, beside I2
+Kong turns OpenAPI operations into MCP tools. A company with an internal
+REST API and no MCP server could offer it to agents through `/mcp`:
+`llmgw ctl mcp add hr --openapi https://hr.internal/openapi.json`, each
+operation a tool, under the same grants, audit, guardrails and rate as I2.
+
+### L7. MCP sign-in by the MCP spec (M) — for llmgw-80, part of I2 P3
+Kong's MCP OAuth2: llmgw's `/mcp` as an OAuth 2.1 resource server (protected
+resource metadata, tokens checked by JWKS against the company's identity
+provider), and a token exchanged for a short-lived one per upstream server
+rather than one long-lived secret.
+
+### L8. A2A (S, when asked)
+Agent-to-agent traffic through the gateway: agent cards' URLs rewritten to
+llmgw's, calls audited and traced. The standard is young; wait for a
+customer.
+
+### L9. llmgw behind Kong, written down (S) — done 2026-10-25
+docs/kong.md, tried with kong:3.9: plain route and ai-proxy, keys passed on;
+Kong 3.9 counts no completion tokens of a stream. The original item follows.
+A customer that already runs Kong puts llmgw behind it: `ai-proxy` with
+`provider: openai` and `upstream_url` at llmgw (3.9, open source), or a plain
+route. A page in docs/ with the configuration, tried against Kong 3.9 in a
+container, and what Kong's token accounting sees.
+
 ## D. Facts worth knowing before starting
 
-- Admin API: `127.0.0.1:9090`, no auth, `ctlClient` in `internal/cli/ctl.go`.
+- Admin API: `127.0.0.1:9090`, behind the token in
+  `/var/lib/llmgw/admin-token` (A5), `ctlClient` in `internal/cli/ctl.go`.
   Metrics on `:9091` with `/prometheus/targets` beside them.
 - Store: `internal/store`, migrations under `internal/store/migrations/{sqlite,postgres}`;
   the PoC runs PostgreSQL under Patroni.

CHANGELOG.md

@@ -18,12 +18,317 @@ is only released once its **exit criterion** is met, not when its code lands.
 | v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released |
 | v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released |
 | v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
-| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines) |
+| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines; v0.14.7: the judge sized for its machine; v0.14.8: sign-in with Active Directory, several keys for one outside provider; v0.14.9: sign-in set up from the console, a node's console that shows the machine) |
+| v0.15.0 | — | The company's MCP tools and VS Code through the gateway; guardrails that ask outside services and put redacted values back; answers to similar questions | released |
 | v1.0.0 | 5 | What phase 5 has left: precise KV routing proven on vLLM, the block cost picker, defragmentation across NVLink domains, prefill/decode disaggregation | planned |
 
 Until v1.0.0 the minor version is the breaking one: configuration formats and Go
 APIs change between minors without a deprecation period.
 
+## v0.15.0 — 2026-10-25
+
+The company's tools and editors through the gateway, and guardrails that
+can lean on others. Its own MCP servers sit behind llmgw's `/mcp` under the
+guardrails, a rate and a breaker. VS Code gets the gateway's models from an
+extension the gateway serves. A guardrail policy can ask an outside
+service, and can put redacted values back in the answer. Chat goes to the
+replica answering soonest. A similar question can get the answer already
+given. A cached prompt can be priced apart. Traces follow the OpenTelemetry
+GenAI conventions, and Kong is written down from trying it. Also an install
+guide with every screen, at `/install`.
+
+**Upgrading.**
+- Migrations 0049 (`mcp_servers`), 0050 (their rate and breaker) and 0051
+  (`projects.cache_similar`), applied when the control plane starts.
+- `llmgw server init` now refuses a machine that would serve models without
+  the memory for them: the starter model and the judges need 12 GB on one
+  machine, where 8 GB was said before. `--no-model`, `--no-guard-model` or
+  `--no-node` make it fit. A running cluster is not checked again.
+- Traces carry the GenAI conventions' `gen_ai.*` attributes beside the
+  `llmgw.*` ones, which stay; docs/install.md, "Tracing", lists them.
+- Nothing else changes until it is turned on: `services` and `restore` in a
+  guardrail policy, `keys cache --similar` with
+  `router.cache.embedding_model`, `cached_per_1k` in a price.
+
+### Added
+
+- **Answers to similar questions.** `llmgw keys cache --similar 0.95` also
+  gives a chat question the answer already given to one like it in the same
+  conversation, compared by `router.cache.embedding_model`, as
+  `x-llmgw-cache: similar` with the similarity. Held in memory beside the
+  exact answers; nothing written to the store but the setting.
+- **A cached prompt costs the cached rate.** `cached_per_1k` in a price
+  charges the prompt tokens an engine or provider read from its cache (the
+  answer's `cached_tokens`) at their own rate; without it they cost what any
+  prompt token does, as before.
+- **Chat goes to the replica answering soonest.** The router keeps each
+  replica's time to first token, smoothed over its last answers, and the
+  `chat` profile scores by it beside the queue and the KV cache: of two
+  replicas of one model, the one on the slower card or the busier host gets
+  less, and a replica not yet measured is tried.
+- **llmgw and Kong, written down** ([docs/kong.md](docs/kong.md)): Kong 3.9
+  in front of llmgw by a plain route or its `ai-proxy`, each team's llmgw
+  key passed on; Kong as a provider behind llmgw; and that Kong 3.9 counts
+  no completion tokens of a streamed answer, so bill from llmgw's usage.
+- **A guardrail that is someone else's service.** A policy's `services`
+  asks Azure Content Safety, Lakera, NeMo or a company's own classifier over
+  HTTP: a body template, keys from files, dotted paths to where the answer
+  says unsafe and why, on input, output or both, block or flag, and the
+  policy's `fail` when it does not answer.
+- **Redacted values put back in the answer.** `restore: true` on a guardrail
+  policy's input: the model reads `[EMAIL_1]`, the client reads its own
+  address in the reply, streamed or not. Only the request's own values, held
+  in memory for its life; the output rules read the answer as the client
+  will.
+- **Traces in the OpenTelemetry GenAI conventions.** An inference request
+  and each attempt at an engine carry `gen_ai.operation.name`, the requested
+  and the answering model, `max_tokens`, `temperature` and `top_p` when set,
+  the session as `gen_ai.conversation.id`, and the input, output and cached
+  tokens, and an attempt the engine that answered as `gen_ai.provider.name`.
+  Tools that read the conventions show llmgw's calls as model calls with no
+  dashboard of their own. No prompt or answer text is ever in a span.
+- **llmgw for VS Code.** An extension of llmgw's own, served by every gateway
+  at `/setup/vscode/llmgw.vsix` with that gateway's address in it, made from
+  the binary itself: plain JavaScript, no dependencies, no build, and so no
+  marketplace needed at a site with no internet. It offers the gateway's chat
+  models to VS Code's language-model API — the chat's model picker, agent
+  mode, and every extension that asks VS Code for a model — with the sizes
+  and the tool and vision capabilities the catalog says, streamed, with tool
+  calls and their results both ways. **VS Code** is a tool on the account
+  page's **Connect your editor**: a key, the one-line install, and an **Open
+  in VS Code** link that hands the key to the extension, which keeps it in
+  VS Code's secret storage. **llmgw: Sign In** opens that page with a state
+  of its own; a link without it is confirmed first, the gateway named.
+- **An install guide with every screen, at `/install`.** From three bare
+  Ubuntu machines to a first answer: what each machine needs and how to
+  check it, the release fetched and checked, the setup screens one by one,
+  the system's state, and a first request with a key from the account page.
+  The screens are a real three-machine install, recorded and kept as
+  terminal frames — sharp at any size and a few kilobytes — with each
+  session played back on request; the administrator's password and every
+  key were masked before it was kept. `/about` says more of what llmgw does
+  (documents, guardrails, sign-in, high availability, pictures and 3D), and
+  its install steps are the ones of today, leading to the guide.
+- **The company's MCP tools under the guardrails, a rate and a breaker.**
+  - **The answer:** the caller's policies read a tool's answer as the prompt it becomes. An instruction planted in a
+    ticket is withheld as a pasted one would be, and personal data is redacted.
+  - **The arguments:** they are read before they leave, so a secret pasted into one does not reach the company system.
+  - **Rate:** `ctl mcp add --rate N` allows each caller N calls a minute to a server.
+  - **Breaker:** five failures in a row take a server out for a minute, its tools off the list.
+
+- **How fast the judge is, measured.** `llmgw ctl guard judge check` (and `POST /api/v1/guard/judges/check`) works
+  out whether the judge keeps up with requests. It times three judgments of a 280-token prompt, then a burst at once,
+  and sets both against the policies' wait. It says ok, tight or slow, about how many requests a second can be judged,
+  and what to do: more cores, more judges, a GPU judge, a longer wait, or `fail: open`. `server init` runs it once the
+  judge is up. If the machine is too slow it says so before the next steps, not after the first refused request.
+
+- **The company's own MCP servers through llmgw's `/mcp`.** `llmgw ctl mcp add jira --url … --secret-file
+  Authorization=FILE` registers one; `/mcp` then lists its tools as `jira.search` beside llmgw's own. Only keys granted
+  them see them: `llmgw keys limits --tools 'jira.*'`, patterns at any level of the tenancy tree. None granted anywhere
+  is none. The server's credentials stay on the control planes, kept in the store sealed or read by reference. The
+  server is told the calling key, person and project. Each call is bounded in time and size, and its progress passes
+  on to a streaming client. Every call is in the audit log with its arguments' size and digest, never their text. The
+  admin API has `/api/v1/mcp/servers`.
+- **The cluster as a read-only MCP server.** Registered as `llmgw ctl mcp add llmgw --url llmgw:admin` and granted
+  as `llmgw.*`, its tools say what `llmgw ctl` does:
+  - `cluster_status`, `list_nodes`, `list_models`, `list_replicas`, `explain_placement`, `node_logs`;
+  - nothing that changes the cluster.
+
+  It is answered in the control plane's own process, so there is no token to keep and every control plane serves it.
+  The admin API serves the same at `POST /api/v1/mcp`. `ctl mcp add --bearer-file` keeps a token, sent as
+  `Authorization: Bearer`.
+
+- **Large models across cards and nodes.** For a model of experts on more than one card, the plan now adds vLLM's
+  `--enable-expert-parallel`. It reads the expert count from `config.json` under any of its names, and sizes a model
+  that also reads pictures by its `text_config`. Several nodes are offered in any number, not only powers of two. Each
+  way now says what it needs of a card, part by part (weights, context, engine), and of all its cards together. This
+  shows in `ctl model plan`, in the console's Serve a model, and in a serve job's first step. The serve form checks that
+  a longer context still fits. `ctl model serve --env KEY=VALUE` puts engine variables such as `NCCL_DEBUG=INFO` on
+  every node. docs/large-models.md has the rules, the commands, and a checklist for bringing a model up on a fabric.
+
+- **Machines of the judges' own in a three-machine install.** The setup screen asks for up to two judge machines after
+  the store's two; `server init` takes `--judge-node user@host` for each. Each is added as a node labelled
+  `llmgw.role: judge`, and the judge's profile selects it, so the judges run there alone. Under load on a 4-core
+  control plane that was also a node, 38 of 60 requests were refused for want of a judgment. With the judges on a node
+  of their own, 2 were. A node with an `llmgw.role` label takes only profiles whose node selector asks for that role,
+  so no other model is placed on the judges' machine.
+
+### Fixed
+
+- **A machine too small for the models it would serve is said so first.**
+  The control plane that is also a node, serving the starter model and two
+  judges, needs about 10.5 GB. On 8 GB the install finished every step and
+  then had no room for the starter. Now the setup's checks fail its memory
+  with what it would serve, and `server init` refuses before it installs
+  anything, naming `--no-model`, `--no-guard-model` and `--no-node`.
+
+- **Turkish pages are Turkish to the end.**
+  - **Errors:** those from the store, catalog and mirror that a page shows ("not found", "not empty", "already in the
+    catalog", the settings changed meanwhile, and others) are now in the page's language.
+  - **Percent:** a percentage is written the language's way: `%93` in Turkish, `93%` in English.
+  - **Footer:** the time at the foot of a page is in the language's date form.
+  - **Stale forms:** they are refused with a sentence in that language rather than "bad request".
+
+- **`llmgw keys limits` no longer clears the limits it was not given.** Setting one limit at a scope wrote the whole
+  row, so `--tools 'jira.*'` alone took away that scope's rate, budget and model list. Only what is given changes now.
+  `--replace` writes every limit, as the console's limits form does.
+
+- **The plan refused tensor parallelism past a model's KV heads.** Qwen3-235B, with four KV heads, had no way to run
+  on eight cards, though vLLM gives each card a copy of a head. A width that is a multiple of the KV heads is now
+  offered, as is one that divides them.
+- **A big model on a slow mirror was failed while it downloaded.** The startup timeout counted the download, so
+  hundreds of gigabytes at a few hundred MB/s ran past twenty minutes and the replica was started again from nothing.
+  The timeout now runs from when a node's download last moved.
+
+- **The judge init installs is no longer replaced by the leader's own.** Just after the control plane started, the
+  leader saw a policy naming a judge that init was still fetching, and served one itself. Its entry replaced init's, so
+  a second judge ran with the leader's sizing, and with judge machines it ran off them. The leader now waits ten minutes
+  after it starts before it serves a judge itself. If someone adds one while it fetches, theirs stays.
+- **A judge the leader serves by itself is sized as init's is.** When a guard policy names a judge the catalog lacks,
+  the leader serves it. That judge reserved 3 GB and no cores, and ran llama.cpp's prompt cache, which grew one to
+  3.2 GB. Without cores of its own, a judgment on a busy node missed its 5 s budget, and `fail: closed` refused the
+  request. It now gets what `server init` gives its judges: 2 GB, two cores each (fewer on a small node), and no
+  prompt cache. Every judge served through `modelserve` runs with `--cache-ram 0`.
+
+## v0.14.9 — 2026-10-19
+
+Sign-in set up from the console, with Active Directory tried before it is
+kept. A node's console now shows what the machine is doing, and a node's web
+page shows what it runs. Also fixes found trying Active Directory against a
+Samba domain controller on the test cluster. Sign-in through it, nested
+groups and a disabled account suspended by the sync all worked there.
+
+**Upgrading.**
+- No migration.
+- `auth.ldap` and `auth.oidc` are in the cluster's settings, as before. A
+  block written into `server.yaml` on a cluster with a store was never
+  read; the log said so. Put it in with `llmgw ctl config edit`, or set it up
+  again from the console's Sign-in screen.
+- A Samba DC's own autogenerated certificate is refused (a negative serial
+  and no SAN). Give it one from a CA.
+
+### Added
+
+- **Sign-in set up from the console.** Under People → Sign-in, `sudo llmgw` shows how people get in: single sign-on,
+  Active Directory and SCIM, each on or off. It also shows the directory's last sync. **E** sets up Entra ID from the
+  tenant and client IDs. **D** sets up Active Directory, and the directory is tried before it is kept: each server
+  reached and bound, how many people may sign in, how many are in each group. Nothing is kept if a step fails. The
+  password, client secret and SCIM token are kept in the store as `store:auth/…`, sealed by the master key. The
+  domain's CA goes into the settings (`auth.ldap.ca`), so no file has to be put on each control plane. The admin API
+  has the same: `GET /api/v1/auth`, `POST /api/v1/auth/ldap/test` and `PUT /api/v1/auth/secrets/{name}`.
+- **A node's page says what it runs.** It now shows the agent's version, marked when the control plane runs another,
+  and the engines with their versions. Each GPU row now names the models loaded on that card, and each engine row its
+  GPUs.
+- **`sudo llmgw` on a node opens on what the machine is doing.** A new Overview screen shows the agent's state and
+  version, each control plane it dials and whether it answers, every GPU with the models on it, the engines running
+  here, the free space for weights and engines, and the agent's warnings of the last hour. It is read on the node
+  itself, every three seconds, so it needs no admin token and works while the control plane is out of reach.
+
+### Fixed
+
+- **A broken `auth.ldap` no longer stops every control plane.** It is the cluster's setting. A CA file that one
+  machine could not use made `llmgw server` exit on every control plane, and no `ctl config` could then put it right.
+  Directory sign-in is now turned off on that control plane, with an error in its log. Local accounts still sign in.
+- **A second `llmgw server` on a machine stops before it competes to lead.** `llmgw server check` (no such command)
+  ran a whole server. Its ports were taken, yet it went on to hold the leader's lease for three hours while the real
+  control planes stood by. `server` now refuses words it does not know and takes its ports before anything else.
+- **The single sign-on and Active Directory guides said `server.yaml`.** On a cluster with a store, `auth` is in the
+  cluster's settings and the file's block is not read. They now say `llmgw ctl config edit`, and what certificate a
+  domain controller needs.
+
+## v0.14.8 — 2026-10-18
+
+Sign-in with an on-premises Active Directory, and Entra ID made safe for
+people in many groups. Also several keys for one outside provider, and a
+measure of what the gateway adds to a request.
+
+**Upgrading.**
+- Migration 0048 adds the `user_directory` table, applied when the control
+  plane starts.
+- Nothing changes until `auth.ldap` is set; docs/install.md, "Active
+  Directory", has an example. It has been tested against a fake directory,
+  not yet against a real domain controller.
+- A judge installed by an earlier `server init` is refitted by the leader
+  by itself (below); the steps given for v0.14.7 are no longer needed.
+
+### Added
+
+- **Sign-in with Active Directory** (`auth.ldap`). People sign in on the gateway's form with their AD username or mail
+  and password, checked against the domain controllers over LDAPS or StartTLS. Groups, nested ones included, give
+  staff roles and places in the tree, as with OIDC. Accounts are tied to the `objectGUID` and made at first sign-in
+  when `create_accounts` is on. Every 15 minutes the leader reads the directory. People disabled or deleted there are
+  suspended and reinstated when they come back; roles and memberships follow their groups. A sync that would suspend
+  more than half the accounts is refused. See docs/install.md, "Active Directory".
+
+- **Several keys for one outside provider.** An external model takes `api_key_files`, each with an optional weight, in
+  place of `api_key_file`. Each key is an endpoint of its own (`external:MODEL#1`, `#2`), so a key answering 429 is
+  passed over while its breaker is open and the request goes to another. A key answering 401 is passed over too, and
+  `external_key_refused` is sent, naming the endpoint, never the key. Usage records which key answered by that name.
+
+- **`llmgw ctl bench overhead`**: what the gateway adds to a request, measured against an engine that answers at
+  once. Gateway and engine run in one process on loopback, so no cluster is needed. The load is sent straight to the
+  engine and through the gateway: bare, with authentication and accounting, and with the guardrails' pattern checks.
+  It reports p50 and p99 latency, what the gateway added, and a rough CPU per thousand requests. On an Apple M5 the
+  gateway added 0.05–0.42 ms at p50 and at most 0.49 ms at p99 (README).
+
+### Fixed
+
+- **Entra ID's groups past 200 no longer take a role away.** For someone in more than 200 groups Entra sends none in
+  the token, only a pointer to Graph. That was read as no groups, so an administrator lost their role at the next
+  sign-in. Their role and memberships are now left as they were, and an error is logged saying what to change in the app
+  registration. docs/install.md has an Entra ID section.
+- **An account made by SCIM or OIDC could not be reinstated.** Setting it active was refused as "never approved".
+- **An upgrade refits the judge an earlier `server init` installed.** Upgrading to v0.14.7 left such a judge as it was:
+  no cores or one, 2 or 4 GB, the prompt cache on. On fewer than eight cores every request was refused until it was
+  fixed by hand. Now the leader finds a judge whose entry is exactly what an earlier init wrote and gives it 2 GB, no
+  prompt cache and cores by today's rule (up to four for one judge, two each for more). It then restarts the judge's
+  replicas onto that. The number of replicas is left as it is. A judge whose entry was changed by hand is left alone,
+  and the log says so once.
+- **A restart is no longer refused for room another restart is giving back.** With every core of a node reserved, two
+  models restarted together, one to need more and one less (as after `ctl model resources` on a judge and a starter),
+  had the first refused: "fits on no node even with this replica stopped", though the second was about to give back
+  its cores. A restart now waits for other restarts under way, up to the drain grace, before it is refused. A drained
+  replica also counts a replica being restarted that still answers as serving, so replicas restarted one after
+  another no longer wait out the five-minute grace each.
+
+## v0.14.7 — 2026-10-16
+
+The judge installed by `server init` sized for what it does. On a machine
+of fewer than eight cores v0.14.6 left every judgment past its budget, so
+`fail: closed` refused every request. A single-machine install of v0.14.7
+on a 4-core VM passed 20 of 20, and on 8 cores 20 of 20 under load.
+
+**Upgrading.**
+- No migration.
+- **Upgrading does not change a judge already in the catalog.** v0.14.6
+  and earlier installed two judges, reserving no cores or one, and 2 or
+  4 GB. To give them what they need:
+  `llmgw ctl model resources qwen3guard-0.6b --memory 2 --cores 2`, then
+  `llmgw ctl restart --model qwen3guard-0.6b`. The two then need four cores
+  on a single machine; below that, a longer `latency_budget` or
+  `fail: open`.
+- On a node whose cores are all reserved, `ctl restart --model` cannot
+  start the new replica beside the old one and waits. Stop the old engine
+  process, and the replica comes back with the new figures.
+
+### Fixed
+
+- **Both judges fit an 8 GB machine again.** v0.14.6 had `server init` reserve 4 GB a judge replica. That figure came
+  from one replica at 3.2 GB after a day of traffic. The growth was llama.cpp's prompt cache, up to 8 GB by default,
+  which a judge never reuses: each text it reads is different. On a 4-core, 8 GB machine the second judge then fitted
+  nowhere ("has 7 GB of memory with 1 free, needs 4"). The judge now runs with `--cache-ram 0`, uses about 1.6 GB and
+  stays there, and reserves 2.
+- **One machine runs one judge, on up to four cores.** A second replica on the same machine shares its cores and is
+  lost with it, so it bought nothing but half the speed of each. `server init` on a single machine now serves one judge
+  on up to four cores (2.3 s a judgment where two on two cores took 4.2). Where there will be other nodes, it serves two
+  on two cores each, as in HA setups. `--judge-replicas` and the setup's Copies still choose.
+- **A judge gets the two cores it needs.** v0.14.6 gave each judge replica one core on machines under eight. A judgment
+  of about 280 tokens takes 8.0 s on one core, 4.2 s on two and 2.3 s on four, against the 5 s budget, so on a 4-core
+  machine every judgment timed out and every request was refused. Where there are two judges, each now reserves two
+  cores (one only below four CPUs), and the one judge of a single machine up to four; the starter has what is left, or
+  shares when nothing is. A judge on two cores stays close to its budget: give it more cores or a GPU, or raise
+  `latency_budget`, where requests must not wait.
+
 ## v0.14.6 — 2026-10-16
 
 Setup that says what each part of an install is and lets every one be

README.md

@@ -40,7 +40,8 @@ Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
 | v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released |
 | v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released |
 | v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
-| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines) |
+| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines; v0.14.7: the judge sized for its machine; v0.14.8: sign-in with Active Directory, several keys for one outside provider; v0.14.9: sign-in set up from the console, a node's console that shows the machine) |
+| v0.15.0 | — | The company's MCP tools and VS Code through the gateway; guardrails that ask outside services and put redacted values back; answers to similar questions | released |
 | v1.0.0 | 5 | What phase 5 has left: precise KV routing proven on vLLM, the block cost picker, defragmentation across NVLink domains, prefill/decode disaggregation | planned |
 
 As of v0.6.0 a cluster runs unattended: agents report their hardware, the
@@ -183,6 +184,8 @@ Every page is also in the binary: `llmgw docs`, or **Manual** in the console.
   cluster that exists
 - [docs/kubernetes.md](docs/kubernetes.md) — the control plane on
   Kubernetes with the Helm chart, and why GPU nodes stay outside it
+- [docs/kong.md](docs/kong.md) — llmgw behind Kong Gateway, or Kong behind
+  llmgw: the configuration, what Kong counts, and which to choose
 - [docs/fabrics.md](docs/fabrics.md) — InfiniBand fabrics and NVLink: what
   llmgw needs of them, checking a node with `llmgw doctor`, and what to look at
   when a replica spanning nodes will not run
@@ -257,6 +260,33 @@ Add `--json` to any command for machine-readable output.
 says nothing about which of six constraints bit, so it prints every candidate
 considered and the reason it was rejected.
 
+## What llmgw adds to a request
+
+Measured with `llmgw ctl bench overhead`. It starts an engine that answers
+at once and a gateway in front of it, in one process on loopback. It sends
+the same load straight to the engine and through the gateway, and reports
+what the gateway added. Run on an Apple M5 (10 CPUs, load about 2) on
+2026-10-16, 10 s a step, 72,000 requests with none failed:
+
+| Through the gateway | added at p50 | added at p99 |
+|---|---|---|
+| bare | 0.06–0.24 ms | 0.10–0.33 ms |
+| with authentication and usage accounting | 0.05–0.32 ms | 0.09–0.37 ms |
+| with the guardrails' pattern checks as well | 0.10–0.42 ms | 0.15–0.49 ms |
+
+The range is across 100 to 2,000 requests a second. The overhead was
+*lowest* at 2,000, most likely because a busier processor keeps its cores
+awake. The guardrails' judge model is not in these figures: a judgment
+takes seconds on CPUs (see [install.md](docs/install.md)) and is a model's
+time, not the gateway's.
+
+The command also prints the gateway's CPU per thousand requests. Gateway,
+engine and load share one process there, so that figure is rough (0.04 to
+0.65 s per thousand in the run above); the latency is the one to quote.
+Figures from a VM on a busy host move with the host: run it where you will
+deploy, and keep the load it prints beside the result. CI runs a one-second
+version on every change (`TestTheGatewaysOverheadIsMeasured`).
+
 ## Running the router
 
 ```sh
@@ -284,8 +314,9 @@ within 1.24x of idle while 500 concurrent batch requests run, against 58x with
 admission off.
 
 Routing profiles select how endpoints are scored: `chat` favours session
-affinity, `batch` favours the shortest queue, `long-context` favours free KV
-cache. Pick one per model in the config, or per request with the
+affinity and the replica that has been answering soonest (its time to first
+token, smoothed), `batch` favours the shortest queue, `long-context` favours
+free KV cache. Pick one per model in the config, or per request with the
 `x-llmgw-profile` header.
 
 ## sim-engine

docs/admin-api.openapi.json

@@ -23,6 +23,9 @@
     {
       "name": "audit"
     },
+    {
+      "name": "auth"
+    },
     {
       "name": "canaries"
     },
@@ -62,6 +65,9 @@
     {
       "name": "keys"
     },
+    {
+      "name": "mcp"
+    },
     {
       "name": "metrics"
     },
@@ -226,6 +232,129 @@
         }
       }
     },
+    "/api/v1/auth": {
+      "get": {
+        "operationId": "getAuth",
+        "summary": "How people sign in: OIDC, a directory, SCIM, and the directory's last sync",
+        "tags": [
+          "auth"
+        ],
+        "responses": {
+          "200": {
+            "description": "OK",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/AuthView"
+                }
+              }
+            }
+          },
+          "401": {
+            "$ref": "#/components/responses/Unauthorized"
+          },
+          "default": {
+            "$ref": "#/components/responses/Error"
+          }
+        }
+      }
+    },
+    "/api/v1/auth/ldap/test": {
+      "post": {
+        "operationId": "postAuthLdapTest",
+        "summary": "Try an auth.ldap block before it is kept: servers, the service account, people and groups",
+        "tags": [
+          "auth"
+        ],
+        "parameters": [
+          {
+            "$ref": "#/components/parameters/Actor"
+          }
+        ],
+        "requestBody": {
+          "required": true,
+          "content": {
+            "application/json": {
+              "schema": {
+                "$ref": "#/components/schemas/LDAPTestRequest"
+              }
+            }
+          }
+        },
+        "responses": {
+          "200": {
+            "description": "OK",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/LDAPTestView"
+                }
+              }
+            }
+          },
+          "401": {
+            "$ref": "#/components/responses/Unauthorized"
+          },
+          "default": {
+            "$ref": "#/components/responses/Error"
+          }
+        }
+      }
+    },
+    "/api/v1/auth/secrets/{name}": {
+      "put": {
+        "operationId": "putAuthSecretsName",
+        "summary": "Keep ldap-bind, oidc-client-secret or scim-token in the store, for settings to name as store:auth/NAME",
+        "tags": [
+          "auth"
+        ],
+        "parameters": [
+          {
+            "name": "name",
+            "in": "path",
+            "required": true,
+            "description": "ldap-bind, oidc-client-secret or scim-token.",
+            "schema": {
+              "type": "string"
+            }
+          },
+          {
+            "$ref": "#/components/parameters/Actor"
+          }
+        ],
+        "requestBody": {
+          "required": true,
+          "content": {
+            "application/json": {
+              "schema": {
+                "$ref": "#/components/schemas/SecretRequest"
+              }
+            }
+          }
+        },
+        "responses": {
+          "200": {
+            "description": "OK",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "type": "object",
+                  "additionalProperties": {
+                    "type": "string"
+                  }
+                }
+              }
+            }
+          },
+          "401": {
+            "$ref": "#/components/responses/Unauthorized"
+          },
+          "default": {
+            "$ref": "#/components/responses/Error"
+          }
+        }
+      }
+    },
     "/api/v1/canaries": {
       "get": {
         "operationId": "getCanaries",
@@ -1155,6 +1284,48 @@
         }
       }
     },
+    "/api/v1/guard/judges/check": {
+      "post": {
+        "operationId": "postGuardJudgesCheck",
+        "summary": "How a judge keeps up: three judgments in turn, then a burst at once, against the budget",
+        "tags": [
+          "guard"
+        ],
+        "parameters": [
+          {
+            "$ref": "#/components/parameters/Actor"
+          }
+        ],
+        "requestBody": {
+          "required": true,
+          "content": {
+            "application/json": {
+              "schema": {
+                "$ref": "#/components/schemas/JudgeCheckRequest"
+              }
+            }
+          }
+        },
+        "responses": {
+          "200": {
+            "description": "OK",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/JudgeCheckView"
+                }
+              }
+            }
+          },
+          "401": {
+            "$ref": "#/components/responses/Unauthorized"
+          },
+          "default": {
+            "$ref": "#/components/responses/Error"
+          }
+        }
+      }
+    },
     "/api/v1/jobs": {
       "get": {
         "operationId": "getJobs",
@@ -1810,6 +1981,215 @@
         }
       }
     },
+    "/api/v1/mcp": {
+      "post": {
+        "operationId": "postMcp",
+        "summary": "This cluster as a read-only MCP server: status, nodes, models, replicas, placement, logs",
+        "description": "JSON-RPC: initialize, tools/list, tools/call. Registered with the gateway as llmgw:admin.",
+        "tags": [
+          "mcp"
+        ],
+        "parameters": [
+          {
+            "$ref": "#/components/parameters/Actor"
+          }
+        ],
+        "requestBody": {
+          "required": true,
+          "content": {
+            "application/json": {
+              "schema": {
+                "type": "object",
+                "additionalProperties": {}
+              }
+            }
+          }
+        },
+        "responses": {
+          "200": {
+            "description": "OK",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "type": "object",
+                  "additionalProperties": {}
+                }
+              }
+            }
+          },
+          "401": {
+            "$ref": "#/components/responses/Unauthorized"
+          },
+          "default": {
+            "$ref": "#/components/responses/Error"
+          }
+        }
+      }
+    },
+    "/api/v1/mcp/servers": {
+      "get": {
+        "operationId": "getMcpServers",
+        "summary": "The company's MCP servers /mcp serves, and how each is reached",
+        "tags": [
+          "mcp"
+        ],
+        "responses": {
+          "200": {
+            "description": "OK",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "type": "array",
+                  "items": {
+                    "$ref": "#/components/schemas/MCPServerView"
+                  }
+                }
+              }
+            }
+          },
+          "401": {
+            "$ref": "#/components/responses/Unauthorized"
+          },
+          "default": {
+            "$ref": "#/components/responses/Error"
+          }
+        }
+      }
+    },
+    "/api/v1/mcp/servers/{name}": {
+      "put": {
+        "operationId": "putMcpServersName",
+        "summary": "Add an MCP server, or change one: its address, headers and the secrets they carry",
+        "tags": [
+          "mcp"
+        ],
+        "parameters": [
+          {
+            "name": "name",
+            "in": "path",
+            "required": true,
+            "description": "The MCP server's name: its tools are NAME.tool in /mcp.",
+            "schema": {
+              "type": "string"
+            }
+          },
+          {
+            "$ref": "#/components/parameters/Actor"
+          }
+        ],
+        "requestBody": {
+          "required": true,
+          "content": {
+            "application/json": {
+              "schema": {
+                "$ref": "#/components/schemas/MCPServerRequest"
+              }
+            }
+          }
+        },
+        "responses": {
+          "200": {
+            "description": "OK",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/MCPServerView"
+                }
+              }
+            }
+          },
+          "401": {
+            "$ref": "#/components/responses/Unauthorized"
+          },
+          "default": {
+            "$ref": "#/components/responses/Error"
+          }
+        }
+      },
+      "delete": {
+        "operationId": "deleteMcpServersName",
+        "summary": "Remove an MCP server: its tools leave /mcp",
+        "tags": [
+          "mcp"
+        ],
+        "parameters": [
+          {
+            "name": "name",
+            "in": "path",
+            "required": true,
+            "description": "The MCP server's name: its tools are NAME.tool in /mcp.",
+            "schema": {
+              "type": "string"
+            }
+          },
+          {
+            "$ref": "#/components/parameters/Actor"
+          }
+        ],
+        "responses": {
+          "200": {
+            "description": "OK",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "type": "object",
+                  "additionalProperties": {
+                    "type": "string"
+                  }
+                }
+              }
+            }
+          },
+          "401": {
+            "$ref": "#/components/responses/Unauthorized"
+          },
+          "default": {
+            "$ref": "#/components/responses/Error"
+          }
+        }
+      }
+    },
+    "/api/v1/mcp/servers/{name}/test": {
+      "post": {
+        "operationId": "postMcpServersNameTest",
+        "summary": "Ask an MCP server for its tools, with its credentials, as /mcp would",
+        "tags": [
+          "mcp"
+        ],
+        "parameters": [
+          {
+            "name": "name",
+            "in": "path",
+            "required": true,
+            "description": "The MCP server's name: its tools are NAME.tool in /mcp.",
+            "schema": {
+              "type": "string"
+            }
+          },
+          {
+            "$ref": "#/components/parameters/Actor"
+          }
+        ],
+        "responses": {
+          "200": {
+            "description": "OK",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/MCPTestView"
+                }
+              }
+            }
+          },
+          "401": {
+            "$ref": "#/components/responses/Unauthorized"
+          },
+          "default": {
+            "$ref": "#/components/responses/Error"
+          }
+        }
+      }
+    },
     "/api/v1/metrics/range": {
       "post": {
         "operationId": "postMetricsRange",
@@ -5351,6 +5731,31 @@
           }
         }
       },
+      "AuthView": {
+        "type": "object",
+        "description": "AuthView is how people sign in on this gateway.",
+        "properties": {
+          "signup_open": {
+            "type": "boolean"
+          },
+          "oidc": {
+            "$ref": "#/components/schemas/OIDCView",
+            "description": "OIDC is single sign-on through an identity provider: Entra ID, Keycloak, Okta."
+          },
+          "ldap": {
+            "$ref": "#/components/schemas/LDAPView",
+            "description": "LDAP is sign-in checked by a directory: Active Directory."
+          },
+          "scim": {
+            "type": "boolean",
+            "description": "SCIM is whether a provider may keep accounts in step."
+          },
+          "base_url": {
+            "type": "string",
+            "description": "BaseURL is the portal's address, when the settings say it: what a provider's redirect address is made from."
+          }
+        }
+      },
       "CanaryPercentRequest": {
         "type": "object",
         "description": "CanaryPercentRequest changes the share of a model's requests its canary takes.",
@@ -5421,6 +5826,21 @@
           }
         }
       },
+      "CheckView": {
+        "type": "object",
+        "description": "CheckView is one step of trying a directory.",
+        "properties": {
+          "step": {
+            "type": "string"
+          },
+          "ok": {
+            "type": "boolean"
+          },
+          "detail": {
+            "type": "string"
+          }
+        }
+      },
       "ClassGPUs": {
         "type": "object",
         "description": "ClassGPUs is how many of a node's devices are of one class, and how many of those are free or kept out of placement.",
@@ -5573,6 +5993,34 @@
           }
         }
       },
+      "DirectorySyncView": {
+        "type": "object",
+        "description": "DirectorySyncView is the last time the leader read the directory.",
+        "properties": {
+          "at": {
+            "type": "string",
+            "format": "date-time"
+          },
+          "people": {
+            "type": "integer"
+          },
+          "accounts": {
+            "type": "integer"
+          },
+          "suspended": {
+            "type": "integer"
+          },
+          "reinstated": {
+            "type": "integer"
+          },
+          "roles_changed": {
+            "type": "integer"
+          },
+          "error": {
+            "type": "string"
+          }
+        }
+      },
       "DomainView": {
         "type": "object",
         "description": "DomainView is one domain: its nodes, its power and whether it is held.",
@@ -6349,6 +6797,70 @@
           }
         }
       },
+      "JudgeCheckRequest": {
+        "type": "object",
+        "description": "JudgeCheckRequest asks how a judge keeps up: Burst judgments at once after three in turn; zero is two for each ready replica.",
+        "properties": {
+          "model": {
+            "type": "string"
+          },
+          "burst": {
+            "type": "integer"
+          }
+        }
+      },
+      "JudgeCheckView": {
+        "type": "object",
+        "description": "JudgeCheckView is how long a judgment took, alone and among others, set against the time a request waits for one.",
+        "properties": {
+          "model": {
+            "type": "string"
+          },
+          "budget_ms": {
+            "type": "integer",
+            "format": "int64"
+          },
+          "ready_replicas": {
+            "type": "integer"
+          },
+          "took_ms": {
+            "type": "array",
+            "items": {
+              "type": "integer",
+              "format": "int64"
+            },
+            "description": "TookMs are three judgments of the same text, one after another."
+          },
+          "slowest_ms": {
+            "type": "integer",
+            "format": "int64"
+          },
+          "burst": {
+            "type": "integer",
+            "description": "Burst judgments at once: the slowest of them, how many gave no verdict, and how many a second the judges read."
+          },
+          "burst_slowest_ms": {
+            "type": "integer",
+            "format": "int64"
+          },
+          "burst_failed": {
+            "type": "integer"
+          },
+          "judgments_per_second": {
+            "type": "number"
+          },
+          "verdict": {
+            "type": "string",
+            "description": "Verdict is ok, tight (past half the budget alone, or past it among others) or slow (past it alone); Advice what to do when not ok."
+          },
+          "advice": {
+            "type": "array",
+            "items": {
+              "type": "string"
+            }
+          }
+        }
+      },
       "JudgeView": {
         "type": "object",
         "description": "JudgeView is one judge model the guardrail policies ask, and whether it can answer them.",
@@ -6389,6 +6901,11 @@
           "wanted_replicas": {
             "type": "integer"
           },
+          "budget_ms": {
+            "type": "integer",
+            "format": "int64",
+            "description": "BudgetMs is how long a request waits for it: the shortest of its policies' latency budgets."
+          },
           "last_error": {
             "type": "string",
             "description": "LastError is its last failure to give a verdict, if any."
@@ -6826,6 +7343,61 @@
           }
         }
       },
+      "LDAPTestRequest": {
+        "type": "object",
+        "description": "LDAPTestRequest is an auth.ldap block to try, as YAML, and the service account's password when it is not kept yet.",
+        "properties": {
+          "settings": {
+            "type": "string"
+          },
+          "password": {
+            "type": "string"
+          }
+        }
+      },
+      "LDAPTestView": {
+        "type": "object",
+        "description": "LDAPTestView is how trying the directory went.",
+        "properties": {
+          "ok": {
+            "type": "boolean"
+          },
+          "checks": {
+            "type": "array",
+            "items": {
+              "$ref": "#/components/schemas/CheckView"
+            }
+          }
+        }
+      },
+      "LDAPView": {
+        "type": "object",
+        "description": "LDAPView is auth.ldap as it is in force, and how its last sync went.",
+        "properties": {
+          "urls": {
+            "type": "array",
+            "items": {
+              "type": "string"
+            }
+          },
+          "base_dn": {
+            "type": "string"
+          },
+          "bind_dn": {
+            "type": "string"
+          },
+          "label": {
+            "type": "string"
+          },
+          "off": {
+            "type": "string",
+            "description": "Off is why this control plane could not use the settings, when it could not; local accounts still sign in."
+          },
+          "last_sync": {
+            "$ref": "#/components/schemas/DirectorySyncView"
+          }
+        }
+      },
       "Live": {
         "type": "object",
         "description": "Live is one of the cluster's adapters as its node last reported it.",
@@ -6941,6 +7513,107 @@
           }
         }
       },
+      "MCPServerRequest": {
+        "type": "object",
+        "description": "MCPServerRequest adds a server or changes one. Secrets are header values to keep in the store, sealed, each then read by reference; HeaderRefs name secrets kept elsewhere (a file, env:, vault:).",
+        "properties": {
+          "url": {
+            "type": "string"
+          },
+          "description": {
+            "type": "string"
+          },
+          "headers": {
+            "type": "object",
+            "additionalProperties": {
+              "type": "string"
+            }
+          },
+          "header_refs": {
+            "type": "object",
+            "additionalProperties": {
+              "type": "string"
+            }
+          },
+          "secrets": {
+            "type": "object",
+            "additionalProperties": {
+              "type": "string"
+            }
+          },
+          "enabled": {
+            "type": "boolean"
+          },
+          "timeout_seconds": {
+            "type": "integer"
+          },
+          "rate_per_minute": {
+            "type": "integer",
+            "description": "RatePerMinute bounds each caller's calls to its tools; a negative number takes the bound away."
+          }
+        }
+      },
+      "MCPServerView": {
+        "type": "object",
+        "description": "MCPServerView is a server as the API shows it: header values only for the headers sent as written, and for the others the reference read.",
+        "properties": {
+          "name": {
+            "type": "string"
+          },
+          "url": {
+            "type": "string"
+          },
+          "description": {
+            "type": "string"
+          },
+          "headers": {
+            "type": "object",
+            "additionalProperties": {
+              "type": "string"
+            }
+          },
+          "header_refs": {
+            "type": "object",
+            "additionalProperties": {
+              "type": "string"
+            }
+          },
+          "enabled": {
+            "type": "boolean"
+          },
+          "timeout_seconds": {
+            "type": "integer"
+          },
+          "rate_per_minute": {
+            "type": "integer"
+          },
+          "created_at": {
+            "type": "string",
+            "format": "date-time"
+          },
+          "created_by": {
+            "type": "string"
+          }
+        }
+      },
+      "MCPTestView": {
+        "type": "object",
+        "description": "MCPTestView is what the server said when asked: its tools, or why not.",
+        "properties": {
+          "ok": {
+            "type": "boolean"
+          },
+          "tools": {
+            "type": "array",
+            "items": {
+              "type": "string"
+            }
+          },
+          "error": {
+            "type": "string"
+          }
+        }
+      },
       "MembershipRequest": {
         "type": "object",
         "description": "MembershipRequest puts a person in an organization, team or project, or with DELETE takes them out.",
@@ -7492,6 +8165,27 @@
           }
         }
       },
+      "OIDCView": {
+        "type": "object",
+        "description": "OIDCView is auth.oidc as it is in force.",
+        "properties": {
+          "issuer": {
+            "type": "string"
+          },
+          "client_id": {
+            "type": "string"
+          },
+          "label": {
+            "type": "string"
+          },
+          "create_accounts": {
+            "type": "boolean"
+          },
+          "no_passwords": {
+            "type": "boolean"
+          }
+        }
+      },
       "OrgView": {
         "type": "object",
         "description": "OrgView is an organization with what is in it.",
@@ -8182,6 +8876,15 @@
           }
         }
       },
+      "SecretRequest": {
+        "type": "object",
+        "description": "SecretRequest is a secret for the store to keep.",
+        "properties": {
+          "value": {
+            "type": "string"
+          }
+        }
+      },
       "SettingsRequest": {
         "type": "object",
         "description": "SettingsRequest is a new version of the cluster's settings, made from the version Base, which must still be the newest.",
@@ -9054,6 +9757,13 @@
             "type": "string",
             "description": "APIKeyFile holds the provider's key, on the control plane. A file rather than the key itself, so the key is in neither the catalog, the store nor the audit log; empty for a provider that asks for none."
           },
+          "api_key_files": {
+            "type": "array",
+            "items": {
+              "$ref": "#/components/schemas/catalog.ExternalKey"
+            },
+            "description": "APIKeyFiles are several keys for the provider, in place of APIKeyFile: one endpoint each (external:MODEL#1, #2…), so a key that answers 429 or 401 is passed over while its breaker is open and the request goes to another. A provider's limits are per key."
+          },
           "peer": {
             "type": "string",
             "description": "Peer names the federated llmgw cluster this model is offered by, for a model that federation added (`llmgw ctl federation add`) rather than one written by hand. Such a model is named PEER/MODEL, comes and goes with the peer's own list, and is sent only while the peer is up."
@@ -9072,6 +9782,20 @@
           }
         }
       },
+      "catalog.ExternalKey": {
+        "type": "object",
+        "description": "ExternalKey is one of a provider's keys.",
+        "properties": {
+          "file": {
+            "type": "string",
+            "description": "File holds the key, on the control plane, as APIKeyFile does."
+          },
+          "weight": {
+            "type": "number",
+            "description": "Weight is its share of the requests against the others'; zero is 1."
+          }
+        }
+      },
       "catalog.GPUNeed": {
         "type": "object",
         "description": "GPUNeed is what card a profile needs.",
@@ -10216,6 +10940,20 @@
           "bits_per_weight": {
             "type": "number",
             "description": "BitsPerWeight is the artifact's precision, 0 when unknown."
+          },
+          "weights_gb": {
+            "type": "number",
+            "description": "What NeedGB is made of, a card's share (or, on the CPU, the node's RAM): the weights, the context's KV cache, and the engine's own."
+          },
+          "context_gb": {
+            "type": "number"
+          },
+          "engine_gb": {
+            "type": "number"
+          },
+          "expert_parallel": {
+            "type": "boolean",
+            "description": "ExpertParallel spreads a mixture-of-experts model's experts over the cards, whole, rather than slicing each across them: vLLM's --enable-expert-parallel, for a model of experts on more than one card."
           }
         }
       },
@@ -10320,6 +11058,15 @@
           "context": {
             "type": "integer"
           },
+          "kv_bytes_per_token": {
+            "type": "integer",
+            "format": "int64",
+            "description": "KVBytesPerToken is one token of context, the whole model: what an option needs for another context is worked out from it. 0 is unknown."
+          },
+          "experts": {
+            "type": "integer",
+            "description": "Experts is a mixture-of-experts model's routed experts; 0 for dense."
+          },
           "gated": {
             "type": "boolean"
           },
@@ -10469,6 +11216,17 @@
           "cpu_cores": {
             "type": "integer",
             "description": "CPUCores is what a replica computes on; zero takes half the smallest node it fits on."
+          },
+          "env": {
+            "type": "object",
+            "additionalProperties": {
+              "type": "string"
+            },
+            "description": "Env is set in the engine's environment, on every node of a replica: NCCL_DEBUG=INFO to see how its cards talk, NCCL_IB_* to steer them. Only the variables engines are tuned with; the catalog refuses others."
+          },
+          "memory_gb": {
+            "type": "integer",
+            "description": "MemoryGB is what a replica on CPUs reserves, in place of the figure worked out from its weights and context: a judge's, whose own sizing is known."
           }
         }
       },

docs/clients.md

@@ -579,6 +579,15 @@ claude mcp add --transport http llmgw https://llm.example.com/mcp \
   --header "Authorization: Bearer $LLMGW_API_KEY"
 ```
 
+**Your company's tools.** Where an administrator has registered the
+company's own MCP servers (Jira, a file server, a database) and your key is
+granted their tools, the same `/mcp` lists them too, each as `server.tool`
+(`jira.search`). Give your agent only llmgw's `/mcp` and your key: the
+servers' own credentials stay with llmgw. A tool your key is not granted is
+not listed, and calling it by name is refused. Each call is in the audit
+log under your key: which tool, the size of what was sent, how it ended and
+how long it took, never what it said.
+
 ### Licences you leave out
 
 A model's licence is in `/v1/models` (`license`: its `id`, `name`, whether

docs/console.md

@@ -2,8 +2,8 @@
 
 `sudo llmgw`, with nothing after it, on a machine llmgw runs on. On a machine
 with nothing installed it opens the setup screens; on a control plane, the
-console described here; on a node, the node's own console, which edits
-`agent.yaml`. Everything on these screens can also be done by command — each
+console described here; on a node, the node's own console, which shows
+what runs on that machine and edits `agent.yaml`. Everything on these screens can also be done by command — each
 screen says which — so a script never needs them and a person never needs
 to remember the commands.
 
@@ -246,6 +246,28 @@ long a key made there may live, and is the expiry of one made without any
 Limits are requests and tokens a minute, a budget per day or month, and the
 models allowed. Each level only narrows what is above it.
 
+**Sign-in** — how people get in besides a password kept here: single
+sign-on, Active Directory and provisioning, each on or off, with the
+directory's last sync (how many people, accounts kept in step, suspended,
+reinstated).
+- **E** sets up Entra ID. Give the tenant ID, the client ID and secret,
+  and the groups for administrators and operators. A provider other than
+  Entra takes its issuer URL in place of the tenant. The form shows the
+  redirect address to register.
+- **D** sets up Active Directory: the servers, where people are, the
+  service account and its password, the domain's CA certificate as a PEM
+  file on this machine, the group whose members may sign in, and groups
+  for roles and teams.
+- Before it is kept, the directory is tried from the control plane: each
+  server reached and bound, how many people may sign in, how many are in
+  each group. If any step fails, nothing is kept and the steps are shown.
+  **T** tries the settings in force.
+- On the provisioning row, Enter makes a SCIM token, shown once.
+- The password, the client secret and the token are kept in the store,
+  sealed by the master key when one is set. The settings name them as
+  `store:auth/…`, so each control plane has them with no file put on it.
+  The CA is written into the settings themselves.
+
 **Guardrails…** on a row lists the guardrail policies server.yaml defines.
 Each policy is marked in one of three ways:
 
@@ -335,5 +357,18 @@ to restart it.
 
 ## The node's console
 
-On a node, `sudo llmgw` opens `agent.yaml` in the same settings view, and the
-manual. The cluster is watched and changed from a control plane.
+On a node, `sudo llmgw` opens on **Overview**: what this machine is doing, read
+on the machine itself, so it works without an admin token and while the control
+plane is out of reach. Refreshed every three seconds, it shows
+
+- llmgw-agent's state and since when, and the version of llmgw;
+- each control plane the agent dials, from `agent.yaml` and the list the
+  leader last sent, and whether it can be reached;
+- each GPU: how busy, its memory, its heat, and the models on it;
+- the engines running here, from the agent's own records: replica, model,
+  engine, GPUs, port, whether the process is still there, and for how long;
+- the free space where weights and engines are kept;
+- the agent's warnings and errors of the last hour.
+
+Below it, `agent.yaml` in the same settings view, and the manual. The cluster
+is watched and changed from a control plane.

docs/data.md

@@ -72,7 +72,9 @@ each answer that may be reused, as the client got it, and a hash of the
 request it answers, in the gateway process's memory — never in the store,
 never on disk — for the project's TTL or until pushed out by newer ones
 (`router.cache.max_bytes`), and gone with a restart. Never under a `no_store`
-policy. Forgetting a person does not reach into it; the answers age out by
+policy. A project that reuses answers to similar questions (`--similar`)
+also has each question's last user message embedded, and the vector kept
+beside its answer, in the same memory and for as long. Forgetting a person does not reach into it; the answers age out by
 the TTL. The store keeps only the setting, and a usage record's mark that it
 was answered from the cache.
 

docs/guardrails.md

@@ -91,6 +91,21 @@ guard:
   value is the same placeholder throughout a request).
 - **`block`** stops it.
 
+**Putting them back.** With `restore: true` in a policy's `input`, the
+values a request had redacted are put back in its answer: the model reads
+"write to [PERSON_1] at [EMAIL_1]", the client reads the name and address
+it sent. Streamed answers too — a placeholder split across two events is
+held until it is whole. Only that request's own values, only into its
+answer; the map is kept in memory for the request and nowhere else. The
+output rules then read the answer as the client will, so a policy that
+redacts personal data in answers still does.
+
+```yaml
+input:
+  pii: {email: redact, phone: redact}
+  restore: true
+```
+
 **Shapes.**
 
 - `system_prompt` is put first.
@@ -258,6 +273,39 @@ themselves.
     context rather than overflowing it and going unjudged.
 - **Cache:** the same text is not asked about twice within five minutes.
 
+## An outside service
+
+A site required to use a particular guardrail — Azure Content Safety,
+Lakera Guard, NeMo Guardrails, a classifier of its own — names it in a
+policy's `services`. It is asked over HTTP beside the patterns and before
+the judge; its "unsafe" blocks or flags like theirs, and a service that does
+not answer is what the policy's `fail` says.
+
+```yaml
+policies:
+  strict:
+    fail: closed
+    services:
+      - name: lakera
+        url: https://api.lakera.ai/v2/guard
+        header_files: {Authorization: /etc/llmgw/secrets/lakera}   # "Bearer …", read each call
+        body: '{"messages": [{"role": "user", "content": {{json .Text}}}]}'
+        blocked: flagged                 # where the answer says unsafe
+        reason: breakdown.0.detector_type
+        on: both                         # input, output or both
+        action: block                    # or flag
+        timeout: 2s
+```
+
+`body` is a template given `.Text` (`{{json .Text}}` writes it as a JSON
+string; empty sends `{"input": …}`). `blocked` and `reason` are dotted paths
+into the JSON answer, numbers for list items; true, a number above zero, a
+non-empty list or a string other than "false" is unsafe. On input the
+service reads the latest user turn; on output the whole answer, once, at its
+end — a streamed answer released a sentence at a time is then stopped or
+flagged as the judge's is. The rule recorded is `service:<name>:<reason>`.
+The service is told the text only, never who sent it.
+
 ## Where a policy applies
 
 `guard.default` applies everywhere. Others are attached to a part of the

docs/install.md

@@ -152,9 +152,9 @@ That is the whole step. In order, it:
    `nvidia-smi` finds any, on CPU otherwise — and waits for it to install its
    engine
 9. closes llmgw's internal ports to everything but the cluster's networks
-10. fetches Qwen3Guard-Gen 0.6B, the guardrails' judge, and runs two
-    replicas on the CPUs of nodes with room — this one, or the first that
-    join ([guardrails.md](guardrails.md#a-judge-model) has one on a GPU too)
+10. fetches Qwen3Guard-Gen 0.6B, the guardrails' judge, and runs it on
+    CPUs: one replica on up to four cores when this machine is the only
+    node, two on two cores each on the nodes of a larger cluster ([guardrails.md](guardrails.md#a-judge-model) has one on a GPU too)
 11. fetches Qwen2.5 0.5B into the mirror and serves it on this node, so the
     first thing anyone opens `/chat` to find is a model that answers. A
     machine without room for it is told so within two minutes, and why.
@@ -187,15 +187,43 @@ demonstration one:
 | Requests | every key limited (60 a minute, 200 000 tokens a minute) and priced, so budgets bind |
 | Guardrails | the `baseline` policy on every request: secrets in prompts and answers refused, card numbers masked, injection attempts recorded, and the judge refusing violence, crime, weapons, child abuse, privacy violations, hate, self-harm and sexual content, in prompts and answers |
 
-The judge runs as two replicas, each on a CPU — about half a second for a
-short text on any CPU with AVX2 — and reads a streamed answer every 600
-characters. The policy fails **closed**: a request the judge cannot read
-within 5 seconds is refused with 503 rather than served unjudged, and
-`judge_unavailable` goes to the notification webhooks. Two replicas are what
-make that bearable; on one small machine, or a VM without AVX2 (see
+The judge runs on CPUs. A judgment of about 280 tokens took 2.3 s on four
+cores, 4.2 s on two and 8.0 s on one (a 4-core VM). It also reads a
+streamed answer every 600 characters. The policy fails **closed**: a request
+the judge cannot read within 5 seconds is refused with 503 rather than
+served unjudged, and `judge_unavailable` goes to the notification webhooks.
+So a judge gets cores of its own: on one machine one replica on up to four,
+and on a larger cluster two replicas of two cores each, spread over its nodes
+so that losing one leaves the other. On a machine of fewer than four cores,
+a busy one, or a VM without AVX2 (see
 [step 6](#6-setting-the-nodes-cpu-model-if-it-is-a-vm)), the judge is the
-first thing to fail, so `fail: open` — serve it, recorded as flagged — or
-`--no-guard-model` is the choice there. Defamation, specialized advice,
+first thing to fail. The choices there are a longer `latency_budget`,
+`fail: open` (serve the request, recorded as flagged), the judge on a GPU,
+or `--no-guard-model`.
+
+init times the judge once it is up. If a judgment takes more than half the
+wait, or more than the wait when several come at once, it says so before the
+next steps, with what to do. The wait itself is not changed without being
+asked. `llmgw ctl guard judge check` does the same at any time:
+
+```
+qwen3guard-0.6b, 2 replica(s) ready, a request waits 5.0 s for it
+  alone:    2.4, 2.3, 2.3 s
+  4 at once: slowest 6.1 s; about 0.7 judgments a second, so about 0.3 requests a second with short answers
+tight: a busy moment passes the budget
+```
+
+In a three-machine install the judges are better on machines of their own.
+On a 4-core control plane that was also a node, 38 of 60 requests under load
+were refused for want of a judgment. With the judges on a node of their own
+(4 cores, two judges of two cores each), 2 were refused. The setup screen asks
+for up to two **judge machines** after the store's two, or give
+`--judge-node user@host` to `server init`, once for each. Each is added as a
+node labelled `llmgw.role: judge`, and the judge's profile selects that
+label, so the judges run only there and nothing else does. Four cores and
+8 GB are enough for two judges.
+
+Defamation, specialized advice,
 intellectual property and elections are left out of what it refuses: they
 catch too much ordinary work. [guardrails.md](guardrails.md) has the rest.
 
@@ -363,7 +391,11 @@ by itself. Three things are worth knowing before you edit it:
 - `default_rpm`, `default_tpm` and `default_price` are what make limits and
   budgets mean anything. Without a price every request costs zero and a budget
   never binds; without the defaults, a project created without limits is
-  metered by nothing at all.
+  metered by nothing at all. A price may set `cached_per_1k` beside
+  `prompt_per_1k`: the prompt tokens the engine or provider read from its
+  cache (the answer's `cached_tokens`) are charged at it, the rest at the
+  prompt rate. A provider that bills cached reads at a tenth is
+  `{prompt_per_1k: 2500, cached_per_1k: 250, completion_per_1k: 10000}`.
 - `control.enabled: true` is what makes this a control plane while its catalog
   is empty. Without it, a file with no models is refused as having nothing to
   serve.
@@ -1319,6 +1351,26 @@ catalog, and a key rotated in the file is used within seconds. A model whose key
 file cannot be read is not offered. Everything else applies as to any model:
 prices and budgets, rate limits, usage records, and key allowlists.
 
+A provider's limits are per key. With several keys, one answering 429 need not
+hold up the model: give them all, each with an optional weight (its share
+against the others'), in place of `api_key_file`:
+
+```yaml
+  external:
+    url: https://api.openai.com
+    model: gpt-4o-mini
+    api_key_files:
+      - {file: /etc/llmgw/secrets/openai-1.key, weight: 2}
+      - {file: /etc/llmgw/secrets/openai-2.key}
+```
+
+Each key is an endpoint of its own, `external:cloud-mini#1`, `#2`. A key that
+answers 429 is passed over while its breaker is open, and the request goes to
+another. One that answers 401 (revoked, expired, mistyped) is passed over too,
+and `external_key_refused` goes to the notification webhooks, naming the
+endpoint and never the key. The usage records say which key answered by the
+same name. A key whose file cannot be read is left out, and the others stay.
+
 **Its prompts leave the cluster.** Decide who may send them: a key or project
 whose allowlist leaves the external model out cannot reach it, as a fallback or
 by name. The cluster page and `ctl models` mark it `external`.
@@ -1909,6 +1961,18 @@ So "why did qwen take four minutes to start" is one trace: find `replica start`
 with `llmgw.model` = qwen, and the node's spans beneath it say whether the
 time went to the download, the launch or the engine loading.
 
+An inference request and each `upstream` attempt also carry the
+[OpenTelemetry GenAI conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/),
+so a tool that reads them — Grafana, Jaeger, Langfuse, Arize Phoenix — shows
+llmgw's calls as model calls without a dashboard made for llmgw:
+`gen_ai.operation.name` (`chat`, `text_completion`, `embeddings`),
+`gen_ai.request.model`, `gen_ai.request.max_tokens`, `temperature` and `top_p`
+when the client set them, `gen_ai.conversation.id` (the session),
+`gen_ai.response.model`, and `gen_ai.usage.input_tokens`, `output_tokens` and
+`cache_read.input_tokens`. On an `upstream` span `gen_ai.provider.name` is the
+engine that answered (`vllm`, `llamacpp`, `external`, …). What was asked and
+what was answered are never in a span.
+
 A node follows the control plane's decision: a start the control plane kept is
 traced on the node whatever the agent's own `sample_ratio`, and one it sampled
 out is not; the agent's ratio decides only for what arrives with no trace, as
@@ -2094,8 +2158,46 @@ router:
     max_entry_bytes: 2097152 # 2 MB
 ```
 
-`llmgw_cache_requests_total{model,result}` counts hits, misses and
-bypasses, `llmgw_cache_bytes` and `llmgw_cache_entries` what is held. The
+### Similar questions
+
+A help desk asked "how do I reset my password" in forty wordings asks the
+model forty times under the exact cache. `--similar` also gives a chat
+question the answer to one like it:
+
+```
+llmgw keys cache --project Acme/support/desk --ttl 24h --similar 0.95
+```
+
+```yaml
+router:
+  cache:
+    embedding_model: bge-m3   # an embedding model in the catalog
+```
+
+- **What is compared:** the last user message, embedded by
+  `router.cache.embedding_model`, against the questions already answered
+  in the same conversation — the same system prompt, the same earlier turns,
+  tools, `response_format` and sampling settings, the same model and
+  project. A like question in another conversation, or after a tool's
+  result, is not taken for this one. The exact cache is asked first, and an
+  exact hit embeds nothing.
+- **How alike:** the cosine similarity of the two embeddings, at least the
+  project's `--similar`. 0.95 is strict; below 0.9 different questions start
+  to meet. A question that differs only in a number or a name ("2+2" and
+  "2+3") can be closer than that, so a project whose questions turn on
+  figures should not use it. A question over 2,000 characters is reused only
+  when it is the same.
+- **Said:** `x-llmgw-cache: similar`, with `x-llmgw-cache-similarity`
+  and `Age`. Recorded and charged as a hit. The same rules as a hit decide
+  which answers are reused and kept; guardrails judge the request before.
+- **The embedding** is asked of the model by the gateway on its own account,
+  as a judge is, within 2 seconds; every miss waits for it. Without the
+  model, or when it does not answer, only the same request is reused. The
+  vectors are kept in memory with their answers and counted in
+  `router.cache.max_bytes`.
+
+`llmgw_cache_requests_total{model,result}` counts hits, similar hits, misses
+and bypasses, `llmgw_cache_bytes` and `llmgw_cache_entries` what is held. The
 console has it as **Cache answers…** on a project.
 
 ## Batches
@@ -2460,6 +2562,79 @@ the same files and keys as `locales/en/`, and its `_meta.*` entries saying how
 it writes numbers and dates. The tests refuse a catalogue that lacks a key a
 page uses.
 
+## The company's MCP servers
+
+llmgw's `/mcp` can serve the company's own MCP servers beside its own tools.
+Each server, a ticket system, a file server or a database, is registered once.
+Its credentials stay on the control planes, and people's agents are given only
+`/mcp` and their llmgw key.
+
+```
+llmgw ctl mcp add jira --url https://jira.internal/mcp \
+    --description "Acme Jira" --secret-file Authorization=/root/jira-token
+llmgw ctl mcp test jira          # its tools, asked with its credentials
+llmgw keys limits --scope team --id Acme/support --tools 'jira.*'
+```
+
+- **Credentials:**
+  - `--secret-file NAME=FILE` reads a header's value once and keeps it in
+    the store, sealed by the master key, so no file has to be on each
+    control plane.
+  - `--header-ref NAME=REF` reads it at each call from a file, `env:`,
+    `vault:` or `store:` instead.
+  - `--header NAME=VALUE` is for what is not secret.
+- **Who may call what:** tools are named `server.tool` and granted as
+  patterns (`jira.*`, `db.query`) at any level of the tenancy tree, as
+  models are.
+  - A level that names any narrows what is above it.
+  - Unlike models, **none granted anywhere is none**: a tool reaches into a
+    company system, so it is granted, never assumed.
+  - The console's limits form has the same field.
+- **The server is told who calls:** `X-Llmgw-Key`, and `X-Llmgw-User` and
+  `X-Llmgw-Project` where there are some. It is called with llmgw's
+  credentials, never the caller's key.
+- **Bounds:** a call has 60 seconds (`--timeout`) and an answer 4 MB. A
+  server's progress is passed on to a client that takes a stream. `--rate N`
+  allows each caller N calls a minute to a server's tools.
+- **Guardrails, both ways:** the caller's policies read a tool's answer as
+  the prompt it becomes.
+  - An instruction planted in a ticket is caught as a pasted one would be
+    (`injection: block`).
+  - Personal data is redacted or refused as the policy says. A refused
+    answer does not reach the agent.
+  - The arguments are read before they leave, so a secret pasted into one
+    does not reach the company system.
+- **A server that keeps failing:** five calls failing in a row take a server
+  out for a minute. Its tools leave the list, and a call is answered at once.
+- **Record:** every call is in `llmgw ctl audit`, with its arguments' size
+  and a digest, never their text. `llmgw_mcp_tool_calls_total` and
+  `llmgw_mcp_tool_call_seconds` count them by server, tool and outcome.
+- **Changes:** a server added, changed or removed on one control plane is
+  served by every one within half a minute; its tool list is read again
+  every five minutes. `--disabled` keeps a server but serves none of its
+  tools.
+
+**This cluster's own, read only.** `llmgw ctl mcp add llmgw --url
+llmgw:admin` registers the cluster itself as a server: tools that say what
+`llmgw ctl` does, and change nothing. It is answered in the control plane's
+own process, with no address and no token to give.
+
+| Tool | Says |
+|---|---|
+| `llmgw.cluster_status` | control planes, store, nodes, models against what they asked for, GPUs, what is unhealthy |
+| `llmgw.list_nodes` | each node's state, GPUs, load, fabrics, agent version and engines |
+| `llmgw.list_models` | each model's replicas ready against wanted, profiles, why one is parked |
+| `llmgw.list_replicas` | each replica's shards: node, GPUs, state (`model` for one model's) |
+| `llmgw.explain_placement` | why a model is not placed, node by node |
+| `llmgw.node_logs` | a node's last agent log lines, or an engine's (`source`) |
+
+Grant it as any other: `llmgw keys limits --scope key --id KEY-ID --tools
+'llmgw.*'`. It is the same as an administrator's view, so grant it to the
+people who would have `ctl`.
+
+Servers over HTTP only for now (the streamable HTTP transport). A server
+that runs as a local process (stdio) is planned.
+
 ## Single sign-on
 
 People can sign in through your identity provider — Keycloak, Entra ID,
@@ -2467,8 +2642,10 @@ Google, Okta, Authentik, anything that speaks OpenID Connect.
 
 1. Register a client there with the redirect address
    `https://<gateway>/login/oidc/callback`.
-2. Put its secret in a file only root reads.
-3. Add to `server.yaml`:
+2. Put its secret in a file the llmgw user reads, on every control plane.
+3. Add this to the cluster's settings with `llmgw ctl config edit`. On a
+   cluster with a store, `auth` is the cluster's: the same block in
+   `server.yaml` is not read, and the log says so.
 
 ```yaml
 auth:
@@ -2517,6 +2694,99 @@ and point it at `https://<gateway>/scim/v2` with that token as the bearer.
   join and leave them. Membership in a group is removed when someone leaves
   it.
 
+### Entra ID
+
+The console sets this up: `sudo llmgw`, People → Sign-in, **E**. What
+follows is what it writes, and what to do in Entra.
+
+Entra ID is an OpenID Connect provider like the others, with two things to
+set in the app registration:
+
+1. **Redirect URI:** under Authentication, add a Web platform with
+   `https://<gateway>/login/oidc/callback`. The issuer is
+   `https://login.microsoftonline.com/<tenant-id>/v2.0`.
+2. **Groups in the token:** under Token configuration, add a groups claim.
+   Choose **Groups assigned to the application**, and assign the groups you
+   map in `roles` and `memberships` to the enterprise application. Entra
+   sends groups by object id unless you choose to emit names
+   (`sAMAccountName` for groups synced from AD), so write them the same way
+   in the mapping.
+
+Entra leaves the groups out of the token for someone in more than 200 of
+them and sends a pointer to Graph instead. llmgw then keeps that person's
+role and memberships as they were, rather than reading it as no groups, and
+logs an error saying so. "Groups assigned to the application" avoids this.
+So does SCIM below, which Entra also speaks: in the enterprise application
+under Provisioning, set the tenant URL to `https://<gateway>/scim/v2` and
+the secret token to the contents of `auth.scim.token_file`.
+
+## Active Directory
+
+With an on-premises Active Directory, people sign in on the gateway's own
+form with their AD username (`ada`, `CORP\ada`, `ada@corp.example`) and
+password. The gateway checks them with the domain controllers over LDAPS.
+
+The console sets this up and tries it first: `sudo llmgw`, People →
+Sign-in, **D**. It keeps the service account's password in the store
+(`bind_password_file: store:auth/ldap-bind`) and writes the CA into the
+settings (`ca:`), so nothing has to be copied to each control plane.
+By hand it is this block in the cluster's settings (`llmgw ctl config
+edit`), where files it names must then be on every control plane:
+
+```yaml
+auth:
+  enabled: true
+  ldap:
+    urls: [ldaps://dc1.corp.example, ldaps://dc2.corp.example]   # tried in order
+    ca_file: /etc/llmgw/corp-ca.pem         # the domain's CA, when not in the system's
+    bind_dn: CN=svc-llmgw,OU=Service,DC=corp,DC=example
+    bind_password_file: /etc/llmgw/secrets/ldap-bind   # root-only
+    base_dn: OU=Staff,DC=corp,DC=example    # where people are
+    user_filter: (memberOf:1.2.840.113556.1.4.1941:=CN=llmgw-users,OU=Groups,DC=corp,DC=example)
+    label: Acme                             # "Sign in with your Acme account"
+    create_accounts: true
+    roles:                                  # by group CN or full DN; nested groups count
+      admin: [llmgw-admins]
+      operator: [llmgw-ops]
+    memberships:
+      ml-engineers: "team:Acme/ml"
+    sync_interval: 15m                      # negative turns the sync off
+```
+
+- **Service account:** an ordinary domain user is enough: it only reads.
+  It finds the person and reads their groups. The person's password is
+  checked by binding as them, and is never stored.
+- **Encryption:** `ldaps://`, or `ldap://` with `start_tls: true`.
+  Plaintext LDAP is refused, since passwords would cross the network.
+- **Certificate:** the domain controller's certificate must name it in
+  its subject alternative names: a name in the CN only is refused, as are
+  certificates with a negative serial number. Samba's own autogenerated
+  certificate is both, so a Samba DC needs one from a CA (`tls certfile`
+  in smb.conf). A CA file a control plane cannot read turns directory
+  sign-in off on that control plane, said in its log. Local accounts still
+  sign in.
+- **Who may sign in:** enabled user objects under `base_dn` that match
+  `user_filter`. The example admits members of `llmgw-users`, nested groups
+  included.
+- **Accounts:** an account is tied to the person's `objectGUID`, so a changed
+  mail or name is still the same account. An account that already exists
+  with the person's mail is taken over. Without `create_accounts`, only
+  people who already have an account can sign in. Local accounts, such as
+  the administrator `server init` made, keep signing in with their own
+  password.
+- **Sync:** every `sync_interval` the leader reads the directory.
+  - Someone disabled or deleted in AD is suspended and their sessions
+    end.
+  - When they are enabled again they are reinstated. An account an
+    administrator suspended by hand stays suspended.
+  - Roles follow groups, and memberships the directory gave are taken away
+    when the person leaves the group.
+  - A sync that would suspend more than half of the directory's accounts is
+    refused and logged, rather than locking everyone out over a wrong
+    `base_dn` or filter.
+- **Passwords:** a person's password is changed in AD. The gateway sends no
+  reset link for these accounts.
+
 ## Secrets in Vault
 
 Every setting that names a file holding a secret — the store's `@address`,
@@ -3088,6 +3358,8 @@ requests. On a slow model, set `timeouts.generation` higher.
 
 - `docs/choosing-models.md` — which models to run and how many, if the answer
   to "we have GPUs, now what" is not already obvious
+- `docs/large-models.md` — a model on several cards or several nodes: how it
+  is split, what each way needs of a card, and bringing one up on a fabric
 - `docs/data.md` — what is stored and what leaves the machine, for the review
   that happens before this is allowed near anyone's documents
 - `docs/reference.md` — every configuration key, generated from the source, so

docs/kong.md

@@ -0,0 +1,95 @@
+# llmgw and Kong
+
+For a site that already runs Kong. The two meet over plain HTTP: Kong in
+front of llmgw, as one more API it governs, or Kong behind llmgw, as an
+OpenAI-compatible provider. Nothing of either is built into the other.
+
+Tried with Kong Gateway 3.9 (`kong:3.9`, the last open-source release; 3.10
+and later need a Kong licence) in DB-less mode, in front of llmgw.
+
+## Kong in front of llmgw
+
+Two ways, both tried.
+
+**A plain route.** Kong forwards the request as it came. Tried with chat
+completions, streamed and not; llmgw's other APIs — Anthropic's
+`/v1/messages`, `/v1/responses`, embeddings, pictures — go the same way,
+since Kong does not read them. Kong adds its own authentication, rate limits
+and logging; llmgw still checks the key, the budget and the guardrails.
+
+```yaml
+_format_version: "3.0"
+services:
+- name: llmgw
+  url: https://llm.example.com          # llmgw's client port
+  routes:
+  - name: llmgw
+    paths: [/llmgw]                      # /llmgw/v1/chat/completions → /v1/chat/completions
+```
+
+**Kong's `ai-proxy` plugin**, for Kong's own token statistics on a chat
+route. llmgw is an OpenAI provider whose `upstream_url` is llmgw:
+
+```yaml
+services:
+- name: llmgw-chat
+  url: https://llm.example.com
+  routes:
+  - name: chat
+    paths: [/chat]
+  plugins:
+  - name: ai-proxy
+    config:
+      route_type: llm/v1/chat
+      logging: {log_statistics: true, log_payloads: false}
+      model:
+        provider: openai
+        name: qwen-coder                 # a model llmgw serves
+        options:
+          upstream_url: https://llm.example.com/v1/chat/completions
+```
+
+**Keys.** Without an `auth` block, `ai-proxy` passes the client's
+`Authorization` header on: each team keeps its own llmgw key, and llmgw's
+budgets, limits, guardrails and usage stay per team. With
+`auth: {header_name: Authorization, header_value: "Bearer sk-llmgw-…"}` Kong
+sends one key of its own instead, and llmgw sees every caller as that key —
+then budgets per team are Kong's to keep.
+
+**What Kong counts.** For a whole answer Kong reads llmgw's `usage` and
+records prompt and completion tokens as llmgw did. For a streamed answer
+Kong 3.9 records an estimate of the prompt tokens and no completion tokens,
+even with `stream_options.include_usage` set — the usage chunk reaches it
+and is not read. llmgw's own accounting counts both; use its usage export
+(`llmgw ctl usage export`) or `/admin/usage` for figures to bill by.
+
+**Timeouts.** A long answer from a model still loading outlasts Kong's
+60-second defaults: set the service's `read_timeout` to the time a model may
+take to start, as for nginx in [install.md](install.md).
+
+## Kong behind llmgw
+
+Not tried here. Kong's AI routes speak OpenAI's API, so llmgw reaches models Kong fronts as
+it reaches any provider: a model whose source is `external`.
+
+```yaml
+- name: kong-gpt
+  source: external
+  external:
+    url: https://kong.example.com/openai     # Kong's route, without /v1
+    model: gpt-4o-mini
+    api_key_file: /etc/llmgw/secrets/kong-consumer.key
+```
+
+The key is the one Kong's own authentication asks of a consumer. As a
+fallback (`fallbacks: [kong-gpt]`) it is used only when the cluster has no
+room.
+
+## Which to choose
+
+Kong in front when the site's API policy says every API goes through Kong:
+llmgw is then one more upstream, and its per-team accounting still holds
+when each team keeps its own key. Kong behind when the site already reaches
+outside providers through Kong and llmgw should too. llmgw does not need
+Kong for anything it does itself — keys, budgets, guardrails, routing to
+its own replicas.

docs/large-models.md

@@ -0,0 +1,145 @@
+# Large models: several cards, several nodes
+
+A model too big for one card runs on several. llmgw chooses how from the
+model's own figures and the cluster as it is; this page is the rules it
+follows, the commands, and what to look at when a first one is brought up
+on new hardware.
+
+## How a model is split
+
+In order, the first that fits:
+
+1. **One card.** Nothing to split.
+2. **Several cards of one node, tensor parallel.** Every layer is cut
+   across the cards, so they talk for every token: they must be on one
+   NVLink domain. The agent reads `nvidia-smi topo -m` and gives a replica
+   the cards with the most NVLink between them
+   ([install.md](install.md#nvlink-if-your-gpus-have-it)).
+3. **Several nodes, pipeline parallel between them.** Each node holds a
+   run of layers on all its cards (tensor parallel inside it), and only a
+   layer's output crosses to the next node. The nodes must share one
+   InfiniBand or RoCE fabric: a replica never spans two, and never a node
+   with none. Any number of nodes, not only two or four.
+
+The plan takes the fewest cards, then the fewest nodes, then the better
+precision up to eight bits.
+
+**Tensor width and the KV heads.** The cards split the model's KV heads
+among them, or past their number each card holds a copy of one. So the
+width divides the KV heads or is a multiple of them: a model with 8 KV
+heads runs 2, 4, 8 or 16 wide, not 12. Qwen3-235B has 4 and runs on 8.
+
+**Mixture of experts.** A model of experts (Qwen3-235B-A22B, DeepSeek-V3,
+Llama 4, Mixtral) on more than one card is served with vLLM's
+`--enable-expert-parallel`: each card holds whole experts rather than a
+slice of every one, which is less traffic per token. It is read from
+`config.json` (`num_experts`, `n_routed_experts`, `num_local_experts`).
+
+**When it does not fit.** A smaller precision first: an FP8 or AWQ
+repository of the same model, or a GGUF quantisation on llama.cpp if one
+node's cards hold it. Then more nodes. `llmgw ctl model plan REPO
+--precision bf16,fp8` shows each precision's fit from the model's config.
+
+## What a way needs
+
+Each way the plan offers says what a replica needs of each card, part by
+part, and of all its cards together:
+
+```
+$ llmgw ctl model plan meta-llama/Llama-3.3-70B-Instruct --format safetensors --context 32768
+FILE      SIZE    BEST WAY        A CARD   OF     ALL CARDS  NOW
+original  131 GB  4 × h100-80g    37.4 GB  72 GB  150 GB     free
+
+Best: original on 4 × h100-80g: 37.4 GB a card (weights 32.9 + context 2.5 + engine 2.0) of 72 usable; 150 GB on 4 cards.
+The context costs 0.31 GB per 1000 tokens, the whole model (--context to plan another).
+```
+
+- **weights:** the files, divided over the cards.
+- **context:** the KV cache for the context planned, divided likewise.
+- **engine:** the engine's own: 2 GB and a fiftieth of the weights for vLLM.
+- **usable:** 90% of a card; the rest is the driver's and fragmentation's.
+
+The console's **Serve a model** shows the same for every way a file runs.
+A context longer than planned is checked against it before anything is
+fetched.
+
+## Serving one
+
+Without flags the plan's best way is taken:
+
+```
+llmgw ctl model serve Qwen/Qwen3-235B-A22B-Instruct-2507-FP8 --format safetensors --context-length 32768
+```
+
+Or a way given by hand, for instance two nodes of eight:
+
+```
+llmgw ctl model serve meta-llama/Llama-3.3-70B-Instruct --format safetensors \
+    --gpu-class h100-80g --gpus 8 --nodes 2 --env NCCL_DEBUG=INFO
+```
+
+`--gpus` is the cards on each node and `--nodes` the nodes. The pipeline
+is the nodes and the tensor width the cards of each.
+
+`--env KEY=VALUE` goes into the engine's environment on every node:
+`NCCL_DEBUG=INFO` while bringing a cluster up, `NCCL_IB_*` to steer it.
+Only the variables engines are tuned with are taken. llmgw already sets
+these:
+- `NCCL_IB_HCA` to the fabric's adapters;
+- `NCCL_IB_GID_INDEX` on RoCE;
+- `NCCL_SOCKET_IFNAME` to the management network;
+- for an NVLink domain spanning nodes, `NCCL_MNNVL_ENABLE`.
+
+The job's first step says what it needs of each card. The weights then
+come to the mirror once and from it to each node. The startup timeout (20
+minutes unless the profile's `startup_timeout` says otherwise) runs from
+when a node's download last moved. So a model of hundreds of gigabytes is
+not failed while it is still arriving, and one whose download stops is
+failed that long after.
+
+## Watching it come up
+
+| What | Where |
+|---|---|
+| The replica and its shards, one per node, each with its download | `llmgw ctl replicas` |
+| Why it is not placed | `llmgw ctl explain-placement MODEL` |
+| A node's engine output (NCCL's lines with `NCCL_DEBUG=INFO`) | `llmgw ctl logs NODE --source SHARD` |
+| The bandwidth between two nodes, rail by rail | `llmgw ctl fabric-test NODE NODE` |
+| Each node's GPUs, NVLink and adapters | `llmgw doctor` on the node; its page under /admin |
+| Every step of a start, across nodes | the `replica start` trace ([install.md](install.md)) |
+
+With `NCCL_DEBUG=INFO`, NCCL says which transport it took between nodes:
+`NET/IB` is the fabric. `NET/Socket` means it fell back to TCP, and the
+model will run, slowly.
+
+## Bringing it up on new hardware
+
+A first run on a fabric, in order. Each step is worth its output if
+something goes wrong.
+
+1. **The nodes.** Add each with its fabric:
+   `llmgw node add host --fabric ib-a:mlx5_0,mlx5_1`. Then
+   `llmgw ctl nodes` shows every node ready, on `ib-a`.
+2. **The fabric.** `llmgw ctl fabric-test node1 node2`: each rail at its
+   rated speed (a 400 Gb/s rail near 45 GB/s).
+3. **One node, several cards.** A dense 70B at bf16 on four or eight
+   cards of one node, by the plan:
+   `llmgw ctl model serve meta-llama/Llama-3.3-70B-Instruct --format safetensors`.
+   Ready, then a question answered.
+4. **Two nodes.** The same model forced over two:
+   `--gpu-class CLASS --gpus 8 --nodes 2 --env NCCL_DEBUG=INFO`, under
+   another `--name`. Ready, answering, and `NET/IB` in both shards' logs.
+5. **Experts.** A mixture-of-experts model on one node's eight cards,
+   for instance `Qwen/Qwen3-235B-A22B-Instruct-2507-FP8`: its profile
+   carries `--enable-expert-parallel` (`llmgw ctl models`, or its
+   definition from the admin API).
+6. **A node lost.** With the two-node replica serving, stop one node's
+   agent (`systemctl stop llmgw-agent`): the replica fails and is placed
+   again once the node is back or another has room. Nothing is left
+   holding cards.
+
+For each, worth keeping:
+- the time from serve to ready;
+- tokens a second from `llmgw ctl bench`;
+- the plan's per-card figure against what `nvidia-smi` shows once loaded;
+- anything it said that was wrong.

docs/onboarding.md

@@ -54,10 +54,20 @@ on the page that creates it and never again.
 
 ### Connecting an editor
 
-The account page's **Connect your editor** makes a key for Continue, Codex or
-Cline and shows what the tool needs with that key, this gateway's address and a
-model already in it:
+The account page's **Connect your editor** makes a key for VS Code, Continue,
+Codex or Cline and shows what the tool needs with that key, this gateway's
+address and a model already in it:
 
+- **VS Code** — llmgw's own extension, downloaded from the gateway at
+  `/setup/vscode/llmgw.vsix` with this gateway's address already in it. It
+  offers every chat model the key may use to VS Code itself: the chat's model
+  picker, agent mode, and any extension that asks VS Code for a language model
+  (Cline's "VS Code LM API" provider among them), with no setup of their own.
+  **llmgw: Sign In** in VS Code opens this page with VS Code chosen; the page's
+  **Open in VS Code** link hands the key back, and the extension keeps it in
+  VS Code's secret storage. A link the extension did not ask for is confirmed
+  first, with the gateway's address shown, since it decides where code is sent.
+  **llmgw: Enter API Key** takes a pasted one instead.
 - **Continue** — an assistant file of its own, `~/.continue/assistants/llmgw.yaml`,
   so their `config.yaml` is not touched.
 - **Codex** — a provider and a profile in `~/.codex/config.toml`, run with

docs/quick-install.md

@@ -8,7 +8,7 @@ explains every choice.
 
 | | |
 |---|---|
-| **A control plane** | One machine, or three for high availability (the store replicated, either one able to take over). Linux, x86-64 or ARM64: Debian 12, Ubuntu 22.04+ or RHEL 10. No GPU needed; 2 GB RAM, 20 GB disk each — 8 GB RAM when it is also a node running the starter model and the two judges. |
+| **A control plane** | One machine, or three for high availability (the store replicated, either one able to take over). Linux, x86-64 or ARM64: Debian 12, Ubuntu 22.04+ or RHEL 10. No GPU needed; 2 GB RAM, 20 GB disk each — 12 GB RAM when it is also a node running the starter model and the two judges, which setup checks before it installs anything; four cores or more, or the judges on machines of their own, keep a judgment within its wait. |
 | **GPU nodes** (optional to start) | The same Linux, with the NVIDIA driver installed. vLLM needs driver 525 or newer and a Turing GPU or newer; a V100 gets an older vLLM. |
 | **root** | on the control plane. On the other two of a three-machine setup, `sudo` (its password is asked for if it needs one); on nodes, `sudo` without a password. |
 | **SSH** | from the control plane to the other machines. The setup screens take a key, the ssh agent or a password; `node add` uses ssh keys. A machine nothing may log into joins with a code instead (`llmgw join`). |

docs/reference.md

@@ -37,6 +37,7 @@ rather than repeated.
 | `default_tpm` | `int` | [Auth](#auth) |
 | `signup_open` | `bool` | [Auth](#auth) |
 | `oidc` | `OIDC` | [Auth](#auth) |
+| `ldap` | `LDAP` | [Auth](#auth) |
 | `scim` | `SCIM` | [Auth](#auth) |
 | `max_key_ttl` | `time.Duration` | [Auth](#auth) |
 | `personal` | `Personal` | [Auth](#auth) |
@@ -52,6 +53,7 @@ rather than repeated.
 | `retain_days` | `int` | [Batch](#batch) |
 | `max_bytes` | `int64` | [Cache](#cache) |
 | `max_entry_bytes` | `int64` | [Cache](#cache) |
+| `embedding_model` | `string` | [Cache](#cache) |
 | `model` | `string` | [Canary](#canary) |
 | `percent` | `int` | [Canary](#canary) |
 | `min_requests` | `int` | [Canary](#canary) |
@@ -105,10 +107,13 @@ rather than repeated.
 | `url` | `string` | [External](#external) |
 | `model` | `string` | [External](#external) |
 | `api_key_file` | `string` | [External](#external) |
+| `api_key_files` | `[]ExternalKey` | [External](#external) |
 | `peer` | `string` | [External](#external) |
 | `api` | `string` | [External](#external) |
 | `workflow` | `string` | [External](#external) |
 | `output` | `string` | [External](#external) |
+| `file` | `string` | [ExternalKey](#externalkey) |
+| `weight` | `float64` | [ExternalKey](#externalkey) |
 | `budgets` | `[]FacilityBudget` | [Facility](#facility) |
 | `margin` | `float64` | [Facility](#facility) |
 | `gpu_watts` | `float64` | [Facility](#facility) |
@@ -144,6 +149,21 @@ rather than repeated.
 | `tokenize_timeout` | `time.Duration` | [KVIndex](#kvindex) |
 | `min_blocks` | `int` | [KVIndex](#kvindex) |
 | `max_blocks` | `int` | [KVIndex](#kvindex) |
+| `urls` | `[]string` | [LDAP](#ldap) |
+| `start_tls` | `bool` | [LDAP](#ldap) |
+| `ca_file` | `string` | [LDAP](#ldap) |
+| `ca` | `string` | [LDAP](#ldap) |
+| `bind_dn` | `string` | [LDAP](#ldap) |
+| `bind_password_file` | `string` | [LDAP](#ldap) |
+| `base_dn` | `string` | [LDAP](#ldap) |
+| `group_base_dn` | `string` | [LDAP](#ldap) |
+| `user_filter` | `string` | [LDAP](#ldap) |
+| `label` | `string` | [LDAP](#ldap) |
+| `create_accounts` | `bool` | [LDAP](#ldap) |
+| `roles` | `map[string][]string` | [LDAP](#ldap) |
+| `memberships` | `map[string]string` | [LDAP](#ldap) |
+| `sync_interval` | `time.Duration` | [LDAP](#ldap) |
+| `timeout` | `time.Duration` | [LDAP](#ldap) |
 | `contact` | `string` | [Landing](#landing) |
 | `notice` | `string` | [Landing](#landing) |
 | `language` | `string` | [Landing](#landing) |
@@ -238,6 +258,7 @@ rather than repeated.
 | `require_capabilities` | `bool` | [Policy](#policy) |
 | `validate_structured` | `bool` | [Policy](#policy) |
 | `judge` | `*JudgePolicy` | [Policy](#policy) |
+| `services` | `[]Service` | [Policy](#policy) |
 | `stream` | `string` | [Policy](#policy) |
 | `enabled` | `bool` | [Prefix](#prefix) |
 | `max_nodes` | `int` | [Prefix](#prefix) |
@@ -301,6 +322,7 @@ rather than repeated.
 | `injection` | `string` | [Rules](#rules) |
 | `deny` | `[]Pattern` | [Rules](#rules) |
 | `max_chars` | `int` | [Rules](#rules) |
+| `restore` | `bool` | [Rules](#rules) |
 | `token_file` | `string` | [SCIM](#scim) |
 | `python` | `string` | [SGLangConfig](#sglangconfig) |
 | `ttft_p99` | `time.Duration` | [SLO](#slo) |
@@ -361,6 +383,15 @@ rather than repeated.
 | `facility` | `Facility` | [Server](#server) |
 | `federation` | `Federation` | [Server](#server) |
 | `endpoints` | `[]Endpoint` | [Server](#server) |
+| `name` | `string` | [Service](#service) |
+| `url` | `string` | [Service](#service) |
+| `header_files` | `map[string]string` | [Service](#service) |
+| `body` | `string` | [Service](#service) |
+| `blocked` | `string` | [Service](#service) |
+| `reason` | `string` | [Service](#service) |
+| `on` | `string` | [Service](#service) |
+| `action` | `string` | [Service](#service) |
+| `timeout` | `time.Duration` | [Service](#service) |
 | `stopped` | `bool` | [ServingSettings](#servingsettings) |
 | `bin` | `string` | [SimConfig](#simconfig) |
 | `python` | `string` | [TRTLLMConfig](#trtllmconfig) |
@@ -394,6 +425,7 @@ rather than repeated.
 | `secret_file` | `string` | [Webhook](#webhook) |
 | `prompt_per_1k` | `int64` | [accounting.Price](#accountingprice) |
 | `completion_per_1k` | `int64` | [accounting.Price](#accountingprice) |
+| `cached_per_1k` | `int64` | [accounting.Price](#accountingprice) |
 | `per_image` | `int64` | [accounting.Price](#accountingprice) |
 | `per_mesh` | `int64` | [accounting.Price](#accountingprice) |
 | `name` | `string` | [admission.Band](#admissionband) |
@@ -432,6 +464,7 @@ rather than repeated.
 | `queue_ttl` | `time.Duration` | [config.Band](#configband) |
 | `prompt_per_1k` | `int64` | [config.Price](#configprice) |
 | `completion_per_1k` | `int64` | [config.Price](#configprice) |
+| `cached_per_1k` | `int64` | [config.Price](#configprice) |
 | `per_image` | `int64` | [config.Price](#configprice) |
 | `per_mesh` | `int64` | [config.Price](#configprice) |
 | `disabled` | `[]string` | [engines.Config](#enginesconfig) |
@@ -616,6 +649,15 @@ OIDC is sign-in through an identity provider.
 
 Takes a [OIDC](#oidc) block.
 
+### `ldap`
+
+`LDAP`
+
+LDAP is sign-in against a directory, Active Directory above all, and
+the accounts kept in step with it.
+
+Takes a [LDAP](#ldap) block.
+
 ### `scim`
 
 `SCIM`
@@ -728,6 +770,15 @@ Past it, the answers used longest ago go first.
 MaxEntryBytes bounds one answer; a longer one is not kept. 1 MB
 unless set.
 
+### `embedding_model`
+
+`string`
+
+EmbeddingModel embeds the questions of projects that reuse answers to
+similar ones (llmgw keys cache --similar): an embedding model in the
+catalog. Unset, those projects reuse only the same request's answer.
+The gateway asks it on its own account, as it asks a judge.
+
 ## Canary
 
 ### `model`
@@ -1103,6 +1154,15 @@ APIKeyFile holds the provider's key, on the control plane. A file
 rather than the key itself, so the key is in neither the catalog, the
 store nor the audit log; empty for a provider that asks for none.
 
+### `api_key_files`
+
+`[]ExternalKey`
+
+APIKeyFiles are several keys for the provider, in place of
+APIKeyFile: one endpoint each (external:MODEL#1, #2…), so a key that
+answers 429 or 401 is passed over while its breaker is open and the
+request goes to another. A provider's limits are per key.
+
 ### `peer`
 
 `string`
@@ -1135,6 +1195,20 @@ plane.
 
 Output is what it makes: pictures, or meshes (see ComfyUI.Output).
 
+## ExternalKey
+
+### `file`
+
+`string`
+
+File holds the key, on the control plane, as APIKeyFile does.
+
+### `weight`
+
+`float64`
+
+Weight is its share of the requests against the others'; zero is 1.
+
 ## Facility
 
 ### `budgets`
@@ -1401,6 +1475,114 @@ MaxBlocks bounds how many of a prompt's blocks are hashed and looked
 up. Beyond it a longer match still helps, but the difference no longer
 decides anything. Zero takes 4096.
 
+## LDAP
+
+### `urls`
+
+`[]string`
+
+URLs are the directory's servers, tried in turn: ldaps:// (636), or
+ldap:// (389) with StartTLS.
+
+### `start_tls`
+
+`bool`
+
+StartTLS upgrades an ldap:// connection before anything is sent. A
+password never crosses a connection without TLS: ldap:// without it
+is refused.
+
+### `ca_file`
+
+`string`
+
+CAFile is the authority the servers' certificates are checked against,
+when it is not the system's: an AD Certificate Services root.
+
+### `ca`
+
+`string`
+
+CA is the same authority written out, PEM: kept in the cluster's
+settings, so no file has to be put on every control plane. Used in
+place of CAFile when both are set.
+
+### `bind_dn`
+
+`string`
+
+BindDN and the file holding its password: a service account that may
+read users and groups. Nothing is written to the directory.
+
+### `bind_password_file`
+
+`string`
+
+### `base_dn`
+
+`string`
+
+BaseDN is where people are searched for, subtree included.
+
+### `group_base_dn`
+
+`string`
+
+GroupBaseDN is where their groups are; empty is BaseDN's domain root.
+
+### `user_filter`
+
+`string`
+
+UserFilter narrows who may sign in, ANDed with the login match: a
+group (memberOf=...) or an OU's people only. Empty is every enabled
+user object.
+
+### `label`
+
+`string`
+
+Label is what the form says to sign in with: "Sign in with your
+<label> account". Empty is "Active Directory".
+
+### `create_accounts`
+
+`bool`
+
+CreateAccounts makes an account, active, for a person the directory
+vouches for and the gateway does not know. Off, only existing accounts
+sign in this way.
+
+### `roles`
+
+`map[string][]string`
+
+Roles maps a staff role (viewer, operator, billing, admin) to the
+groups that give it, by name (CN) or whole DN. Set at every sign-in
+and sync: someone taken out of the group loses the role.
+
+### `memberships`
+
+`map[string]string`
+
+Memberships maps a group, by CN or DN, to a place in the tree —
+org:NAME, team:ORG/TEAM, project:ORG/TEAM/PROJECT — joined at sign-in
+and kept by the sync, which also takes away a place whose group the
+person has left.
+
+### `sync_interval`
+
+`time.Duration`
+
+SyncInterval is how often the leader reads the directory. Zero is
+15m; negative turns the sync off, leaving only sign-in.
+
+### `timeout`
+
+`time.Duration`
+
+Timeout bounds each round trip to a server. Zero is 10s.
+
 ## Landing
 
 ### `contact`
@@ -2136,6 +2318,12 @@ Judge has a judge model on the cluster read the text too.
 
 Takes a [JudgePolicy](#judgepolicy) block.
 
+### `services`
+
+`[]Service`
+
+Services are outside guardrails asked over HTTP, beside the judge.
+
 ### `stream`
 
 `string`
@@ -2610,6 +2798,14 @@ Deny are patterns, matched against the text as a model reads it.
 
 MaxChars refuses text longer than this; zero is no limit.
 
+### `restore`
+
+`bool`
+
+Restore, on input, puts what redact rewrote as placeholders back into
+the answer: the model reads "[EMAIL_1]", the client reads its own
+address. Only the values that request sent, only into its answer.
+
 ## SCIM
 
 ### `token_file`
@@ -3117,6 +3313,67 @@ Endpoints is the phase-1 static replica list. It is only consulted when
 no model declares a profile, which is what makes a router-only smoke test
 against sim-engines possible without a control plane.
 
+## Service
+
+### `name`
+
+`string`
+
+Name says which service decided, in the verdict's rule.
+
+### `url`
+
+`string`
+
+URL is POSTed the body.
+
+### `header_files`
+
+`map[string]string`
+
+HeaderFiles are headers whose values are read from files each time,
+so a rotated key needs no restart: {Authorization: /etc/…/lakera}.
+
+### `body`
+
+`string`
+
+Body is the request, a template given .Text; {{json .Text}} writes it
+as a JSON string. Empty is {"input": <text>}.
+
+### `blocked`
+
+`string`
+
+Blocked is where in the answer the service says the text is unsafe:
+a dotted path, numbers for list items (results.0.flagged). True, a
+number above zero, a non-empty list or string other than "false" is
+unsafe.
+
+### `reason`
+
+`string`
+
+Reason is where it says why, for the verdict; optional.
+
+### `on`
+
+`string`
+
+On: input, output or both. Empty is input.
+
+### `action`
+
+`string`
+
+Action on unsafe: block or flag. Empty is block.
+
+### `timeout`
+
+`time.Duration`
+
+Timeout bounds one call. Empty is 3s; past it the policy's fail says.
+
 ## ServingSettings
 
 ### `stopped`
@@ -3355,6 +3612,15 @@ is sha256=<hex HMAC of the body>.
 
 `int64`
 
+### `cached_per_1k`
+
+`int64`
+
+CachedPer1k is what a prompt token costs that the engine or provider
+read from its cache instead of computing: the part of prompt_tokens
+the answer counts as cached_tokens. Zero charges those at
+PromptPer1k, as before there was a rate for them.
+
 ### `per_image`
 
 `int64`
@@ -3629,6 +3895,14 @@ had rather wait behind people than be turned away.
 
 `int64`
 
+### `cached_per_1k`
+
+`int64`
+
+CachedPer1k is the cost of a prompt token the engine or provider read
+from its cache (the answer's cached_tokens, counted within the
+prompt's). Zero charges them at prompt_per_1k.
+
 ### `per_image`
 
 `int64`

go.mod

@@ -12,6 +12,7 @@ require (
 	github.com/charmbracelet/x/ansi v0.11.6
 	github.com/coreos/go-oidc/v3 v3.21.0
 	github.com/go-jose/go-jose/v4 v4.1.4
+	github.com/go-ldap/ldap/v3 v3.4.15
 	github.com/go-zeromq/zmq4 v0.17.0
 	github.com/jackc/pgx/v5 v5.11.0
 	github.com/knadh/koanf/parsers/yaml v1.1.1
@@ -45,6 +46,7 @@ require (
 require (
 	filippo.io/edwards25519 v1.2.0 // indirect
 	filippo.io/hpke v0.4.0 // indirect
+	github.com/Azure/go-ntlmssp v0.1.1 // indirect
 	github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
 	github.com/cenkalti/backoff/v5 v5.0.3 // indirect
@@ -57,6 +59,7 @@ require (
 	github.com/dustin/go-humanize v1.0.1 // indirect
 	github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f // indirect
 	github.com/fsnotify/fsnotify v1.9.0 // indirect
+	github.com/go-asn1-ber/asn1-ber v1.5.8 // indirect
 	github.com/go-logr/logr v1.4.4 // indirect
 	github.com/go-logr/stdr v1.2.2 // indirect
 	github.com/go-viper/mapstructure/v2 v2.5.0 // indirect

go.sum

@@ -8,8 +8,12 @@ filippo.io/edwards25519 v1.2.0 h1:crnVqOiS4jqYleHd9vaKZ+HKtHfllngJIiOpNpoJsjo=
 filippo.io/edwards25519 v1.2.0/go.mod h1:xzAOLCNug/yB62zG1bQ8uziwrIqIuxhctzJT18Q77mc=
 filippo.io/hpke v0.4.0 h1:p575VVQ6ted4pL+it6M00V/f2qTZITO0zgmdKCkd5+A=
 filippo.io/hpke v0.4.0/go.mod h1:EmAN849/P3qdeK+PCMkDpDm83vRHM5cDipBJ8xbQLVY=
+github.com/Azure/go-ntlmssp v0.1.1 h1:l+FM/EEMb0U9QZE7mKNEDw5Mu3mFiaa2GKOoTSsNDPw=
+github.com/Azure/go-ntlmssp v0.1.1/go.mod h1:NYqdhxd/8aAct/s4qSYZEerdPuH1liG2/X9DiVTbhpk=
 github.com/NVIDIA/go-nvml v0.13.4-0 h1:o3jp9u2x1R9ShFE3v+Aesp55XOSIQFMJz/VGNUcJaNE=
 github.com/NVIDIA/go-nvml v0.13.4-0/go.mod h1:id63qwpoDWpFXwnwM6psDCSqW4BmNu6mWpr4YeQtPGo=
+github.com/alexbrainman/sspi v0.0.0-20250919150558-7d374ff0d59e h1:4dAU9FXIyQktpoUAgOJK3OTFc/xug0PCXYCqU0FgDKI=
+github.com/alexbrainman/sspi v0.0.0-20250919150558-7d374ff0d59e/go.mod h1:cEWa1LVoE5KvSD9ONXsZrj0z6KqySlCCNKHlLzbqAt4=
 github.com/aymanbagabas/go-osc52/v2 v2.0.1 h1:HwpRHbFMcZLEVr42D4p7XBqjyuxQH5SMiErDT4WkJ2k=
 github.com/aymanbagabas/go-osc52/v2 v2.0.1/go.mod h1:uYgXzlJ7ZpABp8OJ+exZzJJhRNQ2ASbcXHWsFqH8hp8=
 github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
@@ -46,8 +50,12 @@ github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f h1:Y/CXytFA4m6
 github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f/go.mod h1:vw97MGsxSvLiUE2X8qFplwetxpGLQrlU1Q9AUEIzCaM=
 github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
 github.com/fsnotify/fsnotify v1.9.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
+github.com/go-asn1-ber/asn1-ber v1.5.8 h1:H9AZkK22UOmfX8J84ubyaZxKJZ3FMHVwn8swoMML7iQ=
+github.com/go-asn1-ber/asn1-ber v1.5.8/go.mod h1:hEBeB/ic+5LoWskz+yKT7vGhhPYkProFKoKdwZRWMe0=
 github.com/go-jose/go-jose/v4 v4.1.4 h1:moDMcTHmvE6Groj34emNPLs/qtYXRVcd6S7NHbHz3kA=
 github.com/go-jose/go-jose/v4 v4.1.4/go.mod h1:x4oUasVrzR7071A4TnHLGSPpNOm2a21K9Kf04k1rs08=
+github.com/go-ldap/ldap/v3 v3.4.15 h1:eRB4f4Ecn1POL1mO86+uUDlecmyWsdiJwh60bAUS4UY=
+github.com/go-ldap/ldap/v3 v3.4.15/go.mod h1:jeD0c98EdZQCKKKKe4kTWs5aCf6y+j8KuYsYLznZSQk=
 github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
 github.com/go-logr/logr v1.4.4 h1:tG4xh9yMsRCAiodLVTxyrkzSZ9+o0L1Kg/+cPVcbP/8=
 github.com/go-logr/logr v1.4.4/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
@@ -69,6 +77,8 @@ github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
 github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
 github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0 h1:/Tnpcb2E0Pz/tN9s3bfEY2Q8ePCEX9iuS+cneUwncnw=
 github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0/go.mod h1:zOBXOsUaBSjKgmH4OGzV1esUpR3oUSCPYVd2cUBjKYY=
+github.com/hashicorp/go-uuid v1.0.3 h1:2gKiV6YVmrJ1i2CKKa9obLvRieoRGviZFL26PcT/Co8=
+github.com/hashicorp/go-uuid v1.0.3/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/bN7x4byOro=
 github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
 github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
 github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
@@ -81,6 +91,18 @@ github.com/jackc/pgx/v5 v5.11.0 h1:IzBBtyK9AHqf98cctWFifYSci2hgQR/cd56wB4p+ogg=
 github.com/jackc/pgx/v5 v5.11.0/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4=
 github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
 github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
+github.com/jcmturner/aescts/v2 v2.0.0 h1:9YKLH6ey7H4eDBXW8khjYslgyqG2xZikXP0EQFKrle8=
+github.com/jcmturner/aescts/v2 v2.0.0/go.mod h1:AiaICIRyfYg35RUkr8yESTqvSy7csK90qZ5xfvvsoNs=
+github.com/jcmturner/dnsutils/v2 v2.0.0 h1:lltnkeZGL0wILNvrNiVCR6Ro5PGU/SeBvVO/8c/iPbo=
+github.com/jcmturner/dnsutils/v2 v2.0.0/go.mod h1:b0TnjGOvI/n42bZa+hmXL+kFJZsFT7G4t3HTlQ184QM=
+github.com/jcmturner/gofork v1.7.6 h1:QH0l3hzAU1tfT3rZCnW5zXl+orbkNMMRGJfdJjHVETg=
+github.com/jcmturner/gofork v1.7.6/go.mod h1:1622LH6i/EZqLloHfE7IeZ0uEJwMSUyQ/nDd82IeqRo=
+github.com/jcmturner/goidentity/v6 v6.0.1 h1:VKnZd2oEIMorCTsFBnJWbExfNN7yZr3EhJAxwOkZg6o=
+github.com/jcmturner/goidentity/v6 v6.0.1/go.mod h1:X1YW3bgtvwAXju7V3LCIMpY0Gbxyjn/mY9zx4tFonSg=
+github.com/jcmturner/gokrb5/v8 v8.4.4 h1:x1Sv4HaTpepFkXbt2IkL29DXRf8sOfZXo8eRKh687T8=
+github.com/jcmturner/gokrb5/v8 v8.4.4/go.mod h1:1btQEpgT6k+unzCwX1KdWMEwPPkkgBtP+F6aCACiMrs=
+github.com/jcmturner/rpc/v2 v2.0.3 h1:7FXXj8Ti1IaVFpSAziCZWNzbNuZmnvw/i6CqLNdWfZY=
+github.com/jcmturner/rpc/v2 v2.0.3/go.mod h1:VUJYCIDm3PVOEHw8sgt091/20OJjskO/YJki3ELg/Hc=
 github.com/klauspost/compress v1.19.1 h1:VsB4HPswih7mmZ8WleSFQ75c/Ui1M4trX5oAsJnhSlk=
 github.com/klauspost/compress v1.19.1/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
 github.com/knadh/koanf/maps v0.1.2 h1:RBfmAW5CnZT+PJ1CVc1QSJKf4Xu9kxfQgYVQSu8hpbo=

internal/account/directory_test.go

@@ -0,0 +1,63 @@
+package account
+
+import (
+	"context"
+	"errors"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+type fakeDirectory struct {
+	asked []string
+	u     store.User
+	err   error
+}
+
+func (f *fakeDirectory) SignIn(_ context.Context, login, _ string) (store.User, error) {
+	f.asked = append(f.asked, login)
+	return f.u, f.err
+}
+
+// With a directory, a name the store has no password for is the
+// directory's to check; an account with a password here — the
+// administrator init made — is still checked here, so a directory that is
+// down locks nobody out.
+func TestPasswordsKeptElsewhereAreTheDirectorysToCheck(t *testing.T) {
+	svc, st := newService(t)
+	ctx := t.Context()
+	hash, _ := HashPassword("local-secret")
+	admin := store.User{ID: "usr-admin", Email: "admin@example.com", Name: "a", PasswordHash: hash,
+		State: store.UserActive, CreatedAt: time.Now()}
+	ada := store.User{ID: "usr-ada", Email: "ada@corp.example", Name: "Ada", PasswordHash: "!ldap",
+		State: store.UserActive, CreatedAt: time.Now()}
+	for _, u := range []store.User{admin, ada} {
+		if err := st.CreateUser(ctx, u); err != nil {
+			t.Fatal(err)
+		}
+	}
+	dir := &fakeDirectory{u: ada}
+	svc.SetDirectory(dir)
+
+	if _, _, err := svc.SignIn(ctx, "admin@example.com", "local-secret"); err != nil || len(dir.asked) != 0 {
+		t.Fatalf("the local administrator: %v, directory asked %v", err, dir.asked)
+	}
+	// A name, not an address, and an account kept by the directory.
+	for _, login := range []string{`CORP\ada`, "ada@corp.example"} {
+		if tok, _, err := svc.SignIn(ctx, login, "ad-secret"); err != nil || tok == "" {
+			t.Errorf("%s: %v", login, err)
+		}
+	}
+	if len(dir.asked) != 2 || dir.asked[0] != `CORP\ada` {
+		t.Errorf("the directory was asked %v", dir.asked)
+	}
+	dir.err = ErrBadCredentials
+	if _, _, err := svc.SignIn(ctx, "ada", "wrong"); !errors.Is(err, ErrBadCredentials) {
+		t.Errorf("a password the directory refused: %v", err)
+	}
+	// A reset link would give a directory's account a password here.
+	if _, _, err := svc.RequestReset(ctx, st, "ada@corp.example"); !errors.Is(err, ErrNoReset) {
+		t.Errorf("a directory's account was sent a reset link: %v", err)
+	}
+}

internal/account/service.go

@@ -80,8 +80,26 @@ type Service struct {
 
 	mu       sync.Mutex
 	attempts map[string]*attemptWindow
+
+	// dir checks the passwords of accounts this store keeps none for.
+	dir Directory
 }
 
+// Directory checks a password against an outside directory — Active
+// Directory over LDAP — and answers the account to sign in: made or kept in
+// step with the directory. Its errors are this package's ErrBadCredentials
+// and ErrSuspended for the person's own, and anything else for the
+// directory's.
+type Directory interface {
+	SignIn(ctx context.Context, login, password string) (store.User, error)
+}
+
+// SetDirectory has passwords for accounts with none here, and names that are
+// not addresses, checked by the directory. An account with a password here
+// — the administrator init made — is still checked here, so a directory that
+// is down locks nobody out.
+func (s *Service) SetDirectory(d Directory) { s.dir = d }
+
 type attemptWindow struct {
 	count int
 	since time.Time
@@ -137,6 +155,7 @@ func (s *Service) SignUp(ctx context.Context, email, name, password string) erro
 // SignIn checks a password and opens a session, returning the session token and
 // the CSRF token that must accompany the forms in it.
 func (s *Service) SignIn(ctx context.Context, email, password string) (token, csrf string, err error) {
+	login := strings.TrimSpace(email)
 	email = store.NormalizeEmail(email)
 	// Counted per address and the client trying it, so somebody guessing
 	// from elsewhere cannot lock the owner out of their own account; and per
@@ -151,6 +170,15 @@ func (s *Service) SignIn(ctx context.Context, email, password string) (token, cs
 	}
 
 	u, err := s.store.UserByEmail(ctx, email)
+	// No password kept here — the directory's, or nobody's — and a
+	// directory to ask: it decides.
+	if s.dir != nil && (errors.Is(err, store.ErrNotFound) || (err == nil && strings.HasPrefix(u.PasswordHash, "!"))) {
+		if u, err = s.dir.SignIn(ctx, login, password); err != nil {
+			return "", "", err
+		}
+		s.clearAttempts(byClient)
+		return s.open(ctx, u)
+	}
 	if errors.Is(err, store.ErrNotFound) {
 		// Verify anyway, against a hash that cannot match. Returning early here
 		// would let a stopwatch enumerate every account on the system.
@@ -180,7 +208,11 @@ func (s *Service) SignIn(ctx context.Context, email, password string) (token, cs
 	case store.UserSuspended:
 		return "", "", ErrSuspended
 	}
+	return s.open(ctx, u)
+}
 
+// open makes a session for a person whose password was right.
+func (s *Service) open(ctx context.Context, u store.User) (token, csrf string, err error) {
 	if token, err = store.NewSessionToken(); err != nil {
 		return "", "", err
 	}
@@ -402,7 +434,10 @@ func (s *Service) RequestReset(ctx context.Context, rs ResetStore, email string)
 		return "", store.User{}, ErrTooManyTries
 	}
 	u, err := rs.UserByEmail(ctx, email)
-	if err != nil || u.PasswordHash == "" || u.State != store.UserActive {
+	// An account whose password is kept elsewhere — "!oidc", "!scim",
+	// "!ldap" — gets no link: setting one here would give it a second way
+	// in that its directory knows nothing of, and outlives its leaving.
+	if err != nil || u.PasswordHash == "" || strings.HasPrefix(u.PasswordHash, "!") || u.State != store.UserActive {
 		return "", store.User{}, ErrNoReset
 	}
 	token, err := rs.CreatePasswordReset(ctx, u.ID, ResetLinkTTL)

internal/accounting/accounting.go

@@ -323,7 +323,7 @@ func (a *Accountant) Record(res Reservation, u store.Usage) {
 	if u.ServedModel != "" && u.ServedModel != u.Model && a.pricing != nil {
 		price = a.pricing.For(u.ServedModel)
 	}
-	u.CostMicros = price.Cost(u.PromptTokens, u.CompletionTokens, u.Images, u.Meshes)
+	u.CostMicros = price.Cost(u.PromptTokens, u.CachedTokens, u.CompletionTokens, u.Images, u.Meshes)
 	for _, b := range res.Budgets {
 		a.budgets.Add(b.Counter, b.Period, u.CostMicros)
 	}

internal/accounting/accounting_test.go

@@ -39,20 +39,37 @@ func TestCostRoundsUpSoSmallRequestsAreNotFree(t *testing.T) {
 	// Rounding down would make a stream of tiny requests cost nothing, which is
 	// exactly the traffic shape a quota exists to bound.
 	p := Price{PromptPer1k: 1000, CompletionPer1k: 3000}
-	if got := p.Cost(1, 0, 0, 0); got != 1 {
+	if got := p.Cost(1, 0, 0, 0, 0); got != 1 {
 		t.Errorf("one prompt token cost %d", got)
 	}
-	if got := p.Cost(1000, 1000, 0, 0); got != 4000 {
+	if got := p.Cost(1000, 0, 1000, 0, 0); got != 4000 {
 		t.Errorf("a thousand of each cost %d, want 4000", got)
 	}
-	if got := p.Cost(0, 0, 0, 0); got != 0 {
+	if got := p.Cost(0, 0, 0, 0, 0); got != 0 {
 		t.Errorf("an empty request cost %d", got)
 	}
 }
 
+func TestACachedPromptCostsTheCachedRate(t *testing.T) {
+	p := Price{PromptPer1k: 1000, CachedPer1k: 100, CompletionPer1k: 3000}
+	// 10,000 prompt tokens of which 9,000 were read from the cache: 1,000 at
+	// the prompt rate and 9,000 at the cached one.
+	if got, want := p.Cost(10000, 9000, 0, 0, 0), int64(1000+900); got != want {
+		t.Errorf("a 90%% cached prompt cost %d, want %d", got, want)
+	}
+	// More cached than prompt is an engine's miscount, not a refund.
+	if got := p.Cost(100, 500, 0, 0, 0); got != 10 {
+		t.Errorf("cached beyond the prompt cost %d, want 10", got)
+	}
+	// No cached rate: cached tokens cost what any prompt token does.
+	if got := (Price{PromptPer1k: 1000}).Cost(10000, 9000, 0, 0, 0); got != 10000 {
+		t.Errorf("without a cached rate a cached prompt cost %d, want 10000", got)
+	}
+}
+
 func TestAPictureCostsItsPriceOnTopOfItsPrompt(t *testing.T) {
 	p := Price{PromptPer1k: 1000, PerImage: 50000}
-	if got := p.Cost(20, 0, 3, 0); got != 20+150000 {
+	if got := p.Cost(20, 0, 0, 3, 0); got != 20+150000 {
 		t.Errorf("three pictures from a 20-token prompt cost %d", got)
 	}
 	if (Price{PerImage: 1}).Zero() {

internal/accounting/cost.go

@@ -18,6 +18,11 @@ import (
 type Price struct {
 	PromptPer1k     int64 `koanf:"prompt_per_1k"`
 	CompletionPer1k int64 `koanf:"completion_per_1k"`
+	// CachedPer1k is what a prompt token costs that the engine or provider
+	// read from its cache instead of computing: the part of prompt_tokens
+	// the answer counts as cached_tokens. Zero charges those at
+	// PromptPer1k, as before there was a rate for them.
+	CachedPer1k int64 `koanf:"cached_per_1k"`
 	// PerImage is what one picture costs, for a model that makes them: a
 	// picture's cost is the card's time, which its prompt's few tokens do
 	// not measure.
@@ -33,15 +38,24 @@ type Price struct {
 // whose budget is spent, which is what makes the plan's budget fallback work
 // (§7.7).
 func (p Price) Zero() bool {
-	return p.PromptPer1k == 0 && p.CompletionPer1k == 0 && p.PerImage == 0 && p.PerMesh == 0
+	return p.PromptPer1k == 0 && p.CompletionPer1k == 0 && p.CachedPer1k == 0 && p.PerImage == 0 && p.PerMesh == 0
 }
 
 // Cost computes what a request spent, rounding up.
 //
 // Up rather than to nearest: rounding down would make a stream of tiny requests
 // free, which is exactly the shape of traffic a quota exists to bound.
-func (p Price) Cost(promptTokens, completionTokens, images, meshes int64) int64 {
-	return divUp(promptTokens*p.PromptPer1k, 1000) + divUp(completionTokens*p.CompletionPer1k, 1000) +
+//
+// cachedTokens are counted within promptTokens, as OpenAI's usage counts
+// them, and are charged at the cached rate instead of the prompt rate.
+func (p Price) Cost(promptTokens, cachedTokens, completionTokens, images, meshes int64) int64 {
+	cachedTokens = min(max(cachedTokens, 0), max(promptTokens, 0))
+	cachedRate := p.CachedPer1k
+	if cachedRate == 0 {
+		cachedRate = p.PromptPer1k
+	}
+	return divUp((promptTokens-cachedTokens)*p.PromptPer1k+cachedTokens*cachedRate, 1000) +
+		divUp(completionTokens*p.CompletionPer1k, 1000) +
 		max(images, 0)*p.PerImage + max(meshes, 0)*p.PerMesh
 }
 

internal/agent/discovery/sensors.go

@@ -74,6 +74,13 @@ func NewSensorSampler() *SensorSampler {
 	return &SensorSampler{sys: "/sys", smi: runSMI, now: time.Now}
 }
 
+// SampleGPUs reads every card's heat, power and how busy it is now, without
+// a sampler's ten seconds of memory: for a screen that asks every few seconds
+// and wants what it asked for. Nil on a machine without nvidia-smi or a card.
+func SampleGPUs(ctx context.Context) []GPUSensor {
+	return (&SensorSampler{smi: runSMI}).gpus(ctx)
+}
+
 func runSMI(ctx context.Context, fields string) ([]byte, error) {
 	ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
 	defer cancel()

internal/cli/answercache.go

@@ -5,6 +5,8 @@ import (
 	"time"
 
 	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
 )
 
 func newAnswerCacheCmd(open opener) *cobra.Command {
@@ -12,6 +14,7 @@ func newAnswerCacheCmd(open opener) *cobra.Command {
 		project   string
 		ttl       time.Duration
 		anyAnswer bool
+		similar   float64
 		off       bool
 	)
 	cmd := &cobra.Command{
@@ -28,15 +31,25 @@ func newAnswerCacheCmd(open opener) *cobra.Command {
 			"a request continuing a stored conversation or under a guardrails no_store\n" +
 			"policy. The guardrails judge a reused answer as they judge any other.\n\n" +
 			"Off for every project until this is run. Without --ttl or --off, says how\n" +
-			"it stands. The answers are held in memory, router.cache bounds how much.",
+			"it stands. The answers are held in memory, router.cache bounds how much.\n\n" +
+			"--similar also answers a chat question that is like one already answered:\n" +
+			"its last user message embedded by router.cache.embedding_model, and given\n" +
+			"the answer whose question reaches that cosine similarity (0.95 is strict,\n" +
+			"0.9 looser), as x-llmgw-cache: similar. The system prompt, the earlier\n" +
+			"turns, tools and the sampling settings must still be the same.",
 		Example: "  llmgw keys cache --project Acme/ml/train --ttl 1h\n" +
 			"  llmgw keys cache --project Acme/ml/eval --ttl 24h --any\n" +
+			"  llmgw keys cache --project Acme/support/desk --ttl 24h --similar 0.95\n" +
 			"  llmgw keys cache --project Acme/ml/train --off",
 		Args: cobra.NoArgs,
 		RunE: func(cmd *cobra.Command, _ []string) error {
 			switch {
-			case off && (ttl != 0 || anyAnswer):
-				return fmt.Errorf("--off, or --ttl with --any: not both")
+			case off && (ttl != 0 || anyAnswer || similar != 0):
+				return fmt.Errorf("--off, or --ttl with --any or --similar: not both")
+			case similar < 0 || similar > 1:
+				return fmt.Errorf("--similar is a cosine similarity between 0 and 1, like 0.95")
+			case similar != 0 && ttl == 0:
+				return fmt.Errorf("--similar goes with --ttl: say how long answers are reused")
 			case ttl < 0:
 				return fmt.Errorf("--ttl is how long an answer is reused, like 1h")
 			case ttl > 0 && ttl < time.Second:
@@ -54,7 +67,7 @@ func newAnswerCacheCmd(open opener) *cobra.Command {
 				return err
 			}
 			if off || ttl > 0 {
-				if err := s.SetAnswerCache(cmd.Context(), p.ID, ttl, anyAnswer); err != nil {
+				if err := s.SetAnswerCache(cmd.Context(), p.ID, store.AnswerCache{TTL: ttl, Any: anyAnswer, Similar: similar}); err != nil {
 					return err
 				}
 			}
@@ -71,6 +84,9 @@ func newAnswerCacheCmd(open opener) *cobra.Command {
 				fmt.Fprintf(cmd.OutOrStdout(), "%s reuses answers for %s: those asked at temperature 0 or with a seed, "+
 					"and embeddings, rerank and score.\n", p.Path, c.TTL)
 			}
+			if c.TTL > 0 && c.Similar > 0 {
+				fmt.Fprintf(cmd.OutOrStdout(), "Chat questions reaching %.2f similarity to one answered get its answer too.\n", c.Similar)
+			}
 			if off || ttl > 0 {
 				fmt.Fprintln(cmd.OutOrStdout(), "Gateways see the change within 30 seconds.")
 			}
@@ -80,6 +96,7 @@ func newAnswerCacheCmd(open opener) *cobra.Command {
 	cmd.Flags().StringVar(&project, "project", "", "project: id, path or unique name")
 	cmd.Flags().DurationVar(&ttl, "ttl", 0, "how long an answer is reused, like 1h")
 	cmd.Flags().BoolVar(&anyAnswer, "any", false, "reuse sampled answers too, not only reproducible ones")
+	cmd.Flags().Float64Var(&similar, "similar", 0, "also answer chat questions this similar to one answered, like 0.95")
 	cmd.Flags().BoolVar(&off, "off", false, "stop reusing answers for this project")
 	return cmd
 }

internal/cli/bench.go

@@ -150,6 +150,7 @@ func newBenchCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 	f.DurationVar(&cfg.SLO.TPOTp99, "tpot-p99", cfg.SLO.TPOTp99, "per-output-token target")
 	f.BoolVar(&save, "save", false,
 		"record the capacity against the model and profile, so the number outlives this terminal")
+	cmd.AddCommand(newBenchOverheadCmd(asJSON))
 	return cmd
 }
 

internal/cli/bench_overhead.go

@@ -0,0 +1,421 @@
+package cli
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"fmt"
+	"io"
+	"log/slog"
+	"net"
+	"net/http"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"runtime"
+	"slices"
+	"sort"
+	"strings"
+	"sync"
+	"syscall"
+	"text/tabwriter"
+	"time"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/router"
+	storepkg "gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// What llmgw adds to a request, measured: the same load straight at an
+// engine that answers at once, and through the gateway in front of it —
+// bare, with authentication and usage accounting, and with the guardrails'
+// pattern checks besides. The engine's own time is the same on both paths,
+// so what differs is the gateway's.
+//
+// Everything runs in this process, on loopback: no cluster, no GPU, and the
+// numbers are this machine's. On a machine shared with other work (a VM on
+// an oversubscribed host) they move with it; compare modes run together,
+// not figures from different runs.
+
+// overheadMode is one way a request reaches the engine.
+type overheadMode struct {
+	name        string
+	gateway     bool
+	auth, guard bool
+}
+
+var overheadModes = []overheadMode{
+	{name: "direct"},
+	{name: "gateway", gateway: true},
+	{name: "gateway+auth", gateway: true, auth: true},
+	{name: "gateway+auth+guard", gateway: true, auth: true, guard: true},
+}
+
+// overheadStep is one mode at one request rate.
+type overheadStep struct {
+	Mode     string        `json:"mode"`
+	Rate     int           `json:"rate"`
+	Sent     int           `json:"sent"`
+	Failed   int           `json:"failed"`
+	P50      time.Duration `json:"p50"`
+	P99      time.Duration `json:"p99"`
+	AddedP50 time.Duration `json:"added_p50"`
+	AddedP99 time.Duration `json:"added_p99"`
+	// CPUPer1k is the process's CPU time per thousand requests beyond what
+	// the direct mode spent at the same rate: the gateway's.
+	CPUPer1k time.Duration `json:"cpu_per_1k"`
+	cpu      time.Duration
+}
+
+func newBenchOverheadCmd(asJSON *bool) *cobra.Command {
+	var (
+		rates []int
+		each  time.Duration
+	)
+	cmd := &cobra.Command{
+		Use:   "overhead",
+		Short: "Measure what the gateway adds to a request, against an engine that answers at once",
+		Long: "Runs an engine that answers at once and a gateway in front of it, in this\n" +
+			"process on loopback, and sends the same load at each rate straight to the\n" +
+			"engine and through the gateway: bare, with authentication and usage\n" +
+			"accounting, and with the guardrails' pattern checks too. Reports each\n" +
+			"request's latency at p50 and p99, what the gateway added to it, and the\n" +
+			"gateway's CPU time per thousand requests.\n\n" +
+			"Needs no cluster. The figures are this machine's: on one shared with other\n" +
+			"work they move with it, so compare the modes of one run, and note the\n" +
+			"machine's load beside any figure you keep.",
+		Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			steps, err := runOverhead(cmd.Context(), rates, each)
+			if err != nil {
+				return err
+			}
+			if *asJSON {
+				return json.NewEncoder(cmd.OutOrStdout()).Encode(steps)
+			}
+			printOverhead(cmd.OutOrStdout(), steps)
+			return nil
+		},
+	}
+	f := cmd.Flags()
+	f.IntSliceVar(&rates, "rates", []int{100, 500, 1000, 2000}, "requests a second to send, one step each")
+	f.DurationVar(&each, "each", 3*time.Second, "how long each step sends for")
+	return cmd
+}
+
+func printOverhead(w io.Writer, steps []overheadStep) {
+	tw := tabwriter.NewWriter(w, 0, 0, 2, ' ', 0)
+	fmt.Fprintln(tw, "MODE\tRATE/S\tSENT\tFAILED\tP50\tP99\t+P50\t+P99\tCPU/1K REQ")
+	for _, s := range steps {
+		added := func(d time.Duration) string {
+			if s.Mode == "direct" {
+				return "-"
+			}
+			return msOf(d)
+		}
+		cpu := "-"
+		if s.Mode != "direct" {
+			cpu = msOf(s.CPUPer1k)
+		}
+		fmt.Fprintf(tw, "%s\t%d\t%d\t%d\t%s\t%s\t%s\t%s\t%s\n", s.Mode, s.Rate, s.Sent, s.Failed,
+			msOf(s.P50), msOf(s.P99), added(s.AddedP50), added(s.AddedP99), cpu)
+	}
+	_ = tw.Flush()
+	fmt.Fprintf(w, "\n%d CPUs, %s. Loopback, in one process: the figures are this machine's.\n",
+		runtime.NumCPU(), loadAverage())
+}
+
+func msOf(d time.Duration) string { return fmt.Sprintf("%.2fms", float64(d)/float64(time.Millisecond)) }
+
+// loadAverage is the machine's, where it can be read, to keep beside the
+// figures.
+func loadAverage() string {
+	if b, err := os.ReadFile("/proc/loadavg"); err == nil {
+		if f := strings.Fields(string(b)); len(f) >= 3 {
+			return "load " + strings.Join(f[:3], " ")
+		}
+	}
+	// macOS: "{ 1.23 1.10 0.98 }".
+	if b, err := exec.Command("sysctl", "-n", "vm.loadavg").Output(); err == nil {
+		if f := strings.Fields(strings.Trim(strings.TrimSpace(string(b)), "{}")); len(f) >= 3 {
+			return "load " + strings.Join(f[:3], " ")
+		}
+	}
+	return "load not read"
+}
+
+// runOverhead measures every mode at every rate.
+func runOverhead(ctx context.Context, rates []int, each time.Duration) ([]overheadStep, error) {
+	engine, err := instantEngine()
+	if err != nil {
+		return nil, err
+	}
+	defer engine.Close()
+	dir, err := os.MkdirTemp("", "llmgw-overhead-")
+	if err != nil {
+		return nil, err
+	}
+	defer os.RemoveAll(dir)
+
+	type target struct {
+		url, key string
+		stop     func()
+	}
+	targets := map[string]target{}
+	defer func() {
+		for _, t := range targets {
+			if t.stop != nil {
+				t.stop()
+			}
+		}
+	}()
+	for _, m := range overheadModes {
+		if !m.gateway {
+			targets[m.name] = target{url: engine.URL}
+			continue
+		}
+		url, key, stop, err := overheadGateway(ctx, engine.URL, filepath.Join(dir, m.name+".db"), m)
+		if err != nil {
+			return nil, fmt.Errorf("%s: %w", m.name, err)
+		}
+		targets[m.name] = target{url: url, key: key, stop: stop}
+	}
+
+	client := &http.Client{Transport: &http.Transport{MaxIdleConns: 1024, MaxIdleConnsPerHost: 1024}, Timeout: 30 * time.Second}
+	var out []overheadStep
+	for _, rate := range rates {
+		var direct overheadStep
+		for _, m := range overheadModes {
+			t := targets[m.name]
+			s := loadStep(ctx, client, t.url, t.key, rate, each)
+			s.Mode = m.name
+			if m.name == "direct" {
+				direct = s
+			} else {
+				s.AddedP50, s.AddedP99 = max(0, s.P50-direct.P50), max(0, s.P99-direct.P99)
+				if s.Sent > 0 {
+					s.CPUPer1k = max(0, s.cpu-direct.cpu) * 1000 / time.Duration(s.Sent)
+				}
+			}
+			out = append(out, s)
+		}
+	}
+	return out, nil
+}
+
+// loadStep sends rate requests a second for d, each in its own goroutine,
+// and measures each one's latency and the process's CPU meanwhile.
+func loadStep(ctx context.Context, client *http.Client, url, key string, rate int, d time.Duration) overheadStep {
+	s := overheadStep{Rate: rate}
+	body := []byte(`{"model":"bench","max_tokens":8,"messages":[{"role":"user","content":"What is the capital of France?"}]}`)
+	var (
+		mu    sync.Mutex
+		lat   []time.Duration
+		wg    sync.WaitGroup
+		total = int(float64(rate) * d.Seconds())
+	)
+	interval := time.Second / time.Duration(max(rate, 1))
+	cpu0 := processCPU()
+	start := time.Now()
+	for i := range total {
+		if ctx.Err() != nil {
+			break
+		}
+		if wait := time.Until(start.Add(time.Duration(i) * interval)); wait > 0 {
+			time.Sleep(wait)
+		}
+		wg.Add(1)
+		go func() {
+			defer wg.Done()
+			req, _ := http.NewRequestWithContext(ctx, http.MethodPost, url+"/v1/chat/completions", bytes.NewReader(body))
+			req.Header.Set("Content-Type", "application/json")
+			if key != "" {
+				req.Header.Set("Authorization", "Bearer "+key)
+			}
+			t := time.Now()
+			resp, err := client.Do(req)
+			ok := err == nil && resp.StatusCode == http.StatusOK
+			if resp != nil {
+				_, _ = io.Copy(io.Discard, resp.Body)
+				resp.Body.Close()
+			}
+			el := time.Since(t)
+			mu.Lock()
+			s.Sent++
+			if ok {
+				lat = append(lat, el)
+			} else {
+				s.Failed++
+			}
+			mu.Unlock()
+		}()
+	}
+	wg.Wait()
+	s.cpu = processCPU() - cpu0
+	s.P50, s.P99 = percentile(lat, 0.50), percentile(lat, 0.99)
+	return s
+}
+
+func percentile(ds []time.Duration, p float64) time.Duration {
+	if len(ds) == 0 {
+		return 0
+	}
+	ds = slices.Clone(ds)
+	sort.Slice(ds, func(i, j int) bool { return ds[i] < ds[j] })
+	i := int(p * float64(len(ds)-1))
+	return ds[i]
+}
+
+// processCPU is the CPU time this process has used, user and system.
+func processCPU() time.Duration {
+	var ru syscall.Rusage
+	if err := syscall.Getrusage(syscall.RUSAGE_SELF, &ru); err != nil {
+		return 0
+	}
+	return time.Duration(ru.Utime.Nano() + ru.Stime.Nano())
+}
+
+// instantEngine is an OpenAI-compatible engine that answers at once, with
+// the metrics the router scrapes.
+func instantEngine() (*httpServer, error) {
+	mux := http.NewServeMux()
+	mux.HandleFunc("GET /metrics", func(w http.ResponseWriter, _ *http.Request) {
+		fmt.Fprint(w, "# TYPE vllm:num_requests_running gauge\nvllm:num_requests_running 0\n"+
+			"# TYPE vllm:num_requests_waiting gauge\nvllm:num_requests_waiting 0\n"+
+			"# TYPE vllm:kv_cache_usage_perc gauge\nvllm:kv_cache_usage_perc 0.1\n")
+	})
+	mux.HandleFunc("GET /health", func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(http.StatusOK) })
+	mux.HandleFunc("POST /v1/chat/completions", func(w http.ResponseWriter, r *http.Request) {
+		_, _ = io.Copy(io.Discard, r.Body)
+		w.Header().Set("Content-Type", "application/json")
+		fmt.Fprint(w, `{"id":"b","object":"chat.completion","model":"bench","choices":[{"index":0,"message":{"role":"assistant","content":"Paris."},"finish_reason":"stop"}],"usage":{"prompt_tokens":14,"completion_tokens":2,"total_tokens":16}}`)
+	})
+	return serveLoopback(mux)
+}
+
+// httpServer is a handler served on a loopback port.
+type httpServer struct {
+	URL string
+	srv *http.Server
+}
+
+func (s *httpServer) Close() { _ = s.srv.Close() }
+
+func serveLoopback(h http.Handler) (*httpServer, error) {
+	ln, err := net.Listen("tcp", "127.0.0.1:0")
+	if err != nil {
+		return nil, err
+	}
+	srv := &http.Server{Handler: h, ReadHeaderTimeout: 10 * time.Second}
+	go func() { _ = srv.Serve(ln) }()
+	return &httpServer{URL: "http://" + ln.Addr().String(), srv: srv}, nil
+}
+
+// overheadGateway starts a gateway in front of the engine as the mode asks,
+// and returns its address and, with auth, a key.
+func overheadGateway(ctx context.Context, engineURL, storePath string, m overheadMode) (string, string, func(), error) {
+	cfg := config.Default()
+	cfg.Listen = "127.0.0.1:0"
+	cfg.Router.ScrapeInterval = 200 * time.Millisecond
+	cfg.Endpoints = []config.Endpoint{{ID: "bench", URL: engineURL, Models: []string{"bench"}}}
+	key := ""
+	if m.auth {
+		var err error
+		if key, err = overheadKey(ctx, storePath); err != nil {
+			return "", "", nil, err
+		}
+		cfg.Auth.Enabled, cfg.Auth.Store = true, storePath
+		// Limits would measure themselves refusing: off.
+		cfg.Auth.DefaultRPM, cfg.Auth.DefaultTPM = 0, 0
+	}
+	rt, err := router.New(cfg, slog.New(slog.DiscardHandler))
+	if err != nil {
+		return "", "", nil, err
+	}
+	if m.guard {
+		g, err := baselineGuard()
+		if err != nil {
+			return "", "", nil, err
+		}
+		rt.SetGuard(g)
+	}
+	rctx, cancel := context.WithCancel(ctx)
+	rt.Start(rctx)
+	srv, err := serveLoopback(rt.Handler())
+	if err != nil {
+		cancel()
+		return "", "", nil, err
+	}
+	stop := func() { srv.Close(); cancel(); rt.Stop() }
+	// Ready once a request goes through: the endpoint is scraped first.
+	deadline := time.Now().Add(10 * time.Second)
+	for {
+		body := `{"model":"bench","messages":[{"role":"user","content":"hi"}]}`
+		req, _ := http.NewRequestWithContext(ctx, http.MethodPost, srv.URL+"/v1/chat/completions", strings.NewReader(body))
+		req.Header.Set("Content-Type", "application/json")
+		if key != "" {
+			req.Header.Set("Authorization", "Bearer "+key)
+		}
+		resp, err := http.DefaultClient.Do(req)
+		if err == nil {
+			_, _ = io.Copy(io.Discard, resp.Body)
+			resp.Body.Close()
+			if resp.StatusCode == http.StatusOK {
+				return srv.URL, key, stop, nil
+			}
+		}
+		if time.Now().After(deadline) {
+			stop()
+			status := "no answer"
+			if err == nil {
+				status = resp.Status
+			}
+			return "", "", nil, fmt.Errorf("the gateway did not serve within 10s (%s)", status)
+		}
+		time.Sleep(100 * time.Millisecond)
+	}
+}
+
+// overheadKey makes a store with one project and a key in it.
+func overheadKey(ctx context.Context, path string) (string, error) {
+	st, err := storepkg.Open(ctx, path)
+	if err != nil {
+		return "", err
+	}
+	defer st.Close()
+	org, err := st.CreateOrg(ctx, "bench")
+	if err != nil {
+		return "", err
+	}
+	team, err := st.CreateTeam(ctx, org, "bench")
+	if err != nil {
+		return "", err
+	}
+	project, err := st.CreateProject(ctx, team, "bench")
+	if err != nil {
+		return "", err
+	}
+	secret, k, err := storepkg.NewKey("key-bench", "bench", project)
+	if err != nil {
+		return "", err
+	}
+	k.CreatedAt = time.Now()
+	return secret, st.CreateKey(ctx, k)
+}
+
+// baselineGuard is the policy init installs, without its judge: the pattern
+// checks every request passes.
+func baselineGuard() (guard.Guard, error) {
+	m, err := config.Parse([]byte(guardSection(false)))
+	if err != nil {
+		return nil, err
+	}
+	cfg, err := config.FromMap(m)
+	if err != nil {
+		return nil, err
+	}
+	return guard.NewGate(cfg.Guard, nil, slog.New(slog.DiscardHandler))
+}

internal/cli/bench_overhead_test.go

@@ -0,0 +1,33 @@
+package cli
+
+import (
+	"bytes"
+	"strings"
+	"testing"
+	"time"
+)
+
+// The short version, so CI sees the gateway's overhead on every change: every
+// mode answers every request, and what the gateway adds is measured, not
+// guessed. No bound is asserted on it, as CI's machines vary; the table is
+// in the test's log.
+func TestTheGatewaysOverheadIsMeasured(t *testing.T) {
+	steps, err := runOverhead(t.Context(), []int{50}, time.Second)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(steps) != len(overheadModes) {
+		t.Fatalf("%d steps for %d modes", len(steps), len(overheadModes))
+	}
+	for _, s := range steps {
+		if s.Sent == 0 || s.Failed > 0 || s.P50 <= 0 || s.P99 < s.P50 {
+			t.Errorf("%s: sent %d failed %d p50 %s p99 %s", s.Mode, s.Sent, s.Failed, s.P50, s.P99)
+		}
+	}
+	var b bytes.Buffer
+	printOverhead(&b, steps)
+	if !strings.Contains(b.String(), "gateway+auth+guard") {
+		t.Errorf("table:\n%s", b.String())
+	}
+	t.Logf("\n%s", b.String())
+}

internal/cli/ctl.go

@@ -186,6 +186,7 @@ func newCtlCmd() *cobra.Command {
 		newCapacityCmd(client, &asJSON),
 		newQuotaCmd(client, &asJSON),
 		newGuardCmd(client, &asJSON),
+		newMCPCmd(client, &asJSON),
 		newKBCmd(client, &asJSON),
 		newTrainCmd(client, &asJSON),
 		newConfigCmd(client, &asJSON),

internal/cli/ctl_mcp.go

@@ -0,0 +1,201 @@
+package cli
+
+import (
+	"encoding/json"
+	"errors"
+	"fmt"
+	"os"
+	"strings"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+)
+
+func newMCPCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
+	c := &cobra.Command{
+		Use:   "mcp",
+		Short: "The company's own MCP servers, served through llmgw's /mcp",
+		Long: "An MCP server registered here — Jira, a file server, a database — is served\n" +
+			"through llmgw's /mcp beside llmgw's own tools, its tools named SERVER.tool.\n" +
+			"Its credentials stay on the control planes: a client gives only its llmgw\n" +
+			"key. A key sees and calls only the tools granted it, as patterns, with\n" +
+			"`llmgw keys limits --tools 'jira.*'` at any level of the tenancy tree;\n" +
+			"none granted anywhere is none. Every call is in `llmgw ctl audit`.",
+	}
+	c.AddCommand(newMCPAddCmd(client, asJSON), newMCPListCmd(client, asJSON), newMCPRemoveCmd(client),
+		newMCPTestCmd(client, asJSON))
+	return c
+}
+
+func newMCPAddCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
+	var req adminapi.MCPServerRequest
+	var headers, refs, secretFiles []string
+	var bearerFile string
+	var disabled bool
+	c := &cobra.Command{
+		Use:   "add NAME --url URL",
+		Short: "Register an MCP server, or change one",
+		Example: "  llmgw ctl mcp add jira --url https://jira.internal/mcp --bearer-file /root/jira-token\n" +
+			"  llmgw ctl mcp add llmgw --url llmgw:admin --description \"this cluster, read only\"\n" +
+			"  llmgw ctl mcp add wiki --url https://wiki.internal/mcp --header-ref Authorization=vault:secret/mcp/wiki#token\n" +
+			"  llmgw ctl mcp add jira --disabled      # off, kept",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			var err error
+			if req.Headers, err = pairs(headers, "--header"); err != nil {
+				return err
+			}
+			if req.HeaderRefs, err = pairs(refs, "--header-ref"); err != nil {
+				return err
+			}
+			files, err := pairs(secretFiles, "--secret-file")
+			if err != nil {
+				return err
+			}
+			// A secret read here and kept in the store, sealed: the file
+			// need not be on every control plane, and is not read again.
+			for header, file := range files {
+				b, err := os.ReadFile(file) //nolint:gosec // a file the operator names
+				if err != nil {
+					return err
+				}
+				if req.Secrets == nil {
+					req.Secrets = map[string]string{}
+				}
+				req.Secrets[header] = strings.TrimSpace(string(b))
+			}
+			if bearerFile != "" {
+				b, err := os.ReadFile(bearerFile) //nolint:gosec // a file the operator names
+				if err != nil {
+					return err
+				}
+				if req.Secrets == nil {
+					req.Secrets = map[string]string{}
+				}
+				req.Secrets["Authorization"] = "Bearer " + strings.TrimSpace(string(b))
+			}
+			if cmd.Flags().Changed("disabled") {
+				on := !disabled
+				req.Enabled = &on
+			}
+			body, _ := json.Marshal(req)
+			var v adminapi.MCPServerView
+			if err := client().put(cmd.Context(), "/api/v1/mcp/servers/"+args[0], body, &v); err != nil {
+				return err
+			}
+			if *asJSON {
+				return json.NewEncoder(cmd.OutOrStdout()).Encode(v)
+			}
+			state := "on"
+			if !v.Enabled {
+				state = "off"
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "%s: %s, %s. Its tools are %s.TOOL in /mcp, to keys granted them "+
+				"(llmgw keys limits --tools '%s.*'). Check it: llmgw ctl mcp test %s\n", v.Name, v.URL, state, v.Name, v.Name, v.Name)
+			return nil
+		},
+	}
+	f := c.Flags()
+	f.StringVar(&req.URL, "url", "", "the server's streamable HTTP address, https://…/mcp; llmgw:admin for this cluster, read only")
+	f.StringVar(&req.Description, "description", "", "what it is, shown beside its tools")
+	f.StringArrayVar(&headers, "header", nil, "NAME=VALUE sent as written, for what is not secret; repeatable")
+	f.StringArrayVar(&refs, "header-ref", nil, "NAME=REF read at each call: a file on every control plane, env:, vault: or store:; repeatable")
+	f.StringArrayVar(&secretFiles, "secret-file", nil, "NAME=FILE read now and kept in the store, sealed: a token; repeatable")
+	f.StringVar(&bearerFile, "bearer-file", "", "a token FILE, read now and kept sealed, sent as Authorization: Bearer")
+	f.BoolVar(&disabled, "disabled", false, "keep it but serve none of its tools")
+	f.IntVar(&req.TimeoutSeconds, "timeout", 0, "seconds one call may take (default 60)")
+	f.IntVar(&req.RatePerMinute, "rate", 0, "calls a minute each caller may make to its tools (default: no bound; -1 takes one away)")
+	return c
+}
+
+func newMCPListCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
+	return &cobra.Command{
+		Use:   "list",
+		Short: "The MCP servers /mcp serves",
+		Args:  cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			var list []adminapi.MCPServerView
+			if err := client().get(cmd.Context(), "/api/v1/mcp/servers", &list); err != nil {
+				return err
+			}
+			if *asJSON {
+				return json.NewEncoder(cmd.OutOrStdout()).Encode(list)
+			}
+			if len(list) == 0 {
+				fmt.Fprintln(cmd.OutOrStdout(), "no MCP servers; add one: llmgw ctl mcp add NAME --url URL")
+				return nil
+			}
+			w := tabTo(cmd.OutOrStdout())
+			fmt.Fprintln(w, "NAME\tSTATE\tURL\tCREDENTIALS\tDESCRIPTION")
+			for _, s := range list {
+				state := "on"
+				if !s.Enabled {
+					state = "off"
+				}
+				var creds []string
+				for h := range s.HeaderRefs {
+					creds = append(creds, h)
+				}
+				fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\n", s.Name, state, s.URL, dash(strings.Join(creds, ",")), s.Description)
+			}
+			return w.Flush()
+		},
+	}
+}
+
+func newMCPRemoveCmd(client func() *ctlClient) *cobra.Command {
+	return &cobra.Command{
+		Use:   "remove NAME",
+		Short: "Remove an MCP server: its tools leave /mcp",
+		Args:  cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			if err := client().delete(cmd.Context(), "/api/v1/mcp/servers/"+args[0], nil); err != nil {
+				return err
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "%s removed; its tools leave /mcp within half a minute\n", args[0])
+			return nil
+		},
+	}
+}
+
+func newMCPTestCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
+	return &cobra.Command{
+		Use:   "test NAME",
+		Short: "Ask an MCP server for its tools, with its credentials, as /mcp would",
+		Args:  cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			var v adminapi.MCPTestView
+			if err := client().post(cmd.Context(), "/api/v1/mcp/servers/"+args[0]+"/test", nil, &v); err != nil {
+				return err
+			}
+			if *asJSON {
+				return json.NewEncoder(cmd.OutOrStdout()).Encode(v)
+			}
+			if !v.OK {
+				return errors.New(args[0] + " did not answer: " + v.Error)
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "%s answers with %d tool(s):\n", args[0], len(v.Tools))
+			for _, t := range v.Tools {
+				fmt.Fprintln(cmd.OutOrStdout(), "  "+t)
+			}
+			return nil
+		},
+	}
+}
+
+// pairs reads NAME=VALUE flags.
+func pairs(list []string, flag string) (map[string]string, error) {
+	if len(list) == 0 {
+		return nil, nil
+	}
+	out := map[string]string{}
+	for _, p := range list {
+		k, v, ok := strings.Cut(p, "=")
+		if !ok || strings.TrimSpace(k) == "" {
+			return nil, fmt.Errorf("%s %q is not NAME=VALUE", flag, p)
+		}
+		out[strings.TrimSpace(k)] = v
+	}
+	return out, nil
+}

internal/cli/guard_judge.go

@@ -1,6 +1,7 @@
 package cli
 
 import (
+	"context"
 	"encoding/json"
 	"errors"
 	"fmt"
@@ -33,7 +34,7 @@ func newGuardJudgeCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 			"then fetches and serves by itself: " + knownJudgeNames() + ".",
 	}
 	c.AddCommand(newGuardJudgeStatusCmd(client, asJSON), newGuardJudgeEnableCmd(client),
-		newGuardJudgeUseCmd(client), newGuardJudgeAskCmd(client, asJSON))
+		newGuardJudgeUseCmd(client), newGuardJudgeAskCmd(client, asJSON), newGuardJudgeCheckCmd(client, asJSON))
 	return c
 }
 
@@ -383,3 +384,70 @@ func newGuardJudgeAskCmd(client func() *ctlClient, asJSON *bool) *cobra.Command
 	c.Flags().StringVar(&answer, "answer", "", "an assistant's reply to TEXT, judged as output is")
 	return c
 }
+
+func newGuardJudgeCheckCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
+	var model string
+	var burst int
+	c := &cobra.Command{
+		Use:   "check",
+		Short: "How a judge keeps up: a judgment alone and among others, against the time a request waits",
+		Long: "Times three judgments of a prompt of about 280 tokens, one after another, then\n" +
+			"a burst of them at once (two for each ready replica unless --burst says),\n" +
+			"and sets both against the policies' latency budget. A judgment past the\n" +
+			"budget is refused under fail: closed; past half of it alone, or past it\n" +
+			"among others, a busy moment refuses requests. Each request is judged once\n" +
+			"for its prompt and again every 600 characters of its answer.",
+		Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			v, err := checkJudge(cmd.Context(), client(), model, burst)
+			if err != nil {
+				return err
+			}
+			if *asJSON {
+				return emitJSON(v)
+			}
+			fmt.Fprint(cmd.OutOrStdout(), judgeCheckText(v))
+			return nil
+		},
+	}
+	c.Flags().StringVar(&model, "model", "", "the judge (default: the one the policies ask)")
+	c.Flags().IntVar(&burst, "burst", 0, "judgments at once (default: two for each ready replica)")
+	return c
+}
+
+func checkJudge(ctx context.Context, c *ctlClient, model string, burst int) (adminapi.JudgeCheckView, error) {
+	body, _ := json.Marshal(adminapi.JudgeCheckRequest{Model: model, Burst: burst})
+	var v adminapi.JudgeCheckView
+	err := c.patient(3*time.Minute).post(ctx, "/api/v1/guard/judges/check", body, &v)
+	return v, err
+}
+
+// judgeCheckText says a check in a few lines, and what to do when it is not
+// fine.
+func judgeCheckText(v adminapi.JudgeCheckView) string {
+	var b strings.Builder
+	took := make([]string, len(v.TookMs))
+	for i, t := range v.TookMs {
+		took[i] = fmt.Sprintf("%.1f", float64(t)/1000)
+	}
+	fmt.Fprintf(&b, "%s, %d replica(s) ready, a request waits %.1f s for it\n", v.Model, v.Ready, float64(v.BudgetMs)/1000)
+	fmt.Fprintf(&b, "  alone:    %s s\n", strings.Join(took, ", "))
+	fmt.Fprintf(&b, "  %d at once: slowest %.1f s", v.Burst, float64(v.BurstSlowestMs)/1000)
+	if v.BurstFailed > 0 {
+		fmt.Fprintf(&b, ", %d gave no verdict", v.BurstFailed)
+	}
+	fmt.Fprintf(&b, "; about %.1f judgments a second, so about %.1f requests a second with short answers\n",
+		v.PerSecond, v.PerSecond/2)
+	switch v.Verdict {
+	case "ok":
+		b.WriteString("ok: well inside the budget\n")
+	case "tight":
+		b.WriteString("tight: a busy moment passes the budget\n")
+	default:
+		b.WriteString("slow: a judgment alone passes the budget\n")
+	}
+	for _, a := range v.Advice {
+		b.WriteString("  - " + a + "\n")
+	}
+	return b.String()
+}

internal/cli/judgekeeper.go

@@ -32,12 +32,34 @@ type judgeKeeper struct {
 	// tried is when each judge was last asked for, so one that cannot be
 	// fetched is not asked for again on every look.
 	tried map[string]time.Time
+	// grace is how long after its first look the keeper serves no judge
+	// itself: init adds its own judge just after the control plane starts,
+	// sized and placed as the install says, and one the leader added first
+	// was what init then kept. Zero waits for nothing.
+	grace time.Duration
+	first time.Time
+
+	// What refitting an earlier install's judge needs: the catalog to change
+	// its profile in, the replicas to restart, and the nodes' CPUs. Any nil
+	// leaves judges as they are.
+	profiles interface {
+		SetProfiles(ctx context.Context, name string, profiles []catalog.Profile, by string) (catalog.Model, error)
+	}
+	restart  func(ids ...string) error
+	replicas func(model string) []string
+	nodeCPUs func() []int
+	// looked is each judge already refitted or found changed by hand, so
+	// each is said once.
+	looked map[string]bool
 }
 
 // judgeRetry is how long a judge whose serving failed waits before it is
 // asked for again.
 const judgeRetry = 10 * time.Minute
 
+// judgeGrace is the keeper's grace on a control plane just started.
+const judgeGrace = 10 * time.Minute
+
 func (k *judgeKeeper) run(ctx context.Context, every time.Duration) {
 	t := time.NewTicker(every)
 	defer t.Stop()
@@ -69,6 +91,13 @@ func (k *judgeKeeper) check() {
 	have := map[string]bool{}
 	for _, m := range k.models() {
 		have[m.Name] = true
+		k.refit(m)
+	}
+	if k.first.IsZero() {
+		k.first = now()
+	}
+	if now().Sub(k.first) < k.grace {
+		return
 	}
 	busy := map[string]bool{}
 	for _, j := range k.serve.Jobs() {
@@ -96,6 +125,15 @@ func (k *judgeKeeper) check() {
 		}
 		k.tried[name] = now()
 		var opts judgeServeOptions
+		if !kj.GPU {
+			// The cores init would give the same number of judges.
+			var cpus []int
+			if k.nodeCPUs != nil {
+				cpus = k.nodeCPUs()
+			}
+			opts.Replicas = 2
+			opts.Cores = refitCores(opts.Replicas, cpus)
+		}
 		if kj.GPU {
 			if opts.GPUClass = roomiestGPUClass(k.gpus()); opts.GPUClass == "" {
 				k.log.Warn("a policy names a judge that runs on a GPU, and this cluster has none",
@@ -105,6 +143,8 @@ func (k *judgeKeeper) check() {
 		}
 		req := kj.serveRequest(opts)
 		req.By = "llmgw (a guard policy names it)"
+		// init, or an operator, may add it while this fetches: theirs stays.
+		req.IfAbsent = true
 		job, err := k.serve.Start(req)
 		if err != nil {
 			k.log.Error("could not start serving a judge a policy names", "judge", name, "err", err)
@@ -114,3 +154,88 @@ func (k *judgeKeeper) check() {
 			"judge", name, "repo", kj.Repo, "file", kj.File, "job", job.ID)
 	}
 }
+
+// refit moves a judge an earlier `server init` installed to what init
+// installs now, and restarts its replicas onto it. Those judges reserved no
+// cores or one, 2 or 4 GB, and ran llama.cpp's prompt cache: on fewer than
+// eight cores every judgment missed its 5 s budget and fail: closed refused
+// every request, until fixed by hand. A judge whose profile differs in any
+// other way was set by somebody, and is left as it is, said once.
+func (k *judgeKeeper) refit(m catalog.Model) {
+	if k.profiles == nil || k.restart == nil || k.replicas == nil || m.Name != defaultJudge || !m.IsGuard() {
+		return
+	}
+	if k.looked == nil {
+		k.looked = map[string]bool{}
+	}
+	if k.looked[m.Name] {
+		return
+	}
+	k.looked[m.Name] = true
+	if !earlierInitJudge(m) {
+		if len(m.Profiles) != 1 || !slices.Contains(m.Profiles[0].Args, "--cache-ram") {
+			k.log.Info("the judge's profile is not one init wrote; left as it is",
+				"judge", m.Name, "see", "llmgw ctl model resources "+m.Name)
+		}
+		return
+	}
+	var cpus []int
+	if k.nodeCPUs != nil {
+		cpus = k.nodeCPUs()
+	}
+	p := m.Profiles[0]
+	p.MinMemoryGB = judgeMemoryGB
+	p.CPUCores = refitCores(m.Scaling.Min, cpus)
+	p.Args = []string{"--ctx-size", "8192", "--cache-ram", "0"}
+	ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
+	defer cancel()
+	if _, err := k.profiles.SetProfiles(ctx, m.Name, []catalog.Profile{p}, "llmgw (the judge an earlier init installed)"); err != nil {
+		k.log.Warn("could not refit the judge an earlier init installed", "judge", m.Name, "err", err)
+		k.looked[m.Name] = false
+		return
+	}
+	ids := k.replicas(m.Name)
+	if len(ids) > 0 {
+		if err := k.restart(ids...); err != nil {
+			k.log.Warn("refitted the judge, but could not restart it", "judge", m.Name, "err", err)
+		}
+	}
+	k.log.Info("refitted the judge an earlier init installed", "judge", m.Name,
+		"memory_gb", p.MinMemoryGB, "cpu_cores", p.CPUCores, "replicas_restarted", len(ids))
+}
+
+// earlierInitJudge is whether a judge's entry is exactly what `server init`
+// before v0.14.7 wrote: one llama.cpp profile on CPUs, an 8192 context and
+// no other args, 2 or 4 GB, none to two cores.
+func earlierInitJudge(m catalog.Model) bool {
+	if len(m.Profiles) != 1 {
+		return false
+	}
+	p := m.Profiles[0]
+	return p.Name == "cpu" && p.Engine == "llamacpp" && p.GPUClass == "" && p.GPU == nil &&
+		len(p.NodeSelector) == 0 && slices.Equal(p.Args, []string{"--ctx-size", "8192"}) &&
+		(p.MinMemoryGB == 2 || p.MinMemoryGB == 4) && p.CPUCores >= 0 && p.CPUCores <= 2
+}
+
+// refitCores is a judge replica's cores by init's rule: up to four for the
+// one judge of a cluster, two each where there are more, fewer where the
+// smallest node cannot hold them all.
+func refitCores(replicas int, cpus []int) int {
+	smallest := 0
+	for _, c := range cpus {
+		if c > 0 && (smallest == 0 || c < smallest) {
+			smallest = c
+		}
+	}
+	if replicas <= 1 {
+		if smallest == 0 {
+			return judgeCores
+		}
+		return min(smallest, judgeCoresAlone)
+	}
+	cores := judgeCores
+	if len(cpus) == 1 && smallest > 0 && replicas*cores > smallest {
+		cores = max(1, smallest/replicas)
+	}
+	return cores
+}

internal/cli/judgekeeper_test.go

@@ -1,6 +1,7 @@
 package cli
 
 import (
+	"context"
 	"errors"
 	"log/slog"
 	"slices"
@@ -51,7 +52,8 @@ func TestAJudgeAPolicyNamesIsFetchedWhenTheCatalogLacksIt(t *testing.T) {
 	// The GPU judge alone: the CPU one is there, and one llmgw does not know
 	// is not guessed at.
 	if len(fs.started) != 1 || fs.started[0].Name != "qwen3guard-4b" || fs.started[0].GPUClass != "6000-24g" ||
-		fs.started[0].Class != catalog.ClassStandard || fs.started[0].GuardFormat != "qwen3guard" || fs.started[0].CPU {
+		fs.started[0].Class != catalog.ClassStandard || fs.started[0].GuardFormat != "qwen3guard" || fs.started[0].CPU ||
+		!fs.started[0].IfAbsent {
 		t.Fatalf("started %+v", fs.started)
 	}
 
@@ -90,8 +92,12 @@ func TestAJudgeAPolicyNamesIsFetchedWhenTheCatalogLacksIt(t *testing.T) {
 	lib = guard.Config{Policies: map[string]guard.Policy{"b": {Judge: &guard.JudgePolicy{Model: "qwen3guard-0.6b", Input: "block"}}}}
 	inCatalog = nil
 	k.gpus = func() []adminapi.ClassUse { return nil }
+	k.nodeCPUs = func() []int { return []int{8, 4} }
 	k.check()
-	if last := fs.started[len(fs.started)-1]; last.Name != "qwen3guard-0.6b" || !last.CPU || last.Replicas != 2 {
+	// Sized as init sizes its two judges: 2 GB, and two cores each on a
+	// cluster whose smallest node has four.
+	if last := fs.started[len(fs.started)-1]; last.Name != "qwen3guard-0.6b" || !last.CPU || last.Replicas != 2 ||
+		last.MemoryGB != judgeMemoryGB || last.CPUCores != refitCores(2, []int{8, 4}) || last.CPUCores == 0 {
 		t.Errorf("CPU judge: %+v", last)
 	}
 	fs.err = errors.New("the mirror is down")
@@ -124,3 +130,108 @@ func TestCategoriesFollowTheJudges(t *testing.T) {
 		t.Errorf("no categories (every one counts) became %v", got)
 	}
 }
+
+type fakeProfiles struct {
+	set map[string][]catalog.Profile
+}
+
+func (f *fakeProfiles) SetProfiles(_ context.Context, name string, ps []catalog.Profile, _ string) (catalog.Model, error) {
+	if f.set == nil {
+		f.set = map[string][]catalog.Profile{}
+	}
+	f.set[name] = ps
+	return catalog.Model{Name: name, Profiles: ps}, nil
+}
+
+// A judge that v0.14.6 or earlier's init installed is moved to today's
+// figures, once, and its replicas restarted onto them; one somebody changed
+// is left alone.
+func TestAnEarlierInstallsJudgeIsRefittedOnce(t *testing.T) {
+	lib := guard.Config{Policies: map[string]guard.Policy{
+		"baseline": {Judge: &guard.JudgePolicy{Model: "qwen3guard-0.6b", Input: "block"}},
+	}}
+	earlier := guardModel("http://10.0.0.5:8081/qwen3guard-0.6b/guard/g.gguf", 2, 1, nil)
+	earlier.Profiles[0].MinMemoryGB, earlier.Profiles[0].Args = 4, []string{"--ctx-size", "8192"}
+	inCatalog := []catalog.Model{earlier}
+	fp := &fakeProfiles{}
+	var restarted []string
+	k := &judgeKeeper{
+		library:  func() guard.Config { return lib },
+		models:   func() []catalog.Model { return inCatalog },
+		serve:    &fakeServe{},
+		log:      slog.New(slog.DiscardHandler),
+		profiles: fp,
+		restart:  func(ids ...string) error { restarted = append(restarted, ids...); return nil },
+		replicas: func(string) []string { return []string{"j-1", "j-2"} },
+		nodeCPUs: func() []int { return []int{4} },
+	}
+	k.check()
+	got := fp.set["qwen3guard-0.6b"]
+	if len(got) != 1 || got[0].MinMemoryGB != 2 || got[0].CPUCores != 2 || !slices.Contains(got[0].Args, "--cache-ram") {
+		t.Fatalf("refitted to %+v", got)
+	}
+	if !slices.Equal(restarted, []string{"j-1", "j-2"}) {
+		t.Errorf("restarted %v", restarted)
+	}
+	k.check()
+	if len(restarted) != 2 {
+		t.Error("refitted twice")
+	}
+
+	// Changed by somebody: an extra arg. Left as it is.
+	mine := guardModel("http://x/g.gguf", 2, 3, nil)
+	mine.Profiles[0].Args = []string{"--ctx-size", "16384"}
+	inCatalog = []catalog.Model{mine}
+	fp.set, k.looked = nil, nil
+	k.check()
+	if fp.set != nil {
+		t.Errorf("a judge set by hand was changed: %+v", fp.set)
+	}
+}
+
+func TestAJudgeIsRefittedToTheCoresInitWouldGive(t *testing.T) {
+	for _, c := range []struct {
+		replicas int
+		cpus     []int
+		want     int
+	}{
+		{1, []int{4}, 4},
+		{1, []int{2}, 2},
+		{1, []int{16}, 4},
+		{2, []int{4}, 2},
+		{2, []int{2}, 1},
+		{2, []int{4, 32}, 2},
+		{2, nil, 2},
+	} {
+		if got := refitCores(c.replicas, c.cpus); got != c.want {
+			t.Errorf("%d judges on %v: %d cores, want %d", c.replicas, c.cpus, got, c.want)
+		}
+	}
+}
+
+// Just after the control plane starts, the judge init is adding is left to
+// init: the keeper serves one itself only after its grace.
+func TestTheKeeperLeavesAJustStartedInstallsJudgeToInit(t *testing.T) {
+	lib := guard.Config{Policies: map[string]guard.Policy{"b": {Judge: &guard.JudgePolicy{Model: "qwen3guard-0.6b", Input: "block"}}}}
+	now := time.Date(2026, 10, 19, 12, 0, 0, 0, time.UTC)
+	fs := &fakeServe{}
+	k := &judgeKeeper{
+		library: func() guard.Config { return lib },
+		models:  func() []catalog.Model { return nil },
+		serve:   fs, grace: judgeGrace,
+		gpus: func() []adminapi.ClassUse { return nil },
+		log:  slog.New(slog.DiscardHandler),
+		now:  func() time.Time { return now },
+	}
+	k.check()
+	now = now.Add(judgeGrace / 2)
+	k.check()
+	if len(fs.started) != 0 {
+		t.Fatalf("served within its grace: %+v", fs.started)
+	}
+	now = now.Add(judgeGrace)
+	k.check()
+	if len(fs.started) != 1 {
+		t.Errorf("never served after its grace: %d", len(fs.started))
+	}
+}

internal/cli/judges_known.go

@@ -142,7 +142,9 @@ func (k knownJudge) serveRequest(o judgeServeOptions) modelserve.Request {
 		}
 		return req
 	}
-	req.CPU, req.CPUCores = true, o.Cores
+	// Sized as init sizes its judge: the memory it measured to need with
+	// no prompt cache, and the cores its rule gives.
+	req.CPU, req.CPUCores, req.MemoryGB = true, o.Cores, judgeMemoryGB
 	if req.Replicas == 0 {
 		req.Replicas = 2
 	}

internal/cli/keys.go

@@ -7,6 +7,7 @@ import (
 	"errors"
 	"fmt"
 	"math"
+	"path"
 	"slices"
 	"strings"
 	"text/tabwriter"
@@ -301,7 +302,8 @@ func newLimitsCmd(open opener) *cobra.Command {
 	var perMember bool
 	var rpm, tpm int
 	var budget float64
-	var models []string
+	var models, tools []string
+	var replace bool
 
 	cmd := &cobra.Command{
 		Use:   "limits",
@@ -357,7 +359,19 @@ func newLimitsCmd(open opener) *cobra.Command {
 					return errors.New("--per-member is for --scope org, team or project")
 				}
 			}
+			// Only what is given changes: setting a tool grant must not
+			// clear the scope's rate or budget. --replace writes the row
+			// whole, a flag not given then no limit, as the console's
+			// form does with every field it shows.
 			l := store.Limits{Scope: level, ScopeID: scopeID}
+			if !replace {
+				switch cur, err := s.LimitsFor(cmd.Context(), level, scopeID); {
+				case err == nil:
+					l = cur
+				case !errors.Is(err, store.ErrNotFound):
+					return err
+				}
+			}
 			if cmd.Flags().Changed("rpm") {
 				l.RPM = &rpm
 			}
@@ -379,6 +393,14 @@ func newLimitsCmd(open opener) *cobra.Command {
 				}
 				l.AllowedModels = models
 			}
+			if cmd.Flags().Changed("tools") {
+				for _, p := range tools {
+					if _, err := path.Match(p, "x.y"); err != nil {
+						return fmt.Errorf("--tools %q is not a pattern: %w", p, err)
+					}
+				}
+				l.AllowedTools = tools
+			}
 			if err := s.PutLimits(cmd.Context(), l); err != nil {
 				return err
 			}
@@ -395,6 +417,10 @@ func newLimitsCmd(open opener) *cobra.Command {
 	cmd.Flags().Float64Var(&budget, "budget", 0, "budget per period, in whole units as the quotas page shows them")
 	cmd.Flags().StringVar(&period, "period", "month", "day | month")
 	cmd.Flags().StringSliceVar(&models, "models", nil, "allowed models; empty allows none")
+	cmd.Flags().BoolVar(&replace, "replace", false,
+		"write every limit at the scope: one not given is then no limit there (default: only what is given changes)")
+	cmd.Flags().StringSliceVar(&tools, "tools", nil,
+		"the company MCP servers' tools granted here, as patterns: jira.*,db.query; none granted anywhere is none")
 	return cmd
 }
 

internal/cli/model_serve.go

@@ -69,6 +69,8 @@ func newModelServeCmd(client func() *ctlClient) *cobra.Command {
 	f.IntVar(&req.Nodes, "nodes", 0, "with --gpu-class: the nodes one replica spans, over a fabric (vLLM only)")
 	f.IntVar(&req.TP, "tp", 0, "the older name of --gpus")
 	_ = f.MarkHidden("tp")
+	f.StringToStringVar(&req.Env, "env", nil,
+		"KEY=VALUE in the engine's environment on every node, e.g. NCCL_DEBUG=INFO; repeatable")
 	f.BoolVar(&req.Fallbacks, "fallbacks", false, "also add the next best ways to run it, used when the first is busy")
 	f.BoolVar(&detach, "detach", false, "start it and return: it runs on the control plane; `llmgw ctl jobs` follows it")
 	f.StringVar(&req.Name, "name", "", "the catalog name (default: from the repository)")

internal/cli/modelplan.go

@@ -291,14 +291,14 @@ func planFiles(cmd *cobra.Command, client func() *ctlClient, repo, format string
 	}
 	fmt.Fprintln(out)
 	w := tabTo(out)
-	fmt.Fprintln(w, "FILE\tSIZE\tBEST WAY\tHOLDS\tOF\tNOW")
+	fmt.Fprintln(w, "FILE\tSIZE\tBEST WAY\tA CARD\tOF\tALL CARDS\tNOW")
 	for _, a := range p.Artifacts {
 		name := a.Name
 		if a.Shards > 1 && a.Format == "gguf" {
 			name += fmt.Sprintf(" (%d parts)", a.Shards)
 		}
 		if a.Best == nil {
-			fmt.Fprintf(w, "%s\t%s\tfits nowhere\t\t\t\n", name, bytesShort(a.Bytes))
+			fmt.Fprintf(w, "%s\t%s\tfits nowhere\t\t\t\t\n", name, bytesShort(a.Bytes))
 			continue
 		}
 		b := a.Best
@@ -306,11 +306,11 @@ func planFiles(cmd *cobra.Command, client func() *ctlClient, repo, format string
 		if b.FreeNow {
 			now = "free"
 		}
-		holds := fmt.Sprintf("%.1f GB a card", b.NeedGB)
+		holds, all := fmt.Sprintf("%.1f GB", b.NeedGB), fmt.Sprintf("%.0f GB", b.TotalGB())
 		if b.CPU {
-			holds = fmt.Sprintf("%.1f GB of RAM", b.NeedGB)
+			holds, all = fmt.Sprintf("%.1f GB of RAM", b.NeedGB), "-"
 		}
-		fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%.0f GB\t%s\n", name, bytesShort(a.Bytes), b.Label(), holds, b.HaveGB, now)
+		fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%.0f GB\t%s\t%s\n", name, bytesShort(a.Bytes), b.Label(), holds, b.HaveGB, all, now)
 	}
 	_ = w.Flush()
 	rec, ok := p.Recommended()
@@ -318,7 +318,11 @@ func planFiles(cmd *cobra.Command, client func() *ctlClient, repo, format string
 		fmt.Fprintln(out, "\nNo file fits this cluster as it is: a smaller context (--context), or more memory.")
 		return errDoesNotFit
 	}
-	fmt.Fprintf(out, "\nBest: %s on %s.\n", rec.Artifact, rec.Label())
+	fmt.Fprintf(out, "\nBest: %s on %s: %s.\n", rec.Artifact, rec.Label(), rec.MemoryText())
+	if p.KVBytesPerToken > 0 {
+		fmt.Fprintf(out, "The context costs %.2f GB per 1000 tokens, the whole model (--context to plan another).\n",
+			modelplan.GB(p.KVBytesPerToken*1000))
+	}
 	if len(p.Options) > 1 {
 		fmt.Fprintln(out, "\nEvery way, best first:")
 		w = tabTo(out)
@@ -327,7 +331,7 @@ func planFiles(cmd *cobra.Command, client func() *ctlClient, repo, format string
 			if o.FreeNow {
 				now = "free"
 			}
-			fmt.Fprintf(w, "  %s\t%s\t%.1f of %.0f GB\t%d host(s)\t%s\n", o.Artifact, o.Label(), o.NeedGB, o.HaveGB, o.Hosts, now)
+			fmt.Fprintf(w, "  %s\t%s\t%s\t%d host(s)\t%s\n", o.Artifact, o.Label(), o.MemoryText(), o.Hosts, now)
 		}
 		_ = w.Flush()
 	}

internal/cli/org_test.go

@@ -131,3 +131,33 @@ func TestKeyBudgetsAreWholeUnits(t *testing.T) {
 		}
 	}
 }
+
+// A limit given alone changes that limit alone: granting a tool keeps the
+// scope's rate. --replace writes the row whole, as the console's form does.
+func TestALimitGivenAloneLeavesTheOthersAtTheScope(t *testing.T) {
+	db := filepath.Join(t.TempDir(), "s.db")
+	mustRun(t, db, "org", "add", "org", "Acme")
+	mustRun(t, db, "org", "add", "team", "platform")
+	mustRun(t, db, "org", "add", "project", "gateway")
+	mustRun(t, db, "keys", "limits", "--scope", "project", "--id", "gateway", "--rpm", "30")
+	mustRun(t, db, "keys", "limits", "--scope", "project", "--id", "gateway", "--tools", "jira.*")
+	read := func() store.Limits {
+		s, err := store.Open(context.Background(), db)
+		if err != nil {
+			t.Fatal(err)
+		}
+		defer s.Close()
+		all, _ := s.AllLimits(context.Background())
+		for _, l := range all {
+			return l
+		}
+		return store.Limits{}
+	}
+	if l := read(); l.RPM == nil || *l.RPM != 30 || len(l.AllowedTools) != 1 {
+		t.Errorf("after a tool grant: rpm %v, tools %v", l.RPM, l.AllowedTools)
+	}
+	mustRun(t, db, "keys", "limits", "--replace", "--scope", "project", "--id", "gateway", "--tpm", "1000")
+	if l := read(); l.RPM != nil || l.AllowedTools != nil || l.TPM == nil {
+		t.Errorf("after --replace: %+v", l)
+	}
+}

internal/cli/server.go

@@ -76,6 +76,7 @@ func newServerCmd() *cobra.Command {
 
 	cmd := &cobra.Command{
 		Use:   "server",
+		Args:  cobra.NoArgs,
 		Short: "Run the router (phase 1; the control plane arrives in phase 3)",
 		Long: "Serves the OpenAI API in front of a set of engine replicas.\n\n" +
 			"In this phase the replica list is static: give it in the config file, or\n" +
@@ -138,6 +139,26 @@ func newServerCmd() *cobra.Command {
 				_ = shutdownTracing(sctx)
 			}()
 
+			// The ports before anything else: a second server on this machine,
+			// whose ports are taken, must stop here, not go on to compete for
+			// the lease and lead a cluster no request can reach.
+			apiLn, err := net.Listen("tcp", cfg.Listen)
+			if err != nil {
+				return fmt.Errorf("listening on %s: %w", cfg.Listen, err)
+			}
+			if cfg.MaxConnections > 0 {
+				// Past this, new connections wait in the kernel's backlog
+				// rather than each taking a goroutine and its buffers.
+				apiLn = netutil.LimitListener(apiLn, cfg.MaxConnections)
+			}
+			adminLn, err := net.Listen("tcp", cfg.AdminListen)
+			if err != nil {
+				_ = apiLn.Close()
+				return fmt.Errorf("listening on %s: %w", cfg.AdminListen, err)
+			}
+			defer apiLn.Close()
+			defer adminLn.Close()
+
 			rt, err := router.New(cfg, log)
 			if err != nil {
 				return err
@@ -418,6 +439,9 @@ func newServerCmd() *cobra.Command {
 					}
 					// The cluster's settings, each change checked against this
 					// machine's own keys before it is kept.
+					api.SetMCP(st)
+					rt.SetAdminMCP(http.HandlerFunc(api.MCP))
+					api.SetAuth(&adminapi.AuthAdmin{Status: rt.AuthStatus, TestLDAP: rt.TestDirectory, Secrets: st, Log: log})
 					api.SetSettings(&adminapi.Settings{Store: st, LocalKeys: config.LocalKeys(),
 						Check: func(body []byte) error { return checkClusterSettings(configPath, body) }})
 					// And the same document on the portal, for administrators.
@@ -585,6 +609,7 @@ func newServerCmd() *cobra.Command {
 						// A judge a policy names and the catalog lacks is
 						// fetched and served, on the leader.
 						keeper := &judgeKeeper{
+							grace:   judgeGrace,
 							library: gate.Library,
 							models:  func() []catalog.Model { return admin.Catalog().Models() },
 							serve:   serve,
@@ -593,6 +618,25 @@ func newServerCmd() *cobra.Command {
 							},
 							gpus: func() []adminapi.ClassUse { return adminAPI.Summary().GPUs },
 							log:  log,
+							// An earlier install's judge, moved to today's figures.
+							profiles: catalogSvc,
+							restart:  admin.Reconciler().Restart,
+							replicas: func(model string) []string {
+								var ids []string
+								for _, r := range admin.Reconciler().ReadyReplicas() {
+									if r.Model == model {
+										ids = append(ids, r.ID)
+									}
+								}
+								return ids
+							},
+							nodeCPUs: func() []int {
+								var out []int
+								for _, n := range admin.Nodes() {
+									out = append(out, n.CPUCount)
+								}
+								return out
+							},
 						}
 						go keeper.run(ctx, 30*time.Second)
 					}
@@ -642,21 +686,6 @@ func newServerCmd() *cobra.Command {
 				ReadTimeout: time.Minute,
 			})
 
-			apiLn, err := net.Listen("tcp", cfg.Listen)
-			if err != nil {
-				return fmt.Errorf("listening on %s: %w", cfg.Listen, err)
-			}
-			if cfg.MaxConnections > 0 {
-				// Past this, new connections wait in the kernel's backlog
-				// rather than each taking a goroutine and its buffers.
-				apiLn = netutil.LimitListener(apiLn, cfg.MaxConnections)
-			}
-			adminLn, err := net.Listen("tcp", cfg.AdminListen)
-			if err != nil {
-				_ = apiLn.Close()
-				return fmt.Errorf("listening on %s: %w", cfg.AdminListen, err)
-			}
-
 			// A metrics-only listener, when one is configured. Started after
 			// AdminHandler, which is where the registry it serves is built.
 			var metricsSrv *http.Server

internal/cli/server_args_test.go

@@ -0,0 +1,39 @@
+package cli
+
+import (
+	"context"
+	"net"
+	"strings"
+	"testing"
+	"time"
+)
+
+// `llmgw server check` once ran a whole second server: words after server
+// that name no subcommand are refused.
+func TestServerTakesNoWordsItDoesNotKnow(t *testing.T) {
+	cmd := newServerCmd()
+	cmd.SetArgs([]string{"check", "--config", "/nonexistent"})
+	cmd.SilenceUsage, cmd.SilenceErrors = true, true
+	if err := cmd.Execute(); err == nil || !strings.Contains(err.Error(), "unknown command") {
+		t.Errorf("err %v", err)
+	}
+}
+
+// A second server whose port is taken stops at once, before the router or
+// any control plane is built.
+func TestAServerWhosePortIsTakenStopsFirst(t *testing.T) {
+	taken, err := net.Listen("tcp", "127.0.0.1:0")
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer taken.Close()
+	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+	defer cancel()
+	cmd := newServerCmd()
+	cmd.SetArgs([]string{"--listen", taken.Addr().String(), "--admin-listen", "127.0.0.1:0",
+		"--endpoint", "a=http://127.0.0.1:1=m"})
+	cmd.SilenceUsage, cmd.SilenceErrors = true, true
+	if err := cmd.ExecuteContext(ctx); err == nil || !strings.Contains(err.Error(), "listening on") {
+		t.Errorf("err %v", err)
+	}
+}

internal/cli/serverinit.go

@@ -80,6 +80,13 @@ type serverInit struct {
 	// setup screen does with the password it was given; when set, patroni
 	// only names them.
 	patroniNodes []pgha.Node
+	// judgeMachines are machines for the guardrails' judges alone, added as
+	// nodes labelled for them; judgeCPUs their cores, once added.
+	judgeMachines []judgeMachine
+	judgeCPUs     []int
+	// judgeNote is what timing the judge found, said before the next
+	// steps when it is not fine.
+	judgeNote string
 	// noMirror, noNode and noModel turn off the parts of a first install that
 	// make it serve something: a mirror, this machine as a node, and a small
 	// model on it.
@@ -234,6 +241,8 @@ func newServerInitCmd() *cobra.Command {
 	f.StringVar(&judgeCats, "judge-categories", strings.Join(defaultJudgeCategories, ","),
 		"what the judge refuses, comma-separated, of: "+strings.Join(judgeCategories, ", "))
 	f.IntVar(&in.judge.Replicas, "judge-replicas", 2, "how many copies of the judge run, each on CPUs")
+	f.Var(judgeMachineFlag{&in.judgeMachines}, "judge-node",
+		"user@host of a machine for the guardrails' judges alone: added as a node, and the judges run only there; repeatable")
 	f.DurationVar(&in.judge.Budget, "judge-budget", 5*time.Second, "how long a request waits for the judge")
 	f.StringVar(&in.language, "language", "", "the portal's language for a browser that asks for neither: en or tr")
 	f.BoolVar(&in.signupOpen, "signup-open", false, "let anyone ask for an account on the portal, to be approved by an administrator")
@@ -257,6 +266,9 @@ func (in *serverInit) run(ctx context.Context) error {
 	if err := in.checkBundle(); err != nil {
 		return fmt.Errorf("offline bundle: %w", err)
 	}
+	if err := in.checkMemory(); err != nil {
+		return err
+	}
 	steps := []struct {
 		name string
 		fn   func(context.Context) error
@@ -317,6 +329,11 @@ func (in *serverInit) run(ctx context.Context) error {
 	if err := in.ensureWebhooks(ctx); err != nil {
 		fmt.Fprintf(in.out, "  !!    webhooks: %v\n", err)
 	}
+	if hasSystemd && len(in.judgeMachines) > 0 {
+		if err := in.ensureJudgeNodes(ctx); err != nil {
+			return fmt.Errorf("judge nodes: %w", err)
+		}
+	}
 	if in.judgeWanted() && in.configJudgesWith(guardModelName) {
 		// Not fatal: without its judge the baseline policy still checks for
 		// secrets and injection, and a judged request is served flagged.
@@ -338,6 +355,9 @@ func (in *serverInit) run(ctx context.Context) error {
 			return fmt.Errorf("standby control planes: %w", err)
 		}
 	}
+	if hasSystemd && in.judgeWanted() && in.configJudgesWith(guardModelName) {
+		in.checkJudge(ctx)
+	}
 	in.nextSteps()
 	return nil
 }
@@ -902,6 +922,9 @@ func (in *serverInit) waitHealthy(ctx context.Context) error {
 }
 
 func (in *serverInit) nextSteps() {
+	if in.judgeNote != "" {
+		fmt.Fprint(in.out, "\n"+in.judgeNote)
+	}
 	fmt.Fprint(in.out, `
 The control plane is up. Next:
 

internal/cli/serverinit_choices.go

@@ -4,6 +4,7 @@ import (
 	"fmt"
 	"math"
 	"net"
+	"os"
 	"path"
 	"slices"
 	"strconv"
@@ -11,6 +12,8 @@ import (
 	"time"
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/pgha"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/preflight"
 )
 
 // What a first install lets a person choose beyond which parts it has: which
@@ -154,6 +157,62 @@ func (s starterSpec) minMemoryGB() int {
 	return max(2, int(math.Ceil(s.SizeGB+1.5)))
 }
 
+// servingMemoryMB is what this machine needs, beyond what the control plane
+// does, for the models it will serve itself: the starter's and the judges'
+// reservations, and a gigabyte more because the scheduler counts a machine's
+// memory in whole gigabytes, rounded down — 7.75 GB is 7. Zero when it serves
+// none of them; what names those it does.
+//
+// Asked before anything is installed: the 8 GB machine the guide was first
+// recorded on finished every step and then had no room for the starter, the
+// one thing a person asks first.
+func (in *serverInit) servingMemoryMB(judgesElsewhere bool) (mb int, what string) {
+	if in.noNode {
+		return 0, ""
+	}
+	gb := 0
+	var parts []string
+	if !in.noModel && (!in.noMirror || in.mirrorURL != "") {
+		gb += in.starterModelSpec().minMemoryGB()
+		parts = append(parts, "the starter model")
+	}
+	if !in.noGuardModel && !judgesElsewhere {
+		n := in.judgeFor().Replicas
+		gb += n * judgeMemoryGB
+		parts = append(parts, fmt.Sprintf("%d judges", n))
+	}
+	if gb == 0 {
+		return 0, ""
+	}
+	return (gb + 1) * 1024, strings.Join(parts, " and ")
+}
+
+// checkMemory refuses, before anything is installed, a machine too small for
+// the models it would serve as well as the control plane.
+func (in *serverInit) checkMemory() error {
+	if in.unitPath == "" {
+		return nil // a development run, served by nothing
+	}
+	meminfo, err := os.ReadFile("/proc/meminfo")
+	if err != nil {
+		return nil
+	}
+	total, ok := pgha.MemTotal(string(meminfo))
+	if !ok {
+		return nil
+	}
+	extra, what := in.servingMemoryMB(len(in.judgeMachines) > 0)
+	if extra == 0 {
+		return nil
+	}
+	have, need := int(total>>20), preflight.Needs[preflight.RoleControl].MemoryMB+extra
+	if have >= need {
+		return nil
+	}
+	return fmt.Errorf("memory: this machine has %d MB; the control plane and %s it would serve need %d. "+
+		"Give it more, or leave them off it: --no-model, --no-guard-model, or --no-node to serve models only on nodes added later", have, what, need)
+}
+
 // defaultContext is the context a client may use of the starter.
 const defaultContext = 4096
 
@@ -194,6 +253,9 @@ func (in *serverInit) judgeFor() judgeSpec {
 	}
 	if j.Replicas <= 0 {
 		j.Replicas = 2
+		if in.singleMachine() {
+			j.Replicas = 1
+		}
 	}
 	if j.Budget <= 0 {
 		j.Budget = 5 * time.Second
@@ -201,6 +263,14 @@ func (in *serverInit) judgeFor() judgeSpec {
 	return j
 }
 
+// singleMachine is an install where this machine is the only node there
+// will be, as far as init knows: a second judge replica there shares the
+// same cores and is lost with the same machine, so it buys nothing but
+// half the speed of each.
+func (in *serverInit) singleMachine() bool {
+	return !in.noNode && !in.standbys && len(in.patroni) == 0 && len(in.patroniNodes) == 0
+}
+
 // The configuration's limits, as init writes them unless told otherwise.
 const (
 	defaultRPM       = 60
@@ -314,6 +384,9 @@ func (in *serverInit) initFlags() []string {
 	for _, t := range in.patroni {
 		add("--patroni", t)
 	}
+	for _, m := range in.judgeMachines {
+		add("--judge-node", m.target)
+	}
 	switch {
 	case in.mirrorURL != "":
 		add("--mirror", in.mirrorURL)
@@ -346,7 +419,10 @@ func (in *serverInit) initFlags() []string {
 	if in.noGuardModel {
 		add("--no-guard-model")
 	} else {
-		j, d := in.judgeFor(), (&serverInit{}).judgeFor()
+		// Against the defaults for an install of the same shape: one judge
+		// on one machine, two otherwise.
+		shape := serverInit{noNode: in.noNode, standbys: in.standbys, patroni: in.patroni, patroniNodes: in.patroniNodes}
+		j, d := in.judgeFor(), shape.judgeFor()
 		if j.Fail != d.Fail {
 			add("--judge-fail", j.Fail)
 		}

internal/cli/serverinit_choices_test.go

@@ -183,12 +183,17 @@ func TestTheJudgesAndTheStarterShareTheCPUsRatherThanEachTakingAll(t *testing.T)
 		in             serverInit
 		judge, starter int
 	}{
-		{serverInit{cpus: 4}, 1, 2},
-		{serverInit{cpus: 8}, 2, 4},
-		{serverInit{cpus: 32}, 2, 8},
-		{serverInit{cpus: 2}, 1, 0}, // nothing left: the starter shares rather than fits nowhere
+		// One machine: one judge, on up to four cores; the starter has the rest.
+		{serverInit{cpus: 4}, 4, 0},
+		{serverInit{cpus: 8}, 4, 4},
+		{serverInit{cpus: 32}, 4, 8},
+		{serverInit{cpus: 2}, 2, 0},
 		{serverInit{cpus: 1}, 0, 0},
-		{serverInit{cpus: 4, noGuardModel: true}, 1, 4},
+		{serverInit{cpus: 4, noGuardModel: true}, judgeCores, 4}, // no judge: all for the starter
+		// Two asked for on one machine: two cores each, one each where two
+		// would not fit both.
+		{serverInit{cpus: 8, judge: judgeSpec{Replicas: 2}}, 2, 4},
+		{serverInit{cpus: 2, judge: judgeSpec{Replicas: 2}}, 1, 0},
 		{serverInit{cpus: 8, judge: judgeSpec{Replicas: 3}}, 2, 2},
 		{serverInit{cpus: 4, noNode: true}, 2, 0}, // on nodes not known yet
 	} {
@@ -197,12 +202,40 @@ func TestTheJudgesAndTheStarterShareTheCPUsRatherThanEachTakingAll(t *testing.T)
 			t.Errorf("%+v: judge %d starter %d, want %d %d", c.in, got.judge, got.starter, c.judge, c.starter)
 		}
 	}
-	m := guardModel("http://x/g.gguf", 2, 1)
-	if p := m.Profiles[0]; p.CPUCores != 1 || p.MinMemoryGB != 4 {
-		t.Errorf("judge profile: %d cores, %d GB", p.CPUCores, p.MinMemoryGB)
+	m := guardModel("http://x/g.gguf", 2, 1, nil)
+	if p := m.Profiles[0]; p.CPUCores != 1 || p.MinMemoryGB != 2 || !slices.Contains(p.Args, "--cache-ram") {
+		t.Errorf("judge profile: %d cores, %d GB, args %v", p.CPUCores, p.MinMemoryGB, p.Args)
 	}
 	s := starterModel(starterChoices[0], defaultContext, 2, "http://x/q.gguf", []string{"6000-24g"}, true)
 	if cpu := s.Profiles[len(s.Profiles)-1]; cpu.Name != "cpu" || cpu.CPUCores != 2 || s.Profiles[0].CPUCores != 0 {
 		t.Errorf("starter: GPU profile %d cores, CPU %+v", s.Profiles[0].CPUCores, cpu)
 	}
 }
+
+// A second judge on the same machine shares its cores and is lost with it:
+// one machine has one, more machines two.
+func TestOneMachineHasOneJudgeAndMoreHaveTwo(t *testing.T) {
+	for _, c := range []struct {
+		in   serverInit
+		want int
+	}{
+		{serverInit{}, 1},
+		{serverInit{noNode: true}, 2},
+		{serverInit{patroni: []string{"a@db1", "b@db2"}}, 2},
+		{serverInit{standbys: true}, 2},
+		{serverInit{judge: judgeSpec{Replicas: 3}}, 3},
+	} {
+		if got := c.in.judgeFor().Replicas; got != c.want {
+			t.Errorf("%+v: %d judges, want %d", c.in, got, c.want)
+		}
+	}
+	// The defaults are left out of the command; a choice against them is in.
+	ha := serverInit{noNode: true, patroni: []string{"a@db1", "b@db2"}, judge: judgeSpec{Replicas: 2}}
+	if f := strings.Join(ha.initFlags(), " "); strings.Contains(f, "--judge-replicas") {
+		t.Errorf("two judges on three machines named: %s", f)
+	}
+	one := serverInit{judge: judgeSpec{Replicas: 2}}
+	if f := strings.Join(one.initFlags(), " "); !strings.Contains(f, "--judge-replicas 2") {
+		t.Errorf("two judges on one machine not named: %s", f)
+	}
+}

internal/cli/serverinit_cluster.go

@@ -485,7 +485,7 @@ func starterModel(spec starterSpec, ctx, cores int, source string, gpuClasses []
 
 // guardModel is the judge's catalog entry: CPU only, so it takes no card from
 // the models people use, on whichever node has room; replicas copies of it.
-func guardModel(source string, replicas, cores int) catalog.Model {
+func guardModel(source string, replicas, cores int, selector map[string]string) catalog.Model {
 	return catalog.Model{
 		Name:        guardModelName,
 		Description: "The guardrails' judge: reads prompts and answers for unsafe content. Not offered to clients.",
@@ -496,10 +496,15 @@ func guardModel(source string, replicas, cores int) catalog.Model {
 		// judge that is not there refuses everything.
 		Scaling: catalog.Scaling{Min: replicas, Max: replicas},
 		Profiles: []catalog.Profile{{
-			// It uses about 3.2 GB with its context: reserving 2 filled nodes
-			// past their memory.
-			Name: "cpu", Engine: "llamacpp", Priority: 100, MinMemoryGB: 4, CPUCores: cores,
-			Args: []string{"--ctx-size", "8192"},
+			// About 1.6 GB with its context, which stays so with no prompt
+			// cache: llama.cpp keeps up to 8 GB of prompts by default, and a
+			// judge's, each a different text, are not asked again. With the
+			// cache one grew to 3.2 GB over a day of traffic; reserving 4 for
+			// that left a 7 GB machine room for only one of the two.
+			Name: "cpu", Engine: "llamacpp", Priority: 100, MinMemoryGB: judgeMemoryGB, CPUCores: cores,
+			Args: []string{"--ctx-size", "8192", "--cache-ram", "0"},
+			// The judge machines, when the install was given some.
+			NodeSelector: selector,
 		}},
 	}
 }
@@ -521,7 +526,8 @@ func (in *serverInit) ensureStarter(ctx context.Context) error {
 func (in *serverInit) ensureGuardModel(ctx context.Context) error {
 	return in.ensureServed(ctx, guardModelName, guardModelSource, guardModelVersion, false,
 		func(url string, _ []string) catalog.Model {
-			return guardModel(url, in.judgeFor().Replicas, in.cpuPlan().judge)
+			cores, selector := in.judgePlacement()
+			return guardModel(url, in.judgeFor().Replicas, cores, selector)
 		})
 }
 
@@ -716,14 +722,16 @@ type cpuSplit struct{ judge, starter int }
 // the starter on one 4-core control plane all ran on all four: under load
 // most judgments missed their budget and the requests were refused.
 //
-// When this machine is the node, by its own CPUs: a core a judge under 8,
-// two from 8, and the starter what is left, up to 8, or no reservation at
-// all when nothing is, rather than a starter that fits nowhere. Otherwise
-// two a judge, on whichever node takes it, and the starter, on a node not
-// known yet, none.
+// A judge needs cores: a judgment of about 280 tokens took 8.0 s on one,
+// 4.2 s on two and 2.3 s on four, against a 5 s budget. On one machine its
+// one replica gets up to four; where there are two, two each (one each only
+// where two would not fit both). The starter has what is left, up to 8, or
+// no reservation at all when nothing is, rather than a starter that fits
+// nowhere. A machine that is not the node reserves two a judge on whichever
+// node takes it, and the starter, on a node not known yet, none.
 func (in *serverInit) cpuPlan() cpuSplit {
 	if in.noNode {
-		return cpuSplit{judge: 2}
+		return cpuSplit{judge: judgeCores}
 	}
 	n := in.cpus
 	if n <= 0 {
@@ -736,13 +744,23 @@ func (in *serverInit) cpuPlan() cpuSplit {
 	if !in.noGuardModel {
 		judges = in.judgeFor().Replicas
 	}
-	judge := 1
-	if n >= 8 {
-		judge = 2
-	}
-	left := n - judges*judge
-	if left < 1 {
-		left = 0
+	judge := judgeCores
+	switch {
+	case judges == 1:
+		judge = min(n, judgeCoresAlone)
+	case judges > 0 && judges*judge > n:
+		judge = max(1, n/judges)
 	}
+	left := max(n-judges*judge, 0)
 	return cpuSplit{judge: judge, starter: min(left, 8)}
 }
+
+// judgeCores and judgeMemoryGB are what a judge replica needs: two cores
+// for a judgment within its budget, and 2 GB for its 1.6 with no prompt
+// cache. judgeCoresAlone is what the one judge of a single machine takes,
+// past which a model this small hardly gets faster.
+const (
+	judgeCores      = 2
+	judgeCoresAlone = 4
+	judgeMemoryGB   = 2
+)

internal/cli/serverinit_cluster_test.go

@@ -171,7 +171,7 @@ func TestTheInstalledConfigurationIsHardenedAndLoads(t *testing.T) {
 }
 
 func TestTheGuardModelIsAValidJudge(t *testing.T) {
-	m := guardModel("http://10.0.0.5:8081/qwen3guard-0.6b/guard/g.gguf", 2, 2)
+	m := guardModel("http://10.0.0.5:8081/qwen3guard-0.6b/guard/g.gguf", 2, 2, nil)
 	c, err := catalog.New([]catalog.Model{m})
 	if err != nil {
 		t.Fatalf("not a valid catalog entry: %v", err)

internal/cli/serverinit_judges.go

@@ -0,0 +1,150 @@
+package cli
+
+import (
+	"context"
+	"fmt"
+	"strconv"
+	"strings"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
+)
+
+// Machines of the judges' own. Under load on a 4-core control plane that was
+// also a node, 38 of 60 requests were refused for want of a judgment; with the
+// judges on a node of their own the same test refused 2. A judge machine is
+// added as a node carrying judgeRoleLabel, and the judge's profile selects
+// that label, so nothing else is placed beside it and it on nothing else.
+
+// judgeRoleLabel marks a node as the judges'.
+const (
+	judgeRoleLabel = placement.RoleLabel
+	judgeRoleValue = "judge"
+)
+
+// judgeMachine is a machine for the judges: user@host, and the connection the
+// setup screen already made to it, when it made one.
+type judgeMachine struct {
+	target string
+	remote onboard.Remote
+}
+
+// ensureJudgeNodes adds each judge machine as a node labelled for the
+// judges, and says how many cores the smallest of them has.
+func (in *serverInit) ensureJudgeNodes(ctx context.Context) error {
+	if len(in.judgeMachines) == 0 {
+		return nil
+	}
+	na := nodeAdd{serverCfg: in.configPath(), plain: true}
+	na.opts.PKI = in.pkiDir
+	na.opts.NoFirewall = in.noFirewall
+	if err := na.prepare(); err != nil {
+		return err
+	}
+	in.judgeCPUs = nil
+	for _, jm := range in.judgeMachines {
+		o := na.target(jm.target)
+		fmt.Fprintf(in.out, "  ..    adding %s as a node for the judges\n", o.Addr)
+		r := onboard.New(o, nil)
+		if jm.remote != nil {
+			r.Remote = jm.remote
+		}
+		res, err := r.Run(ctx)
+		if err != nil {
+			return fmt.Errorf("%s: %w", o.Addr, err)
+		}
+		if err := na.watch(ctx, res.Node, func(onboard.Event) {}); err != nil && err != errNoAdmin {
+			return fmt.Errorf("%s: %w", res.Node, err)
+		}
+		if out, err := r.Remote.Run(ctx, "nproc", nil); err == nil {
+			if n, err := strconv.Atoi(strings.TrimSpace(string(out))); err == nil {
+				in.judgeCPUs = append(in.judgeCPUs, n)
+			}
+		}
+		if _, err := changeStoredSettings(in.configPath(), "a node for the judges, from init", func(doc *settings.Doc) error {
+			return doc.SetYAML([]string{"agents", "nodes", res.Node, "labels", judgeRoleLabel}, judgeRoleValue)
+		}); err != nil {
+			return fmt.Errorf("labelling %s for the judges: %w", res.Node, err)
+		}
+		in.done("%s is a node for the judges alone (%s: %s)", res.Node, judgeRoleLabel, judgeRoleValue)
+	}
+	return nil
+}
+
+// judgePlacement is the judge's cores and where it may run: on the judge
+// machines when there are some, by init's rule over their cores; otherwise
+// on any node, with this machine's plan.
+func (in *serverInit) judgePlacement() (cores int, selector map[string]string) {
+	if len(in.judgeMachines) == 0 {
+		return in.cpuPlan().judge, nil
+	}
+	return refitCores(in.judgeFor().Replicas, in.judgeCPUs), map[string]string{judgeRoleLabel: judgeRoleValue}
+}
+
+// judgeMachineFlag is --judge-node, given again for each machine.
+type judgeMachineFlag struct{ to *[]judgeMachine }
+
+func (f judgeMachineFlag) String() string {
+	if f.to == nil {
+		return ""
+	}
+	var out []string
+	for _, m := range *f.to {
+		out = append(out, m.target)
+	}
+	return strings.Join(out, ",")
+}
+
+func (f judgeMachineFlag) Set(v string) error {
+	if strings.TrimSpace(v) == "" {
+		return fmt.Errorf("--judge-node wants user@host")
+	}
+	*f.to = append(*f.to, judgeMachine{target: strings.TrimSpace(v)})
+	return nil
+}
+
+func (judgeMachineFlag) Type() string { return "user@host" }
+
+// judgeReadyWait is how long init waits for its judge to load before timing
+// it: the weights come from the mirror, and llama.cpp loads in seconds.
+const judgeReadyWait = 15 * time.Minute
+
+// checkJudge times the judge once it is up, and keeps what it found for the
+// closing text: a machine too slow for its judge is said before its first
+// refused request, not after. The budget is not changed without asking.
+func (in *serverInit) checkJudge(ctx context.Context) {
+	c := in.adminAPI()
+	deadline := time.Now().Add(judgeReadyWait)
+	for {
+		var judges []adminapi.JudgeView
+		if err := c.get(ctx, "/api/v1/guard/judges", &judges); err == nil {
+			ready := false
+			for _, j := range judges {
+				ready = ready || (j.Model == guardModelName && j.Ready > 0)
+			}
+			if ready {
+				break
+			}
+		}
+		if time.Now().After(deadline) || ctx.Err() != nil {
+			in.judgeNote = "  !!    the judge was not up to be timed; once it is: llmgw ctl guard judge check\n"
+			return
+		}
+		time.Sleep(10 * time.Second)
+	}
+	fmt.Fprintf(in.out, "  ..    timing the judge\n")
+	v, err := checkJudge(ctx, c, guardModelName, 0)
+	if err != nil {
+		in.judgeNote = "  !!    the judge could not be timed (" + err.Error() + "); later: llmgw ctl guard judge check\n"
+		return
+	}
+	if v.Verdict == "ok" {
+		in.done("the judge reads a prompt in %.1f s, of the %.1f s a request waits; about %.1f requests a second",
+			float64(v.SlowestMs)/1000, float64(v.BudgetMs)/1000, v.PerSecond/2)
+		return
+	}
+	in.judgeNote = "The judge is " + v.Verdict + " on this cluster:\n" + indent(judgeCheckText(v), "  ") + "\n"
+}

internal/cli/serverinit_judges_test.go

@@ -0,0 +1,67 @@
+package cli
+
+import (
+	"strings"
+	"testing"
+)
+
+func TestJudgesGoOnTheirOwnMachinesWhenTheInstallHasSome(t *testing.T) {
+	in := newDefaultServerInit()
+	if _, sel := in.judgePlacement(); sel != nil {
+		t.Errorf("no judge machines, yet a selector: %v", sel)
+	}
+	if err := (judgeMachineFlag{&in.judgeMachines}).Set("ubuntu@10.0.0.21"); err != nil {
+		t.Fatal(err)
+	}
+	_ = (judgeMachineFlag{&in.judgeMachines}).Set("ubuntu@10.0.0.22")
+	in.judge.Replicas, in.judgeCPUs = 2, []int{4, 4}
+	cores, sel := in.judgePlacement()
+	if sel[judgeRoleLabel] != judgeRoleValue || cores != refitCores(2, []int{4, 4}) {
+		t.Errorf("cores %d, selector %v", cores, sel)
+	}
+	m := guardModel("http://x/g.gguf", 2, cores, sel)
+	if m.Profiles[0].NodeSelector[judgeRoleLabel] != judgeRoleValue {
+		t.Errorf("the judge's profile does not select its machines: %+v", m.Profiles[0])
+	}
+	// The same from a script.
+	flags := strings.Join(in.initFlags(), " ")
+	if !strings.Contains(flags, "--judge-node ubuntu@10.0.0.21") || !strings.Contains(flags, "--judge-node ubuntu@10.0.0.22") {
+		t.Errorf("flags %s", flags)
+	}
+	if err := (judgeMachineFlag{&in.judgeMachines}).Set(" "); err == nil {
+		t.Error("an empty --judge-node was taken")
+	}
+}
+
+func TestTheSetupAsksForJudgeMachinesWhenChosen(t *testing.T) {
+	m := newSetupModel("ctl", []string{"10.0.0.5"})
+	m.choice, m.page = 1, pMachines
+	m.mfocus = m.shown() * mfCount // the judges' choice
+	m = press(m, "right").(setupModel)
+	if m.judges != 1 || m.shown() != 3 || m.mfocus != 3*mfCount {
+		t.Fatalf("judges %d, shown %d, focus %d", m.judges, m.shown(), m.mfocus)
+	}
+	if v := m.machinesView(); !strings.Contains(v, "Judge machine 1") || !strings.Contains(v, "Check the 4 machines") {
+		t.Errorf("the page lacks the judge machine:\n%s", v)
+	}
+	for i, addr := range []string{"10.0.0.11", "10.0.0.12", ""} {
+		m.db[i].addr.SetValue(addr)
+		m.db[i].user.SetValue("ubuntu")
+		m.db[i].auth = authAgent
+	}
+	if err := m.validateMachines(); !strings.Contains(err, "judge machine 1 has no address") {
+		t.Errorf("err %q", err)
+	}
+	m.db[2].addr.SetValue("10.0.0.21")
+	if err := m.validateMachines(); err != "" {
+		t.Errorf("err %q", err)
+	}
+	if cmd := m.command(); !strings.Contains(cmd, "--judge-node ubuntu@10.0.0.21") || strings.Contains(cmd, "--patroni ubuntu@10.0.0.21") {
+		t.Errorf("command %s", cmd)
+	}
+	// None again: the judge machine is no longer asked for.
+	m = press(m, "left").(setupModel)
+	if m.judges != 0 || strings.Contains(m.machinesView(), "Judge machine 1") {
+		t.Errorf("judges %d after choosing none", m.judges)
+	}
+}

internal/cli/serverinit_memory_test.go

@@ -0,0 +1,27 @@
+package cli
+
+import (
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/preflight"
+)
+
+func TestTheMemoryAMachineNeedsCountsWhatItServes(t *testing.T) {
+	in := &serverInit{}
+	mb, what := in.servingMemoryMB(false)
+	need := preflight.Needs[preflight.RoleControl].MemoryMB + mb
+	if what == "" || need <= 7941 || need > 11961 {
+		t.Errorf("node, starter and judges here: need %d MB (%q); an 8 GB machine must fail and a 12 GB one pass", need, what)
+	}
+	// Judges on machines of their own take their share off this one.
+	if mb2, _ := in.servingMemoryMB(true); mb2 >= mb {
+		t.Errorf("with judge machines %d MB, without %d", mb2, mb)
+	}
+	// A control plane that serves nothing needs only its own.
+	if mb, what := (&serverInit{noNode: true}).servingMemoryMB(false); mb != 0 || what != "" {
+		t.Errorf("no node: %d %q", mb, what)
+	}
+	if mb, _ := (&serverInit{noModel: true, noGuardModel: true}).servingMemoryMB(false); mb != 0 {
+		t.Errorf("nothing served: %d", mb)
+	}
+}

internal/cli/tui_choices.go

@@ -282,7 +282,7 @@ func newInstallForm(in *serverInit, host string, found []string, ha bool) *insta
 		"The request is served unjudged and marked so in the logs. Nothing is refused for the judge's sake.",
 	}
 	f.judgeCopies = textSetupField("Copies", strconv.Itoa(j.Replicas), "2",
-		"Each takes about 1 GB and two CPU threads on whichever node has room.")
+		"On one machine one copy, on up to four cores; with more machines two, two cores each, on different nodes. Each reserves 2 GB.")
 	f.judgeBudget = textSetupField("Waits at most", durationYAML(j.Budget), "5s",
 		"A short text is judged in about two seconds on CPUs.")
 	judgeFields := []*setupField{f.judgeFail, f.judgeCopies, f.judgeBudget, headingField("Refuses")}

internal/cli/tui_console.go

@@ -53,6 +53,7 @@ const (
 	cAccounts
 	cKeys
 	cOrgs
+	cSignIn
 	cAudit
 	cManual
 	cSettings
@@ -69,7 +70,7 @@ const (
 	cCount
 )
 
-var consoleLabels = [cCount]string{"Overview", "Monitor", "Nodes", "GPUs", "Models", "Replicas", "Usage", "Quotas", "Guardrails", "Knowledge", "Federation", "Mirror", "Jobs", "Logs", "Accounts", "API keys", "Organizations", "Audit", "Manual",
+var consoleLabels = [cCount]string{"Overview", "Monitor", "Nodes", "GPUs", "Models", "Replicas", "Usage", "Quotas", "Guardrails", "Knowledge", "Federation", "Mirror", "Jobs", "Logs", "Accounts", "API keys", "Organizations", "Sign-in", "Audit", "Manual",
 	"server.yaml", "Add a node", "Add a control plane", "Replicate the store", "Serve a model", "Serve the guardrails' judge", "Upgrade the cluster",
 	"Restart all replicas", "Back up now", "Wipe this machine", "Leave"}
 
@@ -140,8 +141,8 @@ type consoleModel struct {
 	detail bool
 	scroll int
 
-	// node is the console of a node: its agent.yaml and nothing to watch,
-	// since the cluster is watched from the control plane.
+	// node is the console of a node: what runs on the machine and its
+	// agent.yaml, since the cluster is watched from the control plane.
 	node bool
 
 	// The settings.
@@ -181,6 +182,7 @@ type consoleModel struct {
 	quotas    []adminapi.QuotaView
 	quotasErr error
 	guard     guardState
+	signIn    signInState
 	kb        kbState
 
 	st     *storeHolder
@@ -190,6 +192,11 @@ type consoleModel struct {
 	docs   docViewer
 	mon    monitorState
 	det    monitorState
+
+	// nodeOv is a node's overview as last read; nodeBusy says a reading is
+	// under way.
+	nodeOv   nodeOverview
+	nodeBusy bool
 }
 
 var usageGroupings = []string{"team", "project", "org"}
@@ -198,9 +205,6 @@ func newConsoleModel(addr, file string) consoleModel {
 	m := consoleModel{client: newCtlClient(addr), spin: newSpinner(), width: 100, height: 30, file: file,
 		st: &storeHolder{file: file}, docs: newDocViewer()}
 	m.node = strings.HasSuffix(file, "agent.yaml")
-	if m.node {
-		m.menu = cSettings
-	}
 	m.loadSettings()
 	return m
 }
@@ -257,7 +261,7 @@ func (m *consoleModel) loadSettings() {
 
 func (m consoleModel) Init() tea.Cmd {
 	if m.node {
-		return m.spin.Tick
+		return tea.Batch(m.fetchNodeOverview(), m.spin.Tick, tea.Tick(3*time.Second, func(time.Time) tea.Msg { return consoleTickMsg{} }))
 	}
 	return tea.Batch(m.fetch(), m.spin.Tick, tea.Tick(3*time.Second, func(time.Time) tea.Msg { return consoleTickMsg{} }))
 }
@@ -320,7 +324,7 @@ func (m consoleModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
 		return m, cmd
 	case consoleTickMsg:
 		if m.node {
-			return m, nil
+			return m.nodeTick()
 		}
 		cmds := []tea.Cmd{m.fetch(), tea.Tick(3*time.Second, func(time.Time) tea.Msg { return consoleTickMsg{} })}
 		if m.menu == cUsage {
@@ -332,6 +336,9 @@ func (m consoleModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
 		if m.menu == cGuard {
 			cmds = append(cmds, m.fetchGuard())
 		}
+		if m.menu == cSignIn {
+			cmds = append(cmds, m.fetchSignIn())
+		}
 		if m.menu == cKB {
 			cmds = append(cmds, m.fetchKB())
 		}
@@ -351,6 +358,8 @@ func (m consoleModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
 			cmds = append(cmds, m.fetchLogs(m.logs.node))
 		}
 		return m, tea.Batch(cmds...)
+	case nodeOverviewMsg:
+		m.nodeOv, m.nodeBusy = nodeOverview(msg), false
 	case consoleDataMsg:
 		m.loaded = true
 		if msg.err != nil {
@@ -395,6 +404,8 @@ func (m consoleModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
 		m.quotas, m.quotasErr = msg.rows, msg.err
 	case consoleKBMsg:
 		m.kb = kbState{view: msg.view, err: msg.err, loaded: true}
+	case consoleSignInMsg:
+		m.signIn = signInState{view: msg.view, err: msg.err, loaded: true, ldap: msg.ldap, oidc: msg.oidc}
 	case consoleGuardMsg:
 		window := m.guard.window
 		m.guard = guardState{view: msg.view, events: msg.events, err: msg.err, window: window}
@@ -423,7 +434,7 @@ func (m consoleModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
 
 func (m consoleModel) entryOK(i int) bool {
 	if m.node {
-		return i == cSettings || i == cManual || i == cWipe || i == cLeave
+		return i == cOverview || i == cSettings || i == cManual || i == cWipe || i == cLeave
 	}
 	return i != cReplicate || !m.storeReplicated()
 }
@@ -553,6 +564,8 @@ func (m consoleModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) {
 			m.menuD = m.quotaMenu(m.quotas[m.row])
 		case cGuard:
 			m.menuD = m.guardAttachmentMenu(m.guard.view.Attachments[m.row])
+		case cSignIn:
+			m.menuD = m.signInMenu(m.signInRowKey())
 		case cKB:
 			m.menuD = m.kbMenu(m.kb.view.Collections[m.row])
 		case cMirror:
@@ -649,6 +662,19 @@ func (m consoleModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) {
 			m.usageExportForm()
 			return m, nil
 		}
+		if m.menu == cSignIn && m.signIn.loaded {
+			m.entraForm()
+			return m, nil
+		}
+	case "d":
+		if m.menu == cSignIn && m.signIn.loaded {
+			m.directoryForm()
+			return m, nil
+		}
+	case "t":
+		if m.menu == cSignIn && m.signIn.view.LDAP != nil {
+			return m, m.tryDirectory()
+		}
 	case "right", "l":
 		if m.menu == cUsage {
 			m.usageBy = (m.usageBy + 1) % len(usageGroupings)
@@ -675,6 +701,8 @@ func (m consoleModel) onEnter() tea.Cmd {
 		return m.fetchQuotas()
 	case cGuard:
 		return m.fetchGuard()
+	case cSignIn:
+		return m.fetchSignIn()
 	case cKB:
 		return m.fetchKB()
 	case cMirror:
@@ -1053,6 +1081,8 @@ func (m consoleModel) rowCount() int {
 		return len(m.quotas)
 	case cGuard:
 		return len(m.guard.view.Attachments)
+	case cSignIn:
+		return len(signInRows)
 	case cKB:
 		return len(m.kb.view.Collections)
 	case cMirror:
@@ -1080,9 +1110,6 @@ func (m consoleModel) rowCount() int {
 func (m consoleModel) View() string {
 	host, _ := os.Hostname()
 	nav := []navItem{{Label: "Watch", Heading: true}}
-	if m.node {
-		nav = nil
-	}
 	for i := 0; i < cCount; i++ {
 		if m.node && !m.entryOK(i) {
 			continue
@@ -1177,6 +1204,9 @@ func (m consoleModel) panel() (title, body, keys string) {
 	}
 	switch m.menu {
 	case cOverview:
+		if m.node {
+			return m.nodeOverviewPanel(pw)
+		}
 		var b bytes.Buffer
 		c := &cobra.Command{}
 		c.SetOut(&b)
@@ -1292,6 +1322,8 @@ func (m consoleModel) panel() (title, body, keys string) {
 		return m.quotasPanel(pw)
 	case cGuard:
 		return m.guardPanel(pw)
+	case cSignIn:
+		return m.signInPanel(pw)
 	case cKB:
 		return m.kbPanel(pw)
 	case cAccounts, cKeys, cOrgs:
@@ -1648,9 +1680,6 @@ func runConsole(ctx context.Context, out io.Writer, file string) error {
 		}
 	}
 	menu := cOverview
-	if strings.HasSuffix(file, "agent.yaml") {
-		menu = cSettings
-	}
 	for {
 		m := newConsoleModel(addr, file)
 		m.menu = menu

internal/cli/tui_hf.go

@@ -107,11 +107,7 @@ func optionLabel(o modelplan.Option) string {
 	if o.FreeNow {
 		now = sOK.Render("free")
 	}
-	need := fmt.Sprintf("%.1f of %.0f GB a card", o.NeedGB, o.HaveGB)
-	if o.CPU {
-		need = fmt.Sprintf("%.1f GB of RAM", o.NeedGB)
-	}
-	return fmt.Sprintf("%s · %s · %s", o.Label(), need, now)
+	return fmt.Sprintf("%s · %s · %s", o.Label(), o.MemoryText(), now)
 }
 
 // repoMenu offers a repository's files, each with the best way it runs here;
@@ -197,8 +193,15 @@ func (m *consoleModel) serveForm(p modelserve.Plan, a modelserve.PlannedArtifact
 	if p.Note != "" {
 		note += "\n" + sWarn.Render("! ") + p.Note
 	}
-	note += fmt.Sprintf("\nPlanned for a %d-token context. The mirror fetches it and checks it, it joins the\n"+
-		"catalog, and a replica is started and asked to say hello.", p.Context)
+	note += fmt.Sprintf("\nMemory for a %d-token context, each way it runs:", p.Context)
+	for _, o := range opts {
+		note += "\n  " + o.Label() + ": " + o.MemoryText()
+	}
+	if p.KVBytesPerToken > 0 {
+		note += fmt.Sprintf("\nThe context costs %.2f GB per 1000 tokens, the whole model; a longer one is checked\n"+
+			"before it is served.", modelplan.GB(p.KVBytesPerToken*1000))
+	}
+	note += "\nThe mirror fetches it and checks it, it joins the catalog, and a replica is started\nand asked to say hello."
 	ctxDefault := "4096"
 	if a.Format == "safetensors" {
 		ctxDefault = strconv.Itoa(p.Context)
@@ -234,6 +237,10 @@ func (m *consoleModel) serveForm(p modelserve.Plan, a modelserve.PlannedArtifact
 				req.File = a.Name
 			}
 			o := opts[slices.Index(where, v[4])]
+			// Asked for a context other than planned: does it still fit?
+			if sized := o.WithContext(p.KVBytesPerToken, ctxLen); !sized.Fits() {
+				return nil, fmt.Errorf("with a %d-token context it needs %s: a shorter context, or another way", ctxLen, sized.MemoryText())
+			}
 			if o.CPU {
 				req.CPU = true
 			} else {

internal/cli/tui_node_overview.go

@@ -0,0 +1,715 @@
+package cli
+
+import (
+	"bufio"
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"net"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"slices"
+	"sort"
+	"strconv"
+	"strings"
+	"sync"
+	"syscall"
+	"time"
+
+	tea "github.com/charmbracelet/bubbletea"
+	"github.com/charmbracelet/lipgloss"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/version"
+)
+
+// A node's overview: what a bare `sudo llmgw` opens on a machine that runs
+// only llmgw-agent. Everything on it is read here — the agent's own files,
+// systemd, the cards and the disks — because a node has no admin token, and
+// the control plane may be the very thing it cannot reach.
+
+// nodeCmdTimeout bounds every command and check the overview runs: a screen
+// refreshed every three seconds cannot wait on a tool that hangs.
+const nodeCmdTimeout = 2 * time.Second
+
+// nodeControlPlanesFile is where the agent keeps the control planes the
+// leader last named (controlPlanesFile in the agent package).
+var nodeControlPlanesFile = "/var/lib/llmgw-agent/control-planes.json"
+
+// nodeProblemLines is how many of the agent's warnings the overview shows.
+const nodeProblemLines = 6
+
+type nodeOverviewMsg nodeOverview
+
+// nodeOverview is everything the overview shows, read in one go.
+type nodeOverview struct {
+	at      time.Time
+	version string
+	agent   agentService
+	planes  []planeCheck
+	gpus    []nodeGPU
+	// gpuNote says why there are no GPU rows: no card, or no answer.
+	gpuNote string
+	// names is the cards' names by index, which do not change and are asked
+	// for once rather than every refresh.
+	names       map[int]string
+	engines     []engineRun
+	enginesNote string
+	disks       []diskUse
+	problems    []string
+	// problemsNote replaces the problems when the journal cannot be read.
+	problemsNote string
+}
+
+// agentService is what systemd says of llmgw-agent.
+type agentService struct {
+	state, sub string
+	since      time.Time
+	// note replaces the rest when systemd cannot say.
+	note string
+}
+
+// planeCheck is one control plane the agent dials, and whether it answers.
+type planeCheck struct {
+	addr string
+	// learned is an address the leader named rather than one in agent.yaml.
+	learned bool
+	err     string
+}
+
+type nodeGPU struct {
+	index int
+	name  string
+	// util is -1 where the driver does not report it.
+	util                  float64
+	memUsedMB, memTotalMB float64
+	tempC                 float64
+	models                []string
+}
+
+// engineRun is one engine the supervisor started here, from its state file.
+type engineRun struct {
+	replica, model, engine string
+	gpus                   []int
+	port, pid              int
+	alive                  bool
+	started                time.Time
+}
+
+type diskUse struct {
+	what, path  string
+	free, total uint64
+	err         string
+}
+
+// nodeProbe reads this machine. Every path and tool is a field, so a test
+// reads a directory of its own instead.
+type nodeProbe struct {
+	configured []string
+	planesFile string
+	workDir    string
+	weightsDir string
+	enginesDir string
+	root       bool
+	names      map[int]string
+
+	run    func(ctx context.Context, name string, args ...string) ([]byte, error)
+	dial   func(ctx context.Context, addr string) error
+	gpus   func(ctx context.Context) []discovery.GPUSensor
+	cards  func(ctx context.Context) ([]discovery.GPU, error)
+	alive  func(pid int) bool
+	statfs func(path string) (free, total uint64, err error)
+	now    func() time.Time
+}
+
+// newNodeProbe reads the machine agent.yaml describes, as the agent would.
+func newNodeProbe(cfg any, names map[int]string) nodeProbe {
+	c, ok := cfg.(agent.Config)
+	if !ok {
+		c = agent.DefaultConfig()
+	}
+	var configured []string
+	if c.Server != "" {
+		configured = append(configured, c.Server)
+	}
+	configured = append(configured, c.Servers...)
+	// The same defaults the agent applies as it starts.
+	weights := c.WeightsCache
+	if weights == "" && len(c.ModelDirs) > 0 {
+		weights = c.ModelDirs[0]
+	}
+	if weights == "" {
+		weights = agent.DefaultWeightsCache
+	}
+	enginesDir := c.EnginesDir
+	if enginesDir == "" {
+		enginesDir = provision.DefaultRoot
+	}
+	work := c.Superv.WorkDir
+	if work == "" {
+		work = "/var/lib/llmgw-agent"
+	}
+	return nodeProbe{
+		configured: configured, planesFile: nodeControlPlanesFile, workDir: work,
+		weightsDir: weights, enginesDir: enginesDir, root: os.Geteuid() == 0, names: names,
+		run: runLocal, dial: dialTCP, gpus: discovery.SampleGPUs,
+		cards: func(ctx context.Context) ([]discovery.GPU, error) { return discovery.DiscoverGPUsSMI(ctx, "") },
+		alive: pidAlive, statfs: statfs, now: time.Now,
+	}
+}
+
+// fetchNodeOverview reads the machine in the background.
+func (m consoleModel) fetchNodeOverview() tea.Cmd {
+	p := newNodeProbe(m.cfg, m.nodeOv.names)
+	return func() tea.Msg { return nodeOverviewMsg(p.gather(context.Background())) }
+}
+
+// nodeTick refreshes the overview while it is on screen, one reading at a
+// time; the other screens of a node watch nothing.
+func (m consoleModel) nodeTick() (consoleModel, tea.Cmd) {
+	tick := tea.Tick(3*time.Second, func(time.Time) tea.Msg { return consoleTickMsg{} })
+	if m.menu != cOverview || m.nodeBusy {
+		return m, tick
+	}
+	m.nodeBusy = true
+	return m, tea.Batch(m.fetchNodeOverview(), tick)
+}
+
+// gather reads everything at once, each part in its own goroutine, so the
+// slowest part bounds the whole.
+func (p nodeProbe) gather(ctx context.Context) nodeOverview {
+	ov := nodeOverview{version: version.Version}
+	var wg sync.WaitGroup
+	do := func(f func()) {
+		wg.Add(1)
+		go func() { defer wg.Done(); f() }()
+	}
+	var engines []engineRun
+	do(func() { ov.agent = p.agentState(ctx) })
+	do(func() { ov.planes = p.checkPlanes(ctx) })
+	do(func() { ov.gpus, ov.names, ov.gpuNote = p.readGPUs(ctx) })
+	do(func() { engines, ov.enginesNote = readShardStates(p.workDir, p.alive) })
+	do(func() { ov.disks = p.readDisks() })
+	do(func() { ov.problems, ov.problemsNote = p.readProblems(ctx) })
+	wg.Wait()
+	ov.engines = engines
+	ov.gpus = modelsOnGPUs(ov.gpus, engines)
+	ov.at = p.now()
+	return ov
+}
+
+// --- the agent ---
+
+func (p nodeProbe) agentState(ctx context.Context) agentService {
+	out, err := p.run(ctx, "systemctl", "show", "llmgw-agent",
+		"-p", "LoadState", "-p", "ActiveState", "-p", "SubState", "-p", "ActiveEnterTimestamp")
+	if err != nil && len(out) == 0 {
+		return agentService{note: "systemctl is not available: " + err.Error()}
+	}
+	return parseSystemctlShow(string(out))
+}
+
+// parseSystemctlShow reads `systemctl show -p …`: one key=value a line.
+func parseSystemctlShow(out string) agentService {
+	kv := map[string]string{}
+	for _, line := range strings.Split(out, "\n") {
+		if k, v, ok := strings.Cut(strings.TrimSpace(line), "="); ok {
+			kv[k] = v
+		}
+	}
+	if kv["LoadState"] == "not-found" {
+		return agentService{note: "llmgw-agent is not installed as a service here"}
+	}
+	s := agentService{state: kv["ActiveState"], sub: kv["SubState"]}
+	if s.state == "" {
+		s.note = "systemd does not say"
+	}
+	s.since = parseSystemdTime(kv["ActiveEnterTimestamp"])
+	return s
+}
+
+// parseSystemdTime reads systemd's "Mon 2026-10-19 08:12:03 CEST", in this
+// machine's zone; zero when it is empty or n/a.
+func parseSystemdTime(v string) time.Time {
+	v = strings.TrimSpace(v)
+	if v == "" || v == "n/a" {
+		return time.Time{}
+	}
+	for _, layout := range []string{"Mon 2006-01-02 15:04:05 MST", "Mon 2006-01-02 15:04:05 -0700", "Mon 2006-01-02 15:04:05"} {
+		if t, err := time.ParseInLocation(layout, v, time.Local); err == nil {
+			return t
+		}
+	}
+	return time.Time{}
+}
+
+// --- the control planes ---
+
+// readControlPlanes is every address the agent dials, in its order: the
+// configured ones, then those the leader named that agent.yaml lacks.
+func readControlPlanes(file string, configured []string) []planeCheck {
+	var out []planeCheck
+	seen := map[string]bool{}
+	add := func(s string, learned bool) {
+		hp := hostPort(s)
+		if hp == "" || seen[hp] {
+			return
+		}
+		seen[hp] = true
+		out = append(out, planeCheck{addr: hp, learned: learned})
+	}
+	for _, s := range configured {
+		add(s, false)
+	}
+	if b, err := os.ReadFile(file); err == nil {
+		var told []string
+		if json.Unmarshal(b, &told) == nil {
+			for _, s := range told {
+				add(s, true)
+			}
+		}
+	}
+	return out
+}
+
+// hostPort is the host:port a control plane's address dials, whether it was
+// written as a URL or not.
+func hostPort(s string) string {
+	s = strings.TrimSpace(s)
+	port := ""
+	switch {
+	case strings.HasPrefix(s, "https://"):
+		s, port = strings.TrimPrefix(s, "https://"), "443"
+	case strings.HasPrefix(s, "http://"):
+		s, port = strings.TrimPrefix(s, "http://"), "80"
+	}
+	s, _, _ = strings.Cut(s, "/")
+	if _, _, err := net.SplitHostPort(s); err != nil && port != "" && s != "" {
+		s = net.JoinHostPort(strings.Trim(s, "[]"), port)
+	}
+	return s
+}
+
+func (p nodeProbe) checkPlanes(ctx context.Context) []planeCheck {
+	planes := readControlPlanes(p.planesFile, p.configured)
+	var wg sync.WaitGroup
+	for i := range planes {
+		wg.Add(1)
+		go func(c *planeCheck) {
+			defer wg.Done()
+			if err := p.dial(ctx, c.addr); err != nil {
+				c.err = err.Error()
+			}
+		}(&planes[i])
+	}
+	wg.Wait()
+	return planes
+}
+
+func dialTCP(ctx context.Context, addr string) error {
+	d := net.Dialer{Timeout: nodeCmdTimeout}
+	c, err := d.DialContext(ctx, "tcp", addr)
+	if err != nil {
+		return err
+	}
+	return c.Close()
+}
+
+// --- the GPUs ---
+
+// readGPUs samples the cards, naming them from discovery the first time.
+func (p nodeProbe) readGPUs(ctx context.Context) ([]nodeGPU, map[int]string, string) {
+	ctx, cancel := context.WithTimeout(ctx, nodeCmdTimeout)
+	defer cancel()
+	sensors := p.gpus(ctx)
+	if len(sensors) == 0 {
+		// Without nvidia-smi discovery says nothing and no error: a machine
+		// without a card. With it and no answer, a driver in trouble.
+		if _, err := p.cards(ctx); err != nil {
+			return nil, p.names, "The GPUs do not answer: " + err.Error()
+		}
+		return nil, p.names, "No GPU on this machine."
+	}
+	names := p.names
+	for _, s := range sensors {
+		if names[s.Index] == "" {
+			names = map[int]string{}
+			if cards, err := p.cards(ctx); err == nil {
+				for _, c := range cards {
+					if c.MIG == nil {
+						names[c.Index] = c.Name
+					}
+				}
+			}
+			break
+		}
+	}
+	var out []nodeGPU
+	for _, s := range sensors {
+		g := nodeGPU{index: s.Index, name: names[s.Index], util: s.UtilPercent, tempC: s.TempC,
+			memUsedMB: s.MemoryUsedMB, memTotalMB: s.MemoryTotalMB}
+		if s.MemoryTotalMB == 0 {
+			g.util = -1
+		}
+		out = append(out, g)
+	}
+	return out, names, ""
+}
+
+// cardOf is the card a device number is on: itself, or a MIG instance's card.
+func cardOf(i int) int {
+	if discovery.IsMIGIndex(i) {
+		return (i - discovery.MIGIndex(0, 0)) / 100
+	}
+	return i
+}
+
+// modelsOnGPUs names, on each card, the models whose engines hold it.
+func modelsOnGPUs(gpus []nodeGPU, runs []engineRun) []nodeGPU {
+	for i := range gpus {
+		gpus[i].models = nil
+		for _, r := range runs {
+			for _, d := range r.gpus {
+				if cardOf(d) == gpus[i].index && !slices.Contains(gpus[i].models, r.model) {
+					gpus[i].models = append(gpus[i].models, r.model)
+				}
+			}
+		}
+	}
+	return gpus
+}
+
+// --- the engines ---
+
+// shardStateFile is the part of the supervisor's shard-*.json the overview
+// reads (shardState in the supervisor package).
+type shardStateFile struct {
+	ShardID   string            `json:"shard_id"`
+	ReplicaID string            `json:"replica_id"`
+	PID       int               `json:"pid"`
+	StartedAt time.Time         `json:"started_at"`
+	Spec      engines.ShardSpec `json:"spec"`
+}
+
+// readShardStates is the engines the supervisor keeps state files for in
+// its work directory, with whether each process is still there.
+func readShardStates(dir string, alive func(int) bool) ([]engineRun, string) {
+	entries, err := os.ReadDir(dir)
+	switch {
+	case errors.Is(err, os.ErrPermission):
+		return nil, "The engines' state is root's: run with sudo."
+	case errors.Is(err, os.ErrNotExist):
+		return nil, ""
+	case err != nil:
+		return nil, err.Error()
+	}
+	var runs []engineRun
+	note := ""
+	for _, e := range entries {
+		name := e.Name()
+		if !strings.HasPrefix(name, "shard-") || !strings.HasSuffix(name, ".json") {
+			continue
+		}
+		b, err := os.ReadFile(filepath.Join(dir, name))
+		if errors.Is(err, os.ErrPermission) {
+			note = "The engines' state is root's: run with sudo."
+			continue
+		}
+		var st shardStateFile
+		if err != nil || json.Unmarshal(b, &st) != nil {
+			continue
+		}
+		replica := st.ReplicaID
+		if replica == "" {
+			replica = st.Spec.ReplicaID
+		}
+		model := st.Spec.Profile.ServedModelName
+		if model == "" {
+			model = modelOfReplica(replica)
+		}
+		runs = append(runs, engineRun{replica: replica, model: model, engine: st.Spec.Profile.Engine,
+			gpus: st.Spec.GPUIndices, port: st.Spec.HTTPPort, pid: st.PID, alive: st.PID > 0 && alive(st.PID),
+			started: st.StartedAt})
+	}
+	sort.Slice(runs, func(i, j int) bool { return runs[i].replica < runs[j].replica })
+	return runs, note
+}
+
+// modelOfReplica is the model a replica id names: the control plane calls a
+// replica the model's name and a suffix.
+func modelOfReplica(id string) string {
+	if i := strings.LastIndexByte(id, '-'); i > 0 {
+		return id[:i]
+	}
+	return id
+}
+
+func pidAlive(pid int) bool {
+	err := syscall.Kill(pid, 0)
+	return err == nil || errors.Is(err, syscall.EPERM)
+}
+
+// --- the disks ---
+
+func (p nodeProbe) readDisks() []diskUse {
+	disks := []diskUse{{what: "weights", path: p.weightsDir}, {what: "engines", path: p.enginesDir}}
+	for i := range disks {
+		// In its own goroutine: statfs on a network mount that has gone away
+		// does not return.
+		type res struct {
+			free, total uint64
+			err         error
+		}
+		ch := make(chan res, 1)
+		go func(path string) {
+			f, t, err := p.statfs(path)
+			ch <- res{f, t, err}
+		}(disks[i].path)
+		select {
+		case r := <-ch:
+			disks[i].free, disks[i].total = r.free, r.total
+			if r.err != nil {
+				disks[i].err = r.err.Error()
+				if errors.Is(r.err, os.ErrNotExist) {
+					disks[i].err = "not there yet"
+				}
+			}
+		case <-time.After(nodeCmdTimeout):
+			disks[i].err = "does not answer"
+		}
+	}
+	return disks
+}
+
+func statfs(path string) (free, total uint64, err error) {
+	var st syscall.Statfs_t
+	if err := syscall.Statfs(path, &st); err != nil {
+		return 0, 0, &os.PathError{Op: "statfs", Path: path, Err: err}
+	}
+	//nolint:unconvert // the field types differ between systems
+	return uint64(st.Bavail) * uint64(st.Bsize), uint64(st.Blocks) * uint64(st.Bsize), nil
+}
+
+// --- the problems ---
+
+func (p nodeProbe) readProblems(ctx context.Context) ([]string, string) {
+	out, err := p.run(ctx, "journalctl", "-u", "llmgw-agent", "-p", "warning", "--since", "-1h",
+		"--no-pager", "-o", "cat", "-n", strconv.Itoa(nodeProblemLines))
+	if err != nil && len(out) == 0 {
+		return nil, "The journal cannot be read: " + err.Error()
+	}
+	lines := parseJournal(string(out), nodeProblemLines)
+	if len(lines) == 0 && !p.root {
+		return nil, "None that this user can see: run with sudo to read the agent's journal."
+	}
+	return lines, ""
+}
+
+// parseJournal keeps the last n lines journalctl printed, without its own
+// remarks.
+func parseJournal(out string, n int) []string {
+	var lines []string
+	sc := bufio.NewScanner(strings.NewReader(out))
+	sc.Buffer(make([]byte, 64*1024), 1024*1024)
+	for sc.Scan() {
+		l := strings.TrimSpace(sc.Text())
+		if l == "" || strings.HasPrefix(l, "-- ") || strings.HasPrefix(l, "Hint: ") {
+			continue
+		}
+		lines = append(lines, l)
+	}
+	if len(lines) > n {
+		lines = lines[len(lines)-n:]
+	}
+	return lines
+}
+
+// runLocal runs a tool on this machine, bounded by nodeCmdTimeout. A tool
+// that is not installed is said so, not waited for.
+func runLocal(ctx context.Context, name string, args ...string) ([]byte, error) {
+	if _, err := exec.LookPath(name); err != nil {
+		return nil, fmt.Errorf("%s is not installed", name)
+	}
+	ctx, cancel := context.WithTimeout(ctx, nodeCmdTimeout)
+	defer cancel()
+	//nolint:gosec // fixed tools and arguments, from this file
+	return exec.CommandContext(ctx, name, args...).Output()
+}
+
+// --- the panel ---
+
+func (m consoleModel) nodeOverviewPanel(pw int) (string, string, string) {
+	keys := "↑↓ choose · PgUp/PgDn scroll · Q leave · refreshed every three seconds"
+	if m.nodeOv.at.IsZero() {
+		return "Overview", "\n  " + m.spin.View() + " reading this machine…", keys
+	}
+	return "Overview", renderNodeOverview(m.nodeOv, pw, time.Now()), keys
+}
+
+// renderNodeOverview draws the overview from what was read.
+func renderNodeOverview(ov nodeOverview, pw int, now time.Time) string {
+	var b strings.Builder
+	head := func(s string) { b.WriteString("\n  " + lipgloss.NewStyle().Bold(true).Render(s) + "\n") }
+	kv := func(k, v string) { fmt.Fprintf(&b, "    %-14s %s\n", sDim.Render(k), v) }
+	line := func(s string) { b.WriteString("    " + s + "\n") }
+
+	head("Agent")
+	a := ov.agent
+	switch {
+	case a.note != "":
+		kv("llmgw-agent", sWarn.Render(a.note))
+	default:
+		state := a.state
+		if a.sub != "" {
+			state += " (" + a.sub + ")"
+		}
+		if a.state == "active" {
+			state = sOK.Render(state)
+		} else {
+			state = sBad.Render(state)
+		}
+		if !a.since.IsZero() {
+			state += sDim.Render(" since " + a.since.Format("2006-01-02 15:04") + ", " + uptime(now.Sub(a.since)) + " ago")
+		}
+		kv("llmgw-agent", state)
+	}
+	kv("llmgw", ov.version)
+
+	head("Control planes")
+	if len(ov.planes) == 0 {
+		line(sWarn.Render("None configured: agent.yaml names no server."))
+	}
+	for _, c := range ov.planes {
+		state := sOK.Render("reachable")
+		if c.err != "" {
+			state = sBad.Render("✗ ") + truncate(c.err, max(20, pw-30))
+		}
+		if c.learned {
+			state += sDim.Render("  (named by the leader)")
+		}
+		fmt.Fprintf(&b, "    %-22s %s\n", c.addr, state)
+	}
+
+	head("GPUs")
+	if len(ov.gpus) == 0 {
+		line(sDim.Render(ov.gpuNote))
+	} else {
+		var rows [][]string
+		for _, g := range ov.gpus {
+			busy, mem, temp := "—", "—", "—"
+			if g.util >= 0 {
+				busy = fmt.Sprintf("%.0f%%", g.util)
+			}
+			if g.memTotalMB > 0 {
+				mem = fmt.Sprintf("%.1f/%.0f GB", g.memUsedMB/1024, g.memTotalMB/1024)
+			}
+			if g.tempC > 0 {
+				temp = fmt.Sprintf("%.0f°C", g.tempC)
+			}
+			models := strings.Join(g.models, ", ")
+			if models == "" {
+				models = sDim.Render("free")
+			}
+			rows = append(rows, []string{strconv.Itoa(g.index), g.name, busy, mem, temp, models})
+		}
+		b.WriteString(indent(table([]string{"GPU", "CARD", "BUSY", "MEMORY", "TEMP", "MODELS"}, rows, -1, pw-2, false), "  ") + "\n")
+	}
+
+	head("Engines running here")
+	switch {
+	case len(ov.engines) > 0:
+		var rows [][]string
+		for _, r := range ov.engines {
+			// The process by its pid, marked when it is no longer there.
+			proc := sOK.Render(strconv.Itoa(r.pid))
+			if !r.alive {
+				proc = sBad.Render(strconv.Itoa(r.pid) + " gone")
+			}
+			gpus := strings.Join(discovery.DeviceLabels(r.gpus), ",")
+			if gpus == "" {
+				gpus = sDim.Render("CPU")
+			}
+			up := "—"
+			if !r.started.IsZero() {
+				up = uptime(now.Sub(r.started))
+			}
+			port := "—"
+			if r.port > 0 {
+				port = strconv.Itoa(r.port)
+			}
+			rows = append(rows, []string{r.replica, r.model, r.engine, gpus, port, proc, up})
+		}
+		b.WriteString(indent(table([]string{"REPLICA", "MODEL", "ENGINE", "GPUS", "PORT", "PID", "UP"}, rows, -1, pw-2, false), "  ") + "\n")
+		if ov.enginesNote != "" {
+			line(sWarn.Render(ov.enginesNote))
+		}
+	case ov.enginesNote != "":
+		line(sWarn.Render(ov.enginesNote))
+	default:
+		line(sDim.Render("None: the control plane has placed nothing here."))
+	}
+
+	head("Disk")
+	for _, d := range ov.disks {
+		use := sWarn.Render(d.err)
+		if d.err == "" {
+			use = fmt.Sprintf("%s free of %s", diskSize(d.free), diskSize(d.total))
+			if d.total > 0 && d.free*10 < d.total {
+				use = sWarn.Render(use)
+			}
+		}
+		fmt.Fprintf(&b, "    %-14s %-28s %s\n", sDim.Render(d.what), d.path, use)
+	}
+
+	head("Recent problems")
+	switch {
+	case ov.problemsNote != "":
+		line(sDim.Render(ov.problemsNote))
+	case len(ov.problems) == 0:
+		line(sOK.Render("None in the last hour."))
+	default:
+		for _, l := range ov.problems {
+			l = truncateWidth(l, max(20, pw-4))
+			if strings.Contains(l, "level=ERROR") {
+				l = sBad.Render(l)
+			} else {
+				l = sWarn.Render(l)
+			}
+			line(l)
+		}
+	}
+	b.WriteString("\n  " + sDim.Render("read at "+ov.at.Format("15:04:05")) + "\n")
+	return b.String()
+}
+
+// uptime is a duration as people say it: 3d4h, 2h15m, 40s.
+func uptime(d time.Duration) string {
+	d = d.Round(time.Second)
+	switch {
+	case d < 0:
+		return "0s"
+	case d >= 24*time.Hour:
+		return fmt.Sprintf("%dd%dh", d/(24*time.Hour), (d%(24*time.Hour))/time.Hour)
+	case d >= time.Hour:
+		return fmt.Sprintf("%dh%dm", d/time.Hour, (d%time.Hour)/time.Minute)
+	case d >= time.Minute:
+		return fmt.Sprintf("%dm%ds", d/time.Minute, (d%time.Minute)/time.Second)
+	}
+	return d.String()
+}
+
+// diskSize is a byte count in the unit disks are sold in.
+func diskSize(n uint64) string {
+	switch {
+	case n >= 1<<40:
+		return fmt.Sprintf("%.1f TB", float64(n)/(1<<40))
+	case n >= 1<<30:
+		return fmt.Sprintf("%.0f GB", float64(n)/(1<<30))
+	}
+	return fmt.Sprintf("%.0f MB", float64(n)/(1<<20))
+}

internal/cli/tui_node_overview_test.go

@@ -0,0 +1,206 @@
+package cli
+
+import (
+	"context"
+	"errors"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/charmbracelet/x/ansi"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+)
+
+func writeNodeFile(t *testing.T, path, body string) {
+	t.Helper()
+	if err := os.WriteFile(path, []byte(body), 0o600); err != nil {
+		t.Fatal(err)
+	}
+}
+
+func TestANodesOverviewDialsTheConfiguredControlPlanesThenTheLeadersOwn(t *testing.T) {
+	dir := t.TempDir()
+	file := filepath.Join(dir, "control-planes.json")
+	writeNodeFile(t, file, `["https://10.0.0.150:7443","10.0.0.140:7443","https://cp3.example/agent"]`)
+	got := readControlPlanes(file, []string{"10.0.0.140:7443", "http://10.0.0.141"})
+	want := []planeCheck{{addr: "10.0.0.140:7443"}, {addr: "10.0.0.141:80"},
+		{addr: "10.0.0.150:7443", learned: true}, {addr: "cp3.example:443", learned: true}}
+	if len(got) != len(want) {
+		t.Fatalf("got %+v, want %+v", got, want)
+	}
+	for i := range want {
+		if got[i] != want[i] {
+			t.Errorf("%d: got %+v, want %+v", i, got[i], want[i])
+		}
+	}
+	// Without the file, what agent.yaml says.
+	if got := readControlPlanes(filepath.Join(dir, "none.json"), []string{"cp:7443"}); len(got) != 1 || got[0].addr != "cp:7443" {
+		t.Errorf("without the file: %+v", got)
+	}
+}
+
+func TestANodesOverviewReadsTheEnginesFromTheSupervisorsStateFiles(t *testing.T) {
+	dir := t.TempDir()
+	writeNodeFile(t, filepath.Join(dir, "shard-qwen3-8b-ab12-0.json"), `{"shard_id":"qwen3-8b-ab12-0","replica_id":"qwen3-8b-ab12",
+		"pid":4242,"started_at":"2026-10-19T08:00:00Z","spec":{"ReplicaID":"qwen3-8b-ab12","GPUIndices":[0,1],"HTTPPort":8101,
+		"Profile":{"Name":"vllm-tp2","Engine":"vllm","ServedModelName":"qwen3-8b"}}}`)
+	writeNodeFile(t, filepath.Join(dir, "shard-bge-m3-7-0.json"), `{"replica_id":"bge-m3-7","pid":99,
+		"spec":{"GPUIndices":[1002],"HTTPPort":8102,"Profile":{"Engine":"llamacpp"}}}`)
+	writeNodeFile(t, filepath.Join(dir, "shard-x.json.tmp"), `{"replica_id":"half-written"}`)
+	writeNodeFile(t, filepath.Join(dir, "control-planes.json"), `[]`)
+	runs, note := readShardStates(dir, func(pid int) bool { return pid == 4242 })
+	if note != "" {
+		t.Fatalf("note %q", note)
+	}
+	if len(runs) != 2 {
+		t.Fatalf("got %+v", runs)
+	}
+	b, q := runs[0], runs[1]
+	if q.replica != "qwen3-8b-ab12" || q.model != "qwen3-8b" || q.engine != "vllm" || q.port != 8101 || !q.alive ||
+		len(q.gpus) != 2 || !q.started.Equal(time.Date(2026, 10, 19, 8, 0, 0, 0, time.UTC)) {
+		t.Errorf("qwen: %+v", q)
+	}
+	// No served name: the model from the replica's id.
+	if b.model != "bge-m3" || b.alive || b.engine != "llamacpp" {
+		t.Errorf("bge: %+v", b)
+	}
+	if runs, note := readShardStates(filepath.Join(dir, "absent"), pidAlive); runs != nil || note != "" {
+		t.Errorf("no work dir: %+v %q", runs, note)
+	}
+}
+
+func TestANodesOverviewNamesTheModelOnEachGPU(t *testing.T) {
+	gpus := []nodeGPU{{index: 0}, {index: 1}, {index: 2}}
+	runs := []engineRun{
+		{model: "qwen3-8b", gpus: []int{0, 1}},
+		{model: "bge-m3", gpus: []int{discovery.MIGIndex(1, 2)}},
+		{model: "qwen3-8b", gpus: []int{1}},
+	}
+	got := modelsOnGPUs(gpus, runs)
+	if strings.Join(got[0].models, ",") != "qwen3-8b" || strings.Join(got[1].models, ",") != "qwen3-8b,bge-m3" ||
+		len(got[2].models) != 0 {
+		t.Errorf("got %+v", got)
+	}
+}
+
+func TestANodesOverviewReadsWhatSystemdSaysOfTheAgent(t *testing.T) {
+	s := parseSystemctlShow("LoadState=loaded\nActiveState=active\nSubState=running\nActiveEnterTimestamp=Mon 2026-10-19 08:12:03 UTC\n")
+	if s.state != "active" || s.sub != "running" || s.note != "" || s.since.IsZero() || s.since.Minute() != 12 {
+		t.Errorf("got %+v", s)
+	}
+	if s := parseSystemctlShow("LoadState=not-found\nActiveState=inactive\nActiveEnterTimestamp=\n"); s.note == "" {
+		t.Errorf("a missing service: %+v", s)
+	}
+	if s := parseSystemctlShow("LoadState=loaded\nActiveState=failed\nSubState=failed\nActiveEnterTimestamp=n/a\n"); s.state != "failed" ||
+		!s.since.IsZero() {
+		t.Errorf("a failed service: %+v", s)
+	}
+}
+
+func TestANodesOverviewKeepsTheLastWarningsOfTheJournal(t *testing.T) {
+	var b strings.Builder
+	for i := 1; i <= 9; i++ {
+		b.WriteString("level=WARN msg=w" + string(rune('0'+i)) + "\n")
+	}
+	got := parseJournal(b.String(), 6)
+	if len(got) != 6 || got[0] != "level=WARN msg=w4" || got[5] != "level=WARN msg=w9" {
+		t.Errorf("got %q", got)
+	}
+	if got := parseJournal("-- No entries --\n", 6); len(got) != 0 {
+		t.Errorf("no entries: %q", got)
+	}
+}
+
+// fakeProbe is a node with two cards, one engine and a broken second control
+// plane, and no tool of the machine running the test touched.
+func fakeProbe(t *testing.T) nodeProbe {
+	dir := t.TempDir()
+	writeNodeFile(t, filepath.Join(dir, "shard-qwen3-8b-ab12-0.json"), `{"replica_id":"qwen3-8b-ab12","pid":4242,
+		"started_at":"2026-10-19T08:00:00Z","spec":{"GPUIndices":[0],"HTTPPort":8101,"Profile":{"Engine":"vllm","ServedModelName":"qwen3-8b"}}}`)
+	return nodeProbe{
+		configured: []string{"10.0.0.140:7443", "10.0.0.150:7443"},
+		planesFile: filepath.Join(dir, "control-planes.json"),
+		workDir:    dir, weightsDir: "/var/lib/llmgw-weights", enginesDir: "/opt/llmgw/engines", root: true,
+		run: func(_ context.Context, name string, _ ...string) ([]byte, error) {
+			switch name {
+			case "systemctl":
+				return []byte("LoadState=loaded\nActiveState=active\nSubState=running\nActiveEnterTimestamp=Mon 2026-10-19 08:00:00 UTC\n"), nil
+			case "journalctl":
+				return []byte("level=WARN msg=\"the control plane does not answer\" server=10.0.0.150:7443\n"), nil
+			}
+			return nil, errors.New(name + " is not installed")
+		},
+		dial: func(_ context.Context, addr string) error {
+			if addr == "10.0.0.150:7443" {
+				return errors.New("dial tcp 10.0.0.150:7443: connect: connection refused")
+			}
+			return nil
+		},
+		gpus: func(context.Context) []discovery.GPUSensor {
+			return []discovery.GPUSensor{
+				{Index: 0, TempC: 61, UtilPercent: 87, MemoryUsedMB: 20480, MemoryTotalMB: 24576},
+				{Index: 1, TempC: 35, UtilPercent: 0, MemoryUsedMB: 300, MemoryTotalMB: 24576},
+			}
+		},
+		cards: func(context.Context) ([]discovery.GPU, error) {
+			return []discovery.GPU{{Index: 0, Name: "NVIDIA RTX A5000"}, {Index: 1, Name: "NVIDIA RTX A5000"}}, nil
+		},
+		alive:  func(int) bool { return true },
+		statfs: func(string) (uint64, uint64, error) { return 120 << 30, 500 << 30, nil },
+		now:    func() time.Time { return time.Date(2026, 10, 19, 11, 12, 0, 0, time.UTC) },
+	}
+}
+
+func TestANodesOverviewShowsWhatRunsOnTheMachine(t *testing.T) {
+	ov := fakeProbe(t).gather(context.Background())
+	out := ansi.Strip(renderNodeOverview(ov, 100, ov.at))
+	t.Log("\n" + out)
+	for _, want := range []string{
+		"llmgw-agent", "active (running)", "3h12m ago",
+		"10.0.0.140:7443", "reachable", "connection refused",
+		"NVIDIA RTX A5000", "87%", "20.0/24 GB", "61°C", "qwen3-8b", "free",
+		"qwen3-8b-ab12", "vllm", "8101", "4242", "3h12m",
+		"/var/lib/llmgw-weights", "120 GB free of 500 GB", "/opt/llmgw/engines",
+		"the control plane does not answer",
+	} {
+		if !strings.Contains(out, want) {
+			t.Errorf("the overview lacks %q", want)
+		}
+	}
+	if ov.names[0] != "NVIDIA RTX A5000" {
+		t.Errorf("the names are not kept for the next reading: %v", ov.names)
+	}
+}
+
+func TestANodeWithoutAGPUSaysSoInOneLine(t *testing.T) {
+	p := fakeProbe(t)
+	p.gpus = func(context.Context) []discovery.GPUSensor { return nil }
+	p.cards = func(context.Context) ([]discovery.GPU, error) { return nil, nil }
+	p.run = func(context.Context, string, ...string) ([]byte, error) { return nil, errors.New("not installed") }
+	ov := p.gather(context.Background())
+	out := ansi.Strip(renderNodeOverview(ov, 100, ov.at))
+	if !strings.Contains(out, "No GPU on this machine.") {
+		t.Errorf("no line saying there is no GPU:\n%s", out)
+	}
+	if strings.Contains(out, "BUSY") {
+		t.Errorf("an empty GPU table:\n%s", out)
+	}
+	// No systemctl or journalctl is said, not a crash.
+	if !strings.Contains(out, "systemctl is not available") || !strings.Contains(out, "The journal cannot be read") {
+		t.Errorf("missing tools are not said:\n%s", out)
+	}
+}
+
+func TestANodesConsoleOpensOnItsOverview(t *testing.T) {
+	m := newConsoleModel("127.0.0.1:1", filepath.Join(t.TempDir(), "agent.yaml"))
+	if m.menu != cOverview || !m.entryOK(cOverview) {
+		t.Fatalf("menu %d", m.menu)
+	}
+	_, body, _ := m.panel()
+	if !strings.Contains(body, "reading this machine") {
+		t.Errorf("before the first reading: %q", body)
+	}
+}

internal/cli/tui_people.go

@@ -395,6 +395,7 @@ func (m *consoleModel) limitsForm(scope store.Scope, id, who string) {
 	if l.AllowedModels != nil {
 		models = strings.Join(l.AllowedModels, ",")
 	}
+	tools := strings.Join(l.AllowedTools, ",")
 	m.openForm(&formDialog{title: "Limits on " + who,
 		note: "Each level only narrows what is above it. Empty is no limit here.",
 		fields: []formField{
@@ -403,9 +404,11 @@ func (m *consoleModel) limitsForm(scope store.Scope, id, who string) {
 			textF("budget", budget, "spend per period, in whole units"),
 			choiceF("per", []string{"day", "month"}, period),
 			textF("models", models, "allowed, separated by commas; \"every model\" for no list"),
+			textF("company tools", tools, "MCP tools granted here, as jira.*,db.query; empty grants none here"),
 		},
 		submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
-			args := []string{"keys", "limits", "--scope=" + string(scope), "--id=" + id}
+			// Every field the form shows, so an emptied one is cleared.
+			args := []string{"keys", "limits", "--replace", "--scope=" + string(scope), "--id=" + id}
 			for i, flag := range []string{"rpm", "tpm"} {
 				if v[i] != "" {
 					if _, err := strconv.Atoi(v[i]); err != nil {
@@ -423,6 +426,9 @@ func (m *consoleModel) limitsForm(scope store.Scope, id, who string) {
 			if v[4] != "every model" {
 				args = append(args, "--models="+strings.ReplaceAll(v[4], " ", ""))
 			}
+			if t := strings.ReplaceAll(v[5], " ", ""); t != "" {
+				args = append(args, "--tools="+t)
+			}
 			return m.runCLI("Limits on "+who, args...), nil
 		}})
 }
@@ -567,13 +573,20 @@ func (m consoleModel) orgMenu(r orgRow) *menuDialog {
 				if c.Any {
 					which = "any"
 				}
+				similar := "off"
+				if c.Similar > 0 {
+					similar = strconv.FormatFloat(c.Similar, 'f', -1, 64)
+				}
 				m.openForm(&formDialog{title: "Cache answers · " + r.path,
 					note: "Off by default. On, the same request to the same model within the time below is\n" +
 						"answered again from the gateway's memory, without the model, and costs nothing.\n" +
-						"Reproducible: only temperature 0 or a seed, and embeddings. Any: sampled answers too.",
+						"Reproducible: only temperature 0 or a seed, and embeddings. Any: sampled answers too.\n" +
+						"Similar: a chat question this alike to one answered gets its answer (needs\n" +
+						"router.cache.embedding_model).",
 					fields: []formField{
 						textF("reuse for", ttl, "like 1h; off reuses none"),
 						choiceF("answers", []string{"reproducible", "any"}, which),
+						textF("similar", similar, "like 0.95; off reuses only the same request"),
 					},
 					submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
 						args := []string{"keys", "cache", "--project=" + r.id}
@@ -587,6 +600,12 @@ func (m consoleModel) orgMenu(r orgRow) *menuDialog {
 						if v[1] == "any" {
 							args = append(args, "--any")
 						}
+						if v[2] != "off" && v[2] != "" && v[2] != "0" {
+							if f, err := strconv.ParseFloat(v[2], 64); err != nil || f <= 0 || f > 1 {
+								return nil, fmt.Errorf("similar is a number like 0.95, or off")
+							}
+							args = append(args, "--similar="+v[2])
+						}
 						return m.runCLI("Cache answers", args...), nil
 					}})
 				return nil
@@ -761,5 +780,8 @@ func limitsCell(l store.Limits) string {
 	if l.AllowedModels != nil {
 		p = append(p, fmt.Sprintf("%d models", len(l.AllowedModels)))
 	}
+	if len(l.AllowedTools) > 0 {
+		p = append(p, fmt.Sprintf("%d tool grants", len(l.AllowedTools)))
+	}
 	return strings.Join(p, ", ")
 }

internal/cli/tui_setup.go

@@ -89,14 +89,21 @@ func (d dbMachine) visible(f int) bool {
 	return true
 }
 
-// dbConn is a database machine as it is reached, whichever way.
+// dbConn is a database machine as it is reached, whichever way, or a judge
+// machine when judge is set.
 type dbConn struct {
 	remote onboard.Remote
 	addr   string
 	ssh    *onboard.GoSSH
 	joined *joinpkg.Remote
+	judge  bool
+	target string
 }
 
+// The machines the page asks for: the store's two, then up to maxJudges for
+// the guardrails' judges alone.
+const maxJudges = 2
+
 func newDBMachine(n int) dbMachine {
 	f := func(ph string, mask bool) textField {
 		t := newTextField()
@@ -119,6 +126,24 @@ func newDBMachine(n int) dbMachine {
 	return m
 }
 
+// newJudgeMachine is the fields of a machine for the judges.
+func newJudgeMachine(n int) dbMachine {
+	d := newDBMachine(n + 2)
+	d.addr.Placeholder = fmt.Sprintf("10.0.0.%d or judge%d.internal", 20+n, n)
+	return d
+}
+
+// shown is how many machines the page asks for.
+func (m setupModel) shown() int { return 2 + m.judges }
+
+// machineTitle names machine i: the store's, or the judges'.
+func machineTitle(i int) string {
+	if i < 2 {
+		return fmt.Sprintf("Database machine %d", i+1)
+	}
+	return fmt.Sprintf("Judge machine %d", i-1)
+}
+
 // field is the text field at a machine's position, nil for the selector.
 func (d *dbMachine) field(i int) *textField {
 	switch i {
@@ -156,8 +181,10 @@ type setupModel struct {
 
 	choices *installForm
 
-	db     [2]dbMachine
-	mfocus int // over both machines' fields, then the check button
+	db [2 + maxJudges]dbMachine
+	// judges is how many judge machines are asked for, none to maxJudges.
+	judges int
+	mfocus int // over the machines' fields, then the judges' choice, then the check button
 
 	conns    []dbConn
 	listener *joinpkg.Listener
@@ -188,9 +215,12 @@ func newSetupModel(host string, ips []string) setupModel {
 	// nodes are added once it is up. Either can still be chosen.
 	in.noNode, in.noModel = true, true
 	in.standbys = true
+	// Two judges, on two of the three machines' nodes.
+	in.judge.Replicas = 2
 	return setupModel{
 		ref: &progRef{}, host: host, ips: ips, in: &in, choices: newInstallForm(&in, host, ips, true),
-		db: [2]dbMachine{newDBMachine(1), newDBMachine(2)}, spin: newSpinner(), width: 100, height: 30,
+		db:   [2 + maxJudges]dbMachine{newDBMachine(1), newDBMachine(2), newJudgeMachine(1), newJudgeMachine(2)},
+		spin: newSpinner(), width: 100, height: 30,
 		log: &[]string{}, logMu: &sync.Mutex{},
 	}
 }
@@ -325,7 +355,7 @@ func (m *setupModel) focusMachine() {
 			}
 		}
 	}
-	if m.mfocus < 2*mfCount {
+	if m.mfocus < m.shown()*mfCount {
 		if t := m.db[m.mfocus/mfCount].field(m.mfocus % mfCount); t != nil {
 			t.Focus()
 		}
@@ -333,7 +363,8 @@ func (m *setupModel) focusMachine() {
 }
 
 func (m setupModel) machinesKey(k tea.KeyMsg) (tea.Model, tea.Cmd) {
-	n := 2*mfCount + 1
+	sel := m.shown() * mfCount
+	n := sel + 2
 	switch k.Type {
 	case tea.KeyTab, tea.KeyDown:
 		m.step(1, n)
@@ -342,7 +373,7 @@ func (m setupModel) machinesKey(k tea.KeyMsg) (tea.Model, tea.Cmd) {
 		m.step(-1, n)
 		return m, nil
 	case tea.KeyEnter:
-		if m.mfocus < 2*mfCount {
+		if m.mfocus <= sel {
 			m.step(1, n)
 			return m, nil
 		}
@@ -359,7 +390,17 @@ func (m setupModel) machinesKey(k tea.KeyMsg) (tea.Model, tea.Cmd) {
 		cmd := m.startCheck()
 		return m, cmd
 	}
-	if m.mfocus == 2*mfCount {
+	if m.mfocus == sel {
+		switch k.String() {
+		case "left", "h":
+			m.judges = (m.judges + maxJudges) % (maxJudges + 1)
+		case "right", "l", " ":
+			m.judges = (m.judges + 1) % (maxJudges + 1)
+		}
+		m.mfocus = m.shown() * mfCount
+		return m, nil
+	}
+	if m.mfocus > sel {
 		return m, nil
 	}
 	d := &m.db[m.mfocus/mfCount]
@@ -389,7 +430,7 @@ func (m setupModel) machinesKey(k tea.KeyMsg) (tea.Model, tea.Cmd) {
 func (m *setupModel) step(d, n int) {
 	for range n {
 		m.mfocus = (m.mfocus + d + n) % n
-		if m.mfocus == 2*mfCount || m.db[m.mfocus/mfCount].visible(m.mfocus%mfCount) {
+		if m.mfocus >= m.shown()*mfCount || m.db[m.mfocus/mfCount].visible(m.mfocus%mfCount) {
 			break
 		}
 	}
@@ -397,7 +438,7 @@ func (m *setupModel) step(d, n int) {
 }
 
 func (m setupModel) allJoined() bool {
-	for _, d := range m.db {
+	for _, d := range m.db[:m.shown()] {
 		if d.auth == authJoin && d.joined == nil {
 			return false
 		}
@@ -409,7 +450,7 @@ func (m setupModel) allJoined() bool {
 // codes, and waits for them; it says whether there is anything to wait for.
 func (m *setupModel) startJoins() (tea.Cmd, bool) {
 	var cmds []tea.Cmd
-	for i := range m.db {
+	for i := range m.db[:m.shown()] {
 		d := &m.db[i]
 		if d.auth != authJoin || d.joined != nil {
 			continue
@@ -427,7 +468,11 @@ func (m *setupModel) startJoins() (tea.Cmd, bool) {
 			m.listener = l
 		}
 		if d.arrive == nil {
-			c, ch := m.listener.Code("database")
+			role := "database"
+			if i >= 2 {
+				role = "judge"
+			}
+			c, ch := m.listener.Code(role)
 			d.code, d.arrive = c.String(), ch
 		}
 		ch, i := d.arrive, i
@@ -438,20 +483,20 @@ func (m *setupModel) startJoins() (tea.Cmd, bool) {
 
 func (m setupModel) validateMachines() string {
 	seen := map[string]bool{m.thisAddr(): true, m.host: true}
-	for i, d := range m.db {
+	for i, d := range m.db[:m.shown()] {
 		if d.auth == authJoin {
 			continue
 		}
-		addr := strings.TrimSpace(d.addr.Value())
+		addr, title := strings.TrimSpace(d.addr.Value()), strings.ToLower(machineTitle(i)[:1])+machineTitle(i)[1:]
 		switch {
 		case addr == "":
-			return fmt.Sprintf("database machine %d has no address", i+1)
+			return title + " has no address"
 		case seen[addr]:
-			return fmt.Sprintf("%s is given twice, or is this machine: the three must be different machines", addr)
+			return fmt.Sprintf("%s is given twice, or is this machine: each must be a different machine", addr)
 		case strings.TrimSpace(d.user.Value()) == "":
-			return fmt.Sprintf("database machine %d has no user to log in as", i+1)
+			return title + " has no user to log in as"
 		case d.auth != authAgent && d.secret.Value() == "":
-			return fmt.Sprintf("database machine %d needs its %s", i+1, authNames[d.auth])
+			return fmt.Sprintf("%s needs its %s", title, authNames[d.auth])
 		}
 		seen[addr] = true
 	}
@@ -467,15 +512,22 @@ func (m *setupModel) startCheck() tea.Cmd {
 		}
 	}
 	m.conns = nil
+	serves, what := m.in.servingMemoryMB(m.judges > 0)
 	machines := []preflight.Machine{{
 		Name: m.host + " (this machine)", Addr: m.thisAddr(), Local: true, Remote: pgha.Local{},
-		Roles: []preflight.Role{preflight.RoleControl, preflight.RoleWitness},
+		Roles:    []preflight.Role{preflight.RoleControl, preflight.RoleWitness},
+		ServesMB: serves, Serves: what,
 	}}
-	for _, d := range m.db {
+	for i, d := range m.db[:m.shown()] {
+		roles, judge := m.dbRoles(), i >= 2
+		if judge {
+			roles = []preflight.Role{preflight.RoleNode}
+		}
 		if d.auth == authJoin {
-			m.conns = append(m.conns, dbConn{remote: d.joined, addr: d.joined.Addr, joined: d.joined})
+			m.conns = append(m.conns, dbConn{remote: d.joined, addr: d.joined.Addr, joined: d.joined,
+				judge: judge, target: "root@" + d.joined.Addr})
 			machines = append(machines, preflight.Machine{Name: d.joined.Hostname + " (" + d.joined.Addr + ", joined)",
-				Addr: d.joined.Addr, Roles: m.dbRoles(), Remote: d.joined})
+				Addr: d.joined.Addr, Roles: roles, Remote: d.joined})
 			continue
 		}
 		g := &onboard.GoSSH{
@@ -488,9 +540,8 @@ func (m *setupModel) startCheck() tea.Cmd {
 		case authPassword:
 			g.Password = d.secret.Value()
 		}
-		m.conns = append(m.conns, dbConn{remote: g, addr: g.Addr, ssh: g})
-		machines = append(machines, preflight.Machine{Name: g.Addr, Addr: g.Addr,
-			Roles: m.dbRoles(), Remote: g})
+		m.conns = append(m.conns, dbConn{remote: g, addr: g.Addr, ssh: g, judge: judge, target: g.User + "@" + g.Addr})
+		machines = append(machines, preflight.Machine{Name: g.Addr, Addr: g.Addr, Roles: roles, Remote: g})
 	}
 	m.checking = true
 	m.report = preflight.Report{}
@@ -533,7 +584,7 @@ func (m setupModel) dbRoles() []preflight.Role {
 // targets are the database machines as user@address, for the command line.
 func (m setupModel) targets() []string {
 	var out []string
-	for _, d := range m.db {
+	for _, d := range m.db[:2] {
 		if d.joined != nil {
 			out = append(out, "root@"+d.joined.Addr)
 			continue
@@ -543,11 +594,25 @@ func (m setupModel) targets() []string {
 	return out
 }
 
+// judgeTargets are the judge machines as user@address.
+func (m setupModel) judgeTargets() []judgeMachine {
+	var out []judgeMachine
+	for _, d := range m.db[2:m.shown()] {
+		if d.joined != nil {
+			out = append(out, judgeMachine{target: "root@" + d.joined.Addr})
+			continue
+		}
+		out = append(out, judgeMachine{target: strings.TrimSpace(d.user.Value()) + "@" + strings.TrimSpace(d.addr.Value())})
+	}
+	return out
+}
+
 // command is what this setup is on the command line, for doing it again
 // from a script — with keys in place, since a password cannot be given there.
 func (m setupModel) command() string {
 	in := *m.in
 	in.patroni = m.targets()
+	in.judgeMachines = m.judgeTargets()
 	return strings.Join(append([]string{"sudo --preserve-env=SSH_AUTH_SOCK llmgw server init"}, in.initFlags()...), " \\\n    ")
 }
 
@@ -556,6 +621,7 @@ func (m setupModel) command() string {
 func (m *setupModel) startRun() tea.Cmd {
 	in := m.in
 	in.patroni = m.targets()
+	in.judgeMachines = nil
 	m.run = newInitModel(in, nil, false)
 	m.run.running, m.run.width, m.run.spin = true, m.width, m.spin
 	ref, conns, logp, mu, listener := m.ref, m.conns, m.log, m.logMu, m.listener
@@ -576,6 +642,11 @@ func (m *setupModel) startRun() tea.Cmd {
 						return err
 					}
 				}
+				if c.judge {
+					// Added as a node once the control plane is up.
+					in.judgeMachines = append(in.judgeMachines, judgeMachine{target: c.target, remote: c.remote})
+					continue
+				}
 				out, err := c.remote.Run(ctx, "hostname", nil)
 				if err != nil {
 					return fmt.Errorf("%s: %v", c.addr, err)
@@ -718,8 +789,12 @@ func (m setupModel) machinesView() string {
 	var b strings.Builder
 	b.WriteString("\n  The two machines that keep the store: Ubuntu 22.04/24.04 or Debian 12.\n" +
 		"  PostgreSQL, Patroni and etcd are installed on them.\n\n")
-	for i, d := range m.db {
+	for i, d := range m.db[:m.shown()] {
 		base := i * mfCount
+		if i == 2 {
+			b.WriteString("  " + sDim.Render("Machines for the guardrails' judges alone: each becomes a node that runs only\n"+
+				"  the judges, so a busy control plane cannot leave a request waiting for a judgment.") + "\n\n")
+		}
 		// Padded before it is coloured: the colour's escape codes count
 		// toward a width and threw the focused row out of line.
 		label := func(f int, s string) string {
@@ -729,7 +804,7 @@ func (m setupModel) machinesView() string {
 			}
 			return "  " + s
 		}
-		fmt.Fprintf(&b, "  %s\n", lipgloss.NewStyle().Bold(true).Render(fmt.Sprintf("Database machine %d", i+1)))
+		fmt.Fprintf(&b, "  %s\n", lipgloss.NewStyle().Bold(true).Render(machineTitle(i)))
 		if d.visible(mfAddr) {
 			fmt.Fprintf(&b, "  %s %s\n", label(mfAddr, "Address"), d.addr.View())
 			fmt.Fprintf(&b, "  %s %s\n", label(mfUser, "User"), d.user.View())
@@ -754,7 +829,22 @@ func (m setupModel) machinesView() string {
 		}
 		b.WriteString("\n")
 	}
-	b.WriteString(button(m.mfocus == 2*mfCount, "Check the three machines") + "\n\n")
+	sel := m.shown() * mfCount
+	var opts []string
+	for n, name := range []string{"none", "one", "two"} {
+		if n == m.judges {
+			opts = append(opts, "("+sOK.Render("•")+") "+name)
+		} else {
+			opts = append(opts, "( ) "+sDim.Render(name))
+		}
+	}
+	judgeLabel := fmt.Sprintf("  %-15s", "Judge machines")
+	if m.mfocus == sel {
+		judgeLabel = fSelected.Render(fmt.Sprintf("› %-15s", "Judge machines"))
+	}
+	fmt.Fprintf(&b, "  %s   %s\n", judgeLabel, strings.Join(opts, "  "))
+	b.WriteString("      " + sDim.Render("Without them the judges run on whichever node has room, added later.") + "\n\n")
+	b.WriteString(button(m.mfocus == sel+1, fmt.Sprintf("Check the %d machines", m.shown()+1)) + "\n\n")
 	b.WriteString("  " + sDim.Render("Passwords are kept in memory only, for this setup.") + "\n")
 	return b.String()
 }
@@ -762,11 +852,11 @@ func (m setupModel) machinesView() string {
 func (m setupModel) waitView() string {
 	var b strings.Builder
 	fmt.Fprintf(&b, "\n  On each machine below, as root. It dials this machine on port %d;\n  nothing dials in. Each code is for one machine and good for an hour.\n\n", joinpkg.DefaultPort)
-	for i, d := range m.db {
+	for i, d := range m.db[:m.shown()] {
 		if d.auth != authJoin {
 			continue
 		}
-		title := lipgloss.NewStyle().Bold(true).Render(fmt.Sprintf("Database machine %d", i+1))
+		title := lipgloss.NewStyle().Bold(true).Render(machineTitle(i))
 		if d.joined != nil {
 			fmt.Fprintf(&b, "  %s  %s joined as %s (%s)\n\n", title, sOK.Render("✓"), d.joined.Hostname, d.joined.Addr)
 			continue
@@ -785,7 +875,7 @@ func (m setupModel) checkView() string {
 	var b strings.Builder
 	b.WriteString("\n")
 	if m.checking {
-		b.WriteString("  " + m.spin.View() + " checking the three machines…\n")
+		b.WriteString("  " + m.spin.View() + " checking the machines…\n")
 		return b.String()
 	}
 	switch m.report.Worst() {
@@ -794,7 +884,7 @@ func (m setupModel) checkView() string {
 	case preflight.Warn:
 		b.WriteString("  " + sWarn.Render("! It can go ahead; what is marked ! is worth a look.") + "\n")
 	default:
-		b.WriteString("  " + sOK.Render("✓ All three are ready.") + "\n")
+		b.WriteString("  " + sOK.Render("✓ All are ready.") + "\n")
 	}
 	machine := ""
 	for _, f := range m.report.Findings {
@@ -820,7 +910,7 @@ func (m setupModel) checkView() string {
 func (m setupModel) planView() string {
 	var b strings.Builder
 	b.WriteString(m.choices.planView(m.width - navWidth - 6))
-	b.WriteString("\n  " + lipgloss.NewStyle().Bold(true).Render("The three machines") + "\n")
+	b.WriteString("\n  " + lipgloss.NewStyle().Bold(true).Render("The machines") + "\n")
 	here := "control plane, leading at first; model mirror; etcd"
 	there := "PostgreSQL under Patroni, etcd"
 	if m.in.standbys {
@@ -832,12 +922,16 @@ func (m setupModel) planView() string {
 	}
 	fmt.Fprintf(&b, "  %s  %s\n      %s\n", lipgloss.NewStyle().Bold(true).Render(m.host), sDim.Render(m.thisAddr()), here)
 	granted := false
-	for _, d := range m.db {
+	for i, d := range m.db[:m.shown()] {
 		addr := strings.TrimSpace(d.addr.Value())
 		if d.joined != nil {
 			addr = d.joined.Hostname + "  " + sDim.Render(d.joined.Addr+", joined")
 		}
-		fmt.Fprintf(&b, "  %s\n      %s\n", lipgloss.NewStyle().Bold(true).Render(addr), there)
+		role := there
+		if i >= 2 {
+			role = "a node for the guardrails' judges alone"
+		}
+		fmt.Fprintf(&b, "  %s\n      %s\n", lipgloss.NewStyle().Bold(true).Render(addr), role)
 		granted = granted || d.sudo.Value() != ""
 	}
 	b.WriteString("\n  The store is replicated from the first to the second; if the primary is lost\n" +
@@ -849,7 +943,7 @@ func (m setupModel) planView() string {
 	for _, l := range strings.Split(m.command(), "\n") {
 		b.WriteString("  " + l + "\n")
 	}
-	for _, d := range m.db {
+	for _, d := range m.db[:m.shown()] {
 		if d.joined != nil {
 			b.WriteString("  " + sDim.Render("(a machine that joined with a code needs ssh for that)") + "\n")
 			break

internal/cli/tui_signin.go

@@ -0,0 +1,664 @@
+package cli
+
+import (
+	"context"
+	"crypto/rand"
+	"encoding/hex"
+	"errors"
+	"fmt"
+	"net/http"
+	"os"
+	"strings"
+	"time"
+
+	tea "github.com/charmbracelet/bubbletea"
+	"go.yaml.in/yaml/v3"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
+)
+
+// How people sign in, as the console shows and sets it: single sign-on
+// through Entra ID or another OpenID provider, an Active Directory checked
+// over LDAPS, and provisioning by SCIM. The secrets these need are kept in
+// the store, so every control plane has them without a file put on each,
+// and a directory is tried before its settings are kept.
+
+// inChainRule is AD's matching rule for nested group membership.
+const inChainRule = "1.2.840.113556.1.4.1941"
+
+type signInState struct {
+	view   adminapi.AuthView
+	err    error
+	loaded bool
+	// ldap and oidc are the blocks the cluster's settings hold now, for the
+	// forms to start from.
+	ldap config.LDAP
+	oidc config.OIDC
+}
+
+type consoleSignInMsg struct {
+	view adminapi.AuthView
+	ldap config.LDAP
+	oidc config.OIDC
+	err  error
+}
+
+// signInRows are the panel's rows, one per way in.
+var signInRows = []string{"oidc", "ldap", "scim"}
+
+func (m consoleModel) fetchSignIn() tea.Cmd {
+	c := m.client
+	return func() tea.Msg {
+		ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+		defer cancel()
+		var msg consoleSignInMsg
+		if msg.err = c.get(ctx, "/api/v1/auth", &msg.view); msg.err != nil {
+			return msg
+		}
+		var cur adminapi.SettingsView
+		if err := c.get(ctx, "/api/v1/settings", &cur); err == nil {
+			msg.ldap, msg.oidc = authBlocks(cur.Body)
+		}
+		return msg
+	}
+}
+
+// authBlocks are auth.ldap and auth.oidc as a settings document has them.
+func authBlocks(body string) (config.LDAP, config.OIDC) {
+	var doc struct {
+		Auth struct {
+			LDAP yaml.Node `yaml:"ldap"`
+			OIDC yaml.Node `yaml:"oidc"`
+		} `yaml:"auth"`
+	}
+	var l config.LDAP
+	var o config.OIDC
+	if yaml.Unmarshal([]byte(body), &doc) != nil {
+		return l, o
+	}
+	if doc.Auth.LDAP.Kind != 0 {
+		if b, err := yaml.Marshal(&doc.Auth.LDAP); err == nil {
+			// Read as kept, whether or not it would validate now.
+			l, _ = config.ParseLDAP(b)
+		}
+	}
+	if doc.Auth.OIDC.Kind != 0 {
+		if m, err := yaml.Marshal(&doc.Auth.OIDC); err == nil {
+			if cfg, err := config.FromMap(map[string]any{"auth": map[string]any{"oidc": yamlMap(m)}}); err == nil {
+				o = cfg.Auth.OIDC
+			}
+		}
+	}
+	return l, o
+}
+
+func yamlMap(b []byte) map[string]any {
+	var m map[string]any
+	_ = yaml.Unmarshal(b, &m)
+	return m
+}
+
+func (m consoleModel) signInPanel(pw int) (string, string, string) {
+	keys := "→ or Enter open · ↑↓ choose · Q leave"
+	if m.inPanel {
+		keys = "↑↓ row · Enter actions · E Entra ID · D Active Directory · T try the directory · ← back · Q leave"
+	}
+	s := m.signIn
+	var b strings.Builder
+	b.WriteString("\n  " + sDim.Render("How people sign in besides a password kept here. E for Entra ID, D for Active Directory.") + "\n\n")
+	if s.err != nil {
+		b.WriteString("  " + sBad.Render(s.err.Error()))
+		return "Sign-in", b.String(), keys
+	}
+	if !s.loaded {
+		b.WriteString("  " + sDim.Render("asking the control plane…"))
+		return "Sign-in", b.String(), keys
+	}
+	v := s.view
+	local := "passwords kept here; accounts made by an administrator"
+	if v.SignupOpen {
+		local = "passwords kept here; anyone may sign up, then wait for approval"
+	}
+	b.WriteString("  " + sTitle.Render("LOCAL") + "  " + sDim.Render(local) + "\n\n")
+
+	var rows [][]string
+	for _, r := range signInRows {
+		switch r {
+		case "oidc":
+			if v.OIDC == nil {
+				rows = append(rows, []string{"Single sign-on (Entra ID, OIDC)", "off", "E to set up"})
+				continue
+			}
+			state := "on"
+			if v.OIDC.NoPasswords {
+				state = "on, the only way in"
+			}
+			rows = append(rows, []string{"Single sign-on (Entra ID, OIDC)", state, or(v.OIDC.Label, v.OIDC.Issuer)})
+		case "ldap":
+			if v.LDAP == nil {
+				rows = append(rows, []string{"Active Directory (LDAP)", "off", "D to set up"})
+				continue
+			}
+			state := "on"
+			if v.LDAP.Off != "" {
+				state = "broken here"
+			}
+			rows = append(rows, []string{"Active Directory (LDAP)", state, strings.Join(v.LDAP.URLs, " ")})
+		case "scim":
+			state, detail := "off", "Enter to make a token"
+			if v.SCIM {
+				state, detail = "on", "at /scim/v2"
+			}
+			rows = append(rows, []string{"Provisioning (SCIM)", state, detail})
+		}
+	}
+	b.WriteString(table([]string{"WAY IN", "STATE", ""}, rows, m.row, pw, m.inPanel) + "\n")
+
+	if v.OIDC != nil {
+		b.WriteString("\n  " + sTitle.Render("SINGLE SIGN-ON") + "\n")
+		b.WriteString("  issuer     " + v.OIDC.Issuer + "\n")
+		b.WriteString("  redirect   " + redirectURL(v.BaseURL) + "\n")
+		b.WriteString("  " + sDim.Render(pickText(v.OIDC.CreateAccounts, "accounts made at first sign-in", "only people with an account sign in")) + "\n")
+	}
+	if l := v.LDAP; l != nil {
+		b.WriteString("\n  " + sTitle.Render("ACTIVE DIRECTORY") + "  " + sDim.Render(l.BaseDN) + "\n")
+		if l.Off != "" {
+			b.WriteString("  " + sBad.Render("off on this control plane: "+l.Off) + "\n")
+		}
+		switch ls := l.LastSync; {
+		case ls == nil:
+			b.WriteString("  " + sDim.Render("not read yet: the leader reads it every sync_interval") + "\n")
+		case ls.Error != "":
+			b.WriteString("  " + sBad.Render("last read "+agoText(ls.At)+": "+ls.Error) + "\n")
+		default:
+			b.WriteString(fmt.Sprintf("  last read %s: %d people there, %d accounts kept in step, %d suspended, %d reinstated\n",
+				agoText(ls.At), ls.People, ls.Accounts, ls.Suspended, ls.Reinstated))
+		}
+	}
+	return "Sign-in", b.String(), keys
+}
+
+func agoText(t time.Time) string { return time.Since(t).Round(time.Second).String() + " ago" }
+
+func pickText(b bool, yes, no string) string {
+	if b {
+		return yes
+	}
+	return no
+}
+
+// redirectURL is what an identity provider sends people back to.
+func redirectURL(base string) string {
+	if base == "" {
+		return "https://<this gateway>/login/oidc/callback"
+	}
+	return strings.TrimRight(base, "/") + "/login/oidc/callback"
+}
+
+// signInMenu is what can be done to one way in.
+func (m consoleModel) signInMenu(row string) *menuDialog {
+	v := m.signIn.view
+	switch row {
+	case "oidc":
+		d := &menuDialog{title: "Single sign-on", items: []menuItem{
+			{"Set up Entra ID…", func(m *consoleModel) tea.Cmd { m.entraForm(); return nil }},
+		}}
+		if v.OIDC != nil {
+			d.items = append(d.items, menuItem{"Turn off…", func(m *consoleModel) tea.Cmd {
+				m.signInOffConfirm("oidc", "Single sign-on", "People sign in with a password kept here again; accounts made through the provider have none.")
+				return nil
+			}})
+		}
+		return d
+	case "ldap":
+		d := &menuDialog{title: "Active Directory", items: []menuItem{
+			{"Set up…", func(m *consoleModel) tea.Cmd { m.directoryForm(); return nil }},
+		}}
+		if v.LDAP != nil {
+			d.items = append(d.items,
+				menuItem{"Try it", func(m *consoleModel) tea.Cmd { return m.tryDirectory() }},
+				menuItem{"Turn off…", func(m *consoleModel) tea.Cmd {
+					m.signInOffConfirm("ldap", "Active Directory", "Its people can no longer sign in; their accounts stay, and are no longer kept in step.")
+					return nil
+				}})
+		}
+		return d
+	default:
+		d := &menuDialog{title: "Provisioning (SCIM)", items: []menuItem{
+			{"Make a new token", func(m *consoleModel) tea.Cmd { return m.scimToken() }},
+		}}
+		if v.SCIM {
+			d.items = append(d.items, menuItem{"Turn off…", func(m *consoleModel) tea.Cmd {
+				m.signInOffConfirm("scim", "Provisioning", "The provider can no longer make, change or suspend accounts here.")
+				return nil
+			}})
+		}
+		return d
+	}
+}
+
+// signInRowKey is the way in the cursor is on.
+func (m consoleModel) signInRowKey() string {
+	if m.row >= 0 && m.row < len(signInRows) {
+		return signInRows[m.row]
+	}
+	return ""
+}
+
+// directoryForm sets up auth.ldap, starting from what is kept.
+func (m *consoleModel) directoryForm() {
+	l := m.signIn.ldap
+	allowed := ""
+	if f := l.UserFilter; strings.HasPrefix(f, "(memberOf:"+inChainRule+":=") {
+		allowed = strings.TrimSuffix(strings.TrimPrefix(f, "(memberOf:"+inChainRule+":="), ")")
+	} else {
+		allowed = f
+	}
+	pwHint := "the service account's password"
+	if l.BindPasswordFile != "" {
+		pwHint = "empty keeps the one kept"
+	}
+	caHint := "a PEM file on this machine; empty uses the system's"
+	if l.CA != "" || l.CAFile != "" {
+		caHint = "empty keeps the one kept; - for the system's"
+	}
+	create := "yes"
+	if l.Enabled() && !l.CreateAccounts {
+		create = "no"
+	}
+	var teams []string
+	for g, place := range l.Memberships {
+		teams = append(teams, g+"="+place)
+	}
+	pw := textF("password", "", pwHint)
+	pw.text.Mask = true
+	m.openForm(&formDialog{title: "Active Directory",
+		note: "People sign in with their AD name and password, checked over LDAPS. Tried before it is kept.",
+		fields: []formField{
+			textF("servers", strings.Join(l.URLs, " "), "ldaps://dc1.corp.example ldaps://dc2.corp.example"),
+			textF("people under", l.BaseDN, "OU=Staff,DC=corp,DC=example"),
+			textF("service account", l.BindDN, "CN=svc-llmgw,OU=Service,DC=corp,DC=example"),
+			pw,
+			textF("CA certificate", "", caHint),
+			textF("who may sign in", allowed, "a group's DN, nested groups count; empty is everyone under it"),
+			textF("administrators", strings.Join(l.Roles["admin"], ", "), "group names or DNs"),
+			textF("operators", strings.Join(l.Roles["operator"], ", "), "group names or DNs"),
+			textF("teams", strings.Join(teams, "; "), "ml-team=team:Acme/ml; sales=org:Sales"),
+			textF("label", l.Label, "the form says: Sign in with your LABEL account"),
+			choiceF("make accounts", []string{"yes", "no"}, create),
+		},
+		submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
+			next, err := directoryFromForm(l, v)
+			if err != nil {
+				return nil, err
+			}
+			return m.saveDirectory(next, v[3]), nil
+		}})
+}
+
+// directoryFromForm is the auth.ldap block the form's values make, over
+// what was kept.
+func directoryFromForm(kept config.LDAP, v []string) (config.LDAP, error) {
+	l := kept
+	l.URLs = strings.Fields(strings.ReplaceAll(v[0], ",", " "))
+	l.BaseDN, l.BindDN = v[1], v[2]
+	if v[3] != "" || l.BindPasswordFile == "" {
+		l.BindPasswordFile = adminapi.AuthSecretRef("ldap-bind")
+	}
+	switch ca := v[4]; ca {
+	case "":
+	case "-":
+		l.CA, l.CAFile = "", ""
+	default:
+		b, err := os.ReadFile(ca) //nolint:gosec // a file the operator names
+		if err != nil {
+			return l, fmt.Errorf("CA certificate: %w", err)
+		}
+		l.CA, l.CAFile = string(b), ""
+	}
+	switch g := v[5]; {
+	case g == "":
+		l.UserFilter = ""
+	case strings.HasPrefix(g, "("):
+		l.UserFilter = g
+	case strings.Contains(g, "="):
+		l.UserFilter = "(memberOf:" + inChainRule + ":=" + g + ")"
+	default:
+		return l, errors.New("who may sign in: a group's DN (CN=…,OU=…,DC=…), or an LDAP filter in parentheses")
+	}
+	l.Roles = map[string][]string{}
+	for role, i := range map[string]int{"admin": 6, "operator": 7} {
+		for _, g := range strings.Split(v[i], ",") {
+			if g = strings.TrimSpace(g); g != "" {
+				l.Roles[role] = append(l.Roles[role], g)
+			}
+		}
+	}
+	if len(l.Roles) == 0 {
+		l.Roles = nil
+	}
+	l.Memberships = nil
+	for _, pair := range strings.Split(v[8], ";") {
+		if pair = strings.TrimSpace(pair); pair == "" {
+			continue
+		}
+		g, place, ok := strings.Cut(pair, "=")
+		if !ok {
+			return l, fmt.Errorf("teams: %q is not GROUP=team:ORG/TEAM", pair)
+		}
+		if l.Memberships == nil {
+			l.Memberships = map[string]string{}
+		}
+		l.Memberships[strings.TrimSpace(g)] = strings.TrimSpace(place)
+	}
+	l.Label, l.CreateAccounts = v[9], v[10] == "yes"
+	return l, l.Validate()
+}
+
+// ldapYAML is a block as the cluster's settings write it.
+func ldapYAML(l config.LDAP) string {
+	m := map[string]any{"urls": l.URLs, "bind_dn": l.BindDN, "bind_password_file": l.BindPasswordFile,
+		"base_dn": l.BaseDN, "create_accounts": l.CreateAccounts}
+	put := func(k, v string) {
+		if v != "" {
+			m[k] = v
+		}
+	}
+	put("ca", l.CA)
+	put("ca_file", l.CAFile)
+	put("user_filter", l.UserFilter)
+	put("group_base_dn", l.GroupBaseDN)
+	put("label", l.Label)
+	if l.StartTLS {
+		m["start_tls"] = true
+	}
+	if len(l.Roles) > 0 {
+		m["roles"] = l.Roles
+	}
+	if len(l.Memberships) > 0 {
+		m["memberships"] = l.Memberships
+	}
+	if l.SyncInterval != 0 {
+		m["sync_interval"] = l.SyncInterval.String()
+	}
+	if l.Timeout != 0 {
+		m["timeout"] = l.Timeout.String()
+	}
+	b, _ := yaml.Marshal(m)
+	return string(b)
+}
+
+// saveDirectory tries the block, and only if every step passes keeps the
+// password and puts the block in the cluster's settings.
+func (m *consoleModel) saveDirectory(l config.LDAP, password string) tea.Cmd {
+	m.busy = "Active Directory"
+	c, block := m.client, ldapYAML(l)
+	return func() tea.Msg {
+		ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)
+		defer cancel()
+		var tv adminapi.LDAPTestView
+		if err := c.do(ctx, http.MethodPost, "/api/v1/auth/ldap/test",
+			adminapi.LDAPTestRequest{Settings: block, Password: password}, &tv); err != nil {
+			return actionDoneMsg{title: "Active Directory", err: err}
+		}
+		report := checksText(tv.Checks)
+		if !tv.OK {
+			return actionDoneMsg{title: "Active Directory", err: errors.New("not kept: the directory did not pass\n\n" + report)}
+		}
+		if password != "" {
+			if err := c.do(ctx, http.MethodPut, "/api/v1/auth/secrets/ldap-bind", adminapi.SecretRequest{Value: password}, nil); err != nil {
+				return actionDoneMsg{title: "Active Directory", err: err}
+			}
+		}
+		said, err := changeClusterSettings(ctx, c, "auth.ldap from the console", func(doc *settings.Doc) error {
+			return doc.SetYAML([]string{"auth", "ldap"}, block)
+		})
+		if err != nil {
+			return actionDoneMsg{title: "Active Directory", err: err}
+		}
+		return actionDoneMsg{title: "Active Directory", text: report + "\n\n" + said}
+	}
+}
+
+// tryDirectory tries the block in force.
+func (m *consoleModel) tryDirectory() tea.Cmd {
+	m.busy = "Trying the directory"
+	c, block := m.client, ldapYAML(m.signIn.ldap)
+	return func() tea.Msg {
+		ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)
+		defer cancel()
+		var tv adminapi.LDAPTestView
+		if err := c.do(ctx, http.MethodPost, "/api/v1/auth/ldap/test", adminapi.LDAPTestRequest{Settings: block}, &tv); err != nil {
+			return actionDoneMsg{title: "The directory", err: err}
+		}
+		if !tv.OK {
+			return actionDoneMsg{title: "The directory", err: errors.New("it did not pass\n\n" + checksText(tv.Checks))}
+		}
+		return actionDoneMsg{title: "The directory", text: checksText(tv.Checks)}
+	}
+}
+
+func checksText(checks []adminapi.CheckView) string {
+	var b strings.Builder
+	for i, c := range checks {
+		if i > 0 {
+			b.WriteString("\n")
+		}
+		mark := sOK.Render("✓")
+		if !c.OK {
+			mark = sBad.Render("✗")
+		}
+		fmt.Fprintf(&b, "%s %s: %s", mark, c.Step, c.Detail)
+	}
+	return b.String()
+}
+
+// entraForm sets up single sign-on through Entra ID, or another provider
+// when an issuer address is given in place of the tenant.
+func (m *consoleModel) entraForm() {
+	o := m.signIn.oidc
+	tenant := strings.TrimSuffix(strings.TrimPrefix(o.Issuer, "https://login.microsoftonline.com/"), "/v2.0")
+	secretHint := "the app registration's client secret"
+	if o.ClientSecretFile != "" {
+		secretHint = "empty keeps the one kept"
+	}
+	create, only := "yes", "no"
+	if o.Enabled() && !o.CreateAccounts {
+		create = "no"
+	}
+	if o.NoPasswords {
+		only = "yes"
+	}
+	secret := textF("client secret", "", secretHint)
+	secret.text.Mask = true
+	m.openForm(&formDialog{title: "Single sign-on: Entra ID",
+		note: "Register an app in Entra with the redirect address " + redirectURL(m.signIn.view.BaseURL) +
+			", and a groups claim of the groups assigned to it.",
+		fields: []formField{
+			textF("tenant", tenant, "the directory (tenant) ID; or another provider's issuer URL"),
+			textF("client ID", o.ClientID, "the application (client) ID"),
+			secret,
+			textF("administrators", strings.Join(o.Roles["admin"], ", "), "group object IDs, as Entra sends them"),
+			textF("operators", strings.Join(o.Roles["operator"], ", "), "group object IDs"),
+			textF("label", or(o.Label, "Microsoft"), "the button says: Sign in with LABEL"),
+			choiceF("make accounts", []string{"yes", "no"}, create),
+			choiceF("only this way", []string{"no", "yes"}, only),
+		},
+		submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
+			next, err := oidcFromForm(o, v)
+			if err != nil {
+				return nil, err
+			}
+			return m.saveOIDC(next, v[2]), nil
+		}})
+}
+
+func oidcFromForm(kept config.OIDC, v []string) (config.OIDC, error) {
+	o := kept
+	switch t := strings.TrimSpace(v[0]); {
+	case t == "":
+		return o, errors.New("tenant: the directory (tenant) ID, or an issuer URL")
+	case strings.Contains(t, "://"):
+		o.Issuer = t
+	default:
+		o.Issuer = "https://login.microsoftonline.com/" + t + "/v2.0"
+	}
+	if o.ClientID = v[1]; o.ClientID == "" {
+		return o, errors.New("client ID: the application (client) ID")
+	}
+	if v[2] != "" || o.ClientSecretFile == "" {
+		if v[2] == "" {
+			return o, errors.New("client secret: the app registration's secret")
+		}
+		o.ClientSecretFile = adminapi.AuthSecretRef("oidc-client-secret")
+	}
+	o.Roles = map[string][]string{}
+	for role, i := range map[string]int{"admin": 3, "operator": 4} {
+		for _, g := range strings.Split(v[i], ",") {
+			if g = strings.TrimSpace(g); g != "" {
+				o.Roles[role] = append(o.Roles[role], g)
+			}
+		}
+	}
+	if len(o.Roles) == 0 {
+		o.Roles = nil
+	}
+	o.Label, o.CreateAccounts, o.NoPasswords = v[5], v[6] == "yes", v[7] == "yes"
+	return o, o.Validate()
+}
+
+func oidcYAML(o config.OIDC) string {
+	m := map[string]any{"issuer": o.Issuer, "client_id": o.ClientID, "client_secret_file": o.ClientSecretFile,
+		"create_accounts": o.CreateAccounts}
+	if o.Label != "" {
+		m["label"] = o.Label
+	}
+	if o.NoPasswords {
+		m["no_passwords"] = true
+	}
+	if o.RedirectURL != "" {
+		m["redirect_url"] = o.RedirectURL
+	}
+	if len(o.Scopes) > 0 {
+		m["scopes"] = o.Scopes
+	}
+	if o.GroupsClaim != "" {
+		m["groups_claim"] = o.GroupsClaim
+	}
+	if len(o.Roles) > 0 {
+		m["roles"] = o.Roles
+	}
+	if len(o.Memberships) > 0 {
+		m["memberships"] = o.Memberships
+	}
+	b, _ := yaml.Marshal(m)
+	return string(b)
+}
+
+func (m *consoleModel) saveOIDC(o config.OIDC, secret string) tea.Cmd {
+	m.busy = "Single sign-on"
+	c, block, redirect := m.client, oidcYAML(o), redirectURL(m.signIn.view.BaseURL)
+	return func() tea.Msg {
+		ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
+		defer cancel()
+		if secret != "" {
+			if err := c.do(ctx, http.MethodPut, "/api/v1/auth/secrets/oidc-client-secret", adminapi.SecretRequest{Value: secret}, nil); err != nil {
+				return actionDoneMsg{title: "Single sign-on", err: err}
+			}
+		}
+		said, err := changeClusterSettings(ctx, c, "auth.oidc from the console", func(doc *settings.Doc) error {
+			return doc.SetYAML([]string{"auth", "oidc"}, block)
+		})
+		if err != nil {
+			return actionDoneMsg{title: "Single sign-on", err: err}
+		}
+		return actionDoneMsg{title: "Single sign-on", text: said + "\n\nIn the app registration the redirect address is " + redirect +
+			". For people in more than 200 groups, emit the groups assigned to the application."}
+	}
+}
+
+// scimToken makes a new provisioning token, keeps it, and shows it once.
+func (m *consoleModel) scimToken() tea.Cmd {
+	m.busy = "Provisioning"
+	c, base := m.client, m.signIn.view.BaseURL
+	return func() tea.Msg {
+		ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
+		defer cancel()
+		b := make([]byte, 32)
+		if _, err := rand.Read(b); err != nil {
+			return actionDoneMsg{title: "Provisioning", err: err}
+		}
+		token := hex.EncodeToString(b)
+		if err := c.do(ctx, http.MethodPut, "/api/v1/auth/secrets/scim-token", adminapi.SecretRequest{Value: token}, nil); err != nil {
+			return actionDoneMsg{title: "Provisioning", err: err}
+		}
+		said, err := changeClusterSettings(ctx, c, "auth.scim from the console", func(doc *settings.Doc) error {
+			return doc.SetYAML([]string{"auth", "scim", "token_file"}, adminapi.AuthSecretRef("scim-token"))
+		})
+		if err != nil {
+			return actionDoneMsg{title: "Provisioning", err: err}
+		}
+		url := "https://<this gateway>/scim/v2"
+		if base != "" {
+			url = strings.TrimRight(base, "/") + "/scim/v2"
+		}
+		return actionDoneMsg{title: "Provisioning", text: "Give the provider the tenant URL " + url + " and this token, shown only now:\n\n  " +
+			token + "\n\n" + said}
+	}
+}
+
+// signInOffConfirm turns one way in off, after asking.
+func (m *consoleModel) signInOffConfirm(key, title, text string) {
+	m.confirmD = &confirmDialog{title: "Turn off " + strings.ToLower(title), text: text,
+		yes: func(m *consoleModel) tea.Cmd {
+			m.busy = title
+			c := m.client
+			return func() tea.Msg {
+				ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
+				defer cancel()
+				said, err := changeClusterSettings(ctx, c, "auth."+key+" off from the console", func(doc *settings.Doc) error {
+					doc.Unset([]string{"auth", key})
+					return nil
+				})
+				return actionDoneMsg{title: title, text: said, err: err}
+			}
+		}}
+}
+
+// changeClusterSettings is changeSettings for the console: the newest
+// settings read, changed, and put in force, said in a sentence.
+func changeClusterSettings(ctx context.Context, c *ctlClient, note string, change func(doc *settings.Doc) error) (string, error) {
+	var cur adminapi.SettingsView
+	if err := c.get(ctx, "/api/v1/settings", &cur); err != nil {
+		return "", err
+	}
+	doc, err := settings.FromBytes("settings", []byte(cur.Body))
+	if err != nil {
+		return "", err
+	}
+	if err := change(doc); err != nil {
+		return "", err
+	}
+	body, err := doc.Bytes()
+	if err != nil {
+		return "", err
+	}
+	before, _ := config.Parse([]byte(cur.Body))
+	after, err := config.Parse(body)
+	if err != nil {
+		return "", err
+	}
+	changed := config.Changed(before, after)
+	if len(changed) == 0 {
+		return "Nothing changed.", nil
+	}
+	var v adminapi.SettingsView
+	if err := c.do(ctx, http.MethodPut, "/api/v1/settings",
+		adminapi.SettingsRequest{Base: cur.Version, Body: string(body), Note: note}, &v); err != nil {
+		return "", err
+	}
+	return fmt.Sprintf("Settings version %d. Every control plane starts again on it within seconds, standbys first; "+
+		"engines keep serving.", v.Version), nil
+}

internal/cli/tui_signin_test.go

@@ -0,0 +1,108 @@
+package cli
+
+import (
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+)
+
+// adForm is the directory form's values, in its order.
+func adForm(password, ca, allowed, teams string) []string {
+	return []string{"ldaps://dc1.corp.example, ldaps://dc2.corp.example", "OU=Staff,DC=corp,DC=example",
+		"CN=svc,OU=Service,DC=corp,DC=example", password, ca, allowed, "llmgw-admins", "", teams, "Acme", "yes"}
+}
+
+func TestTheDirectoryFormMakesTheBlockTheSettingsKeep(t *testing.T) {
+	pem := "-----BEGIN CERTIFICATE-----\nMIIB\n-----END CERTIFICATE-----\n"
+	caFile := filepath.Join(t.TempDir(), "ca.pem")
+	if err := os.WriteFile(caFile, []byte(pem), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	l, err := directoryFromForm(config.LDAP{}, adForm("pw", caFile, "CN=llmgw-users,OU=Groups,DC=corp,DC=example",
+		"ml-team=team:Acme/ml; sales=org:Sales"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	// The group, nested ones counting; the CA written into the settings;
+	// the password named in the store, not a file on one machine.
+	if l.UserFilter != "(memberOf:1.2.840.113556.1.4.1941:=CN=llmgw-users,OU=Groups,DC=corp,DC=example)" ||
+		l.CA != pem || l.BindPasswordFile != "store:auth/ldap-bind" || len(l.URLs) != 2 ||
+		l.Memberships["sales"] != "org:Sales" || l.Roles["admin"][0] != "llmgw-admins" || l.Roles["operator"] != nil {
+		t.Errorf("%+v", l)
+	}
+	// What the settings keep reads back the same.
+	back, err := config.ParseLDAP([]byte(ldapYAML(l)))
+	if err != nil || back.UserFilter != l.UserFilter || back.CA != pem || back.Memberships["ml-team"] != "team:Acme/ml" {
+		t.Errorf("read back %+v %v", back, err)
+	}
+
+	// Changed again with nothing typed for the password or the CA: both kept.
+	again, err := directoryFromForm(l, adForm("", "", "(objectClass=user)", ""))
+	if err != nil || again.CA != pem || again.BindPasswordFile != l.BindPasswordFile || again.UserFilter != "(objectClass=user)" ||
+		again.Memberships != nil {
+		t.Errorf("kept: %+v %v", again, err)
+	}
+	// A group by its bare name cannot be a filter, and a plaintext server is refused.
+	if _, err := directoryFromForm(config.LDAP{}, adForm("pw", "", "llmgw-users", "")); err == nil {
+		t.Error("a bare group name was taken")
+	}
+	plain := adForm("pw", "", "", "")
+	plain[0] = "ldap://dc1"
+	if _, err := directoryFromForm(config.LDAP{}, plain); err == nil || !strings.Contains(err.Error(), "plaintext") {
+		t.Errorf("plaintext: %v", err)
+	}
+}
+
+func TestTheEntraFormMakesTheIssuerFromTheTenant(t *testing.T) {
+	o, err := oidcFromForm(config.OIDC{}, []string{"0f2c-tenant", "app-id", "s3cret", "g-admins", "", "Microsoft", "yes", "no"})
+	if err != nil {
+		t.Fatal(err)
+	}
+	if o.Issuer != "https://login.microsoftonline.com/0f2c-tenant/v2.0" || o.ClientSecretFile != "store:auth/oidc-client-secret" ||
+		o.Roles["admin"][0] != "g-admins" || !o.CreateAccounts || o.NoPasswords {
+		t.Errorf("%+v", o)
+	}
+	// Another provider by its issuer; the secret kept when none is typed.
+	k, err := oidcFromForm(o, []string{"https://sso.example/realms/acme", "llmgw", "", "", "", "Acme SSO", "no", "yes"})
+	if err != nil || k.Issuer != "https://sso.example/realms/acme" || k.ClientSecretFile != o.ClientSecretFile || !k.NoPasswords {
+		t.Errorf("%+v %v", k, err)
+	}
+	if _, err := oidcFromForm(config.OIDC{}, []string{"t", "app", "", "", "", "", "yes", "no"}); err == nil {
+		t.Error("a first setup without a secret was taken")
+	}
+}
+
+func TestTheSignInPanelSaysWhatIsOnAndHowTheDirectoryLastWent(t *testing.T) {
+	m := consoleModel{menu: cSignIn, loaded: true, width: 120, height: 40}
+	m.signIn = signInState{loaded: true, view: adminapi.AuthView{
+		BaseURL: "https://llm.example",
+		OIDC:    &adminapi.OIDCView{Issuer: "https://login.microsoftonline.com/t/v2.0", Label: "Microsoft", CreateAccounts: true},
+		LDAP: &adminapi.LDAPView{URLs: []string{"ldaps://dc1"}, BaseDN: "OU=Staff,DC=corp",
+			LastSync: &adminapi.DirectorySyncView{At: time.Now().Add(-time.Minute), People: 40, Accounts: 12, Suspended: 1}},
+	}}
+	_, body, _ := m.signInPanel(110)
+	for _, want := range []string{"Single sign-on (Entra ID, OIDC)", "Microsoft", "https://llm.example/login/oidc/callback",
+		"Active Directory (LDAP)", "ldaps://dc1", "40 people there, 12 accounts kept in step, 1 suspended", "Provisioning (SCIM)"} {
+		if !strings.Contains(body, want) {
+			t.Errorf("the panel lacks %q:\n%s", want, body)
+		}
+	}
+	m.signIn.view.LDAP.Off = "auth.ldap.ca: holds no certificate"
+	if _, body, _ := m.signInPanel(110); !strings.Contains(body, "broken here") || !strings.Contains(body, "holds no certificate") {
+		t.Errorf("a directory off here is not said:\n%s", body)
+	}
+}
+
+func TestAuthBlocksAreReadFromTheSettingsDocument(t *testing.T) {
+	body := "auth:\n  ldap:\n    urls: [ldaps://dc1]\n    base_dn: DC=corp\n    bind_dn: CN=svc\n    bind_password_file: store:auth/ldap-bind\n" +
+		"    sync_interval: 1m\n  oidc:\n    issuer: https://sso.example\n    client_id: llmgw\n"
+	l, o := authBlocks(body)
+	if l.BaseDN != "DC=corp" || l.SyncInterval != time.Minute || o.ClientID != "llmgw" {
+		t.Errorf("%+v %+v", l, o)
+	}
+}

internal/config/config.go

@@ -442,6 +442,9 @@ type Auth struct {
 
 	// OIDC is sign-in through an identity provider.
 	OIDC OIDC `koanf:"oidc"`
+	// LDAP is sign-in against a directory, Active Directory above all, and
+	// the accounts kept in step with it.
+	LDAP LDAP `koanf:"ldap"`
 	// SCIM is account provisioning by the identity provider.
 	SCIM SCIM `koanf:"scim"`
 
@@ -500,6 +503,10 @@ type Personal struct {
 type Price struct {
 	PromptPer1k     int64 `koanf:"prompt_per_1k"`
 	CompletionPer1k int64 `koanf:"completion_per_1k"`
+	// CachedPer1k is the cost of a prompt token the engine or provider read
+	// from its cache (the answer's cached_tokens, counted within the
+	// prompt's). Zero charges them at prompt_per_1k.
+	CachedPer1k int64 `koanf:"cached_per_1k"`
 	// PerImage is the cost of one picture, for a model that makes them
 	// (/v1/images/generations), on top of its prompt's tokens.
 	PerImage int64 `koanf:"per_image"`
@@ -791,6 +798,11 @@ type Cache struct {
 	// MaxEntryBytes bounds one answer; a longer one is not kept. 1 MB
 	// unless set.
 	MaxEntryBytes int64 `koanf:"max_entry_bytes"`
+	// EmbeddingModel embeds the questions of projects that reuse answers to
+	// similar ones (llmgw keys cache --similar): an embedding model in the
+	// catalog. Unset, those projects reuse only the same request's answer.
+	// The gateway asks it on its own account, as it asks a judge.
+	EmbeddingModel string `koanf:"embedding_model"`
 }
 
 // Default returns the configuration used when nothing is specified.
@@ -968,6 +980,12 @@ func (s Server) Validate() error {
 	if err := s.Auth.OIDC.Validate(); err != nil {
 		return err
 	}
+	if err := s.Auth.LDAP.Validate(); err != nil {
+		return err
+	}
+	if s.Auth.LDAP.Enabled() && !s.Auth.Enabled {
+		return fmt.Errorf("auth.ldap needs auth.enabled: it signs people in to the accounts the store keeps")
+	}
 	if err := s.AgentTLS.Validate(); err != nil {
 		return err
 	}

internal/config/ldap.go

@@ -0,0 +1,160 @@
+package config
+
+import (
+	"errors"
+	"fmt"
+	"net/url"
+	"strings"
+	"time"
+
+	"github.com/knadh/koanf/parsers/yaml"
+	"github.com/knadh/koanf/v2"
+)
+
+// LDAP is sign-in against a directory — Active Directory above all — with
+// the directory's own passwords, and the accounts kept in step with it.
+//
+//	auth:
+//	  ldap:
+//	    urls: [ldaps://dc1.corp.example.com, ldaps://dc2.corp.example.com]
+//	    bind_dn: CN=llmgw,OU=Service Accounts,DC=corp,DC=example,DC=com
+//	    bind_password_file: /etc/llmgw/secrets/ldap-bind
+//	    base_dn: OU=Staff,DC=corp,DC=example,DC=com
+//	    create_accounts: true
+//	    roles: {admin: [llmgw-admins]}
+//	    memberships: {ml-team: "team:Acme/ml"}
+//
+// A person signs in on the portal's form with their sAMAccountName, their
+// userPrincipalName or their mail, and their directory password. The
+// gateway finds them with the bind account, checks the password by binding
+// as them, and reads their groups, nested ones included. Every sync_interval
+// the leader reads the directory again: an account disabled or gone there
+// is suspended here, its sessions ended and its API keys refused.
+type LDAP struct {
+	// URLs are the directory's servers, tried in turn: ldaps:// (636), or
+	// ldap:// (389) with StartTLS.
+	URLs []string `koanf:"urls"`
+	// StartTLS upgrades an ldap:// connection before anything is sent. A
+	// password never crosses a connection without TLS: ldap:// without it
+	// is refused.
+	StartTLS bool `koanf:"start_tls"`
+	// CAFile is the authority the servers' certificates are checked against,
+	// when it is not the system's: an AD Certificate Services root.
+	CAFile string `koanf:"ca_file"`
+	// CA is the same authority written out, PEM: kept in the cluster's
+	// settings, so no file has to be put on every control plane. Used in
+	// place of CAFile when both are set.
+	CA string `koanf:"ca"`
+	// BindDN and the file holding its password: a service account that may
+	// read users and groups. Nothing is written to the directory.
+	BindDN           string `koanf:"bind_dn"`
+	BindPasswordFile string `koanf:"bind_password_file"`
+	// BaseDN is where people are searched for, subtree included.
+	BaseDN string `koanf:"base_dn"`
+	// GroupBaseDN is where their groups are; empty is BaseDN's domain root.
+	GroupBaseDN string `koanf:"group_base_dn"`
+	// UserFilter narrows who may sign in, ANDed with the login match: a
+	// group (memberOf=...) or an OU's people only. Empty is every enabled
+	// user object.
+	UserFilter string `koanf:"user_filter"`
+	// Label is what the form says to sign in with: "Sign in with your
+	// <label> account". Empty is "Active Directory".
+	Label string `koanf:"label"`
+	// CreateAccounts makes an account, active, for a person the directory
+	// vouches for and the gateway does not know. Off, only existing accounts
+	// sign in this way.
+	CreateAccounts bool `koanf:"create_accounts"`
+	// Roles maps a staff role (viewer, operator, billing, admin) to the
+	// groups that give it, by name (CN) or whole DN. Set at every sign-in
+	// and sync: someone taken out of the group loses the role.
+	Roles map[string][]string `koanf:"roles"`
+	// Memberships maps a group, by CN or DN, to a place in the tree —
+	// org:NAME, team:ORG/TEAM, project:ORG/TEAM/PROJECT — joined at sign-in
+	// and kept by the sync, which also takes away a place whose group the
+	// person has left.
+	Memberships map[string]string `koanf:"memberships"`
+	// SyncInterval is how often the leader reads the directory. Zero is
+	// 15m; negative turns the sync off, leaving only sign-in.
+	SyncInterval time.Duration `koanf:"sync_interval"`
+	// Timeout bounds each round trip to a server. Zero is 10s.
+	Timeout time.Duration `koanf:"timeout"`
+}
+
+// Enabled reports whether directory sign-in is configured.
+func (l LDAP) Enabled() bool { return len(l.URLs) > 0 }
+
+// Validate says what in the configuration cannot work.
+func (l LDAP) Validate() error {
+	if !l.Enabled() {
+		return nil
+	}
+	for _, raw := range l.URLs {
+		u, err := url.Parse(raw)
+		if err != nil || u.Host == "" {
+			return fmt.Errorf("auth.ldap.urls: %q is not an ldap:// or ldaps:// address", raw)
+		}
+		switch u.Scheme {
+		case "ldaps":
+		case "ldap":
+			if !l.StartTLS {
+				return fmt.Errorf("auth.ldap.urls: %q is plaintext: use ldaps://, or set start_tls; passwords would cross it", raw)
+			}
+		default:
+			return fmt.Errorf("auth.ldap.urls: %q is not an ldap:// or ldaps:// address", raw)
+		}
+	}
+	switch {
+	case strings.TrimSpace(l.BindDN) == "" || l.BindPasswordFile == "":
+		return errors.New("auth.ldap: bind_dn and bind_password_file are needed, for the account that finds people")
+	case l.CA != "" && !strings.Contains(l.CA, "-----BEGIN CERTIFICATE-----"):
+		return errors.New("auth.ldap.ca: not a PEM certificate")
+	case strings.TrimSpace(l.BaseDN) == "":
+		return errors.New("auth.ldap: base_dn is needed, where people are searched for")
+	case l.UserFilter != "" && !(strings.HasPrefix(l.UserFilter, "(") && strings.HasSuffix(l.UserFilter, ")")):
+		return fmt.Errorf("auth.ldap.user_filter: %q is not an LDAP filter in parentheses", l.UserFilter)
+	}
+	for role := range l.Roles {
+		if !validStaffRole(role) {
+			return fmt.Errorf("auth.ldap.roles: %q is not viewer, operator, billing or admin", role)
+		}
+	}
+	for g, place := range l.Memberships {
+		if !validPlace(place) {
+			return fmt.Errorf("auth.ldap.memberships[%s]: %q is not org:NAME, team:ORG/TEAM or project:ORG/TEAM/PROJECT", g, place)
+		}
+	}
+	return nil
+}
+
+// Interval is SyncInterval with its default; zero means off.
+func (l LDAP) Interval() time.Duration {
+	switch {
+	case l.SyncInterval < 0:
+		return 0
+	case l.SyncInterval == 0:
+		return 15 * time.Minute
+	}
+	return l.SyncInterval
+}
+
+// RoundTrip is Timeout with its default.
+func (l LDAP) RoundTrip() time.Duration {
+	if l.Timeout <= 0 {
+		return 10 * time.Second
+	}
+	return l.Timeout
+}
+
+// ParseLDAP is an auth.ldap block, as YAML, read on its own as the cluster's
+// settings would read it, and checked.
+func ParseLDAP(block []byte) (LDAP, error) {
+	var l LDAP
+	k := koanf.New(".")
+	if err := k.Load(bytesProvider(block), yaml.Parser()); err != nil {
+		return l, fmt.Errorf("auth.ldap: %w", err)
+	}
+	if err := k.UnmarshalWithConf("", &l, koanf.UnmarshalConf{Tag: "koanf"}); err != nil {
+		return l, fmt.Errorf("auth.ldap: %w", err)
+	}
+	return l, l.Validate()
+}

internal/config/oidc.go

@@ -72,21 +72,34 @@ func (o OIDC) Validate() error {
 		return errors.New("auth.oidc.client_id is required")
 	}
 	for role := range o.Roles {
-		switch role {
-		case "viewer", "operator", "billing", "admin":
-		default:
+		if !validStaffRole(role) {
 			return fmt.Errorf("auth.oidc.roles: %q is not viewer, operator, billing or admin", role)
 		}
 	}
 	for g, where := range o.Memberships {
-		kind, path, ok := strings.Cut(where, ":")
-		if !ok || path == "" || (kind != "org" && kind != "team" && kind != "project") {
+		if !validPlace(where) {
 			return fmt.Errorf("auth.oidc.memberships.%s is %q: org:NAME, team:ORG/TEAM or project:ORG/TEAM/PROJECT", g, where)
 		}
 	}
 	return nil
 }
 
+// validStaffRole is a role groups may give: viewer, operator, billing, admin.
+func validStaffRole(role string) bool {
+	switch role {
+	case "viewer", "operator", "billing", "admin":
+		return true
+	}
+	return false
+}
+
+// validPlace is a place in the tree a group may give: org:NAME,
+// team:ORG/TEAM or project:ORG/TEAM/PROJECT.
+func validPlace(where string) bool {
+	kind, path, ok := strings.Cut(where, ":")
+	return ok && path != "" && (kind == "org" || kind == "team" || kind == "project")
+}
+
 // SCIM lets an identity provider provision accounts (/scim/v2): made,
 // suspended and reinstated as they are there, with groups mapped by
 // auth.oidc's roles and memberships.

internal/control/adminapi/api.go

@@ -76,6 +76,8 @@ type API struct {
 	// train is adapter training; nil without a store and agents.
 	train    *TrainAdmin
 	settings *Settings
+	auth     *AuthAdmin
+	mcp      MCPStore
 	// restartAgents tells every connected agent to start again.
 	restartAgents func(reason string) []string
 	// revoke records a removed node, so its certificate is refused.
@@ -129,6 +131,8 @@ func (a *API) Register(mux Mux) {
 	a.registerTrain(mux)
 	a.registerGuard(mux)
 	a.registerSettings(mux)
+	a.registerAuth(mux)
+	a.registerMCP(mux)
 	mux.HandleFunc("POST /api/v1/agents/restart", a.restartAgentsHandler)
 	mux.HandleFunc("GET /api/v1/audit", a.auditEvents)
 	mux.HandleFunc("GET /api/v1/usage/export", a.usageExport)

internal/control/adminapi/auth.go

@@ -0,0 +1,170 @@
+package adminapi
+
+import (
+	"context"
+	"encoding/json"
+	"log/slog"
+	"net/http"
+	"strings"
+	"time"
+)
+
+// AuthAdmin is what the sign-in endpoints need: how people sign in now, a
+// directory's settings tried before they are kept, and where the secrets
+// those settings name are kept for every control plane.
+type AuthAdmin struct {
+	Status func() AuthView
+	// TestLDAP tries an auth.ldap block, as YAML, binding with password
+	// when one is given rather than the one the block names.
+	TestLDAP func(ctx context.Context, settings, password string) ([]CheckView, error)
+	Secrets  interface {
+		SetClusterSecret(ctx context.Context, name, value string) error
+	}
+	Log *slog.Logger
+}
+
+// SetAuth enables the sign-in endpoints.
+func (a *API) SetAuth(x *AuthAdmin) { a.auth = x }
+
+// AuthView is how people sign in on this gateway.
+type AuthView struct {
+	SignupOpen bool `json:"signup_open"`
+	// OIDC is single sign-on through an identity provider: Entra ID,
+	// Keycloak, Okta.
+	OIDC *OIDCView `json:"oidc,omitempty"`
+	// LDAP is sign-in checked by a directory: Active Directory.
+	LDAP *LDAPView `json:"ldap,omitempty"`
+	// SCIM is whether a provider may keep accounts in step.
+	SCIM bool `json:"scim"`
+	// BaseURL is the portal's address, when the settings say it: what a
+	// provider's redirect address is made from.
+	BaseURL string `json:"base_url,omitempty"`
+}
+
+// OIDCView is auth.oidc as it is in force.
+type OIDCView struct {
+	Issuer         string `json:"issuer"`
+	ClientID       string `json:"client_id"`
+	Label          string `json:"label,omitempty"`
+	CreateAccounts bool   `json:"create_accounts"`
+	NoPasswords    bool   `json:"no_passwords"`
+}
+
+// LDAPView is auth.ldap as it is in force, and how its last sync went.
+type LDAPView struct {
+	URLs   []string `json:"urls"`
+	BaseDN string   `json:"base_dn"`
+	BindDN string   `json:"bind_dn"`
+	Label  string   `json:"label,omitempty"`
+	// Off is why this control plane could not use the settings, when it
+	// could not; local accounts still sign in.
+	Off      string             `json:"off,omitempty"`
+	LastSync *DirectorySyncView `json:"last_sync,omitempty"`
+}
+
+// DirectorySyncView is the last time the leader read the directory.
+type DirectorySyncView struct {
+	At           time.Time `json:"at"`
+	People       int       `json:"people"`
+	Accounts     int       `json:"accounts"`
+	Suspended    int       `json:"suspended"`
+	Reinstated   int       `json:"reinstated"`
+	RolesChanged int       `json:"roles_changed"`
+	Error        string    `json:"error,omitempty"`
+}
+
+// CheckView is one step of trying a directory.
+type CheckView struct {
+	Step   string `json:"step"`
+	OK     bool   `json:"ok"`
+	Detail string `json:"detail"`
+}
+
+// LDAPTestRequest is an auth.ldap block to try, as YAML, and the service
+// account's password when it is not kept yet.
+type LDAPTestRequest struct {
+	Settings string `json:"settings"`
+	Password string `json:"password,omitempty"`
+}
+
+// LDAPTestView is how trying the directory went.
+type LDAPTestView struct {
+	OK     bool        `json:"ok"`
+	Checks []CheckView `json:"checks"`
+}
+
+// SecretRequest is a secret for the store to keep.
+type SecretRequest struct {
+	Value string `json:"value"`
+}
+
+// authSecrets are the secrets an operator may give through the API, and
+// the reference the settings then name each by.
+var authSecrets = map[string]bool{"ldap-bind": true, "oidc-client-secret": true, "scim-token": true}
+
+// AuthSecretRef is the reference auth's settings name a kept secret by.
+func AuthSecretRef(name string) string { return "store:auth/" + name }
+
+func (a *API) registerAuth(mux Mux) {
+	need := func(w http.ResponseWriter) bool {
+		if a.auth == nil {
+			writeError(w, http.StatusNotImplemented, "this control plane has no accounts")
+			return false
+		}
+		return true
+	}
+	mux.HandleFunc("GET /api/v1/auth", func(w http.ResponseWriter, r *http.Request) {
+		if !need(w) {
+			return
+		}
+		writeJSON(w, http.StatusOK, a.auth.Status())
+	})
+	mux.HandleFunc("POST /api/v1/auth/ldap/test", func(w http.ResponseWriter, r *http.Request) {
+		if !need(w) {
+			return
+		}
+		var req LDAPTestRequest
+		if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
+			writeError(w, http.StatusBadRequest, err.Error())
+			return
+		}
+		checks, err := a.auth.TestLDAP(r.Context(), req.Settings, req.Password)
+		if err != nil {
+			writeError(w, http.StatusBadRequest, err.Error())
+			return
+		}
+		v := LDAPTestView{OK: len(checks) > 0, Checks: checks}
+		for _, c := range checks {
+			v.OK = v.OK && c.OK
+		}
+		writeJSON(w, http.StatusOK, v)
+	})
+	mux.HandleFunc("PUT /api/v1/auth/secrets/{name}", func(w http.ResponseWriter, r *http.Request) {
+		if !need(w) {
+			return
+		}
+		name := r.PathValue("name")
+		if !authSecrets[name] {
+			writeError(w, http.StatusNotFound, "no such secret: ldap-bind, oidc-client-secret or scim-token")
+			return
+		}
+		var req SecretRequest
+		if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
+			writeError(w, http.StatusBadRequest, err.Error())
+			return
+		}
+		if strings.TrimSpace(req.Value) == "" {
+			writeError(w, http.StatusBadRequest, "an empty secret")
+			return
+		}
+		if err := a.auth.Secrets.SetClusterSecret(r.Context(), "auth/"+name, req.Value); err != nil {
+			writeError(w, http.StatusInternalServerError, err.Error())
+			return
+		}
+		if a.auth.Log != nil {
+			// What changed and who changed it; never the value.
+			a.auth.Log.Info("a sign-in secret was set", "secret", name, "by", actorOf(r))
+		}
+		writeJSON(w, http.StatusOK, map[string]string{"ref": AuthSecretRef(name)})
+	})
+}

internal/control/adminapi/auth_test.go

@@ -0,0 +1,53 @@
+package adminapi
+
+import (
+	"context"
+	"strings"
+	"testing"
+)
+
+type memSecrets map[string]string
+
+func (m memSecrets) SetClusterSecret(_ context.Context, name, value string) error {
+	m[name] = value
+	return nil
+}
+
+func TestSignInIsShownTriedAndItsSecretsKept(t *testing.T) {
+	a, _ := setup(t, testModels())
+	kept := memSecrets{}
+	var tried string
+	a.SetAuth(&AuthAdmin{
+		Status: func() AuthView { return AuthView{LDAP: &LDAPView{URLs: []string{"ldaps://dc1"}}} },
+		TestLDAP: func(_ context.Context, settings, password string) ([]CheckView, error) {
+			tried = settings + "|" + password
+			return []CheckView{{Step: "ldaps://dc1", OK: true}, {Step: "people", OK: false, Detail: "no one"}}, nil
+		},
+		Secrets: kept,
+	})
+	var v AuthView
+	if code := call(t, a, "GET", "/api/v1/auth", "", &v); code != 200 || v.LDAP == nil {
+		t.Fatalf("status: %d %+v", code, v)
+	}
+	// One step failing fails the whole.
+	var tv LDAPTestView
+	if code := call(t, a, "POST", "/api/v1/auth/ldap/test", `{"settings":"urls: [ldaps://dc1]","password":"pw"}`, &tv); code != 200 ||
+		tv.OK || len(tv.Checks) != 2 || tried != "urls: [ldaps://dc1]|pw" {
+		t.Errorf("test: %d %+v %q", code, tv, tried)
+	}
+	var ref map[string]string
+	if code := call(t, a, "PUT", "/api/v1/auth/secrets/ldap-bind", `{"value":"s3cret"}`, &ref); code != 200 ||
+		kept["auth/ldap-bind"] != "s3cret" || ref["ref"] != "store:auth/ldap-bind" {
+		t.Errorf("secret: %d %v %v", code, kept, ref)
+	}
+	// Only the sign-in secrets, and never an empty one.
+	if code := call(t, a, "PUT", "/api/v1/auth/secrets/engine-key", `{"value":"x"}`, nil); code != 404 {
+		t.Errorf("another secret: %d", code)
+	}
+	if code := call(t, a, "PUT", "/api/v1/auth/secrets/ldap-bind", `{"value":"  "}`, nil); code != 400 {
+		t.Errorf("an empty secret: %d", code)
+	}
+	if strings.Contains(kept["auth/ldap-bind"], " ") {
+		t.Error("an empty secret replaced the kept one")
+	}
+}

internal/control/adminapi/guard.go

@@ -209,6 +209,7 @@ type GuardAttachmentRequest struct {
 func (a *API) registerGuard(mux Mux) {
 	mux.HandleFunc("GET /api/v1/guard/judges", a.listJudges)
 	mux.HandleFunc("POST /api/v1/guard/judges/ask", a.askJudge)
+	mux.HandleFunc("POST /api/v1/guard/judges/check", a.checkJudge)
 	mux.HandleFunc("GET /api/v1/guard", func(w http.ResponseWriter, r *http.Request) {
 		if a.guard == nil {
 			writeError(w, http.StatusNotImplemented, "no guard policies: define them under guard in the cluster's settings (llmgw ctl config edit)")

internal/control/adminapi/judges.go

@@ -7,6 +7,7 @@ import (
 	"slices"
 	"sort"
 	"strings"
+	"sync"
 	"time"
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
@@ -30,6 +31,9 @@ type JudgeView struct {
 	Judge     bool `json:"judge"`
 	Ready     int  `json:"ready_replicas"`
 	Wanted    int  `json:"wanted_replicas"`
+	// BudgetMs is how long a request waits for it: the shortest of its
+	// policies' latency budgets.
+	BudgetMs int64 `json:"budget_ms,omitempty"`
 	// LastError is its last failure to give a verdict, if any.
 	LastError   string     `json:"last_error,omitempty"`
 	LastErrorAt *time.Time `json:"last_error_at,omitempty"`
@@ -65,6 +69,9 @@ func (a *API) Judges() []JudgeView {
 			if !slices.Contains(v.Policies, name) {
 				v.Policies = append(v.Policies, name)
 			}
+			if b := p.Judge.Budget().Milliseconds(); v.BudgetMs == 0 || b < v.BudgetMs {
+				v.BudgetMs = b
+			}
 			if p.Fail == "closed" && p.Mode != "shadow" && !slices.Contains(v.Closed, name) {
 				v.Closed = append(v.Closed, name)
 			}
@@ -198,3 +205,148 @@ func (a *API) askJudge(w http.ResponseWriter, r *http.Request) {
 	}
 	writeJSON(w, http.StatusOK, v)
 }
+
+// JudgeCheckRequest asks how a judge keeps up: Burst judgments at once after
+// three in turn; zero is two for each ready replica.
+type JudgeCheckRequest struct {
+	Model string `json:"model,omitempty"`
+	Burst int    `json:"burst,omitempty"`
+}
+
+// JudgeCheckView is how long a judgment took, alone and among others, set
+// against the time a request waits for one.
+type JudgeCheckView struct {
+	Model    string `json:"model"`
+	BudgetMs int64  `json:"budget_ms"`
+	Ready    int    `json:"ready_replicas"`
+	// TookMs are three judgments of the same text, one after another.
+	TookMs    []int64 `json:"took_ms"`
+	SlowestMs int64   `json:"slowest_ms"`
+	// Burst judgments at once: the slowest of them, how many gave no
+	// verdict, and how many a second the judges read.
+	Burst          int     `json:"burst"`
+	BurstSlowestMs int64   `json:"burst_slowest_ms"`
+	BurstFailed    int     `json:"burst_failed"`
+	PerSecond      float64 `json:"judgments_per_second"`
+	// Verdict is ok, tight (past half the budget alone, or past it among
+	// others) or slow (past it alone); Advice what to do when not ok.
+	Verdict string   `json:"verdict"`
+	Advice  []string `json:"advice,omitempty"`
+}
+
+// judgeCheckText is a prompt of about 280 tokens, the length the judges
+// were measured at: ordinary work, so the verdict is safe and the time is
+// what a request pays.
+const judgeCheckText = "Our support team has been asked to write a short guide for new customers who are setting " +
+	"up the gateway for the first time. The guide should explain how to create an API key in the portal, how to " +
+	"choose a model from the list the portal shows, and how to send a first request with curl or with the OpenAI " +
+	"Python client by changing only the base URL and the key. It should also say what the usage page shows, how " +
+	"budgets and rate limits are set by an administrator for each team, and what a person should do when a " +
+	"request is refused because a limit was reached. Please keep the tone friendly and plain, use short " +
+	"paragraphs, and add a small table that lists the three most common error codes with what each means and " +
+	"what to try next. The audience is technical but new to the product, so avoid internal names, and end with " +
+	"a short checklist the reader can follow on their first day. Write it in English, about four hundred words."
+
+func (a *API) checkJudge(w http.ResponseWriter, r *http.Request) {
+	if a.guard == nil || a.guard.Ask == nil {
+		writeError(w, http.StatusNotImplemented, "no guard policies: define them under guard in the cluster's settings (llmgw ctl config edit)")
+		return
+	}
+	var req JudgeCheckRequest
+	_ = json.NewDecoder(r.Body).Decode(&req)
+	var jv *JudgeView
+	for _, j := range a.Judges() {
+		if req.Model == "" || j.Model == req.Model {
+			j := j
+			jv = &j
+			break
+		}
+	}
+	if jv == nil {
+		writeError(w, http.StatusBadRequest, "no policy asks a judge of that name")
+		return
+	}
+	if jv.Ready == 0 {
+		writeError(w, http.StatusConflict, "the judge "+jv.Model+" has no replica ready: "+jv.Problem)
+		return
+	}
+	v := JudgeCheckView{Model: jv.Model, BudgetMs: jv.BudgetMs, Ready: jv.Ready, Burst: req.Burst}
+	if v.Burst <= 0 {
+		v.Burst = 2 * jv.Ready
+	}
+	v.Burst = min(v.Burst, 32)
+	var mu sync.Mutex
+	start := time.Now()
+	// Each text begins differently: the engine keeps what it read before,
+	// and the same text again is answered from that in a tenth of the time
+	// a request's own prompt takes.
+	n := 0
+	text := func() string {
+		mu.Lock()
+		defer mu.Unlock()
+		n++
+		return fmt.Sprintf("Request %d of a check begun at %s.\n", n, start.Format(time.RFC3339Nano)) + judgeCheckText
+	}
+	for range 3 {
+		ans, err := a.guard.Ask(r.Context(), jv.Model, text(), "")
+		if err != nil {
+			writeError(w, http.StatusBadGateway, fmt.Sprintf("the judge %s gave no verdict: %v", jv.Model, err))
+			return
+		}
+		v.TookMs = append(v.TookMs, ans.Took.Milliseconds())
+		v.SlowestMs = max(v.SlowestMs, ans.Took.Milliseconds())
+	}
+	var wg sync.WaitGroup
+	start = time.Now()
+	for range v.Burst {
+		wg.Add(1)
+		go func() {
+			defer wg.Done()
+			ans, err := a.guard.Ask(r.Context(), jv.Model, text(), "")
+			mu.Lock()
+			defer mu.Unlock()
+			if err != nil {
+				v.BurstFailed++
+				return
+			}
+			v.BurstSlowestMs = max(v.BurstSlowestMs, ans.Took.Milliseconds())
+		}()
+	}
+	wg.Wait()
+	if wall := time.Since(start).Seconds(); wall > 0 {
+		v.PerSecond = float64(v.Burst-v.BurstFailed) / wall
+	}
+	v.Verdict, v.Advice = judgeVerdict(v, *jv)
+	writeJSON(w, http.StatusOK, v)
+}
+
+// judgeVerdict reads a check: a judgment alone past the budget refuses every
+// request under fail: closed; past half of it, or past it among others, a
+// busy moment does.
+func judgeVerdict(v JudgeCheckView, jv JudgeView) (string, []string) {
+	verdict := "ok"
+	switch {
+	case v.BudgetMs > 0 && v.SlowestMs > v.BudgetMs:
+		verdict = "slow"
+	case v.BudgetMs > 0 && (v.SlowestMs*2 > v.BudgetMs || v.BurstSlowestMs > v.BudgetMs || v.BurstFailed > 0):
+		verdict = "tight"
+	}
+	if verdict == "ok" {
+		return verdict, nil
+	}
+	refused := "served unjudged"
+	if len(jv.Closed) > 0 {
+		refused = "refused"
+	}
+	advice := []string{
+		fmt.Sprintf("a judgment past %d ms is %s: each request is judged once for its prompt and again every 600 characters of its answer", v.BudgetMs, refused),
+		"more cores for each judge: llmgw ctl model resources " + v.Model + " --cores 4, then llmgw ctl restart --model " + v.Model,
+		"more judges, on machines of their own: llmgw ctl scale " + v.Model + " N, or judge machines at install",
+		"a judge on a GPU: llmgw ctl guard judge use qwen3guard-4b " + v.Model,
+		"a longer wait: guard.policies.<policy>.judge.latency_budget in llmgw ctl config edit",
+	}
+	if len(jv.Closed) > 0 {
+		advice = append(advice, "or fail: open, which serves what the judge did not read, flagged")
+	}
+	return verdict, advice
+}

internal/control/adminapi/judges_test.go

@@ -123,3 +123,44 @@ func TestANodeSaysWhetherItsFabricsAreUp(t *testing.T) {
 		t.Errorf("no fabric: %q", got)
 	}
 }
+
+// A judge alone past its budget is slow; past half of it, or past it among
+// others, tight; and what to do is said, fail: open only where it is closed.
+func TestAJudgeCheckIsReadAgainstTheBudget(t *testing.T) {
+	closed := JudgeView{Model: "qwen3guard-0.6b", Closed: []string{"baseline"}}
+	for _, c := range []struct {
+		alone, burst int64
+		failed       int
+		want         string
+	}{{1200, 2400, 0, "ok"}, {2600, 3000, 0, "tight"}, {1200, 6000, 0, "tight"}, {1200, 2000, 1, "tight"}, {5200, 9000, 0, "slow"}} {
+		v := JudgeCheckView{Model: closed.Model, BudgetMs: 5000, SlowestMs: c.alone, BurstSlowestMs: c.burst, BurstFailed: c.failed}
+		got, advice := judgeVerdict(v, closed)
+		if got != c.want || (got == "ok") != (advice == nil) {
+			t.Errorf("%+v: %s %v", c, got, advice)
+		}
+		if got != "ok" && !strings.Contains(strings.Join(advice, " "), "fail: open") {
+			t.Errorf("a closed policy is not offered fail: open: %v", advice)
+		}
+	}
+	if _, advice := judgeVerdict(JudgeCheckView{BudgetMs: 5000, SlowestMs: 6000}, JudgeView{}); strings.Contains(strings.Join(advice, " "), "fail: open") {
+		t.Error("fail: open offered to a policy already open")
+	}
+}
+
+func TestAJudgeWithNoReplicaIsNotTimed(t *testing.T) {
+	a, _ := setup(t, testModels())
+	lib := guard.Config{Policies: map[string]guard.Policy{
+		"baseline": {Judge: &guard.JudgePolicy{Model: "nowhere-judge", Input: "block", LatencyBudget: 5 * time.Second}}}}
+	a.SetGuard(&GuardAdmin{Library: func() guard.Config { return lib },
+		Ask: func(context.Context, string, string, string) (guard.JudgeAnswer, error) {
+			return guard.JudgeAnswer{}, nil
+		}})
+	if code := call(t, a, "POST", "/api/v1/guard/judges/check", `{}`, nil); code != 409 {
+		t.Errorf("a judge with no replica: %d", code)
+	}
+	var judges []JudgeView
+	call(t, a, "GET", "/api/v1/guard/judges", "", &judges)
+	if len(judges) != 1 || judges[0].BudgetMs != 5000 {
+		t.Errorf("judges %+v", judges)
+	}
+}

internal/control/adminapi/mcp.go

@@ -0,0 +1,240 @@
+package adminapi
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"net/http"
+	"net/url"
+	"sort"
+	"strings"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/mcpclient"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/secrets"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// MCPStore keeps the company's MCP servers and the secrets their headers
+// carry.
+type MCPStore interface {
+	MCPServers(ctx context.Context) ([]store.MCPServer, error)
+	MCPServer(ctx context.Context, name string) (store.MCPServer, error)
+	PutMCPServer(ctx context.Context, m store.MCPServer) error
+	DeleteMCPServer(ctx context.Context, name string) error
+	SetClusterSecret(ctx context.Context, name, value string) error
+}
+
+// SetMCP enables the endpoints for the company's MCP servers.
+func (a *API) SetMCP(st MCPStore) { a.mcp = st }
+
+// MCPServerView is a server as the API shows it: header values only for the
+// headers sent as written, and for the others the reference read.
+type MCPServerView struct {
+	Name           string            `json:"name"`
+	URL            string            `json:"url"`
+	Description    string            `json:"description,omitempty"`
+	Headers        map[string]string `json:"headers,omitempty"`
+	HeaderRefs     map[string]string `json:"header_refs,omitempty"`
+	Enabled        bool              `json:"enabled"`
+	TimeoutSeconds int               `json:"timeout_seconds,omitempty"`
+	RatePerMinute  int               `json:"rate_per_minute,omitempty"`
+	CreatedAt      time.Time         `json:"created_at"`
+	CreatedBy      string            `json:"created_by,omitempty"`
+}
+
+// MCPServerRequest adds a server or changes one. Secrets are header values
+// to keep in the store, sealed, each then read by reference; HeaderRefs name
+// secrets kept elsewhere (a file, env:, vault:).
+type MCPServerRequest struct {
+	URL            string            `json:"url"`
+	Description    string            `json:"description,omitempty"`
+	Headers        map[string]string `json:"headers,omitempty"`
+	HeaderRefs     map[string]string `json:"header_refs,omitempty"`
+	Secrets        map[string]string `json:"secrets,omitempty"`
+	Enabled        *bool             `json:"enabled,omitempty"`
+	TimeoutSeconds int               `json:"timeout_seconds,omitempty"`
+	// RatePerMinute bounds each caller's calls to its tools; a negative
+	// number takes the bound away.
+	RatePerMinute int `json:"rate_per_minute,omitempty"`
+}
+
+// MCPTestView is what the server said when asked: its tools, or why not.
+type MCPTestView struct {
+	OK    bool     `json:"ok"`
+	Tools []string `json:"tools,omitempty"`
+	Error string   `json:"error,omitempty"`
+}
+
+func mcpServerView(m store.MCPServer) MCPServerView {
+	return MCPServerView{Name: m.Name, URL: m.URL, Description: m.Description, Headers: m.Headers,
+		HeaderRefs: m.HeaderRefs, Enabled: m.Enabled, TimeoutSeconds: int(m.Timeout / time.Second), RatePerMinute: m.RatePerMinute,
+		CreatedAt: m.CreatedAt, CreatedBy: m.CreatedBy}
+}
+
+func (a *API) registerMCP(mux Mux) {
+	need := func(w http.ResponseWriter) bool {
+		if a.mcp == nil {
+			writeError(w, http.StatusNotImplemented, "this control plane has no store to keep MCP servers in")
+			return false
+		}
+		return true
+	}
+	mux.HandleFunc("POST /api/v1/mcp", a.MCP)
+	mux.HandleFunc("GET /api/v1/mcp/servers", func(w http.ResponseWriter, r *http.Request) {
+		if !need(w) {
+			return
+		}
+		list, err := a.mcp.MCPServers(r.Context())
+		if err != nil {
+			writeError(w, http.StatusInternalServerError, err.Error())
+			return
+		}
+		out := make([]MCPServerView, len(list))
+		for i, m := range list {
+			out[i] = mcpServerView(m)
+		}
+		writeJSON(w, http.StatusOK, out)
+	})
+	mux.HandleFunc("PUT /api/v1/mcp/servers/{name}", func(w http.ResponseWriter, r *http.Request) {
+		if !need(w) {
+			return
+		}
+		name := r.PathValue("name")
+		var req MCPServerRequest
+		if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
+			writeError(w, http.StatusBadRequest, err.Error())
+			return
+		}
+		m, err := a.mcpServerFrom(r.Context(), name, req, actorOf(r))
+		if err != nil {
+			writeError(w, http.StatusBadRequest, err.Error())
+			return
+		}
+		if err := a.mcp.PutMCPServer(r.Context(), m); err != nil {
+			writeError(w, http.StatusInternalServerError, err.Error())
+			return
+		}
+		writeJSON(w, http.StatusOK, mcpServerView(m))
+	})
+	mux.HandleFunc("DELETE /api/v1/mcp/servers/{name}", func(w http.ResponseWriter, r *http.Request) {
+		if !need(w) {
+			return
+		}
+		switch err := a.mcp.DeleteMCPServer(r.Context(), r.PathValue("name")); {
+		case errors.Is(err, store.ErrNotFound):
+			writeError(w, http.StatusNotFound, "no MCP server "+r.PathValue("name"))
+		case err != nil:
+			writeError(w, http.StatusInternalServerError, err.Error())
+		default:
+			writeJSON(w, http.StatusOK, map[string]string{"removed": r.PathValue("name")})
+		}
+	})
+	mux.HandleFunc("POST /api/v1/mcp/servers/{name}/test", func(w http.ResponseWriter, r *http.Request) {
+		if !need(w) {
+			return
+		}
+		m, err := a.mcp.MCPServer(r.Context(), r.PathValue("name"))
+		if errors.Is(err, store.ErrNotFound) {
+			writeError(w, http.StatusNotFound, "no MCP server "+r.PathValue("name"))
+			return
+		}
+		if err != nil {
+			writeError(w, http.StatusInternalServerError, err.Error())
+			return
+		}
+		writeJSON(w, http.StatusOK, a.testMCPServer(r.Context(), m))
+	})
+}
+
+// mcpServerFrom is the server a request makes, over the one of that name if
+// there is one: its secrets kept in the store, and named by reference.
+func (a *API) mcpServerFrom(ctx context.Context, name string, req MCPServerRequest, by string) (store.MCPServer, error) {
+	if !store.ValidMCPServerName(name) {
+		return store.MCPServer{}, errors.New("a server's name is lowercase letters, digits, - and _, starting with a letter: " +
+			"its tools are NAME.tool")
+	}
+	m, err := a.mcp.MCPServer(ctx, name)
+	switch {
+	case errors.Is(err, store.ErrNotFound):
+		m = store.MCPServer{Name: name, Enabled: true, CreatedBy: by}
+	case err != nil:
+		return m, err
+	}
+	if req.URL != "" {
+		m.URL = req.URL
+	}
+	if u, err := url.Parse(m.URL); m.URL != AdminMCPAddress && (err != nil || (u.Scheme != "https" && u.Scheme != "http") || u.Host == "") {
+		return m, errors.New("the server's url is an http:// or https:// address, or " + AdminMCPAddress + " for this cluster's own")
+	}
+	if req.Description != "" {
+		m.Description = req.Description
+	}
+	if req.Headers != nil {
+		m.Headers = req.Headers
+	}
+	if req.HeaderRefs != nil {
+		m.HeaderRefs = req.HeaderRefs
+	}
+	if m.HeaderRefs == nil {
+		m.HeaderRefs = map[string]string{}
+	}
+	for header, value := range req.Secrets {
+		if strings.TrimSpace(header) == "" || value == "" {
+			return m, errors.New("a secret header needs a name and a value")
+		}
+		key := "mcp/" + name + "/" + header
+		if err := a.mcp.SetClusterSecret(ctx, key, value); err != nil {
+			return m, err
+		}
+		m.HeaderRefs[header] = "store:" + key
+		delete(m.Headers, header)
+	}
+	if req.Enabled != nil {
+		m.Enabled = *req.Enabled
+	}
+	if req.TimeoutSeconds > 0 {
+		m.Timeout = time.Duration(req.TimeoutSeconds) * time.Second
+	}
+	switch {
+	case req.RatePerMinute > 0:
+		m.RatePerMinute = req.RatePerMinute
+	case req.RatePerMinute < 0:
+		m.RatePerMinute = 0
+	}
+	return m, nil
+}
+
+// testMCPServer asks a server for its tools, with its credentials, as /mcp
+// would.
+func (a *API) testMCPServer(ctx context.Context, m store.MCPServer) MCPTestView {
+	ctx, cancel := context.WithTimeout(ctx, 15*time.Second)
+	defer cancel()
+	c := &mcpclient.Client{URL: m.URL, Header: func(ctx context.Context) (http.Header, error) {
+		h := http.Header{}
+		for k, v := range m.Headers {
+			h.Set(k, v)
+		}
+		for k, ref := range m.HeaderRefs {
+			v, err := secrets.ReadString(ctx, ref)
+			if err != nil {
+				return nil, err
+			}
+			h.Set(k, v)
+		}
+		return h, nil
+	}}
+	if m.URL == AdminMCPAddress {
+		c.URL, c.HTTP = InProcessURL, &http.Client{Transport: InProcess(http.HandlerFunc(a.MCP))}
+	}
+	tools, err := c.Tools(ctx)
+	if err != nil {
+		return MCPTestView{Error: err.Error()}
+	}
+	v := MCPTestView{OK: true}
+	for _, t := range tools {
+		v.Tools = append(v.Tools, m.Name+"."+t.Name)
+	}
+	sort.Strings(v.Tools)
+	return v
+}

internal/control/adminapi/mcp_server.go

@@ -0,0 +1,186 @@
+package adminapi
+
+import (
+	"bytes"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+	"net/http/httptest"
+	"net/url"
+	"strconv"
+)
+
+// The cluster as an MCP server, read only: what `llmgw ctl status`, `nodes`,
+// `models`, `replicas`, `explain-placement` and `logs` show, as tools an
+// agent can be given. It answers at POST /api/v1/mcp, behind the admin
+// token, and in-process to the gateway as the MCP server llmgw:admin, which
+// an administrator registers like any other and grants like any other.
+// Nothing here changes the cluster.
+
+// AdminMCPAddress is the address that registers this server with the
+// gateway: called in-process, with no token and no network.
+const AdminMCPAddress = "llmgw:admin"
+
+// adminMCPAnswerBytes bounds one tool's answer: it becomes a prompt.
+const adminMCPAnswerBytes = 256 << 10
+
+var adminMCPTools = []map[string]any{
+	adminTool("cluster_status", "The whole cluster in one answer: control planes, store, nodes ready, models with the replicas they "+
+		"asked for, GPUs in use, and what is unhealthy or idle.", nil, nil),
+	adminTool("list_nodes", "Every node: its state, GPUs and their use, CPU and memory load, fabrics, agent version and engines.", nil, nil),
+	adminTool("list_models", "Every model in the catalog: its replicas ready against wanted, state, profiles and why one is parked.", nil, nil),
+	adminTool("list_replicas", "Every replica, with its shards: the node and GPUs each is on, its state and how long it has run.",
+		map[string]any{"model": map[string]any{"type": "string", "description": "Only this model's replicas."}}, nil),
+	adminTool("explain_placement", "Why a model is not placed: each node, and what kept the model off it.",
+		map[string]any{"model": map[string]any{"type": "string", "description": "The model's name or an alias."}}, []string{"model"}),
+	adminTool("node_logs", "The last lines of a node's agent log, or of one of its engines.",
+		map[string]any{
+			"node":   map[string]any{"type": "string", "description": "The node's id."},
+			"lines":  map[string]any{"type": "integer", "description": "How many lines, up to 300; 100 when absent."},
+			"source": map[string]any{"type": "string", "description": "agent when absent, or a shard id for that engine's output."},
+		}, []string{"node"}),
+}
+
+func adminTool(name, desc string, props map[string]any, required []string) map[string]any {
+	schema := map[string]any{"type": "object", "properties": map[string]any{}}
+	if props != nil {
+		schema["properties"] = props
+	}
+	if len(required) > 0 {
+		schema["required"] = required
+	}
+	return map[string]any{"name": name, "description": desc, "inputSchema": schema,
+		"annotations": map[string]any{"readOnlyHint": true, "openWorldHint": false}}
+}
+
+// MCP serves the cluster as a read-only MCP server: JSON-RPC over POST, the
+// streamable HTTP transport answered as plain JSON, no session.
+func (a *API) MCP(w http.ResponseWriter, r *http.Request) {
+	var msg struct {
+		ID     json.RawMessage `json:"id"`
+		Method string          `json:"method"`
+		Params json.RawMessage `json:"params"`
+	}
+	if err := json.NewDecoder(io.LimitReader(r.Body, 1<<20)).Decode(&msg); err != nil || msg.Method == "" {
+		writeJSON(w, http.StatusBadRequest, map[string]any{"jsonrpc": "2.0", "id": nil,
+			"error": map[string]any{"code": -32600, "message": "expected one JSON-RPC request object"}})
+		return
+	}
+	if len(msg.ID) == 0 {
+		w.WriteHeader(http.StatusAccepted)
+		return
+	}
+	answer := func(result any) {
+		writeJSON(w, http.StatusOK, map[string]any{"jsonrpc": "2.0", "id": msg.ID, "result": result})
+	}
+	switch msg.Method {
+	case "initialize":
+		var p struct {
+			ProtocolVersion string `json:"protocolVersion"`
+		}
+		_ = json.Unmarshal(msg.Params, &p)
+		version := p.ProtocolVersion
+		if version != "2025-03-26" && version != "2025-06-18" {
+			version = "2025-06-18"
+		}
+		answer(map[string]any{"protocolVersion": version, "capabilities": map[string]any{"tools": map[string]any{}},
+			"serverInfo": map[string]any{"name": "llmgw-admin", "version": "1"},
+			"instructions": "This llmgw cluster, read only: its control planes, nodes, models and replicas, why a model is " +
+				"not placed, and a node's logs. Start with cluster_status."})
+	case "ping":
+		answer(map[string]any{})
+	case "tools/list":
+		answer(map[string]any{"tools": adminMCPTools})
+	case "tools/call":
+		var call struct {
+			Name      string `json:"name"`
+			Arguments struct {
+				Model  string `json:"model"`
+				Node   string `json:"node"`
+				Source string `json:"source"`
+				Lines  int    `json:"lines"`
+			} `json:"arguments"`
+		}
+		_ = json.Unmarshal(msg.Params, &call)
+		answer(a.callAdminTool(r, call.Name, call.Arguments.Model, call.Arguments.Node, call.Arguments.Source, call.Arguments.Lines))
+	default:
+		writeJSON(w, http.StatusOK, map[string]any{"jsonrpc": "2.0", "id": msg.ID,
+			"error": map[string]any{"code": -32601, "message": "method not found: " + msg.Method}})
+	}
+}
+
+// callAdminTool answers a tool from the API's own handler for it, so the
+// tool says what `llmgw ctl` would.
+func (a *API) callAdminTool(r *http.Request, name, model, node, source string, lines int) map[string]any {
+	var h http.HandlerFunc
+	q := url.Values{}
+	path := ""
+	switch name {
+	case "cluster_status":
+		h, path = a.summary, "/api/v1/summary"
+	case "list_nodes":
+		h, path = a.listNodes, "/api/v1/nodes"
+	case "list_models":
+		h, path = a.listModels, "/api/v1/models"
+	case "list_replicas":
+		h, path = a.listReplicas, "/api/v1/replicas"
+		if model != "" {
+			q.Set("model", model)
+		}
+	case "explain_placement":
+		if model == "" {
+			return adminToolText("explain_placement needs the model", true)
+		}
+		h, path = a.explainPlacement, "/api/v1/explain-placement"
+		q.Set("model", model)
+	case "node_logs":
+		if node == "" {
+			return adminToolText("node_logs needs the node", true)
+		}
+		if lines <= 0 {
+			lines = 100
+		}
+		h, path = a.nodeLogs, "/api/v1/nodes/"+url.PathEscape(node)+"/logs"
+		q.Set("lines", strconv.Itoa(min(lines, 300)))
+		if source != "" {
+			q.Set("source", source)
+		}
+	default:
+		return adminToolText("no tool "+name, true)
+	}
+	req := httptest.NewRequestWithContext(r.Context(), http.MethodGet, path+"?"+q.Encode(), nil)
+	if name == "node_logs" {
+		req.SetPathValue("id", node)
+	}
+	rec := httptest.NewRecorder()
+	h(rec, req)
+	body := bytes.TrimSpace(rec.Body.Bytes())
+	if rec.Code >= 300 {
+		return adminToolText(fmt.Sprintf("%s: %s", name, body), true)
+	}
+	if len(body) > adminMCPAnswerBytes {
+		body = append(body[:adminMCPAnswerBytes], []byte("… (cut: the answer is larger than an agent should read)")...)
+	}
+	return adminToolText(string(body), false)
+}
+
+func adminToolText(text string, isError bool) map[string]any {
+	return map[string]any{"content": []any{map[string]any{"type": "text", "text": text}}, "isError": isError}
+}
+
+// InProcessURL is the address an in-process client posts to; the transport
+// ignores it.
+const InProcessURL = "http://llmgw-admin/api/v1/mcp"
+
+// InProcess is a transport that answers every request from h, in this
+// process: the gateway's way to the cluster's own MCP server.
+func InProcess(h http.Handler) http.RoundTripper { return inProcess{h} }
+
+type inProcess struct{ h http.Handler }
+
+func (t inProcess) RoundTrip(req *http.Request) (*http.Response, error) {
+	rec := httptest.NewRecorder()
+	t.h.ServeHTTP(rec, req)
+	return rec.Result(), nil
+}

internal/control/adminapi/mcp_server_test.go

@@ -0,0 +1,77 @@
+package adminapi
+
+import (
+	"bytes"
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+)
+
+func adminMCP(t *testing.T, a *API, method string, params any) map[string]any {
+	t.Helper()
+	b, _ := json.Marshal(map[string]any{"jsonrpc": "2.0", "id": 1, "method": method, "params": params})
+	rec := httptest.NewRecorder()
+	a.MCP(rec, httptest.NewRequest(http.MethodPost, "/api/v1/mcp", bytes.NewReader(b)))
+	var resp struct {
+		Result map[string]any `json:"result"`
+	}
+	if rec.Code != 200 || json.Unmarshal(rec.Body.Bytes(), &resp) != nil {
+		t.Fatalf("%s: %d %s", method, rec.Code, rec.Body.String())
+	}
+	return resp.Result
+}
+
+func TestTheClusterIsAReadOnlyMCPServer(t *testing.T) {
+	a, _ := setup(t, testModels())
+	if init := adminMCP(t, a, "initialize", map[string]any{"protocolVersion": "2025-06-18"}); init["protocolVersion"] != "2025-06-18" {
+		t.Errorf("initialize %v", init)
+	}
+	tools := adminMCP(t, a, "tools/list", map[string]any{})["tools"].([]any)
+	var names []string
+	for _, tl := range tools {
+		m := tl.(map[string]any)
+		names = append(names, m["name"].(string))
+		if m["annotations"].(map[string]any)["readOnlyHint"] != true {
+			t.Errorf("%s is not marked read only", m["name"])
+		}
+	}
+	if got := strings.Join(names, ","); got != "cluster_status,list_nodes,list_models,list_replicas,explain_placement,node_logs" {
+		t.Errorf("tools %s", got)
+	}
+	// Each answers what the API would.
+	for _, name := range []string{"cluster_status", "list_nodes", "list_models", "list_replicas"} {
+		res := adminMCP(t, a, "tools/call", map[string]any{"name": name, "arguments": map[string]any{}})
+		text := res["content"].([]any)[0].(map[string]any)["text"].(string)
+		if res["isError"] == true || !json.Valid([]byte(text)) {
+			t.Errorf("%s: %v", name, res)
+		}
+	}
+	if res := adminMCP(t, a, "tools/call", map[string]any{"name": "explain_placement", "arguments": map[string]any{}}); res["isError"] != true {
+		t.Errorf("explain_placement without a model: %v", res)
+	}
+	if res := adminMCP(t, a, "tools/call", map[string]any{"name": "remove_model", "arguments": map[string]any{}}); res["isError"] != true {
+		t.Errorf("a tool that would change something: %v", res)
+	}
+}
+
+// The gateway reaches it in this process, as llmgw:admin.
+func TestTheClustersOwnMCPServerIsReachedInProcess(t *testing.T) {
+	a, _ := setup(t, testModels())
+	c := &http.Client{Transport: InProcess(http.HandlerFunc(a.MCP))}
+	b, _ := json.Marshal(map[string]any{"jsonrpc": "2.0", "id": 1, "method": "tools/list"})
+	resp, err := c.Post(InProcessURL, "application/json", bytes.NewReader(b))
+	if err != nil || resp.StatusCode != 200 {
+		t.Fatalf("%v %v", resp, err)
+	}
+	defer resp.Body.Close()
+	var out struct {
+		Result struct {
+			Tools []any `json:"tools"`
+		} `json:"result"`
+	}
+	if json.NewDecoder(resp.Body).Decode(&out) != nil || len(out.Result.Tools) != 6 {
+		t.Errorf("tools %+v", out)
+	}
+}

internal/control/adminapi/mcp_test.go

@@ -0,0 +1,107 @@
+package adminapi
+
+import (
+	"context"
+	"strings"
+	"sync"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/mcpclient/mcptest"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/secrets"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+type memMCP struct {
+	mu      sync.Mutex
+	servers map[string]store.MCPServer
+	secrets map[string]string
+}
+
+func (m *memMCP) MCPServers(context.Context) ([]store.MCPServer, error) {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+	var out []store.MCPServer
+	for _, s := range m.servers {
+		out = append(out, s)
+	}
+	return out, nil
+}
+
+func (m *memMCP) MCPServer(_ context.Context, name string) (store.MCPServer, error) {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+	s, ok := m.servers[name]
+	if !ok {
+		return s, store.ErrNotFound
+	}
+	return s, nil
+}
+
+func (m *memMCP) PutMCPServer(_ context.Context, s store.MCPServer) error {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+	m.servers[s.Name] = s
+	return nil
+}
+
+func (m *memMCP) DeleteMCPServer(_ context.Context, name string) error {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+	if _, ok := m.servers[name]; !ok {
+		return store.ErrNotFound
+	}
+	delete(m.servers, name)
+	return nil
+}
+
+func (m *memMCP) SetClusterSecret(_ context.Context, name, value string) error {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+	m.secrets[name] = value
+	return nil
+}
+
+func TestAnMCPServerIsRegisteredWithItsTokenKeptAsASecretAndTried(t *testing.T) {
+	srv := mcptest.New()
+	defer srv.Close()
+	a, _ := setup(t, testModels())
+	st := &memMCP{servers: map[string]store.MCPServer{}, secrets: map[string]string{}}
+	a.SetMCP(st)
+	secrets.SetStore(func(_ context.Context, name string) (string, error) {
+		st.mu.Lock()
+		defer st.mu.Unlock()
+		return st.secrets[name], nil
+	})
+	t.Cleanup(func() { secrets.SetStore(nil) })
+
+	var v MCPServerView
+	body := `{"url":"` + srv.URL + `","description":"tickets","headers":{"X-Org":"acme"},"secrets":{"Authorization":"Bearer t0ken"}}`
+	if code := call(t, a, "PUT", "/api/v1/mcp/servers/jira", body, &v); code != 200 {
+		t.Fatalf("add: %d", code)
+	}
+	// The token is in the store, named by reference; the row has none.
+	if st.secrets["mcp/jira/Authorization"] != "Bearer t0ken" || v.HeaderRefs["Authorization"] != "store:mcp/jira/Authorization" ||
+		strings.Contains(v.Headers["Authorization"], "t0ken") || !v.Enabled {
+		t.Errorf("view %+v, secrets %v", v, st.secrets)
+	}
+	var tv MCPTestView
+	if code := call(t, a, "POST", "/api/v1/mcp/servers/jira/test", "", &tv); code != 200 || !tv.OK || len(tv.Tools) != 3 || tv.Tools[1] != "jira.echo" {
+		t.Errorf("test: %d %+v", code, tv)
+	}
+	if code := call(t, a, "PUT", "/api/v1/mcp/servers/Bad.Name", body, nil); code != 400 {
+		t.Errorf("a name with a dot: %d", code)
+	}
+	if code := call(t, a, "PUT", "/api/v1/mcp/servers/x", `{"url":"ftp://x"}`, nil); code != 400 {
+		t.Errorf("not an http address: %d", code)
+	}
+	// Switched off, kept; then removed.
+	if code := call(t, a, "PUT", "/api/v1/mcp/servers/jira", `{"enabled":false}`, &v); code != 200 || v.Enabled || v.URL != srv.URL {
+		t.Errorf("off: %d %+v", code, v)
+	}
+	if code := call(t, a, "DELETE", "/api/v1/mcp/servers/jira", "", nil); code != 200 {
+		t.Errorf("remove: %d", code)
+	}
+	if code := call(t, a, "DELETE", "/api/v1/mcp/servers/jira", "", nil); code != 404 {
+		t.Errorf("remove again: %d", code)
+	}
+}

internal/control/adminapi/routes.go

@@ -67,6 +67,8 @@ var PathParams = map[string]string{
 	"models/{name}":     "The model's name in the catalog.",
 	"presets/{name}":    "The preset's name, as `llmgw ctl model presets` lists it.",
 	"nodes/{id}":        "The node's ID, as `llmgw ctl nodes` lists it.",
+	"secrets/{name}":    "ldap-bind, oidc-client-secret or scim-token.",
+	"servers/{name}":    "The MCP server's name: its tools are NAME.tool in /mcp.",
 	"jobs/{id}":         "The job's ID, as starting it answered.",
 	"runs/{id}":         "The quality run's ID, as starting it answered.",
 	"requests/{id}":     "The request's ID: the x-request-id a client was answered with.",
@@ -280,6 +282,8 @@ var Routes = []Route{
 		Response: []JudgeView{}},
 	{Method: "POST", Path: "/api/v1/guard/judges/ask", Summary: "Have a judge read a text and say what it answered; nothing is kept",
 		Request: JudgeAskRequest{}, Response: JudgeAskView{}},
+	{Method: "POST", Path: "/api/v1/guard/judges/check", Summary: "How a judge keeps up: three judgments in turn, then a burst at once, against the budget",
+		Request: JudgeCheckRequest{}, Response: JudgeCheckView{}},
 	{Method: "POST", Path: "/api/v1/guard/attachments", Summary: "Attach a guardrail policy",
 		Request: GuardAttachmentRequest{}, Response: guard.Attachment{}},
 	{Method: "DELETE", Path: "/api/v1/guard/attachments", Summary: "Detach a guardrail policy",
@@ -396,6 +400,28 @@ var Routes = []Route{
 	{Method: "POST", Path: "/api/v1/settings/rollback", Summary: "Put an earlier version of the cluster's settings back in force",
 		Request: SettingsRollbackRequest{}, Response: SettingsView{}},
 
+	// How people sign in: single sign-on, a directory, and the secrets
+	// they need kept for every control plane.
+	{Method: "GET", Path: "/api/v1/auth", Summary: "How people sign in: OIDC, a directory, SCIM, and the directory's last sync",
+		Response: AuthView{}},
+	{Method: "POST", Path: "/api/v1/auth/ldap/test", Summary: "Try an auth.ldap block before it is kept: servers, the service account, people and groups",
+		Request: LDAPTestRequest{}, Response: LDAPTestView{}},
+	{Method: "PUT", Path: "/api/v1/auth/secrets/{name}", Summary: "Keep ldap-bind, oidc-client-secret or scim-token in the store, for settings to name as store:auth/NAME",
+		Request: SecretRequest{}, Response: map[string]string{}},
+
+	// The company's own MCP servers, served through /mcp.
+	{Method: "POST", Path: "/api/v1/mcp", Summary: "This cluster as a read-only MCP server: status, nodes, models, replicas, placement, logs",
+		Description: "JSON-RPC: initialize, tools/list, tools/call. Registered with the gateway as llmgw:admin.",
+		Request:     map[string]any{}, Response: map[string]any{}},
+	{Method: "GET", Path: "/api/v1/mcp/servers", Summary: "The company's MCP servers /mcp serves, and how each is reached",
+		Response: []MCPServerView{}},
+	{Method: "PUT", Path: "/api/v1/mcp/servers/{name}", Summary: "Add an MCP server, or change one: its address, headers and the secrets they carry",
+		Request: MCPServerRequest{}, Response: MCPServerView{}},
+	{Method: "DELETE", Path: "/api/v1/mcp/servers/{name}", Summary: "Remove an MCP server: its tools leave /mcp",
+		Response: map[string]string{}},
+	{Method: "POST", Path: "/api/v1/mcp/servers/{name}/test", Summary: "Ask an MCP server for its tools, with its credentials, as /mcp would",
+		Response: MCPTestView{}},
+
 	// Usage, requests and the record of changes.
 	{Method: "GET", Path: "/api/v1/usage", Summary: "What each group of the tenancy tree spent",
 		Query: []Param{

internal/control/catalog/catalog.go

@@ -6,6 +6,7 @@ import (
 	"crypto/sha256"
 	"encoding/hex"
 	"encoding/json"
+	"errors"
 	"fmt"
 	"maps"
 	"net/url"
@@ -23,6 +24,9 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
 )
 
+// ErrExists is a model added only if absent that was there already.
+var ErrExists = errors.New("already in the catalog")
+
 // Scaling is how many replicas a model should have.
 type Scaling struct {
 	Min int `json:"min,omitempty" koanf:"min"`
@@ -1533,6 +1537,11 @@ type External struct {
 	// rather than the key itself, so the key is in neither the catalog, the
 	// store nor the audit log; empty for a provider that asks for none.
 	APIKeyFile string `json:"api_key_file,omitempty" koanf:"api_key_file"`
+	// APIKeyFiles are several keys for the provider, in place of
+	// APIKeyFile: one endpoint each (external:MODEL#1, #2…), so a key that
+	// answers 429 or 401 is passed over while its breaker is open and the
+	// request goes to another. A provider's limits are per key.
+	APIKeyFiles []ExternalKey `json:"api_key_files,omitempty" koanf:"api_key_files"`
 	// Peer names the federated llmgw cluster this model is offered by, for
 	// a model that federation added (`llmgw ctl federation add`) rather
 	// than one written by hand. Such a model is named PEER/MODEL, comes and
@@ -1551,6 +1560,14 @@ type External struct {
 	Output string `json:"output,omitempty" koanf:"output"`
 }
 
+// ExternalKey is one of a provider's keys.
+type ExternalKey struct {
+	// File holds the key, on the control plane, as APIKeyFile does.
+	File string `json:"file" koanf:"file"`
+	// Weight is its share of the requests against the others'; zero is 1.
+	Weight float64 `json:"weight,omitempty" koanf:"weight"`
+}
+
 // APIComfyUI is External.API for a ComfyUI server.
 const APIComfyUI = "comfyui"
 
@@ -1568,6 +1585,20 @@ func (e *External) validate(model string, profiles int) error {
 	if strings.TrimSpace(e.Model) == "" {
 		return fmt.Errorf("model %q: external model needs the name the provider knows it by", model)
 	}
+	if e.APIKeyFile != "" && len(e.APIKeyFiles) > 0 {
+		return fmt.Errorf("model %q: api_key_file or api_key_files, not both", model)
+	}
+	for i, k := range e.APIKeyFiles {
+		if strings.TrimSpace(k.File) == "" {
+			return fmt.Errorf("model %q: api_key_files[%d] names no file", model, i)
+		}
+		if k.Weight < 0 {
+			return fmt.Errorf("model %q: api_key_files[%d] has a negative weight", model, i)
+		}
+	}
+	if len(e.APIKeyFiles) > 0 && e.Peer != "" {
+		return fmt.Errorf("model %q: a federated model is reached with its peer's key, not api_key_files", model)
+	}
 	switch e.API {
 	case "", "openai":
 		if e.Workflow != "" || e.Output != "" {

internal/control/catalog/catalog_test.go

@@ -429,3 +429,23 @@ func TestAGuardModelSaysHowItIsReadAndAlwaysRuns(t *testing.T) {
 		t.Error("a judge is not critical by default")
 	}
 }
+
+func TestAProvidersKeysAreOneListOrOneFile(t *testing.T) {
+	ext := func(e External) Model {
+		e.URL, e.Model = "https://api.example.com", "m"
+		return Model{Name: "cloud", Source: "external", External: &e}
+	}
+	for _, bad := range []External{
+		{APIKeyFile: "/k", APIKeyFiles: []ExternalKey{{File: "/k2"}}},
+		{APIKeyFiles: []ExternalKey{{File: " "}}},
+		{APIKeyFiles: []ExternalKey{{File: "/k", Weight: -1}}},
+		{Peer: "p", APIKeyFiles: []ExternalKey{{File: "/k"}}},
+	} {
+		if _, err := New([]Model{ext(bad)}); err == nil {
+			t.Errorf("taken: %+v", bad)
+		}
+	}
+	if _, err := New([]Model{ext(External{APIKeyFiles: []ExternalKey{{File: "/a", Weight: 2}, {File: "/b"}}})}); err != nil {
+		t.Error(err)
+	}
+}

internal/control/control.go

@@ -438,15 +438,6 @@ func (c *Controller) externalEndpoints() []endpoint.Info {
 				continue
 			}
 		}
-		key := ""
-		if f := m.External.APIKeyFile; f != "" {
-			b, err := secrets.Read(context.Background(), f)
-			if err != nil {
-				c.log.Warn("an external model's key could not be read; not offered", "model", m.Name, "err", err)
-				continue
-			}
-			key = strings.TrimSpace(string(b))
-		}
 		// A peer's endpoint says so in its id, which is what usage records
 		// as the replica that answered.
 		id := "external:" + m.Name
@@ -462,14 +453,39 @@ func (c *Controller) externalEndpoints() []endpoint.Info {
 			}
 			engine, workflow = endpoint.EngineComfyUI, w
 		}
-		out = append(out, endpoint.Info{
+		base := endpoint.Info{
 			ID: id, URL: strings.TrimRight(m.External.URL, "/"),
-			Models: append([]string{m.Name}, m.Aliases...), APIKey: key,
+			Models: append([]string{m.Name}, m.Aliases...),
 			Engine: engine, Pool: "external", Weight: 1,
 			UpstreamModel: m.External.Model,
 			Peer:          m.External.Peer, PeerCluster: cluster, PeerTags: tags,
 			Workflow: workflow,
-		})
+		}
+		keys := m.External.APIKeyFiles
+		if len(keys) == 0 {
+			keys = []catalog.ExternalKey{{File: m.External.APIKeyFile}}
+		}
+		for i, k := range keys {
+			e := base
+			// One endpoint a key, numbered, so usage says which answered
+			// and a breaker stands for one key — never the key itself.
+			if len(m.External.APIKeyFiles) > 0 {
+				e.ID = fmt.Sprintf("%s#%d", id, i+1)
+				if k.Weight > 0 {
+					e.Weight = k.Weight
+				}
+			}
+			if k.File != "" {
+				b, err := secrets.Read(context.Background(), k.File)
+				if err != nil {
+					c.log.Warn("an external model's key could not be read; not offered", "model", m.Name,
+						"endpoint", e.ID, "err", err)
+					continue
+				}
+				e.APIKey = strings.TrimSpace(string(b))
+			}
+			out = append(out, e)
+		}
 	}
 	return out
 }

internal/control/external_test.go

@@ -73,3 +73,28 @@ func TestAnExternalModelIsPublishedWithItsKeyAndNothingIsPlacedForIt(t *testing.
 		}
 	}
 }
+
+// Several keys for one provider: an endpoint each, numbered, with its
+// weight; one whose file cannot be read is left out and the others stay.
+func TestEachOfAProvidersKeysIsAnEndpointOfItsOwn(t *testing.T) {
+	dir := t.TempDir()
+	a, c := filepath.Join(dir, "a.key"), filepath.Join(dir, "c.key")
+	_ = os.WriteFile(a, []byte("sk-a\n"), 0o600)
+	_ = os.WriteFile(c, []byte("sk-c"), 0o600)
+	cat, err := catalog.New([]catalog.Model{{Name: "cloud", Source: "external",
+		External: &catalog.External{URL: "https://api.example.com", Model: "gpt-4o-mini",
+			APIKeyFiles: []catalog.ExternalKey{{File: a, Weight: 3}, {File: filepath.Join(dir, "gone")}, {File: c}}}}})
+	if err != nil {
+		t.Fatal(err)
+	}
+	sink := &recordSink{}
+	New(DefaultConfig(), catalog.NewLive(cat), noDispatch{}, sink, slog.New(slog.DiscardHandler)).Publish(t.Context())
+	got := map[string]endpoint.Info{}
+	for _, e := range sink.infos {
+		got[e.ID] = e
+	}
+	if len(got) != 2 || got["external:cloud#1"].APIKey != "sk-a" || got["external:cloud#1"].Weight != 3 ||
+		got["external:cloud#3"].APIKey != "sk-c" || got["external:cloud#3"].Weight != 1 {
+		t.Errorf("published %+v", sink.infos)
+	}
+}

internal/control/modelserve/modelserve.go

@@ -163,6 +163,18 @@ type Request struct {
 	// CPUCores is what a replica computes on; zero takes half the smallest
 	// node it fits on.
 	CPUCores int `json:"cpu_cores,omitempty"`
+	// Env is set in the engine's environment, on every node of a replica:
+	// NCCL_DEBUG=INFO to see how its cards talk, NCCL_IB_* to steer them.
+	// Only the variables engines are tuned with; the catalog refuses others.
+	Env map[string]string `json:"env,omitempty"`
+	// IfAbsent adds the model only if nobody has meanwhile: the leader's own
+	// serving of a judge a policy names gives way to the one init or an
+	// operator added while it fetched, rather than replacing it.
+	IfAbsent bool `json:"-"`
+	// MemoryGB is what a replica on CPUs reserves, in place of the figure
+	// worked out from its weights and context: a judge's, whose own sizing
+	// is known.
+	MemoryGB int `json:"memory_gb,omitempty"`
 	// By is who asked, for the catalog's record.
 	By string `json:"-"`
 }
@@ -453,8 +465,17 @@ func (s *Server) run(j *Job) {
 	for _, o := range layout {
 		where = append(where, o.Label())
 	}
-	s.step(j, "registry", Done, fmt.Sprintf("%s, %.1f GB in %d file(s), revision %s; runs on %s",
-		set.Name, gb(set.Bytes), len(set.Files), repo.ShortRevision(), strings.Join(where, ", else ")))
+	// What the first way needs of each card, for the context asked for.
+	need := ""
+	if first := layout[0]; first.NeedGB > 0 {
+		tokens := req.ContextLength
+		if req.Format != "safetensors" && req.CtxPool > 0 {
+			tokens = req.CtxPool
+		}
+		need = "; needs " + first.WithContext(plan.KVBytesPerToken, tokens).MemoryText()
+	}
+	s.step(j, "registry", Done, fmt.Sprintf("%s, %.1f GB in %d file(s), revision %s; runs on %s%s",
+		set.Name, gb(set.Bytes), len(set.Files), repo.ShortRevision(), strings.Join(where, ", else "), need))
 
 	// 2. The mirror: fetch, and check what arrived against what was published.
 	source, err := s.mirror(ctx, j, repo, set, name)
@@ -504,7 +525,11 @@ func (s *Server) run(j *Job) {
 			req = fitted
 		}
 	}
+	// Under the lock: a job is read by Get while it runs, and the race
+	// detector caught this write beside it.
+	s.mu.Lock()
 	j.Request = req
+	s.mu.Unlock()
 	memGB := int((header.MemoryBytes(set.Bytes, req.CtxPool) + 1<<30 - 1) >> 30)
 	basis := fmt.Sprintf("weights %.1f GB + KV cache %.1f GB for %d tokens + engine",
 		gb(set.Bytes), gb(int64(req.CtxPool)*header.KVBytesPerToken()), req.CtxPool)
@@ -514,6 +539,9 @@ func (s *Server) run(j *Job) {
 		memGB = int(gb(set.Bytes)*1.5) + 2
 		basis = "weights plus half again plus two; the header gave no shape"
 	}
+	if req.MemoryGB > 0 {
+		memGB, basis = req.MemoryGB, "as asked, in place of "+basis
+	}
 	s.step(j, "check", Done, fmt.Sprintf("%s, needs about %d GB (%s)", orUnknown(header.Architecture), memGB, basis))
 
 	cores := req.CPUCores
@@ -556,6 +584,8 @@ func chooseLayout(req Request, plan Plan, planned bool, set hf.FileSet) ([]model
 				o = p
 			}
 		}
+		// A way the plan did not offer is still a model of experts.
+		o.ExpertParallel = engine == "vllm" && plan.Experts > 1 && o.GPUs() > 1
 		out = append(out, o)
 	case len(opts) > 0:
 		out = append(out, opts[0])
@@ -590,7 +620,7 @@ func chooseLayout(req Request, plan Plan, planned bool, set hf.FileSet) ([]model
 func profilesFor(layout []modelplan.Option, req Request, memGB, cores int) []catalog.Profile {
 	var out []catalog.Profile
 	for i, o := range layout {
-		p := catalog.Profile{Engine: o.Engine, Weight: 1, Priority: 100 - 10*i}
+		p := catalog.Profile{Engine: o.Engine, Weight: 1, Priority: 100 - 10*i, Env: req.Env}
 		switch {
 		case o.CPU:
 			p.Name = "cpu"
@@ -607,6 +637,11 @@ func profilesFor(layout []modelplan.Option, req Request, memGB, cores int) []cat
 				p.PP, p.Nodes, p.GPUsPerNode = o.Nodes, o.Nodes, o.GPUsPerNode
 			}
 			p.Args = []string{"--max-model-len", strconv.Itoa(req.ContextLength)}
+			if o.ExpertParallel {
+				// The experts whole on each card rather than each sliced
+				// across them: less traffic between the cards for every token.
+				p.Args = append(p.Args, "--enable-expert-parallel")
+			}
 		}
 		out = append(out, p)
 	}
@@ -627,6 +662,12 @@ func llamaArgs(req Request) []string {
 		n := strconv.Itoa(req.ContextLength)
 		args = append([]string{"--embedding"}, append(args, "--batch-size", n, "--ubatch-size", n)...)
 	}
+	if req.GuardFormat != "" {
+		// No prompt cache: a judge's prompts, each a different text, are
+		// not asked again, and llama.cpp's default of up to 8 GB grew one
+		// judge to 3.2 GB over a day.
+		args = append(args, "--cache-ram", "0")
+	}
 	return args
 }
 
@@ -674,7 +715,18 @@ func (s *Server) addAndStart(ctx context.Context, j *Job, repo hf.Repo, set hf.F
 	}
 	m.Class = req.Class
 	s.step(j, "catalog", Running, "adding "+name)
-	if err := s.deps.Catalog.Add(ctx, m, req.By); err != nil {
+	add := s.deps.Catalog.Add
+	if adder, ok := s.deps.Catalog.(interface {
+		AddNew(ctx context.Context, m catalog.Model, by string) error
+	}); ok && req.IfAbsent {
+		add = adder.AddNew
+	}
+	if err := add(ctx, m, req.By); err != nil {
+		if req.IfAbsent && errors.Is(err, catalog.ErrExists) {
+			s.step(j, "catalog", Skipped, name+" was added meanwhile; left as it is")
+			s.finish(j)
+			return
+		}
 		s.fail(j, "catalog", err)
 		return
 	}

internal/control/modelserve/modelserve_test.go

@@ -428,3 +428,73 @@ func TestAPoolTheCardCannotHoldIsShrunkNotStarted(t *testing.T) {
 		t.Errorf("a pool that fits was changed: %+v %q", got, why)
 	}
 }
+
+// A judge is sized as asked, not from its weights, and runs with no prompt
+// cache: what `server init` gives its own.
+func TestAJudgeIsServedWithTheMemoryAskedAndNoPromptCache(t *testing.T) {
+	files := []hf.File{{Path: "Qwen3Guard-Gen-0.6B-Q8_0.gguf", Bytes: 600 << 20, SHA256: "aa"}}
+	s, _, c, _ := setup(t, files, nil)
+	job, err := s.Start(Request{Repo: "example/Qwen3Guard-Gen-0.6B-GGUF", File: "Qwen3Guard-Gen-0.6B-Q8_0.gguf",
+		GuardFormat: "qwen3guard", CPU: true, CPUCores: 2, MemoryGB: 2, Replicas: 2, CtxPool: 8192, ContextLength: 4096})
+	if err != nil {
+		t.Fatal(err)
+	}
+	wait(t, s, job.ID)
+	if len(c.added) != 1 {
+		t.Fatalf("added %+v", c.added)
+	}
+	p := c.added[0].Profiles[0]
+	if p.MinMemoryGB != 2 || p.CPUCores != 2 || !strings.Contains(strings.Join(p.Args, " "), "--cache-ram 0") {
+		t.Errorf("profile %+v", p)
+	}
+	// Any other model keeps llama.cpp's cache.
+	if strings.Contains(strings.Join(llamaArgs(Request{CtxPool: 8192, ContextLength: 4096}), " "), "cache-ram") {
+		t.Error("a model that is not a judge lost its prompt cache")
+	}
+}
+
+// takenCatalog has the model by the time a job comes to add it.
+type takenCatalog struct{ fakeCatalog }
+
+func (c *takenCatalog) AddNew(context.Context, catalog.Model, string) error {
+	return fmt.Errorf("model %q: %w", "x", catalog.ErrExists)
+}
+
+func TestAJobAddingOnlyIfAbsentGivesWayToAModelAddedMeanwhile(t *testing.T) {
+	files := []hf.File{{Path: "Qwen3Guard-Gen-0.6B-Q8_0.gguf", Bytes: 600 << 20, SHA256: "aa"}}
+	s, _, _, _ := setup(t, files, nil)
+	taken := &takenCatalog{}
+	s.deps.Catalog = taken
+	job, err := s.Start(Request{Repo: "example/Qwen3Guard-Gen-0.6B-GGUF", File: "Qwen3Guard-Gen-0.6B-Q8_0.gguf",
+		GuardFormat: "qwen3guard", CPU: true, CPUCores: 2, MemoryGB: 2, Replicas: 2, IfAbsent: true})
+	if err != nil {
+		t.Fatal(err)
+	}
+	j := wait(t, s, job.ID)
+	if j.State != Done || len(taken.added) != 0 {
+		t.Errorf("job %s, added %d: %+v", j.State, len(taken.added), j.Steps)
+	}
+}
+
+// A model of experts given its cards by hand spreads them still, and the
+// environment asked for reaches every profile.
+func TestAHandGivenLayoutOfExpertsKeepsExpertParallelAndTheEnvironment(t *testing.T) {
+	plan := Plan{Repo: "Qwen/Qwen3-235B-A22B", Experts: 128}
+	req := Request{Format: "safetensors", GPUClass: "h100-80g", GPUsPerNode: 8, Nodes: 2, ContextLength: 8192,
+		Env: map[string]string{"NCCL_DEBUG": "INFO"}}
+	layout, err := chooseLayout(req, plan, true, hf.FileSet{Name: "original"})
+	if err != nil {
+		t.Fatal(err)
+	}
+	profs := profilesFor(layout, req, 0, 0)
+	p := profs[0]
+	if p.TP != 8 || p.PP != 2 || p.Nodes != 2 || !slices.Contains(p.Args, "--enable-expert-parallel") || p.Env["NCCL_DEBUG"] != "INFO" {
+		t.Errorf("profile %+v", p)
+	}
+	// A dense model given the same cards does not.
+	plan.Experts = 0
+	layout, _ = chooseLayout(req, plan, true, hf.FileSet{Name: "original"})
+	if slices.Contains(profilesFor(layout, req, 0, 0)[0].Args, "--enable-expert-parallel") {
+		t.Error("a dense model was given expert parallelism")
+	}
+}

internal/control/modelserve/plan.go

@@ -42,7 +42,12 @@ type Plan struct {
 	License  string `json:"license,omitempty"`
 	Params   int64  `json:"params,omitempty"`
 	Context  int    `json:"context"`
-	Gated    bool   `json:"gated,omitempty"`
+	// KVBytesPerToken is one token of context, the whole model: what an
+	// option needs for another context is worked out from it. 0 is unknown.
+	KVBytesPerToken int64 `json:"kv_bytes_per_token,omitempty"`
+	// Experts is a mixture-of-experts model's routed experts; 0 for dense.
+	Experts int  `json:"experts,omitempty"`
+	Gated   bool `json:"gated,omitempty"`
 	// Servable is false when the nodes' engine cannot run the model at all,
 	// and Reason says why.
 	Servable bool   `json:"servable"`
@@ -144,6 +149,7 @@ func (s *Server) plan(ctx context.Context, repo hf.Repo, req PlanRequest) (Plan,
 		if err == nil {
 			shape.KVBytesPerToken = m.KVBytesPerToken(modelplan.BF16)
 			shape.KVHeads = m.KVHeads
+			shape.Experts = m.Experts
 			if p.Params == 0 {
 				p.Params = m.Params
 			}
@@ -157,6 +163,7 @@ func (s *Server) plan(ctx context.Context, repo hf.Repo, req PlanRequest) (Plan,
 		p.Note = "the model's shape was not readable, so the context's memory is not counted"
 	}
 	shape.Params = p.Params
+	p.KVBytesPerToken, p.Experts = shape.KVBytesPerToken, shape.Experts
 	p.Options = modelplan.Options(arts, shape, s.deps.Pool(), p.Context)
 	if !p.Servable {
 		p.Options = nil

internal/control/modelsvc/modelsvc.go

@@ -21,6 +21,7 @@ import (
 	"errors"
 	"fmt"
 	"log/slog"
+	"slices"
 	"sync"
 	"time"
 
@@ -105,6 +106,17 @@ func (s *Service) Sources(ctx context.Context) (map[string][]catalog.Model, erro
 // than persisted and then skipped on every future start — which would be a
 // model that looks added and never appears.
 func (s *Service) Add(ctx context.Context, m catalog.Model, by string) error {
+	return s.add(ctx, m, by, false)
+}
+
+// AddNew is Add for a model nobody has added meanwhile: one that is in the
+// catalog by now is left as it is, and catalog.ErrExists said. The leader serving a
+// judge a policy names gives way so to the judge init added while it fetched.
+func (s *Service) AddNew(ctx context.Context, m catalog.Model, by string) error {
+	return s.add(ctx, m, by, true)
+}
+
+func (s *Service) add(ctx context.Context, m catalog.Model, by string, onlyNew bool) error {
 	s.mu.Lock()
 	defer s.mu.Unlock()
 
@@ -134,6 +146,9 @@ func (s *Service) Add(ctx context.Context, m catalog.Model, by string) error {
 	next := s.base(stored)
 	for _, e := range stored {
 		if e.Name == m.Name {
+			if onlyNew {
+				return fmt.Errorf("model %q: %w", m.Name, catalog.ErrExists)
+			}
 			continue // this call replaces it
 		}
 		next = append(next, e)
@@ -248,6 +263,19 @@ func (s *Service) Resources(ctx context.Context, name, profile string, memoryGB,
 	return final, err
 }
 
+// SetProfiles replaces a model's profiles, as a model added with them
+// would have: for a model whose engines are to run otherwise, its args among
+// it. Running replicas keep what they were started with until restarted.
+func (s *Service) SetProfiles(ctx context.Context, name string, profiles []catalog.Profile, by string) (catalog.Model, error) {
+	if len(profiles) == 0 {
+		return catalog.Model{}, errors.New("a model needs a profile")
+	}
+	return s.update(ctx, name, by, "replace its profiles", func(m *catalog.Model) error {
+		m.Profiles = slices.Clone(profiles)
+		return nil
+	})
+}
+
 // SetClass sets a model's capacity class: critical, standard or
 // opportunistic, or empty to go back to what its scaling implies.
 func (s *Service) SetClass(ctx context.Context, name, class, by string) (catalog.Model, error) {

internal/control/modelsvc/modelsvc_test.go

@@ -3,6 +3,7 @@ package modelsvc
 import (
 	"context"
 	"encoding/json"
+	"errors"
 	"log/slog"
 	"strings"
 	"testing"
@@ -377,3 +378,25 @@ func TestResourcesAreSetOnAStoredModelAndSurviveARestart(t *testing.T) {
 		t.Error("the cores were lost on reload")
 	}
 }
+
+// The leader serving a judge gives way to the one init added while it
+// fetched: AddNew leaves a model already there as it is.
+func TestAddNewLeavesAModelAddedMeanwhile(t *testing.T) {
+	s, _, rec := newService(t)
+	first := model("judge")
+	first.Description = "init's"
+	if err := s.Add(t.Context(), first, "init"); err != nil {
+		t.Fatal(err)
+	}
+	second := model("judge")
+	second.Description = "the leader's"
+	if err := s.AddNew(t.Context(), second, "llmgw"); !errors.Is(err, catalog.ErrExists) {
+		t.Fatalf("err %v", err)
+	}
+	if m, _ := rec.last.Resolve("judge"); m.Description != "init's" {
+		t.Errorf("replaced: %q", m.Description)
+	}
+	if err := s.AddNew(t.Context(), model("fresh"), "llmgw"); err != nil {
+		t.Errorf("a new model: %v", err)
+	}
+}

internal/control/placement/placement.go

@@ -548,6 +548,10 @@ func hasWeights(source string, n inventory.Node) (present, known bool) {
 // It returns the label rather than a bare false so the rejection can name it:
 // "does not carry ram=32g" is a placement failure someone can act on, and
 // "no candidates were considered" is not.
+// RoleLabel gives a node a role, such as judge: it then takes only profiles
+// whose node selector asks for that role.
+const RoleLabel = "llmgw.role"
+
 func unmatchedLabel(selector, labels map[string]string) (string, bool) {
 	for k, want := range selector {
 		if got, ok := labels[k]; !ok || got != want {
@@ -602,6 +606,13 @@ func (s *Scheduler) eligibleNodes(p Profile, fabric string, avoid []string, reje
 			reject("does not carry " + miss)
 			continue
 		}
+		// A node given a role is kept for what asks for it: the judges'
+		// machine runs the judges alone, so nothing beside them takes the
+		// cores a judgment waits on.
+		if role, ok := n.Labels[RoleLabel]; ok && p.NodeSelector[RoleLabel] != role {
+			reject("kept for " + RoleLabel + "=" + role)
+			continue
+		}
 		// The engine has to be installed here. The field has been on this
 		// struct since placement was written and was never read, so a node
 		// with no vLLM was as good a candidate as one with it: the replica was

internal/control/placement/placement_test.go

@@ -1099,3 +1099,19 @@ func TestAProfileCanAskForAnEngineVersion(t *testing.T) {
 		t.Errorf("the rejection should say what the node has:\n%s", res.Explain())
 	}
 }
+
+func TestANodeGivenARoleTakesOnlyWhatAsksForIt(t *testing.T) {
+	judge := node("judge", "", 0)
+	judge.Labels = map[string]string{RoleLabel: "judge"}
+	s := New(store(judge))
+	// A model that does not ask for the judges' machine is not put there.
+	res := s.Place([]Profile{{Name: "cpu", Engine: "llamacpp"}}, nil)
+	if res.Placement != nil || !strings.Contains(res.Explain(), "kept for llmgw.role=judge") {
+		t.Fatalf("an ordinary model went on the judges' machine:\n%s", res.Explain())
+	}
+	// The judge, which asks for it, is.
+	res = s.Place([]Profile{{Name: "cpu", Engine: "llamacpp", NodeSelector: map[string]string{RoleLabel: "judge"}}}, nil)
+	if res.Placement == nil || res.Placement.Nodes[0].NodeID != "judge" {
+		t.Fatalf("the judge was not placed on its machine:\n%s", res.Explain())
+	}
+}

internal/control/reconciler/reconciler.go

@@ -582,7 +582,11 @@ func (r *Reconciler) expireStuckReplicas() {
 		if rep.State != Starting && rep.State != Placing {
 			continue
 		}
-		if now.Sub(rep.CreatedAt) < r.startupTimeout(rep) {
+		since := rep.CreatedAt
+		if rep.fetchedAt.After(since) {
+			since = rep.fetchedAt
+		}
+		if now.Sub(since) < r.startupTimeout(rep) {
 			continue
 		}
 		rep.State = Failed
@@ -915,7 +919,20 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
 	// A drained replica is stopped once its replacement is serving, or once the
 	// grace period expires. Stopping it the moment it was drained would cut
 	// in-flight requests, which is the thing draining exists to avoid.
-	replaced := len(liveIDs) >= desired
+	//
+	// Serving counts a replica being restarted that still answers: with
+	// replicas restarted one after another, the second's retiring kept the
+	// first's drained one waiting out its grace — and with it the second's
+	// replacement, on a node with no room until the first had gone.
+	serving := len(liveIDs)
+	r.mu.Lock()
+	for _, id := range retiringIDs {
+		if rep, ok := r.replicas[id]; ok && rep.State == Ready {
+			serving++
+		}
+	}
+	r.mu.Unlock()
+	replaced := serving >= desired
 	for _, id := range drainingIDs {
 		r.mu.Lock()
 		rep, ok := r.replicas[id]
@@ -1315,6 +1332,9 @@ func (r *Reconciler) OnFetchProgress(replicaID, shardID, source string, done, to
 		if rep.Shards[i].ID != shardID {
 			continue
 		}
+		if prev := rep.Shards[i].Fetch; prev == nil || done > prev.BytesDone {
+			rep.fetchedAt = r.now()
+		}
 		rep.Shards[i].Fetch = &FetchProgress{
 			Source: source, BytesDone: done, BytesTotal: total,
 		}

internal/control/reconciler/reconciler_test.go

@@ -2436,3 +2436,51 @@ func TestTheRoomAnInPlaceRestartFreesIsItsReplacements(t *testing.T) {
 		}
 	}
 }
+
+// Two models restarted together on a node whose cores are all reserved,
+// one to need more and one less: the judge fits only once the starter
+// has given back its cores. The judge's restart waits for that, rather
+// than being refused as if nothing would ever make room. Met on a 4-core
+// single-machine install after `ctl model resources` on both.
+func TestARestartWaitsForRoomAnotherRestartWillGiveBack(t *testing.T) {
+	cpu := inventory.Node{ID: "a", MgmtAddr: "10.0.0.1", State: inventory.Ready, MemoryTotalMB: 16 * 1024, CPUCount: 4}
+	judge := catalog.Model{Name: "judge", Source: "/m", Scaling: catalog.Scaling{Min: 2, Max: 2},
+		Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 2, CPUCores: 1}}}
+	starter := catalog.Model{Name: "starter", Source: "/m", Scaling: catalog.Scaling{Min: 1, Max: 1},
+		Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 2, CPUCores: 2}}}
+	h := newFleetHarness(t, []catalog.Model{judge, starter}, cpu)
+	h.settle(t, 3)
+	var old []string
+	for _, rep := range h.r.ReadyReplicas() {
+		old = append(old, rep.ID)
+	}
+	if len(old) != 3 {
+		t.Fatalf("%d replicas ready, want 3", len(old))
+	}
+
+	judge.Profiles[0].CPUCores, starter.Profiles[0].CPUCores = 2, 0
+	cat, err := catalog.New([]catalog.Model{judge, starter})
+	if err != nil {
+		t.Fatal(err)
+	}
+	h.r.cat.Store(cat)
+	if err := h.r.Restart(old...); err != nil {
+		t.Fatal(err)
+	}
+	for range 12 {
+		h.r.Reconcile(t.Context())
+		h.markReady()
+		h.advance(10 * time.Second)
+	}
+	for _, id := range old {
+		if st := h.r.RestartState(id); strings.HasPrefix(st, "refused") {
+			t.Errorf("%s: %s", id, st)
+		}
+		if rep, ok := replicaByID(h, id); ok && rep.State == Ready {
+			t.Errorf("%s still serves on its old figures", id)
+		}
+	}
+	if j, s := replicasOf(h, "judge"), replicasOf(h, "starter"); j != 2 || s != 1 {
+		t.Errorf("judge %d, starter %d after the restarts", j, s)
+	}
+}

internal/control/reconciler/replica.go

@@ -160,6 +160,11 @@ type Replica struct {
 	AdoptedAt time.Time
 
 	CreatedAt time.Time
+	// fetchedAt is when a shard's download last moved: the startup timeout
+	// runs from then, so a model of hundreds of gigabytes on a slow mirror
+	// is not failed while it is still arriving, and one whose download
+	// stalls is failed a timeout after it stopped.
+	fetchedAt time.Time
 	ReadyAt   time.Time
 	// LastUsedAt is when a request was last routed to this replica, zero if
 	// none has been since the router started.

internal/control/reconciler/restart.go

@@ -22,6 +22,10 @@ import (
 //     stopped now, and its replacement takes its place. Waiting would change
 //     nothing; it used to wait DrainGrace anyway, five minutes reading only
 //     "restart replacing", which looked like a restart that never would.
+//   - other replicas whose own restarts are under way: the replacement fits
+//     once they have restarted, as a starter edited to reserve fewer cores
+//     gives back what a judge edited to reserve more needs. Waited for, up
+//     to DrainGrace, so that two restarts each waiting on the other end.
 //   - more than it: the replacement fits nowhere even with it stopped — its
 //     profile was edited to need more than any node has free. Stopping it
 //     would leave the model with nothing, so the restart is given up, and
@@ -62,6 +66,8 @@ func (r *Reconciler) retire(ctx context.Context, m catalog.Model, ids []string,
 			r.log.Warn("restarting a replica in place: it is the only thing in its replacement's way",
 				"replica", id, "model", m.Name)
 			r.stopForRestart(ctx, id, "restarted in place: it was the only thing in its replacement's way")
+		case outcome == noRoom && r.now().Sub(since) <= r.cfg.DrainGrace && r.fitsAfterRestarts(m, id):
+			// Another restart under way gives the room back; waited for.
 		case outcome == noRoom || outcome == overLimit:
 			r.refuseRestart(ctx, id, "its replacement fits on no node even with this replica stopped "+
 				"(llmgw ctl explain-placement "+m.Name+")")
@@ -73,6 +79,28 @@ func (r *Reconciler) retire(ctx context.Context, m catalog.Model, ids []string,
 	}
 }
 
+// fitsAfterRestarts is whether a model's replacement for a replica fits with
+// that replica gone and every other on its way out: queued to restart, being
+// replaced, or draining. Their replacements take room too, so this is
+// whether waiting could help, not a promise that it will.
+func (r *Reconciler) fitsAfterRestarts(m catalog.Model, id string) bool {
+	gone := []victim{{ID: id, Model: m.Name}}
+	r.mu.Lock()
+	for oid, o := range r.replicas {
+		if oid == id {
+			continue
+		}
+		if o.State == Draining || !o.RetiringAt.IsZero() || slices.Contains(r.restartQueue[o.Model], oid) {
+			gone = append(gone, victim{ID: oid, Model: o.Model})
+		}
+	}
+	r.mu.Unlock()
+	if len(gone) == 1 {
+		return false
+	}
+	return r.fitsWithout(m, nil, gone)
+}
+
 // stopForRestart drains a replica and stops it at once, and places its
 // replacement in what it freed.
 //

internal/control/reconciler/stuck_test.go

@@ -406,3 +406,34 @@ func TestAMultiNodeReplicaKnowsHowManyShardsItWants(t *testing.T) {
 		t.Errorf("WantShards = %d, want 2", reps[0].WantShards)
 	}
 }
+
+// A model of hundreds of gigabytes on a slow mirror takes longer to arrive
+// than to load: the startup timeout runs from when its download last moved,
+// and a download that stops is failed a timeout after it stopped.
+func TestAReplicaStillDownloadingIsNotFailed(t *testing.T) {
+	m := testModel(1, catalog.Profile{
+		Name: "tp4", GPUClass: "h100-80g", TP: 4, StartupTimeout: 20 * time.Minute,
+	})
+	h := newHarness(t, m, gpuNode("1", 8))
+	h.r.Reconcile(context.Background())
+	rep := h.r.Replicas()[0]
+	shard := rep.Shards[0].ID
+
+	// Forty minutes of download, moving all the while.
+	for i := int64(1); i <= 8; i++ {
+		h.advance(5 * time.Minute)
+		h.r.OnFetchProgress(rep.ID, shard, "mirror", i<<35, 400<<30)
+		h.r.expireStuckReplicas()
+	}
+	if got := h.r.Replicas()[0].State; got == Failed {
+		t.Fatal("a replica whose download kept moving was failed")
+	}
+	// The same figure again is not progress: stalled, it fails a timeout on.
+	h.advance(10 * time.Minute)
+	h.r.OnFetchProgress(rep.ID, shard, "mirror", 8<<35, 400<<30)
+	h.advance(11 * time.Minute)
+	h.r.expireStuckReplicas()
+	if got := h.r.Replicas()[0].State; got != Failed {
+		t.Errorf("a stalled download was never failed: %s", got)
+	}
+}

internal/directory/directory.go

@@ -0,0 +1,395 @@
+// Package directory keeps accounts in step with an outside directory —
+// Active Directory over LDAP — at sign-in and by a periodic sync: made for
+// people the directory vouches for, given the staff role and the places in
+// the tree their groups map to, and suspended when the directory disables or
+// loses them.
+package directory
+
+import (
+	"context"
+	"crypto/rand"
+	"encoding/hex"
+	"errors"
+	"fmt"
+	"slices"
+	"strings"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// Identity is a person as the directory has them.
+type Identity struct {
+	// ID never changes for the person: an AD objectGUID.
+	ID    string
+	DN    string
+	Email string
+	Name  string
+	// Groups are the person's groups, nested ones included, each by its CN
+	// and by its DN, so a mapping may name either.
+	Groups []string
+	// Disabled is an account the directory has turned off.
+	Disabled bool
+}
+
+// Mapping is what a person's groups give them: a staff role and places in
+// the tree. The same shape as auth.oidc's.
+type Mapping struct {
+	Roles       map[string][]string
+	Memberships map[string]string
+}
+
+// Role is the highest staff role the groups give, "" when none.
+func (m Mapping) Role(groups []string) string {
+	for _, r := range []string{store.RoleAdmin, store.RoleOperator, store.RoleBilling, store.RoleViewer} {
+		if slices.ContainsFunc(m.Roles[r], func(g string) bool { return hasGroup(groups, g) }) {
+			return r
+		}
+	}
+	return ""
+}
+
+// Places are the places in the tree the groups give, each once.
+func (m Mapping) Places(groups []string) []string {
+	var out []string
+	for g, where := range m.Memberships {
+		if hasGroup(groups, g) && !slices.Contains(out, where) {
+			out = append(out, where)
+		}
+	}
+	slices.Sort(out)
+	return out
+}
+
+// Groups are every group the mapping names.
+func (m Mapping) Groups() []string {
+	var out []string
+	add := func(g string) {
+		if !slices.ContainsFunc(out, func(o string) bool { return strings.EqualFold(o, g) }) {
+			out = append(out, g)
+		}
+	}
+	for _, gs := range m.Roles {
+		for _, g := range gs {
+			add(g)
+		}
+	}
+	for g := range m.Memberships {
+		add(g)
+	}
+	slices.Sort(out)
+	return out
+}
+
+// hasGroup matches a group's CN or DN as AD does: case aside.
+func hasGroup(groups []string, g string) bool {
+	return slices.ContainsFunc(groups, func(have string) bool { return strings.EqualFold(have, g) })
+}
+
+// Store is what keeping accounts needs of the store.
+type Store interface {
+	UserByEmail(ctx context.Context, email string) (store.User, error)
+	UserByID(ctx context.Context, id string) (store.User, error)
+	CreateUser(ctx context.Context, u store.User) error
+	SetStaffRole(ctx context.Context, id, role string) error
+	SetUserState(ctx context.Context, id string, state store.UserState) error
+	DeleteUserSessions(ctx context.Context, userID string) error
+	AddMembership(ctx context.Context, user string, scope store.Scope, id, role, by string, at time.Time) error
+	RemoveMembership(ctx context.Context, userID string, scope store.Scope, scopeID string, at time.Time) (int, error)
+	MembershipsOf(ctx context.Context, userID string) ([]store.Membership, error)
+	AllOrgs(ctx context.Context) ([]store.Org, error)
+	AllTeams(ctx context.Context) ([]store.TeamPath, error)
+	AllProjects(ctx context.Context) ([]store.ProjectPath, error)
+	DirectoryUser(ctx context.Context, directory, externalID string) (store.DirectoryLink, error)
+	LinkDirectory(ctx context.Context, userID, directory, externalID string, at time.Time) error
+	DirectoryLinks(ctx context.Context, directory string) ([]store.DirectoryLink, error)
+	SetSyncSuspended(ctx context.Context, userID string, suspended bool, at time.Time) error
+}
+
+// Keeper keeps one directory's accounts.
+type Keeper struct {
+	St Store
+	// Name is the directory's, as links and memberships record it: "ldap".
+	Name           string
+	Mapping        Mapping
+	CreateAccounts bool
+	Now            func() time.Time
+}
+
+// The ways a person the directory vouches for is still refused.
+var (
+	ErrNoAccount = errors.New("no account here, and accounts are not made for the directory's people")
+	ErrSuspended = errors.New("the account is suspended here")
+	ErrNotActive = errors.New("the account is not active here")
+	ErrNoEmail   = errors.New("the directory has no mail for this person")
+	// ErrImplausible is a sync that would suspend too many at once: done
+	// nothing, and said.
+	ErrImplausible = errors.New("the directory's answer is implausible; nobody was suspended")
+)
+
+func (k *Keeper) now() time.Time {
+	if k.Now != nil {
+		return k.Now()
+	}
+	return time.Now()
+}
+
+// SignIn is the account for a person the directory has just vouched for:
+// found by their directory id, else their mail, else made; given what their
+// groups map to; refused when it may not sign in.
+func (k *Keeper) SignIn(ctx context.Context, id Identity) (store.User, error) {
+	u, link, err := k.account(ctx, id)
+	if err != nil {
+		return store.User{}, err
+	}
+	// The directory has them enabled, or the bind would have failed: an
+	// account the sync suspended for being disabled there is theirs again.
+	if u.State == store.UserSuspended && link.SuspendedBySync {
+		if u, err = k.reinstate(ctx, u); err != nil {
+			return store.User{}, err
+		}
+	}
+	switch {
+	case u.State == store.UserSuspended:
+		return store.User{}, ErrSuspended
+	case !u.CanSignIn():
+		return store.User{}, fmt.Errorf("%w: %s", ErrNotActive, u.State)
+	}
+	if err := k.apply(ctx, &u, id, false); err != nil {
+		return store.User{}, err
+	}
+	return u, nil
+}
+
+// account finds or makes the account and links it to the directory.
+func (k *Keeper) account(ctx context.Context, id Identity) (store.User, store.DirectoryLink, error) {
+	link, err := k.St.DirectoryUser(ctx, k.Name, id.ID)
+	switch {
+	case err == nil:
+		u, err := k.St.UserByID(ctx, link.UserID)
+		return u, link, err
+	case !errors.Is(err, store.ErrNotFound):
+		return store.User{}, link, err
+	}
+	email := store.NormalizeEmail(id.Email)
+	if email == "" {
+		return store.User{}, link, ErrNoEmail
+	}
+	u, err := k.St.UserByEmail(ctx, email)
+	switch {
+	case errors.Is(err, store.ErrNotFound) && k.CreateAccounts:
+		name := id.Name
+		if name == "" {
+			name = email
+		}
+		u = store.User{ID: "usr-" + randomID(), Email: email, Name: name,
+			// No password works here: the directory's is checked.
+			PasswordHash: "!" + k.Name, State: store.UserActive, CreatedAt: k.now()}
+		if err := k.St.CreateUser(ctx, u); err != nil {
+			return store.User{}, link, err
+		}
+	case errors.Is(err, store.ErrNotFound):
+		return store.User{}, link, fmt.Errorf("%w: %s", ErrNoAccount, email)
+	case err != nil:
+		return store.User{}, link, err
+	}
+	if err := k.St.LinkDirectory(ctx, u.ID, k.Name, id.ID, k.now()); err != nil {
+		return store.User{}, link, err
+	}
+	return u, store.DirectoryLink{UserID: u.ID, Directory: k.Name, ExternalID: id.ID}, nil
+}
+
+// apply gives the account the role and places its groups map to. With
+// remove, a place this directory gave that the groups no longer do is taken
+// away; one given otherwise is never touched.
+func (k *Keeper) apply(ctx context.Context, u *store.User, id Identity, remove bool) error {
+	if len(k.Mapping.Roles) > 0 {
+		if role := k.Mapping.Role(id.Groups); role != u.StaffRole() {
+			if err := k.St.SetStaffRole(ctx, u.ID, role); err != nil {
+				return err
+			}
+			u.IsAdmin, u.Role = role == store.RoleAdmin, role
+			if role == store.RoleAdmin {
+				u.Role = ""
+			}
+		}
+	}
+	want := map[string]store.Scope{}
+	for _, where := range k.Mapping.Places(id.Groups) {
+		scope, sid, err := k.place(ctx, where)
+		if err != nil {
+			continue // a mapping to a place the tree lacks: said by Sync
+		}
+		want[string(scope)+"/"+sid] = scope
+		if err := k.St.AddMembership(ctx, u.ID, scope, sid, store.RoleMember, k.Name, k.now()); err != nil {
+			return err
+		}
+	}
+	if !remove {
+		return nil
+	}
+	have, err := k.St.MembershipsOf(ctx, u.ID)
+	if err != nil {
+		return err
+	}
+	for _, m := range have {
+		if m.AddedBy != k.Name {
+			continue
+		}
+		if _, ok := want[string(m.Scope)+"/"+m.ScopeID]; ok {
+			continue
+		}
+		if _, err := k.St.RemoveMembership(ctx, u.ID, m.Scope, m.ScopeID, k.now()); err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+func (k *Keeper) reinstate(ctx context.Context, u store.User) (store.User, error) {
+	if err := k.St.SetUserState(ctx, u.ID, store.UserActive); err != nil {
+		return u, err
+	}
+	if err := k.St.SetSyncSuspended(ctx, u.ID, false, k.now()); err != nil {
+		return u, err
+	}
+	u.State = store.UserActive
+	return u, nil
+}
+
+// SyncReport is what a sync did.
+type SyncReport struct {
+	Seen, Updated, Suspended, Reinstated int
+	// Unmapped are mappings to places the tree does not have.
+	Unmapped []string
+}
+
+// Sync brings every account the directory keeps in step with what it says
+// now: people disabled or gone there are suspended here (their sessions
+// ended; their keys refuse, as a suspended owner's do), those enabled again
+// that the sync had suspended are reinstated, and everyone's role and places
+// follow their groups. People the directory has that never signed in here
+// are left until they do.
+func (k *Keeper) Sync(ctx context.Context, people []Identity) (SyncReport, error) {
+	var rep SyncReport
+	rep.Unmapped = k.unmapped(ctx)
+	byID := make(map[string]Identity, len(people))
+	for _, p := range people {
+		byID[p.ID] = p
+	}
+	links, err := k.St.DirectoryLinks(ctx, k.Name)
+	if err != nil {
+		return rep, err
+	}
+	// A directory that answers with nobody, or with half the people missing,
+	// is far likelier a wrong base, a filter or a bind account's lost rights
+	// than half the company leaving at once. Nothing is suspended for it.
+	gone := 0
+	for _, l := range links {
+		if p, there := byID[l.ExternalID]; !there || p.Disabled {
+			gone++
+		}
+	}
+	if len(links) > 0 && (len(people) == 0 || (gone >= 3 && gone*2 > len(links))) {
+		return rep, fmt.Errorf("%w: %d of the %d accounts it keeps would be suspended", ErrImplausible, gone, len(links))
+	}
+	for _, l := range links {
+		rep.Seen++
+		u, err := k.St.UserByID(ctx, l.UserID)
+		if err != nil {
+			continue
+		}
+		p, there := byID[l.ExternalID]
+		switch {
+		case !there || p.Disabled:
+			if u.State == store.UserActive {
+				if err := k.St.SetUserState(ctx, u.ID, store.UserSuspended); err != nil {
+					return rep, err
+				}
+				if err := k.St.DeleteUserSessions(ctx, u.ID); err != nil {
+					return rep, err
+				}
+				if err := k.St.SetSyncSuspended(ctx, u.ID, true, k.now()); err != nil {
+					return rep, err
+				}
+				rep.Suspended++
+			}
+		default:
+			if u.State == store.UserSuspended && l.SuspendedBySync {
+				if u, err = k.reinstate(ctx, u); err != nil {
+					return rep, err
+				}
+				rep.Reinstated++
+			}
+			if u.State != store.UserActive {
+				continue
+			}
+			before := u.StaffRole()
+			if err := k.apply(ctx, &u, p, true); err != nil {
+				return rep, err
+			}
+			if u.StaffRole() != before {
+				rep.Updated++
+			}
+		}
+	}
+	return rep, nil
+}
+
+// unmapped are the mapping's places the tree does not have.
+func (k *Keeper) unmapped(ctx context.Context) []string {
+	var out []string
+	for _, where := range k.Mapping.Memberships {
+		if _, _, err := k.place(ctx, where); err != nil && !slices.Contains(out, where) {
+			out = append(out, where)
+		}
+	}
+	slices.Sort(out)
+	return out
+}
+
+// place finds org:NAME, team:ORG/TEAM, project:ORG/TEAM/PROJECT, as
+// auth.oidc's mapping does.
+func (k *Keeper) place(ctx context.Context, where string) (store.Scope, string, error) {
+	kind, path, _ := strings.Cut(where, ":")
+	norm := func(s string) string {
+		parts := strings.Split(s, "/")
+		for i := range parts {
+			parts[i] = strings.TrimSpace(parts[i])
+		}
+		return strings.Join(parts, "/")
+	}
+	same := func(a string) bool { return strings.EqualFold(norm(a), norm(path)) }
+	switch kind {
+	case "org":
+		orgs, err := k.St.AllOrgs(ctx)
+		for _, o := range orgs {
+			if same(o.Name) {
+				return store.ScopeOrg, o.ID, nil
+			}
+		}
+		return "", "", errors.Join(err, errors.New("no such organization"))
+	case "team":
+		teams, err := k.St.AllTeams(ctx)
+		for _, t := range teams {
+			if same(t.OrgName + "/" + t.Name) {
+				return store.ScopeTeam, t.ID, nil
+			}
+		}
+		return "", "", errors.Join(err, errors.New("no such team"))
+	default:
+		projects, err := k.St.AllProjects(ctx)
+		for _, p := range projects {
+			if same(p.Path) {
+				return store.ScopeProject, p.ID, nil
+			}
+		}
+		return "", "", errors.Join(err, errors.New("no such project"))
+	}
+}
+
+func randomID() string {
+	var b [8]byte
+	_, _ = rand.Read(b[:])
+	return hex.EncodeToString(b[:])
+}

internal/directory/directory_test.go

@@ -0,0 +1,142 @@
+package directory
+
+import (
+	"errors"
+	"path/filepath"
+	"slices"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+func newKeeper(t *testing.T) (*Keeper, *store.DB, string) {
+	t.Helper()
+	st, err := store.Open(t.Context(), filepath.Join(t.TempDir(), "s.db"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	t.Cleanup(func() { st.Close() })
+	org, _ := st.CreateOrg(t.Context(), "Acme")
+	team, _ := st.CreateTeam(t.Context(), org, "ml")
+	k := &Keeper{St: st, Name: "ldap", CreateAccounts: true, Mapping: Mapping{
+		Roles:       map[string][]string{"admin": {"llmgw-admins"}, "viewer": {"CN=Staff,OU=Groups,DC=corp,DC=example"}},
+		Memberships: map[string]string{"ml-team": "team:Acme/ml", "ghosts": "team:Acme/nowhere"},
+	}}
+	return k, st, team
+}
+
+var ada = Identity{ID: "guid-ada", DN: "CN=Ada,OU=Staff,DC=corp,DC=example", Email: "Ada@Corp.example", Name: "Ada L",
+	Groups: []string{"ml-team", "CN=ml-team,OU=Groups,DC=corp,DC=example", "Staff", "CN=Staff,OU=Groups,DC=corp,DC=example"}}
+
+func TestAPersonTheDirectoryVouchesForIsMadeAnAccountWithWhatTheirGroupsGive(t *testing.T) {
+	k, st, team := newKeeper(t)
+	u, err := k.SignIn(t.Context(), ada)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if u.Email != "ada@corp.example" || u.Name != "Ada L" || u.PasswordHash != "!ldap" || u.StaffRole() != "viewer" {
+		t.Errorf("account %+v role %q", u, u.StaffRole())
+	}
+	ms, _ := st.MembershipsOf(t.Context(), u.ID)
+	if len(ms) != 1 || ms[0].ScopeID != team || ms[0].AddedBy != "ldap" {
+		t.Errorf("memberships %+v", ms)
+	}
+	// Their mail changes there: still the same account, by its directory id.
+	moved := ada
+	moved.Email = "ada.lovelace@corp.example"
+	again, err := k.SignIn(t.Context(), moved)
+	if err != nil || again.ID != u.ID {
+		t.Errorf("a new mail made a new account: %v %+v", err, again)
+	}
+	// The higher role of two wins.
+	boss := ada
+	boss.Groups = append(slices.Clone(ada.Groups), "llmgw-admins")
+	if u, _ := k.SignIn(t.Context(), boss); u.StaffRole() != "admin" {
+		t.Errorf("role %q", u.StaffRole())
+	}
+}
+
+func TestWithoutAccountsMadeOnlyThoseThatExistSignIn(t *testing.T) {
+	k, _, _ := newKeeper(t)
+	k.CreateAccounts = false
+	if _, err := k.SignIn(t.Context(), ada); !errors.Is(err, ErrNoAccount) {
+		t.Errorf("err %v", err)
+	}
+}
+
+func TestTheSyncSuspendsWhomTheDirectoryDisabledAndReinstatesThemAfter(t *testing.T) {
+	k, st, _ := newKeeper(t)
+	ctx := t.Context()
+	people := []Identity{ada}
+	for _, n := range []string{"b", "c", "d"} {
+		people = append(people, Identity{ID: "guid-" + n, Email: n + "@corp.example"})
+	}
+	ids := map[string]string{}
+	for _, p := range people {
+		u, err := k.SignIn(ctx, p)
+		if err != nil {
+			t.Fatal(err)
+		}
+		ids[p.ID] = u.ID
+	}
+	if err := st.CreateSession(ctx, ids["guid-ada"], "tok", "csrf", time.Hour); err != nil {
+		t.Fatal(err)
+	}
+	// An administrator suspends c here, by hand.
+	_ = st.SetUserState(ctx, ids["guid-c"], store.UserSuspended)
+
+	// Ada is disabled there, and has left ml-team.
+	off := ada
+	off.Disabled = true
+	rep, err := k.Sync(ctx, []Identity{off, people[1], people[2], people[3]})
+	if err != nil {
+		t.Fatal(err)
+	}
+	u, _ := st.UserByID(ctx, ids["guid-ada"])
+	if u.State != store.UserSuspended || rep.Suspended != 1 {
+		t.Errorf("ada %s, report %+v", u.State, rep)
+	}
+	if _, _, err := st.SessionByToken(ctx, "tok"); err == nil {
+		t.Error("a suspended account's session still works")
+	}
+	if !slices.Contains(rep.Unmapped, "team:Acme/nowhere") {
+		t.Errorf("an unmapped place was not said: %+v", rep)
+	}
+
+	// Enabled again, and out of ml-team: reinstated, and its place taken
+	// away. c, suspended by hand, stays so.
+	back := ada
+	back.Groups = []string{"Staff", "CN=Staff,OU=Groups,DC=corp,DC=example"}
+	rep, err = k.Sync(ctx, []Identity{back, people[1], people[2], people[3]})
+	if err != nil {
+		t.Fatal(err)
+	}
+	u, _ = st.UserByID(ctx, ids["guid-ada"])
+	ms, _ := st.MembershipsOf(ctx, u.ID)
+	if u.State != store.UserActive || rep.Reinstated != 1 || len(ms) != 0 {
+		t.Errorf("ada %s, memberships %+v, report %+v", u.State, ms, rep)
+	}
+	if c, _ := st.UserByID(ctx, ids["guid-c"]); c.State != store.UserSuspended {
+		t.Errorf("an account suspended by hand was reinstated: %s", c.State)
+	}
+}
+
+func TestASyncThatWouldSuspendMostIsRefused(t *testing.T) {
+	k, st, _ := newKeeper(t)
+	ctx := t.Context()
+	for _, n := range []string{"a", "b", "c", "d"} {
+		if _, err := k.SignIn(ctx, Identity{ID: "guid-" + n, Email: n + "@corp.example"}); err != nil {
+			t.Fatal(err)
+		}
+	}
+	for _, people := range [][]Identity{nil, {{ID: "guid-a", Email: "a@corp.example"}}} {
+		if _, err := k.Sync(ctx, people); !errors.Is(err, ErrImplausible) {
+			t.Errorf("%d people: %v", len(people), err)
+		}
+	}
+	users, _ := st.Users(ctx, store.UserSuspended)
+	if len(users) != 0 {
+		t.Errorf("suspended %d", len(users))
+	}
+}

internal/directory/ldap.go

@@ -0,0 +1,413 @@
+package directory
+
+import (
+	"context"
+	"crypto/tls"
+	"crypto/x509"
+	"encoding/binary"
+	"encoding/pem"
+	"errors"
+	"fmt"
+	"os"
+	"strconv"
+	"strings"
+
+	"github.com/go-ldap/ldap/v3"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/secrets"
+)
+
+// LDAP is a directory reached over LDAP, written for Active Directory: its
+// attribute names, its disabled flag, and its matching rule for nested
+// groups (LDAP_MATCHING_RULE_IN_CHAIN).
+type LDAP struct {
+	cfg  config.LDAP
+	tls  *tls.Config
+	dial func(url string) (ldapConn, error)
+	// password, when set, is the service account's in place of the one
+	// bind_password_file names: settings tried before they are kept.
+	password string
+}
+
+// WithPassword has the service account bind with pw rather than read it.
+func (d *LDAP) WithPassword(pw string) *LDAP {
+	d.password = pw
+	return d
+}
+
+func (d *LDAP) servicePassword(ctx context.Context) (string, error) {
+	if d.password != "" {
+		return d.password, nil
+	}
+	pw, err := secrets.Read(ctx, d.cfg.BindPasswordFile)
+	if err != nil {
+		return "", fmt.Errorf("auth.ldap.bind_password_file: %w", err)
+	}
+	return strings.TrimSpace(string(pw)), nil
+}
+
+// ldapConn is the part of a connection used here, so tests need no server.
+type ldapConn interface {
+	Bind(user, password string) error
+	Search(*ldap.SearchRequest) (*ldap.SearchResult, error)
+	SearchWithPaging(*ldap.SearchRequest, uint32) (*ldap.SearchResult, error)
+	Close() error
+}
+
+// inChain is AD's matching rule that follows nested group membership.
+const inChain = "1.2.840.113556.1.4.1941"
+
+// uacDisabled is userAccountControl's ACCOUNTDISABLE bit.
+const uacDisabled = 0x2
+
+var personAttrs = []string{"objectGUID", "sAMAccountName", "userPrincipalName", "mail", "displayName", "userAccountControl"}
+
+// ErrBadCredentials is a person the directory does not know, or a password
+// it does not take: said alike, so a form cannot tell which.
+var ErrBadCredentials = errors.New("the directory does not take that name and password")
+
+// NewLDAP prepares a directory from its configuration.
+func NewLDAP(cfg config.LDAP) (*LDAP, error) {
+	tc := &tls.Config{MinVersion: tls.VersionTLS12}
+	ca, where := []byte(cfg.CA), "auth.ldap.ca"
+	if cfg.CA == "" && cfg.CAFile != "" {
+		var err error
+		if ca, err = os.ReadFile(cfg.CAFile); err != nil {
+			return nil, fmt.Errorf("auth.ldap.ca_file: %w", err)
+		}
+		where = "auth.ldap.ca_file " + cfg.CAFile
+	}
+	if len(ca) > 0 {
+		pool, err := certPool(ca)
+		if err != nil {
+			return nil, fmt.Errorf("%s: %w", where, err)
+		}
+		tc.RootCAs = pool
+	}
+	d := &LDAP{cfg: cfg, tls: tc}
+	d.dial = d.dialURL
+	return d, nil
+}
+
+// certPool is the certificates in pem, each of which must parse: a
+// certificate Go refuses (a negative serial, as Samba's own has) is said,
+// not skipped into an empty pool.
+func certPool(data []byte) (*x509.CertPool, error) {
+	pool := x509.NewCertPool()
+	n := 0
+	for rest := data; ; {
+		var b *pem.Block
+		if b, rest = pem.Decode(rest); b == nil {
+			break
+		}
+		if b.Type != "CERTIFICATE" {
+			continue
+		}
+		c, err := x509.ParseCertificate(b.Bytes)
+		if err != nil {
+			return nil, err
+		}
+		pool.AddCert(c)
+		n++
+	}
+	if n == 0 {
+		return nil, errors.New("holds no certificate")
+	}
+	return pool, nil
+}
+
+func (d *LDAP) dialURL(u string) (ldapConn, error) {
+	c, err := ldap.DialURL(u, ldap.DialWithTLSConfig(d.tls))
+	if err != nil {
+		return nil, err
+	}
+	c.SetTimeout(d.cfg.RoundTrip())
+	if strings.HasPrefix(u, "ldap://") {
+		// Validate refuses ldap:// without start_tls: nothing is sent first.
+		if err := c.StartTLS(d.tls); err != nil {
+			c.Close()
+			return nil, fmt.Errorf("StartTLS: %w", err)
+		}
+	}
+	return c, nil
+}
+
+// service is a connection bound as the account that finds people, to the
+// first server that answers.
+func (d *LDAP) service(ctx context.Context) (ldapConn, error) {
+	pw, err := d.servicePassword(ctx)
+	if err != nil {
+		return nil, err
+	}
+	var errs []error
+	for _, u := range d.cfg.URLs {
+		c, err := d.dial(u)
+		if err != nil {
+			errs = append(errs, fmt.Errorf("%s: %w", u, err))
+			continue
+		}
+		if err := c.Bind(d.cfg.BindDN, pw); err != nil {
+			c.Close()
+			// A server that refuses the service account refuses it
+			// everywhere: not tried on the next.
+			return nil, fmt.Errorf("binding as %s on %s: %w", d.cfg.BindDN, u, err)
+		}
+		return c, nil
+	}
+	return nil, fmt.Errorf("no directory server answered: %w", errors.Join(errs...))
+}
+
+// Authenticate checks a person's name and password against the directory
+// and says who they are there, groups and all.
+func (d *LDAP) Authenticate(ctx context.Context, login, password string) (Identity, error) {
+	login = strings.TrimSpace(login)
+	// An empty password is an unauthenticated bind, which succeeds: never
+	// one for a person.
+	if login == "" || password == "" {
+		return Identity{}, ErrBadCredentials
+	}
+	// DOMAIN\user, as Windows writes it.
+	if i := strings.LastIndexByte(login, '\\'); i >= 0 {
+		login = login[i+1:]
+	}
+	c, err := d.service(ctx)
+	if err != nil {
+		return Identity{}, err
+	}
+	defer c.Close()
+	q := ldap.EscapeFilter(login)
+	filter := "(&" + d.people(false) + "(|(sAMAccountName=" + q + ")(userPrincipalName=" + q + ")(mail=" + q + ")))"
+	res, err := c.Search(ldap.NewSearchRequest(d.cfg.BaseDN, ldap.ScopeWholeSubtree, ldap.NeverDerefAliases,
+		2, 0, false, filter, personAttrs, nil))
+	if err != nil {
+		return Identity{}, fmt.Errorf("finding %s: %w", login, err)
+	}
+	if len(res.Entries) != 1 {
+		// None, or two people one name matches: neither is let in.
+		return Identity{}, ErrBadCredentials
+	}
+	id := identity(res.Entries[0])
+	if err := c.Bind(id.DN, password); err != nil {
+		var le *ldap.Error
+		if errors.As(err, &le) && le.ResultCode == ldap.LDAPResultInvalidCredentials {
+			return Identity{}, ErrBadCredentials
+		}
+		return Identity{}, fmt.Errorf("checking %s's password: %w", login, err)
+	}
+	// Back to the service account to read groups, which a person may not.
+	pw, err := d.servicePassword(ctx)
+	if err != nil {
+		return Identity{}, err
+	}
+	if err := c.Bind(d.cfg.BindDN, pw); err != nil {
+		return Identity{}, err
+	}
+	if id.Groups, err = d.groupsOf(c, id.DN); err != nil {
+		return Identity{}, err
+	}
+	return id, nil
+}
+
+// groupsOf are a person's groups, nested ones included, by CN and DN.
+func (d *LDAP) groupsOf(c ldapConn, dn string) ([]string, error) {
+	filter := "(&(objectClass=group)(member:" + inChain + ":=" + ldap.EscapeFilter(dn) + "))"
+	res, err := c.SearchWithPaging(ldap.NewSearchRequest(d.groupBase(), ldap.ScopeWholeSubtree, ldap.NeverDerefAliases,
+		0, 0, false, filter, []string{"cn"}, nil), 500)
+	if err != nil {
+		return nil, fmt.Errorf("reading the groups of %s: %w", dn, err)
+	}
+	var out []string
+	for _, e := range res.Entries {
+		out = append(out, e.GetAttributeValue("cn"), e.DN)
+	}
+	return out, nil
+}
+
+// People are everyone the directory has under the base, enabled or not,
+// each with the mapped groups they are in: what a sync compares against.
+func (d *LDAP) People(ctx context.Context, m Mapping) ([]Identity, error) {
+	c, err := d.service(ctx)
+	if err != nil {
+		return nil, err
+	}
+	defer c.Close()
+	res, err := c.SearchWithPaging(ldap.NewSearchRequest(d.cfg.BaseDN, ldap.ScopeWholeSubtree, ldap.NeverDerefAliases,
+		0, 0, false, d.people(true), personAttrs, nil), 500)
+	if err != nil {
+		return nil, fmt.Errorf("listing people: %w", err)
+	}
+	people := make([]Identity, 0, len(res.Entries))
+	at := map[string]int{}
+	for _, e := range res.Entries {
+		id := identity(e)
+		at[id.ID] = len(people)
+		people = append(people, id)
+	}
+	// Only the groups the mapping names matter: a query each, for its
+	// members nested ones included, rather than one per person.
+	for _, g := range m.Groups() {
+		dns, err := d.groupDNs(c, g)
+		if err != nil {
+			return nil, err
+		}
+		for _, gdn := range dns {
+			filter := "(&" + d.people(true) + "(memberOf:" + inChain + ":=" + ldap.EscapeFilter(gdn) + "))"
+			r, err := c.SearchWithPaging(ldap.NewSearchRequest(d.cfg.BaseDN, ldap.ScopeWholeSubtree,
+				ldap.NeverDerefAliases, 0, 0, false, filter, []string{"objectGUID"}, nil), 500)
+			if err != nil {
+				return nil, fmt.Errorf("reading the members of %s: %w", gdn, err)
+			}
+			for _, e := range r.Entries {
+				if i, ok := at[guid(e.GetRawAttributeValue("objectGUID"))]; ok {
+					people[i].Groups = append(people[i].Groups, g, gdn)
+				}
+			}
+		}
+	}
+	return people, nil
+}
+
+// Check is one step of trying a directory.
+type Check struct {
+	Step   string `json:"step"`
+	OK     bool   `json:"ok"`
+	Detail string `json:"detail"`
+}
+
+// Check tries the directory as configured, before its settings are kept:
+// each server reached and the service account bound on it, how many people
+// may sign in, and how many are in each group the mapping names.
+func (d *LDAP) Check(ctx context.Context, m Mapping) []Check {
+	pw, err := d.servicePassword(ctx)
+	if err != nil {
+		return []Check{{Step: "password", Detail: err.Error()}}
+	}
+	var out []Check
+	var c ldapConn
+	for _, u := range d.cfg.URLs {
+		conn, err := d.dial(u)
+		if err == nil {
+			if err = conn.Bind(d.cfg.BindDN, pw); err != nil {
+				conn.Close()
+			}
+		}
+		if err != nil {
+			out = append(out, Check{Step: u, Detail: err.Error()})
+			continue
+		}
+		out = append(out, Check{Step: u, OK: true, Detail: "reached, bound as " + d.cfg.BindDN})
+		if c == nil {
+			c = conn
+		} else {
+			conn.Close()
+		}
+	}
+	if c == nil {
+		return out
+	}
+	defer c.Close()
+	res, err := c.SearchWithPaging(ldap.NewSearchRequest(d.cfg.BaseDN, ldap.ScopeWholeSubtree, ldap.NeverDerefAliases,
+		0, 0, false, d.people(false), []string{"objectGUID"}, nil), 500)
+	switch {
+	case err != nil:
+		out = append(out, Check{Step: "people", Detail: err.Error()})
+	case len(res.Entries) == 0:
+		out = append(out, Check{Step: "people", Detail: "no enabled user under base_dn matches user_filter: no one could sign in"})
+	default:
+		out = append(out, Check{Step: "people", OK: true,
+			Detail: fmt.Sprintf("%d enabled people under base_dn may sign in", len(res.Entries))})
+	}
+	for _, g := range m.Groups() {
+		dns, err := d.groupDNs(c, g)
+		if err != nil || len(dns) == 0 {
+			detail := "no group of that name under " + d.groupBase()
+			if err != nil {
+				detail = err.Error()
+			}
+			out = append(out, Check{Step: "group " + g, Detail: detail})
+			continue
+		}
+		n := 0
+		for _, gdn := range dns {
+			filter := "(&" + d.people(false) + "(memberOf:" + inChain + ":=" + ldap.EscapeFilter(gdn) + "))"
+			r, err := c.SearchWithPaging(ldap.NewSearchRequest(d.cfg.BaseDN, ldap.ScopeWholeSubtree,
+				ldap.NeverDerefAliases, 0, 0, false, filter, []string{"objectGUID"}, nil), 500)
+			if err == nil {
+				n += len(r.Entries)
+			}
+		}
+		out = append(out, Check{Step: "group " + g, OK: true,
+			Detail: fmt.Sprintf("%d of the people who may sign in are in it, nested groups included", n)})
+	}
+	return out
+}
+
+// groupDNs are the groups a mapping's name means: a DN as it is, a CN
+// looked up.
+func (d *LDAP) groupDNs(c ldapConn, g string) ([]string, error) {
+	if strings.Contains(g, "=") {
+		return []string{g}, nil
+	}
+	res, err := c.Search(ldap.NewSearchRequest(d.groupBase(), ldap.ScopeWholeSubtree, ldap.NeverDerefAliases,
+		0, 0, false, "(&(objectClass=group)(cn="+ldap.EscapeFilter(g)+"))", []string{"cn"}, nil))
+	if err != nil {
+		return nil, fmt.Errorf("finding the group %s: %w", g, err)
+	}
+	var out []string
+	for _, e := range res.Entries {
+		out = append(out, e.DN)
+	}
+	return out, nil
+}
+
+// people is the filter for person accounts under the user filter; enabled
+// ones only, unless all.
+func (d *LDAP) people(all bool) string {
+	f := "(objectCategory=person)(objectClass=user)"
+	if !all {
+		f += "(!(userAccountControl:1.2.840.113556.1.4.803:=" + strconv.Itoa(uacDisabled) + "))"
+	}
+	return "(&" + f + d.cfg.UserFilter + ")"
+}
+
+// groupBase is where groups are searched: GroupBaseDN, else the domain root
+// of BaseDN (its DC= parts).
+func (d *LDAP) groupBase() string {
+	if d.cfg.GroupBaseDN != "" {
+		return d.cfg.GroupBaseDN
+	}
+	var dc []string
+	for _, rdn := range strings.Split(d.cfg.BaseDN, ",") {
+		if strings.HasPrefix(strings.ToUpper(strings.TrimSpace(rdn)), "DC=") {
+			dc = append(dc, strings.TrimSpace(rdn))
+		}
+	}
+	if len(dc) == 0 {
+		return d.cfg.BaseDN
+	}
+	return strings.Join(dc, ",")
+}
+
+func identity(e *ldap.Entry) Identity {
+	email := e.GetAttributeValue("mail")
+	if email == "" {
+		email = e.GetAttributeValue("userPrincipalName")
+	}
+	uac, _ := strconv.Atoi(e.GetAttributeValue("userAccountControl"))
+	return Identity{
+		ID: guid(e.GetRawAttributeValue("objectGUID")), DN: e.DN,
+		Email: email, Name: e.GetAttributeValue("displayName"),
+		Disabled: uac&uacDisabled != 0,
+	}
+}
+
+// guid is an objectGUID as Windows writes it: its first three parts
+// little-endian. Another directory's id is kept as hex.
+func guid(b []byte) string {
+	if len(b) != 16 {
+		return fmt.Sprintf("%x", b)
+	}
+	return fmt.Sprintf("%08x-%04x-%04x-%x-%x", binary.LittleEndian.Uint32(b[0:4]),
+		binary.LittleEndian.Uint16(b[4:6]), binary.LittleEndian.Uint16(b[6:8]), b[8:10], b[10:16])
+}

internal/directory/ldap_test.go

@@ -0,0 +1,162 @@
+package directory
+
+import (
+	"errors"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+
+	"github.com/go-ldap/ldap/v3"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+)
+
+// fakeConn is a directory with one person in it.
+type fakeConn struct {
+	binds    []string
+	filters  []string
+	password string // the person's
+	people   int    // how many the person search finds
+	refused  bool   // the service account's password is wrong
+}
+
+func (f *fakeConn) Bind(user, password string) error {
+	f.binds = append(f.binds, user)
+	if f.refused && strings.HasPrefix(user, "CN=llmgw") {
+		return ldap.NewError(ldap.LDAPResultInvalidCredentials, errors.New("invalid credentials"))
+	}
+	if strings.HasPrefix(user, "CN=Ada") && password != f.password {
+		return ldap.NewError(ldap.LDAPResultInvalidCredentials, errors.New("invalid credentials"))
+	}
+	return nil
+}
+
+func (f *fakeConn) Search(r *ldap.SearchRequest) (*ldap.SearchResult, error) {
+	f.filters = append(f.filters, r.Filter)
+	var out []*ldap.Entry
+	for i := range f.people {
+		e := ldap.NewEntry("CN=Ada,OU=Staff,DC=corp,DC=example", map[string][]string{
+			"mail": {"ada@corp.example"}, "displayName": {"Ada"}, "userAccountControl": {"512"},
+		})
+		e.Attributes = append(e.Attributes, &ldap.EntryAttribute{Name: "objectGUID", ByteValues: [][]byte{guidBytes(byte(i))}})
+		out = append(out, e)
+	}
+	return &ldap.SearchResult{Entries: out}, nil
+}
+
+func (f *fakeConn) SearchWithPaging(r *ldap.SearchRequest, _ uint32) (*ldap.SearchResult, error) {
+	f.filters = append(f.filters, r.Filter)
+	return &ldap.SearchResult{Entries: []*ldap.Entry{
+		ldap.NewEntry("CN=ml-team,OU=Groups,DC=corp,DC=example", map[string][]string{"cn": {"ml-team"}}),
+	}}, nil
+}
+
+func (f *fakeConn) Close() error { return nil }
+
+func guidBytes(last byte) []byte {
+	return []byte{0x78, 0x56, 0x34, 0x12, 0x34, 0x12, 0x78, 0x56, 0x9a, 0xbc, 0xde, 0xf0, 0x12, 0x34, 0x56, last}
+}
+
+func fakeLDAP(t *testing.T, c *fakeConn) *LDAP {
+	t.Helper()
+	pw := filepath.Join(t.TempDir(), "bind")
+	_ = os.WriteFile(pw, []byte("svc-secret\n"), 0o600)
+	d, err := NewLDAP(config.LDAP{URLs: []string{"ldaps://dc1"}, BindDN: "CN=llmgw,OU=Svc,DC=corp,DC=example",
+		BindPasswordFile: pw, BaseDN: "OU=Staff,DC=corp,DC=example", UserFilter: "(memberOf=CN=llm,DC=corp,DC=example)"})
+	if err != nil {
+		t.Fatal(err)
+	}
+	d.dial = func(string) (ldapConn, error) { return c, nil }
+	return d
+}
+
+func TestAPersonIsCheckedByBindingAsThemAndTheirGroupsReadNested(t *testing.T) {
+	c := &fakeConn{password: "ad-secret", people: 1}
+	d := fakeLDAP(t, c)
+	id, err := d.Authenticate(t.Context(), `CORP\ada*)(x=`, "ad-secret")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if id.Email != "ada@corp.example" || id.ID != "12345678-1234-5678-9abc-def012345600" ||
+		!slicesHas(id.Groups, "ml-team") || !slicesHas(id.Groups, "CN=ml-team,OU=Groups,DC=corp,DC=example") {
+		t.Errorf("identity %+v", id)
+	}
+	// The name is escaped into the filter, the domain part dropped; the
+	// user filter is ANDed in; disabled accounts are not found; groups are
+	// read with the in-chain rule.
+	find, groups := c.filters[0], c.filters[1]
+	if !strings.Contains(find, `(sAMAccountName=ada\2a\29\28x=)`) || !strings.Contains(find, "(memberOf=CN=llm,DC=corp,DC=example)") ||
+		!strings.Contains(find, "1.2.840.113556.1.4.803:=2") || !strings.Contains(groups, inChain) {
+		t.Errorf("filters %q", c.filters)
+	}
+	// Service account, then the person, then the service account again.
+	if len(c.binds) != 3 || c.binds[1] != "CN=Ada,OU=Staff,DC=corp,DC=example" {
+		t.Errorf("binds %v", c.binds)
+	}
+}
+
+func TestWrongPasswordsEmptyOnesAndAmbiguousNamesAreRefusedAlike(t *testing.T) {
+	for _, tc := range []struct {
+		people   int
+		password string
+	}{{1, "wrong"}, {1, ""}, {0, "ad-secret"}, {2, "ad-secret"}} {
+		d := fakeLDAP(t, &fakeConn{password: "ad-secret", people: tc.people})
+		if _, err := d.Authenticate(t.Context(), "ada", tc.password); !errors.Is(err, ErrBadCredentials) {
+			t.Errorf("%d found, password %q: %v", tc.people, tc.password, err)
+		}
+	}
+}
+
+func TestPlaintextLDAPIsRefusedWithoutStartTLS(t *testing.T) {
+	base := config.LDAP{BindDN: "CN=x", BindPasswordFile: "/f", BaseDN: "DC=corp"}
+	for _, c := range []struct {
+		urls  []string
+		start bool
+		ok    bool
+	}{{[]string{"ldaps://dc1"}, false, true}, {[]string{"ldap://dc1"}, false, false},
+		{[]string{"ldap://dc1"}, true, true}, {[]string{"http://dc1"}, true, false}} {
+		l := base
+		l.URLs, l.StartTLS = c.urls, c.start
+		if err := l.Validate(); (err == nil) != c.ok {
+			t.Errorf("%v start_tls=%v: %v", c.urls, c.start, err)
+		}
+	}
+}
+
+func slicesHas(xs []string, x string) bool {
+	for _, v := range xs {
+		if v == x {
+			return true
+		}
+	}
+	return false
+}
+
+func TestADirectoryIsTriedStepByStepBeforeItsSettingsAreKept(t *testing.T) {
+	m := Mapping{Roles: map[string][]string{"admin": {"llmgw-admins"}}}
+	checks := fakeLDAP(t, &fakeConn{people: 1}).WithPassword("typed-in").Check(t.Context(), m)
+	if len(checks) != 3 || !checks[0].OK || checks[0].Step != "ldaps://dc1" || !checks[1].OK ||
+		checks[2].Step != "group llmgw-admins" || !checks[2].OK {
+		t.Errorf("checks %+v", checks)
+	}
+	// A refused service account stops at the server, and says why.
+	checks = fakeLDAP(t, &fakeConn{refused: true}).Check(t.Context(), m)
+	if len(checks) != 1 || checks[0].OK || !strings.Contains(checks[0].Detail, "Invalid Credentials") {
+		t.Errorf("refused: %+v", checks)
+	}
+}
+
+func TestACertificateGoCannotReadIsSaidNotSkipped(t *testing.T) {
+	if _, err := certPool([]byte("not a certificate")); err == nil {
+		t.Error("an empty pool was taken")
+	}
+	// A CERTIFICATE block Go cannot parse, as it cannot Samba's own (its
+	// serial is negative): an error, not an empty pool.
+	neg := "-----BEGIN CERTIFICATE-----\nMIIBNTCB3KADAgECAgT/////MAoGCCqGSM49BAMCMA8xDTALBgNVBAMTBHRlc3Qw\n" +
+		"HhcNMjYwMTAxMDAwMDAwWhcNMjgwMTAxMDAwMDAwWjAPMQ0wCwYDVQQDEwR0ZXN0\n" +
+		"MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE\n-----END CERTIFICATE-----\n"
+	if _, err := certPool([]byte(neg)); err == nil {
+		t.Error("a broken certificate made a pool")
+	}
+}

internal/docgen/reference_gen.go

@@ -33,6 +33,7 @@ var Reference = []Block{
 		{Key: "default_tpm", Type: "int", Doc: "", Nested: ""},
 		{Key: "signup_open", Type: "bool", Doc: "SignupOpen serves the signup form. Accounts it creates are pending until\nan operator approves them, so this is not a way in — but turning it off\nmeans accounts are made with `llmgw users create` instead.", Nested: ""},
 		{Key: "oidc", Type: "OIDC", Doc: "OIDC is sign-in through an identity provider.", Nested: "config.OIDC"},
+		{Key: "ldap", Type: "LDAP", Doc: "LDAP is sign-in against a directory, Active Directory above all, and\nthe accounts kept in step with it.", Nested: "config.LDAP"},
 		{Key: "scim", Type: "SCIM", Doc: "SCIM is account provisioning by the identity provider.", Nested: "config.SCIM"},
 		{Key: "max_key_ttl", Type: "time.Duration", Doc: "MaxKeyTTL is how long any API key may live; a key asked for with no\nexpiry is given this. Organizations, teams and projects can set\nshorter ones (llmgw keys ttl). Zero is no limit.", Nested: ""},
 		{Key: "personal", Type: "Personal", Doc: "Personal is what a person may do on their own, outside any project: make\npersonal keys and use the chat page personally, within these limits.", Nested: "config.Personal"},
@@ -52,6 +53,7 @@ var Reference = []Block{
 	{Package: "config", Name: "Cache", Title: "Cache", Doc: "", Fields: []Field{
 		{Key: "max_bytes", Type: "int64", Doc: "MaxBytes bounds every project's answers together; 128 MB unless set.\nPast it, the answers used longest ago go first.", Nested: ""},
 		{Key: "max_entry_bytes", Type: "int64", Doc: "MaxEntryBytes bounds one answer; a longer one is not kept. 1 MB\nunless set.", Nested: ""},
+		{Key: "embedding_model", Type: "string", Doc: "EmbeddingModel embeds the questions of projects that reuse answers to\nsimilar ones (llmgw keys cache --similar): an embedding model in the\ncatalog. Unset, those projects reuse only the same request's answer.\nThe gateway asks it on its own account, as it asks a judge.", Nested: ""},
 	}},
 	{Package: "catalog", Name: "Canary", Title: "Canary", Doc: "", Fields: []Field{
 		{Key: "model", Type: "string", Doc: "Model is the candidate: a catalog model, usually this one at newer\nweights, which clients cannot name themselves.", Nested: ""},
@@ -125,11 +127,16 @@ var Reference = []Block{
 		{Key: "url", Type: "string", Doc: "URL is the provider's base address, without /v1: https://api.openai.com.", Nested: ""},
 		{Key: "model", Type: "string", Doc: "Model is what the provider calls it; requests are rewritten to it.", Nested: ""},
 		{Key: "api_key_file", Type: "string", Doc: "APIKeyFile holds the provider's key, on the control plane. A file\nrather than the key itself, so the key is in neither the catalog, the\nstore nor the audit log; empty for a provider that asks for none.", Nested: ""},
+		{Key: "api_key_files", Type: "[]ExternalKey", Doc: "APIKeyFiles are several keys for the provider, in place of\nAPIKeyFile: one endpoint each (external:MODEL#1, #2…), so a key that\nanswers 429 or 401 is passed over while its breaker is open and the\nrequest goes to another. A provider's limits are per key.", Nested: ""},
 		{Key: "peer", Type: "string", Doc: "Peer names the federated llmgw cluster this model is offered by, for\na model that federation added (`llmgw ctl federation add`) rather\nthan one written by hand. Such a model is named PEER/MODEL, comes and\ngoes with the peer's own list, and is sent only while the peer is up.", Nested: ""},
 		{Key: "api", Type: "string", Doc: "API is what the provider speaks: OpenAI's, unless this says comfyui —\na ComfyUI server, asked for pictures at /v1/images/generations and\nsent the workflow below with the request's prompt in it. Model is then\nthe checkpoint file the workflow loads, as ComfyUI lists it.", Nested: ""},
 		{Key: "workflow", Type: "string", Doc: "Workflow is the ComfyUI workflow, in its API format: a built-in's name\n(flux-schnell, the default, or flux-dev) or a JSON file on the control\nplane.", Nested: ""},
 		{Key: "output", Type: "string", Doc: "Output is what it makes: pictures, or meshes (see ComfyUI.Output).", Nested: ""},
 	}},
+	{Package: "catalog", Name: "ExternalKey", Title: "ExternalKey", Doc: "", Fields: []Field{
+		{Key: "file", Type: "string", Doc: "File holds the key, on the control plane, as APIKeyFile does.", Nested: ""},
+		{Key: "weight", Type: "float64", Doc: "Weight is its share of the requests against the others'; zero is 1.", Nested: ""},
+	}},
 	{Package: "config", Name: "Facility", Title: "Facility", Doc: "", Fields: []Field{
 		{Key: "budgets", Type: "[]FacilityBudget", Doc: "Budgets are the domains with a power cap.", Nested: ""},
 		{Key: "margin", Type: "float64", Doc: "Margin is the share of every cap kept free, 0.1 for ten percent. Zero\nkeeps none, for caps already given derated.", Nested: ""},
@@ -183,6 +190,23 @@ var Reference = []Block{
 		{Key: "min_blocks", Type: "int", Doc: "MinBlocks is the fewest leading blocks a replica must hold to be\npreferred for them; fewer is coincidence — a chat template's opening\ntokens — rather than a prefix worth following. Zero takes 4, which is\n64 tokens at vLLM's usual block size of 16.", Nested: ""},
 		{Key: "max_blocks", Type: "int", Doc: "MaxBlocks bounds how many of a prompt's blocks are hashed and looked\nup. Beyond it a longer match still helps, but the difference no longer\ndecides anything. Zero takes 4096.", Nested: ""},
 	}},
+	{Package: "config", Name: "LDAP", Title: "LDAP", Doc: "", Fields: []Field{
+		{Key: "urls", Type: "[]string", Doc: "URLs are the directory's servers, tried in turn: ldaps:// (636), or\nldap:// (389) with StartTLS.", Nested: ""},
+		{Key: "start_tls", Type: "bool", Doc: "StartTLS upgrades an ldap:// connection before anything is sent. A\npassword never crosses a connection without TLS: ldap:// without it\nis refused.", Nested: ""},
+		{Key: "ca_file", Type: "string", Doc: "CAFile is the authority the servers' certificates are checked against,\nwhen it is not the system's: an AD Certificate Services root.", Nested: ""},
+		{Key: "ca", Type: "string", Doc: "CA is the same authority written out, PEM: kept in the cluster's\nsettings, so no file has to be put on every control plane. Used in\nplace of CAFile when both are set.", Nested: ""},
+		{Key: "bind_dn", Type: "string", Doc: "BindDN and the file holding its password: a service account that may\nread users and groups. Nothing is written to the directory.", Nested: ""},
+		{Key: "bind_password_file", Type: "string", Doc: "", Nested: ""},
+		{Key: "base_dn", Type: "string", Doc: "BaseDN is where people are searched for, subtree included.", Nested: ""},
+		{Key: "group_base_dn", Type: "string", Doc: "GroupBaseDN is where their groups are; empty is BaseDN's domain root.", Nested: ""},
+		{Key: "user_filter", Type: "string", Doc: "UserFilter narrows who may sign in, ANDed with the login match: a\ngroup (memberOf=...) or an OU's people only. Empty is every enabled\nuser object.", Nested: ""},
+		{Key: "label", Type: "string", Doc: "Label is what the form says to sign in with: \"Sign in with your\n<label> account\". Empty is \"Active Directory\".", Nested: ""},
+		{Key: "create_accounts", Type: "bool", Doc: "CreateAccounts makes an account, active, for a person the directory\nvouches for and the gateway does not know. Off, only existing accounts\nsign in this way.", Nested: ""},
+		{Key: "roles", Type: "map[string][]string", Doc: "Roles maps a staff role (viewer, operator, billing, admin) to the\ngroups that give it, by name (CN) or whole DN. Set at every sign-in\nand sync: someone taken out of the group loses the role.", Nested: ""},
+		{Key: "memberships", Type: "map[string]string", Doc: "Memberships maps a group, by CN or DN, to a place in the tree —\norg:NAME, team:ORG/TEAM, project:ORG/TEAM/PROJECT — joined at sign-in\nand kept by the sync, which also takes away a place whose group the\nperson has left.", Nested: ""},
+		{Key: "sync_interval", Type: "time.Duration", Doc: "SyncInterval is how often the leader reads the directory. Zero is\n15m; negative turns the sync off, leaving only sign-in.", Nested: ""},
+		{Key: "timeout", Type: "time.Duration", Doc: "Timeout bounds each round trip to a server. Zero is 10s.", Nested: ""},
+	}},
 	{Package: "config", Name: "Landing", Title: "Landing", Doc: "", Fields: []Field{
 		{Key: "contact", Type: "string", Doc: "Contact is where someone without a key should go — an address, a URL, a\nchannel name. Rendered as a link when it is one, as text otherwise.", Nested: ""},
 		{Key: "notice", Type: "string", Doc: "Notice is an optional line of your own, for whatever the reader needs to\nknow that this page cannot guess.", Nested: ""},
@@ -306,6 +330,7 @@ var Reference = []Block{
 		{Key: "require_capabilities", Type: "bool", Doc: "RequireCapabilities refuses a request offering tools, or sending\nimages, to a model whose catalog entry does not say it takes them.", Nested: ""},
 		{Key: "validate_structured", Type: "bool", Doc: "ValidateStructured checks an answer against the JSON schema the\nrequest asked for in response_format: one that does not fit is\nstopped — or, streamed a sentence at a time, flagged, having been\nsent already.", Nested: ""},
 		{Key: "judge", Type: "*JudgePolicy", Doc: "Judge has a judge model on the cluster read the text too.", Nested: "guard.JudgePolicy"},
+		{Key: "services", Type: "[]Service", Doc: "Services are outside guardrails asked over HTTP, beside the judge.", Nested: ""},
 		{Key: "stream", Type: "string", Doc: "Stream: sentence scans a streamed answer a sentence at a time,\nreleasing each once it passes; buffer_all holds the whole answer\nuntil it is judged. Empty is sentence.", Nested: ""},
 	}},
 	{Package: "config", Name: "Prefix", Title: "Prefix", Doc: "", Fields: []Field{
@@ -383,6 +408,7 @@ var Reference = []Block{
 		{Key: "injection", Type: "string", Doc: "Injection is what is done on signs of prompt injection: flag or block.", Nested: ""},
 		{Key: "deny", Type: "[]Pattern", Doc: "Deny are patterns, matched against the text as a model reads it.", Nested: ""},
 		{Key: "max_chars", Type: "int", Doc: "MaxChars refuses text longer than this; zero is no limit.", Nested: ""},
+		{Key: "restore", Type: "bool", Doc: "Restore, on input, puts what redact rewrote as placeholders back into\nthe answer: the model reads \"[EMAIL_1]\", the client reads its own\naddress. Only the values that request sent, only into its answer.", Nested: ""},
 	}},
 	{Package: "config", Name: "SCIM", Title: "SCIM", Doc: "", Fields: []Field{
 		{Key: "token_file", Type: "string", Doc: "TokenFile holds the bearer token the provider sends. Empty turns\nprovisioning off.", Nested: ""},
@@ -460,6 +486,17 @@ var Reference = []Block{
 		{Key: "federation", Type: "Federation", Doc: "Federation is how this cluster takes part in federation with other\nllmgw clusters. The peers themselves are added at runtime, with\n`llmgw ctl federation add`, and kept in the store.", Nested: "config.Federation"},
 		{Key: "endpoints", Type: "[]Endpoint", Doc: "Endpoints is the phase-1 static replica list. It is only consulted when\nno model declares a profile, which is what makes a router-only smoke test\nagainst sim-engines possible without a control plane.", Nested: ""},
 	}},
+	{Package: "guard", Name: "Service", Title: "Service", Doc: "", Fields: []Field{
+		{Key: "name", Type: "string", Doc: "Name says which service decided, in the verdict's rule.", Nested: ""},
+		{Key: "url", Type: "string", Doc: "URL is POSTed the body.", Nested: ""},
+		{Key: "header_files", Type: "map[string]string", Doc: "HeaderFiles are headers whose values are read from files each time,\nso a rotated key needs no restart: {Authorization: /etc/…/lakera}.", Nested: ""},
+		{Key: "body", Type: "string", Doc: "Body is the request, a template given .Text; {{json .Text}} writes it\nas a JSON string. Empty is {\"input\": <text>}.", Nested: ""},
+		{Key: "blocked", Type: "string", Doc: "Blocked is where in the answer the service says the text is unsafe:\na dotted path, numbers for list items (results.0.flagged). True, a\nnumber above zero, a non-empty list or string other than \"false\" is\nunsafe.", Nested: ""},
+		{Key: "reason", Type: "string", Doc: "Reason is where it says why, for the verdict; optional.", Nested: ""},
+		{Key: "on", Type: "string", Doc: "On: input, output or both. Empty is input.", Nested: ""},
+		{Key: "action", Type: "string", Doc: "Action on unsafe: block or flag. Empty is block.", Nested: ""},
+		{Key: "timeout", Type: "time.Duration", Doc: "Timeout bounds one call. Empty is 3s; past it the policy's fail says.", Nested: ""},
+	}},
 	{Package: "config", Name: "ServingSettings", Title: "ServingSettings", Doc: "", Fields: []Field{
 		{Key: "stopped", Type: "bool", Doc: "Stopped takes every model down and answers every request for one with\n503 (system_stopped), leaving the control planes, agents and mirror\nrunning and every model's definition as it is: llmgw stop sets it,\nllmgw start clears it, and the models come back as their catalog\nentries ask. In force on every control plane within seconds.", Nested: ""},
 	}},
@@ -514,6 +551,7 @@ var Reference = []Block{
 	{Package: "accounting", Name: "Price", Title: "accounting.Price", Doc: "", Fields: []Field{
 		{Key: "prompt_per_1k", Type: "int64", Doc: "", Nested: ""},
 		{Key: "completion_per_1k", Type: "int64", Doc: "", Nested: ""},
+		{Key: "cached_per_1k", Type: "int64", Doc: "CachedPer1k is what a prompt token costs that the engine or provider\nread from its cache instead of computing: the part of prompt_tokens\nthe answer counts as cached_tokens. Zero charges those at\nPromptPer1k, as before there was a rate for them.", Nested: ""},
 		{Key: "per_image", Type: "int64", Doc: "PerImage is what one picture costs, for a model that makes them: a\npicture's cost is the card's time, which its prompt's few tokens do\nnot measure.", Nested: ""},
 		{Key: "per_mesh", Type: "int64", Doc: "PerMesh is what one 3D model costs: minutes of a card, which no\ntoken count measures.", Nested: ""},
 	}},
@@ -562,6 +600,7 @@ var Reference = []Block{
 	{Package: "config", Name: "Price", Title: "config.Price", Doc: "", Fields: []Field{
 		{Key: "prompt_per_1k", Type: "int64", Doc: "", Nested: ""},
 		{Key: "completion_per_1k", Type: "int64", Doc: "", Nested: ""},
+		{Key: "cached_per_1k", Type: "int64", Doc: "CachedPer1k is the cost of a prompt token the engine or provider read\nfrom its cache (the answer's cached_tokens, counted within the\nprompt's). Zero charges them at prompt_per_1k.", Nested: ""},
 		{Key: "per_image", Type: "int64", Doc: "PerImage is the cost of one picture, for a model that makes them\n(/v1/images/generations), on top of its prompt's tokens.", Nested: ""},
 		{Key: "per_mesh", Type: "int64", Doc: "PerMesh is the cost of one 3D model (/v1/3d/generations).", Nested: ""},
 	}},

internal/docgen/render.go

@@ -31,8 +31,10 @@ var Sources = []string{
 	"internal/router/admission/controller.go",
 	"internal/accounting/cost.go",
 	"internal/guard/policy.go",
+	"internal/guard/service.go",
 	"internal/guard/judge.go",
 	"internal/config/oidc.go",
+	"internal/config/ldap.go",
 	"internal/config/firewall.go",
 	"internal/config/serving.go",
 	"internal/config/presets.go",

internal/guard/gate.go

@@ -307,6 +307,15 @@ func (p *placeholders) rewrite(text string, spans []detect.Span) string {
 	return b.String()
 }
 
+// inverse is each placeholder and the value it replaced.
+func (p *placeholders) inverse() map[string]string {
+	out := make(map[string]string, len(p.byValue))
+	for v, ph := range p.byValue {
+		out[ph] = v
+	}
+	return out
+}
+
 var _ Guard = (*Gate)(nil)
 
 // Directives are the policy in force's demands of the gateway. Not in
@@ -406,6 +415,19 @@ func (g *Gate) Input(ctx context.Context, s Subject, body []byte) (Verdict, erro
 	if found && best.action == Block {
 		return done(best), nil
 	}
+	// Outside services next, then the judge: what a pattern already stopped
+	// need not wait for either.
+	if user != "" {
+		sf, serr := g.askServices(ctx, p, true, user)
+		if serr != nil && !v.Shadow {
+			v.Took = g.now().Sub(start)
+			return v, serr
+		}
+		fs = append(fs, sf...)
+		if best, found = strictest(fs); found && best.action == Block {
+			return done(best), nil
+		}
+	}
 	// The judge last: what a pattern already stopped need not wait for it.
 	if jp := p.Judge; jp != nil && jp.Input != "" && user != "" {
 		res, err := g.judger.ask(ctx, jp, user, "")
@@ -432,6 +454,9 @@ func (g *Gate) Input(ctx context.Context, s Subject, body []byte) (Verdict, erro
 			rw.seg.set(rw.text)
 			changed = true
 		}
+		if p.Input.Restore && len(rewrites) > 0 {
+			v.Restore = ph.inverse()
+		}
 		if d.pin(p.SystemPrompt, p.DropClientSystem) {
 			changed = true
 		}

internal/guard/gate_test.go

@@ -309,3 +309,27 @@ func TestAJudgesRefusalNamesTheHazard(t *testing.T) {
 		t.Errorf("%q", got)
 	}
 }
+
+func TestRestoreSaysWhichPlaceholderWasWhichOnlyWhenAsked(t *testing.T) {
+	body := []byte(`{"messages":[{"role":"user","content":"mail ayse@example.com and ayse@example.com"}]}`)
+	for _, c := range []struct {
+		mode    string
+		restore bool
+		want    int
+	}{{"enforce", true, 1}, {"enforce", false, 0}, {"shadow", true, 0}} {
+		p := Policy{Mode: c.mode, Input: Rules{PII: map[string]string{"email": "redact"}, Restore: c.restore}}
+		g := gate(t, lib(map[string]Policy{"pii": p}, "pii"))
+		v, err := g.Input(context.Background(), Subject{}, body)
+		if err != nil {
+			t.Fatal(err)
+		}
+		if len(v.Restore) != c.want {
+			t.Errorf("%s restore=%v: %v", c.mode, c.restore, v.Restore)
+		}
+		for ph, val := range v.Restore {
+			if val != "ayse@example.com" || !strings.HasPrefix(ph, "[EMAIL_") {
+				t.Errorf("%s -> %s", ph, val)
+			}
+		}
+	}
+}

internal/guard/guard.go

@@ -75,6 +75,10 @@ type Verdict struct {
 	// Shadow says the policy only records: the action is what it would have
 	// done, and nothing is done.
 	Shadow bool
+	// Restore maps each placeholder the input was given to the value it
+	// stands for, when the policy puts them back in the answer. Held for
+	// the request's life only: never stored, logged or traced.
+	Restore map[string]string
 }
 
 // Stricter reports whether v outranks w: block over redact over flag over

internal/guard/judge.go

@@ -61,6 +61,9 @@ func (j *JudgePolicy) Name() string { return strings.Join(j.Judges(), ", else ")
 // not say.
 const DefaultJudgeBudget = 250 * time.Millisecond
 
+// Budget is how long each judge is waited for.
+func (j *JudgePolicy) Budget() time.Duration { return j.budget() }
+
 func (j *JudgePolicy) budget() time.Duration {
 	if j.LatencyBudget > 0 {
 		return j.LatencyBudget

internal/guard/output.go

@@ -3,6 +3,7 @@ package guard
 import (
 	"context"
 	"encoding/json"
+	"slices"
 	"strings"
 	"sync"
 	"time"
@@ -55,6 +56,7 @@ func outputWatched(p Policy) bool {
 	o, t := p.Output, p.Tools
 	return len(o.PII) > 0 || o.Injection != "" || len(o.Deny) > 0 || o.MaxChars > 0 || p.ValidateStructured ||
 		p.Judge != nil && p.Judge.Output != "" ||
+		slices.ContainsFunc(p.Services, Service.onOutput) ||
 		len(t.Allow) > 0 || len(t.Deny) > 0 || t.MaxCallsPerTurn > 0 || t.ValidateArguments
 }
 
@@ -82,6 +84,7 @@ type outFilter struct {
 	full    strings.Builder // the whole answer, when it must fit a schema
 	user    string          // the request's last user turn, for the judge
 	sofar   strings.Builder // the answer so far, for the judge
+	served  strings.Builder // the answer so far, for outside services, asked once at its end
 	judged  int             // how much of it the judge has read
 	v       Verdict
 	took    time.Duration
@@ -203,6 +206,21 @@ func (f *outFilter) askJudge(text string, final bool) finding {
 	return f.record([]finding{jf})
 }
 
+// askServices has the outside services read the whole answer, once, at its
+// end. One that cannot be asked under fail: closed stops it.
+func (f *outFilter) askServices(text string) finding {
+	if !slices.ContainsFunc(f.p.Services, Service.onOutput) || text == "" {
+		return finding{}
+	}
+	start := time.Now()
+	fs, err := f.g.askServices(f.ctx, f.p, false, text)
+	f.took += time.Since(start)
+	if err != nil {
+		fs = append(fs, finding{action: Block, category: "service", rule: "service:unavailable"})
+	}
+	return f.record(fs)
+}
+
 // judgeStructured checks a whole answer against the schema the request
 // asked it to fit.
 func (f *outFilter) judgeStructured(text string, enforce bool) finding {
@@ -259,6 +277,9 @@ func (f *outFilter) Buffered(body []byte) []byte {
 			if f.askJudge(t, true).action == Block {
 				blocked = true
 			}
+			if f.askServices(t).action == Block {
+				blocked = true
+			}
 			if out != t {
 				msg["content"] = rawString(out)
 				ch["message"], _ = json.Marshal(msg)
@@ -422,6 +443,12 @@ func (f *outFilter) flush(final bool) (bool, map[int]string, bool) {
 		if i == 0 && f.askJudge(head, final).action == Block {
 			blocked = true
 		}
+		if i == 0 {
+			f.served.WriteString(head)
+			if final && f.askServices(f.served.String()).action == Block {
+				blocked = true
+			}
+		}
 		if head != "" {
 			fd, r := f.judgeText(head)
 			blocked = blocked || fd.action == Block

internal/guard/policy.go

@@ -69,6 +69,8 @@ type Policy struct {
 	ValidateStructured bool `koanf:"validate_structured"`
 	// Judge has a judge model on the cluster read the text too.
 	Judge *JudgePolicy `koanf:"judge"`
+	// Services are outside guardrails asked over HTTP, beside the judge.
+	Services []Service `koanf:"services"`
 	// Stream: sentence scans a streamed answer a sentence at a time,
 	// releasing each once it passes; buffer_all holds the whole answer
 	// until it is judged. Empty is sentence.
@@ -86,6 +88,10 @@ type Rules struct {
 	Deny []Pattern `koanf:"deny"`
 	// MaxChars refuses text longer than this; zero is no limit.
 	MaxChars int `koanf:"max_chars"`
+	// Restore, on input, puts what redact rewrote as placeholders back into
+	// the answer: the model reads "[EMAIL_1]", the client reads its own
+	// address. Only the values that request sent, only into its answer.
+	Restore bool `koanf:"restore"`
 }
 
 // Pattern is a named regular expression and what its match does.
@@ -150,6 +156,11 @@ func (p Policy) validate() error {
 			return err
 		}
 	}
+	for _, sv := range p.Services {
+		if err := sv.validate(); err != nil {
+			return err
+		}
+	}
 	switch p.Tools.OnDenied {
 	case "", "strip", "block":
 	default:
@@ -209,6 +220,12 @@ func Merge(names []string, ps []Policy) Policy {
 		out.RequireCapabilities = out.RequireCapabilities || p.RequireCapabilities
 		out.ValidateStructured = out.ValidateStructured || p.ValidateStructured
 		out.Judge = mergeJudge(out.Judge, p.Judge)
+		// Every service any policy asks, each once.
+		for _, sv := range p.Services {
+			if !slices.ContainsFunc(out.Services, func(e Service) bool { return e.Name == sv.Name }) {
+				out.Services = append(out.Services, sv)
+			}
+		}
 		out.MaxOutputTokens = minSet(out.MaxOutputTokens, p.MaxOutputTokens)
 		if p.SystemPrompt != "" {
 			prompts = append(prompts, p.SystemPrompt)
@@ -253,6 +270,9 @@ func mergeRules(dst *Rules, r Rules) {
 		}
 	}
 	dst.MaxChars = minSet(dst.MaxChars, r.MaxChars)
+	// Any policy may ask: what is put back is the caller's own, to the
+	// caller, and never reaches the model either way.
+	dst.Restore = dst.Restore || r.Restore
 }
 
 func stricterOf(a, b string, order []string) string {

internal/guard/service.go

@@ -0,0 +1,226 @@
+package guard
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+	"net/url"
+	"os"
+	"strconv"
+	"strings"
+	"text/template"
+	"time"
+)
+
+// A guardrail that is someone else's service: Azure Content Safety, Lakera,
+// NeMo Guardrails, a company's own classifier. A site may be required to use
+// one; this asks it beside the patterns and the judge, and its answer counts
+// like theirs. Told only the text, never who sent it, and nothing of the
+// call is kept but its verdict.
+
+// Service is one outside guardrail, asked over HTTP.
+type Service struct {
+	// Name says which service decided, in the verdict's rule.
+	Name string `koanf:"name"`
+	// URL is POSTed the body.
+	URL string `koanf:"url"`
+	// HeaderFiles are headers whose values are read from files each time,
+	// so a rotated key needs no restart: {Authorization: /etc/…/lakera}.
+	HeaderFiles map[string]string `koanf:"header_files"`
+	// Body is the request, a template given .Text; {{json .Text}} writes it
+	// as a JSON string. Empty is {"input": <text>}.
+	Body string `koanf:"body"`
+	// Blocked is where in the answer the service says the text is unsafe:
+	// a dotted path, numbers for list items (results.0.flagged). True, a
+	// number above zero, a non-empty list or string other than "false" is
+	// unsafe.
+	Blocked string `koanf:"blocked"`
+	// Reason is where it says why, for the verdict; optional.
+	Reason string `koanf:"reason"`
+	// On: input, output or both. Empty is input.
+	On string `koanf:"on"`
+	// Action on unsafe: block or flag. Empty is block.
+	Action string `koanf:"action"`
+	// Timeout bounds one call. Empty is 3s; past it the policy's fail says.
+	Timeout time.Duration `koanf:"timeout"`
+}
+
+func (s Service) validate() error {
+	if s.Name == "" {
+		return fmt.Errorf("services: one has no name")
+	}
+	u, err := url.Parse(s.URL)
+	if err != nil || (u.Scheme != "http" && u.Scheme != "https") || u.Host == "" {
+		return fmt.Errorf("services.%s: url %q is not an http(s) address", s.Name, s.URL)
+	}
+	if s.Blocked == "" {
+		return fmt.Errorf("services.%s: blocked says where the answer says unsafe, and is empty", s.Name)
+	}
+	switch s.On {
+	case "", "input", "output", "both":
+	default:
+		return fmt.Errorf("services.%s: on is %q: input, output or both", s.Name, s.On)
+	}
+	switch s.Action {
+	case "", "block", "flag":
+	default:
+		return fmt.Errorf("services.%s: action is %q: block or flag", s.Name, s.Action)
+	}
+	if _, err := s.template(); err != nil {
+		return fmt.Errorf("services.%s: body: %w", s.Name, err)
+	}
+	return nil
+}
+
+func (s Service) onInput() bool  { return s.On == "" || s.On == "input" || s.On == "both" }
+func (s Service) onOutput() bool { return s.On == "output" || s.On == "both" }
+
+func (s Service) action() Action {
+	if s.Action == "flag" {
+		return Flag
+	}
+	return Block
+}
+
+func (s Service) template() (*template.Template, error) {
+	body := s.Body
+	if body == "" {
+		body = `{"input": {{json .Text}}}`
+	}
+	return template.New(s.Name).Funcs(template.FuncMap{"json": func(v any) (string, error) {
+		b, err := json.Marshal(v)
+		return string(b), err
+	}}).Parse(body)
+}
+
+// serviceClient asks every service; each call is bounded by its own timeout.
+var serviceClient = &http.Client{}
+
+// ask sends the service the text and says whether it found it unsafe, and why.
+func (s Service) ask(ctx context.Context, text string) (unsafe bool, reason string, err error) {
+	t, err := s.template()
+	if err != nil {
+		return false, "", err
+	}
+	var body bytes.Buffer
+	if err := t.Execute(&body, struct{ Text string }{text}); err != nil {
+		return false, "", err
+	}
+	timeout := s.Timeout
+	if timeout <= 0 {
+		timeout = 3 * time.Second
+	}
+	ctx, cancel := context.WithTimeout(ctx, timeout)
+	defer cancel()
+	req, err := http.NewRequestWithContext(ctx, http.MethodPost, s.URL, &body)
+	if err != nil {
+		return false, "", err
+	}
+	req.Header.Set("Content-Type", "application/json")
+	for h, file := range s.HeaderFiles {
+		v, err := os.ReadFile(file)
+		if err != nil {
+			return false, "", fmt.Errorf("header %s: %w", h, err)
+		}
+		req.Header.Set(h, strings.TrimSpace(string(v)))
+	}
+	resp, err := serviceClient.Do(req)
+	if err != nil {
+		return false, "", err
+	}
+	defer resp.Body.Close()
+	b, err := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
+	if err != nil {
+		return false, "", err
+	}
+	if resp.StatusCode/100 != 2 {
+		return false, "", fmt.Errorf("answered %d", resp.StatusCode)
+	}
+	var doc any
+	if err := json.Unmarshal(b, &doc); err != nil {
+		return false, "", fmt.Errorf("answer is not JSON: %w", err)
+	}
+	v, ok := at(doc, s.Blocked)
+	if !ok {
+		return false, "", fmt.Errorf("the answer has no %s", s.Blocked)
+	}
+	if s.Reason != "" {
+		if r, ok := at(doc, s.Reason); ok {
+			reason = fmt.Sprint(r)
+		}
+	}
+	return truthy(v), reason, nil
+}
+
+// at is the value at a dotted path in decoded JSON.
+func at(doc any, path string) (any, bool) {
+	cur := doc
+	for _, k := range strings.Split(path, ".") {
+		switch c := cur.(type) {
+		case map[string]any:
+			v, ok := c[k]
+			if !ok {
+				return nil, false
+			}
+			cur = v
+		case []any:
+			i, err := strconv.Atoi(k)
+			if err != nil || i < 0 || i >= len(c) {
+				return nil, false
+			}
+			cur = c[i]
+		default:
+			return nil, false
+		}
+	}
+	return cur, true
+}
+
+func truthy(v any) bool {
+	switch x := v.(type) {
+	case bool:
+		return x
+	case float64:
+		return x > 0
+	case string:
+		return x != "" && !strings.EqualFold(x, "false") && x != "0"
+	case []any:
+		return len(x) > 0
+	case map[string]any:
+		return len(x) > 0
+	}
+	return false
+}
+
+// askServices asks each service of a direction, and says what the policy
+// makes of their answers: a finding per service that found the text unsafe,
+// or could not be asked under fail: closed. failed is true when a service
+// gave no answer, for the caller to refuse under fail: closed.
+func (g *Gate) askServices(ctx context.Context, p Policy, input bool, text string) (fs []finding, failed error) {
+	for _, s := range p.Services {
+		if input && !s.onInput() || !input && !s.onOutput() || text == "" {
+			continue
+		}
+		unsafe, reason, err := s.ask(ctx, text)
+		if err != nil {
+			g.log.Warn("a guardrail service gave no verdict", "service", s.Name, "err", err)
+			if p.Fail == "closed" {
+				failed = fmt.Errorf("the guardrail service %s could not be asked: %w", s.Name, err)
+				continue
+			}
+			g.m.failOpen.Inc()
+			continue
+		}
+		if unsafe {
+			rule := "service:" + s.Name
+			if reason != "" {
+				rule += ":" + reason
+			}
+			fs = append(fs, finding{action: s.action(), category: "service", rule: rule})
+		}
+	}
+	return fs, failed
+}