v0.13.0

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

BACKLOG.md

@@ -761,6 +761,135 @@ portal is not done — the page shows; the console and ctl change.
 
 ---
 
+## E. Every setting takes effect when it is changed (2026-10-05)
+
+Asked for after a site visit: an upgrade left server.yaml without guardrails,
+a node on another subnet stayed lost until firewall_networks was edited by
+hand, and nothing in server.yaml, agent.yaml or a generated unit applies until
+somebody restarts the right service on every machine. The rule from here on:
+**a file holds only what one machine needs to start; everything else is the
+cluster's, kept in the store, changed through ctl, the console or the web,
+audited, and in force on every control plane within seconds.**
+
+### E1. Cluster settings in the store (L) — done 2026-10-05
+Live on the PoC (140, 150): imported as version 1; a guard change applied on
+both without a restart; a router change restarted the standby at once and the
+leader 30s later with /health answering throughout; rollback did the same.
+- Table `cluster_settings` (versioned YAML document, who and why). The
+  cluster-wide part of server.yaml lives there; server.yaml keeps the
+  per-machine keys only (listeners, TLS paths, store DSN, secrets, tracing,
+  monitoring, mirror.api/tls, backup_dir/key, endpoints).
+- First start against a store with no document imports the file's cluster
+  keys (and the baseline guard policy when there is none); after that those
+  keys in a file are ignored, with a warning.
+- Each control plane polls the document's version and its own server.yaml's
+  hash. A change is validated, then applied: `guard` live, the rest by a
+  graceful restart of the process (standbys first, the leader last), which
+  systemd brings back. No "restart now?" anywhere.
+- `llmgw ctl config show|history|get|set|unset|edit|rollback`; the console's
+  settings screen edits the store for cluster keys; admin API
+  `GET/PUT /api/v1/settings`, `/history`, `/rollback`.
+- A standby that becomes leader re-reads the catalog, engine choices and
+  facility state from the store (it kept what it read at start).
+- Retires `appendGuardIfMissing` (server init): the import does it.
+
+### E2. Generated files from settings (L) — firewall on/off done 2026-10-06
+`firewall.enabled` in the cluster's settings reaches nodes over the agent
+stream and control planes through llmgw-firewall-sync.path. Left: the mirror
+machine and store members, the allowed set from membership, the other files.
+
+Every machine runs the agent (root) with roles — node, control plane, mirror,
+store member — and the agent renders and applies what was written once by
+init: the llmgw nft table, the mirror's unit flags, keepalived, etcd and
+Patroni configuration. The firewall's allowed set is the cluster's membership
+(every node, control plane, mirror and store member) plus
+`firewall.extra` from the store; 7443 stays open (mTLS). A node added or
+removed changes every machine's rules. Retires `firewall.Admit` and
+`firewall_networks` in server.yaml (imported into the store).
+
+### E3. agent.yaml down to bootstrap (M)
+agent.yaml keeps the control planes' addresses, its certificate and its id.
+Labels, fabrics, engines, heartbeat and supervisor settings move to the
+store per node and reach the agent over its stream; a change to the file is
+noticed and the agent restarts itself.
+
+### E4. Web settings page (S)
+The same document, edited on /admin/settings, for whoever has no terminal.
+
+---
+
+## F. Company documents: knowledge bases (RAG), then adapters (2026-10-08)
+
+A company wants a model to answer from its own documents. Retrieval is the
+tool for *knowledge* (sources shown, re-indexed in minutes, access per
+tenant); LoRA training is for *behaviour* (vocabulary, format) and has no
+access control — whatever goes into weights reaches everyone who may use the
+adapter. RAG first. Decisions taken: access per collection (no per-document
+ACL yet); a personal key or session names a project or team and gets that
+chain only, never the union of its person's memberships; PostgreSQL store
+only, off on SQLite with a sentence saying why.
+
+### F1. Store: collections, grants, chunks (M)
+- `pgvector` ≥ 0.8 installed on every Patroni member by `server init`
+  (`internal/cli/serverinit_postgres.go`), `CREATE EXTENSION` in the
+  migration; the SQLite migration has no tables, and the store reports the
+  feature unavailable.
+- Tables: `kb_collections` (id, org_id, owner `team:…`/`project:…`/`user:…`,
+  name, embed_model, dims, created_at), `kb_grants` (scope, scope_id,
+  collection, perm read|write — like `guard_attachments`), `kb_documents`
+  (name, digest, bytes, state, chunk count), `kb_chunks` (org_id,
+  collection, doc_id, ord, body, tsv, embedding) `PARTITION BY LIST
+  (org_id)`, a partition per organization, HNSW and GIN indexes per
+  partition.
+- Row-level security on `kb_chunks` against `current_setting('llmgw.org')`
+  and `llmgw.collections`, retrieval through its own role.
+- Rules, refused before anything is written: a grant never crosses
+  organizations; owner and grants resolve to existing scopes; deleting a
+  project/team/org and `users forget` delete their collections (`forget.go`).
+- Done: store tests on PostgreSQL, including an RLS test that a query
+  without the settings returns no rows.
+
+### F2. Ingest (M)
+- `llmgw ctl kb create|list|delete|grant|revoke|docs`, `ctl kb add
+  <collection> <files…>`, admin API under `/api/v1/kb`, audit events.
+- Text from PDF (text layer), DOCX, HTML, Markdown, plain text — in Go, no
+  Python. Split on headings/paragraphs, ~500 tokens with overlap. Embedded
+  through the router at the bronze band, accounted to the collection's
+  owner. One transaction per document; same digest skipped, changed digest
+  replaces. OCR out of scope.
+- Console: a Knowledge screen (collections, documents, grants; add, grant,
+  revoke, delete); `docs/console.md`, `docs/knowledge.md`.
+
+### F3. Retrieval on the request path (M)
+- `llmgw.knowledge: [collections]` in the body (stripped before the
+  engine), or default collections per project/key.
+- Allowed set from the caller's chain (§7.9.2), cached like limits; grants
+  add up down the tree. Personal keys and sessions need `x-llmgw-project`
+  (or the playground's picker) naming a scope the person is a member of.
+- A collection not allowed answers 404, exactly as a missing one.
+- Hybrid search with the filter inside the query (`hnsw.iterative_scan`,
+  `WHERE org_id = $1 AND collection = ANY($2)`), full-text arm merged by
+  reciprocal rank, `/v1/rerank` when a reranker is in the catalog. Top
+  chunks into a system message with document and position;
+  `x-llmgw-sources` lists them.
+- Leaks closed: answer-cache key includes the allowed set; per-organization
+  `cache_salt` on RAG requests; injected passages under `no_store` and the
+  guard's prompt-injection check; returned doc ids in the audit.
+- Done: router tests that tenant B never receives tenant A's chunks
+  (including via cache), that a personal key without a project is refused,
+  metrics (retrieval latency, chunks returned), docs.
+
+### F4. Adapter training (L, after F1–F3)
+A run-to-completion job placed by the control plane like a replica and
+supervised by the agent like an engine, in a Python environment the agent
+installs (TRL + PEFT, QLoRA). Data: Q/A pairs generated from a collection by
+a cluster model through `/v1/batches`, reviewed by people. Output to the
+mirror, into the model's `adapters`, through canary and a quality suite.
+Runs when the scaling schedule frees a GPU; 8–14B dense bases on one card.
+Never train on a collection some users of the adapter may not read.
+
+---
+
 ## D. Facts worth knowing before starting
 
 - Admin API: `127.0.0.1:9090`, no auth, `ctlClient` in `internal/cli/ctl.go`.

CHANGELOG.md

@@ -16,12 +16,237 @@ is only released once its **exit criterion** is met, not when its code lands.
 | v0.9.0 | — | Managing people and the tenancy tree | released |
 | v0.10.0 | — | InfiniBand fabrics and NVLink, doctor, node add, engines installed by the nodes | released |
 | v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released |
+| v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released |
+| v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
 | v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
 
 Until v1.0.0 the minor version is the breaking one: configuration formats and Go
 APIs change between minors without a deprecation period.
 
-## Unreleased
+## v0.13.0 — 2026-10-08
+
+The whole system stopped, started and restarted from one command, a
+guardrails judge that no longer refuses ordinary coding requests, and the
+rest of the field report from v0.10.6.
+
+**Upgrading.** A cluster keeps the judge its policies name: one installed
+before this release goes on asking Llama Guard 3 1B until `llmgw ctl guard
+judge use qwen3guard-0.6b`, which the leader then fetches and serves by
+itself, and which replaces the policies' Llama Guard category codes.
+
+### Added
+
+- **A judge on a GPU and one on CPUs together**: a policy's `judge.models`
+  is a list asked in turn, the first that answers giving the verdict; the
+  next is asked when one has no replica serving, fails, or misses its
+  `latency_budget`. `llmgw ctl guard judge use qwen3guard-4b
+  qwen3guard-0.6b` sets the order for every policy with a judge, and
+  `guard judge status` says which judge stands in for which. A judge on a
+  GPU is served as a `standard` model, so it takes a card only from what
+  may give one up.
+- **A judge a policy names is fetched if it is missing**: the leader
+  fetches the weights of a judge llmgw knows (`qwen3guard-0.6b`, `-4b`,
+  `-8b`, `llama-guard-3-1b`, `-8b`) into the mirror and serves it, when a
+  policy names it and the catalog does not have it — and again if it is
+  removed. `guard judge enable --gpu` serves Qwen3Guard-Gen 4B on the class
+  with the most free cards.
+- **The whole system stops, starts and restarts from one command**: `llmgw
+  stop` takes every model down and answers every request for one with 503
+  (`system_stopped`), keeping the control planes, agents and mirror running
+  and no model's definition changed; `llmgw start` brings every model back.
+  It is the cluster's setting `serving.stopped`, live on every control plane.
+  `llmgw restart` restarts every node's agent and every control plane, the
+  leader last, with the engines serving throughout. `llmgw status` says
+  whether the system serves. One machine's services are `llmgw service
+  status|start|stop|restart`.
+- **A replica says what it computes on**: `ctl replicas` (ON), the console
+  and the admin API show `CUDA` with the GPUs its processes hold memory on
+  and how much, as nvidia-smi sees them, or `CPU` — for every engine, kept by
+  the agent across its own and the control plane's restarts. llama.cpp's
+  layer count is added when it says it, which it does only at `-lv 4`.
+- **`ctl engine-check llamacpp` computes a token on the GPU**: a small GGUF
+  (`--model`, or the smallest in the weights cache) is loaded with every layer
+  on the GPU at `-lv 4` and asked for one token; a build whose CUDA backend
+  does not load fails, by llama.cpp's own account of its layers. With no
+  `--bin` it checks the build the agent installed.
+- **Engine versions and compute capability on nodes**: `ctl nodes` shows
+  `llamacpp b10991` and `6000-24g (7.5)`; the admin API gives
+  `engine_versions` and `compute_capabilities`. A profile's `engine_version`
+  restricts it to nodes whose engine version starts with it, and
+  explain-placement says which version a node has, or which engines.
+- **`ctl restart --max-surge N` and `--max-unavailable N`**: that many of a
+  model's replicas restart at once, beside themselves or stopped first.
+- **Multi-node readiness apart from the node's**: doctor gives both, with
+  fabric findings in a section of their own; `node add` says the multi-node
+  one; `ctl nodes` shows a declared fabric's adapters that are down.
+- **A forgotten password is set through a link**: `/forgot` mails one when
+  the cluster's `mail` settings say how (SMTP, sender, the public
+  `base_url` it points at), saying the same whether or not the address has an
+  account; `/reset` sets the password once and closes every session. Without
+  mail, `llmgw users reset-link` makes the link for an operator to hand over.
+
+### Fixed
+
+- **A replica restarted in place keeps its room**: the replica was stopped
+  and the pass went on to other models before placing its replacement, so a
+  model waiting for room took the freed cards. On the reference cluster a
+  GPU judge waiting for the only card took it from the model being
+  restarted there. The replacement is now placed in the same step.
+- **The judge no longer refuses ordinary coding requests**: Llama Guard 3
+  1B, the judge `server init` installed, could not tell "continue", "devam
+  et" or "how do I kill all child processes in bash" (0.51 to 0.92 unsafe)
+  from harmful requests (0.97 to 1.00), at any quantisation. The judge is
+  now Qwen3Guard-Gen 0.6B (format `qwen3guard`), which gave the same
+  ordinary requests 0.15 at the most and the harmful ones 1.00, on CPUs,
+  in 0.8 GB. Its third verdict, controversial, counts as safe unless the
+  policy says `controversial: unsafe`. `guard judge use qwen3guard-0.6b`
+  moves an installed cluster to it, and replaces Llama Guard's category
+  codes, which Qwen3Guard never names, with the baseline's.
+- **A long message is judged rather than refused**: a judge read the whole
+  last user message, and one with a file pasted into it overflowed the
+  judge's context, so it gave no verdict and `fail: closed` refused the
+  request. It now reads the start and the end, about 6000 characters.
+- **A replica no longer stays "starting" while it serves after a restart**:
+  an agent that registered before its adopted engines had answered their
+  health check never said they had, so the control plane kept them starting
+  and routed nothing to them — on the reference cluster, after `llmgw
+  restart`, the only replica of a model. The agent now reports each shard
+  the registration called not ready once it is.
+- **A failover no longer replaces healthy replicas**: a standby that took
+  over placed at once, before the agents had reconnected and said what runs,
+  so it started a second copy of replicas it did not know of and then drained
+  the healthy ones as surplus — on the reference cluster the guardrails'
+  judge and a model's replica, within twenty seconds of a leader restarting.
+  Placement now waits after a takeover as it does at start.
+
+### Changed
+
+- **A GPU placement llama.cpp says it cannot compute on is refused** when it
+  warns that it found no usable GPU, even where nvidia-smi cannot be asked;
+  where it can, holding no memory on its GPUs refuses it as before.
+
+## v0.12.2 — 2026-10-06
+
+The guardrails turned off and on as a whole, and a judge that refuses
+ordinary requests explained: the hazard it saw is told to the client and
+kept with the request, and `llmgw ctl guard judge ask` shows what it says of
+any text. Nothing to do to upgrade; the store gains a column.
+
+### Added
+
+- **The guardrails turn off and on as a whole**: `llmgw ctl guard off` /
+  `on`, which is `guard.enabled` in the cluster's settings — in force within
+  seconds, the policies and their attachments kept. `ctl status`, `ctl guard
+  list` and the guardrails page say when they are off.
+- **`llmgw ctl guard judge ask TEXT`** has the judge read a text and shows
+  its answer as given and as read — for finding out why ordinary requests
+  are refused. Nothing of the text is kept, not even in the audit.
+
+### Changed
+
+- **A judge's refusal names the hazard**: the client is told `(judge:
+  S6,Specialized Advice)` rather than `(judge)`, and the verdict is kept with
+  the request (`guard_rule`): `ctl request`, the server's log and the
+  guardrails page show it. Llama Guard 3's codes come with their names,
+  S9 Indiscriminate Weapons, without a categories list.
+
+## v0.12.1 — 2026-10-06
+
+The guardrails' judge, served and watched without server init: a judge
+missing, served as the wrong kind of model, or not answering is said on
+`ctl status` and the console, and `llmgw ctl guard judge enable` puts it
+right. Nothing to do to upgrade.
+
+### Added
+
+- **`llmgw ctl guard judge status` and `enable`**: each judge model the
+  policies ask, whether it is in the catalog, as a judge, with replicas
+  ready and answering — and serving the missing one, Llama Guard 3 1B on
+  CPUs under the name the policies give it, with `--replace` for a model of
+  that name that is not a judge. No more `server init` to put a judge back.
+  Also **Do → Serve the guardrails' judge** in the console.
+- **A missing or broken judge is said where it is looked for**: `llmgw ctl
+  status` and the console's Overview list each judge with a problem — not in
+  the catalog, not a judge, no replica ready, or failing to answer — and
+  what that does to requests (refused or unjudged).
+
+### Fixed
+
+- **A Llama Guard file served from the mirror page is the judge**: it was
+  added as a model clients use, under whatever name the form had, so the
+  policies never found it and refused everything they judged
+  (`guard_unavailable`). It is now added as a judge, under the name the
+  policies ask.
+- **`ctl` says so when the API answers with a page**: a command a control
+  plane does not know — the leader on an older release, through a standby —
+  failed with "invalid character '<'"; it now says which and why.
+
+## v0.12.0 — 2026-10-06
+
+A minor version, because where the configuration lives changes: everything in
+server.yaml that is not one machine's own is now the cluster's settings, kept
+in the store, the same on every control plane and in force within seconds of
+a change. Nothing needs doing by hand to upgrade. The first control plane to
+start on v0.12.0 brings its server.yaml's cluster keys into the store, with
+the baseline guard policy when the file had none. From then on, change them
+with `llmgw ctl config` or the console's Settings, not in server.yaml, whose
+cluster keys are ignored, with a warning. Upgrade every control plane (`llmgw
+ctl upgrade` does, the leader last): until the leader runs v0.12.0,
+`llmgw ctl config` through a standby is refused. Nodes need v0.12.0 to follow
+the firewall switch.
+
+### Changed
+
+- **The cluster's settings live in the store, and every change is in force
+  within seconds.** Everything in server.yaml that is not one machine's own
+  (guardrails, limits and prices, routing, engines, notifications, the
+  mirror's policy, declared models, offline, federation, facility budgets) is
+  a versioned document in the store, the same on every control plane.
+  `llmgw ctl config` shows, sets, edits, lists and rolls it back, and the
+  console's Settings screen edits it. A change is checked, kept with who made
+  it and why, and applied: guardrails live, anything else by every control
+  plane starting again on it, standbys first and the leader last. A change to
+  a machine's own keys in its server.yaml is noticed and applied the same way.
+  No "restart now?" any more, and no second copy to keep in step. The first
+  start against a store brings in the file's cluster keys, with the baseline
+  guard policy when the file had none, so an upgraded cluster gets its
+  guardrails; after that, those keys in a server.yaml are ignored, with a
+  warning.
+
+### Added
+
+- **llmgw's firewall can be turned off and on for the whole cluster**:
+  `llmgw ctl config set firewall.enabled false`. Every node's agent takes its
+  table out within seconds, and each control plane's new
+  `llmgw-firewall-sync.path` does the same there; `true` puts the rules back
+  as they were. `firewall_networks` changes reach the nodes the same way.
+- **`llmgw wipe`** removes llmgw from a machine — services, firewall table,
+  configuration, the authority, secrets, state, users — after listing it and
+  asking for the machine's name; `--models`, `--store` and `--all` take
+  weights, the store's database here, backups and the binary too. Also
+  **Do → Wipe this machine** in the console.
+
+### Fixed
+
+- **The mirror is given an address the nodes reach**: `server init` and
+  `mirror init` took the machine's first address for the mirror's `--base`,
+  which on a machine with Docker could be 172.17.0.1 — a node then asked its
+  own bridge for every model and was refused. They now take the source of
+  the default route, else the first address not on a container's or a
+  bridge's interface. And `node add` has the node reach the mirror's files
+  before it is added: one that cannot fails the checks, naming the address
+  and what to change.
+- **A standby that becomes the leader reads the store again**: models added
+  or removed, engine builds chosen and domains held through the leader before
+  it, rather than placing on what it read when it started.
+- **A node on another network joins**: `node add` set up the node's firewall
+  but left the control plane's to the networks it was installed with, so a node
+  on another subnet was installed and then never heard from (lost). It now lets
+  the node in on the control plane it runs on — its subnet, or its address
+  when a router is between them — and says so; a network already let in
+  changes nothing.
+
+## v0.11.4 — 2026-10-05
 
 Two control planes, run in anger on the reference cluster.
 
@@ -53,8 +278,24 @@ Two control planes, run in anger on the reference cluster.
   dashboard as well as the API; a proxy left pointing at a standby showed an
   empty cluster. The leader trusts the control planes' `X-Forwarded-For`.
 
+- `ctl model serve --tools` says in the catalog that a model makes tool
+  calls, for agents and editors.
+
 ### Fixed
 
+- **Continue's agent did nothing.** The configuration the portal writes told
+  Continue every model uses tools, and nothing of its context: a model whose
+  calls came back as text stalled the agent, and prompts over a CPU
+  replica's 4096 tokens were refused. It now says `tool_use` only for a
+  model whose catalog says tools, and the context and longest answer the
+  gateway allows.
+- `ctl model serve` gave llama.cpp four slots whatever was asked: a
+  `--context-length 32768 --ctx-pool 32768` model had slots of 8192 and
+  refused what the gateway admitted. Slots are now what the pool holds of
+  the context. The longest answer grows with the context (a quarter, from
+  1024 to 8192) rather than being 1024 for every model.
+- `ctl upgrade` upgrades the leader last wherever it is run; run on the
+  standby, it upgraded the leader first and the leadership moved twice.
 - **Failover took 40 seconds; it takes a few.** A stopping control plane held
   every agent's stream for the thirty seconds of a graceful shutdown; it now
   gives up its lease first and lets the agents go at once.

api/proto/llmgw/v1/agent.proto

@@ -71,6 +71,8 @@ message ControlMessage {
     FabricTestRequest fabric_test = 9;
     UpgradeAgent upgrade = 10;
     ControlPlanes control_planes = 11;
+    Firewall firewall = 13;
+    RestartAgent restart_agent = 14;
   }
   // trace is the W3C trace context (traceparent, and tracestate when there
   // is one) of the control plane's span that sent this command, so what the
@@ -80,6 +82,27 @@ message ControlMessage {
   map<string, string> trace = 12;
 }
 
+// RestartAgent has the agent start again, as llmgw restart asks of every
+// node: its engines keep serving and are taken over again.
+message RestartAgent {
+  string reason = 1;
+}
+
+// Firewall is whether llmgw's firewall is kept on the node — its nftables
+// table, which lets the cluster's networks alone reach the engines' ports and
+// the node exporter — sent when an agent registers and whenever the
+// cluster's settings change it (firewall.enabled).
+message Firewall {
+  bool enabled = 1;
+  // networks are let in besides those the node reaches the control planes
+  // on: firewall_networks, networks or single addresses.
+  repeated string networks = 2;
+  // engine_ports_from and engine_ports_to are the engines' ports
+  // (control.port_range).
+  int32 engine_ports_from = 3;
+  int32 engine_ports_to = 4;
+}
+
 // ControlPlanes is every control plane an agent may dial, sent by the leader
 // when an agent registers and whenever the list changes. The agent keeps it
 // beside its configuration, so a second control plane added after the node
@@ -250,6 +273,11 @@ message RunningShard {
   // restart could not be compared with the catalog — which is exactly when the
   // catalog is most likely to have been edited.
   string profile_fp = 15;
+
+  // compute is where its engine said it put the model when it became ready,
+  // kept by the agent across its own restarts: a control plane that takes
+  // over, or restarts, learns it here.
+  Compute compute = 16;
 }
 
 message GPU {
@@ -589,6 +617,18 @@ message ProvisionStatus {
   string detail = 5;
 }
 
+// Compute is where an engine runs its model, as it said when it loaded it.
+message Compute {
+  // backend is "CUDA", "Vulkan", "Metal" or "CPU".
+  string backend = 1;
+  // devices are the devices it named: "CUDA0: Quadro RTX 6000 (7.5)".
+  repeated string devices = 2;
+  // layers_offloaded of layers_total went to the devices; both 0 when the
+  // engine did not say.
+  int32 layers_offloaded = 3;
+  int32 layers_total = 4;
+}
+
 // ShardStatus reports a shard's lifecycle transition.
 message ShardStatus {
   string node_id = 1;
@@ -610,6 +650,10 @@ message ShardStatus {
   // going to look.
   FetchProgress fetch = 8;
 
+  // compute is what the engine says it computes on, read from its own output
+  // as it became ready: sent with STATE_READY when the engine says.
+  Compute compute = 9;
+
   enum State {
     STATE_UNSPECIFIED = 0;
     STATE_PENDING = 1;

docs/admin-api.openapi.json

@@ -17,6 +17,9 @@
     }
   ],
   "tags": [
+    {
+      "name": "agents"
+    },
     {
       "name": "audit"
     },
@@ -83,6 +86,9 @@
     {
       "name": "restart"
     },
+    {
+      "name": "settings"
+    },
     {
       "name": "summary"
     },
@@ -97,6 +103,38 @@
     }
   ],
   "paths": {
+    "/api/v1/agents/restart": {
+      "post": {
+        "operationId": "postAgentsRestart",
+        "summary": "Have every node's agent start again; their engines keep serving",
+        "tags": [
+          "agents"
+        ],
+        "parameters": [
+          {
+            "$ref": "#/components/parameters/Actor"
+          }
+        ],
+        "responses": {
+          "200": {
+            "description": "OK",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/AgentRestartView"
+                }
+              }
+            }
+          },
+          "401": {
+            "$ref": "#/components/responses/Unauthorized"
+          },
+          "default": {
+            "$ref": "#/components/responses/Error"
+          }
+        }
+      }
+    },
     "/api/v1/audit": {
       "get": {
         "operationId": "getAudit",
@@ -891,6 +929,78 @@
         }
       }
     },
+    "/api/v1/guard/judges": {
+      "get": {
+        "operationId": "getGuardJudges",
+        "summary": "The judge models the policies ask, and what is wrong with each",
+        "tags": [
+          "guard"
+        ],
+        "responses": {
+          "200": {
+            "description": "OK",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "type": "array",
+                  "items": {
+                    "$ref": "#/components/schemas/JudgeView"
+                  }
+                }
+              }
+            }
+          },
+          "401": {
+            "$ref": "#/components/responses/Unauthorized"
+          },
+          "default": {
+            "$ref": "#/components/responses/Error"
+          }
+        }
+      }
+    },
+    "/api/v1/guard/judges/ask": {
+      "post": {
+        "operationId": "postGuardJudgesAsk",
+        "summary": "Have a judge read a text and say what it answered; nothing is kept",
+        "tags": [
+          "guard"
+        ],
+        "parameters": [
+          {
+            "$ref": "#/components/parameters/Actor"
+          }
+        ],
+        "requestBody": {
+          "required": true,
+          "content": {
+            "application/json": {
+              "schema": {
+                "$ref": "#/components/schemas/JudgeAskRequest"
+              }
+            }
+          }
+        },
+        "responses": {
+          "200": {
+            "description": "OK",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/JudgeAskView"
+                }
+              }
+            }
+          },
+          "401": {
+            "$ref": "#/components/responses/Unauthorized"
+          },
+          "default": {
+            "$ref": "#/components/responses/Error"
+          }
+        }
+      }
+    },
     "/api/v1/jobs": {
       "get": {
         "operationId": "getJobs",
@@ -1156,7 +1266,7 @@
       "delete": {
         "operationId": "deleteModelsName",
         "summary": "Remove a model added at runtime",
-        "description": "A model declared in configuration is refused: it would come back on the next restart.",
+        "description": "A model declared in the cluster's settings is refused: it is changed there.",
         "tags": [
           "models"
         ],
@@ -2963,6 +3073,165 @@
         }
       }
     },
+    "/api/v1/settings": {
+      "get": {
+        "operationId": "getSettings",
+        "summary": "The cluster's settings in force, or one earlier version",
+        "tags": [
+          "settings"
+        ],
+        "parameters": [
+          {
+            "name": "version",
+            "in": "query",
+            "description": "An earlier version; the newest when absent.",
+            "schema": {
+              "type": "string"
+            }
+          }
+        ],
+        "responses": {
+          "200": {
+            "description": "OK",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/SettingsView"
+                }
+              }
+            }
+          },
+          "401": {
+            "$ref": "#/components/responses/Unauthorized"
+          },
+          "default": {
+            "$ref": "#/components/responses/Error"
+          }
+        }
+      },
+      "put": {
+        "operationId": "putSettings",
+        "summary": "Change the cluster's settings: in force on every control plane within seconds",
+        "tags": [
+          "settings"
+        ],
+        "parameters": [
+          {
+            "$ref": "#/components/parameters/Actor"
+          }
+        ],
+        "requestBody": {
+          "required": true,
+          "content": {
+            "application/json": {
+              "schema": {
+                "$ref": "#/components/schemas/SettingsRequest"
+              }
+            }
+          }
+        },
+        "responses": {
+          "200": {
+            "description": "OK",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/SettingsView"
+                }
+              }
+            }
+          },
+          "401": {
+            "$ref": "#/components/responses/Unauthorized"
+          },
+          "default": {
+            "$ref": "#/components/responses/Error"
+          }
+        }
+      }
+    },
+    "/api/v1/settings/history": {
+      "get": {
+        "operationId": "getSettingsHistory",
+        "summary": "Versions of the cluster's settings, newest first",
+        "tags": [
+          "settings"
+        ],
+        "parameters": [
+          {
+            "name": "limit",
+            "in": "query",
+            "description": "How many; 20 when absent.",
+            "schema": {
+              "type": "string"
+            }
+          }
+        ],
+        "responses": {
+          "200": {
+            "description": "OK",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "type": "array",
+                  "items": {
+                    "$ref": "#/components/schemas/SettingsView"
+                  }
+                }
+              }
+            }
+          },
+          "401": {
+            "$ref": "#/components/responses/Unauthorized"
+          },
+          "default": {
+            "$ref": "#/components/responses/Error"
+          }
+        }
+      }
+    },
+    "/api/v1/settings/rollback": {
+      "post": {
+        "operationId": "postSettingsRollback",
+        "summary": "Put an earlier version of the cluster's settings back in force",
+        "tags": [
+          "settings"
+        ],
+        "parameters": [
+          {
+            "$ref": "#/components/parameters/Actor"
+          }
+        ],
+        "requestBody": {
+          "required": true,
+          "content": {
+            "application/json": {
+              "schema": {
+                "$ref": "#/components/schemas/SettingsRollbackRequest"
+              }
+            }
+          }
+        },
+        "responses": {
+          "200": {
+            "description": "OK",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/SettingsView"
+                }
+              }
+            }
+          },
+          "401": {
+            "$ref": "#/components/responses/Unauthorized"
+          },
+          "default": {
+            "$ref": "#/components/responses/Error"
+          }
+        }
+      }
+    },
     "/api/v1/summary": {
       "get": {
         "operationId": "getSummary",
@@ -3320,6 +3589,18 @@
           }
         }
       },
+      "AgentRestartView": {
+        "type": "object",
+        "description": "AgentRestartView names the nodes whose agents were told to restart.",
+        "properties": {
+          "nodes": {
+            "type": "array",
+            "items": {
+              "type": "string"
+            }
+          }
+        }
+      },
       "CanaryPercentRequest": {
         "type": "object",
         "description": "CanaryPercentRequest changes the share of a model's requests its canary takes.",
@@ -3480,6 +3761,29 @@
           }
         }
       },
+      "ComputeView": {
+        "type": "object",
+        "description": "ComputeView is where an engine runs its model, by its own account.",
+        "properties": {
+          "backend": {
+            "type": "string",
+            "description": "Backend is CUDA, Vulkan, Metal, ROCm, GPU or CPU."
+          },
+          "devices": {
+            "type": "array",
+            "items": {
+              "type": "string"
+            }
+          },
+          "layers_offloaded": {
+            "type": "integer",
+            "description": "LayersOffloaded of LayersTotal went to the devices; absent when unsaid."
+          },
+          "layers_total": {
+            "type": "integer"
+          }
+        }
+      },
       "ControlPlane": {
         "type": "object",
         "description": "ControlPlane is one control plane.",
@@ -3937,6 +4241,10 @@
           "category": {
             "type": "string"
           },
+          "rule": {
+            "type": "string",
+            "description": "Rule is what decided it within the category: a pattern's name, or a judge's verdict, \"S6,Specialized Advice\"."
+          },
           "guard_ms": {
             "type": "integer",
             "format": "int64"
@@ -3999,6 +4307,10 @@
         "type": "object",
         "description": "GuardView is the library and where its policies are attached.",
         "properties": {
+          "enabled": {
+            "type": "boolean",
+            "description": "Enabled is whether the guardrails check anything (guard.enabled)."
+          },
           "policies": {
             "type": "array",
             "items": {
@@ -4197,6 +4509,103 @@
           }
         }
       },
+      "JudgeAskRequest": {
+        "type": "object",
+        "description": "JudgeAskRequest is a text to have a judge read.",
+        "properties": {
+          "model": {
+            "type": "string",
+            "description": "Model is the judge; empty is the one the policies ask, when they ask one."
+          },
+          "text": {
+            "type": "string"
+          },
+          "answer": {
+            "type": "string",
+            "description": "Answer, when given, is read as the assistant's reply to Text."
+          }
+        }
+      },
+      "JudgeAskView": {
+        "type": "object",
+        "description": "JudgeAskView is what the judge answered.",
+        "properties": {
+          "model": {
+            "type": "string"
+          },
+          "verdict": {
+            "type": "string",
+            "description": "Verdict is safe, unsafe or controversial, as the gateway reads the answer; a policy's judge.controversial says what controversial counts as."
+          },
+          "categories": {
+            "type": "array",
+            "items": {
+              "type": "string"
+            }
+          },
+          "raw": {
+            "type": "string",
+            "description": "Raw is the judge's answer as it gave it."
+          },
+          "took_ms": {
+            "type": "integer",
+            "format": "int64"
+          }
+        }
+      },
+      "JudgeView": {
+        "type": "object",
+        "description": "JudgeView is one judge model the guardrail policies ask, and whether it can answer them.",
+        "properties": {
+          "model": {
+            "type": "string"
+          },
+          "policies": {
+            "type": "array",
+            "items": {
+              "type": "string"
+            },
+            "description": "Policies are those that ask it; Closed those that refuse what it cannot read (fail: closed)."
+          },
+          "closed": {
+            "type": "array",
+            "items": {
+              "type": "string"
+            }
+          },
+          "then": {
+            "type": "array",
+            "items": {
+              "type": "string"
+            },
+            "description": "Then are the judges its policies ask after it, when it gives no verdict: a CPU judge behind a GPU one."
+          },
+          "in_catalog": {
+            "type": "boolean",
+            "description": "InCatalog is whether a model of that name is in the catalog, and Judge whether it is there as a judge (role: guard)."
+          },
+          "judge": {
+            "type": "boolean"
+          },
+          "ready_replicas": {
+            "type": "integer"
+          },
+          "wanted_replicas": {
+            "type": "integer"
+          },
+          "last_error": {
+            "type": "string",
+            "description": "LastError is its last failure to give a verdict, if any."
+          },
+          "last_error_at": {
+            "type": "string"
+          },
+          "problem": {
+            "type": "string",
+            "description": "Problem says, in words, what is wrong and what to do; empty when nothing is."
+          }
+        }
+      },
       "Live": {
         "type": "object",
         "description": "Live is one of the cluster's adapters as its node last reported it.",
@@ -4428,6 +4837,10 @@
           "source": {
             "type": "string"
           },
+          "role": {
+            "type": "string",
+            "description": "Role is \"guard\" for a guardrails judge, empty for a model clients use."
+          },
           "min_replicas": {
             "type": "integer"
           },
@@ -4676,6 +5089,23 @@
             },
             "description": "Engines are the inference engines installed here, as the agent found them. Absent means the agent is older than the probe and said nothing, which is not the same as having none — placement treats it that way too, or a control plane upgrade would empty a fleet."
           },
+          "engine_versions": {
+            "type": "object",
+            "additionalProperties": {
+              "type": "string"
+            },
+            "description": "EngineVersions are the installed engines with their versions: llamacpp → b10991. ComputeCapabilities are its GPUs' CUDA compute capabilities, 7.0 for a V100, so an engine build's support can be read against them."
+          },
+          "multi_node": {
+            "type": "string",
+            "description": "MultiNode is whether it can take part in replicas spanning nodes: \"ready\", \"down: mlx5_1\" naming the declared adapters whose ports are not active, or empty for a node on no fabric. Its own replicas run either way."
+          },
+          "compute_capabilities": {
+            "type": "array",
+            "items": {
+              "type": "string"
+            }
+          },
           "provision": {
             "type": "array",
             "items": {
@@ -5163,6 +5593,9 @@
           "guard_category": {
             "type": "string"
           },
+          "guard_rule": {
+            "type": "string"
+          },
           "guard_ms": {
             "type": "integer",
             "format": "int64"
@@ -5221,6 +5654,13 @@
           "in_place": {
             "type": "boolean",
             "description": "InPlace stops each replica before starting its replacement, rather than starting the replacement beside it first."
+          },
+          "max_surge": {
+            "type": "integer",
+            "description": "MaxSurge is how many of a model's replicas are replaced beside themselves at once, MaxUnavailable how many are stopped first at once; both zero is one at a time, replaced beside itself."
+          },
+          "max_unavailable": {
+            "type": "integer"
           }
         }
       },
@@ -5350,6 +5790,64 @@
           }
         }
       },
+      "SettingsRequest": {
+        "type": "object",
+        "description": "SettingsRequest is a new version of the cluster's settings, made from the version Base, which must still be the newest.",
+        "properties": {
+          "base": {
+            "type": "integer",
+            "format": "int64"
+          },
+          "body": {
+            "type": "string"
+          },
+          "note": {
+            "type": "string"
+          }
+        }
+      },
+      "SettingsRollbackRequest": {
+        "type": "object",
+        "description": "SettingsRollbackRequest puts an earlier version's settings back in force, as a new version.",
+        "properties": {
+          "version": {
+            "type": "integer",
+            "format": "int64"
+          },
+          "note": {
+            "type": "string"
+          }
+        }
+      },
+      "SettingsView": {
+        "type": "object",
+        "description": "SettingsView is one version of the cluster's settings.",
+        "properties": {
+          "version": {
+            "type": "integer",
+            "format": "int64"
+          },
+          "body": {
+            "type": "string"
+          },
+          "note": {
+            "type": "string"
+          },
+          "at": {
+            "type": "string",
+            "format": "date-time"
+          },
+          "by": {
+            "type": "string"
+          },
+          "local_keys": {
+            "type": "array",
+            "items": {
+              "type": "string"
+            }
+          }
+        }
+      },
       "ShardView": {
         "type": "object",
         "description": "ShardView is one shard of a replica.",
@@ -5386,6 +5884,10 @@
           "memory_bytes": {
             "type": "integer",
             "format": "int64"
+          },
+          "compute": {
+            "$ref": "#/components/schemas/ComputeView",
+            "description": "Compute is where the engine said it put the model as it became ready: absent when it did not say."
           }
         }
       },
@@ -5416,6 +5918,10 @@
         "type": "object",
         "description": "Summary is the whole fleet in one answer.\n\nA per-node listing is the right tool for four nodes and the wrong one for two hundred: the questions that matter at that size — how full am I, what is broken, what is holding cards it is not using — are aggregates, and reading them off a long table is a chore that gets skipped. Everything here is derived from the same state the tables show, so it can never disagree with them; it is a different question asked of the same facts.",
         "properties": {
+          "version": {
+            "type": "string",
+            "description": "Version is the release the answering control plane runs: not the binary on its disk, which a manual install changes underneath it."
+          },
           "nodes": {
             "$ref": "#/components/schemas/NodeTotals"
           },
@@ -5472,6 +5978,21 @@
           "control_planes": {
             "$ref": "#/components/schemas/ControlPlanes",
             "description": "ControlPlanes is the cluster's control planes when there is more than one: which leads, which stand by, and the floating address in front."
+          },
+          "judges": {
+            "type": "array",
+            "items": {
+              "$ref": "#/components/schemas/JudgeView"
+            },
+            "description": "Judges are the guardrails' judge models with something wrong."
+          },
+          "stopped": {
+            "type": "boolean",
+            "description": "Stopped says the operators have stopped the service (llmgw stop): no model runs, and requests for one are answered 503."
+          },
+          "guard_off": {
+            "type": "boolean",
+            "description": "GuardOff says the guardrails are turned off (guard.enabled: false): no request is checked, whatever the policies say."
           }
         }
       },
@@ -5821,7 +6342,7 @@
         "properties": {
           "format": {
             "type": "string",
-            "description": "Format is llamaguard3 (Meta's Llama Guard 3: \"safe\", or \"unsafe\" and the categories), shieldgemma (Google's ShieldGemma: Yes or No), or json ({\"safe\": false, \"categories\": [\"...\"]})."
+            "description": "Format is qwen3guard (Qwen3Guard-Gen: \"Safety: Safe\", \"Unsafe\" or \"Controversial\", and the categories), llamaguard3 (Meta's Llama Guard 3: \"safe\", or \"unsafe\" and the categories), shieldgemma (Google's ShieldGemma: Yes or No), or json ({\"safe\": false, \"categories\": [\"...\"]})."
           },
           "categories": {
             "type": "array",
@@ -5999,6 +6520,10 @@
           "engine": {
             "type": "string"
           },
+          "engine_version": {
+            "type": "string",
+            "description": "EngineVersion restricts the profile to nodes whose installed engine's version starts with it: \"0.11\" takes vLLM 0.11.0 and 0.11.1, not 0.10.2. Empty takes whatever version the node has."
+          },
           "gpu_class": {
             "type": "string"
           },
@@ -6997,15 +7522,23 @@
           },
           "guard_format": {
             "type": "string",
-            "description": "GuardFormat makes it a judge model for the guardrails (role: guard): llamaguard3, shieldgemma or json. Empty serves clients."
+            "description": "GuardFormat makes it a judge model for the guardrails (role: guard): llamaguard3, qwen3guard, shieldgemma or json. Empty serves clients."
+          },
+          "class": {
+            "type": "string",
+            "description": "Class is what it may take from others and they from it when there is not room for both: critical, standard or opportunistic. Empty is the catalog's default for its role."
           },
           "ctx_pool": {
             "type": "integer",
-            "description": "CtxPool is llama.cpp's --ctx-size; ContextLength what one request may use. They default to 16384 and 4096, what four parallel slots divide."
+            "description": "CtxPool is llama.cpp's --ctx-size; ContextLength what one request may use. They default to 16384 and 4096, what four parallel slots divide; llama.cpp is given as many slots as the pool holds requests."
           },
           "context_length": {
             "type": "integer"
           },
+          "tools": {
+            "type": "boolean",
+            "description": "Tools says the model makes tool calls the engine returns as calls: it is written into the catalog, where clients and editors read it. Check a running replica with llmgw ctl tool-check before saying so."
+          },
           "cpu_cores": {
             "type": "integer",
             "description": "CPUCores is what a replica computes on; zero takes half the smallest node it fits on."

docs/clients.md

@@ -184,6 +184,15 @@ asks for, send the results back. Whether a model calls tools well is the
 model's business, not the gateway's; ask your operator which of the served
 models are set up for it.
 
+**Editors' agents need such a model.** Continue's Agent mode, Cline and
+Codex act only through tool calls: with a model whose calls come back as
+text — `<tool_call>{...}` or a JSON block in the answer, `tool_calls` empty —
+they chat but never read a file or run a command. Qwen2.5-Coder is one;
+Qwen3-Coder is not. The configuration the portal writes for Continue says
+`tool_use` only for a model whose catalog entry says tools, and gives
+Continue the model's context and longest answer, so it never sends a request
+the gateway refuses.
+
 ```python
 import json
 

docs/console.md

@@ -270,14 +270,20 @@ page's text.
 
 ## Settings
 
-**server.yaml** (on a node, **agent.yaml**) is every key of the file, as a
-tree of menus built from the configuration reference: the value set, or the
-one that applies when nothing is, its type, and what it is for. Enter changes
-a value or opens a block; on a list or map of blocks, Enter opens an entry and
-**D** removes one. **U** puts a key back to its default. **S** saves: the file
-is checked the way the service reads it before anything is written, and its
-comments are kept. The service reads it when it starts, so the console offers
-to restart it. Another control plane keeps its own copy.
+**server.yaml** (on a node, **agent.yaml**) is every key, as a tree of menus
+built from the configuration reference: the value set, or the one that
+applies when nothing is, its type, and what it is for. Enter changes a value
+or opens a block; on a list or map of blocks, Enter opens an entry and **D**
+removes one. **U** puts a key back to its default. **S** saves.
+
+On a control plane, the cluster's keys are read from and saved to the
+cluster's settings in the store — the same on every control plane, in force
+on all of them within seconds (`llmgw ctl config` does the same from a
+shell) — and this machine's own keys (listeners, certificates, the store's
+address) to its server.yaml, which the server notices and applies by itself.
+Either is checked before anything is kept, and the file's comments survive.
+On a node, agent.yaml is read when the agent starts, so the console offers
+to restart it.
 
 ## Do
 
@@ -313,6 +319,9 @@ to restart it. Another control plane keeps its own copy.
 - **Restart all replicas**, one at a time.
 - **Back up now** — the store and `/etc/llmgw` into one archive
   (`sudo llmgw server backup`).
+- **Wipe this machine** — removes llmgw's services, configuration, state and
+  users from this machine, after listing them and asking for its name
+  (`sudo llmgw wipe`; on a node too).
 - **Leave**.
 
 ## The node's console

docs/guardrails.md

@@ -4,13 +4,30 @@ Written for the operator deciding what the gateway lets through: what a
 policy can check, what it can do, and how one is attached to a team.
 
 `llmgw server init` writes a `baseline` policy that applies to every
-request, with Llama Guard 3 1B as its judge on a CPU; see
-[install.md](install.md#2-the-control-plane) for what it checks. Without a
-`guard:` section in `server.yaml` nothing is checked and nothing costs
-anything. With one, every request is judged once it is authorised and
+request, with Qwen3Guard-Gen 0.6B as its judge on CPUs; see
+[install.md](install.md#2-the-control-plane) for what it checks. The
+policies are part of the cluster's settings, kept in the store: `llmgw ctl
+config edit` (or the console's Settings) changes them, and the change is in
+force on every control plane within seconds, without a restart. A cluster
+whose settings were brought in from a server.yaml without a `guard:` section
+gets the `baseline` policy, without a judge. With no policies, nothing is
+checked and nothing costs anything. With one, every request is judged once it is authorised and
 within its limits, with its stored conversation spliced in, and before it is
 sent to a model; and the answer is judged before the client sees it.
 
+## Turning them off and on
+
+```
+llmgw ctl guard off --note "why"   # no request is checked, whatever the policies say
+llmgw ctl guard on                 # every policy checks again, as it was
+```
+
+That is `guard.enabled` in the cluster's settings, in force on every control
+plane within seconds without a restart. The policies and where they are
+attached are kept. While off, `llmgw ctl status`, `llmgw ctl guard list` and
+the guardrails page say so. To stop one policy rather than all, give it
+`mode: off`, or `mode: shadow` to have it record what it would do.
+
 ## A policy
 
 ```yaml
@@ -97,44 +114,136 @@ guard:
 
 ## A judge model
 
-Patterns find what has a shape. For what does not — violence, self-harm,
-abuse, whatever a moderation model was trained on — a policy can have a
-**judge model on the cluster** read the text:
+Patterns find what has a shape. For what does not — violence, crime,
+sexual content, self-harm, jailbreaks, whatever a moderation model was
+trained on — a policy can have a **judge model on the cluster** read the
+text.
+
+The judge must be in the catalog under the name the policy gives, as a
+judge (`role: guard`), with a replica ready. When it is not, what the
+policy judges is refused with `guard_unavailable` (`fail: closed`) or served
+unjudged (`fail: open`), and `llmgw ctl status` and the console's Overview
+say which judge and why.
+
+### Which judge
+
+| judge | runs on | size | |
+|---|---|---|---|
+| `qwen3guard-0.6b` | CPUs | 0.8 GB | the default: about 2 s a check on two threads |
+| `qwen3guard-4b` | a GPU | 2.5 GB | for a cluster with a card to spare |
+| `qwen3guard-8b` | a GPU | 4.7 GB | |
+| `llama-guard-3-1b` | CPUs | 0.9 GB | the earlier default: refuses ordinary coding requests |
+| `llama-guard-3-8b` | a GPU | 4.6 GB | |
+
+Measured on the reference cluster, as the judge's probability of "unsafe"
+for ordinary coding requests against harmful ones, Llama Guard 3 1B could
+not tell them apart: "continue" 0.51, "continue the code above" 0.89, "how
+do I kill all child processes in bash" 0.92, against 0.97 to 1.00 for
+harmful requests, at any quantisation. Qwen3Guard-Gen 0.6B gave the
+ordinary ones 0.15 at the most and the harmful ones 1.00. Its verdicts are
+safe, unsafe and **controversial**; controversial counts as safe unless the
+policy says `controversial: unsafe`.
+
+A policy that names one of these is enough: the leader fetches its weights
+into the mirror and serves it as a judge, and serves it again if it is
+removed from the catalog. Another judge is served by hand with `--repo`.
+
+```
+llmgw ctl guard judge status              # each judge the policies ask, and what is wrong with it
+llmgw ctl guard judge enable              # Qwen3Guard-Gen 0.6B on CPUs, or the judge the policies ask that is missing
+llmgw ctl guard judge enable --gpu        # Qwen3Guard-Gen 4B on a GPU
+llmgw ctl guard judge use qwen3guard-4b qwen3guard-0.6b   # the order every policy asks them in
+llmgw ctl guard judge enable --replace    # when a model of that name is there, but not as a judge
+```
+
+or **Do → Serve the guardrails' judge** in the console.
+
+### A GPU and CPUs together
+
+`judge.models` is a list asked in turn: the first judge that answers gives
+the verdict. The next is asked when one has no replica serving, fails, or
+does not answer within `latency_budget`; a judge with no replica at all is
+passed over at once. A judge on a GPU first and one on CPUs after it is
+the GPU's speed while it has its card, and the CPUs' when it does not.
+
+```yaml
+judge:
+  models: [qwen3guard-4b, qwen3guard-0.6b]
+  input: block
+  output: block
+  latency_budget: 5s           # each judge's wait
+```
+
+A judge on a GPU is served as a `standard` model, on one card: it takes a
+card only from what may give one up, and waits while every card is busy,
+the judge on CPUs standing in. A judge on CPUs is `critical`, with two
+replicas, because every judged request waits on one. `llmgw ctl guard judge
+status` says, for each judge, which is asked when it gives no verdict.
+
+### When it refuses ordinary requests
+
+A refusal by a judge names the hazard it saw — the client is told `refused
+by the gateway's policy (judge: Violent)` — and the same is kept with the
+request: `llmgw ctl request <id>`, the server's log (`guard_rule`) and the
+guardrails page show it. To see what the judge says of any text, without
+anything being kept:
+
+```
+llmgw ctl guard judge ask 'How do I sort a list in Python?'
+# qwen3guard-0.6b: safe, 1900 ms
+# it answered: "Safety: Safe\nCategories: None"
+```
+
+A judge that says `unsafe` to ordinary text is usually Llama Guard 3 1B
+(`llmgw ctl guard judge use qwen3guard-0.6b`), the wrong weights (another
+repository's, without the judge's template: `judge enable --replace` serves
+the right ones), or a policy counting every category: give its judge
+`categories`. The baseline counts Violent, Non-violent Illegal Acts, Sexual
+Content or Sexual Acts, Suicide & Self-Harm and Jailbreak; PII is the
+patterns', and copyright, unethical acts and politically sensitive topics
+refuse too much ordinary work. While finding out, `judge.input: flag`
+records instead of refusing.
+
+A judge's file served from the mirror page — Llama Guard, Qwen3Guard,
+ShieldGemma — is served as a judge too, under the name the policies ask.
+By hand, the catalog entry is:
 
 ```yaml
 # a catalog model (server.yaml models:, or ctl model add)
-- name: llama-guard-3-1b
-  source: http://mirror:8081/llama-guard-3-1b/…/Llama-Guard-3-1B-Q4_K_M.gguf
+- name: qwen3guard-0.6b
+  source: http://mirror:8081/qwen3guard-0.6b/…/Qwen3Guard-Gen-0.6B.Q8_0.gguf
   role: guard
   guard:
-    format: llamaguard3        # llamaguard3, shieldgemma or json
-    categories: [Violent Crimes, Non-Violent Crimes, Sex-Related Crimes, …]  # names for S1, S2, …
+    format: qwen3guard         # qwen3guard, llamaguard3, shieldgemma or json
   scaling: {min: 2, max: 2}
   profiles: [{name: cpu, engine: llamacpp}]
 
 # in a guard policy
 judge:
-  model: llama-guard-3-1b
+  models: [qwen3guard-0.6b]    # or model: one judge
   input: block                 # flag or block; empty does not ask about input
   output: flag
-  categories: [S1, S10]        # only these count; empty: any unsafe verdict
-  latency_budget: 250ms        # then the policy's fail: open serves (flagged), closed refuses
+  categories: [Violent, Jailbreak]   # only these count; empty: any unsafe verdict
+  controversial: safe          # or unsafe
+  latency_budget: 250ms        # each judge's wait; after the last, fail: open serves (flagged), closed refuses
 ```
 
 A model with `role: guard` is the guardrails' alone:
 
 - it is not in `/v1/models` or the chat page, and is refused (404) to
   every key, as a name and as a fallback;
-- it is `critical` by default and must always run — on demand is refused.
-  Give it two replicas on different nodes, because every judged request
-  waits on it.
+- it must always run — on demand is refused. One on CPUs is `critical`
+  and should have two replicas on different nodes, because every judged
+  request waits on it.
 
 The gateway's calls to it go past admission, accounting and the guardrails
 themselves.
 
 - **Formats:**
-  - `llamaguard3` reads Meta's "safe" / "unsafe
-S1,S10".
+  - `qwen3guard` asks in Qwen3Guard's own words and reads "Safety: Safe |
+    Unsafe | Controversial" and its "Categories:".
+  - `llamaguard3` reads Meta's "safe" / "unsafe\nS1,S10"; give the catalog
+    entry `categories` to have S1 recorded as its name.
   - `shieldgemma` asks Google's model whether the text breaks the listed
     principles (`categories`, each a sentence) and reads Yes or No.
   - `json` expects `{"safe": false, "categories": [...]}`.
@@ -144,6 +253,9 @@ S1,S10".
     is read every 600 characters and at the end, so text between two
     readings has gone out already; `stream: buffer_all` holds it until it
     is read.
+  - **A long text** is read by its start and its end (about 6000
+    characters), so a message with a file pasted into it fits the judge's
+    context rather than overflowing it and going unjudged.
 - **Cache:** the same text is not asked about twice within five minutes.
 
 ## Where a policy applies

docs/install.md

@@ -129,8 +129,9 @@ That is the whole step. In order, it:
    `nvidia-smi` finds any, on CPU otherwise — and waits for it to install its
    engine
 9. closes llmgw's internal ports to everything but the cluster's networks
-10. fetches Llama Guard 3 1B, the guardrails' judge, and runs two replicas
-    on the CPUs of nodes with room — this one, or the first that join
+10. fetches Qwen3Guard-Gen 0.6B, the guardrails' judge, and runs two
+    replicas on the CPUs of nodes with room — this one, or the first that
+    join ([guardrails.md](guardrails.md#a-judge-model) has one on a GPU too)
 11. fetches Qwen2.5 0.5B into the mirror and serves it on this node, so the
     first thing anyone opens `/chat` to find is a model that answers. A
     machine without room for it is told so within two minutes, and why.
@@ -241,8 +242,38 @@ password, change it, and issue yourself a key from the profile page.
 
 ### The configuration
 
-`/etc/llmgw/server.yaml` is yours from here on. Three things in it are worth
-knowing before you edit it:
+There are two halves, and each change to either is in force within seconds,
+on every machine it concerns, without anyone restarting anything.
+
+**The cluster's settings** — guardrails, limits and prices, routing, the
+reconciler, engines, notifications, the mirror's policy, models declared by
+hand, offline, federation, facility budgets, backups' recipients — are kept in
+the store, so every control plane runs on the same ones. Change them with
+
+```
+llmgw ctl config                         # what is in force, and its version
+llmgw ctl config set router.retries 3 --note "flaky upstream"
+llmgw ctl config edit                    # the whole document in $EDITOR
+llmgw ctl config history                 # who changed what, when, and why
+llmgw ctl config rollback 4              # back to version 4, as a new version
+```
+
+or on the console's **Settings** screen. A change is checked before it is
+kept — one that would not load is refused — and recorded in the audit log.
+Guardrails change live; anything else has every control plane start again on
+the new settings, the standbys first and the leader half a minute later, so
+one of them is always serving. Engines keep running through it.
+
+The first control plane to start against a store brings in its server.yaml's
+cluster keys — with a `baseline` guard policy, if the file had none — and
+from then on those keys in any server.yaml are not read (the server says so
+when it starts).
+
+**This machine's own keys** stay in `/etc/llmgw/server.yaml`: `listen`,
+`admin_listen`, `metrics_listen`, `agent_listen`, `agent_tls`, `auth.store`,
+`secrets`, `tracing`, `monitoring`, `mirror.api` and `mirror.tls`, the backup
+directory and key. The server notices the file change and starts again on it
+by itself. Three things are worth knowing before you edit it:
 
 - `admin_listen` is on loopback on purpose, and the server refuses any other
   address unless `admin_remote: true`. Its API under `/api` adds models and
@@ -2101,14 +2132,33 @@ loaded at boot by `llmgw-firewall.service`):
 
 "The cluster's networks" are, on each machine, the network of the interface
 it reaches the control planes (or the other store members) on — or just
-their address, when the route goes through a gateway. Anything else that
-must reach those ports, a Prometheus on another subnet or nodes behind a
-router, goes in `server.yaml` before the machines are set up:
+their address, when the route goes through a gateway. `node add` opens
+the control plane's to each node it adds when the node is on a network not
+already let in: its subnet when the control plane is on it, its address alone
+when a router is between them. Anything else that must reach those ports, a
+Prometheus on another subnet, a second control plane's or a separate mirror's
+for nodes behind a router, goes in `server.yaml`:
 
 ```yaml
 firewall_networks: [10.0.5.0/24, 192.168.1.20]
 ```
 
+**Turning it off and on.** Whether llmgw keeps this firewall is one of the
+cluster's settings:
+
+```
+llmgw ctl config set firewall.enabled false   # every machine's table taken out
+llmgw ctl config set firewall.enabled true    # and put back as it was
+```
+
+It is in force within seconds: each node's agent takes its table out or puts
+it back, and each control plane's `llmgw-firewall-sync.path` (installed by
+`server init` and `server add`; `sudo llmgw firewall sync --install` on one
+set up before) does the same there. The rules files stay, so turning it back
+on restores them. Nodes added while it is off get none. A separate mirror
+machine and the store's members keep theirs for now. `firewall_networks`
+changes reach the nodes the same way.
+
 Only those ports are touched. The table's policy is to accept, so whatever
 else the machine serves — and whatever firewall it already has — is left as
 it was; the client API (`:8080`) is not closed, being meant to be reached.
@@ -2245,6 +2295,68 @@ A cluster set up online becomes offline the same way: `bundle import` on the
 mirror and control plane, then the `offline:` block in `server.yaml` and a
 restart.
 
+## Stopping and starting the system
+
+From a control plane:
+
+```
+llmgw status                     # does it serve; its control planes, nodes, models, store
+llmgw stop --note "maintenance"  # every model taken down, every request answered 503
+llmgw start                      # every model back as its catalog entry asks
+sudo llmgw restart               # every agent and control plane started again, engines serving throughout
+```
+
+`llmgw stop` frees every GPU and answers requests for a model with 503
+(`system_stopped`, with Retry-After) until `llmgw start`. The control planes,
+agents and mirror keep running, so the console, the admin pages and `llmgw
+start` keep working, and no model's definition changes. It is the cluster's
+setting `serving.stopped`, so a restart does not undo it, and it is in the
+audit with its note. `--wait` waits until every replica is down, or every
+model has its replicas again.
+
+`llmgw restart` restarts every node's agent, then the other control planes
+over ssh (`--ssh-user`, the one sudo was run by), then this one — the leader
+last, so it hands over to a standby already restarted. Engines keep serving
+and are taken over again; nothing is placed anew.
+
+One machine's own services are `llmgw service`:
+
+```
+sudo llmgw service status
+sudo llmgw service stop          # agent, control plane, mirror — whichever this machine has
+sudo llmgw service start
+sudo llmgw service restart agent # or name one: server, agent, mirror
+```
+
+Stopping a node's agent leaves its engines serving — the agent takes them
+over again when it starts — unless `llmgw service stop --engines` stops them
+too, and the control plane then places their replicas elsewhere.
+
+## Removing llmgw from a machine
+
+```
+sudo llmgw wipe              # services, configuration, certificates, state, users
+sudo llmgw wipe --models     # and model weights, the mirror's files, engines
+sudo llmgw wipe --store      # and the store's database here (Patroni member, etcd, or local PostgreSQL)
+sudo llmgw wipe --all        # all of that, backups and the binary
+```
+
+It lists everything it will remove and asks for the machine's name (`--yes`
+does not ask), then stops llmgw's services and the engines, takes the
+firewall table out, and deletes `/etc/llmgw` — the authority and secrets
+with it — `/var/lib/llmgw`, the agent's state, the sudo grant, the
+authority from the system's trust, keepalived's configuration when llmgw
+wrote it, and the `llmgw` and `llmgw-engine` users. Packages it installed
+(PostgreSQL, Patroni, etcd, nftables, the node exporter) stay. Run it again
+to finish what failed. A node wiped is then forgotten on the control plane
+with `llmgw node remove <name>`. The console has it too: **Do → Wipe this
+machine**.
+
+This is how a machine with an older install is set up again from nothing:
+wipe it, then install as above. Wiping a control plane without `--store`
+leaves the store with the cluster's accounts, keys and settings in it, and a
+new control plane against that store takes them up.
+
 ## Kubernetes
 
 The control plane runs in Kubernetes from the Helm chart in
@@ -2259,12 +2371,13 @@ On a control plane:
 sudo --preserve-env=SSH_AUTH_SOCK llmgw ctl upgrade v0.10.4
 ```
 
-With two control planes it upgrades the other one first, over ssh as the user
-who ran sudo (`--ssh-user`), with `llmgw upgrade` there; then this machine —
+With two control planes it upgrades the standby first and the leader last,
+whichever this machine is — the other over ssh as the user who ran sudo
+(`--ssh-user`), with `llmgw upgrade` there, and this machine —
 fetching the release for its platform, checking it against the release's
 checksums, keeping the old binary as `/usr/local/bin/llmgw.bak-pre-<version>`,
-restarting `llmgw-server` and waiting for every node to reconnect. A leader
-restarted last hands over to a standby already on the release.
+restarting `llmgw-server` and waiting for every node to reconnect. The
+leader's restart hands over once, to a standby already on the release.
 
 Then the nodes, in waves: one first, then two, four, and so on up to
 `--parallel` (20) at once. A node's agent is told the version over the stream

docs/onboarding.md

@@ -379,16 +379,38 @@ is what someone gets from a borrowed laptop, and the password is what they do no
 have — and it signs every other browser out, which is the point if they are
 changing it because something went wrong.
 
-There is no self-service reset, because this gateway sends no email and a reset
-link with nowhere to send it is not a reset. When someone loses theirs:
+Someone who has forgotten theirs follows **Forgotten your password?** on the
+sign-in page. With mail set up in the cluster's settings, a link to set a new one
+is mailed to their address — the page says the same whether or not the address
+has an account, so nobody can find out who does from it:
+
+```
+llmgw ctl config set mail '{smtp: smtp.example.com:587, from: llmgw@example.com,
+  username: llmgw, password_file: /etc/llmgw/secrets/smtp, base_url: https://llm.example.com}'
+```
+
+`base_url` is where the link points, and it is never taken from the request:
+otherwise a stranger could ask for someone's link from a page of their own. A
+link works once, for an hour, and a newer one replaces it.
+
+Without mail, the page says to ask an operator, who makes the link and hands it
+over however suits:
+
+```
+llmgw users reset-link ayse@example.com --store …    # --base-url without mail.base_url
+```
+
+Or sets a temporary password to give them, and has them change it from their
+profile:
 
 ```
 llmgw users reset-password usr-k4mq… --password "a temporary passphrase" --store …
 ```
 
-Tell them out of band and have them change it from their profile. Their sessions
-are closed — if they lost the password, the reason might be that someone else has
-it. Their API keys keep working; those are separate credentials.
+Either way their sessions are closed — if they lost the password, the reason
+might be that someone else has it. Their API keys keep working; those are
+separate credentials. People who sign in through single sign-on have no password
+here, and get no link.
 
 ## Trying a model in the browser
 

docs/quick-install.md

@@ -52,7 +52,7 @@ install:
 - your administrator account. **The password is shown once. Copy it now.**
 - a model mirror, which downloads model files once for the whole cluster;
 - this machine as a first node, and a small model (Qwen2.5 0.5B) on it;
-- guardrails on every request, with a judge model (Llama Guard 3 1B, two
+- guardrails on every request, with a judge model (Qwen3Guard-Gen 0.6B, two
   replicas) on CPUs — a request it cannot check is refused, not let through — and API keys that expire within 90 days. What is hardened, and how
   to loosen it: [install.md](install.md#2-the-control-plane).
 

docs/reference.md

@@ -98,12 +98,15 @@ rather than repeated.
 | `other_watts` | `float64` | [FacilityBudget](#facilitybudget) |
 | `name` | `string` | [Federation](#federation) |
 | `interval` | `time.Duration` | [Federation](#federation) |
+| `enabled` | `bool` | [FirewallSettings](#firewallsettings) |
 | `format` | `string` | [GuardModel](#guardmodel) |
 | `categories` | `[]string` | [GuardModel](#guardmodel) |
 | `model` | `string` | [JudgePolicy](#judgepolicy) |
+| `models` | `[]string` | [JudgePolicy](#judgepolicy) |
 | `input` | `string` | [JudgePolicy](#judgepolicy) |
 | `output` | `string` | [JudgePolicy](#judgepolicy) |
 | `categories` | `[]string` | [JudgePolicy](#judgepolicy) |
+| `controversial` | `string` | [JudgePolicy](#judgepolicy) |
 | `latency_budget` | `time.Duration` | [JudgePolicy](#judgepolicy) |
 | `tokenize_timeout` | `time.Duration` | [KVIndex](#kvindex) |
 | `min_blocks` | `int` | [KVIndex](#kvindex) |
@@ -202,6 +205,7 @@ rather than repeated.
 | `files` | `[]OriginFile` | [PreviousOrigin](#previousorigin) |
 | `name` | `string` | [Profile](#profile) |
 | `engine` | `string` | [Profile](#profile) |
+| `engine_version` | `string` | [Profile](#profile) |
 | `gpu_class` | `string` | [Profile](#profile) |
 | `tp` | `int` | [Profile](#profile) |
 | `pp` | `int` | [Profile](#profile) |
@@ -287,6 +291,9 @@ rather than repeated.
 | `guard` | `guard.Config` | [Server](#server) |
 | `secrets` | `Secrets` | [Server](#server) |
 | `notify` | `notify.Config` | [Server](#server) |
+| `mail` | `mail.Config` | [Server](#server) |
+| `serving` | `ServingSettings` | [Server](#server) |
+| `firewall` | `FirewallSettings` | [Server](#server) |
 | `firewall_networks` | `[]string` | [Server](#server) |
 | `backup_recipients` | `[]string` | [Server](#server) |
 | `backup_copy_to` | `[]string` | [Server](#server) |
@@ -298,6 +305,7 @@ rather than repeated.
 | `facility` | `Facility` | [Server](#server) |
 | `federation` | `Federation` | [Server](#server) |
 | `endpoints` | `[]Endpoint` | [Server](#server) |
+| `stopped` | `bool` | [ServingSettings](#servingsettings) |
 | `bin` | `string` | [SimConfig](#simconfig) |
 | `python` | `string` | [TRTLLMConfig](#trtllmconfig) |
 | `store` | `string` | [Threads](#threads) |
@@ -370,8 +378,14 @@ rather than repeated.
 | `llamacpp` | `LlamaCppConfig` | [engines.Config](#enginesconfig) |
 | `trtllm` | `TRTLLMConfig` | [engines.Config](#enginesconfig) |
 | `sim` | `SimConfig` | [engines.Config](#enginesconfig) |
+| `enabled` | `*bool` | [guard.Config](#guardconfig) |
 | `default` | `[]string` | [guard.Config](#guardconfig) |
 | `policies` | `map[string]Policy` | [guard.Config](#guardconfig) |
+| `smtp` | `string` | [mail.Config](#mailconfig) |
+| `from` | `string` | [mail.Config](#mailconfig) |
+| `username` | `string` | [mail.Config](#mailconfig) |
+| `password_file` | `string` | [mail.Config](#mailconfig) |
+| `base_url` | `string` | [mail.Config](#mailconfig) |
 | `webhooks` | `[]Webhook` | [notify.Config](#notifyconfig) |
 | `host` | `string` | [notify.Config](#notifyconfig) |
 | `work_dir` | `string` | [supervisor.Config](#supervisorconfig) |
@@ -963,15 +977,30 @@ Interval is how often each peer is asked whether it is ready, timed,
 and what it offers. A peer that does not answer is taken out of
 routing until it does. Zero is 10s.
 
+## FirewallSettings
+
+### `enabled`
+
+`bool`
+
+Enabled keeps llmgw's internal ports — engines, node exporter, the
+agents' stream, the mirror, the store — to the cluster's networks on
+every machine. False removes llmgw's nftables table everywhere, for a
+site whose own firewall does that job; the rules are kept, and true
+puts them back. Changed in the cluster's settings, it is in force on
+every node and control plane within seconds.
+
 ## GuardModel
 
 ### `format`
 
 `string`
 
-Format is llamaguard3 (Meta's Llama Guard 3: "safe", or "unsafe" and
-the categories), shieldgemma (Google's ShieldGemma: Yes or No), or
-json ({"safe": false, "categories": ["..."]}).
+Format is qwen3guard (Qwen3Guard-Gen: "Safety: Safe", "Unsafe" or
+"Controversial", and the categories), llamaguard3 (Meta's Llama Guard
+3: "safe", or "unsafe" and the categories), shieldgemma (Google's
+ShieldGemma: Yes or No), or json ({"safe": false, "categories":
+["..."]}).
 
 ### `categories`
 
@@ -988,6 +1017,15 @@ sentence — and the names reported for llamaguard3's S1, S2, ….
 
 Model is a catalog model with role: guard.
 
+### `models`
+
+`[]string`
+
+Models are judges asked in turn, instead of Model: the first that
+answers gives the verdict. A judge on a GPU first and one on CPUs
+after it is the GPU's speed while it is there, and the CPUs' when it
+is not — busy, failed, or the node down.
+
 ### `input`
 
 `string`
@@ -1006,12 +1044,19 @@ Empty does not ask the judge about that direction.
 Categories, when set, are the only ones that count: Llama Guard's
 codes (S1, S10) or the names the model reports.
 
+### `controversial`
+
+`string`
+
+Controversial is what Qwen3Guard's third verdict counts as: safe
+(the default) or unsafe.
+
 ### `latency_budget`
 
 `time.Duration`
 
-LatencyBudget is how long the judge is waited for before the policy's
-fail applies. Zero is 250ms.
+LatencyBudget is how long each judge is waited for before the next is
+asked, or, after the last, the policy's fail applies. Zero is 250ms.
 
 ## KVIndex
 
@@ -1740,6 +1785,14 @@ before affinity is abandoned.
 
 `string`
 
+### `engine_version`
+
+`string`
+
+EngineVersion restricts the profile to nodes whose installed engine's
+version starts with it: "0.11" takes vLLM 0.11.0 and 0.11.1, not
+0.10.2. Empty takes whatever version the node has.
+
 ### `gpu_class`
 
 `string`
@@ -2441,6 +2494,32 @@ Prometheus and Alertmanager in between. Empty sends nothing.
 
 Takes a [notify.Config](#notifyconfig) block.
 
+### `mail`
+
+`mail.Config`
+
+Mail is where the gateway's few emails go out from: a link to set a
+forgotten password. Unset, none is sent and people ask an operator.
+
+Takes a [mail.Config](#mailconfig) block.
+
+### `serving`
+
+`ServingSettings`
+
+Serving stops and starts the whole cluster's service (llmgw stop, llmgw
+start).
+
+Takes a [ServingSettings](#servingsettings) block.
+
+### `firewall`
+
+`FirewallSettings`
+
+Firewall turns llmgw's firewall on every machine on or off.
+
+Takes a [FirewallSettings](#firewallsettings) block.
+
 ### `firewall_networks`
 
 `[]string`
@@ -2531,6 +2610,18 @@ Endpoints is the phase-1 static replica list. It is only consulted when
 no model declares a profile, which is what makes a router-only smoke test
 against sim-engines possible without a control plane.
 
+## ServingSettings
+
+### `stopped`
+
+`bool`
+
+Stopped takes every model down and answers every request for one with
+503 (system_stopped), leaving the control planes, agents and mirror
+running and every model's definition as it is: llmgw stop sets it,
+llmgw start clears it, and the models come back as their catalog
+entries ask. In force on every control plane within seconds.
+
 ## SimConfig
 
 ### `bin`
@@ -3044,6 +3135,13 @@ Takes a [SimConfig](#simconfig) block.
 
 ## guard.Config
 
+### `enabled`
+
+`*bool`
+
+Enabled turns every policy on or off at once: false checks nothing,
+whatever is defined or attached. Unset is on.
+
 ### `default`
 
 `[]string`
@@ -3052,6 +3150,40 @@ Takes a [SimConfig](#simconfig) block.
 
 `map[string]Policy`
 
+## mail.Config
+
+### `smtp`
+
+`string`
+
+SMTP is the relay, host:port: 587 with STARTTLS, 465 with TLS from the
+start, 25 as the relay allows. Empty sends nothing.
+
+### `from`
+
+`string`
+
+From is the sender, llmgw@example.com or "llmgw <llmgw@example.com>".
+
+### `username`
+
+`string`
+
+Username and PasswordFile log in to the relay, when it wants that.
+The password is a secret reference: a file, env:NAME or vault:path.
+
+### `password_file`
+
+`string`
+
+### `base_url`
+
+`string`
+
+BaseURL is the gateway's public address, which links in mail point
+at: https://llm.example.com. Never taken from a request, whose Host a
+sender chooses and could point a reset link at their own site.
+
 ## notify.Config
 
 ### `webhooks`

internal/account/account_test.go

@@ -448,3 +448,40 @@ func TestAHashThatCannotGetATurnSaysTheServerIsBusy(t *testing.T) {
 		t.Errorf("a hash with no turn: %v", err)
 	}
 }
+
+func TestAForgottenPasswordIsResetOnceThroughALink(t *testing.T) {
+	svc, st := newService(t)
+	ctx := context.Background()
+	must(t, svc.SignUp(ctx, "ayse@example.com", "A", goodPassword))
+	u, _ := st.UserByEmail(ctx, "ayse@example.com")
+
+	// Waiting for approval: no link, said the same as for nobody.
+	if _, _, err := svc.RequestReset(ctx, st, "ayse@example.com"); !errors.Is(err, ErrNoReset) {
+		t.Fatalf("a pending account got a link: %v", err)
+	}
+	if _, _, err := svc.RequestReset(ctx, st, "nobody@example.com"); !errors.Is(err, ErrNoReset) {
+		t.Fatalf("an unknown address: %v", err)
+	}
+	must(t, st.ApproveUser(ctx, u.ID, "proj", "ilker", time.Now()))
+	session, _, err := svc.SignIn(ctx, "ayse@example.com", goodPassword)
+	must(t, err)
+
+	token, got, err := svc.RequestReset(ctx, st, "AYSE@example.com")
+	if err != nil || token == "" || got.ID != u.ID {
+		t.Fatalf("an active account: %q %+v %v", token, got, err)
+	}
+	if err := svc.ResetPassword(ctx, st, st, token, "short"); !errors.Is(err, ErrWeakPassword) {
+		t.Fatalf("a weak password: %v", err)
+	}
+	next := goodPassword + "-new"
+	must(t, svc.ResetPassword(ctx, st, st, token, next))
+	if _, _, err := svc.Resolve(ctx, session); err == nil {
+		t.Error("a session from before the reset still works")
+	}
+	if _, _, err := svc.SignIn(ctx, "ayse@example.com", next); err != nil {
+		t.Errorf("the new password does not sign in: %v", err)
+	}
+	if err := svc.ResetPassword(ctx, st, st, token, next+"2"); !errors.Is(err, ErrNoReset) {
+		t.Errorf("a link used twice: %v", err)
+	}
+}

internal/account/service.go

@@ -373,3 +373,63 @@ func (s *Service) OpenSession(ctx context.Context, userID string) (token, csrf s
 	}
 	return token, csrf, nil
 }
+
+// ResetLinkTTL is how long a password reset link works.
+const ResetLinkTTL = time.Hour
+
+// ResetStore keeps password reset links.
+type ResetStore interface {
+	UserByEmail(ctx context.Context, email string) (store.User, error)
+	CreatePasswordReset(ctx context.Context, userID string, ttl time.Duration) (string, error)
+	PasswordResetUser(ctx context.Context, token string) (string, error)
+	UsePasswordReset(ctx context.Context, token string) (string, error)
+}
+
+// ErrNoReset is an address that gets no reset link: none has an account, the
+// account signs in through single sign-on, or it is not active. Never told to
+// whoever asked, who learns the same either way.
+var ErrNoReset = errors.New("no reset link for that address")
+
+// RequestReset mints a reset link token for the person an address belongs
+// to, when they sign in with a password and are active. A few a quarter of
+// an hour per address: each sends a mail.
+func (s *Service) RequestReset(ctx context.Context, rs ResetStore, email string) (string, store.User, error) {
+	email = store.NormalizeEmail(email)
+	if !validEmail(email) {
+		return "", store.User{}, ErrNoReset
+	}
+	if !s.allowAttempt("reset:"+email, 3) {
+		return "", store.User{}, ErrTooManyTries
+	}
+	u, err := rs.UserByEmail(ctx, email)
+	if err != nil || u.PasswordHash == "" || u.State != store.UserActive {
+		return "", store.User{}, ErrNoReset
+	}
+	token, err := rs.CreatePasswordReset(ctx, u.ID, ResetLinkTTL)
+	if err != nil {
+		return "", store.User{}, err
+	}
+	s.log.Info("a password reset link was made", "user", u.ID)
+	return token, u, nil
+}
+
+// ResetPassword sets a password from a reset link, which is spent, and signs
+// the person out everywhere: if they lost it, someone else may have it.
+func (s *Service) ResetPassword(ctx context.Context, rs ResetStore, pw PasswordChanger, token, next string) error {
+	hash, err := HashPassword(next)
+	if err != nil {
+		return err
+	}
+	userID, err := rs.UsePasswordReset(ctx, token)
+	if err != nil {
+		return ErrNoReset
+	}
+	if err := pw.SetPassword(ctx, userID, hash); err != nil {
+		return err
+	}
+	if err := pw.DeleteUserSessions(ctx, userID); err != nil {
+		return err
+	}
+	s.log.Info("a password was set from a reset link", "user", userID)
+	return nil
+}

internal/agent/agent.go

@@ -36,6 +36,7 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/agentsvc"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/kvevents"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/obs"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
@@ -87,7 +88,11 @@ type Agent struct {
 	// serverList is every control plane to dial: the configured ones, and
 	// any the leader has named since. Read through servers().
 	serverList []string
-	inv        discovery.Inventory
+	// fwMu keeps one firewall change at a time; fwRunner runs its scripts
+	// (nil: this machine's shell).
+	fwMu     sync.Mutex
+	fwRunner firewall.Runner
+	inv      discovery.Inventory
 	// shardReplica remembers which replica a shard belongs to, so a status
 	// report can name both. The control plane keys on the replica.
 	shardReplica map[string]string
@@ -366,6 +371,12 @@ func (a *Agent) settleAdopted(sh *supervisor.Shard, o supervisor.Orphan) {
 		_ = a.sup.Stop(ctx, o.ShardID)
 		return
 	}
+	// What it computes on, for one started before the agent kept that.
+	if a.sup.ComputeOf(o.ShardID) == nil {
+		if c, ok := a.gpuCompute(ctx, o.Spec.GPUIndices, sh.Status().PID); ok {
+			a.sup.SetCompute(o.ShardID, c)
+		}
+	}
 	a.log.Info("adopted engine is serving",
 		"shard", o.ShardID, "replica", o.ReplicaID)
 }
@@ -406,6 +417,17 @@ func (a *Agent) weights() *weights.Fetcher {
 }
 
 // runningShards describes what this agent is supervising, for registration.
+// unready are the shards a registration reported as not yet ready.
+func unready(running []*llmgwv1.RunningShard) map[string]bool {
+	out := map[string]bool{}
+	for _, r := range running {
+		if r.GetState() != agentsvc.StateToWire(supervisor.Ready) {
+			out[r.GetShardId()] = true
+		}
+	}
+	return out
+}
+
 func (a *Agent) runningShards() []*llmgwv1.RunningShard {
 	var out []*llmgwv1.RunningShard
 	for _, st := range a.sup.Statuses() {
@@ -431,6 +453,7 @@ func (a *Agent) runningShards() []*llmgwv1.RunningShard {
 			Pool:       spec.Pool,
 			Fabric:     spec.Fabric,
 			ProfileFp:  spec.ProfileFP,
+			Compute:    computeToWire(a.sup.ComputeOf(st.ID)),
 			LeaderAddr: a.cfg.MgmtAddr,
 			LeaderPort: int32(spec.LeaderPort),
 		})
@@ -559,8 +582,9 @@ func (a *Agent) session(ctx context.Context, server string) error {
 		a.sendLoop(ctx, stream, out)
 	}()
 	defer wg.Wait()
-	go a.watchExits(ctx, out)
+	go a.watchExits(ctx, out, unready(reg.RunningShards))
 	go a.forwardKV(ctx, out)
+	go a.reportCompute(ctx, out)
 
 	for {
 		msg, err := stream.Receive()
@@ -854,6 +878,16 @@ func (a *Agent) handle(ctx context.Context, msg *llmgwv1.ControlMessage, out cha
 		go a.upgrade(ctx, m.Upgrade, out)
 	case *llmgwv1.ControlMessage_ControlPlanes:
 		a.setControlPlanes(m.ControlPlanes)
+	case *llmgwv1.ControlMessage_Firewall:
+		go a.applyFirewall(m.Firewall)
+	case *llmgwv1.ControlMessage_RestartAgent:
+		go func() {
+			a.log.Info("restarting, as the control plane asks; the engines keep serving",
+				"reason", m.RestartAgent.GetReason())
+			// A moment for the log line to leave before the process does.
+			time.Sleep(300 * time.Millisecond)
+			exitForUpgrade()
+		}()
 	case *llmgwv1.ControlMessage_Logs:
 		reply := a.logsReply(m.Logs)
 		// Off the receive loop: a full send queue must not stop commands being
@@ -1123,6 +1157,14 @@ func (a *Agent) startShard(ctx context.Context, m *llmgwv1.StartShard, out chan<
 	} else {
 		err = a.sup.WaitAlive(ctx, sh)
 	}
+	// Where it put the model, by its own account: said with the readiness,
+	// and a GPU placement it computes on the CPU refused even where
+	// nvidia-smi cannot be asked.
+	compute, said, cerr := engineCompute(eng, spec.GPUIndices, sh.Logs(512))
+	if err == nil && cerr != nil {
+		err = cerr
+		_ = a.sup.Stop(ctx, shardID)
+	}
 	if err == nil {
 		err = a.onItsGPUs(ctx, eng, spec, sh.Status().PID)
 		if err != nil {
@@ -1135,7 +1177,19 @@ func (a *Agent) startShard(ctx context.Context, m *llmgwv1.StartShard, out chan<
 		report(supervisor.Failed, err.Error(), sh.Logs(20))
 		return
 	}
-	report(supervisor.Ready, "", nil)
+	// The engine's own account when it gave one, with its layers; else what
+	// nvidia-smi sees the shard holding.
+	if !said {
+		compute, said = a.gpuCompute(ctx, spec.GPUIndices, sh.Status().PID)
+	}
+	if said {
+		// Kept with the shard, so an agent restarted, or a control plane
+		// taking over, still learns it.
+		a.sup.SetCompute(shardID, compute)
+		send(&llmgwv1.ShardStatus{State: agentsvc.StateToWire(supervisor.Ready), Compute: computeToWire(&compute)})
+	} else {
+		report(supervisor.Ready, "", nil)
+	}
 	// Its dying later is reported by the session's watchExits.
 }
 
@@ -1148,7 +1202,14 @@ func (a *Agent) startShard(ctx context.Context, m *llmgwv1.StartShard, out chan<
 // reported the engine dying; and an adopted engine had no watcher at all —
 // on node06 one was killed and its replica stayed "ready" for the control
 // plane, routed to and failing, indefinitely.
-func (a *Agent) watchExits(ctx context.Context, out chan<- *llmgwv1.AgentMessage) {
+//
+// It also reports the shards the registration said were not ready yet once
+// they are. An adopted engine is health-checked after the agent registers,
+// and a shard started on the stream before is reported on that stream; so
+// when the registration came first nothing ever told the control plane, and
+// the replica stayed "starting" while it served — on node05, after a
+// control plane restart, the only replica of a model.
+func (a *Agent) watchExits(ctx context.Context, out chan<- *llmgwv1.AgentMessage, pending map[string]bool) {
 	t := time.NewTicker(time.Second)
 	defer t.Stop()
 	ready := map[string]bool{}
@@ -1157,6 +1218,24 @@ func (a *Agent) watchExits(ctx context.Context, out chan<- *llmgwv1.AgentMessage
 			switch st.State {
 			case supervisor.Ready:
 				ready[st.ID] = true
+				if !pending[st.ID] {
+					continue
+				}
+				delete(pending, st.ID)
+				sh, err := a.sup.Get(st.ID)
+				if err != nil {
+					continue
+				}
+				msg := &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_Shard{Shard: &llmgwv1.ShardStatus{
+					NodeId: a.cfg.NodeID, ReplicaId: sh.Spec.ReplicaID, ShardId: st.ID,
+					State:   agentsvc.StateToWire(st.State),
+					Compute: computeToWire(a.sup.ComputeOf(st.ID)),
+				}}}
+				select {
+				case out <- msg:
+				case <-ctx.Done():
+					return
+				}
 			case supervisor.Failed:
 				if !ready[st.ID] {
 					continue // never ready here: its start reported it
@@ -1366,3 +1445,42 @@ func ResolveManagement(c Config) (addr, iface string) {
 	}
 	return addr, iface
 }
+
+// reportCompute tells the control plane, once a session, what each running
+// shard computes on as soon as that is known: an adopted engine's is found
+// after registration, which carried what was known then.
+func (a *Agent) reportCompute(ctx context.Context, out chan<- *llmgwv1.AgentMessage) {
+	sent := map[string]bool{}
+	t := time.NewTicker(5 * time.Second)
+	defer t.Stop()
+	for range 60 {
+		for _, st := range a.sup.Statuses() {
+			if st.State.Terminal() || sent[st.ID] {
+				continue
+			}
+			c := a.sup.ComputeOf(st.ID)
+			if c == nil {
+				continue
+			}
+			sh, err := a.sup.Get(st.ID)
+			if err != nil {
+				continue
+			}
+			// No state: a report of compute alone, not a transition.
+			msg := &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_Shard{Shard: &llmgwv1.ShardStatus{
+				NodeId: a.cfg.NodeID, ReplicaId: sh.Spec.ReplicaID, ShardId: st.ID, Compute: computeToWire(c),
+			}}}
+			select {
+			case out <- msg:
+				sent[st.ID] = true
+			case <-ctx.Done():
+				return
+			}
+		}
+		select {
+		case <-ctx.Done():
+			return
+		case <-t.C:
+		}
+	}
+}

internal/agent/agent_test.go

@@ -37,11 +37,12 @@ import (
 type statusRecorder struct {
 	ch chan status
 
-	mu      sync.Mutex
-	calls   []adoptCall
-	adopted []reconciler.AdoptedShard
-	unclaim []string // shard ids to report back as unclaimed
-	fetches []fetchReport
+	mu       sync.Mutex
+	calls    []adoptCall
+	adopted  []reconciler.AdoptedShard
+	unclaim  []string // shard ids to report back as unclaimed
+	fetches  []fetchReport
+	computes []reconciler.Compute
 }
 
 type status struct {
@@ -61,6 +62,12 @@ func (r *statusRecorder) OnShardStatus(replicaID, shardID string, st supervisor.
 
 // fetches records download progress, so a test can assert that a fetch is
 // visible from the control plane rather than only in the node's log.
+func (r *statusRecorder) OnShardCompute(replicaID, shardID string, c reconciler.Compute) {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	r.computes = append(r.computes, c)
+}
+
 func (r *statusRecorder) OnFetchProgress(replicaID, shardID, source string, done, total int64) {
 	r.mu.Lock()
 	defer r.mu.Unlock()

internal/agent/engines/compute.go

@@ -0,0 +1,79 @@
+package engines
+
+import (
+	"regexp"
+	"strconv"
+	"strings"
+)
+
+// Compute is where an engine runs its model, as it said while loading it.
+type Compute struct {
+	// Backend is CUDA, Vulkan, Metal, ROCm or CPU.
+	Backend string
+	// Devices are those it named: "CUDA0: Quadro RTX 6000 (7.5)".
+	Devices []string
+	// Offloaded of Total layers went to the devices; 0 and 0 when it did
+	// not say.
+	Offloaded, Total int
+	// NoGPU is the engine saying it found no device it could use.
+	NoGPU bool
+}
+
+var (
+	offloadedRE = regexp.MustCompile(`offloaded (\d+)/(\d+) layers to GPU`)
+	usingRE     = regexp.MustCompile(`using device ([A-Za-z]+\d+) \(([^)]*)\)`)
+	ccRE        = regexp.MustCompile(`Device (\d+): ([^,]+), compute capability ([0-9.]+)`)
+	noDevicesRE = regexp.MustCompile(`found 0 (CUDA|Vulkan|ROCm) devices|no usable GPU found`)
+)
+
+// ReadLlamaCompute reads llama.cpp's start-up output for where it put the
+// model, reporting false when the output says nothing of it.
+func ReadLlamaCompute(lines []string) (Compute, bool) {
+	var c Compute
+	known := false
+	cc := map[string]string{} // device index → compute capability
+	for _, l := range lines {
+		if m := ccRE.FindStringSubmatch(l); m != nil {
+			cc[m[1]] = m[3]
+		}
+	}
+	seen := map[string]bool{}
+	for _, l := range lines {
+		if m := offloadedRE.FindStringSubmatch(l); m != nil {
+			c.Offloaded, _ = strconv.Atoi(m[1])
+			c.Total, _ = strconv.Atoi(m[2])
+			known = true
+		}
+		if m := usingRE.FindStringSubmatch(l); m != nil && !seen[m[1]] {
+			seen[m[1]] = true
+			d := m[1] + ": " + strings.TrimSpace(m[2])
+			idx := strings.TrimLeft(m[1], "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz")
+			if v, ok := cc[idx]; ok && strings.HasPrefix(m[1], "CUDA") {
+				d += " (" + v + ")"
+			}
+			c.Devices = append(c.Devices, d)
+			if c.Backend == "" {
+				c.Backend = strings.TrimRight(m[1], "0123456789")
+			}
+			known = true
+		}
+		if noDevicesRE.MatchString(l) {
+			c.NoGPU, known = true, true
+		}
+	}
+	switch {
+	case c.OnTheCPU():
+		c.Backend = "CPU"
+	case c.Backend == "" && c.Offloaded > 0:
+		c.Backend = "GPU"
+	case c.Backend == "":
+		c.Backend = "CPU"
+	}
+	return c, known
+}
+
+// OnTheCPU is whether a model the engine was given GPUs for is computed on
+// the CPU, by its own account: no device found, or no layer offloaded.
+func (c Compute) OnTheCPU() bool {
+	return c.NoGPU || (c.Total > 0 && c.Offloaded == 0)
+}

internal/agent/engines/compute_test.go

@@ -0,0 +1,32 @@
+package engines
+
+import (
+	"strings"
+	"testing"
+)
+
+func TestLlamaCppSaysWhereItPutTheModel(t *testing.T) {
+	gpu := strings.Split(`ggml_cuda_init: found 1 CUDA devices:
+  Device 0: Quadro RTX 6000, compute capability 7.5, VMM: yes
+load_backend: loaded CUDA backend from /opt/llmgw/engines/llamacpp/b10991/libggml-cuda.so
+llama_model_load_from_file_impl: using device CUDA0 (Quadro RTX 6000) - 23800 MiB free
+load_tensors: offloading 48 repeating layers to GPU
+load_tensors: offloaded 49/49 layers to GPU`, "\n")
+	c, ok := ReadLlamaCompute(gpu)
+	if !ok || c.Backend != "CUDA" || c.Offloaded != 49 || c.Total != 49 || c.OnTheCPU() ||
+		len(c.Devices) != 1 || c.Devices[0] != "CUDA0: Quadro RTX 6000 (7.5)" {
+		t.Errorf("on the GPU: %+v %v", c, ok)
+	}
+
+	cpu := strings.Split(`warning: no usable GPU found, --gpu-layers option will be ignored
+warning: one possible reason is that llama.cpp was compiled without GPU support
+load_tensors: offloaded 0/49 layers to GPU`, "\n")
+	c, ok = ReadLlamaCompute(cpu)
+	if !ok || c.Backend != "CPU" || !c.OnTheCPU() {
+		t.Errorf("fell back to the CPU: %+v %v", c, ok)
+	}
+
+	if _, ok := ReadLlamaCompute([]string{"main: server is listening on http://127.0.0.1:20001"}); ok {
+		t.Error("output that says nothing was read as saying something")
+	}
+}

internal/agent/engines/engine.go

@@ -114,6 +114,10 @@ type ShardSpec struct {
 	// lets the control plane tell, after its own restart, whether a shard that
 	// is still running matches the profile the catalog holds now.
 	ProfileFP string
+
+	// Compute is where the engine said it put the model as it became ready,
+	// kept with the shard's state so an agent restarted later still knows.
+	Compute *Compute `json:",omitempty"`
 }
 
 // Host is where the engine listens.

internal/agent/engines/llamacpp.go

@@ -277,6 +277,9 @@ func (l *LlamaCpp) bin() string {
 	return "llama-server"
 }
 
+// Binary is the llama-server this engine runs.
+func (l *LlamaCpp) Binary() string { return l.bin() }
+
 // FallsBackToCPU says that llama.cpp keeps running when its GPU backend does
 // not load: --n-gpu-layers asks for layers on a GPU it then has none of, and
 // the server comes up and answers on the CPU, at a tenth of the speed, with no

internal/agent/firewall.go

@@ -0,0 +1,89 @@
+package agent
+
+import (
+	"context"
+	"net"
+	"net/netip"
+	"net/url"
+	"strings"
+	"time"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
+)
+
+// applyFirewall keeps llmgw's firewall on this node as the cluster's
+// settings say: the engines' ports and the node exporter's for the networks
+// the node reaches its control planes on, and those named, or no table at
+// all. Off the receive loop, one at a time.
+func (a *Agent) applyFirewall(m *llmgwv1.Firewall) {
+	a.fwMu.Lock()
+	defer a.fwMu.Unlock()
+	ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
+	defer cancel()
+	r := a.fwRunner
+	if r == nil {
+		r = firewall.Local{}
+	}
+	if !m.GetEnabled() {
+		had, err := firewall.Disable(ctx, r)
+		switch {
+		case err != nil:
+			a.log.Warn("could not take llmgw's firewall off this node", "err", err)
+		case had:
+			a.log.Info("llmgw's firewall is off on this node, as the cluster's settings say")
+		}
+		return
+	}
+	a.mu.Lock()
+	servers := append([]string(nil), a.serverList...)
+	a.mu.Unlock()
+	nets, err := firewall.Networks(ctx, r, controlPlaneIPs(ctx, servers))
+	if err != nil {
+		a.log.Warn("could not tell which networks reach the control planes; the firewall is left as it is", "err", err)
+		return
+	}
+	for _, n := range m.GetNetworks() {
+		if p, err := netip.ParsePrefix(n); err == nil {
+			nets = append(nets, p)
+		} else if ad, err := netip.ParseAddr(n); err == nil {
+			nets = append(nets, netip.PrefixFrom(ad, ad.BitLen()))
+		}
+	}
+	from, to := int(m.GetEnginePortsFrom()), int(m.GetEnginePortsTo())
+	ports := []firewall.Port{firewall.NodeExporter}
+	if from > 0 && to >= from {
+		ports = append(ports, firewall.Engines(from, to))
+	}
+	changed, err := firewall.Install(ctx, r, firewall.Spec{Ports: ports, Allow: nets})
+	switch {
+	case err != nil:
+		a.log.Warn("could not put llmgw's firewall on this node", "err", err)
+	case changed:
+		a.log.Info("llmgw's firewall is on this node", "networks", len(nets))
+	}
+}
+
+// controlPlaneIPs are the addresses of the control planes an agent dials.
+func controlPlaneIPs(ctx context.Context, servers []string) []string {
+	var out []string
+	for _, s := range servers {
+		host := s
+		if u, err := url.Parse(s); err == nil && u.Host != "" {
+			host = u.Hostname()
+		}
+		host = strings.Trim(host, "[]")
+		if a, err := netip.ParseAddr(host); err == nil {
+			out = append(out, a.String())
+			continue
+		}
+		ips, err := net.DefaultResolver.LookupNetIP(ctx, "ip", host)
+		if err != nil {
+			continue
+		}
+		for _, ip := range ips {
+			out = append(out, ip.Unmap().String())
+		}
+	}
+	return out
+}

internal/agent/firewall_test.go

@@ -0,0 +1,57 @@
+package agent
+
+import (
+	"context"
+	"errors"
+	"log/slog"
+	"strings"
+	"testing"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+)
+
+// fakeShell answers the firewall package's scripts and records what was
+// installed or removed.
+type fakeShell struct {
+	present   bool
+	installed string
+	deleted   bool
+}
+
+func (f *fakeShell) Run(_ context.Context, script string, stdin []byte) ([]byte, error) {
+	switch {
+	case strings.HasPrefix(script, "ip -o route get 10.0.0.140"):
+		return []byte("10.0.0.140 dev eth0 src 10.0.0.141 uid 0\n2: eth0    inet 10.0.0.141/24 brd 10.0.0.255 scope global eth0\n"), nil
+	case strings.HasPrefix(script, "sudo -n /usr/sbin/nft list table"):
+		if f.present {
+			return nil, nil
+		}
+		return nil, errors.New("no table")
+	case strings.Contains(script, "nft delete table") && !strings.Contains(script, "cat > "):
+		f.deleted, f.present = true, false
+		return nil, nil
+	case strings.HasPrefix(script, "sudo cat "):
+		return nil, nil
+	default:
+		f.installed, f.present = string(stdin), true
+		return []byte("changed\n"), nil
+	}
+}
+
+func TestANodeKeepsOrDropsItsFirewallAsTheClusterSays(t *testing.T) {
+	sh := &fakeShell{}
+	a := &Agent{log: slog.New(slog.DiscardHandler), fwRunner: sh, serverList: []string{"https://10.0.0.140:7443"}}
+
+	a.applyFirewall(&llmgwv1.Firewall{Enabled: true, Networks: []string{"192.168.5.0/24", "10.9.9.9"},
+		EnginePortsFrom: 20000, EnginePortsTo: 20999})
+	for _, want := range []string{"10.0.0.0/24", "192.168.5.0/24", "10.9.9.9/32", "20000-20999", "9100"} {
+		if !strings.Contains(sh.installed, want) {
+			t.Errorf("the rules lack %s:\n%s", want, sh.installed)
+		}
+	}
+
+	a.applyFirewall(&llmgwv1.Firewall{Enabled: false})
+	if !sh.deleted {
+		t.Error("turned off, the table stayed")
+	}
+}

internal/agent/gpucheck.go

@@ -4,10 +4,12 @@ import (
 	"context"
 	"fmt"
 	"os"
+	"slices"
 	"strconv"
 	"strings"
 	"time"
 
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
 )
@@ -153,3 +155,70 @@ func joinIndices(is []int) string {
 	}
 	return strings.Join(s, ", ")
 }
+
+// engineCompute is where an engine said it put its model, from its output
+// so far — llama.cpp's, which says — and false for one that said nothing. A
+// shard given GPUs that its engine says it computes on the CPU is an error,
+// found this way even where nvidia-smi cannot be asked.
+func engineCompute(eng engines.Engine, gpus []int, lines []string) (engines.Compute, bool, error) {
+	if _, ok := eng.(engines.CPUFallback); !ok {
+		return engines.Compute{}, false, nil
+	}
+	read, ok := engines.ReadLlamaCompute(lines)
+	if !ok {
+		return engines.Compute{}, false, nil
+	}
+	if len(gpus) > 0 && read.OnTheCPU() {
+		why := "its GPU backend saw no device it could use"
+		if read.Total > 0 && !read.NoGPU {
+			why = fmt.Sprintf("it offloaded %d of %d layers", read.Offloaded, read.Total)
+		}
+		return read, true, fmt.Errorf("placed on GPU %v but computing on the CPU: %s, by llama.cpp's own account "+
+			"(llmgw ctl engine-check on this node says why)", gpus, why)
+	}
+	return read, true, nil
+}
+
+// computeToWire is a compute as it travels to the control plane.
+func computeToWire(c *engines.Compute) *llmgwv1.Compute {
+	if c == nil {
+		return nil
+	}
+	return &llmgwv1.Compute{Backend: c.Backend, Devices: c.Devices,
+		LayersOffloaded: int32(c.Offloaded), LayersTotal: int32(c.Total)} //nolint:gosec // layer counts
+}
+
+// gpuCompute is where a shard computes as nvidia-smi sees it: the CPU for a
+// shard given no GPU, else the GPUs whose memory its processes hold, and how
+// much. False when nvidia-smi cannot be asked. Engine-agnostic, and needing
+// no log line: llama.cpp says where its layers went only at -lv 4.
+func (a *Agent) gpuCompute(ctx context.Context, gpus []int, pid int) (engines.Compute, bool) {
+	if len(gpus) == 0 {
+		return engines.Compute{Backend: "CPU"}, true
+	}
+	mem, err := gpuMemory(ctx, a.cfg.Discovery.NvidiaSMI)
+	if err != nil || mem == nil || pid <= 0 {
+		return engines.Compute{}, false
+	}
+	group := processGroup(pid)
+	c := engines.Compute{Backend: "CUDA"}
+	for _, g := range mem {
+		if !slices.Contains(gpus, g.Index) {
+			continue
+		}
+		used := 0
+		for _, p := range g.Procs {
+			if holds([]int{p.PID}, pid, group) {
+				used += p.UsedMB
+			}
+		}
+		if used > 0 {
+			c.Devices = append(c.Devices, fmt.Sprintf("GPU %d: %.1f GB", g.Index, float64(used)/1024))
+		}
+	}
+	if len(c.Devices) == 0 {
+		// Placed on GPUs and holding none of their memory.
+		c.Backend = "CPU"
+	}
+	return c, true
+}

internal/agent/gpucheck_test.go

@@ -122,3 +122,55 @@ func TestForeignGPUMemoryLeavesOutTheAgentsOwnShards(t *testing.T) {
 		t.Errorf("no reading came back as %v", foreign)
 	}
 }
+
+func TestLlamaCppsOwnWordOnTheCPUIsEnoughToRefuse(t *testing.T) {
+	eng := fallbackEngine{}
+	// Without nvidia-smi to ask, the engine's own output says it.
+	c, said, err := engineCompute(eng, []int{0}, []string{
+		"warning: no usable GPU found, --gpu-layers option will be ignored",
+		"load_tensors: offloaded 0/49 layers to GPU"})
+	if err == nil || !said || !strings.Contains(err.Error(), "computing on the CPU") || c.Backend != "CPU" {
+		t.Fatalf("a GPU shard llama.cpp put on the CPU: %v %+v", err, c)
+	}
+	c, said, err = engineCompute(eng, []int{0}, []string{
+		"llama_model_load_from_file_impl: using device CUDA0 (Quadro RTX 6000) - 23800 MiB free",
+		"load_tensors: offloaded 49/49 layers to GPU"})
+	if err != nil || !said || c.Backend != "CUDA" || c.Offloaded != 49 || computeToWire(&c).GetLayersOffloaded() != 49 {
+		t.Errorf("on its GPU: %v %+v", err, c)
+	}
+	// A CPU shard is not refused for being on the CPU, and says so.
+	if c, _, err := engineCompute(eng, nil, []string{"load_tensors: offloaded 0/49 layers to GPU"}); err != nil || c.Backend != "CPU" {
+		t.Errorf("a CPU shard: %v %+v", err, c)
+	}
+	// An engine that said nothing says nothing.
+	if _, said, err := engineCompute(eng, []int{0}, []string{"listening"}); err != nil || said {
+		t.Errorf("nothing said: %v", err)
+	}
+}
+
+func TestNvidiaSMISaysWhereAShardComputes(t *testing.T) {
+	oldMem, oldGroup := gpuMemory, processGroup
+	t.Cleanup(func() { gpuMemory, processGroup = oldMem, oldGroup })
+	processGroup = func(pid int) int { return pid }
+	gpuMemory = func(context.Context, string) ([]discovery.GPUMemory, error) {
+		return []discovery.GPUMemory{
+			{Index: 0, UsedMB: 20000, Procs: []discovery.GPUProcess{{PID: 700, UsedMB: 2048}, {PID: 900, UsedMB: 17000}}},
+			{Index: 1, UsedMB: 0},
+		}, nil
+	}
+	a := &Agent{log: slog.Default()}
+	if c, ok := a.gpuCompute(context.Background(), []int{0}, 700); !ok || c.Backend != "CUDA" ||
+		len(c.Devices) != 1 || c.Devices[0] != "GPU 0: 2.0 GB" {
+		t.Errorf("on GPU 0: %+v %v", c, ok)
+	}
+	if c, ok := a.gpuCompute(context.Background(), []int{1}, 700); !ok || c.Backend != "CPU" {
+		t.Errorf("placed on GPU 1, holding none of it: %+v", c)
+	}
+	if c, ok := a.gpuCompute(context.Background(), nil, 700); !ok || c.Backend != "CPU" {
+		t.Errorf("a CPU shard: %+v", c)
+	}
+	gpuMemory = func(context.Context, string) ([]discovery.GPUMemory, error) { return nil, nil }
+	if _, ok := a.gpuCompute(context.Background(), []int{0}, 700); ok {
+		t.Error("no nvidia-smi read as an answer")
+	}
+}

internal/agent/supervisor/supervisor.go

@@ -113,7 +113,10 @@ type Shard struct {
 	exitCode int
 
 	logs *ringBuffer
-	cmd  *exec.Cmd
+	// cmdline is an adopted engine's command line, from its state file: it
+	// has no cmd of this agent's to read it from.
+	cmdline []string
+	cmd     *exec.Cmd
 	// sidecars are the shard's supporting processes, torn down with it.
 	sidecars []*exec.Cmd
 	cancel   context.CancelFunc
@@ -644,6 +647,32 @@ func (s *Supervisor) Get(id string) (*Shard, error) {
 	return sh, nil
 }
 
+// SetCompute records where a shard's engine said it put the model, with the
+// shard's state on disk.
+func (s *Supervisor) SetCompute(id string, c engines.Compute) {
+	s.mu.Lock()
+	sh, ok := s.shards[id]
+	if ok {
+		sh.Spec.Compute = &c
+	}
+	s.mu.Unlock()
+	if ok && sh.pid > 0 {
+		_ = s.writeState(sh)
+	}
+}
+
+// ComputeOf is where a shard's engine was found to compute, nil when not
+// yet known.
+func (s *Supervisor) ComputeOf(id string) *engines.Compute {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	if sh, ok := s.shards[id]; ok && sh.Spec.Compute != nil {
+		c := *sh.Spec.Compute
+		return &c
+	}
+	return nil
+}
+
 // Remove forgets a terminated shard.
 func (s *Supervisor) Remove(id string) {
 	s.mu.Lock()
@@ -703,11 +732,17 @@ type shardState struct {
 
 // writeState records everything needed to recognise and describe the process later.
 func (s *Supervisor) writeState(sh *Shard) error {
+	// An adopted engine was started by an earlier agent: its command line
+	// is the one its state file kept.
+	cmdline := sh.cmdline
+	if sh.cmd != nil {
+		cmdline = sh.cmd.Args
+	}
 	b, err := json.Marshal(shardState{
 		ShardID:   sh.ID,
 		ReplicaID: sh.Spec.ReplicaID,
 		PID:       sh.pid,
-		Cmdline:   sh.cmd.Args,
+		Cmdline:   cmdline,
 		StartedAt: sh.started,
 		Spec:      sh.Spec,
 	})
@@ -804,6 +839,7 @@ func (s *Supervisor) Adopt(o Orphan) (*Shard, error) {
 		started: o.StartedAt,
 		done:    make(chan struct{}),
 		adopted: true,
+		cmdline: o.Cmdline,
 	}
 	s.shards[o.ShardID] = sh
 	s.mu.Unlock()

internal/agent/supervisor/supervisor_test.go

@@ -848,3 +848,60 @@ func eventually(t *testing.T, limit time.Duration, msg string, cond func() bool)
 		time.Sleep(10 * time.Millisecond)
 	}
 }
+
+func TestWhatAShardComputesOnSurvivesTheAgent(t *testing.T) {
+	s, _ := newSupervisor(t)
+	spec := engines.ShardSpec{ReplicaID: "rep-1", ShardID: "a", HTTPPort: 20100}
+	sh, err := s.Start(t.Context(), "a", spec, shellCommand("sleep 30"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	t.Cleanup(func() { _ = s.Stop(context.Background(), "a") })
+	_ = sh
+	s.SetCompute("a", engines.Compute{Backend: "CUDA", Offloaded: 49, Total: 49})
+
+	s2, err := New(Config{WorkDir: s.cfg.WorkDir, StopTimeout: 2 * time.Second},
+		&scriptedHealth{}, slog.New(slog.DiscardHandler))
+	if err != nil {
+		t.Fatal(err)
+	}
+	orphans, err := s2.FindOrphans()
+	if err != nil || len(orphans) != 1 {
+		t.Fatalf("%v %d", err, len(orphans))
+	}
+	if c := orphans[0].Spec.Compute; c == nil || c.Backend != "CUDA" || c.Offloaded != 49 {
+		t.Errorf("not kept: %+v", c)
+	}
+}
+
+func TestAnAdoptedShardsComputeIsKeptWithoutCrashing(t *testing.T) {
+	s, _ := newSupervisor(t)
+	spec := engines.ShardSpec{ReplicaID: "rep-1", ShardID: "a", HTTPPort: 20100}
+	if _, err := s.Start(t.Context(), "a", spec, shellCommand("sleep 30")); err != nil {
+		t.Fatal(err)
+	}
+	// An agent restart: the engine is adopted by a new supervisor, which did
+	// not start it and has no command of its own for it.
+	s2, err := New(Config{WorkDir: s.cfg.WorkDir, StopTimeout: 2 * time.Second},
+		&scriptedHealth{}, slog.New(slog.DiscardHandler))
+	if err != nil {
+		t.Fatal(err)
+	}
+	orphans, _ := s2.FindOrphans()
+	if len(orphans) != 1 {
+		t.Fatalf("%d orphans", len(orphans))
+	}
+	if _, err := s2.Adopt(orphans[0]); err != nil {
+		t.Fatal(err)
+	}
+	t.Cleanup(func() { _ = s2.Stop(context.Background(), "a") })
+	s2.SetCompute("a", engines.Compute{Backend: "CPU"})
+
+	// And a third agent still finds it, with its command line and compute.
+	s3, _ := New(Config{WorkDir: s.cfg.WorkDir, StopTimeout: 2 * time.Second},
+		&scriptedHealth{}, slog.New(slog.DiscardHandler))
+	again, _ := s3.FindOrphans()
+	if len(again) != 1 || len(again[0].Cmdline) == 0 || again[0].Spec.Compute == nil {
+		t.Fatalf("lost: %+v", again)
+	}
+}

internal/agent/watchexits_test.go

@@ -10,6 +10,7 @@ import (
 	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/agentsvc"
 )
 
 type alwaysReady struct{}
@@ -40,7 +41,7 @@ func TestAnAdoptedEngineThatDiesIsReported(t *testing.T) {
 	out := make(chan *llmgwv1.AgentMessage, 4)
 	ctx, cancel := context.WithCancel(t.Context())
 	defer cancel()
-	go a.watchExits(ctx, out)
+	go a.watchExits(ctx, out, nil)
 	time.Sleep(1500 * time.Millisecond) // seen ready
 	_ = cmd.Process.Kill()
 
@@ -54,3 +55,50 @@ func TestAnAdoptedEngineThatDiesIsReported(t *testing.T) {
 		t.Fatal("the adopted engine died and nothing was reported")
 	}
 }
+
+// Registered while its health check still ran: the ready that follows is
+// reported, or the control plane keeps the replica starting while it serves.
+func TestAShardRegisteredStartingIsReportedReady(t *testing.T) {
+	sup, err := supervisor.New(supervisor.Config{WorkDir: t.TempDir(), RunAs: "-"}, alwaysReady{}, slog.New(slog.DiscardHandler))
+	if err != nil {
+		t.Fatal(err)
+	}
+	cmd := exec.Command("sleep", "60")
+	if err := cmd.Start(); err != nil {
+		t.Fatal(err)
+	}
+	defer func() { _ = cmd.Process.Kill(); _ = cmd.Wait() }()
+	sh, err := sup.Adopt(supervisor.Orphan{ShardID: "m-r0", ReplicaID: "m", PID: cmd.Process.Pid,
+		Spec: engines.ShardSpec{ReplicaID: "m", ShardID: "m-r0", HTTPPort: 1}, Alive: true})
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	a := &Agent{sup: sup, log: slog.New(slog.DiscardHandler)}
+	pending := unready(a.runningShards())
+	if !pending["m-r0"] {
+		t.Fatalf("registration reported it ready already: %v", pending)
+	}
+	out := make(chan *llmgwv1.AgentMessage, 4)
+	ctx, cancel := context.WithCancel(t.Context())
+	defer cancel()
+	go a.watchExits(ctx, out, pending)
+	if err := sup.WaitReady(t.Context(), sh, "http://127.0.0.1:1", "m", 5*time.Second); err != nil {
+		t.Fatal(err)
+	}
+
+	select {
+	case m := <-out:
+		s := m.GetShard()
+		if s.GetShardId() != "m-r0" || s.GetState() != agentsvc.StateToWire(supervisor.Ready) {
+			t.Errorf("reported %+v", s)
+		}
+	case <-time.After(10 * time.Second):
+		t.Fatal("it became ready and nothing was reported")
+	}
+	select {
+	case m := <-out:
+		t.Errorf("reported twice: %+v", m.GetShard())
+	case <-time.After(2500 * time.Millisecond):
+	}
+}

internal/audit/audit.go

@@ -12,6 +12,7 @@ import (
 	"net/http"
 	"net/url"
 	"regexp"
+	"slices"
 	"sort"
 	"strings"
 	"time"
@@ -110,11 +111,16 @@ func (w *statusWriter) Flush() {
 	}
 }
 
+// notChanges are POSTs that change nothing and carry text that must not be
+// kept: asking a judge about a sentence.
+var notChanges = []string{"/api/v1/guard/judges/ask"}
+
 // API records every change to the admin API: a request that is not a GET or
 // HEAD, under /api/. actor says who made it.
 func API(sink Sink, actor func(*http.Request) string, log *slog.Logger, next http.Handler) http.Handler {
 	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
-		if r.Method == http.MethodGet || r.Method == http.MethodHead || !strings.HasPrefix(r.URL.Path, "/api/") {
+		if r.Method == http.MethodGet || r.Method == http.MethodHead || !strings.HasPrefix(r.URL.Path, "/api/") ||
+			slices.Contains(notChanges, r.URL.Path) {
 			next.ServeHTTP(w, r)
 			return
 		}

internal/audit/audit_test.go

@@ -72,3 +72,13 @@ func TestABodyLargerThanWhatIsRecordedReachesTheHandlerWhole(t *testing.T) {
 		t.Errorf("the handler was given %d of %d bytes", got, len(big))
 	}
 }
+
+func TestAskingAJudgeKeepsNoText(t *testing.T) {
+	sink := &memSink{}
+	h := API(sink, func(*http.Request) string { return "ilker" }, nil, http.HandlerFunc(func(http.ResponseWriter, *http.Request) {}))
+	h.ServeHTTP(httptest.NewRecorder(), httptest.NewRequest("POST", "/api/v1/guard/judges/ask",
+		strings.NewReader(`{"text":"something a user wrote"}`)))
+	if len(sink.got) != 0 {
+		t.Errorf("a judge's question was recorded: %+v", sink.got)
+	}
+}

internal/cli/backup.go

@@ -54,7 +54,7 @@ func newBackupCmd() *cobra.Command {
 			"the newest archive as llmgw_backup_last_success_timestamp_seconds, and\n" +
 			"BackupTooOld fires when it is more than a day old.",
 		RunE: func(cmd *cobra.Command, _ []string) error {
-			cfg, err := config.Load(configPath)
+			cfg, err := loadEffectiveConfig(configPath)
 			if err != nil {
 				return err
 			}

internal/cli/bundle.go

@@ -323,7 +323,7 @@ func loadServerConfig(path string) (config.Server, bool) {
 	if _, err := os.Stat(path); err != nil {
 		return config.Server{}, false
 	}
-	cfg, err := config.Load(path)
+	cfg, err := loadEffectiveConfig(path)
 	if err != nil {
 		return config.Server{}, false
 	}

internal/cli/clustersettings.go

@@ -0,0 +1,365 @@
+package cli
+
+import (
+	"context"
+	"crypto/sha256"
+	"errors"
+	"fmt"
+	"log/slog"
+	"os"
+	"slices"
+	"strings"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
+	storepkg "gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// settingsPoll is how often a control plane asks whether the cluster's
+// settings, or its own server.yaml, changed.
+const settingsPoll = 5 * time.Second
+
+// clusterSettings is what a control plane runs on, and where it came from:
+// its server.yaml for its own keys, the store's settings for the rest.
+type clusterSettings struct {
+	cfg      config.Server
+	eff      map[string]any // the settings, as written, cfg was made from
+	cluster  map[string]any // the store's document, nil without a store
+	version  int64
+	fileHash [32]byte
+}
+
+// loadClusterSettings reads the configuration a control plane starts on.
+// Without a store it is the file. With one, the store's settings take the
+// place of the file's cluster keys; a store with none yet is given the
+// file's, so the first control plane to start against it brings them in.
+func loadClusterSettings(ctx context.Context, path string, log *slog.Logger) (clusterSettings, error) {
+	var s clusterSettings
+	b, file, err := readSettingsFile(path)
+	if err != nil {
+		return s, err
+	}
+	s.fileHash = sha256.Sum256(b)
+	fileCfg, err := config.FromMap(file)
+	if err != nil {
+		return s, fileError(path, err)
+	}
+	if fileCfg.Auth.Store == "" {
+		s.cfg, s.eff = fileCfg, file
+		return s, nil
+	}
+	st, err := storepkg.Open(ctx, fileCfg.Auth.Store)
+	if err != nil {
+		return s, fmt.Errorf("the store, for the cluster's settings: %w", err)
+	}
+	defer st.Close()
+
+	doc, err := st.ClusterSettings(ctx)
+	if err != nil {
+		return s, fmt.Errorf("reading the cluster's settings: %w", err)
+	}
+	if doc.Version == 0 {
+		if doc, err = importSettings(ctx, st, path, file, log); err != nil {
+			return s, err
+		}
+	} else if ignored := config.Ignored(file); len(ignored) > 0 {
+		log.Warn("server.yaml sets keys the cluster's settings now hold; they are not read "+
+			"(llmgw ctl config shows what is in force)", "keys", strings.Join(ignored, ", "))
+	}
+	// The newest version that this machine can run on. One that cannot — a
+	// path only another control plane has — must not keep this one down,
+	// where nothing could be used to put it right.
+	versions := []storepkg.SettingsVersion{doc}
+	if older, err := st.ClusterSettingsHistory(ctx, 20); err == nil && len(older) > 1 {
+		versions = append(versions, older[1:]...)
+	}
+	var firstErr error
+	for _, v := range versions {
+		cluster, err := config.Parse([]byte(v.Body))
+		if err == nil {
+			eff := config.Layer(file, cluster)
+			var cfg config.Server
+			if cfg, err = config.FromMap(eff); err == nil {
+				if v.Version != doc.Version {
+					log.Error("the cluster's newest settings do not load on this machine; running on an older version",
+						"newest", doc.Version, "running", v.Version, "err", firstErr)
+				}
+				s.cfg, s.eff, s.cluster, s.version = cfg, eff, cluster, doc.Version
+				return s, nil
+			}
+		}
+		if firstErr == nil {
+			firstErr = err
+		}
+	}
+	return s, fmt.Errorf("the cluster's settings (version %d) do not load on this machine: %w", doc.Version, firstErr)
+}
+
+// importSettings records a server.yaml's cluster keys as the cluster's first
+// settings, with the baseline guard policy when the file has none: a cluster
+// installed before guardrails gets them on its first start after an upgrade.
+func importSettings(ctx context.Context, st *storepkg.DB, path string, file map[string]any,
+	log *slog.Logger,
+) (storepkg.SettingsVersion, error) {
+	_, cluster := config.Split(file)
+	note := "imported from " + path
+	if _, ok := cluster["guard"]; !ok {
+		baseline, err := config.Parse([]byte(guardSection(false)))
+		if err == nil {
+			cluster["guard"] = baseline["guard"]
+			note += ", with the baseline guard policy it did not have"
+		}
+	}
+	body, err := config.Marshal(cluster)
+	if err != nil {
+		return storepkg.SettingsVersion{}, err
+	}
+	host, _ := os.Hostname()
+	_, err = st.PutClusterSettings(ctx, 0, string(body), host, note)
+	switch {
+	case errors.Is(err, storepkg.ErrSettingsChanged):
+		// Another control plane brought its own in first: those are the
+		// cluster's.
+	case err != nil:
+		return storepkg.SettingsVersion{}, fmt.Errorf("recording the cluster's settings: %w", err)
+	default:
+		log.Info("the cluster's settings are now kept in the store; server.yaml keeps this machine's own",
+			"from", path)
+	}
+	return st.ClusterSettings(ctx)
+}
+
+func readSettingsFile(path string) ([]byte, map[string]any, error) {
+	if path == "" {
+		return nil, map[string]any{}, nil
+	}
+	b, err := os.ReadFile(path) //nolint:gosec // the configuration
+	if err != nil {
+		return nil, nil, fmt.Errorf("config file: %w", err)
+	}
+	m, err := config.Parse(b)
+	if err != nil {
+		return nil, nil, fmt.Errorf("parsing %s: %w", path, err)
+	}
+	return b, m, nil
+}
+
+func fileError(path string, err error) error {
+	if path == "" {
+		return err
+	}
+	return fmt.Errorf("%s: %w", path, err)
+}
+
+// settingsWatch follows the cluster's settings and this machine's
+// server.yaml, and puts every change in force.
+type settingsWatch struct {
+	path string
+	st   *storepkg.DB // nil: the file only
+	cur  clusterSettings
+	// live applies a change without a restart, reporting whether it could.
+	live func(cfg config.Server, changed []string) bool
+	// restart has the process start again on the new settings.
+	restart func(reason string)
+	log     *slog.Logger
+}
+
+func (w *settingsWatch) run(ctx context.Context) {
+	t := time.NewTicker(settingsPoll)
+	defer t.Stop()
+	for {
+		select {
+		case <-ctx.Done():
+			return
+		case <-t.C:
+			w.check(ctx)
+		}
+	}
+}
+
+// check reads what changed, if anything, and applies it.
+func (w *settingsWatch) check(ctx context.Context) {
+	version := w.cur.version
+	if w.st != nil {
+		v, err := w.st.ClusterSettingsVersion(ctx)
+		if err != nil {
+			return // asked again in a few seconds
+		}
+		version = v
+	}
+	b, file, err := readSettingsFile(w.path)
+	if err != nil {
+		w.log.Error("server.yaml cannot be read; running on what it said before", "err", err)
+		return
+	}
+	hash := sha256.Sum256(b)
+	if version == w.cur.version && hash == w.cur.fileHash {
+		return
+	}
+	cluster := w.cur.cluster
+	eff := file
+	if w.st != nil {
+		if version != w.cur.version {
+			doc, err := w.st.ClusterSettings(ctx)
+			if err != nil {
+				return
+			}
+			if cluster, err = config.Parse([]byte(doc.Body)); err != nil {
+				w.log.Error("the cluster's settings do not parse; running on what was in force",
+					"version", doc.Version, "err", err)
+				w.cur.version, w.cur.fileHash = version, hash
+				return
+			}
+		}
+		eff = config.Layer(file, cluster)
+	}
+	cfg, err := config.FromMap(eff)
+	if err != nil {
+		w.log.Error("the changed settings do not load on this machine; running on what was in force",
+			"version", version, "err", err)
+		w.cur.version, w.cur.fileHash = version, hash
+		return
+	}
+	changed := config.Changed(w.cur.eff, eff)
+	w.cur = clusterSettings{cfg: cfg, eff: eff, cluster: cluster, version: version, fileHash: hash}
+	if len(changed) == 0 {
+		return
+	}
+	if w.live != nil && w.live(cfg, changed) {
+		w.log.Info("settings changed and in force", "version", version, "changed", strings.Join(changed, ", "))
+		return
+	}
+	w.restart(fmt.Sprintf("settings version %d changed %s", version, strings.Join(changed, ", ")))
+}
+
+// liveSettings are the keys a control plane puts in force without starting
+// again: what it reads at every use, or not at all.
+var liveSettings = []string{
+	"guard",
+	// Stopping and starting the service (llmgw stop, llmgw start).
+	"serving",
+	// Sent to the nodes and this machine's firewall unit as they change.
+	"firewall", "firewall_networks",
+	// Read by server init, node add and backups, not by the server.
+	"backup_recipients", "backup_copy_to",
+}
+
+// onlyLive is whether every changed key is one of liveSettings.
+func onlyLive(changed []string) bool {
+	for _, c := range changed {
+		top, _, _ := strings.Cut(c, ".")
+		if !slices.Contains(liveSettings, top) {
+			return false
+		}
+	}
+	return true
+}
+
+// checkClusterSettings refuses a settings document that would not load,
+// laid over the server.yaml at path, or that sets one machine's keys.
+func checkClusterSettings(path string, body []byte) error {
+	cluster, err := config.Parse(body)
+	if err != nil {
+		return err
+	}
+	if err := config.CheckCluster(cluster); err != nil {
+		return err
+	}
+	_, file, err := readSettingsFile(path)
+	if err != nil {
+		return err
+	}
+	_, err = config.FromMap(config.Layer(file, cluster))
+	return err
+}
+
+// loadEffectiveConfig is what the control plane on this machine runs on:
+// its server.yaml with the cluster's settings from the store laid over it,
+// for the commands beside it that read cluster keys — init, node add,
+// backups. The file alone when the store cannot be reached or holds none.
+func loadEffectiveConfig(path string) (config.Server, error) {
+	_, file, err := readSettingsFile(path)
+	if err != nil {
+		return config.Default(), err
+	}
+	fileCfg, err := config.FromMap(file)
+	if err != nil {
+		return fileCfg, fileError(path, err)
+	}
+	cluster, ok := storedSettings(fileCfg)
+	if !ok {
+		return fileCfg, nil
+	}
+	cfg, err := config.FromMap(config.Layer(file, cluster))
+	if err != nil {
+		return fileCfg, nil //nolint:nilerr // the file still says what this machine is
+	}
+	return cfg, nil
+}
+
+// storedSettings are the cluster's settings, when the store a configuration
+// names can be reached and holds some.
+func storedSettings(cfg config.Server) (map[string]any, bool) {
+	if cfg.Auth.Store == "" {
+		return nil, false
+	}
+	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+	defer cancel()
+	st, err := storepkg.Open(ctx, cfg.Auth.Store)
+	if err != nil {
+		return nil, false
+	}
+	defer st.Close()
+	doc, err := st.ClusterSettings(ctx)
+	if err != nil || doc.Version == 0 {
+		return nil, false
+	}
+	cluster, err := config.Parse([]byte(doc.Body))
+	return cluster, err == nil
+}
+
+// changeStoredSettings changes the cluster's settings from a command run on
+// a control plane (init), when the store holds them: changed is false when
+// it does not, and the file is then where the change belongs.
+func changeStoredSettings(path, note string, change func(doc *settings.Doc) error) (changed bool, err error) {
+	_, file, err := readSettingsFile(path)
+	if err != nil {
+		return false, err
+	}
+	fileCfg, err := config.FromMap(file)
+	if err != nil || fileCfg.Auth.Store == "" {
+		return false, nil //nolint:nilerr // no store to keep them in
+	}
+	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+	defer cancel()
+	st, err := storepkg.Open(ctx, fileCfg.Auth.Store)
+	if err != nil {
+		return false, nil //nolint:nilerr // not set up yet: the file it is
+	}
+	defer st.Close()
+	cur, err := st.ClusterSettings(ctx)
+	if err != nil || cur.Version == 0 {
+		return false, err
+	}
+	doc, err := settings.FromBytes("the cluster's settings", []byte(cur.Body))
+	if err != nil {
+		return false, err
+	}
+	if err := change(doc); err != nil {
+		return false, err
+	}
+	body, err := doc.Bytes()
+	if err != nil {
+		return false, err
+	}
+	if err := checkClusterSettings(path, body); err != nil {
+		return false, err
+	}
+	who := "llmgw"
+	if u := os.Getenv("SUDO_USER"); u != "" {
+		who = u
+	}
+	_, err = st.PutClusterSettings(ctx, cur.Version, string(body), who, note)
+	return err == nil, err
+}

internal/cli/clustersettings_test.go

@@ -0,0 +1,139 @@
+package cli
+
+import (
+	"context"
+	"io"
+	"log/slog"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	storepkg "gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+func settingsFixture(t *testing.T, extra string) (path, store string) {
+	t.Helper()
+	dir := t.TempDir()
+	store = filepath.Join(dir, "llmgw.db")
+	path = filepath.Join(dir, "server.yaml")
+	body := "listen: \":8080\"\nagent_insecure: true\nauth:\n  enabled: true\n  store: " + store +
+		"\nrouter:\n  retries: 2\n" + extra
+	if err := os.WriteFile(path, []byte(body), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	return path, store
+}
+
+var quiet = slog.New(slog.NewTextHandler(io.Discard, nil))
+
+func TestTheFirstStartBringsTheFilesSettingsInWithGuardrails(t *testing.T) {
+	path, store := settingsFixture(t, "")
+	s, err := loadClusterSettings(t.Context(), path, quiet)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if s.version != 1 || s.cfg.Router.Retries != 2 {
+		t.Errorf("version %d retries %d", s.version, s.cfg.Router.Retries)
+	}
+	if len(s.cfg.Guard.Policies) == 0 || s.cfg.Guard.Policies["baseline"].Judge != nil {
+		t.Errorf("a file without guard got %+v, want the baseline without a judge", s.cfg.Guard)
+	}
+	st, _ := storepkg.Open(t.Context(), store)
+	defer st.Close()
+	doc, _ := st.ClusterSettings(t.Context())
+	if strings.Contains(doc.Body, "listen") || strings.Contains(doc.Body, store) || !strings.Contains(doc.Body, "retries: 2") {
+		t.Errorf("the store holds a machine's keys, or lost the cluster's:\n%s", doc.Body)
+	}
+
+	// From then on the store's settings are the cluster's, whatever a
+	// file says.
+	_ = os.WriteFile(path, []byte(strings.Replace(mustRead(t, path), "retries: 2", "retries: 9", 1)), 0o600)
+	again, err := loadClusterSettings(t.Context(), path, quiet)
+	if err != nil || again.version != 1 || again.cfg.Router.Retries != 2 {
+		t.Errorf("second start: version %d retries %d err %v", again.version, again.cfg.Router.Retries, err)
+	}
+}
+
+func mustRead(t *testing.T, p string) string {
+	b, err := os.ReadFile(p)
+	if err != nil {
+		t.Fatal(err)
+	}
+	return string(b)
+}
+
+func TestEveryChangeIsAppliedLiveOrByStartingAgain(t *testing.T) {
+	path, store := settingsFixture(t, "")
+	s, err := loadClusterSettings(t.Context(), path, quiet)
+	if err != nil {
+		t.Fatal(err)
+	}
+	st, _ := storepkg.Open(t.Context(), store)
+	defer st.Close()
+	var lived [][]string
+	var restarts []string
+	w := &settingsWatch{path: path, st: st, cur: s, log: quiet,
+		live: func(_ config.Server, changed []string) bool {
+			lived = append(lived, changed)
+			return onlyLive(changed)
+		},
+		restart: func(why string) { restarts = append(restarts, why) },
+	}
+	ctx := context.Background()
+	put := func(body string) {
+		t.Helper()
+		v, _ := st.ClusterSettingsVersion(ctx)
+		if _, err := st.PutClusterSettings(ctx, v, body, "test", ""); err != nil {
+			t.Fatal(err)
+		}
+	}
+	cur, _ := st.ClusterSettings(ctx)
+
+	w.check(ctx) // nothing changed
+	if len(lived)+len(restarts) != 0 {
+		t.Fatalf("acted on no change: %v %v", lived, restarts)
+	}
+
+	put(strings.Replace(cur.Body, "injection: flag", "injection: block", 1))
+	w.check(ctx)
+	if len(restarts) != 0 || len(lived) != 1 || lived[0][0] != "guard" {
+		t.Fatalf("a guard change: lived %v restarts %v", lived, restarts)
+	}
+
+	cur, _ = st.ClusterSettings(ctx)
+	put(strings.Replace(cur.Body, "retries: 2", "retries: 4", 1))
+	w.check(ctx)
+	if len(restarts) != 1 || !strings.Contains(restarts[0], "router") || w.cur.cfg.Router.Retries != 4 {
+		t.Fatalf("a router change: restarts %v retries %d", restarts, w.cur.cfg.Router.Retries)
+	}
+
+	// A machine's own key, in its file.
+	_ = os.WriteFile(path, []byte(strings.Replace(mustRead(t, path), `":8080"`, `":8090"`, 1)), 0o600)
+	w.check(ctx)
+	if len(restarts) != 2 || !strings.Contains(restarts[1], "listen") {
+		t.Fatalf("a listener change: %v", restarts)
+	}
+
+	// Settings that do not load are not put in force.
+	cur, _ = st.ClusterSettings(ctx)
+	put(cur.Body + "control:\n  timezone: Mars/Olympus\n")
+	w.check(ctx)
+	if len(restarts) != 2 || w.cur.cfg.Control.Timezone != "" {
+		t.Errorf("bad settings were applied: %v %q", restarts, w.cur.cfg.Control.Timezone)
+	}
+}
+
+func TestAClusterDocumentCannotSetAMachinesKeys(t *testing.T) {
+	path, _ := settingsFixture(t, "")
+	if err := checkClusterSettings(path, []byte("listen: \":1\"\n")); err == nil {
+		t.Error("listen was accepted")
+	}
+	if err := checkClusterSettings(path, []byte("control:\n  timezone: Mars/Olympus\n")); err == nil {
+		t.Error("a timezone that is not one was accepted")
+	}
+	if err := checkClusterSettings(path, []byte("router:\n  retries: 3\n")); err != nil {
+		t.Error(err)
+	}
+}

internal/cli/ctl.go

@@ -39,6 +39,14 @@ func newCtlClient(addr string, tokenFiles ...string) *ctlClient {
 	}
 }
 
+// patient is the same client waiting up to d for an answer, for a call the
+// server itself may spend minutes on.
+func (c *ctlClient) patient(d time.Duration) *ctlClient {
+	cp := *c
+	cp.http = &http.Client{Timeout: d}
+	return &cp
+}
+
 func (c *ctlClient) get(ctx context.Context, path string, out any) error {
 	return c.do(ctx, http.MethodGet, path, nil, out)
 }
@@ -108,6 +116,13 @@ func (c *ctlClient) do(ctx context.Context, method, path string, body, out any)
 	if out == nil {
 		return nil
 	}
+	// An older control plane answers a path it does not know with its
+	// dashboard: a page, which read as JSON said only "invalid character '<'".
+	if strings.HasPrefix(resp.Header.Get("Content-Type"), "text/html") {
+		return fmt.Errorf("%s %s answered with a page, not the API: the control plane answering "+
+			"(the leader, through a standby) does not know it — is it on an older release? "+
+			"llmgw ctl status says which it runs", method, path)
+	}
 	return json.NewDecoder(resp.Body).Decode(out)
 }
 
@@ -160,11 +175,12 @@ func newCtlCmd() *cobra.Command {
 		newJobsCmd(client, &asJSON),
 		newEngineCheckCmd(),
 		newEngineBuildCmd(client),
-		newToolCheckCmd(),
+		newToolCheckCmd(client),
 		newWeightsCmd(client, &asJSON),
 		newCapacityCmd(client, &asJSON),
 		newQuotaCmd(client, &asJSON),
 		newGuardCmd(client, &asJSON),
+		newConfigCmd(client, &asJSON),
 		newAuditCmd(client, &asJSON),
 		newNotifyCmd(client, &asJSON),
 		newUsageCmd(client, &asJSON),
@@ -216,8 +232,8 @@ func newNodesCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 					// What the node can run, which is the first thing to look
 					// at when a model will not place and the hardware looks
 					// fine.
-					join(n.Engines),
-					classesColumn(n), join(n.Fabrics), n.Replicas, n.LastSeenAgo)
+					enginesColumn(n),
+					classesColumn(n), fabricsColumn(n), n.Replicas, n.LastSeenAgo)
 			}
 			_ = w.Flush()
 
@@ -434,7 +450,7 @@ func newReplicasCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 			}
 
 			w := tab()
-			fmt.Fprintln(w, "REPLICA\tMODEL\tSTATE\tPOOL\tNODES\tAGE\tIDLE\tENDPOINT")
+			fmt.Fprintln(w, "REPLICA\tMODEL\tSTATE\tPOOL\tON\tNODES\tAGE\tIDLE\tENDPOINT")
 			for _, r := range reps {
 				var nodes []string
 				for _, s := range r.Shards {
@@ -452,8 +468,14 @@ func newReplicasCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 				if r.Restart != "" {
 					state += " (restart " + r.Restart + ")"
 				}
-				fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n",
-					r.ID, r.Model, state, r.Pool, join(nodes),
+				// What the engine computes on, by its own account: the first
+				// shard's, which serves.
+				on := "-"
+				if len(r.Shards) > 0 {
+					on = r.Shards[0].Compute.Short()
+				}
+				fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n",
+					r.ID, r.Model, state, r.Pool, on, join(nodes),
 					(time.Duration(r.AgeSec) * time.Second).String(),
 					idleColumn(r.IdleSec), r.Endpoint)
 				// A line rather than a column, because most replicas are not
@@ -463,7 +485,7 @@ func newReplicasCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 					if s.Fetch == nil {
 						continue
 					}
-					fmt.Fprintf(w, "  %s: fetching %s\t\t\t\t\t\t\t\n",
+					fmt.Fprintf(w, "  %s: fetching %s\t\t\t\t\t\t\t\t\n",
 						s.Node, fetchColumn(s.Fetch))
 				}
 			}
@@ -574,7 +596,42 @@ func emitJSON(v any) error {
 // more than one or MIG instances among them: a card cut into two 3g.40gb
 // instances beside a whole card is "a100-80g-mig-3g.40gb×2,a100-80g×1", where
 // the GPUS column's 3 alone would read as three of something.
+// fabricsColumn is the node's fabrics, and when they are not all up, which
+// adapters are down: what keeps it out of replicas spanning nodes.
+func fabricsColumn(n adminapi.NodeView) string {
+	s := join(n.Fabrics)
+	if n.MultiNode != "" && n.MultiNode != "ready" {
+		s += " (" + n.MultiNode + ")"
+	}
+	return s
+}
+
+// enginesColumn is each engine with its version: llamacpp b10991.
+func enginesColumn(n adminapi.NodeView) string {
+	if len(n.EngineVersions) == 0 {
+		return join(n.Engines)
+	}
+	parts := make([]string, 0, len(n.Engines))
+	for _, e := range n.Engines {
+		if v := n.EngineVersions[e]; v != "" && v != "unknown" {
+			e += " " + v
+		}
+		parts = append(parts, e)
+	}
+	return strings.Join(parts, ",")
+}
+
+// classesColumn is the node's GPU classes, with their compute capability
+// when they report one: 6000-24g (7.5).
 func classesColumn(n adminapi.NodeView) string {
+	s := classesOnly(n)
+	if len(n.ComputeCapabilities) > 0 && s != "-" {
+		s += " (" + strings.Join(n.ComputeCapabilities, ",") + ")"
+	}
+	return s
+}
+
+func classesOnly(n adminapi.NodeView) string {
 	counted := len(n.ClassGPUs) > 1
 	for _, c := range n.ClassGPUs {
 		counted = counted || c.MIG

internal/cli/ctl_config.go

@@ -0,0 +1,377 @@
+package cli
+
+import (
+	"bytes"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"strconv"
+	"strings"
+	"time"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
+)
+
+func newConfigCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
+	var version int64
+	c := &cobra.Command{
+		Use:   "config",
+		Short: "The cluster's settings: show, change, and go back",
+		Long: "Everything in server.yaml that is not one machine's own — guardrails,\n" +
+			"limits, pricing, routing, engines, notifications, the mirror's policy —\n" +
+			"is the cluster's, kept in the store and the same on every control plane.\n" +
+			"A change is checked, kept as a new version, and in force on every control\n" +
+			"plane within seconds: live where it can be (guard), otherwise each control\n" +
+			"plane starts again on it, standbys first and the leader last.\n\n" +
+			"With no subcommand, prints the settings in force.\n\n" +
+			"A machine's own keys (" + strings.Join(config.LocalKeys(), ", ") + ")\n" +
+			"stay in its /etc/llmgw/server.yaml; a change there is noticed and applied\n" +
+			"on that machine the same way.",
+		Example: "  llmgw ctl config\n" +
+			"  llmgw ctl config get router.retries\n" +
+			"  llmgw ctl config set router.retries 3 --note 'flaky upstream'\n" +
+			"  llmgw ctl config set firewall_networks '[10.20.0.0/16]'\n" +
+			"  llmgw ctl config edit\n" +
+			"  llmgw ctl config history\n" +
+			"  llmgw ctl config rollback 4",
+		Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			path := "/api/v1/settings"
+			if version > 0 {
+				path += "?version=" + strconv.FormatInt(version, 10)
+			}
+			var v adminapi.SettingsView
+			if err := client().get(cmd.Context(), path, &v); err != nil {
+				return err
+			}
+			if *asJSON {
+				return emitJSON(v)
+			}
+			fmt.Fprintf(cmd.ErrOrStderr(), "# version %d, %s by %s%s\n", v.Version, v.At.Local().Format(time.DateTime),
+				orUnknown(v.By), noteSuffix(v.Note))
+			fmt.Fprint(cmd.OutOrStdout(), v.Body)
+			return nil
+		},
+	}
+	c.Flags().Int64Var(&version, "version", 0, "an earlier version")
+	c.AddCommand(newConfigGetCmd(client), newConfigSetCmd(client), newConfigUnsetCmd(client),
+		newConfigEditCmd(client), newConfigHistoryCmd(client, asJSON), newConfigRollbackCmd(client))
+	return c
+}
+
+func orUnknown(s string) string {
+	if s == "" {
+		return "unknown"
+	}
+	return s
+}
+
+func noteSuffix(n string) string {
+	if n == "" {
+		return ""
+	}
+	return ": " + n
+}
+
+// keyPath reads a dotted key, refusing one machine's own.
+func keyPath(key string) ([]string, error) {
+	path := strings.Split(key, ".")
+	if config.IsLocal(path) {
+		return nil, fmt.Errorf("%s is each machine's own: change it in that machine's /etc/llmgw/server.yaml, "+
+			"where a change is applied by itself too", key)
+	}
+	return path, nil
+}
+
+func newConfigGetCmd(client func() *ctlClient) *cobra.Command {
+	return &cobra.Command{
+		Use:   "get KEY",
+		Short: "One key of the cluster's settings, as YAML",
+		Args:  cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			path, err := keyPath(args[0])
+			if err != nil {
+				return err
+			}
+			var v adminapi.SettingsView
+			if err := client().get(cmd.Context(), "/api/v1/settings", &v); err != nil {
+				return err
+			}
+			doc, err := settings.FromBytes("settings", []byte(v.Body))
+			if err != nil {
+				return err
+			}
+			n := doc.Node(path)
+			if n == nil {
+				return fmt.Errorf("%s is not set in the cluster's settings; its default applies (docs/reference.md)", args[0])
+			}
+			sub, err := settings.FromBytes("value", nil)
+			if err != nil {
+				return err
+			}
+			if err := sub.SetNode([]string{"v"}, n); err != nil {
+				return err
+			}
+			b, err := sub.Bytes()
+			if err != nil {
+				return err
+			}
+			out := strings.TrimPrefix(string(b), "v:")
+			fmt.Fprintln(cmd.OutOrStdout(), strings.TrimSpace(dedent(out)))
+			return nil
+		},
+	}
+}
+
+// dedent takes off the two spaces a block nested under one key carries.
+func dedent(s string) string {
+	lines := strings.Split(s, "\n")
+	for i, l := range lines {
+		lines[i] = strings.TrimPrefix(l, "  ")
+	}
+	return strings.Join(lines, "\n")
+}
+
+// changeSettings reads the newest settings, changes them, and puts the
+// result in force, saying what that does.
+func changeSettings(cmd *cobra.Command, client func() *ctlClient, note string,
+	change func(doc *settings.Doc) error,
+) error {
+	ctx := cmd.Context()
+	var cur adminapi.SettingsView
+	if err := client().get(ctx, "/api/v1/settings", &cur); err != nil {
+		return err
+	}
+	doc, err := settings.FromBytes("settings", []byte(cur.Body))
+	if err != nil {
+		return err
+	}
+	if err := change(doc); err != nil {
+		return err
+	}
+	b, err := doc.Bytes()
+	if err != nil {
+		return err
+	}
+	return putSettings(cmd, client, cur, b, note)
+}
+
+func putSettings(cmd *cobra.Command, client func() *ctlClient, cur adminapi.SettingsView, body []byte, note string) error {
+	before, _ := config.Parse([]byte(cur.Body))
+	after, err := config.Parse(body)
+	if err != nil {
+		return err
+	}
+	changed := config.Changed(before, after)
+	if len(changed) == 0 {
+		fmt.Fprintln(cmd.OutOrStdout(), "nothing changed")
+		return nil
+	}
+	req, _ := json.Marshal(adminapi.SettingsRequest{Base: cur.Version, Body: string(body), Note: note})
+	var v adminapi.SettingsView
+	if err := client().put(cmd.Context(), "/api/v1/settings", req, &v); err != nil {
+		return err
+	}
+	fmt.Fprintf(cmd.OutOrStdout(), "version %d: %s\n", v.Version, strings.Join(changed, ", "))
+	if onlyLive(changed) {
+		fmt.Fprintln(cmd.OutOrStdout(), "in force on every control plane within seconds")
+	} else {
+		fmt.Fprintln(cmd.OutOrStdout(), "every control plane starts again on it within seconds, standbys first, "+
+			"the leader half a minute later; engines keep serving")
+	}
+	return nil
+}
+
+func newConfigSetCmd(client func() *ctlClient) *cobra.Command {
+	var note string
+	c := &cobra.Command{
+		Use:   "set KEY VALUE",
+		Short: "Change one key of the cluster's settings; VALUE is YAML",
+		Example: "  llmgw ctl config set router.retries 3\n" +
+			"  llmgw ctl config set guard.default '[baseline, finance]'\n" +
+			"  llmgw ctl config set notify.webhooks '[{url: https://hooks.example/llmgw}]'",
+		Args: cobra.ExactArgs(2),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			path, err := keyPath(args[0])
+			if err != nil {
+				return err
+			}
+			return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
+				return doc.SetYAML(path, args[1])
+			})
+		},
+	}
+	c.Flags().StringVar(&note, "note", "", "why, kept with the version")
+	return c
+}
+
+func newConfigUnsetCmd(client func() *ctlClient) *cobra.Command {
+	var note string
+	c := &cobra.Command{
+		Use:   "unset KEY",
+		Short: "Remove one key from the cluster's settings, so its default applies",
+		Args:  cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			path, err := keyPath(args[0])
+			if err != nil {
+				return err
+			}
+			return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
+				if doc.Node(path) == nil {
+					return fmt.Errorf("%s is not set", args[0])
+				}
+				doc.Unset(path)
+				return nil
+			})
+		},
+	}
+	c.Flags().StringVar(&note, "note", "", "why, kept with the version")
+	return c
+}
+
+func newConfigEditCmd(client func() *ctlClient) *cobra.Command {
+	var note string
+	c := &cobra.Command{
+		Use:   "edit",
+		Short: "Edit the cluster's settings in $EDITOR",
+		Long: "Opens the settings in force in $EDITOR (vi when unset). Saved, they are\n" +
+			"checked and kept as a new version; a document that does not load is\n" +
+			"opened again with the reason, until it does or is left unchanged.",
+		Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			var cur adminapi.SettingsView
+			if err := client().get(cmd.Context(), "/api/v1/settings", &cur); err != nil {
+				return err
+			}
+			dir, err := os.MkdirTemp("", "llmgw-settings")
+			if err != nil {
+				return err
+			}
+			defer os.RemoveAll(dir)
+			file := filepath.Join(dir, "settings.yaml")
+			body := []byte(cur.Body)
+			problem := ""
+			for {
+				text := body
+				if problem != "" {
+					text = append([]byte("# NOT SAVED: "+strings.ReplaceAll(problem, "\n", " ")+"\n"), stripProblem(body)...)
+				}
+				if err := os.WriteFile(file, text, 0o600); err != nil {
+					return err
+				}
+				if err := runEditor(file, cmd.InOrStdin(), cmd.OutOrStdout(), cmd.ErrOrStderr()); err != nil {
+					return err
+				}
+				edited, err := os.ReadFile(file) //nolint:gosec // written above
+				if err != nil {
+					return err
+				}
+				edited = stripProblem(edited)
+				if bytes.Equal(bytes.TrimSpace(edited), bytes.TrimSpace([]byte(cur.Body))) {
+					fmt.Fprintln(cmd.OutOrStdout(), "nothing changed")
+					return nil
+				}
+				err = putSettings(cmd, client, cur, edited, note)
+				if err == nil {
+					return nil
+				}
+				if strings.Contains(err.Error(), "changed since they were read") {
+					// Somebody else's change came first: opening the editor
+					// again on the old base would only be refused again.
+					return fmt.Errorf("%w; your edit is not saved", err)
+				}
+				body, problem = edited, err.Error()
+			}
+		},
+	}
+	c.Flags().StringVar(&note, "note", "", "why, kept with the version")
+	return c
+}
+
+func stripProblem(b []byte) []byte {
+	if bytes.HasPrefix(b, []byte("# NOT SAVED: ")) {
+		if i := bytes.IndexByte(b, '\n'); i >= 0 {
+			return b[i+1:]
+		}
+	}
+	return b
+}
+
+func runEditor(file string, in io.Reader, out, errOut io.Writer) error {
+	editor := os.Getenv("VISUAL")
+	if editor == "" {
+		editor = os.Getenv("EDITOR")
+	}
+	if editor == "" {
+		editor = "vi"
+	}
+	parts := strings.Fields(editor)
+	c := exec.Command(parts[0], append(parts[1:], file)...) //nolint:gosec // the operator's own editor
+	c.Stdin, c.Stdout, c.Stderr = in, out, errOut
+	if err := c.Run(); err != nil {
+		return fmt.Errorf("%s: %w", editor, err)
+	}
+	return nil
+}
+
+func newConfigHistoryCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
+	var limit int
+	c := &cobra.Command{
+		Use:   "history",
+		Short: "Versions of the cluster's settings, newest first",
+		Args:  cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			var list []adminapi.SettingsView
+			if err := client().get(cmd.Context(), "/api/v1/settings/history?limit="+strconv.Itoa(limit), &list); err != nil {
+				return err
+			}
+			if *asJSON {
+				return emitJSON(list)
+			}
+			w := tabTo(cmd.OutOrStdout())
+			fmt.Fprintln(w, "VERSION\tWHEN\tBY\tNOTE")
+			for _, v := range list {
+				fmt.Fprintf(w, "%d\t%s\t%s\t%s\n", v.Version, v.At.Local().Format(time.DateTime), orUnknown(v.By), v.Note)
+			}
+			return w.Flush()
+		},
+	}
+	c.Flags().IntVar(&limit, "limit", 20, "how many")
+	return c
+}
+
+func newConfigRollbackCmd(client func() *ctlClient) *cobra.Command {
+	var note string
+	c := &cobra.Command{
+		Use:   "rollback VERSION",
+		Short: "Put an earlier version of the cluster's settings back in force, as a new version",
+		Args:  cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			n, err := strconv.ParseInt(args[0], 10, 64)
+			if err != nil || n < 1 {
+				return errors.New("VERSION is a number from 'llmgw ctl config history'")
+			}
+			var cur, old adminapi.SettingsView
+			if err := client().get(cmd.Context(), "/api/v1/settings", &cur); err != nil {
+				return err
+			}
+			if err := client().get(cmd.Context(), "/api/v1/settings?version="+args[0], &old); err != nil {
+				return err
+			}
+			if note == "" {
+				note = "back to version " + args[0]
+			}
+			return putSettings(cmd, client, cur, []byte(old.Body), note)
+		},
+	}
+	c.Flags().StringVar(&note, "note", "", "why, kept with the version")
+	return c
+}

internal/cli/ctl_config_test.go

@@ -0,0 +1,22 @@
+package cli
+
+import (
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+)
+
+func TestAPageInsteadOfTheAPIIsSaidPlainly(t *testing.T) {
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		w.Header().Set("Content-Type", "text/html; charset=utf-8")
+		_, _ = w.Write([]byte("<!doctype html><title>llmgw</title>"))
+	}))
+	defer srv.Close()
+	c := newCtlClient(strings.TrimPrefix(srv.URL, "http://"), "")
+	var out map[string]any
+	err := c.get(t.Context(), "/api/v1/settings", &out)
+	if err == nil || !strings.Contains(err.Error(), "older release") || strings.Contains(err.Error(), "invalid character") {
+		t.Errorf("%v", err)
+	}
+}

internal/cli/ctl_request.go

@@ -76,8 +76,11 @@ func newRequestCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 				fmt.Fprintf(out, "tags      %s\n", strings.Join(kv, ", "))
 			}
 			if v.GuardAction != "" {
-				fmt.Fprintf(out, "guard     %s by policy %s (%s), %d ms\n", v.GuardAction, v.GuardPolicy,
-					or(v.GuardCategory, "-"), v.GuardMs)
+				what := or(v.GuardCategory, "-")
+				if v.GuardRule != "" {
+					what += ": " + v.GuardRule
+				}
+				fmt.Fprintf(out, "guard     %s by policy %s (%s), %d ms\n", v.GuardAction, v.GuardPolicy, what, v.GuardMs)
 			} else if v.GuardMs > 0 {
 				fmt.Fprintf(out, "guard     allowed, %d ms\n", v.GuardMs)
 			}

internal/cli/ctl_restart.go

@@ -12,9 +12,10 @@ import (
 
 func newRestartCmd(client func() *ctlClient) *cobra.Command {
 	var (
-		models  []string
-		all     bool
-		inPlace bool
+		models             []string
+		all                bool
+		inPlace            bool
+		surge, unavailable int
 	)
 	c := &cobra.Command{
 		Use:   "restart [REPLICA...]",
@@ -31,6 +32,10 @@ func newRestartCmd(client func() *ctlClient) *cobra.Command {
 			"profile edited to need more than any node has — the restart is given up\n" +
 			"and the old replica keeps serving: llmgw ctl replicas says so, and why.\n\n" +
 			"--in-place stops each replica first, whatever room there is.\n\n" +
+			"A model's replicas are replaced one at a time. --max-surge N replaces N at\n" +
+			"once, each beside itself, needing room for N more; --max-unavailable N\n" +
+			"stops N at once first, needing no room and leaving N fewer serving while\n" +
+			"they load. Together, the first N are stopped first and the rest surged.\n\n" +
 			"  llmgw ctl restart qwen2.5-7b-3f2a91c0\n" +
 			"  llmgw ctl restart --model qwen2.5-7b\n" +
 			"  llmgw ctl restart --all",
@@ -38,7 +43,8 @@ func newRestartCmd(client func() *ctlClient) *cobra.Command {
 			if len(args) == 0 && len(models) == 0 && !all {
 				return errors.New("name replicas, --model, or --all")
 			}
-			body, _ := json.Marshal(adminapi.RestartRequest{Replicas: args, Models: models, All: all, InPlace: inPlace})
+			body, _ := json.Marshal(adminapi.RestartRequest{Replicas: args, Models: models, All: all, InPlace: inPlace,
+				MaxSurge: surge, MaxUnavailable: unavailable})
 			var res adminapi.RestartResult
 			if err := client().post(cmd.Context(), "/api/v1/restart", body, &res); err != nil {
 				return err
@@ -50,6 +56,9 @@ func newRestartCmd(client func() *ctlClient) *cobra.Command {
 	c.Flags().StringArrayVar(&models, "model", nil, "every replica of this model; repeatable")
 	c.Flags().BoolVar(&all, "all", false, "every ready replica in the cluster")
 	c.Flags().BoolVar(&inPlace, "in-place", false, "stop each replica before starting its replacement")
+	c.Flags().IntVar(&surge, "max-surge", 0, "replicas of a model replaced beside themselves at once (default 1)")
+	c.Flags().IntVar(&unavailable, "max-unavailable", 0,
+		"replicas of a model stopped first at once, each out while its replacement loads")
 	return c
 }
 

internal/cli/ctl_upgrade.go

@@ -68,14 +68,26 @@ func newUpgradeCmd(client func() *ctlClient) *cobra.Command {
 				}
 			}
 			if !skipCP {
-				if othersCP {
-					if err := upgradeOtherControlPlanes(ctx, out, client, v, releases, sshUser); err != nil {
+				// The leader last, wherever this runs: every other control
+				// plane is a standby, upgraded with nothing moving, and the
+				// leader's restart hands over once, to a machine already on
+				// the release.
+				others := func() error {
+					if !othersCP {
+						return nil
+					}
+					return upgradeOtherControlPlanes(ctx, out, client, v, releases, sshUser)
+				}
+				self := func() error { return upgradeControlPlane(ctx, out, client, v, releases, serverSvc) }
+				order := []func() error{others, self}
+				if !thisLeads(ctx, client) {
+					order = []func() error{self, others}
+				}
+				for _, step := range order {
+					if err := step(); err != nil {
 						return err
 					}
 				}
-				if err := upgradeControlPlane(ctx, out, client, v, releases, serverSvc); err != nil {
-					return err
-				}
 			}
 			return upgradeNodes(ctx, out, client, v, nodes, perNode, parallel)
 		},
@@ -92,10 +104,23 @@ func newUpgradeCmd(client func() *ctlClient) *cobra.Command {
 	return c
 }
 
+// thisLeads says whether this machine's control plane leads; one that does
+// not say, alone or older, is taken to.
+func thisLeads(ctx context.Context, client func() *ctlClient) bool {
+	var sum adminapi.Summary
+	if err := client().get(ctx, "/api/v1/summary", &sum); err != nil {
+		return true
+	}
+	for _, cp := range sum.ControlPlanes.Members {
+		if cp.This {
+			return cp.State == "leading"
+		}
+	}
+	return true
+}
+
 // upgradeOtherControlPlanes moves every other control plane of the cluster to
-// v, over ssh with `llmgw upgrade`, before this machine: each is a standby, or
-// becomes one when this one is restarted, and the leadership should move to
-// a machine already on the release. Nothing is changed here if one cannot be
+// v, over ssh with `llmgw upgrade`. Nothing is changed here if one cannot be
 // reached.
 func upgradeOtherControlPlanes(ctx context.Context, out io.Writer, client func() *ctlClient, v, releases, user string) error {
 	var sum adminapi.Summary
@@ -127,8 +152,20 @@ func upgradeOtherControlPlanes(ctx context.Context, out io.Writer, client func()
 	return nil
 }
 
+// controlPlaneStep is what this machine needs to run v: the binary installed,
+// the service restarted, both, or neither. The binary is this command's own;
+// running is what the server says it runs, "" from one too old to say. A
+// binary put in place by hand is not a server running it.
+func controlPlaneStep(binary, running, v string) (install, restart bool) {
+	install = binary != v
+	return install, install || running != v
+}
+
 func upgradeControlPlane(ctx context.Context, out io.Writer, client func() *ctlClient, v, releases, svc string) error {
-	if version.Version == v {
+	var sum adminapi.Summary
+	_ = client().get(ctx, "/api/v1/summary", &sum)
+	install, restart := controlPlaneStep(version.Version, sum.Version, v)
+	if !restart {
 		fmt.Fprintf(out, "this machine already runs %s\n", v)
 		return nil
 	}
@@ -140,12 +177,20 @@ func upgradeControlPlane(ctx context.Context, out io.Writer, client func() *ctlC
 	if err != nil {
 		return err
 	}
-	fmt.Fprintf(out, "== this machine: %s -> %s\n", version.Version, v)
-	backup, err := release.Install(ctx, nil, releases, v, runtime.GOOS, runtime.GOARCH, self)
-	if err != nil {
-		return err
+	backup := "the binary installed by hand before it"
+	if install {
+		fmt.Fprintf(out, "== this machine: %s -> %s\n", version.Version, v)
+		if backup, err = release.Install(ctx, nil, releases, v, runtime.GOOS, runtime.GOARCH, self); err != nil {
+			return err
+		}
+		fmt.Fprintf(out, "   installed, checksum verified; the old binary is %s\n", backup)
+	} else {
+		running := sum.Version
+		if running == "" {
+			running = "an older release"
+		}
+		fmt.Fprintf(out, "== this machine: %s is installed, and %s still runs it; restarting %s\n", v, running, svc)
 	}
-	fmt.Fprintf(out, "   installed, checksum verified; the old binary is %s\n", backup)
 	var before []adminapi.NodeView
 	_ = client().get(ctx, "/api/v1/nodes", &before)
 	if b, err := exec.CommandContext(ctx, "systemctl", "restart", svc).CombinedOutput(); err != nil { //nolint:gosec // fixed command
@@ -245,7 +290,10 @@ func upgradeWave(ctx context.Context, out io.Writer, client func() *ctlClient, v
 	body, _ := json.Marshal(adminapi.UpgradeRequest{Version: v})
 	var failed []string
 	for _, n := range wave {
-		if err := client().post(ctx, "/api/v1/nodes/"+n.ID+"/upgrade", body, nil); err != nil {
+		// The control plane waits for the agent to fetch and check the
+		// release, up to five minutes; fifteen seconds gave up on a node
+		// that was upgrading.
+		if err := client().patient(6*time.Minute).post(ctx, "/api/v1/nodes/"+n.ID+"/upgrade", body, nil); err != nil {
 			failed = append(failed, fmt.Sprintf("%s: %v", n.ID, err))
 		}
 	}

internal/cli/ctl_upgrade_test.go

@@ -111,3 +111,42 @@ func TestAnUpgradeGoesInWavesThatGrowUpToTheLimit(t *testing.T) {
 		t.Errorf("waves %s:\n%s", got, out.String())
 	}
 }
+
+func TestAnUpgradeKnowsWhetherItRunsOnTheLeader(t *testing.T) {
+	sum := adminapi.Summary{}
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		_ = json.NewEncoder(w).Encode(sum)
+	}))
+	defer srv.Close()
+	client := func() *ctlClient { return newCtlClient(srv.URL) }
+	if !thisLeads(t.Context(), client) {
+		t.Error("a control plane with no rival does not lead")
+	}
+	sum.ControlPlanes.Members = []adminapi.ControlPlane{{Name: "a", State: "leading"}, {Name: "b", State: "standing by", This: true}}
+	if thisLeads(t.Context(), client) {
+		t.Error("a standby took itself for the leader: it would upgrade the leader before itself")
+	}
+	sum.ControlPlanes.Members[0].This, sum.ControlPlanes.Members[1].This = true, false
+	if !thisLeads(t.Context(), client) {
+		t.Error("the leader took itself for a standby")
+	}
+}
+
+func TestABinaryInstalledByHandIsNotAServerRunningIt(t *testing.T) {
+	// The binary put in place with install, llmgw-server not restarted: the
+	// upgrade said "already runs" and left the old server running.
+	for _, c := range []struct {
+		binary, running  string
+		install, restart bool
+	}{
+		{"v1.0.0", "v1.0.0", true, true},  // the usual: fetch it, restart
+		{"v1.0.1", "v1.0.0", false, true}, // installed by hand: restart only
+		{"v1.0.1", "", false, true},       // a server too old to say
+		{"v1.0.1", "v1.0.1", false, false},
+	} {
+		install, restart := controlPlaneStep(c.binary, c.running, "v1.0.1")
+		if install != c.install || restart != c.restart {
+			t.Errorf("binary %s, running %q: install %v restart %v", c.binary, c.running, install, restart)
+		}
+	}
+}

internal/cli/doctor.go

@@ -63,9 +63,10 @@ that file when the report is not enough to go on.`,
 				enc := json.NewEncoder(os.Stdout)
 				enc.SetIndent("", "  ")
 				return enc.Encode(struct {
-					Findings []doctor.Finding `json:"findings"`
-					Facts    doctor.Facts     `json:"facts"`
-				}{findings, facts})
+					Findings  []doctor.Finding `json:"findings"`
+					Readiness doctor.Readiness `json:"readiness"`
+					Facts     doctor.Facts     `json:"facts"`
+				}{findings, doctor.ReadinessOf(findings), facts})
 			}
 			printDoctor(os.Stdout, facts, findings, verbose)
 			for _, f := range findings {
@@ -101,12 +102,26 @@ func printDoctor(out io.Writer, f doctor.Facts, findings []doctor.Finding, verbo
 	fmt.Fprintf(out, "%s · %s %s %s · driver %s\n\n", f.Hostname, f.Host.OSID, f.Host.OSVersion,
 		f.Host.Arch, orDash(f.Host.NVIDIADriver))
 
-	for _, x := range findings {
-		fmt.Fprintf(out, "%-4s  %-6s  %s\n", x.Level, x.Area, x.Message)
-		if x.Fix != "" && (x.Level == doctor.Fail || x.Level == doctor.Warn) {
-			fmt.Fprintf(out, "              → %s\n", x.Fix)
+	// The node first; then what replicas spanning nodes would need, which
+	// keeps a node out only when a fabric was declared for it.
+	multi := false
+	for _, pass := range []bool{false, true} {
+		for _, x := range findings {
+			if x.MultiNode() != pass {
+				continue
+			}
+			if pass && !multi {
+				fmt.Fprintln(out, "\nMulti-node (fabrics, RDMA) — for replicas spanning nodes only:")
+				multi = true
+			}
+			fmt.Fprintf(out, "%-4s  %-6s  %s\n", x.Level, x.Area, x.Message)
+			if x.Fix != "" && (x.Level == doctor.Fail || x.Level == doctor.Warn) {
+				fmt.Fprintf(out, "              → %s\n", x.Fix)
+			}
 		}
 	}
+	r := doctor.ReadinessOf(findings)
+	fmt.Fprintf(out, "\nThis node:  %s\nMulti-node: %s\n", r.Node, r.MultiNode)
 
 	w := tabwriter.NewWriter(out, 0, 0, 2, ' ', 0)
 	if len(f.GPUs) > 0 {

internal/cli/enginecheck.go

@@ -4,10 +4,15 @@ import (
 	"context"
 	"fmt"
 	"io"
+	"net"
+	"net/http"
+	"os"
 	"os/exec"
 	"path/filepath"
 	"sort"
+	"strconv"
 	"strings"
+	"sync"
 	"time"
 
 	"github.com/spf13/cobra"
@@ -22,6 +27,7 @@ func newEngineCheckCmd() *cobra.Command {
 		nodes   int
 		tp      int
 		perNode int
+		model   string
 	)
 
 	cmd := &cobra.Command{
@@ -31,6 +37,10 @@ func newEngineCheckCmd() *cobra.Command {
 			"engines rename and remove flags between versions. When that happens the\n" +
 			"failure arrives minutes into a model load, in an engine's own error\n" +
 			"message, on a node nobody was watching.\n\n" +
+			"For llama.cpp on a machine with NVIDIA GPUs it then loads a small GGUF\n" +
+			"(--model, or the smallest in the weights cache) with every layer on the GPU\n" +
+			"and asks for one token: a build whose CUDA backend does not load computes on\n" +
+			"the CPU, and that is where it is caught.\n\n" +
 			"This asks the engine itself. It runs the engine's --help and checks every\n" +
 			"flag llmgw would pass against it, which needs no GPU — a CLI prints its\n" +
 			"help on any machine. Run it after upgrading an engine.\n\n" +
@@ -49,6 +59,11 @@ func newEngineCheckCmd() *cobra.Command {
 			// rarely on PATH: a check that cannot find the engine reports
 			// nothing rather than reporting a problem.
 			cfg := engines.Config{}
+			// The build the agent installed, when none is named: the newest
+			// under /opt/llmgw/engines, which is where every node keeps it.
+			if bin == "" && name == "llamacpp" {
+				bin = installedLlama()
+			}
 			if bin != "" {
 				switch name {
 				case "llamacpp":
@@ -135,6 +150,11 @@ func newEngineCheckCmd() *cobra.Command {
 			var deviceErr error
 			if fb, ok := eng.(engines.CPUFallback); ok && help == "" {
 				deviceErr = checkDevices(cmd.Context(), name, fb, out)
+				// Seeing a GPU is not computing on it: a small model loaded
+				// on it and asked for a token is.
+				if b, ok := eng.(interface{ Binary() string }); ok && deviceErr == nil && model != "none" {
+					deviceErr = computeOnGPU(cmd.Context(), b.Binary(), model, out)
+				}
 			}
 
 			fmt.Fprintf(out, "%s: %d flags checked", name, checked)
@@ -160,7 +180,9 @@ func newEngineCheckCmd() *cobra.Command {
 
 	f := cmd.Flags()
 	f.StringVar(&help, "help-output", "", "a saved --help to check against, instead of running the engine")
-	f.StringVar(&bin, "bin", "", "where the engine lives, when it is not on PATH")
+	f.StringVar(&bin, "bin", "", "where the engine lives (default: the build the agent installed, else PATH)")
+	f.StringVar(&model, "model", "", "a GGUF file to load on the GPU and ask for a token (default: the smallest "+
+		"in this node's weights cache; none skips it)")
 	f.IntVar(&nodes, "nodes", 2, "shape the command for a replica of this many nodes")
 	f.IntVar(&tp, "tp", 8, "tensor parallel degree")
 	f.IntVar(&perNode, "gpus-per-node", 4, "GPUs each node contributes")
@@ -286,3 +308,146 @@ func checkDevices(ctx context.Context, name string, fb engines.CPUFallback, out
 	fmt.Fprintf(out, "computes on: %s\n", strings.Join(devs, "; "))
 	return nil
 }
+
+// llamaRoot is where the agent installs llama.cpp builds.
+var llamaRoot = "/opt/llmgw/engines/llamacpp"
+
+// installedLlama is the llama-server the agent installed here: under
+// llamaRoot/<version-variant>/llama-<build>/, or one level up for older
+// layouts; on a machine with NVIDIA GPUs a CUDA build first, then the newest.
+func installedLlama() string {
+	var found []string
+	for _, pat := range []string{"*/llama-server", "*/*/llama-server"} {
+		m, _ := filepath.Glob(filepath.Join(llamaRoot, pat))
+		found = append(found, m...)
+	}
+	if len(found) == 0 {
+		return ""
+	}
+	sort.Strings(found)
+	if _, err := exec.LookPath("nvidia-smi"); err == nil {
+		for i := len(found) - 1; i >= 0; i-- {
+			if strings.Contains(found[i], "cuda") {
+				return found[i]
+			}
+		}
+	}
+	return found[len(found)-1]
+}
+
+// weightDirs are where a node keeps models it has fetched.
+var weightDirs = []string{"/var/lib/llmgw-weights", "/var/lib/llmgw-models", "/srv/models"}
+
+// smallestGGUF is the smallest GGUF file under dirs, for a check that only
+// needs something to load.
+func smallestGGUF(dirs []string) string {
+	best, size := "", int64(0)
+	for _, d := range dirs {
+		_ = filepath.WalkDir(d, func(p string, e os.DirEntry, err error) error {
+			if err != nil || e.IsDir() || !strings.HasSuffix(strings.ToLower(p), ".gguf") ||
+				strings.Contains(strings.ToLower(filepath.Base(p)), "mmproj") {
+				return nil
+			}
+			if info, err := e.Info(); err == nil && (best == "" || info.Size() < size) {
+				best, size = p, info.Size()
+			}
+			return nil
+		})
+	}
+	return best
+}
+
+// computeOnGPU loads a model with every layer on the GPU, asks for one
+// token, and reads from llama.cpp's own output where the layers went.
+func computeOnGPU(ctx context.Context, bin, model string, out io.Writer) error {
+	if _, err := exec.LookPath("nvidia-smi"); err != nil {
+		return nil
+	}
+	if model == "" {
+		if model = smallestGGUF(weightDirs); model == "" {
+			fmt.Fprintln(out, "no GGUF in this node's weights cache to compute with; --model PATH runs one")
+			return nil
+		}
+	}
+	ln, err := net.Listen("tcp", "127.0.0.1:0")
+	if err != nil {
+		return err
+	}
+	port := ln.Addr().(*net.TCPAddr).Port
+	_ = ln.Close()
+	ctx, cancel := context.WithTimeout(ctx, 3*time.Minute)
+	defer cancel()
+	var logs syncBuffer
+	c := exec.CommandContext(ctx, bin, "-m", model, "--host", "127.0.0.1", "--port", strconv.Itoa(port), //nolint:gosec // the engine's own binary
+		"--n-gpu-layers", "999", "--ctx-size", "512", "--parallel", "1",
+		// Where the layers went is said only at this verbosity.
+		"-lv", "4")
+	c.Stdout, c.Stderr = &logs, &logs
+	if err := c.Start(); err != nil {
+		return fmt.Errorf("starting %s: %w", bin, err)
+	}
+	defer func() { _ = c.Process.Kill(); _ = c.Wait() }()
+	base := "http://127.0.0.1:" + strconv.Itoa(port)
+	start := time.Now()
+	for {
+		resp, err := http.Get(base + "/health") //nolint:gosec,noctx // loopback, bounded by the loop
+		if err == nil {
+			_ = resp.Body.Close()
+			if resp.StatusCode == http.StatusOK {
+				break
+			}
+		}
+		if ctx.Err() != nil {
+			return fmt.Errorf("%s did not come up with %s: %s", bin, filepath.Base(model), tailLines(logs.String(), 5))
+		}
+		time.Sleep(500 * time.Millisecond)
+	}
+	req, _ := http.NewRequestWithContext(ctx, http.MethodPost, base+"/completion",
+		strings.NewReader(`{"prompt":"Hello","n_predict":1}`))
+	req.Header.Set("Content-Type", "application/json")
+	resp, err := http.DefaultClient.Do(req)
+	if err != nil {
+		return fmt.Errorf("asking it for a token: %w", err)
+	}
+	_ = resp.Body.Close()
+	took := time.Since(start)
+	comp, said := engines.ReadLlamaCompute(strings.Split(logs.String(), "\n"))
+	if !said {
+		fmt.Fprintf(out, "computed a token with %s, but llama.cpp did not say where\n", filepath.Base(model))
+		return nil
+	}
+	if comp.OnTheCPU() {
+		return fmt.Errorf("%s computed on the CPU though every layer was asked for on the GPU (%d of %d offloaded): "+
+			"its CUDA backend did not load; ldd on its libggml-cuda.so names the missing library",
+			filepath.Base(model), comp.Offloaded, comp.Total)
+	}
+	fmt.Fprintf(out, "computed a token on %s: %s, %d of %d layers offloaded, in %s\n", comp.Backend,
+		strings.Join(comp.Devices, "; "), comp.Offloaded, comp.Total, took.Round(100*time.Millisecond))
+	return nil
+}
+
+func tailLines(s string, n int) string {
+	l := strings.Split(strings.TrimSpace(s), "\n")
+	if len(l) > n {
+		l = l[len(l)-n:]
+	}
+	return strings.Join(l, " | ")
+}
+
+// syncBuffer is a buffer two writers share.
+type syncBuffer struct {
+	mu sync.Mutex
+	b  strings.Builder
+}
+
+func (s *syncBuffer) Write(p []byte) (int, error) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	return s.b.Write(p)
+}
+
+func (s *syncBuffer) String() string {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	return s.b.String()
+}

internal/cli/enginecheck_gpu_test.go

@@ -0,0 +1,86 @@
+package cli
+
+import (
+	"bytes"
+	"fmt"
+	"net/http"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+)
+
+// TestMain lets the test binary stand in for llama-server: run with
+// LLMGW_FAKE_LLAMA set, it prints what llama.cpp prints and answers.
+func TestMain(m *testing.M) {
+	if mode := os.Getenv("LLMGW_FAKE_LLAMA"); mode != "" {
+		fakeLlama(mode)
+		return
+	}
+	os.Exit(m.Run())
+}
+
+func fakeLlama(mode string) {
+	port := ""
+	for i, a := range os.Args {
+		if a == "--port" && i+1 < len(os.Args) {
+			port = os.Args[i+1]
+		}
+	}
+	if mode == "gpu" {
+		fmt.Println("llama_model_load_from_file_impl: using device CUDA0 (Quadro RTX 6000) - 23800 MiB free")
+		fmt.Println("load_tensors: offloaded 25/25 layers to GPU")
+	} else {
+		fmt.Println("warning: no usable GPU found, --gpu-layers option will be ignored")
+		fmt.Println("load_tensors: offloaded 0/25 layers to GPU")
+	}
+	http.HandleFunc("/health", func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(http.StatusOK) })
+	http.HandleFunc("/completion", func(w http.ResponseWriter, _ *http.Request) { _, _ = w.Write([]byte(`{"content":"!"}`)) })
+	_ = http.ListenAndServe("127.0.0.1:"+port, nil) //nolint:gosec // a test's stand-in
+}
+
+func TestEngineCheckComputesATokenOnTheGPU(t *testing.T) {
+	// nvidia-smi present, so the check runs.
+	dir := t.TempDir()
+	smi := filepath.Join(dir, "nvidia-smi")
+	if err := os.WriteFile(smi, []byte("#!/bin/sh\nexit 0\n"), 0o755); err != nil {
+		t.Fatal(err)
+	}
+	t.Setenv("PATH", dir+string(os.PathListSeparator)+os.Getenv("PATH"))
+	model := filepath.Join(dir, "tiny.gguf")
+	_ = os.WriteFile(model, []byte("GGUF"), 0o644)
+	self, _ := os.Executable()
+
+	t.Setenv("LLMGW_FAKE_LLAMA", "gpu")
+	var out bytes.Buffer
+	if err := computeOnGPU(t.Context(), self, model, &out); err != nil || !strings.Contains(out.String(), "on CUDA") ||
+		!strings.Contains(out.String(), "25 of 25") {
+		t.Fatalf("on the GPU: %v %s", err, out.String())
+	}
+
+	t.Setenv("LLMGW_FAKE_LLAMA", "cpu")
+	out.Reset()
+	if err := computeOnGPU(t.Context(), self, model, &out); err == nil || !strings.Contains(err.Error(), "computed on the CPU") {
+		t.Fatalf("a build that falls back to the CPU: %v %s", err, out.String())
+	}
+}
+
+func TestEngineCheckFindsTheBuildTheAgentInstalled(t *testing.T) {
+	old := llamaRoot
+	llamaRoot = t.TempDir()
+	t.Cleanup(func() { llamaRoot = old })
+	if got := installedLlama(); got != "" {
+		t.Fatalf("nothing installed, found %q", got)
+	}
+	for _, p := range []string{"b10991-cpu/llama-b10991/llama-server", "b10991-cuda-13.3/llama-b10991/llama-server"} {
+		full := filepath.Join(llamaRoot, p)
+		_ = os.MkdirAll(filepath.Dir(full), 0o755)
+		_ = os.WriteFile(full, []byte("x"), 0o755)
+	}
+	dir := t.TempDir()
+	_ = os.WriteFile(filepath.Join(dir, "nvidia-smi"), []byte("#!/bin/sh\n"), 0o755)
+	t.Setenv("PATH", dir)
+	if got := installedLlama(); !strings.Contains(got, "cuda-13.3/llama-b10991/llama-server") {
+		t.Errorf("with GPUs, not the CUDA build: %q", got)
+	}
+}

internal/cli/firewallcmd.go

@@ -0,0 +1,176 @@
+package cli
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"net/netip"
+	"os"
+	"time"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
+)
+
+// firewallStateFile is where a control plane writes what the cluster's
+// settings say of the firewall, for llmgw-firewall-sync to apply as root:
+// the server itself runs as llmgw and cannot change the ruleset.
+const firewallStateFile = "/var/lib/llmgw/firewall.json"
+
+// firewallState is what is written there.
+type firewallState struct {
+	Enabled  bool     `json:"enabled"`
+	Networks []string `json:"networks,omitempty"`
+}
+
+// writeFirewallState records the settings' firewall for this machine's sync
+// unit. Without the state directory — a server not run by systemd — there
+// is no unit to read it, and nothing is written.
+func writeFirewallState(cfg config.Server) error {
+	if _, err := os.Stat("/var/lib/llmgw"); err != nil {
+		return nil //nolint:nilerr // not a control plane set up by init
+	}
+	b, _ := json.Marshal(firewallState{Enabled: cfg.Firewall.Enabled, Networks: cfg.FirewallNetworks})
+	if old, err := os.ReadFile(firewallStateFile); err == nil && string(old) == string(b) {
+		return nil
+	}
+	tmp := firewallStateFile + ".new"
+	if err := os.WriteFile(tmp, b, 0o644); err != nil {
+		return err
+	}
+	return os.Rename(tmp, firewallStateFile)
+}
+
+// The units that apply it: a path unit watching the file, and the oneshot it
+// starts.
+const (
+	firewallSyncPathUnit = "llmgw-firewall-sync.path"
+	firewallSyncUnitFile = `[Unit]
+Description=llmgw firewall: apply the cluster's settings on this control plane
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/bin/llmgw firewall sync
+`
+	firewallSyncPathFile = `[Unit]
+Description=llmgw firewall: follow the cluster's settings on this control plane
+
+[Path]
+PathChanged=` + firewallStateFile + `
+
+[Install]
+WantedBy=multi-user.target
+`
+)
+
+func newFirewallCmd() *cobra.Command {
+	c := &cobra.Command{
+		Use:   "firewall",
+		Short: "llmgw's firewall on this machine",
+		Long: "Whether llmgw keeps a firewall is the cluster's setting firewall.enabled\n" +
+			"(llmgw ctl config set firewall.enabled false). Nodes are told by the\n" +
+			"control plane; a control plane writes it to " + firewallStateFile + ",\n" +
+			"which " + firewallSyncPathUnit + " applies with 'llmgw firewall sync'.",
+	}
+	var install bool
+	sync := &cobra.Command{
+		Use:   "sync",
+		Short: "Apply what the cluster's settings say of the firewall, here (root)",
+		Args:  cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			if os.Geteuid() != 0 {
+				return errors.New("the firewall is changed as root: run it with sudo")
+			}
+			if install {
+				if err := installFirewallSync(cmd.Context(), firewall.Local{}); err != nil {
+					return err
+				}
+				fmt.Fprintln(cmd.OutOrStdout(), firewallSyncPathUnit+" follows the cluster's settings here")
+				if _, err := os.Stat(firewallStateFile); err != nil {
+					return nil // the server writes it when it starts
+				}
+			}
+			b, err := os.ReadFile(firewallStateFile)
+			if err != nil {
+				return err
+			}
+			var st firewallState
+			if err := json.Unmarshal(b, &st); err != nil {
+				return fmt.Errorf("%s: %w", firewallStateFile, err)
+			}
+			ctx, cancel := context.WithTimeout(cmd.Context(), 2*time.Minute)
+			defer cancel()
+			msg, err := syncFirewall(ctx, firewall.Local{}, st)
+			if err != nil {
+				return err
+			}
+			fmt.Fprintln(cmd.OutOrStdout(), msg)
+			return nil
+		},
+	}
+	sync.Flags().BoolVar(&install, "install", false, "first install the units that run this whenever the settings change")
+	c.AddCommand(sync)
+	return c
+}
+
+// installFirewallSync writes the sync units on a machine and has the path
+// unit follow the state file.
+func installFirewallSync(ctx context.Context, r firewall.Runner) error {
+	body := firewallSyncUnitFile + "\x00" + firewallSyncPathFile
+	out, err := r.Run(ctx, `set -e
+t=$(mktemp -d); trap 'rm -rf "$t"' EXIT
+cat > "$t/all"
+python3 - "$t" <<'PY'
+import sys
+d = sys.argv[1]
+a, b = open(d + "/all", "rb").read().split(b"\x00")
+open(d + "/service", "wb").write(a)
+open(d + "/path", "wb").write(b)
+PY
+sudo install -m 644 "$t/service" /etc/systemd/system/llmgw-firewall-sync.service
+sudo install -m 644 "$t/path" /etc/systemd/system/`+firewallSyncPathUnit+`
+sudo systemctl daemon-reload
+sudo systemctl enable --now `+firewallSyncPathUnit+` >/dev/null 2>&1`, []byte(body))
+	if err != nil {
+		return fmt.Errorf("installing %s: %v: %s", firewallSyncPathUnit, err, string(out))
+	}
+	return nil
+}
+
+// syncFirewall puts a firewall state in force on this machine.
+func syncFirewall(ctx context.Context, r firewall.Runner, st firewallState) (string, error) {
+	if !st.Enabled {
+		had, err := firewall.Disable(ctx, r)
+		if err != nil {
+			return "", err
+		}
+		if had {
+			return "llmgw's firewall is off here", nil
+		}
+		return "llmgw's firewall was already off here", nil
+	}
+	loaded, err := firewall.Enable(ctx, r)
+	if err != nil {
+		return "", err
+	}
+	if !loaded {
+		return "no llmgw rules on this machine to put back; 'llmgw server init' writes them", nil
+	}
+	var extra []netip.Prefix
+	for _, n := range st.Networks {
+		if p, err := netip.ParsePrefix(n); err == nil {
+			extra = append(extra, p)
+		} else if a, err := netip.ParseAddr(n); err == nil {
+			extra = append(extra, netip.PrefixFrom(a, a.BitLen()))
+		}
+	}
+	if len(extra) > 0 {
+		if _, err := firewall.Install(ctx, r, firewall.Spec{Allow: extra}); err != nil {
+			return "", err
+		}
+	}
+	return "llmgw's firewall is on here", nil
+}

internal/cli/firewallcmd_test.go

@@ -0,0 +1,51 @@
+package cli
+
+import (
+	"context"
+	"strings"
+	"testing"
+)
+
+type scriptLog struct {
+	scripts []string
+	reply   func(script string) string
+}
+
+func (s *scriptLog) Run(_ context.Context, script string, _ []byte) ([]byte, error) {
+	s.scripts = append(s.scripts, script)
+	if s.reply != nil {
+		return []byte(s.reply(script)), nil
+	}
+	return nil, nil
+}
+
+func TestAControlPlaneFollowsTheFirewallSetting(t *testing.T) {
+	off := &scriptLog{}
+	if _, err := syncFirewall(context.Background(), off, firewallState{Enabled: false}); err != nil {
+		t.Fatal(err)
+	}
+	if !strings.Contains(strings.Join(off.scripts, "\n"), "nft delete table inet llmgw") {
+		t.Errorf("off did not take the table out: %q", off.scripts)
+	}
+
+	none := &scriptLog{reply: func(s string) string {
+		if strings.Contains(s, "echo loaded") {
+			return "none\n"
+		}
+		return ""
+	}}
+	msg, err := syncFirewall(context.Background(), none, firewallState{Enabled: true})
+	if err != nil || !strings.Contains(msg, "server init") {
+		t.Errorf("on, with no rules here: %q %v", msg, err)
+	}
+
+	on := &scriptLog{reply: func(s string) string {
+		if strings.Contains(s, "echo loaded") {
+			return "loaded\n"
+		}
+		return ""
+	}}
+	if msg, err := syncFirewall(context.Background(), on, firewallState{Enabled: true}); err != nil || msg != "llmgw's firewall is on here" {
+		t.Errorf("on: %q %v", msg, err)
+	}
+}

internal/cli/guard.go

@@ -3,18 +3,21 @@ package cli
 import (
 	"encoding/json"
 	"fmt"
+	"strconv"
 	"strings"
 
 	"github.com/spf13/cobra"
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
 )
 
 func newGuardCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 	c := &cobra.Command{
 		Use:   "guard",
 		Short: "Guardrail policies: which apply where",
-		Long: "Policies are defined in server.yaml under guard.policies — what requests and\n" +
+		Long: "Policies are defined in the cluster's settings under guard.policies\n" +
+			"(llmgw ctl config edit; in force everywhere within seconds) — what requests and\n" +
 			"answers are checked for (personal data, secrets, prompt injection, patterns,\n" +
 			"tools) and what is done: flag, redact or block. guard.default applies to\n" +
 			"every request; attach others to an organization, team, project, user or\n" +
@@ -23,6 +26,35 @@ func newGuardCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 			"Targets are written org:Acme, team:Acme/ml, project:Acme/ml/training (or\n" +
 			"with IDs), key:KEY-ID or user:USER-ID. An attachment applies at once.",
 	}
+	c.AddCommand(newGuardJudgeCmd(client, asJSON))
+	for _, sw := range []struct {
+		use, short string
+		on         bool
+	}{
+		{"on", "Turn the guardrails on: every policy checks again", true},
+		{"off", "Turn the guardrails off: no request is checked, whatever the policies say", false},
+	} {
+		var note string
+		cmd := &cobra.Command{
+			Use:   sw.use,
+			Short: sw.short,
+			Long: "guard.enabled in the cluster's settings: in force on every control plane\n" +
+				"within seconds, without a restart. The policies and where they are attached\n" +
+				"stay as they are, and come back as they were when turned on again. While\n" +
+				"off, llmgw ctl status says so.",
+			Args: cobra.NoArgs,
+			RunE: func(cmd *cobra.Command, _ []string) error {
+				if note == "" {
+					note = "guardrails " + sw.use
+				}
+				return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
+					return doc.SetYAML([]string{"guard", "enabled"}, strconv.FormatBool(sw.on))
+				})
+			},
+		}
+		cmd.Flags().StringVar(&note, "note", "", "why, kept with the settings' version")
+		c.AddCommand(cmd)
+	}
 	c.AddCommand(&cobra.Command{
 		Use:   "list",
 		Short: "The policies, and where each is attached",
@@ -36,6 +68,9 @@ func newGuardCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 				return json.NewEncoder(cmd.OutOrStdout()).Encode(v)
 			}
 			out := cmd.OutOrStdout()
+			if !v.Enabled {
+				fmt.Fprintln(out, "The guardrails are OFF: no request is checked (llmgw ctl guard on).")
+			}
 			fmt.Fprintf(out, "Policies: %s\n", strings.Join(v.Policies, ", "))
 			def := strings.Join(v.Default, ", ")
 			if def == "" {

internal/cli/guard_judge.go

@@ -0,0 +1,385 @@
+package cli
+
+import (
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"slices"
+	"strings"
+	"time"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
+)
+
+func newGuardJudgeCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
+	c := &cobra.Command{
+		Use:   "judge",
+		Short: "The judge models the policies ask: whether each can answer, and serving one",
+		Long: "A policy with a judge has a model read what patterns cannot. The judge must\n" +
+			"be in the catalog under the name the policy gives, as a judge (role:\n" +
+			"guard), with a replica ready; otherwise what the policy judges is refused\n" +
+			"(fail: closed, guard_unavailable) or served unjudged (fail: open).\n\n" +
+			"A policy may name several, asked in turn (judge.models): a judge on a GPU\n" +
+			"first and one on CPUs after it is the GPU's speed while it is there, and\n" +
+			"the CPUs' when it is not.\n\n" +
+			"Judges llmgw knows where to fetch, which a policy may name and the leader\n" +
+			"then fetches and serves by itself: " + knownJudgeNames() + ".",
+	}
+	c.AddCommand(newGuardJudgeStatusCmd(client, asJSON), newGuardJudgeEnableCmd(client),
+		newGuardJudgeUseCmd(client), newGuardJudgeAskCmd(client, asJSON))
+	return c
+}
+
+func newGuardJudgeStatusCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
+	return &cobra.Command{
+		Use:   "status",
+		Short: "Each judge the policies ask, and what is wrong with it",
+		Args:  cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			var js []adminapi.JudgeView
+			if err := client().get(cmd.Context(), "/api/v1/guard/judges", &js); err != nil {
+				return err
+			}
+			if *asJSON {
+				return emitJSON(js)
+			}
+			out := cmd.OutOrStdout()
+			if len(js) == 0 {
+				fmt.Fprintln(out, "no policy asks a judge")
+				return nil
+			}
+			w := tabTo(out)
+			fmt.Fprintln(w, "JUDGE\tPOLICIES\tIN CATALOG\tAS JUDGE\tREADY\tSTATE")
+			for _, j := range js {
+				state := "answering"
+				if j.Problem != "" {
+					state = j.Problem
+				}
+				fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%d/%d\t%s\n", j.Model, strings.Join(j.Policies, ","),
+					yesNo(j.InCatalog), yesNo(j.Judge), j.Ready, j.Wanted, state)
+			}
+			for _, j := range js {
+				if len(j.Then) > 0 {
+					fmt.Fprintf(w, "\n%s gives no verdict: %s is asked instead", j.Model, strings.Join(j.Then, ", else "))
+				}
+			}
+			if err := w.Flush(); err != nil {
+				return err
+			}
+			strays, _ := strayJudges(cmd, client, js)
+			sayStrays(out, strays)
+			return nil
+		},
+	}
+}
+
+// strayJudges are models whose weights are a judge's, under a name no policy
+// asks: a judge the policies moved on from, or one served from the mirror
+// page as a model clients use, as before that page knew better.
+func strayJudges(cmd *cobra.Command, client func() *ctlClient, js []adminapi.JudgeView) ([]adminapi.ModelView, error) {
+	var models []adminapi.ModelView
+	if err := client().get(cmd.Context(), "/api/v1/models", &models); err != nil {
+		return nil, err
+	}
+	asked := map[string]bool{}
+	for _, j := range js {
+		asked[j.Model] = true
+	}
+	var out []adminapi.ModelView
+	for _, m := range models {
+		if _, judge := guard.FormatOfWeights(m.Source); !asked[m.Name] && judge {
+			out = append(out, m)
+		}
+	}
+	return out, nil
+}
+
+// sayStrays says what each stray judge is and how to be rid of it.
+func sayStrays(out io.Writer, strays []adminapi.ModelView) {
+	for _, m := range strays {
+		if m.Role == catalog.RoleGuard {
+			fmt.Fprintf(out, "\n%s is a judge no policy asks, still running: llmgw ctl model remove %s\n", m.Name, m.Name)
+			continue
+		}
+		fmt.Fprintf(out, "\n%s looks like a judge's weights but is served as a model clients use, and no "+
+			"policy asks it: llmgw ctl model remove %s\n", m.Name, m.Name)
+	}
+}
+
+func newGuardJudgeEnableCmd(client func() *ctlClient) *cobra.Command {
+	var (
+		model, repo, file, format, gpuClass string
+		replicas, cores                     int
+		gpu, replace, detach                bool
+	)
+	c := &cobra.Command{
+		Use:   "enable",
+		Short: "Serve a judge: Qwen3Guard-Gen 0.6B on CPUs, or with --gpu the 4B on a GPU",
+		Long: "Fetches a judge's weights into the mirror (once; what is there already, its\n" +
+			"checksums matching, is not fetched again), adds it to the catalog as a judge\n" +
+			"(role: guard) and waits for a replica to answer. Which judge: --model, or\n" +
+			"with --gpu " + defaultGPUJudge + ", or else the first the policies ask that is\n" +
+			"missing, or else " + defaultJudge + ".\n\n" +
+			"A judge on CPUs takes no card. One on a GPU takes one card, as a standard\n" +
+			"model: from nothing more important, so it waits while the cards are busy;\n" +
+			"put a judge on CPUs after it in the policies (llmgw ctl guard judge use) so\n" +
+			"there is always one to ask.\n\n" +
+			"Known judges: " + knownJudgeNames() + ". Another is served with --repo,\n" +
+			"--file and --format. A model of that name already in the catalog as a\n" +
+			"model clients use is refused; --replace removes it first. An offline\n" +
+			"cluster gets its judge from its bundle with server init.",
+		Example: "  llmgw ctl guard judge enable\n" +
+			"  llmgw ctl guard judge enable --gpu\n" +
+			"  llmgw ctl guard judge use " + defaultGPUJudge + " " + defaultJudge,
+		Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			ctx := cmd.Context()
+			out := cmd.OutOrStdout()
+			var js []adminapi.JudgeView
+			if err := client().get(ctx, "/api/v1/guard/judges", &js); err != nil {
+				return err
+			}
+			name := model
+			if name == "" && gpu {
+				name = defaultGPUJudge
+			}
+			if name == "" {
+				for _, j := range js {
+					if !j.InCatalog || !j.Judge {
+						name = j.Model
+						break
+					}
+				}
+			}
+			if name == "" {
+				name = defaultJudge
+			}
+
+			k, known := knownJudgeNamed(name)
+			switch {
+			case repo != "":
+				f := format
+				if f == "" {
+					f, _ = guard.FormatOfWeights(repo + "/" + file)
+				}
+				if f == "" || !guard.KnownFormat(f) {
+					return fmt.Errorf("say how its answers are read: --format %s", strings.Join(guard.Formats, "|"))
+				}
+				k = knownJudge{Name: name, Repo: repo, File: file, Format: f, GPU: gpu}
+			case !known:
+				return fmt.Errorf("%s is not a judge llmgw knows where to fetch (%s): give --repo, --file and --format", name, knownJudgeNames())
+			}
+			if gpu {
+				k.GPU = true
+			}
+
+			var models []adminapi.ModelView
+			if err := client().get(ctx, "/api/v1/models", &models); err != nil {
+				return err
+			}
+			if i := slices.IndexFunc(models, func(m adminapi.ModelView) bool { return m.Name == name }); i >= 0 {
+				if models[i].Role == catalog.RoleGuard {
+					fmt.Fprintf(out, "%s is in the catalog as a judge already; llmgw ctl guard judge status says how it is\n", name)
+					return nil
+				}
+				if !replace {
+					return fmt.Errorf("%s is in the catalog as a model clients use, not as a judge: "+
+						"--replace removes it and serves it again as a judge", name)
+				}
+				var res map[string]string
+				if err := client().delete(ctx, "/api/v1/models/"+name, &res); err != nil {
+					return fmt.Errorf("removing %s: %w", name, err)
+				}
+				fmt.Fprintf(out, "%s removed\n", name)
+			}
+
+			opts := judgeServeOptions{Replicas: replicas, Cores: cores, GPUClass: gpuClass}
+			if k.GPU && opts.GPUClass == "" {
+				var s adminapi.Summary
+				if err := client().get(ctx, "/api/v1/summary", &s); err != nil {
+					return err
+				}
+				if opts.GPUClass = roomiestGPUClass(s.GPUs); opts.GPUClass == "" {
+					return fmt.Errorf("this cluster has no GPU for %s: serve %s on CPUs instead", name, defaultJudge)
+				}
+			}
+			fmt.Fprintf(out, "serving %s\n", k)
+			body, _ := json.Marshal(k.serveRequest(opts))
+			var job modelserve.Job
+			if err := client().patient(30*time.Second).post(ctx, "/api/v1/models/serve", body, &job); err != nil {
+				return err
+			}
+			if detach {
+				return detached(cmd, job)
+			}
+			if err := followJob(cmd, client, job, "is the judge"); err != nil {
+				return err
+			}
+			if !slices.ContainsFunc(js, func(j adminapi.JudgeView) bool { return j.Model == name }) {
+				fmt.Fprintf(out, "\nno policy asks %s yet: llmgw ctl guard judge use %s", name, name)
+				if k.GPU {
+					fmt.Fprintf(out, " %s", defaultJudge)
+				}
+				fmt.Fprintln(out)
+			}
+			if strays, _ := strayJudges(cmd, client, js); len(strays) > 0 {
+				sayStrays(out, slices.DeleteFunc(strays, func(m adminapi.ModelView) bool { return m.Name == name }))
+			}
+			return nil
+		},
+	}
+	f := c.Flags()
+	f.StringVar(&model, "model", "", "the judge to serve: "+knownJudgeNames()+", or a name for --repo's")
+	f.BoolVar(&gpu, "gpu", false, "serve it on a GPU ("+defaultGPUJudge+" when no --model)")
+	f.StringVar(&gpuClass, "gpu-class", "", "the GPUs it runs on (default: the class with the most free cards)")
+	f.StringVar(&repo, "repo", "", "the registry repository of a judge llmgw does not know")
+	f.StringVar(&file, "file", "", "the GGUF file in --repo")
+	f.StringVar(&format, "format", "", "how --repo's judge answers: "+strings.Join(guard.Formats, ", "))
+	f.IntVar(&replicas, "replicas", 0, "replicas (default: 2 on CPUs, as every judged request waits on one; 1 on a GPU)")
+	f.IntVar(&cores, "cores", 4, "CPU cores a replica on CPUs computes on")
+	f.BoolVar(&replace, "replace", false, "remove a model of that name that is not a judge first")
+	f.BoolVar(&detach, "detach", false, "start it and return; llmgw ctl jobs follows it")
+	return c
+}
+
+// roomiestGPUClass is the class with the most free cards, else the most
+// cards; empty when the cluster has none.
+func roomiestGPUClass(classes []adminapi.ClassUse) string {
+	best := -1
+	for i, c := range classes {
+		if c.Total == 0 {
+			continue
+		}
+		if best < 0 || c.Free() > classes[best].Free() || (c.Free() == classes[best].Free() && c.Total > classes[best].Total) {
+			best = i
+		}
+	}
+	if best < 0 {
+		return ""
+	}
+	return classes[best].Class
+}
+
+func newGuardJudgeUseCmd(client func() *ctlClient) *cobra.Command {
+	var policies []string
+	var note string
+	c := &cobra.Command{
+		Use:   "use JUDGE...",
+		Short: "Set the judges the policies ask, in the order they are asked",
+		Long: "Sets judge.models of every policy that has a judge, or of those given with\n" +
+			"--policy: the first judge that answers gives the verdict, and the next is\n" +
+			"asked when one has no replica serving, fails, or does not answer within\n" +
+			"the policy's latency_budget. A judge on a GPU first and one on CPUs after\n" +
+			"it is the GPU's speed while it is there, and the CPUs' when it is not.\n\n" +
+			"A judge llmgw knows (" + knownJudgeNames() + ")\n" +
+			"that is not in the catalog is fetched and served by the leader; another\n" +
+			"is served first with llmgw ctl guard judge enable --repo. In force on\n" +
+			"every control plane within seconds.",
+		Example: "  llmgw ctl guard judge use " + defaultJudge + "\n" +
+			"  llmgw ctl guard judge use " + defaultGPUJudge + " " + defaultJudge + "\n" +
+			"  llmgw ctl guard judge use " + defaultGPUJudge + " " + defaultJudge + " --policy strict",
+		Args: cobra.MinimumNArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			for i, a := range args {
+				if slices.Contains(args[:i], a) {
+					return fmt.Errorf("%s is named twice", a)
+				}
+			}
+			if note == "" {
+				note = "judges: " + strings.Join(args, ", else ")
+			}
+			list := "[" + strings.Join(args, ", ") + "]"
+			return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
+				names := policies
+				if len(names) == 0 {
+					for _, p := range doc.MapKeys([]string{"guard", "policies"}) {
+						if doc.Node([]string{"guard", "policies", p, "judge"}) != nil {
+							names = append(names, p)
+						}
+					}
+					if len(names) == 0 {
+						return errors.New("no policy has a judge: name one with --policy, and it is given one " +
+							"that blocks what the judge finds unsafe in requests (judge.input: block)")
+					}
+				}
+				for _, p := range names {
+					base := []string{"guard", "policies", p}
+					if doc.Node(base) == nil {
+						return fmt.Errorf("there is no policy %s (llmgw ctl guard list)", p)
+					}
+					judge := append(slices.Clone(base), "judge")
+					if doc.Node(judge) == nil {
+						if err := doc.SetYAML(append(slices.Clone(judge), "input"), "block"); err != nil {
+							return err
+						}
+					}
+					doc.Unset(append(slices.Clone(judge), "model"))
+					if err := doc.SetYAML(append(slices.Clone(judge), "models"), list); err != nil {
+						return err
+					}
+					cats := append(slices.Clone(judge), "categories")
+					var have []string
+					if n := doc.Node(cats); n != nil {
+						for _, c := range n.Content {
+							have = append(have, c.Value)
+						}
+					}
+					if next := categoriesFor(args, have); next != nil {
+						if err := doc.SetYAML(cats, "["+strings.Join(next, ", ")+"]"); err != nil {
+							return err
+						}
+						fmt.Fprintf(cmd.OutOrStdout(), "%s: categories [%s] are not what %s name; it counts [%s] instead\n",
+							p, strings.Join(have, ", "), strings.Join(args, " or "), strings.Join(next, ", "))
+					}
+				}
+				return nil
+			})
+		},
+	}
+	c.Flags().StringSliceVar(&policies, "policy", nil, "the policies to change (default: every one with a judge)")
+	c.Flags().StringVar(&note, "note", "", "why, kept with the settings' version")
+	return c
+}
+
+func newGuardJudgeAskCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
+	var model, answer string
+	c := &cobra.Command{
+		Use:   "ask TEXT",
+		Short: "Have a judge read a text and say what it answered",
+		Long: "For finding out why a request was refused: the judge reads TEXT as a user's\n" +
+			"message (with --answer, ANSWER as the reply to it), past the cache, and its\n" +
+			"answer is shown as it gave it and as the gateway reads it — \"unsafe\" and\n" +
+			"the hazards it named, S6 Specialized Advice and the like. Nothing is kept:\n" +
+			"not in the audit, the usage or the logs.",
+		Example: "  llmgw ctl guard judge ask 'How do I sort a list in Python?'\n" +
+			"  llmgw ctl guard judge ask 'Write a poem' --answer 'Roses are red'",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			body, _ := json.Marshal(adminapi.JudgeAskRequest{Model: model, Text: args[0], Answer: answer})
+			var v adminapi.JudgeAskView
+			if err := client().patient(45*time.Second).post(cmd.Context(), "/api/v1/guard/judges/ask", body, &v); err != nil {
+				return err
+			}
+			if *asJSON {
+				return emitJSON(v)
+			}
+			out := cmd.OutOrStdout()
+			fmt.Fprintf(out, "%s: %s", v.Model, v.Verdict)
+			if len(v.Categories) > 0 {
+				fmt.Fprintf(out, " (%s)", strings.Join(v.Categories, ", "))
+			}
+			fmt.Fprintf(out, ", %d ms\nit answered: %q\n", v.TookMs, v.Raw)
+			return nil
+		},
+	}
+	c.Flags().StringVar(&model, "model", "", "the judge (default: the one the policies ask)")
+	c.Flags().StringVar(&answer, "answer", "", "an assistant's reply to TEXT, judged as output is")
+	return c
+}

internal/cli/judgekeeper.go

@@ -0,0 +1,116 @@
+package cli
+
+import (
+	"context"
+	"log/slog"
+	"slices"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
+)
+
+// judgeKeeper fetches and serves, on the leader, each judge llmgw knows that
+// a policy names and the catalog does not have: naming qwen3guard-4b in a
+// policy is all it takes to have one, and a judge removed by mistake comes
+// back. A judge it does not know is left to llmgw ctl guard judge enable
+// --repo, said in guard judge status.
+type judgeKeeper struct {
+	library func() guard.Config
+	models  func() []catalog.Model
+	serve   interface {
+		Start(modelserve.Request) (modelserve.Job, error)
+		Jobs() []modelserve.Job
+	}
+	leader func() bool
+	gpus   func() []adminapi.ClassUse
+	log    *slog.Logger
+	now    func() time.Time
+
+	// tried is when each judge was last asked for, so one that cannot be
+	// fetched is not asked for again on every look.
+	tried map[string]time.Time
+}
+
+// judgeRetry is how long a judge whose serving failed waits before it is
+// asked for again.
+const judgeRetry = 10 * time.Minute
+
+func (k *judgeKeeper) run(ctx context.Context, every time.Duration) {
+	t := time.NewTicker(every)
+	defer t.Stop()
+	for {
+		k.check()
+		select {
+		case <-ctx.Done():
+			return
+		case <-t.C:
+		}
+	}
+}
+
+func (k *judgeKeeper) check() {
+	if k.leader != nil && !k.leader() {
+		return
+	}
+	lib := k.library()
+	if !lib.On() {
+		return
+	}
+	if k.tried == nil {
+		k.tried = map[string]time.Time{}
+	}
+	now := time.Now
+	if k.now != nil {
+		now = k.now
+	}
+	have := map[string]bool{}
+	for _, m := range k.models() {
+		have[m.Name] = true
+	}
+	busy := map[string]bool{}
+	for _, j := range k.serve.Jobs() {
+		if j.State == modelserve.Running {
+			busy[j.Model] = true
+			busy[j.Request.Name] = true
+		}
+	}
+	var wanted []string
+	for _, p := range lib.Policies {
+		for _, j := range p.Judge.Judges() {
+			if !slices.Contains(wanted, j) {
+				wanted = append(wanted, j)
+			}
+		}
+	}
+	slices.Sort(wanted)
+	for _, name := range wanted {
+		kj, known := knownJudgeNamed(name)
+		if have[name] || !known || busy[name] {
+			continue
+		}
+		if at, ok := k.tried[name]; ok && now().Sub(at) < judgeRetry {
+			continue
+		}
+		k.tried[name] = now()
+		var opts judgeServeOptions
+		if kj.GPU {
+			if opts.GPUClass = roomiestGPUClass(k.gpus()); opts.GPUClass == "" {
+				k.log.Warn("a policy names a judge that runs on a GPU, and this cluster has none",
+					"judge", name, "instead", defaultJudge)
+				continue
+			}
+		}
+		req := kj.serveRequest(opts)
+		req.By = "llmgw (a guard policy names it)"
+		job, err := k.serve.Start(req)
+		if err != nil {
+			k.log.Error("could not start serving a judge a policy names", "judge", name, "err", err)
+			continue
+		}
+		k.log.Info("serving a judge a policy names, which the catalog did not have",
+			"judge", name, "repo", kj.Repo, "file", kj.File, "job", job.ID)
+	}
+}

internal/cli/judgekeeper_test.go

@@ -0,0 +1,126 @@
+package cli
+
+import (
+	"errors"
+	"log/slog"
+	"slices"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
+)
+
+type fakeServe struct {
+	started []modelserve.Request
+	running []modelserve.Job
+	err     error
+}
+
+func (f *fakeServe) Start(r modelserve.Request) (modelserve.Job, error) {
+	if f.err != nil {
+		return modelserve.Job{}, f.err
+	}
+	f.started = append(f.started, r)
+	return modelserve.Job{ID: "j", Model: r.Name, Request: r, State: modelserve.Running}, nil
+}
+func (f *fakeServe) Jobs() []modelserve.Job { return f.running }
+
+func TestAJudgeAPolicyNamesIsFetchedWhenTheCatalogLacksIt(t *testing.T) {
+	lib := guard.Config{Policies: map[string]guard.Policy{
+		"baseline": {Judge: &guard.JudgePolicy{Models: []string{"qwen3guard-4b", "qwen3guard-0.6b"}, Input: "block"}},
+		"custom":   {Judge: &guard.JudgePolicy{Model: "our-own-judge", Input: "block"}},
+	}}
+	inCatalog := []catalog.Model{{Name: "qwen3guard-0.6b"}}
+	now := time.Date(2026, 10, 8, 12, 0, 0, 0, time.UTC)
+	fs := &fakeServe{}
+	leader := true
+	k := &judgeKeeper{
+		library: func() guard.Config { return lib },
+		models:  func() []catalog.Model { return inCatalog },
+		serve:   fs,
+		leader:  func() bool { return leader },
+		gpus:    func() []adminapi.ClassUse { return []adminapi.ClassUse{{Class: "6000-24g", Total: 1, Used: 1}} },
+		log:     slog.New(slog.DiscardHandler),
+		now:     func() time.Time { return now },
+	}
+
+	k.check()
+	// The GPU judge alone: the CPU one is there, and one llmgw does not know
+	// is not guessed at.
+	if len(fs.started) != 1 || fs.started[0].Name != "qwen3guard-4b" || fs.started[0].GPUClass != "6000-24g" ||
+		fs.started[0].Class != catalog.ClassStandard || fs.started[0].GuardFormat != "qwen3guard" || fs.started[0].CPU {
+		t.Fatalf("started %+v", fs.started)
+	}
+
+	// Being fetched: not asked for again.
+	fs.running = []modelserve.Job{{Model: "qwen3guard-4b", State: modelserve.Running}}
+	k.check()
+	if len(fs.started) != 1 {
+		t.Errorf("asked for again while it is being fetched: %d", len(fs.started))
+	}
+	// Failed: asked for again only after a while.
+	fs.running = nil
+	now = now.Add(time.Minute)
+	k.check()
+	if len(fs.started) != 1 {
+		t.Error("asked for again at once after it failed")
+	}
+	now = now.Add(judgeRetry)
+	k.check()
+	if len(fs.started) != 2 {
+		t.Error("never asked for again")
+	}
+
+	// Only the leader fetches, and nothing while the guardrails are off.
+	leader = false
+	now = now.Add(judgeRetry)
+	k.check()
+	leader = true
+	off := false
+	lib.Enabled = &off
+	k.check()
+	if len(fs.started) != 2 {
+		t.Errorf("a standby or the guardrails off fetched: %d", len(fs.started))
+	}
+
+	// A CPU judge on a cluster without GPUs is served on CPUs, two of it.
+	lib = guard.Config{Policies: map[string]guard.Policy{"b": {Judge: &guard.JudgePolicy{Model: "qwen3guard-0.6b", Input: "block"}}}}
+	inCatalog = nil
+	k.gpus = func() []adminapi.ClassUse { return nil }
+	k.check()
+	if last := fs.started[len(fs.started)-1]; last.Name != "qwen3guard-0.6b" || !last.CPU || last.Replicas != 2 {
+		t.Errorf("CPU judge: %+v", last)
+	}
+	fs.err = errors.New("the mirror is down")
+	lib = guard.Config{Policies: map[string]guard.Policy{"b": {Judge: &guard.JudgePolicy{Model: "qwen3guard-8b", Input: "block"}}}}
+	k.check() // no GPU, so not started, and no panic on the error path either
+}
+
+// A policy moved from Llama Guard's codes to Qwen3Guard would count nothing:
+// its categories become the judge's own.
+func TestCategoriesFollowTheJudges(t *testing.T) {
+	lg := []string{"S1", "S2", "S10"}
+	if got := categoriesFor([]string{"qwen3guard-0.6b"}, lg); !slices.Equal(got, baselineCategories["qwen3guard"]) {
+		t.Errorf("Llama Guard's codes for Qwen3Guard: %v", got)
+	}
+	if got := categoriesFor([]string{"qwen3guard-4b", "qwen3guard-0.6b"}, []string{"Violent", "Jailbreak"}); got != nil {
+		t.Errorf("its own names were replaced: %v", got)
+	}
+	if got := categoriesFor([]string{"llama-guard-3-1b"}, lg); got != nil {
+		t.Errorf("Llama Guard's own codes were replaced: %v", got)
+	}
+	// Both kinds asked in turn: each must have something to count.
+	got := categoriesFor([]string{"qwen3guard-4b", "llama-guard-3-1b"}, []string{"Violent"})
+	if !slices.Contains(got, "Violent") || !slices.Contains(got, "S1") {
+		t.Errorf("mixed: %v", got)
+	}
+	if got := categoriesFor([]string{"our-own-judge"}, lg); got != nil {
+		t.Errorf("a judge llmgw does not know: %v", got)
+	}
+	if got := categoriesFor([]string{"qwen3guard-0.6b"}, nil); got != nil {
+		t.Errorf("no categories (every one counts) became %v", got)
+	}
+}

internal/cli/judges_known.go

@@ -0,0 +1,158 @@
+package cli
+
+import (
+	"fmt"
+	"slices"
+	"strings"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
+)
+
+// knownJudge is a judge model llmgw knows where to fetch and how to serve:
+// what a policy may name without anyone saying where its weights are.
+type knownJudge struct {
+	Name, Repo, File, Format string
+	// GPU is a judge too slow on CPUs to wait for: served on one card.
+	GPU   bool
+	About string
+}
+
+// knownJudges, the default first. Measured on the reference cluster against
+// ordinary coding requests ("continue", "kill the process on port 8080") and
+// harmful ones: Llama Guard 3 1B scored both alike, about one in two
+// ordinary ones refused; Qwen3Guard-Gen 0.6B kept them apart, 0.15 against
+// 1.00 at the worst, at about two seconds a check on two CPU threads.
+var knownJudges = []knownJudge{
+	{Name: "qwen3guard-0.6b", Repo: "mradermacher/Qwen3Guard-Gen-0.6B-GGUF", File: "Qwen3Guard-Gen-0.6B.Q8_0.gguf",
+		Format: "qwen3guard", About: "Qwen3Guard-Gen 0.6B on CPUs, 0.8 GB: the default"},
+	{Name: "qwen3guard-4b", Repo: "mradermacher/Qwen3Guard-Gen-4B-GGUF", File: "Qwen3Guard-Gen-4B.Q4_K_M.gguf",
+		Format: "qwen3guard", GPU: true, About: "Qwen3Guard-Gen 4B on a GPU, 2.5 GB"},
+	{Name: "qwen3guard-8b", Repo: "mradermacher/Qwen3Guard-Gen-8B-GGUF", File: "Qwen3Guard-Gen-8B.Q4_K_M.gguf",
+		Format: "qwen3guard", GPU: true, About: "Qwen3Guard-Gen 8B on a GPU, 4.7 GB"},
+	{Name: "llama-guard-3-1b", Repo: "QuantFactory/Llama-Guard-3-1B-GGUF", File: "Llama-Guard-3-1B.Q4_K_M.gguf",
+		Format: "llamaguard3", About: "Llama Guard 3 1B on CPUs, 0.9 GB: refuses ordinary coding requests"},
+	{Name: "llama-guard-3-8b", Repo: "QuantFactory/Llama-Guard-3-8B-GGUF", File: "Llama-Guard-3-8B.Q4_K_M.gguf",
+		Format: "llamaguard3", GPU: true, About: "Llama Guard 3 8B on a GPU, 4.6 GB"},
+}
+
+const (
+	defaultJudge    = "qwen3guard-0.6b"
+	defaultGPUJudge = "qwen3guard-4b"
+)
+
+// baselineCategories are what the baseline policy counts of each format's
+// verdicts: not PII, which is the patterns', nor what refuses ordinary work
+// — copyright, unethical acts, politically sensitive topics; Llama Guard's
+// defamation, specialized advice, intellectual property and elections.
+var baselineCategories = map[string][]string{
+	"qwen3guard":  {"Violent", "Non-violent Illegal Acts", "Sexual Content or Sexual Acts", "Suicide & Self-Harm", "Jailbreak"},
+	"llamaguard3": {"S1", "S2", "S3", "S4", "S7", "S9", "S10", "S11", "S12"},
+}
+
+// reported is whether a judge of that format ever names category c: its own
+// names, or for Llama Guard its codes.
+func reported(format, c string) bool {
+	if slices.ContainsFunc(guard.DefaultCategories(format), func(n string) bool { return strings.EqualFold(n, c) }) {
+		return true
+	}
+	var n int
+	_, err := fmt.Sscanf(c, "S%d", &n)
+	return format == "llamaguard3" && err == nil && n >= 1 && n <= 14
+}
+
+// categoriesFor is what a policy asking these judges should count, when the
+// categories it has are ones none of them names: a policy moved from Llama
+// Guard's S1 to Qwen3Guard, which says Violent, would otherwise count
+// nothing. Nil keeps what it has; so does a judge llmgw does not know.
+func categoriesFor(judges, have []string) []string {
+	var formats []string
+	for _, j := range judges {
+		k, ok := knownJudgeNamed(j)
+		if !ok {
+			return nil
+		}
+		if !slices.Contains(formats, k.Format) {
+			formats = append(formats, k.Format)
+		}
+	}
+	if len(have) == 0 {
+		return nil
+	}
+	stray := false
+	for _, f := range formats {
+		for _, c := range have {
+			if !reported(f, c) && !slices.ContainsFunc(formats, func(o string) bool { return reported(o, c) }) {
+				stray = true
+			}
+		}
+		// Every judge must have something it counts.
+		if !slices.ContainsFunc(have, func(c string) bool { return reported(f, c) }) {
+			stray = true
+		}
+	}
+	if !stray {
+		return nil
+	}
+	var out []string
+	for _, f := range formats {
+		out = append(out, baselineCategories[f]...)
+	}
+	return out
+}
+
+func knownJudgeNamed(name string) (knownJudge, bool) {
+	i := slices.IndexFunc(knownJudges, func(k knownJudge) bool { return k.Name == name })
+	if i < 0 {
+		return knownJudge{}, false
+	}
+	return knownJudges[i], true
+}
+
+func knownJudgeNames() string {
+	var n []string
+	for _, k := range knownJudges {
+		n = append(n, k.Name)
+	}
+	return strings.Join(n, ", ")
+}
+
+// judgeServeOptions are how a judge is served beyond which weights.
+type judgeServeOptions struct {
+	Replicas, Cores int
+	// GPUClass is the cards a GPU judge runs on; empty lets the plan choose.
+	GPUClass string
+}
+
+// serveRequest is the serve job for a judge. A judge on CPUs is critical,
+// as every judged request waits on it; one on a GPU is standard, so it takes
+// a card only from what may give one up, and the judge on CPUs after it in
+// the policies stands in while it has none.
+func (k knownJudge) serveRequest(o judgeServeOptions) modelserve.Request {
+	req := modelserve.Request{
+		Repo: k.Repo, File: k.File, Name: k.Name, GuardFormat: k.Format,
+		Description: "The guardrails' judge: reads prompts and answers for unsafe content. Not offered to clients.",
+		Replicas:    o.Replicas, CtxPool: 8192, ContextLength: 4096,
+	}
+	if k.GPU {
+		req.GPUClass, req.GPUsPerNode, req.Class = o.GPUClass, 1, catalog.ClassStandard
+		if req.Replicas == 0 {
+			req.Replicas = 1
+		}
+		return req
+	}
+	req.CPU, req.CPUCores = true, o.Cores
+	if req.Replicas == 0 {
+		req.Replicas = 2
+	}
+	return req
+}
+
+func (k knownJudge) String() string {
+	where := "CPUs"
+	if k.GPU {
+		where = "a GPU"
+	}
+	return fmt.Sprintf("%s (%s, %s/%s, on %s)", k.Name, k.Format, k.Repo, k.File, where)
+}

internal/cli/mirrorinit.go

@@ -89,10 +89,7 @@ func mirrorInit(ctx context.Context, out io.Writer, dir, base string, controlPla
 	if err := in.chown(dir, true); err != nil {
 		return err
 	}
-	addr := ""
-	if ips := localAddrs(); len(ips) > 0 {
-		addr = ips[0].String()
-	}
+	addr := primaryAddr()
 	if base == "" {
 		if addr == "" {
 			return errors.New("this machine has no address to give nodes; give one with --base")

internal/cli/model_serve.go

@@ -73,6 +73,8 @@ func newModelServeCmd(client func() *ctlClient) *cobra.Command {
 	f.StringSliceVar(&req.Aliases, "alias", nil, "other names clients may use; repeatable")
 	f.StringVar(&req.Description, "description", "", "what it is for, shown to clients in /v1/models")
 	f.IntVar(&req.Replicas, "replicas", 1, "how many replicas to run")
+	f.BoolVar(&req.Tools, "tools", false,
+		"say it makes tool calls (for agents and editors); check a running one with llmgw ctl tool-check")
 	f.StringVar(&req.GuardFormat, "guard-format", "",
 		"serve it as the guardrails' judge (role: guard), read as llamaguard3, shieldgemma or json; hidden from clients")
 	f.BoolVar(&onDemand, "on-demand", false, "run none until first asked for")

internal/cli/node.go

@@ -5,6 +5,8 @@ import (
 	"errors"
 	"fmt"
 	"io"
+	"net"
+	"net/netip"
 	"os"
 	"slices"
 	"strings"
@@ -16,7 +18,9 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/pgha"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/release"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/version"
 )
@@ -164,23 +168,36 @@ func (na *nodeAdd) prepare() error {
 	// are not known until the node says what it has, and are not applied
 	// here; doctor's verdict on vLLM is a warning either way.
 	if o.VLLMVersion == "" {
-		if cfg, err := config.Load(na.serverCfg); err == nil {
+		if cfg, err := loadEffectiveConfig(na.serverCfg); err == nil {
 			o.VLLMVersion = cfg.WantedEngines()["vllm"]
 			if o.VLLMVersion == "" {
 				o.VLLMVersion = "none"
 			}
 		}
 	}
-	// The node's ports — its engines', its metrics — for the cluster only.
-	if cfg, err := config.Load(na.serverCfg); err == nil && !o.NoFirewall {
+	// The node's ports — its engines', its metrics — for the cluster only,
+	// unless the cluster's settings turned llmgw's firewall off.
+	if cfg, err := loadEffectiveConfig(na.serverCfg); err == nil && !cfg.Firewall.Enabled {
+		o.NoFirewall = true
+	} else if err == nil && !o.NoFirewall {
 		from, to := cfg.EnginePorts()
 		o.FirewallPorts = []firewall.Port{firewall.Engines(from, to), firewall.NodeExporter}
 		o.FirewallAllow, _ = cfg.FirewallAllow()
 	}
+	// Where the mirror serves model files, for the node to be asked to reach.
+	if cfg, err := loadEffectiveConfig(na.serverCfg); err == nil && cfg.Mirror.API != "" {
+		if mc, err := mirror.NewClient(cfg.Mirror.API, cfg.Mirror.TLS, 5*time.Second); err == nil {
+			ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+			if list, err := mc.List(ctx); err == nil {
+				o.MirrorBase = list.Base
+			}
+			cancel()
+		}
+	}
 	// Where the control plane upgrades nodes from, so a node joins on the
 	// same release from the same place: an offline cluster's mirror.
 	if o.Releases == "" {
-		if cfg, err := config.Load(na.serverCfg); err == nil {
+		if cfg, err := loadEffectiveConfig(na.serverCfg); err == nil {
 			o.Releases = cfg.ReleasesBase()
 		}
 		if o.Releases == "" {
@@ -266,6 +283,9 @@ func (na *nodeAdd) run(ctx context.Context, addrs []string, out io.Writer) error
 		if len(addrs) > 1 {
 			title += fmt.Sprintf(" (%d of %d)", i+1, len(addrs))
 		}
+		if !o.NoFirewall {
+			na.letIn(ctx, o.Addr, out)
+		}
 		var node string
 		var err error
 		if tui {
@@ -297,6 +317,37 @@ func (na *nodeAdd) run(ctx context.Context, addrs []string, out io.Writer) error
 	return nil
 }
 
+// letIn opens this control plane's firewall to the node, when it has one
+// and the node is on a network it does not already let in: otherwise a node
+// on another subnet is set up and then never heard from. Said, not fatal.
+func (na *nodeAdd) letIn(ctx context.Context, host string, out io.Writer) {
+	var addrs []string
+	if a, err := netip.ParseAddr(host); err == nil {
+		addrs = []string{a.String()}
+	} else {
+		ips, err := net.DefaultResolver.LookupNetIP(ctx, "ip4", host)
+		if err != nil {
+			fmt.Fprintf(out, "  !!    firewall: cannot resolve %s (%v); if it is on another network, "+
+				"list it in firewall_networks and run 'llmgw server init'\n", host, err)
+			return
+		}
+		for _, ip := range ips {
+			addrs = append(addrs, ip.Unmap().String())
+		}
+	}
+	added, err := firewall.Admit(ctx, pgha.Local{}, addrs)
+	switch {
+	case err != nil:
+		fmt.Fprintf(out, "  !!    firewall: could not let %s in: %v\n", host, err)
+	case len(added) > 0:
+		shown := make([]string, len(added))
+		for i, p := range added {
+			shown[i] = p.String()
+		}
+		fmt.Fprintf(out, "  ok    firewall: this control plane now lets in %s\n", strings.Join(shown, ", "))
+	}
+}
+
 func (na *nodeAdd) runPlain(ctx context.Context, o onboard.Options, title string, out io.Writer) (string, error) {
 	p := &plainProgress{out: out}
 	fmt.Fprintf(out, "%s\n", title)

internal/cli/root.go

@@ -59,8 +59,14 @@ func Root() *cobra.Command {
 		newSelfUpgradeCmd(),
 		newReleaseCmd(),
 		newJoinCmd(),
+		newFirewallCmd(),
+		newWipeCmd(),
 		newDocsCmd(),
 	)
+	// status, start, stop and restart: the whole system; service: this
+	// machine's services.
+	root.AddCommand(newSystemCmds()...)
+	root.AddCommand(newServiceCmd())
 	return root
 }
 

internal/cli/server.go

@@ -16,6 +16,7 @@ import (
 	"slices"
 	"strings"
 	"sync"
+	"sync/atomic"
 	"syscall"
 	"time"
 
@@ -80,10 +81,13 @@ func newServerCmd() *cobra.Command {
 		RunE: func(cmd *cobra.Command, _ []string) error {
 			log := logger()
 
-			cfg, err := config.Load(configPath)
+			// server.yaml for this machine's own keys, the store's settings
+			// for the cluster's: the same on every control plane.
+			boot, err := loadClusterSettings(cmd.Context(), configPath, log)
 			if err != nil {
 				return err
 			}
+			cfg := boot.cfg
 			if listen != "" {
 				cfg.Listen = listen
 			}
@@ -166,7 +170,7 @@ func newServerCmd() *cobra.Command {
 				}
 				gate.SetJudge(rt.Judge, rt.GuardFormat)
 				rt.SetGuard(gate)
-				rt.SetGuardLibrary(cfg.Guard)
+				rt.SetGuardLibrary(gate.Library)
 				if err := rt.RegisterMetrics(gate); err != nil {
 					return fmt.Errorf("guard metrics: %w", err)
 				}
@@ -175,9 +179,14 @@ func newServerCmd() *cobra.Command {
 
 			ctx, stop := signal.NotifyContext(cmd.Context(), syscall.SIGINT, syscall.SIGTERM)
 			defer stop()
+			// Cancelled to start again on changed settings.
+			ctx, cancelRun := context.WithCancel(ctx)
+			defer cancelRun()
+			var restartReason atomic.Pointer[string]
 
 			rt.Start(ctx)
 			defer rt.Stop()
+			rt.SetStopped(cfg.Serving.Stopped)
 
 			// Which cluster this is, to federated peers: the id in the hop
 			// header that stops a request going round two clusters that
@@ -215,6 +224,10 @@ func newServerCmd() *cobra.Command {
 			// The control plane replaces the static endpoint list: agents
 			// register, the reconciler places replicas, and Ready ones are
 			// published to the router (plan §6.4).
+			// The settings in force, for what reads them at each use.
+			var current atomic.Pointer[config.Server]
+			current.Store(&cfg)
+			var agentSvc *agentsvc.Service
 			var agentHandler http.Handler
 			var agentPath string
 			var admin *control.Controller
@@ -239,6 +252,16 @@ func newServerCmd() *cobra.Command {
 					return err
 				}
 				defer ctl.Close()
+				agentSvc = svc
+				// Stopped by the operators (llmgw stop): nothing placed.
+				ctl.Reconciler().SetStopped(cfg.Serving.Stopped)
+				// Whether nodes keep llmgw's firewall, as the settings say now.
+				svc.SetFirewall(func() *llmgwv1.Firewall {
+					c := current.Load()
+					from, to := c.EnginePorts()
+					return &llmgwv1.Firewall{Enabled: c.Firewall.Enabled, Networks: c.FirewallNetworks,
+						EnginePortsFrom: int32(from), EnginePortsTo: int32(to)} //nolint:gosec // ports
+				})
 				ctl.Reconciler().SetNotifier(notifier)
 				admin = ctl
 				agentPath, agentHandler = svc.Handler()
@@ -341,12 +364,20 @@ func newServerCmd() *cobra.Command {
 				evals = newEvalService(cfg, rt, admin, log)
 				api.SetEval(evals)
 				if st := rt.Store(); st != nil && gate != nil {
-					api.SetGuard(&adminapi.GuardAdmin{Library: cfg.Guard, Store: st, Invalidate: gate.Invalidate})
+					api.SetGuard(&adminapi.GuardAdmin{Library: gate.Library, Store: st, Invalidate: gate.Invalidate,
+						JudgeErrors: gate.JudgeErrors, Ask: gate.AskJudge})
 				}
 				if st := rt.Store(); st != nil {
 					// Removing a node revokes its certificate.
 					api.SetRevoker(st.RevokeNode)
+					if agentSvc != nil {
+						api.SetAgentRestarter(agentSvc.RestartAgents)
+					}
 					api.SetAudit(st)
+					// The cluster's settings, each change checked against this
+					// machine's own keys before it is kept.
+					api.SetSettings(&adminapi.Settings{Store: st, LocalKeys: config.LocalKeys(),
+						Check: func(body []byte) error { return checkClusterSettings(configPath, body) }})
 				}
 				// Catalog changes need somewhere to persist, so this is wired
 				// only when there is a store. Without one, the endpoints say so
@@ -378,6 +409,17 @@ func newServerCmd() *cobra.Command {
 					} else {
 						admin.ReplaceCatalog(ctx, built)
 					}
+					// And again on becoming the leader: models added or
+					// removed through the leader before are in the store.
+					if l := admin.Leadership(); l != nil {
+						l.OnLead(func() {
+							if built, err := svc.Load(ctx); err != nil {
+								log.Error("could not read the stored models on becoming the leader", "err", err)
+							} else {
+								admin.ReplaceCatalog(ctx, built)
+							}
+						})
+					}
 					api.SetModels(svc)
 					catalogSvc = svc
 					// And the mirror page, so a model that has just been
@@ -476,6 +518,21 @@ func newServerCmd() *cobra.Command {
 					})
 					go serve.WatchCanaries(ctx, 30*time.Second)
 					adminAPI.SetServe(serve)
+					if gate != nil {
+						// A judge a policy names and the catalog lacks is
+						// fetched and served, on the leader.
+						keeper := &judgeKeeper{
+							library: gate.Library,
+							models:  func() []catalog.Model { return admin.Catalog().Models() },
+							serve:   serve,
+							leader: func() bool {
+								return admin.Leadership() == nil || admin.Leadership().IsLeader()
+							},
+							gpus: func() []adminapi.ClassUse { return adminAPI.Summary().GPUs },
+							log:  log,
+						}
+						go keeper.run(ctx, 30*time.Second)
+					}
 					updateMetrics.Set(serve)
 					go serve.WatchUpdates(ctx, cfg.Mirror.UpdateCheck)
 				}
@@ -627,6 +684,67 @@ func newServerCmd() *cobra.Command {
 				}
 			}
 
+			// Every change to the settings in force: what can change live
+			// does, the rest starts this process again, which systemd brings
+			// back on them.
+			// And this machine's own firewall, through its sync unit.
+			if err := writeFirewallState(cfg); err != nil {
+				log.Warn("could not record the firewall's settings for this machine", "err", err)
+			}
+			watch := &settingsWatch{path: configPath, st: rt.Store(), cur: boot, log: log,
+				live: func(next config.Server, changed []string) bool {
+					if !onlyLive(changed) {
+						return false
+					}
+					current.Store(&next)
+					if slices.ContainsFunc(changed, func(c string) bool { return strings.HasPrefix(c, "serving") }) {
+						rt.SetStopped(next.Serving.Stopped)
+						if admin != nil {
+							admin.Reconciler().SetStopped(next.Serving.Stopped)
+						}
+						log.Warn("the service is " + map[bool]string{true: "stopped", false: "started"}[next.Serving.Stopped] +
+							" by the cluster's settings")
+					}
+					if slices.ContainsFunc(changed, func(c string) bool { return strings.HasPrefix(c, "firewall") }) {
+						if err := writeFirewallState(next); err != nil {
+							log.Warn("could not record the firewall's settings for this machine", "err", err)
+						}
+						if agentSvc != nil {
+							agentSvc.BroadcastFirewall()
+						}
+					}
+					if slices.ContainsFunc(changed, func(c string) bool { return strings.HasPrefix(c, "guard") }) {
+						if gate == nil {
+							// Built at start, and only when there are policies.
+							return len(next.Guard.Policies) == 0
+						}
+						if err := gate.Replace(next.Guard); err != nil {
+							log.Error("the changed guard policies are not valid; the old ones stay in force", "err", err)
+						}
+					}
+					return true
+				},
+				restart: func(reason string) {
+					if os.Getenv("INVOCATION_ID") == "" {
+						log.Warn("the settings changed in a way this process applies only when it starts; "+
+							"it is not run by systemd, so start it again by hand", "why", reason)
+						return
+					}
+					// The leader last, so its standbys are already on the new
+					// settings when it hands over to one of them.
+					delay := time.Duration(0)
+					if admin != nil && admin.Leadership() != nil && admin.Leadership().IsLeader() && rt.Store() != nil {
+						if cps, err := rt.Store().ControlPlanes(ctx); err == nil && len(cps) > 1 {
+							delay = 30 * time.Second
+						}
+					}
+					log.Warn("starting again to apply the settings", "why", reason, "in", delay)
+					restartReason.Store(&reason)
+					time.AfterFunc(delay, cancelRun)
+				},
+			}
+			go watch.run(ctx)
+
 			g, gctx := errgroup.WithContext(ctx)
 			g.Go(func() error { return serve(apiSrv, apiLn) })
 			g.Go(func() error { return serve(adminSrv, adminLn) })
@@ -658,7 +776,12 @@ func newServerCmd() *cobra.Command {
 				_ = adminSrv.Shutdown(sctx)
 				return apiSrv.Shutdown(sctx)
 			})
-			return g.Wait()
+			err = g.Wait()
+			if r := restartReason.Load(); r != nil {
+				// Not a failure, but said as one, so systemd starts it again.
+				return fmt.Errorf("stopped to start again on changed settings (%s)", *r)
+			}
+			return err
 		},
 	}
 
@@ -756,6 +879,14 @@ func startControlPlane(ctx context.Context, cfg config.Server, cat *catalog.Live
 			ctl.SetEngineChoices(control.NewEngineChoices(ctx, st, log))
 			// Domains held for maintenance and labels set on nodes.
 			ctl.Facility().UseStore(ctx, st)
+			// A standby read these when it started; the leader before it may
+			// have changed them since.
+			lead.OnLead(func() {
+				if ch := ctl.EngineChoices(); ch != nil {
+					ch.Reload(ctx)
+				}
+				ctl.Facility().Reload(ctx)
+			})
 			log.Info("competing to be the control plane that places work",
 				"owner", lead.Owner())
 		}

internal/cli/serveradd.go

@@ -155,10 +155,18 @@ func (sa *serverAdd) run(ctx context.Context) error {
 // firewall keeps the new control plane's internal ports to the networks it
 // reaches this one on. Not fatal: it serves without one.
 func (sa *serverAdd) firewall(ctx context.Context) error {
-	cfg, err := config.Load(filepath.Join(sa.etcDir, "server.yaml"))
+	cfg, err := loadEffectiveConfig(filepath.Join(sa.etcDir, "server.yaml"))
 	if err != nil {
 		return err
 	}
+	// What applies the cluster's firewall setting there when it changes.
+	if err := installFirewallSync(ctx, sa.remote); err != nil {
+		fmt.Fprintf(sa.out, "  !!    firewall: %v\n", err)
+	}
+	if !cfg.Firewall.Enabled {
+		sa.kept("firewall: off, as the cluster's settings say (firewall.enabled)")
+		return nil
+	}
 	self, err := sourceIPToward(sa.ip)
 	if err != nil {
 		return err

internal/cli/serverinit.go

@@ -288,12 +288,12 @@ func (in *serverInit) judgeWanted() bool {
 // configuration written before init installed one is never changed, so the
 // model is not fetched for nothing.
 func (in *serverInit) configJudgesWith(model string) bool {
-	cfg, err := config.Load(in.configPath())
+	cfg, err := loadEffectiveConfig(in.configPath())
 	if err != nil {
 		return false
 	}
 	for _, p := range cfg.Guard.Policies {
-		if p.Judge != nil && p.Judge.Model == model {
+		if slices.Contains(p.Judge.Judges(), model) {
 			return true
 		}
 	}
@@ -308,18 +308,21 @@ func guardSection(judge bool) string {
       # judge: {model: ` + guardModelName + `, input: block, output: block, ...}
 `
 	if judge {
-		j = `      # A judge model reads what patterns cannot: violence, weapons, abuse,
-      # self-harm. Two replicas run on CPUs with AVX2, about half a second
-      # for a short text; an answer is read every 600 characters as it
-      # streams. With fail: closed, a request the judge cannot read within
+		j = `      # A judge model reads what patterns cannot: violence, crime, sexual
+      # content, self-harm, jailbreaks. Two replicas run on CPUs, about two
+      # seconds for a short text; an answer is read every 600 characters as
+      # it streams. With fail: closed, a request no judge can read within
       # latency_budget is refused (503); open serves it, flagged, unjudged.
-      # Defamation (S5), specialized advice (S6), intellectual property (S8)
-      # and elections (S13) are left out: they refuse too much ordinary work.
+      # PII is the patterns' (above), and copyright, unethical acts and
+      # politically sensitive topics are left out: they refuse too much
+      # ordinary work. Controversial counts as safe; controversial: unsafe
+      # makes it count. With a GPU: llmgw ctl guard judge enable --gpu, then
+      # llmgw ctl guard judge use ` + defaultGPUJudge + ` ` + guardModelName + `.
       judge:
-        model: ` + guardModelName + `
+        models: [` + guardModelName + `]
         input: block
         output: block
-        categories: [S1, S2, S3, S4, S7, S9, S10, S11, S12]
+        categories: [Violent, Non-violent Illegal Acts, Sexual Content or Sexual Acts, Suicide & Self-Harm, Jailbreak]
         latency_budget: 5s
 `
 	}
@@ -821,3 +824,58 @@ Before people outside this network use it:
   #   in server.yaml, and copy them off this machine
 `)
 }
+
+// primaryAddr is the address other machines most likely reach this one on:
+// the source of its default route, else the first address on an interface
+// that is not a container's or a bridge's. The first address of all is not
+// it on a machine with Docker, whose 172.17.0.1 a node takes for its own.
+func primaryAddr() string {
+	if out, err := exec.Command("ip", "-4", "route", "get", "1.1.1.1").Output(); err == nil {
+		if src := routeSource(string(out)); src != "" {
+			return src
+		}
+	}
+	ifaces, err := net.Interfaces()
+	if err == nil {
+		for _, ifc := range ifaces {
+			if ifc.Flags&net.FlagUp == 0 || virtualInterface(ifc.Name) {
+				continue
+			}
+			addrs, _ := ifc.Addrs()
+			for _, a := range addrs {
+				if n, ok := a.(*net.IPNet); ok && n.IP.To4() != nil && n.IP.IsGlobalUnicast() {
+					return n.IP.String()
+				}
+			}
+		}
+	}
+	if ips := localAddrs(); len(ips) > 0 {
+		return ips[0].String()
+	}
+	return ""
+}
+
+// routeSource reads the src of `ip route get`.
+func routeSource(out string) string {
+	f := strings.Fields(out)
+	for i := 0; i+1 < len(f); i++ {
+		if f[i] == "src" {
+			if ip := net.ParseIP(f[i+1]); ip != nil && !ip.IsLoopback() {
+				return ip.String()
+			}
+		}
+	}
+	return ""
+}
+
+// virtualInterface is an interface containers, bridges or overlays made,
+// whose address no other machine reaches this one on.
+func virtualInterface(name string) bool {
+	for _, p := range []string{"docker", "br-", "virbr", "veth", "cni", "flannel", "cali", "vxlan",
+		"kube", "lxc", "lxd", "podman", "cilium", "weave", "tun", "tap"} {
+		if strings.HasPrefix(name, p) {
+			return true
+		}
+	}
+	return false
+}

internal/cli/serverinit_bundle.go

@@ -12,6 +12,7 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/bundle"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
 )
 
@@ -60,7 +61,7 @@ func (in *serverInit) offline() bool {
 	if in.bundle != "" {
 		return true
 	}
-	cfg, err := config.Load(in.configPath())
+	cfg, err := loadEffectiveConfig(in.configPath())
 	return err == nil && cfg.Offline.Enabled()
 }
 
@@ -76,7 +77,7 @@ func (in *serverInit) mirrorRoot() string {
 // server.yaml that the cluster is offline. With a mirror on another
 // machine, the files are imported there; here only the checksums are kept.
 func (in *serverInit) ensureBundle(ctx context.Context) error {
-	cfg, err := config.Load(in.configPath())
+	cfg, err := loadEffectiveConfig(in.configPath())
 	if err != nil {
 		return err
 	}
@@ -129,6 +130,24 @@ func (in *serverInit) ensureBundle(ctx context.Context) error {
 	if sums == config.DefaultOfflineChecksums {
 		sums = ""
 	}
+	// In the cluster's settings once the store holds them; in server.yaml
+	// before, where the first start brings it in.
+	stored, err := changeStoredSettings(in.configPath(), "offline, from server init --bundle", func(doc *settings.Doc) error {
+		if err := doc.SetYAML([]string{"offline", "mirror"}, files); err != nil {
+			return err
+		}
+		if sums != "" {
+			return doc.SetYAML([]string{"offline", "checksums"}, sums)
+		}
+		return nil
+	})
+	if err != nil {
+		return err
+	}
+	if stored {
+		in.done("the cluster's settings: offline, nodes install from the mirror at %s", files)
+		return nil
+	}
 	if err := appendOfflineBlock(in.configPath(), files, sums); err != nil {
 		return err
 	}

internal/cli/serverinit_cluster.go

@@ -41,11 +41,13 @@ const (
 	starterSource  = "hf:Qwen/Qwen2.5-0.5B-Instruct-GGUF/qwen2.5-0.5b-instruct-q4_k_m.gguf"
 	starterVersion = "starter"
 
-	// The guardrails' judge: Meta's Llama Guard 3 1B, which reads a prompt
-	// or an answer and says whether it is unsafe and why. Q4_K_M is about
-	// 900 MB and judges a short text in about half a second on a CPU.
-	guardModelName    = "llama-guard-3-1b"
-	guardModelSource  = "hf:QuantFactory/Llama-Guard-3-1B-GGUF/Llama-Guard-3-1B.Q4_K_M.gguf"
+	// The guardrails' judge: Qwen3Guard-Gen 0.6B, which reads a prompt or an
+	// answer and says whether it is safe, controversial or unsafe, and why.
+	// Q8_0 is 0.8 GB and judges a short text in about two seconds on two CPU
+	// threads. It replaced Llama Guard 3 1B, which refused about one in two
+	// ordinary coding requests ("continue").
+	guardModelName    = defaultJudge
+	guardModelSource  = "hf:mradermacher/Qwen3Guard-Gen-0.6B-GGUF/Qwen3Guard-Gen-0.6B.Q8_0.gguf"
 	guardModelVersion = "guard"
 
 	// localLabel marks the control plane's own node, so the starter model is
@@ -86,8 +88,8 @@ func (in *serverInit) reachableAddr() string {
 	if len(in.hosts) > 0 {
 		return in.hosts[0]
 	}
-	if ips := localAddrs(); len(ips) > 0 {
-		return ips[0].String()
+	if a := primaryAddr(); a != "" {
+		return a
 	}
 	name, _ := os.Hostname()
 	return name
@@ -443,12 +445,6 @@ func starterModel(source string, gpuClasses []string, pinned bool) catalog.Model
 	}
 }
 
-// llamaGuard3Categories name Llama Guard 3's S1 to S13, so a verdict is
-// recorded as "Violent Crimes" rather than "S1".
-var llamaGuard3Categories = []string{"Violent Crimes", "Non-Violent Crimes", "Sex-Related Crimes",
-	"Child Sexual Exploitation", "Defamation", "Specialized Advice", "Privacy", "Intellectual Property",
-	"Indiscriminate Weapons", "Hate", "Suicide & Self-Harm", "Sexual Content", "Elections"}
-
 // guardModel is the judge's catalog entry: CPU only, so it takes no card from
 // the models people use, on whichever node has room.
 func guardModel(source string) catalog.Model {
@@ -457,7 +453,7 @@ func guardModel(source string) catalog.Model {
 		Description: "The guardrails' judge: reads prompts and answers for unsafe content. Not offered to clients.",
 		Source:      source,
 		Role:        catalog.RoleGuard,
-		Guard:       &catalog.GuardModel{Format: "llamaguard3", Categories: llamaGuard3Categories},
+		Guard:       &catalog.GuardModel{Format: "qwen3guard"},
 		// Two: every judged request waits on it, and with fail: closed a
 		// judge that is not there refuses everything.
 		Scaling: catalog.Scaling{Min: 2, Max: 2},

internal/cli/serverinit_cluster_test.go

@@ -7,12 +7,14 @@ import (
 	"net/http/httptest"
 	"os"
 	"path/filepath"
+	"slices"
 	"strings"
 	"testing"
 	"time"
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
 )
 
 func TestTheInitAgentUnitIsTheShippedUnit(t *testing.T) {
@@ -150,9 +152,17 @@ func TestTheInstalledConfigurationIsHardenedAndLoads(t *testing.T) {
 		if (p.Judge != nil) != judge {
 			t.Errorf("judge=%v: the policy's judge is %+v", judge, p.Judge)
 		}
-		if judge && (p.Judge.Model != guardModelName || p.Judge.Input != "block") {
+		if judge && (!slices.Equal(p.Judge.Judges(), []string{guardModelName}) || p.Judge.Input != "block") {
 			t.Errorf("judge %+v", p.Judge)
 		}
+		// A category the judge never names would never count.
+		if judge {
+			for _, c := range p.Judge.Categories {
+				if !slices.Contains(guard.DefaultCategories("qwen3guard"), c) {
+					t.Errorf("the baseline counts %q, which Qwen3Guard does not name", c)
+				}
+			}
+		}
 		if cfg.Auth.MaxKeyTTL != 90*24*time.Hour || cfg.Auth.SignupOpen || cfg.DebugPprof || cfg.AdminRemote {
 			t.Errorf("not hardened: ttl %v signup %v pprof %v remote %v",
 				cfg.Auth.MaxKeyTTL, cfg.Auth.SignupOpen, cfg.DebugPprof, cfg.AdminRemote)
@@ -161,13 +171,13 @@ func TestTheInstalledConfigurationIsHardenedAndLoads(t *testing.T) {
 }
 
 func TestTheGuardModelIsAValidJudge(t *testing.T) {
-	m := guardModel("http://10.0.0.5:8081/llama-guard-3-1b/guard/g.gguf")
+	m := guardModel("http://10.0.0.5:8081/qwen3guard-0.6b/guard/g.gguf")
 	c, err := catalog.New([]catalog.Model{m})
 	if err != nil {
 		t.Fatalf("not a valid catalog entry: %v", err)
 	}
 	got, _ := c.Resolve(guardModelName)
-	if !got.IsGuard() || got.Guard.Format != "llamaguard3" || len(got.Guard.Categories) != 13 || got.Scaling.Min != 2 {
+	if !got.IsGuard() || got.Guard.Format != "qwen3guard" || got.Scaling.Min != 2 {
 		t.Errorf("%+v", got)
 	}
 	for _, p := range got.Profiles {

internal/cli/serverinit_firewall.go

@@ -19,10 +19,19 @@ import (
 // the engines and etcd when they are here. The client API is not among
 // them; it is meant to be reached.
 func (in *serverInit) ensureFirewall(ctx context.Context) error {
-	cfg, err := config.Load(in.configPath())
+	cfg, err := loadEffectiveConfig(in.configPath())
 	if err != nil {
 		return err
 	}
+	// What applies the cluster's setting here when it changes: the server
+	// writes it, as llmgw; this applies it, as root.
+	if err := in.ensureFirewallSync(ctx); err != nil {
+		return err
+	}
+	if !cfg.Firewall.Enabled {
+		in.kept("firewall: off, as the cluster's settings say (firewall.enabled)")
+		return nil
+	}
 	ports := controlPlanePorts(cfg)
 	if !in.noMirror && in.mirrorURL == "" {
 		ports = append(ports, firewall.MirrorFiles)
@@ -122,3 +131,9 @@ func loopbackListen(listen string) bool {
 	ip := net.ParseIP(host)
 	return host == "localhost" || (ip != nil && ip.IsLoopback())
 }
+
+// ensureFirewallSync installs the units that apply the cluster's firewall
+// setting on this control plane.
+func (in *serverInit) ensureFirewallSync(ctx context.Context) error {
+	return installFirewallSync(ctx, firewall.Local{})
+}

internal/cli/serverinit_test.go

@@ -172,3 +172,22 @@ func TestInitKeepsCertificatesFromAnAuthorityElsewhere(t *testing.T) {
 		t.Error("--host cannot be honoured without the authority, and should say so")
 	}
 }
+
+func TestTheAddressNodesReachIsNotAContainersBridge(t *testing.T) {
+	if got := routeSource("1.1.1.1 via 10.104.130.129 dev ens192 src 10.104.130.140 uid 0\n    cache\n"); got != "10.104.130.140" {
+		t.Errorf("route source %q", got)
+	}
+	if got := routeSource("unreachable"); got != "" {
+		t.Errorf("no route: %q", got)
+	}
+	for _, name := range []string{"docker0", "br-3f2a", "virbr0", "veth12ab", "cni0", "flannel.1"} {
+		if !virtualInterface(name) {
+			t.Errorf("%s taken for a real interface", name)
+		}
+	}
+	for _, name := range []string{"eth0", "ens192", "enp3s0", "bond0", "ib0"} {
+		if virtualInterface(name) {
+			t.Errorf("%s taken for a virtual one", name)
+		}
+	}
+}

internal/cli/service.go

@@ -0,0 +1,187 @@
+package cli
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"os"
+	"os/exec"
+	"runtime"
+	"strings"
+
+	"github.com/spf13/cobra"
+)
+
+// llmgwServices are the units llmgw runs as on a machine, in the order they
+// start: the store's clients after the store, the server before the agent
+// that dials it.
+var llmgwServices = []string{"llmgw-mirror", "llmgw-server", "llmgw-agent"}
+
+// unitDir is where llmgw's units are installed.
+var unitDir = "/etc/systemd/system"
+
+// serviceRunner runs systemctl; a variable so tests stand in for it.
+var serviceRunner = func(ctx context.Context, args ...string) (string, error) {
+	out, err := exec.CommandContext(ctx, "systemctl", args...).CombinedOutput() //nolint:gosec // fixed command
+	return strings.TrimSpace(string(out)), err
+}
+
+// installedServices are the llmgw units this machine has, or those named.
+func installedServices(ctx context.Context, names []string) ([]string, error) {
+	var want []string
+	for _, n := range names {
+		if !strings.HasPrefix(n, "llmgw-") {
+			n = "llmgw-" + n
+		}
+		want = append(want, n)
+	}
+	var have []string
+	for _, u := range llmgwServices {
+		if _, err := os.Stat(unitDir + "/" + u + ".service"); err != nil {
+			continue
+		}
+		if len(want) > 0 && !hasUnit(want, u) {
+			continue
+		}
+		have = append(have, u)
+	}
+	for _, w := range want {
+		if !hasUnit(have, w) {
+			return nil, fmt.Errorf("%s is not installed here (this machine has: %s)", w, strings.Join(present(), ", "))
+		}
+	}
+	if len(have) == 0 {
+		return nil, errors.New("no llmgw service is installed on this machine")
+	}
+	return have, nil
+}
+
+func present() []string {
+	var out []string
+	for _, u := range llmgwServices {
+		if _, err := os.Stat(unitDir + "/" + u + ".service"); err == nil {
+			out = append(out, strings.TrimPrefix(u, "llmgw-"))
+		}
+	}
+	if len(out) == 0 {
+		return []string{"none"}
+	}
+	return out
+}
+
+func hasUnit(list []string, s string) bool {
+	for _, x := range list {
+		if x == s {
+			return true
+		}
+	}
+	return false
+}
+
+// newServiceCmd is llmgw service: this machine's llmgw services, one by one.
+// The whole system is llmgw status, start, stop and restart.
+func newServiceCmd() *cobra.Command {
+	c := &cobra.Command{
+		Use:   "service",
+		Short: "This machine's llmgw services: status, start, stop, restart",
+		Long: "The control plane (server), node agent (agent) and model mirror (mirror) on\n" +
+			"this machine alone, by systemd. The whole system is 'llmgw status', 'llmgw\n" +
+			"stop', 'llmgw start' and 'llmgw restart', run on a control plane.",
+	}
+	c.AddCommand(serviceVerbs()...)
+	return c
+}
+
+func serviceVerbs() []*cobra.Command {
+	status := &cobra.Command{
+		Use:   "status [server|agent|mirror]...",
+		Short: "Whether llmgw's services on this machine run",
+		Long: "Each llmgw service this machine has — the control plane (server), the\n" +
+			"node agent (agent), the model mirror (mirror) — and whether it runs.",
+		RunE: func(cmd *cobra.Command, args []string) error {
+			units, err := localUnits(cmd, args)
+			if err != nil {
+				return err
+			}
+			out := cmd.OutOrStdout()
+			for _, u := range units {
+				state, _ := serviceRunner(cmd.Context(), "is-active", u)
+				since, _ := serviceRunner(cmd.Context(), "show", u, "-p", "ActiveEnterTimestamp", "--value")
+				fmt.Fprintf(out, "%-14s %-10s %s\n", strings.TrimPrefix(u, "llmgw-"), state, since)
+			}
+			return nil
+		},
+	}
+	verb := func(use, short, long, action string, reverse bool) *cobra.Command {
+		var engines bool
+		c := &cobra.Command{
+			Use:   use + " [server|agent|mirror]...",
+			Short: short,
+			Long:  long,
+			RunE: func(cmd *cobra.Command, args []string) error {
+				if os.Geteuid() != 0 {
+					return errors.New("starting and stopping services needs root: run it with sudo")
+				}
+				units, err := localUnits(cmd, args)
+				if err != nil {
+					return err
+				}
+				if reverse {
+					// The agent before the server it dials, the server before
+					// the mirror it reads.
+					for i, j := 0, len(units)-1; i < j; i, j = i+1, j-1 {
+						units[i], units[j] = units[j], units[i]
+					}
+				}
+				out := cmd.OutOrStdout()
+				for _, u := range units {
+					if msg, err := serviceRunner(cmd.Context(), action, u); err != nil {
+						return fmt.Errorf("%s %s: %v %s", action, u, err, msg)
+					}
+					state, _ := serviceRunner(cmd.Context(), "is-active", u)
+					fmt.Fprintf(out, "%-14s %s\n", strings.TrimPrefix(u, "llmgw-"), state)
+				}
+				if action == "stop" && hasUnit(units, "llmgw-agent") {
+					if engines {
+						_, _ = exec.CommandContext(cmd.Context(), "pkill", "-u", "llmgw-engine").CombinedOutput() //nolint:gosec // fixed command
+						fmt.Fprintln(out, "engines   stopped (every process of llmgw-engine)")
+					} else {
+						fmt.Fprintln(out, "\nThe engines keep serving: they outlive the agent, which takes them over again when it starts.\n"+
+							"--engines stops them too; the control plane then places their replicas elsewhere.")
+					}
+				}
+				if action == "stop" && hasUnit(units, "llmgw-server") {
+					fmt.Fprintln(out, "A standby control plane, where there is one, takes over within seconds.")
+				}
+				return nil
+			},
+		}
+		if action == "stop" {
+			c.Flags().BoolVar(&engines, "engines", false, "also stop the engines this node runs")
+		}
+		return c
+	}
+	return []*cobra.Command{
+		status,
+		verb("start", "Start llmgw's services on this machine",
+			"Starts each llmgw service this machine has, or those named: the mirror, the\n"+
+				"control plane, the node agent, in that order.", "start", false),
+		verb("stop", "Stop llmgw's services on this machine",
+			"Stops each llmgw service this machine has, or those named: the agent, the\n"+
+				"control plane, the mirror, in that order. The engines a node runs keep\n"+
+				"serving unless --engines is given; a standby control plane takes over from\n"+
+				"a leader stopped. They stay stopped until 'llmgw start', or the machine\n"+
+				"boots.", "stop", true),
+		verb("restart", "Restart llmgw's services on this machine",
+			"Restarts each llmgw service this machine has, or those named. Engines keep\n"+
+				"serving and are taken over again by the agent; a control plane's settings\n"+
+				"and server.yaml are read anew.", "restart", false),
+	}
+}
+
+func localUnits(cmd *cobra.Command, args []string) ([]string, error) {
+	if runtime.GOOS != "linux" {
+		return nil, fmt.Errorf("llmgw's services run under systemd on Linux; this is %s", runtime.GOOS)
+	}
+	return installedServices(cmd.Context(), args)
+}

internal/cli/service_test.go

@@ -0,0 +1,137 @@
+package cli
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"runtime"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+)
+
+func TestTheServicesHereAreFoundAndAskedAbout(t *testing.T) {
+	old, oldRun := unitDir, serviceRunner
+	unitDir = t.TempDir()
+	t.Cleanup(func() { unitDir, serviceRunner = old, oldRun })
+	for _, u := range []string{"llmgw-server", "llmgw-agent"} {
+		_ = os.WriteFile(filepath.Join(unitDir, u+".service"), []byte("[Unit]"), 0o644)
+	}
+	var asked []string
+	serviceRunner = func(_ context.Context, args ...string) (string, error) {
+		asked = append(asked, strings.Join(args, " "))
+		if args[0] == "is-active" {
+			return "active", nil
+		}
+		return "Wed 2026-10-07 12:00:00 UTC", nil
+	}
+
+	units, err := installedServices(context.Background(), nil)
+	if err != nil || strings.Join(units, ",") != "llmgw-server,llmgw-agent" {
+		t.Fatalf("%v %v", units, err)
+	}
+	if _, err := installedServices(context.Background(), []string{"mirror"}); err == nil ||
+		!strings.Contains(err.Error(), "server, agent") {
+		t.Errorf("a service not here: %v", err)
+	}
+	if u, err := installedServices(context.Background(), []string{"agent"}); err != nil || len(u) != 1 {
+		t.Errorf("one named: %v %v", u, err)
+	}
+
+	if runtime.GOOS != "linux" {
+		return
+	}
+	var out bytes.Buffer
+	c := serviceVerbs()[0]
+	c.SetOut(&out)
+	c.SetContext(context.Background())
+	if err := c.RunE(c, nil); err != nil {
+		t.Fatal(err)
+	}
+	if !strings.Contains(out.String(), "server") || !strings.Contains(out.String(), "active") {
+		t.Errorf("%s", out.String())
+	}
+}
+
+func TestStopSetsTheClustersServingStopped(t *testing.T) {
+	var put adminapi.SettingsRequest
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		switch {
+		case r.URL.Path == "/api/v1/summary":
+			_ = json.NewEncoder(w).Encode(adminapi.Summary{})
+		case r.Method == "GET" && r.URL.Path == "/api/v1/settings":
+			_ = json.NewEncoder(w).Encode(adminapi.SettingsView{Version: 3, Body: "router:\n  retries: 2\n"})
+		case r.Method == "PUT" && r.URL.Path == "/api/v1/settings":
+			_ = json.NewDecoder(r.Body).Decode(&put)
+			_ = json.NewEncoder(w).Encode(adminapi.SettingsView{Version: 4, Body: put.Body})
+		default:
+			http.NotFound(w, r)
+		}
+	}))
+	defer srv.Close()
+	dir := t.TempDir()
+	cfg := filepath.Join(dir, "server.yaml")
+	_ = os.WriteFile(filepath.Join(dir, "token"), []byte("t"), 0o600)
+	_ = os.WriteFile(cfg, []byte("listen: \":8080\"\nagent_insecure: true\nadmin_listen: "+
+		strings.TrimPrefix(srv.URL, "http://")+"\nadmin_remote: true\nadmin_token_file: "+filepath.Join(dir, "token")+"\n"), 0o600)
+
+	var stop *cobra.Command
+	for _, c := range newSystemCmds() {
+		if c.Name() == "stop" {
+			stop = c
+		}
+	}
+	var out bytes.Buffer
+	stop.SetOut(&out)
+	stop.SetArgs([]string{"--config", cfg, "--note", "maintenance"})
+	if err := stop.ExecuteContext(context.Background()); err != nil {
+		t.Fatalf("%v: %s", err, out.String())
+	}
+	if put.Base != 3 || !strings.Contains(put.Body, "stopped: true") || !strings.Contains(put.Body, "retries: 2") ||
+		put.Note != "maintenance" {
+		t.Errorf("sent %+v", put)
+	}
+	if !strings.Contains(out.String(), "stopping") {
+		t.Errorf("%s", out.String())
+	}
+}
+
+// Started while the control plane has not yet taken the setting up: every
+// model of a stopped system has the nothing it asks for, which is not
+// serving.
+func TestStartWaitsForTheControlPlaneToServe(t *testing.T) {
+	old := servicePoll
+	servicePoll = time.Millisecond
+	t.Cleanup(func() { servicePoll = old })
+	answers := []adminapi.Summary{
+		{Stopped: true, Models: adminapi.ModelTotals{Total: 2, Satisfied: 2}},
+		{Models: adminapi.ModelTotals{Total: 2, Satisfied: 0}},
+		{Models: adminapi.ModelTotals{Total: 2, Satisfied: 2}},
+	}
+	asked := 0
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		_ = json.NewEncoder(w).Encode(answers[min(asked, len(answers)-1)])
+		asked++
+	}))
+	defer srv.Close()
+
+	var out bytes.Buffer
+	cmd := &cobra.Command{}
+	cmd.SetOut(&out)
+	cmd.SetContext(context.Background())
+	c := &ctlClient{base: srv.URL, http: srv.Client()}
+	if err := waitForService(cmd, func() *ctlClient { return c }, false); err != nil {
+		t.Fatal(err)
+	}
+	if asked != 3 {
+		t.Errorf("returned after %d answers: %s", asked, out.String())
+	}
+}

internal/cli/status.go

@@ -153,6 +153,19 @@ func printSummary(cmd *cobra.Command, s adminapi.Summary) {
 		fmt.Fprintf(out, "STORE     %s\n", strings.Join(parts, ", "))
 	}
 
+	if s.Stopped {
+		fmt.Fprintln(out, "SERVICE   STOPPED: no model runs and requests are answered 503 (llmgw start)")
+	}
+	if s.GuardOff {
+		fmt.Fprintln(out, "GUARD     off: no request is checked, whatever the policies say (llmgw ctl guard on)")
+	}
+	if len(s.Judges) > 0 {
+		fmt.Fprintln(out, "\nGUARDRAILS' JUDGE")
+		for _, j := range s.Judges {
+			fmt.Fprintf(out, "  %s: %s\n", j.Model, j.Problem)
+		}
+	}
+
 	if len(s.Parked) > 0 {
 		fmt.Fprintln(out, "\nPARKED — failed every time they started; not retried until fixed")
 		for _, p := range s.Parked {

internal/cli/system.go

@@ -0,0 +1,232 @@
+package cli
+
+import (
+	"errors"
+	"fmt"
+	"os"
+	"os/exec"
+	"strings"
+	"time"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
+)
+
+// systemClient is the admin API of the control plane on this machine, as its
+// server.yaml says, with its token.
+func systemClient(configPath string) func() *ctlClient {
+	return func() *ctlClient {
+		addr := "127.0.0.1:9090"
+		var tokens []string
+		if cfg, ok := loadServerConfig(configPath); ok {
+			if cfg.AdminListen != "" {
+				addr = cfg.AdminListen
+				if strings.HasPrefix(addr, ":") {
+					addr = "127.0.0.1" + addr
+				}
+			}
+			tokens = append(tokens, adminTokenPath(configPath, cfg))
+		}
+		return newCtlClient(addr, tokens...)
+	}
+}
+
+// newSystemCmds are llmgw status, stop, start and restart: the whole system,
+// from a control plane.
+func newSystemCmds() []*cobra.Command {
+	var configPath string
+	client := systemClient("/etc/llmgw/server.yaml")
+	bind := func(c *cobra.Command) *cobra.Command {
+		c.Flags().StringVar(&configPath, "config", "/etc/llmgw/server.yaml", "this control plane's configuration")
+		c.PreRun = func(*cobra.Command, []string) { client = systemClient(configPath) }
+		return c
+	}
+	call := func() *ctlClient { return client() }
+
+	status := bind(&cobra.Command{
+		Use:   "status",
+		Short: "The whole system: whether it serves, its control planes, nodes, models and store",
+		Long: "Run on a control plane. Says whether the service is stopped (llmgw stop),\n" +
+			"and everything 'llmgw ctl status' does. One machine's services are\n" +
+			"'llmgw service status'.",
+		Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			var s adminapi.Summary
+			if err := call().get(cmd.Context(), "/api/v1/summary", &s); err != nil {
+				return notAControlPlane(err)
+			}
+			printSummary(cmd, s)
+			return nil
+		},
+	})
+
+	switchService := func(stop bool) *cobra.Command {
+		use, short, long := "start", "Start the whole system serving again, after llmgw stop",
+			"Every model comes back as its catalog entry asks: placed, loaded and\n"+
+				"serving again, on every control plane within seconds. Run on a control\n"+
+				"plane. The services themselves are never stopped by llmgw stop; a\n"+
+				"machine's own are started with 'llmgw service start'."
+		if stop {
+			use, short, long = "stop", "Stop the whole system serving: every model taken down, requests answered 503",
+				"Every model's replicas are taken down, freeing every GPU, and every\n"+
+					"request for a model is answered 503 (system_stopped) until 'llmgw start'.\n"+
+					"The control planes, agents and mirror keep running, so the console, the\n"+
+					"admin pages and llmgw start still work, and no model's definition is\n"+
+					"changed. In force on every control plane within seconds; kept in the\n"+
+					"cluster's settings (serving.stopped), so a restart does not undo it."
+		}
+		var note string
+		var wait bool
+		c := bind(&cobra.Command{
+			Use:   use,
+			Short: short,
+			Long:  long,
+			Args:  cobra.NoArgs,
+			RunE: func(cmd *cobra.Command, _ []string) error {
+				if note == "" {
+					note = "llmgw " + use
+				}
+				var s adminapi.Summary
+				if err := call().get(cmd.Context(), "/api/v1/summary", &s); err != nil {
+					return notAControlPlane(err)
+				}
+				if s.Stopped == stop {
+					fmt.Fprintf(cmd.OutOrStdout(), "the system is already %s\n", map[bool]string{true: "stopped", false: "serving"}[stop])
+					return nil
+				}
+				if err := changeSettings(cmd, call, note, func(doc *settings.Doc) error {
+					return doc.SetYAML([]string{"serving", "stopped"}, fmt.Sprint(stop))
+				}); err != nil {
+					return err
+				}
+				out := cmd.OutOrStdout()
+				if !wait {
+					if stop {
+						fmt.Fprintln(out, "stopping: every model's replicas are being taken down; llmgw status follows it")
+					} else {
+						fmt.Fprintln(out, "starting: every model is placed again as it asks; llmgw status follows it")
+					}
+					return nil
+				}
+				return waitForService(cmd, call, stop)
+			},
+		})
+		c.Flags().StringVar(&note, "note", "", "why, kept with the settings' version and in the audit")
+		c.Flags().BoolVar(&wait, "wait", false, "wait until every replica is down (stop) or the models have theirs (start)")
+		return c
+	}
+
+	var sshUser string
+	restart := bind(&cobra.Command{
+		Use:   "restart",
+		Short: "Restart every control plane and node agent; the engines keep serving",
+		Long: "Every node's agent starts again, then the other control planes over ssh\n" +
+			"as --ssh-user, then this one — the leader last, so it hands over to a\n" +
+			"standby already restarted. The engines keep serving throughout and are\n" +
+			"taken over again; nothing is placed anew. Run on a control plane, with\n" +
+			"sudo. One machine's services are 'llmgw service restart'.",
+		Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			if os.Geteuid() != 0 {
+				return errors.New("restarting this control plane needs root: run it with sudo")
+			}
+			ctx, out := cmd.Context(), cmd.OutOrStdout()
+			var s adminapi.Summary
+			if err := call().get(ctx, "/api/v1/summary", &s); err != nil {
+				return notAControlPlane(err)
+			}
+			var agents adminapi.AgentRestartView
+			if err := call().post(ctx, "/api/v1/agents/restart", []byte("{}"), &agents); err != nil {
+				return fmt.Errorf("restarting the agents: %w", err)
+			}
+			fmt.Fprintf(out, "agents     %d told to restart: %s\n", len(agents.Nodes), strings.Join(agents.Nodes, ", "))
+
+			// The other control planes, standbys before a leader.
+			var others []adminapi.ControlPlane
+			for _, cp := range s.ControlPlanes.Members {
+				if !cp.This {
+					others = append(others, cp)
+				}
+			}
+			// One that cannot be reached is said and left: the rest of the
+			// system still restarts, and the leader still goes last.
+			var failed []string
+			for pass := 0; pass < 2; pass++ {
+				for _, cp := range others {
+					leading := cp.State == "leading"
+					if leading != (pass == 1) {
+						continue
+					}
+					r := &onboard.SSH{Addr: cp.Address, User: sshUser}
+					if res, err := r.Run(ctx, "sudo systemctl restart llmgw-server", nil); err != nil {
+						fmt.Fprintf(out, "%-10s NOT restarted: ssh as %q: %v %s\n", cp.Name, sshUser, err, strings.TrimSpace(string(res)))
+						failed = append(failed, cp.Name+" ("+cp.Address+")")
+						continue
+					}
+					fmt.Fprintf(out, "%-10s restarted (%s)\n", cp.Name, cp.State)
+					time.Sleep(5 * time.Second)
+				}
+			}
+			if _, err := os.Stat(unitDir + "/llmgw-server.service"); err == nil {
+				fmt.Fprintln(out, "this       restarting")
+				if b, err := exec.CommandContext(ctx, "systemctl", "restart", "llmgw-server").CombinedOutput(); err != nil { //nolint:gosec // fixed command
+					return fmt.Errorf("restarting this control plane: %v: %s", err, strings.TrimSpace(string(b)))
+				}
+				fmt.Fprintln(out, "this       restarted")
+			}
+			fmt.Fprintln(out, "\nllmgw status shows the nodes coming back; their replicas are taken over as they are.")
+			if len(failed) > 0 {
+				return fmt.Errorf("not restarted: %s; run `sudo llmgw service restart server` there, "+
+					"or give --ssh-user someone this machine's root can log in as", strings.Join(failed, ", "))
+			}
+			return nil
+		},
+	})
+	restart.Flags().StringVar(&sshUser, "ssh-user", os.Getenv("SUDO_USER"), "who to log in to the other control planes as")
+
+	return []*cobra.Command{status, switchService(true), switchService(false), restart}
+}
+
+// servicePoll is how often --wait asks; a variable so tests need not wait.
+var servicePoll = 5 * time.Second
+
+// waitForService waits until no replica runs (stopped) or every model has
+// the replicas it asks for (started).
+//
+// Only once the control plane says it is in the state asked for: it takes the
+// setting up at its next look at the store, seconds later, and until then a
+// stopped system's models, which ask for nothing, all have what they ask for —
+// so llmgw start --wait returned at once, with nothing started.
+func waitForService(cmd *cobra.Command, call func() *ctlClient, stop bool) error {
+	deadline := time.Now().Add(15 * time.Minute)
+	for time.Now().Before(deadline) {
+		var s adminapi.Summary
+		if err := call().get(cmd.Context(), "/api/v1/summary", &s); err == nil && s.Stopped == stop {
+			if stop && s.Replicas.Ready+s.Replicas.Starting+s.Replicas.Draining == 0 {
+				fmt.Fprintln(cmd.OutOrStdout(), "stopped: no replica runs")
+				return nil
+			}
+			if !stop && s.Models.Satisfied >= s.Models.Total {
+				fmt.Fprintf(cmd.OutOrStdout(), "serving: %d of %d models have the replicas they ask for\n",
+					s.Models.Satisfied, s.Models.Total)
+				return nil
+			}
+		}
+		select {
+		case <-cmd.Context().Done():
+			return cmd.Context().Err()
+		case <-time.After(servicePoll):
+		}
+	}
+	return errors.New("not there within fifteen minutes; llmgw status says where it stands")
+}
+
+// notAControlPlane is a failure to reach the admin API, said as what it
+// most often means.
+func notAControlPlane(err error) error {
+	return fmt.Errorf("%w\n(the whole system is stopped and started from a control plane; this machine's own "+
+		"services are 'llmgw service status|start|stop|restart')", err)
+}

internal/cli/toolcheck.go

@@ -34,10 +34,11 @@ import (
 // wrapper — left it in the content. A differently trained model fixes that. A
 // bigger one from the same family does not: qwen2.5-coder-7b invented
 // <function-call> and failed the same way.
-func newToolCheckCmd() *cobra.Command {
+func newToolCheckCmd(client func() *ctlClient) *cobra.Command {
 	var (
 		endpoint string
 		model    string
+		key      string
 		timeout  time.Duration
 	)
 
@@ -57,10 +58,21 @@ func newToolCheckCmd() *cobra.Command {
 			if endpoint == "" {
 				return fmt.Errorf("--endpoint is required, e.g. http://10.0.0.145:20002")
 			}
+			// A replica's engine demands its own key, and refused the check
+			// with a 401 when none was sent. Asked of the admin API, as bench
+			// does; an endpoint that is no replica's is asked without one.
+			if key == "" {
+				reps, err := listReplicas(cmd, client)
+				if err != nil {
+					cmd.PrintErrf("The admin API could not be asked for the engine key of the replica at "+
+						"--endpoint (%v); an engine that demands one will refuse. Pass --key to give it.\n", err)
+				}
+				key = replicaKey(reps, endpoint)
+			}
 			ctx, cancel := context.WithTimeout(cmd.Context(), timeout)
 			defer cancel()
 
-			res, err := probeToolCall(ctx, endpoint, model)
+			res, err := probeToolCall(ctx, endpoint, model, key)
 			if err != nil {
 				return err
 			}
@@ -75,6 +87,7 @@ func newToolCheckCmd() *cobra.Command {
 	}
 	cmd.Flags().StringVar(&endpoint, "endpoint", "", "engine base URL, e.g. http://10.0.0.145:20002")
 	cmd.Flags().StringVar(&model, "model", "", "model name to send, when the engine needs one")
+	cmd.Flags().StringVar(&key, "key", "", "the engine's key (default: the admin API's, for the replica at --endpoint)")
 	cmd.Flags().DurationVar(&timeout, "timeout", 5*time.Minute, "how long to wait; CPU inference is slow")
 	return cmd
 }
@@ -102,7 +115,7 @@ type ToolProbe struct {
 	Raw     string
 }
 
-func probeToolCall(ctx context.Context, endpoint, model string) (ToolProbe, error) {
+func probeToolCall(ctx context.Context, endpoint, model, key string) (ToolProbe, error) {
 	// A question with one obvious tool answer and no other sensible reply, so a
 	// model that can use tools has no excuse not to.
 	body := map[string]any{
@@ -138,6 +151,9 @@ func probeToolCall(ctx context.Context, endpoint, model string) (ToolProbe, erro
 		return ToolProbe{}, err
 	}
 	req.Header.Set("Content-Type", "application/json")
+	if key != "" {
+		req.Header.Set("Authorization", "Bearer "+key)
+	}
 
 	resp, err := http.DefaultClient.Do(req)
 	if err != nil {
@@ -150,6 +166,10 @@ func probeToolCall(ctx context.Context, endpoint, model string) (ToolProbe, erro
 		return ToolProbe{}, err
 	}
 	if resp.StatusCode != http.StatusOK {
+		if resp.StatusCode == http.StatusUnauthorized {
+			return ToolProbe{}, fmt.Errorf("%s answered 401: the engine's key is the admin API's, "+
+				"for the replica `llmgw ctl replicas` lists at --endpoint; or pass it with --key", url)
+		}
 		return ToolProbe{}, fmt.Errorf("%s returned %d: %s", url, resp.StatusCode, trim(string(raw), 300))
 	}
 	return readToolProbe(raw)

internal/cli/toolcheck_test.go

@@ -1,10 +1,65 @@
 package cli
 
 import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"net/http/httptest"
 	"strings"
 	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
 )
 
+// toolCheck runs the command against an engine that answers only its own key,
+// with an admin API that lists the replica there with that key.
+func toolCheck(t *testing.T, args ...string) (string, error) {
+	t.Helper()
+	engine := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if r.Header.Get("Authorization") != "Bearer engine-key" {
+			http.Error(w, `{"error":{"message":"Invalid API Key"}}`, http.StatusUnauthorized)
+			return
+		}
+		fmt.Fprint(w, `{"choices":[{"message":{"tool_calls":[{"id":"c1","type":"function",
+		  "function":{"name":"get_weather","arguments":"{\"city\":\"Istanbul\"}"}}]}}]}`)
+	}))
+	admin := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		_ = json.NewEncoder(w).Encode([]adminapi.ReplicaView{
+			{ID: "r1", Model: "m", Profile: "gpu", Endpoint: engine.URL, EngineKey: "engine-key"},
+		})
+	}))
+	t.Cleanup(func() { engine.Close(); admin.Close() })
+	t.Setenv("LLMGW_ADMIN_TOKEN", "t")
+	cmd := newToolCheckCmd(func() *ctlClient { return newCtlClient(admin.URL) })
+	var out bytes.Buffer
+	cmd.SetOut(&out)
+	cmd.SetErr(&out)
+	cmd.SetArgs(append([]string{"--endpoint", engine.URL}, args...))
+	err := cmd.ExecuteContext(context.Background())
+	return out.String(), err
+}
+
+func TestToolCheckAsksWithTheEngineKeyTheAdminAPIGives(t *testing.T) {
+	// Every replica's engine demands its key, and the check was sent without
+	// one: it failed with a 401 on every engine in the cluster.
+	out, err := toolCheck(t)
+	if err != nil {
+		t.Fatalf("%v\n%s", err, out)
+	}
+	if !strings.Contains(out, "tool calls work") {
+		t.Errorf("unexpected report:\n%s", out)
+	}
+}
+
+func TestToolCheckSaysWhereTheEngineKeyComesFrom(t *testing.T) {
+	out, err := toolCheck(t, "--key", "wrong")
+	if err == nil || !strings.Contains(err.Error(), "401") || !strings.Contains(err.Error(), "--key") {
+		t.Fatalf("expected the refusal explained, got %v\n%s", err, out)
+	}
+}
+
 func TestAStructuredToolCallIsRecognised(t *testing.T) {
 	p, err := readToolProbe([]byte(`{"choices":[{"message":{"role":"assistant",
 	  "tool_calls":[{"id":"c1","type":"function","function":{"name":"get_weather","arguments":"{\"city\":\"Istanbul\"}"}}]}}]}`))

internal/cli/tui_actions.go

@@ -679,6 +679,8 @@ func (m *consoleModel) doEntry() tea.Cmd {
 			}}
 	case cBackup:
 		return m.runAfter("server", "backup", "--config="+m.file)
+	case cJudge:
+		return m.runAfter("ctl", server, "guard", "judge", "enable", "--replace")
 	}
 	return nil
 }

internal/cli/tui_console.go

@@ -4,6 +4,7 @@ import (
 	"bufio"
 	"bytes"
 	"context"
+	"encoding/json"
 	"fmt"
 	"io"
 	"net"
@@ -58,16 +59,18 @@ const (
 	cAddControl
 	cReplicate
 	cServe
+	cJudge
 	cUpgrade
 	cRestartAll
 	cBackup
+	cWipe
 	cLeave
 	cCount
 )
 
 var consoleLabels = [cCount]string{"Overview", "Monitor", "Nodes", "GPUs", "Models", "Replicas", "Usage", "Quotas", "Guardrails", "Federation", "Mirror", "Jobs", "Logs", "Accounts", "API keys", "Organizations", "Audit", "Manual",
-	"server.yaml", "Add a node", "Add a control plane", "Replicate the store", "Serve a model", "Upgrade the cluster",
-	"Restart all replicas", "Back up now", "Leave"}
+	"server.yaml", "Add a node", "Add a control plane", "Replicate the store", "Serve a model", "Serve the guardrails' judge", "Upgrade the cluster",
+	"Restart all replicas", "Back up now", "Wipe this machine", "Leave"}
 
 // consoleAction is what is done once the screen closes: each runs its own
 // screen, and the console comes back after.
@@ -138,15 +141,18 @@ type consoleModel struct {
 	node bool
 
 	// The settings.
-	file     string
-	doc      *settings.Doc
-	cfg      any // the file as loaded, for the values that apply
-	levels   []settingsLevel
-	docErr   error
-	saveNote string
+	file string
+	doc  *settings.Doc
+	cfg  any // the file as loaded, for the values that apply
+	// settingsBase is the version of the cluster's settings the screen
+	// read, when the cluster keeps them (doc is then layered over them).
+	settingsBase int64
+	levels       []settingsLevel
+	docErr       error
+	saveNote     string
 
 	// A question open as a dialog.
-	dialog      string // "", "target", "edit", "addname", "saved", "confirm-leave", "error"
+	dialog      string // "", "target", "edit", "addname", "wipe", "saved", "note", "confirm-leave", "error"
 	dialogTitle string
 	dialogText  string
 	prompt      textField
@@ -219,6 +225,26 @@ func (m *consoleModel) loadSettings() {
 	} else if !m.node {
 		m.cfg = config.Default()
 	}
+	// The cluster's keys from the cluster's settings, when the control
+	// plane answers with them; its own keys from its file.
+	if !m.node && m.client != nil {
+		ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
+		defer cancel()
+		var v adminapi.SettingsView
+		if err := m.client.get(ctx, "/api/v1/settings", &v); err == nil && v.Version > 0 {
+			if cdoc, err := settings.FromBytes("the cluster's settings", []byte(v.Body)); err == nil {
+				m.doc.Layer(cdoc, config.IsLocal)
+				m.settingsBase = v.Version
+				if file, err := config.ReadFile(m.file); err == nil {
+					if cluster, err := config.Parse([]byte(v.Body)); err == nil {
+						if cfg, err := config.FromMap(config.Layer(file, cluster)); err == nil {
+							m.cfg = cfg
+						}
+					}
+				}
+			}
+		}
+	}
 	if root, err := settings.Root(m.file); err == nil && len(m.levels) == 0 {
 		m.levels = []settingsLevel{{title: filepathBase(m.file), block: root, kind: settings.Block}}
 	}
@@ -383,7 +409,7 @@ func (m consoleModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
 
 func (m consoleModel) entryOK(i int) bool {
 	if m.node {
-		return i == cSettings || i == cManual || i == cLeave
+		return i == cSettings || i == cManual || i == cWipe || i == cLeave
 	}
 	return i != cReplicate || !m.storeReplicated()
 }
@@ -469,7 +495,9 @@ func (m consoleModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) {
 				return m.ask("Replicate the store", "user@address of database machine", 2)
 			case cLeave:
 				return m.leave()
-			case cUpgrade, cRestartAll, cBackup:
+			case cWipe:
+				return m.askWipe()
+			case cUpgrade, cRestartAll, cBackup, cJudge:
 				return m, m.doEntry()
 			case cOverview:
 				return m, nil
@@ -643,10 +671,10 @@ func (m consoleModel) peopleView() bool {
 }
 
 func (m consoleModel) leave() (tea.Model, tea.Cmd) {
-	if m.doc != nil && m.doc.Changed > 0 {
+	if m.doc != nil && m.doc.Pending() > 0 {
 		m.dialog, m.dialogTitle = "confirm-leave", "Unsaved changes"
-		m.dialogText = fmt.Sprintf("%d change(s) to %s are not saved.\nLeave without saving them?\n\n%s",
-			m.doc.Changed, m.file, sDim.Render("Y leave · N stay"))
+		m.dialogText = fmt.Sprintf("%d change(s) are not saved.\nLeave without saving them?\n\n%s",
+			m.doc.Pending(), sDim.Render("Y leave · N stay"))
 		return m, nil
 	}
 	return m, tea.Quit
@@ -663,7 +691,7 @@ func (m consoleModel) ask(title, question string, n int) (tea.Model, tea.Cmd) {
 
 func (m consoleModel) dialogKey(k tea.KeyMsg) (tea.Model, tea.Cmd) {
 	switch m.dialog {
-	case "error":
+	case "error", "note":
 		m.dialog = ""
 		return m, nil
 	case "confirm-leave":
@@ -720,6 +748,13 @@ func (m consoleModel) dialogKey(k tea.KeyMsg) (tea.Model, tea.Cmd) {
 				m.dialogText = floatingQuestion
 				m.prompt.Placeholder = "10.0.0.161/24"
 			}
+		case "wipe":
+			if host, _ := os.Hostname(); v != host {
+				m.dialogText = m.wipeQuestion() + "\n\n" + sBad.Render("that is not this machine's name; nothing is removed")
+				return m, nil
+			}
+			m.action = consoleAction{kind: "wipe"}
+			return m, tea.Quit
 		case "edit":
 			if err := m.doc.Set(m.editPath, m.editKind, m.prompt.Value()); err != nil {
 				m.dialogText = m.editQuestion() + "\n\n" + sBad.Render(err.Error())
@@ -917,28 +952,60 @@ func (m consoleModel) settingsKey(k tea.KeyMsg) (tea.Model, tea.Cmd) {
 }
 
 func (m consoleModel) save() (tea.Model, tea.Cmd) {
-	if m.doc.Changed == 0 {
+	if m.doc.Pending() == 0 {
 		m.dialog, m.dialogTitle, m.dialogText = "error", "Nothing to save", "No change has been made."
 		return m, nil
 	}
-	backup, err := m.doc.Save(func(p string) error { _, err := m.load(p); return err })
-	if err != nil {
+	notSaved := func(err error) (tea.Model, tea.Cmd) {
 		m.dialog, m.dialogTitle = "error", "Not saved"
-		m.dialogText = "The configuration would not load:\n\n" + sBad.Render(err.Error()) + "\n\n" + sDim.Render("Nothing was written. Any key to go back.")
+		m.dialogText = "The configuration would not load:\n\n" + sBad.Render(err.Error()) + "\n\n" +
+			sDim.Render("Nothing was written. Any key to go back.")
 		return m, nil
 	}
+	var said []string
+	// The cluster's settings first: checked by the control plane against
+	// its own file, and in force everywhere once kept.
+	if c := m.doc.Cluster(); c != nil && c.Changed > 0 {
+		body, err := c.Bytes()
+		if err != nil {
+			return notSaved(err)
+		}
+		req, _ := json.Marshal(adminapi.SettingsRequest{Base: m.settingsBase, Body: string(body), Note: "console"})
+		ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+		defer cancel()
+		var v adminapi.SettingsView
+		if err := m.client.put(ctx, "/api/v1/settings", req, &v); err != nil {
+			return notSaved(err)
+		}
+		c.Changed = 0
+		said = append(said, fmt.Sprintf("The cluster's settings are version %d: in force on every control plane "+
+			"within seconds.\nllmgw ctl config history lists the versions; rollback goes back to one.", v.Version))
+	}
+	fileSaved := m.doc.Changed > 0
+	if fileSaved {
+		backup, err := m.doc.Save(func(p string) error { _, err := m.load(p); return err })
+		if err != nil {
+			return notSaved(err)
+		}
+		line := "Saved " + m.file + "."
+		if backup != "" {
+			line += " The old file is " + backup + "."
+		}
+		said = append(said, line)
+	}
 	m.loadSettings()
-	note := ""
-	if backup != "" {
-		note = "The old file is " + backup + ".\n"
+	if !m.node {
+		if fileSaved {
+			said = append(said, "llmgw-server notices the change and applies it within seconds.")
+		}
+		m.dialog, m.dialogTitle = "note", "Saved"
+		m.dialogText = strings.Join(said, "\n\n") + "\n\n" + sDim.Render("Any key to go back.")
+		return m, nil
 	}
 	m.dialog, m.dialogTitle = "saved", "Saved"
-	m.dialogText = "Saved " + m.file + ".\n" + note +
-		fmt.Sprintf("\n%s reads it when it starts. Restart it now?\n", m.service())
-	if !m.node {
-		m.dialogText += sDim.Render("Another control plane keeps its own copy: make the change there too.") + "\n"
-	}
-	m.dialogText += "\n" + sDim.Render("Y restart · N later")
+	m.dialogText = strings.Join(said, "\n") + "\n" +
+		fmt.Sprintf("\n%s reads it when it starts. Restart it now?\n", m.service()) +
+		"\n" + sDim.Render("Y restart · N later")
 	return m, nil
 }
 
@@ -999,7 +1066,7 @@ func (m consoleModel) View() string {
 			nav = append(nav, navItem{Label: "", Heading: true}, navItem{Label: "Help", Heading: true})
 		case cSettings:
 			nav = append(nav, navItem{Label: "", Heading: true}, navItem{Label: "Settings", Heading: true})
-		case cAddNode, cLeave:
+		case cAddNode, cWipe:
 			if (i == cAddNode) != m.node {
 				nav = append(nav, navItem{Label: "", Heading: true}, navItem{Label: "Do", Heading: true})
 			}
@@ -1011,8 +1078,8 @@ func (m consoleModel) View() string {
 		if n := runningJobs(m.data.jobs); i == cJobs && n > 0 {
 			label += fmt.Sprintf(" (%d)", n)
 		}
-		if i == cSettings && m.doc != nil && m.doc.Changed > 0 {
-			label += fmt.Sprintf(" (%d)", m.doc.Changed)
+		if i == cSettings && m.doc != nil && m.doc.Pending() > 0 {
+			label += fmt.Sprintf(" (%d)", m.doc.Pending())
 		}
 		nav = append(nav, navItem{Label: label, Current: i == m.menu, Dim: !m.entryOK(i)})
 	}
@@ -1040,7 +1107,7 @@ func (m consoleModel) View() string {
 		f.DialogTitle = m.dialogTitle
 		f.Dialog = m.dialogText
 		switch m.dialog {
-		case "target", "edit", "addname":
+		case "target", "edit", "addname", "wipe":
 			q := m.dialogText
 			if m.dialog == "target" && m.editKind > 1 {
 				q = fmt.Sprintf("%s %d of %d", q, len(m.asked)+1, int(m.editKind))
@@ -1163,13 +1230,17 @@ func (m consoleModel) panel() (title, body, keys string) {
 			for _, s := range r.Shards {
 				nodes = append(nodes, fmt.Sprintf("%s%v", s.Node, discovery.DeviceLabels(s.GPUs)))
 			}
-			rows = append(rows, []string{r.ID, r.Model, r.State, r.Pool, strings.Join(nodes, " "),
+			compute := "-"
+			if len(r.Shards) > 0 {
+				compute = r.Shards[0].Compute.Short()
+			}
+			rows = append(rows, []string{r.ID, r.Model, r.State, r.Pool, compute, strings.Join(nodes, " "),
 				(time.Duration(r.AgeSec) * time.Second).Round(time.Second).String()})
 		}
 		if m.inPanel {
 			keys = rowKeys
 		}
-		return "Replicas", "\n" + table([]string{"REPLICA", "MODEL", "STATE", "POOL", "ON", "AGE"}, rows, m.row, pw, m.inPanel), keys
+		return "Replicas", "\n" + table([]string{"REPLICA", "MODEL", "STATE", "POOL", "COMPUTE", "ON", "AGE"}, rows, m.row, pw, m.inPanel), keys
 	case cUsage:
 		head := "\n  Last 24 hours, by " + lipgloss.NewStyle().Bold(true).Render(usageGroupings[m.usageBy]) +
 			sDim.Render("   (→ to group by "+usageGroupings[(m.usageBy+1)%len(usageGroupings)]+")") + "\n\n"
@@ -1233,6 +1304,9 @@ func (m consoleModel) panel() (title, body, keys string) {
 		item = "\n  Moves the store onto two machines under Patroni, replicated,\n  so losing one loses nothing.\n\n  " +
 			sDim.Render("On the command line:") +
 			"\n  sudo --preserve-env=SSH_AUTH_SOCK llmgw server init \\\n    --patroni <user>@<a> --patroni <user>@<b> --no-mirror --no-node --no-model"
+	case cJudge:
+		item = "\n  Serves the model the guardrail policies ask as their judge — Llama\n  Guard 3 1B on CPUs, under the name the policies give it — so what they\n  judge is not refused (guard_unavailable). A model of that name that is\n  not a judge is replaced. Overview says when a judge is missing.\n\n  " +
+			sDim.Render("On the command line:") + "\n  llmgw ctl guard judge status\n  llmgw ctl guard judge enable --replace"
 	case cUpgrade:
 		item = "\n  Moves the cluster to a release: this machine first, then each node\n  in turn. Engines keep serving; a node that does not come back stops it.\n\n  " +
 			sDim.Render("On the command line:") + "\n  sudo llmgw ctl upgrade VERSION"
@@ -1242,6 +1316,9 @@ func (m consoleModel) panel() (title, body, keys string) {
 	case cBackup:
 		item = "\n  Writes the store and /etc/llmgw to one archive in " + DefaultBackupDir + ",\n  and keeps the last few. Copy it off this machine.\n\n  " +
 			sDim.Render("On the command line:") + "\n  sudo llmgw server backup"
+	case cWipe:
+		item = "\n  Removes llmgw from this machine: its services, firewall table,\n  configuration, certificates, the authority, secrets, state and users,\n  so it can be set up again from nothing. Model weights, engines, the\n  store's database and backups stay. Asks for the machine's name first.\n\n  " +
+			sDim.Render("On the command line, with --models, --store or --all for more:") + "\n  sudo llmgw wipe"
 	case cLeave:
 		item = "\n  Closes the console. The cluster keeps running."
 	}
@@ -1565,6 +1642,16 @@ func runConsole(ctx context.Context, out io.Writer, file string) error {
 			// Already set up: only the store changes.
 			in.noMirror, in.noNode, in.noModel = true, true, true
 			err = in.runInitTUI(ctx, out)
+		case "wipe":
+			c := Root()
+			c.SetArgs([]string{"wipe", "--yes"})
+			c.SetOut(out)
+			c.SetErr(out)
+			if err := c.ExecuteContext(ctx); err != nil {
+				fmt.Fprintln(out, sBad.Render("✗ ")+err.Error())
+			}
+			// Nothing is left here for the console to show.
+			return nil
 		case "run":
 			c := Root()
 			c.SetArgs(act.targets)
@@ -1610,3 +1697,31 @@ func (m consoleModel) standbyNote() string {
 	}
 	return "\n  " + sDim.Render("Standing by. Shown, and changed, on the leader: "+leader) + "\n"
 }
+
+// askWipe asks for this machine's name before removing llmgw from it, saying
+// what would go.
+func (m consoleModel) askWipe() (tea.Model, tea.Cmd) {
+	if os.Geteuid() != 0 {
+		m.dialog, m.dialogTitle = "error", "Wipe this machine"
+		m.dialogText = "Removing services, users and files under /etc needs root:\nstart the console with sudo llmgw."
+		return m, nil
+	}
+	m.dialog, m.dialogTitle, m.dialogText = "wipe", "Wipe this machine", m.wipeQuestion()
+	m.prompt = newTextField()
+	m.prompt.Width = 40
+	m.prompt.Focus()
+	return m, nil
+}
+
+func (m consoleModel) wipeQuestion() string {
+	host, _ := os.Hostname()
+	w := &wiper{}
+	var b strings.Builder
+	b.WriteString("This removes, on " + host + ":\n\n")
+	for _, s := range w.plan() {
+		b.WriteString("  - " + s.what + "\n")
+	}
+	b.WriteString("\nModel weights, engines, the store's database and backups stay\n" +
+		"(sudo llmgw wipe --all takes them too).\n\nType the machine's name to remove it all:")
+	return b.String()
+}

internal/cli/tui_hf.go

@@ -13,7 +13,6 @@ import (
 
 	tea "github.com/charmbracelet/bubbletea"
 
-	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/modelplan"
@@ -57,7 +56,7 @@ var hfClients sync.Map
 
 func (m consoleModel) registry() *hf.Client {
 	endpoint := ""
-	if cfg, err := config.Load(m.file); err == nil {
+	if cfg, err := loadEffectiveConfig(m.file); err == nil {
 		endpoint = cfg.Mirror.Registry
 	}
 	c, _ := hfClients.LoadOrStore(endpoint, hf.New(endpoint, 20*time.Second))

internal/cli/tui_people.go

@@ -12,7 +12,6 @@ import (
 
 	tea "github.com/charmbracelet/bubbletea"
 
-	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
@@ -118,7 +117,7 @@ func (m consoleModel) fetchPeople() tea.Cmd {
 				}
 			}
 		}
-		if cfg, err := config.Load(file); err == nil {
+		if cfg, err := loadEffectiveConfig(file); err == nil {
 			for name := range cfg.Guard.Policies {
 				d.guardPolicies = append(d.guardPolicies, name)
 			}

internal/cli/tui_setup_test.go

@@ -1,6 +1,9 @@
 package cli
 
 import (
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
 	"os"
 	"path/filepath"
 	"slices"
@@ -164,7 +167,7 @@ func TestTheConsoleWatchesTheClusterAndEditsItsSettings(t *testing.T) {
 	cm = mm.(consoleModel)
 	cm.prompt.SetValue("5")
 	mm = press(cm, "enter", "s")
-	if d := mm.(consoleModel).dialog; d != "saved" {
+	if d := mm.(consoleModel).dialog; d != "note" {
 		t.Fatalf("after saving the dialog is %q: %s", d, mm.(consoleModel).dialogText)
 	}
 	b, _ := os.ReadFile(file)
@@ -244,3 +247,80 @@ func TestTheConsoleShowsTheGuardrails(t *testing.T) {
 		t.Errorf("window %q", guardWindows[c.guard.window])
 	}
 }
+
+func TestTheConsoleEditsTheClustersSettingsInTheStore(t *testing.T) {
+	dir := t.TempDir()
+	file := filepath.Join(dir, "server.yaml")
+	if err := os.WriteFile(file, []byte("listen: \":8080\"\n"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	var put adminapi.SettingsRequest
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		switch {
+		case r.Method == "GET" && r.URL.Path == "/api/v1/settings":
+			_ = json.NewEncoder(w).Encode(adminapi.SettingsView{Version: 7, Body: "router:\n  retries: 2\n"})
+		case r.Method == "PUT" && r.URL.Path == "/api/v1/settings":
+			_ = json.NewDecoder(r.Body).Decode(&put)
+			_ = json.NewEncoder(w).Encode(adminapi.SettingsView{Version: 8, Body: put.Body})
+		default:
+			http.NotFound(w, r)
+		}
+	}))
+	defer srv.Close()
+	m := newConsoleModel(strings.TrimPrefix(srv.URL, "http://"), file)
+	m.width, m.height, m.loaded = 120, 40, true
+	m.menu, m.inPanel = cSettings, true
+	m.loadSettings()
+	for i, r := range m.settingsRows() {
+		if r.label == "router" {
+			m.row = i
+		}
+	}
+	var mm tea.Model = press(m, "enter")
+	cm := mm.(consoleModel)
+	for i, r := range cm.settingsRows() {
+		if r.label == "retries" {
+			cm.row = i
+		}
+	}
+	if v := cm.View(); !strings.Contains(v, "2") {
+		t.Errorf("the cluster's value is not shown:\n%s", v)
+	}
+	mm = press(cm, "enter")
+	cm = mm.(consoleModel)
+	cm.prompt.SetValue("5")
+	mm = press(cm, "enter", "s")
+	if d := mm.(consoleModel).dialog; d != "note" || !strings.Contains(mm.(consoleModel).dialogText, "version 8") {
+		t.Fatalf("after saving: %q %s", d, mm.(consoleModel).dialogText)
+	}
+	if put.Base != 7 || !strings.Contains(put.Body, "retries: 5") {
+		t.Errorf("sent %+v", put)
+	}
+	if b, _ := os.ReadFile(file); string(b) != "listen: \":8080\"\n" {
+		t.Errorf("the cluster's key went into the machine's file:\n%s", b)
+	}
+}
+
+func TestTheConsoleWipesOnlyOnTheMachinesName(t *testing.T) {
+	m := newConsoleModel("127.0.0.1:1", filepath.Join(t.TempDir(), "server.yaml"))
+	m.width, m.height, m.loaded = 120, 40, true
+	m.menu = cWipe
+	m.dialog, m.dialogTitle, m.dialogText = "wipe", "Wipe this machine", m.wipeQuestion()
+	m.prompt = newTextField()
+	m.prompt.Focus()
+	if v := m.View(); !strings.Contains(v, "Type the machine's name") {
+		t.Errorf("the dialog does not ask for the name:\n%s", v)
+	}
+	m.prompt.SetValue("not-this-one")
+	mm := press(m, "enter")
+	if a := mm.(consoleModel).action; a.kind != "" {
+		t.Fatalf("a wrong name wiped: %+v", a)
+	}
+	cm := mm.(consoleModel)
+	host, _ := os.Hostname()
+	cm.prompt.SetValue(host)
+	mm = press(cm, "enter")
+	if a := mm.(consoleModel).action; a.kind != "wipe" {
+		t.Errorf("the right name did not: %+v", a)
+	}
+}

internal/cli/users.go

@@ -3,6 +3,7 @@ package cli
 import (
 	"errors"
 	"fmt"
+	"net/url"
 	"strings"
 	"text/tabwriter"
 	"time"
@@ -34,7 +35,7 @@ func newUsersCmd() *cobra.Command {
 			"Stop an account working, keeping it and its history"),
 		newUserStateCmd(open, "reinstate", store.UserActive,
 			"Let a suspended account work again"),
-		newUserCreateCmd(open), newUserResetPasswordCmd(open),
+		newUserCreateCmd(open), newUserResetPasswordCmd(open), newUserResetLinkCmd(open, &sf),
 		newUserJoinCmd(open), newUserLeaveCmd(open), newUserDeleteCmd(open), newUserForgetCmd(open),
 		newUserAdminCmd(open, "promote", true, "Let an account see the cluster dashboard at /admin"),
 		newUserAdminCmd(open, "demote", false, "Take away access to the cluster dashboard"),
@@ -254,9 +255,10 @@ func newUserResetPasswordCmd(open userOpener) *cobra.Command {
 	cmd := &cobra.Command{
 		Use:   "reset-password <user>",
 		Short: "Set a new password for someone who lost theirs",
-		Long: "There is no self-service reset, because this gateway sends no email and\n" +
-			"a reset link with nowhere to send it is not a reset. Set one here, tell\n" +
-			"them out of band, and have them change it from their profile.\n\n" +
+		Long: "People reset a forgotten password themselves at /forgot when the cluster\n" +
+			"has mail settings (mail.smtp); without them, 'llmgw users reset-link' makes\n" +
+			"a link to hand them. Or set one here, tell them out of band, and have them\n" +
+			"change it from their profile.\n\n" +
 			"Every session they have open is closed: if they lost the password, the\n" +
 			"reason might be that somebody else has it.",
 		Args: cobra.ExactArgs(1),
@@ -532,3 +534,50 @@ func newUserForgetCmd(open userOpener) *cobra.Command {
 	cmd.Flags().BoolVar(&yes, "yes", false, "confirm")
 	return cmd
 }
+
+func newUserResetLinkCmd(open userOpener, sf *storeFlags) *cobra.Command {
+	var base string
+	cmd := &cobra.Command{
+		Use:   "reset-link <user>",
+		Short: "Make a one-time link for someone to set a forgotten password",
+		Long: "For a gateway that sends no mail, or a person whose mail does not arrive:\n" +
+			"the link is printed, to hand them however suits. It works once, for an\n" +
+			"hour, and a newer one replaces it. Setting the password closes every\n" +
+			"session they have open. The address it points at is mail.base_url from\n" +
+			"the cluster's settings, or --base-url.",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			if base == "" {
+				if cfg, err := loadEffectiveConfig(sf.config); err == nil {
+					base = cfg.Mail.BaseURL
+				}
+			}
+			if base == "" {
+				return errors.New("give --base-url, the address people reach this gateway at, or set mail.base_url " +
+					"(llmgw ctl config set mail.base_url https://…)")
+			}
+			s, err := open(cmd)
+			if err != nil {
+				return err
+			}
+			defer s.Close()
+			u, err := resolveUser(cmd.Context(), s, args[0])
+			if err != nil {
+				return err
+			}
+			if u.PasswordHash == "" {
+				return fmt.Errorf("%s signs in through single sign-on: there is no password to set", u.Email)
+			}
+			token, err := s.CreatePasswordReset(cmd.Context(), u.ID, account.ResetLinkTTL)
+			if err != nil {
+				return err
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "%s/reset?token=%s\n", strings.TrimRight(base, "/"), url.QueryEscape(token))
+			cmd.PrintErrf("For %s: works once, for %s. Hand it to them alone; it opens their account.\n",
+				u.Email, account.ResetLinkTTL)
+			return nil
+		},
+	}
+	cmd.Flags().StringVar(&base, "base-url", "", "the address people reach this gateway at (default: mail.base_url)")
+	return cmd
+}

internal/cli/wipe.go

@@ -0,0 +1,277 @@
+package cli
+
+import (
+	"bufio"
+	"errors"
+	"fmt"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"runtime"
+	"strings"
+
+	"github.com/spf13/cobra"
+)
+
+// wipeUnits are the systemd units llmgw installs, stopped first so nothing
+// writes while the rest goes.
+var wipeUnits = []string{
+	"llmgw-server.service", "llmgw-agent.service", "llmgw-mirror.service",
+	firewallSyncPathUnit, "llmgw-firewall-sync.service", "llmgw-firewall.service",
+	"llmgw-backup.timer", "llmgw-backup.service",
+}
+
+// wiper removes llmgw from a machine. root prefixes every path, for tests;
+// run runs a command.
+type wiper struct {
+	root   string
+	run    func(name string, args ...string) error
+	store  bool
+	models bool
+	all    bool
+}
+
+// wipeStep is one thing to remove, said before it is done.
+type wipeStep struct {
+	what string
+	do   func() error
+}
+
+func (w *wiper) path(p string) string { return filepath.Join(w.root, p) }
+
+func (w *wiper) exists(p string) bool {
+	_, err := os.Lstat(w.path(p))
+	return err == nil
+}
+
+func (w *wiper) glob(pattern string) []string {
+	m, _ := filepath.Glob(w.path(pattern))
+	out := make([]string, len(m))
+	for i, p := range m {
+		out[i] = strings.TrimPrefix(p, w.root)
+	}
+	return out
+}
+
+// mentions is whether a file is there and says llmgw: a file of a name
+// another tool also uses (keepalived's, etcd's) is removed only when llmgw
+// wrote it.
+func (w *wiper) mentions(p string) bool {
+	b, err := os.ReadFile(w.path(p))
+	return err == nil && strings.Contains(string(b), "llmgw")
+}
+
+func (w *wiper) remove(paths ...string) func() error {
+	return func() error {
+		var errs []error
+		for _, p := range paths {
+			if err := os.RemoveAll(w.path(p)); err != nil {
+				errs = append(errs, err)
+			}
+		}
+		return errors.Join(errs...)
+	}
+}
+
+// plan is every step, in order, for what is on this machine.
+func (w *wiper) plan() []wipeStep {
+	var steps []wipeStep
+	add := func(what string, do func() error) { steps = append(steps, wipeStep{what, do}) }
+	files := func(what string, paths ...string) {
+		var there []string
+		for _, p := range paths {
+			if w.exists(p) {
+				there = append(there, p)
+			}
+		}
+		if len(there) > 0 {
+			add(what+": "+strings.Join(there, ", "), w.remove(there...))
+		}
+	}
+
+	// The services, stopped and gone.
+	var units []string
+	for _, u := range wipeUnits {
+		if w.exists("/etc/systemd/system/" + u) {
+			units = append(units, u)
+		}
+	}
+	if len(units) > 0 {
+		add("stop and remove "+strings.Join(units, ", "), func() error {
+			for _, u := range units {
+				_ = w.run("systemctl", "disable", "--now", u)
+				if err := os.Remove(w.path("/etc/systemd/system/" + u)); err != nil && !os.IsNotExist(err) {
+					return err
+				}
+			}
+			return w.run("systemctl", "daemon-reload")
+		})
+	}
+	// Engines run as their own user, kept alive across the agent's restarts:
+	// stopped by user, or they would keep their GPUs.
+	if w.userExists("llmgw-engine") {
+		add("stop the engines (every process of llmgw-engine)", func() error {
+			_ = w.run("pkill", "-KILL", "-u", "llmgw-engine")
+			return nil
+		})
+	}
+	add("take llmgw's table out of the firewall (nft inet llmgw), if loaded", func() error {
+		_ = w.run("nft", "delete", "table", "inet", "llmgw")
+		return nil
+	})
+	if w.mentions("/etc/keepalived/keepalived.conf") {
+		add("stop keepalived and remove llmgw's floating address (/etc/keepalived/keepalived.conf)", func() error {
+			_ = w.run("systemctl", "disable", "--now", "keepalived")
+			return w.remove("/etc/keepalived/keepalived.conf")()
+		})
+	}
+
+	files("configuration, certificates, the authority and secrets", "/etc/llmgw")
+	files("state: the admin token, a SQLite store, what the agent kept",
+		"/var/lib/llmgw", "/var/lib/llmgw-agent")
+	files("the sudo grant setup left", "/etc/sudoers.d/llmgw-setup")
+	if w.exists("/usr/local/share/ca-certificates/llmgw-ca.crt") {
+		add("the cluster's authority from the system's trust (llmgw-ca.crt)", func() error {
+			if err := w.remove("/usr/local/share/ca-certificates/llmgw-ca.crt")(); err != nil {
+				return err
+			}
+			_ = w.run("update-ca-certificates", "--fresh")
+			return nil
+		})
+	}
+	if w.exists("/etc/pki/ca-trust/source/anchors/llmgw-ca.pem") {
+		add("the cluster's authority from the system's trust (llmgw-ca.pem)", func() error {
+			if err := w.remove("/etc/pki/ca-trust/source/anchors/llmgw-ca.pem")(); err != nil {
+				return err
+			}
+			_ = w.run("update-ca-trust")
+			return nil
+		})
+	}
+
+	if w.store || w.all {
+		w.planStore(add, files)
+	}
+	if w.models || w.all {
+		files("model weights and the mirror", "/srv/models", "/var/lib/llmgw-weights", "/var/lib/llmgw-models")
+		files("installed engines", "/opt/llmgw")
+	}
+	if w.all {
+		files("backups", DefaultBackupDir)
+	}
+
+	for _, u := range []string{"llmgw-engine", "llmgw"} {
+		if w.userExists(u) {
+			add("the user "+u, func() error { return w.run("userdel", u) })
+		}
+	}
+	if w.all {
+		files("the llmgw binary", "/usr/local/bin/llmgw", "/usr/local/bin/llmgw.prev")
+	}
+	return steps
+}
+
+// planStore is the store, when it is on this machine: a member of the
+// replicated one (Patroni, etcd), or llmgw's database in a local PostgreSQL.
+func (w *wiper) planStore(add func(string, func() error), files func(string, ...string)) {
+	patroni := w.mentions("/etc/patroni/config.yml")
+	if patroni || w.mentions("/etc/default/etcd") {
+		add("stop Patroni and etcd", func() error {
+			_ = w.run("systemctl", "disable", "--now", "patroni")
+			_ = w.run("systemctl", "disable", "--now", "etcd")
+			return nil
+		})
+	}
+	if patroni {
+		data := append(w.glob("/var/lib/postgresql/*/llmgw"), w.glob("/var/lib/pgsql/*/llmgw")...)
+		files("the store's database on this member, and Patroni's configuration",
+			append([]string{"/etc/patroni", "/var/lib/postgresql/.pgpass_patroni", "/root/llmgw-pg"}, data...)...)
+	} else if w.exists("/usr/bin/psql") {
+		add("drop the llmgw database and role from the local PostgreSQL, if there", func() error {
+			_ = w.run("runuser", "-u", "postgres", "--", "psql", "-q",
+				"-c", "DROP DATABASE IF EXISTS llmgw", "-c", "DROP ROLE IF EXISTS llmgw")
+			return nil
+		})
+	}
+	if w.mentions("/etc/default/etcd") {
+		files("etcd's data and configuration", "/var/lib/etcd/default", "/etc/etcd/tls", "/etc/default/etcd")
+	}
+}
+
+func (w *wiper) userExists(name string) bool {
+	b, err := os.ReadFile(w.path("/etc/passwd"))
+	return err == nil && strings.Contains("\n"+string(b), "\n"+name+":")
+}
+
+func newWipeCmd() *cobra.Command {
+	var w wiper
+	var yes bool
+	c := &cobra.Command{
+		Use:   "wipe",
+		Short: "Remove llmgw from this machine: its services, configuration and state",
+		Long: "Stops and removes llmgw's services, takes its firewall table out, and\n" +
+			"deletes its configuration, certificates, the authority, secrets, state\n" +
+			"and users — what server init, node add, server add and mirror init left —\n" +
+			"so the machine can be set up again from nothing. It says everything it\n" +
+			"will remove first, and asks for the machine's name.\n\n" +
+			"Kept unless asked: model weights, the mirror and engines (--models), the\n" +
+			"store's database when it is on this machine (--store), backups and the\n" +
+			"binary (--all, which is all of these). Packages llmgw installed —\n" +
+			"PostgreSQL, Patroni, etcd, nftables, the node exporter — stay.\n\n" +
+			"A node wiped is then forgotten on the control plane with\n" +
+			"'llmgw node remove <name>'.",
+		Example: "  sudo llmgw wipe\n" +
+			"  sudo llmgw wipe --models\n" +
+			"  sudo llmgw wipe --all --yes",
+		Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			if runtime.GOOS != "linux" {
+				return fmt.Errorf("wipe removes llmgw from a Linux machine; this is %s", runtime.GOOS)
+			}
+			if os.Geteuid() != 0 {
+				return errors.New("wipe removes services, users and files under /etc: run it with sudo")
+			}
+			out := cmd.OutOrStdout()
+			w.run = func(name string, args ...string) error {
+				return exec.CommandContext(cmd.Context(), name, args...).Run() //nolint:gosec // fixed commands
+			}
+			steps := w.plan()
+			host, _ := os.Hostname()
+			fmt.Fprintf(out, "On %s, this removes:\n", host)
+			for _, s := range steps {
+				fmt.Fprintf(out, "  - %s\n", s.what)
+			}
+			if !yes {
+				if !isTerminal(out) {
+					return errors.New("nothing removed: give --yes to wipe without being asked")
+				}
+				fmt.Fprintf(out, "\nType this machine's name (%s) to remove all of it: ", host)
+				answer, _ := bufio.NewReader(cmd.InOrStdin()).ReadString('\n')
+				if strings.TrimSpace(answer) != host {
+					return errors.New("nothing removed")
+				}
+			}
+			var failed []string
+			for _, s := range steps {
+				if err := s.do(); err != nil {
+					failed = append(failed, s.what+": "+err.Error())
+					fmt.Fprintf(out, "  !!    %s: %v\n", s.what, err)
+					continue
+				}
+				fmt.Fprintf(out, "  ok    %s\n", s.what)
+			}
+			if len(failed) > 0 {
+				return fmt.Errorf("%d step(s) failed; running wipe again finishes them", len(failed))
+			}
+			fmt.Fprintln(out, "\nllmgw is gone from this machine. If it was a node, forget it on the control plane: "+
+				"llmgw node remove "+host)
+			return nil
+		},
+	}
+	f := c.Flags()
+	f.BoolVar(&yes, "yes", false, "do not ask")
+	f.BoolVar(&w.models, "models", false, "also remove model weights, the mirror's files and installed engines")
+	f.BoolVar(&w.store, "store", false, "also remove the store's database here: a Patroni member's data, etcd's, or llmgw's database in a local PostgreSQL")
+	f.BoolVar(&w.all, "all", false, "everything: --models, --store, backups and the binary")
+	return c
+}

internal/cli/wipe_test.go

@@ -0,0 +1,106 @@
+package cli
+
+import (
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+)
+
+func wipeFixture(t *testing.T) string {
+	t.Helper()
+	root := t.TempDir()
+	for p, body := range map[string]string{
+		"/etc/llmgw/server.yaml":                      "listen: :8080\n",
+		"/etc/llmgw/pki/ca-key.pem":                   "key",
+		"/var/lib/llmgw/admin-token":                  "t",
+		"/var/lib/llmgw-agent/control-planes.json":    "[]",
+		"/etc/systemd/system/llmgw-server.service":    "[Unit]",
+		"/etc/systemd/system/llmgw-agent.service":     "[Unit]",
+		"/etc/sudoers.d/llmgw-setup":                  "x",
+		"/etc/keepalived/keepalived.conf":             "# llmgw: the gateway's floating address\n",
+		"/etc/passwd":                                 "root:x:0:0::/root:/bin/sh\nllmgw:x:999:999::/var/lib/llmgw:/usr/sbin/nologin\nllmgw-engine:x:998:998::/:/usr/sbin/nologin\n",
+		"/srv/models/qwen/v1/q.gguf":                  "w",
+		"/opt/llmgw/engines/llamacpp/b1/llama-server": "e",
+		"/etc/patroni/config.yml":                     "scope: llmgw\n",
+		"/var/lib/postgresql/16/llmgw/PG_VERSION":     "16",
+		"/etc/default/etcd":                           "ETCD_INITIAL_CLUSTER_TOKEN=llmgw\n",
+		"/var/lib/etcd/default/member/snap":           "s",
+		"/usr/local/bin/llmgw":                        "bin",
+	} {
+		full := filepath.Join(root, p)
+		if err := os.MkdirAll(filepath.Dir(full), 0o755); err != nil {
+			t.Fatal(err)
+		}
+		if err := os.WriteFile(full, []byte(body), 0o600); err != nil {
+			t.Fatal(err)
+		}
+	}
+	return root
+}
+
+func TestWipeRemovesTheConfigurationAndKeepsWhatWasNotAsked(t *testing.T) {
+	root := wipeFixture(t)
+	var ran []string
+	w := &wiper{root: root, run: func(name string, args ...string) error {
+		ran = append(ran, name+" "+strings.Join(args, " "))
+		return nil
+	}}
+	for _, s := range w.plan() {
+		if err := s.do(); err != nil {
+			t.Fatalf("%s: %v", s.what, err)
+		}
+	}
+	for _, gone := range []string{"/etc/llmgw", "/var/lib/llmgw", "/var/lib/llmgw-agent",
+		"/etc/systemd/system/llmgw-server.service", "/etc/sudoers.d/llmgw-setup", "/etc/keepalived/keepalived.conf"} {
+		if _, err := os.Stat(filepath.Join(root, gone)); err == nil {
+			t.Errorf("%s is still there", gone)
+		}
+	}
+	for _, kept := range []string{"/srv/models/qwen/v1/q.gguf", "/opt/llmgw/engines", "/etc/patroni/config.yml",
+		"/var/lib/postgresql/16/llmgw", "/var/lib/etcd/default", "/usr/local/bin/llmgw"} {
+		if _, err := os.Stat(filepath.Join(root, kept)); err != nil {
+			t.Errorf("%s was removed without being asked", kept)
+		}
+	}
+	all := strings.Join(ran, "\n")
+	for _, want := range []string{"systemctl disable --now llmgw-server.service", "pkill -KILL -u llmgw-engine",
+		"nft delete table inet llmgw", "systemctl disable --now keepalived", "userdel llmgw-engine", "userdel llmgw"} {
+		if !strings.Contains(all, want) {
+			t.Errorf("did not run %q:\n%s", want, all)
+		}
+	}
+	if strings.Contains(all, "patroni") {
+		t.Error("the store was touched without --store")
+	}
+}
+
+func TestWipeAllTakesTheStoreAndTheModelsToo(t *testing.T) {
+	root := wipeFixture(t)
+	w := &wiper{root: root, all: true, run: func(string, ...string) error { return nil }}
+	for _, s := range w.plan() {
+		if err := s.do(); err != nil {
+			t.Fatalf("%s: %v", s.what, err)
+		}
+	}
+	for _, gone := range []string{"/srv/models", "/opt/llmgw", "/etc/patroni", "/var/lib/postgresql/16/llmgw",
+		"/var/lib/etcd/default", "/etc/default/etcd", "/usr/local/bin/llmgw"} {
+		if _, err := os.Stat(filepath.Join(root, gone)); err == nil {
+			t.Errorf("%s is still there", gone)
+		}
+	}
+}
+
+func TestWipeLeavesAKeepalivedItDidNotWrite(t *testing.T) {
+	root := t.TempDir()
+	p := filepath.Join(root, "/etc/keepalived/keepalived.conf")
+	_ = os.MkdirAll(filepath.Dir(p), 0o755)
+	_ = os.WriteFile(p, []byte("vrrp_instance site_vip {}\n"), 0o600)
+	w := &wiper{root: root, run: func(string, ...string) error { return nil }}
+	for _, s := range w.plan() {
+		_ = s.do()
+	}
+	if _, err := os.Stat(p); err != nil {
+		t.Error("someone else's keepalived configuration was removed")
+	}
+}

internal/config/config.go

@@ -14,12 +14,12 @@ import (
 
 	"github.com/knadh/koanf/parsers/yaml"
 	env "github.com/knadh/koanf/providers/env/v2"
-	"github.com/knadh/koanf/providers/file"
 	"github.com/knadh/koanf/v2"
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/facility"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/mail"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/notify"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/obs"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/secrets"
@@ -147,6 +147,17 @@ type Server struct {
 	// Prometheus and Alertmanager in between. Empty sends nothing.
 	Notify notify.Config `koanf:"notify"`
 
+	// Mail is where the gateway's few emails go out from: a link to set a
+	// forgotten password. Unset, none is sent and people ask an operator.
+	Mail mail.Config `koanf:"mail"`
+
+	// Serving stops and starts the whole cluster's service (llmgw stop, llmgw
+	// start).
+	Serving ServingSettings `koanf:"serving"`
+
+	// Firewall turns llmgw's firewall on every machine on or off.
+	Firewall FirewallSettings `koanf:"firewall"`
+
 	// FirewallNetworks are networks, besides those each machine reaches the
 	// cluster on, that llmgw's internal ports answer: a monitoring host on
 	// another subnet, nodes behind a router. Written into every machine's
@@ -764,6 +775,7 @@ func Default() Server {
 		AgentListen:    ":7443",
 		MaxConnections: 10_000,
 		Mirror:         Mirror{UpdateCheck: 6 * time.Hour},
+		Firewall:       FirewallSettings{Enabled: true},
 		Control: Control{
 			Interval:         5 * time.Second,
 			BaseBackoff:      10 * time.Second,
@@ -823,15 +835,30 @@ func Default() Server {
 // An empty path loads the defaults, which is what makes `llmgw server` with a
 // couple of --endpoint flags usable for a smoke test.
 func Load(path string) (Server, error) {
-	cfg := Default()
-	k := koanf.New(".")
-
+	var b []byte
 	if path != "" {
 		if _, err := os.Stat(path); err != nil {
-			return cfg, fmt.Errorf("config file: %w", err)
+			return Default(), fmt.Errorf("config file: %w", err)
 		}
-		if err := k.Load(file.Provider(path), yaml.Parser()); err != nil {
-			return cfg, fmt.Errorf("parsing %s: %w", path, err)
+		var err error
+		if b, err = os.ReadFile(path); err != nil { //nolint:gosec // the configuration
+			return Default(), fmt.Errorf("config file: %w", err)
+		}
+	}
+	cfg, err := load(b)
+	if err != nil && path != "" && strings.HasPrefix(err.Error(), "parsing settings") {
+		return cfg, fmt.Errorf("parsing %s:%s", path, strings.TrimPrefix(err.Error(), "parsing settings:"))
+	}
+	return cfg, err
+}
+
+// load reads a YAML document of settings over the defaults.
+func load(b []byte) (Server, error) {
+	cfg := Default()
+	k := koanf.New(".")
+	if len(b) > 0 {
+		if err := k.Load(bytesProvider(b), yaml.Parser()); err != nil {
+			return cfg, fmt.Errorf("parsing settings: %w", err)
 		}
 	}
 
@@ -896,6 +923,9 @@ func (s Server) Validate() error {
 		return fmt.Errorf("backup_copy_to needs backup_recipients: an archive holds the certificate authority's key " +
 			"and every secret in the store, and leaves this machine only encrypted")
 	}
+	if err := s.Mail.Validate(); err != nil {
+		return err
+	}
 	if _, err := s.FirewallAllow(); err != nil {
 		return err
 	}

internal/config/firewall.go

@@ -32,3 +32,15 @@ func (s Server) EnginePorts() (int, int) {
 	d := Default().Control.PortRange
 	return d[0], d[1]
 }
+
+// FirewallSettings is whether llmgw keeps a firewall on the cluster's
+// machines.
+type FirewallSettings struct {
+	// Enabled keeps llmgw's internal ports — engines, node exporter, the
+	// agents' stream, the mirror, the store — to the cluster's networks on
+	// every machine. False removes llmgw's nftables table everywhere, for a
+	// site whose own firewall does that job; the rules are kept, and true
+	// puts them back. Changed in the cluster's settings, it is in force on
+	// every node and control plane within seconds.
+	Enabled bool `koanf:"enabled"`
+}

internal/config/layer.go

@@ -0,0 +1,256 @@
+package config
+
+import (
+	"bytes"
+	"errors"
+	"fmt"
+	"maps"
+	"os"
+	"reflect"
+	"slices"
+	"strings"
+
+	"go.yaml.in/yaml/v3"
+)
+
+// localKeys are the keys of server.yaml that belong to one machine: where it
+// listens, its certificates, how it reaches the store, its own secrets and
+// files. Everything else is the cluster's, kept in the store and the same on
+// every control plane (see Layer).
+var localKeys = [][]string{
+	{"listen"}, {"admin_listen"}, {"metrics_listen"}, {"agent_listen"},
+	{"max_connections"}, {"admin_token_file"}, {"admin_remote"}, {"debug_pprof"},
+	{"agent_insecure"}, {"agent_tls"},
+	{"auth", "store"}, {"threads", "store"},
+	{"secrets"}, {"backup_dir"}, {"backup_copy_key"},
+	{"tracing"}, {"monitoring"},
+	{"mirror", "api"}, {"mirror", "tls"},
+	{"endpoints"},
+}
+
+// IsLocal is whether a key path is one machine's own, kept in its
+// server.yaml, rather than the cluster's.
+func IsLocal(path []string) bool {
+	for _, l := range localKeys {
+		if len(path) >= len(l) && slices.Equal(path[:len(l)], l) {
+			return true
+		}
+	}
+	return false
+}
+
+// LocalKeys are the per-machine keys, dotted, for saying which they are.
+func LocalKeys() []string {
+	out := make([]string, len(localKeys))
+	for i, l := range localKeys {
+		out[i] = strings.Join(l, ".")
+	}
+	return out
+}
+
+// ReadFile is a configuration file as YAML maps, empty for no path.
+func ReadFile(path string) (map[string]any, error) {
+	if path == "" {
+		return map[string]any{}, nil
+	}
+	b, err := os.ReadFile(path) //nolint:gosec // the configuration
+	if err != nil {
+		return nil, fmt.Errorf("config file: %w", err)
+	}
+	m, err := Parse(b)
+	if err != nil {
+		return nil, fmt.Errorf("parsing %s: %w", path, err)
+	}
+	return m, nil
+}
+
+// Parse reads a YAML document of settings.
+func Parse(b []byte) (map[string]any, error) {
+	m := map[string]any{}
+	if len(bytes.TrimSpace(b)) == 0 {
+		return m, nil
+	}
+	if err := yaml.Unmarshal(b, &m); err != nil {
+		return nil, err
+	}
+	if m == nil {
+		m = map[string]any{}
+	}
+	return m, nil
+}
+
+// Split separates settings into one machine's and the cluster's.
+func Split(m map[string]any) (local, cluster map[string]any) {
+	local, cluster = map[string]any{}, map[string]any{}
+	split(m, nil, local, cluster)
+	return local, cluster
+}
+
+func split(m map[string]any, path []string, local, cluster map[string]any) {
+	for k, v := range m {
+		p := append(slices.Clone(path), k)
+		switch {
+		case IsLocal(p):
+			local[k] = v
+		case hasLocalBelow(p):
+			sub, ok := v.(map[string]any)
+			if !ok {
+				cluster[k] = v
+				continue
+			}
+			l, c := map[string]any{}, map[string]any{}
+			split(sub, p, l, c)
+			if len(l) > 0 {
+				local[k] = l
+			}
+			if len(c) > 0 {
+				cluster[k] = c
+			}
+		default:
+			cluster[k] = v
+		}
+	}
+}
+
+// hasLocalBelow is whether some local key lies under a path: auth holds
+// store, which is local, beside its policies, which are not.
+func hasLocalBelow(path []string) bool {
+	for _, l := range localKeys {
+		if len(l) > len(path) && slices.Equal(l[:len(path)], path) {
+			return true
+		}
+	}
+	return false
+}
+
+// Layer is what a control plane runs on: its own keys from its server.yaml
+// and every other one from the cluster's settings. The file's cluster keys
+// are not read once the cluster has settings of its own; Ignored names them.
+func Layer(file, cluster map[string]any) map[string]any {
+	local, _ := Split(file)
+	out := clone(local)
+	merge(out, cluster)
+	return out
+}
+
+// Ignored are the cluster keys a file has that the cluster's settings, once
+// there, take the place of.
+func Ignored(file map[string]any) []string {
+	_, c := Split(file)
+	return flatKeys(c, "")
+}
+
+func flatKeys(m map[string]any, prefix string) []string {
+	var out []string
+	for _, k := range slices.Sorted(maps.Keys(m)) {
+		p := prefix + k
+		if sub, ok := m[k].(map[string]any); ok && hasLocalBelow(strings.Split(p, ".")) {
+			out = append(out, flatKeys(sub, p+".")...)
+			continue
+		}
+		out = append(out, p)
+	}
+	return out
+}
+
+// CheckCluster refuses a cluster document that sets one machine's keys: a
+// listener or a certificate path written there would be every control
+// plane's, and none of them could start on it.
+func CheckCluster(m map[string]any) error {
+	local, _ := Split(m)
+	if keys := flatKeys(local, ""); len(keys) > 0 {
+		return fmt.Errorf("%s belong to each machine's server.yaml, not to the cluster's settings",
+			strings.Join(keys, ", "))
+	}
+	return nil
+}
+
+// Changed are the keys whose values differ between two sets of settings,
+// at the top level, and below it where a key mixes local and cluster ones.
+func Changed(a, b map[string]any) []string {
+	var out []string
+	changed(a, b, nil, &out)
+	slices.Sort(out)
+	return out
+}
+
+func changed(a, b map[string]any, path []string, out *[]string) {
+	keys := map[string]bool{}
+	for k := range a {
+		keys[k] = true
+	}
+	for k := range b {
+		keys[k] = true
+	}
+	for k := range keys {
+		p := append(slices.Clone(path), k)
+		va, vb := a[k], b[k]
+		if reflect.DeepEqual(va, vb) {
+			continue
+		}
+		ma, okA := va.(map[string]any)
+		mb, okB := vb.(map[string]any)
+		if hasLocalBelow(p) && (okA || va == nil) && (okB || vb == nil) {
+			changed(ma, mb, p, out)
+			continue
+		}
+		*out = append(*out, strings.Join(p, "."))
+	}
+}
+
+// Marshal writes settings as YAML.
+func Marshal(m map[string]any) ([]byte, error) {
+	if len(m) == 0 {
+		return nil, nil
+	}
+	var b bytes.Buffer
+	enc := yaml.NewEncoder(&b)
+	enc.SetIndent(2)
+	if err := enc.Encode(m); err != nil {
+		return nil, err
+	}
+	return b.Bytes(), enc.Close()
+}
+
+// FromMap is the configuration settings make: laid over the defaults, with
+// LLMGW_ environment overrides, and validated.
+func FromMap(m map[string]any) (Server, error) {
+	b, err := Marshal(m)
+	if err != nil {
+		return Default(), err
+	}
+	return load(b)
+}
+
+// bytesProvider hands koanf a document already read.
+type bytesProvider []byte
+
+func (b bytesProvider) ReadBytes() ([]byte, error) { return b, nil }
+func (b bytesProvider) Read() (map[string]any, error) {
+	return nil, errors.New("bytesProvider reads bytes only")
+}
+
+func clone(m map[string]any) map[string]any {
+	out := make(map[string]any, len(m))
+	for k, v := range m {
+		if sub, ok := v.(map[string]any); ok {
+			v = clone(sub)
+		}
+		out[k] = v
+	}
+	return out
+}
+
+// merge lays src over dst: maps key by key, everything else replaced.
+func merge(dst, src map[string]any) {
+	for k, v := range src {
+		if sv, ok := v.(map[string]any); ok {
+			if dv, ok := dst[k].(map[string]any); ok {
+				merge(dv, sv)
+				continue
+			}
+			v = clone(sv)
+		}
+		dst[k] = v
+	}
+}

internal/config/layer_test.go

@@ -0,0 +1,99 @@
+package config
+
+import (
+	"slices"
+	"strings"
+	"testing"
+)
+
+const machineFile = `
+listen: ":8080"
+agent_listen: ":7443"
+agent_insecure: true
+auth:
+  enabled: true
+  store: /var/lib/llmgw/llmgw.db
+  default_rpm: 60
+mirror:
+  api: http://127.0.0.1:8082
+  update_check: 6h
+guard:
+  default: [baseline]
+  policies:
+    baseline:
+      input:
+        injection: flag
+router:
+  retries: 2
+`
+
+func TestAFileSplitsIntoTheMachinesAndTheClusters(t *testing.T) {
+	file, err := Parse([]byte(machineFile))
+	if err != nil {
+		t.Fatal(err)
+	}
+	local, cluster := Split(file)
+	for _, k := range []string{"listen", "agent_listen", "agent_insecure"} {
+		if _, ok := local[k]; !ok {
+			t.Errorf("%s is not the machine's", k)
+		}
+	}
+	if local["auth"].(map[string]any)["store"] == nil || cluster["auth"].(map[string]any)["store"] != nil {
+		t.Errorf("auth.store went to the cluster: local %v cluster %v", local["auth"], cluster["auth"])
+	}
+	if cluster["auth"].(map[string]any)["default_rpm"] != 60 {
+		t.Errorf("auth's policy stayed with the machine: %v", cluster["auth"])
+	}
+	if local["mirror"].(map[string]any)["api"] == nil || cluster["mirror"].(map[string]any)["update_check"] == nil {
+		t.Errorf("mirror split wrong: local %v cluster %v", local["mirror"], cluster["mirror"])
+	}
+	for _, k := range []string{"guard", "router"} {
+		if _, ok := cluster[k]; !ok {
+			t.Errorf("%s is not the cluster's", k)
+		}
+	}
+	if err := CheckCluster(cluster); err != nil {
+		t.Errorf("the cluster half is refused: %v", err)
+	}
+	if err := CheckCluster(file); err == nil || !strings.Contains(err.Error(), "auth.store") {
+		t.Errorf("a cluster document with a machine's keys: %v", err)
+	}
+}
+
+func TestTheClustersSettingsTakeThePlaceOfTheFiles(t *testing.T) {
+	file, _ := Parse([]byte(machineFile))
+	cluster, _ := Parse([]byte("router:\n  retries: 5\nauth:\n  enabled: true\n  default_rpm: 10\n"))
+	eff := Layer(file, cluster)
+	cfg, err := FromMap(eff)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if cfg.Router.Retries != 5 || cfg.Auth.DefaultRPM != 10 {
+		t.Errorf("cluster values not in force: retries %d rpm %d", cfg.Router.Retries, cfg.Auth.DefaultRPM)
+	}
+	if cfg.Auth.Store != "/var/lib/llmgw/llmgw.db" || cfg.Mirror.API != "http://127.0.0.1:8082" || cfg.Listen != ":8080" {
+		t.Errorf("the machine's own keys lost: %+v", cfg.Auth)
+	}
+	if len(cfg.Guard.Policies) != 0 {
+		t.Error("the file's guard applies though the cluster has settings without one")
+	}
+	if got := Ignored(file); !slices.Contains(got, "guard") || !slices.Contains(got, "auth.default_rpm") ||
+		slices.Contains(got, "auth.store") {
+		t.Errorf("ignored: %v", got)
+	}
+}
+
+func TestChangedNamesWhatMoved(t *testing.T) {
+	a, _ := Parse([]byte(machineFile))
+	b, _ := Parse([]byte(strings.Replace(machineFile, "retries: 2", "retries: 3", 1)))
+	if got := Changed(a, b); !slices.Equal(got, []string{"router"}) {
+		t.Errorf("changed %v", got)
+	}
+	c, _ := Parse([]byte(strings.Replace(machineFile, "store: /var/lib/llmgw/llmgw.db", "store: /x.db", 1)))
+	if got := Changed(a, c); !slices.Equal(got, []string{"auth.store"}) {
+		t.Errorf("changed %v", got)
+	}
+	if got := Changed(a, a); len(got) != 0 {
+		t.Errorf("unchanged: %v", got)
+	}
+}

internal/config/serving.go

@@ -0,0 +1,11 @@
+package config
+
+// ServingSettings is whether the cluster serves at all.
+type ServingSettings struct {
+	// Stopped takes every model down and answers every request for one with
+	// 503 (system_stopped), leaving the control planes, agents and mirror
+	// running and every model's definition as it is: llmgw stop sets it,
+	// llmgw start clears it, and the models come back as their catalog
+	// entries ask. In force on every control plane within seconds.
+	Stopped bool `koanf:"stopped"`
+}

internal/control/adminapi/api.go

@@ -12,6 +12,7 @@ import (
 	"errors"
 	"fmt"
 	"io"
+	"maps"
 	"net/http"
 	"net/url"
 	"slices"
@@ -67,7 +68,10 @@ type API struct {
 	// nil without a store.
 	federation *federation.Service
 	// guard is the guardrail policies' library and attachments.
-	guard *GuardAdmin
+	guard    *GuardAdmin
+	settings *Settings
+	// restartAgents tells every connected agent to start again.
+	restartAgents func(reason string) []string
 	// revoke records a removed node, so its certificate is refused.
 	revoke func(ctx context.Context, nodeID string, at time.Time) error
 
@@ -111,6 +115,8 @@ func (a *API) Register(mux Mux) {
 	mux.HandleFunc("POST /api/v1/models/{name}/resources", a.setResources)
 	a.registerQuotas(mux)
 	a.registerGuard(mux)
+	a.registerSettings(mux)
+	mux.HandleFunc("POST /api/v1/agents/restart", a.restartAgentsHandler)
 	mux.HandleFunc("GET /api/v1/audit", a.auditEvents)
 	mux.HandleFunc("GET /api/v1/usage/export", a.usageExport)
 	mux.HandleFunc("POST /api/v1/notify/test", a.notifyTest)
@@ -226,6 +232,17 @@ type NodeView struct {
 	// which is not the same as having none — placement treats it that way too,
 	// or a control plane upgrade would empty a fleet.
 	Engines []string `json:"engines,omitempty"`
+	// EngineVersions are the installed engines with their versions:
+	// llamacpp → b10991. ComputeCapabilities are its GPUs' CUDA compute
+	// capabilities, 7.0 for a V100, so an engine build's support can be read
+	// against them.
+	EngineVersions map[string]string `json:"engine_versions,omitempty"`
+	// MultiNode is whether it can take part in replicas spanning nodes:
+	// "ready", "down: mlx5_1" naming the declared adapters whose ports are
+	// not active, or empty for a node on no fabric. Its own replicas run
+	// either way.
+	MultiNode           string   `json:"multi_node,omitempty"`
+	ComputeCapabilities []string `json:"compute_capabilities,omitempty"`
 	// Provision is where installing each wanted engine stands. An engine is in
 	// Engines only once its installation here is ready.
 	Provision []inventory.EngineInstall `json:"provision,omitempty"`
@@ -414,10 +431,12 @@ type ModelView struct {
 	Description string   `json:"description,omitempty"`
 	Aliases     []string `json:"aliases,omitempty"`
 	Source      string   `json:"source"`
-	Min         int      `json:"min_replicas"`
-	Max         int      `json:"max_replicas"`
-	Ready       int      `json:"ready_replicas"`
-	Total       int      `json:"total_replicas"`
+	// Role is "guard" for a guardrails judge, empty for a model clients use.
+	Role  string `json:"role,omitempty"`
+	Min   int    `json:"min_replicas"`
+	Max   int    `json:"max_replicas"`
+	Ready int    `json:"ready_replicas"`
+	Total int    `json:"total_replicas"`
 	// Desired is how many it should have now: Min, or for an on-demand
 	// model one while it is being asked for and none otherwise.
 	Desired int `json:"desired_replicas"`
@@ -516,6 +535,30 @@ type ShardView struct {
 	// node's last heartbeat; zero when not measured.
 	CPUCores    float64 `json:"cpu_cores,omitempty"`
 	MemoryBytes int64   `json:"memory_bytes,omitempty"`
+	// Compute is where the engine said it put the model as it became
+	// ready: absent when it did not say.
+	Compute *ComputeView `json:"compute,omitempty"`
+}
+
+// ComputeView is where an engine runs its model, by its own account.
+type ComputeView struct {
+	// Backend is CUDA, Vulkan, Metal, ROCm, GPU or CPU.
+	Backend string   `json:"backend"`
+	Devices []string `json:"devices,omitempty"`
+	// LayersOffloaded of LayersTotal went to the devices; absent when unsaid.
+	LayersOffloaded int `json:"layers_offloaded,omitempty"`
+	LayersTotal     int `json:"layers_total,omitempty"`
+}
+
+// Short is the compute in a few characters: "CUDA 49/49", "CPU".
+func (c *ComputeView) Short() string {
+	if c == nil {
+		return "-"
+	}
+	if c.LayersTotal > 0 && c.Backend != "CPU" {
+		return fmt.Sprintf("%s %d/%d", c.Backend, c.LayersOffloaded, c.LayersTotal)
+	}
+	return c.Backend
 }
 
 // FetchView is a download in flight.
@@ -621,6 +664,16 @@ func (a *API) Nodes() []NodeView {
 			v.Engines = append(v.Engines, name)
 		}
 		slices.Sort(v.Engines)
+		if len(n.EngineVersions) > 0 {
+			v.EngineVersions = maps.Clone(n.EngineVersions)
+		}
+		for _, g := range n.GPUs {
+			if g.ComputeCapability != "" && !slices.Contains(v.ComputeCapabilities, g.ComputeCapability) {
+				v.ComputeCapabilities = append(v.ComputeCapabilities, g.ComputeCapability)
+			}
+		}
+		slices.Sort(v.ComputeCapabilities)
+		v.MultiNode = multiNodeReadiness(n)
 		v.Provision = n.Provision
 		if ch := a.ctl.EngineChoices(); ch != nil && len(n.GPUs) > 0 {
 			chosen := ch.For(n.ID)
@@ -759,6 +812,7 @@ func (a *API) Models() []ModelView {
 			Class: m.EffectiveClass(), Owner: m.Owner, Preempted: h.Preempted, HeldFor: h.HeldFor,
 			External: externalLabel(m), Adapters: m.Adapters,
 			Canary: m.Canary, CanaryOf: a.ctl.Catalog().CanaryOf(m.Name), Eval: m.Eval,
+			Role: m.Role,
 		}
 		for _, p := range m.Profiles {
 			v.Profiles = append(v.Profiles, p.Name)
@@ -827,6 +881,10 @@ func (a *API) Replicas(model string) []ReplicaView {
 					sv.Fetch.Percent = &p
 				}
 			}
+			if c := s.Compute; c != nil {
+				sv.Compute = &ComputeView{Backend: c.Backend, Devices: c.Devices,
+					LayersOffloaded: c.Offloaded, LayersTotal: c.Total}
+			}
 			if l := loads[s.NodeID]; l != nil {
 				if use, ok := l.Shards[s.ID]; ok {
 					sv.CPUCores, sv.MemoryBytes = use.CPUCores, use.RSSBytes
@@ -1471,6 +1529,11 @@ type RestartRequest struct {
 	// InPlace stops each replica before starting its replacement, rather
 	// than starting the replacement beside it first.
 	InPlace bool `json:"in_place,omitempty"`
+	// MaxSurge is how many of a model's replicas are replaced beside
+	// themselves at once, MaxUnavailable how many are stopped first at once;
+	// both zero is one at a time, replaced beside itself.
+	MaxSurge       int `json:"max_surge,omitempty"`
+	MaxUnavailable int `json:"max_unavailable,omitempty"`
 }
 
 // RestartResult lists the replicas queued.
@@ -1494,11 +1557,12 @@ func (a *API) restart(w http.ResponseWriter, r *http.Request) {
 		writeError(w, http.StatusBadRequest, "nothing ready matches; name replicas, models, or all")
 		return
 	}
-	restart := a.ctl.Reconciler().Restart
-	if req.InPlace {
-		restart = a.ctl.Reconciler().RestartInPlace
+	if req.MaxSurge < 0 || req.MaxUnavailable < 0 {
+		writeError(w, http.StatusBadRequest, "max_surge and max_unavailable are counts of replicas, not negative")
+		return
 	}
-	if err := restart(ids...); err != nil {
+	pace := reconciler.RestartPace{MaxSurge: req.MaxSurge, MaxUnavailable: req.MaxUnavailable}
+	if err := a.ctl.Reconciler().RestartPaced(pace, req.InPlace, ids...); err != nil {
 		writeError(w, http.StatusNotFound, err.Error())
 		return
 	}
@@ -1543,6 +1607,7 @@ type RequestView struct {
 	GuardAction   string `json:"guard_action,omitempty"`
 	GuardPolicy   string `json:"guard_policy,omitempty"`
 	GuardCategory string `json:"guard_category,omitempty"`
+	GuardRule     string `json:"guard_rule,omitempty"`
 	GuardMs       int64  `json:"guard_ms,omitempty"`
 	// What the client said about it: OpenAI's user field, and its tags.
 	EndUser string            `json:"end_user,omitempty"`
@@ -1571,7 +1636,7 @@ func (a *API) request(w http.ResponseWriter, r *http.Request) {
 		Model: u.Model, ServedModel: u.ServedModel, Replica: u.Replica, Status: u.Status,
 		LatencyMs: u.LatencyMs, QueueMs: u.QueueMs, TTFTMs: u.TTFTMs,
 		PromptTokens: u.PromptTokens, CompletionTokens: u.CompletionTokens, CostMicros: u.CostMicros,
-		GuardAction: u.GuardAction, GuardPolicy: u.GuardPolicy, GuardCategory: u.GuardCategory, GuardMs: u.GuardMs,
+		GuardAction: u.GuardAction, GuardPolicy: u.GuardPolicy, GuardCategory: u.GuardCategory, GuardRule: u.GuardRule, GuardMs: u.GuardMs,
 		EndUser: u.EndUser, Tags: u.Tags, CacheHit: u.CacheHit,
 	})
 }
@@ -1973,3 +2038,32 @@ func (a *API) SetNodeEngine(ctx context.Context, node, engine, build, by string)
 	}
 	return ch.Set(ctx, node, engine, build, by)
 }
+
+// multiNodeReadiness is whether a node's declared fabrics are up: every
+// adapter they name with an active port. Empty for a node on no fabric.
+func multiNodeReadiness(n inventory.Node) string {
+	if len(n.Fabrics) == 0 {
+		return ""
+	}
+	if len(n.IBPorts) == 0 {
+		return "unknown: no port reported yet"
+	}
+	var down []string
+	for _, f := range n.Fabrics {
+		for _, hca := range f.HCAs {
+			up := false
+			for _, p := range n.IBPorts {
+				if p.HCA == hca && p.Active() {
+					up = true
+				}
+			}
+			if !up && !slices.Contains(down, hca) {
+				down = append(down, hca)
+			}
+		}
+	}
+	if len(down) > 0 {
+		return "down: " + strings.Join(down, ", ")
+	}
+	return "ready"
+}

internal/control/adminapi/guard.go

@@ -21,12 +21,17 @@ type GuardStore interface {
 	DetachGuard(ctx context.Context, a guard.Attachment) (bool, error)
 }
 
-// GuardAdmin is what the guard endpoints need: the library from server.yaml,
-// the store, and a way to have the gate read attachments again at once.
+// GuardAdmin is what the guard endpoints need: the library in force (the
+// cluster's settings), the store, and a way to have the gate read
+// attachments again at once.
 type GuardAdmin struct {
-	Library    guard.Config
+	Library    func() guard.Config
 	Store      GuardStore
 	Invalidate func()
+	// JudgeErrors are each judge's last failure to give a verdict.
+	JudgeErrors func() map[string]guard.JudgeError
+	// Ask has a judge read a text, past the cache.
+	Ask func(ctx context.Context, model, text, answer string) (guard.JudgeAnswer, error)
 }
 
 // SetGuard enables the guard endpoints.
@@ -34,6 +39,8 @@ func (a *API) SetGuard(g *GuardAdmin) { a.guard = g }
 
 // GuardView is the library and where its policies are attached.
 type GuardView struct {
+	// Enabled is whether the guardrails check anything (guard.enabled).
+	Enabled     bool                  `json:"enabled"`
 	Policies    []string              `json:"policies"`
 	Default     []string              `json:"default"`
 	Attachments []GuardAttachmentView `json:"attachments"`
@@ -88,7 +95,10 @@ type GuardEventView struct {
 	Action    string    `json:"action"`
 	Policy    string    `json:"policy"`
 	Category  string    `json:"category,omitempty"`
-	GuardMs   int64     `json:"guard_ms"`
+	// Rule is what decided it within the category: a pattern's name, or a
+	// judge's verdict, "S6,Specialized Advice".
+	Rule    string `json:"rule,omitempty"`
+	GuardMs int64  `json:"guard_ms"`
 }
 
 // policyView sums one policy up.
@@ -197,9 +207,11 @@ type GuardAttachmentRequest struct {
 }
 
 func (a *API) registerGuard(mux Mux) {
+	mux.HandleFunc("GET /api/v1/guard/judges", a.listJudges)
+	mux.HandleFunc("POST /api/v1/guard/judges/ask", a.askJudge)
 	mux.HandleFunc("GET /api/v1/guard", func(w http.ResponseWriter, r *http.Request) {
 		if a.guard == nil {
-			writeError(w, http.StatusNotImplemented, "no guard policies: define them under guard: in server.yaml")
+			writeError(w, http.StatusNotImplemented, "no guard policies: define them under guard in the cluster's settings (llmgw ctl config edit)")
 			return
 		}
 		att, err := a.guard.Store.GuardAttachments(r.Context())
@@ -207,7 +219,7 @@ func (a *API) registerGuard(mux Mux) {
 			writeError(w, http.StatusInternalServerError, err.Error())
 			return
 		}
-		v := GuardView{Default: a.guard.Library.Default, Attachments: []GuardAttachmentView{}}
+		v := GuardView{Enabled: a.guard.Library().On(), Default: a.guard.Library().Default, Attachments: []GuardAttachmentView{}}
 		var names map[string]string
 		if a.quotas != nil {
 			names, _ = a.scopeNames(r.Context())
@@ -215,12 +227,12 @@ func (a *API) registerGuard(mux Mux) {
 		for _, at := range att {
 			v.Attachments = append(v.Attachments, GuardAttachmentView{Attachment: at, Name: names[at.Scope+":"+at.ScopeID]})
 		}
-		for name := range a.guard.Library.Policies {
+		for name := range a.guard.Library().Policies {
 			v.Policies = append(v.Policies, name)
 		}
 		slices.Sort(v.Policies)
 		for _, name := range v.Policies {
-			v.Details = append(v.Details, policyView(name, a.guard.Library.Policies[name]))
+			v.Details = append(v.Details, policyView(name, a.guard.Library().Policies[name]))
 		}
 		writeJSON(w, http.StatusOK, v)
 	})
@@ -268,14 +280,14 @@ func (a *API) registerGuard(mux Mux) {
 		slices.SortStableFunc(v.Counts, func(a, b GuardCountView) int { return int(b.Count - a.Count) })
 		for _, e := range events {
 			v.Events = append(v.Events, GuardEventView{At: e.At, RequestID: e.RequestID, Project: project(e.ProjectID),
-				Model: e.Model, Status: e.Status, Action: e.Action, Policy: e.Policy, Category: e.Category, GuardMs: e.GuardMs})
+				Model: e.Model, Status: e.Status, Action: e.Action, Policy: e.Policy, Category: e.Category, Rule: e.Rule, GuardMs: e.GuardMs})
 		}
 		writeJSON(w, http.StatusOK, v)
 	})
 	change := func(attach bool) http.HandlerFunc {
 		return func(w http.ResponseWriter, r *http.Request) {
 			if a.guard == nil {
-				writeError(w, http.StatusNotImplemented, "no guard policies: define them under guard: in server.yaml")
+				writeError(w, http.StatusNotImplemented, "no guard policies: define them under guard in the cluster's settings (llmgw ctl config edit)")
 				return
 			}
 			var at GuardAttachmentRequest
@@ -289,8 +301,8 @@ func (a *API) registerGuard(mux Mux) {
 				return
 			}
 			att.Policy = at.Policy
-			if _, ok := a.guard.Library.Policies[att.Policy]; !ok && attach {
-				writeError(w, http.StatusBadRequest, fmt.Sprintf("no policy %q in server.yaml's guard.policies", att.Policy))
+			if _, ok := a.guard.Library().Policies[att.Policy]; !ok && attach {
+				writeError(w, http.StatusBadRequest, fmt.Sprintf("no policy %q in the cluster's guard.policies", att.Policy))
 				return
 			}
 			changed := true

internal/control/adminapi/guard_test.go

@@ -36,8 +36,10 @@ func TestGuardPoliciesAreAttachedByTarget(t *testing.T) {
 	st := &memGuard{}
 	invalidated := 0
 	a.SetGuard(&GuardAdmin{
-		Library: guard.Config{Default: []string{"base"}, Policies: map[string]guard.Policy{"base": {}, "finance": {}}},
-		Store:   st, Invalidate: func() { invalidated++ },
+		Library: func() guard.Config {
+			return guard.Config{Default: []string{"base"}, Policies: map[string]guard.Policy{"base": {}, "finance": {}}}
+		},
+		Store: st, Invalidate: func() { invalidated++ },
 	})
 	if code := call(t, a, "POST", "/api/v1/guard/attachments", `{"target":"key:k1","policy":"finance"}`, nil); code != 200 {
 		t.Fatalf("attach: %d", code)
@@ -138,10 +140,11 @@ func (m *memGuardHistory) GuardCounts(context.Context, time.Time) ([]store.Guard
 func TestTheGuardrailsSayWhatEachPolicyDoesAndWhatTheyDid(t *testing.T) {
 	a, _ := setup(t, testModels())
 	h := &memGuardHistory{}
-	a.SetGuard(&GuardAdmin{Store: h, Library: guard.Config{Policies: map[string]guard.Policy{
+	lib := guard.Config{Policies: map[string]guard.Policy{
 		"base": {Fail: "closed", Input: guard.Rules{PII: map[string]string{"email": "redact"}, Injection: "block"},
 			Judge: &guard.JudgePolicy{Model: "llama-guard-3-1b", Input: "block", Categories: []string{"S1", "S9"}}},
-	}}})
+	}}
+	a.SetGuard(&GuardAdmin{Store: h, Library: func() guard.Config { return lib }})
 
 	var v GuardView
 	if code := call(t, a, "GET", "/api/v1/guard", "", &v); code != 200 || len(v.Details) != 1 {

internal/control/adminapi/judges.go

@@ -0,0 +1,200 @@
+package adminapi
+
+import (
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"slices"
+	"sort"
+	"strings"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
+)
+
+// JudgeView is one judge model the guardrail policies ask, and whether it
+// can answer them.
+type JudgeView struct {
+	Model string `json:"model"`
+	// Policies are those that ask it; Closed those that refuse what it
+	// cannot read (fail: closed).
+	Policies []string `json:"policies"`
+	Closed   []string `json:"closed,omitempty"`
+	// Then are the judges its policies ask after it, when it gives no
+	// verdict: a CPU judge behind a GPU one.
+	Then []string `json:"then,omitempty"`
+	// InCatalog is whether a model of that name is in the catalog, and
+	// Judge whether it is there as a judge (role: guard).
+	InCatalog bool `json:"in_catalog"`
+	Judge     bool `json:"judge"`
+	Ready     int  `json:"ready_replicas"`
+	Wanted    int  `json:"wanted_replicas"`
+	// LastError is its last failure to give a verdict, if any.
+	LastError   string     `json:"last_error,omitempty"`
+	LastErrorAt *time.Time `json:"last_error_at,omitempty"`
+	// Problem says, in words, what is wrong and what to do; empty when
+	// nothing is.
+	Problem string `json:"problem,omitempty"`
+}
+
+// judgeErrorFresh is how recent a judge's failure must be to count as a
+// problem now.
+const judgeErrorFresh = 5 * time.Minute
+
+// Judges are the judge models the policies in force ask, each with what is
+// wrong with it, if anything.
+func (a *API) Judges() []JudgeView {
+	if a.guard == nil || a.guard.Library == nil {
+		return nil
+	}
+	lib := a.guard.Library()
+	if !lib.On() {
+		// Off: no judge is asked, so none is missing.
+		return nil
+	}
+	byModel := map[string]*JudgeView{}
+	for name, p := range lib.Policies {
+		judges := p.Judge.Judges()
+		for i, model := range judges {
+			v := byModel[model]
+			if v == nil {
+				v = &JudgeView{Model: model}
+				byModel[model] = v
+			}
+			if !slices.Contains(v.Policies, name) {
+				v.Policies = append(v.Policies, name)
+			}
+			if p.Fail == "closed" && p.Mode != "shadow" && !slices.Contains(v.Closed, name) {
+				v.Closed = append(v.Closed, name)
+			}
+			for _, after := range judges[i+1:] {
+				if !slices.Contains(v.Then, after) {
+					v.Then = append(v.Then, after)
+				}
+			}
+		}
+	}
+	ready := map[string]int{}
+	for _, rep := range a.ctl.Reconciler().Replicas() {
+		if rep.State == reconciler.Ready {
+			ready[rep.Model]++
+		}
+	}
+	var errs map[string]guard.JudgeError
+	if a.guard.JudgeErrors != nil {
+		errs = a.guard.JudgeErrors()
+	}
+	out := make([]JudgeView, 0, len(byModel))
+	for _, v := range byModel {
+		sort.Strings(v.Policies)
+		sort.Strings(v.Closed)
+		if m, ok := a.ctl.Catalog().Resolve(v.Model); ok && m.Name == v.Model {
+			v.InCatalog, v.Judge = true, m.IsGuard()
+			v.Wanted, v.Ready = m.Scaling.Min, ready[m.Name]
+		}
+		if e, ok := errs[v.Model]; ok {
+			at := e.At
+			v.LastError, v.LastErrorAt = e.Err, &at
+		}
+		v.Problem = judgeProblem(*v)
+		out = append(out, *v)
+	}
+	slices.SortFunc(out, func(x, y JudgeView) int {
+		switch {
+		case x.Model < y.Model:
+			return -1
+		case x.Model > y.Model:
+			return 1
+		}
+		return 0
+	})
+	return out
+}
+
+func judgeProblem(v JudgeView) string {
+	consequence := "requests its policies judge are served unjudged"
+	switch {
+	case len(v.Then) > 0:
+		consequence = "its policies ask " + strings.Join(v.Then, ", else ") + " instead"
+	case len(v.Closed) > 0:
+		consequence = "requests its policies judge are refused (guard_unavailable)"
+	}
+	switch {
+	case !v.InCatalog:
+		return fmt.Sprintf("not in the catalog, so %s: llmgw ctl guard judge enable", consequence)
+	case !v.Judge:
+		return fmt.Sprintf("in the catalog as a model clients use, not as a judge (role: guard), so %s: "+
+			"llmgw ctl guard judge enable --replace", consequence)
+	case v.Ready == 0:
+		return fmt.Sprintf("no replica ready, so %s: llmgw ctl replicas --model %s", consequence, v.Model)
+	case v.LastErrorAt != nil && time.Since(*v.LastErrorAt) < judgeErrorFresh:
+		return fmt.Sprintf("gave no verdict %s ago (%s)", time.Since(*v.LastErrorAt).Round(time.Second), v.LastError)
+	}
+	return ""
+}
+
+func (a *API) listJudges(w http.ResponseWriter, _ *http.Request) {
+	if a.guard == nil {
+		writeError(w, http.StatusNotImplemented, "no guard policies: define them under guard in the cluster's settings (llmgw ctl config edit)")
+		return
+	}
+	writeJSON(w, http.StatusOK, a.Judges())
+}
+
+// JudgeAskRequest is a text to have a judge read.
+type JudgeAskRequest struct {
+	// Model is the judge; empty is the one the policies ask, when they ask one.
+	Model string `json:"model,omitempty"`
+	Text  string `json:"text"`
+	// Answer, when given, is read as the assistant's reply to Text.
+	Answer string `json:"answer,omitempty"`
+}
+
+// JudgeAskView is what the judge answered.
+type JudgeAskView struct {
+	Model string `json:"model"`
+	// Verdict is safe, unsafe or controversial, as the gateway reads the
+	// answer; a policy's judge.controversial says what controversial counts as.
+	Verdict    string   `json:"verdict"`
+	Categories []string `json:"categories,omitempty"`
+	// Raw is the judge's answer as it gave it.
+	Raw    string `json:"raw"`
+	TookMs int64  `json:"took_ms"`
+}
+
+func (a *API) askJudge(w http.ResponseWriter, r *http.Request) {
+	if a.guard == nil || a.guard.Ask == nil {
+		writeError(w, http.StatusNotImplemented, "no guard policies: define them under guard in the cluster's settings (llmgw ctl config edit)")
+		return
+	}
+	var req JudgeAskRequest
+	if err := json.NewDecoder(r.Body).Decode(&req); err != nil || strings.TrimSpace(req.Text) == "" {
+		writeError(w, http.StatusBadRequest, "give the text for the judge to read")
+		return
+	}
+	if req.Model == "" {
+		judges := a.Judges()
+		if len(judges) == 0 {
+			writeError(w, http.StatusBadRequest, "no policy asks a judge: give the model")
+			return
+		}
+		req.Model = judges[0].Model
+	}
+	ans, err := a.guard.Ask(r.Context(), req.Model, req.Text, req.Answer)
+	v := JudgeAskView{Model: req.Model, Raw: ans.Raw, Categories: ans.Categories, TookMs: ans.Took.Milliseconds()}
+	if err != nil {
+		writeError(w, http.StatusBadGateway, fmt.Sprintf("the judge %s gave no verdict: %v (it answered %q)", req.Model, err, ans.Raw))
+		return
+	}
+	switch {
+	case ans.Unsafe:
+		v.Verdict = "unsafe"
+	case ans.Controversial:
+		// Qwen3Guard's middle verdict: what it counts as is the policy's.
+		v.Verdict = "controversial"
+	default:
+		v.Verdict = "safe"
+	}
+	writeJSON(w, http.StatusOK, v)
+}

internal/control/adminapi/judges_test.go

@@ -0,0 +1,125 @@
+package adminapi
+
+import (
+	"context"
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
+)
+
+func TestEachJudgeSaysWhatIsWrongWithIt(t *testing.T) {
+	models := append(testModels(),
+		// Served from the mirror page before it knew better: a model
+		// clients use, under the judge's name.
+		catalog.Model{Name: "shield", Source: "/s", Scaling: catalog.Scaling{Min: 1, Max: 1},
+			Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp"}}},
+		catalog.Model{Name: "llama-guard-3-1b", Source: "/g", Role: catalog.RoleGuard,
+			Guard: &catalog.GuardModel{Format: "llamaguard3"}, Scaling: catalog.Scaling{Min: 2, Max: 2},
+			Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp"}}})
+	a, _ := setup(t, models)
+	lib := guard.Config{Default: []string{"baseline"}, Policies: map[string]guard.Policy{
+		"baseline": {Fail: "closed", Judge: &guard.JudgePolicy{Model: "llama-guard-3-1b", Input: "block"}},
+		"strict":   {Fail: "closed", Judge: &guard.JudgePolicy{Model: "missing-judge", Input: "block"}},
+		"shield":   {Fail: "open", Judge: &guard.JudgePolicy{Model: "shield", Input: "flag"}},
+	}}
+	a.SetGuard(&GuardAdmin{Library: func() guard.Config { return lib }})
+
+	var js []JudgeView
+	if code := call(t, a, "GET", "/api/v1/guard/judges", "", &js); code != 200 || len(js) != 3 {
+		t.Fatalf("%d %+v", code, js)
+	}
+	by := map[string]JudgeView{}
+	for _, j := range js {
+		by[j.Model] = j
+	}
+	if j := by["missing-judge"]; j.InCatalog || !strings.Contains(j.Problem, "not in the catalog") ||
+		!strings.Contains(j.Problem, "refused") {
+		t.Errorf("missing: %+v", j)
+	}
+	if j := by["shield"]; !j.InCatalog || j.Judge || !strings.Contains(j.Problem, "--replace") ||
+		!strings.Contains(j.Problem, "unjudged") {
+		t.Errorf("not a judge: %+v", j)
+	}
+	if j := by["llama-guard-3-1b"]; !j.Judge || j.Wanted != 2 || !strings.Contains(j.Problem, "no replica ready") {
+		t.Errorf("not running: %+v", j)
+	}
+
+	var s Summary
+	call(t, a, "GET", "/api/v1/summary", "", &s)
+	if len(s.Judges) != 3 {
+		t.Errorf("the summary does not carry the judges' problems: %+v", s.Judges)
+	}
+
+	// A judge that failed lately says so, once it is otherwise fine.
+	at := time.Now()
+	v := JudgeView{Model: "x", InCatalog: true, Judge: true, Ready: 1, LastError: "deadline exceeded", LastErrorAt: &at}
+	if p := judgeProblem(v); !strings.Contains(p, "deadline exceeded") {
+		t.Errorf("a recent failure: %q", p)
+	}
+	old := at.Add(-time.Hour)
+	v.LastErrorAt = &old
+	if p := judgeProblem(v); p != "" {
+		t.Errorf("an old failure still counts: %q", p)
+	}
+}
+
+func TestAJudgeIsAskedAndItsWordsShown(t *testing.T) {
+	a, _ := setup(t, testModels())
+	lib := guard.Config{Policies: map[string]guard.Policy{
+		"baseline": {Judge: &guard.JudgePolicy{Model: "llama-guard-3-1b", Input: "block"}}}}
+	var asked string
+	a.SetGuard(&GuardAdmin{Library: func() guard.Config { return lib },
+		Ask: func(_ context.Context, model, text, _ string) (guard.JudgeAnswer, error) {
+			asked = model + "|" + text
+			return guard.JudgeAnswer{Model: model, Raw: "unsafe\nS6", Unsafe: true,
+				Categories: []string{"S6", "Specialized Advice"}}, nil
+		}})
+	var v JudgeAskView
+	if code := call(t, a, "POST", "/api/v1/guard/judges/ask", `{"text":"how do I sort a list"}`, &v); code != 200 {
+		t.Fatalf("%d", code)
+	}
+	if asked != "llama-guard-3-1b|how do I sort a list" || v.Verdict != "unsafe" || v.Raw != "unsafe\nS6" ||
+		len(v.Categories) != 2 {
+		t.Errorf("asked %q, got %+v", asked, v)
+	}
+	if code := call(t, a, "POST", "/api/v1/guard/judges/ask", `{"text":"  "}`, nil); code != 400 {
+		t.Errorf("no text: %d", code)
+	}
+}
+
+func TestANodeSaysItsEnginesVersionsAndCapabilities(t *testing.T) {
+	n := gpuNode("volta", 2)
+	n.EngineVersions = map[string]string{"llamacpp": "b10991", "vllm": "0.10.2"}
+	for i := range n.GPUs {
+		n.GPUs[i].ComputeCapability = "7.0"
+	}
+	a, _ := setup(t, testModels(), n)
+	var nodes []NodeView
+	call(t, a, "GET", "/api/v1/nodes", "", &nodes)
+	if len(nodes) != 1 || nodes[0].EngineVersions["vllm"] != "0.10.2" ||
+		len(nodes[0].ComputeCapabilities) != 1 || nodes[0].ComputeCapabilities[0] != "7.0" {
+		t.Errorf("%+v", nodes)
+	}
+}
+
+func TestANodeSaysWhetherItsFabricsAreUp(t *testing.T) {
+	n := gpuNode("ib1", 2, "ib-a")
+	if got := multiNodeReadiness(n); !strings.HasPrefix(got, "unknown") {
+		t.Errorf("no port yet: %q", got)
+	}
+	n.IBPorts = []inventory.IBPort{{HCA: "mlx5_0", State: "DOWN"}}
+	if got := multiNodeReadiness(n); got != "down: mlx5_0" {
+		t.Errorf("down: %q", got)
+	}
+	n.IBPorts[0].State = "ACTIVE"
+	if got := multiNodeReadiness(n); got != "ready" {
+		t.Errorf("up: %q", got)
+	}
+	if got := multiNodeReadiness(gpuNode("plain", 1)); got != "" {
+		t.Errorf("no fabric: %q", got)
+	}
+}

internal/control/adminapi/notify.go

@@ -12,7 +12,7 @@ func (a *API) SetNotifier(n *notify.Notifier) { a.notifier = n }
 // notifyTest sends a test event to every webhook and says how each answered.
 func (a *API) notifyTest(w http.ResponseWriter, r *http.Request) {
 	if a.notifier == nil {
-		writeError(w, http.StatusNotFound, "no webhooks: server.yaml has no notify.webhooks")
+		writeError(w, http.StatusNotFound, "no webhooks: the cluster's settings have no notify.webhooks (llmgw ctl config set notify.webhooks ...)")
 		return
 	}
 	writeJSON(w, http.StatusOK, a.notifier.Test(r.Context()))

internal/control/adminapi/routes.go

@@ -131,7 +131,7 @@ var Routes = []Route{
 		Description: "With the defaults the catalog filled in. By its name only, not an alias.",
 		Response:    catalog.Model{}},
 	{Method: "DELETE", Path: "/api/v1/models/{name}", Summary: "Remove a model added at runtime",
-		Description: "A model declared in configuration is refused: it would come back on the next restart.",
+		Description: "A model declared in the cluster's settings is refused: it is changed there.",
 		Response:    Removed{}},
 	{Method: "POST", Path: "/api/v1/models/{name}/scale", Summary: "Set how many replicas a model runs",
 		Request: ScaleRequest{}, Response: ScaleResponse{}},
@@ -254,11 +254,31 @@ var Routes = []Route{
 			{Name: "since", Description: "How far back: 90m, 24h, 7d; 24h when absent."},
 		},
 		Response: GuardEventsView{}},
+	{Method: "GET", Path: "/api/v1/guard/judges", Summary: "The judge models the policies ask, and what is wrong with each",
+		Response: []JudgeView{}},
+	{Method: "POST", Path: "/api/v1/guard/judges/ask", Summary: "Have a judge read a text and say what it answered; nothing is kept",
+		Request: JudgeAskRequest{}, Response: JudgeAskView{}},
 	{Method: "POST", Path: "/api/v1/guard/attachments", Summary: "Attach a guardrail policy",
 		Request: GuardAttachmentRequest{}, Response: guard.Attachment{}},
 	{Method: "DELETE", Path: "/api/v1/guard/attachments", Summary: "Detach a guardrail policy",
 		Request: GuardAttachmentRequest{}, Response: guard.Attachment{}},
 
+	{Method: "POST", Path: "/api/v1/agents/restart", Summary: "Have every node's agent start again; their engines keep serving",
+		Response: AgentRestartView{}},
+
+	// The cluster's settings: everything in server.yaml that is not one
+	// machine's own, the same on every control plane.
+	{Method: "GET", Path: "/api/v1/settings", Summary: "The cluster's settings in force, or one earlier version",
+		Query:    []Param{{Name: "version", Description: "An earlier version; the newest when absent."}},
+		Response: SettingsView{}},
+	{Method: "GET", Path: "/api/v1/settings/history", Summary: "Versions of the cluster's settings, newest first",
+		Query:    []Param{{Name: "limit", Description: "How many; 20 when absent."}},
+		Response: []SettingsView{}},
+	{Method: "PUT", Path: "/api/v1/settings", Summary: "Change the cluster's settings: in force on every control plane within seconds",
+		Request: SettingsRequest{}, Response: SettingsView{}},
+	{Method: "POST", Path: "/api/v1/settings/rollback", Summary: "Put an earlier version of the cluster's settings back in force",
+		Request: SettingsRollbackRequest{}, Response: SettingsView{}},
+
 	// Usage, requests and the record of changes.
 	{Method: "GET", Path: "/api/v1/usage", Summary: "What each group of the tenancy tree spent",
 		Query: []Param{

internal/control/adminapi/settings.go

@@ -0,0 +1,179 @@
+package adminapi
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"net/http"
+	"strconv"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// SettingsStore keeps the cluster's settings.
+type SettingsStore interface {
+	ClusterSettings(ctx context.Context) (store.SettingsVersion, error)
+	ClusterSettingsAt(ctx context.Context, version int64) (store.SettingsVersion, error)
+	ClusterSettingsHistory(ctx context.Context, limit int) ([]store.SettingsVersion, error)
+	PutClusterSettings(ctx context.Context, base int64, body, by, note string) (int64, error)
+}
+
+// Settings is what the settings endpoints need: where they are kept, and a
+// check that a document would load — laid over this control plane's own
+// server.yaml, validated as it would be at start.
+type Settings struct {
+	Store SettingsStore
+	Check func(body []byte) error
+	// LocalKeys are those each machine's server.yaml keeps, which the
+	// document cannot set.
+	LocalKeys []string
+}
+
+// SetSettings enables the settings endpoints.
+func (a *API) SetSettings(s *Settings) { a.settings = s }
+
+// SettingsView is one version of the cluster's settings.
+type SettingsView struct {
+	Version   int64     `json:"version"`
+	Body      string    `json:"body"`
+	Note      string    `json:"note,omitempty"`
+	At        time.Time `json:"at"`
+	By        string    `json:"by,omitempty"`
+	LocalKeys []string  `json:"local_keys,omitempty"`
+}
+
+// SettingsRequest is a new version of the cluster's settings, made from the
+// version Base, which must still be the newest.
+type SettingsRequest struct {
+	Base int64  `json:"base"`
+	Body string `json:"body"`
+	Note string `json:"note,omitempty"`
+}
+
+// SettingsRollbackRequest puts an earlier version's settings back in force,
+// as a new version.
+type SettingsRollbackRequest struct {
+	Version int64  `json:"version"`
+	Note    string `json:"note,omitempty"`
+}
+
+func settingsView(v store.SettingsVersion) SettingsView {
+	return SettingsView{Version: v.Version, Body: v.Body, Note: v.Note, At: v.At, By: v.By}
+}
+
+func (a *API) registerSettings(mux Mux) {
+	need := func(w http.ResponseWriter) bool {
+		if a.settings == nil {
+			writeError(w, http.StatusNotImplemented, "this control plane has no store to keep the cluster's settings in")
+			return false
+		}
+		return true
+	}
+	mux.HandleFunc("GET /api/v1/settings", func(w http.ResponseWriter, r *http.Request) {
+		if !need(w) {
+			return
+		}
+		var v store.SettingsVersion
+		var err error
+		if q := r.URL.Query().Get("version"); q != "" {
+			n, perr := strconv.ParseInt(q, 10, 64)
+			if perr != nil {
+				writeError(w, http.StatusBadRequest, "version is a number")
+				return
+			}
+			v, err = a.settings.Store.ClusterSettingsAt(r.Context(), n)
+			if err == nil && v.Version == 0 {
+				writeError(w, http.StatusNotFound, "there is no such version")
+				return
+			}
+		} else {
+			v, err = a.settings.Store.ClusterSettings(r.Context())
+		}
+		if err != nil {
+			writeError(w, http.StatusInternalServerError, err.Error())
+			return
+		}
+		out := settingsView(v)
+		out.LocalKeys = a.settings.LocalKeys
+		writeJSON(w, http.StatusOK, out)
+	})
+	mux.HandleFunc("GET /api/v1/settings/history", func(w http.ResponseWriter, r *http.Request) {
+		if !need(w) {
+			return
+		}
+		limit := 20
+		if q := r.URL.Query().Get("limit"); q != "" {
+			if n, err := strconv.Atoi(q); err == nil && n > 0 && n <= 500 {
+				limit = n
+			}
+		}
+		list, err := a.settings.Store.ClusterSettingsHistory(r.Context(), limit)
+		if err != nil {
+			writeError(w, http.StatusInternalServerError, err.Error())
+			return
+		}
+		out := make([]SettingsView, len(list))
+		for i, v := range list {
+			out[i] = settingsView(v)
+		}
+		writeJSON(w, http.StatusOK, out)
+	})
+	put := func(w http.ResponseWriter, r *http.Request, base int64, body, note string) {
+		if err := a.settings.Check([]byte(body)); err != nil {
+			writeError(w, http.StatusBadRequest, err.Error())
+			return
+		}
+		n, err := a.settings.Store.PutClusterSettings(r.Context(), base, body, actorOf(r), note)
+		switch {
+		case errors.Is(err, store.ErrSettingsChanged):
+			writeError(w, http.StatusConflict, err.Error())
+			return
+		case err != nil:
+			writeError(w, http.StatusInternalServerError, err.Error())
+			return
+		}
+		v, err := a.settings.Store.ClusterSettingsAt(r.Context(), n)
+		if err != nil {
+			writeError(w, http.StatusInternalServerError, err.Error())
+			return
+		}
+		writeJSON(w, http.StatusOK, settingsView(v))
+	}
+	mux.HandleFunc("PUT /api/v1/settings", func(w http.ResponseWriter, r *http.Request) {
+		if !need(w) {
+			return
+		}
+		var req SettingsRequest
+		if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
+			writeError(w, http.StatusBadRequest, err.Error())
+			return
+		}
+		put(w, r, req.Base, req.Body, req.Note)
+	})
+	mux.HandleFunc("POST /api/v1/settings/rollback", func(w http.ResponseWriter, r *http.Request) {
+		if !need(w) {
+			return
+		}
+		var req SettingsRollbackRequest
+		if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
+			writeError(w, http.StatusBadRequest, err.Error())
+			return
+		}
+		old, err := a.settings.Store.ClusterSettingsAt(r.Context(), req.Version)
+		if err == nil && old.Version == 0 {
+			writeError(w, http.StatusNotFound, "there is no such version")
+			return
+		}
+		cur, err2 := a.settings.Store.ClusterSettings(r.Context())
+		if err = errors.Join(err, err2); err != nil {
+			writeError(w, http.StatusInternalServerError, err.Error())
+			return
+		}
+		note := req.Note
+		if note == "" {
+			note = "back to version " + strconv.FormatInt(req.Version, 10)
+		}
+		put(w, r, cur.Version, old.Body, note)
+	})
+}

internal/control/adminapi/settings_test.go

@@ -0,0 +1,78 @@
+package adminapi
+
+import (
+	"context"
+	"errors"
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+type memSettings struct{ versions []store.SettingsVersion }
+
+func (m *memSettings) ClusterSettings(context.Context) (store.SettingsVersion, error) {
+	if len(m.versions) == 0 {
+		return store.SettingsVersion{}, nil
+	}
+	return m.versions[len(m.versions)-1], nil
+}
+
+func (m *memSettings) ClusterSettingsAt(_ context.Context, v int64) (store.SettingsVersion, error) {
+	if v < 1 || int(v) > len(m.versions) {
+		return store.SettingsVersion{}, nil
+	}
+	return m.versions[v-1], nil
+}
+
+func (m *memSettings) ClusterSettingsHistory(context.Context, int) ([]store.SettingsVersion, error) {
+	var out []store.SettingsVersion
+	for i := len(m.versions) - 1; i >= 0; i-- {
+		out = append(out, m.versions[i])
+	}
+	return out, nil
+}
+
+func (m *memSettings) PutClusterSettings(_ context.Context, base int64, body, by, note string) (int64, error) {
+	if base != int64(len(m.versions)) {
+		return 0, store.ErrSettingsChanged
+	}
+	m.versions = append(m.versions, store.SettingsVersion{Version: base + 1, Body: body, By: by, Note: note, At: time.Now()})
+	return base + 1, nil
+}
+
+func TestSettingsAreCheckedKeptAndRolledBack(t *testing.T) {
+	a, _ := setup(t, testModels())
+	st := &memSettings{}
+	a.SetSettings(&Settings{Store: st, Check: func(b []byte) error {
+		if strings.Contains(string(b), "listen") {
+			return errors.New("listen is each machine's")
+		}
+		return nil
+	}})
+	var v SettingsView
+	if code := call(t, a, "PUT", "/api/v1/settings", `{"base":0,"body":"router:\n  retries: 2\n"}`, &v); code != 200 || v.Version != 1 {
+		t.Fatalf("first: %d %+v", code, v)
+	}
+	if code := call(t, a, "PUT", "/api/v1/settings", `{"base":1,"body":"listen: :1\n"}`, nil); code != 400 {
+		t.Errorf("a document the check refuses: %d", code)
+	}
+	if code := call(t, a, "PUT", "/api/v1/settings", `{"base":0,"body":"router: {}\n"}`, nil); code != 409 {
+		t.Errorf("a change from an old version: %d", code)
+	}
+	if code := call(t, a, "PUT", "/api/v1/settings", `{"base":1,"body":"router:\n  retries: 5\n"}`, nil); code != 200 {
+		t.Fatalf("second: %d", code)
+	}
+	if code := call(t, a, "POST", "/api/v1/settings/rollback", `{"version":1}`, &v); code != 200 ||
+		v.Version != 3 || v.Body != "router:\n  retries: 2\n" || v.Note != "back to version 1" {
+		t.Errorf("rollback: %d %+v", code, v)
+	}
+	var h []SettingsView
+	if code := call(t, a, "GET", "/api/v1/settings/history", "", &h); code != 200 || len(h) != 3 || h[0].Version != 3 {
+		t.Errorf("history: %d %+v", code, h)
+	}
+	if code := call(t, a, "GET", "/api/v1/settings?version=9", "", nil); code != 404 {
+		t.Errorf("a version that is not there: %d", code)
+	}
+}

internal/control/adminapi/summary.go

@@ -1,6 +1,7 @@
 package adminapi
 
 import (
+	"gitea.mynodes.xyz/ilker/llmgw/internal/version"
 	"sort"
 	"time"
 )
@@ -14,8 +15,11 @@ import (
 // derived from the same state the tables show, so it can never disagree with
 // them; it is a different question asked of the same facts.
 type Summary struct {
-	Nodes NodeTotals `json:"nodes"`
-	GPUs  []ClassUse `json:"gpus"`
+	// Version is the release the answering control plane runs: not the
+	// binary on its disk, which a manual install changes underneath it.
+	Version string     `json:"version,omitempty"`
+	Nodes   NodeTotals `json:"nodes"`
+	GPUs    []ClassUse `json:"gpus"`
 	// Fabrics matter because a multi-node replica may never span two of them,
 	// so free capacity in the wrong fabric is not capacity for the model that
 	// needs it.
@@ -38,6 +42,14 @@ type Summary struct {
 	// ControlPlanes is the cluster's control planes when there is more than
 	// one: which leads, which stand by, and the floating address in front.
 	ControlPlanes ControlPlanes `json:"control_planes,omitzero"`
+	// Judges are the guardrails' judge models with something wrong.
+	Judges []JudgeView `json:"judges,omitempty"`
+	// Stopped says the operators have stopped the service (llmgw stop): no
+	// model runs, and requests for one are answered 503.
+	Stopped bool `json:"stopped,omitempty"`
+	// GuardOff says the guardrails are turned off (guard.enabled: false):
+	// no request is checked, whatever the policies say.
+	GuardOff bool `json:"guard_off,omitempty"`
 }
 
 // StoreMember is one member of a replicated store, as its Patroni reports it.
@@ -144,7 +156,15 @@ const idleThreshold = 10 * time.Minute
 
 // Summary builds the fleet summary.
 func (a *API) Summary() Summary {
-	var s Summary
+	s := Summary{Version: version.Version, Stopped: a.ctl.Reconciler().Stopped()}
+	if a.guard != nil && a.guard.Library != nil && !a.guard.Library().On() {
+		s.GuardOff = true
+	}
+	for _, j := range a.Judges() {
+		if j.Problem != "" {
+			s.Judges = append(s.Judges, j)
+		}
+	}
 	if a.storeStatus != nil {
 		members, err := a.storeStatus()
 		s.Store = members

internal/control/adminapi/system.go

@@ -0,0 +1,22 @@
+package adminapi
+
+import (
+	"net/http"
+)
+
+// AgentRestartView names the nodes whose agents were told to restart.
+type AgentRestartView struct {
+	Nodes []string `json:"nodes"`
+}
+
+// SetAgentRestarter enables POST /api/v1/agents/restart.
+func (a *API) SetAgentRestarter(f func(reason string) []string) { a.restartAgents = f }
+
+func (a *API) restartAgentsHandler(w http.ResponseWriter, r *http.Request) {
+	if a.restartAgents == nil {
+		writeError(w, http.StatusNotImplemented, "this control plane has no agents to restart")
+		return
+	}
+	nodes := a.restartAgents("llmgw restart, by " + actorOf(r))
+	writeJSON(w, http.StatusAccepted, AgentRestartView{Nodes: nodes})
+}

internal/control/agentsvc/convert.go

@@ -169,6 +169,10 @@ func AdoptedFromRegister(nodeID string, shards []*llmgwv1.RunningShard) []reconc
 			Fabric:     s.GetFabric(),
 			ProfileFP:  s.GetProfileFp(),
 		}
+		if c := s.GetCompute(); c != nil {
+			a.Compute = &reconciler.Compute{Backend: c.GetBackend(), Devices: c.GetDevices(),
+				Offloaded: int(c.GetLayersOffloaded()), Total: int(c.GetLayersTotal())}
+		}
 		if a.Rank == 0 && s.GetLeaderAddr() != "" {
 			a.Endpoint = fmt.Sprintf("http://%s:%d", s.GetLeaderAddr(), s.GetHttpPort())
 		}

internal/control/agentsvc/firewall.go

@@ -0,0 +1,61 @@
+package agentsvc
+
+import (
+	"time"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+)
+
+// SetFirewall gives the service whether nodes keep llmgw's firewall, read
+// at each use so a change to the cluster's settings is what is sent next.
+func (s *Service) SetFirewall(f func() *llmgwv1.Firewall) {
+	s.mu.Lock()
+	s.firewall = f
+	s.mu.Unlock()
+}
+
+func (s *Service) firewallMsg() *llmgwv1.ControlMessage {
+	s.mu.RLock()
+	f := s.firewall
+	s.mu.RUnlock()
+	if f == nil {
+		return nil
+	}
+	fw := f()
+	if fw == nil {
+		return nil
+	}
+	return &llmgwv1.ControlMessage{Msg: &llmgwv1.ControlMessage_Firewall{Firewall: fw}}
+}
+
+// BroadcastFirewall tells every connected agent what the cluster's settings
+// now say of the firewall. A node not connected is told when it registers.
+func (s *Service) BroadcastFirewall() {
+	msg := s.firewallMsg()
+	if msg == nil {
+		return
+	}
+	for _, node := range s.ConnectedNodes() {
+		if err := s.send(node, msg); err != nil {
+			s.log.Warn("could not tell a node about the firewall; it is told when it next registers",
+				"node", node, "err", err)
+		}
+	}
+}
+
+// RestartAgents has every connected agent start again, one at a time a
+// moment apart, and names those it told. Their engines keep serving.
+func (s *Service) RestartAgents(reason string) []string {
+	msg := &llmgwv1.ControlMessage{Msg: &llmgwv1.ControlMessage_RestartAgent{
+		RestartAgent: &llmgwv1.RestartAgent{Reason: reason}}}
+	var told []string
+	for _, node := range s.ConnectedNodes() {
+		if err := s.send(node, msg); err != nil {
+			s.log.Warn("could not tell a node to restart its agent", "node", node, "err", err)
+			continue
+		}
+		told = append(told, node)
+		time.Sleep(time.Second)
+	}
+	return told
+}

internal/control/agentsvc/service.go

@@ -36,6 +36,8 @@ type StatusSink interface {
 	// still starting, and treating each report as a state change would reset
 	// the timers that decide when a slow start has become a failed one.
 	OnFetchProgress(replicaID, shardID, source string, done, total int64)
+	// OnShardCompute records where a shard's engine said it runs the model.
+	OnShardCompute(replicaID, shardID string, c reconciler.Compute)
 	Adopt(nodeID string, shards []reconciler.AdoptedShard) (unclaimed []reconciler.AdoptedShard)
 }
 
@@ -75,6 +77,8 @@ type Service struct {
 
 	// controlPlanes lists what agents may dial; see SetControlPlanes.
 	controlPlanes func(context.Context) ([]string, error)
+	// firewall is what nodes are told of llmgw's firewall; see SetFirewall.
+	firewall func() *llmgwv1.Firewall
 
 	// logWaits are the log requests awaiting a reply, by request id.
 	logMu    sync.Mutex
@@ -286,6 +290,12 @@ func (s *Service) Connect(
 			return err
 		}
 	}
+	// Whether to keep llmgw's firewall, as the cluster's settings say now.
+	if msg := s.firewallMsg(); msg != nil {
+		if err := stream.Send(msg); err != nil {
+			return err
+		}
+	}
 
 	// Recover replica state from what this node is already running, before any
 	// reconciliation pass can conclude the cluster is empty and place
@@ -439,6 +449,16 @@ func (s *Service) handle(nodeID string, msg *llmgwv1.AgentMessage) {
 				f.GetSource(), f.GetBytesDone(), f.GetBytesTotal())
 			return
 		}
+		if c := m.Shard.GetCompute(); c != nil {
+			s.status.OnShardCompute(m.Shard.GetReplicaId(), m.Shard.GetShardId(), reconciler.Compute{
+				Backend: c.GetBackend(), Devices: c.GetDevices(),
+				Offloaded: int(c.GetLayersOffloaded()), Total: int(c.GetLayersTotal()),
+			})
+			// Compute alone, with no state: not a transition.
+			if m.Shard.GetState() == llmgwv1.ShardStatus_STATE_UNSPECIFIED {
+				return
+			}
+		}
 		st := StateFromWire(m.Shard.GetState())
 		s.status.OnShardStatus(m.Shard.GetReplicaId(), m.Shard.GetShardId(), st, m.Shard.GetError())
 		if m.Shard.GetState() == llmgwv1.ShardStatus_STATE_FAILED {

internal/control/catalog/catalog.go

@@ -16,6 +16,7 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/facility"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
 )
 
 // Scaling is how many replicas a model should have.
@@ -164,9 +165,13 @@ type SLO struct {
 
 // Profile is how a model runs on one GPU class.
 type Profile struct {
-	Name     string `json:"name,omitempty" koanf:"name"`
-	Engine   string `json:"engine,omitempty" koanf:"engine"`
-	GPUClass string `json:"gpu_class,omitempty" koanf:"gpu_class"`
+	Name   string `json:"name,omitempty" koanf:"name"`
+	Engine string `json:"engine,omitempty" koanf:"engine"`
+	// EngineVersion restricts the profile to nodes whose installed engine's
+	// version starts with it: "0.11" takes vLLM 0.11.0 and 0.11.1, not
+	// 0.10.2. Empty takes whatever version the node has.
+	EngineVersion string `json:"engine_version,omitempty" koanf:"engine_version"`
+	GPUClass      string `json:"gpu_class,omitempty" koanf:"gpu_class"`
 
 	TP int `json:"tp,omitempty" koanf:"tp"`
 	PP int `json:"pp,omitempty" koanf:"pp"`
@@ -382,9 +387,11 @@ func (m Model) IsGuard() bool { return m.Role == RoleGuard }
 
 // GuardModel is how a judge model's answers are read.
 type GuardModel struct {
-	// Format is llamaguard3 (Meta's Llama Guard 3: "safe", or "unsafe" and
-	// the categories), shieldgemma (Google's ShieldGemma: Yes or No), or
-	// json ({"safe": false, "categories": ["..."]}).
+	// Format is qwen3guard (Qwen3Guard-Gen: "Safety: Safe", "Unsafe" or
+	// "Controversial", and the categories), llamaguard3 (Meta's Llama Guard
+	// 3: "safe", or "unsafe" and the categories), shieldgemma (Google's
+	// ShieldGemma: Yes or No), or json ({"safe": false, "categories":
+	// ["..."]}).
 	Format string `json:"format" koanf:"format"`
 	// Categories the model is asked about, for shieldgemma — each a policy
 	// sentence — and the names reported for llamaguard3's S1, S2, ….
@@ -704,12 +711,10 @@ func (m Model) validate() error {
 		}
 	case RoleGuard:
 		if m.Guard == nil {
-			return fmt.Errorf("model %q: a guard model needs guard.format: llamaguard3, shieldgemma or json", m.Name)
+			return fmt.Errorf("model %q: a guard model needs guard.format: %s", m.Name, strings.Join(guard.Formats, ", "))
 		}
-		switch m.Guard.Format {
-		case "llamaguard3", "shieldgemma", "json":
-		default:
-			return fmt.Errorf("model %q: guard.format %q is not llamaguard3, shieldgemma or json", m.Name, m.Guard.Format)
+		if !guard.KnownFormat(m.Guard.Format) {
+			return fmt.Errorf("model %q: guard.format %q is not one of %s", m.Name, m.Guard.Format, strings.Join(guard.Formats, ", "))
 		}
 		if m.Scaling.OnDemand() {
 			// A judge loaded on demand makes the first request of every
@@ -885,18 +890,19 @@ func (m Model) PlacementProfiles() []placement.Profile {
 	out := make([]placement.Profile, 0, len(m.Profiles))
 	for _, p := range m.Profiles {
 		out = append(out, placement.Profile{
-			Name:         p.Name,
-			Engine:       p.Engine,
-			GPUClass:     p.GPUClass,
-			GPUsPerNode:  p.GPUsPerNode,
-			Nodes:        p.Nodes,
-			Priority:     p.Priority,
-			NodeSelector: p.NodeSelector,
-			MinMemoryGB:  p.MinMemoryGB,
-			MinDiskGB:    p.MinDiskGB,
-			CPUCores:     p.CPUCores,
-			PowerWatts:   p.PowerWatts,
-			KeepWithin:   facility.Level(strings.ToLower(strings.TrimSpace(p.KeepWithin))),
+			Name:          p.Name,
+			Engine:        p.Engine,
+			EngineVersion: p.EngineVersion,
+			GPUClass:      p.GPUClass,
+			GPUsPerNode:   p.GPUsPerNode,
+			Nodes:         p.Nodes,
+			Priority:      p.Priority,
+			NodeSelector:  p.NodeSelector,
+			MinMemoryGB:   p.MinMemoryGB,
+			MinDiskGB:     p.MinDiskGB,
+			CPUCores:      p.CPUCores,
+			PowerWatts:    p.PowerWatts,
+			KeepWithin:    facility.Level(strings.ToLower(strings.TrimSpace(p.KeepWithin))),
 			// The model's source, so placement can prefer nodes that already
 			// hold the weights rather than one that must first move them.
 			Source: m.Source,

internal/control/enginechoice.go

@@ -43,6 +43,39 @@ func NewEngineChoices(ctx context.Context, st EngineChoiceStore, log *slog.Logge
 	return c
 }
 
+// Reload reads the choices again, as a standby becoming the leader does, and
+// tells each node whose choice changed.
+func (c *EngineChoices) Reload(ctx context.Context) {
+	got, err := c.st.NodeEngines(ctx)
+	if err != nil {
+		c.log.Error("could not read the engine builds chosen per node again; keeping the ones read before", "err", err)
+		return
+	}
+	c.mu.Lock()
+	var changed []string
+	for node := range got {
+		if !maps.Equal(got[node], c.byNode[node]) {
+			changed = append(changed, node)
+		}
+	}
+	for node := range c.byNode {
+		if _, ok := got[node]; !ok {
+			changed = append(changed, node)
+		}
+	}
+	c.byNode = got
+	notify := c.notify
+	c.mu.Unlock()
+	if notify == nil {
+		return
+	}
+	for _, node := range changed {
+		if err := notify(node); err != nil {
+			c.log.Warn("could not tell a node its engine builds", "node", node, "err", err)
+		}
+	}
+}
+
 // OnChange is called with a node whose choice changed, to tell it.
 func (c *EngineChoices) OnChange(f func(node string) error) {
 	c.mu.Lock()

internal/control/facility.go

@@ -68,6 +68,23 @@ func (f *Facility) UseStore(ctx context.Context, st FacilityStore) {
 	f.mu.Lock()
 	defer f.mu.Unlock()
 	f.st = st
+	f.loadLocked(ctx)
+}
+
+// Reload reads the holds and labels in the store again, as a standby
+// becoming the leader does.
+func (f *Facility) Reload(ctx context.Context) {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	if f.st == nil {
+		return
+	}
+	f.holds = map[facility.Domain]facility.Hold{}
+	f.loadLocked(ctx)
+}
+
+func (f *Facility) loadLocked(ctx context.Context) {
+	st := f.st
 	if holds, err := st.FacilityHolds(ctx); err != nil {
 		f.log.Error("could not read the domains held for maintenance; none is held until they are set again", "err", err)
 	} else {

internal/control/leadership.go

@@ -7,7 +7,9 @@ import (
 	"errors"
 	"log/slog"
 	"os"
+	"slices"
 	"strings"
+	"sync"
 	"sync/atomic"
 	"time"
 
@@ -54,6 +56,27 @@ type Leadership struct {
 	leader atomic.Bool
 	// stopped is set by StepDown: nothing takes the lease again after it.
 	stopped atomic.Bool
+
+	hookMu sync.Mutex
+	onLead []func()
+}
+
+// OnLead is called each time this process becomes the leader, after
+// standing by: what it read at start may have been changed since by the
+// leader before it, and is read again.
+func (l *Leadership) OnLead(f func()) {
+	l.hookMu.Lock()
+	l.onLead = append(l.onLead, f)
+	l.hookMu.Unlock()
+}
+
+func (l *Leadership) becameLeader() {
+	l.hookMu.Lock()
+	hooks := slices.Clone(l.onLead)
+	l.hookMu.Unlock()
+	for _, f := range hooks {
+		go f()
+	}
 }
 
 // NewLeadership prepares leadership over a store.
@@ -133,6 +156,7 @@ func (l *Leadership) attempt(ctx context.Context) {
 	case err == nil:
 		if l.leader.CompareAndSwap(false, true) {
 			l.log.Info("this control plane is now placing work", "owner", l.owner)
+			l.becameLeader()
 		}
 	case errors.Is(err, store.ErrNotLeader):
 		if l.leader.CompareAndSwap(true, false) {

internal/control/leadership_test.go

@@ -49,3 +49,29 @@ func TestAStoppingLeaderHandsOverAtOnceAndNeverTakesItBack(t *testing.T) {
 		t.Errorf("holder name %q from %q", got, b.Owner())
 	}
 }
+
+func TestAStandbyIsToldWhenItBecomesTheLeader(t *testing.T) {
+	st, err := store.Open(t.Context(), filepath.Join(t.TempDir(), "s.db"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer st.Close()
+	log := slog.New(slog.DiscardHandler)
+	a := NewLeadership(st, time.Minute, log)
+	b := NewLeadership(st, time.Minute, log)
+	led := make(chan struct{}, 4)
+	b.OnLead(func() { led <- struct{}{} })
+	ctx := t.Context()
+	a.attempt(ctx)
+	b.attempt(ctx)
+	if !a.IsLeader() || b.IsLeader() {
+		t.Fatal("a should lead, b stand by")
+	}
+	a.StepDown()
+	b.attempt(ctx)
+	select {
+	case <-led:
+	case <-time.After(time.Second):
+		t.Fatal("b became the leader and was not told")
+	}
+}