v0.12.2

Guardrails off and on; a judge's refusal names the hazard; ctl guard judge ask.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

BACKLOG.md

@@ -761,6 +761,63 @@ portal is not done — the page shows; the console and ctl change.
 
 ---
 
+## E. Every setting takes effect when it is changed (2026-10-05)
+
+Asked for after a site visit: an upgrade left server.yaml without guardrails,
+a node on another subnet stayed lost until firewall_networks was edited by
+hand, and nothing in server.yaml, agent.yaml or a generated unit applies until
+somebody restarts the right service on every machine. The rule from here on:
+**a file holds only what one machine needs to start; everything else is the
+cluster's, kept in the store, changed through ctl, the console or the web,
+audited, and in force on every control plane within seconds.**
+
+### E1. Cluster settings in the store (L) — done 2026-10-05
+Live on the PoC (140, 150): imported as version 1; a guard change applied on
+both without a restart; a router change restarted the standby at once and the
+leader 30s later with /health answering throughout; rollback did the same.
+- Table `cluster_settings` (versioned YAML document, who and why). The
+  cluster-wide part of server.yaml lives there; server.yaml keeps the
+  per-machine keys only (listeners, TLS paths, store DSN, secrets, tracing,
+  monitoring, mirror.api/tls, backup_dir/key, endpoints).
+- First start against a store with no document imports the file's cluster
+  keys (and the baseline guard policy when there is none); after that those
+  keys in a file are ignored, with a warning.
+- Each control plane polls the document's version and its own server.yaml's
+  hash. A change is validated, then applied: `guard` live, the rest by a
+  graceful restart of the process (standbys first, the leader last), which
+  systemd brings back. No "restart now?" anywhere.
+- `llmgw ctl config show|history|get|set|unset|edit|rollback`; the console's
+  settings screen edits the store for cluster keys; admin API
+  `GET/PUT /api/v1/settings`, `/history`, `/rollback`.
+- A standby that becomes leader re-reads the catalog, engine choices and
+  facility state from the store (it kept what it read at start).
+- Retires `appendGuardIfMissing` (server init): the import does it.
+
+### E2. Generated files from settings (L) — firewall on/off done 2026-10-06
+`firewall.enabled` in the cluster's settings reaches nodes over the agent
+stream and control planes through llmgw-firewall-sync.path. Left: the mirror
+machine and store members, the allowed set from membership, the other files.
+
+Every machine runs the agent (root) with roles — node, control plane, mirror,
+store member — and the agent renders and applies what was written once by
+init: the llmgw nft table, the mirror's unit flags, keepalived, etcd and
+Patroni configuration. The firewall's allowed set is the cluster's membership
+(every node, control plane, mirror and store member) plus
+`firewall.extra` from the store; 7443 stays open (mTLS). A node added or
+removed changes every machine's rules. Retires `firewall.Admit` and
+`firewall_networks` in server.yaml (imported into the store).
+
+### E3. agent.yaml down to bootstrap (M)
+agent.yaml keeps the control planes' addresses, its certificate and its id.
+Labels, fabrics, engines, heartbeat and supervisor settings move to the
+store per node and reach the agent over its stream; a change to the file is
+noticed and the agent restarts itself.
+
+### E4. Web settings page (S)
+The same document, edited on /admin/settings, for whoever has no terminal.
+
+---
+
 ## D. Facts worth knowing before starting
 
 - Admin API: `127.0.0.1:9090`, no auth, `ctlClient` in `internal/cli/ctl.go`.

CHANGELOG.md

@@ -16,12 +16,134 @@ is only released once its **exit criterion** is met, not when its code lands.
 | v0.9.0 | — | Managing people and the tenancy tree | released |
 | v0.10.0 | — | InfiniBand fabrics and NVLink, doctor, node add, engines installed by the nodes | released |
 | v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released |
+| v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released |
 | v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
 
 Until v1.0.0 the minor version is the breaking one: configuration formats and Go
 APIs change between minors without a deprecation period.
 
-## v0.11.3-rc.1 — 2026-10-05
+## v0.12.2 — 2026-10-06
+
+The guardrails turned off and on as a whole, and a judge that refuses
+ordinary requests explained: the hazard it saw is told to the client and
+kept with the request, and `llmgw ctl guard judge ask` shows what it says of
+any text. Nothing to do to upgrade; the store gains a column.
+
+### Added
+
+- **The guardrails turn off and on as a whole**: `llmgw ctl guard off` /
+  `on`, which is `guard.enabled` in the cluster's settings — in force within
+  seconds, the policies and their attachments kept. `ctl status`, `ctl guard
+  list` and the guardrails page say when they are off.
+- **`llmgw ctl guard judge ask TEXT`** has the judge read a text and shows
+  its answer as given and as read — for finding out why ordinary requests
+  are refused. Nothing of the text is kept, not even in the audit.
+
+### Changed
+
+- **A judge's refusal names the hazard**: the client is told `(judge:
+  S6,Specialized Advice)` rather than `(judge)`, and the verdict is kept with
+  the request (`guard_rule`): `ctl request`, the server's log and the
+  guardrails page show it. Llama Guard 3's codes come with their names,
+  S9 Indiscriminate Weapons, without a categories list.
+
+## v0.12.1 — 2026-10-06
+
+The guardrails' judge, served and watched without server init: a judge
+missing, served as the wrong kind of model, or not answering is said on
+`ctl status` and the console, and `llmgw ctl guard judge enable` puts it
+right. Nothing to do to upgrade.
+
+### Added
+
+- **`llmgw ctl guard judge status` and `enable`**: each judge model the
+  policies ask, whether it is in the catalog, as a judge, with replicas
+  ready and answering — and serving the missing one, Llama Guard 3 1B on
+  CPUs under the name the policies give it, with `--replace` for a model of
+  that name that is not a judge. No more `server init` to put a judge back.
+  Also **Do → Serve the guardrails' judge** in the console.
+- **A missing or broken judge is said where it is looked for**: `llmgw ctl
+  status` and the console's Overview list each judge with a problem — not in
+  the catalog, not a judge, no replica ready, or failing to answer — and
+  what that does to requests (refused or unjudged).
+
+### Fixed
+
+- **A Llama Guard file served from the mirror page is the judge**: it was
+  added as a model clients use, under whatever name the form had, so the
+  policies never found it and refused everything they judged
+  (`guard_unavailable`). It is now added as a judge, under the name the
+  policies ask.
+- **`ctl` says so when the API answers with a page**: a command a control
+  plane does not know — the leader on an older release, through a standby —
+  failed with "invalid character '<'"; it now says which and why.
+
+## v0.12.0 — 2026-10-06
+
+A minor version, because where the configuration lives changes: everything in
+server.yaml that is not one machine's own is now the cluster's settings, kept
+in the store, the same on every control plane and in force within seconds of
+a change. Nothing needs doing by hand to upgrade. The first control plane to
+start on v0.12.0 brings its server.yaml's cluster keys into the store, with
+the baseline guard policy when the file had none. From then on, change them
+with `llmgw ctl config` or the console's Settings, not in server.yaml, whose
+cluster keys are ignored, with a warning. Upgrade every control plane (`llmgw
+ctl upgrade` does, the leader last): until the leader runs v0.12.0,
+`llmgw ctl config` through a standby is refused. Nodes need v0.12.0 to follow
+the firewall switch.
+
+### Changed
+
+- **The cluster's settings live in the store, and every change is in force
+  within seconds.** Everything in server.yaml that is not one machine's own
+  (guardrails, limits and prices, routing, engines, notifications, the
+  mirror's policy, declared models, offline, federation, facility budgets) is
+  a versioned document in the store, the same on every control plane.
+  `llmgw ctl config` shows, sets, edits, lists and rolls it back, and the
+  console's Settings screen edits it. A change is checked, kept with who made
+  it and why, and applied: guardrails live, anything else by every control
+  plane starting again on it, standbys first and the leader last. A change to
+  a machine's own keys in its server.yaml is noticed and applied the same way.
+  No "restart now?" any more, and no second copy to keep in step. The first
+  start against a store brings in the file's cluster keys, with the baseline
+  guard policy when the file had none, so an upgraded cluster gets its
+  guardrails; after that, those keys in a server.yaml are ignored, with a
+  warning.
+
+### Added
+
+- **llmgw's firewall can be turned off and on for the whole cluster**:
+  `llmgw ctl config set firewall.enabled false`. Every node's agent takes its
+  table out within seconds, and each control plane's new
+  `llmgw-firewall-sync.path` does the same there; `true` puts the rules back
+  as they were. `firewall_networks` changes reach the nodes the same way.
+- **`llmgw wipe`** removes llmgw from a machine — services, firewall table,
+  configuration, the authority, secrets, state, users — after listing it and
+  asking for the machine's name; `--models`, `--store` and `--all` take
+  weights, the store's database here, backups and the binary too. Also
+  **Do → Wipe this machine** in the console.
+
+### Fixed
+
+- **The mirror is given an address the nodes reach**: `server init` and
+  `mirror init` took the machine's first address for the mirror's `--base`,
+  which on a machine with Docker could be 172.17.0.1 — a node then asked its
+  own bridge for every model and was refused. They now take the source of
+  the default route, else the first address not on a container's or a
+  bridge's interface. And `node add` has the node reach the mirror's files
+  before it is added: one that cannot fails the checks, naming the address
+  and what to change.
+- **A standby that becomes the leader reads the store again**: models added
+  or removed, engine builds chosen and domains held through the leader before
+  it, rather than placing on what it read when it started.
+- **A node on another network joins**: `node add` set up the node's firewall
+  but left the control plane's to the networks it was installed with, so a node
+  on another subnet was installed and then never heard from (lost). It now lets
+  the node in on the control plane it runs on — its subnet, or its address
+  when a router is between them — and says so; a network already let in
+  changes nothing.
+
+## v0.11.4 — 2026-10-05
 
 Two control planes, run in anger on the reference cluster.
 
@@ -42,9 +164,35 @@ Two control planes, run in anger on the reference cluster.
   come back stops it. The other control plane is upgraded first, over ssh.
 - `server add` says when the mirror's `--api-allow` would refuse the new
   control plane.
+- **Guardrails in the console**: a view of the policies and what each checks,
+  whether their judge models are serving, where they are attached (attach
+  and detach from it), and what they did in a window. `GET
+  /api/v1/guard/events` answers the same, and `GET /api/v1/guard` now says
+  what each policy does.
+
+- **A standby's portal is the leader's.** Its client port passes everything
+  but `/health`, `/ready` and `/leader` to the leader, and its admin port the
+  dashboard as well as the API; a proxy left pointing at a standby showed an
+  empty cluster. The leader trusts the control planes' `X-Forwarded-For`.
+
+- `ctl model serve --tools` says in the catalog that a model makes tool
+  calls, for agents and editors.
 
 ### Fixed
 
+- **Continue's agent did nothing.** The configuration the portal writes told
+  Continue every model uses tools, and nothing of its context: a model whose
+  calls came back as text stalled the agent, and prompts over a CPU
+  replica's 4096 tokens were refused. It now says `tool_use` only for a
+  model whose catalog says tools, and the context and longest answer the
+  gateway allows.
+- `ctl model serve` gave llama.cpp four slots whatever was asked: a
+  `--context-length 32768 --ctx-pool 32768` model had slots of 8192 and
+  refused what the gateway admitted. Slots are now what the pool holds of
+  the context. The longest answer grows with the context (a quarter, from
+  1024 to 8192) rather than being 1024 for every model.
+- `ctl upgrade` upgrades the leader last wherever it is run; run on the
+  standby, it upgraded the leader first and the leadership moved twice.
 - **Failover took 40 seconds; it takes a few.** A stopping control plane held
   every agent's stream for the thirty seconds of a graceful shutdown; it now
   gives up its lease first and lets the agents go at once.
@@ -53,6 +201,7 @@ Two control planes, run in anger on the reference cluster.
 - An agent's reconnect backoff carried over from one outage to the next, so
   each failover made the next one slower; it starts again after a session that
   worked.
+- The `CONTROL` line named no floating address.
 
 ## v0.11.2 — 2026-10-05
 

api/proto/llmgw/v1/agent.proto

@@ -71,6 +71,7 @@ message ControlMessage {
     FabricTestRequest fabric_test = 9;
     UpgradeAgent upgrade = 10;
     ControlPlanes control_planes = 11;
+    Firewall firewall = 13;
   }
   // trace is the W3C trace context (traceparent, and tracestate when there
   // is one) of the control plane's span that sent this command, so what the
@@ -80,6 +81,21 @@ message ControlMessage {
   map<string, string> trace = 12;
 }
 
+// Firewall is whether llmgw's firewall is kept on the node — its nftables
+// table, which lets the cluster's networks alone reach the engines' ports and
+// the node exporter — sent when an agent registers and whenever the
+// cluster's settings change it (firewall.enabled).
+message Firewall {
+  bool enabled = 1;
+  // networks are let in besides those the node reaches the control planes
+  // on: firewall_networks, networks or single addresses.
+  repeated string networks = 2;
+  // engine_ports_from and engine_ports_to are the engines' ports
+  // (control.port_range).
+  int32 engine_ports_from = 3;
+  int32 engine_ports_to = 4;
+}
+
 // ControlPlanes is every control plane an agent may dial, sent by the leader
 // when an agent registers and whenever the list changes. The agent keeps it
 // beside its configuration, so a second control plane added after the node

docs/admin-api.openapi.json

@@ -83,6 +83,9 @@
     {
       "name": "restart"
     },
+    {
+      "name": "settings"
+    },
     {
       "name": "summary"
     },
@@ -854,6 +857,115 @@
         }
       }
     },
+    "/api/v1/guard/events": {
+      "get": {
+        "operationId": "getGuardEvents",
+        "summary": "What the guardrails did: totals and the newest requests",
+        "tags": [
+          "guard"
+        ],
+        "parameters": [
+          {
+            "name": "since",
+            "in": "query",
+            "description": "How far back: 90m, 24h, 7d; 24h when absent.",
+            "schema": {
+              "type": "string"
+            }
+          }
+        ],
+        "responses": {
+          "200": {
+            "description": "OK",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/GuardEventsView"
+                }
+              }
+            }
+          },
+          "401": {
+            "$ref": "#/components/responses/Unauthorized"
+          },
+          "default": {
+            "$ref": "#/components/responses/Error"
+          }
+        }
+      }
+    },
+    "/api/v1/guard/judges": {
+      "get": {
+        "operationId": "getGuardJudges",
+        "summary": "The judge models the policies ask, and what is wrong with each",
+        "tags": [
+          "guard"
+        ],
+        "responses": {
+          "200": {
+            "description": "OK",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "type": "array",
+                  "items": {
+                    "$ref": "#/components/schemas/JudgeView"
+                  }
+                }
+              }
+            }
+          },
+          "401": {
+            "$ref": "#/components/responses/Unauthorized"
+          },
+          "default": {
+            "$ref": "#/components/responses/Error"
+          }
+        }
+      }
+    },
+    "/api/v1/guard/judges/ask": {
+      "post": {
+        "operationId": "postGuardJudgesAsk",
+        "summary": "Have a judge read a text and say what it answered; nothing is kept",
+        "tags": [
+          "guard"
+        ],
+        "parameters": [
+          {
+            "$ref": "#/components/parameters/Actor"
+          }
+        ],
+        "requestBody": {
+          "required": true,
+          "content": {
+            "application/json": {
+              "schema": {
+                "$ref": "#/components/schemas/JudgeAskRequest"
+              }
+            }
+          }
+        },
+        "responses": {
+          "200": {
+            "description": "OK",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/JudgeAskView"
+                }
+              }
+            }
+          },
+          "401": {
+            "$ref": "#/components/responses/Unauthorized"
+          },
+          "default": {
+            "$ref": "#/components/responses/Error"
+          }
+        }
+      }
+    },
     "/api/v1/jobs": {
       "get": {
         "operationId": "getJobs",
@@ -1119,7 +1231,7 @@
       "delete": {
         "operationId": "deleteModelsName",
         "summary": "Remove a model added at runtime",
-        "description": "A model declared in configuration is refused: it would come back on the next restart.",
+        "description": "A model declared in the cluster's settings is refused: it is changed there.",
         "tags": [
           "models"
         ],
@@ -2926,6 +3038,165 @@
         }
       }
     },
+    "/api/v1/settings": {
+      "get": {
+        "operationId": "getSettings",
+        "summary": "The cluster's settings in force, or one earlier version",
+        "tags": [
+          "settings"
+        ],
+        "parameters": [
+          {
+            "name": "version",
+            "in": "query",
+            "description": "An earlier version; the newest when absent.",
+            "schema": {
+              "type": "string"
+            }
+          }
+        ],
+        "responses": {
+          "200": {
+            "description": "OK",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/SettingsView"
+                }
+              }
+            }
+          },
+          "401": {
+            "$ref": "#/components/responses/Unauthorized"
+          },
+          "default": {
+            "$ref": "#/components/responses/Error"
+          }
+        }
+      },
+      "put": {
+        "operationId": "putSettings",
+        "summary": "Change the cluster's settings: in force on every control plane within seconds",
+        "tags": [
+          "settings"
+        ],
+        "parameters": [
+          {
+            "$ref": "#/components/parameters/Actor"
+          }
+        ],
+        "requestBody": {
+          "required": true,
+          "content": {
+            "application/json": {
+              "schema": {
+                "$ref": "#/components/schemas/SettingsRequest"
+              }
+            }
+          }
+        },
+        "responses": {
+          "200": {
+            "description": "OK",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/SettingsView"
+                }
+              }
+            }
+          },
+          "401": {
+            "$ref": "#/components/responses/Unauthorized"
+          },
+          "default": {
+            "$ref": "#/components/responses/Error"
+          }
+        }
+      }
+    },
+    "/api/v1/settings/history": {
+      "get": {
+        "operationId": "getSettingsHistory",
+        "summary": "Versions of the cluster's settings, newest first",
+        "tags": [
+          "settings"
+        ],
+        "parameters": [
+          {
+            "name": "limit",
+            "in": "query",
+            "description": "How many; 20 when absent.",
+            "schema": {
+              "type": "string"
+            }
+          }
+        ],
+        "responses": {
+          "200": {
+            "description": "OK",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "type": "array",
+                  "items": {
+                    "$ref": "#/components/schemas/SettingsView"
+                  }
+                }
+              }
+            }
+          },
+          "401": {
+            "$ref": "#/components/responses/Unauthorized"
+          },
+          "default": {
+            "$ref": "#/components/responses/Error"
+          }
+        }
+      }
+    },
+    "/api/v1/settings/rollback": {
+      "post": {
+        "operationId": "postSettingsRollback",
+        "summary": "Put an earlier version of the cluster's settings back in force",
+        "tags": [
+          "settings"
+        ],
+        "parameters": [
+          {
+            "$ref": "#/components/parameters/Actor"
+          }
+        ],
+        "requestBody": {
+          "required": true,
+          "content": {
+            "application/json": {
+              "schema": {
+                "$ref": "#/components/schemas/SettingsRollbackRequest"
+              }
+            }
+          }
+        },
+        "responses": {
+          "200": {
+            "description": "OK",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/SettingsView"
+                }
+              }
+            }
+          },
+          "401": {
+            "$ref": "#/components/responses/Unauthorized"
+          },
+          "default": {
+            "$ref": "#/components/responses/Error"
+          }
+        }
+      }
+    },
     "/api/v1/summary": {
       "get": {
         "operationId": "getSummary",
@@ -3849,10 +4120,127 @@
           }
         }
       },
+      "GuardCountView": {
+        "type": "object",
+        "description": "GuardCountView is how many requests one policy did one thing to.",
+        "properties": {
+          "project": {
+            "type": "string"
+          },
+          "action": {
+            "type": "string"
+          },
+          "policy": {
+            "type": "string"
+          },
+          "category": {
+            "type": "string"
+          },
+          "count": {
+            "type": "integer",
+            "format": "int64"
+          }
+        }
+      },
+      "GuardEventView": {
+        "type": "object",
+        "description": "GuardEventView is one request the guardrails did something to.",
+        "properties": {
+          "at": {
+            "type": "string",
+            "format": "date-time"
+          },
+          "request_id": {
+            "type": "string"
+          },
+          "project": {
+            "type": "string"
+          },
+          "model": {
+            "type": "string"
+          },
+          "status": {
+            "type": "integer"
+          },
+          "action": {
+            "type": "string"
+          },
+          "policy": {
+            "type": "string"
+          },
+          "category": {
+            "type": "string"
+          },
+          "rule": {
+            "type": "string",
+            "description": "Rule is what decided it within the category: a pattern's name, or a judge's verdict, \"S6,Specialized Advice\"."
+          },
+          "guard_ms": {
+            "type": "integer",
+            "format": "int64"
+          }
+        }
+      },
+      "GuardEventsView": {
+        "type": "object",
+        "description": "GuardEventsView is what the guardrails did since a moment: totals, and the newest requests.",
+        "properties": {
+          "since": {
+            "type": "string",
+            "format": "date-time"
+          },
+          "counts": {
+            "type": "array",
+            "items": {
+              "$ref": "#/components/schemas/GuardCountView"
+            }
+          },
+          "events": {
+            "type": "array",
+            "items": {
+              "$ref": "#/components/schemas/GuardEventView"
+            }
+          }
+        }
+      },
+      "GuardPolicyView": {
+        "type": "object",
+        "description": "GuardPolicyView is what one policy does, in brief.",
+        "properties": {
+          "name": {
+            "type": "string"
+          },
+          "mode": {
+            "type": "string",
+            "description": "Mode is enforce, shadow or off; Fail is open or closed."
+          },
+          "fail": {
+            "type": "string"
+          },
+          "no_store": {
+            "type": "boolean"
+          },
+          "judge": {
+            "type": "string",
+            "description": "Judge is the judge model, when one reads the text."
+          },
+          "checks": {
+            "type": "array",
+            "items": {
+              "type": "string"
+            },
+            "description": "Checks are what it looks for, one line each: \"input: pii email → redact\"."
+          }
+        }
+      },
       "GuardView": {
         "type": "object",
         "description": "GuardView is the library and where its policies are attached.",
         "properties": {
+          "enabled": {
+            "type": "boolean",
+            "description": "Enabled is whether the guardrails check anything (guard.enabled)."
+          },
           "policies": {
             "type": "array",
             "items": {
@@ -3870,6 +4258,13 @@
             "items": {
               "$ref": "#/components/schemas/GuardAttachmentView"
             }
+          },
+          "details": {
+            "type": "array",
+            "items": {
+              "$ref": "#/components/schemas/GuardPolicyView"
+            },
+            "description": "Details says what each policy does, in the order of Policies."
           }
         }
       },
@@ -4044,6 +4439,96 @@
           }
         }
       },
+      "JudgeAskRequest": {
+        "type": "object",
+        "description": "JudgeAskRequest is a text to have a judge read.",
+        "properties": {
+          "model": {
+            "type": "string",
+            "description": "Model is the judge; empty is the one the policies ask, when they ask one."
+          },
+          "text": {
+            "type": "string"
+          },
+          "answer": {
+            "type": "string",
+            "description": "Answer, when given, is read as the assistant's reply to Text."
+          }
+        }
+      },
+      "JudgeAskView": {
+        "type": "object",
+        "description": "JudgeAskView is what the judge answered.",
+        "properties": {
+          "model": {
+            "type": "string"
+          },
+          "verdict": {
+            "type": "string",
+            "description": "Verdict is safe or unsafe, as the gateway reads the answer."
+          },
+          "categories": {
+            "type": "array",
+            "items": {
+              "type": "string"
+            }
+          },
+          "raw": {
+            "type": "string",
+            "description": "Raw is the judge's answer as it gave it."
+          },
+          "took_ms": {
+            "type": "integer",
+            "format": "int64"
+          }
+        }
+      },
+      "JudgeView": {
+        "type": "object",
+        "description": "JudgeView is one judge model the guardrail policies ask, and whether it can answer them.",
+        "properties": {
+          "model": {
+            "type": "string"
+          },
+          "policies": {
+            "type": "array",
+            "items": {
+              "type": "string"
+            },
+            "description": "Policies are those that ask it; Closed those that refuse what it cannot read (fail: closed)."
+          },
+          "closed": {
+            "type": "array",
+            "items": {
+              "type": "string"
+            }
+          },
+          "in_catalog": {
+            "type": "boolean",
+            "description": "InCatalog is whether a model of that name is in the catalog, and Judge whether it is there as a judge (role: guard)."
+          },
+          "judge": {
+            "type": "boolean"
+          },
+          "ready_replicas": {
+            "type": "integer"
+          },
+          "wanted_replicas": {
+            "type": "integer"
+          },
+          "last_error": {
+            "type": "string",
+            "description": "LastError is its last failure to give a verdict, if any."
+          },
+          "last_error_at": {
+            "type": "string"
+          },
+          "problem": {
+            "type": "string",
+            "description": "Problem says, in words, what is wrong and what to do; empty when nothing is."
+          }
+        }
+      },
       "Live": {
         "type": "object",
         "description": "Live is one of the cluster's adapters as its node last reported it.",
@@ -5010,6 +5495,9 @@
           "guard_category": {
             "type": "string"
           },
+          "guard_rule": {
+            "type": "string"
+          },
           "guard_ms": {
             "type": "integer",
             "format": "int64"
@@ -5197,6 +5685,64 @@
           }
         }
       },
+      "SettingsRequest": {
+        "type": "object",
+        "description": "SettingsRequest is a new version of the cluster's settings, made from the version Base, which must still be the newest.",
+        "properties": {
+          "base": {
+            "type": "integer",
+            "format": "int64"
+          },
+          "body": {
+            "type": "string"
+          },
+          "note": {
+            "type": "string"
+          }
+        }
+      },
+      "SettingsRollbackRequest": {
+        "type": "object",
+        "description": "SettingsRollbackRequest puts an earlier version's settings back in force, as a new version.",
+        "properties": {
+          "version": {
+            "type": "integer",
+            "format": "int64"
+          },
+          "note": {
+            "type": "string"
+          }
+        }
+      },
+      "SettingsView": {
+        "type": "object",
+        "description": "SettingsView is one version of the cluster's settings.",
+        "properties": {
+          "version": {
+            "type": "integer",
+            "format": "int64"
+          },
+          "body": {
+            "type": "string"
+          },
+          "note": {
+            "type": "string"
+          },
+          "at": {
+            "type": "string",
+            "format": "date-time"
+          },
+          "by": {
+            "type": "string"
+          },
+          "local_keys": {
+            "type": "array",
+            "items": {
+              "type": "string"
+            }
+          }
+        }
+      },
       "ShardView": {
         "type": "object",
         "description": "ShardView is one shard of a replica.",
@@ -5263,6 +5809,10 @@
         "type": "object",
         "description": "Summary is the whole fleet in one answer.\n\nA per-node listing is the right tool for four nodes and the wrong one for two hundred: the questions that matter at that size — how full am I, what is broken, what is holding cards it is not using — are aggregates, and reading them off a long table is a chore that gets skipped. Everything here is derived from the same state the tables show, so it can never disagree with them; it is a different question asked of the same facts.",
         "properties": {
+          "version": {
+            "type": "string",
+            "description": "Version is the release the answering control plane runs: not the binary on its disk, which a manual install changes underneath it."
+          },
           "nodes": {
             "$ref": "#/components/schemas/NodeTotals"
           },
@@ -5319,6 +5869,17 @@
           "control_planes": {
             "$ref": "#/components/schemas/ControlPlanes",
             "description": "ControlPlanes is the cluster's control planes when there is more than one: which leads, which stand by, and the floating address in front."
+          },
+          "judges": {
+            "type": "array",
+            "items": {
+              "$ref": "#/components/schemas/JudgeView"
+            },
+            "description": "Judges are the guardrails' judge models with something wrong."
+          },
+          "guard_off": {
+            "type": "boolean",
+            "description": "GuardOff says the guardrails are turned off (guard.enabled: false): no request is checked, whatever the policies say."
           }
         }
       },
@@ -6848,11 +7409,15 @@
           },
           "ctx_pool": {
             "type": "integer",
-            "description": "CtxPool is llama.cpp's --ctx-size; ContextLength what one request may use. They default to 16384 and 4096, what four parallel slots divide."
+            "description": "CtxPool is llama.cpp's --ctx-size; ContextLength what one request may use. They default to 16384 and 4096, what four parallel slots divide; llama.cpp is given as many slots as the pool holds requests."
           },
           "context_length": {
             "type": "integer"
           },
+          "tools": {
+            "type": "boolean",
+            "description": "Tools says the model makes tool calls the engine returns as calls: it is written into the catalog, where clients and editors read it. Check a running replica with llmgw ctl tool-check before saying so."
+          },
           "cpu_cores": {
             "type": "integer",
             "description": "CPUCores is what a replica computes on; zero takes half the smallest node it fits on."

docs/clients.md

@@ -184,6 +184,15 @@ asks for, send the results back. Whether a model calls tools well is the
 model's business, not the gateway's; ask your operator which of the served
 models are set up for it.
 
+**Editors' agents need such a model.** Continue's Agent mode, Cline and
+Codex act only through tool calls: with a model whose calls come back as
+text — `<tool_call>{...}` or a JSON block in the answer, `tool_calls` empty —
+they chat but never read a file or run a command. Qwen2.5-Coder is one;
+Qwen3-Coder is not. The configuration the portal writes for Continue says
+`tool_use` only for a model whose catalog entry says tools, and gives
+Continue the model's context and longest answer, so it never sends a request
+the gateway refuses.
+
 ```python
 import json
 

docs/console.md

@@ -270,14 +270,20 @@ page's text.
 
 ## Settings
 
-**server.yaml** (on a node, **agent.yaml**) is every key of the file, as a
-tree of menus built from the configuration reference: the value set, or the
-one that applies when nothing is, its type, and what it is for. Enter changes
-a value or opens a block; on a list or map of blocks, Enter opens an entry and
-**D** removes one. **U** puts a key back to its default. **S** saves: the file
-is checked the way the service reads it before anything is written, and its
-comments are kept. The service reads it when it starts, so the console offers
-to restart it. Another control plane keeps its own copy.
+**server.yaml** (on a node, **agent.yaml**) is every key, as a tree of menus
+built from the configuration reference: the value set, or the one that
+applies when nothing is, its type, and what it is for. Enter changes a value
+or opens a block; on a list or map of blocks, Enter opens an entry and **D**
+removes one. **U** puts a key back to its default. **S** saves.
+
+On a control plane, the cluster's keys are read from and saved to the
+cluster's settings in the store — the same on every control plane, in force
+on all of them within seconds (`llmgw ctl config` does the same from a
+shell) — and this machine's own keys (listeners, certificates, the store's
+address) to its server.yaml, which the server notices and applies by itself.
+Either is checked before anything is kept, and the file's comments survive.
+On a node, agent.yaml is read when the agent starts, so the console offers
+to restart it.
 
 ## Do
 
@@ -313,6 +319,9 @@ to restart it. Another control plane keeps its own copy.
 - **Restart all replicas**, one at a time.
 - **Back up now** — the store and `/etc/llmgw` into one archive
   (`sudo llmgw server backup`).
+- **Wipe this machine** — removes llmgw's services, configuration, state and
+  users from this machine, after listing them and asking for its name
+  (`sudo llmgw wipe`; on a node too).
 - **Leave**.
 
 ## The node's console

docs/guardrails.md

@@ -5,12 +5,29 @@ policy can check, what it can do, and how one is attached to a team.
 
 `llmgw server init` writes a `baseline` policy that applies to every
 request, with Llama Guard 3 1B as its judge on a CPU; see
-[install.md](install.md#2-the-control-plane) for what it checks. Without a
-`guard:` section in `server.yaml` nothing is checked and nothing costs
-anything. With one, every request is judged once it is authorised and
+[install.md](install.md#2-the-control-plane) for what it checks. The
+policies are part of the cluster's settings, kept in the store: `llmgw ctl
+config edit` (or the console's Settings) changes them, and the change is in
+force on every control plane within seconds, without a restart. A cluster
+whose settings were brought in from a server.yaml without a `guard:` section
+gets the `baseline` policy, without a judge. With no policies, nothing is
+checked and nothing costs anything. With one, every request is judged once it is authorised and
 within its limits, with its stored conversation spliced in, and before it is
 sent to a model; and the answer is judged before the client sees it.
 
+## Turning them off and on
+
+```
+llmgw ctl guard off --note "why"   # no request is checked, whatever the policies say
+llmgw ctl guard on                 # every policy checks again, as it was
+```
+
+That is `guard.enabled` in the cluster's settings, in force on every control
+plane within seconds without a restart. The policies and where they are
+attached are kept. While off, `llmgw ctl status`, `llmgw ctl guard list` and
+the guardrails page say so. To stop one policy rather than all, give it
+`mode: off`, or `mode: shadow` to have it record what it would do.
+
 ## A policy
 
 ```yaml
@@ -99,7 +116,42 @@ guard:
 
 Patterns find what has a shape. For what does not — violence, self-harm,
 abuse, whatever a moderation model was trained on — a policy can have a
-**judge model on the cluster** read the text:
+**judge model on the cluster** read the text.
+
+The judge must be in the catalog under the name the policy gives, as a
+judge (`role: guard`), with a replica ready. When it is not, what the
+policy judges is refused with `guard_unavailable` (`fail: closed`) or served
+unjudged (`fail: open`), and `llmgw ctl status` and the console's Overview
+say which judge and why. To serve it:
+
+```
+llmgw ctl guard judge status           # each judge the policies ask, and what is wrong with it
+llmgw ctl guard judge enable           # Llama Guard 3 1B on CPUs, under the name the policies ask
+llmgw ctl guard judge enable --replace # when a model of that name is there, but not as a judge
+```
+
+or **Do → Serve the guardrails' judge** in the console.
+
+**When it refuses ordinary requests.** A refusal by a judge names the hazard
+it saw — the client is told `refused by the gateway's policy (judge:
+S6,Specialized Advice)` — and the same is kept with the request: `llmgw ctl
+request <id>`, the server's log (`guard_rule`) and the guardrails page show
+it. To see what the judge says of any text, without anything being kept:
+
+```
+llmgw ctl guard judge ask 'How do I sort a list in Python?'
+# llama-guard-3-1b: safe, 140 ms
+# it answered: "safe"
+```
+
+A judge that says `unsafe` to ordinary text is usually the wrong weights
+(another repository's or quantisation's, without Llama Guard's template:
+`judge enable --replace` serves the right ones), or a policy counting every
+hazard: give its judge `categories`, leaving out S5, S6, S8 and S13 as the
+baseline does. While finding out, `judge.input: flag` records instead of
+refusing. A Llama Guard file
+served from the mirror page is served as a judge too, under the name the
+policies ask. By hand, the catalog entry is:
 
 ```yaml
 # a catalog model (server.yaml models:, or ctl model add)
@@ -209,3 +261,14 @@ Each request's usage row has the action, policy, category and the time the
 checks took. The access log has the action, stage and policy. Neither has
 the text, a matched value, or the placeholder map: the map exists only in
 the gateway's memory while the request is served.
+
+## In the console
+
+`sudo llmgw` has a **Guardrails** view: each policy with its mode, what it
+does when it cannot judge, and what it checks; whether each judge model it
+leans on has replicas ready — a policy that fails closed refuses everything
+while its judge has none; where policies are attached, with A to attach one
+and Enter on a row to detach it; and what they did in the last day (W for an
+hour or a week), totals first and the newest requests after, each with the
+request id `llmgw ctl request` reads. The same is at `GET /api/v1/guard` and
+`GET /api/v1/guard/events?since=24h`, and on the portal's Guardrails page.

docs/install.md

@@ -241,8 +241,38 @@ password, change it, and issue yourself a key from the profile page.
 
 ### The configuration
 
-`/etc/llmgw/server.yaml` is yours from here on. Three things in it are worth
-knowing before you edit it:
+There are two halves, and each change to either is in force within seconds,
+on every machine it concerns, without anyone restarting anything.
+
+**The cluster's settings** — guardrails, limits and prices, routing, the
+reconciler, engines, notifications, the mirror's policy, models declared by
+hand, offline, federation, facility budgets, backups' recipients — are kept in
+the store, so every control plane runs on the same ones. Change them with
+
+```
+llmgw ctl config                         # what is in force, and its version
+llmgw ctl config set router.retries 3 --note "flaky upstream"
+llmgw ctl config edit                    # the whole document in $EDITOR
+llmgw ctl config history                 # who changed what, when, and why
+llmgw ctl config rollback 4              # back to version 4, as a new version
+```
+
+or on the console's **Settings** screen. A change is checked before it is
+kept — one that would not load is refused — and recorded in the audit log.
+Guardrails change live; anything else has every control plane start again on
+the new settings, the standbys first and the leader half a minute later, so
+one of them is always serving. Engines keep running through it.
+
+The first control plane to start against a store brings in its server.yaml's
+cluster keys — with a `baseline` guard policy, if the file had none — and
+from then on those keys in any server.yaml are not read (the server says so
+when it starts).
+
+**This machine's own keys** stay in `/etc/llmgw/server.yaml`: `listen`,
+`admin_listen`, `metrics_listen`, `agent_listen`, `agent_tls`, `auth.store`,
+`secrets`, `tracing`, `monitoring`, `mirror.api` and `mirror.tls`, the backup
+directory and key. The server notices the file change and starts again on it
+by itself. Three things are worth knowing before you edit it:
 
 - `admin_listen` is on loopback on purpose, and the server refuses any other
   address unless `admin_remote: true`. Its API under `/api` adds models and
@@ -1940,8 +1970,12 @@ Without it, put a load balancer in front that checks `/ready` — not
   `CONTROL` line: which leads, which stand by, one that does not answer, and
   whether the floating address is on the leader. A standby with nothing to
   take over from is the thing to see before it is needed.
-- **Either machine's console.** A standby's admin API passes everything under
-  `/api/` to the leader, over the port control planes already reach each
+- **Either machine, the leader's cluster.** A standby passes its client port
+  — the portal, its admin pages, the API — to the leader's, keeping only
+  `/health`, `/ready` and `/leader` to describe itself, so a proxy pointed at
+  the wrong machine still shows the cluster; the leader trusts a control
+  plane's `X-Forwarded-For`, so clients keep their own address. Its admin
+  API passes everything but `/debug/` and `/metrics` to the leader, over the port control planes already reach each
   other on and with the cluster's certificates both ways, so `sudo llmgw` and
   `llmgw ctl` show the same cluster on both — the leader's — and a change made
   on the standby is made by the leader. A node's certificate is refused
@@ -2097,14 +2131,33 @@ loaded at boot by `llmgw-firewall.service`):
 
 "The cluster's networks" are, on each machine, the network of the interface
 it reaches the control planes (or the other store members) on — or just
-their address, when the route goes through a gateway. Anything else that
-must reach those ports, a Prometheus on another subnet or nodes behind a
-router, goes in `server.yaml` before the machines are set up:
+their address, when the route goes through a gateway. `node add` opens
+the control plane's to each node it adds when the node is on a network not
+already let in: its subnet when the control plane is on it, its address alone
+when a router is between them. Anything else that must reach those ports, a
+Prometheus on another subnet, a second control plane's or a separate mirror's
+for nodes behind a router, goes in `server.yaml`:
 
 ```yaml
 firewall_networks: [10.0.5.0/24, 192.168.1.20]
 ```
 
+**Turning it off and on.** Whether llmgw keeps this firewall is one of the
+cluster's settings:
+
+```
+llmgw ctl config set firewall.enabled false   # every machine's table taken out
+llmgw ctl config set firewall.enabled true    # and put back as it was
+```
+
+It is in force within seconds: each node's agent takes its table out or puts
+it back, and each control plane's `llmgw-firewall-sync.path` (installed by
+`server init` and `server add`; `sudo llmgw firewall sync --install` on one
+set up before) does the same there. The rules files stay, so turning it back
+on restores them. Nodes added while it is off get none. A separate mirror
+machine and the store's members keep theirs for now. `firewall_networks`
+changes reach the nodes the same way.
+
 Only those ports are touched. The table's policy is to accept, so whatever
 else the machine serves — and whatever firewall it already has — is left as
 it was; the client API (`:8080`) is not closed, being meant to be reached.
@@ -2241,6 +2294,31 @@ A cluster set up online becomes offline the same way: `bundle import` on the
 mirror and control plane, then the `offline:` block in `server.yaml` and a
 restart.
 
+## Removing llmgw from a machine
+
+```
+sudo llmgw wipe              # services, configuration, certificates, state, users
+sudo llmgw wipe --models     # and model weights, the mirror's files, engines
+sudo llmgw wipe --store      # and the store's database here (Patroni member, etcd, or local PostgreSQL)
+sudo llmgw wipe --all        # all of that, backups and the binary
+```
+
+It lists everything it will remove and asks for the machine's name (`--yes`
+does not ask), then stops llmgw's services and the engines, takes the
+firewall table out, and deletes `/etc/llmgw` — the authority and secrets
+with it — `/var/lib/llmgw`, the agent's state, the sudo grant, the
+authority from the system's trust, keepalived's configuration when llmgw
+wrote it, and the `llmgw` and `llmgw-engine` users. Packages it installed
+(PostgreSQL, Patroni, etcd, nftables, the node exporter) stay. Run it again
+to finish what failed. A node wiped is then forgotten on the control plane
+with `llmgw node remove <name>`. The console has it too: **Do → Wipe this
+machine**.
+
+This is how a machine with an older install is set up again from nothing:
+wipe it, then install as above. Wiping a control plane without `--store`
+leaves the store with the cluster's accounts, keys and settings in it, and a
+new control plane against that store takes them up.
+
 ## Kubernetes
 
 The control plane runs in Kubernetes from the Helm chart in
@@ -2255,12 +2333,13 @@ On a control plane:
 sudo --preserve-env=SSH_AUTH_SOCK llmgw ctl upgrade v0.10.4
 ```
 
-With two control planes it upgrades the other one first, over ssh as the user
-who ran sudo (`--ssh-user`), with `llmgw upgrade` there; then this machine —
+With two control planes it upgrades the standby first and the leader last,
+whichever this machine is — the other over ssh as the user who ran sudo
+(`--ssh-user`), with `llmgw upgrade` there, and this machine —
 fetching the release for its platform, checking it against the release's
 checksums, keeping the old binary as `/usr/local/bin/llmgw.bak-pre-<version>`,
-restarting `llmgw-server` and waiting for every node to reconnect. A leader
-restarted last hands over to a standby already on the release.
+restarting `llmgw-server` and waiting for every node to reconnect. The
+leader's restart hands over once, to a standby already on the release.
 
 Then the nodes, in waves: one first, then two, four, and so on up to
 `--parallel` (20) at once. A node's agent is told the version over the stream

docs/reference.md

@@ -98,6 +98,7 @@ rather than repeated.
 | `other_watts` | `float64` | [FacilityBudget](#facilitybudget) |
 | `name` | `string` | [Federation](#federation) |
 | `interval` | `time.Duration` | [Federation](#federation) |
+| `enabled` | `bool` | [FirewallSettings](#firewallsettings) |
 | `format` | `string` | [GuardModel](#guardmodel) |
 | `categories` | `[]string` | [GuardModel](#guardmodel) |
 | `model` | `string` | [JudgePolicy](#judgepolicy) |
@@ -287,6 +288,7 @@ rather than repeated.
 | `guard` | `guard.Config` | [Server](#server) |
 | `secrets` | `Secrets` | [Server](#server) |
 | `notify` | `notify.Config` | [Server](#server) |
+| `firewall` | `FirewallSettings` | [Server](#server) |
 | `firewall_networks` | `[]string` | [Server](#server) |
 | `backup_recipients` | `[]string` | [Server](#server) |
 | `backup_copy_to` | `[]string` | [Server](#server) |
@@ -370,6 +372,7 @@ rather than repeated.
 | `llamacpp` | `LlamaCppConfig` | [engines.Config](#enginesconfig) |
 | `trtllm` | `TRTLLMConfig` | [engines.Config](#enginesconfig) |
 | `sim` | `SimConfig` | [engines.Config](#enginesconfig) |
+| `enabled` | `*bool` | [guard.Config](#guardconfig) |
 | `default` | `[]string` | [guard.Config](#guardconfig) |
 | `policies` | `map[string]Policy` | [guard.Config](#guardconfig) |
 | `webhooks` | `[]Webhook` | [notify.Config](#notifyconfig) |
@@ -963,6 +966,19 @@ Interval is how often each peer is asked whether it is ready, timed,
 and what it offers. A peer that does not answer is taken out of
 routing until it does. Zero is 10s.
 
+## FirewallSettings
+
+### `enabled`
+
+`bool`
+
+Enabled keeps llmgw's internal ports — engines, node exporter, the
+agents' stream, the mirror, the store — to the cluster's networks on
+every machine. False removes llmgw's nftables table everywhere, for a
+site whose own firewall does that job; the rules are kept, and true
+puts them back. Changed in the cluster's settings, it is in force on
+every node and control plane within seconds.
+
 ## GuardModel
 
 ### `format`
@@ -2441,6 +2457,14 @@ Prometheus and Alertmanager in between. Empty sends nothing.
 
 Takes a [notify.Config](#notifyconfig) block.
 
+### `firewall`
+
+`FirewallSettings`
+
+Firewall turns llmgw's firewall on every machine on or off.
+
+Takes a [FirewallSettings](#firewallsettings) block.
+
 ### `firewall_networks`
 
 `[]string`
@@ -3044,6 +3068,13 @@ Takes a [SimConfig](#simconfig) block.
 
 ## guard.Config
 
+### `enabled`
+
+`*bool`
+
+Enabled turns every policy on or off at once: false checks nothing,
+whatever is defined or attached. Unset is on.
+
 ### `default`
 
 `[]string`

internal/agent/agent.go

@@ -36,6 +36,7 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/agentsvc"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/kvevents"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/obs"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
@@ -87,7 +88,11 @@ type Agent struct {
 	// serverList is every control plane to dial: the configured ones, and
 	// any the leader has named since. Read through servers().
 	serverList []string
-	inv        discovery.Inventory
+	// fwMu keeps one firewall change at a time; fwRunner runs its scripts
+	// (nil: this machine's shell).
+	fwMu     sync.Mutex
+	fwRunner firewall.Runner
+	inv      discovery.Inventory
 	// shardReplica remembers which replica a shard belongs to, so a status
 	// report can name both. The control plane keys on the replica.
 	shardReplica map[string]string
@@ -854,6 +859,8 @@ func (a *Agent) handle(ctx context.Context, msg *llmgwv1.ControlMessage, out cha
 		go a.upgrade(ctx, m.Upgrade, out)
 	case *llmgwv1.ControlMessage_ControlPlanes:
 		a.setControlPlanes(m.ControlPlanes)
+	case *llmgwv1.ControlMessage_Firewall:
+		go a.applyFirewall(m.Firewall)
 	case *llmgwv1.ControlMessage_Logs:
 		reply := a.logsReply(m.Logs)
 		// Off the receive loop: a full send queue must not stop commands being

internal/agent/firewall.go

@@ -0,0 +1,89 @@
+package agent
+
+import (
+	"context"
+	"net"
+	"net/netip"
+	"net/url"
+	"strings"
+	"time"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
+)
+
+// applyFirewall keeps llmgw's firewall on this node as the cluster's
+// settings say: the engines' ports and the node exporter's for the networks
+// the node reaches its control planes on, and those named, or no table at
+// all. Off the receive loop, one at a time.
+func (a *Agent) applyFirewall(m *llmgwv1.Firewall) {
+	a.fwMu.Lock()
+	defer a.fwMu.Unlock()
+	ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
+	defer cancel()
+	r := a.fwRunner
+	if r == nil {
+		r = firewall.Local{}
+	}
+	if !m.GetEnabled() {
+		had, err := firewall.Disable(ctx, r)
+		switch {
+		case err != nil:
+			a.log.Warn("could not take llmgw's firewall off this node", "err", err)
+		case had:
+			a.log.Info("llmgw's firewall is off on this node, as the cluster's settings say")
+		}
+		return
+	}
+	a.mu.Lock()
+	servers := append([]string(nil), a.serverList...)
+	a.mu.Unlock()
+	nets, err := firewall.Networks(ctx, r, controlPlaneIPs(ctx, servers))
+	if err != nil {
+		a.log.Warn("could not tell which networks reach the control planes; the firewall is left as it is", "err", err)
+		return
+	}
+	for _, n := range m.GetNetworks() {
+		if p, err := netip.ParsePrefix(n); err == nil {
+			nets = append(nets, p)
+		} else if ad, err := netip.ParseAddr(n); err == nil {
+			nets = append(nets, netip.PrefixFrom(ad, ad.BitLen()))
+		}
+	}
+	from, to := int(m.GetEnginePortsFrom()), int(m.GetEnginePortsTo())
+	ports := []firewall.Port{firewall.NodeExporter}
+	if from > 0 && to >= from {
+		ports = append(ports, firewall.Engines(from, to))
+	}
+	changed, err := firewall.Install(ctx, r, firewall.Spec{Ports: ports, Allow: nets})
+	switch {
+	case err != nil:
+		a.log.Warn("could not put llmgw's firewall on this node", "err", err)
+	case changed:
+		a.log.Info("llmgw's firewall is on this node", "networks", len(nets))
+	}
+}
+
+// controlPlaneIPs are the addresses of the control planes an agent dials.
+func controlPlaneIPs(ctx context.Context, servers []string) []string {
+	var out []string
+	for _, s := range servers {
+		host := s
+		if u, err := url.Parse(s); err == nil && u.Host != "" {
+			host = u.Hostname()
+		}
+		host = strings.Trim(host, "[]")
+		if a, err := netip.ParseAddr(host); err == nil {
+			out = append(out, a.String())
+			continue
+		}
+		ips, err := net.DefaultResolver.LookupNetIP(ctx, "ip", host)
+		if err != nil {
+			continue
+		}
+		for _, ip := range ips {
+			out = append(out, ip.Unmap().String())
+		}
+	}
+	return out
+}

internal/agent/firewall_test.go

@@ -0,0 +1,57 @@
+package agent
+
+import (
+	"context"
+	"errors"
+	"log/slog"
+	"strings"
+	"testing"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+)
+
+// fakeShell answers the firewall package's scripts and records what was
+// installed or removed.
+type fakeShell struct {
+	present   bool
+	installed string
+	deleted   bool
+}
+
+func (f *fakeShell) Run(_ context.Context, script string, stdin []byte) ([]byte, error) {
+	switch {
+	case strings.HasPrefix(script, "ip -o route get 10.0.0.140"):
+		return []byte("10.0.0.140 dev eth0 src 10.0.0.141 uid 0\n2: eth0    inet 10.0.0.141/24 brd 10.0.0.255 scope global eth0\n"), nil
+	case strings.HasPrefix(script, "sudo -n /usr/sbin/nft list table"):
+		if f.present {
+			return nil, nil
+		}
+		return nil, errors.New("no table")
+	case strings.Contains(script, "nft delete table") && !strings.Contains(script, "cat > "):
+		f.deleted, f.present = true, false
+		return nil, nil
+	case strings.HasPrefix(script, "sudo cat "):
+		return nil, nil
+	default:
+		f.installed, f.present = string(stdin), true
+		return []byte("changed\n"), nil
+	}
+}
+
+func TestANodeKeepsOrDropsItsFirewallAsTheClusterSays(t *testing.T) {
+	sh := &fakeShell{}
+	a := &Agent{log: slog.New(slog.DiscardHandler), fwRunner: sh, serverList: []string{"https://10.0.0.140:7443"}}
+
+	a.applyFirewall(&llmgwv1.Firewall{Enabled: true, Networks: []string{"192.168.5.0/24", "10.9.9.9"},
+		EnginePortsFrom: 20000, EnginePortsTo: 20999})
+	for _, want := range []string{"10.0.0.0/24", "192.168.5.0/24", "10.9.9.9/32", "20000-20999", "9100"} {
+		if !strings.Contains(sh.installed, want) {
+			t.Errorf("the rules lack %s:\n%s", want, sh.installed)
+		}
+	}
+
+	a.applyFirewall(&llmgwv1.Firewall{Enabled: false})
+	if !sh.deleted {
+		t.Error("turned off, the table stayed")
+	}
+}

internal/audit/audit.go

@@ -12,6 +12,7 @@ import (
 	"net/http"
 	"net/url"
 	"regexp"
+	"slices"
 	"sort"
 	"strings"
 	"time"
@@ -110,11 +111,16 @@ func (w *statusWriter) Flush() {
 	}
 }
 
+// notChanges are POSTs that change nothing and carry text that must not be
+// kept: asking a judge about a sentence.
+var notChanges = []string{"/api/v1/guard/judges/ask"}
+
 // API records every change to the admin API: a request that is not a GET or
 // HEAD, under /api/. actor says who made it.
 func API(sink Sink, actor func(*http.Request) string, log *slog.Logger, next http.Handler) http.Handler {
 	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
-		if r.Method == http.MethodGet || r.Method == http.MethodHead || !strings.HasPrefix(r.URL.Path, "/api/") {
+		if r.Method == http.MethodGet || r.Method == http.MethodHead || !strings.HasPrefix(r.URL.Path, "/api/") ||
+			slices.Contains(notChanges, r.URL.Path) {
 			next.ServeHTTP(w, r)
 			return
 		}

internal/audit/audit_test.go

@@ -72,3 +72,13 @@ func TestABodyLargerThanWhatIsRecordedReachesTheHandlerWhole(t *testing.T) {
 		t.Errorf("the handler was given %d of %d bytes", got, len(big))
 	}
 }
+
+func TestAskingAJudgeKeepsNoText(t *testing.T) {
+	sink := &memSink{}
+	h := API(sink, func(*http.Request) string { return "ilker" }, nil, http.HandlerFunc(func(http.ResponseWriter, *http.Request) {}))
+	h.ServeHTTP(httptest.NewRecorder(), httptest.NewRequest("POST", "/api/v1/guard/judges/ask",
+		strings.NewReader(`{"text":"something a user wrote"}`)))
+	if len(sink.got) != 0 {
+		t.Errorf("a judge's question was recorded: %+v", sink.got)
+	}
+}

internal/cli/backup.go

@@ -54,7 +54,7 @@ func newBackupCmd() *cobra.Command {
 			"the newest archive as llmgw_backup_last_success_timestamp_seconds, and\n" +
 			"BackupTooOld fires when it is more than a day old.",
 		RunE: func(cmd *cobra.Command, _ []string) error {
-			cfg, err := config.Load(configPath)
+			cfg, err := loadEffectiveConfig(configPath)
 			if err != nil {
 				return err
 			}

internal/cli/bundle.go

@@ -323,7 +323,7 @@ func loadServerConfig(path string) (config.Server, bool) {
 	if _, err := os.Stat(path); err != nil {
 		return config.Server{}, false
 	}
-	cfg, err := config.Load(path)
+	cfg, err := loadEffectiveConfig(path)
 	if err != nil {
 		return config.Server{}, false
 	}

internal/cli/clustersettings.go

@@ -0,0 +1,363 @@
+package cli
+
+import (
+	"context"
+	"crypto/sha256"
+	"errors"
+	"fmt"
+	"log/slog"
+	"os"
+	"slices"
+	"strings"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
+	storepkg "gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// settingsPoll is how often a control plane asks whether the cluster's
+// settings, or its own server.yaml, changed.
+const settingsPoll = 5 * time.Second
+
+// clusterSettings is what a control plane runs on, and where it came from:
+// its server.yaml for its own keys, the store's settings for the rest.
+type clusterSettings struct {
+	cfg      config.Server
+	eff      map[string]any // the settings, as written, cfg was made from
+	cluster  map[string]any // the store's document, nil without a store
+	version  int64
+	fileHash [32]byte
+}
+
+// loadClusterSettings reads the configuration a control plane starts on.
+// Without a store it is the file. With one, the store's settings take the
+// place of the file's cluster keys; a store with none yet is given the
+// file's, so the first control plane to start against it brings them in.
+func loadClusterSettings(ctx context.Context, path string, log *slog.Logger) (clusterSettings, error) {
+	var s clusterSettings
+	b, file, err := readSettingsFile(path)
+	if err != nil {
+		return s, err
+	}
+	s.fileHash = sha256.Sum256(b)
+	fileCfg, err := config.FromMap(file)
+	if err != nil {
+		return s, fileError(path, err)
+	}
+	if fileCfg.Auth.Store == "" {
+		s.cfg, s.eff = fileCfg, file
+		return s, nil
+	}
+	st, err := storepkg.Open(ctx, fileCfg.Auth.Store)
+	if err != nil {
+		return s, fmt.Errorf("the store, for the cluster's settings: %w", err)
+	}
+	defer st.Close()
+
+	doc, err := st.ClusterSettings(ctx)
+	if err != nil {
+		return s, fmt.Errorf("reading the cluster's settings: %w", err)
+	}
+	if doc.Version == 0 {
+		if doc, err = importSettings(ctx, st, path, file, log); err != nil {
+			return s, err
+		}
+	} else if ignored := config.Ignored(file); len(ignored) > 0 {
+		log.Warn("server.yaml sets keys the cluster's settings now hold; they are not read "+
+			"(llmgw ctl config shows what is in force)", "keys", strings.Join(ignored, ", "))
+	}
+	// The newest version that this machine can run on. One that cannot — a
+	// path only another control plane has — must not keep this one down,
+	// where nothing could be used to put it right.
+	versions := []storepkg.SettingsVersion{doc}
+	if older, err := st.ClusterSettingsHistory(ctx, 20); err == nil && len(older) > 1 {
+		versions = append(versions, older[1:]...)
+	}
+	var firstErr error
+	for _, v := range versions {
+		cluster, err := config.Parse([]byte(v.Body))
+		if err == nil {
+			eff := config.Layer(file, cluster)
+			var cfg config.Server
+			if cfg, err = config.FromMap(eff); err == nil {
+				if v.Version != doc.Version {
+					log.Error("the cluster's newest settings do not load on this machine; running on an older version",
+						"newest", doc.Version, "running", v.Version, "err", firstErr)
+				}
+				s.cfg, s.eff, s.cluster, s.version = cfg, eff, cluster, doc.Version
+				return s, nil
+			}
+		}
+		if firstErr == nil {
+			firstErr = err
+		}
+	}
+	return s, fmt.Errorf("the cluster's settings (version %d) do not load on this machine: %w", doc.Version, firstErr)
+}
+
+// importSettings records a server.yaml's cluster keys as the cluster's first
+// settings, with the baseline guard policy when the file has none: a cluster
+// installed before guardrails gets them on its first start after an upgrade.
+func importSettings(ctx context.Context, st *storepkg.DB, path string, file map[string]any,
+	log *slog.Logger,
+) (storepkg.SettingsVersion, error) {
+	_, cluster := config.Split(file)
+	note := "imported from " + path
+	if _, ok := cluster["guard"]; !ok {
+		baseline, err := config.Parse([]byte(guardSection(false)))
+		if err == nil {
+			cluster["guard"] = baseline["guard"]
+			note += ", with the baseline guard policy it did not have"
+		}
+	}
+	body, err := config.Marshal(cluster)
+	if err != nil {
+		return storepkg.SettingsVersion{}, err
+	}
+	host, _ := os.Hostname()
+	_, err = st.PutClusterSettings(ctx, 0, string(body), host, note)
+	switch {
+	case errors.Is(err, storepkg.ErrSettingsChanged):
+		// Another control plane brought its own in first: those are the
+		// cluster's.
+	case err != nil:
+		return storepkg.SettingsVersion{}, fmt.Errorf("recording the cluster's settings: %w", err)
+	default:
+		log.Info("the cluster's settings are now kept in the store; server.yaml keeps this machine's own",
+			"from", path)
+	}
+	return st.ClusterSettings(ctx)
+}
+
+func readSettingsFile(path string) ([]byte, map[string]any, error) {
+	if path == "" {
+		return nil, map[string]any{}, nil
+	}
+	b, err := os.ReadFile(path) //nolint:gosec // the configuration
+	if err != nil {
+		return nil, nil, fmt.Errorf("config file: %w", err)
+	}
+	m, err := config.Parse(b)
+	if err != nil {
+		return nil, nil, fmt.Errorf("parsing %s: %w", path, err)
+	}
+	return b, m, nil
+}
+
+func fileError(path string, err error) error {
+	if path == "" {
+		return err
+	}
+	return fmt.Errorf("%s: %w", path, err)
+}
+
+// settingsWatch follows the cluster's settings and this machine's
+// server.yaml, and puts every change in force.
+type settingsWatch struct {
+	path string
+	st   *storepkg.DB // nil: the file only
+	cur  clusterSettings
+	// live applies a change without a restart, reporting whether it could.
+	live func(cfg config.Server, changed []string) bool
+	// restart has the process start again on the new settings.
+	restart func(reason string)
+	log     *slog.Logger
+}
+
+func (w *settingsWatch) run(ctx context.Context) {
+	t := time.NewTicker(settingsPoll)
+	defer t.Stop()
+	for {
+		select {
+		case <-ctx.Done():
+			return
+		case <-t.C:
+			w.check(ctx)
+		}
+	}
+}
+
+// check reads what changed, if anything, and applies it.
+func (w *settingsWatch) check(ctx context.Context) {
+	version := w.cur.version
+	if w.st != nil {
+		v, err := w.st.ClusterSettingsVersion(ctx)
+		if err != nil {
+			return // asked again in a few seconds
+		}
+		version = v
+	}
+	b, file, err := readSettingsFile(w.path)
+	if err != nil {
+		w.log.Error("server.yaml cannot be read; running on what it said before", "err", err)
+		return
+	}
+	hash := sha256.Sum256(b)
+	if version == w.cur.version && hash == w.cur.fileHash {
+		return
+	}
+	cluster := w.cur.cluster
+	eff := file
+	if w.st != nil {
+		if version != w.cur.version {
+			doc, err := w.st.ClusterSettings(ctx)
+			if err != nil {
+				return
+			}
+			if cluster, err = config.Parse([]byte(doc.Body)); err != nil {
+				w.log.Error("the cluster's settings do not parse; running on what was in force",
+					"version", doc.Version, "err", err)
+				w.cur.version, w.cur.fileHash = version, hash
+				return
+			}
+		}
+		eff = config.Layer(file, cluster)
+	}
+	cfg, err := config.FromMap(eff)
+	if err != nil {
+		w.log.Error("the changed settings do not load on this machine; running on what was in force",
+			"version", version, "err", err)
+		w.cur.version, w.cur.fileHash = version, hash
+		return
+	}
+	changed := config.Changed(w.cur.eff, eff)
+	w.cur = clusterSettings{cfg: cfg, eff: eff, cluster: cluster, version: version, fileHash: hash}
+	if len(changed) == 0 {
+		return
+	}
+	if w.live != nil && w.live(cfg, changed) {
+		w.log.Info("settings changed and in force", "version", version, "changed", strings.Join(changed, ", "))
+		return
+	}
+	w.restart(fmt.Sprintf("settings version %d changed %s", version, strings.Join(changed, ", ")))
+}
+
+// liveSettings are the keys a control plane puts in force without starting
+// again: what it reads at every use, or not at all.
+var liveSettings = []string{
+	"guard",
+	// Sent to the nodes and this machine's firewall unit as they change.
+	"firewall", "firewall_networks",
+	// Read by server init, node add and backups, not by the server.
+	"backup_recipients", "backup_copy_to",
+}
+
+// onlyLive is whether every changed key is one of liveSettings.
+func onlyLive(changed []string) bool {
+	for _, c := range changed {
+		top, _, _ := strings.Cut(c, ".")
+		if !slices.Contains(liveSettings, top) {
+			return false
+		}
+	}
+	return true
+}
+
+// checkClusterSettings refuses a settings document that would not load,
+// laid over the server.yaml at path, or that sets one machine's keys.
+func checkClusterSettings(path string, body []byte) error {
+	cluster, err := config.Parse(body)
+	if err != nil {
+		return err
+	}
+	if err := config.CheckCluster(cluster); err != nil {
+		return err
+	}
+	_, file, err := readSettingsFile(path)
+	if err != nil {
+		return err
+	}
+	_, err = config.FromMap(config.Layer(file, cluster))
+	return err
+}
+
+// loadEffectiveConfig is what the control plane on this machine runs on:
+// its server.yaml with the cluster's settings from the store laid over it,
+// for the commands beside it that read cluster keys — init, node add,
+// backups. The file alone when the store cannot be reached or holds none.
+func loadEffectiveConfig(path string) (config.Server, error) {
+	_, file, err := readSettingsFile(path)
+	if err != nil {
+		return config.Default(), err
+	}
+	fileCfg, err := config.FromMap(file)
+	if err != nil {
+		return fileCfg, fileError(path, err)
+	}
+	cluster, ok := storedSettings(fileCfg)
+	if !ok {
+		return fileCfg, nil
+	}
+	cfg, err := config.FromMap(config.Layer(file, cluster))
+	if err != nil {
+		return fileCfg, nil //nolint:nilerr // the file still says what this machine is
+	}
+	return cfg, nil
+}
+
+// storedSettings are the cluster's settings, when the store a configuration
+// names can be reached and holds some.
+func storedSettings(cfg config.Server) (map[string]any, bool) {
+	if cfg.Auth.Store == "" {
+		return nil, false
+	}
+	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+	defer cancel()
+	st, err := storepkg.Open(ctx, cfg.Auth.Store)
+	if err != nil {
+		return nil, false
+	}
+	defer st.Close()
+	doc, err := st.ClusterSettings(ctx)
+	if err != nil || doc.Version == 0 {
+		return nil, false
+	}
+	cluster, err := config.Parse([]byte(doc.Body))
+	return cluster, err == nil
+}
+
+// changeStoredSettings changes the cluster's settings from a command run on
+// a control plane (init), when the store holds them: changed is false when
+// it does not, and the file is then where the change belongs.
+func changeStoredSettings(path, note string, change func(doc *settings.Doc) error) (changed bool, err error) {
+	_, file, err := readSettingsFile(path)
+	if err != nil {
+		return false, err
+	}
+	fileCfg, err := config.FromMap(file)
+	if err != nil || fileCfg.Auth.Store == "" {
+		return false, nil //nolint:nilerr // no store to keep them in
+	}
+	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+	defer cancel()
+	st, err := storepkg.Open(ctx, fileCfg.Auth.Store)
+	if err != nil {
+		return false, nil //nolint:nilerr // not set up yet: the file it is
+	}
+	defer st.Close()
+	cur, err := st.ClusterSettings(ctx)
+	if err != nil || cur.Version == 0 {
+		return false, err
+	}
+	doc, err := settings.FromBytes("the cluster's settings", []byte(cur.Body))
+	if err != nil {
+		return false, err
+	}
+	if err := change(doc); err != nil {
+		return false, err
+	}
+	body, err := doc.Bytes()
+	if err != nil {
+		return false, err
+	}
+	if err := checkClusterSettings(path, body); err != nil {
+		return false, err
+	}
+	who := "llmgw"
+	if u := os.Getenv("SUDO_USER"); u != "" {
+		who = u
+	}
+	_, err = st.PutClusterSettings(ctx, cur.Version, string(body), who, note)
+	return err == nil, err
+}

internal/cli/clustersettings_test.go

@@ -0,0 +1,139 @@
+package cli
+
+import (
+	"context"
+	"io"
+	"log/slog"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	storepkg "gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+func settingsFixture(t *testing.T, extra string) (path, store string) {
+	t.Helper()
+	dir := t.TempDir()
+	store = filepath.Join(dir, "llmgw.db")
+	path = filepath.Join(dir, "server.yaml")
+	body := "listen: \":8080\"\nagent_insecure: true\nauth:\n  enabled: true\n  store: " + store +
+		"\nrouter:\n  retries: 2\n" + extra
+	if err := os.WriteFile(path, []byte(body), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	return path, store
+}
+
+var quiet = slog.New(slog.NewTextHandler(io.Discard, nil))
+
+func TestTheFirstStartBringsTheFilesSettingsInWithGuardrails(t *testing.T) {
+	path, store := settingsFixture(t, "")
+	s, err := loadClusterSettings(t.Context(), path, quiet)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if s.version != 1 || s.cfg.Router.Retries != 2 {
+		t.Errorf("version %d retries %d", s.version, s.cfg.Router.Retries)
+	}
+	if len(s.cfg.Guard.Policies) == 0 || s.cfg.Guard.Policies["baseline"].Judge != nil {
+		t.Errorf("a file without guard got %+v, want the baseline without a judge", s.cfg.Guard)
+	}
+	st, _ := storepkg.Open(t.Context(), store)
+	defer st.Close()
+	doc, _ := st.ClusterSettings(t.Context())
+	if strings.Contains(doc.Body, "listen") || strings.Contains(doc.Body, store) || !strings.Contains(doc.Body, "retries: 2") {
+		t.Errorf("the store holds a machine's keys, or lost the cluster's:\n%s", doc.Body)
+	}
+
+	// From then on the store's settings are the cluster's, whatever a
+	// file says.
+	_ = os.WriteFile(path, []byte(strings.Replace(mustRead(t, path), "retries: 2", "retries: 9", 1)), 0o600)
+	again, err := loadClusterSettings(t.Context(), path, quiet)
+	if err != nil || again.version != 1 || again.cfg.Router.Retries != 2 {
+		t.Errorf("second start: version %d retries %d err %v", again.version, again.cfg.Router.Retries, err)
+	}
+}
+
+func mustRead(t *testing.T, p string) string {
+	b, err := os.ReadFile(p)
+	if err != nil {
+		t.Fatal(err)
+	}
+	return string(b)
+}
+
+func TestEveryChangeIsAppliedLiveOrByStartingAgain(t *testing.T) {
+	path, store := settingsFixture(t, "")
+	s, err := loadClusterSettings(t.Context(), path, quiet)
+	if err != nil {
+		t.Fatal(err)
+	}
+	st, _ := storepkg.Open(t.Context(), store)
+	defer st.Close()
+	var lived [][]string
+	var restarts []string
+	w := &settingsWatch{path: path, st: st, cur: s, log: quiet,
+		live: func(_ config.Server, changed []string) bool {
+			lived = append(lived, changed)
+			return onlyLive(changed)
+		},
+		restart: func(why string) { restarts = append(restarts, why) },
+	}
+	ctx := context.Background()
+	put := func(body string) {
+		t.Helper()
+		v, _ := st.ClusterSettingsVersion(ctx)
+		if _, err := st.PutClusterSettings(ctx, v, body, "test", ""); err != nil {
+			t.Fatal(err)
+		}
+	}
+	cur, _ := st.ClusterSettings(ctx)
+
+	w.check(ctx) // nothing changed
+	if len(lived)+len(restarts) != 0 {
+		t.Fatalf("acted on no change: %v %v", lived, restarts)
+	}
+
+	put(strings.Replace(cur.Body, "injection: flag", "injection: block", 1))
+	w.check(ctx)
+	if len(restarts) != 0 || len(lived) != 1 || lived[0][0] != "guard" {
+		t.Fatalf("a guard change: lived %v restarts %v", lived, restarts)
+	}
+
+	cur, _ = st.ClusterSettings(ctx)
+	put(strings.Replace(cur.Body, "retries: 2", "retries: 4", 1))
+	w.check(ctx)
+	if len(restarts) != 1 || !strings.Contains(restarts[0], "router") || w.cur.cfg.Router.Retries != 4 {
+		t.Fatalf("a router change: restarts %v retries %d", restarts, w.cur.cfg.Router.Retries)
+	}
+
+	// A machine's own key, in its file.
+	_ = os.WriteFile(path, []byte(strings.Replace(mustRead(t, path), `":8080"`, `":8090"`, 1)), 0o600)
+	w.check(ctx)
+	if len(restarts) != 2 || !strings.Contains(restarts[1], "listen") {
+		t.Fatalf("a listener change: %v", restarts)
+	}
+
+	// Settings that do not load are not put in force.
+	cur, _ = st.ClusterSettings(ctx)
+	put(cur.Body + "control:\n  timezone: Mars/Olympus\n")
+	w.check(ctx)
+	if len(restarts) != 2 || w.cur.cfg.Control.Timezone != "" {
+		t.Errorf("bad settings were applied: %v %q", restarts, w.cur.cfg.Control.Timezone)
+	}
+}
+
+func TestAClusterDocumentCannotSetAMachinesKeys(t *testing.T) {
+	path, _ := settingsFixture(t, "")
+	if err := checkClusterSettings(path, []byte("listen: \":1\"\n")); err == nil {
+		t.Error("listen was accepted")
+	}
+	if err := checkClusterSettings(path, []byte("control:\n  timezone: Mars/Olympus\n")); err == nil {
+		t.Error("a timezone that is not one was accepted")
+	}
+	if err := checkClusterSettings(path, []byte("router:\n  retries: 3\n")); err != nil {
+		t.Error(err)
+	}
+}

internal/cli/controlplanes.go

@@ -46,7 +46,7 @@ func newControlPlaneProbe(st *store.DB, listen string) *controlPlaneProbe {
 	self, _ := os.Hostname()
 	return &controlPlaneProbe{st: st, self: self,
 		leaderURL: func(host string) string { return "http://" + net.JoinHostPort(host, port) + "/leader" },
-		floating:  floatingAddress(floatingPath),
+		floating:  floatingPath,
 		client:    &http.Client{Timeout: time.Second}}
 }
 

internal/cli/controlplanes_test.go

@@ -132,3 +132,11 @@ func TestTheFloatingAddressIsReadFromWhatServerAddWrote(t *testing.T) {
 		t.Errorf("no file read as %q", got)
 	}
 }
+
+func TestTheProbeReadsTheFloatingAddressFromItsFile(t *testing.T) {
+	// The file, not what it held when the process started: server add
+	// writes it after restarting the control plane it adds.
+	if p := newControlPlaneProbe(nil, ":8080"); p.floating != floatingPath {
+		t.Errorf("the probe reads %q, not %s", p.floating, floatingPath)
+	}
+}

internal/cli/ctl.go

@@ -39,6 +39,14 @@ func newCtlClient(addr string, tokenFiles ...string) *ctlClient {
 	}
 }
 
+// patient is the same client waiting up to d for an answer, for a call the
+// server itself may spend minutes on.
+func (c *ctlClient) patient(d time.Duration) *ctlClient {
+	cp := *c
+	cp.http = &http.Client{Timeout: d}
+	return &cp
+}
+
 func (c *ctlClient) get(ctx context.Context, path string, out any) error {
 	return c.do(ctx, http.MethodGet, path, nil, out)
 }
@@ -108,6 +116,13 @@ func (c *ctlClient) do(ctx context.Context, method, path string, body, out any)
 	if out == nil {
 		return nil
 	}
+	// An older control plane answers a path it does not know with its
+	// dashboard: a page, which read as JSON said only "invalid character '<'".
+	if strings.HasPrefix(resp.Header.Get("Content-Type"), "text/html") {
+		return fmt.Errorf("%s %s answered with a page, not the API: the control plane answering "+
+			"(the leader, through a standby) does not know it — is it on an older release? "+
+			"llmgw ctl status says which it runs", method, path)
+	}
 	return json.NewDecoder(resp.Body).Decode(out)
 }
 
@@ -165,6 +180,7 @@ func newCtlCmd() *cobra.Command {
 		newCapacityCmd(client, &asJSON),
 		newQuotaCmd(client, &asJSON),
 		newGuardCmd(client, &asJSON),
+		newConfigCmd(client, &asJSON),
 		newAuditCmd(client, &asJSON),
 		newNotifyCmd(client, &asJSON),
 		newUsageCmd(client, &asJSON),

internal/cli/ctl_config.go

@@ -0,0 +1,377 @@
+package cli
+
+import (
+	"bytes"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"strconv"
+	"strings"
+	"time"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
+)
+
+func newConfigCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
+	var version int64
+	c := &cobra.Command{
+		Use:   "config",
+		Short: "The cluster's settings: show, change, and go back",
+		Long: "Everything in server.yaml that is not one machine's own — guardrails,\n" +
+			"limits, pricing, routing, engines, notifications, the mirror's policy —\n" +
+			"is the cluster's, kept in the store and the same on every control plane.\n" +
+			"A change is checked, kept as a new version, and in force on every control\n" +
+			"plane within seconds: live where it can be (guard), otherwise each control\n" +
+			"plane starts again on it, standbys first and the leader last.\n\n" +
+			"With no subcommand, prints the settings in force.\n\n" +
+			"A machine's own keys (" + strings.Join(config.LocalKeys(), ", ") + ")\n" +
+			"stay in its /etc/llmgw/server.yaml; a change there is noticed and applied\n" +
+			"on that machine the same way.",
+		Example: "  llmgw ctl config\n" +
+			"  llmgw ctl config get router.retries\n" +
+			"  llmgw ctl config set router.retries 3 --note 'flaky upstream'\n" +
+			"  llmgw ctl config set firewall_networks '[10.20.0.0/16]'\n" +
+			"  llmgw ctl config edit\n" +
+			"  llmgw ctl config history\n" +
+			"  llmgw ctl config rollback 4",
+		Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			path := "/api/v1/settings"
+			if version > 0 {
+				path += "?version=" + strconv.FormatInt(version, 10)
+			}
+			var v adminapi.SettingsView
+			if err := client().get(cmd.Context(), path, &v); err != nil {
+				return err
+			}
+			if *asJSON {
+				return emitJSON(v)
+			}
+			fmt.Fprintf(cmd.ErrOrStderr(), "# version %d, %s by %s%s\n", v.Version, v.At.Local().Format(time.DateTime),
+				orUnknown(v.By), noteSuffix(v.Note))
+			fmt.Fprint(cmd.OutOrStdout(), v.Body)
+			return nil
+		},
+	}
+	c.Flags().Int64Var(&version, "version", 0, "an earlier version")
+	c.AddCommand(newConfigGetCmd(client), newConfigSetCmd(client), newConfigUnsetCmd(client),
+		newConfigEditCmd(client), newConfigHistoryCmd(client, asJSON), newConfigRollbackCmd(client))
+	return c
+}
+
+func orUnknown(s string) string {
+	if s == "" {
+		return "unknown"
+	}
+	return s
+}
+
+func noteSuffix(n string) string {
+	if n == "" {
+		return ""
+	}
+	return ": " + n
+}
+
+// keyPath reads a dotted key, refusing one machine's own.
+func keyPath(key string) ([]string, error) {
+	path := strings.Split(key, ".")
+	if config.IsLocal(path) {
+		return nil, fmt.Errorf("%s is each machine's own: change it in that machine's /etc/llmgw/server.yaml, "+
+			"where a change is applied by itself too", key)
+	}
+	return path, nil
+}
+
+func newConfigGetCmd(client func() *ctlClient) *cobra.Command {
+	return &cobra.Command{
+		Use:   "get KEY",
+		Short: "One key of the cluster's settings, as YAML",
+		Args:  cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			path, err := keyPath(args[0])
+			if err != nil {
+				return err
+			}
+			var v adminapi.SettingsView
+			if err := client().get(cmd.Context(), "/api/v1/settings", &v); err != nil {
+				return err
+			}
+			doc, err := settings.FromBytes("settings", []byte(v.Body))
+			if err != nil {
+				return err
+			}
+			n := doc.Node(path)
+			if n == nil {
+				return fmt.Errorf("%s is not set in the cluster's settings; its default applies (docs/reference.md)", args[0])
+			}
+			sub, err := settings.FromBytes("value", nil)
+			if err != nil {
+				return err
+			}
+			if err := sub.SetNode([]string{"v"}, n); err != nil {
+				return err
+			}
+			b, err := sub.Bytes()
+			if err != nil {
+				return err
+			}
+			out := strings.TrimPrefix(string(b), "v:")
+			fmt.Fprintln(cmd.OutOrStdout(), strings.TrimSpace(dedent(out)))
+			return nil
+		},
+	}
+}
+
+// dedent takes off the two spaces a block nested under one key carries.
+func dedent(s string) string {
+	lines := strings.Split(s, "\n")
+	for i, l := range lines {
+		lines[i] = strings.TrimPrefix(l, "  ")
+	}
+	return strings.Join(lines, "\n")
+}
+
+// changeSettings reads the newest settings, changes them, and puts the
+// result in force, saying what that does.
+func changeSettings(cmd *cobra.Command, client func() *ctlClient, note string,
+	change func(doc *settings.Doc) error,
+) error {
+	ctx := cmd.Context()
+	var cur adminapi.SettingsView
+	if err := client().get(ctx, "/api/v1/settings", &cur); err != nil {
+		return err
+	}
+	doc, err := settings.FromBytes("settings", []byte(cur.Body))
+	if err != nil {
+		return err
+	}
+	if err := change(doc); err != nil {
+		return err
+	}
+	b, err := doc.Bytes()
+	if err != nil {
+		return err
+	}
+	return putSettings(cmd, client, cur, b, note)
+}
+
+func putSettings(cmd *cobra.Command, client func() *ctlClient, cur adminapi.SettingsView, body []byte, note string) error {
+	before, _ := config.Parse([]byte(cur.Body))
+	after, err := config.Parse(body)
+	if err != nil {
+		return err
+	}
+	changed := config.Changed(before, after)
+	if len(changed) == 0 {
+		fmt.Fprintln(cmd.OutOrStdout(), "nothing changed")
+		return nil
+	}
+	req, _ := json.Marshal(adminapi.SettingsRequest{Base: cur.Version, Body: string(body), Note: note})
+	var v adminapi.SettingsView
+	if err := client().put(cmd.Context(), "/api/v1/settings", req, &v); err != nil {
+		return err
+	}
+	fmt.Fprintf(cmd.OutOrStdout(), "version %d: %s\n", v.Version, strings.Join(changed, ", "))
+	if onlyLive(changed) {
+		fmt.Fprintln(cmd.OutOrStdout(), "in force on every control plane within seconds")
+	} else {
+		fmt.Fprintln(cmd.OutOrStdout(), "every control plane starts again on it within seconds, standbys first, "+
+			"the leader half a minute later; engines keep serving")
+	}
+	return nil
+}
+
+func newConfigSetCmd(client func() *ctlClient) *cobra.Command {
+	var note string
+	c := &cobra.Command{
+		Use:   "set KEY VALUE",
+		Short: "Change one key of the cluster's settings; VALUE is YAML",
+		Example: "  llmgw ctl config set router.retries 3\n" +
+			"  llmgw ctl config set guard.default '[baseline, finance]'\n" +
+			"  llmgw ctl config set notify.webhooks '[{url: https://hooks.example/llmgw}]'",
+		Args: cobra.ExactArgs(2),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			path, err := keyPath(args[0])
+			if err != nil {
+				return err
+			}
+			return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
+				return doc.SetYAML(path, args[1])
+			})
+		},
+	}
+	c.Flags().StringVar(&note, "note", "", "why, kept with the version")
+	return c
+}
+
+func newConfigUnsetCmd(client func() *ctlClient) *cobra.Command {
+	var note string
+	c := &cobra.Command{
+		Use:   "unset KEY",
+		Short: "Remove one key from the cluster's settings, so its default applies",
+		Args:  cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			path, err := keyPath(args[0])
+			if err != nil {
+				return err
+			}
+			return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
+				if doc.Node(path) == nil {
+					return fmt.Errorf("%s is not set", args[0])
+				}
+				doc.Unset(path)
+				return nil
+			})
+		},
+	}
+	c.Flags().StringVar(&note, "note", "", "why, kept with the version")
+	return c
+}
+
+func newConfigEditCmd(client func() *ctlClient) *cobra.Command {
+	var note string
+	c := &cobra.Command{
+		Use:   "edit",
+		Short: "Edit the cluster's settings in $EDITOR",
+		Long: "Opens the settings in force in $EDITOR (vi when unset). Saved, they are\n" +
+			"checked and kept as a new version; a document that does not load is\n" +
+			"opened again with the reason, until it does or is left unchanged.",
+		Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			var cur adminapi.SettingsView
+			if err := client().get(cmd.Context(), "/api/v1/settings", &cur); err != nil {
+				return err
+			}
+			dir, err := os.MkdirTemp("", "llmgw-settings")
+			if err != nil {
+				return err
+			}
+			defer os.RemoveAll(dir)
+			file := filepath.Join(dir, "settings.yaml")
+			body := []byte(cur.Body)
+			problem := ""
+			for {
+				text := body
+				if problem != "" {
+					text = append([]byte("# NOT SAVED: "+strings.ReplaceAll(problem, "\n", " ")+"\n"), stripProblem(body)...)
+				}
+				if err := os.WriteFile(file, text, 0o600); err != nil {
+					return err
+				}
+				if err := runEditor(file, cmd.InOrStdin(), cmd.OutOrStdout(), cmd.ErrOrStderr()); err != nil {
+					return err
+				}
+				edited, err := os.ReadFile(file) //nolint:gosec // written above
+				if err != nil {
+					return err
+				}
+				edited = stripProblem(edited)
+				if bytes.Equal(bytes.TrimSpace(edited), bytes.TrimSpace([]byte(cur.Body))) {
+					fmt.Fprintln(cmd.OutOrStdout(), "nothing changed")
+					return nil
+				}
+				err = putSettings(cmd, client, cur, edited, note)
+				if err == nil {
+					return nil
+				}
+				if strings.Contains(err.Error(), "changed since they were read") {
+					// Somebody else's change came first: opening the editor
+					// again on the old base would only be refused again.
+					return fmt.Errorf("%w; your edit is not saved", err)
+				}
+				body, problem = edited, err.Error()
+			}
+		},
+	}
+	c.Flags().StringVar(&note, "note", "", "why, kept with the version")
+	return c
+}
+
+func stripProblem(b []byte) []byte {
+	if bytes.HasPrefix(b, []byte("# NOT SAVED: ")) {
+		if i := bytes.IndexByte(b, '\n'); i >= 0 {
+			return b[i+1:]
+		}
+	}
+	return b
+}
+
+func runEditor(file string, in io.Reader, out, errOut io.Writer) error {
+	editor := os.Getenv("VISUAL")
+	if editor == "" {
+		editor = os.Getenv("EDITOR")
+	}
+	if editor == "" {
+		editor = "vi"
+	}
+	parts := strings.Fields(editor)
+	c := exec.Command(parts[0], append(parts[1:], file)...) //nolint:gosec // the operator's own editor
+	c.Stdin, c.Stdout, c.Stderr = in, out, errOut
+	if err := c.Run(); err != nil {
+		return fmt.Errorf("%s: %w", editor, err)
+	}
+	return nil
+}
+
+func newConfigHistoryCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
+	var limit int
+	c := &cobra.Command{
+		Use:   "history",
+		Short: "Versions of the cluster's settings, newest first",
+		Args:  cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			var list []adminapi.SettingsView
+			if err := client().get(cmd.Context(), "/api/v1/settings/history?limit="+strconv.Itoa(limit), &list); err != nil {
+				return err
+			}
+			if *asJSON {
+				return emitJSON(list)
+			}
+			w := tabTo(cmd.OutOrStdout())
+			fmt.Fprintln(w, "VERSION\tWHEN\tBY\tNOTE")
+			for _, v := range list {
+				fmt.Fprintf(w, "%d\t%s\t%s\t%s\n", v.Version, v.At.Local().Format(time.DateTime), orUnknown(v.By), v.Note)
+			}
+			return w.Flush()
+		},
+	}
+	c.Flags().IntVar(&limit, "limit", 20, "how many")
+	return c
+}
+
+func newConfigRollbackCmd(client func() *ctlClient) *cobra.Command {
+	var note string
+	c := &cobra.Command{
+		Use:   "rollback VERSION",
+		Short: "Put an earlier version of the cluster's settings back in force, as a new version",
+		Args:  cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			n, err := strconv.ParseInt(args[0], 10, 64)
+			if err != nil || n < 1 {
+				return errors.New("VERSION is a number from 'llmgw ctl config history'")
+			}
+			var cur, old adminapi.SettingsView
+			if err := client().get(cmd.Context(), "/api/v1/settings", &cur); err != nil {
+				return err
+			}
+			if err := client().get(cmd.Context(), "/api/v1/settings?version="+args[0], &old); err != nil {
+				return err
+			}
+			if note == "" {
+				note = "back to version " + args[0]
+			}
+			return putSettings(cmd, client, cur, []byte(old.Body), note)
+		},
+	}
+	c.Flags().StringVar(&note, "note", "", "why, kept with the version")
+	return c
+}

internal/cli/ctl_config_test.go

@@ -0,0 +1,22 @@
+package cli
+
+import (
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+)
+
+func TestAPageInsteadOfTheAPIIsSaidPlainly(t *testing.T) {
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		w.Header().Set("Content-Type", "text/html; charset=utf-8")
+		_, _ = w.Write([]byte("<!doctype html><title>llmgw</title>"))
+	}))
+	defer srv.Close()
+	c := newCtlClient(strings.TrimPrefix(srv.URL, "http://"), "")
+	var out map[string]any
+	err := c.get(t.Context(), "/api/v1/settings", &out)
+	if err == nil || !strings.Contains(err.Error(), "older release") || strings.Contains(err.Error(), "invalid character") {
+		t.Errorf("%v", err)
+	}
+}

internal/cli/ctl_request.go

@@ -76,8 +76,11 @@ func newRequestCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 				fmt.Fprintf(out, "tags      %s\n", strings.Join(kv, ", "))
 			}
 			if v.GuardAction != "" {
-				fmt.Fprintf(out, "guard     %s by policy %s (%s), %d ms\n", v.GuardAction, v.GuardPolicy,
-					or(v.GuardCategory, "-"), v.GuardMs)
+				what := or(v.GuardCategory, "-")
+				if v.GuardRule != "" {
+					what += ": " + v.GuardRule
+				}
+				fmt.Fprintf(out, "guard     %s by policy %s (%s), %d ms\n", v.GuardAction, v.GuardPolicy, what, v.GuardMs)
 			} else if v.GuardMs > 0 {
 				fmt.Fprintf(out, "guard     allowed, %d ms\n", v.GuardMs)
 			}

internal/cli/ctl_upgrade.go

@@ -68,14 +68,26 @@ func newUpgradeCmd(client func() *ctlClient) *cobra.Command {
 				}
 			}
 			if !skipCP {
-				if othersCP {
-					if err := upgradeOtherControlPlanes(ctx, out, client, v, releases, sshUser); err != nil {
+				// The leader last, wherever this runs: every other control
+				// plane is a standby, upgraded with nothing moving, and the
+				// leader's restart hands over once, to a machine already on
+				// the release.
+				others := func() error {
+					if !othersCP {
+						return nil
+					}
+					return upgradeOtherControlPlanes(ctx, out, client, v, releases, sshUser)
+				}
+				self := func() error { return upgradeControlPlane(ctx, out, client, v, releases, serverSvc) }
+				order := []func() error{others, self}
+				if !thisLeads(ctx, client) {
+					order = []func() error{self, others}
+				}
+				for _, step := range order {
+					if err := step(); err != nil {
 						return err
 					}
 				}
-				if err := upgradeControlPlane(ctx, out, client, v, releases, serverSvc); err != nil {
-					return err
-				}
 			}
 			return upgradeNodes(ctx, out, client, v, nodes, perNode, parallel)
 		},
@@ -92,10 +104,23 @@ func newUpgradeCmd(client func() *ctlClient) *cobra.Command {
 	return c
 }
 
+// thisLeads says whether this machine's control plane leads; one that does
+// not say, alone or older, is taken to.
+func thisLeads(ctx context.Context, client func() *ctlClient) bool {
+	var sum adminapi.Summary
+	if err := client().get(ctx, "/api/v1/summary", &sum); err != nil {
+		return true
+	}
+	for _, cp := range sum.ControlPlanes.Members {
+		if cp.This {
+			return cp.State == "leading"
+		}
+	}
+	return true
+}
+
 // upgradeOtherControlPlanes moves every other control plane of the cluster to
-// v, over ssh with `llmgw upgrade`, before this machine: each is a standby, or
-// becomes one when this one is restarted, and the leadership should move to
-// a machine already on the release. Nothing is changed here if one cannot be
+// v, over ssh with `llmgw upgrade`. Nothing is changed here if one cannot be
 // reached.
 func upgradeOtherControlPlanes(ctx context.Context, out io.Writer, client func() *ctlClient, v, releases, user string) error {
 	var sum adminapi.Summary
@@ -127,8 +152,20 @@ func upgradeOtherControlPlanes(ctx context.Context, out io.Writer, client func()
 	return nil
 }
 
+// controlPlaneStep is what this machine needs to run v: the binary installed,
+// the service restarted, both, or neither. The binary is this command's own;
+// running is what the server says it runs, "" from one too old to say. A
+// binary put in place by hand is not a server running it.
+func controlPlaneStep(binary, running, v string) (install, restart bool) {
+	install = binary != v
+	return install, install || running != v
+}
+
 func upgradeControlPlane(ctx context.Context, out io.Writer, client func() *ctlClient, v, releases, svc string) error {
-	if version.Version == v {
+	var sum adminapi.Summary
+	_ = client().get(ctx, "/api/v1/summary", &sum)
+	install, restart := controlPlaneStep(version.Version, sum.Version, v)
+	if !restart {
 		fmt.Fprintf(out, "this machine already runs %s\n", v)
 		return nil
 	}
@@ -140,12 +177,20 @@ func upgradeControlPlane(ctx context.Context, out io.Writer, client func() *ctlC
 	if err != nil {
 		return err
 	}
-	fmt.Fprintf(out, "== this machine: %s -> %s\n", version.Version, v)
-	backup, err := release.Install(ctx, nil, releases, v, runtime.GOOS, runtime.GOARCH, self)
-	if err != nil {
-		return err
+	backup := "the binary installed by hand before it"
+	if install {
+		fmt.Fprintf(out, "== this machine: %s -> %s\n", version.Version, v)
+		if backup, err = release.Install(ctx, nil, releases, v, runtime.GOOS, runtime.GOARCH, self); err != nil {
+			return err
+		}
+		fmt.Fprintf(out, "   installed, checksum verified; the old binary is %s\n", backup)
+	} else {
+		running := sum.Version
+		if running == "" {
+			running = "an older release"
+		}
+		fmt.Fprintf(out, "== this machine: %s is installed, and %s still runs it; restarting %s\n", v, running, svc)
 	}
-	fmt.Fprintf(out, "   installed, checksum verified; the old binary is %s\n", backup)
 	var before []adminapi.NodeView
 	_ = client().get(ctx, "/api/v1/nodes", &before)
 	if b, err := exec.CommandContext(ctx, "systemctl", "restart", svc).CombinedOutput(); err != nil { //nolint:gosec // fixed command
@@ -245,7 +290,10 @@ func upgradeWave(ctx context.Context, out io.Writer, client func() *ctlClient, v
 	body, _ := json.Marshal(adminapi.UpgradeRequest{Version: v})
 	var failed []string
 	for _, n := range wave {
-		if err := client().post(ctx, "/api/v1/nodes/"+n.ID+"/upgrade", body, nil); err != nil {
+		// The control plane waits for the agent to fetch and check the
+		// release, up to five minutes; fifteen seconds gave up on a node
+		// that was upgrading.
+		if err := client().patient(6*time.Minute).post(ctx, "/api/v1/nodes/"+n.ID+"/upgrade", body, nil); err != nil {
 			failed = append(failed, fmt.Sprintf("%s: %v", n.ID, err))
 		}
 	}

internal/cli/ctl_upgrade_test.go

@@ -111,3 +111,42 @@ func TestAnUpgradeGoesInWavesThatGrowUpToTheLimit(t *testing.T) {
 		t.Errorf("waves %s:\n%s", got, out.String())
 	}
 }
+
+func TestAnUpgradeKnowsWhetherItRunsOnTheLeader(t *testing.T) {
+	sum := adminapi.Summary{}
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		_ = json.NewEncoder(w).Encode(sum)
+	}))
+	defer srv.Close()
+	client := func() *ctlClient { return newCtlClient(srv.URL) }
+	if !thisLeads(t.Context(), client) {
+		t.Error("a control plane with no rival does not lead")
+	}
+	sum.ControlPlanes.Members = []adminapi.ControlPlane{{Name: "a", State: "leading"}, {Name: "b", State: "standing by", This: true}}
+	if thisLeads(t.Context(), client) {
+		t.Error("a standby took itself for the leader: it would upgrade the leader before itself")
+	}
+	sum.ControlPlanes.Members[0].This, sum.ControlPlanes.Members[1].This = true, false
+	if !thisLeads(t.Context(), client) {
+		t.Error("the leader took itself for a standby")
+	}
+}
+
+func TestABinaryInstalledByHandIsNotAServerRunningIt(t *testing.T) {
+	// The binary put in place with install, llmgw-server not restarted: the
+	// upgrade said "already runs" and left the old server running.
+	for _, c := range []struct {
+		binary, running  string
+		install, restart bool
+	}{
+		{"v1.0.0", "v1.0.0", true, true},  // the usual: fetch it, restart
+		{"v1.0.1", "v1.0.0", false, true}, // installed by hand: restart only
+		{"v1.0.1", "", false, true},       // a server too old to say
+		{"v1.0.1", "v1.0.1", false, false},
+	} {
+		install, restart := controlPlaneStep(c.binary, c.running, "v1.0.1")
+		if install != c.install || restart != c.restart {
+			t.Errorf("binary %s, running %q: install %v restart %v", c.binary, c.running, install, restart)
+		}
+	}
+}

internal/cli/firewallcmd.go

@@ -0,0 +1,176 @@
+package cli
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"net/netip"
+	"os"
+	"time"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
+)
+
+// firewallStateFile is where a control plane writes what the cluster's
+// settings say of the firewall, for llmgw-firewall-sync to apply as root:
+// the server itself runs as llmgw and cannot change the ruleset.
+const firewallStateFile = "/var/lib/llmgw/firewall.json"
+
+// firewallState is what is written there.
+type firewallState struct {
+	Enabled  bool     `json:"enabled"`
+	Networks []string `json:"networks,omitempty"`
+}
+
+// writeFirewallState records the settings' firewall for this machine's sync
+// unit. Without the state directory — a server not run by systemd — there
+// is no unit to read it, and nothing is written.
+func writeFirewallState(cfg config.Server) error {
+	if _, err := os.Stat("/var/lib/llmgw"); err != nil {
+		return nil //nolint:nilerr // not a control plane set up by init
+	}
+	b, _ := json.Marshal(firewallState{Enabled: cfg.Firewall.Enabled, Networks: cfg.FirewallNetworks})
+	if old, err := os.ReadFile(firewallStateFile); err == nil && string(old) == string(b) {
+		return nil
+	}
+	tmp := firewallStateFile + ".new"
+	if err := os.WriteFile(tmp, b, 0o644); err != nil {
+		return err
+	}
+	return os.Rename(tmp, firewallStateFile)
+}
+
+// The units that apply it: a path unit watching the file, and the oneshot it
+// starts.
+const (
+	firewallSyncPathUnit = "llmgw-firewall-sync.path"
+	firewallSyncUnitFile = `[Unit]
+Description=llmgw firewall: apply the cluster's settings on this control plane
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/bin/llmgw firewall sync
+`
+	firewallSyncPathFile = `[Unit]
+Description=llmgw firewall: follow the cluster's settings on this control plane
+
+[Path]
+PathChanged=` + firewallStateFile + `
+
+[Install]
+WantedBy=multi-user.target
+`
+)
+
+func newFirewallCmd() *cobra.Command {
+	c := &cobra.Command{
+		Use:   "firewall",
+		Short: "llmgw's firewall on this machine",
+		Long: "Whether llmgw keeps a firewall is the cluster's setting firewall.enabled\n" +
+			"(llmgw ctl config set firewall.enabled false). Nodes are told by the\n" +
+			"control plane; a control plane writes it to " + firewallStateFile + ",\n" +
+			"which " + firewallSyncPathUnit + " applies with 'llmgw firewall sync'.",
+	}
+	var install bool
+	sync := &cobra.Command{
+		Use:   "sync",
+		Short: "Apply what the cluster's settings say of the firewall, here (root)",
+		Args:  cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			if os.Geteuid() != 0 {
+				return errors.New("the firewall is changed as root: run it with sudo")
+			}
+			if install {
+				if err := installFirewallSync(cmd.Context(), firewall.Local{}); err != nil {
+					return err
+				}
+				fmt.Fprintln(cmd.OutOrStdout(), firewallSyncPathUnit+" follows the cluster's settings here")
+				if _, err := os.Stat(firewallStateFile); err != nil {
+					return nil // the server writes it when it starts
+				}
+			}
+			b, err := os.ReadFile(firewallStateFile)
+			if err != nil {
+				return err
+			}
+			var st firewallState
+			if err := json.Unmarshal(b, &st); err != nil {
+				return fmt.Errorf("%s: %w", firewallStateFile, err)
+			}
+			ctx, cancel := context.WithTimeout(cmd.Context(), 2*time.Minute)
+			defer cancel()
+			msg, err := syncFirewall(ctx, firewall.Local{}, st)
+			if err != nil {
+				return err
+			}
+			fmt.Fprintln(cmd.OutOrStdout(), msg)
+			return nil
+		},
+	}
+	sync.Flags().BoolVar(&install, "install", false, "first install the units that run this whenever the settings change")
+	c.AddCommand(sync)
+	return c
+}
+
+// installFirewallSync writes the sync units on a machine and has the path
+// unit follow the state file.
+func installFirewallSync(ctx context.Context, r firewall.Runner) error {
+	body := firewallSyncUnitFile + "\x00" + firewallSyncPathFile
+	out, err := r.Run(ctx, `set -e
+t=$(mktemp -d); trap 'rm -rf "$t"' EXIT
+cat > "$t/all"
+python3 - "$t" <<'PY'
+import sys
+d = sys.argv[1]
+a, b = open(d + "/all", "rb").read().split(b"\x00")
+open(d + "/service", "wb").write(a)
+open(d + "/path", "wb").write(b)
+PY
+sudo install -m 644 "$t/service" /etc/systemd/system/llmgw-firewall-sync.service
+sudo install -m 644 "$t/path" /etc/systemd/system/`+firewallSyncPathUnit+`
+sudo systemctl daemon-reload
+sudo systemctl enable --now `+firewallSyncPathUnit+` >/dev/null 2>&1`, []byte(body))
+	if err != nil {
+		return fmt.Errorf("installing %s: %v: %s", firewallSyncPathUnit, err, string(out))
+	}
+	return nil
+}
+
+// syncFirewall puts a firewall state in force on this machine.
+func syncFirewall(ctx context.Context, r firewall.Runner, st firewallState) (string, error) {
+	if !st.Enabled {
+		had, err := firewall.Disable(ctx, r)
+		if err != nil {
+			return "", err
+		}
+		if had {
+			return "llmgw's firewall is off here", nil
+		}
+		return "llmgw's firewall was already off here", nil
+	}
+	loaded, err := firewall.Enable(ctx, r)
+	if err != nil {
+		return "", err
+	}
+	if !loaded {
+		return "no llmgw rules on this machine to put back; 'llmgw server init' writes them", nil
+	}
+	var extra []netip.Prefix
+	for _, n := range st.Networks {
+		if p, err := netip.ParsePrefix(n); err == nil {
+			extra = append(extra, p)
+		} else if a, err := netip.ParseAddr(n); err == nil {
+			extra = append(extra, netip.PrefixFrom(a, a.BitLen()))
+		}
+	}
+	if len(extra) > 0 {
+		if _, err := firewall.Install(ctx, r, firewall.Spec{Allow: extra}); err != nil {
+			return "", err
+		}
+	}
+	return "llmgw's firewall is on here", nil
+}

internal/cli/firewallcmd_test.go

@@ -0,0 +1,51 @@
+package cli
+
+import (
+	"context"
+	"strings"
+	"testing"
+)
+
+type scriptLog struct {
+	scripts []string
+	reply   func(script string) string
+}
+
+func (s *scriptLog) Run(_ context.Context, script string, _ []byte) ([]byte, error) {
+	s.scripts = append(s.scripts, script)
+	if s.reply != nil {
+		return []byte(s.reply(script)), nil
+	}
+	return nil, nil
+}
+
+func TestAControlPlaneFollowsTheFirewallSetting(t *testing.T) {
+	off := &scriptLog{}
+	if _, err := syncFirewall(context.Background(), off, firewallState{Enabled: false}); err != nil {
+		t.Fatal(err)
+	}
+	if !strings.Contains(strings.Join(off.scripts, "\n"), "nft delete table inet llmgw") {
+		t.Errorf("off did not take the table out: %q", off.scripts)
+	}
+
+	none := &scriptLog{reply: func(s string) string {
+		if strings.Contains(s, "echo loaded") {
+			return "none\n"
+		}
+		return ""
+	}}
+	msg, err := syncFirewall(context.Background(), none, firewallState{Enabled: true})
+	if err != nil || !strings.Contains(msg, "server init") {
+		t.Errorf("on, with no rules here: %q %v", msg, err)
+	}
+
+	on := &scriptLog{reply: func(s string) string {
+		if strings.Contains(s, "echo loaded") {
+			return "loaded\n"
+		}
+		return ""
+	}}
+	if msg, err := syncFirewall(context.Background(), on, firewallState{Enabled: true}); err != nil || msg != "llmgw's firewall is on here" {
+		t.Errorf("on: %q %v", msg, err)
+	}
+}

internal/cli/guard.go

@@ -3,18 +3,21 @@ package cli
 import (
 	"encoding/json"
 	"fmt"
+	"strconv"
 	"strings"
 
 	"github.com/spf13/cobra"
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
 )
 
 func newGuardCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 	c := &cobra.Command{
 		Use:   "guard",
 		Short: "Guardrail policies: which apply where",
-		Long: "Policies are defined in server.yaml under guard.policies — what requests and\n" +
+		Long: "Policies are defined in the cluster's settings under guard.policies\n" +
+			"(llmgw ctl config edit; in force everywhere within seconds) — what requests and\n" +
 			"answers are checked for (personal data, secrets, prompt injection, patterns,\n" +
 			"tools) and what is done: flag, redact or block. guard.default applies to\n" +
 			"every request; attach others to an organization, team, project, user or\n" +
@@ -23,6 +26,35 @@ func newGuardCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 			"Targets are written org:Acme, team:Acme/ml, project:Acme/ml/training (or\n" +
 			"with IDs), key:KEY-ID or user:USER-ID. An attachment applies at once.",
 	}
+	c.AddCommand(newGuardJudgeCmd(client, asJSON))
+	for _, sw := range []struct {
+		use, short string
+		on         bool
+	}{
+		{"on", "Turn the guardrails on: every policy checks again", true},
+		{"off", "Turn the guardrails off: no request is checked, whatever the policies say", false},
+	} {
+		var note string
+		cmd := &cobra.Command{
+			Use:   sw.use,
+			Short: sw.short,
+			Long: "guard.enabled in the cluster's settings: in force on every control plane\n" +
+				"within seconds, without a restart. The policies and where they are attached\n" +
+				"stay as they are, and come back as they were when turned on again. While\n" +
+				"off, llmgw ctl status says so.",
+			Args: cobra.NoArgs,
+			RunE: func(cmd *cobra.Command, _ []string) error {
+				if note == "" {
+					note = "guardrails " + sw.use
+				}
+				return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
+					return doc.SetYAML([]string{"guard", "enabled"}, strconv.FormatBool(sw.on))
+				})
+			},
+		}
+		cmd.Flags().StringVar(&note, "note", "", "why, kept with the settings' version")
+		c.AddCommand(cmd)
+	}
 	c.AddCommand(&cobra.Command{
 		Use:   "list",
 		Short: "The policies, and where each is attached",
@@ -36,6 +68,9 @@ func newGuardCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 				return json.NewEncoder(cmd.OutOrStdout()).Encode(v)
 			}
 			out := cmd.OutOrStdout()
+			if !v.Enabled {
+				fmt.Fprintln(out, "The guardrails are OFF: no request is checked (llmgw ctl guard on).")
+			}
 			fmt.Fprintf(out, "Policies: %s\n", strings.Join(v.Policies, ", "))
 			def := strings.Join(v.Default, ", ")
 			if def == "" {

internal/cli/guard_judge.go

@@ -0,0 +1,225 @@
+package cli
+
+import (
+	"encoding/json"
+	"errors"
+	"fmt"
+	"strings"
+	"time"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
+)
+
+// The judge the guardrails are installed with: Meta's Llama Guard 3 1B,
+// from the repository server init fetches it from.
+const (
+	guardModelRepo = "QuantFactory/Llama-Guard-3-1B-GGUF"
+	guardModelFile = "Llama-Guard-3-1B.Q4_K_M.gguf"
+)
+
+func newGuardJudgeCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
+	c := &cobra.Command{
+		Use:   "judge",
+		Short: "The judge models the policies ask: whether each can answer, and serving one",
+		Long: "A policy with a judge has a model read what patterns cannot. The judge must\n" +
+			"be in the catalog under the name the policy gives, as a judge (role:\n" +
+			"guard), with a replica ready; otherwise what the policy judges is refused\n" +
+			"(fail: closed, guard_unavailable) or served unjudged (fail: open).",
+	}
+	c.AddCommand(newGuardJudgeStatusCmd(client, asJSON), newGuardJudgeEnableCmd(client), newGuardJudgeAskCmd(client, asJSON))
+	return c
+}
+
+func newGuardJudgeStatusCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
+	return &cobra.Command{
+		Use:   "status",
+		Short: "Each judge the policies ask, and what is wrong with it",
+		Args:  cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			var js []adminapi.JudgeView
+			if err := client().get(cmd.Context(), "/api/v1/guard/judges", &js); err != nil {
+				return err
+			}
+			if *asJSON {
+				return emitJSON(js)
+			}
+			out := cmd.OutOrStdout()
+			if len(js) == 0 {
+				fmt.Fprintln(out, "no policy asks a judge")
+				return nil
+			}
+			w := tabTo(out)
+			fmt.Fprintln(w, "JUDGE\tPOLICIES\tIN CATALOG\tAS JUDGE\tREADY\tSTATE")
+			for _, j := range js {
+				state := "answering"
+				if j.Problem != "" {
+					state = j.Problem
+				}
+				fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%d/%d\t%s\n", j.Model, strings.Join(j.Policies, ","),
+					yesNo(j.InCatalog), yesNo(j.Judge), j.Ready, j.Wanted, state)
+			}
+			if err := w.Flush(); err != nil {
+				return err
+			}
+			strays, _ := strayJudges(cmd, client, js)
+			for _, s := range strays {
+				fmt.Fprintf(out, "\n%s looks like a judge's weights but is served as a model clients use, and no "+
+					"policy asks it: llmgw ctl model remove %s\n", s, s)
+			}
+			return nil
+		},
+	}
+}
+
+// strayJudges are models whose weights are a judge's but which are in the
+// catalog as models clients use, under a name no policy asks: what serving
+// Llama Guard from the mirror page did before it knew better.
+func strayJudges(cmd *cobra.Command, client func() *ctlClient, js []adminapi.JudgeView) ([]string, error) {
+	var models []adminapi.ModelView
+	if err := client().get(cmd.Context(), "/api/v1/models", &models); err != nil {
+		return nil, err
+	}
+	asked := map[string]bool{}
+	for _, j := range js {
+		asked[j.Model] = true
+	}
+	var out []string
+	for _, m := range models {
+		if !asked[m.Name] && strings.Contains(strings.ToLower(m.Source), "llama-guard") {
+			out = append(out, m.Name)
+		}
+	}
+	return out, nil
+}
+
+func newGuardJudgeEnableCmd(client func() *ctlClient) *cobra.Command {
+	var (
+		model, repo, file string
+		replicas, cores   int
+		replace, detach   bool
+	)
+	c := &cobra.Command{
+		Use:   "enable",
+		Short: "Serve the judge the policies ask, as a judge: Llama Guard 3 1B on CPUs",
+		Long: "Fetches Llama Guard 3 1B into the mirror (once), adds it to the catalog under\n" +
+			"the name the policies give its judge, as a judge (role: guard,\n" +
+			"format llamaguard3), on CPUs so it takes no card, and waits for a replica\n" +
+			"to answer. With no --model, the judge the policies ask that is missing.\n\n" +
+			"A model of that name already in the catalog as a model clients use is\n" +
+			"refused; --replace removes it first. An offline cluster gets its judge\n" +
+			"from its bundle with server init.",
+		Example: "  llmgw ctl guard judge enable\n" +
+			"  llmgw ctl guard judge enable --replace",
+		Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			ctx := cmd.Context()
+			out := cmd.OutOrStdout()
+			var js []adminapi.JudgeView
+			if err := client().get(ctx, "/api/v1/guard/judges", &js); err != nil {
+				return err
+			}
+			var target *adminapi.JudgeView
+			for i, j := range js {
+				if (model != "" && j.Model == model) || (model == "" && (!j.InCatalog || !j.Judge)) {
+					target = &js[i]
+					break
+				}
+			}
+			switch {
+			case target == nil && model != "":
+				target = &adminapi.JudgeView{Model: model}
+				fmt.Fprintf(out, "no policy asks %s yet; it is served, and a policy's judge.model names it\n", model)
+			case target == nil && len(js) == 0:
+				return errors.New("no policy asks a judge: give one a judge (llmgw ctl config edit, guard.policies.<name>.judge.model)")
+			case target == nil:
+				fmt.Fprintln(out, "every judge the policies ask is in the catalog, as a judge")
+				return nil
+			}
+			if target.InCatalog && target.Judge {
+				fmt.Fprintf(out, "%s is in the catalog as a judge already; llmgw ctl guard judge status says how it is\n", target.Model)
+				return nil
+			}
+			if target.InCatalog && !target.Judge {
+				if !replace {
+					return fmt.Errorf("%s is in the catalog as a model clients use, not as a judge: "+
+						"--replace removes it and serves it again as a judge", target.Model)
+				}
+				var res map[string]string
+				if err := client().delete(ctx, "/api/v1/models/"+target.Model, &res); err != nil {
+					return fmt.Errorf("removing %s: %w", target.Model, err)
+				}
+				fmt.Fprintf(out, "%s removed\n", target.Model)
+			}
+			req := modelserve.Request{
+				Repo: repo, File: file, Name: target.Model, GuardFormat: "llamaguard3",
+				Description: "The guardrails' judge: reads prompts and answers for unsafe content. Not offered to clients.",
+				CPU:         true, Replicas: replicas, CPUCores: cores,
+				CtxPool: 8192, ContextLength: 4096,
+			}
+			body, _ := json.Marshal(req)
+			var job modelserve.Job
+			if err := client().patient(30*time.Second).post(ctx, "/api/v1/models/serve", body, &job); err != nil {
+				return err
+			}
+			if detach {
+				return detached(cmd, job)
+			}
+			if err := followJob(cmd, client, job, "is the judge"); err != nil {
+				return err
+			}
+			if strays, _ := strayJudges(cmd, client, js); len(strays) > 0 {
+				fmt.Fprintf(out, "\nno policy asks %s, served from a judge's weights as a model clients use: "+
+					"llmgw ctl model remove %s\n", strings.Join(strays, ", "), strings.Join(strays, " "))
+			}
+			return nil
+		},
+	}
+	f := c.Flags()
+	f.StringVar(&model, "model", "", "the judge to serve (default: the one the policies ask that is missing)")
+	f.StringVar(&repo, "repo", guardModelRepo, "the registry repository its weights come from")
+	f.StringVar(&file, "file", guardModelFile, "the file in it")
+	f.IntVar(&replicas, "replicas", 2, "replicas: every judged request waits on one")
+	f.IntVar(&cores, "cores", 4, "CPU cores a replica computes on")
+	f.BoolVar(&replace, "replace", false, "remove a model of that name that is not a judge first")
+	f.BoolVar(&detach, "detach", false, "start it and return; llmgw ctl jobs follows it")
+	return c
+}
+
+func newGuardJudgeAskCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
+	var model, answer string
+	c := &cobra.Command{
+		Use:   "ask TEXT",
+		Short: "Have a judge read a text and say what it answered",
+		Long: "For finding out why a request was refused: the judge reads TEXT as a user's\n" +
+			"message (with --answer, ANSWER as the reply to it), past the cache, and its\n" +
+			"answer is shown as it gave it and as the gateway reads it — \"unsafe\" and\n" +
+			"the hazards it named, S6 Specialized Advice and the like. Nothing is kept:\n" +
+			"not in the audit, the usage or the logs.",
+		Example: "  llmgw ctl guard judge ask 'How do I sort a list in Python?'\n" +
+			"  llmgw ctl guard judge ask 'Write a poem' --answer 'Roses are red'",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			body, _ := json.Marshal(adminapi.JudgeAskRequest{Model: model, Text: args[0], Answer: answer})
+			var v adminapi.JudgeAskView
+			if err := client().patient(45*time.Second).post(cmd.Context(), "/api/v1/guard/judges/ask", body, &v); err != nil {
+				return err
+			}
+			if *asJSON {
+				return emitJSON(v)
+			}
+			out := cmd.OutOrStdout()
+			fmt.Fprintf(out, "%s: %s", v.Model, v.Verdict)
+			if len(v.Categories) > 0 {
+				fmt.Fprintf(out, " (%s)", strings.Join(v.Categories, ", "))
+			}
+			fmt.Fprintf(out, ", %d ms\nit answered: %q\n", v.TookMs, v.Raw)
+			return nil
+		},
+	}
+	c.Flags().StringVar(&model, "model", "", "the judge (default: the one the policies ask)")
+	c.Flags().StringVar(&answer, "answer", "", "an assistant's reply to TEXT, judged as output is")
+	return c
+}

internal/cli/leaderforward.go

@@ -7,6 +7,7 @@ import (
 	"errors"
 	"fmt"
 	"log/slog"
+	"net"
 	"net/http"
 	"net/http/httputil"
 	"net/url"
@@ -16,6 +17,7 @@ import (
 
 	"golang.org/x/net/http2"
 
+	"gitea.mynodes.xyz/ilker/llmgw/internal/clientip"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
@@ -53,7 +55,7 @@ type leaderForward struct {
 // process is not it.
 func (f *leaderForward) standby(next http.Handler) http.Handler {
 	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
-		if !strings.HasPrefix(r.URL.Path, "/api/") || f.lead.IsLeader() {
+		if adminStaysHere(r.URL.Path) || f.lead.IsLeader() {
 			next.ServeHTTP(w, r)
 			return
 		}
@@ -90,6 +92,89 @@ func (f *leaderForward) standby(next http.Handler) http.Handler {
 	})
 }
 
+// adminStaysHere are the admin paths about this process rather than the
+// cluster: its profile, its own metrics, its own health. Never forwarded, and
+// never answered for another — pprof dumps the heap, keys and all.
+func adminStaysHere(path string) bool {
+	return strings.HasPrefix(path, "/debug/") || path == "/metrics" || path == "/health"
+}
+
+// publicStaysHere are the client port's paths that must describe this
+// process: a load balancer and keepalived ask them which machine to send
+// traffic to, and a standby saying the leader is ready would draw it.
+func publicStaysHere(path string) bool {
+	return path == "/health" || path == "/ready" || path == "/leader" || path == "/metrics"
+}
+
+// public wraps the client-facing handler: on a standby, everything but its
+// own health goes to the leader's client port — the portal, its admin pages
+// and the API, which on a standby would show an empty cluster and serve
+// nothing. Plain HTTP, as the port is; the leader trusts a control plane's
+// X-Forwarded-For, so a client is still known by its own address.
+func (f *leaderForward) public(next http.Handler, port string) http.Handler {
+	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		// Once only: a request another control plane forwarded is answered
+		// here, leading or not, rather than sent on while the lease moves.
+		if publicStaysHere(r.URL.Path) || f.lead.IsLeader() || r.Header.Get(hdrForwardedBy) != "" {
+			next.ServeHTTP(w, r)
+			return
+		}
+		target, _, err := f.leader(r.Context())
+		if err != nil || target.Hostname() == "" {
+			next.ServeHTTP(w, r)
+			return
+		}
+		host := net.JoinHostPort(target.Hostname(), port)
+		p := &httputil.ReverseProxy{
+			Rewrite: func(pr *httputil.ProxyRequest) {
+				pr.Out.URL.Scheme, pr.Out.URL.Host = "http", host
+				// The name the client used, for the portal's links and cookies.
+				pr.Out.Host = pr.In.Host
+				peer, _, _ := net.SplitHostPort(pr.In.RemoteAddr)
+				xff := pr.In.Header.Get("X-Forwarded-For")
+				if xff != "" {
+					xff += ", "
+				}
+				pr.Out.Header.Set("X-Forwarded-For", xff+peer)
+				for _, h := range []string{"X-Forwarded-Proto", "X-Forwarded-Host"} {
+					if v := pr.In.Header.Get(h); v != "" {
+						pr.Out.Header.Set(h, v)
+					}
+				}
+				pr.Out.Header.Set(hdrForwardedBy, f.self)
+			},
+			FlushInterval: -1,
+			ErrorHandler: func(w http.ResponseWriter, _ *http.Request, err error) {
+				writeForwardError(w, http.StatusBadGateway, "this control plane is standing by, and the leader did not answer: "+err.Error())
+			},
+		}
+		p.ServeHTTP(w, r)
+	})
+}
+
+// trustControlPlanes keeps the client port trusting the other control
+// planes' X-Forwarded-For, besides router.trusted_proxies: a standby passes
+// requests on, and the client is the address it was given, not the standby.
+// Read again every half minute, for one added since.
+func (f *leaderForward) trustControlPlanes(ctx context.Context, configured []string) {
+	for {
+		list := slices.Clone(configured)
+		if cps, err := f.st.ControlPlanes(ctx); err == nil {
+			for _, cp := range cps {
+				if u, err := url.Parse(cp.URL); err == nil && u.Hostname() != "" {
+					list = append(list, u.Hostname())
+				}
+			}
+		}
+		clientip.SetTrusted(list)
+		select {
+		case <-ctx.Done():
+			return
+		case <-time.After(30 * time.Second):
+		}
+	}
+}
+
 // leader is where the lease's holder is reached, and its name.
 func (f *leaderForward) leader(ctx context.Context) (*url.URL, string, error) {
 	ctx, cancel := context.WithTimeout(ctx, 3*time.Second)
@@ -99,6 +184,10 @@ func (f *leaderForward) leader(ctx context.Context) (*url.URL, string, error) {
 		return nil, "", errors.New("no control plane holds the lease right now")
 	}
 	name := control.HolderName(lease.Owner)
+	if name == f.self {
+		// This machine's previous process, whose lease has not run out.
+		return nil, name, errors.New("the lease is still held by this machine's previous run; it runs out within seconds")
+	}
 	cps, err := f.st.ControlPlanes(ctx)
 	if err != nil {
 		return nil, name, err
@@ -117,7 +206,7 @@ func (f *leaderForward) leader(ctx context.Context) (*url.URL, string, error) {
 func (f *leaderForward) leaderSide(admin http.Handler) http.Handler {
 	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 		path := strings.TrimPrefix(r.URL.Path, forwardPrefix)
-		if !strings.HasPrefix(path, "/api/") {
+		if adminStaysHere(path) {
 			http.NotFound(w, r)
 			return
 		}

internal/cli/leaderforward_test.go

@@ -4,6 +4,7 @@ import (
 	"encoding/json"
 	"io"
 	"log/slog"
+	"net"
 	"net/http"
 	"net/http/httptest"
 	"path/filepath"
@@ -140,9 +141,15 @@ func TestAStandbysAdminAPIIsTheLeaders(t *testing.T) {
 	if got["actor"] != "root@ctl-b" {
 		t.Errorf("who asked was lost: %q", got["actor"])
 	}
-	// What is not the API stays on the machine: the dashboard, pprof.
-	if _, got := c.ask(t, "/debug/pprof/"); got["path"] != "local" {
-		t.Errorf("a non-API path was forwarded: %v", got)
+	// The dashboard's pages are the leader's too.
+	if _, got := c.ask(t, "/nodes/cuda01"); got["path"] != "/nodes/cuda01" {
+		t.Errorf("a dashboard page was answered here: %v", got)
+	}
+	// What is about this process stays on it: pprof, its metrics.
+	for _, p := range []string{"/debug/pprof/", "/metrics"} {
+		if _, got := c.ask(t, p); got["path"] != "local" {
+			t.Errorf("%s was forwarded: %v", p, got)
+		}
 	}
 }
 
@@ -186,3 +193,52 @@ func TestAStandbyWithNoLeaderSaysSo(t *testing.T) {
 		t.Errorf("%d %v", resp.StatusCode, got)
 	}
 }
+
+func TestAStandbysPortalIsTheLeaders(t *testing.T) {
+	c := newForwardCluster(t)
+	// The leader's client port, on the address the store lists for it.
+	var got http.Header
+	var gotHost string
+	leader := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		got, gotHost = r.Header.Clone(), r.Host
+		_, _ = w.Write([]byte("leader " + r.URL.Path))
+	}))
+	defer leader.Close()
+	_, port, _ := net.SplitHostPort(strings.TrimPrefix(leader.URL, "http://"))
+
+	local := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { _, _ = w.Write([]byte("standby " + r.URL.Path)) })
+	srv := httptest.NewServer(c.standby.public(local, port))
+	defer srv.Close()
+	get := func(path string, hdr map[string]string) string {
+		req, _ := http.NewRequestWithContext(t.Context(), http.MethodGet, srv.URL+path, nil)
+		req.Host = "llmgw.example"
+		for k, v := range hdr {
+			req.Header.Set(k, v)
+		}
+		resp, err := http.DefaultClient.Do(req)
+		if err != nil {
+			t.Fatal(err)
+		}
+		defer resp.Body.Close()
+		b, _ := io.ReadAll(resp.Body)
+		return string(b)
+	}
+
+	if body := get("/admin/replicas", map[string]string{"X-Forwarded-For": "203.0.113.7", "X-Forwarded-Proto": "https"}); body != "leader /admin/replicas" {
+		t.Fatalf("the portal was answered by %q", body)
+	}
+	if gotHost != "llmgw.example" || got.Get("X-Forwarded-Proto") != "https" ||
+		!strings.HasPrefix(got.Get("X-Forwarded-For"), "203.0.113.7, ") || got.Get(hdrForwardedBy) != "ctl-b" {
+		t.Errorf("host %q, headers %v", gotHost, got)
+	}
+	// What a load balancer asks is about this machine.
+	for _, p := range []string{"/ready", "/health", "/leader"} {
+		if body := get(p, nil); body != "standby "+p {
+			t.Errorf("%s answered by %q", p, body)
+		}
+	}
+	// Forwarded once only.
+	if body := get("/v1/models", map[string]string{hdrForwardedBy: "ctl-c"}); body != "standby /v1/models" {
+		t.Errorf("a forwarded request was forwarded again: %q", body)
+	}
+}

internal/cli/mirrorinit.go

@@ -89,10 +89,7 @@ func mirrorInit(ctx context.Context, out io.Writer, dir, base string, controlPla
 	if err := in.chown(dir, true); err != nil {
 		return err
 	}
-	addr := ""
-	if ips := localAddrs(); len(ips) > 0 {
-		addr = ips[0].String()
-	}
+	addr := primaryAddr()
 	if base == "" {
 		if addr == "" {
 			return errors.New("this machine has no address to give nodes; give one with --base")

internal/cli/model_serve.go

@@ -73,6 +73,8 @@ func newModelServeCmd(client func() *ctlClient) *cobra.Command {
 	f.StringSliceVar(&req.Aliases, "alias", nil, "other names clients may use; repeatable")
 	f.StringVar(&req.Description, "description", "", "what it is for, shown to clients in /v1/models")
 	f.IntVar(&req.Replicas, "replicas", 1, "how many replicas to run")
+	f.BoolVar(&req.Tools, "tools", false,
+		"say it makes tool calls (for agents and editors); check a running one with llmgw ctl tool-check")
 	f.StringVar(&req.GuardFormat, "guard-format", "",
 		"serve it as the guardrails' judge (role: guard), read as llamaguard3, shieldgemma or json; hidden from clients")
 	f.BoolVar(&onDemand, "on-demand", false, "run none until first asked for")

internal/cli/node.go

@@ -5,6 +5,8 @@ import (
 	"errors"
 	"fmt"
 	"io"
+	"net"
+	"net/netip"
 	"os"
 	"slices"
 	"strings"
@@ -16,7 +18,9 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/pgha"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/release"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/version"
 )
@@ -164,23 +168,36 @@ func (na *nodeAdd) prepare() error {
 	// are not known until the node says what it has, and are not applied
 	// here; doctor's verdict on vLLM is a warning either way.
 	if o.VLLMVersion == "" {
-		if cfg, err := config.Load(na.serverCfg); err == nil {
+		if cfg, err := loadEffectiveConfig(na.serverCfg); err == nil {
 			o.VLLMVersion = cfg.WantedEngines()["vllm"]
 			if o.VLLMVersion == "" {
 				o.VLLMVersion = "none"
 			}
 		}
 	}
-	// The node's ports — its engines', its metrics — for the cluster only.
-	if cfg, err := config.Load(na.serverCfg); err == nil && !o.NoFirewall {
+	// The node's ports — its engines', its metrics — for the cluster only,
+	// unless the cluster's settings turned llmgw's firewall off.
+	if cfg, err := loadEffectiveConfig(na.serverCfg); err == nil && !cfg.Firewall.Enabled {
+		o.NoFirewall = true
+	} else if err == nil && !o.NoFirewall {
 		from, to := cfg.EnginePorts()
 		o.FirewallPorts = []firewall.Port{firewall.Engines(from, to), firewall.NodeExporter}
 		o.FirewallAllow, _ = cfg.FirewallAllow()
 	}
+	// Where the mirror serves model files, for the node to be asked to reach.
+	if cfg, err := loadEffectiveConfig(na.serverCfg); err == nil && cfg.Mirror.API != "" {
+		if mc, err := mirror.NewClient(cfg.Mirror.API, cfg.Mirror.TLS, 5*time.Second); err == nil {
+			ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+			if list, err := mc.List(ctx); err == nil {
+				o.MirrorBase = list.Base
+			}
+			cancel()
+		}
+	}
 	// Where the control plane upgrades nodes from, so a node joins on the
 	// same release from the same place: an offline cluster's mirror.
 	if o.Releases == "" {
-		if cfg, err := config.Load(na.serverCfg); err == nil {
+		if cfg, err := loadEffectiveConfig(na.serverCfg); err == nil {
 			o.Releases = cfg.ReleasesBase()
 		}
 		if o.Releases == "" {
@@ -266,6 +283,9 @@ func (na *nodeAdd) run(ctx context.Context, addrs []string, out io.Writer) error
 		if len(addrs) > 1 {
 			title += fmt.Sprintf(" (%d of %d)", i+1, len(addrs))
 		}
+		if !o.NoFirewall {
+			na.letIn(ctx, o.Addr, out)
+		}
 		var node string
 		var err error
 		if tui {
@@ -297,6 +317,37 @@ func (na *nodeAdd) run(ctx context.Context, addrs []string, out io.Writer) error
 	return nil
 }
 
+// letIn opens this control plane's firewall to the node, when it has one
+// and the node is on a network it does not already let in: otherwise a node
+// on another subnet is set up and then never heard from. Said, not fatal.
+func (na *nodeAdd) letIn(ctx context.Context, host string, out io.Writer) {
+	var addrs []string
+	if a, err := netip.ParseAddr(host); err == nil {
+		addrs = []string{a.String()}
+	} else {
+		ips, err := net.DefaultResolver.LookupNetIP(ctx, "ip4", host)
+		if err != nil {
+			fmt.Fprintf(out, "  !!    firewall: cannot resolve %s (%v); if it is on another network, "+
+				"list it in firewall_networks and run 'llmgw server init'\n", host, err)
+			return
+		}
+		for _, ip := range ips {
+			addrs = append(addrs, ip.Unmap().String())
+		}
+	}
+	added, err := firewall.Admit(ctx, pgha.Local{}, addrs)
+	switch {
+	case err != nil:
+		fmt.Fprintf(out, "  !!    firewall: could not let %s in: %v\n", host, err)
+	case len(added) > 0:
+		shown := make([]string, len(added))
+		for i, p := range added {
+			shown[i] = p.String()
+		}
+		fmt.Fprintf(out, "  ok    firewall: this control plane now lets in %s\n", strings.Join(shown, ", "))
+	}
+}
+
 func (na *nodeAdd) runPlain(ctx context.Context, o onboard.Options, title string, out io.Writer) (string, error) {
 	p := &plainProgress{out: out}
 	fmt.Fprintf(out, "%s\n", title)

internal/cli/root.go

@@ -59,6 +59,8 @@ func Root() *cobra.Command {
 		newSelfUpgradeCmd(),
 		newReleaseCmd(),
 		newJoinCmd(),
+		newFirewallCmd(),
+		newWipeCmd(),
 		newDocsCmd(),
 	)
 	return root

internal/cli/server.go

@@ -16,6 +16,7 @@ import (
 	"slices"
 	"strings"
 	"sync"
+	"sync/atomic"
 	"syscall"
 	"time"
 
@@ -80,10 +81,13 @@ func newServerCmd() *cobra.Command {
 		RunE: func(cmd *cobra.Command, _ []string) error {
 			log := logger()
 
-			cfg, err := config.Load(configPath)
+			// server.yaml for this machine's own keys, the store's settings
+			// for the cluster's: the same on every control plane.
+			boot, err := loadClusterSettings(cmd.Context(), configPath, log)
 			if err != nil {
 				return err
 			}
+			cfg := boot.cfg
 			if listen != "" {
 				cfg.Listen = listen
 			}
@@ -166,7 +170,7 @@ func newServerCmd() *cobra.Command {
 				}
 				gate.SetJudge(rt.Judge, rt.GuardFormat)
 				rt.SetGuard(gate)
-				rt.SetGuardLibrary(cfg.Guard)
+				rt.SetGuardLibrary(gate.Library)
 				if err := rt.RegisterMetrics(gate); err != nil {
 					return fmt.Errorf("guard metrics: %w", err)
 				}
@@ -175,6 +179,10 @@ func newServerCmd() *cobra.Command {
 
 			ctx, stop := signal.NotifyContext(cmd.Context(), syscall.SIGINT, syscall.SIGTERM)
 			defer stop()
+			// Cancelled to start again on changed settings.
+			ctx, cancelRun := context.WithCancel(ctx)
+			defer cancelRun()
+			var restartReason atomic.Pointer[string]
 
 			rt.Start(ctx)
 			defer rt.Stop()
@@ -215,6 +223,10 @@ func newServerCmd() *cobra.Command {
 			// The control plane replaces the static endpoint list: agents
 			// register, the reconciler places replicas, and Ready ones are
 			// published to the router (plan §6.4).
+			// The settings in force, for what reads them at each use.
+			var current atomic.Pointer[config.Server]
+			current.Store(&cfg)
+			var agentSvc *agentsvc.Service
 			var agentHandler http.Handler
 			var agentPath string
 			var admin *control.Controller
@@ -239,6 +251,14 @@ func newServerCmd() *cobra.Command {
 					return err
 				}
 				defer ctl.Close()
+				agentSvc = svc
+				// Whether nodes keep llmgw's firewall, as the settings say now.
+				svc.SetFirewall(func() *llmgwv1.Firewall {
+					c := current.Load()
+					from, to := c.EnginePorts()
+					return &llmgwv1.Firewall{Enabled: c.Firewall.Enabled, Networks: c.FirewallNetworks,
+						EnginePortsFrom: int32(from), EnginePortsTo: int32(to)} //nolint:gosec // ports
+				})
 				ctl.Reconciler().SetNotifier(notifier)
 				admin = ctl
 				agentPath, agentHandler = svc.Handler()
@@ -341,12 +361,17 @@ func newServerCmd() *cobra.Command {
 				evals = newEvalService(cfg, rt, admin, log)
 				api.SetEval(evals)
 				if st := rt.Store(); st != nil && gate != nil {
-					api.SetGuard(&adminapi.GuardAdmin{Library: cfg.Guard, Store: st, Invalidate: gate.Invalidate})
+					api.SetGuard(&adminapi.GuardAdmin{Library: gate.Library, Store: st, Invalidate: gate.Invalidate,
+						JudgeErrors: gate.JudgeErrors, Ask: gate.AskJudge})
 				}
 				if st := rt.Store(); st != nil {
 					// Removing a node revokes its certificate.
 					api.SetRevoker(st.RevokeNode)
 					api.SetAudit(st)
+					// The cluster's settings, each change checked against this
+					// machine's own keys before it is kept.
+					api.SetSettings(&adminapi.Settings{Store: st, LocalKeys: config.LocalKeys(),
+						Check: func(body []byte) error { return checkClusterSettings(configPath, body) }})
 				}
 				// Catalog changes need somewhere to persist, so this is wired
 				// only when there is a store. Without one, the endpoints say so
@@ -378,6 +403,17 @@ func newServerCmd() *cobra.Command {
 					} else {
 						admin.ReplaceCatalog(ctx, built)
 					}
+					// And again on becoming the leader: models added or
+					// removed through the leader before are in the store.
+					if l := admin.Leadership(); l != nil {
+						l.OnLead(func() {
+							if built, err := svc.Load(ctx); err != nil {
+								log.Error("could not read the stored models on becoming the leader", "err", err)
+							} else {
+								admin.ReplaceCatalog(ctx, built)
+							}
+						})
+					}
 					api.SetModels(svc)
 					catalogSvc = svc
 					// And the mirror page, so a model that has just been
@@ -513,6 +549,9 @@ func newServerCmd() *cobra.Command {
 			served := adminHandler
 			if forward != nil {
 				served = forward.standby(adminHandler)
+				_, port, _ := net.SplitHostPort(cfg.Listen)
+				apiSrv.Handler = forward.public(apiSrv.Handler, port)
+				go forward.trustControlPlanes(ctx, cfg.Router.TrustedProxies)
 			}
 			adminSrv := withLimits(&http.Server{
 				Handler:     router.AdminGuard(router.AdminAuth(adminToken, cfg.DebugPprof, served)),
@@ -624,6 +663,59 @@ func newServerCmd() *cobra.Command {
 				}
 			}
 
+			// Every change to the settings in force: what can change live
+			// does, the rest starts this process again, which systemd brings
+			// back on them.
+			// And this machine's own firewall, through its sync unit.
+			if err := writeFirewallState(cfg); err != nil {
+				log.Warn("could not record the firewall's settings for this machine", "err", err)
+			}
+			watch := &settingsWatch{path: configPath, st: rt.Store(), cur: boot, log: log,
+				live: func(next config.Server, changed []string) bool {
+					if !onlyLive(changed) {
+						return false
+					}
+					current.Store(&next)
+					if slices.ContainsFunc(changed, func(c string) bool { return strings.HasPrefix(c, "firewall") }) {
+						if err := writeFirewallState(next); err != nil {
+							log.Warn("could not record the firewall's settings for this machine", "err", err)
+						}
+						if agentSvc != nil {
+							agentSvc.BroadcastFirewall()
+						}
+					}
+					if slices.ContainsFunc(changed, func(c string) bool { return strings.HasPrefix(c, "guard") }) {
+						if gate == nil {
+							// Built at start, and only when there are policies.
+							return len(next.Guard.Policies) == 0
+						}
+						if err := gate.Replace(next.Guard); err != nil {
+							log.Error("the changed guard policies are not valid; the old ones stay in force", "err", err)
+						}
+					}
+					return true
+				},
+				restart: func(reason string) {
+					if os.Getenv("INVOCATION_ID") == "" {
+						log.Warn("the settings changed in a way this process applies only when it starts; "+
+							"it is not run by systemd, so start it again by hand", "why", reason)
+						return
+					}
+					// The leader last, so its standbys are already on the new
+					// settings when it hands over to one of them.
+					delay := time.Duration(0)
+					if admin != nil && admin.Leadership() != nil && admin.Leadership().IsLeader() && rt.Store() != nil {
+						if cps, err := rt.Store().ControlPlanes(ctx); err == nil && len(cps) > 1 {
+							delay = 30 * time.Second
+						}
+					}
+					log.Warn("starting again to apply the settings", "why", reason, "in", delay)
+					restartReason.Store(&reason)
+					time.AfterFunc(delay, cancelRun)
+				},
+			}
+			go watch.run(ctx)
+
 			g, gctx := errgroup.WithContext(ctx)
 			g.Go(func() error { return serve(apiSrv, apiLn) })
 			g.Go(func() error { return serve(adminSrv, adminLn) })
@@ -655,7 +747,12 @@ func newServerCmd() *cobra.Command {
 				_ = adminSrv.Shutdown(sctx)
 				return apiSrv.Shutdown(sctx)
 			})
-			return g.Wait()
+			err = g.Wait()
+			if r := restartReason.Load(); r != nil {
+				// Not a failure, but said as one, so systemd starts it again.
+				return fmt.Errorf("stopped to start again on changed settings (%s)", *r)
+			}
+			return err
 		},
 	}
 
@@ -753,6 +850,14 @@ func startControlPlane(ctx context.Context, cfg config.Server, cat *catalog.Live
 			ctl.SetEngineChoices(control.NewEngineChoices(ctx, st, log))
 			// Domains held for maintenance and labels set on nodes.
 			ctl.Facility().UseStore(ctx, st)
+			// A standby read these when it started; the leader before it may
+			// have changed them since.
+			lead.OnLead(func() {
+				if ch := ctl.EngineChoices(); ch != nil {
+					ch.Reload(ctx)
+				}
+				ctl.Facility().Reload(ctx)
+			})
 			log.Info("competing to be the control plane that places work",
 				"owner", lead.Owner())
 		}

internal/cli/serveradd.go

@@ -155,10 +155,18 @@ func (sa *serverAdd) run(ctx context.Context) error {
 // firewall keeps the new control plane's internal ports to the networks it
 // reaches this one on. Not fatal: it serves without one.
 func (sa *serverAdd) firewall(ctx context.Context) error {
-	cfg, err := config.Load(filepath.Join(sa.etcDir, "server.yaml"))
+	cfg, err := loadEffectiveConfig(filepath.Join(sa.etcDir, "server.yaml"))
 	if err != nil {
 		return err
 	}
+	// What applies the cluster's firewall setting there when it changes.
+	if err := installFirewallSync(ctx, sa.remote); err != nil {
+		fmt.Fprintf(sa.out, "  !!    firewall: %v\n", err)
+	}
+	if !cfg.Firewall.Enabled {
+		sa.kept("firewall: off, as the cluster's settings say (firewall.enabled)")
+		return nil
+	}
 	self, err := sourceIPToward(sa.ip)
 	if err != nil {
 		return err

internal/cli/serverinit.go

@@ -288,7 +288,7 @@ func (in *serverInit) judgeWanted() bool {
 // configuration written before init installed one is never changed, so the
 // model is not fetched for nothing.
 func (in *serverInit) configJudgesWith(model string) bool {
-	cfg, err := config.Load(in.configPath())
+	cfg, err := loadEffectiveConfig(in.configPath())
 	if err != nil {
 		return false
 	}
@@ -821,3 +821,58 @@ Before people outside this network use it:
   #   in server.yaml, and copy them off this machine
 `)
 }
+
+// primaryAddr is the address other machines most likely reach this one on:
+// the source of its default route, else the first address on an interface
+// that is not a container's or a bridge's. The first address of all is not
+// it on a machine with Docker, whose 172.17.0.1 a node takes for its own.
+func primaryAddr() string {
+	if out, err := exec.Command("ip", "-4", "route", "get", "1.1.1.1").Output(); err == nil {
+		if src := routeSource(string(out)); src != "" {
+			return src
+		}
+	}
+	ifaces, err := net.Interfaces()
+	if err == nil {
+		for _, ifc := range ifaces {
+			if ifc.Flags&net.FlagUp == 0 || virtualInterface(ifc.Name) {
+				continue
+			}
+			addrs, _ := ifc.Addrs()
+			for _, a := range addrs {
+				if n, ok := a.(*net.IPNet); ok && n.IP.To4() != nil && n.IP.IsGlobalUnicast() {
+					return n.IP.String()
+				}
+			}
+		}
+	}
+	if ips := localAddrs(); len(ips) > 0 {
+		return ips[0].String()
+	}
+	return ""
+}
+
+// routeSource reads the src of `ip route get`.
+func routeSource(out string) string {
+	f := strings.Fields(out)
+	for i := 0; i+1 < len(f); i++ {
+		if f[i] == "src" {
+			if ip := net.ParseIP(f[i+1]); ip != nil && !ip.IsLoopback() {
+				return ip.String()
+			}
+		}
+	}
+	return ""
+}
+
+// virtualInterface is an interface containers, bridges or overlays made,
+// whose address no other machine reaches this one on.
+func virtualInterface(name string) bool {
+	for _, p := range []string{"docker", "br-", "virbr", "veth", "cni", "flannel", "cali", "vxlan",
+		"kube", "lxc", "lxd", "podman", "cilium", "weave", "tun", "tap"} {
+		if strings.HasPrefix(name, p) {
+			return true
+		}
+	}
+	return false
+}

internal/cli/serverinit_bundle.go

@@ -12,6 +12,7 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/bundle"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
 )
 
@@ -60,7 +61,7 @@ func (in *serverInit) offline() bool {
 	if in.bundle != "" {
 		return true
 	}
-	cfg, err := config.Load(in.configPath())
+	cfg, err := loadEffectiveConfig(in.configPath())
 	return err == nil && cfg.Offline.Enabled()
 }
 
@@ -76,7 +77,7 @@ func (in *serverInit) mirrorRoot() string {
 // server.yaml that the cluster is offline. With a mirror on another
 // machine, the files are imported there; here only the checksums are kept.
 func (in *serverInit) ensureBundle(ctx context.Context) error {
-	cfg, err := config.Load(in.configPath())
+	cfg, err := loadEffectiveConfig(in.configPath())
 	if err != nil {
 		return err
 	}
@@ -129,6 +130,24 @@ func (in *serverInit) ensureBundle(ctx context.Context) error {
 	if sums == config.DefaultOfflineChecksums {
 		sums = ""
 	}
+	// In the cluster's settings once the store holds them; in server.yaml
+	// before, where the first start brings it in.
+	stored, err := changeStoredSettings(in.configPath(), "offline, from server init --bundle", func(doc *settings.Doc) error {
+		if err := doc.SetYAML([]string{"offline", "mirror"}, files); err != nil {
+			return err
+		}
+		if sums != "" {
+			return doc.SetYAML([]string{"offline", "checksums"}, sums)
+		}
+		return nil
+	})
+	if err != nil {
+		return err
+	}
+	if stored {
+		in.done("the cluster's settings: offline, nodes install from the mirror at %s", files)
+		return nil
+	}
 	if err := appendOfflineBlock(in.configPath(), files, sums); err != nil {
 		return err
 	}

internal/cli/serverinit_cluster.go

@@ -86,8 +86,8 @@ func (in *serverInit) reachableAddr() string {
 	if len(in.hosts) > 0 {
 		return in.hosts[0]
 	}
-	if ips := localAddrs(); len(ips) > 0 {
-		return ips[0].String()
+	if a := primaryAddr(); a != "" {
+		return a
 	}
 	name, _ := os.Hostname()
 	return name

internal/cli/serverinit_firewall.go

@@ -19,10 +19,19 @@ import (
 // the engines and etcd when they are here. The client API is not among
 // them; it is meant to be reached.
 func (in *serverInit) ensureFirewall(ctx context.Context) error {
-	cfg, err := config.Load(in.configPath())
+	cfg, err := loadEffectiveConfig(in.configPath())
 	if err != nil {
 		return err
 	}
+	// What applies the cluster's setting here when it changes: the server
+	// writes it, as llmgw; this applies it, as root.
+	if err := in.ensureFirewallSync(ctx); err != nil {
+		return err
+	}
+	if !cfg.Firewall.Enabled {
+		in.kept("firewall: off, as the cluster's settings say (firewall.enabled)")
+		return nil
+	}
 	ports := controlPlanePorts(cfg)
 	if !in.noMirror && in.mirrorURL == "" {
 		ports = append(ports, firewall.MirrorFiles)
@@ -122,3 +131,9 @@ func loopbackListen(listen string) bool {
 	ip := net.ParseIP(host)
 	return host == "localhost" || (ip != nil && ip.IsLoopback())
 }
+
+// ensureFirewallSync installs the units that apply the cluster's firewall
+// setting on this control plane.
+func (in *serverInit) ensureFirewallSync(ctx context.Context) error {
+	return installFirewallSync(ctx, firewall.Local{})
+}

internal/cli/serverinit_test.go

@@ -172,3 +172,22 @@ func TestInitKeepsCertificatesFromAnAuthorityElsewhere(t *testing.T) {
 		t.Error("--host cannot be honoured without the authority, and should say so")
 	}
 }
+
+func TestTheAddressNodesReachIsNotAContainersBridge(t *testing.T) {
+	if got := routeSource("1.1.1.1 via 10.104.130.129 dev ens192 src 10.104.130.140 uid 0\n    cache\n"); got != "10.104.130.140" {
+		t.Errorf("route source %q", got)
+	}
+	if got := routeSource("unreachable"); got != "" {
+		t.Errorf("no route: %q", got)
+	}
+	for _, name := range []string{"docker0", "br-3f2a", "virbr0", "veth12ab", "cni0", "flannel.1"} {
+		if !virtualInterface(name) {
+			t.Errorf("%s taken for a real interface", name)
+		}
+	}
+	for _, name := range []string{"eth0", "ens192", "enp3s0", "bond0", "ib0"} {
+		if virtualInterface(name) {
+			t.Errorf("%s taken for a virtual one", name)
+		}
+	}
+}

internal/cli/status.go

@@ -153,6 +153,16 @@ func printSummary(cmd *cobra.Command, s adminapi.Summary) {
 		fmt.Fprintf(out, "STORE     %s\n", strings.Join(parts, ", "))
 	}
 
+	if s.GuardOff {
+		fmt.Fprintln(out, "GUARD     off: no request is checked, whatever the policies say (llmgw ctl guard on)")
+	}
+	if len(s.Judges) > 0 {
+		fmt.Fprintln(out, "\nGUARDRAILS' JUDGE")
+		for _, j := range s.Judges {
+			fmt.Fprintf(out, "  %s: %s\n", j.Model, j.Problem)
+		}
+	}
+
 	if len(s.Parked) > 0 {
 		fmt.Fprintln(out, "\nPARKED — failed every time they started; not retried until fixed")
 		for _, p := range s.Parked {

internal/cli/tui_actions.go

@@ -679,6 +679,8 @@ func (m *consoleModel) doEntry() tea.Cmd {
 			}}
 	case cBackup:
 		return m.runAfter("server", "backup", "--config="+m.file)
+	case cJudge:
+		return m.runAfter("ctl", server, "guard", "judge", "enable", "--replace")
 	}
 	return nil
 }

internal/cli/tui_console.go

@@ -4,6 +4,7 @@ import (
 	"bufio"
 	"bytes"
 	"context"
+	"encoding/json"
 	"fmt"
 	"io"
 	"net"
@@ -43,6 +44,7 @@ const (
 	cReplicas
 	cUsage
 	cQuotas
+	cGuard
 	cFederation
 	cMirror
 	cJobs
@@ -57,16 +59,18 @@ const (
 	cAddControl
 	cReplicate
 	cServe
+	cJudge
 	cUpgrade
 	cRestartAll
 	cBackup
+	cWipe
 	cLeave
 	cCount
 )
 
-var consoleLabels = [cCount]string{"Overview", "Monitor", "Nodes", "GPUs", "Models", "Replicas", "Usage", "Quotas", "Federation", "Mirror", "Jobs", "Logs", "Accounts", "API keys", "Organizations", "Audit", "Manual",
-	"server.yaml", "Add a node", "Add a control plane", "Replicate the store", "Serve a model", "Upgrade the cluster",
-	"Restart all replicas", "Back up now", "Leave"}
+var consoleLabels = [cCount]string{"Overview", "Monitor", "Nodes", "GPUs", "Models", "Replicas", "Usage", "Quotas", "Guardrails", "Federation", "Mirror", "Jobs", "Logs", "Accounts", "API keys", "Organizations", "Audit", "Manual",
+	"server.yaml", "Add a node", "Add a control plane", "Replicate the store", "Serve a model", "Serve the guardrails' judge", "Upgrade the cluster",
+	"Restart all replicas", "Back up now", "Wipe this machine", "Leave"}
 
 // consoleAction is what is done once the screen closes: each runs its own
 // screen, and the console comes back after.
@@ -137,15 +141,18 @@ type consoleModel struct {
 	node bool
 
 	// The settings.
-	file     string
-	doc      *settings.Doc
-	cfg      any // the file as loaded, for the values that apply
-	levels   []settingsLevel
-	docErr   error
-	saveNote string
+	file string
+	doc  *settings.Doc
+	cfg  any // the file as loaded, for the values that apply
+	// settingsBase is the version of the cluster's settings the screen
+	// read, when the cluster keeps them (doc is then layered over them).
+	settingsBase int64
+	levels       []settingsLevel
+	docErr       error
+	saveNote     string
 
 	// A question open as a dialog.
-	dialog      string // "", "target", "edit", "addname", "saved", "confirm-leave", "error"
+	dialog      string // "", "target", "edit", "addname", "wipe", "saved", "note", "confirm-leave", "error"
 	dialogTitle string
 	dialogText  string
 	prompt      textField
@@ -169,6 +176,7 @@ type consoleModel struct {
 
 	quotas    []adminapi.QuotaView
 	quotasErr error
+	guard     guardState
 
 	st     *storeHolder
 	people peopleData
@@ -217,6 +225,26 @@ func (m *consoleModel) loadSettings() {
 	} else if !m.node {
 		m.cfg = config.Default()
 	}
+	// The cluster's keys from the cluster's settings, when the control
+	// plane answers with them; its own keys from its file.
+	if !m.node && m.client != nil {
+		ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
+		defer cancel()
+		var v adminapi.SettingsView
+		if err := m.client.get(ctx, "/api/v1/settings", &v); err == nil && v.Version > 0 {
+			if cdoc, err := settings.FromBytes("the cluster's settings", []byte(v.Body)); err == nil {
+				m.doc.Layer(cdoc, config.IsLocal)
+				m.settingsBase = v.Version
+				if file, err := config.ReadFile(m.file); err == nil {
+					if cluster, err := config.Parse([]byte(v.Body)); err == nil {
+						if cfg, err := config.FromMap(config.Layer(file, cluster)); err == nil {
+							m.cfg = cfg
+						}
+					}
+				}
+			}
+		}
+	}
 	if root, err := settings.Root(m.file); err == nil && len(m.levels) == 0 {
 		m.levels = []settingsLevel{{title: filepathBase(m.file), block: root, kind: settings.Block}}
 	}
@@ -292,6 +320,9 @@ func (m consoleModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
 		if m.menu == cQuotas {
 			cmds = append(cmds, m.fetchQuotas())
 		}
+		if m.menu == cGuard {
+			cmds = append(cmds, m.fetchGuard())
+		}
 		if m.menu == cMirror {
 			cmds = append(cmds, m.fetchMirror())
 		}
@@ -350,6 +381,9 @@ func (m consoleModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
 		m.people = peopleData(msg)
 	case consoleQuotasMsg:
 		m.quotas, m.quotasErr = msg.rows, msg.err
+	case consoleGuardMsg:
+		window := m.guard.window
+		m.guard = guardState{view: msg.view, events: msg.events, err: msg.err, window: window}
 	case evalSuitesMsg:
 		return m.onEvalSuites(msg)
 	case evalPollMsg:
@@ -375,7 +409,7 @@ func (m consoleModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
 
 func (m consoleModel) entryOK(i int) bool {
 	if m.node {
-		return i == cSettings || i == cManual || i == cLeave
+		return i == cSettings || i == cManual || i == cWipe || i == cLeave
 	}
 	return i != cReplicate || !m.storeReplicated()
 }
@@ -461,7 +495,9 @@ func (m consoleModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) {
 				return m.ask("Replicate the store", "user@address of database machine", 2)
 			case cLeave:
 				return m.leave()
-			case cUpgrade, cRestartAll, cBackup:
+			case cWipe:
+				return m.askWipe()
+			case cUpgrade, cRestartAll, cBackup, cJudge:
 				return m, m.doEntry()
 			case cOverview:
 				return m, nil
@@ -501,6 +537,8 @@ func (m consoleModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) {
 			m.menuD = m.replicaMenu(m.data.replicas[m.row])
 		case cQuotas:
 			m.menuD = m.quotaMenu(m.quotas[m.row])
+		case cGuard:
+			m.menuD = m.guardAttachmentMenu(m.guard.view.Attachments[m.row])
 		case cMirror:
 			m.menuD = m.mirrorMenu(m.mirrorGroups()[m.row])
 		case cJobs:
@@ -556,6 +594,8 @@ func (m consoleModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) {
 		switch m.menu {
 		case cQuotas:
 			m.quotaForm(adminapi.QuotaView{})
+		case cGuard:
+			m.guardAttachForm()
 		case cModels:
 			m.addModelForm()
 		case cMirror:
@@ -567,6 +607,11 @@ func (m consoleModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) {
 		case cOrgs:
 			m.createOrgForm()
 		}
+	case "w":
+		if m.menu == cGuard {
+			m.guard.window = (m.guard.window + 1) % len(guardWindows)
+			return m, m.fetchGuard()
+		}
 	case "i":
 		if m.menu == cMirror {
 			m.mirrorImportOllamaForm()
@@ -604,6 +649,8 @@ func (m consoleModel) onEnter() tea.Cmd {
 		return m.fetchUsage()
 	case cQuotas:
 		return m.fetchQuotas()
+	case cGuard:
+		return m.fetchGuard()
 	case cMirror:
 		return m.fetchMirror()
 	case cMonitor:
@@ -624,10 +671,10 @@ func (m consoleModel) peopleView() bool {
 }
 
 func (m consoleModel) leave() (tea.Model, tea.Cmd) {
-	if m.doc != nil && m.doc.Changed > 0 {
+	if m.doc != nil && m.doc.Pending() > 0 {
 		m.dialog, m.dialogTitle = "confirm-leave", "Unsaved changes"
-		m.dialogText = fmt.Sprintf("%d change(s) to %s are not saved.\nLeave without saving them?\n\n%s",
-			m.doc.Changed, m.file, sDim.Render("Y leave · N stay"))
+		m.dialogText = fmt.Sprintf("%d change(s) are not saved.\nLeave without saving them?\n\n%s",
+			m.doc.Pending(), sDim.Render("Y leave · N stay"))
 		return m, nil
 	}
 	return m, tea.Quit
@@ -644,7 +691,7 @@ func (m consoleModel) ask(title, question string, n int) (tea.Model, tea.Cmd) {
 
 func (m consoleModel) dialogKey(k tea.KeyMsg) (tea.Model, tea.Cmd) {
 	switch m.dialog {
-	case "error":
+	case "error", "note":
 		m.dialog = ""
 		return m, nil
 	case "confirm-leave":
@@ -701,6 +748,13 @@ func (m consoleModel) dialogKey(k tea.KeyMsg) (tea.Model, tea.Cmd) {
 				m.dialogText = floatingQuestion
 				m.prompt.Placeholder = "10.0.0.161/24"
 			}
+		case "wipe":
+			if host, _ := os.Hostname(); v != host {
+				m.dialogText = m.wipeQuestion() + "\n\n" + sBad.Render("that is not this machine's name; nothing is removed")
+				return m, nil
+			}
+			m.action = consoleAction{kind: "wipe"}
+			return m, tea.Quit
 		case "edit":
 			if err := m.doc.Set(m.editPath, m.editKind, m.prompt.Value()); err != nil {
 				m.dialogText = m.editQuestion() + "\n\n" + sBad.Render(err.Error())
@@ -898,28 +952,60 @@ func (m consoleModel) settingsKey(k tea.KeyMsg) (tea.Model, tea.Cmd) {
 }
 
 func (m consoleModel) save() (tea.Model, tea.Cmd) {
-	if m.doc.Changed == 0 {
+	if m.doc.Pending() == 0 {
 		m.dialog, m.dialogTitle, m.dialogText = "error", "Nothing to save", "No change has been made."
 		return m, nil
 	}
-	backup, err := m.doc.Save(func(p string) error { _, err := m.load(p); return err })
-	if err != nil {
+	notSaved := func(err error) (tea.Model, tea.Cmd) {
 		m.dialog, m.dialogTitle = "error", "Not saved"
-		m.dialogText = "The configuration would not load:\n\n" + sBad.Render(err.Error()) + "\n\n" + sDim.Render("Nothing was written. Any key to go back.")
+		m.dialogText = "The configuration would not load:\n\n" + sBad.Render(err.Error()) + "\n\n" +
+			sDim.Render("Nothing was written. Any key to go back.")
 		return m, nil
 	}
+	var said []string
+	// The cluster's settings first: checked by the control plane against
+	// its own file, and in force everywhere once kept.
+	if c := m.doc.Cluster(); c != nil && c.Changed > 0 {
+		body, err := c.Bytes()
+		if err != nil {
+			return notSaved(err)
+		}
+		req, _ := json.Marshal(adminapi.SettingsRequest{Base: m.settingsBase, Body: string(body), Note: "console"})
+		ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+		defer cancel()
+		var v adminapi.SettingsView
+		if err := m.client.put(ctx, "/api/v1/settings", req, &v); err != nil {
+			return notSaved(err)
+		}
+		c.Changed = 0
+		said = append(said, fmt.Sprintf("The cluster's settings are version %d: in force on every control plane "+
+			"within seconds.\nllmgw ctl config history lists the versions; rollback goes back to one.", v.Version))
+	}
+	fileSaved := m.doc.Changed > 0
+	if fileSaved {
+		backup, err := m.doc.Save(func(p string) error { _, err := m.load(p); return err })
+		if err != nil {
+			return notSaved(err)
+		}
+		line := "Saved " + m.file + "."
+		if backup != "" {
+			line += " The old file is " + backup + "."
+		}
+		said = append(said, line)
+	}
 	m.loadSettings()
-	note := ""
-	if backup != "" {
-		note = "The old file is " + backup + ".\n"
+	if !m.node {
+		if fileSaved {
+			said = append(said, "llmgw-server notices the change and applies it within seconds.")
+		}
+		m.dialog, m.dialogTitle = "note", "Saved"
+		m.dialogText = strings.Join(said, "\n\n") + "\n\n" + sDim.Render("Any key to go back.")
+		return m, nil
 	}
 	m.dialog, m.dialogTitle = "saved", "Saved"
-	m.dialogText = "Saved " + m.file + ".\n" + note +
-		fmt.Sprintf("\n%s reads it when it starts. Restart it now?\n", m.service())
-	if !m.node {
-		m.dialogText += sDim.Render("Another control plane keeps its own copy: make the change there too.") + "\n"
-	}
-	m.dialogText += "\n" + sDim.Render("Y restart · N later")
+	m.dialogText = strings.Join(said, "\n") + "\n" +
+		fmt.Sprintf("\n%s reads it when it starts. Restart it now?\n", m.service()) +
+		"\n" + sDim.Render("Y restart · N later")
 	return m, nil
 }
 
@@ -939,6 +1025,8 @@ func (m consoleModel) rowCount() int {
 		return len(m.usage)
 	case cQuotas:
 		return len(m.quotas)
+	case cGuard:
+		return len(m.guard.view.Attachments)
 	case cMirror:
 		return len(m.mirrorGroups())
 	case cJobs:
@@ -978,7 +1066,7 @@ func (m consoleModel) View() string {
 			nav = append(nav, navItem{Label: "", Heading: true}, navItem{Label: "Help", Heading: true})
 		case cSettings:
 			nav = append(nav, navItem{Label: "", Heading: true}, navItem{Label: "Settings", Heading: true})
-		case cAddNode, cLeave:
+		case cAddNode, cWipe:
 			if (i == cAddNode) != m.node {
 				nav = append(nav, navItem{Label: "", Heading: true}, navItem{Label: "Do", Heading: true})
 			}
@@ -990,8 +1078,8 @@ func (m consoleModel) View() string {
 		if n := runningJobs(m.data.jobs); i == cJobs && n > 0 {
 			label += fmt.Sprintf(" (%d)", n)
 		}
-		if i == cSettings && m.doc != nil && m.doc.Changed > 0 {
-			label += fmt.Sprintf(" (%d)", m.doc.Changed)
+		if i == cSettings && m.doc != nil && m.doc.Pending() > 0 {
+			label += fmt.Sprintf(" (%d)", m.doc.Pending())
 		}
 		nav = append(nav, navItem{Label: label, Current: i == m.menu, Dim: !m.entryOK(i)})
 	}
@@ -1019,7 +1107,7 @@ func (m consoleModel) View() string {
 		f.DialogTitle = m.dialogTitle
 		f.Dialog = m.dialogText
 		switch m.dialog {
-		case "target", "edit", "addname":
+		case "target", "edit", "addname", "wipe":
 			q := m.dialogText
 			if m.dialog == "target" && m.editKind > 1 {
 				q = fmt.Sprintf("%s %d of %d", q, len(m.asked)+1, int(m.editKind))
@@ -1166,6 +1254,8 @@ func (m consoleModel) panel() (title, body, keys string) {
 		return "Usage", head + table([]string{"WHO", "REQUESTS", "TOKENS", "ERRORS", "SPENT"}, rows, m.row, pw, m.inPanel), keys
 	case cQuotas:
 		return m.quotasPanel(pw)
+	case cGuard:
+		return m.guardPanel(pw)
 	case cAccounts, cKeys, cOrgs:
 		return m.peoplePanel(pw)
 	case cManual:
@@ -1210,6 +1300,9 @@ func (m consoleModel) panel() (title, body, keys string) {
 		item = "\n  Moves the store onto two machines under Patroni, replicated,\n  so losing one loses nothing.\n\n  " +
 			sDim.Render("On the command line:") +
 			"\n  sudo --preserve-env=SSH_AUTH_SOCK llmgw server init \\\n    --patroni <user>@<a> --patroni <user>@<b> --no-mirror --no-node --no-model"
+	case cJudge:
+		item = "\n  Serves the model the guardrail policies ask as their judge — Llama\n  Guard 3 1B on CPUs, under the name the policies give it — so what they\n  judge is not refused (guard_unavailable). A model of that name that is\n  not a judge is replaced. Overview says when a judge is missing.\n\n  " +
+			sDim.Render("On the command line:") + "\n  llmgw ctl guard judge status\n  llmgw ctl guard judge enable --replace"
 	case cUpgrade:
 		item = "\n  Moves the cluster to a release: this machine first, then each node\n  in turn. Engines keep serving; a node that does not come back stops it.\n\n  " +
 			sDim.Render("On the command line:") + "\n  sudo llmgw ctl upgrade VERSION"
@@ -1219,6 +1312,9 @@ func (m consoleModel) panel() (title, body, keys string) {
 	case cBackup:
 		item = "\n  Writes the store and /etc/llmgw to one archive in " + DefaultBackupDir + ",\n  and keeps the last few. Copy it off this machine.\n\n  " +
 			sDim.Render("On the command line:") + "\n  sudo llmgw server backup"
+	case cWipe:
+		item = "\n  Removes llmgw from this machine: its services, firewall table,\n  configuration, certificates, the authority, secrets, state and users,\n  so it can be set up again from nothing. Model weights, engines, the\n  store's database and backups stay. Asks for the machine's name first.\n\n  " +
+			sDim.Render("On the command line, with --models, --store or --all for more:") + "\n  sudo llmgw wipe"
 	case cLeave:
 		item = "\n  Closes the console. The cluster keeps running."
 	}
@@ -1542,6 +1638,16 @@ func runConsole(ctx context.Context, out io.Writer, file string) error {
 			// Already set up: only the store changes.
 			in.noMirror, in.noNode, in.noModel = true, true, true
 			err = in.runInitTUI(ctx, out)
+		case "wipe":
+			c := Root()
+			c.SetArgs([]string{"wipe", "--yes"})
+			c.SetOut(out)
+			c.SetErr(out)
+			if err := c.ExecuteContext(ctx); err != nil {
+				fmt.Fprintln(out, sBad.Render("✗ ")+err.Error())
+			}
+			// Nothing is left here for the console to show.
+			return nil
 		case "run":
 			c := Root()
 			c.SetArgs(act.targets)
@@ -1587,3 +1693,31 @@ func (m consoleModel) standbyNote() string {
 	}
 	return "\n  " + sDim.Render("Standing by. Shown, and changed, on the leader: "+leader) + "\n"
 }
+
+// askWipe asks for this machine's name before removing llmgw from it, saying
+// what would go.
+func (m consoleModel) askWipe() (tea.Model, tea.Cmd) {
+	if os.Geteuid() != 0 {
+		m.dialog, m.dialogTitle = "error", "Wipe this machine"
+		m.dialogText = "Removing services, users and files under /etc needs root:\nstart the console with sudo llmgw."
+		return m, nil
+	}
+	m.dialog, m.dialogTitle, m.dialogText = "wipe", "Wipe this machine", m.wipeQuestion()
+	m.prompt = newTextField()
+	m.prompt.Width = 40
+	m.prompt.Focus()
+	return m, nil
+}
+
+func (m consoleModel) wipeQuestion() string {
+	host, _ := os.Hostname()
+	w := &wiper{}
+	var b strings.Builder
+	b.WriteString("This removes, on " + host + ":\n\n")
+	for _, s := range w.plan() {
+		b.WriteString("  - " + s.what + "\n")
+	}
+	b.WriteString("\nModel weights, engines, the store's database and backups stay\n" +
+		"(sudo llmgw wipe --all takes them too).\n\nType the machine's name to remove it all:")
+	return b.String()
+}

internal/cli/tui_guard.go

@@ -0,0 +1,189 @@
+package cli
+
+import (
+	"context"
+	"fmt"
+	"net/http"
+	"strings"
+	"time"
+
+	tea "github.com/charmbracelet/bubbletea"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+)
+
+// The guardrails, as the console shows them: the policies server.yaml
+// defines and what each checks, where they are attached, whether the judge
+// models they lean on are serving, and what they did in a window.
+
+// guardWindows are the windows W steps through.
+var guardWindows = []string{"24h", "7d", "1h"}
+
+type guardState struct {
+	view   adminapi.GuardView
+	events adminapi.GuardEventsView
+	err    error
+	// window is an index into guardWindows.
+	window int
+}
+
+type consoleGuardMsg struct {
+	view   adminapi.GuardView
+	events adminapi.GuardEventsView
+	err    error
+}
+
+func (m consoleModel) fetchGuard() tea.Cmd {
+	c, since := m.client, guardWindows[m.guard.window]
+	return func() tea.Msg {
+		ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+		defer cancel()
+		var msg consoleGuardMsg
+		if msg.err = c.get(ctx, "/api/v1/guard", &msg.view); msg.err != nil {
+			return msg
+		}
+		// What they did is a second question; a store without the record
+		// still shows the policies.
+		_ = c.get(ctx, "/api/v1/guard/events?since="+since, &msg.events)
+		return msg
+	}
+}
+
+func (m consoleModel) guardPanel(pw int) (string, string, string) {
+	keys := "→ or Enter open · ↑↓ choose · Q leave"
+	if m.inPanel {
+		keys = "↑↓ attachment · Enter actions · A attach · W window · ← back · Q leave"
+	}
+	g := m.guard
+	var b strings.Builder
+	b.WriteString("\n  " + sDim.Render("What requests and answers are checked for, where, and what that did. A to attach a policy.") + "\n\n")
+	if g.err != nil {
+		if strings.Contains(g.err.Error(), "501") || strings.Contains(g.err.Error(), "no guard policies") {
+			b.WriteString("  " + sDim.Render("No policies: define them under guard: in server.yaml (Settings), then restart."))
+		} else {
+			b.WriteString("  " + sBad.Render(g.err.Error()))
+		}
+		return "Guardrails", b.String(), keys
+	}
+	if len(g.view.Details) == 0 && len(g.view.Policies) == 0 {
+		b.WriteString("  " + sDim.Render("asking the control plane…"))
+		return "Guardrails", b.String(), keys
+	}
+
+	def := "none"
+	if len(g.view.Default) > 0 {
+		def = strings.Join(g.view.Default, ", ")
+	}
+	b.WriteString("  " + sTitle.Render("POLICIES") + "  " + sDim.Render("on every request: "+def) + "\n")
+	for _, p := range g.view.Details {
+		line := fmt.Sprintf("  %-14s %s · fail %s", p.Name, p.Mode, p.Fail)
+		if p.NoStore {
+			line += " · keeps nothing"
+		}
+		if p.Judge != "" {
+			line += " · judge " + p.Judge + " " + m.judgeState(p.Judge)
+		}
+		b.WriteString(line + "\n")
+		for _, c := range p.Checks {
+			b.WriteString("  " + strings.Repeat(" ", 15) + sDim.Render(c) + "\n")
+		}
+	}
+
+	b.WriteString("\n  " + sTitle.Render("ATTACHED") + "  " + sDim.Render("besides the default, to part of the tenancy tree, a person or a key") + "\n")
+	if len(g.view.Attachments) == 0 {
+		b.WriteString("  " + sDim.Render("Nowhere. A to attach one.") + "\n")
+	} else {
+		var rows [][]string
+		for _, a := range g.view.Attachments {
+			who := a.Name
+			if who == "" {
+				who = a.ScopeID
+			}
+			rows = append(rows, []string{a.Scope, who, a.Policy})
+		}
+		b.WriteString(table([]string{"SCOPE", "WHO", "POLICY"}, rows, m.row, pw, m.inPanel) + "\n")
+	}
+
+	window := guardWindows[g.window]
+	b.WriteString("\n  " + sTitle.Render("IN THE LAST "+strings.ToUpper(window)) + "  " + sDim.Render("W for another window") + "\n")
+	if len(g.events.Counts) == 0 {
+		b.WriteString("  " + sDim.Render("Nothing flagged, redacted or blocked.") + "\n")
+	} else {
+		var rows [][]string
+		for i, c := range g.events.Counts {
+			if i == 10 {
+				break
+			}
+			rows = append(rows, []string{c.Action, c.Policy, dash(c.Category), dash(c.Project), fmt.Sprint(c.Count)})
+		}
+		b.WriteString(table([]string{"ACTION", "POLICY", "CATEGORY", "PROJECT", "REQUESTS"}, rows, -1, pw, false) + "\n")
+	}
+	if len(g.events.Events) > 0 {
+		b.WriteString("\n  " + sTitle.Render("NEWEST") + "\n")
+		var rows [][]string
+		for i, e := range g.events.Events {
+			if i == 10 {
+				break
+			}
+			rows = append(rows, []string{e.At.Local().Format("01-02 15:04:05"), e.Action, e.Policy, dash(e.Category),
+				e.Model, fmt.Sprint(e.Status), fmt.Sprintf("%dms", e.GuardMs), e.RequestID})
+		}
+		b.WriteString(table([]string{"AT", "ACTION", "POLICY", "CATEGORY", "MODEL", "STATUS", "GUARD", "REQUEST"}, rows, -1, pw, false))
+	}
+	return "Guardrails", b.String(), keys
+}
+
+// judgeState is how many of a judge model's replicas are ready: a policy
+// that fails closed refuses everything while it has none.
+func (m consoleModel) judgeState(model string) string {
+	for _, md := range m.data.models {
+		if md.Name == model {
+			s := fmt.Sprintf("(%d/%d ready)", md.Ready, md.Desired)
+			if md.Ready == 0 {
+				return sBad.Render(s)
+			}
+			return sOK.Render(s)
+		}
+	}
+	return sBad.Render("(not in the catalog)")
+}
+
+// guardAttachForm attaches a policy.
+func (m *consoleModel) guardAttachForm() {
+	if len(m.guard.view.Policies) == 0 {
+		return
+	}
+	m.openForm(&formDialog{title: "Attach a guardrail policy",
+		note: "Requests from there are checked by it as well as by the default.",
+		fields: []formField{
+			choiceF("to", []string{"project", "team", "org", "user", "key"}, "project"),
+			textF("which", "", "ORG/TEAM/PROJECT, ORG/TEAM, ORG, or the ID"),
+			choiceF("policy", m.guard.view.Policies, m.guard.view.Policies[0]),
+		},
+		submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
+			if strings.TrimSpace(v[1]) == "" {
+				return nil, fmt.Errorf("which %s?", v[0])
+			}
+			req := adminapi.GuardAttachmentRequest{Target: v[0] + ":" + strings.TrimSpace(v[1]), Policy: v[2]}
+			return m.call("Attach "+v[2], http.MethodPost, "/api/v1/guard/attachments", req, nil), nil
+		}})
+}
+
+// guardAttachmentMenu is what can be done to one attachment.
+func (m consoleModel) guardAttachmentMenu(a adminapi.GuardAttachmentView) *menuDialog {
+	who := a.Name
+	if who == "" {
+		who = a.ScopeID
+	}
+	return &menuDialog{title: a.Policy + " on " + a.Scope + " " + who, items: []menuItem{
+		{"Detach…", func(m *consoleModel) tea.Cmd {
+			m.confirmD = &confirmDialog{title: "Detach " + a.Policy,
+				text: "Requests from " + who + " are then checked by the default policies alone.",
+				yes: func(m *consoleModel) tea.Cmd {
+					req := adminapi.GuardAttachmentRequest{Target: a.Scope + ":" + a.ScopeID, Policy: a.Policy}
+					return m.call("Detach "+a.Policy, http.MethodDelete, "/api/v1/guard/attachments", req, nil)
+				}}
+			return nil
+		}},
+	}}
+}

internal/cli/tui_hf.go

@@ -13,7 +13,6 @@ import (
 
 	tea "github.com/charmbracelet/bubbletea"
 
-	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/modelplan"
@@ -57,7 +56,7 @@ var hfClients sync.Map
 
 func (m consoleModel) registry() *hf.Client {
 	endpoint := ""
-	if cfg, err := config.Load(m.file); err == nil {
+	if cfg, err := loadEffectiveConfig(m.file); err == nil {
 		endpoint = cfg.Mirror.Registry
 	}
 	c, _ := hfClients.LoadOrStore(endpoint, hf.New(endpoint, 20*time.Second))

internal/cli/tui_people.go

@@ -12,7 +12,6 @@ import (
 
 	tea "github.com/charmbracelet/bubbletea"
 
-	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
@@ -118,7 +117,7 @@ func (m consoleModel) fetchPeople() tea.Cmd {
 				}
 			}
 		}
-		if cfg, err := config.Load(file); err == nil {
+		if cfg, err := loadEffectiveConfig(file); err == nil {
 			for name := range cfg.Guard.Policies {
 				d.guardPolicies = append(d.guardPolicies, name)
 			}

internal/cli/tui_setup_test.go

@@ -1,6 +1,9 @@
 package cli
 
 import (
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
 	"os"
 	"path/filepath"
 	"slices"
@@ -164,7 +167,7 @@ func TestTheConsoleWatchesTheClusterAndEditsItsSettings(t *testing.T) {
 	cm = mm.(consoleModel)
 	cm.prompt.SetValue("5")
 	mm = press(cm, "enter", "s")
-	if d := mm.(consoleModel).dialog; d != "saved" {
+	if d := mm.(consoleModel).dialog; d != "note" {
 		t.Fatalf("after saving the dialog is %q: %s", d, mm.(consoleModel).dialogText)
 	}
 	b, _ := os.ReadFile(file)
@@ -218,3 +221,106 @@ func TestTheConsoleOnAStandbySaysWhoseClusterItShows(t *testing.T) {
 		t.Error("the leader's console says it is standing by")
 	}
 }
+
+func TestTheConsoleShowsTheGuardrails(t *testing.T) {
+	m := consoleModel{menu: cGuard, loaded: true, width: 140, height: 60}
+	m.data.models = []adminapi.ModelView{{Name: "llama-guard-3-1b", Ready: 2, Desired: 2}}
+	m.guard.view = adminapi.GuardView{
+		Policies: []string{"baseline"}, Default: []string{"baseline"},
+		Details: []adminapi.GuardPolicyView{{Name: "baseline", Mode: "enforce", Fail: "closed", Judge: "llama-guard-3-1b",
+			Checks: []string{"input: pii email → redact"}}},
+		Attachments: []adminapi.GuardAttachmentView{{Name: "acme/ml"}},
+	}
+	m.guard.view.Attachments[0].Scope, m.guard.view.Attachments[0].Policy = "team", "baseline"
+	m.guard.events = adminapi.GuardEventsView{Counts: []adminapi.GuardCountView{{Action: "block", Policy: "baseline", Category: "S1", Count: 7}}}
+	v := m.View()
+	for _, want := range []string{"Guardrails", "baseline", "fail closed", "judge llama-guard-3-1b (2/2 ready)",
+		"input: pii email → redact", "acme/ml", "IN THE LAST 24H", "S1"} {
+		if !strings.Contains(v, want) {
+			t.Errorf("%q missing:\n%s", want, v)
+		}
+	}
+	// W moves the window, and asks again.
+	m.inPanel = true
+	mm, cmd := m.Update(tea.KeyMsg{Type: tea.KeyRunes, Runes: []rune("w")})
+	if c := mm.(consoleModel); guardWindows[c.guard.window] != "7d" || cmd == nil {
+		t.Errorf("window %q", guardWindows[c.guard.window])
+	}
+}
+
+func TestTheConsoleEditsTheClustersSettingsInTheStore(t *testing.T) {
+	dir := t.TempDir()
+	file := filepath.Join(dir, "server.yaml")
+	if err := os.WriteFile(file, []byte("listen: \":8080\"\n"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	var put adminapi.SettingsRequest
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		switch {
+		case r.Method == "GET" && r.URL.Path == "/api/v1/settings":
+			_ = json.NewEncoder(w).Encode(adminapi.SettingsView{Version: 7, Body: "router:\n  retries: 2\n"})
+		case r.Method == "PUT" && r.URL.Path == "/api/v1/settings":
+			_ = json.NewDecoder(r.Body).Decode(&put)
+			_ = json.NewEncoder(w).Encode(adminapi.SettingsView{Version: 8, Body: put.Body})
+		default:
+			http.NotFound(w, r)
+		}
+	}))
+	defer srv.Close()
+	m := newConsoleModel(strings.TrimPrefix(srv.URL, "http://"), file)
+	m.width, m.height, m.loaded = 120, 40, true
+	m.menu, m.inPanel = cSettings, true
+	m.loadSettings()
+	for i, r := range m.settingsRows() {
+		if r.label == "router" {
+			m.row = i
+		}
+	}
+	var mm tea.Model = press(m, "enter")
+	cm := mm.(consoleModel)
+	for i, r := range cm.settingsRows() {
+		if r.label == "retries" {
+			cm.row = i
+		}
+	}
+	if v := cm.View(); !strings.Contains(v, "2") {
+		t.Errorf("the cluster's value is not shown:\n%s", v)
+	}
+	mm = press(cm, "enter")
+	cm = mm.(consoleModel)
+	cm.prompt.SetValue("5")
+	mm = press(cm, "enter", "s")
+	if d := mm.(consoleModel).dialog; d != "note" || !strings.Contains(mm.(consoleModel).dialogText, "version 8") {
+		t.Fatalf("after saving: %q %s", d, mm.(consoleModel).dialogText)
+	}
+	if put.Base != 7 || !strings.Contains(put.Body, "retries: 5") {
+		t.Errorf("sent %+v", put)
+	}
+	if b, _ := os.ReadFile(file); string(b) != "listen: \":8080\"\n" {
+		t.Errorf("the cluster's key went into the machine's file:\n%s", b)
+	}
+}
+
+func TestTheConsoleWipesOnlyOnTheMachinesName(t *testing.T) {
+	m := newConsoleModel("127.0.0.1:1", filepath.Join(t.TempDir(), "server.yaml"))
+	m.width, m.height, m.loaded = 120, 40, true
+	m.menu = cWipe
+	m.dialog, m.dialogTitle, m.dialogText = "wipe", "Wipe this machine", m.wipeQuestion()
+	m.prompt = newTextField()
+	m.prompt.Focus()
+	if v := m.View(); !strings.Contains(v, "Type the machine's name") {
+		t.Errorf("the dialog does not ask for the name:\n%s", v)
+	}
+	m.prompt.SetValue("not-this-one")
+	mm := press(m, "enter")
+	if a := mm.(consoleModel).action; a.kind != "" {
+		t.Fatalf("a wrong name wiped: %+v", a)
+	}
+	cm := mm.(consoleModel)
+	host, _ := os.Hostname()
+	cm.prompt.SetValue(host)
+	mm = press(cm, "enter")
+	if a := mm.(consoleModel).action; a.kind != "wipe" {
+		t.Errorf("the right name did not: %+v", a)
+	}
+}

internal/cli/wipe.go

@@ -0,0 +1,277 @@
+package cli
+
+import (
+	"bufio"
+	"errors"
+	"fmt"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"runtime"
+	"strings"
+
+	"github.com/spf13/cobra"
+)
+
+// wipeUnits are the systemd units llmgw installs, stopped first so nothing
+// writes while the rest goes.
+var wipeUnits = []string{
+	"llmgw-server.service", "llmgw-agent.service", "llmgw-mirror.service",
+	firewallSyncPathUnit, "llmgw-firewall-sync.service", "llmgw-firewall.service",
+	"llmgw-backup.timer", "llmgw-backup.service",
+}
+
+// wiper removes llmgw from a machine. root prefixes every path, for tests;
+// run runs a command.
+type wiper struct {
+	root   string
+	run    func(name string, args ...string) error
+	store  bool
+	models bool
+	all    bool
+}
+
+// wipeStep is one thing to remove, said before it is done.
+type wipeStep struct {
+	what string
+	do   func() error
+}
+
+func (w *wiper) path(p string) string { return filepath.Join(w.root, p) }
+
+func (w *wiper) exists(p string) bool {
+	_, err := os.Lstat(w.path(p))
+	return err == nil
+}
+
+func (w *wiper) glob(pattern string) []string {
+	m, _ := filepath.Glob(w.path(pattern))
+	out := make([]string, len(m))
+	for i, p := range m {
+		out[i] = strings.TrimPrefix(p, w.root)
+	}
+	return out
+}
+
+// mentions is whether a file is there and says llmgw: a file of a name
+// another tool also uses (keepalived's, etcd's) is removed only when llmgw
+// wrote it.
+func (w *wiper) mentions(p string) bool {
+	b, err := os.ReadFile(w.path(p))
+	return err == nil && strings.Contains(string(b), "llmgw")
+}
+
+func (w *wiper) remove(paths ...string) func() error {
+	return func() error {
+		var errs []error
+		for _, p := range paths {
+			if err := os.RemoveAll(w.path(p)); err != nil {
+				errs = append(errs, err)
+			}
+		}
+		return errors.Join(errs...)
+	}
+}
+
+// plan is every step, in order, for what is on this machine.
+func (w *wiper) plan() []wipeStep {
+	var steps []wipeStep
+	add := func(what string, do func() error) { steps = append(steps, wipeStep{what, do}) }
+	files := func(what string, paths ...string) {
+		var there []string
+		for _, p := range paths {
+			if w.exists(p) {
+				there = append(there, p)
+			}
+		}
+		if len(there) > 0 {
+			add(what+": "+strings.Join(there, ", "), w.remove(there...))
+		}
+	}
+
+	// The services, stopped and gone.
+	var units []string
+	for _, u := range wipeUnits {
+		if w.exists("/etc/systemd/system/" + u) {
+			units = append(units, u)
+		}
+	}
+	if len(units) > 0 {
+		add("stop and remove "+strings.Join(units, ", "), func() error {
+			for _, u := range units {
+				_ = w.run("systemctl", "disable", "--now", u)
+				if err := os.Remove(w.path("/etc/systemd/system/" + u)); err != nil && !os.IsNotExist(err) {
+					return err
+				}
+			}
+			return w.run("systemctl", "daemon-reload")
+		})
+	}
+	// Engines run as their own user, kept alive across the agent's restarts:
+	// stopped by user, or they would keep their GPUs.
+	if w.userExists("llmgw-engine") {
+		add("stop the engines (every process of llmgw-engine)", func() error {
+			_ = w.run("pkill", "-KILL", "-u", "llmgw-engine")
+			return nil
+		})
+	}
+	add("take llmgw's table out of the firewall (nft inet llmgw), if loaded", func() error {
+		_ = w.run("nft", "delete", "table", "inet", "llmgw")
+		return nil
+	})
+	if w.mentions("/etc/keepalived/keepalived.conf") {
+		add("stop keepalived and remove llmgw's floating address (/etc/keepalived/keepalived.conf)", func() error {
+			_ = w.run("systemctl", "disable", "--now", "keepalived")
+			return w.remove("/etc/keepalived/keepalived.conf")()
+		})
+	}
+
+	files("configuration, certificates, the authority and secrets", "/etc/llmgw")
+	files("state: the admin token, a SQLite store, what the agent kept",
+		"/var/lib/llmgw", "/var/lib/llmgw-agent")
+	files("the sudo grant setup left", "/etc/sudoers.d/llmgw-setup")
+	if w.exists("/usr/local/share/ca-certificates/llmgw-ca.crt") {
+		add("the cluster's authority from the system's trust (llmgw-ca.crt)", func() error {
+			if err := w.remove("/usr/local/share/ca-certificates/llmgw-ca.crt")(); err != nil {
+				return err
+			}
+			_ = w.run("update-ca-certificates", "--fresh")
+			return nil
+		})
+	}
+	if w.exists("/etc/pki/ca-trust/source/anchors/llmgw-ca.pem") {
+		add("the cluster's authority from the system's trust (llmgw-ca.pem)", func() error {
+			if err := w.remove("/etc/pki/ca-trust/source/anchors/llmgw-ca.pem")(); err != nil {
+				return err
+			}
+			_ = w.run("update-ca-trust")
+			return nil
+		})
+	}
+
+	if w.store || w.all {
+		w.planStore(add, files)
+	}
+	if w.models || w.all {
+		files("model weights and the mirror", "/srv/models", "/var/lib/llmgw-weights", "/var/lib/llmgw-models")
+		files("installed engines", "/opt/llmgw")
+	}
+	if w.all {
+		files("backups", DefaultBackupDir)
+	}
+
+	for _, u := range []string{"llmgw-engine", "llmgw"} {
+		if w.userExists(u) {
+			add("the user "+u, func() error { return w.run("userdel", u) })
+		}
+	}
+	if w.all {
+		files("the llmgw binary", "/usr/local/bin/llmgw", "/usr/local/bin/llmgw.prev")
+	}
+	return steps
+}
+
+// planStore is the store, when it is on this machine: a member of the
+// replicated one (Patroni, etcd), or llmgw's database in a local PostgreSQL.
+func (w *wiper) planStore(add func(string, func() error), files func(string, ...string)) {
+	patroni := w.mentions("/etc/patroni/config.yml")
+	if patroni || w.mentions("/etc/default/etcd") {
+		add("stop Patroni and etcd", func() error {
+			_ = w.run("systemctl", "disable", "--now", "patroni")
+			_ = w.run("systemctl", "disable", "--now", "etcd")
+			return nil
+		})
+	}
+	if patroni {
+		data := append(w.glob("/var/lib/postgresql/*/llmgw"), w.glob("/var/lib/pgsql/*/llmgw")...)
+		files("the store's database on this member, and Patroni's configuration",
+			append([]string{"/etc/patroni", "/var/lib/postgresql/.pgpass_patroni", "/root/llmgw-pg"}, data...)...)
+	} else if w.exists("/usr/bin/psql") {
+		add("drop the llmgw database and role from the local PostgreSQL, if there", func() error {
+			_ = w.run("runuser", "-u", "postgres", "--", "psql", "-q",
+				"-c", "DROP DATABASE IF EXISTS llmgw", "-c", "DROP ROLE IF EXISTS llmgw")
+			return nil
+		})
+	}
+	if w.mentions("/etc/default/etcd") {
+		files("etcd's data and configuration", "/var/lib/etcd/default", "/etc/etcd/tls", "/etc/default/etcd")
+	}
+}
+
+func (w *wiper) userExists(name string) bool {
+	b, err := os.ReadFile(w.path("/etc/passwd"))
+	return err == nil && strings.Contains("\n"+string(b), "\n"+name+":")
+}
+
+func newWipeCmd() *cobra.Command {
+	var w wiper
+	var yes bool
+	c := &cobra.Command{
+		Use:   "wipe",
+		Short: "Remove llmgw from this machine: its services, configuration and state",
+		Long: "Stops and removes llmgw's services, takes its firewall table out, and\n" +
+			"deletes its configuration, certificates, the authority, secrets, state\n" +
+			"and users — what server init, node add, server add and mirror init left —\n" +
+			"so the machine can be set up again from nothing. It says everything it\n" +
+			"will remove first, and asks for the machine's name.\n\n" +
+			"Kept unless asked: model weights, the mirror and engines (--models), the\n" +
+			"store's database when it is on this machine (--store), backups and the\n" +
+			"binary (--all, which is all of these). Packages llmgw installed —\n" +
+			"PostgreSQL, Patroni, etcd, nftables, the node exporter — stay.\n\n" +
+			"A node wiped is then forgotten on the control plane with\n" +
+			"'llmgw node remove <name>'.",
+		Example: "  sudo llmgw wipe\n" +
+			"  sudo llmgw wipe --models\n" +
+			"  sudo llmgw wipe --all --yes",
+		Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			if runtime.GOOS != "linux" {
+				return fmt.Errorf("wipe removes llmgw from a Linux machine; this is %s", runtime.GOOS)
+			}
+			if os.Geteuid() != 0 {
+				return errors.New("wipe removes services, users and files under /etc: run it with sudo")
+			}
+			out := cmd.OutOrStdout()
+			w.run = func(name string, args ...string) error {
+				return exec.CommandContext(cmd.Context(), name, args...).Run() //nolint:gosec // fixed commands
+			}
+			steps := w.plan()
+			host, _ := os.Hostname()
+			fmt.Fprintf(out, "On %s, this removes:\n", host)
+			for _, s := range steps {
+				fmt.Fprintf(out, "  - %s\n", s.what)
+			}
+			if !yes {
+				if !isTerminal(out) {
+					return errors.New("nothing removed: give --yes to wipe without being asked")
+				}
+				fmt.Fprintf(out, "\nType this machine's name (%s) to remove all of it: ", host)
+				answer, _ := bufio.NewReader(cmd.InOrStdin()).ReadString('\n')
+				if strings.TrimSpace(answer) != host {
+					return errors.New("nothing removed")
+				}
+			}
+			var failed []string
+			for _, s := range steps {
+				if err := s.do(); err != nil {
+					failed = append(failed, s.what+": "+err.Error())
+					fmt.Fprintf(out, "  !!    %s: %v\n", s.what, err)
+					continue
+				}
+				fmt.Fprintf(out, "  ok    %s\n", s.what)
+			}
+			if len(failed) > 0 {
+				return fmt.Errorf("%d step(s) failed; running wipe again finishes them", len(failed))
+			}
+			fmt.Fprintln(out, "\nllmgw is gone from this machine. If it was a node, forget it on the control plane: "+
+				"llmgw node remove "+host)
+			return nil
+		},
+	}
+	f := c.Flags()
+	f.BoolVar(&yes, "yes", false, "do not ask")
+	f.BoolVar(&w.models, "models", false, "also remove model weights, the mirror's files and installed engines")
+	f.BoolVar(&w.store, "store", false, "also remove the store's database here: a Patroni member's data, etcd's, or llmgw's database in a local PostgreSQL")
+	f.BoolVar(&w.all, "all", false, "everything: --models, --store, backups and the binary")
+	return c
+}

internal/cli/wipe_test.go

@@ -0,0 +1,106 @@
+package cli
+
+import (
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+)
+
+func wipeFixture(t *testing.T) string {
+	t.Helper()
+	root := t.TempDir()
+	for p, body := range map[string]string{
+		"/etc/llmgw/server.yaml":                      "listen: :8080\n",
+		"/etc/llmgw/pki/ca-key.pem":                   "key",
+		"/var/lib/llmgw/admin-token":                  "t",
+		"/var/lib/llmgw-agent/control-planes.json":    "[]",
+		"/etc/systemd/system/llmgw-server.service":    "[Unit]",
+		"/etc/systemd/system/llmgw-agent.service":     "[Unit]",
+		"/etc/sudoers.d/llmgw-setup":                  "x",
+		"/etc/keepalived/keepalived.conf":             "# llmgw: the gateway's floating address\n",
+		"/etc/passwd":                                 "root:x:0:0::/root:/bin/sh\nllmgw:x:999:999::/var/lib/llmgw:/usr/sbin/nologin\nllmgw-engine:x:998:998::/:/usr/sbin/nologin\n",
+		"/srv/models/qwen/v1/q.gguf":                  "w",
+		"/opt/llmgw/engines/llamacpp/b1/llama-server": "e",
+		"/etc/patroni/config.yml":                     "scope: llmgw\n",
+		"/var/lib/postgresql/16/llmgw/PG_VERSION":     "16",
+		"/etc/default/etcd":                           "ETCD_INITIAL_CLUSTER_TOKEN=llmgw\n",
+		"/var/lib/etcd/default/member/snap":           "s",
+		"/usr/local/bin/llmgw":                        "bin",
+	} {
+		full := filepath.Join(root, p)
+		if err := os.MkdirAll(filepath.Dir(full), 0o755); err != nil {
+			t.Fatal(err)
+		}
+		if err := os.WriteFile(full, []byte(body), 0o600); err != nil {
+			t.Fatal(err)
+		}
+	}
+	return root
+}
+
+func TestWipeRemovesTheConfigurationAndKeepsWhatWasNotAsked(t *testing.T) {
+	root := wipeFixture(t)
+	var ran []string
+	w := &wiper{root: root, run: func(name string, args ...string) error {
+		ran = append(ran, name+" "+strings.Join(args, " "))
+		return nil
+	}}
+	for _, s := range w.plan() {
+		if err := s.do(); err != nil {
+			t.Fatalf("%s: %v", s.what, err)
+		}
+	}
+	for _, gone := range []string{"/etc/llmgw", "/var/lib/llmgw", "/var/lib/llmgw-agent",
+		"/etc/systemd/system/llmgw-server.service", "/etc/sudoers.d/llmgw-setup", "/etc/keepalived/keepalived.conf"} {
+		if _, err := os.Stat(filepath.Join(root, gone)); err == nil {
+			t.Errorf("%s is still there", gone)
+		}
+	}
+	for _, kept := range []string{"/srv/models/qwen/v1/q.gguf", "/opt/llmgw/engines", "/etc/patroni/config.yml",
+		"/var/lib/postgresql/16/llmgw", "/var/lib/etcd/default", "/usr/local/bin/llmgw"} {
+		if _, err := os.Stat(filepath.Join(root, kept)); err != nil {
+			t.Errorf("%s was removed without being asked", kept)
+		}
+	}
+	all := strings.Join(ran, "\n")
+	for _, want := range []string{"systemctl disable --now llmgw-server.service", "pkill -KILL -u llmgw-engine",
+		"nft delete table inet llmgw", "systemctl disable --now keepalived", "userdel llmgw-engine", "userdel llmgw"} {
+		if !strings.Contains(all, want) {
+			t.Errorf("did not run %q:\n%s", want, all)
+		}
+	}
+	if strings.Contains(all, "patroni") {
+		t.Error("the store was touched without --store")
+	}
+}
+
+func TestWipeAllTakesTheStoreAndTheModelsToo(t *testing.T) {
+	root := wipeFixture(t)
+	w := &wiper{root: root, all: true, run: func(string, ...string) error { return nil }}
+	for _, s := range w.plan() {
+		if err := s.do(); err != nil {
+			t.Fatalf("%s: %v", s.what, err)
+		}
+	}
+	for _, gone := range []string{"/srv/models", "/opt/llmgw", "/etc/patroni", "/var/lib/postgresql/16/llmgw",
+		"/var/lib/etcd/default", "/etc/default/etcd", "/usr/local/bin/llmgw"} {
+		if _, err := os.Stat(filepath.Join(root, gone)); err == nil {
+			t.Errorf("%s is still there", gone)
+		}
+	}
+}
+
+func TestWipeLeavesAKeepalivedItDidNotWrite(t *testing.T) {
+	root := t.TempDir()
+	p := filepath.Join(root, "/etc/keepalived/keepalived.conf")
+	_ = os.MkdirAll(filepath.Dir(p), 0o755)
+	_ = os.WriteFile(p, []byte("vrrp_instance site_vip {}\n"), 0o600)
+	w := &wiper{root: root, run: func(string, ...string) error { return nil }}
+	for _, s := range w.plan() {
+		_ = s.do()
+	}
+	if _, err := os.Stat(p); err != nil {
+		t.Error("someone else's keepalived configuration was removed")
+	}
+}

internal/config/config.go

@@ -14,7 +14,6 @@ import (
 
 	"github.com/knadh/koanf/parsers/yaml"
 	env "github.com/knadh/koanf/providers/env/v2"
-	"github.com/knadh/koanf/providers/file"
 	"github.com/knadh/koanf/v2"
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
@@ -147,6 +146,9 @@ type Server struct {
 	// Prometheus and Alertmanager in between. Empty sends nothing.
 	Notify notify.Config `koanf:"notify"`
 
+	// Firewall turns llmgw's firewall on every machine on or off.
+	Firewall FirewallSettings `koanf:"firewall"`
+
 	// FirewallNetworks are networks, besides those each machine reaches the
 	// cluster on, that llmgw's internal ports answer: a monitoring host on
 	// another subnet, nodes behind a router. Written into every machine's
@@ -764,6 +766,7 @@ func Default() Server {
 		AgentListen:    ":7443",
 		MaxConnections: 10_000,
 		Mirror:         Mirror{UpdateCheck: 6 * time.Hour},
+		Firewall:       FirewallSettings{Enabled: true},
 		Control: Control{
 			Interval:         5 * time.Second,
 			BaseBackoff:      10 * time.Second,
@@ -823,15 +826,30 @@ func Default() Server {
 // An empty path loads the defaults, which is what makes `llmgw server` with a
 // couple of --endpoint flags usable for a smoke test.
 func Load(path string) (Server, error) {
-	cfg := Default()
-	k := koanf.New(".")
-
+	var b []byte
 	if path != "" {
 		if _, err := os.Stat(path); err != nil {
-			return cfg, fmt.Errorf("config file: %w", err)
+			return Default(), fmt.Errorf("config file: %w", err)
 		}
-		if err := k.Load(file.Provider(path), yaml.Parser()); err != nil {
-			return cfg, fmt.Errorf("parsing %s: %w", path, err)
+		var err error
+		if b, err = os.ReadFile(path); err != nil { //nolint:gosec // the configuration
+			return Default(), fmt.Errorf("config file: %w", err)
+		}
+	}
+	cfg, err := load(b)
+	if err != nil && path != "" && strings.HasPrefix(err.Error(), "parsing settings") {
+		return cfg, fmt.Errorf("parsing %s:%s", path, strings.TrimPrefix(err.Error(), "parsing settings:"))
+	}
+	return cfg, err
+}
+
+// load reads a YAML document of settings over the defaults.
+func load(b []byte) (Server, error) {
+	cfg := Default()
+	k := koanf.New(".")
+	if len(b) > 0 {
+		if err := k.Load(bytesProvider(b), yaml.Parser()); err != nil {
+			return cfg, fmt.Errorf("parsing settings: %w", err)
 		}
 	}
 

internal/config/firewall.go

@@ -32,3 +32,15 @@ func (s Server) EnginePorts() (int, int) {
 	d := Default().Control.PortRange
 	return d[0], d[1]
 }
+
+// FirewallSettings is whether llmgw keeps a firewall on the cluster's
+// machines.
+type FirewallSettings struct {
+	// Enabled keeps llmgw's internal ports — engines, node exporter, the
+	// agents' stream, the mirror, the store — to the cluster's networks on
+	// every machine. False removes llmgw's nftables table everywhere, for a
+	// site whose own firewall does that job; the rules are kept, and true
+	// puts them back. Changed in the cluster's settings, it is in force on
+	// every node and control plane within seconds.
+	Enabled bool `koanf:"enabled"`
+}

internal/config/layer.go

@@ -0,0 +1,256 @@
+package config
+
+import (
+	"bytes"
+	"errors"
+	"fmt"
+	"maps"
+	"os"
+	"reflect"
+	"slices"
+	"strings"
+
+	"go.yaml.in/yaml/v3"
+)
+
+// localKeys are the keys of server.yaml that belong to one machine: where it
+// listens, its certificates, how it reaches the store, its own secrets and
+// files. Everything else is the cluster's, kept in the store and the same on
+// every control plane (see Layer).
+var localKeys = [][]string{
+	{"listen"}, {"admin_listen"}, {"metrics_listen"}, {"agent_listen"},
+	{"max_connections"}, {"admin_token_file"}, {"admin_remote"}, {"debug_pprof"},
+	{"agent_insecure"}, {"agent_tls"},
+	{"auth", "store"}, {"threads", "store"},
+	{"secrets"}, {"backup_dir"}, {"backup_copy_key"},
+	{"tracing"}, {"monitoring"},
+	{"mirror", "api"}, {"mirror", "tls"},
+	{"endpoints"},
+}
+
+// IsLocal is whether a key path is one machine's own, kept in its
+// server.yaml, rather than the cluster's.
+func IsLocal(path []string) bool {
+	for _, l := range localKeys {
+		if len(path) >= len(l) && slices.Equal(path[:len(l)], l) {
+			return true
+		}
+	}
+	return false
+}
+
+// LocalKeys are the per-machine keys, dotted, for saying which they are.
+func LocalKeys() []string {
+	out := make([]string, len(localKeys))
+	for i, l := range localKeys {
+		out[i] = strings.Join(l, ".")
+	}
+	return out
+}
+
+// ReadFile is a configuration file as YAML maps, empty for no path.
+func ReadFile(path string) (map[string]any, error) {
+	if path == "" {
+		return map[string]any{}, nil
+	}
+	b, err := os.ReadFile(path) //nolint:gosec // the configuration
+	if err != nil {
+		return nil, fmt.Errorf("config file: %w", err)
+	}
+	m, err := Parse(b)
+	if err != nil {
+		return nil, fmt.Errorf("parsing %s: %w", path, err)
+	}
+	return m, nil
+}
+
+// Parse reads a YAML document of settings.
+func Parse(b []byte) (map[string]any, error) {
+	m := map[string]any{}
+	if len(bytes.TrimSpace(b)) == 0 {
+		return m, nil
+	}
+	if err := yaml.Unmarshal(b, &m); err != nil {
+		return nil, err
+	}
+	if m == nil {
+		m = map[string]any{}
+	}
+	return m, nil
+}
+
+// Split separates settings into one machine's and the cluster's.
+func Split(m map[string]any) (local, cluster map[string]any) {
+	local, cluster = map[string]any{}, map[string]any{}
+	split(m, nil, local, cluster)
+	return local, cluster
+}
+
+func split(m map[string]any, path []string, local, cluster map[string]any) {
+	for k, v := range m {
+		p := append(slices.Clone(path), k)
+		switch {
+		case IsLocal(p):
+			local[k] = v
+		case hasLocalBelow(p):
+			sub, ok := v.(map[string]any)
+			if !ok {
+				cluster[k] = v
+				continue
+			}
+			l, c := map[string]any{}, map[string]any{}
+			split(sub, p, l, c)
+			if len(l) > 0 {
+				local[k] = l
+			}
+			if len(c) > 0 {
+				cluster[k] = c
+			}
+		default:
+			cluster[k] = v
+		}
+	}
+}
+
+// hasLocalBelow is whether some local key lies under a path: auth holds
+// store, which is local, beside its policies, which are not.
+func hasLocalBelow(path []string) bool {
+	for _, l := range localKeys {
+		if len(l) > len(path) && slices.Equal(l[:len(path)], path) {
+			return true
+		}
+	}
+	return false
+}
+
+// Layer is what a control plane runs on: its own keys from its server.yaml
+// and every other one from the cluster's settings. The file's cluster keys
+// are not read once the cluster has settings of its own; Ignored names them.
+func Layer(file, cluster map[string]any) map[string]any {
+	local, _ := Split(file)
+	out := clone(local)
+	merge(out, cluster)
+	return out
+}
+
+// Ignored are the cluster keys a file has that the cluster's settings, once
+// there, take the place of.
+func Ignored(file map[string]any) []string {
+	_, c := Split(file)
+	return flatKeys(c, "")
+}
+
+func flatKeys(m map[string]any, prefix string) []string {
+	var out []string
+	for _, k := range slices.Sorted(maps.Keys(m)) {
+		p := prefix + k
+		if sub, ok := m[k].(map[string]any); ok && hasLocalBelow(strings.Split(p, ".")) {
+			out = append(out, flatKeys(sub, p+".")...)
+			continue
+		}
+		out = append(out, p)
+	}
+	return out
+}
+
+// CheckCluster refuses a cluster document that sets one machine's keys: a
+// listener or a certificate path written there would be every control
+// plane's, and none of them could start on it.
+func CheckCluster(m map[string]any) error {
+	local, _ := Split(m)
+	if keys := flatKeys(local, ""); len(keys) > 0 {
+		return fmt.Errorf("%s belong to each machine's server.yaml, not to the cluster's settings",
+			strings.Join(keys, ", "))
+	}
+	return nil
+}
+
+// Changed are the keys whose values differ between two sets of settings,
+// at the top level, and below it where a key mixes local and cluster ones.
+func Changed(a, b map[string]any) []string {
+	var out []string
+	changed(a, b, nil, &out)
+	slices.Sort(out)
+	return out
+}
+
+func changed(a, b map[string]any, path []string, out *[]string) {
+	keys := map[string]bool{}
+	for k := range a {
+		keys[k] = true
+	}
+	for k := range b {
+		keys[k] = true
+	}
+	for k := range keys {
+		p := append(slices.Clone(path), k)
+		va, vb := a[k], b[k]
+		if reflect.DeepEqual(va, vb) {
+			continue
+		}
+		ma, okA := va.(map[string]any)
+		mb, okB := vb.(map[string]any)
+		if hasLocalBelow(p) && (okA || va == nil) && (okB || vb == nil) {
+			changed(ma, mb, p, out)
+			continue
+		}
+		*out = append(*out, strings.Join(p, "."))
+	}
+}
+
+// Marshal writes settings as YAML.
+func Marshal(m map[string]any) ([]byte, error) {
+	if len(m) == 0 {
+		return nil, nil
+	}
+	var b bytes.Buffer
+	enc := yaml.NewEncoder(&b)
+	enc.SetIndent(2)
+	if err := enc.Encode(m); err != nil {
+		return nil, err
+	}
+	return b.Bytes(), enc.Close()
+}
+
+// FromMap is the configuration settings make: laid over the defaults, with
+// LLMGW_ environment overrides, and validated.
+func FromMap(m map[string]any) (Server, error) {
+	b, err := Marshal(m)
+	if err != nil {
+		return Default(), err
+	}
+	return load(b)
+}
+
+// bytesProvider hands koanf a document already read.
+type bytesProvider []byte
+
+func (b bytesProvider) ReadBytes() ([]byte, error) { return b, nil }
+func (b bytesProvider) Read() (map[string]any, error) {
+	return nil, errors.New("bytesProvider reads bytes only")
+}
+
+func clone(m map[string]any) map[string]any {
+	out := make(map[string]any, len(m))
+	for k, v := range m {
+		if sub, ok := v.(map[string]any); ok {
+			v = clone(sub)
+		}
+		out[k] = v
+	}
+	return out
+}
+
+// merge lays src over dst: maps key by key, everything else replaced.
+func merge(dst, src map[string]any) {
+	for k, v := range src {
+		if sv, ok := v.(map[string]any); ok {
+			if dv, ok := dst[k].(map[string]any); ok {
+				merge(dv, sv)
+				continue
+			}
+			v = clone(sv)
+		}
+		dst[k] = v
+	}
+}

internal/config/layer_test.go

@@ -0,0 +1,99 @@
+package config
+
+import (
+	"slices"
+	"strings"
+	"testing"
+)
+
+const machineFile = `
+listen: ":8080"
+agent_listen: ":7443"
+agent_insecure: true
+auth:
+  enabled: true
+  store: /var/lib/llmgw/llmgw.db
+  default_rpm: 60
+mirror:
+  api: http://127.0.0.1:8082
+  update_check: 6h
+guard:
+  default: [baseline]
+  policies:
+    baseline:
+      input:
+        injection: flag
+router:
+  retries: 2
+`
+
+func TestAFileSplitsIntoTheMachinesAndTheClusters(t *testing.T) {
+	file, err := Parse([]byte(machineFile))
+	if err != nil {
+		t.Fatal(err)
+	}
+	local, cluster := Split(file)
+	for _, k := range []string{"listen", "agent_listen", "agent_insecure"} {
+		if _, ok := local[k]; !ok {
+			t.Errorf("%s is not the machine's", k)
+		}
+	}
+	if local["auth"].(map[string]any)["store"] == nil || cluster["auth"].(map[string]any)["store"] != nil {
+		t.Errorf("auth.store went to the cluster: local %v cluster %v", local["auth"], cluster["auth"])
+	}
+	if cluster["auth"].(map[string]any)["default_rpm"] != 60 {
+		t.Errorf("auth's policy stayed with the machine: %v", cluster["auth"])
+	}
+	if local["mirror"].(map[string]any)["api"] == nil || cluster["mirror"].(map[string]any)["update_check"] == nil {
+		t.Errorf("mirror split wrong: local %v cluster %v", local["mirror"], cluster["mirror"])
+	}
+	for _, k := range []string{"guard", "router"} {
+		if _, ok := cluster[k]; !ok {
+			t.Errorf("%s is not the cluster's", k)
+		}
+	}
+	if err := CheckCluster(cluster); err != nil {
+		t.Errorf("the cluster half is refused: %v", err)
+	}
+	if err := CheckCluster(file); err == nil || !strings.Contains(err.Error(), "auth.store") {
+		t.Errorf("a cluster document with a machine's keys: %v", err)
+	}
+}
+
+func TestTheClustersSettingsTakeThePlaceOfTheFiles(t *testing.T) {
+	file, _ := Parse([]byte(machineFile))
+	cluster, _ := Parse([]byte("router:\n  retries: 5\nauth:\n  enabled: true\n  default_rpm: 10\n"))
+	eff := Layer(file, cluster)
+	cfg, err := FromMap(eff)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if cfg.Router.Retries != 5 || cfg.Auth.DefaultRPM != 10 {
+		t.Errorf("cluster values not in force: retries %d rpm %d", cfg.Router.Retries, cfg.Auth.DefaultRPM)
+	}
+	if cfg.Auth.Store != "/var/lib/llmgw/llmgw.db" || cfg.Mirror.API != "http://127.0.0.1:8082" || cfg.Listen != ":8080" {
+		t.Errorf("the machine's own keys lost: %+v", cfg.Auth)
+	}
+	if len(cfg.Guard.Policies) != 0 {
+		t.Error("the file's guard applies though the cluster has settings without one")
+	}
+	if got := Ignored(file); !slices.Contains(got, "guard") || !slices.Contains(got, "auth.default_rpm") ||
+		slices.Contains(got, "auth.store") {
+		t.Errorf("ignored: %v", got)
+	}
+}
+
+func TestChangedNamesWhatMoved(t *testing.T) {
+	a, _ := Parse([]byte(machineFile))
+	b, _ := Parse([]byte(strings.Replace(machineFile, "retries: 2", "retries: 3", 1)))
+	if got := Changed(a, b); !slices.Equal(got, []string{"router"}) {
+		t.Errorf("changed %v", got)
+	}
+	c, _ := Parse([]byte(strings.Replace(machineFile, "store: /var/lib/llmgw/llmgw.db", "store: /x.db", 1)))
+	if got := Changed(a, c); !slices.Equal(got, []string{"auth.store"}) {
+		t.Errorf("changed %v", got)
+	}
+	if got := Changed(a, a); len(got) != 0 {
+		t.Errorf("unchanged: %v", got)
+	}
+}

internal/control/adminapi/api.go

@@ -67,7 +67,8 @@ type API struct {
 	// nil without a store.
 	federation *federation.Service
 	// guard is the guardrail policies' library and attachments.
-	guard *GuardAdmin
+	guard    *GuardAdmin
+	settings *Settings
 	// revoke records a removed node, so its certificate is refused.
 	revoke func(ctx context.Context, nodeID string, at time.Time) error
 
@@ -111,6 +112,7 @@ func (a *API) Register(mux Mux) {
 	mux.HandleFunc("POST /api/v1/models/{name}/resources", a.setResources)
 	a.registerQuotas(mux)
 	a.registerGuard(mux)
+	a.registerSettings(mux)
 	mux.HandleFunc("GET /api/v1/audit", a.auditEvents)
 	mux.HandleFunc("GET /api/v1/usage/export", a.usageExport)
 	mux.HandleFunc("POST /api/v1/notify/test", a.notifyTest)
@@ -1543,6 +1545,7 @@ type RequestView struct {
 	GuardAction   string `json:"guard_action,omitempty"`
 	GuardPolicy   string `json:"guard_policy,omitempty"`
 	GuardCategory string `json:"guard_category,omitempty"`
+	GuardRule     string `json:"guard_rule,omitempty"`
 	GuardMs       int64  `json:"guard_ms,omitempty"`
 	// What the client said about it: OpenAI's user field, and its tags.
 	EndUser string            `json:"end_user,omitempty"`
@@ -1571,7 +1574,7 @@ func (a *API) request(w http.ResponseWriter, r *http.Request) {
 		Model: u.Model, ServedModel: u.ServedModel, Replica: u.Replica, Status: u.Status,
 		LatencyMs: u.LatencyMs, QueueMs: u.QueueMs, TTFTMs: u.TTFTMs,
 		PromptTokens: u.PromptTokens, CompletionTokens: u.CompletionTokens, CostMicros: u.CostMicros,
-		GuardAction: u.GuardAction, GuardPolicy: u.GuardPolicy, GuardCategory: u.GuardCategory, GuardMs: u.GuardMs,
+		GuardAction: u.GuardAction, GuardPolicy: u.GuardPolicy, GuardCategory: u.GuardCategory, GuardRule: u.GuardRule, GuardMs: u.GuardMs,
 		EndUser: u.EndUser, Tags: u.Tags, CacheHit: u.CacheHit,
 	})
 }

internal/control/adminapi/guard.go

@@ -4,11 +4,14 @@ import (
 	"context"
 	"encoding/json"
 	"fmt"
+	"maps"
 	"net/http"
 	"slices"
 	"strings"
+	"time"
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
 )
 
 // GuardStore is where policy attachments are kept.
@@ -18,12 +21,17 @@ type GuardStore interface {
 	DetachGuard(ctx context.Context, a guard.Attachment) (bool, error)
 }
 
-// GuardAdmin is what the guard endpoints need: the library from server.yaml,
-// the store, and a way to have the gate read attachments again at once.
+// GuardAdmin is what the guard endpoints need: the library in force (the
+// cluster's settings), the store, and a way to have the gate read
+// attachments again at once.
 type GuardAdmin struct {
-	Library    guard.Config
+	Library    func() guard.Config
 	Store      GuardStore
 	Invalidate func()
+	// JudgeErrors are each judge's last failure to give a verdict.
+	JudgeErrors func() map[string]guard.JudgeError
+	// Ask has a judge read a text, past the cache.
+	Ask func(ctx context.Context, model, text, answer string) (guard.JudgeAnswer, error)
 }
 
 // SetGuard enables the guard endpoints.
@@ -31,9 +39,122 @@ func (a *API) SetGuard(g *GuardAdmin) { a.guard = g }
 
 // GuardView is the library and where its policies are attached.
 type GuardView struct {
+	// Enabled is whether the guardrails check anything (guard.enabled).
+	Enabled     bool                  `json:"enabled"`
 	Policies    []string              `json:"policies"`
 	Default     []string              `json:"default"`
 	Attachments []GuardAttachmentView `json:"attachments"`
+	// Details says what each policy does, in the order of Policies.
+	Details []GuardPolicyView `json:"details"`
+}
+
+// GuardPolicyView is what one policy does, in brief.
+type GuardPolicyView struct {
+	Name string `json:"name"`
+	// Mode is enforce, shadow or off; Fail is open or closed.
+	Mode    string `json:"mode"`
+	Fail    string `json:"fail"`
+	NoStore bool   `json:"no_store,omitempty"`
+	// Judge is the judge model, when one reads the text.
+	Judge string `json:"judge,omitempty"`
+	// Checks are what it looks for, one line each: "input: pii email → redact".
+	Checks []string `json:"checks"`
+}
+
+// GuardHistory is what the guardrails did, from the usage record; the store
+// has it, and a store that does not leaves the events endpoint unanswered.
+type GuardHistory interface {
+	GuardEvents(ctx context.Context, since time.Time, limit int) ([]store.GuardEvent, error)
+	GuardCounts(ctx context.Context, since time.Time) ([]store.GuardCount, error)
+}
+
+// GuardEventsView is what the guardrails did since a moment: totals, and the
+// newest requests.
+type GuardEventsView struct {
+	Since  time.Time        `json:"since"`
+	Counts []GuardCountView `json:"counts"`
+	Events []GuardEventView `json:"events"`
+}
+
+// GuardCountView is how many requests one policy did one thing to.
+type GuardCountView struct {
+	Project  string `json:"project,omitempty"`
+	Action   string `json:"action"`
+	Policy   string `json:"policy"`
+	Category string `json:"category,omitempty"`
+	Count    int64  `json:"count"`
+}
+
+// GuardEventView is one request the guardrails did something to.
+type GuardEventView struct {
+	At        time.Time `json:"at"`
+	RequestID string    `json:"request_id"`
+	Project   string    `json:"project,omitempty"`
+	Model     string    `json:"model"`
+	Status    int       `json:"status"`
+	Action    string    `json:"action"`
+	Policy    string    `json:"policy"`
+	Category  string    `json:"category,omitempty"`
+	// Rule is what decided it within the category: a pattern's name, or a
+	// judge's verdict, "S6,Specialized Advice".
+	Rule    string `json:"rule,omitempty"`
+	GuardMs int64  `json:"guard_ms"`
+}
+
+// policyView sums one policy up.
+func policyView(name string, p guard.Policy) GuardPolicyView {
+	v := GuardPolicyView{Name: name, Mode: or(p.Mode, "enforce"), Fail: or(p.Fail, "open"), NoStore: p.NoStore, Checks: []string{}}
+	rules := func(dir string, r guard.Rules) {
+		for _, kind := range slices.Sorted(maps.Keys(r.PII)) {
+			v.Checks = append(v.Checks, fmt.Sprintf("%s: pii %s → %s", dir, kind, r.PII[kind]))
+		}
+		if r.Injection != "" {
+			v.Checks = append(v.Checks, fmt.Sprintf("%s: prompt injection → %s", dir, r.Injection))
+		}
+		for _, d := range r.Deny {
+			v.Checks = append(v.Checks, fmt.Sprintf("%s: pattern %s → %s", dir, d.Name, or(d.Action, "block")))
+		}
+		if r.MaxChars > 0 {
+			v.Checks = append(v.Checks, fmt.Sprintf("%s: at most %d characters", dir, r.MaxChars))
+		}
+	}
+	rules("input", p.Input)
+	rules("output", p.Output)
+	if j := p.Judge; j != nil {
+		v.Judge = j.Model
+		var dirs []string
+		if j.Input != "" {
+			dirs = append(dirs, "input → "+j.Input)
+		}
+		if j.Output != "" {
+			dirs = append(dirs, "output → "+j.Output)
+		}
+		line := "judge " + j.Model + ": " + strings.Join(dirs, ", ")
+		if len(j.Categories) > 0 {
+			line += " (" + strings.Join(j.Categories, " ") + ")"
+		}
+		v.Checks = append(v.Checks, line)
+	}
+	if len(p.Tools.Allow) > 0 {
+		v.Checks = append(v.Checks, "tools: only "+strings.Join(p.Tools.Allow, ", "))
+	}
+	if len(p.Tools.Deny) > 0 {
+		v.Checks = append(v.Checks, "tools: never "+strings.Join(p.Tools.Deny, ", "))
+	}
+	if p.MaxOutputTokens > 0 {
+		v.Checks = append(v.Checks, fmt.Sprintf("answers of at most %d tokens", p.MaxOutputTokens))
+	}
+	if p.SystemPrompt != "" {
+		v.Checks = append(v.Checks, "a system prompt of its own")
+	}
+	return v
+}
+
+func or(s, dflt string) string {
+	if s == "" {
+		return dflt
+	}
+	return s
 }
 
 // GuardAttachmentView is one attachment, with the name people know its
@@ -43,6 +164,15 @@ type GuardAttachmentView struct {
 	Name string `json:"name,omitempty"`
 }
 
+// guardHistory is where what the guardrails did is read, when there is one.
+func (a *API) guardHistory() (GuardHistory, bool) {
+	if a.guard == nil {
+		return nil, false
+	}
+	h, ok := a.guard.Store.(GuardHistory)
+	return h, ok
+}
+
 // guardTarget reads where a policy is to be attached.
 func (a *API) guardTarget(ctx context.Context, target string) (guard.Attachment, error) {
 	kind, what, ok := strings.Cut(strings.TrimSpace(target), ":")
@@ -77,9 +207,11 @@ type GuardAttachmentRequest struct {
 }
 
 func (a *API) registerGuard(mux Mux) {
+	mux.HandleFunc("GET /api/v1/guard/judges", a.listJudges)
+	mux.HandleFunc("POST /api/v1/guard/judges/ask", a.askJudge)
 	mux.HandleFunc("GET /api/v1/guard", func(w http.ResponseWriter, r *http.Request) {
 		if a.guard == nil {
-			writeError(w, http.StatusNotImplemented, "no guard policies: define them under guard: in server.yaml")
+			writeError(w, http.StatusNotImplemented, "no guard policies: define them under guard in the cluster's settings (llmgw ctl config edit)")
 			return
 		}
 		att, err := a.guard.Store.GuardAttachments(r.Context())
@@ -87,7 +219,7 @@ func (a *API) registerGuard(mux Mux) {
 			writeError(w, http.StatusInternalServerError, err.Error())
 			return
 		}
-		v := GuardView{Default: a.guard.Library.Default, Attachments: []GuardAttachmentView{}}
+		v := GuardView{Enabled: a.guard.Library().On(), Default: a.guard.Library().Default, Attachments: []GuardAttachmentView{}}
 		var names map[string]string
 		if a.quotas != nil {
 			names, _ = a.scopeNames(r.Context())
@@ -95,16 +227,67 @@ func (a *API) registerGuard(mux Mux) {
 		for _, at := range att {
 			v.Attachments = append(v.Attachments, GuardAttachmentView{Attachment: at, Name: names[at.Scope+":"+at.ScopeID]})
 		}
-		for name := range a.guard.Library.Policies {
+		for name := range a.guard.Library().Policies {
 			v.Policies = append(v.Policies, name)
 		}
 		slices.Sort(v.Policies)
+		for _, name := range v.Policies {
+			v.Details = append(v.Details, policyView(name, a.guard.Library().Policies[name]))
+		}
+		writeJSON(w, http.StatusOK, v)
+	})
+	mux.HandleFunc("GET /api/v1/guard/events", func(w http.ResponseWriter, r *http.Request) {
+		h, ok := a.guardHistory()
+		if !ok {
+			writeError(w, http.StatusNotImplemented, "no guard policies, or no store to record what they did")
+			return
+		}
+		since := 24 * time.Hour
+		if q := r.URL.Query().Get("since"); q != "" {
+			d, err := parseSince(q)
+			if err != nil {
+				writeError(w, http.StatusBadRequest, err.Error())
+				return
+			}
+			since = d
+		}
+		from := time.Now().Add(-since)
+		counts, err := h.GuardCounts(r.Context(), from)
+		if err != nil {
+			writeError(w, http.StatusInternalServerError, err.Error())
+			return
+		}
+		events, err := h.GuardEvents(r.Context(), from, 100)
+		if err != nil {
+			writeError(w, http.StatusInternalServerError, err.Error())
+			return
+		}
+		var names map[string]string
+		if a.quotas != nil {
+			names, _ = a.scopeNames(r.Context())
+		}
+		project := func(id string) string {
+			if n := names["project:"+id]; n != "" {
+				return n
+			}
+			return id
+		}
+		v := GuardEventsView{Since: from, Counts: []GuardCountView{}, Events: []GuardEventView{}}
+		for _, c := range counts {
+			v.Counts = append(v.Counts, GuardCountView{Project: project(c.ProjectID), Action: c.Action,
+				Policy: c.Policy, Category: c.Category, Count: c.Count})
+		}
+		slices.SortStableFunc(v.Counts, func(a, b GuardCountView) int { return int(b.Count - a.Count) })
+		for _, e := range events {
+			v.Events = append(v.Events, GuardEventView{At: e.At, RequestID: e.RequestID, Project: project(e.ProjectID),
+				Model: e.Model, Status: e.Status, Action: e.Action, Policy: e.Policy, Category: e.Category, Rule: e.Rule, GuardMs: e.GuardMs})
+		}
 		writeJSON(w, http.StatusOK, v)
 	})
 	change := func(attach bool) http.HandlerFunc {
 		return func(w http.ResponseWriter, r *http.Request) {
 			if a.guard == nil {
-				writeError(w, http.StatusNotImplemented, "no guard policies: define them under guard: in server.yaml")
+				writeError(w, http.StatusNotImplemented, "no guard policies: define them under guard in the cluster's settings (llmgw ctl config edit)")
 				return
 			}
 			var at GuardAttachmentRequest
@@ -118,8 +301,8 @@ func (a *API) registerGuard(mux Mux) {
 				return
 			}
 			att.Policy = at.Policy
-			if _, ok := a.guard.Library.Policies[att.Policy]; !ok && attach {
-				writeError(w, http.StatusBadRequest, fmt.Sprintf("no policy %q in server.yaml's guard.policies", att.Policy))
+			if _, ok := a.guard.Library().Policies[att.Policy]; !ok && attach {
+				writeError(w, http.StatusBadRequest, fmt.Sprintf("no policy %q in the cluster's guard.policies", att.Policy))
 				return
 			}
 			changed := true

internal/control/adminapi/guard_test.go

@@ -36,8 +36,10 @@ func TestGuardPoliciesAreAttachedByTarget(t *testing.T) {
 	st := &memGuard{}
 	invalidated := 0
 	a.SetGuard(&GuardAdmin{
-		Library: guard.Config{Default: []string{"base"}, Policies: map[string]guard.Policy{"base": {}, "finance": {}}},
-		Store:   st, Invalidate: func() { invalidated++ },
+		Library: func() guard.Config {
+			return guard.Config{Default: []string{"base"}, Policies: map[string]guard.Policy{"base": {}, "finance": {}}}
+		},
+		Store: st, Invalidate: func() { invalidated++ },
 	})
 	if code := call(t, a, "POST", "/api/v1/guard/attachments", `{"target":"key:k1","policy":"finance"}`, nil); code != 200 {
 		t.Fatalf("attach: %d", code)
@@ -116,3 +118,55 @@ func TestUsageIsExportedAsCSVThatASpreadsheetCannotRun(t *testing.T) {
 		t.Errorf("%d %s %+v", rec.Code, body, m)
 	}
 }
+
+// memGuardHistory is a guard store that also remembers what was done.
+type memGuardHistory struct {
+	memGuard
+	since time.Time
+}
+
+func (m *memGuardHistory) GuardEvents(_ context.Context, since time.Time, _ int) ([]store.GuardEvent, error) {
+	m.since = since
+	return []store.GuardEvent{{RequestID: "r1", Model: "m", Status: 400, Action: "block", Policy: "base", Category: "S1", GuardMs: 410}}, nil
+}
+
+func (m *memGuardHistory) GuardCounts(context.Context, time.Time) ([]store.GuardCount, error) {
+	return []store.GuardCount{
+		{Action: "flag", Policy: "base", Category: "email", Count: 2},
+		{Action: "block", Policy: "base", Category: "S1", Count: 7},
+	}, nil
+}
+
+func TestTheGuardrailsSayWhatEachPolicyDoesAndWhatTheyDid(t *testing.T) {
+	a, _ := setup(t, testModels())
+	h := &memGuardHistory{}
+	lib := guard.Config{Policies: map[string]guard.Policy{
+		"base": {Fail: "closed", Input: guard.Rules{PII: map[string]string{"email": "redact"}, Injection: "block"},
+			Judge: &guard.JudgePolicy{Model: "llama-guard-3-1b", Input: "block", Categories: []string{"S1", "S9"}}},
+	}}
+	a.SetGuard(&GuardAdmin{Store: h, Library: func() guard.Config { return lib }})
+
+	var v GuardView
+	if code := call(t, a, "GET", "/api/v1/guard", "", &v); code != 200 || len(v.Details) != 1 {
+		t.Fatalf("%d %+v", code, v)
+	}
+	d := v.Details[0]
+	want := []string{"input: pii email → redact", "input: prompt injection → block", "judge llama-guard-3-1b: input → block (S1 S9)"}
+	if d.Mode != "enforce" || d.Fail != "closed" || d.Judge != "llama-guard-3-1b" || !slices.Equal(d.Checks, want) {
+		t.Errorf("%+v", d)
+	}
+
+	var ev GuardEventsView
+	if code := call(t, a, "GET", "/api/v1/guard/events?since=1h", "", &ev); code != 200 {
+		t.Fatalf("events: %d", code)
+	}
+	if time.Since(h.since) > time.Hour+time.Minute || time.Since(h.since) < time.Hour-time.Minute {
+		t.Errorf("asked since %s", h.since)
+	}
+	if len(ev.Counts) != 2 || ev.Counts[0].Count != 7 || len(ev.Events) != 1 || ev.Events[0].GuardMs != 410 {
+		t.Errorf("most first, and the events: %+v", ev)
+	}
+	if code := call(t, a, "GET", "/api/v1/guard/events?since=soon", "", nil); code != 400 {
+		t.Errorf("a bad window: %d", code)
+	}
+}

internal/control/adminapi/judges.go

@@ -0,0 +1,181 @@
+package adminapi
+
+import (
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"slices"
+	"sort"
+	"strings"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
+)
+
+// JudgeView is one judge model the guardrail policies ask, and whether it
+// can answer them.
+type JudgeView struct {
+	Model string `json:"model"`
+	// Policies are those that ask it; Closed those that refuse what it
+	// cannot read (fail: closed).
+	Policies []string `json:"policies"`
+	Closed   []string `json:"closed,omitempty"`
+	// InCatalog is whether a model of that name is in the catalog, and
+	// Judge whether it is there as a judge (role: guard).
+	InCatalog bool `json:"in_catalog"`
+	Judge     bool `json:"judge"`
+	Ready     int  `json:"ready_replicas"`
+	Wanted    int  `json:"wanted_replicas"`
+	// LastError is its last failure to give a verdict, if any.
+	LastError   string     `json:"last_error,omitempty"`
+	LastErrorAt *time.Time `json:"last_error_at,omitempty"`
+	// Problem says, in words, what is wrong and what to do; empty when
+	// nothing is.
+	Problem string `json:"problem,omitempty"`
+}
+
+// judgeErrorFresh is how recent a judge's failure must be to count as a
+// problem now.
+const judgeErrorFresh = 5 * time.Minute
+
+// Judges are the judge models the policies in force ask, each with what is
+// wrong with it, if anything.
+func (a *API) Judges() []JudgeView {
+	if a.guard == nil || a.guard.Library == nil {
+		return nil
+	}
+	lib := a.guard.Library()
+	if !lib.On() {
+		// Off: no judge is asked, so none is missing.
+		return nil
+	}
+	byModel := map[string]*JudgeView{}
+	for name, p := range lib.Policies {
+		if p.Judge == nil || p.Judge.Model == "" {
+			continue
+		}
+		v := byModel[p.Judge.Model]
+		if v == nil {
+			v = &JudgeView{Model: p.Judge.Model}
+			byModel[p.Judge.Model] = v
+		}
+		v.Policies = append(v.Policies, name)
+		if p.Fail == "closed" && p.Mode != "shadow" {
+			v.Closed = append(v.Closed, name)
+		}
+	}
+	ready := map[string]int{}
+	for _, rep := range a.ctl.Reconciler().Replicas() {
+		if rep.State == reconciler.Ready {
+			ready[rep.Model]++
+		}
+	}
+	var errs map[string]guard.JudgeError
+	if a.guard.JudgeErrors != nil {
+		errs = a.guard.JudgeErrors()
+	}
+	out := make([]JudgeView, 0, len(byModel))
+	for _, v := range byModel {
+		sort.Strings(v.Policies)
+		sort.Strings(v.Closed)
+		if m, ok := a.ctl.Catalog().Resolve(v.Model); ok && m.Name == v.Model {
+			v.InCatalog, v.Judge = true, m.IsGuard()
+			v.Wanted, v.Ready = m.Scaling.Min, ready[m.Name]
+		}
+		if e, ok := errs[v.Model]; ok {
+			at := e.At
+			v.LastError, v.LastErrorAt = e.Err, &at
+		}
+		v.Problem = judgeProblem(*v)
+		out = append(out, *v)
+	}
+	slices.SortFunc(out, func(x, y JudgeView) int {
+		switch {
+		case x.Model < y.Model:
+			return -1
+		case x.Model > y.Model:
+			return 1
+		}
+		return 0
+	})
+	return out
+}
+
+func judgeProblem(v JudgeView) string {
+	consequence := "requests its policies judge are served unjudged"
+	if len(v.Closed) > 0 {
+		consequence = "requests its policies judge are refused (guard_unavailable)"
+	}
+	switch {
+	case !v.InCatalog:
+		return fmt.Sprintf("not in the catalog, so %s: llmgw ctl guard judge enable", consequence)
+	case !v.Judge:
+		return fmt.Sprintf("in the catalog as a model clients use, not as a judge (role: guard), so %s: "+
+			"llmgw ctl guard judge enable --replace", consequence)
+	case v.Ready == 0:
+		return fmt.Sprintf("no replica ready, so %s: llmgw ctl replicas --model %s", consequence, v.Model)
+	case v.LastErrorAt != nil && time.Since(*v.LastErrorAt) < judgeErrorFresh:
+		return fmt.Sprintf("gave no verdict %s ago (%s)", time.Since(*v.LastErrorAt).Round(time.Second), v.LastError)
+	}
+	return ""
+}
+
+func (a *API) listJudges(w http.ResponseWriter, _ *http.Request) {
+	if a.guard == nil {
+		writeError(w, http.StatusNotImplemented, "no guard policies: define them under guard in the cluster's settings (llmgw ctl config edit)")
+		return
+	}
+	writeJSON(w, http.StatusOK, a.Judges())
+}
+
+// JudgeAskRequest is a text to have a judge read.
+type JudgeAskRequest struct {
+	// Model is the judge; empty is the one the policies ask, when they ask one.
+	Model string `json:"model,omitempty"`
+	Text  string `json:"text"`
+	// Answer, when given, is read as the assistant's reply to Text.
+	Answer string `json:"answer,omitempty"`
+}
+
+// JudgeAskView is what the judge answered.
+type JudgeAskView struct {
+	Model string `json:"model"`
+	// Verdict is safe or unsafe, as the gateway reads the answer.
+	Verdict    string   `json:"verdict"`
+	Categories []string `json:"categories,omitempty"`
+	// Raw is the judge's answer as it gave it.
+	Raw    string `json:"raw"`
+	TookMs int64  `json:"took_ms"`
+}
+
+func (a *API) askJudge(w http.ResponseWriter, r *http.Request) {
+	if a.guard == nil || a.guard.Ask == nil {
+		writeError(w, http.StatusNotImplemented, "no guard policies: define them under guard in the cluster's settings (llmgw ctl config edit)")
+		return
+	}
+	var req JudgeAskRequest
+	if err := json.NewDecoder(r.Body).Decode(&req); err != nil || strings.TrimSpace(req.Text) == "" {
+		writeError(w, http.StatusBadRequest, "give the text for the judge to read")
+		return
+	}
+	if req.Model == "" {
+		judges := a.Judges()
+		if len(judges) == 0 {
+			writeError(w, http.StatusBadRequest, "no policy asks a judge: give the model")
+			return
+		}
+		req.Model = judges[0].Model
+	}
+	ans, err := a.guard.Ask(r.Context(), req.Model, req.Text, req.Answer)
+	v := JudgeAskView{Model: req.Model, Raw: ans.Raw, Categories: ans.Categories, TookMs: ans.Took.Milliseconds()}
+	if err != nil {
+		writeError(w, http.StatusBadGateway, fmt.Sprintf("the judge %s gave no verdict: %v (it answered %q)", req.Model, err, ans.Raw))
+		return
+	}
+	v.Verdict = "safe"
+	if ans.Unsafe {
+		v.Verdict = "unsafe"
+	}
+	writeJSON(w, http.StatusOK, v)
+}

internal/control/adminapi/judges_test.go

@@ -0,0 +1,91 @@
+package adminapi
+
+import (
+	"context"
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
+)
+
+func TestEachJudgeSaysWhatIsWrongWithIt(t *testing.T) {
+	models := append(testModels(),
+		// Served from the mirror page before it knew better: a model
+		// clients use, under the judge's name.
+		catalog.Model{Name: "shield", Source: "/s", Scaling: catalog.Scaling{Min: 1, Max: 1},
+			Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp"}}},
+		catalog.Model{Name: "llama-guard-3-1b", Source: "/g", Role: catalog.RoleGuard,
+			Guard: &catalog.GuardModel{Format: "llamaguard3"}, Scaling: catalog.Scaling{Min: 2, Max: 2},
+			Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp"}}})
+	a, _ := setup(t, models)
+	lib := guard.Config{Default: []string{"baseline"}, Policies: map[string]guard.Policy{
+		"baseline": {Fail: "closed", Judge: &guard.JudgePolicy{Model: "llama-guard-3-1b", Input: "block"}},
+		"strict":   {Fail: "closed", Judge: &guard.JudgePolicy{Model: "missing-judge", Input: "block"}},
+		"shield":   {Fail: "open", Judge: &guard.JudgePolicy{Model: "shield", Input: "flag"}},
+	}}
+	a.SetGuard(&GuardAdmin{Library: func() guard.Config { return lib }})
+
+	var js []JudgeView
+	if code := call(t, a, "GET", "/api/v1/guard/judges", "", &js); code != 200 || len(js) != 3 {
+		t.Fatalf("%d %+v", code, js)
+	}
+	by := map[string]JudgeView{}
+	for _, j := range js {
+		by[j.Model] = j
+	}
+	if j := by["missing-judge"]; j.InCatalog || !strings.Contains(j.Problem, "not in the catalog") ||
+		!strings.Contains(j.Problem, "refused") {
+		t.Errorf("missing: %+v", j)
+	}
+	if j := by["shield"]; !j.InCatalog || j.Judge || !strings.Contains(j.Problem, "--replace") ||
+		!strings.Contains(j.Problem, "unjudged") {
+		t.Errorf("not a judge: %+v", j)
+	}
+	if j := by["llama-guard-3-1b"]; !j.Judge || j.Wanted != 2 || !strings.Contains(j.Problem, "no replica ready") {
+		t.Errorf("not running: %+v", j)
+	}
+
+	var s Summary
+	call(t, a, "GET", "/api/v1/summary", "", &s)
+	if len(s.Judges) != 3 {
+		t.Errorf("the summary does not carry the judges' problems: %+v", s.Judges)
+	}
+
+	// A judge that failed lately says so, once it is otherwise fine.
+	at := time.Now()
+	v := JudgeView{Model: "x", InCatalog: true, Judge: true, Ready: 1, LastError: "deadline exceeded", LastErrorAt: &at}
+	if p := judgeProblem(v); !strings.Contains(p, "deadline exceeded") {
+		t.Errorf("a recent failure: %q", p)
+	}
+	old := at.Add(-time.Hour)
+	v.LastErrorAt = &old
+	if p := judgeProblem(v); p != "" {
+		t.Errorf("an old failure still counts: %q", p)
+	}
+}
+
+func TestAJudgeIsAskedAndItsWordsShown(t *testing.T) {
+	a, _ := setup(t, testModels())
+	lib := guard.Config{Policies: map[string]guard.Policy{
+		"baseline": {Judge: &guard.JudgePolicy{Model: "llama-guard-3-1b", Input: "block"}}}}
+	var asked string
+	a.SetGuard(&GuardAdmin{Library: func() guard.Config { return lib },
+		Ask: func(_ context.Context, model, text, _ string) (guard.JudgeAnswer, error) {
+			asked = model + "|" + text
+			return guard.JudgeAnswer{Model: model, Raw: "unsafe\nS6", Unsafe: true,
+				Categories: []string{"S6", "Specialized Advice"}}, nil
+		}})
+	var v JudgeAskView
+	if code := call(t, a, "POST", "/api/v1/guard/judges/ask", `{"text":"how do I sort a list"}`, &v); code != 200 {
+		t.Fatalf("%d", code)
+	}
+	if asked != "llama-guard-3-1b|how do I sort a list" || v.Verdict != "unsafe" || v.Raw != "unsafe\nS6" ||
+		len(v.Categories) != 2 {
+		t.Errorf("asked %q, got %+v", asked, v)
+	}
+	if code := call(t, a, "POST", "/api/v1/guard/judges/ask", `{"text":"  "}`, nil); code != 400 {
+		t.Errorf("no text: %d", code)
+	}
+}

internal/control/adminapi/notify.go

@@ -12,7 +12,7 @@ func (a *API) SetNotifier(n *notify.Notifier) { a.notifier = n }
 // notifyTest sends a test event to every webhook and says how each answered.
 func (a *API) notifyTest(w http.ResponseWriter, r *http.Request) {
 	if a.notifier == nil {
-		writeError(w, http.StatusNotFound, "no webhooks: server.yaml has no notify.webhooks")
+		writeError(w, http.StatusNotFound, "no webhooks: the cluster's settings have no notify.webhooks (llmgw ctl config set notify.webhooks ...)")
 		return
 	}
 	writeJSON(w, http.StatusOK, a.notifier.Test(r.Context()))

internal/control/adminapi/routes.go

@@ -131,7 +131,7 @@ var Routes = []Route{
 		Description: "With the defaults the catalog filled in. By its name only, not an alias.",
 		Response:    catalog.Model{}},
 	{Method: "DELETE", Path: "/api/v1/models/{name}", Summary: "Remove a model added at runtime",
-		Description: "A model declared in configuration is refused: it would come back on the next restart.",
+		Description: "A model declared in the cluster's settings is refused: it is changed there.",
 		Response:    Removed{}},
 	{Method: "POST", Path: "/api/v1/models/{name}/scale", Summary: "Set how many replicas a model runs",
 		Request: ScaleRequest{}, Response: ScaleResponse{}},
@@ -249,11 +249,33 @@ var Routes = []Route{
 		Response: Removed{}},
 	{Method: "GET", Path: "/api/v1/guard", Summary: "The guardrail policies and where they are attached",
 		Response: GuardView{}},
+	{Method: "GET", Path: "/api/v1/guard/events", Summary: "What the guardrails did: totals and the newest requests",
+		Query: []Param{
+			{Name: "since", Description: "How far back: 90m, 24h, 7d; 24h when absent."},
+		},
+		Response: GuardEventsView{}},
+	{Method: "GET", Path: "/api/v1/guard/judges", Summary: "The judge models the policies ask, and what is wrong with each",
+		Response: []JudgeView{}},
+	{Method: "POST", Path: "/api/v1/guard/judges/ask", Summary: "Have a judge read a text and say what it answered; nothing is kept",
+		Request: JudgeAskRequest{}, Response: JudgeAskView{}},
 	{Method: "POST", Path: "/api/v1/guard/attachments", Summary: "Attach a guardrail policy",
 		Request: GuardAttachmentRequest{}, Response: guard.Attachment{}},
 	{Method: "DELETE", Path: "/api/v1/guard/attachments", Summary: "Detach a guardrail policy",
 		Request: GuardAttachmentRequest{}, Response: guard.Attachment{}},
 
+	// The cluster's settings: everything in server.yaml that is not one
+	// machine's own, the same on every control plane.
+	{Method: "GET", Path: "/api/v1/settings", Summary: "The cluster's settings in force, or one earlier version",
+		Query:    []Param{{Name: "version", Description: "An earlier version; the newest when absent."}},
+		Response: SettingsView{}},
+	{Method: "GET", Path: "/api/v1/settings/history", Summary: "Versions of the cluster's settings, newest first",
+		Query:    []Param{{Name: "limit", Description: "How many; 20 when absent."}},
+		Response: []SettingsView{}},
+	{Method: "PUT", Path: "/api/v1/settings", Summary: "Change the cluster's settings: in force on every control plane within seconds",
+		Request: SettingsRequest{}, Response: SettingsView{}},
+	{Method: "POST", Path: "/api/v1/settings/rollback", Summary: "Put an earlier version of the cluster's settings back in force",
+		Request: SettingsRollbackRequest{}, Response: SettingsView{}},
+
 	// Usage, requests and the record of changes.
 	{Method: "GET", Path: "/api/v1/usage", Summary: "What each group of the tenancy tree spent",
 		Query: []Param{

internal/control/adminapi/settings.go

@@ -0,0 +1,179 @@
+package adminapi
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"net/http"
+	"strconv"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// SettingsStore keeps the cluster's settings.
+type SettingsStore interface {
+	ClusterSettings(ctx context.Context) (store.SettingsVersion, error)
+	ClusterSettingsAt(ctx context.Context, version int64) (store.SettingsVersion, error)
+	ClusterSettingsHistory(ctx context.Context, limit int) ([]store.SettingsVersion, error)
+	PutClusterSettings(ctx context.Context, base int64, body, by, note string) (int64, error)
+}
+
+// Settings is what the settings endpoints need: where they are kept, and a
+// check that a document would load — laid over this control plane's own
+// server.yaml, validated as it would be at start.
+type Settings struct {
+	Store SettingsStore
+	Check func(body []byte) error
+	// LocalKeys are those each machine's server.yaml keeps, which the
+	// document cannot set.
+	LocalKeys []string
+}
+
+// SetSettings enables the settings endpoints.
+func (a *API) SetSettings(s *Settings) { a.settings = s }
+
+// SettingsView is one version of the cluster's settings.
+type SettingsView struct {
+	Version   int64     `json:"version"`
+	Body      string    `json:"body"`
+	Note      string    `json:"note,omitempty"`
+	At        time.Time `json:"at"`
+	By        string    `json:"by,omitempty"`
+	LocalKeys []string  `json:"local_keys,omitempty"`
+}
+
+// SettingsRequest is a new version of the cluster's settings, made from the
+// version Base, which must still be the newest.
+type SettingsRequest struct {
+	Base int64  `json:"base"`
+	Body string `json:"body"`
+	Note string `json:"note,omitempty"`
+}
+
+// SettingsRollbackRequest puts an earlier version's settings back in force,
+// as a new version.
+type SettingsRollbackRequest struct {
+	Version int64  `json:"version"`
+	Note    string `json:"note,omitempty"`
+}
+
+func settingsView(v store.SettingsVersion) SettingsView {
+	return SettingsView{Version: v.Version, Body: v.Body, Note: v.Note, At: v.At, By: v.By}
+}
+
+func (a *API) registerSettings(mux Mux) {
+	need := func(w http.ResponseWriter) bool {
+		if a.settings == nil {
+			writeError(w, http.StatusNotImplemented, "this control plane has no store to keep the cluster's settings in")
+			return false
+		}
+		return true
+	}
+	mux.HandleFunc("GET /api/v1/settings", func(w http.ResponseWriter, r *http.Request) {
+		if !need(w) {
+			return
+		}
+		var v store.SettingsVersion
+		var err error
+		if q := r.URL.Query().Get("version"); q != "" {
+			n, perr := strconv.ParseInt(q, 10, 64)
+			if perr != nil {
+				writeError(w, http.StatusBadRequest, "version is a number")
+				return
+			}
+			v, err = a.settings.Store.ClusterSettingsAt(r.Context(), n)
+			if err == nil && v.Version == 0 {
+				writeError(w, http.StatusNotFound, "there is no such version")
+				return
+			}
+		} else {
+			v, err = a.settings.Store.ClusterSettings(r.Context())
+		}
+		if err != nil {
+			writeError(w, http.StatusInternalServerError, err.Error())
+			return
+		}
+		out := settingsView(v)
+		out.LocalKeys = a.settings.LocalKeys
+		writeJSON(w, http.StatusOK, out)
+	})
+	mux.HandleFunc("GET /api/v1/settings/history", func(w http.ResponseWriter, r *http.Request) {
+		if !need(w) {
+			return
+		}
+		limit := 20
+		if q := r.URL.Query().Get("limit"); q != "" {
+			if n, err := strconv.Atoi(q); err == nil && n > 0 && n <= 500 {
+				limit = n
+			}
+		}
+		list, err := a.settings.Store.ClusterSettingsHistory(r.Context(), limit)
+		if err != nil {
+			writeError(w, http.StatusInternalServerError, err.Error())
+			return
+		}
+		out := make([]SettingsView, len(list))
+		for i, v := range list {
+			out[i] = settingsView(v)
+		}
+		writeJSON(w, http.StatusOK, out)
+	})
+	put := func(w http.ResponseWriter, r *http.Request, base int64, body, note string) {
+		if err := a.settings.Check([]byte(body)); err != nil {
+			writeError(w, http.StatusBadRequest, err.Error())
+			return
+		}
+		n, err := a.settings.Store.PutClusterSettings(r.Context(), base, body, actorOf(r), note)
+		switch {
+		case errors.Is(err, store.ErrSettingsChanged):
+			writeError(w, http.StatusConflict, err.Error())
+			return
+		case err != nil:
+			writeError(w, http.StatusInternalServerError, err.Error())
+			return
+		}
+		v, err := a.settings.Store.ClusterSettingsAt(r.Context(), n)
+		if err != nil {
+			writeError(w, http.StatusInternalServerError, err.Error())
+			return
+		}
+		writeJSON(w, http.StatusOK, settingsView(v))
+	}
+	mux.HandleFunc("PUT /api/v1/settings", func(w http.ResponseWriter, r *http.Request) {
+		if !need(w) {
+			return
+		}
+		var req SettingsRequest
+		if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
+			writeError(w, http.StatusBadRequest, err.Error())
+			return
+		}
+		put(w, r, req.Base, req.Body, req.Note)
+	})
+	mux.HandleFunc("POST /api/v1/settings/rollback", func(w http.ResponseWriter, r *http.Request) {
+		if !need(w) {
+			return
+		}
+		var req SettingsRollbackRequest
+		if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
+			writeError(w, http.StatusBadRequest, err.Error())
+			return
+		}
+		old, err := a.settings.Store.ClusterSettingsAt(r.Context(), req.Version)
+		if err == nil && old.Version == 0 {
+			writeError(w, http.StatusNotFound, "there is no such version")
+			return
+		}
+		cur, err2 := a.settings.Store.ClusterSettings(r.Context())
+		if err = errors.Join(err, err2); err != nil {
+			writeError(w, http.StatusInternalServerError, err.Error())
+			return
+		}
+		note := req.Note
+		if note == "" {
+			note = "back to version " + strconv.FormatInt(req.Version, 10)
+		}
+		put(w, r, cur.Version, old.Body, note)
+	})
+}

internal/control/adminapi/settings_test.go

@@ -0,0 +1,78 @@
+package adminapi
+
+import (
+	"context"
+	"errors"
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+type memSettings struct{ versions []store.SettingsVersion }
+
+func (m *memSettings) ClusterSettings(context.Context) (store.SettingsVersion, error) {
+	if len(m.versions) == 0 {
+		return store.SettingsVersion{}, nil
+	}
+	return m.versions[len(m.versions)-1], nil
+}
+
+func (m *memSettings) ClusterSettingsAt(_ context.Context, v int64) (store.SettingsVersion, error) {
+	if v < 1 || int(v) > len(m.versions) {
+		return store.SettingsVersion{}, nil
+	}
+	return m.versions[v-1], nil
+}
+
+func (m *memSettings) ClusterSettingsHistory(context.Context, int) ([]store.SettingsVersion, error) {
+	var out []store.SettingsVersion
+	for i := len(m.versions) - 1; i >= 0; i-- {
+		out = append(out, m.versions[i])
+	}
+	return out, nil
+}
+
+func (m *memSettings) PutClusterSettings(_ context.Context, base int64, body, by, note string) (int64, error) {
+	if base != int64(len(m.versions)) {
+		return 0, store.ErrSettingsChanged
+	}
+	m.versions = append(m.versions, store.SettingsVersion{Version: base + 1, Body: body, By: by, Note: note, At: time.Now()})
+	return base + 1, nil
+}
+
+func TestSettingsAreCheckedKeptAndRolledBack(t *testing.T) {
+	a, _ := setup(t, testModels())
+	st := &memSettings{}
+	a.SetSettings(&Settings{Store: st, Check: func(b []byte) error {
+		if strings.Contains(string(b), "listen") {
+			return errors.New("listen is each machine's")
+		}
+		return nil
+	}})
+	var v SettingsView
+	if code := call(t, a, "PUT", "/api/v1/settings", `{"base":0,"body":"router:\n  retries: 2\n"}`, &v); code != 200 || v.Version != 1 {
+		t.Fatalf("first: %d %+v", code, v)
+	}
+	if code := call(t, a, "PUT", "/api/v1/settings", `{"base":1,"body":"listen: :1\n"}`, nil); code != 400 {
+		t.Errorf("a document the check refuses: %d", code)
+	}
+	if code := call(t, a, "PUT", "/api/v1/settings", `{"base":0,"body":"router: {}\n"}`, nil); code != 409 {
+		t.Errorf("a change from an old version: %d", code)
+	}
+	if code := call(t, a, "PUT", "/api/v1/settings", `{"base":1,"body":"router:\n  retries: 5\n"}`, nil); code != 200 {
+		t.Fatalf("second: %d", code)
+	}
+	if code := call(t, a, "POST", "/api/v1/settings/rollback", `{"version":1}`, &v); code != 200 ||
+		v.Version != 3 || v.Body != "router:\n  retries: 2\n" || v.Note != "back to version 1" {
+		t.Errorf("rollback: %d %+v", code, v)
+	}
+	var h []SettingsView
+	if code := call(t, a, "GET", "/api/v1/settings/history", "", &h); code != 200 || len(h) != 3 || h[0].Version != 3 {
+		t.Errorf("history: %d %+v", code, h)
+	}
+	if code := call(t, a, "GET", "/api/v1/settings?version=9", "", nil); code != 404 {
+		t.Errorf("a version that is not there: %d", code)
+	}
+}

internal/control/adminapi/summary.go

@@ -1,6 +1,7 @@
 package adminapi
 
 import (
+	"gitea.mynodes.xyz/ilker/llmgw/internal/version"
 	"sort"
 	"time"
 )
@@ -14,8 +15,11 @@ import (
 // derived from the same state the tables show, so it can never disagree with
 // them; it is a different question asked of the same facts.
 type Summary struct {
-	Nodes NodeTotals `json:"nodes"`
-	GPUs  []ClassUse `json:"gpus"`
+	// Version is the release the answering control plane runs: not the
+	// binary on its disk, which a manual install changes underneath it.
+	Version string     `json:"version,omitempty"`
+	Nodes   NodeTotals `json:"nodes"`
+	GPUs    []ClassUse `json:"gpus"`
 	// Fabrics matter because a multi-node replica may never span two of them,
 	// so free capacity in the wrong fabric is not capacity for the model that
 	// needs it.
@@ -38,6 +42,11 @@ type Summary struct {
 	// ControlPlanes is the cluster's control planes when there is more than
 	// one: which leads, which stand by, and the floating address in front.
 	ControlPlanes ControlPlanes `json:"control_planes,omitzero"`
+	// Judges are the guardrails' judge models with something wrong.
+	Judges []JudgeView `json:"judges,omitempty"`
+	// GuardOff says the guardrails are turned off (guard.enabled: false):
+	// no request is checked, whatever the policies say.
+	GuardOff bool `json:"guard_off,omitempty"`
 }
 
 // StoreMember is one member of a replicated store, as its Patroni reports it.
@@ -144,7 +153,15 @@ const idleThreshold = 10 * time.Minute
 
 // Summary builds the fleet summary.
 func (a *API) Summary() Summary {
-	var s Summary
+	s := Summary{Version: version.Version}
+	if a.guard != nil && a.guard.Library != nil && !a.guard.Library().On() {
+		s.GuardOff = true
+	}
+	for _, j := range a.Judges() {
+		if j.Problem != "" {
+			s.Judges = append(s.Judges, j)
+		}
+	}
 	if a.storeStatus != nil {
 		members, err := a.storeStatus()
 		s.Store = members

internal/control/agentsvc/firewall.go

@@ -0,0 +1,42 @@
+package agentsvc
+
+import (
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+)
+
+// SetFirewall gives the service whether nodes keep llmgw's firewall, read
+// at each use so a change to the cluster's settings is what is sent next.
+func (s *Service) SetFirewall(f func() *llmgwv1.Firewall) {
+	s.mu.Lock()
+	s.firewall = f
+	s.mu.Unlock()
+}
+
+func (s *Service) firewallMsg() *llmgwv1.ControlMessage {
+	s.mu.RLock()
+	f := s.firewall
+	s.mu.RUnlock()
+	if f == nil {
+		return nil
+	}
+	fw := f()
+	if fw == nil {
+		return nil
+	}
+	return &llmgwv1.ControlMessage{Msg: &llmgwv1.ControlMessage_Firewall{Firewall: fw}}
+}
+
+// BroadcastFirewall tells every connected agent what the cluster's settings
+// now say of the firewall. A node not connected is told when it registers.
+func (s *Service) BroadcastFirewall() {
+	msg := s.firewallMsg()
+	if msg == nil {
+		return
+	}
+	for _, node := range s.ConnectedNodes() {
+		if err := s.send(node, msg); err != nil {
+			s.log.Warn("could not tell a node about the firewall; it is told when it next registers",
+				"node", node, "err", err)
+		}
+	}
+}

internal/control/agentsvc/service.go

@@ -75,6 +75,8 @@ type Service struct {
 
 	// controlPlanes lists what agents may dial; see SetControlPlanes.
 	controlPlanes func(context.Context) ([]string, error)
+	// firewall is what nodes are told of llmgw's firewall; see SetFirewall.
+	firewall func() *llmgwv1.Firewall
 
 	// logWaits are the log requests awaiting a reply, by request id.
 	logMu    sync.Mutex
@@ -286,6 +288,12 @@ func (s *Service) Connect(
 			return err
 		}
 	}
+	// Whether to keep llmgw's firewall, as the cluster's settings say now.
+	if msg := s.firewallMsg(); msg != nil {
+		if err := stream.Send(msg); err != nil {
+			return err
+		}
+	}
 
 	// Recover replica state from what this node is already running, before any
 	// reconciliation pass can conclude the cluster is empty and place

internal/control/enginechoice.go

@@ -43,6 +43,39 @@ func NewEngineChoices(ctx context.Context, st EngineChoiceStore, log *slog.Logge
 	return c
 }
 
+// Reload reads the choices again, as a standby becoming the leader does, and
+// tells each node whose choice changed.
+func (c *EngineChoices) Reload(ctx context.Context) {
+	got, err := c.st.NodeEngines(ctx)
+	if err != nil {
+		c.log.Error("could not read the engine builds chosen per node again; keeping the ones read before", "err", err)
+		return
+	}
+	c.mu.Lock()
+	var changed []string
+	for node := range got {
+		if !maps.Equal(got[node], c.byNode[node]) {
+			changed = append(changed, node)
+		}
+	}
+	for node := range c.byNode {
+		if _, ok := got[node]; !ok {
+			changed = append(changed, node)
+		}
+	}
+	c.byNode = got
+	notify := c.notify
+	c.mu.Unlock()
+	if notify == nil {
+		return
+	}
+	for _, node := range changed {
+		if err := notify(node); err != nil {
+			c.log.Warn("could not tell a node its engine builds", "node", node, "err", err)
+		}
+	}
+}
+
 // OnChange is called with a node whose choice changed, to tell it.
 func (c *EngineChoices) OnChange(f func(node string) error) {
 	c.mu.Lock()

internal/control/facility.go

@@ -68,6 +68,23 @@ func (f *Facility) UseStore(ctx context.Context, st FacilityStore) {
 	f.mu.Lock()
 	defer f.mu.Unlock()
 	f.st = st
+	f.loadLocked(ctx)
+}
+
+// Reload reads the holds and labels in the store again, as a standby
+// becoming the leader does.
+func (f *Facility) Reload(ctx context.Context) {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	if f.st == nil {
+		return
+	}
+	f.holds = map[facility.Domain]facility.Hold{}
+	f.loadLocked(ctx)
+}
+
+func (f *Facility) loadLocked(ctx context.Context) {
+	st := f.st
 	if holds, err := st.FacilityHolds(ctx); err != nil {
 		f.log.Error("could not read the domains held for maintenance; none is held until they are set again", "err", err)
 	} else {

internal/control/leadership.go

@@ -7,7 +7,9 @@ import (
 	"errors"
 	"log/slog"
 	"os"
+	"slices"
 	"strings"
+	"sync"
 	"sync/atomic"
 	"time"
 
@@ -54,6 +56,27 @@ type Leadership struct {
 	leader atomic.Bool
 	// stopped is set by StepDown: nothing takes the lease again after it.
 	stopped atomic.Bool
+
+	hookMu sync.Mutex
+	onLead []func()
+}
+
+// OnLead is called each time this process becomes the leader, after
+// standing by: what it read at start may have been changed since by the
+// leader before it, and is read again.
+func (l *Leadership) OnLead(f func()) {
+	l.hookMu.Lock()
+	l.onLead = append(l.onLead, f)
+	l.hookMu.Unlock()
+}
+
+func (l *Leadership) becameLeader() {
+	l.hookMu.Lock()
+	hooks := slices.Clone(l.onLead)
+	l.hookMu.Unlock()
+	for _, f := range hooks {
+		go f()
+	}
 }
 
 // NewLeadership prepares leadership over a store.
@@ -133,6 +156,7 @@ func (l *Leadership) attempt(ctx context.Context) {
 	case err == nil:
 		if l.leader.CompareAndSwap(false, true) {
 			l.log.Info("this control plane is now placing work", "owner", l.owner)
+			l.becameLeader()
 		}
 	case errors.Is(err, store.ErrNotLeader):
 		if l.leader.CompareAndSwap(true, false) {

internal/control/leadership_test.go

@@ -49,3 +49,29 @@ func TestAStoppingLeaderHandsOverAtOnceAndNeverTakesItBack(t *testing.T) {
 		t.Errorf("holder name %q from %q", got, b.Owner())
 	}
 }
+
+func TestAStandbyIsToldWhenItBecomesTheLeader(t *testing.T) {
+	st, err := store.Open(t.Context(), filepath.Join(t.TempDir(), "s.db"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer st.Close()
+	log := slog.New(slog.DiscardHandler)
+	a := NewLeadership(st, time.Minute, log)
+	b := NewLeadership(st, time.Minute, log)
+	led := make(chan struct{}, 4)
+	b.OnLead(func() { led <- struct{}{} })
+	ctx := t.Context()
+	a.attempt(ctx)
+	b.attempt(ctx)
+	if !a.IsLeader() || b.IsLeader() {
+		t.Fatal("a should lead, b stand by")
+	}
+	a.StepDown()
+	b.attempt(ctx)
+	select {
+	case <-led:
+	case <-time.After(time.Second):
+		t.Fatal("b became the leader and was not told")
+	}
+}

internal/control/modelserve/modelserve.go

@@ -140,9 +140,14 @@ type Request struct {
 	// llamaguard3, shieldgemma or json. Empty serves clients.
 	GuardFormat string `json:"guard_format,omitempty"`
 	// CtxPool is llama.cpp's --ctx-size; ContextLength what one request may
-	// use. They default to 16384 and 4096, what four parallel slots divide.
+	// use. They default to 16384 and 4096, what four parallel slots divide;
+	// llama.cpp is given as many slots as the pool holds requests.
 	CtxPool       int `json:"ctx_pool,omitempty"`
 	ContextLength int `json:"context_length,omitempty"`
+	// Tools says the model makes tool calls the engine returns as calls: it
+	// is written into the catalog, where clients and editors read it. Check
+	// a running replica with llmgw ctl tool-check before saying so.
+	Tools bool `json:"tools,omitempty"`
 	// CPUCores is what a replica computes on; zero takes half the smallest
 	// node it fits on.
 	CPUCores int `json:"cpu_cores,omitempty"`
@@ -261,6 +266,10 @@ func (s *Server) Start(req Request) (Job, error) {
 	if req.ContextLength <= 0 {
 		req.ContextLength = 4096
 	}
+	if req.ContextLength > req.CtxPool {
+		return Job{}, fmt.Errorf("a request of %d tokens cannot fit a pool of %d: give --ctx-pool at least --context-length",
+			req.ContextLength, req.CtxPool)
+	}
 	s.mu.Lock()
 	s.seq++
 	j := &Job{ID: fmt.Sprintf("serve-%d-%d", time.Now().Unix(), s.seq), Request: req,
@@ -545,12 +554,12 @@ func profilesFor(layout []modelplan.Option, req Request, memGB, cores int) []cat
 		switch {
 		case o.CPU:
 			p.Name = "cpu"
-			p.Args = []string{"--ctx-size", strconv.Itoa(req.CtxPool)}
+			p.Args = llamaArgs(req)
 			p.MinMemoryGB, p.CPUCores = memGB, cores
 		case o.Engine == "llamacpp":
 			p.Name = fmt.Sprintf("gpu-%dx%s", o.GPUsPerNode, o.Class)
 			p.GPUClass, p.GPUsPerNode = o.Class, o.GPUsPerNode
-			p.Args = []string{"--ctx-size", strconv.Itoa(req.CtxPool)}
+			p.Args = llamaArgs(req)
 		default:
 			p.Name = fmt.Sprintf("gpu-%dx%s", o.GPUs(), o.Class)
 			p.GPUClass, p.TP = o.Class, o.GPUsPerNode
@@ -564,6 +573,23 @@ func profilesFor(layout []modelplan.Option, req Request, memGB, cores int) []cat
 	return out
 }
 
+// llamaArgs size llama.cpp's pool and its slots together. The pool is shared
+// by the slots, so slots of the context the catalog promises are as many as
+// the pool holds: the engine's default of four turned a 32768 pool into
+// slots of 8192, and the gateway, told 32768, sent requests the engine then
+// refused.
+func llamaArgs(req Request) []string {
+	return []string{"--ctx-size", strconv.Itoa(req.CtxPool),
+		"--parallel", strconv.Itoa(max(1, req.CtxPool/req.ContextLength))}
+}
+
+// maxOutput is the longest answer the catalog allows: a quarter of the
+// context, from 1024 to 8192. 1024 everywhere cut an editor's agent off in
+// the middle of the file it was writing.
+func maxOutput(contextLength int) int {
+	return min(8192, max(1024, contextLength/4))
+}
+
 // addAndStart puts the model in the catalog with its profile, waits for a
 // replica and asks it to answer.
 // hardwareExists is whether the cluster has what the ways chosen need, busy
@@ -583,10 +609,10 @@ func (s *Server) addAndStart(ctx context.Context, j *Job, repo hf.Repo, set hf.F
 	// 4. The catalog.
 	m := catalog.Model{
 		Name: name, Aliases: req.Aliases, Description: req.Description, Source: source,
-		RoutingProfile: "chat", ContextLength: req.ContextLength, MaxOutput: 1024,
+		RoutingProfile: "chat", ContextLength: req.ContextLength, MaxOutput: maxOutput(req.ContextLength),
 		Profiles:     profs,
 		Scaling:      catalog.Scaling{Min: req.Replicas, Max: max(req.Replicas, 1)},
-		Capabilities: catalog.Capabilities{Vision: req.Projector != ""},
+		Capabilities: catalog.Capabilities{Vision: req.Projector != "", Tools: req.Tools},
 		Origin:       &catalog.Origin{Repo: repo.ID, Revision: repo.Revision, Files: originFiles(set)},
 	}
 	if req.GuardFormat != "" {

internal/control/modelserve/slots_test.go

@@ -0,0 +1,31 @@
+package modelserve
+
+import (
+	"slices"
+	"testing"
+)
+
+func TestLlamaCppIsGivenTheSlotsTheCatalogPromises(t *testing.T) {
+	// A 32768 pool for 32768-token requests is one slot: the engine's own
+	// four made slots of 8192, and requests the gateway admitted were refused.
+	for _, c := range []struct {
+		pool, ctx int
+		want      []string
+	}{
+		{32768, 32768, []string{"--ctx-size", "32768", "--parallel", "1"}},
+		{16384, 4096, []string{"--ctx-size", "16384", "--parallel", "4"}},
+		{65536, 16384, []string{"--ctx-size", "65536", "--parallel", "4"}},
+	} {
+		if got := llamaArgs(Request{CtxPool: c.pool, ContextLength: c.ctx}); !slices.Equal(got, c.want) {
+			t.Errorf("pool %d, context %d: %v", c.pool, c.ctx, got)
+		}
+	}
+}
+
+func TestTheLongestAnswerGrowsWithTheContext(t *testing.T) {
+	for ctx, want := range map[int]int{2048: 1024, 4096: 1024, 16384: 4096, 32768: 8192, 131072: 8192} {
+		if got := maxOutput(ctx); got != want {
+			t.Errorf("context %d: max output %d, want %d", ctx, got, want)
+		}
+	}
+}

internal/control/modelsvc/modelsvc.go

@@ -109,7 +109,7 @@ func (s *Service) Add(ctx context.Context, m catalog.Model, by string) error {
 	defer s.mu.Unlock()
 
 	if s.Declared(m.Name) {
-		return fmt.Errorf("model %q is declared in configuration; edit that rather than adding a second definition", m.Name)
+		return fmt.Errorf("model %q is declared in the cluster's settings; edit it there (llmgw ctl config edit) rather than adding a second definition", m.Name)
 	}
 	// A model with no GPU is all host memory, and one that declares none
 	// fits everywhere as far as placement can tell — however many of it
@@ -170,7 +170,7 @@ func (s *Service) Remove(ctx context.Context, name string) error {
 	defer s.mu.Unlock()
 
 	if s.Declared(name) {
-		return fmt.Errorf("model %q is declared in configuration; remove it there, or it comes back on the next restart", name)
+		return fmt.Errorf("model %q is declared in the cluster's settings, under models; remove it there: llmgw ctl config edit", name)
 	}
 
 	stored, err := s.storedModels(ctx)
@@ -368,7 +368,7 @@ func (s *Service) update(ctx context.Context, name, by, what string, change func
 
 	if s.Declared(name) {
 		return catalog.Model{}, fmt.Errorf(
-			"model %q is declared in configuration; %s there, or it reverts on the next restart", name, what)
+			"model %q is declared in the cluster's settings, under models; %s there: llmgw ctl config edit", name, what)
 	}
 	stored, err := s.storedModels(ctx)
 	if err != nil {

internal/control/modelsvc/modelsvc_test.go

@@ -96,7 +96,7 @@ func TestConfigurationWinsOverTheDatabase(t *testing.T) {
 	if err == nil {
 		t.Fatal("a stored model was allowed to shadow a declared one")
 	}
-	if !strings.Contains(err.Error(), "declared in configuration") {
+	if !strings.Contains(err.Error(), "declared in the cluster's settings") {
 		t.Errorf("the error does not say why: %v", err)
 	}
 }
@@ -109,7 +109,7 @@ func TestADeclaredModelCannotBeRemovedAtRuntime(t *testing.T) {
 	if err == nil {
 		t.Fatal("a declared model was removed at runtime")
 	}
-	if !strings.Contains(err.Error(), "comes back") {
+	if !strings.Contains(err.Error(), "llmgw ctl config edit") {
 		t.Errorf("the error does not explain the consequence: %v", err)
 	}
 }
@@ -264,7 +264,7 @@ func TestADeclaredModelCannotBeScaledHere(t *testing.T) {
 	if err == nil {
 		t.Fatal("expected scaling a declared model to be refused")
 	}
-	if !strings.Contains(err.Error(), "configuration") {
+	if !strings.Contains(err.Error(), "cluster's settings") {
 		t.Errorf("the error should say where to change it; got %q", err)
 	}
 }

internal/docgen/reference_gen.go

@@ -117,6 +117,9 @@ var Reference = []Block{
 		{Key: "name", Type: "string", Doc: "Name is this cluster as a peer reads it, in the federated_from tag\nsent with a request to a peer set to receive tags (`ctl federation add\n--send-tags`). Empty is this machine's host name.", Nested: ""},
 		{Key: "interval", Type: "time.Duration", Doc: "Interval is how often each peer is asked whether it is ready, timed,\nand what it offers. A peer that does not answer is taken out of\nrouting until it does. Zero is 10s.", Nested: ""},
 	}},
+	{Package: "config", Name: "FirewallSettings", Title: "FirewallSettings", Doc: "", Fields: []Field{
+		{Key: "enabled", Type: "bool", Doc: "Enabled keeps llmgw's internal ports — engines, node exporter, the\nagents' stream, the mirror, the store — to the cluster's networks on\nevery machine. False removes llmgw's nftables table everywhere, for a\nsite whose own firewall does that job; the rules are kept, and true\nputs them back. Changed in the cluster's settings, it is in force on\nevery node and control plane within seconds.", Nested: ""},
+	}},
 	{Package: "catalog", Name: "GuardModel", Title: "GuardModel", Doc: "", Fields: []Field{
 		{Key: "format", Type: "string", Doc: "Format is llamaguard3 (Meta's Llama Guard 3: \"safe\", or \"unsafe\" and\nthe categories), shieldgemma (Google's ShieldGemma: Yes or No), or\njson ({\"safe\": false, \"categories\": [\"...\"]}).", Nested: ""},
 		{Key: "categories", Type: "[]string", Doc: "Categories the model is asked about, for shieldgemma — each a policy\nsentence — and the names reported for llamaguard3's S1, S2, ….", Nested: ""},
@@ -363,6 +366,7 @@ var Reference = []Block{
 		{Key: "guard", Type: "guard.Config", Doc: "Guard is the library of guardrail policies: what requests and answers\nare checked for, and what is done. Applied to every request through\nguard.default, and to parts of the tenancy tree by attaching them\n(llmgw ctl guard attach). Empty checks nothing.", Nested: "guard.Config"},
 		{Key: "secrets", Type: "Secrets", Doc: "Secrets is where secrets kept outside files are read from, and the key\nthe store's own secrets are sealed under.", Nested: "config.Secrets"},
 		{Key: "notify", Type: "notify.Config", Doc: "Notify sends what needs a person — a model parked, a node lost, a\nbudget spent, new weights published — to webhooks, signed, without a\nPrometheus and Alertmanager in between. Empty sends nothing.", Nested: "notify.Config"},
+		{Key: "firewall", Type: "FirewallSettings", Doc: "Firewall turns llmgw's firewall on every machine on or off.", Nested: "config.FirewallSettings"},
 		{Key: "firewall_networks", Type: "[]string", Doc: "FirewallNetworks are networks, besides those each machine reaches the\ncluster on, that llmgw's internal ports answer: a monitoring host on\nanother subnet, nodes behind a router. Written into every machine's\nfirewall by `server init`, `node add` and `server add`.", Nested: ""},
 		{Key: "backup_recipients", Type: "[]string", Doc: "BackupRecipients encrypt every archive `llmgw server backup` writes, to\nwhoever holds one of their keys: age recipients (age1…) or SSH public\nkeys (ssh-ed25519 …, ssh-rsa …), as `age` takes them. An archive holds\nthe certificate authority's key and every secret in the store, and is\ncopied off the machine; encrypted, the copy is safe wherever it lands.\nEmpty writes archives unencrypted, readable by root only.", Nested: ""},
 		{Key: "backup_copy_to", Type: "[]string", Doc: "BackupCopyTo is where each archive is copied once written, so it\nsurvives this machine: a directory (a mounted share, another disk) or\nssh://user@host[:port]/path, reached with BackupCopyKey and checked\nagainst /etc/llmgw/backup_known_hosts, which learns a host the first\ntime. The same number of archives is kept there as here. Refused\nwithout backup_recipients: an archive leaves the machine encrypted.", Nested: ""},
@@ -480,6 +484,7 @@ var Reference = []Block{
 		{Key: "sim", Type: "SimConfig", Doc: "", Nested: "engines.SimConfig"},
 	}},
 	{Package: "guard", Name: "Config", Title: "guard.Config", Doc: "", Fields: []Field{
+		{Key: "enabled", Type: "*bool", Doc: "Enabled turns every policy on or off at once: false checks nothing,\nwhatever is defined or attached. Unset is on.", Nested: ""},
 		{Key: "default", Type: "[]string", Doc: "", Nested: ""},
 		{Key: "policies", Type: "map[string]Policy", Doc: "", Nested: ""},
 	}},

internal/docgen/render.go

@@ -30,6 +30,7 @@ var Sources = []string{
 	"internal/guard/policy.go",
 	"internal/guard/judge.go",
 	"internal/config/oidc.go",
+	"internal/config/firewall.go",
 	"internal/notify/notify.go",
 	"internal/secrets/secrets.go",
 	"internal/obs/trace.go",

internal/firewall/firewall.go

@@ -11,10 +11,13 @@
 package firewall
 
 import (
+	"bytes"
 	"context"
 	"errors"
 	"fmt"
 	"net/netip"
+	"os"
+	"os/exec"
 	"slices"
 	"strconv"
 	"strings"
@@ -317,3 +320,87 @@ func Present(ctx context.Context, r Runner) bool {
 	_, err := r.Run(ctx, "sudo -n /usr/sbin/nft list table inet "+Table+" >/dev/null 2>&1", nil)
 	return err == nil
 }
+
+// Admit lets addresses through a machine's ruleset when it has one: each
+// one's network, as Networks finds it, unless a network already there holds
+// it. A machine without llmgw's rules is left without them. What was added
+// is returned, nothing when every address was already let in.
+func Admit(ctx context.Context, r Runner, addrs []string) ([]netip.Prefix, error) {
+	old, err := r.Run(ctx, "sudo cat "+RulesFile+" 2>/dev/null || true", nil)
+	if err != nil {
+		return nil, err
+	}
+	prev := parse(string(old))
+	if len(prev.Ports) == 0 {
+		return nil, nil
+	}
+	var missing []string
+	for _, s := range addrs {
+		a, err := netip.ParseAddr(s)
+		if err != nil {
+			continue
+		}
+		if !slices.ContainsFunc(prev.Allow, func(p netip.Prefix) bool { return p.Contains(a) }) {
+			missing = append(missing, s)
+		}
+	}
+	if len(missing) == 0 {
+		return nil, nil
+	}
+	nets, err := Networks(ctx, r, missing)
+	if err != nil {
+		return nil, err
+	}
+	if _, err := Install(ctx, r, Spec{Allow: nets}); err != nil {
+		return nil, err
+	}
+	return nets, nil
+}
+
+// Disable takes llmgw's table out of a machine's ruleset and stops it being
+// loaded at boot, keeping the rules file: Install, or enabling the unit,
+// puts it back as it was. Reports whether there was anything to take out.
+func Disable(ctx context.Context, r Runner) (bool, error) {
+	had := Present(ctx, r)
+	out, err := r.Run(ctx, `sudo systemctl disable --now `+Unit+` >/dev/null 2>&1 || true
+if sudo /usr/sbin/nft list table inet `+Table+` >/dev/null 2>&1; then
+  sudo /usr/sbin/nft delete table inet `+Table+`
+fi`, nil)
+	if err != nil {
+		return false, fmt.Errorf("%v: %s", err, strings.TrimSpace(string(out)))
+	}
+	return had, nil
+}
+
+// Enable loads the rules file a machine already has and has it loaded at
+// boot again, after Disable. Reports false when there is no rules file.
+func Enable(ctx context.Context, r Runner) (bool, error) {
+	out, err := r.Run(ctx, `[ -f `+RulesFile+` ] || { echo none; exit 0; }
+sudo systemctl enable `+Unit+` >/dev/null 2>&1
+sudo /usr/sbin/nft -f `+RulesFile+`
+sudo systemctl start `+Unit+` >/dev/null 2>&1 || true
+echo loaded`, nil)
+	if err != nil {
+		return false, fmt.Errorf("%v: %s", err, strings.TrimSpace(string(out)))
+	}
+	return strings.TrimSpace(string(out)) == "loaded", nil
+}
+
+// Local runs scripts on this machine. As root, sudo in them runs its command
+// directly, for a machine that has no sudo.
+type Local struct{}
+
+// Run runs script with sh.
+func (Local) Run(ctx context.Context, script string, stdin []byte) ([]byte, error) {
+	if os.Geteuid() == 0 {
+		script = "sudo() { \"$@\"; }\n" + script
+	}
+	cmd := exec.CommandContext(ctx, "sh", "-c", script)
+	if stdin != nil {
+		cmd.Stdin = bytes.NewReader(stdin)
+	}
+	var out bytes.Buffer
+	cmd.Stdout, cmd.Stderr = &out, &out
+	err := cmd.Run()
+	return out.Bytes(), err
+}

internal/firewall/firewall_test.go

@@ -1,6 +1,7 @@
 package firewall
 
 import (
+	"context"
 	"net/netip"
 	"strings"
 	"testing"
@@ -58,3 +59,54 @@ func TestARulesetReadsBackAsItsSpec(t *testing.T) {
 		t.Errorf("read back as\n%s\nfrom\n%s", Script(got), Script(in))
 	}
 }
+
+// fakeMachine answers the scripts Admit runs: the ruleset file, the route,
+// and an install, which it records.
+type fakeMachine struct {
+	rules     string
+	route     string
+	installed []byte
+}
+
+func (f *fakeMachine) Run(_ context.Context, script string, stdin []byte) ([]byte, error) {
+	switch {
+	case strings.HasPrefix(script, "sudo cat "):
+		return []byte(f.rules), nil
+	case strings.HasPrefix(script, "ip -o route get"):
+		return []byte(f.route), nil
+	default:
+		f.installed = stdin
+		return []byte("changed\n"), nil
+	}
+}
+
+func TestANodeOnAnotherNetworkIsLetIn(t *testing.T) {
+	rules := Script(Spec{Ports: []Port{AgentPort, MirrorFiles},
+		Allow: []netip.Prefix{netip.MustParsePrefix("10.104.130.128/25")}})
+	m := &fakeMachine{rules: rules,
+		route: "10.20.0.5 via 10.104.130.129 dev eth0 src 10.104.130.140 uid 0\n" +
+			"2: eth0    inet 10.104.130.140/25 brd 10.104.130.255 scope global eth0\n"}
+
+	added, err := Admit(context.Background(), m, []string{"10.104.130.200"})
+	if err != nil || len(added) != 0 || m.installed != nil {
+		t.Fatalf("an address the rules hold: added %v, err %v, installed %v", added, err, m.installed != nil)
+	}
+
+	added, err = Admit(context.Background(), m, []string{"10.20.0.5", "gpu01"})
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(added) != 1 || added[0].String() != "10.20.0.5/32" {
+		t.Errorf("added %v, want the routed node's address alone", added)
+	}
+	got := parse(strings.Split(string(m.installed), "\x00")[0])
+	if len(got.Ports) != 2 || len(got.Allow) != 2 {
+		t.Errorf("installed %+v, want both ports kept and both networks", got)
+	}
+
+	// No llmgw rules on the machine: none are made.
+	bare := &fakeMachine{}
+	if added, err := Admit(context.Background(), bare, []string{"10.20.0.5"}); err != nil || added != nil || bare.installed != nil {
+		t.Errorf("a machine without rules: added %v, err %v", added, err)
+	}
+}

internal/guard/gate.go

@@ -5,10 +5,12 @@ import (
 	"encoding/json"
 	"fmt"
 	"log/slog"
+	"maps"
 	"regexp"
 	"sort"
 	"strings"
 	"sync"
+	"sync/atomic"
 	"time"
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/guard/detect"
@@ -36,12 +38,15 @@ type Gate struct {
 	log   *slog.Logger
 	now   func() time.Time
 
-	deny   map[string]*regexp.Regexp // by pattern regex
+	deny   atomic.Pointer[map[string]*regexp.Regexp] // by pattern regex
 	tools  toolMemo
 	judger *judge
 	m      *gateMetrics
 	// alarm is told each time a judge gives no verdict; nil tells no one.
 	alarm func(model string, err error)
+	// lastErr is each judge's last failure to give a verdict.
+	errMu   sync.Mutex
+	lastErr map[string]JudgeError
 
 	mu       sync.Mutex
 	attached map[string][]string // "scope/id" → policy names
@@ -59,16 +64,53 @@ func NewGate(cfg Config, store Attachments, log *slog.Logger) (*Gate, error) {
 		log = slog.Default()
 	}
 	g := &Gate{cfg: cfg, store: store, log: log, now: time.Now,
-		deny: map[string]*regexp.Regexp{}, merged: map[string]Policy{}, m: newGateMetrics()}
+		merged: map[string]Policy{}, m: newGateMetrics()}
+	g.deny.Store(compileDeny(cfg))
+	g.reload(context.Background())
+	return g, nil
+}
+
+func compileDeny(cfg Config) *map[string]*regexp.Regexp {
+	deny := map[string]*regexp.Regexp{}
 	for _, p := range cfg.Policies {
 		for _, r := range []Rules{p.Input, p.Output} {
 			for _, d := range r.Deny {
-				g.deny[d.Regex] = regexp.MustCompile(d.Regex)
+				deny[d.Regex] = regexp.MustCompile(d.Regex)
 			}
 		}
 	}
+	return &deny
+}
+
+// Replace puts a new library of policies in force, for requests from the
+// next one on: the cluster's settings changed. Refused, and the old one
+// kept, when it is not valid.
+func (g *Gate) Replace(cfg Config) error {
+	if err := cfg.Validate(); err != nil {
+		return err
+	}
+	// The patterns first: a policy naming one must find it compiled.
+	old := *g.deny.Load()
+	next := *compileDeny(cfg)
+	for k, v := range old {
+		if _, ok := next[k]; !ok {
+			next[k] = v
+		}
+	}
+	g.deny.Store(&next)
+	g.mu.Lock()
+	g.cfg = cfg
+	g.merged = map[string]Policy{}
+	g.mu.Unlock()
 	g.reload(context.Background())
-	return g, nil
+	return nil
+}
+
+// Library is the policies in force.
+func (g *Gate) Library() Config {
+	g.mu.Lock()
+	defer g.mu.Unlock()
+	return g.cfg
 }
 
 // attachmentsTTL is how stale an attachment may be: one made in the console
@@ -92,7 +134,7 @@ func (g *Gate) reload(ctx context.Context) {
 	g.attached = map[string][]string{}
 	for _, a := range list {
 		if _, ok := g.cfg.Policies[a.Policy]; !ok {
-			g.log.Warn("a guard policy is attached that server.yaml does not define; ignored",
+			g.log.Warn("a guard policy is attached that the cluster's settings do not define; ignored",
 				"policy", a.Policy, "scope", a.Scope, "id", a.ScopeID)
 			continue
 		}
@@ -114,6 +156,10 @@ func (g *Gate) Invalidate() { g.reload(context.Background()) }
 // from.
 func (g *Gate) Policy(s Subject) (Policy, string) {
 	g.mu.Lock()
+	if !g.cfg.On() {
+		g.mu.Unlock()
+		return Policy{}, ""
+	}
 	if !g.loading && g.store != nil && g.now().Sub(g.loadedAt) > attachmentsTTL {
 		g.loading = true
 		go g.reload(context.WithoutCancel(context.Background()))
@@ -188,7 +234,7 @@ func (g *Gate) judge(text string, r Rules) ([]finding, []detect.Span) {
 	if len(r.Deny) > 0 {
 		norm := detect.Normalize(text)
 		for _, d := range r.Deny {
-			re := g.deny[d.Regex]
+			re := (*g.deny.Load())[d.Regex]
 			if re == nil {
 				continue
 			}
@@ -286,7 +332,7 @@ func (g *Gate) Input(ctx context.Context, s Subject, body []byte) (Verdict, erro
 	done := func(f finding) Verdict {
 		v.Action, v.Category, v.Rule = f.action, f.category, f.rule
 		if f.action == Block {
-			v.Message = fmt.Sprintf("this request was refused by the gateway's policy (%s)", f.category)
+			v.Message = fmt.Sprintf("this request was refused by the gateway's policy (%s)", refusedFor(f))
 			v.Body = nil
 		}
 		v.Took = g.now().Sub(start)
@@ -410,11 +456,30 @@ func (g *Gate) Input(ctx context.Context, s Subject, body []byte) (Verdict, erro
 func (g *Gate) SetJudgeAlarm(f func(model string, err error)) { g.alarm = f }
 
 func (g *Gate) judgeFailed(model string, err error) {
+	g.errMu.Lock()
+	if g.lastErr == nil {
+		g.lastErr = map[string]JudgeError{}
+	}
+	g.lastErr[model] = JudgeError{Err: err.Error(), At: g.now()}
+	g.errMu.Unlock()
 	if g.alarm != nil {
 		g.alarm(model, err)
 	}
 }
 
+// JudgeError is a judge's last failure to give a verdict.
+type JudgeError struct {
+	Err string
+	At  time.Time
+}
+
+// JudgeErrors are each judge's last failure to give a verdict, by model.
+func (g *Gate) JudgeErrors() map[string]JudgeError {
+	g.errMu.Lock()
+	defer g.errMu.Unlock()
+	return maps.Clone(g.lastErr)
+}
+
 func judgeFinding(p Policy, action string, res judgment, err error) finding {
 	if err != nil {
 		// Unanswered: closed stops it, open lets it through, said.
@@ -463,3 +528,14 @@ func toolAllowed(t Tools, name string) bool {
 	}
 	return false
 }
+
+// refusedFor is what a refusal names: the category, and for a judge the
+// hazard it saw — "judge: S6,Specialized Advice" — so whoever is refused
+// can say why, and a judge refusing ordinary questions can be told apart.
+// Never the text.
+func refusedFor(f finding) string {
+	if f.category == "judge" && f.rule != "" && f.rule != "unavailable" {
+		return "judge: " + f.rule
+	}
+	return f.category
+}

internal/guard/gate_test.go

@@ -278,3 +278,34 @@ func TestCapabilitiesAndStructuredAnswersWhenAskedFor(t *testing.T) {
 		}
 	}
 }
+
+func TestTheGuardrailsTurnOffAndOnAsAWhole(t *testing.T) {
+	off := false
+	cfg := lib(map[string]Policy{"base": {Input: Rules{PII: map[string]string{"secret": "block"}}}}, "base")
+	cfg.Enabled = &off
+	g := gate(t, cfg)
+	secret := chat("user", "my key is sk-proj-abcdefghijklmnopqrstuvwx1234")
+	if v, err := g.Input(context.Background(), Subject{}, secret); err != nil || v.Action != "" {
+		t.Errorf("off, a secret was acted on: %+v %v", v, err)
+	}
+	on := true
+	cfg.Enabled = &on
+	if err := g.Replace(cfg); err != nil {
+		t.Fatal(err)
+	}
+	if v, _ := g.Input(context.Background(), Subject{}, secret); v.Action != Block {
+		t.Errorf("on again, a secret went through: %+v", v)
+	}
+	if !(Config{}).On() {
+		t.Error("unset is not on")
+	}
+}
+
+func TestAJudgesRefusalNamesTheHazard(t *testing.T) {
+	if got := refusedFor(finding{category: "judge", rule: "S6,Specialized Advice"}); got != "judge: S6,Specialized Advice" {
+		t.Errorf("%q", got)
+	}
+	if got := refusedFor(finding{category: "secret", rule: "pii:secret"}); got != "secret" {
+		t.Errorf("%q", got)
+	}
+}

internal/guard/judge.go

@@ -294,6 +294,10 @@ func parseJudgment(format string, names []string, body []byte, answer string) (j
 	}
 	switch format {
 	case "llamaguard3":
+		if len(names) == 0 {
+			// Its own: S1 is Violent Crimes, S9 Indiscriminate Weapons.
+			names = llamaGuard3Categories
+		}
 		lines := strings.Split(text, "\n")
 		switch strings.ToLower(strings.TrimSpace(lines[0])) {
 		case "safe":
@@ -369,3 +373,53 @@ func DefaultCategories(format string) []string {
 	}
 	return nil
 }
+
+// JudgeAnswer is what a judge said of a text: its words as it gave them,
+// and as the gateway reads them.
+type JudgeAnswer struct {
+	Model      string
+	Raw        string
+	Unsafe     bool
+	Categories []string
+	Took       time.Duration
+}
+
+// AskJudge has a judge read a text, past the cache, and says what it
+// answered: for an operator finding out why ordinary requests are refused.
+// With answer, the judge reads it as the assistant's reply to text.
+func (g *Gate) AskJudge(ctx context.Context, model, text, answer string) (JudgeAnswer, error) {
+	out := JudgeAnswer{Model: model}
+	if g.judger == nil {
+		return out, errNoJudge
+	}
+	format, names, ok := g.judger.format(model)
+	if !ok {
+		return out, fmt.Errorf("%w: %q is not a model with role: guard", errNoJudge, model)
+	}
+	ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
+	defer cancel()
+	start := time.Now()
+	path, body := judgeRequest(model, format, names, text, answer)
+	reply, err := g.judger.call(ctx, model, path, body)
+	out.Took = time.Since(start)
+	if err != nil {
+		return out, err
+	}
+	var resp struct {
+		Choices []struct {
+			Message struct {
+				Content string `json:"content"`
+			} `json:"message"`
+			Text string `json:"text"`
+		} `json:"choices"`
+	}
+	if json.Unmarshal(reply, &resp) == nil && len(resp.Choices) > 0 {
+		out.Raw = strings.TrimSpace(resp.Choices[0].Message.Content + resp.Choices[0].Text)
+	}
+	res, err := parseJudgment(format, names, reply, answer)
+	if err != nil {
+		return out, err
+	}
+	out.Unsafe, out.Categories = res.unsafe, res.categories
+	return out, nil
+}

internal/guard/policy.go

@@ -25,10 +25,16 @@ import (
 // Policies are attached to parts of the tenancy tree in the store; Default
 // applies to every request.
 type Config struct {
+	// Enabled turns every policy on or off at once: false checks nothing,
+	// whatever is defined or attached. Unset is on.
+	Enabled  *bool             `koanf:"enabled"`
 	Default  []string          `koanf:"default"`
 	Policies map[string]Policy `koanf:"policies"`
 }
 
+// On is whether the guardrails check anything.
+func (c Config) On() bool { return c.Enabled == nil || *c.Enabled }
+
 // Policy is one named set of rules.
 type Policy struct {
 	// Mode: enforce acts, shadow only records what it would have done, off

internal/onboard/onboard.go

@@ -69,7 +69,11 @@ type Options struct {
 	// NoFirewall leaves the machine's firewall as it is. Otherwise its llmgw
 	// ports — FirewallPorts — answer only the networks it reaches the
 	// control planes on, and FirewallAllow.
-	NoFirewall    bool
+	NoFirewall bool
+	// MirrorBase is where the mirror serves model files, http://host:8081:
+	// the node is asked to reach it before it is added, since a node that
+	// cannot fetches no model.
+	MirrorBase    string
 	FirewallPorts []firewall.Port
 	FirewallAllow []netip.Prefix
 
@@ -461,6 +465,9 @@ func (r *Runner) check(ctx context.Context, res *Result) error {
 		// A missing firewall is what this run is about to write.
 		rep.Findings = slices.DeleteFunc(rep.Findings, func(f doctor.Finding) bool { return f.Area == "firewall" })
 	}
+	if f, ok := r.mirrorReach(ctx); ok {
+		rep.Findings = append(rep.Findings, f)
+	}
 	res.Facts, res.Findings = rep.Facts, rep.Findings
 
 	fails := 0
@@ -502,6 +509,37 @@ func (r *Runner) check(ctx context.Context, res *Result) error {
 	return nil
 }
 
+// mirrorReach asks the node to reach the mirror's files: a base address the
+// node cannot reach (another machine's interface, Docker's bridge, a port
+// closed to it) otherwise shows up as every model failing to download.
+func (r *Runner) mirrorReach(ctx context.Context) (doctor.Finding, bool) {
+	if r.Opts.MirrorBase == "" {
+		return doctor.Finding{}, false
+	}
+	u, err := url.Parse(r.Opts.MirrorBase)
+	if err != nil || u.Hostname() == "" {
+		return doctor.Finding{}, false
+	}
+	port := u.Port()
+	if port == "" {
+		port = "80"
+	}
+	base := strings.TrimRight(r.Opts.MirrorBase, "/") + "/"
+	out, _ := r.Remote.Run(ctx, `if command -v curl >/dev/null 2>&1; then
+  curl -s -o /dev/null -w '%{http_code}' --max-time 5 `+shellQuote(base)+` || true
+else
+  timeout 5 bash -c `+shellQuote("</dev/tcp/"+u.Hostname()+"/"+port)+` 2>/dev/null && echo 200 || echo 000
+fi`, nil)
+	code := strings.TrimSpace(lastLine(out))
+	if code != "" && code != "000" {
+		return doctor.Finding{Level: doctor.OK, Area: "mirror", Message: "reaches the mirror's files at " + base}, true
+	}
+	return doctor.Finding{Level: doctor.Fail, Area: "mirror",
+		Message: "cannot reach the mirror's files at " + base + ": every model it is given would fail to download",
+		Fix: "on the mirror's machine, give llmgw-mirror a --base this node reaches (systemctl cat llmgw-mirror), " +
+			"and let this node through to port " + port}, true
+}
+
 // suggestFabrics groups the active adapters by the fabric they report being
 // on: the subnet manager they see, or the network of their RoCE address.
 // Names are placeholders for the operator: a name is what ties one node's

internal/onboard/onboard_test.go

@@ -32,6 +32,8 @@ type fakeMachine struct {
 	files    map[string][]byte
 	scripts  []string
 	download error
+	// mirror is what curl prints for the mirror's address: 000 unreachable.
+	mirror string
 }
 
 type doctorReport struct {
@@ -64,6 +66,8 @@ func (f *fakeMachine) Run(_ context.Context, script string, stdin []byte) ([]byt
 		return nil, nil
 	case strings.Contains(script, "systemctl restart llmgw-agent"):
 		return []byte("active\n"), nil
+	case strings.Contains(script, "curl -s -o /dev/null -w"):
+		return []byte(f.mirror), nil
 	}
 	return nil, nil
 }
@@ -389,3 +393,30 @@ func TestTheSudoGrantRemovesItselfWithinTheHour(t *testing.T) {
 		t.Errorf("not shell: %v %s\n%s", err, out, inner)
 	}
 }
+
+func TestANodeThatCannotReachTheMirrorIsNotAdded(t *testing.T) {
+	m := newFake()
+	m.mirror = "000"
+	r, events := runner(t, m, Options{Addr: "10.0.0.9", MirrorBase: "http://172.17.0.1:8081"})
+	var res Result
+	err := r.check(t.Context(), &res)
+	if !errors.Is(err, ErrChecks) {
+		t.Fatalf("a node that cannot reach the mirror was let through: %v", err)
+	}
+	found := false
+	for _, e := range *events {
+		if e.Level == Fail && strings.Contains(e.Detail, "http://172.17.0.1:8081/") && strings.Contains(e.Fix, "--base") {
+			found = true
+		}
+	}
+	if !found {
+		t.Errorf("no failure naming the address and the fix: %+v", *events)
+	}
+
+	ok := newFake()
+	ok.mirror = "404"
+	r, _ = runner(t, ok, Options{Addr: "10.0.0.9", MirrorBase: "http://10.0.0.146:8081"})
+	if err := r.check(t.Context(), &res); err != nil {
+		t.Errorf("a node that reaches the mirror: %v", err)
+	}
+}

internal/router/guard.go

@@ -191,7 +191,7 @@ func guardFields(u *store.Usage, in, out guard.Verdict) {
 	if out.Stricter(in) {
 		v = out
 	}
-	u.GuardAction, u.GuardPolicy, u.GuardCategory = string(v.Action), v.Policy, v.Category
+	u.GuardAction, u.GuardPolicy, u.GuardCategory, u.GuardRule = string(v.Action), v.Policy, v.Category, v.Rule
 	if v.Shadow && v.Action != guard.Allow {
 		u.GuardAction = "shadow-" + u.GuardAction
 	}

internal/router/guard_test.go

@@ -34,7 +34,7 @@ func (g *wordGuard) Input(_ context.Context, s guard.Subject, body []byte) (guar
 	case g.err != nil:
 		return guard.Verdict{}, g.err
 	case bytes.Contains(body, []byte("forbidden")):
-		return guard.Verdict{Action: guard.Block, Policy: "words", Category: "test"}, nil
+		return guard.Verdict{Action: guard.Block, Policy: "words", Category: "test", Rule: "S6,Specialized Advice"}, nil
 	case bytes.Contains(body, []byte("alice@example.com")):
 		return guard.Verdict{Action: guard.Redact, Policy: "pii", Category: "email",
 			Body: bytes.ReplaceAll(body, []byte("alice@example.com"), []byte("[EMAIL_1]"))}, nil
@@ -157,6 +157,7 @@ func TestABlockedRequestIsRecordedWithItsVerdictAndCostsNothing(t *testing.T) {
 		}
 		if ok {
 			if u.GuardAction != "blocked" || u.GuardPolicy != "words" || u.GuardCategory != "test" ||
+				u.GuardRule != "S6,Specialized Advice" ||
 				u.CostMicros != 0 || u.PromptTokens != 0 || u.Status != http.StatusBadRequest {
 				t.Errorf("recorded %+v", u)
 			}

internal/router/handlers.go

@@ -208,6 +208,10 @@ func (r *Router) Handler() http.Handler {
 		// So the example on the profile page names a model that is actually
 		// running, rather than a placeholder that gets sent as-is.
 		pages.ServedModels(r.exampleModels)
+		pages.SetModelFacts(func(name string) web.ModelFacts {
+			_, m := r.resolve(name)
+			return web.ModelFacts{ContextLength: m.ContextLength, MaxOutput: m.MaxOutput, Tools: m.Capabilities.Tools}
+		})
 		pages.SetPersonalOff(r.personalOff)
 		pages.SetLlamaCppVersion(r.llamaVersion)
 		pages.SetPersonalDefaults(r.personalDefaults)
@@ -235,7 +239,7 @@ func (r *Router) Handler() http.Handler {
 				return names, p.NoStore && p.Mode == "enforce"
 			})
 		}
-		if r.store != nil && len(r.guardLibrary.Policies) > 0 {
+		if r.store != nil && r.guardLibrary != nil {
 			pages.RegisterAdminGuard(mux, r.store, r.guardLibrary)
 		}
 		pages.Register(mux)
@@ -1213,9 +1217,11 @@ func (r *Router) accessLog(req *parsedRequest, res dispatchResult, status int, t
 	}
 	// What a guard did, by name: never what it matched.
 	if v := req.guardIn; res.guard.Stricter(v) {
-		attrs = append(attrs, "guard", string(res.guard.Action), "guard_stage", "output", "guard_policy", res.guard.Policy)
+		attrs = append(attrs, "guard", string(res.guard.Action), "guard_stage", "output", "guard_policy", res.guard.Policy,
+			"guard_category", res.guard.Category, "guard_rule", res.guard.Rule)
 	} else if v.Action != "" {
-		attrs = append(attrs, "guard", string(v.Action), "guard_stage", "input", "guard_policy", v.Policy)
+		attrs = append(attrs, "guard", string(v.Action), "guard_stage", "input", "guard_policy", v.Policy,
+			"guard_category", v.Category, "guard_rule", v.Rule)
 	}
 	if res.outcome.Stopped {
 		attrs = append(attrs, "stopped_by_guard", true)

internal/router/router.go

@@ -108,7 +108,7 @@ type Router struct {
 	// guard judges requests and answers; nil judges nothing. guardLibrary
 	// is the policies it was built from, for the pages that show them.
 	guard        guard.Guard
-	guardLibrary guard.Config
+	guardLibrary func() guard.Config
 	// oidc is sign-in through an identity provider, and scim provisioning
 	// by it.
 	oidc config.OIDC
@@ -187,7 +187,7 @@ func (r *Router) SetMirror(m web.MirrorAdmin) { r.mirror = m }
 func (r *Router) SetGuard(g guard.Guard) { r.guard = g }
 
 // SetGuardLibrary is the policies the guard was built from, for the pages.
-func (r *Router) SetGuardLibrary(c guard.Config) { r.guardLibrary = c }
+func (r *Router) SetGuardLibrary(c func() guard.Config) { r.guardLibrary = c }
 
 // SetRegistry gives the mirror page somewhere to look models up.
 func (r *Router) SetRegistry(reg web.Registry) { r.registry = reg }

internal/settings/settings.go

@@ -126,15 +126,61 @@ type Doc struct {
 	doc  yaml.Node
 	// Changed counts edits not yet saved.
 	Changed int
+
+	// cluster holds the keys that are not the machine's own, when the
+	// cluster keeps them elsewhere (the store); isLocal says which are.
+	cluster *Doc
+	isLocal func([]string) bool
+}
+
+// Layer has the keys isLocal does not claim read and written in cluster
+// instead of this document: one screen over a machine's file and the
+// cluster's settings.
+func (d *Doc) Layer(cluster *Doc, isLocal func([]string) bool) {
+	d.cluster, d.isLocal = cluster, isLocal
+}
+
+// Cluster is the layered cluster document, nil when there is none.
+func (d *Doc) Cluster() *Doc { return d.cluster }
+
+// Pending counts edits not yet saved, in this document and its layer.
+func (d *Doc) Pending() int {
+	if d.cluster != nil {
+		return d.Changed + d.cluster.Changed
+	}
+	return d.Changed
+}
+
+// at is the document a path is kept in.
+func (d *Doc) at(path []string) *Doc {
+	if d.cluster != nil && len(path) > 0 && !d.isLocal(path) {
+		return d.cluster
+	}
+	return d
 }
 
 // Open reads a file; a missing one is an empty document.
 func Open(path string) (*Doc, error) {
-	d := &Doc{Path: path}
 	b, err := os.ReadFile(path)
 	if err != nil && !os.IsNotExist(err) {
 		return nil, err
 	}
+	return parseDoc(path, b)
+}
+
+// FromBytes reads a document held somewhere other than a file: the
+// cluster's settings, kept in the store. It has no Path, so it is not Saved.
+func FromBytes(name string, b []byte) (*Doc, error) {
+	d, err := parseDoc(name, b)
+	if err != nil {
+		return nil, err
+	}
+	d.Path = ""
+	return d, nil
+}
+
+func parseDoc(path string, b []byte) (*Doc, error) {
+	d := &Doc{Path: path}
 	if len(bytes.TrimSpace(b)) > 0 {
 		if err := yaml.Unmarshal(b, &d.doc); err != nil {
 			return nil, fmt.Errorf("%s: %w", path, err)
@@ -163,6 +209,9 @@ func (d *Doc) Bytes() ([]byte, error) {
 
 // Node is the node at a path of keys and list indexes, or nil.
 func (d *Doc) Node(path []string) *yaml.Node {
+	if t := d.at(path); t != d {
+		return t.Node(path)
+	}
 	n := d.root
 	for _, p := range path {
 		n = child(n, p)
@@ -209,6 +258,9 @@ func (d *Doc) ensure(path []string) *yaml.Node {
 // Set writes one key's value: a scalar, a list of lines or a map of
 // "key = value" lines, by its kind. It checks the value first.
 func (d *Doc) Set(path []string, k Kind, value string) error {
+	if t := d.at(path); t != d {
+		return t.Set(path, k, value)
+	}
 	if len(path) == 0 {
 		return fmt.Errorf("no key")
 	}
@@ -216,6 +268,42 @@ func (d *Doc) Set(path []string, k Kind, value string) error {
 	if err != nil {
 		return err
 	}
+	return d.place(path, n)
+}
+
+// SetYAML writes one key's value given as YAML: a scalar, [a, b], {k: v},
+// or a whole block.
+func (d *Doc) SetYAML(path []string, value string) error {
+	if t := d.at(path); t != d {
+		return t.SetYAML(path, value)
+	}
+	if len(path) == 0 {
+		return fmt.Errorf("no key")
+	}
+	var doc yaml.Node
+	if err := yaml.Unmarshal([]byte(value), &doc); err != nil {
+		return fmt.Errorf("%q is not YAML: %w", value, err)
+	}
+	n := &yaml.Node{Kind: yaml.ScalarNode, Tag: "!!null", Value: ""}
+	if len(doc.Content) > 0 {
+		n = doc.Content[0]
+	}
+	return d.place(path, n)
+}
+
+// SetNode writes a node, as read from another document, at path.
+func (d *Doc) SetNode(path []string, n *yaml.Node) error {
+	if t := d.at(path); t != d {
+		return t.SetNode(path, n)
+	}
+	if len(path) == 0 {
+		return fmt.Errorf("no key")
+	}
+	c := *n
+	return d.place(path, &c)
+}
+
+func (d *Doc) place(path []string, n *yaml.Node) error {
 	parent := d.ensure(path[:len(path)-1])
 	if parent == nil {
 		return fmt.Errorf("%s is not a block", strings.Join(path[:len(path)-1], "."))
@@ -237,6 +325,10 @@ func (d *Doc) Set(path []string, k Kind, value string) error {
 
 // Unset removes a key, so its default applies.
 func (d *Doc) Unset(path []string) {
+	if t := d.at(path); t != d {
+		t.Unset(path)
+		return
+	}
 	if len(path) == 0 {
 		return
 	}
@@ -264,6 +356,9 @@ func (d *Doc) Unset(path []string) {
 
 // Append adds an empty block to the list at path, and returns its index.
 func (d *Doc) Append(path []string) int {
+	if t := d.at(path); t != d {
+		return t.Append(path)
+	}
 	parent := d.ensure(path[:len(path)-1])
 	key := path[len(path)-1]
 	seq := child(parent, key)
@@ -283,6 +378,9 @@ func (d *Doc) Append(path []string) int {
 
 // AddEntry adds an empty block under a name to the map at path.
 func (d *Doc) AddEntry(path []string, name string) error {
+	if t := d.at(path); t != d {
+		return t.AddEntry(path, name)
+	}
 	m := d.ensure(path)
 	if m == nil || m.Kind != yaml.MappingNode {
 		return fmt.Errorf("%s is not a map", strings.Join(path, "."))

internal/store/copy.go

@@ -58,6 +58,8 @@ var copyOrder = []string{
 	// Or a removed node's certificate would work again in the new store.
 	"revoked_nodes",
 	"guard_attachments",
+	// Or the cluster would start the new store on server.yaml's settings again.
+	"cluster_settings",
 	// Who changed what is not lost by moving the store.
 	"audit_events",
 	"key_ttl",

internal/store/guard.go

@@ -50,6 +50,7 @@ type GuardEvent struct {
 	Action    string
 	Policy    string
 	Category  string
+	Rule      string
 	GuardMs   int64
 }
 
@@ -57,14 +58,14 @@ type GuardEvent struct {
 // flagged, redacted, blocked, or would have in shadow — since a moment.
 func (s *DB) GuardEvents(ctx context.Context, since time.Time, limit int) ([]GuardEvent, error) {
 	return queryAll(ctx, s.db, `SELECT ts, request_id, key_id, COALESCE(user_id, ''), project_id, model, status,
-		guard_action, guard_policy, guard_category, guard_ms
+		guard_action, guard_policy, guard_category, guard_rule, guard_ms
 		FROM usage WHERE ts >= ? AND guard_action <> '' ORDER BY ts DESC LIMIT ?`,
 		[]any{millis(since), limit},
 		func(rows *sql.Rows) (GuardEvent, error) {
 			var e GuardEvent
 			var ts int64
 			err := rows.Scan(&ts, &e.RequestID, &e.KeyID, &e.UserID, &e.ProjectID, &e.Model, &e.Status,
-				&e.Action, &e.Policy, &e.Category, &e.GuardMs)
+				&e.Action, &e.Policy, &e.Category, &e.Rule, &e.GuardMs)
 			e.At = fromMillis(ts)
 			return e, err
 		})

internal/store/migrations/postgres/0032_cluster_settings.sql

@@ -0,0 +1,10 @@
+-- Ported from the SQLite migration of the same number.
+-- The cluster's settings: everything in server.yaml that is not one
+-- machine's own, as a versioned YAML document every control plane follows.
+CREATE TABLE cluster_settings (
+  version    BIGINT PRIMARY KEY,
+  body       TEXT NOT NULL,
+  note       TEXT NOT NULL DEFAULT '',
+  created_at BIGINT NOT NULL,
+  created_by TEXT NOT NULL DEFAULT ''
+);

internal/store/migrations/postgres/0033_usage_guard_rule.sql

@@ -0,0 +1,2 @@
+-- Ported from the SQLite migration of the same number.
+ALTER TABLE usage ADD COLUMN guard_rule TEXT NOT NULL DEFAULT '';

internal/store/migrations/sqlite/0032_cluster_settings.sql

@@ -0,0 +1,13 @@
+-- The cluster's settings: everything in server.yaml that is not one
+-- machine's own (listeners, certificates, the store's address), as a YAML
+-- document every control plane reads and follows. Each change is a new row,
+-- so what was in force before is there to compare with and go back to, and
+-- two edits made at once cannot both win: the second is refused for naming
+-- a version that is no longer the newest.
+CREATE TABLE cluster_settings (
+  version    INTEGER PRIMARY KEY,
+  body       TEXT NOT NULL,
+  note       TEXT NOT NULL DEFAULT '',
+  created_at INTEGER NOT NULL,
+  created_by TEXT NOT NULL DEFAULT ''
+);

internal/store/migrations/sqlite/0033_usage_guard_rule.sql

@@ -0,0 +1,5 @@
+-- What in a guard policy decided a request, beside its category: the
+-- pattern's name, or the judge's verdict — "S6,Specialized Advice" — so a
+-- request a judge refused says which hazard it saw, and a judge that
+-- refuses ordinary questions can be told from one that is right to.
+ALTER TABLE usage ADD COLUMN guard_rule TEXT NOT NULL DEFAULT '';

internal/store/settings.go

@@ -0,0 +1,98 @@
+package store
+
+import (
+	"context"
+	"database/sql"
+	"errors"
+	"time"
+)
+
+// ErrSettingsChanged is a change made against a version of the cluster's
+// settings that is no longer the newest: somebody else changed them first.
+var ErrSettingsChanged = errors.New("the cluster's settings were changed since they were read; read them again")
+
+// SettingsVersion is one version of the cluster's settings.
+type SettingsVersion struct {
+	Version int64
+	Body    string
+	Note    string
+	At      time.Time
+	By      string
+}
+
+// ClusterSettings is the newest version, Version 0 when there is none yet.
+func (s *DB) ClusterSettings(ctx context.Context) (SettingsVersion, error) {
+	return s.settingsWhere(ctx, `version = (SELECT MAX(version) FROM cluster_settings)`)
+}
+
+// ClusterSettingsAt is one version, Version 0 when there is no such version.
+func (s *DB) ClusterSettingsAt(ctx context.Context, version int64) (SettingsVersion, error) {
+	return s.settingsWhere(ctx, `version = ?`, version)
+}
+
+func (s *DB) settingsWhere(ctx context.Context, where string, args ...any) (SettingsVersion, error) {
+	var v SettingsVersion
+	var at int64
+	err := s.db.QueryRowContext(ctx,
+		`SELECT version, body, note, created_at, created_by FROM cluster_settings WHERE `+where, args...).
+		Scan(&v.Version, &v.Body, &v.Note, &at, &v.By)
+	if errors.Is(err, sql.ErrNoRows) {
+		return SettingsVersion{}, nil
+	}
+	v.At = time.UnixMilli(at)
+	return v, err
+}
+
+// ClusterSettingsVersion is the newest version's number, 0 when there is
+// none: what every control plane asks every few seconds.
+func (s *DB) ClusterSettingsVersion(ctx context.Context) (int64, error) {
+	var v sql.NullInt64
+	err := s.db.QueryRowContext(ctx, `SELECT MAX(version) FROM cluster_settings`).Scan(&v)
+	return v.Int64, err
+}
+
+// PutClusterSettings records a new version made from base, the version the
+// change was read from, and returns its number. Refused with
+// ErrSettingsChanged when base is not the newest.
+func (s *DB) PutClusterSettings(ctx context.Context, base int64, body, by, note string) (int64, error) {
+	cur, err := s.ClusterSettingsVersion(ctx)
+	if err != nil {
+		return 0, err
+	}
+	if cur != base {
+		return 0, ErrSettingsChanged
+	}
+	// The key settles a race the read above cannot: of two writers from the
+	// same base, one inserts base+1 and the other is refused.
+	_, err = s.db.ExecContext(ctx,
+		`INSERT INTO cluster_settings (version, body, note, created_at, created_by) VALUES (?, ?, ?, ?, ?)`,
+		base+1, body, note, time.Now().UnixMilli(), by)
+	if isUniqueViolation(err) {
+		return 0, ErrSettingsChanged
+	}
+	if err != nil {
+		return 0, err
+	}
+	return base + 1, nil
+}
+
+// ClusterSettingsHistory is the newest versions first, at most limit.
+func (s *DB) ClusterSettingsHistory(ctx context.Context, limit int) ([]SettingsVersion, error) {
+	rows, err := s.db.QueryContext(ctx,
+		`SELECT version, body, note, created_at, created_by FROM cluster_settings ORDER BY version DESC LIMIT ?`, limit)
+	if err != nil {
+		return nil, err
+	}
+	defer rows.Close()
+	var out []SettingsVersion
+	for rows.Next() {
+		var v SettingsVersion
+		var at int64
+		if err := rows.Scan(&v.Version, &v.Body, &v.Note, &at, &v.By); err != nil {
+			return nil, err
+		}
+		v.At = time.UnixMilli(at)
+		out = append(out, v)
+	}
+	return out, rows.Err()
+}

internal/store/settings_test.go

@@ -0,0 +1,39 @@
+package store
+
+import (
+	"errors"
+	"testing"
+)
+
+func TestSettingsAreVersionedAndTwoEditsCannotBothWin(t *testing.T) {
+	s := capacityStore(t)
+	ctx := t.Context()
+	if v, err := s.ClusterSettings(ctx); err != nil || v.Version != 0 {
+		t.Fatalf("empty store: %+v %v", v, err)
+	}
+	v1, err := s.PutClusterSettings(ctx, 0, "router: {}\n", "import", "from server.yaml")
+	if err != nil || v1 != 1 {
+		t.Fatalf("first: %d %v", v1, err)
+	}
+	if _, err := s.PutClusterSettings(ctx, 0, "x: 1\n", "late", ""); !errors.Is(err, ErrSettingsChanged) {
+		t.Fatalf("a change from an old version: %v", err)
+	}
+	if _, err := s.PutClusterSettings(ctx, 1, "guard: {}\n", "ilker", "guard"); err != nil {
+		t.Fatal(err)
+	}
+	cur, _ := s.ClusterSettings(ctx)
+	if cur.Version != 2 || cur.Body != "guard: {}\n" || cur.By != "ilker" {
+		t.Errorf("newest: %+v", cur)
+	}
+	if n, _ := s.ClusterSettingsVersion(ctx); n != 2 {
+		t.Errorf("version %d", n)
+	}
+	old, _ := s.ClusterSettingsAt(ctx, 1)
+	if old.Body != "router: {}\n" || old.Note != "from server.yaml" {
+		t.Errorf("version 1: %+v", old)
+	}
+	h, _ := s.ClusterSettingsHistory(ctx, 10)
+	if len(h) != 2 || h[0].Version != 2 {
+		t.Errorf("history: %+v", h)
+	}
+}