From 6289b01ed615172a32fde5a5c40a0fc72101161c Mon Sep 17 00:00:00 2001 From: Ilker Manap Date: Mon, 5 Oct 2026 15:25:23 +0200 Subject: [PATCH] The cluster's settings live in the store, and every change takes effect Everything in server.yaml that is not one machine's own is now a versioned document in the store (cluster_settings), the same on every control plane. The first start against a store imports the file's cluster keys, with the baseline guard policy when there is none; after that those keys in a file are ignored, with a warning. Each control plane polls the document's version and its own server.yaml's hash. A change is validated and applied: guard live (Gate.Replace), anything else by a graceful restart that systemd brings back, standbys first and the leader 30s later. A version that does not load on a machine is not applied; at start the newest one that loads is used, so a bad edit cannot keep every control plane down. llmgw ctl config show/get/set/unset/edit/history/rollback, the admin API under /api/v1/settings, and the console's Settings screen, which layers the store's document over the machine's file, write it. Commands beside the control plane that read cluster keys (init, node add, server add, backup, upgrade) read the layered configuration; init --bundle writes offline to the store once it holds the settings. A standby that becomes the leader re-reads models, engine choices and facility holds from the store. Retires appendGuardIfMissing: the import does it. Co-Authored-By: Claude Opus 5.5 (1M context) --- BACKLOG.md | 50 +++ CHANGELOG.md | 37 +- docs/admin-api.openapi.json | 222 ++++++++++- docs/console.md | 22 +- docs/guardrails.md | 10 +- docs/install.md | 34 +- internal/cli/backup.go | 2 +- internal/cli/bundle.go | 2 +- internal/cli/clustersettings.go | 361 +++++++++++++++++ internal/cli/clustersettings_test.go | 139 +++++++ internal/cli/ctl.go | 1 + internal/cli/ctl_config.go | 377 ++++++++++++++++++ internal/cli/node.go | 6 +- internal/cli/server.go | 84 +++- internal/cli/serveradd.go | 2 +- internal/cli/serverinit.go | 16 +- internal/cli/serverinit_bundle.go | 23 +- internal/cli/serverinit_cluster.go | 38 -- internal/cli/serverinit_cluster_test.go | 37 -- internal/cli/serverinit_firewall.go | 2 +- internal/cli/serverinit_test.go | 6 +- internal/cli/tui_console.go | 106 +++-- internal/cli/tui_hf.go | 3 +- internal/cli/tui_people.go | 3 +- internal/cli/tui_setup_test.go | 58 ++- internal/config/config.go | 28 +- internal/config/layer.go | 256 ++++++++++++ internal/config/layer_test.go | 99 +++++ internal/control/adminapi/api.go | 4 +- internal/control/adminapi/guard.go | 21 +- internal/control/adminapi/guard_test.go | 11 +- internal/control/adminapi/notify.go | 2 +- internal/control/adminapi/routes.go | 15 +- internal/control/adminapi/settings.go | 179 +++++++++ internal/control/adminapi/settings_test.go | 78 ++++ internal/control/enginechoice.go | 33 ++ internal/control/facility.go | 17 + internal/control/leadership.go | 24 ++ internal/control/leadership_test.go | 26 ++ internal/control/modelsvc/modelsvc.go | 6 +- internal/control/modelsvc/modelsvc_test.go | 6 +- internal/guard/gate.go | 50 ++- internal/router/handlers.go | 2 +- internal/router/router.go | 4 +- internal/settings/settings.go | 100 ++++- internal/store/copy.go | 2 + .../postgres/0032_cluster_settings.sql | 10 + .../sqlite/0032_cluster_settings.sql | 13 + internal/store/settings.go | 98 +++++ internal/store/settings_test.go | 39 ++ internal/web/admin_guard.go | 7 +- internal/web/admin_mirror_test.go | 5 +- internal/web/i18n_people_test.go | 7 +- internal/web/locales/en/guard.json | 4 +- internal/web/locales/en/mirror.json | 6 +- internal/web/locales/tr/guard.json | 4 +- internal/web/locales/tr/mirror.json | 6 +- 57 files changed, 2579 insertions(+), 224 deletions(-) create mode 100644 internal/cli/clustersettings.go create mode 100644 internal/cli/clustersettings_test.go create mode 100644 internal/cli/ctl_config.go create mode 100644 internal/config/layer.go create mode 100644 internal/config/layer_test.go create mode 100644 internal/control/adminapi/settings.go create mode 100644 internal/control/adminapi/settings_test.go create mode 100644 internal/store/migrations/postgres/0032_cluster_settings.sql create mode 100644 internal/store/migrations/sqlite/0032_cluster_settings.sql create mode 100644 internal/store/settings.go create mode 100644 internal/store/settings_test.go diff --git a/BACKLOG.md b/BACKLOG.md index 4be277b..70a092c 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -761,6 +761,56 @@ portal is not done — the page shows; the console and ctl change. --- +## E. Every setting takes effect when it is changed (2026-10-05) + +Asked for after a site visit: an upgrade left server.yaml without guardrails, +a node on another subnet stayed lost until firewall_networks was edited by +hand, and nothing in server.yaml, agent.yaml or a generated unit applies until +somebody restarts the right service on every machine. The rule from here on: +**a file holds only what one machine needs to start; everything else is the +cluster's, kept in the store, changed through ctl, the console or the web, +audited, and in force on every control plane within seconds.** + +### E1. Cluster settings in the store (L) — in progress +- Table `cluster_settings` (versioned YAML document, who and why). The + cluster-wide part of server.yaml lives there; server.yaml keeps the + per-machine keys only (listeners, TLS paths, store DSN, secrets, tracing, + monitoring, mirror.api/tls, backup_dir/key, endpoints). +- First start against a store with no document imports the file's cluster + keys (and the baseline guard policy when there is none); after that those + keys in a file are ignored, with a warning. +- Each control plane polls the document's version and its own server.yaml's + hash. A change is validated, then applied: `guard` live, the rest by a + graceful restart of the process (standbys first, the leader last), which + systemd brings back. No "restart now?" anywhere. +- `llmgw ctl config show|history|get|set|unset|edit|rollback`; the console's + settings screen edits the store for cluster keys; admin API + `GET/PUT /api/v1/settings`, `/history`, `/rollback`. +- A standby that becomes leader re-reads the catalog, engine choices and + facility state from the store (it kept what it read at start). +- Retires `appendGuardIfMissing` (server init): the import does it. + +### E2. Generated files from settings (L) +Every machine runs the agent (root) with roles — node, control plane, mirror, +store member — and the agent renders and applies what was written once by +init: the llmgw nft table, the mirror's unit flags, keepalived, etcd and +Patroni configuration. The firewall's allowed set is the cluster's membership +(every node, control plane, mirror and store member) plus +`firewall.extra` from the store; 7443 stays open (mTLS). A node added or +removed changes every machine's rules. Retires `firewall.Admit` and +`firewall_networks` in server.yaml (imported into the store). + +### E3. agent.yaml down to bootstrap (M) +agent.yaml keeps the control planes' addresses, its certificate and its id. +Labels, fabrics, engines, heartbeat and supervisor settings move to the +store per node and reach the agent over its stream; a change to the file is +noticed and the agent restarts itself. + +### E4. Web settings page (S) +The same document, edited on /admin/settings, for whoever has no terminal. + +--- + ## D. Facts worth knowing before starting - Admin API: `127.0.0.1:9090`, no auth, `ctlClient` in `internal/cli/ctl.go`. diff --git a/CHANGELOG.md b/CHANGELOG.md index 3069acf..7957de7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -23,8 +23,29 @@ APIs change between minors without a deprecation period. ## Unreleased +### Changed + +- **The cluster's settings live in the store, and every change is in force + within seconds.** Everything in server.yaml that is not one machine's own + (guardrails, limits and prices, routing, engines, notifications, the + mirror's policy, declared models, offline, federation, facility budgets) is + a versioned document in the store, the same on every control plane. + `llmgw ctl config` shows, sets, edits, lists and rolls it back, and the + console's Settings screen edits it. A change is checked, kept with who made + it and why, and applied: guardrails live, anything else by every control + plane starting again on it, standbys first and the leader last. A change to + a machine's own keys in its server.yaml is noticed and applied the same way. + No "restart now?" any more, and no second copy to keep in step. The first + start against a store brings in the file's cluster keys, with the baseline + guard policy when the file had none, so an upgraded cluster gets its + guardrails; after that, those keys in a server.yaml are ignored, with a + warning. + ### Fixed +- **A standby that becomes the leader reads the store again**: models added + or removed, engine builds chosen and domains held through the leader before + it, rather than placing on what it read when it started. - **A node on another network joins**: `node add` set up the node's firewall but left the control plane's to the networks it was installed with, so a node on another subnet was installed and then never heard from (lost). It now lets @@ -32,22 +53,6 @@ APIs change between minors without a deprecation period. when a router is between them — and says so; a network already let in changes nothing. -- **An upgraded control plane gets the guardrails**: `ctl upgrade` replaces the - binary but leaves server.yaml as it is, and one written before guardrails has no - `guard:` section, so nothing was checked. `llmgw server init`, run again on such - a machine, now adds the baseline policy (and fetches the judge model), keeping - the old file beside it as `server.yaml.bak-