v0.14.1: changelog and README

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

BACKLOG.md

@@ -569,9 +569,12 @@ portal is not done — the page shows; the console and ctl change.
     client tests against a fake admin API and the OpenAPI document;
     acceptance tests with Terraform and OpenTofu against the fake; one
     apply, empty plan, in-place change and destroy against a real `llmgw
-    server` run by hand. Left: organizations, teams, projects, people and
-    keys are not in the admin API (they are `llmgw org`/`users`/`keys`
-    against the store), so not in the provider; a model's definition beyond
+    server` run by hand. Organizations, teams, projects, keys and
+    memberships are in the admin API since 2026-10-08 (`/api/v1/orgs`,
+    `teams`, `projects`, `keys`, `users/{id}/memberships`) and in the
+    provider (`llmgw_org`, `llmgw_team`, `llmgw_project`, `llmgw_key`,
+    `llmgw_membership`); accounts are still made by sign-up, invitation or
+    OIDC/SCIM. Left: a model's definition beyond
     scaling, class, owner and adapters is changed by replacing it (the API
     has no edit, and `resources` is not used yet); `canary`, `slo`,
     `scaling.schedule` and `prewarm`, `guard` role and `origin` are not in
@@ -793,10 +796,19 @@ leader 30s later with /health answering throughout; rollback did the same.
   facility state from the store (it kept what it read at start).
 - Retires `appendGuardIfMissing` (server init): the import does it.
 
-### E2. Generated files from settings (L) — firewall on/off done 2026-10-06
+### E2. Generated files from settings (L) — firewall on/off done 2026-10-06, every machine 2026-10-09
 `firewall.enabled` in the cluster's settings reaches nodes over the agent
-stream and control planes through llmgw-firewall-sync.path. Left: the mirror
-machine and store members, the allowed set from membership, the other files.
+stream and control planes through llmgw-firewall-sync.path. Since
+2026-10-09 the mirror machine too (the leader PUTs /v1/firewall, the mirror
+writes `.firewall.json`, the same path unit applies it) and store members
+that run no llmgw (the leader keeps it in `cluster_state`, a root timer
+reads the member's own copy with psql); `firewall_networks` reach all of
+them, and the mirror lets in the network of every node and control plane
+the leader knows. Tried on the PoC: off and on, and a network added, on
+control planes, mirror, store member and nodes. Left: the allowed set
+replaced by membership rather than added to (a network taken out of the
+settings stays let in), the other files (the mirror's unit flags,
+keepalived, etcd and Patroni configuration).
 
 Every machine runs the agent (root) with roles — node, control plane, mirror,
 store member — and the agent renders and applies what was written once by
@@ -807,14 +819,34 @@ Patroni configuration. The firewall's allowed set is the cluster's membership
 removed changes every machine's rules. Retires `firewall.Admit` and
 `firewall_networks` in server.yaml (imported into the store).
 
-### E3. agent.yaml down to bootstrap (M)
+### E3. agent.yaml down to bootstrap (M) — done 2026-10-08 (branch `settings-e`)
 agent.yaml keeps the control planes' addresses, its certificate and its id.
 Labels, fabrics, engines, heartbeat and supervisor settings move to the
 store per node and reach the agent over its stream; a change to the file is
 noticed and the agent restarts itself.
+Built as a section of the cluster's settings rather than a store of its own,
+so history, audit, `ctl config`, the console and the portal page cover it:
+`agents.default` and `agents.nodes.<id>` (config.AgentSettings; bootstrap
+keys node_id/server(s)/hostname/mgmt_addr/tls refused; each node's merge
+loaded as an agent would, so a bad fabric is refused on save). A live key:
+the control plane sends ControlMessage.AgentSettings on register and, a
+second apart per node, on change; the agent compares with what it started on,
+checks it loads beside its agent.yaml (else logs and keeps running), keeps it
+in /var/lib/llmgw-agent/agent-settings.yaml and restarts (engines adopted).
+At start: agent.yaml < kept settings < LLMGW_ env; kept settings that no
+longer load are set aside, never a reason not to start. agent.yaml is
+watched every 10 s and restarted on when it changes and loads. Left: file
+keys are not imported into the settings (they keep working under them);
+`node add` still writes a full agent.yaml; labels have two homes now
+(`ctl label`, which wins, and agents.*.labels).
 
-### E4. Web settings page (S)
+### E4. Web settings page (S) — done 2026-10-08 (branch `settings-e`)
 The same document, edited on /admin/settings, for whoever has no terminal.
+Built: administrators only (staff roles get 404: webhook secrets), checked by
+the same function as PUT /api/v1/settings, made from a base version or
+refused on a newer one (the edit kept in the editor), history with view and
+roll back as a new version, audit with the size not the body, English and
+Turkish, a Settings link in the menu.
 
 ---
 
@@ -843,6 +875,12 @@ Built (`internal/store/knowledge.go`, migration 0035 PostgreSQL only):
   partial HNSW index per collection on its organization's partition
   (`embedding::vector(dims)`, `WHERE collection = '<id>'`). EXPLAIN on
   20k rows: partition pruned, that index used, policy applied.
+  Later (scale, 2026-10-08): a partial GIN per collection too
+  (`kb_tsv_<id>`, backfilled at open); `vec_type` per collection, `halfvec`
+  on pgvector ≥ 0.7 (up to 4000 dims); `SET LOCAL hnsw.ef_search` to
+  candidates(k) clamped 40..1000 (EXPLAIN: 40 rows without it, all with
+  it); the text arm takes candidates by `ts_rank` in a subquery and scores
+  only those.
 - RLS FORCEd on `kb_chunks` against `llmgw.kb_org`/`llmgw.kb_collections`
   (set with set_config per transaction), one role — the table owner — not
   a separate one. Chunks are deleted by the foreign-key cascade from their
@@ -985,7 +1023,7 @@ a request without the header, `none` turns it off, an explicit header with
 k=2 gives two sources and no more, llmgw-kb lists owners by path. Left: returned passages
 are not audited per request (the header and the kept request record them).
 
-### F4. Adapter training (L, after F1–F3)
+### F4. Adapter training (L, after F1–F3) — done 2026-10-08 (branch `adapters-f4`)
 A run-to-completion job placed by the control plane like a replica and
 supervised by the agent like an engine, in a Python environment the agent
 installs (TRL + PEFT, QLoRA). Data: Q/A pairs generated from a collection by
@@ -993,6 +1031,37 @@ a cluster model through `/v1/batches`, reviewed by people. Output to the
 mirror, into the model's `adapters`, through canary and a quality suite.
 Runs when the scaling schedule frees a GPU; 8–14B dense bases on one card.
 Never train on a collection some users of the adapter may not read.
+Tried 2026-10-08 on the PoC, by hand: Qwen2.5-0.5B-Instruct (fp16) with a
+LoRA (r=16, every linear layer, 8.8M trainable), trained beside the 35B in
+cuda01's last 2.3 GB (peak 2158 MiB on the card, 64 s for 138 steps, PEFT in a
+venv over the vLLM engine's torch). Data: 90 Turkish Q/A pairs a cluster
+model wrote from the nine Demo/politikalar documents, through the gateway.
+On nine questions it had not seen, phrased otherwise: base 0/9, 6 epochs
+4/9, 4 epochs at a lower rate 2/9. Wrong answers mixed facts across
+documents (the meal card's amount, the office day). The same nine through
+retrieval: 9/9 with sources. So F4 is for format, vocabulary and tone, and
+facts come from F1–F3; a bigger base on a free card would do better but not
+change that. The training itself fits a small slot on a shared card, which
+the job design can rely on for small bases.
+
+Built: `ctl kb dataset` (POST /api/v1/kb/collections/{id}/dataset, NDJSON
+with each pair's collection); train_jobs/train_artifacts (migrations 0037,
+0038 owner); internal/control/train (queue, one job per node, artifacts in
+1 MiB pieces, a node that goes fails its job); internal/agent/train (the
+newest vLLM's python3 + `uv pip install --no-deps --target` peft 0.21.2 and
+accelerate 1.15.0; llama.cpp's conversion code for the installed release,
+untarred from its source; the card with the most free memory, if enough;
+runs as llmgw-engine); `ctl train start|list|show --follow|cancel|rm|
+download|publish`; the mirror's PUT /v1/models/{model}/{version}/{file};
+publish uploads the GGUF (llama.cpp) or PEFT files (vLLM) and sets the
+adapter after checkAdapters. Owners: catalog.Adapter.Owner, enforced in the
+router (404, hidden from /v1/models); a job's owner defaults to its single
+collection's owner, must be able to read every collection the pairs came
+from, and pairs from a knowledge base are never public. Tried on the PoC:
+dataset of 90 pairs in 89 s; a job trained beside the 35B (peak 2214 MiB,
+138 steps in about a minute) and sent back PEFT and GGUF.
+Left: a job dies with its agent's stream (no resume); no console screen for
+training yet; quality gating of a published adapter is by hand (ctl eval).
 
 ---
 

CHANGELOG.md

@@ -18,12 +18,110 @@ is only released once its **exit criterion** is met, not when its code lands.
 | v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released |
 | v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released |
 | v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
-| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released |
+| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine) |
 | v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
 
 Until v1.0.0 the minor version is the breaking one: configuration formats and Go
 APIs change between minors without a deprecation period.
 
+## Unreleased
+
+## v0.14.1 — 2026-10-09
+
+Adapters trained on the nodes from a knowledge base and kept private to
+their owners; the cluster's and the agents' settings edited in one place;
+the tenancy tree and keys over the admin API and in Terraform; knowledge
+bases that hold up with many documents; the firewall setting on every
+machine.
+
+**Upgrading.**
+- **Nodes** need this release to follow `agents.*` in the cluster's
+  settings; until a node is upgraded its agent.yaml is what it runs on.
+- **A separate mirror machine and store members that run nothing else**
+  follow the firewall setting once what reads it is installed: run `sudo
+  llmgw mirror init` on the mirror again with the flags it was set up with,
+  and `llmgw server init --patroni` again for the store's members. A control
+  plane set up before v0.12.0 needs `sudo llmgw firewall sync --install`.
+- **The store's PostgreSQL memory** applies after each member is restarted,
+  one at a time, the leader last; until then it runs as before.
+- The store gains tables and columns; nothing to do. Existing knowledge-base
+  collections get their full-text index when a control plane starts, and
+  keep `vector` storage; only new collections use `halfvec`, and only on
+  pgvector 0.7 or later.
+- Cached knowledge-base passages in vLLM are made again once, under the new
+  `cache_salt`.
+
+### Added
+
+- **The cluster's settings on the portal** (BACKLOG E4): `/admin/settings`
+  for administrators. It edits the document `llmgw ctl config edit` opens:
+  checked as a control plane would load it, refused when made from a
+  version since replaced, with the history to view and roll back. Changes
+  are audited by size, never by contents.
+- **The nodes' agent settings in the cluster's settings** (BACKLOG E3):
+  `agents.default` and `agents.nodes.<id>` hold everything agent.yaml can
+  say but the agent's id, control planes, address and certificate. They are
+  sent to the agents as they change; an agent starts again on new settings,
+  or on an edited agent.yaml, with its engines serving throughout. Settings
+  that would not load are refused on save, and set aside on a node rather
+  than keep its agent from starting. Nodes need this release to follow
+  them.
+- **LoRA adapters trained on the nodes** (BACKLOG F4): `llmgw ctl kb dataset`
+  writes question/answer pairs from a collection with a model of the
+  cluster. `llmgw ctl train start` trains an adapter on a node with its
+  installed vLLM's torch and a small overlay, on a card with enough memory
+  free (beside an engine, if it leaves that much), converting to GGUF for
+  llama.cpp. `llmgw ctl train publish` keeps it on the mirror (which now
+  accepts uploads) and makes it the model's adapter.
+- **Adapters with owners.** An adapter's `owner` (org, team, project, user
+  or key) is the only part of the tree that sees it; to anyone else it is a
+  model that does not exist. An adapter trained on a knowledge base belongs
+  to a part of the tree that may read it, and is never made public.
+- The knowledge bases' vLLM `cache_salt` follows the collections searched,
+  not the organization, so two teams of one organization no longer share
+  cached passages.
+- **Organizations, teams, projects, API keys and memberships in the admin
+  API**: `/api/v1/orgs` (the tree), `POST /api/v1/teams` and `projects`,
+  `PATCH` to rename or move, `DELETE` for what is empty; `/api/v1/keys` to
+  issue (the key is in the answer, once), list and revoke; `/api/v1/users`
+  and `/api/v1/users/{id}/memberships`. They were only `llmgw org`, `keys`
+  and `users` on a control plane before. The Terraform provider manages them
+  as `llmgw_org`, `llmgw_team`, `llmgw_project`, `llmgw_key` (the key in a
+  sensitive `secret`) and `llmgw_membership`.
+- **The firewall setting reaches every machine.** A separate mirror and the
+  store's members follow `firewall.enabled` and `firewall_networks` as nodes
+  and control planes did: the leading control plane tells the mirror over
+  its API, and keeps the state in the store for members that run no llmgw,
+  which a timer on each reads. The mirror also lets in the network of every
+  node the leader knows. `mirror init` and `server init --patroni` install
+  what follows it; run them again on a cluster set up before.
+
+### Changed
+
+- **The store's PostgreSQL is sized to its machine's memory.** Before, it ran
+  on PostgreSQL's defaults (128MB cache) under Patroni, or on 96MB on one
+  machine. Now each member gets a share of its own RAM, a smaller one where
+  a control plane or agent shares the machine, and the share is read again
+  every time `server init --patroni` runs. The new cache is applied by
+  restarting one member at a time, the leader last.
+- **Knowledge-base searches hold up with many documents.** New collections
+  store their index as `halfvec` where pgvector is 0.7 or later (half the
+  memory, and embeddings up to 4000 dimensions instead of 2000); older
+  collections, and clusters on pgvector 0.6, keep `vector`. Each collection
+  has its own full-text index, as it has its own vector index, made for
+  existing collections when the control plane starts. The text arm ranks by
+  the cheap measure first and scores only those candidates.
+
+### Fixed
+
+- A knowledge-base search asking for more than 10 passages
+  (`X-Llmgw-Knowledge-K`) got at most 40 candidates per collection from the
+  vector index, whatever it asked for: pgvector's `hnsw.ef_search` is now
+  raised to the number wanted, up to 1000.
+- `llmgw server backup` and `restore` against a replicated store over TLS
+  failed looking for `~/.postgresql/root.crt`: pg_dump was given
+  `sslmode=verify-full` without the cluster's authority.
+
 ## v0.14.0 — 2026-10-08
 
 Knowledge bases: a company's documents, put in by people from their own

README.md

@@ -3,8 +3,9 @@
 A single-binary LLM gateway for clusters with multiple InfiniBand fabrics,
 NVLink domains and heterogeneous GPUs.
 
-It serves LLMs running on your own GPU servers (vLLM / SGLang) behind one
-OpenAI-compatible endpoint. Design document: [llmgw-plan.md](llmgw-plan.md).
+It serves LLMs running on your own GPU and CPU servers (vLLM, SGLang,
+llama.cpp, TensorRT-LLM) behind one OpenAI- and Anthropic-compatible
+endpoint, and answers from your own documents. Design document: [llmgw-plan.md](llmgw-plan.md).
 
 ## Getting started
 
@@ -37,6 +38,9 @@ Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
 | v0.9.0 | — | Managing people and the tenancy tree | released |
 | v0.10.0 | — | InfiniBand fabrics and NVLink, doctor, node add, engines installed by the nodes | released |
 | v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released |
+| v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released |
+| v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
+| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine) |
 | v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
 
 As of v0.6.0 a cluster runs unattended: agents report their hardware, the
@@ -44,38 +48,67 @@ control plane places replicas — including ones larger than a single machine
 models load when someone asks for them and stop when nobody does, and keys,
 budgets and rate limits decide who may use any of it.
 
-Since v0.11.0, not yet released (see [CHANGELOG.md](CHANGELOG.md),
-"Unreleased"):
+Since v0.11.0 (each release's detail is in [CHANGELOG.md](CHANGELOG.md)):
 
-- **Security hardening** — signed releases only, a token on the admin API,
-  engines not run as root, TLS for the agent port, etcd and Patroni, a
-  firewall for llmgw's internal ports, encrypted backups, bounded
-  connections, sign-in throttling, a Content-Security-Policy.
-- **Guardrails** — policies that check requests and answers (PII, secrets,
-  prompt injection, deny patterns, a judge model such as Llama Guard), in
-  observe or enforce mode, attached to any part of the tenancy tree:
+- **Adapters, and the tenancy tree over the API** (v0.14.1). `llmgw ctl
+  kb dataset` and `llmgw ctl train` make a LoRA adapter from a collection
+  on a node's spare GPU memory, kept on the mirror and seen only by its
+  owner. Organizations, teams, projects, keys and memberships are in the
+  admin API and the Terraform provider. The agents' settings are edited
+  with the cluster's, on the portal too, and the firewall setting reaches
+  every machine.
+- **Knowledge bases** (v0.14.0). People put a company's documents in from
+  their own machines with `llmgw-kb`, a separate download for Windows,
+  macOS and Linux that reads PDF, Office, OpenDocument, mail and many more
+  formats and sends text and vectors, never the files. Collections are kept
+  apart per tenant, down to row-level security in PostgreSQL. A chat
+  request, or a part of the tree by default, is answered from them with
+  its sources: [docs/knowledge.md](docs/knowledge.md).
+- **One command for the whole system** (v0.13.0): `llmgw stop`, `start` and
+  `restart`. Qwen3Guard is the guardrails' judge, on a GPU and on CPUs
+  together.
+- **Settings that apply when changed** (v0.12.0). The cluster's
+  configuration lives in the store, the same on every control plane and in
+  force within seconds (`llmgw ctl config`, the console's Settings). The
+  firewall can be switched on and off for the whole cluster.
+- **Security hardening** (v0.11.2): signed releases only, a token on the
+  admin API, engines not run as root, TLS for the agent port, etcd and
+  Patroni, a firewall for llmgw's internal ports, encrypted backups,
+  bounded connections, sign-in throttling, a Content-Security-Policy.
+- **Guardrails** (v0.11.2). Policies check requests and answers for PII,
+  secrets, prompt injection, deny patterns, and with a judge model; they
+  observe or enforce, and attach to any part of the tenancy tree:
   [docs/guardrails.md](docs/guardrails.md).
-- **Administration** — an audit log of every change (`ctl audit`), staff
-  roles (viewer, operator, billing, admin), single sign-on through OpenID
-  Connect and provisioning through SCIM 2.0, key lifetime limits and
-  rotation with overlap (`keys ttl`, `keys rotate`), usage export as CSV
-  (`ctl usage export`).
-- **Notifications** — webhooks, signed or Slack-shaped, for a model parked,
-  a node lost, a budget spent and new weights published (`notify.webhooks`,
-  `ctl notify test`), and Alertmanager receiver templates.
-- **Traffic and models** — request tags, canaries of new weights with
-  automatic rollback, quality suites gating updates, a response cache,
-  kept and replayed requests, batches, Anthropic's Messages API, more of
-  OpenAI's (structured output, images, rerank, audio, moderations),
-  scaling by the clock, SLO-aware admission, TensorRT-LLM, MIG, models
-  imported from Ollama, routing by what each vLLM replica's cache holds,
-  LoRA adapters loaded on demand, other llmgw clusters as providers, and
-  placement that knows racks, PDUs and power caps.
-- **Operations** — `server restore` and off-machine backup copies, secrets
-  in Vault, forgetting a person and keeping a project's requests in the
-  cluster, offline installs from a bundle, a Helm chart, tracing from the
-  control plane to the engines, the admin API in OpenAPI with a Terraform
-  provider, and the portal in English and Turkish with its docs at /docs.
+- **Administration** (v0.11.2):
+  - an audit log of every change (`ctl audit`);
+  - staff roles (viewer, operator, billing, admin);
+  - single sign-on through OpenID Connect and provisioning through SCIM
+    2.0;
+  - key lifetime limits, and rotation with overlap (`keys ttl`, `keys
+    rotate`);
+  - usage export as CSV (`ctl usage export`).
+- **Notifications** (v0.11.2): webhooks, signed or Slack-shaped, for a
+  model parked, a node lost, a budget spent and new weights published, and
+  Alertmanager receiver templates.
+- **Traffic and models** (v0.11.2):
+  - request tags, canaries of new weights with automatic rollback, and
+    quality suites gating updates;
+  - a response cache, kept and replayed requests, and batches;
+  - Anthropic's Messages API, and more of OpenAI's (structured output,
+    images, rerank, audio, moderations);
+  - scaling by the clock, and SLO-aware admission;
+  - TensorRT-LLM, MIG, and models imported from Ollama;
+  - routing by what each vLLM replica's cache holds, and LoRA adapters
+    loaded on demand;
+  - other llmgw clusters as providers, and placement that knows racks,
+    PDUs and power caps.
+- **Operations** (v0.11.2):
+  - `server restore`, off-machine backup copies, and secrets in Vault;
+  - forgetting a person, and keeping a project's requests in the cluster;
+  - offline installs from a bundle, and a Helm chart;
+  - tracing from the control plane to the engines;
+  - the admin API in OpenAPI, with a Terraform provider;
+  - the portal in English and Turkish, with its docs at /docs.
 
 Every one of these is in the console as well as on the command line:
 [docs/console.md](docs/console.md).
@@ -124,6 +157,9 @@ Every page is also in the binary: `llmgw docs`, or **Manual** in the console.
   official OpenAI and Anthropic SDKs, and llmgw's extensions
 - [docs/guardrails.md](docs/guardrails.md) — checking requests and answers:
   policies, detectors, a judge model, and where each applies
+- [docs/knowledge.md](docs/knowledge.md) — knowledge bases: collections,
+  who may read them, `llmgw-kb` on people's machines, and answers from
+  documents
 - [docs/choosing-models.md](docs/choosing-models.md) — which models to run,
   how many of each, and where the weights should live
 
@@ -152,7 +188,7 @@ Every page is also in the binary: `llmgw docs`, or **Manual** in the console.
 ## Building
 
 ```sh
-make build      # bin/llmgw
+make build      # bin/llmgw and bin/llmgw-kb
 make test       # unit tests
 make race       # with the race detector
 ```
@@ -169,6 +205,7 @@ llmgw ctl cordon gpu-a-01            # stop new placement, leave what runs
 sudo llmgw doctor                    # on a node: what its GPUs, NVLink and fabric need
 llmgw ctl audit --since 24h          # who changed what
 llmgw ctl usage export --from 2026-09-01 --to 2026-10-01 --by project,model
+llmgw ctl kb list                    # knowledge-base collections, grants, defaults
 ```
 
 Bare `llmgw` opens the console, which does all of this from one screen.

api/proto/llmgw/v1/agent.proto

@@ -28,6 +28,8 @@ message AgentMessage {
     FabricTestReply fabric_test = 6;
     UpgradeReply upgrade = 7;
     KVEvents kv_events = 8;
+    TrainStatus train_status = 9;
+    TrainArtifact train_artifact = 10;
   }
 }
 
@@ -73,6 +75,9 @@ message ControlMessage {
     ControlPlanes control_planes = 11;
     Firewall firewall = 13;
     RestartAgent restart_agent = 14;
+    AgentSettings agent_settings = 15;
+    TrainStart train_start = 16;
+    TrainCancel train_cancel = 17;
   }
   // trace is the W3C trace context (traceparent, and tracestate when there
   // is one) of the control plane's span that sent this command, so what the
@@ -103,6 +108,17 @@ message Firewall {
   int32 engine_ports_to = 4;
 }
 
+// AgentSettings is what the cluster's settings say of this node's agent
+// (agents.default, with agents.nodes.<id> over it), sent when it registers
+// and whenever they change. The agent keeps it beside agent.yaml, which
+// holds only what it needs to start; over that file, it is what the agent
+// runs on. A change from what it runs on has the agent start again on it,
+// its engines serving throughout.
+message AgentSettings {
+  // yaml is agent.yaml's keys, as YAML; empty for none.
+  string yaml = 1;
+}
+
 // ControlPlanes is every control plane an agent may dial, sent by the leader
 // when an agent registers and whenever the list changes. The agent keeps it
 // beside its configuration, so a second control plane added after the node
@@ -872,3 +888,55 @@ message LoraAdapter {
   string name = 1;
   string source = 2;
 }
+
+// TrainStart has a node train a LoRA adapter (BACKLOG F4): on a base model's
+// weights from its registry, on question/answer pairs, with a Python trainer
+// the agent carries, in an environment it makes over an installed engine's.
+// One at a time per node.
+message TrainStart {
+  string job_id = 1;
+  // base_repo and base_revision name the base model's weights on its
+  // registry (Hugging Face): safetensors, not the GGUF a catalog may serve.
+  string base_repo = 2;
+  string base_revision = 3;
+  // data is the training pairs, NDJSON of {"q", "a"}.
+  bytes data = 4;
+  // params is the trainer's settings as JSON: rank, alpha, epochs, lr,
+  // batch, max_len.
+  string params = 5;
+  // gpu_memory_mib is what the job needs of a card; the agent starts it
+  // only when one has that much free, which lets it run beside an engine.
+  int32 gpu_memory_mib = 6;
+  // gguf asks for the adapter in llama.cpp's format too.
+  bool gguf = 7;
+}
+
+// TrainCancel stops a training job.
+message TrainCancel {
+  string job_id = 1;
+}
+
+// TrainStatus is how a training job is going, sent as it changes.
+message TrainStatus {
+  string job_id = 1;
+  // state is preparing, running, done or failed.
+  string state = 2;
+  int32 step = 3;
+  int32 steps = 4;
+  double loss = 5;
+  // message is what is being done, or why it failed.
+  string message = 6;
+  // peak_mib is the most the job held of the card.
+  int32 peak_mib = 7;
+}
+
+// TrainArtifact is a piece of what a training job made, sent before its
+// final status: a gzipped tar of the PEFT adapter (kind peft), or the GGUF
+// adapter (kind gguf).
+message TrainArtifact {
+  string job_id = 1;
+  string kind = 2;
+  int32 seq = 3;
+  bytes data = 4;
+  bool last = 5;
+}

docs/admin-api.md

@@ -8,8 +8,9 @@ description is.
 anything they do can be done with `curl`, a generated client or the
 [Terraform provider](../terraform-provider-llmgw/README.md). It adds and
 removes models, scales them, cordons and upgrades nodes, sets capacity
-quotas, attaches guardrail policies, and reads the fleet, usage and the audit
-log.
+quotas, attaches guardrail policies, adds organizations, teams and projects,
+issues and revokes API keys, puts people in the tree, and reads the fleet,
+usage and the audit log.
 
 ## Reaching it
 
@@ -99,6 +100,14 @@ curl -s "${H[@]}" -X POST $API/models/qwen-small/scale -d '{"min": 0, "max": 2,
 # A team's capacity quota.
 curl -s "${H[@]}" -X PUT $API/quotas -d '{"scope": "team:Acme/ml", "limit": {"gpus": 8}}'
 
+# An organization, a team in it and a project in that; then a key for the
+# project. The answer holds the key once: only its hash is kept.
+curl -s "${H[@]}" -X POST $API/orgs -d '{"name": "Acme"}'
+curl -s "${H[@]}" -X POST $API/teams -d '{"name": "ml", "in": "Acme"}'
+curl -s "${H[@]}" -X POST $API/projects -d '{"name": "search", "in": "Acme/ml"}'
+curl -s "${H[@]}" -X POST $API/keys -d '{"name": "ci", "project": "Acme/ml/search", "expires_in": "720h"}'
+curl -s "${H[@]}" -X POST $API/users/ayse@example.com/memberships -d '{"target": "team:Acme/ml"}'
+
 # Remove the model; a DELETE carries the content type too.
 curl -s "${H[@]}" -X DELETE $API/models/qwen-small
 ```
@@ -149,11 +158,11 @@ go generate ./internal/docgen
 
 ## What it does not cover
 
-- **Organizations, teams, projects, people and API keys.** These live in the
-  store and are managed with `llmgw org`, `llmgw users` and `llmgw keys` on the
-  control plane, or in the web console's `/admin` pages on the client port
-  behind a session. The admin API reads the tree only where a quota or a
-  policy names a part of it.
+- **Making people's accounts.** People sign up, are invited, or come from
+  the identity provider (OIDC, SCIM); the API lists them and puts them in
+  the tree, and `llmgw users` approves, disables and deletes them. Key
+  lifetime limits and spending limits are `llmgw keys ttl` and `llmgw keys
+  limits`.
 - **The client-facing API** (`/v1/chat/completions` and the rest), which is
   OpenAI's and Anthropic's, on `listen`, with API keys.
 - **The rest of the admin listener**: `/metrics`, `/health`, the dashboard

docs/install.md

@@ -201,6 +201,16 @@ this by running the same command again, on the control plane that holds the
 authority: etcd is rebuilt on TLS with Patroni in maintenance mode, so
 PostgreSQL serves throughout and no failover happens.
 
+Each member's PostgreSQL is sized to the machine's memory, read each time
+the command runs: a quarter of it as cache (`shared_buffers`), three quarters
+counted on for the planner, and room to sort and build indexes, all capped. A
+member that also runs a control plane or an agent gets less: about 15% as
+cache, so the gateway is not pushed into swap. A knowledge base's vector
+index is searched well only from memory, so give a store machine more RAM and
+run the command again. The new cache takes a PostgreSQL restart, which Patroni
+does one member at a time, the leader last. The leader's restart costs a few
+seconds without writes, and there is no failover.
+
 Every step looks before it acts. Running it again finishes an install that
 stopped halfway and changes nothing on one that did not; it never overwrites
 `server.yaml`, the authority, or an existing account's password.
@@ -258,8 +268,14 @@ llmgw ctl config history                 # who changed what, when, and why
 llmgw ctl config rollback 4              # back to version 4, as a new version
 ```
 
-or on the console's **Settings** screen. A change is checked before it is
-kept — one that would not load is refused — and recorded in the audit log.
+or on the console's **Settings** screen, or in a browser on the portal's
+**Settings** page (`/admin/settings`, for administrators only: the document
+can hold webhook secrets). The page edits the same document, lists its
+history, and puts an earlier version back. A change is checked before it is
+kept, so one that would not load is refused. A change made from a version
+someone else has replaced since is refused too, rather than overwriting
+theirs. Every change is recorded in the audit log, with the document's size
+but not its contents.
 Guardrails change live; anything else has every control plane start again on
 the new settings, the standbys first and the leader half a minute later, so
 one of them is always serving. Engines keep running through it.
@@ -269,6 +285,33 @@ cluster keys — with a `baseline` guard policy, if the file had none — and
 from then on those keys in any server.yaml are not read (the server says so
 when it starts).
 
+**The nodes' agents** are set the same way, under `agents`. `agents.default`
+is for every node, and `agents.nodes.<node-id>` is for one node, laid over
+the default:
+
+```
+llmgw ctl config set agents.default.heartbeat_interval 5s
+llmgw ctl config set agents.nodes.cuda01.labels.gpu rtx6000
+llmgw ctl config edit                    # fabrics, engines, supervisor, discovery…
+```
+
+Each node's `agent.yaml` then needs only what its agent must know before it
+can be told anything: `node_id`, `server` or `servers`, `hostname`,
+`mgmt_addr` and `tls`. Those keys are refused under `agents`.
+
+- **Precedence.** Anything else in the file is still read, with the
+  cluster's settings over it.
+- **Applying a change.** The control plane sends each node its settings, a
+  second apart. An agent whose settings changed keeps them beside its
+  `agent.yaml` (in `/var/lib/llmgw-agent/agent-settings.yaml`) and starts
+  again on them, its engines serving throughout.
+- **Bad settings.** Settings that would not load are refused when they are
+  saved. If they somehow do not load beside one node's own file, that agent
+  keeps running on what it has and logs why.
+- **Editing `agent.yaml` itself.** The agent notices the change within ten
+  seconds and starts again on it. If the edited file does not load, the
+  agent reports it and keeps running on what it had.
+
 **This machine's own keys** stay in `/etc/llmgw/server.yaml`: `listen`,
 `admin_listen`, `metrics_listen`, `agent_listen`, `agent_tls`, `auth.store`,
 `secrets`, `tracing`, `monitoring`, `mirror.api` and `mirror.tls`, the backup
@@ -1703,9 +1746,11 @@ the file again.
 
 On a machine that also runs the control plane, PostgreSQL's defaults assume it
 owns the box. A gateway swapped out to give the database cache is a worse
-outcome than a smaller cache, so cap it: `shared_buffers = 96MB` and
-`max_connections = 40` are enough for a store whose working set is measured in
-megabytes.
+outcome than a smaller cache, so size it to the machine. `server init` does this
+for a PostgreSQL it installs: `shared_buffers` about 15% of memory (at least
+96MB), `effective_cache_size` half, `work_mem` and `maintenance_work_mem`
+scaled to it, and `max_connections = 40`. A store with a large knowledge base
+wants a machine of its own, where its vector index fits in memory.
 
 ## Tagging requests
 
@@ -2151,13 +2196,29 @@ llmgw ctl config set firewall.enabled false   # every machine's table taken out
 llmgw ctl config set firewall.enabled true    # and put back as it was
 ```
 
-It is in force within seconds: each node's agent takes its table out or puts
-it back, and each control plane's `llmgw-firewall-sync.path` (installed by
-`server init` and `server add`; `sudo llmgw firewall sync --install` on one
-set up before) does the same there. The rules files stay, so turning it back
-on restores them. Nodes added while it is off get none. A separate mirror
-machine and the store's members keep theirs for now. `firewall_networks`
-changes reach the nodes the same way.
+It is in force on every machine within a minute, each in its own way:
+
+- **nodes**: the agent takes its table out or puts it back;
+- **control planes**: `llmgw-firewall-sync.path`, installed by `server
+  init` and `server add` (`sudo llmgw firewall sync --install` on one set up
+  before);
+- **a separate mirror machine**: the leading control plane tells the mirror,
+  which writes it to `.firewall.json` in its directory, and the same path
+  unit applies it. `mirror init` installs it; on a mirror set up before, run
+  `sudo llmgw mirror init` again with the same flags;
+- **store members that are nothing else**: they run no llmgw, so
+  `llmgw-firewall-follow.timer` reads what the leading control plane keeps in
+  the store, from the member's own copy, once a minute. `server init
+  --patroni` installs it; run it again on a cluster set up before.
+
+The rules files stay, so turning it back on restores them. Nodes added while
+it is off get none. Networks added to `firewall_networks` reach every machine
+the same way. One taken out of it stays let in until it is taken out of
+`/etc/llmgw/firewall.nft` by hand: the rules are only ever added to.
+
+The leading control plane also tells the mirror machine where every node and
+control plane is, so a node on a network the mirror had not let in can fetch
+its weights without a `firewall_networks` entry.
 
 Only those ports are touched. The table's policy is to accept, so whatever
 else the machine serves — and whatever firewall it already has — is left as

docs/knowledge.md

@@ -45,6 +45,11 @@ is available to everyone who may use them.
   A collection stays with the model it was made with: vectors from two
   models cannot be compared.
 
+  With pgvector 0.7 or later a collection's index is kept at half
+  precision (`halfvec`), which halves its memory and takes models of up to
+  4000 dimensions; with an older pgvector the limit is 2000. A collection
+  keeps the kind it was made with.
+
 ## For administrators
 
 ### Make a collection
@@ -317,9 +322,11 @@ Behaviour to know about:
   (Codex, Claude Code) work the same way, since they are answered as chat
   completions.
 - **Separate prefix caches.** On vLLM replicas, a request given passages
-  carries a `cache_salt` derived from its organization. One organization's
-  passages then never share cached blocks with another's, so how fast an
-  answer starts says nothing about what another tenant asked.
+  carries a `cache_salt` derived from the collections they came from.
+  Requests that could be given the same passages still share cached blocks,
+  and no one else's do: not another organization's, nor another team's of
+  the same one. How fast an answer starts therefore says nothing about what
+  someone else asked.
 - **No answer cache.** Answers from knowledge bases are never served from
   the answer cache, because two keys of one project can read different
   collections.
@@ -340,6 +347,49 @@ a product code or a name.
 Metrics: `llmgw_kb_retrievals_total{result}` (found, nothing, refused) and
 `llmgw_kb_retrieval_seconds`.
 
+## Training an adapter on a collection
+
+An adapter (LoRA) changes how a model *writes*: the organisation's words,
+format and tone. It does not reliably teach facts. On the PoC, a small
+model trained on nine policy documents answered 4 of 9 rephrased questions
+right and mixed facts up on the rest, while the same questions through the
+knowledge base got 9 of 9. Use both: the knowledge base for what answers
+say, an adapter for how they sound.
+
+```sh
+# 1. Questions and answers from the collection, written by a model of the
+#    cluster. Read the file through and correct it.
+llmgw ctl kb dataset Acme/policies --model qwen2.5-coder-14b-instruct --out policies.jsonl
+
+# 2. Train on a node. --gpu-memory is what the job needs of a card; it
+#    runs beside an engine when the engine leaves that much free.
+llmgw ctl train start policies-tone --model qwen2.5-0.5b-instruct \
+  --base-repo Qwen/Qwen2.5-0.5B-Instruct --node cuda01 --gpu-memory 2300 --data policies.jsonl
+llmgw ctl train show JOB --follow
+
+# 3. Make it the model's adapter, kept on the mirror.
+llmgw ctl train publish JOB --restart
+```
+
+The node trains with its installed vLLM's torch and a small overlay (peft
+and accelerate), as the engines' user. For a model llama.cpp serves, it also
+converts the adapter to GGUF with the installed llama.cpp's own code. What
+it makes is kept by the control plane (`llmgw ctl train download`).
+
+An adapter is private like its collection:
+
+- **It has an owner.** An adapter trained on a collection carries some of
+  what the collection says in its weights, so it belongs to a part of the
+  tree.
+  - By default the owner is the collection's owner.
+  - `--owner` names another, which must be able to read every collection
+    the pairs came from.
+  - Only callers in the owner's part of the tree see the adapter. To anyone
+    else it is a model that does not exist, absent from `/v1/models` too.
+- **It is public only by choice.** `--public` makes an adapter every
+  caller's. It is refused for pairs from a knowledge base, so it is only for
+  pairs written by hand.
+
 ## How it is kept apart
 
 Passages live in PostgreSQL, partitioned by organization, with an index per

docs/reference.md

@@ -18,6 +18,7 @@ rather than repeated.
 | `name` | `string` | [Adapter](#adapter) |
 | `source` | `string` | [Adapter](#adapter) |
 | `scale` | `float64` | [Adapter](#adapter) |
+| `owner` | `string` | [Adapter](#adapter) |
 | `enabled` | `bool` | [Admission](#admission) |
 | `bands` | `[]Band` | [Admission](#admission) |
 | `default_band` | `string` | [Admission](#admission) |
@@ -26,6 +27,8 @@ rather than repeated.
 | `queue_capacity_per_band` | `int` | [Admission](#admission) |
 | `queue_capacity_total` | `int` | [Admission](#admission) |
 | `interval` | `time.Duration` | [Admission](#admission) |
+| `default` | `map[string]any` | [AgentSettings](#agentsettings) |
+| `nodes` | `map[string]map[string]any` | [AgentSettings](#agentsettings) |
 | `enabled` | `bool` | [Auth](#auth) |
 | `store` | `string` | [Auth](#auth) |
 | `cache_ttl` | `time.Duration` | [Auth](#auth) |
@@ -294,6 +297,7 @@ rather than repeated.
 | `mail` | `mail.Config` | [Server](#server) |
 | `serving` | `ServingSettings` | [Server](#server) |
 | `firewall` | `FirewallSettings` | [Server](#server) |
+| `agents` | `AgentSettings` | [Server](#server) |
 | `firewall_networks` | `[]string` | [Server](#server) |
 | `backup_recipients` | `[]string` | [Server](#server) |
 | `backup_copy_to` | `[]string` | [Server](#server) |
@@ -423,6 +427,16 @@ weights are.
 
 Scale is how strongly it is applied; zero takes 1.
 
+### `owner`
+
+`string`
+
+Owner is the part of the tenancy tree the adapter belongs to —
+org:ID, team:ID, project:ID, user:ID or key:ID — and only callers in
+it see it: to anyone else it is a model that does not exist. An
+adapter trained on an organisation's documents carries some of what
+they say in its weights. Empty is every caller the model allows.
+
 ## Admission
 
 ### `enabled`
@@ -469,6 +483,21 @@ Zero takes half of all the bands' capacities together.
 
 `time.Duration`
 
+## AgentSettings
+
+### `default`
+
+`map[string]any`
+
+Default is agent.yaml's keys for every node.
+
+### `nodes`
+
+`map[string]map[string]any`
+
+Nodes are agent.yaml's keys for one node each, by node id, over
+Default.
+
 ## Auth
 
 ### `enabled`
@@ -2520,6 +2549,18 @@ Firewall turns llmgw's firewall on every machine on or off.
 
 Takes a [FirewallSettings](#firewallsettings) block.
 
+### `agents`
+
+`AgentSettings`
+
+Agents are the nodes' agent settings: agent.yaml's keys but those an
+agent needs to start, for every node (default) and for one
+(nodes.<id>). Sent to the agents as they change; an agent whose
+settings changed starts again on them, its engines serving
+throughout.
+
+Takes a [AgentSettings](#agentsettings) block.
+
 ### `firewall_networks`
 
 `[]string`

internal/agent/agent.go

@@ -34,6 +34,7 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/train"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/agentsvc"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
@@ -136,6 +137,13 @@ type Agent struct {
 	// by mu.
 	kv     *kvevents.Mirror
 	kvSubs map[string]*kvevents.Subscriber
+
+	// settingsCheck says whether the cluster's settings for this agent
+	// would load beside its agent.yaml; see SetSettingsCheck.
+	settingsCheck func([]byte) error
+
+	// trainer runs LoRA training jobs, one at a time (BACKLOG F4).
+	trainer *train.Runner
 }
 
 // connectHTTPClient is the HTTP client connect uses; an interface so tests can
@@ -182,6 +190,18 @@ func New(cfg Config, httpClient connectHTTPClient, log *slog.Logger) (*Agent, er
 		kvSubs:       map[string]*kvevents.Subscriber{},
 	}
 	a.prov = &provision.Installer{Root: cfg.EnginesDir, Log: log, OnStatus: a.provisionStatus}
+	runAs := cfg.Superv.RunAs
+	if runAs == "" {
+		runAs = supervisor.DefaultRunAs
+	}
+	// Where the engines are, as the installer has it when the file says
+	// nothing.
+	enginesDir := cfg.EnginesDir
+	if enginesDir == "" {
+		enginesDir = provision.DefaultRoot
+	}
+	a.trainer = train.New(train.Env{EnginesDir: enginesDir, WorkDir: filepath.Join(cfg.Superv.WorkDir, "trainer"),
+		RunAs: runAs, Log: log})
 	return a, nil
 }
 
@@ -880,6 +900,12 @@ func (a *Agent) handle(ctx context.Context, msg *llmgwv1.ControlMessage, out cha
 		a.setControlPlanes(m.ControlPlanes)
 	case *llmgwv1.ControlMessage_Firewall:
 		go a.applyFirewall(m.Firewall)
+	case *llmgwv1.ControlMessage_AgentSettings:
+		go a.applySettings(m.AgentSettings)
+	case *llmgwv1.ControlMessage_TrainStart:
+		go a.trainJob(ctx, m.TrainStart, out)
+	case *llmgwv1.ControlMessage_TrainCancel:
+		a.trainer.Cancel(m.TrainCancel.GetJobId())
 	case *llmgwv1.ControlMessage_RestartAgent:
 		go func() {
 			a.log.Info("restarting, as the control plane asks; the engines keep serving",

internal/agent/config.go

@@ -103,6 +103,11 @@ type Config struct {
 	Tracing obs.Tracing `koanf:"tracing"`
 
 	Version string
+
+	// Settings is the cluster's settings for this agent it was started on,
+	// as the control plane sent them: what a newly sent version is
+	// compared with.
+	Settings string `koanf:"-"`
 }
 
 // DefaultConfig returns sensible defaults.

internal/agent/settings.go

@@ -0,0 +1,109 @@
+package agent
+
+import (
+	"context"
+	"crypto/sha256"
+	"os"
+	"path/filepath"
+	"time"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+)
+
+// The cluster's settings for this agent (BACKLOG E3): what agent.yaml can
+// say but the agent's id, control planes, address and certificate, kept by
+// the control planes and sent over the stream. Kept here beside agent.yaml,
+// which then holds only what the agent needs to start; at start the two are
+// read together, these over the file.
+
+// SettingsFile keeps the settings the control plane last sent.
+var SettingsFile = "/var/lib/llmgw-agent/agent-settings.yaml"
+
+// ReadSettings is what SettingsFile holds, nil when nothing.
+func ReadSettings() []byte {
+	b, err := os.ReadFile(SettingsFile)
+	if err != nil {
+		return nil
+	}
+	return b
+}
+
+// SetSettingsCheck says how to tell whether settings would load beside this
+// agent's own file; settings that would not are not taken.
+func (a *Agent) SetSettingsCheck(f func([]byte) error) { a.settingsCheck = f }
+
+// applySettings takes the settings the control plane sent: nothing when they
+// are what this agent runs on, and otherwise kept and started again on.
+func (a *Agent) applySettings(m *llmgwv1.AgentSettings) {
+	next := m.GetYaml()
+	if next == a.cfg.Settings {
+		return
+	}
+	if a.settingsCheck != nil {
+		if err := a.settingsCheck([]byte(next)); err != nil {
+			// Kept running on what it has: settings that will not load
+			// here must not take the node out of service.
+			a.log.Error("the cluster's settings for this agent do not load beside its agent.yaml; running on what it has",
+				"err", err)
+			return
+		}
+	}
+	if err := os.MkdirAll(filepath.Dir(SettingsFile), 0o755); err != nil {
+		a.log.Error("could not keep the cluster's settings for this agent", "err", err)
+		return
+	}
+	tmp := SettingsFile + ".new"
+	if err := os.WriteFile(tmp, []byte(next), 0o600); err != nil {
+		a.log.Error("could not keep the cluster's settings for this agent", "err", err)
+		return
+	}
+	if err := os.Rename(tmp, SettingsFile); err != nil {
+		a.log.Error("could not keep the cluster's settings for this agent", "err", err)
+		return
+	}
+	a.log.Info("the cluster's settings for this agent changed; starting again on them, the engines keep serving")
+	time.Sleep(300 * time.Millisecond)
+	exitForUpgrade()
+}
+
+// WatchFile starts the agent again when its configuration file changes and
+// still loads: agent.yaml is read only at start, and a change to it that
+// waited for somebody to restart the agent was a change nobody saw take
+// effect.
+func (a *Agent) WatchFile(ctx context.Context, path string, loads func() error) {
+	if path == "" {
+		return
+	}
+	sum := func() [32]byte {
+		b, _ := os.ReadFile(path) //nolint:gosec // the agent's own configuration
+		return sha256.Sum256(b)
+	}
+	was := sum()
+	t := time.NewTicker(watchEvery)
+	defer t.Stop()
+	for {
+		select {
+		case <-ctx.Done():
+			return
+		case <-t.C:
+		}
+		now := sum()
+		if now == was {
+			continue
+		}
+		if err := loads(); err != nil {
+			// Once per change: an editor saving halfway must not stop the
+			// agent, and a broken file is reported, not restarted into.
+			a.log.Error("agent.yaml changed and does not load; running on what it had", "path", path, "err", err)
+			was = now
+			continue
+		}
+		a.log.Info("agent.yaml changed; starting again on it, the engines keep serving", "path", path)
+		time.Sleep(300 * time.Millisecond)
+		exitForUpgrade()
+		return
+	}
+}
+
+// watchEvery is how often agent.yaml is looked at.
+var watchEvery = 10 * time.Second

internal/agent/settings_test.go

@@ -0,0 +1,88 @@
+package agent
+
+import (
+	"errors"
+	"os"
+	"path/filepath"
+	"sync/atomic"
+	"testing"
+	"time"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+)
+
+func settingsAgent(t *testing.T, running string) (*Agent, chan struct{}) {
+	t.Helper()
+	oldFile, oldExit := SettingsFile, exitForUpgrade
+	SettingsFile = filepath.Join(t.TempDir(), "agent-settings.yaml")
+	exited := make(chan struct{}, 1)
+	exitForUpgrade = func() { exited <- struct{}{} }
+	t.Cleanup(func() { SettingsFile, exitForUpgrade = oldFile, oldExit })
+	cfg := Config{Server: "127.0.0.1:7443", NodeID: "n1", WeightsCache: t.TempDir(), EnginesDir: t.TempDir(), Settings: running}
+	cfg.Superv.WorkDir = t.TempDir()
+	a, err := New(cfg, nil, nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	return a, exited
+}
+
+// exitedSoon is whether the agent exited within a second: it waits a
+// moment for its log line to leave first.
+func exitedSoon(ch chan struct{}) bool {
+	select {
+	case <-ch:
+		return true
+	case <-time.After(time.Second):
+		return false
+	}
+}
+
+func TestAnAgentOnItsSettingsStaysAndOneOffThemStartsAgain(t *testing.T) {
+	a, exited := settingsAgent(t, "heartbeat_interval: 5s\n")
+	a.applySettings(&llmgwv1.AgentSettings{Yaml: "heartbeat_interval: 5s\n"})
+	if exitedSoon(exited) || ReadSettings() != nil {
+		t.Error("settings it already runs on restarted it")
+	}
+	a.applySettings(&llmgwv1.AgentSettings{Yaml: "heartbeat_interval: 2s\n"})
+	if !exitedSoon(exited) || string(ReadSettings()) != "heartbeat_interval: 2s\n" {
+		t.Errorf("new settings: kept %q", ReadSettings())
+	}
+}
+
+func TestSettingsThatWouldNotLoadAreNotTaken(t *testing.T) {
+	a, exited := settingsAgent(t, "")
+	a.SetSettingsCheck(func([]byte) error { return errors.New("fabric ib-a lists no HCAs") })
+	a.applySettings(&llmgwv1.AgentSettings{Yaml: "fabrics: [{name: ib-a}]\n"})
+	if exitedSoon(exited) || ReadSettings() != nil {
+		t.Error("settings that do not load were taken")
+	}
+}
+
+func TestAChangedAgentYamlThatLoadsStartsTheAgentAgain(t *testing.T) {
+	a, exited := settingsAgent(t, "")
+	old := watchEvery
+	watchEvery = 10 * time.Millisecond
+	t.Cleanup(func() { watchEvery = old })
+	p := filepath.Join(t.TempDir(), "agent.yaml")
+	_ = os.WriteFile(p, []byte("node_id: n1\n"), 0o600)
+	var loads atomic.Bool
+	check := func() error {
+		if loads.Load() {
+			return nil
+		}
+		return errors.New("broken")
+	}
+	go a.WatchFile(t.Context(), p, check)
+	time.Sleep(30 * time.Millisecond)
+	// Saved broken: reported, not restarted into.
+	_ = os.WriteFile(p, []byte("node_id: [\n"), 0o600)
+	if exitedSoon(exited) {
+		t.Fatal("a broken agent.yaml restarted the agent")
+	}
+	loads.Store(true)
+	_ = os.WriteFile(p, []byte("node_id: n1\nlabels: {a: b}\n"), 0o600)
+	if !exitedSoon(exited) {
+		t.Error("a changed agent.yaml did not restart the agent")
+	}
+}

internal/agent/supervisor/runas.go

@@ -3,6 +3,7 @@ package supervisor
 import (
 	"fmt"
 	"os"
+	"os/exec"
 	"os/user"
 	"path/filepath"
 	"strconv"
@@ -90,3 +91,25 @@ func credentialFor(u *user.User, workDir string) (*runAs, error) {
 func (r *runAs) env() []string {
 	return []string{"HOME=" + r.home, "XDG_CACHE_HOME=" + filepath.Join(r.home, ".cache"), "USER=" + r.name, "LOGNAME=" + r.name}
 }
+
+// AsEngineUser has cmd run as the engine user, as an engine would, with its
+// home under workDir: for work the agent does that is not an engine but
+// reads what an engine reads — training an adapter. It answers that home
+// and a way to hand a path to the user, or why it runs as this process.
+func AsEngineUser(cmd *exec.Cmd, name, workDir string) (home string, give func(path string) error, why string) {
+	r, why := resolveRunAs(name, workDir)
+	if r == nil {
+		return "", func(string) error { return nil }, why
+	}
+	cmd.SysProcAttr = &syscall.SysProcAttr{Credential: r.cred, Setpgid: true}
+	cmd.Env = append(cmd.Env, r.env()...)
+	uid, gid := int(r.cred.Uid), int(r.cred.Gid)
+	return r.home, func(path string) error {
+		return filepath.WalkDir(path, func(p string, _ os.DirEntry, err error) error {
+			if err != nil {
+				return err
+			}
+			return os.Lchown(p, uid, gid)
+		})
+	}, ""
+}

internal/agent/train/runner.go

@@ -0,0 +1,574 @@
+// Package train runs LoRA training jobs on a node (BACKLOG F4): with an
+// installed engine's Python — its torch and transformers, already matched to
+// the node's CUDA — and a small overlay of what training needs besides, so a
+// node that serves can train without a second multi-gigabyte environment.
+package train
+
+import (
+	"archive/tar"
+	"bufio"
+	"bytes"
+	"compress/gzip"
+	"context"
+	"crypto/sha256"
+	_ "embed"
+	"encoding/hex"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"log/slog"
+	"net/http"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"sort"
+	"strconv"
+	"strings"
+	"sync"
+	"syscall"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
+)
+
+//go:embed trainer.py
+var trainerPy []byte
+
+// overlayPackages are what training needs besides an engine's Python, pinned
+// and installed without their dependencies: those are the engine's own, and
+// letting an installer resolve them pulled a second torch.
+var overlayPackages = []string{"peft==0.21.2", "accelerate==1.15.0"}
+
+// Job is one training job, as the control plane sent it.
+type Job struct {
+	ID           string
+	BaseRepo     string
+	BaseRevision string
+	// Data is NDJSON of {"q", "a"}.
+	Data []byte
+	// Params is the trainer's settings as JSON.
+	Params       string
+	GPUMemoryMiB int
+	GGUF         bool
+}
+
+// Status is how a job is going.
+type Status struct {
+	State   string
+	Message string
+	Step    int
+	Steps   int
+	Loss    float64
+	PeakMiB int
+}
+
+// Env is where a node keeps what training needs.
+type Env struct {
+	// EnginesDir is where engines are installed (the agent's engines_dir).
+	EnginesDir string
+	// WorkDir is the trainer's own directory: its overlay, the llama.cpp
+	// conversion code, and each job's files while it runs.
+	WorkDir string
+	// RunAs is the user jobs run as: the engines' user.
+	RunAs string
+	Log   *slog.Logger
+	// Python, UV, ConvDir and GPUs replace what would be found, for tests.
+	Python, UV, ConvDir string
+	GPUs                func(ctx context.Context) ([]GPU, error)
+}
+
+// GPU is one card and how much of it is free.
+type GPU struct {
+	Index   int
+	FreeMiB int
+}
+
+// Runner runs one job at a time.
+type Runner struct {
+	env    Env
+	mu     sync.Mutex
+	cancel context.CancelFunc
+	job    string
+}
+
+// New is a runner.
+func New(env Env) *Runner {
+	if env.Log == nil {
+		env.Log = slog.New(slog.DiscardHandler)
+	}
+	return &Runner{env: env}
+}
+
+// Cancel stops a job if it is the one running.
+func (r *Runner) Cancel(id string) {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	if r.job == id && r.cancel != nil {
+		r.cancel()
+	}
+}
+
+// ErrBusy is a node already training.
+var ErrBusy = errors.New("this node is already training")
+
+// Run runs a job to its end, telling status as it goes and artifact each
+// piece of what it made. It answers the error it failed with.
+func (r *Runner) Run(ctx context.Context, job Job, status func(Status), artifact func(kind string, seq int, data []byte, last bool) error) error {
+	r.mu.Lock()
+	if r.job != "" {
+		r.mu.Unlock()
+		return ErrBusy
+	}
+	ctx, cancel := context.WithCancel(ctx)
+	r.job, r.cancel = job.ID, cancel
+	r.mu.Unlock()
+	defer func() {
+		cancel()
+		r.mu.Lock()
+		r.job, r.cancel = "", nil
+		r.mu.Unlock()
+	}()
+
+	status(Status{State: "preparing", Message: "preparing the trainer"})
+	py, err := r.python()
+	if err != nil {
+		return err
+	}
+	overlay, err := r.overlay(ctx, py)
+	if err != nil {
+		return fmt.Errorf("installing %s: %w", strings.Join(overlayPackages, ", "), err)
+	}
+	conv := ""
+	if job.GGUF {
+		if conv, err = r.conversion(ctx); err != nil {
+			return fmt.Errorf("fetching llama.cpp's conversion code: %w", err)
+		}
+	}
+	gpu, err := r.pickGPU(ctx, job.GPUMemoryMiB)
+	if err != nil {
+		return err
+	}
+
+	dir := filepath.Join(r.env.WorkDir, "jobs", job.ID)
+	if err := os.MkdirAll(dir, 0o755); err != nil {
+		return err
+	}
+	defer os.RemoveAll(dir)
+	out := filepath.Join(dir, "adapter")
+	var params map[string]any
+	if job.Params != "" {
+		if err := json.Unmarshal([]byte(job.Params), &params); err != nil {
+			return fmt.Errorf("the job's params: %w", err)
+		}
+	}
+	spec, _ := json.Marshal(map[string]any{"base_repo": job.BaseRepo, "base_revision": job.BaseRevision,
+		"data": filepath.Join(dir, "data.jsonl"), "out": out, "params": params})
+	for name, b := range map[string][]byte{"data.jsonl": job.Data, "job.json": spec, "trainer.py": trainerPy} {
+		if err := os.WriteFile(filepath.Join(dir, name), b, 0o644); err != nil {
+			return err
+		}
+	}
+
+	var baseDir string
+	err = r.python3(ctx, py, dir, []string{"PYTHONPATH=" + overlay, "CUDA_VISIBLE_DEVICES=" + strconv.Itoa(gpu)},
+		[]string{filepath.Join(dir, "trainer.py"), filepath.Join(dir, "job.json")},
+		func(line map[string]any) {
+			s := Status{State: str(line["state"]), Message: str(line["message"]), Step: num(line["step"]),
+				Steps: num(line["steps"]), PeakMiB: num(line["peak_mib"])}
+			if v, ok := line["loss"].(float64); ok {
+				s.Loss = v
+			}
+			if b := str(line["base_dir"]); b != "" {
+				baseDir = b
+			}
+			if s.State == "trained" {
+				s.State = "running"
+				s.Message = "trained; packing the adapter"
+			}
+			status(s)
+		})
+	if err != nil {
+		return err
+	}
+
+	if job.GGUF {
+		status(Status{State: "running", Message: "converting the adapter to GGUF for llama.cpp"})
+		gguf := filepath.Join(dir, "adapter.gguf")
+		err := r.python3(ctx, py, dir, []string{"PYTHONPATH=" + strings.Join([]string{filepath.Join(conv, "gguf-py"), conv, overlay}, ":")},
+			[]string{filepath.Join(conv, "convert_lora_to_gguf.py"), "--base", baseDir, "--outtype", "f16", "--outfile", gguf, out}, nil)
+		if err != nil {
+			return fmt.Errorf("converting to GGUF: %w", err)
+		}
+		if err := sendFile(gguf, "gguf", artifact); err != nil {
+			return err
+		}
+	}
+	var tgz bytes.Buffer
+	if err := tarGz(&tgz, out); err != nil {
+		return err
+	}
+	return sendBytes(tgz.Bytes(), "peft", artifact)
+}
+
+func str(v any) string {
+	s, _ := v.(string)
+	return s
+}
+
+func num(v any) int {
+	f, _ := v.(float64)
+	return int(f)
+}
+
+// python3 runs a Python script as the engines' user, passing each JSON line
+// it prints to line, and answers the last of what it wrote to stderr when it
+// fails.
+func (r *Runner) python3(ctx context.Context, py, dir string, env, args []string, line func(map[string]any)) error {
+	cmd := exec.CommandContext(ctx, py, args...)
+	cmd.Dir = dir
+	cmd.Env = append(os.Environ(), "PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True", "TOKENIZERS_PARALLELISM=false",
+		"HF_HUB_DISABLE_PROGRESS_BARS=1", "PYTHONUNBUFFERED=1")
+	cmd.Env = append(cmd.Env, env...)
+	home, give, why := supervisor.AsEngineUser(cmd, r.env.RunAs, r.env.WorkDir)
+	if why != "" {
+		r.env.Log.Warn(why)
+	}
+	if home != "" {
+		cmd.Env = append(cmd.Env, "HF_HOME="+filepath.Join(home, ".cache", "huggingface"))
+	} else {
+		cmd.Env = append(cmd.Env, "HF_HOME="+filepath.Join(r.env.WorkDir, "hf"))
+	}
+	if err := give(dir); err != nil {
+		return fmt.Errorf("handing %s to %s: %w", dir, r.env.RunAs, err)
+	}
+	// The whole process group goes when the job is cancelled: torch starts
+	// workers of its own.
+	if cmd.SysProcAttr == nil {
+		cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
+	}
+	cmd.Cancel = func() error { return syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL) }
+	stdout, err := cmd.StdoutPipe()
+	if err != nil {
+		return err
+	}
+	var stderr tailBuffer
+	cmd.Stderr = &stderr
+	if err := cmd.Start(); err != nil {
+		return err
+	}
+	sc := bufio.NewScanner(stdout)
+	sc.Buffer(make([]byte, 64<<10), 1<<20)
+	for sc.Scan() {
+		var m map[string]any
+		if line != nil && json.Unmarshal(sc.Bytes(), &m) == nil {
+			line(m)
+		}
+	}
+	err = cmd.Wait()
+	if ctx.Err() != nil {
+		return errors.New("cancelled")
+	}
+	if err != nil {
+		return fmt.Errorf("%v: %s", err, stderr.last())
+	}
+	return nil
+}
+
+// tailBuffer keeps the end of what a process wrote: the reason it failed is
+// at the end, and the rest is progress bars.
+type tailBuffer struct{ b []byte }
+
+func (t *tailBuffer) Write(p []byte) (int, error) {
+	t.b = append(t.b, p...)
+	if len(t.b) > 16<<10 {
+		t.b = t.b[len(t.b)-16<<10:]
+	}
+	return len(p), nil
+}
+
+func (t *tailBuffer) last() string {
+	lines := strings.Split(strings.TrimSpace(string(t.b)), "\n")
+	if len(lines) > 3 {
+		lines = lines[len(lines)-3:]
+	}
+	return strings.Join(lines, " | ")
+}
+
+// python is the newest installed vLLM's interpreter: its torch is built for
+// this node's CUDA.
+func (r *Runner) python() (string, error) {
+	if r.env.Python != "" {
+		return r.env.Python, nil
+	}
+	found, _ := filepath.Glob(filepath.Join(r.env.EnginesDir, "vllm", "*", "bin", "python3"))
+	if len(found) == 0 {
+		return "", errors.New("training needs vLLM installed on this node (its torch is what trains); " +
+			"llmgw ctl engine-build, or serving any vLLM model here, installs it")
+	}
+	sort.Strings(found)
+	return found[len(found)-1], nil
+}
+
+func (r *Runner) uv() (string, error) {
+	if r.env.UV != "" {
+		return r.env.UV, nil
+	}
+	var found []string
+	_ = filepath.WalkDir(filepath.Join(r.env.EnginesDir, "uv"), func(p string, d os.DirEntry, err error) error {
+		if err == nil && !d.IsDir() && d.Name() == "uv" {
+			found = append(found, p)
+		}
+		return nil
+	})
+	if len(found) == 0 {
+		return "", errors.New("no uv under " + filepath.Join(r.env.EnginesDir, "uv"))
+	}
+	sort.Strings(found)
+	return found[len(found)-1], nil
+}
+
+// overlay installs the pinned packages once per Python and set of pins.
+func (r *Runner) overlay(ctx context.Context, py string) (string, error) {
+	sum := sha256.Sum256([]byte(py + "\x00" + strings.Join(overlayPackages, " ")))
+	dir := filepath.Join(r.env.WorkDir, "overlay-"+hex.EncodeToString(sum[:6]))
+	if _, err := os.Stat(filepath.Join(dir, ".done")); err == nil {
+		return dir, nil
+	}
+	uv, err := r.uv()
+	if err != nil {
+		return "", err
+	}
+	tmp := dir + ".new"
+	_ = os.RemoveAll(tmp)
+	args := append([]string{"pip", "install", "--quiet", "--python", py, "--no-deps", "--target", tmp}, overlayPackages...)
+	cmd := exec.CommandContext(ctx, uv, args...)
+	if out, err := cmd.CombinedOutput(); err != nil {
+		return "", fmt.Errorf("%v: %s", err, strings.TrimSpace(string(out)))
+	}
+	_ = os.WriteFile(filepath.Join(tmp, ".done"), nil, 0o644)
+	_ = os.RemoveAll(dir)
+	if err := os.Rename(tmp, dir); err != nil {
+		return "", err
+	}
+	// Readable by the engines' user, which runs the trainer.
+	_ = filepath.WalkDir(dir, func(p string, d os.DirEntry, err error) error {
+		if err == nil {
+			mode := os.FileMode(0o644)
+			if d.IsDir() {
+				mode = 0o755
+			}
+			_ = os.Chmod(p, mode)
+		}
+		return nil
+	})
+	return dir, nil
+}
+
+// llamaCppTag is the release of the newest llama.cpp installed: its
+// conversion code is the one whose GGUF that llama.cpp reads.
+func (r *Runner) llamaCppTag() (string, error) {
+	found, _ := filepath.Glob(filepath.Join(r.env.EnginesDir, "llamacpp", "b*"))
+	if len(found) == 0 {
+		return "", errors.New("a GGUF adapter needs llama.cpp installed on this node, to match its release")
+	}
+	sort.Strings(found)
+	tag, _, _ := strings.Cut(filepath.Base(found[len(found)-1]), "-")
+	return tag, nil
+}
+
+// conversionSource is where llama.cpp's source is fetched from, by tag.
+var conversionSource = "https://codeload.github.com/ggml-org/llama.cpp/tar.gz/refs/tags/"
+
+// conversion fetches llama.cpp's LoRA conversion code for the installed
+// release, once.
+func (r *Runner) conversion(ctx context.Context) (string, error) {
+	if r.env.ConvDir != "" {
+		return r.env.ConvDir, nil
+	}
+	tag, err := r.llamaCppTag()
+	if err != nil {
+		return "", err
+	}
+	dir := filepath.Join(r.env.WorkDir, "llama.cpp-"+tag)
+	if _, err := os.Stat(filepath.Join(dir, "convert_lora_to_gguf.py")); err == nil {
+		return dir, nil
+	}
+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, conversionSource+tag, nil)
+	if err != nil {
+		return "", err
+	}
+	res, err := http.DefaultClient.Do(req)
+	if err != nil {
+		return "", err
+	}
+	defer res.Body.Close()
+	if res.StatusCode != http.StatusOK {
+		return "", fmt.Errorf("%s answered %s", conversionSource+tag, res.Status)
+	}
+	tmp := dir + ".new"
+	_ = os.RemoveAll(tmp)
+	if err := untarConversion(res.Body, tmp); err != nil {
+		return "", err
+	}
+	_ = os.RemoveAll(dir)
+	return dir, os.Rename(tmp, dir)
+}
+
+// untarConversion keeps, of llama.cpp's source, the conversion scripts and
+// the Python packages they import.
+func untarConversion(r io.Reader, dir string) error {
+	gz, err := gzip.NewReader(r)
+	if err != nil {
+		return err
+	}
+	tr := tar.NewReader(gz)
+	kept := 0
+	for {
+		h, err := tr.Next()
+		if errors.Is(err, io.EOF) {
+			break
+		}
+		if err != nil {
+			return err
+		}
+		_, rel, ok := strings.Cut(h.Name, "/")
+		if !ok || h.Typeflag != tar.TypeReg || strings.Contains(rel, "..") {
+			continue
+		}
+		if rel != "convert_lora_to_gguf.py" && rel != "convert_hf_to_gguf.py" &&
+			!strings.HasPrefix(rel, "conversion/") && !strings.HasPrefix(rel, "gguf-py/") {
+			continue
+		}
+		p := filepath.Join(dir, filepath.FromSlash(rel))
+		if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
+			return err
+		}
+		f, err := os.OpenFile(p, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o644)
+		if err != nil {
+			return err
+		}
+		_, err = io.Copy(f, io.LimitReader(tr, 64<<20))
+		f.Close()
+		if err != nil {
+			return err
+		}
+		kept++
+	}
+	if kept == 0 {
+		return errors.New("no conversion code in llama.cpp's source")
+	}
+	return nil
+}
+
+// pickGPU is the card with the most memory free, if that is enough.
+func (r *Runner) pickGPU(ctx context.Context, need int) (int, error) {
+	list := r.env.GPUs
+	if list == nil {
+		list = nvidiaSMI
+	}
+	gpus, err := list(ctx)
+	if err != nil {
+		return 0, fmt.Errorf("reading the cards' free memory: %w", err)
+	}
+	if len(gpus) == 0 {
+		return 0, errors.New("this node has no GPU to train on")
+	}
+	best := gpus[0]
+	for _, g := range gpus[1:] {
+		if g.FreeMiB > best.FreeMiB {
+			best = g
+		}
+	}
+	if best.FreeMiB < need {
+		return 0, fmt.Errorf("no card here has %d MiB free (the most free is GPU %d with %d MiB); "+
+			"a smaller base, or a card freed for it, would do", need, best.Index, best.FreeMiB)
+	}
+	return best.Index, nil
+}
+
+func nvidiaSMI(ctx context.Context) ([]GPU, error) {
+	ctx, cancel := context.WithTimeout(ctx, 15*time.Second)
+	defer cancel()
+	out, err := exec.CommandContext(ctx, "nvidia-smi", "--query-gpu=index,memory.free", "--format=csv,noheader,nounits").Output()
+	if err != nil {
+		return nil, err
+	}
+	return parseGPUs(string(out)), nil
+}
+
+func parseGPUs(s string) []GPU {
+	var out []GPU
+	for _, line := range strings.Split(strings.TrimSpace(s), "\n") {
+		a, b, ok := strings.Cut(line, ",")
+		if !ok {
+			continue
+		}
+		i, err1 := strconv.Atoi(strings.TrimSpace(a))
+		f, err2 := strconv.Atoi(strings.TrimSpace(b))
+		if err1 == nil && err2 == nil {
+			out = append(out, GPU{Index: i, FreeMiB: f})
+		}
+	}
+	return out
+}
+
+// tarGz packs a directory.
+func tarGz(w io.Writer, dir string) error {
+	gz := gzip.NewWriter(w)
+	tw := tar.NewWriter(gz)
+	err := filepath.WalkDir(dir, func(p string, d os.DirEntry, err error) error {
+		if err != nil || d.IsDir() {
+			return err
+		}
+		rel, _ := filepath.Rel(dir, p)
+		info, err := d.Info()
+		if err != nil {
+			return err
+		}
+		if err := tw.WriteHeader(&tar.Header{Name: filepath.ToSlash(rel), Mode: 0o644, Size: info.Size(), ModTime: info.ModTime()}); err != nil {
+			return err
+		}
+		f, err := os.Open(p)
+		if err != nil {
+			return err
+		}
+		defer f.Close()
+		_, err = io.Copy(tw, f)
+		return err
+	})
+	if err != nil {
+		return err
+	}
+	if err := tw.Close(); err != nil {
+		return err
+	}
+	return gz.Close()
+}
+
+// artifactChunk is how much of an artifact goes in one message.
+const artifactChunk = 1 << 20
+
+func sendFile(path, kind string, send func(string, int, []byte, bool) error) error {
+	b, err := os.ReadFile(path)
+	if err != nil {
+		return err
+	}
+	return sendBytes(b, kind, send)
+}
+
+func sendBytes(b []byte, kind string, send func(string, int, []byte, bool) error) error {
+	seq := 0
+	for {
+		n := min(artifactChunk, len(b))
+		last := n == len(b)
+		if err := send(kind, seq, b[:n], last); err != nil {
+			return err
+		}
+		if last {
+			return nil
+		}
+		b = b[n:]
+		seq++
+	}
+}

internal/agent/train/runner_test.go

@@ -0,0 +1,164 @@
+package train
+
+import (
+	"archive/tar"
+	"bytes"
+	"compress/gzip"
+	"context"
+	"io"
+	"os"
+	"path/filepath"
+	"strings"
+	"sync"
+	"testing"
+)
+
+// fakePython stands in for an engine's interpreter: as the trainer it
+// prints progress and writes an adapter; as the converter it writes a GGUF;
+// told to fail, it fails.
+const fakePython = `#!/bin/sh
+case "$1" in
+*trainer.py)
+  out=$(sed -n 's/.*"out": *"\([^"]*\)".*/\1/p' "$2")
+  grep -q FAIL "$(dirname "$2")/data.jsonl" && { echo "CUDA out of memory" >&2; exit 1; }
+  echo '{"state":"preparing","message":"fetching base"}'
+  echo '{"state":"running","step":5,"steps":10,"loss":0.9,"peak_mib":1900}'
+  mkdir -p "$out"; echo '{"r":16}' > "$out/adapter_config.json"; echo weights > "$out/adapter_model.safetensors"
+  echo '{"state":"trained","step":10,"steps":10,"peak_mib":2100,"base_dir":"/base"}'
+  ;;
+*convert_lora_to_gguf.py)
+  while [ $# -gt 0 ]; do [ "$1" = "--outfile" ] && echo GGUF > "$2"; shift; done
+  ;;
+esac
+`
+
+func runner(t *testing.T, free int) *Runner {
+	t.Helper()
+	dir := t.TempDir()
+	py := filepath.Join(dir, "python3")
+	_ = os.WriteFile(py, []byte(fakePython), 0o755)
+	uv := filepath.Join(dir, "uv")
+	_ = os.WriteFile(uv, []byte("#!/bin/sh\nwhile [ $# -gt 0 ]; do [ \"$1\" = \"--target\" ] && mkdir -p \"$2\"; shift; done\n"), 0o755)
+	return New(Env{WorkDir: filepath.Join(dir, "work"), Python: py, UV: uv, ConvDir: dir,
+		GPUs: func(context.Context) ([]GPU, error) { return []GPU{{0, 500}, {1, free}}, nil }})
+}
+
+type sink struct {
+	mu       sync.Mutex
+	statuses []Status
+	parts    map[string][]byte
+	lasts    int
+}
+
+func (s *sink) status(st Status) { s.mu.Lock(); s.statuses = append(s.statuses, st); s.mu.Unlock() }
+func (s *sink) artifact(kind string, _ int, b []byte, last bool) error {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	if s.parts == nil {
+		s.parts = map[string][]byte{}
+	}
+	s.parts[kind] = append(s.parts[kind], b...)
+	if last {
+		s.lasts++
+	}
+	return nil
+}
+
+func TestAJobTrainsOnTheFreestCardAndSendsWhatItMade(t *testing.T) {
+	r := runner(t, 3000)
+	s := &sink{}
+	err := r.Run(context.Background(), Job{ID: "j1", BaseRepo: "Qwen/x", Data: []byte(`{"q":"a","a":"b"}`), GPUMemoryMiB: 2500, GGUF: true},
+		s.status, s.artifact)
+	if err != nil {
+		t.Fatal(err)
+	}
+	var running bool
+	for _, st := range s.statuses {
+		if st.State == "running" && st.Step == 5 && st.Steps == 10 && st.PeakMiB == 1900 {
+			running = true
+		}
+	}
+	if !running {
+		t.Errorf("statuses %+v", s.statuses)
+	}
+	if string(s.parts["gguf"]) != "GGUF\n" || s.lasts != 2 {
+		t.Errorf("gguf %q, lasts %d", s.parts["gguf"], s.lasts)
+	}
+	files := map[string]string{}
+	gz, err := gzip.NewReader(bytes.NewReader(s.parts["peft"]))
+	if err != nil {
+		t.Fatal(err)
+	}
+	tr := tar.NewReader(gz)
+	for {
+		h, err := tr.Next()
+		if err != nil {
+			break
+		}
+		b, _ := io.ReadAll(tr)
+		files[h.Name] = string(b)
+	}
+	if files["adapter_model.safetensors"] != "weights\n" || files["adapter_config.json"] == "" {
+		t.Errorf("peft archive %v", files)
+	}
+	// Nothing of the job is left on the node.
+	if entries, _ := os.ReadDir(filepath.Join(r.env.WorkDir, "jobs")); len(entries) != 0 {
+		t.Errorf("left behind: %v", entries)
+	}
+}
+
+func TestAJobWaitsForNoCardItDoesNotFit(t *testing.T) {
+	r := runner(t, 1000)
+	err := r.Run(context.Background(), Job{ID: "j1", Data: []byte(`{"q":"a","a":"b"}`), GPUMemoryMiB: 2500}, (&sink{}).status, (&sink{}).artifact)
+	if err == nil || !strings.Contains(err.Error(), "no card here has 2500 MiB free") {
+		t.Errorf("got %v", err)
+	}
+}
+
+func TestATrainerThatFailsSaysWhy(t *testing.T) {
+	r := runner(t, 3000)
+	err := r.Run(context.Background(), Job{ID: "j1", Data: []byte(`{"q":"FAIL","a":"b"}`)}, (&sink{}).status, (&sink{}).artifact)
+	if err == nil || !strings.Contains(err.Error(), "CUDA out of memory") {
+		t.Errorf("got %v", err)
+	}
+}
+
+func TestOneJobAtATime(t *testing.T) {
+	r := runner(t, 3000)
+	r.job = "other"
+	if err := r.Run(context.Background(), Job{ID: "j2"}, (&sink{}).status, (&sink{}).artifact); err != ErrBusy {
+		t.Errorf("got %v", err)
+	}
+}
+
+func TestCardsAreReadFromNvidiaSMI(t *testing.T) {
+	got := parseGPUs("0, 2310\n1, 80000\nbad line\n")
+	if len(got) != 2 || got[1].FreeMiB != 80000 || got[0].Index != 0 {
+		t.Errorf("parsed %+v", got)
+	}
+}
+
+func TestOnlyTheConversionCodeIsKeptOfLlamaCpp(t *testing.T) {
+	var buf bytes.Buffer
+	gz := gzip.NewWriter(&buf)
+	tw := tar.NewWriter(gz)
+	for _, name := range []string{"llama.cpp-b1/convert_lora_to_gguf.py", "llama.cpp-b1/conversion/qwen.py",
+		"llama.cpp-b1/gguf-py/gguf/__init__.py", "llama.cpp-b1/src/llama.cpp", "llama.cpp-b1/../evil"} {
+		_ = tw.WriteHeader(&tar.Header{Name: name, Mode: 0o644, Size: 1, Typeflag: tar.TypeReg})
+		_, _ = tw.Write([]byte("x"))
+	}
+	_ = tw.Close()
+	_ = gz.Close()
+	dir := filepath.Join(t.TempDir(), "conv")
+	if err := untarConversion(&buf, dir); err != nil {
+		t.Fatal(err)
+	}
+	for _, want := range []string{"convert_lora_to_gguf.py", "conversion/qwen.py", "gguf-py/gguf/__init__.py"} {
+		if _, err := os.Stat(filepath.Join(dir, want)); err != nil {
+			t.Errorf("%s not kept", want)
+		}
+	}
+	if _, err := os.Stat(filepath.Join(dir, "src")); err == nil {
+		t.Error("the rest of llama.cpp was kept")
+	}
+}

internal/agent/train/trainer.py

@@ -0,0 +1,101 @@
+# llmgw's LoRA trainer (BACKLOG F4), run by the node agent with an installed
+# engine's Python (its torch and transformers) and a small overlay (peft,
+# accelerate). Reads a job, trains, writes the PEFT adapter, and says how it
+# goes as one JSON object a line on standard output.
+import json, math, os, random, sys, threading, subprocess, time
+
+def say(**kw):
+    print(json.dumps(kw), flush=True)
+
+job = json.load(open(sys.argv[1], encoding="utf-8"))
+p = {"rank": 16, "alpha": 32, "dropout": 0.05, "epochs": 6, "lr": 2e-4, "batch": 4, "max_len": 512, "seed": 0}
+p.update(job.get("params") or {})
+
+say(state="preparing", message="loading libraries")
+import torch
+from huggingface_hub import snapshot_download
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from peft import LoraConfig, get_peft_model
+
+random.seed(p["seed"]); torch.manual_seed(p["seed"])
+if not torch.cuda.is_available():
+    raise SystemExit("no CUDA device is visible to the trainer")
+
+say(state="preparing", message="fetching " + job["base_repo"])
+base_dir = snapshot_download(job["base_repo"], revision=job.get("base_revision") or None,
+                             allow_patterns=["*.json", "*.safetensors", "*.txt", "*.model", "*.tiktoken"])
+say(state="preparing", message="loading the base model", base_dir=base_dir)
+
+# bf16 where the card has it; fp16 with a loss scaler where it does not
+# (Turing and older).
+bf16 = torch.cuda.is_bf16_supported()
+dtype = torch.bfloat16 if bf16 else torch.float16
+tok = AutoTokenizer.from_pretrained(base_dir)
+if tok.pad_token_id is None:
+    tok.pad_token = tok.eos_token
+model = AutoModelForCausalLM.from_pretrained(base_dir, dtype=dtype).to("cuda")
+
+rows = [json.loads(l) for l in open(job["data"], encoding="utf-8") if l.strip()]
+rows = [r for r in rows if r.get("q") and r.get("a")]
+if not rows:
+    raise SystemExit("no {q, a} pairs in the data")
+
+def encode(r):
+    prompt = tok.apply_chat_template([{"role": "user", "content": r["q"]}], add_generation_prompt=True, tokenize=False)
+    full = tok.apply_chat_template([{"role": "user", "content": r["q"]}, {"role": "assistant", "content": r["a"]}], tokenize=False)
+    pi = tok(prompt, add_special_tokens=False)["input_ids"]
+    fi = tok(full, add_special_tokens=False)["input_ids"][: p["max_len"]]
+    return fi, ([-100] * len(pi) + fi[len(pi):])[: len(fi)]
+
+data = [encode(r) for r in rows]
+
+model.gradient_checkpointing_enable()
+model.enable_input_require_grads()
+model = get_peft_model(model, LoraConfig(r=int(p["rank"]), lora_alpha=int(p["alpha"]), lora_dropout=float(p["dropout"]),
+    task_type="CAUSAL_LM", target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]))
+trainable = [x for x in model.parameters() if x.requires_grad]
+for x in trainable:  # fp32 trainable weights, so a loss scaler can unscale them
+    x.data = x.data.float()
+
+peak = {"mib": 0}
+stop = threading.Event()
+def watch():
+    while not stop.is_set():
+        try:
+            out = subprocess.run(["nvidia-smi", "--query-compute-apps=pid,used_memory", "--format=csv,noheader,nounits"],
+                                 capture_output=True, text=True, timeout=5).stdout
+            for line in out.splitlines():
+                pid, mib = [v.strip() for v in line.split(",")]
+                if int(pid) == os.getpid():
+                    peak["mib"] = max(peak["mib"], int(mib))
+        except Exception:
+            pass
+        time.sleep(0.5)
+threading.Thread(target=watch, daemon=True).start()
+
+opt = torch.optim.AdamW(trainable, lr=float(p["lr"]), weight_decay=0.0)
+scaler = torch.amp.GradScaler("cuda", enabled=not bf16)
+bs, epochs = int(p["batch"]), int(p["epochs"])
+steps = epochs * math.ceil(len(data) / bs)
+warm = max(1, steps // 10)
+sched = torch.optim.lr_scheduler.LambdaLR(opt, lambda s: min(1.0, (s + 1) / warm) * max(0.0, 1 - s / steps))
+model.train(); step = 0; last = time.time()
+for ep in range(epochs):
+    random.shuffle(data)
+    for i in range(0, len(data), bs):
+        batch = data[i:i + bs]; n = max(len(x[0]) for x in batch)
+        ids = torch.tensor([x[0] + [tok.pad_token_id] * (n - len(x[0])) for x in batch], device="cuda")
+        lab = torch.tensor([x[1] + [-100] * (n - len(x[1])) for x in batch], device="cuda")
+        att = torch.tensor([[1] * len(x[0]) + [0] * (n - len(x[0])) for x in batch], device="cuda")
+        with torch.autocast("cuda", dtype=dtype):
+            loss = model(input_ids=ids, attention_mask=att, labels=lab).loss
+        scaler.scale(loss).backward(); scaler.unscale_(opt)
+        torch.nn.utils.clip_grad_norm_(trainable, 1.0)
+        scaler.step(opt); scaler.update(); opt.zero_grad(set_to_none=True); sched.step(); step += 1
+        if time.time() - last > 2 or step == steps:
+            last = time.time()
+            say(state="running", message="training", step=step, steps=steps, loss=round(loss.item(), 4), peak_mib=peak["mib"])
+
+model.save_pretrained(job["out"])
+stop.set()
+say(state="trained", step=step, steps=steps, peak_mib=peak["mib"], base_dir=base_dir)

internal/agent/trainjob.go

@@ -0,0 +1,46 @@
+package agent
+
+import (
+	"context"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/train"
+)
+
+// trainJob runs a training job the control plane sent, telling it how the
+// job goes and sending what it made. A stream that ends ends the job: the
+// control plane, which sees the node go, marks it failed, and it is started
+// again from the beginning rather than resumed.
+func (a *Agent) trainJob(ctx context.Context, m *llmgwv1.TrainStart, out chan<- *llmgwv1.AgentMessage) {
+	id := m.GetJobId()
+	send := func(msg *llmgwv1.AgentMessage) bool {
+		select {
+		case out <- msg:
+			return true
+		case <-ctx.Done():
+			return false
+		}
+	}
+	status := func(s train.Status) {
+		send(&llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_TrainStatus{TrainStatus: &llmgwv1.TrainStatus{
+			JobId: id, State: s.State, Message: s.Message, Step: int32(s.Step), Steps: int32(s.Steps), //nolint:gosec // counts
+			Loss: s.Loss, PeakMib: int32(s.PeakMiB)}}}) //nolint:gosec // MiB
+	}
+	artifact := func(kind string, seq int, data []byte, last bool) error {
+		if !send(&llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_TrainArtifact{TrainArtifact: &llmgwv1.TrainArtifact{
+			JobId: id, Kind: kind, Seq: int32(seq), Data: data, Last: last}}}) { //nolint:gosec // chunk count
+			return ctx.Err()
+		}
+		return nil
+	}
+	a.log.Info("training an adapter", "job", id, "base", m.GetBaseRepo())
+	err := a.trainer.Run(ctx, train.Job{ID: id, BaseRepo: m.GetBaseRepo(), BaseRevision: m.GetBaseRevision(),
+		Data: m.GetData(), Params: m.GetParams(), GPUMemoryMiB: int(m.GetGpuMemoryMib()), GGUF: m.GetGguf()}, status, artifact)
+	if err != nil {
+		a.log.Warn("training failed", "job", id, "err", err)
+		status(train.Status{State: "failed", Message: err.Error()})
+		return
+	}
+	a.log.Info("trained an adapter", "job", id)
+	status(train.Status{State: "done"})
+}

internal/cli/agent.go

@@ -40,10 +40,23 @@ func newAgentCmd() *cobra.Command {
 		RunE: func(cmd *cobra.Command, _ []string) error {
 			log := logger()
 
-			cfg, err := config.LoadAgent(configPath)
+			// The cluster's settings for this agent over its file, as the
+			// control plane last sent them. Settings that no longer load
+			// beside the file are set aside rather than keeping the agent
+			// from starting: the control plane sends them again, and says
+			// why they are refused.
+			settings := agent.ReadSettings()
+			cfg, err := config.LoadAgentWith(configPath, settings)
+			if err != nil && settings != nil {
+				log.Warn("the cluster's settings kept for this agent do not load beside its agent.yaml; starting without them",
+					"err", err)
+				settings = nil
+				cfg, err = config.LoadAgent(configPath)
+			}
 			if err != nil {
 				return err
 			}
+			cfg.Settings = string(settings)
 			if server != "" {
 				cfg.Server = server
 			}
@@ -89,9 +102,17 @@ func newAgentCmd() *cobra.Command {
 			if err != nil {
 				return err
 			}
+			a.SetSettingsCheck(func(b []byte) error {
+				_, err := config.LoadAgentWith(configPath, b)
+				return err
+			})
 
 			ctx, stop := signal.NotifyContext(cmd.Context(), syscall.SIGINT, syscall.SIGTERM)
 			defer stop()
+			go a.WatchFile(ctx, configPath, func() error {
+				_, err := config.LoadAgentWith(configPath, agent.ReadSettings())
+				return err
+			})
 
 			log.Info("agent starting",
 				"node", cfg.NodeID, "server", cfg.Server,

internal/cli/backup.go

@@ -249,7 +249,14 @@ func pgEnv(dsn string) ([]string, error) {
 		set("PGPASSWORD", pw)
 	}
 	set("PGDATABASE", strings.TrimPrefix(u.Path, "/"))
-	set("PGSSLMODE", u.Query().Get("sslmode"))
+	// The TLS settings go with the mode: verify-full without the cluster's
+	// authority sent libpq looking for ~/.postgresql/root.crt, and the dump
+	// failed on every control plane whose store is replicated.
+	q := u.Query()
+	set("PGSSLMODE", q.Get("sslmode"))
+	set("PGSSLROOTCERT", q.Get("sslrootcert"))
+	set("PGSSLCERT", q.Get("sslcert"))
+	set("PGSSLKEY", q.Get("sslkey"))
 	// A standby if there is one, else whichever answers — not the address's
 	// read-write: a dump only reads, it need not load the primary, and one
 	// that insists on the primary fails exactly while the cluster is failing

internal/cli/backup_test.go

@@ -238,3 +238,17 @@ func TestAStoreOfSeveralHostsIsGivenToPgDumpAsLibpqLists(t *testing.T) {
 		t.Error("an empty host in the list was accepted")
 	}
 }
+
+func TestPgDumpIsGivenTheClusterAuthorityAlongWithVerifyFull(t *testing.T) {
+	// Without it libpq looked for ~/.postgresql/root.crt and the dump failed.
+	env, err := pgEnv("postgres://llmgw:p@10.0.0.149:5432,10.0.0.150:5432/llmgw?sslmode=verify-full&sslrootcert=%2Fetc%2Fllmgw%2Fpki%2Fca.crt&target_session_attrs=read-write")
+	if err != nil {
+		t.Fatal(err)
+	}
+	got := strings.Join(env, " ")
+	for _, want := range []string{"PGSSLMODE=verify-full", "PGSSLROOTCERT=/etc/llmgw/pki/ca.crt"} {
+		if !strings.Contains(got, want) {
+			t.Errorf("%s is not in %s", want, got)
+		}
+	}
+}

internal/cli/clustersettings.go

@@ -241,6 +241,8 @@ var liveSettings = []string{
 	"serving",
 	// Sent to the nodes and this machine's firewall unit as they change.
 	"firewall", "firewall_networks",
+	// Sent to the nodes' agents as they change.
+	"agents",
 	// Read by server init, node add and backups, not by the server.
 	"backup_recipients", "backup_copy_to",
 }

internal/cli/ctl.go

@@ -116,6 +116,11 @@ func (c *ctlClient) do(ctx context.Context, method, path string, body, out any)
 	if out == nil {
 		return nil
 	}
+	// A stream, as it comes: NDJSON a long run writes line by line.
+	if w, ok := out.(io.Writer); ok {
+		_, err := io.Copy(w, resp.Body)
+		return err
+	}
 	// An older control plane answers a path it does not know with its
 	// dashboard: a page, which read as JSON said only "invalid character '<'".
 	if strings.HasPrefix(resp.Header.Get("Content-Type"), "text/html") {
@@ -181,6 +186,7 @@ func newCtlCmd() *cobra.Command {
 		newQuotaCmd(client, &asJSON),
 		newGuardCmd(client, &asJSON),
 		newKBCmd(client, &asJSON),
+		newTrainCmd(client, &asJSON),
 		newConfigCmd(client, &asJSON),
 		newAuditCmd(client, &asJSON),
 		newNotifyCmd(client, &asJSON),

internal/cli/firewallcmd.go

@@ -7,12 +7,14 @@ import (
 	"fmt"
 	"net/netip"
 	"os"
+	"strings"
 	"time"
 
 	"github.com/spf13/cobra"
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
 )
 
 // firewallStateFile is where a control plane writes what the cluster's
@@ -20,11 +22,9 @@ import (
 // the server itself runs as llmgw and cannot change the ruleset.
 const firewallStateFile = "/var/lib/llmgw/firewall.json"
 
-// firewallState is what is written there.
-type firewallState struct {
-	Enabled  bool     `json:"enabled"`
-	Networks []string `json:"networks,omitempty"`
-}
+// firewallState is what is written there: the same as the mirror is told
+// and keeps beside its models.
+type firewallState = mirror.FirewallState
 
 // writeFirewallState records the settings' firewall for this machine's sync
 // unit. Without the state directory — a server not run by systemd — there
@@ -46,25 +46,34 @@ func writeFirewallState(cfg config.Server) error {
 
 // The units that apply it: a path unit watching the file, and the oneshot it
 // starts.
-const (
-	firewallSyncPathUnit = "llmgw-firewall-sync.path"
-	firewallSyncUnitFile = `[Unit]
-Description=llmgw firewall: apply the cluster's settings on this control plane
+const firewallSyncPathUnit = "llmgw-firewall-sync.path"
+
+// firewallSyncUnits are the two units for a state file: on a control plane
+// the one its server writes, on a mirror the one the control plane has the
+// mirror write.
+func firewallSyncUnits(state string) (service, pathUnit string) {
+	arg := ""
+	if state != firewallStateFile {
+		arg = " --state " + state
+	}
+	service = `[Unit]
+Description=llmgw firewall: apply the cluster's settings on this machine
 
 [Service]
 Type=oneshot
-ExecStart=/usr/local/bin/llmgw firewall sync
+ExecStart=/usr/local/bin/llmgw firewall sync` + arg + `
 `
-	firewallSyncPathFile = `[Unit]
-Description=llmgw firewall: follow the cluster's settings on this control plane
+	pathUnit = `[Unit]
+Description=llmgw firewall: follow the cluster's settings on this machine
 
 [Path]
-PathChanged=` + firewallStateFile + `
+PathChanged=` + state + `
 
 [Install]
 WantedBy=multi-user.target
 `
-)
+	return service, pathUnit
+}
 
 func newFirewallCmd() *cobra.Command {
 	c := &cobra.Command{
@@ -76,6 +85,7 @@ func newFirewallCmd() *cobra.Command {
 			"which " + firewallSyncPathUnit + " applies with 'llmgw firewall sync'.",
 	}
 	var install bool
+	state := firewallStateFile
 	sync := &cobra.Command{
 		Use:   "sync",
 		Short: "Apply what the cluster's settings say of the firewall, here (root)",
@@ -85,21 +95,21 @@ func newFirewallCmd() *cobra.Command {
 				return errors.New("the firewall is changed as root: run it with sudo")
 			}
 			if install {
-				if err := installFirewallSync(cmd.Context(), firewall.Local{}); err != nil {
+				if err := installFirewallSync(cmd.Context(), firewall.Local{}, state); err != nil {
 					return err
 				}
 				fmt.Fprintln(cmd.OutOrStdout(), firewallSyncPathUnit+" follows the cluster's settings here")
-				if _, err := os.Stat(firewallStateFile); err != nil {
-					return nil // the server writes it when it starts
+				if _, err := os.Stat(state); err != nil {
+					return nil // written when the settings are first told
 				}
 			}
-			b, err := os.ReadFile(firewallStateFile)
+			b, err := os.ReadFile(state)
 			if err != nil {
 				return err
 			}
 			var st firewallState
 			if err := json.Unmarshal(b, &st); err != nil {
-				return fmt.Errorf("%s: %w", firewallStateFile, err)
+				return fmt.Errorf("%s: %w", state, err)
 			}
 			ctx, cancel := context.WithTimeout(cmd.Context(), 2*time.Minute)
 			defer cancel()
@@ -112,14 +122,17 @@ func newFirewallCmd() *cobra.Command {
 		},
 	}
 	sync.Flags().BoolVar(&install, "install", false, "first install the units that run this whenever the settings change")
+	sync.Flags().StringVar(&state, "state", firewallStateFile,
+		"the file the settings are written to: a control plane's, or "+mirror.FirewallFile+" in a mirror's directory")
 	c.AddCommand(sync)
 	return c
 }
 
 // installFirewallSync writes the sync units on a machine and has the path
 // unit follow the state file.
-func installFirewallSync(ctx context.Context, r firewall.Runner) error {
-	body := firewallSyncUnitFile + "\x00" + firewallSyncPathFile
+func installFirewallSync(ctx context.Context, r firewall.Runner, state string) error {
+	service, pathUnit := firewallSyncUnits(state)
+	body := service + "\x00" + pathUnit
 	out, err := r.Run(ctx, `set -e
 t=$(mktemp -d); trap 'rm -rf "$t"' EXIT
 cat > "$t/all"
@@ -172,5 +185,20 @@ func syncFirewall(ctx context.Context, r firewall.Runner, st firewallState) (str
 			return "", err
 		}
 	}
+	// The cluster's machines: a node on a network this one has not let in
+	// yet, which the mirror's files are for.
+	if len(st.Toward) > 0 {
+		added, err := firewall.Admit(ctx, r, st.Toward)
+		if err != nil {
+			return "", err
+		}
+		if len(added) > 0 {
+			shown := make([]string, len(added))
+			for i, p := range added {
+				shown[i] = p.String()
+			}
+			return "llmgw's firewall is on here, and now lets in " + strings.Join(shown, ", "), nil
+		}
+	}
 	return "llmgw's firewall is on here", nil
 }

internal/cli/firewallcmd_test.go

@@ -2,8 +2,11 @@ package cli
 
 import (
 	"context"
+	"net/netip"
 	"strings"
 	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
 )
 
 type scriptLog struct {
@@ -49,3 +52,32 @@ func TestAControlPlaneFollowsTheFirewallSetting(t *testing.T) {
 		t.Errorf("on: %q %v", msg, err)
 	}
 }
+
+func TestAMachineLetsInTheClusterMachinesItIsTold(t *testing.T) {
+	// The rules hold 10.0.0.0/24; a node on 192.168.7.0/24 is let in, as
+	// the route to it says, and one already in is not asked about.
+	r := &scriptLog{reply: func(s string) string {
+		switch {
+		case strings.Contains(s, "echo loaded"):
+			return "loaded\n"
+		case strings.Contains(s, "cat /etc/llmgw/firewall.nft"):
+			return firewallRulesFor("10.0.0.0/24")
+		case strings.Contains(s, "ip -o route get 192.168.7.20"):
+			return "192.168.7.20 dev eth1 src 192.168.7.1 uid 0\n2: eth1    inet 192.168.7.1/24 brd 192.168.7.255 scope global eth1\n"
+		case strings.Contains(s, "nft -f"):
+			return "changed\n"
+		}
+		return ""
+	}}
+	msg, err := syncFirewall(context.Background(), r, firewallState{Enabled: true, Toward: []string{"10.0.0.147", "192.168.7.20"}})
+	if err != nil || !strings.Contains(msg, "192.168.7.0/24") {
+		t.Fatalf("%q %v", msg, err)
+	}
+	if strings.Contains(strings.Join(r.scripts, "\n"), "route get 10.0.0.147") {
+		t.Error("asked the route to a machine already let in")
+	}
+}
+
+func firewallRulesFor(nets string) string {
+	return firewall.Script(firewall.Spec{Ports: []firewall.Port{firewall.MirrorFiles}, Allow: []netip.Prefix{netip.MustParsePrefix(nets)}})
+}

internal/cli/firewallpub.go

@@ -0,0 +1,146 @@
+package cli
+
+import (
+	"context"
+	"encoding/json"
+	"log/slog"
+	"net"
+	"net/netip"
+	"net/url"
+	"slices"
+	"sync"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// firewallStateKey is where, in the store, the firewall's state is kept
+// for the store's members to read.
+const firewallStateKey = "firewall"
+
+// firewallPublisher tells the machines that cannot ask a control plane
+// what the cluster's settings say of their firewall: the mirror, over its
+// API, and the store's members, through the store they hold a copy of.
+// Nodes are told over their agents' streams and control planes by their
+// own servers. On the leader only, at start, at every change to the
+// firewall's settings, and every few minutes for nodes come and gone.
+type firewallPublisher struct {
+	settings func() config.Server
+	st       *store.DB
+	nodes    func() []string
+	leader   func() bool
+	log      *slog.Logger
+
+	mu     sync.Mutex
+	mirror *mirror.Client
+	kick   chan struct{}
+}
+
+func newFirewallPublisher(settings func() config.Server, st *store.DB, nodes func() []string,
+	leader func() bool, log *slog.Logger) *firewallPublisher {
+	return &firewallPublisher{settings: settings, st: st, nodes: nodes, leader: leader, log: log,
+		kick: make(chan struct{}, 1)}
+}
+
+// setMirror has the mirror told too.
+func (p *firewallPublisher) setMirror(m *mirror.Client) {
+	p.mu.Lock()
+	p.mirror = m
+	p.mu.Unlock()
+	p.Kick()
+}
+
+// Kick has the state worked out and told again now.
+func (p *firewallPublisher) Kick() {
+	select {
+	case p.kick <- struct{}{}:
+	default:
+	}
+}
+
+func (p *firewallPublisher) run(ctx context.Context, every time.Duration) {
+	t := time.NewTicker(every)
+	defer t.Stop()
+	for {
+		p.once(ctx)
+		select {
+		case <-ctx.Done():
+			return
+		case <-t.C:
+		case <-p.kick:
+		}
+	}
+}
+
+// state is what the settings say now, with the cluster's machines.
+func (p *firewallPublisher) state(ctx context.Context) mirror.FirewallState {
+	cfg := p.settings()
+	st := mirror.FirewallState{Enabled: cfg.Firewall.Enabled, Networks: slices.Clone(cfg.FirewallNetworks)}
+	var toward []string
+	if p.nodes != nil {
+		toward = append(toward, p.nodes()...)
+	}
+	if p.st != nil {
+		if cps, err := p.st.ControlPlanes(ctx); err == nil {
+			for _, c := range cps {
+				toward = append(toward, c.URL)
+			}
+		}
+	}
+	for _, t := range toward {
+		if a, ok := hostAddr(t); ok {
+			st.Toward = append(st.Toward, a)
+		}
+	}
+	slices.Sort(st.Toward)
+	st.Toward = slices.Compact(st.Toward)
+	return st
+}
+
+// hostAddr is the address in an address, host:port or URL; names are not
+// looked up here.
+func hostAddr(s string) (string, bool) {
+	if u, err := url.Parse(s); err == nil && u.Host != "" {
+		s = u.Hostname()
+	} else if h, _, err := net.SplitHostPort(s); err == nil {
+		s = h
+	}
+	a, err := netip.ParseAddr(s)
+	if err != nil {
+		return "", false
+	}
+	return a.Unmap().String(), true
+}
+
+func (p *firewallPublisher) once(ctx context.Context) {
+	if p.leader != nil && !p.leader() {
+		return
+	}
+	ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
+	defer cancel()
+	st := p.state(ctx)
+	p.mu.Lock()
+	m := p.mirror
+	p.mu.Unlock()
+	if m != nil {
+		changed, err := m.SetFirewall(ctx, st)
+		switch {
+		case err != nil:
+			p.log.Warn("could not tell the mirror what the settings say of its firewall", "err", err)
+		case changed:
+			p.log.Info("the mirror's firewall follows the cluster's settings", "enabled", st.Enabled)
+		}
+	}
+	if p.st != nil {
+		b, _ := json.Marshal(st)
+		changed, err := p.st.PutClusterState(ctx, firewallStateKey, string(b))
+		switch {
+		case err != nil:
+			p.log.Warn("could not record the firewall's state for the store's members", "err", err)
+		case changed:
+			p.log.Info("the store's members' firewall follows the cluster's settings", "enabled", st.Enabled)
+		}
+	}
+}

internal/cli/firewallpub_test.go

@@ -0,0 +1,59 @@
+package cli
+
+import (
+	"context"
+	"encoding/json"
+	"log/slog"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
+)
+
+func TestTheLeaderTellsTheMirrorAndTheStoreWhatTheFirewallSettingsSay(t *testing.T) {
+	ctx := context.Background()
+	st, err := store.OpenSQLite(ctx, filepath.Join(t.TempDir(), "s.db"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer st.Close()
+	if err := st.PutControlPlane(ctx, store.ControlPlane{URL: "https://10.0.0.150:7443", Name: "ctl02"}); err != nil {
+		t.Fatal(err)
+	}
+	root := t.TempDir()
+	srv := httptest.NewServer(mirror.NewAPI(mirror.NewManager(root, slog.New(slog.DiscardHandler)), "http://m:8081").Handler())
+	defer srv.Close()
+	mc, _ := mirror.NewClient(srv.URL, tlsconf.Config{}, 0)
+
+	cfg := config.Server{Firewall: config.FirewallSettings{Enabled: false}, FirewallNetworks: []string{"192.168.5.0/24"}}
+	leader := false
+	p := newFirewallPublisher(func() config.Server { return cfg }, st,
+		func() []string { return []string{"10.0.0.147", "", "10.0.0.147", "node-by-name"} },
+		func() bool { return leader }, slog.New(slog.DiscardHandler))
+	p.setMirror(mc)
+
+	p.once(ctx)
+	if _, err := st.ClusterState(ctx, firewallStateKey); err == nil {
+		t.Error("a standby told the store")
+	}
+	leader = true
+	p.once(ctx)
+	var got mirror.FirewallState
+	v, err := st.ClusterState(ctx, firewallStateKey)
+	if err != nil || json.Unmarshal([]byte(v), &got) != nil {
+		t.Fatalf("the store holds %q %v", v, err)
+	}
+	if got.Enabled || got.Networks[0] != "192.168.5.0/24" || len(got.Toward) != 2 ||
+		got.Toward[0] != "10.0.0.147" || got.Toward[1] != "10.0.0.150" {
+		t.Errorf("state %+v", got)
+	}
+	b, err := os.ReadFile(filepath.Join(root, mirror.FirewallFile))
+	if err != nil || string(b) != v {
+		t.Errorf("the mirror holds %s, the store %s", b, v)
+	}
+}

internal/cli/kb.go

@@ -1,10 +1,14 @@
 package cli
 
 import (
+	"bytes"
 	"encoding/json"
 	"fmt"
+	"io"
 	"net/url"
+	"os"
 	"strings"
+	"time"
 
 	"github.com/spf13/cobra"
 
@@ -212,6 +216,45 @@ func newKBCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 	def.Flags().BoolVar(&clear, "clear", false, "no default any more")
 	c.AddCommand(def)
 
+	var dsModel, dsOut string
+	var dsPer, dsMax int
+	dataset := &cobra.Command{
+		Use:   "dataset COLLECTION --model MODEL",
+		Short: "Write training data from a collection: questions and answers, for an adapter",
+		Long: "A chat model of the cluster reads each chunk of the collection and writes\n" +
+			"questions a person might ask of it, with the answers it holds, one JSON object\n" +
+			"a line ({\"q\", \"a\", \"doc\", \"ord\"}). Read it through and correct it before\n" +
+			"training on it (llmgw ctl train): what the model got wrong, an adapter learns.\n\n" +
+			"An adapter learns how answers sound — the organisation's words, format and\n" +
+			"tone — far better than what they say; the facts stay the knowledge base's.",
+		Example: "  llmgw ctl kb dataset Acme/policies --model qwen2.5-coder-14b-instruct --out policies.jsonl",
+		Args:    cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			w := cmd.OutOrStdout()
+			if dsOut != "" {
+				f, err := os.Create(dsOut)
+				if err != nil {
+					return err
+				}
+				defer f.Close()
+				w = f
+			}
+			lw := &datasetWriter{pairs: w, problems: cmd.ErrOrStderr()}
+			body, _ := json.Marshal(adminapi.KBDatasetRequest{Model: dsModel, PerChunk: dsPer, MaxChunks: dsMax})
+			err := client().patient(6*time.Hour).do(cmd.Context(), "POST",
+				"/api/v1/kb/collections/"+url.PathEscape(args[0])+"/dataset", json.RawMessage(body), lw)
+			lw.flush()
+			fmt.Fprintf(cmd.ErrOrStderr(), "%d pairs from %d chunks; %d chunks gave none\n", lw.n, len(lw.chunks), lw.failed)
+			return err
+		},
+	}
+	dataset.Flags().StringVar(&dsModel, "model", "", "the chat model that writes the questions and answers")
+	dataset.Flags().StringVar(&dsOut, "out", "", "the file to write (default: standard output)")
+	dataset.Flags().IntVar(&dsPer, "per-chunk", 8, "pairs from each chunk")
+	dataset.Flags().IntVar(&dsMax, "max-chunks", 500, "the most chunks read")
+	_ = dataset.MarkFlagRequired("model")
+	c.AddCommand(dataset)
+
 	var k int
 	search := &cobra.Command{
 		Use:   "search COLLECTION QUERY...",
@@ -240,3 +283,50 @@ func newKBCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 	c.AddCommand(search)
 	return c
 }
+
+// datasetWriter splits the dataset stream: pairs to the file, problems to
+// the terminal, counted as they pass.
+type datasetWriter struct {
+	pairs, problems io.Writer
+	buf             []byte
+	n, failed       int
+	chunks          map[string]bool
+}
+
+func (d *datasetWriter) Write(p []byte) (int, error) {
+	d.buf = append(d.buf, p...)
+	for {
+		i := bytes.IndexByte(d.buf, '\n')
+		if i < 0 {
+			return len(p), nil
+		}
+		d.line(d.buf[:i])
+		d.buf = d.buf[i+1:]
+	}
+}
+
+func (d *datasetWriter) flush() {
+	if len(bytes.TrimSpace(d.buf)) > 0 {
+		d.line(d.buf)
+	}
+	d.buf = nil
+}
+
+func (d *datasetWriter) line(b []byte) {
+	var l adminapi.KBDatasetLine
+	if json.Unmarshal(b, &l) != nil {
+		return
+	}
+	if d.chunks == nil {
+		d.chunks = map[string]bool{}
+	}
+	d.chunks[fmt.Sprintf("%s#%d", l.Doc, l.Ord)] = true
+	if l.Error != "" {
+		d.failed++
+		fmt.Fprintf(d.problems, "  %s #%d: %s\n", l.Doc, l.Ord, l.Error)
+		return
+	}
+	d.n++
+	out, _ := json.Marshal(l)
+	fmt.Fprintln(d.pairs, string(out))
+}

internal/cli/mirrorinit.go

@@ -8,6 +8,7 @@ import (
 	"net"
 	"net/netip"
 	"os"
+	"path/filepath"
 	"runtime"
 	"strings"
 	"time"
@@ -65,6 +66,14 @@ Safe to run again: it changes only what differs.`,
 			}
 			if !noFirewall {
 				mirrorFirewall(cmd.Context(), cmd.OutOrStdout(), controlPlanes)
+				// And after, as the cluster's settings say: the control plane
+				// tells the mirror, which writes it beside its models.
+				state := filepath.Join(*dir, mirror.FirewallFile)
+				if err := installFirewallSync(cmd.Context(), firewall.Local{}, state); err != nil {
+					fmt.Fprintf(cmd.OutOrStdout(), "  !!    firewall: %v; the cluster's settings will not change it here\n", err)
+				} else {
+					fmt.Fprintln(cmd.OutOrStdout(), "  ok    firewall: follows the cluster's settings ("+firewallSyncPathUnit+")")
+				}
 			}
 			return nil
 		},

internal/cli/server.go

@@ -49,6 +49,7 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/fleetmetrics"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelsvc"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/train"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/kvevents"
@@ -232,6 +233,9 @@ func newServerCmd() *cobra.Command {
 			var agentHandler http.Handler
 			var agentPath string
 			var admin *control.Controller
+			// What the settings say of the firewall, for the mirror and the
+			// store's members; nil without a control plane.
+			var fwPub *firewallPublisher
 			updateMetrics := &modelserve.UpdateCollector{}
 			// The store's cluster, when it is one; read by the metrics and
 			// by `ctl status`.
@@ -263,9 +267,31 @@ func newServerCmd() *cobra.Command {
 					return &llmgwv1.Firewall{Enabled: c.Firewall.Enabled, Networks: c.FirewallNetworks,
 						EnginePortsFrom: int32(from), EnginePortsTo: int32(to)} //nolint:gosec // ports
 				})
+				// What each node's agent runs on, as the settings say now.
+				svc.SetAgentSettings(func(node string) string {
+					b, err := config.Marshal(current.Load().Agents.For(node))
+					if err != nil {
+						log.Warn("could not write a node's agent settings", "node", node, "err", err)
+						return ""
+					}
+					return string(b)
+				})
 				ctl.Reconciler().SetNotifier(notifier)
 				admin = ctl
 				agentPath, agentHandler = svc.Handler()
+				fwPub = newFirewallPublisher(func() config.Server { return *current.Load() }, rt.Store(),
+					func() []string {
+						var out []string
+						for _, n := range ctl.Inventory().Nodes() {
+							out = append(out, n.MgmtAddr)
+						}
+						return out
+					},
+					func() bool { return ctl.Leadership() == nil || ctl.Leadership().IsLeader() }, log)
+				go fwPub.run(ctx, 5*time.Minute)
+				if l := ctl.Leadership(); l != nil {
+					l.OnLead(fwPub.Kick)
+				}
 
 				// Before AdminHandler, which is where the registry is built —
 				// registering after it silently publishes nothing, which is
@@ -378,10 +404,19 @@ func newServerCmd() *cobra.Command {
 					// Knowledge bases, with the cluster's own models to embed a
 					// probe (a collection's dimension) and an operator's search.
 					api.SetKB(&adminapi.KBAdmin{Store: st, Embed: embedder(rt.Replay)})
+					// LoRA adapters trained on the nodes.
+					if agentSvc != nil {
+						trainer := train.New(st, agentSvc, log)
+						agentSvc.SetTrain(trainer)
+						go trainer.Run(ctx)
+						api.SetTrain(&adminapi.TrainAdmin{Store: st, Service: trainer})
+					}
 					// The cluster's settings, each change checked against this
 					// machine's own keys before it is kept.
 					api.SetSettings(&adminapi.Settings{Store: st, LocalKeys: config.LocalKeys(),
 						Check: func(body []byte) error { return checkClusterSettings(configPath, body) }})
+					// And the same document on the portal, for administrators.
+					rt.SetSettingsCheck(func(body []byte) error { return checkClusterSettings(configPath, body) }, config.LocalKeys())
 				}
 				// Catalog changes need somewhere to persist, so this is wired
 				// only when there is a store. Without one, the endpoints say so
@@ -438,6 +473,7 @@ func newServerCmd() *cobra.Command {
 					api.SetUsage(st)
 					// Capacity quotas, set and read over this API.
 					api.SetQuotas(st)
+					api.SetTenancy(st)
 				}
 				api.Register(adminMux)
 
@@ -499,6 +535,16 @@ func newServerCmd() *cobra.Command {
 					return fmt.Errorf("the model mirror: %w", err)
 				}
 				rt.SetMirror(mc)
+				if fwPub != nil {
+					fwPub.setMirror(mc)
+				}
+				// Adapters trained on the nodes are kept there too; sending one
+				// takes longer than the mirror's usual answers.
+				if adminAPI != nil {
+					if up, err := mirror.NewClient(cfg.Mirror.API, cfg.Mirror.TLS, 10*time.Minute); err == nil {
+						adminAPI.SetTrainMirror(up)
+					}
+				}
 				// Browsing the registry is what turns "paste a resolve URL"
 				// into "find the model". It needs a way out to the internet
 				// from this machine, which the mirror's own fetching does not.
@@ -716,6 +762,13 @@ func newServerCmd() *cobra.Command {
 						if agentSvc != nil {
 							agentSvc.BroadcastFirewall()
 						}
+						if fwPub != nil {
+							fwPub.Kick()
+						}
+					}
+					if slices.ContainsFunc(changed, func(c string) bool { return strings.HasPrefix(c, "agents") }) && agentSvc != nil {
+						// A moment apart per node, so off this goroutine.
+						go agentSvc.BroadcastAgentSettings()
 					}
 					if slices.ContainsFunc(changed, func(c string) bool { return strings.HasPrefix(c, "guard") }) {
 						if gate == nil {

internal/cli/serveradd.go

@@ -160,7 +160,7 @@ func (sa *serverAdd) firewall(ctx context.Context) error {
 		return err
 	}
 	// What applies the cluster's firewall setting there when it changes.
-	if err := installFirewallSync(ctx, sa.remote); err != nil {
+	if err := installFirewallSync(ctx, sa.remote, firewallStateFile); err != nil {
 		fmt.Fprintf(sa.out, "  !!    firewall: %v\n", err)
 	}
 	if !cfg.Firewall.Enabled {

internal/cli/serverinit_firewall.go

@@ -93,6 +93,11 @@ func (in *serverInit) storeFirewall(ctx context.Context, members []pgha.Node, se
 				} else {
 					in.kept("firewall on %s", m.Name)
 				}
+				// And after, as the cluster's settings say, read from the
+				// member's own copy of the store.
+				if err := firewall.InstallFollower(ctx, m.Remote); err != nil {
+					fmt.Fprintf(in.out, "  !!    firewall on %s: %v; the cluster's settings will not change it\n", m.Name, err)
+				}
 				continue
 			}
 		}
@@ -135,5 +140,5 @@ func loopbackListen(listen string) bool {
 // ensureFirewallSync installs the units that apply the cluster's firewall
 // setting on this control plane.
 func (in *serverInit) ensureFirewallSync(ctx context.Context) error {
-	return installFirewallSync(ctx, firewall.Local{})
+	return installFirewallSync(ctx, firewall.Local{}, firewallStateFile)
 }

internal/cli/serverinit_postgres.go

@@ -16,6 +16,7 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/pgha"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
 )
 
@@ -348,15 +349,23 @@ func (in *serverInit) setupPostgres(ctx context.Context) (string, error) {
 	}
 
 	if installed {
-		// Its defaults assume it owns the machine. A gateway swapped out to
-		// give the database cache is worse than a smaller cache, and a store
-		// whose working set is megabytes needs little. Only a PostgreSQL init
-		// installed itself: one that was already here may serve something
-		// else that needs its connections.
-		for _, q := range []string{
-			"ALTER SYSTEM SET shared_buffers = '96MB'",
-			"ALTER SYSTEM SET max_connections = 40",
-		} {
+		// Sized to the machine's memory, as one it shares with the gateway:
+		// a gateway swapped out to give the database cache is worse than a
+		// smaller cache, but a knowledge base's vector index is searched
+		// well only from memory. Only a PostgreSQL init installed itself:
+		// one that was already here may serve something else that needs its
+		// connections.
+		meminfo, _ := os.ReadFile("/proc/meminfo")
+		total, ok := pgha.MemTotal(string(meminfo))
+		if !ok {
+			total = 1 << 30
+		}
+		mem := pgha.MemorySettings(total, true)
+		qs := []string{"ALTER SYSTEM SET max_connections = 40"}
+		for _, k := range []string{"shared_buffers", "effective_cache_size", "maintenance_work_mem", "work_mem"} {
+			qs = append(qs, "ALTER SYSTEM SET "+k+" = '"+mem[k]+"'")
+		}
+		for _, q := range qs {
 			if _, err := in.psql(ctx, q); err != nil {
 				return "", err
 			}
@@ -367,7 +376,8 @@ func (in *serverInit) setupPostgres(ctx context.Context) (string, error) {
 		if err := in.waitForPostgres(ctx); err != nil {
 			return "", err
 		}
-		in.done("PostgreSQL sized for a machine it shares (shared_buffers 96MB, 40 connections)")
+		in.done("PostgreSQL sized for a machine it shares, with %d MB of memory (shared_buffers %s, work_mem %s, 40 connections)",
+			total>>20, mem["shared_buffers"], mem["work_mem"])
 	}
 
 	in.setupPgvector(ctx, discovery.DiscoverHost())

internal/cli/train.go

@@ -0,0 +1,293 @@
+package cli
+
+import (
+	"encoding/json"
+	"fmt"
+	"io"
+	"os"
+	"strings"
+	"time"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+func newTrainCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
+	c := &cobra.Command{
+		Use:   "train",
+		Short: "Train LoRA adapters on the nodes, on question/answer pairs",
+		Long: "A node trains an adapter for a catalog model on pairs (llmgw ctl kb dataset\n" +
+			"writes them from a knowledge base; read them through first). It trains with\n" +
+			"its installed vLLM's torch and a small overlay, as the engines' user, on a card\n" +
+			"with --gpu-memory free — beside an engine, if it leaves that much. What it\n" +
+			"makes is kept by the control plane: llmgw ctl train download, or publish it\n" +
+			"as the model's adapter.\n\n" +
+			"An adapter learns how answers sound — words, format, tone — far better than\n" +
+			"what they say. Facts belong to the knowledge base.",
+	}
+
+	var model, base, rev, node, data, owner string
+	var public bool
+	var gpuMem int
+	var epochs, rank int
+	var lr float64
+	var noGGUF bool
+	start := &cobra.Command{
+		Use:   "start NAME --model MODEL --base-repo OWNER/NAME --node NODE --data FILE",
+		Short: "Queue an adapter's training on a node",
+		Example: "  llmgw ctl train start policies-tone --model qwen2.5-0.5b-instruct \\\n" +
+			"    --base-repo Qwen/Qwen2.5-0.5B-Instruct --node cuda01 --gpu-memory 2300 --data policies.jsonl",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			b, err := os.ReadFile(data)
+			if err != nil {
+				return err
+			}
+			req := adminapi.TrainRequest{Name: args[0], Model: model, BaseRepo: base, BaseRevision: rev, Node: node,
+				GPUMemoryMiB: gpuMem, Data: string(b), Params: map[string]any{}, Owner: owner, Public: public}
+			if cmd.Flags().Changed("epochs") {
+				req.Params["epochs"] = epochs
+			}
+			if cmd.Flags().Changed("rank") {
+				req.Params["rank"] = rank
+			}
+			if cmd.Flags().Changed("lr") {
+				req.Params["lr"] = lr
+			}
+			if noGGUF {
+				f := false
+				req.GGUF = &f
+			}
+			body, _ := json.Marshal(req)
+			var v adminapi.TrainJobView
+			if err := client().patient(2*time.Minute).do(cmd.Context(), "POST", "/api/v1/train/jobs", json.RawMessage(body), &v); err != nil {
+				return err
+			}
+			who := "every caller of " + v.Model
+			if v.Owner != "" {
+				who = v.Owner + " only"
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "queued %s: %s for %s on %s, %d pairs; it will be seen by %s\n",
+				v.ID, v.Name, v.Model, v.Node, v.Pairs, who)
+			fmt.Fprintf(cmd.OutOrStdout(), "follow it with: llmgw ctl train show %s --follow\n", v.ID)
+			return nil
+		},
+	}
+	f := start.Flags()
+	f.StringVar(&model, "model", "", "the catalog model the adapter is for")
+	f.StringVar(&base, "base-repo", "", "the weights to train on, as safetensors on Hugging Face (owner/name)")
+	f.StringVar(&rev, "base-revision", "", "a revision of them (default: the newest)")
+	f.StringVar(&node, "node", "", "the node that trains")
+	f.IntVar(&gpuMem, "gpu-memory", 4096, "MiB of a card the job needs free")
+	f.StringVar(&data, "data", "", "the pairs, NDJSON of {\"q\", \"a\"}")
+	f.IntVar(&epochs, "epochs", 6, "passes over the pairs")
+	f.IntVar(&rank, "rank", 16, "the adapter's rank")
+	f.Float64Var(&lr, "lr", 2e-4, "the learning rate")
+	f.BoolVar(&noGGUF, "no-gguf", false, "skip the llama.cpp GGUF, even for a model llama.cpp serves")
+	f.StringVar(&owner, "owner", "", "who the adapter belongs to — org:, team:, project: (path or ID), user: or key: — and only they see it "+
+		"(default: the knowledge base's owner, when the pairs come from one)")
+	f.BoolVar(&public, "public", false, "every caller of the model sees it; refused for pairs from a knowledge base")
+	for _, req := range []string{"model", "base-repo", "node", "data"} {
+		_ = start.MarkFlagRequired(req)
+	}
+	c.AddCommand(start)
+
+	c.AddCommand(&cobra.Command{
+		Use:   "list",
+		Short: "Training jobs, newest first",
+		Args:  cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			var jobs []adminapi.TrainJobView
+			if err := client().get(cmd.Context(), "/api/v1/train/jobs", &jobs); err != nil {
+				return err
+			}
+			if *asJSON {
+				return json.NewEncoder(cmd.OutOrStdout()).Encode(jobs)
+			}
+			if len(jobs) == 0 {
+				fmt.Fprintln(cmd.OutOrStdout(), "No training jobs (llmgw ctl train start).")
+				return nil
+			}
+			w := tab()
+			fmt.Fprintln(w, "JOB\tADAPTER\tMODEL\tOWNER\tNODE\tSTATE\tPROGRESS\tLOSS\tPEAK\tCREATED")
+			for _, j := range jobs {
+				fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n", j.ID, j.Name, j.Model, or(j.Owner, "everyone"), j.Node, j.State,
+					trainProgress(j), trainLoss(j), trainPeak(j), j.CreatedAt.Local().Format("01-02 15:04"))
+			}
+			return w.Flush()
+		},
+	})
+
+	var follow bool
+	show := &cobra.Command{
+		Use:   "show JOB",
+		Short: "How a training job goes, and what it made",
+		Args:  cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			out := cmd.OutOrStdout()
+			last := ""
+			for {
+				var j adminapi.TrainJobView
+				if err := client().get(cmd.Context(), "/api/v1/train/jobs/"+args[0], &j); err != nil {
+					return err
+				}
+				if *asJSON {
+					return json.NewEncoder(out).Encode(j)
+				}
+				line := fmt.Sprintf("%s  %s  %s  loss %s  peak %s  %s", j.State, trainProgress(j), j.Node, trainLoss(j), trainPeak(j), j.Message)
+				if !follow {
+					fmt.Fprintf(out, "%s: adapter %s for %s, owned by %s, on %s (base %s), %d pairs\n", j.ID, j.Name, j.Model,
+						or(j.Owner, "everyone"), j.Node, j.BaseRepo, j.Pairs)
+					fmt.Fprintln(out, line)
+					if len(j.Artifacts) > 0 {
+						fmt.Fprintf(out, "made: %s (llmgw ctl train download %s --kind KIND)\n", strings.Join(j.Artifacts, ", "), j.ID)
+					}
+					return nil
+				}
+				if line != last {
+					fmt.Fprintf(out, "%s  %s\n", time.Now().Format("15:04:05"), line)
+					last = line
+				}
+				if store.TrainOver(j.State) {
+					if j.State != store.TrainDone {
+						return fmt.Errorf("%s %s", j.ID, j.State)
+					}
+					return nil
+				}
+				select {
+				case <-cmd.Context().Done():
+					return nil
+				case <-time.After(3 * time.Second):
+				}
+			}
+		},
+	}
+	show.Flags().BoolVarP(&follow, "follow", "f", false, "follow it until it ends")
+	c.AddCommand(show)
+
+	c.AddCommand(&cobra.Command{
+		Use:   "cancel JOB",
+		Short: "Stop a training job",
+		Args:  cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			if err := client().do(cmd.Context(), "POST", "/api/v1/train/jobs/"+args[0]+"/cancel", nil, nil); err != nil {
+				return err
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "%s cancelled\n", args[0])
+			return nil
+		},
+	})
+	c.AddCommand(&cobra.Command{
+		Use:   "rm JOB",
+		Short: "Remove an ended training job and what it made",
+		Args:  cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			if err := client().do(cmd.Context(), "DELETE", "/api/v1/train/jobs/"+args[0], nil, nil); err != nil {
+				return err
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "%s removed\n", args[0])
+			return nil
+		},
+	})
+
+	var restart bool
+	publish := &cobra.Command{
+		Use:   "publish JOB",
+		Short: "Make what a finished job made its model's adapter, kept on the mirror",
+		Long: "The GGUF for a model llama.cpp serves, the PEFT files for one vLLM serves, are kept\n" +
+			"on the mirror and added to the model's adapters; a request then asks for the\n" +
+			"adapter by its name, as a model. llama.cpp loads adapters when a replica starts:\n" +
+			"--restart replaces the model's replicas, one at a time, to load it.",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			var v adminapi.TrainPublishView
+			if err := client().patient(15*time.Minute).do(cmd.Context(), "POST", "/api/v1/train/jobs/"+args[0]+"/publish", nil, &v); err != nil {
+				return err
+			}
+			out := cmd.OutOrStdout()
+			fmt.Fprintf(out, "%s is %s's adapter, from %s\n", v.Adapter, v.Model, v.Source)
+			if v.Owner != "" {
+				fmt.Fprintf(out, "only %s sees it; to anyone else it is a model that does not exist\n", v.Owner)
+			}
+			if restart && strings.Contains(v.Note, "restart") {
+				body, _ := json.Marshal(adminapi.RestartRequest{Models: []string{v.Model}})
+				var res adminapi.RestartResult
+				if err := client().post(cmd.Context(), "/api/v1/restart", body, &res); err != nil {
+					return err
+				}
+				fmt.Fprintf(out, "%d replica(s) of %s restarting to load it; then ask for model %s\n", len(res.Queued), v.Model, v.Adapter)
+				return nil
+			}
+			fmt.Fprintln(out, v.Note)
+			return nil
+		},
+	}
+	publish.Flags().BoolVar(&restart, "restart", false, "restart the model's replicas to load it (llama.cpp)")
+	c.AddCommand(publish)
+
+	var kind, outFile string
+	download := &cobra.Command{
+		Use:   "download JOB",
+		Short: "Save what a training job made: the PEFT adapter (peft) or the GGUF (gguf)",
+		Args:  cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			if outFile == "" {
+				outFile = args[0] + ".tar.gz"
+				if kind == "gguf" {
+					outFile = args[0] + ".gguf"
+				}
+			}
+			f, err := os.Create(outFile)
+			if err != nil {
+				return err
+			}
+			defer f.Close()
+			cw := &countWriter{w: f}
+			if err := client().patient(10*time.Minute).do(cmd.Context(), "GET",
+				"/api/v1/train/jobs/"+args[0]+"/artifacts/"+kind, nil, io.Writer(cw)); err != nil {
+				_ = os.Remove(outFile)
+				return err
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "%s: %s\n", outFile, humanBytes(cw.n))
+			return nil
+		},
+	}
+	download.Flags().StringVar(&kind, "kind", "peft", "peft (a gzipped tar of the PEFT adapter) or gguf")
+	download.Flags().StringVarP(&outFile, "out", "o", "", "the file to write")
+	c.AddCommand(download)
+	return c
+}
+
+type countWriter struct {
+	w io.Writer
+	n int64
+}
+
+func (c *countWriter) Write(p []byte) (int, error) {
+	n, err := c.w.Write(p)
+	c.n += int64(n)
+	return n, err
+}
+
+func trainProgress(j adminapi.TrainJobView) string {
+	if j.Steps == 0 {
+		return "-"
+	}
+	return fmt.Sprintf("%d/%d", j.Step, j.Steps)
+}
+
+func trainLoss(j adminapi.TrainJobView) string {
+	if j.Loss == 0 {
+		return "-"
+	}
+	return fmt.Sprintf("%.3f", j.Loss)
+}
+
+func trainPeak(j adminapi.TrainJobView) string {
+	if j.PeakMiB == 0 {
+		return "-"
+	}
+	return fmt.Sprintf("%d MiB", j.PeakMiB)
+}

internal/config/agent.go

@@ -18,7 +18,12 @@ import (
 //
 // An empty path loads the defaults, which with --server and --node-id on the
 // command line is enough for a smoke test on a node with no IB.
-func LoadAgent(path string) (agent.Config, error) {
+func LoadAgent(path string) (agent.Config, error) { return LoadAgentWith(path, nil) }
+
+// LoadAgentWith is LoadAgent with the cluster's settings for this node
+// (agents.default and agents.nodes.<id>, as YAML) laid over the file: the
+// file is what the agent needs to start, the settings what it runs on.
+func LoadAgentWith(path string, settings []byte) (agent.Config, error) {
 	cfg := agent.DefaultConfig()
 	k := koanf.New(".")
 
@@ -30,6 +35,11 @@ func LoadAgent(path string) (agent.Config, error) {
 			return cfg, fmt.Errorf("parsing %s: %w", path, err)
 		}
 	}
+	if len(settings) > 0 {
+		if err := k.Load(bytesProvider(settings), yaml.Parser()); err != nil {
+			return cfg, fmt.Errorf("parsing the cluster's settings for this agent: %w", err)
+		}
+	}
 
 	err := k.Load(env.Provider(".", env.Opt{
 		Prefix: "LLMGW_",

internal/config/agents.go

@@ -0,0 +1,110 @@
+package config
+
+import (
+	"fmt"
+	"slices"
+	"sort"
+)
+
+// AgentSettings are the nodes' agent settings, kept with the cluster's
+// (BACKLOG E3): everything agent.yaml can say except what an agent needs to
+// start — its id, the control planes to dial, its address and its
+// certificate. Default applies to every node, and Nodes, by node id, over
+// it; both over the node's own agent.yaml.
+type AgentSettings struct {
+	// Default is agent.yaml's keys for every node.
+	Default map[string]any `koanf:"default"`
+	// Nodes are agent.yaml's keys for one node each, by node id, over
+	// Default.
+	Nodes map[string]map[string]any `koanf:"nodes"`
+}
+
+// agentBootstrapKeys are what an agent needs before it can be told anything:
+// they stay in agent.yaml.
+var agentBootstrapKeys = []string{"node_id", "server", "servers", "hostname", "mgmt_addr", "tls"}
+
+// For is the settings for one node: Default with the node's own over it.
+// Empty when there are none.
+func (a AgentSettings) For(node string) map[string]any {
+	out := clone(stringKeys(a.Default))
+	if n, ok := a.Nodes[node]; ok {
+		merge(out, stringKeys(n))
+	}
+	return out
+}
+
+// Check refuses settings that set an agent's bootstrap keys, or that would
+// not load as an agent's configuration for some node.
+func (a AgentSettings) Check() error {
+	check := func(where string, m map[string]any) error {
+		for _, k := range agentBootstrapKeys {
+			if _, ok := m[k]; ok {
+				return fmt.Errorf("%s.%s stays in each node's agent.yaml: an agent needs it before it can be told anything", where, k)
+			}
+		}
+		return nil
+	}
+	if err := check("agents.default", stringKeys(a.Default)); err != nil {
+		return err
+	}
+	nodes := make([]string, 0, len(a.Nodes))
+	for n := range a.Nodes {
+		nodes = append(nodes, n)
+	}
+	sort.Strings(nodes)
+	for _, n := range nodes {
+		if err := check("agents.nodes."+n, stringKeys(a.Nodes[n])); err != nil {
+			return err
+		}
+	}
+	// Each node's settings as its agent would load them, over nothing: what
+	// its own file adds cannot be known here, and what is wrong in the
+	// settings alone is wrong on every node.
+	for _, n := range append([]string{""}, nodes...) {
+		b, err := Marshal(a.For(n))
+		if err != nil {
+			return err
+		}
+		if _, err := LoadAgentWith("", b); err != nil {
+			where := "agents.default"
+			if n != "" {
+				where = "agents.nodes." + n
+			}
+			return fmt.Errorf("%s: %w", where, err)
+		}
+	}
+	return nil
+}
+
+// AgentBootstrapKeys are the keys agent.yaml keeps, for saying so.
+func AgentBootstrapKeys() []string { return slices.Clone(agentBootstrapKeys) }
+
+// stringKeys turns YAML's map[any]any, wherever it hides, into the
+// map[string]any the rest of this package merges.
+func stringKeys(m map[string]any) map[string]any {
+	out := make(map[string]any, len(m))
+	for k, v := range m {
+		out[k] = normalize(v)
+	}
+	return out
+}
+
+func normalize(v any) any {
+	switch t := v.(type) {
+	case map[string]any:
+		return stringKeys(t)
+	case map[any]any:
+		m := make(map[string]any, len(t))
+		for k, vv := range t {
+			m[fmt.Sprint(k)] = normalize(vv)
+		}
+		return m
+	case []any:
+		out := make([]any, len(t))
+		for i, vv := range t {
+			out[i] = normalize(vv)
+		}
+		return out
+	}
+	return v
+}

internal/config/agents_test.go

@@ -0,0 +1,79 @@
+package config
+
+import (
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+)
+
+func TestANodesAgentSettingsAreTheDefaultWithItsOwnOver(t *testing.T) {
+	a := AgentSettings{
+		Default: map[string]any{"heartbeat_interval": "5s", "supervisor": map[string]any{"work_dir": "/srv/w", "x": 1}},
+		Nodes: map[string]map[string]any{
+			"cuda01": {"supervisor": map[string]any{"work_dir": "/fast/w"}, "labels": map[string]any{"gpu": "rtx6000"}},
+		},
+	}
+	got := a.For("cuda01")
+	sup := got["supervisor"].(map[string]any)
+	if sup["work_dir"] != "/fast/w" || sup["x"] != 1 || got["heartbeat_interval"] != "5s" || got["labels"] == nil {
+		t.Errorf("cuda01: %v", got)
+	}
+	if got := a.For("node01"); got["labels"] != nil || got["supervisor"].(map[string]any)["work_dir"] != "/srv/w" {
+		t.Errorf("node01: %v", got)
+	}
+	if got := (AgentSettings{}).For("x"); len(got) != 0 {
+		t.Errorf("none: %v", got)
+	}
+}
+
+func TestAgentSettingsKeepBootstrapKeysInTheFileAndMustLoad(t *testing.T) {
+	for _, a := range []AgentSettings{
+		{Default: map[string]any{"servers": []any{"x:7443"}}},
+		{Nodes: map[string]map[string]any{"n1": {"node_id": "other"}}},
+		{Nodes: map[string]map[string]any{"n1": {"tls": map[string]any{"ca": "/x"}}}},
+	} {
+		if err := a.Check(); err == nil || !strings.Contains(err.Error(), "stays in each node's agent.yaml") {
+			t.Errorf("%+v: %v", a, err)
+		}
+	}
+	// A fabric with no adapters would not load on the node it names.
+	bad := AgentSettings{Nodes: map[string]map[string]any{"n1": {"fabrics": []any{map[string]any{"name": "ib-a"}}}}}
+	if err := bad.Check(); err == nil || !strings.Contains(err.Error(), "agents.nodes.n1") || !strings.Contains(err.Error(), "no HCAs") {
+		t.Errorf("a fabric without HCAs: %v", err)
+	}
+	if err := (AgentSettings{Default: map[string]any{"heartbeat_interval": "3s"}}).Check(); err != nil {
+		t.Errorf("a plain setting: %v", err)
+	}
+}
+
+func TestTheClusterSettingsCarryAgentsAndRefuseBadOnes(t *testing.T) {
+	good, err := Parse([]byte("agents:\n  default:\n    heartbeat_interval: 3s\n  nodes:\n    cuda01:\n      labels: {gpu: rtx6000}\n"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	cfg, err := FromMap(good)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if cfg.Agents.For("cuda01")["labels"] == nil {
+		t.Errorf("agents not read: %+v", cfg.Agents)
+	}
+	bad, _ := Parse([]byte("agents:\n  default:\n    mgmt_addr: 10.0.0.1\n"))
+	if _, err := FromMap(bad); err == nil {
+		t.Error("a bootstrap key in the cluster's settings was taken")
+	}
+}
+
+func TestTheSettingsAreLaidOverTheAgentsFile(t *testing.T) {
+	p := filepath.Join(t.TempDir(), "agent.yaml")
+	_ = os.WriteFile(p, []byte("node_id: n1\nserver: cp:7443\nheartbeat_interval: 9s\nlabels: {rack: r1}\n"), 0o600)
+	cfg, err := LoadAgentWith(p, []byte("heartbeat_interval: 2s\nlabels: {gpu: a100}\n"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	if cfg.NodeID != "n1" || cfg.HeartbeatInterval != 2*time.Second || cfg.Labels["rack"] != "r1" || cfg.Labels["gpu"] != "a100" {
+		t.Errorf("loaded %+v", cfg)
+	}
+}

internal/config/config.go

@@ -158,6 +158,13 @@ type Server struct {
 	// Firewall turns llmgw's firewall on every machine on or off.
 	Firewall FirewallSettings `koanf:"firewall"`
 
+	// Agents are the nodes' agent settings: agent.yaml's keys but those an
+	// agent needs to start, for every node (default) and for one
+	// (nodes.<id>). Sent to the agents as they change; an agent whose
+	// settings changed starts again on them, its engines serving
+	// throughout.
+	Agents AgentSettings `koanf:"agents"`
+
 	// FirewallNetworks are networks, besides those each machine reaches the
 	// cluster on, that llmgw's internal ports answer: a monitoring host on
 	// another subnet, nodes behind a router. Written into every machine's
@@ -887,6 +894,9 @@ func load(b []byte) (Server, error) {
 // Validate rejects configurations that would otherwise fail confusingly at
 // request time.
 func (s Server) Validate() error {
+	if err := s.Agents.Check(); err != nil {
+		return err
+	}
 	for i, o := range s.EngineOverrides {
 		if o.GPUClass == "" && len(o.Nodes) == 0 && len(o.Labels) == 0 {
 			return fmt.Errorf("engine_overrides[%d] picks no nodes: give it a gpu_class, nodes or labels", i)

internal/control/adminapi/api.go

@@ -70,7 +70,11 @@ type API struct {
 	// guard is the guardrail policies' library and attachments.
 	guard *GuardAdmin
 	// kb is knowledge bases; nil without a store.
-	kb       *KBAdmin
+	kb *KBAdmin
+	// tenancy is organizations, keys and people; nil without a store.
+	tenancy TenancyStore
+	// train is adapter training; nil without a store and agents.
+	train    *TrainAdmin
 	settings *Settings
 	// restartAgents tells every connected agent to start again.
 	restartAgents func(reason string) []string
@@ -117,6 +121,8 @@ func (a *API) Register(mux Mux) {
 	mux.HandleFunc("POST /api/v1/models/{name}/resources", a.setResources)
 	a.registerQuotas(mux)
 	a.registerKB(mux)
+	a.registerTenancy(mux)
+	a.registerTrain(mux)
 	a.registerGuard(mux)
 	a.registerSettings(mux)
 	mux.HandleFunc("POST /api/v1/agents/restart", a.restartAgentsHandler)
@@ -1963,32 +1969,9 @@ func (a *API) setAdapters(w http.ResponseWriter, r *http.Request) {
 		writeError(w, http.StatusBadRequest, "no model "+name)
 		return
 	}
-	base, baseErr := gguf.KindFromURL(r.Context(), safedial.Client(safedial.ControlPlane, 30*time.Second), m.Source)
-	have := map[string]bool{}
-	for _, ad := range m.Adapters {
-		have[ad.Source] = true
-	}
-	// The file is read only where llama.cpp will load it: a vLLM adapter is
-	// a PEFT directory, which has no header to read, and vLLM itself refuses
-	// one that does not fit the model when it is loaded.
-	ggufs := slices.ContainsFunc(m.Profiles, func(p catalog.Profile) bool { return p.Engine == "llamacpp" })
-	for _, ad := range req.Adapters {
-		if have[ad.Source] || !ggufs {
-			continue // already checked when it was attached
-		}
-		h, err := gguf.KindFromURL(r.Context(), safedial.Client(safedial.ControlPlane, 30*time.Second), ad.Source)
-		switch {
-		case err != nil:
-			writeError(w, http.StatusBadRequest, fmt.Sprintf("adapter %s: %v", ad.Name, err))
-			return
-		case !h.IsAdapter():
-			writeError(w, http.StatusBadRequest, fmt.Sprintf("adapter %s: %s is a model, not a LoRA adapter", ad.Name, ad.Source))
-			return
-		case baseErr == nil && h.Architecture != base.Architecture:
-			writeError(w, http.StatusBadRequest, fmt.Sprintf("adapter %s is for %s models and %s is %s",
-				ad.Name, h.Architecture, m.Name, base.Architecture))
-			return
-		}
+	if err := checkAdapters(r.Context(), m, req.Adapters); err != nil {
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
 	}
 	out, err := a.models.SetAdapters(r.Context(), m.Name, req.Adapters, actorOf(r))
 	if err != nil {
@@ -2070,3 +2053,36 @@ func multiNodeReadiness(n inventory.Node) string {
 	}
 	return "ready"
 }
+
+// checkAdapters reads each adapter a model does not have yet, where
+// llama.cpp will load it, to refuse one that is not a LoRA adapter for the
+// model's architecture: such an adapter is a replica that fails to start,
+// found minutes later on a node.
+func checkAdapters(ctx context.Context, m catalog.Model, adapters []catalog.Adapter) error {
+	// A vLLM adapter is a PEFT directory, which has no header to read, and
+	// vLLM itself refuses one that does not fit the model when it is loaded.
+	if !slices.ContainsFunc(m.Profiles, func(p catalog.Profile) bool { return p.Engine == "llamacpp" }) {
+		return nil
+	}
+	client := safedial.Client(safedial.ControlPlane, 30*time.Second)
+	base, baseErr := gguf.KindFromURL(ctx, client, m.Source)
+	have := map[string]bool{}
+	for _, ad := range m.Adapters {
+		have[ad.Source] = true
+	}
+	for _, ad := range adapters {
+		if have[ad.Source] {
+			continue // checked when it was attached
+		}
+		h, err := gguf.KindFromURL(ctx, client, ad.Source)
+		switch {
+		case err != nil:
+			return fmt.Errorf("adapter %s: %v", ad.Name, err)
+		case !h.IsAdapter():
+			return fmt.Errorf("adapter %s: %s is a model, not a LoRA adapter", ad.Name, ad.Source)
+		case baseErr == nil && h.Architecture != base.Architecture:
+			return fmt.Errorf("adapter %s is for %s models and %s is %s", ad.Name, h.Architecture, m.Name, base.Architecture)
+		}
+	}
+	return nil
+}

internal/control/adminapi/kb.go

@@ -30,6 +30,9 @@ type KBStore interface {
 	DeleteDocument(ctx context.Context, collection, name string) (bool, error)
 	SearchChunks(ctx context.Context, q store.KBSearch) ([]store.KBHit, error)
 	SetKBDefault(ctx context.Context, d store.KBDefault) error
+	CollectionChunks(ctx context.Context, c store.KBCollection, limit int) ([]store.KBHit, error)
+	CollectionsFor(ctx context.Context, e store.Effective, write bool) ([]store.KBCollection, error)
+	EffectiveFor(ctx context.Context, scope store.Scope, id, org string) (store.Effective, error)
 	KBDefaults(ctx context.Context) ([]store.KBDefault, error)
 }
 
@@ -161,6 +164,7 @@ func (a *API) registerKB(mux Mux) {
 	mux.HandleFunc("DELETE /api/v1/kb/grants", a.kbGrant(false))
 	mux.HandleFunc("POST /api/v1/kb/search", a.kbSearch)
 	mux.HandleFunc("POST /api/v1/kb/defaults", a.kbSetDefault)
+	mux.HandleFunc("POST /api/v1/kb/collections/{id}/dataset", a.kbDataset)
 }
 
 // kbReady answers for a gateway without knowledge bases, and says whether

internal/control/adminapi/kbdataset.go

@@ -0,0 +1,147 @@
+package adminapi
+
+import (
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"regexp"
+	"strings"
+)
+
+// Training data from a collection (BACKLOG F4): questions a person might
+// ask of each chunk and the answers it holds, written by a model of the
+// cluster, for people to read through before an adapter is trained on them.
+// What an adapter learns well is how the organisation's answers sound; what
+// they say is still the knowledge base's to give.
+
+// KBDatasetRequest asks for training data from a collection.
+type KBDatasetRequest struct {
+	// Model writes the questions and answers: a chat model of the catalog.
+	Model string `json:"model"`
+	// PerChunk is how many pairs each chunk gives; 8 when 0.
+	PerChunk int `json:"per_chunk,omitempty"`
+	// MaxChunks bounds how many chunks are read; 500 when 0.
+	MaxChunks int `json:"max_chunks,omitempty"`
+}
+
+// KBDatasetLine is one line of the answer: a pair, or why a chunk gave none.
+type KBDatasetLine struct {
+	Q   string `json:"q,omitempty"`
+	A   string `json:"a,omitempty"`
+	Doc string `json:"doc"`
+	Ord int    `json:"ord"`
+	// Collection is where the pair came from: an adapter trained on it
+	// belongs to a part of the tree that may read it.
+	Collection string `json:"collection,omitempty"`
+	Error      string `json:"error,omitempty"`
+}
+
+const kbDatasetPrompt = `From the passage of an organisation's document below, write %d different questions an employee might ask, each with a short, correct answer.
+Rules: use only what the passage says. Phrase the questions differently from each other (formal, everyday, short, indirect). Answers are one or two sentences and keep the passage's numbers exactly. Leave out e-mail addresses and people's names. Write in the passage's own language.
+Output: one JSON object per line, {"q": "...", "a": "..."}, and nothing else.
+
+DOCUMENT %s:
+%s`
+
+var thinkBlock = regexp.MustCompile(`(?s)<think>.*?</think>`)
+
+func (a *API) kbDataset(w http.ResponseWriter, r *http.Request) {
+	c, ok := a.kbCollection(w, r)
+	if !ok {
+		return
+	}
+	if a.replay == nil {
+		writeError(w, http.StatusNotImplemented, "no router here to ask a model")
+		return
+	}
+	var in KBDatasetRequest
+	if err := json.NewDecoder(r.Body).Decode(&in); err != nil {
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	}
+	m, found := a.ctl.Catalog().Resolve(in.Model)
+	if !found {
+		writeError(w, http.StatusBadRequest, "no model "+in.Model+" in the catalog")
+		return
+	}
+	if m.Capabilities.Embeddings {
+		writeError(w, http.StatusBadRequest, m.Name+" embeds; questions are written by a chat model")
+		return
+	}
+	per := in.PerChunk
+	if per <= 0 || per > 30 {
+		per = 8
+	}
+	limit := in.MaxChunks
+	if limit <= 0 || limit > 5000 {
+		limit = 500
+	}
+	chunks, err := a.kb.Store.CollectionChunks(r.Context(), c, limit)
+	if err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	w.Header().Set("Content-Type", "application/x-ndjson")
+	w.WriteHeader(http.StatusOK)
+	enc := json.NewEncoder(w)
+	flush := func() {
+		if f, ok := w.(http.Flusher); ok {
+			f.Flush()
+		}
+	}
+	for _, ch := range chunks {
+		if r.Context().Err() != nil {
+			return
+		}
+		body, _ := json.Marshal(map[string]any{"model": m.Name, "temperature": 0.7, "max_tokens": 200 * per,
+			"messages": []map[string]string{{"role": "user", "content": fmt.Sprintf(kbDatasetPrompt, per, ch.Document, ch.Body)}}})
+		status, out, _, err := a.replay(r.Context(), "/v1/chat/completions", m.Name, body)
+		line := KBDatasetLine{Doc: ch.Document, Ord: ch.Ord, Collection: c.ID}
+		if err == nil && status != http.StatusOK {
+			err = fmt.Errorf("%d: %s", status, strings.TrimSpace(string(out)))
+		}
+		if err != nil {
+			line.Error = err.Error()
+			_ = enc.Encode(line)
+			flush()
+			continue
+		}
+		pairs := datasetPairs(out)
+		if len(pairs) == 0 {
+			line.Error = "the model gave no pairs it could read"
+			_ = enc.Encode(line)
+		}
+		for _, p := range pairs {
+			p.Doc, p.Ord, p.Collection = ch.Document, ch.Ord, c.ID
+			_ = enc.Encode(p)
+		}
+		flush()
+	}
+}
+
+// datasetPairs reads the {"q","a"} lines out of a chat completion.
+func datasetPairs(completion []byte) []KBDatasetLine {
+	var res struct {
+		Choices []struct {
+			Message struct {
+				Content string `json:"content"`
+			} `json:"message"`
+		} `json:"choices"`
+	}
+	if json.Unmarshal(completion, &res) != nil || len(res.Choices) == 0 {
+		return nil
+	}
+	text := thinkBlock.ReplaceAllString(res.Choices[0].Message.Content, "")
+	var out []KBDatasetLine
+	for _, line := range strings.Split(text, "\n") {
+		line = strings.TrimSuffix(strings.TrimSpace(line), ",")
+		if !strings.HasPrefix(line, "{") || !strings.HasSuffix(line, "}") {
+			continue
+		}
+		var p struct{ Q, A string }
+		if json.Unmarshal([]byte(line), &p) == nil && strings.TrimSpace(p.Q) != "" && strings.TrimSpace(p.A) != "" {
+			out = append(out, KBDatasetLine{Q: strings.TrimSpace(p.Q), A: strings.TrimSpace(p.A)})
+		}
+	}
+	return out
+}

internal/control/adminapi/kbdataset_test.go

@@ -0,0 +1,91 @@
+package adminapi
+
+import (
+	"context"
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+func TestPairsAreReadOutOfWhatAModelWrote(t *testing.T) {
+	completion := `{"choices":[{"message":{"content":"<think>let me see</think>\n{\"q\": \"Babalık izni kaç gün?\", \"a\": \"10 iş günü.\"},\nnot json\n{\"q\": \"\", \"a\": \"empty question\"}\n{\"q\": \"Ne zaman?\", \"a\": \"Doğumdan sonra.\"}"}}]}`
+	got := datasetPairs([]byte(completion))
+	if len(got) != 2 || got[0].Q != "Babalık izni kaç gün?" || got[1].A != "Doğumdan sonra." {
+		t.Errorf("pairs %+v", got)
+	}
+	if datasetPairs([]byte(`{"choices":[]}`)) != nil {
+		t.Error("pairs from no choice")
+	}
+}
+
+func TestADatasetIsWrittenChunkByChunk(t *testing.T) {
+	dsn := os.Getenv("LLMGW_TEST_POSTGRES")
+	if dsn == "" {
+		t.Skip("set LLMGW_TEST_POSTGRES to a pgvector database to run this")
+	}
+	ctx := context.Background()
+	s, err := store.OpenPostgres(ctx, dsn)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer s.Close()
+	if s.KnowledgeAvailable() != nil {
+		t.Skip("no pgvector")
+	}
+	models := append(kbModels(), catalog.Model{Name: "writer", Source: "/w",
+		Profiles: []catalog.Profile{{Name: "cpu", GPUClass: "h100-80g", TP: 1}}})
+	a, _ := setup(t, models)
+	a.SetKB(&KBAdmin{Store: s})
+	asked := 0
+	a.SetReplayer(func(_ context.Context, path, model string, body []byte) (int, []byte, time.Duration, error) {
+		asked++
+		if path != "/v1/chat/completions" || model != "writer" || !strings.Contains(string(body), "Write in the passage") {
+			t.Errorf("asked %s %s %s", path, model, body)
+		}
+		if strings.Contains(string(body), "bozuk") {
+			return 200, []byte(`{"choices":[{"message":{"content":"sorry"}}]}`), 0, nil
+		}
+		return 200, []byte(`{"choices":[{"message":{"content":"{\"q\":\"Soru?\",\"a\":\"Cevap.\"}"}}]}`), 0, nil
+	})
+	org, _ := s.CreateOrg(ctx, "ds-"+strings.ToLower(store.NewID("")))
+	c, err := s.CreateCollection(ctx, store.KBCollection{OwnerScope: store.ScopeOrg, OwnerID: org, Name: "d", EmbedModel: "bge-m3", Dims: 3})
+	if err != nil {
+		t.Fatal(err)
+	}
+	for _, d := range []string{"a.md", "b.md"} {
+		body := "iyi metin"
+		if d == "b.md" {
+			body = "bozuk metin"
+		}
+		if _, _, err := s.PutDocument(ctx, c, store.KBDocument{Name: d, Digest: d}, []store.KBChunk{{Body: body, Embedding: []float32{1, 0, 0}}}); err != nil {
+			t.Fatal(err)
+		}
+	}
+	mux := http.NewServeMux()
+	a.Register(mux)
+	rec := httptest.NewRecorder()
+	mux.ServeHTTP(rec, httptest.NewRequest("POST", "/api/v1/kb/collections/"+c.ID+"/dataset", strings.NewReader(`{"model":"writer","per_chunk":3}`)))
+	var lines []KBDatasetLine
+	for _, l := range strings.Split(strings.TrimSpace(rec.Body.String()), "\n") {
+		var x KBDatasetLine
+		_ = json.Unmarshal([]byte(l), &x)
+		lines = append(lines, x)
+	}
+	if rec.Code != 200 || asked != 2 || len(lines) != 2 || lines[0].Q != "Soru?" || lines[0].Doc != "a.md" ||
+		lines[1].Doc != "b.md" || lines[1].Error == "" {
+		t.Errorf("%d asked %d: %+v", rec.Code, asked, lines)
+	}
+	// An embedding model writes no questions.
+	rec = httptest.NewRecorder()
+	mux.ServeHTTP(rec, httptest.NewRequest("POST", "/api/v1/kb/collections/"+c.ID+"/dataset", strings.NewReader(`{"model":"bge-m3"}`)))
+	if rec.Code != http.StatusBadRequest {
+		t.Errorf("an embedding model: %d", rec.Code)
+	}
+}

internal/control/adminapi/routes.go

@@ -67,6 +67,12 @@ var PathParams = map[string]string{
 	"federation/{peer}": "The federation peer's name, as `llmgw ctl federation list` lists it.",
 	"holds/{domain}":    "The domain as level:name, rack:r12-07, as `llmgw ctl facility` lists it.",
 	"collections/{id}":  "The collection's ID, its name when that is unique, or ORG/NAME.",
+	"orgs/{id}":         "The organization's ID or name.",
+	"teams/{id}":        "The team's ID.",
+	"projects/{id}":     "The project's ID.",
+	"keys/{id}":         "The key's ID, as listed (not the key itself).",
+	"users/{id}":        "The person's ID or email.",
+	"artifacts/{kind}":  "What the job made: peft, a gzipped tar of the PEFT adapter, or gguf, the adapter for llama.cpp.",
 }
 
 // Routes is every endpoint Register mounts.
@@ -288,10 +294,77 @@ var Routes = []Route{
 		Description: "On an organization, team, project or key; the narrowest one with a default wins. No collections clears it. " +
 			"A request's own X-Llmgw-Knowledge overrides it, and X-Llmgw-Knowledge: none turns it off.",
 		Request: KBDefaultRequest{}, Response: KBDefaultView{}},
+	{Method: "POST", Path: "/api/v1/kb/collections/{id}/dataset", Summary: "Training data from a collection: questions and answers a chat model writes from each chunk",
+		Description: "NDJSON, a line per pair as each chunk is read, or a line with error for a chunk that gave none. For people to correct before an adapter is trained on it.",
+		Request:     KBDatasetRequest{}, Response: KBDatasetLine{}},
 	{Method: "POST", Path: "/api/v1/kb/search", Summary: "Try a search of a collection, as an administrator",
 		Description: "No tenancy applies: this is for checking what a collection holds and finds.",
 		Request:     KBSearchRequest{}, Response: []KBHitView{}},
 
+	// LoRA adapters trained on the nodes.
+	// Organizations, teams, projects, keys and people.
+	{Method: "GET", Path: "/api/v1/orgs", Summary: "The tenancy tree: every organization, its teams and their projects",
+		Response: []OrgView{}},
+	{Method: "POST", Path: "/api/v1/orgs", Summary: "Add an organization",
+		Request: TenancyCreateRequest{}, Response: TenancyItem{}, Status: http.StatusCreated},
+	{Method: "PATCH", Path: "/api/v1/orgs/{id}", Summary: "Rename an organization",
+		Request: TenancyChangeRequest{}, Response: TenancyItem{}},
+	{Method: "DELETE", Path: "/api/v1/orgs/{id}", Summary: "Remove an organization that has no teams",
+		Response: map[string]any{}},
+	{Method: "POST", Path: "/api/v1/teams", Summary: "Add a team to an organization",
+		Request: TenancyCreateRequest{}, Response: TenancyItem{}, Status: http.StatusCreated},
+	{Method: "PATCH", Path: "/api/v1/teams/{id}", Summary: "Rename a team, or move it to another organization",
+		Description: "Everything under it moves too, and from then on inherits the new organization's limits.",
+		Request:     TenancyChangeRequest{}, Response: TenancyItem{}},
+	{Method: "DELETE", Path: "/api/v1/teams/{id}", Summary: "Remove a team that has no projects",
+		Response: map[string]any{}},
+	{Method: "POST", Path: "/api/v1/projects", Summary: "Add a project to a team",
+		Request: TenancyCreateRequest{}, Response: TenancyItem{}, Status: http.StatusCreated},
+	{Method: "PATCH", Path: "/api/v1/projects/{id}", Summary: "Rename a project, or move it to another team",
+		Request: TenancyChangeRequest{}, Response: TenancyItem{}},
+	{Method: "DELETE", Path: "/api/v1/projects/{id}", Summary: "Remove a project with no people, keys or collections",
+		Description: "A project that ever had a key keeps it, revoked, because usage refers to it; rename such a project instead.",
+		Response:    map[string]any{}},
+	{Method: "GET", Path: "/api/v1/keys", Summary: "API keys, with their project and owner; never the keys themselves",
+		Query:    []Param{{Name: "project", Description: "Only this project's keys: ORG/TEAM/PROJECT or its ID."}},
+		Response: []KeyView{}},
+	{Method: "POST", Path: "/api/v1/keys", Summary: "Issue an API key",
+		Description: "The answer holds the key, once: only its hash is kept. A lifetime longer than a scope above it allows is refused.",
+		Request:     KeyCreateRequest{}, Response: KeyCreated{}, Status: http.StatusCreated},
+	{Method: "GET", Path: "/api/v1/keys/{id}", Summary: "One API key", Response: KeyView{}},
+	{Method: "DELETE", Path: "/api/v1/keys/{id}", Summary: "Revoke an API key",
+		Description: "A soft delete: usage refers to it. A gateway that has it cached accepts it until auth.cache_ttl.",
+		Response:    map[string]any{}},
+	{Method: "GET", Path: "/api/v1/users", Summary: "People, with where each is a member",
+		Query:    []Param{{Name: "state", Description: "Only people in this state: pending, active or disabled."}},
+		Response: []UserView{}},
+	{Method: "GET", Path: "/api/v1/users/{id}", Summary: "One person, with where they are a member", Response: UserView{}},
+	{Method: "POST", Path: "/api/v1/users/{id}/memberships", Summary: "Put a person in an organization, team or project",
+		Description: "Already there, only the role changes.",
+		Request:     MembershipRequest{}, Response: UserView{}},
+	{Method: "DELETE", Path: "/api/v1/users/{id}/memberships", Summary: "Take a person out of an organization, team or project",
+		Description: "Their keys in projects they no longer reach are revoked.",
+		Request:     MembershipRequest{}, Response: UserView{}},
+
+	{Method: "POST", Path: "/api/v1/train/jobs", Summary: "Queue a LoRA adapter's training on a node",
+		Description: "The node trains when its agent is there, nothing else trains on it, and a card has gpu_memory_mib free; " +
+			"the pairs are NDJSON of {\"q\", \"a\"}. An adapter learns how answers sound far better than what they say.",
+		Request: TrainRequest{}, Response: TrainJobView{}, Status: http.StatusCreated},
+	{Method: "GET", Path: "/api/v1/train/jobs", Summary: "Training jobs, newest first",
+		Query:    []Param{{Name: "limit", Description: "How many; 50 when absent.", Type: "integer"}},
+		Response: []TrainJobView{}},
+	{Method: "GET", Path: "/api/v1/train/jobs/{id}", Summary: "One training job: how it goes, and what it made",
+		Response: TrainJobView{}},
+	{Method: "POST", Path: "/api/v1/train/jobs/{id}/cancel", Summary: "Stop a training job",
+		Response: TrainJobView{}},
+	{Method: "DELETE", Path: "/api/v1/train/jobs/{id}", Summary: "Remove an ended training job and what it made",
+		Response: map[string]any{}},
+	{Method: "POST", Path: "/api/v1/train/jobs/{id}/publish", Summary: "Make what a finished job made the model's adapter",
+		Description: "Kept on the mirror — the GGUF for a model llama.cpp serves, the PEFT files for one vLLM serves — and added to the model's adapters after the same check as POST /api/v1/models/{name}/adapters.",
+		Response:    TrainPublishView{}},
+	{Method: "GET", Path: "/api/v1/train/jobs/{id}/artifacts/{kind}", Summary: "What a training job made: peft (a gzipped tar) or gguf",
+		Response: map[string]any{}},
+
 	{Method: "POST", Path: "/api/v1/agents/restart", Summary: "Have every node's agent start again; their engines keep serving",
 		Response: AgentRestartView{}},
 

internal/control/adminapi/tenancy.go

@@ -0,0 +1,658 @@
+package adminapi
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"net/http"
+	"strings"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// The tenancy tree, keys and people's memberships, administered over the
+// API as `llmgw org`, `keys` and `users` do against the store: so that
+// they can be run from elsewhere, and kept in Terraform.
+
+// TenancyStore is the part of the store the tenancy endpoints need.
+type TenancyStore interface {
+	Tree(ctx context.Context) ([]store.TreeOrg, error)
+	CreateOrg(ctx context.Context, name string) (string, error)
+	CreateTeam(ctx context.Context, orgID, name string) (string, error)
+	CreateProject(ctx context.Context, teamID, name string) (string, error)
+	RenameOrg(ctx context.Context, id, name string) error
+	RenameTeam(ctx context.Context, id, name string) error
+	RenameProject(ctx context.Context, id, name string) error
+	MoveTeam(ctx context.Context, teamID, orgID string) error
+	MoveProject(ctx context.Context, projectID, teamID string) error
+	DeleteOrg(ctx context.Context, id string) error
+	DeleteTeam(ctx context.Context, id string) error
+	DeleteProject(ctx context.Context, id string) error
+
+	AllKeys(ctx context.Context) ([]store.KeyRow, error)
+	KeyByID(ctx context.Context, id string) (store.Key, error)
+	CreateKey(ctx context.Context, k store.Key) error
+	CreateUserKey(ctx context.Context, userID string, k store.Key) error
+	RevokeKey(ctx context.Context, id string, at time.Time) error
+
+	Users(ctx context.Context, state store.UserState) ([]store.User, error)
+	UserByID(ctx context.Context, id string) (store.User, error)
+	UserByEmail(ctx context.Context, email string) (store.User, error)
+	MembershipsOf(ctx context.Context, userID string) ([]store.Membership, error)
+	AddMembership(ctx context.Context, userID string, scope store.Scope, scopeID, role, by string, at time.Time) error
+	RemoveMembership(ctx context.Context, userID string, scope store.Scope, scopeID string, at time.Time) (int, error)
+}
+
+// SetTenancy enables the tenancy endpoints. The quota store must be set
+// too: names are resolved through it.
+func (a *API) SetTenancy(t TenancyStore) { a.tenancy = t }
+
+// OrgView is an organization with what is in it.
+type OrgView struct {
+	ID      string     `json:"id"`
+	Name    string     `json:"name"`
+	Members int        `json:"members"`
+	Teams   []TeamView `json:"teams"`
+}
+
+// TeamView is a team with its projects.
+type TeamView struct {
+	ID       string        `json:"id"`
+	Name     string        `json:"name"`
+	Org      string        `json:"org"`
+	Members  int           `json:"members"`
+	Projects []ProjectView `json:"projects"`
+}
+
+// ProjectView is a project, with how many people and keys are in it.
+type ProjectView struct {
+	ID   string `json:"id"`
+	Name string `json:"name"`
+	Team string `json:"team"`
+	// Path is ORG/TEAM/PROJECT by name.
+	Path       string `json:"path"`
+	Users      int    `json:"users"`
+	Keys       int    `json:"keys"`
+	ActiveKeys int    `json:"active_keys"`
+}
+
+// TenancyCreateRequest adds an organization, a team or a project.
+type TenancyCreateRequest struct {
+	Name string `json:"name"`
+	// In is the organization a team goes in, or the team a project goes in
+	// (ORG/TEAM), by name or ID; nothing for an organization.
+	In string `json:"in,omitempty"`
+}
+
+// TenancyChangeRequest renames an organization, team or project, or moves
+// a team to another organization or a project to another team.
+type TenancyChangeRequest struct {
+	Name string `json:"name,omitempty"`
+	// In is where it moves to, by name or ID; not for an organization.
+	In string `json:"in,omitempty"`
+}
+
+// TenancyItem is an organization, team or project added or changed.
+type TenancyItem struct {
+	ID   string `json:"id"`
+	Name string `json:"name,omitempty"`
+	// In is the organization a team is in, or the team a project is in, by
+	// ID.
+	In string `json:"in,omitempty"`
+}
+
+// KeyView is a key as listed: never the key itself, which is not kept.
+type KeyView struct {
+	ID   string `json:"id"`
+	Name string `json:"name"`
+	// Prefix is the key's first characters, to tell it apart.
+	Prefix  string `json:"prefix"`
+	Project string `json:"project,omitempty"`
+	// ProjectPath is ORG/TEAM/PROJECT, or empty for a personal key.
+	ProjectPath    string    `json:"project_path,omitempty"`
+	User           string    `json:"user,omitempty"`
+	UserEmail      string    `json:"user_email,omitempty"`
+	Band           string    `json:"band,omitempty"`
+	RoutingProfile string    `json:"routing_profile,omitempty"`
+	State          string    `json:"state"`
+	CreatedAt      time.Time `json:"created_at"`
+	ExpiresAt      time.Time `json:"expires_at,omitzero"`
+	RevokedAt      time.Time `json:"revoked_at,omitzero"`
+	LastUsedAt     time.Time `json:"last_used_at,omitzero"`
+}
+
+// KeyCreateRequest issues a key: a project's, a person's in a project they
+// are a member of, or a person's own with no project.
+type KeyCreateRequest struct {
+	Name string `json:"name"`
+	// Project is ORG/TEAM/PROJECT or its ID.
+	Project string `json:"project,omitempty"`
+	// User is the person it belongs to, by ID or email.
+	User           string `json:"user,omitempty"`
+	Band           string `json:"band,omitempty"`
+	RoutingProfile string `json:"routing_profile,omitempty"`
+	// ExpiresIn is its lifetime, as 720h; none never expires, unless a
+	// scope above it limits how long keys live.
+	ExpiresIn string `json:"expires_in,omitempty"`
+}
+
+// KeyCreated is a new key, with the key itself: this answer is the only
+// time it is given.
+type KeyCreated struct {
+	KeyView
+	Secret string `json:"secret"`
+}
+
+// UserView is a person, with where they are a member.
+type UserView struct {
+	ID    string `json:"id"`
+	Email string `json:"email"`
+	Name  string `json:"name,omitempty"`
+	State string `json:"state"`
+	// Role is the person's part in running the gateway, if any: viewer,
+	// operator, billing or admin.
+	Role        string           `json:"role,omitempty"`
+	Memberships []MembershipView `json:"memberships"`
+	CreatedAt   time.Time        `json:"created_at"`
+	LastLoginAt time.Time        `json:"last_login_at,omitzero"`
+}
+
+// MembershipView is one place a person belongs.
+type MembershipView struct {
+	Target string    `json:"target"`
+	Path   string    `json:"path,omitempty"`
+	Role   string    `json:"role"`
+	At     time.Time `json:"at"`
+	By     string    `json:"by,omitempty"`
+}
+
+// MembershipRequest puts a person in an organization, team or project, or
+// with DELETE takes them out.
+type MembershipRequest struct {
+	// Target is org:NAME, team:ORG/TEAM or project:ORG/TEAM/PROJECT, or the
+	// ID after the colon.
+	Target string `json:"target"`
+	// Role is member or manager; member when absent. Ignored by DELETE.
+	Role string `json:"role,omitempty"`
+}
+
+func (a *API) registerTenancy(mux Mux) {
+	mux.HandleFunc("GET /api/v1/orgs", a.tenancyTree)
+	for _, level := range []store.Scope{store.ScopeOrg, store.ScopeTeam, store.ScopeProject} {
+		path := "/api/v1/" + string(level) + "s"
+		mux.HandleFunc("POST "+path, a.tenancyCreate(level))
+		mux.HandleFunc("PATCH "+path+"/{id}", a.tenancyChange(level))
+		mux.HandleFunc("DELETE "+path+"/{id}", a.tenancyDelete(level))
+	}
+	mux.HandleFunc("GET /api/v1/keys", a.keyList)
+	mux.HandleFunc("POST /api/v1/keys", a.keyCreate)
+	mux.HandleFunc("GET /api/v1/keys/{id}", a.keyGet)
+	mux.HandleFunc("DELETE /api/v1/keys/{id}", a.keyRevoke)
+	mux.HandleFunc("GET /api/v1/users", a.userList)
+	mux.HandleFunc("GET /api/v1/users/{id}", a.userGet)
+	mux.HandleFunc("POST /api/v1/users/{id}/memberships", a.membership(true))
+	mux.HandleFunc("DELETE /api/v1/users/{id}/memberships", a.membership(false))
+}
+
+// tenancyReady answers for a control plane without the store, and says
+// whether it did.
+func (a *API) tenancyReady(w http.ResponseWriter) bool {
+	if a.tenancy == nil || a.quotas == nil {
+		writeError(w, http.StatusNotImplemented, "organizations, keys and people need the store, which this control plane runs without")
+		return false
+	}
+	return true
+}
+
+// tenancyStatus is the status a store error answers with.
+func tenancyStatus(err error) int {
+	switch {
+	case errors.Is(err, store.ErrNotFound):
+		return http.StatusNotFound
+	case errors.Is(err, store.ErrDuplicateName), errors.Is(err, store.ErrNotEmpty):
+		return http.StatusConflict
+	}
+	return http.StatusBadRequest
+}
+
+func (a *API) tenancyTree(w http.ResponseWriter, r *http.Request) {
+	if !a.tenancyReady(w) {
+		return
+	}
+	tree, err := a.tenancy.Tree(r.Context())
+	if err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	out := []OrgView{}
+	for _, o := range tree {
+		ov := OrgView{ID: o.ID, Name: o.Name, Members: o.Members, Teams: []TeamView{}}
+		for _, t := range o.Teams {
+			tv := TeamView{ID: t.ID, Name: t.Name, Org: o.ID, Members: t.Members, Projects: []ProjectView{}}
+			for _, p := range t.Projects {
+				tv.Projects = append(tv.Projects, ProjectView{ID: p.ID, Name: p.Name, Team: t.ID,
+					Path: o.Name + "/" + t.Name + "/" + p.Name, Users: p.Users, Keys: p.Keys, ActiveKeys: p.ActiveKeys})
+			}
+			ov.Teams = append(ov.Teams, tv)
+		}
+		out = append(out, ov)
+	}
+	writeJSON(w, http.StatusOK, out)
+}
+
+// parentOf is the level something at level sits in.
+func parentOf(level store.Scope) store.Scope {
+	if level == store.ScopeProject {
+		return store.ScopeTeam
+	}
+	return store.ScopeOrg
+}
+
+// levelNoun is a level as a sentence says it.
+func levelNoun(level store.Scope) string {
+	switch level {
+	case store.ScopeOrg:
+		return "an organization"
+	case store.ScopeTeam:
+		return "a team"
+	}
+	return "a project"
+}
+
+func (a *API) tenancyCreate(level store.Scope) http.HandlerFunc {
+	return func(w http.ResponseWriter, r *http.Request) {
+		if !a.tenancyReady(w) {
+			return
+		}
+		var in TenancyCreateRequest
+		if err := json.NewDecoder(r.Body).Decode(&in); err != nil {
+			writeError(w, http.StatusBadRequest, err.Error())
+			return
+		}
+		in.Name = strings.TrimSpace(in.Name)
+		if in.Name == "" {
+			writeError(w, http.StatusBadRequest, "a name is needed")
+			return
+		}
+		ctx := r.Context()
+		var id, parent string
+		var err error
+		if level != store.ScopeOrg {
+			if in.In == "" {
+				writeError(w, http.StatusBadRequest, fmt.Sprintf("%s goes in %s: give in", levelNoun(level), levelNoun(parentOf(level))))
+				return
+			}
+			if _, parent, _, err = a.resolveScope(ctx, string(parentOf(level))+":"+in.In); err != nil {
+				writeError(w, http.StatusNotFound, err.Error())
+				return
+			}
+		}
+		switch level {
+		case store.ScopeOrg:
+			id, err = a.tenancy.CreateOrg(ctx, in.Name)
+		case store.ScopeTeam:
+			id, err = a.tenancy.CreateTeam(ctx, parent, in.Name)
+		default:
+			id, err = a.tenancy.CreateProject(ctx, parent, in.Name)
+		}
+		if errors.Is(err, store.ErrDuplicateName) {
+			writeError(w, http.StatusConflict, fmt.Sprintf("there is already %s named %q there", levelNoun(level), in.Name))
+			return
+		}
+		if err != nil {
+			writeError(w, tenancyStatus(err), err.Error())
+			return
+		}
+		writeJSON(w, http.StatusCreated, TenancyItem{ID: id, Name: in.Name, In: parent})
+	}
+}
+
+func (a *API) tenancyChange(level store.Scope) http.HandlerFunc {
+	return func(w http.ResponseWriter, r *http.Request) {
+		if !a.tenancyReady(w) {
+			return
+		}
+		var in TenancyChangeRequest
+		if err := json.NewDecoder(r.Body).Decode(&in); err != nil {
+			writeError(w, http.StatusBadRequest, err.Error())
+			return
+		}
+		ctx := r.Context()
+		_, id, _, err := a.resolveScope(ctx, string(level)+":"+r.PathValue("id"))
+		if err != nil {
+			writeError(w, http.StatusNotFound, err.Error())
+			return
+		}
+		if in.Name == "" && in.In == "" {
+			writeError(w, http.StatusBadRequest, "nothing to change: give name, in or both")
+			return
+		}
+		out := TenancyItem{ID: id}
+		if in.In != "" {
+			if level == store.ScopeOrg {
+				writeError(w, http.StatusBadRequest, "an organization is in nothing; only a team or a project moves")
+				return
+			}
+			_, to, _, err := a.resolveScope(ctx, string(parentOf(level))+":"+in.In)
+			if err != nil {
+				writeError(w, http.StatusNotFound, err.Error())
+				return
+			}
+			if level == store.ScopeTeam {
+				err = a.tenancy.MoveTeam(ctx, id, to)
+			} else {
+				err = a.tenancy.MoveProject(ctx, id, to)
+			}
+			if err != nil {
+				writeError(w, tenancyStatus(err), err.Error())
+				return
+			}
+			out.In = to
+		}
+		if name := strings.TrimSpace(in.Name); name != "" {
+			switch level {
+			case store.ScopeOrg:
+				err = a.tenancy.RenameOrg(ctx, id, name)
+			case store.ScopeTeam:
+				err = a.tenancy.RenameTeam(ctx, id, name)
+			default:
+				err = a.tenancy.RenameProject(ctx, id, name)
+			}
+			if err != nil {
+				writeError(w, tenancyStatus(err), err.Error())
+				return
+			}
+			out.Name = name
+		}
+		writeJSON(w, http.StatusOK, out)
+	}
+}
+
+func (a *API) tenancyDelete(level store.Scope) http.HandlerFunc {
+	return func(w http.ResponseWriter, r *http.Request) {
+		if !a.tenancyReady(w) {
+			return
+		}
+		ctx := r.Context()
+		_, id, _, err := a.resolveScope(ctx, string(level)+":"+r.PathValue("id"))
+		if err != nil {
+			writeError(w, http.StatusNotFound, err.Error())
+			return
+		}
+		switch level {
+		case store.ScopeOrg:
+			err = a.tenancy.DeleteOrg(ctx, id)
+		case store.ScopeTeam:
+			err = a.tenancy.DeleteTeam(ctx, id)
+		default:
+			err = a.tenancy.DeleteProject(ctx, id)
+		}
+		if err != nil {
+			// Only what is empty goes: the store says what is still in it.
+			writeError(w, tenancyStatus(err), err.Error())
+			return
+		}
+		writeJSON(w, http.StatusOK, map[string]any{"id": id, "deleted": true})
+	}
+}
+
+// keyState is a key's state at an instant.
+func keyState(k store.Key, now time.Time) string {
+	switch {
+	case !k.RevokedAt.IsZero():
+		return "revoked"
+	case !k.ExpiresAt.IsZero() && !now.Before(k.ExpiresAt):
+		return "expired"
+	}
+	return "active"
+}
+
+func keyView(k store.Key, path, email string, now time.Time) KeyView {
+	// The store says "Acme / ml / search"; a path given to the API is
+	// written Acme/ml/search, and is answered the same way.
+	path = strings.ReplaceAll(path, " / ", "/")
+	return KeyView{ID: k.ID, Name: k.Name, Prefix: k.Prefix, Project: k.ProjectID, ProjectPath: path,
+		User: k.UserID, UserEmail: email, Band: k.Band, RoutingProfile: k.RoutingProfile, State: keyState(k, now),
+		CreatedAt: k.CreatedAt, ExpiresAt: k.ExpiresAt, RevokedAt: k.RevokedAt, LastUsedAt: k.LastUsedAt}
+}
+
+func (a *API) keyList(w http.ResponseWriter, r *http.Request) {
+	if !a.tenancyReady(w) {
+		return
+	}
+	project := ""
+	if p := r.URL.Query().Get("project"); p != "" {
+		var err error
+		if _, project, _, err = a.resolveScope(r.Context(), "project:"+p); err != nil {
+			writeError(w, http.StatusNotFound, err.Error())
+			return
+		}
+	}
+	rows, err := a.tenancy.AllKeys(r.Context())
+	if err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	now := time.Now()
+	out := []KeyView{}
+	for _, k := range rows {
+		if project == "" || k.ProjectID == project {
+			out = append(out, keyView(k.Key, k.ProjectPath, k.OwnerEmail, now))
+		}
+	}
+	writeJSON(w, http.StatusOK, out)
+}
+
+func (a *API) keyGet(w http.ResponseWriter, r *http.Request) {
+	if !a.tenancyReady(w) {
+		return
+	}
+	rows, err := a.tenancy.AllKeys(r.Context())
+	if err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	for _, k := range rows {
+		if k.ID == r.PathValue("id") {
+			writeJSON(w, http.StatusOK, keyView(k.Key, k.ProjectPath, k.OwnerEmail, time.Now()))
+			return
+		}
+	}
+	writeError(w, http.StatusNotFound, "no key "+r.PathValue("id"))
+}
+
+// findUser reads a person by ID or email.
+func (a *API) findUser(ctx context.Context, ref string) (store.User, error) {
+	ref = strings.TrimSpace(ref)
+	if u, err := a.tenancy.UserByID(ctx, ref); err == nil {
+		return u, nil
+	}
+	if strings.Contains(ref, "@") {
+		if u, err := a.tenancy.UserByEmail(ctx, ref); err == nil {
+			return u, nil
+		}
+	}
+	return store.User{}, fmt.Errorf("%w: no person %q", store.ErrNotFound, ref)
+}
+
+func (a *API) keyCreate(w http.ResponseWriter, r *http.Request) {
+	if !a.tenancyReady(w) {
+		return
+	}
+	var in KeyCreateRequest
+	if err := json.NewDecoder(r.Body).Decode(&in); err != nil {
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	}
+	ctx := r.Context()
+	if strings.TrimSpace(in.Name) == "" {
+		writeError(w, http.StatusBadRequest, "a key needs a name")
+		return
+	}
+	if in.Project == "" && in.User == "" {
+		writeError(w, http.StatusBadRequest, "a key belongs to a project, a person, or a person in a project: give project, user or both")
+		return
+	}
+	var owner store.User
+	var err error
+	if in.User != "" {
+		if owner, err = a.findUser(ctx, in.User); err != nil {
+			writeError(w, http.StatusNotFound, err.Error())
+			return
+		}
+	}
+	project, path := "", ""
+	if in.Project != "" {
+		if _, project, path, err = a.resolveScope(ctx, "project:"+in.Project); err != nil {
+			writeError(w, http.StatusNotFound, err.Error())
+			return
+		}
+	}
+	var expires time.Duration
+	if in.ExpiresIn != "" {
+		if expires, err = time.ParseDuration(in.ExpiresIn); err != nil || expires <= 0 {
+			writeError(w, http.StatusBadRequest, fmt.Sprintf("expires_in %q is not a positive duration such as 720h", in.ExpiresIn))
+			return
+		}
+	}
+	secret, k, err := store.NewKey(store.NewID("key-"), strings.TrimSpace(in.Name), project)
+	if err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	k.CreatedAt = time.Now()
+	k.Band, k.RoutingProfile = in.Band, in.RoutingProfile
+	if expires > 0 {
+		k.ExpiresAt = k.CreatedAt.Add(expires)
+	}
+	if owner.ID != "" {
+		err = a.tenancy.CreateUserKey(ctx, owner.ID, k)
+	} else {
+		err = a.tenancy.CreateKey(ctx, k)
+	}
+	switch {
+	case errors.Is(err, store.ErrNotMember):
+		writeError(w, http.StatusConflict, fmt.Sprintf("%s is not a member of %s; add the membership first", owner.Email, path))
+		return
+	case errors.Is(err, store.ErrKeyTTL):
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	case err != nil:
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	k.UserID = owner.ID
+	writeJSON(w, http.StatusCreated, KeyCreated{KeyView: keyView(k, path, owner.Email, time.Now()), Secret: secret})
+}
+
+func (a *API) keyRevoke(w http.ResponseWriter, r *http.Request) {
+	if !a.tenancyReady(w) {
+		return
+	}
+	id := r.PathValue("id")
+	if _, err := a.tenancy.KeyByID(r.Context(), id); err != nil {
+		writeError(w, tenancyStatus(err), err.Error())
+		return
+	}
+	if err := a.tenancy.RevokeKey(r.Context(), id, time.Now()); err != nil {
+		writeError(w, tenancyStatus(err), err.Error())
+		return
+	}
+	// A key held in a gateway's cache keeps working until auth.cache_ttl.
+	writeJSON(w, http.StatusOK, map[string]any{"id": id, "revoked": true})
+}
+
+func (a *API) userView(ctx context.Context, u store.User) UserView {
+	role := u.Role
+	if u.IsAdmin {
+		role = "admin"
+	}
+	v := UserView{ID: u.ID, Email: u.Email, Name: u.Name, State: string(u.State), Role: role,
+		Memberships: []MembershipView{}, CreatedAt: u.CreatedAt, LastLoginAt: u.LastLoginAt}
+	if ms, err := a.tenancy.MembershipsOf(ctx, u.ID); err == nil {
+		for _, m := range ms {
+			v.Memberships = append(v.Memberships, MembershipView{Target: string(m.Scope) + ":" + m.ScopeID,
+				Path: m.Path, Role: m.Role, At: m.AddedAt, By: m.AddedBy})
+		}
+	}
+	return v
+}
+
+func (a *API) userList(w http.ResponseWriter, r *http.Request) {
+	if !a.tenancyReady(w) {
+		return
+	}
+	us, err := a.tenancy.Users(r.Context(), store.UserState(r.URL.Query().Get("state")))
+	if err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	out := []UserView{}
+	for _, u := range us {
+		out = append(out, a.userView(r.Context(), u))
+	}
+	writeJSON(w, http.StatusOK, out)
+}
+
+func (a *API) userGet(w http.ResponseWriter, r *http.Request) {
+	if !a.tenancyReady(w) {
+		return
+	}
+	u, err := a.findUser(r.Context(), r.PathValue("id"))
+	if err != nil {
+		writeError(w, http.StatusNotFound, err.Error())
+		return
+	}
+	writeJSON(w, http.StatusOK, a.userView(r.Context(), u))
+}
+
+func (a *API) membership(add bool) http.HandlerFunc {
+	return func(w http.ResponseWriter, r *http.Request) {
+		if !a.tenancyReady(w) {
+			return
+		}
+		var in MembershipRequest
+		if err := json.NewDecoder(r.Body).Decode(&in); err != nil {
+			writeError(w, http.StatusBadRequest, err.Error())
+			return
+		}
+		ctx := r.Context()
+		u, err := a.findUser(ctx, r.PathValue("id"))
+		if err != nil {
+			writeError(w, http.StatusNotFound, err.Error())
+			return
+		}
+		scope, id, _, err := a.resolveScope(ctx, in.Target)
+		if err != nil {
+			writeError(w, http.StatusBadRequest, err.Error())
+			return
+		}
+		if add {
+			role := in.Role
+			if role == "" {
+				role = store.RoleMember
+			}
+			if role != store.RoleMember && role != store.RoleManager {
+				writeError(w, http.StatusBadRequest, fmt.Sprintf("role %q: member or manager", role))
+				return
+			}
+			if err := a.tenancy.AddMembership(ctx, u.ID, scope, id, role, actorOf(r), time.Now()); err != nil {
+				writeError(w, tenancyStatus(err), err.Error())
+				return
+			}
+		} else {
+			// The person's keys in projects they no longer reach are
+			// revoked with it, by the store.
+			if _, err := a.tenancy.RemoveMembership(ctx, u.ID, scope, id, time.Now()); err != nil {
+				if errors.Is(err, store.ErrNotFound) {
+					err = errors.New(u.Email + " is not a member there")
+				}
+				writeError(w, http.StatusNotFound, err.Error())
+				return
+			}
+		}
+		writeJSON(w, http.StatusOK, a.userView(ctx, u))
+	}
+}

internal/control/adminapi/tenancy_test.go

@@ -0,0 +1,171 @@
+package adminapi
+
+import (
+	"context"
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+func TestTheTenancyTreeKeysAndMembershipsAreAdministered(t *testing.T) {
+	ctx := context.Background()
+	a, _ := setup(t, testModels())
+	s, err := store.OpenSQLite(ctx, filepath.Join(t.TempDir(), "s.db"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer s.Close()
+	a.SetQuotas(s)
+	a.SetTenancy(s)
+	mux := http.NewServeMux()
+	a.Register(mux)
+	do := func(method, path, body string, out any) int {
+		t.Helper()
+		rec := httptest.NewRecorder()
+		mux.ServeHTTP(rec, httptest.NewRequest(method, path, strings.NewReader(body)))
+		if out != nil && rec.Code < 300 {
+			if err := json.Unmarshal(rec.Body.Bytes(), out); err != nil {
+				t.Fatalf("%s %s: %v (%s)", method, path, err, rec.Body.String())
+			}
+		}
+		if rec.Code >= 300 {
+			t.Logf("%s %s: %d %s", method, path, rec.Code, strings.TrimSpace(rec.Body.String()))
+		}
+		return rec.Code
+	}
+	var made TenancyItem
+	if c := do("POST", "/api/v1/orgs", `{"name":"Acme"}`, &made); c != http.StatusCreated {
+		t.Fatalf("adding an organization: %d", c)
+	}
+	acme := made.ID
+	if c := do("POST", "/api/v1/orgs", `{"name":"Acme"}`, nil); c != http.StatusConflict {
+		t.Errorf("a second Acme: %d", c)
+	}
+	if c := do("POST", "/api/v1/teams", `{"name":"hr"}`, nil); c != http.StatusBadRequest {
+		t.Errorf("a team in nothing: %d", c)
+	}
+	do("POST", "/api/v1/teams", `{"name":"hr","in":"Acme"}`, &made)
+	hr := made.ID
+	if c := do("POST", "/api/v1/projects", `{"name":"crm","in":"Acme/hr"}`, &made); c != http.StatusCreated || made.In != hr {
+		t.Fatalf("adding a project: %d %v", c, made)
+	}
+	crm := made.ID
+	do("POST", "/api/v1/orgs", `{"name":"Other"}`, &made)
+	other := made.ID
+
+	// Renamed, and moved with what is in it.
+	if c := do("PATCH", "/api/v1/projects/"+crm, `{"name":"crm2"}`, nil); c != http.StatusOK {
+		t.Errorf("renaming: %d", c)
+	}
+	if c := do("PATCH", "/api/v1/teams/"+hr, `{"in":"Other"}`, nil); c != http.StatusOK {
+		t.Errorf("moving: %d", c)
+	}
+	if c := do("PATCH", "/api/v1/orgs/"+acme, `{"in":"Other"}`, nil); c != http.StatusBadRequest {
+		t.Errorf("moving an organization: %d", c)
+	}
+	var tree []OrgView
+	do("GET", "/api/v1/orgs", "", &tree)
+	if len(tree) != 2 || tree[1].ID != other || len(tree[1].Teams) != 1 ||
+		tree[1].Teams[0].Projects[0].Path != "Other/hr/crm2" {
+		t.Fatalf("the tree is %+v", tree)
+	}
+
+	// A project's key: given once, listed without it.
+	var key KeyCreated
+	rec := httptest.NewRecorder()
+	mux.ServeHTTP(rec, httptest.NewRequest("POST", "/api/v1/keys",
+		strings.NewReader(`{"name":"svc","project":"Other/hr/crm2","expires_in":"24h"}`)))
+	if rec.Code != http.StatusCreated {
+		t.Fatalf("issuing a key: %d %s", rec.Code, rec.Body.String())
+	}
+	_ = json.Unmarshal(rec.Body.Bytes(), &key)
+	if !strings.HasPrefix(key.Secret, "sk-llmgw-") || key.Project != crm || key.ExpiresAt.IsZero() {
+		t.Errorf("the key issued: %+v", key)
+	}
+	if k, err := s.KeyByHash(ctx, store.HashKey(key.Secret)); err != nil || k.ID != key.ID {
+		t.Errorf("the key does not authenticate: %v", err)
+	}
+	var keys []KeyView
+	do("GET", "/api/v1/keys?project="+crm, "", &keys)
+	if len(keys) != 1 || keys[0].ID != key.ID || keys[0].State != "active" || keys[0].ProjectPath != "Other/hr/crm2" {
+		t.Errorf("the project's keys: %+v", keys)
+	}
+	raw := httptest.NewRecorder()
+	mux.ServeHTTP(raw, httptest.NewRequest("GET", "/api/v1/keys", nil))
+	if strings.Contains(raw.Body.String(), key.Secret) || strings.Contains(raw.Body.String(), "secret") {
+		t.Error("a listing holds the key")
+	}
+	if c := do("POST", "/api/v1/keys", `{"name":"x","project":"Other/hr/crm2","expires_in":"soon"}`, nil); c != http.StatusBadRequest {
+		t.Errorf("a lifetime that is not one: %d", c)
+	}
+	if c := do("DELETE", "/api/v1/projects/"+crm, "", nil); c != http.StatusConflict {
+		t.Errorf("removing a project with a key: %d", c)
+	}
+	if c := do("DELETE", "/api/v1/keys/"+key.ID, "", nil); c != http.StatusOK {
+		t.Errorf("revoking: %d", c)
+	}
+	var one KeyView
+	do("GET", "/api/v1/keys/"+key.ID, "", &one)
+	if one.State != "revoked" {
+		t.Errorf("after revoking, %+v", one)
+	}
+
+	// A person: a key in a project only as its member, revoked when they
+	// leave it.
+	if err := s.CreateUser(ctx, store.User{ID: "u1", Email: "ayse@example.com", PasswordHash: "x",
+		State: store.UserActive, CreatedAt: time.Now()}); err != nil {
+		t.Fatal(err)
+	}
+	if c := do("POST", "/api/v1/keys", `{"name":"mine","project":"`+crm+`","user":"ayse@example.com"}`, nil); c != http.StatusConflict {
+		t.Errorf("a key where its person is no member: %d", c)
+	}
+	var u UserView
+	if c := do("POST", "/api/v1/users/ayse@example.com/memberships", `{"target":"project:Other/hr/crm2"}`, &u); c != http.StatusOK ||
+		len(u.Memberships) != 1 || u.Memberships[0].Target != "project:"+crm || u.Memberships[0].Role != "member" {
+		t.Fatalf("joining: %d %+v", c, u)
+	}
+	var mine KeyCreated
+	rec = httptest.NewRecorder()
+	mux.ServeHTTP(rec, httptest.NewRequest("POST", "/api/v1/keys",
+		strings.NewReader(`{"name":"mine","project":"`+crm+`","user":"u1"}`)))
+	_ = json.Unmarshal(rec.Body.Bytes(), &mine)
+	if rec.Code != http.StatusCreated || mine.User != "u1" {
+		t.Fatalf("a member's key: %d %s", rec.Code, rec.Body.String())
+	}
+	if c := do("DELETE", "/api/v1/users/u1/memberships", `{"target":"project:`+crm+`"}`, &u); c != http.StatusOK || len(u.Memberships) != 0 {
+		t.Errorf("leaving: %d %+v", c, u)
+	}
+	do("GET", "/api/v1/keys/"+mine.ID, "", &one)
+	if one.State != "revoked" {
+		t.Errorf("the key of a person who left: %+v", one)
+	}
+	if c := do("DELETE", "/api/v1/users/u1/memberships", `{"target":"project:`+crm+`"}`, nil); c != http.StatusNotFound {
+		t.Errorf("leaving twice: %d", c)
+	}
+	var people []UserView
+	do("GET", "/api/v1/users", "", &people)
+	if len(people) != 1 || people[0].Email != "ayse@example.com" {
+		t.Errorf("people: %+v", people)
+	}
+
+	// Only what is empty is removed.
+	if c := do("DELETE", "/api/v1/orgs/Acme", "", nil); c != http.StatusOK {
+		t.Errorf("removing the empty Acme by name: %d", c)
+	}
+	if c := do("DELETE", "/api/v1/orgs/"+other, "", nil); c != http.StatusConflict {
+		t.Errorf("removing an organization with a team: %d", c)
+	}
+}
+
+func TestTenancyNeedsTheStore(t *testing.T) {
+	a, _ := setup(t, testModels())
+	if c := call(t, a, "GET", "/api/v1/orgs", "", nil); c != http.StatusNotImplemented {
+		t.Errorf("without a store: %d", c)
+	}
+}

internal/control/adminapi/train.go

@@ -0,0 +1,559 @@
+package adminapi
+
+import (
+	"archive/tar"
+	"bufio"
+	"bytes"
+	"compress/gzip"
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"net/http"
+	"regexp"
+	"slices"
+	"strconv"
+	"strings"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/train"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// Training LoRA adapters (BACKLOG F4), administered: a job names the model
+// the adapter is for, the weights to train on, the node and how much of a
+// card it needs, and the pairs; the node trains when it can, and what it
+// made is kept here to download or publish.
+
+// TrainStore is where jobs are kept.
+type TrainStore interface {
+	CreateTrainJob(ctx context.Context, j store.TrainJob) error
+	TrainJobs(ctx context.Context, limit int) ([]store.TrainJob, error)
+	TrainJob(ctx context.Context, id string) (store.TrainJob, error)
+	TrainArtifact(ctx context.Context, job, kind string) ([]byte, error)
+	TrainArtifactKinds(ctx context.Context, job string) ([]string, error)
+	DeleteTrainJob(ctx context.Context, id string) error
+}
+
+// TrainAdmin is what the training endpoints need.
+type TrainAdmin struct {
+	Store   TrainStore
+	Service *train.Service
+	// Mirror keeps what a job made where the nodes fetch weights; nil
+	// leaves publishing off.
+	Mirror TrainMirror
+}
+
+// TrainMirror is the model mirror, as publishing needs it.
+type TrainMirror interface {
+	Upload(ctx context.Context, model, version, file string, body []byte, source string) (mirror.FetchResponse, error)
+}
+
+// TrainPublishView is an adapter published: where the nodes fetch it from,
+// and what to do for replicas to load it.
+type TrainPublishView struct {
+	Adapter string `json:"adapter"`
+	Model   string `json:"model"`
+	Source  string `json:"source"`
+	// Owner is who sees it; empty is every caller of the model.
+	Owner string `json:"owner,omitempty"`
+	Note  string `json:"note,omitempty"`
+}
+
+// SetTrain enables the training endpoints.
+func (a *API) SetTrain(t *TrainAdmin) { a.train = t }
+
+// SetTrainMirror gives publishing the mirror, which is set up after the
+// training endpoints are.
+func (a *API) SetTrainMirror(m TrainMirror) {
+	if a.train != nil {
+		a.train.Mirror = m
+	}
+}
+
+// TrainRequest queues a job.
+type TrainRequest struct {
+	// Name is the adapter's: what requests will ask for it by.
+	Name string `json:"name"`
+	// Model is the catalog model the adapter is for.
+	Model string `json:"model"`
+	// BaseRepo and BaseRevision are the weights trained on: the model's own
+	// on its registry, in safetensors (Hugging Face), not a GGUF.
+	BaseRepo     string `json:"base_repo"`
+	BaseRevision string `json:"base_revision,omitempty"`
+	// Node trains it; GPUMemoryMiB is what it needs of a card, which can be
+	// what an engine there leaves free.
+	Node         string `json:"node"`
+	GPUMemoryMiB int    `json:"gpu_memory_mib"`
+	// Owner is who the adapter will belong to: org:, team:, project: (by
+	// path or ID), user: or key:. Only callers there see it. When the
+	// pairs come from one knowledge base, it defaults to that collection's
+	// owner; it must be able to read every collection they come from.
+	Owner string `json:"owner,omitempty"`
+	// Public makes the adapter every caller's its model allows. Refused for
+	// pairs from a knowledge base.
+	Public bool `json:"public,omitempty"`
+	// Data is the pairs, NDJSON of {"q", "a"} (llmgw ctl kb dataset).
+	Data string `json:"data"`
+	// Params tune the trainer: rank, alpha, dropout, epochs, lr, batch,
+	// max_len, seed.
+	Params map[string]any `json:"params,omitempty"`
+	// GGUF also makes the adapter in llama.cpp's format; when absent, it
+	// does when the model is served by llama.cpp.
+	GGUF *bool `json:"gguf,omitempty"`
+}
+
+// TrainJobView is a job as listed.
+type TrainJobView struct {
+	ID           string    `json:"id"`
+	Name         string    `json:"name"`
+	Model        string    `json:"model"`
+	BaseRepo     string    `json:"base_repo"`
+	BaseRevision string    `json:"base_revision,omitempty"`
+	Node         string    `json:"node"`
+	Owner        string    `json:"owner,omitempty"`
+	GPUMemoryMiB int       `json:"gpu_memory_mib"`
+	Params       string    `json:"params,omitempty"`
+	GGUF         bool      `json:"gguf"`
+	Pairs        int       `json:"pairs"`
+	State        string    `json:"state"`
+	Step         int       `json:"step"`
+	Steps        int       `json:"steps"`
+	Loss         float64   `json:"loss"`
+	Message      string    `json:"message,omitempty"`
+	PeakMiB      int       `json:"peak_mib"`
+	CreatedAt    time.Time `json:"created_at"`
+	StartedAt    time.Time `json:"started_at,omitzero"`
+	FinishedAt   time.Time `json:"finished_at,omitzero"`
+	CreatedBy    string    `json:"created_by,omitempty"`
+	// Artifacts are what it made: peft (a gzipped tar), gguf.
+	Artifacts []string `json:"artifacts,omitempty"`
+}
+
+func trainView(j store.TrainJob) TrainJobView {
+	return TrainJobView{ID: j.ID, Name: j.Name, Model: j.Model, BaseRepo: j.BaseRepo, BaseRevision: j.BaseRevision,
+		Node: j.Node, Owner: j.Owner, GPUMemoryMiB: j.GPUMemoryMiB, Params: j.Params, GGUF: j.GGUF, Pairs: j.Pairs, State: j.State,
+		Step: j.Step, Steps: j.Steps, Loss: j.Loss, Message: j.Message, PeakMiB: j.PeakMiB, CreatedAt: j.CreatedAt,
+		StartedAt: j.StartedAt, FinishedAt: j.FinishedAt, CreatedBy: j.CreatedBy}
+}
+
+func (a *API) registerTrain(mux Mux) {
+	mux.HandleFunc("POST /api/v1/train/jobs", a.trainCreate)
+	mux.HandleFunc("GET /api/v1/train/jobs", a.trainList)
+	mux.HandleFunc("GET /api/v1/train/jobs/{id}", a.trainGet)
+	mux.HandleFunc("POST /api/v1/train/jobs/{id}/cancel", a.trainCancel)
+	mux.HandleFunc("DELETE /api/v1/train/jobs/{id}", a.trainDelete)
+	mux.HandleFunc("GET /api/v1/train/jobs/{id}/artifacts/{kind}", a.trainArtifact)
+	mux.HandleFunc("POST /api/v1/train/jobs/{id}/publish", a.trainPublish)
+}
+
+func (a *API) trainReady(w http.ResponseWriter) bool {
+	if a.train == nil {
+		writeError(w, http.StatusNotImplemented, "training needs the store and the nodes' agents, which this control plane runs without")
+		return false
+	}
+	return true
+}
+
+// trainMaxData bounds a job's pairs.
+const trainMaxData = 64 << 20
+
+var adapterName = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._-]{0,99}$`)
+
+// trainParams are the trainer's settings, with what each may be.
+var trainParams = map[string][2]float64{
+	"rank": {1, 256}, "alpha": {1, 1024}, "dropout": {0, 0.9}, "epochs": {1, 100},
+	"lr": {1e-7, 1e-2}, "batch": {1, 128}, "max_len": {16, 32768}, "seed": {0, 1 << 31},
+}
+
+func (a *API) trainCreate(w http.ResponseWriter, r *http.Request) {
+	if !a.trainReady(w) {
+		return
+	}
+	var in TrainRequest
+	if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, trainMaxData+1<<20)).Decode(&in); err != nil {
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	}
+	bad := func(format string, args ...any) {
+		writeError(w, http.StatusBadRequest, fmt.Sprintf(format, args...))
+	}
+	if !adapterName.MatchString(in.Name) {
+		bad("an adapter's name is letters, digits, . _ and -, up to 100: %q is not", in.Name)
+		return
+	}
+	m, ok := a.ctl.Catalog().Resolve(in.Model)
+	if !ok {
+		bad("no model %s in the catalog", in.Model)
+		return
+	}
+	if _, _, _, taken := a.ctl.Catalog().ResolveAdapter(in.Name); taken || in.Name == m.Name {
+		bad("%s is already a model's or an adapter's name", in.Name)
+		return
+	}
+	if strings.Count(in.BaseRepo, "/") != 1 || strings.ContainsAny(in.BaseRepo, " :") {
+		bad("base_repo is the weights' registry repository, owner/name: %q is not", in.BaseRepo)
+		return
+	}
+	if _, ok := a.ctl.Inventory().Get(in.Node); !ok {
+		bad("no node %s", in.Node)
+		return
+	}
+	if in.GPUMemoryMiB <= 0 {
+		bad("gpu_memory_mib is what the job needs of a card, in MiB")
+		return
+	}
+	pairs, colls, err := countPairs(in.Data)
+	if err != nil {
+		bad("%v", err)
+		return
+	}
+	owner, err := a.trainOwner(r.Context(), in, colls)
+	if err != nil {
+		bad("%v", err)
+		return
+	}
+	for k, v := range in.Params {
+		lim, known := trainParams[k]
+		f, isNum := v.(float64)
+		if !known || !isNum || f < lim[0] || f > lim[1] {
+			bad("params.%s: the trainer takes rank, alpha, dropout, epochs, lr, batch, max_len and seed, each a number in its range", k)
+			return
+		}
+	}
+	params := ""
+	if len(in.Params) > 0 {
+		b, _ := json.Marshal(in.Params)
+		params = string(b)
+	}
+	gguf := false
+	for _, p := range m.Profiles {
+		if p.Engine == "llamacpp" {
+			gguf = true
+		}
+	}
+	if in.GGUF != nil {
+		gguf = *in.GGUF
+	}
+	j := store.TrainJob{ID: store.NewID("train-"), Name: in.Name, Model: m.Name, BaseRepo: in.BaseRepo,
+		BaseRevision: in.BaseRevision, Node: in.Node, Owner: owner, GPUMemoryMiB: in.GPUMemoryMiB, Params: params, GGUF: gguf,
+		Data: []byte(in.Data), Pairs: pairs, CreatedBy: actorOf(r)}
+	if err := a.train.Store.CreateTrainJob(r.Context(), j); err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	// At once rather than at the next look, when the node is here.
+	go a.train.Service.Dispatch(context.WithoutCancel(r.Context()))
+	j.State = store.TrainQueued
+	writeJSON(w, http.StatusCreated, trainView(j))
+}
+
+// trainOwner is who a job's adapter will belong to, as scope:ID: one that
+// may read every knowledge base the pairs come from. Pairs from a knowledge
+// base never make an adapter everyone's.
+func (a *API) trainOwner(ctx context.Context, in TrainRequest, colls []string) (string, error) {
+	if in.Public {
+		if len(colls) > 0 {
+			return "", errors.New("these pairs come from a knowledge base, and an adapter trained on them " +
+				"carries some of what it says: it belongs to a part of the tree that may read it, not to everyone")
+		}
+		if in.Owner != "" {
+			return "", errors.New("an adapter is public or has an owner, not both")
+		}
+		return "", nil
+	}
+	if len(colls) > 0 && a.kb == nil {
+		return "", errors.New("the pairs name knowledge bases, which this control plane cannot read")
+	}
+	var found []store.KBCollection
+	for _, id := range colls {
+		c, err := a.kb.Store.Collection(ctx, id)
+		if err != nil {
+			return "", fmt.Errorf("the pairs come from collection %s, which is not here: %w", id, err)
+		}
+		found = append(found, c)
+	}
+	if in.Owner == "" {
+		if len(found) == 1 {
+			return string(found[0].OwnerScope) + ":" + found[0].OwnerID, nil
+		}
+		return "", errors.New("name the adapter's owner (org:, team:, project:, user: or key:), or make it public " +
+			"when its pairs are not a knowledge base's")
+	}
+	scope, id, err := a.kbTarget(ctx, in.Owner)
+	if err != nil {
+		return "", err
+	}
+	for _, c := range found {
+		e, err := a.kb.Store.EffectiveFor(ctx, scope, id, c.OrgID)
+		if err != nil {
+			return "", fmt.Errorf("%s: %w", in.Owner, err)
+		}
+		readable, err := a.kb.Store.CollectionsFor(ctx, e, false)
+		if err != nil {
+			return "", err
+		}
+		if !slices.ContainsFunc(readable, func(x store.KBCollection) bool { return x.ID == c.ID }) {
+			return "", fmt.Errorf("%s may not read collection %s, which the pairs come from: "+
+				"an adapter trained on it would tell its callers what it says", in.Owner, c.Name)
+		}
+	}
+	return string(scope) + ":" + id, nil
+}
+
+// countPairs checks a job's data: NDJSON of {"q", "a"}, at least one, and
+// says which knowledge bases the pairs come from.
+func countPairs(data string) (int, []string, error) {
+	if len(data) > trainMaxData {
+		return 0, nil, fmt.Errorf("the data is over %d MiB", trainMaxData>>20)
+	}
+	seen := map[string]bool{}
+	var colls []string
+	n := 0
+	sc := bufio.NewScanner(strings.NewReader(data))
+	sc.Buffer(make([]byte, 64<<10), 1<<20)
+	line := 0
+	for sc.Scan() {
+		line++
+		b := bytes.TrimSpace(sc.Bytes())
+		if len(b) == 0 {
+			continue
+		}
+		var p struct{ Q, A, Collection string }
+		if err := json.Unmarshal(b, &p); err != nil {
+			return 0, nil, fmt.Errorf("data line %d is not JSON: %v", line, err)
+		}
+		if strings.TrimSpace(p.Q) == "" || strings.TrimSpace(p.A) == "" {
+			return 0, nil, fmt.Errorf("data line %d has no q or no a", line)
+		}
+		if p.Collection != "" && !seen[p.Collection] {
+			seen[p.Collection] = true
+			colls = append(colls, p.Collection)
+		}
+		n++
+	}
+	if err := sc.Err(); err != nil {
+		return 0, nil, err
+	}
+	if n == 0 {
+		return 0, nil, errors.New("the data holds no pairs: NDJSON of {\"q\", \"a\"}, as llmgw ctl kb dataset writes")
+	}
+	return n, colls, nil
+}
+
+func (a *API) trainList(w http.ResponseWriter, r *http.Request) {
+	if !a.trainReady(w) {
+		return
+	}
+	limit := 50
+	if n, err := strconv.Atoi(r.URL.Query().Get("limit")); err == nil && n > 0 && n <= 500 {
+		limit = n
+	}
+	jobs, err := a.train.Store.TrainJobs(r.Context(), limit)
+	if err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	out := []TrainJobView{}
+	for _, j := range jobs {
+		out = append(out, trainView(j))
+	}
+	writeJSON(w, http.StatusOK, out)
+}
+
+func (a *API) trainJob(w http.ResponseWriter, r *http.Request) (store.TrainJob, bool) {
+	if !a.trainReady(w) {
+		return store.TrainJob{}, false
+	}
+	j, err := a.train.Store.TrainJob(r.Context(), r.PathValue("id"))
+	if errors.Is(err, store.ErrNotFound) {
+		writeError(w, http.StatusNotFound, "no training job "+r.PathValue("id"))
+		return j, false
+	}
+	if err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return j, false
+	}
+	return j, true
+}
+
+func (a *API) trainGet(w http.ResponseWriter, r *http.Request) {
+	j, ok := a.trainJob(w, r)
+	if !ok {
+		return
+	}
+	v := trainView(j)
+	v.Artifacts, _ = a.train.Store.TrainArtifactKinds(r.Context(), j.ID)
+	writeJSON(w, http.StatusOK, v)
+}
+
+func (a *API) trainCancel(w http.ResponseWriter, r *http.Request) {
+	j, ok := a.trainJob(w, r)
+	if !ok {
+		return
+	}
+	if err := a.train.Service.Cancel(r.Context(), j.ID); err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	j, _ = a.train.Store.TrainJob(r.Context(), j.ID)
+	writeJSON(w, http.StatusOK, trainView(j))
+}
+
+func (a *API) trainDelete(w http.ResponseWriter, r *http.Request) {
+	j, ok := a.trainJob(w, r)
+	if !ok {
+		return
+	}
+	if !store.TrainOver(j.State) {
+		writeError(w, http.StatusConflict, "the job has not ended: cancel it first")
+		return
+	}
+	if err := a.train.Store.DeleteTrainJob(r.Context(), j.ID); err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	writeJSON(w, http.StatusOK, map[string]any{"id": j.ID, "deleted": true})
+}
+
+func (a *API) trainArtifact(w http.ResponseWriter, r *http.Request) {
+	j, ok := a.trainJob(w, r)
+	if !ok {
+		return
+	}
+	kind := r.PathValue("kind")
+	b, err := a.train.Store.TrainArtifact(r.Context(), j.ID, kind)
+	if errors.Is(err, store.ErrNotFound) {
+		writeError(w, http.StatusNotFound, "the job made no "+kind)
+		return
+	}
+	if err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	name := j.Name + ".tar.gz"
+	w.Header().Set("Content-Type", "application/gzip")
+	if kind == "gguf" {
+		name = j.Name + ".gguf"
+		w.Header().Set("Content-Type", "application/octet-stream")
+	}
+	w.Header().Set("Content-Disposition", `attachment; filename="`+name+`"`)
+	w.Header().Set("Content-Length", strconv.Itoa(len(b)))
+	_, _ = w.Write(b)
+}
+
+// trainPublish puts what a job made on the mirror and makes it the model's
+// adapter: the GGUF for a model llama.cpp serves, the PEFT directory for
+// one vLLM serves.
+func (a *API) trainPublish(w http.ResponseWriter, r *http.Request) {
+	j, ok := a.trainJob(w, r)
+	if !ok {
+		return
+	}
+	if a.train.Mirror == nil {
+		writeError(w, http.StatusNotImplemented, "publishing needs the model mirror (mirror.api in the settings); "+
+			"llmgw ctl train download, then llmgw mirror import there, does it by hand")
+		return
+	}
+	if a.models == nil {
+		writeError(w, http.StatusNotImplemented, "this control plane has no store for the catalog")
+		return
+	}
+	if j.State != store.TrainDone {
+		writeError(w, http.StatusConflict, "the job has not finished: it is "+j.State)
+		return
+	}
+	m, found := a.ctl.Catalog().Resolve(j.Model)
+	if !found {
+		writeError(w, http.StatusConflict, "the model "+j.Model+" is no longer in the catalog")
+		return
+	}
+	for _, ad := range m.Adapters {
+		if ad.Name == j.Name {
+			writeError(w, http.StatusConflict, j.Name+" is already "+m.Name+"'s adapter; train under another name")
+			return
+		}
+	}
+	llama := slices.ContainsFunc(m.Profiles, func(p catalog.Profile) bool { return p.Engine == "llamacpp" })
+	source := "trained by llmgw job " + j.ID + " on " + j.Node + " from " + j.BaseRepo
+	var url string
+	if llama {
+		b, err := a.train.Store.TrainArtifact(r.Context(), j.ID, "gguf")
+		if err != nil {
+			writeError(w, http.StatusConflict, "the job made no GGUF, which llama.cpp needs; train again without --no-gguf")
+			return
+		}
+		res, err := a.train.Mirror.Upload(r.Context(), j.Name, j.ID, j.Name+".gguf", b, source)
+		if err != nil {
+			writeError(w, http.StatusBadGateway, "keeping it on the mirror: "+err.Error())
+			return
+		}
+		url = res.URL
+	} else {
+		b, err := a.train.Store.TrainArtifact(r.Context(), j.ID, "peft")
+		if err != nil {
+			writeError(w, http.StatusConflict, "the job made no PEFT adapter")
+			return
+		}
+		files, err := untarAll(b)
+		if err != nil {
+			writeError(w, http.StatusInternalServerError, err.Error())
+			return
+		}
+		for name, data := range files {
+			res, err := a.train.Mirror.Upload(r.Context(), j.Name, j.ID, name, data, source)
+			if err != nil {
+				writeError(w, http.StatusBadGateway, "keeping "+name+" on the mirror: "+err.Error())
+				return
+			}
+			url = res.URL[:strings.LastIndex(res.URL, "/")+1]
+		}
+	}
+	ad := catalog.Adapter{Name: j.Name, Source: url, Owner: j.Owner}
+	if err := checkAdapters(r.Context(), m, []catalog.Adapter{ad}); err != nil {
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	}
+	if _, err := a.models.SetAdapters(r.Context(), m.Name, append(slices.Clone(m.Adapters), ad), actorOf(r)); err != nil {
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	}
+	v := TrainPublishView{Adapter: j.Name, Model: m.Name, Source: url, Owner: j.Owner}
+	if llama {
+		v.Note = "llama.cpp loads adapters when a replica starts: restart " + m.Name + "'s replicas (llmgw ctl restart " +
+			m.Name + ") and ask for model " + j.Name
+	} else {
+		v.Note = "ask for model " + j.Name + "; a replica loads it on the first request"
+	}
+	writeJSON(w, http.StatusOK, v)
+}
+
+// untarAll reads a gzipped tar of a PEFT directory's files.
+func untarAll(b []byte) (map[string][]byte, error) {
+	gz, err := gzip.NewReader(bytes.NewReader(b))
+	if err != nil {
+		return nil, err
+	}
+	tr := tar.NewReader(gz)
+	out := map[string][]byte{}
+	for {
+		h, err := tr.Next()
+		if errors.Is(err, io.EOF) {
+			return out, nil
+		}
+		if err != nil {
+			return nil, err
+		}
+		if h.Typeflag != tar.TypeReg || strings.ContainsAny(h.Name, "/\\") {
+			continue
+		}
+		data, err := io.ReadAll(io.LimitReader(tr, 4<<30))
+		if err != nil {
+			return nil, err
+		}
+		out[h.Name] = data
+	}
+}

internal/control/adminapi/train_test.go

@@ -0,0 +1,145 @@
+package adminapi
+
+import (
+	"context"
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/train"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+type noAgents struct{}
+
+func (noAgents) Send(string, *llmgwv1.ControlMessage) error { return nil }
+func (noAgents) Connected(string) bool                      { return false }
+
+func TestATrainingJobIsCheckedBeforeItIsQueued(t *testing.T) {
+	models := append(testModels(), catalog.Model{Name: "small", Source: "/s",
+		Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp"}}})
+	a, _ := setup(t, models, gpuNode("cuda01", 1))
+	s, err := store.OpenSQLite(context.Background(), filepath.Join(t.TempDir(), "s.db"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer s.Close()
+	a.SetTrain(&TrainAdmin{Store: s, Service: train.New(s, noAgents{}, nil)})
+	mux := http.NewServeMux()
+	a.Register(mux)
+	post := func(body string) (int, string) {
+		rec := httptest.NewRecorder()
+		mux.ServeHTTP(rec, httptest.NewRequest("POST", "/api/v1/train/jobs", strings.NewReader(body)))
+		return rec.Code, rec.Body.String()
+	}
+	good := map[string]any{"name": "tone", "model": "small", "base_repo": "Qwen/Qwen2.5-0.5B-Instruct", "node": "cuda01",
+		"gpu_memory_mib": 2300, "data": "{\"q\":\"a\",\"a\":\"b\"}\n{\"q\":\"c\",\"a\":\"d\"}\n", "params": map[string]any{"epochs": 3},
+		"public": true}
+	with := func(k string, v any) string {
+		m := map[string]any{}
+		for kk, vv := range good {
+			m[kk] = vv
+		}
+		m[k] = v
+		b, _ := json.Marshal(m)
+		return string(b)
+	}
+	for _, c := range []struct {
+		field string
+		value any
+		says  string
+	}{
+		{"name", "bad name", "letters, digits"},
+		{"model", "nope", "no model nope"},
+		{"name", "small", "already a model"},
+		{"base_repo", "Qwen", "owner/name"},
+		{"node", "gone", "no node gone"},
+		{"data", "{\"q\":\"a\"}\n", "no q or no a"},
+		{"data", "", "no pairs"},
+		{"params", map[string]any{"epochs": 0}, "params.epochs"},
+		{"params", map[string]any{"momentum": 0.9}, "params.momentum"},
+		{"public", false, "name the adapter's owner"},
+		{"data", "{\"q\":\"a\",\"a\":\"b\",\"collection\":\"kb_aaaaaaaaaaaaaaaa\"}\n", "from a knowledge base"},
+	} {
+		if code, body := post(with(c.field, c.value)); code != http.StatusBadRequest || !strings.Contains(body, c.says) {
+			t.Errorf("%s=%v: %d %s", c.field, c.value, code, body)
+		}
+	}
+	code, body := post(with("name", "tone"))
+	var v TrainJobView
+	_ = json.Unmarshal([]byte(body), &v)
+	if code != http.StatusCreated || v.Pairs != 2 || !v.GGUF || v.State != "queued" || v.Params != `{"epochs":3}` {
+		t.Fatalf("%d %s", code, body)
+	}
+	rec := httptest.NewRecorder()
+	mux.ServeHTTP(rec, httptest.NewRequest("DELETE", "/api/v1/train/jobs/"+v.ID, nil))
+	if rec.Code != http.StatusConflict {
+		t.Errorf("deleting a queued job: %d", rec.Code)
+	}
+	rec = httptest.NewRecorder()
+	mux.ServeHTTP(rec, httptest.NewRequest("GET", "/api/v1/train/jobs/"+v.ID+"/artifacts/peft", nil))
+	if rec.Code != http.StatusNotFound {
+		t.Errorf("an artifact not made: %d", rec.Code)
+	}
+}
+
+func TestAnAdapterTrainedOnAKnowledgeBaseBelongsToWhoMayReadIt(t *testing.T) {
+	dsn := os.Getenv("LLMGW_TEST_POSTGRES")
+	if dsn == "" {
+		t.Skip("set LLMGW_TEST_POSTGRES to a pgvector database to run this")
+	}
+	ctx := context.Background()
+	s, err := store.OpenPostgres(ctx, dsn)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer s.Close()
+	if s.KnowledgeAvailable() != nil {
+		t.Skip("no pgvector")
+	}
+	models := append(testModels(), catalog.Model{Name: "small", Source: "/s",
+		Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp"}}})
+	a, _ := setup(t, models, gpuNode("cuda01", 1))
+	a.SetQuotas(s)
+	a.SetKB(&KBAdmin{Store: s})
+	a.SetTrain(&TrainAdmin{Store: s, Service: train.New(s, noAgents{}, nil)})
+	sfx := strings.ToLower(store.NewID(""))
+	org, _ := s.CreateOrg(ctx, "Tr"+sfx)
+	hr, _ := s.CreateTeam(ctx, org, "hr")
+	_, _ = s.CreateProject(ctx, hr, "docs")
+	sales, _ := s.CreateTeam(ctx, org, "sales")
+	_, _ = s.CreateProject(ctx, sales, "crm")
+	c, err := s.CreateCollection(ctx, store.KBCollection{OwnerScope: store.ScopeTeam, OwnerID: hr, Name: "p", EmbedModel: "bge-m3", Dims: 3})
+	if err != nil {
+		t.Fatal(err)
+	}
+	mux := http.NewServeMux()
+	a.Register(mux)
+	post := func(extra string) (int, TrainJobView, string) {
+		body := `{"name":"tone-` + store.NewID("") + `","model":"small","base_repo":"Qwen/Q","node":"cuda01","gpu_memory_mib":2000,` +
+			`"data":"{\"q\":\"a\",\"a\":\"b\",\"collection\":\"` + c.ID + `\"}\n"` + extra + `}`
+		rec := httptest.NewRecorder()
+		mux.ServeHTTP(rec, httptest.NewRequest("POST", "/api/v1/train/jobs", strings.NewReader(body)))
+		var v TrainJobView
+		_ = json.Unmarshal(rec.Body.Bytes(), &v)
+		return rec.Code, v, rec.Body.String()
+	}
+	if code, v, body := post(""); code != http.StatusCreated || v.Owner != "team:"+hr {
+		t.Errorf("by default: %d %s", code, body)
+	}
+	if code, v, body := post(`,"owner":"project:Tr` + sfx + `/hr/docs"`); code != http.StatusCreated || !strings.HasPrefix(v.Owner, "project:") {
+		t.Errorf("a project of the team: %d %s", code, body)
+	}
+	if code, _, body := post(`,"owner":"project:Tr` + sfx + `/sales/crm"`); code != http.StatusBadRequest || !strings.Contains(body, "may not read") {
+		t.Errorf("a project that cannot read it: %d %s", code, body)
+	}
+	if code, _, body := post(`,"public":true`); code != http.StatusBadRequest || !strings.Contains(body, "not to everyone") {
+		t.Errorf("public: %d %s", code, body)
+	}
+}

internal/control/agentsvc/agentsettings.go

@@ -0,0 +1,50 @@
+package agentsvc
+
+import (
+	"time"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+)
+
+// SetAgentSettings gives the service each node's agent settings from the
+// cluster's (BACKLOG E3), as YAML, read at each use so a change is what is
+// sent next.
+func (s *Service) SetAgentSettings(f func(node string) string) {
+	s.mu.Lock()
+	s.agentSettings = f
+	s.mu.Unlock()
+}
+
+func (s *Service) agentSettingsMsg(node string) *llmgwv1.ControlMessage {
+	s.mu.RLock()
+	f := s.agentSettings
+	s.mu.RUnlock()
+	if f == nil {
+		return nil
+	}
+	return &llmgwv1.ControlMessage{Msg: &llmgwv1.ControlMessage_AgentSettings{
+		AgentSettings: &llmgwv1.AgentSettings{Yaml: f(node)}}}
+}
+
+// agentSettingsGap is the pause between telling one node and the next: an
+// agent whose settings changed starts again, and a whole cluster's agents
+// doing so in the same second would all re-register at once.
+var agentSettingsGap = time.Second
+
+// BroadcastAgentSettings tells every connected agent what the cluster's
+// settings now say it runs on, a moment apart. A node not connected is told
+// when it registers.
+func (s *Service) BroadcastAgentSettings() {
+	for _, node := range s.ConnectedNodes() {
+		msg := s.agentSettingsMsg(node)
+		if msg == nil {
+			return
+		}
+		if err := s.send(node, msg); err != nil {
+			s.log.Warn("could not tell a node its agent settings; it is told when it next registers",
+				"node", node, "err", err)
+			continue
+		}
+		time.Sleep(agentSettingsGap)
+	}
+}

internal/control/agentsvc/agentsettings_test.go

@@ -0,0 +1,28 @@
+package agentsvc
+
+import (
+	"log/slog"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+)
+
+func TestEachNodeIsToldItsOwnAgentSettings(t *testing.T) {
+	s := New(inventory.NewStore(), nil, false, slog.New(slog.DiscardHandler))
+	if s.agentSettingsMsg("n1") != nil {
+		t.Error("told settings with nothing to read them from")
+	}
+	s.SetAgentSettings(func(node string) string {
+		if node == "cuda01" {
+			return "labels:\n  gpu: rtx6000\n"
+		}
+		return ""
+	})
+	if got := s.agentSettingsMsg("cuda01").GetAgentSettings().GetYaml(); got != "labels:\n  gpu: rtx6000\n" {
+		t.Errorf("cuda01 told %q", got)
+	}
+	// None is said too, so an agent whose settings were removed drops them.
+	if m := s.agentSettingsMsg("node01"); m == nil || m.GetAgentSettings().GetYaml() != "" {
+		t.Errorf("node01 told %v", m)
+	}
+}

internal/control/agentsvc/service.go

@@ -79,6 +79,11 @@ type Service struct {
 	controlPlanes func(context.Context) ([]string, error)
 	// firewall is what nodes are told of llmgw's firewall; see SetFirewall.
 	firewall func() *llmgwv1.Firewall
+	// agentSettings is what each node's agent is told to run on; see
+	// SetAgentSettings.
+	agentSettings func(node string) string
+	// train is told how training jobs go; see SetTrain.
+	train TrainHandler
 
 	// logWaits are the log requests awaiting a reply, by request id.
 	logMu    sync.Mutex
@@ -296,6 +301,13 @@ func (s *Service) Connect(
 			return err
 		}
 	}
+	// What the cluster's settings say this agent runs on. An agent already
+	// on them ignores it; one that is not starts again on them.
+	if msg := s.agentSettingsMsg(nodeID); msg != nil {
+		if err := stream.Send(msg); err != nil {
+			return err
+		}
+	}
 
 	// Recover replica state from what this node is already running, before any
 	// reconciliation pass can conclude the cluster is empty and place
@@ -358,6 +370,9 @@ func (s *Service) readLoop(ctx context.Context,
 		msg, err := stream.Receive()
 		if err != nil {
 			c.close()
+			if h := s.trainHandler(); h != nil {
+				h.OnNodeGone(nodeID)
+			}
 			if ctx.Err() != nil || errors.Is(err, io.EOF) {
 				s.log.Info("agent disconnected", "node", nodeID)
 				return
@@ -495,6 +510,16 @@ func (s *Service) handle(nodeID string, msg *llmgwv1.AgentMessage) {
 			ch <- m.Upgrade
 		}
 
+	case *llmgwv1.AgentMessage_TrainStatus:
+		if h := s.trainHandler(); h != nil {
+			h.OnTrainStatus(nodeID, m.TrainStatus)
+		}
+
+	case *llmgwv1.AgentMessage_TrainArtifact:
+		if h := s.trainHandler(); h != nil {
+			h.OnTrainArtifact(nodeID, m.TrainArtifact)
+		}
+
 	case *llmgwv1.AgentMessage_Logs:
 		s.logMu.Lock()
 		ch, ok := s.logWaits[m.Logs.GetRequestId()]

internal/control/agentsvc/train.go

@@ -0,0 +1,35 @@
+package agentsvc
+
+import (
+	"slices"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+)
+
+// TrainHandler is told how training jobs on the nodes go (BACKLOG F4).
+type TrainHandler interface {
+	OnTrainStatus(node string, st *llmgwv1.TrainStatus)
+	OnTrainArtifact(node string, a *llmgwv1.TrainArtifact)
+	// OnNodeGone is a node's stream ending: a job running there ended with
+	// it.
+	OnNodeGone(node string)
+}
+
+// SetTrain gives the service what training jobs' messages go to.
+func (s *Service) SetTrain(h TrainHandler) {
+	s.mu.Lock()
+	s.train = h
+	s.mu.Unlock()
+}
+
+func (s *Service) trainHandler() TrainHandler {
+	s.mu.RLock()
+	defer s.mu.RUnlock()
+	return s.train
+}
+
+// Send gives a connected node's agent a command.
+func (s *Service) Send(node string, msg *llmgwv1.ControlMessage) error { return s.send(node, msg) }
+
+// Connected is whether a node's agent holds a stream to this control plane.
+func (s *Service) Connected(node string) bool { return slices.Contains(s.ConnectedNodes(), node) }

internal/control/catalog/catalog.go

@@ -1127,6 +1127,27 @@ type Adapter struct {
 	Source string `json:"source" koanf:"source"`
 	// Scale is how strongly it is applied; zero takes 1.
 	Scale float64 `json:"scale,omitempty" koanf:"scale"`
+	// Owner is the part of the tenancy tree the adapter belongs to —
+	// org:ID, team:ID, project:ID, user:ID or key:ID — and only callers in
+	// it see it: to anyone else it is a model that does not exist. An
+	// adapter trained on an organisation's documents carries some of what
+	// they say in its weights. Empty is every caller the model allows.
+	Owner string `json:"owner,omitempty" koanf:"owner"`
+}
+
+// ValidAdapterOwner checks an adapter's owner is scope:ID.
+func ValidAdapterOwner(owner string) error {
+	if owner == "" {
+		return nil
+	}
+	scope, id, ok := strings.Cut(owner, ":")
+	switch {
+	case !ok || id == "":
+		return fmt.Errorf("an adapter's owner is scope:ID, %q is not", owner)
+	case scope != "org" && scope != "team" && scope != "project" && scope != "user" && scope != "key":
+		return fmt.Errorf("an adapter's owner is an org, team, project, user or key, not %q", scope)
+	}
+	return nil
 }
 
 // EffectiveScale is the scale an adapter is applied at.
@@ -1175,6 +1196,9 @@ func validateAdapters(m Model) error {
 	}
 	seen := map[string]bool{}
 	for _, a := range m.Adapters {
+		if err := ValidAdapterOwner(a.Owner); err != nil {
+			return fmt.Errorf("model %q adapter %q: %w", m.Name, a.Name, err)
+		}
 		switch {
 		case a.Name == "" || a.Source == "":
 			return fmt.Errorf("model %q: an adapter needs a name and a source", m.Name)

internal/control/train/service.go

@@ -0,0 +1,196 @@
+// Package train runs LoRA training jobs from the control plane (BACKLOG F4):
+// queued in the store, sent to the node each names when its agent is there
+// and has nothing else training, followed as the agent reports, and what it
+// makes kept in the store.
+package train
+
+import (
+	"context"
+	"log/slog"
+	"sync"
+	"time"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// Store is where jobs are kept.
+type Store interface {
+	QueuedTrainJobs(ctx context.Context) ([]store.TrainJob, error)
+	ActiveTrainJobs(ctx context.Context) ([]store.TrainJob, error)
+	TrainJob(ctx context.Context, id string) (store.TrainJob, error)
+	UpdateTrainJob(ctx context.Context, j store.TrainJob) error
+	PutTrainArtifact(ctx context.Context, job, kind string, data []byte) error
+}
+
+// Agents is how jobs reach the nodes.
+type Agents interface {
+	Send(node string, msg *llmgwv1.ControlMessage) error
+	Connected(node string) bool
+}
+
+// Service sends queued jobs and records how they go.
+type Service struct {
+	store  Store
+	agents Agents
+	log    *slog.Logger
+
+	mu sync.Mutex
+	// pieces are artifacts arriving, by job and kind.
+	pieces map[[2]string][]byte
+}
+
+// New is a service.
+func New(s Store, a Agents, log *slog.Logger) *Service {
+	if log == nil {
+		log = slog.New(slog.DiscardHandler)
+	}
+	return &Service{store: s, agents: a, log: log, pieces: map[[2]string][]byte{}}
+}
+
+// Every is how often queued jobs are looked at.
+var Every = 5 * time.Second
+
+// Run sends queued jobs until ctx ends.
+func (s *Service) Run(ctx context.Context) {
+	t := time.NewTicker(Every)
+	defer t.Stop()
+	for {
+		s.Dispatch(ctx)
+		select {
+		case <-ctx.Done():
+			return
+		case <-t.C:
+		}
+	}
+}
+
+// Dispatch sends each queued job whose node is here and not already
+// training, one per node.
+func (s *Service) Dispatch(ctx context.Context) {
+	queued, err := s.store.QueuedTrainJobs(ctx)
+	if err != nil || len(queued) == 0 {
+		return
+	}
+	active, err := s.store.ActiveTrainJobs(ctx)
+	if err != nil {
+		return
+	}
+	busy := map[string]bool{}
+	for _, j := range active {
+		busy[j.Node] = true
+	}
+	for _, q := range queued {
+		if busy[q.Node] || !s.agents.Connected(q.Node) {
+			continue
+		}
+		j, err := s.store.TrainJob(ctx, q.ID)
+		if err != nil {
+			continue
+		}
+		err = s.agents.Send(j.Node, &llmgwv1.ControlMessage{Msg: &llmgwv1.ControlMessage_TrainStart{TrainStart: &llmgwv1.TrainStart{
+			JobId: j.ID, BaseRepo: j.BaseRepo, BaseRevision: j.BaseRevision, Data: j.Data, Params: j.Params,
+			GpuMemoryMib: int32(j.GPUMemoryMiB), Gguf: j.GGUF}}}) //nolint:gosec // MiB
+		if err != nil {
+			s.log.Warn("could not send a training job to its node", "job", j.ID, "node", j.Node, "err", err)
+			continue
+		}
+		j.State, j.StartedAt, j.Message = store.TrainStarting, time.Now(), "sent to "+j.Node
+		_ = s.store.UpdateTrainJob(ctx, j)
+		busy[j.Node] = true
+		s.log.Info("training job sent", "job", j.ID, "node", j.Node, "adapter", j.Name)
+	}
+}
+
+// OnTrainStatus records how a job goes, as its node says.
+func (s *Service) OnTrainStatus(node string, st *llmgwv1.TrainStatus) {
+	ctx := context.Background()
+	j, err := s.store.TrainJob(ctx, st.GetJobId())
+	if err != nil || j.Node != node || store.TrainOver(j.State) {
+		return
+	}
+	j.State = st.GetState()
+	if m := st.GetMessage(); m != "" || j.State == store.TrainDone {
+		j.Message = m
+	}
+	if st.GetSteps() > 0 {
+		j.Step, j.Steps = int(st.GetStep()), int(st.GetSteps())
+	}
+	if st.GetLoss() > 0 {
+		j.Loss = st.GetLoss()
+	}
+	if p := int(st.GetPeakMib()); p > j.PeakMiB {
+		j.PeakMiB = p
+	}
+	if store.TrainOver(j.State) {
+		j.FinishedAt = time.Now()
+		s.mu.Lock()
+		for k := range s.pieces {
+			if k[0] == j.ID {
+				delete(s.pieces, k)
+			}
+		}
+		s.mu.Unlock()
+		s.log.Info("training job ended", "job", j.ID, "state", j.State, "message", j.Message)
+	}
+	if err := s.store.UpdateTrainJob(ctx, j); err != nil {
+		s.log.Warn("could not record a training job's progress", "job", j.ID, "err", err)
+	}
+}
+
+// OnTrainArtifact gathers what a job made, and keeps it when it is whole.
+func (s *Service) OnTrainArtifact(node string, a *llmgwv1.TrainArtifact) {
+	key := [2]string{a.GetJobId(), a.GetKind()}
+	s.mu.Lock()
+	if a.GetSeq() == 0 {
+		s.pieces[key] = nil
+	}
+	s.pieces[key] = append(s.pieces[key], a.GetData()...)
+	whole := s.pieces[key]
+	if a.GetLast() {
+		delete(s.pieces, key)
+	}
+	s.mu.Unlock()
+	if !a.GetLast() {
+		return
+	}
+	if err := s.store.PutTrainArtifact(context.Background(), key[0], key[1], whole); err != nil {
+		s.log.Warn("could not keep what a training job made", "job", key[0], "kind", key[1], "err", err)
+	}
+}
+
+// OnNodeGone fails the jobs a node was running: its agent's stream ended,
+// and the job with it.
+func (s *Service) OnNodeGone(node string) {
+	ctx := context.Background()
+	active, err := s.store.ActiveTrainJobs(ctx)
+	if err != nil {
+		return
+	}
+	for _, j := range active {
+		if j.Node != node {
+			continue
+		}
+		j.State, j.FinishedAt = store.TrainFailed, time.Now()
+		j.Message = "the node's agent went away while it trained; start the job again"
+		_ = s.store.UpdateTrainJob(ctx, j)
+	}
+}
+
+// Cancel stops a job: one still queued is not sent, one running is stopped
+// on its node.
+func (s *Service) Cancel(ctx context.Context, id string) error {
+	j, err := s.store.TrainJob(ctx, id)
+	if err != nil {
+		return err
+	}
+	if store.TrainOver(j.State) {
+		return nil
+	}
+	if j.State != store.TrainQueued && s.agents.Connected(j.Node) {
+		_ = s.agents.Send(j.Node, &llmgwv1.ControlMessage{Msg: &llmgwv1.ControlMessage_TrainCancel{
+			TrainCancel: &llmgwv1.TrainCancel{JobId: id}}})
+	}
+	j.State, j.FinishedAt, j.Message = store.TrainCancelled, time.Now(), "cancelled"
+	return s.store.UpdateTrainJob(ctx, j)
+}

internal/control/train/service_test.go

@@ -0,0 +1,124 @@
+package train
+
+import (
+	"context"
+	"path/filepath"
+	"sync"
+	"testing"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+type fakeAgents struct {
+	mu        sync.Mutex
+	connected map[string]bool
+	sent      []*llmgwv1.ControlMessage
+}
+
+func (f *fakeAgents) Send(_ string, m *llmgwv1.ControlMessage) error {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	f.sent = append(f.sent, m)
+	return nil
+}
+func (f *fakeAgents) Connected(n string) bool {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	return f.connected[n]
+}
+
+func setup(t *testing.T) (*Service, *store.DB, *fakeAgents) {
+	t.Helper()
+	s, err := store.OpenSQLite(context.Background(), filepath.Join(t.TempDir(), "s.db"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	t.Cleanup(func() { _ = s.Close() })
+	a := &fakeAgents{connected: map[string]bool{}}
+	return New(s, a, nil), s, a
+}
+
+func job(t *testing.T, s *store.DB, id, node string) {
+	t.Helper()
+	if err := s.CreateTrainJob(context.Background(), store.TrainJob{ID: id, Name: "a-" + id, Model: "m", BaseRepo: "o/n",
+		Node: node, GPUMemoryMiB: 2000, Data: []byte(`{"q":"x","a":"y"}`), Pairs: 1, GGUF: true}); err != nil {
+		t.Fatal(err)
+	}
+}
+
+func TestAJobGoesToItsNodeWhenItIsHereAndFree(t *testing.T) {
+	ctx := context.Background()
+	svc, s, a := setup(t)
+	job(t, s, "j1", "cuda01")
+	job(t, s, "j2", "cuda01")
+	svc.Dispatch(ctx)
+	if len(a.sent) != 0 {
+		t.Fatal("sent to a node that is not here")
+	}
+	a.connected["cuda01"] = true
+	svc.Dispatch(ctx)
+	svc.Dispatch(ctx)
+	if len(a.sent) != 1 {
+		t.Fatalf("sent %d jobs to one node", len(a.sent))
+	}
+	start := a.sent[0].GetTrainStart()
+	if start.GetJobId() != "j1" || start.GetGpuMemoryMib() != 2000 || !start.GetGguf() || string(start.GetData()) == "" {
+		t.Errorf("sent %+v", start)
+	}
+	if j, _ := s.TrainJob(ctx, "j1"); j.State != store.TrainStarting || j.StartedAt.IsZero() {
+		t.Errorf("j1 %+v", j)
+	}
+
+	// Progress, then what it made in pieces, then done: the next is sent.
+	svc.OnTrainStatus("cuda01", &llmgwv1.TrainStatus{JobId: "j1", State: "running", Step: 5, Steps: 10, Loss: 0.8, PeakMib: 1900})
+	svc.OnTrainArtifact("cuda01", &llmgwv1.TrainArtifact{JobId: "j1", Kind: "gguf", Seq: 0, Data: []byte("GG")})
+	svc.OnTrainArtifact("cuda01", &llmgwv1.TrainArtifact{JobId: "j1", Kind: "gguf", Seq: 1, Data: []byte("UF"), Last: true})
+	svc.OnTrainStatus("cuda01", &llmgwv1.TrainStatus{JobId: "j1", State: "done"})
+	j, _ := s.TrainJob(ctx, "j1")
+	if j.State != store.TrainDone || j.Step != 5 || j.PeakMiB != 1900 || j.Loss != 0.8 || j.FinishedAt.IsZero() {
+		t.Errorf("j1 after %+v", j)
+	}
+	if b, _ := s.TrainArtifact(ctx, "j1", "gguf"); string(b) != "GGUF" {
+		t.Errorf("artifact %q", b)
+	}
+	// A late message does not bring an ended job back.
+	svc.OnTrainStatus("cuda01", &llmgwv1.TrainStatus{JobId: "j1", State: "running"})
+	if j, _ := s.TrainJob(ctx, "j1"); j.State != store.TrainDone {
+		t.Errorf("revived: %s", j.State)
+	}
+	// Nor does another node speak for it.
+	svc.Dispatch(ctx)
+	if len(a.sent) != 2 || a.sent[1].GetTrainStart().GetJobId() != "j2" {
+		t.Fatalf("j2 not sent: %d", len(a.sent))
+	}
+	svc.OnTrainStatus("node01", &llmgwv1.TrainStatus{JobId: "j2", State: "done"})
+	if j, _ := s.TrainJob(ctx, "j2"); j.State == store.TrainDone {
+		t.Error("another node ended the job")
+	}
+	// The node goes: its job fails, saying why.
+	svc.OnNodeGone("cuda01")
+	if j, _ := s.TrainJob(ctx, "j2"); j.State != store.TrainFailed || j.Message == "" {
+		t.Errorf("after the node went: %+v", j)
+	}
+}
+
+func TestCancellingAJob(t *testing.T) {
+	ctx := context.Background()
+	svc, s, a := setup(t)
+	job(t, s, "j1", "cuda01")
+	if err := svc.Cancel(ctx, "j1"); err != nil {
+		t.Fatal(err)
+	}
+	a.connected["cuda01"] = true
+	svc.Dispatch(ctx)
+	if len(a.sent) != 0 {
+		t.Error("a cancelled job was sent")
+	}
+	job(t, s, "j2", "cuda01")
+	svc.Dispatch(ctx)
+	_ = svc.Cancel(ctx, "j2")
+	if len(a.sent) != 2 || a.sent[1].GetTrainCancel().GetJobId() != "j2" {
+		t.Errorf("the node was not told: %+v", a.sent)
+	}
+}

internal/docgen/reference_gen.go

@@ -8,6 +8,7 @@ var Reference = []Block{
 		{Key: "name", Type: "string", Doc: "Name is what clients ask for, like a model name.", Nested: ""},
 		{Key: "source", Type: "string", Doc: "Source is the adapter: a GGUF file for llama.cpp; for vLLM a PEFT\nadapter directory (a mirror version ending in \"/\", a path on the node,\nor a Hugging Face repository id). Fetched by each node as the model's\nweights are.", Nested: ""},
 		{Key: "scale", Type: "float64", Doc: "Scale is how strongly it is applied; zero takes 1.", Nested: ""},
+		{Key: "owner", Type: "string", Doc: "Owner is the part of the tenancy tree the adapter belongs to —\norg:ID, team:ID, project:ID, user:ID or key:ID — and only callers in\nit see it: to anyone else it is a model that does not exist. An\nadapter trained on an organisation's documents carries some of what\nthey say in its weights. Empty is every caller the model allows.", Nested: ""},
 	}},
 	{Package: "config", Name: "Admission", Title: "Admission", Doc: "", Fields: []Field{
 		{Key: "enabled", Type: "bool", Doc: "Enabled turns the layer on. Off, every request goes straight through,\nwhich is the v0.2.0 behaviour.", Nested: ""},
@@ -19,6 +20,10 @@ var Reference = []Block{
 		{Key: "queue_capacity_total", Type: "int", Doc: "QueueCapacityTotal bounds everything queued across bands. Reached, a\nrequest is not refused while a lower band has any queued: the newest of\nthe lowest band is turned away instead, with a 503 and Retry-After.\nZero takes half of all the bands' capacities together.", Nested: ""},
 		{Key: "interval", Type: "time.Duration", Doc: "", Nested: ""},
 	}},
+	{Package: "config", Name: "AgentSettings", Title: "AgentSettings", Doc: "", Fields: []Field{
+		{Key: "default", Type: "map[string]any", Doc: "Default is agent.yaml's keys for every node.", Nested: ""},
+		{Key: "nodes", Type: "map[string]map[string]any", Doc: "Nodes are agent.yaml's keys for one node each, by node id, over\nDefault.", Nested: ""},
+	}},
 	{Package: "config", Name: "Auth", Title: "Auth", Doc: "", Fields: []Field{
 		{Key: "enabled", Type: "bool", Doc: "", Nested: ""},
 		{Key: "store", Type: "string", Doc: "Store is the SQLite file holding keys, tenancy and usage.", Nested: ""},
@@ -372,6 +377,7 @@ var Reference = []Block{
 		{Key: "mail", Type: "mail.Config", Doc: "Mail is where the gateway's few emails go out from: a link to set a\nforgotten password. Unset, none is sent and people ask an operator.", Nested: "mail.Config"},
 		{Key: "serving", Type: "ServingSettings", Doc: "Serving stops and starts the whole cluster's service (llmgw stop, llmgw\nstart).", Nested: "config.ServingSettings"},
 		{Key: "firewall", Type: "FirewallSettings", Doc: "Firewall turns llmgw's firewall on every machine on or off.", Nested: "config.FirewallSettings"},
+		{Key: "agents", Type: "AgentSettings", Doc: "Agents are the nodes' agent settings: agent.yaml's keys but those an\nagent needs to start, for every node (default) and for one\n(nodes.<id>). Sent to the agents as they change; an agent whose\nsettings changed starts again on them, its engines serving\nthroughout.", Nested: "config.AgentSettings"},
 		{Key: "firewall_networks", Type: "[]string", Doc: "FirewallNetworks are networks, besides those each machine reaches the\ncluster on, that llmgw's internal ports answer: a monitoring host on\nanother subnet, nodes behind a router. Written into every machine's\nfirewall by `server init`, `node add` and `server add`.", Nested: ""},
 		{Key: "backup_recipients", Type: "[]string", Doc: "BackupRecipients encrypt every archive `llmgw server backup` writes, to\nwhoever holds one of their keys: age recipients (age1…) or SSH public\nkeys (ssh-ed25519 …, ssh-rsa …), as `age` takes them. An archive holds\nthe certificate authority's key and every secret in the store, and is\ncopied off the machine; encrypted, the copy is safe wherever it lands.\nEmpty writes archives unencrypted, readable by root only.", Nested: ""},
 		{Key: "backup_copy_to", Type: "[]string", Doc: "BackupCopyTo is where each archive is copied once written, so it\nsurvives this machine: a directory (a mounted share, another disk) or\nssh://user@host[:port]/path, reached with BackupCopyKey and checked\nagainst /etc/llmgw/backup_known_hosts, which learns a host the first\ntime. The same number of archives is kept there as here. Refused\nwithout backup_recipients: an archive leaves the machine encrypted.", Nested: ""},

internal/docgen/render.go

@@ -32,6 +32,7 @@ var Sources = []string{
 	"internal/config/oidc.go",
 	"internal/config/firewall.go",
 	"internal/config/serving.go",
+	"internal/config/agents.go",
 	"internal/mail/mail.go",
 	"internal/notify/notify.go",
 	"internal/secrets/secrets.go",

internal/firewall/follow.go

@@ -0,0 +1,154 @@
+package firewall
+
+import (
+	"context"
+	"fmt"
+	"strings"
+)
+
+// A store member that is nothing else runs no llmgw process: Patroni, etcd
+// and PostgreSQL only. What the cluster's settings say of its firewall it
+// reads from its own copy of the store, where the leading control plane
+// keeps it (the cluster_state table, key firewall), once a minute, by a
+// root timer running FollowScript. It turns llmgw's table off and on, and
+// lets in the networks the settings add; what `server init` let in stays.
+
+// FollowScript is the script, as python3 runs it. LLMGW_FW_STATE and
+// LLMGW_FW_RULES replace the store and the rules file, for its tests.
+const FollowScript = `#!/usr/bin/python3
+# Written by llmgw: llmgw's firewall on this store member, as the cluster's
+# settings say. Run by llmgw-firewall-follow.timer.
+import ipaddress, json, os, subprocess, sys, tempfile
+
+RULES = os.environ.get("LLMGW_FW_RULES", "` + RulesFile + `")
+
+def run(*a):
+    return subprocess.run(a, capture_output=True, text=True)
+
+raw = os.environ.get("LLMGW_FW_STATE")
+if raw is None:
+    r = run("runuser", "-u", "postgres", "--", "psql", "-d", "llmgw", "-tAqc",
+            "SELECT value FROM cluster_state WHERE key = 'firewall'")
+    # No table yet, or nothing told: a control plane older than this, or
+    # none leading. Left as it is.
+    if r.returncode != 0:
+        sys.exit(0)
+    raw = r.stdout
+if not raw.strip():
+    sys.exit(0)
+st = json.loads(raw)
+present = run("nft", "list", "table", "inet", "` + Table + `").returncode == 0
+
+if not st.get("enabled", True):
+    if present:
+        run("systemctl", "disable", "--now", "` + Unit + `")
+        run("nft", "delete", "table", "inet", "` + Table + `")
+        print("llmgw's firewall is off here")
+    sys.exit(0)
+
+if not os.path.exists(RULES):
+    sys.exit(0)
+lines = open(RULES).read().split("\n")
+want = []
+for n in st.get("networks") or []:
+    try:
+        want.append(ipaddress.ip_network(n, strict=False))
+    except ValueError:
+        pass
+
+def saddr(l):
+    s = l.strip()
+    return s.startswith("ip saddr {") or s.startswith("ip6 saddr {")
+
+changed = False
+for fam, kw in ((4, "ip saddr {"), (6, "ip6 saddr {")):
+    add = [str(n) for n in want if n.version == fam]
+    if not add:
+        continue
+    same = [i for i, l in enumerate(lines) if l.strip().startswith(kw)]
+    if same:
+        i = same[0]
+        l = lines[i]
+        a, b = l.index("{") + 1, l.index("}")
+        have = [x.strip() for x in l[a:b].split(",") if x.strip()]
+        new = [x for x in add if x not in have]
+        if new:
+            lines[i] = l[:a] + " " + ", ".join(have + new) + " " + l[b:]
+            changed = True
+        continue
+    other = [i for i, l in enumerate(lines) if saddr(l)]
+    if not other:
+        continue
+    l = lines[other[0]]
+    indent = l[:len(l) - len(l.lstrip())]
+    lines.insert(other[0] + 1, indent + kw + " " + ", ".join(add) + " }" + l[l.index("}") + 1:])
+    changed = True
+
+if changed:
+    fd, tmp = tempfile.mkstemp(dir=os.path.dirname(RULES))
+    with os.fdopen(fd, "w") as f:
+        f.write("\n".join(lines))
+    if run("nft", "-c", "-f", tmp).returncode != 0:
+        os.unlink(tmp)
+        print("the settings' networks make a ruleset nft refuses; left as it was", file=sys.stderr)
+        sys.exit(1)
+    os.chmod(tmp, 0o644)
+    os.replace(tmp, RULES)
+if changed or not present:
+    run("systemctl", "enable", "` + Unit + `")
+    if run("nft", "-f", RULES).returncode != 0:
+        sys.exit(1)
+    print("llmgw's firewall is on here")
+`
+
+// FollowerPath is where FollowScript is installed.
+const FollowerPath = "/usr/local/lib/llmgw/firewall-follow"
+
+// FollowerTimer is the timer that runs it.
+const FollowerTimer = "llmgw-firewall-follow.timer"
+
+const followService = `[Unit]
+Description=llmgw firewall: follow the cluster's settings on this store member
+After=postgresql.service patroni.service
+
+[Service]
+Type=oneshot
+ExecStart=/usr/bin/python3 ` + FollowerPath + `
+`
+
+const followTimer = `[Unit]
+Description=llmgw firewall: follow the cluster's settings on this store member
+
+[Timer]
+OnBootSec=1min
+OnUnitActiveSec=1min
+
+[Install]
+WantedBy=timers.target
+`
+
+// InstallFollower puts FollowScript and its timer on a store member.
+func InstallFollower(ctx context.Context, r Runner) error {
+	body := FollowScript + "\x00" + followService + "\x00" + followTimer
+	out, err := r.Run(ctx, `set -e
+t=$(mktemp -d); trap 'rm -rf "$t"' EXIT
+cat > "$t/all"
+python3 - "$t" <<'PY'
+import sys
+d = sys.argv[1]
+s, svc, tim = open(d + "/all", "rb").read().split(b"\x00")
+open(d + "/script", "wb").write(s)
+open(d + "/service", "wb").write(svc)
+open(d + "/timer", "wb").write(tim)
+PY
+sudo install -d -m 755 /usr/local/lib/llmgw
+sudo install -m 755 "$t/script" `+FollowerPath+`
+sudo install -m 644 "$t/service" /etc/systemd/system/llmgw-firewall-follow.service
+sudo install -m 644 "$t/timer" /etc/systemd/system/`+FollowerTimer+`
+sudo systemctl daemon-reload
+sudo systemctl enable --now `+FollowerTimer+` >/dev/null 2>&1`, []byte(body))
+	if err != nil {
+		return fmt.Errorf("installing %s: %v: %s", FollowerTimer, err, strings.TrimSpace(string(out)))
+	}
+	return nil
+}

internal/firewall/follow_test.go

@@ -0,0 +1,101 @@
+package firewall
+
+import (
+	"net/netip"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"strings"
+	"testing"
+)
+
+// follow runs FollowScript against a rules file and a state, with nft and
+// systemctl replaced by scripts that log what they were asked and answer as
+// a machine whose table is loaded or not.
+func follow(t *testing.T, rules, state string, loaded bool) (string, string, string) {
+	t.Helper()
+	if _, err := exec.LookPath("python3"); err != nil {
+		t.Skip("no python3")
+	}
+	dir := t.TempDir()
+	bin := filepath.Join(dir, "bin")
+	_ = os.Mkdir(bin, 0o755)
+	log := filepath.Join(dir, "log")
+	list := "exit 1"
+	if loaded {
+		list = "exit 0"
+	}
+	for name, body := range map[string]string{
+		"nft":       `echo "nft $*" >> ` + log + "\n" + `[ "$1" = list ] && ` + list + "\nexit 0\n",
+		"systemctl": `echo "systemctl $*" >> ` + log + "\nexit 0\n",
+	} {
+		if err := os.WriteFile(filepath.Join(bin, name), []byte("#!/bin/sh\n"+body), 0o755); err != nil {
+			t.Fatal(err)
+		}
+	}
+	rf := filepath.Join(dir, "firewall.nft")
+	if rules != "" {
+		_ = os.WriteFile(rf, []byte(rules), 0o644)
+	}
+	script := filepath.Join(dir, "follow")
+	_ = os.WriteFile(script, []byte(FollowScript), 0o755)
+	cmd := exec.Command("python3", script)
+	cmd.Env = append(os.Environ(), "PATH="+bin+":"+os.Getenv("PATH"), "LLMGW_FW_STATE="+state, "LLMGW_FW_RULES="+rf)
+	out, err := cmd.CombinedOutput()
+	if err != nil {
+		t.Fatalf("%v: %s", err, out)
+	}
+	got, _ := os.ReadFile(rf)
+	calls, _ := os.ReadFile(log)
+	return string(out), string(got), string(calls)
+}
+
+var memberRules = Script(Spec{Ports: []Port{Postgres, EtcdClient}, Allow: mustPrefixes("10.0.0.0/24")})
+
+func TestAStoreMemberFollowsTheSettings(t *testing.T) {
+	// Off: the table goes, the rules stay for later.
+	out, rules, calls := follow(t, memberRules, `{"enabled":false}`, true)
+	if !strings.Contains(out, "off") || !strings.Contains(calls, "nft delete table inet llmgw") ||
+		!strings.Contains(calls, "systemctl disable --now "+Unit) || rules != memberRules {
+		t.Errorf("off: %q %q", out, calls)
+	}
+	// On again: loaded from the rules kept.
+	out, _, calls = follow(t, memberRules, `{"enabled":true}`, false)
+	if !strings.Contains(out, "on") || !strings.Contains(calls, "nft -f") {
+		t.Errorf("on: %q %q", out, calls)
+	}
+	// On and loaded, nothing new: nothing done.
+	out, _, calls = follow(t, memberRules, `{"enabled":true,"networks":["10.0.0.0/24"]}`, true)
+	if out != "" || strings.Contains(calls, "nft -f") {
+		t.Errorf("unchanged: %q %q", out, calls)
+	}
+	// Networks the settings add, of both families, are let in beside what
+	// init let in; the result reads back as the same ports.
+	_, rules, calls = follow(t, memberRules, `{"enabled":true,"networks":["192.168.5.0/24","fd00::/64","nonsense"]}`, true)
+	if !strings.Contains(calls, "nft -c -f") || !strings.Contains(calls, "nft -f") {
+		t.Errorf("not checked and loaded: %q", calls)
+	}
+	back := parse(rules)
+	var nets []string
+	for _, p := range back.Allow {
+		nets = append(nets, p.String())
+	}
+	if strings.Join(nets, ",") != "10.0.0.0/24,192.168.5.0/24,fd00::/64" || len(back.Ports) != 2 {
+		t.Errorf("rules now:\n%s", rules)
+	}
+	// Nothing told yet, or no rules: left alone.
+	if out, _, calls := follow(t, memberRules, "", true); out != "" || calls != "" {
+		t.Errorf("nothing told: %q %q", out, calls)
+	}
+	if out, _, calls := follow(t, "", `{"enabled":true}`, false); out != "" || strings.Contains(calls, "nft -f") {
+		t.Errorf("no rules: %q %q", out, calls)
+	}
+}
+
+func mustPrefixes(ss ...string) []netip.Prefix {
+	var out []netip.Prefix
+	for _, s := range ss {
+		out = append(out, netip.MustParsePrefix(s))
+	}
+	return out
+}

internal/mirror/api.go

@@ -4,6 +4,7 @@ import (
 	"encoding/json"
 	"errors"
 	"fmt"
+	"io"
 	"net/http"
 	"os"
 	"strings"
@@ -37,7 +38,9 @@ func (a *API) Handler() http.Handler {
 	mux.HandleFunc("POST /v1/models", a.fetch)
 	mux.HandleFunc("GET /v1/models/{model}/{version}", a.status)
 	mux.HandleFunc("DELETE /v1/models/{model}/{version}", a.remove)
+	mux.HandleFunc("PUT /v1/models/{model}/{version}/{file}", a.upload)
 	mux.HandleFunc("GET /v1/health", a.health)
+	mux.HandleFunc("PUT /v1/firewall", a.putFirewall)
 	return mux
 }
 
@@ -199,3 +202,59 @@ func LastPathElement(u string) string {
 	}
 	return u
 }
+
+// maxUpload bounds a file sent to the mirror rather than fetched by it.
+const maxUpload = 8 << 30
+
+// upload keeps a file the caller sends — an adapter a node trained, which
+// is on no registry to fetch from — with the same discipline as an import:
+// its checksum and source beside it, and an existing version never
+// replaced. X-Llmgw-Source says where it came from.
+func (a *API) upload(w http.ResponseWriter, r *http.Request) {
+	e := Entry{Model: r.PathValue("model"), Version: r.PathValue("version"), File: r.PathValue("file")}
+	for _, n := range []string{e.Model, e.Version, e.File} {
+		if err := ValidName(n); err != nil {
+			fail(w, http.StatusBadRequest, err)
+			return
+		}
+	}
+	if _, err := os.Stat(e.Path(a.m.root)); err == nil {
+		fail(w, http.StatusConflict, fmt.Errorf("%s/%s/%s: %w", e.Model, e.Version, e.File, ErrExists))
+		return
+	}
+	tmp, err := os.CreateTemp(a.m.root, ".upload-*")
+	if err != nil {
+		fail(w, http.StatusInternalServerError, err)
+		return
+	}
+	defer os.Remove(tmp.Name())
+	n, err := io.Copy(tmp, http.MaxBytesReader(w, r.Body, maxUpload))
+	if cerr := tmp.Close(); err == nil {
+		err = cerr
+	}
+	if err != nil {
+		fail(w, http.StatusBadRequest, fmt.Errorf("receiving the file: %w", err))
+		return
+	}
+	if n == 0 {
+		fail(w, http.StatusBadRequest, errors.New("the file is empty"))
+		return
+	}
+	source := strings.TrimSpace(r.Header.Get("X-Llmgw-Source"))
+	if source == "" {
+		source = "uploaded"
+	}
+	added, err := Import(r.Context(), a.m.root, tmp.Name(), e, "", source, nil)
+	switch {
+	case errors.Is(err, ErrExists):
+		fail(w, http.StatusConflict, err)
+		return
+	case err != nil:
+		fail(w, http.StatusBadRequest, err)
+		return
+	}
+	writeJSON(w, http.StatusCreated, FetchResponse{
+		Status: Status{Model: e.Model, Version: e.Version, File: e.File, State: Present, BytesDone: added.Entry.Bytes, BytesTotal: added.Entry.Bytes},
+		URL:    e.URL(a.base),
+	})
+}

internal/mirror/client.go

@@ -100,6 +100,36 @@ func (c *Client) Fetch(ctx context.Context, req FetchRequest) (FetchResponse, er
 	return out, err
 }
 
+// Upload sends a file for the mirror to keep at model/version/file, with
+// where it came from; it answers the URL nodes fetch it from.
+func (c *Client) Upload(ctx context.Context, model, version, file string, body []byte, source string) (FetchResponse, error) {
+	var out FetchResponse
+	req, err := http.NewRequestWithContext(ctx, http.MethodPut,
+		c.base+"/v1/models/"+url.PathEscape(model)+"/"+url.PathEscape(version)+"/"+url.PathEscape(file), bytes.NewReader(body))
+	if err != nil {
+		return out, err
+	}
+	req.Header.Set("Content-Type", "application/octet-stream")
+	req.Header.Set("X-Llmgw-Source", source)
+	resp, err := c.hc.Do(req)
+	if err != nil {
+		return out, fmt.Errorf("reaching the mirror at %s: %w", c.base, err)
+	}
+	defer resp.Body.Close()
+	raw, err := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
+	if err != nil {
+		return out, err
+	}
+	if resp.StatusCode >= 300 {
+		var e struct {
+			Error string `json:"error"`
+		}
+		_ = json.Unmarshal(raw, &e)
+		return out, &APIError{Status: resp.StatusCode, Msg: e.Error}
+	}
+	return out, json.Unmarshal(raw, &out)
+}
+
 // Status is where one model version stands.
 func (c *Client) Status(ctx context.Context, model, version string) (FetchResponse, error) {
 	var out FetchResponse

internal/mirror/firewall.go

@@ -0,0 +1,65 @@
+package mirror
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"net/http"
+	"os"
+	"path/filepath"
+)
+
+// The mirror's machine keeps llmgw's firewall as the cluster's settings
+// say, as every other machine does. The mirror runs unprivileged and cannot
+// change the ruleset: the control plane tells it what the settings say, it
+// writes that beside its models, and a root path unit on the machine
+// applies it with `llmgw firewall sync`.
+
+// FirewallState is what the cluster's settings say of a machine's firewall.
+type FirewallState struct {
+	Enabled bool `json:"enabled"`
+	// Networks are let in besides those the machine reaches others on.
+	Networks []string `json:"networks,omitempty"`
+	// Toward are the cluster's machines, by address: each one's network is
+	// let in, as the machine's route to it says.
+	Toward []string `json:"toward,omitempty"`
+}
+
+// FirewallFile is where, in the mirror's directory, the state is written.
+// Hidden, so it is never taken for a model.
+const FirewallFile = ".firewall.json"
+
+func (a *API) putFirewall(w http.ResponseWriter, r *http.Request) {
+	var st FirewallState
+	if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 1<<20)).Decode(&st); err != nil {
+		fail(w, http.StatusBadRequest, err)
+		return
+	}
+	b, _ := json.Marshal(st)
+	p := filepath.Join(a.m.root, FirewallFile)
+	changed := false
+	if old, err := os.ReadFile(p); err != nil || !bytes.Equal(old, b) {
+		// Renamed into place: the path unit sees one change, of a whole file.
+		tmp := p + ".new"
+		if err := os.WriteFile(tmp, b, 0o644); err != nil {
+			fail(w, http.StatusInternalServerError, err)
+			return
+		}
+		if err := os.Rename(tmp, p); err != nil {
+			fail(w, http.StatusInternalServerError, err)
+			return
+		}
+		changed = true
+	}
+	writeJSON(w, http.StatusOK, map[string]bool{"changed": changed})
+}
+
+// SetFirewall tells the mirror what the cluster's settings say of its
+// machine's firewall; it answers whether that changed anything.
+func (c *Client) SetFirewall(ctx context.Context, st FirewallState) (bool, error) {
+	var out struct {
+		Changed bool `json:"changed"`
+	}
+	err := c.do(ctx, http.MethodPut, "/v1/firewall", st, &out)
+	return out.Changed, err
+}

internal/mirror/firewall_test.go

@@ -0,0 +1,38 @@
+package mirror
+
+import (
+	"context"
+	"encoding/json"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
+)
+
+func TestTheMirrorKeepsWhatTheSettingsSayOfItsFirewall(t *testing.T) {
+	root := t.TempDir()
+	srv := httptest.NewServer(NewAPI(NewManager(root, discardLog()), "http://m:8081").Handler())
+	defer srv.Close()
+	c, err := NewClient(srv.URL, tlsconf.Config{}, 0)
+	if err != nil {
+		t.Fatal(err)
+	}
+	ctx := context.Background()
+	st := FirewallState{Enabled: true, Networks: []string{"192.168.5.0/24"}, Toward: []string{"10.0.0.147"}}
+	if changed, err := c.SetFirewall(ctx, st); err != nil || !changed {
+		t.Fatalf("first: %v %v", changed, err)
+	}
+	if changed, err := c.SetFirewall(ctx, st); err != nil || changed {
+		t.Errorf("the same again: %v %v", changed, err)
+	}
+	var back FirewallState
+	b, _ := os.ReadFile(filepath.Join(root, FirewallFile))
+	if err := json.Unmarshal(b, &back); err != nil || !back.Enabled || back.Toward[0] != "10.0.0.147" {
+		t.Errorf("kept %s", b)
+	}
+	if all, err := c.List(ctx); err != nil || len(all.Models) != 0 {
+		t.Errorf("the file is taken for a model: %+v %v", all, err)
+	}
+}

internal/mirror/upload_test.go

@@ -0,0 +1,45 @@
+package mirror
+
+import (
+	"context"
+	"net/http/httptest"
+	"os"
+	"strings"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
+)
+
+func TestAFileSentToTheMirrorIsKeptWithItsChecksumAndNeverReplaced(t *testing.T) {
+	root := t.TempDir()
+	srv := httptest.NewServer(NewAPI(NewManager(root, nil), "http://mirror:8081").Handler())
+	defer srv.Close()
+	c, err := NewClient(srv.URL, tlsconf.Config{}, 0)
+	if err != nil {
+		t.Fatal(err)
+	}
+	ctx := context.Background()
+	got, err := c.Upload(ctx, "politika-tonu", "train-1", "politika-tonu.gguf", []byte("GGUF adapter"), "trained by job train-1")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if got.URL != "http://mirror:8081/politika-tonu/train-1/politika-tonu.gguf" || got.Status.BytesTotal != 12 {
+		t.Errorf("answered %+v", got)
+	}
+	e := Entry{Model: "politika-tonu", Version: "train-1", File: "politika-tonu.gguf"}
+	if b, _ := os.ReadFile(e.Path(root)); string(b) != "GGUF adapter" {
+		t.Errorf("kept %q", b)
+	}
+	if b, _ := os.ReadFile(sourcePath(e.Path(root))); !strings.Contains(string(b), "train-1") {
+		t.Errorf("source %q", b)
+	}
+	if _, err := os.Stat(checksumPath(e.Path(root))); err != nil {
+		t.Errorf("no checksum: %v", err)
+	}
+	if _, err := c.Upload(ctx, "politika-tonu", "train-1", "politika-tonu.gguf", []byte("other"), "x"); err == nil {
+		t.Error("an existing version was replaced")
+	}
+	if _, err := c.Upload(ctx, "../etc", "v", "f", []byte("x"), "x"); err == nil {
+		t.Error("a path outside the mirror was taken")
+	}
+}

internal/pgha/memory.go

@@ -0,0 +1,171 @@
+package pgha
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"slices"
+	"strconv"
+	"strings"
+	"time"
+)
+
+// PostgreSQL's own defaults — 128MB of cache, 4MB to sort in — are for a
+// machine it barely uses. A knowledge base's vector index is searched well
+// only from memory: one that does not fit is read from disk on every search.
+// So each member is sized to the memory it has, read when setup runs, and
+// again on every later run, so a machine given more memory gets the use of it.
+
+// MemorySettings are PostgreSQL's memory parameters for a machine with total
+// bytes of memory. shared is a machine that also runs a control plane or an
+// agent: the database is given less of it, since a gateway swapped out to
+// give the database cache is worse than a smaller cache.
+func MemorySettings(total int64, shared bool) map[string]string {
+	const mb = 1 << 20
+	ram := total / mb
+	// Shares of memory, in MB: the cache PostgreSQL keeps itself, what the
+	// planner may count on the kernel caching besides, what building or
+	// vacuuming an index may take, and what one sort may.
+	buffers, cache, maint, sort := ram/4, ram*3/4, ram/16, ram/256
+	if shared {
+		buffers, cache, maint, sort = ram*3/20, ram/2, ram/32, ram/512
+	}
+	clamp := func(v, lo, hi int64) int64 { return min(max(v, lo), hi) }
+	return map[string]string{
+		"shared_buffers":       strconv.FormatInt(clamp(buffers, 96, 32<<10), 10) + "MB",
+		"effective_cache_size": strconv.FormatInt(clamp(cache, 256, 1<<20), 10) + "MB",
+		"maintenance_work_mem": strconv.FormatInt(clamp(maint, 64, 2<<10), 10) + "MB",
+		"work_mem":             strconv.FormatInt(clamp(sort, 4, 64), 10) + "MB",
+	}
+}
+
+// MemTotal reads the machine's memory, in bytes, from /proc/meminfo.
+func MemTotal(meminfo string) (int64, bool) {
+	for _, line := range strings.Split(meminfo, "\n") {
+		f := strings.Fields(line)
+		if len(f) >= 2 && f[0] == "MemTotal:" {
+			kb, err := strconv.ParseInt(f[1], 10, 64)
+			return kb << 10, err == nil && kb > 0
+		}
+	}
+	return 0, false
+}
+
+// memory sizes each member to its machine, in its own configuration — the
+// machines need not be alike. What takes effect on reload is applied so;
+// shared_buffers needs a restart, given one member at a time, the leader
+// last, each waited on before the next.
+func (s *Setup) memory(ctx context.Context, sec Secrets) error {
+	var restart []Node
+	for _, d := range s.DB {
+		out, err := d.Remote.Run(ctx, `cat /proc/meminfo
+if [ -f /etc/llmgw/server.yaml ] || [ -f /etc/llmgw/agent.yaml ]; then echo "llmgw-shared"; fi`, nil)
+		total, ok := MemTotal(string(out))
+		if err != nil || !ok {
+			s.emit("memory", false, "%s: could not read its memory; PostgreSQL left as it is", d.Name)
+			continue
+		}
+		shared := strings.Contains(string(out), "llmgw-shared")
+		params := MemorySettings(total, shared)
+		body, _ := json.Marshal(params)
+		out, err = d.Remote.Run(ctx, `set -e
+test -f /etc/patroni/config.yml || { echo missing; exit 0; }
+want=$(cat)
+r=$(sudo env WANT="$want" python3 - <<'PY'
+import json, os, yaml
+p = "/etc/patroni/config.yml"
+want = json.loads(os.environ["WANT"])
+c = yaml.safe_load(open(p))
+pg = c.setdefault("postgresql", {})
+params = pg.get("parameters") or {}
+pg["parameters"] = params
+if all(str(params.get(k)) == v for k, v in want.items()):
+    print("same")
+else:
+    restart = str(params.get("shared_buffers")) != want["shared_buffers"]
+    params.update(want)
+    open(p + ".new", "w").write("# llmgw: the control planes' store, kept by Patroni, its leader lock in etcd.\n" +
+        yaml.safe_dump(c, default_flow_style=False, sort_keys=False))
+    print("restart" if restart else "changed")
+PY
+)
+if [ "$r" != same ]; then
+  sudo chown root:postgres /etc/patroni/config.yml.new
+  sudo chmod 640 /etc/patroni/config.yml.new
+  sudo mv -f /etc/patroni/config.yml.new /etc/patroni/config.yml
+  sudo systemctl reload patroni
+fi
+echo "$r"`, body)
+		if err != nil {
+			return fmt.Errorf("on %s: %v: %s", d.Name, err, strings.TrimSpace(string(out)))
+		}
+		what := fmt.Sprintf("%s: %d MB of memory%s; shared_buffers %s, effective_cache_size %s, maintenance_work_mem %s, work_mem %s",
+			d.Name, total>>20, map[bool]string{true: ", shared with llmgw"}[shared],
+			params["shared_buffers"], params["effective_cache_size"], params["maintenance_work_mem"], params["work_mem"])
+		switch r := strings.TrimSpace(lastLine(out)); r {
+		case "same", "missing":
+			s.emit("memory", false, "%s", what)
+		case "restart":
+			s.emit("memory", true, "%s", what)
+			restart = append(restart, d)
+		default:
+			s.emit("memory", true, "%s", what)
+		}
+	}
+	// Patroni takes a reload in its own time: restarted before it has written
+	// the new configuration, PostgreSQL comes back on the old one. So each is
+	// waited on until Patroni says a restart is pending — and a member found
+	// pending, by a run that stopped before restarting it, is restarted too.
+	pending := func(c Cluster, name string) bool {
+		for _, m := range c.Members {
+			if m.Name == name {
+				return m.PendingRestart
+			}
+		}
+		return false
+	}
+	var c Cluster
+	err := s.waitFor(ctx, s.dbHosts(), func(got Cluster) bool {
+		c = got
+		for _, d := range restart {
+			if !pending(got, d.Name) {
+				return false
+			}
+		}
+		return true
+	})
+	if err != nil {
+		return fmt.Errorf("waiting for Patroni to take the new configuration: %w", err)
+	}
+	for _, d := range s.DB {
+		if pending(c, d.Name) && !slices.ContainsFunc(restart, func(n Node) bool { return n.Name == d.Name }) {
+			restart = append(restart, d)
+		}
+	}
+	if len(restart) == 0 {
+		return nil
+	}
+	// The leader last: its restart is the only one the control planes see,
+	// a few seconds without writes, and by then the rest are back.
+	slices.SortStableFunc(restart, func(a, b Node) int {
+		al, bl := a.Name == c.Leader(), b.Name == c.Leader()
+		switch {
+		case al && !bl:
+			return 1
+		case bl && !al:
+			return -1
+		}
+		return 0
+	})
+	for _, d := range restart {
+		if _, err := restJSON(ctx, d.IP, http.MethodPost, "/restart", []byte(`{"restart_pending": true}`), sec.RestAPI, 2*time.Minute); err != nil {
+			return fmt.Errorf("restarting PostgreSQL on %s for its new shared_buffers: %w", d.Name, err)
+		}
+		if err := s.waitFor(ctx, s.dbHosts(), func(c Cluster) bool { return c.Complete(len(s.DB)) }); err != nil {
+			return fmt.Errorf("after restarting %s: %w", d.Name, err)
+		}
+		s.emit("memory", true, "%s: PostgreSQL restarted with the new shared_buffers", d.Name)
+	}
+	return nil
+}

internal/pgha/pgha_test.go

@@ -465,3 +465,97 @@ func TestAControlPlaneWithoutTheAuthorityConnectsByTLSOnceTheClusterHasIt(t *tes
 		t.Errorf("TLS asked of a cluster without it: %s", dsn)
 	}
 }
+
+func TestPostgreSQLIsGivenAShareOfTheMachinesMemory(t *testing.T) {
+	total, ok := MemTotal("MemTotal:        4009848 kB\nMemFree:          976000 kB\n")
+	if !ok || total != 4009848<<10 {
+		t.Fatalf("MemTotal %d %v", total, ok)
+	}
+	own := MemorySettings(total, false)
+	if own["shared_buffers"] != "978MB" || own["effective_cache_size"] != "2936MB" ||
+		own["maintenance_work_mem"] != "244MB" || own["work_mem"] != "15MB" {
+		t.Errorf("a machine of its own: %v", own)
+	}
+	// Beside a control plane, less of it.
+	if sh := MemorySettings(total, true); sh["shared_buffers"] != "587MB" || sh["work_mem"] != "7MB" {
+		t.Errorf("a shared machine: %v", sh)
+	}
+	// Never below what PostgreSQL needs, never past what it can use.
+	if tiny := MemorySettings(512<<20, true); tiny["shared_buffers"] != "96MB" || tiny["work_mem"] != "4MB" {
+		t.Errorf("a small machine: %v", tiny)
+	}
+	if big := MemorySettings(1<<40, false); big["shared_buffers"] != "32768MB" || big["maintenance_work_mem"] != "2048MB" {
+		t.Errorf("a large machine: %v", big)
+	}
+}
+
+func TestNewSharedBuffersRestartTheReplicasBeforeTheLeader(t *testing.T) {
+	var mu sync.Mutex
+	var restarted []string
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if r.Method == http.MethodPost && r.URL.Path == "/restart" {
+			if _, pass, _ := r.BasicAuth(); pass != "pw" {
+				w.WriteHeader(http.StatusUnauthorized)
+				return
+			}
+			host, _, _ := strings.Cut(r.Host, ":")
+			mu.Lock()
+			restarted = append(restarted, host)
+			mu.Unlock()
+		}
+	}))
+	defer srv.Close()
+	u, _ := url.Parse(srv.URL)
+	old := restPort
+	restPort, _ = strconv.Atoi(u.Port())
+	defer func() { restPort = old }()
+
+	answers := map[string]string{"cat /proc/meminfo": "MemTotal: 4009848 kB\n", "WANT": "restart\n"}
+	leader := &machine{name: "a", answers: answers}
+	replica := &machine{name: "b", answers: answers}
+	up := Cluster{Members: []MemberState{{Name: "a", Role: "leader", State: "running", PendingRestart: true},
+		{Name: "b", Role: "replica", State: "streaming", PendingRestart: true}}}
+	s := Setup{
+		DB:     []Node{{Member{"a", "127.0.0.1"}, leader}, {Member{"b", "localhost"}, replica}},
+		Status: func(context.Context, []string) (Cluster, error) { return up, nil },
+		Wait:   time.Second,
+	}
+	if err := s.memory(context.Background(), Secrets{RestAPI: "pw"}); err != nil {
+		t.Fatal(err)
+	}
+	if strings.Join(restarted, " ") != "localhost 127.0.0.1" {
+		t.Errorf("restarted %v; the replica first, then the leader", restarted)
+	}
+}
+
+func TestAMemberLeftPendingARestartIsRestartedOnTheNextRun(t *testing.T) {
+	// A run that stopped after writing the configuration, or restarted before
+	// Patroni had taken it: nothing to change, but PostgreSQL still on the old.
+	var restarted []string
+	srv := httptest.NewServer(http.HandlerFunc(func(_ http.ResponseWriter, r *http.Request) {
+		if r.URL.Path == "/restart" {
+			host, _, _ := strings.Cut(r.Host, ":")
+			restarted = append(restarted, host)
+		}
+	}))
+	defer srv.Close()
+	u, _ := url.Parse(srv.URL)
+	old := restPort
+	restPort, _ = strconv.Atoi(u.Port())
+	defer func() { restPort = old }()
+
+	answers := map[string]string{"cat /proc/meminfo": "MemTotal: 4009848 kB\n", "WANT": "same\n"}
+	up := Cluster{Members: []MemberState{{Name: "a", Role: "leader", State: "running"},
+		{Name: "b", Role: "replica", State: "streaming", PendingRestart: true}}}
+	s := Setup{
+		DB:     []Node{{Member{"a", "127.0.0.1"}, &machine{answers: answers}}, {Member{"b", "localhost"}, &machine{answers: answers}}},
+		Status: func(context.Context, []string) (Cluster, error) { return up, nil },
+		Wait:   time.Second,
+	}
+	if err := s.memory(context.Background(), Secrets{RestAPI: "pw"}); err != nil {
+		t.Fatal(err)
+	}
+	if strings.Join(restarted, " ") != "localhost" {
+		t.Errorf("restarted %v; only the member pending", restarted)
+	}
+}

internal/pgha/setup.go

@@ -128,6 +128,9 @@ func (s *Setup) Run(ctx context.Context) (string, error) {
 	if err := s.ensurePatroniTLS(ctx, sec); err != nil {
 		return "", fmt.Errorf("patroni api: %w", err)
 	}
+	if err := s.memory(ctx, sec); err != nil {
+		return "", fmt.Errorf("memory: %w", err)
+	}
 	if err := s.database(ctx); err != nil {
 		return "", fmt.Errorf("database: %w", err)
 	}

internal/pgha/status.go

@@ -19,6 +19,9 @@ type MemberState struct {
 	State    string `json:"state"`
 	Host     string `json:"host"`
 	Timeline int    `json:"timeline"`
+	// PendingRestart is a parameter changed that PostgreSQL takes only on
+	// restart.
+	PendingRestart bool `json:"pending_restart"`
 	// Lag is how far behind the leader a replica is, in bytes; -1 when
 	// Patroni does not know, which is itself worth an alert.
 	Lag    int64           `json:"-"`

internal/router/adapterowner_test.go

@@ -0,0 +1,61 @@
+package router
+
+import (
+	"crypto/sha256"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/auth"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+func TestAnOwnedAdapterIsSeenOnlyInItsPartOfTheTree(t *testing.T) {
+	e := newFakeEngine(t, "qwen2.5-14b-instruct")
+	rt, owner := pricedRouter(t, e.srv.URL, "tone")
+	ctx := t.Context()
+	key, _ := rt.store.KeyByID(ctx, "key-1")
+	team, _ := rt.store.CreateTeam(ctx, mustOrgOf(t, rt), "other-team")
+	other, _ := rt.store.CreateProject(ctx, team, "other")
+	otherSecret := store.KeyPrefix + strings.Repeat("b", 43)
+	sum := sha256.Sum256([]byte(otherSecret))
+	if err := rt.store.CreateKey(ctx, store.Key{ID: "key-2", Hash: sum[:], Prefix: "bbbb", Name: "o", ProjectID: other, CreatedAt: time.Now()}); err != nil {
+		t.Fatal(err)
+	}
+	rt.storeModels([]catalog.Model{{Name: "qwen2.5-14b-instruct", Aliases: []string{"q14"},
+		Adapters: []catalog.Adapter{{Name: "tone", Source: "/t.gguf", Owner: "project:" + key.ProjectID}}}})
+	h := rt.Handler()
+	ask := func(secret string) int {
+		req := httptest.NewRequest(http.MethodPost, "/v1/chat/completions",
+			strings.NewReader(`{"model":"tone","messages":[{"role":"user","content":"hi"}]}`))
+		if secret != "" {
+			req.Header.Set("Authorization", "Bearer "+secret)
+		}
+		rec := httptest.NewRecorder()
+		h.ServeHTTP(rec, req)
+		return rec.Code
+	}
+	if code := ask(owner); code != http.StatusOK {
+		t.Errorf("its own project: %d", code)
+	}
+	if code := ask(otherSecret); code != http.StatusNotFound {
+		t.Errorf("another project: %d", code)
+	}
+	list := func(secret string) string {
+		req := httptest.NewRequest(http.MethodGet, "/v1/models", nil)
+		req.Header.Set("Authorization", "Bearer "+secret)
+		rec := httptest.NewRecorder()
+		h.ServeHTTP(rec, req)
+		return rec.Body.String()
+	}
+	if !strings.Contains(list(owner), `"tone"`) || strings.Contains(list(otherSecret), `"tone"`) {
+		t.Errorf("listed: own %s\nother %s", list(owner), list(otherSecret))
+	}
+	// No identity, no owned adapter.
+	if rt.adapterHidden(auth.Principal{}, false, "tone") != true {
+		t.Error("an owned adapter was shown to a caller with no identity")
+	}
+}

internal/router/handlers.go

@@ -247,6 +247,9 @@ func (r *Router) Handler() http.Handler {
 		if r.store != nil && r.guardLibrary != nil {
 			pages.RegisterAdminGuard(mux, r.store, r.guardLibrary)
 		}
+		if r.store != nil && r.settingsCheck != nil {
+			pages.RegisterAdminSettings(mux, r.store, r.settingsCheck, r.settingsLocal)
+		}
 		pages.Register(mux)
 		// The docs portal, with the menu of whoever is signed in, and only to
 		// them when the site asked for that.
@@ -409,9 +412,9 @@ type parsedRequest struct {
 	guardIn guard.Verdict
 	// knowledge is set when passages from knowledge bases were added.
 	knowledge bool
-	// kbOrg is the organization whose passages were added, for the prefix
+	// kbSalt is the collections whose passages were added, for the prefix
 	// cache's salt.
-	kbOrg string
+	kbSalt string
 	// noStore is a policy that keeps nothing of this request.
 	noStore bool
 	// inCluster is a project whose requests may not leave the cluster.
@@ -481,7 +484,8 @@ func (r *Router) handleInference(path string) http.HandlerFunc {
 			}
 		}
 
-		if r.hiddenModel(parsed.Model) {
+		p0, authed := auth.FromContext(req.Context())
+		if r.hiddenModel(parsed.Model) || r.adapterHidden(p0, authed, parsed.Model) {
 			// Said as a model that is not there: it is not, for clients.
 			writeErrorCode(w, http.StatusNotFound, "invalid_request_error", "model_not_found",
 				fmt.Sprintf("the model %q does not exist", parsed.Model))
@@ -1038,7 +1042,7 @@ func (r *Router) handleModels(w http.ResponseWriter, req *http.Request) {
 	principal, authed := auth.FromContext(req.Context())
 	names := make([]string, 0, len(seen))
 	for n := range seen {
-		if authed && !r.permits(principal, n) || r.hiddenModel(n) {
+		if authed && !r.permits(principal, n) || r.hiddenModel(n) || r.adapterHidden(principal, authed, n) {
 			continue
 		}
 		names = append(names, n)

internal/router/rag.go

@@ -10,6 +10,7 @@ import (
 	"fmt"
 	"net/http"
 	"net/url"
+	"sort"
 	"strconv"
 	"strings"
 	"time"
@@ -225,7 +226,14 @@ func (r *Router) retrieve(ctx context.Context, w http.ResponseWriter, req *parse
 	req.UpstreamBody = body
 	req.PromptTokens = len(body) / 4
 	req.knowledge = true
-	req.kbOrg = e.OrgID
+	// The collections searched, which is who else could have had these
+	// passages added: the prefix cache is shared among exactly them.
+	ids := make([]string, len(colls))
+	for i, c := range colls {
+		ids[i] = c.ID
+	}
+	sort.Strings(ids)
+	req.kbSalt = strings.Join(ids, ",")
 	// Set now: the proxy writes the headers when the answer starts.
 	w.Header().Set(sourcesHeader, strings.Join(sources, ","))
 	result := "found"
@@ -429,14 +437,16 @@ func (r *Router) kbDefaults(ctx context.Context, p auth.Principal, project strin
 	return names
 }
 
-// withCacheSalt sets vLLM's cache_salt to one derived from an organization:
-// the same for its requests, so they still share a cache among themselves.
-func withCacheSalt(body []byte, org string) []byte {
+// withCacheSalt sets vLLM's cache_salt to one derived from key — the
+// collections a request's passages came from — so requests that can see the
+// same passages still share a cache, and nobody else's do: not another
+// organization's, nor another team's of the same one.
+func withCacheSalt(body []byte, key string) []byte {
 	var top map[string]json.RawMessage
 	if json.Unmarshal(body, &top) != nil {
 		return body
 	}
-	sum := sha256.Sum256([]byte("llmgw-kb\x00" + org))
+	sum := sha256.Sum256([]byte("llmgw-kb\x00" + key))
 	top["cache_salt"], _ = json.Marshal(hex.EncodeToString(sum[:16]))
 	out, err := json.Marshal(top)
 	if err != nil {

internal/router/rag_test.go

@@ -52,9 +52,9 @@ func TestPassagesJoinTheLeadingSystemMessage(t *testing.T) {
 	}
 }
 
-func TestTheCacheSaltIsAnOrganizations(t *testing.T) {
+func TestTheCacheSaltFollowsTheCollections(t *testing.T) {
 	body := []byte(`{"model":"m","messages":[],"temperature":0}`)
-	a1, a2, b := withCacheSalt(body, "org-a"), withCacheSalt(body, "org-a"), withCacheSalt(body, "org-b")
+	a1, a2, b := withCacheSalt(body, "kb_a,kb_b"), withCacheSalt(body, "kb_a,kb_b"), withCacheSalt(body, "kb_a")
 	var x, y struct {
 		Salt string  `json:"cache_salt"`
 		T    float64 `json:"temperature"`
@@ -202,7 +202,7 @@ func mustID(t *testing.T, s *store.DB, name string) string {
 	return ""
 }
 
-func TestAVLLMReplicaGetsTheOrganizationsSaltOnlyWithPassages(t *testing.T) {
+func TestAVLLMReplicaGetsASaltOnlyWithPassages(t *testing.T) {
 	dsn := os.Getenv("LLMGW_TEST_POSTGRES")
 	if dsn == "" {
 		t.Skip("set LLMGW_TEST_POSTGRES to a pgvector database to run this")

internal/router/router.go

@@ -177,6 +177,10 @@ type Router struct {
 	// collections for kbDefaultTTL.
 	kbDefMu    sync.Mutex
 	kbDefCache map[string]kbDefaultEntry
+
+	// settingsCheck and settingsLocal mount the portal's settings page.
+	settingsCheck func([]byte) error
+	settingsLocal []string
 }
 
 // SetClusterView gives the router the data the /admin page renders.
@@ -200,6 +204,13 @@ func (r *Router) SetGuard(g guard.Guard) { r.guard = g }
 // SetGuardLibrary is the policies the guard was built from, for the pages.
 func (r *Router) SetGuardLibrary(c func() guard.Config) { r.guardLibrary = c }
 
+// SetSettingsCheck enables the portal's settings page: check says whether a
+// document would load as the cluster's settings beside this machine's own
+// server.yaml, and local are the keys that file keeps.
+func (r *Router) SetSettingsCheck(check func([]byte) error, local []string) {
+	r.settingsCheck, r.settingsLocal = check, local
+}
+
 // SetStopped refuses, or again serves, every request for a model.
 func (r *Router) SetStopped(stopped bool) { r.stopped.Store(stopped) }
 
@@ -702,7 +713,7 @@ func (r *Router) storeModels(models []catalog.Model) {
 		// is applied to the request on the way there.
 		for i, a := range entry.Adapters {
 			m[a.Name] = entry
-			ads[a.Name] = adapterRef{base: entry.Name, index: i, scale: a.EffectiveScale(),
+			ads[a.Name] = adapterRef{base: entry.Name, owner: a.Owner, index: i, scale: a.EffectiveScale(),
 				limit: entry.MaxLoadedAdapters, path: engines.RuntimeLoRAPath(a.Name, a.Source)}
 		}
 	}
@@ -724,7 +735,9 @@ func (r *Router) storeModels(models []catalog.Model) {
 
 // adapterRef is where an adapter runs and how it is applied.
 type adapterRef struct {
-	base  string
+	base string
+	// owner is who sees it (catalog.Adapter.Owner); empty is everyone.
+	owner string
 	index int
 	scale float64
 	// limit is the model's max_loaded_adapters: above zero, the adapter
@@ -746,6 +759,33 @@ func (r *Router) adapter(name string) (adapterRef, bool) {
 	return adapterRef{}, false
 }
 
+// adapterHidden reports whether name is an adapter its owner keeps from this
+// caller: one outside the part of the tree it belongs to, or one with no
+// identity at all. Such a caller is told there is no such model.
+func (r *Router) adapterHidden(p auth.Principal, authed bool, name string) bool {
+	ad, ok := r.adapter(name)
+	if !ok || ad.owner == "" {
+		return false
+	}
+	if !authed {
+		return true
+	}
+	scope, id, _ := strings.Cut(ad.owner, ":")
+	switch scope {
+	case "org":
+		return p.OrgID != id
+	case "team":
+		return p.TeamID != id
+	case "project":
+		return p.ProjectID != id
+	case "user":
+		return p.UserID != id
+	case "key":
+		return p.KeyID != id
+	}
+	return true
+}
+
 // LastUsed reports when each endpoint was last sent a request, satisfying
 // control.UsageSource so the control plane can tell a busy replica from one
 // holding its cards out of habit.
@@ -1079,13 +1119,13 @@ func (r *Router) dispatchOne(ctx context.Context, w http.ResponseWriter, req *pa
 		if isAdapter && !onDemand && dec.Endpoint.Engine != "vllm" {
 			body = withAdapter(body, model, ad.index, ad.scale)
 		}
-		// Passages of one organization's documents never share cached
-		// blocks with another's: a prefix found warm would say what another
-		// tenant asked about, by how fast its answer started. vLLM keys its
-		// prefix cache by this salt; other engines are not sent a field
-		// they may refuse.
-		if req.kbOrg != "" && dec.Endpoint.Engine == "vllm" && req.ContentType == "" {
-			body = withCacheSalt(body, req.kbOrg)
+		// Passages share cached blocks only with requests that could have
+		// been given the same ones: a prefix found warm would otherwise say
+		// what another tenant asked about, by how fast its answer started.
+		// vLLM keys its prefix cache by this salt; other engines are not
+		// sent a field they may refuse.
+		if req.kbSalt != "" && dec.Endpoint.Engine == "vllm" && req.ContentType == "" {
+			body = withCacheSalt(body, req.kbSalt)
 		}
 		// What computes the answer, said before the proxy writes the headers.
 		// A model can have replicas on cards and on CPUs at once, and the two

internal/store/clusterstate.go

@@ -0,0 +1,30 @@
+package store
+
+import (
+	"context"
+	"database/sql"
+	"errors"
+	"time"
+)
+
+// PutClusterState keeps a value under a key, for machines that read the
+// store rather than a control plane; it reports whether it changed.
+func (s *DB) PutClusterState(ctx context.Context, key, value string) (bool, error) {
+	if old, err := s.ClusterState(ctx, key); err == nil && old == value {
+		return false, nil
+	}
+	_, err := s.db.ExecContext(ctx, `INSERT INTO cluster_state (key, value, updated_at) VALUES (?, ?, ?)
+		ON CONFLICT (key) DO UPDATE SET value = excluded.value, updated_at = excluded.updated_at`,
+		key, value, millis(time.Now()))
+	return err == nil, err
+}
+
+// ClusterState reads a value; ErrNotFound when there is none.
+func (s *DB) ClusterState(ctx context.Context, key string) (string, error) {
+	var v string
+	err := s.db.QueryRowContext(ctx, `SELECT value FROM cluster_state WHERE key = ?`, key).Scan(&v)
+	if errors.Is(err, sql.ErrNoRows) {
+		return "", ErrNotFound
+	}
+	return v, err
+}

internal/store/copy.go

@@ -73,6 +73,9 @@ var copyOrder = []string{
 	"eval_runs",
 	"eval_cases",
 	"federation_peers",
+	// Before their artifacts, which point at them.
+	"train_jobs",
+	"train_artifacts",
 }
 
 // notCopied are tables deliberately left behind.
@@ -83,7 +86,8 @@ var copyOrder = []string{
 // the exact moment somebody is watching a migration and wondering why nothing
 // is being placed.
 var notCopied = map[string]string{
-	"leadership": "a lease on a running process, meaningless in a new store",
+	"leadership":    "a lease on a running process, meaningless in a new store",
+	"cluster_state": "worked out again by the leader from the settings and the machines, within minutes",
 }
 
 // CopyResult is how many rows each table received.

internal/store/knowledge.go

@@ -29,9 +29,13 @@ import (
 // ErrKnowledgeOff is a store that cannot hold knowledge bases.
 var ErrKnowledgeOff = errors.New("knowledge bases are off")
 
-// MaxEmbeddingDims is the most dimensions pgvector can index (HNSW on vector).
+// MaxEmbeddingDims is the most dimensions pgvector can index with HNSW: on
+// vector; halfvec takes up to MaxHalfvecDims.
 const MaxEmbeddingDims = 2000
 
+// MaxHalfvecDims is the most dimensions an HNSW index on halfvec takes.
+const MaxHalfvecDims = 4000
+
 // KBPerm is what a grant allows.
 type KBPerm string
 
@@ -50,8 +54,11 @@ type KBCollection struct {
 	Name       string
 	EmbedModel string
 	Dims       int
-	CreatedAt  time.Time
-	CreatedBy  string
+	// VecType is how its vectors are indexed: vector, or halfvec where
+	// pgvector has it (0.7 and later), at half the index's memory.
+	VecType   string
+	CreatedAt time.Time
+	CreatedBy string
 }
 
 // KBGrant lets a scope, and everything below it, read or write a collection.
@@ -133,11 +140,16 @@ func (s *DB) ensureKnowledge(ctx context.Context) error {
 				"as postgres, run CREATE EXTENSION vector in it (%v)", ErrKnowledgeOff, err)
 		}
 	}
+	var version string
+	if err := s.db.QueryRowContext(ctx, `SELECT extversion FROM pg_extension WHERE extname = 'vector'`).Scan(&version); err == nil {
+		s.kbHalfvec = versionAtLeast(version, 0, 7)
+	}
 	var exists bool
 	if err := s.db.QueryRowContext(ctx, `SELECT to_regclass('kb_chunks') IS NOT NULL`).Scan(&exists); err != nil {
 		return fmt.Errorf("%w: %v", ErrKnowledgeOff, err)
 	}
 	if exists {
+		s.kbBackfillIndexes(ctx)
 		return nil
 	}
 	err := s.inTx(ctx, func(c conn) error {
@@ -197,6 +209,52 @@ func kbVectorIndex(collection string) string {
 	return "kb_vec_" + strings.TrimPrefix(collection, "kb_")
 }
 
+// kbTextIndex is the name of a collection's full-text index.
+func kbTextIndex(collection string) string { return "kb_tsv_" + strings.TrimPrefix(collection, "kb_") }
+
+// cast is the type a collection's vectors are compared as, which its index
+// is built on: the query must say the same for the planner to use it.
+func (c KBCollection) cast() string {
+	t := c.VecType
+	if t != "halfvec" {
+		t = "vector"
+	}
+	return "::" + t + "(" + strconv.Itoa(c.Dims) + ")"
+}
+
+// versionAtLeast compares an extension's "major.minor.patch".
+func versionAtLeast(v string, major, minor int) bool {
+	parts := strings.SplitN(v, ".", 3)
+	if len(parts) < 2 {
+		return false
+	}
+	ma, err1 := strconv.Atoi(parts[0])
+	mi, err2 := strconv.Atoi(parts[1])
+	if err1 != nil || err2 != nil {
+		return false
+	}
+	return ma > major || ma == major && mi >= minor
+}
+
+// kbBackfillIndexes gives collections made before their full-text index
+// existed one. Best effort, at open: a search still works without it.
+func (s *DB) kbBackfillIndexes(ctx context.Context) {
+	cs, err := queryAll(ctx, s.db, `SELECT id, org_id FROM kb_collections`, nil, func(r *sql.Rows) ([2]string, error) {
+		var x [2]string
+		return x, r.Scan(&x[0], &x[1])
+	})
+	if err != nil {
+		return
+	}
+	for _, c := range cs {
+		if !kbIDPattern.MatchString(c[0]) {
+			continue
+		}
+		_, _ = s.db.ExecContext(ctx, `CREATE INDEX IF NOT EXISTS `+kbTextIndex(c[0])+` ON `+kbPartition(c[1])+
+			` USING gin (tsv) WHERE collection = `+sqlLiteral(c[0]))
+	}
+}
+
 func sqlLiteral(v string) string { return "'" + strings.ReplaceAll(v, "'", "''") + "'" }
 
 // kbScope sets, for this transaction, which rows of kb_chunks it may touch.
@@ -263,14 +321,22 @@ func (s *DB) CreateCollection(ctx context.Context, c KBCollection) (KBCollection
 		return KBCollection{}, errors.New("a collection needs a name")
 	case c.EmbedModel == "":
 		return KBCollection{}, errors.New("a collection needs the embedding model its documents are embedded with")
-	case c.Dims < 1 || c.Dims > MaxEmbeddingDims:
-		return KBCollection{}, fmt.Errorf("%d dimensions: an indexed embedding has 1 to %d", c.Dims, MaxEmbeddingDims)
+	case c.Dims < 1 || c.Dims > MaxHalfvecDims || c.Dims > MaxEmbeddingDims && !s.kbHalfvec:
+		limit := MaxEmbeddingDims
+		if s.kbHalfvec {
+			limit = MaxHalfvecDims
+		}
+		return KBCollection{}, fmt.Errorf("%d dimensions: an indexed embedding has 1 to %d", c.Dims, limit)
 	case c.OwnerScope == ScopeUser && c.OrgID == "":
 		return KBCollection{}, errors.New("a person's collection names the organization it belongs to")
 	case c.OwnerScope == ScopeKey:
 		return KBCollection{}, errors.New("a key cannot own a collection; its project or its person can")
 	}
 	c.ID = NewID("kb_")
+	c.VecType = "vector"
+	if s.kbHalfvec {
+		c.VecType = "halfvec"
+	}
 	if c.CreatedAt.IsZero() {
 		c.CreatedAt = time.Now()
 	}
@@ -292,9 +358,9 @@ func (s *DB) CreateCollection(ctx context.Context, c KBCollection) (KBCollection
 			return fmt.Errorf("%w: collection %q", ErrDuplicateName, c.Name)
 		}
 		if _, err := tx.ExecContext(ctx, `INSERT INTO kb_collections
-			(id, org_id, owner_scope, owner_id, name, embed_model, dims, created_at, created_by)
-			VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`,
-			c.ID, c.OrgID, string(c.OwnerScope), c.OwnerID, c.Name, c.EmbedModel, c.Dims,
+			(id, org_id, owner_scope, owner_id, name, embed_model, dims, vec_type, created_at, created_by)
+			VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
+			c.ID, c.OrgID, string(c.OwnerScope), c.OwnerID, c.Name, c.EmbedModel, c.Dims, c.VecType,
 			millis(c.CreatedAt), c.CreatedBy); err != nil {
 			return err
 		}
@@ -303,12 +369,18 @@ func (s *DB) CreateCollection(ctx context.Context, c KBCollection) (KBCollection
 			` PARTITION OF kb_chunks FOR VALUES IN (`+sqlLiteral(c.OrgID)+`)`); err != nil {
 			return err
 		}
-		// A partial index per collection, on its organization's partition
+		// Partial indexes per collection, on its organization's partition
 		// only: the dimension is the collection's, and a search, which
-		// names one collection at a time, uses exactly this one.
-		_, err = tx.ExecContext(ctx, `CREATE INDEX `+kbVectorIndex(c.ID)+` ON `+part+
-			` USING hnsw ((embedding::vector(`+strconv.Itoa(c.Dims)+`)) vector_cosine_ops)`+
-			` WHERE collection = `+sqlLiteral(c.ID))
+		// names one collection at a time, uses exactly these — a text
+		// search over one collection does not walk its whole
+		// organization's index.
+		if _, err := tx.ExecContext(ctx, `CREATE INDEX `+kbVectorIndex(c.ID)+` ON `+part+
+			` USING hnsw ((embedding`+c.cast()+`) `+c.VecType+`_cosine_ops)`+
+			` WHERE collection = `+sqlLiteral(c.ID)); err != nil {
+			return err
+		}
+		_, err = tx.ExecContext(ctx, `CREATE INDEX `+kbTextIndex(c.ID)+` ON `+part+
+			` USING gin (tsv) WHERE collection = `+sqlLiteral(c.ID))
 		return err
 	})
 	if err != nil {
@@ -317,13 +389,13 @@ func (s *DB) CreateCollection(ctx context.Context, c KBCollection) (KBCollection
 	return c, nil
 }
 
-const kbCollectionCols = `id, org_id, owner_scope, owner_id, name, embed_model, dims, created_at, created_by`
+const kbCollectionCols = `id, org_id, owner_scope, owner_id, name, embed_model, dims, vec_type, created_at, created_by`
 
 func scanCollection(sc scanner) (KBCollection, error) {
 	var c KBCollection
 	var scope string
 	var at int64
-	err := sc.Scan(&c.ID, &c.OrgID, &scope, &c.OwnerID, &c.Name, &c.EmbedModel, &c.Dims, &at, &c.CreatedBy)
+	err := sc.Scan(&c.ID, &c.OrgID, &scope, &c.OwnerID, &c.Name, &c.EmbedModel, &c.Dims, &c.VecType, &at, &c.CreatedBy)
 	c.OwnerScope, c.CreatedAt = Scope(scope), fromMillis(at)
 	return c, err
 }
@@ -377,7 +449,10 @@ func (s *DB) DeleteCollection(ctx context.Context, id string) error {
 		if _, err := c.ExecContext(ctx, `DELETE FROM kb_defaults WHERE collections = ''`); err != nil {
 			return err
 		}
-		_, err = c.ExecContext(ctx, `DROP INDEX IF EXISTS `+kbVectorIndex(id))
+		if _, err := c.ExecContext(ctx, `DROP INDEX IF EXISTS `+kbVectorIndex(id)); err != nil {
+			return err
+		}
+		_, err = c.ExecContext(ctx, `DROP INDEX IF EXISTS `+kbTextIndex(id))
 		return err
 	})
 }
@@ -729,6 +804,14 @@ func (s *DB) SearchChunks(ctx context.Context, q KBSearch) ([]KBHit, error) {
 		if err := kbScope(ctx, tx, q.OrgID, ids); err != nil {
 			return err
 		}
+		// pgvector's HNSW search returns at most ef_search rows (40 by
+		// default), whatever LIMIT asks: more candidates need it raised.
+		if len(q.Embeddings) > 0 {
+			ef := min(max(candidates(q.K), 40), 1000)
+			if _, err := tx.ExecContext(ctx, `SET LOCAL hnsw.ef_search = `+strconv.Itoa(ef)); err != nil {
+				return err
+			}
+		}
 		for _, c := range q.Collections {
 			// The literal collection and cast are what let the planner use
 			// that collection's partial index.
@@ -739,7 +822,7 @@ func (s *DB) SearchChunks(ctx context.Context, q KBSearch) ([]KBHit, error) {
 				if len(v) != c.Dims {
 					return fmt.Errorf("the query has %d dimensions; collection %s takes %d", len(v), c.Name, c.Dims)
 				}
-				cast := `::vector(` + strconv.Itoa(c.Dims) + `)`
+				cast := c.cast()
 				sim = `1 - (k.embedding` + cast + ` <=> ?` + cast + `)`
 				args = append(args, vectorText(v))
 				if err := query(tx, false, `SELECT k.collection, k.doc_id, d.name, k.ord, k.body, `+sim+`, 0::float8
@@ -753,15 +836,20 @@ func (s *DB) SearchChunks(ctx context.Context, q KBSearch) ([]KBHit, error) {
 			if tsq != "" {
 				// The text search's finds are measured against the question's
 				// meaning too, so the two are compared on one scale.
+				// Candidates by the cheap rank first, from the collection's
+				// own index; the share of words held and the similarity are
+				// worked out for those alone, not for every chunk that holds
+				// one word of the question.
 				targs := append([]any{}, args...)
 				for _, t := range terms {
 					targs = append(targs, t)
 				}
-				targs = append(targs, tsq, q.OrgID, candidates(q.K))
+				targs = append(targs, q.OrgID, tsq, tsq, candidates(q.K))
 				if err := query(tx, true, `SELECT k.collection, k.doc_id, d.name, k.ord, k.body, `+sim+`, `+coverage+` AS coverage
-					FROM kb_chunks k JOIN kb_documents d ON d.id = k.doc_id, to_tsquery('simple', ?) tq
-					WHERE k.org_id = ? AND k.collection = `+lit+` AND k.tsv @@ tq
-					ORDER BY coverage DESC, ts_rank(k.tsv, tq) DESC LIMIT ?`, targs...); err != nil {
+					FROM (SELECT * FROM kb_chunks k
+					      WHERE k.org_id = ? AND k.collection = `+lit+` AND k.tsv @@ to_tsquery('simple', ?)
+					      ORDER BY ts_rank(k.tsv, to_tsquery('simple', ?)) DESC LIMIT ?) k
+					JOIN kb_documents d ON d.id = k.doc_id`, targs...); err != nil {
 					return err
 				}
 			}
@@ -887,8 +975,10 @@ func (s *DB) kbForgetScope(ctx context.Context, c conn, scope Scope, id string)
 			return err
 		}
 		if kbIDPattern.MatchString(cid) {
-			if _, err := c.ExecContext(ctx, `DROP INDEX IF EXISTS `+kbVectorIndex(cid)); err != nil {
-				return err
+			for _, idx := range []string{kbVectorIndex(cid), kbTextIndex(cid)} {
+				if _, err := c.ExecContext(ctx, `DROP INDEX IF EXISTS `+idx); err != nil {
+					return err
+				}
 			}
 		}
 	}
@@ -1026,3 +1116,32 @@ func (s *DB) KBDefaultFor(ctx context.Context, e Effective) ([]string, error) {
 	}
 	return nil, nil
 }
+
+// CollectionChunks lists a collection's chunks in document order, for
+// building training data from them (BACKLOG F4). At most limit.
+func (s *DB) CollectionChunks(ctx context.Context, c KBCollection, limit int) ([]KBHit, error) {
+	if err := s.KnowledgeAvailable(); err != nil {
+		return nil, err
+	}
+	if !kbIDPattern.MatchString(c.ID) {
+		return nil, ErrNotFound
+	}
+	var out []KBHit
+	err := s.inTx(ctx, func(tx conn) error {
+		if err := kbScope(ctx, tx, c.OrgID, []string{c.ID}); err != nil {
+			return err
+		}
+		var err error
+		out, err = queryAll(ctx, tx, `SELECT k.collection, k.doc_id, d.name, k.ord, k.body
+			FROM kb_chunks k JOIN kb_documents d ON d.id = k.doc_id
+			WHERE k.org_id = ? AND k.collection = `+sqlLiteral(c.ID)+`
+			ORDER BY d.name, k.ord LIMIT ?`, []any{c.OrgID, limit},
+			func(r *sql.Rows) (KBHit, error) {
+				var h KBHit
+				err := r.Scan(&h.Collection, &h.DocID, &h.Document, &h.Ord, &h.Body)
+				return h, err
+			})
+		return err
+	})
+	return out, err
+}

internal/store/knowledge_test.go

@@ -3,6 +3,7 @@ package store
 import (
 	"context"
 	"errors"
+	"fmt"
 	"math"
 	"strings"
 	"testing"
@@ -320,10 +321,85 @@ func TestPostgresScopesWithCollectionsAreNotDeleted(t *testing.T) {
 	if err := s.DeleteProject(ctx, "proj2"); err != nil {
 		t.Errorf("deleting the project once its collection is gone: %v", err)
 	}
+	for _, idx := range []string{kbVectorIndex(c.ID), kbTextIndex(c.ID)} {
+		if indexExists(t, s, idx) {
+			t.Errorf("the collection's index %s is left behind", idx)
+		}
+	}
+}
+
+func indexExists(t *testing.T, s *DB, name string) bool {
+	t.Helper()
 	var n int
-	if err := s.raw.QueryRowContext(ctx, `SELECT COUNT(*) FROM pg_indexes WHERE indexname = $1`,
-		kbVectorIndex(c.ID)).Scan(&n); err != nil || n != 0 {
-		t.Errorf("the collection's index is left behind (%d, %v)", n, err)
+	if err := s.raw.QueryRowContext(context.Background(), `SELECT COUNT(*) FROM pg_indexes WHERE indexname = $1`,
+		name).Scan(&n); err != nil {
+		t.Fatal(err)
+	}
+	return n == 1
+}
+
+func TestPostgresCollectionsHaveTheirOwnIndexes(t *testing.T) {
+	ctx := context.Background()
+	s := openKB(t)
+	kbTwoOrgs(t, s)
+	c := mustCollection(t, s, ScopeProject, "proj1", "", "docs")
+	if !indexExists(t, s, kbVectorIndex(c.ID)) || !indexExists(t, s, kbTextIndex(c.ID)) {
+		t.Fatal("a new collection lacks its vector or text index")
+	}
+	// The test image's pgvector has halfvec; a collection there uses it, and
+	// is searched as it is indexed.
+	if !s.kbHalfvec || c.VecType != "halfvec" {
+		t.Errorf("halfvec %v, the collection's type %q", s.kbHalfvec, c.VecType)
+	}
+	if got, err := s.Collection(ctx, c.ID); err != nil || got.VecType != c.VecType {
+		t.Errorf("read back %q, %v", got.VecType, err)
+	}
+	var def string
+	if err := s.raw.QueryRowContext(ctx, `SELECT indexdef FROM pg_indexes WHERE indexname = $1`, kbVectorIndex(c.ID)).Scan(&def); err != nil ||
+		!strings.Contains(def, "halfvec_cosine_ops") {
+		t.Errorf("the vector index is %q (%v)", def, err)
+	}
+	if _, err := s.CreateCollection(ctx, KBCollection{OwnerScope: ScopeProject, OwnerID: "proj1",
+		Name: "wide", EmbedModel: "emb", Dims: 3072}); err != nil {
+		t.Errorf("3072 dimensions, which halfvec indexes: %v", err)
+	}
+	if _, err := s.CreateCollection(ctx, KBCollection{OwnerScope: ScopeProject, OwnerID: "proj1",
+		Name: "too-wide", EmbedModel: "emb", Dims: MaxHalfvecDims + 1}); err == nil {
+		t.Error("more dimensions than any index takes were accepted")
+	}
+
+	// A collection made before text indexes were gets one when the store
+	// opens.
+	if _, err := s.raw.ExecContext(ctx, `DROP INDEX `+kbTextIndex(c.ID)); err != nil {
+		t.Fatal(err)
+	}
+	s.kbBackfillIndexes(ctx)
+	if !indexExists(t, s, kbTextIndex(c.ID)) {
+		t.Error("the text index was not put back")
+	}
+
+	// Asking for more than HNSW's default 40 is answered in full. (A table
+	// this small is read without its index: with one, 40 is all HNSW gives
+	// unless ef_search is raised, as an EXPLAIN on 20000 rows showed.)
+	var chunks []KBChunk
+	for i := range 120 {
+		chunks = append(chunks, KBChunk{Body: fmt.Sprintf("parça %d izin", i), Embedding: unit(i % 3)})
+	}
+	if _, _, err := s.PutDocument(ctx, c, KBDocument{Name: "big.md", Digest: "big"}, chunks); err != nil {
+		t.Fatal(err)
+	}
+	hits, err := s.SearchChunks(ctx, KBSearch{OrgID: "org1", Collections: []KBCollection{c},
+		Embeddings: map[string][]float32{"emb": unit(0)}, K: 50})
+	if err != nil || len(hits) != 50 {
+		t.Errorf("K=50 gave %d hits, %v", len(hits), err)
+	}
+}
+
+func TestVersionAtLeast(t *testing.T) {
+	for v, want := range map[string]bool{"0.6.0": false, "0.7.0": true, "0.8.7": true, "1.0": true, "0": false, "x.y": false} {
+		if got := versionAtLeast(v, 0, 7); got != want {
+			t.Errorf("%s: %v", v, got)
+		}
 	}
 }
 

internal/store/migrations/postgres/0037_train_jobs.sql

@@ -0,0 +1,42 @@
+-- Training jobs (BACKLOG F4): a LoRA adapter trained on a node, from a base
+-- model's weights and question/answer pairs, and what it made. The pairs
+-- and the adapter are kept here, in the store every control plane reads:
+-- they are megabytes, and a job's record is worth little without them.
+CREATE TABLE train_jobs (
+  id             TEXT PRIMARY KEY,
+  -- name is the adapter's: what clients will ask for it by.
+  name           TEXT NOT NULL,
+  -- model is the catalog model the adapter is for; base_repo and
+  -- base_revision the weights it is trained on.
+  model          TEXT NOT NULL,
+  base_repo      TEXT NOT NULL,
+  base_revision  TEXT NOT NULL DEFAULT '',
+  node           TEXT NOT NULL,
+  gpu_memory_mib INTEGER NOT NULL,
+  params         TEXT NOT NULL DEFAULT '',
+  gguf           INTEGER NOT NULL DEFAULT 0,
+  data           BYTEA NOT NULL,
+  pairs          INTEGER NOT NULL,
+  -- state is queued, starting, preparing, running, done, failed or
+  -- cancelled.
+  state          TEXT NOT NULL,
+  step           INTEGER NOT NULL DEFAULT 0,
+  steps          INTEGER NOT NULL DEFAULT 0,
+  loss           DOUBLE PRECISION NOT NULL DEFAULT 0,
+  message        TEXT NOT NULL DEFAULT '',
+  peak_mib       INTEGER NOT NULL DEFAULT 0,
+  created_at     BIGINT NOT NULL,
+  started_at     BIGINT NOT NULL DEFAULT 0,
+  finished_at    BIGINT NOT NULL DEFAULT 0,
+  created_by     TEXT NOT NULL DEFAULT ''
+);
+CREATE INDEX train_jobs_by_state ON train_jobs (state);
+
+-- What a job made: the PEFT adapter as a gzipped tar, and for llama.cpp the
+-- adapter as GGUF.
+CREATE TABLE train_artifacts (
+  job_id TEXT NOT NULL REFERENCES train_jobs(id) ON DELETE CASCADE,
+  kind   TEXT NOT NULL,
+  data   BYTEA NOT NULL,
+  PRIMARY KEY (job_id, kind)
+);

internal/store/migrations/postgres/0038_train_owner.sql

@@ -0,0 +1,5 @@
+-- Who an adapter belongs to (BACKLOG F4): scope:ID in the tenancy tree, or
+-- empty for every caller its model allows. An adapter trained on a
+-- knowledge base's documents carries some of what they say, and belongs to
+-- a part of the tree that may read them.
+ALTER TABLE train_jobs ADD COLUMN owner TEXT NOT NULL DEFAULT '';

internal/store/migrations/postgres/0039_kb_vec_type.sql

@@ -0,0 +1,5 @@
+-- How a collection's vectors are indexed: vector (single precision) or
+-- halfvec (half, pgvector 0.7 and later), which halves the index a search
+-- has to hold in memory for the same recall. Fixed when the collection is
+-- made: its index and its queries name the same type.
+ALTER TABLE kb_collections ADD COLUMN vec_type TEXT NOT NULL DEFAULT 'vector';

internal/store/migrations/postgres/0040_cluster_state.sql

@@ -0,0 +1,7 @@
+-- What a control plane works out for machines that cannot ask it: the
+-- store's members read it from their own copy of the store.
+CREATE TABLE cluster_state (
+  key        TEXT PRIMARY KEY,
+  value      TEXT NOT NULL,
+  updated_at BIGINT NOT NULL
+);

internal/store/migrations/sqlite/0037_train_jobs.sql

@@ -0,0 +1,42 @@
+-- Training jobs (BACKLOG F4): a LoRA adapter trained on a node, from a base
+-- model's weights and question/answer pairs, and what it made. The pairs
+-- and the adapter are kept here, in the store every control plane reads:
+-- they are megabytes, and a job's record is worth little without them.
+CREATE TABLE train_jobs (
+  id             TEXT PRIMARY KEY,
+  -- name is the adapter's: what clients will ask for it by.
+  name           TEXT NOT NULL,
+  -- model is the catalog model the adapter is for; base_repo and
+  -- base_revision the weights it is trained on.
+  model          TEXT NOT NULL,
+  base_repo      TEXT NOT NULL,
+  base_revision  TEXT NOT NULL DEFAULT '',
+  node           TEXT NOT NULL,
+  gpu_memory_mib INTEGER NOT NULL,
+  params         TEXT NOT NULL DEFAULT '',
+  gguf           INTEGER NOT NULL DEFAULT 0,
+  data           BLOB NOT NULL,
+  pairs          INTEGER NOT NULL,
+  -- state is queued, starting, preparing, running, done, failed or
+  -- cancelled.
+  state          TEXT NOT NULL,
+  step           INTEGER NOT NULL DEFAULT 0,
+  steps          INTEGER NOT NULL DEFAULT 0,
+  loss           REAL NOT NULL DEFAULT 0,
+  message        TEXT NOT NULL DEFAULT '',
+  peak_mib       INTEGER NOT NULL DEFAULT 0,
+  created_at     INTEGER NOT NULL,
+  started_at     INTEGER NOT NULL DEFAULT 0,
+  finished_at    INTEGER NOT NULL DEFAULT 0,
+  created_by     TEXT NOT NULL DEFAULT ''
+);
+CREATE INDEX train_jobs_by_state ON train_jobs (state);
+
+-- What a job made: the PEFT adapter as a gzipped tar, and for llama.cpp the
+-- adapter as GGUF.
+CREATE TABLE train_artifacts (
+  job_id TEXT NOT NULL REFERENCES train_jobs(id) ON DELETE CASCADE,
+  kind   TEXT NOT NULL,
+  data   BLOB NOT NULL,
+  PRIMARY KEY (job_id, kind)
+);

internal/store/migrations/sqlite/0038_train_owner.sql

@@ -0,0 +1,5 @@
+-- Who an adapter belongs to (BACKLOG F4): scope:ID in the tenancy tree, or
+-- empty for every caller its model allows. An adapter trained on a
+-- knowledge base's documents carries some of what they say, and belongs to
+-- a part of the tree that may read them.
+ALTER TABLE train_jobs ADD COLUMN owner TEXT NOT NULL DEFAULT '';

internal/store/migrations/sqlite/0040_cluster_state.sql

@@ -0,0 +1,7 @@
+-- What a control plane works out for machines that cannot ask it: the
+-- store's members read it from their own copy of the store.
+CREATE TABLE cluster_state (
+  key        TEXT PRIMARY KEY,
+  value      TEXT NOT NULL,
+  updated_at INTEGER NOT NULL
+);

internal/store/sqlite.go

@@ -52,6 +52,9 @@ type DB struct {
 	// kbErr is why this store cannot hold knowledge bases, nil when it can;
 	// set once at open (knowledge.go).
 	kbErr error
+	// kbHalfvec is pgvector 0.7 or later here: new collections index their
+	// vectors as halfvec.
+	kbHalfvec bool
 }
 
 // OpenSQLite opens (creating if needed) a database and brings it up to date.

internal/store/train.go

@@ -0,0 +1,203 @@
+package store
+
+import (
+	"context"
+	"database/sql"
+	"errors"
+	"time"
+)
+
+// TrainJob is a LoRA adapter trained on a node (BACKLOG F4).
+type TrainJob struct {
+	ID           string
+	Name         string
+	Model        string
+	BaseRepo     string
+	BaseRevision string
+	Node         string
+	// Owner is who the adapter belongs to, scope:ID; empty is everyone.
+	Owner        string
+	GPUMemoryMiB int
+	Params       string
+	GGUF         bool
+	// Data is the pairs, NDJSON; Pairs how many.
+	Data  []byte
+	Pairs int
+
+	State      string
+	Step       int
+	Steps      int
+	Loss       float64
+	Message    string
+	PeakMiB    int
+	CreatedAt  time.Time
+	StartedAt  time.Time
+	FinishedAt time.Time
+	CreatedBy  string
+}
+
+// The states of a training job.
+const (
+	TrainQueued    = "queued"
+	TrainStarting  = "starting"
+	TrainDone      = "done"
+	TrainFailed    = "failed"
+	TrainCancelled = "cancelled"
+)
+
+// TrainOver is whether a state is one a job ends in.
+func TrainOver(state string) bool {
+	return state == TrainDone || state == TrainFailed || state == TrainCancelled
+}
+
+// CreateTrainJob queues a job.
+func (s *DB) CreateTrainJob(ctx context.Context, j TrainJob) error {
+	if j.CreatedAt.IsZero() {
+		j.CreatedAt = time.Now()
+	}
+	gguf := 0
+	if j.GGUF {
+		gguf = 1
+	}
+	_, err := s.db.ExecContext(ctx, `INSERT INTO train_jobs (id, name, model, base_repo, base_revision, node, owner, gpu_memory_mib,
+		params, gguf, data, pairs, state, created_at, created_by) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
+		j.ID, j.Name, j.Model, j.BaseRepo, j.BaseRevision, j.Node, j.Owner, j.GPUMemoryMiB, j.Params, gguf, j.Data, j.Pairs,
+		TrainQueued, millis(j.CreatedAt), j.CreatedBy)
+	return err
+}
+
+const trainCols = `id, name, model, base_repo, base_revision, node, owner, gpu_memory_mib, params, gguf, pairs, state, step, steps,
+	loss, message, peak_mib, created_at, started_at, finished_at, created_by`
+
+func scanTrainJob(sc scanner) (TrainJob, error) {
+	var j TrainJob
+	var gguf int
+	var created, started, finished int64
+	err := sc.Scan(&j.ID, &j.Name, &j.Model, &j.BaseRepo, &j.BaseRevision, &j.Node, &j.Owner, &j.GPUMemoryMiB, &j.Params, &gguf,
+		&j.Pairs, &j.State, &j.Step, &j.Steps, &j.Loss, &j.Message, &j.PeakMiB, &created, &started, &finished, &j.CreatedBy)
+	j.GGUF = gguf != 0
+	j.CreatedAt = fromMillis(created)
+	if started > 0 {
+		j.StartedAt = fromMillis(started)
+	}
+	if finished > 0 {
+		j.FinishedAt = fromMillis(finished)
+	}
+	return j, err
+}
+
+// TrainJobs lists jobs, newest first.
+func (s *DB) TrainJobs(ctx context.Context, limit int) ([]TrainJob, error) {
+	return queryAll(ctx, s.db, `SELECT `+trainCols+` FROM train_jobs ORDER BY created_at DESC LIMIT ?`, []any{limit},
+		func(r *sql.Rows) (TrainJob, error) { return scanTrainJob(r) })
+}
+
+// TrainJob reads one job, with its data.
+func (s *DB) TrainJob(ctx context.Context, id string) (TrainJob, error) {
+	j, err := scanTrainJob(s.db.QueryRowContext(ctx, `SELECT `+trainCols+` FROM train_jobs WHERE id = ?`, id))
+	if errors.Is(err, sql.ErrNoRows) {
+		return TrainJob{}, ErrNotFound
+	}
+	if err != nil {
+		return j, err
+	}
+	err = s.db.QueryRowContext(ctx, `SELECT data FROM train_jobs WHERE id = ?`, id).Scan(&j.Data)
+	return j, err
+}
+
+// QueuedTrainJobs are the jobs waiting for their node, oldest first.
+func (s *DB) QueuedTrainJobs(ctx context.Context) ([]TrainJob, error) {
+	return queryAll(ctx, s.db, `SELECT `+trainCols+` FROM train_jobs WHERE state = ? ORDER BY created_at`, []any{TrainQueued},
+		func(r *sql.Rows) (TrainJob, error) { return scanTrainJob(r) })
+}
+
+// ActiveTrainJobs are the jobs sent to a node and not ended.
+func (s *DB) ActiveTrainJobs(ctx context.Context) ([]TrainJob, error) {
+	return queryAll(ctx, s.db, `SELECT `+trainCols+` FROM train_jobs WHERE state NOT IN (?, ?, ?, ?)`,
+		[]any{TrainQueued, TrainDone, TrainFailed, TrainCancelled},
+		func(r *sql.Rows) (TrainJob, error) { return scanTrainJob(r) })
+}
+
+// UpdateTrainJob records how a job goes; a job that has ended is not
+// changed again, so a late message cannot bring one back.
+func (s *DB) UpdateTrainJob(ctx context.Context, j TrainJob) error {
+	var started, finished int64
+	if !j.StartedAt.IsZero() {
+		started = millis(j.StartedAt)
+	}
+	if !j.FinishedAt.IsZero() {
+		finished = millis(j.FinishedAt)
+	}
+	_, err := s.db.ExecContext(ctx, `UPDATE train_jobs SET state = ?, step = ?, steps = ?, loss = ?, message = ?,
+		peak_mib = ?, started_at = ?, finished_at = ? WHERE id = ? AND state NOT IN (?, ?, ?)`,
+		j.State, j.Step, j.Steps, j.Loss, j.Message, j.PeakMiB, started, finished, j.ID,
+		TrainDone, TrainFailed, TrainCancelled)
+	return err
+}
+
+// PutTrainArtifact keeps what a job made.
+func (s *DB) PutTrainArtifact(ctx context.Context, job, kind string, data []byte) error {
+	_, err := s.db.ExecContext(ctx, `INSERT INTO train_artifacts (job_id, kind, data) VALUES (?, ?, ?)
+		ON CONFLICT (job_id, kind) DO UPDATE SET data = excluded.data`, job, kind, data)
+	return err
+}
+
+// TrainArtifact reads what a job made, of one kind.
+func (s *DB) TrainArtifact(ctx context.Context, job, kind string) ([]byte, error) {
+	var b []byte
+	err := s.db.QueryRowContext(ctx, `SELECT data FROM train_artifacts WHERE job_id = ? AND kind = ?`, job, kind).Scan(&b)
+	if errors.Is(err, sql.ErrNoRows) {
+		return nil, ErrNotFound
+	}
+	return b, err
+}
+
+// TrainArtifactKinds lists what a job made.
+func (s *DB) TrainArtifactKinds(ctx context.Context, job string) ([]string, error) {
+	return queryAll(ctx, s.db, `SELECT kind FROM train_artifacts WHERE job_id = ? ORDER BY kind`, []any{job},
+		func(r *sql.Rows) (string, error) {
+			var k string
+			return k, r.Scan(&k)
+		})
+}
+
+// DeleteTrainJob removes a job and what it made.
+func (s *DB) DeleteTrainJob(ctx context.Context, id string) error {
+	return s.inTx(ctx, func(c conn) error {
+		if _, err := c.ExecContext(ctx, `DELETE FROM train_artifacts WHERE job_id = ?`, id); err != nil {
+			return err
+		}
+		res, err := c.ExecContext(ctx, `DELETE FROM train_jobs WHERE id = ?`, id)
+		if err != nil {
+			return err
+		}
+		if n, _ := res.RowsAffected(); n == 0 {
+			return ErrNotFound
+		}
+		return nil
+	})
+}
+
+// EffectiveFor is a scope's place in the tree, as a caller in it would have
+// it, for asking what it may read: an organization, a team or a project
+// with what is above it, a key as it resolves, a person in an organization.
+func (s *DB) EffectiveFor(ctx context.Context, scope Scope, id, org string) (Effective, error) {
+	switch scope {
+	case ScopeOrg:
+		return Effective{OrgID: id}, nil
+	case ScopeTeam:
+		var o string
+		err := s.db.QueryRowContext(ctx, `SELECT org_id FROM teams WHERE id = ?`, id).Scan(&o)
+		if errors.Is(err, sql.ErrNoRows) {
+			return Effective{}, ErrNotFound
+		}
+		return Effective{TeamID: id, OrgID: o}, err
+	case ScopeProject:
+		return s.ResolveFor(ctx, "", "", id)
+	case ScopeKey:
+		return s.Resolve(ctx, id)
+	case ScopeUser:
+		return Effective{UserID: id, OrgID: org}, nil
+	}
+	return Effective{}, errors.New("no such scope " + string(scope))
+}

internal/web/account.go

@@ -90,7 +90,9 @@ type AccountPages struct {
 	audit audit.Sink
 	// hasGuard is set when /admin/guard is registered, and guardLibrary
 	// with it: the policies in force, whose judges the mirror page serves.
-	hasGuard     bool
+	hasGuard bool
+	// hasSettings is set when /admin/settings is registered.
+	hasSettings  bool
 	guardLibrary func() guard.Config
 	// guardFor names the guardrail policies in force for someone, and
 	// whether they keep nothing; nil when there are none.

internal/web/admin_settings.go

@@ -0,0 +1,174 @@
+package web
+
+import (
+	"context"
+	"errors"
+	"net/http"
+	"strconv"
+	"strings"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// The cluster's settings, edited in the browser (BACKLOG E4): the document
+// `llmgw ctl config edit` opens, for whoever has no terminal on a control
+// plane. The same rules hold here as there — a version is checked as a
+// control plane would load it before it is kept, it is made from the newest
+// version or refused, and every change is in force on every control plane
+// within seconds and in the audit log.
+
+// SettingsAdmin is where the cluster's settings are kept.
+type SettingsAdmin interface {
+	ClusterSettings(ctx context.Context) (store.SettingsVersion, error)
+	ClusterSettingsAt(ctx context.Context, version int64) (store.SettingsVersion, error)
+	ClusterSettingsHistory(ctx context.Context, limit int) ([]store.SettingsVersion, error)
+	PutClusterSettings(ctx context.Context, base int64, body, by, note string) (int64, error)
+}
+
+type settingsPageData struct {
+	Nav     Nav
+	Version string
+	CSRF    string
+	// Current is the version in force; Shown the one in the editor, which
+	// is an earlier one when ?version= asked for it.
+	Current, Shown store.SettingsVersion
+	Earlier        bool
+	// Body is what the editor holds: the version shown, or what was sent
+	// and refused, so a refusal does not lose the edit.
+	Body      string
+	Note      string
+	History   []settingsRow
+	LocalKeys []string
+	Notice    string
+	Error     string
+}
+
+type settingsRow struct {
+	Version  int64
+	At, By   string
+	Note     string
+	InForce  bool
+	Selected bool
+}
+
+// RegisterAdminSettings mounts /admin/settings, for administrators only:
+// the settings hold webhook addresses and the like, so even reading them is
+// not every staff role's.
+func (p *AccountPages) RegisterAdminSettings(mux *http.ServeMux, s SettingsAdmin, check func([]byte) error, localKeys []string) {
+	p.hasSettings = true
+	page := func(w http.ResponseWriter, r *http.Request, admin store.User, d settingsPageData) {
+		ctx, l := r.Context(), langOf(r)
+		d.Nav, d.Version, d.CSRF, d.LocalKeys = p.navFor(r, admin, "settings"), p.version, csrfFor(r), localKeys
+		cur, err := s.ClusterSettings(ctx)
+		if err != nil {
+			d.Error = l.Err(err)
+		}
+		d.Current = cur
+		if d.Shown.Version == 0 {
+			d.Shown = cur
+		}
+		d.Earlier = d.Shown.Version != cur.Version
+		if d.Body == "" {
+			d.Body = d.Shown.Body
+		}
+		if hist, err := s.ClusterSettingsHistory(ctx, 30); err == nil {
+			for _, v := range hist {
+				d.History = append(d.History, settingsRow{Version: v.Version, At: l.Date(v.At.UTC()) + " " + v.At.UTC().Format("15:04"),
+					By: v.By, Note: v.Note, InForce: v.Version == cur.Version, Selected: v.Version == d.Shown.Version})
+			}
+		}
+		noStore(w)
+		render(w, r, "admin_settings", d)
+	}
+	adminsOnly := func(next func(w http.ResponseWriter, r *http.Request, admin store.User)) http.HandlerFunc {
+		return p.adminOnly(func(w http.ResponseWriter, r *http.Request, admin store.User) {
+			if !admin.IsAdmin {
+				http.NotFound(w, r)
+				return
+			}
+			if r.Method == http.MethodPost {
+				sess, _, _ := p.session(r)
+				if !p.accounts.CheckCSRF(sess, r.FormValue("csrf")) {
+					http.Error(w, "bad request", http.StatusBadRequest)
+					return
+				}
+			}
+			next(w, r, admin)
+		})
+	}
+
+	mux.HandleFunc("GET /admin/settings", adminsOnly(func(w http.ResponseWriter, r *http.Request, admin store.User) {
+		var d settingsPageData
+		if q := r.URL.Query().Get("version"); q != "" {
+			n, err := strconv.ParseInt(q, 10, 64)
+			if v, verr := s.ClusterSettingsAt(r.Context(), n); err == nil && verr == nil && v.Version != 0 {
+				d.Shown = v
+			} else {
+				d.Error = langOf(r).T("settings.err.no_version")
+			}
+		}
+		page(w, r, admin, d)
+	}))
+
+	mux.HandleFunc("POST /admin/settings", adminsOnly(func(w http.ResponseWriter, r *http.Request, admin store.User) {
+		l := langOf(r)
+		body := strings.ReplaceAll(r.FormValue("body"), "\r\n", "\n")
+		note := strings.TrimSpace(r.FormValue("note"))
+		base, _ := strconv.ParseInt(r.FormValue("base"), 10, 64)
+		// The audit line records the form; the document can hold webhook
+		// secrets, so it records the document's size, not the document.
+		r.PostForm.Set("body", "("+strconv.Itoa(len(body))+" bytes)")
+		refuse := func(msg string) {
+			noteAudit(r, "cluster settings", "", msg)
+			page(w, r, admin, settingsPageData{Body: body, Note: note, Error: msg})
+		}
+		if cur, err := s.ClusterSettings(r.Context()); err == nil && cur.Body == body {
+			refuse(l.T("settings.err.unchanged"))
+			return
+		}
+		if err := check([]byte(body)); err != nil {
+			refuse(l.T("settings.err.invalid") + " " + err.Error())
+			return
+		}
+		n, err := s.PutClusterSettings(r.Context(), base, body, admin.Email, note)
+		switch {
+		case errors.Is(err, store.ErrSettingsChanged):
+			refuse(l.T("settings.err.changed"))
+			return
+		case err != nil:
+			refuse(l.Err(err))
+			return
+		}
+		noteAudit(r, "cluster settings", "version "+strconv.FormatInt(n, 10), "")
+		page(w, r, admin, settingsPageData{Notice: l.T("settings.saved", n)})
+	}))
+
+	mux.HandleFunc("POST /admin/settings/rollback", adminsOnly(func(w http.ResponseWriter, r *http.Request, admin store.User) {
+		l := langOf(r)
+		n, _ := strconv.ParseInt(r.FormValue("version"), 10, 64)
+		old, err := s.ClusterSettingsAt(r.Context(), n)
+		if err != nil || old.Version == 0 {
+			page(w, r, admin, settingsPageData{Error: l.T("settings.err.no_version")})
+			return
+		}
+		cur, err := s.ClusterSettings(r.Context())
+		if err != nil {
+			page(w, r, admin, settingsPageData{Error: l.Err(err)})
+			return
+		}
+		if err := check([]byte(old.Body)); err != nil {
+			// A version kept before a change to server.yaml may no longer
+			// load beside it; refused here rather than breaking the cluster.
+			noteAudit(r, "cluster settings", "", err.Error())
+			page(w, r, admin, settingsPageData{Error: l.T("settings.err.invalid") + " " + err.Error()})
+			return
+		}
+		v, err := s.PutClusterSettings(r.Context(), cur.Version, old.Body, admin.Email, "back to version "+strconv.FormatInt(n, 10))
+		if err != nil {
+			page(w, r, admin, settingsPageData{Error: l.Err(err)})
+			return
+		}
+		noteAudit(r, "cluster settings", "version "+strconv.FormatInt(v, 10)+" from "+strconv.FormatInt(n, 10), "")
+		page(w, r, admin, settingsPageData{Notice: l.T("settings.rolled_back", n, v)})
+	}))
+}

internal/web/admin_settings_test.go

@@ -0,0 +1,143 @@
+package web
+
+import (
+	"context"
+	"errors"
+	"io"
+	"log/slog"
+	"net/http"
+	"net/url"
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+type fakeSettings struct{ versions []store.SettingsVersion }
+
+func (f *fakeSettings) ClusterSettings(context.Context) (store.SettingsVersion, error) {
+	if len(f.versions) == 0 {
+		return store.SettingsVersion{}, nil
+	}
+	return f.versions[len(f.versions)-1], nil
+}
+
+func (f *fakeSettings) ClusterSettingsAt(_ context.Context, v int64) (store.SettingsVersion, error) {
+	for _, s := range f.versions {
+		if s.Version == v {
+			return s, nil
+		}
+	}
+	return store.SettingsVersion{}, nil
+}
+
+func (f *fakeSettings) ClusterSettingsHistory(context.Context, int) ([]store.SettingsVersion, error) {
+	out := make([]store.SettingsVersion, len(f.versions))
+	for i, v := range f.versions {
+		out[len(f.versions)-1-i] = v
+	}
+	return out, nil
+}
+
+func (f *fakeSettings) PutClusterSettings(_ context.Context, base int64, body, by, note string) (int64, error) {
+	cur, _ := f.ClusterSettings(context.Background())
+	if base != cur.Version {
+		return 0, store.ErrSettingsChanged
+	}
+	n := cur.Version + 1
+	f.versions = append(f.versions, store.SettingsVersion{Version: n, Body: body, By: by, Note: note, At: time.Now()})
+	return n, nil
+}
+
+type auditSink struct{ events []store.AuditEvent }
+
+func (a *auditSink) WriteAudit(_ context.Context, e store.AuditEvent) error {
+	a.events = append(a.events, e)
+	return nil
+}
+
+func clusterSettingsPage(t *testing.T, user store.User) (*http.ServeMux, *fakeSettings, *auditSink) {
+	t.Helper()
+	s := &fakeSettings{versions: []store.SettingsVersion{{Version: 1, Body: "guard:\n  enabled: true\n", By: "init", At: time.Now()}}}
+	p := NewAccountPages(fakeAccounts{user: user}, nil, nil, "v0", slog.New(slog.NewTextHandler(io.Discard, nil)))
+	sink := &auditSink{}
+	p.SetAudit(sink)
+	mux := http.NewServeMux()
+	p.RegisterAdminSettings(mux, s, func(b []byte) error {
+		if strings.Contains(string(b), "bogus") {
+			return errors.New("unknown key bogus")
+		}
+		return nil
+	}, []string{"listen", "store"})
+	return mux, s, sink
+}
+
+func TestTheSettingsPageIsForAdministratorsOnly(t *testing.T) {
+	for _, u := range []store.User{
+		{ID: "u1", Email: "v@example.com", Role: store.RoleViewer},
+		{ID: "u2", Email: "o@example.com", Role: store.RoleOperator},
+		{ID: "u3", Email: "b@example.com", Role: store.RoleBilling},
+	} {
+		mux, _, _ := clusterSettingsPage(t, u)
+		if w := mirrorGet(t, mux, "/admin/settings"); w.Code != http.StatusNotFound {
+			t.Errorf("%s saw the settings: %d", u.Role, w.Code)
+		}
+	}
+	mux, _, _ := clusterSettingsPage(t, store.User{ID: "a", Email: "a@example.com", IsAdmin: true})
+	body := mirrorGet(t, mux, "/admin/settings").Body.String()
+	for _, want := range []string{"guard:\n  enabled: true", "Version 1, in force", "<code>listen</code>", `href="/admin/settings" aria-current="page"`} {
+		if !strings.Contains(body, want) {
+			t.Errorf("no %q in:\n%s", want, body)
+		}
+	}
+}
+
+func TestSettingsAreCheckedAppliedAndRolledBack(t *testing.T) {
+	mux, s, sink := clusterSettingsPage(t, store.User{ID: "a", Email: "a@example.com", IsAdmin: true})
+	post := func(form url.Values) string {
+		form.Set("csrf", "tok")
+		return mirrorPost(t, mux, "/admin/settings", form).Body.String()
+	}
+	// Refused: does not load. The edit stays in the editor; nothing kept.
+	body := post(url.Values{"base": {"1"}, "body": {"bogus: 1\n"}})
+	if !strings.Contains(body, "unknown key bogus") || !strings.Contains(body, "bogus: 1") || len(s.versions) != 1 {
+		t.Errorf("an invalid document: %d versions\n%s", len(s.versions), body)
+	}
+	// Applied, with its note and who.
+	body = post(url.Values{"base": {"1"}, "body": {"guard:\r\n  enabled: false\r\n"}, "note": {"maintenance"}})
+	if !strings.Contains(body, "Version 2 is in force") || len(s.versions) != 2 ||
+		s.versions[1].Body != "guard:\n  enabled: false\n" || s.versions[1].By != "a@example.com" || s.versions[1].Note != "maintenance" {
+		t.Errorf("applying: %+v\n%s", s.versions, body)
+	}
+	// Made from a version no longer the newest: refused, not overwritten.
+	body = post(url.Values{"base": {"1"}, "body": {"guard:\n  enabled: true\n  x: 1\n"}})
+	if !strings.Contains(body, "someone changed the settings") || len(s.versions) != 2 {
+		t.Errorf("a stale edit: %d versions\n%s", len(s.versions), body)
+	}
+	// An earlier version is shown, and put back as a new one.
+	if body := mirrorGet(t, mux, "/admin/settings?version=1").Body.String(); !strings.Contains(body, "Put version 1 back in force") {
+		t.Errorf("viewing version 1:\n%s", body)
+	}
+	w := mirrorPost(t, mux, "/admin/settings/rollback", url.Values{"csrf": {"tok"}, "version": {"1"}})
+	if len(s.versions) != 3 || s.versions[2].Body != s.versions[0].Body || !strings.Contains(w.Body.String(), "as version 3") {
+		t.Errorf("rollback: %+v\n%s", s.versions, w.Body.String())
+	}
+	// Without the CSRF token, nothing.
+	if w := mirrorPost(t, mux, "/admin/settings", url.Values{"base": {"3"}, "body": {"a: 1\n"}}); w.Code != http.StatusBadRequest || len(s.versions) != 3 {
+		t.Errorf("no token: %d", w.Code)
+	}
+	// Audited, without the document itself: it can hold secrets.
+	var saw bool
+	for _, e := range sink.events {
+		if strings.Contains(e.Detail, "enabled") {
+			t.Errorf("the document is in the audit log: %+v", e)
+		}
+		if e.Action == "POST /admin/settings" && strings.Contains(e.Detail, "version 2") {
+			saw = true
+		}
+	}
+	if !saw {
+		t.Errorf("no audit line for version 2: %+v", sink.events)
+	}
+}

internal/web/locales/en/nav.json

@@ -10,5 +10,6 @@
   "nav.chat": "Chat",
   "nav.account": "Account",
   "nav.sign_out": "Sign out",
-  "nav.docs": "Docs"
+  "nav.docs": "Docs",
+  "nav.settings": "Settings"
 }

internal/web/locales/en/settings.json

@@ -0,0 +1,28 @@
+{
+  "settings.title": "Settings",
+  "settings.sub": "The cluster's settings: everything that is not one machine's own, the same on every control plane, in force within seconds of a change. <code>llmgw ctl config edit</code> edits the same document.",
+  "settings.current": "Version %d, in force",
+  "settings.earlier": "Version %d",
+  "settings.earlier_sub": "An earlier version; version %d is in force. Rolling back keeps this one as a new version, so nothing is lost.",
+  "settings.rollback": "Put version %d back in force",
+  "settings.back_to_current": "back to the version in force",
+  "settings.local_keys": "Not here, because each machine's server.yaml keeps them:",
+  "settings.note": "Why",
+  "settings.note_hint": "kept with the version, for whoever reads the history",
+  "settings.save": "Check and apply",
+  "settings.save_hint": "Checked as a control plane would load it before it is kept; a change made since this page was opened is not overwritten.",
+  "settings.saved": "Version %d is in force; every control plane applies it within seconds.",
+  "settings.rolled_back": "Version %d's settings are back in force, as version %d.",
+  "settings.history": "History",
+  "settings.col.version": "Version",
+  "settings.col.at": "When",
+  "settings.col.by": "By",
+  "settings.col.note": "Why",
+  "settings.in_force": "in force",
+  "settings.view": "view",
+  "settings.no_history": "No versions yet.",
+  "settings.err.unchanged": "Nothing was changed.",
+  "settings.err.invalid": "Not applied: a control plane could not load it.",
+  "settings.err.changed": "Not applied: someone changed the settings since this page was opened. Your edit is below; compare it with the version now in force and apply it again.",
+  "settings.err.no_version": "There is no such version."
+}

internal/web/locales/tr/nav.json

@@ -10,5 +10,6 @@
   "nav.chat": "Sohbet",
   "nav.account": "Hesap",
   "nav.sign_out": "Çıkış yap",
-  "nav.docs": "Belgeler"
+  "nav.docs": "Belgeler",
+  "nav.settings": "Ayarlar"
 }

internal/web/locales/tr/settings.json

@@ -0,0 +1,28 @@
+{
+  "settings.title": "Ayarlar",
+  "settings.sub": "Kümenin ayarları: tek bir makineye ait olmayan her şey; her kontrol düzleminde aynı, değiştikten saniyeler sonra yürürlükte. <code>llmgw ctl config edit</code> aynı belgeyi düzenler.",
+  "settings.current": "Sürüm %d, yürürlükte",
+  "settings.earlier": "Sürüm %d",
+  "settings.earlier_sub": "Önceki bir sürüm; yürürlükte olan sürüm %d. Geri almak bunu yeni bir sürüm olarak kaydeder, hiçbir şey kaybolmaz.",
+  "settings.rollback": "%d numaralı sürümü yeniden yürürlüğe koy",
+  "settings.back_to_current": "yürürlükteki sürüme dön",
+  "settings.local_keys": "Burada değil, çünkü her makinenin kendi server.yaml dosyası tutar:",
+  "settings.note": "Neden",
+  "settings.note_hint": "sürümle birlikte saklanır, geçmişi okuyacak kişi için",
+  "settings.save": "Denetle ve uygula",
+  "settings.save_hint": "Saklanmadan önce bir kontrol düzleminin yükleyeceği gibi denetlenir; bu sayfa açıldıktan sonra yapılmış bir değişikliğin üzerine yazılmaz.",
+  "settings.saved": "%d numaralı sürüm yürürlükte; her kontrol düzlemi saniyeler içinde uygular.",
+  "settings.rolled_back": "%d numaralı sürümün ayarları yeniden yürürlükte; yeni sürüm numarası %d.",
+  "settings.history": "Geçmiş",
+  "settings.col.version": "Sürüm",
+  "settings.col.at": "Ne zaman",
+  "settings.col.by": "Kim",
+  "settings.col.note": "Neden",
+  "settings.in_force": "yürürlükte",
+  "settings.view": "görüntüle",
+  "settings.no_history": "Henüz sürüm yok.",
+  "settings.err.unchanged": "Hiçbir şey değiştirilmedi.",
+  "settings.err.invalid": "Uygulanmadı: bir kontrol düzlemi bunu yükleyemezdi.",
+  "settings.err.changed": "Uygulanmadı: bu sayfa açıldıktan sonra biri ayarları değiştirdi. Düzenlemeniz aşağıda; yürürlükteki sürümle karşılaştırıp yeniden uygulayın.",
+  "settings.err.no_version": "Böyle bir sürüm yok."
+}

internal/web/nav.go

@@ -26,6 +26,9 @@ type Nav struct {
 	Mirror, Quotas bool
 	// Guard is whether the guardrails page was mounted.
 	Guard bool
+	// Settings offers the cluster's settings: mounted, and to an
+	// administrator only.
+	Settings bool
 	// CSRF signs the sign-out form.
 	CSRF string
 	// Org offers the organization page: to an administrator, and to anyone
@@ -38,13 +41,14 @@ type navKey struct{}
 // navFor builds the menu for a signed-in person.
 func (p *AccountPages) navFor(r *http.Request, user store.User, current string) Nav {
 	n := Nav{
-		Current: current,
-		Admin:   user.Staff(),
-		Role:    user.StaffRole(),
-		Mirror:  p.hasMirror,
-		Quotas:  p.hasQuotas,
-		Guard:   p.hasGuard,
-		CSRF:    csrfFor(r),
+		Current:  current,
+		Admin:    user.Staff(),
+		Role:     user.StaffRole(),
+		Mirror:   p.hasMirror,
+		Quotas:   p.hasQuotas,
+		Guard:    p.hasGuard,
+		Settings: p.hasSettings && user.IsAdmin,
+		CSRF:     csrfFor(r),
 	}
 	// Only when there is an organization to show: a link that leads to "there
 	// are none" on every page is noise.

internal/web/static/base.css

@@ -905,3 +905,13 @@ a.sel-link.small { font-size: .78rem; color: var(--muted); margin-left: .4rem; }
 /* The language switcher at the foot of every page. */
 .langs a { color: var(--muted); }
 .langs b { font-weight: 600; }
+
+/* The cluster's settings, edited as the YAML document they are. */
+.settings-form textarea, pre.settings-body {
+  width: 100%; box-sizing: border-box; font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace;
+  font-size: .85rem; line-height: 1.4; padding: .6rem .7rem; border: 1px solid var(--line);
+  border-radius: 6px; background: var(--card); color: var(--fg); tab-size: 2; overflow-x: auto;
+}
+.settings-form label { display: block; margin: .8rem 0; }
+.settings-form input[type=text] { width: 100%; box-sizing: border-box; }
+.sr-only { position: absolute; width: 1px; height: 1px; overflow: hidden; clip: rect(0 0 0 0); white-space: nowrap; }

internal/web/templates/admin_settings.html

@@ -0,0 +1,50 @@
+{{define "admin_settings"}}<!doctype html>
+<html lang="{{lang}}"><head>
+<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
+<title>{{t "settings.title"}} · llmgw</title><link rel="stylesheet" href="{{stylesheet}}">
+</head><body><div class="wrap">
+<header>
+  <h1>{{t "settings.title"}}</h1>
+  <p class="sub">{{th "settings.sub"}}</p>
+  {{template "nav" .Nav}}
+</header>
+{{if .Error}}<p class="error" role="alert">{{.Error}}</p>{{end}}
+{{if .Notice}}<p class="ok" role="status">{{.Notice}}</p>{{end}}
+
+{{if .Earlier}}
+<h2>{{t "settings.earlier" .Shown.Version}}</h2>
+<p class="sub">{{t "settings.earlier_sub" .Current.Version}}</p>
+<pre class="settings-body">{{.Body}}</pre>
+<form method="post" action="/admin/settings/rollback">
+  <input type="hidden" name="csrf" value="{{.CSRF}}">
+  <input type="hidden" name="version" value="{{.Shown.Version}}">
+  <button type="submit">{{t "settings.rollback" .Shown.Version}}</button>
+  <a href="/admin/settings">{{t "settings.back_to_current"}}</a>
+</form>
+{{else}}
+<h2>{{t "settings.current" .Current.Version}}</h2>
+{{if .LocalKeys}}<p class="sub">{{t "settings.local_keys"}} {{range $i, $k := .LocalKeys}}{{if $i}}, {{end}}<code>{{$k}}</code>{{end}}.</p>{{end}}
+<form method="post" action="/admin/settings" class="settings-form">
+  <input type="hidden" name="csrf" value="{{.CSRF}}">
+  <input type="hidden" name="base" value="{{.Current.Version}}">
+  <label for="settings-body" class="sr-only">{{t "settings.title"}}</label>
+  <textarea id="settings-body" name="body" rows="30" spellcheck="false" class="mono">{{.Body}}</textarea>
+  <label>{{t "settings.note"}} <span class="opt">{{t "settings.note_hint"}}</span>
+    <input type="text" name="note" value="{{.Note}}" maxlength="200"></label>
+  <button type="submit">{{t "settings.save"}}</button>
+  <p class="sub">{{t "settings.save_hint"}}</p>
+</form>
+{{end}}
+
+<h2>{{t "settings.history"}}</h2>
+{{if .History}}
+<table class="rows">
+<tr><th class="num">{{t "settings.col.version"}}</th><th>{{t "settings.col.at"}}</th><th>{{t "settings.col.by"}}</th><th>{{t "settings.col.note"}}</th><th></th></tr>
+{{range .History}}<tr{{if .Selected}} aria-current="true"{{end}}>
+  <td class="num">{{.Version}}</td><td>{{.At}}</td><td>{{.By}}</td><td>{{.Note}}</td>
+  <td>{{if .InForce}}<strong>{{t "settings.in_force"}}</strong>{{else}}<a href="/admin/settings?version={{.Version}}">{{t "settings.view"}}</a>{{end}}</td>
+</tr>{{end}}
+</table>
+{{else}}<p class="sub">{{t "settings.no_history"}}</p>{{end}}
+<footer>llmgw {{.Version}}</footer>
+</div></body></html>{{end}}

internal/web/templates/nav.html

@@ -20,6 +20,9 @@
   {{- if .Guard}}
   <a href="/admin/guard"{{if eq .Current "guard"}} aria-current="page"{{end}}>{{t "nav.guard"}}</a>
   {{- end}}
+  {{- if .Settings}}
+  <a href="/admin/settings"{{if eq .Current "settings"}} aria-current="page"{{end}}>{{t "nav.settings"}}</a>
+  {{- end}}
   {{- /* An organization is a panel of the people page for an administrator;
        /org is the managers' view, and leads an administrator there. */}}
   <span class="menu-gap" aria-hidden="true"></span>

terraform-provider-llmgw/README.md

@@ -15,13 +15,17 @@ API is reached and what it asks of a request.
 | `llmgw_model` | A model added to the catalog at runtime, as `llmgw ctl model add` adds one. Scaling, class, owner and adapters change in place; any other change to the definition replaces the model, which stops its replicas while it is added again. |
 | `llmgw_quota` | A capacity quota on an organization, team or project: a guarantee, a limit, or both, in GPUs, CPU cores and memory. |
 | `llmgw_guard_attachment` | A guardrail policy attached to an organization, team, project, key or person. The policies are defined in `server.yaml`'s `guard:` section; this says where they apply. |
+| `llmgw_org`, `llmgw_team`, `llmgw_project` | The tenancy tree. A team names its organization and a project its team, by path or ID; a new name renames in place, a new parent moves it with everything under it. Only something empty is removed, and a project that has ever had a key is never empty (the revoked key stays for usage history): rename it rather than destroy it. |
+| `llmgw_key` | An API key in a project, for a person in a project they belong to, or a person's own. The key is in `secret` (sensitive) from the apply that made it; the gateway keeps only a hash. Any change makes a new key, and one revoked or expired outside Terraform is made again. Destroying it revokes it. |
+| `llmgw_membership` | A person in an organization, team or project, as member or manager. People are not made here: they sign up, are invited, or come from the identity provider. Taking a membership away revokes their keys in projects they no longer reach. |
 | `data.llmgw_models` | Every model in the catalog, with its replicas now. |
 | `data.llmgw_nodes` | Every node, with its cards and state now. |
 
-Not here, because the admin API does not serve them: organizations, teams,
-projects, people and API keys. Those live in the store and are managed with
-`llmgw org`, `llmgw users` and `llmgw keys` on the control plane. A quota or
-an attachment names a team that already exists, by its path or its ID.
+Not here: people's accounts (sign-up, invitation, OIDC/SCIM, and `llmgw
+users` to approve, disable or delete), and key lifetime and spending limits
+(`llmgw keys ttl`, `llmgw keys limits`). A quota or an attachment names a
+part of the tree by its path or its ID; refer to an `llmgw_team`'s `id` to
+have Terraform make the team first.
 
 Models declared in the control plane's `server.yaml` are not managed here
 either: the cluster refuses to remove or change them through the API, since
@@ -107,8 +111,13 @@ with `version = "0.1.0"` beside `source` in `required_providers`.
   found again by the ID, so renaming the team in the store does not lose it.
 - **Import** works for quotas, by `team:ID` as `GET /api/v1/quotas` lists it
   (the `scope` is then that ID; write it so in the configuration, or the next
-  plan replaces the quota with itself), and for attachments, by
-  `scope:SCOPE-ID:policy` as their `id` shows. Models cannot be imported yet.
+  plan replaces the quota with itself), for attachments, by
+  `scope:SCOPE-ID:policy` as their `id` shows, for organizations, teams,
+  projects and keys by their IDs (an imported key has no `secret`), and for
+  memberships by `USER-ID/scope:SCOPE-ID`. Models cannot be imported yet.
+- **A project's `path` follows a move of its team at the next refresh**: the
+  project did not change, so the apply that moved the team does not rewrite
+  it.
 
 ## Developing it
 

terraform-provider-llmgw/examples/main.tf

@@ -66,6 +66,33 @@ resource "llmgw_model" "llama_70b" {
   fallbacks = [llmgw_model.qwen_small.name]
 }
 
+# A team and a project in Acme, a key for its CI, and a person in the team.
+resource "llmgw_team" "search" {
+  name = "search"
+  org  = "Acme"
+}
+
+resource "llmgw_project" "ranking" {
+  name = "ranking"
+  team = llmgw_team.search.id
+}
+
+resource "llmgw_key" "ranking_ci" {
+  name       = "ci"
+  project    = llmgw_project.ranking.id
+  expires_in = "2160h"
+}
+
+resource "llmgw_membership" "ayse" {
+  user   = "ayse@example.com"
+  target = "team:${llmgw_team.search.id}"
+}
+
+output "ranking_ci_key" {
+  value     = llmgw_key.ranking_ci.secret
+  sensitive = true
+}
+
 # The ml team may hold at most eight GPUs, and is always given two.
 resource "llmgw_quota" "ml" {
   scope     = "team:Acme/ml"

terraform-provider-llmgw/internal/client/client.go

@@ -40,6 +40,22 @@ var Endpoints = []string{
 	"GET /api/v1/guard",
 	"POST /api/v1/guard/attachments",
 	"DELETE /api/v1/guard/attachments",
+	"GET /api/v1/orgs",
+	"POST /api/v1/orgs",
+	"PATCH /api/v1/orgs/{id}",
+	"DELETE /api/v1/orgs/{id}",
+	"POST /api/v1/teams",
+	"PATCH /api/v1/teams/{id}",
+	"DELETE /api/v1/teams/{id}",
+	"POST /api/v1/projects",
+	"PATCH /api/v1/projects/{id}",
+	"DELETE /api/v1/projects/{id}",
+	"POST /api/v1/keys",
+	"GET /api/v1/keys/{id}",
+	"DELETE /api/v1/keys/{id}",
+	"GET /api/v1/users/{id}",
+	"POST /api/v1/users/{id}/memberships",
+	"DELETE /api/v1/users/{id}/memberships",
 }
 
 // Client is a connection to one control plane's admin API.
@@ -436,3 +452,166 @@ func (c *Client) Attach(ctx context.Context, target, policy string) (Attachment,
 func (c *Client) Detach(ctx context.Context, target, policy string) error {
 	return c.do(ctx, "DELETE /api/v1/guard/attachments", nil, GuardAttachmentRequest{Target: target, Policy: policy}, nil)
 }
+
+// --- the tenancy tree, keys and memberships ---
+
+// OrgView is an organization with its teams.
+type OrgView struct {
+	ID    string     `json:"id"`
+	Name  string     `json:"name"`
+	Teams []TeamView `json:"teams"`
+}
+
+// TeamView is a team with its projects.
+type TeamView struct {
+	ID       string        `json:"id"`
+	Name     string        `json:"name"`
+	Org      string        `json:"org"`
+	Projects []ProjectView `json:"projects"`
+}
+
+// ProjectView is a project.
+type ProjectView struct {
+	ID   string `json:"id"`
+	Name string `json:"name"`
+	Team string `json:"team"`
+	Path string `json:"path"`
+}
+
+// TenancyCreateRequest adds an organization, a team or a project.
+type TenancyCreateRequest struct {
+	Name string `json:"name"`
+	In   string `json:"in,omitempty"`
+}
+
+// TenancyChangeRequest renames or moves one.
+type TenancyChangeRequest struct {
+	Name string `json:"name,omitempty"`
+	In   string `json:"in,omitempty"`
+}
+
+// TenancyItem is one added or changed, by ID.
+type TenancyItem struct {
+	ID   string `json:"id"`
+	Name string `json:"name,omitempty"`
+	In   string `json:"in,omitempty"`
+}
+
+// Levels of the tree, as the API's paths name them.
+const (
+	LevelOrg     = "orgs"
+	LevelTeam    = "teams"
+	LevelProject = "projects"
+)
+
+// Tree reads every organization, team and project.
+func (c *Client) Tree(ctx context.Context) ([]OrgView, error) {
+	var out []OrgView
+	return out, c.do(ctx, "GET /api/v1/orgs", nil, nil, &out)
+}
+
+// Create adds an organization, team or project.
+func (c *Client) Create(ctx context.Context, level string, in TenancyCreateRequest) (TenancyItem, error) {
+	var out TenancyItem
+	return out, c.do(ctx, "POST /api/v1/"+level, nil, in, &out)
+}
+
+// Change renames an organization, team or project, or moves a team or a
+// project.
+func (c *Client) Change(ctx context.Context, level, id string, in TenancyChangeRequest) (TenancyItem, error) {
+	var out TenancyItem
+	return out, c.do(ctx, "PATCH /api/v1/"+level+"/{id}", []string{id}, in, &out)
+}
+
+// Remove removes an organization, team or project with nothing in it.
+func (c *Client) Remove(ctx context.Context, level, id string) error {
+	return c.do(ctx, "DELETE /api/v1/"+level+"/{id}", []string{id}, nil, nil)
+}
+
+// KeyView is a key, never the key itself.
+type KeyView struct {
+	ID             string    `json:"id"`
+	Name           string    `json:"name"`
+	Prefix         string    `json:"prefix"`
+	Project        string    `json:"project,omitempty"`
+	User           string    `json:"user,omitempty"`
+	Band           string    `json:"band,omitempty"`
+	RoutingProfile string    `json:"routing_profile,omitempty"`
+	State          string    `json:"state"`
+	ExpiresAt      time.Time `json:"expires_at,omitzero"`
+}
+
+// KeyCreateRequest issues a key.
+type KeyCreateRequest struct {
+	Name           string `json:"name"`
+	Project        string `json:"project,omitempty"`
+	User           string `json:"user,omitempty"`
+	Band           string `json:"band,omitempty"`
+	RoutingProfile string `json:"routing_profile,omitempty"`
+	ExpiresIn      string `json:"expires_in,omitempty"`
+}
+
+// KeyCreated is a new key with the key itself, given this once.
+type KeyCreated struct {
+	ID        string    `json:"id"`
+	Prefix    string    `json:"prefix"`
+	Project   string    `json:"project,omitempty"`
+	User      string    `json:"user,omitempty"`
+	State     string    `json:"state"`
+	ExpiresAt time.Time `json:"expires_at,omitzero"`
+	Secret    string    `json:"secret"`
+}
+
+// CreateKey issues a key.
+func (c *Client) CreateKey(ctx context.Context, in KeyCreateRequest) (KeyCreated, error) {
+	var out KeyCreated
+	return out, c.do(ctx, "POST /api/v1/keys", nil, in, &out)
+}
+
+// Key reads a key.
+func (c *Client) Key(ctx context.Context, id string) (KeyView, error) {
+	var out KeyView
+	return out, c.do(ctx, "GET /api/v1/keys/{id}", []string{id}, nil, &out)
+}
+
+// RevokeKey stops a key working.
+func (c *Client) RevokeKey(ctx context.Context, id string) error {
+	return c.do(ctx, "DELETE /api/v1/keys/{id}", []string{id}, nil, nil)
+}
+
+// UserView is a person with where they are a member.
+type UserView struct {
+	ID          string           `json:"id"`
+	Email       string           `json:"email"`
+	Memberships []MembershipView `json:"memberships"`
+}
+
+// MembershipView is one place a person belongs, target as scope:ID.
+type MembershipView struct {
+	Target string `json:"target"`
+	Role   string `json:"role"`
+}
+
+// MembershipRequest names where a person goes or leaves.
+type MembershipRequest struct {
+	Target string `json:"target"`
+	Role   string `json:"role,omitempty"`
+}
+
+// User reads a person, by ID or email.
+func (c *Client) User(ctx context.Context, id string) (UserView, error) {
+	var out UserView
+	return out, c.do(ctx, "GET /api/v1/users/{id}", []string{id}, nil, &out)
+}
+
+// Join puts a person in an organization, team or project, or changes their
+// role there.
+func (c *Client) Join(ctx context.Context, user string, in MembershipRequest) (UserView, error) {
+	var out UserView
+	return out, c.do(ctx, "POST /api/v1/users/{id}/memberships", []string{user}, in, &out)
+}
+
+// Leave takes a person out.
+func (c *Client) Leave(ctx context.Context, user, target string) error {
+	return c.do(ctx, "DELETE /api/v1/users/{id}/memberships", []string{user}, MembershipRequest{Target: target}, nil)
+}