Backlog F5: Qwen3-Embedding-4B measured, on llama.cpp and vLLM, and against bge-m3

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Docs: pictures from ComfyUI
2026-10-09 21:36:23 +02:00 · 2026-10-09 21:26:31 +02:00 · 2026-10-09 21:22:02 +02:00 · 2026-10-09 21:15:50 +02:00 · 2026-10-09 18:03:00 +02:00 · 2026-10-09 15:44:47 +02:00
119 changed files with 12251 additions and 453 deletions
@@ -569,9 +569,12 @@ portal is not done — the page shows; the console and ctl change.
    client tests against a fake admin API and the OpenAPI document;
    acceptance tests with Terraform and OpenTofu against the fake; one
    apply, empty plan, in-place change and destroy against a real `llmgw
-    server` run by hand. Left: organizations, teams, projects, people and
-    keys are not in the admin API (they are `llmgw org`/`users`/`keys`
-    against the store), so not in the provider; a model's definition beyond
+    server` run by hand. Organizations, teams, projects, keys and
+    memberships are in the admin API since 2026-10-08 (`/api/v1/orgs`,
+    `teams`, `projects`, `keys`, `users/{id}/memberships`) and in the
+    provider (`llmgw_org`, `llmgw_team`, `llmgw_project`, `llmgw_key`,
+    `llmgw_membership`); accounts are still made by sign-up, invitation or
+    OIDC/SCIM. Left: a model's definition beyond
    scaling, class, owner and adapters is changed by replacing it (the API
    has no edit, and `resources` is not used yet); `canary`, `slo`,
    `scaling.schedule` and `prewarm`, `guard` role and `origin` are not in
@@ -793,10 +796,19 @@ leader 30s later with /health answering throughout; rollback did the same.
  facility state from the store (it kept what it read at start).
 - Retires `appendGuardIfMissing` (server init): the import does it.

-### E2. Generated files from settings (L) — firewall on/off done 2026-10-06
+### E2. Generated files from settings (L) — firewall on/off done 2026-10-06, every machine 2026-10-09
 `firewall.enabled` in the cluster's settings reaches nodes over the agent
-stream and control planes through llmgw-firewall-sync.path. Left: the mirror
-machine and store members, the allowed set from membership, the other files.
+stream and control planes through llmgw-firewall-sync.path. Since
+2026-10-09 the mirror machine too (the leader PUTs /v1/firewall, the mirror
+writes `.firewall.json`, the same path unit applies it) and store members
+that run no llmgw (the leader keeps it in `cluster_state`, a root timer
+reads the member's own copy with psql); `firewall_networks` reach all of
+them, and the mirror lets in the network of every node and control plane
+the leader knows. Tried on the PoC: off and on, and a network added, on
+control planes, mirror, store member and nodes. Left: the allowed set
+replaced by membership rather than added to (a network taken out of the
+settings stays let in), the other files (the mirror's unit flags,
+keepalived, etcd and Patroni configuration).

 Every machine runs the agent (root) with roles — node, control plane, mirror,
 store member — and the agent renders and applies what was written once by
@@ -807,14 +819,34 @@ Patroni configuration. The firewall's allowed set is the cluster's membership
 removed changes every machine's rules. Retires `firewall.Admit` and
 `firewall_networks` in server.yaml (imported into the store).

-### E3. agent.yaml down to bootstrap (M)
+### E3. agent.yaml down to bootstrap (M) — done 2026-10-08 (branch `settings-e`)
 agent.yaml keeps the control planes' addresses, its certificate and its id.
 Labels, fabrics, engines, heartbeat and supervisor settings move to the
 store per node and reach the agent over its stream; a change to the file is
 noticed and the agent restarts itself.
+Built as a section of the cluster's settings rather than a store of its own,
+so history, audit, `ctl config`, the console and the portal page cover it:
+`agents.default` and `agents.nodes.<id>` (config.AgentSettings; bootstrap
+keys node_id/server(s)/hostname/mgmt_addr/tls refused; each node's merge
+loaded as an agent would, so a bad fabric is refused on save). A live key:
+the control plane sends ControlMessage.AgentSettings on register and, a
+second apart per node, on change; the agent compares with what it started on,
+checks it loads beside its agent.yaml (else logs and keeps running), keeps it
+in /var/lib/llmgw-agent/agent-settings.yaml and restarts (engines adopted).
+At start: agent.yaml < kept settings < LLMGW_ env; kept settings that no
+longer load are set aside, never a reason not to start. agent.yaml is
+watched every 10 s and restarted on when it changes and loads. Left: file
+keys are not imported into the settings (they keep working under them);
+`node add` still writes a full agent.yaml; labels have two homes now
+(`ctl label`, which wins, and agents.*.labels).

-### E4. Web settings page (S)
+### E4. Web settings page (S) — done 2026-10-08 (branch `settings-e`)
 The same document, edited on /admin/settings, for whoever has no terminal.
+Built: administrators only (staff roles get 404: webhook secrets), checked by
+the same function as PUT /api/v1/settings, made from a base version or
+refused on a newer one (the edit kept in the editor), history with view and
+roll back as a new version, audit with the size not the body, English and
+Turkish, a Settings link in the menu.

 ---

@@ -843,6 +875,12 @@ Built (`internal/store/knowledge.go`, migration 0035 PostgreSQL only):
  partial HNSW index per collection on its organization's partition
  (`embedding::vector(dims)`, `WHERE collection = '<id>'`). EXPLAIN on
  20k rows: partition pruned, that index used, policy applied.
+  Later (scale, 2026-10-08): a partial GIN per collection too
+  (`kb_tsv_<id>`, backfilled at open); `vec_type` per collection, `halfvec`
+  on pgvector ≥ 0.7 (up to 4000 dims); `SET LOCAL hnsw.ef_search` to
+  candidates(k) clamped 40..1000 (EXPLAIN: 40 rows without it, all with
+  it); the text arm takes candidates by `ts_rank` in a subquery and scores
+  only those.
 - RLS FORCEd on `kb_chunks` against `llmgw.kb_org`/`llmgw.kb_collections`
  (set with set_config per transaction), one role — the table owner — not
  a separate one. Chunks are deleted by the foreign-key cascade from their
@@ -985,7 +1023,7 @@ a request without the header, `none` turns it off, an explicit header with
 k=2 gives two sources and no more, llmgw-kb lists owners by path. Left: returned passages
 are not audited per request (the header and the kept request record them).

-### F4. Adapter training (L, after F1–F3)
+### F4. Adapter training (L, after F1–F3) — done 2026-10-08 (branch `adapters-f4`)
 A run-to-completion job placed by the control plane like a replica and
 supervised by the agent like an engine, in a Python environment the agent
 installs (TRL + PEFT, QLoRA). Data: Q/A pairs generated from a collection by
@@ -993,9 +1031,76 @@ a cluster model through `/v1/batches`, reviewed by people. Output to the
 mirror, into the model's `adapters`, through canary and a quality suite.
 Runs when the scaling schedule frees a GPU; 8–14B dense bases on one card.
 Never train on a collection some users of the adapter may not read.
+Tried 2026-10-08 on the PoC, by hand: Qwen2.5-0.5B-Instruct (fp16) with a
+LoRA (r=16, every linear layer, 8.8M trainable), trained beside the 35B in
+cuda01's last 2.3 GB (peak 2158 MiB on the card, 64 s for 138 steps, PEFT in a
+venv over the vLLM engine's torch). Data: 90 Turkish Q/A pairs a cluster
+model wrote from the nine Demo/politikalar documents, through the gateway.
+On nine questions it had not seen, phrased otherwise: base 0/9, 6 epochs
+4/9, 4 epochs at a lower rate 2/9. Wrong answers mixed facts across
+documents (the meal card's amount, the office day). The same nine through
+retrieval: 9/9 with sources. So F4 is for format, vocabulary and tone, and
+facts come from F1–F3; a bigger base on a free card would do better but not
+change that. The training itself fits a small slot on a shared card, which
+the job design can rely on for small bases.
+
+Built: `ctl kb dataset` (POST /api/v1/kb/collections/{id}/dataset, NDJSON
+with each pair's collection); train_jobs/train_artifacts (migrations 0037,
+0038 owner); internal/control/train (queue, one job per node, artifacts in
+1 MiB pieces, a node that goes fails its job); internal/agent/train (the
+newest vLLM's python3 + `uv pip install --no-deps --target` peft 0.21.2 and
+accelerate 1.15.0; llama.cpp's conversion code for the installed release,
+untarred from its source; the card with the most free memory, if enough;
+runs as llmgw-engine); `ctl train start|list|show --follow|cancel|rm|
+download|publish`; the mirror's PUT /v1/models/{model}/{version}/{file};
+publish uploads the GGUF (llama.cpp) or PEFT files (vLLM) and sets the
+adapter after checkAdapters. Owners: catalog.Adapter.Owner, enforced in the
+router (404, hidden from /v1/models); a job's owner defaults to its single
+collection's owner, must be able to read every collection the pairs came
+from, and pairs from a knowledge base are never public. Tried on the PoC:
+dataset of 90 pairs in 89 s; a job trained beside the 35B (peak 2214 MiB,
+138 steps in about a minute) and sent back PEFT and GGUF.
+Left: a job dies with its agent's stream (no resume); no console screen for
+training yet; quality gating of a published adapter is by hand (ctl eval).

 ---

+### F5. Large imports (M) — measured 2026-10-09, open
+The metallurgy corpus (2,314 files, 108,779 chunks, bge-m3 on the one
+GPU) went in at about 25 chunks/s, the same over a phone hotspot and the
+LAN. The engine's slots were the limit: llama.cpp `--parallel 4` embeds
+four texts per pass, while llmgw-kb sends 32 a request from 6 workers. The
+usage rows show 4.6 s a request on average, 1.8 s of it in the gateway's
+queue and 0.04 s in the guard. The GPU sat at 18–50% and the store machine
+at a third of its CPUs.
+- An embedding model's profile with many slots by default (`--parallel`
+  32–64, `--ctx-size` to match, `--batch-size`/`--ubatch-size` at least the
+  longest chunk), or `ctl model serve` choosing that for a model with
+  `capabilities.embeddings`. With the default 512, one 516-token chunk failed.
+- Embedding on the gateway's side of a document PUT, so the client sends
+  text only: today an embeddings answer is ~21 KB of JSON a chunk, and
+  the text and vector go up again (~11 KB).
+- The store next: HNSW inserts took half of one of pg01's two cores; at a
+  million chunks the vectors alone outgrow its 4 GB. halfvec (pgvector ≥
+  0.7) and a larger store machine.
+- Done: a batch of embeddings checked text by text against the context
+  (branch `embeddings-batch`); it was refused as one over-long prompt.
+- Done: `query_prefix` on an embedding model, before questions only.
+- Qwen3-Embedding-4B on the RTX 6000 (2026-10-09): llama.cpp Q8 1,575
+  tok/s as first set up (32k cache, 16 slots), 2,700 with a 16k cache,
+  FlashAttention and ubatch 1–2k; f16 no better. vLLM fp16 (`--runner
+  pooling`) 6,200 direct, 4,700–4,900 through the gateway in the import,
+  and its vectors match llama.cpp's (cosine 0.9995). The 103,579 chunks
+  took 3 h. Wanted: an embedding model's GPU profile on vLLM and CPU
+  profile on llama.cpp, which needs a source per profile (the catalog has
+  one per model; the import swapped the entry by hand), and an admission
+  queue limit that suits bulk imports (30 s turned some away).
+- Retrieval on 40 questions written from the corpus, document in the top
+  5: bge-m3 31/40 (Turkish questions 13/21), Qwen3-Embedding-4B 37/40
+  (18/21), with its query prefix 36/40 and first place 31/40 instead of
+  28 (MRR 0.738 → 0.789 → 0.838). Next: translating the question with the
+  chat model, and a reranker.
+
 ## D. Facts worth knowing before starting

 - Admin API: `127.0.0.1:9090`, no auth, `ctlClient` in `internal/cli/ctl.go`.
@@ -18,12 +18,110 @@ is only released once its **exit criterion** is met, not when its code lands.
 | v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released |
 | v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released |
 | v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
-| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released |
+| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine) |
 | v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |

 Until v1.0.0 the minor version is the breaking one: configuration formats and Go
 APIs change between minors without a deprecation period.

+## Unreleased
+
+## v0.14.1 — 2026-10-09
+
+Adapters trained on the nodes from a knowledge base and kept private to
+their owners; the cluster's and the agents' settings edited in one place;
+the tenancy tree and keys over the admin API and in Terraform; knowledge
+bases that hold up with many documents; the firewall setting on every
+machine.
+
+**Upgrading.**
+- **Nodes** need this release to follow `agents.*` in the cluster's
+  settings; until a node is upgraded its agent.yaml is what it runs on.
+- **A separate mirror machine and store members that run nothing else**
+  follow the firewall setting once what reads it is installed: run `sudo
+  llmgw mirror init` on the mirror again with the flags it was set up with,
+  and `llmgw server init --patroni` again for the store's members. A control
+  plane set up before v0.12.0 needs `sudo llmgw firewall sync --install`.
+- **The store's PostgreSQL memory** applies after each member is restarted,
+  one at a time, the leader last; until then it runs as before.
+- The store gains tables and columns; nothing to do. Existing knowledge-base
+  collections get their full-text index when a control plane starts, and
+  keep `vector` storage; only new collections use `halfvec`, and only on
+  pgvector 0.7 or later.
+- Cached knowledge-base passages in vLLM are made again once, under the new
+  `cache_salt`.
+
+### Added
+
+- **The cluster's settings on the portal** (BACKLOG E4): `/admin/settings`
+  for administrators. It edits the document `llmgw ctl config edit` opens:
+  checked as a control plane would load it, refused when made from a
+  version since replaced, with the history to view and roll back. Changes
+  are audited by size, never by contents.
+- **The nodes' agent settings in the cluster's settings** (BACKLOG E3):
+  `agents.default` and `agents.nodes.<id>` hold everything agent.yaml can
+  say but the agent's id, control planes, address and certificate. They are
+  sent to the agents as they change; an agent starts again on new settings,
+  or on an edited agent.yaml, with its engines serving throughout. Settings
+  that would not load are refused on save, and set aside on a node rather
+  than keep its agent from starting. Nodes need this release to follow
+  them.
+- **LoRA adapters trained on the nodes** (BACKLOG F4): `llmgw ctl kb dataset`
+  writes question/answer pairs from a collection with a model of the
+  cluster. `llmgw ctl train start` trains an adapter on a node with its
+  installed vLLM's torch and a small overlay, on a card with enough memory
+  free (beside an engine, if it leaves that much), converting to GGUF for
+  llama.cpp. `llmgw ctl train publish` keeps it on the mirror (which now
+  accepts uploads) and makes it the model's adapter.
+- **Adapters with owners.** An adapter's `owner` (org, team, project, user
+  or key) is the only part of the tree that sees it; to anyone else it is a
+  model that does not exist. An adapter trained on a knowledge base belongs
+  to a part of the tree that may read it, and is never made public.
+- The knowledge bases' vLLM `cache_salt` follows the collections searched,
+  not the organization, so two teams of one organization no longer share
+  cached passages.
+- **Organizations, teams, projects, API keys and memberships in the admin
+  API**: `/api/v1/orgs` (the tree), `POST /api/v1/teams` and `projects`,
+  `PATCH` to rename or move, `DELETE` for what is empty; `/api/v1/keys` to
+  issue (the key is in the answer, once), list and revoke; `/api/v1/users`
+  and `/api/v1/users/{id}/memberships`. They were only `llmgw org`, `keys`
+  and `users` on a control plane before. The Terraform provider manages them
+  as `llmgw_org`, `llmgw_team`, `llmgw_project`, `llmgw_key` (the key in a
+  sensitive `secret`) and `llmgw_membership`.
+- **The firewall setting reaches every machine.** A separate mirror and the
+  store's members follow `firewall.enabled` and `firewall_networks` as nodes
+  and control planes did: the leading control plane tells the mirror over
+  its API, and keeps the state in the store for members that run no llmgw,
+  which a timer on each reads. The mirror also lets in the network of every
+  node the leader knows. `mirror init` and `server init --patroni` install
+  what follows it; run them again on a cluster set up before.
+
+### Changed
+
+- **The store's PostgreSQL is sized to its machine's memory.** Before, it ran
+  on PostgreSQL's defaults (128MB cache) under Patroni, or on 96MB on one
+  machine. Now each member gets a share of its own RAM, a smaller one where
+  a control plane or agent shares the machine, and the share is read again
+  every time `server init --patroni` runs. The new cache is applied by
+  restarting one member at a time, the leader last.
+- **Knowledge-base searches hold up with many documents.** New collections
+  store their index as `halfvec` where pgvector is 0.7 or later (half the
+  memory, and embeddings up to 4000 dimensions instead of 2000); older
+  collections, and clusters on pgvector 0.6, keep `vector`. Each collection
+  has its own full-text index, as it has its own vector index, made for
+  existing collections when the control plane starts. The text arm ranks by
+  the cheap measure first and scores only those candidates.
+
+### Fixed
+
+- A knowledge-base search asking for more than 10 passages
+  (`X-Llmgw-Knowledge-K`) got at most 40 candidates per collection from the
+  vector index, whatever it asked for: pgvector's `hnsw.ef_search` is now
+  raised to the number wanted, up to 1000.
+- `llmgw server backup` and `restore` against a replicated store over TLS
+  failed looking for `~/.postgresql/root.crt`: pg_dump was given
+  `sslmode=verify-full` without the cluster's authority.
+
 ## v0.14.0 — 2026-10-08

 Knowledge bases: a company's documents, put in by people from their own
@@ -3,8 +3,9 @@
 A single-binary LLM gateway for clusters with multiple InfiniBand fabrics,
 NVLink domains and heterogeneous GPUs.

-It serves LLMs running on your own GPU servers (vLLM / SGLang) behind one
-OpenAI-compatible endpoint. Design document: [llmgw-plan.md](llmgw-plan.md).
+It serves LLMs running on your own GPU and CPU servers (vLLM, SGLang,
+llama.cpp, TensorRT-LLM) behind one OpenAI- and Anthropic-compatible
+endpoint, and answers from your own documents. Design document: [llmgw-plan.md](llmgw-plan.md).

 ## Getting started

@@ -37,6 +38,9 @@ Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
 | v0.9.0 | — | Managing people and the tenancy tree | released |
 | v0.10.0 | — | InfiniBand fabrics and NVLink, doctor, node add, engines installed by the nodes | released |
 | v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released |
+| v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released |
+| v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
+| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine) |
 | v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |

 As of v0.6.0 a cluster runs unattended: agents report their hardware, the
@@ -44,38 +48,67 @@ control plane places replicas — including ones larger than a single machine
 models load when someone asks for them and stop when nobody does, and keys,
 budgets and rate limits decide who may use any of it.

-Since v0.11.0, not yet released (see [CHANGELOG.md](CHANGELOG.md),
-"Unreleased"):
+Since v0.11.0 (each release's detail is in [CHANGELOG.md](CHANGELOG.md)):

- **Security hardening** — signed releases only, a token on the admin API,
-  engines not run as root, TLS for the agent port, etcd and Patroni, a
-  firewall for llmgw's internal ports, encrypted backups, bounded
-  connections, sign-in throttling, a Content-Security-Policy.
- **Guardrails** — policies that check requests and answers (PII, secrets,
-  prompt injection, deny patterns, a judge model such as Llama Guard), in
-  observe or enforce mode, attached to any part of the tenancy tree:
+- **Adapters, and the tenancy tree over the API** (v0.14.1). `llmgw ctl
+  kb dataset` and `llmgw ctl train` make a LoRA adapter from a collection
+  on a node's spare GPU memory, kept on the mirror and seen only by its
+  owner. Organizations, teams, projects, keys and memberships are in the
+  admin API and the Terraform provider. The agents' settings are edited
+  with the cluster's, on the portal too, and the firewall setting reaches
+  every machine.
+- **Knowledge bases** (v0.14.0). People put a company's documents in from
+  their own machines with `llmgw-kb`, a separate download for Windows,
+  macOS and Linux that reads PDF, Office, OpenDocument, mail and many more
+  formats and sends text and vectors, never the files. Collections are kept
+  apart per tenant, down to row-level security in PostgreSQL. A chat
+  request, or a part of the tree by default, is answered from them with
+  its sources: [docs/knowledge.md](docs/knowledge.md).
+- **One command for the whole system** (v0.13.0): `llmgw stop`, `start` and
+  `restart`. Qwen3Guard is the guardrails' judge, on a GPU and on CPUs
+  together.
+- **Settings that apply when changed** (v0.12.0). The cluster's
+  configuration lives in the store, the same on every control plane and in
+  force within seconds (`llmgw ctl config`, the console's Settings). The
+  firewall can be switched on and off for the whole cluster.
+- **Security hardening** (v0.11.2): signed releases only, a token on the
+  admin API, engines not run as root, TLS for the agent port, etcd and
+  Patroni, a firewall for llmgw's internal ports, encrypted backups,
+  bounded connections, sign-in throttling, a Content-Security-Policy.
+- **Guardrails** (v0.11.2). Policies check requests and answers for PII,
+  secrets, prompt injection, deny patterns, and with a judge model; they
+  observe or enforce, and attach to any part of the tenancy tree:
  [docs/guardrails.md](docs/guardrails.md).
- **Administration** — an audit log of every change (`ctl audit`), staff
-  roles (viewer, operator, billing, admin), single sign-on through OpenID
-  Connect and provisioning through SCIM 2.0, key lifetime limits and
-  rotation with overlap (`keys ttl`, `keys rotate`), usage export as CSV
-  (`ctl usage export`).
- **Notifications** — webhooks, signed or Slack-shaped, for a model parked,
-  a node lost, a budget spent and new weights published (`notify.webhooks`,
-  `ctl notify test`), and Alertmanager receiver templates.
- **Traffic and models** — request tags, canaries of new weights with
-  automatic rollback, quality suites gating updates, a response cache,
-  kept and replayed requests, batches, Anthropic's Messages API, more of
-  OpenAI's (structured output, images, rerank, audio, moderations),
-  scaling by the clock, SLO-aware admission, TensorRT-LLM, MIG, models
-  imported from Ollama, routing by what each vLLM replica's cache holds,
-  LoRA adapters loaded on demand, other llmgw clusters as providers, and
-  placement that knows racks, PDUs and power caps.
- **Operations** — `server restore` and off-machine backup copies, secrets
-  in Vault, forgetting a person and keeping a project's requests in the
-  cluster, offline installs from a bundle, a Helm chart, tracing from the
-  control plane to the engines, the admin API in OpenAPI with a Terraform
-  provider, and the portal in English and Turkish with its docs at /docs.
+- **Administration** (v0.11.2):
+  - an audit log of every change (`ctl audit`);
+  - staff roles (viewer, operator, billing, admin);
+  - single sign-on through OpenID Connect and provisioning through SCIM
+    2.0;
+  - key lifetime limits, and rotation with overlap (`keys ttl`, `keys
+    rotate`);
+  - usage export as CSV (`ctl usage export`).
+- **Notifications** (v0.11.2): webhooks, signed or Slack-shaped, for a
+  model parked, a node lost, a budget spent and new weights published, and
+  Alertmanager receiver templates.
+- **Traffic and models** (v0.11.2):
+  - request tags, canaries of new weights with automatic rollback, and
+    quality suites gating updates;
+  - a response cache, kept and replayed requests, and batches;
+  - Anthropic's Messages API, and more of OpenAI's (structured output,
+    images, rerank, audio, moderations);
+  - scaling by the clock, and SLO-aware admission;
+  - TensorRT-LLM, MIG, and models imported from Ollama;
+  - routing by what each vLLM replica's cache holds, and LoRA adapters
+    loaded on demand;
+  - other llmgw clusters as providers, and placement that knows racks,
+    PDUs and power caps.
+- **Operations** (v0.11.2):
+  - `server restore`, off-machine backup copies, and secrets in Vault;
+  - forgetting a person, and keeping a project's requests in the cluster;
+  - offline installs from a bundle, and a Helm chart;
+  - tracing from the control plane to the engines;
+  - the admin API in OpenAPI, with a Terraform provider;
+  - the portal in English and Turkish, with its docs at /docs.

 Every one of these is in the console as well as on the command line:
 [docs/console.md](docs/console.md).
@@ -124,6 +157,9 @@ Every page is also in the binary: `llmgw docs`, or **Manual** in the console.
  official OpenAI and Anthropic SDKs, and llmgw's extensions
 - [docs/guardrails.md](docs/guardrails.md) — checking requests and answers:
  policies, detectors, a judge model, and where each applies
+- [docs/knowledge.md](docs/knowledge.md) — knowledge bases: collections,
+  who may read them, `llmgw-kb` on people's machines, and answers from
+  documents
 - [docs/choosing-models.md](docs/choosing-models.md) — which models to run,
  how many of each, and where the weights should live

@@ -152,7 +188,7 @@ Every page is also in the binary: `llmgw docs`, or **Manual** in the console.
 ## Building

 ```sh
-make build      # bin/llmgw
+make build      # bin/llmgw and bin/llmgw-kb
 make test       # unit tests
 make race       # with the race detector
 ```
@@ -169,6 +205,7 @@ llmgw ctl cordon gpu-a-01            # stop new placement, leave what runs
 sudo llmgw doctor                    # on a node: what its GPUs, NVLink and fabric need
 llmgw ctl audit --since 24h          # who changed what
 llmgw ctl usage export --from 2026-09-01 --to 2026-10-01 --by project,model
+llmgw ctl kb list                    # knowledge-base collections, grants, defaults
 ```

 Bare `llmgw` opens the console, which does all of this from one screen.
@@ -28,6 +28,8 @@ message AgentMessage {
    FabricTestReply fabric_test = 6;
    UpgradeReply upgrade = 7;
    KVEvents kv_events = 8;
+    TrainStatus train_status = 9;
+    TrainArtifact train_artifact = 10;
  }
 }

@@ -73,6 +75,9 @@ message ControlMessage {
    ControlPlanes control_planes = 11;
    Firewall firewall = 13;
    RestartAgent restart_agent = 14;
+    AgentSettings agent_settings = 15;
+    TrainStart train_start = 16;
+    TrainCancel train_cancel = 17;
  }
  // trace is the W3C trace context (traceparent, and tracestate when there
  // is one) of the control plane's span that sent this command, so what the
@@ -103,6 +108,17 @@ message Firewall {
  int32 engine_ports_to = 4;
 }

+// AgentSettings is what the cluster's settings say of this node's agent
+// (agents.default, with agents.nodes.<id> over it), sent when it registers
+// and whenever they change. The agent keeps it beside agent.yaml, which
+// holds only what it needs to start; over that file, it is what the agent
+// runs on. A change from what it runs on has the agent start again on it,
+// its engines serving throughout.
+message AgentSettings {
+  // yaml is agent.yaml's keys, as YAML; empty for none.
+  string yaml = 1;
+}
+
 // ControlPlanes is every control plane an agent may dial, sent by the leader
 // when an agent registers and whenever the list changes. The agent keeps it
 // beside its configuration, so a second control plane added after the node
@@ -872,3 +888,55 @@ message LoraAdapter {
  string name = 1;
  string source = 2;
 }
+
+// TrainStart has a node train a LoRA adapter (BACKLOG F4): on a base model's
+// weights from its registry, on question/answer pairs, with a Python trainer
+// the agent carries, in an environment it makes over an installed engine's.
+// One at a time per node.
+message TrainStart {
+  string job_id = 1;
+  // base_repo and base_revision name the base model's weights on its
+  // registry (Hugging Face): safetensors, not the GGUF a catalog may serve.
+  string base_repo = 2;
+  string base_revision = 3;
+  // data is the training pairs, NDJSON of {"q", "a"}.
+  bytes data = 4;
+  // params is the trainer's settings as JSON: rank, alpha, epochs, lr,
+  // batch, max_len.
+  string params = 5;
+  // gpu_memory_mib is what the job needs of a card; the agent starts it
+  // only when one has that much free, which lets it run beside an engine.
+  int32 gpu_memory_mib = 6;
+  // gguf asks for the adapter in llama.cpp's format too.
+  bool gguf = 7;
+}
+
+// TrainCancel stops a training job.
+message TrainCancel {
+  string job_id = 1;
+}
+
+// TrainStatus is how a training job is going, sent as it changes.
+message TrainStatus {
+  string job_id = 1;
+  // state is preparing, running, done or failed.
+  string state = 2;
+  int32 step = 3;
+  int32 steps = 4;
+  double loss = 5;
+  // message is what is being done, or why it failed.
+  string message = 6;
+  // peak_mib is the most the job held of the card.
+  int32 peak_mib = 7;
+}
+
+// TrainArtifact is a piece of what a training job made, sent before its
+// final status: a gzipped tar of the PEFT adapter (kind peft), or the GGUF
+// adapter (kind gguf).
+message TrainArtifact {
+  string job_id = 1;
+  string kind = 2;
+  int32 seq = 3;
+  bytes data = 4;
+  bool last = 5;
+}
@@ -8,8 +8,9 @@ description is.
 anything they do can be done with `curl`, a generated client or the
 [Terraform provider](../terraform-provider-llmgw/README.md). It adds and
 removes models, scales them, cordons and upgrades nodes, sets capacity
-quotas, attaches guardrail policies, and reads the fleet, usage and the audit
-log.
+quotas, attaches guardrail policies, adds organizations, teams and projects,
+issues and revokes API keys, puts people in the tree, and reads the fleet,
+usage and the audit log.

 ## Reaching it

@@ -99,6 +100,14 @@ curl -s "${H[@]}" -X POST $API/models/qwen-small/scale -d '{"min": 0, "max": 2,
 # A team's capacity quota.
 curl -s "${H[@]}" -X PUT $API/quotas -d '{"scope": "team:Acme/ml", "limit": {"gpus": 8}}'

+# An organization, a team in it and a project in that; then a key for the
+# project. The answer holds the key once: only its hash is kept.
+curl -s "${H[@]}" -X POST $API/orgs -d '{"name": "Acme"}'
+curl -s "${H[@]}" -X POST $API/teams -d '{"name": "ml", "in": "Acme"}'
+curl -s "${H[@]}" -X POST $API/projects -d '{"name": "search", "in": "Acme/ml"}'
+curl -s "${H[@]}" -X POST $API/keys -d '{"name": "ci", "project": "Acme/ml/search", "expires_in": "720h"}'
+curl -s "${H[@]}" -X POST $API/users/ayse@example.com/memberships -d '{"target": "team:Acme/ml"}'
+
 # Remove the model; a DELETE carries the content type too.
 curl -s "${H[@]}" -X DELETE $API/models/qwen-small
 ```
@@ -149,11 +158,11 @@ go generate ./internal/docgen

 ## What it does not cover

- **Organizations, teams, projects, people and API keys.** These live in the
-  store and are managed with `llmgw org`, `llmgw users` and `llmgw keys` on the
-  control plane, or in the web console's `/admin` pages on the client port
-  behind a session. The admin API reads the tree only where a quota or a
-  policy names a part of it.
+- **Making people's accounts.** People sign up, are invited, or come from
+  the identity provider (OIDC, SCIM); the API lists them and puts them in
+  the tree, and `llmgw users` approves, disables and deletes them. Key
+  lifetime limits and spending limits are `llmgw keys ttl` and `llmgw keys
+  limits`.
 - **The client-facing API** (`/v1/chat/completions` and the rest), which is
  OpenAI's and Anthropic's, on `listen`, with API keys.
 - **The rest of the admin listener**: `/metrics`, `/health`, the dashboard
@@ -201,6 +201,16 @@ this by running the same command again, on the control plane that holds the
 authority: etcd is rebuilt on TLS with Patroni in maintenance mode, so
 PostgreSQL serves throughout and no failover happens.

+Each member's PostgreSQL is sized to the machine's memory, read each time
+the command runs: a quarter of it as cache (`shared_buffers`), three quarters
+counted on for the planner, and room to sort and build indexes, all capped. A
+member that also runs a control plane or an agent gets less: about 15% as
+cache, so the gateway is not pushed into swap. A knowledge base's vector
+index is searched well only from memory, so give a store machine more RAM and
+run the command again. The new cache takes a PostgreSQL restart, which Patroni
+does one member at a time, the leader last. The leader's restart costs a few
+seconds without writes, and there is no failover.
+
 Every step looks before it acts. Running it again finishes an install that
 stopped halfway and changes nothing on one that did not; it never overwrites
 `server.yaml`, the authority, or an existing account's password.
@@ -258,8 +268,14 @@ llmgw ctl config history                 # who changed what, when, and why
 llmgw ctl config rollback 4              # back to version 4, as a new version
 ```

-or on the console's **Settings** screen. A change is checked before it is
-kept — one that would not load is refused — and recorded in the audit log.
+or on the console's **Settings** screen, or in a browser on the portal's
+**Settings** page (`/admin/settings`, for administrators only: the document
+can hold webhook secrets). The page edits the same document, lists its
+history, and puts an earlier version back. A change is checked before it is
+kept, so one that would not load is refused. A change made from a version
+someone else has replaced since is refused too, rather than overwriting
+theirs. Every change is recorded in the audit log, with the document's size
+but not its contents.
 Guardrails change live; anything else has every control plane start again on
 the new settings, the standbys first and the leader half a minute later, so
 one of them is always serving. Engines keep running through it.
@@ -269,6 +285,33 @@ cluster keys — with a `baseline` guard policy, if the file had none — and
 from then on those keys in any server.yaml are not read (the server says so
 when it starts).

+**The nodes' agents** are set the same way, under `agents`. `agents.default`
+is for every node, and `agents.nodes.<node-id>` is for one node, laid over
+the default:
+
+```
+llmgw ctl config set agents.default.heartbeat_interval 5s
+llmgw ctl config set agents.nodes.cuda01.labels.gpu rtx6000
+llmgw ctl config edit                    # fabrics, engines, supervisor, discovery…
+```
+
+Each node's `agent.yaml` then needs only what its agent must know before it
+can be told anything: `node_id`, `server` or `servers`, `hostname`,
+`mgmt_addr` and `tls`. Those keys are refused under `agents`.
+
+- **Precedence.** Anything else in the file is still read, with the
+  cluster's settings over it.
+- **Applying a change.** The control plane sends each node its settings, a
+  second apart. An agent whose settings changed keeps them beside its
+  `agent.yaml` (in `/var/lib/llmgw-agent/agent-settings.yaml`) and starts
+  again on them, its engines serving throughout.
+- **Bad settings.** Settings that would not load are refused when they are
+  saved. If they somehow do not load beside one node's own file, that agent
+  keeps running on what it has and logs why.
+- **Editing `agent.yaml` itself.** The agent notices the change within ten
+  seconds and starts again on it. If the edited file does not load, the
+  agent reports it and keeps running on what it had.
+
 **This machine's own keys** stay in `/etc/llmgw/server.yaml`: `listen`,
 `admin_listen`, `metrics_listen`, `agent_listen`, `agent_tls`, `auth.store`,
 `secrets`, `tracing`, `monitoring`, `mirror.api` and `mirror.tls`, the backup
@@ -1227,6 +1270,51 @@ prices and budgets, rate limits, usage records, and key allowlists.
 whose allowlist leaves the external model out cannot reach it, as a fallback or
 by name. The cluster page and `ctl models` mark it `external`.

+### Pictures from ComfyUI
+
+A ComfyUI server is an external model with `api: comfyui`. Clients ask it for
+pictures at OpenAI's `/v1/images/generations`; llmgw fills a ComfyUI workflow
+with the request, queues it, waits, and sends the pictures back as `b64_json`:
+
+```yaml
+- name: flux-schnell
+  source: external
+  external:
+    url: http://10.0.0.147:8188                  # ComfyUI, started with --listen
+    model: flux1-schnell-fp8.safetensors         # the checkpoint, as ComfyUI lists it
+    api: comfyui
+    workflow: flux-schnell                       # or flux-dev, or a JSON file
+```
+
+`flux-schnell` (the default) and `flux-dev` are built in. They suit the
+single-file FLUX checkpoints that ComfyUI loads with `CheckpointLoaderSimple`.
+For any other model, export the workflow from ComfyUI with **Save (API)** and
+name the file. In the file, put placeholders where the request's values go:
+
+- `{{prompt}}` and `{{negative_prompt}}` are the prompts.
+- `{{width}}` and `{{height}}` are the size.
+- `{{n}}` is the batch size.
+- `{{seed}}` is the seed.
+- `{{steps|4}}` and `{{guidance|3.5}}` take the value after the `|` when the
+  request leaves them out.
+- `{{checkpoint}}` is the `model` above.
+
+A request is OpenAI's: `prompt`, `n` (at most 4), `size` (each side a multiple
+of 16, from 256 to 2048) and `response_format: b64_json`. Beyond those, it may
+send `seed`, `steps`, `negative_prompt` and `guidance`. The seed used comes
+back in `x-llmgw-seed`. `url` is not offered, because pictures are kept
+nowhere. ComfyUI also writes the workflow into each PNG; llmgw strips it
+before the picture is sent.
+
+Requests for pictures pass the same checks as any other request: keys,
+allowlists, limits, admission, and the guardrails on the prompt. A client that
+leaves takes its workflow off the ComfyUI queue. Usage is recorded, but for
+now it is charged as the prompt's tokens; a per-picture price is not yet
+available.
+
+Start ComfyUI with `--disable-smart-memory` on a card that llmgw also places
+models on. Without it, ComfyUI keeps the model on the card after each picture.
+
 ### Federating clusters

 Another llmgw cluster — another site, another team's GPUs — can be a provider
@@ -1703,9 +1791,11 @@ the file again.

 On a machine that also runs the control plane, PostgreSQL's defaults assume it
 owns the box. A gateway swapped out to give the database cache is a worse
-outcome than a smaller cache, so cap it: `shared_buffers = 96MB` and
-`max_connections = 40` are enough for a store whose working set is measured in
-megabytes.
+outcome than a smaller cache, so size it to the machine. `server init` does this
+for a PostgreSQL it installs: `shared_buffers` about 15% of memory (at least
+96MB), `effective_cache_size` half, `work_mem` and `maintenance_work_mem`
+scaled to it, and `max_connections = 40`. A store with a large knowledge base
+wants a machine of its own, where its vector index fits in memory.

 ## Tagging requests

@@ -2151,13 +2241,29 @@ llmgw ctl config set firewall.enabled false   # every machine's table taken out
 llmgw ctl config set firewall.enabled true    # and put back as it was
 ```

-It is in force within seconds: each node's agent takes its table out or puts
-it back, and each control plane's `llmgw-firewall-sync.path` (installed by
-`server init` and `server add`; `sudo llmgw firewall sync --install` on one
-set up before) does the same there. The rules files stay, so turning it back
-on restores them. Nodes added while it is off get none. A separate mirror
-machine and the store's members keep theirs for now. `firewall_networks`
-changes reach the nodes the same way.
+It is in force on every machine within a minute, each in its own way:
+
+- **nodes**: the agent takes its table out or puts it back;
+- **control planes**: `llmgw-firewall-sync.path`, installed by `server
+  init` and `server add` (`sudo llmgw firewall sync --install` on one set up
+  before);
+- **a separate mirror machine**: the leading control plane tells the mirror,
+  which writes it to `.firewall.json` in its directory, and the same path
+  unit applies it. `mirror init` installs it; on a mirror set up before, run
+  `sudo llmgw mirror init` again with the same flags;
+- **store members that are nothing else**: they run no llmgw, so
+  `llmgw-firewall-follow.timer` reads what the leading control plane keeps in
+  the store, from the member's own copy, once a minute. `server init
+  --patroni` installs it; run it again on a cluster set up before.
+
+The rules files stay, so turning it back on restores them. Nodes added while
+it is off get none. Networks added to `firewall_networks` reach every machine
+the same way. One taken out of it stays let in until it is taken out of
+`/etc/llmgw/firewall.nft` by hand: the rules are only ever added to.
+
+The leading control plane also tells the mirror machine where every node and
+control plane is, so a node on a network the mirror had not let in can fetch
+its weights without a `firewall_networks` entry.

 Only those ports are touched. The table's policy is to accept, so whatever
 else the machine serves — and whatever firewall it already has — is left as
@@ -45,6 +45,20 @@ is available to everyone who may use them.
  A collection stays with the model it was made with: vectors from two
  models cannot be compared.

+  Some embedding models are trained with an instruction before the
+  question and none before the documents. Give it as the entry's
+  `query_prefix`; llmgw puts it before questions only. For
+  Qwen3-Embedding:
+
+  ```yaml
+  query_prefix: "Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery: "
+  ```
+
+  With pgvector 0.7 or later a collection's index is kept at half
+  precision (`halfvec`), which halves its memory and takes models of up to
+  4000 dimensions; with an older pgvector the limit is 2000. A collection
+  keeps the kind it was made with.
+
 ## For administrators

 ### Make a collection
@@ -317,9 +331,11 @@ Behaviour to know about:
  (Codex, Claude Code) work the same way, since they are answered as chat
  completions.
 - **Separate prefix caches.** On vLLM replicas, a request given passages
-  carries a `cache_salt` derived from its organization. One organization's
-  passages then never share cached blocks with another's, so how fast an
-  answer starts says nothing about what another tenant asked.
+  carries a `cache_salt` derived from the collections they came from.
+  Requests that could be given the same passages still share cached blocks,
+  and no one else's do: not another organization's, nor another team's of
+  the same one. How fast an answer starts therefore says nothing about what
+  someone else asked.
 - **No answer cache.** Answers from knowledge bases are never served from
  the answer cache, because two keys of one project can read different
  collections.
@@ -340,6 +356,49 @@ a product code or a name.
 Metrics: `llmgw_kb_retrievals_total{result}` (found, nothing, refused) and
 `llmgw_kb_retrieval_seconds`.

+## Training an adapter on a collection
+
+An adapter (LoRA) changes how a model *writes*: the organisation's words,
+format and tone. It does not reliably teach facts. On the PoC, a small
+model trained on nine policy documents answered 4 of 9 rephrased questions
+right and mixed facts up on the rest, while the same questions through the
+knowledge base got 9 of 9. Use both: the knowledge base for what answers
+say, an adapter for how they sound.
+
+```sh
+# 1. Questions and answers from the collection, written by a model of the
+#    cluster. Read the file through and correct it.
+llmgw ctl kb dataset Acme/policies --model qwen2.5-coder-14b-instruct --out policies.jsonl
+
+# 2. Train on a node. --gpu-memory is what the job needs of a card; it
+#    runs beside an engine when the engine leaves that much free.
+llmgw ctl train start policies-tone --model qwen2.5-0.5b-instruct \
+  --base-repo Qwen/Qwen2.5-0.5B-Instruct --node cuda01 --gpu-memory 2300 --data policies.jsonl
+llmgw ctl train show JOB --follow
+
+# 3. Make it the model's adapter, kept on the mirror.
+llmgw ctl train publish JOB --restart
+```
+
+The node trains with its installed vLLM's torch and a small overlay (peft
+and accelerate), as the engines' user. For a model llama.cpp serves, it also
+converts the adapter to GGUF with the installed llama.cpp's own code. What
+it makes is kept by the control plane (`llmgw ctl train download`).
+
+An adapter is private like its collection:
+
+- **It has an owner.** An adapter trained on a collection carries some of
+  what the collection says in its weights, so it belongs to a part of the
+  tree.
+  - By default the owner is the collection's owner.
+  - `--owner` names another, which must be able to read every collection
+    the pairs came from.
+  - Only callers in the owner's part of the tree see the adapter. To anyone
+    else it is a model that does not exist, absent from `/v1/models` too.
+- **It is public only by choice.** `--public` makes an adapter every
+  caller's. It is refused for pairs from a knowledge base, so it is only for
+  pairs written by hand.
+
 ## How it is kept apart

 Passages live in PostgreSQL, partitioned by organization, with an index per
@@ -18,6 +18,7 @@ rather than repeated.
 | `name` | `string` | [Adapter](#adapter) |
 | `source` | `string` | [Adapter](#adapter) |
 | `scale` | `float64` | [Adapter](#adapter) |
+| `owner` | `string` | [Adapter](#adapter) |
 | `enabled` | `bool` | [Admission](#admission) |
 | `bands` | `[]Band` | [Admission](#admission) |
 | `default_band` | `string` | [Admission](#admission) |
@@ -26,6 +27,8 @@ rather than repeated.
 | `queue_capacity_per_band` | `int` | [Admission](#admission) |
 | `queue_capacity_total` | `int` | [Admission](#admission) |
 | `interval` | `time.Duration` | [Admission](#admission) |
+| `default` | `map[string]any` | [AgentSettings](#agentsettings) |
+| `nodes` | `map[string]map[string]any` | [AgentSettings](#agentsettings) |
 | `enabled` | `bool` | [Auth](#auth) |
 | `store` | `string` | [Auth](#auth) |
 | `cache_ttl` | `time.Duration` | [Auth](#auth) |
@@ -57,6 +60,7 @@ rather than repeated.
 | `tools` | `bool` | [Capabilities](#capabilities) |
 | `vision` | `bool` | [Capabilities](#capabilities) |
 | `embeddings` | `bool` | [Capabilities](#capabilities) |
+| `images` | `bool` | [Capabilities](#capabilities) |
 | `interval` | `time.Duration` | [Control](#control) |
 | `base_backoff` | `time.Duration` | [Control](#control) |
 | `max_backoff` | `time.Duration` | [Control](#control) |
@@ -73,6 +77,7 @@ rather than repeated.
 | `api_key` | `string` | [Endpoint](#endpoint) |
 | `models` | `[]string` | [Endpoint](#endpoint) |
 | `engine` | `string` | [Endpoint](#endpoint) |
+| `workflow` | `string` | [Endpoint](#endpoint) |
 | `pool` | `string` | [Endpoint](#endpoint) |
 | `weight` | `float64` | [Endpoint](#endpoint) |
 | `priority` | `int` | [Endpoint](#endpoint) |
@@ -87,6 +92,8 @@ rather than repeated.
 | `model` | `string` | [External](#external) |
 | `api_key_file` | `string` | [External](#external) |
 | `peer` | `string` | [External](#external) |
+| `api` | `string` | [External](#external) |
+| `workflow` | `string` | [External](#external) |
 | `budgets` | `[]FacilityBudget` | [Facility](#facility) |
 | `margin` | `float64` | [Facility](#facility) |
 | `gpu_watts` | `float64` | [Facility](#facility) |
@@ -142,6 +149,7 @@ rather than repeated.
 | `context_length` | `int` | [Model](#model) |
 | `max_output` | `int` | [Model](#model) |
 | `capabilities` | `Capabilities` | [Model](#model) |
+| `query_prefix` | `string` | [Model](#model) |
 | `role` | `string` | [Model](#model) |
 | `guard` | `*GuardModel` | [Model](#model) |
 | `origin` | `*Origin` | [Model](#model) |
@@ -294,6 +302,7 @@ rather than repeated.
 | `mail` | `mail.Config` | [Server](#server) |
 | `serving` | `ServingSettings` | [Server](#server) |
 | `firewall` | `FirewallSettings` | [Server](#server) |
+| `agents` | `AgentSettings` | [Server](#server) |
 | `firewall_networks` | `[]string` | [Server](#server) |
 | `backup_recipients` | `[]string` | [Server](#server) |
 | `backup_copy_to` | `[]string` | [Server](#server) |
@@ -423,6 +432,16 @@ weights are.

 Scale is how strongly it is applied; zero takes 1.

+### `owner`
+
+`string`
+
+Owner is the part of the tenancy tree the adapter belongs to —
+org:ID, team:ID, project:ID, user:ID or key:ID — and only callers in
+it see it: to anyone else it is a model that does not exist. An
+adapter trained on an organisation's documents carries some of what
+they say in its weights. Empty is every caller the model allows.
+
 ## Admission

 ### `enabled`
@@ -469,6 +488,21 @@ Zero takes half of all the bands' capacities together.

 `time.Duration`

+## AgentSettings
+
+### `default`
+
+`map[string]any`
+
+Default is agent.yaml's keys for every node.
+
+### `nodes`
+
+`map[string]map[string]any`
+
+Nodes are agent.yaml's keys for one node each, by node id, over
+Default.
+
 ## Auth

 ### `enabled`
@@ -686,6 +720,14 @@ token is MaxTTFTRatio times this model's. Zero is 50, 0.05 and 1.5.

 `bool`

+### `images`
+
+`bool`
+
+Images is a model that makes pictures, asked for at
+/v1/images/generations. Implied by an external model whose API is
+ComfyUI's.
+
 ## Control

 ### `interval`
@@ -799,6 +841,13 @@ with one.
 Engine names the inference engine, which decides what its metrics are
 called. Empty means vLLM's names.

+### `workflow`
+
+`string`
+
+Workflow is the ComfyUI workflow for engine: comfyui, a built-in's
+name or its JSON; flux-schnell unless set.
+
 ### `pool`

 `string`
@@ -890,6 +939,23 @@ a model that federation added (`llmgw ctl federation add`) rather
 than one written by hand. Such a model is named PEER/MODEL, comes and
 goes with the peer's own list, and is sent only while the peer is up.

+### `api`
+
+`string`
+
+API is what the provider speaks: OpenAI's, unless this says comfyui —
+a ComfyUI server, asked for pictures at /v1/images/generations and
+sent the workflow below with the request's prompt in it. Model is then
+the checkpoint file the workflow loads, as ComfyUI lists it.
+
+### `workflow`
+
+`string`
+
+Workflow is the ComfyUI workflow, in its API format: a built-in's name
+(flux-schnell, the default, or flux-dev) or a JSON file on the control
+plane.
+
 ## Facility

 ### `budgets`
@@ -1360,6 +1426,16 @@ Capabilities is what this model can do beyond plain completion.

 Takes a [Capabilities](#capabilities) block.

+### `query_prefix`
+
+`string`
+
+QueryPrefix is put before a question an embedding model embeds for a
+knowledge-base search; documents are embedded without it. Models
+trained with an instruction on the query side want it, Qwen3-Embedding
+"Instruct: Given a web search query, retrieve relevant passages that
+answer the query\nQuery: " for one.
+
 ### `role`

 `string`
@@ -2520,6 +2596,18 @@ Firewall turns llmgw's firewall on every machine on or off.

 Takes a [FirewallSettings](#firewallsettings) block.

+### `agents`
+
+`AgentSettings`
+
+Agents are the nodes' agent settings: agent.yaml's keys but those an
+agent needs to start, for every node (default) and for one
+(nodes.<id>). Sent to the agents as they change; an agent whose
+settings changed starts again on them, its engines serving
+throughout.
+
+Takes a [AgentSettings](#agentsettings) block.
+
 ### `firewall_networks`

 `[]string`
@@ -34,6 +34,7 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/train"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/agentsvc"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
@@ -136,6 +137,13 @@ type Agent struct {
 	// by mu.
 	kv     *kvevents.Mirror
 	kvSubs map[string]*kvevents.Subscriber
+
+	// settingsCheck says whether the cluster's settings for this agent
+	// would load beside its agent.yaml; see SetSettingsCheck.
+	settingsCheck func([]byte) error
+
+	// trainer runs LoRA training jobs, one at a time (BACKLOG F4).
+	trainer *train.Runner
 }

 // connectHTTPClient is the HTTP client connect uses; an interface so tests can
@@ -182,6 +190,18 @@ func New(cfg Config, httpClient connectHTTPClient, log *slog.Logger) (*Agent, er
 		kvSubs:       map[string]*kvevents.Subscriber{},
 	}
 	a.prov = &provision.Installer{Root: cfg.EnginesDir, Log: log, OnStatus: a.provisionStatus}
+	runAs := cfg.Superv.RunAs
+	if runAs == "" {
+		runAs = supervisor.DefaultRunAs
+	}
+	// Where the engines are, as the installer has it when the file says
+	// nothing.
+	enginesDir := cfg.EnginesDir
+	if enginesDir == "" {
+		enginesDir = provision.DefaultRoot
+	}
+	a.trainer = train.New(train.Env{EnginesDir: enginesDir, WorkDir: filepath.Join(cfg.Superv.WorkDir, "trainer"),
+		RunAs: runAs, Log: log})
 	return a, nil
 }

@@ -880,6 +900,12 @@ func (a *Agent) handle(ctx context.Context, msg *llmgwv1.ControlMessage, out cha
 		a.setControlPlanes(m.ControlPlanes)
 	case *llmgwv1.ControlMessage_Firewall:
 		go a.applyFirewall(m.Firewall)
+	case *llmgwv1.ControlMessage_AgentSettings:
+		go a.applySettings(m.AgentSettings)
+	case *llmgwv1.ControlMessage_TrainStart:
+		go a.trainJob(ctx, m.TrainStart, out)
+	case *llmgwv1.ControlMessage_TrainCancel:
+		a.trainer.Cancel(m.TrainCancel.GetJobId())
 	case *llmgwv1.ControlMessage_RestartAgent:
 		go func() {
 			a.log.Info("restarting, as the control plane asks; the engines keep serving",
@@ -103,6 +103,11 @@ type Config struct {
 	Tracing obs.Tracing `koanf:"tracing"`

 	Version string
+
+	// Settings is the cluster's settings for this agent it was started on,
+	// as the control plane sent them: what a newly sent version is
+	// compared with.
+	Settings string `koanf:"-"`
 }

 // DefaultConfig returns sensible defaults.
@@ -0,0 +1,109 @@
+package agent
+
+import (
+	"context"
+	"crypto/sha256"
+	"os"
+	"path/filepath"
+	"time"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+)
+
+// The cluster's settings for this agent (BACKLOG E3): what agent.yaml can
+// say but the agent's id, control planes, address and certificate, kept by
+// the control planes and sent over the stream. Kept here beside agent.yaml,
+// which then holds only what the agent needs to start; at start the two are
+// read together, these over the file.
+
+// SettingsFile keeps the settings the control plane last sent.
+var SettingsFile = "/var/lib/llmgw-agent/agent-settings.yaml"
+
+// ReadSettings is what SettingsFile holds, nil when nothing.
+func ReadSettings() []byte {
+	b, err := os.ReadFile(SettingsFile)
+	if err != nil {
+		return nil
+	}
+	return b
+}
+
+// SetSettingsCheck says how to tell whether settings would load beside this
+// agent's own file; settings that would not are not taken.
+func (a *Agent) SetSettingsCheck(f func([]byte) error) { a.settingsCheck = f }
+
+// applySettings takes the settings the control plane sent: nothing when they
+// are what this agent runs on, and otherwise kept and started again on.
+func (a *Agent) applySettings(m *llmgwv1.AgentSettings) {
+	next := m.GetYaml()
+	if next == a.cfg.Settings {
+		return
+	}
+	if a.settingsCheck != nil {
+		if err := a.settingsCheck([]byte(next)); err != nil {
+			// Kept running on what it has: settings that will not load
+			// here must not take the node out of service.
+			a.log.Error("the cluster's settings for this agent do not load beside its agent.yaml; running on what it has",
+				"err", err)
+			return
+		}
+	}
+	if err := os.MkdirAll(filepath.Dir(SettingsFile), 0o755); err != nil {
+		a.log.Error("could not keep the cluster's settings for this agent", "err", err)
+		return
+	}
+	tmp := SettingsFile + ".new"
+	if err := os.WriteFile(tmp, []byte(next), 0o600); err != nil {
+		a.log.Error("could not keep the cluster's settings for this agent", "err", err)
+		return
+	}
+	if err := os.Rename(tmp, SettingsFile); err != nil {
+		a.log.Error("could not keep the cluster's settings for this agent", "err", err)
+		return
+	}
+	a.log.Info("the cluster's settings for this agent changed; starting again on them, the engines keep serving")
+	time.Sleep(300 * time.Millisecond)
+	exitForUpgrade()
+}
+
+// WatchFile starts the agent again when its configuration file changes and
+// still loads: agent.yaml is read only at start, and a change to it that
+// waited for somebody to restart the agent was a change nobody saw take
+// effect.
+func (a *Agent) WatchFile(ctx context.Context, path string, loads func() error) {
+	if path == "" {
+		return
+	}
+	sum := func() [32]byte {
+		b, _ := os.ReadFile(path) //nolint:gosec // the agent's own configuration
+		return sha256.Sum256(b)
+	}
+	was := sum()
+	t := time.NewTicker(watchEvery)
+	defer t.Stop()
+	for {
+		select {
+		case <-ctx.Done():
+			return
+		case <-t.C:
+		}
+		now := sum()
+		if now == was {
+			continue
+		}
+		if err := loads(); err != nil {
+			// Once per change: an editor saving halfway must not stop the
+			// agent, and a broken file is reported, not restarted into.
+			a.log.Error("agent.yaml changed and does not load; running on what it had", "path", path, "err", err)
+			was = now
+			continue
+		}
+		a.log.Info("agent.yaml changed; starting again on it, the engines keep serving", "path", path)
+		time.Sleep(300 * time.Millisecond)
+		exitForUpgrade()
+		return
+	}
+}
+
+// watchEvery is how often agent.yaml is looked at.
+var watchEvery = 10 * time.Second
@@ -0,0 +1,88 @@
+package agent
+
+import (
+	"errors"
+	"os"
+	"path/filepath"
+	"sync/atomic"
+	"testing"
+	"time"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+)
+
+func settingsAgent(t *testing.T, running string) (*Agent, chan struct{}) {
+	t.Helper()
+	oldFile, oldExit := SettingsFile, exitForUpgrade
+	SettingsFile = filepath.Join(t.TempDir(), "agent-settings.yaml")
+	exited := make(chan struct{}, 1)
+	exitForUpgrade = func() { exited <- struct{}{} }
+	t.Cleanup(func() { SettingsFile, exitForUpgrade = oldFile, oldExit })
+	cfg := Config{Server: "127.0.0.1:7443", NodeID: "n1", WeightsCache: t.TempDir(), EnginesDir: t.TempDir(), Settings: running}
+	cfg.Superv.WorkDir = t.TempDir()
+	a, err := New(cfg, nil, nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	return a, exited
+}
+
+// exitedSoon is whether the agent exited within a second: it waits a
+// moment for its log line to leave first.
+func exitedSoon(ch chan struct{}) bool {
+	select {
+	case <-ch:
+		return true
+	case <-time.After(time.Second):
+		return false
+	}
+}
+
+func TestAnAgentOnItsSettingsStaysAndOneOffThemStartsAgain(t *testing.T) {
+	a, exited := settingsAgent(t, "heartbeat_interval: 5s\n")
+	a.applySettings(&llmgwv1.AgentSettings{Yaml: "heartbeat_interval: 5s\n"})
+	if exitedSoon(exited) || ReadSettings() != nil {
+		t.Error("settings it already runs on restarted it")
+	}
+	a.applySettings(&llmgwv1.AgentSettings{Yaml: "heartbeat_interval: 2s\n"})
+	if !exitedSoon(exited) || string(ReadSettings()) != "heartbeat_interval: 2s\n" {
+		t.Errorf("new settings: kept %q", ReadSettings())
+	}
+}
+
+func TestSettingsThatWouldNotLoadAreNotTaken(t *testing.T) {
+	a, exited := settingsAgent(t, "")
+	a.SetSettingsCheck(func([]byte) error { return errors.New("fabric ib-a lists no HCAs") })
+	a.applySettings(&llmgwv1.AgentSettings{Yaml: "fabrics: [{name: ib-a}]\n"})
+	if exitedSoon(exited) || ReadSettings() != nil {
+		t.Error("settings that do not load were taken")
+	}
+}
+
+func TestAChangedAgentYamlThatLoadsStartsTheAgentAgain(t *testing.T) {
+	a, exited := settingsAgent(t, "")
+	old := watchEvery
+	watchEvery = 10 * time.Millisecond
+	t.Cleanup(func() { watchEvery = old })
+	p := filepath.Join(t.TempDir(), "agent.yaml")
+	_ = os.WriteFile(p, []byte("node_id: n1\n"), 0o600)
+	var loads atomic.Bool
+	check := func() error {
+		if loads.Load() {
+			return nil
+		}
+		return errors.New("broken")
+	}
+	go a.WatchFile(t.Context(), p, check)
+	time.Sleep(30 * time.Millisecond)
+	// Saved broken: reported, not restarted into.
+	_ = os.WriteFile(p, []byte("node_id: [\n"), 0o600)
+	if exitedSoon(exited) {
+		t.Fatal("a broken agent.yaml restarted the agent")
+	}
+	loads.Store(true)
+	_ = os.WriteFile(p, []byte("node_id: n1\nlabels: {a: b}\n"), 0o600)
+	if !exitedSoon(exited) {
+		t.Error("a changed agent.yaml did not restart the agent")
+	}
+}
@@ -3,6 +3,7 @@ package supervisor
 import (
 	"fmt"
 	"os"
+	"os/exec"
 	"os/user"
 	"path/filepath"
 	"strconv"
@@ -90,3 +91,25 @@ func credentialFor(u *user.User, workDir string) (*runAs, error) {
 func (r *runAs) env() []string {
 	return []string{"HOME=" + r.home, "XDG_CACHE_HOME=" + filepath.Join(r.home, ".cache"), "USER=" + r.name, "LOGNAME=" + r.name}
 }
+
+// AsEngineUser has cmd run as the engine user, as an engine would, with its
+// home under workDir: for work the agent does that is not an engine but
+// reads what an engine reads — training an adapter. It answers that home
+// and a way to hand a path to the user, or why it runs as this process.
+func AsEngineUser(cmd *exec.Cmd, name, workDir string) (home string, give func(path string) error, why string) {
+	r, why := resolveRunAs(name, workDir)
+	if r == nil {
+		return "", func(string) error { return nil }, why
+	}
+	cmd.SysProcAttr = &syscall.SysProcAttr{Credential: r.cred, Setpgid: true}
+	cmd.Env = append(cmd.Env, r.env()...)
+	uid, gid := int(r.cred.Uid), int(r.cred.Gid)
+	return r.home, func(path string) error {
+		return filepath.WalkDir(path, func(p string, _ os.DirEntry, err error) error {
+			if err != nil {
+				return err
+			}
+			return os.Lchown(p, uid, gid)
+		})
+	}, ""
+}
@@ -0,0 +1,574 @@
+// Package train runs LoRA training jobs on a node (BACKLOG F4): with an
+// installed engine's Python — its torch and transformers, already matched to
+// the node's CUDA — and a small overlay of what training needs besides, so a
+// node that serves can train without a second multi-gigabyte environment.
+package train
+
+import (
+	"archive/tar"
+	"bufio"
+	"bytes"
+	"compress/gzip"
+	"context"
+	"crypto/sha256"
+	_ "embed"
+	"encoding/hex"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"log/slog"
+	"net/http"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"sort"
+	"strconv"
+	"strings"
+	"sync"
+	"syscall"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
+)
+
+//go:embed trainer.py
+var trainerPy []byte
+
+// overlayPackages are what training needs besides an engine's Python, pinned
+// and installed without their dependencies: those are the engine's own, and
+// letting an installer resolve them pulled a second torch.
+var overlayPackages = []string{"peft==0.21.2", "accelerate==1.15.0"}
+
+// Job is one training job, as the control plane sent it.
+type Job struct {
+	ID           string
+	BaseRepo     string
+	BaseRevision string
+	// Data is NDJSON of {"q", "a"}.
+	Data []byte
+	// Params is the trainer's settings as JSON.
+	Params       string
+	GPUMemoryMiB int
+	GGUF         bool
+}
+
+// Status is how a job is going.
+type Status struct {
+	State   string
+	Message string
+	Step    int
+	Steps   int
+	Loss    float64
+	PeakMiB int
+}
+
+// Env is where a node keeps what training needs.
+type Env struct {
+	// EnginesDir is where engines are installed (the agent's engines_dir).
+	EnginesDir string
+	// WorkDir is the trainer's own directory: its overlay, the llama.cpp
+	// conversion code, and each job's files while it runs.
+	WorkDir string
+	// RunAs is the user jobs run as: the engines' user.
+	RunAs string
+	Log   *slog.Logger
+	// Python, UV, ConvDir and GPUs replace what would be found, for tests.
+	Python, UV, ConvDir string
+	GPUs                func(ctx context.Context) ([]GPU, error)
+}
+
+// GPU is one card and how much of it is free.
+type GPU struct {
+	Index   int
+	FreeMiB int
+}
+
+// Runner runs one job at a time.
+type Runner struct {
+	env    Env
+	mu     sync.Mutex
+	cancel context.CancelFunc
+	job    string
+}
+
+// New is a runner.
+func New(env Env) *Runner {
+	if env.Log == nil {
+		env.Log = slog.New(slog.DiscardHandler)
+	}
+	return &Runner{env: env}
+}
+
+// Cancel stops a job if it is the one running.
+func (r *Runner) Cancel(id string) {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	if r.job == id && r.cancel != nil {
+		r.cancel()
+	}
+}
+
+// ErrBusy is a node already training.
+var ErrBusy = errors.New("this node is already training")
+
+// Run runs a job to its end, telling status as it goes and artifact each
+// piece of what it made. It answers the error it failed with.
+func (r *Runner) Run(ctx context.Context, job Job, status func(Status), artifact func(kind string, seq int, data []byte, last bool) error) error {
+	r.mu.Lock()
+	if r.job != "" {
+		r.mu.Unlock()
+		return ErrBusy
+	}
+	ctx, cancel := context.WithCancel(ctx)
+	r.job, r.cancel = job.ID, cancel
+	r.mu.Unlock()
+	defer func() {
+		cancel()
+		r.mu.Lock()
+		r.job, r.cancel = "", nil
+		r.mu.Unlock()
+	}()
+
+	status(Status{State: "preparing", Message: "preparing the trainer"})
+	py, err := r.python()
+	if err != nil {
+		return err
+	}
+	overlay, err := r.overlay(ctx, py)
+	if err != nil {
+		return fmt.Errorf("installing %s: %w", strings.Join(overlayPackages, ", "), err)
+	}
+	conv := ""
+	if job.GGUF {
+		if conv, err = r.conversion(ctx); err != nil {
+			return fmt.Errorf("fetching llama.cpp's conversion code: %w", err)
+		}
+	}
+	gpu, err := r.pickGPU(ctx, job.GPUMemoryMiB)
+	if err != nil {
+		return err
+	}
+
+	dir := filepath.Join(r.env.WorkDir, "jobs", job.ID)
+	if err := os.MkdirAll(dir, 0o755); err != nil {
+		return err
+	}
+	defer os.RemoveAll(dir)
+	out := filepath.Join(dir, "adapter")
+	var params map[string]any
+	if job.Params != "" {
+		if err := json.Unmarshal([]byte(job.Params), &params); err != nil {
+			return fmt.Errorf("the job's params: %w", err)
+		}
+	}
+	spec, _ := json.Marshal(map[string]any{"base_repo": job.BaseRepo, "base_revision": job.BaseRevision,
+		"data": filepath.Join(dir, "data.jsonl"), "out": out, "params": params})
+	for name, b := range map[string][]byte{"data.jsonl": job.Data, "job.json": spec, "trainer.py": trainerPy} {
+		if err := os.WriteFile(filepath.Join(dir, name), b, 0o644); err != nil {
+			return err
+		}
+	}
+
+	var baseDir string
+	err = r.python3(ctx, py, dir, []string{"PYTHONPATH=" + overlay, "CUDA_VISIBLE_DEVICES=" + strconv.Itoa(gpu)},
+		[]string{filepath.Join(dir, "trainer.py"), filepath.Join(dir, "job.json")},
+		func(line map[string]any) {
+			s := Status{State: str(line["state"]), Message: str(line["message"]), Step: num(line["step"]),
+				Steps: num(line["steps"]), PeakMiB: num(line["peak_mib"])}
+			if v, ok := line["loss"].(float64); ok {
+				s.Loss = v
+			}
+			if b := str(line["base_dir"]); b != "" {
+				baseDir = b
+			}
+			if s.State == "trained" {
+				s.State = "running"
+				s.Message = "trained; packing the adapter"
+			}
+			status(s)
+		})
+	if err != nil {
+		return err
+	}
+
+	if job.GGUF {
+		status(Status{State: "running", Message: "converting the adapter to GGUF for llama.cpp"})
+		gguf := filepath.Join(dir, "adapter.gguf")
+		err := r.python3(ctx, py, dir, []string{"PYTHONPATH=" + strings.Join([]string{filepath.Join(conv, "gguf-py"), conv, overlay}, ":")},
+			[]string{filepath.Join(conv, "convert_lora_to_gguf.py"), "--base", baseDir, "--outtype", "f16", "--outfile", gguf, out}, nil)
+		if err != nil {
+			return fmt.Errorf("converting to GGUF: %w", err)
+		}
+		if err := sendFile(gguf, "gguf", artifact); err != nil {
+			return err
+		}
+	}
+	var tgz bytes.Buffer
+	if err := tarGz(&tgz, out); err != nil {
+		return err
+	}
+	return sendBytes(tgz.Bytes(), "peft", artifact)
+}
+
+func str(v any) string {
+	s, _ := v.(string)
+	return s
+}
+
+func num(v any) int {
+	f, _ := v.(float64)
+	return int(f)
+}
+
+// python3 runs a Python script as the engines' user, passing each JSON line
+// it prints to line, and answers the last of what it wrote to stderr when it
+// fails.
+func (r *Runner) python3(ctx context.Context, py, dir string, env, args []string, line func(map[string]any)) error {
+	cmd := exec.CommandContext(ctx, py, args...)
+	cmd.Dir = dir
+	cmd.Env = append(os.Environ(), "PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True", "TOKENIZERS_PARALLELISM=false",
+		"HF_HUB_DISABLE_PROGRESS_BARS=1", "PYTHONUNBUFFERED=1")
+	cmd.Env = append(cmd.Env, env...)
+	home, give, why := supervisor.AsEngineUser(cmd, r.env.RunAs, r.env.WorkDir)
+	if why != "" {
+		r.env.Log.Warn(why)
+	}
+	if home != "" {
+		cmd.Env = append(cmd.Env, "HF_HOME="+filepath.Join(home, ".cache", "huggingface"))
+	} else {
+		cmd.Env = append(cmd.Env, "HF_HOME="+filepath.Join(r.env.WorkDir, "hf"))
+	}
+	if err := give(dir); err != nil {
+		return fmt.Errorf("handing %s to %s: %w", dir, r.env.RunAs, err)
+	}
+	// The whole process group goes when the job is cancelled: torch starts
+	// workers of its own.
+	if cmd.SysProcAttr == nil {
+		cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
+	}
+	cmd.Cancel = func() error { return syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL) }
+	stdout, err := cmd.StdoutPipe()
+	if err != nil {
+		return err
+	}
+	var stderr tailBuffer
+	cmd.Stderr = &stderr
+	if err := cmd.Start(); err != nil {
+		return err
+	}
+	sc := bufio.NewScanner(stdout)
+	sc.Buffer(make([]byte, 64<<10), 1<<20)
+	for sc.Scan() {
+		var m map[string]any
+		if line != nil && json.Unmarshal(sc.Bytes(), &m) == nil {
+			line(m)
+		}
+	}
+	err = cmd.Wait()
+	if ctx.Err() != nil {
+		return errors.New("cancelled")
+	}
+	if err != nil {
+		return fmt.Errorf("%v: %s", err, stderr.last())
+	}
+	return nil
+}
+
+// tailBuffer keeps the end of what a process wrote: the reason it failed is
+// at the end, and the rest is progress bars.
+type tailBuffer struct{ b []byte }
+
+func (t *tailBuffer) Write(p []byte) (int, error) {
+	t.b = append(t.b, p...)
+	if len(t.b) > 16<<10 {
+		t.b = t.b[len(t.b)-16<<10:]
+	}
+	return len(p), nil
+}
+
+func (t *tailBuffer) last() string {
+	lines := strings.Split(strings.TrimSpace(string(t.b)), "\n")
+	if len(lines) > 3 {
+		lines = lines[len(lines)-3:]
+	}
+	return strings.Join(lines, " | ")
+}
+
+// python is the newest installed vLLM's interpreter: its torch is built for
+// this node's CUDA.
+func (r *Runner) python() (string, error) {
+	if r.env.Python != "" {
+		return r.env.Python, nil
+	}
+	found, _ := filepath.Glob(filepath.Join(r.env.EnginesDir, "vllm", "*", "bin", "python3"))
+	if len(found) == 0 {
+		return "", errors.New("training needs vLLM installed on this node (its torch is what trains); " +
+			"llmgw ctl engine-build, or serving any vLLM model here, installs it")
+	}
+	sort.Strings(found)
+	return found[len(found)-1], nil
+}
+
+func (r *Runner) uv() (string, error) {
+	if r.env.UV != "" {
+		return r.env.UV, nil
+	}
+	var found []string
+	_ = filepath.WalkDir(filepath.Join(r.env.EnginesDir, "uv"), func(p string, d os.DirEntry, err error) error {
+		if err == nil && !d.IsDir() && d.Name() == "uv" {
+			found = append(found, p)
+		}
+		return nil
+	})
+	if len(found) == 0 {
+		return "", errors.New("no uv under " + filepath.Join(r.env.EnginesDir, "uv"))
+	}
+	sort.Strings(found)
+	return found[len(found)-1], nil
+}
+
+// overlay installs the pinned packages once per Python and set of pins.
+func (r *Runner) overlay(ctx context.Context, py string) (string, error) {
+	sum := sha256.Sum256([]byte(py + "\x00" + strings.Join(overlayPackages, " ")))
+	dir := filepath.Join(r.env.WorkDir, "overlay-"+hex.EncodeToString(sum[:6]))
+	if _, err := os.Stat(filepath.Join(dir, ".done")); err == nil {
+		return dir, nil
+	}
+	uv, err := r.uv()
+	if err != nil {
+		return "", err
+	}
+	tmp := dir + ".new"
+	_ = os.RemoveAll(tmp)
+	args := append([]string{"pip", "install", "--quiet", "--python", py, "--no-deps", "--target", tmp}, overlayPackages...)
+	cmd := exec.CommandContext(ctx, uv, args...)
+	if out, err := cmd.CombinedOutput(); err != nil {
+		return "", fmt.Errorf("%v: %s", err, strings.TrimSpace(string(out)))
+	}
+	_ = os.WriteFile(filepath.Join(tmp, ".done"), nil, 0o644)
+	_ = os.RemoveAll(dir)
+	if err := os.Rename(tmp, dir); err != nil {
+		return "", err
+	}
+	// Readable by the engines' user, which runs the trainer.
+	_ = filepath.WalkDir(dir, func(p string, d os.DirEntry, err error) error {
+		if err == nil {
+			mode := os.FileMode(0o644)
+			if d.IsDir() {
+				mode = 0o755
+			}
+			_ = os.Chmod(p, mode)
+		}
+		return nil
+	})
+	return dir, nil
+}
+
+// llamaCppTag is the release of the newest llama.cpp installed: its
+// conversion code is the one whose GGUF that llama.cpp reads.
+func (r *Runner) llamaCppTag() (string, error) {
+	found, _ := filepath.Glob(filepath.Join(r.env.EnginesDir, "llamacpp", "b*"))
+	if len(found) == 0 {
+		return "", errors.New("a GGUF adapter needs llama.cpp installed on this node, to match its release")
+	}
+	sort.Strings(found)
+	tag, _, _ := strings.Cut(filepath.Base(found[len(found)-1]), "-")
+	return tag, nil
+}
+
+// conversionSource is where llama.cpp's source is fetched from, by tag.
+var conversionSource = "https://codeload.github.com/ggml-org/llama.cpp/tar.gz/refs/tags/"
+
+// conversion fetches llama.cpp's LoRA conversion code for the installed
+// release, once.
+func (r *Runner) conversion(ctx context.Context) (string, error) {
+	if r.env.ConvDir != "" {
+		return r.env.ConvDir, nil
+	}
+	tag, err := r.llamaCppTag()
+	if err != nil {
+		return "", err
+	}
+	dir := filepath.Join(r.env.WorkDir, "llama.cpp-"+tag)
+	if _, err := os.Stat(filepath.Join(dir, "convert_lora_to_gguf.py")); err == nil {
+		return dir, nil
+	}
+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, conversionSource+tag, nil)
+	if err != nil {
+		return "", err
+	}
+	res, err := http.DefaultClient.Do(req)
+	if err != nil {
+		return "", err
+	}
+	defer res.Body.Close()
+	if res.StatusCode != http.StatusOK {
+		return "", fmt.Errorf("%s answered %s", conversionSource+tag, res.Status)
+	}
+	tmp := dir + ".new"
+	_ = os.RemoveAll(tmp)
+	if err := untarConversion(res.Body, tmp); err != nil {
+		return "", err
+	}
+	_ = os.RemoveAll(dir)
+	return dir, os.Rename(tmp, dir)
+}
+
+// untarConversion keeps, of llama.cpp's source, the conversion scripts and
+// the Python packages they import.
+func untarConversion(r io.Reader, dir string) error {
+	gz, err := gzip.NewReader(r)
+	if err != nil {
+		return err
+	}
+	tr := tar.NewReader(gz)
+	kept := 0
+	for {
+		h, err := tr.Next()
+		if errors.Is(err, io.EOF) {
+			break
+		}
+		if err != nil {
+			return err
+		}
+		_, rel, ok := strings.Cut(h.Name, "/")
+		if !ok || h.Typeflag != tar.TypeReg || strings.Contains(rel, "..") {
+			continue
+		}
+		if rel != "convert_lora_to_gguf.py" && rel != "convert_hf_to_gguf.py" &&
+			!strings.HasPrefix(rel, "conversion/") && !strings.HasPrefix(rel, "gguf-py/") {
+			continue
+		}
+		p := filepath.Join(dir, filepath.FromSlash(rel))
+		if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
+			return err
+		}
+		f, err := os.OpenFile(p, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o644)
+		if err != nil {
+			return err
+		}
+		_, err = io.Copy(f, io.LimitReader(tr, 64<<20))
+		f.Close()
+		if err != nil {
+			return err
+		}
+		kept++
+	}
+	if kept == 0 {
+		return errors.New("no conversion code in llama.cpp's source")
+	}
+	return nil
+}
+
+// pickGPU is the card with the most memory free, if that is enough.
+func (r *Runner) pickGPU(ctx context.Context, need int) (int, error) {
+	list := r.env.GPUs
+	if list == nil {
+		list = nvidiaSMI
+	}
+	gpus, err := list(ctx)
+	if err != nil {
+		return 0, fmt.Errorf("reading the cards' free memory: %w", err)
+	}
+	if len(gpus) == 0 {
+		return 0, errors.New("this node has no GPU to train on")
+	}
+	best := gpus[0]
+	for _, g := range gpus[1:] {
+		if g.FreeMiB > best.FreeMiB {
+			best = g
+		}
+	}
+	if best.FreeMiB < need {
+		return 0, fmt.Errorf("no card here has %d MiB free (the most free is GPU %d with %d MiB); "+
+			"a smaller base, or a card freed for it, would do", need, best.Index, best.FreeMiB)
+	}
+	return best.Index, nil
+}
+
+func nvidiaSMI(ctx context.Context) ([]GPU, error) {
+	ctx, cancel := context.WithTimeout(ctx, 15*time.Second)
+	defer cancel()
+	out, err := exec.CommandContext(ctx, "nvidia-smi", "--query-gpu=index,memory.free", "--format=csv,noheader,nounits").Output()
+	if err != nil {
+		return nil, err
+	}
+	return parseGPUs(string(out)), nil
+}
+
+func parseGPUs(s string) []GPU {
+	var out []GPU
+	for _, line := range strings.Split(strings.TrimSpace(s), "\n") {
+		a, b, ok := strings.Cut(line, ",")
+		if !ok {
+			continue
+		}
+		i, err1 := strconv.Atoi(strings.TrimSpace(a))
+		f, err2 := strconv.Atoi(strings.TrimSpace(b))
+		if err1 == nil && err2 == nil {
+			out = append(out, GPU{Index: i, FreeMiB: f})
+		}
+	}
+	return out
+}
+
+// tarGz packs a directory.
+func tarGz(w io.Writer, dir string) error {
+	gz := gzip.NewWriter(w)
+	tw := tar.NewWriter(gz)
+	err := filepath.WalkDir(dir, func(p string, d os.DirEntry, err error) error {
+		if err != nil || d.IsDir() {
+			return err
+		}
+		rel, _ := filepath.Rel(dir, p)
+		info, err := d.Info()
+		if err != nil {
+			return err
+		}
+		if err := tw.WriteHeader(&tar.Header{Name: filepath.ToSlash(rel), Mode: 0o644, Size: info.Size(), ModTime: info.ModTime()}); err != nil {
+			return err
+		}
+		f, err := os.Open(p)
+		if err != nil {
+			return err
+		}
+		defer f.Close()
+		_, err = io.Copy(tw, f)
+		return err
+	})
+	if err != nil {
+		return err
+	}
+	if err := tw.Close(); err != nil {
+		return err
+	}
+	return gz.Close()
+}
+
+// artifactChunk is how much of an artifact goes in one message.
+const artifactChunk = 1 << 20
+
+func sendFile(path, kind string, send func(string, int, []byte, bool) error) error {
+	b, err := os.ReadFile(path)
+	if err != nil {
+		return err
+	}
+	return sendBytes(b, kind, send)
+}
+
+func sendBytes(b []byte, kind string, send func(string, int, []byte, bool) error) error {
+	seq := 0
+	for {
+		n := min(artifactChunk, len(b))
+		last := n == len(b)
+		if err := send(kind, seq, b[:n], last); err != nil {
+			return err
+		}
+		if last {
+			return nil
+		}
+		b = b[n:]
+		seq++
+	}
+}
@@ -0,0 +1,164 @@
+package train
+
+import (
+	"archive/tar"
+	"bytes"
+	"compress/gzip"
+	"context"
+	"io"
+	"os"
+	"path/filepath"
+	"strings"
+	"sync"
+	"testing"
+)
+
+// fakePython stands in for an engine's interpreter: as the trainer it
+// prints progress and writes an adapter; as the converter it writes a GGUF;
+// told to fail, it fails.
+const fakePython = `#!/bin/sh
+case "$1" in
+*trainer.py)
+  out=$(sed -n 's/.*"out": *"\([^"]*\)".*/\1/p' "$2")
+  grep -q FAIL "$(dirname "$2")/data.jsonl" && { echo "CUDA out of memory" >&2; exit 1; }
+  echo '{"state":"preparing","message":"fetching base"}'
+  echo '{"state":"running","step":5,"steps":10,"loss":0.9,"peak_mib":1900}'
+  mkdir -p "$out"; echo '{"r":16}' > "$out/adapter_config.json"; echo weights > "$out/adapter_model.safetensors"
+  echo '{"state":"trained","step":10,"steps":10,"peak_mib":2100,"base_dir":"/base"}'
+  ;;
+*convert_lora_to_gguf.py)
+  while [ $# -gt 0 ]; do [ "$1" = "--outfile" ] && echo GGUF > "$2"; shift; done
+  ;;
+esac
+`
+
+func runner(t *testing.T, free int) *Runner {
+	t.Helper()
+	dir := t.TempDir()
+	py := filepath.Join(dir, "python3")
+	_ = os.WriteFile(py, []byte(fakePython), 0o755)
+	uv := filepath.Join(dir, "uv")
+	_ = os.WriteFile(uv, []byte("#!/bin/sh\nwhile [ $# -gt 0 ]; do [ \"$1\" = \"--target\" ] && mkdir -p \"$2\"; shift; done\n"), 0o755)
+	return New(Env{WorkDir: filepath.Join(dir, "work"), Python: py, UV: uv, ConvDir: dir,
+		GPUs: func(context.Context) ([]GPU, error) { return []GPU{{0, 500}, {1, free}}, nil }})
+}
+
+type sink struct {
+	mu       sync.Mutex
+	statuses []Status
+	parts    map[string][]byte
+	lasts    int
+}
+
+func (s *sink) status(st Status) { s.mu.Lock(); s.statuses = append(s.statuses, st); s.mu.Unlock() }
+func (s *sink) artifact(kind string, _ int, b []byte, last bool) error {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	if s.parts == nil {
+		s.parts = map[string][]byte{}
+	}
+	s.parts[kind] = append(s.parts[kind], b...)
+	if last {
+		s.lasts++
+	}
+	return nil
+}
+
+func TestAJobTrainsOnTheFreestCardAndSendsWhatItMade(t *testing.T) {
+	r := runner(t, 3000)
+	s := &sink{}
+	err := r.Run(context.Background(), Job{ID: "j1", BaseRepo: "Qwen/x", Data: []byte(`{"q":"a","a":"b"}`), GPUMemoryMiB: 2500, GGUF: true},
+		s.status, s.artifact)
+	if err != nil {
+		t.Fatal(err)
+	}
+	var running bool
+	for _, st := range s.statuses {
+		if st.State == "running" && st.Step == 5 && st.Steps == 10 && st.PeakMiB == 1900 {
+			running = true
+		}
+	}
+	if !running {
+		t.Errorf("statuses %+v", s.statuses)
+	}
+	if string(s.parts["gguf"]) != "GGUF\n" || s.lasts != 2 {
+		t.Errorf("gguf %q, lasts %d", s.parts["gguf"], s.lasts)
+	}
+	files := map[string]string{}
+	gz, err := gzip.NewReader(bytes.NewReader(s.parts["peft"]))
+	if err != nil {
+		t.Fatal(err)
+	}
+	tr := tar.NewReader(gz)
+	for {
+		h, err := tr.Next()
+		if err != nil {
+			break
+		}
+		b, _ := io.ReadAll(tr)
+		files[h.Name] = string(b)
+	}
+	if files["adapter_model.safetensors"] != "weights\n" || files["adapter_config.json"] == "" {
+		t.Errorf("peft archive %v", files)
+	}
+	// Nothing of the job is left on the node.
+	if entries, _ := os.ReadDir(filepath.Join(r.env.WorkDir, "jobs")); len(entries) != 0 {
+		t.Errorf("left behind: %v", entries)
+	}
+}
+
+func TestAJobWaitsForNoCardItDoesNotFit(t *testing.T) {
+	r := runner(t, 1000)
+	err := r.Run(context.Background(), Job{ID: "j1", Data: []byte(`{"q":"a","a":"b"}`), GPUMemoryMiB: 2500}, (&sink{}).status, (&sink{}).artifact)
+	if err == nil || !strings.Contains(err.Error(), "no card here has 2500 MiB free") {
+		t.Errorf("got %v", err)
+	}
+}
+
+func TestATrainerThatFailsSaysWhy(t *testing.T) {
+	r := runner(t, 3000)
+	err := r.Run(context.Background(), Job{ID: "j1", Data: []byte(`{"q":"FAIL","a":"b"}`)}, (&sink{}).status, (&sink{}).artifact)
+	if err == nil || !strings.Contains(err.Error(), "CUDA out of memory") {
+		t.Errorf("got %v", err)
+	}
+}
+
+func TestOneJobAtATime(t *testing.T) {
+	r := runner(t, 3000)
+	r.job = "other"
+	if err := r.Run(context.Background(), Job{ID: "j2"}, (&sink{}).status, (&sink{}).artifact); err != ErrBusy {
+		t.Errorf("got %v", err)
+	}
+}
+
+func TestCardsAreReadFromNvidiaSMI(t *testing.T) {
+	got := parseGPUs("0, 2310\n1, 80000\nbad line\n")
+	if len(got) != 2 || got[1].FreeMiB != 80000 || got[0].Index != 0 {
+		t.Errorf("parsed %+v", got)
+	}
+}
+
+func TestOnlyTheConversionCodeIsKeptOfLlamaCpp(t *testing.T) {
+	var buf bytes.Buffer
+	gz := gzip.NewWriter(&buf)
+	tw := tar.NewWriter(gz)
+	for _, name := range []string{"llama.cpp-b1/convert_lora_to_gguf.py", "llama.cpp-b1/conversion/qwen.py",
+		"llama.cpp-b1/gguf-py/gguf/__init__.py", "llama.cpp-b1/src/llama.cpp", "llama.cpp-b1/../evil"} {
+		_ = tw.WriteHeader(&tar.Header{Name: name, Mode: 0o644, Size: 1, Typeflag: tar.TypeReg})
+		_, _ = tw.Write([]byte("x"))
+	}
+	_ = tw.Close()
+	_ = gz.Close()
+	dir := filepath.Join(t.TempDir(), "conv")
+	if err := untarConversion(&buf, dir); err != nil {
+		t.Fatal(err)
+	}
+	for _, want := range []string{"convert_lora_to_gguf.py", "conversion/qwen.py", "gguf-py/gguf/__init__.py"} {
+		if _, err := os.Stat(filepath.Join(dir, want)); err != nil {
+			t.Errorf("%s not kept", want)
+		}
+	}
+	if _, err := os.Stat(filepath.Join(dir, "src")); err == nil {
+		t.Error("the rest of llama.cpp was kept")
+	}
+}
@@ -0,0 +1,101 @@
+# llmgw's LoRA trainer (BACKLOG F4), run by the node agent with an installed
+# engine's Python (its torch and transformers) and a small overlay (peft,
+# accelerate). Reads a job, trains, writes the PEFT adapter, and says how it
+# goes as one JSON object a line on standard output.
+import json, math, os, random, sys, threading, subprocess, time
+
+def say(**kw):
+    print(json.dumps(kw), flush=True)
+
+job = json.load(open(sys.argv[1], encoding="utf-8"))
+p = {"rank": 16, "alpha": 32, "dropout": 0.05, "epochs": 6, "lr": 2e-4, "batch": 4, "max_len": 512, "seed": 0}
+p.update(job.get("params") or {})
+
+say(state="preparing", message="loading libraries")
+import torch
+from huggingface_hub import snapshot_download
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from peft import LoraConfig, get_peft_model
+
+random.seed(p["seed"]); torch.manual_seed(p["seed"])
+if not torch.cuda.is_available():
+    raise SystemExit("no CUDA device is visible to the trainer")
+
+say(state="preparing", message="fetching " + job["base_repo"])
+base_dir = snapshot_download(job["base_repo"], revision=job.get("base_revision") or None,
+                             allow_patterns=["*.json", "*.safetensors", "*.txt", "*.model", "*.tiktoken"])
+say(state="preparing", message="loading the base model", base_dir=base_dir)
+
+# bf16 where the card has it; fp16 with a loss scaler where it does not
+# (Turing and older).
+bf16 = torch.cuda.is_bf16_supported()
+dtype = torch.bfloat16 if bf16 else torch.float16
+tok = AutoTokenizer.from_pretrained(base_dir)
+if tok.pad_token_id is None:
+    tok.pad_token = tok.eos_token
+model = AutoModelForCausalLM.from_pretrained(base_dir, dtype=dtype).to("cuda")
+
+rows = [json.loads(l) for l in open(job["data"], encoding="utf-8") if l.strip()]
+rows = [r for r in rows if r.get("q") and r.get("a")]
+if not rows:
+    raise SystemExit("no {q, a} pairs in the data")
+
+def encode(r):
+    prompt = tok.apply_chat_template([{"role": "user", "content": r["q"]}], add_generation_prompt=True, tokenize=False)
+    full = tok.apply_chat_template([{"role": "user", "content": r["q"]}, {"role": "assistant", "content": r["a"]}], tokenize=False)
+    pi = tok(prompt, add_special_tokens=False)["input_ids"]
+    fi = tok(full, add_special_tokens=False)["input_ids"][: p["max_len"]]
+    return fi, ([-100] * len(pi) + fi[len(pi):])[: len(fi)]
+
+data = [encode(r) for r in rows]
+
+model.gradient_checkpointing_enable()
+model.enable_input_require_grads()
+model = get_peft_model(model, LoraConfig(r=int(p["rank"]), lora_alpha=int(p["alpha"]), lora_dropout=float(p["dropout"]),
+    task_type="CAUSAL_LM", target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]))
+trainable = [x for x in model.parameters() if x.requires_grad]
+for x in trainable:  # fp32 trainable weights, so a loss scaler can unscale them
+    x.data = x.data.float()
+
+peak = {"mib": 0}
+stop = threading.Event()
+def watch():
+    while not stop.is_set():
+        try:
+            out = subprocess.run(["nvidia-smi", "--query-compute-apps=pid,used_memory", "--format=csv,noheader,nounits"],
+                                 capture_output=True, text=True, timeout=5).stdout
+            for line in out.splitlines():
+                pid, mib = [v.strip() for v in line.split(",")]
+                if int(pid) == os.getpid():
+                    peak["mib"] = max(peak["mib"], int(mib))
+        except Exception:
+            pass
+        time.sleep(0.5)
+threading.Thread(target=watch, daemon=True).start()
+
+opt = torch.optim.AdamW(trainable, lr=float(p["lr"]), weight_decay=0.0)
+scaler = torch.amp.GradScaler("cuda", enabled=not bf16)
+bs, epochs = int(p["batch"]), int(p["epochs"])
+steps = epochs * math.ceil(len(data) / bs)
+warm = max(1, steps // 10)
+sched = torch.optim.lr_scheduler.LambdaLR(opt, lambda s: min(1.0, (s + 1) / warm) * max(0.0, 1 - s / steps))
+model.train(); step = 0; last = time.time()
+for ep in range(epochs):
+    random.shuffle(data)
+    for i in range(0, len(data), bs):
+        batch = data[i:i + bs]; n = max(len(x[0]) for x in batch)
+        ids = torch.tensor([x[0] + [tok.pad_token_id] * (n - len(x[0])) for x in batch], device="cuda")
+        lab = torch.tensor([x[1] + [-100] * (n - len(x[1])) for x in batch], device="cuda")
+        att = torch.tensor([[1] * len(x[0]) + [0] * (n - len(x[0])) for x in batch], device="cuda")
+        with torch.autocast("cuda", dtype=dtype):
+            loss = model(input_ids=ids, attention_mask=att, labels=lab).loss
+        scaler.scale(loss).backward(); scaler.unscale_(opt)
+        torch.nn.utils.clip_grad_norm_(trainable, 1.0)
+        scaler.step(opt); scaler.update(); opt.zero_grad(set_to_none=True); sched.step(); step += 1
+        if time.time() - last > 2 or step == steps:
+            last = time.time()
+            say(state="running", message="training", step=step, steps=steps, loss=round(loss.item(), 4), peak_mib=peak["mib"])
+
+model.save_pretrained(job["out"])
+stop.set()
+say(state="trained", step=step, steps=steps, peak_mib=peak["mib"], base_dir=base_dir)
@@ -0,0 +1,46 @@
+package agent
+
+import (
+	"context"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/train"
+)
+
+// trainJob runs a training job the control plane sent, telling it how the
+// job goes and sending what it made. A stream that ends ends the job: the
+// control plane, which sees the node go, marks it failed, and it is started
+// again from the beginning rather than resumed.
+func (a *Agent) trainJob(ctx context.Context, m *llmgwv1.TrainStart, out chan<- *llmgwv1.AgentMessage) {
+	id := m.GetJobId()
+	send := func(msg *llmgwv1.AgentMessage) bool {
+		select {
+		case out <- msg:
+			return true
+		case <-ctx.Done():
+			return false
+		}
+	}
+	status := func(s train.Status) {
+		send(&llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_TrainStatus{TrainStatus: &llmgwv1.TrainStatus{
+			JobId: id, State: s.State, Message: s.Message, Step: int32(s.Step), Steps: int32(s.Steps), //nolint:gosec // counts
+			Loss: s.Loss, PeakMib: int32(s.PeakMiB)}}}) //nolint:gosec // MiB
+	}
+	artifact := func(kind string, seq int, data []byte, last bool) error {
+		if !send(&llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_TrainArtifact{TrainArtifact: &llmgwv1.TrainArtifact{
+			JobId: id, Kind: kind, Seq: int32(seq), Data: data, Last: last}}}) { //nolint:gosec // chunk count
+			return ctx.Err()
+		}
+		return nil
+	}
+	a.log.Info("training an adapter", "job", id, "base", m.GetBaseRepo())
+	err := a.trainer.Run(ctx, train.Job{ID: id, BaseRepo: m.GetBaseRepo(), BaseRevision: m.GetBaseRevision(),
+		Data: m.GetData(), Params: m.GetParams(), GPUMemoryMiB: int(m.GetGpuMemoryMib()), GGUF: m.GetGguf()}, status, artifact)
+	if err != nil {
+		a.log.Warn("training failed", "job", id, "err", err)
+		status(train.Status{State: "failed", Message: err.Error()})
+		return
+	}
+	a.log.Info("trained an adapter", "job", id)
+	status(train.Status{State: "done"})
+}
@@ -40,10 +40,23 @@ func newAgentCmd() *cobra.Command {
 		RunE: func(cmd *cobra.Command, _ []string) error {
 			log := logger()

-			cfg, err := config.LoadAgent(configPath)
+			// The cluster's settings for this agent over its file, as the
+			// control plane last sent them. Settings that no longer load
+			// beside the file are set aside rather than keeping the agent
+			// from starting: the control plane sends them again, and says
+			// why they are refused.
+			settings := agent.ReadSettings()
+			cfg, err := config.LoadAgentWith(configPath, settings)
+			if err != nil && settings != nil {
+				log.Warn("the cluster's settings kept for this agent do not load beside its agent.yaml; starting without them",
+					"err", err)
+				settings = nil
+				cfg, err = config.LoadAgent(configPath)
+			}
 			if err != nil {
 				return err
 			}
+			cfg.Settings = string(settings)
 			if server != "" {
 				cfg.Server = server
 			}
@@ -89,9 +102,17 @@ func newAgentCmd() *cobra.Command {
 			if err != nil {
 				return err
 			}
+			a.SetSettingsCheck(func(b []byte) error {
+				_, err := config.LoadAgentWith(configPath, b)
+				return err
+			})

 			ctx, stop := signal.NotifyContext(cmd.Context(), syscall.SIGINT, syscall.SIGTERM)
 			defer stop()
+			go a.WatchFile(ctx, configPath, func() error {
+				_, err := config.LoadAgentWith(configPath, agent.ReadSettings())
+				return err
+			})

 			log.Info("agent starting",
 				"node", cfg.NodeID, "server", cfg.Server,
@@ -249,7 +249,14 @@ func pgEnv(dsn string) ([]string, error) {
 		set("PGPASSWORD", pw)
 	}
 	set("PGDATABASE", strings.TrimPrefix(u.Path, "/"))
-	set("PGSSLMODE", u.Query().Get("sslmode"))
+	// The TLS settings go with the mode: verify-full without the cluster's
+	// authority sent libpq looking for ~/.postgresql/root.crt, and the dump
+	// failed on every control plane whose store is replicated.
+	q := u.Query()
+	set("PGSSLMODE", q.Get("sslmode"))
+	set("PGSSLROOTCERT", q.Get("sslrootcert"))
+	set("PGSSLCERT", q.Get("sslcert"))
+	set("PGSSLKEY", q.Get("sslkey"))
 	// A standby if there is one, else whichever answers — not the address's
 	// read-write: a dump only reads, it need not load the primary, and one
 	// that insists on the primary fails exactly while the cluster is failing
@@ -238,3 +238,17 @@ func TestAStoreOfSeveralHostsIsGivenToPgDumpAsLibpqLists(t *testing.T) {
 		t.Error("an empty host in the list was accepted")
 	}
 }
+
+func TestPgDumpIsGivenTheClusterAuthorityAlongWithVerifyFull(t *testing.T) {
+	// Without it libpq looked for ~/.postgresql/root.crt and the dump failed.
+	env, err := pgEnv("postgres://llmgw:p@10.0.0.149:5432,10.0.0.150:5432/llmgw?sslmode=verify-full&sslrootcert=%2Fetc%2Fllmgw%2Fpki%2Fca.crt&target_session_attrs=read-write")
+	if err != nil {
+		t.Fatal(err)
+	}
+	got := strings.Join(env, " ")
+	for _, want := range []string{"PGSSLMODE=verify-full", "PGSSLROOTCERT=/etc/llmgw/pki/ca.crt"} {
+		if !strings.Contains(got, want) {
+			t.Errorf("%s is not in %s", want, got)
+		}
+	}
+}
@@ -241,6 +241,8 @@ var liveSettings = []string{
 	"serving",
 	// Sent to the nodes and this machine's firewall unit as they change.
 	"firewall", "firewall_networks",
+	// Sent to the nodes' agents as they change.
+	"agents",
 	// Read by server init, node add and backups, not by the server.
 	"backup_recipients", "backup_copy_to",
 }
@@ -116,6 +116,11 @@ func (c *ctlClient) do(ctx context.Context, method, path string, body, out any)
 	if out == nil {
 		return nil
 	}
+	// A stream, as it comes: NDJSON a long run writes line by line.
+	if w, ok := out.(io.Writer); ok {
+		_, err := io.Copy(w, resp.Body)
+		return err
+	}
 	// An older control plane answers a path it does not know with its
 	// dashboard: a page, which read as JSON said only "invalid character '<'".
 	if strings.HasPrefix(resp.Header.Get("Content-Type"), "text/html") {
@@ -181,6 +186,7 @@ func newCtlCmd() *cobra.Command {
 		newQuotaCmd(client, &asJSON),
 		newGuardCmd(client, &asJSON),
 		newKBCmd(client, &asJSON),
+		newTrainCmd(client, &asJSON),
 		newConfigCmd(client, &asJSON),
 		newAuditCmd(client, &asJSON),
 		newNotifyCmd(client, &asJSON),
@@ -7,12 +7,14 @@ import (
 	"fmt"
 	"net/netip"
 	"os"
+	"strings"
 	"time"

 	"github.com/spf13/cobra"

 	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
 )

 // firewallStateFile is where a control plane writes what the cluster's
@@ -20,11 +22,9 @@ import (
 // the server itself runs as llmgw and cannot change the ruleset.
 const firewallStateFile = "/var/lib/llmgw/firewall.json"

-// firewallState is what is written there.
-type firewallState struct {
-	Enabled  bool     `json:"enabled"`
-	Networks []string `json:"networks,omitempty"`
-}
+// firewallState is what is written there: the same as the mirror is told
+// and keeps beside its models.
+type firewallState = mirror.FirewallState

 // writeFirewallState records the settings' firewall for this machine's sync
 // unit. Without the state directory — a server not run by systemd — there
@@ -46,25 +46,34 @@ func writeFirewallState(cfg config.Server) error {

 // The units that apply it: a path unit watching the file, and the oneshot it
 // starts.
-const (
-	firewallSyncPathUnit = "llmgw-firewall-sync.path"
-	firewallSyncUnitFile = `[Unit]
-Description=llmgw firewall: apply the cluster's settings on this control plane
+const firewallSyncPathUnit = "llmgw-firewall-sync.path"
+
+// firewallSyncUnits are the two units for a state file: on a control plane
+// the one its server writes, on a mirror the one the control plane has the
+// mirror write.
+func firewallSyncUnits(state string) (service, pathUnit string) {
+	arg := ""
+	if state != firewallStateFile {
+		arg = " --state " + state
+	}
+	service = `[Unit]
+Description=llmgw firewall: apply the cluster's settings on this machine

 [Service]
 Type=oneshot
-ExecStart=/usr/local/bin/llmgw firewall sync
+ExecStart=/usr/local/bin/llmgw firewall sync` + arg + `
 `
-	firewallSyncPathFile = `[Unit]
-Description=llmgw firewall: follow the cluster's settings on this control plane
+	pathUnit = `[Unit]
+Description=llmgw firewall: follow the cluster's settings on this machine

 [Path]
-PathChanged=` + firewallStateFile + `
+PathChanged=` + state + `

 [Install]
 WantedBy=multi-user.target
 `
-)
+	return service, pathUnit
+}

 func newFirewallCmd() *cobra.Command {
 	c := &cobra.Command{
@@ -76,6 +85,7 @@ func newFirewallCmd() *cobra.Command {
 			"which " + firewallSyncPathUnit + " applies with 'llmgw firewall sync'.",
 	}
 	var install bool
+	state := firewallStateFile
 	sync := &cobra.Command{
 		Use:   "sync",
 		Short: "Apply what the cluster's settings say of the firewall, here (root)",
@@ -85,21 +95,21 @@ func newFirewallCmd() *cobra.Command {
 				return errors.New("the firewall is changed as root: run it with sudo")
 			}
 			if install {
-				if err := installFirewallSync(cmd.Context(), firewall.Local{}); err != nil {
+				if err := installFirewallSync(cmd.Context(), firewall.Local{}, state); err != nil {
 					return err
 				}
 				fmt.Fprintln(cmd.OutOrStdout(), firewallSyncPathUnit+" follows the cluster's settings here")
-				if _, err := os.Stat(firewallStateFile); err != nil {
-					return nil // the server writes it when it starts
+				if _, err := os.Stat(state); err != nil {
+					return nil // written when the settings are first told
 				}
 			}
-			b, err := os.ReadFile(firewallStateFile)
+			b, err := os.ReadFile(state)
 			if err != nil {
 				return err
 			}
 			var st firewallState
 			if err := json.Unmarshal(b, &st); err != nil {
-				return fmt.Errorf("%s: %w", firewallStateFile, err)
+				return fmt.Errorf("%s: %w", state, err)
 			}
 			ctx, cancel := context.WithTimeout(cmd.Context(), 2*time.Minute)
 			defer cancel()
@@ -112,14 +122,17 @@ func newFirewallCmd() *cobra.Command {
 		},
 	}
 	sync.Flags().BoolVar(&install, "install", false, "first install the units that run this whenever the settings change")
+	sync.Flags().StringVar(&state, "state", firewallStateFile,
+		"the file the settings are written to: a control plane's, or "+mirror.FirewallFile+" in a mirror's directory")
 	c.AddCommand(sync)
 	return c
 }

 // installFirewallSync writes the sync units on a machine and has the path
 // unit follow the state file.
-func installFirewallSync(ctx context.Context, r firewall.Runner) error {
-	body := firewallSyncUnitFile + "\x00" + firewallSyncPathFile
+func installFirewallSync(ctx context.Context, r firewall.Runner, state string) error {
+	service, pathUnit := firewallSyncUnits(state)
+	body := service + "\x00" + pathUnit
 	out, err := r.Run(ctx, `set -e
 t=$(mktemp -d); trap 'rm -rf "$t"' EXIT
 cat > "$t/all"
@@ -172,5 +185,20 @@ func syncFirewall(ctx context.Context, r firewall.Runner, st firewallState) (str
 			return "", err
 		}
 	}
+	// The cluster's machines: a node on a network this one has not let in
+	// yet, which the mirror's files are for.
+	if len(st.Toward) > 0 {
+		added, err := firewall.Admit(ctx, r, st.Toward)
+		if err != nil {
+			return "", err
+		}
+		if len(added) > 0 {
+			shown := make([]string, len(added))
+			for i, p := range added {
+				shown[i] = p.String()
+			}
+			return "llmgw's firewall is on here, and now lets in " + strings.Join(shown, ", "), nil
+		}
+	}
 	return "llmgw's firewall is on here", nil
 }
@@ -2,8 +2,11 @@ package cli

 import (
 	"context"
+	"net/netip"
 	"strings"
 	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
 )

 type scriptLog struct {
@@ -49,3 +52,32 @@ func TestAControlPlaneFollowsTheFirewallSetting(t *testing.T) {
 		t.Errorf("on: %q %v", msg, err)
 	}
 }
+
+func TestAMachineLetsInTheClusterMachinesItIsTold(t *testing.T) {
+	// The rules hold 10.0.0.0/24; a node on 192.168.7.0/24 is let in, as
+	// the route to it says, and one already in is not asked about.
+	r := &scriptLog{reply: func(s string) string {
+		switch {
+		case strings.Contains(s, "echo loaded"):
+			return "loaded\n"
+		case strings.Contains(s, "cat /etc/llmgw/firewall.nft"):
+			return firewallRulesFor("10.0.0.0/24")
+		case strings.Contains(s, "ip -o route get 192.168.7.20"):
+			return "192.168.7.20 dev eth1 src 192.168.7.1 uid 0\n2: eth1    inet 192.168.7.1/24 brd 192.168.7.255 scope global eth1\n"
+		case strings.Contains(s, "nft -f"):
+			return "changed\n"
+		}
+		return ""
+	}}
+	msg, err := syncFirewall(context.Background(), r, firewallState{Enabled: true, Toward: []string{"10.0.0.147", "192.168.7.20"}})
+	if err != nil || !strings.Contains(msg, "192.168.7.0/24") {
+		t.Fatalf("%q %v", msg, err)
+	}
+	if strings.Contains(strings.Join(r.scripts, "\n"), "route get 10.0.0.147") {
+		t.Error("asked the route to a machine already let in")
+	}
+}
+
+func firewallRulesFor(nets string) string {
+	return firewall.Script(firewall.Spec{Ports: []firewall.Port{firewall.MirrorFiles}, Allow: []netip.Prefix{netip.MustParsePrefix(nets)}})
+}
@@ -0,0 +1,146 @@
+package cli
+
+import (
+	"context"
+	"encoding/json"
+	"log/slog"
+	"net"
+	"net/netip"
+	"net/url"
+	"slices"
+	"sync"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// firewallStateKey is where, in the store, the firewall's state is kept
+// for the store's members to read.
+const firewallStateKey = "firewall"
+
+// firewallPublisher tells the machines that cannot ask a control plane
+// what the cluster's settings say of their firewall: the mirror, over its
+// API, and the store's members, through the store they hold a copy of.
+// Nodes are told over their agents' streams and control planes by their
+// own servers. On the leader only, at start, at every change to the
+// firewall's settings, and every few minutes for nodes come and gone.
+type firewallPublisher struct {
+	settings func() config.Server
+	st       *store.DB
+	nodes    func() []string
+	leader   func() bool
+	log      *slog.Logger
+
+	mu     sync.Mutex
+	mirror *mirror.Client
+	kick   chan struct{}
+}
+
+func newFirewallPublisher(settings func() config.Server, st *store.DB, nodes func() []string,
+	leader func() bool, log *slog.Logger) *firewallPublisher {
+	return &firewallPublisher{settings: settings, st: st, nodes: nodes, leader: leader, log: log,
+		kick: make(chan struct{}, 1)}
+}
+
+// setMirror has the mirror told too.
+func (p *firewallPublisher) setMirror(m *mirror.Client) {
+	p.mu.Lock()
+	p.mirror = m
+	p.mu.Unlock()
+	p.Kick()
+}
+
+// Kick has the state worked out and told again now.
+func (p *firewallPublisher) Kick() {
+	select {
+	case p.kick <- struct{}{}:
+	default:
+	}
+}
+
+func (p *firewallPublisher) run(ctx context.Context, every time.Duration) {
+	t := time.NewTicker(every)
+	defer t.Stop()
+	for {
+		p.once(ctx)
+		select {
+		case <-ctx.Done():
+			return
+		case <-t.C:
+		case <-p.kick:
+		}
+	}
+}
+
+// state is what the settings say now, with the cluster's machines.
+func (p *firewallPublisher) state(ctx context.Context) mirror.FirewallState {
+	cfg := p.settings()
+	st := mirror.FirewallState{Enabled: cfg.Firewall.Enabled, Networks: slices.Clone(cfg.FirewallNetworks)}
+	var toward []string
+	if p.nodes != nil {
+		toward = append(toward, p.nodes()...)
+	}
+	if p.st != nil {
+		if cps, err := p.st.ControlPlanes(ctx); err == nil {
+			for _, c := range cps {
+				toward = append(toward, c.URL)
+			}
+		}
+	}
+	for _, t := range toward {
+		if a, ok := hostAddr(t); ok {
+			st.Toward = append(st.Toward, a)
+		}
+	}
+	slices.Sort(st.Toward)
+	st.Toward = slices.Compact(st.Toward)
+	return st
+}
+
+// hostAddr is the address in an address, host:port or URL; names are not
+// looked up here.
+func hostAddr(s string) (string, bool) {
+	if u, err := url.Parse(s); err == nil && u.Host != "" {
+		s = u.Hostname()
+	} else if h, _, err := net.SplitHostPort(s); err == nil {
+		s = h
+	}
+	a, err := netip.ParseAddr(s)
+	if err != nil {
+		return "", false
+	}
+	return a.Unmap().String(), true
+}
+
+func (p *firewallPublisher) once(ctx context.Context) {
+	if p.leader != nil && !p.leader() {
+		return
+	}
+	ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
+	defer cancel()
+	st := p.state(ctx)
+	p.mu.Lock()
+	m := p.mirror
+	p.mu.Unlock()
+	if m != nil {
+		changed, err := m.SetFirewall(ctx, st)
+		switch {
+		case err != nil:
+			p.log.Warn("could not tell the mirror what the settings say of its firewall", "err", err)
+		case changed:
+			p.log.Info("the mirror's firewall follows the cluster's settings", "enabled", st.Enabled)
+		}
+	}
+	if p.st != nil {
+		b, _ := json.Marshal(st)
+		changed, err := p.st.PutClusterState(ctx, firewallStateKey, string(b))
+		switch {
+		case err != nil:
+			p.log.Warn("could not record the firewall's state for the store's members", "err", err)
+		case changed:
+			p.log.Info("the store's members' firewall follows the cluster's settings", "enabled", st.Enabled)
+		}
+	}
+}
@@ -0,0 +1,59 @@
+package cli
+
+import (
+	"context"
+	"encoding/json"
+	"log/slog"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
+)
+
+func TestTheLeaderTellsTheMirrorAndTheStoreWhatTheFirewallSettingsSay(t *testing.T) {
+	ctx := context.Background()
+	st, err := store.OpenSQLite(ctx, filepath.Join(t.TempDir(), "s.db"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer st.Close()
+	if err := st.PutControlPlane(ctx, store.ControlPlane{URL: "https://10.0.0.150:7443", Name: "ctl02"}); err != nil {
+		t.Fatal(err)
+	}
+	root := t.TempDir()
+	srv := httptest.NewServer(mirror.NewAPI(mirror.NewManager(root, slog.New(slog.DiscardHandler)), "http://m:8081").Handler())
+	defer srv.Close()
+	mc, _ := mirror.NewClient(srv.URL, tlsconf.Config{}, 0)
+
+	cfg := config.Server{Firewall: config.FirewallSettings{Enabled: false}, FirewallNetworks: []string{"192.168.5.0/24"}}
+	leader := false
+	p := newFirewallPublisher(func() config.Server { return cfg }, st,
+		func() []string { return []string{"10.0.0.147", "", "10.0.0.147", "node-by-name"} },
+		func() bool { return leader }, slog.New(slog.DiscardHandler))
+	p.setMirror(mc)
+
+	p.once(ctx)
+	if _, err := st.ClusterState(ctx, firewallStateKey); err == nil {
+		t.Error("a standby told the store")
+	}
+	leader = true
+	p.once(ctx)
+	var got mirror.FirewallState
+	v, err := st.ClusterState(ctx, firewallStateKey)
+	if err != nil || json.Unmarshal([]byte(v), &got) != nil {
+		t.Fatalf("the store holds %q %v", v, err)
+	}
+	if got.Enabled || got.Networks[0] != "192.168.5.0/24" || len(got.Toward) != 2 ||
+		got.Toward[0] != "10.0.0.147" || got.Toward[1] != "10.0.0.150" {
+		t.Errorf("state %+v", got)
+	}
+	b, err := os.ReadFile(filepath.Join(root, mirror.FirewallFile))
+	if err != nil || string(b) != v {
+		t.Errorf("the mirror holds %s, the store %s", b, v)
+	}
+}
@@ -1,10 +1,14 @@
 package cli

 import (
+	"bytes"
 	"encoding/json"
 	"fmt"
+	"io"
 	"net/url"
+	"os"
 	"strings"
+	"time"

 	"github.com/spf13/cobra"

@@ -212,6 +216,45 @@ func newKBCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 	def.Flags().BoolVar(&clear, "clear", false, "no default any more")
 	c.AddCommand(def)

+	var dsModel, dsOut string
+	var dsPer, dsMax int
+	dataset := &cobra.Command{
+		Use:   "dataset COLLECTION --model MODEL",
+		Short: "Write training data from a collection: questions and answers, for an adapter",
+		Long: "A chat model of the cluster reads each chunk of the collection and writes\n" +
+			"questions a person might ask of it, with the answers it holds, one JSON object\n" +
+			"a line ({\"q\", \"a\", \"doc\", \"ord\"}). Read it through and correct it before\n" +
+			"training on it (llmgw ctl train): what the model got wrong, an adapter learns.\n\n" +
+			"An adapter learns how answers sound — the organisation's words, format and\n" +
+			"tone — far better than what they say; the facts stay the knowledge base's.",
+		Example: "  llmgw ctl kb dataset Acme/policies --model qwen2.5-coder-14b-instruct --out policies.jsonl",
+		Args:    cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			w := cmd.OutOrStdout()
+			if dsOut != "" {
+				f, err := os.Create(dsOut)
+				if err != nil {
+					return err
+				}
+				defer f.Close()
+				w = f
+			}
+			lw := &datasetWriter{pairs: w, problems: cmd.ErrOrStderr()}
+			body, _ := json.Marshal(adminapi.KBDatasetRequest{Model: dsModel, PerChunk: dsPer, MaxChunks: dsMax})
+			err := client().patient(6*time.Hour).do(cmd.Context(), "POST",
+				"/api/v1/kb/collections/"+url.PathEscape(args[0])+"/dataset", json.RawMessage(body), lw)
+			lw.flush()
+			fmt.Fprintf(cmd.ErrOrStderr(), "%d pairs from %d chunks; %d chunks gave none\n", lw.n, len(lw.chunks), lw.failed)
+			return err
+		},
+	}
+	dataset.Flags().StringVar(&dsModel, "model", "", "the chat model that writes the questions and answers")
+	dataset.Flags().StringVar(&dsOut, "out", "", "the file to write (default: standard output)")
+	dataset.Flags().IntVar(&dsPer, "per-chunk", 8, "pairs from each chunk")
+	dataset.Flags().IntVar(&dsMax, "max-chunks", 500, "the most chunks read")
+	_ = dataset.MarkFlagRequired("model")
+	c.AddCommand(dataset)
+
 	var k int
 	search := &cobra.Command{
 		Use:   "search COLLECTION QUERY...",
@@ -240,3 +283,50 @@ func newKBCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 	c.AddCommand(search)
 	return c
 }
+
+// datasetWriter splits the dataset stream: pairs to the file, problems to
+// the terminal, counted as they pass.
+type datasetWriter struct {
+	pairs, problems io.Writer
+	buf             []byte
+	n, failed       int
+	chunks          map[string]bool
+}
+
+func (d *datasetWriter) Write(p []byte) (int, error) {
+	d.buf = append(d.buf, p...)
+	for {
+		i := bytes.IndexByte(d.buf, '\n')
+		if i < 0 {
+			return len(p), nil
+		}
+		d.line(d.buf[:i])
+		d.buf = d.buf[i+1:]
+	}
+}
+
+func (d *datasetWriter) flush() {
+	if len(bytes.TrimSpace(d.buf)) > 0 {
+		d.line(d.buf)
+	}
+	d.buf = nil
+}
+
+func (d *datasetWriter) line(b []byte) {
+	var l adminapi.KBDatasetLine
+	if json.Unmarshal(b, &l) != nil {
+		return
+	}
+	if d.chunks == nil {
+		d.chunks = map[string]bool{}
+	}
+	d.chunks[fmt.Sprintf("%s#%d", l.Doc, l.Ord)] = true
+	if l.Error != "" {
+		d.failed++
+		fmt.Fprintf(d.problems, "  %s #%d: %s\n", l.Doc, l.Ord, l.Error)
+		return
+	}
+	d.n++
+	out, _ := json.Marshal(l)
+	fmt.Fprintln(d.pairs, string(out))
+}
@@ -8,6 +8,7 @@ import (
 	"net"
 	"net/netip"
 	"os"
+	"path/filepath"
 	"runtime"
 	"strings"
 	"time"
@@ -65,6 +66,14 @@ Safe to run again: it changes only what differs.`,
 			}
 			if !noFirewall {
 				mirrorFirewall(cmd.Context(), cmd.OutOrStdout(), controlPlanes)
+				// And after, as the cluster's settings say: the control plane
+				// tells the mirror, which writes it beside its models.
+				state := filepath.Join(*dir, mirror.FirewallFile)
+				if err := installFirewallSync(cmd.Context(), firewall.Local{}, state); err != nil {
+					fmt.Fprintf(cmd.OutOrStdout(), "  !!    firewall: %v; the cluster's settings will not change it here\n", err)
+				} else {
+					fmt.Fprintln(cmd.OutOrStdout(), "  ok    firewall: follows the cluster's settings ("+firewallSyncPathUnit+")")
+				}
 			}
 			return nil
 		},
@@ -49,6 +49,7 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/fleetmetrics"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelsvc"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/train"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/kvevents"
@@ -232,6 +233,9 @@ func newServerCmd() *cobra.Command {
 			var agentHandler http.Handler
 			var agentPath string
 			var admin *control.Controller
+			// What the settings say of the firewall, for the mirror and the
+			// store's members; nil without a control plane.
+			var fwPub *firewallPublisher
 			updateMetrics := &modelserve.UpdateCollector{}
 			// The store's cluster, when it is one; read by the metrics and
 			// by `ctl status`.
@@ -263,9 +267,31 @@ func newServerCmd() *cobra.Command {
 					return &llmgwv1.Firewall{Enabled: c.Firewall.Enabled, Networks: c.FirewallNetworks,
 						EnginePortsFrom: int32(from), EnginePortsTo: int32(to)} //nolint:gosec // ports
 				})
+				// What each node's agent runs on, as the settings say now.
+				svc.SetAgentSettings(func(node string) string {
+					b, err := config.Marshal(current.Load().Agents.For(node))
+					if err != nil {
+						log.Warn("could not write a node's agent settings", "node", node, "err", err)
+						return ""
+					}
+					return string(b)
+				})
 				ctl.Reconciler().SetNotifier(notifier)
 				admin = ctl
 				agentPath, agentHandler = svc.Handler()
+				fwPub = newFirewallPublisher(func() config.Server { return *current.Load() }, rt.Store(),
+					func() []string {
+						var out []string
+						for _, n := range ctl.Inventory().Nodes() {
+							out = append(out, n.MgmtAddr)
+						}
+						return out
+					},
+					func() bool { return ctl.Leadership() == nil || ctl.Leadership().IsLeader() }, log)
+				go fwPub.run(ctx, 5*time.Minute)
+				if l := ctl.Leadership(); l != nil {
+					l.OnLead(fwPub.Kick)
+				}

 				// Before AdminHandler, which is where the registry is built —
 				// registering after it silently publishes nothing, which is
@@ -378,10 +404,19 @@ func newServerCmd() *cobra.Command {
 					// Knowledge bases, with the cluster's own models to embed a
 					// probe (a collection's dimension) and an operator's search.
 					api.SetKB(&adminapi.KBAdmin{Store: st, Embed: embedder(rt.Replay)})
+					// LoRA adapters trained on the nodes.
+					if agentSvc != nil {
+						trainer := train.New(st, agentSvc, log)
+						agentSvc.SetTrain(trainer)
+						go trainer.Run(ctx)
+						api.SetTrain(&adminapi.TrainAdmin{Store: st, Service: trainer})
+					}
 					// The cluster's settings, each change checked against this
 					// machine's own keys before it is kept.
 					api.SetSettings(&adminapi.Settings{Store: st, LocalKeys: config.LocalKeys(),
 						Check: func(body []byte) error { return checkClusterSettings(configPath, body) }})
+					// And the same document on the portal, for administrators.
+					rt.SetSettingsCheck(func(body []byte) error { return checkClusterSettings(configPath, body) }, config.LocalKeys())
 				}
 				// Catalog changes need somewhere to persist, so this is wired
 				// only when there is a store. Without one, the endpoints say so
@@ -438,6 +473,7 @@ func newServerCmd() *cobra.Command {
 					api.SetUsage(st)
 					// Capacity quotas, set and read over this API.
 					api.SetQuotas(st)
+					api.SetTenancy(st)
 				}
 				api.Register(adminMux)

@@ -499,6 +535,16 @@ func newServerCmd() *cobra.Command {
 					return fmt.Errorf("the model mirror: %w", err)
 				}
 				rt.SetMirror(mc)
+				if fwPub != nil {
+					fwPub.setMirror(mc)
+				}
+				// Adapters trained on the nodes are kept there too; sending one
+				// takes longer than the mirror's usual answers.
+				if adminAPI != nil {
+					if up, err := mirror.NewClient(cfg.Mirror.API, cfg.Mirror.TLS, 10*time.Minute); err == nil {
+						adminAPI.SetTrainMirror(up)
+					}
+				}
 				// Browsing the registry is what turns "paste a resolve URL"
 				// into "find the model". It needs a way out to the internet
 				// from this machine, which the mirror's own fetching does not.
@@ -716,6 +762,13 @@ func newServerCmd() *cobra.Command {
 						if agentSvc != nil {
 							agentSvc.BroadcastFirewall()
 						}
+						if fwPub != nil {
+							fwPub.Kick()
+						}
+					}
+					if slices.ContainsFunc(changed, func(c string) bool { return strings.HasPrefix(c, "agents") }) && agentSvc != nil {
+						// A moment apart per node, so off this goroutine.
+						go agentSvc.BroadcastAgentSettings()
 					}
 					if slices.ContainsFunc(changed, func(c string) bool { return strings.HasPrefix(c, "guard") }) {
 						if gate == nil {
@@ -160,7 +160,7 @@ func (sa *serverAdd) firewall(ctx context.Context) error {
 		return err
 	}
 	// What applies the cluster's firewall setting there when it changes.
-	if err := installFirewallSync(ctx, sa.remote); err != nil {
+	if err := installFirewallSync(ctx, sa.remote, firewallStateFile); err != nil {
 		fmt.Fprintf(sa.out, "  !!    firewall: %v\n", err)
 	}
 	if !cfg.Firewall.Enabled {
@@ -93,6 +93,11 @@ func (in *serverInit) storeFirewall(ctx context.Context, members []pgha.Node, se
 				} else {
 					in.kept("firewall on %s", m.Name)
 				}
+				// And after, as the cluster's settings say, read from the
+				// member's own copy of the store.
+				if err := firewall.InstallFollower(ctx, m.Remote); err != nil {
+					fmt.Fprintf(in.out, "  !!    firewall on %s: %v; the cluster's settings will not change it\n", m.Name, err)
+				}
 				continue
 			}
 		}
@@ -135,5 +140,5 @@ func loopbackListen(listen string) bool {
 // ensureFirewallSync installs the units that apply the cluster's firewall
 // setting on this control plane.
 func (in *serverInit) ensureFirewallSync(ctx context.Context) error {
-	return installFirewallSync(ctx, firewall.Local{})
+	return installFirewallSync(ctx, firewall.Local{}, firewallStateFile)
 }
@@ -16,6 +16,7 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/pgha"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
 )

@@ -348,15 +349,23 @@ func (in *serverInit) setupPostgres(ctx context.Context) (string, error) {
 	}

 	if installed {
-		// Its defaults assume it owns the machine. A gateway swapped out to
-		// give the database cache is worse than a smaller cache, and a store
-		// whose working set is megabytes needs little. Only a PostgreSQL init
-		// installed itself: one that was already here may serve something
-		// else that needs its connections.
-		for _, q := range []string{
-			"ALTER SYSTEM SET shared_buffers = '96MB'",
-			"ALTER SYSTEM SET max_connections = 40",
-		} {
+		// Sized to the machine's memory, as one it shares with the gateway:
+		// a gateway swapped out to give the database cache is worse than a
+		// smaller cache, but a knowledge base's vector index is searched
+		// well only from memory. Only a PostgreSQL init installed itself:
+		// one that was already here may serve something else that needs its
+		// connections.
+		meminfo, _ := os.ReadFile("/proc/meminfo")
+		total, ok := pgha.MemTotal(string(meminfo))
+		if !ok {
+			total = 1 << 30
+		}
+		mem := pgha.MemorySettings(total, true)
+		qs := []string{"ALTER SYSTEM SET max_connections = 40"}
+		for _, k := range []string{"shared_buffers", "effective_cache_size", "maintenance_work_mem", "work_mem"} {
+			qs = append(qs, "ALTER SYSTEM SET "+k+" = '"+mem[k]+"'")
+		}
+		for _, q := range qs {
 			if _, err := in.psql(ctx, q); err != nil {
 				return "", err
 			}
@@ -367,7 +376,8 @@ func (in *serverInit) setupPostgres(ctx context.Context) (string, error) {
 		if err := in.waitForPostgres(ctx); err != nil {
 			return "", err
 		}
-		in.done("PostgreSQL sized for a machine it shares (shared_buffers 96MB, 40 connections)")
+		in.done("PostgreSQL sized for a machine it shares, with %d MB of memory (shared_buffers %s, work_mem %s, 40 connections)",
+			total>>20, mem["shared_buffers"], mem["work_mem"])
 	}

 	in.setupPgvector(ctx, discovery.DiscoverHost())
@@ -0,0 +1,293 @@
+package cli
+
+import (
+	"encoding/json"
+	"fmt"
+	"io"
+	"os"
+	"strings"
+	"time"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+func newTrainCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
+	c := &cobra.Command{
+		Use:   "train",
+		Short: "Train LoRA adapters on the nodes, on question/answer pairs",
+		Long: "A node trains an adapter for a catalog model on pairs (llmgw ctl kb dataset\n" +
+			"writes them from a knowledge base; read them through first). It trains with\n" +
+			"its installed vLLM's torch and a small overlay, as the engines' user, on a card\n" +
+			"with --gpu-memory free — beside an engine, if it leaves that much. What it\n" +
+			"makes is kept by the control plane: llmgw ctl train download, or publish it\n" +
+			"as the model's adapter.\n\n" +
+			"An adapter learns how answers sound — words, format, tone — far better than\n" +
+			"what they say. Facts belong to the knowledge base.",
+	}
+
+	var model, base, rev, node, data, owner string
+	var public bool
+	var gpuMem int
+	var epochs, rank int
+	var lr float64
+	var noGGUF bool
+	start := &cobra.Command{
+		Use:   "start NAME --model MODEL --base-repo OWNER/NAME --node NODE --data FILE",
+		Short: "Queue an adapter's training on a node",
+		Example: "  llmgw ctl train start policies-tone --model qwen2.5-0.5b-instruct \\\n" +
+			"    --base-repo Qwen/Qwen2.5-0.5B-Instruct --node cuda01 --gpu-memory 2300 --data policies.jsonl",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			b, err := os.ReadFile(data)
+			if err != nil {
+				return err
+			}
+			req := adminapi.TrainRequest{Name: args[0], Model: model, BaseRepo: base, BaseRevision: rev, Node: node,
+				GPUMemoryMiB: gpuMem, Data: string(b), Params: map[string]any{}, Owner: owner, Public: public}
+			if cmd.Flags().Changed("epochs") {
+				req.Params["epochs"] = epochs
+			}
+			if cmd.Flags().Changed("rank") {
+				req.Params["rank"] = rank
+			}
+			if cmd.Flags().Changed("lr") {
+				req.Params["lr"] = lr
+			}
+			if noGGUF {
+				f := false
+				req.GGUF = &f
+			}
+			body, _ := json.Marshal(req)
+			var v adminapi.TrainJobView
+			if err := client().patient(2*time.Minute).do(cmd.Context(), "POST", "/api/v1/train/jobs", json.RawMessage(body), &v); err != nil {
+				return err
+			}
+			who := "every caller of " + v.Model
+			if v.Owner != "" {
+				who = v.Owner + " only"
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "queued %s: %s for %s on %s, %d pairs; it will be seen by %s\n",
+				v.ID, v.Name, v.Model, v.Node, v.Pairs, who)
+			fmt.Fprintf(cmd.OutOrStdout(), "follow it with: llmgw ctl train show %s --follow\n", v.ID)
+			return nil
+		},
+	}
+	f := start.Flags()
+	f.StringVar(&model, "model", "", "the catalog model the adapter is for")
+	f.StringVar(&base, "base-repo", "", "the weights to train on, as safetensors on Hugging Face (owner/name)")
+	f.StringVar(&rev, "base-revision", "", "a revision of them (default: the newest)")
+	f.StringVar(&node, "node", "", "the node that trains")
+	f.IntVar(&gpuMem, "gpu-memory", 4096, "MiB of a card the job needs free")
+	f.StringVar(&data, "data", "", "the pairs, NDJSON of {\"q\", \"a\"}")
+	f.IntVar(&epochs, "epochs", 6, "passes over the pairs")
+	f.IntVar(&rank, "rank", 16, "the adapter's rank")
+	f.Float64Var(&lr, "lr", 2e-4, "the learning rate")
+	f.BoolVar(&noGGUF, "no-gguf", false, "skip the llama.cpp GGUF, even for a model llama.cpp serves")
+	f.StringVar(&owner, "owner", "", "who the adapter belongs to — org:, team:, project: (path or ID), user: or key: — and only they see it "+
+		"(default: the knowledge base's owner, when the pairs come from one)")
+	f.BoolVar(&public, "public", false, "every caller of the model sees it; refused for pairs from a knowledge base")
+	for _, req := range []string{"model", "base-repo", "node", "data"} {
+		_ = start.MarkFlagRequired(req)
+	}
+	c.AddCommand(start)
+
+	c.AddCommand(&cobra.Command{
+		Use:   "list",
+		Short: "Training jobs, newest first",
+		Args:  cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			var jobs []adminapi.TrainJobView
+			if err := client().get(cmd.Context(), "/api/v1/train/jobs", &jobs); err != nil {
+				return err
+			}
+			if *asJSON {
+				return json.NewEncoder(cmd.OutOrStdout()).Encode(jobs)
+			}
+			if len(jobs) == 0 {
+				fmt.Fprintln(cmd.OutOrStdout(), "No training jobs (llmgw ctl train start).")
+				return nil
+			}
+			w := tab()
+			fmt.Fprintln(w, "JOB\tADAPTER\tMODEL\tOWNER\tNODE\tSTATE\tPROGRESS\tLOSS\tPEAK\tCREATED")
+			for _, j := range jobs {
+				fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n", j.ID, j.Name, j.Model, or(j.Owner, "everyone"), j.Node, j.State,
+					trainProgress(j), trainLoss(j), trainPeak(j), j.CreatedAt.Local().Format("01-02 15:04"))
+			}
+			return w.Flush()
+		},
+	})
+
+	var follow bool
+	show := &cobra.Command{
+		Use:   "show JOB",
+		Short: "How a training job goes, and what it made",
+		Args:  cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			out := cmd.OutOrStdout()
+			last := ""
+			for {
+				var j adminapi.TrainJobView
+				if err := client().get(cmd.Context(), "/api/v1/train/jobs/"+args[0], &j); err != nil {
+					return err
+				}
+				if *asJSON {
+					return json.NewEncoder(out).Encode(j)
+				}
+				line := fmt.Sprintf("%s  %s  %s  loss %s  peak %s  %s", j.State, trainProgress(j), j.Node, trainLoss(j), trainPeak(j), j.Message)
+				if !follow {
+					fmt.Fprintf(out, "%s: adapter %s for %s, owned by %s, on %s (base %s), %d pairs\n", j.ID, j.Name, j.Model,
+						or(j.Owner, "everyone"), j.Node, j.BaseRepo, j.Pairs)
+					fmt.Fprintln(out, line)
+					if len(j.Artifacts) > 0 {
+						fmt.Fprintf(out, "made: %s (llmgw ctl train download %s --kind KIND)\n", strings.Join(j.Artifacts, ", "), j.ID)
+					}
+					return nil
+				}
+				if line != last {
+					fmt.Fprintf(out, "%s  %s\n", time.Now().Format("15:04:05"), line)
+					last = line
+				}
+				if store.TrainOver(j.State) {
+					if j.State != store.TrainDone {
+						return fmt.Errorf("%s %s", j.ID, j.State)
+					}
+					return nil
+				}
+				select {
+				case <-cmd.Context().Done():
+					return nil
+				case <-time.After(3 * time.Second):
+				}
+			}
+		},
+	}
+	show.Flags().BoolVarP(&follow, "follow", "f", false, "follow it until it ends")
+	c.AddCommand(show)
+
+	c.AddCommand(&cobra.Command{
+		Use:   "cancel JOB",
+		Short: "Stop a training job",
+		Args:  cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			if err := client().do(cmd.Context(), "POST", "/api/v1/train/jobs/"+args[0]+"/cancel", nil, nil); err != nil {
+				return err
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "%s cancelled\n", args[0])
+			return nil
+		},
+	})
+	c.AddCommand(&cobra.Command{
+		Use:   "rm JOB",
+		Short: "Remove an ended training job and what it made",
+		Args:  cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			if err := client().do(cmd.Context(), "DELETE", "/api/v1/train/jobs/"+args[0], nil, nil); err != nil {
+				return err
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "%s removed\n", args[0])
+			return nil
+		},
+	})
+
+	var restart bool
+	publish := &cobra.Command{
+		Use:   "publish JOB",
+		Short: "Make what a finished job made its model's adapter, kept on the mirror",
+		Long: "The GGUF for a model llama.cpp serves, the PEFT files for one vLLM serves, are kept\n" +
+			"on the mirror and added to the model's adapters; a request then asks for the\n" +
+			"adapter by its name, as a model. llama.cpp loads adapters when a replica starts:\n" +
+			"--restart replaces the model's replicas, one at a time, to load it.",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			var v adminapi.TrainPublishView
+			if err := client().patient(15*time.Minute).do(cmd.Context(), "POST", "/api/v1/train/jobs/"+args[0]+"/publish", nil, &v); err != nil {
+				return err
+			}
+			out := cmd.OutOrStdout()
+			fmt.Fprintf(out, "%s is %s's adapter, from %s\n", v.Adapter, v.Model, v.Source)
+			if v.Owner != "" {
+				fmt.Fprintf(out, "only %s sees it; to anyone else it is a model that does not exist\n", v.Owner)
+			}
+			if restart && strings.Contains(v.Note, "restart") {
+				body, _ := json.Marshal(adminapi.RestartRequest{Models: []string{v.Model}})
+				var res adminapi.RestartResult
+				if err := client().post(cmd.Context(), "/api/v1/restart", body, &res); err != nil {
+					return err
+				}
+				fmt.Fprintf(out, "%d replica(s) of %s restarting to load it; then ask for model %s\n", len(res.Queued), v.Model, v.Adapter)
+				return nil
+			}
+			fmt.Fprintln(out, v.Note)
+			return nil
+		},
+	}
+	publish.Flags().BoolVar(&restart, "restart", false, "restart the model's replicas to load it (llama.cpp)")
+	c.AddCommand(publish)
+
+	var kind, outFile string
+	download := &cobra.Command{
+		Use:   "download JOB",
+		Short: "Save what a training job made: the PEFT adapter (peft) or the GGUF (gguf)",
+		Args:  cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			if outFile == "" {
+				outFile = args[0] + ".tar.gz"
+				if kind == "gguf" {
+					outFile = args[0] + ".gguf"
+				}
+			}
+			f, err := os.Create(outFile)
+			if err != nil {
+				return err
+			}
+			defer f.Close()
+			cw := &countWriter{w: f}
+			if err := client().patient(10*time.Minute).do(cmd.Context(), "GET",
+				"/api/v1/train/jobs/"+args[0]+"/artifacts/"+kind, nil, io.Writer(cw)); err != nil {
+				_ = os.Remove(outFile)
+				return err
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "%s: %s\n", outFile, humanBytes(cw.n))
+			return nil
+		},
+	}
+	download.Flags().StringVar(&kind, "kind", "peft", "peft (a gzipped tar of the PEFT adapter) or gguf")
+	download.Flags().StringVarP(&outFile, "out", "o", "", "the file to write")
+	c.AddCommand(download)
+	return c
+}
+
+type countWriter struct {
+	w io.Writer
+	n int64
+}
+
+func (c *countWriter) Write(p []byte) (int, error) {
+	n, err := c.w.Write(p)
+	c.n += int64(n)
+	return n, err
+}
+
+func trainProgress(j adminapi.TrainJobView) string {
+	if j.Steps == 0 {
+		return "-"
+	}
+	return fmt.Sprintf("%d/%d", j.Step, j.Steps)
+}
+
+func trainLoss(j adminapi.TrainJobView) string {
+	if j.Loss == 0 {
+		return "-"
+	}
+	return fmt.Sprintf("%.3f", j.Loss)
+}
+
+func trainPeak(j adminapi.TrainJobView) string {
+	if j.PeakMiB == 0 {
+		return "-"
+	}
+	return fmt.Sprintf("%d MiB", j.PeakMiB)
+}
@@ -0,0 +1,529 @@
+// Package comfy has images made by ComfyUI: a workflow filled in with what
+// was asked for, queued, waited on, and its pictures fetched.
+//
+// ComfyUI speaks no OpenAI. A workflow is a graph of nodes that is queued with
+// POST /prompt, is done when GET /history/{id} lists it, and leaves its
+// pictures to be read one by one from GET /view. What the graph is depends on
+// the model — FLUX schnell is not FLUX dev, and neither is SDXL — so the graph
+// is a template the catalog names, with placeholders where a request's
+// prompt, size and seed go.
+package comfy
+
+import (
+	"bytes"
+	"context"
+	"crypto/rand"
+	"embed"
+	"encoding/binary"
+	"encoding/hex"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"math/big"
+	"net/http"
+	"net/url"
+	"slices"
+	"strconv"
+	"strings"
+	"time"
+)
+
+//go:embed workflows/*.json
+var builtin embed.FS
+
+// DefaultWorkflow is the template used when the catalog names none.
+const DefaultWorkflow = "flux-schnell"
+
+// Builtins are the workflows that come with llmgw, by name.
+func Builtins() []string {
+	entries, _ := builtin.ReadDir("workflows")
+	out := make([]string, 0, len(entries))
+	for _, e := range entries {
+		out = append(out, strings.TrimSuffix(e.Name(), ".json"))
+	}
+	return out
+}
+
+// Template is a workflow in ComfyUI's API format, with placeholders.
+//
+// A placeholder is a string value "{{name}}", or "{{name|default}}" for one a
+// request may leave out. One that is the whole value becomes the value
+// itself, a number where the node wants a number; one inside a longer string
+// is written into it. The names are those of Params.
+type Template struct {
+	graph map[string]any
+}
+
+// Load reads a template: a built-in's name, or the workflow's JSON itself.
+func Load(nameOrJSON string) (*Template, error) {
+	s := strings.TrimSpace(nameOrJSON)
+	if s == "" {
+		s = DefaultWorkflow
+	}
+	raw := []byte(s)
+	if !strings.HasPrefix(s, "{") {
+		b, err := builtin.ReadFile("workflows/" + s + ".json")
+		if err != nil {
+			return nil, fmt.Errorf("no built-in workflow %q (there are %s)", s, strings.Join(Builtins(), ", "))
+		}
+		raw = b
+	}
+	var g map[string]any
+	if err := json.Unmarshal(raw, &g); err != nil {
+		return nil, fmt.Errorf("the workflow is not JSON: %w", err)
+	}
+	if len(g) == 0 {
+		return nil, errors.New("the workflow has no nodes")
+	}
+	for id, n := range g {
+		node, ok := n.(map[string]any)
+		if !ok || node["class_type"] == nil {
+			// The UI's own save format has a "nodes" list instead, and is
+			// not something /prompt accepts.
+			return nil, fmt.Errorf("node %q has no class_type: the workflow must be saved in API format", id)
+		}
+	}
+	return &Template{graph: g}, nil
+}
+
+// Params are what a request decides. A zero number is one not given, which
+// leaves it to the template's default.
+type Params struct {
+	Checkpoint     string
+	Prompt         string
+	NegativePrompt string
+	Width, Height  int
+	N              int
+	Seed           int64
+	Steps          int
+	Guidance       float64
+}
+
+func (p Params) value(name string) (any, bool) {
+	switch name {
+	case "checkpoint":
+		return p.Checkpoint, p.Checkpoint != ""
+	case "prompt":
+		return p.Prompt, true
+	case "negative_prompt":
+		return p.NegativePrompt, true
+	case "width":
+		return p.Width, p.Width > 0
+	case "height":
+		return p.Height, p.Height > 0
+	case "n":
+		return max(p.N, 1), true
+	case "seed":
+		return p.Seed, true
+	case "steps":
+		return p.Steps, p.Steps > 0
+	case "guidance":
+		return p.Guidance, p.Guidance > 0
+	}
+	return nil, false
+}
+
+// Fill is the workflow for one request.
+func (t *Template) Fill(p Params) (map[string]any, error) {
+	var missing []string
+	var fill func(v any) any
+	fill = func(v any) any {
+		switch v := v.(type) {
+		case map[string]any:
+			out := make(map[string]any, len(v))
+			for k, x := range v {
+				out[k] = fill(x)
+			}
+			return out
+		case []any:
+			out := make([]any, len(v))
+			for i, x := range v {
+				out[i] = fill(x)
+			}
+			return out
+		case string:
+			return fillString(v, p, &missing)
+		}
+		return v
+	}
+	g := fill(t.graph).(map[string]any)
+	if len(missing) > 0 {
+		slices.Sort(missing)
+		return nil, fmt.Errorf("the workflow wants %s, which the request did not give and the workflow has no default for",
+			strings.Join(slices.Compact(missing), ", "))
+	}
+	return g, nil
+}
+
+func fillString(s string, p Params, missing *[]string) any {
+	if !strings.Contains(s, "{{") {
+		return s
+	}
+	resolve := func(ph string) (any, bool) {
+		name, def, hasDef := strings.Cut(ph, "|")
+		name = strings.TrimSpace(name)
+		if v, ok := p.value(name); ok {
+			return v, true
+		}
+		if hasDef {
+			def = strings.TrimSpace(def)
+			if f, err := strconv.ParseFloat(def, 64); err == nil {
+				return f, true
+			}
+			return def, true
+		}
+		*missing = append(*missing, name)
+		return nil, false
+	}
+	// The whole value: what the node gets is the value, not its text.
+	if strings.HasPrefix(s, "{{") && strings.HasSuffix(s, "}}") && strings.Count(s, "{{") == 1 {
+		v, _ := resolve(s[2 : len(s)-2])
+		return v
+	}
+	var b strings.Builder
+	for {
+		i := strings.Index(s, "{{")
+		j := strings.Index(s, "}}")
+		if i < 0 || j < i {
+			b.WriteString(s)
+			return b.String()
+		}
+		b.WriteString(s[:i])
+		if v, ok := resolve(s[i+2 : j]); ok {
+			fmt.Fprint(&b, v)
+		}
+		s = s[j+2:]
+	}
+}
+
+// RandomSeed is a seed for a request that named none. ComfyUI's are 64-bit,
+// but JSON numbers are read as doubles on the way: kept to 2^53.
+func RandomSeed() int64 {
+	n, err := rand.Int(rand.Reader, big.NewInt(1<<53))
+	if err != nil {
+		return time.Now().UnixNano() & (1<<53 - 1)
+	}
+	return n.Int64()
+}
+
+// Image is one picture made.
+type Image struct {
+	Data        []byte
+	ContentType string
+}
+
+// Error is a failure, said as what it means for the request.
+type Error struct {
+	Msg string
+	// Retryable is a failure of this server, which another one might not
+	// have: it could not be reached, or it answered 5xx.
+	Retryable bool
+}
+
+func (e *Error) Error() string { return e.Msg }
+
+// Client queues workflows and waits for them.
+type Client struct {
+	HTTP *http.Client
+	// Poll is how often a queued workflow is asked after.
+	Poll time.Duration
+}
+
+// Generate runs a workflow on the ComfyUI at base and returns its pictures,
+// in the order of the nodes that made them. A context that ends takes the
+// workflow out of the queue, or interrupts it if it is running, so a client
+// that went away does not leave the card working for nobody.
+func (c *Client) Generate(ctx context.Context, base, apiKey string, workflow map[string]any) ([]Image, error) {
+	base = strings.TrimRight(base, "/")
+	client := make([]byte, 8)
+	_, _ = rand.Read(client)
+	body, err := json.Marshal(map[string]any{"prompt": workflow, "client_id": "llmgw-" + hex.EncodeToString(client)})
+	if err != nil {
+		return nil, err
+	}
+	var queued struct {
+		PromptID   string          `json:"prompt_id"`
+		Error      json.RawMessage `json:"error"`
+		NodeErrors json.RawMessage `json:"node_errors"`
+	}
+	status, err := c.do(ctx, http.MethodPost, base+"/prompt", apiKey, body, &queued)
+	if err != nil {
+		return nil, err
+	}
+	if status != http.StatusOK || queued.PromptID == "" {
+		// A workflow ComfyUI will not run: a checkpoint it does not have, a
+		// node it does not know. The catalog's to fix, not the client's, and
+		// no other replica set up the same way would take it either.
+		return nil, &Error{Msg: "ComfyUI refused the workflow: " + refusal(queued.Error, queued.NodeErrors)}
+	}
+	id := queued.PromptID
+
+	poll := c.Poll
+	if poll <= 0 {
+		poll = 250 * time.Millisecond
+	}
+	t := time.NewTicker(poll)
+	defer t.Stop()
+	for {
+		select {
+		case <-ctx.Done():
+			c.cancel(context.WithoutCancel(ctx), base, apiKey, id)
+			return nil, ctx.Err()
+		case <-t.C:
+		}
+		var hist map[string]historyEntry
+		if _, err := c.do(ctx, http.MethodGet, base+"/history/"+url.PathEscape(id), apiKey, nil, &hist); err != nil {
+			if ctx.Err() != nil {
+				continue // said by the select
+			}
+			return nil, err
+		}
+		h, ok := hist[id]
+		if !ok {
+			continue
+		}
+		if h.Status.StatusStr == "error" {
+			return nil, &Error{Msg: "the workflow failed in ComfyUI: " + h.failure()}
+		}
+		if !h.Status.Completed {
+			continue
+		}
+		return c.fetch(ctx, base, apiKey, h)
+	}
+}
+
+type historyEntry struct {
+	Status struct {
+		StatusStr string              `json:"status_str"`
+		Completed bool                `json:"completed"`
+		Messages  [][]json.RawMessage `json:"messages"`
+	} `json:"status"`
+	Outputs map[string]struct {
+		Images []struct {
+			Filename  string `json:"filename"`
+			Subfolder string `json:"subfolder"`
+			Type      string `json:"type"`
+		} `json:"images"`
+	} `json:"outputs"`
+}
+
+// failure is what ComfyUI said went wrong, from its execution_error.
+func (h historyEntry) failure() string {
+	for _, m := range h.Status.Messages {
+		if len(m) < 2 || string(m[0]) != `"execution_error"` {
+			continue
+		}
+		var e struct {
+			NodeType string `json:"node_type"`
+			Message  string `json:"exception_message"`
+		}
+		if json.Unmarshal(m[1], &e) == nil && e.Message != "" {
+			return strings.TrimSpace(e.NodeType + ": " + strings.TrimSpace(e.Message))
+		}
+	}
+	return "no reason given"
+}
+
+func (c *Client) fetch(ctx context.Context, base, apiKey string, h historyEntry) ([]Image, error) {
+	nodes := make([]string, 0, len(h.Outputs))
+	for id := range h.Outputs {
+		nodes = append(nodes, id)
+	}
+	// Numerically where they are numbers, which is how ComfyUI names them.
+	slices.SortFunc(nodes, func(a, b string) int {
+		x, ea := strconv.Atoi(a)
+		y, eb := strconv.Atoi(b)
+		if ea == nil && eb == nil {
+			return x - y
+		}
+		return strings.Compare(a, b)
+	})
+	var out []Image
+	for _, n := range nodes {
+		for _, im := range h.Outputs[n].Images {
+			q := url.Values{"filename": {im.Filename}, "subfolder": {im.Subfolder}, "type": {im.Type}}
+			img, err := c.get(ctx, base+"/view?"+q.Encode(), apiKey)
+			if err != nil {
+				return nil, err
+			}
+			out = append(out, img)
+		}
+	}
+	if len(out) == 0 {
+		return nil, &Error{Msg: "the workflow finished without a picture: it has no SaveImage or PreviewImage node"}
+	}
+	return out, nil
+}
+
+// cancel takes a workflow out of the queue, or stops it if it is the one
+// running. Best effort: the server may be gone, which also stops it.
+func (c *Client) cancel(ctx context.Context, base, apiKey, id string) {
+	ctx, done := context.WithTimeout(ctx, 5*time.Second)
+	defer done()
+	body, _ := json.Marshal(map[string]any{"delete": []string{id}})
+	_, _ = c.do(ctx, http.MethodPost, base+"/queue", apiKey, body, nil)
+	var q struct {
+		Running [][]json.RawMessage `json:"queue_running"`
+	}
+	if _, err := c.do(ctx, http.MethodGet, base+"/queue", apiKey, nil, &q); err != nil {
+		return
+	}
+	for _, r := range q.Running {
+		if len(r) > 1 && string(r[1]) == strconv.Quote(id) {
+			_, _ = c.do(ctx, http.MethodPost, base+"/interrupt", apiKey, []byte("{}"), nil)
+			return
+		}
+	}
+}
+
+func (c *Client) client() *http.Client {
+	if c.HTTP != nil {
+		return c.HTTP
+	}
+	return http.DefaultClient
+}
+
+func (c *Client) request(ctx context.Context, method, u, apiKey string, body []byte) (*http.Response, error) {
+	req, err := http.NewRequestWithContext(ctx, method, u, bytes.NewReader(body))
+	if err != nil {
+		return nil, err
+	}
+	if body != nil {
+		req.Header.Set("Content-Type", "application/json")
+	}
+	if apiKey != "" {
+		req.Header.Set("Authorization", "Bearer "+apiKey)
+	}
+	resp, err := c.client().Do(req)
+	if err != nil {
+		if ctx.Err() != nil {
+			return nil, ctx.Err()
+		}
+		return nil, &Error{Msg: "ComfyUI could not be reached: " + err.Error(), Retryable: true}
+	}
+	return resp, nil
+}
+
+// do sends a request and reads a JSON answer into out. A 5xx is an error;
+// any other status is returned with whatever body could be read.
+func (c *Client) do(ctx context.Context, method, u, apiKey string, body []byte, out any) (int, error) {
+	resp, err := c.request(ctx, method, u, apiKey, body)
+	if err != nil {
+		return 0, err
+	}
+	defer resp.Body.Close()
+	b, err := io.ReadAll(io.LimitReader(resp.Body, 16<<20))
+	if err != nil {
+		return 0, &Error{Msg: "reading ComfyUI's answer: " + err.Error(), Retryable: true}
+	}
+	if resp.StatusCode >= 500 {
+		return resp.StatusCode, &Error{Msg: fmt.Sprintf("ComfyUI answered %d: %s", resp.StatusCode, snippet(b)), Retryable: true}
+	}
+	if out != nil && len(b) > 0 {
+		if err := json.Unmarshal(b, out); err != nil && resp.StatusCode == http.StatusOK {
+			return resp.StatusCode, &Error{Msg: "ComfyUI's answer is not JSON: " + snippet(b)}
+		}
+	}
+	return resp.StatusCode, nil
+}
+
+func (c *Client) get(ctx context.Context, u, apiKey string) (Image, error) {
+	resp, err := c.request(ctx, http.MethodGet, u, apiKey, nil)
+	if err != nil {
+		return Image{}, err
+	}
+	defer resp.Body.Close()
+	b, err := io.ReadAll(io.LimitReader(resp.Body, 64<<20))
+	if err != nil {
+		return Image{}, &Error{Msg: "reading a picture from ComfyUI: " + err.Error(), Retryable: true}
+	}
+	if resp.StatusCode != http.StatusOK {
+		return Image{}, &Error{Msg: fmt.Sprintf("ComfyUI answered %d for a picture it made", resp.StatusCode)}
+	}
+	ct := resp.Header.Get("Content-Type")
+	if ct == "" || ct == "application/octet-stream" {
+		ct = http.DetectContentType(b)
+	}
+	if ct == "image/png" {
+		b = withoutPNGText(b)
+	}
+	return Image{Data: b, ContentType: ct}, nil
+}
+
+// withoutPNGText is a PNG without its text chunks. ComfyUI writes the whole
+// workflow into every picture it saves — the prompt, and the checkpoint and
+// nodes behind it — and what a picture says about how it was made is the
+// operator's to publish, not something to hand every client. A file that is
+// not a well-formed PNG is returned as it came.
+func withoutPNGText(b []byte) []byte {
+	const sig = "\x89PNG\r\n\x1a\n"
+	if len(b) < len(sig) || string(b[:len(sig)]) != sig {
+		return b
+	}
+	out := make([]byte, 0, len(b))
+	out = append(out, b[:len(sig)]...)
+	for i := len(sig); i < len(b); {
+		if i+12 > len(b) {
+			return b
+		}
+		n := int(binary.BigEndian.Uint32(b[i:]))
+		end := i + 12 + n
+		if n < 0 || end > len(b) {
+			return b
+		}
+		switch string(b[i+4 : i+8]) {
+		case "tEXt", "iTXt", "zTXt":
+		default:
+			out = append(out, b[i:end]...)
+		}
+		i = end
+	}
+	return out
+}
+
+// refusal is what /prompt said when it would not queue a workflow.
+func refusal(e, nodes json.RawMessage) string {
+	var top struct {
+		Message string `json:"message"`
+		Details string `json:"details"`
+	}
+	msg := snippet(e)
+	if json.Unmarshal(e, &top) == nil && top.Message != "" {
+		msg = top.Message
+		if top.Details != "" {
+			msg += ": " + top.Details
+		}
+	}
+	var perNode map[string]struct {
+		ClassType string `json:"class_type"`
+		Errors    []struct {
+			Message string `json:"message"`
+			Details string `json:"details"`
+		} `json:"errors"`
+	}
+	if json.Unmarshal(nodes, &perNode) == nil {
+		var parts []string
+		for _, n := range perNode {
+			for _, er := range n.Errors {
+				parts = append(parts, strings.TrimSpace(n.ClassType+": "+er.Message+" "+er.Details))
+			}
+		}
+		slices.Sort(parts)
+		if len(parts) > 0 {
+			msg += " (" + strings.Join(parts, "; ") + ")"
+		}
+	}
+	if msg == "" {
+		msg = "no reason given"
+	}
+	return msg
+}
+
+func snippet(b []byte) string {
+	s := strings.TrimSpace(string(b))
+	if len(s) > 300 {
+		s = s[:300] + "…"
+	}
+	return s
+}
@@ -0,0 +1,227 @@
+package comfy
+
+import (
+	"context"
+	"encoding/binary"
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"sync"
+	"testing"
+	"time"
+)
+
+func TestTheBuiltInWorkflowsFillIn(t *testing.T) {
+	for _, name := range Builtins() {
+		tpl, err := Load(name)
+		if err != nil {
+			t.Fatalf("%s: %v", name, err)
+		}
+		g, err := tpl.Fill(Params{Checkpoint: "flux.safetensors", Prompt: `a "red" fox`, Width: 768, Height: 512, N: 2, Seed: 42})
+		if err != nil {
+			t.Fatalf("%s: %v", name, err)
+		}
+		b, _ := json.Marshal(g)
+		s := string(b)
+		for _, want := range []string{`"ckpt_name":"flux.safetensors"`, `"text":"a \"red\" fox"`, `"width":768`,
+			`"height":512`, `"batch_size":2`, `"seed":42`} {
+			if !strings.Contains(s, want) {
+				t.Errorf("%s: %s not in %s", name, want, s)
+			}
+		}
+		if strings.Contains(s, "{{") {
+			t.Errorf("%s: a placeholder was left: %s", name, s)
+		}
+	}
+}
+
+func TestAStepCountGivenReplacesTheDefault(t *testing.T) {
+	tpl, _ := Load("flux-schnell")
+	g, _ := tpl.Fill(Params{Checkpoint: "c", Prompt: "p", Width: 512, Height: 512})
+	if steps := g["5"].(map[string]any)["inputs"].(map[string]any)["steps"]; steps != 4.0 {
+		t.Errorf("default steps = %v, want 4", steps)
+	}
+	g, _ = tpl.Fill(Params{Checkpoint: "c", Prompt: "p", Width: 512, Height: 512, Steps: 8})
+	if steps := g["5"].(map[string]any)["inputs"].(map[string]any)["steps"]; steps != 8 {
+		t.Errorf("steps = %v, want 8", steps)
+	}
+}
+
+func TestAPlaceholderWithoutAValueOrADefaultIsSaid(t *testing.T) {
+	tpl, err := Load(`{"1":{"class_type":"X","inputs":{"a":"{{checkpoint}}","b":"size {{width}}x{{height}}"}}}`)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if _, err := tpl.Fill(Params{Width: 5}); err == nil || !strings.Contains(err.Error(), "checkpoint, height") {
+		t.Fatalf("err = %v", err)
+	}
+	g, err := tpl.Fill(Params{Checkpoint: "c", Width: 5, Height: 6})
+	if err != nil {
+		t.Fatal(err)
+	}
+	if b := g["1"].(map[string]any)["inputs"].(map[string]any)["b"]; b != "size 5x6" {
+		t.Errorf("b = %v", b)
+	}
+}
+
+func TestAWorkflowInTheUIsFormatIsRefused(t *testing.T) {
+	if _, err := Load(`{"nodes":[{"id":1}],"links":[]}`); err == nil || !strings.Contains(err.Error(), "API format") {
+		t.Fatalf("err = %v", err)
+	}
+	if _, err := Load("sdxl-turbo"); err == nil {
+		t.Fatal("an unknown built-in was loaded")
+	}
+}
+
+// fakeComfy is just enough of ComfyUI: /prompt queues, /history answers
+// after a few asks, /view serves a picture.
+type fakeComfy struct {
+	mu          sync.Mutex
+	asked       int
+	fail        string
+	refuse      bool
+	deleted     []string
+	queued      map[string]any
+	readyAt     int
+	srv         *httptest.Server
+	interrupted bool
+}
+
+func newFakeComfy(t *testing.T) *fakeComfy {
+	f := &fakeComfy{readyAt: 2}
+	f.srv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		f.mu.Lock()
+		defer f.mu.Unlock()
+		switch {
+		case r.Method == http.MethodPost && r.URL.Path == "/prompt":
+			if f.refuse {
+				w.WriteHeader(http.StatusBadRequest)
+				fmt.Fprint(w, `{"error":{"message":"Prompt outputs failed validation","details":""},"node_errors":{"1":{"class_type":"CheckpointLoaderSimple","errors":[{"message":"Value not in list","details":"ckpt_name: 'x' not in []"}]}}}`)
+				return
+			}
+			var b struct {
+				Prompt map[string]any `json:"prompt"`
+			}
+			_ = json.NewDecoder(r.Body).Decode(&b)
+			f.queued = b.Prompt
+			fmt.Fprint(w, `{"prompt_id":"p1","number":1,"node_errors":{}}`)
+		case r.URL.Path == "/history/p1":
+			f.asked++
+			if f.asked < f.readyAt {
+				fmt.Fprint(w, `{}`)
+				return
+			}
+			if f.fail != "" {
+				fmt.Fprintf(w, `{"p1":{"status":{"status_str":"error","completed":false,"messages":[["execution_start",{}],["execution_error",{"node_type":"KSampler","exception_message":%q}]]},"outputs":{}}}`, f.fail)
+				return
+			}
+			fmt.Fprint(w, `{"p1":{"status":{"status_str":"success","completed":true,"messages":[]},"outputs":{"7":{"images":[{"filename":"a.png","subfolder":"","type":"temp"},{"filename":"b.png","subfolder":"","type":"temp"}]}}}}`)
+		case r.URL.Path == "/view":
+			w.Header().Set("Content-Type", "image/png")
+			fmt.Fprint(w, "PNG:"+r.URL.Query().Get("filename"))
+		case r.URL.Path == "/queue" && r.Method == http.MethodPost:
+			var b struct {
+				Delete []string `json:"delete"`
+			}
+			_ = json.NewDecoder(r.Body).Decode(&b)
+			f.deleted = append(f.deleted, b.Delete...)
+		case r.URL.Path == "/queue":
+			fmt.Fprint(w, `{"queue_running":[[1,"p1",{},{},[]]],"queue_pending":[]}`)
+		case r.URL.Path == "/interrupt":
+			f.interrupted = true
+		default:
+			http.NotFound(w, r)
+		}
+	}))
+	t.Cleanup(f.srv.Close)
+	return f
+}
+
+func TestGenerateWaitsForTheWorkflowAndFetchesItsPictures(t *testing.T) {
+	f := newFakeComfy(t)
+	c := &Client{Poll: time.Millisecond}
+	images, err := c.Generate(t.Context(), f.srv.URL, "", map[string]any{"1": map[string]any{"class_type": "X"}})
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(images) != 2 || string(images[0].Data) != "PNG:a.png" || images[1].ContentType != "image/png" {
+		t.Fatalf("images = %+v", images)
+	}
+	if f.queued["1"] == nil {
+		t.Error("the workflow was not what was queued")
+	}
+}
+
+func TestAWorkflowThatFailsSaysWhy(t *testing.T) {
+	f := newFakeComfy(t)
+	f.fail = "CUDA out of memory"
+	_, err := (&Client{Poll: time.Millisecond}).Generate(t.Context(), f.srv.URL, "", map[string]any{})
+	if err == nil || !strings.Contains(err.Error(), "KSampler: CUDA out of memory") {
+		t.Fatalf("err = %v", err)
+	}
+}
+
+func TestAWorkflowComfyUIRefusesIsNotRetried(t *testing.T) {
+	f := newFakeComfy(t)
+	f.refuse = true
+	_, err := (&Client{Poll: time.Millisecond}).Generate(t.Context(), f.srv.URL, "", map[string]any{})
+	ce, ok := err.(*Error)
+	if !ok || ce.Retryable || !strings.Contains(ce.Msg, "Value not in list") {
+		t.Fatalf("err = %#v", err)
+	}
+}
+
+func TestAComfyUIThatCannotBeReachedIsRetryable(t *testing.T) {
+	f := newFakeComfy(t)
+	f.srv.Close()
+	_, err := (&Client{Poll: time.Millisecond}).Generate(t.Context(), f.srv.URL, "", map[string]any{})
+	if ce, ok := err.(*Error); !ok || !ce.Retryable {
+		t.Fatalf("err = %#v", err)
+	}
+}
+
+func TestAClientThatLeavesTakesItsWorkflowOffTheCard(t *testing.T) {
+	f := newFakeComfy(t)
+	f.readyAt = 1 << 30
+	ctx, cancel := context.WithTimeout(t.Context(), 30*time.Millisecond)
+	defer cancel()
+	_, err := (&Client{Poll: time.Millisecond}).Generate(ctx, f.srv.URL, "", map[string]any{})
+	if err == nil {
+		t.Fatal("no error")
+	}
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	if len(f.deleted) != 1 || f.deleted[0] != "p1" || !f.interrupted {
+		t.Errorf("deleted %v, interrupted %v", f.deleted, f.interrupted)
+	}
+}
+
+func TestAPictureLosesTheWorkflowComfyUIWroteIntoIt(t *testing.T) {
+	chunk := func(kind, data string) []byte {
+		b := make([]byte, 0, 12+len(data))
+		b = binary.BigEndian.AppendUint32(b, uint32(len(data)))
+		b = append(b, kind...)
+		b = append(b, data...)
+		return append(b, 0, 0, 0, 0) // a CRC nobody here checks
+	}
+	png := []byte("\x89PNG\r\n\x1a\n")
+	png = append(png, chunk("IHDR", "0123456789abc")...)
+	png = append(png, chunk("tEXt", "prompt\x00{\"1\":...}")...)
+	png = append(png, chunk("IDAT", "pixels")...)
+	png = append(png, chunk("iTXt", "workflow\x00...")...)
+	png = append(png, chunk("IEND", "")...)
+	got := string(withoutPNGText(png))
+	if strings.Contains(got, "prompt") || strings.Contains(got, "workflow") {
+		t.Errorf("text left in: %q", got)
+	}
+	for _, kind := range []string{"IHDR", "IDAT", "pixels", "IEND"} {
+		if !strings.Contains(got, kind) {
+			t.Errorf("%s gone: %q", kind, got)
+		}
+	}
+	if broken := png[:30]; string(withoutPNGText(broken)) != string(broken) {
+		t.Error("a truncated PNG was changed")
+	}
+}
@@ -0,0 +1,12 @@
+{
+  "1": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": "{{checkpoint}}"}},
+  "2": {"class_type": "CLIPTextEncode", "inputs": {"text": "{{prompt}}", "clip": ["1", 1]}},
+  "3": {"class_type": "CLIPTextEncode", "inputs": {"text": "{{negative_prompt}}", "clip": ["1", 1]}},
+  "8": {"class_type": "FluxGuidance", "inputs": {"guidance": "{{guidance|3.5}}", "conditioning": ["2", 0]}},
+  "4": {"class_type": "EmptySD3LatentImage", "inputs": {"width": "{{width}}", "height": "{{height}}", "batch_size": "{{n}}"}},
+  "5": {"class_type": "KSampler", "inputs": {
+    "seed": "{{seed}}", "steps": "{{steps|20}}", "cfg": 1, "sampler_name": "euler", "scheduler": "simple", "denoise": 1,
+    "model": ["1", 0], "positive": ["8", 0], "negative": ["3", 0], "latent_image": ["4", 0]}},
+  "6": {"class_type": "VAEDecode", "inputs": {"samples": ["5", 0], "vae": ["1", 2]}},
+  "7": {"class_type": "PreviewImage", "inputs": {"images": ["6", 0]}}
+}
@@ -0,0 +1,11 @@
+{
+  "1": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": "{{checkpoint}}"}},
+  "2": {"class_type": "CLIPTextEncode", "inputs": {"text": "{{prompt}}", "clip": ["1", 1]}},
+  "3": {"class_type": "CLIPTextEncode", "inputs": {"text": "{{negative_prompt}}", "clip": ["1", 1]}},
+  "4": {"class_type": "EmptySD3LatentImage", "inputs": {"width": "{{width}}", "height": "{{height}}", "batch_size": "{{n}}"}},
+  "5": {"class_type": "KSampler", "inputs": {
+    "seed": "{{seed}}", "steps": "{{steps|4}}", "cfg": 1, "sampler_name": "euler", "scheduler": "simple", "denoise": 1,
+    "model": ["1", 0], "positive": ["2", 0], "negative": ["3", 0], "latent_image": ["4", 0]}},
+  "6": {"class_type": "VAEDecode", "inputs": {"samples": ["5", 0], "vae": ["1", 2]}},
+  "7": {"class_type": "PreviewImage", "inputs": {"images": ["6", 0]}}
+}
@@ -18,7 +18,12 @@ import (
 //
 // An empty path loads the defaults, which with --server and --node-id on the
 // command line is enough for a smoke test on a node with no IB.
-func LoadAgent(path string) (agent.Config, error) {
+func LoadAgent(path string) (agent.Config, error) { return LoadAgentWith(path, nil) }
+
+// LoadAgentWith is LoadAgent with the cluster's settings for this node
+// (agents.default and agents.nodes.<id>, as YAML) laid over the file: the
+// file is what the agent needs to start, the settings what it runs on.
+func LoadAgentWith(path string, settings []byte) (agent.Config, error) {
 	cfg := agent.DefaultConfig()
 	k := koanf.New(".")

@@ -30,6 +35,11 @@ func LoadAgent(path string) (agent.Config, error) {
 			return cfg, fmt.Errorf("parsing %s: %w", path, err)
 		}
 	}
+	if len(settings) > 0 {
+		if err := k.Load(bytesProvider(settings), yaml.Parser()); err != nil {
+			return cfg, fmt.Errorf("parsing the cluster's settings for this agent: %w", err)
+		}
+	}

 	err := k.Load(env.Provider(".", env.Opt{
 		Prefix: "LLMGW_",
@@ -0,0 +1,110 @@
+package config
+
+import (
+	"fmt"
+	"slices"
+	"sort"
+)
+
+// AgentSettings are the nodes' agent settings, kept with the cluster's
+// (BACKLOG E3): everything agent.yaml can say except what an agent needs to
+// start — its id, the control planes to dial, its address and its
+// certificate. Default applies to every node, and Nodes, by node id, over
+// it; both over the node's own agent.yaml.
+type AgentSettings struct {
+	// Default is agent.yaml's keys for every node.
+	Default map[string]any `koanf:"default"`
+	// Nodes are agent.yaml's keys for one node each, by node id, over
+	// Default.
+	Nodes map[string]map[string]any `koanf:"nodes"`
+}
+
+// agentBootstrapKeys are what an agent needs before it can be told anything:
+// they stay in agent.yaml.
+var agentBootstrapKeys = []string{"node_id", "server", "servers", "hostname", "mgmt_addr", "tls"}
+
+// For is the settings for one node: Default with the node's own over it.
+// Empty when there are none.
+func (a AgentSettings) For(node string) map[string]any {
+	out := clone(stringKeys(a.Default))
+	if n, ok := a.Nodes[node]; ok {
+		merge(out, stringKeys(n))
+	}
+	return out
+}
+
+// Check refuses settings that set an agent's bootstrap keys, or that would
+// not load as an agent's configuration for some node.
+func (a AgentSettings) Check() error {
+	check := func(where string, m map[string]any) error {
+		for _, k := range agentBootstrapKeys {
+			if _, ok := m[k]; ok {
+				return fmt.Errorf("%s.%s stays in each node's agent.yaml: an agent needs it before it can be told anything", where, k)
+			}
+		}
+		return nil
+	}
+	if err := check("agents.default", stringKeys(a.Default)); err != nil {
+		return err
+	}
+	nodes := make([]string, 0, len(a.Nodes))
+	for n := range a.Nodes {
+		nodes = append(nodes, n)
+	}
+	sort.Strings(nodes)
+	for _, n := range nodes {
+		if err := check("agents.nodes."+n, stringKeys(a.Nodes[n])); err != nil {
+			return err
+		}
+	}
+	// Each node's settings as its agent would load them, over nothing: what
+	// its own file adds cannot be known here, and what is wrong in the
+	// settings alone is wrong on every node.
+	for _, n := range append([]string{""}, nodes...) {
+		b, err := Marshal(a.For(n))
+		if err != nil {
+			return err
+		}
+		if _, err := LoadAgentWith("", b); err != nil {
+			where := "agents.default"
+			if n != "" {
+				where = "agents.nodes." + n
+			}
+			return fmt.Errorf("%s: %w", where, err)
+		}
+	}
+	return nil
+}
+
+// AgentBootstrapKeys are the keys agent.yaml keeps, for saying so.
+func AgentBootstrapKeys() []string { return slices.Clone(agentBootstrapKeys) }
+
+// stringKeys turns YAML's map[any]any, wherever it hides, into the
+// map[string]any the rest of this package merges.
+func stringKeys(m map[string]any) map[string]any {
+	out := make(map[string]any, len(m))
+	for k, v := range m {
+		out[k] = normalize(v)
+	}
+	return out
+}
+
+func normalize(v any) any {
+	switch t := v.(type) {
+	case map[string]any:
+		return stringKeys(t)
+	case map[any]any:
+		m := make(map[string]any, len(t))
+		for k, vv := range t {
+			m[fmt.Sprint(k)] = normalize(vv)
+		}
+		return m
+	case []any:
+		out := make([]any, len(t))
+		for i, vv := range t {
+			out[i] = normalize(vv)
+		}
+		return out
+	}
+	return v
+}
@@ -0,0 +1,79 @@
+package config
+
+import (
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+)
+
+func TestANodesAgentSettingsAreTheDefaultWithItsOwnOver(t *testing.T) {
+	a := AgentSettings{
+		Default: map[string]any{"heartbeat_interval": "5s", "supervisor": map[string]any{"work_dir": "/srv/w", "x": 1}},
+		Nodes: map[string]map[string]any{
+			"cuda01": {"supervisor": map[string]any{"work_dir": "/fast/w"}, "labels": map[string]any{"gpu": "rtx6000"}},
+		},
+	}
+	got := a.For("cuda01")
+	sup := got["supervisor"].(map[string]any)
+	if sup["work_dir"] != "/fast/w" || sup["x"] != 1 || got["heartbeat_interval"] != "5s" || got["labels"] == nil {
+		t.Errorf("cuda01: %v", got)
+	}
+	if got := a.For("node01"); got["labels"] != nil || got["supervisor"].(map[string]any)["work_dir"] != "/srv/w" {
+		t.Errorf("node01: %v", got)
+	}
+	if got := (AgentSettings{}).For("x"); len(got) != 0 {
+		t.Errorf("none: %v", got)
+	}
+}
+
+func TestAgentSettingsKeepBootstrapKeysInTheFileAndMustLoad(t *testing.T) {
+	for _, a := range []AgentSettings{
+		{Default: map[string]any{"servers": []any{"x:7443"}}},
+		{Nodes: map[string]map[string]any{"n1": {"node_id": "other"}}},
+		{Nodes: map[string]map[string]any{"n1": {"tls": map[string]any{"ca": "/x"}}}},
+	} {
+		if err := a.Check(); err == nil || !strings.Contains(err.Error(), "stays in each node's agent.yaml") {
+			t.Errorf("%+v: %v", a, err)
+		}
+	}
+	// A fabric with no adapters would not load on the node it names.
+	bad := AgentSettings{Nodes: map[string]map[string]any{"n1": {"fabrics": []any{map[string]any{"name": "ib-a"}}}}}
+	if err := bad.Check(); err == nil || !strings.Contains(err.Error(), "agents.nodes.n1") || !strings.Contains(err.Error(), "no HCAs") {
+		t.Errorf("a fabric without HCAs: %v", err)
+	}
+	if err := (AgentSettings{Default: map[string]any{"heartbeat_interval": "3s"}}).Check(); err != nil {
+		t.Errorf("a plain setting: %v", err)
+	}
+}
+
+func TestTheClusterSettingsCarryAgentsAndRefuseBadOnes(t *testing.T) {
+	good, err := Parse([]byte("agents:\n  default:\n    heartbeat_interval: 3s\n  nodes:\n    cuda01:\n      labels: {gpu: rtx6000}\n"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	cfg, err := FromMap(good)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if cfg.Agents.For("cuda01")["labels"] == nil {
+		t.Errorf("agents not read: %+v", cfg.Agents)
+	}
+	bad, _ := Parse([]byte("agents:\n  default:\n    mgmt_addr: 10.0.0.1\n"))
+	if _, err := FromMap(bad); err == nil {
+		t.Error("a bootstrap key in the cluster's settings was taken")
+	}
+}
+
+func TestTheSettingsAreLaidOverTheAgentsFile(t *testing.T) {
+	p := filepath.Join(t.TempDir(), "agent.yaml")
+	_ = os.WriteFile(p, []byte("node_id: n1\nserver: cp:7443\nheartbeat_interval: 9s\nlabels: {rack: r1}\n"), 0o600)
+	cfg, err := LoadAgentWith(p, []byte("heartbeat_interval: 2s\nlabels: {gpu: a100}\n"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	if cfg.NodeID != "n1" || cfg.HeartbeatInterval != 2*time.Second || cfg.Labels["rack"] != "r1" || cfg.Labels["gpu"] != "a100" {
+		t.Errorf("loaded %+v", cfg)
+	}
+}
@@ -158,6 +158,13 @@ type Server struct {
 	// Firewall turns llmgw's firewall on every machine on or off.
 	Firewall FirewallSettings `koanf:"firewall"`

+	// Agents are the nodes' agent settings: agent.yaml's keys but those an
+	// agent needs to start, for every node (default) and for one
+	// (nodes.<id>). Sent to the agents as they change; an agent whose
+	// settings changed starts again on them, its engines serving
+	// throughout.
+	Agents AgentSettings `koanf:"agents"`
+
 	// FirewallNetworks are networks, besides those each machine reaches the
 	// cluster on, that llmgw's internal ports answer: a monitoring host on
 	// another subnet, nodes behind a router. Written into every machine's
@@ -716,7 +723,10 @@ type Endpoint struct {
 	Models []string `koanf:"models"`
 	// Engine names the inference engine, which decides what its metrics are
 	// called. Empty means vLLM's names.
-	Engine   string  `koanf:"engine"`
+	Engine string `koanf:"engine"`
+	// Workflow is the ComfyUI workflow for engine: comfyui, a built-in's
+	// name or its JSON; flux-schnell unless set.
+	Workflow string  `koanf:"workflow"`
 	Pool     string  `koanf:"pool"`
 	Weight   float64 `koanf:"weight"`
 	Priority int     `koanf:"priority"`
@@ -887,6 +897,9 @@ func load(b []byte) (Server, error) {
 // Validate rejects configurations that would otherwise fail confusingly at
 // request time.
 func (s Server) Validate() error {
+	if err := s.Agents.Check(); err != nil {
+		return err
+	}
 	for i, o := range s.EngineOverrides {
 		if o.GPUClass == "" && len(o.Nodes) == 0 && len(o.Labels) == 0 {
 			return fmt.Errorf("engine_overrides[%d] picks no nodes: give it a gpu_class, nodes or labels", i)
@@ -70,7 +70,11 @@ type API struct {
 	// guard is the guardrail policies' library and attachments.
 	guard *GuardAdmin
 	// kb is knowledge bases; nil without a store.
-	kb       *KBAdmin
+	kb *KBAdmin
+	// tenancy is organizations, keys and people; nil without a store.
+	tenancy TenancyStore
+	// train is adapter training; nil without a store and agents.
+	train    *TrainAdmin
 	settings *Settings
 	// restartAgents tells every connected agent to start again.
 	restartAgents func(reason string) []string
@@ -117,6 +121,8 @@ func (a *API) Register(mux Mux) {
 	mux.HandleFunc("POST /api/v1/models/{name}/resources", a.setResources)
 	a.registerQuotas(mux)
 	a.registerKB(mux)
+	a.registerTenancy(mux)
+	a.registerTrain(mux)
 	a.registerGuard(mux)
 	a.registerSettings(mux)
 	mux.HandleFunc("POST /api/v1/agents/restart", a.restartAgentsHandler)
@@ -1963,32 +1969,9 @@ func (a *API) setAdapters(w http.ResponseWriter, r *http.Request) {
 		writeError(w, http.StatusBadRequest, "no model "+name)
 		return
 	}
-	base, baseErr := gguf.KindFromURL(r.Context(), safedial.Client(safedial.ControlPlane, 30*time.Second), m.Source)
-	have := map[string]bool{}
-	for _, ad := range m.Adapters {
-		have[ad.Source] = true
-	}
-	// The file is read only where llama.cpp will load it: a vLLM adapter is
-	// a PEFT directory, which has no header to read, and vLLM itself refuses
-	// one that does not fit the model when it is loaded.
-	ggufs := slices.ContainsFunc(m.Profiles, func(p catalog.Profile) bool { return p.Engine == "llamacpp" })
-	for _, ad := range req.Adapters {
-		if have[ad.Source] || !ggufs {
-			continue // already checked when it was attached
-		}
-		h, err := gguf.KindFromURL(r.Context(), safedial.Client(safedial.ControlPlane, 30*time.Second), ad.Source)
-		switch {
-		case err != nil:
-			writeError(w, http.StatusBadRequest, fmt.Sprintf("adapter %s: %v", ad.Name, err))
-			return
-		case !h.IsAdapter():
-			writeError(w, http.StatusBadRequest, fmt.Sprintf("adapter %s: %s is a model, not a LoRA adapter", ad.Name, ad.Source))
-			return
-		case baseErr == nil && h.Architecture != base.Architecture:
-			writeError(w, http.StatusBadRequest, fmt.Sprintf("adapter %s is for %s models and %s is %s",
-				ad.Name, h.Architecture, m.Name, base.Architecture))
-			return
-		}
+	if err := checkAdapters(r.Context(), m, req.Adapters); err != nil {
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
 	}
 	out, err := a.models.SetAdapters(r.Context(), m.Name, req.Adapters, actorOf(r))
 	if err != nil {
@@ -2070,3 +2053,36 @@ func multiNodeReadiness(n inventory.Node) string {
 	}
 	return "ready"
 }
+
+// checkAdapters reads each adapter a model does not have yet, where
+// llama.cpp will load it, to refuse one that is not a LoRA adapter for the
+// model's architecture: such an adapter is a replica that fails to start,
+// found minutes later on a node.
+func checkAdapters(ctx context.Context, m catalog.Model, adapters []catalog.Adapter) error {
+	// A vLLM adapter is a PEFT directory, which has no header to read, and
+	// vLLM itself refuses one that does not fit the model when it is loaded.
+	if !slices.ContainsFunc(m.Profiles, func(p catalog.Profile) bool { return p.Engine == "llamacpp" }) {
+		return nil
+	}
+	client := safedial.Client(safedial.ControlPlane, 30*time.Second)
+	base, baseErr := gguf.KindFromURL(ctx, client, m.Source)
+	have := map[string]bool{}
+	for _, ad := range m.Adapters {
+		have[ad.Source] = true
+	}
+	for _, ad := range adapters {
+		if have[ad.Source] {
+			continue // checked when it was attached
+		}
+		h, err := gguf.KindFromURL(ctx, client, ad.Source)
+		switch {
+		case err != nil:
+			return fmt.Errorf("adapter %s: %v", ad.Name, err)
+		case !h.IsAdapter():
+			return fmt.Errorf("adapter %s: %s is a model, not a LoRA adapter", ad.Name, ad.Source)
+		case baseErr == nil && h.Architecture != base.Architecture:
+			return fmt.Errorf("adapter %s is for %s models and %s is %s", ad.Name, h.Architecture, m.Name, base.Architecture)
+		}
+	}
+	return nil
+}
@@ -30,6 +30,9 @@ type KBStore interface {
 	DeleteDocument(ctx context.Context, collection, name string) (bool, error)
 	SearchChunks(ctx context.Context, q store.KBSearch) ([]store.KBHit, error)
 	SetKBDefault(ctx context.Context, d store.KBDefault) error
+	CollectionChunks(ctx context.Context, c store.KBCollection, limit int) ([]store.KBHit, error)
+	CollectionsFor(ctx context.Context, e store.Effective, write bool) ([]store.KBCollection, error)
+	EffectiveFor(ctx context.Context, scope store.Scope, id, org string) (store.Effective, error)
 	KBDefaults(ctx context.Context) ([]store.KBDefault, error)
 }

@@ -161,6 +164,7 @@ func (a *API) registerKB(mux Mux) {
 	mux.HandleFunc("DELETE /api/v1/kb/grants", a.kbGrant(false))
 	mux.HandleFunc("POST /api/v1/kb/search", a.kbSearch)
 	mux.HandleFunc("POST /api/v1/kb/defaults", a.kbSetDefault)
+	mux.HandleFunc("POST /api/v1/kb/collections/{id}/dataset", a.kbDataset)
 }

 // kbReady answers for a gateway without knowledge bases, and says whether
@@ -495,7 +499,11 @@ func (a *API) kbSearch(w http.ResponseWriter, r *http.Request) {
 	// Said, not swallowed: a search that quietly lost its meaning half
 	// looks like a collection that does not know the answer.
 	if a.kb.Embed != nil {
-		v, err := a.kb.Embed(r.Context(), c.EmbedModel, in.Query)
+		question := in.Query
+		if m, ok := a.ctl.Catalog().Resolve(c.EmbedModel); ok {
+			question = m.QueryPrefix + question
+		}
+		v, err := a.kb.Embed(r.Context(), c.EmbedModel, question)
 		if err != nil {
 			writeError(w, http.StatusBadGateway, c.EmbedModel+" did not embed the question: "+err.Error())
 			return
@@ -0,0 +1,147 @@
+package adminapi
+
+import (
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"regexp"
+	"strings"
+)
+
+// Training data from a collection (BACKLOG F4): questions a person might
+// ask of each chunk and the answers it holds, written by a model of the
+// cluster, for people to read through before an adapter is trained on them.
+// What an adapter learns well is how the organisation's answers sound; what
+// they say is still the knowledge base's to give.
+
+// KBDatasetRequest asks for training data from a collection.
+type KBDatasetRequest struct {
+	// Model writes the questions and answers: a chat model of the catalog.
+	Model string `json:"model"`
+	// PerChunk is how many pairs each chunk gives; 8 when 0.
+	PerChunk int `json:"per_chunk,omitempty"`
+	// MaxChunks bounds how many chunks are read; 500 when 0.
+	MaxChunks int `json:"max_chunks,omitempty"`
+}
+
+// KBDatasetLine is one line of the answer: a pair, or why a chunk gave none.
+type KBDatasetLine struct {
+	Q   string `json:"q,omitempty"`
+	A   string `json:"a,omitempty"`
+	Doc string `json:"doc"`
+	Ord int    `json:"ord"`
+	// Collection is where the pair came from: an adapter trained on it
+	// belongs to a part of the tree that may read it.
+	Collection string `json:"collection,omitempty"`
+	Error      string `json:"error,omitempty"`
+}
+
+const kbDatasetPrompt = `From the passage of an organisation's document below, write %d different questions an employee might ask, each with a short, correct answer.
+Rules: use only what the passage says. Phrase the questions differently from each other (formal, everyday, short, indirect). Answers are one or two sentences and keep the passage's numbers exactly. Leave out e-mail addresses and people's names. Write in the passage's own language.
+Output: one JSON object per line, {"q": "...", "a": "..."}, and nothing else.
+
+DOCUMENT %s:
+%s`
+
+var thinkBlock = regexp.MustCompile(`(?s)<think>.*?</think>`)
+
+func (a *API) kbDataset(w http.ResponseWriter, r *http.Request) {
+	c, ok := a.kbCollection(w, r)
+	if !ok {
+		return
+	}
+	if a.replay == nil {
+		writeError(w, http.StatusNotImplemented, "no router here to ask a model")
+		return
+	}
+	var in KBDatasetRequest
+	if err := json.NewDecoder(r.Body).Decode(&in); err != nil {
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	}
+	m, found := a.ctl.Catalog().Resolve(in.Model)
+	if !found {
+		writeError(w, http.StatusBadRequest, "no model "+in.Model+" in the catalog")
+		return
+	}
+	if m.Capabilities.Embeddings {
+		writeError(w, http.StatusBadRequest, m.Name+" embeds; questions are written by a chat model")
+		return
+	}
+	per := in.PerChunk
+	if per <= 0 || per > 30 {
+		per = 8
+	}
+	limit := in.MaxChunks
+	if limit <= 0 || limit > 5000 {
+		limit = 500
+	}
+	chunks, err := a.kb.Store.CollectionChunks(r.Context(), c, limit)
+	if err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	w.Header().Set("Content-Type", "application/x-ndjson")
+	w.WriteHeader(http.StatusOK)
+	enc := json.NewEncoder(w)
+	flush := func() {
+		if f, ok := w.(http.Flusher); ok {
+			f.Flush()
+		}
+	}
+	for _, ch := range chunks {
+		if r.Context().Err() != nil {
+			return
+		}
+		body, _ := json.Marshal(map[string]any{"model": m.Name, "temperature": 0.7, "max_tokens": 200 * per,
+			"messages": []map[string]string{{"role": "user", "content": fmt.Sprintf(kbDatasetPrompt, per, ch.Document, ch.Body)}}})
+		status, out, _, err := a.replay(r.Context(), "/v1/chat/completions", m.Name, body)
+		line := KBDatasetLine{Doc: ch.Document, Ord: ch.Ord, Collection: c.ID}
+		if err == nil && status != http.StatusOK {
+			err = fmt.Errorf("%d: %s", status, strings.TrimSpace(string(out)))
+		}
+		if err != nil {
+			line.Error = err.Error()
+			_ = enc.Encode(line)
+			flush()
+			continue
+		}
+		pairs := datasetPairs(out)
+		if len(pairs) == 0 {
+			line.Error = "the model gave no pairs it could read"
+			_ = enc.Encode(line)
+		}
+		for _, p := range pairs {
+			p.Doc, p.Ord, p.Collection = ch.Document, ch.Ord, c.ID
+			_ = enc.Encode(p)
+		}
+		flush()
+	}
+}
+
+// datasetPairs reads the {"q","a"} lines out of a chat completion.
+func datasetPairs(completion []byte) []KBDatasetLine {
+	var res struct {
+		Choices []struct {
+			Message struct {
+				Content string `json:"content"`
+			} `json:"message"`
+		} `json:"choices"`
+	}
+	if json.Unmarshal(completion, &res) != nil || len(res.Choices) == 0 {
+		return nil
+	}
+	text := thinkBlock.ReplaceAllString(res.Choices[0].Message.Content, "")
+	var out []KBDatasetLine
+	for _, line := range strings.Split(text, "\n") {
+		line = strings.TrimSuffix(strings.TrimSpace(line), ",")
+		if !strings.HasPrefix(line, "{") || !strings.HasSuffix(line, "}") {
+			continue
+		}
+		var p struct{ Q, A string }
+		if json.Unmarshal([]byte(line), &p) == nil && strings.TrimSpace(p.Q) != "" && strings.TrimSpace(p.A) != "" {
+			out = append(out, KBDatasetLine{Q: strings.TrimSpace(p.Q), A: strings.TrimSpace(p.A)})
+		}
+	}
+	return out
+}
@@ -0,0 +1,91 @@
+package adminapi
+
+import (
+	"context"
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+func TestPairsAreReadOutOfWhatAModelWrote(t *testing.T) {
+	completion := `{"choices":[{"message":{"content":"<think>let me see</think>\n{\"q\": \"Babalık izni kaç gün?\", \"a\": \"10 iş günü.\"},\nnot json\n{\"q\": \"\", \"a\": \"empty question\"}\n{\"q\": \"Ne zaman?\", \"a\": \"Doğumdan sonra.\"}"}}]}`
+	got := datasetPairs([]byte(completion))
+	if len(got) != 2 || got[0].Q != "Babalık izni kaç gün?" || got[1].A != "Doğumdan sonra." {
+		t.Errorf("pairs %+v", got)
+	}
+	if datasetPairs([]byte(`{"choices":[]}`)) != nil {
+		t.Error("pairs from no choice")
+	}
+}
+
+func TestADatasetIsWrittenChunkByChunk(t *testing.T) {
+	dsn := os.Getenv("LLMGW_TEST_POSTGRES")
+	if dsn == "" {
+		t.Skip("set LLMGW_TEST_POSTGRES to a pgvector database to run this")
+	}
+	ctx := context.Background()
+	s, err := store.OpenPostgres(ctx, dsn)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer s.Close()
+	if s.KnowledgeAvailable() != nil {
+		t.Skip("no pgvector")
+	}
+	models := append(kbModels(), catalog.Model{Name: "writer", Source: "/w",
+		Profiles: []catalog.Profile{{Name: "cpu", GPUClass: "h100-80g", TP: 1}}})
+	a, _ := setup(t, models)
+	a.SetKB(&KBAdmin{Store: s})
+	asked := 0
+	a.SetReplayer(func(_ context.Context, path, model string, body []byte) (int, []byte, time.Duration, error) {
+		asked++
+		if path != "/v1/chat/completions" || model != "writer" || !strings.Contains(string(body), "Write in the passage") {
+			t.Errorf("asked %s %s %s", path, model, body)
+		}
+		if strings.Contains(string(body), "bozuk") {
+			return 200, []byte(`{"choices":[{"message":{"content":"sorry"}}]}`), 0, nil
+		}
+		return 200, []byte(`{"choices":[{"message":{"content":"{\"q\":\"Soru?\",\"a\":\"Cevap.\"}"}}]}`), 0, nil
+	})
+	org, _ := s.CreateOrg(ctx, "ds-"+strings.ToLower(store.NewID("")))
+	c, err := s.CreateCollection(ctx, store.KBCollection{OwnerScope: store.ScopeOrg, OwnerID: org, Name: "d", EmbedModel: "bge-m3", Dims: 3})
+	if err != nil {
+		t.Fatal(err)
+	}
+	for _, d := range []string{"a.md", "b.md"} {
+		body := "iyi metin"
+		if d == "b.md" {
+			body = "bozuk metin"
+		}
+		if _, _, err := s.PutDocument(ctx, c, store.KBDocument{Name: d, Digest: d}, []store.KBChunk{{Body: body, Embedding: []float32{1, 0, 0}}}); err != nil {
+			t.Fatal(err)
+		}
+	}
+	mux := http.NewServeMux()
+	a.Register(mux)
+	rec := httptest.NewRecorder()
+	mux.ServeHTTP(rec, httptest.NewRequest("POST", "/api/v1/kb/collections/"+c.ID+"/dataset", strings.NewReader(`{"model":"writer","per_chunk":3}`)))
+	var lines []KBDatasetLine
+	for _, l := range strings.Split(strings.TrimSpace(rec.Body.String()), "\n") {
+		var x KBDatasetLine
+		_ = json.Unmarshal([]byte(l), &x)
+		lines = append(lines, x)
+	}
+	if rec.Code != 200 || asked != 2 || len(lines) != 2 || lines[0].Q != "Soru?" || lines[0].Doc != "a.md" ||
+		lines[1].Doc != "b.md" || lines[1].Error == "" {
+		t.Errorf("%d asked %d: %+v", rec.Code, asked, lines)
+	}
+	// An embedding model writes no questions.
+	rec = httptest.NewRecorder()
+	mux.ServeHTTP(rec, httptest.NewRequest("POST", "/api/v1/kb/collections/"+c.ID+"/dataset", strings.NewReader(`{"model":"bge-m3"}`)))
+	if rec.Code != http.StatusBadRequest {
+		t.Errorf("an embedding model: %d", rec.Code)
+	}
+}
@@ -67,6 +67,12 @@ var PathParams = map[string]string{
 	"federation/{peer}": "The federation peer's name, as `llmgw ctl federation list` lists it.",
 	"holds/{domain}":    "The domain as level:name, rack:r12-07, as `llmgw ctl facility` lists it.",
 	"collections/{id}":  "The collection's ID, its name when that is unique, or ORG/NAME.",
+	"orgs/{id}":         "The organization's ID or name.",
+	"teams/{id}":        "The team's ID.",
+	"projects/{id}":     "The project's ID.",
+	"keys/{id}":         "The key's ID, as listed (not the key itself).",
+	"users/{id}":        "The person's ID or email.",
+	"artifacts/{kind}":  "What the job made: peft, a gzipped tar of the PEFT adapter, or gguf, the adapter for llama.cpp.",
 }

 // Routes is every endpoint Register mounts.
@@ -288,10 +294,77 @@ var Routes = []Route{
 		Description: "On an organization, team, project or key; the narrowest one with a default wins. No collections clears it. " +
 			"A request's own X-Llmgw-Knowledge overrides it, and X-Llmgw-Knowledge: none turns it off.",
 		Request: KBDefaultRequest{}, Response: KBDefaultView{}},
+	{Method: "POST", Path: "/api/v1/kb/collections/{id}/dataset", Summary: "Training data from a collection: questions and answers a chat model writes from each chunk",
+		Description: "NDJSON, a line per pair as each chunk is read, or a line with error for a chunk that gave none. For people to correct before an adapter is trained on it.",
+		Request:     KBDatasetRequest{}, Response: KBDatasetLine{}},
 	{Method: "POST", Path: "/api/v1/kb/search", Summary: "Try a search of a collection, as an administrator",
 		Description: "No tenancy applies: this is for checking what a collection holds and finds.",
 		Request:     KBSearchRequest{}, Response: []KBHitView{}},

+	// LoRA adapters trained on the nodes.
+	// Organizations, teams, projects, keys and people.
+	{Method: "GET", Path: "/api/v1/orgs", Summary: "The tenancy tree: every organization, its teams and their projects",
+		Response: []OrgView{}},
+	{Method: "POST", Path: "/api/v1/orgs", Summary: "Add an organization",
+		Request: TenancyCreateRequest{}, Response: TenancyItem{}, Status: http.StatusCreated},
+	{Method: "PATCH", Path: "/api/v1/orgs/{id}", Summary: "Rename an organization",
+		Request: TenancyChangeRequest{}, Response: TenancyItem{}},
+	{Method: "DELETE", Path: "/api/v1/orgs/{id}", Summary: "Remove an organization that has no teams",
+		Response: map[string]any{}},
+	{Method: "POST", Path: "/api/v1/teams", Summary: "Add a team to an organization",
+		Request: TenancyCreateRequest{}, Response: TenancyItem{}, Status: http.StatusCreated},
+	{Method: "PATCH", Path: "/api/v1/teams/{id}", Summary: "Rename a team, or move it to another organization",
+		Description: "Everything under it moves too, and from then on inherits the new organization's limits.",
+		Request:     TenancyChangeRequest{}, Response: TenancyItem{}},
+	{Method: "DELETE", Path: "/api/v1/teams/{id}", Summary: "Remove a team that has no projects",
+		Response: map[string]any{}},
+	{Method: "POST", Path: "/api/v1/projects", Summary: "Add a project to a team",
+		Request: TenancyCreateRequest{}, Response: TenancyItem{}, Status: http.StatusCreated},
+	{Method: "PATCH", Path: "/api/v1/projects/{id}", Summary: "Rename a project, or move it to another team",
+		Request: TenancyChangeRequest{}, Response: TenancyItem{}},
+	{Method: "DELETE", Path: "/api/v1/projects/{id}", Summary: "Remove a project with no people, keys or collections",
+		Description: "A project that ever had a key keeps it, revoked, because usage refers to it; rename such a project instead.",
+		Response:    map[string]any{}},
+	{Method: "GET", Path: "/api/v1/keys", Summary: "API keys, with their project and owner; never the keys themselves",
+		Query:    []Param{{Name: "project", Description: "Only this project's keys: ORG/TEAM/PROJECT or its ID."}},
+		Response: []KeyView{}},
+	{Method: "POST", Path: "/api/v1/keys", Summary: "Issue an API key",
+		Description: "The answer holds the key, once: only its hash is kept. A lifetime longer than a scope above it allows is refused.",
+		Request:     KeyCreateRequest{}, Response: KeyCreated{}, Status: http.StatusCreated},
+	{Method: "GET", Path: "/api/v1/keys/{id}", Summary: "One API key", Response: KeyView{}},
+	{Method: "DELETE", Path: "/api/v1/keys/{id}", Summary: "Revoke an API key",
+		Description: "A soft delete: usage refers to it. A gateway that has it cached accepts it until auth.cache_ttl.",
+		Response:    map[string]any{}},
+	{Method: "GET", Path: "/api/v1/users", Summary: "People, with where each is a member",
+		Query:    []Param{{Name: "state", Description: "Only people in this state: pending, active or disabled."}},
+		Response: []UserView{}},
+	{Method: "GET", Path: "/api/v1/users/{id}", Summary: "One person, with where they are a member", Response: UserView{}},
+	{Method: "POST", Path: "/api/v1/users/{id}/memberships", Summary: "Put a person in an organization, team or project",
+		Description: "Already there, only the role changes.",
+		Request:     MembershipRequest{}, Response: UserView{}},
+	{Method: "DELETE", Path: "/api/v1/users/{id}/memberships", Summary: "Take a person out of an organization, team or project",
+		Description: "Their keys in projects they no longer reach are revoked.",
+		Request:     MembershipRequest{}, Response: UserView{}},
+
+	{Method: "POST", Path: "/api/v1/train/jobs", Summary: "Queue a LoRA adapter's training on a node",
+		Description: "The node trains when its agent is there, nothing else trains on it, and a card has gpu_memory_mib free; " +
+			"the pairs are NDJSON of {\"q\", \"a\"}. An adapter learns how answers sound far better than what they say.",
+		Request: TrainRequest{}, Response: TrainJobView{}, Status: http.StatusCreated},
+	{Method: "GET", Path: "/api/v1/train/jobs", Summary: "Training jobs, newest first",
+		Query:    []Param{{Name: "limit", Description: "How many; 50 when absent.", Type: "integer"}},
+		Response: []TrainJobView{}},
+	{Method: "GET", Path: "/api/v1/train/jobs/{id}", Summary: "One training job: how it goes, and what it made",
+		Response: TrainJobView{}},
+	{Method: "POST", Path: "/api/v1/train/jobs/{id}/cancel", Summary: "Stop a training job",
+		Response: TrainJobView{}},
+	{Method: "DELETE", Path: "/api/v1/train/jobs/{id}", Summary: "Remove an ended training job and what it made",
+		Response: map[string]any{}},
+	{Method: "POST", Path: "/api/v1/train/jobs/{id}/publish", Summary: "Make what a finished job made the model's adapter",
+		Description: "Kept on the mirror — the GGUF for a model llama.cpp serves, the PEFT files for one vLLM serves — and added to the model's adapters after the same check as POST /api/v1/models/{name}/adapters.",
+		Response:    TrainPublishView{}},
+	{Method: "GET", Path: "/api/v1/train/jobs/{id}/artifacts/{kind}", Summary: "What a training job made: peft (a gzipped tar) or gguf",
+		Response: map[string]any{}},
+
 	{Method: "POST", Path: "/api/v1/agents/restart", Summary: "Have every node's agent start again; their engines keep serving",
 		Response: AgentRestartView{}},

@@ -0,0 +1,658 @@
+package adminapi
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"net/http"
+	"strings"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// The tenancy tree, keys and people's memberships, administered over the
+// API as `llmgw org`, `keys` and `users` do against the store: so that
+// they can be run from elsewhere, and kept in Terraform.
+
+// TenancyStore is the part of the store the tenancy endpoints need.
+type TenancyStore interface {
+	Tree(ctx context.Context) ([]store.TreeOrg, error)
+	CreateOrg(ctx context.Context, name string) (string, error)
+	CreateTeam(ctx context.Context, orgID, name string) (string, error)
+	CreateProject(ctx context.Context, teamID, name string) (string, error)
+	RenameOrg(ctx context.Context, id, name string) error
+	RenameTeam(ctx context.Context, id, name string) error
+	RenameProject(ctx context.Context, id, name string) error
+	MoveTeam(ctx context.Context, teamID, orgID string) error
+	MoveProject(ctx context.Context, projectID, teamID string) error
+	DeleteOrg(ctx context.Context, id string) error
+	DeleteTeam(ctx context.Context, id string) error
+	DeleteProject(ctx context.Context, id string) error
+
+	AllKeys(ctx context.Context) ([]store.KeyRow, error)
+	KeyByID(ctx context.Context, id string) (store.Key, error)
+	CreateKey(ctx context.Context, k store.Key) error
+	CreateUserKey(ctx context.Context, userID string, k store.Key) error
+	RevokeKey(ctx context.Context, id string, at time.Time) error
+
+	Users(ctx context.Context, state store.UserState) ([]store.User, error)
+	UserByID(ctx context.Context, id string) (store.User, error)
+	UserByEmail(ctx context.Context, email string) (store.User, error)
+	MembershipsOf(ctx context.Context, userID string) ([]store.Membership, error)
+	AddMembership(ctx context.Context, userID string, scope store.Scope, scopeID, role, by string, at time.Time) error
+	RemoveMembership(ctx context.Context, userID string, scope store.Scope, scopeID string, at time.Time) (int, error)
+}
+
+// SetTenancy enables the tenancy endpoints. The quota store must be set
+// too: names are resolved through it.
+func (a *API) SetTenancy(t TenancyStore) { a.tenancy = t }
+
+// OrgView is an organization with what is in it.
+type OrgView struct {
+	ID      string     `json:"id"`
+	Name    string     `json:"name"`
+	Members int        `json:"members"`
+	Teams   []TeamView `json:"teams"`
+}
+
+// TeamView is a team with its projects.
+type TeamView struct {
+	ID       string        `json:"id"`
+	Name     string        `json:"name"`
+	Org      string        `json:"org"`
+	Members  int           `json:"members"`
+	Projects []ProjectView `json:"projects"`
+}
+
+// ProjectView is a project, with how many people and keys are in it.
+type ProjectView struct {
+	ID   string `json:"id"`
+	Name string `json:"name"`
+	Team string `json:"team"`
+	// Path is ORG/TEAM/PROJECT by name.
+	Path       string `json:"path"`
+	Users      int    `json:"users"`
+	Keys       int    `json:"keys"`
+	ActiveKeys int    `json:"active_keys"`
+}
+
+// TenancyCreateRequest adds an organization, a team or a project.
+type TenancyCreateRequest struct {
+	Name string `json:"name"`
+	// In is the organization a team goes in, or the team a project goes in
+	// (ORG/TEAM), by name or ID; nothing for an organization.
+	In string `json:"in,omitempty"`
+}
+
+// TenancyChangeRequest renames an organization, team or project, or moves
+// a team to another organization or a project to another team.
+type TenancyChangeRequest struct {
+	Name string `json:"name,omitempty"`
+	// In is where it moves to, by name or ID; not for an organization.
+	In string `json:"in,omitempty"`
+}
+
+// TenancyItem is an organization, team or project added or changed.
+type TenancyItem struct {
+	ID   string `json:"id"`
+	Name string `json:"name,omitempty"`
+	// In is the organization a team is in, or the team a project is in, by
+	// ID.
+	In string `json:"in,omitempty"`
+}
+
+// KeyView is a key as listed: never the key itself, which is not kept.
+type KeyView struct {
+	ID   string `json:"id"`
+	Name string `json:"name"`
+	// Prefix is the key's first characters, to tell it apart.
+	Prefix  string `json:"prefix"`
+	Project string `json:"project,omitempty"`
+	// ProjectPath is ORG/TEAM/PROJECT, or empty for a personal key.
+	ProjectPath    string    `json:"project_path,omitempty"`
+	User           string    `json:"user,omitempty"`
+	UserEmail      string    `json:"user_email,omitempty"`
+	Band           string    `json:"band,omitempty"`
+	RoutingProfile string    `json:"routing_profile,omitempty"`
+	State          string    `json:"state"`
+	CreatedAt      time.Time `json:"created_at"`
+	ExpiresAt      time.Time `json:"expires_at,omitzero"`
+	RevokedAt      time.Time `json:"revoked_at,omitzero"`
+	LastUsedAt     time.Time `json:"last_used_at,omitzero"`
+}
+
+// KeyCreateRequest issues a key: a project's, a person's in a project they
+// are a member of, or a person's own with no project.
+type KeyCreateRequest struct {
+	Name string `json:"name"`
+	// Project is ORG/TEAM/PROJECT or its ID.
+	Project string `json:"project,omitempty"`
+	// User is the person it belongs to, by ID or email.
+	User           string `json:"user,omitempty"`
+	Band           string `json:"band,omitempty"`
+	RoutingProfile string `json:"routing_profile,omitempty"`
+	// ExpiresIn is its lifetime, as 720h; none never expires, unless a
+	// scope above it limits how long keys live.
+	ExpiresIn string `json:"expires_in,omitempty"`
+}
+
+// KeyCreated is a new key, with the key itself: this answer is the only
+// time it is given.
+type KeyCreated struct {
+	KeyView
+	Secret string `json:"secret"`
+}
+
+// UserView is a person, with where they are a member.
+type UserView struct {
+	ID    string `json:"id"`
+	Email string `json:"email"`
+	Name  string `json:"name,omitempty"`
+	State string `json:"state"`
+	// Role is the person's part in running the gateway, if any: viewer,
+	// operator, billing or admin.
+	Role        string           `json:"role,omitempty"`
+	Memberships []MembershipView `json:"memberships"`
+	CreatedAt   time.Time        `json:"created_at"`
+	LastLoginAt time.Time        `json:"last_login_at,omitzero"`
+}
+
+// MembershipView is one place a person belongs.
+type MembershipView struct {
+	Target string    `json:"target"`
+	Path   string    `json:"path,omitempty"`
+	Role   string    `json:"role"`
+	At     time.Time `json:"at"`
+	By     string    `json:"by,omitempty"`
+}
+
+// MembershipRequest puts a person in an organization, team or project, or
+// with DELETE takes them out.
+type MembershipRequest struct {
+	// Target is org:NAME, team:ORG/TEAM or project:ORG/TEAM/PROJECT, or the
+	// ID after the colon.
+	Target string `json:"target"`
+	// Role is member or manager; member when absent. Ignored by DELETE.
+	Role string `json:"role,omitempty"`
+}
+
+func (a *API) registerTenancy(mux Mux) {
+	mux.HandleFunc("GET /api/v1/orgs", a.tenancyTree)
+	for _, level := range []store.Scope{store.ScopeOrg, store.ScopeTeam, store.ScopeProject} {
+		path := "/api/v1/" + string(level) + "s"
+		mux.HandleFunc("POST "+path, a.tenancyCreate(level))
+		mux.HandleFunc("PATCH "+path+"/{id}", a.tenancyChange(level))
+		mux.HandleFunc("DELETE "+path+"/{id}", a.tenancyDelete(level))
+	}
+	mux.HandleFunc("GET /api/v1/keys", a.keyList)
+	mux.HandleFunc("POST /api/v1/keys", a.keyCreate)
+	mux.HandleFunc("GET /api/v1/keys/{id}", a.keyGet)
+	mux.HandleFunc("DELETE /api/v1/keys/{id}", a.keyRevoke)
+	mux.HandleFunc("GET /api/v1/users", a.userList)
+	mux.HandleFunc("GET /api/v1/users/{id}", a.userGet)
+	mux.HandleFunc("POST /api/v1/users/{id}/memberships", a.membership(true))
+	mux.HandleFunc("DELETE /api/v1/users/{id}/memberships", a.membership(false))
+}
+
+// tenancyReady answers for a control plane without the store, and says
+// whether it did.
+func (a *API) tenancyReady(w http.ResponseWriter) bool {
+	if a.tenancy == nil || a.quotas == nil {
+		writeError(w, http.StatusNotImplemented, "organizations, keys and people need the store, which this control plane runs without")
+		return false
+	}
+	return true
+}
+
+// tenancyStatus is the status a store error answers with.
+func tenancyStatus(err error) int {
+	switch {
+	case errors.Is(err, store.ErrNotFound):
+		return http.StatusNotFound
+	case errors.Is(err, store.ErrDuplicateName), errors.Is(err, store.ErrNotEmpty):
+		return http.StatusConflict
+	}
+	return http.StatusBadRequest
+}
+
+func (a *API) tenancyTree(w http.ResponseWriter, r *http.Request) {
+	if !a.tenancyReady(w) {
+		return
+	}
+	tree, err := a.tenancy.Tree(r.Context())
+	if err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	out := []OrgView{}
+	for _, o := range tree {
+		ov := OrgView{ID: o.ID, Name: o.Name, Members: o.Members, Teams: []TeamView{}}
+		for _, t := range o.Teams {
+			tv := TeamView{ID: t.ID, Name: t.Name, Org: o.ID, Members: t.Members, Projects: []ProjectView{}}
+			for _, p := range t.Projects {
+				tv.Projects = append(tv.Projects, ProjectView{ID: p.ID, Name: p.Name, Team: t.ID,
+					Path: o.Name + "/" + t.Name + "/" + p.Name, Users: p.Users, Keys: p.Keys, ActiveKeys: p.ActiveKeys})
+			}
+			ov.Teams = append(ov.Teams, tv)
+		}
+		out = append(out, ov)
+	}
+	writeJSON(w, http.StatusOK, out)
+}
+
+// parentOf is the level something at level sits in.
+func parentOf(level store.Scope) store.Scope {
+	if level == store.ScopeProject {
+		return store.ScopeTeam
+	}
+	return store.ScopeOrg
+}
+
+// levelNoun is a level as a sentence says it.
+func levelNoun(level store.Scope) string {
+	switch level {
+	case store.ScopeOrg:
+		return "an organization"
+	case store.ScopeTeam:
+		return "a team"
+	}
+	return "a project"
+}
+
+func (a *API) tenancyCreate(level store.Scope) http.HandlerFunc {
+	return func(w http.ResponseWriter, r *http.Request) {
+		if !a.tenancyReady(w) {
+			return
+		}
+		var in TenancyCreateRequest
+		if err := json.NewDecoder(r.Body).Decode(&in); err != nil {
+			writeError(w, http.StatusBadRequest, err.Error())
+			return
+		}
+		in.Name = strings.TrimSpace(in.Name)
+		if in.Name == "" {
+			writeError(w, http.StatusBadRequest, "a name is needed")
+			return
+		}
+		ctx := r.Context()
+		var id, parent string
+		var err error
+		if level != store.ScopeOrg {
+			if in.In == "" {
+				writeError(w, http.StatusBadRequest, fmt.Sprintf("%s goes in %s: give in", levelNoun(level), levelNoun(parentOf(level))))
+				return
+			}
+			if _, parent, _, err = a.resolveScope(ctx, string(parentOf(level))+":"+in.In); err != nil {
+				writeError(w, http.StatusNotFound, err.Error())
+				return
+			}
+		}
+		switch level {
+		case store.ScopeOrg:
+			id, err = a.tenancy.CreateOrg(ctx, in.Name)
+		case store.ScopeTeam:
+			id, err = a.tenancy.CreateTeam(ctx, parent, in.Name)
+		default:
+			id, err = a.tenancy.CreateProject(ctx, parent, in.Name)
+		}
+		if errors.Is(err, store.ErrDuplicateName) {
+			writeError(w, http.StatusConflict, fmt.Sprintf("there is already %s named %q there", levelNoun(level), in.Name))
+			return
+		}
+		if err != nil {
+			writeError(w, tenancyStatus(err), err.Error())
+			return
+		}
+		writeJSON(w, http.StatusCreated, TenancyItem{ID: id, Name: in.Name, In: parent})
+	}
+}
+
+func (a *API) tenancyChange(level store.Scope) http.HandlerFunc {
+	return func(w http.ResponseWriter, r *http.Request) {
+		if !a.tenancyReady(w) {
+			return
+		}
+		var in TenancyChangeRequest
+		if err := json.NewDecoder(r.Body).Decode(&in); err != nil {
+			writeError(w, http.StatusBadRequest, err.Error())
+			return
+		}
+		ctx := r.Context()
+		_, id, _, err := a.resolveScope(ctx, string(level)+":"+r.PathValue("id"))
+		if err != nil {
+			writeError(w, http.StatusNotFound, err.Error())
+			return
+		}
+		if in.Name == "" && in.In == "" {
+			writeError(w, http.StatusBadRequest, "nothing to change: give name, in or both")
+			return
+		}
+		out := TenancyItem{ID: id}
+		if in.In != "" {
+			if level == store.ScopeOrg {
+				writeError(w, http.StatusBadRequest, "an organization is in nothing; only a team or a project moves")
+				return
+			}
+			_, to, _, err := a.resolveScope(ctx, string(parentOf(level))+":"+in.In)
+			if err != nil {
+				writeError(w, http.StatusNotFound, err.Error())
+				return
+			}
+			if level == store.ScopeTeam {
+				err = a.tenancy.MoveTeam(ctx, id, to)
+			} else {
+				err = a.tenancy.MoveProject(ctx, id, to)
+			}
+			if err != nil {
+				writeError(w, tenancyStatus(err), err.Error())
+				return
+			}
+			out.In = to
+		}
+		if name := strings.TrimSpace(in.Name); name != "" {
+			switch level {
+			case store.ScopeOrg:
+				err = a.tenancy.RenameOrg(ctx, id, name)
+			case store.ScopeTeam:
+				err = a.tenancy.RenameTeam(ctx, id, name)
+			default:
+				err = a.tenancy.RenameProject(ctx, id, name)
+			}
+			if err != nil {
+				writeError(w, tenancyStatus(err), err.Error())
+				return
+			}
+			out.Name = name
+		}
+		writeJSON(w, http.StatusOK, out)
+	}
+}
+
+func (a *API) tenancyDelete(level store.Scope) http.HandlerFunc {
+	return func(w http.ResponseWriter, r *http.Request) {
+		if !a.tenancyReady(w) {
+			return
+		}
+		ctx := r.Context()
+		_, id, _, err := a.resolveScope(ctx, string(level)+":"+r.PathValue("id"))
+		if err != nil {
+			writeError(w, http.StatusNotFound, err.Error())
+			return
+		}
+		switch level {
+		case store.ScopeOrg:
+			err = a.tenancy.DeleteOrg(ctx, id)
+		case store.ScopeTeam:
+			err = a.tenancy.DeleteTeam(ctx, id)
+		default:
+			err = a.tenancy.DeleteProject(ctx, id)
+		}
+		if err != nil {
+			// Only what is empty goes: the store says what is still in it.
+			writeError(w, tenancyStatus(err), err.Error())
+			return
+		}
+		writeJSON(w, http.StatusOK, map[string]any{"id": id, "deleted": true})
+	}
+}
+
+// keyState is a key's state at an instant.
+func keyState(k store.Key, now time.Time) string {
+	switch {
+	case !k.RevokedAt.IsZero():
+		return "revoked"
+	case !k.ExpiresAt.IsZero() && !now.Before(k.ExpiresAt):
+		return "expired"
+	}
+	return "active"
+}
+
+func keyView(k store.Key, path, email string, now time.Time) KeyView {
+	// The store says "Acme / ml / search"; a path given to the API is
+	// written Acme/ml/search, and is answered the same way.
+	path = strings.ReplaceAll(path, " / ", "/")
+	return KeyView{ID: k.ID, Name: k.Name, Prefix: k.Prefix, Project: k.ProjectID, ProjectPath: path,
+		User: k.UserID, UserEmail: email, Band: k.Band, RoutingProfile: k.RoutingProfile, State: keyState(k, now),
+		CreatedAt: k.CreatedAt, ExpiresAt: k.ExpiresAt, RevokedAt: k.RevokedAt, LastUsedAt: k.LastUsedAt}
+}
+
+func (a *API) keyList(w http.ResponseWriter, r *http.Request) {
+	if !a.tenancyReady(w) {
+		return
+	}
+	project := ""
+	if p := r.URL.Query().Get("project"); p != "" {
+		var err error
+		if _, project, _, err = a.resolveScope(r.Context(), "project:"+p); err != nil {
+			writeError(w, http.StatusNotFound, err.Error())
+			return
+		}
+	}
+	rows, err := a.tenancy.AllKeys(r.Context())
+	if err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	now := time.Now()
+	out := []KeyView{}
+	for _, k := range rows {
+		if project == "" || k.ProjectID == project {
+			out = append(out, keyView(k.Key, k.ProjectPath, k.OwnerEmail, now))
+		}
+	}
+	writeJSON(w, http.StatusOK, out)
+}
+
+func (a *API) keyGet(w http.ResponseWriter, r *http.Request) {
+	if !a.tenancyReady(w) {
+		return
+	}
+	rows, err := a.tenancy.AllKeys(r.Context())
+	if err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	for _, k := range rows {
+		if k.ID == r.PathValue("id") {
+			writeJSON(w, http.StatusOK, keyView(k.Key, k.ProjectPath, k.OwnerEmail, time.Now()))
+			return
+		}
+	}
+	writeError(w, http.StatusNotFound, "no key "+r.PathValue("id"))
+}
+
+// findUser reads a person by ID or email.
+func (a *API) findUser(ctx context.Context, ref string) (store.User, error) {
+	ref = strings.TrimSpace(ref)
+	if u, err := a.tenancy.UserByID(ctx, ref); err == nil {
+		return u, nil
+	}
+	if strings.Contains(ref, "@") {
+		if u, err := a.tenancy.UserByEmail(ctx, ref); err == nil {
+			return u, nil
+		}
+	}
+	return store.User{}, fmt.Errorf("%w: no person %q", store.ErrNotFound, ref)
+}
+
+func (a *API) keyCreate(w http.ResponseWriter, r *http.Request) {
+	if !a.tenancyReady(w) {
+		return
+	}
+	var in KeyCreateRequest
+	if err := json.NewDecoder(r.Body).Decode(&in); err != nil {
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	}
+	ctx := r.Context()
+	if strings.TrimSpace(in.Name) == "" {
+		writeError(w, http.StatusBadRequest, "a key needs a name")
+		return
+	}
+	if in.Project == "" && in.User == "" {
+		writeError(w, http.StatusBadRequest, "a key belongs to a project, a person, or a person in a project: give project, user or both")
+		return
+	}
+	var owner store.User
+	var err error
+	if in.User != "" {
+		if owner, err = a.findUser(ctx, in.User); err != nil {
+			writeError(w, http.StatusNotFound, err.Error())
+			return
+		}
+	}
+	project, path := "", ""
+	if in.Project != "" {
+		if _, project, path, err = a.resolveScope(ctx, "project:"+in.Project); err != nil {
+			writeError(w, http.StatusNotFound, err.Error())
+			return
+		}
+	}
+	var expires time.Duration
+	if in.ExpiresIn != "" {
+		if expires, err = time.ParseDuration(in.ExpiresIn); err != nil || expires <= 0 {
+			writeError(w, http.StatusBadRequest, fmt.Sprintf("expires_in %q is not a positive duration such as 720h", in.ExpiresIn))
+			return
+		}
+	}
+	secret, k, err := store.NewKey(store.NewID("key-"), strings.TrimSpace(in.Name), project)
+	if err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	k.CreatedAt = time.Now()
+	k.Band, k.RoutingProfile = in.Band, in.RoutingProfile
+	if expires > 0 {
+		k.ExpiresAt = k.CreatedAt.Add(expires)
+	}
+	if owner.ID != "" {
+		err = a.tenancy.CreateUserKey(ctx, owner.ID, k)
+	} else {
+		err = a.tenancy.CreateKey(ctx, k)
+	}
+	switch {
+	case errors.Is(err, store.ErrNotMember):
+		writeError(w, http.StatusConflict, fmt.Sprintf("%s is not a member of %s; add the membership first", owner.Email, path))
+		return
+	case errors.Is(err, store.ErrKeyTTL):
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	case err != nil:
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	k.UserID = owner.ID
+	writeJSON(w, http.StatusCreated, KeyCreated{KeyView: keyView(k, path, owner.Email, time.Now()), Secret: secret})
+}
+
+func (a *API) keyRevoke(w http.ResponseWriter, r *http.Request) {
+	if !a.tenancyReady(w) {
+		return
+	}
+	id := r.PathValue("id")
+	if _, err := a.tenancy.KeyByID(r.Context(), id); err != nil {
+		writeError(w, tenancyStatus(err), err.Error())
+		return
+	}
+	if err := a.tenancy.RevokeKey(r.Context(), id, time.Now()); err != nil {
+		writeError(w, tenancyStatus(err), err.Error())
+		return
+	}
+	// A key held in a gateway's cache keeps working until auth.cache_ttl.
+	writeJSON(w, http.StatusOK, map[string]any{"id": id, "revoked": true})
+}
+
+func (a *API) userView(ctx context.Context, u store.User) UserView {
+	role := u.Role
+	if u.IsAdmin {
+		role = "admin"
+	}
+	v := UserView{ID: u.ID, Email: u.Email, Name: u.Name, State: string(u.State), Role: role,
+		Memberships: []MembershipView{}, CreatedAt: u.CreatedAt, LastLoginAt: u.LastLoginAt}
+	if ms, err := a.tenancy.MembershipsOf(ctx, u.ID); err == nil {
+		for _, m := range ms {
+			v.Memberships = append(v.Memberships, MembershipView{Target: string(m.Scope) + ":" + m.ScopeID,
+				Path: m.Path, Role: m.Role, At: m.AddedAt, By: m.AddedBy})
+		}
+	}
+	return v
+}
+
+func (a *API) userList(w http.ResponseWriter, r *http.Request) {
+	if !a.tenancyReady(w) {
+		return
+	}
+	us, err := a.tenancy.Users(r.Context(), store.UserState(r.URL.Query().Get("state")))
+	if err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	out := []UserView{}
+	for _, u := range us {
+		out = append(out, a.userView(r.Context(), u))
+	}
+	writeJSON(w, http.StatusOK, out)
+}
+
+func (a *API) userGet(w http.ResponseWriter, r *http.Request) {
+	if !a.tenancyReady(w) {
+		return
+	}
+	u, err := a.findUser(r.Context(), r.PathValue("id"))
+	if err != nil {
+		writeError(w, http.StatusNotFound, err.Error())
+		return
+	}
+	writeJSON(w, http.StatusOK, a.userView(r.Context(), u))
+}
+
+func (a *API) membership(add bool) http.HandlerFunc {
+	return func(w http.ResponseWriter, r *http.Request) {
+		if !a.tenancyReady(w) {
+			return
+		}
+		var in MembershipRequest
+		if err := json.NewDecoder(r.Body).Decode(&in); err != nil {
+			writeError(w, http.StatusBadRequest, err.Error())
+			return
+		}
+		ctx := r.Context()
+		u, err := a.findUser(ctx, r.PathValue("id"))
+		if err != nil {
+			writeError(w, http.StatusNotFound, err.Error())
+			return
+		}
+		scope, id, _, err := a.resolveScope(ctx, in.Target)
+		if err != nil {
+			writeError(w, http.StatusBadRequest, err.Error())
+			return
+		}
+		if add {
+			role := in.Role
+			if role == "" {
+				role = store.RoleMember
+			}
+			if role != store.RoleMember && role != store.RoleManager {
+				writeError(w, http.StatusBadRequest, fmt.Sprintf("role %q: member or manager", role))
+				return
+			}
+			if err := a.tenancy.AddMembership(ctx, u.ID, scope, id, role, actorOf(r), time.Now()); err != nil {
+				writeError(w, tenancyStatus(err), err.Error())
+				return
+			}
+		} else {
+			// The person's keys in projects they no longer reach are
+			// revoked with it, by the store.
+			if _, err := a.tenancy.RemoveMembership(ctx, u.ID, scope, id, time.Now()); err != nil {
+				if errors.Is(err, store.ErrNotFound) {
+					err = errors.New(u.Email + " is not a member there")
+				}
+				writeError(w, http.StatusNotFound, err.Error())
+				return
+			}
+		}
+		writeJSON(w, http.StatusOK, a.userView(ctx, u))
+	}
+}
@@ -0,0 +1,171 @@
+package adminapi
+
+import (
+	"context"
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+func TestTheTenancyTreeKeysAndMembershipsAreAdministered(t *testing.T) {
+	ctx := context.Background()
+	a, _ := setup(t, testModels())
+	s, err := store.OpenSQLite(ctx, filepath.Join(t.TempDir(), "s.db"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer s.Close()
+	a.SetQuotas(s)
+	a.SetTenancy(s)
+	mux := http.NewServeMux()
+	a.Register(mux)
+	do := func(method, path, body string, out any) int {
+		t.Helper()
+		rec := httptest.NewRecorder()
+		mux.ServeHTTP(rec, httptest.NewRequest(method, path, strings.NewReader(body)))
+		if out != nil && rec.Code < 300 {
+			if err := json.Unmarshal(rec.Body.Bytes(), out); err != nil {
+				t.Fatalf("%s %s: %v (%s)", method, path, err, rec.Body.String())
+			}
+		}
+		if rec.Code >= 300 {
+			t.Logf("%s %s: %d %s", method, path, rec.Code, strings.TrimSpace(rec.Body.String()))
+		}
+		return rec.Code
+	}
+	var made TenancyItem
+	if c := do("POST", "/api/v1/orgs", `{"name":"Acme"}`, &made); c != http.StatusCreated {
+		t.Fatalf("adding an organization: %d", c)
+	}
+	acme := made.ID
+	if c := do("POST", "/api/v1/orgs", `{"name":"Acme"}`, nil); c != http.StatusConflict {
+		t.Errorf("a second Acme: %d", c)
+	}
+	if c := do("POST", "/api/v1/teams", `{"name":"hr"}`, nil); c != http.StatusBadRequest {
+		t.Errorf("a team in nothing: %d", c)
+	}
+	do("POST", "/api/v1/teams", `{"name":"hr","in":"Acme"}`, &made)
+	hr := made.ID
+	if c := do("POST", "/api/v1/projects", `{"name":"crm","in":"Acme/hr"}`, &made); c != http.StatusCreated || made.In != hr {
+		t.Fatalf("adding a project: %d %v", c, made)
+	}
+	crm := made.ID
+	do("POST", "/api/v1/orgs", `{"name":"Other"}`, &made)
+	other := made.ID
+
+	// Renamed, and moved with what is in it.
+	if c := do("PATCH", "/api/v1/projects/"+crm, `{"name":"crm2"}`, nil); c != http.StatusOK {
+		t.Errorf("renaming: %d", c)
+	}
+	if c := do("PATCH", "/api/v1/teams/"+hr, `{"in":"Other"}`, nil); c != http.StatusOK {
+		t.Errorf("moving: %d", c)
+	}
+	if c := do("PATCH", "/api/v1/orgs/"+acme, `{"in":"Other"}`, nil); c != http.StatusBadRequest {
+		t.Errorf("moving an organization: %d", c)
+	}
+	var tree []OrgView
+	do("GET", "/api/v1/orgs", "", &tree)
+	if len(tree) != 2 || tree[1].ID != other || len(tree[1].Teams) != 1 ||
+		tree[1].Teams[0].Projects[0].Path != "Other/hr/crm2" {
+		t.Fatalf("the tree is %+v", tree)
+	}
+
+	// A project's key: given once, listed without it.
+	var key KeyCreated
+	rec := httptest.NewRecorder()
+	mux.ServeHTTP(rec, httptest.NewRequest("POST", "/api/v1/keys",
+		strings.NewReader(`{"name":"svc","project":"Other/hr/crm2","expires_in":"24h"}`)))
+	if rec.Code != http.StatusCreated {
+		t.Fatalf("issuing a key: %d %s", rec.Code, rec.Body.String())
+	}
+	_ = json.Unmarshal(rec.Body.Bytes(), &key)
+	if !strings.HasPrefix(key.Secret, "sk-llmgw-") || key.Project != crm || key.ExpiresAt.IsZero() {
+		t.Errorf("the key issued: %+v", key)
+	}
+	if k, err := s.KeyByHash(ctx, store.HashKey(key.Secret)); err != nil || k.ID != key.ID {
+		t.Errorf("the key does not authenticate: %v", err)
+	}
+	var keys []KeyView
+	do("GET", "/api/v1/keys?project="+crm, "", &keys)
+	if len(keys) != 1 || keys[0].ID != key.ID || keys[0].State != "active" || keys[0].ProjectPath != "Other/hr/crm2" {
+		t.Errorf("the project's keys: %+v", keys)
+	}
+	raw := httptest.NewRecorder()
+	mux.ServeHTTP(raw, httptest.NewRequest("GET", "/api/v1/keys", nil))
+	if strings.Contains(raw.Body.String(), key.Secret) || strings.Contains(raw.Body.String(), "secret") {
+		t.Error("a listing holds the key")
+	}
+	if c := do("POST", "/api/v1/keys", `{"name":"x","project":"Other/hr/crm2","expires_in":"soon"}`, nil); c != http.StatusBadRequest {
+		t.Errorf("a lifetime that is not one: %d", c)
+	}
+	if c := do("DELETE", "/api/v1/projects/"+crm, "", nil); c != http.StatusConflict {
+		t.Errorf("removing a project with a key: %d", c)
+	}
+	if c := do("DELETE", "/api/v1/keys/"+key.ID, "", nil); c != http.StatusOK {
+		t.Errorf("revoking: %d", c)
+	}
+	var one KeyView
+	do("GET", "/api/v1/keys/"+key.ID, "", &one)
+	if one.State != "revoked" {
+		t.Errorf("after revoking, %+v", one)
+	}
+
+	// A person: a key in a project only as its member, revoked when they
+	// leave it.
+	if err := s.CreateUser(ctx, store.User{ID: "u1", Email: "ayse@example.com", PasswordHash: "x",
+		State: store.UserActive, CreatedAt: time.Now()}); err != nil {
+		t.Fatal(err)
+	}
+	if c := do("POST", "/api/v1/keys", `{"name":"mine","project":"`+crm+`","user":"ayse@example.com"}`, nil); c != http.StatusConflict {
+		t.Errorf("a key where its person is no member: %d", c)
+	}
+	var u UserView
+	if c := do("POST", "/api/v1/users/ayse@example.com/memberships", `{"target":"project:Other/hr/crm2"}`, &u); c != http.StatusOK ||
+		len(u.Memberships) != 1 || u.Memberships[0].Target != "project:"+crm || u.Memberships[0].Role != "member" {
+		t.Fatalf("joining: %d %+v", c, u)
+	}
+	var mine KeyCreated
+	rec = httptest.NewRecorder()
+	mux.ServeHTTP(rec, httptest.NewRequest("POST", "/api/v1/keys",
+		strings.NewReader(`{"name":"mine","project":"`+crm+`","user":"u1"}`)))
+	_ = json.Unmarshal(rec.Body.Bytes(), &mine)
+	if rec.Code != http.StatusCreated || mine.User != "u1" {
+		t.Fatalf("a member's key: %d %s", rec.Code, rec.Body.String())
+	}
+	if c := do("DELETE", "/api/v1/users/u1/memberships", `{"target":"project:`+crm+`"}`, &u); c != http.StatusOK || len(u.Memberships) != 0 {
+		t.Errorf("leaving: %d %+v", c, u)
+	}
+	do("GET", "/api/v1/keys/"+mine.ID, "", &one)
+	if one.State != "revoked" {
+		t.Errorf("the key of a person who left: %+v", one)
+	}
+	if c := do("DELETE", "/api/v1/users/u1/memberships", `{"target":"project:`+crm+`"}`, nil); c != http.StatusNotFound {
+		t.Errorf("leaving twice: %d", c)
+	}
+	var people []UserView
+	do("GET", "/api/v1/users", "", &people)
+	if len(people) != 1 || people[0].Email != "ayse@example.com" {
+		t.Errorf("people: %+v", people)
+	}
+
+	// Only what is empty is removed.
+	if c := do("DELETE", "/api/v1/orgs/Acme", "", nil); c != http.StatusOK {
+		t.Errorf("removing the empty Acme by name: %d", c)
+	}
+	if c := do("DELETE", "/api/v1/orgs/"+other, "", nil); c != http.StatusConflict {
+		t.Errorf("removing an organization with a team: %d", c)
+	}
+}
+
+func TestTenancyNeedsTheStore(t *testing.T) {
+	a, _ := setup(t, testModels())
+	if c := call(t, a, "GET", "/api/v1/orgs", "", nil); c != http.StatusNotImplemented {
+		t.Errorf("without a store: %d", c)
+	}
+}
@@ -0,0 +1,559 @@
+package adminapi
+
+import (
+	"archive/tar"
+	"bufio"
+	"bytes"
+	"compress/gzip"
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"net/http"
+	"regexp"
+	"slices"
+	"strconv"
+	"strings"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/train"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// Training LoRA adapters (BACKLOG F4), administered: a job names the model
+// the adapter is for, the weights to train on, the node and how much of a
+// card it needs, and the pairs; the node trains when it can, and what it
+// made is kept here to download or publish.
+
+// TrainStore is where jobs are kept.
+type TrainStore interface {
+	CreateTrainJob(ctx context.Context, j store.TrainJob) error
+	TrainJobs(ctx context.Context, limit int) ([]store.TrainJob, error)
+	TrainJob(ctx context.Context, id string) (store.TrainJob, error)
+	TrainArtifact(ctx context.Context, job, kind string) ([]byte, error)
+	TrainArtifactKinds(ctx context.Context, job string) ([]string, error)
+	DeleteTrainJob(ctx context.Context, id string) error
+}
+
+// TrainAdmin is what the training endpoints need.
+type TrainAdmin struct {
+	Store   TrainStore
+	Service *train.Service
+	// Mirror keeps what a job made where the nodes fetch weights; nil
+	// leaves publishing off.
+	Mirror TrainMirror
+}
+
+// TrainMirror is the model mirror, as publishing needs it.
+type TrainMirror interface {
+	Upload(ctx context.Context, model, version, file string, body []byte, source string) (mirror.FetchResponse, error)
+}
+
+// TrainPublishView is an adapter published: where the nodes fetch it from,
+// and what to do for replicas to load it.
+type TrainPublishView struct {
+	Adapter string `json:"adapter"`
+	Model   string `json:"model"`
+	Source  string `json:"source"`
+	// Owner is who sees it; empty is every caller of the model.
+	Owner string `json:"owner,omitempty"`
+	Note  string `json:"note,omitempty"`
+}
+
+// SetTrain enables the training endpoints.
+func (a *API) SetTrain(t *TrainAdmin) { a.train = t }
+
+// SetTrainMirror gives publishing the mirror, which is set up after the
+// training endpoints are.
+func (a *API) SetTrainMirror(m TrainMirror) {
+	if a.train != nil {
+		a.train.Mirror = m
+	}
+}
+
+// TrainRequest queues a job.
+type TrainRequest struct {
+	// Name is the adapter's: what requests will ask for it by.
+	Name string `json:"name"`
+	// Model is the catalog model the adapter is for.
+	Model string `json:"model"`
+	// BaseRepo and BaseRevision are the weights trained on: the model's own
+	// on its registry, in safetensors (Hugging Face), not a GGUF.
+	BaseRepo     string `json:"base_repo"`
+	BaseRevision string `json:"base_revision,omitempty"`
+	// Node trains it; GPUMemoryMiB is what it needs of a card, which can be
+	// what an engine there leaves free.
+	Node         string `json:"node"`
+	GPUMemoryMiB int    `json:"gpu_memory_mib"`
+	// Owner is who the adapter will belong to: org:, team:, project: (by
+	// path or ID), user: or key:. Only callers there see it. When the
+	// pairs come from one knowledge base, it defaults to that collection's
+	// owner; it must be able to read every collection they come from.
+	Owner string `json:"owner,omitempty"`
+	// Public makes the adapter every caller's its model allows. Refused for
+	// pairs from a knowledge base.
+	Public bool `json:"public,omitempty"`
+	// Data is the pairs, NDJSON of {"q", "a"} (llmgw ctl kb dataset).
+	Data string `json:"data"`
+	// Params tune the trainer: rank, alpha, dropout, epochs, lr, batch,
+	// max_len, seed.
+	Params map[string]any `json:"params,omitempty"`
+	// GGUF also makes the adapter in llama.cpp's format; when absent, it
+	// does when the model is served by llama.cpp.
+	GGUF *bool `json:"gguf,omitempty"`
+}
+
+// TrainJobView is a job as listed.
+type TrainJobView struct {
+	ID           string    `json:"id"`
+	Name         string    `json:"name"`
+	Model        string    `json:"model"`
+	BaseRepo     string    `json:"base_repo"`
+	BaseRevision string    `json:"base_revision,omitempty"`
+	Node         string    `json:"node"`
+	Owner        string    `json:"owner,omitempty"`
+	GPUMemoryMiB int       `json:"gpu_memory_mib"`
+	Params       string    `json:"params,omitempty"`
+	GGUF         bool      `json:"gguf"`
+	Pairs        int       `json:"pairs"`
+	State        string    `json:"state"`
+	Step         int       `json:"step"`
+	Steps        int       `json:"steps"`
+	Loss         float64   `json:"loss"`
+	Message      string    `json:"message,omitempty"`
+	PeakMiB      int       `json:"peak_mib"`
+	CreatedAt    time.Time `json:"created_at"`
+	StartedAt    time.Time `json:"started_at,omitzero"`
+	FinishedAt   time.Time `json:"finished_at,omitzero"`
+	CreatedBy    string    `json:"created_by,omitempty"`
+	// Artifacts are what it made: peft (a gzipped tar), gguf.
+	Artifacts []string `json:"artifacts,omitempty"`
+}
+
+func trainView(j store.TrainJob) TrainJobView {
+	return TrainJobView{ID: j.ID, Name: j.Name, Model: j.Model, BaseRepo: j.BaseRepo, BaseRevision: j.BaseRevision,
+		Node: j.Node, Owner: j.Owner, GPUMemoryMiB: j.GPUMemoryMiB, Params: j.Params, GGUF: j.GGUF, Pairs: j.Pairs, State: j.State,
+		Step: j.Step, Steps: j.Steps, Loss: j.Loss, Message: j.Message, PeakMiB: j.PeakMiB, CreatedAt: j.CreatedAt,
+		StartedAt: j.StartedAt, FinishedAt: j.FinishedAt, CreatedBy: j.CreatedBy}
+}
+
+func (a *API) registerTrain(mux Mux) {
+	mux.HandleFunc("POST /api/v1/train/jobs", a.trainCreate)
+	mux.HandleFunc("GET /api/v1/train/jobs", a.trainList)
+	mux.HandleFunc("GET /api/v1/train/jobs/{id}", a.trainGet)
+	mux.HandleFunc("POST /api/v1/train/jobs/{id}/cancel", a.trainCancel)
+	mux.HandleFunc("DELETE /api/v1/train/jobs/{id}", a.trainDelete)
+	mux.HandleFunc("GET /api/v1/train/jobs/{id}/artifacts/{kind}", a.trainArtifact)
+	mux.HandleFunc("POST /api/v1/train/jobs/{id}/publish", a.trainPublish)
+}
+
+func (a *API) trainReady(w http.ResponseWriter) bool {
+	if a.train == nil {
+		writeError(w, http.StatusNotImplemented, "training needs the store and the nodes' agents, which this control plane runs without")
+		return false
+	}
+	return true
+}
+
+// trainMaxData bounds a job's pairs.
+const trainMaxData = 64 << 20
+
+var adapterName = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._-]{0,99}$`)
+
+// trainParams are the trainer's settings, with what each may be.
+var trainParams = map[string][2]float64{
+	"rank": {1, 256}, "alpha": {1, 1024}, "dropout": {0, 0.9}, "epochs": {1, 100},
+	"lr": {1e-7, 1e-2}, "batch": {1, 128}, "max_len": {16, 32768}, "seed": {0, 1 << 31},
+}
+
+func (a *API) trainCreate(w http.ResponseWriter, r *http.Request) {
+	if !a.trainReady(w) {
+		return
+	}
+	var in TrainRequest
+	if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, trainMaxData+1<<20)).Decode(&in); err != nil {
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	}
+	bad := func(format string, args ...any) {
+		writeError(w, http.StatusBadRequest, fmt.Sprintf(format, args...))
+	}
+	if !adapterName.MatchString(in.Name) {
+		bad("an adapter's name is letters, digits, . _ and -, up to 100: %q is not", in.Name)
+		return
+	}
+	m, ok := a.ctl.Catalog().Resolve(in.Model)
+	if !ok {
+		bad("no model %s in the catalog", in.Model)
+		return
+	}
+	if _, _, _, taken := a.ctl.Catalog().ResolveAdapter(in.Name); taken || in.Name == m.Name {
+		bad("%s is already a model's or an adapter's name", in.Name)
+		return
+	}
+	if strings.Count(in.BaseRepo, "/") != 1 || strings.ContainsAny(in.BaseRepo, " :") {
+		bad("base_repo is the weights' registry repository, owner/name: %q is not", in.BaseRepo)
+		return
+	}
+	if _, ok := a.ctl.Inventory().Get(in.Node); !ok {
+		bad("no node %s", in.Node)
+		return
+	}
+	if in.GPUMemoryMiB <= 0 {
+		bad("gpu_memory_mib is what the job needs of a card, in MiB")
+		return
+	}
+	pairs, colls, err := countPairs(in.Data)
+	if err != nil {
+		bad("%v", err)
+		return
+	}
+	owner, err := a.trainOwner(r.Context(), in, colls)
+	if err != nil {
+		bad("%v", err)
+		return
+	}
+	for k, v := range in.Params {
+		lim, known := trainParams[k]
+		f, isNum := v.(float64)
+		if !known || !isNum || f < lim[0] || f > lim[1] {
+			bad("params.%s: the trainer takes rank, alpha, dropout, epochs, lr, batch, max_len and seed, each a number in its range", k)
+			return
+		}
+	}
+	params := ""
+	if len(in.Params) > 0 {
+		b, _ := json.Marshal(in.Params)
+		params = string(b)
+	}
+	gguf := false
+	for _, p := range m.Profiles {
+		if p.Engine == "llamacpp" {
+			gguf = true
+		}
+	}
+	if in.GGUF != nil {
+		gguf = *in.GGUF
+	}
+	j := store.TrainJob{ID: store.NewID("train-"), Name: in.Name, Model: m.Name, BaseRepo: in.BaseRepo,
+		BaseRevision: in.BaseRevision, Node: in.Node, Owner: owner, GPUMemoryMiB: in.GPUMemoryMiB, Params: params, GGUF: gguf,
+		Data: []byte(in.Data), Pairs: pairs, CreatedBy: actorOf(r)}
+	if err := a.train.Store.CreateTrainJob(r.Context(), j); err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	// At once rather than at the next look, when the node is here.
+	go a.train.Service.Dispatch(context.WithoutCancel(r.Context()))
+	j.State = store.TrainQueued
+	writeJSON(w, http.StatusCreated, trainView(j))
+}
+
+// trainOwner is who a job's adapter will belong to, as scope:ID: one that
+// may read every knowledge base the pairs come from. Pairs from a knowledge
+// base never make an adapter everyone's.
+func (a *API) trainOwner(ctx context.Context, in TrainRequest, colls []string) (string, error) {
+	if in.Public {
+		if len(colls) > 0 {
+			return "", errors.New("these pairs come from a knowledge base, and an adapter trained on them " +
+				"carries some of what it says: it belongs to a part of the tree that may read it, not to everyone")
+		}
+		if in.Owner != "" {
+			return "", errors.New("an adapter is public or has an owner, not both")
+		}
+		return "", nil
+	}
+	if len(colls) > 0 && a.kb == nil {
+		return "", errors.New("the pairs name knowledge bases, which this control plane cannot read")
+	}
+	var found []store.KBCollection
+	for _, id := range colls {
+		c, err := a.kb.Store.Collection(ctx, id)
+		if err != nil {
+			return "", fmt.Errorf("the pairs come from collection %s, which is not here: %w", id, err)
+		}
+		found = append(found, c)
+	}
+	if in.Owner == "" {
+		if len(found) == 1 {
+			return string(found[0].OwnerScope) + ":" + found[0].OwnerID, nil
+		}
+		return "", errors.New("name the adapter's owner (org:, team:, project:, user: or key:), or make it public " +
+			"when its pairs are not a knowledge base's")
+	}
+	scope, id, err := a.kbTarget(ctx, in.Owner)
+	if err != nil {
+		return "", err
+	}
+	for _, c := range found {
+		e, err := a.kb.Store.EffectiveFor(ctx, scope, id, c.OrgID)
+		if err != nil {
+			return "", fmt.Errorf("%s: %w", in.Owner, err)
+		}
+		readable, err := a.kb.Store.CollectionsFor(ctx, e, false)
+		if err != nil {
+			return "", err
+		}
+		if !slices.ContainsFunc(readable, func(x store.KBCollection) bool { return x.ID == c.ID }) {
+			return "", fmt.Errorf("%s may not read collection %s, which the pairs come from: "+
+				"an adapter trained on it would tell its callers what it says", in.Owner, c.Name)
+		}
+	}
+	return string(scope) + ":" + id, nil
+}
+
+// countPairs checks a job's data: NDJSON of {"q", "a"}, at least one, and
+// says which knowledge bases the pairs come from.
+func countPairs(data string) (int, []string, error) {
+	if len(data) > trainMaxData {
+		return 0, nil, fmt.Errorf("the data is over %d MiB", trainMaxData>>20)
+	}
+	seen := map[string]bool{}
+	var colls []string
+	n := 0
+	sc := bufio.NewScanner(strings.NewReader(data))
+	sc.Buffer(make([]byte, 64<<10), 1<<20)
+	line := 0
+	for sc.Scan() {
+		line++
+		b := bytes.TrimSpace(sc.Bytes())
+		if len(b) == 0 {
+			continue
+		}
+		var p struct{ Q, A, Collection string }
+		if err := json.Unmarshal(b, &p); err != nil {
+			return 0, nil, fmt.Errorf("data line %d is not JSON: %v", line, err)
+		}
+		if strings.TrimSpace(p.Q) == "" || strings.TrimSpace(p.A) == "" {
+			return 0, nil, fmt.Errorf("data line %d has no q or no a", line)
+		}
+		if p.Collection != "" && !seen[p.Collection] {
+			seen[p.Collection] = true
+			colls = append(colls, p.Collection)
+		}
+		n++
+	}
+	if err := sc.Err(); err != nil {
+		return 0, nil, err
+	}
+	if n == 0 {
+		return 0, nil, errors.New("the data holds no pairs: NDJSON of {\"q\", \"a\"}, as llmgw ctl kb dataset writes")
+	}
+	return n, colls, nil
+}
+
+func (a *API) trainList(w http.ResponseWriter, r *http.Request) {
+	if !a.trainReady(w) {
+		return
+	}
+	limit := 50
+	if n, err := strconv.Atoi(r.URL.Query().Get("limit")); err == nil && n > 0 && n <= 500 {
+		limit = n
+	}
+	jobs, err := a.train.Store.TrainJobs(r.Context(), limit)
+	if err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	out := []TrainJobView{}
+	for _, j := range jobs {
+		out = append(out, trainView(j))
+	}
+	writeJSON(w, http.StatusOK, out)
+}
+
+func (a *API) trainJob(w http.ResponseWriter, r *http.Request) (store.TrainJob, bool) {
+	if !a.trainReady(w) {
+		return store.TrainJob{}, false
+	}
+	j, err := a.train.Store.TrainJob(r.Context(), r.PathValue("id"))
+	if errors.Is(err, store.ErrNotFound) {
+		writeError(w, http.StatusNotFound, "no training job "+r.PathValue("id"))
+		return j, false
+	}
+	if err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return j, false
+	}
+	return j, true
+}
+
+func (a *API) trainGet(w http.ResponseWriter, r *http.Request) {
+	j, ok := a.trainJob(w, r)
+	if !ok {
+		return
+	}
+	v := trainView(j)
+	v.Artifacts, _ = a.train.Store.TrainArtifactKinds(r.Context(), j.ID)
+	writeJSON(w, http.StatusOK, v)
+}
+
+func (a *API) trainCancel(w http.ResponseWriter, r *http.Request) {
+	j, ok := a.trainJob(w, r)
+	if !ok {
+		return
+	}
+	if err := a.train.Service.Cancel(r.Context(), j.ID); err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	j, _ = a.train.Store.TrainJob(r.Context(), j.ID)
+	writeJSON(w, http.StatusOK, trainView(j))
+}
+
+func (a *API) trainDelete(w http.ResponseWriter, r *http.Request) {
+	j, ok := a.trainJob(w, r)
+	if !ok {
+		return
+	}
+	if !store.TrainOver(j.State) {
+		writeError(w, http.StatusConflict, "the job has not ended: cancel it first")
+		return
+	}
+	if err := a.train.Store.DeleteTrainJob(r.Context(), j.ID); err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	writeJSON(w, http.StatusOK, map[string]any{"id": j.ID, "deleted": true})
+}
+
+func (a *API) trainArtifact(w http.ResponseWriter, r *http.Request) {
+	j, ok := a.trainJob(w, r)
+	if !ok {
+		return
+	}
+	kind := r.PathValue("kind")
+	b, err := a.train.Store.TrainArtifact(r.Context(), j.ID, kind)
+	if errors.Is(err, store.ErrNotFound) {
+		writeError(w, http.StatusNotFound, "the job made no "+kind)
+		return
+	}
+	if err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	name := j.Name + ".tar.gz"
+	w.Header().Set("Content-Type", "application/gzip")
+	if kind == "gguf" {
+		name = j.Name + ".gguf"
+		w.Header().Set("Content-Type", "application/octet-stream")
+	}
+	w.Header().Set("Content-Disposition", `attachment; filename="`+name+`"`)
+	w.Header().Set("Content-Length", strconv.Itoa(len(b)))
+	_, _ = w.Write(b)
+}
+
+// trainPublish puts what a job made on the mirror and makes it the model's
+// adapter: the GGUF for a model llama.cpp serves, the PEFT directory for
+// one vLLM serves.
+func (a *API) trainPublish(w http.ResponseWriter, r *http.Request) {
+	j, ok := a.trainJob(w, r)
+	if !ok {
+		return
+	}
+	if a.train.Mirror == nil {
+		writeError(w, http.StatusNotImplemented, "publishing needs the model mirror (mirror.api in the settings); "+
+			"llmgw ctl train download, then llmgw mirror import there, does it by hand")
+		return
+	}
+	if a.models == nil {
+		writeError(w, http.StatusNotImplemented, "this control plane has no store for the catalog")
+		return
+	}
+	if j.State != store.TrainDone {
+		writeError(w, http.StatusConflict, "the job has not finished: it is "+j.State)
+		return
+	}
+	m, found := a.ctl.Catalog().Resolve(j.Model)
+	if !found {
+		writeError(w, http.StatusConflict, "the model "+j.Model+" is no longer in the catalog")
+		return
+	}
+	for _, ad := range m.Adapters {
+		if ad.Name == j.Name {
+			writeError(w, http.StatusConflict, j.Name+" is already "+m.Name+"'s adapter; train under another name")
+			return
+		}
+	}
+	llama := slices.ContainsFunc(m.Profiles, func(p catalog.Profile) bool { return p.Engine == "llamacpp" })
+	source := "trained by llmgw job " + j.ID + " on " + j.Node + " from " + j.BaseRepo
+	var url string
+	if llama {
+		b, err := a.train.Store.TrainArtifact(r.Context(), j.ID, "gguf")
+		if err != nil {
+			writeError(w, http.StatusConflict, "the job made no GGUF, which llama.cpp needs; train again without --no-gguf")
+			return
+		}
+		res, err := a.train.Mirror.Upload(r.Context(), j.Name, j.ID, j.Name+".gguf", b, source)
+		if err != nil {
+			writeError(w, http.StatusBadGateway, "keeping it on the mirror: "+err.Error())
+			return
+		}
+		url = res.URL
+	} else {
+		b, err := a.train.Store.TrainArtifact(r.Context(), j.ID, "peft")
+		if err != nil {
+			writeError(w, http.StatusConflict, "the job made no PEFT adapter")
+			return
+		}
+		files, err := untarAll(b)
+		if err != nil {
+			writeError(w, http.StatusInternalServerError, err.Error())
+			return
+		}
+		for name, data := range files {
+			res, err := a.train.Mirror.Upload(r.Context(), j.Name, j.ID, name, data, source)
+			if err != nil {
+				writeError(w, http.StatusBadGateway, "keeping "+name+" on the mirror: "+err.Error())
+				return
+			}
+			url = res.URL[:strings.LastIndex(res.URL, "/")+1]
+		}
+	}
+	ad := catalog.Adapter{Name: j.Name, Source: url, Owner: j.Owner}
+	if err := checkAdapters(r.Context(), m, []catalog.Adapter{ad}); err != nil {
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	}
+	if _, err := a.models.SetAdapters(r.Context(), m.Name, append(slices.Clone(m.Adapters), ad), actorOf(r)); err != nil {
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	}
+	v := TrainPublishView{Adapter: j.Name, Model: m.Name, Source: url, Owner: j.Owner}
+	if llama {
+		v.Note = "llama.cpp loads adapters when a replica starts: restart " + m.Name + "'s replicas (llmgw ctl restart " +
+			m.Name + ") and ask for model " + j.Name
+	} else {
+		v.Note = "ask for model " + j.Name + "; a replica loads it on the first request"
+	}
+	writeJSON(w, http.StatusOK, v)
+}
+
+// untarAll reads a gzipped tar of a PEFT directory's files.
+func untarAll(b []byte) (map[string][]byte, error) {
+	gz, err := gzip.NewReader(bytes.NewReader(b))
+	if err != nil {
+		return nil, err
+	}
+	tr := tar.NewReader(gz)
+	out := map[string][]byte{}
+	for {
+		h, err := tr.Next()
+		if errors.Is(err, io.EOF) {
+			return out, nil
+		}
+		if err != nil {
+			return nil, err
+		}
+		if h.Typeflag != tar.TypeReg || strings.ContainsAny(h.Name, "/\\") {
+			continue
+		}
+		data, err := io.ReadAll(io.LimitReader(tr, 4<<30))
+		if err != nil {
+			return nil, err
+		}
+		out[h.Name] = data
+	}
+}
@@ -0,0 +1,145 @@
+package adminapi
+
+import (
+	"context"
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/train"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+type noAgents struct{}
+
+func (noAgents) Send(string, *llmgwv1.ControlMessage) error { return nil }
+func (noAgents) Connected(string) bool                      { return false }
+
+func TestATrainingJobIsCheckedBeforeItIsQueued(t *testing.T) {
+	models := append(testModels(), catalog.Model{Name: "small", Source: "/s",
+		Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp"}}})
+	a, _ := setup(t, models, gpuNode("cuda01", 1))
+	s, err := store.OpenSQLite(context.Background(), filepath.Join(t.TempDir(), "s.db"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer s.Close()
+	a.SetTrain(&TrainAdmin{Store: s, Service: train.New(s, noAgents{}, nil)})
+	mux := http.NewServeMux()
+	a.Register(mux)
+	post := func(body string) (int, string) {
+		rec := httptest.NewRecorder()
+		mux.ServeHTTP(rec, httptest.NewRequest("POST", "/api/v1/train/jobs", strings.NewReader(body)))
+		return rec.Code, rec.Body.String()
+	}
+	good := map[string]any{"name": "tone", "model": "small", "base_repo": "Qwen/Qwen2.5-0.5B-Instruct", "node": "cuda01",
+		"gpu_memory_mib": 2300, "data": "{\"q\":\"a\",\"a\":\"b\"}\n{\"q\":\"c\",\"a\":\"d\"}\n", "params": map[string]any{"epochs": 3},
+		"public": true}
+	with := func(k string, v any) string {
+		m := map[string]any{}
+		for kk, vv := range good {
+			m[kk] = vv
+		}
+		m[k] = v
+		b, _ := json.Marshal(m)
+		return string(b)
+	}
+	for _, c := range []struct {
+		field string
+		value any
+		says  string
+	}{
+		{"name", "bad name", "letters, digits"},
+		{"model", "nope", "no model nope"},
+		{"name", "small", "already a model"},
+		{"base_repo", "Qwen", "owner/name"},
+		{"node", "gone", "no node gone"},
+		{"data", "{\"q\":\"a\"}\n", "no q or no a"},
+		{"data", "", "no pairs"},
+		{"params", map[string]any{"epochs": 0}, "params.epochs"},
+		{"params", map[string]any{"momentum": 0.9}, "params.momentum"},
+		{"public", false, "name the adapter's owner"},
+		{"data", "{\"q\":\"a\",\"a\":\"b\",\"collection\":\"kb_aaaaaaaaaaaaaaaa\"}\n", "from a knowledge base"},
+	} {
+		if code, body := post(with(c.field, c.value)); code != http.StatusBadRequest || !strings.Contains(body, c.says) {
+			t.Errorf("%s=%v: %d %s", c.field, c.value, code, body)
+		}
+	}
+	code, body := post(with("name", "tone"))
+	var v TrainJobView
+	_ = json.Unmarshal([]byte(body), &v)
+	if code != http.StatusCreated || v.Pairs != 2 || !v.GGUF || v.State != "queued" || v.Params != `{"epochs":3}` {
+		t.Fatalf("%d %s", code, body)
+	}
+	rec := httptest.NewRecorder()
+	mux.ServeHTTP(rec, httptest.NewRequest("DELETE", "/api/v1/train/jobs/"+v.ID, nil))
+	if rec.Code != http.StatusConflict {
+		t.Errorf("deleting a queued job: %d", rec.Code)
+	}
+	rec = httptest.NewRecorder()
+	mux.ServeHTTP(rec, httptest.NewRequest("GET", "/api/v1/train/jobs/"+v.ID+"/artifacts/peft", nil))
+	if rec.Code != http.StatusNotFound {
+		t.Errorf("an artifact not made: %d", rec.Code)
+	}
+}
+
+func TestAnAdapterTrainedOnAKnowledgeBaseBelongsToWhoMayReadIt(t *testing.T) {
+	dsn := os.Getenv("LLMGW_TEST_POSTGRES")
+	if dsn == "" {
+		t.Skip("set LLMGW_TEST_POSTGRES to a pgvector database to run this")
+	}
+	ctx := context.Background()
+	s, err := store.OpenPostgres(ctx, dsn)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer s.Close()
+	if s.KnowledgeAvailable() != nil {
+		t.Skip("no pgvector")
+	}
+	models := append(testModels(), catalog.Model{Name: "small", Source: "/s",
+		Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp"}}})
+	a, _ := setup(t, models, gpuNode("cuda01", 1))
+	a.SetQuotas(s)
+	a.SetKB(&KBAdmin{Store: s})
+	a.SetTrain(&TrainAdmin{Store: s, Service: train.New(s, noAgents{}, nil)})
+	sfx := strings.ToLower(store.NewID(""))
+	org, _ := s.CreateOrg(ctx, "Tr"+sfx)
+	hr, _ := s.CreateTeam(ctx, org, "hr")
+	_, _ = s.CreateProject(ctx, hr, "docs")
+	sales, _ := s.CreateTeam(ctx, org, "sales")
+	_, _ = s.CreateProject(ctx, sales, "crm")
+	c, err := s.CreateCollection(ctx, store.KBCollection{OwnerScope: store.ScopeTeam, OwnerID: hr, Name: "p", EmbedModel: "bge-m3", Dims: 3})
+	if err != nil {
+		t.Fatal(err)
+	}
+	mux := http.NewServeMux()
+	a.Register(mux)
+	post := func(extra string) (int, TrainJobView, string) {
+		body := `{"name":"tone-` + store.NewID("") + `","model":"small","base_repo":"Qwen/Q","node":"cuda01","gpu_memory_mib":2000,` +
+			`"data":"{\"q\":\"a\",\"a\":\"b\",\"collection\":\"` + c.ID + `\"}\n"` + extra + `}`
+		rec := httptest.NewRecorder()
+		mux.ServeHTTP(rec, httptest.NewRequest("POST", "/api/v1/train/jobs", strings.NewReader(body)))
+		var v TrainJobView
+		_ = json.Unmarshal(rec.Body.Bytes(), &v)
+		return rec.Code, v, rec.Body.String()
+	}
+	if code, v, body := post(""); code != http.StatusCreated || v.Owner != "team:"+hr {
+		t.Errorf("by default: %d %s", code, body)
+	}
+	if code, v, body := post(`,"owner":"project:Tr` + sfx + `/hr/docs"`); code != http.StatusCreated || !strings.HasPrefix(v.Owner, "project:") {
+		t.Errorf("a project of the team: %d %s", code, body)
+	}
+	if code, _, body := post(`,"owner":"project:Tr` + sfx + `/sales/crm"`); code != http.StatusBadRequest || !strings.Contains(body, "may not read") {
+		t.Errorf("a project that cannot read it: %d %s", code, body)
+	}
+	if code, _, body := post(`,"public":true`); code != http.StatusBadRequest || !strings.Contains(body, "not to everyone") {
+		t.Errorf("public: %d %s", code, body)
+	}
+}
@@ -0,0 +1,50 @@
+package agentsvc
+
+import (
+	"time"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+)
+
+// SetAgentSettings gives the service each node's agent settings from the
+// cluster's (BACKLOG E3), as YAML, read at each use so a change is what is
+// sent next.
+func (s *Service) SetAgentSettings(f func(node string) string) {
+	s.mu.Lock()
+	s.agentSettings = f
+	s.mu.Unlock()
+}
+
+func (s *Service) agentSettingsMsg(node string) *llmgwv1.ControlMessage {
+	s.mu.RLock()
+	f := s.agentSettings
+	s.mu.RUnlock()
+	if f == nil {
+		return nil
+	}
+	return &llmgwv1.ControlMessage{Msg: &llmgwv1.ControlMessage_AgentSettings{
+		AgentSettings: &llmgwv1.AgentSettings{Yaml: f(node)}}}
+}
+
+// agentSettingsGap is the pause between telling one node and the next: an
+// agent whose settings changed starts again, and a whole cluster's agents
+// doing so in the same second would all re-register at once.
+var agentSettingsGap = time.Second
+
+// BroadcastAgentSettings tells every connected agent what the cluster's
+// settings now say it runs on, a moment apart. A node not connected is told
+// when it registers.
+func (s *Service) BroadcastAgentSettings() {
+	for _, node := range s.ConnectedNodes() {
+		msg := s.agentSettingsMsg(node)
+		if msg == nil {
+			return
+		}
+		if err := s.send(node, msg); err != nil {
+			s.log.Warn("could not tell a node its agent settings; it is told when it next registers",
+				"node", node, "err", err)
+			continue
+		}
+		time.Sleep(agentSettingsGap)
+	}
+}
@@ -0,0 +1,28 @@
+package agentsvc
+
+import (
+	"log/slog"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+)
+
+func TestEachNodeIsToldItsOwnAgentSettings(t *testing.T) {
+	s := New(inventory.NewStore(), nil, false, slog.New(slog.DiscardHandler))
+	if s.agentSettingsMsg("n1") != nil {
+		t.Error("told settings with nothing to read them from")
+	}
+	s.SetAgentSettings(func(node string) string {
+		if node == "cuda01" {
+			return "labels:\n  gpu: rtx6000\n"
+		}
+		return ""
+	})
+	if got := s.agentSettingsMsg("cuda01").GetAgentSettings().GetYaml(); got != "labels:\n  gpu: rtx6000\n" {
+		t.Errorf("cuda01 told %q", got)
+	}
+	// None is said too, so an agent whose settings were removed drops them.
+	if m := s.agentSettingsMsg("node01"); m == nil || m.GetAgentSettings().GetYaml() != "" {
+		t.Errorf("node01 told %v", m)
+	}
+}
@@ -79,6 +79,11 @@ type Service struct {
 	controlPlanes func(context.Context) ([]string, error)
 	// firewall is what nodes are told of llmgw's firewall; see SetFirewall.
 	firewall func() *llmgwv1.Firewall
+	// agentSettings is what each node's agent is told to run on; see
+	// SetAgentSettings.
+	agentSettings func(node string) string
+	// train is told how training jobs go; see SetTrain.
+	train TrainHandler

 	// logWaits are the log requests awaiting a reply, by request id.
 	logMu    sync.Mutex
@@ -296,6 +301,13 @@ func (s *Service) Connect(
 			return err
 		}
 	}
+	// What the cluster's settings say this agent runs on. An agent already
+	// on them ignores it; one that is not starts again on them.
+	if msg := s.agentSettingsMsg(nodeID); msg != nil {
+		if err := stream.Send(msg); err != nil {
+			return err
+		}
+	}

 	// Recover replica state from what this node is already running, before any
 	// reconciliation pass can conclude the cluster is empty and place
@@ -358,6 +370,9 @@ func (s *Service) readLoop(ctx context.Context,
 		msg, err := stream.Receive()
 		if err != nil {
 			c.close()
+			if h := s.trainHandler(); h != nil {
+				h.OnNodeGone(nodeID)
+			}
 			if ctx.Err() != nil || errors.Is(err, io.EOF) {
 				s.log.Info("agent disconnected", "node", nodeID)
 				return
@@ -495,6 +510,16 @@ func (s *Service) handle(nodeID string, msg *llmgwv1.AgentMessage) {
 			ch <- m.Upgrade
 		}

+	case *llmgwv1.AgentMessage_TrainStatus:
+		if h := s.trainHandler(); h != nil {
+			h.OnTrainStatus(nodeID, m.TrainStatus)
+		}
+
+	case *llmgwv1.AgentMessage_TrainArtifact:
+		if h := s.trainHandler(); h != nil {
+			h.OnTrainArtifact(nodeID, m.TrainArtifact)
+		}
+
 	case *llmgwv1.AgentMessage_Logs:
 		s.logMu.Lock()
 		ch, ok := s.logWaits[m.Logs.GetRequestId()]
@@ -0,0 +1,35 @@
+package agentsvc
+
+import (
+	"slices"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+)
+
+// TrainHandler is told how training jobs on the nodes go (BACKLOG F4).
+type TrainHandler interface {
+	OnTrainStatus(node string, st *llmgwv1.TrainStatus)
+	OnTrainArtifact(node string, a *llmgwv1.TrainArtifact)
+	// OnNodeGone is a node's stream ending: a job running there ended with
+	// it.
+	OnNodeGone(node string)
+}
+
+// SetTrain gives the service what training jobs' messages go to.
+func (s *Service) SetTrain(h TrainHandler) {
+	s.mu.Lock()
+	s.train = h
+	s.mu.Unlock()
+}
+
+func (s *Service) trainHandler() TrainHandler {
+	s.mu.RLock()
+	defer s.mu.RUnlock()
+	return s.train
+}
+
+// Send gives a connected node's agent a command.
+func (s *Service) Send(node string, msg *llmgwv1.ControlMessage) error { return s.send(node, msg) }
+
+// Connected is whether a node's agent holds a stream to this control plane.
+func (s *Service) Connected(node string) bool { return slices.Contains(s.ConnectedNodes(), node) }
@@ -14,6 +14,7 @@ import (

 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/comfy"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/facility"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
@@ -363,6 +364,13 @@ type Model struct {
 	// Capabilities is what this model can do beyond plain completion.
 	Capabilities Capabilities `json:"capabilities,omitempty" koanf:"capabilities"`

+	// QueryPrefix is put before a question an embedding model embeds for a
+	// knowledge-base search; documents are embedded without it. Models
+	// trained with an instruction on the query side want it, Qwen3-Embedding
+	// "Instruct: Given a web search query, retrieve relevant passages that
+	// answer the query\nQuery: " for one.
+	QueryPrefix string `json:"query_prefix,omitempty" koanf:"query_prefix"`
+
 	// Role "guard" is a model that judges the gateway's traffic for the
 	// guardrails rather than serving clients: hidden from /v1/models and the
 	// chat page, refused to any key that names it, critical by default, and
@@ -450,6 +458,15 @@ type Capabilities struct {
 	Tools      bool `json:"tools,omitempty" koanf:"tools"`
 	Vision     bool `json:"vision,omitempty" koanf:"vision"`
 	Embeddings bool `json:"embeddings,omitempty" koanf:"embeddings"`
+	// Images is a model that makes pictures, asked for at
+	// /v1/images/generations. Implied by an external model whose API is
+	// ComfyUI's.
+	Images bool `json:"images,omitempty" koanf:"images"`
+}
+
+// MakesImages reports whether a model answers /v1/images/generations.
+func (m Model) MakesImages() bool {
+	return m.Capabilities.Images || m.External != nil && m.External.API == APIComfyUI
 }

 // Catalog is the set of models the cluster should serve.
@@ -1094,8 +1111,20 @@ type External struct {
 	// than one written by hand. Such a model is named PEER/MODEL, comes and
 	// goes with the peer's own list, and is sent only while the peer is up.
 	Peer string `json:"peer,omitempty" koanf:"peer"`
+	// API is what the provider speaks: OpenAI's, unless this says comfyui —
+	// a ComfyUI server, asked for pictures at /v1/images/generations and
+	// sent the workflow below with the request's prompt in it. Model is then
+	// the checkpoint file the workflow loads, as ComfyUI lists it.
+	API string `json:"api,omitempty" koanf:"api"`
+	// Workflow is the ComfyUI workflow, in its API format: a built-in's name
+	// (flux-schnell, the default, or flux-dev) or a JSON file on the control
+	// plane.
+	Workflow string `json:"workflow,omitempty" koanf:"workflow"`
 }

+// APIComfyUI is External.API for a ComfyUI server.
+const APIComfyUI = "comfyui"
+
 // IsPeer reports whether a model is offered by a federated cluster.
 func (m Model) IsPeer() bool { return m.External != nil && m.External.Peer != "" }

@@ -1110,6 +1139,23 @@ func (e *External) validate(model string, profiles int) error {
 	if strings.TrimSpace(e.Model) == "" {
 		return fmt.Errorf("model %q: external model needs the name the provider knows it by", model)
 	}
+	switch e.API {
+	case "", "openai":
+		if e.Workflow != "" {
+			return fmt.Errorf("model %q: a workflow is for api: comfyui", model)
+		}
+	case APIComfyUI:
+		if e.Peer != "" {
+			return fmt.Errorf("model %q: a federated model speaks OpenAI's API, not ComfyUI's", model)
+		}
+		// A file is read where the router runs; a name is checked here.
+		if w := e.Workflow; w != "" && !strings.ContainsAny(w, "/.") && !slices.Contains(comfy.Builtins(), w) {
+			return fmt.Errorf("model %q: no built-in workflow %q (there are %s; or name a JSON file)",
+				model, w, strings.Join(comfy.Builtins(), ", "))
+		}
+	default:
+		return fmt.Errorf("model %q: external api %q is not openai or comfyui", model, e.API)
+	}
 	return nil
 }

@@ -1127,6 +1173,27 @@ type Adapter struct {
 	Source string `json:"source" koanf:"source"`
 	// Scale is how strongly it is applied; zero takes 1.
 	Scale float64 `json:"scale,omitempty" koanf:"scale"`
+	// Owner is the part of the tenancy tree the adapter belongs to —
+	// org:ID, team:ID, project:ID, user:ID or key:ID — and only callers in
+	// it see it: to anyone else it is a model that does not exist. An
+	// adapter trained on an organisation's documents carries some of what
+	// they say in its weights. Empty is every caller the model allows.
+	Owner string `json:"owner,omitempty" koanf:"owner"`
+}
+
+// ValidAdapterOwner checks an adapter's owner is scope:ID.
+func ValidAdapterOwner(owner string) error {
+	if owner == "" {
+		return nil
+	}
+	scope, id, ok := strings.Cut(owner, ":")
+	switch {
+	case !ok || id == "":
+		return fmt.Errorf("an adapter's owner is scope:ID, %q is not", owner)
+	case scope != "org" && scope != "team" && scope != "project" && scope != "user" && scope != "key":
+		return fmt.Errorf("an adapter's owner is an org, team, project, user or key, not %q", scope)
+	}
+	return nil
 }

 // EffectiveScale is the scale an adapter is applied at.
@@ -1175,6 +1242,9 @@ func validateAdapters(m Model) error {
 	}
 	seen := map[string]bool{}
 	for _, a := range m.Adapters {
+		if err := ValidAdapterOwner(a.Owner); err != nil {
+			return fmt.Errorf("model %q adapter %q: %w", m.Name, a.Name, err)
+		}
 		switch {
 		case a.Name == "" || a.Source == "":
 			return fmt.Errorf("model %q: an adapter needs a name and a source", m.Name)
@@ -9,6 +9,7 @@ import (
 	"encoding/hex"
 	"fmt"
 	"log/slog"
+	"os"
 	"strings"
 	"sync"
 	"time"
@@ -412,12 +413,27 @@ func (c *Controller) externalEndpoints() []endpoint.Info {
 		if m.IsPeer() {
 			id = "peer:" + m.Name
 		}
+		engine, workflow := endpoint.EngineExternal, ""
+		if m.External.API == catalog.APIComfyUI {
+			engine, workflow = endpoint.EngineComfyUI, m.External.Workflow
+			// A file is read here, on the control plane the catalog says it
+			// is on, and sent as the workflow itself.
+			if strings.ContainsAny(workflow, "/.") {
+				b, err := os.ReadFile(workflow)
+				if err != nil {
+					c.log.Warn("a ComfyUI model's workflow could not be read; not offered", "model", m.Name, "err", err)
+					continue
+				}
+				workflow = string(b)
+			}
+		}
 		out = append(out, endpoint.Info{
 			ID: id, URL: strings.TrimRight(m.External.URL, "/"),
 			Models: append([]string{m.Name}, m.Aliases...), APIKey: key,
-			Engine: endpoint.EngineExternal, Pool: "external", Weight: 1,
+			Engine: engine, Pool: "external", Weight: 1,
 			UpstreamModel: m.External.Model,
 			Peer:          m.External.Peer, PeerCluster: cluster, PeerTags: tags,
+			Workflow: workflow,
 		})
 	}
 	return out
@@ -427,8 +443,9 @@ func (c *Controller) externalEndpoints() []endpoint.Info {
 func fingerprint(infos []endpoint.Info) string {
 	var b strings.Builder
 	for _, i := range infos {
-		fmt.Fprintf(&b, "%s|%s|%s|%s|%v|%g|%d|%s|%s|%s|%t|%x;", i.ID, i.URL, i.Engine, i.Pool, i.Models, i.Weight,
-			i.Priority, i.UpstreamModel, i.Peer, i.PeerCluster, i.PeerTags, sha256.Sum256([]byte(i.APIKey)))
+		fmt.Fprintf(&b, "%s|%s|%s|%s|%v|%g|%d|%s|%s|%s|%t|%x|%x;", i.ID, i.URL, i.Engine, i.Pool, i.Models, i.Weight,
+			i.Priority, i.UpstreamModel, i.Peer, i.PeerCluster, i.PeerTags, sha256.Sum256([]byte(i.APIKey)),
+			sha256.Sum256([]byte(i.Workflow)))
 	}
 	return b.String()
 }
@@ -0,0 +1,196 @@
+// Package train runs LoRA training jobs from the control plane (BACKLOG F4):
+// queued in the store, sent to the node each names when its agent is there
+// and has nothing else training, followed as the agent reports, and what it
+// makes kept in the store.
+package train
+
+import (
+	"context"
+	"log/slog"
+	"sync"
+	"time"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// Store is where jobs are kept.
+type Store interface {
+	QueuedTrainJobs(ctx context.Context) ([]store.TrainJob, error)
+	ActiveTrainJobs(ctx context.Context) ([]store.TrainJob, error)
+	TrainJob(ctx context.Context, id string) (store.TrainJob, error)
+	UpdateTrainJob(ctx context.Context, j store.TrainJob) error
+	PutTrainArtifact(ctx context.Context, job, kind string, data []byte) error
+}
+
+// Agents is how jobs reach the nodes.
+type Agents interface {
+	Send(node string, msg *llmgwv1.ControlMessage) error
+	Connected(node string) bool
+}
+
+// Service sends queued jobs and records how they go.
+type Service struct {
+	store  Store
+	agents Agents
+	log    *slog.Logger
+
+	mu sync.Mutex
+	// pieces are artifacts arriving, by job and kind.
+	pieces map[[2]string][]byte
+}
+
+// New is a service.
+func New(s Store, a Agents, log *slog.Logger) *Service {
+	if log == nil {
+		log = slog.New(slog.DiscardHandler)
+	}
+	return &Service{store: s, agents: a, log: log, pieces: map[[2]string][]byte{}}
+}
+
+// Every is how often queued jobs are looked at.
+var Every = 5 * time.Second
+
+// Run sends queued jobs until ctx ends.
+func (s *Service) Run(ctx context.Context) {
+	t := time.NewTicker(Every)
+	defer t.Stop()
+	for {
+		s.Dispatch(ctx)
+		select {
+		case <-ctx.Done():
+			return
+		case <-t.C:
+		}
+	}
+}
+
+// Dispatch sends each queued job whose node is here and not already
+// training, one per node.
+func (s *Service) Dispatch(ctx context.Context) {
+	queued, err := s.store.QueuedTrainJobs(ctx)
+	if err != nil || len(queued) == 0 {
+		return
+	}
+	active, err := s.store.ActiveTrainJobs(ctx)
+	if err != nil {
+		return
+	}
+	busy := map[string]bool{}
+	for _, j := range active {
+		busy[j.Node] = true
+	}
+	for _, q := range queued {
+		if busy[q.Node] || !s.agents.Connected(q.Node) {
+			continue
+		}
+		j, err := s.store.TrainJob(ctx, q.ID)
+		if err != nil {
+			continue
+		}
+		err = s.agents.Send(j.Node, &llmgwv1.ControlMessage{Msg: &llmgwv1.ControlMessage_TrainStart{TrainStart: &llmgwv1.TrainStart{
+			JobId: j.ID, BaseRepo: j.BaseRepo, BaseRevision: j.BaseRevision, Data: j.Data, Params: j.Params,
+			GpuMemoryMib: int32(j.GPUMemoryMiB), Gguf: j.GGUF}}}) //nolint:gosec // MiB
+		if err != nil {
+			s.log.Warn("could not send a training job to its node", "job", j.ID, "node", j.Node, "err", err)
+			continue
+		}
+		j.State, j.StartedAt, j.Message = store.TrainStarting, time.Now(), "sent to "+j.Node
+		_ = s.store.UpdateTrainJob(ctx, j)
+		busy[j.Node] = true
+		s.log.Info("training job sent", "job", j.ID, "node", j.Node, "adapter", j.Name)
+	}
+}
+
+// OnTrainStatus records how a job goes, as its node says.
+func (s *Service) OnTrainStatus(node string, st *llmgwv1.TrainStatus) {
+	ctx := context.Background()
+	j, err := s.store.TrainJob(ctx, st.GetJobId())
+	if err != nil || j.Node != node || store.TrainOver(j.State) {
+		return
+	}
+	j.State = st.GetState()
+	if m := st.GetMessage(); m != "" || j.State == store.TrainDone {
+		j.Message = m
+	}
+	if st.GetSteps() > 0 {
+		j.Step, j.Steps = int(st.GetStep()), int(st.GetSteps())
+	}
+	if st.GetLoss() > 0 {
+		j.Loss = st.GetLoss()
+	}
+	if p := int(st.GetPeakMib()); p > j.PeakMiB {
+		j.PeakMiB = p
+	}
+	if store.TrainOver(j.State) {
+		j.FinishedAt = time.Now()
+		s.mu.Lock()
+		for k := range s.pieces {
+			if k[0] == j.ID {
+				delete(s.pieces, k)
+			}
+		}
+		s.mu.Unlock()
+		s.log.Info("training job ended", "job", j.ID, "state", j.State, "message", j.Message)
+	}
+	if err := s.store.UpdateTrainJob(ctx, j); err != nil {
+		s.log.Warn("could not record a training job's progress", "job", j.ID, "err", err)
+	}
+}
+
+// OnTrainArtifact gathers what a job made, and keeps it when it is whole.
+func (s *Service) OnTrainArtifact(node string, a *llmgwv1.TrainArtifact) {
+	key := [2]string{a.GetJobId(), a.GetKind()}
+	s.mu.Lock()
+	if a.GetSeq() == 0 {
+		s.pieces[key] = nil
+	}
+	s.pieces[key] = append(s.pieces[key], a.GetData()...)
+	whole := s.pieces[key]
+	if a.GetLast() {
+		delete(s.pieces, key)
+	}
+	s.mu.Unlock()
+	if !a.GetLast() {
+		return
+	}
+	if err := s.store.PutTrainArtifact(context.Background(), key[0], key[1], whole); err != nil {
+		s.log.Warn("could not keep what a training job made", "job", key[0], "kind", key[1], "err", err)
+	}
+}
+
+// OnNodeGone fails the jobs a node was running: its agent's stream ended,
+// and the job with it.
+func (s *Service) OnNodeGone(node string) {
+	ctx := context.Background()
+	active, err := s.store.ActiveTrainJobs(ctx)
+	if err != nil {
+		return
+	}
+	for _, j := range active {
+		if j.Node != node {
+			continue
+		}
+		j.State, j.FinishedAt = store.TrainFailed, time.Now()
+		j.Message = "the node's agent went away while it trained; start the job again"
+		_ = s.store.UpdateTrainJob(ctx, j)
+	}
+}
+
+// Cancel stops a job: one still queued is not sent, one running is stopped
+// on its node.
+func (s *Service) Cancel(ctx context.Context, id string) error {
+	j, err := s.store.TrainJob(ctx, id)
+	if err != nil {
+		return err
+	}
+	if store.TrainOver(j.State) {
+		return nil
+	}
+	if j.State != store.TrainQueued && s.agents.Connected(j.Node) {
+		_ = s.agents.Send(j.Node, &llmgwv1.ControlMessage{Msg: &llmgwv1.ControlMessage_TrainCancel{
+			TrainCancel: &llmgwv1.TrainCancel{JobId: id}}})
+	}
+	j.State, j.FinishedAt, j.Message = store.TrainCancelled, time.Now(), "cancelled"
+	return s.store.UpdateTrainJob(ctx, j)
+}
@@ -0,0 +1,124 @@
+package train
+
+import (
+	"context"
+	"path/filepath"
+	"sync"
+	"testing"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+type fakeAgents struct {
+	mu        sync.Mutex
+	connected map[string]bool
+	sent      []*llmgwv1.ControlMessage
+}
+
+func (f *fakeAgents) Send(_ string, m *llmgwv1.ControlMessage) error {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	f.sent = append(f.sent, m)
+	return nil
+}
+func (f *fakeAgents) Connected(n string) bool {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	return f.connected[n]
+}
+
+func setup(t *testing.T) (*Service, *store.DB, *fakeAgents) {
+	t.Helper()
+	s, err := store.OpenSQLite(context.Background(), filepath.Join(t.TempDir(), "s.db"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	t.Cleanup(func() { _ = s.Close() })
+	a := &fakeAgents{connected: map[string]bool{}}
+	return New(s, a, nil), s, a
+}
+
+func job(t *testing.T, s *store.DB, id, node string) {
+	t.Helper()
+	if err := s.CreateTrainJob(context.Background(), store.TrainJob{ID: id, Name: "a-" + id, Model: "m", BaseRepo: "o/n",
+		Node: node, GPUMemoryMiB: 2000, Data: []byte(`{"q":"x","a":"y"}`), Pairs: 1, GGUF: true}); err != nil {
+		t.Fatal(err)
+	}
+}
+
+func TestAJobGoesToItsNodeWhenItIsHereAndFree(t *testing.T) {
+	ctx := context.Background()
+	svc, s, a := setup(t)
+	job(t, s, "j1", "cuda01")
+	job(t, s, "j2", "cuda01")
+	svc.Dispatch(ctx)
+	if len(a.sent) != 0 {
+		t.Fatal("sent to a node that is not here")
+	}
+	a.connected["cuda01"] = true
+	svc.Dispatch(ctx)
+	svc.Dispatch(ctx)
+	if len(a.sent) != 1 {
+		t.Fatalf("sent %d jobs to one node", len(a.sent))
+	}
+	start := a.sent[0].GetTrainStart()
+	if start.GetJobId() != "j1" || start.GetGpuMemoryMib() != 2000 || !start.GetGguf() || string(start.GetData()) == "" {
+		t.Errorf("sent %+v", start)
+	}
+	if j, _ := s.TrainJob(ctx, "j1"); j.State != store.TrainStarting || j.StartedAt.IsZero() {
+		t.Errorf("j1 %+v", j)
+	}
+
+	// Progress, then what it made in pieces, then done: the next is sent.
+	svc.OnTrainStatus("cuda01", &llmgwv1.TrainStatus{JobId: "j1", State: "running", Step: 5, Steps: 10, Loss: 0.8, PeakMib: 1900})
+	svc.OnTrainArtifact("cuda01", &llmgwv1.TrainArtifact{JobId: "j1", Kind: "gguf", Seq: 0, Data: []byte("GG")})
+	svc.OnTrainArtifact("cuda01", &llmgwv1.TrainArtifact{JobId: "j1", Kind: "gguf", Seq: 1, Data: []byte("UF"), Last: true})
+	svc.OnTrainStatus("cuda01", &llmgwv1.TrainStatus{JobId: "j1", State: "done"})
+	j, _ := s.TrainJob(ctx, "j1")
+	if j.State != store.TrainDone || j.Step != 5 || j.PeakMiB != 1900 || j.Loss != 0.8 || j.FinishedAt.IsZero() {
+		t.Errorf("j1 after %+v", j)
+	}
+	if b, _ := s.TrainArtifact(ctx, "j1", "gguf"); string(b) != "GGUF" {
+		t.Errorf("artifact %q", b)
+	}
+	// A late message does not bring an ended job back.
+	svc.OnTrainStatus("cuda01", &llmgwv1.TrainStatus{JobId: "j1", State: "running"})
+	if j, _ := s.TrainJob(ctx, "j1"); j.State != store.TrainDone {
+		t.Errorf("revived: %s", j.State)
+	}
+	// Nor does another node speak for it.
+	svc.Dispatch(ctx)
+	if len(a.sent) != 2 || a.sent[1].GetTrainStart().GetJobId() != "j2" {
+		t.Fatalf("j2 not sent: %d", len(a.sent))
+	}
+	svc.OnTrainStatus("node01", &llmgwv1.TrainStatus{JobId: "j2", State: "done"})
+	if j, _ := s.TrainJob(ctx, "j2"); j.State == store.TrainDone {
+		t.Error("another node ended the job")
+	}
+	// The node goes: its job fails, saying why.
+	svc.OnNodeGone("cuda01")
+	if j, _ := s.TrainJob(ctx, "j2"); j.State != store.TrainFailed || j.Message == "" {
+		t.Errorf("after the node went: %+v", j)
+	}
+}
+
+func TestCancellingAJob(t *testing.T) {
+	ctx := context.Background()
+	svc, s, a := setup(t)
+	job(t, s, "j1", "cuda01")
+	if err := svc.Cancel(ctx, "j1"); err != nil {
+		t.Fatal(err)
+	}
+	a.connected["cuda01"] = true
+	svc.Dispatch(ctx)
+	if len(a.sent) != 0 {
+		t.Error("a cancelled job was sent")
+	}
+	job(t, s, "j2", "cuda01")
+	svc.Dispatch(ctx)
+	_ = svc.Cancel(ctx, "j2")
+	if len(a.sent) != 2 || a.sent[1].GetTrainCancel().GetJobId() != "j2" {
+		t.Errorf("the node was not told: %+v", a.sent)
+	}
+}
@@ -8,6 +8,7 @@ var Reference = []Block{
 		{Key: "name", Type: "string", Doc: "Name is what clients ask for, like a model name.", Nested: ""},
 		{Key: "source", Type: "string", Doc: "Source is the adapter: a GGUF file for llama.cpp; for vLLM a PEFT\nadapter directory (a mirror version ending in \"/\", a path on the node,\nor a Hugging Face repository id). Fetched by each node as the model's\nweights are.", Nested: ""},
 		{Key: "scale", Type: "float64", Doc: "Scale is how strongly it is applied; zero takes 1.", Nested: ""},
+		{Key: "owner", Type: "string", Doc: "Owner is the part of the tenancy tree the adapter belongs to —\norg:ID, team:ID, project:ID, user:ID or key:ID — and only callers in\nit see it: to anyone else it is a model that does not exist. An\nadapter trained on an organisation's documents carries some of what\nthey say in its weights. Empty is every caller the model allows.", Nested: ""},
 	}},
 	{Package: "config", Name: "Admission", Title: "Admission", Doc: "", Fields: []Field{
 		{Key: "enabled", Type: "bool", Doc: "Enabled turns the layer on. Off, every request goes straight through,\nwhich is the v0.2.0 behaviour.", Nested: ""},
@@ -19,6 +20,10 @@ var Reference = []Block{
 		{Key: "queue_capacity_total", Type: "int", Doc: "QueueCapacityTotal bounds everything queued across bands. Reached, a\nrequest is not refused while a lower band has any queued: the newest of\nthe lowest band is turned away instead, with a 503 and Retry-After.\nZero takes half of all the bands' capacities together.", Nested: ""},
 		{Key: "interval", Type: "time.Duration", Doc: "", Nested: ""},
 	}},
+	{Package: "config", Name: "AgentSettings", Title: "AgentSettings", Doc: "", Fields: []Field{
+		{Key: "default", Type: "map[string]any", Doc: "Default is agent.yaml's keys for every node.", Nested: ""},
+		{Key: "nodes", Type: "map[string]map[string]any", Doc: "Nodes are agent.yaml's keys for one node each, by node id, over\nDefault.", Nested: ""},
+	}},
 	{Package: "config", Name: "Auth", Title: "Auth", Doc: "", Fields: []Field{
 		{Key: "enabled", Type: "bool", Doc: "", Nested: ""},
 		{Key: "store", Type: "string", Doc: "Store is the SQLite file holding keys, tenancy and usage.", Nested: ""},
@@ -59,6 +64,7 @@ var Reference = []Block{
 		{Key: "tools", Type: "bool", Doc: "", Nested: ""},
 		{Key: "vision", Type: "bool", Doc: "", Nested: ""},
 		{Key: "embeddings", Type: "bool", Doc: "", Nested: ""},
+		{Key: "images", Type: "bool", Doc: "Images is a model that makes pictures, asked for at\n/v1/images/generations. Implied by an external model whose API is\nComfyUI's.", Nested: ""},
 	}},
 	{Package: "config", Name: "Control", Title: "Control", Doc: "", Fields: []Field{
 		{Key: "interval", Type: "time.Duration", Doc: "Interval is how often desired and actual state are compared.", Nested: ""},
@@ -79,6 +85,7 @@ var Reference = []Block{
 		{Key: "api_key", Type: "string", Doc: "APIKey is sent to the engine as a bearer token, for an engine started\nwith one.", Nested: ""},
 		{Key: "models", Type: "[]string", Doc: "", Nested: ""},
 		{Key: "engine", Type: "string", Doc: "Engine names the inference engine, which decides what its metrics are\ncalled. Empty means vLLM's names.", Nested: ""},
+		{Key: "workflow", Type: "string", Doc: "Workflow is the ComfyUI workflow for engine: comfyui, a built-in's\nname or its JSON; flux-schnell unless set.", Nested: ""},
 		{Key: "pool", Type: "string", Doc: "", Nested: ""},
 		{Key: "weight", Type: "float64", Doc: "", Nested: ""},
 		{Key: "priority", Type: "int", Doc: "", Nested: ""},
@@ -99,6 +106,8 @@ var Reference = []Block{
 		{Key: "model", Type: "string", Doc: "Model is what the provider calls it; requests are rewritten to it.", Nested: ""},
 		{Key: "api_key_file", Type: "string", Doc: "APIKeyFile holds the provider's key, on the control plane. A file\nrather than the key itself, so the key is in neither the catalog, the\nstore nor the audit log; empty for a provider that asks for none.", Nested: ""},
 		{Key: "peer", Type: "string", Doc: "Peer names the federated llmgw cluster this model is offered by, for\na model that federation added (`llmgw ctl federation add`) rather\nthan one written by hand. Such a model is named PEER/MODEL, comes and\ngoes with the peer's own list, and is sent only while the peer is up.", Nested: ""},
+		{Key: "api", Type: "string", Doc: "API is what the provider speaks: OpenAI's, unless this says comfyui —\na ComfyUI server, asked for pictures at /v1/images/generations and\nsent the workflow below with the request's prompt in it. Model is then\nthe checkpoint file the workflow loads, as ComfyUI lists it.", Nested: ""},
+		{Key: "workflow", Type: "string", Doc: "Workflow is the ComfyUI workflow, in its API format: a built-in's name\n(flux-schnell, the default, or flux-dev) or a JSON file on the control\nplane.", Nested: ""},
 	}},
 	{Package: "config", Name: "Facility", Title: "Facility", Doc: "", Fields: []Field{
 		{Key: "budgets", Type: "[]FacilityBudget", Doc: "Budgets are the domains with a power cap.", Nested: ""},
@@ -176,6 +185,7 @@ var Reference = []Block{
 		{Key: "context_length", Type: "int", Doc: "ContextLength is how many tokens one request may use, prompt and\ncompletion together, and MaxOutput bounds the completion alone. Zero\nmeans unknown, which is what every model was until now.\n\nThe gateway knew neither, so an over-long prompt was not refused: it was\nadmitted, given a replica, and failed at the engine, and a client sizing\nits prompts from /v1/models had nothing to read. Worse, the number is not\nthe one an operator writes in the engine's arguments — llama.cpp divides\n--ctx-size across its parallel slots, so a pool of 8192 across four slots\nis 2048 a request, and nothing said so.", Nested: ""},
 		{Key: "max_output", Type: "int", Doc: "", Nested: ""},
 		{Key: "capabilities", Type: "Capabilities", Doc: "Capabilities is what this model can do beyond plain completion.", Nested: "catalog.Capabilities"},
+		{Key: "query_prefix", Type: "string", Doc: "QueryPrefix is put before a question an embedding model embeds for a\nknowledge-base search; documents are embedded without it. Models\ntrained with an instruction on the query side want it, Qwen3-Embedding\n\"Instruct: Given a web search query, retrieve relevant passages that\nanswer the query\\nQuery: \" for one.", Nested: ""},
 		{Key: "role", Type: "string", Doc: "Role \"guard\" is a model that judges the gateway's traffic for the\nguardrails rather than serving clients: hidden from /v1/models and the\nchat page, refused to any key that names it, critical by default, and\nalways running. Guard says how its answers are read.", Nested: ""},
 		{Key: "guard", Type: "*GuardModel", Doc: "", Nested: "catalog.GuardModel"},
 		{Key: "origin", Type: "*Origin", Doc: "Origin is the registry repository the weights came from, at which\ncommit, and each file's checksum there. Written when a model is served\nfrom the registry; for one that was not, worked out from the mirror's\nrecord of where it fetched the file. It is what lets the control plane\nsay that the repository has published different weights since, and\nmove the model to them.", Nested: "catalog.Origin"},
@@ -372,6 +382,7 @@ var Reference = []Block{
 		{Key: "mail", Type: "mail.Config", Doc: "Mail is where the gateway's few emails go out from: a link to set a\nforgotten password. Unset, none is sent and people ask an operator.", Nested: "mail.Config"},
 		{Key: "serving", Type: "ServingSettings", Doc: "Serving stops and starts the whole cluster's service (llmgw stop, llmgw\nstart).", Nested: "config.ServingSettings"},
 		{Key: "firewall", Type: "FirewallSettings", Doc: "Firewall turns llmgw's firewall on every machine on or off.", Nested: "config.FirewallSettings"},
+		{Key: "agents", Type: "AgentSettings", Doc: "Agents are the nodes' agent settings: agent.yaml's keys but those an\nagent needs to start, for every node (default) and for one\n(nodes.<id>). Sent to the agents as they change; an agent whose\nsettings changed starts again on them, its engines serving\nthroughout.", Nested: "config.AgentSettings"},
 		{Key: "firewall_networks", Type: "[]string", Doc: "FirewallNetworks are networks, besides those each machine reaches the\ncluster on, that llmgw's internal ports answer: a monitoring host on\nanother subnet, nodes behind a router. Written into every machine's\nfirewall by `server init`, `node add` and `server add`.", Nested: ""},
 		{Key: "backup_recipients", Type: "[]string", Doc: "BackupRecipients encrypt every archive `llmgw server backup` writes, to\nwhoever holds one of their keys: age recipients (age1…) or SSH public\nkeys (ssh-ed25519 …, ssh-rsa …), as `age` takes them. An archive holds\nthe certificate authority's key and every secret in the store, and is\ncopied off the machine; encrypted, the copy is safe wherever it lands.\nEmpty writes archives unencrypted, readable by root only.", Nested: ""},
 		{Key: "backup_copy_to", Type: "[]string", Doc: "BackupCopyTo is where each archive is copied once written, so it\nsurvives this machine: a directory (a mounted share, another disk) or\nssh://user@host[:port]/path, reached with BackupCopyKey and checked\nagainst /etc/llmgw/backup_known_hosts, which learns a host the first\ntime. The same number of archives is kept there as here. Refused\nwithout backup_recipients: an archive leaves the machine encrypted.", Nested: ""},
@@ -32,6 +32,7 @@ var Sources = []string{
 	"internal/config/oidc.go",
 	"internal/config/firewall.go",
 	"internal/config/serving.go",
+	"internal/config/agents.go",
 	"internal/mail/mail.go",
 	"internal/notify/notify.go",
 	"internal/secrets/secrets.go",
@@ -0,0 +1,154 @@
+package firewall
+
+import (
+	"context"
+	"fmt"
+	"strings"
+)
+
+// A store member that is nothing else runs no llmgw process: Patroni, etcd
+// and PostgreSQL only. What the cluster's settings say of its firewall it
+// reads from its own copy of the store, where the leading control plane
+// keeps it (the cluster_state table, key firewall), once a minute, by a
+// root timer running FollowScript. It turns llmgw's table off and on, and
+// lets in the networks the settings add; what `server init` let in stays.
+
+// FollowScript is the script, as python3 runs it. LLMGW_FW_STATE and
+// LLMGW_FW_RULES replace the store and the rules file, for its tests.
+const FollowScript = `#!/usr/bin/python3
+# Written by llmgw: llmgw's firewall on this store member, as the cluster's
+# settings say. Run by llmgw-firewall-follow.timer.
+import ipaddress, json, os, subprocess, sys, tempfile
+
+RULES = os.environ.get("LLMGW_FW_RULES", "` + RulesFile + `")
+
+def run(*a):
+    return subprocess.run(a, capture_output=True, text=True)
+
+raw = os.environ.get("LLMGW_FW_STATE")
+if raw is None:
+    r = run("runuser", "-u", "postgres", "--", "psql", "-d", "llmgw", "-tAqc",
+            "SELECT value FROM cluster_state WHERE key = 'firewall'")
+    # No table yet, or nothing told: a control plane older than this, or
+    # none leading. Left as it is.
+    if r.returncode != 0:
+        sys.exit(0)
+    raw = r.stdout
+if not raw.strip():
+    sys.exit(0)
+st = json.loads(raw)
+present = run("nft", "list", "table", "inet", "` + Table + `").returncode == 0
+
+if not st.get("enabled", True):
+    if present:
+        run("systemctl", "disable", "--now", "` + Unit + `")
+        run("nft", "delete", "table", "inet", "` + Table + `")
+        print("llmgw's firewall is off here")
+    sys.exit(0)
+
+if not os.path.exists(RULES):
+    sys.exit(0)
+lines = open(RULES).read().split("\n")
+want = []
+for n in st.get("networks") or []:
+    try:
+        want.append(ipaddress.ip_network(n, strict=False))
+    except ValueError:
+        pass
+
+def saddr(l):
+    s = l.strip()
+    return s.startswith("ip saddr {") or s.startswith("ip6 saddr {")
+
+changed = False
+for fam, kw in ((4, "ip saddr {"), (6, "ip6 saddr {")):
+    add = [str(n) for n in want if n.version == fam]
+    if not add:
+        continue
+    same = [i for i, l in enumerate(lines) if l.strip().startswith(kw)]
+    if same:
+        i = same[0]
+        l = lines[i]
+        a, b = l.index("{") + 1, l.index("}")
+        have = [x.strip() for x in l[a:b].split(",") if x.strip()]
+        new = [x for x in add if x not in have]
+        if new:
+            lines[i] = l[:a] + " " + ", ".join(have + new) + " " + l[b:]
+            changed = True
+        continue
+    other = [i for i, l in enumerate(lines) if saddr(l)]
+    if not other:
+        continue
+    l = lines[other[0]]
+    indent = l[:len(l) - len(l.lstrip())]
+    lines.insert(other[0] + 1, indent + kw + " " + ", ".join(add) + " }" + l[l.index("}") + 1:])
+    changed = True
+
+if changed:
+    fd, tmp = tempfile.mkstemp(dir=os.path.dirname(RULES))
+    with os.fdopen(fd, "w") as f:
+        f.write("\n".join(lines))
+    if run("nft", "-c", "-f", tmp).returncode != 0:
+        os.unlink(tmp)
+        print("the settings' networks make a ruleset nft refuses; left as it was", file=sys.stderr)
+        sys.exit(1)
+    os.chmod(tmp, 0o644)
+    os.replace(tmp, RULES)
+if changed or not present:
+    run("systemctl", "enable", "` + Unit + `")
+    if run("nft", "-f", RULES).returncode != 0:
+        sys.exit(1)
+    print("llmgw's firewall is on here")
+`
+
+// FollowerPath is where FollowScript is installed.
+const FollowerPath = "/usr/local/lib/llmgw/firewall-follow"
+
+// FollowerTimer is the timer that runs it.
+const FollowerTimer = "llmgw-firewall-follow.timer"
+
+const followService = `[Unit]
+Description=llmgw firewall: follow the cluster's settings on this store member
+After=postgresql.service patroni.service
+
+[Service]
+Type=oneshot
+ExecStart=/usr/bin/python3 ` + FollowerPath + `
+`
+
+const followTimer = `[Unit]
+Description=llmgw firewall: follow the cluster's settings on this store member
+
+[Timer]
+OnBootSec=1min
+OnUnitActiveSec=1min
+
+[Install]
+WantedBy=timers.target
+`
+
+// InstallFollower puts FollowScript and its timer on a store member.
+func InstallFollower(ctx context.Context, r Runner) error {
+	body := FollowScript + "\x00" + followService + "\x00" + followTimer
+	out, err := r.Run(ctx, `set -e
+t=$(mktemp -d); trap 'rm -rf "$t"' EXIT
+cat > "$t/all"
+python3 - "$t" <<'PY'
+import sys
+d = sys.argv[1]
+s, svc, tim = open(d + "/all", "rb").read().split(b"\x00")
+open(d + "/script", "wb").write(s)
+open(d + "/service", "wb").write(svc)
+open(d + "/timer", "wb").write(tim)
+PY
+sudo install -d -m 755 /usr/local/lib/llmgw
+sudo install -m 755 "$t/script" `+FollowerPath+`
+sudo install -m 644 "$t/service" /etc/systemd/system/llmgw-firewall-follow.service
+sudo install -m 644 "$t/timer" /etc/systemd/system/`+FollowerTimer+`
+sudo systemctl daemon-reload
+sudo systemctl enable --now `+FollowerTimer+` >/dev/null 2>&1`, []byte(body))
+	if err != nil {
+		return fmt.Errorf("installing %s: %v: %s", FollowerTimer, err, strings.TrimSpace(string(out)))
+	}
+	return nil
+}
@@ -0,0 +1,101 @@
+package firewall
+
+import (
+	"net/netip"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"strings"
+	"testing"
+)
+
+// follow runs FollowScript against a rules file and a state, with nft and
+// systemctl replaced by scripts that log what they were asked and answer as
+// a machine whose table is loaded or not.
+func follow(t *testing.T, rules, state string, loaded bool) (string, string, string) {
+	t.Helper()
+	if _, err := exec.LookPath("python3"); err != nil {
+		t.Skip("no python3")
+	}
+	dir := t.TempDir()
+	bin := filepath.Join(dir, "bin")
+	_ = os.Mkdir(bin, 0o755)
+	log := filepath.Join(dir, "log")
+	list := "exit 1"
+	if loaded {
+		list = "exit 0"
+	}
+	for name, body := range map[string]string{
+		"nft":       `echo "nft $*" >> ` + log + "\n" + `[ "$1" = list ] && ` + list + "\nexit 0\n",
+		"systemctl": `echo "systemctl $*" >> ` + log + "\nexit 0\n",
+	} {
+		if err := os.WriteFile(filepath.Join(bin, name), []byte("#!/bin/sh\n"+body), 0o755); err != nil {
+			t.Fatal(err)
+		}
+	}
+	rf := filepath.Join(dir, "firewall.nft")
+	if rules != "" {
+		_ = os.WriteFile(rf, []byte(rules), 0o644)
+	}
+	script := filepath.Join(dir, "follow")
+	_ = os.WriteFile(script, []byte(FollowScript), 0o755)
+	cmd := exec.Command("python3", script)
+	cmd.Env = append(os.Environ(), "PATH="+bin+":"+os.Getenv("PATH"), "LLMGW_FW_STATE="+state, "LLMGW_FW_RULES="+rf)
+	out, err := cmd.CombinedOutput()
+	if err != nil {
+		t.Fatalf("%v: %s", err, out)
+	}
+	got, _ := os.ReadFile(rf)
+	calls, _ := os.ReadFile(log)
+	return string(out), string(got), string(calls)
+}
+
+var memberRules = Script(Spec{Ports: []Port{Postgres, EtcdClient}, Allow: mustPrefixes("10.0.0.0/24")})
+
+func TestAStoreMemberFollowsTheSettings(t *testing.T) {
+	// Off: the table goes, the rules stay for later.
+	out, rules, calls := follow(t, memberRules, `{"enabled":false}`, true)
+	if !strings.Contains(out, "off") || !strings.Contains(calls, "nft delete table inet llmgw") ||
+		!strings.Contains(calls, "systemctl disable --now "+Unit) || rules != memberRules {
+		t.Errorf("off: %q %q", out, calls)
+	}
+	// On again: loaded from the rules kept.
+	out, _, calls = follow(t, memberRules, `{"enabled":true}`, false)
+	if !strings.Contains(out, "on") || !strings.Contains(calls, "nft -f") {
+		t.Errorf("on: %q %q", out, calls)
+	}
+	// On and loaded, nothing new: nothing done.
+	out, _, calls = follow(t, memberRules, `{"enabled":true,"networks":["10.0.0.0/24"]}`, true)
+	if out != "" || strings.Contains(calls, "nft -f") {
+		t.Errorf("unchanged: %q %q", out, calls)
+	}
+	// Networks the settings add, of both families, are let in beside what
+	// init let in; the result reads back as the same ports.
+	_, rules, calls = follow(t, memberRules, `{"enabled":true,"networks":["192.168.5.0/24","fd00::/64","nonsense"]}`, true)
+	if !strings.Contains(calls, "nft -c -f") || !strings.Contains(calls, "nft -f") {
+		t.Errorf("not checked and loaded: %q", calls)
+	}
+	back := parse(rules)
+	var nets []string
+	for _, p := range back.Allow {
+		nets = append(nets, p.String())
+	}
+	if strings.Join(nets, ",") != "10.0.0.0/24,192.168.5.0/24,fd00::/64" || len(back.Ports) != 2 {
+		t.Errorf("rules now:\n%s", rules)
+	}
+	// Nothing told yet, or no rules: left alone.
+	if out, _, calls := follow(t, memberRules, "", true); out != "" || calls != "" {
+		t.Errorf("nothing told: %q %q", out, calls)
+	}
+	if out, _, calls := follow(t, "", `{"enabled":true}`, false); out != "" || strings.Contains(calls, "nft -f") {
+		t.Errorf("no rules: %q %q", out, calls)
+	}
+}
+
+func mustPrefixes(ss ...string) []netip.Prefix {
+	var out []netip.Prefix
+	for _, s := range ss {
+		out = append(out, netip.MustParsePrefix(s))
+	}
+	return out
+}
@@ -4,6 +4,7 @@ import (
 	"encoding/json"
 	"errors"
 	"fmt"
+	"io"
 	"net/http"
 	"os"
 	"strings"
@@ -37,7 +38,9 @@ func (a *API) Handler() http.Handler {
 	mux.HandleFunc("POST /v1/models", a.fetch)
 	mux.HandleFunc("GET /v1/models/{model}/{version}", a.status)
 	mux.HandleFunc("DELETE /v1/models/{model}/{version}", a.remove)
+	mux.HandleFunc("PUT /v1/models/{model}/{version}/{file}", a.upload)
 	mux.HandleFunc("GET /v1/health", a.health)
+	mux.HandleFunc("PUT /v1/firewall", a.putFirewall)
 	return mux
 }

@@ -199,3 +202,59 @@ func LastPathElement(u string) string {
 	}
 	return u
 }
+
+// maxUpload bounds a file sent to the mirror rather than fetched by it.
+const maxUpload = 8 << 30
+
+// upload keeps a file the caller sends — an adapter a node trained, which
+// is on no registry to fetch from — with the same discipline as an import:
+// its checksum and source beside it, and an existing version never
+// replaced. X-Llmgw-Source says where it came from.
+func (a *API) upload(w http.ResponseWriter, r *http.Request) {
+	e := Entry{Model: r.PathValue("model"), Version: r.PathValue("version"), File: r.PathValue("file")}
+	for _, n := range []string{e.Model, e.Version, e.File} {
+		if err := ValidName(n); err != nil {
+			fail(w, http.StatusBadRequest, err)
+			return
+		}
+	}
+	if _, err := os.Stat(e.Path(a.m.root)); err == nil {
+		fail(w, http.StatusConflict, fmt.Errorf("%s/%s/%s: %w", e.Model, e.Version, e.File, ErrExists))
+		return
+	}
+	tmp, err := os.CreateTemp(a.m.root, ".upload-*")
+	if err != nil {
+		fail(w, http.StatusInternalServerError, err)
+		return
+	}
+	defer os.Remove(tmp.Name())
+	n, err := io.Copy(tmp, http.MaxBytesReader(w, r.Body, maxUpload))
+	if cerr := tmp.Close(); err == nil {
+		err = cerr
+	}
+	if err != nil {
+		fail(w, http.StatusBadRequest, fmt.Errorf("receiving the file: %w", err))
+		return
+	}
+	if n == 0 {
+		fail(w, http.StatusBadRequest, errors.New("the file is empty"))
+		return
+	}
+	source := strings.TrimSpace(r.Header.Get("X-Llmgw-Source"))
+	if source == "" {
+		source = "uploaded"
+	}
+	added, err := Import(r.Context(), a.m.root, tmp.Name(), e, "", source, nil)
+	switch {
+	case errors.Is(err, ErrExists):
+		fail(w, http.StatusConflict, err)
+		return
+	case err != nil:
+		fail(w, http.StatusBadRequest, err)
+		return
+	}
+	writeJSON(w, http.StatusCreated, FetchResponse{
+		Status: Status{Model: e.Model, Version: e.Version, File: e.File, State: Present, BytesDone: added.Entry.Bytes, BytesTotal: added.Entry.Bytes},
+		URL:    e.URL(a.base),
+	})
+}
@@ -100,6 +100,36 @@ func (c *Client) Fetch(ctx context.Context, req FetchRequest) (FetchResponse, er
 	return out, err
 }

+// Upload sends a file for the mirror to keep at model/version/file, with
+// where it came from; it answers the URL nodes fetch it from.
+func (c *Client) Upload(ctx context.Context, model, version, file string, body []byte, source string) (FetchResponse, error) {
+	var out FetchResponse
+	req, err := http.NewRequestWithContext(ctx, http.MethodPut,
+		c.base+"/v1/models/"+url.PathEscape(model)+"/"+url.PathEscape(version)+"/"+url.PathEscape(file), bytes.NewReader(body))
+	if err != nil {
+		return out, err
+	}
+	req.Header.Set("Content-Type", "application/octet-stream")
+	req.Header.Set("X-Llmgw-Source", source)
+	resp, err := c.hc.Do(req)
+	if err != nil {
+		return out, fmt.Errorf("reaching the mirror at %s: %w", c.base, err)
+	}
+	defer resp.Body.Close()
+	raw, err := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
+	if err != nil {
+		return out, err
+	}
+	if resp.StatusCode >= 300 {
+		var e struct {
+			Error string `json:"error"`
+		}
+		_ = json.Unmarshal(raw, &e)
+		return out, &APIError{Status: resp.StatusCode, Msg: e.Error}
+	}
+	return out, json.Unmarshal(raw, &out)
+}
+
 // Status is where one model version stands.
 func (c *Client) Status(ctx context.Context, model, version string) (FetchResponse, error) {
 	var out FetchResponse
@@ -0,0 +1,65 @@
+package mirror
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"net/http"
+	"os"
+	"path/filepath"
+)
+
+// The mirror's machine keeps llmgw's firewall as the cluster's settings
+// say, as every other machine does. The mirror runs unprivileged and cannot
+// change the ruleset: the control plane tells it what the settings say, it
+// writes that beside its models, and a root path unit on the machine
+// applies it with `llmgw firewall sync`.
+
+// FirewallState is what the cluster's settings say of a machine's firewall.
+type FirewallState struct {
+	Enabled bool `json:"enabled"`
+	// Networks are let in besides those the machine reaches others on.
+	Networks []string `json:"networks,omitempty"`
+	// Toward are the cluster's machines, by address: each one's network is
+	// let in, as the machine's route to it says.
+	Toward []string `json:"toward,omitempty"`
+}
+
+// FirewallFile is where, in the mirror's directory, the state is written.
+// Hidden, so it is never taken for a model.
+const FirewallFile = ".firewall.json"
+
+func (a *API) putFirewall(w http.ResponseWriter, r *http.Request) {
+	var st FirewallState
+	if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 1<<20)).Decode(&st); err != nil {
+		fail(w, http.StatusBadRequest, err)
+		return
+	}
+	b, _ := json.Marshal(st)
+	p := filepath.Join(a.m.root, FirewallFile)
+	changed := false
+	if old, err := os.ReadFile(p); err != nil || !bytes.Equal(old, b) {
+		// Renamed into place: the path unit sees one change, of a whole file.
+		tmp := p + ".new"
+		if err := os.WriteFile(tmp, b, 0o644); err != nil {
+			fail(w, http.StatusInternalServerError, err)
+			return
+		}
+		if err := os.Rename(tmp, p); err != nil {
+			fail(w, http.StatusInternalServerError, err)
+			return
+		}
+		changed = true
+	}
+	writeJSON(w, http.StatusOK, map[string]bool{"changed": changed})
+}
+
+// SetFirewall tells the mirror what the cluster's settings say of its
+// machine's firewall; it answers whether that changed anything.
+func (c *Client) SetFirewall(ctx context.Context, st FirewallState) (bool, error) {
+	var out struct {
+		Changed bool `json:"changed"`
+	}
+	err := c.do(ctx, http.MethodPut, "/v1/firewall", st, &out)
+	return out.Changed, err
+}
@@ -0,0 +1,38 @@
+package mirror
+
+import (
+	"context"
+	"encoding/json"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
+)
+
+func TestTheMirrorKeepsWhatTheSettingsSayOfItsFirewall(t *testing.T) {
+	root := t.TempDir()
+	srv := httptest.NewServer(NewAPI(NewManager(root, discardLog()), "http://m:8081").Handler())
+	defer srv.Close()
+	c, err := NewClient(srv.URL, tlsconf.Config{}, 0)
+	if err != nil {
+		t.Fatal(err)
+	}
+	ctx := context.Background()
+	st := FirewallState{Enabled: true, Networks: []string{"192.168.5.0/24"}, Toward: []string{"10.0.0.147"}}
+	if changed, err := c.SetFirewall(ctx, st); err != nil || !changed {
+		t.Fatalf("first: %v %v", changed, err)
+	}
+	if changed, err := c.SetFirewall(ctx, st); err != nil || changed {
+		t.Errorf("the same again: %v %v", changed, err)
+	}
+	var back FirewallState
+	b, _ := os.ReadFile(filepath.Join(root, FirewallFile))
+	if err := json.Unmarshal(b, &back); err != nil || !back.Enabled || back.Toward[0] != "10.0.0.147" {
+		t.Errorf("kept %s", b)
+	}
+	if all, err := c.List(ctx); err != nil || len(all.Models) != 0 {
+		t.Errorf("the file is taken for a model: %+v %v", all, err)
+	}
+}
@@ -0,0 +1,45 @@
+package mirror
+
+import (
+	"context"
+	"net/http/httptest"
+	"os"
+	"strings"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
+)
+
+func TestAFileSentToTheMirrorIsKeptWithItsChecksumAndNeverReplaced(t *testing.T) {
+	root := t.TempDir()
+	srv := httptest.NewServer(NewAPI(NewManager(root, nil), "http://mirror:8081").Handler())
+	defer srv.Close()
+	c, err := NewClient(srv.URL, tlsconf.Config{}, 0)
+	if err != nil {
+		t.Fatal(err)
+	}
+	ctx := context.Background()
+	got, err := c.Upload(ctx, "politika-tonu", "train-1", "politika-tonu.gguf", []byte("GGUF adapter"), "trained by job train-1")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if got.URL != "http://mirror:8081/politika-tonu/train-1/politika-tonu.gguf" || got.Status.BytesTotal != 12 {
+		t.Errorf("answered %+v", got)
+	}
+	e := Entry{Model: "politika-tonu", Version: "train-1", File: "politika-tonu.gguf"}
+	if b, _ := os.ReadFile(e.Path(root)); string(b) != "GGUF adapter" {
+		t.Errorf("kept %q", b)
+	}
+	if b, _ := os.ReadFile(sourcePath(e.Path(root))); !strings.Contains(string(b), "train-1") {
+		t.Errorf("source %q", b)
+	}
+	if _, err := os.Stat(checksumPath(e.Path(root))); err != nil {
+		t.Errorf("no checksum: %v", err)
+	}
+	if _, err := c.Upload(ctx, "politika-tonu", "train-1", "politika-tonu.gguf", []byte("other"), "x"); err == nil {
+		t.Error("an existing version was replaced")
+	}
+	if _, err := c.Upload(ctx, "../etc", "v", "f", []byte("x"), "x"); err == nil {
+		t.Error("a path outside the mirror was taken")
+	}
+}
@@ -0,0 +1,171 @@
+package pgha
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"slices"
+	"strconv"
+	"strings"
+	"time"
+)
+
+// PostgreSQL's own defaults — 128MB of cache, 4MB to sort in — are for a
+// machine it barely uses. A knowledge base's vector index is searched well
+// only from memory: one that does not fit is read from disk on every search.
+// So each member is sized to the memory it has, read when setup runs, and
+// again on every later run, so a machine given more memory gets the use of it.
+
+// MemorySettings are PostgreSQL's memory parameters for a machine with total
+// bytes of memory. shared is a machine that also runs a control plane or an
+// agent: the database is given less of it, since a gateway swapped out to
+// give the database cache is worse than a smaller cache.
+func MemorySettings(total int64, shared bool) map[string]string {
+	const mb = 1 << 20
+	ram := total / mb
+	// Shares of memory, in MB: the cache PostgreSQL keeps itself, what the
+	// planner may count on the kernel caching besides, what building or
+	// vacuuming an index may take, and what one sort may.
+	buffers, cache, maint, sort := ram/4, ram*3/4, ram/16, ram/256
+	if shared {
+		buffers, cache, maint, sort = ram*3/20, ram/2, ram/32, ram/512
+	}
+	clamp := func(v, lo, hi int64) int64 { return min(max(v, lo), hi) }
+	return map[string]string{
+		"shared_buffers":       strconv.FormatInt(clamp(buffers, 96, 32<<10), 10) + "MB",
+		"effective_cache_size": strconv.FormatInt(clamp(cache, 256, 1<<20), 10) + "MB",
+		"maintenance_work_mem": strconv.FormatInt(clamp(maint, 64, 2<<10), 10) + "MB",
+		"work_mem":             strconv.FormatInt(clamp(sort, 4, 64), 10) + "MB",
+	}
+}
+
+// MemTotal reads the machine's memory, in bytes, from /proc/meminfo.
+func MemTotal(meminfo string) (int64, bool) {
+	for _, line := range strings.Split(meminfo, "\n") {
+		f := strings.Fields(line)
+		if len(f) >= 2 && f[0] == "MemTotal:" {
+			kb, err := strconv.ParseInt(f[1], 10, 64)
+			return kb << 10, err == nil && kb > 0
+		}
+	}
+	return 0, false
+}
+
+// memory sizes each member to its machine, in its own configuration — the
+// machines need not be alike. What takes effect on reload is applied so;
+// shared_buffers needs a restart, given one member at a time, the leader
+// last, each waited on before the next.
+func (s *Setup) memory(ctx context.Context, sec Secrets) error {
+	var restart []Node
+	for _, d := range s.DB {
+		out, err := d.Remote.Run(ctx, `cat /proc/meminfo
+if [ -f /etc/llmgw/server.yaml ] || [ -f /etc/llmgw/agent.yaml ]; then echo "llmgw-shared"; fi`, nil)
+		total, ok := MemTotal(string(out))
+		if err != nil || !ok {
+			s.emit("memory", false, "%s: could not read its memory; PostgreSQL left as it is", d.Name)
+			continue
+		}
+		shared := strings.Contains(string(out), "llmgw-shared")
+		params := MemorySettings(total, shared)
+		body, _ := json.Marshal(params)
+		out, err = d.Remote.Run(ctx, `set -e
+test -f /etc/patroni/config.yml || { echo missing; exit 0; }
+want=$(cat)
+r=$(sudo env WANT="$want" python3 - <<'PY'
+import json, os, yaml
+p = "/etc/patroni/config.yml"
+want = json.loads(os.environ["WANT"])
+c = yaml.safe_load(open(p))
+pg = c.setdefault("postgresql", {})
+params = pg.get("parameters") or {}
+pg["parameters"] = params
+if all(str(params.get(k)) == v for k, v in want.items()):
+    print("same")
+else:
+    restart = str(params.get("shared_buffers")) != want["shared_buffers"]
+    params.update(want)
+    open(p + ".new", "w").write("# llmgw: the control planes' store, kept by Patroni, its leader lock in etcd.\n" +
+        yaml.safe_dump(c, default_flow_style=False, sort_keys=False))
+    print("restart" if restart else "changed")
+PY
+)
+if [ "$r" != same ]; then
+  sudo chown root:postgres /etc/patroni/config.yml.new
+  sudo chmod 640 /etc/patroni/config.yml.new
+  sudo mv -f /etc/patroni/config.yml.new /etc/patroni/config.yml
+  sudo systemctl reload patroni
+fi
+echo "$r"`, body)
+		if err != nil {
+			return fmt.Errorf("on %s: %v: %s", d.Name, err, strings.TrimSpace(string(out)))
+		}
+		what := fmt.Sprintf("%s: %d MB of memory%s; shared_buffers %s, effective_cache_size %s, maintenance_work_mem %s, work_mem %s",
+			d.Name, total>>20, map[bool]string{true: ", shared with llmgw"}[shared],
+			params["shared_buffers"], params["effective_cache_size"], params["maintenance_work_mem"], params["work_mem"])
+		switch r := strings.TrimSpace(lastLine(out)); r {
+		case "same", "missing":
+			s.emit("memory", false, "%s", what)
+		case "restart":
+			s.emit("memory", true, "%s", what)
+			restart = append(restart, d)
+		default:
+			s.emit("memory", true, "%s", what)
+		}
+	}
+	// Patroni takes a reload in its own time: restarted before it has written
+	// the new configuration, PostgreSQL comes back on the old one. So each is
+	// waited on until Patroni says a restart is pending — and a member found
+	// pending, by a run that stopped before restarting it, is restarted too.
+	pending := func(c Cluster, name string) bool {
+		for _, m := range c.Members {
+			if m.Name == name {
+				return m.PendingRestart
+			}
+		}
+		return false
+	}
+	var c Cluster
+	err := s.waitFor(ctx, s.dbHosts(), func(got Cluster) bool {
+		c = got
+		for _, d := range restart {
+			if !pending(got, d.Name) {
+				return false
+			}
+		}
+		return true
+	})
+	if err != nil {
+		return fmt.Errorf("waiting for Patroni to take the new configuration: %w", err)
+	}
+	for _, d := range s.DB {
+		if pending(c, d.Name) && !slices.ContainsFunc(restart, func(n Node) bool { return n.Name == d.Name }) {
+			restart = append(restart, d)
+		}
+	}
+	if len(restart) == 0 {
+		return nil
+	}
+	// The leader last: its restart is the only one the control planes see,
+	// a few seconds without writes, and by then the rest are back.
+	slices.SortStableFunc(restart, func(a, b Node) int {
+		al, bl := a.Name == c.Leader(), b.Name == c.Leader()
+		switch {
+		case al && !bl:
+			return 1
+		case bl && !al:
+			return -1
+		}
+		return 0
+	})
+	for _, d := range restart {
+		if _, err := restJSON(ctx, d.IP, http.MethodPost, "/restart", []byte(`{"restart_pending": true}`), sec.RestAPI, 2*time.Minute); err != nil {
+			return fmt.Errorf("restarting PostgreSQL on %s for its new shared_buffers: %w", d.Name, err)
+		}
+		if err := s.waitFor(ctx, s.dbHosts(), func(c Cluster) bool { return c.Complete(len(s.DB)) }); err != nil {
+			return fmt.Errorf("after restarting %s: %w", d.Name, err)
+		}
+		s.emit("memory", true, "%s: PostgreSQL restarted with the new shared_buffers", d.Name)
+	}
+	return nil
+}
@@ -465,3 +465,97 @@ func TestAControlPlaneWithoutTheAuthorityConnectsByTLSOnceTheClusterHasIt(t *tes
 		t.Errorf("TLS asked of a cluster without it: %s", dsn)
 	}
 }
+
+func TestPostgreSQLIsGivenAShareOfTheMachinesMemory(t *testing.T) {
+	total, ok := MemTotal("MemTotal:        4009848 kB\nMemFree:          976000 kB\n")
+	if !ok || total != 4009848<<10 {
+		t.Fatalf("MemTotal %d %v", total, ok)
+	}
+	own := MemorySettings(total, false)
+	if own["shared_buffers"] != "978MB" || own["effective_cache_size"] != "2936MB" ||
+		own["maintenance_work_mem"] != "244MB" || own["work_mem"] != "15MB" {
+		t.Errorf("a machine of its own: %v", own)
+	}
+	// Beside a control plane, less of it.
+	if sh := MemorySettings(total, true); sh["shared_buffers"] != "587MB" || sh["work_mem"] != "7MB" {
+		t.Errorf("a shared machine: %v", sh)
+	}
+	// Never below what PostgreSQL needs, never past what it can use.
+	if tiny := MemorySettings(512<<20, true); tiny["shared_buffers"] != "96MB" || tiny["work_mem"] != "4MB" {
+		t.Errorf("a small machine: %v", tiny)
+	}
+	if big := MemorySettings(1<<40, false); big["shared_buffers"] != "32768MB" || big["maintenance_work_mem"] != "2048MB" {
+		t.Errorf("a large machine: %v", big)
+	}
+}
+
+func TestNewSharedBuffersRestartTheReplicasBeforeTheLeader(t *testing.T) {
+	var mu sync.Mutex
+	var restarted []string
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if r.Method == http.MethodPost && r.URL.Path == "/restart" {
+			if _, pass, _ := r.BasicAuth(); pass != "pw" {
+				w.WriteHeader(http.StatusUnauthorized)
+				return
+			}
+			host, _, _ := strings.Cut(r.Host, ":")
+			mu.Lock()
+			restarted = append(restarted, host)
+			mu.Unlock()
+		}
+	}))
+	defer srv.Close()
+	u, _ := url.Parse(srv.URL)
+	old := restPort
+	restPort, _ = strconv.Atoi(u.Port())
+	defer func() { restPort = old }()
+
+	answers := map[string]string{"cat /proc/meminfo": "MemTotal: 4009848 kB\n", "WANT": "restart\n"}
+	leader := &machine{name: "a", answers: answers}
+	replica := &machine{name: "b", answers: answers}
+	up := Cluster{Members: []MemberState{{Name: "a", Role: "leader", State: "running", PendingRestart: true},
+		{Name: "b", Role: "replica", State: "streaming", PendingRestart: true}}}
+	s := Setup{
+		DB:     []Node{{Member{"a", "127.0.0.1"}, leader}, {Member{"b", "localhost"}, replica}},
+		Status: func(context.Context, []string) (Cluster, error) { return up, nil },
+		Wait:   time.Second,
+	}
+	if err := s.memory(context.Background(), Secrets{RestAPI: "pw"}); err != nil {
+		t.Fatal(err)
+	}
+	if strings.Join(restarted, " ") != "localhost 127.0.0.1" {
+		t.Errorf("restarted %v; the replica first, then the leader", restarted)
+	}
+}
+
+func TestAMemberLeftPendingARestartIsRestartedOnTheNextRun(t *testing.T) {
+	// A run that stopped after writing the configuration, or restarted before
+	// Patroni had taken it: nothing to change, but PostgreSQL still on the old.
+	var restarted []string
+	srv := httptest.NewServer(http.HandlerFunc(func(_ http.ResponseWriter, r *http.Request) {
+		if r.URL.Path == "/restart" {
+			host, _, _ := strings.Cut(r.Host, ":")
+			restarted = append(restarted, host)
+		}
+	}))
+	defer srv.Close()
+	u, _ := url.Parse(srv.URL)
+	old := restPort
+	restPort, _ = strconv.Atoi(u.Port())
+	defer func() { restPort = old }()
+
+	answers := map[string]string{"cat /proc/meminfo": "MemTotal: 4009848 kB\n", "WANT": "same\n"}
+	up := Cluster{Members: []MemberState{{Name: "a", Role: "leader", State: "running"},
+		{Name: "b", Role: "replica", State: "streaming", PendingRestart: true}}}
+	s := Setup{
+		DB:     []Node{{Member{"a", "127.0.0.1"}, &machine{answers: answers}}, {Member{"b", "localhost"}, &machine{answers: answers}}},
+		Status: func(context.Context, []string) (Cluster, error) { return up, nil },
+		Wait:   time.Second,
+	}
+	if err := s.memory(context.Background(), Secrets{RestAPI: "pw"}); err != nil {
+		t.Fatal(err)
+	}
+	if strings.Join(restarted, " ") != "localhost" {
+		t.Errorf("restarted %v; only the member pending", restarted)
+	}
+}
@@ -128,6 +128,9 @@ func (s *Setup) Run(ctx context.Context) (string, error) {
 	if err := s.ensurePatroniTLS(ctx, sec); err != nil {
 		return "", fmt.Errorf("patroni api: %w", err)
 	}
+	if err := s.memory(ctx, sec); err != nil {
+		return "", fmt.Errorf("memory: %w", err)
+	}
 	if err := s.database(ctx); err != nil {
 		return "", fmt.Errorf("database: %w", err)
 	}
@@ -19,6 +19,9 @@ type MemberState struct {
 	State    string `json:"state"`
 	Host     string `json:"host"`
 	Timeline int    `json:"timeline"`
+	// PendingRestart is a parameter changed that PostgreSQL takes only on
+	// restart.
+	PendingRestart bool `json:"pending_restart"`
 	// Lag is how far behind the leader a replica is, in bytes; -1 when
 	// Patroni does not know, which is itself worth an alert.
 	Lag    int64           `json:"-"`
@@ -0,0 +1,61 @@
+package router
+
+import (
+	"crypto/sha256"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/auth"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+func TestAnOwnedAdapterIsSeenOnlyInItsPartOfTheTree(t *testing.T) {
+	e := newFakeEngine(t, "qwen2.5-14b-instruct")
+	rt, owner := pricedRouter(t, e.srv.URL, "tone")
+	ctx := t.Context()
+	key, _ := rt.store.KeyByID(ctx, "key-1")
+	team, _ := rt.store.CreateTeam(ctx, mustOrgOf(t, rt), "other-team")
+	other, _ := rt.store.CreateProject(ctx, team, "other")
+	otherSecret := store.KeyPrefix + strings.Repeat("b", 43)
+	sum := sha256.Sum256([]byte(otherSecret))
+	if err := rt.store.CreateKey(ctx, store.Key{ID: "key-2", Hash: sum[:], Prefix: "bbbb", Name: "o", ProjectID: other, CreatedAt: time.Now()}); err != nil {
+		t.Fatal(err)
+	}
+	rt.storeModels([]catalog.Model{{Name: "qwen2.5-14b-instruct", Aliases: []string{"q14"},
+		Adapters: []catalog.Adapter{{Name: "tone", Source: "/t.gguf", Owner: "project:" + key.ProjectID}}}})
+	h := rt.Handler()
+	ask := func(secret string) int {
+		req := httptest.NewRequest(http.MethodPost, "/v1/chat/completions",
+			strings.NewReader(`{"model":"tone","messages":[{"role":"user","content":"hi"}]}`))
+		if secret != "" {
+			req.Header.Set("Authorization", "Bearer "+secret)
+		}
+		rec := httptest.NewRecorder()
+		h.ServeHTTP(rec, req)
+		return rec.Code
+	}
+	if code := ask(owner); code != http.StatusOK {
+		t.Errorf("its own project: %d", code)
+	}
+	if code := ask(otherSecret); code != http.StatusNotFound {
+		t.Errorf("another project: %d", code)
+	}
+	list := func(secret string) string {
+		req := httptest.NewRequest(http.MethodGet, "/v1/models", nil)
+		req.Header.Set("Authorization", "Bearer "+secret)
+		rec := httptest.NewRecorder()
+		h.ServeHTTP(rec, req)
+		return rec.Body.String()
+	}
+	if !strings.Contains(list(owner), `"tone"`) || strings.Contains(list(otherSecret), `"tone"`) {
+		t.Errorf("listed: own %s\nother %s", list(owner), list(otherSecret))
+	}
+	// No identity, no owned adapter.
+	if rt.adapterHidden(auth.Principal{}, false, "tone") != true {
+		t.Error("an owned adapter was shown to a caller with no identity")
+	}
+}
@@ -42,7 +42,7 @@ func (r *Router) budgetFallback(p auth.Principal, req *parsedRequest) (accountin
 		// answer than saying the prompt is too large, and it would be done
 		// silently, to a request that was already being quietly moved.
 		canonical, cand := r.resolve(name)
-		if why := checkContextFits(cand, req.PromptTokens, req.MaxTokens); why != "" {
+		if why := checkContextFits(cand, req.contextTokens(), req.MaxTokens); why != "" {
 			continue
 		}
 		res, err := r.acct.Check(p, canonical, req.estimatedTokens())
@@ -48,12 +48,23 @@ type Info struct {
 	Peer        string
 	PeerCluster string
 	PeerTags    bool
+	// Workflow is the ComfyUI workflow sent for an endpoint whose engine is
+	// EngineComfyUI: a built-in's name or the workflow's JSON.
+	Workflow string
 }

 // EngineExternal marks an endpoint that is a provider outside the cluster.
 // It publishes no metrics, so it is never scraped and never reads as stale.
 const EngineExternal = "external"

+// EngineComfyUI marks a ComfyUI server. The router speaks ComfyUI's API to
+// it, not OpenAI's, and it publishes no metrics either.
+const EngineComfyUI = "comfyui"
+
+// Scraped reports whether an engine publishes metrics the router reads. One
+// that does not is never read as stale for having none.
+func Scraped(engine string) bool { return engine != EngineExternal && engine != EngineComfyUI }
+
 // Serves reports whether the replica serves the given model.
 func (i Info) Serves(model string) bool {
 	return slices.Contains(i.Models, model)
@@ -257,7 +268,7 @@ func (r *Registry) Snapshots(model string) []Snapshot {
 			continue
 		}
 		var age time.Duration
-		stale := l.info.Engine != EngineExternal
+		stale := Scraped(l.info.Engine)
 		if !l.scrapedAt.IsZero() {
 			age = now.Sub(l.scrapedAt)
 			stale = age > r.cfg.StaleAfter
@@ -102,6 +102,8 @@ func (r *Router) Handler() http.Handler {
 	mux.HandleFunc("POST /v1/score", r.handleInference("/v1/score"))
 	mux.HandleFunc("POST /v1/audio/transcriptions", r.handleInference("/v1/audio/transcriptions"))
 	mux.HandleFunc("POST /v1/audio/translations", r.handleInference("/v1/audio/translations"))
+	// Pictures, made by ComfyUI from a workflow the catalog names.
+	mux.HandleFunc("POST "+imagesPath, r.handleInference(imagesPath))
 	// Answered by the guardrails' judge model.
 	mux.HandleFunc("POST /v1/moderations", r.handleModerations)
 	// Codex, and whatever else speaks only the Responses API, translated to
@@ -247,6 +249,9 @@ func (r *Router) Handler() http.Handler {
 		if r.store != nil && r.guardLibrary != nil {
 			pages.RegisterAdminGuard(mux, r.store, r.guardLibrary)
 		}
+		if r.store != nil && r.settingsCheck != nil {
+			pages.RegisterAdminSettings(mux, r.store, r.settingsCheck, r.settingsLocal)
+		}
 		pages.Register(mux)
 		// The docs portal, with the menu of whoever is signed in, and only to
 		// them when the site asked for that.
@@ -388,7 +393,11 @@ type parsedRequest struct {
 	ProfileOverride string
 	PromptPrefix    string
 	PromptTokens    int
-	MaxTokens       int
+	// ContextTokens is what has to fit in one context: the prompt, except
+	// for embeddings, where each input is embedded on its own and the
+	// longest one is what a context must hold.
+	ContextTokens int
+	MaxTokens     int

 	// ThreadID is the conversation this request continues, empty for the
 	// stateless path every request took before threads existed.
@@ -409,9 +418,9 @@ type parsedRequest struct {
 	guardIn guard.Verdict
 	// knowledge is set when passages from knowledge bases were added.
 	knowledge bool
-	// kbOrg is the organization whose passages were added, for the prefix
+	// kbSalt is the collections whose passages were added, for the prefix
 	// cache's salt.
-	kbOrg string
+	kbSalt string
 	// noStore is a policy that keeps nothing of this request.
 	noStore bool
 	// inCluster is a project whose requests may not leave the cluster.
@@ -470,18 +479,27 @@ func (r *Router) handleInference(path string) http.HandlerFunc {
 			return
 		}

+		// A picture asked of a model that writes text, or the other way
+		// round, is refused before it waits for either.
+		if why := r.imageRouteRefusal(path, parsed.Model, parsed.UpstreamBody); why != "" {
+			writeError(w, http.StatusBadRequest, "invalid_request_error", why)
+			r.m.observeRequest(r.metricModel(parsed.Model), "", http.StatusBadRequest, time.Since(start), nil, 0)
+			return
+		}
+
 		// Before authentication work and before admission: a request that
 		// cannot fit should not consume a quota reservation or a queue slot on
 		// its way to being refused.
 		if _, entry := r.resolve(parsed.Model); entry.ContextLength > 0 {
-			if why := checkContextFits(entry, parsed.PromptTokens, parsed.MaxTokens); why != "" {
+			if why := checkContextFits(entry, parsed.contextTokens(), parsed.MaxTokens); why != "" {
 				writeError(w, http.StatusBadRequest, "invalid_request_error", why)
 				r.m.observeRequest(r.metricModel(parsed.Model), "", http.StatusBadRequest, time.Since(start), nil, 0)
 				return
 			}
 		}

-		if r.hiddenModel(parsed.Model) {
+		p0, authed := auth.FromContext(req.Context())
+		if r.hiddenModel(parsed.Model) || r.adapterHidden(p0, authed, parsed.Model) {
 			// Said as a model that is not there: it is not, for clients.
 			writeErrorCode(w, http.StatusNotFound, "invalid_request_error", "model_not_found",
 				fmt.Sprintf("the model %q does not exist", parsed.Model))
@@ -821,6 +839,12 @@ func (r *Router) parse(req *http.Request, path string) (*parsedRequest, error) {
 	// plan §14). Measured over the whole prompt, not the routing prefix —
 	// see promptBytes.
 	p.PromptTokens = promptBytes(&b) / 4
+	p.ContextTokens = p.PromptTokens
+	if path == "/v1/embeddings" {
+		if longest, ok := longestInput(b.Input); ok {
+			p.ContextTokens = longest / 4
+		}
+	}

 	// Tags are the gateway's: an engine has no use for them, and its logs
 	// are no place for what a client wrote there.
@@ -893,6 +917,29 @@ func promptPrefix(b *inferenceBody) string {
 // bounded head of the conversation; a token estimate wants all of it, and using
 // one for the other meant TPM limits undercounted exactly the requests that
 // consume the most, and nothing could tell whether a prompt fit in a context.
+// longestInput is the length of an embeddings request's longest input, when
+// input is a list of strings: a batch of texts, each its own context.
+func longestInput(raw json.RawMessage) (int, bool) {
+	var list []string
+	if len(raw) == 0 || raw[0] != '[' || json.Unmarshal(raw, &list) != nil || len(list) == 0 {
+		return 0, false
+	}
+	n := 0
+	for _, s := range list {
+		n = max(n, len(s))
+	}
+	return n, true
+}
+
+// contextTokens is what has to fit in one context; requests built inside
+// the router set only PromptTokens.
+func (p *parsedRequest) contextTokens() int {
+	if p.ContextTokens > 0 {
+		return p.ContextTokens
+	}
+	return p.PromptTokens
+}
+
 func promptBytes(b *inferenceBody) int {
 	n := 0
 	add := func(raw json.RawMessage) {
@@ -1038,7 +1085,7 @@ func (r *Router) handleModels(w http.ResponseWriter, req *http.Request) {
 	principal, authed := auth.FromContext(req.Context())
 	names := make([]string, 0, len(seen))
 	for n := range seen {
-		if authed && !r.permits(principal, n) || r.hiddenModel(n) {
+		if authed && !r.permits(principal, n) || r.hiddenModel(n) || r.adapterHidden(principal, authed, n) {
 			continue
 		}
 		names = append(names, n)
@@ -1084,7 +1131,7 @@ func (r *Router) handleModels(w http.ResponseWriter, req *http.Request) {
 		}
 		entry["capabilities"] = map[string]bool{
 			"tools": m.Capabilities.Tools, "vision": m.Capabilities.Vision,
-			"embeddings": m.Capabilities.Embeddings,
+			"embeddings": m.Capabilities.Embeddings, "images": m.MakesImages(),
 		}
 		data = append(data, entry)
 	}
@@ -0,0 +1,200 @@
+package router
+
+import (
+	"context"
+	"encoding/base64"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"net/http"
+	"strconv"
+	"strings"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/comfy"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/router/proxy"
+)
+
+// /v1/images/generations, OpenAI's shape, made by ComfyUI: the request goes
+// through the same checks as any other — key, allowlist, limits, admission,
+// guardrails on its prompt — and only at the replica is it turned into a
+// workflow, queued and waited on, and its pictures sent back as base64.
+
+const imagesPath = "/v1/images/generations"
+
+// imageRequest is what a request for pictures asks. Beyond OpenAI's fields,
+// those a diffusion model has and DALL·E does not: seed, steps,
+// negative_prompt and guidance.
+type imageRequest struct {
+	Prompt         string   `json:"prompt"`
+	N              int      `json:"n"`
+	Size           string   `json:"size"`
+	ResponseFormat string   `json:"response_format"`
+	Seed           *int64   `json:"seed"`
+	Steps          int      `json:"steps"`
+	NegativePrompt string   `json:"negative_prompt"`
+	Guidance       *float64 `json:"guidance"`
+
+	width, height int
+}
+
+// The bounds of a request. A FLUX latent is 1/16 of the picture, so a side
+// that is not a multiple of 16 is not one it can make; the rest keep one
+// request from holding a card for the length of a film.
+const (
+	maxImages   = 4
+	minSide     = 256
+	maxSide     = 2048
+	maxSteps    = 100
+	defaultSide = 1024
+)
+
+func parseImageRequest(body []byte) (imageRequest, error) {
+	var in imageRequest
+	if err := json.Unmarshal(body, &in); err != nil {
+		return in, err
+	}
+	if strings.TrimSpace(in.Prompt) == "" {
+		return in, errors.New("prompt is required")
+	}
+	switch {
+	case in.N == 0:
+		in.N = 1
+	case in.N < 0 || in.N > maxImages:
+		return in, fmt.Errorf("n is between 1 and %d", maxImages)
+	}
+	switch in.ResponseFormat {
+	case "", "b64_json":
+	case "url":
+		// A URL would have to be served by somebody and kept for some time,
+		// and a picture made from a prompt is the prompt's owner's: it is
+		// not kept here.
+		return in, errors.New(`response_format "url" is not offered: the picture comes back as b64_json and is kept nowhere`)
+	default:
+		return in, fmt.Errorf("response_format %q is not b64_json", in.ResponseFormat)
+	}
+	in.width, in.height = defaultSide, defaultSide
+	if s := strings.TrimSpace(in.Size); s != "" && s != "auto" {
+		w, h, ok := strings.Cut(strings.ToLower(s), "x")
+		wi, err1 := strconv.Atoi(w)
+		hi, err2 := strconv.Atoi(h)
+		if !ok || err1 != nil || err2 != nil {
+			return in, fmt.Errorf("size %q is not WIDTHxHEIGHT", s)
+		}
+		in.width, in.height = wi, hi
+	}
+	for _, side := range []int{in.width, in.height} {
+		if side < minSide || side > maxSide || side%16 != 0 {
+			return in, fmt.Errorf("size %dx%d: each side is a multiple of 16 from %d to %d", in.width, in.height, minSide, maxSide)
+		}
+	}
+	if in.Steps < 0 || in.Steps > maxSteps {
+		return in, fmt.Errorf("steps is between 1 and %d", maxSteps)
+	}
+	if in.Guidance != nil && (*in.Guidance <= 0 || *in.Guidance > 30) {
+		return in, errors.New("guidance is above 0 and at most 30")
+	}
+	return in, nil
+}
+
+// imageRouteRefusal says why a request cannot be answered at the path it was
+// sent to, before it takes a place in a queue: a picture asked of a model
+// that writes text, or text of one that makes pictures. Empty when it can,
+// and for a model the catalog does not know, which a hand-configured
+// endpoint may serve as it likes.
+func (r *Router) imageRouteRefusal(path, model string, body []byte) string {
+	m, known := r.modelMap()[model]
+	if path == imagesPath {
+		if known && !m.MakesImages() {
+			return fmt.Sprintf("%s does not make pictures", m.Name)
+		}
+		if _, err := parseImageRequest(body); err != nil {
+			return err.Error()
+		}
+		return ""
+	}
+	if known && m.External != nil && m.External.API == catalog.APIComfyUI {
+		return fmt.Sprintf("%s makes pictures: ask for them at %s", m.Name, imagesPath)
+	}
+	return ""
+}
+
+// generateImages answers a request at a ComfyUI replica. It returns what the
+// proxy would: a failure of this replica that another may not have is
+// retryable, and nothing is written until the pictures are all in hand.
+func (r *Router) generateImages(ctx context.Context, w http.ResponseWriter, req *parsedRequest, ep endpoint.Info, body []byte) (proxy.Outcome, error) {
+	if req.Path != imagesPath {
+		return proxy.Outcome{}, fmt.Errorf("%s is a ComfyUI server: it answers %s only", ep.ID, imagesPath)
+	}
+	in, err := parseImageRequest(body)
+	if err != nil {
+		return proxy.Outcome{}, err
+	}
+	var upstream struct {
+		Model string `json:"model"`
+	}
+	_ = json.Unmarshal(body, &upstream)
+	tpl, err := comfy.Load(ep.Workflow)
+	if err != nil {
+		return proxy.Outcome{}, fmt.Errorf("%s: %w", ep.ID, err)
+	}
+	seed := comfy.RandomSeed()
+	if in.Seed != nil {
+		seed = *in.Seed
+	}
+	p := comfy.Params{
+		Checkpoint: upstream.Model, Prompt: in.Prompt, NegativePrompt: in.NegativePrompt,
+		Width: in.width, Height: in.height, N: in.N, Seed: seed, Steps: in.Steps,
+	}
+	if in.Guidance != nil {
+		p.Guidance = *in.Guidance
+	}
+	workflow, err := tpl.Fill(p)
+	if err != nil {
+		return proxy.Outcome{}, fmt.Errorf("%s: %w", ep.ID, err)
+	}
+
+	if t := r.cfg.Timeouts.Total; t > 0 {
+		var cancel context.CancelFunc
+		ctx, cancel = context.WithTimeout(ctx, t)
+		defer cancel()
+	}
+	start := time.Now()
+	images, err := r.comfy.Generate(ctx, ep.URL, ep.APIKey, workflow)
+	if err != nil {
+		var ce *comfy.Error
+		switch {
+		case errors.As(err, &ce):
+			return proxy.Outcome{Retryable: ce.Retryable}, err
+		case errors.Is(err, context.Canceled) && ctxCauseIsClient(ctx):
+			return proxy.Outcome{}, proxy.ErrClientGone
+		case errors.Is(err, context.DeadlineExceeded):
+			return proxy.Outcome{}, fmt.Errorf("%s made no picture within %s", ep.ID, r.cfg.Timeouts.Total)
+		}
+		return proxy.Outcome{}, err
+	}
+
+	data := make([]map[string]any, 0, len(images))
+	for _, im := range images {
+		data = append(data, map[string]any{"b64_json": base64.StdEncoding.EncodeToString(im.Data)})
+	}
+	out, err := json.Marshal(map[string]any{"created": time.Now().Unix(), "data": data})
+	if err != nil {
+		return proxy.Outcome{}, err
+	}
+	// The seed that made them, so a picture liked can be made again.
+	w.Header().Set("x-llmgw-seed", strconv.FormatInt(seed, 10))
+	w.Header().Set("Content-Type", "application/json")
+	w.WriteHeader(http.StatusOK)
+	_, _ = w.Write(out)
+	took := time.Since(start)
+	return proxy.Outcome{Wrote: true, Status: http.StatusOK, Bytes: int64(len(out)), TTFT: took, Decode: took}, nil
+}
+
+// ctxCauseIsClient reports whether a context ended because the client went
+// away, rather than because a deadline set here passed.
+func ctxCauseIsClient(ctx context.Context) bool {
+	return !errors.Is(context.Cause(ctx), context.DeadlineExceeded)
+}
@@ -0,0 +1,153 @@
+package router
+
+import (
+	"bytes"
+	"encoding/base64"
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"sync"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/comfy"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
+)
+
+// imageRouter has a ComfyUI model, flux, beside a text model, and returns
+// what the ComfyUI server was sent.
+func imageRouter(t *testing.T) (*Router, *map[string]any, *sync.Mutex) {
+	t.Helper()
+	var mu sync.Mutex
+	var queued map[string]any
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		switch r.URL.Path {
+		case "/metrics":
+			t.Error("ComfyUI was scraped for metrics")
+		case "/prompt":
+			var b struct {
+				Prompt map[string]any `json:"prompt"`
+			}
+			_ = json.NewDecoder(r.Body).Decode(&b)
+			mu.Lock()
+			queued = b.Prompt
+			mu.Unlock()
+			fmt.Fprint(w, `{"prompt_id":"p1"}`)
+		case "/history/p1":
+			fmt.Fprint(w, `{"p1":{"status":{"status_str":"success","completed":true},"outputs":{"7":{"images":[{"filename":"a.png","type":"temp"}]}}}}`)
+		case "/view":
+			w.Header().Set("Content-Type", "image/png")
+			fmt.Fprint(w, "png-bytes")
+		default:
+			http.NotFound(w, r)
+		}
+	}))
+	t.Cleanup(srv.Close)
+
+	cfg := baseConfig()
+	cfg.Models = []catalog.Model{catModel("chat", nil), {Name: "flux", Source: "external",
+		External: &catalog.External{URL: srv.URL, Model: "flux1-schnell-fp8.safetensors", API: catalog.APIComfyUI}}}
+	rt := newTestRouter(t, cfg)
+	rt.comfy = &comfy.Client{Poll: time.Millisecond}
+	rt.SetEndpoints(t.Context(), []endpoint.Info{
+		{ID: "external:flux", URL: srv.URL, Models: []string{"flux"}, Engine: endpoint.EngineComfyUI,
+			Pool: "external", Weight: 1, UpstreamModel: "flux1-schnell-fp8.safetensors"},
+	})
+	return rt, &queued, &mu
+}
+
+func postImages(t *testing.T, h http.Handler, path string, body any) *httptest.ResponseRecorder {
+	t.Helper()
+	b, _ := json.Marshal(body)
+	req := httptest.NewRequest(http.MethodPost, path, bytes.NewReader(b))
+	req.Header.Set("Content-Type", "application/json")
+	rec := httptest.NewRecorder()
+	h.ServeHTTP(rec, req)
+	return rec
+}
+
+func TestAPictureIsMadeByComfyUIAndComesBackAsBase64(t *testing.T) {
+	rt, queued, mu := imageRouter(t)
+	rec := postImages(t, rt.Handler(), imagesPath, map[string]any{
+		"model": "flux", "prompt": "a lighthouse at dusk", "size": "768x512", "seed": 7})
+	if rec.Code != http.StatusOK {
+		t.Fatalf("status %d: %s", rec.Code, rec.Body.String())
+	}
+	var out struct {
+		Data []struct {
+			B64 string `json:"b64_json"`
+		} `json:"data"`
+	}
+	if err := json.Unmarshal(rec.Body.Bytes(), &out); err != nil || len(out.Data) != 1 {
+		t.Fatalf("body %s: %v", rec.Body.String(), err)
+	}
+	if b, _ := base64.StdEncoding.DecodeString(out.Data[0].B64); string(b) != "png-bytes" {
+		t.Errorf("picture = %q", b)
+	}
+	if rec.Header().Get("x-llmgw-seed") != "7" {
+		t.Errorf("seed header = %q", rec.Header().Get("x-llmgw-seed"))
+	}
+	mu.Lock()
+	defer mu.Unlock()
+	wf, _ := json.Marshal(*queued)
+	for _, want := range []string{`"ckpt_name":"flux1-schnell-fp8.safetensors"`, `"text":"a lighthouse at dusk"`,
+		`"width":768`, `"height":512`, `"seed":7`} {
+		if !strings.Contains(string(wf), want) {
+			t.Errorf("%s not in the workflow sent: %s", want, wf)
+		}
+	}
+}
+
+func TestPicturesAndTextAreAskedForAtTheirOwnPaths(t *testing.T) {
+	rt, _, _ := imageRouter(t)
+	h := rt.Handler()
+	for name, c := range map[string]struct {
+		path string
+		body map[string]any
+		want string
+	}{
+		"text of flux":     {"/v1/chat/completions", map[string]any{"model": "flux", "messages": []any{}}, "ask for them at " + imagesPath},
+		"picture of chat":  {imagesPath, map[string]any{"model": "chat", "prompt": "x"}, "does not make pictures"},
+		"no prompt":        {imagesPath, map[string]any{"model": "flux"}, "prompt is required"},
+		"odd size":         {imagesPath, map[string]any{"model": "flux", "prompt": "x", "size": "1000x1000"}, "multiple of 16"},
+		"url wanted":       {imagesPath, map[string]any{"model": "flux", "prompt": "x", "response_format": "url"}, "b64_json"},
+		"too many at once": {imagesPath, map[string]any{"model": "flux", "prompt": "x", "n": 9}, "n is between"},
+	} {
+		rec := postImages(t, h, c.path, c.body)
+		if rec.Code != http.StatusBadRequest || !strings.Contains(rec.Body.String(), c.want) {
+			t.Errorf("%s: %d %s", name, rec.Code, rec.Body.String())
+		}
+	}
+}
+
+func TestAComfyUIModelSaysItMakesPictures(t *testing.T) {
+	rt, _, _ := imageRouter(t)
+	req := httptest.NewRequest(http.MethodGet, "/v1/models", nil)
+	rec := httptest.NewRecorder()
+	rt.Handler().ServeHTTP(rec, req)
+	if !strings.Contains(rec.Body.String(), `"images":true`) {
+		t.Errorf("/v1/models: %s", rec.Body.String())
+	}
+}
+
+func TestAComfyUIModelIsCheckedInTheCatalog(t *testing.T) {
+	ok := catalog.Model{Name: "f", Source: "external", External: &catalog.External{
+		URL: "http://10.0.0.147:8188", Model: "flux1-dev.safetensors", API: catalog.APIComfyUI, Workflow: "flux-dev"}}
+	if _, err := catalog.New([]catalog.Model{ok}); err != nil {
+		t.Fatalf("refused: %v", err)
+	}
+	for name, e := range map[string]catalog.External{
+		"unknown api":       {URL: "http://a:1", Model: "m", API: "a1111"},
+		"unknown workflow":  {URL: "http://a:1", Model: "m", API: catalog.APIComfyUI, Workflow: "sdxl"},
+		"workflow, openai":  {URL: "http://a:1", Model: "m", Workflow: "flux-dev"},
+		"comfyui from peer": {URL: "http://a:1", Model: "m", API: catalog.APIComfyUI, Peer: "other"},
+	} {
+		e := e
+		if _, err := catalog.New([]catalog.Model{{Name: "f", Source: "external", External: &e}}); err == nil {
+			t.Errorf("%s: accepted", name)
+		}
+	}
+}
@@ -10,6 +10,7 @@ import (
 	"fmt"
 	"net/http"
 	"net/url"
+	"sort"
 	"strconv"
 	"strings"
 	"time"
@@ -225,7 +226,14 @@ func (r *Router) retrieve(ctx context.Context, w http.ResponseWriter, req *parse
 	req.UpstreamBody = body
 	req.PromptTokens = len(body) / 4
 	req.knowledge = true
-	req.kbOrg = e.OrgID
+	// The collections searched, which is who else could have had these
+	// passages added: the prefix cache is shared among exactly them.
+	ids := make([]string, len(colls))
+	for i, c := range colls {
+		ids[i] = c.ID
+	}
+	sort.Strings(ids)
+	req.kbSalt = strings.Join(ids, ",")
 	// Set now: the proxy writes the headers when the answer starts.
 	w.Header().Set(sourcesHeader, strings.Join(sources, ","))
 	result := "found"
@@ -268,6 +276,10 @@ func splitList(v string) []string {
 // ordinary path, so the caller's limits, budget and guardrails apply to it
 // and it is counted to them.
 func (r *Router) embedFor(ctx context.Context, p auth.Principal, model, text string) ([]float32, error) {
+	// A question, not a document: what the model wants before a query.
+	if _, m := r.resolve(model); m.QueryPrefix != "" {
+		text = m.QueryPrefix + text
+	}
 	body, _ := json.Marshal(map[string]any{"model": model, "input": []string{text}})
 	hreq, err := http.NewRequestWithContext(auth.WithPrincipal(ctx, p), http.MethodPost, "/v1/embeddings", bytes.NewReader(body))
 	if err != nil {
@@ -429,14 +441,16 @@ func (r *Router) kbDefaults(ctx context.Context, p auth.Principal, project strin
 	return names
 }

-// withCacheSalt sets vLLM's cache_salt to one derived from an organization:
-// the same for its requests, so they still share a cache among themselves.
-func withCacheSalt(body []byte, org string) []byte {
+// withCacheSalt sets vLLM's cache_salt to one derived from key — the
+// collections a request's passages came from — so requests that can see the
+// same passages still share a cache, and nobody else's do: not another
+// organization's, nor another team's of the same one.
+func withCacheSalt(body []byte, key string) []byte {
 	var top map[string]json.RawMessage
 	if json.Unmarshal(body, &top) != nil {
 		return body
 	}
-	sum := sha256.Sum256([]byte("llmgw-kb\x00" + org))
+	sum := sha256.Sum256([]byte("llmgw-kb\x00" + key))
 	top["cache_salt"], _ = json.Marshal(hex.EncodeToString(sum[:16]))
 	out, err := json.Marshal(top)
 	if err != nil {
@@ -9,7 +9,9 @@ import (
 	"strings"
 	"testing"

+	"gitea.mynodes.xyz/ilker/llmgw/internal/auth"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/kb/kbsync"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
 )
@@ -52,9 +54,9 @@ func TestPassagesJoinTheLeadingSystemMessage(t *testing.T) {
 	}
 }

-func TestTheCacheSaltIsAnOrganizations(t *testing.T) {
+func TestTheCacheSaltFollowsTheCollections(t *testing.T) {
 	body := []byte(`{"model":"m","messages":[],"temperature":0}`)
-	a1, a2, b := withCacheSalt(body, "org-a"), withCacheSalt(body, "org-a"), withCacheSalt(body, "org-b")
+	a1, a2, b := withCacheSalt(body, "kb_a,kb_b"), withCacheSalt(body, "kb_a,kb_b"), withCacheSalt(body, "kb_a")
 	var x, y struct {
 		Salt string  `json:"cache_salt"`
 		T    float64 `json:"temperature"`
@@ -202,7 +204,7 @@ func mustID(t *testing.T, s *store.DB, name string) string {
 	return ""
 }

-func TestAVLLMReplicaGetsTheOrganizationsSaltOnlyWithPassages(t *testing.T) {
+func TestAVLLMReplicaGetsASaltOnlyWithPassages(t *testing.T) {
 	dsn := os.Getenv("LLMGW_TEST_POSTGRES")
 	if dsn == "" {
 		t.Skip("set LLMGW_TEST_POSTGRES to a pgvector database to run this")
@@ -253,3 +255,25 @@ func TestAVLLMReplicaGetsTheOrganizationsSaltOnlyWithPassages(t *testing.T) {
 		t.Errorf("a salt without passages:\n%s", got)
 	}
 }
+
+func TestAQuestionIsEmbeddedWithTheModelsQueryPrefix(t *testing.T) {
+	// Qwen3-Embedding and its like are trained with an instruction before
+	// the query, never before the documents: the documents come through
+	// /v1/embeddings as they are; the question gets the prefix here.
+	e := newFakeEngine(t, "emb")
+	cfg := baseConfig(config.Endpoint{ID: "a", URL: e.srv.URL, Models: []string{"emb"}})
+	m := catModel("emb", nil)
+	m.Capabilities.Embeddings = true
+	m.QueryPrefix = "Instruct: retrieve passages\nQuery: "
+	cfg.Models = []catalog.Model{m}
+	rt := newTestRouter(t, cfg)
+	// The test engine answers with the input's length in bytes and words.
+	v, err := rt.embedFor(t.Context(), auth.Principal{}, "emb", "kaç gün izin var")
+	if err != nil {
+		t.Fatal(err)
+	}
+	want := "Instruct: retrieve passages\nQuery: kaç gün izin var"
+	if int(v[0]) != len(want) || int(v[1]) != len(strings.Fields(want)) {
+		t.Errorf("embedded %v, not %q", v, want)
+	}
+}
@@ -29,6 +29,7 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/accounting"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/auth"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/comfy"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
@@ -53,7 +54,9 @@ type Router struct {
 	reg    *endpoint.Registry
 	poller *scrape.Poller
 	px     *proxy.Proxy
-	m      *metrics
+	// comfy asks ComfyUI replicas for pictures.
+	comfy *comfy.Client
+	m     *metrics
 	// adm is nil when admission control is disabled.
 	adm *admission.Controller

@@ -177,6 +180,10 @@ type Router struct {
 	// collections for kbDefaultTTL.
 	kbDefMu    sync.Mutex
 	kbDefCache map[string]kbDefaultEntry
+
+	// settingsCheck and settingsLocal mount the portal's settings page.
+	settingsCheck func([]byte) error
+	settingsLocal []string
 }

 // SetClusterView gives the router the data the /admin page renders.
@@ -200,6 +207,13 @@ func (r *Router) SetGuard(g guard.Guard) { r.guard = g }
 // SetGuardLibrary is the policies the guard was built from, for the pages.
 func (r *Router) SetGuardLibrary(c func() guard.Config) { r.guardLibrary = c }

+// SetSettingsCheck enables the portal's settings page: check says whether a
+// document would load as the cluster's settings beside this machine's own
+// server.yaml, and local are the keys that file keeps.
+func (r *Router) SetSettingsCheck(check func([]byte) error, local []string) {
+	r.settingsCheck, r.settingsLocal = check, local
+}
+
 // SetStopped refuses, or again serves, every request for a model.
 func (r *Router) SetStopped(stopped bool) { r.stopped.Store(stopped) }

@@ -300,6 +314,7 @@ func New(cfg config.Server, log *slog.Logger) (*Router, error) {
 		kv:        kvindex.New(),
 		kvClient:  newKVClient(),
 		m:         metrics,
+		comfy:     &comfy.Client{HTTP: &http.Client{Transport: &http.Transport{DialContext: (&net.Dialer{Timeout: cfg.Router.Timeouts.Dial}).DialContext}}},
 		px: proxy.New(proxy.Config{
 			DialTimeout:           cfg.Router.Timeouts.Dial,
 			ResponseHeaderTimeout: cfg.Router.Timeouts.ResponseHeader,
@@ -476,6 +491,7 @@ func toInfos(eps []config.Endpoint) []endpoint.Info {
 			Pool:     e.Pool,
 			Weight:   e.Weight,
 			Priority: e.Priority,
+			Workflow: e.Workflow,
 		})
 	}
 	return out
@@ -702,7 +718,7 @@ func (r *Router) storeModels(models []catalog.Model) {
 		// is applied to the request on the way there.
 		for i, a := range entry.Adapters {
 			m[a.Name] = entry
-			ads[a.Name] = adapterRef{base: entry.Name, index: i, scale: a.EffectiveScale(),
+			ads[a.Name] = adapterRef{base: entry.Name, owner: a.Owner, index: i, scale: a.EffectiveScale(),
 				limit: entry.MaxLoadedAdapters, path: engines.RuntimeLoRAPath(a.Name, a.Source)}
 		}
 	}
@@ -724,7 +740,9 @@ func (r *Router) storeModels(models []catalog.Model) {

 // adapterRef is where an adapter runs and how it is applied.
 type adapterRef struct {
-	base  string
+	base string
+	// owner is who sees it (catalog.Adapter.Owner); empty is everyone.
+	owner string
 	index int
 	scale float64
 	// limit is the model's max_loaded_adapters: above zero, the adapter
@@ -746,6 +764,33 @@ func (r *Router) adapter(name string) (adapterRef, bool) {
 	return adapterRef{}, false
 }

+// adapterHidden reports whether name is an adapter its owner keeps from this
+// caller: one outside the part of the tree it belongs to, or one with no
+// identity at all. Such a caller is told there is no such model.
+func (r *Router) adapterHidden(p auth.Principal, authed bool, name string) bool {
+	ad, ok := r.adapter(name)
+	if !ok || ad.owner == "" {
+		return false
+	}
+	if !authed {
+		return true
+	}
+	scope, id, _ := strings.Cut(ad.owner, ":")
+	switch scope {
+	case "org":
+		return p.OrgID != id
+	case "team":
+		return p.TeamID != id
+	case "project":
+		return p.ProjectID != id
+	case "user":
+		return p.UserID != id
+	case "key":
+		return p.KeyID != id
+	}
+	return true
+}
+
 // LastUsed reports when each endpoint was last sent a request, satisfying
 // control.UsageSource so the control plane can tell a busy replica from one
 // holding its cards out of habit.
@@ -1079,13 +1124,13 @@ func (r *Router) dispatchOne(ctx context.Context, w http.ResponseWriter, req *pa
 		if isAdapter && !onDemand && dec.Endpoint.Engine != "vllm" {
 			body = withAdapter(body, model, ad.index, ad.scale)
 		}
-		// Passages of one organization's documents never share cached
-		// blocks with another's: a prefix found warm would say what another
-		// tenant asked about, by how fast its answer started. vLLM keys its
-		// prefix cache by this salt; other engines are not sent a field
-		// they may refuse.
-		if req.kbOrg != "" && dec.Endpoint.Engine == "vllm" && req.ContentType == "" {
-			body = withCacheSalt(body, req.kbOrg)
+		// Passages share cached blocks only with requests that could have
+		// been given the same ones: a prefix found warm would otherwise say
+		// what another tenant asked about, by how fast its answer started.
+		// vLLM keys its prefix cache by this salt; other engines are not
+		// sent a field they may refuse.
+		if req.kbSalt != "" && dec.Endpoint.Engine == "vllm" && req.ContentType == "" {
+			body = withCacheSalt(body, req.kbSalt)
 		}
 		// What computes the answer, said before the proxy writes the headers.
 		// A model can have replicas on cards and on CPUs at once, and the two
@@ -1122,7 +1167,13 @@ func (r *Router) dispatchOne(ctx context.Context, w http.ResponseWriter, req *pa
 		}
 		// A filter per attempt, for the model this attempt is served by.
 		filter := r.outputFilter(uctx, req, model, &preq)
-		out, perr := r.px.Do(uctx, w, preq)
+		var out proxy.Outcome
+		var perr error
+		if dec.Endpoint.Engine == endpoint.EngineComfyUI {
+			out, perr = r.generateImages(uctx, w, req, dec.Endpoint.Info, body)
+		} else {
+			out, perr = r.px.Do(uctx, w, preq)
+		}
 		releaseAdapter()
 		uspan.SetAttributes(attribute.Int("http.response.status_code", out.Status))
 		endSpan(uspan, perr)
@@ -783,3 +783,28 @@ func TestAStoppedServiceAnswers503AndStillListsModels(t *testing.T) {
 		t.Errorf("started again, still refused: %s", rec.Body.String())
 	}
 }
+
+func TestABatchOfEmbeddingsIsCheckedTextByText(t *testing.T) {
+	// llmgw-kb sends many passages in one request; each is embedded on its
+	// own, so their sum over the context is no reason to refuse them. One
+	// passage over the context still is.
+	e := newFakeEngine(t, "emb")
+	cfg := baseConfig(config.Endpoint{ID: "a", URL: e.srv.URL, Models: []string{"emb"}})
+	m := catModel("emb", nil)
+	m.ContextLength = 2048
+	cfg.Models = []catalog.Model{m}
+	rt := newTestRouter(t, cfg)
+	send := func(input []string) *httptest.ResponseRecorder {
+		b, _ := json.Marshal(map[string]any{"model": "emb", "input": input})
+		rec := httptest.NewRecorder()
+		rt.Handler().ServeHTTP(rec, httptest.NewRequest(http.MethodPost, "/v1/embeddings", bytes.NewReader(b)))
+		return rec
+	}
+	passage := strings.Repeat("x", 6000) // about 1500 tokens
+	if rec := send([]string{passage, passage, passage, passage}); rec.Code == http.StatusBadRequest {
+		t.Errorf("four passages that each fit were refused: %s", rec.Body.String())
+	}
+	if rec := send([]string{"short", strings.Repeat("x", 40_000)}); rec.Code != http.StatusBadRequest {
+		t.Errorf("a passage over the context was let through: %d", rec.Code)
+	}
+}
@@ -96,8 +96,8 @@ func (p *Poller) Sync(ctx context.Context) {

 	want := make(map[string]endpoint.Info, len(infos))
 	for _, in := range infos {
-		if in.Engine == endpoint.EngineExternal {
-			continue // a provider elsewhere publishes no /metrics
+		if !endpoint.Scraped(in.Engine) {
+			continue // a provider elsewhere, or ComfyUI, publishes no /metrics
 		}
 		want[in.ID] = in
 	}
@@ -0,0 +1,30 @@
+package store
+
+import (
+	"context"
+	"database/sql"
+	"errors"
+	"time"
+)
+
+// PutClusterState keeps a value under a key, for machines that read the
+// store rather than a control plane; it reports whether it changed.
+func (s *DB) PutClusterState(ctx context.Context, key, value string) (bool, error) {
+	if old, err := s.ClusterState(ctx, key); err == nil && old == value {
+		return false, nil
+	}
+	_, err := s.db.ExecContext(ctx, `INSERT INTO cluster_state (key, value, updated_at) VALUES (?, ?, ?)
+		ON CONFLICT (key) DO UPDATE SET value = excluded.value, updated_at = excluded.updated_at`,
+		key, value, millis(time.Now()))
+	return err == nil, err
+}
+
+// ClusterState reads a value; ErrNotFound when there is none.
+func (s *DB) ClusterState(ctx context.Context, key string) (string, error) {
+	var v string
+	err := s.db.QueryRowContext(ctx, `SELECT value FROM cluster_state WHERE key = ?`, key).Scan(&v)
+	if errors.Is(err, sql.ErrNoRows) {
+		return "", ErrNotFound
+	}
+	return v, err
+}
@@ -73,6 +73,9 @@ var copyOrder = []string{
 	"eval_runs",
 	"eval_cases",
 	"federation_peers",
+	// Before their artifacts, which point at them.
+	"train_jobs",
+	"train_artifacts",
 }

 // notCopied are tables deliberately left behind.
@@ -83,7 +86,8 @@ var copyOrder = []string{
 // the exact moment somebody is watching a migration and wondering why nothing
 // is being placed.
 var notCopied = map[string]string{
-	"leadership": "a lease on a running process, meaningless in a new store",
+	"leadership":    "a lease on a running process, meaningless in a new store",
+	"cluster_state": "worked out again by the leader from the settings and the machines, within minutes",
 }

 // CopyResult is how many rows each table received.
@@ -29,9 +29,13 @@ import (
 // ErrKnowledgeOff is a store that cannot hold knowledge bases.
 var ErrKnowledgeOff = errors.New("knowledge bases are off")

-// MaxEmbeddingDims is the most dimensions pgvector can index (HNSW on vector).
+// MaxEmbeddingDims is the most dimensions pgvector can index with HNSW: on
+// vector; halfvec takes up to MaxHalfvecDims.
 const MaxEmbeddingDims = 2000

+// MaxHalfvecDims is the most dimensions an HNSW index on halfvec takes.
+const MaxHalfvecDims = 4000
+
 // KBPerm is what a grant allows.
 type KBPerm string

@@ -50,8 +54,11 @@ type KBCollection struct {
 	Name       string
 	EmbedModel string
 	Dims       int
-	CreatedAt  time.Time
-	CreatedBy  string
+	// VecType is how its vectors are indexed: vector, or halfvec where
+	// pgvector has it (0.7 and later), at half the index's memory.
+	VecType   string
+	CreatedAt time.Time
+	CreatedBy string
 }

 // KBGrant lets a scope, and everything below it, read or write a collection.
@@ -133,11 +140,16 @@ func (s *DB) ensureKnowledge(ctx context.Context) error {
 				"as postgres, run CREATE EXTENSION vector in it (%v)", ErrKnowledgeOff, err)
 		}
 	}
+	var version string
+	if err := s.db.QueryRowContext(ctx, `SELECT extversion FROM pg_extension WHERE extname = 'vector'`).Scan(&version); err == nil {
+		s.kbHalfvec = versionAtLeast(version, 0, 7)
+	}
 	var exists bool
 	if err := s.db.QueryRowContext(ctx, `SELECT to_regclass('kb_chunks') IS NOT NULL`).Scan(&exists); err != nil {
 		return fmt.Errorf("%w: %v", ErrKnowledgeOff, err)
 	}
 	if exists {
+		s.kbBackfillIndexes(ctx)
 		return nil
 	}
 	err := s.inTx(ctx, func(c conn) error {
@@ -197,6 +209,52 @@ func kbVectorIndex(collection string) string {
 	return "kb_vec_" + strings.TrimPrefix(collection, "kb_")
 }

+// kbTextIndex is the name of a collection's full-text index.
+func kbTextIndex(collection string) string { return "kb_tsv_" + strings.TrimPrefix(collection, "kb_") }
+
+// cast is the type a collection's vectors are compared as, which its index
+// is built on: the query must say the same for the planner to use it.
+func (c KBCollection) cast() string {
+	t := c.VecType
+	if t != "halfvec" {
+		t = "vector"
+	}
+	return "::" + t + "(" + strconv.Itoa(c.Dims) + ")"
+}
+
+// versionAtLeast compares an extension's "major.minor.patch".
+func versionAtLeast(v string, major, minor int) bool {
+	parts := strings.SplitN(v, ".", 3)
+	if len(parts) < 2 {
+		return false
+	}
+	ma, err1 := strconv.Atoi(parts[0])
+	mi, err2 := strconv.Atoi(parts[1])
+	if err1 != nil || err2 != nil {
+		return false
+	}
+	return ma > major || ma == major && mi >= minor
+}
+
+// kbBackfillIndexes gives collections made before their full-text index
+// existed one. Best effort, at open: a search still works without it.
+func (s *DB) kbBackfillIndexes(ctx context.Context) {
+	cs, err := queryAll(ctx, s.db, `SELECT id, org_id FROM kb_collections`, nil, func(r *sql.Rows) ([2]string, error) {
+		var x [2]string
+		return x, r.Scan(&x[0], &x[1])
+	})
+	if err != nil {
+		return
+	}
+	for _, c := range cs {
+		if !kbIDPattern.MatchString(c[0]) {
+			continue
+		}
+		_, _ = s.db.ExecContext(ctx, `CREATE INDEX IF NOT EXISTS `+kbTextIndex(c[0])+` ON `+kbPartition(c[1])+
+			` USING gin (tsv) WHERE collection = `+sqlLiteral(c[0]))
+	}
+}
+
 func sqlLiteral(v string) string { return "'" + strings.ReplaceAll(v, "'", "''") + "'" }

 // kbScope sets, for this transaction, which rows of kb_chunks it may touch.
@@ -263,14 +321,22 @@ func (s *DB) CreateCollection(ctx context.Context, c KBCollection) (KBCollection
 		return KBCollection{}, errors.New("a collection needs a name")
 	case c.EmbedModel == "":
 		return KBCollection{}, errors.New("a collection needs the embedding model its documents are embedded with")
-	case c.Dims < 1 || c.Dims > MaxEmbeddingDims:
-		return KBCollection{}, fmt.Errorf("%d dimensions: an indexed embedding has 1 to %d", c.Dims, MaxEmbeddingDims)
+	case c.Dims < 1 || c.Dims > MaxHalfvecDims || c.Dims > MaxEmbeddingDims && !s.kbHalfvec:
+		limit := MaxEmbeddingDims
+		if s.kbHalfvec {
+			limit = MaxHalfvecDims
+		}
+		return KBCollection{}, fmt.Errorf("%d dimensions: an indexed embedding has 1 to %d", c.Dims, limit)
 	case c.OwnerScope == ScopeUser && c.OrgID == "":
 		return KBCollection{}, errors.New("a person's collection names the organization it belongs to")
 	case c.OwnerScope == ScopeKey:
 		return KBCollection{}, errors.New("a key cannot own a collection; its project or its person can")
 	}
 	c.ID = NewID("kb_")
+	c.VecType = "vector"
+	if s.kbHalfvec {
+		c.VecType = "halfvec"
+	}
 	if c.CreatedAt.IsZero() {
 		c.CreatedAt = time.Now()
 	}
@@ -292,9 +358,9 @@ func (s *DB) CreateCollection(ctx context.Context, c KBCollection) (KBCollection
 			return fmt.Errorf("%w: collection %q", ErrDuplicateName, c.Name)
 		}
 		if _, err := tx.ExecContext(ctx, `INSERT INTO kb_collections
-			(id, org_id, owner_scope, owner_id, name, embed_model, dims, created_at, created_by)
-			VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`,
-			c.ID, c.OrgID, string(c.OwnerScope), c.OwnerID, c.Name, c.EmbedModel, c.Dims,
+			(id, org_id, owner_scope, owner_id, name, embed_model, dims, vec_type, created_at, created_by)
+			VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
+			c.ID, c.OrgID, string(c.OwnerScope), c.OwnerID, c.Name, c.EmbedModel, c.Dims, c.VecType,
 			millis(c.CreatedAt), c.CreatedBy); err != nil {
 			return err
 		}
@@ -303,12 +369,18 @@ func (s *DB) CreateCollection(ctx context.Context, c KBCollection) (KBCollection
 			` PARTITION OF kb_chunks FOR VALUES IN (`+sqlLiteral(c.OrgID)+`)`); err != nil {
 			return err
 		}
-		// A partial index per collection, on its organization's partition
+		// Partial indexes per collection, on its organization's partition
 		// only: the dimension is the collection's, and a search, which
-		// names one collection at a time, uses exactly this one.
-		_, err = tx.ExecContext(ctx, `CREATE INDEX `+kbVectorIndex(c.ID)+` ON `+part+
-			` USING hnsw ((embedding::vector(`+strconv.Itoa(c.Dims)+`)) vector_cosine_ops)`+
-			` WHERE collection = `+sqlLiteral(c.ID))
+		// names one collection at a time, uses exactly these — a text
+		// search over one collection does not walk its whole
+		// organization's index.
+		if _, err := tx.ExecContext(ctx, `CREATE INDEX `+kbVectorIndex(c.ID)+` ON `+part+
+			` USING hnsw ((embedding`+c.cast()+`) `+c.VecType+`_cosine_ops)`+
+			` WHERE collection = `+sqlLiteral(c.ID)); err != nil {
+			return err
+		}
+		_, err = tx.ExecContext(ctx, `CREATE INDEX `+kbTextIndex(c.ID)+` ON `+part+
+			` USING gin (tsv) WHERE collection = `+sqlLiteral(c.ID))
 		return err
 	})
 	if err != nil {
@@ -317,13 +389,13 @@ func (s *DB) CreateCollection(ctx context.Context, c KBCollection) (KBCollection
 	return c, nil
 }

-const kbCollectionCols = `id, org_id, owner_scope, owner_id, name, embed_model, dims, created_at, created_by`
+const kbCollectionCols = `id, org_id, owner_scope, owner_id, name, embed_model, dims, vec_type, created_at, created_by`

 func scanCollection(sc scanner) (KBCollection, error) {
 	var c KBCollection
 	var scope string
 	var at int64
-	err := sc.Scan(&c.ID, &c.OrgID, &scope, &c.OwnerID, &c.Name, &c.EmbedModel, &c.Dims, &at, &c.CreatedBy)
+	err := sc.Scan(&c.ID, &c.OrgID, &scope, &c.OwnerID, &c.Name, &c.EmbedModel, &c.Dims, &c.VecType, &at, &c.CreatedBy)
 	c.OwnerScope, c.CreatedAt = Scope(scope), fromMillis(at)
 	return c, err
 }
@@ -377,7 +449,10 @@ func (s *DB) DeleteCollection(ctx context.Context, id string) error {
 		if _, err := c.ExecContext(ctx, `DELETE FROM kb_defaults WHERE collections = ''`); err != nil {
 			return err
 		}
-		_, err = c.ExecContext(ctx, `DROP INDEX IF EXISTS `+kbVectorIndex(id))
+		if _, err := c.ExecContext(ctx, `DROP INDEX IF EXISTS `+kbVectorIndex(id)); err != nil {
+			return err
+		}
+		_, err = c.ExecContext(ctx, `DROP INDEX IF EXISTS `+kbTextIndex(id))
 		return err
 	})
 }
@@ -729,6 +804,14 @@ func (s *DB) SearchChunks(ctx context.Context, q KBSearch) ([]KBHit, error) {
 		if err := kbScope(ctx, tx, q.OrgID, ids); err != nil {
 			return err
 		}
+		// pgvector's HNSW search returns at most ef_search rows (40 by
+		// default), whatever LIMIT asks: more candidates need it raised.
+		if len(q.Embeddings) > 0 {
+			ef := min(max(candidates(q.K), 40), 1000)
+			if _, err := tx.ExecContext(ctx, `SET LOCAL hnsw.ef_search = `+strconv.Itoa(ef)); err != nil {
+				return err
+			}
+		}
 		for _, c := range q.Collections {
 			// The literal collection and cast are what let the planner use
 			// that collection's partial index.
@@ -739,7 +822,7 @@ func (s *DB) SearchChunks(ctx context.Context, q KBSearch) ([]KBHit, error) {
 				if len(v) != c.Dims {
 					return fmt.Errorf("the query has %d dimensions; collection %s takes %d", len(v), c.Name, c.Dims)
 				}
-				cast := `::vector(` + strconv.Itoa(c.Dims) + `)`
+				cast := c.cast()
 				sim = `1 - (k.embedding` + cast + ` <=> ?` + cast + `)`
 				args = append(args, vectorText(v))
 				if err := query(tx, false, `SELECT k.collection, k.doc_id, d.name, k.ord, k.body, `+sim+`, 0::float8
@@ -753,15 +836,20 @@ func (s *DB) SearchChunks(ctx context.Context, q KBSearch) ([]KBHit, error) {
 			if tsq != "" {
 				// The text search's finds are measured against the question's
 				// meaning too, so the two are compared on one scale.
+				// Candidates by the cheap rank first, from the collection's
+				// own index; the share of words held and the similarity are
+				// worked out for those alone, not for every chunk that holds
+				// one word of the question.
 				targs := append([]any{}, args...)
 				for _, t := range terms {
 					targs = append(targs, t)
 				}
-				targs = append(targs, tsq, q.OrgID, candidates(q.K))
+				targs = append(targs, q.OrgID, tsq, tsq, candidates(q.K))
 				if err := query(tx, true, `SELECT k.collection, k.doc_id, d.name, k.ord, k.body, `+sim+`, `+coverage+` AS coverage
-					FROM kb_chunks k JOIN kb_documents d ON d.id = k.doc_id, to_tsquery('simple', ?) tq
-					WHERE k.org_id = ? AND k.collection = `+lit+` AND k.tsv @@ tq
-					ORDER BY coverage DESC, ts_rank(k.tsv, tq) DESC LIMIT ?`, targs...); err != nil {
+					FROM (SELECT * FROM kb_chunks k
+					      WHERE k.org_id = ? AND k.collection = `+lit+` AND k.tsv @@ to_tsquery('simple', ?)
+					      ORDER BY ts_rank(k.tsv, to_tsquery('simple', ?)) DESC LIMIT ?) k
+					JOIN kb_documents d ON d.id = k.doc_id`, targs...); err != nil {
 					return err
 				}
 			}
@@ -887,8 +975,10 @@ func (s *DB) kbForgetScope(ctx context.Context, c conn, scope Scope, id string)
 			return err
 		}
 		if kbIDPattern.MatchString(cid) {
-			if _, err := c.ExecContext(ctx, `DROP INDEX IF EXISTS `+kbVectorIndex(cid)); err != nil {
-				return err
+			for _, idx := range []string{kbVectorIndex(cid), kbTextIndex(cid)} {
+				if _, err := c.ExecContext(ctx, `DROP INDEX IF EXISTS `+idx); err != nil {
+					return err
+				}
 			}
 		}
 	}
@@ -1026,3 +1116,32 @@ func (s *DB) KBDefaultFor(ctx context.Context, e Effective) ([]string, error) {
 	}
 	return nil, nil
 }
+
+// CollectionChunks lists a collection's chunks in document order, for
+// building training data from them (BACKLOG F4). At most limit.
+func (s *DB) CollectionChunks(ctx context.Context, c KBCollection, limit int) ([]KBHit, error) {
+	if err := s.KnowledgeAvailable(); err != nil {
+		return nil, err
+	}
+	if !kbIDPattern.MatchString(c.ID) {
+		return nil, ErrNotFound
+	}
+	var out []KBHit
+	err := s.inTx(ctx, func(tx conn) error {
+		if err := kbScope(ctx, tx, c.OrgID, []string{c.ID}); err != nil {
+			return err
+		}
+		var err error
+		out, err = queryAll(ctx, tx, `SELECT k.collection, k.doc_id, d.name, k.ord, k.body
+			FROM kb_chunks k JOIN kb_documents d ON d.id = k.doc_id
+			WHERE k.org_id = ? AND k.collection = `+sqlLiteral(c.ID)+`
+			ORDER BY d.name, k.ord LIMIT ?`, []any{c.OrgID, limit},
+			func(r *sql.Rows) (KBHit, error) {
+				var h KBHit
+				err := r.Scan(&h.Collection, &h.DocID, &h.Document, &h.Ord, &h.Body)
+				return h, err
+			})
+		return err
+	})
+	return out, err
+}
@@ -3,6 +3,7 @@ package store
 import (
 	"context"
 	"errors"
+	"fmt"
 	"math"
 	"strings"
 	"testing"
@@ -320,10 +321,85 @@ func TestPostgresScopesWithCollectionsAreNotDeleted(t *testing.T) {
 	if err := s.DeleteProject(ctx, "proj2"); err != nil {
 		t.Errorf("deleting the project once its collection is gone: %v", err)
 	}
+	for _, idx := range []string{kbVectorIndex(c.ID), kbTextIndex(c.ID)} {
+		if indexExists(t, s, idx) {
+			t.Errorf("the collection's index %s is left behind", idx)
+		}
+	}
+}
+
+func indexExists(t *testing.T, s *DB, name string) bool {
+	t.Helper()
 	var n int
-	if err := s.raw.QueryRowContext(ctx, `SELECT COUNT(*) FROM pg_indexes WHERE indexname = $1`,
-		kbVectorIndex(c.ID)).Scan(&n); err != nil || n != 0 {
-		t.Errorf("the collection's index is left behind (%d, %v)", n, err)
+	if err := s.raw.QueryRowContext(context.Background(), `SELECT COUNT(*) FROM pg_indexes WHERE indexname = $1`,
+		name).Scan(&n); err != nil {
+		t.Fatal(err)
+	}
+	return n == 1
+}
+
+func TestPostgresCollectionsHaveTheirOwnIndexes(t *testing.T) {
+	ctx := context.Background()
+	s := openKB(t)
+	kbTwoOrgs(t, s)
+	c := mustCollection(t, s, ScopeProject, "proj1", "", "docs")
+	if !indexExists(t, s, kbVectorIndex(c.ID)) || !indexExists(t, s, kbTextIndex(c.ID)) {
+		t.Fatal("a new collection lacks its vector or text index")
+	}
+	// The test image's pgvector has halfvec; a collection there uses it, and
+	// is searched as it is indexed.
+	if !s.kbHalfvec || c.VecType != "halfvec" {
+		t.Errorf("halfvec %v, the collection's type %q", s.kbHalfvec, c.VecType)
+	}
+	if got, err := s.Collection(ctx, c.ID); err != nil || got.VecType != c.VecType {
+		t.Errorf("read back %q, %v", got.VecType, err)
+	}
+	var def string
+	if err := s.raw.QueryRowContext(ctx, `SELECT indexdef FROM pg_indexes WHERE indexname = $1`, kbVectorIndex(c.ID)).Scan(&def); err != nil ||
+		!strings.Contains(def, "halfvec_cosine_ops") {
+		t.Errorf("the vector index is %q (%v)", def, err)
+	}
+	if _, err := s.CreateCollection(ctx, KBCollection{OwnerScope: ScopeProject, OwnerID: "proj1",
+		Name: "wide", EmbedModel: "emb", Dims: 3072}); err != nil {
+		t.Errorf("3072 dimensions, which halfvec indexes: %v", err)
+	}
+	if _, err := s.CreateCollection(ctx, KBCollection{OwnerScope: ScopeProject, OwnerID: "proj1",
+		Name: "too-wide", EmbedModel: "emb", Dims: MaxHalfvecDims + 1}); err == nil {
+		t.Error("more dimensions than any index takes were accepted")
+	}
+
+	// A collection made before text indexes were gets one when the store
+	// opens.
+	if _, err := s.raw.ExecContext(ctx, `DROP INDEX `+kbTextIndex(c.ID)); err != nil {
+		t.Fatal(err)
+	}
+	s.kbBackfillIndexes(ctx)
+	if !indexExists(t, s, kbTextIndex(c.ID)) {
+		t.Error("the text index was not put back")
+	}
+
+	// Asking for more than HNSW's default 40 is answered in full. (A table
+	// this small is read without its index: with one, 40 is all HNSW gives
+	// unless ef_search is raised, as an EXPLAIN on 20000 rows showed.)
+	var chunks []KBChunk
+	for i := range 120 {
+		chunks = append(chunks, KBChunk{Body: fmt.Sprintf("parça %d izin", i), Embedding: unit(i % 3)})
+	}
+	if _, _, err := s.PutDocument(ctx, c, KBDocument{Name: "big.md", Digest: "big"}, chunks); err != nil {
+		t.Fatal(err)
+	}
+	hits, err := s.SearchChunks(ctx, KBSearch{OrgID: "org1", Collections: []KBCollection{c},
+		Embeddings: map[string][]float32{"emb": unit(0)}, K: 50})
+	if err != nil || len(hits) != 50 {
+		t.Errorf("K=50 gave %d hits, %v", len(hits), err)
+	}
+}
+
+func TestVersionAtLeast(t *testing.T) {
+	for v, want := range map[string]bool{"0.6.0": false, "0.7.0": true, "0.8.7": true, "1.0": true, "0": false, "x.y": false} {
+		if got := versionAtLeast(v, 0, 7); got != want {
+			t.Errorf("%s: %v", v, got)
+		}
 	}
 }

@@ -0,0 +1,42 @@
+-- Training jobs (BACKLOG F4): a LoRA adapter trained on a node, from a base
+-- model's weights and question/answer pairs, and what it made. The pairs
+-- and the adapter are kept here, in the store every control plane reads:
+-- they are megabytes, and a job's record is worth little without them.
+CREATE TABLE train_jobs (
+  id             TEXT PRIMARY KEY,
+  -- name is the adapter's: what clients will ask for it by.
+  name           TEXT NOT NULL,
+  -- model is the catalog model the adapter is for; base_repo and
+  -- base_revision the weights it is trained on.
+  model          TEXT NOT NULL,
+  base_repo      TEXT NOT NULL,
+  base_revision  TEXT NOT NULL DEFAULT '',
+  node           TEXT NOT NULL,
+  gpu_memory_mib INTEGER NOT NULL,
+  params         TEXT NOT NULL DEFAULT '',
+  gguf           INTEGER NOT NULL DEFAULT 0,
+  data           BYTEA NOT NULL,
+  pairs          INTEGER NOT NULL,
+  -- state is queued, starting, preparing, running, done, failed or
+  -- cancelled.
+  state          TEXT NOT NULL,
+  step           INTEGER NOT NULL DEFAULT 0,
+  steps          INTEGER NOT NULL DEFAULT 0,
+  loss           DOUBLE PRECISION NOT NULL DEFAULT 0,
+  message        TEXT NOT NULL DEFAULT '',
+  peak_mib       INTEGER NOT NULL DEFAULT 0,
+  created_at     BIGINT NOT NULL,
+  started_at     BIGINT NOT NULL DEFAULT 0,
+  finished_at    BIGINT NOT NULL DEFAULT 0,
+  created_by     TEXT NOT NULL DEFAULT ''
+);
+CREATE INDEX train_jobs_by_state ON train_jobs (state);
+
+-- What a job made: the PEFT adapter as a gzipped tar, and for llama.cpp the
+-- adapter as GGUF.
+CREATE TABLE train_artifacts (
+  job_id TEXT NOT NULL REFERENCES train_jobs(id) ON DELETE CASCADE,
+  kind   TEXT NOT NULL,
+  data   BYTEA NOT NULL,
+  PRIMARY KEY (job_id, kind)
+);
@@ -0,0 +1,5 @@
+-- Who an adapter belongs to (BACKLOG F4): scope:ID in the tenancy tree, or
+-- empty for every caller its model allows. An adapter trained on a
+-- knowledge base's documents carries some of what they say, and belongs to
+-- a part of the tree that may read them.
+ALTER TABLE train_jobs ADD COLUMN owner TEXT NOT NULL DEFAULT '';
@@ -0,0 +1,5 @@
+-- How a collection's vectors are indexed: vector (single precision) or
+-- halfvec (half, pgvector 0.7 and later), which halves the index a search
+-- has to hold in memory for the same recall. Fixed when the collection is
+-- made: its index and its queries name the same type.
+ALTER TABLE kb_collections ADD COLUMN vec_type TEXT NOT NULL DEFAULT 'vector';
@@ -0,0 +1,7 @@
+-- What a control plane works out for machines that cannot ask it: the
+-- store's members read it from their own copy of the store.
+CREATE TABLE cluster_state (
+  key        TEXT PRIMARY KEY,
+  value      TEXT NOT NULL,
+  updated_at BIGINT NOT NULL
+);
@@ -0,0 +1,42 @@
+-- Training jobs (BACKLOG F4): a LoRA adapter trained on a node, from a base
+-- model's weights and question/answer pairs, and what it made. The pairs
+-- and the adapter are kept here, in the store every control plane reads:
+-- they are megabytes, and a job's record is worth little without them.
+CREATE TABLE train_jobs (
+  id             TEXT PRIMARY KEY,
+  -- name is the adapter's: what clients will ask for it by.
+  name           TEXT NOT NULL,
+  -- model is the catalog model the adapter is for; base_repo and
+  -- base_revision the weights it is trained on.
+  model          TEXT NOT NULL,
+  base_repo      TEXT NOT NULL,
+  base_revision  TEXT NOT NULL DEFAULT '',
+  node           TEXT NOT NULL,
+  gpu_memory_mib INTEGER NOT NULL,
+  params         TEXT NOT NULL DEFAULT '',
+  gguf           INTEGER NOT NULL DEFAULT 0,
+  data           BLOB NOT NULL,
+  pairs          INTEGER NOT NULL,
+  -- state is queued, starting, preparing, running, done, failed or
+  -- cancelled.
+  state          TEXT NOT NULL,
+  step           INTEGER NOT NULL DEFAULT 0,
+  steps          INTEGER NOT NULL DEFAULT 0,
+  loss           REAL NOT NULL DEFAULT 0,
+  message        TEXT NOT NULL DEFAULT '',
+  peak_mib       INTEGER NOT NULL DEFAULT 0,
+  created_at     INTEGER NOT NULL,
+  started_at     INTEGER NOT NULL DEFAULT 0,
+  finished_at    INTEGER NOT NULL DEFAULT 0,
+  created_by     TEXT NOT NULL DEFAULT ''
+);
+CREATE INDEX train_jobs_by_state ON train_jobs (state);
+
+-- What a job made: the PEFT adapter as a gzipped tar, and for llama.cpp the
+-- adapter as GGUF.
+CREATE TABLE train_artifacts (
+  job_id TEXT NOT NULL REFERENCES train_jobs(id) ON DELETE CASCADE,
+  kind   TEXT NOT NULL,
+  data   BLOB NOT NULL,
+  PRIMARY KEY (job_id, kind)
+);
@@ -0,0 +1,5 @@
+-- Who an adapter belongs to (BACKLOG F4): scope:ID in the tenancy tree, or
+-- empty for every caller its model allows. An adapter trained on a
+-- knowledge base's documents carries some of what they say, and belongs to
+-- a part of the tree that may read them.
+ALTER TABLE train_jobs ADD COLUMN owner TEXT NOT NULL DEFAULT '';
@@ -0,0 +1,7 @@
+-- What a control plane works out for machines that cannot ask it: the
+-- store's members read it from their own copy of the store.
+CREATE TABLE cluster_state (
+  key        TEXT PRIMARY KEY,
+  value      TEXT NOT NULL,
+  updated_at INTEGER NOT NULL
+);
@@ -52,6 +52,9 @@ type DB struct {
 	// kbErr is why this store cannot hold knowledge bases, nil when it can;
 	// set once at open (knowledge.go).
 	kbErr error
+	// kbHalfvec is pgvector 0.7 or later here: new collections index their
+	// vectors as halfvec.
+	kbHalfvec bool
 }

 // OpenSQLite opens (creating if needed) a database and brings it up to date.
@@ -0,0 +1,203 @@
+package store
+
+import (
+	"context"
+	"database/sql"
+	"errors"
+	"time"
+)
+
+// TrainJob is a LoRA adapter trained on a node (BACKLOG F4).
+type TrainJob struct {
+	ID           string
+	Name         string
+	Model        string
+	BaseRepo     string
+	BaseRevision string
+	Node         string
+	// Owner is who the adapter belongs to, scope:ID; empty is everyone.
+	Owner        string
+	GPUMemoryMiB int
+	Params       string
+	GGUF         bool
+	// Data is the pairs, NDJSON; Pairs how many.
+	Data  []byte
+	Pairs int
+
+	State      string
+	Step       int
+	Steps      int
+	Loss       float64
+	Message    string
+	PeakMiB    int
+	CreatedAt  time.Time
+	StartedAt  time.Time
+	FinishedAt time.Time
+	CreatedBy  string
+}
+
+// The states of a training job.
+const (
+	TrainQueued    = "queued"
+	TrainStarting  = "starting"
+	TrainDone      = "done"
+	TrainFailed    = "failed"
+	TrainCancelled = "cancelled"
+)
+
+// TrainOver is whether a state is one a job ends in.
+func TrainOver(state string) bool {
+	return state == TrainDone || state == TrainFailed || state == TrainCancelled
+}
+
+// CreateTrainJob queues a job.
+func (s *DB) CreateTrainJob(ctx context.Context, j TrainJob) error {
+	if j.CreatedAt.IsZero() {
+		j.CreatedAt = time.Now()
+	}
+	gguf := 0
+	if j.GGUF {
+		gguf = 1
+	}
+	_, err := s.db.ExecContext(ctx, `INSERT INTO train_jobs (id, name, model, base_repo, base_revision, node, owner, gpu_memory_mib,
+		params, gguf, data, pairs, state, created_at, created_by) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
+		j.ID, j.Name, j.Model, j.BaseRepo, j.BaseRevision, j.Node, j.Owner, j.GPUMemoryMiB, j.Params, gguf, j.Data, j.Pairs,
+		TrainQueued, millis(j.CreatedAt), j.CreatedBy)
+	return err
+}
+
+const trainCols = `id, name, model, base_repo, base_revision, node, owner, gpu_memory_mib, params, gguf, pairs, state, step, steps,
+	loss, message, peak_mib, created_at, started_at, finished_at, created_by`
+
+func scanTrainJob(sc scanner) (TrainJob, error) {
+	var j TrainJob
+	var gguf int
+	var created, started, finished int64
+	err := sc.Scan(&j.ID, &j.Name, &j.Model, &j.BaseRepo, &j.BaseRevision, &j.Node, &j.Owner, &j.GPUMemoryMiB, &j.Params, &gguf,
+		&j.Pairs, &j.State, &j.Step, &j.Steps, &j.Loss, &j.Message, &j.PeakMiB, &created, &started, &finished, &j.CreatedBy)
+	j.GGUF = gguf != 0
+	j.CreatedAt = fromMillis(created)
+	if started > 0 {
+		j.StartedAt = fromMillis(started)
+	}
+	if finished > 0 {
+		j.FinishedAt = fromMillis(finished)
+	}
+	return j, err
+}
+
+// TrainJobs lists jobs, newest first.
+func (s *DB) TrainJobs(ctx context.Context, limit int) ([]TrainJob, error) {
+	return queryAll(ctx, s.db, `SELECT `+trainCols+` FROM train_jobs ORDER BY created_at DESC LIMIT ?`, []any{limit},
+		func(r *sql.Rows) (TrainJob, error) { return scanTrainJob(r) })
+}
+
+// TrainJob reads one job, with its data.
+func (s *DB) TrainJob(ctx context.Context, id string) (TrainJob, error) {
+	j, err := scanTrainJob(s.db.QueryRowContext(ctx, `SELECT `+trainCols+` FROM train_jobs WHERE id = ?`, id))
+	if errors.Is(err, sql.ErrNoRows) {
+		return TrainJob{}, ErrNotFound
+	}
+	if err != nil {
+		return j, err
+	}
+	err = s.db.QueryRowContext(ctx, `SELECT data FROM train_jobs WHERE id = ?`, id).Scan(&j.Data)
+	return j, err
+}
+
+// QueuedTrainJobs are the jobs waiting for their node, oldest first.
+func (s *DB) QueuedTrainJobs(ctx context.Context) ([]TrainJob, error) {
+	return queryAll(ctx, s.db, `SELECT `+trainCols+` FROM train_jobs WHERE state = ? ORDER BY created_at`, []any{TrainQueued},
+		func(r *sql.Rows) (TrainJob, error) { return scanTrainJob(r) })
+}
+
+// ActiveTrainJobs are the jobs sent to a node and not ended.
+func (s *DB) ActiveTrainJobs(ctx context.Context) ([]TrainJob, error) {
+	return queryAll(ctx, s.db, `SELECT `+trainCols+` FROM train_jobs WHERE state NOT IN (?, ?, ?, ?)`,
+		[]any{TrainQueued, TrainDone, TrainFailed, TrainCancelled},
+		func(r *sql.Rows) (TrainJob, error) { return scanTrainJob(r) })
+}
+
+// UpdateTrainJob records how a job goes; a job that has ended is not
+// changed again, so a late message cannot bring one back.
+func (s *DB) UpdateTrainJob(ctx context.Context, j TrainJob) error {
+	var started, finished int64
+	if !j.StartedAt.IsZero() {
+		started = millis(j.StartedAt)
+	}
+	if !j.FinishedAt.IsZero() {
+		finished = millis(j.FinishedAt)
+	}
+	_, err := s.db.ExecContext(ctx, `UPDATE train_jobs SET state = ?, step = ?, steps = ?, loss = ?, message = ?,
+		peak_mib = ?, started_at = ?, finished_at = ? WHERE id = ? AND state NOT IN (?, ?, ?)`,
+		j.State, j.Step, j.Steps, j.Loss, j.Message, j.PeakMiB, started, finished, j.ID,
+		TrainDone, TrainFailed, TrainCancelled)
+	return err
+}
+
+// PutTrainArtifact keeps what a job made.
+func (s *DB) PutTrainArtifact(ctx context.Context, job, kind string, data []byte) error {
+	_, err := s.db.ExecContext(ctx, `INSERT INTO train_artifacts (job_id, kind, data) VALUES (?, ?, ?)
+		ON CONFLICT (job_id, kind) DO UPDATE SET data = excluded.data`, job, kind, data)
+	return err
+}
+
+// TrainArtifact reads what a job made, of one kind.
+func (s *DB) TrainArtifact(ctx context.Context, job, kind string) ([]byte, error) {
+	var b []byte
+	err := s.db.QueryRowContext(ctx, `SELECT data FROM train_artifacts WHERE job_id = ? AND kind = ?`, job, kind).Scan(&b)
+	if errors.Is(err, sql.ErrNoRows) {
+		return nil, ErrNotFound
+	}
+	return b, err
+}
+
+// TrainArtifactKinds lists what a job made.
+func (s *DB) TrainArtifactKinds(ctx context.Context, job string) ([]string, error) {
+	return queryAll(ctx, s.db, `SELECT kind FROM train_artifacts WHERE job_id = ? ORDER BY kind`, []any{job},
+		func(r *sql.Rows) (string, error) {
+			var k string
+			return k, r.Scan(&k)
+		})
+}
+
+// DeleteTrainJob removes a job and what it made.
+func (s *DB) DeleteTrainJob(ctx context.Context, id string) error {
+	return s.inTx(ctx, func(c conn) error {
+		if _, err := c.ExecContext(ctx, `DELETE FROM train_artifacts WHERE job_id = ?`, id); err != nil {
+			return err
+		}
+		res, err := c.ExecContext(ctx, `DELETE FROM train_jobs WHERE id = ?`, id)
+		if err != nil {
+			return err
+		}
+		if n, _ := res.RowsAffected(); n == 0 {
+			return ErrNotFound
+		}
+		return nil
+	})
+}
+
+// EffectiveFor is a scope's place in the tree, as a caller in it would have
+// it, for asking what it may read: an organization, a team or a project
+// with what is above it, a key as it resolves, a person in an organization.
+func (s *DB) EffectiveFor(ctx context.Context, scope Scope, id, org string) (Effective, error) {
+	switch scope {
+	case ScopeOrg:
+		return Effective{OrgID: id}, nil
+	case ScopeTeam:
+		var o string
+		err := s.db.QueryRowContext(ctx, `SELECT org_id FROM teams WHERE id = ?`, id).Scan(&o)
+		if errors.Is(err, sql.ErrNoRows) {
+			return Effective{}, ErrNotFound
+		}
+		return Effective{TeamID: id, OrgID: o}, err
+	case ScopeProject:
+		return s.ResolveFor(ctx, "", "", id)
+	case ScopeKey:
+		return s.Resolve(ctx, id)
+	case ScopeUser:
+		return Effective{UserID: id, OrgID: org}, nil
+	}
+	return Effective{}, errors.New("no such scope " + string(scope))
+}
@@ -90,7 +90,9 @@ type AccountPages struct {
 	audit audit.Sink
 	// hasGuard is set when /admin/guard is registered, and guardLibrary
 	// with it: the policies in force, whose judges the mirror page serves.
-	hasGuard     bool
+	hasGuard bool
+	// hasSettings is set when /admin/settings is registered.
+	hasSettings  bool
 	guardLibrary func() guard.Config
 	// guardFor names the guardrail policies in force for someone, and
 	// whether they keep nothing; nil when there are none.
@@ -325,6 +327,10 @@ func (p *AccountPages) logout(w http.ResponseWriter, r *http.Request) {
 		}
 	}
 	clearSessionCookie(w, r)
+	// What the pages kept in this browser goes with the session: the chat's
+	// conversation, htmx's copies of the pages visited. The chat keys its
+	// own by account as well, for a browser that ignores this header.
+	w.Header().Set("Clear-Site-Data", `"storage"`)
 	http.Redirect(w, r, "/login", http.StatusSeeOther)
 }

@@ -0,0 +1,174 @@
+package web
+
+import (
+	"context"
+	"errors"
+	"net/http"
+	"strconv"
+	"strings"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// The cluster's settings, edited in the browser (BACKLOG E4): the document
+// `llmgw ctl config edit` opens, for whoever has no terminal on a control
+// plane. The same rules hold here as there — a version is checked as a
+// control plane would load it before it is kept, it is made from the newest
+// version or refused, and every change is in force on every control plane
+// within seconds and in the audit log.
+
+// SettingsAdmin is where the cluster's settings are kept.
+type SettingsAdmin interface {
+	ClusterSettings(ctx context.Context) (store.SettingsVersion, error)
+	ClusterSettingsAt(ctx context.Context, version int64) (store.SettingsVersion, error)
+	ClusterSettingsHistory(ctx context.Context, limit int) ([]store.SettingsVersion, error)
+	PutClusterSettings(ctx context.Context, base int64, body, by, note string) (int64, error)
+}
+
+type settingsPageData struct {
+	Nav     Nav
+	Version string
+	CSRF    string
+	// Current is the version in force; Shown the one in the editor, which
+	// is an earlier one when ?version= asked for it.
+	Current, Shown store.SettingsVersion
+	Earlier        bool
+	// Body is what the editor holds: the version shown, or what was sent
+	// and refused, so a refusal does not lose the edit.
+	Body      string
+	Note      string
+	History   []settingsRow
+	LocalKeys []string
+	Notice    string
+	Error     string
+}
+
+type settingsRow struct {
+	Version  int64
+	At, By   string
+	Note     string
+	InForce  bool
+	Selected bool
+}
+
+// RegisterAdminSettings mounts /admin/settings, for administrators only:
+// the settings hold webhook addresses and the like, so even reading them is
+// not every staff role's.
+func (p *AccountPages) RegisterAdminSettings(mux *http.ServeMux, s SettingsAdmin, check func([]byte) error, localKeys []string) {
+	p.hasSettings = true
+	page := func(w http.ResponseWriter, r *http.Request, admin store.User, d settingsPageData) {
+		ctx, l := r.Context(), langOf(r)
+		d.Nav, d.Version, d.CSRF, d.LocalKeys = p.navFor(r, admin, "settings"), p.version, csrfFor(r), localKeys
+		cur, err := s.ClusterSettings(ctx)
+		if err != nil {
+			d.Error = l.Err(err)
+		}
+		d.Current = cur
+		if d.Shown.Version == 0 {
+			d.Shown = cur
+		}
+		d.Earlier = d.Shown.Version != cur.Version
+		if d.Body == "" {
+			d.Body = d.Shown.Body
+		}
+		if hist, err := s.ClusterSettingsHistory(ctx, 30); err == nil {
+			for _, v := range hist {
+				d.History = append(d.History, settingsRow{Version: v.Version, At: l.Date(v.At.UTC()) + " " + v.At.UTC().Format("15:04"),
+					By: v.By, Note: v.Note, InForce: v.Version == cur.Version, Selected: v.Version == d.Shown.Version})
+			}
+		}
+		noStore(w)
+		render(w, r, "admin_settings", d)
+	}
+	adminsOnly := func(next func(w http.ResponseWriter, r *http.Request, admin store.User)) http.HandlerFunc {
+		return p.adminOnly(func(w http.ResponseWriter, r *http.Request, admin store.User) {
+			if !admin.IsAdmin {
+				http.NotFound(w, r)
+				return
+			}
+			if r.Method == http.MethodPost {
+				sess, _, _ := p.session(r)
+				if !p.accounts.CheckCSRF(sess, r.FormValue("csrf")) {
+					http.Error(w, "bad request", http.StatusBadRequest)
+					return
+				}
+			}
+			next(w, r, admin)
+		})
+	}
+
+	mux.HandleFunc("GET /admin/settings", adminsOnly(func(w http.ResponseWriter, r *http.Request, admin store.User) {
+		var d settingsPageData
+		if q := r.URL.Query().Get("version"); q != "" {
+			n, err := strconv.ParseInt(q, 10, 64)
+			if v, verr := s.ClusterSettingsAt(r.Context(), n); err == nil && verr == nil && v.Version != 0 {
+				d.Shown = v
+			} else {
+				d.Error = langOf(r).T("settings.err.no_version")
+			}
+		}
+		page(w, r, admin, d)
+	}))
+
+	mux.HandleFunc("POST /admin/settings", adminsOnly(func(w http.ResponseWriter, r *http.Request, admin store.User) {
+		l := langOf(r)
+		body := strings.ReplaceAll(r.FormValue("body"), "\r\n", "\n")
+		note := strings.TrimSpace(r.FormValue("note"))
+		base, _ := strconv.ParseInt(r.FormValue("base"), 10, 64)
+		// The audit line records the form; the document can hold webhook
+		// secrets, so it records the document's size, not the document.
+		r.PostForm.Set("body", "("+strconv.Itoa(len(body))+" bytes)")
+		refuse := func(msg string) {
+			noteAudit(r, "cluster settings", "", msg)
+			page(w, r, admin, settingsPageData{Body: body, Note: note, Error: msg})
+		}
+		if cur, err := s.ClusterSettings(r.Context()); err == nil && cur.Body == body {
+			refuse(l.T("settings.err.unchanged"))
+			return
+		}
+		if err := check([]byte(body)); err != nil {
+			refuse(l.T("settings.err.invalid") + " " + err.Error())
+			return
+		}
+		n, err := s.PutClusterSettings(r.Context(), base, body, admin.Email, note)
+		switch {
+		case errors.Is(err, store.ErrSettingsChanged):
+			refuse(l.T("settings.err.changed"))
+			return
+		case err != nil:
+			refuse(l.Err(err))
+			return
+		}
+		noteAudit(r, "cluster settings", "version "+strconv.FormatInt(n, 10), "")
+		page(w, r, admin, settingsPageData{Notice: l.T("settings.saved", n)})
+	}))
+
+	mux.HandleFunc("POST /admin/settings/rollback", adminsOnly(func(w http.ResponseWriter, r *http.Request, admin store.User) {
+		l := langOf(r)
+		n, _ := strconv.ParseInt(r.FormValue("version"), 10, 64)
+		old, err := s.ClusterSettingsAt(r.Context(), n)
+		if err != nil || old.Version == 0 {
+			page(w, r, admin, settingsPageData{Error: l.T("settings.err.no_version")})
+			return
+		}
+		cur, err := s.ClusterSettings(r.Context())
+		if err != nil {
+			page(w, r, admin, settingsPageData{Error: l.Err(err)})
+			return
+		}
+		if err := check([]byte(old.Body)); err != nil {
+			// A version kept before a change to server.yaml may no longer
+			// load beside it; refused here rather than breaking the cluster.
+			noteAudit(r, "cluster settings", "", err.Error())
+			page(w, r, admin, settingsPageData{Error: l.T("settings.err.invalid") + " " + err.Error()})
+			return
+		}
+		v, err := s.PutClusterSettings(r.Context(), cur.Version, old.Body, admin.Email, "back to version "+strconv.FormatInt(n, 10))
+		if err != nil {
+			page(w, r, admin, settingsPageData{Error: l.Err(err)})
+			return
+		}
+		noteAudit(r, "cluster settings", "version "+strconv.FormatInt(v, 10)+" from "+strconv.FormatInt(n, 10), "")
+		page(w, r, admin, settingsPageData{Notice: l.T("settings.rolled_back", n, v)})
+	}))
+}
@@ -0,0 +1,143 @@
+package web
+
+import (
+	"context"
+	"errors"
+	"io"
+	"log/slog"
+	"net/http"
+	"net/url"
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+type fakeSettings struct{ versions []store.SettingsVersion }
+
+func (f *fakeSettings) ClusterSettings(context.Context) (store.SettingsVersion, error) {
+	if len(f.versions) == 0 {
+		return store.SettingsVersion{}, nil
+	}
+	return f.versions[len(f.versions)-1], nil
+}
+
+func (f *fakeSettings) ClusterSettingsAt(_ context.Context, v int64) (store.SettingsVersion, error) {
+	for _, s := range f.versions {
+		if s.Version == v {
+			return s, nil
+		}
+	}
+	return store.SettingsVersion{}, nil
+}
+
+func (f *fakeSettings) ClusterSettingsHistory(context.Context, int) ([]store.SettingsVersion, error) {
+	out := make([]store.SettingsVersion, len(f.versions))
+	for i, v := range f.versions {
+		out[len(f.versions)-1-i] = v
+	}
+	return out, nil
+}
+
+func (f *fakeSettings) PutClusterSettings(_ context.Context, base int64, body, by, note string) (int64, error) {
+	cur, _ := f.ClusterSettings(context.Background())
+	if base != cur.Version {
+		return 0, store.ErrSettingsChanged
+	}
+	n := cur.Version + 1
+	f.versions = append(f.versions, store.SettingsVersion{Version: n, Body: body, By: by, Note: note, At: time.Now()})
+	return n, nil
+}
+
+type auditSink struct{ events []store.AuditEvent }
+
+func (a *auditSink) WriteAudit(_ context.Context, e store.AuditEvent) error {
+	a.events = append(a.events, e)
+	return nil
+}
+
+func clusterSettingsPage(t *testing.T, user store.User) (*http.ServeMux, *fakeSettings, *auditSink) {
+	t.Helper()
+	s := &fakeSettings{versions: []store.SettingsVersion{{Version: 1, Body: "guard:\n  enabled: true\n", By: "init", At: time.Now()}}}
+	p := NewAccountPages(fakeAccounts{user: user}, nil, nil, "v0", slog.New(slog.NewTextHandler(io.Discard, nil)))
+	sink := &auditSink{}
+	p.SetAudit(sink)
+	mux := http.NewServeMux()
+	p.RegisterAdminSettings(mux, s, func(b []byte) error {
+		if strings.Contains(string(b), "bogus") {
+			return errors.New("unknown key bogus")
+		}
+		return nil
+	}, []string{"listen", "store"})
+	return mux, s, sink
+}
+
+func TestTheSettingsPageIsForAdministratorsOnly(t *testing.T) {
+	for _, u := range []store.User{
+		{ID: "u1", Email: "v@example.com", Role: store.RoleViewer},
+		{ID: "u2", Email: "o@example.com", Role: store.RoleOperator},
+		{ID: "u3", Email: "b@example.com", Role: store.RoleBilling},
+	} {
+		mux, _, _ := clusterSettingsPage(t, u)
+		if w := mirrorGet(t, mux, "/admin/settings"); w.Code != http.StatusNotFound {
+			t.Errorf("%s saw the settings: %d", u.Role, w.Code)
+		}
+	}
+	mux, _, _ := clusterSettingsPage(t, store.User{ID: "a", Email: "a@example.com", IsAdmin: true})
+	body := mirrorGet(t, mux, "/admin/settings").Body.String()
+	for _, want := range []string{"guard:\n  enabled: true", "Version 1, in force", "<code>listen</code>", `href="/admin/settings" aria-current="page"`} {
+		if !strings.Contains(body, want) {
+			t.Errorf("no %q in:\n%s", want, body)
+		}
+	}
+}
+
+func TestSettingsAreCheckedAppliedAndRolledBack(t *testing.T) {
+	mux, s, sink := clusterSettingsPage(t, store.User{ID: "a", Email: "a@example.com", IsAdmin: true})
+	post := func(form url.Values) string {
+		form.Set("csrf", "tok")
+		return mirrorPost(t, mux, "/admin/settings", form).Body.String()
+	}
+	// Refused: does not load. The edit stays in the editor; nothing kept.
+	body := post(url.Values{"base": {"1"}, "body": {"bogus: 1\n"}})
+	if !strings.Contains(body, "unknown key bogus") || !strings.Contains(body, "bogus: 1") || len(s.versions) != 1 {
+		t.Errorf("an invalid document: %d versions\n%s", len(s.versions), body)
+	}
+	// Applied, with its note and who.
+	body = post(url.Values{"base": {"1"}, "body": {"guard:\r\n  enabled: false\r\n"}, "note": {"maintenance"}})
+	if !strings.Contains(body, "Version 2 is in force") || len(s.versions) != 2 ||
+		s.versions[1].Body != "guard:\n  enabled: false\n" || s.versions[1].By != "a@example.com" || s.versions[1].Note != "maintenance" {
+		t.Errorf("applying: %+v\n%s", s.versions, body)
+	}
+	// Made from a version no longer the newest: refused, not overwritten.
+	body = post(url.Values{"base": {"1"}, "body": {"guard:\n  enabled: true\n  x: 1\n"}})
+	if !strings.Contains(body, "someone changed the settings") || len(s.versions) != 2 {
+		t.Errorf("a stale edit: %d versions\n%s", len(s.versions), body)
+	}
+	// An earlier version is shown, and put back as a new one.
+	if body := mirrorGet(t, mux, "/admin/settings?version=1").Body.String(); !strings.Contains(body, "Put version 1 back in force") {
+		t.Errorf("viewing version 1:\n%s", body)
+	}
+	w := mirrorPost(t, mux, "/admin/settings/rollback", url.Values{"csrf": {"tok"}, "version": {"1"}})
+	if len(s.versions) != 3 || s.versions[2].Body != s.versions[0].Body || !strings.Contains(w.Body.String(), "as version 3") {
+		t.Errorf("rollback: %+v\n%s", s.versions, w.Body.String())
+	}
+	// Without the CSRF token, nothing.
+	if w := mirrorPost(t, mux, "/admin/settings", url.Values{"base": {"3"}, "body": {"a: 1\n"}}); w.Code != http.StatusBadRequest || len(s.versions) != 3 {
+		t.Errorf("no token: %d", w.Code)
+	}
+	// Audited, without the document itself: it can hold secrets.
+	var saw bool
+	for _, e := range sink.events {
+		if strings.Contains(e.Detail, "enabled") {
+			t.Errorf("the document is in the audit log: %+v", e)
+		}
+		if e.Action == "POST /admin/settings" && strings.Contains(e.Detail, "version 2") {
+			saw = true
+		}
+	}
+	if !saw {
+		t.Errorf("no audit line for version 2: %+v", sink.events)
+	}
+}
Author	SHA1	Message	Date
Ilker ManapandClaude Opus 5.5	3894e8e5fd	Backlog F5: Qwen3-Embedding-4B measured, on llama.cpp and vLLM, and against bge-m3 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-09 21:36:23 +02:00
Ilker ManapandClaude Opus 5.5	751533f853	Docs: pictures from ComfyUI Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-09 21:26:31 +02:00
Ilker ManapandClaude Opus 5.5	79a503f824	A picture from ComfyUI loses the workflow written into it ComfyUI saves the whole workflow — prompt, checkpoint, nodes — as PNG text chunks in every picture. They are taken out before the picture is sent. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-09 21:22:02 +02:00
Ilker ManapandClaude Opus 5.5	2f06c0e513	Pictures from ComfyUI at /v1/images/generations An external model with api: comfyui is a ComfyUI server. A request for pictures goes through the same key, allowlist, limit, admission and guard checks as any other; at the replica it becomes a workflow — a built-in (flux-schnell, flux-dev) or a JSON file in API format, with {{prompt}}, {{width}}, {{seed}} and the like filled in — queued, waited on, and its pictures sent back as b64_json. A client that leaves takes its workflow off the queue or interrupts it. ComfyUI is never scraped for metrics. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-09 21:15:50 +02:00
Ilker ManapandClaude Opus 5.5	6b7750a0dd	An embedding model's query_prefix, put before questions and never before documents Qwen3-Embedding and models like it are trained with an instruction on the query side only. The prefix goes before the question a knowledge-base search embeds, at answer time and in the admin API's search; documents reach /v1/embeddings from llmgw-kb as they are. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-09 18:03:00 +02:00
Ilker ManapandClaude Opus 5.5	a133270725	The chat's kept conversation belongs to the account that kept it It sat in localStorage under one fixed key, so the next person to sign in on the same browser opened the chat on the last one's conversation. It is now kept under the account's id, anyone else's is dropped when the page loads, and signing out sends Clear-Site-Data: "storage" so nothing the pages kept outlives the session. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-09 15:44:47 +02:00
Ilker ManapandClaude Opus 5.5	8932502768	Backlog F5: where a large knowledge-base import spends its time Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-09 15:19:35 +02:00
Ilker ManapandClaude Opus 5.5	92733317a9	A batch of embeddings is checked text by text against the model's context llmgw-kb sends many passages in one /v1/embeddings request. Their sizes were added together and compared with the context, so a batch of passages that each fit was refused as one over-long prompt. The longest input is what has to fit; the total still counts for usage and limits. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-09 11:43:21 +02:00
Ilker ManapandClaude Opus 5.5	37ad1657ee	v0.14.1: changelog and README Release / build (push) Successful in 1m59s Details Release / notes (push) Successful in 5s Details CI / test (push) Successful in 8m37s Details Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-09 10:11:10 +02:00
Ilker ManapandClaude Opus 5.5	8df78a4d41	The firewall setting reaches the mirror machine and the store's members The leading control plane tells the mirror over its API (PUT /v1/firewall, kept as .firewall.json beside the models and applied by the same path unit control planes use), and keeps the state in the store's cluster_state for members that run no llmgw, which a root timer reads from the member's own copy. The mirror also lets in the network of every node and control plane the leader knows. Tried on the PoC: off, on and a network added, on every kind of machine. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-09 08:53:17 +02:00
Ilker ManapandClaude Opus 5.5	5be3da505b	Keys' project paths written as the API takes them, Acme/ml/search Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-09 06:46:12 +02:00
Ilker ManapandClaude Opus 5.5	15a209e652	Terraform: organizations, teams, projects, keys and memberships llmgw_org, llmgw_team and llmgw_project rename and move in place; llmgw_key holds the key in a sensitive secret and is made again when revoked or expired outside Terraform; llmgw_membership puts a person in the tree. Tried with OpenTofu against the fake admin API, drift and import included. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-09 06:42:07 +02:00
Ilker ManapandClaude Opus 5.5	90e0942481	Organizations, teams, projects, keys and memberships in the admin API What llmgw org, keys and users did against the store on a control plane can now be done remotely and by automation: the tree read, added to, renamed, moved and emptied; keys issued (the key in the answer, once), listed and revoked, with the scopes' lifetime limits; people put in and taken out of the tree, their keys in projects they no longer reach revoked with it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-08 22:42:43 +02:00
Ilker ManapandClaude Opus 5.5	9c675e7293	Knowledge bases at scale: halfvec, a text index per collection, ef_search raised to the candidates wanted A search asking for more than 10 passages got at most 40 candidates from HNSW, pgvector's default ef_search. The full-text arm scored every chunk holding a word of the question before limiting, over the whole organization's GIN index; it now takes candidates by ts_rank from the collection's own partial index and scores those. New collections index halfvec on pgvector 0.7 and later (up to 4000 dimensions). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-08 22:08:29 +02:00
Ilker ManapandClaude Opus 5.5	42d3b7ea22	The store's PostgreSQL sized to its machine's memory Each Patroni member gets a share of its own RAM in its own config.yml: shared_buffers a quarter (15% beside a control plane or agent), effective_cache_size, maintenance_work_mem and work_mem scaled to it, all clamped. It is read again on every server init --patroni, so a machine given more memory gets the use of it. A single-machine PostgreSQL init installs is sized the same way, as a shared machine, instead of a fixed 96MB. The step itself went in with `85025df` by accident, in an early form that restarted PostgreSQL before Patroni had taken the reload, leaving the old shared_buffers with a restart pending. It now waits for Patroni to report the restart pending, restarts with restart_pending (replicas first, the leader last), and restarts a member it finds pending from an earlier run. Applied to the PoC: pg01 978MB, ctl02 (control plane beside it) 587MB, both from 128MB; streaming, lag 0, gateways unaffected. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-08 21:48:53 +02:00
Ilker ManapandClaude Opus 5.5	85025dfb03	Trained adapters published, private to their owners like the knowledge bases they come from An adapter now has an owner (org, team, project, user or key): only callers there see it, and to anyone else it is a model that does not exist, absent from /v1/models too. A training job's owner defaults to the one collection its pairs come from, must be able to read every collection they come from, and pairs from a knowledge base are never made public. ctl train publish keeps the GGUF or PEFT files on the mirror, which accepts uploads now, and sets the owned adapter after the usual architecture check. The knowledge bases' cache_salt follows the collections searched rather than the organization, so two teams of one organization no longer share cached passages. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-08 21:30:50 +02:00
Ilker ManapandClaude Opus 5.5	964b35ae65	Backups reach a replicated store: pg_dump is given the cluster's authority with verify-full pgEnv passed sslmode=verify-full but dropped sslrootcert, so libpq looked for ~/.postgresql/root.crt and both backup and restore failed against the Patroni store. sslrootcert, sslcert and sslkey now go along as PGSSL*. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-08 21:04:46 +02:00
Ilker ManapandClaude Opus 5.5	88cbf11570	Training finds the engines where the installer puts them when agent.yaml names no directory Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-08 20:55:50 +02:00
Ilker ManapandClaude Opus 5.5	79b3e9f887	LoRA adapters trained on the nodes: llmgw ctl train A job names the catalog model the adapter is for, the safetensors weights to train on, the node and the MiB of a card it needs, and question/answer pairs. The control plane sends it when the node's agent is there and nothing else trains on it; the agent trains with its installed vLLM's torch and a pinned overlay of peft and accelerate (no second environment), as the engines' user, on the card with that much free — beside an engine, if it leaves that much — converts the adapter to GGUF with the installed llama.cpp's own conversion code when the model is served by llama.cpp, and sends what it made back in pieces, kept in the store to download. Progress, loss and peak card memory are reported as it goes; a node that goes fails its job. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-08 20:25:57 +02:00
Ilker ManapandClaude Opus 5.5	ae95ac8261	Training data from a collection: llmgw ctl kb dataset A chat model of the cluster reads each chunk and writes questions a person might ask of it with the answers it holds, streamed as NDJSON a chunk at a time, for people to correct before an adapter is trained on it. A chunk that gave no pairs says so on its own line. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-08 20:04:37 +02:00
Ilker ManapandClaude Opus 5.5	0a6a586dfe	Backlog F4: a LoRA trained in the GPU's spare 2 GB on the PoC, and what it learned Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-08 19:31:53 +02:00
Ilker ManapandClaude Opus 5.5	c52b73feec	Reference: the agents section of the cluster's settings Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-08 18:07:00 +02:00
Ilker ManapandClaude Opus 5.5	dc243f8840	The nodes' agent settings kept with the cluster's, agent.yaml down to what an agent needs to start agents.default and agents.nodes.<id> in the cluster's settings hold everything agent.yaml can say but the agent's id, control planes, address and certificate, which stay in the file and are refused there. The control plane sends each node its settings on register and, a second apart, when they change; an agent whose settings changed checks they load beside its file, keeps them and starts again on them, engines serving throughout. An edited agent.yaml that loads restarts the agent too. History, audit, ctl config, the console and the portal's settings page cover them as the rest of the settings. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-08 18:06:36 +02:00
Ilker ManapandClaude Opus 5.5	659f20dc9a	The cluster's settings edited on the portal, by administrators /admin/settings edits the document llmgw ctl config edit opens: checked as a control plane would load it, refused when made from a version since replaced (the edit kept), its history viewed and rolled back as a new version, every change audited by size and never by contents, since it can hold webhook secrets. Administrators only; in English and Turkish. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-08 17:23:13 +02:00
Ilker ManapandClaude Opus 5.5	6bc6a67fb0	README: the releases since v0.11.0 as released, and knowledge bases CI / test (push) Successful in 7m51s Details Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-08 17:17:40 +02:00