v0.14.1: changelog and README

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The firewall setting reaches the mirror machine and the store's members
2026-10-09 10:11:10 +02:00 · 2026-10-09 08:53:17 +02:00 · 2026-10-09 06:46:12 +02:00 · 2026-10-09 06:42:07 +02:00 · 2026-10-08 22:42:43 +02:00 · 2026-10-08 22:08:29 +02:00
83 changed files with 9271 additions and 230 deletions
@@ -569,9 +569,12 @@ portal is not done — the page shows; the console and ctl change.
    client tests against a fake admin API and the OpenAPI document;
    acceptance tests with Terraform and OpenTofu against the fake; one
    apply, empty plan, in-place change and destroy against a real `llmgw
-    server` run by hand. Left: organizations, teams, projects, people and
-    keys are not in the admin API (they are `llmgw org`/`users`/`keys`
-    against the store), so not in the provider; a model's definition beyond
+    server` run by hand. Organizations, teams, projects, keys and
+    memberships are in the admin API since 2026-10-08 (`/api/v1/orgs`,
+    `teams`, `projects`, `keys`, `users/{id}/memberships`) and in the
+    provider (`llmgw_org`, `llmgw_team`, `llmgw_project`, `llmgw_key`,
+    `llmgw_membership`); accounts are still made by sign-up, invitation or
+    OIDC/SCIM. Left: a model's definition beyond
    scaling, class, owner and adapters is changed by replacing it (the API
    has no edit, and `resources` is not used yet); `canary`, `slo`,
    `scaling.schedule` and `prewarm`, `guard` role and `origin` are not in
@@ -793,10 +796,19 @@ leader 30s later with /health answering throughout; rollback did the same.
  facility state from the store (it kept what it read at start).
 - Retires `appendGuardIfMissing` (server init): the import does it.

-### E2. Generated files from settings (L) — firewall on/off done 2026-10-06
+### E2. Generated files from settings (L) — firewall on/off done 2026-10-06, every machine 2026-10-09
 `firewall.enabled` in the cluster's settings reaches nodes over the agent
-stream and control planes through llmgw-firewall-sync.path. Left: the mirror
-machine and store members, the allowed set from membership, the other files.
+stream and control planes through llmgw-firewall-sync.path. Since
+2026-10-09 the mirror machine too (the leader PUTs /v1/firewall, the mirror
+writes `.firewall.json`, the same path unit applies it) and store members
+that run no llmgw (the leader keeps it in `cluster_state`, a root timer
+reads the member's own copy with psql); `firewall_networks` reach all of
+them, and the mirror lets in the network of every node and control plane
+the leader knows. Tried on the PoC: off and on, and a network added, on
+control planes, mirror, store member and nodes. Left: the allowed set
+replaced by membership rather than added to (a network taken out of the
+settings stays let in), the other files (the mirror's unit flags,
+keepalived, etcd and Patroni configuration).

 Every machine runs the agent (root) with roles — node, control plane, mirror,
 store member — and the agent renders and applies what was written once by
@@ -863,6 +875,12 @@ Built (`internal/store/knowledge.go`, migration 0035 PostgreSQL only):
  partial HNSW index per collection on its organization's partition
  (`embedding::vector(dims)`, `WHERE collection = '<id>'`). EXPLAIN on
  20k rows: partition pruned, that index used, policy applied.
+  Later (scale, 2026-10-08): a partial GIN per collection too
+  (`kb_tsv_<id>`, backfilled at open); `vec_type` per collection, `halfvec`
+  on pgvector ≥ 0.7 (up to 4000 dims); `SET LOCAL hnsw.ef_search` to
+  candidates(k) clamped 40..1000 (EXPLAIN: 40 rows without it, all with
+  it); the text arm takes candidates by `ts_rank` in a subquery and scores
+  only those.
 - RLS FORCEd on `kb_chunks` against `llmgw.kb_org`/`llmgw.kb_collections`
  (set with set_config per transaction), one role — the table owner — not
  a separate one. Chunks are deleted by the foreign-key cascade from their
@@ -1005,7 +1023,7 @@ a request without the header, `none` turns it off, an explicit header with
 k=2 gives two sources and no more, llmgw-kb lists owners by path. Left: returned passages
 are not audited per request (the header and the kept request record them).

-### F4. Adapter training (L, after F1–F3)
+### F4. Adapter training (L, after F1–F3) — done 2026-10-08 (branch `adapters-f4`)
 A run-to-completion job placed by the control plane like a replica and
 supervised by the agent like an engine, in a Python environment the agent
 installs (TRL + PEFT, QLoRA). Data: Q/A pairs generated from a collection by
@@ -1013,6 +1031,37 @@ a cluster model through `/v1/batches`, reviewed by people. Output to the
 mirror, into the model's `adapters`, through canary and a quality suite.
 Runs when the scaling schedule frees a GPU; 8–14B dense bases on one card.
 Never train on a collection some users of the adapter may not read.
+Tried 2026-10-08 on the PoC, by hand: Qwen2.5-0.5B-Instruct (fp16) with a
+LoRA (r=16, every linear layer, 8.8M trainable), trained beside the 35B in
+cuda01's last 2.3 GB (peak 2158 MiB on the card, 64 s for 138 steps, PEFT in a
+venv over the vLLM engine's torch). Data: 90 Turkish Q/A pairs a cluster
+model wrote from the nine Demo/politikalar documents, through the gateway.
+On nine questions it had not seen, phrased otherwise: base 0/9, 6 epochs
+4/9, 4 epochs at a lower rate 2/9. Wrong answers mixed facts across
+documents (the meal card's amount, the office day). The same nine through
+retrieval: 9/9 with sources. So F4 is for format, vocabulary and tone, and
+facts come from F1–F3; a bigger base on a free card would do better but not
+change that. The training itself fits a small slot on a shared card, which
+the job design can rely on for small bases.
+
+Built: `ctl kb dataset` (POST /api/v1/kb/collections/{id}/dataset, NDJSON
+with each pair's collection); train_jobs/train_artifacts (migrations 0037,
+0038 owner); internal/control/train (queue, one job per node, artifacts in
+1 MiB pieces, a node that goes fails its job); internal/agent/train (the
+newest vLLM's python3 + `uv pip install --no-deps --target` peft 0.21.2 and
+accelerate 1.15.0; llama.cpp's conversion code for the installed release,
+untarred from its source; the card with the most free memory, if enough;
+runs as llmgw-engine); `ctl train start|list|show --follow|cancel|rm|
+download|publish`; the mirror's PUT /v1/models/{model}/{version}/{file};
+publish uploads the GGUF (llama.cpp) or PEFT files (vLLM) and sets the
+adapter after checkAdapters. Owners: catalog.Adapter.Owner, enforced in the
+router (404, hidden from /v1/models); a job's owner defaults to its single
+collection's owner, must be able to read every collection the pairs came
+from, and pairs from a knowledge base are never public. Tried on the PoC:
+dataset of 90 pairs in 89 s; a job trained beside the 35B (peak 2214 MiB,
+138 steps in about a minute) and sent back PEFT and GGUF.
+Left: a job dies with its agent's stream (no resume); no console screen for
+training yet; quality gating of a published adapter is by hand (ctl eval).

 ---

@@ -18,7 +18,7 @@ is only released once its **exit criterion** is met, not when its code lands.
 | v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released |
 | v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released |
 | v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
-| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released |
+| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine) |
 | v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |

 Until v1.0.0 the minor version is the breaking one: configuration formats and Go
@@ -26,6 +26,31 @@ APIs change between minors without a deprecation period.

 ## Unreleased

+## v0.14.1 — 2026-10-09
+
+Adapters trained on the nodes from a knowledge base and kept private to
+their owners; the cluster's and the agents' settings edited in one place;
+the tenancy tree and keys over the admin API and in Terraform; knowledge
+bases that hold up with many documents; the firewall setting on every
+machine.
+
+**Upgrading.**
+- **Nodes** need this release to follow `agents.*` in the cluster's
+  settings; until a node is upgraded its agent.yaml is what it runs on.
+- **A separate mirror machine and store members that run nothing else**
+  follow the firewall setting once what reads it is installed: run `sudo
+  llmgw mirror init` on the mirror again with the flags it was set up with,
+  and `llmgw server init --patroni` again for the store's members. A control
+  plane set up before v0.12.0 needs `sudo llmgw firewall sync --install`.
+- **The store's PostgreSQL memory** applies after each member is restarted,
+  one at a time, the leader last; until then it runs as before.
+- The store gains tables and columns; nothing to do. Existing knowledge-base
+  collections get their full-text index when a control plane starts, and
+  keep `vector` storage; only new collections use `halfvec`, and only on
+  pgvector 0.7 or later.
+- Cached knowledge-base passages in vLLM are made again once, under the new
+  `cache_salt`.
+
 ### Added

 - **The cluster's settings on the portal** (BACKLOG E4): `/admin/settings`
@@ -41,6 +66,61 @@ APIs change between minors without a deprecation period.
  that would not load are refused on save, and set aside on a node rather
  than keep its agent from starting. Nodes need this release to follow
  them.
+- **LoRA adapters trained on the nodes** (BACKLOG F4): `llmgw ctl kb dataset`
+  writes question/answer pairs from a collection with a model of the
+  cluster. `llmgw ctl train start` trains an adapter on a node with its
+  installed vLLM's torch and a small overlay, on a card with enough memory
+  free (beside an engine, if it leaves that much), converting to GGUF for
+  llama.cpp. `llmgw ctl train publish` keeps it on the mirror (which now
+  accepts uploads) and makes it the model's adapter.
+- **Adapters with owners.** An adapter's `owner` (org, team, project, user
+  or key) is the only part of the tree that sees it; to anyone else it is a
+  model that does not exist. An adapter trained on a knowledge base belongs
+  to a part of the tree that may read it, and is never made public.
+- The knowledge bases' vLLM `cache_salt` follows the collections searched,
+  not the organization, so two teams of one organization no longer share
+  cached passages.
+- **Organizations, teams, projects, API keys and memberships in the admin
+  API**: `/api/v1/orgs` (the tree), `POST /api/v1/teams` and `projects`,
+  `PATCH` to rename or move, `DELETE` for what is empty; `/api/v1/keys` to
+  issue (the key is in the answer, once), list and revoke; `/api/v1/users`
+  and `/api/v1/users/{id}/memberships`. They were only `llmgw org`, `keys`
+  and `users` on a control plane before. The Terraform provider manages them
+  as `llmgw_org`, `llmgw_team`, `llmgw_project`, `llmgw_key` (the key in a
+  sensitive `secret`) and `llmgw_membership`.
+- **The firewall setting reaches every machine.** A separate mirror and the
+  store's members follow `firewall.enabled` and `firewall_networks` as nodes
+  and control planes did: the leading control plane tells the mirror over
+  its API, and keeps the state in the store for members that run no llmgw,
+  which a timer on each reads. The mirror also lets in the network of every
+  node the leader knows. `mirror init` and `server init --patroni` install
+  what follows it; run them again on a cluster set up before.
+
+### Changed
+
+- **The store's PostgreSQL is sized to its machine's memory.** Before, it ran
+  on PostgreSQL's defaults (128MB cache) under Patroni, or on 96MB on one
+  machine. Now each member gets a share of its own RAM, a smaller one where
+  a control plane or agent shares the machine, and the share is read again
+  every time `server init --patroni` runs. The new cache is applied by
+  restarting one member at a time, the leader last.
+- **Knowledge-base searches hold up with many documents.** New collections
+  store their index as `halfvec` where pgvector is 0.7 or later (half the
+  memory, and embeddings up to 4000 dimensions instead of 2000); older
+  collections, and clusters on pgvector 0.6, keep `vector`. Each collection
+  has its own full-text index, as it has its own vector index, made for
+  existing collections when the control plane starts. The text arm ranks by
+  the cheap measure first and scores only those candidates.
+
+### Fixed
+
+- A knowledge-base search asking for more than 10 passages
+  (`X-Llmgw-Knowledge-K`) got at most 40 candidates per collection from the
+  vector index, whatever it asked for: pgvector's `hnsw.ef_search` is now
+  raised to the number wanted, up to 1000.
+- `llmgw server backup` and `restore` against a replicated store over TLS
+  failed looking for `~/.postgresql/root.crt`: pg_dump was given
+  `sslmode=verify-full` without the cluster's authority.

 ## v0.14.0 — 2026-10-08

@@ -40,7 +40,7 @@ Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
 | v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released |
 | v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released |
 | v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
-| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released |
+| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine) |
 | v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |

 As of v0.6.0 a cluster runs unattended: agents report their hardware, the
@@ -50,6 +50,13 @@ budgets and rate limits decide who may use any of it.

 Since v0.11.0 (each release's detail is in [CHANGELOG.md](CHANGELOG.md)):

+- **Adapters, and the tenancy tree over the API** (v0.14.1). `llmgw ctl
+  kb dataset` and `llmgw ctl train` make a LoRA adapter from a collection
+  on a node's spare GPU memory, kept on the mirror and seen only by its
+  owner. Organizations, teams, projects, keys and memberships are in the
+  admin API and the Terraform provider. The agents' settings are edited
+  with the cluster's, on the portal too, and the firewall setting reaches
+  every machine.
 - **Knowledge bases** (v0.14.0). People put a company's documents in from
  their own machines with `llmgw-kb`, a separate download for Windows,
  macOS and Linux that reads PDF, Office, OpenDocument, mail and many more
@@ -106,6 +106,8 @@ type AgentMessage struct {
 	//	*AgentMessage_FabricTest
 	//	*AgentMessage_Upgrade
 	//	*AgentMessage_KvEvents
+	//	*AgentMessage_TrainStatus
+	//	*AgentMessage_TrainArtifact
 	Msg           isAgentMessage_Msg `protobuf_oneof:"msg"`
 	unknownFields protoimpl.UnknownFields
 	sizeCache     protoimpl.SizeCache
@@ -220,6 +222,24 @@ func (x *AgentMessage) GetKvEvents() *KVEvents {
 	return nil
 }

+func (x *AgentMessage) GetTrainStatus() *TrainStatus {
+	if x != nil {
+		if x, ok := x.Msg.(*AgentMessage_TrainStatus); ok {
+			return x.TrainStatus
+		}
+	}
+	return nil
+}
+
+func (x *AgentMessage) GetTrainArtifact() *TrainArtifact {
+	if x != nil {
+		if x, ok := x.Msg.(*AgentMessage_TrainArtifact); ok {
+			return x.TrainArtifact
+		}
+	}
+	return nil
+}
+
 type isAgentMessage_Msg interface {
 	isAgentMessage_Msg()
 }
@@ -256,6 +276,14 @@ type AgentMessage_KvEvents struct {
 	KvEvents *KVEvents `protobuf:"bytes,8,opt,name=kv_events,json=kvEvents,proto3,oneof"`
 }

+type AgentMessage_TrainStatus struct {
+	TrainStatus *TrainStatus `protobuf:"bytes,9,opt,name=train_status,json=trainStatus,proto3,oneof"`
+}
+
+type AgentMessage_TrainArtifact struct {
+	TrainArtifact *TrainArtifact `protobuf:"bytes,10,opt,name=train_artifact,json=trainArtifact,proto3,oneof"`
+}
+
 func (*AgentMessage_Register) isAgentMessage_Msg() {}

 func (*AgentMessage_Heartbeat) isAgentMessage_Msg() {}
@@ -272,6 +300,10 @@ func (*AgentMessage_Upgrade) isAgentMessage_Msg() {}

 func (*AgentMessage_KvEvents) isAgentMessage_Msg() {}

+func (*AgentMessage_TrainStatus) isAgentMessage_Msg() {}
+
+func (*AgentMessage_TrainArtifact) isAgentMessage_Msg() {}
+
 // KVEvents is a change to what one vLLM replica's prefix cache holds, read
 // by the agent from the engine's KV cache events and sent on for the router's
 // precise index (plan §7.5 tier 2).
@@ -392,6 +424,8 @@ type ControlMessage struct {
 	//	*ControlMessage_Firewall
 	//	*ControlMessage_RestartAgent
 	//	*ControlMessage_AgentSettings
+	//	*ControlMessage_TrainStart
+	//	*ControlMessage_TrainCancel
 	Msg isControlMessage_Msg `protobuf_oneof:"msg"`
 	// trace is the W3C trace context (traceparent, and tracestate when there
 	// is one) of the control plane's span that sent this command, so what the
@@ -566,6 +600,24 @@ func (x *ControlMessage) GetAgentSettings() *AgentSettings {
 	return nil
 }

+func (x *ControlMessage) GetTrainStart() *TrainStart {
+	if x != nil {
+		if x, ok := x.Msg.(*ControlMessage_TrainStart); ok {
+			return x.TrainStart
+		}
+	}
+	return nil
+}
+
+func (x *ControlMessage) GetTrainCancel() *TrainCancel {
+	if x != nil {
+		if x, ok := x.Msg.(*ControlMessage_TrainCancel); ok {
+			return x.TrainCancel
+		}
+	}
+	return nil
+}
+
 func (x *ControlMessage) GetTrace() map[string]string {
 	if x != nil {
 		return x.Trace
@@ -633,6 +685,14 @@ type ControlMessage_AgentSettings struct {
 	AgentSettings *AgentSettings `protobuf:"bytes,15,opt,name=agent_settings,json=agentSettings,proto3,oneof"`
 }

+type ControlMessage_TrainStart struct {
+	TrainStart *TrainStart `protobuf:"bytes,16,opt,name=train_start,json=trainStart,proto3,oneof"`
+}
+
+type ControlMessage_TrainCancel struct {
+	TrainCancel *TrainCancel `protobuf:"bytes,17,opt,name=train_cancel,json=trainCancel,proto3,oneof"`
+}
+
 func (*ControlMessage_Start) isControlMessage_Msg() {}

 func (*ControlMessage_Stop) isControlMessage_Msg() {}
@@ -661,6 +721,10 @@ func (*ControlMessage_RestartAgent) isControlMessage_Msg() {}

 func (*ControlMessage_AgentSettings) isControlMessage_Msg() {}

+func (*ControlMessage_TrainStart) isControlMessage_Msg() {}
+
+func (*ControlMessage_TrainCancel) isControlMessage_Msg() {}
+
 // RestartAgent has the agent start again, as llmgw restart asks of every
 // node: its engines keep serving and are taken over again.
 type RestartAgent struct {
@@ -4699,11 +4763,335 @@ func (x *LoraAdapter) GetSource() string {
 	return ""
 }

+// TrainStart has a node train a LoRA adapter (BACKLOG F4): on a base model's
+// weights from its registry, on question/answer pairs, with a Python trainer
+// the agent carries, in an environment it makes over an installed engine's.
+// One at a time per node.
+type TrainStart struct {
+	state protoimpl.MessageState `protogen:"open.v1"`
+	JobId string                 `protobuf:"bytes,1,opt,name=job_id,json=jobId,proto3" json:"job_id,omitempty"`
+	// base_repo and base_revision name the base model's weights on its
+	// registry (Hugging Face): safetensors, not the GGUF a catalog may serve.
+	BaseRepo     string `protobuf:"bytes,2,opt,name=base_repo,json=baseRepo,proto3" json:"base_repo,omitempty"`
+	BaseRevision string `protobuf:"bytes,3,opt,name=base_revision,json=baseRevision,proto3" json:"base_revision,omitempty"`
+	// data is the training pairs, NDJSON of {"q", "a"}.
+	Data []byte `protobuf:"bytes,4,opt,name=data,proto3" json:"data,omitempty"`
+	// params is the trainer's settings as JSON: rank, alpha, epochs, lr,
+	// batch, max_len.
+	Params string `protobuf:"bytes,5,opt,name=params,proto3" json:"params,omitempty"`
+	// gpu_memory_mib is what the job needs of a card; the agent starts it
+	// only when one has that much free, which lets it run beside an engine.
+	GpuMemoryMib int32 `protobuf:"varint,6,opt,name=gpu_memory_mib,json=gpuMemoryMib,proto3" json:"gpu_memory_mib,omitempty"`
+	// gguf asks for the adapter in llama.cpp's format too.
+	Gguf          bool `protobuf:"varint,7,opt,name=gguf,proto3" json:"gguf,omitempty"`
+	unknownFields protoimpl.UnknownFields
+	sizeCache     protoimpl.SizeCache
+}
+
+func (x *TrainStart) Reset() {
+	*x = TrainStart{}
+	mi := &file_llmgw_v1_agent_proto_msgTypes[44]
+	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+	ms.StoreMessageInfo(mi)
+}
+
+func (x *TrainStart) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*TrainStart) ProtoMessage() {}
+
+func (x *TrainStart) ProtoReflect() protoreflect.Message {
+	mi := &file_llmgw_v1_agent_proto_msgTypes[44]
+	if x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use TrainStart.ProtoReflect.Descriptor instead.
+func (*TrainStart) Descriptor() ([]byte, []int) {
+	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{44}
+}
+
+func (x *TrainStart) GetJobId() string {
+	if x != nil {
+		return x.JobId
+	}
+	return ""
+}
+
+func (x *TrainStart) GetBaseRepo() string {
+	if x != nil {
+		return x.BaseRepo
+	}
+	return ""
+}
+
+func (x *TrainStart) GetBaseRevision() string {
+	if x != nil {
+		return x.BaseRevision
+	}
+	return ""
+}
+
+func (x *TrainStart) GetData() []byte {
+	if x != nil {
+		return x.Data
+	}
+	return nil
+}
+
+func (x *TrainStart) GetParams() string {
+	if x != nil {
+		return x.Params
+	}
+	return ""
+}
+
+func (x *TrainStart) GetGpuMemoryMib() int32 {
+	if x != nil {
+		return x.GpuMemoryMib
+	}
+	return 0
+}
+
+func (x *TrainStart) GetGguf() bool {
+	if x != nil {
+		return x.Gguf
+	}
+	return false
+}
+
+// TrainCancel stops a training job.
+type TrainCancel struct {
+	state         protoimpl.MessageState `protogen:"open.v1"`
+	JobId         string                 `protobuf:"bytes,1,opt,name=job_id,json=jobId,proto3" json:"job_id,omitempty"`
+	unknownFields protoimpl.UnknownFields
+	sizeCache     protoimpl.SizeCache
+}
+
+func (x *TrainCancel) Reset() {
+	*x = TrainCancel{}
+	mi := &file_llmgw_v1_agent_proto_msgTypes[45]
+	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+	ms.StoreMessageInfo(mi)
+}
+
+func (x *TrainCancel) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*TrainCancel) ProtoMessage() {}
+
+func (x *TrainCancel) ProtoReflect() protoreflect.Message {
+	mi := &file_llmgw_v1_agent_proto_msgTypes[45]
+	if x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use TrainCancel.ProtoReflect.Descriptor instead.
+func (*TrainCancel) Descriptor() ([]byte, []int) {
+	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{45}
+}
+
+func (x *TrainCancel) GetJobId() string {
+	if x != nil {
+		return x.JobId
+	}
+	return ""
+}
+
+// TrainStatus is how a training job is going, sent as it changes.
+type TrainStatus struct {
+	state protoimpl.MessageState `protogen:"open.v1"`
+	JobId string                 `protobuf:"bytes,1,opt,name=job_id,json=jobId,proto3" json:"job_id,omitempty"`
+	// state is preparing, running, done or failed.
+	State string  `protobuf:"bytes,2,opt,name=state,proto3" json:"state,omitempty"`
+	Step  int32   `protobuf:"varint,3,opt,name=step,proto3" json:"step,omitempty"`
+	Steps int32   `protobuf:"varint,4,opt,name=steps,proto3" json:"steps,omitempty"`
+	Loss  float64 `protobuf:"fixed64,5,opt,name=loss,proto3" json:"loss,omitempty"`
+	// message is what is being done, or why it failed.
+	Message string `protobuf:"bytes,6,opt,name=message,proto3" json:"message,omitempty"`
+	// peak_mib is the most the job held of the card.
+	PeakMib       int32 `protobuf:"varint,7,opt,name=peak_mib,json=peakMib,proto3" json:"peak_mib,omitempty"`
+	unknownFields protoimpl.UnknownFields
+	sizeCache     protoimpl.SizeCache
+}
+
+func (x *TrainStatus) Reset() {
+	*x = TrainStatus{}
+	mi := &file_llmgw_v1_agent_proto_msgTypes[46]
+	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+	ms.StoreMessageInfo(mi)
+}
+
+func (x *TrainStatus) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*TrainStatus) ProtoMessage() {}
+
+func (x *TrainStatus) ProtoReflect() protoreflect.Message {
+	mi := &file_llmgw_v1_agent_proto_msgTypes[46]
+	if x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use TrainStatus.ProtoReflect.Descriptor instead.
+func (*TrainStatus) Descriptor() ([]byte, []int) {
+	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{46}
+}
+
+func (x *TrainStatus) GetJobId() string {
+	if x != nil {
+		return x.JobId
+	}
+	return ""
+}
+
+func (x *TrainStatus) GetState() string {
+	if x != nil {
+		return x.State
+	}
+	return ""
+}
+
+func (x *TrainStatus) GetStep() int32 {
+	if x != nil {
+		return x.Step
+	}
+	return 0
+}
+
+func (x *TrainStatus) GetSteps() int32 {
+	if x != nil {
+		return x.Steps
+	}
+	return 0
+}
+
+func (x *TrainStatus) GetLoss() float64 {
+	if x != nil {
+		return x.Loss
+	}
+	return 0
+}
+
+func (x *TrainStatus) GetMessage() string {
+	if x != nil {
+		return x.Message
+	}
+	return ""
+}
+
+func (x *TrainStatus) GetPeakMib() int32 {
+	if x != nil {
+		return x.PeakMib
+	}
+	return 0
+}
+
+// TrainArtifact is a piece of what a training job made, sent before its
+// final status: a gzipped tar of the PEFT adapter (kind peft), or the GGUF
+// adapter (kind gguf).
+type TrainArtifact struct {
+	state         protoimpl.MessageState `protogen:"open.v1"`
+	JobId         string                 `protobuf:"bytes,1,opt,name=job_id,json=jobId,proto3" json:"job_id,omitempty"`
+	Kind          string                 `protobuf:"bytes,2,opt,name=kind,proto3" json:"kind,omitempty"`
+	Seq           int32                  `protobuf:"varint,3,opt,name=seq,proto3" json:"seq,omitempty"`
+	Data          []byte                 `protobuf:"bytes,4,opt,name=data,proto3" json:"data,omitempty"`
+	Last          bool                   `protobuf:"varint,5,opt,name=last,proto3" json:"last,omitempty"`
+	unknownFields protoimpl.UnknownFields
+	sizeCache     protoimpl.SizeCache
+}
+
+func (x *TrainArtifact) Reset() {
+	*x = TrainArtifact{}
+	mi := &file_llmgw_v1_agent_proto_msgTypes[47]
+	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+	ms.StoreMessageInfo(mi)
+}
+
+func (x *TrainArtifact) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*TrainArtifact) ProtoMessage() {}
+
+func (x *TrainArtifact) ProtoReflect() protoreflect.Message {
+	mi := &file_llmgw_v1_agent_proto_msgTypes[47]
+	if x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use TrainArtifact.ProtoReflect.Descriptor instead.
+func (*TrainArtifact) Descriptor() ([]byte, []int) {
+	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{47}
+}
+
+func (x *TrainArtifact) GetJobId() string {
+	if x != nil {
+		return x.JobId
+	}
+	return ""
+}
+
+func (x *TrainArtifact) GetKind() string {
+	if x != nil {
+		return x.Kind
+	}
+	return ""
+}
+
+func (x *TrainArtifact) GetSeq() int32 {
+	if x != nil {
+		return x.Seq
+	}
+	return 0
+}
+
+func (x *TrainArtifact) GetData() []byte {
+	if x != nil {
+		return x.Data
+	}
+	return nil
+}
+
+func (x *TrainArtifact) GetLast() bool {
+	if x != nil {
+		return x.Last
+	}
+	return false
+}
+
 var File_llmgw_v1_agent_proto protoreflect.FileDescriptor

 const file_llmgw_v1_agent_proto_rawDesc = "" +
 	"\n" +
-	"\x14llmgw/v1/agent.proto\x12\bllmgw.v1\"\xb9\x03\n" +
+	"\x14llmgw/v1/agent.proto\x12\bllmgw.v1\"\xb7\x04\n" +
 	"\fAgentMessage\x120\n" +
 	"\bregister\x18\x01 \x01(\v2\x12.llmgw.v1.RegisterH\x00R\bregister\x123\n" +
 	"\theartbeat\x18\x02 \x01(\v2\x13.llmgw.v1.HeartbeatH\x00R\theartbeat\x12-\n" +
@@ -4714,7 +5102,10 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
 	"\vfabric_test\x18\x06 \x01(\v2\x19.llmgw.v1.FabricTestReplyH\x00R\n" +
 	"fabricTest\x122\n" +
 	"\aupgrade\x18\a \x01(\v2\x16.llmgw.v1.UpgradeReplyH\x00R\aupgrade\x121\n" +
-	"\tkv_events\x18\b \x01(\v2\x12.llmgw.v1.KVEventsH\x00R\bkvEventsB\x05\n" +
+	"\tkv_events\x18\b \x01(\v2\x12.llmgw.v1.KVEventsH\x00R\bkvEvents\x12:\n" +
+	"\ftrain_status\x18\t \x01(\v2\x15.llmgw.v1.TrainStatusH\x00R\vtrainStatus\x12@\n" +
+	"\x0etrain_artifact\x18\n" +
+	" \x01(\v2\x17.llmgw.v1.TrainArtifactH\x00R\rtrainArtifactB\x05\n" +
 	"\x03msg\"\xb5\x01\n" +
 	"\bKVEvents\x12\x1d\n" +
 	"\n" +
@@ -4725,7 +5116,7 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
 	"clearFirst\x12\x16\n" +
 	"\x06stored\x18\x04 \x03(\x06R\x06stored\x12\x18\n" +
 	"\aremoved\x18\x05 \x03(\x06R\aremoved\x12\x18\n" +
-	"\aproblem\x18\x06 \x01(\tR\aproblem\"\x83\a\n" +
+	"\aproblem\x18\x06 \x01(\tR\aproblem\"\xf8\a\n" +
 	"\x0eControlMessage\x12,\n" +
 	"\x05start\x18\x01 \x01(\v2\x14.llmgw.v1.StartShardH\x00R\x05start\x12)\n" +
 	"\x04stop\x18\x02 \x01(\v2\x13.llmgw.v1.StopShardH\x00R\x04stop\x12,\n" +
@@ -4743,7 +5134,10 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
 	"\x0econtrol_planes\x18\v \x01(\v2\x17.llmgw.v1.ControlPlanesH\x00R\rcontrolPlanes\x120\n" +
 	"\bfirewall\x18\r \x01(\v2\x12.llmgw.v1.FirewallH\x00R\bfirewall\x12=\n" +
 	"\rrestart_agent\x18\x0e \x01(\v2\x16.llmgw.v1.RestartAgentH\x00R\frestartAgent\x12@\n" +
-	"\x0eagent_settings\x18\x0f \x01(\v2\x17.llmgw.v1.AgentSettingsH\x00R\ragentSettings\x129\n" +
+	"\x0eagent_settings\x18\x0f \x01(\v2\x17.llmgw.v1.AgentSettingsH\x00R\ragentSettings\x127\n" +
+	"\vtrain_start\x18\x10 \x01(\v2\x14.llmgw.v1.TrainStartH\x00R\n" +
+	"trainStart\x12:\n" +
+	"\ftrain_cancel\x18\x11 \x01(\v2\x15.llmgw.v1.TrainCancelH\x00R\vtrainCancel\x129\n" +
 	"\x05trace\x18\f \x03(\v2#.llmgw.v1.ControlMessage.TraceEntryR\x05trace\x1a8\n" +
 	"\n" +
 	"TraceEntry\x12\x10\n" +
@@ -5150,7 +5544,32 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
 	"\x05error\x18\x03 \x01(\tR\x05error\"9\n" +
 	"\vLoraAdapter\x12\x12\n" +
 	"\x04name\x18\x01 \x01(\tR\x04name\x12\x16\n" +
-	"\x06source\x18\x02 \x01(\tR\x06source2O\n" +
+	"\x06source\x18\x02 \x01(\tR\x06source\"\xcb\x01\n" +
+	"\n" +
+	"TrainStart\x12\x15\n" +
+	"\x06job_id\x18\x01 \x01(\tR\x05jobId\x12\x1b\n" +
+	"\tbase_repo\x18\x02 \x01(\tR\bbaseRepo\x12#\n" +
+	"\rbase_revision\x18\x03 \x01(\tR\fbaseRevision\x12\x12\n" +
+	"\x04data\x18\x04 \x01(\fR\x04data\x12\x16\n" +
+	"\x06params\x18\x05 \x01(\tR\x06params\x12$\n" +
+	"\x0egpu_memory_mib\x18\x06 \x01(\x05R\fgpuMemoryMib\x12\x12\n" +
+	"\x04gguf\x18\a \x01(\bR\x04gguf\"$\n" +
+	"\vTrainCancel\x12\x15\n" +
+	"\x06job_id\x18\x01 \x01(\tR\x05jobId\"\xad\x01\n" +
+	"\vTrainStatus\x12\x15\n" +
+	"\x06job_id\x18\x01 \x01(\tR\x05jobId\x12\x14\n" +
+	"\x05state\x18\x02 \x01(\tR\x05state\x12\x12\n" +
+	"\x04step\x18\x03 \x01(\x05R\x04step\x12\x14\n" +
+	"\x05steps\x18\x04 \x01(\x05R\x05steps\x12\x12\n" +
+	"\x04loss\x18\x05 \x01(\x01R\x04loss\x12\x18\n" +
+	"\amessage\x18\x06 \x01(\tR\amessage\x12\x19\n" +
+	"\bpeak_mib\x18\a \x01(\x05R\apeakMib\"t\n" +
+	"\rTrainArtifact\x12\x15\n" +
+	"\x06job_id\x18\x01 \x01(\tR\x05jobId\x12\x12\n" +
+	"\x04kind\x18\x02 \x01(\tR\x04kind\x12\x10\n" +
+	"\x03seq\x18\x03 \x01(\x05R\x03seq\x12\x12\n" +
+	"\x04data\x18\x04 \x01(\fR\x04data\x12\x12\n" +
+	"\x04last\x18\x05 \x01(\bR\x04last2O\n" +
 	"\fAgentService\x12?\n" +
 	"\aConnect\x12\x16.llmgw.v1.AgentMessage\x1a\x18.llmgw.v1.ControlMessage(\x010\x01B\x93\x01\n" +
 	"\fcom.llmgw.v1B\n" +
@@ -5169,7 +5588,7 @@ func file_llmgw_v1_agent_proto_rawDescGZIP() []byte {
 }

 var file_llmgw_v1_agent_proto_enumTypes = make([]protoimpl.EnumInfo, 1)
-var file_llmgw_v1_agent_proto_msgTypes = make([]protoimpl.MessageInfo, 52)
+var file_llmgw_v1_agent_proto_msgTypes = make([]protoimpl.MessageInfo, 56)
 var file_llmgw_v1_agent_proto_goTypes = []any{
 	(ShardStatus_State)(0),    // 0: llmgw.v1.ShardStatus.State
 	(*AgentMessage)(nil),      // 1: llmgw.v1.AgentMessage
@@ -5216,14 +5635,18 @@ var file_llmgw_v1_agent_proto_goTypes = []any{
 	(*UpgradeAgent)(nil),      // 42: llmgw.v1.UpgradeAgent
 	(*UpgradeReply)(nil),      // 43: llmgw.v1.UpgradeReply
 	(*LoraAdapter)(nil),       // 44: llmgw.v1.LoraAdapter
-	nil,                       // 45: llmgw.v1.ControlMessage.TraceEntry
-	nil,                       // 46: llmgw.v1.Register.LabelsEntry
-	nil,                       // 47: llmgw.v1.Register.EngineVersionsEntry
-	nil,                       // 48: llmgw.v1.GPU.NicPathsEntry
-	nil,                       // 49: llmgw.v1.IBPort.CountersEntry
-	nil,                       // 50: llmgw.v1.Heartbeat.EngineVersionsEntry
-	nil,                       // 51: llmgw.v1.EngineWant.OfflineSha256Entry
-	nil,                       // 52: llmgw.v1.EngineProfile.EnvEntry
+	(*TrainStart)(nil),        // 45: llmgw.v1.TrainStart
+	(*TrainCancel)(nil),       // 46: llmgw.v1.TrainCancel
+	(*TrainStatus)(nil),       // 47: llmgw.v1.TrainStatus
+	(*TrainArtifact)(nil),     // 48: llmgw.v1.TrainArtifact
+	nil,                       // 49: llmgw.v1.ControlMessage.TraceEntry
+	nil,                       // 50: llmgw.v1.Register.LabelsEntry
+	nil,                       // 51: llmgw.v1.Register.EngineVersionsEntry
+	nil,                       // 52: llmgw.v1.GPU.NicPathsEntry
+	nil,                       // 53: llmgw.v1.IBPort.CountersEntry
+	nil,                       // 54: llmgw.v1.Heartbeat.EngineVersionsEntry
+	nil,                       // 55: llmgw.v1.EngineWant.OfflineSha256Entry
+	nil,                       // 56: llmgw.v1.EngineProfile.EnvEntry
 }
 var file_llmgw_v1_agent_proto_depIdxs = []int32{
 	8,  // 0: llmgw.v1.AgentMessage.register:type_name -> llmgw.v1.Register
@@ -5234,62 +5657,66 @@ var file_llmgw_v1_agent_proto_depIdxs = []int32{
 	41, // 5: llmgw.v1.AgentMessage.fabric_test:type_name -> llmgw.v1.FabricTestReply
 	43, // 6: llmgw.v1.AgentMessage.upgrade:type_name -> llmgw.v1.UpgradeReply
 	2,  // 7: llmgw.v1.AgentMessage.kv_events:type_name -> llmgw.v1.KVEvents
-	30, // 8: llmgw.v1.ControlMessage.start:type_name -> llmgw.v1.StartShard
-	33, // 9: llmgw.v1.ControlMessage.stop:type_name -> llmgw.v1.StopShard
-	34, // 10: llmgw.v1.ControlMessage.drain:type_name -> llmgw.v1.DrainShard
-	35, // 11: llmgw.v1.ControlMessage.ack:type_name -> llmgw.v1.Ack
-	32, // 12: llmgw.v1.ControlMessage.remove_weights:type_name -> llmgw.v1.RemoveWeights
-	24, // 13: llmgw.v1.ControlMessage.provision:type_name -> llmgw.v1.Provision
-	36, // 14: llmgw.v1.ControlMessage.logs:type_name -> llmgw.v1.LogsRequest
-	38, // 15: llmgw.v1.ControlMessage.fabric_scan:type_name -> llmgw.v1.FabricScanRequest
-	40, // 16: llmgw.v1.ControlMessage.fabric_test:type_name -> llmgw.v1.FabricTestRequest
-	42, // 17: llmgw.v1.ControlMessage.upgrade:type_name -> llmgw.v1.UpgradeAgent
-	7,  // 18: llmgw.v1.ControlMessage.control_planes:type_name -> llmgw.v1.ControlPlanes
-	5,  // 19: llmgw.v1.ControlMessage.firewall:type_name -> llmgw.v1.Firewall
-	4,  // 20: llmgw.v1.ControlMessage.restart_agent:type_name -> llmgw.v1.RestartAgent
-	6,  // 21: llmgw.v1.ControlMessage.agent_settings:type_name -> llmgw.v1.AgentSettings
-	45, // 22: llmgw.v1.ControlMessage.trace:type_name -> llmgw.v1.ControlMessage.TraceEntry
-	46, // 23: llmgw.v1.Register.labels:type_name -> llmgw.v1.Register.LabelsEntry
-	12, // 24: llmgw.v1.Register.gpus:type_name -> llmgw.v1.GPU
-	14, // 25: llmgw.v1.Register.fabrics:type_name -> llmgw.v1.FabricAttachment
-	47, // 26: llmgw.v1.Register.engine_versions:type_name -> llmgw.v1.Register.EngineVersionsEntry
-	11, // 27: llmgw.v1.Register.running_shards:type_name -> llmgw.v1.RunningShard
-	15, // 28: llmgw.v1.Register.weights:type_name -> llmgw.v1.LocalWeights
-	10, // 29: llmgw.v1.Register.storage:type_name -> llmgw.v1.Storage
-	9,  // 30: llmgw.v1.Register.host:type_name -> llmgw.v1.Host
-	21, // 31: llmgw.v1.Register.imex:type_name -> llmgw.v1.IMEX
-	0,  // 32: llmgw.v1.RunningShard.state:type_name -> llmgw.v1.ShardStatus.State
-	27, // 33: llmgw.v1.RunningShard.compute:type_name -> llmgw.v1.Compute
-	48, // 34: llmgw.v1.GPU.nic_paths:type_name -> llmgw.v1.GPU.NicPathsEntry
-	13, // 35: llmgw.v1.GPU.mig:type_name -> llmgw.v1.MIGDevice
-	18, // 36: llmgw.v1.NodeLoad.shards:type_name -> llmgw.v1.ShardLoad
-	17, // 37: llmgw.v1.NodeLoad.gpus:type_name -> llmgw.v1.GPUSensor
-	49, // 38: llmgw.v1.IBPort.counters:type_name -> llmgw.v1.IBPort.CountersEntry
-	20, // 39: llmgw.v1.IBPort.subnet_managers:type_name -> llmgw.v1.SubnetManager
-	10, // 40: llmgw.v1.Heartbeat.storage:type_name -> llmgw.v1.Storage
-	15, // 41: llmgw.v1.Heartbeat.weights:type_name -> llmgw.v1.LocalWeights
-	50, // 42: llmgw.v1.Heartbeat.engine_versions:type_name -> llmgw.v1.Heartbeat.EngineVersionsEntry
-	26, // 43: llmgw.v1.Heartbeat.provision:type_name -> llmgw.v1.ProvisionStatus
-	16, // 44: llmgw.v1.Heartbeat.load:type_name -> llmgw.v1.NodeLoad
-	19, // 45: llmgw.v1.Heartbeat.ib_ports:type_name -> llmgw.v1.IBPort
-	14, // 46: llmgw.v1.Heartbeat.fabrics:type_name -> llmgw.v1.FabricAttachment
-	22, // 47: llmgw.v1.Heartbeat.gpu_fabric:type_name -> llmgw.v1.GPUFabricState
-	21, // 48: llmgw.v1.Heartbeat.imex:type_name -> llmgw.v1.IMEX
-	25, // 49: llmgw.v1.Provision.engines:type_name -> llmgw.v1.EngineWant
-	51, // 50: llmgw.v1.EngineWant.offline_sha256:type_name -> llmgw.v1.EngineWant.OfflineSha256Entry
-	0,  // 51: llmgw.v1.ShardStatus.state:type_name -> llmgw.v1.ShardStatus.State
-	29, // 52: llmgw.v1.ShardStatus.fetch:type_name -> llmgw.v1.FetchProgress
-	27, // 53: llmgw.v1.ShardStatus.compute:type_name -> llmgw.v1.Compute
-	31, // 54: llmgw.v1.StartShard.profile:type_name -> llmgw.v1.EngineProfile
-	52, // 55: llmgw.v1.EngineProfile.env:type_name -> llmgw.v1.EngineProfile.EnvEntry
-	44, // 56: llmgw.v1.EngineProfile.adapters:type_name -> llmgw.v1.LoraAdapter
-	1,  // 57: llmgw.v1.AgentService.Connect:input_type -> llmgw.v1.AgentMessage
-	3,  // 58: llmgw.v1.AgentService.Connect:output_type -> llmgw.v1.ControlMessage
-	58, // [58:59] is the sub-list for method output_type
-	57, // [57:58] is the sub-list for method input_type
-	57, // [57:57] is the sub-list for extension type_name
-	57, // [57:57] is the sub-list for extension extendee
-	0,  // [0:57] is the sub-list for field type_name
+	47, // 8: llmgw.v1.AgentMessage.train_status:type_name -> llmgw.v1.TrainStatus
+	48, // 9: llmgw.v1.AgentMessage.train_artifact:type_name -> llmgw.v1.TrainArtifact
+	30, // 10: llmgw.v1.ControlMessage.start:type_name -> llmgw.v1.StartShard
+	33, // 11: llmgw.v1.ControlMessage.stop:type_name -> llmgw.v1.StopShard
+	34, // 12: llmgw.v1.ControlMessage.drain:type_name -> llmgw.v1.DrainShard
+	35, // 13: llmgw.v1.ControlMessage.ack:type_name -> llmgw.v1.Ack
+	32, // 14: llmgw.v1.ControlMessage.remove_weights:type_name -> llmgw.v1.RemoveWeights
+	24, // 15: llmgw.v1.ControlMessage.provision:type_name -> llmgw.v1.Provision
+	36, // 16: llmgw.v1.ControlMessage.logs:type_name -> llmgw.v1.LogsRequest
+	38, // 17: llmgw.v1.ControlMessage.fabric_scan:type_name -> llmgw.v1.FabricScanRequest
+	40, // 18: llmgw.v1.ControlMessage.fabric_test:type_name -> llmgw.v1.FabricTestRequest
+	42, // 19: llmgw.v1.ControlMessage.upgrade:type_name -> llmgw.v1.UpgradeAgent
+	7,  // 20: llmgw.v1.ControlMessage.control_planes:type_name -> llmgw.v1.ControlPlanes
+	5,  // 21: llmgw.v1.ControlMessage.firewall:type_name -> llmgw.v1.Firewall
+	4,  // 22: llmgw.v1.ControlMessage.restart_agent:type_name -> llmgw.v1.RestartAgent
+	6,  // 23: llmgw.v1.ControlMessage.agent_settings:type_name -> llmgw.v1.AgentSettings
+	45, // 24: llmgw.v1.ControlMessage.train_start:type_name -> llmgw.v1.TrainStart
+	46, // 25: llmgw.v1.ControlMessage.train_cancel:type_name -> llmgw.v1.TrainCancel
+	49, // 26: llmgw.v1.ControlMessage.trace:type_name -> llmgw.v1.ControlMessage.TraceEntry
+	50, // 27: llmgw.v1.Register.labels:type_name -> llmgw.v1.Register.LabelsEntry
+	12, // 28: llmgw.v1.Register.gpus:type_name -> llmgw.v1.GPU
+	14, // 29: llmgw.v1.Register.fabrics:type_name -> llmgw.v1.FabricAttachment
+	51, // 30: llmgw.v1.Register.engine_versions:type_name -> llmgw.v1.Register.EngineVersionsEntry
+	11, // 31: llmgw.v1.Register.running_shards:type_name -> llmgw.v1.RunningShard
+	15, // 32: llmgw.v1.Register.weights:type_name -> llmgw.v1.LocalWeights
+	10, // 33: llmgw.v1.Register.storage:type_name -> llmgw.v1.Storage
+	9,  // 34: llmgw.v1.Register.host:type_name -> llmgw.v1.Host
+	21, // 35: llmgw.v1.Register.imex:type_name -> llmgw.v1.IMEX
+	0,  // 36: llmgw.v1.RunningShard.state:type_name -> llmgw.v1.ShardStatus.State
+	27, // 37: llmgw.v1.RunningShard.compute:type_name -> llmgw.v1.Compute
+	52, // 38: llmgw.v1.GPU.nic_paths:type_name -> llmgw.v1.GPU.NicPathsEntry
+	13, // 39: llmgw.v1.GPU.mig:type_name -> llmgw.v1.MIGDevice
+	18, // 40: llmgw.v1.NodeLoad.shards:type_name -> llmgw.v1.ShardLoad
+	17, // 41: llmgw.v1.NodeLoad.gpus:type_name -> llmgw.v1.GPUSensor
+	53, // 42: llmgw.v1.IBPort.counters:type_name -> llmgw.v1.IBPort.CountersEntry
+	20, // 43: llmgw.v1.IBPort.subnet_managers:type_name -> llmgw.v1.SubnetManager
+	10, // 44: llmgw.v1.Heartbeat.storage:type_name -> llmgw.v1.Storage
+	15, // 45: llmgw.v1.Heartbeat.weights:type_name -> llmgw.v1.LocalWeights
+	54, // 46: llmgw.v1.Heartbeat.engine_versions:type_name -> llmgw.v1.Heartbeat.EngineVersionsEntry
+	26, // 47: llmgw.v1.Heartbeat.provision:type_name -> llmgw.v1.ProvisionStatus
+	16, // 48: llmgw.v1.Heartbeat.load:type_name -> llmgw.v1.NodeLoad
+	19, // 49: llmgw.v1.Heartbeat.ib_ports:type_name -> llmgw.v1.IBPort
+	14, // 50: llmgw.v1.Heartbeat.fabrics:type_name -> llmgw.v1.FabricAttachment
+	22, // 51: llmgw.v1.Heartbeat.gpu_fabric:type_name -> llmgw.v1.GPUFabricState
+	21, // 52: llmgw.v1.Heartbeat.imex:type_name -> llmgw.v1.IMEX
+	25, // 53: llmgw.v1.Provision.engines:type_name -> llmgw.v1.EngineWant
+	55, // 54: llmgw.v1.EngineWant.offline_sha256:type_name -> llmgw.v1.EngineWant.OfflineSha256Entry
+	0,  // 55: llmgw.v1.ShardStatus.state:type_name -> llmgw.v1.ShardStatus.State
+	29, // 56: llmgw.v1.ShardStatus.fetch:type_name -> llmgw.v1.FetchProgress
+	27, // 57: llmgw.v1.ShardStatus.compute:type_name -> llmgw.v1.Compute
+	31, // 58: llmgw.v1.StartShard.profile:type_name -> llmgw.v1.EngineProfile
+	56, // 59: llmgw.v1.EngineProfile.env:type_name -> llmgw.v1.EngineProfile.EnvEntry
+	44, // 60: llmgw.v1.EngineProfile.adapters:type_name -> llmgw.v1.LoraAdapter
+	1,  // 61: llmgw.v1.AgentService.Connect:input_type -> llmgw.v1.AgentMessage
+	3,  // 62: llmgw.v1.AgentService.Connect:output_type -> llmgw.v1.ControlMessage
+	62, // [62:63] is the sub-list for method output_type
+	61, // [61:62] is the sub-list for method input_type
+	61, // [61:61] is the sub-list for extension type_name
+	61, // [61:61] is the sub-list for extension extendee
+	0,  // [0:61] is the sub-list for field type_name
 }

 func init() { file_llmgw_v1_agent_proto_init() }
@@ -5306,6 +5733,8 @@ func file_llmgw_v1_agent_proto_init() {
 		(*AgentMessage_FabricTest)(nil),
 		(*AgentMessage_Upgrade)(nil),
 		(*AgentMessage_KvEvents)(nil),
+		(*AgentMessage_TrainStatus)(nil),
+		(*AgentMessage_TrainArtifact)(nil),
 	}
 	file_llmgw_v1_agent_proto_msgTypes[2].OneofWrappers = []any{
 		(*ControlMessage_Start)(nil),
@@ -5322,6 +5751,8 @@ func file_llmgw_v1_agent_proto_init() {
 		(*ControlMessage_Firewall)(nil),
 		(*ControlMessage_RestartAgent)(nil),
 		(*ControlMessage_AgentSettings)(nil),
+		(*ControlMessage_TrainStart)(nil),
+		(*ControlMessage_TrainCancel)(nil),
 	}
 	type x struct{}
 	out := protoimpl.TypeBuilder{
@@ -5329,7 +5760,7 @@ func file_llmgw_v1_agent_proto_init() {
 			GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
 			RawDescriptor: unsafe.Slice(unsafe.StringData(file_llmgw_v1_agent_proto_rawDesc), len(file_llmgw_v1_agent_proto_rawDesc)),
 			NumEnums:      1,
-			NumMessages:   52,
+			NumMessages:   56,
 			NumExtensions: 0,
 			NumServices:   1,
 		},
@@ -28,6 +28,8 @@ message AgentMessage {
    FabricTestReply fabric_test = 6;
    UpgradeReply upgrade = 7;
    KVEvents kv_events = 8;
+    TrainStatus train_status = 9;
+    TrainArtifact train_artifact = 10;
  }
 }

@@ -74,6 +76,8 @@ message ControlMessage {
    Firewall firewall = 13;
    RestartAgent restart_agent = 14;
    AgentSettings agent_settings = 15;
+    TrainStart train_start = 16;
+    TrainCancel train_cancel = 17;
  }
  // trace is the W3C trace context (traceparent, and tracestate when there
  // is one) of the control plane's span that sent this command, so what the
@@ -884,3 +888,55 @@ message LoraAdapter {
  string name = 1;
  string source = 2;
 }
+
+// TrainStart has a node train a LoRA adapter (BACKLOG F4): on a base model's
+// weights from its registry, on question/answer pairs, with a Python trainer
+// the agent carries, in an environment it makes over an installed engine's.
+// One at a time per node.
+message TrainStart {
+  string job_id = 1;
+  // base_repo and base_revision name the base model's weights on its
+  // registry (Hugging Face): safetensors, not the GGUF a catalog may serve.
+  string base_repo = 2;
+  string base_revision = 3;
+  // data is the training pairs, NDJSON of {"q", "a"}.
+  bytes data = 4;
+  // params is the trainer's settings as JSON: rank, alpha, epochs, lr,
+  // batch, max_len.
+  string params = 5;
+  // gpu_memory_mib is what the job needs of a card; the agent starts it
+  // only when one has that much free, which lets it run beside an engine.
+  int32 gpu_memory_mib = 6;
+  // gguf asks for the adapter in llama.cpp's format too.
+  bool gguf = 7;
+}
+
+// TrainCancel stops a training job.
+message TrainCancel {
+  string job_id = 1;
+}
+
+// TrainStatus is how a training job is going, sent as it changes.
+message TrainStatus {
+  string job_id = 1;
+  // state is preparing, running, done or failed.
+  string state = 2;
+  int32 step = 3;
+  int32 steps = 4;
+  double loss = 5;
+  // message is what is being done, or why it failed.
+  string message = 6;
+  // peak_mib is the most the job held of the card.
+  int32 peak_mib = 7;
+}
+
+// TrainArtifact is a piece of what a training job made, sent before its
+// final status: a gzipped tar of the PEFT adapter (kind peft), or the GGUF
+// adapter (kind gguf).
+message TrainArtifact {
+  string job_id = 1;
+  string kind = 2;
+  int32 seq = 3;
+  bytes data = 4;
+  bool last = 5;
+}
@@ -8,8 +8,9 @@ description is.
 anything they do can be done with `curl`, a generated client or the
 [Terraform provider](../terraform-provider-llmgw/README.md). It adds and
 removes models, scales them, cordons and upgrades nodes, sets capacity
-quotas, attaches guardrail policies, and reads the fleet, usage and the audit
-log.
+quotas, attaches guardrail policies, adds organizations, teams and projects,
+issues and revokes API keys, puts people in the tree, and reads the fleet,
+usage and the audit log.

 ## Reaching it

@@ -99,6 +100,14 @@ curl -s "${H[@]}" -X POST $API/models/qwen-small/scale -d '{"min": 0, "max": 2,
 # A team's capacity quota.
 curl -s "${H[@]}" -X PUT $API/quotas -d '{"scope": "team:Acme/ml", "limit": {"gpus": 8}}'

+# An organization, a team in it and a project in that; then a key for the
+# project. The answer holds the key once: only its hash is kept.
+curl -s "${H[@]}" -X POST $API/orgs -d '{"name": "Acme"}'
+curl -s "${H[@]}" -X POST $API/teams -d '{"name": "ml", "in": "Acme"}'
+curl -s "${H[@]}" -X POST $API/projects -d '{"name": "search", "in": "Acme/ml"}'
+curl -s "${H[@]}" -X POST $API/keys -d '{"name": "ci", "project": "Acme/ml/search", "expires_in": "720h"}'
+curl -s "${H[@]}" -X POST $API/users/ayse@example.com/memberships -d '{"target": "team:Acme/ml"}'
+
 # Remove the model; a DELETE carries the content type too.
 curl -s "${H[@]}" -X DELETE $API/models/qwen-small
 ```
@@ -149,11 +158,11 @@ go generate ./internal/docgen

 ## What it does not cover

- **Organizations, teams, projects, people and API keys.** These live in the
-  store and are managed with `llmgw org`, `llmgw users` and `llmgw keys` on the
-  control plane, or in the web console's `/admin` pages on the client port
-  behind a session. The admin API reads the tree only where a quota or a
-  policy names a part of it.
+- **Making people's accounts.** People sign up, are invited, or come from
+  the identity provider (OIDC, SCIM); the API lists them and puts them in
+  the tree, and `llmgw users` approves, disables and deletes them. Key
+  lifetime limits and spending limits are `llmgw keys ttl` and `llmgw keys
+  limits`.
 - **The client-facing API** (`/v1/chat/completions` and the rest), which is
  OpenAI's and Anthropic's, on `listen`, with API keys.
 - **The rest of the admin listener**: `/metrics`, `/health`, the dashboard
@@ -201,6 +201,16 @@ this by running the same command again, on the control plane that holds the
 authority: etcd is rebuilt on TLS with Patroni in maintenance mode, so
 PostgreSQL serves throughout and no failover happens.

+Each member's PostgreSQL is sized to the machine's memory, read each time
+the command runs: a quarter of it as cache (`shared_buffers`), three quarters
+counted on for the planner, and room to sort and build indexes, all capped. A
+member that also runs a control plane or an agent gets less: about 15% as
+cache, so the gateway is not pushed into swap. A knowledge base's vector
+index is searched well only from memory, so give a store machine more RAM and
+run the command again. The new cache takes a PostgreSQL restart, which Patroni
+does one member at a time, the leader last. The leader's restart costs a few
+seconds without writes, and there is no failover.
+
 Every step looks before it acts. Running it again finishes an install that
 stopped halfway and changes nothing on one that did not; it never overwrites
 `server.yaml`, the authority, or an existing account's password.
@@ -1736,9 +1746,11 @@ the file again.

 On a machine that also runs the control plane, PostgreSQL's defaults assume it
 owns the box. A gateway swapped out to give the database cache is a worse
-outcome than a smaller cache, so cap it: `shared_buffers = 96MB` and
-`max_connections = 40` are enough for a store whose working set is measured in
-megabytes.
+outcome than a smaller cache, so size it to the machine. `server init` does this
+for a PostgreSQL it installs: `shared_buffers` about 15% of memory (at least
+96MB), `effective_cache_size` half, `work_mem` and `maintenance_work_mem`
+scaled to it, and `max_connections = 40`. A store with a large knowledge base
+wants a machine of its own, where its vector index fits in memory.

 ## Tagging requests

@@ -2184,13 +2196,29 @@ llmgw ctl config set firewall.enabled false   # every machine's table taken out
 llmgw ctl config set firewall.enabled true    # and put back as it was
 ```

-It is in force within seconds: each node's agent takes its table out or puts
-it back, and each control plane's `llmgw-firewall-sync.path` (installed by
-`server init` and `server add`; `sudo llmgw firewall sync --install` on one
-set up before) does the same there. The rules files stay, so turning it back
-on restores them. Nodes added while it is off get none. A separate mirror
-machine and the store's members keep theirs for now. `firewall_networks`
-changes reach the nodes the same way.
+It is in force on every machine within a minute, each in its own way:
+
+- **nodes**: the agent takes its table out or puts it back;
+- **control planes**: `llmgw-firewall-sync.path`, installed by `server
+  init` and `server add` (`sudo llmgw firewall sync --install` on one set up
+  before);
+- **a separate mirror machine**: the leading control plane tells the mirror,
+  which writes it to `.firewall.json` in its directory, and the same path
+  unit applies it. `mirror init` installs it; on a mirror set up before, run
+  `sudo llmgw mirror init` again with the same flags;
+- **store members that are nothing else**: they run no llmgw, so
+  `llmgw-firewall-follow.timer` reads what the leading control plane keeps in
+  the store, from the member's own copy, once a minute. `server init
+  --patroni` installs it; run it again on a cluster set up before.
+
+The rules files stay, so turning it back on restores them. Nodes added while
+it is off get none. Networks added to `firewall_networks` reach every machine
+the same way. One taken out of it stays let in until it is taken out of
+`/etc/llmgw/firewall.nft` by hand: the rules are only ever added to.
+
+The leading control plane also tells the mirror machine where every node and
+control plane is, so a node on a network the mirror had not let in can fetch
+its weights without a `firewall_networks` entry.

 Only those ports are touched. The table's policy is to accept, so whatever
 else the machine serves — and whatever firewall it already has — is left as
@@ -45,6 +45,11 @@ is available to everyone who may use them.
  A collection stays with the model it was made with: vectors from two
  models cannot be compared.

+  With pgvector 0.7 or later a collection's index is kept at half
+  precision (`halfvec`), which halves its memory and takes models of up to
+  4000 dimensions; with an older pgvector the limit is 2000. A collection
+  keeps the kind it was made with.
+
 ## For administrators

 ### Make a collection
@@ -317,9 +322,11 @@ Behaviour to know about:
  (Codex, Claude Code) work the same way, since they are answered as chat
  completions.
 - **Separate prefix caches.** On vLLM replicas, a request given passages
-  carries a `cache_salt` derived from its organization. One organization's
-  passages then never share cached blocks with another's, so how fast an
-  answer starts says nothing about what another tenant asked.
+  carries a `cache_salt` derived from the collections they came from.
+  Requests that could be given the same passages still share cached blocks,
+  and no one else's do: not another organization's, nor another team's of
+  the same one. How fast an answer starts therefore says nothing about what
+  someone else asked.
 - **No answer cache.** Answers from knowledge bases are never served from
  the answer cache, because two keys of one project can read different
  collections.
@@ -340,6 +347,49 @@ a product code or a name.
 Metrics: `llmgw_kb_retrievals_total{result}` (found, nothing, refused) and
 `llmgw_kb_retrieval_seconds`.

+## Training an adapter on a collection
+
+An adapter (LoRA) changes how a model *writes*: the organisation's words,
+format and tone. It does not reliably teach facts. On the PoC, a small
+model trained on nine policy documents answered 4 of 9 rephrased questions
+right and mixed facts up on the rest, while the same questions through the
+knowledge base got 9 of 9. Use both: the knowledge base for what answers
+say, an adapter for how they sound.
+
+```sh
+# 1. Questions and answers from the collection, written by a model of the
+#    cluster. Read the file through and correct it.
+llmgw ctl kb dataset Acme/policies --model qwen2.5-coder-14b-instruct --out policies.jsonl
+
+# 2. Train on a node. --gpu-memory is what the job needs of a card; it
+#    runs beside an engine when the engine leaves that much free.
+llmgw ctl train start policies-tone --model qwen2.5-0.5b-instruct \
+  --base-repo Qwen/Qwen2.5-0.5B-Instruct --node cuda01 --gpu-memory 2300 --data policies.jsonl
+llmgw ctl train show JOB --follow
+
+# 3. Make it the model's adapter, kept on the mirror.
+llmgw ctl train publish JOB --restart
+```
+
+The node trains with its installed vLLM's torch and a small overlay (peft
+and accelerate), as the engines' user. For a model llama.cpp serves, it also
+converts the adapter to GGUF with the installed llama.cpp's own code. What
+it makes is kept by the control plane (`llmgw ctl train download`).
+
+An adapter is private like its collection:
+
+- **It has an owner.** An adapter trained on a collection carries some of
+  what the collection says in its weights, so it belongs to a part of the
+  tree.
+  - By default the owner is the collection's owner.
+  - `--owner` names another, which must be able to read every collection
+    the pairs came from.
+  - Only callers in the owner's part of the tree see the adapter. To anyone
+    else it is a model that does not exist, absent from `/v1/models` too.
+- **It is public only by choice.** `--public` makes an adapter every
+  caller's. It is refused for pairs from a knowledge base, so it is only for
+  pairs written by hand.
+
 ## How it is kept apart

 Passages live in PostgreSQL, partitioned by organization, with an index per
@@ -18,6 +18,7 @@ rather than repeated.
 | `name` | `string` | [Adapter](#adapter) |
 | `source` | `string` | [Adapter](#adapter) |
 | `scale` | `float64` | [Adapter](#adapter) |
+| `owner` | `string` | [Adapter](#adapter) |
 | `enabled` | `bool` | [Admission](#admission) |
 | `bands` | `[]Band` | [Admission](#admission) |
 | `default_band` | `string` | [Admission](#admission) |
@@ -426,6 +427,16 @@ weights are.

 Scale is how strongly it is applied; zero takes 1.

+### `owner`
+
+`string`
+
+Owner is the part of the tenancy tree the adapter belongs to —
+org:ID, team:ID, project:ID, user:ID or key:ID — and only callers in
+it see it: to anyone else it is a model that does not exist. An
+adapter trained on an organisation's documents carries some of what
+they say in its weights. Empty is every caller the model allows.
+
 ## Admission

 ### `enabled`
@@ -34,6 +34,7 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/train"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/agentsvc"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
@@ -140,6 +141,9 @@ type Agent struct {
 	// settingsCheck says whether the cluster's settings for this agent
 	// would load beside its agent.yaml; see SetSettingsCheck.
 	settingsCheck func([]byte) error
+
+	// trainer runs LoRA training jobs, one at a time (BACKLOG F4).
+	trainer *train.Runner
 }

 // connectHTTPClient is the HTTP client connect uses; an interface so tests can
@@ -186,6 +190,18 @@ func New(cfg Config, httpClient connectHTTPClient, log *slog.Logger) (*Agent, er
 		kvSubs:       map[string]*kvevents.Subscriber{},
 	}
 	a.prov = &provision.Installer{Root: cfg.EnginesDir, Log: log, OnStatus: a.provisionStatus}
+	runAs := cfg.Superv.RunAs
+	if runAs == "" {
+		runAs = supervisor.DefaultRunAs
+	}
+	// Where the engines are, as the installer has it when the file says
+	// nothing.
+	enginesDir := cfg.EnginesDir
+	if enginesDir == "" {
+		enginesDir = provision.DefaultRoot
+	}
+	a.trainer = train.New(train.Env{EnginesDir: enginesDir, WorkDir: filepath.Join(cfg.Superv.WorkDir, "trainer"),
+		RunAs: runAs, Log: log})
 	return a, nil
 }

@@ -886,6 +902,10 @@ func (a *Agent) handle(ctx context.Context, msg *llmgwv1.ControlMessage, out cha
 		go a.applyFirewall(m.Firewall)
 	case *llmgwv1.ControlMessage_AgentSettings:
 		go a.applySettings(m.AgentSettings)
+	case *llmgwv1.ControlMessage_TrainStart:
+		go a.trainJob(ctx, m.TrainStart, out)
+	case *llmgwv1.ControlMessage_TrainCancel:
+		a.trainer.Cancel(m.TrainCancel.GetJobId())
 	case *llmgwv1.ControlMessage_RestartAgent:
 		go func() {
 			a.log.Info("restarting, as the control plane asks; the engines keep serving",
@@ -3,6 +3,7 @@ package supervisor
 import (
 	"fmt"
 	"os"
+	"os/exec"
 	"os/user"
 	"path/filepath"
 	"strconv"
@@ -90,3 +91,25 @@ func credentialFor(u *user.User, workDir string) (*runAs, error) {
 func (r *runAs) env() []string {
 	return []string{"HOME=" + r.home, "XDG_CACHE_HOME=" + filepath.Join(r.home, ".cache"), "USER=" + r.name, "LOGNAME=" + r.name}
 }
+
+// AsEngineUser has cmd run as the engine user, as an engine would, with its
+// home under workDir: for work the agent does that is not an engine but
+// reads what an engine reads — training an adapter. It answers that home
+// and a way to hand a path to the user, or why it runs as this process.
+func AsEngineUser(cmd *exec.Cmd, name, workDir string) (home string, give func(path string) error, why string) {
+	r, why := resolveRunAs(name, workDir)
+	if r == nil {
+		return "", func(string) error { return nil }, why
+	}
+	cmd.SysProcAttr = &syscall.SysProcAttr{Credential: r.cred, Setpgid: true}
+	cmd.Env = append(cmd.Env, r.env()...)
+	uid, gid := int(r.cred.Uid), int(r.cred.Gid)
+	return r.home, func(path string) error {
+		return filepath.WalkDir(path, func(p string, _ os.DirEntry, err error) error {
+			if err != nil {
+				return err
+			}
+			return os.Lchown(p, uid, gid)
+		})
+	}, ""
+}
@@ -0,0 +1,574 @@
+// Package train runs LoRA training jobs on a node (BACKLOG F4): with an
+// installed engine's Python — its torch and transformers, already matched to
+// the node's CUDA — and a small overlay of what training needs besides, so a
+// node that serves can train without a second multi-gigabyte environment.
+package train
+
+import (
+	"archive/tar"
+	"bufio"
+	"bytes"
+	"compress/gzip"
+	"context"
+	"crypto/sha256"
+	_ "embed"
+	"encoding/hex"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"log/slog"
+	"net/http"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"sort"
+	"strconv"
+	"strings"
+	"sync"
+	"syscall"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
+)
+
+//go:embed trainer.py
+var trainerPy []byte
+
+// overlayPackages are what training needs besides an engine's Python, pinned
+// and installed without their dependencies: those are the engine's own, and
+// letting an installer resolve them pulled a second torch.
+var overlayPackages = []string{"peft==0.21.2", "accelerate==1.15.0"}
+
+// Job is one training job, as the control plane sent it.
+type Job struct {
+	ID           string
+	BaseRepo     string
+	BaseRevision string
+	// Data is NDJSON of {"q", "a"}.
+	Data []byte
+	// Params is the trainer's settings as JSON.
+	Params       string
+	GPUMemoryMiB int
+	GGUF         bool
+}
+
+// Status is how a job is going.
+type Status struct {
+	State   string
+	Message string
+	Step    int
+	Steps   int
+	Loss    float64
+	PeakMiB int
+}
+
+// Env is where a node keeps what training needs.
+type Env struct {
+	// EnginesDir is where engines are installed (the agent's engines_dir).
+	EnginesDir string
+	// WorkDir is the trainer's own directory: its overlay, the llama.cpp
+	// conversion code, and each job's files while it runs.
+	WorkDir string
+	// RunAs is the user jobs run as: the engines' user.
+	RunAs string
+	Log   *slog.Logger
+	// Python, UV, ConvDir and GPUs replace what would be found, for tests.
+	Python, UV, ConvDir string
+	GPUs                func(ctx context.Context) ([]GPU, error)
+}
+
+// GPU is one card and how much of it is free.
+type GPU struct {
+	Index   int
+	FreeMiB int
+}
+
+// Runner runs one job at a time.
+type Runner struct {
+	env    Env
+	mu     sync.Mutex
+	cancel context.CancelFunc
+	job    string
+}
+
+// New is a runner.
+func New(env Env) *Runner {
+	if env.Log == nil {
+		env.Log = slog.New(slog.DiscardHandler)
+	}
+	return &Runner{env: env}
+}
+
+// Cancel stops a job if it is the one running.
+func (r *Runner) Cancel(id string) {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	if r.job == id && r.cancel != nil {
+		r.cancel()
+	}
+}
+
+// ErrBusy is a node already training.
+var ErrBusy = errors.New("this node is already training")
+
+// Run runs a job to its end, telling status as it goes and artifact each
+// piece of what it made. It answers the error it failed with.
+func (r *Runner) Run(ctx context.Context, job Job, status func(Status), artifact func(kind string, seq int, data []byte, last bool) error) error {
+	r.mu.Lock()
+	if r.job != "" {
+		r.mu.Unlock()
+		return ErrBusy
+	}
+	ctx, cancel := context.WithCancel(ctx)
+	r.job, r.cancel = job.ID, cancel
+	r.mu.Unlock()
+	defer func() {
+		cancel()
+		r.mu.Lock()
+		r.job, r.cancel = "", nil
+		r.mu.Unlock()
+	}()
+
+	status(Status{State: "preparing", Message: "preparing the trainer"})
+	py, err := r.python()
+	if err != nil {
+		return err
+	}
+	overlay, err := r.overlay(ctx, py)
+	if err != nil {
+		return fmt.Errorf("installing %s: %w", strings.Join(overlayPackages, ", "), err)
+	}
+	conv := ""
+	if job.GGUF {
+		if conv, err = r.conversion(ctx); err != nil {
+			return fmt.Errorf("fetching llama.cpp's conversion code: %w", err)
+		}
+	}
+	gpu, err := r.pickGPU(ctx, job.GPUMemoryMiB)
+	if err != nil {
+		return err
+	}
+
+	dir := filepath.Join(r.env.WorkDir, "jobs", job.ID)
+	if err := os.MkdirAll(dir, 0o755); err != nil {
+		return err
+	}
+	defer os.RemoveAll(dir)
+	out := filepath.Join(dir, "adapter")
+	var params map[string]any
+	if job.Params != "" {
+		if err := json.Unmarshal([]byte(job.Params), &params); err != nil {
+			return fmt.Errorf("the job's params: %w", err)
+		}
+	}
+	spec, _ := json.Marshal(map[string]any{"base_repo": job.BaseRepo, "base_revision": job.BaseRevision,
+		"data": filepath.Join(dir, "data.jsonl"), "out": out, "params": params})
+	for name, b := range map[string][]byte{"data.jsonl": job.Data, "job.json": spec, "trainer.py": trainerPy} {
+		if err := os.WriteFile(filepath.Join(dir, name), b, 0o644); err != nil {
+			return err
+		}
+	}
+
+	var baseDir string
+	err = r.python3(ctx, py, dir, []string{"PYTHONPATH=" + overlay, "CUDA_VISIBLE_DEVICES=" + strconv.Itoa(gpu)},
+		[]string{filepath.Join(dir, "trainer.py"), filepath.Join(dir, "job.json")},
+		func(line map[string]any) {
+			s := Status{State: str(line["state"]), Message: str(line["message"]), Step: num(line["step"]),
+				Steps: num(line["steps"]), PeakMiB: num(line["peak_mib"])}
+			if v, ok := line["loss"].(float64); ok {
+				s.Loss = v
+			}
+			if b := str(line["base_dir"]); b != "" {
+				baseDir = b
+			}
+			if s.State == "trained" {
+				s.State = "running"
+				s.Message = "trained; packing the adapter"
+			}
+			status(s)
+		})
+	if err != nil {
+		return err
+	}
+
+	if job.GGUF {
+		status(Status{State: "running", Message: "converting the adapter to GGUF for llama.cpp"})
+		gguf := filepath.Join(dir, "adapter.gguf")
+		err := r.python3(ctx, py, dir, []string{"PYTHONPATH=" + strings.Join([]string{filepath.Join(conv, "gguf-py"), conv, overlay}, ":")},
+			[]string{filepath.Join(conv, "convert_lora_to_gguf.py"), "--base", baseDir, "--outtype", "f16", "--outfile", gguf, out}, nil)
+		if err != nil {
+			return fmt.Errorf("converting to GGUF: %w", err)
+		}
+		if err := sendFile(gguf, "gguf", artifact); err != nil {
+			return err
+		}
+	}
+	var tgz bytes.Buffer
+	if err := tarGz(&tgz, out); err != nil {
+		return err
+	}
+	return sendBytes(tgz.Bytes(), "peft", artifact)
+}
+
+func str(v any) string {
+	s, _ := v.(string)
+	return s
+}
+
+func num(v any) int {
+	f, _ := v.(float64)
+	return int(f)
+}
+
+// python3 runs a Python script as the engines' user, passing each JSON line
+// it prints to line, and answers the last of what it wrote to stderr when it
+// fails.
+func (r *Runner) python3(ctx context.Context, py, dir string, env, args []string, line func(map[string]any)) error {
+	cmd := exec.CommandContext(ctx, py, args...)
+	cmd.Dir = dir
+	cmd.Env = append(os.Environ(), "PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True", "TOKENIZERS_PARALLELISM=false",
+		"HF_HUB_DISABLE_PROGRESS_BARS=1", "PYTHONUNBUFFERED=1")
+	cmd.Env = append(cmd.Env, env...)
+	home, give, why := supervisor.AsEngineUser(cmd, r.env.RunAs, r.env.WorkDir)
+	if why != "" {
+		r.env.Log.Warn(why)
+	}
+	if home != "" {
+		cmd.Env = append(cmd.Env, "HF_HOME="+filepath.Join(home, ".cache", "huggingface"))
+	} else {
+		cmd.Env = append(cmd.Env, "HF_HOME="+filepath.Join(r.env.WorkDir, "hf"))
+	}
+	if err := give(dir); err != nil {
+		return fmt.Errorf("handing %s to %s: %w", dir, r.env.RunAs, err)
+	}
+	// The whole process group goes when the job is cancelled: torch starts
+	// workers of its own.
+	if cmd.SysProcAttr == nil {
+		cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
+	}
+	cmd.Cancel = func() error { return syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL) }
+	stdout, err := cmd.StdoutPipe()
+	if err != nil {
+		return err
+	}
+	var stderr tailBuffer
+	cmd.Stderr = &stderr
+	if err := cmd.Start(); err != nil {
+		return err
+	}
+	sc := bufio.NewScanner(stdout)
+	sc.Buffer(make([]byte, 64<<10), 1<<20)
+	for sc.Scan() {
+		var m map[string]any
+		if line != nil && json.Unmarshal(sc.Bytes(), &m) == nil {
+			line(m)
+		}
+	}
+	err = cmd.Wait()
+	if ctx.Err() != nil {
+		return errors.New("cancelled")
+	}
+	if err != nil {
+		return fmt.Errorf("%v: %s", err, stderr.last())
+	}
+	return nil
+}
+
+// tailBuffer keeps the end of what a process wrote: the reason it failed is
+// at the end, and the rest is progress bars.
+type tailBuffer struct{ b []byte }
+
+func (t *tailBuffer) Write(p []byte) (int, error) {
+	t.b = append(t.b, p...)
+	if len(t.b) > 16<<10 {
+		t.b = t.b[len(t.b)-16<<10:]
+	}
+	return len(p), nil
+}
+
+func (t *tailBuffer) last() string {
+	lines := strings.Split(strings.TrimSpace(string(t.b)), "\n")
+	if len(lines) > 3 {
+		lines = lines[len(lines)-3:]
+	}
+	return strings.Join(lines, " | ")
+}
+
+// python is the newest installed vLLM's interpreter: its torch is built for
+// this node's CUDA.
+func (r *Runner) python() (string, error) {
+	if r.env.Python != "" {
+		return r.env.Python, nil
+	}
+	found, _ := filepath.Glob(filepath.Join(r.env.EnginesDir, "vllm", "*", "bin", "python3"))
+	if len(found) == 0 {
+		return "", errors.New("training needs vLLM installed on this node (its torch is what trains); " +
+			"llmgw ctl engine-build, or serving any vLLM model here, installs it")
+	}
+	sort.Strings(found)
+	return found[len(found)-1], nil
+}
+
+func (r *Runner) uv() (string, error) {
+	if r.env.UV != "" {
+		return r.env.UV, nil
+	}
+	var found []string
+	_ = filepath.WalkDir(filepath.Join(r.env.EnginesDir, "uv"), func(p string, d os.DirEntry, err error) error {
+		if err == nil && !d.IsDir() && d.Name() == "uv" {
+			found = append(found, p)
+		}
+		return nil
+	})
+	if len(found) == 0 {
+		return "", errors.New("no uv under " + filepath.Join(r.env.EnginesDir, "uv"))
+	}
+	sort.Strings(found)
+	return found[len(found)-1], nil
+}
+
+// overlay installs the pinned packages once per Python and set of pins.
+func (r *Runner) overlay(ctx context.Context, py string) (string, error) {
+	sum := sha256.Sum256([]byte(py + "\x00" + strings.Join(overlayPackages, " ")))
+	dir := filepath.Join(r.env.WorkDir, "overlay-"+hex.EncodeToString(sum[:6]))
+	if _, err := os.Stat(filepath.Join(dir, ".done")); err == nil {
+		return dir, nil
+	}
+	uv, err := r.uv()
+	if err != nil {
+		return "", err
+	}
+	tmp := dir + ".new"
+	_ = os.RemoveAll(tmp)
+	args := append([]string{"pip", "install", "--quiet", "--python", py, "--no-deps", "--target", tmp}, overlayPackages...)
+	cmd := exec.CommandContext(ctx, uv, args...)
+	if out, err := cmd.CombinedOutput(); err != nil {
+		return "", fmt.Errorf("%v: %s", err, strings.TrimSpace(string(out)))
+	}
+	_ = os.WriteFile(filepath.Join(tmp, ".done"), nil, 0o644)
+	_ = os.RemoveAll(dir)
+	if err := os.Rename(tmp, dir); err != nil {
+		return "", err
+	}
+	// Readable by the engines' user, which runs the trainer.
+	_ = filepath.WalkDir(dir, func(p string, d os.DirEntry, err error) error {
+		if err == nil {
+			mode := os.FileMode(0o644)
+			if d.IsDir() {
+				mode = 0o755
+			}
+			_ = os.Chmod(p, mode)
+		}
+		return nil
+	})
+	return dir, nil
+}
+
+// llamaCppTag is the release of the newest llama.cpp installed: its
+// conversion code is the one whose GGUF that llama.cpp reads.
+func (r *Runner) llamaCppTag() (string, error) {
+	found, _ := filepath.Glob(filepath.Join(r.env.EnginesDir, "llamacpp", "b*"))
+	if len(found) == 0 {
+		return "", errors.New("a GGUF adapter needs llama.cpp installed on this node, to match its release")
+	}
+	sort.Strings(found)
+	tag, _, _ := strings.Cut(filepath.Base(found[len(found)-1]), "-")
+	return tag, nil
+}
+
+// conversionSource is where llama.cpp's source is fetched from, by tag.
+var conversionSource = "https://codeload.github.com/ggml-org/llama.cpp/tar.gz/refs/tags/"
+
+// conversion fetches llama.cpp's LoRA conversion code for the installed
+// release, once.
+func (r *Runner) conversion(ctx context.Context) (string, error) {
+	if r.env.ConvDir != "" {
+		return r.env.ConvDir, nil
+	}
+	tag, err := r.llamaCppTag()
+	if err != nil {
+		return "", err
+	}
+	dir := filepath.Join(r.env.WorkDir, "llama.cpp-"+tag)
+	if _, err := os.Stat(filepath.Join(dir, "convert_lora_to_gguf.py")); err == nil {
+		return dir, nil
+	}
+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, conversionSource+tag, nil)
+	if err != nil {
+		return "", err
+	}
+	res, err := http.DefaultClient.Do(req)
+	if err != nil {
+		return "", err
+	}
+	defer res.Body.Close()
+	if res.StatusCode != http.StatusOK {
+		return "", fmt.Errorf("%s answered %s", conversionSource+tag, res.Status)
+	}
+	tmp := dir + ".new"
+	_ = os.RemoveAll(tmp)
+	if err := untarConversion(res.Body, tmp); err != nil {
+		return "", err
+	}
+	_ = os.RemoveAll(dir)
+	return dir, os.Rename(tmp, dir)
+}
+
+// untarConversion keeps, of llama.cpp's source, the conversion scripts and
+// the Python packages they import.
+func untarConversion(r io.Reader, dir string) error {
+	gz, err := gzip.NewReader(r)
+	if err != nil {
+		return err
+	}
+	tr := tar.NewReader(gz)
+	kept := 0
+	for {
+		h, err := tr.Next()
+		if errors.Is(err, io.EOF) {
+			break
+		}
+		if err != nil {
+			return err
+		}
+		_, rel, ok := strings.Cut(h.Name, "/")
+		if !ok || h.Typeflag != tar.TypeReg || strings.Contains(rel, "..") {
+			continue
+		}
+		if rel != "convert_lora_to_gguf.py" && rel != "convert_hf_to_gguf.py" &&
+			!strings.HasPrefix(rel, "conversion/") && !strings.HasPrefix(rel, "gguf-py/") {
+			continue
+		}
+		p := filepath.Join(dir, filepath.FromSlash(rel))
+		if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
+			return err
+		}
+		f, err := os.OpenFile(p, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o644)
+		if err != nil {
+			return err
+		}
+		_, err = io.Copy(f, io.LimitReader(tr, 64<<20))
+		f.Close()
+		if err != nil {
+			return err
+		}
+		kept++
+	}
+	if kept == 0 {
+		return errors.New("no conversion code in llama.cpp's source")
+	}
+	return nil
+}
+
+// pickGPU is the card with the most memory free, if that is enough.
+func (r *Runner) pickGPU(ctx context.Context, need int) (int, error) {
+	list := r.env.GPUs
+	if list == nil {
+		list = nvidiaSMI
+	}
+	gpus, err := list(ctx)
+	if err != nil {
+		return 0, fmt.Errorf("reading the cards' free memory: %w", err)
+	}
+	if len(gpus) == 0 {
+		return 0, errors.New("this node has no GPU to train on")
+	}
+	best := gpus[0]
+	for _, g := range gpus[1:] {
+		if g.FreeMiB > best.FreeMiB {
+			best = g
+		}
+	}
+	if best.FreeMiB < need {
+		return 0, fmt.Errorf("no card here has %d MiB free (the most free is GPU %d with %d MiB); "+
+			"a smaller base, or a card freed for it, would do", need, best.Index, best.FreeMiB)
+	}
+	return best.Index, nil
+}
+
+func nvidiaSMI(ctx context.Context) ([]GPU, error) {
+	ctx, cancel := context.WithTimeout(ctx, 15*time.Second)
+	defer cancel()
+	out, err := exec.CommandContext(ctx, "nvidia-smi", "--query-gpu=index,memory.free", "--format=csv,noheader,nounits").Output()
+	if err != nil {
+		return nil, err
+	}
+	return parseGPUs(string(out)), nil
+}
+
+func parseGPUs(s string) []GPU {
+	var out []GPU
+	for _, line := range strings.Split(strings.TrimSpace(s), "\n") {
+		a, b, ok := strings.Cut(line, ",")
+		if !ok {
+			continue
+		}
+		i, err1 := strconv.Atoi(strings.TrimSpace(a))
+		f, err2 := strconv.Atoi(strings.TrimSpace(b))
+		if err1 == nil && err2 == nil {
+			out = append(out, GPU{Index: i, FreeMiB: f})
+		}
+	}
+	return out
+}
+
+// tarGz packs a directory.
+func tarGz(w io.Writer, dir string) error {
+	gz := gzip.NewWriter(w)
+	tw := tar.NewWriter(gz)
+	err := filepath.WalkDir(dir, func(p string, d os.DirEntry, err error) error {
+		if err != nil || d.IsDir() {
+			return err
+		}
+		rel, _ := filepath.Rel(dir, p)
+		info, err := d.Info()
+		if err != nil {
+			return err
+		}
+		if err := tw.WriteHeader(&tar.Header{Name: filepath.ToSlash(rel), Mode: 0o644, Size: info.Size(), ModTime: info.ModTime()}); err != nil {
+			return err
+		}
+		f, err := os.Open(p)
+		if err != nil {
+			return err
+		}
+		defer f.Close()
+		_, err = io.Copy(tw, f)
+		return err
+	})
+	if err != nil {
+		return err
+	}
+	if err := tw.Close(); err != nil {
+		return err
+	}
+	return gz.Close()
+}
+
+// artifactChunk is how much of an artifact goes in one message.
+const artifactChunk = 1 << 20
+
+func sendFile(path, kind string, send func(string, int, []byte, bool) error) error {
+	b, err := os.ReadFile(path)
+	if err != nil {
+		return err
+	}
+	return sendBytes(b, kind, send)
+}
+
+func sendBytes(b []byte, kind string, send func(string, int, []byte, bool) error) error {
+	seq := 0
+	for {
+		n := min(artifactChunk, len(b))
+		last := n == len(b)
+		if err := send(kind, seq, b[:n], last); err != nil {
+			return err
+		}
+		if last {
+			return nil
+		}
+		b = b[n:]
+		seq++
+	}
+}
@@ -0,0 +1,164 @@
+package train
+
+import (
+	"archive/tar"
+	"bytes"
+	"compress/gzip"
+	"context"
+	"io"
+	"os"
+	"path/filepath"
+	"strings"
+	"sync"
+	"testing"
+)
+
+// fakePython stands in for an engine's interpreter: as the trainer it
+// prints progress and writes an adapter; as the converter it writes a GGUF;
+// told to fail, it fails.
+const fakePython = `#!/bin/sh
+case "$1" in
+*trainer.py)
+  out=$(sed -n 's/.*"out": *"\([^"]*\)".*/\1/p' "$2")
+  grep -q FAIL "$(dirname "$2")/data.jsonl" && { echo "CUDA out of memory" >&2; exit 1; }
+  echo '{"state":"preparing","message":"fetching base"}'
+  echo '{"state":"running","step":5,"steps":10,"loss":0.9,"peak_mib":1900}'
+  mkdir -p "$out"; echo '{"r":16}' > "$out/adapter_config.json"; echo weights > "$out/adapter_model.safetensors"
+  echo '{"state":"trained","step":10,"steps":10,"peak_mib":2100,"base_dir":"/base"}'
+  ;;
+*convert_lora_to_gguf.py)
+  while [ $# -gt 0 ]; do [ "$1" = "--outfile" ] && echo GGUF > "$2"; shift; done
+  ;;
+esac
+`
+
+func runner(t *testing.T, free int) *Runner {
+	t.Helper()
+	dir := t.TempDir()
+	py := filepath.Join(dir, "python3")
+	_ = os.WriteFile(py, []byte(fakePython), 0o755)
+	uv := filepath.Join(dir, "uv")
+	_ = os.WriteFile(uv, []byte("#!/bin/sh\nwhile [ $# -gt 0 ]; do [ \"$1\" = \"--target\" ] && mkdir -p \"$2\"; shift; done\n"), 0o755)
+	return New(Env{WorkDir: filepath.Join(dir, "work"), Python: py, UV: uv, ConvDir: dir,
+		GPUs: func(context.Context) ([]GPU, error) { return []GPU{{0, 500}, {1, free}}, nil }})
+}
+
+type sink struct {
+	mu       sync.Mutex
+	statuses []Status
+	parts    map[string][]byte
+	lasts    int
+}
+
+func (s *sink) status(st Status) { s.mu.Lock(); s.statuses = append(s.statuses, st); s.mu.Unlock() }
+func (s *sink) artifact(kind string, _ int, b []byte, last bool) error {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	if s.parts == nil {
+		s.parts = map[string][]byte{}
+	}
+	s.parts[kind] = append(s.parts[kind], b...)
+	if last {
+		s.lasts++
+	}
+	return nil
+}
+
+func TestAJobTrainsOnTheFreestCardAndSendsWhatItMade(t *testing.T) {
+	r := runner(t, 3000)
+	s := &sink{}
+	err := r.Run(context.Background(), Job{ID: "j1", BaseRepo: "Qwen/x", Data: []byte(`{"q":"a","a":"b"}`), GPUMemoryMiB: 2500, GGUF: true},
+		s.status, s.artifact)
+	if err != nil {
+		t.Fatal(err)
+	}
+	var running bool
+	for _, st := range s.statuses {
+		if st.State == "running" && st.Step == 5 && st.Steps == 10 && st.PeakMiB == 1900 {
+			running = true
+		}
+	}
+	if !running {
+		t.Errorf("statuses %+v", s.statuses)
+	}
+	if string(s.parts["gguf"]) != "GGUF\n" || s.lasts != 2 {
+		t.Errorf("gguf %q, lasts %d", s.parts["gguf"], s.lasts)
+	}
+	files := map[string]string{}
+	gz, err := gzip.NewReader(bytes.NewReader(s.parts["peft"]))
+	if err != nil {
+		t.Fatal(err)
+	}
+	tr := tar.NewReader(gz)
+	for {
+		h, err := tr.Next()
+		if err != nil {
+			break
+		}
+		b, _ := io.ReadAll(tr)
+		files[h.Name] = string(b)
+	}
+	if files["adapter_model.safetensors"] != "weights\n" || files["adapter_config.json"] == "" {
+		t.Errorf("peft archive %v", files)
+	}
+	// Nothing of the job is left on the node.
+	if entries, _ := os.ReadDir(filepath.Join(r.env.WorkDir, "jobs")); len(entries) != 0 {
+		t.Errorf("left behind: %v", entries)
+	}
+}
+
+func TestAJobWaitsForNoCardItDoesNotFit(t *testing.T) {
+	r := runner(t, 1000)
+	err := r.Run(context.Background(), Job{ID: "j1", Data: []byte(`{"q":"a","a":"b"}`), GPUMemoryMiB: 2500}, (&sink{}).status, (&sink{}).artifact)
+	if err == nil || !strings.Contains(err.Error(), "no card here has 2500 MiB free") {
+		t.Errorf("got %v", err)
+	}
+}
+
+func TestATrainerThatFailsSaysWhy(t *testing.T) {
+	r := runner(t, 3000)
+	err := r.Run(context.Background(), Job{ID: "j1", Data: []byte(`{"q":"FAIL","a":"b"}`)}, (&sink{}).status, (&sink{}).artifact)
+	if err == nil || !strings.Contains(err.Error(), "CUDA out of memory") {
+		t.Errorf("got %v", err)
+	}
+}
+
+func TestOneJobAtATime(t *testing.T) {
+	r := runner(t, 3000)
+	r.job = "other"
+	if err := r.Run(context.Background(), Job{ID: "j2"}, (&sink{}).status, (&sink{}).artifact); err != ErrBusy {
+		t.Errorf("got %v", err)
+	}
+}
+
+func TestCardsAreReadFromNvidiaSMI(t *testing.T) {
+	got := parseGPUs("0, 2310\n1, 80000\nbad line\n")
+	if len(got) != 2 || got[1].FreeMiB != 80000 || got[0].Index != 0 {
+		t.Errorf("parsed %+v", got)
+	}
+}
+
+func TestOnlyTheConversionCodeIsKeptOfLlamaCpp(t *testing.T) {
+	var buf bytes.Buffer
+	gz := gzip.NewWriter(&buf)
+	tw := tar.NewWriter(gz)
+	for _, name := range []string{"llama.cpp-b1/convert_lora_to_gguf.py", "llama.cpp-b1/conversion/qwen.py",
+		"llama.cpp-b1/gguf-py/gguf/__init__.py", "llama.cpp-b1/src/llama.cpp", "llama.cpp-b1/../evil"} {
+		_ = tw.WriteHeader(&tar.Header{Name: name, Mode: 0o644, Size: 1, Typeflag: tar.TypeReg})
+		_, _ = tw.Write([]byte("x"))
+	}
+	_ = tw.Close()
+	_ = gz.Close()
+	dir := filepath.Join(t.TempDir(), "conv")
+	if err := untarConversion(&buf, dir); err != nil {
+		t.Fatal(err)
+	}
+	for _, want := range []string{"convert_lora_to_gguf.py", "conversion/qwen.py", "gguf-py/gguf/__init__.py"} {
+		if _, err := os.Stat(filepath.Join(dir, want)); err != nil {
+			t.Errorf("%s not kept", want)
+		}
+	}
+	if _, err := os.Stat(filepath.Join(dir, "src")); err == nil {
+		t.Error("the rest of llama.cpp was kept")
+	}
+}
@@ -0,0 +1,101 @@
+# llmgw's LoRA trainer (BACKLOG F4), run by the node agent with an installed
+# engine's Python (its torch and transformers) and a small overlay (peft,
+# accelerate). Reads a job, trains, writes the PEFT adapter, and says how it
+# goes as one JSON object a line on standard output.
+import json, math, os, random, sys, threading, subprocess, time
+
+def say(**kw):
+    print(json.dumps(kw), flush=True)
+
+job = json.load(open(sys.argv[1], encoding="utf-8"))
+p = {"rank": 16, "alpha": 32, "dropout": 0.05, "epochs": 6, "lr": 2e-4, "batch": 4, "max_len": 512, "seed": 0}
+p.update(job.get("params") or {})
+
+say(state="preparing", message="loading libraries")
+import torch
+from huggingface_hub import snapshot_download
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from peft import LoraConfig, get_peft_model
+
+random.seed(p["seed"]); torch.manual_seed(p["seed"])
+if not torch.cuda.is_available():
+    raise SystemExit("no CUDA device is visible to the trainer")
+
+say(state="preparing", message="fetching " + job["base_repo"])
+base_dir = snapshot_download(job["base_repo"], revision=job.get("base_revision") or None,
+                             allow_patterns=["*.json", "*.safetensors", "*.txt", "*.model", "*.tiktoken"])
+say(state="preparing", message="loading the base model", base_dir=base_dir)
+
+# bf16 where the card has it; fp16 with a loss scaler where it does not
+# (Turing and older).
+bf16 = torch.cuda.is_bf16_supported()
+dtype = torch.bfloat16 if bf16 else torch.float16
+tok = AutoTokenizer.from_pretrained(base_dir)
+if tok.pad_token_id is None:
+    tok.pad_token = tok.eos_token
+model = AutoModelForCausalLM.from_pretrained(base_dir, dtype=dtype).to("cuda")
+
+rows = [json.loads(l) for l in open(job["data"], encoding="utf-8") if l.strip()]
+rows = [r for r in rows if r.get("q") and r.get("a")]
+if not rows:
+    raise SystemExit("no {q, a} pairs in the data")
+
+def encode(r):
+    prompt = tok.apply_chat_template([{"role": "user", "content": r["q"]}], add_generation_prompt=True, tokenize=False)
+    full = tok.apply_chat_template([{"role": "user", "content": r["q"]}, {"role": "assistant", "content": r["a"]}], tokenize=False)
+    pi = tok(prompt, add_special_tokens=False)["input_ids"]
+    fi = tok(full, add_special_tokens=False)["input_ids"][: p["max_len"]]
+    return fi, ([-100] * len(pi) + fi[len(pi):])[: len(fi)]
+
+data = [encode(r) for r in rows]
+
+model.gradient_checkpointing_enable()
+model.enable_input_require_grads()
+model = get_peft_model(model, LoraConfig(r=int(p["rank"]), lora_alpha=int(p["alpha"]), lora_dropout=float(p["dropout"]),
+    task_type="CAUSAL_LM", target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]))
+trainable = [x for x in model.parameters() if x.requires_grad]
+for x in trainable:  # fp32 trainable weights, so a loss scaler can unscale them
+    x.data = x.data.float()
+
+peak = {"mib": 0}
+stop = threading.Event()
+def watch():
+    while not stop.is_set():
+        try:
+            out = subprocess.run(["nvidia-smi", "--query-compute-apps=pid,used_memory", "--format=csv,noheader,nounits"],
+                                 capture_output=True, text=True, timeout=5).stdout
+            for line in out.splitlines():
+                pid, mib = [v.strip() for v in line.split(",")]
+                if int(pid) == os.getpid():
+                    peak["mib"] = max(peak["mib"], int(mib))
+        except Exception:
+            pass
+        time.sleep(0.5)
+threading.Thread(target=watch, daemon=True).start()
+
+opt = torch.optim.AdamW(trainable, lr=float(p["lr"]), weight_decay=0.0)
+scaler = torch.amp.GradScaler("cuda", enabled=not bf16)
+bs, epochs = int(p["batch"]), int(p["epochs"])
+steps = epochs * math.ceil(len(data) / bs)
+warm = max(1, steps // 10)
+sched = torch.optim.lr_scheduler.LambdaLR(opt, lambda s: min(1.0, (s + 1) / warm) * max(0.0, 1 - s / steps))
+model.train(); step = 0; last = time.time()
+for ep in range(epochs):
+    random.shuffle(data)
+    for i in range(0, len(data), bs):
+        batch = data[i:i + bs]; n = max(len(x[0]) for x in batch)
+        ids = torch.tensor([x[0] + [tok.pad_token_id] * (n - len(x[0])) for x in batch], device="cuda")
+        lab = torch.tensor([x[1] + [-100] * (n - len(x[1])) for x in batch], device="cuda")
+        att = torch.tensor([[1] * len(x[0]) + [0] * (n - len(x[0])) for x in batch], device="cuda")
+        with torch.autocast("cuda", dtype=dtype):
+            loss = model(input_ids=ids, attention_mask=att, labels=lab).loss
+        scaler.scale(loss).backward(); scaler.unscale_(opt)
+        torch.nn.utils.clip_grad_norm_(trainable, 1.0)
+        scaler.step(opt); scaler.update(); opt.zero_grad(set_to_none=True); sched.step(); step += 1
+        if time.time() - last > 2 or step == steps:
+            last = time.time()
+            say(state="running", message="training", step=step, steps=steps, loss=round(loss.item(), 4), peak_mib=peak["mib"])
+
+model.save_pretrained(job["out"])
+stop.set()
+say(state="trained", step=step, steps=steps, peak_mib=peak["mib"], base_dir=base_dir)
@@ -0,0 +1,46 @@
+package agent
+
+import (
+	"context"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/train"
+)
+
+// trainJob runs a training job the control plane sent, telling it how the
+// job goes and sending what it made. A stream that ends ends the job: the
+// control plane, which sees the node go, marks it failed, and it is started
+// again from the beginning rather than resumed.
+func (a *Agent) trainJob(ctx context.Context, m *llmgwv1.TrainStart, out chan<- *llmgwv1.AgentMessage) {
+	id := m.GetJobId()
+	send := func(msg *llmgwv1.AgentMessage) bool {
+		select {
+		case out <- msg:
+			return true
+		case <-ctx.Done():
+			return false
+		}
+	}
+	status := func(s train.Status) {
+		send(&llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_TrainStatus{TrainStatus: &llmgwv1.TrainStatus{
+			JobId: id, State: s.State, Message: s.Message, Step: int32(s.Step), Steps: int32(s.Steps), //nolint:gosec // counts
+			Loss: s.Loss, PeakMib: int32(s.PeakMiB)}}}) //nolint:gosec // MiB
+	}
+	artifact := func(kind string, seq int, data []byte, last bool) error {
+		if !send(&llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_TrainArtifact{TrainArtifact: &llmgwv1.TrainArtifact{
+			JobId: id, Kind: kind, Seq: int32(seq), Data: data, Last: last}}}) { //nolint:gosec // chunk count
+			return ctx.Err()
+		}
+		return nil
+	}
+	a.log.Info("training an adapter", "job", id, "base", m.GetBaseRepo())
+	err := a.trainer.Run(ctx, train.Job{ID: id, BaseRepo: m.GetBaseRepo(), BaseRevision: m.GetBaseRevision(),
+		Data: m.GetData(), Params: m.GetParams(), GPUMemoryMiB: int(m.GetGpuMemoryMib()), GGUF: m.GetGguf()}, status, artifact)
+	if err != nil {
+		a.log.Warn("training failed", "job", id, "err", err)
+		status(train.Status{State: "failed", Message: err.Error()})
+		return
+	}
+	a.log.Info("trained an adapter", "job", id)
+	status(train.Status{State: "done"})
+}
@@ -249,7 +249,14 @@ func pgEnv(dsn string) ([]string, error) {
 		set("PGPASSWORD", pw)
 	}
 	set("PGDATABASE", strings.TrimPrefix(u.Path, "/"))
-	set("PGSSLMODE", u.Query().Get("sslmode"))
+	// The TLS settings go with the mode: verify-full without the cluster's
+	// authority sent libpq looking for ~/.postgresql/root.crt, and the dump
+	// failed on every control plane whose store is replicated.
+	q := u.Query()
+	set("PGSSLMODE", q.Get("sslmode"))
+	set("PGSSLROOTCERT", q.Get("sslrootcert"))
+	set("PGSSLCERT", q.Get("sslcert"))
+	set("PGSSLKEY", q.Get("sslkey"))
 	// A standby if there is one, else whichever answers — not the address's
 	// read-write: a dump only reads, it need not load the primary, and one
 	// that insists on the primary fails exactly while the cluster is failing
@@ -238,3 +238,17 @@ func TestAStoreOfSeveralHostsIsGivenToPgDumpAsLibpqLists(t *testing.T) {
 		t.Error("an empty host in the list was accepted")
 	}
 }
+
+func TestPgDumpIsGivenTheClusterAuthorityAlongWithVerifyFull(t *testing.T) {
+	// Without it libpq looked for ~/.postgresql/root.crt and the dump failed.
+	env, err := pgEnv("postgres://llmgw:p@10.0.0.149:5432,10.0.0.150:5432/llmgw?sslmode=verify-full&sslrootcert=%2Fetc%2Fllmgw%2Fpki%2Fca.crt&target_session_attrs=read-write")
+	if err != nil {
+		t.Fatal(err)
+	}
+	got := strings.Join(env, " ")
+	for _, want := range []string{"PGSSLMODE=verify-full", "PGSSLROOTCERT=/etc/llmgw/pki/ca.crt"} {
+		if !strings.Contains(got, want) {
+			t.Errorf("%s is not in %s", want, got)
+		}
+	}
+}
@@ -116,6 +116,11 @@ func (c *ctlClient) do(ctx context.Context, method, path string, body, out any)
 	if out == nil {
 		return nil
 	}
+	// A stream, as it comes: NDJSON a long run writes line by line.
+	if w, ok := out.(io.Writer); ok {
+		_, err := io.Copy(w, resp.Body)
+		return err
+	}
 	// An older control plane answers a path it does not know with its
 	// dashboard: a page, which read as JSON said only "invalid character '<'".
 	if strings.HasPrefix(resp.Header.Get("Content-Type"), "text/html") {
@@ -181,6 +186,7 @@ func newCtlCmd() *cobra.Command {
 		newQuotaCmd(client, &asJSON),
 		newGuardCmd(client, &asJSON),
 		newKBCmd(client, &asJSON),
+		newTrainCmd(client, &asJSON),
 		newConfigCmd(client, &asJSON),
 		newAuditCmd(client, &asJSON),
 		newNotifyCmd(client, &asJSON),
@@ -7,12 +7,14 @@ import (
 	"fmt"
 	"net/netip"
 	"os"
+	"strings"
 	"time"

 	"github.com/spf13/cobra"

 	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
 )

 // firewallStateFile is where a control plane writes what the cluster's
@@ -20,11 +22,9 @@ import (
 // the server itself runs as llmgw and cannot change the ruleset.
 const firewallStateFile = "/var/lib/llmgw/firewall.json"

-// firewallState is what is written there.
-type firewallState struct {
-	Enabled  bool     `json:"enabled"`
-	Networks []string `json:"networks,omitempty"`
-}
+// firewallState is what is written there: the same as the mirror is told
+// and keeps beside its models.
+type firewallState = mirror.FirewallState

 // writeFirewallState records the settings' firewall for this machine's sync
 // unit. Without the state directory — a server not run by systemd — there
@@ -46,25 +46,34 @@ func writeFirewallState(cfg config.Server) error {

 // The units that apply it: a path unit watching the file, and the oneshot it
 // starts.
-const (
-	firewallSyncPathUnit = "llmgw-firewall-sync.path"
-	firewallSyncUnitFile = `[Unit]
-Description=llmgw firewall: apply the cluster's settings on this control plane
+const firewallSyncPathUnit = "llmgw-firewall-sync.path"
+
+// firewallSyncUnits are the two units for a state file: on a control plane
+// the one its server writes, on a mirror the one the control plane has the
+// mirror write.
+func firewallSyncUnits(state string) (service, pathUnit string) {
+	arg := ""
+	if state != firewallStateFile {
+		arg = " --state " + state
+	}
+	service = `[Unit]
+Description=llmgw firewall: apply the cluster's settings on this machine

 [Service]
 Type=oneshot
-ExecStart=/usr/local/bin/llmgw firewall sync
+ExecStart=/usr/local/bin/llmgw firewall sync` + arg + `
 `
-	firewallSyncPathFile = `[Unit]
-Description=llmgw firewall: follow the cluster's settings on this control plane
+	pathUnit = `[Unit]
+Description=llmgw firewall: follow the cluster's settings on this machine

 [Path]
-PathChanged=` + firewallStateFile + `
+PathChanged=` + state + `

 [Install]
 WantedBy=multi-user.target
 `
-)
+	return service, pathUnit
+}

 func newFirewallCmd() *cobra.Command {
 	c := &cobra.Command{
@@ -76,6 +85,7 @@ func newFirewallCmd() *cobra.Command {
 			"which " + firewallSyncPathUnit + " applies with 'llmgw firewall sync'.",
 	}
 	var install bool
+	state := firewallStateFile
 	sync := &cobra.Command{
 		Use:   "sync",
 		Short: "Apply what the cluster's settings say of the firewall, here (root)",
@@ -85,21 +95,21 @@ func newFirewallCmd() *cobra.Command {
 				return errors.New("the firewall is changed as root: run it with sudo")
 			}
 			if install {
-				if err := installFirewallSync(cmd.Context(), firewall.Local{}); err != nil {
+				if err := installFirewallSync(cmd.Context(), firewall.Local{}, state); err != nil {
 					return err
 				}
 				fmt.Fprintln(cmd.OutOrStdout(), firewallSyncPathUnit+" follows the cluster's settings here")
-				if _, err := os.Stat(firewallStateFile); err != nil {
-					return nil // the server writes it when it starts
+				if _, err := os.Stat(state); err != nil {
+					return nil // written when the settings are first told
 				}
 			}
-			b, err := os.ReadFile(firewallStateFile)
+			b, err := os.ReadFile(state)
 			if err != nil {
 				return err
 			}
 			var st firewallState
 			if err := json.Unmarshal(b, &st); err != nil {
-				return fmt.Errorf("%s: %w", firewallStateFile, err)
+				return fmt.Errorf("%s: %w", state, err)
 			}
 			ctx, cancel := context.WithTimeout(cmd.Context(), 2*time.Minute)
 			defer cancel()
@@ -112,14 +122,17 @@ func newFirewallCmd() *cobra.Command {
 		},
 	}
 	sync.Flags().BoolVar(&install, "install", false, "first install the units that run this whenever the settings change")
+	sync.Flags().StringVar(&state, "state", firewallStateFile,
+		"the file the settings are written to: a control plane's, or "+mirror.FirewallFile+" in a mirror's directory")
 	c.AddCommand(sync)
 	return c
 }

 // installFirewallSync writes the sync units on a machine and has the path
 // unit follow the state file.
-func installFirewallSync(ctx context.Context, r firewall.Runner) error {
-	body := firewallSyncUnitFile + "\x00" + firewallSyncPathFile
+func installFirewallSync(ctx context.Context, r firewall.Runner, state string) error {
+	service, pathUnit := firewallSyncUnits(state)
+	body := service + "\x00" + pathUnit
 	out, err := r.Run(ctx, `set -e
 t=$(mktemp -d); trap 'rm -rf "$t"' EXIT
 cat > "$t/all"
@@ -172,5 +185,20 @@ func syncFirewall(ctx context.Context, r firewall.Runner, st firewallState) (str
 			return "", err
 		}
 	}
+	// The cluster's machines: a node on a network this one has not let in
+	// yet, which the mirror's files are for.
+	if len(st.Toward) > 0 {
+		added, err := firewall.Admit(ctx, r, st.Toward)
+		if err != nil {
+			return "", err
+		}
+		if len(added) > 0 {
+			shown := make([]string, len(added))
+			for i, p := range added {
+				shown[i] = p.String()
+			}
+			return "llmgw's firewall is on here, and now lets in " + strings.Join(shown, ", "), nil
+		}
+	}
 	return "llmgw's firewall is on here", nil
 }
@@ -2,8 +2,11 @@ package cli

 import (
 	"context"
+	"net/netip"
 	"strings"
 	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
 )

 type scriptLog struct {
@@ -49,3 +52,32 @@ func TestAControlPlaneFollowsTheFirewallSetting(t *testing.T) {
 		t.Errorf("on: %q %v", msg, err)
 	}
 }
+
+func TestAMachineLetsInTheClusterMachinesItIsTold(t *testing.T) {
+	// The rules hold 10.0.0.0/24; a node on 192.168.7.0/24 is let in, as
+	// the route to it says, and one already in is not asked about.
+	r := &scriptLog{reply: func(s string) string {
+		switch {
+		case strings.Contains(s, "echo loaded"):
+			return "loaded\n"
+		case strings.Contains(s, "cat /etc/llmgw/firewall.nft"):
+			return firewallRulesFor("10.0.0.0/24")
+		case strings.Contains(s, "ip -o route get 192.168.7.20"):
+			return "192.168.7.20 dev eth1 src 192.168.7.1 uid 0\n2: eth1    inet 192.168.7.1/24 brd 192.168.7.255 scope global eth1\n"
+		case strings.Contains(s, "nft -f"):
+			return "changed\n"
+		}
+		return ""
+	}}
+	msg, err := syncFirewall(context.Background(), r, firewallState{Enabled: true, Toward: []string{"10.0.0.147", "192.168.7.20"}})
+	if err != nil || !strings.Contains(msg, "192.168.7.0/24") {
+		t.Fatalf("%q %v", msg, err)
+	}
+	if strings.Contains(strings.Join(r.scripts, "\n"), "route get 10.0.0.147") {
+		t.Error("asked the route to a machine already let in")
+	}
+}
+
+func firewallRulesFor(nets string) string {
+	return firewall.Script(firewall.Spec{Ports: []firewall.Port{firewall.MirrorFiles}, Allow: []netip.Prefix{netip.MustParsePrefix(nets)}})
+}
@@ -0,0 +1,146 @@
+package cli
+
+import (
+	"context"
+	"encoding/json"
+	"log/slog"
+	"net"
+	"net/netip"
+	"net/url"
+	"slices"
+	"sync"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// firewallStateKey is where, in the store, the firewall's state is kept
+// for the store's members to read.
+const firewallStateKey = "firewall"
+
+// firewallPublisher tells the machines that cannot ask a control plane
+// what the cluster's settings say of their firewall: the mirror, over its
+// API, and the store's members, through the store they hold a copy of.
+// Nodes are told over their agents' streams and control planes by their
+// own servers. On the leader only, at start, at every change to the
+// firewall's settings, and every few minutes for nodes come and gone.
+type firewallPublisher struct {
+	settings func() config.Server
+	st       *store.DB
+	nodes    func() []string
+	leader   func() bool
+	log      *slog.Logger
+
+	mu     sync.Mutex
+	mirror *mirror.Client
+	kick   chan struct{}
+}
+
+func newFirewallPublisher(settings func() config.Server, st *store.DB, nodes func() []string,
+	leader func() bool, log *slog.Logger) *firewallPublisher {
+	return &firewallPublisher{settings: settings, st: st, nodes: nodes, leader: leader, log: log,
+		kick: make(chan struct{}, 1)}
+}
+
+// setMirror has the mirror told too.
+func (p *firewallPublisher) setMirror(m *mirror.Client) {
+	p.mu.Lock()
+	p.mirror = m
+	p.mu.Unlock()
+	p.Kick()
+}
+
+// Kick has the state worked out and told again now.
+func (p *firewallPublisher) Kick() {
+	select {
+	case p.kick <- struct{}{}:
+	default:
+	}
+}
+
+func (p *firewallPublisher) run(ctx context.Context, every time.Duration) {
+	t := time.NewTicker(every)
+	defer t.Stop()
+	for {
+		p.once(ctx)
+		select {
+		case <-ctx.Done():
+			return
+		case <-t.C:
+		case <-p.kick:
+		}
+	}
+}
+
+// state is what the settings say now, with the cluster's machines.
+func (p *firewallPublisher) state(ctx context.Context) mirror.FirewallState {
+	cfg := p.settings()
+	st := mirror.FirewallState{Enabled: cfg.Firewall.Enabled, Networks: slices.Clone(cfg.FirewallNetworks)}
+	var toward []string
+	if p.nodes != nil {
+		toward = append(toward, p.nodes()...)
+	}
+	if p.st != nil {
+		if cps, err := p.st.ControlPlanes(ctx); err == nil {
+			for _, c := range cps {
+				toward = append(toward, c.URL)
+			}
+		}
+	}
+	for _, t := range toward {
+		if a, ok := hostAddr(t); ok {
+			st.Toward = append(st.Toward, a)
+		}
+	}
+	slices.Sort(st.Toward)
+	st.Toward = slices.Compact(st.Toward)
+	return st
+}
+
+// hostAddr is the address in an address, host:port or URL; names are not
+// looked up here.
+func hostAddr(s string) (string, bool) {
+	if u, err := url.Parse(s); err == nil && u.Host != "" {
+		s = u.Hostname()
+	} else if h, _, err := net.SplitHostPort(s); err == nil {
+		s = h
+	}
+	a, err := netip.ParseAddr(s)
+	if err != nil {
+		return "", false
+	}
+	return a.Unmap().String(), true
+}
+
+func (p *firewallPublisher) once(ctx context.Context) {
+	if p.leader != nil && !p.leader() {
+		return
+	}
+	ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
+	defer cancel()
+	st := p.state(ctx)
+	p.mu.Lock()
+	m := p.mirror
+	p.mu.Unlock()
+	if m != nil {
+		changed, err := m.SetFirewall(ctx, st)
+		switch {
+		case err != nil:
+			p.log.Warn("could not tell the mirror what the settings say of its firewall", "err", err)
+		case changed:
+			p.log.Info("the mirror's firewall follows the cluster's settings", "enabled", st.Enabled)
+		}
+	}
+	if p.st != nil {
+		b, _ := json.Marshal(st)
+		changed, err := p.st.PutClusterState(ctx, firewallStateKey, string(b))
+		switch {
+		case err != nil:
+			p.log.Warn("could not record the firewall's state for the store's members", "err", err)
+		case changed:
+			p.log.Info("the store's members' firewall follows the cluster's settings", "enabled", st.Enabled)
+		}
+	}
+}
@@ -0,0 +1,59 @@
+package cli
+
+import (
+	"context"
+	"encoding/json"
+	"log/slog"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
+)
+
+func TestTheLeaderTellsTheMirrorAndTheStoreWhatTheFirewallSettingsSay(t *testing.T) {
+	ctx := context.Background()
+	st, err := store.OpenSQLite(ctx, filepath.Join(t.TempDir(), "s.db"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer st.Close()
+	if err := st.PutControlPlane(ctx, store.ControlPlane{URL: "https://10.0.0.150:7443", Name: "ctl02"}); err != nil {
+		t.Fatal(err)
+	}
+	root := t.TempDir()
+	srv := httptest.NewServer(mirror.NewAPI(mirror.NewManager(root, slog.New(slog.DiscardHandler)), "http://m:8081").Handler())
+	defer srv.Close()
+	mc, _ := mirror.NewClient(srv.URL, tlsconf.Config{}, 0)
+
+	cfg := config.Server{Firewall: config.FirewallSettings{Enabled: false}, FirewallNetworks: []string{"192.168.5.0/24"}}
+	leader := false
+	p := newFirewallPublisher(func() config.Server { return cfg }, st,
+		func() []string { return []string{"10.0.0.147", "", "10.0.0.147", "node-by-name"} },
+		func() bool { return leader }, slog.New(slog.DiscardHandler))
+	p.setMirror(mc)
+
+	p.once(ctx)
+	if _, err := st.ClusterState(ctx, firewallStateKey); err == nil {
+		t.Error("a standby told the store")
+	}
+	leader = true
+	p.once(ctx)
+	var got mirror.FirewallState
+	v, err := st.ClusterState(ctx, firewallStateKey)
+	if err != nil || json.Unmarshal([]byte(v), &got) != nil {
+		t.Fatalf("the store holds %q %v", v, err)
+	}
+	if got.Enabled || got.Networks[0] != "192.168.5.0/24" || len(got.Toward) != 2 ||
+		got.Toward[0] != "10.0.0.147" || got.Toward[1] != "10.0.0.150" {
+		t.Errorf("state %+v", got)
+	}
+	b, err := os.ReadFile(filepath.Join(root, mirror.FirewallFile))
+	if err != nil || string(b) != v {
+		t.Errorf("the mirror holds %s, the store %s", b, v)
+	}
+}
@@ -1,10 +1,14 @@
 package cli

 import (
+	"bytes"
 	"encoding/json"
 	"fmt"
+	"io"
 	"net/url"
+	"os"
 	"strings"
+	"time"

 	"github.com/spf13/cobra"

@@ -212,6 +216,45 @@ func newKBCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 	def.Flags().BoolVar(&clear, "clear", false, "no default any more")
 	c.AddCommand(def)

+	var dsModel, dsOut string
+	var dsPer, dsMax int
+	dataset := &cobra.Command{
+		Use:   "dataset COLLECTION --model MODEL",
+		Short: "Write training data from a collection: questions and answers, for an adapter",
+		Long: "A chat model of the cluster reads each chunk of the collection and writes\n" +
+			"questions a person might ask of it, with the answers it holds, one JSON object\n" +
+			"a line ({\"q\", \"a\", \"doc\", \"ord\"}). Read it through and correct it before\n" +
+			"training on it (llmgw ctl train): what the model got wrong, an adapter learns.\n\n" +
+			"An adapter learns how answers sound — the organisation's words, format and\n" +
+			"tone — far better than what they say; the facts stay the knowledge base's.",
+		Example: "  llmgw ctl kb dataset Acme/policies --model qwen2.5-coder-14b-instruct --out policies.jsonl",
+		Args:    cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			w := cmd.OutOrStdout()
+			if dsOut != "" {
+				f, err := os.Create(dsOut)
+				if err != nil {
+					return err
+				}
+				defer f.Close()
+				w = f
+			}
+			lw := &datasetWriter{pairs: w, problems: cmd.ErrOrStderr()}
+			body, _ := json.Marshal(adminapi.KBDatasetRequest{Model: dsModel, PerChunk: dsPer, MaxChunks: dsMax})
+			err := client().patient(6*time.Hour).do(cmd.Context(), "POST",
+				"/api/v1/kb/collections/"+url.PathEscape(args[0])+"/dataset", json.RawMessage(body), lw)
+			lw.flush()
+			fmt.Fprintf(cmd.ErrOrStderr(), "%d pairs from %d chunks; %d chunks gave none\n", lw.n, len(lw.chunks), lw.failed)
+			return err
+		},
+	}
+	dataset.Flags().StringVar(&dsModel, "model", "", "the chat model that writes the questions and answers")
+	dataset.Flags().StringVar(&dsOut, "out", "", "the file to write (default: standard output)")
+	dataset.Flags().IntVar(&dsPer, "per-chunk", 8, "pairs from each chunk")
+	dataset.Flags().IntVar(&dsMax, "max-chunks", 500, "the most chunks read")
+	_ = dataset.MarkFlagRequired("model")
+	c.AddCommand(dataset)
+
 	var k int
 	search := &cobra.Command{
 		Use:   "search COLLECTION QUERY...",
@@ -240,3 +283,50 @@ func newKBCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 	c.AddCommand(search)
 	return c
 }
+
+// datasetWriter splits the dataset stream: pairs to the file, problems to
+// the terminal, counted as they pass.
+type datasetWriter struct {
+	pairs, problems io.Writer
+	buf             []byte
+	n, failed       int
+	chunks          map[string]bool
+}
+
+func (d *datasetWriter) Write(p []byte) (int, error) {
+	d.buf = append(d.buf, p...)
+	for {
+		i := bytes.IndexByte(d.buf, '\n')
+		if i < 0 {
+			return len(p), nil
+		}
+		d.line(d.buf[:i])
+		d.buf = d.buf[i+1:]
+	}
+}
+
+func (d *datasetWriter) flush() {
+	if len(bytes.TrimSpace(d.buf)) > 0 {
+		d.line(d.buf)
+	}
+	d.buf = nil
+}
+
+func (d *datasetWriter) line(b []byte) {
+	var l adminapi.KBDatasetLine
+	if json.Unmarshal(b, &l) != nil {
+		return
+	}
+	if d.chunks == nil {
+		d.chunks = map[string]bool{}
+	}
+	d.chunks[fmt.Sprintf("%s#%d", l.Doc, l.Ord)] = true
+	if l.Error != "" {
+		d.failed++
+		fmt.Fprintf(d.problems, "  %s #%d: %s\n", l.Doc, l.Ord, l.Error)
+		return
+	}
+	d.n++
+	out, _ := json.Marshal(l)
+	fmt.Fprintln(d.pairs, string(out))
+}
@@ -8,6 +8,7 @@ import (
 	"net"
 	"net/netip"
 	"os"
+	"path/filepath"
 	"runtime"
 	"strings"
 	"time"
@@ -65,6 +66,14 @@ Safe to run again: it changes only what differs.`,
 			}
 			if !noFirewall {
 				mirrorFirewall(cmd.Context(), cmd.OutOrStdout(), controlPlanes)
+				// And after, as the cluster's settings say: the control plane
+				// tells the mirror, which writes it beside its models.
+				state := filepath.Join(*dir, mirror.FirewallFile)
+				if err := installFirewallSync(cmd.Context(), firewall.Local{}, state); err != nil {
+					fmt.Fprintf(cmd.OutOrStdout(), "  !!    firewall: %v; the cluster's settings will not change it here\n", err)
+				} else {
+					fmt.Fprintln(cmd.OutOrStdout(), "  ok    firewall: follows the cluster's settings ("+firewallSyncPathUnit+")")
+				}
 			}
 			return nil
 		},
@@ -49,6 +49,7 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/fleetmetrics"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelsvc"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/train"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/kvevents"
@@ -232,6 +233,9 @@ func newServerCmd() *cobra.Command {
 			var agentHandler http.Handler
 			var agentPath string
 			var admin *control.Controller
+			// What the settings say of the firewall, for the mirror and the
+			// store's members; nil without a control plane.
+			var fwPub *firewallPublisher
 			updateMetrics := &modelserve.UpdateCollector{}
 			// The store's cluster, when it is one; read by the metrics and
 			// by `ctl status`.
@@ -275,6 +279,19 @@ func newServerCmd() *cobra.Command {
 				ctl.Reconciler().SetNotifier(notifier)
 				admin = ctl
 				agentPath, agentHandler = svc.Handler()
+				fwPub = newFirewallPublisher(func() config.Server { return *current.Load() }, rt.Store(),
+					func() []string {
+						var out []string
+						for _, n := range ctl.Inventory().Nodes() {
+							out = append(out, n.MgmtAddr)
+						}
+						return out
+					},
+					func() bool { return ctl.Leadership() == nil || ctl.Leadership().IsLeader() }, log)
+				go fwPub.run(ctx, 5*time.Minute)
+				if l := ctl.Leadership(); l != nil {
+					l.OnLead(fwPub.Kick)
+				}

 				// Before AdminHandler, which is where the registry is built —
 				// registering after it silently publishes nothing, which is
@@ -387,6 +404,13 @@ func newServerCmd() *cobra.Command {
 					// Knowledge bases, with the cluster's own models to embed a
 					// probe (a collection's dimension) and an operator's search.
 					api.SetKB(&adminapi.KBAdmin{Store: st, Embed: embedder(rt.Replay)})
+					// LoRA adapters trained on the nodes.
+					if agentSvc != nil {
+						trainer := train.New(st, agentSvc, log)
+						agentSvc.SetTrain(trainer)
+						go trainer.Run(ctx)
+						api.SetTrain(&adminapi.TrainAdmin{Store: st, Service: trainer})
+					}
 					// The cluster's settings, each change checked against this
 					// machine's own keys before it is kept.
 					api.SetSettings(&adminapi.Settings{Store: st, LocalKeys: config.LocalKeys(),
@@ -449,6 +473,7 @@ func newServerCmd() *cobra.Command {
 					api.SetUsage(st)
 					// Capacity quotas, set and read over this API.
 					api.SetQuotas(st)
+					api.SetTenancy(st)
 				}
 				api.Register(adminMux)

@@ -510,6 +535,16 @@ func newServerCmd() *cobra.Command {
 					return fmt.Errorf("the model mirror: %w", err)
 				}
 				rt.SetMirror(mc)
+				if fwPub != nil {
+					fwPub.setMirror(mc)
+				}
+				// Adapters trained on the nodes are kept there too; sending one
+				// takes longer than the mirror's usual answers.
+				if adminAPI != nil {
+					if up, err := mirror.NewClient(cfg.Mirror.API, cfg.Mirror.TLS, 10*time.Minute); err == nil {
+						adminAPI.SetTrainMirror(up)
+					}
+				}
 				// Browsing the registry is what turns "paste a resolve URL"
 				// into "find the model". It needs a way out to the internet
 				// from this machine, which the mirror's own fetching does not.
@@ -727,6 +762,9 @@ func newServerCmd() *cobra.Command {
 						if agentSvc != nil {
 							agentSvc.BroadcastFirewall()
 						}
+						if fwPub != nil {
+							fwPub.Kick()
+						}
 					}
 					if slices.ContainsFunc(changed, func(c string) bool { return strings.HasPrefix(c, "agents") }) && agentSvc != nil {
 						// A moment apart per node, so off this goroutine.
@@ -160,7 +160,7 @@ func (sa *serverAdd) firewall(ctx context.Context) error {
 		return err
 	}
 	// What applies the cluster's firewall setting there when it changes.
-	if err := installFirewallSync(ctx, sa.remote); err != nil {
+	if err := installFirewallSync(ctx, sa.remote, firewallStateFile); err != nil {
 		fmt.Fprintf(sa.out, "  !!    firewall: %v\n", err)
 	}
 	if !cfg.Firewall.Enabled {
@@ -93,6 +93,11 @@ func (in *serverInit) storeFirewall(ctx context.Context, members []pgha.Node, se
 				} else {
 					in.kept("firewall on %s", m.Name)
 				}
+				// And after, as the cluster's settings say, read from the
+				// member's own copy of the store.
+				if err := firewall.InstallFollower(ctx, m.Remote); err != nil {
+					fmt.Fprintf(in.out, "  !!    firewall on %s: %v; the cluster's settings will not change it\n", m.Name, err)
+				}
 				continue
 			}
 		}
@@ -135,5 +140,5 @@ func loopbackListen(listen string) bool {
 // ensureFirewallSync installs the units that apply the cluster's firewall
 // setting on this control plane.
 func (in *serverInit) ensureFirewallSync(ctx context.Context) error {
-	return installFirewallSync(ctx, firewall.Local{})
+	return installFirewallSync(ctx, firewall.Local{}, firewallStateFile)
 }
@@ -16,6 +16,7 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/pgha"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
 )

@@ -348,15 +349,23 @@ func (in *serverInit) setupPostgres(ctx context.Context) (string, error) {
 	}

 	if installed {
-		// Its defaults assume it owns the machine. A gateway swapped out to
-		// give the database cache is worse than a smaller cache, and a store
-		// whose working set is megabytes needs little. Only a PostgreSQL init
-		// installed itself: one that was already here may serve something
-		// else that needs its connections.
-		for _, q := range []string{
-			"ALTER SYSTEM SET shared_buffers = '96MB'",
-			"ALTER SYSTEM SET max_connections = 40",
-		} {
+		// Sized to the machine's memory, as one it shares with the gateway:
+		// a gateway swapped out to give the database cache is worse than a
+		// smaller cache, but a knowledge base's vector index is searched
+		// well only from memory. Only a PostgreSQL init installed itself:
+		// one that was already here may serve something else that needs its
+		// connections.
+		meminfo, _ := os.ReadFile("/proc/meminfo")
+		total, ok := pgha.MemTotal(string(meminfo))
+		if !ok {
+			total = 1 << 30
+		}
+		mem := pgha.MemorySettings(total, true)
+		qs := []string{"ALTER SYSTEM SET max_connections = 40"}
+		for _, k := range []string{"shared_buffers", "effective_cache_size", "maintenance_work_mem", "work_mem"} {
+			qs = append(qs, "ALTER SYSTEM SET "+k+" = '"+mem[k]+"'")
+		}
+		for _, q := range qs {
 			if _, err := in.psql(ctx, q); err != nil {
 				return "", err
 			}
@@ -367,7 +376,8 @@ func (in *serverInit) setupPostgres(ctx context.Context) (string, error) {
 		if err := in.waitForPostgres(ctx); err != nil {
 			return "", err
 		}
-		in.done("PostgreSQL sized for a machine it shares (shared_buffers 96MB, 40 connections)")
+		in.done("PostgreSQL sized for a machine it shares, with %d MB of memory (shared_buffers %s, work_mem %s, 40 connections)",
+			total>>20, mem["shared_buffers"], mem["work_mem"])
 	}

 	in.setupPgvector(ctx, discovery.DiscoverHost())
@@ -0,0 +1,293 @@
+package cli
+
+import (
+	"encoding/json"
+	"fmt"
+	"io"
+	"os"
+	"strings"
+	"time"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+func newTrainCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
+	c := &cobra.Command{
+		Use:   "train",
+		Short: "Train LoRA adapters on the nodes, on question/answer pairs",
+		Long: "A node trains an adapter for a catalog model on pairs (llmgw ctl kb dataset\n" +
+			"writes them from a knowledge base; read them through first). It trains with\n" +
+			"its installed vLLM's torch and a small overlay, as the engines' user, on a card\n" +
+			"with --gpu-memory free — beside an engine, if it leaves that much. What it\n" +
+			"makes is kept by the control plane: llmgw ctl train download, or publish it\n" +
+			"as the model's adapter.\n\n" +
+			"An adapter learns how answers sound — words, format, tone — far better than\n" +
+			"what they say. Facts belong to the knowledge base.",
+	}
+
+	var model, base, rev, node, data, owner string
+	var public bool
+	var gpuMem int
+	var epochs, rank int
+	var lr float64
+	var noGGUF bool
+	start := &cobra.Command{
+		Use:   "start NAME --model MODEL --base-repo OWNER/NAME --node NODE --data FILE",
+		Short: "Queue an adapter's training on a node",
+		Example: "  llmgw ctl train start policies-tone --model qwen2.5-0.5b-instruct \\\n" +
+			"    --base-repo Qwen/Qwen2.5-0.5B-Instruct --node cuda01 --gpu-memory 2300 --data policies.jsonl",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			b, err := os.ReadFile(data)
+			if err != nil {
+				return err
+			}
+			req := adminapi.TrainRequest{Name: args[0], Model: model, BaseRepo: base, BaseRevision: rev, Node: node,
+				GPUMemoryMiB: gpuMem, Data: string(b), Params: map[string]any{}, Owner: owner, Public: public}
+			if cmd.Flags().Changed("epochs") {
+				req.Params["epochs"] = epochs
+			}
+			if cmd.Flags().Changed("rank") {
+				req.Params["rank"] = rank
+			}
+			if cmd.Flags().Changed("lr") {
+				req.Params["lr"] = lr
+			}
+			if noGGUF {
+				f := false
+				req.GGUF = &f
+			}
+			body, _ := json.Marshal(req)
+			var v adminapi.TrainJobView
+			if err := client().patient(2*time.Minute).do(cmd.Context(), "POST", "/api/v1/train/jobs", json.RawMessage(body), &v); err != nil {
+				return err
+			}
+			who := "every caller of " + v.Model
+			if v.Owner != "" {
+				who = v.Owner + " only"
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "queued %s: %s for %s on %s, %d pairs; it will be seen by %s\n",
+				v.ID, v.Name, v.Model, v.Node, v.Pairs, who)
+			fmt.Fprintf(cmd.OutOrStdout(), "follow it with: llmgw ctl train show %s --follow\n", v.ID)
+			return nil
+		},
+	}
+	f := start.Flags()
+	f.StringVar(&model, "model", "", "the catalog model the adapter is for")
+	f.StringVar(&base, "base-repo", "", "the weights to train on, as safetensors on Hugging Face (owner/name)")
+	f.StringVar(&rev, "base-revision", "", "a revision of them (default: the newest)")
+	f.StringVar(&node, "node", "", "the node that trains")
+	f.IntVar(&gpuMem, "gpu-memory", 4096, "MiB of a card the job needs free")
+	f.StringVar(&data, "data", "", "the pairs, NDJSON of {\"q\", \"a\"}")
+	f.IntVar(&epochs, "epochs", 6, "passes over the pairs")
+	f.IntVar(&rank, "rank", 16, "the adapter's rank")
+	f.Float64Var(&lr, "lr", 2e-4, "the learning rate")
+	f.BoolVar(&noGGUF, "no-gguf", false, "skip the llama.cpp GGUF, even for a model llama.cpp serves")
+	f.StringVar(&owner, "owner", "", "who the adapter belongs to — org:, team:, project: (path or ID), user: or key: — and only they see it "+
+		"(default: the knowledge base's owner, when the pairs come from one)")
+	f.BoolVar(&public, "public", false, "every caller of the model sees it; refused for pairs from a knowledge base")
+	for _, req := range []string{"model", "base-repo", "node", "data"} {
+		_ = start.MarkFlagRequired(req)
+	}
+	c.AddCommand(start)
+
+	c.AddCommand(&cobra.Command{
+		Use:   "list",
+		Short: "Training jobs, newest first",
+		Args:  cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			var jobs []adminapi.TrainJobView
+			if err := client().get(cmd.Context(), "/api/v1/train/jobs", &jobs); err != nil {
+				return err
+			}
+			if *asJSON {
+				return json.NewEncoder(cmd.OutOrStdout()).Encode(jobs)
+			}
+			if len(jobs) == 0 {
+				fmt.Fprintln(cmd.OutOrStdout(), "No training jobs (llmgw ctl train start).")
+				return nil
+			}
+			w := tab()
+			fmt.Fprintln(w, "JOB\tADAPTER\tMODEL\tOWNER\tNODE\tSTATE\tPROGRESS\tLOSS\tPEAK\tCREATED")
+			for _, j := range jobs {
+				fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n", j.ID, j.Name, j.Model, or(j.Owner, "everyone"), j.Node, j.State,
+					trainProgress(j), trainLoss(j), trainPeak(j), j.CreatedAt.Local().Format("01-02 15:04"))
+			}
+			return w.Flush()
+		},
+	})
+
+	var follow bool
+	show := &cobra.Command{
+		Use:   "show JOB",
+		Short: "How a training job goes, and what it made",
+		Args:  cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			out := cmd.OutOrStdout()
+			last := ""
+			for {
+				var j adminapi.TrainJobView
+				if err := client().get(cmd.Context(), "/api/v1/train/jobs/"+args[0], &j); err != nil {
+					return err
+				}
+				if *asJSON {
+					return json.NewEncoder(out).Encode(j)
+				}
+				line := fmt.Sprintf("%s  %s  %s  loss %s  peak %s  %s", j.State, trainProgress(j), j.Node, trainLoss(j), trainPeak(j), j.Message)
+				if !follow {
+					fmt.Fprintf(out, "%s: adapter %s for %s, owned by %s, on %s (base %s), %d pairs\n", j.ID, j.Name, j.Model,
+						or(j.Owner, "everyone"), j.Node, j.BaseRepo, j.Pairs)
+					fmt.Fprintln(out, line)
+					if len(j.Artifacts) > 0 {
+						fmt.Fprintf(out, "made: %s (llmgw ctl train download %s --kind KIND)\n", strings.Join(j.Artifacts, ", "), j.ID)
+					}
+					return nil
+				}
+				if line != last {
+					fmt.Fprintf(out, "%s  %s\n", time.Now().Format("15:04:05"), line)
+					last = line
+				}
+				if store.TrainOver(j.State) {
+					if j.State != store.TrainDone {
+						return fmt.Errorf("%s %s", j.ID, j.State)
+					}
+					return nil
+				}
+				select {
+				case <-cmd.Context().Done():
+					return nil
+				case <-time.After(3 * time.Second):
+				}
+			}
+		},
+	}
+	show.Flags().BoolVarP(&follow, "follow", "f", false, "follow it until it ends")
+	c.AddCommand(show)
+
+	c.AddCommand(&cobra.Command{
+		Use:   "cancel JOB",
+		Short: "Stop a training job",
+		Args:  cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			if err := client().do(cmd.Context(), "POST", "/api/v1/train/jobs/"+args[0]+"/cancel", nil, nil); err != nil {
+				return err
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "%s cancelled\n", args[0])
+			return nil
+		},
+	})
+	c.AddCommand(&cobra.Command{
+		Use:   "rm JOB",
+		Short: "Remove an ended training job and what it made",
+		Args:  cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			if err := client().do(cmd.Context(), "DELETE", "/api/v1/train/jobs/"+args[0], nil, nil); err != nil {
+				return err
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "%s removed\n", args[0])
+			return nil
+		},
+	})
+
+	var restart bool
+	publish := &cobra.Command{
+		Use:   "publish JOB",
+		Short: "Make what a finished job made its model's adapter, kept on the mirror",
+		Long: "The GGUF for a model llama.cpp serves, the PEFT files for one vLLM serves, are kept\n" +
+			"on the mirror and added to the model's adapters; a request then asks for the\n" +
+			"adapter by its name, as a model. llama.cpp loads adapters when a replica starts:\n" +
+			"--restart replaces the model's replicas, one at a time, to load it.",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			var v adminapi.TrainPublishView
+			if err := client().patient(15*time.Minute).do(cmd.Context(), "POST", "/api/v1/train/jobs/"+args[0]+"/publish", nil, &v); err != nil {
+				return err
+			}
+			out := cmd.OutOrStdout()
+			fmt.Fprintf(out, "%s is %s's adapter, from %s\n", v.Adapter, v.Model, v.Source)
+			if v.Owner != "" {
+				fmt.Fprintf(out, "only %s sees it; to anyone else it is a model that does not exist\n", v.Owner)
+			}
+			if restart && strings.Contains(v.Note, "restart") {
+				body, _ := json.Marshal(adminapi.RestartRequest{Models: []string{v.Model}})
+				var res adminapi.RestartResult
+				if err := client().post(cmd.Context(), "/api/v1/restart", body, &res); err != nil {
+					return err
+				}
+				fmt.Fprintf(out, "%d replica(s) of %s restarting to load it; then ask for model %s\n", len(res.Queued), v.Model, v.Adapter)
+				return nil
+			}
+			fmt.Fprintln(out, v.Note)
+			return nil
+		},
+	}
+	publish.Flags().BoolVar(&restart, "restart", false, "restart the model's replicas to load it (llama.cpp)")
+	c.AddCommand(publish)
+
+	var kind, outFile string
+	download := &cobra.Command{
+		Use:   "download JOB",
+		Short: "Save what a training job made: the PEFT adapter (peft) or the GGUF (gguf)",
+		Args:  cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			if outFile == "" {
+				outFile = args[0] + ".tar.gz"
+				if kind == "gguf" {
+					outFile = args[0] + ".gguf"
+				}
+			}
+			f, err := os.Create(outFile)
+			if err != nil {
+				return err
+			}
+			defer f.Close()
+			cw := &countWriter{w: f}
+			if err := client().patient(10*time.Minute).do(cmd.Context(), "GET",
+				"/api/v1/train/jobs/"+args[0]+"/artifacts/"+kind, nil, io.Writer(cw)); err != nil {
+				_ = os.Remove(outFile)
+				return err
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "%s: %s\n", outFile, humanBytes(cw.n))
+			return nil
+		},
+	}
+	download.Flags().StringVar(&kind, "kind", "peft", "peft (a gzipped tar of the PEFT adapter) or gguf")
+	download.Flags().StringVarP(&outFile, "out", "o", "", "the file to write")
+	c.AddCommand(download)
+	return c
+}
+
+type countWriter struct {
+	w io.Writer
+	n int64
+}
+
+func (c *countWriter) Write(p []byte) (int, error) {
+	n, err := c.w.Write(p)
+	c.n += int64(n)
+	return n, err
+}
+
+func trainProgress(j adminapi.TrainJobView) string {
+	if j.Steps == 0 {
+		return "-"
+	}
+	return fmt.Sprintf("%d/%d", j.Step, j.Steps)
+}
+
+func trainLoss(j adminapi.TrainJobView) string {
+	if j.Loss == 0 {
+		return "-"
+	}
+	return fmt.Sprintf("%.3f", j.Loss)
+}
+
+func trainPeak(j adminapi.TrainJobView) string {
+	if j.PeakMiB == 0 {
+		return "-"
+	}
+	return fmt.Sprintf("%d MiB", j.PeakMiB)
+}
@@ -70,7 +70,11 @@ type API struct {
 	// guard is the guardrail policies' library and attachments.
 	guard *GuardAdmin
 	// kb is knowledge bases; nil without a store.
-	kb       *KBAdmin
+	kb *KBAdmin
+	// tenancy is organizations, keys and people; nil without a store.
+	tenancy TenancyStore
+	// train is adapter training; nil without a store and agents.
+	train    *TrainAdmin
 	settings *Settings
 	// restartAgents tells every connected agent to start again.
 	restartAgents func(reason string) []string
@@ -117,6 +121,8 @@ func (a *API) Register(mux Mux) {
 	mux.HandleFunc("POST /api/v1/models/{name}/resources", a.setResources)
 	a.registerQuotas(mux)
 	a.registerKB(mux)
+	a.registerTenancy(mux)
+	a.registerTrain(mux)
 	a.registerGuard(mux)
 	a.registerSettings(mux)
 	mux.HandleFunc("POST /api/v1/agents/restart", a.restartAgentsHandler)
@@ -1963,32 +1969,9 @@ func (a *API) setAdapters(w http.ResponseWriter, r *http.Request) {
 		writeError(w, http.StatusBadRequest, "no model "+name)
 		return
 	}
-	base, baseErr := gguf.KindFromURL(r.Context(), safedial.Client(safedial.ControlPlane, 30*time.Second), m.Source)
-	have := map[string]bool{}
-	for _, ad := range m.Adapters {
-		have[ad.Source] = true
-	}
-	// The file is read only where llama.cpp will load it: a vLLM adapter is
-	// a PEFT directory, which has no header to read, and vLLM itself refuses
-	// one that does not fit the model when it is loaded.
-	ggufs := slices.ContainsFunc(m.Profiles, func(p catalog.Profile) bool { return p.Engine == "llamacpp" })
-	for _, ad := range req.Adapters {
-		if have[ad.Source] || !ggufs {
-			continue // already checked when it was attached
-		}
-		h, err := gguf.KindFromURL(r.Context(), safedial.Client(safedial.ControlPlane, 30*time.Second), ad.Source)
-		switch {
-		case err != nil:
-			writeError(w, http.StatusBadRequest, fmt.Sprintf("adapter %s: %v", ad.Name, err))
-			return
-		case !h.IsAdapter():
-			writeError(w, http.StatusBadRequest, fmt.Sprintf("adapter %s: %s is a model, not a LoRA adapter", ad.Name, ad.Source))
-			return
-		case baseErr == nil && h.Architecture != base.Architecture:
-			writeError(w, http.StatusBadRequest, fmt.Sprintf("adapter %s is for %s models and %s is %s",
-				ad.Name, h.Architecture, m.Name, base.Architecture))
-			return
-		}
+	if err := checkAdapters(r.Context(), m, req.Adapters); err != nil {
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
 	}
 	out, err := a.models.SetAdapters(r.Context(), m.Name, req.Adapters, actorOf(r))
 	if err != nil {
@@ -2070,3 +2053,36 @@ func multiNodeReadiness(n inventory.Node) string {
 	}
 	return "ready"
 }
+
+// checkAdapters reads each adapter a model does not have yet, where
+// llama.cpp will load it, to refuse one that is not a LoRA adapter for the
+// model's architecture: such an adapter is a replica that fails to start,
+// found minutes later on a node.
+func checkAdapters(ctx context.Context, m catalog.Model, adapters []catalog.Adapter) error {
+	// A vLLM adapter is a PEFT directory, which has no header to read, and
+	// vLLM itself refuses one that does not fit the model when it is loaded.
+	if !slices.ContainsFunc(m.Profiles, func(p catalog.Profile) bool { return p.Engine == "llamacpp" }) {
+		return nil
+	}
+	client := safedial.Client(safedial.ControlPlane, 30*time.Second)
+	base, baseErr := gguf.KindFromURL(ctx, client, m.Source)
+	have := map[string]bool{}
+	for _, ad := range m.Adapters {
+		have[ad.Source] = true
+	}
+	for _, ad := range adapters {
+		if have[ad.Source] {
+			continue // checked when it was attached
+		}
+		h, err := gguf.KindFromURL(ctx, client, ad.Source)
+		switch {
+		case err != nil:
+			return fmt.Errorf("adapter %s: %v", ad.Name, err)
+		case !h.IsAdapter():
+			return fmt.Errorf("adapter %s: %s is a model, not a LoRA adapter", ad.Name, ad.Source)
+		case baseErr == nil && h.Architecture != base.Architecture:
+			return fmt.Errorf("adapter %s is for %s models and %s is %s", ad.Name, h.Architecture, m.Name, base.Architecture)
+		}
+	}
+	return nil
+}
@@ -30,6 +30,9 @@ type KBStore interface {
 	DeleteDocument(ctx context.Context, collection, name string) (bool, error)
 	SearchChunks(ctx context.Context, q store.KBSearch) ([]store.KBHit, error)
 	SetKBDefault(ctx context.Context, d store.KBDefault) error
+	CollectionChunks(ctx context.Context, c store.KBCollection, limit int) ([]store.KBHit, error)
+	CollectionsFor(ctx context.Context, e store.Effective, write bool) ([]store.KBCollection, error)
+	EffectiveFor(ctx context.Context, scope store.Scope, id, org string) (store.Effective, error)
 	KBDefaults(ctx context.Context) ([]store.KBDefault, error)
 }

@@ -161,6 +164,7 @@ func (a *API) registerKB(mux Mux) {
 	mux.HandleFunc("DELETE /api/v1/kb/grants", a.kbGrant(false))
 	mux.HandleFunc("POST /api/v1/kb/search", a.kbSearch)
 	mux.HandleFunc("POST /api/v1/kb/defaults", a.kbSetDefault)
+	mux.HandleFunc("POST /api/v1/kb/collections/{id}/dataset", a.kbDataset)
 }

 // kbReady answers for a gateway without knowledge bases, and says whether
@@ -0,0 +1,147 @@
+package adminapi
+
+import (
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"regexp"
+	"strings"
+)
+
+// Training data from a collection (BACKLOG F4): questions a person might
+// ask of each chunk and the answers it holds, written by a model of the
+// cluster, for people to read through before an adapter is trained on them.
+// What an adapter learns well is how the organisation's answers sound; what
+// they say is still the knowledge base's to give.
+
+// KBDatasetRequest asks for training data from a collection.
+type KBDatasetRequest struct {
+	// Model writes the questions and answers: a chat model of the catalog.
+	Model string `json:"model"`
+	// PerChunk is how many pairs each chunk gives; 8 when 0.
+	PerChunk int `json:"per_chunk,omitempty"`
+	// MaxChunks bounds how many chunks are read; 500 when 0.
+	MaxChunks int `json:"max_chunks,omitempty"`
+}
+
+// KBDatasetLine is one line of the answer: a pair, or why a chunk gave none.
+type KBDatasetLine struct {
+	Q   string `json:"q,omitempty"`
+	A   string `json:"a,omitempty"`
+	Doc string `json:"doc"`
+	Ord int    `json:"ord"`
+	// Collection is where the pair came from: an adapter trained on it
+	// belongs to a part of the tree that may read it.
+	Collection string `json:"collection,omitempty"`
+	Error      string `json:"error,omitempty"`
+}
+
+const kbDatasetPrompt = `From the passage of an organisation's document below, write %d different questions an employee might ask, each with a short, correct answer.
+Rules: use only what the passage says. Phrase the questions differently from each other (formal, everyday, short, indirect). Answers are one or two sentences and keep the passage's numbers exactly. Leave out e-mail addresses and people's names. Write in the passage's own language.
+Output: one JSON object per line, {"q": "...", "a": "..."}, and nothing else.
+
+DOCUMENT %s:
+%s`
+
+var thinkBlock = regexp.MustCompile(`(?s)<think>.*?</think>`)
+
+func (a *API) kbDataset(w http.ResponseWriter, r *http.Request) {
+	c, ok := a.kbCollection(w, r)
+	if !ok {
+		return
+	}
+	if a.replay == nil {
+		writeError(w, http.StatusNotImplemented, "no router here to ask a model")
+		return
+	}
+	var in KBDatasetRequest
+	if err := json.NewDecoder(r.Body).Decode(&in); err != nil {
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	}
+	m, found := a.ctl.Catalog().Resolve(in.Model)
+	if !found {
+		writeError(w, http.StatusBadRequest, "no model "+in.Model+" in the catalog")
+		return
+	}
+	if m.Capabilities.Embeddings {
+		writeError(w, http.StatusBadRequest, m.Name+" embeds; questions are written by a chat model")
+		return
+	}
+	per := in.PerChunk
+	if per <= 0 || per > 30 {
+		per = 8
+	}
+	limit := in.MaxChunks
+	if limit <= 0 || limit > 5000 {
+		limit = 500
+	}
+	chunks, err := a.kb.Store.CollectionChunks(r.Context(), c, limit)
+	if err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	w.Header().Set("Content-Type", "application/x-ndjson")
+	w.WriteHeader(http.StatusOK)
+	enc := json.NewEncoder(w)
+	flush := func() {
+		if f, ok := w.(http.Flusher); ok {
+			f.Flush()
+		}
+	}
+	for _, ch := range chunks {
+		if r.Context().Err() != nil {
+			return
+		}
+		body, _ := json.Marshal(map[string]any{"model": m.Name, "temperature": 0.7, "max_tokens": 200 * per,
+			"messages": []map[string]string{{"role": "user", "content": fmt.Sprintf(kbDatasetPrompt, per, ch.Document, ch.Body)}}})
+		status, out, _, err := a.replay(r.Context(), "/v1/chat/completions", m.Name, body)
+		line := KBDatasetLine{Doc: ch.Document, Ord: ch.Ord, Collection: c.ID}
+		if err == nil && status != http.StatusOK {
+			err = fmt.Errorf("%d: %s", status, strings.TrimSpace(string(out)))
+		}
+		if err != nil {
+			line.Error = err.Error()
+			_ = enc.Encode(line)
+			flush()
+			continue
+		}
+		pairs := datasetPairs(out)
+		if len(pairs) == 0 {
+			line.Error = "the model gave no pairs it could read"
+			_ = enc.Encode(line)
+		}
+		for _, p := range pairs {
+			p.Doc, p.Ord, p.Collection = ch.Document, ch.Ord, c.ID
+			_ = enc.Encode(p)
+		}
+		flush()
+	}
+}
+
+// datasetPairs reads the {"q","a"} lines out of a chat completion.
+func datasetPairs(completion []byte) []KBDatasetLine {
+	var res struct {
+		Choices []struct {
+			Message struct {
+				Content string `json:"content"`
+			} `json:"message"`
+		} `json:"choices"`
+	}
+	if json.Unmarshal(completion, &res) != nil || len(res.Choices) == 0 {
+		return nil
+	}
+	text := thinkBlock.ReplaceAllString(res.Choices[0].Message.Content, "")
+	var out []KBDatasetLine
+	for _, line := range strings.Split(text, "\n") {
+		line = strings.TrimSuffix(strings.TrimSpace(line), ",")
+		if !strings.HasPrefix(line, "{") || !strings.HasSuffix(line, "}") {
+			continue
+		}
+		var p struct{ Q, A string }
+		if json.Unmarshal([]byte(line), &p) == nil && strings.TrimSpace(p.Q) != "" && strings.TrimSpace(p.A) != "" {
+			out = append(out, KBDatasetLine{Q: strings.TrimSpace(p.Q), A: strings.TrimSpace(p.A)})
+		}
+	}
+	return out
+}
@@ -0,0 +1,91 @@
+package adminapi
+
+import (
+	"context"
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+func TestPairsAreReadOutOfWhatAModelWrote(t *testing.T) {
+	completion := `{"choices":[{"message":{"content":"<think>let me see</think>\n{\"q\": \"Babalık izni kaç gün?\", \"a\": \"10 iş günü.\"},\nnot json\n{\"q\": \"\", \"a\": \"empty question\"}\n{\"q\": \"Ne zaman?\", \"a\": \"Doğumdan sonra.\"}"}}]}`
+	got := datasetPairs([]byte(completion))
+	if len(got) != 2 || got[0].Q != "Babalık izni kaç gün?" || got[1].A != "Doğumdan sonra." {
+		t.Errorf("pairs %+v", got)
+	}
+	if datasetPairs([]byte(`{"choices":[]}`)) != nil {
+		t.Error("pairs from no choice")
+	}
+}
+
+func TestADatasetIsWrittenChunkByChunk(t *testing.T) {
+	dsn := os.Getenv("LLMGW_TEST_POSTGRES")
+	if dsn == "" {
+		t.Skip("set LLMGW_TEST_POSTGRES to a pgvector database to run this")
+	}
+	ctx := context.Background()
+	s, err := store.OpenPostgres(ctx, dsn)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer s.Close()
+	if s.KnowledgeAvailable() != nil {
+		t.Skip("no pgvector")
+	}
+	models := append(kbModels(), catalog.Model{Name: "writer", Source: "/w",
+		Profiles: []catalog.Profile{{Name: "cpu", GPUClass: "h100-80g", TP: 1}}})
+	a, _ := setup(t, models)
+	a.SetKB(&KBAdmin{Store: s})
+	asked := 0
+	a.SetReplayer(func(_ context.Context, path, model string, body []byte) (int, []byte, time.Duration, error) {
+		asked++
+		if path != "/v1/chat/completions" || model != "writer" || !strings.Contains(string(body), "Write in the passage") {
+			t.Errorf("asked %s %s %s", path, model, body)
+		}
+		if strings.Contains(string(body), "bozuk") {
+			return 200, []byte(`{"choices":[{"message":{"content":"sorry"}}]}`), 0, nil
+		}
+		return 200, []byte(`{"choices":[{"message":{"content":"{\"q\":\"Soru?\",\"a\":\"Cevap.\"}"}}]}`), 0, nil
+	})
+	org, _ := s.CreateOrg(ctx, "ds-"+strings.ToLower(store.NewID("")))
+	c, err := s.CreateCollection(ctx, store.KBCollection{OwnerScope: store.ScopeOrg, OwnerID: org, Name: "d", EmbedModel: "bge-m3", Dims: 3})
+	if err != nil {
+		t.Fatal(err)
+	}
+	for _, d := range []string{"a.md", "b.md"} {
+		body := "iyi metin"
+		if d == "b.md" {
+			body = "bozuk metin"
+		}
+		if _, _, err := s.PutDocument(ctx, c, store.KBDocument{Name: d, Digest: d}, []store.KBChunk{{Body: body, Embedding: []float32{1, 0, 0}}}); err != nil {
+			t.Fatal(err)
+		}
+	}
+	mux := http.NewServeMux()
+	a.Register(mux)
+	rec := httptest.NewRecorder()
+	mux.ServeHTTP(rec, httptest.NewRequest("POST", "/api/v1/kb/collections/"+c.ID+"/dataset", strings.NewReader(`{"model":"writer","per_chunk":3}`)))
+	var lines []KBDatasetLine
+	for _, l := range strings.Split(strings.TrimSpace(rec.Body.String()), "\n") {
+		var x KBDatasetLine
+		_ = json.Unmarshal([]byte(l), &x)
+		lines = append(lines, x)
+	}
+	if rec.Code != 200 || asked != 2 || len(lines) != 2 || lines[0].Q != "Soru?" || lines[0].Doc != "a.md" ||
+		lines[1].Doc != "b.md" || lines[1].Error == "" {
+		t.Errorf("%d asked %d: %+v", rec.Code, asked, lines)
+	}
+	// An embedding model writes no questions.
+	rec = httptest.NewRecorder()
+	mux.ServeHTTP(rec, httptest.NewRequest("POST", "/api/v1/kb/collections/"+c.ID+"/dataset", strings.NewReader(`{"model":"bge-m3"}`)))
+	if rec.Code != http.StatusBadRequest {
+		t.Errorf("an embedding model: %d", rec.Code)
+	}
+}
@@ -67,6 +67,12 @@ var PathParams = map[string]string{
 	"federation/{peer}": "The federation peer's name, as `llmgw ctl federation list` lists it.",
 	"holds/{domain}":    "The domain as level:name, rack:r12-07, as `llmgw ctl facility` lists it.",
 	"collections/{id}":  "The collection's ID, its name when that is unique, or ORG/NAME.",
+	"orgs/{id}":         "The organization's ID or name.",
+	"teams/{id}":        "The team's ID.",
+	"projects/{id}":     "The project's ID.",
+	"keys/{id}":         "The key's ID, as listed (not the key itself).",
+	"users/{id}":        "The person's ID or email.",
+	"artifacts/{kind}":  "What the job made: peft, a gzipped tar of the PEFT adapter, or gguf, the adapter for llama.cpp.",
 }

 // Routes is every endpoint Register mounts.
@@ -288,10 +294,77 @@ var Routes = []Route{
 		Description: "On an organization, team, project or key; the narrowest one with a default wins. No collections clears it. " +
 			"A request's own X-Llmgw-Knowledge overrides it, and X-Llmgw-Knowledge: none turns it off.",
 		Request: KBDefaultRequest{}, Response: KBDefaultView{}},
+	{Method: "POST", Path: "/api/v1/kb/collections/{id}/dataset", Summary: "Training data from a collection: questions and answers a chat model writes from each chunk",
+		Description: "NDJSON, a line per pair as each chunk is read, or a line with error for a chunk that gave none. For people to correct before an adapter is trained on it.",
+		Request:     KBDatasetRequest{}, Response: KBDatasetLine{}},
 	{Method: "POST", Path: "/api/v1/kb/search", Summary: "Try a search of a collection, as an administrator",
 		Description: "No tenancy applies: this is for checking what a collection holds and finds.",
 		Request:     KBSearchRequest{}, Response: []KBHitView{}},

+	// LoRA adapters trained on the nodes.
+	// Organizations, teams, projects, keys and people.
+	{Method: "GET", Path: "/api/v1/orgs", Summary: "The tenancy tree: every organization, its teams and their projects",
+		Response: []OrgView{}},
+	{Method: "POST", Path: "/api/v1/orgs", Summary: "Add an organization",
+		Request: TenancyCreateRequest{}, Response: TenancyItem{}, Status: http.StatusCreated},
+	{Method: "PATCH", Path: "/api/v1/orgs/{id}", Summary: "Rename an organization",
+		Request: TenancyChangeRequest{}, Response: TenancyItem{}},
+	{Method: "DELETE", Path: "/api/v1/orgs/{id}", Summary: "Remove an organization that has no teams",
+		Response: map[string]any{}},
+	{Method: "POST", Path: "/api/v1/teams", Summary: "Add a team to an organization",
+		Request: TenancyCreateRequest{}, Response: TenancyItem{}, Status: http.StatusCreated},
+	{Method: "PATCH", Path: "/api/v1/teams/{id}", Summary: "Rename a team, or move it to another organization",
+		Description: "Everything under it moves too, and from then on inherits the new organization's limits.",
+		Request:     TenancyChangeRequest{}, Response: TenancyItem{}},
+	{Method: "DELETE", Path: "/api/v1/teams/{id}", Summary: "Remove a team that has no projects",
+		Response: map[string]any{}},
+	{Method: "POST", Path: "/api/v1/projects", Summary: "Add a project to a team",
+		Request: TenancyCreateRequest{}, Response: TenancyItem{}, Status: http.StatusCreated},
+	{Method: "PATCH", Path: "/api/v1/projects/{id}", Summary: "Rename a project, or move it to another team",
+		Request: TenancyChangeRequest{}, Response: TenancyItem{}},
+	{Method: "DELETE", Path: "/api/v1/projects/{id}", Summary: "Remove a project with no people, keys or collections",
+		Description: "A project that ever had a key keeps it, revoked, because usage refers to it; rename such a project instead.",
+		Response:    map[string]any{}},
+	{Method: "GET", Path: "/api/v1/keys", Summary: "API keys, with their project and owner; never the keys themselves",
+		Query:    []Param{{Name: "project", Description: "Only this project's keys: ORG/TEAM/PROJECT or its ID."}},
+		Response: []KeyView{}},
+	{Method: "POST", Path: "/api/v1/keys", Summary: "Issue an API key",
+		Description: "The answer holds the key, once: only its hash is kept. A lifetime longer than a scope above it allows is refused.",
+		Request:     KeyCreateRequest{}, Response: KeyCreated{}, Status: http.StatusCreated},
+	{Method: "GET", Path: "/api/v1/keys/{id}", Summary: "One API key", Response: KeyView{}},
+	{Method: "DELETE", Path: "/api/v1/keys/{id}", Summary: "Revoke an API key",
+		Description: "A soft delete: usage refers to it. A gateway that has it cached accepts it until auth.cache_ttl.",
+		Response:    map[string]any{}},
+	{Method: "GET", Path: "/api/v1/users", Summary: "People, with where each is a member",
+		Query:    []Param{{Name: "state", Description: "Only people in this state: pending, active or disabled."}},
+		Response: []UserView{}},
+	{Method: "GET", Path: "/api/v1/users/{id}", Summary: "One person, with where they are a member", Response: UserView{}},
+	{Method: "POST", Path: "/api/v1/users/{id}/memberships", Summary: "Put a person in an organization, team or project",
+		Description: "Already there, only the role changes.",
+		Request:     MembershipRequest{}, Response: UserView{}},
+	{Method: "DELETE", Path: "/api/v1/users/{id}/memberships", Summary: "Take a person out of an organization, team or project",
+		Description: "Their keys in projects they no longer reach are revoked.",
+		Request:     MembershipRequest{}, Response: UserView{}},
+
+	{Method: "POST", Path: "/api/v1/train/jobs", Summary: "Queue a LoRA adapter's training on a node",
+		Description: "The node trains when its agent is there, nothing else trains on it, and a card has gpu_memory_mib free; " +
+			"the pairs are NDJSON of {\"q\", \"a\"}. An adapter learns how answers sound far better than what they say.",
+		Request: TrainRequest{}, Response: TrainJobView{}, Status: http.StatusCreated},
+	{Method: "GET", Path: "/api/v1/train/jobs", Summary: "Training jobs, newest first",
+		Query:    []Param{{Name: "limit", Description: "How many; 50 when absent.", Type: "integer"}},
+		Response: []TrainJobView{}},
+	{Method: "GET", Path: "/api/v1/train/jobs/{id}", Summary: "One training job: how it goes, and what it made",
+		Response: TrainJobView{}},
+	{Method: "POST", Path: "/api/v1/train/jobs/{id}/cancel", Summary: "Stop a training job",
+		Response: TrainJobView{}},
+	{Method: "DELETE", Path: "/api/v1/train/jobs/{id}", Summary: "Remove an ended training job and what it made",
+		Response: map[string]any{}},
+	{Method: "POST", Path: "/api/v1/train/jobs/{id}/publish", Summary: "Make what a finished job made the model's adapter",
+		Description: "Kept on the mirror — the GGUF for a model llama.cpp serves, the PEFT files for one vLLM serves — and added to the model's adapters after the same check as POST /api/v1/models/{name}/adapters.",
+		Response:    TrainPublishView{}},
+	{Method: "GET", Path: "/api/v1/train/jobs/{id}/artifacts/{kind}", Summary: "What a training job made: peft (a gzipped tar) or gguf",
+		Response: map[string]any{}},
+
 	{Method: "POST", Path: "/api/v1/agents/restart", Summary: "Have every node's agent start again; their engines keep serving",
 		Response: AgentRestartView{}},

@@ -0,0 +1,658 @@
+package adminapi
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"net/http"
+	"strings"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// The tenancy tree, keys and people's memberships, administered over the
+// API as `llmgw org`, `keys` and `users` do against the store: so that
+// they can be run from elsewhere, and kept in Terraform.
+
+// TenancyStore is the part of the store the tenancy endpoints need.
+type TenancyStore interface {
+	Tree(ctx context.Context) ([]store.TreeOrg, error)
+	CreateOrg(ctx context.Context, name string) (string, error)
+	CreateTeam(ctx context.Context, orgID, name string) (string, error)
+	CreateProject(ctx context.Context, teamID, name string) (string, error)
+	RenameOrg(ctx context.Context, id, name string) error
+	RenameTeam(ctx context.Context, id, name string) error
+	RenameProject(ctx context.Context, id, name string) error
+	MoveTeam(ctx context.Context, teamID, orgID string) error
+	MoveProject(ctx context.Context, projectID, teamID string) error
+	DeleteOrg(ctx context.Context, id string) error
+	DeleteTeam(ctx context.Context, id string) error
+	DeleteProject(ctx context.Context, id string) error
+
+	AllKeys(ctx context.Context) ([]store.KeyRow, error)
+	KeyByID(ctx context.Context, id string) (store.Key, error)
+	CreateKey(ctx context.Context, k store.Key) error
+	CreateUserKey(ctx context.Context, userID string, k store.Key) error
+	RevokeKey(ctx context.Context, id string, at time.Time) error
+
+	Users(ctx context.Context, state store.UserState) ([]store.User, error)
+	UserByID(ctx context.Context, id string) (store.User, error)
+	UserByEmail(ctx context.Context, email string) (store.User, error)
+	MembershipsOf(ctx context.Context, userID string) ([]store.Membership, error)
+	AddMembership(ctx context.Context, userID string, scope store.Scope, scopeID, role, by string, at time.Time) error
+	RemoveMembership(ctx context.Context, userID string, scope store.Scope, scopeID string, at time.Time) (int, error)
+}
+
+// SetTenancy enables the tenancy endpoints. The quota store must be set
+// too: names are resolved through it.
+func (a *API) SetTenancy(t TenancyStore) { a.tenancy = t }
+
+// OrgView is an organization with what is in it.
+type OrgView struct {
+	ID      string     `json:"id"`
+	Name    string     `json:"name"`
+	Members int        `json:"members"`
+	Teams   []TeamView `json:"teams"`
+}
+
+// TeamView is a team with its projects.
+type TeamView struct {
+	ID       string        `json:"id"`
+	Name     string        `json:"name"`
+	Org      string        `json:"org"`
+	Members  int           `json:"members"`
+	Projects []ProjectView `json:"projects"`
+}
+
+// ProjectView is a project, with how many people and keys are in it.
+type ProjectView struct {
+	ID   string `json:"id"`
+	Name string `json:"name"`
+	Team string `json:"team"`
+	// Path is ORG/TEAM/PROJECT by name.
+	Path       string `json:"path"`
+	Users      int    `json:"users"`
+	Keys       int    `json:"keys"`
+	ActiveKeys int    `json:"active_keys"`
+}
+
+// TenancyCreateRequest adds an organization, a team or a project.
+type TenancyCreateRequest struct {
+	Name string `json:"name"`
+	// In is the organization a team goes in, or the team a project goes in
+	// (ORG/TEAM), by name or ID; nothing for an organization.
+	In string `json:"in,omitempty"`
+}
+
+// TenancyChangeRequest renames an organization, team or project, or moves
+// a team to another organization or a project to another team.
+type TenancyChangeRequest struct {
+	Name string `json:"name,omitempty"`
+	// In is where it moves to, by name or ID; not for an organization.
+	In string `json:"in,omitempty"`
+}
+
+// TenancyItem is an organization, team or project added or changed.
+type TenancyItem struct {
+	ID   string `json:"id"`
+	Name string `json:"name,omitempty"`
+	// In is the organization a team is in, or the team a project is in, by
+	// ID.
+	In string `json:"in,omitempty"`
+}
+
+// KeyView is a key as listed: never the key itself, which is not kept.
+type KeyView struct {
+	ID   string `json:"id"`
+	Name string `json:"name"`
+	// Prefix is the key's first characters, to tell it apart.
+	Prefix  string `json:"prefix"`
+	Project string `json:"project,omitempty"`
+	// ProjectPath is ORG/TEAM/PROJECT, or empty for a personal key.
+	ProjectPath    string    `json:"project_path,omitempty"`
+	User           string    `json:"user,omitempty"`
+	UserEmail      string    `json:"user_email,omitempty"`
+	Band           string    `json:"band,omitempty"`
+	RoutingProfile string    `json:"routing_profile,omitempty"`
+	State          string    `json:"state"`
+	CreatedAt      time.Time `json:"created_at"`
+	ExpiresAt      time.Time `json:"expires_at,omitzero"`
+	RevokedAt      time.Time `json:"revoked_at,omitzero"`
+	LastUsedAt     time.Time `json:"last_used_at,omitzero"`
+}
+
+// KeyCreateRequest issues a key: a project's, a person's in a project they
+// are a member of, or a person's own with no project.
+type KeyCreateRequest struct {
+	Name string `json:"name"`
+	// Project is ORG/TEAM/PROJECT or its ID.
+	Project string `json:"project,omitempty"`
+	// User is the person it belongs to, by ID or email.
+	User           string `json:"user,omitempty"`
+	Band           string `json:"band,omitempty"`
+	RoutingProfile string `json:"routing_profile,omitempty"`
+	// ExpiresIn is its lifetime, as 720h; none never expires, unless a
+	// scope above it limits how long keys live.
+	ExpiresIn string `json:"expires_in,omitempty"`
+}
+
+// KeyCreated is a new key, with the key itself: this answer is the only
+// time it is given.
+type KeyCreated struct {
+	KeyView
+	Secret string `json:"secret"`
+}
+
+// UserView is a person, with where they are a member.
+type UserView struct {
+	ID    string `json:"id"`
+	Email string `json:"email"`
+	Name  string `json:"name,omitempty"`
+	State string `json:"state"`
+	// Role is the person's part in running the gateway, if any: viewer,
+	// operator, billing or admin.
+	Role        string           `json:"role,omitempty"`
+	Memberships []MembershipView `json:"memberships"`
+	CreatedAt   time.Time        `json:"created_at"`
+	LastLoginAt time.Time        `json:"last_login_at,omitzero"`
+}
+
+// MembershipView is one place a person belongs.
+type MembershipView struct {
+	Target string    `json:"target"`
+	Path   string    `json:"path,omitempty"`
+	Role   string    `json:"role"`
+	At     time.Time `json:"at"`
+	By     string    `json:"by,omitempty"`
+}
+
+// MembershipRequest puts a person in an organization, team or project, or
+// with DELETE takes them out.
+type MembershipRequest struct {
+	// Target is org:NAME, team:ORG/TEAM or project:ORG/TEAM/PROJECT, or the
+	// ID after the colon.
+	Target string `json:"target"`
+	// Role is member or manager; member when absent. Ignored by DELETE.
+	Role string `json:"role,omitempty"`
+}
+
+func (a *API) registerTenancy(mux Mux) {
+	mux.HandleFunc("GET /api/v1/orgs", a.tenancyTree)
+	for _, level := range []store.Scope{store.ScopeOrg, store.ScopeTeam, store.ScopeProject} {
+		path := "/api/v1/" + string(level) + "s"
+		mux.HandleFunc("POST "+path, a.tenancyCreate(level))
+		mux.HandleFunc("PATCH "+path+"/{id}", a.tenancyChange(level))
+		mux.HandleFunc("DELETE "+path+"/{id}", a.tenancyDelete(level))
+	}
+	mux.HandleFunc("GET /api/v1/keys", a.keyList)
+	mux.HandleFunc("POST /api/v1/keys", a.keyCreate)
+	mux.HandleFunc("GET /api/v1/keys/{id}", a.keyGet)
+	mux.HandleFunc("DELETE /api/v1/keys/{id}", a.keyRevoke)
+	mux.HandleFunc("GET /api/v1/users", a.userList)
+	mux.HandleFunc("GET /api/v1/users/{id}", a.userGet)
+	mux.HandleFunc("POST /api/v1/users/{id}/memberships", a.membership(true))
+	mux.HandleFunc("DELETE /api/v1/users/{id}/memberships", a.membership(false))
+}
+
+// tenancyReady answers for a control plane without the store, and says
+// whether it did.
+func (a *API) tenancyReady(w http.ResponseWriter) bool {
+	if a.tenancy == nil || a.quotas == nil {
+		writeError(w, http.StatusNotImplemented, "organizations, keys and people need the store, which this control plane runs without")
+		return false
+	}
+	return true
+}
+
+// tenancyStatus is the status a store error answers with.
+func tenancyStatus(err error) int {
+	switch {
+	case errors.Is(err, store.ErrNotFound):
+		return http.StatusNotFound
+	case errors.Is(err, store.ErrDuplicateName), errors.Is(err, store.ErrNotEmpty):
+		return http.StatusConflict
+	}
+	return http.StatusBadRequest
+}
+
+func (a *API) tenancyTree(w http.ResponseWriter, r *http.Request) {
+	if !a.tenancyReady(w) {
+		return
+	}
+	tree, err := a.tenancy.Tree(r.Context())
+	if err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	out := []OrgView{}
+	for _, o := range tree {
+		ov := OrgView{ID: o.ID, Name: o.Name, Members: o.Members, Teams: []TeamView{}}
+		for _, t := range o.Teams {
+			tv := TeamView{ID: t.ID, Name: t.Name, Org: o.ID, Members: t.Members, Projects: []ProjectView{}}
+			for _, p := range t.Projects {
+				tv.Projects = append(tv.Projects, ProjectView{ID: p.ID, Name: p.Name, Team: t.ID,
+					Path: o.Name + "/" + t.Name + "/" + p.Name, Users: p.Users, Keys: p.Keys, ActiveKeys: p.ActiveKeys})
+			}
+			ov.Teams = append(ov.Teams, tv)
+		}
+		out = append(out, ov)
+	}
+	writeJSON(w, http.StatusOK, out)
+}
+
+// parentOf is the level something at level sits in.
+func parentOf(level store.Scope) store.Scope {
+	if level == store.ScopeProject {
+		return store.ScopeTeam
+	}
+	return store.ScopeOrg
+}
+
+// levelNoun is a level as a sentence says it.
+func levelNoun(level store.Scope) string {
+	switch level {
+	case store.ScopeOrg:
+		return "an organization"
+	case store.ScopeTeam:
+		return "a team"
+	}
+	return "a project"
+}
+
+func (a *API) tenancyCreate(level store.Scope) http.HandlerFunc {
+	return func(w http.ResponseWriter, r *http.Request) {
+		if !a.tenancyReady(w) {
+			return
+		}
+		var in TenancyCreateRequest
+		if err := json.NewDecoder(r.Body).Decode(&in); err != nil {
+			writeError(w, http.StatusBadRequest, err.Error())
+			return
+		}
+		in.Name = strings.TrimSpace(in.Name)
+		if in.Name == "" {
+			writeError(w, http.StatusBadRequest, "a name is needed")
+			return
+		}
+		ctx := r.Context()
+		var id, parent string
+		var err error
+		if level != store.ScopeOrg {
+			if in.In == "" {
+				writeError(w, http.StatusBadRequest, fmt.Sprintf("%s goes in %s: give in", levelNoun(level), levelNoun(parentOf(level))))
+				return
+			}
+			if _, parent, _, err = a.resolveScope(ctx, string(parentOf(level))+":"+in.In); err != nil {
+				writeError(w, http.StatusNotFound, err.Error())
+				return
+			}
+		}
+		switch level {
+		case store.ScopeOrg:
+			id, err = a.tenancy.CreateOrg(ctx, in.Name)
+		case store.ScopeTeam:
+			id, err = a.tenancy.CreateTeam(ctx, parent, in.Name)
+		default:
+			id, err = a.tenancy.CreateProject(ctx, parent, in.Name)
+		}
+		if errors.Is(err, store.ErrDuplicateName) {
+			writeError(w, http.StatusConflict, fmt.Sprintf("there is already %s named %q there", levelNoun(level), in.Name))
+			return
+		}
+		if err != nil {
+			writeError(w, tenancyStatus(err), err.Error())
+			return
+		}
+		writeJSON(w, http.StatusCreated, TenancyItem{ID: id, Name: in.Name, In: parent})
+	}
+}
+
+func (a *API) tenancyChange(level store.Scope) http.HandlerFunc {
+	return func(w http.ResponseWriter, r *http.Request) {
+		if !a.tenancyReady(w) {
+			return
+		}
+		var in TenancyChangeRequest
+		if err := json.NewDecoder(r.Body).Decode(&in); err != nil {
+			writeError(w, http.StatusBadRequest, err.Error())
+			return
+		}
+		ctx := r.Context()
+		_, id, _, err := a.resolveScope(ctx, string(level)+":"+r.PathValue("id"))
+		if err != nil {
+			writeError(w, http.StatusNotFound, err.Error())
+			return
+		}
+		if in.Name == "" && in.In == "" {
+			writeError(w, http.StatusBadRequest, "nothing to change: give name, in or both")
+			return
+		}
+		out := TenancyItem{ID: id}
+		if in.In != "" {
+			if level == store.ScopeOrg {
+				writeError(w, http.StatusBadRequest, "an organization is in nothing; only a team or a project moves")
+				return
+			}
+			_, to, _, err := a.resolveScope(ctx, string(parentOf(level))+":"+in.In)
+			if err != nil {
+				writeError(w, http.StatusNotFound, err.Error())
+				return
+			}
+			if level == store.ScopeTeam {
+				err = a.tenancy.MoveTeam(ctx, id, to)
+			} else {
+				err = a.tenancy.MoveProject(ctx, id, to)
+			}
+			if err != nil {
+				writeError(w, tenancyStatus(err), err.Error())
+				return
+			}
+			out.In = to
+		}
+		if name := strings.TrimSpace(in.Name); name != "" {
+			switch level {
+			case store.ScopeOrg:
+				err = a.tenancy.RenameOrg(ctx, id, name)
+			case store.ScopeTeam:
+				err = a.tenancy.RenameTeam(ctx, id, name)
+			default:
+				err = a.tenancy.RenameProject(ctx, id, name)
+			}
+			if err != nil {
+				writeError(w, tenancyStatus(err), err.Error())
+				return
+			}
+			out.Name = name
+		}
+		writeJSON(w, http.StatusOK, out)
+	}
+}
+
+func (a *API) tenancyDelete(level store.Scope) http.HandlerFunc {
+	return func(w http.ResponseWriter, r *http.Request) {
+		if !a.tenancyReady(w) {
+			return
+		}
+		ctx := r.Context()
+		_, id, _, err := a.resolveScope(ctx, string(level)+":"+r.PathValue("id"))
+		if err != nil {
+			writeError(w, http.StatusNotFound, err.Error())
+			return
+		}
+		switch level {
+		case store.ScopeOrg:
+			err = a.tenancy.DeleteOrg(ctx, id)
+		case store.ScopeTeam:
+			err = a.tenancy.DeleteTeam(ctx, id)
+		default:
+			err = a.tenancy.DeleteProject(ctx, id)
+		}
+		if err != nil {
+			// Only what is empty goes: the store says what is still in it.
+			writeError(w, tenancyStatus(err), err.Error())
+			return
+		}
+		writeJSON(w, http.StatusOK, map[string]any{"id": id, "deleted": true})
+	}
+}
+
+// keyState is a key's state at an instant.
+func keyState(k store.Key, now time.Time) string {
+	switch {
+	case !k.RevokedAt.IsZero():
+		return "revoked"
+	case !k.ExpiresAt.IsZero() && !now.Before(k.ExpiresAt):
+		return "expired"
+	}
+	return "active"
+}
+
+func keyView(k store.Key, path, email string, now time.Time) KeyView {
+	// The store says "Acme / ml / search"; a path given to the API is
+	// written Acme/ml/search, and is answered the same way.
+	path = strings.ReplaceAll(path, " / ", "/")
+	return KeyView{ID: k.ID, Name: k.Name, Prefix: k.Prefix, Project: k.ProjectID, ProjectPath: path,
+		User: k.UserID, UserEmail: email, Band: k.Band, RoutingProfile: k.RoutingProfile, State: keyState(k, now),
+		CreatedAt: k.CreatedAt, ExpiresAt: k.ExpiresAt, RevokedAt: k.RevokedAt, LastUsedAt: k.LastUsedAt}
+}
+
+func (a *API) keyList(w http.ResponseWriter, r *http.Request) {
+	if !a.tenancyReady(w) {
+		return
+	}
+	project := ""
+	if p := r.URL.Query().Get("project"); p != "" {
+		var err error
+		if _, project, _, err = a.resolveScope(r.Context(), "project:"+p); err != nil {
+			writeError(w, http.StatusNotFound, err.Error())
+			return
+		}
+	}
+	rows, err := a.tenancy.AllKeys(r.Context())
+	if err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	now := time.Now()
+	out := []KeyView{}
+	for _, k := range rows {
+		if project == "" || k.ProjectID == project {
+			out = append(out, keyView(k.Key, k.ProjectPath, k.OwnerEmail, now))
+		}
+	}
+	writeJSON(w, http.StatusOK, out)
+}
+
+func (a *API) keyGet(w http.ResponseWriter, r *http.Request) {
+	if !a.tenancyReady(w) {
+		return
+	}
+	rows, err := a.tenancy.AllKeys(r.Context())
+	if err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	for _, k := range rows {
+		if k.ID == r.PathValue("id") {
+			writeJSON(w, http.StatusOK, keyView(k.Key, k.ProjectPath, k.OwnerEmail, time.Now()))
+			return
+		}
+	}
+	writeError(w, http.StatusNotFound, "no key "+r.PathValue("id"))
+}
+
+// findUser reads a person by ID or email.
+func (a *API) findUser(ctx context.Context, ref string) (store.User, error) {
+	ref = strings.TrimSpace(ref)
+	if u, err := a.tenancy.UserByID(ctx, ref); err == nil {
+		return u, nil
+	}
+	if strings.Contains(ref, "@") {
+		if u, err := a.tenancy.UserByEmail(ctx, ref); err == nil {
+			return u, nil
+		}
+	}
+	return store.User{}, fmt.Errorf("%w: no person %q", store.ErrNotFound, ref)
+}
+
+func (a *API) keyCreate(w http.ResponseWriter, r *http.Request) {
+	if !a.tenancyReady(w) {
+		return
+	}
+	var in KeyCreateRequest
+	if err := json.NewDecoder(r.Body).Decode(&in); err != nil {
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	}
+	ctx := r.Context()
+	if strings.TrimSpace(in.Name) == "" {
+		writeError(w, http.StatusBadRequest, "a key needs a name")
+		return
+	}
+	if in.Project == "" && in.User == "" {
+		writeError(w, http.StatusBadRequest, "a key belongs to a project, a person, or a person in a project: give project, user or both")
+		return
+	}
+	var owner store.User
+	var err error
+	if in.User != "" {
+		if owner, err = a.findUser(ctx, in.User); err != nil {
+			writeError(w, http.StatusNotFound, err.Error())
+			return
+		}
+	}
+	project, path := "", ""
+	if in.Project != "" {
+		if _, project, path, err = a.resolveScope(ctx, "project:"+in.Project); err != nil {
+			writeError(w, http.StatusNotFound, err.Error())
+			return
+		}
+	}
+	var expires time.Duration
+	if in.ExpiresIn != "" {
+		if expires, err = time.ParseDuration(in.ExpiresIn); err != nil || expires <= 0 {
+			writeError(w, http.StatusBadRequest, fmt.Sprintf("expires_in %q is not a positive duration such as 720h", in.ExpiresIn))
+			return
+		}
+	}
+	secret, k, err := store.NewKey(store.NewID("key-"), strings.TrimSpace(in.Name), project)
+	if err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	k.CreatedAt = time.Now()
+	k.Band, k.RoutingProfile = in.Band, in.RoutingProfile
+	if expires > 0 {
+		k.ExpiresAt = k.CreatedAt.Add(expires)
+	}
+	if owner.ID != "" {
+		err = a.tenancy.CreateUserKey(ctx, owner.ID, k)
+	} else {
+		err = a.tenancy.CreateKey(ctx, k)
+	}
+	switch {
+	case errors.Is(err, store.ErrNotMember):
+		writeError(w, http.StatusConflict, fmt.Sprintf("%s is not a member of %s; add the membership first", owner.Email, path))
+		return
+	case errors.Is(err, store.ErrKeyTTL):
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	case err != nil:
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	k.UserID = owner.ID
+	writeJSON(w, http.StatusCreated, KeyCreated{KeyView: keyView(k, path, owner.Email, time.Now()), Secret: secret})
+}
+
+func (a *API) keyRevoke(w http.ResponseWriter, r *http.Request) {
+	if !a.tenancyReady(w) {
+		return
+	}
+	id := r.PathValue("id")
+	if _, err := a.tenancy.KeyByID(r.Context(), id); err != nil {
+		writeError(w, tenancyStatus(err), err.Error())
+		return
+	}
+	if err := a.tenancy.RevokeKey(r.Context(), id, time.Now()); err != nil {
+		writeError(w, tenancyStatus(err), err.Error())
+		return
+	}
+	// A key held in a gateway's cache keeps working until auth.cache_ttl.
+	writeJSON(w, http.StatusOK, map[string]any{"id": id, "revoked": true})
+}
+
+func (a *API) userView(ctx context.Context, u store.User) UserView {
+	role := u.Role
+	if u.IsAdmin {
+		role = "admin"
+	}
+	v := UserView{ID: u.ID, Email: u.Email, Name: u.Name, State: string(u.State), Role: role,
+		Memberships: []MembershipView{}, CreatedAt: u.CreatedAt, LastLoginAt: u.LastLoginAt}
+	if ms, err := a.tenancy.MembershipsOf(ctx, u.ID); err == nil {
+		for _, m := range ms {
+			v.Memberships = append(v.Memberships, MembershipView{Target: string(m.Scope) + ":" + m.ScopeID,
+				Path: m.Path, Role: m.Role, At: m.AddedAt, By: m.AddedBy})
+		}
+	}
+	return v
+}
+
+func (a *API) userList(w http.ResponseWriter, r *http.Request) {
+	if !a.tenancyReady(w) {
+		return
+	}
+	us, err := a.tenancy.Users(r.Context(), store.UserState(r.URL.Query().Get("state")))
+	if err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	out := []UserView{}
+	for _, u := range us {
+		out = append(out, a.userView(r.Context(), u))
+	}
+	writeJSON(w, http.StatusOK, out)
+}
+
+func (a *API) userGet(w http.ResponseWriter, r *http.Request) {
+	if !a.tenancyReady(w) {
+		return
+	}
+	u, err := a.findUser(r.Context(), r.PathValue("id"))
+	if err != nil {
+		writeError(w, http.StatusNotFound, err.Error())
+		return
+	}
+	writeJSON(w, http.StatusOK, a.userView(r.Context(), u))
+}
+
+func (a *API) membership(add bool) http.HandlerFunc {
+	return func(w http.ResponseWriter, r *http.Request) {
+		if !a.tenancyReady(w) {
+			return
+		}
+		var in MembershipRequest
+		if err := json.NewDecoder(r.Body).Decode(&in); err != nil {
+			writeError(w, http.StatusBadRequest, err.Error())
+			return
+		}
+		ctx := r.Context()
+		u, err := a.findUser(ctx, r.PathValue("id"))
+		if err != nil {
+			writeError(w, http.StatusNotFound, err.Error())
+			return
+		}
+		scope, id, _, err := a.resolveScope(ctx, in.Target)
+		if err != nil {
+			writeError(w, http.StatusBadRequest, err.Error())
+			return
+		}
+		if add {
+			role := in.Role
+			if role == "" {
+				role = store.RoleMember
+			}
+			if role != store.RoleMember && role != store.RoleManager {
+				writeError(w, http.StatusBadRequest, fmt.Sprintf("role %q: member or manager", role))
+				return
+			}
+			if err := a.tenancy.AddMembership(ctx, u.ID, scope, id, role, actorOf(r), time.Now()); err != nil {
+				writeError(w, tenancyStatus(err), err.Error())
+				return
+			}
+		} else {
+			// The person's keys in projects they no longer reach are
+			// revoked with it, by the store.
+			if _, err := a.tenancy.RemoveMembership(ctx, u.ID, scope, id, time.Now()); err != nil {
+				if errors.Is(err, store.ErrNotFound) {
+					err = errors.New(u.Email + " is not a member there")
+				}
+				writeError(w, http.StatusNotFound, err.Error())
+				return
+			}
+		}
+		writeJSON(w, http.StatusOK, a.userView(ctx, u))
+	}
+}
@@ -0,0 +1,171 @@
+package adminapi
+
+import (
+	"context"
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+func TestTheTenancyTreeKeysAndMembershipsAreAdministered(t *testing.T) {
+	ctx := context.Background()
+	a, _ := setup(t, testModels())
+	s, err := store.OpenSQLite(ctx, filepath.Join(t.TempDir(), "s.db"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer s.Close()
+	a.SetQuotas(s)
+	a.SetTenancy(s)
+	mux := http.NewServeMux()
+	a.Register(mux)
+	do := func(method, path, body string, out any) int {
+		t.Helper()
+		rec := httptest.NewRecorder()
+		mux.ServeHTTP(rec, httptest.NewRequest(method, path, strings.NewReader(body)))
+		if out != nil && rec.Code < 300 {
+			if err := json.Unmarshal(rec.Body.Bytes(), out); err != nil {
+				t.Fatalf("%s %s: %v (%s)", method, path, err, rec.Body.String())
+			}
+		}
+		if rec.Code >= 300 {
+			t.Logf("%s %s: %d %s", method, path, rec.Code, strings.TrimSpace(rec.Body.String()))
+		}
+		return rec.Code
+	}
+	var made TenancyItem
+	if c := do("POST", "/api/v1/orgs", `{"name":"Acme"}`, &made); c != http.StatusCreated {
+		t.Fatalf("adding an organization: %d", c)
+	}
+	acme := made.ID
+	if c := do("POST", "/api/v1/orgs", `{"name":"Acme"}`, nil); c != http.StatusConflict {
+		t.Errorf("a second Acme: %d", c)
+	}
+	if c := do("POST", "/api/v1/teams", `{"name":"hr"}`, nil); c != http.StatusBadRequest {
+		t.Errorf("a team in nothing: %d", c)
+	}
+	do("POST", "/api/v1/teams", `{"name":"hr","in":"Acme"}`, &made)
+	hr := made.ID
+	if c := do("POST", "/api/v1/projects", `{"name":"crm","in":"Acme/hr"}`, &made); c != http.StatusCreated || made.In != hr {
+		t.Fatalf("adding a project: %d %v", c, made)
+	}
+	crm := made.ID
+	do("POST", "/api/v1/orgs", `{"name":"Other"}`, &made)
+	other := made.ID
+
+	// Renamed, and moved with what is in it.
+	if c := do("PATCH", "/api/v1/projects/"+crm, `{"name":"crm2"}`, nil); c != http.StatusOK {
+		t.Errorf("renaming: %d", c)
+	}
+	if c := do("PATCH", "/api/v1/teams/"+hr, `{"in":"Other"}`, nil); c != http.StatusOK {
+		t.Errorf("moving: %d", c)
+	}
+	if c := do("PATCH", "/api/v1/orgs/"+acme, `{"in":"Other"}`, nil); c != http.StatusBadRequest {
+		t.Errorf("moving an organization: %d", c)
+	}
+	var tree []OrgView
+	do("GET", "/api/v1/orgs", "", &tree)
+	if len(tree) != 2 || tree[1].ID != other || len(tree[1].Teams) != 1 ||
+		tree[1].Teams[0].Projects[0].Path != "Other/hr/crm2" {
+		t.Fatalf("the tree is %+v", tree)
+	}
+
+	// A project's key: given once, listed without it.
+	var key KeyCreated
+	rec := httptest.NewRecorder()
+	mux.ServeHTTP(rec, httptest.NewRequest("POST", "/api/v1/keys",
+		strings.NewReader(`{"name":"svc","project":"Other/hr/crm2","expires_in":"24h"}`)))
+	if rec.Code != http.StatusCreated {
+		t.Fatalf("issuing a key: %d %s", rec.Code, rec.Body.String())
+	}
+	_ = json.Unmarshal(rec.Body.Bytes(), &key)
+	if !strings.HasPrefix(key.Secret, "sk-llmgw-") || key.Project != crm || key.ExpiresAt.IsZero() {
+		t.Errorf("the key issued: %+v", key)
+	}
+	if k, err := s.KeyByHash(ctx, store.HashKey(key.Secret)); err != nil || k.ID != key.ID {
+		t.Errorf("the key does not authenticate: %v", err)
+	}
+	var keys []KeyView
+	do("GET", "/api/v1/keys?project="+crm, "", &keys)
+	if len(keys) != 1 || keys[0].ID != key.ID || keys[0].State != "active" || keys[0].ProjectPath != "Other/hr/crm2" {
+		t.Errorf("the project's keys: %+v", keys)
+	}
+	raw := httptest.NewRecorder()
+	mux.ServeHTTP(raw, httptest.NewRequest("GET", "/api/v1/keys", nil))
+	if strings.Contains(raw.Body.String(), key.Secret) || strings.Contains(raw.Body.String(), "secret") {
+		t.Error("a listing holds the key")
+	}
+	if c := do("POST", "/api/v1/keys", `{"name":"x","project":"Other/hr/crm2","expires_in":"soon"}`, nil); c != http.StatusBadRequest {
+		t.Errorf("a lifetime that is not one: %d", c)
+	}
+	if c := do("DELETE", "/api/v1/projects/"+crm, "", nil); c != http.StatusConflict {
+		t.Errorf("removing a project with a key: %d", c)
+	}
+	if c := do("DELETE", "/api/v1/keys/"+key.ID, "", nil); c != http.StatusOK {
+		t.Errorf("revoking: %d", c)
+	}
+	var one KeyView
+	do("GET", "/api/v1/keys/"+key.ID, "", &one)
+	if one.State != "revoked" {
+		t.Errorf("after revoking, %+v", one)
+	}
+
+	// A person: a key in a project only as its member, revoked when they
+	// leave it.
+	if err := s.CreateUser(ctx, store.User{ID: "u1", Email: "ayse@example.com", PasswordHash: "x",
+		State: store.UserActive, CreatedAt: time.Now()}); err != nil {
+		t.Fatal(err)
+	}
+	if c := do("POST", "/api/v1/keys", `{"name":"mine","project":"`+crm+`","user":"ayse@example.com"}`, nil); c != http.StatusConflict {
+		t.Errorf("a key where its person is no member: %d", c)
+	}
+	var u UserView
+	if c := do("POST", "/api/v1/users/ayse@example.com/memberships", `{"target":"project:Other/hr/crm2"}`, &u); c != http.StatusOK ||
+		len(u.Memberships) != 1 || u.Memberships[0].Target != "project:"+crm || u.Memberships[0].Role != "member" {
+		t.Fatalf("joining: %d %+v", c, u)
+	}
+	var mine KeyCreated
+	rec = httptest.NewRecorder()
+	mux.ServeHTTP(rec, httptest.NewRequest("POST", "/api/v1/keys",
+		strings.NewReader(`{"name":"mine","project":"`+crm+`","user":"u1"}`)))
+	_ = json.Unmarshal(rec.Body.Bytes(), &mine)
+	if rec.Code != http.StatusCreated || mine.User != "u1" {
+		t.Fatalf("a member's key: %d %s", rec.Code, rec.Body.String())
+	}
+	if c := do("DELETE", "/api/v1/users/u1/memberships", `{"target":"project:`+crm+`"}`, &u); c != http.StatusOK || len(u.Memberships) != 0 {
+		t.Errorf("leaving: %d %+v", c, u)
+	}
+	do("GET", "/api/v1/keys/"+mine.ID, "", &one)
+	if one.State != "revoked" {
+		t.Errorf("the key of a person who left: %+v", one)
+	}
+	if c := do("DELETE", "/api/v1/users/u1/memberships", `{"target":"project:`+crm+`"}`, nil); c != http.StatusNotFound {
+		t.Errorf("leaving twice: %d", c)
+	}
+	var people []UserView
+	do("GET", "/api/v1/users", "", &people)
+	if len(people) != 1 || people[0].Email != "ayse@example.com" {
+		t.Errorf("people: %+v", people)
+	}
+
+	// Only what is empty is removed.
+	if c := do("DELETE", "/api/v1/orgs/Acme", "", nil); c != http.StatusOK {
+		t.Errorf("removing the empty Acme by name: %d", c)
+	}
+	if c := do("DELETE", "/api/v1/orgs/"+other, "", nil); c != http.StatusConflict {
+		t.Errorf("removing an organization with a team: %d", c)
+	}
+}
+
+func TestTenancyNeedsTheStore(t *testing.T) {
+	a, _ := setup(t, testModels())
+	if c := call(t, a, "GET", "/api/v1/orgs", "", nil); c != http.StatusNotImplemented {
+		t.Errorf("without a store: %d", c)
+	}
+}
@@ -0,0 +1,559 @@
+package adminapi
+
+import (
+	"archive/tar"
+	"bufio"
+	"bytes"
+	"compress/gzip"
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"net/http"
+	"regexp"
+	"slices"
+	"strconv"
+	"strings"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/train"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// Training LoRA adapters (BACKLOG F4), administered: a job names the model
+// the adapter is for, the weights to train on, the node and how much of a
+// card it needs, and the pairs; the node trains when it can, and what it
+// made is kept here to download or publish.
+
+// TrainStore is where jobs are kept.
+type TrainStore interface {
+	CreateTrainJob(ctx context.Context, j store.TrainJob) error
+	TrainJobs(ctx context.Context, limit int) ([]store.TrainJob, error)
+	TrainJob(ctx context.Context, id string) (store.TrainJob, error)
+	TrainArtifact(ctx context.Context, job, kind string) ([]byte, error)
+	TrainArtifactKinds(ctx context.Context, job string) ([]string, error)
+	DeleteTrainJob(ctx context.Context, id string) error
+}
+
+// TrainAdmin is what the training endpoints need.
+type TrainAdmin struct {
+	Store   TrainStore
+	Service *train.Service
+	// Mirror keeps what a job made where the nodes fetch weights; nil
+	// leaves publishing off.
+	Mirror TrainMirror
+}
+
+// TrainMirror is the model mirror, as publishing needs it.
+type TrainMirror interface {
+	Upload(ctx context.Context, model, version, file string, body []byte, source string) (mirror.FetchResponse, error)
+}
+
+// TrainPublishView is an adapter published: where the nodes fetch it from,
+// and what to do for replicas to load it.
+type TrainPublishView struct {
+	Adapter string `json:"adapter"`
+	Model   string `json:"model"`
+	Source  string `json:"source"`
+	// Owner is who sees it; empty is every caller of the model.
+	Owner string `json:"owner,omitempty"`
+	Note  string `json:"note,omitempty"`
+}
+
+// SetTrain enables the training endpoints.
+func (a *API) SetTrain(t *TrainAdmin) { a.train = t }
+
+// SetTrainMirror gives publishing the mirror, which is set up after the
+// training endpoints are.
+func (a *API) SetTrainMirror(m TrainMirror) {
+	if a.train != nil {
+		a.train.Mirror = m
+	}
+}
+
+// TrainRequest queues a job.
+type TrainRequest struct {
+	// Name is the adapter's: what requests will ask for it by.
+	Name string `json:"name"`
+	// Model is the catalog model the adapter is for.
+	Model string `json:"model"`
+	// BaseRepo and BaseRevision are the weights trained on: the model's own
+	// on its registry, in safetensors (Hugging Face), not a GGUF.
+	BaseRepo     string `json:"base_repo"`
+	BaseRevision string `json:"base_revision,omitempty"`
+	// Node trains it; GPUMemoryMiB is what it needs of a card, which can be
+	// what an engine there leaves free.
+	Node         string `json:"node"`
+	GPUMemoryMiB int    `json:"gpu_memory_mib"`
+	// Owner is who the adapter will belong to: org:, team:, project: (by
+	// path or ID), user: or key:. Only callers there see it. When the
+	// pairs come from one knowledge base, it defaults to that collection's
+	// owner; it must be able to read every collection they come from.
+	Owner string `json:"owner,omitempty"`
+	// Public makes the adapter every caller's its model allows. Refused for
+	// pairs from a knowledge base.
+	Public bool `json:"public,omitempty"`
+	// Data is the pairs, NDJSON of {"q", "a"} (llmgw ctl kb dataset).
+	Data string `json:"data"`
+	// Params tune the trainer: rank, alpha, dropout, epochs, lr, batch,
+	// max_len, seed.
+	Params map[string]any `json:"params,omitempty"`
+	// GGUF also makes the adapter in llama.cpp's format; when absent, it
+	// does when the model is served by llama.cpp.
+	GGUF *bool `json:"gguf,omitempty"`
+}
+
+// TrainJobView is a job as listed.
+type TrainJobView struct {
+	ID           string    `json:"id"`
+	Name         string    `json:"name"`
+	Model        string    `json:"model"`
+	BaseRepo     string    `json:"base_repo"`
+	BaseRevision string    `json:"base_revision,omitempty"`
+	Node         string    `json:"node"`
+	Owner        string    `json:"owner,omitempty"`
+	GPUMemoryMiB int       `json:"gpu_memory_mib"`
+	Params       string    `json:"params,omitempty"`
+	GGUF         bool      `json:"gguf"`
+	Pairs        int       `json:"pairs"`
+	State        string    `json:"state"`
+	Step         int       `json:"step"`
+	Steps        int       `json:"steps"`
+	Loss         float64   `json:"loss"`
+	Message      string    `json:"message,omitempty"`
+	PeakMiB      int       `json:"peak_mib"`
+	CreatedAt    time.Time `json:"created_at"`
+	StartedAt    time.Time `json:"started_at,omitzero"`
+	FinishedAt   time.Time `json:"finished_at,omitzero"`
+	CreatedBy    string    `json:"created_by,omitempty"`
+	// Artifacts are what it made: peft (a gzipped tar), gguf.
+	Artifacts []string `json:"artifacts,omitempty"`
+}
+
+func trainView(j store.TrainJob) TrainJobView {
+	return TrainJobView{ID: j.ID, Name: j.Name, Model: j.Model, BaseRepo: j.BaseRepo, BaseRevision: j.BaseRevision,
+		Node: j.Node, Owner: j.Owner, GPUMemoryMiB: j.GPUMemoryMiB, Params: j.Params, GGUF: j.GGUF, Pairs: j.Pairs, State: j.State,
+		Step: j.Step, Steps: j.Steps, Loss: j.Loss, Message: j.Message, PeakMiB: j.PeakMiB, CreatedAt: j.CreatedAt,
+		StartedAt: j.StartedAt, FinishedAt: j.FinishedAt, CreatedBy: j.CreatedBy}
+}
+
+func (a *API) registerTrain(mux Mux) {
+	mux.HandleFunc("POST /api/v1/train/jobs", a.trainCreate)
+	mux.HandleFunc("GET /api/v1/train/jobs", a.trainList)
+	mux.HandleFunc("GET /api/v1/train/jobs/{id}", a.trainGet)
+	mux.HandleFunc("POST /api/v1/train/jobs/{id}/cancel", a.trainCancel)
+	mux.HandleFunc("DELETE /api/v1/train/jobs/{id}", a.trainDelete)
+	mux.HandleFunc("GET /api/v1/train/jobs/{id}/artifacts/{kind}", a.trainArtifact)
+	mux.HandleFunc("POST /api/v1/train/jobs/{id}/publish", a.trainPublish)
+}
+
+func (a *API) trainReady(w http.ResponseWriter) bool {
+	if a.train == nil {
+		writeError(w, http.StatusNotImplemented, "training needs the store and the nodes' agents, which this control plane runs without")
+		return false
+	}
+	return true
+}
+
+// trainMaxData bounds a job's pairs.
+const trainMaxData = 64 << 20
+
+var adapterName = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._-]{0,99}$`)
+
+// trainParams are the trainer's settings, with what each may be.
+var trainParams = map[string][2]float64{
+	"rank": {1, 256}, "alpha": {1, 1024}, "dropout": {0, 0.9}, "epochs": {1, 100},
+	"lr": {1e-7, 1e-2}, "batch": {1, 128}, "max_len": {16, 32768}, "seed": {0, 1 << 31},
+}
+
+func (a *API) trainCreate(w http.ResponseWriter, r *http.Request) {
+	if !a.trainReady(w) {
+		return
+	}
+	var in TrainRequest
+	if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, trainMaxData+1<<20)).Decode(&in); err != nil {
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	}
+	bad := func(format string, args ...any) {
+		writeError(w, http.StatusBadRequest, fmt.Sprintf(format, args...))
+	}
+	if !adapterName.MatchString(in.Name) {
+		bad("an adapter's name is letters, digits, . _ and -, up to 100: %q is not", in.Name)
+		return
+	}
+	m, ok := a.ctl.Catalog().Resolve(in.Model)
+	if !ok {
+		bad("no model %s in the catalog", in.Model)
+		return
+	}
+	if _, _, _, taken := a.ctl.Catalog().ResolveAdapter(in.Name); taken || in.Name == m.Name {
+		bad("%s is already a model's or an adapter's name", in.Name)
+		return
+	}
+	if strings.Count(in.BaseRepo, "/") != 1 || strings.ContainsAny(in.BaseRepo, " :") {
+		bad("base_repo is the weights' registry repository, owner/name: %q is not", in.BaseRepo)
+		return
+	}
+	if _, ok := a.ctl.Inventory().Get(in.Node); !ok {
+		bad("no node %s", in.Node)
+		return
+	}
+	if in.GPUMemoryMiB <= 0 {
+		bad("gpu_memory_mib is what the job needs of a card, in MiB")
+		return
+	}
+	pairs, colls, err := countPairs(in.Data)
+	if err != nil {
+		bad("%v", err)
+		return
+	}
+	owner, err := a.trainOwner(r.Context(), in, colls)
+	if err != nil {
+		bad("%v", err)
+		return
+	}
+	for k, v := range in.Params {
+		lim, known := trainParams[k]
+		f, isNum := v.(float64)
+		if !known || !isNum || f < lim[0] || f > lim[1] {
+			bad("params.%s: the trainer takes rank, alpha, dropout, epochs, lr, batch, max_len and seed, each a number in its range", k)
+			return
+		}
+	}
+	params := ""
+	if len(in.Params) > 0 {
+		b, _ := json.Marshal(in.Params)
+		params = string(b)
+	}
+	gguf := false
+	for _, p := range m.Profiles {
+		if p.Engine == "llamacpp" {
+			gguf = true
+		}
+	}
+	if in.GGUF != nil {
+		gguf = *in.GGUF
+	}
+	j := store.TrainJob{ID: store.NewID("train-"), Name: in.Name, Model: m.Name, BaseRepo: in.BaseRepo,
+		BaseRevision: in.BaseRevision, Node: in.Node, Owner: owner, GPUMemoryMiB: in.GPUMemoryMiB, Params: params, GGUF: gguf,
+		Data: []byte(in.Data), Pairs: pairs, CreatedBy: actorOf(r)}
+	if err := a.train.Store.CreateTrainJob(r.Context(), j); err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	// At once rather than at the next look, when the node is here.
+	go a.train.Service.Dispatch(context.WithoutCancel(r.Context()))
+	j.State = store.TrainQueued
+	writeJSON(w, http.StatusCreated, trainView(j))
+}
+
+// trainOwner is who a job's adapter will belong to, as scope:ID: one that
+// may read every knowledge base the pairs come from. Pairs from a knowledge
+// base never make an adapter everyone's.
+func (a *API) trainOwner(ctx context.Context, in TrainRequest, colls []string) (string, error) {
+	if in.Public {
+		if len(colls) > 0 {
+			return "", errors.New("these pairs come from a knowledge base, and an adapter trained on them " +
+				"carries some of what it says: it belongs to a part of the tree that may read it, not to everyone")
+		}
+		if in.Owner != "" {
+			return "", errors.New("an adapter is public or has an owner, not both")
+		}
+		return "", nil
+	}
+	if len(colls) > 0 && a.kb == nil {
+		return "", errors.New("the pairs name knowledge bases, which this control plane cannot read")
+	}
+	var found []store.KBCollection
+	for _, id := range colls {
+		c, err := a.kb.Store.Collection(ctx, id)
+		if err != nil {
+			return "", fmt.Errorf("the pairs come from collection %s, which is not here: %w", id, err)
+		}
+		found = append(found, c)
+	}
+	if in.Owner == "" {
+		if len(found) == 1 {
+			return string(found[0].OwnerScope) + ":" + found[0].OwnerID, nil
+		}
+		return "", errors.New("name the adapter's owner (org:, team:, project:, user: or key:), or make it public " +
+			"when its pairs are not a knowledge base's")
+	}
+	scope, id, err := a.kbTarget(ctx, in.Owner)
+	if err != nil {
+		return "", err
+	}
+	for _, c := range found {
+		e, err := a.kb.Store.EffectiveFor(ctx, scope, id, c.OrgID)
+		if err != nil {
+			return "", fmt.Errorf("%s: %w", in.Owner, err)
+		}
+		readable, err := a.kb.Store.CollectionsFor(ctx, e, false)
+		if err != nil {
+			return "", err
+		}
+		if !slices.ContainsFunc(readable, func(x store.KBCollection) bool { return x.ID == c.ID }) {
+			return "", fmt.Errorf("%s may not read collection %s, which the pairs come from: "+
+				"an adapter trained on it would tell its callers what it says", in.Owner, c.Name)
+		}
+	}
+	return string(scope) + ":" + id, nil
+}
+
+// countPairs checks a job's data: NDJSON of {"q", "a"}, at least one, and
+// says which knowledge bases the pairs come from.
+func countPairs(data string) (int, []string, error) {
+	if len(data) > trainMaxData {
+		return 0, nil, fmt.Errorf("the data is over %d MiB", trainMaxData>>20)
+	}
+	seen := map[string]bool{}
+	var colls []string
+	n := 0
+	sc := bufio.NewScanner(strings.NewReader(data))
+	sc.Buffer(make([]byte, 64<<10), 1<<20)
+	line := 0
+	for sc.Scan() {
+		line++
+		b := bytes.TrimSpace(sc.Bytes())
+		if len(b) == 0 {
+			continue
+		}
+		var p struct{ Q, A, Collection string }
+		if err := json.Unmarshal(b, &p); err != nil {
+			return 0, nil, fmt.Errorf("data line %d is not JSON: %v", line, err)
+		}
+		if strings.TrimSpace(p.Q) == "" || strings.TrimSpace(p.A) == "" {
+			return 0, nil, fmt.Errorf("data line %d has no q or no a", line)
+		}
+		if p.Collection != "" && !seen[p.Collection] {
+			seen[p.Collection] = true
+			colls = append(colls, p.Collection)
+		}
+		n++
+	}
+	if err := sc.Err(); err != nil {
+		return 0, nil, err
+	}
+	if n == 0 {
+		return 0, nil, errors.New("the data holds no pairs: NDJSON of {\"q\", \"a\"}, as llmgw ctl kb dataset writes")
+	}
+	return n, colls, nil
+}
+
+func (a *API) trainList(w http.ResponseWriter, r *http.Request) {
+	if !a.trainReady(w) {
+		return
+	}
+	limit := 50
+	if n, err := strconv.Atoi(r.URL.Query().Get("limit")); err == nil && n > 0 && n <= 500 {
+		limit = n
+	}
+	jobs, err := a.train.Store.TrainJobs(r.Context(), limit)
+	if err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	out := []TrainJobView{}
+	for _, j := range jobs {
+		out = append(out, trainView(j))
+	}
+	writeJSON(w, http.StatusOK, out)
+}
+
+func (a *API) trainJob(w http.ResponseWriter, r *http.Request) (store.TrainJob, bool) {
+	if !a.trainReady(w) {
+		return store.TrainJob{}, false
+	}
+	j, err := a.train.Store.TrainJob(r.Context(), r.PathValue("id"))
+	if errors.Is(err, store.ErrNotFound) {
+		writeError(w, http.StatusNotFound, "no training job "+r.PathValue("id"))
+		return j, false
+	}
+	if err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return j, false
+	}
+	return j, true
+}
+
+func (a *API) trainGet(w http.ResponseWriter, r *http.Request) {
+	j, ok := a.trainJob(w, r)
+	if !ok {
+		return
+	}
+	v := trainView(j)
+	v.Artifacts, _ = a.train.Store.TrainArtifactKinds(r.Context(), j.ID)
+	writeJSON(w, http.StatusOK, v)
+}
+
+func (a *API) trainCancel(w http.ResponseWriter, r *http.Request) {
+	j, ok := a.trainJob(w, r)
+	if !ok {
+		return
+	}
+	if err := a.train.Service.Cancel(r.Context(), j.ID); err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	j, _ = a.train.Store.TrainJob(r.Context(), j.ID)
+	writeJSON(w, http.StatusOK, trainView(j))
+}
+
+func (a *API) trainDelete(w http.ResponseWriter, r *http.Request) {
+	j, ok := a.trainJob(w, r)
+	if !ok {
+		return
+	}
+	if !store.TrainOver(j.State) {
+		writeError(w, http.StatusConflict, "the job has not ended: cancel it first")
+		return
+	}
+	if err := a.train.Store.DeleteTrainJob(r.Context(), j.ID); err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	writeJSON(w, http.StatusOK, map[string]any{"id": j.ID, "deleted": true})
+}
+
+func (a *API) trainArtifact(w http.ResponseWriter, r *http.Request) {
+	j, ok := a.trainJob(w, r)
+	if !ok {
+		return
+	}
+	kind := r.PathValue("kind")
+	b, err := a.train.Store.TrainArtifact(r.Context(), j.ID, kind)
+	if errors.Is(err, store.ErrNotFound) {
+		writeError(w, http.StatusNotFound, "the job made no "+kind)
+		return
+	}
+	if err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	name := j.Name + ".tar.gz"
+	w.Header().Set("Content-Type", "application/gzip")
+	if kind == "gguf" {
+		name = j.Name + ".gguf"
+		w.Header().Set("Content-Type", "application/octet-stream")
+	}
+	w.Header().Set("Content-Disposition", `attachment; filename="`+name+`"`)
+	w.Header().Set("Content-Length", strconv.Itoa(len(b)))
+	_, _ = w.Write(b)
+}
+
+// trainPublish puts what a job made on the mirror and makes it the model's
+// adapter: the GGUF for a model llama.cpp serves, the PEFT directory for
+// one vLLM serves.
+func (a *API) trainPublish(w http.ResponseWriter, r *http.Request) {
+	j, ok := a.trainJob(w, r)
+	if !ok {
+		return
+	}
+	if a.train.Mirror == nil {
+		writeError(w, http.StatusNotImplemented, "publishing needs the model mirror (mirror.api in the settings); "+
+			"llmgw ctl train download, then llmgw mirror import there, does it by hand")
+		return
+	}
+	if a.models == nil {
+		writeError(w, http.StatusNotImplemented, "this control plane has no store for the catalog")
+		return
+	}
+	if j.State != store.TrainDone {
+		writeError(w, http.StatusConflict, "the job has not finished: it is "+j.State)
+		return
+	}
+	m, found := a.ctl.Catalog().Resolve(j.Model)
+	if !found {
+		writeError(w, http.StatusConflict, "the model "+j.Model+" is no longer in the catalog")
+		return
+	}
+	for _, ad := range m.Adapters {
+		if ad.Name == j.Name {
+			writeError(w, http.StatusConflict, j.Name+" is already "+m.Name+"'s adapter; train under another name")
+			return
+		}
+	}
+	llama := slices.ContainsFunc(m.Profiles, func(p catalog.Profile) bool { return p.Engine == "llamacpp" })
+	source := "trained by llmgw job " + j.ID + " on " + j.Node + " from " + j.BaseRepo
+	var url string
+	if llama {
+		b, err := a.train.Store.TrainArtifact(r.Context(), j.ID, "gguf")
+		if err != nil {
+			writeError(w, http.StatusConflict, "the job made no GGUF, which llama.cpp needs; train again without --no-gguf")
+			return
+		}
+		res, err := a.train.Mirror.Upload(r.Context(), j.Name, j.ID, j.Name+".gguf", b, source)
+		if err != nil {
+			writeError(w, http.StatusBadGateway, "keeping it on the mirror: "+err.Error())
+			return
+		}
+		url = res.URL
+	} else {
+		b, err := a.train.Store.TrainArtifact(r.Context(), j.ID, "peft")
+		if err != nil {
+			writeError(w, http.StatusConflict, "the job made no PEFT adapter")
+			return
+		}
+		files, err := untarAll(b)
+		if err != nil {
+			writeError(w, http.StatusInternalServerError, err.Error())
+			return
+		}
+		for name, data := range files {
+			res, err := a.train.Mirror.Upload(r.Context(), j.Name, j.ID, name, data, source)
+			if err != nil {
+				writeError(w, http.StatusBadGateway, "keeping "+name+" on the mirror: "+err.Error())
+				return
+			}
+			url = res.URL[:strings.LastIndex(res.URL, "/")+1]
+		}
+	}
+	ad := catalog.Adapter{Name: j.Name, Source: url, Owner: j.Owner}
+	if err := checkAdapters(r.Context(), m, []catalog.Adapter{ad}); err != nil {
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	}
+	if _, err := a.models.SetAdapters(r.Context(), m.Name, append(slices.Clone(m.Adapters), ad), actorOf(r)); err != nil {
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	}
+	v := TrainPublishView{Adapter: j.Name, Model: m.Name, Source: url, Owner: j.Owner}
+	if llama {
+		v.Note = "llama.cpp loads adapters when a replica starts: restart " + m.Name + "'s replicas (llmgw ctl restart " +
+			m.Name + ") and ask for model " + j.Name
+	} else {
+		v.Note = "ask for model " + j.Name + "; a replica loads it on the first request"
+	}
+	writeJSON(w, http.StatusOK, v)
+}
+
+// untarAll reads a gzipped tar of a PEFT directory's files.
+func untarAll(b []byte) (map[string][]byte, error) {
+	gz, err := gzip.NewReader(bytes.NewReader(b))
+	if err != nil {
+		return nil, err
+	}
+	tr := tar.NewReader(gz)
+	out := map[string][]byte{}
+	for {
+		h, err := tr.Next()
+		if errors.Is(err, io.EOF) {
+			return out, nil
+		}
+		if err != nil {
+			return nil, err
+		}
+		if h.Typeflag != tar.TypeReg || strings.ContainsAny(h.Name, "/\\") {
+			continue
+		}
+		data, err := io.ReadAll(io.LimitReader(tr, 4<<30))
+		if err != nil {
+			return nil, err
+		}
+		out[h.Name] = data
+	}
+}
@@ -0,0 +1,145 @@
+package adminapi
+
+import (
+	"context"
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/train"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+type noAgents struct{}
+
+func (noAgents) Send(string, *llmgwv1.ControlMessage) error { return nil }
+func (noAgents) Connected(string) bool                      { return false }
+
+func TestATrainingJobIsCheckedBeforeItIsQueued(t *testing.T) {
+	models := append(testModels(), catalog.Model{Name: "small", Source: "/s",
+		Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp"}}})
+	a, _ := setup(t, models, gpuNode("cuda01", 1))
+	s, err := store.OpenSQLite(context.Background(), filepath.Join(t.TempDir(), "s.db"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer s.Close()
+	a.SetTrain(&TrainAdmin{Store: s, Service: train.New(s, noAgents{}, nil)})
+	mux := http.NewServeMux()
+	a.Register(mux)
+	post := func(body string) (int, string) {
+		rec := httptest.NewRecorder()
+		mux.ServeHTTP(rec, httptest.NewRequest("POST", "/api/v1/train/jobs", strings.NewReader(body)))
+		return rec.Code, rec.Body.String()
+	}
+	good := map[string]any{"name": "tone", "model": "small", "base_repo": "Qwen/Qwen2.5-0.5B-Instruct", "node": "cuda01",
+		"gpu_memory_mib": 2300, "data": "{\"q\":\"a\",\"a\":\"b\"}\n{\"q\":\"c\",\"a\":\"d\"}\n", "params": map[string]any{"epochs": 3},
+		"public": true}
+	with := func(k string, v any) string {
+		m := map[string]any{}
+		for kk, vv := range good {
+			m[kk] = vv
+		}
+		m[k] = v
+		b, _ := json.Marshal(m)
+		return string(b)
+	}
+	for _, c := range []struct {
+		field string
+		value any
+		says  string
+	}{
+		{"name", "bad name", "letters, digits"},
+		{"model", "nope", "no model nope"},
+		{"name", "small", "already a model"},
+		{"base_repo", "Qwen", "owner/name"},
+		{"node", "gone", "no node gone"},
+		{"data", "{\"q\":\"a\"}\n", "no q or no a"},
+		{"data", "", "no pairs"},
+		{"params", map[string]any{"epochs": 0}, "params.epochs"},
+		{"params", map[string]any{"momentum": 0.9}, "params.momentum"},
+		{"public", false, "name the adapter's owner"},
+		{"data", "{\"q\":\"a\",\"a\":\"b\",\"collection\":\"kb_aaaaaaaaaaaaaaaa\"}\n", "from a knowledge base"},
+	} {
+		if code, body := post(with(c.field, c.value)); code != http.StatusBadRequest || !strings.Contains(body, c.says) {
+			t.Errorf("%s=%v: %d %s", c.field, c.value, code, body)
+		}
+	}
+	code, body := post(with("name", "tone"))
+	var v TrainJobView
+	_ = json.Unmarshal([]byte(body), &v)
+	if code != http.StatusCreated || v.Pairs != 2 || !v.GGUF || v.State != "queued" || v.Params != `{"epochs":3}` {
+		t.Fatalf("%d %s", code, body)
+	}
+	rec := httptest.NewRecorder()
+	mux.ServeHTTP(rec, httptest.NewRequest("DELETE", "/api/v1/train/jobs/"+v.ID, nil))
+	if rec.Code != http.StatusConflict {
+		t.Errorf("deleting a queued job: %d", rec.Code)
+	}
+	rec = httptest.NewRecorder()
+	mux.ServeHTTP(rec, httptest.NewRequest("GET", "/api/v1/train/jobs/"+v.ID+"/artifacts/peft", nil))
+	if rec.Code != http.StatusNotFound {
+		t.Errorf("an artifact not made: %d", rec.Code)
+	}
+}
+
+func TestAnAdapterTrainedOnAKnowledgeBaseBelongsToWhoMayReadIt(t *testing.T) {
+	dsn := os.Getenv("LLMGW_TEST_POSTGRES")
+	if dsn == "" {
+		t.Skip("set LLMGW_TEST_POSTGRES to a pgvector database to run this")
+	}
+	ctx := context.Background()
+	s, err := store.OpenPostgres(ctx, dsn)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer s.Close()
+	if s.KnowledgeAvailable() != nil {
+		t.Skip("no pgvector")
+	}
+	models := append(testModels(), catalog.Model{Name: "small", Source: "/s",
+		Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp"}}})
+	a, _ := setup(t, models, gpuNode("cuda01", 1))
+	a.SetQuotas(s)
+	a.SetKB(&KBAdmin{Store: s})
+	a.SetTrain(&TrainAdmin{Store: s, Service: train.New(s, noAgents{}, nil)})
+	sfx := strings.ToLower(store.NewID(""))
+	org, _ := s.CreateOrg(ctx, "Tr"+sfx)
+	hr, _ := s.CreateTeam(ctx, org, "hr")
+	_, _ = s.CreateProject(ctx, hr, "docs")
+	sales, _ := s.CreateTeam(ctx, org, "sales")
+	_, _ = s.CreateProject(ctx, sales, "crm")
+	c, err := s.CreateCollection(ctx, store.KBCollection{OwnerScope: store.ScopeTeam, OwnerID: hr, Name: "p", EmbedModel: "bge-m3", Dims: 3})
+	if err != nil {
+		t.Fatal(err)
+	}
+	mux := http.NewServeMux()
+	a.Register(mux)
+	post := func(extra string) (int, TrainJobView, string) {
+		body := `{"name":"tone-` + store.NewID("") + `","model":"small","base_repo":"Qwen/Q","node":"cuda01","gpu_memory_mib":2000,` +
+			`"data":"{\"q\":\"a\",\"a\":\"b\",\"collection\":\"` + c.ID + `\"}\n"` + extra + `}`
+		rec := httptest.NewRecorder()
+		mux.ServeHTTP(rec, httptest.NewRequest("POST", "/api/v1/train/jobs", strings.NewReader(body)))
+		var v TrainJobView
+		_ = json.Unmarshal(rec.Body.Bytes(), &v)
+		return rec.Code, v, rec.Body.String()
+	}
+	if code, v, body := post(""); code != http.StatusCreated || v.Owner != "team:"+hr {
+		t.Errorf("by default: %d %s", code, body)
+	}
+	if code, v, body := post(`,"owner":"project:Tr` + sfx + `/hr/docs"`); code != http.StatusCreated || !strings.HasPrefix(v.Owner, "project:") {
+		t.Errorf("a project of the team: %d %s", code, body)
+	}
+	if code, _, body := post(`,"owner":"project:Tr` + sfx + `/sales/crm"`); code != http.StatusBadRequest || !strings.Contains(body, "may not read") {
+		t.Errorf("a project that cannot read it: %d %s", code, body)
+	}
+	if code, _, body := post(`,"public":true`); code != http.StatusBadRequest || !strings.Contains(body, "not to everyone") {
+		t.Errorf("public: %d %s", code, body)
+	}
+}
@@ -82,6 +82,8 @@ type Service struct {
 	// agentSettings is what each node's agent is told to run on; see
 	// SetAgentSettings.
 	agentSettings func(node string) string
+	// train is told how training jobs go; see SetTrain.
+	train TrainHandler

 	// logWaits are the log requests awaiting a reply, by request id.
 	logMu    sync.Mutex
@@ -368,6 +370,9 @@ func (s *Service) readLoop(ctx context.Context,
 		msg, err := stream.Receive()
 		if err != nil {
 			c.close()
+			if h := s.trainHandler(); h != nil {
+				h.OnNodeGone(nodeID)
+			}
 			if ctx.Err() != nil || errors.Is(err, io.EOF) {
 				s.log.Info("agent disconnected", "node", nodeID)
 				return
@@ -505,6 +510,16 @@ func (s *Service) handle(nodeID string, msg *llmgwv1.AgentMessage) {
 			ch <- m.Upgrade
 		}

+	case *llmgwv1.AgentMessage_TrainStatus:
+		if h := s.trainHandler(); h != nil {
+			h.OnTrainStatus(nodeID, m.TrainStatus)
+		}
+
+	case *llmgwv1.AgentMessage_TrainArtifact:
+		if h := s.trainHandler(); h != nil {
+			h.OnTrainArtifact(nodeID, m.TrainArtifact)
+		}
+
 	case *llmgwv1.AgentMessage_Logs:
 		s.logMu.Lock()
 		ch, ok := s.logWaits[m.Logs.GetRequestId()]
@@ -0,0 +1,35 @@
+package agentsvc
+
+import (
+	"slices"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+)
+
+// TrainHandler is told how training jobs on the nodes go (BACKLOG F4).
+type TrainHandler interface {
+	OnTrainStatus(node string, st *llmgwv1.TrainStatus)
+	OnTrainArtifact(node string, a *llmgwv1.TrainArtifact)
+	// OnNodeGone is a node's stream ending: a job running there ended with
+	// it.
+	OnNodeGone(node string)
+}
+
+// SetTrain gives the service what training jobs' messages go to.
+func (s *Service) SetTrain(h TrainHandler) {
+	s.mu.Lock()
+	s.train = h
+	s.mu.Unlock()
+}
+
+func (s *Service) trainHandler() TrainHandler {
+	s.mu.RLock()
+	defer s.mu.RUnlock()
+	return s.train
+}
+
+// Send gives a connected node's agent a command.
+func (s *Service) Send(node string, msg *llmgwv1.ControlMessage) error { return s.send(node, msg) }
+
+// Connected is whether a node's agent holds a stream to this control plane.
+func (s *Service) Connected(node string) bool { return slices.Contains(s.ConnectedNodes(), node) }
@@ -1127,6 +1127,27 @@ type Adapter struct {
 	Source string `json:"source" koanf:"source"`
 	// Scale is how strongly it is applied; zero takes 1.
 	Scale float64 `json:"scale,omitempty" koanf:"scale"`
+	// Owner is the part of the tenancy tree the adapter belongs to —
+	// org:ID, team:ID, project:ID, user:ID or key:ID — and only callers in
+	// it see it: to anyone else it is a model that does not exist. An
+	// adapter trained on an organisation's documents carries some of what
+	// they say in its weights. Empty is every caller the model allows.
+	Owner string `json:"owner,omitempty" koanf:"owner"`
+}
+
+// ValidAdapterOwner checks an adapter's owner is scope:ID.
+func ValidAdapterOwner(owner string) error {
+	if owner == "" {
+		return nil
+	}
+	scope, id, ok := strings.Cut(owner, ":")
+	switch {
+	case !ok || id == "":
+		return fmt.Errorf("an adapter's owner is scope:ID, %q is not", owner)
+	case scope != "org" && scope != "team" && scope != "project" && scope != "user" && scope != "key":
+		return fmt.Errorf("an adapter's owner is an org, team, project, user or key, not %q", scope)
+	}
+	return nil
 }

 // EffectiveScale is the scale an adapter is applied at.
@@ -1175,6 +1196,9 @@ func validateAdapters(m Model) error {
 	}
 	seen := map[string]bool{}
 	for _, a := range m.Adapters {
+		if err := ValidAdapterOwner(a.Owner); err != nil {
+			return fmt.Errorf("model %q adapter %q: %w", m.Name, a.Name, err)
+		}
 		switch {
 		case a.Name == "" || a.Source == "":
 			return fmt.Errorf("model %q: an adapter needs a name and a source", m.Name)
@@ -0,0 +1,196 @@
+// Package train runs LoRA training jobs from the control plane (BACKLOG F4):
+// queued in the store, sent to the node each names when its agent is there
+// and has nothing else training, followed as the agent reports, and what it
+// makes kept in the store.
+package train
+
+import (
+	"context"
+	"log/slog"
+	"sync"
+	"time"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// Store is where jobs are kept.
+type Store interface {
+	QueuedTrainJobs(ctx context.Context) ([]store.TrainJob, error)
+	ActiveTrainJobs(ctx context.Context) ([]store.TrainJob, error)
+	TrainJob(ctx context.Context, id string) (store.TrainJob, error)
+	UpdateTrainJob(ctx context.Context, j store.TrainJob) error
+	PutTrainArtifact(ctx context.Context, job, kind string, data []byte) error
+}
+
+// Agents is how jobs reach the nodes.
+type Agents interface {
+	Send(node string, msg *llmgwv1.ControlMessage) error
+	Connected(node string) bool
+}
+
+// Service sends queued jobs and records how they go.
+type Service struct {
+	store  Store
+	agents Agents
+	log    *slog.Logger
+
+	mu sync.Mutex
+	// pieces are artifacts arriving, by job and kind.
+	pieces map[[2]string][]byte
+}
+
+// New is a service.
+func New(s Store, a Agents, log *slog.Logger) *Service {
+	if log == nil {
+		log = slog.New(slog.DiscardHandler)
+	}
+	return &Service{store: s, agents: a, log: log, pieces: map[[2]string][]byte{}}
+}
+
+// Every is how often queued jobs are looked at.
+var Every = 5 * time.Second
+
+// Run sends queued jobs until ctx ends.
+func (s *Service) Run(ctx context.Context) {
+	t := time.NewTicker(Every)
+	defer t.Stop()
+	for {
+		s.Dispatch(ctx)
+		select {
+		case <-ctx.Done():
+			return
+		case <-t.C:
+		}
+	}
+}
+
+// Dispatch sends each queued job whose node is here and not already
+// training, one per node.
+func (s *Service) Dispatch(ctx context.Context) {
+	queued, err := s.store.QueuedTrainJobs(ctx)
+	if err != nil || len(queued) == 0 {
+		return
+	}
+	active, err := s.store.ActiveTrainJobs(ctx)
+	if err != nil {
+		return
+	}
+	busy := map[string]bool{}
+	for _, j := range active {
+		busy[j.Node] = true
+	}
+	for _, q := range queued {
+		if busy[q.Node] || !s.agents.Connected(q.Node) {
+			continue
+		}
+		j, err := s.store.TrainJob(ctx, q.ID)
+		if err != nil {
+			continue
+		}
+		err = s.agents.Send(j.Node, &llmgwv1.ControlMessage{Msg: &llmgwv1.ControlMessage_TrainStart{TrainStart: &llmgwv1.TrainStart{
+			JobId: j.ID, BaseRepo: j.BaseRepo, BaseRevision: j.BaseRevision, Data: j.Data, Params: j.Params,
+			GpuMemoryMib: int32(j.GPUMemoryMiB), Gguf: j.GGUF}}}) //nolint:gosec // MiB
+		if err != nil {
+			s.log.Warn("could not send a training job to its node", "job", j.ID, "node", j.Node, "err", err)
+			continue
+		}
+		j.State, j.StartedAt, j.Message = store.TrainStarting, time.Now(), "sent to "+j.Node
+		_ = s.store.UpdateTrainJob(ctx, j)
+		busy[j.Node] = true
+		s.log.Info("training job sent", "job", j.ID, "node", j.Node, "adapter", j.Name)
+	}
+}
+
+// OnTrainStatus records how a job goes, as its node says.
+func (s *Service) OnTrainStatus(node string, st *llmgwv1.TrainStatus) {
+	ctx := context.Background()
+	j, err := s.store.TrainJob(ctx, st.GetJobId())
+	if err != nil || j.Node != node || store.TrainOver(j.State) {
+		return
+	}
+	j.State = st.GetState()
+	if m := st.GetMessage(); m != "" || j.State == store.TrainDone {
+		j.Message = m
+	}
+	if st.GetSteps() > 0 {
+		j.Step, j.Steps = int(st.GetStep()), int(st.GetSteps())
+	}
+	if st.GetLoss() > 0 {
+		j.Loss = st.GetLoss()
+	}
+	if p := int(st.GetPeakMib()); p > j.PeakMiB {
+		j.PeakMiB = p
+	}
+	if store.TrainOver(j.State) {
+		j.FinishedAt = time.Now()
+		s.mu.Lock()
+		for k := range s.pieces {
+			if k[0] == j.ID {
+				delete(s.pieces, k)
+			}
+		}
+		s.mu.Unlock()
+		s.log.Info("training job ended", "job", j.ID, "state", j.State, "message", j.Message)
+	}
+	if err := s.store.UpdateTrainJob(ctx, j); err != nil {
+		s.log.Warn("could not record a training job's progress", "job", j.ID, "err", err)
+	}
+}
+
+// OnTrainArtifact gathers what a job made, and keeps it when it is whole.
+func (s *Service) OnTrainArtifact(node string, a *llmgwv1.TrainArtifact) {
+	key := [2]string{a.GetJobId(), a.GetKind()}
+	s.mu.Lock()
+	if a.GetSeq() == 0 {
+		s.pieces[key] = nil
+	}
+	s.pieces[key] = append(s.pieces[key], a.GetData()...)
+	whole := s.pieces[key]
+	if a.GetLast() {
+		delete(s.pieces, key)
+	}
+	s.mu.Unlock()
+	if !a.GetLast() {
+		return
+	}
+	if err := s.store.PutTrainArtifact(context.Background(), key[0], key[1], whole); err != nil {
+		s.log.Warn("could not keep what a training job made", "job", key[0], "kind", key[1], "err", err)
+	}
+}
+
+// OnNodeGone fails the jobs a node was running: its agent's stream ended,
+// and the job with it.
+func (s *Service) OnNodeGone(node string) {
+	ctx := context.Background()
+	active, err := s.store.ActiveTrainJobs(ctx)
+	if err != nil {
+		return
+	}
+	for _, j := range active {
+		if j.Node != node {
+			continue
+		}
+		j.State, j.FinishedAt = store.TrainFailed, time.Now()
+		j.Message = "the node's agent went away while it trained; start the job again"
+		_ = s.store.UpdateTrainJob(ctx, j)
+	}
+}
+
+// Cancel stops a job: one still queued is not sent, one running is stopped
+// on its node.
+func (s *Service) Cancel(ctx context.Context, id string) error {
+	j, err := s.store.TrainJob(ctx, id)
+	if err != nil {
+		return err
+	}
+	if store.TrainOver(j.State) {
+		return nil
+	}
+	if j.State != store.TrainQueued && s.agents.Connected(j.Node) {
+		_ = s.agents.Send(j.Node, &llmgwv1.ControlMessage{Msg: &llmgwv1.ControlMessage_TrainCancel{
+			TrainCancel: &llmgwv1.TrainCancel{JobId: id}}})
+	}
+	j.State, j.FinishedAt, j.Message = store.TrainCancelled, time.Now(), "cancelled"
+	return s.store.UpdateTrainJob(ctx, j)
+}
@@ -0,0 +1,124 @@
+package train
+
+import (
+	"context"
+	"path/filepath"
+	"sync"
+	"testing"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+type fakeAgents struct {
+	mu        sync.Mutex
+	connected map[string]bool
+	sent      []*llmgwv1.ControlMessage
+}
+
+func (f *fakeAgents) Send(_ string, m *llmgwv1.ControlMessage) error {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	f.sent = append(f.sent, m)
+	return nil
+}
+func (f *fakeAgents) Connected(n string) bool {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	return f.connected[n]
+}
+
+func setup(t *testing.T) (*Service, *store.DB, *fakeAgents) {
+	t.Helper()
+	s, err := store.OpenSQLite(context.Background(), filepath.Join(t.TempDir(), "s.db"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	t.Cleanup(func() { _ = s.Close() })
+	a := &fakeAgents{connected: map[string]bool{}}
+	return New(s, a, nil), s, a
+}
+
+func job(t *testing.T, s *store.DB, id, node string) {
+	t.Helper()
+	if err := s.CreateTrainJob(context.Background(), store.TrainJob{ID: id, Name: "a-" + id, Model: "m", BaseRepo: "o/n",
+		Node: node, GPUMemoryMiB: 2000, Data: []byte(`{"q":"x","a":"y"}`), Pairs: 1, GGUF: true}); err != nil {
+		t.Fatal(err)
+	}
+}
+
+func TestAJobGoesToItsNodeWhenItIsHereAndFree(t *testing.T) {
+	ctx := context.Background()
+	svc, s, a := setup(t)
+	job(t, s, "j1", "cuda01")
+	job(t, s, "j2", "cuda01")
+	svc.Dispatch(ctx)
+	if len(a.sent) != 0 {
+		t.Fatal("sent to a node that is not here")
+	}
+	a.connected["cuda01"] = true
+	svc.Dispatch(ctx)
+	svc.Dispatch(ctx)
+	if len(a.sent) != 1 {
+		t.Fatalf("sent %d jobs to one node", len(a.sent))
+	}
+	start := a.sent[0].GetTrainStart()
+	if start.GetJobId() != "j1" || start.GetGpuMemoryMib() != 2000 || !start.GetGguf() || string(start.GetData()) == "" {
+		t.Errorf("sent %+v", start)
+	}
+	if j, _ := s.TrainJob(ctx, "j1"); j.State != store.TrainStarting || j.StartedAt.IsZero() {
+		t.Errorf("j1 %+v", j)
+	}
+
+	// Progress, then what it made in pieces, then done: the next is sent.
+	svc.OnTrainStatus("cuda01", &llmgwv1.TrainStatus{JobId: "j1", State: "running", Step: 5, Steps: 10, Loss: 0.8, PeakMib: 1900})
+	svc.OnTrainArtifact("cuda01", &llmgwv1.TrainArtifact{JobId: "j1", Kind: "gguf", Seq: 0, Data: []byte("GG")})
+	svc.OnTrainArtifact("cuda01", &llmgwv1.TrainArtifact{JobId: "j1", Kind: "gguf", Seq: 1, Data: []byte("UF"), Last: true})
+	svc.OnTrainStatus("cuda01", &llmgwv1.TrainStatus{JobId: "j1", State: "done"})
+	j, _ := s.TrainJob(ctx, "j1")
+	if j.State != store.TrainDone || j.Step != 5 || j.PeakMiB != 1900 || j.Loss != 0.8 || j.FinishedAt.IsZero() {
+		t.Errorf("j1 after %+v", j)
+	}
+	if b, _ := s.TrainArtifact(ctx, "j1", "gguf"); string(b) != "GGUF" {
+		t.Errorf("artifact %q", b)
+	}
+	// A late message does not bring an ended job back.
+	svc.OnTrainStatus("cuda01", &llmgwv1.TrainStatus{JobId: "j1", State: "running"})
+	if j, _ := s.TrainJob(ctx, "j1"); j.State != store.TrainDone {
+		t.Errorf("revived: %s", j.State)
+	}
+	// Nor does another node speak for it.
+	svc.Dispatch(ctx)
+	if len(a.sent) != 2 || a.sent[1].GetTrainStart().GetJobId() != "j2" {
+		t.Fatalf("j2 not sent: %d", len(a.sent))
+	}
+	svc.OnTrainStatus("node01", &llmgwv1.TrainStatus{JobId: "j2", State: "done"})
+	if j, _ := s.TrainJob(ctx, "j2"); j.State == store.TrainDone {
+		t.Error("another node ended the job")
+	}
+	// The node goes: its job fails, saying why.
+	svc.OnNodeGone("cuda01")
+	if j, _ := s.TrainJob(ctx, "j2"); j.State != store.TrainFailed || j.Message == "" {
+		t.Errorf("after the node went: %+v", j)
+	}
+}
+
+func TestCancellingAJob(t *testing.T) {
+	ctx := context.Background()
+	svc, s, a := setup(t)
+	job(t, s, "j1", "cuda01")
+	if err := svc.Cancel(ctx, "j1"); err != nil {
+		t.Fatal(err)
+	}
+	a.connected["cuda01"] = true
+	svc.Dispatch(ctx)
+	if len(a.sent) != 0 {
+		t.Error("a cancelled job was sent")
+	}
+	job(t, s, "j2", "cuda01")
+	svc.Dispatch(ctx)
+	_ = svc.Cancel(ctx, "j2")
+	if len(a.sent) != 2 || a.sent[1].GetTrainCancel().GetJobId() != "j2" {
+		t.Errorf("the node was not told: %+v", a.sent)
+	}
+}
@@ -8,6 +8,7 @@ var Reference = []Block{
 		{Key: "name", Type: "string", Doc: "Name is what clients ask for, like a model name.", Nested: ""},
 		{Key: "source", Type: "string", Doc: "Source is the adapter: a GGUF file for llama.cpp; for vLLM a PEFT\nadapter directory (a mirror version ending in \"/\", a path on the node,\nor a Hugging Face repository id). Fetched by each node as the model's\nweights are.", Nested: ""},
 		{Key: "scale", Type: "float64", Doc: "Scale is how strongly it is applied; zero takes 1.", Nested: ""},
+		{Key: "owner", Type: "string", Doc: "Owner is the part of the tenancy tree the adapter belongs to —\norg:ID, team:ID, project:ID, user:ID or key:ID — and only callers in\nit see it: to anyone else it is a model that does not exist. An\nadapter trained on an organisation's documents carries some of what\nthey say in its weights. Empty is every caller the model allows.", Nested: ""},
 	}},
 	{Package: "config", Name: "Admission", Title: "Admission", Doc: "", Fields: []Field{
 		{Key: "enabled", Type: "bool", Doc: "Enabled turns the layer on. Off, every request goes straight through,\nwhich is the v0.2.0 behaviour.", Nested: ""},
@@ -0,0 +1,154 @@
+package firewall
+
+import (
+	"context"
+	"fmt"
+	"strings"
+)
+
+// A store member that is nothing else runs no llmgw process: Patroni, etcd
+// and PostgreSQL only. What the cluster's settings say of its firewall it
+// reads from its own copy of the store, where the leading control plane
+// keeps it (the cluster_state table, key firewall), once a minute, by a
+// root timer running FollowScript. It turns llmgw's table off and on, and
+// lets in the networks the settings add; what `server init` let in stays.
+
+// FollowScript is the script, as python3 runs it. LLMGW_FW_STATE and
+// LLMGW_FW_RULES replace the store and the rules file, for its tests.
+const FollowScript = `#!/usr/bin/python3
+# Written by llmgw: llmgw's firewall on this store member, as the cluster's
+# settings say. Run by llmgw-firewall-follow.timer.
+import ipaddress, json, os, subprocess, sys, tempfile
+
+RULES = os.environ.get("LLMGW_FW_RULES", "` + RulesFile + `")
+
+def run(*a):
+    return subprocess.run(a, capture_output=True, text=True)
+
+raw = os.environ.get("LLMGW_FW_STATE")
+if raw is None:
+    r = run("runuser", "-u", "postgres", "--", "psql", "-d", "llmgw", "-tAqc",
+            "SELECT value FROM cluster_state WHERE key = 'firewall'")
+    # No table yet, or nothing told: a control plane older than this, or
+    # none leading. Left as it is.
+    if r.returncode != 0:
+        sys.exit(0)
+    raw = r.stdout
+if not raw.strip():
+    sys.exit(0)
+st = json.loads(raw)
+present = run("nft", "list", "table", "inet", "` + Table + `").returncode == 0
+
+if not st.get("enabled", True):
+    if present:
+        run("systemctl", "disable", "--now", "` + Unit + `")
+        run("nft", "delete", "table", "inet", "` + Table + `")
+        print("llmgw's firewall is off here")
+    sys.exit(0)
+
+if not os.path.exists(RULES):
+    sys.exit(0)
+lines = open(RULES).read().split("\n")
+want = []
+for n in st.get("networks") or []:
+    try:
+        want.append(ipaddress.ip_network(n, strict=False))
+    except ValueError:
+        pass
+
+def saddr(l):
+    s = l.strip()
+    return s.startswith("ip saddr {") or s.startswith("ip6 saddr {")
+
+changed = False
+for fam, kw in ((4, "ip saddr {"), (6, "ip6 saddr {")):
+    add = [str(n) for n in want if n.version == fam]
+    if not add:
+        continue
+    same = [i for i, l in enumerate(lines) if l.strip().startswith(kw)]
+    if same:
+        i = same[0]
+        l = lines[i]
+        a, b = l.index("{") + 1, l.index("}")
+        have = [x.strip() for x in l[a:b].split(",") if x.strip()]
+        new = [x for x in add if x not in have]
+        if new:
+            lines[i] = l[:a] + " " + ", ".join(have + new) + " " + l[b:]
+            changed = True
+        continue
+    other = [i for i, l in enumerate(lines) if saddr(l)]
+    if not other:
+        continue
+    l = lines[other[0]]
+    indent = l[:len(l) - len(l.lstrip())]
+    lines.insert(other[0] + 1, indent + kw + " " + ", ".join(add) + " }" + l[l.index("}") + 1:])
+    changed = True
+
+if changed:
+    fd, tmp = tempfile.mkstemp(dir=os.path.dirname(RULES))
+    with os.fdopen(fd, "w") as f:
+        f.write("\n".join(lines))
+    if run("nft", "-c", "-f", tmp).returncode != 0:
+        os.unlink(tmp)
+        print("the settings' networks make a ruleset nft refuses; left as it was", file=sys.stderr)
+        sys.exit(1)
+    os.chmod(tmp, 0o644)
+    os.replace(tmp, RULES)
+if changed or not present:
+    run("systemctl", "enable", "` + Unit + `")
+    if run("nft", "-f", RULES).returncode != 0:
+        sys.exit(1)
+    print("llmgw's firewall is on here")
+`
+
+// FollowerPath is where FollowScript is installed.
+const FollowerPath = "/usr/local/lib/llmgw/firewall-follow"
+
+// FollowerTimer is the timer that runs it.
+const FollowerTimer = "llmgw-firewall-follow.timer"
+
+const followService = `[Unit]
+Description=llmgw firewall: follow the cluster's settings on this store member
+After=postgresql.service patroni.service
+
+[Service]
+Type=oneshot
+ExecStart=/usr/bin/python3 ` + FollowerPath + `
+`
+
+const followTimer = `[Unit]
+Description=llmgw firewall: follow the cluster's settings on this store member
+
+[Timer]
+OnBootSec=1min
+OnUnitActiveSec=1min
+
+[Install]
+WantedBy=timers.target
+`
+
+// InstallFollower puts FollowScript and its timer on a store member.
+func InstallFollower(ctx context.Context, r Runner) error {
+	body := FollowScript + "\x00" + followService + "\x00" + followTimer
+	out, err := r.Run(ctx, `set -e
+t=$(mktemp -d); trap 'rm -rf "$t"' EXIT
+cat > "$t/all"
+python3 - "$t" <<'PY'
+import sys
+d = sys.argv[1]
+s, svc, tim = open(d + "/all", "rb").read().split(b"\x00")
+open(d + "/script", "wb").write(s)
+open(d + "/service", "wb").write(svc)
+open(d + "/timer", "wb").write(tim)
+PY
+sudo install -d -m 755 /usr/local/lib/llmgw
+sudo install -m 755 "$t/script" `+FollowerPath+`
+sudo install -m 644 "$t/service" /etc/systemd/system/llmgw-firewall-follow.service
+sudo install -m 644 "$t/timer" /etc/systemd/system/`+FollowerTimer+`
+sudo systemctl daemon-reload
+sudo systemctl enable --now `+FollowerTimer+` >/dev/null 2>&1`, []byte(body))
+	if err != nil {
+		return fmt.Errorf("installing %s: %v: %s", FollowerTimer, err, strings.TrimSpace(string(out)))
+	}
+	return nil
+}
@@ -0,0 +1,101 @@
+package firewall
+
+import (
+	"net/netip"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"strings"
+	"testing"
+)
+
+// follow runs FollowScript against a rules file and a state, with nft and
+// systemctl replaced by scripts that log what they were asked and answer as
+// a machine whose table is loaded or not.
+func follow(t *testing.T, rules, state string, loaded bool) (string, string, string) {
+	t.Helper()
+	if _, err := exec.LookPath("python3"); err != nil {
+		t.Skip("no python3")
+	}
+	dir := t.TempDir()
+	bin := filepath.Join(dir, "bin")
+	_ = os.Mkdir(bin, 0o755)
+	log := filepath.Join(dir, "log")
+	list := "exit 1"
+	if loaded {
+		list = "exit 0"
+	}
+	for name, body := range map[string]string{
+		"nft":       `echo "nft $*" >> ` + log + "\n" + `[ "$1" = list ] && ` + list + "\nexit 0\n",
+		"systemctl": `echo "systemctl $*" >> ` + log + "\nexit 0\n",
+	} {
+		if err := os.WriteFile(filepath.Join(bin, name), []byte("#!/bin/sh\n"+body), 0o755); err != nil {
+			t.Fatal(err)
+		}
+	}
+	rf := filepath.Join(dir, "firewall.nft")
+	if rules != "" {
+		_ = os.WriteFile(rf, []byte(rules), 0o644)
+	}
+	script := filepath.Join(dir, "follow")
+	_ = os.WriteFile(script, []byte(FollowScript), 0o755)
+	cmd := exec.Command("python3", script)
+	cmd.Env = append(os.Environ(), "PATH="+bin+":"+os.Getenv("PATH"), "LLMGW_FW_STATE="+state, "LLMGW_FW_RULES="+rf)
+	out, err := cmd.CombinedOutput()
+	if err != nil {
+		t.Fatalf("%v: %s", err, out)
+	}
+	got, _ := os.ReadFile(rf)
+	calls, _ := os.ReadFile(log)
+	return string(out), string(got), string(calls)
+}
+
+var memberRules = Script(Spec{Ports: []Port{Postgres, EtcdClient}, Allow: mustPrefixes("10.0.0.0/24")})
+
+func TestAStoreMemberFollowsTheSettings(t *testing.T) {
+	// Off: the table goes, the rules stay for later.
+	out, rules, calls := follow(t, memberRules, `{"enabled":false}`, true)
+	if !strings.Contains(out, "off") || !strings.Contains(calls, "nft delete table inet llmgw") ||
+		!strings.Contains(calls, "systemctl disable --now "+Unit) || rules != memberRules {
+		t.Errorf("off: %q %q", out, calls)
+	}
+	// On again: loaded from the rules kept.
+	out, _, calls = follow(t, memberRules, `{"enabled":true}`, false)
+	if !strings.Contains(out, "on") || !strings.Contains(calls, "nft -f") {
+		t.Errorf("on: %q %q", out, calls)
+	}
+	// On and loaded, nothing new: nothing done.
+	out, _, calls = follow(t, memberRules, `{"enabled":true,"networks":["10.0.0.0/24"]}`, true)
+	if out != "" || strings.Contains(calls, "nft -f") {
+		t.Errorf("unchanged: %q %q", out, calls)
+	}
+	// Networks the settings add, of both families, are let in beside what
+	// init let in; the result reads back as the same ports.
+	_, rules, calls = follow(t, memberRules, `{"enabled":true,"networks":["192.168.5.0/24","fd00::/64","nonsense"]}`, true)
+	if !strings.Contains(calls, "nft -c -f") || !strings.Contains(calls, "nft -f") {
+		t.Errorf("not checked and loaded: %q", calls)
+	}
+	back := parse(rules)
+	var nets []string
+	for _, p := range back.Allow {
+		nets = append(nets, p.String())
+	}
+	if strings.Join(nets, ",") != "10.0.0.0/24,192.168.5.0/24,fd00::/64" || len(back.Ports) != 2 {
+		t.Errorf("rules now:\n%s", rules)
+	}
+	// Nothing told yet, or no rules: left alone.
+	if out, _, calls := follow(t, memberRules, "", true); out != "" || calls != "" {
+		t.Errorf("nothing told: %q %q", out, calls)
+	}
+	if out, _, calls := follow(t, "", `{"enabled":true}`, false); out != "" || strings.Contains(calls, "nft -f") {
+		t.Errorf("no rules: %q %q", out, calls)
+	}
+}
+
+func mustPrefixes(ss ...string) []netip.Prefix {
+	var out []netip.Prefix
+	for _, s := range ss {
+		out = append(out, netip.MustParsePrefix(s))
+	}
+	return out
+}
@@ -4,6 +4,7 @@ import (
 	"encoding/json"
 	"errors"
 	"fmt"
+	"io"
 	"net/http"
 	"os"
 	"strings"
@@ -37,7 +38,9 @@ func (a *API) Handler() http.Handler {
 	mux.HandleFunc("POST /v1/models", a.fetch)
 	mux.HandleFunc("GET /v1/models/{model}/{version}", a.status)
 	mux.HandleFunc("DELETE /v1/models/{model}/{version}", a.remove)
+	mux.HandleFunc("PUT /v1/models/{model}/{version}/{file}", a.upload)
 	mux.HandleFunc("GET /v1/health", a.health)
+	mux.HandleFunc("PUT /v1/firewall", a.putFirewall)
 	return mux
 }

@@ -199,3 +202,59 @@ func LastPathElement(u string) string {
 	}
 	return u
 }
+
+// maxUpload bounds a file sent to the mirror rather than fetched by it.
+const maxUpload = 8 << 30
+
+// upload keeps a file the caller sends — an adapter a node trained, which
+// is on no registry to fetch from — with the same discipline as an import:
+// its checksum and source beside it, and an existing version never
+// replaced. X-Llmgw-Source says where it came from.
+func (a *API) upload(w http.ResponseWriter, r *http.Request) {
+	e := Entry{Model: r.PathValue("model"), Version: r.PathValue("version"), File: r.PathValue("file")}
+	for _, n := range []string{e.Model, e.Version, e.File} {
+		if err := ValidName(n); err != nil {
+			fail(w, http.StatusBadRequest, err)
+			return
+		}
+	}
+	if _, err := os.Stat(e.Path(a.m.root)); err == nil {
+		fail(w, http.StatusConflict, fmt.Errorf("%s/%s/%s: %w", e.Model, e.Version, e.File, ErrExists))
+		return
+	}
+	tmp, err := os.CreateTemp(a.m.root, ".upload-*")
+	if err != nil {
+		fail(w, http.StatusInternalServerError, err)
+		return
+	}
+	defer os.Remove(tmp.Name())
+	n, err := io.Copy(tmp, http.MaxBytesReader(w, r.Body, maxUpload))
+	if cerr := tmp.Close(); err == nil {
+		err = cerr
+	}
+	if err != nil {
+		fail(w, http.StatusBadRequest, fmt.Errorf("receiving the file: %w", err))
+		return
+	}
+	if n == 0 {
+		fail(w, http.StatusBadRequest, errors.New("the file is empty"))
+		return
+	}
+	source := strings.TrimSpace(r.Header.Get("X-Llmgw-Source"))
+	if source == "" {
+		source = "uploaded"
+	}
+	added, err := Import(r.Context(), a.m.root, tmp.Name(), e, "", source, nil)
+	switch {
+	case errors.Is(err, ErrExists):
+		fail(w, http.StatusConflict, err)
+		return
+	case err != nil:
+		fail(w, http.StatusBadRequest, err)
+		return
+	}
+	writeJSON(w, http.StatusCreated, FetchResponse{
+		Status: Status{Model: e.Model, Version: e.Version, File: e.File, State: Present, BytesDone: added.Entry.Bytes, BytesTotal: added.Entry.Bytes},
+		URL:    e.URL(a.base),
+	})
+}
@@ -100,6 +100,36 @@ func (c *Client) Fetch(ctx context.Context, req FetchRequest) (FetchResponse, er
 	return out, err
 }

+// Upload sends a file for the mirror to keep at model/version/file, with
+// where it came from; it answers the URL nodes fetch it from.
+func (c *Client) Upload(ctx context.Context, model, version, file string, body []byte, source string) (FetchResponse, error) {
+	var out FetchResponse
+	req, err := http.NewRequestWithContext(ctx, http.MethodPut,
+		c.base+"/v1/models/"+url.PathEscape(model)+"/"+url.PathEscape(version)+"/"+url.PathEscape(file), bytes.NewReader(body))
+	if err != nil {
+		return out, err
+	}
+	req.Header.Set("Content-Type", "application/octet-stream")
+	req.Header.Set("X-Llmgw-Source", source)
+	resp, err := c.hc.Do(req)
+	if err != nil {
+		return out, fmt.Errorf("reaching the mirror at %s: %w", c.base, err)
+	}
+	defer resp.Body.Close()
+	raw, err := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
+	if err != nil {
+		return out, err
+	}
+	if resp.StatusCode >= 300 {
+		var e struct {
+			Error string `json:"error"`
+		}
+		_ = json.Unmarshal(raw, &e)
+		return out, &APIError{Status: resp.StatusCode, Msg: e.Error}
+	}
+	return out, json.Unmarshal(raw, &out)
+}
+
 // Status is where one model version stands.
 func (c *Client) Status(ctx context.Context, model, version string) (FetchResponse, error) {
 	var out FetchResponse
@@ -0,0 +1,65 @@
+package mirror
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"net/http"
+	"os"
+	"path/filepath"
+)
+
+// The mirror's machine keeps llmgw's firewall as the cluster's settings
+// say, as every other machine does. The mirror runs unprivileged and cannot
+// change the ruleset: the control plane tells it what the settings say, it
+// writes that beside its models, and a root path unit on the machine
+// applies it with `llmgw firewall sync`.
+
+// FirewallState is what the cluster's settings say of a machine's firewall.
+type FirewallState struct {
+	Enabled bool `json:"enabled"`
+	// Networks are let in besides those the machine reaches others on.
+	Networks []string `json:"networks,omitempty"`
+	// Toward are the cluster's machines, by address: each one's network is
+	// let in, as the machine's route to it says.
+	Toward []string `json:"toward,omitempty"`
+}
+
+// FirewallFile is where, in the mirror's directory, the state is written.
+// Hidden, so it is never taken for a model.
+const FirewallFile = ".firewall.json"
+
+func (a *API) putFirewall(w http.ResponseWriter, r *http.Request) {
+	var st FirewallState
+	if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 1<<20)).Decode(&st); err != nil {
+		fail(w, http.StatusBadRequest, err)
+		return
+	}
+	b, _ := json.Marshal(st)
+	p := filepath.Join(a.m.root, FirewallFile)
+	changed := false
+	if old, err := os.ReadFile(p); err != nil || !bytes.Equal(old, b) {
+		// Renamed into place: the path unit sees one change, of a whole file.
+		tmp := p + ".new"
+		if err := os.WriteFile(tmp, b, 0o644); err != nil {
+			fail(w, http.StatusInternalServerError, err)
+			return
+		}
+		if err := os.Rename(tmp, p); err != nil {
+			fail(w, http.StatusInternalServerError, err)
+			return
+		}
+		changed = true
+	}
+	writeJSON(w, http.StatusOK, map[string]bool{"changed": changed})
+}
+
+// SetFirewall tells the mirror what the cluster's settings say of its
+// machine's firewall; it answers whether that changed anything.
+func (c *Client) SetFirewall(ctx context.Context, st FirewallState) (bool, error) {
+	var out struct {
+		Changed bool `json:"changed"`
+	}
+	err := c.do(ctx, http.MethodPut, "/v1/firewall", st, &out)
+	return out.Changed, err
+}
@@ -0,0 +1,38 @@
+package mirror
+
+import (
+	"context"
+	"encoding/json"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
+)
+
+func TestTheMirrorKeepsWhatTheSettingsSayOfItsFirewall(t *testing.T) {
+	root := t.TempDir()
+	srv := httptest.NewServer(NewAPI(NewManager(root, discardLog()), "http://m:8081").Handler())
+	defer srv.Close()
+	c, err := NewClient(srv.URL, tlsconf.Config{}, 0)
+	if err != nil {
+		t.Fatal(err)
+	}
+	ctx := context.Background()
+	st := FirewallState{Enabled: true, Networks: []string{"192.168.5.0/24"}, Toward: []string{"10.0.0.147"}}
+	if changed, err := c.SetFirewall(ctx, st); err != nil || !changed {
+		t.Fatalf("first: %v %v", changed, err)
+	}
+	if changed, err := c.SetFirewall(ctx, st); err != nil || changed {
+		t.Errorf("the same again: %v %v", changed, err)
+	}
+	var back FirewallState
+	b, _ := os.ReadFile(filepath.Join(root, FirewallFile))
+	if err := json.Unmarshal(b, &back); err != nil || !back.Enabled || back.Toward[0] != "10.0.0.147" {
+		t.Errorf("kept %s", b)
+	}
+	if all, err := c.List(ctx); err != nil || len(all.Models) != 0 {
+		t.Errorf("the file is taken for a model: %+v %v", all, err)
+	}
+}
@@ -0,0 +1,45 @@
+package mirror
+
+import (
+	"context"
+	"net/http/httptest"
+	"os"
+	"strings"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
+)
+
+func TestAFileSentToTheMirrorIsKeptWithItsChecksumAndNeverReplaced(t *testing.T) {
+	root := t.TempDir()
+	srv := httptest.NewServer(NewAPI(NewManager(root, nil), "http://mirror:8081").Handler())
+	defer srv.Close()
+	c, err := NewClient(srv.URL, tlsconf.Config{}, 0)
+	if err != nil {
+		t.Fatal(err)
+	}
+	ctx := context.Background()
+	got, err := c.Upload(ctx, "politika-tonu", "train-1", "politika-tonu.gguf", []byte("GGUF adapter"), "trained by job train-1")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if got.URL != "http://mirror:8081/politika-tonu/train-1/politika-tonu.gguf" || got.Status.BytesTotal != 12 {
+		t.Errorf("answered %+v", got)
+	}
+	e := Entry{Model: "politika-tonu", Version: "train-1", File: "politika-tonu.gguf"}
+	if b, _ := os.ReadFile(e.Path(root)); string(b) != "GGUF adapter" {
+		t.Errorf("kept %q", b)
+	}
+	if b, _ := os.ReadFile(sourcePath(e.Path(root))); !strings.Contains(string(b), "train-1") {
+		t.Errorf("source %q", b)
+	}
+	if _, err := os.Stat(checksumPath(e.Path(root))); err != nil {
+		t.Errorf("no checksum: %v", err)
+	}
+	if _, err := c.Upload(ctx, "politika-tonu", "train-1", "politika-tonu.gguf", []byte("other"), "x"); err == nil {
+		t.Error("an existing version was replaced")
+	}
+	if _, err := c.Upload(ctx, "../etc", "v", "f", []byte("x"), "x"); err == nil {
+		t.Error("a path outside the mirror was taken")
+	}
+}
@@ -0,0 +1,171 @@
+package pgha
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"slices"
+	"strconv"
+	"strings"
+	"time"
+)
+
+// PostgreSQL's own defaults — 128MB of cache, 4MB to sort in — are for a
+// machine it barely uses. A knowledge base's vector index is searched well
+// only from memory: one that does not fit is read from disk on every search.
+// So each member is sized to the memory it has, read when setup runs, and
+// again on every later run, so a machine given more memory gets the use of it.
+
+// MemorySettings are PostgreSQL's memory parameters for a machine with total
+// bytes of memory. shared is a machine that also runs a control plane or an
+// agent: the database is given less of it, since a gateway swapped out to
+// give the database cache is worse than a smaller cache.
+func MemorySettings(total int64, shared bool) map[string]string {
+	const mb = 1 << 20
+	ram := total / mb
+	// Shares of memory, in MB: the cache PostgreSQL keeps itself, what the
+	// planner may count on the kernel caching besides, what building or
+	// vacuuming an index may take, and what one sort may.
+	buffers, cache, maint, sort := ram/4, ram*3/4, ram/16, ram/256
+	if shared {
+		buffers, cache, maint, sort = ram*3/20, ram/2, ram/32, ram/512
+	}
+	clamp := func(v, lo, hi int64) int64 { return min(max(v, lo), hi) }
+	return map[string]string{
+		"shared_buffers":       strconv.FormatInt(clamp(buffers, 96, 32<<10), 10) + "MB",
+		"effective_cache_size": strconv.FormatInt(clamp(cache, 256, 1<<20), 10) + "MB",
+		"maintenance_work_mem": strconv.FormatInt(clamp(maint, 64, 2<<10), 10) + "MB",
+		"work_mem":             strconv.FormatInt(clamp(sort, 4, 64), 10) + "MB",
+	}
+}
+
+// MemTotal reads the machine's memory, in bytes, from /proc/meminfo.
+func MemTotal(meminfo string) (int64, bool) {
+	for _, line := range strings.Split(meminfo, "\n") {
+		f := strings.Fields(line)
+		if len(f) >= 2 && f[0] == "MemTotal:" {
+			kb, err := strconv.ParseInt(f[1], 10, 64)
+			return kb << 10, err == nil && kb > 0
+		}
+	}
+	return 0, false
+}
+
+// memory sizes each member to its machine, in its own configuration — the
+// machines need not be alike. What takes effect on reload is applied so;
+// shared_buffers needs a restart, given one member at a time, the leader
+// last, each waited on before the next.
+func (s *Setup) memory(ctx context.Context, sec Secrets) error {
+	var restart []Node
+	for _, d := range s.DB {
+		out, err := d.Remote.Run(ctx, `cat /proc/meminfo
+if [ -f /etc/llmgw/server.yaml ] || [ -f /etc/llmgw/agent.yaml ]; then echo "llmgw-shared"; fi`, nil)
+		total, ok := MemTotal(string(out))
+		if err != nil || !ok {
+			s.emit("memory", false, "%s: could not read its memory; PostgreSQL left as it is", d.Name)
+			continue
+		}
+		shared := strings.Contains(string(out), "llmgw-shared")
+		params := MemorySettings(total, shared)
+		body, _ := json.Marshal(params)
+		out, err = d.Remote.Run(ctx, `set -e
+test -f /etc/patroni/config.yml || { echo missing; exit 0; }
+want=$(cat)
+r=$(sudo env WANT="$want" python3 - <<'PY'
+import json, os, yaml
+p = "/etc/patroni/config.yml"
+want = json.loads(os.environ["WANT"])
+c = yaml.safe_load(open(p))
+pg = c.setdefault("postgresql", {})
+params = pg.get("parameters") or {}
+pg["parameters"] = params
+if all(str(params.get(k)) == v for k, v in want.items()):
+    print("same")
+else:
+    restart = str(params.get("shared_buffers")) != want["shared_buffers"]
+    params.update(want)
+    open(p + ".new", "w").write("# llmgw: the control planes' store, kept by Patroni, its leader lock in etcd.\n" +
+        yaml.safe_dump(c, default_flow_style=False, sort_keys=False))
+    print("restart" if restart else "changed")
+PY
+)
+if [ "$r" != same ]; then
+  sudo chown root:postgres /etc/patroni/config.yml.new
+  sudo chmod 640 /etc/patroni/config.yml.new
+  sudo mv -f /etc/patroni/config.yml.new /etc/patroni/config.yml
+  sudo systemctl reload patroni
+fi
+echo "$r"`, body)
+		if err != nil {
+			return fmt.Errorf("on %s: %v: %s", d.Name, err, strings.TrimSpace(string(out)))
+		}
+		what := fmt.Sprintf("%s: %d MB of memory%s; shared_buffers %s, effective_cache_size %s, maintenance_work_mem %s, work_mem %s",
+			d.Name, total>>20, map[bool]string{true: ", shared with llmgw"}[shared],
+			params["shared_buffers"], params["effective_cache_size"], params["maintenance_work_mem"], params["work_mem"])
+		switch r := strings.TrimSpace(lastLine(out)); r {
+		case "same", "missing":
+			s.emit("memory", false, "%s", what)
+		case "restart":
+			s.emit("memory", true, "%s", what)
+			restart = append(restart, d)
+		default:
+			s.emit("memory", true, "%s", what)
+		}
+	}
+	// Patroni takes a reload in its own time: restarted before it has written
+	// the new configuration, PostgreSQL comes back on the old one. So each is
+	// waited on until Patroni says a restart is pending — and a member found
+	// pending, by a run that stopped before restarting it, is restarted too.
+	pending := func(c Cluster, name string) bool {
+		for _, m := range c.Members {
+			if m.Name == name {
+				return m.PendingRestart
+			}
+		}
+		return false
+	}
+	var c Cluster
+	err := s.waitFor(ctx, s.dbHosts(), func(got Cluster) bool {
+		c = got
+		for _, d := range restart {
+			if !pending(got, d.Name) {
+				return false
+			}
+		}
+		return true
+	})
+	if err != nil {
+		return fmt.Errorf("waiting for Patroni to take the new configuration: %w", err)
+	}
+	for _, d := range s.DB {
+		if pending(c, d.Name) && !slices.ContainsFunc(restart, func(n Node) bool { return n.Name == d.Name }) {
+			restart = append(restart, d)
+		}
+	}
+	if len(restart) == 0 {
+		return nil
+	}
+	// The leader last: its restart is the only one the control planes see,
+	// a few seconds without writes, and by then the rest are back.
+	slices.SortStableFunc(restart, func(a, b Node) int {
+		al, bl := a.Name == c.Leader(), b.Name == c.Leader()
+		switch {
+		case al && !bl:
+			return 1
+		case bl && !al:
+			return -1
+		}
+		return 0
+	})
+	for _, d := range restart {
+		if _, err := restJSON(ctx, d.IP, http.MethodPost, "/restart", []byte(`{"restart_pending": true}`), sec.RestAPI, 2*time.Minute); err != nil {
+			return fmt.Errorf("restarting PostgreSQL on %s for its new shared_buffers: %w", d.Name, err)
+		}
+		if err := s.waitFor(ctx, s.dbHosts(), func(c Cluster) bool { return c.Complete(len(s.DB)) }); err != nil {
+			return fmt.Errorf("after restarting %s: %w", d.Name, err)
+		}
+		s.emit("memory", true, "%s: PostgreSQL restarted with the new shared_buffers", d.Name)
+	}
+	return nil
+}
@@ -465,3 +465,97 @@ func TestAControlPlaneWithoutTheAuthorityConnectsByTLSOnceTheClusterHasIt(t *tes
 		t.Errorf("TLS asked of a cluster without it: %s", dsn)
 	}
 }
+
+func TestPostgreSQLIsGivenAShareOfTheMachinesMemory(t *testing.T) {
+	total, ok := MemTotal("MemTotal:        4009848 kB\nMemFree:          976000 kB\n")
+	if !ok || total != 4009848<<10 {
+		t.Fatalf("MemTotal %d %v", total, ok)
+	}
+	own := MemorySettings(total, false)
+	if own["shared_buffers"] != "978MB" || own["effective_cache_size"] != "2936MB" ||
+		own["maintenance_work_mem"] != "244MB" || own["work_mem"] != "15MB" {
+		t.Errorf("a machine of its own: %v", own)
+	}
+	// Beside a control plane, less of it.
+	if sh := MemorySettings(total, true); sh["shared_buffers"] != "587MB" || sh["work_mem"] != "7MB" {
+		t.Errorf("a shared machine: %v", sh)
+	}
+	// Never below what PostgreSQL needs, never past what it can use.
+	if tiny := MemorySettings(512<<20, true); tiny["shared_buffers"] != "96MB" || tiny["work_mem"] != "4MB" {
+		t.Errorf("a small machine: %v", tiny)
+	}
+	if big := MemorySettings(1<<40, false); big["shared_buffers"] != "32768MB" || big["maintenance_work_mem"] != "2048MB" {
+		t.Errorf("a large machine: %v", big)
+	}
+}
+
+func TestNewSharedBuffersRestartTheReplicasBeforeTheLeader(t *testing.T) {
+	var mu sync.Mutex
+	var restarted []string
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if r.Method == http.MethodPost && r.URL.Path == "/restart" {
+			if _, pass, _ := r.BasicAuth(); pass != "pw" {
+				w.WriteHeader(http.StatusUnauthorized)
+				return
+			}
+			host, _, _ := strings.Cut(r.Host, ":")
+			mu.Lock()
+			restarted = append(restarted, host)
+			mu.Unlock()
+		}
+	}))
+	defer srv.Close()
+	u, _ := url.Parse(srv.URL)
+	old := restPort
+	restPort, _ = strconv.Atoi(u.Port())
+	defer func() { restPort = old }()
+
+	answers := map[string]string{"cat /proc/meminfo": "MemTotal: 4009848 kB\n", "WANT": "restart\n"}
+	leader := &machine{name: "a", answers: answers}
+	replica := &machine{name: "b", answers: answers}
+	up := Cluster{Members: []MemberState{{Name: "a", Role: "leader", State: "running", PendingRestart: true},
+		{Name: "b", Role: "replica", State: "streaming", PendingRestart: true}}}
+	s := Setup{
+		DB:     []Node{{Member{"a", "127.0.0.1"}, leader}, {Member{"b", "localhost"}, replica}},
+		Status: func(context.Context, []string) (Cluster, error) { return up, nil },
+		Wait:   time.Second,
+	}
+	if err := s.memory(context.Background(), Secrets{RestAPI: "pw"}); err != nil {
+		t.Fatal(err)
+	}
+	if strings.Join(restarted, " ") != "localhost 127.0.0.1" {
+		t.Errorf("restarted %v; the replica first, then the leader", restarted)
+	}
+}
+
+func TestAMemberLeftPendingARestartIsRestartedOnTheNextRun(t *testing.T) {
+	// A run that stopped after writing the configuration, or restarted before
+	// Patroni had taken it: nothing to change, but PostgreSQL still on the old.
+	var restarted []string
+	srv := httptest.NewServer(http.HandlerFunc(func(_ http.ResponseWriter, r *http.Request) {
+		if r.URL.Path == "/restart" {
+			host, _, _ := strings.Cut(r.Host, ":")
+			restarted = append(restarted, host)
+		}
+	}))
+	defer srv.Close()
+	u, _ := url.Parse(srv.URL)
+	old := restPort
+	restPort, _ = strconv.Atoi(u.Port())
+	defer func() { restPort = old }()
+
+	answers := map[string]string{"cat /proc/meminfo": "MemTotal: 4009848 kB\n", "WANT": "same\n"}
+	up := Cluster{Members: []MemberState{{Name: "a", Role: "leader", State: "running"},
+		{Name: "b", Role: "replica", State: "streaming", PendingRestart: true}}}
+	s := Setup{
+		DB:     []Node{{Member{"a", "127.0.0.1"}, &machine{answers: answers}}, {Member{"b", "localhost"}, &machine{answers: answers}}},
+		Status: func(context.Context, []string) (Cluster, error) { return up, nil },
+		Wait:   time.Second,
+	}
+	if err := s.memory(context.Background(), Secrets{RestAPI: "pw"}); err != nil {
+		t.Fatal(err)
+	}
+	if strings.Join(restarted, " ") != "localhost" {
+		t.Errorf("restarted %v; only the member pending", restarted)
+	}
+}
@@ -128,6 +128,9 @@ func (s *Setup) Run(ctx context.Context) (string, error) {
 	if err := s.ensurePatroniTLS(ctx, sec); err != nil {
 		return "", fmt.Errorf("patroni api: %w", err)
 	}
+	if err := s.memory(ctx, sec); err != nil {
+		return "", fmt.Errorf("memory: %w", err)
+	}
 	if err := s.database(ctx); err != nil {
 		return "", fmt.Errorf("database: %w", err)
 	}
@@ -19,6 +19,9 @@ type MemberState struct {
 	State    string `json:"state"`
 	Host     string `json:"host"`
 	Timeline int    `json:"timeline"`
+	// PendingRestart is a parameter changed that PostgreSQL takes only on
+	// restart.
+	PendingRestart bool `json:"pending_restart"`
 	// Lag is how far behind the leader a replica is, in bytes; -1 when
 	// Patroni does not know, which is itself worth an alert.
 	Lag    int64           `json:"-"`
@@ -0,0 +1,61 @@
+package router
+
+import (
+	"crypto/sha256"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/auth"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+func TestAnOwnedAdapterIsSeenOnlyInItsPartOfTheTree(t *testing.T) {
+	e := newFakeEngine(t, "qwen2.5-14b-instruct")
+	rt, owner := pricedRouter(t, e.srv.URL, "tone")
+	ctx := t.Context()
+	key, _ := rt.store.KeyByID(ctx, "key-1")
+	team, _ := rt.store.CreateTeam(ctx, mustOrgOf(t, rt), "other-team")
+	other, _ := rt.store.CreateProject(ctx, team, "other")
+	otherSecret := store.KeyPrefix + strings.Repeat("b", 43)
+	sum := sha256.Sum256([]byte(otherSecret))
+	if err := rt.store.CreateKey(ctx, store.Key{ID: "key-2", Hash: sum[:], Prefix: "bbbb", Name: "o", ProjectID: other, CreatedAt: time.Now()}); err != nil {
+		t.Fatal(err)
+	}
+	rt.storeModels([]catalog.Model{{Name: "qwen2.5-14b-instruct", Aliases: []string{"q14"},
+		Adapters: []catalog.Adapter{{Name: "tone", Source: "/t.gguf", Owner: "project:" + key.ProjectID}}}})
+	h := rt.Handler()
+	ask := func(secret string) int {
+		req := httptest.NewRequest(http.MethodPost, "/v1/chat/completions",
+			strings.NewReader(`{"model":"tone","messages":[{"role":"user","content":"hi"}]}`))
+		if secret != "" {
+			req.Header.Set("Authorization", "Bearer "+secret)
+		}
+		rec := httptest.NewRecorder()
+		h.ServeHTTP(rec, req)
+		return rec.Code
+	}
+	if code := ask(owner); code != http.StatusOK {
+		t.Errorf("its own project: %d", code)
+	}
+	if code := ask(otherSecret); code != http.StatusNotFound {
+		t.Errorf("another project: %d", code)
+	}
+	list := func(secret string) string {
+		req := httptest.NewRequest(http.MethodGet, "/v1/models", nil)
+		req.Header.Set("Authorization", "Bearer "+secret)
+		rec := httptest.NewRecorder()
+		h.ServeHTTP(rec, req)
+		return rec.Body.String()
+	}
+	if !strings.Contains(list(owner), `"tone"`) || strings.Contains(list(otherSecret), `"tone"`) {
+		t.Errorf("listed: own %s\nother %s", list(owner), list(otherSecret))
+	}
+	// No identity, no owned adapter.
+	if rt.adapterHidden(auth.Principal{}, false, "tone") != true {
+		t.Error("an owned adapter was shown to a caller with no identity")
+	}
+}
@@ -412,9 +412,9 @@ type parsedRequest struct {
 	guardIn guard.Verdict
 	// knowledge is set when passages from knowledge bases were added.
 	knowledge bool
-	// kbOrg is the organization whose passages were added, for the prefix
+	// kbSalt is the collections whose passages were added, for the prefix
 	// cache's salt.
-	kbOrg string
+	kbSalt string
 	// noStore is a policy that keeps nothing of this request.
 	noStore bool
 	// inCluster is a project whose requests may not leave the cluster.
@@ -484,7 +484,8 @@ func (r *Router) handleInference(path string) http.HandlerFunc {
 			}
 		}

-		if r.hiddenModel(parsed.Model) {
+		p0, authed := auth.FromContext(req.Context())
+		if r.hiddenModel(parsed.Model) || r.adapterHidden(p0, authed, parsed.Model) {
 			// Said as a model that is not there: it is not, for clients.
 			writeErrorCode(w, http.StatusNotFound, "invalid_request_error", "model_not_found",
 				fmt.Sprintf("the model %q does not exist", parsed.Model))
@@ -1041,7 +1042,7 @@ func (r *Router) handleModels(w http.ResponseWriter, req *http.Request) {
 	principal, authed := auth.FromContext(req.Context())
 	names := make([]string, 0, len(seen))
 	for n := range seen {
-		if authed && !r.permits(principal, n) || r.hiddenModel(n) {
+		if authed && !r.permits(principal, n) || r.hiddenModel(n) || r.adapterHidden(principal, authed, n) {
 			continue
 		}
 		names = append(names, n)
@@ -10,6 +10,7 @@ import (
 	"fmt"
 	"net/http"
 	"net/url"
+	"sort"
 	"strconv"
 	"strings"
 	"time"
@@ -225,7 +226,14 @@ func (r *Router) retrieve(ctx context.Context, w http.ResponseWriter, req *parse
 	req.UpstreamBody = body
 	req.PromptTokens = len(body) / 4
 	req.knowledge = true
-	req.kbOrg = e.OrgID
+	// The collections searched, which is who else could have had these
+	// passages added: the prefix cache is shared among exactly them.
+	ids := make([]string, len(colls))
+	for i, c := range colls {
+		ids[i] = c.ID
+	}
+	sort.Strings(ids)
+	req.kbSalt = strings.Join(ids, ",")
 	// Set now: the proxy writes the headers when the answer starts.
 	w.Header().Set(sourcesHeader, strings.Join(sources, ","))
 	result := "found"
@@ -429,14 +437,16 @@ func (r *Router) kbDefaults(ctx context.Context, p auth.Principal, project strin
 	return names
 }

-// withCacheSalt sets vLLM's cache_salt to one derived from an organization:
-// the same for its requests, so they still share a cache among themselves.
-func withCacheSalt(body []byte, org string) []byte {
+// withCacheSalt sets vLLM's cache_salt to one derived from key — the
+// collections a request's passages came from — so requests that can see the
+// same passages still share a cache, and nobody else's do: not another
+// organization's, nor another team's of the same one.
+func withCacheSalt(body []byte, key string) []byte {
 	var top map[string]json.RawMessage
 	if json.Unmarshal(body, &top) != nil {
 		return body
 	}
-	sum := sha256.Sum256([]byte("llmgw-kb\x00" + org))
+	sum := sha256.Sum256([]byte("llmgw-kb\x00" + key))
 	top["cache_salt"], _ = json.Marshal(hex.EncodeToString(sum[:16]))
 	out, err := json.Marshal(top)
 	if err != nil {
@@ -52,9 +52,9 @@ func TestPassagesJoinTheLeadingSystemMessage(t *testing.T) {
 	}
 }

-func TestTheCacheSaltIsAnOrganizations(t *testing.T) {
+func TestTheCacheSaltFollowsTheCollections(t *testing.T) {
 	body := []byte(`{"model":"m","messages":[],"temperature":0}`)
-	a1, a2, b := withCacheSalt(body, "org-a"), withCacheSalt(body, "org-a"), withCacheSalt(body, "org-b")
+	a1, a2, b := withCacheSalt(body, "kb_a,kb_b"), withCacheSalt(body, "kb_a,kb_b"), withCacheSalt(body, "kb_a")
 	var x, y struct {
 		Salt string  `json:"cache_salt"`
 		T    float64 `json:"temperature"`
@@ -202,7 +202,7 @@ func mustID(t *testing.T, s *store.DB, name string) string {
 	return ""
 }

-func TestAVLLMReplicaGetsTheOrganizationsSaltOnlyWithPassages(t *testing.T) {
+func TestAVLLMReplicaGetsASaltOnlyWithPassages(t *testing.T) {
 	dsn := os.Getenv("LLMGW_TEST_POSTGRES")
 	if dsn == "" {
 		t.Skip("set LLMGW_TEST_POSTGRES to a pgvector database to run this")
@@ -713,7 +713,7 @@ func (r *Router) storeModels(models []catalog.Model) {
 		// is applied to the request on the way there.
 		for i, a := range entry.Adapters {
 			m[a.Name] = entry
-			ads[a.Name] = adapterRef{base: entry.Name, index: i, scale: a.EffectiveScale(),
+			ads[a.Name] = adapterRef{base: entry.Name, owner: a.Owner, index: i, scale: a.EffectiveScale(),
 				limit: entry.MaxLoadedAdapters, path: engines.RuntimeLoRAPath(a.Name, a.Source)}
 		}
 	}
@@ -735,7 +735,9 @@ func (r *Router) storeModels(models []catalog.Model) {

 // adapterRef is where an adapter runs and how it is applied.
 type adapterRef struct {
-	base  string
+	base string
+	// owner is who sees it (catalog.Adapter.Owner); empty is everyone.
+	owner string
 	index int
 	scale float64
 	// limit is the model's max_loaded_adapters: above zero, the adapter
@@ -757,6 +759,33 @@ func (r *Router) adapter(name string) (adapterRef, bool) {
 	return adapterRef{}, false
 }

+// adapterHidden reports whether name is an adapter its owner keeps from this
+// caller: one outside the part of the tree it belongs to, or one with no
+// identity at all. Such a caller is told there is no such model.
+func (r *Router) adapterHidden(p auth.Principal, authed bool, name string) bool {
+	ad, ok := r.adapter(name)
+	if !ok || ad.owner == "" {
+		return false
+	}
+	if !authed {
+		return true
+	}
+	scope, id, _ := strings.Cut(ad.owner, ":")
+	switch scope {
+	case "org":
+		return p.OrgID != id
+	case "team":
+		return p.TeamID != id
+	case "project":
+		return p.ProjectID != id
+	case "user":
+		return p.UserID != id
+	case "key":
+		return p.KeyID != id
+	}
+	return true
+}
+
 // LastUsed reports when each endpoint was last sent a request, satisfying
 // control.UsageSource so the control plane can tell a busy replica from one
 // holding its cards out of habit.
@@ -1090,13 +1119,13 @@ func (r *Router) dispatchOne(ctx context.Context, w http.ResponseWriter, req *pa
 		if isAdapter && !onDemand && dec.Endpoint.Engine != "vllm" {
 			body = withAdapter(body, model, ad.index, ad.scale)
 		}
-		// Passages of one organization's documents never share cached
-		// blocks with another's: a prefix found warm would say what another
-		// tenant asked about, by how fast its answer started. vLLM keys its
-		// prefix cache by this salt; other engines are not sent a field
-		// they may refuse.
-		if req.kbOrg != "" && dec.Endpoint.Engine == "vllm" && req.ContentType == "" {
-			body = withCacheSalt(body, req.kbOrg)
+		// Passages share cached blocks only with requests that could have
+		// been given the same ones: a prefix found warm would otherwise say
+		// what another tenant asked about, by how fast its answer started.
+		// vLLM keys its prefix cache by this salt; other engines are not
+		// sent a field they may refuse.
+		if req.kbSalt != "" && dec.Endpoint.Engine == "vllm" && req.ContentType == "" {
+			body = withCacheSalt(body, req.kbSalt)
 		}
 		// What computes the answer, said before the proxy writes the headers.
 		// A model can have replicas on cards and on CPUs at once, and the two
@@ -0,0 +1,30 @@
+package store
+
+import (
+	"context"
+	"database/sql"
+	"errors"
+	"time"
+)
+
+// PutClusterState keeps a value under a key, for machines that read the
+// store rather than a control plane; it reports whether it changed.
+func (s *DB) PutClusterState(ctx context.Context, key, value string) (bool, error) {
+	if old, err := s.ClusterState(ctx, key); err == nil && old == value {
+		return false, nil
+	}
+	_, err := s.db.ExecContext(ctx, `INSERT INTO cluster_state (key, value, updated_at) VALUES (?, ?, ?)
+		ON CONFLICT (key) DO UPDATE SET value = excluded.value, updated_at = excluded.updated_at`,
+		key, value, millis(time.Now()))
+	return err == nil, err
+}
+
+// ClusterState reads a value; ErrNotFound when there is none.
+func (s *DB) ClusterState(ctx context.Context, key string) (string, error) {
+	var v string
+	err := s.db.QueryRowContext(ctx, `SELECT value FROM cluster_state WHERE key = ?`, key).Scan(&v)
+	if errors.Is(err, sql.ErrNoRows) {
+		return "", ErrNotFound
+	}
+	return v, err
+}
@@ -73,6 +73,9 @@ var copyOrder = []string{
 	"eval_runs",
 	"eval_cases",
 	"federation_peers",
+	// Before their artifacts, which point at them.
+	"train_jobs",
+	"train_artifacts",
 }

 // notCopied are tables deliberately left behind.
@@ -83,7 +86,8 @@ var copyOrder = []string{
 // the exact moment somebody is watching a migration and wondering why nothing
 // is being placed.
 var notCopied = map[string]string{
-	"leadership": "a lease on a running process, meaningless in a new store",
+	"leadership":    "a lease on a running process, meaningless in a new store",
+	"cluster_state": "worked out again by the leader from the settings and the machines, within minutes",
 }

 // CopyResult is how many rows each table received.
@@ -29,9 +29,13 @@ import (
 // ErrKnowledgeOff is a store that cannot hold knowledge bases.
 var ErrKnowledgeOff = errors.New("knowledge bases are off")

-// MaxEmbeddingDims is the most dimensions pgvector can index (HNSW on vector).
+// MaxEmbeddingDims is the most dimensions pgvector can index with HNSW: on
+// vector; halfvec takes up to MaxHalfvecDims.
 const MaxEmbeddingDims = 2000

+// MaxHalfvecDims is the most dimensions an HNSW index on halfvec takes.
+const MaxHalfvecDims = 4000
+
 // KBPerm is what a grant allows.
 type KBPerm string

@@ -50,8 +54,11 @@ type KBCollection struct {
 	Name       string
 	EmbedModel string
 	Dims       int
-	CreatedAt  time.Time
-	CreatedBy  string
+	// VecType is how its vectors are indexed: vector, or halfvec where
+	// pgvector has it (0.7 and later), at half the index's memory.
+	VecType   string
+	CreatedAt time.Time
+	CreatedBy string
 }

 // KBGrant lets a scope, and everything below it, read or write a collection.
@@ -133,11 +140,16 @@ func (s *DB) ensureKnowledge(ctx context.Context) error {
 				"as postgres, run CREATE EXTENSION vector in it (%v)", ErrKnowledgeOff, err)
 		}
 	}
+	var version string
+	if err := s.db.QueryRowContext(ctx, `SELECT extversion FROM pg_extension WHERE extname = 'vector'`).Scan(&version); err == nil {
+		s.kbHalfvec = versionAtLeast(version, 0, 7)
+	}
 	var exists bool
 	if err := s.db.QueryRowContext(ctx, `SELECT to_regclass('kb_chunks') IS NOT NULL`).Scan(&exists); err != nil {
 		return fmt.Errorf("%w: %v", ErrKnowledgeOff, err)
 	}
 	if exists {
+		s.kbBackfillIndexes(ctx)
 		return nil
 	}
 	err := s.inTx(ctx, func(c conn) error {
@@ -197,6 +209,52 @@ func kbVectorIndex(collection string) string {
 	return "kb_vec_" + strings.TrimPrefix(collection, "kb_")
 }

+// kbTextIndex is the name of a collection's full-text index.
+func kbTextIndex(collection string) string { return "kb_tsv_" + strings.TrimPrefix(collection, "kb_") }
+
+// cast is the type a collection's vectors are compared as, which its index
+// is built on: the query must say the same for the planner to use it.
+func (c KBCollection) cast() string {
+	t := c.VecType
+	if t != "halfvec" {
+		t = "vector"
+	}
+	return "::" + t + "(" + strconv.Itoa(c.Dims) + ")"
+}
+
+// versionAtLeast compares an extension's "major.minor.patch".
+func versionAtLeast(v string, major, minor int) bool {
+	parts := strings.SplitN(v, ".", 3)
+	if len(parts) < 2 {
+		return false
+	}
+	ma, err1 := strconv.Atoi(parts[0])
+	mi, err2 := strconv.Atoi(parts[1])
+	if err1 != nil || err2 != nil {
+		return false
+	}
+	return ma > major || ma == major && mi >= minor
+}
+
+// kbBackfillIndexes gives collections made before their full-text index
+// existed one. Best effort, at open: a search still works without it.
+func (s *DB) kbBackfillIndexes(ctx context.Context) {
+	cs, err := queryAll(ctx, s.db, `SELECT id, org_id FROM kb_collections`, nil, func(r *sql.Rows) ([2]string, error) {
+		var x [2]string
+		return x, r.Scan(&x[0], &x[1])
+	})
+	if err != nil {
+		return
+	}
+	for _, c := range cs {
+		if !kbIDPattern.MatchString(c[0]) {
+			continue
+		}
+		_, _ = s.db.ExecContext(ctx, `CREATE INDEX IF NOT EXISTS `+kbTextIndex(c[0])+` ON `+kbPartition(c[1])+
+			` USING gin (tsv) WHERE collection = `+sqlLiteral(c[0]))
+	}
+}
+
 func sqlLiteral(v string) string { return "'" + strings.ReplaceAll(v, "'", "''") + "'" }

 // kbScope sets, for this transaction, which rows of kb_chunks it may touch.
@@ -263,14 +321,22 @@ func (s *DB) CreateCollection(ctx context.Context, c KBCollection) (KBCollection
 		return KBCollection{}, errors.New("a collection needs a name")
 	case c.EmbedModel == "":
 		return KBCollection{}, errors.New("a collection needs the embedding model its documents are embedded with")
-	case c.Dims < 1 || c.Dims > MaxEmbeddingDims:
-		return KBCollection{}, fmt.Errorf("%d dimensions: an indexed embedding has 1 to %d", c.Dims, MaxEmbeddingDims)
+	case c.Dims < 1 || c.Dims > MaxHalfvecDims || c.Dims > MaxEmbeddingDims && !s.kbHalfvec:
+		limit := MaxEmbeddingDims
+		if s.kbHalfvec {
+			limit = MaxHalfvecDims
+		}
+		return KBCollection{}, fmt.Errorf("%d dimensions: an indexed embedding has 1 to %d", c.Dims, limit)
 	case c.OwnerScope == ScopeUser && c.OrgID == "":
 		return KBCollection{}, errors.New("a person's collection names the organization it belongs to")
 	case c.OwnerScope == ScopeKey:
 		return KBCollection{}, errors.New("a key cannot own a collection; its project or its person can")
 	}
 	c.ID = NewID("kb_")
+	c.VecType = "vector"
+	if s.kbHalfvec {
+		c.VecType = "halfvec"
+	}
 	if c.CreatedAt.IsZero() {
 		c.CreatedAt = time.Now()
 	}
@@ -292,9 +358,9 @@ func (s *DB) CreateCollection(ctx context.Context, c KBCollection) (KBCollection
 			return fmt.Errorf("%w: collection %q", ErrDuplicateName, c.Name)
 		}
 		if _, err := tx.ExecContext(ctx, `INSERT INTO kb_collections
-			(id, org_id, owner_scope, owner_id, name, embed_model, dims, created_at, created_by)
-			VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`,
-			c.ID, c.OrgID, string(c.OwnerScope), c.OwnerID, c.Name, c.EmbedModel, c.Dims,
+			(id, org_id, owner_scope, owner_id, name, embed_model, dims, vec_type, created_at, created_by)
+			VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
+			c.ID, c.OrgID, string(c.OwnerScope), c.OwnerID, c.Name, c.EmbedModel, c.Dims, c.VecType,
 			millis(c.CreatedAt), c.CreatedBy); err != nil {
 			return err
 		}
@@ -303,12 +369,18 @@ func (s *DB) CreateCollection(ctx context.Context, c KBCollection) (KBCollection
 			` PARTITION OF kb_chunks FOR VALUES IN (`+sqlLiteral(c.OrgID)+`)`); err != nil {
 			return err
 		}
-		// A partial index per collection, on its organization's partition
+		// Partial indexes per collection, on its organization's partition
 		// only: the dimension is the collection's, and a search, which
-		// names one collection at a time, uses exactly this one.
-		_, err = tx.ExecContext(ctx, `CREATE INDEX `+kbVectorIndex(c.ID)+` ON `+part+
-			` USING hnsw ((embedding::vector(`+strconv.Itoa(c.Dims)+`)) vector_cosine_ops)`+
-			` WHERE collection = `+sqlLiteral(c.ID))
+		// names one collection at a time, uses exactly these — a text
+		// search over one collection does not walk its whole
+		// organization's index.
+		if _, err := tx.ExecContext(ctx, `CREATE INDEX `+kbVectorIndex(c.ID)+` ON `+part+
+			` USING hnsw ((embedding`+c.cast()+`) `+c.VecType+`_cosine_ops)`+
+			` WHERE collection = `+sqlLiteral(c.ID)); err != nil {
+			return err
+		}
+		_, err = tx.ExecContext(ctx, `CREATE INDEX `+kbTextIndex(c.ID)+` ON `+part+
+			` USING gin (tsv) WHERE collection = `+sqlLiteral(c.ID))
 		return err
 	})
 	if err != nil {
@@ -317,13 +389,13 @@ func (s *DB) CreateCollection(ctx context.Context, c KBCollection) (KBCollection
 	return c, nil
 }

-const kbCollectionCols = `id, org_id, owner_scope, owner_id, name, embed_model, dims, created_at, created_by`
+const kbCollectionCols = `id, org_id, owner_scope, owner_id, name, embed_model, dims, vec_type, created_at, created_by`

 func scanCollection(sc scanner) (KBCollection, error) {
 	var c KBCollection
 	var scope string
 	var at int64
-	err := sc.Scan(&c.ID, &c.OrgID, &scope, &c.OwnerID, &c.Name, &c.EmbedModel, &c.Dims, &at, &c.CreatedBy)
+	err := sc.Scan(&c.ID, &c.OrgID, &scope, &c.OwnerID, &c.Name, &c.EmbedModel, &c.Dims, &c.VecType, &at, &c.CreatedBy)
 	c.OwnerScope, c.CreatedAt = Scope(scope), fromMillis(at)
 	return c, err
 }
@@ -377,7 +449,10 @@ func (s *DB) DeleteCollection(ctx context.Context, id string) error {
 		if _, err := c.ExecContext(ctx, `DELETE FROM kb_defaults WHERE collections = ''`); err != nil {
 			return err
 		}
-		_, err = c.ExecContext(ctx, `DROP INDEX IF EXISTS `+kbVectorIndex(id))
+		if _, err := c.ExecContext(ctx, `DROP INDEX IF EXISTS `+kbVectorIndex(id)); err != nil {
+			return err
+		}
+		_, err = c.ExecContext(ctx, `DROP INDEX IF EXISTS `+kbTextIndex(id))
 		return err
 	})
 }
@@ -729,6 +804,14 @@ func (s *DB) SearchChunks(ctx context.Context, q KBSearch) ([]KBHit, error) {
 		if err := kbScope(ctx, tx, q.OrgID, ids); err != nil {
 			return err
 		}
+		// pgvector's HNSW search returns at most ef_search rows (40 by
+		// default), whatever LIMIT asks: more candidates need it raised.
+		if len(q.Embeddings) > 0 {
+			ef := min(max(candidates(q.K), 40), 1000)
+			if _, err := tx.ExecContext(ctx, `SET LOCAL hnsw.ef_search = `+strconv.Itoa(ef)); err != nil {
+				return err
+			}
+		}
 		for _, c := range q.Collections {
 			// The literal collection and cast are what let the planner use
 			// that collection's partial index.
@@ -739,7 +822,7 @@ func (s *DB) SearchChunks(ctx context.Context, q KBSearch) ([]KBHit, error) {
 				if len(v) != c.Dims {
 					return fmt.Errorf("the query has %d dimensions; collection %s takes %d", len(v), c.Name, c.Dims)
 				}
-				cast := `::vector(` + strconv.Itoa(c.Dims) + `)`
+				cast := c.cast()
 				sim = `1 - (k.embedding` + cast + ` <=> ?` + cast + `)`
 				args = append(args, vectorText(v))
 				if err := query(tx, false, `SELECT k.collection, k.doc_id, d.name, k.ord, k.body, `+sim+`, 0::float8
@@ -753,15 +836,20 @@ func (s *DB) SearchChunks(ctx context.Context, q KBSearch) ([]KBHit, error) {
 			if tsq != "" {
 				// The text search's finds are measured against the question's
 				// meaning too, so the two are compared on one scale.
+				// Candidates by the cheap rank first, from the collection's
+				// own index; the share of words held and the similarity are
+				// worked out for those alone, not for every chunk that holds
+				// one word of the question.
 				targs := append([]any{}, args...)
 				for _, t := range terms {
 					targs = append(targs, t)
 				}
-				targs = append(targs, tsq, q.OrgID, candidates(q.K))
+				targs = append(targs, q.OrgID, tsq, tsq, candidates(q.K))
 				if err := query(tx, true, `SELECT k.collection, k.doc_id, d.name, k.ord, k.body, `+sim+`, `+coverage+` AS coverage
-					FROM kb_chunks k JOIN kb_documents d ON d.id = k.doc_id, to_tsquery('simple', ?) tq
-					WHERE k.org_id = ? AND k.collection = `+lit+` AND k.tsv @@ tq
-					ORDER BY coverage DESC, ts_rank(k.tsv, tq) DESC LIMIT ?`, targs...); err != nil {
+					FROM (SELECT * FROM kb_chunks k
+					      WHERE k.org_id = ? AND k.collection = `+lit+` AND k.tsv @@ to_tsquery('simple', ?)
+					      ORDER BY ts_rank(k.tsv, to_tsquery('simple', ?)) DESC LIMIT ?) k
+					JOIN kb_documents d ON d.id = k.doc_id`, targs...); err != nil {
 					return err
 				}
 			}
@@ -887,8 +975,10 @@ func (s *DB) kbForgetScope(ctx context.Context, c conn, scope Scope, id string)
 			return err
 		}
 		if kbIDPattern.MatchString(cid) {
-			if _, err := c.ExecContext(ctx, `DROP INDEX IF EXISTS `+kbVectorIndex(cid)); err != nil {
-				return err
+			for _, idx := range []string{kbVectorIndex(cid), kbTextIndex(cid)} {
+				if _, err := c.ExecContext(ctx, `DROP INDEX IF EXISTS `+idx); err != nil {
+					return err
+				}
 			}
 		}
 	}
@@ -1026,3 +1116,32 @@ func (s *DB) KBDefaultFor(ctx context.Context, e Effective) ([]string, error) {
 	}
 	return nil, nil
 }
+
+// CollectionChunks lists a collection's chunks in document order, for
+// building training data from them (BACKLOG F4). At most limit.
+func (s *DB) CollectionChunks(ctx context.Context, c KBCollection, limit int) ([]KBHit, error) {
+	if err := s.KnowledgeAvailable(); err != nil {
+		return nil, err
+	}
+	if !kbIDPattern.MatchString(c.ID) {
+		return nil, ErrNotFound
+	}
+	var out []KBHit
+	err := s.inTx(ctx, func(tx conn) error {
+		if err := kbScope(ctx, tx, c.OrgID, []string{c.ID}); err != nil {
+			return err
+		}
+		var err error
+		out, err = queryAll(ctx, tx, `SELECT k.collection, k.doc_id, d.name, k.ord, k.body
+			FROM kb_chunks k JOIN kb_documents d ON d.id = k.doc_id
+			WHERE k.org_id = ? AND k.collection = `+sqlLiteral(c.ID)+`
+			ORDER BY d.name, k.ord LIMIT ?`, []any{c.OrgID, limit},
+			func(r *sql.Rows) (KBHit, error) {
+				var h KBHit
+				err := r.Scan(&h.Collection, &h.DocID, &h.Document, &h.Ord, &h.Body)
+				return h, err
+			})
+		return err
+	})
+	return out, err
+}
@@ -3,6 +3,7 @@ package store
 import (
 	"context"
 	"errors"
+	"fmt"
 	"math"
 	"strings"
 	"testing"
@@ -320,10 +321,85 @@ func TestPostgresScopesWithCollectionsAreNotDeleted(t *testing.T) {
 	if err := s.DeleteProject(ctx, "proj2"); err != nil {
 		t.Errorf("deleting the project once its collection is gone: %v", err)
 	}
+	for _, idx := range []string{kbVectorIndex(c.ID), kbTextIndex(c.ID)} {
+		if indexExists(t, s, idx) {
+			t.Errorf("the collection's index %s is left behind", idx)
+		}
+	}
+}
+
+func indexExists(t *testing.T, s *DB, name string) bool {
+	t.Helper()
 	var n int
-	if err := s.raw.QueryRowContext(ctx, `SELECT COUNT(*) FROM pg_indexes WHERE indexname = $1`,
-		kbVectorIndex(c.ID)).Scan(&n); err != nil || n != 0 {
-		t.Errorf("the collection's index is left behind (%d, %v)", n, err)
+	if err := s.raw.QueryRowContext(context.Background(), `SELECT COUNT(*) FROM pg_indexes WHERE indexname = $1`,
+		name).Scan(&n); err != nil {
+		t.Fatal(err)
+	}
+	return n == 1
+}
+
+func TestPostgresCollectionsHaveTheirOwnIndexes(t *testing.T) {
+	ctx := context.Background()
+	s := openKB(t)
+	kbTwoOrgs(t, s)
+	c := mustCollection(t, s, ScopeProject, "proj1", "", "docs")
+	if !indexExists(t, s, kbVectorIndex(c.ID)) || !indexExists(t, s, kbTextIndex(c.ID)) {
+		t.Fatal("a new collection lacks its vector or text index")
+	}
+	// The test image's pgvector has halfvec; a collection there uses it, and
+	// is searched as it is indexed.
+	if !s.kbHalfvec || c.VecType != "halfvec" {
+		t.Errorf("halfvec %v, the collection's type %q", s.kbHalfvec, c.VecType)
+	}
+	if got, err := s.Collection(ctx, c.ID); err != nil || got.VecType != c.VecType {
+		t.Errorf("read back %q, %v", got.VecType, err)
+	}
+	var def string
+	if err := s.raw.QueryRowContext(ctx, `SELECT indexdef FROM pg_indexes WHERE indexname = $1`, kbVectorIndex(c.ID)).Scan(&def); err != nil ||
+		!strings.Contains(def, "halfvec_cosine_ops") {
+		t.Errorf("the vector index is %q (%v)", def, err)
+	}
+	if _, err := s.CreateCollection(ctx, KBCollection{OwnerScope: ScopeProject, OwnerID: "proj1",
+		Name: "wide", EmbedModel: "emb", Dims: 3072}); err != nil {
+		t.Errorf("3072 dimensions, which halfvec indexes: %v", err)
+	}
+	if _, err := s.CreateCollection(ctx, KBCollection{OwnerScope: ScopeProject, OwnerID: "proj1",
+		Name: "too-wide", EmbedModel: "emb", Dims: MaxHalfvecDims + 1}); err == nil {
+		t.Error("more dimensions than any index takes were accepted")
+	}
+
+	// A collection made before text indexes were gets one when the store
+	// opens.
+	if _, err := s.raw.ExecContext(ctx, `DROP INDEX `+kbTextIndex(c.ID)); err != nil {
+		t.Fatal(err)
+	}
+	s.kbBackfillIndexes(ctx)
+	if !indexExists(t, s, kbTextIndex(c.ID)) {
+		t.Error("the text index was not put back")
+	}
+
+	// Asking for more than HNSW's default 40 is answered in full. (A table
+	// this small is read without its index: with one, 40 is all HNSW gives
+	// unless ef_search is raised, as an EXPLAIN on 20000 rows showed.)
+	var chunks []KBChunk
+	for i := range 120 {
+		chunks = append(chunks, KBChunk{Body: fmt.Sprintf("parça %d izin", i), Embedding: unit(i % 3)})
+	}
+	if _, _, err := s.PutDocument(ctx, c, KBDocument{Name: "big.md", Digest: "big"}, chunks); err != nil {
+		t.Fatal(err)
+	}
+	hits, err := s.SearchChunks(ctx, KBSearch{OrgID: "org1", Collections: []KBCollection{c},
+		Embeddings: map[string][]float32{"emb": unit(0)}, K: 50})
+	if err != nil || len(hits) != 50 {
+		t.Errorf("K=50 gave %d hits, %v", len(hits), err)
+	}
+}
+
+func TestVersionAtLeast(t *testing.T) {
+	for v, want := range map[string]bool{"0.6.0": false, "0.7.0": true, "0.8.7": true, "1.0": true, "0": false, "x.y": false} {
+		if got := versionAtLeast(v, 0, 7); got != want {
+			t.Errorf("%s: %v", v, got)
+		}
 	}
 }

@@ -0,0 +1,42 @@
+-- Training jobs (BACKLOG F4): a LoRA adapter trained on a node, from a base
+-- model's weights and question/answer pairs, and what it made. The pairs
+-- and the adapter are kept here, in the store every control plane reads:
+-- they are megabytes, and a job's record is worth little without them.
+CREATE TABLE train_jobs (
+  id             TEXT PRIMARY KEY,
+  -- name is the adapter's: what clients will ask for it by.
+  name           TEXT NOT NULL,
+  -- model is the catalog model the adapter is for; base_repo and
+  -- base_revision the weights it is trained on.
+  model          TEXT NOT NULL,
+  base_repo      TEXT NOT NULL,
+  base_revision  TEXT NOT NULL DEFAULT '',
+  node           TEXT NOT NULL,
+  gpu_memory_mib INTEGER NOT NULL,
+  params         TEXT NOT NULL DEFAULT '',
+  gguf           INTEGER NOT NULL DEFAULT 0,
+  data           BYTEA NOT NULL,
+  pairs          INTEGER NOT NULL,
+  -- state is queued, starting, preparing, running, done, failed or
+  -- cancelled.
+  state          TEXT NOT NULL,
+  step           INTEGER NOT NULL DEFAULT 0,
+  steps          INTEGER NOT NULL DEFAULT 0,
+  loss           DOUBLE PRECISION NOT NULL DEFAULT 0,
+  message        TEXT NOT NULL DEFAULT '',
+  peak_mib       INTEGER NOT NULL DEFAULT 0,
+  created_at     BIGINT NOT NULL,
+  started_at     BIGINT NOT NULL DEFAULT 0,
+  finished_at    BIGINT NOT NULL DEFAULT 0,
+  created_by     TEXT NOT NULL DEFAULT ''
+);
+CREATE INDEX train_jobs_by_state ON train_jobs (state);
+
+-- What a job made: the PEFT adapter as a gzipped tar, and for llama.cpp the
+-- adapter as GGUF.
+CREATE TABLE train_artifacts (
+  job_id TEXT NOT NULL REFERENCES train_jobs(id) ON DELETE CASCADE,
+  kind   TEXT NOT NULL,
+  data   BYTEA NOT NULL,
+  PRIMARY KEY (job_id, kind)
+);
@@ -0,0 +1,5 @@
+-- Who an adapter belongs to (BACKLOG F4): scope:ID in the tenancy tree, or
+-- empty for every caller its model allows. An adapter trained on a
+-- knowledge base's documents carries some of what they say, and belongs to
+-- a part of the tree that may read them.
+ALTER TABLE train_jobs ADD COLUMN owner TEXT NOT NULL DEFAULT '';
@@ -0,0 +1,5 @@
+-- How a collection's vectors are indexed: vector (single precision) or
+-- halfvec (half, pgvector 0.7 and later), which halves the index a search
+-- has to hold in memory for the same recall. Fixed when the collection is
+-- made: its index and its queries name the same type.
+ALTER TABLE kb_collections ADD COLUMN vec_type TEXT NOT NULL DEFAULT 'vector';
@@ -0,0 +1,7 @@
+-- What a control plane works out for machines that cannot ask it: the
+-- store's members read it from their own copy of the store.
+CREATE TABLE cluster_state (
+  key        TEXT PRIMARY KEY,
+  value      TEXT NOT NULL,
+  updated_at BIGINT NOT NULL
+);
@@ -0,0 +1,42 @@
+-- Training jobs (BACKLOG F4): a LoRA adapter trained on a node, from a base
+-- model's weights and question/answer pairs, and what it made. The pairs
+-- and the adapter are kept here, in the store every control plane reads:
+-- they are megabytes, and a job's record is worth little without them.
+CREATE TABLE train_jobs (
+  id             TEXT PRIMARY KEY,
+  -- name is the adapter's: what clients will ask for it by.
+  name           TEXT NOT NULL,
+  -- model is the catalog model the adapter is for; base_repo and
+  -- base_revision the weights it is trained on.
+  model          TEXT NOT NULL,
+  base_repo      TEXT NOT NULL,
+  base_revision  TEXT NOT NULL DEFAULT '',
+  node           TEXT NOT NULL,
+  gpu_memory_mib INTEGER NOT NULL,
+  params         TEXT NOT NULL DEFAULT '',
+  gguf           INTEGER NOT NULL DEFAULT 0,
+  data           BLOB NOT NULL,
+  pairs          INTEGER NOT NULL,
+  -- state is queued, starting, preparing, running, done, failed or
+  -- cancelled.
+  state          TEXT NOT NULL,
+  step           INTEGER NOT NULL DEFAULT 0,
+  steps          INTEGER NOT NULL DEFAULT 0,
+  loss           REAL NOT NULL DEFAULT 0,
+  message        TEXT NOT NULL DEFAULT '',
+  peak_mib       INTEGER NOT NULL DEFAULT 0,
+  created_at     INTEGER NOT NULL,
+  started_at     INTEGER NOT NULL DEFAULT 0,
+  finished_at    INTEGER NOT NULL DEFAULT 0,
+  created_by     TEXT NOT NULL DEFAULT ''
+);
+CREATE INDEX train_jobs_by_state ON train_jobs (state);
+
+-- What a job made: the PEFT adapter as a gzipped tar, and for llama.cpp the
+-- adapter as GGUF.
+CREATE TABLE train_artifacts (
+  job_id TEXT NOT NULL REFERENCES train_jobs(id) ON DELETE CASCADE,
+  kind   TEXT NOT NULL,
+  data   BLOB NOT NULL,
+  PRIMARY KEY (job_id, kind)
+);
@@ -0,0 +1,5 @@
+-- Who an adapter belongs to (BACKLOG F4): scope:ID in the tenancy tree, or
+-- empty for every caller its model allows. An adapter trained on a
+-- knowledge base's documents carries some of what they say, and belongs to
+-- a part of the tree that may read them.
+ALTER TABLE train_jobs ADD COLUMN owner TEXT NOT NULL DEFAULT '';
@@ -0,0 +1,7 @@
+-- What a control plane works out for machines that cannot ask it: the
+-- store's members read it from their own copy of the store.
+CREATE TABLE cluster_state (
+  key        TEXT PRIMARY KEY,
+  value      TEXT NOT NULL,
+  updated_at INTEGER NOT NULL
+);
@@ -52,6 +52,9 @@ type DB struct {
 	// kbErr is why this store cannot hold knowledge bases, nil when it can;
 	// set once at open (knowledge.go).
 	kbErr error
+	// kbHalfvec is pgvector 0.7 or later here: new collections index their
+	// vectors as halfvec.
+	kbHalfvec bool
 }

 // OpenSQLite opens (creating if needed) a database and brings it up to date.
@@ -0,0 +1,203 @@
+package store
+
+import (
+	"context"
+	"database/sql"
+	"errors"
+	"time"
+)
+
+// TrainJob is a LoRA adapter trained on a node (BACKLOG F4).
+type TrainJob struct {
+	ID           string
+	Name         string
+	Model        string
+	BaseRepo     string
+	BaseRevision string
+	Node         string
+	// Owner is who the adapter belongs to, scope:ID; empty is everyone.
+	Owner        string
+	GPUMemoryMiB int
+	Params       string
+	GGUF         bool
+	// Data is the pairs, NDJSON; Pairs how many.
+	Data  []byte
+	Pairs int
+
+	State      string
+	Step       int
+	Steps      int
+	Loss       float64
+	Message    string
+	PeakMiB    int
+	CreatedAt  time.Time
+	StartedAt  time.Time
+	FinishedAt time.Time
+	CreatedBy  string
+}
+
+// The states of a training job.
+const (
+	TrainQueued    = "queued"
+	TrainStarting  = "starting"
+	TrainDone      = "done"
+	TrainFailed    = "failed"
+	TrainCancelled = "cancelled"
+)
+
+// TrainOver is whether a state is one a job ends in.
+func TrainOver(state string) bool {
+	return state == TrainDone || state == TrainFailed || state == TrainCancelled
+}
+
+// CreateTrainJob queues a job.
+func (s *DB) CreateTrainJob(ctx context.Context, j TrainJob) error {
+	if j.CreatedAt.IsZero() {
+		j.CreatedAt = time.Now()
+	}
+	gguf := 0
+	if j.GGUF {
+		gguf = 1
+	}
+	_, err := s.db.ExecContext(ctx, `INSERT INTO train_jobs (id, name, model, base_repo, base_revision, node, owner, gpu_memory_mib,
+		params, gguf, data, pairs, state, created_at, created_by) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
+		j.ID, j.Name, j.Model, j.BaseRepo, j.BaseRevision, j.Node, j.Owner, j.GPUMemoryMiB, j.Params, gguf, j.Data, j.Pairs,
+		TrainQueued, millis(j.CreatedAt), j.CreatedBy)
+	return err
+}
+
+const trainCols = `id, name, model, base_repo, base_revision, node, owner, gpu_memory_mib, params, gguf, pairs, state, step, steps,
+	loss, message, peak_mib, created_at, started_at, finished_at, created_by`
+
+func scanTrainJob(sc scanner) (TrainJob, error) {
+	var j TrainJob
+	var gguf int
+	var created, started, finished int64
+	err := sc.Scan(&j.ID, &j.Name, &j.Model, &j.BaseRepo, &j.BaseRevision, &j.Node, &j.Owner, &j.GPUMemoryMiB, &j.Params, &gguf,
+		&j.Pairs, &j.State, &j.Step, &j.Steps, &j.Loss, &j.Message, &j.PeakMiB, &created, &started, &finished, &j.CreatedBy)
+	j.GGUF = gguf != 0
+	j.CreatedAt = fromMillis(created)
+	if started > 0 {
+		j.StartedAt = fromMillis(started)
+	}
+	if finished > 0 {
+		j.FinishedAt = fromMillis(finished)
+	}
+	return j, err
+}
+
+// TrainJobs lists jobs, newest first.
+func (s *DB) TrainJobs(ctx context.Context, limit int) ([]TrainJob, error) {
+	return queryAll(ctx, s.db, `SELECT `+trainCols+` FROM train_jobs ORDER BY created_at DESC LIMIT ?`, []any{limit},
+		func(r *sql.Rows) (TrainJob, error) { return scanTrainJob(r) })
+}
+
+// TrainJob reads one job, with its data.
+func (s *DB) TrainJob(ctx context.Context, id string) (TrainJob, error) {
+	j, err := scanTrainJob(s.db.QueryRowContext(ctx, `SELECT `+trainCols+` FROM train_jobs WHERE id = ?`, id))
+	if errors.Is(err, sql.ErrNoRows) {
+		return TrainJob{}, ErrNotFound
+	}
+	if err != nil {
+		return j, err
+	}
+	err = s.db.QueryRowContext(ctx, `SELECT data FROM train_jobs WHERE id = ?`, id).Scan(&j.Data)
+	return j, err
+}
+
+// QueuedTrainJobs are the jobs waiting for their node, oldest first.
+func (s *DB) QueuedTrainJobs(ctx context.Context) ([]TrainJob, error) {
+	return queryAll(ctx, s.db, `SELECT `+trainCols+` FROM train_jobs WHERE state = ? ORDER BY created_at`, []any{TrainQueued},
+		func(r *sql.Rows) (TrainJob, error) { return scanTrainJob(r) })
+}
+
+// ActiveTrainJobs are the jobs sent to a node and not ended.
+func (s *DB) ActiveTrainJobs(ctx context.Context) ([]TrainJob, error) {
+	return queryAll(ctx, s.db, `SELECT `+trainCols+` FROM train_jobs WHERE state NOT IN (?, ?, ?, ?)`,
+		[]any{TrainQueued, TrainDone, TrainFailed, TrainCancelled},
+		func(r *sql.Rows) (TrainJob, error) { return scanTrainJob(r) })
+}
+
+// UpdateTrainJob records how a job goes; a job that has ended is not
+// changed again, so a late message cannot bring one back.
+func (s *DB) UpdateTrainJob(ctx context.Context, j TrainJob) error {
+	var started, finished int64
+	if !j.StartedAt.IsZero() {
+		started = millis(j.StartedAt)
+	}
+	if !j.FinishedAt.IsZero() {
+		finished = millis(j.FinishedAt)
+	}
+	_, err := s.db.ExecContext(ctx, `UPDATE train_jobs SET state = ?, step = ?, steps = ?, loss = ?, message = ?,
+		peak_mib = ?, started_at = ?, finished_at = ? WHERE id = ? AND state NOT IN (?, ?, ?)`,
+		j.State, j.Step, j.Steps, j.Loss, j.Message, j.PeakMiB, started, finished, j.ID,
+		TrainDone, TrainFailed, TrainCancelled)
+	return err
+}
+
+// PutTrainArtifact keeps what a job made.
+func (s *DB) PutTrainArtifact(ctx context.Context, job, kind string, data []byte) error {
+	_, err := s.db.ExecContext(ctx, `INSERT INTO train_artifacts (job_id, kind, data) VALUES (?, ?, ?)
+		ON CONFLICT (job_id, kind) DO UPDATE SET data = excluded.data`, job, kind, data)
+	return err
+}
+
+// TrainArtifact reads what a job made, of one kind.
+func (s *DB) TrainArtifact(ctx context.Context, job, kind string) ([]byte, error) {
+	var b []byte
+	err := s.db.QueryRowContext(ctx, `SELECT data FROM train_artifacts WHERE job_id = ? AND kind = ?`, job, kind).Scan(&b)
+	if errors.Is(err, sql.ErrNoRows) {
+		return nil, ErrNotFound
+	}
+	return b, err
+}
+
+// TrainArtifactKinds lists what a job made.
+func (s *DB) TrainArtifactKinds(ctx context.Context, job string) ([]string, error) {
+	return queryAll(ctx, s.db, `SELECT kind FROM train_artifacts WHERE job_id = ? ORDER BY kind`, []any{job},
+		func(r *sql.Rows) (string, error) {
+			var k string
+			return k, r.Scan(&k)
+		})
+}
+
+// DeleteTrainJob removes a job and what it made.
+func (s *DB) DeleteTrainJob(ctx context.Context, id string) error {
+	return s.inTx(ctx, func(c conn) error {
+		if _, err := c.ExecContext(ctx, `DELETE FROM train_artifacts WHERE job_id = ?`, id); err != nil {
+			return err
+		}
+		res, err := c.ExecContext(ctx, `DELETE FROM train_jobs WHERE id = ?`, id)
+		if err != nil {
+			return err
+		}
+		if n, _ := res.RowsAffected(); n == 0 {
+			return ErrNotFound
+		}
+		return nil
+	})
+}
+
+// EffectiveFor is a scope's place in the tree, as a caller in it would have
+// it, for asking what it may read: an organization, a team or a project
+// with what is above it, a key as it resolves, a person in an organization.
+func (s *DB) EffectiveFor(ctx context.Context, scope Scope, id, org string) (Effective, error) {
+	switch scope {
+	case ScopeOrg:
+		return Effective{OrgID: id}, nil
+	case ScopeTeam:
+		var o string
+		err := s.db.QueryRowContext(ctx, `SELECT org_id FROM teams WHERE id = ?`, id).Scan(&o)
+		if errors.Is(err, sql.ErrNoRows) {
+			return Effective{}, ErrNotFound
+		}
+		return Effective{TeamID: id, OrgID: o}, err
+	case ScopeProject:
+		return s.ResolveFor(ctx, "", "", id)
+	case ScopeKey:
+		return s.Resolve(ctx, id)
+	case ScopeUser:
+		return Effective{UserID: id, OrgID: org}, nil
+	}
+	return Effective{}, errors.New("no such scope " + string(scope))
+}
@@ -15,13 +15,17 @@ API is reached and what it asks of a request.
 | `llmgw_model` | A model added to the catalog at runtime, as `llmgw ctl model add` adds one. Scaling, class, owner and adapters change in place; any other change to the definition replaces the model, which stops its replicas while it is added again. |
 | `llmgw_quota` | A capacity quota on an organization, team or project: a guarantee, a limit, or both, in GPUs, CPU cores and memory. |
 | `llmgw_guard_attachment` | A guardrail policy attached to an organization, team, project, key or person. The policies are defined in `server.yaml`'s `guard:` section; this says where they apply. |
+| `llmgw_org`, `llmgw_team`, `llmgw_project` | The tenancy tree. A team names its organization and a project its team, by path or ID; a new name renames in place, a new parent moves it with everything under it. Only something empty is removed, and a project that has ever had a key is never empty (the revoked key stays for usage history): rename it rather than destroy it. |
+| `llmgw_key` | An API key in a project, for a person in a project they belong to, or a person's own. The key is in `secret` (sensitive) from the apply that made it; the gateway keeps only a hash. Any change makes a new key, and one revoked or expired outside Terraform is made again. Destroying it revokes it. |
+| `llmgw_membership` | A person in an organization, team or project, as member or manager. People are not made here: they sign up, are invited, or come from the identity provider. Taking a membership away revokes their keys in projects they no longer reach. |
 | `data.llmgw_models` | Every model in the catalog, with its replicas now. |
 | `data.llmgw_nodes` | Every node, with its cards and state now. |

-Not here, because the admin API does not serve them: organizations, teams,
-projects, people and API keys. Those live in the store and are managed with
-`llmgw org`, `llmgw users` and `llmgw keys` on the control plane. A quota or
-an attachment names a team that already exists, by its path or its ID.
+Not here: people's accounts (sign-up, invitation, OIDC/SCIM, and `llmgw
+users` to approve, disable or delete), and key lifetime and spending limits
+(`llmgw keys ttl`, `llmgw keys limits`). A quota or an attachment names a
+part of the tree by its path or its ID; refer to an `llmgw_team`'s `id` to
+have Terraform make the team first.

 Models declared in the control plane's `server.yaml` are not managed here
 either: the cluster refuses to remove or change them through the API, since
@@ -107,8 +111,13 @@ with `version = "0.1.0"` beside `source` in `required_providers`.
  found again by the ID, so renaming the team in the store does not lose it.
 - **Import** works for quotas, by `team:ID` as `GET /api/v1/quotas` lists it
  (the `scope` is then that ID; write it so in the configuration, or the next
-  plan replaces the quota with itself), and for attachments, by
-  `scope:SCOPE-ID:policy` as their `id` shows. Models cannot be imported yet.
+  plan replaces the quota with itself), for attachments, by
+  `scope:SCOPE-ID:policy` as their `id` shows, for organizations, teams,
+  projects and keys by their IDs (an imported key has no `secret`), and for
+  memberships by `USER-ID/scope:SCOPE-ID`. Models cannot be imported yet.
+- **A project's `path` follows a move of its team at the next refresh**: the
+  project did not change, so the apply that moved the team does not rewrite
+  it.

 ## Developing it

@@ -66,6 +66,33 @@ resource "llmgw_model" "llama_70b" {
  fallbacks = [llmgw_model.qwen_small.name]
 }

+# A team and a project in Acme, a key for its CI, and a person in the team.
+resource "llmgw_team" "search" {
+  name = "search"
+  org  = "Acme"
+}
+
+resource "llmgw_project" "ranking" {
+  name = "ranking"
+  team = llmgw_team.search.id
+}
+
+resource "llmgw_key" "ranking_ci" {
+  name       = "ci"
+  project    = llmgw_project.ranking.id
+  expires_in = "2160h"
+}
+
+resource "llmgw_membership" "ayse" {
+  user   = "ayse@example.com"
+  target = "team:${llmgw_team.search.id}"
+}
+
+output "ranking_ci_key" {
+  value     = llmgw_key.ranking_ci.secret
+  sensitive = true
+}
+
 # The ml team may hold at most eight GPUs, and is always given two.
 resource "llmgw_quota" "ml" {
  scope     = "team:Acme/ml"
@@ -40,6 +40,22 @@ var Endpoints = []string{
 	"GET /api/v1/guard",
 	"POST /api/v1/guard/attachments",
 	"DELETE /api/v1/guard/attachments",
+	"GET /api/v1/orgs",
+	"POST /api/v1/orgs",
+	"PATCH /api/v1/orgs/{id}",
+	"DELETE /api/v1/orgs/{id}",
+	"POST /api/v1/teams",
+	"PATCH /api/v1/teams/{id}",
+	"DELETE /api/v1/teams/{id}",
+	"POST /api/v1/projects",
+	"PATCH /api/v1/projects/{id}",
+	"DELETE /api/v1/projects/{id}",
+	"POST /api/v1/keys",
+	"GET /api/v1/keys/{id}",
+	"DELETE /api/v1/keys/{id}",
+	"GET /api/v1/users/{id}",
+	"POST /api/v1/users/{id}/memberships",
+	"DELETE /api/v1/users/{id}/memberships",
 }

 // Client is a connection to one control plane's admin API.
@@ -436,3 +452,166 @@ func (c *Client) Attach(ctx context.Context, target, policy string) (Attachment,
 func (c *Client) Detach(ctx context.Context, target, policy string) error {
 	return c.do(ctx, "DELETE /api/v1/guard/attachments", nil, GuardAttachmentRequest{Target: target, Policy: policy}, nil)
 }
+
+// --- the tenancy tree, keys and memberships ---
+
+// OrgView is an organization with its teams.
+type OrgView struct {
+	ID    string     `json:"id"`
+	Name  string     `json:"name"`
+	Teams []TeamView `json:"teams"`
+}
+
+// TeamView is a team with its projects.
+type TeamView struct {
+	ID       string        `json:"id"`
+	Name     string        `json:"name"`
+	Org      string        `json:"org"`
+	Projects []ProjectView `json:"projects"`
+}
+
+// ProjectView is a project.
+type ProjectView struct {
+	ID   string `json:"id"`
+	Name string `json:"name"`
+	Team string `json:"team"`
+	Path string `json:"path"`
+}
+
+// TenancyCreateRequest adds an organization, a team or a project.
+type TenancyCreateRequest struct {
+	Name string `json:"name"`
+	In   string `json:"in,omitempty"`
+}
+
+// TenancyChangeRequest renames or moves one.
+type TenancyChangeRequest struct {
+	Name string `json:"name,omitempty"`
+	In   string `json:"in,omitempty"`
+}
+
+// TenancyItem is one added or changed, by ID.
+type TenancyItem struct {
+	ID   string `json:"id"`
+	Name string `json:"name,omitempty"`
+	In   string `json:"in,omitempty"`
+}
+
+// Levels of the tree, as the API's paths name them.
+const (
+	LevelOrg     = "orgs"
+	LevelTeam    = "teams"
+	LevelProject = "projects"
+)
+
+// Tree reads every organization, team and project.
+func (c *Client) Tree(ctx context.Context) ([]OrgView, error) {
+	var out []OrgView
+	return out, c.do(ctx, "GET /api/v1/orgs", nil, nil, &out)
+}
+
+// Create adds an organization, team or project.
+func (c *Client) Create(ctx context.Context, level string, in TenancyCreateRequest) (TenancyItem, error) {
+	var out TenancyItem
+	return out, c.do(ctx, "POST /api/v1/"+level, nil, in, &out)
+}
+
+// Change renames an organization, team or project, or moves a team or a
+// project.
+func (c *Client) Change(ctx context.Context, level, id string, in TenancyChangeRequest) (TenancyItem, error) {
+	var out TenancyItem
+	return out, c.do(ctx, "PATCH /api/v1/"+level+"/{id}", []string{id}, in, &out)
+}
+
+// Remove removes an organization, team or project with nothing in it.
+func (c *Client) Remove(ctx context.Context, level, id string) error {
+	return c.do(ctx, "DELETE /api/v1/"+level+"/{id}", []string{id}, nil, nil)
+}
+
+// KeyView is a key, never the key itself.
+type KeyView struct {
+	ID             string    `json:"id"`
+	Name           string    `json:"name"`
+	Prefix         string    `json:"prefix"`
+	Project        string    `json:"project,omitempty"`
+	User           string    `json:"user,omitempty"`
+	Band           string    `json:"band,omitempty"`
+	RoutingProfile string    `json:"routing_profile,omitempty"`
+	State          string    `json:"state"`
+	ExpiresAt      time.Time `json:"expires_at,omitzero"`
+}
+
+// KeyCreateRequest issues a key.
+type KeyCreateRequest struct {
+	Name           string `json:"name"`
+	Project        string `json:"project,omitempty"`
+	User           string `json:"user,omitempty"`
+	Band           string `json:"band,omitempty"`
+	RoutingProfile string `json:"routing_profile,omitempty"`
+	ExpiresIn      string `json:"expires_in,omitempty"`
+}
+
+// KeyCreated is a new key with the key itself, given this once.
+type KeyCreated struct {
+	ID        string    `json:"id"`
+	Prefix    string    `json:"prefix"`
+	Project   string    `json:"project,omitempty"`
+	User      string    `json:"user,omitempty"`
+	State     string    `json:"state"`
+	ExpiresAt time.Time `json:"expires_at,omitzero"`
+	Secret    string    `json:"secret"`
+}
+
+// CreateKey issues a key.
+func (c *Client) CreateKey(ctx context.Context, in KeyCreateRequest) (KeyCreated, error) {
+	var out KeyCreated
+	return out, c.do(ctx, "POST /api/v1/keys", nil, in, &out)
+}
+
+// Key reads a key.
+func (c *Client) Key(ctx context.Context, id string) (KeyView, error) {
+	var out KeyView
+	return out, c.do(ctx, "GET /api/v1/keys/{id}", []string{id}, nil, &out)
+}
+
+// RevokeKey stops a key working.
+func (c *Client) RevokeKey(ctx context.Context, id string) error {
+	return c.do(ctx, "DELETE /api/v1/keys/{id}", []string{id}, nil, nil)
+}
+
+// UserView is a person with where they are a member.
+type UserView struct {
+	ID          string           `json:"id"`
+	Email       string           `json:"email"`
+	Memberships []MembershipView `json:"memberships"`
+}
+
+// MembershipView is one place a person belongs, target as scope:ID.
+type MembershipView struct {
+	Target string `json:"target"`
+	Role   string `json:"role"`
+}
+
+// MembershipRequest names where a person goes or leaves.
+type MembershipRequest struct {
+	Target string `json:"target"`
+	Role   string `json:"role,omitempty"`
+}
+
+// User reads a person, by ID or email.
+func (c *Client) User(ctx context.Context, id string) (UserView, error) {
+	var out UserView
+	return out, c.do(ctx, "GET /api/v1/users/{id}", []string{id}, nil, &out)
+}
+
+// Join puts a person in an organization, team or project, or changes their
+// role there.
+func (c *Client) Join(ctx context.Context, user string, in MembershipRequest) (UserView, error) {
+	var out UserView
+	return out, c.do(ctx, "POST /api/v1/users/{id}/memberships", []string{user}, in, &out)
+}
+
+// Leave takes a person out.
+func (c *Client) Leave(ctx context.Context, user, target string) error {
+	return c.do(ctx, "DELETE /api/v1/users/{id}/memberships", []string{user}, MembershipRequest{Target: target}, nil)
+}
@@ -101,17 +101,32 @@ var bodies = []struct {
 	{"GET /api/v1/guard", nil, "", GuardView{}, "GuardView"},
 	{"POST /api/v1/guard/attachments", GuardAttachmentRequest{}, "GuardAttachmentRequest", Attachment{}, "guard.Attachment"},
 	{"DELETE /api/v1/guard/attachments", GuardAttachmentRequest{}, "GuardAttachmentRequest", nil, ""},
+	{"GET /api/v1/orgs", nil, "", OrgView{}, "OrgView"},
+	{"POST /api/v1/orgs", TenancyCreateRequest{}, "TenancyCreateRequest", TenancyItem{}, "TenancyItem"},
+	{"PATCH /api/v1/orgs/{id}", TenancyChangeRequest{}, "TenancyChangeRequest", TenancyItem{}, "TenancyItem"},
+	{"POST /api/v1/teams", TenancyCreateRequest{}, "TenancyCreateRequest", TenancyItem{}, "TenancyItem"},
+	{"PATCH /api/v1/teams/{id}", TenancyChangeRequest{}, "TenancyChangeRequest", TenancyItem{}, "TenancyItem"},
+	{"POST /api/v1/projects", TenancyCreateRequest{}, "TenancyCreateRequest", TenancyItem{}, "TenancyItem"},
+	{"PATCH /api/v1/projects/{id}", TenancyChangeRequest{}, "TenancyChangeRequest", TenancyItem{}, "TenancyItem"},
+	{"POST /api/v1/keys", KeyCreateRequest{}, "KeyCreateRequest", KeyCreated{}, "KeyCreated"},
+	{"GET /api/v1/keys/{id}", nil, "", KeyView{}, "KeyView"},
+	{"GET /api/v1/users/{id}", nil, "", UserView{}, "UserView"},
+	{"POST /api/v1/users/{id}/memberships", MembershipRequest{}, "MembershipRequest", UserView{}, "UserView"},
+	{"DELETE /api/v1/users/{id}/memberships", MembershipRequest{}, "MembershipRequest", UserView{}, "UserView"},
 }

 // nested are the client's types inside those bodies, and their schemas.
 var nested = map[reflect.Type]string{
-	reflect.TypeOf(Profile{}):      "catalog.Profile",
-	reflect.TypeOf(Scaling{}):      "catalog.Scaling",
-	reflect.TypeOf(External{}):     "catalog.External",
-	reflect.TypeOf(Adapter{}):      "catalog.Adapter",
-	reflect.TypeOf(Capabilities{}): "catalog.Capabilities",
-	reflect.TypeOf(Resources{}):    "store.Resources",
-	reflect.TypeOf(Attachment{}):   "GuardAttachmentView",
+	reflect.TypeOf(Profile{}):        "catalog.Profile",
+	reflect.TypeOf(Scaling{}):        "catalog.Scaling",
+	reflect.TypeOf(External{}):       "catalog.External",
+	reflect.TypeOf(Adapter{}):        "catalog.Adapter",
+	reflect.TypeOf(Capabilities{}):   "catalog.Capabilities",
+	reflect.TypeOf(Resources{}):      "store.Resources",
+	reflect.TypeOf(Attachment{}):     "GuardAttachmentView",
+	reflect.TypeOf(TeamView{}):       "TeamView",
+	reflect.TypeOf(ProjectView{}):    "ProjectView",
+	reflect.TypeOf(MembershipView{}): "MembershipView",
 }

 func TestEveryBodyTheClientSendsAndReadsMatchesTheDocument(t *testing.T) {
@@ -32,24 +32,33 @@ type API struct {
 	attachments []client.Attachment
 	nodes       []client.NodeView
 	actors      []string
-	// scopes maps scope:ID to the path people know it by.
+	// scopes maps scope:ID to the path people know it by, rebuilt from
+	// the tree whenever it changes.
 	scopes map[string]string
+	tenancy
 }

 // New builds a fake with a small tenancy tree — Acme, Acme/ml,
-// Acme/ml/training — two guard policies and the nodes given.
+// Acme/ml/training — one person, ayse@example.com (u1), two guard policies
+// and the nodes given.
 func New(token string, nodes ...client.NodeView) *API {
-	return &API{
+	a := &API{
 		Token:  token,
 		models: map[string]client.Model{},
 		quotas: map[string]client.QuotaView{},
 		nodes:  nodes,
-		scopes: map[string]string{
-			"org:o1":     "Acme",
-			"team:t1":    "Acme / ml",
-			"project:p1": "Acme/ml/training",
+		tenancy: tenancy{
+			levels: map[string][]node{
+				client.LevelOrg:     {{ID: "o1", Name: "Acme"}},
+				client.LevelTeam:    {{ID: "t1", Name: "ml", In: "o1"}},
+				client.LevelProject: {{ID: "p1", Name: "training", In: "t1"}},
+			},
+			keys:  map[string]client.KeyView{},
+			users: map[string]*client.UserView{"u1": {ID: "u1", Email: "ayse@example.com", Memberships: []client.MembershipView{}}},
 		},
 	}
+	a.rebuild()
+	return a
 }

 // Actors are who each change said it was made by, in order.
@@ -153,6 +162,7 @@ func (a *API) mux() *http.ServeMux {
 	mux.HandleFunc("GET /api/v1/guard", a.guard)
 	mux.HandleFunc("POST /api/v1/guard/attachments", a.attach(true))
 	mux.HandleFunc("DELETE /api/v1/guard/attachments", a.attach(false))
+	a.tenancyRoutes(mux)
 	return mux
 }

@@ -0,0 +1,393 @@
+package fakeapi
+
+import (
+	"cmp"
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"slices"
+	"strings"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/terraform-provider-llmgw/internal/client"
+)
+
+// node is an organization, team or project; In is its parent's ID.
+type node struct{ ID, Name, In string }
+
+// tenancy is the tree, the keys and the people.
+type tenancy struct {
+	levels map[string][]node
+	keys   map[string]client.KeyView
+	users  map[string]*client.UserView
+	seq    int
+}
+
+// parent is the level a level's nodes are in, and the scope it is named by.
+var parent = map[string]string{client.LevelTeam: client.LevelOrg, client.LevelProject: client.LevelTeam}
+
+var scopeOf = map[string]string{client.LevelOrg: "org", client.LevelTeam: "team", client.LevelProject: "project"}
+
+func (a *API) find(level, id string) (node, int) {
+	for i, n := range a.levels[level] {
+		if n.ID == id {
+			return n, i
+		}
+	}
+	return node{}, -1
+}
+
+// rebuild names every part of the tree as the real API does.
+func (a *API) rebuild() {
+	a.scopes = map[string]string{}
+	for _, o := range a.levels[client.LevelOrg] {
+		a.scopes["org:"+o.ID] = o.Name
+	}
+	for _, t := range a.levels[client.LevelTeam] {
+		o, _ := a.find(client.LevelOrg, t.In)
+		a.scopes["team:"+t.ID] = o.Name + " / " + t.Name
+	}
+	for _, p := range a.levels[client.LevelProject] {
+		t, _ := a.find(client.LevelTeam, p.In)
+		o, _ := a.find(client.LevelOrg, t.In)
+		a.scopes["project:"+p.ID] = o.Name + "/" + t.Name + "/" + p.Name
+	}
+}
+
+// resolveAt reads a level's node by ID or path.
+func (a *API) resolveAt(level, ref string) (string, error) {
+	key, err := a.resolve(scopeOf[level] + ":" + ref)
+	if err != nil {
+		return "", err
+	}
+	_, id, _ := strings.Cut(key, ":")
+	return id, nil
+}
+
+// RenameBehindTerraform renames a part of the tree as someone at a terminal
+// would.
+func (a *API) RenameBehindTerraform(level, id, name string) {
+	a.mu.Lock()
+	defer a.mu.Unlock()
+	if _, i := a.find(level, id); i >= 0 {
+		a.levels[level][i].Name = name
+		a.rebuild()
+	}
+}
+
+// RevokeBehindTerraform revokes a key as someone at a terminal would.
+func (a *API) RevokeBehindTerraform(id string) {
+	a.mu.Lock()
+	defer a.mu.Unlock()
+	k := a.keys[id]
+	k.State = "revoked"
+	a.keys[id] = k
+}
+
+// Keys are the keys as the fake holds them.
+func (a *API) Keys() map[string]client.KeyView {
+	a.mu.Lock()
+	defer a.mu.Unlock()
+	out := map[string]client.KeyView{}
+	for id, k := range a.keys {
+		out[id] = k
+	}
+	return out
+}
+
+// Memberships are where a person is, as the fake holds it.
+func (a *API) Memberships(user string) []client.MembershipView {
+	a.mu.Lock()
+	defer a.mu.Unlock()
+	return slices.Clone(a.users[user].Memberships)
+}
+
+// Has says whether a part of the tree is there, by ID.
+func (a *API) Has(level, id string) bool {
+	a.mu.Lock()
+	defer a.mu.Unlock()
+	_, i := a.find(level, id)
+	return i >= 0
+}
+
+func (a *API) tenancyRoutes(mux *http.ServeMux) {
+	mux.HandleFunc("GET /api/v1/orgs", a.tree)
+	for _, level := range []string{client.LevelOrg, client.LevelTeam, client.LevelProject} {
+		mux.HandleFunc("POST /api/v1/"+level, a.create(level))
+		mux.HandleFunc("PATCH /api/v1/"+level+"/{id}", a.changeNode(level))
+		mux.HandleFunc("DELETE /api/v1/"+level+"/{id}", a.removeNode(level))
+	}
+	mux.HandleFunc("POST /api/v1/keys", a.createKey)
+	mux.HandleFunc("GET /api/v1/keys/{id}", a.getKey)
+	mux.HandleFunc("DELETE /api/v1/keys/{id}", a.revokeKey)
+	mux.HandleFunc("GET /api/v1/users/{id}", a.getUser)
+	mux.HandleFunc("POST /api/v1/users/{id}/memberships", a.membership(true))
+	mux.HandleFunc("DELETE /api/v1/users/{id}/memberships", a.membership(false))
+}
+
+func (a *API) tree(w http.ResponseWriter, _ *http.Request) {
+	a.mu.Lock()
+	defer a.mu.Unlock()
+	out := []client.OrgView{}
+	for _, o := range a.levels[client.LevelOrg] {
+		ov := client.OrgView{ID: o.ID, Name: o.Name, Teams: []client.TeamView{}}
+		for _, t := range a.levels[client.LevelTeam] {
+			if t.In != o.ID {
+				continue
+			}
+			tv := client.TeamView{ID: t.ID, Name: t.Name, Org: o.ID, Projects: []client.ProjectView{}}
+			for _, p := range a.levels[client.LevelProject] {
+				if p.In == t.ID {
+					tv.Projects = append(tv.Projects, client.ProjectView{ID: p.ID, Name: p.Name, Team: t.ID,
+						Path: a.scopes["project:"+p.ID]})
+				}
+			}
+			ov.Teams = append(ov.Teams, tv)
+		}
+		out = append(out, ov)
+	}
+	writeJSON(w, http.StatusOK, out)
+}
+
+func (a *API) taken(level, in, name, except string) bool {
+	for _, n := range a.levels[level] {
+		if n.In == in && n.Name == name && n.ID != except {
+			return true
+		}
+	}
+	return false
+}
+
+func (a *API) create(level string) http.HandlerFunc {
+	return func(w http.ResponseWriter, r *http.Request) {
+		var req client.TenancyCreateRequest
+		if err := json.NewDecoder(r.Body).Decode(&req); err != nil || req.Name == "" {
+			writeError(w, http.StatusBadRequest, "a name is needed")
+			return
+		}
+		a.mu.Lock()
+		defer a.mu.Unlock()
+		n := node{Name: req.Name}
+		if p, ok := parent[level]; ok {
+			id, err := a.resolveAt(p, req.In)
+			if err != nil {
+				writeError(w, http.StatusNotFound, err.Error())
+				return
+			}
+			n.In = id
+		}
+		if a.taken(level, n.In, n.Name, "") {
+			writeError(w, http.StatusConflict, "there is already one named "+n.Name+" there")
+			return
+		}
+		a.seq++
+		n.ID = fmt.Sprintf("%s-%d", scopeOf[level], a.seq)
+		a.levels[level] = append(a.levels[level], n)
+		a.rebuild()
+		writeJSON(w, http.StatusCreated, client.TenancyItem{ID: n.ID, Name: n.Name, In: n.In})
+	}
+}
+
+func (a *API) changeNode(level string) http.HandlerFunc {
+	return func(w http.ResponseWriter, r *http.Request) {
+		var req client.TenancyChangeRequest
+		if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
+			writeError(w, http.StatusBadRequest, err.Error())
+			return
+		}
+		a.mu.Lock()
+		defer a.mu.Unlock()
+		id, err := a.resolveAt(level, r.PathValue("id"))
+		if err != nil {
+			writeError(w, http.StatusNotFound, err.Error())
+			return
+		}
+		n, i := a.find(level, id)
+		out := client.TenancyItem{ID: id}
+		if req.In != "" {
+			p, ok := parent[level]
+			if !ok {
+				writeError(w, http.StatusBadRequest, "an organization is in nothing")
+				return
+			}
+			if n.In, err = a.resolveAt(p, req.In); err != nil {
+				writeError(w, http.StatusNotFound, err.Error())
+				return
+			}
+			out.In = n.In
+		}
+		if req.Name != "" {
+			n.Name, out.Name = req.Name, req.Name
+		}
+		if a.taken(level, n.In, n.Name, id) {
+			writeError(w, http.StatusConflict, "there is already one named "+n.Name+" there")
+			return
+		}
+		a.levels[level][i] = n
+		a.rebuild()
+		writeJSON(w, http.StatusOK, out)
+	}
+}
+
+func (a *API) removeNode(level string) http.HandlerFunc {
+	return func(w http.ResponseWriter, r *http.Request) {
+		a.mu.Lock()
+		defer a.mu.Unlock()
+		id, err := a.resolveAt(level, r.PathValue("id"))
+		if err != nil {
+			writeError(w, http.StatusNotFound, err.Error())
+			return
+		}
+		for child, p := range parent {
+			if p != level {
+				continue
+			}
+			for _, c := range a.levels[child] {
+				if c.In == id {
+					writeError(w, http.StatusConflict, "not empty")
+					return
+				}
+			}
+		}
+		for _, k := range a.keys {
+			if k.Project == id {
+				writeError(w, http.StatusConflict, "not empty: it still has a key")
+				return
+			}
+		}
+		_, i := a.find(level, id)
+		a.levels[level] = slices.Delete(a.levels[level], i, i+1)
+		a.rebuild()
+		writeJSON(w, http.StatusOK, map[string]any{"id": id, "deleted": true})
+	}
+}
+
+func (a *API) createKey(w http.ResponseWriter, r *http.Request) {
+	var req client.KeyCreateRequest
+	if err := json.NewDecoder(r.Body).Decode(&req); err != nil || req.Name == "" {
+		writeError(w, http.StatusBadRequest, "a key needs a name")
+		return
+	}
+	a.mu.Lock()
+	defer a.mu.Unlock()
+	k := client.KeyView{Name: req.Name, Band: req.Band, RoutingProfile: req.RoutingProfile, State: "active"}
+	if req.Project != "" {
+		id, err := a.resolveAt(client.LevelProject, req.Project)
+		if err != nil {
+			writeError(w, http.StatusNotFound, err.Error())
+			return
+		}
+		k.Project = id
+	}
+	if req.User != "" {
+		u, ok := a.user(req.User)
+		if !ok {
+			writeError(w, http.StatusNotFound, "no person "+req.User)
+			return
+		}
+		if k.Project != "" && !slices.ContainsFunc(u.Memberships, func(m client.MembershipView) bool {
+			return m.Target == "project:"+k.Project
+		}) {
+			writeError(w, http.StatusConflict, u.Email+" is not a member there")
+			return
+		}
+		k.User = u.ID
+	}
+	if req.ExpiresIn != "" {
+		d, err := time.ParseDuration(req.ExpiresIn)
+		if err != nil || d <= 0 {
+			writeError(w, http.StatusBadRequest, "expires_in is not a duration")
+			return
+		}
+		k.ExpiresAt = time.Now().Add(d).UTC().Truncate(time.Second)
+	}
+	a.seq++
+	k.ID = fmt.Sprintf("key-%d", a.seq)
+	secret := fmt.Sprintf("sk-llmgw-fake%08d", a.seq)
+	k.Prefix = secret[:12]
+	a.keys[k.ID] = k
+	writeJSON(w, http.StatusCreated, client.KeyCreated{ID: k.ID, Prefix: k.Prefix, Project: k.Project, User: k.User,
+		State: k.State, ExpiresAt: k.ExpiresAt, Secret: secret})
+}
+
+func (a *API) getKey(w http.ResponseWriter, r *http.Request) {
+	a.mu.Lock()
+	defer a.mu.Unlock()
+	k, ok := a.keys[r.PathValue("id")]
+	if !ok {
+		writeError(w, http.StatusNotFound, "no key "+r.PathValue("id"))
+		return
+	}
+	writeJSON(w, http.StatusOK, k)
+}
+
+func (a *API) revokeKey(w http.ResponseWriter, r *http.Request) {
+	a.mu.Lock()
+	defer a.mu.Unlock()
+	k, ok := a.keys[r.PathValue("id")]
+	if !ok {
+		writeError(w, http.StatusNotFound, "no key "+r.PathValue("id"))
+		return
+	}
+	k.State = "revoked"
+	a.keys[k.ID] = k
+	writeJSON(w, http.StatusOK, map[string]any{"id": k.ID, "revoked": true})
+}
+
+func (a *API) user(ref string) (*client.UserView, bool) {
+	if u, ok := a.users[ref]; ok {
+		return u, true
+	}
+	for _, u := range a.users {
+		if strings.EqualFold(u.Email, ref) {
+			return u, true
+		}
+	}
+	return nil, false
+}
+
+func (a *API) getUser(w http.ResponseWriter, r *http.Request) {
+	a.mu.Lock()
+	defer a.mu.Unlock()
+	u, ok := a.user(r.PathValue("id"))
+	if !ok {
+		writeError(w, http.StatusNotFound, "no person "+r.PathValue("id"))
+		return
+	}
+	writeJSON(w, http.StatusOK, u)
+}
+
+func (a *API) membership(join bool) http.HandlerFunc {
+	return func(w http.ResponseWriter, r *http.Request) {
+		var req client.MembershipRequest
+		if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
+			writeError(w, http.StatusBadRequest, err.Error())
+			return
+		}
+		a.mu.Lock()
+		defer a.mu.Unlock()
+		u, ok := a.user(r.PathValue("id"))
+		if !ok {
+			writeError(w, http.StatusNotFound, "no person "+r.PathValue("id"))
+			return
+		}
+		key, err := a.resolve(req.Target)
+		if err != nil {
+			writeError(w, http.StatusBadRequest, err.Error())
+			return
+		}
+		i := slices.IndexFunc(u.Memberships, func(m client.MembershipView) bool { return m.Target == key })
+		switch {
+		case join && i >= 0:
+			u.Memberships[i].Role = cmp.Or(req.Role, "member")
+		case join:
+			u.Memberships = append(u.Memberships, client.MembershipView{Target: key, Role: cmp.Or(req.Role, "member")})
+		case i < 0:
+			writeError(w, http.StatusNotFound, u.Email+" is not a member there")
+			return
+		default:
+			u.Memberships = slices.Delete(u.Memberships, i, i+1)
+		}
+		writeJSON(w, http.StatusOK, u)
+	}
+}
@@ -104,7 +104,8 @@ func (p *llmgwProvider) Configure(ctx context.Context, req provider.ConfigureReq
 }

 func (p *llmgwProvider) Resources(context.Context) []func() resource.Resource {
-	return []func() resource.Resource{newModelResource, newQuotaResource, newGuardAttachmentResource}
+	return []func() resource.Resource{newModelResource, newQuotaResource, newGuardAttachmentResource,
+		newOrgResource, newTeamResource, newProjectResource, newKeyResource, newMembershipResource}
 }

 func (p *llmgwProvider) DataSources(context.Context) []func() datasource.DataSource {
@@ -0,0 +1,532 @@
+package provider
+
+import (
+	"context"
+	"fmt"
+	"strings"
+
+	"github.com/hashicorp/terraform-plugin-framework/path"
+	"github.com/hashicorp/terraform-plugin-framework/resource"
+	"github.com/hashicorp/terraform-plugin-framework/resource/schema"
+	"github.com/hashicorp/terraform-plugin-framework/resource/schema/planmodifier"
+	"github.com/hashicorp/terraform-plugin-framework/resource/schema/stringplanmodifier"
+	"github.com/hashicorp/terraform-plugin-framework/tfsdk"
+	"github.com/hashicorp/terraform-plugin-framework/types"
+
+	"gitea.mynodes.xyz/ilker/llmgw/terraform-provider-llmgw/internal/client"
+)
+
+// Organizations, teams and projects: one resource each, written once. A
+// team names the organization it is in and a project the team, by path or
+// ID; changing that moves it, with everything under it, rather than
+// replacing it, as `llmgw org move` does.
+
+type tenancyResource struct {
+	c *client.Client
+	// level is the API's path for it; kind the resource's name; parent the
+	// attribute naming what it is in, empty for an organization.
+	level, kind, parent string
+}
+
+func newOrgResource() resource.Resource {
+	return &tenancyResource{level: client.LevelOrg, kind: "org"}
+}
+
+func newTeamResource() resource.Resource {
+	return &tenancyResource{level: client.LevelTeam, kind: "team", parent: "org"}
+}
+
+func newProjectResource() resource.Resource {
+	return &tenancyResource{level: client.LevelProject, kind: "project", parent: "team"}
+}
+
+func (r *tenancyResource) Metadata(_ context.Context, req resource.MetadataRequest, resp *resource.MetadataResponse) {
+	resp.TypeName = req.ProviderTypeName + "_" + r.kind
+}
+
+func (r *tenancyResource) Schema(_ context.Context, _ resource.SchemaRequest, resp *resource.SchemaResponse) {
+	keep := []planmodifier.String{stringplanmodifier.UseStateForUnknown()}
+	attrs := map[string]schema.Attribute{
+		"id": schema.StringAttribute{Computed: true, PlanModifiers: keep,
+			Description: "The " + r.kind + "'s ID. Import by it."},
+		"name": schema.StringAttribute{Required: true,
+			Description: "Its name; changing it renames it, keeping its ID and everything under it."},
+	}
+	desc := "An organization: the top of the tenancy tree. Removed only when it has no teams."
+	switch r.kind {
+	case "team":
+		desc = "A team in an organization. Removed only when it has no projects."
+		attrs["org"] = schema.StringAttribute{Required: true,
+			Description: "The organization it is in, by name or ID; changing it moves the team and its projects."}
+		attrs["org_id"] = schema.StringAttribute{Computed: true, Description: "The organization's ID."}
+	case "project":
+		desc = "A project in a team: where keys, people and spending meet. Removed only when it has no people and no collections, " +
+			"and has never had a key: a revoked key stays, because usage refers to it. Rename such a project rather than destroying it."
+		attrs["team"] = schema.StringAttribute{Required: true,
+			Description: "The team it is in, as ORG/TEAM or its ID; changing it moves the project."}
+		attrs["team_id"] = schema.StringAttribute{Computed: true, Description: "The team's ID."}
+		attrs["path"] = schema.StringAttribute{Computed: true, Description: "ORG/TEAM/PROJECT, by name."}
+	}
+	resp.Schema = schema.Schema{Description: desc, Attributes: attrs}
+}
+
+func (r *tenancyResource) Configure(_ context.Context, req resource.ConfigureRequest, _ *resource.ConfigureResponse) {
+	if c, ok := clientFrom(req.ProviderData); ok {
+		r.c = c
+	}
+}
+
+// found is a node as the tree has it.
+type found struct{ name, in, path string }
+
+// lookup finds a node of this resource's level in the tree by ID.
+func (r *tenancyResource) lookup(ctx context.Context, id string) (found, bool, error) {
+	tree, err := r.c.Tree(ctx)
+	if err != nil {
+		return found{}, false, err
+	}
+	for _, o := range tree {
+		if r.level == client.LevelOrg && o.ID == id {
+			return found{name: o.Name}, true, nil
+		}
+		for _, t := range o.Teams {
+			if r.level == client.LevelTeam && t.ID == id {
+				return found{name: t.Name, in: o.ID}, true, nil
+			}
+			for _, p := range t.Projects {
+				if r.level == client.LevelProject && p.ID == id {
+					return found{name: p.Name, in: t.ID, path: p.Path}, true, nil
+				}
+			}
+		}
+	}
+	return found{}, false, nil
+}
+
+// set writes what the API says into the state: the computed attributes,
+// and, when the API has moved on from what the state said, the configured
+// ones too, so the next plan shows the difference.
+func (r *tenancyResource) set(ctx context.Context, st *tfsdk.State, id string, f found, fromAPI bool) {
+	st.SetAttribute(ctx, path.Root("id"), types.StringValue(id))
+	if fromAPI {
+		st.SetAttribute(ctx, path.Root("name"), types.StringValue(f.name))
+	}
+	if r.parent == "" {
+		return
+	}
+	var was types.String
+	st.GetAttribute(ctx, path.Root(r.parent+"_id"), &was)
+	if fromAPI && was.ValueString() != f.in {
+		st.SetAttribute(ctx, path.Root(r.parent), types.StringValue(f.in))
+	}
+	st.SetAttribute(ctx, path.Root(r.parent+"_id"), types.StringValue(f.in))
+	if r.level == client.LevelProject {
+		st.SetAttribute(ctx, path.Root("path"), types.StringValue(f.path))
+	}
+}
+
+func (r *tenancyResource) Create(ctx context.Context, req resource.CreateRequest, resp *resource.CreateResponse) {
+	var name, in types.String
+	resp.Diagnostics.Append(req.Plan.GetAttribute(ctx, path.Root("name"), &name)...)
+	if r.parent != "" {
+		resp.Diagnostics.Append(req.Plan.GetAttribute(ctx, path.Root(r.parent), &in)...)
+	}
+	if resp.Diagnostics.HasError() {
+		return
+	}
+	item, err := r.c.Create(ctx, r.level, client.TenancyCreateRequest{Name: name.ValueString(), In: in.ValueString()})
+	if err != nil {
+		resp.Diagnostics.AddError("Adding "+r.kind+" "+name.ValueString(), err.Error())
+		return
+	}
+	f, ok, err := r.lookup(ctx, item.ID)
+	if err != nil || !ok {
+		resp.Diagnostics.AddError("Reading back "+r.kind+" "+name.ValueString(), fmt.Sprint("not in the tree after adding it ", err))
+		return
+	}
+	resp.State.Raw = req.Plan.Raw
+	r.set(ctx, &resp.State, item.ID, f, false)
+}
+
+func (r *tenancyResource) Read(ctx context.Context, req resource.ReadRequest, resp *resource.ReadResponse) {
+	var id types.String
+	resp.Diagnostics.Append(req.State.GetAttribute(ctx, path.Root("id"), &id)...)
+	if resp.Diagnostics.HasError() {
+		return
+	}
+	f, ok, err := r.lookup(ctx, id.ValueString())
+	if err != nil {
+		resp.Diagnostics.AddError("Reading the tenancy tree", err.Error())
+		return
+	}
+	if !ok {
+		resp.State.RemoveResource(ctx)
+		return
+	}
+	r.set(ctx, &resp.State, id.ValueString(), f, true)
+}
+
+func (r *tenancyResource) Update(ctx context.Context, req resource.UpdateRequest, resp *resource.UpdateResponse) {
+	var id, name, wasName, in, wasIn types.String
+	resp.Diagnostics.Append(req.State.GetAttribute(ctx, path.Root("id"), &id)...)
+	resp.Diagnostics.Append(req.Plan.GetAttribute(ctx, path.Root("name"), &name)...)
+	resp.Diagnostics.Append(req.State.GetAttribute(ctx, path.Root("name"), &wasName)...)
+	if r.parent != "" {
+		resp.Diagnostics.Append(req.Plan.GetAttribute(ctx, path.Root(r.parent), &in)...)
+		resp.Diagnostics.Append(req.State.GetAttribute(ctx, path.Root(r.parent), &wasIn)...)
+	}
+	if resp.Diagnostics.HasError() {
+		return
+	}
+	var change client.TenancyChangeRequest
+	if !name.Equal(wasName) {
+		change.Name = name.ValueString()
+	}
+	if r.parent != "" && !in.Equal(wasIn) {
+		change.In = in.ValueString()
+	}
+	if change.Name != "" || change.In != "" {
+		if _, err := r.c.Change(ctx, r.level, id.ValueString(), change); err != nil {
+			resp.Diagnostics.AddError("Changing "+r.kind+" "+wasName.ValueString(), err.Error())
+			return
+		}
+	}
+	f, ok, err := r.lookup(ctx, id.ValueString())
+	if err != nil || !ok {
+		resp.Diagnostics.AddError("Reading back "+r.kind+" "+name.ValueString(), fmt.Sprint("not in the tree after changing it ", err))
+		return
+	}
+	resp.State.Raw = req.Plan.Raw
+	r.set(ctx, &resp.State, id.ValueString(), f, false)
+}
+
+func (r *tenancyResource) Delete(ctx context.Context, req resource.DeleteRequest, resp *resource.DeleteResponse) {
+	var id, name types.String
+	resp.Diagnostics.Append(req.State.GetAttribute(ctx, path.Root("id"), &id)...)
+	resp.Diagnostics.Append(req.State.GetAttribute(ctx, path.Root("name"), &name)...)
+	if resp.Diagnostics.HasError() {
+		return
+	}
+	if err := r.c.Remove(ctx, r.level, id.ValueString()); err != nil && !client.IsNotFound(err) {
+		resp.Diagnostics.AddError("Removing "+r.kind+" "+name.ValueString(), err.Error())
+	}
+}
+
+func (r *tenancyResource) ImportState(ctx context.Context, req resource.ImportStateRequest, resp *resource.ImportStateResponse) {
+	resource.ImportStatePassthroughID(ctx, path.Root("id"), req, resp)
+}
+
+// --- keys ---
+
+// The key resource is an API key. Its secret is known only to the apply
+// that made it, and kept in the state, marked sensitive: the gateway keeps
+// only a hash. A key revoked or expired outside Terraform is made again.
+
+type keyResource struct{ c *client.Client }
+
+func newKeyResource() resource.Resource { return &keyResource{} }
+
+type keyModel struct {
+	ID             types.String `tfsdk:"id"`
+	Name           types.String `tfsdk:"name"`
+	Project        types.String `tfsdk:"project"`
+	User           types.String `tfsdk:"user"`
+	Band           types.String `tfsdk:"band"`
+	RoutingProfile types.String `tfsdk:"routing_profile"`
+	ExpiresIn      types.String `tfsdk:"expires_in"`
+	Secret         types.String `tfsdk:"secret"`
+	Prefix         types.String `tfsdk:"prefix"`
+	ExpiresAt      types.String `tfsdk:"expires_at"`
+	ProjectID      types.String `tfsdk:"project_id"`
+	UserID         types.String `tfsdk:"user_id"`
+}
+
+func (r *keyResource) Metadata(_ context.Context, req resource.MetadataRequest, resp *resource.MetadataResponse) {
+	resp.TypeName = req.ProviderTypeName + "_key"
+}
+
+func (r *keyResource) Schema(_ context.Context, _ resource.SchemaRequest, resp *resource.SchemaResponse) {
+	replace := []planmodifier.String{stringplanmodifier.RequiresReplace()}
+	keep := []planmodifier.String{stringplanmodifier.UseStateForUnknown()}
+	resp.Schema = schema.Schema{
+		Description: "An API key: a project's, a person's in a project they are a member of, or a person's own. " +
+			"Every change makes a new key; one revoked or expired outside Terraform is made again on the next apply.",
+		Attributes: map[string]schema.Attribute{
+			"id": schema.StringAttribute{Computed: true, PlanModifiers: keep,
+				Description: "The key's ID, not the key. Import by it; an imported key has no secret."},
+			"name": schema.StringAttribute{Required: true, PlanModifiers: replace},
+			"project": schema.StringAttribute{Optional: true, PlanModifiers: replace,
+				Description: "The project it belongs to, as ORG/TEAM/PROJECT or its ID."},
+			"user": schema.StringAttribute{Optional: true, PlanModifiers: replace,
+				Description: "The person it belongs to, by ID or email; with a project, they must be a member of it."},
+			"band": schema.StringAttribute{Optional: true, PlanModifiers: replace,
+				Description: "Admission band; inherited when unset."},
+			"routing_profile": schema.StringAttribute{Optional: true, PlanModifiers: replace,
+				Description: "Routing profile; inherited when unset."},
+			"expires_in": schema.StringAttribute{Optional: true, PlanModifiers: replace,
+				Description: "Lifetime, such as 720h; never, unless a scope above it limits how long keys live."},
+			"secret": schema.StringAttribute{Computed: true, Sensitive: true, PlanModifiers: keep,
+				Description: "The key itself."},
+			"prefix":     schema.StringAttribute{Computed: true, PlanModifiers: keep, Description: "Its first characters, as listings show it."},
+			"expires_at": schema.StringAttribute{Computed: true, PlanModifiers: keep, Description: "When it stops working, RFC 3339; empty for never."},
+			"project_id": schema.StringAttribute{Computed: true, PlanModifiers: keep},
+			"user_id":    schema.StringAttribute{Computed: true, PlanModifiers: keep},
+		},
+	}
+}
+
+func (r *keyResource) Configure(_ context.Context, req resource.ConfigureRequest, _ *resource.ConfigureResponse) {
+	if c, ok := clientFrom(req.ProviderData); ok {
+		r.c = c
+	}
+}
+
+func (r *keyResource) Create(ctx context.Context, req resource.CreateRequest, resp *resource.CreateResponse) {
+	var plan keyModel
+	resp.Diagnostics.Append(req.Plan.Get(ctx, &plan)...)
+	if resp.Diagnostics.HasError() {
+		return
+	}
+	k, err := r.c.CreateKey(ctx, client.KeyCreateRequest{Name: plan.Name.ValueString(), Project: plan.Project.ValueString(),
+		User: plan.User.ValueString(), Band: plan.Band.ValueString(), RoutingProfile: plan.RoutingProfile.ValueString(),
+		ExpiresIn: plan.ExpiresIn.ValueString()})
+	if err != nil {
+		resp.Diagnostics.AddError("Issuing key "+plan.Name.ValueString(), err.Error())
+		return
+	}
+	plan.ID, plan.Secret, plan.Prefix = types.StringValue(k.ID), types.StringValue(k.Secret), types.StringValue(k.Prefix)
+	plan.ProjectID, plan.UserID = types.StringValue(k.Project), types.StringValue(k.User)
+	plan.ExpiresAt = types.StringValue("")
+	if !k.ExpiresAt.IsZero() {
+		plan.ExpiresAt = types.StringValue(k.ExpiresAt.Format("2006-01-02T15:04:05Z07:00"))
+	}
+	resp.Diagnostics.Append(resp.State.Set(ctx, &plan)...)
+}
+
+func (r *keyResource) Read(ctx context.Context, req resource.ReadRequest, resp *resource.ReadResponse) {
+	var st keyModel
+	resp.Diagnostics.Append(req.State.Get(ctx, &st)...)
+	if resp.Diagnostics.HasError() {
+		return
+	}
+	k, err := r.c.Key(ctx, st.ID.ValueString())
+	if client.IsNotFound(err) || err == nil && k.State != "active" {
+		resp.State.RemoveResource(ctx)
+		return
+	}
+	if err != nil {
+		resp.Diagnostics.AddError("Reading key "+st.ID.ValueString(), err.Error())
+		return
+	}
+	if st.Name.IsNull() {
+		// Imported: what it was made with comes from the API.
+		st.Name, st.Project, st.User = types.StringValue(k.Name), strOrNull(k.Project), strOrNull(k.User)
+		st.Band, st.RoutingProfile = strOrNull(k.Band), strOrNull(k.RoutingProfile)
+		st.Prefix, st.ProjectID, st.UserID = types.StringValue(k.Prefix), types.StringValue(k.Project), types.StringValue(k.User)
+		st.ExpiresAt = types.StringValue("")
+		if !k.ExpiresAt.IsZero() {
+			st.ExpiresAt = types.StringValue(k.ExpiresAt.Format("2006-01-02T15:04:05Z07:00"))
+		}
+	}
+	resp.Diagnostics.Append(resp.State.Set(ctx, &st)...)
+}
+
+// Update is never called: every attribute replaces the key.
+func (r *keyResource) Update(ctx context.Context, req resource.UpdateRequest, resp *resource.UpdateResponse) {
+	var plan keyModel
+	resp.Diagnostics.Append(req.Plan.Get(ctx, &plan)...)
+	resp.Diagnostics.Append(resp.State.Set(ctx, &plan)...)
+}
+
+func (r *keyResource) Delete(ctx context.Context, req resource.DeleteRequest, resp *resource.DeleteResponse) {
+	var st keyModel
+	resp.Diagnostics.Append(req.State.Get(ctx, &st)...)
+	if resp.Diagnostics.HasError() {
+		return
+	}
+	if err := r.c.RevokeKey(ctx, st.ID.ValueString()); err != nil && !client.IsNotFound(err) {
+		resp.Diagnostics.AddError("Revoking key "+st.Name.ValueString(), err.Error())
+	}
+}
+
+func (r *keyResource) ImportState(ctx context.Context, req resource.ImportStateRequest, resp *resource.ImportStateResponse) {
+	resource.ImportStatePassthroughID(ctx, path.Root("id"), req, resp)
+}
+
+// --- memberships ---
+
+// The membership resource puts a person in an organization, team or
+// project. People themselves are not made here: they sign up, are invited
+// or come from the identity provider.
+
+type membershipResource struct{ c *client.Client }
+
+func newMembershipResource() resource.Resource { return &membershipResource{} }
+
+type membershipModel struct {
+	ID     types.String `tfsdk:"id"`
+	User   types.String `tfsdk:"user"`
+	Target types.String `tfsdk:"target"`
+	Role   types.String `tfsdk:"role"`
+	UserID types.String `tfsdk:"user_id"`
+	// Scope is the target resolved, scope:ID.
+	Scope types.String `tfsdk:"scope"`
+}
+
+func (r *membershipResource) Metadata(_ context.Context, req resource.MetadataRequest, resp *resource.MetadataResponse) {
+	resp.TypeName = req.ProviderTypeName + "_membership"
+}
+
+func (r *membershipResource) Schema(_ context.Context, _ resource.SchemaRequest, resp *resource.SchemaResponse) {
+	replace := []planmodifier.String{stringplanmodifier.RequiresReplace()}
+	keep := []planmodifier.String{stringplanmodifier.UseStateForUnknown()}
+	resp.Schema = schema.Schema{
+		Description: "A person's membership of an organization, team or project. Taking it away revokes their keys in projects they no longer reach.",
+		Attributes: map[string]schema.Attribute{
+			"id": schema.StringAttribute{Computed: true, PlanModifiers: keep,
+				Description: "USER-ID/scope:SCOPE-ID. Import by it."},
+			"user": schema.StringAttribute{Required: true, PlanModifiers: replace,
+				Description: "The person, by ID or email."},
+			"target": schema.StringAttribute{Required: true, PlanModifiers: replace,
+				Description: "org:NAME, team:ORG/TEAM or project:ORG/TEAM/PROJECT, or the ID after the colon."},
+			"role": schema.StringAttribute{Optional: true,
+				Description: "member or manager; member when unset."},
+			"user_id": schema.StringAttribute{Computed: true, PlanModifiers: keep},
+			"scope":   schema.StringAttribute{Computed: true, PlanModifiers: keep, Description: "The target as scope:ID."},
+		},
+	}
+}
+
+func (r *membershipResource) Configure(_ context.Context, req resource.ConfigureRequest, _ *resource.ConfigureResponse) {
+	if c, ok := clientFrom(req.ProviderData); ok {
+		r.c = c
+	}
+}
+
+// resolveTarget is a target as scope:ID, read from the tree.
+func resolveTarget(ctx context.Context, c *client.Client, target string) (string, error) {
+	kind, what, ok := strings.Cut(strings.TrimSpace(target), ":")
+	if !ok || what == "" {
+		return "", fmt.Errorf("%q is not org:, team: or project: and a name or ID", target)
+	}
+	tree, err := c.Tree(ctx)
+	if err != nil {
+		return "", err
+	}
+	same := func(a, b string) bool { return strings.EqualFold(strings.TrimSpace(a), strings.TrimSpace(b)) }
+	for _, o := range tree {
+		if kind == "org" && (o.ID == what || same(o.Name, what)) {
+			return "org:" + o.ID, nil
+		}
+		for _, t := range o.Teams {
+			if kind == "team" && (t.ID == what || same(o.Name+"/"+t.Name, what)) {
+				return "team:" + t.ID, nil
+			}
+			for _, p := range t.Projects {
+				if kind == "project" && (p.ID == what || same(p.Path, what)) {
+					return "project:" + p.ID, nil
+				}
+			}
+		}
+	}
+	return "", fmt.Errorf("there is no %s %q", kind, what)
+}
+
+func roleOf(v types.String) string {
+	if v.IsNull() || v.ValueString() == "" {
+		return "member"
+	}
+	return v.ValueString()
+}
+
+func (r *membershipResource) join(ctx context.Context, m *membershipModel) error {
+	scope := m.Scope.ValueString()
+	if m.Scope.IsNull() || m.Scope.IsUnknown() {
+		var err error
+		if scope, err = resolveTarget(ctx, r.c, m.Target.ValueString()); err != nil {
+			return err
+		}
+	}
+	u, err := r.c.Join(ctx, m.User.ValueString(), client.MembershipRequest{Target: scope, Role: roleOf(m.Role)})
+	if err != nil {
+		return err
+	}
+	m.ID, m.UserID, m.Scope = types.StringValue(u.ID+"/"+scope), types.StringValue(u.ID), types.StringValue(scope)
+	return nil
+}
+
+func (r *membershipResource) Create(ctx context.Context, req resource.CreateRequest, resp *resource.CreateResponse) {
+	var plan membershipModel
+	resp.Diagnostics.Append(req.Plan.Get(ctx, &plan)...)
+	if resp.Diagnostics.HasError() {
+		return
+	}
+	if err := r.join(ctx, &plan); err != nil {
+		resp.Diagnostics.AddError("Putting "+plan.User.ValueString()+" in "+plan.Target.ValueString(), err.Error())
+		return
+	}
+	resp.Diagnostics.Append(resp.State.Set(ctx, &plan)...)
+}
+
+func (r *membershipResource) Read(ctx context.Context, req resource.ReadRequest, resp *resource.ReadResponse) {
+	var st membershipModel
+	resp.Diagnostics.Append(req.State.Get(ctx, &st)...)
+	if resp.Diagnostics.HasError() {
+		return
+	}
+	u, err := r.c.User(ctx, st.UserID.ValueString())
+	if client.IsNotFound(err) {
+		resp.State.RemoveResource(ctx)
+		return
+	}
+	if err != nil {
+		resp.Diagnostics.AddError("Reading "+st.User.ValueString(), err.Error())
+		return
+	}
+	for _, m := range u.Memberships {
+		if m.Target == st.Scope.ValueString() {
+			// Unset stays unset while the role is the default.
+			if !(st.Role.IsNull() && m.Role == "member") {
+				st.Role = types.StringValue(m.Role)
+			}
+			resp.Diagnostics.Append(resp.State.Set(ctx, &st)...)
+			return
+		}
+	}
+	resp.State.RemoveResource(ctx)
+}
+
+func (r *membershipResource) Update(ctx context.Context, req resource.UpdateRequest, resp *resource.UpdateResponse) {
+	var plan membershipModel
+	resp.Diagnostics.Append(req.Plan.Get(ctx, &plan)...)
+	if resp.Diagnostics.HasError() {
+		return
+	}
+	if err := r.join(ctx, &plan); err != nil {
+		resp.Diagnostics.AddError("Changing "+plan.User.ValueString()+"'s role in "+plan.Target.ValueString(), err.Error())
+		return
+	}
+	resp.Diagnostics.Append(resp.State.Set(ctx, &plan)...)
+}
+
+func (r *membershipResource) Delete(ctx context.Context, req resource.DeleteRequest, resp *resource.DeleteResponse) {
+	var st membershipModel
+	resp.Diagnostics.Append(req.State.Get(ctx, &st)...)
+	if resp.Diagnostics.HasError() {
+		return
+	}
+	if err := r.c.Leave(ctx, st.UserID.ValueString(), st.Scope.ValueString()); err != nil && !client.IsNotFound(err) {
+		resp.Diagnostics.AddError("Taking "+st.User.ValueString()+" out of "+st.Target.ValueString(), err.Error())
+	}
+}
+
+func (r *membershipResource) ImportState(ctx context.Context, req resource.ImportStateRequest, resp *resource.ImportStateResponse) {
+	user, scope, ok := strings.Cut(req.ID, "/")
+	if !ok || !strings.Contains(scope, ":") {
+		resp.Diagnostics.AddError("Not a membership ID", fmt.Sprintf("%q is not USER-ID/scope:SCOPE-ID, as id shows it", req.ID))
+		return
+	}
+	st := membershipModel{ID: types.StringValue(req.ID), User: types.StringValue(user), Target: types.StringValue(scope),
+		Role: types.StringNull(), UserID: types.StringValue(user), Scope: types.StringValue(scope)}
+	resp.Diagnostics.Append(resp.State.Set(ctx, &st)...)
+}
@@ -0,0 +1,167 @@
+package provider
+
+import (
+	"fmt"
+	"strings"
+	"testing"
+
+	"github.com/hashicorp/terraform-plugin-testing/helper/resource"
+	"github.com/hashicorp/terraform-plugin-testing/plancheck"
+	"github.com/hashicorp/terraform-plugin-testing/terraform"
+
+	"gitea.mynodes.xyz/ilker/llmgw/terraform-provider-llmgw/internal/client"
+)
+
+func tenancyConfig(project, team, role string) string {
+	roleLine := ""
+	if role != "" {
+		roleLine = fmt.Sprintf("  role   = %q\n", role)
+	}
+	return fmt.Sprintf(`
+resource "llmgw_org" "globex" {
+  name = "Globex"
+}
+
+resource "llmgw_team" "ops" {
+  name = "ops"
+  org  = %s
+}
+
+resource "llmgw_project" "web" {
+  name = %q
+  team = llmgw_team.ops.id
+}
+
+resource "llmgw_membership" "ayse" {
+  user   = "ayse@example.com"
+  target = "project:${llmgw_project.web.id}"
+%s}
+
+# Keys in the project that is not destroyed: one that ever had a key stays.
+resource "llmgw_key" "ci" {
+  name       = "ci"
+  project    = "Acme/ml/training"
+  expires_in = "720h"
+}
+`, team, project, roleLine)
+}
+
+func TestAccTheTreeKeysAndMembershipsAreManaged(t *testing.T) {
+	api, cfg := fake(t)
+	var team, project string
+	resource.Test(t, resource.TestCase{
+		ProtoV6ProviderFactories: factories,
+		CheckDestroy: func(*terraform.State) error {
+			if api.Has(client.LevelOrg, "org-1") || api.Has(client.LevelTeam, team) || api.Has(client.LevelProject, project) {
+				return fmt.Errorf("a part of the tree is left")
+			}
+			for _, k := range api.Keys() {
+				if k.State != "revoked" {
+					return fmt.Errorf("key %s is %s", k.ID, k.State)
+				}
+			}
+			if m := api.Memberships("u1"); len(m) != 0 {
+				return fmt.Errorf("memberships left: %+v", m)
+			}
+			return nil
+		},
+		Steps: []resource.TestStep{
+			{
+				Config: cfg + tenancyConfig("web", "llmgw_org.globex.name", ""),
+				Check: resource.ComposeAggregateTestCheckFunc(
+					resource.TestCheckResourceAttr("llmgw_project.web", "path", "Globex/ops/web"),
+					resource.TestCheckResourceAttrPair("llmgw_team.ops", "org_id", "llmgw_org.globex", "id"),
+					func(s *terraform.State) error {
+						got := s.RootModule().Resources["llmgw_membership.ayse"].Primary.Attributes["scope"]
+						if want := "project:" + s.RootModule().Resources["llmgw_project.web"].Primary.ID; got != want {
+							return fmt.Errorf("scope %q, not %q", got, want)
+						}
+						return nil
+					},
+					resource.TestCheckResourceAttr("llmgw_membership.ayse", "user_id", "u1"),
+					resource.TestCheckResourceAttr("llmgw_key.ci", "project_id", "p1"),
+					resource.TestCheckResourceAttrWith("llmgw_key.ci", "secret", func(v string) error {
+						if !strings.HasPrefix(v, "sk-llmgw-") {
+							return fmt.Errorf("secret %q", v)
+						}
+						return nil
+					}),
+					resource.TestCheckResourceAttrWith("llmgw_key.ci", "expires_at", func(v string) error {
+						if v == "" {
+							return fmt.Errorf("no expiry")
+						}
+						return nil
+					}),
+					func(s *terraform.State) error {
+						team = s.RootModule().Resources["llmgw_team.ops"].Primary.ID
+						project = s.RootModule().Resources["llmgw_project.web"].Primary.ID
+						return nil
+					},
+				),
+			},
+			{
+				// Renamed and the role changed, in place.
+				Config: cfg + tenancyConfig("site", "llmgw_org.globex.name", "manager"),
+				ConfigPlanChecks: resource.ConfigPlanChecks{PreApply: []plancheck.PlanCheck{
+					plancheck.ExpectResourceAction("llmgw_project.web", plancheck.ResourceActionUpdate),
+					plancheck.ExpectResourceAction("llmgw_membership.ayse", plancheck.ResourceActionUpdate),
+					plancheck.ExpectResourceAction("llmgw_key.ci", plancheck.ResourceActionNoop),
+				}},
+				Check: resource.ComposeAggregateTestCheckFunc(
+					resource.TestCheckResourceAttr("llmgw_project.web", "path", "Globex/ops/site"),
+					func(*terraform.State) error {
+						if m := api.Memberships("u1"); len(m) != 1 || m[0].Role != "manager" {
+							return fmt.Errorf("memberships %+v", m)
+						}
+						return nil
+					},
+				),
+			},
+			{
+				// Moved to another organization with its project.
+				Config: cfg + tenancyConfig("site", `"Acme"`, "manager"),
+				ConfigPlanChecks: resource.ConfigPlanChecks{PreApply: []plancheck.PlanCheck{
+					plancheck.ExpectResourceAction("llmgw_team.ops", plancheck.ResourceActionUpdate),
+				}},
+				// The project's path follows at the next refresh: the
+				// project itself did not change.
+				Check: resource.TestCheckResourceAttr("llmgw_team.ops", "org_id", "o1"),
+			},
+			{
+				// Renamed behind Terraform, and the key revoked: the name is
+				// put back, the key made again.
+				PreConfig: func() {
+					api.RenameBehindTerraform(client.LevelProject, project, "renamed")
+					for id, k := range api.Keys() {
+						if k.Name == "ci" && k.State == "active" {
+							api.RevokeBehindTerraform(id)
+						}
+					}
+				},
+				Config: cfg + tenancyConfig("site", `"Acme"`, "manager"),
+				ConfigPlanChecks: resource.ConfigPlanChecks{PreApply: []plancheck.PlanCheck{
+					plancheck.ExpectResourceAction("llmgw_project.web", plancheck.ResourceActionUpdate),
+					plancheck.ExpectResourceAction("llmgw_key.ci", plancheck.ResourceActionCreate),
+				}},
+				Check: resource.TestCheckResourceAttr("llmgw_project.web", "path", "Acme/ops/site"),
+			},
+			{
+				ResourceName:      "llmgw_org.globex",
+				ImportState:       true,
+				ImportStateVerify: true,
+			},
+			{
+				ResourceName:            "llmgw_membership.ayse",
+				ImportState:             true,
+				ImportStateVerify:       true,
+				ImportStateVerifyIgnore: []string{"user", "target"},
+			},
+			{
+				ResourceName:            "llmgw_key.ci",
+				ImportState:             true,
+				ImportStateVerify:       true,
+				ImportStateVerifyIgnore: []string{"secret", "expires_in", "project"},
+			},
+		},
+	})
+}
Author	SHA1	Message	Date
Ilker ManapandClaude Opus 5.5	37ad1657ee	v0.14.1: changelog and README Release / build (push) Successful in 1m59s Details Release / notes (push) Successful in 5s Details CI / test (push) Successful in 8m37s Details Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-09 10:11:10 +02:00
Ilker ManapandClaude Opus 5.5	8df78a4d41	The firewall setting reaches the mirror machine and the store's members The leading control plane tells the mirror over its API (PUT /v1/firewall, kept as .firewall.json beside the models and applied by the same path unit control planes use), and keeps the state in the store's cluster_state for members that run no llmgw, which a root timer reads from the member's own copy. The mirror also lets in the network of every node and control plane the leader knows. Tried on the PoC: off, on and a network added, on every kind of machine. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-09 08:53:17 +02:00
Ilker ManapandClaude Opus 5.5	5be3da505b	Keys' project paths written as the API takes them, Acme/ml/search Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-09 06:46:12 +02:00
Ilker ManapandClaude Opus 5.5	15a209e652	Terraform: organizations, teams, projects, keys and memberships llmgw_org, llmgw_team and llmgw_project rename and move in place; llmgw_key holds the key in a sensitive secret and is made again when revoked or expired outside Terraform; llmgw_membership puts a person in the tree. Tried with OpenTofu against the fake admin API, drift and import included. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-09 06:42:07 +02:00
Ilker ManapandClaude Opus 5.5	90e0942481	Organizations, teams, projects, keys and memberships in the admin API What llmgw org, keys and users did against the store on a control plane can now be done remotely and by automation: the tree read, added to, renamed, moved and emptied; keys issued (the key in the answer, once), listed and revoked, with the scopes' lifetime limits; people put in and taken out of the tree, their keys in projects they no longer reach revoked with it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-08 22:42:43 +02:00
Ilker ManapandClaude Opus 5.5	9c675e7293	Knowledge bases at scale: halfvec, a text index per collection, ef_search raised to the candidates wanted A search asking for more than 10 passages got at most 40 candidates from HNSW, pgvector's default ef_search. The full-text arm scored every chunk holding a word of the question before limiting, over the whole organization's GIN index; it now takes candidates by ts_rank from the collection's own partial index and scores those. New collections index halfvec on pgvector 0.7 and later (up to 4000 dimensions). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-08 22:08:29 +02:00
Ilker ManapandClaude Opus 5.5	42d3b7ea22	The store's PostgreSQL sized to its machine's memory Each Patroni member gets a share of its own RAM in its own config.yml: shared_buffers a quarter (15% beside a control plane or agent), effective_cache_size, maintenance_work_mem and work_mem scaled to it, all clamped. It is read again on every server init --patroni, so a machine given more memory gets the use of it. A single-machine PostgreSQL init installs is sized the same way, as a shared machine, instead of a fixed 96MB. The step itself went in with `85025df` by accident, in an early form that restarted PostgreSQL before Patroni had taken the reload, leaving the old shared_buffers with a restart pending. It now waits for Patroni to report the restart pending, restarts with restart_pending (replicas first, the leader last), and restarts a member it finds pending from an earlier run. Applied to the PoC: pg01 978MB, ctl02 (control plane beside it) 587MB, both from 128MB; streaming, lag 0, gateways unaffected. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-08 21:48:53 +02:00
Ilker ManapandClaude Opus 5.5	85025dfb03	Trained adapters published, private to their owners like the knowledge bases they come from An adapter now has an owner (org, team, project, user or key): only callers there see it, and to anyone else it is a model that does not exist, absent from /v1/models too. A training job's owner defaults to the one collection its pairs come from, must be able to read every collection they come from, and pairs from a knowledge base are never made public. ctl train publish keeps the GGUF or PEFT files on the mirror, which accepts uploads now, and sets the owned adapter after the usual architecture check. The knowledge bases' cache_salt follows the collections searched rather than the organization, so two teams of one organization no longer share cached passages. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-08 21:30:50 +02:00
Ilker ManapandClaude Opus 5.5	964b35ae65	Backups reach a replicated store: pg_dump is given the cluster's authority with verify-full pgEnv passed sslmode=verify-full but dropped sslrootcert, so libpq looked for ~/.postgresql/root.crt and both backup and restore failed against the Patroni store. sslrootcert, sslcert and sslkey now go along as PGSSL*. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-08 21:04:46 +02:00
Ilker ManapandClaude Opus 5.5	88cbf11570	Training finds the engines where the installer puts them when agent.yaml names no directory Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-08 20:55:50 +02:00
Ilker ManapandClaude Opus 5.5	79b3e9f887	LoRA adapters trained on the nodes: llmgw ctl train A job names the catalog model the adapter is for, the safetensors weights to train on, the node and the MiB of a card it needs, and question/answer pairs. The control plane sends it when the node's agent is there and nothing else trains on it; the agent trains with its installed vLLM's torch and a pinned overlay of peft and accelerate (no second environment), as the engines' user, on the card with that much free — beside an engine, if it leaves that much — converts the adapter to GGUF with the installed llama.cpp's own conversion code when the model is served by llama.cpp, and sends what it made back in pieces, kept in the store to download. Progress, loss and peak card memory are reported as it goes; a node that goes fails its job. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-08 20:25:57 +02:00
Ilker ManapandClaude Opus 5.5	ae95ac8261	Training data from a collection: llmgw ctl kb dataset A chat model of the cluster reads each chunk and writes questions a person might ask of it with the answers it holds, streamed as NDJSON a chunk at a time, for people to correct before an adapter is trained on it. A chunk that gave no pairs says so on its own line. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-08 20:04:37 +02:00
Ilker ManapandClaude Opus 5.5	0a6a586dfe	Backlog F4: a LoRA trained in the GPU's spare 2 GB on the PoC, and what it learned Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-08 19:31:53 +02:00