v0.16.0: changelog and README

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

BACKLOG.md

@@ -30,24 +30,21 @@ What is left, most needed first. The detail is in the item each names.
 **Soon: the customer's experience, and HA.**
 4. ~~**C24**~~: done 2026-10-21. The store's, catalog's and mirror's errors, percent signs, footer times and stale
    forms are in the page's language. Left: the /docs documents, and a language kept per person (cookie only today).
-5. **J4**: the model mirror on more than one machine. Code done 2026-10-29: mirrors kept in step (`mirror serve
-   --peer`, removals carried), `mirror.peers` and failover in the control plane, nodes given the file on a mirror that
-   answers, `mirror add-peer` and the setup's second mirror. Tried 2026-10-29 on two VMs:
-   - `add-peer` set the second up over ssh;
-   - a file came across in ~30 s, checksum matching;
-   - a version added on the second while the first was off reached the first when it came back;
-   - one removed on the first while the second was off was removed there when it came back, not copied back.
-
-   Left: the same with a control plane and a node, a model served while the first mirror is off. That part (failover
-   in the control plane, the node given the other mirror's file) is so far tried only in tests.
-6. **F5**: large imports limited by the embedding engine's slots. (M)
+5. ~~**J4**~~: done 2026-10-29. The done-criterion held on the PoC: with mirror 146 stopped, `ctl model serve` of a
+   new model was fetched by the second mirror (VM 167, 10.0.0.152), placed and answered; a node without a file
+   146's address names was given 152's; what was added meanwhile reached 146 when it came back; a removal was
+   carried both ways. Found and fixed there: add-peer on a mirror machine of its own, files without checksums never
+   copied, and `mirror remove` neither recorded nor seen by the running mirror.
+6. **F5**: large imports. Done 2026-10-29: the gateway embeds what `llmgw-kb` sends as text; the slots and
+   halfvec were done before. Left: a larger store machine before a million chunks (the operator's), and the
+   embedding model on a GPU for a real import.
 
 **From Kong** (section L): L1–L5 and L9 done (v0.15.0); left of L4 is routing by cost between outside
 providers, of L5 cache-write and audio prices. L6 and L7 are llmgw-80's; L8 (A2A) when asked.
 
 **Models we can trust** (section M, 2026-10-25): M1 done (pickles refused; the explorer lists only what this
 cluster's hardware holds, with CPU and GPU tok/s). Next M2 (every download pinned and checked against Hugging
-Face), then M5 (engines without the internet), M3, M4, M6.
+Face, done 2026-10-30), then M5 (engines without the internet), M3, M4, M6.
 
 **Governance.**
 7. **C1**: a tenant-admin role, and forms hidden from viewers.
@@ -1133,13 +1130,19 @@ four texts per pass, while llmgw-kb sends 32 a request from 6 workers. The
 usage rows show 4.6 s a request on average, 1.8 s of it in the gateway's
 queue and 0.04 s in the guard. The GPU sat at 18–50% and the store machine
 at a third of its CPUs.
-- An embedding model's profile with many slots by default (`--parallel`
+- Done for `ctl model serve` (32 slots of 1024 tokens; `ctl model context
+  MODEL 1024 --slots 32` resizes one served before): an embedding model's profile with many slots by default (`--parallel`
   32–64, `--ctx-size` to match, `--batch-size`/`--ubatch-size` at least the
   longest chunk), or `ctl model serve` choosing that for a model with
   `capabilities.embeddings`. With the default 512, one 516-token chunk failed.
-- Embedding on the gateway's side of a document PUT, so the client sends
-  text only: today an embeddings answer is ~21 KB of JSON a chunk, and
-  the text and vector go up again (~11 KB).
+- Done (2026-10-29): embedding on the gateway's side of a document PUT.
+  `llmgw-kb` sends text only to a gateway that says `server_embeds`; the
+  gateway embeds through its own `/v1/embeddings` as the caller, 32 a batch,
+  four batches at once, waiting out a busy engine. On 40 DTIC files the
+  upload went from 25.7 to 3.5 MiB, and the ~21 KB a chunk that came down
+  is gone; on 6 files both ways gave the same 32 chunks and the same search
+  answer. Speed is the engine's either way: the PoC's bge-m3 on node02's two
+  CPU cores embeds ~0.2 chunks/s, so a real import still wants it on a GPU.
 - The store next: HNSW inserts took half of one of pg01's two cores; at a
   million chunks the vectors alone outgrow its 4 GB. halfvec (pgvector ≥
   0.7) and a larger store machine.
@@ -1652,7 +1655,7 @@ their own (4 cores, 32 GB, two cores each) the same test refused 2.
 - Done when: the setup installs the judges on the machines given for them,
   and the load test above refuses no more than it did by hand.
 
-### J4. The model mirror on more than one machine (M)
+### J4. The model mirror on more than one machine (M) — done 2026-10-29, tried on the PoC
 In an HA install the mirror stays on the first machine. While it is down,
 models the nodes already have keep serving and new ones wait for it.
 - A second mirror kept in step (each machine of the store, or one more), the
@@ -1820,7 +1823,14 @@ pickle files. `allow_pickle: true` on a model lets it through, on purpose. On th
 PoC, `triposr` (`hf:stabilityai/TripoSR/model.ckpt`) is the one model that
 needs it.
 
-### M2. Every download pinned and checked against Hugging Face (M)
+### M2. Every download pinned and checked against Hugging Face (M) — done 2026-10-30
+Done: what is added from Hugging Face is pinned (the file's commit URL, or
+`--revision` for vLLM/SGLang) with each file's sha256 in the origin; the node
+refuses and deletes a copy that is not it, naming both, and keeps the
+checksum beside a cached file; the mirror checks what `ctl model serve`
+fetches, `mirror add` a Hugging Face URL. Left: models declared in the
+settings are not pinned, the admin page's and the console's mirror fetches
+send no sha256, and adapters by hub id are not pinned.
 - `hf:org/repo/file` follows `main` and is checked by nothing; a bare
   `org/repo` given to vLLM is downloaded by vLLM, of whatever `main` is.
   Both: resolve to the commit when the model is added, keep it in the
@@ -1855,6 +1865,45 @@ cluster, loopback), systemd hardening of the agent's engines
 (M4) serve only after its eval suite passes, so a model trained to answer
 wrongly on a trigger has at least been asked the suite.
 
+## N. Strata, all the way up (written 2026-10-30)
+
+Strata (opt-in engine since 2026-10-29) is wired at the agent: adapter, `llmgw strata-serve`, install pinned by
+digest, metrics in vLLM's names, GPU memory of child processes. Above the agent it is invisible: a Strata model is
+served only from a definition written by hand. In order:
+
+**A. What is broken.**
+- N1. A Strata named by hand (`engines.strata.dir`) was never reported: no Probe. Done 2026-10-30.
+- N2. Its engine runs in a session of its own and survived a stop, holding the card and the RAM: the supervisor
+  now finds what left the group before signalling it. Done 2026-10-30.
+- N3. Host memory: the experts live in RAM (40 GB for Qwen3.8-Flash-Next IQ2_XS, 72–136 GB for Unsloth's) and
+  nothing reserved it. A Strata profile added now reserves what its size takes, from the table (N5), unless it says
+  `min_memory_gb`. Done 2026-10-30.
+- N4. Pascal and Volta: installed (CUDA 12.9 build) and then refused (7.0 floor). Floor now 6.0. Done 2026-10-30.
+
+**B. Offered where models are chosen.**
+- N5. A table of Strata's families and sizes (qwen, coder, swift, unsloth, gyro, nvfp4, qwen36, deepseek41) with
+  download and RAM, from Strata 0.2.7's setup (internal/strata/models.go); the catalog refuses an unknown family or
+  size, and requires `--size`. Done 2026-10-30. Speeds measured: Qwen3.8-Flash-Next IQ2_XS 70–87 tok/s on an RTX 6000.
+- N6. `modelplan`, `ctl model plan` and `ctl model serve` offer "Strata on one card + N GB RAM" for a model it runs;
+  a large MoE that fits no card is not "fits nowhere". Done 2026-10-30, without a speed (one measurement so far).
+- N7. The web serve page, the explorer's fit, the console's serve form: the same choice. Done 2026-10-30.
+- N8. `--context`/`--parallel` tied to `context_length`/`max_output`; `ctl model context` takes Strata.
+
+**C. Operating it.**
+- N9. `ctl engine-check strata`: the flags of setup.py and server.py, which drift between versions.
+- N10. `ctl engine-build NODE strata cuda12|cuda13`, as vLLM has.
+- N11. Disk for the prepared pack and draft layer (and DeepSeek V4.1's 240–280 GB), cleaned with the weights; a
+  first start's preparation within the startup timeout.
+- N12. docs/install.md: Strata in the engines table, a section with a profile, its memory and first-run time;
+  `doctor` and `server init` checking or offering it.
+
+**D. What Strata can do that llmgw does not ask of it yet.**
+- N13. Several cards of one node (`--gpus`, `--gpu-roles`): larger packs on a customer's multi-GPU machines.
+- N14. `--calibrate` once per node and model, kept.
+- N15. Vision, KV types, experts from SSD when RAM is short (`--low-ram`) in the plan and forms.
+- N16. Versions apart from llmgw's releases (Strata's signed update feed, `--rollback-engine`); offline bundles.
+- N17. `cache_salt` for KB isolation if Strata caches prefixes; `ctl bench` with its metrics path.
+
 ## D. Facts worth knowing before starting
 
 - Admin API: `127.0.0.1:9090`, behind the token in

CHANGELOG.md

@@ -19,12 +19,118 @@ is only released once its **exit criterion** is met, not when its code lands.
 | v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released |
 | v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
 | v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines; v0.14.7: the judge sized for its machine; v0.14.8: sign-in with Active Directory, several keys for one outside provider; v0.14.9: sign-in set up from the console, a node's console that shows the machine) |
-| v0.15.0 | — | The company's MCP tools and VS Code through the gateway; guardrails that ask outside services and put redacted values back; answers to similar questions | released (v0.15.1: Strata, a 125B model of experts on one card; a second mirror kept in step; pickled weights refused; a served model's context changed) |
+| v0.15.0 | — | The company's MCP tools and VS Code through the gateway; guardrails that ask outside services and put redacted values back; answers to similar questions | released (v0.15.1: Strata, a 125B model of experts on one card; a second mirror kept in step; pickled weights refused; a served model's context changed; v0.15.2: a served model moved to more cards; v0.15.3: a model stopped and kept, or removed with its files) |
+| v0.16.0 | — | Weights pinned to a commit and checked against what Hugging Face published; documents embedded by the gateway; Strata offered where models are chosen; cluster upgrades that say how far they are | released |
 | v1.0.0 | 5 | What phase 5 has left: precise KV routing proven on vLLM, the block cost picker, defragmentation across NVLink domains, prefill/decode disaggregation | planned |
 
 Until v1.0.0 the minor version is the breaking one: configuration formats and Go
 APIs change between minors without a deprecation period.
 
+## v0.16.0 — 2026-10-30
+
+Weights pinned to the commit they were added at and checked, by the nodes and the mirror, against the sha256
+Hugging Face published; the documents `llmgw-kb` sends embedded by the gateway; Strata offered by the plan, `ctl
+model serve`, the explorer and the serve forms; and cluster upgrades that tell the nodes together, say how far they
+are, and do not fail a node that came back on the version.
+
+**Upgrading.**
+- No migrations. Upgrade the control planes, then the nodes (`llmgw ctl upgrade v0.16.0`), then each mirror's host
+  (`sudo llmgw upgrade v0.16.0 --service llmgw-mirror`).
+- A node checks what it fetches against the catalog's checksums only once it runs this version; until then it
+  fetches as before.
+- A model added from Hugging Face is now pinned when it is added, which needs the registry reachable from the
+  control plane; one it cannot be looked up in is refused, saying why. Models already in the catalog are left as
+  they are.
+- A Strata profile must say `--size` as well as `--family`, and both must be ones Strata runs.
+- `llmgw-kb` sends text alone to a gateway of this version; an older `llmgw-kb` keeps working, sending vectors.
+
+### Added
+
+- **Strata offered where models are chosen.** For a repository of a family Strata runs, `ctl model plan` offers "one
+  card with Strata, experts in RAM" with the RAM it takes, `ctl model serve --engine strata` (or the plan's choice)
+  builds its profile, the Hugging Face explorer lists it as running here, and the mirror's serve form serves such a
+  version on Strata. Strata's families and sizes are known: the catalog refuses one it does not run and reserves the
+  RAM a size keeps its experts in.
+- **The gateway embeds the documents `llmgw-kb` sends.** A document may be sent as its passages' text alone; the
+  gateway embeds them through its own `/v1/embeddings` as the caller, 32 at a time and four batches at once,
+  waiting out a busy engine rather than failing the document. About a kilobyte a passage leaves the machine instead
+  of a vector down and up again, some thirty times more. `llmgw-kb` does so with a gateway that says it can
+  (`server_embeds` in a collection's answer); `--embed-here` keeps the old way.
+- **Weights pinned to a commit, and checked against what Hugging Face published.** A model added from Hugging Face
+  (`hf:`, a resolve URL on a branch, or a repository vLLM or SGLang downloads) is pinned to the commit it is at
+  then — the file's URL at that commit, or `--revision` — and each file's published sha256 goes in its origin. A node
+  refuses and deletes a copy that is not it, naming both checksums; the mirror does the same for what `ctl model
+  serve` fetches, and `llmgw mirror add` for a Hugging Face URL (`--sha256` for others). Nodes need this version to
+  check; older ones fetch as before.
+
+### Fixed
+
+- **A cluster upgrade that sat silent, or stopped on a node that had upgraded.**
+  - The nodes of a wave are told together, each saying when it is done, and what is still awaited is said every 30
+    seconds; before, each was told in turn, up to five minutes each, with nothing on the screen.
+  - The control plane takes the node registering again on the version as done: the agent's answer, sent a moment
+    before it exits, could be lost, and a node that had upgraded was failed as "older than v0.10.4".
+  - The agent's download no longer ends with its connection (a control plane restarting, leadership moving), and the
+    control plane waits longer than the download may take; a second request while one runs is refused.
+  - A cordoned or draining node upgraded is done; a node whose idle model stopped meanwhile is done once nothing on it
+    is starting; a node not connected is passed over and named at the end rather than stopping every node after it.
+  - The console's node upgrade waits as long as the control plane, and offers no development version.
+  - systemd never gives up restarting the agent (`StartLimitIntervalSec=0`), in units written from now on.
+
+- **Strata.** One named by hand in `agent.yaml` was never reported, so never placed on; its engine, which leaves the
+  server's process group, is now stopped with it; Pascal and Volta, which its CUDA 12 build runs, were refused.
+- **A name with a slash in it, asked through a standby control plane.** `llmgw ctl kb delete Demo/hr` on a standby
+  was answered 405: the standby passed the request on to the leader with `Demo%2Fhr` unescaped, two segments instead
+  of one. The path now reaches the leader as it was sent.
+
+## v0.15.3 — 2026-10-29
+
+A model stopped and kept, or removed with its files; the second mirror fixed where it was tried.
+
+**Upgrading.**
+- No migrations. The control planes carry stop, start and purge; the nodes need nothing new.
+- The mirrors carry the fixes: on each mirror's machine, `sudo llmgw upgrade v0.15.3 --service llmgw-mirror`.
+  A mirror with a peer then works out the checksums its older files lack, once, which reads every such file.
+
+### Added
+
+- **A model stopped and kept, and a model removed with its files.** `llmgw ctl model stop|start MODEL`: none of its
+  replicas run, it stays in the catalog with its files, and start runs it again scaled as it was; requests are told an
+  administrator stopped it. `llmgw ctl model remove MODEL --purge` (`DELETE /api/v1/models/{name}?purge=true`) also
+  deletes every version of it on the mirror, a second mirror following, and the nodes' copies once its replicas have
+  stopped; what another model uses stays. Both are in the console's model menu.
+
+### Fixed
+
+- **`llmgw mirror add-peer` on a mirror with a machine of its own.** It set the second mirror up and then
+  failed for want of the control plane's server.yaml; now it prints the `llmgw ctl config set mirror.peers`
+  to run on a control plane.
+- **Models fetched before checksums were kept reach the second mirror.** A mirror kept in step copies only what it
+  can check, and those files had no sha256; a mirror with peers now works theirs out and records them, once.
+- **A file removed with `llmgw mirror remove` is gone from the mirror's answers too.** The running mirror kept
+  reporting a file it had fetched as present for a week after it was removed, so a control plane could place a
+  model on a file that was not there; and the removal was not recorded, so a mirror kept in step copied it back.
+
+## v0.15.2 — 2026-10-29
+
+A served model moved to more cards, or fewer, while it is served.
+
+**Upgrading.**
+- No migrations, and nothing changes until the command is used. The control
+  planes carry it; the nodes need nothing new.
+
+### Added
+
+- **A served model moved to more cards, or fewer.** `llmgw ctl model gpus
+  MODEL N [--profile P] [--in-place]`, `POST /api/v1/models/{name}/gpus`, and
+  GPUs… in the console's model menu. How many cards of one node each replica
+  uses: tensor parallel for vLLM, SGLang and TensorRT-LLM, checked against
+  the model's KV heads, and the layers split for llama.cpp. Refused for a
+  count no node has, or whose cards would not hold the model and its
+  context, with the figures; its replicas are then started again on it, one
+  by one, beside the old — or, with `--in-place` when there is no room
+  beside them, each stopped first.
+
 ## v0.15.1 — 2026-10-29
 
 A 125B model on one card, a second mirror, and weights that run code

README.md

@@ -41,7 +41,8 @@ Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
 | v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released |
 | v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
 | v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines; v0.14.7: the judge sized for its machine; v0.14.8: sign-in with Active Directory, several keys for one outside provider; v0.14.9: sign-in set up from the console, a node's console that shows the machine) |
-| v0.15.0 | — | The company's MCP tools and VS Code through the gateway; guardrails that ask outside services and put redacted values back; answers to similar questions | released (v0.15.1: Strata, a 125B model of experts on one card; a second mirror kept in step; pickled weights refused; a served model's context changed) |
+| v0.15.0 | — | The company's MCP tools and VS Code through the gateway; guardrails that ask outside services and put redacted values back; answers to similar questions | released (v0.15.1: Strata, a 125B model of experts on one card; a second mirror kept in step; pickled weights refused; a served model's context changed; v0.15.2: a served model moved to more cards; v0.15.3: a model stopped and kept, or removed with its files) |
+| v0.16.0 | — | Weights pinned to a commit and checked against what Hugging Face published; documents embedded by the gateway; Strata offered where models are chosen; cluster upgrades that say how far they are | released |
 | v1.0.0 | 5 | What phase 5 has left: precise KV routing proven on vLLM, the block cost picker, defragmentation across NVLink domains, prefill/decode disaggregation | planned |
 
 As of v0.6.0 a cluster runs unattended: agents report their hardware, the

api/gen/llmgw/v1/agent.pb.go

@@ -3849,7 +3849,11 @@ type EngineProfile struct {
 	KvEvents bool `protobuf:"varint,13,opt,name=kv_events,json=kvEvents,proto3" json:"kv_events,omitempty"`
 	// allow_pickle lets the node fetch weights in a pickle format, which run
 	// code when loaded: the catalog's allow_pickle, set on purpose.
-	AllowPickle   bool `protobuf:"varint,14,opt,name=allow_pickle,json=allowPickle,proto3" json:"allow_pickle,omitempty"`
+	AllowPickle bool `protobuf:"varint,14,opt,name=allow_pickle,json=allowPickle,proto3" json:"allow_pickle,omitempty"`
+	// model_sha256 is the sha256 the registry published for the file
+	// model_path names, from the catalog's origin: the node refuses a copy
+	// that is not it. Empty when nothing is known to check against.
+	ModelSha256   string `protobuf:"bytes,15,opt,name=model_sha256,json=modelSha256,proto3" json:"model_sha256,omitempty"`
 	unknownFields protoimpl.UnknownFields
 	sizeCache     protoimpl.SizeCache
 }
@@ -3982,6 +3986,13 @@ func (x *EngineProfile) GetAllowPickle() bool {
 	return false
 }
 
+func (x *EngineProfile) GetModelSha256() string {
+	if x != nil {
+		return x.ModelSha256
+	}
+	return ""
+}
+
 // RemoveWeights deletes cached model files a node no longer needs.
 //
 // The paths are computed by the control plane from the catalog, but the agent
@@ -5536,7 +5547,7 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
 	"profile_fp\x18\x0f \x01(\tR\tprofileFp\x12\x17\n" +
 	"\aapi_key\x18\x10 \x01(\tR\x06apiKey\x12\x1f\n" +
 	"\vcpu_threads\x18\x11 \x01(\x05R\n" +
-	"cpuThreads\"\xf0\x03\n" +
+	"cpuThreads\"\x93\x04\n" +
 	"\rEngineProfile\x12\x12\n" +
 	"\x04name\x18\x01 \x01(\tR\x04name\x12\x16\n" +
 	"\x06engine\x18\x02 \x01(\tR\x06engine\x12\x1b\n" +
@@ -5553,7 +5564,8 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
 	"\x11served_model_name\x18\v \x01(\tR\x0fservedModelName\x121\n" +
 	"\badapters\x18\f \x03(\v2\x15.llmgw.v1.LoraAdapterR\badapters\x12\x1b\n" +
 	"\tkv_events\x18\r \x01(\bR\bkvEvents\x12!\n" +
-	"\fallow_pickle\x18\x0e \x01(\bR\vallowPickle\x1a6\n" +
+	"\fallow_pickle\x18\x0e \x01(\bR\vallowPickle\x12!\n" +
+	"\fmodel_sha256\x18\x0f \x01(\tR\vmodelSha256\x1a6\n" +
 	"\bEnvEntry\x12\x10\n" +
 	"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
 	"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"%\n" +

api/proto/llmgw/v1/agent.proto

@@ -767,6 +767,10 @@ message EngineProfile {
   // allow_pickle lets the node fetch weights in a pickle format, which run
   // code when loaded: the catalog's allow_pickle, set on purpose.
   bool allow_pickle = 14;
+  // model_sha256 is the sha256 the registry published for the file
+  // model_path names, from the catalog's origin: the node refuses a copy
+  // that is not it. Empty when nothing is known to check against.
+  string model_sha256 = 15;
 }
 
 // RemoveWeights deletes cached model files a node no longer needs.

cmd/llmgw-kb/main.go

@@ -331,6 +331,8 @@ hidden files, version control folders and Office lock files are not.`,
 	f.BoolVar(&o.Delete, "delete", false, "remove documents this machine sent whose files are gone")
 	f.BoolVar(&o.Retry, "retry", false, "read again files skipped before")
 	f.IntVar(&o.Workers, "workers", 4, "files read and sent at once")
+	f.BoolVar(&o.EmbedHere, "embed-here", false,
+		"embed on this machine and send the vectors, even to a gateway that embeds text it is sent")
 	f.StringVar(&ocr, "ocr-languages", "tur+eng", "Tesseract's languages for images and scanned pages")
 	f.Int64Var(&maxMiB, "max-size", 256, "skip files larger than this many MiB")
 	f.BoolVarP(&verbose, "verbose", "v", false, "list unchanged files too")

deploy/systemd/llmgw-agent.service

@@ -2,6 +2,9 @@
 Description=llmgw node agent
 After=network-online.target
 Wants=network-online.target
+# Never given up on: a new release that fails its first starts, or an upgrade
+# that restarts it twice in a minute, would otherwise leave the node off.
+StartLimitIntervalSec=0
 
 [Service]
 ExecStart=/usr/local/bin/llmgw agent --config /etc/llmgw/agent.yaml

docs/admin-api.openapi.json

@@ -2387,7 +2387,7 @@
       "delete": {
         "operationId": "deleteModelsName",
         "summary": "Remove a model added at runtime",
-        "description": "A model declared in the cluster's settings is refused: it is changed there.",
+        "description": "A model declared in the cluster's settings is refused: it is changed there. With purge, its files go too: every version of it the mirror holds (a peer mirror follows), and the nodes' copies once its replicas have stopped, waiting up to a minute for them; what another model uses stays. The answer is then a PurgeView.",
         "tags": [
           "models"
         ],
@@ -2401,6 +2401,14 @@
               "type": "string"
             }
           },
+          {
+            "name": "purge",
+            "in": "query",
+            "description": "Delete its files from the mirror and the nodes as well.",
+            "schema": {
+              "type": "boolean"
+            }
+          },
           {
             "$ref": "#/components/parameters/Actor"
           }
@@ -2964,6 +2972,58 @@
         }
       }
     },
+    "/api/v1/models/{name}/gpus": {
+      "post": {
+        "operationId": "postModelsNameGpus",
+        "summary": "Spread a served model over more cards or fewer: its profile, checked against its KV heads, its cards' memory and the nodes, then its replicas restarted",
+        "description": "Tensor parallel for vLLM, SGLang and TensorRT-LLM, a layer split for llama.cpp, on one node. With no room beside the running replicas, in_place stops each first.",
+        "tags": [
+          "models"
+        ],
+        "parameters": [
+          {
+            "name": "name",
+            "in": "path",
+            "required": true,
+            "description": "The model's name in the catalog.",
+            "schema": {
+              "type": "string"
+            }
+          },
+          {
+            "$ref": "#/components/parameters/Actor"
+          }
+        ],
+        "requestBody": {
+          "required": true,
+          "content": {
+            "application/json": {
+              "schema": {
+                "$ref": "#/components/schemas/modelserve.GPUsRequest"
+              }
+            }
+          }
+        },
+        "responses": {
+          "200": {
+            "description": "OK",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/modelserve.GPUsChange"
+                }
+              }
+            }
+          },
+          "401": {
+            "$ref": "#/components/responses/Unauthorized"
+          },
+          "default": {
+            "$ref": "#/components/responses/Error"
+          }
+        }
+      }
+    },
     "/api/v1/models/{name}/headroom": {
       "get": {
         "operationId": "getModelsNameHeadroom",
@@ -3348,6 +3408,89 @@
         }
       }
     },
+    "/api/v1/models/{name}/start": {
+      "post": {
+        "operationId": "postModelsNameStart",
+        "summary": "Run a stopped model again, scaled as it was",
+        "tags": [
+          "models"
+        ],
+        "parameters": [
+          {
+            "name": "name",
+            "in": "path",
+            "required": true,
+            "description": "The model's name in the catalog.",
+            "schema": {
+              "type": "string"
+            }
+          },
+          {
+            "$ref": "#/components/parameters/Actor"
+          }
+        ],
+        "responses": {
+          "200": {
+            "description": "OK",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/ScaleResponse"
+                }
+              }
+            }
+          },
+          "401": {
+            "$ref": "#/components/responses/Unauthorized"
+          },
+          "default": {
+            "$ref": "#/components/responses/Error"
+          }
+        }
+      }
+    },
+    "/api/v1/models/{name}/stop": {
+      "post": {
+        "operationId": "postModelsNameStop",
+        "summary": "Take a model out of service and keep it",
+        "description": "None of its replicas run, and requests are told an administrator stopped it; what it was scaled to is kept for start.",
+        "tags": [
+          "models"
+        ],
+        "parameters": [
+          {
+            "name": "name",
+            "in": "path",
+            "required": true,
+            "description": "The model's name in the catalog.",
+            "schema": {
+              "type": "string"
+            }
+          },
+          {
+            "$ref": "#/components/parameters/Actor"
+          }
+        ],
+        "responses": {
+          "200": {
+            "description": "OK",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/ScaleResponse"
+                }
+              }
+            }
+          },
+          "401": {
+            "$ref": "#/components/responses/Unauthorized"
+          },
+          "default": {
+            "$ref": "#/components/responses/Error"
+          }
+        }
+      }
+    },
     "/api/v1/models/{name}/target": {
       "get": {
         "operationId": "getModelsNameTarget",
@@ -7807,6 +7950,10 @@
           "max_output": {
             "type": "integer"
           },
+          "stopped": {
+            "type": "boolean",
+            "description": "Stopped is a model an administrator took out of service, kept."
+          },
           "canary": {
             "$ref": "#/components/schemas/catalog.Canary",
             "description": "Canary is a share of its requests sent to a candidate; CanaryOf is set on the candidate, naming the model it is a canary of."
@@ -8887,6 +9034,10 @@
           },
           "on_demand": {
             "type": "boolean"
+          },
+          "stopped": {
+            "type": "boolean",
+            "description": "Stopped is a model taken out of service, kept, until it is started."
           }
         }
       },
@@ -10012,6 +10163,10 @@
           "scaling": {
             "$ref": "#/components/schemas/catalog.Scaling"
           },
+          "stopped": {
+            "$ref": "#/components/schemas/catalog.Scaling",
+            "description": "Stopped is the scaling a model had when an administrator stopped it: kept in the catalog with none running (Scaling 0/0) until started again with this. Nil for a model not stopped."
+          },
           "class": {
             "type": "string",
             "description": "Class is how much this model's capacity is worth against another's, when there is not enough for both: critical, standard or opportunistic. Empty is standard, or opportunistic for a model loaded on demand. See EffectiveClass."
@@ -11031,6 +11186,13 @@
           "expert_parallel": {
             "type": "boolean",
             "description": "ExpertParallel spreads a mixture-of-experts model's experts over the cards, whole, rather than slicing each across them: vLLM's --enable-expert-parallel, for a model of experts on more than one card."
+          },
+          "strata_family": {
+            "type": "string",
+            "description": "StrataFamily and StrataSize are Strata's --family and --size, for an option on Strata: one card, and the experts in the node's RAM, which NeedGB and HaveGB are then about."
+          },
+          "strata_size": {
+            "type": "string"
           }
         }
       },
@@ -11133,6 +11295,67 @@
           }
         }
       },
+      "modelserve.GPUsChange": {
+        "type": "object",
+        "description": "GPUsChange is what was changed, and the replicas started again on it.",
+        "properties": {
+          "model": {
+            "type": "string"
+          },
+          "profile": {
+            "type": "string"
+          },
+          "engine": {
+            "type": "string"
+          },
+          "gpu_class": {
+            "type": "string"
+          },
+          "before": {
+            "type": "integer"
+          },
+          "after": {
+            "type": "integer"
+          },
+          "args": {
+            "type": "array",
+            "items": {
+              "type": "string"
+            }
+          },
+          "per_card": {
+            "type": "string",
+            "description": "PerCard is what each card holds after, when it could be worked out."
+          },
+          "restarted": {
+            "type": "integer"
+          },
+          "in_place": {
+            "type": "boolean"
+          },
+          "note": {
+            "type": "string",
+            "description": "Note is what could not be checked, said once."
+          }
+        }
+      },
+      "modelserve.GPUsRequest": {
+        "type": "object",
+        "description": "GPUsRequest is how many cards a replica is to use.",
+        "properties": {
+          "gpus": {
+            "type": "integer"
+          },
+          "profile": {
+            "type": "string",
+            "description": "Profile is the one to change, when the model has more than one profile on cards."
+          },
+          "in_place": {
+            "type": "boolean",
+            "description": "InPlace stops each replica before its replacement starts: for when no node has the cards free beside it. It does not answer while the new one loads."
+          }
+        }
+      },
       "modelserve.Job": {
         "type": "object",
         "description": "Job is one model being served, and where it stands.",
@@ -11320,6 +11543,10 @@
             "type": "boolean",
             "description": "Fallbacks adds the next best ways, lower in priority, so a model whose first choice is busy runs on another rather than waiting."
           },
+          "engine": {
+            "type": "string",
+            "description": "Engine is the engine the file runs on: \"strata\" for a family Strata runs, \"llamacpp\" to keep it off Strata; empty, the plan's best."
+          },
           "name": {
             "type": "string",
             "description": "Name is the catalog name; the repository suggests one otherwise."

docs/client-api.openapi.json

@@ -2890,6 +2890,10 @@
           "writable": {
             "type": "boolean",
             "description": "Writable is whether this caller may add and remove documents."
+          },
+          "server_embeds": {
+            "type": "boolean",
+            "description": "ServerEmbeds says a document may be sent as text alone: the gateway embeds its chunks, as the caller, with the collection's model."
           }
         }
       },

docs/install.md

@@ -703,6 +703,7 @@ Nothing to do: the agent installs them once the node joins, under
 | vLLM | nodes with an NVIDIA GPU it can run on | a Python environment built by uv from PyPI, about 8 GB |
 | SGLang | only when you ask for it | the same way as vLLM |
 | TensorRT-LLM | only nodes where you installed it | not installed by llmgw: see [TensorRT-LLM, which you bring](#tensorrt-llm-which-you-bring) |
+| Strata | only when you ask for it (`engines: {strata: "0.2.7"}`) | its source, pinned by digest, in a Python environment, and its prebuilt engine for CUDA 13 (driver 580+) or CUDA 12.9 (Pascal to Ada, driver 575+); see [Strata](#strata-a-large-model-of-experts-on-one-card) |
 
 vLLM is resolved as PyPI stood a day after that vLLM was published, so every
 node gets the same dependencies whenever it is built, and installed with
@@ -745,6 +746,28 @@ you installed yourself on a node, name its path in that node's `agent.yaml`
 (`engines.llamacpp.bin`, `engines.vllm.bin`, `engines.sglang.python`,
 `engines.trtllm.python`): the agent then installs none of its own beside it.
 
+#### Strata: a large model of experts on one card
+
+Strata runs a few mixture-of-experts models — Qwen3.8-Flash-Next (125B) and its
+Swift, Coder, Unsloth, Gyro and NVFP4 versions, Qwen3.6-35B-A3B, DeepSeek
+V4.1-Flash — with the experts in the node's RAM and the rest on one card of 12
+GB or more. Where it is installed, the plan offers it for a repository of one
+of those families:
+
+```
+llmgw ctl model plan ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF
+llmgw ctl model serve ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF \
+  --file Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00001-of-00002.gguf --engine strata
+```
+
+The Hugging Face explorer shows the same ("1× … with Strata, N GB of RAM"),
+and the mirror's serve form serves such a version on Strata. A Strata profile
+says its family and size (`--family qwen --size IQ2_XS`), which the catalog
+checks against what Strata runs, and reserves the RAM the size keeps its experts
+in (48 GB and the engine's 4 for IQ2_XS) unless it says `min_memory_gb`. Its
+first start prepares the model on the node, which takes minutes, and fetches a
+draft layer of about 5 GB from the internet.
+
 #### Choosing a node's vLLM build
 
 Which build a GPU node installs is chosen on the cluster page, in the **vLLM
@@ -897,6 +920,24 @@ the node — an adapter given by its id, a TensorRT-LLM model's — is refused
 when the model is added, unless it says `allow_pickle`: mirror its
 safetensors and give that instead.
 
+**Weights are pinned to a commit and checked.** A model added with
+`ctl model add`, the API or the console, whose weights come from Hugging Face,
+is pinned to the commit the repository is at when it is added:
+
+- `hf:org/repo/file` and a `…/resolve/main/…` URL become that commit's URL;
+- a repository vLLM or SGLang downloads itself is given `--revision`;
+- the sha256 Hugging Face published for each file goes in the model's origin
+  (its definition, `GET /api/v1/models/NAME`, shows it).
+
+A node then refuses a copy of the file that is not it, naming both checksums,
+and deletes it; a cached copy is checked once and its checksum kept beside it.
+The mirror does the same for a model `ctl model serve` fetches, and
+`llmgw mirror add` for a Hugging Face URL (`--sha256` for any other). A new
+upload to the repository reaches the cluster only through `ctl model update`,
+which says what changed. Not pinned: models declared in the cluster's settings,
+which are not added this way, and a vLLM model whose arguments already say
+`--revision` keeps that one.
+
 Put them where the node can read them. `scp`, a shared filesystem, or the
 engine's own downloader are all valid, and which you want depends on how many
 nodes you have.
@@ -1629,6 +1670,30 @@ Not yet: models declared in server.yaml cannot name a peer's model as a
 fallback (the catalog in the file is checked before the peers are read), and
 a peer behind a private CA needs that CA in the system's trust store.
 
+### Stopping a model, and removing one with its files
+
+```
+llmgw ctl model stop qwen-7b             # none of its replicas run; it is kept
+llmgw ctl model start qwen-7b            # runs again, scaled as it was
+llmgw ctl model remove qwen-7b           # leaves the catalog; its files stay
+llmgw ctl model remove qwen-7b --purge   # its files on the mirror and the nodes too
+```
+
+- **Stop** keeps the model in the catalog with its files and what it was
+  scaled to; a request for it is told an administrator stopped it. `ctl
+  models` shows it as stopped. A `ctl scale` while it is stopped starts it at
+  that number.
+- **Remove** leaves its files on the mirror and the nodes, for adding it
+  again without fetching it. `ctl weights` lists such files on the nodes.
+- **Remove --purge** deletes every version of it on the mirror (a second
+  mirror follows), its adapters and the weights its last update replaced,
+  and the nodes' copies once its replicas have stopped. What another model
+  uses — a canary's weights, a shared adapter — stays. It cannot be undone:
+  adding it again fetches it again.
+
+The console has the same in a model's menu: Stop… or Start, Remove…, and
+Remove with its files….
+
 ### Changing a served model's context
 
 How much one request may use is set when a model is served. To change it
@@ -1648,6 +1713,36 @@ it, one by one, a new one ready before the old one stops. The console has the
 same under a model's menu, Context length…. A model declared in the cluster's
 settings is changed there (`llmgw ctl config edit`).
 
+### Moving a served model to more cards
+
+How many cards of one node each replica uses can be changed while the model
+is served:
+
+```
+llmgw ctl model gpus qwen3-coder-30b 2              # two cards a replica
+llmgw ctl model gpus llama-70b 4 --profile h100     # the model has several profiles on cards
+llmgw ctl model gpus qwen3-coder-30b 2 --in-place   # no room beside the running replica
+```
+
+vLLM, SGLang and TensorRT-LLM spread the model by tensor parallel (`tp`),
+which needs the count to divide the model's KV heads or be divided by them:
+a model with 8 splits 2, 4, 8 or 16 ways, not 3 or 12. llama.cpp has no
+tensor parallel; it splits the layers over the cards (`gpus_per_node`), and
+a `--tensor-split` written for the old cards is dropped. ComfyUI and Strata
+run on one card. A count no node has is refused, and so is one whose cards
+would not hold the weights and the context — with the figures; fewer cards
+with a long context may need `llmgw ctl model context` first.
+
+Its replicas are then started again on it, each new one ready beside the old
+before the old stops. That needs the cards free beside it: on a node of two
+cards with the model on one, two do not fit until it is gone, and the command
+says so. `--in-place` stops each replica first and starts its replacement on
+the cards it held and the free ones, and the model does not answer while the
+new one loads — a model with several replicas keeps answering on the others.
+A replica spread over several nodes is laid out by hand, with a fabric
+(`nodes`, `gpus_per_node` in the profile). The console has the same under a
+model's menu, GPUs….
+
 ### Trying new weights on some of the traffic first
 
 When a model's repository publishes new weights, `llmgw ctl model update`
@@ -2553,6 +2648,12 @@ is down, and new ones wait for it. With a second, the cluster carries on.
   ```
   sudo llmgw mirror add-peer ubuntu@10.0.0.7 --control-plane 10.0.0.5 --control-plane 10.0.0.6 --control-plane 10.0.0.7
   ```
+
+  It logs in with root's ssh key on the mirror's machine, so that key must
+  be accepted for the user named on the second machine. When the mirror has
+  a machine of its own, without the control plane, the command cannot change
+  the cluster's settings itself: it prints the `llmgw ctl config set
+  mirror.peers …` to run on a control plane.
 - **Kept in step:** each mirror reads the other every thirty seconds and
   copies what it lacks, keeping a copy only if its checksum matches.
   - A model removed from one is removed from the other, even if that one

docs/knowledge.md

@@ -3,9 +3,9 @@
 A knowledge base is a collection of a company's documents that the cluster
 can answer from. People put their documents into it from their own machines
 with `llmgw-kb`. The tool reads the files on that machine, cuts them into
-passages ("chunks"), has the gateway embed each passage with the person's own
-key, and sends the text and the vectors. The files themselves never leave
-the machine.
+passages ("chunks") and sends their text; the gateway embeds each passage
+with the person's own key and keeps the text and the vectors. The files
+themselves never leave the machine.
 
 ## Why retrieval, and not training
 
@@ -159,10 +159,12 @@ llmgw ctl config set router.admission.bands '[{name: gold, priority: 300, satura
 llmgw keys create import --project Acme/docs --band bulk
 ```
 
-The people's questions then go first, and the import waits behind them. A
-request that waits as long as its band allows is answered 504 with
-`Retry-After`, and `llmgw-kb` sends it again rather than counting the
-passage as failed. It gives up after 15 minutes.
+The people's questions then go first, and the import waits behind them. The
+gateway embeds a document's passages 32 at a time, four batches at once,
+enough to fill an embedding replica's slots; a batch told to wait (`Retry-After`)
+waits up to five minutes before the document is answered the same, and
+`llmgw-kb` sends it again rather than counting it as failed. It gives up
+after 15 minutes.
 
 ## For everyone: `llmgw-kb`
 
@@ -225,6 +227,7 @@ not changed.
 | `--include`, `--exclude` | Glob patterns. `*.pdf` matches a file name; `reports/**/*.xlsx` matches a path. |
 | `--hidden` | Reads hidden files and folders too (left out by default). |
 | `-v` | Lists unchanged files too. |
+| `--embed-here` | Embeds the passages from this machine and sends the vectors. With a gateway that does not embed them itself it does so anyway. |
 
 Documents are named by their path under the folder, with forward slashes,
 on every system. With several folders, each folder's own name comes first.
@@ -259,13 +262,12 @@ scanned page with no OCR.
 
 ### What leaves your machine
 
-For each new or changed document, two things leave the machine, both over
-the gateway's TLS connection:
-
-- **The passages, to be embedded.** They go through `/v1/embeddings` like
-  any request of yours. Your key's limits, budget and guardrails apply.
-- **The document itself.** Its name, its file's SHA-256 digest and size,
-  and the passages with their vectors.
+For each new or changed document, one request leaves the machine, over the
+gateway's TLS connection: its name, its file's SHA-256 digest and size, and
+its passages' text. The gateway embeds the passages through `/v1/embeddings`
+as you, so your key's limits, budget and guardrails apply as if you had
+asked. That is about a kilobyte a passage; embedding here (`--embed-here`)
+brings each vector down and sends it up again, some thirty times more.
 
 The file itself is never sent. `--dry-run` shows exactly what would be.
 

docs/reference.md

@@ -192,6 +192,7 @@ rather than repeated.
 | `allow_pickle` | `bool` | [Model](#model) |
 | `profiles` | `[]Profile` | [Model](#model) |
 | `scaling` | `Scaling` | [Model](#model) |
+| `stopped` | `*Scaling` | [Model](#model) |
 | `class` | `string` | [Model](#model) |
 | `owner` | `string` | [Model](#model) |
 | `external` | `*External` | [Model](#model) |
@@ -1824,6 +1825,16 @@ no safetensors or GGUF form and whose files were checked.
 
 Takes a [Scaling](#scaling) block.
 
+### `stopped`
+
+`*Scaling`
+
+Stopped is the scaling a model had when an administrator stopped it:
+kept in the catalog with none running (Scaling 0/0) until started
+again with this. Nil for a model not stopped.
+
+Takes a [Scaling](#scaling) block.
+
 ### `class`
 
 `string`

internal/agent/agent.go

@@ -47,7 +47,9 @@ import (
 
 // Agent runs on a GPU node.
 type Agent struct {
-	cfg Config
+	// upgrading is true while an upgrade downloads.
+	upgrading atomic.Bool
+	cfg       Config
 	// answered says the current session heard from its control plane, which
 	// starts the reconnect backoff again when it ends.
 	answered atomic.Bool
@@ -878,6 +880,7 @@ func (a *Agent) removeWeights(paths []string) {
 			a.log.Warn("could not remove cached weights", "path", p, "err", err)
 			continue
 		}
+		_ = os.Remove(weights.ChecksumPath(abs))
 		a.log.Info("removed cached weights no model asks for", "path", p)
 	}
 }
@@ -1133,7 +1136,7 @@ func (a *Agent) startShard(ctx context.Context, m *llmgwv1.StartShard, out chan<
 			},
 		})
 	})
-	if path, err := a.weights().Ensure(ctx, spec.Profile.ModelPath, spec.Profile.AllowPickle); err != nil {
+	if path, err := a.weights().EnsureChecked(ctx, spec.Profile.ModelPath, spec.Profile.AllowPickle, spec.Profile.ModelSHA256); err != nil {
 		stop()
 		report(supervisor.Failed, err.Error(), nil)
 		return

internal/agent/engines/engine.go

@@ -50,6 +50,10 @@ type Profile struct {
 	// AllowPickle lets the node fetch weights in a pickle format, which run
 	// code when loaded; the catalog's allow_pickle.
 	AllowPickle bool `json:"allow_pickle,omitempty"`
+	// ModelSHA256 is the sha256 the registry published for the file
+	// ModelPath names; the node refuses a copy that is not it. Empty when
+	// nothing is known to check against.
+	ModelSHA256 string `json:"model_sha256,omitempty"`
 }
 
 // ShardSpec is one shard of one replica on this node.

internal/agent/engines/strata.go

@@ -1,10 +1,12 @@
 package engines
 
 import (
+	"context"
 	"fmt"
 	"net"
 	"os"
 	"path/filepath"
+	"regexp"
 	"strconv"
 	"sync/atomic"
 )
@@ -93,13 +95,42 @@ func (s *Strata) python() string {
 	return filepath.Join(s.cfg.Dir, ".venv", "bin", "python")
 }
 
+var strataVersion = regexp.MustCompile(`PRODUCT_VERSION\s*=\s*"([^"]+)"`)
+
+// Probe reports the Strata a configuration names, by the version its source
+// states: a directory with its setup and server, an interpreter, and the
+// engine binary. Without it a Strata named by hand was never reported, and
+// so never placed on.
+func (s *Strata) Probe(context.Context) (string, error) {
+	dir := s.dir()
+	if dir == "" {
+		return "", fmt.Errorf("no Strata directory is configured")
+	}
+	for _, f := range []string{filepath.Join(dir, "setup.py"), filepath.Join(dir, "serve", "server.py"),
+		s.python(), filepath.Join(s.engineDir(), "strata")} {
+		if _, err := os.Stat(f); err != nil {
+			return "", fmt.Errorf("%s is not a whole Strata: %w", dir, err)
+		}
+	}
+	b, err := os.ReadFile(filepath.Join(dir, "strata_cli", "__init__.py"))
+	if err != nil {
+		return Unknown, nil //nolint:nilerr // whole, of a version it does not say
+	}
+	if m := strataVersion.FindSubmatch(b); m != nil {
+		return string(m[1]), nil
+	}
+	return Unknown, nil
+}
+
 // Validate checks the profile: one card, on one node. Strata splits a model
 // over a node's cards by layer, but that is not driven from here yet.
 func (*Strata) Validate(p Profile, gpus []GPUInfo) error {
 	if p.Nodes > 1 || p.TP > 1 || p.PP > 1 || p.GPUsPerNode > 1 {
 		return fmt.Errorf("profile %q: strata runs on one GPU of one node", p.Name)
 	}
-	if err := requireComputeCapability(p, gpus, 7.0, "strata"); err != nil {
+	// Pascal and Volta run its CUDA 12 build, which the install picks for
+	// them.
+	if err := requireComputeCapability(p, gpus, 6.0, "strata"); err != nil {
 		return err
 	}
 	return validateCommon(p, gpus)

internal/agent/engines/strata_test.go

@@ -1,6 +1,7 @@
 package engines
 
 import (
+	"os"
 	"path/filepath"
 	"slices"
 	"strings"
@@ -67,3 +68,33 @@ func TestStrataSetupFlagsThatMoveFilesAreRefused(t *testing.T) {
 		t.Errorf("an ordinary profile was refused: %v", err)
 	}
 }
+
+func TestAStrataNamedByHandIsReportedWithItsVersion(t *testing.T) {
+	dir := t.TempDir()
+	for _, f := range []string{"setup.py", "serve/server.py", ".venv/bin/python", "engine/strata"} {
+		p := filepath.Join(dir, f)
+		if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
+			t.Fatal(err)
+		}
+		if err := os.WriteFile(p, nil, 0o755); err != nil {
+			t.Fatal(err)
+		}
+	}
+	s := &Strata{cfg: StrataConfig{Dir: dir}}
+	if v, err := s.Probe(t.Context()); err != nil || v != Unknown {
+		t.Fatalf("%q %v", v, err)
+	}
+	_ = os.MkdirAll(filepath.Join(dir, "strata_cli"), 0o755)
+	_ = os.WriteFile(filepath.Join(dir, "strata_cli", "__init__.py"), []byte("PRODUCT_VERSION = \"0.2.7\"\n"), 0o644)
+	if v, _ := s.Probe(t.Context()); v != "0.2.7" {
+		t.Errorf("version %q", v)
+	}
+	r := NewRegistry(Config{Strata: StrataConfig{Dir: dir}})
+	if got := r.ProbeConfigured(t.Context()); got["strata"] != "0.2.7" {
+		t.Errorf("not reported: %v", got)
+	}
+	_ = os.Remove(filepath.Join(dir, "engine", "strata"))
+	if _, err := s.Probe(t.Context()); err == nil {
+		t.Error("reported without its engine")
+	}
+}

internal/agent/supervisor/process_unix.go

@@ -4,6 +4,7 @@ import (
 	"bytes"
 	"os"
 	"strconv"
+	"strings"
 	"syscall"
 )
 
@@ -63,3 +64,71 @@ func isZombie(pid int) bool {
 	}
 	return b[i+2] == 'Z'
 }
+
+// strays are a process's descendants that left its group, as an engine that
+// starts its worker in a session of its own does (Strata): signalling the
+// group misses them, and one left behind holds the card and the RAM the next
+// replica needs. Found before the group is signalled, while they are still
+// its descendants: after, they belong to init. Linux only; elsewhere none.
+func strays(pid int) []int {
+	entries, err := os.ReadDir("/proc")
+	if err != nil {
+		return nil
+	}
+	parent := map[int]int{}
+	group := map[int]int{}
+	for _, e := range entries {
+		p, err := strconv.Atoi(e.Name())
+		if err != nil {
+			continue
+		}
+		if pp, pg, ok := parentAndGroup(p); ok {
+			parent[p], group[p] = pp, pg
+		}
+	}
+	var out []int
+	for p := range parent {
+		if group[p] == pid || p == pid {
+			continue
+		}
+		// Bounded: no engine's tree is this deep, and a pid reused while
+		// /proc was read must not loop.
+		for q, n := parent[p], 0; q > 1 && n < 32; q, n = parent[q], n+1 {
+			if q == pid {
+				out = append(out, p)
+				break
+			}
+		}
+	}
+	return out
+}
+
+// parentAndGroup reads a process's parent and group from /proc.
+func parentAndGroup(pid int) (int, int, bool) {
+	b, err := os.ReadFile("/proc/" + strconv.Itoa(pid) + "/stat")
+	if err != nil {
+		return 0, 0, false
+	}
+	i := bytes.LastIndexByte(b, ')')
+	if i < 0 {
+		return 0, 0, false
+	}
+	f := strings.Fields(string(b[i+1:]))
+	if len(f) < 3 {
+		return 0, 0, false
+	}
+	pp, err1 := strconv.Atoi(f[1])
+	pg, err2 := strconv.Atoi(f[2])
+	return pp, pg, err1 == nil && err2 == nil
+}
+
+// signalStrays signals each stray, and its own group when it leads one.
+func signalStrays(pids []int, sig syscall.Signal) {
+	for _, p := range pids {
+		if _, pg, ok := parentAndGroup(p); ok && pg == p {
+			_ = syscall.Kill(-p, sig)
+			continue
+		}
+		_ = syscall.Kill(p, sig)
+	}
+}

internal/agent/supervisor/supervisor.go

@@ -582,10 +582,24 @@ func (s *Supervisor) Stop(ctx context.Context, id string) (err error) {
 	}
 
 	// Signal the group, not the process: engines fork workers that hold GPU
-	// memory, and those outlive a signal sent only to the leader.
+	// memory, and those outlive a signal sent only to the leader. And what
+	// left the group, found while it is still the leader's.
+	left := strays(pid)
 	if err := killGroup(pid, syscall.SIGTERM); err != nil && !errors.Is(err, os.ErrProcessDone) {
 		s.log.Warn("SIGTERM failed", "shard", id, "err", err)
 	}
+	signalStrays(left, syscall.SIGTERM)
+	defer func() {
+		// Given as long as the group to go; then killed.
+		deadline := time.Now().Add(5 * time.Second)
+		for time.Now().Before(deadline) && slices.ContainsFunc(left, processAlive) {
+			time.Sleep(100 * time.Millisecond)
+		}
+		if slices.ContainsFunc(left, processAlive) {
+			s.log.Warn("killing an engine's processes that left its group", "shard", id, "pids", left)
+			signalStrays(left, syscall.SIGKILL)
+		}
+	}()
 
 	// force kills the group outright. exec.Cmd's cancel does this for shards we
 	// started; an adopted shard has none, so signal its group directly.
@@ -598,6 +612,7 @@ func (s *Supervisor) Stop(ctx context.Context, id string) (err error) {
 		if err := killGroup(pid, syscall.SIGKILL); err != nil && !errors.Is(err, os.ErrProcessDone) {
 			s.log.Warn("SIGKILL failed", "shard", id, "err", err)
 		}
+		signalStrays(left, syscall.SIGKILL)
 	}
 
 	select {

internal/agent/supervisor/supervisor_test.go

@@ -8,6 +8,7 @@ import (
 	"log/slog"
 	"net"
 	"os"
+	"os/exec"
 	"path/filepath"
 	"slices"
 	"strings"
@@ -981,3 +982,38 @@ func TestAnOutputFileIsEmptiedPastItsBound(t *testing.T) {
 		t.Errorf("a line written after the file was emptied was lost: %v", sh.Logs(8))
 	}
 }
+
+// An engine's worker in a session of its own, as Strata's is, is not in the
+// group a stop signals: it is found and stopped as well.
+func TestStopKillsAWorkerThatLeftTheGroup(t *testing.T) {
+	if _, err := os.Stat("/proc/self/stat"); err != nil {
+		t.Skip("needs /proc")
+	}
+	if _, err := exec.LookPath("setsid"); err != nil {
+		t.Skip("needs setsid")
+	}
+	s, _ := newSupervisor(t)
+	marker := filepath.Join(t.TempDir(), "worker.pid")
+	script := fmt.Sprintf(`setsid sh -c 'echo $$ > %s; sleep 30' & sleep 30`, marker)
+	if _, err := s.Start(t.Context(), "a", engines.ShardSpec{}, shellCommand(script)); err != nil {
+		t.Fatal(err)
+	}
+	var worker int
+	for deadline := time.Now().Add(3 * time.Second); time.Now().Before(deadline) && worker == 0; time.Sleep(10 * time.Millisecond) {
+		if b, err := os.ReadFile(marker); err == nil {
+			_, _ = fmt.Sscanf(strings.TrimSpace(string(b)), "%d", &worker)
+		}
+	}
+	if worker == 0 {
+		t.Fatal("the worker never started")
+	}
+	if err := s.Stop(t.Context(), "a"); err != nil {
+		t.Fatal(err)
+	}
+	for deadline := time.Now().Add(8 * time.Second); time.Now().Before(deadline); time.Sleep(50 * time.Millisecond) {
+		if !processAlive(worker) {
+			return
+		}
+	}
+	t.Errorf("the worker %d that left the group survived the stop", worker)
+}

internal/agent/upgrade.go

@@ -35,12 +35,25 @@ func (a *Agent) upgrade(ctx context.Context, m *llmgwv1.UpgradeAgent, out chan<-
 		select {
 		case out <- &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_Upgrade{Upgrade: reply}}:
 		case <-ctx.Done():
+		case <-time.After(5 * time.Second):
 		}
 	}
+	// One at a time: a second request while one downloads would race it to
+	// the same file.
+	if !a.upgrading.CompareAndSwap(false, true) {
+		reply.Error = "an upgrade is already under way on this node"
+		send()
+		return
+	}
+	defer a.upgrading.Store(false)
 	a.log.Info("upgrading", "to", m.GetVersion(), "from", a.cfg.Version)
 	self, err := upgradeTarget()
 	if err == nil {
-		dctx, cancel := context.WithTimeout(ctx, 10*time.Minute)
+		// Not the stream's context: a stream that breaks while the release
+		// downloads — a control plane restarting, leadership moving — cancelled
+		// the upgrade with nobody told. The node registering again on the
+		// version is what the control plane watches for.
+		dctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), upgradeDownload)
 		_, err = release.Install(dctx, nil, m.GetReleases(), m.GetVersion(), runtime.GOOS, runtime.GOARCH, self)
 		cancel()
 	}
@@ -53,6 +66,13 @@ func (a *Agent) upgrade(ctx context.Context, m *llmgwv1.UpgradeAgent, out chan<-
 	send()
 	a.log.Info("upgraded; restarting into the new version", "to", m.GetVersion())
 	// A moment for the reply to leave before the stream goes.
-	time.Sleep(500 * time.Millisecond)
+	time.Sleep(upgradeLinger)
 	exitForUpgrade()
 }
+
+// upgradeDownload bounds an upgrade's download; the control plane waits
+// longer (agentsvc.UpgradeWait). upgradeLinger is the moment its answer has
+// to leave before the agent exits; a variable so tests need not wait it.
+const upgradeDownload = 10 * time.Minute
+
+var upgradeLinger = 2 * time.Second

internal/agent/weights/fetch.go

@@ -94,6 +94,13 @@ var ErrNotLocal = errors.New("the weights are not on this node and the source sa
 // A source in a pickle format, or a set with such a file in it, is refused
 // before anything is fetched unless allowPickle; see Pickled.
 func (f *Fetcher) Ensure(ctx context.Context, source string, allowPickle bool) (path string, err error) {
+	return f.EnsureChecked(ctx, source, allowPickle, "")
+}
+
+// EnsureChecked is Ensure for a file whose sha256 the registry published: a
+// copy that is not it, downloaded or cached, is refused rather than served.
+// An empty sha256 checks nothing.
+func (f *Fetcher) EnsureChecked(ctx context.Context, source string, allowPickle bool, sha256 string) (path string, err error) {
 	ctx, span := obs.Tracer(tracerName).Start(ctx, "fetch weights",
 		trace.WithAttributes(attribute.String("llmgw.weights.source", redact(source))))
 	defer func() {
@@ -103,7 +110,7 @@ func (f *Fetcher) Ensure(ctx context.Context, source string, allowPickle bool) (
 	if !allowPickle && !IsSet(source) && Pickled(source) {
 		return "", refusePickle(source)
 	}
-	return f.ensure(ctx, source, allowPickle)
+	return f.ensure(ctx, source, allowPickle, strings.ToLower(sha256))
 }
 
 // tracerName names the spans this package opens.
@@ -125,7 +132,7 @@ func redact(source string) string {
 	return u.String()
 }
 
-func (f *Fetcher) ensure(ctx context.Context, source string, allowPickle bool) (string, error) {
+func (f *Fetcher) ensure(ctx context.Context, source string, allowPickle bool, want string) (string, error) {
 	if source == "" {
 		return "", fmt.Errorf("no source given")
 	}
@@ -156,8 +163,13 @@ func (f *Fetcher) ensure(ctx context.Context, source string, allowPickle bool) (
 		return "", err
 	}
 	if _, err := os.Stat(target); err == nil {
-		outcome(ctx, "cached")
-		return target, nil
+		if ok, err := f.cachedIs(target, want); err != nil {
+			return "", err
+		} else if ok {
+			outcome(ctx, "cached")
+			return target, nil
+		}
+		// Not the file the catalog names: fetched again below, and checked.
 	}
 
 	f.mu.Lock()
@@ -179,7 +191,7 @@ func (f *Fetcher) ensure(ctx context.Context, source string, allowPickle bool) (
 	f.mu.Unlock()
 
 	outcome(ctx, "downloaded")
-	call.path, call.err = f.fetch(ctx, source, target)
+	call.path, call.err = f.fetchChecked(ctx, source, target, want)
 	close(call.done)
 
 	f.mu.Lock()
@@ -193,6 +205,80 @@ func (f *Fetcher) fetch(ctx context.Context, source, target string) (string, err
 	return f.fetchAs(ctx, source, source, target)
 }
 
+// fetchChecked is fetch for a file whose sha256 is known: a download that
+// is not it is deleted and refused, naming both, and one that is has its
+// checksum kept beside it so a cached copy is not read again at each start.
+func (f *Fetcher) fetchChecked(ctx context.Context, source, target, want string) (string, error) {
+	got, err := f.fetch(ctx, source, target)
+	if err != nil || want == "" {
+		return got, err
+	}
+	sum, err := fileSHA256(target)
+	if err != nil {
+		return "", err
+	}
+	if sum != want {
+		_ = os.Remove(target)
+		return "", fmt.Errorf("%s arrived with sha256 %s, and the registry published %s for it: refused and deleted",
+			redact(source), sum, want)
+	}
+	_ = writeChecksum(target, sum)
+	return target, nil
+}
+
+// cachedIs says whether a cached file is the one whose sha256 is want: by
+// the checksum kept beside it, or worked out once and then kept. One that
+// is not is deleted, to be fetched again. Any file is, when want is empty.
+func (f *Fetcher) cachedIs(target, want string) (bool, error) {
+	kept := readChecksum(target)
+	if want == "" || kept == want {
+		return true, nil
+	}
+	if kept != "" {
+		// Checked when it was fetched, as another file: the same address
+		// gives the same bytes again, and another model may be serving
+		// these. Refused, and left.
+		return false, fmt.Errorf("the copy of this file here has sha256 %s, as fetched, and the catalog names %s: refused",
+			kept, want)
+	}
+	sum, err := fileSHA256(target)
+	if err != nil {
+		return false, err
+	}
+	if sum == want {
+		_ = writeChecksum(target, sum)
+		return true, nil
+	}
+	if f.Log != nil {
+		f.Log.Warn("a cached file is not the one the catalog names; fetching it again",
+			"path", target, "sha256", sum, "want", want)
+	}
+	_ = os.Remove(checksumPath(target))
+	return false, os.Remove(target)
+}
+
+// checksumPath is where a cached file's sha256 is kept: beside it, hidden,
+// so no scan takes it for weights.
+func checksumPath(file string) string {
+	return filepath.Join(filepath.Dir(file), "."+filepath.Base(file)+".sha256")
+}
+
+func writeChecksum(file, sum string) error {
+	return os.WriteFile(checksumPath(file), []byte(sum+"\n"), 0o644) //nolint:gosec // a checksum, for anyone to read
+}
+
+func readChecksum(file string) string {
+	b, err := os.ReadFile(checksumPath(file)) //nolint:gosec // beside a file this fetcher wrote
+	if err != nil {
+		return ""
+	}
+	return strings.TrimSpace(string(b))
+}
+
+// ChecksumPath is where the sha256 of a cached file is kept, for whoever
+// deletes the file to delete with it.
+func ChecksumPath(file string) string { return checksumPath(file) }
+
 // fetchAs downloads one URL but reports progress under another name.
 //
 // The two differ for a set: the shards waiting on it asked for the version, not

internal/agent/weights/fetch_test.go

@@ -1,6 +1,8 @@
 package weights
 
 import (
+	"crypto/sha256"
+	"encoding/hex"
 	"errors"
 	"net/http"
 	"net/http/httptest"
@@ -287,3 +289,75 @@ func TestAnAbsolutePathThatIsMissingIsStillAnError(t *testing.T) {
 		t.Errorf("err = %v, want ErrNotLocal", err)
 	}
 }
+
+// A file whose sha256 the registry published is served only as that file:
+// changed upstream after the model was added, it is refused with both
+// checksums named, and a cached copy that is not it is fetched again.
+func TestAFileThatIsNotWhatTheRegistryPublishedIsRefused(t *testing.T) {
+	body := "the weights as published"
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		_, _ = w.Write([]byte(body))
+	}))
+	defer srv.Close()
+	sum := sha256.Sum256([]byte(body))
+	want := hex.EncodeToString(sum[:])
+	f := &Fetcher{CacheDir: t.TempDir()}
+	src := srv.URL + "/model.gguf"
+
+	path, err := f.EnsureChecked(t.Context(), src, false, strings.ToUpper(want))
+	if err != nil {
+		t.Fatal(err)
+	}
+	if readChecksum(path) != want {
+		t.Errorf("the checksum was not kept beside it: %q", readChecksum(path))
+	}
+
+	// The cached copy changed under it: fetched again, and the right one kept.
+	if err := os.WriteFile(path, []byte("tampered"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	_ = os.Remove(checksumPath(path))
+	if _, err := f.EnsureChecked(t.Context(), src, false, want); err != nil {
+		t.Fatal(err)
+	}
+	if b, _ := os.ReadFile(path); string(b) != body {
+		t.Errorf("the cached copy that was not it was served: %q", b)
+	}
+
+	// Upstream changed after the model was added.
+	body = "weights someone replaced"
+	f2 := &Fetcher{CacheDir: t.TempDir()}
+	_, err = f2.EnsureChecked(t.Context(), src, false, want)
+	if err == nil || !strings.Contains(err.Error(), want) || !strings.Contains(err.Error(), "sha256") {
+		t.Fatalf("not refused, or without both checksums: %v", err)
+	}
+	if _, err := os.Stat(CachePathFor(f2.CacheDir, src)); !os.IsNotExist(err) {
+		t.Error("the file that was not it was kept")
+	}
+	// With nothing to check against, it is fetched as before.
+	if _, err := f2.Ensure(t.Context(), src, false); err != nil {
+		t.Error(err)
+	}
+}
+
+// A cached file checked when it was fetched, that another model names with
+// another checksum, is refused and left for the model it is.
+func TestACheckedFileAnotherChecksumNamesIsRefusedAndLeft(t *testing.T) {
+	body := "the weights"
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { _, _ = w.Write([]byte(body)) }))
+	defer srv.Close()
+	sum := sha256.Sum256([]byte(body))
+	f := &Fetcher{CacheDir: t.TempDir()}
+	src := srv.URL + "/m.gguf"
+	path, err := f.EnsureChecked(t.Context(), src, false, hex.EncodeToString(sum[:]))
+	if err != nil {
+		t.Fatal(err)
+	}
+	wrong := strings.Repeat("0", 64)
+	if _, err := f.EnsureChecked(t.Context(), src, false, wrong); err == nil || !strings.Contains(err.Error(), wrong) {
+		t.Fatalf("not refused: %v", err)
+	}
+	if b, err := os.ReadFile(path); err != nil || string(b) != body {
+		t.Errorf("the checked file was not left: %v", err)
+	}
+}

internal/cli/ctl.go

@@ -321,6 +321,8 @@ func locationColumn(n adminapi.NodeView) string {
 // actually served, never what is merely catalogued.
 func modelStatus(m adminapi.ModelView) string {
 	switch {
+	case m.Stopped:
+		return "stopped (llmgw ctl model start)"
 	case m.Parked:
 		return "parked"
 	case m.External != "":

internal/cli/ctl_upgrade.go

@@ -11,11 +11,13 @@ import (
 	"runtime"
 	"slices"
 	"strings"
+	"sync"
 	"time"
 
 	"github.com/spf13/cobra"
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/agentsvc"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/release"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/version"
@@ -244,6 +246,7 @@ func upgradeNodes(ctx context.Context, out io.Writer, client func() *ctlClient,
 	}
 	slices.SortFunc(nodes, func(a, b adminapi.NodeView) int { return strings.Compare(a.ID, b.ID) })
 	var todo []adminapi.NodeView
+	var away []string
 	skipped := 0
 	for _, n := range nodes {
 		if len(only) > 0 && !slices.Contains(only, n.ID) {
@@ -253,8 +256,17 @@ func upgradeNodes(ctx context.Context, out io.Writer, client func() *ctlClient,
 			skipped++
 			continue
 		}
+		// A node that is not connected cannot be told, and one tried first
+		// stopped every run at it: passed over, and named at the end.
+		if n.State == "lost" || n.State == "unreachable" {
+			away = append(away, n.ID)
+			continue
+		}
 		todo = append(todo, n)
 	}
+	if len(away) > 0 {
+		fmt.Fprintf(out, "Passed over, not connected: %s\n", strings.Join(away, ", "))
+	}
 	parallel = max(parallel, 1)
 	start, size, done := time.Now(), 1, 0
 	for len(todo) > 0 {
@@ -296,7 +308,10 @@ func upgradeNodes(ctx context.Context, out io.Writer, client func() *ctlClient,
 }
 
 // upgradeWave tells every node in the wave at once, and waits until each is
-// back on v, ready, with as many ready replicas as it had.
+// back on v and serving what it served.
+//
+// Told together, not one after another: each answer can take minutes, and a
+// wave of twenty waited for them in turn with nothing on the screen.
 func upgradeWave(ctx context.Context, out io.Writer, client func() *ctlClient, v string,
 	wave []adminapi.NodeView, perNode time.Duration,
 ) error {
@@ -308,35 +323,61 @@ func upgradeWave(ctx context.Context, out io.Writer, client func() *ctlClient, v
 	}
 	fmt.Fprintf(out, "== %d node(s): %s\n", len(wave), abbreviate(ids, 8))
 	body, _ := json.Marshal(adminapi.UpgradeRequest{Version: v})
-	var failed []string
+	var (
+		mu     sync.Mutex
+		failed []string
+		wg     sync.WaitGroup
+	)
 	for _, n := range wave {
-		// The control plane waits for the agent to fetch and check the
-		// release, up to five minutes; fifteen seconds gave up on a node
-		// that was upgrading.
-		if err := client().patient(6*time.Minute).post(ctx, "/api/v1/nodes/"+n.ID+"/upgrade", body, nil); err != nil {
-			failed = append(failed, fmt.Sprintf("%s: %v", n.ID, err))
-		}
+		wg.Add(1)
+		go func() {
+			defer wg.Done()
+			// The control plane waits for the agent's answer, or for it to
+			// come back on v, longer than the agent gives its download.
+			err := client().patient(agentsvc.UpgradeWait+time.Minute).post(ctx, "/api/v1/nodes/"+n.ID+"/upgrade", body, nil)
+			mu.Lock()
+			defer mu.Unlock()
+			if err != nil {
+				failed = append(failed, fmt.Sprintf("%s: %v", n.ID, err))
+				fmt.Fprintf(out, "   %s: not upgraded: %v\n", n.ID, err)
+				return
+			}
+			fmt.Fprintf(out, "   %s: on %s, coming back\n", n.ID, v)
+		}()
 	}
-	if len(failed) > 0 {
-		return fmt.Errorf("not told to upgrade:\n  %s\n`llmgw ctl logs NODE` says more", strings.Join(failed, "\n  "))
-	}
-	waiting := slices.Clone(ids)
+	wg.Wait()
+	waiting := slices.DeleteFunc(slices.Clone(ids), func(id string) bool {
+		return slices.ContainsFunc(failed, func(f string) bool { return strings.HasPrefix(f, id+": ") })
+	})
 	deadline := time.Now().Add(perNode)
-	for {
+	backAt := map[string]time.Time{}
+	said := time.Now()
+	for len(waiting) > 0 {
 		var now []adminapi.NodeView
 		_ = client().get(ctx, "/api/v1/nodes", &now)
-		after := readyReplicasByNode(ctx, client)
+		var reps []adminapi.ReplicaView
+		_ = client().get(ctx, "/api/v1/replicas", &reps)
+		after, starting := replicasByNode(reps)
 		state := map[string]adminapi.NodeView{}
 		for _, x := range now {
 			state[x.ID] = x
 		}
 		waiting = slices.DeleteFunc(waiting, func(id string) bool {
 			cur := state[id]
-			return cur.AgentVersion == v && cur.State == "ready" && after[id] >= before[id]
+			// Connected and on v: ready, or cordoned or draining as it
+			// was, which an upgrade does not undo.
+			if cur.AgentVersion != v || (cur.State != "ready" && cur.State != "cordoned" && cur.State != "draining") {
+				return false
+			}
+			if _, ok := backAt[id]; !ok {
+				backAt[id] = time.Now()
+			}
+			// Serving what it served; or, a while on, nothing on it still
+			// starting: a model that went idle meanwhile is not one lost.
+			return after[id] >= before[id] || (time.Since(backAt[id]) > 90*time.Second && starting[id] == 0)
 		})
 		if len(waiting) == 0 {
-			fmt.Fprintf(out, "   done in %s\n", time.Since(start).Round(time.Second))
-			return nil
+			break
 		}
 		if time.Now().After(deadline) {
 			var why []string
@@ -345,11 +386,49 @@ func upgradeWave(ctx context.Context, out io.Writer, client func() *ctlClient, v
 				why = append(why, fmt.Sprintf("%s (version %q, state %q, replicas %d/%d)",
 					id, cur.AgentVersion, cur.State, after[id], before[id]))
 			}
-			return fmt.Errorf("not back within %s:\n  %s\n`llmgw ctl logs NODE` says why, and each keeps its "+
-				"previous binary beside the new one", perNode, strings.Join(why, "\n  "))
+			failed = append(failed, fmt.Sprintf("not back within %s: %s", perNode, strings.Join(why, ", ")))
+			break
+		}
+		if time.Since(said) >= 30*time.Second {
+			var w []string
+			for _, id := range waiting {
+				cur := state[id]
+				w = append(w, fmt.Sprintf("%s (%s, %s, replicas %d/%d)", id, orUnknown(cur.AgentVersion), cur.State,
+					after[id], before[id]))
+			}
+			fmt.Fprintf(out, "   waiting for %s\n", abbreviate(w, 8))
+			said = time.Now()
 		}
 		time.Sleep(2 * time.Second)
 	}
+	if len(failed) > 0 {
+		return fmt.Errorf("not upgraded:\n  %s\n`llmgw ctl logs NODE` says why; each keeps its previous binary beside the new one",
+			strings.Join(failed, "\n  "))
+	}
+	fmt.Fprintf(out, "   done in %s\n", time.Since(start).Round(time.Second))
+	return nil
+}
+
+// replicasByNode counts each node's ready replicas, and those still placing
+// or starting.
+func replicasByNode(reps []adminapi.ReplicaView) (ready, starting map[string]int) {
+	ready, starting = map[string]int{}, map[string]int{}
+	for _, r := range reps {
+		seen := map[string]bool{}
+		for _, sh := range r.Shards {
+			if seen[sh.Node] {
+				continue
+			}
+			seen[sh.Node] = true
+			switch r.State {
+			case "ready":
+				ready[sh.Node]++
+			case "placing", "starting":
+				starting[sh.Node]++
+			}
+		}
+	}
+	return ready, starting
 }
 
 // abbreviate lists up to n names, and says how many more.

internal/cli/ctl_upgrade_test.go

@@ -6,6 +6,7 @@ import (
 	"fmt"
 	"net/http"
 	"net/http/httptest"
+	"slices"
 	"strings"
 	"sync"
 	"testing"
@@ -21,6 +22,8 @@ type fakeFleet struct {
 	versions map[string]string
 	stuck    map[string]bool
 	told     []string
+	// states are nodes' states other than ready.
+	states map[string]string
 }
 
 func (f *fakeFleet) server(t *testing.T) *httptest.Server {
@@ -31,7 +34,11 @@ func (f *fakeFleet) server(t *testing.T) *httptest.Server {
 		case r.URL.Path == "/api/v1/nodes":
 			var out []adminapi.NodeView
 			for id, v := range f.versions {
-				out = append(out, adminapi.NodeView{ID: id, State: "ready", AgentVersion: v})
+				st := "ready"
+				if f.states[id] != "" {
+					st = f.states[id]
+				}
+				out = append(out, adminapi.NodeView{ID: id, State: st, AgentVersion: v})
 			}
 			_ = json.NewEncoder(w).Encode(out)
 		case r.URL.Path == "/api/v1/replicas":
@@ -201,3 +208,23 @@ func TestUpgradingAControlPlaneAloneSaysItsNodesWereNot(t *testing.T) {
 		t.Errorf("said something with every node on the version: %s", out.String())
 	}
 }
+
+// A cordoned node upgraded is done, as it was; a node not connected is passed
+// over rather than stopping every node after it.
+func TestAnUpgradeTakesACordonedNodeAndPassesOverOneAway(t *testing.T) {
+	f := &fakeFleet{versions: map[string]string{"a": "v1.0.0", "b": "v1.0.0", "c": "v1.0.0"},
+		states: map[string]string{"a": "lost", "b": "cordoned"}}
+	srv := f.server(t)
+	var out bytes.Buffer
+	err := upgradeNodes(t.Context(), &out, func() *ctlClient { return newCtlClient(srv.URL) }, "v1.0.1", nil, 5*time.Second, 2)
+	if slices.Contains(f.told, "a") || !slices.Contains(f.told, "b") || !slices.Contains(f.told, "c") {
+		t.Errorf("told %v", f.told)
+	}
+	if !strings.Contains(out.String(), "Passed over, not connected: a") {
+		t.Errorf("output:\n%s", out.String())
+	}
+	// The one away is named at the end, behind.
+	if err == nil || !strings.Contains(err.Error(), "a (v1.0.0, lost)") {
+		t.Errorf("err %v", err)
+	}
+}

internal/cli/leaderforward.go

@@ -73,8 +73,10 @@ func (f *leaderForward) standby(next http.Handler) http.Handler {
 		p := &httputil.ReverseProxy{
 			Rewrite: func(pr *httputil.ProxyRequest) {
 				pr.Out.URL.Scheme, pr.Out.URL.Host = "https", target.Host
+				// Escaped as it came: a name with a slash in it, Demo/hr
+				// as Demo%2Fhr, is one segment, not two.
 				pr.Out.URL.Path = forwardPrefix + pr.In.URL.Path
-				pr.Out.URL.RawPath = ""
+				pr.Out.URL.RawPath = forwardPrefix + pr.In.URL.EscapedPath()
 				pr.Out.Host = target.Host
 				// The leader's own token is not this one's, and the
 				// certificate is what it believes.
@@ -221,7 +223,7 @@ func (f *leaderForward) leaderSide(admin http.Handler) http.Handler {
 			return
 		}
 		r2 := r.Clone(r.Context())
-		r2.URL.Path, r2.URL.RawPath = path, ""
+		r2.URL.Path, r2.URL.RawPath = path, strings.TrimPrefix(r.URL.EscapedPath(), forwardPrefix)
 		w.Header().Set(hdrAnsweredBy, f.self)
 		admin.ServeHTTP(w, r2)
 	})

internal/cli/leaderforward_test.go

@@ -77,7 +77,7 @@ func newForwardCluster(t *testing.T) *forwardCluster {
 
 	// The leader's admin API: says whose it is, and who it was asked by.
 	admin := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
-		_ = json.NewEncoder(w).Encode(map[string]string{"path": r.URL.Path, "by": r.Header.Get(hdrForwardedBy),
+		_ = json.NewEncoder(w).Encode(map[string]string{"path": r.URL.Path, "raw": r.URL.EscapedPath(), "by": r.Header.Get(hdrForwardedBy),
 			"auth": r.Header.Get("Authorization"), "actor": r.Header.Get("X-Llmgw-Actor")})
 	})
 	mux := http.NewServeMux()
@@ -242,3 +242,13 @@ func TestAStandbysPortalIsTheLeaders(t *testing.T) {
 		t.Errorf("a forwarded request was forwarded again: %q", body)
 	}
 }
+
+// A name with a slash in it, escaped, stays one segment on the leader: a
+// collection Demo/hr removed through a standby was a 405 there.
+func TestAnEscapedSlashReachesTheLeaderEscaped(t *testing.T) {
+	c := newForwardCluster(t)
+	resp, got := c.ask(t, "/api/v1/kb/collections/Demo%2Fhr")
+	if resp.StatusCode != http.StatusOK || got["raw"] != "/api/v1/kb/collections/Demo%2Fhr" {
+		t.Fatalf("%d %v", resp.StatusCode, got)
+	}
+}

internal/cli/mirror.go

@@ -16,6 +16,7 @@ import (
 	"github.com/spf13/cobra"
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
 )
@@ -57,7 +58,7 @@ func newMirrorCmd() *cobra.Command {
 }
 
 func newMirrorAddCmd(dir *string) *cobra.Command {
-	var model, version, as string
+	var model, version, as, sha, registry string
 
 	c := &cobra.Command{
 		Use:   "add <url>...",
@@ -79,7 +80,7 @@ func newMirrorAddCmd(dir *string) *cobra.Command {
 			if model == "" || version == "" {
 				return fmt.Errorf("--model and --version are required: they are the layout")
 			}
-			if as != "" && len(args) > 1 {
+			if (as != "" || sha != "") && len(args) > 1 {
 				return fmt.Errorf("--as names one file; with %d URLs each keeps its own name", len(args))
 			}
 			out := cmd.OutOrStdout()
@@ -90,6 +91,21 @@ func newMirrorAddCmd(dir *string) *cobra.Command {
 					file = mirror.LastPathElement(source)
 				}
 				e := mirror.Entry{Model: model, Version: version, File: file}
+				// From the registry: what it published for the file, to
+				// compare with what arrives.
+				want := strings.ToLower(sha)
+				if id, rev, path, ok := hf.FileSource(source, registry); ok && want == "" {
+					repo, err := hf.New(registry, 30*time.Second).RepoAt(cmd.Context(), id, rev)
+					if err != nil {
+						return fmt.Errorf("looking up %s in the registry, for the sha256 it published: %w "+
+							"(--sha256 gives it instead)", id, err)
+					}
+					for _, f := range repo.Files {
+						if f.Path == path {
+							want = f.SHA256
+						}
+					}
+				}
 				fmt.Fprintf(out, "fetching %s\n", source)
 				added, err := mirror.Add(cmd.Context(), *dir, source, e, func(done, total int64) {
 					if total > 0 {
@@ -102,6 +118,14 @@ func newMirrorAddCmd(dir *string) *cobra.Command {
 				if err != nil {
 					return err
 				}
+				if want != "" && added.SHA256 != want {
+					_ = mirror.Remove(*dir, e)
+					return fmt.Errorf("%s arrived with sha256 %s, and %s was published for it: deleted",
+						source, added.SHA256, want)
+				}
+				if want != "" {
+					fmt.Fprintln(out, "  sha256 matches the one published")
+				}
 				fmt.Fprintf(out, "%s/%s/%s  %s\n  sha256 %s\n",
 					added.Entry.Model, added.Entry.Version, added.Entry.File,
 					fileSize(added.Entry.Bytes), added.SHA256)
@@ -123,6 +147,8 @@ func newMirrorAddCmd(dir *string) *cobra.Command {
 	c.Flags().StringVar(&model, "model", "", "the model's name, one directory level")
 	c.Flags().StringVar(&version, "version", "", "the version, one directory level; a date works well")
 	c.Flags().StringVar(&as, "as", "", "the filename to save as, for one URL; taken from the URL when omitted")
+	c.Flags().StringVar(&sha, "sha256", "", "the sha256 the file must have, for one URL; looked up for a Hugging Face URL")
+	c.Flags().StringVar(&registry, "registry", hf.DefaultEndpoint, "the registry a resolve URL is looked up in")
 	return c
 }
 
@@ -238,6 +264,10 @@ func newMirrorRemoveCmd(dir *string) *cobra.Command {
 			if err := mirror.Remove(*dir, e); err != nil {
 				return err
 			}
+			// Kept for a mirror kept in step, which otherwise copies it back.
+			if err := mirror.RecordRemoval(*dir, model, version); err != nil {
+				return err
+			}
 			fmt.Fprintf(cmd.OutOrStdout(), "removed %s/%s/%s\n", model, version, file)
 			return nil
 		},

internal/cli/mirror_peer.go

@@ -34,6 +34,8 @@ step with this one: each copies what the other holds, checksum checked, and
 a model removed from one is removed from the other. This mirror is told of
 it, and the cluster's settings name it (mirror.peers), so while either
 machine is down the control planes use the other and nodes fetch from it.
+On a mirror machine without the control plane's settings file, it prints
+the llmgw ctl config set to run on a control plane instead.
 
 HOST needs room for every model this mirror holds: it copies them all.`,
 		Example: "  sudo llmgw mirror add-peer ubuntu@10.0.0.12 --control-plane 10.0.0.10 --control-plane 10.0.0.12",
@@ -123,6 +125,19 @@ func addMirrorPeer(ctx context.Context, out io.Writer, remote onboard.Remote, ho
 	fmt.Fprintf(out, "  ok    this mirror keeps in step with %s\n", peerAPI)
 
 	// The control planes told of both.
+	return nameMirrorPeer(out, configPath, peerAPI)
+}
+
+// nameMirrorPeer names a peer in the cluster's settings. A mirror on a
+// machine of its own has neither the settings file nor the store: it says
+// what to run on a control plane.
+func nameMirrorPeer(out io.Writer, configPath, peerAPI string) error {
+	if _, err := os.Stat(configPath); errors.Is(err, os.ErrNotExist) {
+		fmt.Fprintf(out, "  !!    this machine has no %s, so the cluster's settings are not changed here. On a control plane:\n"+
+			"          sudo llmgw ctl config set mirror.peers '[%s]'\n"+
+			"        (with any peers already named, which llmgw ctl config get mirror.peers shows)\n", configPath, peerAPI)
+		return nil
+	}
 	if err := addMirrorPeerSetting(configPath, peerAPI); err != nil {
 		return err
 	}

internal/cli/mirror_peer_test.go

@@ -1,6 +1,7 @@
 package cli
 
 import (
+	"bytes"
 	"os"
 	"path/filepath"
 	"strings"
@@ -72,3 +73,13 @@ func TestTheSetupOffersASecondMirrorOnAStoreMachine(t *testing.T) {
 		t.Errorf("without standbys: %v", err)
 	}
 }
+
+func TestAMirrorOnAMachineOfItsOwnSaysWhatToRunOnAControlPlane(t *testing.T) {
+	var out bytes.Buffer
+	if err := nameMirrorPeer(&out, filepath.Join(t.TempDir(), "server.yaml"), "http://10.0.0.12:8082"); err != nil {
+		t.Fatal(err)
+	}
+	if !strings.Contains(out.String(), "llmgw ctl config set mirror.peers '[http://10.0.0.12:8082]'") {
+		t.Errorf("%s", out.String())
+	}
+}

internal/cli/model.go

@@ -34,7 +34,7 @@ func newModelCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 		Use:   "model",
 		Short: "Add or remove models while the cluster is running",
 	}
-	c.AddCommand(newModelContextCmd(client), newModelAddCmd(client), newModelPresetsCmd(client, asJSON), newModelRemoveCmd(client), newModelPlanCmd(client),
+	c.AddCommand(newModelStopCmd(client), newModelStartCmd(client), newModelContextCmd(client), newModelGPUsCmd(client), newModelAddCmd(client), newModelPresetsCmd(client, asJSON), newModelRemoveCmd(client), newModelPlanCmd(client),
 		newResourcesCmd(client), newClassCmd(client), newOwnerCmd(client),
 		newTargetCmd(client, asJSON), newAdaptersCmd(client),
 		newModelServeCmd(client), newModelUpdatesCmd(client, asJSON), newModelUpdateCmd(client),
@@ -163,21 +163,104 @@ func readModelFile(path string) (catalog.Model, error) {
 }
 
 func newModelRemoveCmd(client func() *ctlClient) *cobra.Command {
-	return &cobra.Command{
+	var purge bool
+	c := &cobra.Command{
 		Use:   "remove <name>",
-		Short: "Remove a model that was added at runtime",
+		Short: "Remove a model that was added at runtime, and with --purge its files",
 		Long: "Only models added at runtime. One declared in server.yaml would come\n" +
 			"back on the next restart, so removing it here would leave the cluster\n" +
 			"disagreeing with its own configuration until then.\n\n" +
 			"Its replicas are stopped on the next pass, after the endpoint has been\n" +
-			"withdrawn, so traffic stops before the engine does.",
+			"withdrawn, so traffic stops before the engine does.\n\n" +
+			"Its files stay on the mirror and the nodes, for adding it again. With\n" +
+			"--purge they go too: every version of it on the mirror (a mirror kept in\n" +
+			"step follows), and the nodes' copies once its replicas have stopped.\n" +
+			"What another model uses stays. To take a model out of service and keep\n" +
+			"it, use llmgw ctl model stop.",
+		Example: "  llmgw ctl model remove qwen-7b\n" +
+			"  llmgw ctl model remove qwen-7b --purge   # its files on the mirror and the nodes too",
 		Args: cobra.ExactArgs(1),
 		RunE: func(cmd *cobra.Command, args []string) error {
-			var res map[string]string
-			if err := client().delete(cmd.Context(), "/api/v1/models/"+args[0], &res); err != nil {
+			out := cmd.OutOrStdout()
+			if !purge {
+				var res map[string]string
+				if err := client().delete(cmd.Context(), "/api/v1/models/"+args[0], &res); err != nil {
+					return err
+				}
+				fmt.Fprintf(out, "%s removed. Its files stay on the mirror and the nodes; --purge deletes them.\n", args[0])
+				return nil
+			}
+			var v adminapi.PurgeView
+			if err := client().patient(3*time.Minute).delete(cmd.Context(), "/api/v1/models/"+args[0]+"?purge=true", &v); err != nil {
 				return err
 			}
-			fmt.Fprintf(cmd.OutOrStdout(), "%s removed.\n", args[0])
+			fmt.Fprintf(out, "%s removed.\n", v.Removed)
+			for _, m := range v.Mirror {
+				fmt.Fprintf(out, "  mirror  %s deleted\n", m)
+			}
+			for _, m := range v.Kept {
+				fmt.Fprintf(out, "  mirror  %s kept: another model uses it\n", m)
+			}
+			var freed int64
+			for _, n := range v.Nodes {
+				if n.Error != "" {
+					fmt.Fprintf(out, "  node    %s: %s\n", n.Node, n.Error)
+					continue
+				}
+				freed += n.Bytes
+				fmt.Fprintf(out, "  node    %s: %d file(s), %s\n", n.Node, n.Files, humanBytes(n.Bytes))
+			}
+			if len(v.Mirror) == 0 && len(v.Nodes) == 0 && v.Note == "" {
+				fmt.Fprintln(out, "  it had no files of its own on the mirror or the nodes")
+			}
+			if freed > 0 {
+				fmt.Fprintf(out, "  %s freed on the nodes\n", humanBytes(freed))
+			}
+			if v.Note != "" {
+				fmt.Fprintln(out, "  "+v.Note)
+			}
+			return nil
+		},
+	}
+	c.Flags().BoolVar(&purge, "purge", false, "delete its files from the mirror and the nodes as well")
+	return c
+}
+
+func newModelStopCmd(client func() *ctlClient) *cobra.Command {
+	return &cobra.Command{
+		Use:   "stop <model>",
+		Short: "Take a model out of service and keep it: none of its replicas run until it is started",
+		Long: "Its replicas stop; it stays in the catalog with its files, and a request\n" +
+			"for it is told an administrator stopped it. What it was scaled to is kept:\n" +
+			"llmgw ctl model start runs it again as it was.",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			var v adminapi.ScaleResponse
+			if err := client().post(cmd.Context(), "/api/v1/models/"+args[0]+"/stop", nil, &v); err != nil {
+				return err
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "%s stopped: its replicas stop on the next pass. It is kept; "+
+				"llmgw ctl model start %s runs it again.\n", v.Model, v.Model)
+			return nil
+		},
+	}
+}
+
+func newModelStartCmd(client func() *ctlClient) *cobra.Command {
+	return &cobra.Command{
+		Use:   "start <model>",
+		Short: "Run a stopped model again, scaled as it was when it was stopped",
+		Args:  cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			var v adminapi.ScaleResponse
+			if err := client().post(cmd.Context(), "/api/v1/models/"+args[0]+"/start", nil, &v); err != nil {
+				return err
+			}
+			how := fmt.Sprintf("%d replica(s)", v.Min)
+			if v.OnDemand {
+				how = "on demand"
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "%s started: %s, max %d. Watch it with llmgw ctl models.\n", v.Model, how, v.Max)
 			return nil
 		},
 	}
@@ -442,3 +525,57 @@ func newModelContextCmd(client func() *ctlClient) *cobra.Command {
 	c.Flags().IntVar(&slots, "slots", 0, "llama.cpp: requests a replica answers at once (default: as it is)")
 	return c
 }
+
+func newModelGPUsCmd(client func() *ctlClient) *cobra.Command {
+	var profile string
+	var inPlace bool
+	c := &cobra.Command{
+		Use:   "gpus <model> <cards>",
+		Short: "Spread a served model over more cards, or fewer",
+		Long: "How many cards of one node each replica uses. vLLM, SGLang and TensorRT-LLM\n" +
+			"spread the model by tensor parallel, which needs the count to divide the\n" +
+			"model's KV heads or be divided by them; llama.cpp splits its layers over the\n" +
+			"cards. It refuses a count a node does not have, and one whose cards would not\n" +
+			"hold the model and its context, saying what each would need. Its replicas are\n" +
+			"then started again on it, each new one beside the old before it stops.\n\n" +
+			"With no node free enough beside the running replicas — a node of two cards,\n" +
+			"the model on one — --in-place stops each replica first: the model does not\n" +
+			"answer while the new one loads.\n\n" +
+			"--profile names the profile for a model with more than one on cards.",
+		Example: "  llmgw ctl model gpus qwen3-coder-30b 2\n" +
+			"  llmgw ctl model gpus llama-70b 4 --profile h100 --in-place",
+		Args: cobra.ExactArgs(2),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			n, err := strconv.Atoi(args[1])
+			if err != nil {
+				return fmt.Errorf("the cards are a whole number, not %q", args[1])
+			}
+			body, _ := json.Marshal(modelserve.GPUsRequest{GPUs: n, Profile: profile, InPlace: inPlace})
+			var v modelserve.GPUsChange
+			if err := client().patient(time.Minute).post(cmd.Context(), "/api/v1/models/"+args[0]+"/gpus", body, &v); err != nil {
+				return err
+			}
+			out := cmd.OutOrStdout()
+			fmt.Fprintf(out, "%s: %d → %d %s card(s) a replica (%s, %s)\n", v.Model, v.Before, v.After, v.Class, v.Profile, v.Engine)
+			if v.PerCard != "" {
+				fmt.Fprintln(out, "  "+v.PerCard)
+			}
+			if len(v.Args) > 0 {
+				fmt.Fprintln(out, "  args: "+strings.Join(v.Args, " "))
+			}
+			if v.Note != "" {
+				fmt.Fprintln(out, "  note: "+v.Note)
+			}
+			switch {
+			case v.Restarted > 0 && v.InPlace:
+				fmt.Fprintf(out, "%d replica(s) stopped and started again on it, one by one: llmgw ctl replicas --model %s\n", v.Restarted, v.Model)
+			case v.Restarted > 0:
+				fmt.Fprintf(out, "%d replica(s) started again on it, each new one before the old stops: llmgw ctl replicas --model %s\n", v.Restarted, v.Model)
+			}
+			return nil
+		},
+	}
+	c.Flags().StringVar(&profile, "profile", "", "the profile to change, for a model with several on cards")
+	c.Flags().BoolVar(&inPlace, "in-place", false, "stop each replica before its replacement starts, when there is no room beside it")
+	return c
+}

internal/cli/model_serve.go

@@ -64,6 +64,8 @@ func newModelServeCmd(client func() *ctlClient) *cobra.Command {
 		"--embedding, 32 slots of 1024 tokens unless --ctx-pool or --context-length says otherwise")
 	f.StringVar(&req.Format, "format", "gguf", "gguf, run by llama.cpp, or safetensors: the original weights, run by vLLM on GPUs")
 	f.BoolVar(&req.CPU, "cpu", false, "run it on the CPU, whatever the plan says")
+	f.StringVar(&req.Engine, "engine", "", "strata, for a family Strata runs: one card and the experts in RAM; "+
+		"llamacpp to keep it off Strata (default: the plan's best)")
 	f.StringVar(&req.GPUClass, "gpu-class", "", "run it on GPUs of this class, whatever the plan says")
 	f.IntVar(&req.GPUsPerNode, "gpus", 0, "with --gpu-class: the cards one replica takes on each node (default 1)")
 	f.IntVar(&req.Nodes, "nodes", 0, "with --gpu-class: the nodes one replica spans, over a fabric (vLLM only)")

internal/cli/modelplan.go

@@ -310,6 +310,9 @@ func planFiles(cmd *cobra.Command, client func() *ctlClient, repo, format string
 		if b.CPU {
 			holds, all = fmt.Sprintf("%.1f GB of RAM", b.NeedGB), "-"
 		}
+		if b.Engine == "strata" {
+			holds, all = fmt.Sprintf("%.0f GB of RAM", b.NeedGB), "1 card"
+		}
 		fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%.0f GB\t%s\t%s\n", name, bytesShort(a.Bytes), b.Label(), holds, b.HaveGB, all, now)
 	}
 	_ = w.Flush()
@@ -339,6 +342,10 @@ func planFiles(cmd *cobra.Command, client func() *ctlClient, repo, format string
 	if p.Format == "safetensors" {
 		fmt.Fprint(out, " --format safetensors")
 	}
+	if rec.Engine == "strata" {
+		fmt.Fprintf(out, " --file %s --engine strata\n", rec.Artifact)
+		return nil
+	}
 	fmt.Fprintln(out, " --fallbacks")
 	return nil
 }

internal/cli/server.go

@@ -453,6 +453,9 @@ func newServerCmd() *cobra.Command {
 				if st := rt.Store(); st != nil {
 					svc := modelsvc.New(cfg.Models, st, admin, log)
 					svc.SetCheck(servableCheck(cfg.WantedEngines()["llamacpp"]))
+					// What is added from the registry is pinned to the
+					// commit it is at, with each file's sha256 for the nodes.
+					svc.SetRegistry(hf.New(cfg.Mirror.Registry, 30*time.Second))
 					// Other llmgw clusters' models, offered as PEER/MODEL:
 					// read before the catalog is built, so a model that falls
 					// back to one is whole at startup.
@@ -583,6 +586,10 @@ func newServerCmd() *cobra.Command {
 					agentsvc.SetSourceRewrite(src.Rewrite)
 				}
 				rt.SetMirror(mc)
+				if adminAPI != nil {
+					// A model removed with its files is removed from it too.
+					adminAPI.SetPurgeMirror(mc)
+				}
 				if fwPub != nil {
 					fwPub.setMirror(mc)
 				}
@@ -607,7 +614,7 @@ func newServerCmd() *cobra.Command {
 						// Updates: the catalog in force, the service that
 						// changes it, and what replaces replicas.
 						Current: func() []catalog.Model { return admin.Catalog().Models() },
-						Models:  catalogSvc, Restart: admin.Reconciler().Restart,
+						Models:  catalogSvc, Restart: admin.Reconciler().Restart, RestartInPlace: admin.Reconciler().RestartInPlace,
 						Pool: adminAPI.PoolNodes, Notify: notifier,
 						// Canaries: how the router saw each side serve.
 						CanaryStats: rt.CanaryStats, ResetCanary: rt.ResetCanary,

internal/cli/serverinit_cluster.go

@@ -64,6 +64,9 @@ const agentUnit = `[Unit]
 Description=llmgw node agent
 After=network-online.target
 Wants=network-online.target
+# Never given up on: a new release that fails its first starts, or an upgrade
+# that restarts it twice in a minute, would otherwise leave the node off.
+StartLimitIntervalSec=0
 
 [Service]
 ExecStart=/usr/local/bin/llmgw agent --config /etc/llmgw/agent.yaml

internal/cli/tui_actions.go

@@ -16,6 +16,7 @@ import (
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/agentsvc"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/facility"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
@@ -97,10 +98,21 @@ func (m *consoleModel) openForm(f *formDialog) {
 
 // call runs one admin API request in the background and reports it.
 func (m *consoleModel) call(title, method, path string, body any, say func(raw json.RawMessage) string) tea.Cmd {
+	return m.callWaiting(title, method, path, body, say, 60*time.Second)
+}
+
+// callWaiting is call for a change the server may spend longer on than the
+// console's usual wait.
+func (m *consoleModel) callWaiting(title, method, path string, body any, say func(raw json.RawMessage) string,
+	wait time.Duration,
+) tea.Cmd {
 	m.busy = title
 	c := m.client
+	if wait > 60*time.Second {
+		c = c.patient(wait)
+	}
 	return func() tea.Msg {
-		ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
+		ctx, cancel := context.WithTimeout(context.Background(), wait)
 		defer cancel()
 		var raw json.RawMessage
 		// A change may answer with no body at all.
@@ -356,6 +368,31 @@ func (m consoleModel) modelMenu(md adminapi.ModelView) *menuDialog {
 				}})
 			return nil
 		}},
+		{"GPUs…", func(m *consoleModel) tea.Cmd {
+			profiles := append([]string{"(its only one on cards)"}, md.Profiles...)
+			m.openForm(&formDialog{title: "Cards of " + md.Name,
+				note: "How many cards of one node each replica uses: tensor parallel for vLLM, SGLang and TensorRT-LLM, " +
+					"the layers split for llama.cpp. Checked against the model's heads, its cards' memory and the nodes; " +
+					"its replicas are then started again on it, one by one.",
+				fields: []formField{
+					choiceF("profile", profiles, profiles[0]),
+					textF("cards", "", "a replica's, on one node"),
+					choiceF("in place", []string{"no", "yes"}, "no"),
+				},
+				submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
+					n, err := strconv.Atoi(v[1])
+					if err != nil || n < 1 {
+						return nil, fmt.Errorf("the cards are a whole number from 1")
+					}
+					profile := v[0]
+					if profile == profiles[0] {
+						profile = ""
+					}
+					return m.call("Cards of "+md.Name, http.MethodPost, path+"/gpus",
+						modelserve.GPUsRequest{GPUs: n, Profile: profile, InPlace: v[2] == "yes"}, gpusSays), nil
+				}})
+			return nil
+		}},
 		{"Resources…", func(m *consoleModel) tea.Cmd {
 			profiles := md.Profiles
 			if len(profiles) == 0 {
@@ -454,13 +491,40 @@ func (m consoleModel) modelMenu(md adminapi.ModelView) *menuDialog {
 			return m.call("Retry "+md.Name, http.MethodPost, path+"/retry", map[string]string{}, nil)
 		}})
 	}
+	if md.Stopped {
+		items = append(items, menuItem{"Start", func(m *consoleModel) tea.Cmd {
+			return m.call("Start "+md.Name, http.MethodPost, path+"/start", nil, func(json.RawMessage) string {
+				return md.Name + " runs again, scaled as it was when it was stopped."
+			})
+		}})
+	} else {
+		items = append(items, menuItem{"Stop…", func(m *consoleModel) tea.Cmd {
+			m.confirmD = &confirmDialog{title: "Stop " + md.Name,
+				text: "Its replicas stop; it stays in the catalog with its files, and Start runs it again as it was.",
+				yes: func(m *consoleModel) tea.Cmd {
+					return m.call("Stop "+md.Name, http.MethodPost, path+"/stop", nil, func(json.RawMessage) string {
+						return md.Name + " is stopped and kept."
+					})
+				}}
+			return nil
+		}})
+	}
 	items = append(items, menuItem{"Remove…", func(m *consoleModel) tea.Cmd {
 		m.confirmD = &confirmDialog{title: "Remove " + md.Name,
-			text: "Its replicas stop and it leaves the catalog. A model from server.yaml cannot be removed here.",
+			text: "Its replicas stop and it leaves the catalog; its files stay on the mirror and the nodes. " +
+				"A model from server.yaml cannot be removed here.",
 			yes: func(m *consoleModel) tea.Cmd {
 				return m.call("Remove "+md.Name, http.MethodDelete, path, nil, nil)
 			}}
 		return nil
+	}}, menuItem{"Remove with its files…", func(m *consoleModel) tea.Cmd {
+		m.confirmD = &confirmDialog{title: "Remove " + md.Name + " and its files",
+			text: "It leaves the catalog, every version of it is deleted from the mirror, and the nodes' copies once " +
+				"its replicas have stopped. What another model uses stays. This cannot be undone: adding it again fetches it again.",
+			yes: func(m *consoleModel) tea.Cmd {
+				return m.callWaiting("Remove "+md.Name, http.MethodDelete, path+"?purge=true", nil, purgeSays, 3*time.Minute)
+			}}
+		return nil
 	}})
 	return &menuDialog{title: md.Name, items: items}
 }
@@ -508,9 +572,15 @@ func (m consoleModel) nodeMenu(n adminapi.NodeView) *menuDialog {
 		menuItem{"Upgrade its agent…", func(m *consoleModel) tea.Cmd {
 			m.openForm(&formDialog{title: "Upgrade " + n.ID,
 				note:   "It fetches the release for its platform, checks it, swaps it in and restarts; engines keep serving.",
-				fields: []formField{textF("version", version.Version, "a release, like v0.11.1")},
+				fields: []formField{textF("version", releaseOrEmpty(version.Version), "a release, like v0.11.1")},
 				submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
-					return m.call("Upgrade "+n.ID, http.MethodPost, path+"/upgrade", adminapi.UpgradeRequest{Version: v[0]}, nil), nil
+					if !release.Tag.MatchString(v[0]) {
+						return nil, fmt.Errorf("a release is vMAJOR.MINOR.PATCH, like v0.15.3")
+					}
+					// As long as the control plane waits: the download, the
+					// restart, the node back on the version.
+					return m.callWaiting("Upgrade "+n.ID, http.MethodPost, path+"/upgrade", adminapi.UpgradeRequest{Version: v[0]},
+						func(json.RawMessage) string { return n.ID + " is on " + v[0] + "." }, agentsvc.UpgradeWait+time.Minute), nil
 				}})
 			return nil
 		}},
@@ -943,3 +1013,62 @@ func contextSays(raw json.RawMessage) string {
 	}
 	return s
 }
+
+// gpusSays is a change of cards in a sentence or two.
+func gpusSays(raw json.RawMessage) string {
+	var v modelserve.GPUsChange
+	if json.Unmarshal(raw, &v) != nil {
+		return "Done."
+	}
+	s := fmt.Sprintf("%s: %d → %d %s card(s) a replica.", v.Model, v.Before, v.After, v.Class)
+	if v.PerCard != "" {
+		s += "\n" + v.PerCard
+	}
+	if v.Note != "" {
+		s += "\n" + v.Note
+	}
+	switch {
+	case v.Restarted > 0 && v.InPlace:
+		s += fmt.Sprintf("\n%d replica(s) stopped and started again on it, one by one.", v.Restarted)
+	case v.Restarted > 0:
+		s += fmt.Sprintf("\n%d replica(s) started again on it, one by one.", v.Restarted)
+	}
+	return s
+}
+
+// purgeSays is a model removed with its files, in a few lines.
+func purgeSays(raw json.RawMessage) string {
+	var v adminapi.PurgeView
+	if json.Unmarshal(raw, &v) != nil {
+		return "Done."
+	}
+	s := v.Removed + " removed."
+	if len(v.Mirror) > 0 {
+		s += fmt.Sprintf("\nMirror: %d version(s) deleted.", len(v.Mirror))
+	}
+	if len(v.Kept) > 0 {
+		s += "\nKept on the mirror, another model uses them: " + strings.Join(v.Kept, ", ")
+	}
+	var files int
+	var freed int64
+	for _, n := range v.Nodes {
+		files += n.Files
+		freed += n.Bytes
+	}
+	if files > 0 {
+		s += fmt.Sprintf("\nNodes: %d file(s), %s freed.", files, humanBytes(freed))
+	}
+	if v.Note != "" {
+		s += "\n" + v.Note
+	}
+	return s
+}
+
+// releaseOrEmpty is this build's version when it is a release, else empty:
+// a development build's is not one a node can fetch.
+func releaseOrEmpty(v string) string {
+	if release.Tag.MatchString(v) {
+		return v
+	}
+	return ""
+}

internal/cli/tui_hf.go

@@ -206,6 +206,9 @@ func (m *consoleModel) serveForm(p modelserve.Plan, a modelserve.PlannedArtifact
 	if a.Format == "safetensors" {
 		ctxDefault = strconv.Itoa(p.Context)
 	}
+	if len(opts) > 0 && opts[0].Engine == "strata" {
+		ctxDefault = "32768"
+	}
 	fields := []formField{
 		textF("name", hf.ModelName(p.Repo), "what clients ask for"),
 		textF("replicas", "1", "0 runs none until it is first asked for"),
@@ -241,12 +244,18 @@ func (m *consoleModel) serveForm(p modelserve.Plan, a modelserve.PlannedArtifact
 			if sized := o.WithContext(p.KVBytesPerToken, ctxLen); !sized.Fits() {
 				return nil, fmt.Errorf("with a %d-token context it needs %s: a shorter context, or another way", ctxLen, sized.MemoryText())
 			}
-			if o.CPU {
+			switch {
+			case o.Engine == "strata":
+				// One card, the experts in RAM; Strata reads the pictures
+				// with a reader of its own, not this projector.
+				req.Engine, req.GPUClass, proj = "strata", o.Class, -1
+			case o.CPU:
 				req.CPU = true
-			} else {
+			default:
+				req.Engine = "llamacpp"
 				req.GPUClass, req.GPUsPerNode, req.Nodes = o.Class, o.GPUsPerNode, max(o.Nodes, 1)
 			}
-			req.Fallbacks = strings.HasPrefix(v[5], "the next") && len(opts) > 1
+			req.Fallbacks = strings.HasPrefix(v[5], "the next") && len(opts) > 1 && o.Engine != "strata"
 			if proj >= 0 && !strings.HasPrefix(v[proj], "none") {
 				req.Projector = v[proj]
 			}

internal/control/adminapi/api.go

@@ -60,6 +60,9 @@ type API struct {
 	controlPlanes func() ControlPlanes
 	// serve runs registry-to-replica jobs; nil without a mirror and a store.
 	serve *modelserve.Server
+	// purgeMirror is the mirror a model removed with its files is removed
+	// from; nil without one.
+	purgeMirror PurgeMirror
 	// history is the Prometheus the console's charts read from.
 	history History
 	// audit reads the audit log.
@@ -123,6 +126,8 @@ func (a *API) Register(mux Mux) {
 	mux.HandleFunc("POST /api/v1/models", a.addModel)
 	mux.HandleFunc("GET /api/v1/models/{name}", a.getModel)
 	mux.HandleFunc("DELETE /api/v1/models/{name}", a.removeModel)
+	mux.HandleFunc("POST /api/v1/models/{name}/stop", a.stopModel)
+	mux.HandleFunc("POST /api/v1/models/{name}/start", a.startModel)
 	mux.HandleFunc("POST /api/v1/models/{name}/scale", a.scaleModel)
 	mux.HandleFunc("POST /api/v1/models/{name}/resources", a.setResources)
 	a.registerQuotas(mux)
@@ -206,6 +211,7 @@ func (a *API) Register(mux Mux) {
 	mux.HandleFunc("POST /api/v1/updates/check", a.checkUpdates)
 	mux.HandleFunc("POST /api/v1/models/{name}/update", a.updateModel)
 	mux.HandleFunc("POST /api/v1/models/{name}/context", a.setModelContext)
+	mux.HandleFunc("POST /api/v1/models/{name}/gpus", a.setModelGPUs)
 	mux.HandleFunc("POST /api/v1/models/{name}/rollback", a.rollbackModel)
 	a.registerCanary(mux)
 	a.registerEval(mux)
@@ -450,6 +456,8 @@ type ModelView struct {
 	// answer; 0 when the catalog does not say.
 	ContextLength int `json:"context_length,omitempty"`
 	MaxOutput     int `json:"max_output,omitempty"`
+	// Stopped is a model an administrator took out of service, kept.
+	Stopped bool `json:"stopped,omitempty"`
 	// Canary is a share of its requests sent to a candidate; CanaryOf is
 	// set on the candidate, naming the model it is a canary of.
 	Canary   *catalog.Canary `json:"canary,omitempty"`
@@ -842,7 +850,7 @@ func (a *API) Models() []ModelView {
 		h := health[m.Name]
 		v := ModelView{
 			Name: m.Name, Description: m.Description, Aliases: m.Aliases, Source: m.Source,
-			ContextLength: m.ContextLength, MaxOutput: m.MaxOutput,
+			ContextLength: m.ContextLength, MaxOutput: m.MaxOutput, Stopped: m.Stopped != nil,
 			Min: m.Scaling.Min, Max: m.Scaling.Max,
 			Ready: ready[m.Name], Total: total[m.Name],
 			Fallbacks: m.Fallbacks,
@@ -944,6 +952,8 @@ type Models interface {
 	Add(ctx context.Context, m catalog.Model, by string) error
 	Remove(ctx context.Context, name string) error
 	Scale(ctx context.Context, name string, s catalog.Scaling, by string) (catalog.Model, error)
+	Stop(ctx context.Context, name, by string) (catalog.Model, error)
+	Start(ctx context.Context, name, by string) (catalog.Model, error)
 	Resources(ctx context.Context, name, profile string, memoryGB, cpuCores int, by string) (catalog.Model, error)
 	SetClass(ctx context.Context, name, class, by string) (catalog.Model, error)
 	SetOwner(ctx context.Context, name, owner, by string) (catalog.Model, error)
@@ -1193,6 +1203,8 @@ type ScaleResponse struct {
 	Max         int           `json:"max"`
 	IdleTimeout time.Duration `json:"idle_timeout,omitempty"`
 	OnDemand    bool          `json:"on_demand"`
+	// Stopped is a model taken out of service, kept, until it is started.
+	Stopped bool `json:"stopped,omitempty"`
 }
 
 func (a *API) scaleModel(w http.ResponseWriter, r *http.Request) {
@@ -1226,6 +1238,32 @@ func (a *API) scaleModel(w http.ResponseWriter, r *http.Request) {
 	})
 }
 
+// stopModel takes a model out of service, kept.
+func (a *API) stopModel(w http.ResponseWriter, r *http.Request) {
+	a.stopOrStart(w, r, a.models.Stop)
+}
+
+// startModel runs a stopped model again.
+func (a *API) startModel(w http.ResponseWriter, r *http.Request) {
+	a.stopOrStart(w, r, a.models.Start)
+}
+
+func (a *API) stopOrStart(w http.ResponseWriter, r *http.Request, f func(context.Context, string, string) (catalog.Model, error)) {
+	if a.models == nil {
+		writeError(w, http.StatusNotImplemented, "this control plane has no store")
+		return
+	}
+	m, err := f(r.Context(), r.PathValue("name"), actorOf(r))
+	if err != nil {
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	}
+	writeJSON(w, http.StatusOK, ScaleResponse{
+		Model: m.Name, Min: m.Scaling.Min, Max: m.Scaling.Max,
+		IdleTimeout: m.Scaling.IdleTimeout, OnDemand: m.Scaling.OnDemand(), Stopped: m.Stopped != nil,
+	})
+}
+
 // setResources sets a model's reserved memory and cores. Placed replicas keep
 // what they were started with until they are restarted.
 func (a *API) setResources(w http.ResponseWriter, r *http.Request) {
@@ -1252,6 +1290,10 @@ func (a *API) removeModel(w http.ResponseWriter, r *http.Request) {
 		return
 	}
 	name := r.PathValue("name")
+	if r.URL.Query().Get("purge") == "true" {
+		a.purgeModel(w, r, name)
+		return
+	}
 	if err := a.models.Remove(r.Context(), name); err != nil {
 		writeError(w, http.StatusBadRequest, err.Error())
 		return
@@ -1774,6 +1816,26 @@ func (a *API) setModelContext(w http.ResponseWriter, r *http.Request) {
 	writeJSON(w, http.StatusOK, change)
 }
 
+// setModelGPUs spreads a served model over more cards, or fewer, and
+// starts its replicas again on it.
+func (a *API) setModelGPUs(w http.ResponseWriter, r *http.Request) {
+	if a.serve == nil {
+		writeError(w, http.StatusNotImplemented, "this control plane cannot change a model's cards")
+		return
+	}
+	var req modelserve.GPUsRequest
+	if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	}
+	change, err := a.serve.SetGPUs(r.Context(), r.PathValue("name"), req, actorOf(r))
+	if err != nil {
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	}
+	writeJSON(w, http.StatusOK, change)
+}
+
 func (a *API) listUpdates(w http.ResponseWriter, _ *http.Request) {
 	out := make([]modelserve.UpdateInfo, 0)
 	if a.serve != nil {
@@ -1877,7 +1939,7 @@ func PoolNodesOf(nodes []NodeView) []modelplan.PoolNode {
 			continue
 		}
 		p := modelplan.PoolNode{ID: n.ID, GPUs: n.GPUTotal, FreeGPUs: n.GPUFree, HostMemGB: n.MemoryGB,
-			HostFreeGB: n.MemoryGB}
+			HostFreeGB: n.MemoryGB, Engines: n.Engines}
 		if len(n.Classes) > 0 {
 			p.Class = n.Classes[0]
 			p.GPUMemGB = modelplan.MemFromClass(p.Class)

internal/control/adminapi/api_test.go

@@ -749,3 +749,11 @@ func TestANodesCardInTheWrongSlotIsAWarning(t *testing.T) {
 		t.Errorf("warnings %q", warned)
 	}
 }
+
+func (f *fakeModels) Stop(_ context.Context, name, _ string) (catalog.Model, error) {
+	return catalog.Model{Name: name, Stopped: &catalog.Scaling{Min: 1, Max: 1}}, f.err
+}
+
+func (f *fakeModels) Start(_ context.Context, name, _ string) (catalog.Model, error) {
+	return catalog.Model{Name: name, Scaling: catalog.Scaling{Min: 1, Max: 1}}, f.err
+}

internal/control/adminapi/purge.go

@@ -0,0 +1,230 @@
+package adminapi
+
+import (
+	"context"
+	"log/slog"
+	"net/http"
+	"net/url"
+	"path/filepath"
+	"slices"
+	"strings"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
+)
+
+// A model removed with its files: the catalog entry, every version of it the
+// mirror holds (and so, carried by its removal, a peer mirror's), and the
+// copies the nodes fetched, once its replicas have stopped. What another
+// model still uses stays.
+
+// PurgeMirror is what a purge asks of the mirror.
+type PurgeMirror interface {
+	List(ctx context.Context) (mirror.ModelsResponse, error)
+	Remove(ctx context.Context, model, version string) ([]string, error)
+}
+
+// SetPurgeMirror gives removing a model with its files the mirror.
+func (a *API) SetPurgeMirror(m PurgeMirror) { a.purgeMirror = m }
+
+// purgeWait is how long a purge waits for the model's replicas to stop
+// before deleting the nodes' copies; after it, they are left for llmgw ctl
+// weights --collect.
+var purgeWait = 60 * time.Second
+
+// PurgeView is what a model removed with its files took with it.
+type PurgeView struct {
+	Removed string `json:"removed"`
+	// Mirror are the versions removed from the mirror, model/version.
+	Mirror []string `json:"mirror,omitempty"`
+	// Kept are versions another model still uses, left on the mirror.
+	Kept  []string        `json:"kept,omitempty"`
+	Nodes []CollectResult `json:"nodes,omitempty"`
+	Note  string          `json:"note,omitempty"`
+}
+
+// purgeModel removes a model from the catalog and its files from the mirror
+// and the nodes.
+func (a *API) purgeModel(w http.ResponseWriter, r *http.Request, name string) {
+	m, ok := a.ctl.Catalog().Resolve(name)
+	if !ok {
+		writeError(w, http.StatusNotFound, "no model "+name+" in the catalog")
+		return
+	}
+	name = m.Name
+	others := []catalog.Model{}
+	for _, o := range a.ctl.Catalog().Models() {
+		if o.Name != name {
+			others = append(others, o)
+		}
+	}
+	if err := a.models.Remove(r.Context(), name); err != nil {
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	}
+	a.presetDeclined(r.Context(), name, actorOf(r))
+	out := PurgeView{Removed: name}
+
+	sources := purgeSources(m)
+	if a.purgeMirror == nil {
+		out.Note = "this control plane has no mirror; its files there, if any, are left"
+	} else {
+		out.Mirror, out.Kept, out.Note = purgeFromMirror(r.Context(), a.purgeMirror, sources, others)
+	}
+
+	// The nodes' copies, once nothing runs on them.
+	deadline := time.Now().Add(purgeWait)
+	for a.runs(name) && time.Now().Before(deadline) {
+		select {
+		case <-r.Context().Done():
+			return
+		case <-time.After(time.Second):
+		}
+	}
+	if a.runs(name) {
+		out.Note = strings.TrimSpace(out.Note + " Its replicas had not stopped yet: its copies on the nodes are " +
+			"left, and llmgw ctl weights --collect deletes them.")
+		writeJSON(w, http.StatusOK, out)
+		return
+	}
+	out.Nodes = a.purgeFromNodes(r.Context(), sources, others)
+	writeJSON(w, http.StatusOK, out)
+}
+
+// runs says whether any replica of a model is still on a node.
+func (a *API) runs(model string) bool {
+	for _, rep := range a.ctl.Reconciler().Replicas() {
+		if rep.Model == model {
+			return true
+		}
+	}
+	return false
+}
+
+// purgeSources are the files a model fetched: its weights, its adapters,
+// and the weights its last update replaced.
+func purgeSources(m catalog.Model) []string {
+	out := m.WeightSources()
+	for _, ad := range m.Adapters {
+		out = append(out, ad.Source)
+	}
+	if m.Origin != nil && m.Origin.Previous != nil && m.Origin.Previous.Source != "" {
+		out = append(out, m.Origin.Previous.Source)
+	}
+	return out
+}
+
+// inUse is every file the other models fetch.
+func inUse(others []catalog.Model) []string {
+	var out []string
+	for _, o := range others {
+		out = append(out, purgeSources(o)...)
+	}
+	return out
+}
+
+// purgeFromMirror removes from the mirror every version of the model
+// directories the sources are in, but those another model uses.
+func purgeFromMirror(ctx context.Context, mc PurgeMirror, sources []string, others []catalog.Model) (removed, kept []string, note string) {
+	held, err := mc.List(ctx)
+	if err != nil {
+		return nil, nil, "the mirror could not be read (" + err.Error() + "): its files there are left"
+	}
+	dirs := map[string]bool{}
+	for _, s := range sources {
+		if model, _, ok := mirrorPlace(s); ok {
+			dirs[model] = true
+		}
+	}
+	used := map[string]bool{}
+	for _, s := range inUse(others) {
+		if model, version, ok := mirrorPlace(s); ok {
+			used[model+"/"+version] = true
+		}
+	}
+	seen := map[string]bool{}
+	for _, st := range held.Models {
+		key := st.Model + "/" + st.Version
+		if !dirs[st.Model] || seen[key] {
+			continue
+		}
+		seen[key] = true
+		if used[key] {
+			kept = append(kept, key)
+			continue
+		}
+		if _, err := mc.Remove(ctx, st.Model, st.Version); err != nil {
+			note = strings.TrimSpace(note + " " + key + " could not be removed from the mirror: " + err.Error() + ".")
+			continue
+		}
+		removed = append(removed, key)
+	}
+	slices.Sort(removed)
+	slices.Sort(kept)
+	return removed, kept, note
+}
+
+// mirrorPlace is the model directory and version a mirror URL names:
+// BASE/MODEL/VERSION/FILE.
+func mirrorPlace(source string) (model, version string, ok bool) {
+	u, err := url.Parse(source)
+	if err != nil || (u.Scheme != "http" && u.Scheme != "https") {
+		return "", "", false
+	}
+	// A mirror serves from its root: a registry's address is longer.
+	parts := strings.Split(strings.Trim(u.Path, "/"), "/")
+	if len(parts) != 3 {
+		return "", "", false
+	}
+	model, version = parts[0], parts[1]
+	if mirror.ValidName(model) != nil || mirror.ValidName(version) != nil {
+		return "", "", false
+	}
+	return model, version, true
+}
+
+// purgeFromNodes deletes the nodes' copies of the sources: the files llmgw
+// fetched for them under any mirror's address, that no other model wants.
+func (a *API) purgeFromNodes(ctx context.Context, sources []string, others []catalog.Model) []CollectResult {
+	rm := a.ctl.WeightRemover()
+	if rm == nil {
+		return nil
+	}
+	names := map[string]bool{}
+	for _, s := range sources {
+		names[filepath.Base(s)] = true
+	}
+	wanted := inUse(others)
+	var out []CollectResult
+	for _, n := range a.ctl.Inventory().Nodes() {
+		size := map[string]int64{}
+		have := make([]string, 0, len(n.Weights))
+		for _, w := range n.Weights {
+			have = append(have, w.Path)
+			size[w.Path] = w.SizeBytes
+		}
+		var paths []string
+		for _, p := range weights.Orphans(have, wanted) {
+			// The cache name is a hash of the address, then the file's own
+			// name: the address may be either mirror's, the name is the same.
+			if _, file, ok := strings.Cut(filepath.Base(p), "-"); ok && names[file] {
+				paths = append(paths, p)
+			}
+		}
+		if len(paths) == 0 {
+			continue
+		}
+		res := CollectResult{Node: n.ID, Files: len(paths)}
+		for _, p := range paths {
+			res.Bytes += size[p]
+		}
+		if err := rm.RemoveWeights(ctx, n.ID, paths); err != nil {
+			res.Error, res.Files, res.Bytes = err.Error(), 0, 0
+			slog.Warn("a purged model's copies could not be deleted on a node", "node", n.ID, "err", err)
+		}
+		out = append(out, res)
+	}
+	return out
+}

internal/control/adminapi/purge_test.go

@@ -0,0 +1,68 @@
+package adminapi
+
+import (
+	"context"
+	"slices"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
+)
+
+type purgeMirrorFake struct {
+	held    []mirror.Status
+	removed []string
+}
+
+func (f *purgeMirrorFake) List(context.Context) (mirror.ModelsResponse, error) {
+	return mirror.ModelsResponse{Base: "http://m:8081", Models: f.held}, nil
+}
+
+func (f *purgeMirrorFake) Remove(_ context.Context, model, version string) ([]string, error) {
+	f.removed = append(f.removed, model+"/"+version)
+	return nil, nil
+}
+
+func TestAPurgeRemovesEveryVersionOfTheModelButWhatAnotherUses(t *testing.T) {
+	f := &purgeMirrorFake{held: []mirror.Status{
+		{Model: "qwen", Version: "v2", File: "a.gguf"},
+		{Model: "qwen", Version: "v2", File: "b.gguf"},
+		{Model: "qwen", Version: "v1", File: "a.gguf"},
+		{Model: "qwen", Version: "v3", File: "a.gguf"},
+		{Model: "other", Version: "v1", File: "o.gguf"},
+		{Model: "lora", Version: "t1", File: "l.gguf"},
+	}}
+	m := catalog.Model{Name: "qwen", Source: "http://m:8081/qwen/v2/a.gguf",
+		Adapters: []catalog.Adapter{{Name: "tone", Source: "http://m:8081/lora/t1/l.gguf"}}}
+	// The canary runs v3: it stays.
+	others := []catalog.Model{{Name: "qwen-canary", Source: "http://other-mirror:8081/qwen/v3/a.gguf"},
+		{Name: "other", Source: "http://m:8081/other/v1/o.gguf"}}
+	removed, kept, note := purgeFromMirror(t.Context(), f, purgeSources(m), others)
+	if note != "" {
+		t.Fatal(note)
+	}
+	if !slices.Equal(removed, []string{"lora/t1", "qwen/v1", "qwen/v2"}) || !slices.Equal(kept, []string{"qwen/v3"}) {
+		t.Errorf("removed %v, kept %v", removed, kept)
+	}
+	if slices.Contains(f.removed, "other/v1") || len(f.removed) != 3 {
+		t.Errorf("asked to remove %v", f.removed)
+	}
+}
+
+func TestOnlyAMirrorsAddressNamesAPlaceOnIt(t *testing.T) {
+	for src, want := range map[string]string{
+		"http://10.0.0.146:8081/qwen/abc/q4.gguf":         "qwen/abc",
+		"https://huggingface.co/o/r/resolve/main/q4.gguf": "",
+		"/srv/models/q4.gguf":                             "",
+		"Qwen/Qwen2.5-7B":                                 "",
+	} {
+		model, version, ok := mirrorPlace(src)
+		got := ""
+		if ok {
+			got = model + "/" + version
+		}
+		if got != want {
+			t.Errorf("%s: %q, want %q", src, got, want)
+		}
+	}
+}

internal/control/adminapi/routes.go

@@ -155,8 +155,16 @@ var Routes = []Route{
 		Description: "With the defaults the catalog filled in. By its name only, not an alias.",
 		Response:    catalog.Model{}},
 	{Method: "DELETE", Path: "/api/v1/models/{name}", Summary: "Remove a model added at runtime",
-		Description: "A model declared in the cluster's settings is refused: it is changed there.",
-		Response:    Removed{}},
+		Description: "A model declared in the cluster's settings is refused: it is changed there. With purge, its " +
+			"files go too: every version of it the mirror holds (a peer mirror follows), and the nodes' copies once its " +
+			"replicas have stopped, waiting up to a minute for them; what another model uses stays. The answer is then a PurgeView.",
+		Query:    []Param{{Name: "purge", Type: "boolean", Description: "Delete its files from the mirror and the nodes as well."}},
+		Response: Removed{}},
+	{Method: "POST", Path: "/api/v1/models/{name}/stop", Summary: "Take a model out of service and keep it",
+		Description: "None of its replicas run, and requests are told an administrator stopped it; what it was scaled to is kept for start.",
+		Response:    ScaleResponse{}},
+	{Method: "POST", Path: "/api/v1/models/{name}/start", Summary: "Run a stopped model again, scaled as it was",
+		Response: ScaleResponse{}},
 	{Method: "POST", Path: "/api/v1/models/{name}/scale", Summary: "Set how many replicas a model runs",
 		Request: ScaleRequest{}, Response: ScaleResponse{}},
 	{Method: "POST", Path: "/api/v1/models/{name}/resources", Summary: "Set what a model's profiles reserve of a node",
@@ -215,6 +223,9 @@ var Routes = []Route{
 		Request:     modelserve.Change{}, OptionalBody: true, Response: modelserve.Job{}, Status: http.StatusAccepted},
 	{Method: "POST", Path: "/api/v1/models/{name}/context", Summary: "Change a served model's context: the catalog, its engines' arguments and memory, then its replicas restarted",
 		Request: modelserve.ContextRequest{}, Response: modelserve.ContextChange{}},
+	{Method: "POST", Path: "/api/v1/models/{name}/gpus", Summary: "Spread a served model over more cards or fewer: its profile, checked against its KV heads, its cards' memory and the nodes, then its replicas restarted",
+		Description: "Tensor parallel for vLLM, SGLang and TensorRT-LLM, a layer split for llama.cpp, on one node. With no room beside the running replicas, in_place stops each first.",
+		Request:     modelserve.GPUsRequest{}, Response: modelserve.GPUsChange{}},
 	{Method: "POST", Path: "/api/v1/models/{name}/rollback", Summary: "Go back to the weights the last update replaced",
 		Response: Result{}},
 	{Method: "POST", Path: "/api/v1/models/{name}/origin", Summary: "Record the repository a model's weights came from",

internal/control/agentsvc/convert.go

@@ -236,6 +236,7 @@ func StartShardFromCommand(cmd reconciler.ShardCommand) *llmgwv1.StartShard {
 			Adapters:    adaptersToWire(p.Adapters),
 			KvEvents:    p.KVEvents,
 			AllowPickle: p.AllowPickle,
+			ModelSha256: p.ModelSHA256,
 		},
 		StartupTimeoutMs: cmd.StartupTimeout.Milliseconds(),
 		Pool:             cmd.Pool,
@@ -274,6 +275,7 @@ func ShardSpecFromStart(m *llmgwv1.StartShard) (engines.ShardSpec, time.Duration
 			Adapters:    adaptersFromWire(p.GetAdapters()),
 			KVEvents:    p.GetKvEvents(),
 			AllowPickle: p.GetAllowPickle(),
+			ModelSHA256: p.GetModelSha256(),
 		},
 	}
 	return spec, time.Duration(m.GetStartupTimeoutMs()) * time.Millisecond

internal/control/agentsvc/heartbeat_test.go

@@ -3,6 +3,7 @@ package agentsvc
 import (
 	"log/slog"
 	"testing"
+	"time"
 
 	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
@@ -284,6 +285,28 @@ func TestAnUpgradeWaitsForTheAgentsAnswer(t *testing.T) {
 	}
 }
 
+// The agent's answer lost — it exits a moment after sending it — and the node
+// back on the version: the upgrade is done, not five minutes of waiting and a
+// failure blaming an old agent.
+func TestAnUpgradeIsDoneWhenTheNodeComesBackOnTheVersion(t *testing.T) {
+	s := newTestService()
+	nodeWith(s, inventory.Node{ID: "n1", State: inventory.Ready, AgentVersion: "v1.2.2"})
+	c := &conn{nodeID: "n1", send: make(chan *llmgwv1.ControlMessage, 4), closed: make(chan struct{})}
+	s.conns["n1"] = c
+	done := make(chan error, 1)
+	go func() { done <- s.Upgrade(t.Context(), "n1", "v1.2.3", "https://r") }()
+	<-c.send
+	nodeWith(s, inventory.Node{ID: "n1", State: inventory.Ready, AgentVersion: "v1.2.3"})
+	select {
+	case err := <-done:
+		if err != nil {
+			t.Fatal(err)
+		}
+	case <-time.After(10 * time.Second):
+		t.Fatal("still waiting for an answer the node will never send")
+	}
+}
+
 func TestAnotherTenantOnACardIsJudgedEveryHeartbeat(t *testing.T) {
 	// Found on cuda01: ComfyUI held 5 GB when the agent started, so the card
 	// was kept out of placement. ComfyUI let go of it and the card stayed

internal/control/agentsvc/service.go

@@ -1158,21 +1158,45 @@ func (s *Service) Upgrade(ctx context.Context, nodeID, version, releases string)
 		return err
 	}
 	s.log.Info("told a node to upgrade", "node", nodeID, "version", version)
-	// A download of some tens of megabytes on a slow link, and no longer: an
-	// agent too old to know this message never answers.
-	ctx, cancel := context.WithTimeout(ctx, 5*time.Minute)
+	// Longer than the agent gives its download, so its own answer — done, or
+	// why not — is what is heard. Its answer can still be lost: it exits a
+	// moment after sending it, and a stream that breaks takes it along. The
+	// node registering again on the version says the same, and is watched for.
+	ctx, cancel := context.WithTimeout(ctx, UpgradeWait)
 	defer cancel()
-	select {
-	case r := <-ch:
-		if r.GetError() != "" {
-			return errors.New(r.GetError())
+	tick := time.NewTicker(2 * time.Second)
+	defer tick.Stop()
+	for {
+		select {
+		case r := <-ch:
+			if r.GetError() != "" {
+				return errors.New(r.GetError())
+			}
+			return nil
+		case <-tick.C:
+			if n, ok := s.inv.Get(nodeID); ok && n.AgentVersion == version {
+				return nil
+			}
+		case <-ctx.Done():
+			n, ok := s.inv.Get(nodeID)
+			switch {
+			case !slices.Contains(s.ConnectedNodes(), nodeID):
+				return fmt.Errorf("node %s went away during the upgrade and has not come back: on it, "+
+					"`systemctl status llmgw-agent` and `journalctl -u llmgw-agent` say why", nodeID)
+			case ok && n.AgentVersion != "":
+				return fmt.Errorf("node %s did not answer the upgrade and still runs %s: its download is slow or stuck, "+
+					"or its agent is older than v0.10.4, which cannot upgrade itself", nodeID, n.AgentVersion)
+			default:
+				return fmt.Errorf("node %s did not answer the upgrade; an agent older than v0.10.4 cannot upgrade itself", nodeID)
+			}
 		}
-		return nil
-	case <-ctx.Done():
-		return fmt.Errorf("node %s did not answer the upgrade; an agent older than v0.10.4 cannot upgrade itself", nodeID)
 	}
 }
 
+// UpgradeWait is how long the control plane waits for an agent to upgrade:
+// longer than the agent gives its download (agent.upgradeDownload).
+const UpgradeWait = 11 * time.Minute
+
 // wearFrom is a card's wear, when the agent read it.
 func wearFrom(g *llmgwv1.GPUSensor) *inventory.GPUWear {
 	if !g.GetHasWear() {

internal/control/catalog/catalog.go

@@ -22,6 +22,7 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/facility"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/strata"
 )
 
 // ErrExists is a model added only if absent that was there already.
@@ -332,6 +333,10 @@ type Model struct {
 	AllowPickle bool      `json:"allow_pickle,omitempty" koanf:"allow_pickle"`
 	Profiles    []Profile `json:"profiles,omitempty" koanf:"profiles"`
 	Scaling     Scaling   `json:"scaling,omitempty" koanf:"scaling"`
+	// Stopped is the scaling a model had when an administrator stopped it:
+	// kept in the catalog with none running (Scaling 0/0) until started
+	// again with this. Nil for a model not stopped.
+	Stopped *Scaling `json:"stopped,omitempty" koanf:"stopped"`
 	// Class is how much this model's capacity is worth against another's,
 	// when there is not enough for both: critical, standard or opportunistic.
 	// Empty is standard, or opportunistic for a model loaded on demand. See
@@ -1197,10 +1202,14 @@ func (m Model) validate() error {
 			if p.Nodes > 1 || p.TP > 1 || p.PP > 1 || p.GPUsPerNode > 1 {
 				return fmt.Errorf("model %q profile %q: strata runs a replica on one GPU of one node", m.Name, p.Name)
 			}
-			if !slices.ContainsFunc(p.Args, func(a string) bool { return a == "--family" || strings.HasPrefix(a, "--family=") }) {
+			family, size := strata.ArgValue(p.Args, "--family"), strata.ArgValue(p.Args, "--size")
+			if family == "" || size == "" {
 				return fmt.Errorf("model %q profile %q: strata needs the model's family and size in args "+
 					"(--family qwen --size IQ2_XS)", m.Name, p.Name)
 			}
+			if _, err := strata.SizeOf(family, size); err != nil {
+				return fmt.Errorf("model %q profile %q: %w", m.Name, p.Name, err)
+			}
 			if strings.HasSuffix(strings.ToLower(strings.TrimSuffix(m.SourceOf(p), "/")), ".gguf") {
 				return fmt.Errorf("model %q profile %q: strata prepares a model from all of its GGUF files; "+
 					"give the directory they are in as the source", m.Name, p.Name)
@@ -1323,6 +1332,26 @@ func (m Model) SourceOf(p Profile) string {
 	return m.Source
 }
 
+// OriginSHA256 is the sha256 the registry published for the one file a
+// source names, from the model's origin: "" for a set, a repository, or a
+// file the origin does not list.
+func (m Model) OriginSHA256(source string) string {
+	if m.Origin == nil || strings.HasSuffix(source, "/") {
+		return ""
+	}
+	p := source
+	if u, err := url.Parse(source); err == nil && u.Path != "" {
+		p = u.Path
+	}
+	name := path.Base(p)
+	for _, f := range m.Origin.Files {
+		if f.SHA256 != "" && path.Base(f.Path) == name {
+			return f.SHA256
+		}
+	}
+	return ""
+}
+
 // WeightSources are every source a model's replicas may load: the model's
 // and each profile's of its own, once each. What a node must not have
 // collected from under it.
@@ -1398,6 +1427,7 @@ func (m Model) EngineProfile(name string) (engines.Profile, bool) {
 			Adapters:        m.engineAdapters(),
 			KVEvents:        p.KVEvents,
 			AllowPickle:     m.AllowPickle,
+			ModelSHA256:     m.OriginSHA256(m.SourceOf(p)),
 		}.WithRuntimeLoRA(m.MaxLoadedAdapters), true
 	}
 	return engines.Profile{}, false

internal/control/catalog/origin_sha_test.go

@@ -0,0 +1,27 @@
+package catalog
+
+import "testing"
+
+func TestTheFileASourceNamesIsGivenItsPublishedChecksum(t *testing.T) {
+	m := Model{Name: "q", Source: "http://10.0.0.146:8081/q/abc/q4.gguf", Origin: &Origin{Repo: "o/r", Revision: "abc",
+		Files: []OriginFile{{Path: "sub/q4.gguf", SHA256: "aaaa"}, {Path: "config.json"}}}}
+	for src, want := range map[string]string{
+		m.Source: "aaaa",
+		"https://huggingface.co/o/r/resolve/abc/sub/q4.gguf": "aaaa",
+		"http://10.0.0.146:8081/q/abc/":                      "", // a set
+		"o/r":                                                "",
+		"http://10.0.0.146:8081/q/abc/config.json":           "", // no checksum published
+	} {
+		if got := m.OriginSHA256(src); got != want {
+			t.Errorf("%s: %q, want %q", src, got, want)
+		}
+	}
+	m.Profiles = []Profile{{Name: "cpu", Engine: "llamacpp"}}
+	if ep, ok := m.EngineProfile("cpu"); !ok || ep.ModelSHA256 != "aaaa" {
+		t.Errorf("the node is not told it: %+v", ep)
+	}
+	m.Origin = nil
+	if m.OriginSHA256(m.Source) != "" {
+		t.Error("a checksum with no origin")
+	}
+}

internal/control/catalog/strata_test.go

@@ -20,11 +20,17 @@ func TestAStrataProfileNamesItsFamilyAndRunsOnOneCard(t *testing.T) {
 		"family and size": {Name: "x", Source: "http://mirror:8081/x/1/", Profiles: []Profile{{
 			Name: "p", Engine: "strata", GPUClass: "6000-24g", Args: []string{"--size", "IQ2_XS"}}}},
 		"all of its GGUF files": {Name: "x", Source: "http://mirror:8081/x/1/x-00001-of-00002.gguf", Profiles: []Profile{{
+			Name: "p", Engine: "strata", GPUClass: "6000-24g", Args: []string{"--family", "qwen", "--size", "IQ2_XS"}}}},
+		"no family \"llama\"": {Name: "x", Source: "http://mirror:8081/x/1/", Profiles: []Profile{{
+			Name: "p", Engine: "strata", GPUClass: "6000-24g", Args: []string{"--family", "llama", "--size", "IQ2_XS"}}}},
+		"comes in IQ1_M": {Name: "x", Source: "http://mirror:8081/x/1/", Profiles: []Profile{{
+			Name: "p", Engine: "strata", GPUClass: "6000-24g", Args: []string{"--family", "coder", "--size", "IQ2_XS"}}}},
+		"family and size ": {Name: "x", Source: "http://mirror:8081/x/1/", Profiles: []Profile{{
 			Name: "p", Engine: "strata", GPUClass: "6000-24g", Args: []string{"--family", "qwen"}}}},
 		"--gguf-dir": {Name: "x", Source: "http://mirror:8081/x/1/", Profiles: []Profile{{
 			Name: "p", Engine: "strata", GPUClass: "6000-24g", Args: []string{"--family", "qwen", "--gguf-dir", "/etc"}}}},
 	} {
-		if _, err := New([]Model{bad}); err == nil || !strings.Contains(err.Error(), want) {
+		if _, err := New([]Model{bad}); err == nil || !strings.Contains(err.Error(), strings.TrimSpace(want)) {
 			t.Errorf("expected %q, got %v", want, err)
 		}
 	}

internal/control/modelserve/context.go

@@ -49,11 +49,13 @@ type ContextChange struct {
 }
 
 // shape is what sizes a context: a token's KV cache, the context the model
-// was trained for, and its weights.
+// was trained for, and its weights; and its KV heads, which decide how many
+// cards tensor parallel can spread it over (0 unknown).
 type shape struct {
 	kvPerToken int64
 	trained    int
 	weights    int64
+	kvHeads    int
 }
 
 // SetContext changes a served model's context.
@@ -135,12 +137,14 @@ func (s *Server) shapeOf(ctx context.Context, m catalog.Model) (shape, string) {
 			return sh, "its header could not be read (" + err.Error() + "), so the memory it needs is not checked"
 		}
 		sh.kvPerToken, sh.trained = h.KVBytesPerToken(), int(h.ContextLength) //nolint:gosec // a header's context
+		sh.kvHeads = int(h.HeadCountKV)                                       //nolint:gosec // a header's head count
 	case m.Origin != nil && m.Origin.Repo != "":
 		mm, err := modelplan.FetchModel(ctx, m.Origin.Repo)
 		if err != nil {
 			return sh, "its config could not be read (" + err.Error() + "), so the memory it needs is not checked"
 		}
 		sh.kvPerToken, sh.trained = mm.KVBytesPerToken(modelplan.BF16), mm.MaxContext
+		sh.kvHeads = mm.KVHeads
 	default:
 		return sh, "where it came from is not known, so the memory it needs is not checked"
 	}

internal/control/modelserve/gpus.go

@@ -0,0 +1,285 @@
+package modelserve
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"slices"
+	"strings"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/modelplan"
+)
+
+// A served model moved to more cards, or fewer: how many each replica
+// spreads it over, on one node. The profile changes — tensor parallel for
+// vLLM, SGLang and TensorRT-LLM, llama.cpp's layer split for llama.cpp —
+// checked against the model's KV heads, what each card would hold, and the
+// nodes there are; then its replicas start again on it.
+
+// GPUsRequest is how many cards a replica is to use.
+type GPUsRequest struct {
+	GPUs int `json:"gpus"`
+	// Profile is the one to change, when the model has more than one
+	// profile on cards.
+	Profile string `json:"profile,omitempty"`
+	// InPlace stops each replica before its replacement starts: for when no
+	// node has the cards free beside it. It does not answer while the new
+	// one loads.
+	InPlace bool `json:"in_place,omitempty"`
+}
+
+// GPUsChange is what was changed, and the replicas started again on it.
+type GPUsChange struct {
+	Model   string   `json:"model"`
+	Profile string   `json:"profile"`
+	Engine  string   `json:"engine"`
+	Class   string   `json:"gpu_class"`
+	Before  int      `json:"before"`
+	After   int      `json:"after"`
+	Args    []string `json:"args"`
+	// PerCard is what each card holds after, when it could be worked out.
+	PerCard   string `json:"per_card,omitempty"`
+	Restarted int    `json:"restarted"`
+	InPlace   bool   `json:"in_place,omitempty"`
+	// Note is what could not be checked, said once.
+	Note string `json:"note,omitempty"`
+}
+
+// tpFlags are each engine's own spelling of a tensor-parallel width, which
+// the agent sets from the profile: a profile's own would contradict it.
+var tpFlags = map[string][]string{
+	"vllm":   {"--tensor-parallel-size", "-tp"},
+	"sglang": {"--tp-size", "--tensor-parallel-size", "--tp"},
+	"trtllm": {"--tp_size"},
+	// llama.cpp's proportions were for the cards it had.
+	"llamacpp": {"--tensor-split", "-ts"},
+}
+
+// SetGPUs changes how many cards a served model's replicas use.
+func (s *Server) SetGPUs(ctx context.Context, name string, req GPUsRequest, by string) (GPUsChange, error) {
+	if s.deps.Current == nil || s.deps.Models == nil {
+		return GPUsChange{}, errors.New("this control plane cannot change the catalog")
+	}
+	var m catalog.Model
+	found := false
+	for _, c := range s.deps.Current() {
+		if c.Name == name {
+			m, found = c, true
+		}
+	}
+	switch {
+	case !found:
+		return GPUsChange{}, fmt.Errorf("no model %q in the catalog", name)
+	case s.deps.Models.Declared(name):
+		return GPUsChange{}, fmt.Errorf("model %q is declared in the cluster's settings: change its profile there (llmgw ctl config edit)", name)
+	case req.GPUs < 1 || req.GPUs > 64:
+		return GPUsChange{}, errors.New("a replica uses 1 to 64 cards")
+	}
+	i, err := gpuProfile(m, req.Profile)
+	if err != nil {
+		return GPUsChange{}, err
+	}
+	p := m.Profiles[i]
+	before := max(p.GPUsPerNode, p.TP, 1) * max(p.Nodes, p.PP, 1)
+	out := GPUsChange{Model: name, Profile: p.Name, Engine: p.Engine, Class: p.GPUClass, Before: before, After: req.GPUs}
+	if before == req.GPUs {
+		return out, fmt.Errorf("%s's %s profile already uses %d card(s)", name, p.Name, before)
+	}
+
+	sh, note := s.shapeOf(ctx, m)
+	next, err := respread(p, req.GPUs, sh)
+	if err != nil {
+		return out, err
+	}
+	out.Args = next.Args
+	switch kv := int64(contextOf(m, p)) * sh.kvPerToken; {
+	case sh.weights == 0 || sh.kvPerToken == 0 || contextOf(m, p) == 0:
+		if note == "" {
+			note = "its context is not known, so what each card holds is not checked"
+		}
+	default:
+		need := modelplan.CardNeed(p.Engine, sh.weights, kv, req.GPUs)
+		out.PerCard = fmt.Sprintf("%.1f GB a card of %d", need, req.GPUs)
+		if budget := modelplan.CardBudget(p.GPUClass); budget > 0 {
+			out.PerCard += fmt.Sprintf(", of %.0f usable", budget)
+			if need > budget {
+				return out, fmt.Errorf("on %d %s cards each would hold %.1f GB, more than the %.0f GB they hold: "+
+					"more cards, or a shorter context (llmgw ctl model context)", req.GPUs, p.GPUClass, need, budget)
+			}
+		}
+	}
+	out.Note = note
+
+	ids := s.replicasOf(name)
+	inPlace, err := s.roomFor(p.GPUClass, req.GPUs, name, len(ids) > 0, req.InPlace)
+	if err != nil {
+		return out, err
+	}
+	profiles := slices.Clone(m.Profiles)
+	profiles[i] = next
+	if _, err := s.deps.Models.SetProfiles(ctx, name, profiles, by); err != nil {
+		return out, err
+	}
+	if len(ids) == 0 {
+		return out, nil
+	}
+	restart := s.deps.Restart
+	if inPlace {
+		restart, out.InPlace = s.deps.RestartInPlace, true
+	}
+	if restart == nil {
+		return out, errors.New("the profile is changed, and this control plane cannot start its replicas again: llmgw ctl restart")
+	}
+	if err := restart(ids...); err != nil {
+		return out, fmt.Errorf("the profile is changed, and its replicas could not be started again on it: %w", err)
+	}
+	out.Restarted = len(ids)
+	return out, nil
+}
+
+// gpuProfile is the model's profile on cards to change: the one named, or
+// its only one.
+func gpuProfile(m catalog.Model, named string) (int, error) {
+	var onCards []int
+	for i, p := range m.Profiles {
+		if named != "" && p.Name == named {
+			if p.CPUOnly() {
+				return 0, fmt.Errorf("%s's %s profile runs on CPUs; give it a gpu_class first", m.Name, p.Name)
+			}
+			return i, nil
+		}
+		if !p.CPUOnly() {
+			onCards = append(onCards, i)
+		}
+	}
+	switch {
+	case named != "":
+		return 0, fmt.Errorf("%s has no profile %q", m.Name, named)
+	case len(onCards) == 0:
+		return 0, fmt.Errorf("%s runs on CPUs only; a profile on cards needs a gpu_class", m.Name)
+	case len(onCards) > 1:
+		var names []string
+		for _, i := range onCards {
+			names = append(names, m.Profiles[i].Name)
+		}
+		return 0, fmt.Errorf("%s has several profiles on cards (%s): name one", m.Name, strings.Join(names, ", "))
+	}
+	return onCards[0], nil
+}
+
+// respread is a profile spread over n cards of one node.
+func respread(p catalog.Profile, n int, sh shape) (catalog.Profile, error) {
+	for _, f := range tpFlags[p.Engine] {
+		p.Args = dropArg(p.Args, f)
+	}
+	switch p.Engine {
+	case "vllm", "sglang", "trtllm":
+		if !modelplan.SplitsHeads(sh.kvHeads, n) {
+			return p, fmt.Errorf("the model has %d KV heads, which tensor parallel cannot spread over %d cards: "+
+				"a number that divides them, or that they divide", sh.kvHeads, n)
+		}
+		p.TP, p.GPUsPerNode = n, n
+	case "llamacpp":
+		// llama.cpp has no tensor parallel: it splits the layers over a
+		// node's cards, in proportion to their free memory.
+		p.TP, p.GPUsPerNode = 1, n
+	default:
+		return p, fmt.Errorf("a %s replica runs on one card", p.Engine)
+	}
+	// One node: a replica spanning nodes is laid out by hand, with a fabric.
+	p.Nodes, p.PP = 1, 0
+	return p, nil
+}
+
+// contextOf is the tokens a profile's KV cache holds: llama.cpp's pool, or
+// the engine's limit, or the model's context.
+func contextOf(m catalog.Model, p catalog.Profile) int {
+	switch p.Engine {
+	case "llamacpp":
+		if n := argInt(p.Args, "--ctx-size"); n > 0 {
+			return n
+		}
+	case "vllm":
+		if n := argInt(p.Args, "--max-model-len"); n > 0 {
+			return n
+		}
+	case "sglang":
+		if n := argInt(p.Args, "--context-length"); n > 0 {
+			return n
+		}
+	}
+	return m.ContextLength
+}
+
+func (s *Server) replicasOf(model string) []string {
+	var ids []string
+	if s.deps.Fleet != nil {
+		for _, r := range s.deps.Fleet.Replicas() {
+			if r.Model == model {
+				ids = append(ids, r.ID)
+			}
+		}
+	}
+	return ids
+}
+
+// roomFor says whether the new replicas fit: a node with n cards of the
+// class at all, and free beside the running ones — or, asked to start them
+// in place, once the model's own are stopped. True is in place.
+func (s *Server) roomFor(class string, n int, model string, running, inPlace bool) (bool, error) {
+	if s.deps.Pool == nil {
+		return inPlace, nil
+	}
+	own := map[string]int{}
+	if s.deps.Fleet != nil {
+		for _, r := range s.deps.Fleet.Replicas() {
+			if r.Model == model {
+				for _, sh := range r.Shards {
+					own[sh.NodeID] += len(sh.GPUIndices)
+				}
+			}
+		}
+	}
+	most, freeMost, besideOK, inPlaceOK := 0, 0, false, false
+	for _, node := range s.deps.Pool() {
+		for _, k := range modelplan.Kinds(node) {
+			if k.Class != class {
+				continue
+			}
+			most, freeMost = max(most, k.GPUs), max(freeMost, k.FreeGPUs)
+			besideOK = besideOK || k.FreeGPUs >= n
+			inPlaceOK = inPlaceOK || k.FreeGPUs+own[node.ID] >= n
+		}
+	}
+	switch {
+	case most == 0:
+		return false, fmt.Errorf("no ready node has %s cards", class)
+	case most < n:
+		return false, fmt.Errorf("no node has %d %s cards; the most one has is %d", n, class, most)
+	case !running || besideOK:
+		return false, nil
+	case !inPlaceOK:
+		return false, fmt.Errorf("no node has %d %s cards free, even with this model's own stopped (%d at most free now): "+
+			"other models hold them", n, class, freeMost)
+	case !inPlace:
+		return false, fmt.Errorf("no node has %d %s cards free beside the running replica (%d at most); "+
+			"--in-place stops each replica first, and the model does not answer while the new one loads", n, class, freeMost)
+	}
+	return true, nil
+}
+
+// dropArg is arguments without a flag and its value.
+func dropArg(args []string, flag string) []string {
+	out := make([]string, 0, len(args))
+	for i := 0; i < len(args); i++ {
+		switch {
+		case args[i] == flag:
+			i++ // its value
+		case strings.HasPrefix(args[i], flag+"="):
+		default:
+			out = append(out, args[i])
+		}
+	}
+	return out
+}

internal/control/modelserve/gpus_test.go

@@ -0,0 +1,151 @@
+package modelserve
+
+import (
+	"context"
+	"slices"
+	"strings"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/modelplan"
+)
+
+// gpuSetup is the update setup with qwen on one L40S card of llama.cpp, and
+// a pool of one node of four, of which free are free.
+func gpuSetup(t *testing.T, free int) (*Server, *updModels, *restartFleet, *[]string) {
+	t.Helper()
+	s, _, models, _, fleet := updSetup(t)
+	m := models.models["qwen"]
+	m.ContextLength = 8192
+	m.Origin = &catalog.Origin{Repo: "o/r", Files: []catalog.OriginFile{{Path: "q4.gguf", Bytes: 20 << 30}}}
+	m.Profiles = []catalog.Profile{
+		{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 8},
+		{Name: "gpu", Engine: "llamacpp", GPUClass: "l40s-48g", TP: 1, GPUsPerNode: 1,
+			Args: []string{"--ctx-size", "8192", "--tensor-split", "1"}},
+	}
+	models.models["qwen"] = m
+	s.deps.Pool = func() []modelplan.PoolNode {
+		return []modelplan.PoolNode{{ID: "g1", Class: "l40s-48g", GPUMemGB: 48, GPUs: 4, FreeGPUs: free}}
+	}
+	var inPlace []string
+	s.deps.RestartInPlace = func(ids ...string) error { inPlace = append(inPlace, ids...); return nil }
+	return s, models, fleet, &inPlace
+}
+
+func TestAModelIsSpreadOverMoreCardsAndItsReplicasStartedAgainBesideTheOld(t *testing.T) {
+	s, models, fleet, inPlace := gpuSetup(t, 3)
+	got, err := s.SetGPUs(context.Background(), "qwen", GPUsRequest{GPUs: 2}, "admin")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if got.Profile != "gpu" || got.Before != 1 || got.After != 2 || got.Restarted != 2 || got.InPlace {
+		t.Fatalf("%+v", got)
+	}
+	p := models.models["qwen"].Profiles[1]
+	// llama.cpp splits layers over a node's cards: gpus_per_node, not tp,
+	// and the old card's proportions are gone.
+	if p.GPUsPerNode != 2 || p.TP != 1 || p.Nodes != 1 || !slices.Equal(p.Args, []string{"--ctx-size", "8192"}) {
+		t.Errorf("profile %+v", p)
+	}
+	if !strings.Contains(got.PerCard, "a card of 2") {
+		t.Errorf("per card %q", got.PerCard)
+	}
+	if len(*inPlace) != 0 {
+		t.Errorf("stopped first though there was room: %v", *inPlace)
+	}
+	for _, r := range fleet.Replicas() {
+		if !strings.Contains(r.ID, "-new") {
+			t.Errorf("%s was not started again", r.ID)
+		}
+	}
+	if models.models["qwen"].Profiles[0].GPUsPerNode != 0 {
+		t.Error("the CPU profile was changed too")
+	}
+}
+
+func TestWithNoRoomBesideIt_InPlaceIsAskedFor(t *testing.T) {
+	// One card free and the model's replicas on the others: two do not fit
+	// beside them.
+	s, _, fleet, inPlace := gpuSetup(t, 1)
+	fleet.reps[0].Shards = []reconciler.Shard{{NodeID: "g1", GPUIndices: []int{1}}}
+	fleet.reps[1].Shards = []reconciler.Shard{{NodeID: "g1", GPUIndices: []int{2}}}
+	_, err := s.SetGPUs(context.Background(), "qwen", GPUsRequest{GPUs: 2}, "admin")
+	if err == nil || !strings.Contains(err.Error(), "--in-place") {
+		t.Fatalf("not asked for in place: %v", err)
+	}
+	got, err := s.SetGPUs(context.Background(), "qwen", GPUsRequest{GPUs: 2, InPlace: true}, "admin")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if !got.InPlace || len(*inPlace) != 2 {
+		t.Errorf("%+v, stopped first: %v", got, *inPlace)
+	}
+}
+
+func TestMoreCardsThanANodeHasOrOthersHoldAreRefused(t *testing.T) {
+	s, models, _, _ := gpuSetup(t, 4)
+	if _, err := s.SetGPUs(context.Background(), "qwen", GPUsRequest{GPUs: 8}, "admin"); err == nil ||
+		!strings.Contains(err.Error(), "the most one has is 4") {
+		t.Errorf("eight cards on a node of four: %v", err)
+	}
+	s, _, _, _ = gpuSetup(t, 0)
+	if _, err := s.SetGPUs(context.Background(), "qwen", GPUsRequest{GPUs: 2, InPlace: true}, "admin"); err == nil ||
+		!strings.Contains(err.Error(), "other models hold them") {
+		t.Errorf("cards other models hold: %v", err)
+	}
+	if models.models["qwen"].Profiles[1].GPUsPerNode != 1 {
+		t.Error("a refused change was kept")
+	}
+}
+
+func TestFewerCardsThanTheModelNeedsAreRefusedWithTheFigures(t *testing.T) {
+	s, models, _, _ := gpuSetup(t, 3)
+	m := models.models["qwen"]
+	m.Profiles[1].GPUsPerNode, m.Origin.Files[0].Bytes = 4, 120<<30
+	models.models["qwen"] = m
+	_, err := s.SetGPUs(context.Background(), "qwen", GPUsRequest{GPUs: 2}, "admin")
+	if err == nil || !strings.Contains(err.Error(), "GB") {
+		t.Errorf("120 GB on two 48 GB cards: %v", err)
+	}
+}
+
+func TestTensorParallelFollowsTheKVHeads(t *testing.T) {
+	p := catalog.Profile{Name: "gpu", Engine: "vllm", GPUClass: "h100-80g", TP: 2, GPUsPerNode: 2,
+		Args: []string{"--tensor-parallel-size", "2", "--max-model-len", "32768"}}
+	if _, err := respread(p, 3, shape{kvHeads: 8}); err == nil || !strings.Contains(err.Error(), "8 KV heads") {
+		t.Errorf("three ways for eight heads: %v", err)
+	}
+	next, err := respread(p, 4, shape{kvHeads: 8})
+	if err != nil {
+		t.Fatal(err)
+	}
+	// The agent sets the width from tp; the profile's own spelling went.
+	if next.TP != 4 || next.GPUsPerNode != 4 || !slices.Equal(next.Args, []string{"--max-model-len", "32768"}) {
+		t.Errorf("%+v", next)
+	}
+	for _, eng := range []string{"comfyui", "strata"} {
+		if _, err := respread(catalog.Profile{Name: "x", Engine: eng}, 2, shape{}); err == nil {
+			t.Errorf("%s spread over two cards", eng)
+		}
+	}
+}
+
+func TestAModelOnCPUsOrWithSeveralCardProfilesSaysWhy(t *testing.T) {
+	s, models, _, _ := gpuSetup(t, 3)
+	m := models.models["qwen"]
+	m.Profiles = append(m.Profiles, catalog.Profile{Name: "gpu2", Engine: "vllm", GPUClass: "l40s-48g", TP: 1})
+	models.models["qwen"] = m
+	if _, err := s.SetGPUs(context.Background(), "qwen", GPUsRequest{GPUs: 2}, "admin"); err == nil ||
+		!strings.Contains(err.Error(), "name one") {
+		t.Errorf("two profiles on cards and none named: %v", err)
+	}
+	if _, err := s.SetGPUs(context.Background(), "qwen", GPUsRequest{GPUs: 2, Profile: "cpu"}, "admin"); err == nil ||
+		!strings.Contains(err.Error(), "gpu_class") {
+		t.Errorf("a CPU profile: %v", err)
+	}
+	if _, err := s.SetGPUs(context.Background(), "qwen", GPUsRequest{GPUs: 1, Profile: "gpu"}, "admin"); err == nil ||
+		!strings.Contains(err.Error(), "already") {
+		t.Errorf("the same count: %v", err)
+	}
+}

internal/control/modelserve/modelserve.go

@@ -82,6 +82,9 @@ type Deps struct {
 	Current func() []catalog.Model
 	Models  Models
 	Restart func(ids ...string) error
+	// RestartInPlace stops each replica before its replacement starts, for
+	// when there is no room beside it.
+	RestartInPlace func(ids ...string) error
 
 	// Pool is the fleet's ready nodes as a plan sees them: what decides
 	// which file suits the hardware, and how it is spread.
@@ -131,6 +134,9 @@ type Request struct {
 	// Fallbacks adds the next best ways, lower in priority, so a model whose
 	// first choice is busy runs on another rather than waiting.
 	Fallbacks bool `json:"fallbacks,omitempty"`
+	// Engine is the engine the file runs on: "strata" for a family Strata
+	// runs, "llamacpp" to keep it off Strata; empty, the plan's best.
+	Engine string `json:"engine,omitempty"`
 	// Name is the catalog name; the repository suggests one otherwise.
 	Name        string   `json:"name,omitempty"`
 	Aliases     []string `json:"aliases,omitempty"`
@@ -485,6 +491,23 @@ func (s *Server) run(j *Job) {
 	}
 
 	// 3. Whether the nodes' engine loads it, and how much memory it needs.
+	if layout[0].Engine == "strata" {
+		// Strata prepares it on the node from the directory of its files,
+		// and keeps the experts in the RAM the plan counted.
+		if !strings.HasSuffix(source, "/") {
+			source = strings.TrimSuffix(source, mirror.LastPathElement(source))
+		}
+		if !req.sized {
+			req.ContextLength = strataContext
+		}
+		s.step(j, "check", Done, fmt.Sprintf("Strata's %s %s, prepared on the node; %.0f GB of RAM for its experts",
+			layout[0].StrataFamily, layout[0].StrataSize, layout[0].NeedGB))
+		s.mu.Lock()
+		j.Request = req
+		s.mu.Unlock()
+		s.addAndStart(ctx, j, repo, set, name, source, profilesFor(layout, req, 0, 0), hardwareExists(layout))
+		return
+	}
 	if req.Format == "safetensors" {
 		// vLLM reads the model's shape from its config.json, which came
 		// with the weights.
@@ -563,10 +586,21 @@ func chooseLayout(req Request, plan Plan, planned bool, set hf.FileSet) ([]model
 	}
 	var opts []modelplan.Option
 	for _, o := range plan.Options {
-		if o.Artifact == artifact {
+		onStrata := o.Engine == "strata"
+		if o.Artifact == artifact && (req.Engine == "" || (req.Engine == "strata") == onStrata) {
 			opts = append(opts, o)
 		}
 	}
+	if req.Engine == "strata" {
+		// Strata's way only, on the class asked for if one was.
+		i := slices.IndexFunc(opts, func(o modelplan.Option) bool { return req.GPUClass == "" || o.Class == req.GPUClass })
+		if i < 0 {
+			return nil, fmt.Errorf("Strata cannot run %s here: it runs a few families only (`llmgw ctl model plan %s` "+
+				"says whether this is one), on a node with Strata installed, a card of 12 GB or more and the RAM for its experts",
+				set.Name, plan.Repo)
+		}
+		return []modelplan.Option{opts[i]}, nil
+	}
 	var out []modelplan.Option
 	switch {
 	case req.CPU || req.GPUClass != "":
@@ -597,7 +631,7 @@ func chooseLayout(req Request, plan Plan, planned bool, set hf.FileSet) ([]model
 	default:
 		out = append(out, modelplan.Option{Artifact: artifact, Format: req.Format, Engine: engine, CPU: true})
 	}
-	if req.Fallbacks {
+	if req.Fallbacks && out[0].Engine != "strata" {
 		shape := func(o modelplan.Option) string {
 			return fmt.Sprintf("%v/%s/%d/%d", o.CPU, o.Class, o.GPUsPerNode, o.Nodes)
 		}
@@ -622,6 +656,12 @@ func profilesFor(layout []modelplan.Option, req Request, memGB, cores int) []cat
 	for i, o := range layout {
 		p := catalog.Profile{Engine: o.Engine, Weight: 1, Priority: 100 - 10*i, Env: req.Env}
 		switch {
+		case o.Engine == "strata":
+			p.Name = "strata-" + o.Class
+			p.GPUClass, p.GPUsPerNode = o.Class, 1
+			p.Args = []string{"--family", o.StrataFamily, "--size", o.StrataSize,
+				"--context", strconv.Itoa(req.ContextLength), "--vision", "no"}
+			p.MinMemoryGB = int(o.NeedGB)
 		case o.CPU:
 			p.Name = "cpu"
 			p.Args = llamaArgs(req)
@@ -671,6 +711,10 @@ func llamaArgs(req Request) []string {
 	return args
 }
 
+// strataContext is a Strata model's context when its caller did not say:
+// Strata's own default.
+const strataContext = 32768
+
 // An embedding model's slots, when its caller did not size them: passages
 // of up to a thousand tokens, thirty-two at a time.
 const (
@@ -845,6 +889,9 @@ func (s *Server) mirror(ctx context.Context, j *Job, repo hf.Repo, set hf.FileSe
 		}
 		resp, err := s.deps.Mirror.Fetch(ctx, mirror.FetchRequest{
 			Source: repo.DownloadURL("", f.Path), Model: name, Version: version, File: file,
+			// The mirror deletes a copy that is not what the registry
+			// published, rather than holding it for the nodes.
+			SHA256: f.SHA256,
 		})
 		// Already there is not a failure: the same commit fetched before.
 		if err != nil && !strings.Contains(err.Error(), "already") {

internal/control/modelserve/plan.go

@@ -9,6 +9,7 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/modelplan"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/strata"
 )
 
 // Planning: which of a repository's files suits the cluster's hardware, and
@@ -168,6 +169,18 @@ func (s *Server) plan(ctx context.Context, repo hf.Repo, req PlanRequest) (Plan,
 	if !p.Servable {
 		p.Options = nil
 	}
+	// A family Strata runs: its files on one card with the experts in RAM.
+	// Strata reads them itself, whether or not llama.cpp could.
+	if fam, ok := strata.ForRepo(repo.ID); ok && p.Format == "gguf" {
+		if so := modelplan.StrataOptions(fam.Name, arts, s.deps.Pool()); len(so) > 0 {
+			p.Options = append(p.Options, so...)
+			modelplan.Rank(p.Options)
+			if !p.Servable {
+				p.Note = strings.TrimSpace(p.Note + " llama.cpp cannot run it (" + p.Reason + "); Strata can.")
+				p.Servable, p.Reason = true, ""
+			}
+		}
+	}
 	for _, a := range arts {
 		pa := PlannedArtifact{Artifact: a}
 		for i := range p.Options {

internal/control/modelserve/strata_test.go

@@ -0,0 +1,37 @@
+package modelserve
+
+import (
+	"slices"
+	"strings"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/modelplan"
+)
+
+func TestAStrataWayIsChosenAndMadeAProfile(t *testing.T) {
+	set := hf.FileSet{Name: "Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00001-of-00002.gguf"}
+	strataWay := modelplan.Option{Artifact: set.Name, Format: "gguf", Engine: "strata", Class: "6000-24g",
+		GPUsPerNode: 1, Nodes: 1, NeedGB: 52, HaveGB: 213, Hosts: 1, StrataFamily: "qwen", StrataSize: "IQ2_XS"}
+	cpuWay := modelplan.Option{Artifact: set.Name, Format: "gguf", Engine: "llamacpp", CPU: true, NeedGB: 75, HaveGB: 213, Hosts: 1}
+	plan := Plan{Repo: "ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF", Options: []modelplan.Option{strataWay, cpuWay}}
+
+	got, err := chooseLayout(Request{Format: "gguf", Fallbacks: true}, plan, true, set)
+	if err != nil || len(got) != 1 || got[0].Engine != "strata" {
+		t.Fatalf("the plan's best, without llama.cpp fallbacks: %+v %v", got, err)
+	}
+	if got, _ := chooseLayout(Request{Format: "gguf", Engine: "llamacpp"}, plan, true, set); len(got) != 1 || !got[0].CPU {
+		t.Errorf("kept off Strata: %+v", got)
+	}
+	if _, err := chooseLayout(Request{Format: "gguf", Engine: "strata", GPUClass: "h100-80g"}, plan, true, set); err == nil {
+		t.Error("Strata on a class it was not planned for")
+	}
+
+	ps := profilesFor(got, Request{ContextLength: 32768}, 0, 0)
+	p := ps[0]
+	if p.Engine != "strata" || p.GPUClass != "6000-24g" || p.GPUsPerNode != 1 || p.MinMemoryGB != 52 ||
+		!slices.Equal(p.Args, []string{"--family", "qwen", "--size", "IQ2_XS", "--context", "32768", "--vision", "no"}) ||
+		!strings.HasPrefix(p.Name, "strata-") {
+		t.Errorf("%+v", p)
+	}
+}

internal/control/modelserve/updates.go

@@ -35,6 +35,7 @@ type Models interface {
 	SetSource(ctx context.Context, name, source string, origin *catalog.Origin, by string) (catalog.Model, error)
 	SetOrigin(ctx context.Context, name string, origin *catalog.Origin) error
 	SetContext(ctx context.Context, name string, contextLength, maxOutput int, profiles []catalog.Profile, by string) (catalog.Model, error)
+	SetProfiles(ctx context.Context, name string, profiles []catalog.Profile, by string) (catalog.Model, error)
 	// For a canary: the candidate added beside the model and taken out
 	// again, and the share of requests it is given.
 	Add(ctx context.Context, m catalog.Model, by string) error

internal/control/modelserve/updates_test.go

@@ -112,6 +112,15 @@ func (c *updModels) SetContext(_ context.Context, n string, ctxLen, maxOut int,
 	c.models[n] = m
 	return m, nil
 }
+
+func (c *updModels) SetProfiles(_ context.Context, n string, ps []catalog.Profile, _ string) (catalog.Model, error) {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	m := c.models[n]
+	m.Profiles = ps
+	c.models[n] = m
+	return m, nil
+}
 func (c *updModels) all() []catalog.Model {
 	c.mu.Lock()
 	defer c.mu.Unlock()

internal/control/modelsvc/modelsvc.go

@@ -27,6 +27,7 @@ import (
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/strata"
 )
 
 // Store is the persistence this needs.
@@ -51,6 +52,8 @@ type Service struct {
 	// check refuses a model that cannot run before it is added; see
 	// SetCheck.
 	check func(context.Context, catalog.Model) error
+	// registry pins what is added from it to a commit; nil pins nothing.
+	registry Registry
 
 	// federated are the models federated clusters offer; see SetFederated.
 	federated func() []catalog.Model
@@ -127,12 +130,31 @@ func (s *Service) add(ctx context.Context, m catalog.Model, by string, onlyNew b
 	// fits everywhere as far as placement can tell — however many of it
 	// there already are. Required of what is added now; models stored before
 	// this still load, and say what they are missing.
+	m.Profiles = slices.Clone(m.Profiles)
+	for i := range m.Profiles {
+		// Strata keeps a model's experts in the host's memory, tens of
+		// gigabytes of it: placed as if it took none, two replicas, or one
+		// beside a CPU model, filled a node until one was killed. What its
+		// size takes, from Strata's table, unless the model says.
+		if p := &m.Profiles[i]; p.Engine == "strata" && p.MinMemoryGB <= 0 {
+			size, err := strata.SizeOf(strata.ArgValue(p.Args, "--family"), strata.ArgValue(p.Args, "--size"))
+			if err != nil {
+				return fmt.Errorf("model %q profile %q: %w", m.Name, p.Name, err)
+			}
+			p.MinMemoryGB = size.MemoryGB()
+		}
+	}
 	for _, p := range m.Profiles {
 		if p.CPUOnly() && p.MinMemoryGB <= 0 {
 			return fmt.Errorf("model %q profile %q runs without a GPU and declares no memory (min_memory_gb): "+
 				"placement could not tell when a node is full", m.Name, p.Name)
 		}
 	}
+	if s.registry != nil {
+		if err := pin(ctx, s.registry, &m); err != nil {
+			return err
+		}
+	}
 	if s.check != nil {
 		if err := s.check(ctx, m); err != nil {
 			return err
@@ -228,6 +250,8 @@ func (s *Service) Remove(ctx context.Context, name string) error {
 func (s *Service) Scale(ctx context.Context, name string, sc catalog.Scaling, by string) (catalog.Model, error) {
 	final, err := s.update(ctx, name, by, "change its scaling", func(m *catalog.Model) error {
 		m.Scaling = sc
+		// A number set is what it runs now: a stopped model is started by it.
+		m.Stopped = nil
 		return nil
 	})
 	if err == nil {
@@ -237,6 +261,39 @@ func (s *Service) Scale(ctx context.Context, name string, sc catalog.Scaling, by
 	return final, err
 }
 
+// Stop takes a model out of service and keeps it: none of its replicas run,
+// and what it was scaled to is kept for Start.
+func (s *Service) Stop(ctx context.Context, name, by string) (catalog.Model, error) {
+	final, err := s.update(ctx, name, by, "stop it", func(m *catalog.Model) error {
+		if m.Stopped != nil {
+			return fmt.Errorf("model %q is already stopped; llmgw ctl model start %s runs it again", name, name)
+		}
+		was := m.Scaling
+		m.Stopped = &was
+		m.Scaling = catalog.Scaling{IdleTimeout: was.IdleTimeout}
+		return nil
+	})
+	if err == nil {
+		s.log.Info("model stopped", "model", name, "by", by)
+	}
+	return final, err
+}
+
+// Start runs a stopped model again, scaled as it was when it was stopped.
+func (s *Service) Start(ctx context.Context, name, by string) (catalog.Model, error) {
+	final, err := s.update(ctx, name, by, "start it", func(m *catalog.Model) error {
+		if m.Stopped == nil {
+			return fmt.Errorf("model %q is not stopped; llmgw ctl scale changes how many replicas it runs", name)
+		}
+		m.Scaling, m.Stopped = *m.Stopped, nil
+		return nil
+	})
+	if err == nil {
+		s.log.Info("model started", "model", name, "min", final.Scaling.Min, "max", final.Scaling.Max, "by", by)
+	}
+	return final, err
+}
+
 // Resources sets what a model's profile reserves of a node: host memory, and
 // the CPU cores a replica without a GPU computes on. An empty profile name
 // sets every profile.

internal/control/modelsvc/modelsvc_test.go

@@ -400,3 +400,61 @@ func TestAddNewLeavesAModelAddedMeanwhile(t *testing.T) {
 		t.Errorf("a new model: %v", err)
 	}
 }
+
+func TestAStoppedModelRunsNoneAndStartsAsItWas(t *testing.T) {
+	s, _, rec := newService(t)
+	if err := s.Add(t.Context(), model("added"), "ilker"); err != nil {
+		t.Fatal(err)
+	}
+	if _, err := s.Scale(t.Context(), "added", catalog.Scaling{Min: 2, Max: 4, IdleTimeout: 20 * time.Minute}, "ilker"); err != nil {
+		t.Fatal(err)
+	}
+	got, err := s.Stop(t.Context(), "added", "ilker")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if got.Scaling.Min != 0 || got.Scaling.Max != 0 || got.Stopped == nil || got.Stopped.Min != 2 {
+		t.Fatalf("stopped: %+v %+v", got.Scaling, got.Stopped)
+	}
+	if m, _ := rec.last.Resolve("added"); m.Scaling.Max != 0 || m.Scaling.OnDemand() {
+		t.Errorf("the catalog in force runs it: %+v", m.Scaling)
+	}
+	if _, err := s.Stop(t.Context(), "added", "ilker"); err == nil {
+		t.Error("stopped twice")
+	}
+	got, err = s.Start(t.Context(), "added", "ilker")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if got.Scaling.Min != 2 || got.Scaling.Max != 4 || got.Scaling.IdleTimeout != 20*time.Minute || got.Stopped != nil {
+		t.Errorf("started: %+v %+v", got.Scaling, got.Stopped)
+	}
+	if _, err := s.Start(t.Context(), "added", "ilker"); err == nil {
+		t.Error("started what was not stopped")
+	}
+	// A scale while stopped starts it at that number.
+	_, _ = s.Stop(t.Context(), "added", "ilker")
+	if got, _ := s.Scale(t.Context(), "added", catalog.Scaling{Min: 1}, "ilker"); got.Stopped != nil {
+		t.Error("still stopped after a scale")
+	}
+}
+
+func TestAStrataModelReservesTheMemoryItsExpertsTake(t *testing.T) {
+	s, _, rec := newService(t)
+	m := model("big")
+	m.Source = "http://mirror:8081/big/v1/"
+	m.Profiles = []catalog.Profile{{Name: "gpu", Engine: "strata", GPUClass: "6000-24g", GPUsPerNode: 1,
+		Args: []string{"--family", "qwen", "--size", "IQ2_XS"}}}
+	if err := s.Add(t.Context(), m, "ilker"); err != nil {
+		t.Fatal(err)
+	}
+	got, _ := rec.last.Resolve("big")
+	if got.Profiles[0].MinMemoryGB != 52 {
+		t.Errorf("IQ2_XS reserves %d GB, want Strata's 48 and 4 for the engine", got.Profiles[0].MinMemoryGB)
+	}
+	m.Name = "big2"
+	m.Profiles[0].Args = []string{"--family", "qwen", "--size", "IQ9"}
+	if err := s.Add(t.Context(), m, "ilker"); err == nil || !strings.Contains(err.Error(), "comes in") {
+		t.Errorf("an unknown size: %v", err)
+	}
+}

internal/control/modelsvc/pin.go

@@ -0,0 +1,133 @@
+package modelsvc
+
+import (
+	"context"
+	"fmt"
+	"regexp"
+	"slices"
+	"strings"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
+)
+
+// A model added with weights from the registry is pinned to the commit they
+// were at: a file named by hf: or by a resolve URL on a branch becomes that
+// commit's URL, and a repository an engine downloads itself is given
+// --revision. The registry's sha256 of each file goes in the model's origin,
+// for the nodes to check what they fetch against. Without it, a new upload
+// to the repository reached the nodes that fetched after it, unseen, beside
+// nodes that still had the old one.
+
+// Registry is where a repository's commit and files are looked up.
+type Registry interface {
+	RepoAt(ctx context.Context, id, rev string) (hf.Repo, error)
+	Base() string
+}
+
+// SetRegistry has every model added from now on pinned against it.
+func (s *Service) SetRegistry(r Registry) { s.registry = r }
+
+// engineDownloads are the engines that download a repository named by its id
+// themselves, and take --revision.
+var engineDownloads = map[string]bool{"vllm": true, "sglang": true}
+
+var repoID = regexp.MustCompile(`^[A-Za-z0-9][\w.-]*/[\w.-]+$`)
+
+// pin pins a model's registry sources, in place. One already given an origin
+// — served through the mirror, which records it — is left as it is.
+func pin(ctx context.Context, reg Registry, m *catalog.Model) error {
+	if m.Origin != nil {
+		return nil
+	}
+	repos := map[string]hf.Repo{}
+	lookup := func(id, rev string) (hf.Repo, error) {
+		key := id + "@" + rev
+		if r, ok := repos[key]; ok {
+			return r, nil
+		}
+		r, err := reg.RepoAt(ctx, id, rev)
+		if err != nil {
+			return r, fmt.Errorf("model %q: looking up %s in the registry, to pin it to a commit: %w", m.Name, id, err)
+		}
+		repos[key] = r
+		return r, nil
+	}
+	record := func(r hf.Repo, files ...hf.File) {
+		if m.Origin == nil {
+			m.Origin = &catalog.Origin{Repo: r.ID, Revision: r.Revision}
+		}
+		if m.Origin.Repo != r.ID || m.Origin.Revision != r.Revision {
+			return // one origin a model; a second repository is pinned, not recorded
+		}
+		for _, f := range files {
+			m.Origin.Files = append(m.Origin.Files, catalog.OriginFile{Path: f.Path, SHA256: f.SHA256, Bytes: f.Bytes})
+		}
+	}
+
+	// One file: its commit's URL.
+	pinned := map[string]string{}
+	for _, src := range m.WeightSources() {
+		id, rev, file, ok := hf.FileSource(src, reg.Base())
+		if !ok {
+			continue
+		}
+		r, err := lookup(id, rev)
+		if err != nil {
+			return err
+		}
+		i := slices.IndexFunc(r.Files, func(f hf.File) bool { return f.Path == file })
+		if i < 0 {
+			return fmt.Errorf("model %q: %s has no file %s at %s", m.Name, id, file, r.ShortRevision())
+		}
+		pinned[src] = r.DownloadURL(reg.Base(), file)
+		record(r, r.Files[i])
+	}
+	if to, ok := pinned[m.Source]; ok {
+		m.Source = to
+	}
+	for i := range m.Profiles {
+		if to, ok := pinned[m.Profiles[i].Source]; ok {
+			m.Profiles[i].Source = to
+		}
+	}
+
+	// A repository an engine downloads: --revision.
+	for i := range m.Profiles {
+		p := &m.Profiles[i]
+		src := m.SourceOf(*p)
+		if !engineDownloads[p.Engine] || !repoID.MatchString(src) {
+			continue
+		}
+		r, err := lookup(src, argValue(p.Args, "--revision"))
+		if err != nil {
+			return err
+		}
+		if argValue(p.Args, "--revision") == "" {
+			p.Args = append(slices.Clone(p.Args), "--revision", r.Revision)
+		}
+		var weights []hf.File
+		for _, f := range r.Files {
+			if f.IsWeights() {
+				weights = append(weights, f)
+			}
+		}
+		if m.Origin == nil || m.Origin.Repo != r.ID || len(m.Origin.Files) == 0 {
+			record(r, weights...)
+		}
+	}
+	return nil
+}
+
+// argValue is a flag's value in arguments, "" when absent.
+func argValue(args []string, flag string) string {
+	for i, a := range args {
+		if a == flag && i+1 < len(args) {
+			return args[i+1]
+		}
+		if v, ok := strings.CutPrefix(a, flag+"="); ok {
+			return v
+		}
+	}
+	return ""
+}

internal/control/modelsvc/pin_test.go

@@ -0,0 +1,88 @@
+package modelsvc
+
+import (
+	"context"
+	"errors"
+	"slices"
+	"strings"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
+)
+
+const commit = "0123456789abcdef0123456789abcdef01234567"
+
+type fakeRegistry struct{ asked []string }
+
+func (f *fakeRegistry) Base() string { return "https://huggingface.co" }
+
+func (f *fakeRegistry) RepoAt(_ context.Context, id, rev string) (hf.Repo, error) {
+	f.asked = append(f.asked, id+"@"+rev)
+	if id == "gone/repo" {
+		return hf.Repo{}, errors.New("404")
+	}
+	return hf.Repo{ID: id, Revision: commit, Files: []hf.File{
+		{Path: "q4.gguf", Bytes: 10, SHA256: "aaaa"},
+		{Path: "model-00001-of-00002.safetensors", Bytes: 20, SHA256: "bbbb"},
+		{Path: "config.json", Bytes: 1},
+	}}, nil
+}
+
+func TestAFileFromTheRegistryIsPinnedToItsCommit(t *testing.T) {
+	for _, src := range []string{"hf:o/r/q4.gguf", "https://huggingface.co/o/r/resolve/main/q4.gguf"} {
+		m := catalog.Model{Name: "q", Source: src, Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp"}}}
+		if err := pin(t.Context(), &fakeRegistry{}, &m); err != nil {
+			t.Fatal(err)
+		}
+		if m.Source != "https://huggingface.co/o/r/resolve/"+commit+"/q4.gguf" {
+			t.Errorf("%s became %s", src, m.Source)
+		}
+		if m.Origin == nil || m.Origin.Revision != commit || m.OriginSHA256(m.Source) != "aaaa" {
+			t.Errorf("%s: origin %+v", src, m.Origin)
+		}
+	}
+}
+
+func TestARepositoryAnEngineDownloadsIsGivenItsRevision(t *testing.T) {
+	m := catalog.Model{Name: "q", Source: "o/r", Profiles: []catalog.Profile{
+		{Name: "gpu", Engine: "vllm", Args: []string{"--max-model-len", "4096"}},
+		{Name: "cpu", Engine: "llamacpp", Source: "hf:o/r/q4.gguf"}}}
+	if err := pin(t.Context(), &fakeRegistry{}, &m); err != nil {
+		t.Fatal(err)
+	}
+	if a := m.Profiles[0].Args; !slices.Equal(a, []string{"--max-model-len", "4096", "--revision", commit}) {
+		t.Errorf("vllm args %v", a)
+	}
+	if !strings.Contains(m.Profiles[1].Source, commit) {
+		t.Errorf("the llama.cpp profile's file: %s", m.Profiles[1].Source)
+	}
+	// A revision given is the one looked up, and kept.
+	reg := &fakeRegistry{}
+	m = catalog.Model{Name: "q", Source: "o/r", Profiles: []catalog.Profile{{Name: "gpu", Engine: "vllm", Args: []string{"--revision", "v2"}}}}
+	if err := pin(t.Context(), reg, &m); err != nil {
+		t.Fatal(err)
+	}
+	if !slices.Contains(reg.asked, "o/r@v2") || argValue(m.Profiles[0].Args, "--revision") != "v2" {
+		t.Errorf("asked %v, args %v", reg.asked, m.Profiles[0].Args)
+	}
+}
+
+func TestWhatIsNotTheRegistrysIsLeftAndWhatCannotBePinnedIsRefused(t *testing.T) {
+	m := catalog.Model{Name: "q", Source: "http://10.0.0.146:8081/q/abc/q4.gguf", Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp"}}}
+	reg := &fakeRegistry{}
+	if err := pin(t.Context(), reg, &m); err != nil || len(reg.asked) != 0 || m.Origin != nil {
+		t.Errorf("a mirror's file: %v %v %+v", err, reg.asked, m.Origin)
+	}
+	// Served through the mirror, with its origin: left alone.
+	m = catalog.Model{Name: "q", Source: "hf:o/r/q4.gguf", Origin: &catalog.Origin{Repo: "o/r"}}
+	if err := pin(t.Context(), reg, &m); err != nil || m.Source != "hf:o/r/q4.gguf" {
+		t.Errorf("an origin given was not kept: %v %s", err, m.Source)
+	}
+	for _, src := range []string{"hf:gone/repo/q4.gguf", "hf:o/r/missing.gguf"} {
+		m = catalog.Model{Name: "q", Source: src}
+		if err := pin(t.Context(), reg, &m); err == nil {
+			t.Errorf("%s was not refused", src)
+		}
+	}
+}

internal/docgen/reference_gen.go

@@ -244,6 +244,7 @@ var Reference = []Block{
 		{Key: "allow_pickle", Type: "bool", Doc: "AllowPickle lets the nodes fetch this model's weights, or its\nadapters', in a pickle format (.bin, .pt, .pth, .ckpt), which run code\nas they are loaded. Off, such a source is refused when the model is\nadded and by the node before it fetches anything. For a model that has\nno safetensors or GGUF form and whose files were checked.", Nested: ""},
 		{Key: "profiles", Type: "[]Profile", Doc: "", Nested: ""},
 		{Key: "scaling", Type: "Scaling", Doc: "", Nested: "catalog.Scaling"},
+		{Key: "stopped", Type: "*Scaling", Doc: "Stopped is the scaling a model had when an administrator stopped it:\nkept in the catalog with none running (Scaling 0/0) until started\nagain with this. Nil for a model not stopped.", Nested: "catalog.Scaling"},
 		{Key: "class", Type: "string", Doc: "Class is how much this model's capacity is worth against another's,\nwhen there is not enough for both: critical, standard or opportunistic.\nEmpty is standard, or opportunistic for a model loaded on demand. See\nEffectiveClass.", Nested: ""},
 		{Key: "owner", Type: "string", Doc: "Owner is the part of the tenancy tree whose capacity quota this model\ncounts against, written org:ID, team:ID or project:ID. Empty is shared:\nheld by the cluster, counted against nobody.", Nested: ""},
 		{Key: "external", Type: "*External", Doc: "External is a model served by someone else — an OpenAI-compatible API\noutside the cluster — rather than by replicas placed here. Such a model\nhas no profiles and nothing is placed for it; the router sends its\nrequests to the provider. Most useful named in another model's\nfallbacks: used only when the cluster has no capacity left. Its\nprompts leave the cluster, so which keys may name it is worth an\nallowlist.", Nested: "catalog.External"},

internal/hf/hf.go

@@ -715,3 +715,35 @@ func ModelName(repoID string) string {
 	name = strings.TrimSuffix(name, "-gguf")
 	return strings.ToLower(name)
 }
+
+// FileSource reads a source naming one file of a registry repository: hf:
+// with the file, or a resolve URL on the registry. A resolve URL already at
+// a commit is still looked up, for the file's sha256.
+func FileSource(src, base string) (id, rev, file string, ok bool) {
+	if rest, found := strings.CutPrefix(src, "hf:"); found {
+		parts := strings.Split(rest, "/")
+		if len(parts) < 3 {
+			return "", "", "", false
+		}
+		return parts[0] + "/" + parts[1], "", strings.Join(parts[2:], "/"), true
+	}
+	u, err := url.Parse(src)
+	if err != nil || (u.Scheme != "https" && u.Scheme != "http") || !isRegistryHost(u.Host, base) {
+		return "", "", "", false
+	}
+	parts := strings.Split(strings.Trim(u.Path, "/"), "/")
+	if len(parts) < 5 || parts[2] != "resolve" {
+		return "", "", "", false
+	}
+	return parts[0] + "/" + parts[1], parts[3], strings.Join(parts[4:], "/"), true
+}
+
+// isRegistryHost says whether a host is the registry's: Hugging Face's own, or
+// the endpoint the cluster names instead.
+func isRegistryHost(host, base string) bool {
+	if host == "huggingface.co" {
+		return true
+	}
+	b, err := url.Parse(base)
+	return err == nil && b.Host != "" && b.Host == host
+}

internal/kb/kbsync/api.go

@@ -63,6 +63,8 @@ type Collection struct {
 	EmbedModel string `json:"embed_model"`
 	Dims       int    `json:"dims"`
 	Writable   bool   `json:"writable"`
+	// ServerEmbeds is a gateway that embeds a document sent as text alone.
+	ServerEmbeds bool `json:"server_embeds"`
 }
 
 // RemoteDoc is a document the collection holds.
@@ -82,8 +84,9 @@ type Manifest struct {
 
 // ChunkOut is one chunk as sent.
 type ChunkOut struct {
-	Text      string    `json:"text"`
-	Embedding []float32 `json:"embedding"`
+	Text string `json:"text"`
+	// Embedding is empty when the gateway embeds the chunk.
+	Embedding []float32 `json:"embedding,omitempty"`
 }
 
 // DocOut is one document as sent.

internal/kb/kbsync/sync.go

@@ -46,8 +46,11 @@ type Options struct {
 	Retry bool
 	// Workers is how many files are read and sent at once; 0 is 4.
 	Workers int
-	Extract extract.Options
-	Chunk   chunk.Options
+	// EmbedHere embeds the chunks on this machine even when the gateway
+	// would: a vector up and down for each, rather than the text alone.
+	EmbedHere bool
+	Extract   extract.Options
+	Chunk     chunk.Options
 }
 
 // Event is something a sync did to one document, for showing progress.
@@ -359,8 +362,12 @@ func (s *syncer) send(ctx context.Context, f file, name, digest, text string) {
 		texts[i] = c.Text()
 		size += int64(len(texts[i]))
 	}
-	// About ten bytes of JSON a dimension.
-	size += int64(len(cs) * s.coll.Dims * 10)
+	// Sent as text alone when the gateway embeds it; else about ten bytes
+	// of JSON a dimension with each chunk.
+	serverEmbeds := s.coll.ServerEmbeds && !s.o.EmbedHere
+	if !serverEmbeds {
+		size += int64(len(cs) * s.coll.Dims * 10)
+	}
 	if size > maxDocJSON {
 		s.fail(ctx, f, name, digest, fmt.Errorf("%d MiB to send is over the gateway's limit for one document; split the file", size>>20))
 		return
@@ -369,12 +376,19 @@ func (s *syncer) send(ctx context.Context, f file, name, digest, text string) {
 		s.emit(Event{Kind: "would-send", Name: name, Chunks: len(cs), Bytes: size})
 		return
 	}
+	out := DocOut{Name: name, Digest: digest, Bytes: f.size, Chunks: make([]ChunkOut, len(cs))}
+	if serverEmbeds {
+		for i := range cs {
+			out.Chunks[i] = ChunkOut{Text: texts[i]}
+		}
+		s.put(ctx, f, name, digest, out, len(cs), size)
+		return
+	}
 	vecs, err := s.c.Embed(ctx, s.coll.EmbedModel, texts)
 	if err != nil {
 		s.fail(ctx, f, name, digest, fmt.Errorf("embedding with %s: %w", s.coll.EmbedModel, err))
 		return
 	}
-	out := DocOut{Name: name, Digest: digest, Bytes: f.size, Chunks: make([]ChunkOut, len(cs))}
 	for i := range cs {
 		if len(vecs[i]) != s.coll.Dims {
 			s.fail(ctx, f, name, digest, fmt.Errorf("%s gave %d dimensions and the collection takes %d",
@@ -383,12 +397,17 @@ func (s *syncer) send(ctx context.Context, f file, name, digest, text string) {
 		}
 		out.Chunks[i] = ChunkOut{Text: texts[i], Embedding: vecs[i]}
 	}
+	s.put(ctx, f, name, digest, out, len(cs), size)
+}
+
+// put sends a document and records it.
+func (s *syncer) put(ctx context.Context, f file, name, digest string, out DocOut, chunks int, size int64) {
 	if _, err := s.c.Put(ctx, s.coll.ID, out); err != nil {
 		s.fail(ctx, f, name, digest, err)
 		return
 	}
-	s.record(ctx, s.entry(f, name, digest, Sent, "", len(cs)))
-	s.emit(Event{Kind: "sent", Name: name, Chunks: len(cs), Bytes: size})
+	s.record(ctx, s.entry(f, name, digest, Sent, "", chunks))
+	s.emit(Event{Kind: "sent", Name: name, Chunks: chunks, Bytes: size})
 }
 
 func (s *syncer) fail(ctx context.Context, f file, name, digest string, err error) {

internal/kb/kbsync/sync_test.go

@@ -22,6 +22,10 @@ type fakeGateway struct {
 	puts      int
 	embedded  int
 	busyFirst bool
+	// serverEmbeds is a gateway that embeds text it is sent; vectors
+	// counts chunks that came with one.
+	serverEmbeds bool
+	vectors      int
 }
 
 func (g *fakeGateway) serve(t *testing.T) *Client {
@@ -37,6 +41,7 @@ func (g *fakeGateway) serve(t *testing.T) *Client {
 		}
 		m := Manifest{Collection: coll}
 		m.Collection.Writable = g.writable
+		m.Collection.ServerEmbeds = g.serverEmbeds
 		for _, d := range g.docs {
 			m.Documents = append(m.Documents, d)
 		}
@@ -48,6 +53,11 @@ func (g *fakeGateway) serve(t *testing.T) *Client {
 		g.mu.Lock()
 		defer g.mu.Unlock()
 		g.puts++
+		for _, c := range d.Chunks {
+			if len(c.Embedding) > 0 {
+				g.vectors++
+			}
+		}
 		g.docs[d.Name] = RemoteDoc{Name: d.Name, Digest: d.Digest, Chunks: len(d.Chunks)}
 		_, _ = w.Write([]byte(`{"changed":true}`))
 	})
@@ -270,3 +280,22 @@ func TestFiltersAndNames(t *testing.T) {
 		t.Errorf("an empty folder gave %v", err)
 	}
 }
+
+func TestAGatewayThatEmbedsIsSentTextAlone(t *testing.T) {
+	g, c, l, dir := setup(t)
+	g.serverEmbeds = true
+	rep, k := run(t, c, l, Options{Collection: "ik", Roots: []string{dir}})
+	if len(k["sent"]) != 2 || rep.Failed != 0 {
+		t.Fatalf("%+v %v", rep, k)
+	}
+	if g.embedded != 0 || g.vectors != 0 {
+		t.Errorf("embedded here: %d asked, %d vectors sent", g.embedded, g.vectors)
+	}
+
+	// --embed-here sends the vectors all the same.
+	write(t, dir, "politika/yeni.md", "Yeni kural.\n")
+	run(t, c, l, Options{Collection: "ik", Roots: []string{dir}, EmbedHere: true})
+	if g.embedded == 0 || g.vectors == 0 {
+		t.Errorf("--embed-here: %d asked, %d vectors sent", g.embedded, g.vectors)
+	}
+}

internal/mirror/api.go

@@ -75,6 +75,9 @@ type FetchRequest struct {
 	// empty, which is right almost always and wrong visibly rather than
 	// quietly when it is not.
 	File string `json:"file"`
+	// SHA256 is what the registry published for the file: a download that
+	// is not it is deleted and the fetch fails. Empty checks nothing.
+	SHA256 string `json:"sha256,omitempty"`
 }
 
 // FetchResponse reports where a download stands, and where it will be.
@@ -102,7 +105,7 @@ func (a *API) fetch(w http.ResponseWriter, r *http.Request) {
 	}
 
 	e := Entry{Model: req.Model, Version: req.Version, File: req.File}
-	st, err := a.m.Start(req.Source, e)
+	st, err := a.m.StartChecked(req.Source, e, strings.ToLower(req.SHA256))
 	switch {
 	case errors.Is(err, ErrAlreadyFetching):
 		// Not an error the caller needs to handle: a control plane that asks

internal/mirror/manager.go

@@ -112,6 +112,12 @@ func (m *Manager) Start(source string, e Entry) (Status, error) {
 	return m.start(source, e, "", m.client)
 }
 
+// StartChecked is Start for a file whose sha256 the registry published: one
+// that arrives otherwise is deleted, and the fetch fails saying both.
+func (m *Manager) StartChecked(source string, e Entry, sha string) (Status, error) {
+	return m.start(source, e, sha, m.client)
+}
+
 // startCopy fetches a file a peer mirror holds, from a private address,
 // and keeps it only if its checksum is the peer's.
 func (m *Manager) startCopy(source string, e Entry, sha string) (Status, error) {
@@ -163,9 +169,10 @@ func (m *Manager) run(source string, e Entry, key, expect string, client *http.C
 	})
 
 	if err == nil && expect != "" && added.SHA256 != expect {
-		// Not what the peer holds: a file changed or cut short on the way.
+		// Not what the registry published, or what the peer holds: a file
+		// changed upstream, or on the way.
 		_ = Remove(m.root, e)
-		err = fmt.Errorf("the copy's sha256 %s is not the peer's %s", added.SHA256, expect)
+		err = fmt.Errorf("%s arrived with sha256 %s, not the %s expected of it: deleted", e.File, added.SHA256, expect)
 	}
 	if err == nil {
 		// Added again after it was removed: the new one stands.
@@ -210,7 +217,7 @@ func (m *Manager) Status(model, version string) (Status, bool) {
 
 	m.mu.Lock()
 	for _, st := range m.running {
-		if st.Model == model && st.Version == version {
+		if st.Model == model && st.Version == version && !m.goneLocked(*st) {
 			out, found = merge(out, *st, found)
 		}
 	}
@@ -327,7 +334,9 @@ func (m *Manager) All() ([]Status, error) {
 	}
 
 	for key, st := range known {
-		if _, onDisk := seen[key]; onDisk {
+		if _, onDisk := seen[key]; onDisk || st.State == Present {
+			// Present and not on the disk: removed since, by hand or by
+			// llmgw mirror remove. Not reported as here, which it is not.
 			continue
 		}
 		out = append(out, st)
@@ -335,6 +344,16 @@ func (m *Manager) All() ([]Status, error) {
 	return out, nil
 }
 
+// goneLocked says whether a finished download's file has been removed since.
+// Called with m.mu held.
+func (m *Manager) goneLocked(st Status) bool {
+	if st.State != Present {
+		return false
+	}
+	_, err := os.Stat(filepath.Join(m.root, st.Model, st.Version, st.File))
+	return errors.Is(err, os.ErrNotExist)
+}
+
 // keptFinished is how long a finished download's status is remembered.
 const keptFinished = 7 * 24 * time.Hour
 

internal/mirror/peer.go

@@ -3,6 +3,7 @@ package mirror
 import (
 	"context"
 	"log/slog"
+	"path/filepath"
 	"strings"
 	"time"
 )
@@ -24,6 +25,13 @@ type Peers struct {
 
 // Run reads the peers until ctx ends.
 func (p *Peers) Run(ctx context.Context) {
+	// Files fetched before checksums were kept have none, and a peer copies
+	// only what it can check: they are given theirs, beside the sync.
+	go func() {
+		if n := p.M.FillChecksums(ctx); n > 0 {
+			p.Log.Info("recorded the checksums of files that had none, so peer mirrors can copy them", "files", n)
+		}
+	}()
 	t := time.NewTicker(PeerInterval)
 	defer t.Stop()
 	for {
@@ -122,3 +130,33 @@ func (p *Peers) SyncOnce(ctx context.Context, c *Client) {
 		}
 	}
 }
+
+// FillChecksums records the sha256 of every file the mirror holds without
+// one, and says how many it recorded.
+func (m *Manager) FillChecksums(ctx context.Context) int {
+	entries, err := List(m.root)
+	if err != nil {
+		return 0
+	}
+	n := 0
+	for _, e := range entries {
+		if ctx.Err() != nil {
+			break
+		}
+		path := filepath.Join(m.root, e.Model, e.Version, e.File)
+		if readChecksum(path) != "" || m.isRunning(e.Model, e.Version, e.File) {
+			continue
+		}
+		sum, err := sha256File(path)
+		if err != nil {
+			m.log.Warn("could not read a file's checksum", "file", path, "err", err)
+			continue
+		}
+		if err := writeChecksum(path, sum); err != nil {
+			m.log.Warn("could not record a file's checksum", "file", path, "err", err)
+			continue
+		}
+		n++
+	}
+	return n
+}

internal/mirror/peer_test.go

@@ -5,6 +5,7 @@ import (
 	"net/http/httptest"
 	"os"
 	"path/filepath"
+	"strings"
 	"testing"
 	"time"
 
@@ -172,3 +173,101 @@ func TestAModelsAddressIsMovedToAMirrorThatAnswers(t *testing.T) {
 		t.Errorf("none up: %s", got)
 	}
 }
+
+// A file fetched before checksums were kept is given one, and is then copied.
+func TestAFileWithoutAChecksumIsGivenOneAndThenCopied(t *testing.T) {
+	a, b := newTestMirror(t), newTestMirror(t)
+	path := filepath.Join(a.root, "old", "v1", "m.gguf")
+	if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
+		t.Fatal(err)
+	}
+	if err := os.WriteFile(path, []byte("fetched long ago"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	pb := &Peers{M: b.m, Log: discardLog()}
+	pb.SyncOnce(t.Context(), a.client(t))
+	time.Sleep(100 * time.Millisecond)
+	if b.has("old", "v1") {
+		t.Fatal("copied without a checksum to check it by")
+	}
+	if n := a.m.FillChecksums(t.Context()); n != 1 || readChecksum(path) == "" {
+		t.Fatalf("recorded %d", n)
+	}
+	if a.m.FillChecksums(t.Context()) != 0 {
+		t.Error("recorded again")
+	}
+	pb.SyncOnce(t.Context(), a.client(t))
+	waitPresent(t, b, "old", "v1")
+}
+
+// A file removed outside the running mirror is not reported as here, and its
+// removal, recorded, reaches a peer instead of the peer's copy coming back.
+func TestAFileRemovedByHandIsNotReportedAndItsRemovalIsCarried(t *testing.T) {
+	blob := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		_, _ = w.Write([]byte("weights"))
+	}))
+	defer blob.Close()
+	a, b := newTestMirror(t), newTestMirror(t)
+	e := Entry{Model: "qwen", Version: "v1", File: "m.gguf"}
+	if _, err := a.m.Start(blob.URL+"/m.gguf", e); err != nil {
+		t.Fatal(err)
+	}
+	waitPresent(t, a, "qwen", "v1")
+	pb := &Peers{M: b.m, Log: discardLog()}
+	pb.SyncOnce(t.Context(), a.client(t))
+	waitPresent(t, b, "qwen", "v1")
+
+	time.Sleep(10 * time.Millisecond)
+	if err := Remove(a.root, e); err != nil {
+		t.Fatal(err)
+	}
+	if err := RecordRemoval(a.root, "qwen", "v1"); err != nil {
+		t.Fatal(err)
+	}
+	if a.has("qwen", "v1") {
+		t.Fatal("reported as here after it was removed")
+	}
+	all, _ := a.m.All()
+	for _, st := range all {
+		if st.Model == "qwen" {
+			t.Fatalf("listed after it was removed: %+v", st)
+		}
+	}
+	pa := &Peers{M: a.m, Log: discardLog()}
+	pa.SyncOnce(t.Context(), b.client(t))
+	time.Sleep(200 * time.Millisecond)
+	if a.has("qwen", "v1") {
+		t.Fatal("copied back from the peer")
+	}
+	pb.SyncOnce(t.Context(), a.client(t))
+	if b.has("qwen", "v1") {
+		t.Error("the peer kept it")
+	}
+}
+
+// A fetch given the registry's sha256 keeps only that file.
+func TestAFetchThatIsNotWhatTheRegistryPublishedIsDeleted(t *testing.T) {
+	served := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		_, _ = w.Write([]byte("replaced upstream"))
+	}))
+	defer served.Close()
+	tm := newTestMirror(t)
+	e := Entry{Model: "qwen", Version: "v1", File: "m.gguf"}
+	if _, err := tm.m.StartChecked(served.URL+"/m.gguf", e, "aaaa"); err != nil {
+		t.Fatal(err)
+	}
+	deadline := time.Now().Add(5 * time.Second)
+	var st Status
+	for time.Now().Before(deadline) {
+		if st, _ = tm.m.Status("qwen", "v1"); st.State == Failed {
+			break
+		}
+		time.Sleep(20 * time.Millisecond)
+	}
+	if st.State != Failed || !strings.Contains(st.Err, "aaaa") {
+		t.Fatalf("%+v", st)
+	}
+	if _, err := os.Stat(e.Path(tm.root)); !os.IsNotExist(err) {
+		t.Error("kept")
+	}
+}

internal/mirror/tombstone.go

@@ -57,3 +57,22 @@ func (m *Manager) clearTombstone(model, version string) {
 		m.writeTombstones(t)
 	}
 }
+
+// RecordRemoval keeps a version's removal for a mirror's peers when the
+// removal was made outside the running mirror, by llmgw mirror remove: once
+// none of its files is left, so a peer removes its copy rather than this
+// mirror copying it back.
+func RecordRemoval(root, model, version string) error {
+	entries, err := List(root)
+	if err != nil {
+		return err
+	}
+	for _, e := range entries {
+		if e.Model == model && e.Version == version {
+			return nil
+		}
+	}
+	m := &Manager{root: root}
+	m.recordTombstone(model, version, time.Now())
+	return nil
+}

internal/modelplan/fit.go

@@ -155,10 +155,7 @@ func fitsOn(m Model, h Hardware, d Demand, p Precision, context int) []Fit {
 		if nodes > 1 && h.Fabric == "" {
 			continue
 		}
-		// The KV heads are split among the cards, or past their number each
-		// card holds a copy of one: the width divides them or they divide
-		// it. A model with 8 KV heads splits 2, 4, 8 or 16 ways, not 12.
-		if m.KVHeads > 0 && m.KVHeads%width != 0 && width%m.KVHeads != 0 {
+		if !SplitsHeads(m.KVHeads, width) {
 			continue
 		}
 

internal/modelplan/options.go

@@ -53,6 +53,8 @@ type PoolNode struct {
 	// card beside it another. Empty means Class, GPUMemGB, GPUs and FreeGPUs
 	// say it all.
 	Kinds []PoolKind
+	// Engines are the engines the node has installed.
+	Engines []string
 }
 
 // PoolKind is one class of a node's devices.
@@ -71,6 +73,9 @@ func (n PoolNode) kinds() []PoolKind {
 	return []PoolKind{{Class: n.Class, GPUMemGB: n.GPUMemGB, GPUs: n.GPUs, FreeGPUs: n.FreeGPUs}}
 }
 
+// Kinds is a node's devices by class: its one class, or each of several.
+func Kinds(n PoolNode) []PoolKind { return n.kinds() }
+
 // isMIG reports whether a class names MIG instances: "a100-80g-mig-3g.40gb".
 // A replica runs on exactly one, never a group of them, so a plan offers one
 // instance and nothing wider. Read from the class label, as the card's memory
@@ -107,6 +112,11 @@ type Option struct {
 	// cards, whole, rather than slicing each across them: vLLM's
 	// --enable-expert-parallel, for a model of experts on more than one card.
 	ExpertParallel bool `json:"expert_parallel,omitempty"`
+	// StrataFamily and StrataSize are Strata's --family and --size, for an
+	// option on Strata: one card, and the experts in the node's RAM, which
+	// NeedGB and HaveGB are then about.
+	StrataFamily string `json:"strata_family,omitempty"`
+	StrataSize   string `json:"strata_size,omitempty"`
 }
 
 // GPUs is every card one replica takes.
@@ -127,7 +137,7 @@ func (o Option) Fits() bool { return o.NeedGB <= o.HaveGB }
 // WithContext is the option for another context: kvPerToken a token's KV
 // cache, the whole model; the weights and the engine's share stay.
 func (o Option) WithContext(kvPerToken int64, tokens int) Option {
-	if kvPerToken <= 0 || tokens <= 0 {
+	if kvPerToken <= 0 || tokens <= 0 || o.Engine == "strata" {
 		return o
 	}
 	ctx := GB(int64(tokens) * kvPerToken)
@@ -141,6 +151,9 @@ func (o Option) WithContext(kvPerToken int64, tokens int) Option {
 
 // MemoryText says what it needs, part by part.
 func (o Option) MemoryText() string {
+	if o.Engine == "strata" {
+		return fmt.Sprintf("%.0f GB of the node's RAM for its experts, of %.0f usable, and one card", o.NeedGB, o.HaveGB)
+	}
 	parts := fmt.Sprintf("weights %.1f + context %.1f + engine %.1f", o.WeightsGB, o.ContextGB, o.EngineGB)
 	if o.WeightsGB == 0 && o.ContextGB == 0 {
 		parts = ""
@@ -171,6 +184,9 @@ func (o Option) Label() string {
 	if o.Nodes > 1 {
 		return fmt.Sprintf("%d nodes × %d %s on %s", o.Nodes, o.GPUsPerNode, o.Class, o.Fabric)
 	}
+	if o.Engine == "strata" {
+		return fmt.Sprintf("1 × %s with Strata, experts in RAM", o.Class)
+	}
 	return fmt.Sprintf("%d × %s", o.GPUsPerNode, o.Class)
 }
 
@@ -263,12 +279,7 @@ func vllmOptions(a Artifact, kv int64, sh Shape, nodes []PoolNode, bpw float64)
 	perCard := func(width int) int64 {
 		return (a.Bytes+kv)/int64(width) + overheadBytes(a.Bytes/int64(width))
 	}
-	// Tensor parallel splits the KV heads among the cards, or past their
-	// number gives each card a copy of one: so the width divides the heads,
-	// or the heads divide the width. Qwen3-235B has four and runs on eight.
-	divides := func(w int) bool {
-		return sh.KVHeads == 0 || sh.KVHeads%w == 0 || w%sh.KVHeads == 0
-	}
+	divides := func(w int) bool { return SplitsHeads(sh.KVHeads, w) }
 	for _, c := range classes(nodes) {
 		budget := float64(c.mem) * gpuShare
 		opt := func(perNode, n int, fabric string) Option {
@@ -516,3 +527,13 @@ func CardNeed(engine string, weights, kv int64, cards int) float64 {
 // CardBudget is what a card of a class may be planned to hold, in GB; 0 for
 // a class whose memory its name does not say.
 func CardBudget(class string) float64 { return float64(MemFromClass(class)) * gpuShare }
+
+// SplitsHeads reports whether tensor parallel can spread a model of kvHeads
+// KV heads over width cards. It splits the heads among the cards, or past
+// their number gives each card a copy of one: so the width divides the
+// heads, or the heads divide the width. Qwen3-235B has four and runs on
+// eight; a model with 8 splits 2, 4, 8 or 16 ways, not 12. 0 heads is
+// unknown, and allowed.
+func SplitsHeads(kvHeads, width int) bool {
+	return kvHeads == 0 || width <= 1 || kvHeads%width == 0 || width%kvHeads == 0
+}

internal/modelplan/strata.go

@@ -0,0 +1,60 @@
+package modelplan
+
+import (
+	"slices"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/strata"
+)
+
+// Strata runs a few mixture-of-experts models with their experts in the
+// node's RAM and one card for the rest: a model that fits no card, or only
+// many, served from one. Offered for the files of a family it runs, on nodes
+// that have it installed, a card of at least strataMinCardGB and the RAM.
+
+// strataMinCardGB is the smallest card Strata is worth running on: what its
+// guide recommends.
+const strataMinCardGB = 12
+
+// StrataOptions are the ways a family's files could run on Strata here.
+func StrataOptions(family string, arts []Artifact, nodes []PoolNode) []Option {
+	var out []Option
+	for _, a := range arts {
+		size, ok := strata.SizeForFile(family, a.Name)
+		if !ok {
+			continue
+		}
+		need := float64(size.MemoryGB())
+		byClass := map[string]*Option{}
+		var order []string
+		for _, n := range nodes {
+			if !slices.Contains(n.Engines, "strata") || n.HostMemGB <= 0 {
+				continue
+			}
+			have := float64(n.HostMemGB) * hostShare
+			if need > have {
+				continue
+			}
+			for _, k := range n.kinds() {
+				if k.Class == "" || isMIG(k.Class) || k.GPUMemGB < strataMinCardGB || k.GPUs < 1 {
+					continue
+				}
+				o := byClass[k.Class]
+				if o == nil {
+					o = &Option{Artifact: a.Name, Format: a.Format, Engine: "strata", Class: k.Class,
+						GPUsPerNode: 1, Nodes: 1, NeedGB: need, StrataFamily: family, StrataSize: size.Name}
+					byClass[k.Class] = o
+					order = append(order, k.Class)
+				}
+				o.Hosts++
+				o.HaveGB = max(o.HaveGB, have)
+				if k.FreeGPUs > 0 && float64(n.HostFreeGB) >= need {
+					o.FreeNow = true
+				}
+			}
+		}
+		for _, c := range order {
+			out = append(out, *byClass[c])
+		}
+	}
+	return out
+}

internal/modelplan/strata_test.go

@@ -0,0 +1,32 @@
+package modelplan
+
+import "testing"
+
+func TestStrataIsOfferedOnANodeWithItTheCardAndTheRAM(t *testing.T) {
+	arts := []Artifact{{Name: "Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00001-of-00002.gguf", Format: "gguf", Bytes: 68e9},
+		{Name: "something-else-Q4_K_M.gguf", Format: "gguf", Bytes: 5e9}}
+	nodes := []PoolNode{
+		{ID: "cuda01", Class: "6000-24g", GPUMemGB: 24, GPUs: 1, FreeGPUs: 1, HostMemGB: 251, HostFreeGB: 200,
+			Engines: []string{"llamacpp", "strata"}},
+		{ID: "no-strata", Class: "6000-24g", GPUMemGB: 24, GPUs: 1, FreeGPUs: 1, HostMemGB: 251, Engines: []string{"llamacpp"}},
+		{ID: "small-ram", Class: "6000-24g", GPUMemGB: 24, GPUs: 1, FreeGPUs: 1, HostMemGB: 32, Engines: []string{"strata"}},
+		{ID: "small-card", Class: "t4-8g", GPUMemGB: 8, GPUs: 1, FreeGPUs: 1, HostMemGB: 251, Engines: []string{"strata"}},
+	}
+	opts := StrataOptions("qwen", arts, nodes)
+	if len(opts) != 1 {
+		t.Fatalf("%+v", opts)
+	}
+	o := opts[0]
+	if o.Engine != "strata" || o.StrataSize != "IQ2_XS" || o.Hosts != 1 || !o.FreeNow || o.NeedGB != 52 || o.GPUs() != 1 {
+		t.Errorf("%+v", o)
+	}
+	// It ranks before llama.cpp on the CPU, and says what it needs.
+	all := append(Options(arts[:1], Shape{}, nodes, 16384), opts...)
+	Rank(all)
+	if all[0].Engine != "strata" {
+		t.Errorf("first is %+v", all[0])
+	}
+	if o.Label() != "1 × 6000-24g with Strata, experts in RAM" || o.MemoryText() == "" {
+		t.Errorf("%q %q", o.Label(), o.MemoryText())
+	}
+}

internal/router/chat_load.go

@@ -68,9 +68,13 @@ func (r *Router) loadStatus(model string) modelStatus {
 	if best != nil {
 		return *best
 	}
-	if _, m := r.resolve(model); m.Scaling.OnDemand() {
+	_, m := r.resolve(model)
+	if m.Scaling.OnDemand() {
 		return modelStatus{State: "waiting", Message: "waiting for a node with room for it"}
 	}
+	if m.Stopped != nil {
+		return modelStatus{State: "stopped", Message: "an administrator stopped this model; it is kept, and runs again when started"}
+	}
 	return modelStatus{State: "stopped", Message: "this model is not running"}
 }
 

internal/router/kb.go

@@ -53,6 +53,9 @@ type KBCollectionView struct {
 	Dims       int    `json:"dims"`
 	// Writable is whether this caller may add and remove documents.
 	Writable bool `json:"writable"`
+	// ServerEmbeds says a document may be sent as text alone: the gateway
+	// embeds its chunks, as the caller, with the collection's model.
+	ServerEmbeds bool `json:"server_embeds"`
 }
 
 // KBDocumentView is a document as the manifest lists it.
@@ -77,8 +80,10 @@ type KBPutDocument struct {
 	Digest string `json:"digest"`
 	Bytes  int64  `json:"bytes"`
 	Chunks []struct {
-		Text      string    `json:"text"`
-		Embedding []float32 `json:"embedding"`
+		Text string `json:"text"`
+		// Embedding is the chunk's vector; with none on any chunk the
+		// gateway embeds them.
+		Embedding []float32 `json:"embedding,omitempty"`
 	} `json:"chunks"`
 }
 
@@ -124,7 +129,7 @@ func (r *Router) kbCaller(w http.ResponseWriter, req *http.Request) (auth.Princi
 
 func (r *Router) kbView(ctx context.Context, c storepkg.KBCollection, writable bool) KBCollectionView {
 	return KBCollectionView{ID: c.ID, Name: c.Name, Owner: string(c.OwnerScope) + ":" + c.OwnerID,
-		OwnerName: r.store.OwnerName(ctx, c.OwnerScope, c.OwnerID), EmbedModel: c.EmbedModel, Dims: c.Dims, Writable: writable}
+		OwnerName: r.store.OwnerName(ctx, c.OwnerScope, c.OwnerID), EmbedModel: c.EmbedModel, Dims: c.Dims, Writable: writable, ServerEmbeds: true}
 }
 
 // kbCollection finds a collection by id or name among those the caller may
@@ -281,6 +286,19 @@ func (r *Router) handleKBPut(w http.ResponseWriter, req *http.Request) {
 			fmt.Sprintf("%d chunks: a document has at most %d", len(in.Chunks), kbMaxChunks))
 		return
 	}
+	// Text alone: the gateway embeds it, as the caller, so their limits and
+	// usage count it as if they had asked.
+	embedded := 0
+	for _, ch := range in.Chunks {
+		if len(ch.Embedding) > 0 {
+			embedded++
+		}
+	}
+	if embedded > 0 && embedded < len(in.Chunks) {
+		writeError(w, http.StatusBadRequest, "invalid_request_error",
+			"some chunks have an embedding and some not: send every chunk's, or none for the gateway to embed")
+		return
+	}
 	chunks := make([]storepkg.KBChunk, len(in.Chunks))
 	for i, ch := range in.Chunks {
 		if !utf8.ValidString(ch.Text) || strings.TrimSpace(ch.Text) == "" {
@@ -292,7 +310,7 @@ func (r *Router) handleKBPut(w http.ResponseWriter, req *http.Request) {
 				fmt.Sprintf("chunk %d is over %d characters", i, kbMaxChunkRunes))
 			return
 		}
-		if len(ch.Embedding) != c.Dims {
+		if embedded > 0 && len(ch.Embedding) != c.Dims {
 			writeError(w, http.StatusBadRequest, "invalid_request_error",
 				fmt.Sprintf("chunk %d has %d dimensions; collection %s is embedded by %s in %d", i,
 					len(ch.Embedding), c.Name, c.EmbedModel, c.Dims))
@@ -300,6 +318,28 @@ func (r *Router) handleKBPut(w http.ResponseWriter, req *http.Request) {
 		}
 		chunks[i] = storepkg.KBChunk{Body: strings.ToValidUTF8(ch.Text, ""), Embedding: ch.Embedding}
 	}
+	if embedded == 0 {
+		texts := make([]string, len(chunks))
+		for i := range chunks {
+			texts[i] = chunks[i].Body
+		}
+		vecs, err := r.kbEmbed(req, c, texts)
+		if err != nil {
+			var ee *kbEmbedError
+			if errors.As(err, &ee) {
+				if ee.retryAfter != "" {
+					w.Header().Set("Retry-After", ee.retryAfter)
+				}
+				writeError(w, ee.status, "invalid_request_error", err.Error())
+				return
+			}
+			writeError(w, http.StatusBadGateway, "server_error", err.Error())
+			return
+		}
+		for i := range chunks {
+			chunks[i].Embedding = vecs[i]
+		}
+	}
 	d, changed, err := r.store.PutDocument(req.Context(), c, storepkg.KBDocument{Name: name, Digest: in.Digest,
 		Bytes: in.Bytes, AddedBy: kbActor(p)}, chunks)
 	if err != nil {

internal/router/kb_embed.go

@@ -0,0 +1,134 @@
+package router
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"strconv"
+	"sync"
+	"time"
+
+	storepkg "gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// A document sent as text alone is embedded here, through the gateway's own
+// /v1/embeddings as the caller: their key, limits and usage, as if they had
+// asked. The client then sends a kilobyte a chunk instead of the vector's
+// twenty in each direction.
+
+const (
+	// kbEmbedBatch is how many chunks go in one embeddings request, and
+	// kbEmbedAtOnce how many such requests are in flight for one document:
+	// enough to fill an embedding replica's slots.
+	kbEmbedBatch  = 32
+	kbEmbedAtOnce = 4
+	// kbEmbedWait is how long one batch waits out answers that say to try
+	// again, before the document is refused with the same advice.
+	kbEmbedWait = 5 * time.Minute
+)
+
+// kbEmbedError is an embeddings answer that was not a success, passed on as
+// the API gave it.
+type kbEmbedError struct {
+	status     int
+	retryAfter string
+	msg        string
+}
+
+func (e *kbEmbedError) Error() string { return e.msg }
+
+// kbEmbed embeds texts with the collection's model, as the request's caller.
+func (r *Router) kbEmbed(req *http.Request, c storepkg.KBCollection, texts []string) ([][]float32, error) {
+	ctx, cancel := context.WithCancel(req.Context())
+	defer cancel()
+	out := make([][]float32, len(texts))
+	var (
+		mu    sync.Mutex
+		first error
+		wg    sync.WaitGroup
+	)
+	slots := make(chan struct{}, kbEmbedAtOnce)
+	for start := 0; start < len(texts); start += kbEmbedBatch {
+		end := min(start+kbEmbedBatch, len(texts))
+		wg.Add(1)
+		slots <- struct{}{}
+		go func() {
+			defer wg.Done()
+			defer func() { <-slots }()
+			vecs, err := r.kbEmbedBatch(ctx, req, c, texts[start:end])
+			mu.Lock()
+			defer mu.Unlock()
+			if err != nil {
+				if first == nil {
+					first = err
+					cancel()
+				}
+				return
+			}
+			copy(out[start:end], vecs)
+		}()
+	}
+	wg.Wait()
+	if first != nil {
+		return nil, first
+	}
+	return out, nil
+}
+
+// kbEmbedBatch embeds one batch, waiting out answers that say when to try
+// again.
+func (r *Router) kbEmbedBatch(ctx context.Context, req *http.Request, c storepkg.KBCollection, texts []string) ([][]float32, error) {
+	call := mediaCall{req: req.WithContext(ctx)}
+	deadline := time.Now().Add(kbEmbedWait)
+	for {
+		status, h, b, err := r.self(call, http.MethodPost, "/v1/embeddings",
+			map[string]any{"model": c.EmbedModel, "input": texts, "encoding_format": "float"})
+		if err != nil {
+			return nil, err
+		}
+		if status < 300 {
+			return kbVectors(b, len(texts), c)
+		}
+		retry := h.Get("Retry-After")
+		if retry == "" || (status != http.StatusTooManyRequests && status < 500) || time.Now().After(deadline) {
+			return nil, &kbEmbedError{status: status, retryAfter: retry,
+				msg: fmt.Sprintf("embedding with %s: %s", c.EmbedModel, errorMessage(b))}
+		}
+		secs, _ := strconv.Atoi(retry)
+		select {
+		case <-ctx.Done():
+			return nil, ctx.Err()
+		case <-time.After(time.Duration(max(secs, 1)) * time.Second):
+		}
+	}
+}
+
+// kbVectors reads an embeddings answer: one vector a text, of the
+// collection's dimensions.
+func kbVectors(b []byte, n int, c storepkg.KBCollection) ([][]float32, error) {
+	var res struct {
+		Data []struct {
+			Index     int       `json:"index"`
+			Embedding []float32 `json:"embedding"`
+		} `json:"data"`
+	}
+	if err := json.Unmarshal(b, &res); err != nil {
+		return nil, fmt.Errorf("embedding with %s: the answer is not JSON: %w", c.EmbedModel, err)
+	}
+	if len(res.Data) != n {
+		return nil, fmt.Errorf("embedding with %s: %d of %d texts came back", c.EmbedModel, len(res.Data), n)
+	}
+	out := make([][]float32, n)
+	for _, d := range res.Data {
+		if d.Index < 0 || d.Index >= n || out[d.Index] != nil {
+			return nil, fmt.Errorf("embedding with %s: an embedding came back with index %d", c.EmbedModel, d.Index)
+		}
+		if len(d.Embedding) != c.Dims {
+			return nil, &kbEmbedError{status: http.StatusBadRequest,
+				msg: fmt.Sprintf("%s gave %d dimensions and collection %s takes %d", c.EmbedModel, len(d.Embedding), c.Name, c.Dims)}
+		}
+		out[d.Index] = d.Embedding
+	}
+	return out, nil
+}

internal/router/kb_embed_test.go

@@ -0,0 +1,47 @@
+package router
+
+import (
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// A document's text is embedded through the gateway's own API as the
+// caller, in batches, each vector in its chunk's place.
+func TestTheGatewayEmbedsADocumentSentAsText(t *testing.T) {
+	e := newFakeEngine(t, "qwen2.5-14b-instruct")
+	rt, secret := pricedRouter(t, e.srv.URL)
+	_ = rt.Handler() // mounts the API the embedding goes through
+	texts := make([]string, kbEmbedBatch*2+5)
+	for i := range texts {
+		texts[i] = strings.Repeat("söz ", i+1)
+	}
+	req := httptest.NewRequestWithContext(t.Context(), http.MethodPut, "/v1/kb/collections/x/documents", nil)
+	req.Header.Set("Authorization", "Bearer "+secret)
+	c := store.KBCollection{Name: "x", EmbedModel: "qwen2.5-14b-instruct", Dims: 3}
+	vecs, err := rt.kbEmbed(req, c, texts)
+	if err != nil {
+		t.Fatal(err)
+	}
+	for i, v := range vecs {
+		// The fake engine's vector is the text's bytes, its words, and 1.
+		if len(v) != 3 || int(v[1]) != i+1 || int(v[0]) != len(texts[i]) {
+			t.Fatalf("chunk %d got %v", i, v)
+		}
+	}
+
+	// A model of other dimensions is refused, saying so.
+	c.Dims = 4
+	if _, err := rt.kbEmbed(req, c, texts[:2]); err == nil || !strings.Contains(err.Error(), "dimensions") {
+		t.Errorf("4 dimensions: %v", err)
+	}
+	// Without the caller's key, nothing is embedded.
+	bare := httptest.NewRequestWithContext(t.Context(), http.MethodPut, "/v1/kb/collections/x/documents", nil)
+	c.Dims = 3
+	if _, err := rt.kbEmbed(bare, c, texts[:1]); err == nil {
+		t.Error("embedded without a key")
+	}
+}

internal/router/kb_test.go

@@ -206,3 +206,36 @@ func TestAPersonalKeyReadsOnlyInTheProjectItNames(t *testing.T) {
 		t.Errorf("a project the person is not in: %d", code)
 	}
 }
+
+func TestADocumentSentAsTextIsEmbeddedByTheGateway(t *testing.T) {
+	rt := kbRouter(t)
+	h := rt.Handler()
+	ctx := t.Context()
+	s := rt.store
+	sfx := strings.ToLower(store.NewID(""))
+	org, _ := s.CreateOrg(ctx, "e-"+sfx)
+	team, _ := s.CreateTeam(ctx, org, "t")
+	prj, _ := s.CreateProject(ctx, team, "p")
+	// The fake engine embeds in three dimensions.
+	if _, err := s.CreateCollection(ctx, store.KBCollection{OwnerScope: store.ScopeProject, OwnerID: prj,
+		Name: "metin", EmbedModel: "qwen2.5-14b-instruct", Dims: 3}); err != nil {
+		t.Fatal(err)
+	}
+	key := kbKey(t, s, prj, "")
+	base := "/v1/kb/collections/metin/"
+	if code, _, body := call(t, h, key, http.MethodGet, base+"manifest", nil); !strings.Contains(body, `"server_embeds":true`) {
+		t.Errorf("manifest: %d %s", code, body)
+	}
+	textOnly := map[string]any{"name": "a.md", "digest": "d1", "bytes": 10, "chunks": []map[string]any{
+		{"text": "birinci parça"}, {"text": "ikinci parça burada"}}}
+	code, out, body := call(t, h, key, http.MethodPut, base+"documents", textOnly)
+	if code != http.StatusOK || out["chunks"] != float64(2) {
+		t.Fatalf("text only: %d %s", code, body)
+	}
+	// Some with a vector and some without is refused.
+	mixed := map[string]any{"name": "b.md", "digest": "d2", "bytes": 10, "chunks": []map[string]any{
+		{"text": "bir", "embedding": []float32{1, 0, 0}}, {"text": "iki"}}}
+	if code, _, body := call(t, h, key, http.MethodPut, base+"documents", mixed); code != http.StatusBadRequest {
+		t.Errorf("mixed: %d %s", code, body)
+	}
+}

internal/strata/models.go

@@ -0,0 +1,174 @@
+package strata
+
+import (
+	"fmt"
+	"slices"
+	"strings"
+)
+
+// What Strata runs, as its setup (0.2.7, the version llmgw installs) lists
+// it: a few model families, each in a few sizes, with what a size takes.
+// Written down here so a model is checked when it is added rather than when
+// a node's setup refuses it minutes later, and so the plan can offer Strata
+// for a model it runs, with the memory to reserve.
+
+// Family is a model Strata runs, from one publisher's files.
+type Family struct {
+	// Name is setup's --family.
+	Name  string
+	Title string
+	// Repo is the Hugging Face repository its files are in; empty when
+	// Strata fetches them itself from several.
+	Repo string
+	// Subdir is whether each size's files are in a directory of its own,
+	// named as the size, in the repository.
+	Subdir bool
+	// Engine is a family that runs on an engine path of its own, with a
+	// download of its own (DeepSeek V4.1).
+	Engine bool
+}
+
+// Size is one of a family's sizes: setup's --model, which llmgw's profile
+// says as --size.
+type Size struct {
+	Name     string
+	Families []string
+	// DownloadGB is what is fetched; DiskGB what it takes on the node once
+	// prepared, when more than the download.
+	DownloadGB, DiskGB float64
+	// RAMGB is the host memory that keeps every expert in RAM: what a
+	// replica reserves. Strata can read the rest from the disk with less,
+	// much slower.
+	RAMGB float64
+	// NVIDIAOnly is a size whose kernels only NVIDIA cards have.
+	NVIDIAOnly bool
+}
+
+// Families are the families Strata 0.2.7 runs.
+var Families = []Family{
+	{Name: "qwen", Title: "Qwen3.8-Flash-Next", Repo: "ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF", Subdir: true},
+	{Name: "swift", Title: "Swift 1.5 (Qwen3.8-Flash-Next, shorter thinking)", Repo: "ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF"},
+	{Name: "coder", Title: "Qwen3.8-Flash-Next Coder", Repo: "ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF", Subdir: true},
+	{Name: "unsloth", Title: "Qwen3.8-Flash-Next (Unsloth)", Repo: "unsloth/Qwen3.8-Flash-Next-GGUF", Subdir: true},
+	{Name: "gyro", Title: "Qwen3.8-Flash-Next (Gyro)", Repo: "agentionai/Qwen3.8-Flash-Next-Gyro-GGUF"},
+	{Name: "nvfp4", Title: "Qwen3.8-Flash-Next (NVFP4)"},
+	{Name: "qwen36", Title: "Qwen3.6-35B-A3B", Repo: "unsloth/Qwen3.6-35B-A3B-MTP-GGUF"},
+	{Name: "deepseek41", Title: "DeepSeek V4.1-Flash", Repo: "ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF", Engine: true},
+}
+
+// Sizes are the sizes Strata 0.2.7 runs, and which families have each.
+var Sizes = []Size{
+	{Name: "Q2_0", Families: []string{"qwen"}, DownloadGB: 66.4, RAMGB: 48},
+	{Name: "IQ2_XS", Families: []string{"qwen", "swift"}, DownloadGB: 68.0, RAMGB: 48},
+	{Name: "IQ3_XXS", Families: []string{"qwen", "swift"}, DownloadGB: 75.8, RAMGB: 60},
+	{Name: "IQ3_S", Families: []string{"qwen", "swift"}, DownloadGB: 83.6, RAMGB: 62},
+	{Name: "IQ1_M", Families: []string{"coder"}, DownloadGB: 58.4, RAMGB: 32},
+	{Name: "UD-IQ4_XS", Families: []string{"unsloth"}, DownloadGB: 93.7, RAMGB: 80},
+	{Name: "UD-Q4_K_XL", Families: []string{"unsloth", "qwen36"}, DownloadGB: 111.3, RAMGB: 94},
+	{Name: "UD-Q5_K_XL", Families: []string{"unsloth"}, DownloadGB: 158.3, RAMGB: 108},
+	{Name: "UD-Q6_K_XL", Families: []string{"unsloth", "qwen36"}, DownloadGB: 169.2, RAMGB: 118},
+	{Name: "Q8_0", Families: []string{"unsloth", "qwen36"}, DownloadGB: 188.2, RAMGB: 136},
+	{Name: "Gyro-S", Families: []string{"gyro"}, DownloadGB: 58.5, RAMGB: 27.6},
+	{Name: "Gyro-M", Families: []string{"gyro"}, DownloadGB: 92.0, RAMGB: 35},
+	{Name: "NVFP4", Families: []string{"nvfp4"}, DownloadGB: 128, RAMGB: 80, NVIDIAOnly: true},
+	{Name: "NVFP4-NVIDIA", Families: []string{"nvfp4"}, DownloadGB: 128, RAMGB: 80, NVIDIAOnly: true},
+	{Name: "NVFP4-QAD", Families: []string{"nvfp4"}, DownloadGB: 102.2, DiskGB: 212.2, RAMGB: 80, NVIDIAOnly: true},
+	{Name: "IST-DAS-1.75bpw", Families: []string{"deepseek41"}, DownloadGB: 239, RAMGB: 34},
+	{Name: "IST-DAS-2.00bpw", Families: []string{"deepseek41"}, DownloadGB: 257, RAMGB: 40},
+	{Name: "IST-DAS-2.25bpw", Families: []string{"deepseek41"}, DownloadGB: 274, RAMGB: 44},
+	{Name: "UD-Q8_K_XL", Families: []string{"deepseek41"}, DownloadGB: 520, RAMGB: 306},
+}
+
+// qwen36 is smaller than the Flash-Next sizes of the same names: its own
+// figures.
+var qwen36RAM = map[string]float64{"UD-Q4_K_XL": 32, "UD-Q6_K_XL": 44, "Q8_0": 52}
+var qwen36Download = map[string]float64{"UD-Q4_K_XL": 22.9, "UD-Q6_K_XL": 32.6, "Q8_0": 37.8}
+
+// FamilyNamed is a family by its --family.
+func FamilyNamed(name string) (Family, bool) {
+	i := slices.IndexFunc(Families, func(f Family) bool { return f.Name == name })
+	if i < 0 {
+		return Family{}, false
+	}
+	return Families[i], true
+}
+
+// SizeOf is a family's size by its --size, with that family's figures.
+func SizeOf(family, size string) (Size, error) {
+	if _, ok := FamilyNamed(family); !ok {
+		return Size{}, fmt.Errorf("strata runs no family %q; it runs %s", family, strings.Join(familyNames(), ", "))
+	}
+	var names []string
+	for _, s := range Sizes {
+		if !slices.Contains(s.Families, family) {
+			continue
+		}
+		names = append(names, s.Name)
+		if s.Name == size {
+			if family == "qwen36" {
+				s.RAMGB, s.DownloadGB = qwen36RAM[size], qwen36Download[size]
+			}
+			return s, nil
+		}
+	}
+	return Size{}, fmt.Errorf("strata's %s comes in %s, not %q", family, strings.Join(names, ", "), size)
+}
+
+// ForRepo is the family whose files a Hugging Face repository holds.
+func ForRepo(repo string) (Family, bool) {
+	i := slices.IndexFunc(Families, func(f Family) bool { return f.Repo != "" && strings.EqualFold(f.Repo, repo) })
+	if i < 0 {
+		return Family{}, false
+	}
+	return Families[i], true
+}
+
+// SizesOf are a family's sizes, with its figures.
+func SizesOf(family string) []Size {
+	var out []Size
+	for _, s := range Sizes {
+		if slices.Contains(s.Families, family) {
+			got, _ := SizeOf(family, s.Name)
+			out = append(out, got)
+		}
+	}
+	return out
+}
+
+// MemoryGB is what a replica of a size reserves of the host: its experts,
+// and the engine and server beside them.
+func (s Size) MemoryGB() int { return int(s.RAMGB+0.999) + 4 }
+
+func familyNames() []string {
+	out := make([]string, len(Families))
+	for i, f := range Families {
+		out[i] = f.Name
+	}
+	return out
+}
+
+// ArgValue is a flag's value in a profile's arguments, "" when absent.
+func ArgValue(args []string, flag string) string {
+	for i, a := range args {
+		if a == flag && i+1 < len(args) {
+			return args[i+1]
+		}
+		if v, ok := strings.CutPrefix(a, flag+"="); ok {
+			return v
+		}
+	}
+	return ""
+}
+
+// SizeForFile is the size of a family a file is, by its name: the longest
+// size name in it, so UD-Q4_K_XL is not read as Q4_K_XL's shorter cousins.
+func SizeForFile(family, file string) (Size, bool) {
+	upper := strings.ToUpper(file)
+	var best Size
+	for _, s := range SizesOf(family) {
+		if strings.Contains(upper, strings.ToUpper(s.Name)) && len(s.Name) > len(best.Name) {
+			best = s
+		}
+	}
+	return best, best.Name != ""
+}

internal/strata/models_test.go

@@ -0,0 +1,29 @@
+package strata
+
+import (
+	"strings"
+	"testing"
+)
+
+func TestASizeIsKnownForTheFamiliesThatHaveIt(t *testing.T) {
+	if s, err := SizeOf("qwen", "IQ2_XS"); err != nil || s.RAMGB != 48 || s.MemoryGB() != 52 {
+		t.Errorf("%+v %v", s, err)
+	}
+	if s, err := SizeOf("qwen36", "Q8_0"); err != nil || s.RAMGB != 52 {
+		t.Errorf("qwen36's own figures: %+v %v", s, err)
+	}
+	if _, err := SizeOf("coder", "IQ2_XS"); err == nil || !strings.Contains(err.Error(), "IQ1_M") {
+		t.Errorf("coder in IQ2_XS: %v", err)
+	}
+	if _, err := SizeOf("llama", "Q8_0"); err == nil || !strings.Contains(err.Error(), "deepseek41") {
+		t.Errorf("an unknown family: %v", err)
+	}
+	if f, ok := ForRepo("istA-daslab/qwen3.8-flash-next-gsq-rco-gguf"); !ok || f.Name != "qwen" {
+		t.Errorf("by repository: %+v", f)
+	}
+	for _, f := range Families {
+		if len(SizesOf(f.Name)) == 0 {
+			t.Errorf("%s has no sizes", f.Name)
+		}
+	}
+}

internal/web/admin_mirror.go

@@ -823,7 +823,11 @@ func (p *AccountPages) openRepo(r *http.Request, d *adminMirrorData, repo string
 			Version: got.ShortRevision(),
 		}
 		rs.Verdict = p.verdictFor(langOf(r), got.Architecture, set.Name)
-		if fit != nil && rs.Verdict.Servable {
+		// A family Strata runs: one card and the experts in RAM, whether
+		// or not llama.cpp could load it.
+		if layout, ok := fit.strataLayout(langOf(r), got.ID, set.Name, set.Bytes); ok {
+			rs.Verdict, rs.GPULayout = gguf.Verdict{Checked: true, Servable: true}, layout
+		} else if fit != nil && rs.Verdict.Servable {
 			if v, ok := fit.holds(langOf(r), set.Name, set.Bytes, got.Params); !ok {
 				rs.Verdict = v
 			} else if gguf.IsGGUF(set.Name) {

internal/web/admin_serve.go

@@ -4,14 +4,18 @@ import (
 	"context"
 	"errors"
 	"fmt"
-	"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
+	"slices"
 	"strconv"
 	"strings"
 	"time"
 
+	"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
+
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/strata"
 )
 
 // CatalogAdmin adds a model to the catalogue in force.
@@ -76,6 +80,11 @@ type serveForm struct {
 	// CtxPool is the engine context pool the form starts with, and the one
 	// the memory figure assumes.
 	CtxPool int
+	// Context is the context length the form starts with.
+	Context int
+	// StrataFamily and StrataSize are Strata's --family and --size, for a
+	// version of a family Strata runs, served on it.
+	StrataFamily, StrataSize string
 
 	// Refusal is set when this version cannot be served as it stands, and says
 	// why. Offered and then failing at placement is worse than not offered.
@@ -122,6 +131,35 @@ func planServe(l *Lang, st mirror.Status, files []mirror.Status, base string,
 		st.BytesTotal = total
 	}
 
+	f.Context = 4096
+	// Files of a family Strata runs, on a fleet that has Strata: served on
+	// it, from the version's directory, whether or not llama.cpp knows them.
+	if f.Engine == "llamacpp" && strataInstalled(fleet) {
+		if id, _, _, ok := hf.FileSource(st.Source, hf.DefaultEndpoint); ok {
+			if fam, ok := strata.ForRepo(id); ok {
+				if size, ok := strata.SizeForFile(fam.Name, st.File); ok {
+					f.Engine, f.StrataFamily, f.StrataSize = "strata", fam.Name, size.Name
+					f.Source = strings.TrimSuffix(mirror.Entry{Model: st.Model, Version: st.Version}.URL(base), "/") + "/"
+					f.Context, f.Profile = 32768, "gpu"
+					f.Placements, f.FreeGPUs = placementsFor(l, fleet, f.Engine)
+					for _, pc := range f.Placements {
+						if pc.Selected {
+							f.Profile = pc.profile()
+						}
+					}
+					f.GPUs = 1
+					f.MinMemoryGB = size.MemoryGB()
+					f.MemoryBasis = l.T("serve.basis.strata", fam.Title, size.Name)
+					if cat != nil && cat.Declared(st.Model) {
+						f.Already = true
+						f.Refusal = l.T("serve.refuse.declared", st.Model)
+					}
+					return f
+				}
+			}
+		}
+	}
+
 	switch f.Engine {
 	case "":
 		f.Refusal = l.T("serve.refuse.no_engine", extOf(st.File))
@@ -220,6 +258,19 @@ func extOf(file string) string {
 	return file
 }
 
+// strataInstalled reports whether a node of the fleet has Strata.
+func strataInstalled(fleet Source) bool {
+	if fleet == nil {
+		return false
+	}
+	for _, n := range fleet.Nodes() {
+		if slices.Contains(n.Engines, "strata") {
+			return true
+		}
+	}
+	return false
+}
+
 // runsAnywhere reports whether some node has this engine.
 //
 // True when nothing is known, so a fleet of agents older than the engine probe
@@ -304,6 +355,14 @@ func buildModel(l *Lang, get func(string) string) (catalog.Model, error) {
 	}
 	place := placeChoice{Value: strings.TrimSpace(get("placement"))}
 	switch class, onGPU := strings.CutPrefix(place.Value, "gpu:"); {
+	case onGPU && class != "" && engine == "strata":
+		// One card; the model's family and size, and the context.
+		p.GPUClass, p.GPUsPerNode, place.GPUs = class, 1, 1
+		family, size := strings.TrimSpace(get("strata_family")), strings.TrimSpace(get("strata_size"))
+		if _, err := strata.SizeOf(family, size); err != nil {
+			return m, err
+		}
+		p.Args = []string{"--family", family, "--size", size, "--context", strconv.Itoa(ctxLen), "--vision", "no"}
 	case onGPU && class != "":
 		// Placement is decided by gpu_class, which is what the selector
 		// sets; the count is gpus_per_node for llama.cpp, which spreads
@@ -330,7 +389,7 @@ func buildModel(l *Lang, get func(string) string) (catalog.Model, error) {
 	// The pool, not the per-request limit: llama.cpp divides it across its
 	// parallel slots, so the number a client may use is smaller and is what
 	// context_length above says.
-	if v := strings.TrimSpace(get("ctx_size")); v != "" {
+	if v := strings.TrimSpace(get("ctx_size")); v != "" && engine != "strata" {
 		if _, err := positive(l, v, "serve.what.ctx_pool"); err != nil {
 			return m, err
 		}

internal/web/admin_serve_test.go

@@ -12,6 +12,7 @@ import (
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
 )
 
@@ -546,3 +547,46 @@ func TestTheCPUWithGPUsFreeIsAChoiceMadeOnPurpose(t *testing.T) {
 		t.Error("with no GPU free the CPU is the only choice and should not be questioned")
 	}
 }
+
+func (f *fakeCatalog) Stop(_ context.Context, name, _ string) (catalog.Model, error) {
+	return catalog.Model{Name: name, Stopped: &catalog.Scaling{Min: 1, Max: 1}}, f.err
+}
+
+func (f *fakeCatalog) Start(_ context.Context, name, _ string) (catalog.Model, error) {
+	return catalog.Model{Name: name, Scaling: catalog.Scaling{Min: 1, Max: 1}}, f.err
+}
+
+// A version of a family Strata runs, on a fleet with Strata, is served on it:
+// one card, its family and size, the experts' memory, from the directory.
+func TestAStrataFamilysVersionIsServedOnStrata(t *testing.T) {
+	st := held("q38", "abc123", "Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00001-of-00002.gguf")
+	st.Source = "https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF/resolve/abc/IQ2_XS/" + st.File
+	files := []mirror.Status{st, held("q38", "abc123", "Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00002-of-00002.gguf")}
+	f := planServe(english, st, files, "http://mirror:8081", fakeFleet{gpus: 1, engines: []string{"llamacpp", "strata"}, memoryGB: 251},
+		nil, func(string) gguf.Verdict { return gguf.Verdict{Checked: true, Reason: "unknown architecture"} })
+	if f.Engine != "strata" || f.StrataFamily != "qwen" || f.StrataSize != "IQ2_XS" || f.Refusal != "" ||
+		f.MinMemoryGB != 52 || f.Context != 32768 || !strings.HasSuffix(f.Source, "/q38/abc123/") {
+		t.Fatalf("%+v", f)
+	}
+	// Without Strata on any node, llama.cpp as before, and its refusal.
+	if f := planServe(english, st, files, "http://mirror:8081", fakeFleet{gpus: 1, engines: []string{"llamacpp"}},
+		nil, func(string) gguf.Verdict { return gguf.Verdict{Checked: true, Reason: "unknown architecture"} }); f.Engine != "llamacpp" || f.Refusal == "" {
+		t.Errorf("%+v", f)
+	}
+	form := url.Values{"name": {"q38"}, "source": {f.Source}, "engine": {"strata"}, "replicas": {"1"},
+		"context_length": {"32768"}, "placement": {"gpu:6000-24g"}, "min_memory_gb": {"52"},
+		"strata_family": {"qwen"}, "strata_size": {"IQ2_XS"}}
+	m, err := buildModel(english, form.Get)
+	if err != nil {
+		t.Fatal(err)
+	}
+	p := m.Profiles[0]
+	if p.Engine != "strata" || p.GPUClass != "6000-24g" || p.GPUsPerNode != 1 || p.MinMemoryGB != 52 ||
+		strings.Join(p.Args, " ") != "--family qwen --size IQ2_XS --context 32768 --vision no" {
+		t.Errorf("%+v", p)
+	}
+	form.Set("strata_size", "IQ9")
+	if _, err := buildModel(english, form.Get); err == nil {
+		t.Error("an unknown size was taken")
+	}
+}

internal/web/explore_fit.go

@@ -9,6 +9,7 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/modelplan"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/strata"
 )
 
 // The explorer lists what this cluster can run, judged by the nodes it has
@@ -137,8 +138,43 @@ var pictureTasks = []string{"text-to-image", "image-to-image", "image-to-3d", "t
 // people want. A repository of Q8 only is judged kindly; its page says.
 const ggufBitsPerWeight = 4.8
 
+// strataLayout is how a file of a family Strata runs would run on it here,
+// and whether it could.
+func (c *clusterFit) strataLayout(l *Lang, repo, name string, bytes int64) (string, bool) {
+	if c == nil {
+		return "", false
+	}
+	fam, ok := strata.ForRepo(repo)
+	if !ok {
+		return "", false
+	}
+	opts := modelplan.StrataOptions(fam.Name, []modelplan.Artifact{{Name: name, Format: "gguf", Bytes: bytes}}, c.pool)
+	if len(opts) == 0 {
+		return "", false
+	}
+	return l.T("explore.strata", opts[0].Class, l.Num(opts[0].NeedGB, 0)), true
+}
+
+// strataRuns is the smallest of a Strata family's sizes that runs here, as
+// a layout, for a repository of the family.
+func (c *clusterFit) strataRuns(l *Lang, repo string) (string, bool) {
+	fam, ok := strata.ForRepo(repo)
+	if !ok {
+		return "", false
+	}
+	for _, s := range strata.SizesOf(fam.Name) {
+		if layout, ok := c.strataLayout(l, repo, s.Name+".gguf", int64(s.DownloadGB*1e9)); ok {
+			return layout, true
+		}
+	}
+	return "", false
+}
+
 // runs reports whether some node here could hold the repository's model.
 func (c *clusterFit) runs(m hf.Model) bool {
+	if _, ok := c.strataRuns(english, m.ID); ok {
+		return true
+	}
 	params := m.Params()
 	if slices.Contains(pictureTasks, m.PipelineTag) || m.HasTag("diffusers") {
 		b := m.WeightBytes()
@@ -329,6 +365,9 @@ func layoutLabel(l *Lang, o modelplan.Option, links modelplan.Links) string {
 // rowGPU is a listed repository's speed on the cards: its GGUF at Q4_K_M
 // on llama.cpp, its safetensors on vLLM, whichever is faster here.
 func (c *clusterFit) rowGPU(l *Lang, m hf.Model) (string, string) {
+	if layout, ok := c.strataRuns(l, m.ID); ok {
+		return "", layout
+	}
 	if !c.gpus() || m.Params() <= 0 {
 		return "", ""
 	}

internal/web/locales/en/mirror.json

@@ -256,5 +256,6 @@
   "mirror.err.not_adapter": "that file is a model, not a LoRA adapter",
   "mirror.err.wrong_arch": "that adapter is for %s models and %s is %s",
   "mirror.msg.adapter_added": "%[1]s: adapter %[2]s added; ask for it by that name once the replicas are restarted (llmgw ctl restart --model %[1]s)",
-  "mirror.err.adapter_op": "add or remove an adapter"
+  "mirror.err.adapter_op": "add or remove an adapter",
+  "explore.strata": "1× %s with Strata, %s GB of RAM"
 }

internal/web/locales/en/serve.json

@@ -81,5 +81,7 @@
   "serve.replicas_n.one": "%d replica",
   "serve.replicas_n.other": "%d replicas",
   "serve.msg.judge": "It is the guardrails' judge (role: guard): policies naming %s ask it; clients do not see it.",
-  "serve.msg.judge_renamed": "Its weights are a judge's, so it is the guardrails' judge (role: guard), named %[2]s as the policies ask rather than %[1]s."
+  "serve.msg.judge_renamed": "Its weights are a judge's, so it is the guardrails' judge (role: guard), named %[2]s as the policies ask rather than %[1]s.",
+  "serve.basis.strata": "what Strata keeps of %s %s's experts in RAM, and 4 GB for its engine and server",
+  "serve.strata": "Strata runs this (%s, %s): one card, and the experts in the node's RAM. The first start prepares the model on the node, which takes minutes."
 }

internal/web/locales/tr/mirror.json

@@ -249,5 +249,6 @@
   "mirror.err.not_adapter": "bu dosya bir LoRA adaptörü değil, bir model",
   "mirror.err.wrong_arch": "bu adaptör %s modelleri için; %s ise %s",
   "mirror.msg.adapter_added": "%[1]s: %[2]s adaptörü eklendi; replikalar yeniden başlatılınca bu adla isteyin (llmgw ctl restart --model %[1]s)",
-  "mirror.err.adapter_op": "bir adaptör ekleyin ya da kaldırın"
+  "mirror.err.adapter_op": "bir adaptör ekleyin ya da kaldırın",
+  "explore.strata": "Strata ile 1× %s, %s GB RAM"
 }

internal/web/locales/tr/serve.json

@@ -78,5 +78,7 @@
   "serve.msg.added": "%s kataloğa eklendi, %s: %s; bir sonraki geçişte yerleştirilecek",
   "serve.replicas_n.other": "%d replika",
   "serve.msg.judge": "Guardrail'lerin judge modeli olarak eklendi (role: guard): %s adını veren politikalar ona soruyor; istemciler görmüyor.",
-  "serve.msg.judge_renamed": "Ağırlıkları bir judge modelinin olduğu için guardrail judge'ı olarak eklendi (role: guard); adı %[1]s yerine politikaların istediği gibi %[2]s."
+  "serve.msg.judge_renamed": "Ağırlıkları bir judge modelinin olduğu için guardrail judge'ı olarak eklendi (role: guard); adı %[1]s yerine politikaların istediği gibi %[2]s.",
+  "serve.basis.strata": "Strata'nın %s %s için RAM'de tuttuğu uzmanlar, artı motoru ve sunucusu için 4 GB",
+  "serve.strata": "Bunu Strata çalıştırır (%s, %s): tek kart, uzmanlar node'un RAM'inde. İlk açılışta model node üzerinde hazırlanır; bu birkaç dakika sürer."
 }

internal/web/templates/admin_mirror.html

@@ -133,8 +133,11 @@
   <input type="hidden" name="csrf" value="{{$.CSRF}}">
   <input type="hidden" name="source" value="{{.Source}}">
   <input type="hidden" name="engine" value="{{.Engine}}">
+  {{if .StrataFamily}}<input type="hidden" name="strata_family" value="{{.StrataFamily}}">
+  <input type="hidden" name="strata_size" value="{{.StrataSize}}">{{end}}
   <p class="sub">{{th "serve.serving" .Source .Engine}}
   {{if .Files}}{{t "serve.whole_version" .Files}}{{end}}</p>
+  {{if .StrataFamily}}<p class="note">{{t "serve.strata" .StrataFamily .StrataSize}}</p>{{end}}
   <div class="field-row">
     <label>{{t "serve.name"}} <input type="text" name="name" value="{{.Model}}" required></label>
     <label>{{t "serve.aliases"}} <span class="opt">{{t "serve.aliases_hint"}}</span>
@@ -154,11 +157,11 @@
            placeholder="{{t "serve.description_example"}}"></label>
   <div class="field-row">
     <label>{{t "serve.context_length"}}
-      <input type="text" name="context_length" value="4096" required></label>
+      <input type="text" name="context_length" value="{{.Context}}" required></label>
     <label>{{t "serve.max_output"}} <span class="opt">{{t "serve.optional"}}</span>
       <input type="text" name="max_output" value="1024"></label>
-    <label>{{t "serve.ctx_pool"}} <span class="opt">{{t "serve.ctx_pool_hint"}}</span>
-      <input type="text" name="ctx_size" value="{{.CtxPool}}"></label>
+    {{if ne .Engine "strata"}}<label>{{t "serve.ctx_pool"}} <span class="opt">{{t "serve.ctx_pool_hint"}}</span>
+      <input type="text" name="ctx_size" value="{{.CtxPool}}"></label>{{end}}
   </div>
   {{/* Where it runs is a choice with consequences, so it is a choice: the
        GPU classes the fleet has, and the CPU for llama.cpp. A profile name
@@ -169,8 +172,8 @@
       <select name="placement">
         {{range .Placements}}<option value="{{.Value}}"{{if .Selected}} selected{{end}}>{{.Label}}</option>{{end}}
       </select></label>
-    <label>{{t "serve.gpus"}} <span class="opt">{{t "serve.gpus_hint"}}</span>
-      <input type="text" name="gpus" value="{{.GPUs}}" size="3"></label>
+    {{if ne .Engine "strata"}}<label>{{t "serve.gpus"}} <span class="opt">{{t "serve.gpus_hint"}}</span>
+      <input type="text" name="gpus" value="{{.GPUs}}" size="3"></label>{{end}}
     {{if eq .Engine "llamacpp"}}<label>{{t "serve.parallel"}} <span class="opt">{{t "serve.parallel_hint"}}</span>
       <input type="text" name="parallel" value="{{.Parallel}}" size="3"></label>{{end}}
   </div>