v0.15.3: changelog and README

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A model can be stopped and kept, or removed with its files from the mirror and the nodes
2026-10-29 20:54:53 +01:00 · 2026-10-29 20:45:48 +01:00 · 2026-10-29 20:01:47 +01:00 · 2026-10-29 19:25:00 +01:00 · 2026-10-29 18:36:59 +01:00 · 2026-10-29 18:30:30 +01:00
36 changed files with 1724 additions and 41 deletions
@@ -30,16 +30,11 @@ What is left, most needed first. The detail is in the item each names.
 **Soon: the customer's experience, and HA.**
 4. ~~**C24**~~: done 2026-10-21. The store's, catalog's and mirror's errors, percent signs, footer times and stale
   forms are in the page's language. Left: the /docs documents, and a language kept per person (cookie only today).
-5. **J4**: the model mirror on more than one machine. Code done 2026-10-29: mirrors kept in step (`mirror serve
-   --peer`, removals carried), `mirror.peers` and failover in the control plane, nodes given the file on a mirror that
-   answers, `mirror add-peer` and the setup's second mirror. Tried 2026-10-29 on two VMs:
-   - `add-peer` set the second up over ssh;
-   - a file came across in ~30 s, checksum matching;
-   - a version added on the second while the first was off reached the first when it came back;
-   - one removed on the first while the second was off was removed there when it came back, not copied back.
-
-   Left: the same with a control plane and a node, a model served while the first mirror is off. That part (failover
-   in the control plane, the node given the other mirror's file) is so far tried only in tests.
+5. ~~**J4**~~: done 2026-10-29. The done-criterion held on the PoC: with mirror 146 stopped, `ctl model serve` of a
+   new model was fetched by the second mirror (VM 167, 10.0.0.152), placed and answered; a node without a file
+   146's address names was given 152's; what was added meanwhile reached 146 when it came back; a removal was
+   carried both ways. Found and fixed there: add-peer on a mirror machine of its own, files without checksums never
+   copied, and `mirror remove` neither recorded nor seen by the running mirror.
 6. **F5**: large imports limited by the embedding engine's slots. (M)

 **From Kong** (section L): L1–L5 and L9 done (v0.15.0); left of L4 is routing by cost between outside
@@ -1652,7 +1647,7 @@ their own (4 cores, 32 GB, two cores each) the same test refused 2.
 - Done when: the setup installs the judges on the machines given for them,
  and the load test above refuses no more than it did by hand.

-### J4. The model mirror on more than one machine (M)
+### J4. The model mirror on more than one machine (M) — done 2026-10-29, tried on the PoC
 In an HA install the mirror stays on the first machine. While it is down,
 models the nodes already have keep serving and new ones wait for it.
 - A second mirror kept in step (each machine of the store, or one more), the
@@ -19,12 +19,60 @@ is only released once its **exit criterion** is met, not when its code lands.
 | v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released |
 | v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
 | v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines; v0.14.7: the judge sized for its machine; v0.14.8: sign-in with Active Directory, several keys for one outside provider; v0.14.9: sign-in set up from the console, a node's console that shows the machine) |
-| v0.15.0 | — | The company's MCP tools and VS Code through the gateway; guardrails that ask outside services and put redacted values back; answers to similar questions | released (v0.15.1: Strata, a 125B model of experts on one card; a second mirror kept in step; pickled weights refused; a served model's context changed) |
+| v0.15.0 | — | The company's MCP tools and VS Code through the gateway; guardrails that ask outside services and put redacted values back; answers to similar questions | released (v0.15.1: Strata, a 125B model of experts on one card; a second mirror kept in step; pickled weights refused; a served model's context changed; v0.15.2: a served model moved to more cards; v0.15.3: a model stopped and kept, or removed with its files) |
 | v1.0.0 | 5 | What phase 5 has left: precise KV routing proven on vLLM, the block cost picker, defragmentation across NVLink domains, prefill/decode disaggregation | planned |

 Until v1.0.0 the minor version is the breaking one: configuration formats and Go
 APIs change between minors without a deprecation period.

+## v0.15.3 — 2026-10-29
+
+A model stopped and kept, or removed with its files; the second mirror fixed where it was tried.
+
+**Upgrading.**
+- No migrations. The control planes carry stop, start and purge; the nodes need nothing new.
+- The mirrors carry the fixes: on each mirror's machine, `sudo llmgw upgrade v0.15.3 --service llmgw-mirror`.
+  A mirror with a peer then works out the checksums its older files lack, once, which reads every such file.
+
+### Added
+
+- **A model stopped and kept, and a model removed with its files.** `llmgw ctl model stop|start MODEL`: none of its
+  replicas run, it stays in the catalog with its files, and start runs it again scaled as it was; requests are told an
+  administrator stopped it. `llmgw ctl model remove MODEL --purge` (`DELETE /api/v1/models/{name}?purge=true`) also
+  deletes every version of it on the mirror, a second mirror following, and the nodes' copies once its replicas have
+  stopped; what another model uses stays. Both are in the console's model menu.
+
+### Fixed
+
+- **`llmgw mirror add-peer` on a mirror with a machine of its own.** It set the second mirror up and then
+  failed for want of the control plane's server.yaml; now it prints the `llmgw ctl config set mirror.peers`
+  to run on a control plane.
+- **Models fetched before checksums were kept reach the second mirror.** A mirror kept in step copies only what it
+  can check, and those files had no sha256; a mirror with peers now works theirs out and records them, once.
+- **A file removed with `llmgw mirror remove` is gone from the mirror's answers too.** The running mirror kept
+  reporting a file it had fetched as present for a week after it was removed, so a control plane could place a
+  model on a file that was not there; and the removal was not recorded, so a mirror kept in step copied it back.
+
+## v0.15.2 — 2026-10-29
+
+A served model moved to more cards, or fewer, while it is served.
+
+**Upgrading.**
+- No migrations, and nothing changes until the command is used. The control
+  planes carry it; the nodes need nothing new.
+
+### Added
+
+- **A served model moved to more cards, or fewer.** `llmgw ctl model gpus
+  MODEL N [--profile P] [--in-place]`, `POST /api/v1/models/{name}/gpus`, and
+  GPUs… in the console's model menu. How many cards of one node each replica
+  uses: tensor parallel for vLLM, SGLang and TensorRT-LLM, checked against
+  the model's KV heads, and the layers split for llama.cpp. Refused for a
+  count no node has, or whose cards would not hold the model and its
+  context, with the figures; its replicas are then started again on it, one
+  by one, beside the old — or, with `--in-place` when there is no room
+  beside them, each stopped first.
+
 ## v0.15.1 — 2026-10-29

 A 125B model on one card, a second mirror, and weights that run code
@@ -41,7 +41,7 @@ Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
 | v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released |
 | v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
 | v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines; v0.14.7: the judge sized for its machine; v0.14.8: sign-in with Active Directory, several keys for one outside provider; v0.14.9: sign-in set up from the console, a node's console that shows the machine) |
-| v0.15.0 | — | The company's MCP tools and VS Code through the gateway; guardrails that ask outside services and put redacted values back; answers to similar questions | released (v0.15.1: Strata, a 125B model of experts on one card; a second mirror kept in step; pickled weights refused; a served model's context changed) |
+| v0.15.0 | — | The company's MCP tools and VS Code through the gateway; guardrails that ask outside services and put redacted values back; answers to similar questions | released (v0.15.1: Strata, a 125B model of experts on one card; a second mirror kept in step; pickled weights refused; a served model's context changed; v0.15.2: a served model moved to more cards; v0.15.3: a model stopped and kept, or removed with its files) |
 | v1.0.0 | 5 | What phase 5 has left: precise KV routing proven on vLLM, the block cost picker, defragmentation across NVLink domains, prefill/decode disaggregation | planned |

 As of v0.6.0 a cluster runs unattended: agents report their hardware, the
@@ -2387,7 +2387,7 @@
      "delete": {
        "operationId": "deleteModelsName",
        "summary": "Remove a model added at runtime",
-        "description": "A model declared in the cluster's settings is refused: it is changed there.",
+        "description": "A model declared in the cluster's settings is refused: it is changed there. With purge, its files go too: every version of it the mirror holds (a peer mirror follows), and the nodes' copies once its replicas have stopped, waiting up to a minute for them; what another model uses stays. The answer is then a PurgeView.",
        "tags": [
          "models"
        ],
@@ -2401,6 +2401,14 @@
              "type": "string"
            }
          },
+          {
+            "name": "purge",
+            "in": "query",
+            "description": "Delete its files from the mirror and the nodes as well.",
+            "schema": {
+              "type": "boolean"
+            }
+          },
          {
            "$ref": "#/components/parameters/Actor"
          }
@@ -2964,6 +2972,58 @@
        }
      }
    },
+    "/api/v1/models/{name}/gpus": {
+      "post": {
+        "operationId": "postModelsNameGpus",
+        "summary": "Spread a served model over more cards or fewer: its profile, checked against its KV heads, its cards' memory and the nodes, then its replicas restarted",
+        "description": "Tensor parallel for vLLM, SGLang and TensorRT-LLM, a layer split for llama.cpp, on one node. With no room beside the running replicas, in_place stops each first.",
+        "tags": [
+          "models"
+        ],
+        "parameters": [
+          {
+            "name": "name",
+            "in": "path",
+            "required": true,
+            "description": "The model's name in the catalog.",
+            "schema": {
+              "type": "string"
+            }
+          },
+          {
+            "$ref": "#/components/parameters/Actor"
+          }
+        ],
+        "requestBody": {
+          "required": true,
+          "content": {
+            "application/json": {
+              "schema": {
+                "$ref": "#/components/schemas/modelserve.GPUsRequest"
+              }
+            }
+          }
+        },
+        "responses": {
+          "200": {
+            "description": "OK",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/modelserve.GPUsChange"
+                }
+              }
+            }
+          },
+          "401": {
+            "$ref": "#/components/responses/Unauthorized"
+          },
+          "default": {
+            "$ref": "#/components/responses/Error"
+          }
+        }
+      }
+    },
    "/api/v1/models/{name}/headroom": {
      "get": {
        "operationId": "getModelsNameHeadroom",
@@ -3348,6 +3408,89 @@
        }
      }
    },
+    "/api/v1/models/{name}/start": {
+      "post": {
+        "operationId": "postModelsNameStart",
+        "summary": "Run a stopped model again, scaled as it was",
+        "tags": [
+          "models"
+        ],
+        "parameters": [
+          {
+            "name": "name",
+            "in": "path",
+            "required": true,
+            "description": "The model's name in the catalog.",
+            "schema": {
+              "type": "string"
+            }
+          },
+          {
+            "$ref": "#/components/parameters/Actor"
+          }
+        ],
+        "responses": {
+          "200": {
+            "description": "OK",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/ScaleResponse"
+                }
+              }
+            }
+          },
+          "401": {
+            "$ref": "#/components/responses/Unauthorized"
+          },
+          "default": {
+            "$ref": "#/components/responses/Error"
+          }
+        }
+      }
+    },
+    "/api/v1/models/{name}/stop": {
+      "post": {
+        "operationId": "postModelsNameStop",
+        "summary": "Take a model out of service and keep it",
+        "description": "None of its replicas run, and requests are told an administrator stopped it; what it was scaled to is kept for start.",
+        "tags": [
+          "models"
+        ],
+        "parameters": [
+          {
+            "name": "name",
+            "in": "path",
+            "required": true,
+            "description": "The model's name in the catalog.",
+            "schema": {
+              "type": "string"
+            }
+          },
+          {
+            "$ref": "#/components/parameters/Actor"
+          }
+        ],
+        "responses": {
+          "200": {
+            "description": "OK",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/ScaleResponse"
+                }
+              }
+            }
+          },
+          "401": {
+            "$ref": "#/components/responses/Unauthorized"
+          },
+          "default": {
+            "$ref": "#/components/responses/Error"
+          }
+        }
+      }
+    },
    "/api/v1/models/{name}/target": {
      "get": {
        "operationId": "getModelsNameTarget",
@@ -7807,6 +7950,10 @@
          "max_output": {
            "type": "integer"
          },
+          "stopped": {
+            "type": "boolean",
+            "description": "Stopped is a model an administrator took out of service, kept."
+          },
          "canary": {
            "$ref": "#/components/schemas/catalog.Canary",
            "description": "Canary is a share of its requests sent to a candidate; CanaryOf is set on the candidate, naming the model it is a canary of."
@@ -8887,6 +9034,10 @@
          },
          "on_demand": {
            "type": "boolean"
+          },
+          "stopped": {
+            "type": "boolean",
+            "description": "Stopped is a model taken out of service, kept, until it is started."
          }
        }
      },
@@ -10012,6 +10163,10 @@
          "scaling": {
            "$ref": "#/components/schemas/catalog.Scaling"
          },
+          "stopped": {
+            "$ref": "#/components/schemas/catalog.Scaling",
+            "description": "Stopped is the scaling a model had when an administrator stopped it: kept in the catalog with none running (Scaling 0/0) until started again with this. Nil for a model not stopped."
+          },
          "class": {
            "type": "string",
            "description": "Class is how much this model's capacity is worth against another's, when there is not enough for both: critical, standard or opportunistic. Empty is standard, or opportunistic for a model loaded on demand. See EffectiveClass."
@@ -11133,6 +11288,67 @@
          }
        }
      },
+      "modelserve.GPUsChange": {
+        "type": "object",
+        "description": "GPUsChange is what was changed, and the replicas started again on it.",
+        "properties": {
+          "model": {
+            "type": "string"
+          },
+          "profile": {
+            "type": "string"
+          },
+          "engine": {
+            "type": "string"
+          },
+          "gpu_class": {
+            "type": "string"
+          },
+          "before": {
+            "type": "integer"
+          },
+          "after": {
+            "type": "integer"
+          },
+          "args": {
+            "type": "array",
+            "items": {
+              "type": "string"
+            }
+          },
+          "per_card": {
+            "type": "string",
+            "description": "PerCard is what each card holds after, when it could be worked out."
+          },
+          "restarted": {
+            "type": "integer"
+          },
+          "in_place": {
+            "type": "boolean"
+          },
+          "note": {
+            "type": "string",
+            "description": "Note is what could not be checked, said once."
+          }
+        }
+      },
+      "modelserve.GPUsRequest": {
+        "type": "object",
+        "description": "GPUsRequest is how many cards a replica is to use.",
+        "properties": {
+          "gpus": {
+            "type": "integer"
+          },
+          "profile": {
+            "type": "string",
+            "description": "Profile is the one to change, when the model has more than one profile on cards."
+          },
+          "in_place": {
+            "type": "boolean",
+            "description": "InPlace stops each replica before its replacement starts: for when no node has the cards free beside it. It does not answer while the new one loads."
+          }
+        }
+      },
      "modelserve.Job": {
        "type": "object",
        "description": "Job is one model being served, and where it stands.",
@@ -1629,6 +1629,30 @@ Not yet: models declared in server.yaml cannot name a peer's model as a
 fallback (the catalog in the file is checked before the peers are read), and
 a peer behind a private CA needs that CA in the system's trust store.

+### Stopping a model, and removing one with its files
+
+```
+llmgw ctl model stop qwen-7b             # none of its replicas run; it is kept
+llmgw ctl model start qwen-7b            # runs again, scaled as it was
+llmgw ctl model remove qwen-7b           # leaves the catalog; its files stay
+llmgw ctl model remove qwen-7b --purge   # its files on the mirror and the nodes too
+```
+
+- **Stop** keeps the model in the catalog with its files and what it was
+  scaled to; a request for it is told an administrator stopped it. `ctl
+  models` shows it as stopped. A `ctl scale` while it is stopped starts it at
+  that number.
+- **Remove** leaves its files on the mirror and the nodes, for adding it
+  again without fetching it. `ctl weights` lists such files on the nodes.
+- **Remove --purge** deletes every version of it on the mirror (a second
+  mirror follows), its adapters and the weights its last update replaced,
+  and the nodes' copies once its replicas have stopped. What another model
+  uses — a canary's weights, a shared adapter — stays. It cannot be undone:
+  adding it again fetches it again.
+
+The console has the same in a model's menu: Stop… or Start, Remove…, and
+Remove with its files….
+
 ### Changing a served model's context

 How much one request may use is set when a model is served. To change it
@@ -1648,6 +1672,36 @@ it, one by one, a new one ready before the old one stops. The console has the
 same under a model's menu, Context length…. A model declared in the cluster's
 settings is changed there (`llmgw ctl config edit`).

+### Moving a served model to more cards
+
+How many cards of one node each replica uses can be changed while the model
+is served:
+
+```
+llmgw ctl model gpus qwen3-coder-30b 2              # two cards a replica
+llmgw ctl model gpus llama-70b 4 --profile h100     # the model has several profiles on cards
+llmgw ctl model gpus qwen3-coder-30b 2 --in-place   # no room beside the running replica
+```
+
+vLLM, SGLang and TensorRT-LLM spread the model by tensor parallel (`tp`),
+which needs the count to divide the model's KV heads or be divided by them:
+a model with 8 splits 2, 4, 8 or 16 ways, not 3 or 12. llama.cpp has no
+tensor parallel; it splits the layers over the cards (`gpus_per_node`), and
+a `--tensor-split` written for the old cards is dropped. ComfyUI and Strata
+run on one card. A count no node has is refused, and so is one whose cards
+would not hold the weights and the context — with the figures; fewer cards
+with a long context may need `llmgw ctl model context` first.
+
+Its replicas are then started again on it, each new one ready beside the old
+before the old stops. That needs the cards free beside it: on a node of two
+cards with the model on one, two do not fit until it is gone, and the command
+says so. `--in-place` stops each replica first and starts its replacement on
+the cards it held and the free ones, and the model does not answer while the
+new one loads — a model with several replicas keeps answering on the others.
+A replica spread over several nodes is laid out by hand, with a fabric
+(`nodes`, `gpus_per_node` in the profile). The console has the same under a
+model's menu, GPUs….
+
 ### Trying new weights on some of the traffic first

 When a model's repository publishes new weights, `llmgw ctl model update`
@@ -2553,6 +2607,12 @@ is down, and new ones wait for it. With a second, the cluster carries on.
  ```
  sudo llmgw mirror add-peer ubuntu@10.0.0.7 --control-plane 10.0.0.5 --control-plane 10.0.0.6 --control-plane 10.0.0.7
  ```
+
+  It logs in with root's ssh key on the mirror's machine, so that key must
+  be accepted for the user named on the second machine. When the mirror has
+  a machine of its own, without the control plane, the command cannot change
+  the cluster's settings itself: it prints the `llmgw ctl config set
+  mirror.peers …` to run on a control plane.
 - **Kept in step:** each mirror reads the other every thirty seconds and
  copies what it lacks, keeping a copy only if its checksum matches.
  - A model removed from one is removed from the other, even if that one
@@ -192,6 +192,7 @@ rather than repeated.
 | `allow_pickle` | `bool` | [Model](#model) |
 | `profiles` | `[]Profile` | [Model](#model) |
 | `scaling` | `Scaling` | [Model](#model) |
+| `stopped` | `*Scaling` | [Model](#model) |
 | `class` | `string` | [Model](#model) |
 | `owner` | `string` | [Model](#model) |
 | `external` | `*External` | [Model](#model) |
@@ -1824,6 +1825,16 @@ no safetensors or GGUF form and whose files were checked.

 Takes a [Scaling](#scaling) block.

+### `stopped`
+
+`*Scaling`
+
+Stopped is the scaling a model had when an administrator stopped it:
+kept in the catalog with none running (Scaling 0/0) until started
+again with this. Nil for a model not stopped.
+
+Takes a [Scaling](#scaling) block.
+
 ### `class`

 `string`
@@ -321,6 +321,8 @@ func locationColumn(n adminapi.NodeView) string {
 // actually served, never what is merely catalogued.
 func modelStatus(m adminapi.ModelView) string {
 	switch {
+	case m.Stopped:
+		return "stopped (llmgw ctl model start)"
 	case m.Parked:
 		return "parked"
 	case m.External != "":
@@ -238,6 +238,10 @@ func newMirrorRemoveCmd(dir *string) *cobra.Command {
 			if err := mirror.Remove(*dir, e); err != nil {
 				return err
 			}
+			// Kept for a mirror kept in step, which otherwise copies it back.
+			if err := mirror.RecordRemoval(*dir, model, version); err != nil {
+				return err
+			}
 			fmt.Fprintf(cmd.OutOrStdout(), "removed %s/%s/%s\n", model, version, file)
 			return nil
 		},
@@ -34,6 +34,8 @@ step with this one: each copies what the other holds, checksum checked, and
 a model removed from one is removed from the other. This mirror is told of
 it, and the cluster's settings name it (mirror.peers), so while either
 machine is down the control planes use the other and nodes fetch from it.
+On a mirror machine without the control plane's settings file, it prints
+the llmgw ctl config set to run on a control plane instead.

 HOST needs room for every model this mirror holds: it copies them all.`,
 		Example: "  sudo llmgw mirror add-peer ubuntu@10.0.0.12 --control-plane 10.0.0.10 --control-plane 10.0.0.12",
@@ -123,6 +125,19 @@ func addMirrorPeer(ctx context.Context, out io.Writer, remote onboard.Remote, ho
 	fmt.Fprintf(out, "  ok    this mirror keeps in step with %s\n", peerAPI)

 	// The control planes told of both.
+	return nameMirrorPeer(out, configPath, peerAPI)
+}
+
+// nameMirrorPeer names a peer in the cluster's settings. A mirror on a
+// machine of its own has neither the settings file nor the store: it says
+// what to run on a control plane.
+func nameMirrorPeer(out io.Writer, configPath, peerAPI string) error {
+	if _, err := os.Stat(configPath); errors.Is(err, os.ErrNotExist) {
+		fmt.Fprintf(out, "  !!    this machine has no %s, so the cluster's settings are not changed here. On a control plane:\n"+
+			"          sudo llmgw ctl config set mirror.peers '[%s]'\n"+
+			"        (with any peers already named, which llmgw ctl config get mirror.peers shows)\n", configPath, peerAPI)
+		return nil
+	}
 	if err := addMirrorPeerSetting(configPath, peerAPI); err != nil {
 		return err
 	}
@@ -1,6 +1,7 @@
 package cli

 import (
+	"bytes"
 	"os"
 	"path/filepath"
 	"strings"
@@ -72,3 +73,13 @@ func TestTheSetupOffersASecondMirrorOnAStoreMachine(t *testing.T) {
 		t.Errorf("without standbys: %v", err)
 	}
 }
+
+func TestAMirrorOnAMachineOfItsOwnSaysWhatToRunOnAControlPlane(t *testing.T) {
+	var out bytes.Buffer
+	if err := nameMirrorPeer(&out, filepath.Join(t.TempDir(), "server.yaml"), "http://10.0.0.12:8082"); err != nil {
+		t.Fatal(err)
+	}
+	if !strings.Contains(out.String(), "llmgw ctl config set mirror.peers '[http://10.0.0.12:8082]'") {
+		t.Errorf("%s", out.String())
+	}
+}
@@ -34,7 +34,7 @@ func newModelCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 		Use:   "model",
 		Short: "Add or remove models while the cluster is running",
 	}
-	c.AddCommand(newModelContextCmd(client), newModelAddCmd(client), newModelPresetsCmd(client, asJSON), newModelRemoveCmd(client), newModelPlanCmd(client),
+	c.AddCommand(newModelStopCmd(client), newModelStartCmd(client), newModelContextCmd(client), newModelGPUsCmd(client), newModelAddCmd(client), newModelPresetsCmd(client, asJSON), newModelRemoveCmd(client), newModelPlanCmd(client),
 		newResourcesCmd(client), newClassCmd(client), newOwnerCmd(client),
 		newTargetCmd(client, asJSON), newAdaptersCmd(client),
 		newModelServeCmd(client), newModelUpdatesCmd(client, asJSON), newModelUpdateCmd(client),
@@ -163,21 +163,104 @@ func readModelFile(path string) (catalog.Model, error) {
 }

 func newModelRemoveCmd(client func() *ctlClient) *cobra.Command {
-	return &cobra.Command{
+	var purge bool
+	c := &cobra.Command{
 		Use:   "remove <name>",
-		Short: "Remove a model that was added at runtime",
+		Short: "Remove a model that was added at runtime, and with --purge its files",
 		Long: "Only models added at runtime. One declared in server.yaml would come\n" +
 			"back on the next restart, so removing it here would leave the cluster\n" +
 			"disagreeing with its own configuration until then.\n\n" +
 			"Its replicas are stopped on the next pass, after the endpoint has been\n" +
-			"withdrawn, so traffic stops before the engine does.",
+			"withdrawn, so traffic stops before the engine does.\n\n" +
+			"Its files stay on the mirror and the nodes, for adding it again. With\n" +
+			"--purge they go too: every version of it on the mirror (a mirror kept in\n" +
+			"step follows), and the nodes' copies once its replicas have stopped.\n" +
+			"What another model uses stays. To take a model out of service and keep\n" +
+			"it, use llmgw ctl model stop.",
+		Example: "  llmgw ctl model remove qwen-7b\n" +
+			"  llmgw ctl model remove qwen-7b --purge   # its files on the mirror and the nodes too",
 		Args: cobra.ExactArgs(1),
 		RunE: func(cmd *cobra.Command, args []string) error {
-			var res map[string]string
-			if err := client().delete(cmd.Context(), "/api/v1/models/"+args[0], &res); err != nil {
+			out := cmd.OutOrStdout()
+			if !purge {
+				var res map[string]string
+				if err := client().delete(cmd.Context(), "/api/v1/models/"+args[0], &res); err != nil {
+					return err
+				}
+				fmt.Fprintf(out, "%s removed. Its files stay on the mirror and the nodes; --purge deletes them.\n", args[0])
+				return nil
+			}
+			var v adminapi.PurgeView
+			if err := client().patient(3*time.Minute).delete(cmd.Context(), "/api/v1/models/"+args[0]+"?purge=true", &v); err != nil {
 				return err
 			}
-			fmt.Fprintf(cmd.OutOrStdout(), "%s removed.\n", args[0])
+			fmt.Fprintf(out, "%s removed.\n", v.Removed)
+			for _, m := range v.Mirror {
+				fmt.Fprintf(out, "  mirror  %s deleted\n", m)
+			}
+			for _, m := range v.Kept {
+				fmt.Fprintf(out, "  mirror  %s kept: another model uses it\n", m)
+			}
+			var freed int64
+			for _, n := range v.Nodes {
+				if n.Error != "" {
+					fmt.Fprintf(out, "  node    %s: %s\n", n.Node, n.Error)
+					continue
+				}
+				freed += n.Bytes
+				fmt.Fprintf(out, "  node    %s: %d file(s), %s\n", n.Node, n.Files, humanBytes(n.Bytes))
+			}
+			if len(v.Mirror) == 0 && len(v.Nodes) == 0 && v.Note == "" {
+				fmt.Fprintln(out, "  it had no files of its own on the mirror or the nodes")
+			}
+			if freed > 0 {
+				fmt.Fprintf(out, "  %s freed on the nodes\n", humanBytes(freed))
+			}
+			if v.Note != "" {
+				fmt.Fprintln(out, "  "+v.Note)
+			}
+			return nil
+		},
+	}
+	c.Flags().BoolVar(&purge, "purge", false, "delete its files from the mirror and the nodes as well")
+	return c
+}
+
+func newModelStopCmd(client func() *ctlClient) *cobra.Command {
+	return &cobra.Command{
+		Use:   "stop <model>",
+		Short: "Take a model out of service and keep it: none of its replicas run until it is started",
+		Long: "Its replicas stop; it stays in the catalog with its files, and a request\n" +
+			"for it is told an administrator stopped it. What it was scaled to is kept:\n" +
+			"llmgw ctl model start runs it again as it was.",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			var v adminapi.ScaleResponse
+			if err := client().post(cmd.Context(), "/api/v1/models/"+args[0]+"/stop", nil, &v); err != nil {
+				return err
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "%s stopped: its replicas stop on the next pass. It is kept; "+
+				"llmgw ctl model start %s runs it again.\n", v.Model, v.Model)
+			return nil
+		},
+	}
+}
+
+func newModelStartCmd(client func() *ctlClient) *cobra.Command {
+	return &cobra.Command{
+		Use:   "start <model>",
+		Short: "Run a stopped model again, scaled as it was when it was stopped",
+		Args:  cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			var v adminapi.ScaleResponse
+			if err := client().post(cmd.Context(), "/api/v1/models/"+args[0]+"/start", nil, &v); err != nil {
+				return err
+			}
+			how := fmt.Sprintf("%d replica(s)", v.Min)
+			if v.OnDemand {
+				how = "on demand"
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "%s started: %s, max %d. Watch it with llmgw ctl models.\n", v.Model, how, v.Max)
 			return nil
 		},
 	}
@@ -442,3 +525,57 @@ func newModelContextCmd(client func() *ctlClient) *cobra.Command {
 	c.Flags().IntVar(&slots, "slots", 0, "llama.cpp: requests a replica answers at once (default: as it is)")
 	return c
 }
+
+func newModelGPUsCmd(client func() *ctlClient) *cobra.Command {
+	var profile string
+	var inPlace bool
+	c := &cobra.Command{
+		Use:   "gpus <model> <cards>",
+		Short: "Spread a served model over more cards, or fewer",
+		Long: "How many cards of one node each replica uses. vLLM, SGLang and TensorRT-LLM\n" +
+			"spread the model by tensor parallel, which needs the count to divide the\n" +
+			"model's KV heads or be divided by them; llama.cpp splits its layers over the\n" +
+			"cards. It refuses a count a node does not have, and one whose cards would not\n" +
+			"hold the model and its context, saying what each would need. Its replicas are\n" +
+			"then started again on it, each new one beside the old before it stops.\n\n" +
+			"With no node free enough beside the running replicas — a node of two cards,\n" +
+			"the model on one — --in-place stops each replica first: the model does not\n" +
+			"answer while the new one loads.\n\n" +
+			"--profile names the profile for a model with more than one on cards.",
+		Example: "  llmgw ctl model gpus qwen3-coder-30b 2\n" +
+			"  llmgw ctl model gpus llama-70b 4 --profile h100 --in-place",
+		Args: cobra.ExactArgs(2),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			n, err := strconv.Atoi(args[1])
+			if err != nil {
+				return fmt.Errorf("the cards are a whole number, not %q", args[1])
+			}
+			body, _ := json.Marshal(modelserve.GPUsRequest{GPUs: n, Profile: profile, InPlace: inPlace})
+			var v modelserve.GPUsChange
+			if err := client().patient(time.Minute).post(cmd.Context(), "/api/v1/models/"+args[0]+"/gpus", body, &v); err != nil {
+				return err
+			}
+			out := cmd.OutOrStdout()
+			fmt.Fprintf(out, "%s: %d → %d %s card(s) a replica (%s, %s)\n", v.Model, v.Before, v.After, v.Class, v.Profile, v.Engine)
+			if v.PerCard != "" {
+				fmt.Fprintln(out, "  "+v.PerCard)
+			}
+			if len(v.Args) > 0 {
+				fmt.Fprintln(out, "  args: "+strings.Join(v.Args, " "))
+			}
+			if v.Note != "" {
+				fmt.Fprintln(out, "  note: "+v.Note)
+			}
+			switch {
+			case v.Restarted > 0 && v.InPlace:
+				fmt.Fprintf(out, "%d replica(s) stopped and started again on it, one by one: llmgw ctl replicas --model %s\n", v.Restarted, v.Model)
+			case v.Restarted > 0:
+				fmt.Fprintf(out, "%d replica(s) started again on it, each new one before the old stops: llmgw ctl replicas --model %s\n", v.Restarted, v.Model)
+			}
+			return nil
+		},
+	}
+	c.Flags().StringVar(&profile, "profile", "", "the profile to change, for a model with several on cards")
+	c.Flags().BoolVar(&inPlace, "in-place", false, "stop each replica before its replacement starts, when there is no room beside it")
+	return c
+}
@@ -583,6 +583,10 @@ func newServerCmd() *cobra.Command {
 					agentsvc.SetSourceRewrite(src.Rewrite)
 				}
 				rt.SetMirror(mc)
+				if adminAPI != nil {
+					// A model removed with its files is removed from it too.
+					adminAPI.SetPurgeMirror(mc)
+				}
 				if fwPub != nil {
 					fwPub.setMirror(mc)
 				}
@@ -607,7 +611,7 @@ func newServerCmd() *cobra.Command {
 						// Updates: the catalog in force, the service that
 						// changes it, and what replaces replicas.
 						Current: func() []catalog.Model { return admin.Catalog().Models() },
-						Models:  catalogSvc, Restart: admin.Reconciler().Restart,
+						Models:  catalogSvc, Restart: admin.Reconciler().Restart, RestartInPlace: admin.Reconciler().RestartInPlace,
 						Pool: adminAPI.PoolNodes, Notify: notifier,
 						// Canaries: how the router saw each side serve.
 						CanaryStats: rt.CanaryStats, ResetCanary: rt.ResetCanary,
@@ -97,10 +97,21 @@ func (m *consoleModel) openForm(f *formDialog) {

 // call runs one admin API request in the background and reports it.
 func (m *consoleModel) call(title, method, path string, body any, say func(raw json.RawMessage) string) tea.Cmd {
+	return m.callWaiting(title, method, path, body, say, 60*time.Second)
+}
+
+// callWaiting is call for a change the server may spend longer on than the
+// console's usual wait.
+func (m *consoleModel) callWaiting(title, method, path string, body any, say func(raw json.RawMessage) string,
+	wait time.Duration,
+) tea.Cmd {
 	m.busy = title
 	c := m.client
+	if wait > 60*time.Second {
+		c = c.patient(wait)
+	}
 	return func() tea.Msg {
-		ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
+		ctx, cancel := context.WithTimeout(context.Background(), wait)
 		defer cancel()
 		var raw json.RawMessage
 		// A change may answer with no body at all.
@@ -356,6 +367,31 @@ func (m consoleModel) modelMenu(md adminapi.ModelView) *menuDialog {
 				}})
 			return nil
 		}},
+		{"GPUs…", func(m *consoleModel) tea.Cmd {
+			profiles := append([]string{"(its only one on cards)"}, md.Profiles...)
+			m.openForm(&formDialog{title: "Cards of " + md.Name,
+				note: "How many cards of one node each replica uses: tensor parallel for vLLM, SGLang and TensorRT-LLM, " +
+					"the layers split for llama.cpp. Checked against the model's heads, its cards' memory and the nodes; " +
+					"its replicas are then started again on it, one by one.",
+				fields: []formField{
+					choiceF("profile", profiles, profiles[0]),
+					textF("cards", "", "a replica's, on one node"),
+					choiceF("in place", []string{"no", "yes"}, "no"),
+				},
+				submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
+					n, err := strconv.Atoi(v[1])
+					if err != nil || n < 1 {
+						return nil, fmt.Errorf("the cards are a whole number from 1")
+					}
+					profile := v[0]
+					if profile == profiles[0] {
+						profile = ""
+					}
+					return m.call("Cards of "+md.Name, http.MethodPost, path+"/gpus",
+						modelserve.GPUsRequest{GPUs: n, Profile: profile, InPlace: v[2] == "yes"}, gpusSays), nil
+				}})
+			return nil
+		}},
 		{"Resources…", func(m *consoleModel) tea.Cmd {
 			profiles := md.Profiles
 			if len(profiles) == 0 {
@@ -454,13 +490,40 @@ func (m consoleModel) modelMenu(md adminapi.ModelView) *menuDialog {
 			return m.call("Retry "+md.Name, http.MethodPost, path+"/retry", map[string]string{}, nil)
 		}})
 	}
+	if md.Stopped {
+		items = append(items, menuItem{"Start", func(m *consoleModel) tea.Cmd {
+			return m.call("Start "+md.Name, http.MethodPost, path+"/start", nil, func(json.RawMessage) string {
+				return md.Name + " runs again, scaled as it was when it was stopped."
+			})
+		}})
+	} else {
+		items = append(items, menuItem{"Stop…", func(m *consoleModel) tea.Cmd {
+			m.confirmD = &confirmDialog{title: "Stop " + md.Name,
+				text: "Its replicas stop; it stays in the catalog with its files, and Start runs it again as it was.",
+				yes: func(m *consoleModel) tea.Cmd {
+					return m.call("Stop "+md.Name, http.MethodPost, path+"/stop", nil, func(json.RawMessage) string {
+						return md.Name + " is stopped and kept."
+					})
+				}}
+			return nil
+		}})
+	}
 	items = append(items, menuItem{"Remove…", func(m *consoleModel) tea.Cmd {
 		m.confirmD = &confirmDialog{title: "Remove " + md.Name,
-			text: "Its replicas stop and it leaves the catalog. A model from server.yaml cannot be removed here.",
+			text: "Its replicas stop and it leaves the catalog; its files stay on the mirror and the nodes. " +
+				"A model from server.yaml cannot be removed here.",
 			yes: func(m *consoleModel) tea.Cmd {
 				return m.call("Remove "+md.Name, http.MethodDelete, path, nil, nil)
 			}}
 		return nil
+	}}, menuItem{"Remove with its files…", func(m *consoleModel) tea.Cmd {
+		m.confirmD = &confirmDialog{title: "Remove " + md.Name + " and its files",
+			text: "It leaves the catalog, every version of it is deleted from the mirror, and the nodes' copies once " +
+				"its replicas have stopped. What another model uses stays. This cannot be undone: adding it again fetches it again.",
+			yes: func(m *consoleModel) tea.Cmd {
+				return m.callWaiting("Remove "+md.Name, http.MethodDelete, path+"?purge=true", nil, purgeSays, 3*time.Minute)
+			}}
+		return nil
 	}})
 	return &menuDialog{title: md.Name, items: items}
 }
@@ -943,3 +1006,53 @@ func contextSays(raw json.RawMessage) string {
 	}
 	return s
 }
+
+// gpusSays is a change of cards in a sentence or two.
+func gpusSays(raw json.RawMessage) string {
+	var v modelserve.GPUsChange
+	if json.Unmarshal(raw, &v) != nil {
+		return "Done."
+	}
+	s := fmt.Sprintf("%s: %d → %d %s card(s) a replica.", v.Model, v.Before, v.After, v.Class)
+	if v.PerCard != "" {
+		s += "\n" + v.PerCard
+	}
+	if v.Note != "" {
+		s += "\n" + v.Note
+	}
+	switch {
+	case v.Restarted > 0 && v.InPlace:
+		s += fmt.Sprintf("\n%d replica(s) stopped and started again on it, one by one.", v.Restarted)
+	case v.Restarted > 0:
+		s += fmt.Sprintf("\n%d replica(s) started again on it, one by one.", v.Restarted)
+	}
+	return s
+}
+
+// purgeSays is a model removed with its files, in a few lines.
+func purgeSays(raw json.RawMessage) string {
+	var v adminapi.PurgeView
+	if json.Unmarshal(raw, &v) != nil {
+		return "Done."
+	}
+	s := v.Removed + " removed."
+	if len(v.Mirror) > 0 {
+		s += fmt.Sprintf("\nMirror: %d version(s) deleted.", len(v.Mirror))
+	}
+	if len(v.Kept) > 0 {
+		s += "\nKept on the mirror, another model uses them: " + strings.Join(v.Kept, ", ")
+	}
+	var files int
+	var freed int64
+	for _, n := range v.Nodes {
+		files += n.Files
+		freed += n.Bytes
+	}
+	if files > 0 {
+		s += fmt.Sprintf("\nNodes: %d file(s), %s freed.", files, humanBytes(freed))
+	}
+	if v.Note != "" {
+		s += "\n" + v.Note
+	}
+	return s
+}
@@ -60,6 +60,9 @@ type API struct {
 	controlPlanes func() ControlPlanes
 	// serve runs registry-to-replica jobs; nil without a mirror and a store.
 	serve *modelserve.Server
+	// purgeMirror is the mirror a model removed with its files is removed
+	// from; nil without one.
+	purgeMirror PurgeMirror
 	// history is the Prometheus the console's charts read from.
 	history History
 	// audit reads the audit log.
@@ -123,6 +126,8 @@ func (a *API) Register(mux Mux) {
 	mux.HandleFunc("POST /api/v1/models", a.addModel)
 	mux.HandleFunc("GET /api/v1/models/{name}", a.getModel)
 	mux.HandleFunc("DELETE /api/v1/models/{name}", a.removeModel)
+	mux.HandleFunc("POST /api/v1/models/{name}/stop", a.stopModel)
+	mux.HandleFunc("POST /api/v1/models/{name}/start", a.startModel)
 	mux.HandleFunc("POST /api/v1/models/{name}/scale", a.scaleModel)
 	mux.HandleFunc("POST /api/v1/models/{name}/resources", a.setResources)
 	a.registerQuotas(mux)
@@ -206,6 +211,7 @@ func (a *API) Register(mux Mux) {
 	mux.HandleFunc("POST /api/v1/updates/check", a.checkUpdates)
 	mux.HandleFunc("POST /api/v1/models/{name}/update", a.updateModel)
 	mux.HandleFunc("POST /api/v1/models/{name}/context", a.setModelContext)
+	mux.HandleFunc("POST /api/v1/models/{name}/gpus", a.setModelGPUs)
 	mux.HandleFunc("POST /api/v1/models/{name}/rollback", a.rollbackModel)
 	a.registerCanary(mux)
 	a.registerEval(mux)
@@ -450,6 +456,8 @@ type ModelView struct {
 	// answer; 0 when the catalog does not say.
 	ContextLength int `json:"context_length,omitempty"`
 	MaxOutput     int `json:"max_output,omitempty"`
+	// Stopped is a model an administrator took out of service, kept.
+	Stopped bool `json:"stopped,omitempty"`
 	// Canary is a share of its requests sent to a candidate; CanaryOf is
 	// set on the candidate, naming the model it is a canary of.
 	Canary   *catalog.Canary `json:"canary,omitempty"`
@@ -842,7 +850,7 @@ func (a *API) Models() []ModelView {
 		h := health[m.Name]
 		v := ModelView{
 			Name: m.Name, Description: m.Description, Aliases: m.Aliases, Source: m.Source,
-			ContextLength: m.ContextLength, MaxOutput: m.MaxOutput,
+			ContextLength: m.ContextLength, MaxOutput: m.MaxOutput, Stopped: m.Stopped != nil,
 			Min: m.Scaling.Min, Max: m.Scaling.Max,
 			Ready: ready[m.Name], Total: total[m.Name],
 			Fallbacks: m.Fallbacks,
@@ -944,6 +952,8 @@ type Models interface {
 	Add(ctx context.Context, m catalog.Model, by string) error
 	Remove(ctx context.Context, name string) error
 	Scale(ctx context.Context, name string, s catalog.Scaling, by string) (catalog.Model, error)
+	Stop(ctx context.Context, name, by string) (catalog.Model, error)
+	Start(ctx context.Context, name, by string) (catalog.Model, error)
 	Resources(ctx context.Context, name, profile string, memoryGB, cpuCores int, by string) (catalog.Model, error)
 	SetClass(ctx context.Context, name, class, by string) (catalog.Model, error)
 	SetOwner(ctx context.Context, name, owner, by string) (catalog.Model, error)
@@ -1193,6 +1203,8 @@ type ScaleResponse struct {
 	Max         int           `json:"max"`
 	IdleTimeout time.Duration `json:"idle_timeout,omitempty"`
 	OnDemand    bool          `json:"on_demand"`
+	// Stopped is a model taken out of service, kept, until it is started.
+	Stopped bool `json:"stopped,omitempty"`
 }

 func (a *API) scaleModel(w http.ResponseWriter, r *http.Request) {
@@ -1226,6 +1238,32 @@ func (a *API) scaleModel(w http.ResponseWriter, r *http.Request) {
 	})
 }

+// stopModel takes a model out of service, kept.
+func (a *API) stopModel(w http.ResponseWriter, r *http.Request) {
+	a.stopOrStart(w, r, a.models.Stop)
+}
+
+// startModel runs a stopped model again.
+func (a *API) startModel(w http.ResponseWriter, r *http.Request) {
+	a.stopOrStart(w, r, a.models.Start)
+}
+
+func (a *API) stopOrStart(w http.ResponseWriter, r *http.Request, f func(context.Context, string, string) (catalog.Model, error)) {
+	if a.models == nil {
+		writeError(w, http.StatusNotImplemented, "this control plane has no store")
+		return
+	}
+	m, err := f(r.Context(), r.PathValue("name"), actorOf(r))
+	if err != nil {
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	}
+	writeJSON(w, http.StatusOK, ScaleResponse{
+		Model: m.Name, Min: m.Scaling.Min, Max: m.Scaling.Max,
+		IdleTimeout: m.Scaling.IdleTimeout, OnDemand: m.Scaling.OnDemand(), Stopped: m.Stopped != nil,
+	})
+}
+
 // setResources sets a model's reserved memory and cores. Placed replicas keep
 // what they were started with until they are restarted.
 func (a *API) setResources(w http.ResponseWriter, r *http.Request) {
@@ -1252,6 +1290,10 @@ func (a *API) removeModel(w http.ResponseWriter, r *http.Request) {
 		return
 	}
 	name := r.PathValue("name")
+	if r.URL.Query().Get("purge") == "true" {
+		a.purgeModel(w, r, name)
+		return
+	}
 	if err := a.models.Remove(r.Context(), name); err != nil {
 		writeError(w, http.StatusBadRequest, err.Error())
 		return
@@ -1774,6 +1816,26 @@ func (a *API) setModelContext(w http.ResponseWriter, r *http.Request) {
 	writeJSON(w, http.StatusOK, change)
 }

+// setModelGPUs spreads a served model over more cards, or fewer, and
+// starts its replicas again on it.
+func (a *API) setModelGPUs(w http.ResponseWriter, r *http.Request) {
+	if a.serve == nil {
+		writeError(w, http.StatusNotImplemented, "this control plane cannot change a model's cards")
+		return
+	}
+	var req modelserve.GPUsRequest
+	if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	}
+	change, err := a.serve.SetGPUs(r.Context(), r.PathValue("name"), req, actorOf(r))
+	if err != nil {
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	}
+	writeJSON(w, http.StatusOK, change)
+}
+
 func (a *API) listUpdates(w http.ResponseWriter, _ *http.Request) {
 	out := make([]modelserve.UpdateInfo, 0)
 	if a.serve != nil {
@@ -749,3 +749,11 @@ func TestANodesCardInTheWrongSlotIsAWarning(t *testing.T) {
 		t.Errorf("warnings %q", warned)
 	}
 }
+
+func (f *fakeModels) Stop(_ context.Context, name, _ string) (catalog.Model, error) {
+	return catalog.Model{Name: name, Stopped: &catalog.Scaling{Min: 1, Max: 1}}, f.err
+}
+
+func (f *fakeModels) Start(_ context.Context, name, _ string) (catalog.Model, error) {
+	return catalog.Model{Name: name, Scaling: catalog.Scaling{Min: 1, Max: 1}}, f.err
+}
@@ -0,0 +1,230 @@
+package adminapi
+
+import (
+	"context"
+	"log/slog"
+	"net/http"
+	"net/url"
+	"path/filepath"
+	"slices"
+	"strings"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
+)
+
+// A model removed with its files: the catalog entry, every version of it the
+// mirror holds (and so, carried by its removal, a peer mirror's), and the
+// copies the nodes fetched, once its replicas have stopped. What another
+// model still uses stays.
+
+// PurgeMirror is what a purge asks of the mirror.
+type PurgeMirror interface {
+	List(ctx context.Context) (mirror.ModelsResponse, error)
+	Remove(ctx context.Context, model, version string) ([]string, error)
+}
+
+// SetPurgeMirror gives removing a model with its files the mirror.
+func (a *API) SetPurgeMirror(m PurgeMirror) { a.purgeMirror = m }
+
+// purgeWait is how long a purge waits for the model's replicas to stop
+// before deleting the nodes' copies; after it, they are left for llmgw ctl
+// weights --collect.
+var purgeWait = 60 * time.Second
+
+// PurgeView is what a model removed with its files took with it.
+type PurgeView struct {
+	Removed string `json:"removed"`
+	// Mirror are the versions removed from the mirror, model/version.
+	Mirror []string `json:"mirror,omitempty"`
+	// Kept are versions another model still uses, left on the mirror.
+	Kept  []string        `json:"kept,omitempty"`
+	Nodes []CollectResult `json:"nodes,omitempty"`
+	Note  string          `json:"note,omitempty"`
+}
+
+// purgeModel removes a model from the catalog and its files from the mirror
+// and the nodes.
+func (a *API) purgeModel(w http.ResponseWriter, r *http.Request, name string) {
+	m, ok := a.ctl.Catalog().Resolve(name)
+	if !ok {
+		writeError(w, http.StatusNotFound, "no model "+name+" in the catalog")
+		return
+	}
+	name = m.Name
+	others := []catalog.Model{}
+	for _, o := range a.ctl.Catalog().Models() {
+		if o.Name != name {
+			others = append(others, o)
+		}
+	}
+	if err := a.models.Remove(r.Context(), name); err != nil {
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	}
+	a.presetDeclined(r.Context(), name, actorOf(r))
+	out := PurgeView{Removed: name}
+
+	sources := purgeSources(m)
+	if a.purgeMirror == nil {
+		out.Note = "this control plane has no mirror; its files there, if any, are left"
+	} else {
+		out.Mirror, out.Kept, out.Note = purgeFromMirror(r.Context(), a.purgeMirror, sources, others)
+	}
+
+	// The nodes' copies, once nothing runs on them.
+	deadline := time.Now().Add(purgeWait)
+	for a.runs(name) && time.Now().Before(deadline) {
+		select {
+		case <-r.Context().Done():
+			return
+		case <-time.After(time.Second):
+		}
+	}
+	if a.runs(name) {
+		out.Note = strings.TrimSpace(out.Note + " Its replicas had not stopped yet: its copies on the nodes are " +
+			"left, and llmgw ctl weights --collect deletes them.")
+		writeJSON(w, http.StatusOK, out)
+		return
+	}
+	out.Nodes = a.purgeFromNodes(r.Context(), sources, others)
+	writeJSON(w, http.StatusOK, out)
+}
+
+// runs says whether any replica of a model is still on a node.
+func (a *API) runs(model string) bool {
+	for _, rep := range a.ctl.Reconciler().Replicas() {
+		if rep.Model == model {
+			return true
+		}
+	}
+	return false
+}
+
+// purgeSources are the files a model fetched: its weights, its adapters,
+// and the weights its last update replaced.
+func purgeSources(m catalog.Model) []string {
+	out := m.WeightSources()
+	for _, ad := range m.Adapters {
+		out = append(out, ad.Source)
+	}
+	if m.Origin != nil && m.Origin.Previous != nil && m.Origin.Previous.Source != "" {
+		out = append(out, m.Origin.Previous.Source)
+	}
+	return out
+}
+
+// inUse is every file the other models fetch.
+func inUse(others []catalog.Model) []string {
+	var out []string
+	for _, o := range others {
+		out = append(out, purgeSources(o)...)
+	}
+	return out
+}
+
+// purgeFromMirror removes from the mirror every version of the model
+// directories the sources are in, but those another model uses.
+func purgeFromMirror(ctx context.Context, mc PurgeMirror, sources []string, others []catalog.Model) (removed, kept []string, note string) {
+	held, err := mc.List(ctx)
+	if err != nil {
+		return nil, nil, "the mirror could not be read (" + err.Error() + "): its files there are left"
+	}
+	dirs := map[string]bool{}
+	for _, s := range sources {
+		if model, _, ok := mirrorPlace(s); ok {
+			dirs[model] = true
+		}
+	}
+	used := map[string]bool{}
+	for _, s := range inUse(others) {
+		if model, version, ok := mirrorPlace(s); ok {
+			used[model+"/"+version] = true
+		}
+	}
+	seen := map[string]bool{}
+	for _, st := range held.Models {
+		key := st.Model + "/" + st.Version
+		if !dirs[st.Model] || seen[key] {
+			continue
+		}
+		seen[key] = true
+		if used[key] {
+			kept = append(kept, key)
+			continue
+		}
+		if _, err := mc.Remove(ctx, st.Model, st.Version); err != nil {
+			note = strings.TrimSpace(note + " " + key + " could not be removed from the mirror: " + err.Error() + ".")
+			continue
+		}
+		removed = append(removed, key)
+	}
+	slices.Sort(removed)
+	slices.Sort(kept)
+	return removed, kept, note
+}
+
+// mirrorPlace is the model directory and version a mirror URL names:
+// BASE/MODEL/VERSION/FILE.
+func mirrorPlace(source string) (model, version string, ok bool) {
+	u, err := url.Parse(source)
+	if err != nil || (u.Scheme != "http" && u.Scheme != "https") {
+		return "", "", false
+	}
+	// A mirror serves from its root: a registry's address is longer.
+	parts := strings.Split(strings.Trim(u.Path, "/"), "/")
+	if len(parts) != 3 {
+		return "", "", false
+	}
+	model, version = parts[0], parts[1]
+	if mirror.ValidName(model) != nil || mirror.ValidName(version) != nil {
+		return "", "", false
+	}
+	return model, version, true
+}
+
+// purgeFromNodes deletes the nodes' copies of the sources: the files llmgw
+// fetched for them under any mirror's address, that no other model wants.
+func (a *API) purgeFromNodes(ctx context.Context, sources []string, others []catalog.Model) []CollectResult {
+	rm := a.ctl.WeightRemover()
+	if rm == nil {
+		return nil
+	}
+	names := map[string]bool{}
+	for _, s := range sources {
+		names[filepath.Base(s)] = true
+	}
+	wanted := inUse(others)
+	var out []CollectResult
+	for _, n := range a.ctl.Inventory().Nodes() {
+		size := map[string]int64{}
+		have := make([]string, 0, len(n.Weights))
+		for _, w := range n.Weights {
+			have = append(have, w.Path)
+			size[w.Path] = w.SizeBytes
+		}
+		var paths []string
+		for _, p := range weights.Orphans(have, wanted) {
+			// The cache name is a hash of the address, then the file's own
+			// name: the address may be either mirror's, the name is the same.
+			if _, file, ok := strings.Cut(filepath.Base(p), "-"); ok && names[file] {
+				paths = append(paths, p)
+			}
+		}
+		if len(paths) == 0 {
+			continue
+		}
+		res := CollectResult{Node: n.ID, Files: len(paths)}
+		for _, p := range paths {
+			res.Bytes += size[p]
+		}
+		if err := rm.RemoveWeights(ctx, n.ID, paths); err != nil {
+			res.Error, res.Files, res.Bytes = err.Error(), 0, 0
+			slog.Warn("a purged model's copies could not be deleted on a node", "node", n.ID, "err", err)
+		}
+		out = append(out, res)
+	}
+	return out
+}
@@ -0,0 +1,68 @@
+package adminapi
+
+import (
+	"context"
+	"slices"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
+)
+
+type purgeMirrorFake struct {
+	held    []mirror.Status
+	removed []string
+}
+
+func (f *purgeMirrorFake) List(context.Context) (mirror.ModelsResponse, error) {
+	return mirror.ModelsResponse{Base: "http://m:8081", Models: f.held}, nil
+}
+
+func (f *purgeMirrorFake) Remove(_ context.Context, model, version string) ([]string, error) {
+	f.removed = append(f.removed, model+"/"+version)
+	return nil, nil
+}
+
+func TestAPurgeRemovesEveryVersionOfTheModelButWhatAnotherUses(t *testing.T) {
+	f := &purgeMirrorFake{held: []mirror.Status{
+		{Model: "qwen", Version: "v2", File: "a.gguf"},
+		{Model: "qwen", Version: "v2", File: "b.gguf"},
+		{Model: "qwen", Version: "v1", File: "a.gguf"},
+		{Model: "qwen", Version: "v3", File: "a.gguf"},
+		{Model: "other", Version: "v1", File: "o.gguf"},
+		{Model: "lora", Version: "t1", File: "l.gguf"},
+	}}
+	m := catalog.Model{Name: "qwen", Source: "http://m:8081/qwen/v2/a.gguf",
+		Adapters: []catalog.Adapter{{Name: "tone", Source: "http://m:8081/lora/t1/l.gguf"}}}
+	// The canary runs v3: it stays.
+	others := []catalog.Model{{Name: "qwen-canary", Source: "http://other-mirror:8081/qwen/v3/a.gguf"},
+		{Name: "other", Source: "http://m:8081/other/v1/o.gguf"}}
+	removed, kept, note := purgeFromMirror(t.Context(), f, purgeSources(m), others)
+	if note != "" {
+		t.Fatal(note)
+	}
+	if !slices.Equal(removed, []string{"lora/t1", "qwen/v1", "qwen/v2"}) || !slices.Equal(kept, []string{"qwen/v3"}) {
+		t.Errorf("removed %v, kept %v", removed, kept)
+	}
+	if slices.Contains(f.removed, "other/v1") || len(f.removed) != 3 {
+		t.Errorf("asked to remove %v", f.removed)
+	}
+}
+
+func TestOnlyAMirrorsAddressNamesAPlaceOnIt(t *testing.T) {
+	for src, want := range map[string]string{
+		"http://10.0.0.146:8081/qwen/abc/q4.gguf":         "qwen/abc",
+		"https://huggingface.co/o/r/resolve/main/q4.gguf": "",
+		"/srv/models/q4.gguf":                             "",
+		"Qwen/Qwen2.5-7B":                                 "",
+	} {
+		model, version, ok := mirrorPlace(src)
+		got := ""
+		if ok {
+			got = model + "/" + version
+		}
+		if got != want {
+			t.Errorf("%s: %q, want %q", src, got, want)
+		}
+	}
+}
@@ -155,8 +155,16 @@ var Routes = []Route{
 		Description: "With the defaults the catalog filled in. By its name only, not an alias.",
 		Response:    catalog.Model{}},
 	{Method: "DELETE", Path: "/api/v1/models/{name}", Summary: "Remove a model added at runtime",
-		Description: "A model declared in the cluster's settings is refused: it is changed there.",
-		Response:    Removed{}},
+		Description: "A model declared in the cluster's settings is refused: it is changed there. With purge, its " +
+			"files go too: every version of it the mirror holds (a peer mirror follows), and the nodes' copies once its " +
+			"replicas have stopped, waiting up to a minute for them; what another model uses stays. The answer is then a PurgeView.",
+		Query:    []Param{{Name: "purge", Type: "boolean", Description: "Delete its files from the mirror and the nodes as well."}},
+		Response: Removed{}},
+	{Method: "POST", Path: "/api/v1/models/{name}/stop", Summary: "Take a model out of service and keep it",
+		Description: "None of its replicas run, and requests are told an administrator stopped it; what it was scaled to is kept for start.",
+		Response:    ScaleResponse{}},
+	{Method: "POST", Path: "/api/v1/models/{name}/start", Summary: "Run a stopped model again, scaled as it was",
+		Response: ScaleResponse{}},
 	{Method: "POST", Path: "/api/v1/models/{name}/scale", Summary: "Set how many replicas a model runs",
 		Request: ScaleRequest{}, Response: ScaleResponse{}},
 	{Method: "POST", Path: "/api/v1/models/{name}/resources", Summary: "Set what a model's profiles reserve of a node",
@@ -215,6 +223,9 @@ var Routes = []Route{
 		Request:     modelserve.Change{}, OptionalBody: true, Response: modelserve.Job{}, Status: http.StatusAccepted},
 	{Method: "POST", Path: "/api/v1/models/{name}/context", Summary: "Change a served model's context: the catalog, its engines' arguments and memory, then its replicas restarted",
 		Request: modelserve.ContextRequest{}, Response: modelserve.ContextChange{}},
+	{Method: "POST", Path: "/api/v1/models/{name}/gpus", Summary: "Spread a served model over more cards or fewer: its profile, checked against its KV heads, its cards' memory and the nodes, then its replicas restarted",
+		Description: "Tensor parallel for vLLM, SGLang and TensorRT-LLM, a layer split for llama.cpp, on one node. With no room beside the running replicas, in_place stops each first.",
+		Request:     modelserve.GPUsRequest{}, Response: modelserve.GPUsChange{}},
 	{Method: "POST", Path: "/api/v1/models/{name}/rollback", Summary: "Go back to the weights the last update replaced",
 		Response: Result{}},
 	{Method: "POST", Path: "/api/v1/models/{name}/origin", Summary: "Record the repository a model's weights came from",
@@ -332,6 +332,10 @@ type Model struct {
 	AllowPickle bool      `json:"allow_pickle,omitempty" koanf:"allow_pickle"`
 	Profiles    []Profile `json:"profiles,omitempty" koanf:"profiles"`
 	Scaling     Scaling   `json:"scaling,omitempty" koanf:"scaling"`
+	// Stopped is the scaling a model had when an administrator stopped it:
+	// kept in the catalog with none running (Scaling 0/0) until started
+	// again with this. Nil for a model not stopped.
+	Stopped *Scaling `json:"stopped,omitempty" koanf:"stopped"`
 	// Class is how much this model's capacity is worth against another's,
 	// when there is not enough for both: critical, standard or opportunistic.
 	// Empty is standard, or opportunistic for a model loaded on demand. See
@@ -49,11 +49,13 @@ type ContextChange struct {
 }

 // shape is what sizes a context: a token's KV cache, the context the model
-// was trained for, and its weights.
+// was trained for, and its weights; and its KV heads, which decide how many
+// cards tensor parallel can spread it over (0 unknown).
 type shape struct {
 	kvPerToken int64
 	trained    int
 	weights    int64
+	kvHeads    int
 }

 // SetContext changes a served model's context.
@@ -135,12 +137,14 @@ func (s *Server) shapeOf(ctx context.Context, m catalog.Model) (shape, string) {
 			return sh, "its header could not be read (" + err.Error() + "), so the memory it needs is not checked"
 		}
 		sh.kvPerToken, sh.trained = h.KVBytesPerToken(), int(h.ContextLength) //nolint:gosec // a header's context
+		sh.kvHeads = int(h.HeadCountKV)                                       //nolint:gosec // a header's head count
 	case m.Origin != nil && m.Origin.Repo != "":
 		mm, err := modelplan.FetchModel(ctx, m.Origin.Repo)
 		if err != nil {
 			return sh, "its config could not be read (" + err.Error() + "), so the memory it needs is not checked"
 		}
 		sh.kvPerToken, sh.trained = mm.KVBytesPerToken(modelplan.BF16), mm.MaxContext
+		sh.kvHeads = mm.KVHeads
 	default:
 		return sh, "where it came from is not known, so the memory it needs is not checked"
 	}
@@ -0,0 +1,285 @@
+package modelserve
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"slices"
+	"strings"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/modelplan"
+)
+
+// A served model moved to more cards, or fewer: how many each replica
+// spreads it over, on one node. The profile changes — tensor parallel for
+// vLLM, SGLang and TensorRT-LLM, llama.cpp's layer split for llama.cpp —
+// checked against the model's KV heads, what each card would hold, and the
+// nodes there are; then its replicas start again on it.
+
+// GPUsRequest is how many cards a replica is to use.
+type GPUsRequest struct {
+	GPUs int `json:"gpus"`
+	// Profile is the one to change, when the model has more than one
+	// profile on cards.
+	Profile string `json:"profile,omitempty"`
+	// InPlace stops each replica before its replacement starts: for when no
+	// node has the cards free beside it. It does not answer while the new
+	// one loads.
+	InPlace bool `json:"in_place,omitempty"`
+}
+
+// GPUsChange is what was changed, and the replicas started again on it.
+type GPUsChange struct {
+	Model   string   `json:"model"`
+	Profile string   `json:"profile"`
+	Engine  string   `json:"engine"`
+	Class   string   `json:"gpu_class"`
+	Before  int      `json:"before"`
+	After   int      `json:"after"`
+	Args    []string `json:"args"`
+	// PerCard is what each card holds after, when it could be worked out.
+	PerCard   string `json:"per_card,omitempty"`
+	Restarted int    `json:"restarted"`
+	InPlace   bool   `json:"in_place,omitempty"`
+	// Note is what could not be checked, said once.
+	Note string `json:"note,omitempty"`
+}
+
+// tpFlags are each engine's own spelling of a tensor-parallel width, which
+// the agent sets from the profile: a profile's own would contradict it.
+var tpFlags = map[string][]string{
+	"vllm":   {"--tensor-parallel-size", "-tp"},
+	"sglang": {"--tp-size", "--tensor-parallel-size", "--tp"},
+	"trtllm": {"--tp_size"},
+	// llama.cpp's proportions were for the cards it had.
+	"llamacpp": {"--tensor-split", "-ts"},
+}
+
+// SetGPUs changes how many cards a served model's replicas use.
+func (s *Server) SetGPUs(ctx context.Context, name string, req GPUsRequest, by string) (GPUsChange, error) {
+	if s.deps.Current == nil || s.deps.Models == nil {
+		return GPUsChange{}, errors.New("this control plane cannot change the catalog")
+	}
+	var m catalog.Model
+	found := false
+	for _, c := range s.deps.Current() {
+		if c.Name == name {
+			m, found = c, true
+		}
+	}
+	switch {
+	case !found:
+		return GPUsChange{}, fmt.Errorf("no model %q in the catalog", name)
+	case s.deps.Models.Declared(name):
+		return GPUsChange{}, fmt.Errorf("model %q is declared in the cluster's settings: change its profile there (llmgw ctl config edit)", name)
+	case req.GPUs < 1 || req.GPUs > 64:
+		return GPUsChange{}, errors.New("a replica uses 1 to 64 cards")
+	}
+	i, err := gpuProfile(m, req.Profile)
+	if err != nil {
+		return GPUsChange{}, err
+	}
+	p := m.Profiles[i]
+	before := max(p.GPUsPerNode, p.TP, 1) * max(p.Nodes, p.PP, 1)
+	out := GPUsChange{Model: name, Profile: p.Name, Engine: p.Engine, Class: p.GPUClass, Before: before, After: req.GPUs}
+	if before == req.GPUs {
+		return out, fmt.Errorf("%s's %s profile already uses %d card(s)", name, p.Name, before)
+	}
+
+	sh, note := s.shapeOf(ctx, m)
+	next, err := respread(p, req.GPUs, sh)
+	if err != nil {
+		return out, err
+	}
+	out.Args = next.Args
+	switch kv := int64(contextOf(m, p)) * sh.kvPerToken; {
+	case sh.weights == 0 || sh.kvPerToken == 0 || contextOf(m, p) == 0:
+		if note == "" {
+			note = "its context is not known, so what each card holds is not checked"
+		}
+	default:
+		need := modelplan.CardNeed(p.Engine, sh.weights, kv, req.GPUs)
+		out.PerCard = fmt.Sprintf("%.1f GB a card of %d", need, req.GPUs)
+		if budget := modelplan.CardBudget(p.GPUClass); budget > 0 {
+			out.PerCard += fmt.Sprintf(", of %.0f usable", budget)
+			if need > budget {
+				return out, fmt.Errorf("on %d %s cards each would hold %.1f GB, more than the %.0f GB they hold: "+
+					"more cards, or a shorter context (llmgw ctl model context)", req.GPUs, p.GPUClass, need, budget)
+			}
+		}
+	}
+	out.Note = note
+
+	ids := s.replicasOf(name)
+	inPlace, err := s.roomFor(p.GPUClass, req.GPUs, name, len(ids) > 0, req.InPlace)
+	if err != nil {
+		return out, err
+	}
+	profiles := slices.Clone(m.Profiles)
+	profiles[i] = next
+	if _, err := s.deps.Models.SetProfiles(ctx, name, profiles, by); err != nil {
+		return out, err
+	}
+	if len(ids) == 0 {
+		return out, nil
+	}
+	restart := s.deps.Restart
+	if inPlace {
+		restart, out.InPlace = s.deps.RestartInPlace, true
+	}
+	if restart == nil {
+		return out, errors.New("the profile is changed, and this control plane cannot start its replicas again: llmgw ctl restart")
+	}
+	if err := restart(ids...); err != nil {
+		return out, fmt.Errorf("the profile is changed, and its replicas could not be started again on it: %w", err)
+	}
+	out.Restarted = len(ids)
+	return out, nil
+}
+
+// gpuProfile is the model's profile on cards to change: the one named, or
+// its only one.
+func gpuProfile(m catalog.Model, named string) (int, error) {
+	var onCards []int
+	for i, p := range m.Profiles {
+		if named != "" && p.Name == named {
+			if p.CPUOnly() {
+				return 0, fmt.Errorf("%s's %s profile runs on CPUs; give it a gpu_class first", m.Name, p.Name)
+			}
+			return i, nil
+		}
+		if !p.CPUOnly() {
+			onCards = append(onCards, i)
+		}
+	}
+	switch {
+	case named != "":
+		return 0, fmt.Errorf("%s has no profile %q", m.Name, named)
+	case len(onCards) == 0:
+		return 0, fmt.Errorf("%s runs on CPUs only; a profile on cards needs a gpu_class", m.Name)
+	case len(onCards) > 1:
+		var names []string
+		for _, i := range onCards {
+			names = append(names, m.Profiles[i].Name)
+		}
+		return 0, fmt.Errorf("%s has several profiles on cards (%s): name one", m.Name, strings.Join(names, ", "))
+	}
+	return onCards[0], nil
+}
+
+// respread is a profile spread over n cards of one node.
+func respread(p catalog.Profile, n int, sh shape) (catalog.Profile, error) {
+	for _, f := range tpFlags[p.Engine] {
+		p.Args = dropArg(p.Args, f)
+	}
+	switch p.Engine {
+	case "vllm", "sglang", "trtllm":
+		if !modelplan.SplitsHeads(sh.kvHeads, n) {
+			return p, fmt.Errorf("the model has %d KV heads, which tensor parallel cannot spread over %d cards: "+
+				"a number that divides them, or that they divide", sh.kvHeads, n)
+		}
+		p.TP, p.GPUsPerNode = n, n
+	case "llamacpp":
+		// llama.cpp has no tensor parallel: it splits the layers over a
+		// node's cards, in proportion to their free memory.
+		p.TP, p.GPUsPerNode = 1, n
+	default:
+		return p, fmt.Errorf("a %s replica runs on one card", p.Engine)
+	}
+	// One node: a replica spanning nodes is laid out by hand, with a fabric.
+	p.Nodes, p.PP = 1, 0
+	return p, nil
+}
+
+// contextOf is the tokens a profile's KV cache holds: llama.cpp's pool, or
+// the engine's limit, or the model's context.
+func contextOf(m catalog.Model, p catalog.Profile) int {
+	switch p.Engine {
+	case "llamacpp":
+		if n := argInt(p.Args, "--ctx-size"); n > 0 {
+			return n
+		}
+	case "vllm":
+		if n := argInt(p.Args, "--max-model-len"); n > 0 {
+			return n
+		}
+	case "sglang":
+		if n := argInt(p.Args, "--context-length"); n > 0 {
+			return n
+		}
+	}
+	return m.ContextLength
+}
+
+func (s *Server) replicasOf(model string) []string {
+	var ids []string
+	if s.deps.Fleet != nil {
+		for _, r := range s.deps.Fleet.Replicas() {
+			if r.Model == model {
+				ids = append(ids, r.ID)
+			}
+		}
+	}
+	return ids
+}
+
+// roomFor says whether the new replicas fit: a node with n cards of the
+// class at all, and free beside the running ones — or, asked to start them
+// in place, once the model's own are stopped. True is in place.
+func (s *Server) roomFor(class string, n int, model string, running, inPlace bool) (bool, error) {
+	if s.deps.Pool == nil {
+		return inPlace, nil
+	}
+	own := map[string]int{}
+	if s.deps.Fleet != nil {
+		for _, r := range s.deps.Fleet.Replicas() {
+			if r.Model == model {
+				for _, sh := range r.Shards {
+					own[sh.NodeID] += len(sh.GPUIndices)
+				}
+			}
+		}
+	}
+	most, freeMost, besideOK, inPlaceOK := 0, 0, false, false
+	for _, node := range s.deps.Pool() {
+		for _, k := range modelplan.Kinds(node) {
+			if k.Class != class {
+				continue
+			}
+			most, freeMost = max(most, k.GPUs), max(freeMost, k.FreeGPUs)
+			besideOK = besideOK || k.FreeGPUs >= n
+			inPlaceOK = inPlaceOK || k.FreeGPUs+own[node.ID] >= n
+		}
+	}
+	switch {
+	case most == 0:
+		return false, fmt.Errorf("no ready node has %s cards", class)
+	case most < n:
+		return false, fmt.Errorf("no node has %d %s cards; the most one has is %d", n, class, most)
+	case !running || besideOK:
+		return false, nil
+	case !inPlaceOK:
+		return false, fmt.Errorf("no node has %d %s cards free, even with this model's own stopped (%d at most free now): "+
+			"other models hold them", n, class, freeMost)
+	case !inPlace:
+		return false, fmt.Errorf("no node has %d %s cards free beside the running replica (%d at most); "+
+			"--in-place stops each replica first, and the model does not answer while the new one loads", n, class, freeMost)
+	}
+	return true, nil
+}
+
+// dropArg is arguments without a flag and its value.
+func dropArg(args []string, flag string) []string {
+	out := make([]string, 0, len(args))
+	for i := 0; i < len(args); i++ {
+		switch {
+		case args[i] == flag:
+			i++ // its value
+		case strings.HasPrefix(args[i], flag+"="):
+		default:
+			out = append(out, args[i])
+		}
+	}
+	return out
+}
@@ -0,0 +1,151 @@
+package modelserve
+
+import (
+	"context"
+	"slices"
+	"strings"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/modelplan"
+)
+
+// gpuSetup is the update setup with qwen on one L40S card of llama.cpp, and
+// a pool of one node of four, of which free are free.
+func gpuSetup(t *testing.T, free int) (*Server, *updModels, *restartFleet, *[]string) {
+	t.Helper()
+	s, _, models, _, fleet := updSetup(t)
+	m := models.models["qwen"]
+	m.ContextLength = 8192
+	m.Origin = &catalog.Origin{Repo: "o/r", Files: []catalog.OriginFile{{Path: "q4.gguf", Bytes: 20 << 30}}}
+	m.Profiles = []catalog.Profile{
+		{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 8},
+		{Name: "gpu", Engine: "llamacpp", GPUClass: "l40s-48g", TP: 1, GPUsPerNode: 1,
+			Args: []string{"--ctx-size", "8192", "--tensor-split", "1"}},
+	}
+	models.models["qwen"] = m
+	s.deps.Pool = func() []modelplan.PoolNode {
+		return []modelplan.PoolNode{{ID: "g1", Class: "l40s-48g", GPUMemGB: 48, GPUs: 4, FreeGPUs: free}}
+	}
+	var inPlace []string
+	s.deps.RestartInPlace = func(ids ...string) error { inPlace = append(inPlace, ids...); return nil }
+	return s, models, fleet, &inPlace
+}
+
+func TestAModelIsSpreadOverMoreCardsAndItsReplicasStartedAgainBesideTheOld(t *testing.T) {
+	s, models, fleet, inPlace := gpuSetup(t, 3)
+	got, err := s.SetGPUs(context.Background(), "qwen", GPUsRequest{GPUs: 2}, "admin")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if got.Profile != "gpu" || got.Before != 1 || got.After != 2 || got.Restarted != 2 || got.InPlace {
+		t.Fatalf("%+v", got)
+	}
+	p := models.models["qwen"].Profiles[1]
+	// llama.cpp splits layers over a node's cards: gpus_per_node, not tp,
+	// and the old card's proportions are gone.
+	if p.GPUsPerNode != 2 || p.TP != 1 || p.Nodes != 1 || !slices.Equal(p.Args, []string{"--ctx-size", "8192"}) {
+		t.Errorf("profile %+v", p)
+	}
+	if !strings.Contains(got.PerCard, "a card of 2") {
+		t.Errorf("per card %q", got.PerCard)
+	}
+	if len(*inPlace) != 0 {
+		t.Errorf("stopped first though there was room: %v", *inPlace)
+	}
+	for _, r := range fleet.Replicas() {
+		if !strings.Contains(r.ID, "-new") {
+			t.Errorf("%s was not started again", r.ID)
+		}
+	}
+	if models.models["qwen"].Profiles[0].GPUsPerNode != 0 {
+		t.Error("the CPU profile was changed too")
+	}
+}
+
+func TestWithNoRoomBesideIt_InPlaceIsAskedFor(t *testing.T) {
+	// One card free and the model's replicas on the others: two do not fit
+	// beside them.
+	s, _, fleet, inPlace := gpuSetup(t, 1)
+	fleet.reps[0].Shards = []reconciler.Shard{{NodeID: "g1", GPUIndices: []int{1}}}
+	fleet.reps[1].Shards = []reconciler.Shard{{NodeID: "g1", GPUIndices: []int{2}}}
+	_, err := s.SetGPUs(context.Background(), "qwen", GPUsRequest{GPUs: 2}, "admin")
+	if err == nil || !strings.Contains(err.Error(), "--in-place") {
+		t.Fatalf("not asked for in place: %v", err)
+	}
+	got, err := s.SetGPUs(context.Background(), "qwen", GPUsRequest{GPUs: 2, InPlace: true}, "admin")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if !got.InPlace || len(*inPlace) != 2 {
+		t.Errorf("%+v, stopped first: %v", got, *inPlace)
+	}
+}
+
+func TestMoreCardsThanANodeHasOrOthersHoldAreRefused(t *testing.T) {
+	s, models, _, _ := gpuSetup(t, 4)
+	if _, err := s.SetGPUs(context.Background(), "qwen", GPUsRequest{GPUs: 8}, "admin"); err == nil ||
+		!strings.Contains(err.Error(), "the most one has is 4") {
+		t.Errorf("eight cards on a node of four: %v", err)
+	}
+	s, _, _, _ = gpuSetup(t, 0)
+	if _, err := s.SetGPUs(context.Background(), "qwen", GPUsRequest{GPUs: 2, InPlace: true}, "admin"); err == nil ||
+		!strings.Contains(err.Error(), "other models hold them") {
+		t.Errorf("cards other models hold: %v", err)
+	}
+	if models.models["qwen"].Profiles[1].GPUsPerNode != 1 {
+		t.Error("a refused change was kept")
+	}
+}
+
+func TestFewerCardsThanTheModelNeedsAreRefusedWithTheFigures(t *testing.T) {
+	s, models, _, _ := gpuSetup(t, 3)
+	m := models.models["qwen"]
+	m.Profiles[1].GPUsPerNode, m.Origin.Files[0].Bytes = 4, 120<<30
+	models.models["qwen"] = m
+	_, err := s.SetGPUs(context.Background(), "qwen", GPUsRequest{GPUs: 2}, "admin")
+	if err == nil || !strings.Contains(err.Error(), "GB") {
+		t.Errorf("120 GB on two 48 GB cards: %v", err)
+	}
+}
+
+func TestTensorParallelFollowsTheKVHeads(t *testing.T) {
+	p := catalog.Profile{Name: "gpu", Engine: "vllm", GPUClass: "h100-80g", TP: 2, GPUsPerNode: 2,
+		Args: []string{"--tensor-parallel-size", "2", "--max-model-len", "32768"}}
+	if _, err := respread(p, 3, shape{kvHeads: 8}); err == nil || !strings.Contains(err.Error(), "8 KV heads") {
+		t.Errorf("three ways for eight heads: %v", err)
+	}
+	next, err := respread(p, 4, shape{kvHeads: 8})
+	if err != nil {
+		t.Fatal(err)
+	}
+	// The agent sets the width from tp; the profile's own spelling went.
+	if next.TP != 4 || next.GPUsPerNode != 4 || !slices.Equal(next.Args, []string{"--max-model-len", "32768"}) {
+		t.Errorf("%+v", next)
+	}
+	for _, eng := range []string{"comfyui", "strata"} {
+		if _, err := respread(catalog.Profile{Name: "x", Engine: eng}, 2, shape{}); err == nil {
+			t.Errorf("%s spread over two cards", eng)
+		}
+	}
+}
+
+func TestAModelOnCPUsOrWithSeveralCardProfilesSaysWhy(t *testing.T) {
+	s, models, _, _ := gpuSetup(t, 3)
+	m := models.models["qwen"]
+	m.Profiles = append(m.Profiles, catalog.Profile{Name: "gpu2", Engine: "vllm", GPUClass: "l40s-48g", TP: 1})
+	models.models["qwen"] = m
+	if _, err := s.SetGPUs(context.Background(), "qwen", GPUsRequest{GPUs: 2}, "admin"); err == nil ||
+		!strings.Contains(err.Error(), "name one") {
+		t.Errorf("two profiles on cards and none named: %v", err)
+	}
+	if _, err := s.SetGPUs(context.Background(), "qwen", GPUsRequest{GPUs: 2, Profile: "cpu"}, "admin"); err == nil ||
+		!strings.Contains(err.Error(), "gpu_class") {
+		t.Errorf("a CPU profile: %v", err)
+	}
+	if _, err := s.SetGPUs(context.Background(), "qwen", GPUsRequest{GPUs: 1, Profile: "gpu"}, "admin"); err == nil ||
+		!strings.Contains(err.Error(), "already") {
+		t.Errorf("the same count: %v", err)
+	}
+}
@@ -82,6 +82,9 @@ type Deps struct {
 	Current func() []catalog.Model
 	Models  Models
 	Restart func(ids ...string) error
+	// RestartInPlace stops each replica before its replacement starts, for
+	// when there is no room beside it.
+	RestartInPlace func(ids ...string) error

 	// Pool is the fleet's ready nodes as a plan sees them: what decides
 	// which file suits the hardware, and how it is spread.
@@ -35,6 +35,7 @@ type Models interface {
 	SetSource(ctx context.Context, name, source string, origin *catalog.Origin, by string) (catalog.Model, error)
 	SetOrigin(ctx context.Context, name string, origin *catalog.Origin) error
 	SetContext(ctx context.Context, name string, contextLength, maxOutput int, profiles []catalog.Profile, by string) (catalog.Model, error)
+	SetProfiles(ctx context.Context, name string, profiles []catalog.Profile, by string) (catalog.Model, error)
 	// For a canary: the candidate added beside the model and taken out
 	// again, and the share of requests it is given.
 	Add(ctx context.Context, m catalog.Model, by string) error
@@ -112,6 +112,15 @@ func (c *updModels) SetContext(_ context.Context, n string, ctxLen, maxOut int,
 	c.models[n] = m
 	return m, nil
 }
+
+func (c *updModels) SetProfiles(_ context.Context, n string, ps []catalog.Profile, _ string) (catalog.Model, error) {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	m := c.models[n]
+	m.Profiles = ps
+	c.models[n] = m
+	return m, nil
+}
 func (c *updModels) all() []catalog.Model {
 	c.mu.Lock()
 	defer c.mu.Unlock()
@@ -228,6 +228,8 @@ func (s *Service) Remove(ctx context.Context, name string) error {
 func (s *Service) Scale(ctx context.Context, name string, sc catalog.Scaling, by string) (catalog.Model, error) {
 	final, err := s.update(ctx, name, by, "change its scaling", func(m *catalog.Model) error {
 		m.Scaling = sc
+		// A number set is what it runs now: a stopped model is started by it.
+		m.Stopped = nil
 		return nil
 	})
 	if err == nil {
@@ -237,6 +239,39 @@ func (s *Service) Scale(ctx context.Context, name string, sc catalog.Scaling, by
 	return final, err
 }

+// Stop takes a model out of service and keeps it: none of its replicas run,
+// and what it was scaled to is kept for Start.
+func (s *Service) Stop(ctx context.Context, name, by string) (catalog.Model, error) {
+	final, err := s.update(ctx, name, by, "stop it", func(m *catalog.Model) error {
+		if m.Stopped != nil {
+			return fmt.Errorf("model %q is already stopped; llmgw ctl model start %s runs it again", name, name)
+		}
+		was := m.Scaling
+		m.Stopped = &was
+		m.Scaling = catalog.Scaling{IdleTimeout: was.IdleTimeout}
+		return nil
+	})
+	if err == nil {
+		s.log.Info("model stopped", "model", name, "by", by)
+	}
+	return final, err
+}
+
+// Start runs a stopped model again, scaled as it was when it was stopped.
+func (s *Service) Start(ctx context.Context, name, by string) (catalog.Model, error) {
+	final, err := s.update(ctx, name, by, "start it", func(m *catalog.Model) error {
+		if m.Stopped == nil {
+			return fmt.Errorf("model %q is not stopped; llmgw ctl scale changes how many replicas it runs", name)
+		}
+		m.Scaling, m.Stopped = *m.Stopped, nil
+		return nil
+	})
+	if err == nil {
+		s.log.Info("model started", "model", name, "min", final.Scaling.Min, "max", final.Scaling.Max, "by", by)
+	}
+	return final, err
+}
+
 // Resources sets what a model's profile reserves of a node: host memory, and
 // the CPU cores a replica without a GPU computes on. An empty profile name
 // sets every profile.
@@ -400,3 +400,41 @@ func TestAddNewLeavesAModelAddedMeanwhile(t *testing.T) {
 		t.Errorf("a new model: %v", err)
 	}
 }
+
+func TestAStoppedModelRunsNoneAndStartsAsItWas(t *testing.T) {
+	s, _, rec := newService(t)
+	if err := s.Add(t.Context(), model("added"), "ilker"); err != nil {
+		t.Fatal(err)
+	}
+	if _, err := s.Scale(t.Context(), "added", catalog.Scaling{Min: 2, Max: 4, IdleTimeout: 20 * time.Minute}, "ilker"); err != nil {
+		t.Fatal(err)
+	}
+	got, err := s.Stop(t.Context(), "added", "ilker")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if got.Scaling.Min != 0 || got.Scaling.Max != 0 || got.Stopped == nil || got.Stopped.Min != 2 {
+		t.Fatalf("stopped: %+v %+v", got.Scaling, got.Stopped)
+	}
+	if m, _ := rec.last.Resolve("added"); m.Scaling.Max != 0 || m.Scaling.OnDemand() {
+		t.Errorf("the catalog in force runs it: %+v", m.Scaling)
+	}
+	if _, err := s.Stop(t.Context(), "added", "ilker"); err == nil {
+		t.Error("stopped twice")
+	}
+	got, err = s.Start(t.Context(), "added", "ilker")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if got.Scaling.Min != 2 || got.Scaling.Max != 4 || got.Scaling.IdleTimeout != 20*time.Minute || got.Stopped != nil {
+		t.Errorf("started: %+v %+v", got.Scaling, got.Stopped)
+	}
+	if _, err := s.Start(t.Context(), "added", "ilker"); err == nil {
+		t.Error("started what was not stopped")
+	}
+	// A scale while stopped starts it at that number.
+	_, _ = s.Stop(t.Context(), "added", "ilker")
+	if got, _ := s.Scale(t.Context(), "added", catalog.Scaling{Min: 1}, "ilker"); got.Stopped != nil {
+		t.Error("still stopped after a scale")
+	}
+}
@@ -244,6 +244,7 @@ var Reference = []Block{
 		{Key: "allow_pickle", Type: "bool", Doc: "AllowPickle lets the nodes fetch this model's weights, or its\nadapters', in a pickle format (.bin, .pt, .pth, .ckpt), which run code\nas they are loaded. Off, such a source is refused when the model is\nadded and by the node before it fetches anything. For a model that has\nno safetensors or GGUF form and whose files were checked.", Nested: ""},
 		{Key: "profiles", Type: "[]Profile", Doc: "", Nested: ""},
 		{Key: "scaling", Type: "Scaling", Doc: "", Nested: "catalog.Scaling"},
+		{Key: "stopped", Type: "*Scaling", Doc: "Stopped is the scaling a model had when an administrator stopped it:\nkept in the catalog with none running (Scaling 0/0) until started\nagain with this. Nil for a model not stopped.", Nested: "catalog.Scaling"},
 		{Key: "class", Type: "string", Doc: "Class is how much this model's capacity is worth against another's,\nwhen there is not enough for both: critical, standard or opportunistic.\nEmpty is standard, or opportunistic for a model loaded on demand. See\nEffectiveClass.", Nested: ""},
 		{Key: "owner", Type: "string", Doc: "Owner is the part of the tenancy tree whose capacity quota this model\ncounts against, written org:ID, team:ID or project:ID. Empty is shared:\nheld by the cluster, counted against nobody.", Nested: ""},
 		{Key: "external", Type: "*External", Doc: "External is a model served by someone else — an OpenAI-compatible API\noutside the cluster — rather than by replicas placed here. Such a model\nhas no profiles and nothing is placed for it; the router sends its\nrequests to the provider. Most useful named in another model's\nfallbacks: used only when the cluster has no capacity left. Its\nprompts leave the cluster, so which keys may name it is worth an\nallowlist.", Nested: "catalog.External"},
@@ -210,7 +210,7 @@ func (m *Manager) Status(model, version string) (Status, bool) {

 	m.mu.Lock()
 	for _, st := range m.running {
-		if st.Model == model && st.Version == version {
+		if st.Model == model && st.Version == version && !m.goneLocked(*st) {
 			out, found = merge(out, *st, found)
 		}
 	}
@@ -327,7 +327,9 @@ func (m *Manager) All() ([]Status, error) {
 	}

 	for key, st := range known {
-		if _, onDisk := seen[key]; onDisk {
+		if _, onDisk := seen[key]; onDisk || st.State == Present {
+			// Present and not on the disk: removed since, by hand or by
+			// llmgw mirror remove. Not reported as here, which it is not.
 			continue
 		}
 		out = append(out, st)
@@ -335,6 +337,16 @@ func (m *Manager) All() ([]Status, error) {
 	return out, nil
 }

+// goneLocked says whether a finished download's file has been removed since.
+// Called with m.mu held.
+func (m *Manager) goneLocked(st Status) bool {
+	if st.State != Present {
+		return false
+	}
+	_, err := os.Stat(filepath.Join(m.root, st.Model, st.Version, st.File))
+	return errors.Is(err, os.ErrNotExist)
+}
+
 // keptFinished is how long a finished download's status is remembered.
 const keptFinished = 7 * 24 * time.Hour

@@ -3,6 +3,7 @@ package mirror
 import (
 	"context"
 	"log/slog"
+	"path/filepath"
 	"strings"
 	"time"
 )
@@ -24,6 +25,13 @@ type Peers struct {

 // Run reads the peers until ctx ends.
 func (p *Peers) Run(ctx context.Context) {
+	// Files fetched before checksums were kept have none, and a peer copies
+	// only what it can check: they are given theirs, beside the sync.
+	go func() {
+		if n := p.M.FillChecksums(ctx); n > 0 {
+			p.Log.Info("recorded the checksums of files that had none, so peer mirrors can copy them", "files", n)
+		}
+	}()
 	t := time.NewTicker(PeerInterval)
 	defer t.Stop()
 	for {
@@ -122,3 +130,33 @@ func (p *Peers) SyncOnce(ctx context.Context, c *Client) {
 		}
 	}
 }
+
+// FillChecksums records the sha256 of every file the mirror holds without
+// one, and says how many it recorded.
+func (m *Manager) FillChecksums(ctx context.Context) int {
+	entries, err := List(m.root)
+	if err != nil {
+		return 0
+	}
+	n := 0
+	for _, e := range entries {
+		if ctx.Err() != nil {
+			break
+		}
+		path := filepath.Join(m.root, e.Model, e.Version, e.File)
+		if readChecksum(path) != "" || m.isRunning(e.Model, e.Version, e.File) {
+			continue
+		}
+		sum, err := sha256File(path)
+		if err != nil {
+			m.log.Warn("could not read a file's checksum", "file", path, "err", err)
+			continue
+		}
+		if err := writeChecksum(path, sum); err != nil {
+			m.log.Warn("could not record a file's checksum", "file", path, "err", err)
+			continue
+		}
+		n++
+	}
+	return n
+}
@@ -172,3 +172,74 @@ func TestAModelsAddressIsMovedToAMirrorThatAnswers(t *testing.T) {
 		t.Errorf("none up: %s", got)
 	}
 }
+
+// A file fetched before checksums were kept is given one, and is then copied.
+func TestAFileWithoutAChecksumIsGivenOneAndThenCopied(t *testing.T) {
+	a, b := newTestMirror(t), newTestMirror(t)
+	path := filepath.Join(a.root, "old", "v1", "m.gguf")
+	if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
+		t.Fatal(err)
+	}
+	if err := os.WriteFile(path, []byte("fetched long ago"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	pb := &Peers{M: b.m, Log: discardLog()}
+	pb.SyncOnce(t.Context(), a.client(t))
+	time.Sleep(100 * time.Millisecond)
+	if b.has("old", "v1") {
+		t.Fatal("copied without a checksum to check it by")
+	}
+	if n := a.m.FillChecksums(t.Context()); n != 1 || readChecksum(path) == "" {
+		t.Fatalf("recorded %d", n)
+	}
+	if a.m.FillChecksums(t.Context()) != 0 {
+		t.Error("recorded again")
+	}
+	pb.SyncOnce(t.Context(), a.client(t))
+	waitPresent(t, b, "old", "v1")
+}
+
+// A file removed outside the running mirror is not reported as here, and its
+// removal, recorded, reaches a peer instead of the peer's copy coming back.
+func TestAFileRemovedByHandIsNotReportedAndItsRemovalIsCarried(t *testing.T) {
+	blob := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		_, _ = w.Write([]byte("weights"))
+	}))
+	defer blob.Close()
+	a, b := newTestMirror(t), newTestMirror(t)
+	e := Entry{Model: "qwen", Version: "v1", File: "m.gguf"}
+	if _, err := a.m.Start(blob.URL+"/m.gguf", e); err != nil {
+		t.Fatal(err)
+	}
+	waitPresent(t, a, "qwen", "v1")
+	pb := &Peers{M: b.m, Log: discardLog()}
+	pb.SyncOnce(t.Context(), a.client(t))
+	waitPresent(t, b, "qwen", "v1")
+
+	time.Sleep(10 * time.Millisecond)
+	if err := Remove(a.root, e); err != nil {
+		t.Fatal(err)
+	}
+	if err := RecordRemoval(a.root, "qwen", "v1"); err != nil {
+		t.Fatal(err)
+	}
+	if a.has("qwen", "v1") {
+		t.Fatal("reported as here after it was removed")
+	}
+	all, _ := a.m.All()
+	for _, st := range all {
+		if st.Model == "qwen" {
+			t.Fatalf("listed after it was removed: %+v", st)
+		}
+	}
+	pa := &Peers{M: a.m, Log: discardLog()}
+	pa.SyncOnce(t.Context(), b.client(t))
+	time.Sleep(200 * time.Millisecond)
+	if a.has("qwen", "v1") {
+		t.Fatal("copied back from the peer")
+	}
+	pb.SyncOnce(t.Context(), a.client(t))
+	if b.has("qwen", "v1") {
+		t.Error("the peer kept it")
+	}
+}
@@ -57,3 +57,22 @@ func (m *Manager) clearTombstone(model, version string) {
 		m.writeTombstones(t)
 	}
 }
+
+// RecordRemoval keeps a version's removal for a mirror's peers when the
+// removal was made outside the running mirror, by llmgw mirror remove: once
+// none of its files is left, so a peer removes its copy rather than this
+// mirror copying it back.
+func RecordRemoval(root, model, version string) error {
+	entries, err := List(root)
+	if err != nil {
+		return err
+	}
+	for _, e := range entries {
+		if e.Model == model && e.Version == version {
+			return nil
+		}
+	}
+	m := &Manager{root: root}
+	m.recordTombstone(model, version, time.Now())
+	return nil
+}
@@ -155,10 +155,7 @@ func fitsOn(m Model, h Hardware, d Demand, p Precision, context int) []Fit {
 		if nodes > 1 && h.Fabric == "" {
 			continue
 		}
-		// The KV heads are split among the cards, or past their number each
-		// card holds a copy of one: the width divides them or they divide
-		// it. A model with 8 KV heads splits 2, 4, 8 or 16 ways, not 12.
-		if m.KVHeads > 0 && m.KVHeads%width != 0 && width%m.KVHeads != 0 {
+		if !SplitsHeads(m.KVHeads, width) {
 			continue
 		}

@@ -71,6 +71,9 @@ func (n PoolNode) kinds() []PoolKind {
 	return []PoolKind{{Class: n.Class, GPUMemGB: n.GPUMemGB, GPUs: n.GPUs, FreeGPUs: n.FreeGPUs}}
 }

+// Kinds is a node's devices by class: its one class, or each of several.
+func Kinds(n PoolNode) []PoolKind { return n.kinds() }
+
 // isMIG reports whether a class names MIG instances: "a100-80g-mig-3g.40gb".
 // A replica runs on exactly one, never a group of them, so a plan offers one
 // instance and nothing wider. Read from the class label, as the card's memory
@@ -263,12 +266,7 @@ func vllmOptions(a Artifact, kv int64, sh Shape, nodes []PoolNode, bpw float64)
 	perCard := func(width int) int64 {
 		return (a.Bytes+kv)/int64(width) + overheadBytes(a.Bytes/int64(width))
 	}
-	// Tensor parallel splits the KV heads among the cards, or past their
-	// number gives each card a copy of one: so the width divides the heads,
-	// or the heads divide the width. Qwen3-235B has four and runs on eight.
-	divides := func(w int) bool {
-		return sh.KVHeads == 0 || sh.KVHeads%w == 0 || w%sh.KVHeads == 0
-	}
+	divides := func(w int) bool { return SplitsHeads(sh.KVHeads, w) }
 	for _, c := range classes(nodes) {
 		budget := float64(c.mem) * gpuShare
 		opt := func(perNode, n int, fabric string) Option {
@@ -516,3 +514,13 @@ func CardNeed(engine string, weights, kv int64, cards int) float64 {
 // CardBudget is what a card of a class may be planned to hold, in GB; 0 for
 // a class whose memory its name does not say.
 func CardBudget(class string) float64 { return float64(MemFromClass(class)) * gpuShare }
+
+// SplitsHeads reports whether tensor parallel can spread a model of kvHeads
+// KV heads over width cards. It splits the heads among the cards, or past
+// their number gives each card a copy of one: so the width divides the
+// heads, or the heads divide the width. Qwen3-235B has four and runs on
+// eight; a model with 8 splits 2, 4, 8 or 16 ways, not 12. 0 heads is
+// unknown, and allowed.
+func SplitsHeads(kvHeads, width int) bool {
+	return kvHeads == 0 || width <= 1 || kvHeads%width == 0 || width%kvHeads == 0
+}
@@ -68,9 +68,13 @@ func (r *Router) loadStatus(model string) modelStatus {
 	if best != nil {
 		return *best
 	}
-	if _, m := r.resolve(model); m.Scaling.OnDemand() {
+	_, m := r.resolve(model)
+	if m.Scaling.OnDemand() {
 		return modelStatus{State: "waiting", Message: "waiting for a node with room for it"}
 	}
+	if m.Stopped != nil {
+		return modelStatus{State: "stopped", Message: "an administrator stopped this model; it is kept, and runs again when started"}
+	}
 	return modelStatus{State: "stopped", Message: "this model is not running"}
 }

@@ -546,3 +546,11 @@ func TestTheCPUWithGPUsFreeIsAChoiceMadeOnPurpose(t *testing.T) {
 		t.Error("with no GPU free the CPU is the only choice and should not be questioned")
 	}
 }
+
+func (f *fakeCatalog) Stop(_ context.Context, name, _ string) (catalog.Model, error) {
+	return catalog.Model{Name: name, Stopped: &catalog.Scaling{Min: 1, Max: 1}}, f.err
+}
+
+func (f *fakeCatalog) Start(_ context.Context, name, _ string) (catalog.Model, error) {
+	return catalog.Model{Name: name, Scaling: catalog.Scaling{Min: 1, Max: 1}}, f.err
+}
Author	SHA1	Message	Date
Ilker ManapandClaude Opus 5.5	01ad8c9207	v0.15.3: changelog and README CI / test (push) Successful in 11m44s Details Release / build (push) Successful in 5m43s Details Release / notes (push) Successful in 4s Details Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-29 20:54:53 +01:00
Ilker ManapandClaude Opus 5.5	59187cdae5	A model can be stopped and kept, or removed with its files from the mirror and the nodes CI / test (push) Successful in 10m37s Details Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-29 20:45:48 +01:00
Ilker ManapandClaude Opus 5.5	5726ceb84e	BACKLOG: J4 done, tried on the PoC with the first mirror stopped CI / test (push) Successful in 9m21s Details Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-29 20:01:47 +01:00
Ilker ManapandClaude Opus 5.5	435ffe1171	A file removed outside the running mirror is no longer reported, and its removal reaches the peers Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-29 19:25:00 +01:00
Ilker ManapandClaude Opus 5.5	785e06ecbf	A mirror with peers records the checksums of files that had none, so they are copied too Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-29 18:36:59 +01:00
Ilker ManapandClaude Opus 5.5	0033a530c7	mirror add-peer on a mirror machine of its own says what to run on a control plane Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-29 18:30:30 +01:00
Ilker ManapandClaude Opus 5.5	8cd376615b	v0.15.2: changelog and README Release / build (push) Successful in 4m44s Details Release / notes (push) Successful in 4s Details CI / test (push) Successful in 9m18s Details Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-29 16:46:15 +01:00
Ilker ManapandClaude Opus 5.5	e1cff0d4e3	A served model can be moved to more cards, or fewer CI / test (push) Successful in 10m28s Details llmgw ctl model gpus MODEL N [--profile P] [--in-place], POST /api/v1/models/{name}/gpus and the console's GPUs…: how many cards of one node each replica uses. Tensor parallel for vLLM, SGLang and TensorRT-LLM, held to the model's KV heads (modelplan.SplitsHeads, now one rule for the planner and this); the layers split for llama.cpp, its old --tensor-split dropped. Refused for a count no node has, for cards that would not hold the weights and the context, and for engines on one card. The replicas start again beside the old ones; with no room beside them the command says so, and --in-place stops each first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-29 16:40:49 +01:00