A model can be stopped and kept, or removed with its files from the mirror and the nodes
CI / test (push) Successful in 10m37s

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-29 20:45:48 +01:00
1 parent 5726ceb84e
commit 59187cdae5
19 files changed
+760 -14

No files matched your search

+104 -1
View File
@@ -2387,7 +2387,7 @@
"delete": {
"operationId": "deleteModelsName",
"summary": "Remove a model added at runtime",
"description": "A model declared in the cluster's settings is refused: it is changed there.",
"description": "A model declared in the cluster's settings is refused: it is changed there. With purge, its files go too: every version of it the mirror holds (a peer mirror follows), and the nodes' copies once its replicas have stopped, waiting up to a minute for them; what another model uses stays. The answer is then a PurgeView.",
"tags": [
"models"
],
@@ -2401,6 +2401,14 @@
"type": "string"
}
},
{
"name": "purge",
"in": "query",
"description": "Delete its files from the mirror and the nodes as well.",
"schema": {
"type": "boolean"
}
},
{
"$ref": "#/components/parameters/Actor"
}
@@ -3400,6 +3408,89 @@
}
}
},
"/api/v1/models/{name}/start": {
"post": {
"operationId": "postModelsNameStart",
"summary": "Run a stopped model again, scaled as it was",
"tags": [
"models"
],
"parameters": [
{
"name": "name",
"in": "path",
"required": true,
"description": "The model's name in the catalog.",
"schema": {
"type": "string"
}
},
{
"$ref": "#/components/parameters/Actor"
}
],
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ScaleResponse"
}
}
}
},
"401": {
"$ref": "#/components/responses/Unauthorized"
},
"default": {
"$ref": "#/components/responses/Error"
}
}
}
},
"/api/v1/models/{name}/stop": {
"post": {
"operationId": "postModelsNameStop",
"summary": "Take a model out of service and keep it",
"description": "None of its replicas run, and requests are told an administrator stopped it; what it was scaled to is kept for start.",
"tags": [
"models"
],
"parameters": [
{
"name": "name",
"in": "path",
"required": true,
"description": "The model's name in the catalog.",
"schema": {
"type": "string"
}
},
{
"$ref": "#/components/parameters/Actor"
}
],
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ScaleResponse"
}
}
}
},
"401": {
"$ref": "#/components/responses/Unauthorized"
},
"default": {
"$ref": "#/components/responses/Error"
}
}
}
},
"/api/v1/models/{name}/target": {
"get": {
"operationId": "getModelsNameTarget",
@@ -7859,6 +7950,10 @@
"max_output": {
"type": "integer"
},
"stopped": {
"type": "boolean",
"description": "Stopped is a model an administrator took out of service, kept."
},
"canary": {
"$ref": "#/components/schemas/catalog.Canary",
"description": "Canary is a share of its requests sent to a candidate; CanaryOf is set on the candidate, naming the model it is a canary of."
@@ -8939,6 +9034,10 @@
},
"on_demand": {
"type": "boolean"
},
"stopped": {
"type": "boolean",
"description": "Stopped is a model taken out of service, kept, until it is started."
}
}
},
@@ -10064,6 +10163,10 @@
"scaling": {
"$ref": "#/components/schemas/catalog.Scaling"
},
"stopped": {
"$ref": "#/components/schemas/catalog.Scaling",
"description": "Stopped is the scaling a model had when an administrator stopped it: kept in the catalog with none running (Scaling 0/0) until started again with this. Nil for a model not stopped."
},
"class": {
"type": "string",
"description": "Class is how much this model's capacity is worth against another's, when there is not enough for both: critical, standard or opportunistic. Empty is standard, or opportunistic for a model loaded on demand. See EffectiveClass."
+24
View File
@@ -1629,6 +1629,30 @@ Not yet: models declared in server.yaml cannot name a peer's model as a
fallback (the catalog in the file is checked before the peers are read), and
a peer behind a private CA needs that CA in the system's trust store.
### Stopping a model, and removing one with its files
```
llmgw ctl model stop qwen-7b # none of its replicas run; it is kept
llmgw ctl model start qwen-7b # runs again, scaled as it was
llmgw ctl model remove qwen-7b # leaves the catalog; its files stay
llmgw ctl model remove qwen-7b --purge # its files on the mirror and the nodes too
```
- **Stop** keeps the model in the catalog with its files and what it was
scaled to; a request for it is told an administrator stopped it. `ctl
models` shows it as stopped. A `ctl scale` while it is stopped starts it at
that number.
- **Remove** leaves its files on the mirror and the nodes, for adding it
again without fetching it. `ctl weights` lists such files on the nodes.
- **Remove --purge** deletes every version of it on the mirror (a second
mirror follows), its adapters and the weights its last update replaced,
and the nodes' copies once its replicas have stopped. What another model
uses — a canary's weights, a shared adapter — stays. It cannot be undone:
adding it again fetches it again.
The console has the same in a model's menu: Stop… or Start, Remove…, and
Remove with its files….
### Changing a served model's context
How much one request may use is set when a model is served. To change it
+11
View File
@@ -192,6 +192,7 @@ rather than repeated.
| `allow_pickle` | `bool` | [Model](#model) |
| `profiles` | `[]Profile` | [Model](#model) |
| `scaling` | `Scaling` | [Model](#model) |
| `stopped` | `*Scaling` | [Model](#model) |
| `class` | `string` | [Model](#model) |
| `owner` | `string` | [Model](#model) |
| `external` | `*External` | [Model](#model) |
@@ -1824,6 +1825,16 @@ no safetensors or GGUF form and whose files were checked.
Takes a [Scaling](#scaling) block.
### `stopped`
`*Scaling`
Stopped is the scaling a model had when an administrator stopped it:
kept in the catalog with none running (Scaling 0/0) until started
again with this. Nil for a model not stopped.
Takes a [Scaling](#scaling) block.
### `class`
`string`