A model can be stopped and kept, or removed with its files from the mirror and the nodes
CI / test (push) Successful in 10m37s
CI / test (push) Successful in 10m37s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
5726ceb84e
commit
59187cdae5
19 files changed
+760
-14
No files matched your search
+104
-1
@@ -2387,7 +2387,7 @@
|
||||
"delete": {
|
||||
"operationId": "deleteModelsName",
|
||||
"summary": "Remove a model added at runtime",
|
||||
"description": "A model declared in the cluster's settings is refused: it is changed there.",
|
||||
"description": "A model declared in the cluster's settings is refused: it is changed there. With purge, its files go too: every version of it the mirror holds (a peer mirror follows), and the nodes' copies once its replicas have stopped, waiting up to a minute for them; what another model uses stays. The answer is then a PurgeView.",
|
||||
"tags": [
|
||||
"models"
|
||||
],
|
||||
@@ -2401,6 +2401,14 @@
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "purge",
|
||||
"in": "query",
|
||||
"description": "Delete its files from the mirror and the nodes as well.",
|
||||
"schema": {
|
||||
"type": "boolean"
|
||||
}
|
||||
},
|
||||
{
|
||||
"$ref": "#/components/parameters/Actor"
|
||||
}
|
||||
@@ -3400,6 +3408,89 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/models/{name}/start": {
|
||||
"post": {
|
||||
"operationId": "postModelsNameStart",
|
||||
"summary": "Run a stopped model again, scaled as it was",
|
||||
"tags": [
|
||||
"models"
|
||||
],
|
||||
"parameters": [
|
||||
{
|
||||
"name": "name",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"description": "The model's name in the catalog.",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
{
|
||||
"$ref": "#/components/parameters/Actor"
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/ScaleResponse"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"$ref": "#/components/responses/Unauthorized"
|
||||
},
|
||||
"default": {
|
||||
"$ref": "#/components/responses/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/models/{name}/stop": {
|
||||
"post": {
|
||||
"operationId": "postModelsNameStop",
|
||||
"summary": "Take a model out of service and keep it",
|
||||
"description": "None of its replicas run, and requests are told an administrator stopped it; what it was scaled to is kept for start.",
|
||||
"tags": [
|
||||
"models"
|
||||
],
|
||||
"parameters": [
|
||||
{
|
||||
"name": "name",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"description": "The model's name in the catalog.",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
{
|
||||
"$ref": "#/components/parameters/Actor"
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/ScaleResponse"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"$ref": "#/components/responses/Unauthorized"
|
||||
},
|
||||
"default": {
|
||||
"$ref": "#/components/responses/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/models/{name}/target": {
|
||||
"get": {
|
||||
"operationId": "getModelsNameTarget",
|
||||
@@ -7859,6 +7950,10 @@
|
||||
"max_output": {
|
||||
"type": "integer"
|
||||
},
|
||||
"stopped": {
|
||||
"type": "boolean",
|
||||
"description": "Stopped is a model an administrator took out of service, kept."
|
||||
},
|
||||
"canary": {
|
||||
"$ref": "#/components/schemas/catalog.Canary",
|
||||
"description": "Canary is a share of its requests sent to a candidate; CanaryOf is set on the candidate, naming the model it is a canary of."
|
||||
@@ -8939,6 +9034,10 @@
|
||||
},
|
||||
"on_demand": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"stopped": {
|
||||
"type": "boolean",
|
||||
"description": "Stopped is a model taken out of service, kept, until it is started."
|
||||
}
|
||||
}
|
||||
},
|
||||
@@ -10064,6 +10163,10 @@
|
||||
"scaling": {
|
||||
"$ref": "#/components/schemas/catalog.Scaling"
|
||||
},
|
||||
"stopped": {
|
||||
"$ref": "#/components/schemas/catalog.Scaling",
|
||||
"description": "Stopped is the scaling a model had when an administrator stopped it: kept in the catalog with none running (Scaling 0/0) until started again with this. Nil for a model not stopped."
|
||||
},
|
||||
"class": {
|
||||
"type": "string",
|
||||
"description": "Class is how much this model's capacity is worth against another's, when there is not enough for both: critical, standard or opportunistic. Empty is standard, or opportunistic for a model loaded on demand. See EffectiveClass."
|
||||
|
||||
@@ -1629,6 +1629,30 @@ Not yet: models declared in server.yaml cannot name a peer's model as a
|
||||
fallback (the catalog in the file is checked before the peers are read), and
|
||||
a peer behind a private CA needs that CA in the system's trust store.
|
||||
|
||||
### Stopping a model, and removing one with its files
|
||||
|
||||
```
|
||||
llmgw ctl model stop qwen-7b # none of its replicas run; it is kept
|
||||
llmgw ctl model start qwen-7b # runs again, scaled as it was
|
||||
llmgw ctl model remove qwen-7b # leaves the catalog; its files stay
|
||||
llmgw ctl model remove qwen-7b --purge # its files on the mirror and the nodes too
|
||||
```
|
||||
|
||||
- **Stop** keeps the model in the catalog with its files and what it was
|
||||
scaled to; a request for it is told an administrator stopped it. `ctl
|
||||
models` shows it as stopped. A `ctl scale` while it is stopped starts it at
|
||||
that number.
|
||||
- **Remove** leaves its files on the mirror and the nodes, for adding it
|
||||
again without fetching it. `ctl weights` lists such files on the nodes.
|
||||
- **Remove --purge** deletes every version of it on the mirror (a second
|
||||
mirror follows), its adapters and the weights its last update replaced,
|
||||
and the nodes' copies once its replicas have stopped. What another model
|
||||
uses — a canary's weights, a shared adapter — stays. It cannot be undone:
|
||||
adding it again fetches it again.
|
||||
|
||||
The console has the same in a model's menu: Stop… or Start, Remove…, and
|
||||
Remove with its files….
|
||||
|
||||
### Changing a served model's context
|
||||
|
||||
How much one request may use is set when a model is served. To change it
|
||||
|
||||
@@ -192,6 +192,7 @@ rather than repeated.
|
||||
| `allow_pickle` | `bool` | [Model](#model) |
|
||||
| `profiles` | `[]Profile` | [Model](#model) |
|
||||
| `scaling` | `Scaling` | [Model](#model) |
|
||||
| `stopped` | `*Scaling` | [Model](#model) |
|
||||
| `class` | `string` | [Model](#model) |
|
||||
| `owner` | `string` | [Model](#model) |
|
||||
| `external` | `*External` | [Model](#model) |
|
||||
@@ -1824,6 +1825,16 @@ no safetensors or GGUF form and whose files were checked.
|
||||
|
||||
Takes a [Scaling](#scaling) block.
|
||||
|
||||
### `stopped`
|
||||
|
||||
`*Scaling`
|
||||
|
||||
Stopped is the scaling a model had when an administrator stopped it:
|
||||
kept in the catalog with none running (Scaling 0/0) until started
|
||||
again with this. Nil for a model not stopped.
|
||||
|
||||
Takes a [Scaling](#scaling) block.
|
||||
|
||||
### `class`
|
||||
|
||||
`string`
|
||||
|
||||
Reference in new issue
Block a user