A model can be stopped and kept, or removed with its files from the mirror and the nodes
CI / test (push) Successful in 10m37s
CI / test (push) Successful in 10m37s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
5726ceb84e
commit
59187cdae5
19 files changed
+760
-14
No files matched your search
@@ -1629,6 +1629,30 @@ Not yet: models declared in server.yaml cannot name a peer's model as a
|
||||
fallback (the catalog in the file is checked before the peers are read), and
|
||||
a peer behind a private CA needs that CA in the system's trust store.
|
||||
|
||||
### Stopping a model, and removing one with its files
|
||||
|
||||
```
|
||||
llmgw ctl model stop qwen-7b # none of its replicas run; it is kept
|
||||
llmgw ctl model start qwen-7b # runs again, scaled as it was
|
||||
llmgw ctl model remove qwen-7b # leaves the catalog; its files stay
|
||||
llmgw ctl model remove qwen-7b --purge # its files on the mirror and the nodes too
|
||||
```
|
||||
|
||||
- **Stop** keeps the model in the catalog with its files and what it was
|
||||
scaled to; a request for it is told an administrator stopped it. `ctl
|
||||
models` shows it as stopped. A `ctl scale` while it is stopped starts it at
|
||||
that number.
|
||||
- **Remove** leaves its files on the mirror and the nodes, for adding it
|
||||
again without fetching it. `ctl weights` lists such files on the nodes.
|
||||
- **Remove --purge** deletes every version of it on the mirror (a second
|
||||
mirror follows), its adapters and the weights its last update replaced,
|
||||
and the nodes' copies once its replicas have stopped. What another model
|
||||
uses — a canary's weights, a shared adapter — stays. It cannot be undone:
|
||||
adding it again fetches it again.
|
||||
|
||||
The console has the same in a model's menu: Stop… or Start, Remove…, and
|
||||
Remove with its files….
|
||||
|
||||
### Changing a served model's context
|
||||
|
||||
How much one request may use is set when a model is served. To change it
|
||||
|
||||
Reference in new issue
Block a user