llmgw stop, start and restart act on the whole system
Issue #3 asked for commands to stop and start the system, and 05aa21c gave
only one machine's services. Now:
- llmgw stop sets serving.stopped in the cluster's settings, live: the
reconciler wants every model at no replicas, freeing every GPU, and the
gateway answers every POST under /v1/ with 503 system_stopped. The control
planes, agents and mirror keep running, and no model's definition changes.
- llmgw start clears it; every model is placed again as it asks.
- llmgw restart has every agent restart over its stream (RestartAgent), then
the other control planes over ssh, standbys before a leader, then this
one; engines keep serving.
- llmgw status is the cluster's summary, saying when it is stopped.
One machine's services moved to llmgw service status|start|stop|restart.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
05aa21c4d4
commit
7245dbe1dd
29 files changed
+891
-254
No files matched your search
@@ -17,6 +17,9 @@
|
||||
}
|
||||
],
|
||||
"tags": [
|
||||
{
|
||||
"name": "agents"
|
||||
},
|
||||
{
|
||||
"name": "audit"
|
||||
},
|
||||
@@ -100,6 +103,38 @@
|
||||
}
|
||||
],
|
||||
"paths": {
|
||||
"/api/v1/agents/restart": {
|
||||
"post": {
|
||||
"operationId": "postAgentsRestart",
|
||||
"summary": "Have every node's agent start again; their engines keep serving",
|
||||
"tags": [
|
||||
"agents"
|
||||
],
|
||||
"parameters": [
|
||||
{
|
||||
"$ref": "#/components/parameters/Actor"
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/AgentRestartView"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"$ref": "#/components/responses/Unauthorized"
|
||||
},
|
||||
"default": {
|
||||
"$ref": "#/components/responses/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/audit": {
|
||||
"get": {
|
||||
"operationId": "getAudit",
|
||||
@@ -3554,6 +3589,18 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"AgentRestartView": {
|
||||
"type": "object",
|
||||
"description": "AgentRestartView names the nodes whose agents were told to restart.",
|
||||
"properties": {
|
||||
"nodes": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"CanaryPercentRequest": {
|
||||
"type": "object",
|
||||
"description": "CanaryPercentRequest changes the share of a model's requests its canary takes.",
|
||||
@@ -5928,6 +5975,10 @@
|
||||
},
|
||||
"description": "Judges are the guardrails' judge models with something wrong."
|
||||
},
|
||||
"stopped": {
|
||||
"type": "boolean",
|
||||
"description": "Stopped says the operators have stopped the service (llmgw stop): no model runs, and requests for one are answered 503."
|
||||
},
|
||||
"guard_off": {
|
||||
"type": "boolean",
|
||||
"description": "GuardOff says the guardrails are turned off (guard.enabled: false): no request is checked, whatever the policies say."
|
||||
|
||||
+31
-9
@@ -2294,20 +2294,42 @@ A cluster set up online becomes offline the same way: `bundle import` on the
|
||||
mirror and control plane, then the `offline:` block in `server.yaml` and a
|
||||
restart.
|
||||
|
||||
## Stopping and starting a machine
|
||||
## Stopping and starting the system
|
||||
|
||||
From a control plane:
|
||||
|
||||
```
|
||||
sudo llmgw status # each llmgw service here, and whether it runs
|
||||
sudo llmgw stop # agent, control plane, mirror — whichever this machine has
|
||||
sudo llmgw start
|
||||
sudo llmgw restart agent # or name one: server, agent, mirror
|
||||
llmgw status # does it serve; its control planes, nodes, models, store
|
||||
llmgw stop --note "maintenance" # every model taken down, every request answered 503
|
||||
llmgw start # every model back as its catalog entry asks
|
||||
sudo llmgw restart # every agent and control plane started again, engines serving throughout
|
||||
```
|
||||
|
||||
`llmgw stop` frees every GPU and answers requests for a model with 503
|
||||
(`system_stopped`, with Retry-After) until `llmgw start`. The control planes,
|
||||
agents and mirror keep running, so the console, the admin pages and `llmgw
|
||||
start` keep working, and no model's definition changes. It is the cluster's
|
||||
setting `serving.stopped`, so a restart does not undo it, and it is in the
|
||||
audit with its note. `--wait` waits until every replica is down, or every
|
||||
model has its replicas again.
|
||||
|
||||
`llmgw restart` restarts every node's agent, then the other control planes
|
||||
over ssh (`--ssh-user`, the one sudo was run by), then this one — the leader
|
||||
last, so it hands over to a standby already restarted. Engines keep serving
|
||||
and are taken over again; nothing is placed anew.
|
||||
|
||||
One machine's own services are `llmgw service`:
|
||||
|
||||
```
|
||||
sudo llmgw service status
|
||||
sudo llmgw service stop # agent, control plane, mirror — whichever this machine has
|
||||
sudo llmgw service start
|
||||
sudo llmgw service restart agent # or name one: server, agent, mirror
|
||||
```
|
||||
|
||||
Stopping a node's agent leaves its engines serving — the agent takes them
|
||||
over again when it starts — unless `llmgw stop --engines` stops them too, and
|
||||
the control plane then places their replicas elsewhere. Stopping the leading
|
||||
control plane hands over to a standby within seconds. The whole cluster's
|
||||
state is `llmgw ctl status`.
|
||||
over again when it starts — unless `llmgw service stop --engines` stops them
|
||||
too, and the control plane then places their replicas elsewhere.
|
||||
|
||||
## Removing llmgw from a machine
|
||||
|
||||
|
||||
@@ -290,6 +290,7 @@ rather than repeated.
|
||||
| `secrets` | `Secrets` | [Server](#server) |
|
||||
| `notify` | `notify.Config` | [Server](#server) |
|
||||
| `mail` | `mail.Config` | [Server](#server) |
|
||||
| `serving` | `ServingSettings` | [Server](#server) |
|
||||
| `firewall` | `FirewallSettings` | [Server](#server) |
|
||||
| `firewall_networks` | `[]string` | [Server](#server) |
|
||||
| `backup_recipients` | `[]string` | [Server](#server) |
|
||||
@@ -302,6 +303,7 @@ rather than repeated.
|
||||
| `facility` | `Facility` | [Server](#server) |
|
||||
| `federation` | `Federation` | [Server](#server) |
|
||||
| `endpoints` | `[]Endpoint` | [Server](#server) |
|
||||
| `stopped` | `bool` | [ServingSettings](#servingsettings) |
|
||||
| `bin` | `string` | [SimConfig](#simconfig) |
|
||||
| `python` | `string` | [TRTLLMConfig](#trtllmconfig) |
|
||||
| `store` | `string` | [Threads](#threads) |
|
||||
@@ -2481,6 +2483,15 @@ forgotten password. Unset, none is sent and people ask an operator.
|
||||
|
||||
Takes a [mail.Config](#mailconfig) block.
|
||||
|
||||
### `serving`
|
||||
|
||||
`ServingSettings`
|
||||
|
||||
Serving stops and starts the whole cluster's service (llmgw stop, llmgw
|
||||
start).
|
||||
|
||||
Takes a [ServingSettings](#servingsettings) block.
|
||||
|
||||
### `firewall`
|
||||
|
||||
`FirewallSettings`
|
||||
@@ -2579,6 +2590,18 @@ Endpoints is the phase-1 static replica list. It is only consulted when
|
||||
no model declares a profile, which is what makes a router-only smoke test
|
||||
against sim-engines possible without a control plane.
|
||||
|
||||
## ServingSettings
|
||||
|
||||
### `stopped`
|
||||
|
||||
`bool`
|
||||
|
||||
Stopped takes every model down and answers every request for one with
|
||||
503 (system_stopped), leaving the control planes, agents and mirror
|
||||
running and every model's definition as it is: llmgw stop sets it,
|
||||
llmgw start clears it, and the models come back as their catalog
|
||||
entries ask. In force on every control plane within seconds.
|
||||
|
||||
## SimConfig
|
||||
|
||||
### `bin`
|
||||
|
||||
Reference in new issue
Block a user