ctl upgrade in waves, and the other control plane first
One node at a time was hours on a fleet of hundreds. Nodes now go one, then two, four, up to --parallel at once, each wave waited on whole; a wave that does not come back stops it. The other control plane is upgraded first over ssh, so a leader restarted last hands over to a standby on the release. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
964b659849
commit
c5e388807a
4 files changed
+245
-77
No files matched your search
+21
-15
@@ -2249,27 +2249,33 @@ joined with `node add`. Why, and how, is in [kubernetes.md](kubernetes.md).
|
||||
|
||||
## Upgrading
|
||||
|
||||
On the control plane:
|
||||
On a control plane:
|
||||
|
||||
```
|
||||
sudo llmgw ctl upgrade v0.10.4
|
||||
sudo --preserve-env=SSH_AUTH_SOCK llmgw ctl upgrade v0.10.4
|
||||
```
|
||||
|
||||
It upgrades this machine first — fetching the release for its platform,
|
||||
checking it against the release's checksums, keeping the old binary as
|
||||
`/usr/local/bin/llmgw.bak-pre-<version>`, restarting `llmgw-server` and waiting
|
||||
for every node to reconnect — and then each node in turn. A node's agent is told
|
||||
the version over the stream it already holds, fetches and checks the release
|
||||
itself, swaps it in and restarts; no ssh into the nodes is needed. Release
|
||||
candidates (`v0.10.4-rc.2`) work the same way.
|
||||
With two control planes it upgrades the other one first, over ssh as the user
|
||||
who ran sudo (`--ssh-user`), with `llmgw upgrade` there; then this machine —
|
||||
fetching the release for its platform, checking it against the release's
|
||||
checksums, keeping the old binary as `/usr/local/bin/llmgw.bak-pre-<version>`,
|
||||
restarting `llmgw-server` and waiting for every node to reconnect. A leader
|
||||
restarted last hands over to a standby already on the release.
|
||||
|
||||
Then the nodes, in waves: one first, then two, four, and so on up to
|
||||
`--parallel` (20) at once. A node's agent is told the version over the stream
|
||||
it already holds, fetches and checks the release itself, swaps it in and
|
||||
restarts; no ssh into the nodes is needed. A fleet of hundreds takes minutes,
|
||||
not hours. Release candidates (`v0.10.4-rc.2`) work the same way.
|
||||
|
||||
A node counts as done when it is back on the new version, ready, with as many
|
||||
ready replicas as before. One that is not within `--timeout` (three minutes)
|
||||
stops the upgrade there, so a bad release reaches one node rather than all of
|
||||
them; `llmgw ctl logs <node>` says why, and the node keeps its previous binary
|
||||
beside the new one. Nodes already on the version are skipped, so running the
|
||||
command again resumes where it stopped. `--nodes a,b` limits it to some nodes,
|
||||
and `--skip-control-plane` leaves this machine alone.
|
||||
ready replicas as before. A wave not back within `--timeout` (three minutes)
|
||||
stops the upgrade there — the first wave is one node, so a bad release reaches
|
||||
one node rather than all of them; `llmgw ctl logs <node>` says why, and the node
|
||||
keeps its previous binary beside the new one. Nodes already on the version are
|
||||
skipped, so running the command again resumes where it stopped. `--nodes a,b`
|
||||
limits it to some nodes, `--skip-control-plane` leaves the control planes alone,
|
||||
and `--other-control-planes=false` this machine's partner.
|
||||
|
||||
**Nothing stops serving.** Engines are started with `KillMode=process`, so they
|
||||
keep running when the agent restarts, and the agent reports them back on its
|
||||
|
||||
Reference in new issue
Block a user