ctl upgrade in waves, and the other control plane first

One node at a time was hours on a fleet of hundreds. Nodes now go one, then
two, four, up to --parallel at once, each wave waited on whole; a wave that
does not come back stops it. The other control plane is upgraded first over
ssh, so a leader restarted last hands over to a standby on the release.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-05 02:43:33 +02:00
1 parent 964b659849
commit c5e388807a
4 files changed
+245 -77

No files matched your search

+21 -15
View File
@@ -2249,27 +2249,33 @@ joined with `node add`. Why, and how, is in [kubernetes.md](kubernetes.md).
## Upgrading
On the control plane:
On a control plane:
```
sudo llmgw ctl upgrade v0.10.4
sudo --preserve-env=SSH_AUTH_SOCK llmgw ctl upgrade v0.10.4
```
It upgrades this machine first — fetching the release for its platform,
checking it against the release's checksums, keeping the old binary as
`/usr/local/bin/llmgw.bak-pre-<version>`, restarting `llmgw-server` and waiting
for every node to reconnect — and then each node in turn. A node's agent is told
the version over the stream it already holds, fetches and checks the release
itself, swaps it in and restarts; no ssh into the nodes is needed. Release
candidates (`v0.10.4-rc.2`) work the same way.
With two control planes it upgrades the other one first, over ssh as the user
who ran sudo (`--ssh-user`), with `llmgw upgrade` there; then this machine —
fetching the release for its platform, checking it against the release's
checksums, keeping the old binary as `/usr/local/bin/llmgw.bak-pre-<version>`,
restarting `llmgw-server` and waiting for every node to reconnect. A leader
restarted last hands over to a standby already on the release.
Then the nodes, in waves: one first, then two, four, and so on up to
`--parallel` (20) at once. A node's agent is told the version over the stream
it already holds, fetches and checks the release itself, swaps it in and
restarts; no ssh into the nodes is needed. A fleet of hundreds takes minutes,
not hours. Release candidates (`v0.10.4-rc.2`) work the same way.
A node counts as done when it is back on the new version, ready, with as many
ready replicas as before. One that is not within `--timeout` (three minutes)
stops the upgrade there, so a bad release reaches one node rather than all of
them; `llmgw ctl logs <node>` says why, and the node keeps its previous binary
beside the new one. Nodes already on the version are skipped, so running the
command again resumes where it stopped. `--nodes a,b` limits it to some nodes,
and `--skip-control-plane` leaves this machine alone.
ready replicas as before. A wave not back within `--timeout` (three minutes)
stops the upgrade there — the first wave is one node, so a bad release reaches
one node rather than all of them; `llmgw ctl logs <node>` says why, and the node
keeps its previous binary beside the new one. Nodes already on the version are
skipped, so running the command again resumes where it stopped. `--nodes a,b`
limits it to some nodes, `--skip-control-plane` leaves the control planes alone,
and `--other-control-planes=false` this machine's partner.
**Nothing stops serving.** Engines are started with `KillMode=process`, so they
keep running when the agent restarts, and the agent reports them back on its