Upgrade the cluster with one command, over the agents' own streams
Release / build (push) Successful in 1m5s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m42s

Every upgrade so far has been a script of mine copying a binary to each
machine over ssh, restarting it and polling until it came back — which is
the job, but not something the product could do, and which needed ssh into
every node.

`sudo llmgw ctl upgrade VERSION` on the control plane does it. This machine
first: the release for its platform, checked against the release's
checksums, the old binary kept beside it, llmgw-server restarted, and a wait
until every node that was ready has reconnected. Then each node in turn: the
control plane tells its agent the version over the stream it already holds;
the agent fetches and checks the release itself, swaps it in with a rename,
answers, and exits with status 75 so systemd starts the new binary. Engines
keep running, as for any agent restart, and are adopted again.

A node is done when it is back on the new version, ready, with as many ready
replicas as before; one that is not within the timeout stops the run before
the next node, so a bad release reaches one machine. Nodes already on the
version are skipped, so running it again resumes.

The fetch-and-verify is one package now, shared with onboarding's idea of
what a release version looks like.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-09-30 02:54:36 +02:00
1 parent 3bf4b7b709
commit 116e75f8cb
19 files changed
+1082 -90

No files matched your search

+29 -32
View File
@@ -909,55 +909,52 @@ refuses every request.
## Upgrading
One binary on every machine, so an upgrade is a copy and a restart. The part
worth care is the copy, not the order.
On the control plane:
```
sudo llmgw ctl upgrade v0.10.4
```
It upgrades this machine first — fetching the release for its platform,
checking it against the release's checksums, keeping the old binary as
`/usr/local/bin/llmgw.bak-pre-<version>`, restarting `llmgw-server` and waiting
for every node to reconnect — and then each node in turn. A node's agent is told
the version over the stream it already holds, fetches and checks the release
itself, swaps it in and restarts; no ssh into the nodes is needed. Release
candidates (`v0.10.4-rc.2`) work the same way.
A node counts as done when it is back on the new version, ready, with as many
ready replicas as before. One that is not within `--timeout` (three minutes)
stops the upgrade there, so a bad release reaches one node rather than all of
them; `llmgw ctl logs <node>` says why, and the node keeps its previous binary
beside the new one. Nodes already on the version are skipped, so running the
command again resumes where it stopped. `--nodes a,b` limits it to some nodes,
and `--skip-control-plane` leaves this machine alone.
**Nothing stops serving.** Engines are started with `KillMode=process`, so they
keep running when the agent restarts, and the agent reports them back on its
next connection. The control plane holds no replica state across its own
restart either — it rebuilds the fleet from what the agents report. A rolling
upgrade of the whole cluster interrupts no request that is not in flight at the
moment of the restart.
restart either — it rebuilds the fleet from what the agents report.
**A mixed fleet is fine while you roll.** The agent protocol only ever gains
fields, and both sides treat an absent one as unknown rather than as a default.
Order does not matter; finish the roll the same day anyway, because
`ctl nodes` reports each agent's version and a fleet that disagrees with itself
is one more thing to hold in your head.
```
VERSION=v0.9.0
# copy to every machine first, install nowhere yet
for h in 10.0.0.140 10.0.0.141 10.0.0.142 10.0.0.146; do
ssh $h 'cat > /tmp/llmgw-new; stat -c %s /tmp/llmgw-new' < llmgw-$VERSION-linux-amd64
done
```
**Check every size before installing anything.** This is not defensive writing:
on a host whose login shell prints anything, `scp` fails outright, and the
obvious workaround — `ssh -n host 'cat > file'` — points stdin at `/dev/null`
and writes a *zero-byte file* while reporting success. Installing that on a
fleet takes the fleet down. Compare the number on each machine with the local
one, then:
```
sudo install -m0755 /tmp/llmgw-new /usr/local/bin/llmgw
sudo systemctl restart llmgw-agent # or llmgw-server, or llmgw-mirror
llmgw --version
```
Agents older than v0.10.4 cannot upgrade themselves: move those by hand once,
installing the release binary and restarting `llmgw-agent`, and every upgrade
after that is the command above. The mirror, which is not an agent, is
upgraded the same way by hand.
Then confirm what you have, rather than assuming:
```
llmgw ctl nodes # every agent back, LAST SEEN in seconds
llmgw ctl nodes # every agent on the new version, LAST SEEN in seconds
llmgw ctl replicas # same replicas, adopted rather than replaced
```
A replica may come back marked **`ready (stale profile)`**. That means it is
running a profile that has since been edited — the reconciler leaves a serving
replica alone on purpose, because churning one costs more than the difference.
Restart it when you want the new profile to take effect; the flag is there so
the choice is yours rather than silent.
`llmgw ctl restart <replica>` replaces it with one on the new profile, starting
the new one before stopping the old.
## Backing up