Upgrade the cluster with one command, over the agents' own streams
Every upgrade so far has been a script of mine copying a binary to each machine over ssh, restarting it and polling until it came back — which is the job, but not something the product could do, and which needed ssh into every node. `sudo llmgw ctl upgrade VERSION` on the control plane does it. This machine first: the release for its platform, checked against the release's checksums, the old binary kept beside it, llmgw-server restarted, and a wait until every node that was ready has reconnected. Then each node in turn: the control plane tells its agent the version over the stream it already holds; the agent fetches and checks the release itself, swaps it in with a rename, answers, and exits with status 75 so systemd starts the new binary. Engines keep running, as for any agent restart, and are adopted again. A node is done when it is back on the new version, ready, with as many ready replicas as before; one that is not within the timeout stops the run before the next node, so a bad release reaches one machine. Nodes already on the version are skipped, so running it again resumes. The fetch-and-verify is one package now, shared with onboarding's idea of what a release version looks like. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
3bf4b7b709
commit
116e75f8cb
19 files changed
+1082
-90
No files matched your search
+29
-32
@@ -909,55 +909,52 @@ refuses every request.
|
||||
|
||||
## Upgrading
|
||||
|
||||
One binary on every machine, so an upgrade is a copy and a restart. The part
|
||||
worth care is the copy, not the order.
|
||||
On the control plane:
|
||||
|
||||
```
|
||||
sudo llmgw ctl upgrade v0.10.4
|
||||
```
|
||||
|
||||
It upgrades this machine first — fetching the release for its platform,
|
||||
checking it against the release's checksums, keeping the old binary as
|
||||
`/usr/local/bin/llmgw.bak-pre-<version>`, restarting `llmgw-server` and waiting
|
||||
for every node to reconnect — and then each node in turn. A node's agent is told
|
||||
the version over the stream it already holds, fetches and checks the release
|
||||
itself, swaps it in and restarts; no ssh into the nodes is needed. Release
|
||||
candidates (`v0.10.4-rc.2`) work the same way.
|
||||
|
||||
A node counts as done when it is back on the new version, ready, with as many
|
||||
ready replicas as before. One that is not within `--timeout` (three minutes)
|
||||
stops the upgrade there, so a bad release reaches one node rather than all of
|
||||
them; `llmgw ctl logs <node>` says why, and the node keeps its previous binary
|
||||
beside the new one. Nodes already on the version are skipped, so running the
|
||||
command again resumes where it stopped. `--nodes a,b` limits it to some nodes,
|
||||
and `--skip-control-plane` leaves this machine alone.
|
||||
|
||||
**Nothing stops serving.** Engines are started with `KillMode=process`, so they
|
||||
keep running when the agent restarts, and the agent reports them back on its
|
||||
next connection. The control plane holds no replica state across its own
|
||||
restart either — it rebuilds the fleet from what the agents report. A rolling
|
||||
upgrade of the whole cluster interrupts no request that is not in flight at the
|
||||
moment of the restart.
|
||||
restart either — it rebuilds the fleet from what the agents report.
|
||||
|
||||
**A mixed fleet is fine while you roll.** The agent protocol only ever gains
|
||||
fields, and both sides treat an absent one as unknown rather than as a default.
|
||||
Order does not matter; finish the roll the same day anyway, because
|
||||
`ctl nodes` reports each agent's version and a fleet that disagrees with itself
|
||||
is one more thing to hold in your head.
|
||||
|
||||
```
|
||||
VERSION=v0.9.0
|
||||
# copy to every machine first, install nowhere yet
|
||||
for h in 10.0.0.140 10.0.0.141 10.0.0.142 10.0.0.146; do
|
||||
ssh $h 'cat > /tmp/llmgw-new; stat -c %s /tmp/llmgw-new' < llmgw-$VERSION-linux-amd64
|
||||
done
|
||||
```
|
||||
|
||||
**Check every size before installing anything.** This is not defensive writing:
|
||||
on a host whose login shell prints anything, `scp` fails outright, and the
|
||||
obvious workaround — `ssh -n host 'cat > file'` — points stdin at `/dev/null`
|
||||
and writes a *zero-byte file* while reporting success. Installing that on a
|
||||
fleet takes the fleet down. Compare the number on each machine with the local
|
||||
one, then:
|
||||
|
||||
```
|
||||
sudo install -m0755 /tmp/llmgw-new /usr/local/bin/llmgw
|
||||
sudo systemctl restart llmgw-agent # or llmgw-server, or llmgw-mirror
|
||||
llmgw --version
|
||||
```
|
||||
Agents older than v0.10.4 cannot upgrade themselves: move those by hand once,
|
||||
installing the release binary and restarting `llmgw-agent`, and every upgrade
|
||||
after that is the command above. The mirror, which is not an agent, is
|
||||
upgraded the same way by hand.
|
||||
|
||||
Then confirm what you have, rather than assuming:
|
||||
|
||||
```
|
||||
llmgw ctl nodes # every agent back, LAST SEEN in seconds
|
||||
llmgw ctl nodes # every agent on the new version, LAST SEEN in seconds
|
||||
llmgw ctl replicas # same replicas, adopted rather than replaced
|
||||
```
|
||||
|
||||
A replica may come back marked **`ready (stale profile)`**. That means it is
|
||||
running a profile that has since been edited — the reconciler leaves a serving
|
||||
replica alone on purpose, because churning one costs more than the difference.
|
||||
Restart it when you want the new profile to take effect; the flag is there so
|
||||
the choice is yours rather than silent.
|
||||
`llmgw ctl restart <replica>` replaces it with one on the new profile, starting
|
||||
the new one before stopping the old.
|
||||
|
||||
## Backing up
|
||||
|
||||
|
||||
Reference in new issue
Block a user