Upgrade the cluster with one command, over the agents' own streams

Every upgrade so far has been a script of mine copying a binary to each
machine over ssh, restarting it and polling until it came back — which is
the job, but not something the product could do, and which needed ssh into
every node.

`sudo llmgw ctl upgrade VERSION` on the control plane does it. This machine
first: the release for its platform, checked against the release's
checksums, the old binary kept beside it, llmgw-server restarted, and a wait
until every node that was ready has reconnected. Then each node in turn: the
control plane tells its agent the version over the stream it already holds;
the agent fetches and checks the release itself, swaps it in with a rename,
answers, and exits with status 75 so systemd starts the new binary. Engines
keep running, as for any agent restart, and are adopted again.

A node is done when it is back on the new version, ready, with as many ready
replicas as before; one that is not within the timeout stops the run before
the next node, so a bad release reaches one machine. Nodes already on the
version are skipped, so running it again resumes.

The fetch-and-verify is one package now, shared with onboarding's idea of
what a release version looks like.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-09-30 02:54:36 +02:00
1 parent 3bf4b7b709
commit 116e75f8cb
19 files changed
+1082 -90

No files matched your search

+11
View File
@@ -211,6 +211,17 @@ func (c *Controller) WeightRemover() WeightRemover {
return r
}
// Upgrader can tell a node's agent to upgrade itself.
type Upgrader interface {
Upgrade(ctx context.Context, nodeID, version, releases string) error
}
// Upgrader returns the dispatcher's upgrade capability, or nil.
func (c *Controller) Upgrader() Upgrader {
u, _ := c.disp.(Upgrader)
return u
}
// LogReader can ask a node for the recent lines of one of its logs.
type LogReader interface {
Logs(ctx context.Context, nodeID, source string, lines int) (out, sources []string, err error)