Upgrade the cluster with one command, over the agents' own streams

Every upgrade so far has been a script of mine copying a binary to each
machine over ssh, restarting it and polling until it came back — which is
the job, but not something the product could do, and which needed ssh into
every node.

`sudo llmgw ctl upgrade VERSION` on the control plane does it. This machine
first: the release for its platform, checked against the release's
checksums, the old binary kept beside it, llmgw-server restarted, and a wait
until every node that was ready has reconnected. Then each node in turn: the
control plane tells its agent the version over the stream it already holds;
the agent fetches and checks the release itself, swaps it in with a rename,
answers, and exits with status 75 so systemd starts the new binary. Engines
keep running, as for any agent restart, and are adopted again.

A node is done when it is back on the new version, ready, with as many ready
replicas as before; one that is not within the timeout stops the run before
the next node, so a bad release reaches one machine. Nodes already on the
version are skipped, so running it again resumes.

The fetch-and-verify is one package now, shared with onboarding's idea of
what a release version looks like.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-09-30 02:54:36 +02:00
1 parent 3bf4b7b709
commit 116e75f8cb
19 files changed
+1082 -90

No files matched your search

+27
View File
@@ -26,6 +26,7 @@ message AgentMessage {
LogsReply logs = 4;
FabricScanReply fabric_scan = 5;
FabricTestReply fabric_test = 6;
UpgradeReply upgrade = 7;
}
}
@@ -41,6 +42,7 @@ message ControlMessage {
LogsRequest logs = 7;
FabricScanRequest fabric_scan = 8;
FabricTestRequest fabric_test = 9;
UpgradeAgent upgrade = 10;
}
}
@@ -649,3 +651,28 @@ message FabricTestReply {
string error = 4;
int64 took_ms = 5;
}
// UpgradeAgent tells an agent to replace its own binary with a release and
// restart.
//
// The agent fetches the release for its own platform and checks it against
// the release's checksums before replacing anything, keeps the old binary
// beside the new, then exits so systemd starts the new one. Engines keep
// running throughout, as for any agent restart, and are adopted on the way
// back. Over the stream the agent already holds, so upgrading a fleet needs
// no ssh into it.
message UpgradeAgent {
string request_id = 1;
// version must be a release: v0.10.4 or v0.10.4-rc.2.
string version = 2;
// releases is the base URL release files are downloaded from.
string releases = 3;
}
// UpgradeReply says whether the new binary is in place. Sent before the
// agent exits; an empty error means it will be back on the new version.
message UpgradeReply {
string request_id = 1;
string version = 2;
string error = 3;
}