A bundle carries what a cluster would download — llmgw's release with its
signed checksums, every llama.cpp build per architecture with GitHub's
checksums, model files pinned and checked against Hugging Face's — into a
directory or a tar whose manifest sha256 is carried apart. Import checks
everything, the release against the signing key, before anything is put in
the mirror. With offline.mirror set, nodes install llama.cpp from the
mirror against checksums the control plane holds and sends over the agent
stream; releases default to the mirror for node add, ctl upgrade and agent
upgrades. server init and the setup form install from a bundle.
The Helm chart runs one control plane with SQLite on a claim or an external
PostgreSQL, Secrets for the agent certificate and admin token, Services,
Ingress and ServiceMonitor; deploy/container/Dockerfile builds its image.
GPU nodes stay outside Kubernetes, as docs/kubernetes.md explains.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
trusted.pub now holds the release signing key, so self-upgrade, agent
upgrade and node onboarding refuse a release whose checksums are not signed
with it. Onboarding fetched the release on the node and compared it with a
checksum from the same server; the control plane now fetches it for the
node's architecture with release.Fetch, checks signature and checksum, and
hands the bytes over. Release addresses must be https (loopback excepted).
LLMGW_RELEASE_KEY overrides the key only in development builds. The CI
release job fails instead of publishing an unsigned release while a key is
built in.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ed25519 over checksums.txt: llmgw release keygen makes the pair, CI signs
with the RELEASE_SIGNING_KEY secret when it is set, and release.Install
refuses an unsigned or wrongly signed release when the build embeds a
trusted key (internal/release/trusted.pub, empty for now) or
LLMGW_RELEASE_KEY names one.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every upgrade so far has been a script of mine copying a binary to each
machine over ssh, restarting it and polling until it came back — which is
the job, but not something the product could do, and which needed ssh into
every node.
`sudo llmgw ctl upgrade VERSION` on the control plane does it. This machine
first: the release for its platform, checked against the release's
checksums, the old binary kept beside it, llmgw-server restarted, and a wait
until every node that was ready has reconnected. Then each node in turn: the
control plane tells its agent the version over the stream it already holds;
the agent fetches and checks the release itself, swaps it in with a rename,
answers, and exits with status 75 so systemd starts the new binary. Engines
keep running, as for any agent restart, and are adopted again.
A node is done when it is back on the new version, ready, with as many ready
replicas as before; one that is not within the timeout stops the run before
the next node, so a bad release reaches one machine. Nodes already on the
version are skipped, so running it again resumes.
The fetch-and-verify is one package now, shared with onboarding's idea of
what a release version looks like.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
