llmgw/notes/c18.md

C18 — Kubernetes/Helm and an air-gapped bundle

README's documentation list should gain a line for docs/kubernetes.md (the control plane in Kubernetes from the Helm chart, and why GPU nodes stay outside it) — README.md was not edited here.

CHANGELOG

Offline sites and Kubernetes

• Offline bundles. llmgw bundle create (on any machine with internet) collects what a cluster would otherwise download — llmgw's release for each --arch with its signed checksums, every llama.cpp build for those architectures (or the --variant ones) with the checksums GitHub publishes, the starter model and guardrails' judge, and any --model name/version=hf:owner/repo/file pinned to the registry's commit and its checksum — each checked as it arrives, into a directory or one .tar (--tar), with a manifest whose sha256 it prints to carry apart from the bundle. Run again into the same directory, it downloads only what is missing. llmgw bundle show --verify reads one back.
• llmgw bundle import checks every file against the manifest and llmgw's release against the project's signing key, and only then puts them into the mirror — releases and engines under _llmgw/, models where the mirror keeps models, with their checksums — so a damaged bundle leaves the mirror as it was; a different file under a version already there is refused. On a control plane it also records the engines' checksums in offline.checksums (/etc/llmgw/offline.sha256); llmgw bundle trust does that alone, for a mirror on another machine.
• offline.mirror in server.yaml makes the cluster reach nothing outside: nodes install llama.cpp from the mirror, each file held to the checksum the control plane sends them over the agent's authenticated connection — never one from the mirror — and releases defaults to the mirror's copy, so node add, ctl upgrade and agents upgrading fetch from it; no registry is asked about newer weights. A release may now come over plain HTTP when the build trusts a signing key, since the signature holds whatever carries it. vLLM and SGLang come from PyPI and are said to be unsupported on an offline node rather than attempted.
• llmgw server init --bundle PATH --bundle-sha256 …, and the same two fields on the setup form: installs a control plane offline — the bundle into the mirror, the checksums trusted, offline: written to server.yaml, the starter model and judge served from the bundle or skipped when it does not carry them.
• docs/data.md no longer says only weights and ctl plan reach outside: it lists everything that does, and what an offline cluster changes.
• A Helm chart for the control plane (deploy/helm/llmgw): one replica in a StatefulSet, SQLite on a claim or an external PostgreSQL, the agent port's certificate and the admin token from Secrets, the API and metrics Service, a LoadBalancer for the agents' mutual TLS, an optional Ingress and ServiceMonitor, config laid into server.yaml, and the offline checksums from a ConfigMap; read-only root, non-root, ready on /leader. deploy/container/Dockerfile builds the image it runs. GPU nodes stay outside Kubernetes and join with node add (docs/kubernetes.md says why).

BACKLOG

Done (branch features-c2): offline bundles — llmgw bundle create (releases with signed checksums, every llama.cpp build per arch/variant with GitHub's checksums, starter/judge and named Hugging Face model files pinned and checked; directory or tar; manifest sha256), bundle import (checked, all-or-nothing, into the mirror under _llmgw/), bundle trust, bundle show --verify; offline.mirror/offline.checksums in server.yaml, with nodes installing llama.cpp from the mirror against checksums the control plane sends over the agent stream (proto EngineWant.offline_*), releases defaulting to the mirror for node add, ctl upgrade and agent upgrades, update checks off; server init --bundle and the setup form's bundle fields; docs/data.md's offline claim corrected; install.md "Offline sites". A Helm chart for the control plane (deploy/helm/llmgw, one replica, SQLite PVC or external PostgreSQL, Secrets, Services, Ingress, ServiceMonitor) with deploy/container/Dockerfile and docs/kubernetes.md; GPU nodes stay outside Kubernetes by design. Left: vLLM/SGLang offline (a wheelhouse of several GB per CUDA build and a uv --find-links install path); the NVML release binary and uv in bundles; the setup wizard's preflight still fails a machine without internet; join's printed curl from the mirror checks nothing; models in a bundle are single files (no whole-repository safetensors sets yet); the chart runs one replica (a standby is never ready, so HA in Kubernetes needs a different readiness design), has not been installed on a real cluster, and ships no agent DaemonSet; no published container image.