Files
llmgw/notes/c18.md
T
Ilker ManapandClaude Opus 5.5 bae16426a2 Offline sites and Kubernetes: llmgw bundle create, import and trust, offline.mirror, server init --bundle; a Helm chart for the control plane
A bundle carries what a cluster would download — llmgw's release with its
signed checksums, every llama.cpp build per architecture with GitHub's
checksums, model files pinned and checked against Hugging Face's — into a
directory or a tar whose manifest sha256 is carried apart. Import checks
everything, the release against the signing key, before anything is put in
the mirror. With offline.mirror set, nodes install llama.cpp from the
mirror against checksums the control plane holds and sends over the agent
stream; releases default to the mirror for node add, ctl upgrade and agent
upgrades. server init and the setup form install from a bundle.

The Helm chart runs one control plane with SQLite on a claim or an external
PostgreSQL, Secrets for the agent certificate and admin token, Services,
Ingress and ServiceMonitor; deploy/container/Dockerfile builds its image.
GPU nodes stay outside Kubernetes, as docs/kubernetes.md explains.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 00:13:39 +02:00

79 lines
4.8 KiB
Markdown

# C18 — Kubernetes/Helm and an air-gapped bundle
README's documentation list should gain a line for `docs/kubernetes.md`
(the control plane in Kubernetes from the Helm chart, and why GPU nodes stay
outside it) — README.md was not edited here.
## CHANGELOG
### Offline sites and Kubernetes
- **Offline bundles.** `llmgw bundle create` (on any machine with internet)
collects what a cluster would otherwise download — llmgw's release for
each `--arch` with its signed checksums, every llama.cpp build for those
architectures (or the `--variant` ones) with the checksums GitHub
publishes, the starter model and guardrails' judge, and any `--model
name/version=hf:owner/repo/file` pinned to the registry's commit and its
checksum — each checked as it arrives, into a directory or one `.tar`
(`--tar`), with a manifest whose sha256 it prints to carry apart from the
bundle. Run again into the same directory, it downloads only what is
missing. `llmgw bundle show --verify` reads one back.
- **`llmgw bundle import`** checks every file against the manifest and
llmgw's release against the project's signing key, and only then puts them
into the mirror — releases and engines under `_llmgw/`, models where the
mirror keeps models, with their checksums — so a damaged bundle leaves the
mirror as it was; a different file under a version already there is
refused. On a control plane it also records the engines' checksums in
`offline.checksums` (`/etc/llmgw/offline.sha256`); `llmgw bundle trust`
does that alone, for a mirror on another machine.
- **`offline.mirror` in server.yaml** makes the cluster reach nothing
outside: nodes install llama.cpp from the mirror, each file held to the
checksum the control plane sends them over the agent's authenticated
connection — never one from the mirror — and `releases` defaults to the
mirror's copy, so `node add`, `ctl upgrade` and agents upgrading fetch
from it; no registry is asked about newer weights. A release may now come
over plain HTTP when the build trusts a signing key, since the signature
holds whatever carries it. vLLM and SGLang come from PyPI and are said to
be unsupported on an offline node rather than attempted.
- **`llmgw server init --bundle PATH --bundle-sha256 …`**, and the same two
fields on the setup form: installs a control plane offline — the bundle
into the mirror, the checksums trusted, `offline:` written to server.yaml,
the starter model and judge served from the bundle or skipped when it does
not carry them.
- `docs/data.md` no longer says only weights and `ctl plan` reach outside:
it lists everything that does, and what an offline cluster changes.
- **A Helm chart for the control plane** (`deploy/helm/llmgw`): one replica
in a StatefulSet, SQLite on a claim or an external PostgreSQL, the agent
port's certificate and the admin token from Secrets, the API and metrics
Service, a LoadBalancer for the agents' mutual TLS, an optional Ingress
and ServiceMonitor, `config` laid into server.yaml, and the offline
checksums from a ConfigMap; read-only root, non-root, ready on `/leader`.
`deploy/container/Dockerfile` builds the image it runs. GPU nodes stay
outside Kubernetes and join with `node add` (docs/kubernetes.md says why).
## BACKLOG
**Done (branch `features-c2`):** offline bundles — `llmgw bundle create`
(releases with signed checksums, every llama.cpp build per arch/variant
with GitHub's checksums, starter/judge and named Hugging Face model files
pinned and checked; directory or tar; manifest sha256), `bundle import`
(checked, all-or-nothing, into the mirror under `_llmgw/`), `bundle trust`,
`bundle show --verify`; `offline.mirror`/`offline.checksums` in server.yaml,
with nodes installing llama.cpp from the mirror against checksums the
control plane sends over the agent stream (proto `EngineWant.offline_*`),
releases defaulting to the mirror for `node add`, `ctl upgrade` and agent
upgrades, update checks off; `server init --bundle` and the setup form's
bundle fields; docs/data.md's offline claim corrected; install.md "Offline
sites". A Helm chart for the control plane (`deploy/helm/llmgw`, one replica,
SQLite PVC or external PostgreSQL, Secrets, Services, Ingress,
ServiceMonitor) with `deploy/container/Dockerfile` and docs/kubernetes.md;
GPU nodes stay outside Kubernetes by design. Left: vLLM/SGLang offline (a
wheelhouse of several GB per CUDA build and a uv `--find-links` install
path); the NVML release binary and uv in bundles; the setup wizard's
preflight still fails a machine without internet; `join`'s printed `curl`
from the mirror checks nothing; models in a bundle are single files
(no whole-repository safetensors sets yet); the chart runs one replica (a
standby is never ready, so HA in Kubernetes needs a different readiness
design), has not been installed on a real cluster, and ships no agent
DaemonSet; no published container image.