A bundle carries what a cluster would download — llmgw's release with its signed checksums, every llama.cpp build per architecture with GitHub's checksums, model files pinned and checked against Hugging Face's — into a directory or a tar whose manifest sha256 is carried apart. Import checks everything, the release against the signing key, before anything is put in the mirror. With offline.mirror set, nodes install llama.cpp from the mirror against checksums the control plane holds and sends over the agent stream; releases default to the mirror for node add, ctl upgrade and agent upgrades. server init and the setup form install from a bundle. The Helm chart runs one control plane with SQLite on a claim or an external PostgreSQL, Secrets for the agent certificate and admin token, Services, Ingress and ServiceMonitor; deploy/container/Dockerfile builds its image. GPU nodes stay outside Kubernetes, as docs/kubernetes.md explains. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
79 lines
4.8 KiB
Markdown
79 lines
4.8 KiB
Markdown
# C18 — Kubernetes/Helm and an air-gapped bundle
|
|
|
|
README's documentation list should gain a line for `docs/kubernetes.md`
|
|
(the control plane in Kubernetes from the Helm chart, and why GPU nodes stay
|
|
outside it) — README.md was not edited here.
|
|
|
|
## CHANGELOG
|
|
|
|
### Offline sites and Kubernetes
|
|
|
|
- **Offline bundles.** `llmgw bundle create` (on any machine with internet)
|
|
collects what a cluster would otherwise download — llmgw's release for
|
|
each `--arch` with its signed checksums, every llama.cpp build for those
|
|
architectures (or the `--variant` ones) with the checksums GitHub
|
|
publishes, the starter model and guardrails' judge, and any `--model
|
|
name/version=hf:owner/repo/file` pinned to the registry's commit and its
|
|
checksum — each checked as it arrives, into a directory or one `.tar`
|
|
(`--tar`), with a manifest whose sha256 it prints to carry apart from the
|
|
bundle. Run again into the same directory, it downloads only what is
|
|
missing. `llmgw bundle show --verify` reads one back.
|
|
- **`llmgw bundle import`** checks every file against the manifest and
|
|
llmgw's release against the project's signing key, and only then puts them
|
|
into the mirror — releases and engines under `_llmgw/`, models where the
|
|
mirror keeps models, with their checksums — so a damaged bundle leaves the
|
|
mirror as it was; a different file under a version already there is
|
|
refused. On a control plane it also records the engines' checksums in
|
|
`offline.checksums` (`/etc/llmgw/offline.sha256`); `llmgw bundle trust`
|
|
does that alone, for a mirror on another machine.
|
|
- **`offline.mirror` in server.yaml** makes the cluster reach nothing
|
|
outside: nodes install llama.cpp from the mirror, each file held to the
|
|
checksum the control plane sends them over the agent's authenticated
|
|
connection — never one from the mirror — and `releases` defaults to the
|
|
mirror's copy, so `node add`, `ctl upgrade` and agents upgrading fetch
|
|
from it; no registry is asked about newer weights. A release may now come
|
|
over plain HTTP when the build trusts a signing key, since the signature
|
|
holds whatever carries it. vLLM and SGLang come from PyPI and are said to
|
|
be unsupported on an offline node rather than attempted.
|
|
- **`llmgw server init --bundle PATH --bundle-sha256 …`**, and the same two
|
|
fields on the setup form: installs a control plane offline — the bundle
|
|
into the mirror, the checksums trusted, `offline:` written to server.yaml,
|
|
the starter model and judge served from the bundle or skipped when it does
|
|
not carry them.
|
|
- `docs/data.md` no longer says only weights and `ctl plan` reach outside:
|
|
it lists everything that does, and what an offline cluster changes.
|
|
- **A Helm chart for the control plane** (`deploy/helm/llmgw`): one replica
|
|
in a StatefulSet, SQLite on a claim or an external PostgreSQL, the agent
|
|
port's certificate and the admin token from Secrets, the API and metrics
|
|
Service, a LoadBalancer for the agents' mutual TLS, an optional Ingress
|
|
and ServiceMonitor, `config` laid into server.yaml, and the offline
|
|
checksums from a ConfigMap; read-only root, non-root, ready on `/leader`.
|
|
`deploy/container/Dockerfile` builds the image it runs. GPU nodes stay
|
|
outside Kubernetes and join with `node add` (docs/kubernetes.md says why).
|
|
|
|
## BACKLOG
|
|
|
|
**Done (branch `features-c2`):** offline bundles — `llmgw bundle create`
|
|
(releases with signed checksums, every llama.cpp build per arch/variant
|
|
with GitHub's checksums, starter/judge and named Hugging Face model files
|
|
pinned and checked; directory or tar; manifest sha256), `bundle import`
|
|
(checked, all-or-nothing, into the mirror under `_llmgw/`), `bundle trust`,
|
|
`bundle show --verify`; `offline.mirror`/`offline.checksums` in server.yaml,
|
|
with nodes installing llama.cpp from the mirror against checksums the
|
|
control plane sends over the agent stream (proto `EngineWant.offline_*`),
|
|
releases defaulting to the mirror for `node add`, `ctl upgrade` and agent
|
|
upgrades, update checks off; `server init --bundle` and the setup form's
|
|
bundle fields; docs/data.md's offline claim corrected; install.md "Offline
|
|
sites". A Helm chart for the control plane (`deploy/helm/llmgw`, one replica,
|
|
SQLite PVC or external PostgreSQL, Secrets, Services, Ingress,
|
|
ServiceMonitor) with `deploy/container/Dockerfile` and docs/kubernetes.md;
|
|
GPU nodes stay outside Kubernetes by design. Left: vLLM/SGLang offline (a
|
|
wheelhouse of several GB per CUDA build and a uv `--find-links` install
|
|
path); the NVML release binary and uv in bundles; the setup wizard's
|
|
preflight still fails a machine without internet; `join`'s printed `curl`
|
|
from the mirror checks nothing; models in a bundle are single files
|
|
(no whole-repository safetensors sets yet); the chart runs one replica (a
|
|
standby is never ready, so HA in Kubernetes needs a different readiness
|
|
design), has not been installed on a real cluster, and ships no agent
|
|
DaemonSet; no published container image.
|