Merge C18 offline bundles and the Helm chart
Conflicts: server.go imports (both kept); agent.pb.go regenerated with buf from the merged proto. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
commit
6f85c17fc5
51 files changed
+4177
-63
No files matched your search
@@ -0,0 +1,78 @@
|
||||
# C18 — Kubernetes/Helm and an air-gapped bundle
|
||||
|
||||
README's documentation list should gain a line for `docs/kubernetes.md`
|
||||
(the control plane in Kubernetes from the Helm chart, and why GPU nodes stay
|
||||
outside it) — README.md was not edited here.
|
||||
|
||||
## CHANGELOG
|
||||
|
||||
### Offline sites and Kubernetes
|
||||
|
||||
- **Offline bundles.** `llmgw bundle create` (on any machine with internet)
|
||||
collects what a cluster would otherwise download — llmgw's release for
|
||||
each `--arch` with its signed checksums, every llama.cpp build for those
|
||||
architectures (or the `--variant` ones) with the checksums GitHub
|
||||
publishes, the starter model and guardrails' judge, and any `--model
|
||||
name/version=hf:owner/repo/file` pinned to the registry's commit and its
|
||||
checksum — each checked as it arrives, into a directory or one `.tar`
|
||||
(`--tar`), with a manifest whose sha256 it prints to carry apart from the
|
||||
bundle. Run again into the same directory, it downloads only what is
|
||||
missing. `llmgw bundle show --verify` reads one back.
|
||||
- **`llmgw bundle import`** checks every file against the manifest and
|
||||
llmgw's release against the project's signing key, and only then puts them
|
||||
into the mirror — releases and engines under `_llmgw/`, models where the
|
||||
mirror keeps models, with their checksums — so a damaged bundle leaves the
|
||||
mirror as it was; a different file under a version already there is
|
||||
refused. On a control plane it also records the engines' checksums in
|
||||
`offline.checksums` (`/etc/llmgw/offline.sha256`); `llmgw bundle trust`
|
||||
does that alone, for a mirror on another machine.
|
||||
- **`offline.mirror` in server.yaml** makes the cluster reach nothing
|
||||
outside: nodes install llama.cpp from the mirror, each file held to the
|
||||
checksum the control plane sends them over the agent's authenticated
|
||||
connection — never one from the mirror — and `releases` defaults to the
|
||||
mirror's copy, so `node add`, `ctl upgrade` and agents upgrading fetch
|
||||
from it; no registry is asked about newer weights. A release may now come
|
||||
over plain HTTP when the build trusts a signing key, since the signature
|
||||
holds whatever carries it. vLLM and SGLang come from PyPI and are said to
|
||||
be unsupported on an offline node rather than attempted.
|
||||
- **`llmgw server init --bundle PATH --bundle-sha256 …`**, and the same two
|
||||
fields on the setup form: installs a control plane offline — the bundle
|
||||
into the mirror, the checksums trusted, `offline:` written to server.yaml,
|
||||
the starter model and judge served from the bundle or skipped when it does
|
||||
not carry them.
|
||||
- `docs/data.md` no longer says only weights and `ctl plan` reach outside:
|
||||
it lists everything that does, and what an offline cluster changes.
|
||||
- **A Helm chart for the control plane** (`deploy/helm/llmgw`): one replica
|
||||
in a StatefulSet, SQLite on a claim or an external PostgreSQL, the agent
|
||||
port's certificate and the admin token from Secrets, the API and metrics
|
||||
Service, a LoadBalancer for the agents' mutual TLS, an optional Ingress
|
||||
and ServiceMonitor, `config` laid into server.yaml, and the offline
|
||||
checksums from a ConfigMap; read-only root, non-root, ready on `/leader`.
|
||||
`deploy/container/Dockerfile` builds the image it runs. GPU nodes stay
|
||||
outside Kubernetes and join with `node add` (docs/kubernetes.md says why).
|
||||
|
||||
## BACKLOG
|
||||
|
||||
**Done (branch `features-c2`):** offline bundles — `llmgw bundle create`
|
||||
(releases with signed checksums, every llama.cpp build per arch/variant
|
||||
with GitHub's checksums, starter/judge and named Hugging Face model files
|
||||
pinned and checked; directory or tar; manifest sha256), `bundle import`
|
||||
(checked, all-or-nothing, into the mirror under `_llmgw/`), `bundle trust`,
|
||||
`bundle show --verify`; `offline.mirror`/`offline.checksums` in server.yaml,
|
||||
with nodes installing llama.cpp from the mirror against checksums the
|
||||
control plane sends over the agent stream (proto `EngineWant.offline_*`),
|
||||
releases defaulting to the mirror for `node add`, `ctl upgrade` and agent
|
||||
upgrades, update checks off; `server init --bundle` and the setup form's
|
||||
bundle fields; docs/data.md's offline claim corrected; install.md "Offline
|
||||
sites". A Helm chart for the control plane (`deploy/helm/llmgw`, one replica,
|
||||
SQLite PVC or external PostgreSQL, Secrets, Services, Ingress,
|
||||
ServiceMonitor) with `deploy/container/Dockerfile` and docs/kubernetes.md;
|
||||
GPU nodes stay outside Kubernetes by design. Left: vLLM/SGLang offline (a
|
||||
wheelhouse of several GB per CUDA build and a uv `--find-links` install
|
||||
path); the NVML release binary and uv in bundles; the setup wizard's
|
||||
preflight still fails a machine without internet; `join`'s printed `curl`
|
||||
from the mirror checks nothing; models in a bundle are single files
|
||||
(no whole-repository safetensors sets yet); the chart runs one replica (a
|
||||
standby is never ready, so HA in Kubernetes needs a different readiness
|
||||
design), has not been installed on a real cluster, and ships no agent
|
||||
DaemonSet; no published container image.
|
||||
Reference in new issue
Block a user