Offline sites and Kubernetes: llmgw bundle create, import and trust, offline.mirror, server init --bundle; a Helm chart for the control plane
A bundle carries what a cluster would download — llmgw's release with its signed checksums, every llama.cpp build per architecture with GitHub's checksums, model files pinned and checked against Hugging Face's — into a directory or a tar whose manifest sha256 is carried apart. Import checks everything, the release against the signing key, before anything is put in the mirror. With offline.mirror set, nodes install llama.cpp from the mirror against checksums the control plane holds and sends over the agent stream; releases default to the mirror for node add, ctl upgrade and agent upgrades. server init and the setup form install from a bundle. The Helm chart runs one control plane with SQLite on a claim or an external PostgreSQL, Secrets for the agent certificate and admin token, Services, Ingress and ServiceMonitor; deploy/container/Dockerfile builds its image. GPU nodes stay outside Kubernetes, as docs/kubernetes.md explains. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
e59ff52932
commit
bae16426a2
51 files changed
+4177
-63
No files matched your search
+135
@@ -1245,6 +1245,139 @@ doctor` on a node says when the table is missing. `--no-firewall` on any of
|
||||
the four commands leaves the firewall alone; `sudo systemctl disable --now
|
||||
llmgw-firewall` removes it.
|
||||
|
||||
## Offline sites
|
||||
|
||||
Everything above downloads: the nodes fetch llama.cpp from GitHub and vLLM from
|
||||
PyPI, `node add` and upgrades fetch llmgw's releases, and the mirror fetches
|
||||
models from Hugging Face. A site with no route to the internet gets all of that
|
||||
in a **bundle**, made somewhere that has one and carried in.
|
||||
|
||||
### Making the bundle
|
||||
|
||||
On any machine with internet and llmgw — a laptop will do:
|
||||
|
||||
```
|
||||
llmgw bundle create --out llmgw-bundle --arch amd64 --tar
|
||||
```
|
||||
|
||||
That carries:
|
||||
|
||||
- **llmgw's release** for each `--arch` (default: this llmgw's own version;
|
||||
`--version` for another), with `checksums.txt` and its signature, checked
|
||||
against the project's signing key as `ctl upgrade` checks it.
|
||||
- **llama.cpp** at the version this llmgw installs, every build for each
|
||||
architecture — CPU, CUDA 13.3, CUDA 12.8 (amd64), Vulkan — each checked
|
||||
against the checksum GitHub publishes, the same check a node does online.
|
||||
`--variant cpu --variant cuda-13.3` carries only those, for a site whose
|
||||
hardware you know; `--engine llamacpp=b10991` names the version.
|
||||
- **The starter model and the guardrails' judge** that `server init` serves
|
||||
(about 1.3 GB; `--no-starter-models` leaves them out), and any
|
||||
`--model name/version=hf:owner/repo/file` you add, pinned to the commit the
|
||||
registry had and checked against the checksum it publishes.
|
||||
|
||||
It ends by printing the manifest's **sha256**. Carry that apart from the bundle
|
||||
— in the ticket, read out over the phone — because it is what says the bundle
|
||||
that arrived is the one that was made. `llmgw bundle show llmgw-bundle.tar
|
||||
--verify` reads every file back, worth doing once it is on the disk that
|
||||
carries it. Run `create` again with the same `--out` (without `--tar`) to add
|
||||
to a bundle; what is there already is not downloaded again.
|
||||
|
||||
What a bundle does **not** carry: vLLM and SGLang, which are Python
|
||||
environments of several gigabytes from PyPI. Offline nodes serve with llama.cpp;
|
||||
on a GPU node that needs vLLM, install it by hand and name it with `node add
|
||||
--vllm-bin` (the node reports vLLM as unsupported until then). Nor the
|
||||
operating system's packages — the OpenMP runtime llama.cpp needs, PostgreSQL
|
||||
with `--postgres` — which come from each machine's own package sources: at an
|
||||
offline site, its distribution mirror.
|
||||
|
||||
### Installing from it
|
||||
|
||||
On the control plane, with the bundle on a local disk:
|
||||
|
||||
```
|
||||
sudo llmgw server init --bundle /media/usb/llmgw-bundle.tar --bundle-sha256 3f2a…
|
||||
```
|
||||
|
||||
(The console's setup form has the same two fields.) Init then does what it
|
||||
always does, with these differences:
|
||||
|
||||
1. Before the control plane starts, the bundle goes into the mirror beside it:
|
||||
every file checked against the manifest, the release against the signing
|
||||
key, and only then put in place. llmgw's releases and the engines go under
|
||||
`/srv/models/_llmgw/`, the models where the mirror keeps models.
|
||||
2. The engines' checksums are written to `/etc/llmgw/offline.sha256`. That
|
||||
file, on the control plane, is what each node's engine files are held to.
|
||||
3. `server.yaml` gets
|
||||
|
||||
```yaml
|
||||
offline:
|
||||
mirror: http://10.0.0.5:8081 # the mirror's file port, as nodes reach it
|
||||
```
|
||||
|
||||
and from then on the cluster asks nothing of the internet: nodes install
|
||||
llama.cpp from the mirror, `releases` defaults to the mirror's copy (so
|
||||
`node add` and `ctl upgrade` fetch from it), and the registry is not asked
|
||||
about newer weights.
|
||||
4. The starter model and the judge are served from the mirror when the bundle
|
||||
carries them, and skipped — not waited for — when it does not. Without the
|
||||
judge, the guard policy is written without one.
|
||||
|
||||
With the mirror on another machine (`--mirror`), import the bundle there first
|
||||
and then run init with `--bundle` as above: the control plane records only the
|
||||
checksums.
|
||||
|
||||
```
|
||||
# on the mirror machine
|
||||
sudo llmgw bundle import /media/usb/llmgw-bundle.tar --sha256 3f2a…
|
||||
```
|
||||
|
||||
Nodes join as usual with `sudo llmgw node add`: it fetches the node's binary
|
||||
from the mirror and checks its signature, and the agent installs llama.cpp
|
||||
from the mirror. Nothing on the node needs the internet.
|
||||
|
||||
### How it stays checked
|
||||
|
||||
Online, a node trusts GitHub, over TLS, for the checksum of the llama.cpp it
|
||||
installs, and the project's signing key for llmgw. Offline it is the same
|
||||
strength by a different route: the checksum GitHub published is recorded when
|
||||
the bundle is made, the bundle is held to its sha256 when it is imported, and
|
||||
the control plane keeps the checksums and tells each node, over its mutually
|
||||
authenticated connection, what every file it fetches must be. The mirror serves
|
||||
plain HTTP and is never asked for a checksum: a file changed on it, or on the
|
||||
way from it, is refused by the node. Releases carry the project's signature,
|
||||
which holds whatever server they come from — so plain HTTP from the mirror is
|
||||
allowed for releases only because this build trusts a signing key.
|
||||
|
||||
### Later
|
||||
|
||||
A new llmgw release or llama.cpp version is another bundle. Make it the same
|
||||
way — `--version v0.12.0`, `--engine llamacpp=b11200` — and on the control plane
|
||||
(and the mirror, if separate):
|
||||
|
||||
```
|
||||
sudo llmgw bundle import new-bundle.tar --sha256 …
|
||||
```
|
||||
|
||||
The control plane sees the new checksums without a restart. Then set
|
||||
`engines.llamacpp` in `server.yaml` to the new version and restart
|
||||
`llmgw-server`, or run `sudo llmgw ctl upgrade v0.12.0` for the release. A
|
||||
mirror on its own machine, which is not an agent, is upgraded from its own
|
||||
copy: `sudo llmgw upgrade v0.12.0 --service llmgw-mirror --releases
|
||||
http://127.0.0.1:8081/_llmgw/releases`. A control plane whose bundle lacks the
|
||||
llama.cpp version it asks for says so in its log at start, and the node says
|
||||
which `bundle create` would carry it. More models: `--model` in a bundle, or
|
||||
copy the files into the mirror by hand — it is a directory.
|
||||
|
||||
A cluster set up online becomes offline the same way: `bundle import` on the
|
||||
mirror and control plane, then the `offline:` block in `server.yaml` and a
|
||||
restart.
|
||||
|
||||
## Kubernetes
|
||||
|
||||
The control plane runs in Kubernetes from the Helm chart in
|
||||
`deploy/helm/llmgw`; GPU nodes are best kept outside it, as ordinary hosts
|
||||
joined with `node add`. Why, and how, is in [kubernetes.md](kubernetes.md).
|
||||
|
||||
## Upgrading
|
||||
|
||||
On the control plane:
|
||||
@@ -1488,6 +1621,8 @@ requests. On a slow model, set `timeouts.generation` higher.
|
||||
wrote about
|
||||
- `deploy/mirror/README.md` — the mirror in more detail, including the layout
|
||||
and what belongs on which port
|
||||
- `docs/kubernetes.md` — the control plane in Kubernetes from the Helm chart,
|
||||
and why GPU nodes stay outside it
|
||||
- `deploy/monitoring/README.md` — Prometheus, Grafana and the alerts, with what
|
||||
each one assumes about this system
|
||||
- `docs/fabrics.md` — InfiniBand fabrics and NVLink: what llmgw needs of them,
|
||||
|
||||
Reference in new issue
Block a user