Merge C18 offline bundles and the Helm chart

Conflicts: server.go imports (both kept); agent.pb.go regenerated with buf
from the merged proto.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-05 00:15:47 +02:00
commit 6f85c17fc5
51 files changed
+4177 -63

No files matched your search

+135
View File
@@ -1861,6 +1861,139 @@ doctor` on a node says when the table is missing. `--no-firewall` on any of
the four commands leaves the firewall alone; `sudo systemctl disable --now
llmgw-firewall` removes it.
## Offline sites
Everything above downloads: the nodes fetch llama.cpp from GitHub and vLLM from
PyPI, `node add` and upgrades fetch llmgw's releases, and the mirror fetches
models from Hugging Face. A site with no route to the internet gets all of that
in a **bundle**, made somewhere that has one and carried in.
### Making the bundle
On any machine with internet and llmgw — a laptop will do:
```
llmgw bundle create --out llmgw-bundle --arch amd64 --tar
```
That carries:
- **llmgw's release** for each `--arch` (default: this llmgw's own version;
`--version` for another), with `checksums.txt` and its signature, checked
against the project's signing key as `ctl upgrade` checks it.
- **llama.cpp** at the version this llmgw installs, every build for each
architecture — CPU, CUDA 13.3, CUDA 12.8 (amd64), Vulkan — each checked
against the checksum GitHub publishes, the same check a node does online.
`--variant cpu --variant cuda-13.3` carries only those, for a site whose
hardware you know; `--engine llamacpp=b10991` names the version.
- **The starter model and the guardrails' judge** that `server init` serves
(about 1.3 GB; `--no-starter-models` leaves them out), and any
`--model name/version=hf:owner/repo/file` you add, pinned to the commit the
registry had and checked against the checksum it publishes.
It ends by printing the manifest's **sha256**. Carry that apart from the bundle
— in the ticket, read out over the phone — because it is what says the bundle
that arrived is the one that was made. `llmgw bundle show llmgw-bundle.tar
--verify` reads every file back, worth doing once it is on the disk that
carries it. Run `create` again with the same `--out` (without `--tar`) to add
to a bundle; what is there already is not downloaded again.
What a bundle does **not** carry: vLLM and SGLang, which are Python
environments of several gigabytes from PyPI. Offline nodes serve with llama.cpp;
on a GPU node that needs vLLM, install it by hand and name it with `node add
--vllm-bin` (the node reports vLLM as unsupported until then). Nor the
operating system's packages — the OpenMP runtime llama.cpp needs, PostgreSQL
with `--postgres` — which come from each machine's own package sources: at an
offline site, its distribution mirror.
### Installing from it
On the control plane, with the bundle on a local disk:
```
sudo llmgw server init --bundle /media/usb/llmgw-bundle.tar --bundle-sha256 3f2a…
```
(The console's setup form has the same two fields.) Init then does what it
always does, with these differences:
1. Before the control plane starts, the bundle goes into the mirror beside it:
every file checked against the manifest, the release against the signing
key, and only then put in place. llmgw's releases and the engines go under
`/srv/models/_llmgw/`, the models where the mirror keeps models.
2. The engines' checksums are written to `/etc/llmgw/offline.sha256`. That
file, on the control plane, is what each node's engine files are held to.
3. `server.yaml` gets
```yaml
offline:
mirror: http://10.0.0.5:8081 # the mirror's file port, as nodes reach it
```
and from then on the cluster asks nothing of the internet: nodes install
llama.cpp from the mirror, `releases` defaults to the mirror's copy (so
`node add` and `ctl upgrade` fetch from it), and the registry is not asked
about newer weights.
4. The starter model and the judge are served from the mirror when the bundle
carries them, and skipped — not waited for — when it does not. Without the
judge, the guard policy is written without one.
With the mirror on another machine (`--mirror`), import the bundle there first
and then run init with `--bundle` as above: the control plane records only the
checksums.
```
# on the mirror machine
sudo llmgw bundle import /media/usb/llmgw-bundle.tar --sha256 3f2a…
```
Nodes join as usual with `sudo llmgw node add`: it fetches the node's binary
from the mirror and checks its signature, and the agent installs llama.cpp
from the mirror. Nothing on the node needs the internet.
### How it stays checked
Online, a node trusts GitHub, over TLS, for the checksum of the llama.cpp it
installs, and the project's signing key for llmgw. Offline it is the same
strength by a different route: the checksum GitHub published is recorded when
the bundle is made, the bundle is held to its sha256 when it is imported, and
the control plane keeps the checksums and tells each node, over its mutually
authenticated connection, what every file it fetches must be. The mirror serves
plain HTTP and is never asked for a checksum: a file changed on it, or on the
way from it, is refused by the node. Releases carry the project's signature,
which holds whatever server they come from — so plain HTTP from the mirror is
allowed for releases only because this build trusts a signing key.
### Later
A new llmgw release or llama.cpp version is another bundle. Make it the same
way — `--version v0.12.0`, `--engine llamacpp=b11200` — and on the control plane
(and the mirror, if separate):
```
sudo llmgw bundle import new-bundle.tar --sha256 …
```
The control plane sees the new checksums without a restart. Then set
`engines.llamacpp` in `server.yaml` to the new version and restart
`llmgw-server`, or run `sudo llmgw ctl upgrade v0.12.0` for the release. A
mirror on its own machine, which is not an agent, is upgraded from its own
copy: `sudo llmgw upgrade v0.12.0 --service llmgw-mirror --releases
http://127.0.0.1:8081/_llmgw/releases`. A control plane whose bundle lacks the
llama.cpp version it asks for says so in its log at start, and the node says
which `bundle create` would carry it. More models: `--model` in a bundle, or
copy the files into the mirror by hand — it is a directory.
A cluster set up online becomes offline the same way: `bundle import` on the
mirror and control plane, then the `offline:` block in `server.yaml` and a
restart.
## Kubernetes
The control plane runs in Kubernetes from the Helm chart in
`deploy/helm/llmgw`; GPU nodes are best kept outside it, as ordinary hosts
joined with `node add`. Why, and how, is in [kubernetes.md](kubernetes.md).
## Upgrading
On the control plane:
@@ -2132,6 +2265,8 @@ requests. On a slow model, set `timeouts.generation` higher.
wrote about
- `deploy/mirror/README.md` — the mirror in more detail, including the layout
and what belongs on which port
- `docs/kubernetes.md` — the control plane in Kubernetes from the Helm chart,
and why GPU nodes stay outside it
- `deploy/monitoring/README.md` — Prometheus, Grafana and the alerts, with what
each one assumes about this system
- `docs/fabrics.md` — InfiniBand fabrics and NVLink: what llmgw needs of them,