Merge C18 offline bundles and the Helm chart
Conflicts: server.go imports (both kept); agent.pb.go regenerated with buf from the merged proto. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
commit
6f85c17fc5
51 files changed
+4177
-63
No files matched your search
+135
@@ -1861,6 +1861,139 @@ doctor` on a node says when the table is missing. `--no-firewall` on any of
|
||||
the four commands leaves the firewall alone; `sudo systemctl disable --now
|
||||
llmgw-firewall` removes it.
|
||||
|
||||
## Offline sites
|
||||
|
||||
Everything above downloads: the nodes fetch llama.cpp from GitHub and vLLM from
|
||||
PyPI, `node add` and upgrades fetch llmgw's releases, and the mirror fetches
|
||||
models from Hugging Face. A site with no route to the internet gets all of that
|
||||
in a **bundle**, made somewhere that has one and carried in.
|
||||
|
||||
### Making the bundle
|
||||
|
||||
On any machine with internet and llmgw — a laptop will do:
|
||||
|
||||
```
|
||||
llmgw bundle create --out llmgw-bundle --arch amd64 --tar
|
||||
```
|
||||
|
||||
That carries:
|
||||
|
||||
- **llmgw's release** for each `--arch` (default: this llmgw's own version;
|
||||
`--version` for another), with `checksums.txt` and its signature, checked
|
||||
against the project's signing key as `ctl upgrade` checks it.
|
||||
- **llama.cpp** at the version this llmgw installs, every build for each
|
||||
architecture — CPU, CUDA 13.3, CUDA 12.8 (amd64), Vulkan — each checked
|
||||
against the checksum GitHub publishes, the same check a node does online.
|
||||
`--variant cpu --variant cuda-13.3` carries only those, for a site whose
|
||||
hardware you know; `--engine llamacpp=b10991` names the version.
|
||||
- **The starter model and the guardrails' judge** that `server init` serves
|
||||
(about 1.3 GB; `--no-starter-models` leaves them out), and any
|
||||
`--model name/version=hf:owner/repo/file` you add, pinned to the commit the
|
||||
registry had and checked against the checksum it publishes.
|
||||
|
||||
It ends by printing the manifest's **sha256**. Carry that apart from the bundle
|
||||
— in the ticket, read out over the phone — because it is what says the bundle
|
||||
that arrived is the one that was made. `llmgw bundle show llmgw-bundle.tar
|
||||
--verify` reads every file back, worth doing once it is on the disk that
|
||||
carries it. Run `create` again with the same `--out` (without `--tar`) to add
|
||||
to a bundle; what is there already is not downloaded again.
|
||||
|
||||
What a bundle does **not** carry: vLLM and SGLang, which are Python
|
||||
environments of several gigabytes from PyPI. Offline nodes serve with llama.cpp;
|
||||
on a GPU node that needs vLLM, install it by hand and name it with `node add
|
||||
--vllm-bin` (the node reports vLLM as unsupported until then). Nor the
|
||||
operating system's packages — the OpenMP runtime llama.cpp needs, PostgreSQL
|
||||
with `--postgres` — which come from each machine's own package sources: at an
|
||||
offline site, its distribution mirror.
|
||||
|
||||
### Installing from it
|
||||
|
||||
On the control plane, with the bundle on a local disk:
|
||||
|
||||
```
|
||||
sudo llmgw server init --bundle /media/usb/llmgw-bundle.tar --bundle-sha256 3f2a…
|
||||
```
|
||||
|
||||
(The console's setup form has the same two fields.) Init then does what it
|
||||
always does, with these differences:
|
||||
|
||||
1. Before the control plane starts, the bundle goes into the mirror beside it:
|
||||
every file checked against the manifest, the release against the signing
|
||||
key, and only then put in place. llmgw's releases and the engines go under
|
||||
`/srv/models/_llmgw/`, the models where the mirror keeps models.
|
||||
2. The engines' checksums are written to `/etc/llmgw/offline.sha256`. That
|
||||
file, on the control plane, is what each node's engine files are held to.
|
||||
3. `server.yaml` gets
|
||||
|
||||
```yaml
|
||||
offline:
|
||||
mirror: http://10.0.0.5:8081 # the mirror's file port, as nodes reach it
|
||||
```
|
||||
|
||||
and from then on the cluster asks nothing of the internet: nodes install
|
||||
llama.cpp from the mirror, `releases` defaults to the mirror's copy (so
|
||||
`node add` and `ctl upgrade` fetch from it), and the registry is not asked
|
||||
about newer weights.
|
||||
4. The starter model and the judge are served from the mirror when the bundle
|
||||
carries them, and skipped — not waited for — when it does not. Without the
|
||||
judge, the guard policy is written without one.
|
||||
|
||||
With the mirror on another machine (`--mirror`), import the bundle there first
|
||||
and then run init with `--bundle` as above: the control plane records only the
|
||||
checksums.
|
||||
|
||||
```
|
||||
# on the mirror machine
|
||||
sudo llmgw bundle import /media/usb/llmgw-bundle.tar --sha256 3f2a…
|
||||
```
|
||||
|
||||
Nodes join as usual with `sudo llmgw node add`: it fetches the node's binary
|
||||
from the mirror and checks its signature, and the agent installs llama.cpp
|
||||
from the mirror. Nothing on the node needs the internet.
|
||||
|
||||
### How it stays checked
|
||||
|
||||
Online, a node trusts GitHub, over TLS, for the checksum of the llama.cpp it
|
||||
installs, and the project's signing key for llmgw. Offline it is the same
|
||||
strength by a different route: the checksum GitHub published is recorded when
|
||||
the bundle is made, the bundle is held to its sha256 when it is imported, and
|
||||
the control plane keeps the checksums and tells each node, over its mutually
|
||||
authenticated connection, what every file it fetches must be. The mirror serves
|
||||
plain HTTP and is never asked for a checksum: a file changed on it, or on the
|
||||
way from it, is refused by the node. Releases carry the project's signature,
|
||||
which holds whatever server they come from — so plain HTTP from the mirror is
|
||||
allowed for releases only because this build trusts a signing key.
|
||||
|
||||
### Later
|
||||
|
||||
A new llmgw release or llama.cpp version is another bundle. Make it the same
|
||||
way — `--version v0.12.0`, `--engine llamacpp=b11200` — and on the control plane
|
||||
(and the mirror, if separate):
|
||||
|
||||
```
|
||||
sudo llmgw bundle import new-bundle.tar --sha256 …
|
||||
```
|
||||
|
||||
The control plane sees the new checksums without a restart. Then set
|
||||
`engines.llamacpp` in `server.yaml` to the new version and restart
|
||||
`llmgw-server`, or run `sudo llmgw ctl upgrade v0.12.0` for the release. A
|
||||
mirror on its own machine, which is not an agent, is upgraded from its own
|
||||
copy: `sudo llmgw upgrade v0.12.0 --service llmgw-mirror --releases
|
||||
http://127.0.0.1:8081/_llmgw/releases`. A control plane whose bundle lacks the
|
||||
llama.cpp version it asks for says so in its log at start, and the node says
|
||||
which `bundle create` would carry it. More models: `--model` in a bundle, or
|
||||
copy the files into the mirror by hand — it is a directory.
|
||||
|
||||
A cluster set up online becomes offline the same way: `bundle import` on the
|
||||
mirror and control plane, then the `offline:` block in `server.yaml` and a
|
||||
restart.
|
||||
|
||||
## Kubernetes
|
||||
|
||||
The control plane runs in Kubernetes from the Helm chart in
|
||||
`deploy/helm/llmgw`; GPU nodes are best kept outside it, as ordinary hosts
|
||||
joined with `node add`. Why, and how, is in [kubernetes.md](kubernetes.md).
|
||||
|
||||
## Upgrading
|
||||
|
||||
On the control plane:
|
||||
@@ -2132,6 +2265,8 @@ requests. On a slow model, set `timeouts.generation` higher.
|
||||
wrote about
|
||||
- `deploy/mirror/README.md` — the mirror in more detail, including the layout
|
||||
and what belongs on which port
|
||||
- `docs/kubernetes.md` — the control plane in Kubernetes from the Helm chart,
|
||||
and why GPU nodes stay outside it
|
||||
- `deploy/monitoring/README.md` — Prometheus, Grafana and the alerts, with what
|
||||
each one assumes about this system
|
||||
- `docs/fabrics.md` — InfiniBand fabrics and NVLink: what llmgw needs of them,
|
||||
|
||||
Reference in new issue
Block a user