Merge C18 offline bundles and the Helm chart

Conflicts: server.go imports (both kept); agent.pb.go regenerated with buf
from the merged proto.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-05 00:15:47 +02:00
commit 6f85c17fc5
51 files changed
+4177 -63

No files matched your search

+50 -10
View File
@@ -126,22 +126,62 @@ your own hardware, and the answer comes back. The router talks to engines, the
engines talk to nothing, the agents talk to the control plane. All of it is
inside the cluster.
Two things can reach outside, both started by an operator rather than by a
request:
Around the serving path, a cluster set up the ordinary way does reach out.
None of it carries a prompt, an answer, a key or a user — it is software and
public model files coming in — but a review should know each one:
**Fetching weights.** When a model's `source` is a URL, the agent downloads it.
Point that at an internal mirror and nothing leaves at all — which is also the
**Installing engines.** Each node's agent downloads the llama.cpp build for its
hardware from GitHub, checked against the checksum GitHub publishes, and vLLM
(SGLang when asked) from PyPI, with `uv` from GitHub. On every node, when it
joins and when the engine version changes.
**Releases.** `llmgw node add`, `llmgw ctl upgrade` and `llmgw upgrade` fetch
llmgw's own release binaries from the project's release server, checked against
its signed checksums.
**Fetching weights.** When a model's `source` is a URL, the agent downloads it;
the mirror fetches what it is asked to, from Hugging Face or another URL, and
`server init` has it fetch the starter model and the guardrails' judge. Point
sources at the mirror and the nodes fetch only from it — which is also the
faster arrangement, because eight nodes pulling forty gigabytes from the
internet is a day's work and from a local mirror it is minutes. A `source` that
is a path on disk is never fetched.
**`llmgw ctl plan`.** Reads a model's architecture from huggingface.co to work
out whether it fits your hardware. An operator command run at planning time,
not part of serving, and not run by anything automatically.
**The registry.** With a mirror configured, the control plane asks Hugging Face
every six hours whether the models served from it have newer files
(`mirror.update_check`), and `/admin/mirror` searches it. `llmgw ctl plan`
reads a model's architecture from huggingface.co to work out whether it fits
your hardware.
There is no telemetry, no licence check, and no call home. The binary is
static, built with `CGO_ENABLED=0`, and runs in a network with no route to the
internet as long as the weights are already on disk.
There is no telemetry, no licence check, and no call home.
### A cluster with no way out
A site with no route to the internet installs from an **offline bundle**
instead, and then nothing at all leaves the cluster (docs/install.md, "Offline
sites"). The bundle is made on a machine that has internet — llmgw's release
binaries with their signed checksums, the llama.cpp builds with the checksums
GitHub publishes, the model files with the checksums Hugging Face publishes,
each checked as it is downloaded — carried in, and imported into the mirror.
With `offline.mirror` set in `server.yaml`, nodes install engines from the
mirror and never from GitHub, releases come from the mirror, and the registry
is not asked about newer weights.
Verification is not weakened by any of this. A release is held to the project's
signing key wherever it is fetched from. An engine file is held to the checksum
recorded when the bundle was made, which the control plane keeps and hands each
node over the agent's authenticated connection: the mirror serves plain HTTP,
and nothing a node runs is taken on the mirror's word. The bundle itself is
checked against the sha256 its maker printed, carried to the site apart from
the bundle.
What does not come in a bundle: vLLM and SGLang, which are Python environments
from PyPI — an offline node serves with llama.cpp unless one is installed on it
by hand — and the operating system's own packages (the OpenMP runtime the
engine needs, PostgreSQL when chosen), which come from the machines' package
sources: at an offline site, the distribution mirror it already has. Searching
the registry from `/admin/mirror` and `llmgw ctl plan` need the internet and
do not work offline; `mirror.registry` can point the first at an internal copy.
## What the software cannot do for you