Merge C18 offline bundles and the Helm chart
Conflicts: server.go imports (both kept); agent.pb.go regenerated with buf from the merged proto. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
commit
6f85c17fc5
51 files changed
+4177
-63
No files matched your search
+50
-10
@@ -126,22 +126,62 @@ your own hardware, and the answer comes back. The router talks to engines, the
|
||||
engines talk to nothing, the agents talk to the control plane. All of it is
|
||||
inside the cluster.
|
||||
|
||||
Two things can reach outside, both started by an operator rather than by a
|
||||
request:
|
||||
Around the serving path, a cluster set up the ordinary way does reach out.
|
||||
None of it carries a prompt, an answer, a key or a user — it is software and
|
||||
public model files coming in — but a review should know each one:
|
||||
|
||||
**Fetching weights.** When a model's `source` is a URL, the agent downloads it.
|
||||
Point that at an internal mirror and nothing leaves at all — which is also the
|
||||
**Installing engines.** Each node's agent downloads the llama.cpp build for its
|
||||
hardware from GitHub, checked against the checksum GitHub publishes, and vLLM
|
||||
(SGLang when asked) from PyPI, with `uv` from GitHub. On every node, when it
|
||||
joins and when the engine version changes.
|
||||
|
||||
**Releases.** `llmgw node add`, `llmgw ctl upgrade` and `llmgw upgrade` fetch
|
||||
llmgw's own release binaries from the project's release server, checked against
|
||||
its signed checksums.
|
||||
|
||||
**Fetching weights.** When a model's `source` is a URL, the agent downloads it;
|
||||
the mirror fetches what it is asked to, from Hugging Face or another URL, and
|
||||
`server init` has it fetch the starter model and the guardrails' judge. Point
|
||||
sources at the mirror and the nodes fetch only from it — which is also the
|
||||
faster arrangement, because eight nodes pulling forty gigabytes from the
|
||||
internet is a day's work and from a local mirror it is minutes. A `source` that
|
||||
is a path on disk is never fetched.
|
||||
|
||||
**`llmgw ctl plan`.** Reads a model's architecture from huggingface.co to work
|
||||
out whether it fits your hardware. An operator command run at planning time,
|
||||
not part of serving, and not run by anything automatically.
|
||||
**The registry.** With a mirror configured, the control plane asks Hugging Face
|
||||
every six hours whether the models served from it have newer files
|
||||
(`mirror.update_check`), and `/admin/mirror` searches it. `llmgw ctl plan`
|
||||
reads a model's architecture from huggingface.co to work out whether it fits
|
||||
your hardware.
|
||||
|
||||
There is no telemetry, no licence check, and no call home. The binary is
|
||||
static, built with `CGO_ENABLED=0`, and runs in a network with no route to the
|
||||
internet as long as the weights are already on disk.
|
||||
There is no telemetry, no licence check, and no call home.
|
||||
|
||||
### A cluster with no way out
|
||||
|
||||
A site with no route to the internet installs from an **offline bundle**
|
||||
instead, and then nothing at all leaves the cluster (docs/install.md, "Offline
|
||||
sites"). The bundle is made on a machine that has internet — llmgw's release
|
||||
binaries with their signed checksums, the llama.cpp builds with the checksums
|
||||
GitHub publishes, the model files with the checksums Hugging Face publishes,
|
||||
each checked as it is downloaded — carried in, and imported into the mirror.
|
||||
With `offline.mirror` set in `server.yaml`, nodes install engines from the
|
||||
mirror and never from GitHub, releases come from the mirror, and the registry
|
||||
is not asked about newer weights.
|
||||
|
||||
Verification is not weakened by any of this. A release is held to the project's
|
||||
signing key wherever it is fetched from. An engine file is held to the checksum
|
||||
recorded when the bundle was made, which the control plane keeps and hands each
|
||||
node over the agent's authenticated connection: the mirror serves plain HTTP,
|
||||
and nothing a node runs is taken on the mirror's word. The bundle itself is
|
||||
checked against the sha256 its maker printed, carried to the site apart from
|
||||
the bundle.
|
||||
|
||||
What does not come in a bundle: vLLM and SGLang, which are Python environments
|
||||
from PyPI — an offline node serves with llama.cpp unless one is installed on it
|
||||
by hand — and the operating system's own packages (the OpenMP runtime the
|
||||
engine needs, PostgreSQL when chosen), which come from the machines' package
|
||||
sources: at an offline site, the distribution mirror it already has. Searching
|
||||
the registry from `/admin/mirror` and `llmgw ctl plan` need the internet and
|
||||
do not work offline; `mirror.registry` can point the first at an internal copy.
|
||||
|
||||
## What the software cannot do for you
|
||||
|
||||
|
||||
Reference in new issue
Block a user