Offline sites and Kubernetes: llmgw bundle create, import and trust, offline.mirror, server init --bundle; a Helm chart for the control plane
A bundle carries what a cluster would download — llmgw's release with its signed checksums, every llama.cpp build per architecture with GitHub's checksums, model files pinned and checked against Hugging Face's — into a directory or a tar whose manifest sha256 is carried apart. Import checks everything, the release against the signing key, before anything is put in the mirror. With offline.mirror set, nodes install llama.cpp from the mirror against checksums the control plane holds and sends over the agent stream; releases default to the mirror for node add, ctl upgrade and agent upgrades. server init and the setup form install from a bundle. The Helm chart runs one control plane with SQLite on a claim or an external PostgreSQL, Secrets for the agent certificate and admin token, Services, Ingress and ServiceMonitor; deploy/container/Dockerfile builds its image. GPU nodes stay outside Kubernetes, as docs/kubernetes.md explains. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
e59ff52932
commit
bae16426a2
51 files changed
+4177
-63
No files matched your search
+50
-10
@@ -95,22 +95,62 @@ your own hardware, and the answer comes back. The router talks to engines, the
|
||||
engines talk to nothing, the agents talk to the control plane. All of it is
|
||||
inside the cluster.
|
||||
|
||||
Two things can reach outside, both started by an operator rather than by a
|
||||
request:
|
||||
Around the serving path, a cluster set up the ordinary way does reach out.
|
||||
None of it carries a prompt, an answer, a key or a user — it is software and
|
||||
public model files coming in — but a review should know each one:
|
||||
|
||||
**Fetching weights.** When a model's `source` is a URL, the agent downloads it.
|
||||
Point that at an internal mirror and nothing leaves at all — which is also the
|
||||
**Installing engines.** Each node's agent downloads the llama.cpp build for its
|
||||
hardware from GitHub, checked against the checksum GitHub publishes, and vLLM
|
||||
(SGLang when asked) from PyPI, with `uv` from GitHub. On every node, when it
|
||||
joins and when the engine version changes.
|
||||
|
||||
**Releases.** `llmgw node add`, `llmgw ctl upgrade` and `llmgw upgrade` fetch
|
||||
llmgw's own release binaries from the project's release server, checked against
|
||||
its signed checksums.
|
||||
|
||||
**Fetching weights.** When a model's `source` is a URL, the agent downloads it;
|
||||
the mirror fetches what it is asked to, from Hugging Face or another URL, and
|
||||
`server init` has it fetch the starter model and the guardrails' judge. Point
|
||||
sources at the mirror and the nodes fetch only from it — which is also the
|
||||
faster arrangement, because eight nodes pulling forty gigabytes from the
|
||||
internet is a day's work and from a local mirror it is minutes. A `source` that
|
||||
is a path on disk is never fetched.
|
||||
|
||||
**`llmgw ctl plan`.** Reads a model's architecture from huggingface.co to work
|
||||
out whether it fits your hardware. An operator command run at planning time,
|
||||
not part of serving, and not run by anything automatically.
|
||||
**The registry.** With a mirror configured, the control plane asks Hugging Face
|
||||
every six hours whether the models served from it have newer files
|
||||
(`mirror.update_check`), and `/admin/mirror` searches it. `llmgw ctl plan`
|
||||
reads a model's architecture from huggingface.co to work out whether it fits
|
||||
your hardware.
|
||||
|
||||
There is no telemetry, no licence check, and no call home. The binary is
|
||||
static, built with `CGO_ENABLED=0`, and runs in a network with no route to the
|
||||
internet as long as the weights are already on disk.
|
||||
There is no telemetry, no licence check, and no call home.
|
||||
|
||||
### A cluster with no way out
|
||||
|
||||
A site with no route to the internet installs from an **offline bundle**
|
||||
instead, and then nothing at all leaves the cluster (docs/install.md, "Offline
|
||||
sites"). The bundle is made on a machine that has internet — llmgw's release
|
||||
binaries with their signed checksums, the llama.cpp builds with the checksums
|
||||
GitHub publishes, the model files with the checksums Hugging Face publishes,
|
||||
each checked as it is downloaded — carried in, and imported into the mirror.
|
||||
With `offline.mirror` set in `server.yaml`, nodes install engines from the
|
||||
mirror and never from GitHub, releases come from the mirror, and the registry
|
||||
is not asked about newer weights.
|
||||
|
||||
Verification is not weakened by any of this. A release is held to the project's
|
||||
signing key wherever it is fetched from. An engine file is held to the checksum
|
||||
recorded when the bundle was made, which the control plane keeps and hands each
|
||||
node over the agent's authenticated connection: the mirror serves plain HTTP,
|
||||
and nothing a node runs is taken on the mirror's word. The bundle itself is
|
||||
checked against the sha256 its maker printed, carried to the site apart from
|
||||
the bundle.
|
||||
|
||||
What does not come in a bundle: vLLM and SGLang, which are Python environments
|
||||
from PyPI — an offline node serves with llama.cpp unless one is installed on it
|
||||
by hand — and the operating system's own packages (the OpenMP runtime the
|
||||
engine needs, PostgreSQL when chosen), which come from the machines' package
|
||||
sources: at an offline site, the distribution mirror it already has. Searching
|
||||
the registry from `/admin/mirror` and `llmgw ctl plan` need the internet and
|
||||
do not work offline; `mirror.registry` can point the first at an internal copy.
|
||||
|
||||
## What the software cannot do for you
|
||||
|
||||
|
||||
Reference in new issue
Block a user