Merge C18 offline bundles and the Helm chart

Conflicts: server.go imports (both kept); agent.pb.go regenerated with buf
from the merged proto.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-05 00:15:47 +02:00
commit 6f85c17fc5
51 files changed
+4177 -63

No files matched your search

+12
View File
@@ -52,6 +52,18 @@ cluster, a backup and adding a node change this machine or another over ssh;
those close the screen and run in the terminal, so their output is there to
read, and Enter brings the console back.
## Setting up
On a machine with nothing installed, the control plane is set up from one
form (`sudo llmgw server init`): the administrator's email, other names nodes
reach the machine by, and ticks for PostgreSQL, the model mirror, this machine
as a node, the starter model and the guardrails' judge. **Offline bundle**
and **Its sha256** are for a site with no internet: the path of a bundle made
by `llmgw bundle create`, and the sha256 it printed. Given, the bundle goes
into the mirror, every file checked, and the cluster is set up to fetch
nothing from outside (install.md, "Offline sites"). A bundle with no sha256,
or with the mirror unticked, is said in the form and nothing starts.
## Watch
**Overview** is `llmgw ctl status`: nodes, GPUs by class, fabrics, replicas,
+50 -10
View File
@@ -126,22 +126,62 @@ your own hardware, and the answer comes back. The router talks to engines, the
engines talk to nothing, the agents talk to the control plane. All of it is
inside the cluster.
Two things can reach outside, both started by an operator rather than by a
request:
Around the serving path, a cluster set up the ordinary way does reach out.
None of it carries a prompt, an answer, a key or a user — it is software and
public model files coming in — but a review should know each one:
**Fetching weights.** When a model's `source` is a URL, the agent downloads it.
Point that at an internal mirror and nothing leaves at all — which is also the
**Installing engines.** Each node's agent downloads the llama.cpp build for its
hardware from GitHub, checked against the checksum GitHub publishes, and vLLM
(SGLang when asked) from PyPI, with `uv` from GitHub. On every node, when it
joins and when the engine version changes.
**Releases.** `llmgw node add`, `llmgw ctl upgrade` and `llmgw upgrade` fetch
llmgw's own release binaries from the project's release server, checked against
its signed checksums.
**Fetching weights.** When a model's `source` is a URL, the agent downloads it;
the mirror fetches what it is asked to, from Hugging Face or another URL, and
`server init` has it fetch the starter model and the guardrails' judge. Point
sources at the mirror and the nodes fetch only from it — which is also the
faster arrangement, because eight nodes pulling forty gigabytes from the
internet is a day's work and from a local mirror it is minutes. A `source` that
is a path on disk is never fetched.
**`llmgw ctl plan`.** Reads a model's architecture from huggingface.co to work
out whether it fits your hardware. An operator command run at planning time,
not part of serving, and not run by anything automatically.
**The registry.** With a mirror configured, the control plane asks Hugging Face
every six hours whether the models served from it have newer files
(`mirror.update_check`), and `/admin/mirror` searches it. `llmgw ctl plan`
reads a model's architecture from huggingface.co to work out whether it fits
your hardware.
There is no telemetry, no licence check, and no call home. The binary is
static, built with `CGO_ENABLED=0`, and runs in a network with no route to the
internet as long as the weights are already on disk.
There is no telemetry, no licence check, and no call home.
### A cluster with no way out
A site with no route to the internet installs from an **offline bundle**
instead, and then nothing at all leaves the cluster (docs/install.md, "Offline
sites"). The bundle is made on a machine that has internet — llmgw's release
binaries with their signed checksums, the llama.cpp builds with the checksums
GitHub publishes, the model files with the checksums Hugging Face publishes,
each checked as it is downloaded — carried in, and imported into the mirror.
With `offline.mirror` set in `server.yaml`, nodes install engines from the
mirror and never from GitHub, releases come from the mirror, and the registry
is not asked about newer weights.
Verification is not weakened by any of this. A release is held to the project's
signing key wherever it is fetched from. An engine file is held to the checksum
recorded when the bundle was made, which the control plane keeps and hands each
node over the agent's authenticated connection: the mirror serves plain HTTP,
and nothing a node runs is taken on the mirror's word. The bundle itself is
checked against the sha256 its maker printed, carried to the site apart from
the bundle.
What does not come in a bundle: vLLM and SGLang, which are Python environments
from PyPI — an offline node serves with llama.cpp unless one is installed on it
by hand — and the operating system's own packages (the OpenMP runtime the
engine needs, PostgreSQL when chosen), which come from the machines' package
sources: at an offline site, the distribution mirror it already has. Searching
the registry from `/admin/mirror` and `llmgw ctl plan` need the internet and
do not work offline; `mirror.registry` can point the first at an internal copy.
## What the software cannot do for you
+135
View File
@@ -1861,6 +1861,139 @@ doctor` on a node says when the table is missing. `--no-firewall` on any of
the four commands leaves the firewall alone; `sudo systemctl disable --now
llmgw-firewall` removes it.
## Offline sites
Everything above downloads: the nodes fetch llama.cpp from GitHub and vLLM from
PyPI, `node add` and upgrades fetch llmgw's releases, and the mirror fetches
models from Hugging Face. A site with no route to the internet gets all of that
in a **bundle**, made somewhere that has one and carried in.
### Making the bundle
On any machine with internet and llmgw — a laptop will do:
```
llmgw bundle create --out llmgw-bundle --arch amd64 --tar
```
That carries:
- **llmgw's release** for each `--arch` (default: this llmgw's own version;
`--version` for another), with `checksums.txt` and its signature, checked
against the project's signing key as `ctl upgrade` checks it.
- **llama.cpp** at the version this llmgw installs, every build for each
architecture — CPU, CUDA 13.3, CUDA 12.8 (amd64), Vulkan — each checked
against the checksum GitHub publishes, the same check a node does online.
`--variant cpu --variant cuda-13.3` carries only those, for a site whose
hardware you know; `--engine llamacpp=b10991` names the version.
- **The starter model and the guardrails' judge** that `server init` serves
(about 1.3 GB; `--no-starter-models` leaves them out), and any
`--model name/version=hf:owner/repo/file` you add, pinned to the commit the
registry had and checked against the checksum it publishes.
It ends by printing the manifest's **sha256**. Carry that apart from the bundle
— in the ticket, read out over the phone — because it is what says the bundle
that arrived is the one that was made. `llmgw bundle show llmgw-bundle.tar
--verify` reads every file back, worth doing once it is on the disk that
carries it. Run `create` again with the same `--out` (without `--tar`) to add
to a bundle; what is there already is not downloaded again.
What a bundle does **not** carry: vLLM and SGLang, which are Python
environments of several gigabytes from PyPI. Offline nodes serve with llama.cpp;
on a GPU node that needs vLLM, install it by hand and name it with `node add
--vllm-bin` (the node reports vLLM as unsupported until then). Nor the
operating system's packages — the OpenMP runtime llama.cpp needs, PostgreSQL
with `--postgres` — which come from each machine's own package sources: at an
offline site, its distribution mirror.
### Installing from it
On the control plane, with the bundle on a local disk:
```
sudo llmgw server init --bundle /media/usb/llmgw-bundle.tar --bundle-sha256 3f2a…
```
(The console's setup form has the same two fields.) Init then does what it
always does, with these differences:
1. Before the control plane starts, the bundle goes into the mirror beside it:
every file checked against the manifest, the release against the signing
key, and only then put in place. llmgw's releases and the engines go under
`/srv/models/_llmgw/`, the models where the mirror keeps models.
2. The engines' checksums are written to `/etc/llmgw/offline.sha256`. That
file, on the control plane, is what each node's engine files are held to.
3. `server.yaml` gets
```yaml
offline:
mirror: http://10.0.0.5:8081 # the mirror's file port, as nodes reach it
```
and from then on the cluster asks nothing of the internet: nodes install
llama.cpp from the mirror, `releases` defaults to the mirror's copy (so
`node add` and `ctl upgrade` fetch from it), and the registry is not asked
about newer weights.
4. The starter model and the judge are served from the mirror when the bundle
carries them, and skipped — not waited for — when it does not. Without the
judge, the guard policy is written without one.
With the mirror on another machine (`--mirror`), import the bundle there first
and then run init with `--bundle` as above: the control plane records only the
checksums.
```
# on the mirror machine
sudo llmgw bundle import /media/usb/llmgw-bundle.tar --sha256 3f2a…
```
Nodes join as usual with `sudo llmgw node add`: it fetches the node's binary
from the mirror and checks its signature, and the agent installs llama.cpp
from the mirror. Nothing on the node needs the internet.
### How it stays checked
Online, a node trusts GitHub, over TLS, for the checksum of the llama.cpp it
installs, and the project's signing key for llmgw. Offline it is the same
strength by a different route: the checksum GitHub published is recorded when
the bundle is made, the bundle is held to its sha256 when it is imported, and
the control plane keeps the checksums and tells each node, over its mutually
authenticated connection, what every file it fetches must be. The mirror serves
plain HTTP and is never asked for a checksum: a file changed on it, or on the
way from it, is refused by the node. Releases carry the project's signature,
which holds whatever server they come from — so plain HTTP from the mirror is
allowed for releases only because this build trusts a signing key.
### Later
A new llmgw release or llama.cpp version is another bundle. Make it the same
way — `--version v0.12.0`, `--engine llamacpp=b11200` — and on the control plane
(and the mirror, if separate):
```
sudo llmgw bundle import new-bundle.tar --sha256 …
```
The control plane sees the new checksums without a restart. Then set
`engines.llamacpp` in `server.yaml` to the new version and restart
`llmgw-server`, or run `sudo llmgw ctl upgrade v0.12.0` for the release. A
mirror on its own machine, which is not an agent, is upgraded from its own
copy: `sudo llmgw upgrade v0.12.0 --service llmgw-mirror --releases
http://127.0.0.1:8081/_llmgw/releases`. A control plane whose bundle lacks the
llama.cpp version it asks for says so in its log at start, and the node says
which `bundle create` would carry it. More models: `--model` in a bundle, or
copy the files into the mirror by hand — it is a directory.
A cluster set up online becomes offline the same way: `bundle import` on the
mirror and control plane, then the `offline:` block in `server.yaml` and a
restart.
## Kubernetes
The control plane runs in Kubernetes from the Helm chart in
`deploy/helm/llmgw`; GPU nodes are best kept outside it, as ordinary hosts
joined with `node add`. Why, and how, is in [kubernetes.md](kubernetes.md).
## Upgrading
On the control plane:
@@ -2132,6 +2265,8 @@ requests. On a slow model, set `timeouts.generation` higher.
wrote about
- `deploy/mirror/README.md` — the mirror in more detail, including the layout
and what belongs on which port
- `docs/kubernetes.md` — the control plane in Kubernetes from the Helm chart,
and why GPU nodes stay outside it
- `deploy/monitoring/README.md` — Prometheus, Grafana and the alerts, with what
each one assumes about this system
- `docs/fabrics.md` — InfiniBand fabrics and NVLink: what llmgw needs of them,
+228
View File
@@ -0,0 +1,228 @@
# llmgw and Kubernetes
The control plane runs in Kubernetes, from the Helm chart in
`deploy/helm/llmgw`. The GPU nodes do not: they stay ordinary machines,
joined with `llmgw node add` as on any other cluster. This page says why,
and how the two meet.
## What goes where
| Part | Where | Why |
|---|---|---|
| Control plane: the API on 8080, the agents' port 7443, the scheduler | A StatefulSet, from the chart | A Go binary with a store and three ports: what Kubernetes runs well |
| Store | A file on a PersistentVolumeClaim (SQLite), or your PostgreSQL | Either, as on a host |
| GPU and CPU nodes | Outside, as hosts | See below |
| Model mirror | Outside, on a machine with a large disk | Every node pulls tens of gigabytes from it over plain HTTP; a pod with a volume that size is a host in all but name |
### Why the nodes stay outside
The agent on a node is not a service that happens to use a GPU. It installs
the engines on the machine — the llama.cpp build for its driver, vLLM in a
Python environment — checks them against their publisher's checksums and
runs them; it chooses which GPUs each replica gets, by index and by NVLink
topology; it starts engines as processes that outlive its own restarts, so a
node upgrades while it serves; it writes the machine's firewall; and it
replaces its own binary when the cluster upgrades. All of that is root on
the host, which is exactly what a container is built to prevent.
Run as a DaemonSet it would need to be privileged, with the host's network,
process namespace and filesystem, and the NVIDIA device plugin would have to
be kept off its GPUs — or the two would hand the same card to two owners.
What is left is the host agent with a container's indirection around it, and
none of the scheduling, isolation or restart guarantees that make a pod
worth having. So the chart does not ship one, and this is the
recommendation:
- **Keep GPU machines out of Kubernetes**, and add them with `node add`.
- **If they are already Kubernetes workers**, dedicate them: taint them so
no other pod lands there, keep the NVIDIA device plugin off them (most
installs select nodes by label — leave the label off), and run the llmgw
agent on the host, where `node add` puts it. llmgw then owns the GPUs, and
Kubernetes schedules nothing that would want them.
## The image
The project publishes binaries, not an image. Build one from the repository
and push it where the cluster pulls from — at an offline site, the internal
registry:
```
docker build -f deploy/container/Dockerfile --build-arg VERSION=v0.11.1 \
-t registry.internal/llmgw:v0.11.1 .
docker push registry.internal/llmgw:v0.11.1
```
It is the static binary on a distroless base: no shell, running as user
65532, with a read-only root filesystem in the pod.
## The agent port's certificate
Agents and the control plane authenticate each other with certificates from
llmgw's own authority, and the TLS ends in the pod — so the authority's key
stays outside the cluster, on the machine you will add nodes from, and the
pod is given only its own certificate.
Decide first the address agents will dial: the `-agents` Service is a
LoadBalancer, so give it a fixed `agentService.loadBalancerIP` or a DNS name
that points at it. That address has to be in the certificate:
```
llmgw ctl cert init-ca --dir ./pki
llmgw ctl cert issue-server llmgw-ctl --host 10.0.0.200 --host llmgw-agents.example.com \
--dir ./pki --out ./pki
kubectl -n llmgw create secret generic llmgw-agent-tls \
--from-file=ca.pem=pki/ca.pem \
--from-file=server.pem=pki/server.pem \
--from-file=server-key.pem=pki/server-key.pem
```
Keep `pki/ca-key.pem` as you would on a host: whoever holds it can make any
node's identity.
## Installing
```yaml
# values.yaml
image:
repository: registry.internal/llmgw
tag: v0.11.1
agentTLS:
existingSecret: llmgw-agent-tls
agentService:
loadBalancerIP: 10.0.0.200
ingress:
enabled: true
className: nginx
hosts: [{host: llm.example.com, paths: [{path: /, pathType: Prefix}]}]
tls: [{secretName: llm-example-com, hosts: [llm.example.com]}]
config:
mirror:
api: http://models.internal:8082
# The address the router reaches nodes' engines from is a Kubernetes
# node's, once the pod's traffic leaves the cluster: the nodes' firewalls
# must let it in.
firewall_networks: ["10.0.1.0/24"]
```
```
helm install llmgw deploy/helm/llmgw -n llmgw -f values.yaml
```
`config` is `server.yaml`: any key of [reference.md](reference.md), laid
under what the chart sets itself — the listeners, where the certificates,
the admin token and the store are mounted. A changed `config` restarts the
pod.
**The store.** By default SQLite, on a 10 GiB claim. For PostgreSQL, put its
address in a Secret and point the chart at it:
```
kubectl -n llmgw create secret generic llmgw-store \
--from-literal=dsn='postgres://llmgw:…@pg.internal:5432/llmgw?sslmode=verify-full'
```
```yaml
store:
type: postgres
postgres:
existingSecret: llmgw-store
```
**One control plane.** The chart runs one replica. Two against PostgreSQL
would elect a leader and keep a standby, but a standby serves nothing and is
never ready, so a StatefulSet rollout would wait on it for ever. Kubernetes
replaces the one pod within seconds when it goes, and agents reconnect on
their own; the replicas on the nodes keep serving meanwhile.
**Probes.** Alive is `/health`. Ready is `/leader`, which on PostgreSQL waits
for the lease of the pod it replaced to run out — not `/ready`, which waits
for a model replica, and would keep a cluster with no node yet from ever
accepting one.
**Monitoring.** The metrics port, 9091, serves metrics and nothing else;
`serviceMonitor.enabled: true` makes a ServiceMonitor for the Prometheus
Operator, and `deploy/monitoring` holds the rules and dashboards.
## Using it
The admin API listens on the pod's loopback, as on a host — it can dump keys
from the heap when profiling is on. Reach it through the pod:
```
kubectl -n llmgw exec -it statefulset/llmgw -- llmgw users create you@example.com --password '…'
kubectl -n llmgw exec -it statefulset/llmgw -- llmgw users role you@example.com admin
kubectl -n llmgw exec -it statefulset/llmgw -- llmgw ctl status
```
or from your machine, through a port-forward:
```
kubectl -n llmgw port-forward statefulset/llmgw 9090:9090 &
export LLMGW_ADMIN_TOKEN=$(kubectl -n llmgw get secret llmgw-admin -o jsonpath='{.data.admin-token}' | base64 -d)
llmgw ctl status
```
The token is made by the chart and kept across upgrades
(`adminToken.existingSecret` brings your own).
## Adding nodes
From the machine holding `pki/`, with the port-forward above running and
`LLMGW_ADMIN_TOKEN` set:
```
sudo -E llmgw node add ubuntu@gpu01 \
--server https://10.0.0.200:7443 --pki ./pki --admin 127.0.0.1:9090
```
`--server` is the address in the certificate. The node is installed, checked
and started over ssh exactly as on any cluster ([node-onboarding.md](node-onboarding.md)),
and dials the control plane in the pod. Its firewall comes from
`firewall_networks` in the server configuration `node add` reads (`--config`,
`/etc/llmgw/server.yaml` by default): copy the chart's out of the cluster
first, or give `--no-firewall` and keep the engines' ports open to the
Kubernetes nodes by hand.
```
kubectl -n llmgw get configmap llmgw -o jsonpath='{.data.server\.yaml}' > server.yaml
sudo -E llmgw node add ubuntu@gpu01 --config ./server.yaml --server https://10.0.0.200:7443 \
--pki ./pki --admin 127.0.0.1:9090
```
## Offline
An offline site ([install.md](install.md), "Offline sites") works the same
way with the control plane in a pod. Import the bundle on the mirror's
machine with `llmgw bundle import`, record its engines' checksums in a
file, and give that file to the chart as a ConfigMap:
```
llmgw bundle trust llmgw-bundle.tar --sha256 3f2a… --file ./offline.sha256
kubectl -n llmgw create configmap llmgw-offline --from-file=offline.sha256
```
```yaml
offlineChecksums:
configMap: llmgw-offline
config:
offline:
mirror: http://models.internal:8081
```
A later bundle: `bundle trust` into the same file, and replace the ConfigMap
(`kubectl create configmap … --dry-run=client -o yaml | kubectl apply -f -`).
The control plane reads the file again when it changes; there is no restart.
## What has and has not been tried
The chart is checked with `helm lint` and `helm template` for SQLite,
PostgreSQL, an Ingress, a ServiceMonitor and an offline cluster, and refuses
what would not work — no image, no certificate, a PostgreSQL store without
its address, more than one replica. The image has been run the way the
chart runs it — read-only root, user 65532, the same files mounted at the
same paths, SQLite on a volume — and answered `/health`, `/leader` and
`/metrics`, with `llmgw ctl` and `llmgw users` working through `exec`.
It has **not** been installed on a Kubernetes cluster, and no node has yet
joined a control plane running in one. Until it has, a control plane on a
host is the well-trodden path.
+36
View File
@@ -139,6 +139,8 @@ rather than repeated.
| `roles` | `map[string][]string` | [OIDC](#oidc) |
| `memberships` | `map[string]string` | [OIDC](#oidc) |
| `no_passwords` | `bool` | [OIDC](#oidc) |
| `mirror` | `string` | [Offline](#offline) |
| `checksums` | `string` | [Offline](#offline) |
| `nvidia_smi` | `string` | [Options](#options) |
| `ib_root` | `string` | [Options](#options) |
| `prefer_nvml` | `bool` | [Options](#options) |
@@ -257,6 +259,7 @@ rather than repeated.
| `monitoring` | `Monitoring` | [Server](#server) |
| `tracing` | `obs.Tracing` | [Server](#server) |
| `mirror` | `Mirror` | [Server](#server) |
| `offline` | `Offline` | [Server](#server) |
| `engines` | `map[string]string` | [Server](#server) |
| `engine_overrides` | `[]EngineOverride` | [Server](#server) |
| `backup_dir` | `string` | [Server](#server) |
@@ -1273,6 +1276,29 @@ PasswordSignIn off refuses passwords on the sign-in form: people come
in through the provider only. Administrators still can, so a broken
provider does not lock everybody out.
## Offline
### `mirror`
`string`
Mirror is the mirror's file address, as nodes reach it:
http://10.0.0.5:8081 — the port model files are served on, not the
management API. Set, the cluster is offline: nodes install engines
from it and never from their publishers, releases come from it unless
releases names another place, and no registry is asked whether a
model has newer weights. Python engines (vLLM, SGLang) come from PyPI,
which no bundle carries; on an offline node they are installed by hand.
### `checksums`
`string`
Checksums is the file of engine checksums this control plane trusts,
kept by llmgw bundle import and llmgw bundle trust; every engine file
a node fetches from the mirror is checked against it. Read again
whenever it changes. Empty is /etc/llmgw/offline.sha256.
## Options
### `nvidia_smi`
@@ -2128,6 +2154,16 @@ Empty means there is none, and /admin/mirror stays unmounted.
Takes a [Mirror](#mirror) block.
### `offline`
`Offline`
Offline is a cluster that reaches nothing outside itself: its nodes
install engines, and upgrades fetch releases, from the mirror, out of
a bundle carried in (llmgw bundle create, llmgw bundle import).
Takes a [Offline](#offline) block.
### `engines`
`map[string]string`