Offline sites and Kubernetes: llmgw bundle create, import and trust, offline.mirror, server init --bundle; a Helm chart for the control plane
A bundle carries what a cluster would download — llmgw's release with its signed checksums, every llama.cpp build per architecture with GitHub's checksums, model files pinned and checked against Hugging Face's — into a directory or a tar whose manifest sha256 is carried apart. Import checks everything, the release against the signing key, before anything is put in the mirror. With offline.mirror set, nodes install llama.cpp from the mirror against checksums the control plane holds and sends over the agent stream; releases default to the mirror for node add, ctl upgrade and agent upgrades. server init and the setup form install from a bundle. The Helm chart runs one control plane with SQLite on a claim or an external PostgreSQL, Secrets for the agent certificate and admin token, Services, Ingress and ServiceMonitor; deploy/container/Dockerfile builds its image. GPU nodes stay outside Kubernetes, as docs/kubernetes.md explains. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
e59ff52932
commit
bae16426a2
51 files changed
+4177
-63
No files matched your search
@@ -52,6 +52,18 @@ cluster, a backup and adding a node change this machine or another over ssh;
|
||||
those close the screen and run in the terminal, so their output is there to
|
||||
read, and Enter brings the console back.
|
||||
|
||||
## Setting up
|
||||
|
||||
On a machine with nothing installed, the control plane is set up from one
|
||||
form (`sudo llmgw server init`): the administrator's email, other names nodes
|
||||
reach the machine by, and ticks for PostgreSQL, the model mirror, this machine
|
||||
as a node, the starter model and the guardrails' judge. **Offline bundle**
|
||||
and **Its sha256** are for a site with no internet: the path of a bundle made
|
||||
by `llmgw bundle create`, and the sha256 it printed. Given, the bundle goes
|
||||
into the mirror, every file checked, and the cluster is set up to fetch
|
||||
nothing from outside (install.md, "Offline sites"). A bundle with no sha256,
|
||||
or with the mirror unticked, is said in the form and nothing starts.
|
||||
|
||||
## Watch
|
||||
|
||||
**Overview** is `llmgw ctl status`: nodes, GPUs by class, fabrics, replicas,
|
||||
|
||||
+50
-10
@@ -95,22 +95,62 @@ your own hardware, and the answer comes back. The router talks to engines, the
|
||||
engines talk to nothing, the agents talk to the control plane. All of it is
|
||||
inside the cluster.
|
||||
|
||||
Two things can reach outside, both started by an operator rather than by a
|
||||
request:
|
||||
Around the serving path, a cluster set up the ordinary way does reach out.
|
||||
None of it carries a prompt, an answer, a key or a user — it is software and
|
||||
public model files coming in — but a review should know each one:
|
||||
|
||||
**Fetching weights.** When a model's `source` is a URL, the agent downloads it.
|
||||
Point that at an internal mirror and nothing leaves at all — which is also the
|
||||
**Installing engines.** Each node's agent downloads the llama.cpp build for its
|
||||
hardware from GitHub, checked against the checksum GitHub publishes, and vLLM
|
||||
(SGLang when asked) from PyPI, with `uv` from GitHub. On every node, when it
|
||||
joins and when the engine version changes.
|
||||
|
||||
**Releases.** `llmgw node add`, `llmgw ctl upgrade` and `llmgw upgrade` fetch
|
||||
llmgw's own release binaries from the project's release server, checked against
|
||||
its signed checksums.
|
||||
|
||||
**Fetching weights.** When a model's `source` is a URL, the agent downloads it;
|
||||
the mirror fetches what it is asked to, from Hugging Face or another URL, and
|
||||
`server init` has it fetch the starter model and the guardrails' judge. Point
|
||||
sources at the mirror and the nodes fetch only from it — which is also the
|
||||
faster arrangement, because eight nodes pulling forty gigabytes from the
|
||||
internet is a day's work and from a local mirror it is minutes. A `source` that
|
||||
is a path on disk is never fetched.
|
||||
|
||||
**`llmgw ctl plan`.** Reads a model's architecture from huggingface.co to work
|
||||
out whether it fits your hardware. An operator command run at planning time,
|
||||
not part of serving, and not run by anything automatically.
|
||||
**The registry.** With a mirror configured, the control plane asks Hugging Face
|
||||
every six hours whether the models served from it have newer files
|
||||
(`mirror.update_check`), and `/admin/mirror` searches it. `llmgw ctl plan`
|
||||
reads a model's architecture from huggingface.co to work out whether it fits
|
||||
your hardware.
|
||||
|
||||
There is no telemetry, no licence check, and no call home. The binary is
|
||||
static, built with `CGO_ENABLED=0`, and runs in a network with no route to the
|
||||
internet as long as the weights are already on disk.
|
||||
There is no telemetry, no licence check, and no call home.
|
||||
|
||||
### A cluster with no way out
|
||||
|
||||
A site with no route to the internet installs from an **offline bundle**
|
||||
instead, and then nothing at all leaves the cluster (docs/install.md, "Offline
|
||||
sites"). The bundle is made on a machine that has internet — llmgw's release
|
||||
binaries with their signed checksums, the llama.cpp builds with the checksums
|
||||
GitHub publishes, the model files with the checksums Hugging Face publishes,
|
||||
each checked as it is downloaded — carried in, and imported into the mirror.
|
||||
With `offline.mirror` set in `server.yaml`, nodes install engines from the
|
||||
mirror and never from GitHub, releases come from the mirror, and the registry
|
||||
is not asked about newer weights.
|
||||
|
||||
Verification is not weakened by any of this. A release is held to the project's
|
||||
signing key wherever it is fetched from. An engine file is held to the checksum
|
||||
recorded when the bundle was made, which the control plane keeps and hands each
|
||||
node over the agent's authenticated connection: the mirror serves plain HTTP,
|
||||
and nothing a node runs is taken on the mirror's word. The bundle itself is
|
||||
checked against the sha256 its maker printed, carried to the site apart from
|
||||
the bundle.
|
||||
|
||||
What does not come in a bundle: vLLM and SGLang, which are Python environments
|
||||
from PyPI — an offline node serves with llama.cpp unless one is installed on it
|
||||
by hand — and the operating system's own packages (the OpenMP runtime the
|
||||
engine needs, PostgreSQL when chosen), which come from the machines' package
|
||||
sources: at an offline site, the distribution mirror it already has. Searching
|
||||
the registry from `/admin/mirror` and `llmgw ctl plan` need the internet and
|
||||
do not work offline; `mirror.registry` can point the first at an internal copy.
|
||||
|
||||
## What the software cannot do for you
|
||||
|
||||
|
||||
+135
@@ -1245,6 +1245,139 @@ doctor` on a node says when the table is missing. `--no-firewall` on any of
|
||||
the four commands leaves the firewall alone; `sudo systemctl disable --now
|
||||
llmgw-firewall` removes it.
|
||||
|
||||
## Offline sites
|
||||
|
||||
Everything above downloads: the nodes fetch llama.cpp from GitHub and vLLM from
|
||||
PyPI, `node add` and upgrades fetch llmgw's releases, and the mirror fetches
|
||||
models from Hugging Face. A site with no route to the internet gets all of that
|
||||
in a **bundle**, made somewhere that has one and carried in.
|
||||
|
||||
### Making the bundle
|
||||
|
||||
On any machine with internet and llmgw — a laptop will do:
|
||||
|
||||
```
|
||||
llmgw bundle create --out llmgw-bundle --arch amd64 --tar
|
||||
```
|
||||
|
||||
That carries:
|
||||
|
||||
- **llmgw's release** for each `--arch` (default: this llmgw's own version;
|
||||
`--version` for another), with `checksums.txt` and its signature, checked
|
||||
against the project's signing key as `ctl upgrade` checks it.
|
||||
- **llama.cpp** at the version this llmgw installs, every build for each
|
||||
architecture — CPU, CUDA 13.3, CUDA 12.8 (amd64), Vulkan — each checked
|
||||
against the checksum GitHub publishes, the same check a node does online.
|
||||
`--variant cpu --variant cuda-13.3` carries only those, for a site whose
|
||||
hardware you know; `--engine llamacpp=b10991` names the version.
|
||||
- **The starter model and the guardrails' judge** that `server init` serves
|
||||
(about 1.3 GB; `--no-starter-models` leaves them out), and any
|
||||
`--model name/version=hf:owner/repo/file` you add, pinned to the commit the
|
||||
registry had and checked against the checksum it publishes.
|
||||
|
||||
It ends by printing the manifest's **sha256**. Carry that apart from the bundle
|
||||
— in the ticket, read out over the phone — because it is what says the bundle
|
||||
that arrived is the one that was made. `llmgw bundle show llmgw-bundle.tar
|
||||
--verify` reads every file back, worth doing once it is on the disk that
|
||||
carries it. Run `create` again with the same `--out` (without `--tar`) to add
|
||||
to a bundle; what is there already is not downloaded again.
|
||||
|
||||
What a bundle does **not** carry: vLLM and SGLang, which are Python
|
||||
environments of several gigabytes from PyPI. Offline nodes serve with llama.cpp;
|
||||
on a GPU node that needs vLLM, install it by hand and name it with `node add
|
||||
--vllm-bin` (the node reports vLLM as unsupported until then). Nor the
|
||||
operating system's packages — the OpenMP runtime llama.cpp needs, PostgreSQL
|
||||
with `--postgres` — which come from each machine's own package sources: at an
|
||||
offline site, its distribution mirror.
|
||||
|
||||
### Installing from it
|
||||
|
||||
On the control plane, with the bundle on a local disk:
|
||||
|
||||
```
|
||||
sudo llmgw server init --bundle /media/usb/llmgw-bundle.tar --bundle-sha256 3f2a…
|
||||
```
|
||||
|
||||
(The console's setup form has the same two fields.) Init then does what it
|
||||
always does, with these differences:
|
||||
|
||||
1. Before the control plane starts, the bundle goes into the mirror beside it:
|
||||
every file checked against the manifest, the release against the signing
|
||||
key, and only then put in place. llmgw's releases and the engines go under
|
||||
`/srv/models/_llmgw/`, the models where the mirror keeps models.
|
||||
2. The engines' checksums are written to `/etc/llmgw/offline.sha256`. That
|
||||
file, on the control plane, is what each node's engine files are held to.
|
||||
3. `server.yaml` gets
|
||||
|
||||
```yaml
|
||||
offline:
|
||||
mirror: http://10.0.0.5:8081 # the mirror's file port, as nodes reach it
|
||||
```
|
||||
|
||||
and from then on the cluster asks nothing of the internet: nodes install
|
||||
llama.cpp from the mirror, `releases` defaults to the mirror's copy (so
|
||||
`node add` and `ctl upgrade` fetch from it), and the registry is not asked
|
||||
about newer weights.
|
||||
4. The starter model and the judge are served from the mirror when the bundle
|
||||
carries them, and skipped — not waited for — when it does not. Without the
|
||||
judge, the guard policy is written without one.
|
||||
|
||||
With the mirror on another machine (`--mirror`), import the bundle there first
|
||||
and then run init with `--bundle` as above: the control plane records only the
|
||||
checksums.
|
||||
|
||||
```
|
||||
# on the mirror machine
|
||||
sudo llmgw bundle import /media/usb/llmgw-bundle.tar --sha256 3f2a…
|
||||
```
|
||||
|
||||
Nodes join as usual with `sudo llmgw node add`: it fetches the node's binary
|
||||
from the mirror and checks its signature, and the agent installs llama.cpp
|
||||
from the mirror. Nothing on the node needs the internet.
|
||||
|
||||
### How it stays checked
|
||||
|
||||
Online, a node trusts GitHub, over TLS, for the checksum of the llama.cpp it
|
||||
installs, and the project's signing key for llmgw. Offline it is the same
|
||||
strength by a different route: the checksum GitHub published is recorded when
|
||||
the bundle is made, the bundle is held to its sha256 when it is imported, and
|
||||
the control plane keeps the checksums and tells each node, over its mutually
|
||||
authenticated connection, what every file it fetches must be. The mirror serves
|
||||
plain HTTP and is never asked for a checksum: a file changed on it, or on the
|
||||
way from it, is refused by the node. Releases carry the project's signature,
|
||||
which holds whatever server they come from — so plain HTTP from the mirror is
|
||||
allowed for releases only because this build trusts a signing key.
|
||||
|
||||
### Later
|
||||
|
||||
A new llmgw release or llama.cpp version is another bundle. Make it the same
|
||||
way — `--version v0.12.0`, `--engine llamacpp=b11200` — and on the control plane
|
||||
(and the mirror, if separate):
|
||||
|
||||
```
|
||||
sudo llmgw bundle import new-bundle.tar --sha256 …
|
||||
```
|
||||
|
||||
The control plane sees the new checksums without a restart. Then set
|
||||
`engines.llamacpp` in `server.yaml` to the new version and restart
|
||||
`llmgw-server`, or run `sudo llmgw ctl upgrade v0.12.0` for the release. A
|
||||
mirror on its own machine, which is not an agent, is upgraded from its own
|
||||
copy: `sudo llmgw upgrade v0.12.0 --service llmgw-mirror --releases
|
||||
http://127.0.0.1:8081/_llmgw/releases`. A control plane whose bundle lacks the
|
||||
llama.cpp version it asks for says so in its log at start, and the node says
|
||||
which `bundle create` would carry it. More models: `--model` in a bundle, or
|
||||
copy the files into the mirror by hand — it is a directory.
|
||||
|
||||
A cluster set up online becomes offline the same way: `bundle import` on the
|
||||
mirror and control plane, then the `offline:` block in `server.yaml` and a
|
||||
restart.
|
||||
|
||||
## Kubernetes
|
||||
|
||||
The control plane runs in Kubernetes from the Helm chart in
|
||||
`deploy/helm/llmgw`; GPU nodes are best kept outside it, as ordinary hosts
|
||||
joined with `node add`. Why, and how, is in [kubernetes.md](kubernetes.md).
|
||||
|
||||
## Upgrading
|
||||
|
||||
On the control plane:
|
||||
@@ -1488,6 +1621,8 @@ requests. On a slow model, set `timeouts.generation` higher.
|
||||
wrote about
|
||||
- `deploy/mirror/README.md` — the mirror in more detail, including the layout
|
||||
and what belongs on which port
|
||||
- `docs/kubernetes.md` — the control plane in Kubernetes from the Helm chart,
|
||||
and why GPU nodes stay outside it
|
||||
- `deploy/monitoring/README.md` — Prometheus, Grafana and the alerts, with what
|
||||
each one assumes about this system
|
||||
- `docs/fabrics.md` — InfiniBand fabrics and NVLink: what llmgw needs of them,
|
||||
|
||||
@@ -0,0 +1,228 @@
|
||||
# llmgw and Kubernetes
|
||||
|
||||
The control plane runs in Kubernetes, from the Helm chart in
|
||||
`deploy/helm/llmgw`. The GPU nodes do not: they stay ordinary machines,
|
||||
joined with `llmgw node add` as on any other cluster. This page says why,
|
||||
and how the two meet.
|
||||
|
||||
## What goes where
|
||||
|
||||
| Part | Where | Why |
|
||||
|---|---|---|
|
||||
| Control plane: the API on 8080, the agents' port 7443, the scheduler | A StatefulSet, from the chart | A Go binary with a store and three ports: what Kubernetes runs well |
|
||||
| Store | A file on a PersistentVolumeClaim (SQLite), or your PostgreSQL | Either, as on a host |
|
||||
| GPU and CPU nodes | Outside, as hosts | See below |
|
||||
| Model mirror | Outside, on a machine with a large disk | Every node pulls tens of gigabytes from it over plain HTTP; a pod with a volume that size is a host in all but name |
|
||||
|
||||
### Why the nodes stay outside
|
||||
|
||||
The agent on a node is not a service that happens to use a GPU. It installs
|
||||
the engines on the machine — the llama.cpp build for its driver, vLLM in a
|
||||
Python environment — checks them against their publisher's checksums and
|
||||
runs them; it chooses which GPUs each replica gets, by index and by NVLink
|
||||
topology; it starts engines as processes that outlive its own restarts, so a
|
||||
node upgrades while it serves; it writes the machine's firewall; and it
|
||||
replaces its own binary when the cluster upgrades. All of that is root on
|
||||
the host, which is exactly what a container is built to prevent.
|
||||
|
||||
Run as a DaemonSet it would need to be privileged, with the host's network,
|
||||
process namespace and filesystem, and the NVIDIA device plugin would have to
|
||||
be kept off its GPUs — or the two would hand the same card to two owners.
|
||||
What is left is the host agent with a container's indirection around it, and
|
||||
none of the scheduling, isolation or restart guarantees that make a pod
|
||||
worth having. So the chart does not ship one, and this is the
|
||||
recommendation:
|
||||
|
||||
- **Keep GPU machines out of Kubernetes**, and add them with `node add`.
|
||||
- **If they are already Kubernetes workers**, dedicate them: taint them so
|
||||
no other pod lands there, keep the NVIDIA device plugin off them (most
|
||||
installs select nodes by label — leave the label off), and run the llmgw
|
||||
agent on the host, where `node add` puts it. llmgw then owns the GPUs, and
|
||||
Kubernetes schedules nothing that would want them.
|
||||
|
||||
## The image
|
||||
|
||||
The project publishes binaries, not an image. Build one from the repository
|
||||
and push it where the cluster pulls from — at an offline site, the internal
|
||||
registry:
|
||||
|
||||
```
|
||||
docker build -f deploy/container/Dockerfile --build-arg VERSION=v0.11.1 \
|
||||
-t registry.internal/llmgw:v0.11.1 .
|
||||
docker push registry.internal/llmgw:v0.11.1
|
||||
```
|
||||
|
||||
It is the static binary on a distroless base: no shell, running as user
|
||||
65532, with a read-only root filesystem in the pod.
|
||||
|
||||
## The agent port's certificate
|
||||
|
||||
Agents and the control plane authenticate each other with certificates from
|
||||
llmgw's own authority, and the TLS ends in the pod — so the authority's key
|
||||
stays outside the cluster, on the machine you will add nodes from, and the
|
||||
pod is given only its own certificate.
|
||||
|
||||
Decide first the address agents will dial: the `-agents` Service is a
|
||||
LoadBalancer, so give it a fixed `agentService.loadBalancerIP` or a DNS name
|
||||
that points at it. That address has to be in the certificate:
|
||||
|
||||
```
|
||||
llmgw ctl cert init-ca --dir ./pki
|
||||
llmgw ctl cert issue-server llmgw-ctl --host 10.0.0.200 --host llmgw-agents.example.com \
|
||||
--dir ./pki --out ./pki
|
||||
kubectl -n llmgw create secret generic llmgw-agent-tls \
|
||||
--from-file=ca.pem=pki/ca.pem \
|
||||
--from-file=server.pem=pki/server.pem \
|
||||
--from-file=server-key.pem=pki/server-key.pem
|
||||
```
|
||||
|
||||
Keep `pki/ca-key.pem` as you would on a host: whoever holds it can make any
|
||||
node's identity.
|
||||
|
||||
## Installing
|
||||
|
||||
```yaml
|
||||
# values.yaml
|
||||
image:
|
||||
repository: registry.internal/llmgw
|
||||
tag: v0.11.1
|
||||
agentTLS:
|
||||
existingSecret: llmgw-agent-tls
|
||||
agentService:
|
||||
loadBalancerIP: 10.0.0.200
|
||||
ingress:
|
||||
enabled: true
|
||||
className: nginx
|
||||
hosts: [{host: llm.example.com, paths: [{path: /, pathType: Prefix}]}]
|
||||
tls: [{secretName: llm-example-com, hosts: [llm.example.com]}]
|
||||
config:
|
||||
mirror:
|
||||
api: http://models.internal:8082
|
||||
# The address the router reaches nodes' engines from is a Kubernetes
|
||||
# node's, once the pod's traffic leaves the cluster: the nodes' firewalls
|
||||
# must let it in.
|
||||
firewall_networks: ["10.0.1.0/24"]
|
||||
```
|
||||
|
||||
```
|
||||
helm install llmgw deploy/helm/llmgw -n llmgw -f values.yaml
|
||||
```
|
||||
|
||||
`config` is `server.yaml`: any key of [reference.md](reference.md), laid
|
||||
under what the chart sets itself — the listeners, where the certificates,
|
||||
the admin token and the store are mounted. A changed `config` restarts the
|
||||
pod.
|
||||
|
||||
**The store.** By default SQLite, on a 10 GiB claim. For PostgreSQL, put its
|
||||
address in a Secret and point the chart at it:
|
||||
|
||||
```
|
||||
kubectl -n llmgw create secret generic llmgw-store \
|
||||
--from-literal=dsn='postgres://llmgw:…@pg.internal:5432/llmgw?sslmode=verify-full'
|
||||
```
|
||||
|
||||
```yaml
|
||||
store:
|
||||
type: postgres
|
||||
postgres:
|
||||
existingSecret: llmgw-store
|
||||
```
|
||||
|
||||
**One control plane.** The chart runs one replica. Two against PostgreSQL
|
||||
would elect a leader and keep a standby, but a standby serves nothing and is
|
||||
never ready, so a StatefulSet rollout would wait on it for ever. Kubernetes
|
||||
replaces the one pod within seconds when it goes, and agents reconnect on
|
||||
their own; the replicas on the nodes keep serving meanwhile.
|
||||
|
||||
**Probes.** Alive is `/health`. Ready is `/leader`, which on PostgreSQL waits
|
||||
for the lease of the pod it replaced to run out — not `/ready`, which waits
|
||||
for a model replica, and would keep a cluster with no node yet from ever
|
||||
accepting one.
|
||||
|
||||
**Monitoring.** The metrics port, 9091, serves metrics and nothing else;
|
||||
`serviceMonitor.enabled: true` makes a ServiceMonitor for the Prometheus
|
||||
Operator, and `deploy/monitoring` holds the rules and dashboards.
|
||||
|
||||
## Using it
|
||||
|
||||
The admin API listens on the pod's loopback, as on a host — it can dump keys
|
||||
from the heap when profiling is on. Reach it through the pod:
|
||||
|
||||
```
|
||||
kubectl -n llmgw exec -it statefulset/llmgw -- llmgw users create you@example.com --password '…'
|
||||
kubectl -n llmgw exec -it statefulset/llmgw -- llmgw users role you@example.com admin
|
||||
kubectl -n llmgw exec -it statefulset/llmgw -- llmgw ctl status
|
||||
```
|
||||
|
||||
or from your machine, through a port-forward:
|
||||
|
||||
```
|
||||
kubectl -n llmgw port-forward statefulset/llmgw 9090:9090 &
|
||||
export LLMGW_ADMIN_TOKEN=$(kubectl -n llmgw get secret llmgw-admin -o jsonpath='{.data.admin-token}' | base64 -d)
|
||||
llmgw ctl status
|
||||
```
|
||||
|
||||
The token is made by the chart and kept across upgrades
|
||||
(`adminToken.existingSecret` brings your own).
|
||||
|
||||
## Adding nodes
|
||||
|
||||
From the machine holding `pki/`, with the port-forward above running and
|
||||
`LLMGW_ADMIN_TOKEN` set:
|
||||
|
||||
```
|
||||
sudo -E llmgw node add ubuntu@gpu01 \
|
||||
--server https://10.0.0.200:7443 --pki ./pki --admin 127.0.0.1:9090
|
||||
```
|
||||
|
||||
`--server` is the address in the certificate. The node is installed, checked
|
||||
and started over ssh exactly as on any cluster ([node-onboarding.md](node-onboarding.md)),
|
||||
and dials the control plane in the pod. Its firewall comes from
|
||||
`firewall_networks` in the server configuration `node add` reads (`--config`,
|
||||
`/etc/llmgw/server.yaml` by default): copy the chart's out of the cluster
|
||||
first, or give `--no-firewall` and keep the engines' ports open to the
|
||||
Kubernetes nodes by hand.
|
||||
|
||||
```
|
||||
kubectl -n llmgw get configmap llmgw -o jsonpath='{.data.server\.yaml}' > server.yaml
|
||||
sudo -E llmgw node add ubuntu@gpu01 --config ./server.yaml --server https://10.0.0.200:7443 \
|
||||
--pki ./pki --admin 127.0.0.1:9090
|
||||
```
|
||||
|
||||
## Offline
|
||||
|
||||
An offline site ([install.md](install.md), "Offline sites") works the same
|
||||
way with the control plane in a pod. Import the bundle on the mirror's
|
||||
machine with `llmgw bundle import`, record its engines' checksums in a
|
||||
file, and give that file to the chart as a ConfigMap:
|
||||
|
||||
```
|
||||
llmgw bundle trust llmgw-bundle.tar --sha256 3f2a… --file ./offline.sha256
|
||||
kubectl -n llmgw create configmap llmgw-offline --from-file=offline.sha256
|
||||
```
|
||||
|
||||
```yaml
|
||||
offlineChecksums:
|
||||
configMap: llmgw-offline
|
||||
config:
|
||||
offline:
|
||||
mirror: http://models.internal:8081
|
||||
```
|
||||
|
||||
A later bundle: `bundle trust` into the same file, and replace the ConfigMap
|
||||
(`kubectl create configmap … --dry-run=client -o yaml | kubectl apply -f -`).
|
||||
The control plane reads the file again when it changes; there is no restart.
|
||||
|
||||
## What has and has not been tried
|
||||
|
||||
The chart is checked with `helm lint` and `helm template` for SQLite,
|
||||
PostgreSQL, an Ingress, a ServiceMonitor and an offline cluster, and refuses
|
||||
what would not work — no image, no certificate, a PostgreSQL store without
|
||||
its address, more than one replica. The image has been run the way the
|
||||
chart runs it — read-only root, user 65532, the same files mounted at the
|
||||
same paths, SQLite on a volume — and answered `/health`, `/leader` and
|
||||
`/metrics`, with `llmgw ctl` and `llmgw users` working through `exec`.
|
||||
|
||||
It has **not** been installed on a Kubernetes cluster, and no node has yet
|
||||
joined a control plane running in one. Until it has, a control plane on a
|
||||
host is the well-trodden path.
|
||||
@@ -120,6 +120,8 @@ rather than repeated.
|
||||
| `roles` | `map[string][]string` | [OIDC](#oidc) |
|
||||
| `memberships` | `map[string]string` | [OIDC](#oidc) |
|
||||
| `no_passwords` | `bool` | [OIDC](#oidc) |
|
||||
| `mirror` | `string` | [Offline](#offline) |
|
||||
| `checksums` | `string` | [Offline](#offline) |
|
||||
| `nvidia_smi` | `string` | [Options](#options) |
|
||||
| `ib_root` | `string` | [Options](#options) |
|
||||
| `prefer_nvml` | `bool` | [Options](#options) |
|
||||
@@ -226,6 +228,7 @@ rather than repeated.
|
||||
| `monitoring` | `Monitoring` | [Server](#server) |
|
||||
| `tracing` | `Tracing` | [Server](#server) |
|
||||
| `mirror` | `Mirror` | [Server](#server) |
|
||||
| `offline` | `Offline` | [Server](#server) |
|
||||
| `engines` | `map[string]string` | [Server](#server) |
|
||||
| `engine_overrides` | `[]EngineOverride` | [Server](#server) |
|
||||
| `backup_dir` | `string` | [Server](#server) |
|
||||
@@ -1084,6 +1087,29 @@ PasswordSignIn off refuses passwords on the sign-in form: people come
|
||||
in through the provider only. Administrators still can, so a broken
|
||||
provider does not lock everybody out.
|
||||
|
||||
## Offline
|
||||
|
||||
### `mirror`
|
||||
|
||||
`string`
|
||||
|
||||
Mirror is the mirror's file address, as nodes reach it:
|
||||
http://10.0.0.5:8081 — the port model files are served on, not the
|
||||
management API. Set, the cluster is offline: nodes install engines
|
||||
from it and never from their publishers, releases come from it unless
|
||||
releases names another place, and no registry is asked whether a
|
||||
model has newer weights. Python engines (vLLM, SGLang) come from PyPI,
|
||||
which no bundle carries; on an offline node they are installed by hand.
|
||||
|
||||
### `checksums`
|
||||
|
||||
`string`
|
||||
|
||||
Checksums is the file of engine checksums this control plane trusts,
|
||||
kept by llmgw bundle import and llmgw bundle trust; every engine file
|
||||
a node fetches from the mirror is checked against it. Read again
|
||||
whenever it changes. Empty is /etc/llmgw/offline.sha256.
|
||||
|
||||
## Options
|
||||
|
||||
### `nvidia_smi`
|
||||
@@ -1834,6 +1860,16 @@ Empty means there is none, and /admin/mirror stays unmounted.
|
||||
|
||||
Takes a [Mirror](#mirror) block.
|
||||
|
||||
### `offline`
|
||||
|
||||
`Offline`
|
||||
|
||||
Offline is a cluster that reaches nothing outside itself: its nodes
|
||||
install engines, and upgrades fetch releases, from the mirror, out of
|
||||
a bundle carried in (llmgw bundle create, llmgw bundle import).
|
||||
|
||||
Takes a [Offline](#offline) block.
|
||||
|
||||
### `engines`
|
||||
|
||||
`map[string]string`
|
||||
|
||||
Reference in new issue
Block a user