Merge C18 offline bundles and the Helm chart
Conflicts: server.go imports (both kept); agent.pb.go regenerated with buf from the merged proto. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
commit
6f85c17fc5
51 files changed
+4177
-63
No files matched your search
@@ -52,6 +52,18 @@ cluster, a backup and adding a node change this machine or another over ssh;
|
||||
those close the screen and run in the terminal, so their output is there to
|
||||
read, and Enter brings the console back.
|
||||
|
||||
## Setting up
|
||||
|
||||
On a machine with nothing installed, the control plane is set up from one
|
||||
form (`sudo llmgw server init`): the administrator's email, other names nodes
|
||||
reach the machine by, and ticks for PostgreSQL, the model mirror, this machine
|
||||
as a node, the starter model and the guardrails' judge. **Offline bundle**
|
||||
and **Its sha256** are for a site with no internet: the path of a bundle made
|
||||
by `llmgw bundle create`, and the sha256 it printed. Given, the bundle goes
|
||||
into the mirror, every file checked, and the cluster is set up to fetch
|
||||
nothing from outside (install.md, "Offline sites"). A bundle with no sha256,
|
||||
or with the mirror unticked, is said in the form and nothing starts.
|
||||
|
||||
## Watch
|
||||
|
||||
**Overview** is `llmgw ctl status`: nodes, GPUs by class, fabrics, replicas,
|
||||
|
||||
+50
-10
@@ -126,22 +126,62 @@ your own hardware, and the answer comes back. The router talks to engines, the
|
||||
engines talk to nothing, the agents talk to the control plane. All of it is
|
||||
inside the cluster.
|
||||
|
||||
Two things can reach outside, both started by an operator rather than by a
|
||||
request:
|
||||
Around the serving path, a cluster set up the ordinary way does reach out.
|
||||
None of it carries a prompt, an answer, a key or a user — it is software and
|
||||
public model files coming in — but a review should know each one:
|
||||
|
||||
**Fetching weights.** When a model's `source` is a URL, the agent downloads it.
|
||||
Point that at an internal mirror and nothing leaves at all — which is also the
|
||||
**Installing engines.** Each node's agent downloads the llama.cpp build for its
|
||||
hardware from GitHub, checked against the checksum GitHub publishes, and vLLM
|
||||
(SGLang when asked) from PyPI, with `uv` from GitHub. On every node, when it
|
||||
joins and when the engine version changes.
|
||||
|
||||
**Releases.** `llmgw node add`, `llmgw ctl upgrade` and `llmgw upgrade` fetch
|
||||
llmgw's own release binaries from the project's release server, checked against
|
||||
its signed checksums.
|
||||
|
||||
**Fetching weights.** When a model's `source` is a URL, the agent downloads it;
|
||||
the mirror fetches what it is asked to, from Hugging Face or another URL, and
|
||||
`server init` has it fetch the starter model and the guardrails' judge. Point
|
||||
sources at the mirror and the nodes fetch only from it — which is also the
|
||||
faster arrangement, because eight nodes pulling forty gigabytes from the
|
||||
internet is a day's work and from a local mirror it is minutes. A `source` that
|
||||
is a path on disk is never fetched.
|
||||
|
||||
**`llmgw ctl plan`.** Reads a model's architecture from huggingface.co to work
|
||||
out whether it fits your hardware. An operator command run at planning time,
|
||||
not part of serving, and not run by anything automatically.
|
||||
**The registry.** With a mirror configured, the control plane asks Hugging Face
|
||||
every six hours whether the models served from it have newer files
|
||||
(`mirror.update_check`), and `/admin/mirror` searches it. `llmgw ctl plan`
|
||||
reads a model's architecture from huggingface.co to work out whether it fits
|
||||
your hardware.
|
||||
|
||||
There is no telemetry, no licence check, and no call home. The binary is
|
||||
static, built with `CGO_ENABLED=0`, and runs in a network with no route to the
|
||||
internet as long as the weights are already on disk.
|
||||
There is no telemetry, no licence check, and no call home.
|
||||
|
||||
### A cluster with no way out
|
||||
|
||||
A site with no route to the internet installs from an **offline bundle**
|
||||
instead, and then nothing at all leaves the cluster (docs/install.md, "Offline
|
||||
sites"). The bundle is made on a machine that has internet — llmgw's release
|
||||
binaries with their signed checksums, the llama.cpp builds with the checksums
|
||||
GitHub publishes, the model files with the checksums Hugging Face publishes,
|
||||
each checked as it is downloaded — carried in, and imported into the mirror.
|
||||
With `offline.mirror` set in `server.yaml`, nodes install engines from the
|
||||
mirror and never from GitHub, releases come from the mirror, and the registry
|
||||
is not asked about newer weights.
|
||||
|
||||
Verification is not weakened by any of this. A release is held to the project's
|
||||
signing key wherever it is fetched from. An engine file is held to the checksum
|
||||
recorded when the bundle was made, which the control plane keeps and hands each
|
||||
node over the agent's authenticated connection: the mirror serves plain HTTP,
|
||||
and nothing a node runs is taken on the mirror's word. The bundle itself is
|
||||
checked against the sha256 its maker printed, carried to the site apart from
|
||||
the bundle.
|
||||
|
||||
What does not come in a bundle: vLLM and SGLang, which are Python environments
|
||||
from PyPI — an offline node serves with llama.cpp unless one is installed on it
|
||||
by hand — and the operating system's own packages (the OpenMP runtime the
|
||||
engine needs, PostgreSQL when chosen), which come from the machines' package
|
||||
sources: at an offline site, the distribution mirror it already has. Searching
|
||||
the registry from `/admin/mirror` and `llmgw ctl plan` need the internet and
|
||||
do not work offline; `mirror.registry` can point the first at an internal copy.
|
||||
|
||||
## What the software cannot do for you
|
||||
|
||||
|
||||
+135
@@ -1861,6 +1861,139 @@ doctor` on a node says when the table is missing. `--no-firewall` on any of
|
||||
the four commands leaves the firewall alone; `sudo systemctl disable --now
|
||||
llmgw-firewall` removes it.
|
||||
|
||||
## Offline sites
|
||||
|
||||
Everything above downloads: the nodes fetch llama.cpp from GitHub and vLLM from
|
||||
PyPI, `node add` and upgrades fetch llmgw's releases, and the mirror fetches
|
||||
models from Hugging Face. A site with no route to the internet gets all of that
|
||||
in a **bundle**, made somewhere that has one and carried in.
|
||||
|
||||
### Making the bundle
|
||||
|
||||
On any machine with internet and llmgw — a laptop will do:
|
||||
|
||||
```
|
||||
llmgw bundle create --out llmgw-bundle --arch amd64 --tar
|
||||
```
|
||||
|
||||
That carries:
|
||||
|
||||
- **llmgw's release** for each `--arch` (default: this llmgw's own version;
|
||||
`--version` for another), with `checksums.txt` and its signature, checked
|
||||
against the project's signing key as `ctl upgrade` checks it.
|
||||
- **llama.cpp** at the version this llmgw installs, every build for each
|
||||
architecture — CPU, CUDA 13.3, CUDA 12.8 (amd64), Vulkan — each checked
|
||||
against the checksum GitHub publishes, the same check a node does online.
|
||||
`--variant cpu --variant cuda-13.3` carries only those, for a site whose
|
||||
hardware you know; `--engine llamacpp=b10991` names the version.
|
||||
- **The starter model and the guardrails' judge** that `server init` serves
|
||||
(about 1.3 GB; `--no-starter-models` leaves them out), and any
|
||||
`--model name/version=hf:owner/repo/file` you add, pinned to the commit the
|
||||
registry had and checked against the checksum it publishes.
|
||||
|
||||
It ends by printing the manifest's **sha256**. Carry that apart from the bundle
|
||||
— in the ticket, read out over the phone — because it is what says the bundle
|
||||
that arrived is the one that was made. `llmgw bundle show llmgw-bundle.tar
|
||||
--verify` reads every file back, worth doing once it is on the disk that
|
||||
carries it. Run `create` again with the same `--out` (without `--tar`) to add
|
||||
to a bundle; what is there already is not downloaded again.
|
||||
|
||||
What a bundle does **not** carry: vLLM and SGLang, which are Python
|
||||
environments of several gigabytes from PyPI. Offline nodes serve with llama.cpp;
|
||||
on a GPU node that needs vLLM, install it by hand and name it with `node add
|
||||
--vllm-bin` (the node reports vLLM as unsupported until then). Nor the
|
||||
operating system's packages — the OpenMP runtime llama.cpp needs, PostgreSQL
|
||||
with `--postgres` — which come from each machine's own package sources: at an
|
||||
offline site, its distribution mirror.
|
||||
|
||||
### Installing from it
|
||||
|
||||
On the control plane, with the bundle on a local disk:
|
||||
|
||||
```
|
||||
sudo llmgw server init --bundle /media/usb/llmgw-bundle.tar --bundle-sha256 3f2a…
|
||||
```
|
||||
|
||||
(The console's setup form has the same two fields.) Init then does what it
|
||||
always does, with these differences:
|
||||
|
||||
1. Before the control plane starts, the bundle goes into the mirror beside it:
|
||||
every file checked against the manifest, the release against the signing
|
||||
key, and only then put in place. llmgw's releases and the engines go under
|
||||
`/srv/models/_llmgw/`, the models where the mirror keeps models.
|
||||
2. The engines' checksums are written to `/etc/llmgw/offline.sha256`. That
|
||||
file, on the control plane, is what each node's engine files are held to.
|
||||
3. `server.yaml` gets
|
||||
|
||||
```yaml
|
||||
offline:
|
||||
mirror: http://10.0.0.5:8081 # the mirror's file port, as nodes reach it
|
||||
```
|
||||
|
||||
and from then on the cluster asks nothing of the internet: nodes install
|
||||
llama.cpp from the mirror, `releases` defaults to the mirror's copy (so
|
||||
`node add` and `ctl upgrade` fetch from it), and the registry is not asked
|
||||
about newer weights.
|
||||
4. The starter model and the judge are served from the mirror when the bundle
|
||||
carries them, and skipped — not waited for — when it does not. Without the
|
||||
judge, the guard policy is written without one.
|
||||
|
||||
With the mirror on another machine (`--mirror`), import the bundle there first
|
||||
and then run init with `--bundle` as above: the control plane records only the
|
||||
checksums.
|
||||
|
||||
```
|
||||
# on the mirror machine
|
||||
sudo llmgw bundle import /media/usb/llmgw-bundle.tar --sha256 3f2a…
|
||||
```
|
||||
|
||||
Nodes join as usual with `sudo llmgw node add`: it fetches the node's binary
|
||||
from the mirror and checks its signature, and the agent installs llama.cpp
|
||||
from the mirror. Nothing on the node needs the internet.
|
||||
|
||||
### How it stays checked
|
||||
|
||||
Online, a node trusts GitHub, over TLS, for the checksum of the llama.cpp it
|
||||
installs, and the project's signing key for llmgw. Offline it is the same
|
||||
strength by a different route: the checksum GitHub published is recorded when
|
||||
the bundle is made, the bundle is held to its sha256 when it is imported, and
|
||||
the control plane keeps the checksums and tells each node, over its mutually
|
||||
authenticated connection, what every file it fetches must be. The mirror serves
|
||||
plain HTTP and is never asked for a checksum: a file changed on it, or on the
|
||||
way from it, is refused by the node. Releases carry the project's signature,
|
||||
which holds whatever server they come from — so plain HTTP from the mirror is
|
||||
allowed for releases only because this build trusts a signing key.
|
||||
|
||||
### Later
|
||||
|
||||
A new llmgw release or llama.cpp version is another bundle. Make it the same
|
||||
way — `--version v0.12.0`, `--engine llamacpp=b11200` — and on the control plane
|
||||
(and the mirror, if separate):
|
||||
|
||||
```
|
||||
sudo llmgw bundle import new-bundle.tar --sha256 …
|
||||
```
|
||||
|
||||
The control plane sees the new checksums without a restart. Then set
|
||||
`engines.llamacpp` in `server.yaml` to the new version and restart
|
||||
`llmgw-server`, or run `sudo llmgw ctl upgrade v0.12.0` for the release. A
|
||||
mirror on its own machine, which is not an agent, is upgraded from its own
|
||||
copy: `sudo llmgw upgrade v0.12.0 --service llmgw-mirror --releases
|
||||
http://127.0.0.1:8081/_llmgw/releases`. A control plane whose bundle lacks the
|
||||
llama.cpp version it asks for says so in its log at start, and the node says
|
||||
which `bundle create` would carry it. More models: `--model` in a bundle, or
|
||||
copy the files into the mirror by hand — it is a directory.
|
||||
|
||||
A cluster set up online becomes offline the same way: `bundle import` on the
|
||||
mirror and control plane, then the `offline:` block in `server.yaml` and a
|
||||
restart.
|
||||
|
||||
## Kubernetes
|
||||
|
||||
The control plane runs in Kubernetes from the Helm chart in
|
||||
`deploy/helm/llmgw`; GPU nodes are best kept outside it, as ordinary hosts
|
||||
joined with `node add`. Why, and how, is in [kubernetes.md](kubernetes.md).
|
||||
|
||||
## Upgrading
|
||||
|
||||
On the control plane:
|
||||
@@ -2132,6 +2265,8 @@ requests. On a slow model, set `timeouts.generation` higher.
|
||||
wrote about
|
||||
- `deploy/mirror/README.md` — the mirror in more detail, including the layout
|
||||
and what belongs on which port
|
||||
- `docs/kubernetes.md` — the control plane in Kubernetes from the Helm chart,
|
||||
and why GPU nodes stay outside it
|
||||
- `deploy/monitoring/README.md` — Prometheus, Grafana and the alerts, with what
|
||||
each one assumes about this system
|
||||
- `docs/fabrics.md` — InfiniBand fabrics and NVLink: what llmgw needs of them,
|
||||
|
||||
@@ -0,0 +1,228 @@
|
||||
# llmgw and Kubernetes
|
||||
|
||||
The control plane runs in Kubernetes, from the Helm chart in
|
||||
`deploy/helm/llmgw`. The GPU nodes do not: they stay ordinary machines,
|
||||
joined with `llmgw node add` as on any other cluster. This page says why,
|
||||
and how the two meet.
|
||||
|
||||
## What goes where
|
||||
|
||||
| Part | Where | Why |
|
||||
|---|---|---|
|
||||
| Control plane: the API on 8080, the agents' port 7443, the scheduler | A StatefulSet, from the chart | A Go binary with a store and three ports: what Kubernetes runs well |
|
||||
| Store | A file on a PersistentVolumeClaim (SQLite), or your PostgreSQL | Either, as on a host |
|
||||
| GPU and CPU nodes | Outside, as hosts | See below |
|
||||
| Model mirror | Outside, on a machine with a large disk | Every node pulls tens of gigabytes from it over plain HTTP; a pod with a volume that size is a host in all but name |
|
||||
|
||||
### Why the nodes stay outside
|
||||
|
||||
The agent on a node is not a service that happens to use a GPU. It installs
|
||||
the engines on the machine — the llama.cpp build for its driver, vLLM in a
|
||||
Python environment — checks them against their publisher's checksums and
|
||||
runs them; it chooses which GPUs each replica gets, by index and by NVLink
|
||||
topology; it starts engines as processes that outlive its own restarts, so a
|
||||
node upgrades while it serves; it writes the machine's firewall; and it
|
||||
replaces its own binary when the cluster upgrades. All of that is root on
|
||||
the host, which is exactly what a container is built to prevent.
|
||||
|
||||
Run as a DaemonSet it would need to be privileged, with the host's network,
|
||||
process namespace and filesystem, and the NVIDIA device plugin would have to
|
||||
be kept off its GPUs — or the two would hand the same card to two owners.
|
||||
What is left is the host agent with a container's indirection around it, and
|
||||
none of the scheduling, isolation or restart guarantees that make a pod
|
||||
worth having. So the chart does not ship one, and this is the
|
||||
recommendation:
|
||||
|
||||
- **Keep GPU machines out of Kubernetes**, and add them with `node add`.
|
||||
- **If they are already Kubernetes workers**, dedicate them: taint them so
|
||||
no other pod lands there, keep the NVIDIA device plugin off them (most
|
||||
installs select nodes by label — leave the label off), and run the llmgw
|
||||
agent on the host, where `node add` puts it. llmgw then owns the GPUs, and
|
||||
Kubernetes schedules nothing that would want them.
|
||||
|
||||
## The image
|
||||
|
||||
The project publishes binaries, not an image. Build one from the repository
|
||||
and push it where the cluster pulls from — at an offline site, the internal
|
||||
registry:
|
||||
|
||||
```
|
||||
docker build -f deploy/container/Dockerfile --build-arg VERSION=v0.11.1 \
|
||||
-t registry.internal/llmgw:v0.11.1 .
|
||||
docker push registry.internal/llmgw:v0.11.1
|
||||
```
|
||||
|
||||
It is the static binary on a distroless base: no shell, running as user
|
||||
65532, with a read-only root filesystem in the pod.
|
||||
|
||||
## The agent port's certificate
|
||||
|
||||
Agents and the control plane authenticate each other with certificates from
|
||||
llmgw's own authority, and the TLS ends in the pod — so the authority's key
|
||||
stays outside the cluster, on the machine you will add nodes from, and the
|
||||
pod is given only its own certificate.
|
||||
|
||||
Decide first the address agents will dial: the `-agents` Service is a
|
||||
LoadBalancer, so give it a fixed `agentService.loadBalancerIP` or a DNS name
|
||||
that points at it. That address has to be in the certificate:
|
||||
|
||||
```
|
||||
llmgw ctl cert init-ca --dir ./pki
|
||||
llmgw ctl cert issue-server llmgw-ctl --host 10.0.0.200 --host llmgw-agents.example.com \
|
||||
--dir ./pki --out ./pki
|
||||
kubectl -n llmgw create secret generic llmgw-agent-tls \
|
||||
--from-file=ca.pem=pki/ca.pem \
|
||||
--from-file=server.pem=pki/server.pem \
|
||||
--from-file=server-key.pem=pki/server-key.pem
|
||||
```
|
||||
|
||||
Keep `pki/ca-key.pem` as you would on a host: whoever holds it can make any
|
||||
node's identity.
|
||||
|
||||
## Installing
|
||||
|
||||
```yaml
|
||||
# values.yaml
|
||||
image:
|
||||
repository: registry.internal/llmgw
|
||||
tag: v0.11.1
|
||||
agentTLS:
|
||||
existingSecret: llmgw-agent-tls
|
||||
agentService:
|
||||
loadBalancerIP: 10.0.0.200
|
||||
ingress:
|
||||
enabled: true
|
||||
className: nginx
|
||||
hosts: [{host: llm.example.com, paths: [{path: /, pathType: Prefix}]}]
|
||||
tls: [{secretName: llm-example-com, hosts: [llm.example.com]}]
|
||||
config:
|
||||
mirror:
|
||||
api: http://models.internal:8082
|
||||
# The address the router reaches nodes' engines from is a Kubernetes
|
||||
# node's, once the pod's traffic leaves the cluster: the nodes' firewalls
|
||||
# must let it in.
|
||||
firewall_networks: ["10.0.1.0/24"]
|
||||
```
|
||||
|
||||
```
|
||||
helm install llmgw deploy/helm/llmgw -n llmgw -f values.yaml
|
||||
```
|
||||
|
||||
`config` is `server.yaml`: any key of [reference.md](reference.md), laid
|
||||
under what the chart sets itself — the listeners, where the certificates,
|
||||
the admin token and the store are mounted. A changed `config` restarts the
|
||||
pod.
|
||||
|
||||
**The store.** By default SQLite, on a 10 GiB claim. For PostgreSQL, put its
|
||||
address in a Secret and point the chart at it:
|
||||
|
||||
```
|
||||
kubectl -n llmgw create secret generic llmgw-store \
|
||||
--from-literal=dsn='postgres://llmgw:…@pg.internal:5432/llmgw?sslmode=verify-full'
|
||||
```
|
||||
|
||||
```yaml
|
||||
store:
|
||||
type: postgres
|
||||
postgres:
|
||||
existingSecret: llmgw-store
|
||||
```
|
||||
|
||||
**One control plane.** The chart runs one replica. Two against PostgreSQL
|
||||
would elect a leader and keep a standby, but a standby serves nothing and is
|
||||
never ready, so a StatefulSet rollout would wait on it for ever. Kubernetes
|
||||
replaces the one pod within seconds when it goes, and agents reconnect on
|
||||
their own; the replicas on the nodes keep serving meanwhile.
|
||||
|
||||
**Probes.** Alive is `/health`. Ready is `/leader`, which on PostgreSQL waits
|
||||
for the lease of the pod it replaced to run out — not `/ready`, which waits
|
||||
for a model replica, and would keep a cluster with no node yet from ever
|
||||
accepting one.
|
||||
|
||||
**Monitoring.** The metrics port, 9091, serves metrics and nothing else;
|
||||
`serviceMonitor.enabled: true` makes a ServiceMonitor for the Prometheus
|
||||
Operator, and `deploy/monitoring` holds the rules and dashboards.
|
||||
|
||||
## Using it
|
||||
|
||||
The admin API listens on the pod's loopback, as on a host — it can dump keys
|
||||
from the heap when profiling is on. Reach it through the pod:
|
||||
|
||||
```
|
||||
kubectl -n llmgw exec -it statefulset/llmgw -- llmgw users create you@example.com --password '…'
|
||||
kubectl -n llmgw exec -it statefulset/llmgw -- llmgw users role you@example.com admin
|
||||
kubectl -n llmgw exec -it statefulset/llmgw -- llmgw ctl status
|
||||
```
|
||||
|
||||
or from your machine, through a port-forward:
|
||||
|
||||
```
|
||||
kubectl -n llmgw port-forward statefulset/llmgw 9090:9090 &
|
||||
export LLMGW_ADMIN_TOKEN=$(kubectl -n llmgw get secret llmgw-admin -o jsonpath='{.data.admin-token}' | base64 -d)
|
||||
llmgw ctl status
|
||||
```
|
||||
|
||||
The token is made by the chart and kept across upgrades
|
||||
(`adminToken.existingSecret` brings your own).
|
||||
|
||||
## Adding nodes
|
||||
|
||||
From the machine holding `pki/`, with the port-forward above running and
|
||||
`LLMGW_ADMIN_TOKEN` set:
|
||||
|
||||
```
|
||||
sudo -E llmgw node add ubuntu@gpu01 \
|
||||
--server https://10.0.0.200:7443 --pki ./pki --admin 127.0.0.1:9090
|
||||
```
|
||||
|
||||
`--server` is the address in the certificate. The node is installed, checked
|
||||
and started over ssh exactly as on any cluster ([node-onboarding.md](node-onboarding.md)),
|
||||
and dials the control plane in the pod. Its firewall comes from
|
||||
`firewall_networks` in the server configuration `node add` reads (`--config`,
|
||||
`/etc/llmgw/server.yaml` by default): copy the chart's out of the cluster
|
||||
first, or give `--no-firewall` and keep the engines' ports open to the
|
||||
Kubernetes nodes by hand.
|
||||
|
||||
```
|
||||
kubectl -n llmgw get configmap llmgw -o jsonpath='{.data.server\.yaml}' > server.yaml
|
||||
sudo -E llmgw node add ubuntu@gpu01 --config ./server.yaml --server https://10.0.0.200:7443 \
|
||||
--pki ./pki --admin 127.0.0.1:9090
|
||||
```
|
||||
|
||||
## Offline
|
||||
|
||||
An offline site ([install.md](install.md), "Offline sites") works the same
|
||||
way with the control plane in a pod. Import the bundle on the mirror's
|
||||
machine with `llmgw bundle import`, record its engines' checksums in a
|
||||
file, and give that file to the chart as a ConfigMap:
|
||||
|
||||
```
|
||||
llmgw bundle trust llmgw-bundle.tar --sha256 3f2a… --file ./offline.sha256
|
||||
kubectl -n llmgw create configmap llmgw-offline --from-file=offline.sha256
|
||||
```
|
||||
|
||||
```yaml
|
||||
offlineChecksums:
|
||||
configMap: llmgw-offline
|
||||
config:
|
||||
offline:
|
||||
mirror: http://models.internal:8081
|
||||
```
|
||||
|
||||
A later bundle: `bundle trust` into the same file, and replace the ConfigMap
|
||||
(`kubectl create configmap … --dry-run=client -o yaml | kubectl apply -f -`).
|
||||
The control plane reads the file again when it changes; there is no restart.
|
||||
|
||||
## What has and has not been tried
|
||||
|
||||
The chart is checked with `helm lint` and `helm template` for SQLite,
|
||||
PostgreSQL, an Ingress, a ServiceMonitor and an offline cluster, and refuses
|
||||
what would not work — no image, no certificate, a PostgreSQL store without
|
||||
its address, more than one replica. The image has been run the way the
|
||||
chart runs it — read-only root, user 65532, the same files mounted at the
|
||||
same paths, SQLite on a volume — and answered `/health`, `/leader` and
|
||||
`/metrics`, with `llmgw ctl` and `llmgw users` working through `exec`.
|
||||
|
||||
It has **not** been installed on a Kubernetes cluster, and no node has yet
|
||||
joined a control plane running in one. Until it has, a control plane on a
|
||||
host is the well-trodden path.
|
||||
@@ -139,6 +139,8 @@ rather than repeated.
|
||||
| `roles` | `map[string][]string` | [OIDC](#oidc) |
|
||||
| `memberships` | `map[string]string` | [OIDC](#oidc) |
|
||||
| `no_passwords` | `bool` | [OIDC](#oidc) |
|
||||
| `mirror` | `string` | [Offline](#offline) |
|
||||
| `checksums` | `string` | [Offline](#offline) |
|
||||
| `nvidia_smi` | `string` | [Options](#options) |
|
||||
| `ib_root` | `string` | [Options](#options) |
|
||||
| `prefer_nvml` | `bool` | [Options](#options) |
|
||||
@@ -257,6 +259,7 @@ rather than repeated.
|
||||
| `monitoring` | `Monitoring` | [Server](#server) |
|
||||
| `tracing` | `obs.Tracing` | [Server](#server) |
|
||||
| `mirror` | `Mirror` | [Server](#server) |
|
||||
| `offline` | `Offline` | [Server](#server) |
|
||||
| `engines` | `map[string]string` | [Server](#server) |
|
||||
| `engine_overrides` | `[]EngineOverride` | [Server](#server) |
|
||||
| `backup_dir` | `string` | [Server](#server) |
|
||||
@@ -1273,6 +1276,29 @@ PasswordSignIn off refuses passwords on the sign-in form: people come
|
||||
in through the provider only. Administrators still can, so a broken
|
||||
provider does not lock everybody out.
|
||||
|
||||
## Offline
|
||||
|
||||
### `mirror`
|
||||
|
||||
`string`
|
||||
|
||||
Mirror is the mirror's file address, as nodes reach it:
|
||||
http://10.0.0.5:8081 — the port model files are served on, not the
|
||||
management API. Set, the cluster is offline: nodes install engines
|
||||
from it and never from their publishers, releases come from it unless
|
||||
releases names another place, and no registry is asked whether a
|
||||
model has newer weights. Python engines (vLLM, SGLang) come from PyPI,
|
||||
which no bundle carries; on an offline node they are installed by hand.
|
||||
|
||||
### `checksums`
|
||||
|
||||
`string`
|
||||
|
||||
Checksums is the file of engine checksums this control plane trusts,
|
||||
kept by llmgw bundle import and llmgw bundle trust; every engine file
|
||||
a node fetches from the mirror is checked against it. Read again
|
||||
whenever it changes. Empty is /etc/llmgw/offline.sha256.
|
||||
|
||||
## Options
|
||||
|
||||
### `nvidia_smi`
|
||||
@@ -2128,6 +2154,16 @@ Empty means there is none, and /admin/mirror stays unmounted.
|
||||
|
||||
Takes a [Mirror](#mirror) block.
|
||||
|
||||
### `offline`
|
||||
|
||||
`Offline`
|
||||
|
||||
Offline is a cluster that reaches nothing outside itself: its nodes
|
||||
install engines, and upgrades fetch releases, from the mirror, out of
|
||||
a bundle carried in (llmgw bundle create, llmgw bundle import).
|
||||
|
||||
Takes a [Offline](#offline) block.
|
||||
|
||||
### `engines`
|
||||
|
||||
`map[string]string`
|
||||
|
||||
Reference in new issue
Block a user