llmgw/docs/kubernetes.md

# llmgw and Kubernetes

The control plane runs in Kubernetes, from the Helm chart in
`deploy/helm/llmgw`. The GPU nodes do not: they stay ordinary machines,
joined with `llmgw node add` as on any other cluster. This page says why,
and how the two meet.

## What goes where

| Part | Where | Why |
|---|---|---|
| Control plane: the API on 8080, the agents' port 7443, the scheduler | A StatefulSet, from the chart | A Go binary with a store and three ports: what Kubernetes runs well |
| Store | A file on a PersistentVolumeClaim (SQLite), or your PostgreSQL | Either, as on a host |
| GPU and CPU nodes | Outside, as hosts | See below |
| Model mirror | Outside, on a machine with a large disk | Every node pulls tens of gigabytes from it over plain HTTP; a pod with a volume that size is a host in all but name |

### Why the nodes stay outside

The agent on a node is not a service that happens to use a GPU. It installs
the engines on the machine — the llama.cpp build for its driver, vLLM in a
Python environment — checks them against their publisher's checksums and
runs them; it chooses which GPUs each replica gets, by index and by NVLink
topology; it starts engines as processes that outlive its own restarts, so a
node upgrades while it serves; it writes the machine's firewall; and it
replaces its own binary when the cluster upgrades. All of that is root on
the host, which is exactly what a container is built to prevent.

Run as a DaemonSet it would need to be privileged, with the host's network,
process namespace and filesystem, and the NVIDIA device plugin would have to
be kept off its GPUs — or the two would hand the same card to two owners.
What is left is the host agent with a container's indirection around it, and
none of the scheduling, isolation or restart guarantees that make a pod
worth having. So the chart does not ship one, and this is the
recommendation:

- **Keep GPU machines out of Kubernetes**, and add them with `node add`.
- **If they are already Kubernetes workers**, dedicate them: taint them so
  no other pod lands there, keep the NVIDIA device plugin off them (most
  installs select nodes by label — leave the label off), and run the llmgw
  agent on the host, where `node add` puts it. llmgw then owns the GPUs, and
  Kubernetes schedules nothing that would want them.

## The image

The project publishes binaries, not an image. Build one from the repository
and push it where the cluster pulls from — at an offline site, the internal
registry:

```
docker build -f deploy/container/Dockerfile --build-arg VERSION=v0.11.1 \
  -t registry.internal/llmgw:v0.11.1 .
docker push registry.internal/llmgw:v0.11.1
```

It is the static binary on a distroless base: no shell, running as user
65532, with a read-only root filesystem in the pod.

## The agent port's certificate

Agents and the control plane authenticate each other with certificates from
llmgw's own authority, and the TLS ends in the pod — so the authority's key
stays outside the cluster, on the machine you will add nodes from, and the
pod is given only its own certificate.

Decide first the address agents will dial: the `-agents` Service is a
LoadBalancer, so give it a fixed `agentService.loadBalancerIP` or a DNS name
that points at it. That address has to be in the certificate:

```
llmgw ctl cert init-ca --dir ./pki
llmgw ctl cert issue-server llmgw-ctl --host 10.0.0.200 --host llmgw-agents.example.com \
  --dir ./pki --out ./pki
kubectl -n llmgw create secret generic llmgw-agent-tls \
  --from-file=ca.pem=pki/ca.pem \
  --from-file=server.pem=pki/server.pem \
  --from-file=server-key.pem=pki/server-key.pem
```

Keep `pki/ca-key.pem` as you would on a host: whoever holds it can make any
node's identity.

## Installing

```yaml
# values.yaml
image:
  repository: registry.internal/llmgw
  tag: v0.11.1
agentTLS:
  existingSecret: llmgw-agent-tls
agentService:
  loadBalancerIP: 10.0.0.200
ingress:
  enabled: true
  className: nginx
  hosts: [{host: llm.example.com, paths: [{path: /, pathType: Prefix}]}]
  tls: [{secretName: llm-example-com, hosts: [llm.example.com]}]
config:
  mirror:
    api: http://models.internal:8082
  # The address the router reaches nodes' engines from is a Kubernetes
  # node's, once the pod's traffic leaves the cluster: the nodes' firewalls
  # must let it in.
  firewall_networks: ["10.0.1.0/24"]
```

```
helm install llmgw deploy/helm/llmgw -n llmgw -f values.yaml
```

`config` is `server.yaml`: any key of [reference.md](reference.md), laid
under what the chart sets itself — the listeners, where the certificates,
the admin token and the store are mounted. A changed `config` restarts the
pod.

**The store.** By default SQLite, on a 10 GiB claim. For PostgreSQL, put its
address in a Secret and point the chart at it:

```
kubectl -n llmgw create secret generic llmgw-store \
  --from-literal=dsn='postgres://llmgw:…@pg.internal:5432/llmgw?sslmode=verify-full'
```

```yaml
store:
  type: postgres
  postgres:
    existingSecret: llmgw-store
```

**One control plane.** The chart runs one replica. Two against PostgreSQL
would elect a leader and keep a standby, but a standby serves nothing and is
never ready, so a StatefulSet rollout would wait on it for ever. Kubernetes
replaces the one pod within seconds when it goes, and agents reconnect on
their own; the replicas on the nodes keep serving meanwhile.

**Probes.** Alive is `/health`. Ready is `/leader`, which on PostgreSQL waits
for the lease of the pod it replaced to run out — not `/ready`, which waits
for a model replica, and would keep a cluster with no node yet from ever
accepting one.

**Monitoring.** The metrics port, 9091, serves metrics and nothing else;
`serviceMonitor.enabled: true` makes a ServiceMonitor for the Prometheus
Operator, and `deploy/monitoring` holds the rules and dashboards.

## Using it

The admin API listens on the pod's loopback, as on a host — it can dump keys
from the heap when profiling is on. Reach it through the pod:

```
kubectl -n llmgw exec -it statefulset/llmgw -- llmgw users create you@example.com --password '…'
kubectl -n llmgw exec -it statefulset/llmgw -- llmgw users role you@example.com admin
kubectl -n llmgw exec -it statefulset/llmgw -- llmgw ctl status
```

or from your machine, through a port-forward:

```
kubectl -n llmgw port-forward statefulset/llmgw 9090:9090 &
export LLMGW_ADMIN_TOKEN=$(kubectl -n llmgw get secret llmgw-admin -o jsonpath='{.data.admin-token}' | base64 -d)
llmgw ctl status
```

The token is made by the chart and kept across upgrades
(`adminToken.existingSecret` brings your own).

## Adding nodes

From the machine holding `pki/`, with the port-forward above running and
`LLMGW_ADMIN_TOKEN` set:

```
sudo -E llmgw node add ubuntu@gpu01 \
  --server https://10.0.0.200:7443 --pki ./pki --admin 127.0.0.1:9090
```

`--server` is the address in the certificate. The node is installed, checked
and started over ssh exactly as on any cluster ([node-onboarding.md](node-onboarding.md)),
and dials the control plane in the pod. Its firewall comes from
`firewall_networks` in the server configuration `node add` reads (`--config`,
`/etc/llmgw/server.yaml` by default): copy the chart's out of the cluster
first, or give `--no-firewall` and keep the engines' ports open to the
Kubernetes nodes by hand.

```
kubectl -n llmgw get configmap llmgw -o jsonpath='{.data.server\.yaml}' > server.yaml
sudo -E llmgw node add ubuntu@gpu01 --config ./server.yaml --server https://10.0.0.200:7443 \
  --pki ./pki --admin 127.0.0.1:9090
```

## Offline

An offline site ([install.md](install.md), "Offline sites") works the same
way with the control plane in a pod. Import the bundle on the mirror's
machine with `llmgw bundle import`, record its engines' checksums in a
file, and give that file to the chart as a ConfigMap:

```
llmgw bundle trust llmgw-bundle.tar --sha256 3f2a… --file ./offline.sha256
kubectl -n llmgw create configmap llmgw-offline --from-file=offline.sha256
```

```yaml
offlineChecksums:
  configMap: llmgw-offline
config:
  offline:
    mirror: http://models.internal:8081
```

A later bundle: `bundle trust` into the same file, and replace the ConfigMap
(`kubectl create configmap … --dry-run=client -o yaml | kubectl apply -f -`).
The control plane reads the file again when it changes; there is no restart.

## What has and has not been tried

The chart is checked with `helm lint` and `helm template` for SQLite,
PostgreSQL, an Ingress, a ServiceMonitor and an offline cluster, and refuses
what would not work — no image, no certificate, a PostgreSQL store without
its address, more than one replica. The image has been run the way the
chart runs it — read-only root, user 65532, the same files mounted at the
same paths, SQLite on a volume — and answered `/health`, `/leader` and
`/metrics`, with `llmgw ctl` and `llmgw users` working through `exec`.

It has **not** been installed on a Kubernetes cluster, and no node has yet
joined a control plane running in one. Until it has, a control plane on a
host is the well-trodden path.