llmgw/docs/kubernetes.md

llmgw and Kubernetes

The control plane runs in Kubernetes, from the Helm chart in deploy/helm/llmgw. The GPU nodes do not: they stay ordinary machines, joined with llmgw node add as on any other cluster. This page says why, and how the two meet.

What goes where

Part	Where	Why
Control plane: the API on 8080, the agents' port 7443, the scheduler	A StatefulSet, from the chart	A Go binary with a store and three ports: what Kubernetes runs well
Store	A file on a PersistentVolumeClaim (SQLite), or your PostgreSQL	Either, as on a host
GPU and CPU nodes	Outside, as hosts	See below
Model mirror	Outside, on a machine with a large disk	Every node pulls tens of gigabytes from it over plain HTTP; a pod with a volume that size is a host in all but name

Why the nodes stay outside

The agent on a node is not a service that happens to use a GPU. It installs the engines on the machine — the llama.cpp build for its driver, vLLM in a Python environment — checks them against their publisher's checksums and runs them; it chooses which GPUs each replica gets, by index and by NVLink topology; it starts engines as processes that outlive its own restarts, so a node upgrades while it serves; it writes the machine's firewall; and it replaces its own binary when the cluster upgrades. All of that is root on the host, which is exactly what a container is built to prevent.

Run as a DaemonSet it would need to be privileged, with the host's network, process namespace and filesystem, and the NVIDIA device plugin would have to be kept off its GPUs — or the two would hand the same card to two owners. What is left is the host agent with a container's indirection around it, and none of the scheduling, isolation or restart guarantees that make a pod worth having. So the chart does not ship one, and this is the recommendation:

• Keep GPU machines out of Kubernetes, and add them with node add.
• If they are already Kubernetes workers, dedicate them: taint them so no other pod lands there, keep the NVIDIA device plugin off them (most installs select nodes by label — leave the label off), and run the llmgw agent on the host, where node add puts it. llmgw then owns the GPUs, and Kubernetes schedules nothing that would want them.

The image

The project publishes binaries, not an image. Build one from the repository and push it where the cluster pulls from — at an offline site, the internal registry:

docker build -f deploy/container/Dockerfile --build-arg VERSION=v0.11.1 \
  -t registry.internal/llmgw:v0.11.1 .
docker push registry.internal/llmgw:v0.11.1

It is the static binary on a distroless base: no shell, running as user 65532, with a read-only root filesystem in the pod.

The agent port's certificate

Agents and the control plane authenticate each other with certificates from llmgw's own authority, and the TLS ends in the pod — so the authority's key stays outside the cluster, on the machine you will add nodes from, and the pod is given only its own certificate.

Decide first the address agents will dial: the -agents Service is a LoadBalancer, so give it a fixed agentService.loadBalancerIP or a DNS name that points at it. That address has to be in the certificate:

llmgw ctl cert init-ca --dir ./pki
llmgw ctl cert issue-server llmgw-ctl --host 10.0.0.200 --host llmgw-agents.example.com \
  --dir ./pki --out ./pki
kubectl -n llmgw create secret generic llmgw-agent-tls \
  --from-file=ca.pem=pki/ca.pem \
  --from-file=server.pem=pki/server.pem \
  --from-file=server-key.pem=pki/server-key.pem

Keep pki/ca-key.pem as you would on a host: whoever holds it can make any node's identity.

Installing

# values.yaml
image:
  repository: registry.internal/llmgw
  tag: v0.11.1
agentTLS:
  existingSecret: llmgw-agent-tls
agentService:
  loadBalancerIP: 10.0.0.200
ingress:
  enabled: true
  className: nginx
  hosts: [{host: llm.example.com, paths: [{path: /, pathType: Prefix}]}]
  tls: [{secretName: llm-example-com, hosts: [llm.example.com]}]
config:
  mirror:
    api: http://models.internal:8082
  # The address the router reaches nodes' engines from is a Kubernetes
  # node's, once the pod's traffic leaves the cluster: the nodes' firewalls
  # must let it in.
  firewall_networks: ["10.0.1.0/24"]

helm install llmgw deploy/helm/llmgw -n llmgw -f values.yaml

config is server.yaml: any key of reference.md, laid under what the chart sets itself — the listeners, where the certificates, the admin token and the store are mounted. A changed config restarts the pod.

The store. By default SQLite, on a 10 GiB claim. For PostgreSQL, put its address in a Secret and point the chart at it:

kubectl -n llmgw create secret generic llmgw-store \
  --from-literal=dsn='postgres://llmgw:…@pg.internal:5432/llmgw?sslmode=verify-full'

store:
  type: postgres
  postgres:
    existingSecret: llmgw-store

One control plane. The chart runs one replica. Two against PostgreSQL would elect a leader and keep a standby, but a standby serves nothing and is never ready, so a StatefulSet rollout would wait on it for ever. Kubernetes replaces the one pod within seconds when it goes, and agents reconnect on their own; the replicas on the nodes keep serving meanwhile.

Probes. Alive is /health. Ready is /leader, which on PostgreSQL waits for the lease of the pod it replaced to run out — not /ready, which waits for a model replica, and would keep a cluster with no node yet from ever accepting one.

Monitoring. The metrics port, 9091, serves metrics and nothing else; serviceMonitor.enabled: true makes a ServiceMonitor for the Prometheus Operator, and deploy/monitoring holds the rules and dashboards.

Using it

The admin API listens on the pod's loopback, as on a host — it can dump keys from the heap when profiling is on. Reach it through the pod:

kubectl -n llmgw exec -it statefulset/llmgw -- llmgw users create you@example.com --password '…'
kubectl -n llmgw exec -it statefulset/llmgw -- llmgw users role you@example.com admin
kubectl -n llmgw exec -it statefulset/llmgw -- llmgw ctl status

or from your machine, through a port-forward:

kubectl -n llmgw port-forward statefulset/llmgw 9090:9090 &
export LLMGW_ADMIN_TOKEN=$(kubectl -n llmgw get secret llmgw-admin -o jsonpath='{.data.admin-token}' | base64 -d)
llmgw ctl status

The token is made by the chart and kept across upgrades (adminToken.existingSecret brings your own).

Adding nodes

From the machine holding pki/, with the port-forward above running and LLMGW_ADMIN_TOKEN set:

sudo -E llmgw node add ubuntu@gpu01 \
  --server https://10.0.0.200:7443 --pki ./pki --admin 127.0.0.1:9090

--server is the address in the certificate. The node is installed, checked and started over ssh exactly as on any cluster (node-onboarding.md), and dials the control plane in the pod. Its firewall comes from firewall_networks in the server configuration node add reads (--config, /etc/llmgw/server.yaml by default): copy the chart's out of the cluster first, or give --no-firewall and keep the engines' ports open to the Kubernetes nodes by hand.

kubectl -n llmgw get configmap llmgw -o jsonpath='{.data.server\.yaml}' > server.yaml
sudo -E llmgw node add ubuntu@gpu01 --config ./server.yaml --server https://10.0.0.200:7443 \
  --pki ./pki --admin 127.0.0.1:9090

Offline

An offline site (install.md, "Offline sites") works the same way with the control plane in a pod. Import the bundle on the mirror's machine with llmgw bundle import, record its engines' checksums in a file, and give that file to the chart as a ConfigMap:

llmgw bundle trust llmgw-bundle.tar --sha256 3f2a… --file ./offline.sha256
kubectl -n llmgw create configmap llmgw-offline --from-file=offline.sha256

offlineChecksums:
  configMap: llmgw-offline
config:
  offline:
    mirror: http://models.internal:8081

A later bundle: bundle trust into the same file, and replace the ConfigMap (kubectl create configmap … --dry-run=client -o yaml | kubectl apply -f -). The control plane reads the file again when it changes; there is no restart.

What has and has not been tried

The chart is checked with helm lint and helm template for SQLite, PostgreSQL, an Ingress, a ServiceMonitor and an offline cluster, and refuses what would not work — no image, no certificate, a PostgreSQL store without its address, more than one replica. The image has been run the way the chart runs it — read-only root, user 65532, the same files mounted at the same paths, SQLite on a volume — and answered /health, /leader and /metrics, with llmgw ctl and llmgw users working through exec.

It has not been installed on a Kubernetes cluster, and no node has yet joined a control plane running in one. Until it has, a control plane on a host is the well-trodden path.