Merge C18 offline bundles and the Helm chart

Conflicts: server.go imports (both kept); agent.pb.go regenerated with buf
from the merged proto.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-05 00:15:47 +02:00
commit 6f85c17fc5
51 files changed
+4177 -63

No files matched your search

+40
View File
@@ -0,0 +1,40 @@
# The llmgw control plane as a container image, for deploy/helm/llmgw.
#
# The project publishes binaries, not an image, so this is built where the
# image is to live:
#
# docker build -f deploy/container/Dockerfile --build-arg VERSION=v0.11.1 \
# -t registry.internal/llmgw:v0.11.1 .
# docker push registry.internal/llmgw:v0.11.1
#
# The binary is static (CGO_ENABLED=0, as every release is), so the image is
# that one file on a base with nothing else in it: no shell, no package
# manager. `kubectl exec ... -- llmgw ctl status` still works, because it
# runs the binary rather than a shell.
#
# Only the control plane runs from it. An agent installs engines on its
# machine, manages its GPUs and supervises processes there as root, which is
# what a container is built to prevent: docs/kubernetes.md.
FROM golang:1.26 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
ARG VERSION=dev
RUN CGO_ENABLED=0 go build -trimpath \
-ldflags "-s -w -X gitea.mynodes.xyz/ilker/llmgw/internal/version.Version=${VERSION}" \
-o /out/llmgw ./cmd/llmgw && \
mkdir -p /out/state
FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /out/llmgw /usr/local/bin/llmgw
# The chart mounts server.yaml, the certificates and the token under
# /etc/llmgw, and the SQLite store, when it is one, under /var/lib/llmgw —
# owned by the user the server runs as, so a volume Docker fills from the
# image is writable without Kubernetes' fsGroup.
COPY --from=build --chown=65532:65532 /out/state /var/lib/llmgw
USER 65532:65532
EXPOSE 8080 7443 9091
ENTRYPOINT ["/usr/local/bin/llmgw"]
CMD ["server", "--config", "/etc/llmgw/server.yaml"]
+17
View File
@@ -0,0 +1,17 @@
apiVersion: v2
name: llmgw
description: >-
The llmgw control plane — the OpenAI-compatible gateway, the router and the
scheduler that places models on GPU and CPU nodes. The nodes themselves run
outside Kubernetes; see docs/kubernetes.md.
type: application
# The chart's own version; appVersion is the llmgw release the image holds.
version: 0.1.0
appVersion: "v0.11.1"
home: https://gitea.mynodes.xyz/ilker/llmgw
sources:
- https://gitea.mynodes.xyz/ilker/llmgw
keywords:
- llm
- gateway
- inference
+27
View File
@@ -0,0 +1,27 @@
The llmgw control plane is starting as {{ include "llmgw.fullname" . }}.
The admin API listens on the pod's loopback only. To reach it from here:
kubectl -n {{ .Release.Namespace }} port-forward statefulset/{{ include "llmgw.fullname" . }} 9090:9090 &
export LLMGW_ADMIN_TOKEN=$(kubectl -n {{ .Release.Namespace }} get secret {{ include "llmgw.adminSecret" . }} \
-o jsonpath='{.data.{{ if .Values.adminToken.existingSecret }}{{ .Values.adminToken.secretKey }}{{ else }}admin-token{{ end }}}' | base64 -d)
llmgw ctl status
The first administrator:
kubectl -n {{ .Release.Namespace }} exec -it statefulset/{{ include "llmgw.fullname" . }} -- \
llmgw users create you@example.com --password '<a password>'
kubectl -n {{ .Release.Namespace }} exec -it statefulset/{{ include "llmgw.fullname" . }} -- \
llmgw users role you@example.com admin
Agents connect to the {{ include "llmgw.fullname" . }}-agents Service on port {{ .Values.agentService.port }}:
kubectl -n {{ .Release.Namespace }} get service {{ include "llmgw.fullname" . }}-agents
Its address must be in the certificate in {{ .Values.agentTLS.existingSecret }}. Nodes join from
the machine holding llmgw's authority:
sudo llmgw node add ubuntu@gpu01 --server https://<that address>:{{ .Values.agentService.port }} \
--pki ./pki --admin 127.0.0.1:9090
docs/kubernetes.md has the rest, and why the nodes stay outside Kubernetes.
+74
View File
@@ -0,0 +1,74 @@
{{/* The release's name for everything the chart makes. */}}
{{- define "llmgw.fullname" -}}
{{- if contains .Chart.Name .Release.Name -}}
{{- .Release.Name | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- printf "%s-%s" .Release.Name .Chart.Name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{- end -}}
{{- define "llmgw.selectorLabels" -}}
app.kubernetes.io/name: {{ .Chart.Name }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end -}}
{{- define "llmgw.labels" -}}
{{ include "llmgw.selectorLabels" . }}
app.kubernetes.io/version: {{ .Values.image.tag | default .Chart.AppVersion | quote }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
helm.sh/chart: {{ printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" }}
{{- end -}}
{{- define "llmgw.image" -}}
{{- if not .Values.image.repository -}}
{{- fail "image.repository: the image built from deploy/container/Dockerfile, pushed where the cluster can pull it (the project publishes binaries, not an image)" -}}
{{- end -}}
{{- printf "%s:%s" .Values.image.repository (.Values.image.tag | default .Chart.AppVersion) -}}
{{- end -}}
{{- define "llmgw.adminSecret" -}}
{{- .Values.adminToken.existingSecret | default (printf "%s-admin" (include "llmgw.fullname" .)) -}}
{{- end -}}
{{/*
server.yaml: what the chart decides — listeners, where the certificates,
the token and the store are mounted — laid over .Values.config. The chart's
keys win, because each names a port or a path the pod is built around.
*/}}
{{- define "llmgw.serverConfig" -}}
{{- $store := "/var/lib/llmgw/llmgw.db" -}}
{{- if eq .Values.store.type "postgres" -}}
{{- $store = "@/etc/llmgw/store/dsn" -}}
{{- end -}}
{{- $base := dict
"listen" ":8080"
"admin_listen" "127.0.0.1:9090"
"metrics_listen" ":9091"
"agent_listen" ":7443"
"admin_token_file" "/etc/llmgw/admin/admin-token"
"agent_tls" (dict "ca" "/etc/llmgw/tls/ca.pem" "cert" "/etc/llmgw/tls/server.pem" "key" "/etc/llmgw/tls/server-key.pem")
"auth" (dict "store" $store)
-}}
{{- if .Values.offlineChecksums.configMap -}}
{{- $_ := set $base "offline" (dict "checksums" "/etc/llmgw/offline/offline.sha256") -}}
{{- end -}}
{{- $cfg := mustMergeOverwrite (deepCopy (.Values.config | default dict)) $base -}}
# Made by the llmgw Helm chart from its values. Every key: docs/reference.md
{{ toYaml $cfg }}
{{- end -}}
{{/* Refuses values that would make a pod that cannot work. */}}
{{- define "llmgw.validate" -}}
{{- if not (has .Values.store.type (list "sqlite" "postgres")) -}}
{{- fail (printf "store.type %q: sqlite or postgres" .Values.store.type) -}}
{{- end -}}
{{- if ne (int .Values.replicas) 1 -}}
{{- fail "replicas: this chart runs one control plane; values.yaml says why" -}}
{{- end -}}
{{- if and (eq .Values.store.type "postgres") (not .Values.store.postgres.existingSecret) -}}
{{- fail "store.postgres.existingSecret: a Secret holding the PostgreSQL address" -}}
{{- end -}}
{{- if not .Values.agentTLS.existingSecret -}}
{{- fail "agentTLS.existingSecret: a Secret with ca.pem, server.pem and server-key.pem from llmgw's authority (see values.yaml)" -}}
{{- end -}}
{{- end -}}
@@ -0,0 +1,10 @@
{{- include "llmgw.validate" . -}}
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ include "llmgw.fullname" . }}
labels:
{{- include "llmgw.labels" . | nindent 4 }}
data:
server.yaml: |
{{- include "llmgw.serverConfig" . | nindent 4 }}
+37
View File
@@ -0,0 +1,37 @@
{{- if .Values.ingress.enabled }}
# The OpenAI-compatible API, the portal and the chat page on port 8080. Never
# the agent port: its mutual TLS cannot pass through an HTTP proxy.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: {{ include "llmgw.fullname" . }}
labels:
{{- include "llmgw.labels" . | nindent 4 }}
{{- with .Values.ingress.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
spec:
{{- with .Values.ingress.className }}
ingressClassName: {{ . }}
{{- end }}
{{- with .Values.ingress.tls }}
tls:
{{- toYaml . | nindent 4 }}
{{- end }}
rules:
{{- range .Values.ingress.hosts }}
- host: {{ .host | quote }}
http:
paths:
{{- range .paths }}
- path: {{ .path }}
pathType: {{ .pathType }}
backend:
service:
name: {{ include "llmgw.fullname" $ }}
port:
name: api
{{- end }}
{{- end }}
{{- end }}
@@ -0,0 +1,19 @@
{{- if not .Values.adminToken.existingSecret }}
{{- $name := include "llmgw.adminSecret" . }}
{{- $token := randAlphaNum 64 }}
{{- /* Kept across upgrades: a new token on every upgrade would lock out every console holding the old one. */}}
{{- with lookup "v1" "Secret" .Release.Namespace $name }}
{{- $token = index .data "admin-token" | b64dec }}
{{- end }}
apiVersion: v1
kind: Secret
metadata:
name: {{ $name }}
labels:
{{- include "llmgw.labels" . | nindent 4 }}
annotations:
helm.sh/resource-policy: keep
type: Opaque
data:
admin-token: {{ $token | b64enc | quote }}
{{- end }}
@@ -0,0 +1,21 @@
{{- if .Values.serviceMonitor.enabled }}
# For the Prometheus Operator: the metrics port, which serves metrics and
# nothing else. deploy/monitoring holds the rules and dashboards that read them.
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: {{ include "llmgw.fullname" . }}
labels:
{{- include "llmgw.labels" . | nindent 4 }}
{{- with .Values.serviceMonitor.labels }}
{{- toYaml . | nindent 4 }}
{{- end }}
spec:
selector:
matchLabels:
{{- include "llmgw.selectorLabels" . | nindent 6 }}
endpoints:
- port: metrics
path: /metrics
interval: {{ .Values.serviceMonitor.interval }}
{{- end }}
+70
View File
@@ -0,0 +1,70 @@
# The API and the metrics, for the cluster and an Ingress.
apiVersion: v1
kind: Service
metadata:
name: {{ include "llmgw.fullname" . }}
labels:
{{- include "llmgw.labels" . | nindent 4 }}
{{- with .Values.service.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
spec:
type: {{ .Values.service.type }}
selector:
{{- include "llmgw.selectorLabels" . | nindent 4 }}
ports:
- name: api
port: {{ .Values.service.port }}
targetPort: api
- name: metrics
port: {{ .Values.service.metricsPort }}
targetPort: metrics
---
# Where agents on the nodes connect. Mutual TLS ends in the pod: whatever is
# in between must pass TCP through untouched.
apiVersion: v1
kind: Service
metadata:
name: {{ include "llmgw.fullname" . }}-agents
labels:
{{- include "llmgw.labels" . | nindent 4 }}
{{- with .Values.agentService.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
spec:
type: {{ .Values.agentService.type }}
{{- if and (eq .Values.agentService.type "LoadBalancer") .Values.agentService.loadBalancerIP }}
loadBalancerIP: {{ .Values.agentService.loadBalancerIP }}
{{- end }}
{{- with .Values.agentService.loadBalancerSourceRanges }}
loadBalancerSourceRanges:
{{- toYaml . | nindent 4 }}
{{- end }}
{{- if has .Values.agentService.type (list "LoadBalancer" "NodePort") }}
externalTrafficPolicy: {{ .Values.agentService.externalTrafficPolicy }}
{{- end }}
selector:
{{- include "llmgw.selectorLabels" . | nindent 4 }}
ports:
- name: agents
port: {{ .Values.agentService.port }}
targetPort: agents
---
# The StatefulSet's own, giving each pod a stable name.
apiVersion: v1
kind: Service
metadata:
name: {{ include "llmgw.fullname" . }}-headless
labels:
{{- include "llmgw.labels" . | nindent 4 }}
spec:
clusterIP: None
publishNotReadyAddresses: true
selector:
{{- include "llmgw.selectorLabels" . | nindent 4 }}
ports:
- name: api
port: 8080
targetPort: api
@@ -0,0 +1,181 @@
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: {{ include "llmgw.fullname" . }}
labels:
{{- include "llmgw.labels" . | nindent 4 }}
spec:
# One control plane: Kubernetes replaces it within seconds when it goes,
# and the agents reconnect on their own. See replicas in values.yaml.
replicas: 1
serviceName: {{ include "llmgw.fullname" . }}-headless
selector:
matchLabels:
{{- include "llmgw.selectorLabels" . | nindent 6 }}
template:
metadata:
labels:
{{- include "llmgw.selectorLabels" . | nindent 8 }}
{{- with .Values.podLabels }}
{{- toYaml . | nindent 8 }}
{{- end }}
annotations:
# server.yaml is read at start: a changed one restarts the pod.
checksum/config: {{ include "llmgw.serverConfig" . | sha256sum }}
{{- with .Values.podAnnotations }}
{{- toYaml . | nindent 8 }}
{{- end }}
spec:
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
# Agents reconnect within seconds of a control plane going; a long
# grace period only delays the one that replaces it.
terminationGracePeriodSeconds: 30
securityContext:
runAsNonRoot: true
runAsUser: 65532
runAsGroup: 65532
fsGroup: 65532
seccompProfile:
type: RuntimeDefault
containers:
- name: llmgw
image: {{ include "llmgw.image" . }}
imagePullPolicy: {{ .Values.image.pullPolicy }}
args: ["server", "--config", "/etc/llmgw/server.yaml"]
env:
# For `kubectl exec ... -- llmgw ctl`, which reads the token
# from here; the server reads admin_token_file.
- name: LLMGW_ADMIN_TOKEN
valueFrom:
secretKeyRef:
name: {{ include "llmgw.adminSecret" . }}
key: {{ if .Values.adminToken.existingSecret }}{{ .Values.adminToken.secretKey }}{{ else }}admin-token{{ end }}
ports:
- name: api
containerPort: 8080
- name: agents
containerPort: 7443
- name: metrics
containerPort: 9091
# Up is /health. Ready is /leader: a control plane holds the
# agents' connections and places models only once it leads — on
# PostgreSQL, once the lease of the one it replaced has run out. A
# cluster with no node yet is still ready: /ready would wait for a
# replica, and nodes could then never connect.
startupProbe:
httpGet: {path: /health, port: api}
periodSeconds: 5
failureThreshold: 60
livenessProbe:
httpGet: {path: /health, port: api}
periodSeconds: 10
failureThreshold: 6
readinessProbe:
httpGet: {path: /leader, port: api}
periodSeconds: 5
failureThreshold: 2
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
resources:
{{- toYaml .Values.resources | nindent 12 }}
volumeMounts:
- name: config
mountPath: /etc/llmgw/server.yaml
subPath: server.yaml
readOnly: true
- name: agent-tls
mountPath: /etc/llmgw/tls
readOnly: true
- name: admin-token
mountPath: /etc/llmgw/admin
readOnly: true
{{- if eq .Values.store.type "postgres" }}
- name: store
mountPath: /etc/llmgw/store
readOnly: true
{{- end }}
{{- if .Values.offlineChecksums.configMap }}
- name: offline
mountPath: /etc/llmgw/offline
readOnly: true
{{- end }}
- name: data
mountPath: /var/lib/llmgw
- name: tmp
mountPath: /tmp
volumes:
- name: config
configMap:
name: {{ include "llmgw.fullname" . }}
- name: agent-tls
secret:
secretName: {{ .Values.agentTLS.existingSecret }}
defaultMode: 0440
items:
- {key: ca.pem, path: ca.pem}
- {key: server.pem, path: server.pem}
- {key: server-key.pem, path: server-key.pem}
- name: admin-token
secret:
secretName: {{ include "llmgw.adminSecret" . }}
defaultMode: 0440
items:
- key: {{ if .Values.adminToken.existingSecret }}{{ .Values.adminToken.secretKey }}{{ else }}admin-token{{ end }}
path: admin-token
{{- if eq .Values.store.type "postgres" }}
- name: store
secret:
secretName: {{ .Values.store.postgres.existingSecret }}
defaultMode: 0440
items:
- key: {{ .Values.store.postgres.secretKey }}
path: dsn
{{- end }}
{{- if .Values.offlineChecksums.configMap }}
- name: offline
configMap:
name: {{ .Values.offlineChecksums.configMap }}
items:
- key: {{ .Values.offlineChecksums.key }}
path: offline.sha256
{{- end }}
- name: tmp
emptyDir: {}
{{- if eq .Values.store.type "postgres" }}
# The store is PostgreSQL; nothing here outlives the pod.
- name: data
emptyDir: {}
{{- end }}
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- if eq .Values.store.type "sqlite" }}
volumeClaimTemplates:
- metadata:
name: data
labels:
{{- include "llmgw.selectorLabels" . | nindent 10 }}
spec:
accessModes: ["ReadWriteOnce"]
{{- with .Values.store.sqlite.storageClass }}
storageClassName: {{ . }}
{{- end }}
resources:
requests:
storage: {{ .Values.store.sqlite.size }}
{{- end }}
+132
View File
@@ -0,0 +1,132 @@
# The llmgw control plane in Kubernetes. docs/kubernetes.md says what this
# runs, what it does not, and how nodes outside the cluster join it.
image:
# Where you pushed the image built from deploy/container/Dockerfile. The
# project publishes binaries, not an image, so this has no default.
repository: ""
# Empty takes the chart's appVersion.
tag: ""
pullPolicy: IfNotPresent
imagePullSecrets: []
# One control plane, always. Two against PostgreSQL would elect a leader and
# keep a standby, but a standby serves nothing and is never ready, so a
# rollout would wait on it for ever. Until the chart handles that, it runs
# one: Kubernetes replaces it within seconds when it goes, and the agents
# reconnect on their own.
replicas: 1
store:
# sqlite: a file on a PersistentVolumeClaim, for one replica.
# postgres: a PostgreSQL you run, its address in a Secret.
type: sqlite
sqlite:
size: 10Gi
storageClass: ""
postgres:
# A Secret holding the address, postgres://user:password@host:5432/llmgw?sslmode=verify-full
existingSecret: ""
secretKey: dsn
# The agent port's certificate. It comes from llmgw's own authority, not a
# public one — agents present certificates from the same authority — so it
# is made outside the cluster, where the authority's key stays:
#
# llmgw ctl cert init-ca --dir ./pki
# llmgw ctl cert issue-server llmgw-ctl --host <agent address> --dir ./pki --out ./pki
# kubectl create secret generic llmgw-agent-tls \
# --from-file=ca.pem=pki/ca.pem \
# --from-file=server.pem=pki/server.pem \
# --from-file=server-key.pem=pki/server-key.pem
#
# --host is every address agents dial: agentService's load balancer address
# or DNS name.
agentTLS:
existingSecret: ""
# The admin API's token (llmgw ctl, the console). Empty makes one, kept in a
# Secret across upgrades.
adminToken:
existingSecret: ""
secretKey: admin-token
# server.yaml beyond what the chart sets (listeners, TLS, store, admin
# token): any key of docs/reference.md, laid over the chart's. An offline
# cluster sets offline.mirror here, and offlineChecksums below.
config:
control:
enabled: true
auth:
enabled: true
signup_open: false
default_rpm: 60
default_tpm: 200000
default_price:
prompt_per_1k: 1000
completion_per_1k: 1000
max_key_ttl: 2160h
# mirror:
# api: https://models.internal:8082
# offline:
# mirror: http://models.internal:8081
# firewall_networks: ["10.42.0.0/16"] # the pods' network, so nodes' engines answer the router
# On an offline cluster, a ConfigMap or Secret holding the engine checksums
# 'llmgw bundle trust' writes (offline.sha256), mounted where offline.checksums
# points. Empty mounts none.
offlineChecksums:
configMap: ""
key: offline.sha256
# The OpenAI-compatible API and the metrics, inside the cluster.
service:
type: ClusterIP
port: 8080
metricsPort: 9091
annotations: {}
# Where agents connect, with mutual TLS. Nodes run outside the cluster, so
# this is a load balancer by default; the TLS ends in the pod, never at a
# proxy in between.
agentService:
type: LoadBalancer
port: 7443
annotations: {}
loadBalancerIP: ""
loadBalancerSourceRanges: []
# Keeps the agents' addresses as they are, which the node's identity does
# not need but the logs and the audit read better with.
externalTrafficPolicy: Local
ingress:
enabled: false
className: ""
annotations: {}
hosts:
- host: llm.example.com
paths:
- path: /
pathType: Prefix
tls: []
# - secretName: llm-example-com
# hosts: [llm.example.com]
# A ServiceMonitor for the Prometheus Operator, scraping the metrics port.
serviceMonitor:
enabled: false
interval: 30s
labels: {}
resources:
requests:
cpu: 500m
memory: 512Mi
limits:
memory: 2Gi
nodeSelector: {}
tolerations: []
affinity: {}
podAnnotations: {}
podLabels: {}
+7
View File
@@ -25,6 +25,13 @@ An HTTP server serving a directory. That is the whole design.
model.safetensors
```
At an offline site the mirror also holds what the nodes would otherwise fetch
from the internet — llmgw's releases and the llama.cpp builds — under
`_llmgw/releases/` and `_llmgw/engines/`, put there by `llmgw bundle import`
(docs/install.md, "Offline sites"). They are not models: `mirror list` and
`mirror unused` pass over them, and a node checks each against what the
control plane says it must be, not against anything the mirror says.
The date in the path is the point of the layout, not decoration. A model
repository that changes upstream changes what you are running, and the first
sign is a prompt that worked last week answering differently this week with