Merge C18 offline bundles and the Helm chart
Conflicts: server.go imports (both kept); agent.pb.go regenerated with buf from the merged proto. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
commit
6f85c17fc5
51 files changed
+4177
-63
No files matched your search
@@ -0,0 +1,40 @@
|
||||
# The llmgw control plane as a container image, for deploy/helm/llmgw.
|
||||
#
|
||||
# The project publishes binaries, not an image, so this is built where the
|
||||
# image is to live:
|
||||
#
|
||||
# docker build -f deploy/container/Dockerfile --build-arg VERSION=v0.11.1 \
|
||||
# -t registry.internal/llmgw:v0.11.1 .
|
||||
# docker push registry.internal/llmgw:v0.11.1
|
||||
#
|
||||
# The binary is static (CGO_ENABLED=0, as every release is), so the image is
|
||||
# that one file on a base with nothing else in it: no shell, no package
|
||||
# manager. `kubectl exec ... -- llmgw ctl status` still works, because it
|
||||
# runs the binary rather than a shell.
|
||||
#
|
||||
# Only the control plane runs from it. An agent installs engines on its
|
||||
# machine, manages its GPUs and supervises processes there as root, which is
|
||||
# what a container is built to prevent: docs/kubernetes.md.
|
||||
|
||||
FROM golang:1.26 AS build
|
||||
WORKDIR /src
|
||||
COPY go.mod go.sum ./
|
||||
RUN go mod download
|
||||
COPY . .
|
||||
ARG VERSION=dev
|
||||
RUN CGO_ENABLED=0 go build -trimpath \
|
||||
-ldflags "-s -w -X gitea.mynodes.xyz/ilker/llmgw/internal/version.Version=${VERSION}" \
|
||||
-o /out/llmgw ./cmd/llmgw && \
|
||||
mkdir -p /out/state
|
||||
|
||||
FROM gcr.io/distroless/static-debian12:nonroot
|
||||
COPY --from=build /out/llmgw /usr/local/bin/llmgw
|
||||
# The chart mounts server.yaml, the certificates and the token under
|
||||
# /etc/llmgw, and the SQLite store, when it is one, under /var/lib/llmgw —
|
||||
# owned by the user the server runs as, so a volume Docker fills from the
|
||||
# image is writable without Kubernetes' fsGroup.
|
||||
COPY --from=build --chown=65532:65532 /out/state /var/lib/llmgw
|
||||
USER 65532:65532
|
||||
EXPOSE 8080 7443 9091
|
||||
ENTRYPOINT ["/usr/local/bin/llmgw"]
|
||||
CMD ["server", "--config", "/etc/llmgw/server.yaml"]
|
||||
@@ -0,0 +1,17 @@
|
||||
apiVersion: v2
|
||||
name: llmgw
|
||||
description: >-
|
||||
The llmgw control plane — the OpenAI-compatible gateway, the router and the
|
||||
scheduler that places models on GPU and CPU nodes. The nodes themselves run
|
||||
outside Kubernetes; see docs/kubernetes.md.
|
||||
type: application
|
||||
# The chart's own version; appVersion is the llmgw release the image holds.
|
||||
version: 0.1.0
|
||||
appVersion: "v0.11.1"
|
||||
home: https://gitea.mynodes.xyz/ilker/llmgw
|
||||
sources:
|
||||
- https://gitea.mynodes.xyz/ilker/llmgw
|
||||
keywords:
|
||||
- llm
|
||||
- gateway
|
||||
- inference
|
||||
@@ -0,0 +1,27 @@
|
||||
The llmgw control plane is starting as {{ include "llmgw.fullname" . }}.
|
||||
|
||||
The admin API listens on the pod's loopback only. To reach it from here:
|
||||
|
||||
kubectl -n {{ .Release.Namespace }} port-forward statefulset/{{ include "llmgw.fullname" . }} 9090:9090 &
|
||||
export LLMGW_ADMIN_TOKEN=$(kubectl -n {{ .Release.Namespace }} get secret {{ include "llmgw.adminSecret" . }} \
|
||||
-o jsonpath='{.data.{{ if .Values.adminToken.existingSecret }}{{ .Values.adminToken.secretKey }}{{ else }}admin-token{{ end }}}' | base64 -d)
|
||||
llmgw ctl status
|
||||
|
||||
The first administrator:
|
||||
|
||||
kubectl -n {{ .Release.Namespace }} exec -it statefulset/{{ include "llmgw.fullname" . }} -- \
|
||||
llmgw users create you@example.com --password '<a password>'
|
||||
kubectl -n {{ .Release.Namespace }} exec -it statefulset/{{ include "llmgw.fullname" . }} -- \
|
||||
llmgw users role you@example.com admin
|
||||
|
||||
Agents connect to the {{ include "llmgw.fullname" . }}-agents Service on port {{ .Values.agentService.port }}:
|
||||
|
||||
kubectl -n {{ .Release.Namespace }} get service {{ include "llmgw.fullname" . }}-agents
|
||||
|
||||
Its address must be in the certificate in {{ .Values.agentTLS.existingSecret }}. Nodes join from
|
||||
the machine holding llmgw's authority:
|
||||
|
||||
sudo llmgw node add ubuntu@gpu01 --server https://<that address>:{{ .Values.agentService.port }} \
|
||||
--pki ./pki --admin 127.0.0.1:9090
|
||||
|
||||
docs/kubernetes.md has the rest, and why the nodes stay outside Kubernetes.
|
||||
@@ -0,0 +1,74 @@
|
||||
{{/* The release's name for everything the chart makes. */}}
|
||||
{{- define "llmgw.fullname" -}}
|
||||
{{- if contains .Chart.Name .Release.Name -}}
|
||||
{{- .Release.Name | trunc 63 | trimSuffix "-" -}}
|
||||
{{- else -}}
|
||||
{{- printf "%s-%s" .Release.Name .Chart.Name | trunc 63 | trimSuffix "-" -}}
|
||||
{{- end -}}
|
||||
{{- end -}}
|
||||
|
||||
{{- define "llmgw.selectorLabels" -}}
|
||||
app.kubernetes.io/name: {{ .Chart.Name }}
|
||||
app.kubernetes.io/instance: {{ .Release.Name }}
|
||||
{{- end -}}
|
||||
|
||||
{{- define "llmgw.labels" -}}
|
||||
{{ include "llmgw.selectorLabels" . }}
|
||||
app.kubernetes.io/version: {{ .Values.image.tag | default .Chart.AppVersion | quote }}
|
||||
app.kubernetes.io/managed-by: {{ .Release.Service }}
|
||||
helm.sh/chart: {{ printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" }}
|
||||
{{- end -}}
|
||||
|
||||
{{- define "llmgw.image" -}}
|
||||
{{- if not .Values.image.repository -}}
|
||||
{{- fail "image.repository: the image built from deploy/container/Dockerfile, pushed where the cluster can pull it (the project publishes binaries, not an image)" -}}
|
||||
{{- end -}}
|
||||
{{- printf "%s:%s" .Values.image.repository (.Values.image.tag | default .Chart.AppVersion) -}}
|
||||
{{- end -}}
|
||||
|
||||
{{- define "llmgw.adminSecret" -}}
|
||||
{{- .Values.adminToken.existingSecret | default (printf "%s-admin" (include "llmgw.fullname" .)) -}}
|
||||
{{- end -}}
|
||||
|
||||
{{/*
|
||||
server.yaml: what the chart decides — listeners, where the certificates,
|
||||
the token and the store are mounted — laid over .Values.config. The chart's
|
||||
keys win, because each names a port or a path the pod is built around.
|
||||
*/}}
|
||||
{{- define "llmgw.serverConfig" -}}
|
||||
{{- $store := "/var/lib/llmgw/llmgw.db" -}}
|
||||
{{- if eq .Values.store.type "postgres" -}}
|
||||
{{- $store = "@/etc/llmgw/store/dsn" -}}
|
||||
{{- end -}}
|
||||
{{- $base := dict
|
||||
"listen" ":8080"
|
||||
"admin_listen" "127.0.0.1:9090"
|
||||
"metrics_listen" ":9091"
|
||||
"agent_listen" ":7443"
|
||||
"admin_token_file" "/etc/llmgw/admin/admin-token"
|
||||
"agent_tls" (dict "ca" "/etc/llmgw/tls/ca.pem" "cert" "/etc/llmgw/tls/server.pem" "key" "/etc/llmgw/tls/server-key.pem")
|
||||
"auth" (dict "store" $store)
|
||||
-}}
|
||||
{{- if .Values.offlineChecksums.configMap -}}
|
||||
{{- $_ := set $base "offline" (dict "checksums" "/etc/llmgw/offline/offline.sha256") -}}
|
||||
{{- end -}}
|
||||
{{- $cfg := mustMergeOverwrite (deepCopy (.Values.config | default dict)) $base -}}
|
||||
# Made by the llmgw Helm chart from its values. Every key: docs/reference.md
|
||||
{{ toYaml $cfg }}
|
||||
{{- end -}}
|
||||
|
||||
{{/* Refuses values that would make a pod that cannot work. */}}
|
||||
{{- define "llmgw.validate" -}}
|
||||
{{- if not (has .Values.store.type (list "sqlite" "postgres")) -}}
|
||||
{{- fail (printf "store.type %q: sqlite or postgres" .Values.store.type) -}}
|
||||
{{- end -}}
|
||||
{{- if ne (int .Values.replicas) 1 -}}
|
||||
{{- fail "replicas: this chart runs one control plane; values.yaml says why" -}}
|
||||
{{- end -}}
|
||||
{{- if and (eq .Values.store.type "postgres") (not .Values.store.postgres.existingSecret) -}}
|
||||
{{- fail "store.postgres.existingSecret: a Secret holding the PostgreSQL address" -}}
|
||||
{{- end -}}
|
||||
{{- if not .Values.agentTLS.existingSecret -}}
|
||||
{{- fail "agentTLS.existingSecret: a Secret with ca.pem, server.pem and server-key.pem from llmgw's authority (see values.yaml)" -}}
|
||||
{{- end -}}
|
||||
{{- end -}}
|
||||
@@ -0,0 +1,10 @@
|
||||
{{- include "llmgw.validate" . -}}
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: {{ include "llmgw.fullname" . }}
|
||||
labels:
|
||||
{{- include "llmgw.labels" . | nindent 4 }}
|
||||
data:
|
||||
server.yaml: |
|
||||
{{- include "llmgw.serverConfig" . | nindent 4 }}
|
||||
@@ -0,0 +1,37 @@
|
||||
{{- if .Values.ingress.enabled }}
|
||||
# The OpenAI-compatible API, the portal and the chat page on port 8080. Never
|
||||
# the agent port: its mutual TLS cannot pass through an HTTP proxy.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: {{ include "llmgw.fullname" . }}
|
||||
labels:
|
||||
{{- include "llmgw.labels" . | nindent 4 }}
|
||||
{{- with .Values.ingress.annotations }}
|
||||
annotations:
|
||||
{{- toYaml . | nindent 4 }}
|
||||
{{- end }}
|
||||
spec:
|
||||
{{- with .Values.ingress.className }}
|
||||
ingressClassName: {{ . }}
|
||||
{{- end }}
|
||||
{{- with .Values.ingress.tls }}
|
||||
tls:
|
||||
{{- toYaml . | nindent 4 }}
|
||||
{{- end }}
|
||||
rules:
|
||||
{{- range .Values.ingress.hosts }}
|
||||
- host: {{ .host | quote }}
|
||||
http:
|
||||
paths:
|
||||
{{- range .paths }}
|
||||
- path: {{ .path }}
|
||||
pathType: {{ .pathType }}
|
||||
backend:
|
||||
service:
|
||||
name: {{ include "llmgw.fullname" $ }}
|
||||
port:
|
||||
name: api
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
@@ -0,0 +1,19 @@
|
||||
{{- if not .Values.adminToken.existingSecret }}
|
||||
{{- $name := include "llmgw.adminSecret" . }}
|
||||
{{- $token := randAlphaNum 64 }}
|
||||
{{- /* Kept across upgrades: a new token on every upgrade would lock out every console holding the old one. */}}
|
||||
{{- with lookup "v1" "Secret" .Release.Namespace $name }}
|
||||
{{- $token = index .data "admin-token" | b64dec }}
|
||||
{{- end }}
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: {{ $name }}
|
||||
labels:
|
||||
{{- include "llmgw.labels" . | nindent 4 }}
|
||||
annotations:
|
||||
helm.sh/resource-policy: keep
|
||||
type: Opaque
|
||||
data:
|
||||
admin-token: {{ $token | b64enc | quote }}
|
||||
{{- end }}
|
||||
@@ -0,0 +1,21 @@
|
||||
{{- if .Values.serviceMonitor.enabled }}
|
||||
# For the Prometheus Operator: the metrics port, which serves metrics and
|
||||
# nothing else. deploy/monitoring holds the rules and dashboards that read them.
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: {{ include "llmgw.fullname" . }}
|
||||
labels:
|
||||
{{- include "llmgw.labels" . | nindent 4 }}
|
||||
{{- with .Values.serviceMonitor.labels }}
|
||||
{{- toYaml . | nindent 4 }}
|
||||
{{- end }}
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
{{- include "llmgw.selectorLabels" . | nindent 6 }}
|
||||
endpoints:
|
||||
- port: metrics
|
||||
path: /metrics
|
||||
interval: {{ .Values.serviceMonitor.interval }}
|
||||
{{- end }}
|
||||
@@ -0,0 +1,70 @@
|
||||
# The API and the metrics, for the cluster and an Ingress.
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: {{ include "llmgw.fullname" . }}
|
||||
labels:
|
||||
{{- include "llmgw.labels" . | nindent 4 }}
|
||||
{{- with .Values.service.annotations }}
|
||||
annotations:
|
||||
{{- toYaml . | nindent 4 }}
|
||||
{{- end }}
|
||||
spec:
|
||||
type: {{ .Values.service.type }}
|
||||
selector:
|
||||
{{- include "llmgw.selectorLabels" . | nindent 4 }}
|
||||
ports:
|
||||
- name: api
|
||||
port: {{ .Values.service.port }}
|
||||
targetPort: api
|
||||
- name: metrics
|
||||
port: {{ .Values.service.metricsPort }}
|
||||
targetPort: metrics
|
||||
---
|
||||
# Where agents on the nodes connect. Mutual TLS ends in the pod: whatever is
|
||||
# in between must pass TCP through untouched.
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: {{ include "llmgw.fullname" . }}-agents
|
||||
labels:
|
||||
{{- include "llmgw.labels" . | nindent 4 }}
|
||||
{{- with .Values.agentService.annotations }}
|
||||
annotations:
|
||||
{{- toYaml . | nindent 4 }}
|
||||
{{- end }}
|
||||
spec:
|
||||
type: {{ .Values.agentService.type }}
|
||||
{{- if and (eq .Values.agentService.type "LoadBalancer") .Values.agentService.loadBalancerIP }}
|
||||
loadBalancerIP: {{ .Values.agentService.loadBalancerIP }}
|
||||
{{- end }}
|
||||
{{- with .Values.agentService.loadBalancerSourceRanges }}
|
||||
loadBalancerSourceRanges:
|
||||
{{- toYaml . | nindent 4 }}
|
||||
{{- end }}
|
||||
{{- if has .Values.agentService.type (list "LoadBalancer" "NodePort") }}
|
||||
externalTrafficPolicy: {{ .Values.agentService.externalTrafficPolicy }}
|
||||
{{- end }}
|
||||
selector:
|
||||
{{- include "llmgw.selectorLabels" . | nindent 4 }}
|
||||
ports:
|
||||
- name: agents
|
||||
port: {{ .Values.agentService.port }}
|
||||
targetPort: agents
|
||||
---
|
||||
# The StatefulSet's own, giving each pod a stable name.
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: {{ include "llmgw.fullname" . }}-headless
|
||||
labels:
|
||||
{{- include "llmgw.labels" . | nindent 4 }}
|
||||
spec:
|
||||
clusterIP: None
|
||||
publishNotReadyAddresses: true
|
||||
selector:
|
||||
{{- include "llmgw.selectorLabels" . | nindent 4 }}
|
||||
ports:
|
||||
- name: api
|
||||
port: 8080
|
||||
targetPort: api
|
||||
@@ -0,0 +1,181 @@
|
||||
apiVersion: apps/v1
|
||||
kind: StatefulSet
|
||||
metadata:
|
||||
name: {{ include "llmgw.fullname" . }}
|
||||
labels:
|
||||
{{- include "llmgw.labels" . | nindent 4 }}
|
||||
spec:
|
||||
# One control plane: Kubernetes replaces it within seconds when it goes,
|
||||
# and the agents reconnect on their own. See replicas in values.yaml.
|
||||
replicas: 1
|
||||
serviceName: {{ include "llmgw.fullname" . }}-headless
|
||||
selector:
|
||||
matchLabels:
|
||||
{{- include "llmgw.selectorLabels" . | nindent 6 }}
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
{{- include "llmgw.selectorLabels" . | nindent 8 }}
|
||||
{{- with .Values.podLabels }}
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
annotations:
|
||||
# server.yaml is read at start: a changed one restarts the pod.
|
||||
checksum/config: {{ include "llmgw.serverConfig" . | sha256sum }}
|
||||
{{- with .Values.podAnnotations }}
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
spec:
|
||||
{{- with .Values.imagePullSecrets }}
|
||||
imagePullSecrets:
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
# Agents reconnect within seconds of a control plane going; a long
|
||||
# grace period only delays the one that replaces it.
|
||||
terminationGracePeriodSeconds: 30
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 65532
|
||||
runAsGroup: 65532
|
||||
fsGroup: 65532
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
containers:
|
||||
- name: llmgw
|
||||
image: {{ include "llmgw.image" . }}
|
||||
imagePullPolicy: {{ .Values.image.pullPolicy }}
|
||||
args: ["server", "--config", "/etc/llmgw/server.yaml"]
|
||||
env:
|
||||
# For `kubectl exec ... -- llmgw ctl`, which reads the token
|
||||
# from here; the server reads admin_token_file.
|
||||
- name: LLMGW_ADMIN_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: {{ include "llmgw.adminSecret" . }}
|
||||
key: {{ if .Values.adminToken.existingSecret }}{{ .Values.adminToken.secretKey }}{{ else }}admin-token{{ end }}
|
||||
ports:
|
||||
- name: api
|
||||
containerPort: 8080
|
||||
- name: agents
|
||||
containerPort: 7443
|
||||
- name: metrics
|
||||
containerPort: 9091
|
||||
# Up is /health. Ready is /leader: a control plane holds the
|
||||
# agents' connections and places models only once it leads — on
|
||||
# PostgreSQL, once the lease of the one it replaced has run out. A
|
||||
# cluster with no node yet is still ready: /ready would wait for a
|
||||
# replica, and nodes could then never connect.
|
||||
startupProbe:
|
||||
httpGet: {path: /health, port: api}
|
||||
periodSeconds: 5
|
||||
failureThreshold: 60
|
||||
livenessProbe:
|
||||
httpGet: {path: /health, port: api}
|
||||
periodSeconds: 10
|
||||
failureThreshold: 6
|
||||
readinessProbe:
|
||||
httpGet: {path: /leader, port: api}
|
||||
periodSeconds: 5
|
||||
failureThreshold: 2
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop: ["ALL"]
|
||||
resources:
|
||||
{{- toYaml .Values.resources | nindent 12 }}
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /etc/llmgw/server.yaml
|
||||
subPath: server.yaml
|
||||
readOnly: true
|
||||
- name: agent-tls
|
||||
mountPath: /etc/llmgw/tls
|
||||
readOnly: true
|
||||
- name: admin-token
|
||||
mountPath: /etc/llmgw/admin
|
||||
readOnly: true
|
||||
{{- if eq .Values.store.type "postgres" }}
|
||||
- name: store
|
||||
mountPath: /etc/llmgw/store
|
||||
readOnly: true
|
||||
{{- end }}
|
||||
{{- if .Values.offlineChecksums.configMap }}
|
||||
- name: offline
|
||||
mountPath: /etc/llmgw/offline
|
||||
readOnly: true
|
||||
{{- end }}
|
||||
- name: data
|
||||
mountPath: /var/lib/llmgw
|
||||
- name: tmp
|
||||
mountPath: /tmp
|
||||
volumes:
|
||||
- name: config
|
||||
configMap:
|
||||
name: {{ include "llmgw.fullname" . }}
|
||||
- name: agent-tls
|
||||
secret:
|
||||
secretName: {{ .Values.agentTLS.existingSecret }}
|
||||
defaultMode: 0440
|
||||
items:
|
||||
- {key: ca.pem, path: ca.pem}
|
||||
- {key: server.pem, path: server.pem}
|
||||
- {key: server-key.pem, path: server-key.pem}
|
||||
- name: admin-token
|
||||
secret:
|
||||
secretName: {{ include "llmgw.adminSecret" . }}
|
||||
defaultMode: 0440
|
||||
items:
|
||||
- key: {{ if .Values.adminToken.existingSecret }}{{ .Values.adminToken.secretKey }}{{ else }}admin-token{{ end }}
|
||||
path: admin-token
|
||||
{{- if eq .Values.store.type "postgres" }}
|
||||
- name: store
|
||||
secret:
|
||||
secretName: {{ .Values.store.postgres.existingSecret }}
|
||||
defaultMode: 0440
|
||||
items:
|
||||
- key: {{ .Values.store.postgres.secretKey }}
|
||||
path: dsn
|
||||
{{- end }}
|
||||
{{- if .Values.offlineChecksums.configMap }}
|
||||
- name: offline
|
||||
configMap:
|
||||
name: {{ .Values.offlineChecksums.configMap }}
|
||||
items:
|
||||
- key: {{ .Values.offlineChecksums.key }}
|
||||
path: offline.sha256
|
||||
{{- end }}
|
||||
- name: tmp
|
||||
emptyDir: {}
|
||||
{{- if eq .Values.store.type "postgres" }}
|
||||
# The store is PostgreSQL; nothing here outlives the pod.
|
||||
- name: data
|
||||
emptyDir: {}
|
||||
{{- end }}
|
||||
{{- with .Values.nodeSelector }}
|
||||
nodeSelector:
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
{{- with .Values.tolerations }}
|
||||
tolerations:
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
{{- with .Values.affinity }}
|
||||
affinity:
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
{{- if eq .Values.store.type "sqlite" }}
|
||||
volumeClaimTemplates:
|
||||
- metadata:
|
||||
name: data
|
||||
labels:
|
||||
{{- include "llmgw.selectorLabels" . | nindent 10 }}
|
||||
spec:
|
||||
accessModes: ["ReadWriteOnce"]
|
||||
{{- with .Values.store.sqlite.storageClass }}
|
||||
storageClassName: {{ . }}
|
||||
{{- end }}
|
||||
resources:
|
||||
requests:
|
||||
storage: {{ .Values.store.sqlite.size }}
|
||||
{{- end }}
|
||||
@@ -0,0 +1,132 @@
|
||||
# The llmgw control plane in Kubernetes. docs/kubernetes.md says what this
|
||||
# runs, what it does not, and how nodes outside the cluster join it.
|
||||
|
||||
image:
|
||||
# Where you pushed the image built from deploy/container/Dockerfile. The
|
||||
# project publishes binaries, not an image, so this has no default.
|
||||
repository: ""
|
||||
# Empty takes the chart's appVersion.
|
||||
tag: ""
|
||||
pullPolicy: IfNotPresent
|
||||
imagePullSecrets: []
|
||||
|
||||
# One control plane, always. Two against PostgreSQL would elect a leader and
|
||||
# keep a standby, but a standby serves nothing and is never ready, so a
|
||||
# rollout would wait on it for ever. Until the chart handles that, it runs
|
||||
# one: Kubernetes replaces it within seconds when it goes, and the agents
|
||||
# reconnect on their own.
|
||||
replicas: 1
|
||||
|
||||
store:
|
||||
# sqlite: a file on a PersistentVolumeClaim, for one replica.
|
||||
# postgres: a PostgreSQL you run, its address in a Secret.
|
||||
type: sqlite
|
||||
sqlite:
|
||||
size: 10Gi
|
||||
storageClass: ""
|
||||
postgres:
|
||||
# A Secret holding the address, postgres://user:password@host:5432/llmgw?sslmode=verify-full
|
||||
existingSecret: ""
|
||||
secretKey: dsn
|
||||
|
||||
# The agent port's certificate. It comes from llmgw's own authority, not a
|
||||
# public one — agents present certificates from the same authority — so it
|
||||
# is made outside the cluster, where the authority's key stays:
|
||||
#
|
||||
# llmgw ctl cert init-ca --dir ./pki
|
||||
# llmgw ctl cert issue-server llmgw-ctl --host <agent address> --dir ./pki --out ./pki
|
||||
# kubectl create secret generic llmgw-agent-tls \
|
||||
# --from-file=ca.pem=pki/ca.pem \
|
||||
# --from-file=server.pem=pki/server.pem \
|
||||
# --from-file=server-key.pem=pki/server-key.pem
|
||||
#
|
||||
# --host is every address agents dial: agentService's load balancer address
|
||||
# or DNS name.
|
||||
agentTLS:
|
||||
existingSecret: ""
|
||||
|
||||
# The admin API's token (llmgw ctl, the console). Empty makes one, kept in a
|
||||
# Secret across upgrades.
|
||||
adminToken:
|
||||
existingSecret: ""
|
||||
secretKey: admin-token
|
||||
|
||||
# server.yaml beyond what the chart sets (listeners, TLS, store, admin
|
||||
# token): any key of docs/reference.md, laid over the chart's. An offline
|
||||
# cluster sets offline.mirror here, and offlineChecksums below.
|
||||
config:
|
||||
control:
|
||||
enabled: true
|
||||
auth:
|
||||
enabled: true
|
||||
signup_open: false
|
||||
default_rpm: 60
|
||||
default_tpm: 200000
|
||||
default_price:
|
||||
prompt_per_1k: 1000
|
||||
completion_per_1k: 1000
|
||||
max_key_ttl: 2160h
|
||||
# mirror:
|
||||
# api: https://models.internal:8082
|
||||
# offline:
|
||||
# mirror: http://models.internal:8081
|
||||
# firewall_networks: ["10.42.0.0/16"] # the pods' network, so nodes' engines answer the router
|
||||
|
||||
# On an offline cluster, a ConfigMap or Secret holding the engine checksums
|
||||
# 'llmgw bundle trust' writes (offline.sha256), mounted where offline.checksums
|
||||
# points. Empty mounts none.
|
||||
offlineChecksums:
|
||||
configMap: ""
|
||||
key: offline.sha256
|
||||
|
||||
# The OpenAI-compatible API and the metrics, inside the cluster.
|
||||
service:
|
||||
type: ClusterIP
|
||||
port: 8080
|
||||
metricsPort: 9091
|
||||
annotations: {}
|
||||
|
||||
# Where agents connect, with mutual TLS. Nodes run outside the cluster, so
|
||||
# this is a load balancer by default; the TLS ends in the pod, never at a
|
||||
# proxy in between.
|
||||
agentService:
|
||||
type: LoadBalancer
|
||||
port: 7443
|
||||
annotations: {}
|
||||
loadBalancerIP: ""
|
||||
loadBalancerSourceRanges: []
|
||||
# Keeps the agents' addresses as they are, which the node's identity does
|
||||
# not need but the logs and the audit read better with.
|
||||
externalTrafficPolicy: Local
|
||||
|
||||
ingress:
|
||||
enabled: false
|
||||
className: ""
|
||||
annotations: {}
|
||||
hosts:
|
||||
- host: llm.example.com
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
tls: []
|
||||
# - secretName: llm-example-com
|
||||
# hosts: [llm.example.com]
|
||||
|
||||
# A ServiceMonitor for the Prometheus Operator, scraping the metrics port.
|
||||
serviceMonitor:
|
||||
enabled: false
|
||||
interval: 30s
|
||||
labels: {}
|
||||
|
||||
resources:
|
||||
requests:
|
||||
cpu: 500m
|
||||
memory: 512Mi
|
||||
limits:
|
||||
memory: 2Gi
|
||||
|
||||
nodeSelector: {}
|
||||
tolerations: []
|
||||
affinity: {}
|
||||
podAnnotations: {}
|
||||
podLabels: {}
|
||||
@@ -25,6 +25,13 @@ An HTTP server serving a directory. That is the whole design.
|
||||
model.safetensors
|
||||
```
|
||||
|
||||
At an offline site the mirror also holds what the nodes would otherwise fetch
|
||||
from the internet — llmgw's releases and the llama.cpp builds — under
|
||||
`_llmgw/releases/` and `_llmgw/engines/`, put there by `llmgw bundle import`
|
||||
(docs/install.md, "Offline sites"). They are not models: `mirror list` and
|
||||
`mirror unused` pass over them, and a node checks each against what the
|
||||
control plane says it must be, not against anything the mirror says.
|
||||
|
||||
The date in the path is the point of the layout, not decoration. A model
|
||||
repository that changes upstream changes what you are running, and the first
|
||||
sign is a prompt that worked last week answering differently this week with
|
||||
|
||||
Reference in new issue
Block a user