Offline sites and Kubernetes: llmgw bundle create, import and trust, offline.mirror, server init --bundle; a Helm chart for the control plane
A bundle carries what a cluster would download — llmgw's release with its signed checksums, every llama.cpp build per architecture with GitHub's checksums, model files pinned and checked against Hugging Face's — into a directory or a tar whose manifest sha256 is carried apart. Import checks everything, the release against the signing key, before anything is put in the mirror. With offline.mirror set, nodes install llama.cpp from the mirror against checksums the control plane holds and sends over the agent stream; releases default to the mirror for node add, ctl upgrade and agent upgrades. server init and the setup form install from a bundle. The Helm chart runs one control plane with SQLite on a claim or an external PostgreSQL, Secrets for the agent certificate and admin token, Services, Ingress and ServiceMonitor; deploy/container/Dockerfile builds its image. GPU nodes stay outside Kubernetes, as docs/kubernetes.md explains. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
e59ff52932
commit
bae16426a2
51 files changed
+4177
-63
No files matched your search
@@ -0,0 +1,17 @@
|
||||
apiVersion: v2
|
||||
name: llmgw
|
||||
description: >-
|
||||
The llmgw control plane — the OpenAI-compatible gateway, the router and the
|
||||
scheduler that places models on GPU and CPU nodes. The nodes themselves run
|
||||
outside Kubernetes; see docs/kubernetes.md.
|
||||
type: application
|
||||
# The chart's own version; appVersion is the llmgw release the image holds.
|
||||
version: 0.1.0
|
||||
appVersion: "v0.11.1"
|
||||
home: https://gitea.mynodes.xyz/ilker/llmgw
|
||||
sources:
|
||||
- https://gitea.mynodes.xyz/ilker/llmgw
|
||||
keywords:
|
||||
- llm
|
||||
- gateway
|
||||
- inference
|
||||
@@ -0,0 +1,27 @@
|
||||
The llmgw control plane is starting as {{ include "llmgw.fullname" . }}.
|
||||
|
||||
The admin API listens on the pod's loopback only. To reach it from here:
|
||||
|
||||
kubectl -n {{ .Release.Namespace }} port-forward statefulset/{{ include "llmgw.fullname" . }} 9090:9090 &
|
||||
export LLMGW_ADMIN_TOKEN=$(kubectl -n {{ .Release.Namespace }} get secret {{ include "llmgw.adminSecret" . }} \
|
||||
-o jsonpath='{.data.{{ if .Values.adminToken.existingSecret }}{{ .Values.adminToken.secretKey }}{{ else }}admin-token{{ end }}}' | base64 -d)
|
||||
llmgw ctl status
|
||||
|
||||
The first administrator:
|
||||
|
||||
kubectl -n {{ .Release.Namespace }} exec -it statefulset/{{ include "llmgw.fullname" . }} -- \
|
||||
llmgw users create you@example.com --password '<a password>'
|
||||
kubectl -n {{ .Release.Namespace }} exec -it statefulset/{{ include "llmgw.fullname" . }} -- \
|
||||
llmgw users role you@example.com admin
|
||||
|
||||
Agents connect to the {{ include "llmgw.fullname" . }}-agents Service on port {{ .Values.agentService.port }}:
|
||||
|
||||
kubectl -n {{ .Release.Namespace }} get service {{ include "llmgw.fullname" . }}-agents
|
||||
|
||||
Its address must be in the certificate in {{ .Values.agentTLS.existingSecret }}. Nodes join from
|
||||
the machine holding llmgw's authority:
|
||||
|
||||
sudo llmgw node add ubuntu@gpu01 --server https://<that address>:{{ .Values.agentService.port }} \
|
||||
--pki ./pki --admin 127.0.0.1:9090
|
||||
|
||||
docs/kubernetes.md has the rest, and why the nodes stay outside Kubernetes.
|
||||
@@ -0,0 +1,74 @@
|
||||
{{/* The release's name for everything the chart makes. */}}
|
||||
{{- define "llmgw.fullname" -}}
|
||||
{{- if contains .Chart.Name .Release.Name -}}
|
||||
{{- .Release.Name | trunc 63 | trimSuffix "-" -}}
|
||||
{{- else -}}
|
||||
{{- printf "%s-%s" .Release.Name .Chart.Name | trunc 63 | trimSuffix "-" -}}
|
||||
{{- end -}}
|
||||
{{- end -}}
|
||||
|
||||
{{- define "llmgw.selectorLabels" -}}
|
||||
app.kubernetes.io/name: {{ .Chart.Name }}
|
||||
app.kubernetes.io/instance: {{ .Release.Name }}
|
||||
{{- end -}}
|
||||
|
||||
{{- define "llmgw.labels" -}}
|
||||
{{ include "llmgw.selectorLabels" . }}
|
||||
app.kubernetes.io/version: {{ .Values.image.tag | default .Chart.AppVersion | quote }}
|
||||
app.kubernetes.io/managed-by: {{ .Release.Service }}
|
||||
helm.sh/chart: {{ printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" }}
|
||||
{{- end -}}
|
||||
|
||||
{{- define "llmgw.image" -}}
|
||||
{{- if not .Values.image.repository -}}
|
||||
{{- fail "image.repository: the image built from deploy/container/Dockerfile, pushed where the cluster can pull it (the project publishes binaries, not an image)" -}}
|
||||
{{- end -}}
|
||||
{{- printf "%s:%s" .Values.image.repository (.Values.image.tag | default .Chart.AppVersion) -}}
|
||||
{{- end -}}
|
||||
|
||||
{{- define "llmgw.adminSecret" -}}
|
||||
{{- .Values.adminToken.existingSecret | default (printf "%s-admin" (include "llmgw.fullname" .)) -}}
|
||||
{{- end -}}
|
||||
|
||||
{{/*
|
||||
server.yaml: what the chart decides — listeners, where the certificates,
|
||||
the token and the store are mounted — laid over .Values.config. The chart's
|
||||
keys win, because each names a port or a path the pod is built around.
|
||||
*/}}
|
||||
{{- define "llmgw.serverConfig" -}}
|
||||
{{- $store := "/var/lib/llmgw/llmgw.db" -}}
|
||||
{{- if eq .Values.store.type "postgres" -}}
|
||||
{{- $store = "@/etc/llmgw/store/dsn" -}}
|
||||
{{- end -}}
|
||||
{{- $base := dict
|
||||
"listen" ":8080"
|
||||
"admin_listen" "127.0.0.1:9090"
|
||||
"metrics_listen" ":9091"
|
||||
"agent_listen" ":7443"
|
||||
"admin_token_file" "/etc/llmgw/admin/admin-token"
|
||||
"agent_tls" (dict "ca" "/etc/llmgw/tls/ca.pem" "cert" "/etc/llmgw/tls/server.pem" "key" "/etc/llmgw/tls/server-key.pem")
|
||||
"auth" (dict "store" $store)
|
||||
-}}
|
||||
{{- if .Values.offlineChecksums.configMap -}}
|
||||
{{- $_ := set $base "offline" (dict "checksums" "/etc/llmgw/offline/offline.sha256") -}}
|
||||
{{- end -}}
|
||||
{{- $cfg := mustMergeOverwrite (deepCopy (.Values.config | default dict)) $base -}}
|
||||
# Made by the llmgw Helm chart from its values. Every key: docs/reference.md
|
||||
{{ toYaml $cfg }}
|
||||
{{- end -}}
|
||||
|
||||
{{/* Refuses values that would make a pod that cannot work. */}}
|
||||
{{- define "llmgw.validate" -}}
|
||||
{{- if not (has .Values.store.type (list "sqlite" "postgres")) -}}
|
||||
{{- fail (printf "store.type %q: sqlite or postgres" .Values.store.type) -}}
|
||||
{{- end -}}
|
||||
{{- if ne (int .Values.replicas) 1 -}}
|
||||
{{- fail "replicas: this chart runs one control plane; values.yaml says why" -}}
|
||||
{{- end -}}
|
||||
{{- if and (eq .Values.store.type "postgres") (not .Values.store.postgres.existingSecret) -}}
|
||||
{{- fail "store.postgres.existingSecret: a Secret holding the PostgreSQL address" -}}
|
||||
{{- end -}}
|
||||
{{- if not .Values.agentTLS.existingSecret -}}
|
||||
{{- fail "agentTLS.existingSecret: a Secret with ca.pem, server.pem and server-key.pem from llmgw's authority (see values.yaml)" -}}
|
||||
{{- end -}}
|
||||
{{- end -}}
|
||||
@@ -0,0 +1,10 @@
|
||||
{{- include "llmgw.validate" . -}}
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: {{ include "llmgw.fullname" . }}
|
||||
labels:
|
||||
{{- include "llmgw.labels" . | nindent 4 }}
|
||||
data:
|
||||
server.yaml: |
|
||||
{{- include "llmgw.serverConfig" . | nindent 4 }}
|
||||
@@ -0,0 +1,37 @@
|
||||
{{- if .Values.ingress.enabled }}
|
||||
# The OpenAI-compatible API, the portal and the chat page on port 8080. Never
|
||||
# the agent port: its mutual TLS cannot pass through an HTTP proxy.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: {{ include "llmgw.fullname" . }}
|
||||
labels:
|
||||
{{- include "llmgw.labels" . | nindent 4 }}
|
||||
{{- with .Values.ingress.annotations }}
|
||||
annotations:
|
||||
{{- toYaml . | nindent 4 }}
|
||||
{{- end }}
|
||||
spec:
|
||||
{{- with .Values.ingress.className }}
|
||||
ingressClassName: {{ . }}
|
||||
{{- end }}
|
||||
{{- with .Values.ingress.tls }}
|
||||
tls:
|
||||
{{- toYaml . | nindent 4 }}
|
||||
{{- end }}
|
||||
rules:
|
||||
{{- range .Values.ingress.hosts }}
|
||||
- host: {{ .host | quote }}
|
||||
http:
|
||||
paths:
|
||||
{{- range .paths }}
|
||||
- path: {{ .path }}
|
||||
pathType: {{ .pathType }}
|
||||
backend:
|
||||
service:
|
||||
name: {{ include "llmgw.fullname" $ }}
|
||||
port:
|
||||
name: api
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
@@ -0,0 +1,19 @@
|
||||
{{- if not .Values.adminToken.existingSecret }}
|
||||
{{- $name := include "llmgw.adminSecret" . }}
|
||||
{{- $token := randAlphaNum 64 }}
|
||||
{{- /* Kept across upgrades: a new token on every upgrade would lock out every console holding the old one. */}}
|
||||
{{- with lookup "v1" "Secret" .Release.Namespace $name }}
|
||||
{{- $token = index .data "admin-token" | b64dec }}
|
||||
{{- end }}
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: {{ $name }}
|
||||
labels:
|
||||
{{- include "llmgw.labels" . | nindent 4 }}
|
||||
annotations:
|
||||
helm.sh/resource-policy: keep
|
||||
type: Opaque
|
||||
data:
|
||||
admin-token: {{ $token | b64enc | quote }}
|
||||
{{- end }}
|
||||
@@ -0,0 +1,21 @@
|
||||
{{- if .Values.serviceMonitor.enabled }}
|
||||
# For the Prometheus Operator: the metrics port, which serves metrics and
|
||||
# nothing else. deploy/monitoring holds the rules and dashboards that read them.
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: {{ include "llmgw.fullname" . }}
|
||||
labels:
|
||||
{{- include "llmgw.labels" . | nindent 4 }}
|
||||
{{- with .Values.serviceMonitor.labels }}
|
||||
{{- toYaml . | nindent 4 }}
|
||||
{{- end }}
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
{{- include "llmgw.selectorLabels" . | nindent 6 }}
|
||||
endpoints:
|
||||
- port: metrics
|
||||
path: /metrics
|
||||
interval: {{ .Values.serviceMonitor.interval }}
|
||||
{{- end }}
|
||||
@@ -0,0 +1,70 @@
|
||||
# The API and the metrics, for the cluster and an Ingress.
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: {{ include "llmgw.fullname" . }}
|
||||
labels:
|
||||
{{- include "llmgw.labels" . | nindent 4 }}
|
||||
{{- with .Values.service.annotations }}
|
||||
annotations:
|
||||
{{- toYaml . | nindent 4 }}
|
||||
{{- end }}
|
||||
spec:
|
||||
type: {{ .Values.service.type }}
|
||||
selector:
|
||||
{{- include "llmgw.selectorLabels" . | nindent 4 }}
|
||||
ports:
|
||||
- name: api
|
||||
port: {{ .Values.service.port }}
|
||||
targetPort: api
|
||||
- name: metrics
|
||||
port: {{ .Values.service.metricsPort }}
|
||||
targetPort: metrics
|
||||
---
|
||||
# Where agents on the nodes connect. Mutual TLS ends in the pod: whatever is
|
||||
# in between must pass TCP through untouched.
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: {{ include "llmgw.fullname" . }}-agents
|
||||
labels:
|
||||
{{- include "llmgw.labels" . | nindent 4 }}
|
||||
{{- with .Values.agentService.annotations }}
|
||||
annotations:
|
||||
{{- toYaml . | nindent 4 }}
|
||||
{{- end }}
|
||||
spec:
|
||||
type: {{ .Values.agentService.type }}
|
||||
{{- if and (eq .Values.agentService.type "LoadBalancer") .Values.agentService.loadBalancerIP }}
|
||||
loadBalancerIP: {{ .Values.agentService.loadBalancerIP }}
|
||||
{{- end }}
|
||||
{{- with .Values.agentService.loadBalancerSourceRanges }}
|
||||
loadBalancerSourceRanges:
|
||||
{{- toYaml . | nindent 4 }}
|
||||
{{- end }}
|
||||
{{- if has .Values.agentService.type (list "LoadBalancer" "NodePort") }}
|
||||
externalTrafficPolicy: {{ .Values.agentService.externalTrafficPolicy }}
|
||||
{{- end }}
|
||||
selector:
|
||||
{{- include "llmgw.selectorLabels" . | nindent 4 }}
|
||||
ports:
|
||||
- name: agents
|
||||
port: {{ .Values.agentService.port }}
|
||||
targetPort: agents
|
||||
---
|
||||
# The StatefulSet's own, giving each pod a stable name.
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: {{ include "llmgw.fullname" . }}-headless
|
||||
labels:
|
||||
{{- include "llmgw.labels" . | nindent 4 }}
|
||||
spec:
|
||||
clusterIP: None
|
||||
publishNotReadyAddresses: true
|
||||
selector:
|
||||
{{- include "llmgw.selectorLabels" . | nindent 4 }}
|
||||
ports:
|
||||
- name: api
|
||||
port: 8080
|
||||
targetPort: api
|
||||
@@ -0,0 +1,181 @@
|
||||
apiVersion: apps/v1
|
||||
kind: StatefulSet
|
||||
metadata:
|
||||
name: {{ include "llmgw.fullname" . }}
|
||||
labels:
|
||||
{{- include "llmgw.labels" . | nindent 4 }}
|
||||
spec:
|
||||
# One control plane: Kubernetes replaces it within seconds when it goes,
|
||||
# and the agents reconnect on their own. See replicas in values.yaml.
|
||||
replicas: 1
|
||||
serviceName: {{ include "llmgw.fullname" . }}-headless
|
||||
selector:
|
||||
matchLabels:
|
||||
{{- include "llmgw.selectorLabels" . | nindent 6 }}
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
{{- include "llmgw.selectorLabels" . | nindent 8 }}
|
||||
{{- with .Values.podLabels }}
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
annotations:
|
||||
# server.yaml is read at start: a changed one restarts the pod.
|
||||
checksum/config: {{ include "llmgw.serverConfig" . | sha256sum }}
|
||||
{{- with .Values.podAnnotations }}
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
spec:
|
||||
{{- with .Values.imagePullSecrets }}
|
||||
imagePullSecrets:
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
# Agents reconnect within seconds of a control plane going; a long
|
||||
# grace period only delays the one that replaces it.
|
||||
terminationGracePeriodSeconds: 30
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 65532
|
||||
runAsGroup: 65532
|
||||
fsGroup: 65532
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
containers:
|
||||
- name: llmgw
|
||||
image: {{ include "llmgw.image" . }}
|
||||
imagePullPolicy: {{ .Values.image.pullPolicy }}
|
||||
args: ["server", "--config", "/etc/llmgw/server.yaml"]
|
||||
env:
|
||||
# For `kubectl exec ... -- llmgw ctl`, which reads the token
|
||||
# from here; the server reads admin_token_file.
|
||||
- name: LLMGW_ADMIN_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: {{ include "llmgw.adminSecret" . }}
|
||||
key: {{ if .Values.adminToken.existingSecret }}{{ .Values.adminToken.secretKey }}{{ else }}admin-token{{ end }}
|
||||
ports:
|
||||
- name: api
|
||||
containerPort: 8080
|
||||
- name: agents
|
||||
containerPort: 7443
|
||||
- name: metrics
|
||||
containerPort: 9091
|
||||
# Up is /health. Ready is /leader: a control plane holds the
|
||||
# agents' connections and places models only once it leads — on
|
||||
# PostgreSQL, once the lease of the one it replaced has run out. A
|
||||
# cluster with no node yet is still ready: /ready would wait for a
|
||||
# replica, and nodes could then never connect.
|
||||
startupProbe:
|
||||
httpGet: {path: /health, port: api}
|
||||
periodSeconds: 5
|
||||
failureThreshold: 60
|
||||
livenessProbe:
|
||||
httpGet: {path: /health, port: api}
|
||||
periodSeconds: 10
|
||||
failureThreshold: 6
|
||||
readinessProbe:
|
||||
httpGet: {path: /leader, port: api}
|
||||
periodSeconds: 5
|
||||
failureThreshold: 2
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop: ["ALL"]
|
||||
resources:
|
||||
{{- toYaml .Values.resources | nindent 12 }}
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /etc/llmgw/server.yaml
|
||||
subPath: server.yaml
|
||||
readOnly: true
|
||||
- name: agent-tls
|
||||
mountPath: /etc/llmgw/tls
|
||||
readOnly: true
|
||||
- name: admin-token
|
||||
mountPath: /etc/llmgw/admin
|
||||
readOnly: true
|
||||
{{- if eq .Values.store.type "postgres" }}
|
||||
- name: store
|
||||
mountPath: /etc/llmgw/store
|
||||
readOnly: true
|
||||
{{- end }}
|
||||
{{- if .Values.offlineChecksums.configMap }}
|
||||
- name: offline
|
||||
mountPath: /etc/llmgw/offline
|
||||
readOnly: true
|
||||
{{- end }}
|
||||
- name: data
|
||||
mountPath: /var/lib/llmgw
|
||||
- name: tmp
|
||||
mountPath: /tmp
|
||||
volumes:
|
||||
- name: config
|
||||
configMap:
|
||||
name: {{ include "llmgw.fullname" . }}
|
||||
- name: agent-tls
|
||||
secret:
|
||||
secretName: {{ .Values.agentTLS.existingSecret }}
|
||||
defaultMode: 0440
|
||||
items:
|
||||
- {key: ca.pem, path: ca.pem}
|
||||
- {key: server.pem, path: server.pem}
|
||||
- {key: server-key.pem, path: server-key.pem}
|
||||
- name: admin-token
|
||||
secret:
|
||||
secretName: {{ include "llmgw.adminSecret" . }}
|
||||
defaultMode: 0440
|
||||
items:
|
||||
- key: {{ if .Values.adminToken.existingSecret }}{{ .Values.adminToken.secretKey }}{{ else }}admin-token{{ end }}
|
||||
path: admin-token
|
||||
{{- if eq .Values.store.type "postgres" }}
|
||||
- name: store
|
||||
secret:
|
||||
secretName: {{ .Values.store.postgres.existingSecret }}
|
||||
defaultMode: 0440
|
||||
items:
|
||||
- key: {{ .Values.store.postgres.secretKey }}
|
||||
path: dsn
|
||||
{{- end }}
|
||||
{{- if .Values.offlineChecksums.configMap }}
|
||||
- name: offline
|
||||
configMap:
|
||||
name: {{ .Values.offlineChecksums.configMap }}
|
||||
items:
|
||||
- key: {{ .Values.offlineChecksums.key }}
|
||||
path: offline.sha256
|
||||
{{- end }}
|
||||
- name: tmp
|
||||
emptyDir: {}
|
||||
{{- if eq .Values.store.type "postgres" }}
|
||||
# The store is PostgreSQL; nothing here outlives the pod.
|
||||
- name: data
|
||||
emptyDir: {}
|
||||
{{- end }}
|
||||
{{- with .Values.nodeSelector }}
|
||||
nodeSelector:
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
{{- with .Values.tolerations }}
|
||||
tolerations:
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
{{- with .Values.affinity }}
|
||||
affinity:
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
{{- if eq .Values.store.type "sqlite" }}
|
||||
volumeClaimTemplates:
|
||||
- metadata:
|
||||
name: data
|
||||
labels:
|
||||
{{- include "llmgw.selectorLabels" . | nindent 10 }}
|
||||
spec:
|
||||
accessModes: ["ReadWriteOnce"]
|
||||
{{- with .Values.store.sqlite.storageClass }}
|
||||
storageClassName: {{ . }}
|
||||
{{- end }}
|
||||
resources:
|
||||
requests:
|
||||
storage: {{ .Values.store.sqlite.size }}
|
||||
{{- end }}
|
||||
@@ -0,0 +1,132 @@
|
||||
# The llmgw control plane in Kubernetes. docs/kubernetes.md says what this
|
||||
# runs, what it does not, and how nodes outside the cluster join it.
|
||||
|
||||
image:
|
||||
# Where you pushed the image built from deploy/container/Dockerfile. The
|
||||
# project publishes binaries, not an image, so this has no default.
|
||||
repository: ""
|
||||
# Empty takes the chart's appVersion.
|
||||
tag: ""
|
||||
pullPolicy: IfNotPresent
|
||||
imagePullSecrets: []
|
||||
|
||||
# One control plane, always. Two against PostgreSQL would elect a leader and
|
||||
# keep a standby, but a standby serves nothing and is never ready, so a
|
||||
# rollout would wait on it for ever. Until the chart handles that, it runs
|
||||
# one: Kubernetes replaces it within seconds when it goes, and the agents
|
||||
# reconnect on their own.
|
||||
replicas: 1
|
||||
|
||||
store:
|
||||
# sqlite: a file on a PersistentVolumeClaim, for one replica.
|
||||
# postgres: a PostgreSQL you run, its address in a Secret.
|
||||
type: sqlite
|
||||
sqlite:
|
||||
size: 10Gi
|
||||
storageClass: ""
|
||||
postgres:
|
||||
# A Secret holding the address, postgres://user:password@host:5432/llmgw?sslmode=verify-full
|
||||
existingSecret: ""
|
||||
secretKey: dsn
|
||||
|
||||
# The agent port's certificate. It comes from llmgw's own authority, not a
|
||||
# public one — agents present certificates from the same authority — so it
|
||||
# is made outside the cluster, where the authority's key stays:
|
||||
#
|
||||
# llmgw ctl cert init-ca --dir ./pki
|
||||
# llmgw ctl cert issue-server llmgw-ctl --host <agent address> --dir ./pki --out ./pki
|
||||
# kubectl create secret generic llmgw-agent-tls \
|
||||
# --from-file=ca.pem=pki/ca.pem \
|
||||
# --from-file=server.pem=pki/server.pem \
|
||||
# --from-file=server-key.pem=pki/server-key.pem
|
||||
#
|
||||
# --host is every address agents dial: agentService's load balancer address
|
||||
# or DNS name.
|
||||
agentTLS:
|
||||
existingSecret: ""
|
||||
|
||||
# The admin API's token (llmgw ctl, the console). Empty makes one, kept in a
|
||||
# Secret across upgrades.
|
||||
adminToken:
|
||||
existingSecret: ""
|
||||
secretKey: admin-token
|
||||
|
||||
# server.yaml beyond what the chart sets (listeners, TLS, store, admin
|
||||
# token): any key of docs/reference.md, laid over the chart's. An offline
|
||||
# cluster sets offline.mirror here, and offlineChecksums below.
|
||||
config:
|
||||
control:
|
||||
enabled: true
|
||||
auth:
|
||||
enabled: true
|
||||
signup_open: false
|
||||
default_rpm: 60
|
||||
default_tpm: 200000
|
||||
default_price:
|
||||
prompt_per_1k: 1000
|
||||
completion_per_1k: 1000
|
||||
max_key_ttl: 2160h
|
||||
# mirror:
|
||||
# api: https://models.internal:8082
|
||||
# offline:
|
||||
# mirror: http://models.internal:8081
|
||||
# firewall_networks: ["10.42.0.0/16"] # the pods' network, so nodes' engines answer the router
|
||||
|
||||
# On an offline cluster, a ConfigMap or Secret holding the engine checksums
|
||||
# 'llmgw bundle trust' writes (offline.sha256), mounted where offline.checksums
|
||||
# points. Empty mounts none.
|
||||
offlineChecksums:
|
||||
configMap: ""
|
||||
key: offline.sha256
|
||||
|
||||
# The OpenAI-compatible API and the metrics, inside the cluster.
|
||||
service:
|
||||
type: ClusterIP
|
||||
port: 8080
|
||||
metricsPort: 9091
|
||||
annotations: {}
|
||||
|
||||
# Where agents connect, with mutual TLS. Nodes run outside the cluster, so
|
||||
# this is a load balancer by default; the TLS ends in the pod, never at a
|
||||
# proxy in between.
|
||||
agentService:
|
||||
type: LoadBalancer
|
||||
port: 7443
|
||||
annotations: {}
|
||||
loadBalancerIP: ""
|
||||
loadBalancerSourceRanges: []
|
||||
# Keeps the agents' addresses as they are, which the node's identity does
|
||||
# not need but the logs and the audit read better with.
|
||||
externalTrafficPolicy: Local
|
||||
|
||||
ingress:
|
||||
enabled: false
|
||||
className: ""
|
||||
annotations: {}
|
||||
hosts:
|
||||
- host: llm.example.com
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
tls: []
|
||||
# - secretName: llm-example-com
|
||||
# hosts: [llm.example.com]
|
||||
|
||||
# A ServiceMonitor for the Prometheus Operator, scraping the metrics port.
|
||||
serviceMonitor:
|
||||
enabled: false
|
||||
interval: 30s
|
||||
labels: {}
|
||||
|
||||
resources:
|
||||
requests:
|
||||
cpu: 500m
|
||||
memory: 512Mi
|
||||
limits:
|
||||
memory: 2Gi
|
||||
|
||||
nodeSelector: {}
|
||||
tolerations: []
|
||||
affinity: {}
|
||||
podAnnotations: {}
|
||||
podLabels: {}
|
||||
Reference in new issue
Block a user