A bundle carries what a cluster would download — llmgw's release with its
signed checksums, every llama.cpp build per architecture with GitHub's
checksums, model files pinned and checked against Hugging Face's — into a
directory or a tar whose manifest sha256 is carried apart. Import checks
everything, the release against the signing key, before anything is put in
the mirror. With offline.mirror set, nodes install llama.cpp from the
mirror against checksums the control plane holds and sends over the agent
stream; releases default to the mirror for node add, ctl upgrade and agent
upgrades. server init and the setup form install from a bundle.
The Helm chart runs one control plane with SQLite on a claim or an external
PostgreSQL, Secrets for the agent certificate and admin token, Services,
Ingress and ServiceMonitor; deploy/container/Dockerfile builds its image.
GPU nodes stay outside Kubernetes, as docs/kubernetes.md explains.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The README's status stopped at v0.9 and listed gaps since closed; onboarding
still said there was no SSO and no roles.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Monitor draws CPU, memory, GPUs, requests, tokens, latency, network and disk
as Braille charts over 5m/1h/24h/7d, then per node, card and model
sparklines; node and model pages have their own. History is read from
monitoring.prometheus through POST /api/v1/metrics/range, or without it
from what the console saw. The control plane serves the cluster's machines
for Prometheus http_sd at /prometheus/targets on its metrics port, and node
add installs prometheus-node-exporter.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The console starts a serve or update job over the admin API and stays,
instead of closing to follow the download. Jobs lists them with their
progress and steps; the sidebar counts those running. ctl model serve and
update take --detach; ctl jobs lists jobs and follows one.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Ranking what was free first made every file come out on the CPU on a cluster
whose one card was in use. The plan now ranks by hardware alone, free now
only breaking a tie; a model served onto cards that are busy is added and
waits for them, the job ending as waiting rather than failed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The control plane reads the model's shape from one GGUF header or its
config.json and lists, for every file, the best way it runs here: one card,
several of a node's (by layer for llama.cpp, tensor-parallel for vLLM),
several nodes' over one fabric (vLLM, pipeline-parallel between them), or a
CPU node; what each card holds and whether it is free now. ctl model plan
prints it; model serve takes its file and way unless told, gives a GGUF on
GPUs a GPU profile, and with --fallbacks adds the next best ways. The
console shows it per file and asks for the way and the fallback.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
model serve takes --projector, fetched beside the weights and loaded by
llama.cpp with --mmproj, and --format safetensors with --gpu-class and --tp,
which fetches the weights with their config and tokenizer and runs them on
vLLM. The console offers both, F switching between GGUF and original
weights. The mirror view is one row per version with its files on Enter.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every document and a page for every command, built from their help, read
like info: topics, contents, find. ? opens the console's own page, which is
new and covers every screen.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>