The console never mentioned them. A Guardrails view shows each policy with
its mode, fail behaviour and checks, whether its judge models have replicas
ready, where policies are attached (attach and detach from it), and what
they did in a window. The admin API gains GET /api/v1/guard/events, and GET
/api/v1/guard says what each policy does.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
With the agent server closed rather than drained, the process could exit
before Run's release had reached the store, and the standby waited out the
lease. The lease is now given up first, synchronously, and nothing takes it
again after.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The console and ctl on a standby showed its own empty view: nodes and
replicas live in the leader's memory. A standby now passes /api/ to the
leader over the agent port, with the cluster's certificates both ways; the
leader answers only a certificate naming a control plane the store lists,
and only while it leads. Control plane certificates are issued for client
use as well, and server add reissues older ones.
An agent's reconnect backoff starts again after a session that worked, so
each failover it lives through no longer makes the next one slower.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A control plane being stopped held every agent's stream for the thirty
seconds of a graceful shutdown, so the standby led within four seconds and
had no nodes for thirty. The agent server is now closed, not drained. A
control plane that loses its lease while it lives on lets its agents go,
rather than keeping nodes the new leader cannot reach.
ctl status and the console's Overview say which control plane leads, which
stand by, and whether the floating address is on the leader. The console
adds a control plane (Do → Add a control plane), asking for the machine and
the floating address. server add says when the mirror would refuse the new
control plane.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Conflicts with facility-aware placement (config, control, admin API routes,
store test tables) all keep both; the generated reference and OpenAPI
document regenerated.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Conflicts with the LoRA work: vLLM's arguments, the engine profile, router
metrics and the scheduler's request all keep both; agent.pb.go regenerated
from the merged proto; the two test fakes of vLLM given separate names.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ctl federation add/list/remove/sync keeps peers in the store (migration
0029); their /v1/models are offered here as external models named
PEER/MODEL and kept in step, a model still named by a fallback kept
withdrawn. Peers are checked at /ready, timed, and taken out of routing
while down. Requests to a peer carry x-llmgw-hop and are refused (508) when
they come back to a cluster they passed through; residency and no_store
keep requests off peers; X-Request-Id and, with --send-tags, attribution
tags go along. ctl model fallbacks sets overflow to a peer. Console view,
metrics, admin API routes and docs.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Plan §7.5 tier 2, opt-in per profile with kv_events. The agent starts vLLM
publishing KV cache events to a loopback port it listens on (pure-Go zmq4,
msgpack in both of vLLM's encodings), with sha256_cbor hashing and a fixed
PYTHONHASHSEED; it mirrors each engine's blocks and sends net changes up the
existing stream, with a full copy on every new one and a reset on gaps,
restarts and clears. The control plane hands them to the router, for the
sending node's own replicas only. The router tokenizes through a replica's
/tokenize, hashes the tokens as vLLM does (checked against cbor2/hashlib
golden values), and scores replicas by the share of leading blocks held,
falling back to the approximate tree. New metrics beside
llmgw_prefix_decisions_total. Untested against a running vLLM.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Nodes say where they are with llmgw.io/site, room, row, rack and pdu labels,
from agent.yaml or `llmgw ctl label` (kept in the store, over the agent's).
Placement spreads a model's replicas across racks and PDUs, keeps a
multi-node replica within one domain when its profile says keep_within,
places nothing in a domain held with `llmgw ctl facility hold` (optionally
draining it), and refuses a shard that would take a domain past its power
cap from facility.budgets, counting declared, derived and measured draw.
Explain-placement names the domain, and keeps the rejections of a placed
replica; `ctl nodes` gains LOCATION and --domain; `ctl facility` lists the
domains; the console's Nodes view shows and sets location and holds.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A vLLM model can carry PEFT adapters, registered at start, or with
max_loaded_adapters held at most N per replica: the router loads an adapter
through vLLM's runtime LoRA API when a request first wants it, sends the
adapter's requests to a replica that holds it while that one is not
saturated, loads each cold adapter once, and unloads the least recently used
idle one to make room. llama.cpp keeps loading every adapter at start.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Conflicts: server.go imports (both kept); agent.pb.go regenerated with buf
from the merged proto.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A bundle carries what a cluster would download — llmgw's release with its
signed checksums, every llama.cpp build per architecture with GitHub's
checksums, model files pinned and checked against Hugging Face's — into a
directory or a tar whose manifest sha256 is carried apart. Import checks
everything, the release against the signing key, before anything is put in
the mirror. With offline.mirror set, nodes install llama.cpp from the
mirror against checksums the control plane holds and sends over the agent
stream; releases default to the mirror for node add, ctl upgrade and agent
upgrades. server init and the setup form install from a bundle.
The Helm chart runs one control plane with SQLite on a claim or an external
PostgreSQL, Secrets for the agent certificate and admin token, Services,
Ingress and ServiceMonitor; deploy/container/Dockerfile builds its image.
GPU nodes stay outside Kubernetes, as docs/kubernetes.md explains.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Conflicts: docs.go (manual, OpenAPI and suites all kept), modelserve Job
(trace and eval both kept). The eval routes are added to the OpenAPI route
table, update and promote documented with their optional body, and the
document regenerated.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Conflicts: config imports and docgen sources (both kept); agent.pb.go
regenerated with buf from the merged proto (MIG devices and trace context).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A suite is a set of cases in YAML or JSON lines, each scored by exact,
contains, regex, a number within a tolerance, a JSON Schema subset, or a
judge model with a rubric; weighed into one score against a threshold.
internal/bench/eval runs one on the control plane through Router.Replay, as
the gateway's own requests, and keeps every run and its cases in the store
(migration 0028: eval_runs, eval_cases).
- llmgw ctl eval MODEL --suite FILE|NAME, --history, --run, --suites; exits
non-zero below the score. Suites by name from control.eval_dir, or the
two built in from deploy/eval (arithmetic, json-format).
- model update and promote take --eval SUITE --min-score N, or a model's
standing gate (ctl model gate, eval: in the catalog): a promote asks the
canary first and leaves it below the score; an update tries the new
weights as a canary given no share, then promotes or removes it.
- The console's model menu: ask a suite with its progress, its runs, its
gate.
- The audit middleware no longer hands a handler a body cut off at 64 KiB.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A reconcile pass that does something is a trace, with placement (where it
went, or every candidate turned down), scale-up/down, drain, stop, eviction,
preemption, reaping and restarts beneath it. A replica's start is a span open
from placement until it serves; the start command carries its context in a
new ControlMessage.trace field, and the agent continues the trace through
the weights fetch, engine launch, rendezvous and readiness wait. Stops,
drains, adoption and engine installs are traced on the node, model jobs are
a trace with a span per step linked to the replica's start, and agent.yaml
takes the same tracing block as server.yaml. Off, every span is the no-op one
and commands carry nothing.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Engine trtllm runs trtllm-serve on one node, through a launcher of
llmgw's that demands the replica's key and turns on the Prometheus
endpoint its queue and cache gauges are read from; the router reads it
at /prometheus/metrics. Brought by the operator (engines.trtllm.python),
not installed; checked by ctl engine-check trtllm. Written against the
documented CLI, server and metrics and tested with stand-ins: not yet
run on a datacenter GPU.
mirror import-ollama copies a model's GGUF, and a vision model's
projector, from Ollama's directory into the mirror, checked against the
digest Ollama named each blob by; mirror import does the same for any
local file. The console's Mirror view imports from Ollama with I.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Discovery reports a card in MIG mode as its instances instead of the whole
card, on both paths: nvidia-smi (-L names them, -q -x gives their memory,
ids and processes) and NVML (GetMigMode, GetMaxMigDeviceCount,
GetMigDeviceHandleByIndex, behind an interface so it is tested with a fake).
Each instance has its MIG UUID, its own memory, and a class from the card's
and the profile: a100-80g-mig-3g.40gb. Placement numbers instances
1000 + 100 x card + instance and every view shows them as card/migN.
A replica on MIG runs on exactly one instance: a process opens one MIG
device and instances have no NVLink between them, so tp, pp, gpus or nodes
above one on a MIG class are refused in catalog validation and again in
placement, and the planner offers one instance or none. The agent gives
the engine CUDA_VISIBLE_DEVICES=MIG-<uuid> (no Vulkan device variable on
MIG). Memory and processes are per instance; heat, power and wear the
card's; utilisation is left out rather than reported as zero. A card in MIG
mode with nothing carved stays out of placement with the reason, and
llmgw doctor reports each card's mode and instances and warns of the states
in which nothing can run. ctl nodes, ctl status, the console and the plan
count devices by class. docs/install.md has a MIG section.
Written against fakes of nvidia-smi and NVML output; not yet run on MIG
hardware.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
llmgw keys cache --project P --ttl 1h [--any] | --off, and Cache answers… on
the console's project menu. The same request — project, endpoint, canonical
model and the body without stream, stream_options, user, metadata, store and
service_tier, keys sorted — is answered from the gateway's memory for the
TTL. Only reproducible answers (temperature 0 or a seed; embeddings, rerank,
score) unless --any; only a whole 200 from the model asked for, not a
fallback's, a canary's, or one a guardrail changed.
Looked up after the limits and the input guardrails; the output policy
judges a reused answer; no_store and stored conversations bypass it.
Streamed answers are kept put back together and streamed back as chunks.
x-llmgw-cache: hit|miss|bypass, Age on a hit, Cache-Control: no-cache or
x-llmgw-cache: no to bypass. A hit is recorded with no tokens and no cost,
marked cache_hit (migration 0027) and shown by ctl request. An LRU bounded
by router.cache.max_bytes and max_entry_bytes; llmgw_cache_requests_total,
llmgw_cache_bytes, llmgw_cache_entries.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
adminapi.Routes lists every endpoint with its query parameters and Go values
of the bodies it reads and answers with; internal/docgen/openapi turns that
into docs/admin-api.openapi.json by reflection, the way encoding/json encodes
the types, with the fields' comments as descriptions. A test fails when an
endpoint is registered and not in the table, another when the committed
document is stale. The binary embeds it and serves it with the token.
Handlers that wrote map literals and decoded anonymous structs now use named
types, so the document can see them. GET /api/v1/models/{name} answers with a
model's whole definition as it is in force, which a tool managing it needs.
docs/admin-api.md: reaching the API, the token, what a request must carry.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
notify.webhooks sends signed JSON or Slack-compatible messages, queued,
retried, de-duplicated hourly; ctl notify test checks each webhook.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
env is an allowlist of tuning families less path, plugin, library, address
and credential names (NCCL_*_PLUGIN, CUDA_INJECTION64_PATH and the like
got through the old denylist); llama.cpp's LLAMA_ARG_* are checked as
flags. args refuse --trust-remote-code, --log-file, --slot-save-path,
--tool-parser-plugin, --middleware, --allowed-local-media-path and others
that read, write or load, and what the agent sets itself. Checked in the
catalog and again on the node before an engine starts.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Monitor draws CPU, memory, GPUs, requests, tokens, latency, network and disk
as Braille charts over 5m/1h/24h/7d, then per node, card and model
sparklines; node and model pages have their own. History is read from
monitoring.prometheus through POST /api/v1/metrics/range, or without it
from what the console saw. The control plane serves the cluster's machines
for Prometheus http_sd at /prometheus/targets on its metrics port, and node
add installs prometheus-node-exporter.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The console starts a serve or update job over the admin API and stays,
instead of closing to follow the download. Jobs lists them with their
progress and steps; the sidebar counts those running. ctl model serve and
update take --detach; ctl jobs lists jobs and follows one.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Ranking what was free first made every file come out on the CPU on a cluster
whose one card was in use. The plan now ranks by hardware alone, free now
only breaking a tie; a model served onto cards that are busy is added and
waits for them, the job ending as waiting rather than failed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The control plane reads the model's shape from one GGUF header or its
config.json and lists, for every file, the best way it runs here: one card,
several of a node's (by layer for llama.cpp, tensor-parallel for vLLM),
several nodes' over one fabric (vLLM, pipeline-parallel between them), or a
CPU node; what each card holds and whether it is free now. ctl model plan
prints it; model serve takes its file and way unless told, gives a GGUF on
GPUs a GPU profile, and with --fallbacks adds the next best ways. The
console shows it per file and asks for the way and the fallback.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
model serve takes --projector, fetched beside the weights and loaded by
llama.cpp with --mmproj, and --format safetensors with --gpu-class and --tp,
which fetches the weights with their config and tokenizer and runs them on
vLLM. The console offers both, F switching between GGUF and original
weights. The mirror view is one row per version with its files on Enter.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
