A proxy left pointing at a standby showed an empty cluster: the portal and
the admin pages read the standby's own memory. A standby now passes its
client port to the leader's, all but /health, /ready and /leader, which
must describe the machine a load balancer is choosing; and its admin port
passes the dashboard as well as the API. The leader trusts the control
planes' X-Forwarded-For, so clients keep their own address.
The changelog's candidate section is "Unreleased" again: release notes
for an rc read that section, and v0.11.3-rc.1 got none.
The e2e wait for two endpoints is 30s: under the race detector on a busy
runner, 15s was missed once.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The console never mentioned them. A Guardrails view shows each policy with
its mode, fail behaviour and checks, whether its judge models have replicas
ready, where policies are attached (attach and detach from it), and what
they did in a window. The admin API gains GET /api/v1/guard/events, and GET
/api/v1/guard says what each policy does.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The change to read it each time left the constructor storing the address
itself, so the file read each time was one named 10.0.0.161.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
One node at a time was hours on a fleet of hundreds. Nodes now go one, then
two, four, up to --parallel at once, each wave waited on whole; a wave that
does not come back stops it. The other control plane is upgraded first over
ssh, so a leader restarted last hands over to a standby on the release.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
With the agent server closed rather than drained, the process could exit
before Run's release had reached the store, and the standby waited out the
lease. The lease is now given up first, synchronously, and nothing takes it
again after.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
server add records it after restarting the new control plane, which then
never named it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The console and ctl on a standby showed its own empty view: nodes and
replicas live in the leader's memory. A standby now passes /api/ to the
leader over the agent port, with the cluster's certificates both ways; the
leader answers only a certificate naming a control plane the store lists,
and only while it leads. Control plane certificates are issued for client
use as well, and server add reissues older ones.
An agent's reconnect backoff starts again after a session that worked, so
each failover it lives through no longer makes the next one slower.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
keepalived's file holds its password and is root's alone; the control plane
runs as llmgw and could not read it, so the CONTROL line never named the
floating address.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A control plane being stopped held every agent's stream for the thirty
seconds of a graceful shutdown, so the standby led within four seconds and
had no nodes for thirty. The agent server is now closed, not drained. A
control plane that loses its lease while it lives on lets its agents go,
rather than keeping nodes the new leader cannot reach.
ctl status and the console's Overview say which control plane leads, which
stand by, and whether the floating address is on the leader. The console
adds a control plane (Do → Add a control plane), asking for the machine and
the floating address. server add says when the mirror would refuse the new
control plane.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Conflicts with facility-aware placement (config, control, admin API routes,
store test tables) all keep both; the generated reference and OpenAPI
document regenerated.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Conflicts with the LoRA work: vLLM's arguments, the engine profile, router
metrics and the scheduler's request all keep both; agent.pb.go regenerated
from the merged proto; the two test fakes of vLLM given separate names.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ctl federation add/list/remove/sync keeps peers in the store (migration
0029); their /v1/models are offered here as external models named
PEER/MODEL and kept in step, a model still named by a fallback kept
withdrawn. Peers are checked at /ready, timed, and taken out of routing
while down. Requests to a peer carry x-llmgw-hop and are refused (508) when
they come back to a cluster they passed through; residency and no_store
keep requests off peers; X-Request-Id and, with --send-tags, attribution
tags go along. ctl model fallbacks sets overflow to a peer. Console view,
metrics, admin API routes and docs.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Plan §7.5 tier 2, opt-in per profile with kv_events. The agent starts vLLM
publishing KV cache events to a loopback port it listens on (pure-Go zmq4,
msgpack in both of vLLM's encodings), with sha256_cbor hashing and a fixed
PYTHONHASHSEED; it mirrors each engine's blocks and sends net changes up the
existing stream, with a full copy on every new one and a reset on gaps,
restarts and clears. The control plane hands them to the router, for the
sending node's own replicas only. The router tokenizes through a replica's
/tokenize, hashes the tokens as vLLM does (checked against cbor2/hashlib
golden values), and scores replicas by the share of leading blocks held,
falling back to the approximate tree. New metrics beside
llmgw_prefix_decisions_total. Untested against a running vLLM.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Nodes say where they are with llmgw.io/site, room, row, rack and pdu labels,
from agent.yaml or `llmgw ctl label` (kept in the store, over the agent's).
Placement spreads a model's replicas across racks and PDUs, keeps a
multi-node replica within one domain when its profile says keep_within,
places nothing in a domain held with `llmgw ctl facility hold` (optionally
draining it), and refuses a shard that would take a domain past its power
cap from facility.budgets, counting declared, derived and measured draw.
Explain-placement names the domain, and keeps the rejections of a placed
replica; `ctl nodes` gains LOCATION and --domain; `ctl facility` lists the
domains; the console's Nodes view shows and sets location and holds.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A vLLM model can carry PEFT adapters, registered at start, or with
max_loaded_adapters held at most N per replica: the router loads an adapter
through vLLM's runtime LoRA API when a request first wants it, sends the
adapter's requests to a replica that holds it while that one is not
saturated, loads each cold adapter once, and unloads the least recently used
idle one to make room. llama.cpp keeps loading every adapter at start.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The sampler sent the key given for --target, which for the gateway is a
client's, so llama.cpp and TensorRT-LLM refused every read and the run
reported an idle engine. The engine key now comes from the admin API for the
replica at --metrics' address, or --metrics-key; unreadable metrics stop the
run before any load, and a step that read none shows why.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The router read SGLang replicas with vLLM's names, and SGLang was started
without --enable-metrics, so it served no /metrics: every scrape failed as
if unreachable and the replica was routed to blind. It now gets
--enable-metrics and a mapping of its own (sglang:num_running_reqs,
num_queue_reqs, token_usage, the token counters and num_requests_total, as
0.5.21 publishes them). engine-check asks python -m sglang.launch_server
--help instead of Python's own help, and SGLang is given --tp-size rather
than the --tp argparse only takes as an abbreviation.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Conflicts: server.go imports (both kept); agent.pb.go regenerated with buf
from the merged proto.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A bundle carries what a cluster would download — llmgw's release with its
signed checksums, every llama.cpp build per architecture with GitHub's
checksums, model files pinned and checked against Hugging Face's — into a
directory or a tar whose manifest sha256 is carried apart. Import checks
everything, the release against the signing key, before anything is put in
the mirror. With offline.mirror set, nodes install llama.cpp from the
mirror against checksums the control plane holds and sends over the agent
stream; releases default to the mirror for node add, ctl upgrade and agent
upgrades. server init and the setup form install from a bundle.
The Helm chart runs one control plane with SQLite on a claim or an external
PostgreSQL, Secrets for the agent certificate and admin token, Services,
Ingress and ServiceMonitor; deploy/container/Dockerfile builds its image.
GPU nodes stay outside Kubernetes, as docs/kubernetes.md explains.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every template and handler message in internal/web goes through a JSON
catalogue under internal/web/locales/<code>/, with English as the fallback.
The language is the person's choice (a cookie set by /_lang/{code}), else
Accept-Language, else landing.language. Tests refuse a key some language
lacks, a translation with other arguments than English, and template text
left in English.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Conflicts: docs.go (manual, OpenAPI and suites all kept), modelserve Job
(trace and eval both kept). The eval routes are added to the OpenAPI route
table, update and promote documented with their optional body, and the
document regenerated.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Conflicts: config imports and docgen sources (both kept); agent.pb.go
regenerated with buf from the merged proto (MIG devices and trace context).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A suite is a set of cases in YAML or JSON lines, each scored by exact,
contains, regex, a number within a tolerance, a JSON Schema subset, or a
judge model with a rubric; weighed into one score against a threshold.
internal/bench/eval runs one on the control plane through Router.Replay, as
the gateway's own requests, and keeps every run and its cases in the store
(migration 0028: eval_runs, eval_cases).
- llmgw ctl eval MODEL --suite FILE|NAME, --history, --run, --suites; exits
non-zero below the score. Suites by name from control.eval_dir, or the
two built in from deploy/eval (arithmetic, json-format).
- model update and promote take --eval SUITE --min-score N, or a model's
standing gate (ctl model gate, eval: in the catalog): a promote asks the
canary first and leaves it below the score; an update tries the new
weights as a canary given no share, then promotes or removes it.
- The console's model menu: ask a suite with its progress, its runs, its
gate.
- The audit middleware no longer hands a handler a body cut off at 64 KiB.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A reconcile pass that does something is a trace, with placement (where it
went, or every candidate turned down), scale-up/down, drain, stop, eviction,
preemption, reaping and restarts beneath it. A replica's start is a span open
from placement until it serves; the start command carries its context in a
new ControlMessage.trace field, and the agent continues the trace through
the weights fetch, engine launch, rendezvous and readiness wait. Stops,
drains, adoption and engine installs are traced on the node, model jobs are
a trace with a span per step linked to the replica's start, and agent.yaml
takes the same tracing block as server.yaml. Off, every span is the no-op one
and commands carry nothing.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Engine trtllm runs trtllm-serve on one node, through a launcher of
llmgw's that demands the replica's key and turns on the Prometheus
endpoint its queue and cache gauges are read from; the router reads it
at /prometheus/metrics. Brought by the operator (engines.trtllm.python),
not installed; checked by ctl engine-check trtllm. Written against the
documented CLI, server and metrics and tested with stand-ins: not yet
run on a datacenter GPU.
mirror import-ollama copies a model's GGUF, and a vision model's
projector, from Ollama's directory into the mirror, checked against the
digest Ollama named each blob by; mirror import does the same for any
local file. The console's Mirror view imports from Ollama with I.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Discovery reports a card in MIG mode as its instances instead of the whole
card, on both paths: nvidia-smi (-L names them, -q -x gives their memory,
ids and processes) and NVML (GetMigMode, GetMaxMigDeviceCount,
GetMigDeviceHandleByIndex, behind an interface so it is tested with a fake).
Each instance has its MIG UUID, its own memory, and a class from the card's
and the profile: a100-80g-mig-3g.40gb. Placement numbers instances
1000 + 100 x card + instance and every view shows them as card/migN.
A replica on MIG runs on exactly one instance: a process opens one MIG
device and instances have no NVLink between them, so tp, pp, gpus or nodes
above one on a MIG class are refused in catalog validation and again in
placement, and the planner offers one instance or none. The agent gives
the engine CUDA_VISIBLE_DEVICES=MIG-<uuid> (no Vulkan device variable on
MIG). Memory and processes are per instance; heat, power and wear the
card's; utilisation is left out rather than reported as zero. A card in MIG
mode with nothing carved stays out of placement with the reason, and
llmgw doctor reports each card's mode and instances and warns of the states
in which nothing can run. ctl nodes, ctl status, the console and the plan
count devices by class. docs/install.md has a MIG section.
Written against fakes of nvidia-smi and NVML output; not yet run on MIG
hardware.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
llmgw keys cache --project P --ttl 1h [--any] | --off, and Cache answers… on
the console's project menu. The same request — project, endpoint, canonical
model and the body without stream, stream_options, user, metadata, store and
service_tier, keys sorted — is answered from the gateway's memory for the
TTL. Only reproducible answers (temperature 0 or a seed; embeddings, rerank,
score) unless --any; only a whole 200 from the model asked for, not a
fallback's, a canary's, or one a guardrail changed.
Looked up after the limits and the input guardrails; the output policy
judges a reused answer; no_store and stored conversations bypass it.
Streamed answers are kept put back together and streamed back as chunks.
x-llmgw-cache: hit|miss|bypass, Age on a hit, Cache-Control: no-cache or
x-llmgw-cache: no to bypass. A hit is recorded with no tokens and no cost,
marked cache_hit (migration 0027) and shown by ctl request. An LRU bounded
by router.cache.max_bytes and max_entry_bytes; llmgw_cache_requests_total,
llmgw_cache_bytes, llmgw_cache_entries.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The manual the binary carries, rendered on the client port: a sidebar of
the documents in the console's reading order (now one list, llmgw.Manual,
for both), heading anchors, links between documents kept in the portal and
links to files outside the binary sent to the repository, no script.
Public by default; landing.docs_sign_in sends visitors to sign in first.
docs/clients.md shows the official OpenAI and Anthropic SDKs against
llmgw in Python, TypeScript, Go and curl, with llmgw's headers; the
portal's index, the landing page and the account page link to it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A starter model with no room failed init after ten minutes and left the
firewall and the judge uninstalled.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
server init (and the setup screens) write guard.default [baseline] —
secrets refused, cards masked, injection recorded, the judge refusing
S1-S4, S7, S9-S12 in prompts and answers, fail open after 5s — fetch the
judge into the mirror and run it on a node's CPU. --no-guard-model leaves
it out. The closing text names what init cannot do: TLS on the client
port and backup recipients.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The README's status stopped at v0.9 and listed gaps since closed; onboarding
still said there was no SSO and no roles.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
