Each Patroni member gets a share of its own RAM in its own config.yml:
shared_buffers a quarter (15% beside a control plane or agent),
effective_cache_size, maintenance_work_mem and work_mem scaled to it, all
clamped. It is read again on every server init --patroni, so a machine
given more memory gets the use of it. A single-machine PostgreSQL init
installs is sized the same way, as a shared machine, instead of a fixed
96MB.
The step itself went in with 85025df by accident, in an early form that
restarted PostgreSQL before Patroni had taken the reload, leaving the old
shared_buffers with a restart pending. It now waits for Patroni to report
the restart pending, restarts with restart_pending (replicas first, the
leader last), and restarts a member it finds pending from an earlier run.
Applied to the PoC: pg01 978MB, ctl02 (control plane beside it) 587MB,
both from 128MB; streaming, lag 0, gateways unaffected.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
agents.default and agents.nodes.<id> in the cluster's settings hold
everything agent.yaml can say but the agent's id, control planes, address and
certificate, which stay in the file and are refused there. The control plane
sends each node its settings on register and, a second apart, when they
change; an agent whose settings changed checks they load beside its file,
keeps them and starts again on them, engines serving throughout. An edited
agent.yaml that loads restarts the agent too. History, audit, ctl config, the
console and the portal's settings page cover them as the rest of the
settings.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
/admin/settings edits the document llmgw ctl config edit opens: checked as a
control plane would load it, refused when made from a version since replaced
(the edit kept), its history viewed and rolled back as a new version, every
change audited by size and never by contents, since it can hold webhook
secrets. Administrators only; in English and Turkish.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Llama Guard 3 1B could not tell ordinary coding requests from harmful ones:
on the PoC it scored "continue" 0.51, "continue the code above" 0.89 and
"how do I kill all child processes in bash" 0.92 unsafe, against 0.97 to
1.00 for harmful requests, at Q4_K_M and Q8_0 alike, with its own template.
A site had "continue" refused. Qwen3Guard-Gen 0.6B gave the ordinary ones
0.15 at the most and the harmful ones 1.00, in 0.8 GB on CPUs.
- format qwen3guard: its prompt written out as its template writes it, for
requests and answers, and its Safe / Unsafe / Controversial verdict read;
controversial counts as the policy's judge.controversial says, safe by
default.
- judge.models: judges asked in turn, the first that answers deciding; a
failure is recorded against the judge that failed. guard judge use sets
the order, replacing categories none of the judges names.
- known judges (Qwen3Guard 0.6B/4B/8B, Llama Guard 1B/8B): guard judge
enable serves any of them, --gpu the 4B on the roomiest GPU class as a
standard model; the leader fetches and serves one a policy names that the
catalog lacks.
- server init installs qwen3guard-0.6b with the baseline's categories.
- a judge reads a long text by its start and end, which fit its context,
rather than overflowing it and refusing the request under fail: closed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Issue #3 asked for commands to stop and start the system, and 05aa21c gave
only one machine's services. Now:
- llmgw stop sets serving.stopped in the cluster's settings, live: the
reconciler wants every model at no replicas, freeing every GPU, and the
gateway answers every POST under /v1/ with 503 system_stopped. The control
planes, agents and mirror keep running, and no model's definition changes.
- llmgw start clears it; every model is placed again as it asks.
- llmgw restart has every agent restart over its stream (RestartAgent), then
the other control planes over ssh, standbys before a leader, then this
one; engines keep serving.
- llmgw status is the cluster's summary, saying when it is stopped.
One machine's services moved to llmgw service status|start|stop|restart.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
There was no command to stop or start llmgw on a machine (issue #3). These
act on the llmgw units the machine has - mirror, server, agent - or those
named, in dependency order, saying what each did. stop leaves a node's
engines serving unless --engines.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
firewall.enabled (default true) is a cluster setting, applied live: the
leader sends a Firewall message to every agent when it registers and when
the setting or firewall_networks changes, and the agent (root) takes llmgw's
nft table out or installs it. A control plane writes the state to
/var/lib/llmgw/firewall.json, which llmgw-firewall-sync.path applies as root
with `llmgw firewall sync`; server init and server add install the units,
`llmgw firewall sync --install` on one set up before. node add and server
add put no firewall on a machine while the setting is off.
`llmgw wipe` removes llmgw from a machine after listing what goes and asking
for its name: units, engines' processes, the nft table, /etc/llmgw,
/var/lib/llmgw, the agent's state, the sudo grant, the CA from the trust
store, keepalived's configuration when llmgw wrote it, and the users.
--models, --store (Patroni member, etcd, or the local database) and --all
take more. The console has it under Do.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Everything in server.yaml that is not one machine's own is now a versioned
document in the store (cluster_settings), the same on every control plane.
The first start against a store imports the file's cluster keys, with the
baseline guard policy when there is none; after that those keys in a file
are ignored, with a warning.
Each control plane polls the document's version and its own server.yaml's
hash. A change is validated and applied: guard live (Gate.Replace), anything
else by a graceful restart that systemd brings back, standbys first and the
leader 30s later. A version that does not load on a machine is not applied;
at start the newest one that loads is used, so a bad edit cannot keep every
control plane down.
llmgw ctl config show/get/set/unset/edit/history/rollback, the admin API
under /api/v1/settings, and the console's Settings screen, which layers the
store's document over the machine's file, write it. Commands beside the
control plane that read cluster keys (init, node add, server add, backup,
upgrade) read the layered configuration; init --bundle writes offline to
the store once it holds the settings.
A standby that becomes the leader re-reads models, engine choices and
facility holds from the store.
Retires appendGuardIfMissing: the import does it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The control plane's firewall answered only the networks it was installed
with, so a node on another subnet was set up, could not reach 7443 and
showed as lost. node add now admits the node's network (or its address,
when routed) into the local ruleset, keeping everything already there.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Run on the standby, it upgraded the leader first, then itself, and the
leadership moved twice.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A proxy left pointing at a standby showed an empty cluster: the portal and
the admin pages read the standby's own memory. A standby now passes its
client port to the leader's, all but /health, /ready and /leader, which
must describe the machine a load balancer is choosing; and its admin port
passes the dashboard as well as the API. The leader trusts the control
planes' X-Forwarded-For, so clients keep their own address.
The changelog's candidate section is "Unreleased" again: release notes
for an rc read that section, and v0.11.3-rc.1 got none.
The e2e wait for two endpoints is 30s: under the race detector on a busy
runner, 15s was missed once.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
One node at a time was hours on a fleet of hundreds. Nodes now go one, then
two, four, up to --parallel at once, each wave waited on whole; a wave that
does not come back stops it. The other control plane is upgraded first over
ssh, so a leader restarted last hands over to a standby on the release.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The console and ctl on a standby showed its own empty view: nodes and
replicas live in the leader's memory. A standby now passes /api/ to the
leader over the agent port, with the cluster's certificates both ways; the
leader answers only a certificate naming a control plane the store lists,
and only while it leads. Control plane certificates are issued for client
use as well, and server add reissues older ones.
An agent's reconnect backoff starts again after a session that worked, so
each failover it lives through no longer makes the next one slower.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A control plane being stopped held every agent's stream for the thirty
seconds of a graceful shutdown, so the standby led within four seconds and
had no nodes for thirty. The agent server is now closed, not drained. A
control plane that loses its lease while it lives on lets its agents go,
rather than keeping nodes the new leader cannot reach.
ctl status and the console's Overview say which control plane leads, which
stand by, and whether the floating address is on the leader. The console
adds a control plane (Do → Add a control plane), asking for the machine and
the floating address. server add says when the mirror would refuse the new
control plane.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Conflicts with facility-aware placement (config, control, admin API routes,
store test tables) all keep both; the generated reference and OpenAPI
document regenerated.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Conflicts with the LoRA work: vLLM's arguments, the engine profile, router
metrics and the scheduler's request all keep both; agent.pb.go regenerated
from the merged proto; the two test fakes of vLLM given separate names.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ctl federation add/list/remove/sync keeps peers in the store (migration
0029); their /v1/models are offered here as external models named
PEER/MODEL and kept in step, a model still named by a fallback kept
withdrawn. Peers are checked at /ready, timed, and taken out of routing
while down. Requests to a peer carry x-llmgw-hop and are refused (508) when
they come back to a cluster they passed through; residency and no_store
keep requests off peers; X-Request-Id and, with --send-tags, attribution
tags go along. ctl model fallbacks sets overflow to a peer. Console view,
metrics, admin API routes and docs.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Plan §7.5 tier 2, opt-in per profile with kv_events. The agent starts vLLM
publishing KV cache events to a loopback port it listens on (pure-Go zmq4,
msgpack in both of vLLM's encodings), with sha256_cbor hashing and a fixed
PYTHONHASHSEED; it mirrors each engine's blocks and sends net changes up the
existing stream, with a full copy on every new one and a reset on gaps,
restarts and clears. The control plane hands them to the router, for the
sending node's own replicas only. The router tokenizes through a replica's
/tokenize, hashes the tokens as vLLM does (checked against cbor2/hashlib
golden values), and scores replicas by the share of leading blocks held,
falling back to the approximate tree. New metrics beside
llmgw_prefix_decisions_total. Untested against a running vLLM.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Nodes say where they are with llmgw.io/site, room, row, rack and pdu labels,
from agent.yaml or `llmgw ctl label` (kept in the store, over the agent's).
Placement spreads a model's replicas across racks and PDUs, keeps a
multi-node replica within one domain when its profile says keep_within,
places nothing in a domain held with `llmgw ctl facility hold` (optionally
draining it), and refuses a shard that would take a domain past its power
cap from facility.budgets, counting declared, derived and measured draw.
Explain-placement names the domain, and keeps the rejections of a placed
replica; `ctl nodes` gains LOCATION and --domain; `ctl facility` lists the
domains; the console's Nodes view shows and sets location and holds.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A vLLM model can carry PEFT adapters, registered at start, or with
max_loaded_adapters held at most N per replica: the router loads an adapter
through vLLM's runtime LoRA API when a request first wants it, sends the
adapter's requests to a replica that holds it while that one is not
saturated, loads each cold adapter once, and unloads the least recently used
idle one to make room. llama.cpp keeps loading every adapter at start.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The router read SGLang replicas with vLLM's names, and SGLang was started
without --enable-metrics, so it served no /metrics: every scrape failed as
if unreachable and the replica was routed to blind. It now gets
--enable-metrics and a mapping of its own (sglang:num_running_reqs,
num_queue_reqs, token_usage, the token counters and num_requests_total, as
0.5.21 publishes them). engine-check asks python -m sglang.launch_server
--help instead of Python's own help, and SGLang is given --tp-size rather
than the --tp argparse only takes as an abbreviation.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Conflicts: server.go imports (both kept); agent.pb.go regenerated with buf
from the merged proto.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A bundle carries what a cluster would download — llmgw's release with its
signed checksums, every llama.cpp build per architecture with GitHub's
checksums, model files pinned and checked against Hugging Face's — into a
directory or a tar whose manifest sha256 is carried apart. Import checks
everything, the release against the signing key, before anything is put in
the mirror. With offline.mirror set, nodes install llama.cpp from the
mirror against checksums the control plane holds and sends over the agent
stream; releases default to the mirror for node add, ctl upgrade and agent
upgrades. server init and the setup form install from a bundle.
The Helm chart runs one control plane with SQLite on a claim or an external
PostgreSQL, Secrets for the agent certificate and admin token, Services,
Ingress and ServiceMonitor; deploy/container/Dockerfile builds its image.
GPU nodes stay outside Kubernetes, as docs/kubernetes.md explains.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every template and handler message in internal/web goes through a JSON
catalogue under internal/web/locales/<code>/, with English as the fallback.
The language is the person's choice (a cookie set by /_lang/{code}), else
Accept-Language, else landing.language. Tests refuse a key some language
lacks, a translation with other arguments than English, and template text
left in English.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Conflicts: docs.go (manual, OpenAPI and suites all kept), modelserve Job
(trace and eval both kept). The eval routes are added to the OpenAPI route
table, update and promote documented with their optional body, and the
document regenerated.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Conflicts: config imports and docgen sources (both kept); agent.pb.go
regenerated with buf from the merged proto (MIG devices and trace context).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A suite is a set of cases in YAML or JSON lines, each scored by exact,
contains, regex, a number within a tolerance, a JSON Schema subset, or a
judge model with a rubric; weighed into one score against a threshold.
internal/bench/eval runs one on the control plane through Router.Replay, as
the gateway's own requests, and keeps every run and its cases in the store
(migration 0028: eval_runs, eval_cases).
- llmgw ctl eval MODEL --suite FILE|NAME, --history, --run, --suites; exits
non-zero below the score. Suites by name from control.eval_dir, or the
two built in from deploy/eval (arithmetic, json-format).
- model update and promote take --eval SUITE --min-score N, or a model's
standing gate (ctl model gate, eval: in the catalog): a promote asks the
canary first and leaves it below the score; an update tries the new
weights as a canary given no share, then promotes or removes it.
- The console's model menu: ask a suite with its progress, its runs, its
gate.
- The audit middleware no longer hands a handler a body cut off at 64 KiB.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A reconcile pass that does something is a trace, with placement (where it
went, or every candidate turned down), scale-up/down, drain, stop, eviction,
preemption, reaping and restarts beneath it. A replica's start is a span open
from placement until it serves; the start command carries its context in a
new ControlMessage.trace field, and the agent continues the trace through
the weights fetch, engine launch, rendezvous and readiness wait. Stops,
drains, adoption and engine installs are traced on the node, model jobs are
a trace with a span per step linked to the replica's start, and agent.yaml
takes the same tracing block as server.yaml. Off, every span is the no-op one
and commands carry nothing.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Engine trtllm runs trtllm-serve on one node, through a launcher of
llmgw's that demands the replica's key and turns on the Prometheus
endpoint its queue and cache gauges are read from; the router reads it
at /prometheus/metrics. Brought by the operator (engines.trtllm.python),
not installed; checked by ctl engine-check trtllm. Written against the
documented CLI, server and metrics and tested with stand-ins: not yet
run on a datacenter GPU.
mirror import-ollama copies a model's GGUF, and a vision model's
projector, from Ollama's directory into the mirror, checked against the
digest Ollama named each blob by; mirror import does the same for any
local file. The console's Mirror view imports from Ollama with I.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Discovery reports a card in MIG mode as its instances instead of the whole
card, on both paths: nvidia-smi (-L names them, -q -x gives their memory,
ids and processes) and NVML (GetMigMode, GetMaxMigDeviceCount,
GetMigDeviceHandleByIndex, behind an interface so it is tested with a fake).
Each instance has its MIG UUID, its own memory, and a class from the card's
and the profile: a100-80g-mig-3g.40gb. Placement numbers instances
1000 + 100 x card + instance and every view shows them as card/migN.
A replica on MIG runs on exactly one instance: a process opens one MIG
device and instances have no NVLink between them, so tp, pp, gpus or nodes
above one on a MIG class are refused in catalog validation and again in
placement, and the planner offers one instance or none. The agent gives
the engine CUDA_VISIBLE_DEVICES=MIG-<uuid> (no Vulkan device variable on
MIG). Memory and processes are per instance; heat, power and wear the
card's; utilisation is left out rather than reported as zero. A card in MIG
mode with nothing carved stays out of placement with the reason, and
llmgw doctor reports each card's mode and instances and warns of the states
in which nothing can run. ctl nodes, ctl status, the console and the plan
count devices by class. docs/install.md has a MIG section.
Written against fakes of nvidia-smi and NVML output; not yet run on MIG
hardware.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
llmgw keys cache --project P --ttl 1h [--any] | --off, and Cache answers… on
the console's project menu. The same request — project, endpoint, canonical
model and the body without stream, stream_options, user, metadata, store and
service_tier, keys sorted — is answered from the gateway's memory for the
TTL. Only reproducible answers (temperature 0 or a seed; embeddings, rerank,
score) unless --any; only a whole 200 from the model asked for, not a
fallback's, a canary's, or one a guardrail changed.
Looked up after the limits and the input guardrails; the output policy
judges a reused answer; no_store and stored conversations bypass it.
Streamed answers are kept put back together and streamed back as chunks.
x-llmgw-cache: hit|miss|bypass, Age on a hit, Cache-Control: no-cache or
x-llmgw-cache: no to bypass. A hit is recorded with no tokens and no cost,
marked cache_hit (migration 0027) and shown by ctl request. An LRU bounded
by router.cache.max_bytes and max_entry_bytes; llmgw_cache_requests_total,
llmgw_cache_bytes, llmgw_cache_entries.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
adminapi.Routes lists every endpoint with its query parameters and Go values
of the bodies it reads and answers with; internal/docgen/openapi turns that
into docs/admin-api.openapi.json by reflection, the way encoding/json encodes
the types, with the fields' comments as descriptions. A test fails when an
endpoint is registered and not in the table, another when the committed
document is stale. The binary embeds it and serves it with the token.
Handlers that wrote map literals and decoded anonymous structs now use named
types, so the document can see them. GET /api/v1/models/{name} answers with a
model's whole definition as it is in force, which a tool managing it needs.
docs/admin-api.md: reaching the API, the token, what a request must carry.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
