Each Patroni member gets a share of its own RAM in its own config.yml:
shared_buffers a quarter (15% beside a control plane or agent),
effective_cache_size, maintenance_work_mem and work_mem scaled to it, all
clamped. It is read again on every server init --patroni, so a machine
given more memory gets the use of it. A single-machine PostgreSQL init
installs is sized the same way, as a shared machine, instead of a fixed
96MB.
The step itself went in with 85025df by accident, in an early form that
restarted PostgreSQL before Patroni had taken the reload, leaving the old
shared_buffers with a restart pending. It now waits for Patroni to report
the restart pending, restarts with restart_pending (replicas first, the
leader last), and restarts a member it finds pending from an earlier run.
Applied to the PoC: pg01 978MB, ctl02 (control plane beside it) 587MB,
both from 128MB; streaming, lag 0, gateways unaffected.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
An adapter now has an owner (org, team, project, user or key): only callers
there see it, and to anyone else it is a model that does not exist, absent
from /v1/models too. A training job's owner defaults to the one collection
its pairs come from, must be able to read every collection they come from,
and pairs from a knowledge base are never made public. ctl train publish
keeps the GGUF or PEFT files on the mirror, which accepts uploads now, and
sets the owned adapter after the usual architecture check.
The knowledge bases' cache_salt follows the collections searched rather
than the organization, so two teams of one organization no longer share
cached passages.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
pgEnv passed sslmode=verify-full but dropped sslrootcert, so libpq looked
for ~/.postgresql/root.crt and both backup and restore failed against the
Patroni store. sslrootcert, sslcert and sslkey now go along as PGSSL*.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A job names the catalog model the adapter is for, the safetensors weights to
train on, the node and the MiB of a card it needs, and question/answer pairs.
The control plane sends it when the node's agent is there and nothing else
trains on it; the agent trains with its installed vLLM's torch and a pinned
overlay of peft and accelerate (no second environment), as the engines'
user, on the card with that much free — beside an engine, if it leaves that
much — converts the adapter to GGUF with the installed llama.cpp's own
conversion code when the model is served by llama.cpp, and sends what it
made back in pieces, kept in the store to download. Progress, loss and peak
card memory are reported as it goes; a node that goes fails its job.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A chat model of the cluster reads each chunk and writes questions a person
might ask of it with the answers it holds, streamed as NDJSON a chunk at a
time, for people to correct before an adapter is trained on it. A chunk that
gave no pairs says so on its own line.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
agents.default and agents.nodes.<id> in the cluster's settings hold
everything agent.yaml can say but the agent's id, control planes, address and
certificate, which stay in the file and are refused there. The control plane
sends each node its settings on register and, a second apart, when they
change; an agent whose settings changed checks they load beside its file,
keeps them and starts again on them, engines serving throughout. An edited
agent.yaml that loads restarts the agent too. History, audit, ctl config, the
console and the portal's settings page cover them as the rest of the
settings.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
/admin/settings edits the document llmgw ctl config edit opens: checked as a
control plane would load it, refused when made from a version since replaced
(the edit kept), its history viewed and rolled back as a new version, every
change audited by size and never by contents, since it can hold webhook
secrets. Administrators only; in English and Turkish.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On an organization, team, project or key, the narrowest winning: llmgw ctl
kb default, POST /api/v1/kb/defaults, and the console's Knowledge screen. A
request's own X-Llmgw-Knowledge overrides it and none turns it off. A default
never refuses a request: a collection the caller cannot read is left out, and
a failed search answers without passages. Removing a collection takes it out
of every default.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Collections listed with owner, model, documents and grants; A makes one; on a
row its documents, grant, revoke, a search to try, delete. docs/knowledge.md
says what administrators do and how people use llmgw-kb on Windows, macOS
and Linux: what it reads, what leaves their machine, where its ledger is.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Collections made for an organization, team, project or person, their
dimension asked of the embedding model; granted and revoked by path, never
outside their organization; their documents listed and removed; and a search
an operator can try. In the OpenAPI document.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Collections owned by an organization, team, project or person; grants that
add up down the tree and never cross an organization; chunks in PostgreSQL
with pgvector, partitioned by organization, an HNSW index per collection, and
row-level security as a second line behind each query's filter. Search fuses
nearest-neighbour and full-text ranks. Off on SQLite and without pgvector,
saying why. server init and the HA database setup install and enable it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
judge ask showed Qwen3Guard's controversial verdict as "safe (Violent)"; it
now says controversial, which a policy's judge.controversial counts. The
stray-judge note called a judge no policy asks any more "served as a model
clients use"; a model with role: guard is now said to be a judge no policy
asks.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every replica's engine demands its own key, and the check sent none, so it
failed with a 401 on every engine in the cluster. The key is asked of the
admin API for the replica at --endpoint, as bench does, or given with --key.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Llama Guard 3 1B could not tell ordinary coding requests from harmful ones:
on the PoC it scored "continue" 0.51, "continue the code above" 0.89 and
"how do I kill all child processes in bash" 0.92 unsafe, against 0.97 to
1.00 for harmful requests, at Q4_K_M and Q8_0 alike, with its own template.
A site had "continue" refused. Qwen3Guard-Gen 0.6B gave the ordinary ones
0.15 at the most and the harmful ones 1.00, in 0.8 GB on CPUs.
- format qwen3guard: its prompt written out as its template writes it, for
requests and answers, and its Safe / Unsafe / Controversial verdict read;
controversial counts as the policy's judge.controversial says, safe by
default.
- judge.models: judges asked in turn, the first that answers deciding; a
failure is recorded against the judge that failed. guard judge use sets
the order, replacing categories none of the judges names.
- known judges (Qwen3Guard 0.6B/4B/8B, Llama Guard 1B/8B): guard judge
enable serves any of them, --gpu the 4B on the roomiest GPU class as a
standard model; the leader fetches and serves one a policy names that the
catalog lacks.
- server init installs qwen3guard-0.6b with the baseline's categories.
- a judge reads a long text by its start and end, which fit its context,
rather than overflowing it and refusing the request under fail: closed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every model of a stopped system asks for no replica and so has what it asks
for. Until the control plane took the setting up, seconds later, start
--wait saw 18 of 18 satisfied and returned at once, with nothing started.
It now counts only once the control plane says it is in the state asked
for, both ways.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
One standby that ssh could not log in to stopped the whole restart before
this control plane's own. It is now said and left, the rest restarts - the
leader last - and the command fails at the end naming what was not
restarted and how to do it there.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Issue #3 asked for commands to stop and start the system, and 05aa21c gave
only one machine's services. Now:
- llmgw stop sets serving.stopped in the cluster's settings, live: the
reconciler wants every model at no replicas, freeing every GPU, and the
gateway answers every POST under /v1/ with 503 system_stopped. The control
planes, agents and mirror keep running, and no model's definition changes.
- llmgw start clears it; every model is placed again as it asks.
- llmgw restart has every agent restart over its stream (RestartAgent), then
the other control planes over ssh, standbys before a leader, then this
one; engines keep serving.
- llmgw status is the cluster's summary, saying when it is stopped.
One machine's services moved to llmgw service status|start|stop|restart.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
There was no command to stop or start llmgw on a machine (issue #3). These
act on the llmgw units the machine has - mirror, server, agent - or those
named, in dependency order, saying what each did. stop leaves a node's
engines serving unless --engines.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
SetCompute on an engine adopted from an earlier agent called writeState,
which read the command line from sh.cmd - nil for a process this agent did
not start - and the agent crashed at every start on the PoC's nodes until
fixed. An adopted shard now keeps the command line its state file had.
engine-check looked for llama-server one directory too shallow; the agent
installs it under <version-variant>/llama-<build>/. It now looks at both,
preferring a CUDA build on a machine with NVIDIA GPUs. Run on cuda01: a
token computed on CUDA0, 25 of 25 layers offloaded.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Checked against the installed llama.cpp (b10991): at its default verbosity
it prints neither the devices it uses nor the layers it offloads, so the
start-up output said nothing and every replica's ON column stayed empty.
The agent now takes compute from nvidia-smi - the GPUs the shard's processes
hold memory on, and how much - or CPU, for every engine, falling back to it
when llama.cpp's output says nothing; adopted engines get it too, and each
session reports it once per shard as a compute-only status. engine-check
runs its GPU token at -lv 4, where llama.cpp does say where its layers went.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
/forgot mails a link to set a new password when the cluster's mail settings
say how - SMTP relay, sender, and the public base_url the link points at,
never the request's Host - and says the same whether or not the address has
an account. /reset sets it once, closing every session. Links are kept by
their hash in password_resets, good for an hour, a newer one replacing an
older. Without mail, /forgot says to ask an operator, and llmgw users
reset-link makes the link to hand over. Single sign-on accounts get none.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
doctor sums findings up as two readinesses - the node's, which decides
whether it joins, and multi-node, its fabrics for replicas spanning nodes -
prints fabric findings in a section of their own, and gives both in --json.
node add says the multi-node one. NodeView.multi_node says whether a node's
declared fabrics are up, naming adapters that are down, and ctl nodes shows
it beside the fabric.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A model's replicas restarted one at a time, always. RestartPace says how
many restart at once: MaxSurge beside themselves (replacement first),
MaxUnavailable stopped first (no room needed, out while loading); the
reconciler begins that many per model per pass. The admin API takes
max_surge and max_unavailable on POST /api/v1/restart.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
NodeView carries engine_versions and compute_capabilities; ctl nodes shows
llamacpp b10991 and 6000-24g (7.5). A profile's engine_version restricts it
to nodes whose engine version starts with it, and explain-placement says
which version a node has, or which engines it has when it lacks the one
asked for.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Seeing a device is not computing on it. For llama.cpp on a machine with
NVIDIA GPUs, engine-check now loads a small GGUF (--model, or the smallest in
the weights cache) with every layer on the GPU, asks for one token, and
reads from llama.cpp's own output where the layers went: a build that
computes on the CPU fails, naming the fix. With no --bin it checks the build
the agent installed under /opt/llmgw/engines.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
llama.cpp's start-up output is read as a shard becomes ready: the devices it
used, its backend, and the layers it offloaded. That goes up with READY in a
new ShardStatus.compute, is kept on the shard, and is shown by the admin
API, ctl replicas (ON) and the console (COMPUTE). A shard given GPUs whose
engine says it found no usable GPU or offloaded no layer is stopped and
failed, even where nvidia-smi cannot be asked.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
guard.enabled (unset is on) makes every policy check nothing while false,
live: llmgw ctl guard off/on set it in the cluster's settings, and ctl
status, ctl guard list and the guardrails page say when they are off.
A refusal by a judge names the hazard it saw - judge: S6,Specialized
Advice - and the verdict is kept with the request in a new usage column,
guard_rule, shown by ctl request, the request log and the guardrails page.
llmgw ctl guard judge ask has a judge read a text, past the cache, and shows
its raw answer and how it is read; the endpoint is kept out of the audit so
no text is recorded.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
llmgw ctl guard judge status lists each judge model the policies ask:
in the catalog, as a judge, replicas ready, its last failure to answer, and
what is wrong in words. enable serves the missing one through model serve -
Llama Guard 3 1B on CPUs, role guard, under the name the policies give it -
with --replace for a model of that name that is not a judge; the console has
it under Do. GET /api/v1/guard/judges, and the summary's judges, put the
problem on ctl status and the console's Overview.
A Llama Guard file served from the mirror page is added as a judge, under
the name the policies ask, rather than as a model clients use the policies
never find. ctl says plainly when the API answers with a page.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
server init and mirror init took the first non-loopback address for the
mirror's --base, Docker's 172.17.0.1 on a machine with Docker; nodes asked
their own bridge for every model and were refused. primaryAddr takes the
default route's source, else the first address on an interface that is not
a container's, bridge's or overlay's. node add has the node reach the
mirror's files during the checks and fails, naming the address and the fix,
when it cannot.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
firewall.enabled (default true) is a cluster setting, applied live: the
leader sends a Firewall message to every agent when it registers and when
the setting or firewall_networks changes, and the agent (root) takes llmgw's
nft table out or installs it. A control plane writes the state to
/var/lib/llmgw/firewall.json, which llmgw-firewall-sync.path applies as root
with `llmgw firewall sync`; server init and server add install the units,
`llmgw firewall sync --install` on one set up before. node add and server
add put no firewall on a machine while the setting is off.
`llmgw wipe` removes llmgw from a machine after listing what goes and asking
for its name: units, engines' processes, the nft table, /etc/llmgw,
/var/lib/llmgw, the agent's state, the sudo grant, the CA from the trust
store, keepalived's configuration when llmgw wrote it, and the users.
--models, --store (Patroni member, etcd, or the local database) and --all
take more. The console has it under Do.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Everything in server.yaml that is not one machine's own is now a versioned
document in the store (cluster_settings), the same on every control plane.
The first start against a store imports the file's cluster keys, with the
baseline guard policy when there is none; after that those keys in a file
are ignored, with a warning.
Each control plane polls the document's version and its own server.yaml's
hash. A change is validated and applied: guard live (Gate.Replace), anything
else by a graceful restart that systemd brings back, standbys first and the
leader 30s later. A version that does not load on a machine is not applied;
at start the newest one that loads is used, so a bad edit cannot keep every
control plane down.
llmgw ctl config show/get/set/unset/edit/history/rollback, the admin API
under /api/v1/settings, and the console's Settings screen, which layers the
store's document over the machine's file, write it. Commands beside the
control plane that read cluster keys (init, node add, server add, backup,
upgrade) read the layered configuration; init --bundle writes offline to
the store once it holds the settings.
A standby that becomes the leader re-reads models, engine choices and
facility holds from the store.
Retires appendGuardIfMissing: the import does it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The control plane's firewall answered only the networks it was installed
with, so a node on another subnet was set up, could not reach 7443 and
showed as lost. node add now admits the node's network (or its address,
when routed) into the local ruleset, keeping everything already there.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A binary installed by hand made ctl upgrade say this machine already ran
the release and leave the old server running. And a node had 15 seconds
to answer an upgrade the control plane gives five minutes, so a node that
was upgrading stopped the run.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
An upgrade replaces the binary and leaves server.yaml alone, so a control
plane installed before guardrails stayed unguarded. Run again, init now
appends the baseline guard section (and so fetches the judge) when the file
has no guard key at all, keeping the old file as server.yaml.bak-<time>.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A --context-length 32768 --ctx-pool 32768 model got llama.cpp's default four
slots of 8192, and refused what the gateway admitted; --parallel is now
what the pool holds of the context. max_output was 1024 for every model,
which cut an editor's agent off mid-file; it is now a quarter of the
context, from 1024 to 8192. --tools writes capabilities.tools, which the
Continue configuration now reads, as it reads max_output.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Run on the standby, it upgraded the leader first, then itself, and the
leadership moved twice.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A proxy left pointing at a standby showed an empty cluster: the portal and
the admin pages read the standby's own memory. A standby now passes its
client port to the leader's, all but /health, /ready and /leader, which
must describe the machine a load balancer is choosing; and its admin port
passes the dashboard as well as the API. The leader trusts the control
planes' X-Forwarded-For, so clients keep their own address.
The changelog's candidate section is "Unreleased" again: release notes
for an rc read that section, and v0.11.3-rc.1 got none.
The e2e wait for two endpoints is 30s: under the race detector on a busy
runner, 15s was missed once.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The console never mentioned them. A Guardrails view shows each policy with
its mode, fail behaviour and checks, whether its judge models have replicas
ready, where policies are attached (attach and detach from it), and what
they did in a window. The admin API gains GET /api/v1/guard/events, and GET
/api/v1/guard says what each policy does.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The change to read it each time left the constructor storing the address
itself, so the file read each time was one named 10.0.0.161.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
One node at a time was hours on a fleet of hundreds. Nodes now go one, then
two, four, up to --parallel at once, each wave waited on whole; a wave that
does not come back stops it. The other control plane is upgraded first over
ssh, so a leader restarted last hands over to a standby on the release.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
With the agent server closed rather than drained, the process could exit
before Run's release had reached the store, and the standby waited out the
lease. The lease is now given up first, synchronously, and nothing takes it
again after.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
server add records it after restarting the new control plane, which then
never named it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The console and ctl on a standby showed its own empty view: nodes and
replicas live in the leader's memory. A standby now passes /api/ to the
leader over the agent port, with the cluster's certificates both ways; the
leader answers only a certificate naming a control plane the store lists,
and only while it leads. Control plane certificates are issued for client
use as well, and server add reissues older ones.
An agent's reconnect backoff starts again after a session that worked, so
each failover it lives through no longer makes the next one slower.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
keepalived's file holds its password and is root's alone; the control plane
runs as llmgw and could not read it, so the CONTROL line never named the
floating address.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A control plane being stopped held every agent's stream for the thirty
seconds of a graceful shutdown, so the standby led within four seconds and
had no nodes for thirty. The agent server is now closed, not drained. A
control plane that loses its lease while it lives on lets its agents go,
rather than keeping nodes the new leader cannot reach.
ctl status and the console's Overview say which control plane leads, which
stand by, and whether the floating address is on the leader. The console
adds a control plane (Do → Add a control plane), asking for the machine and
the floating address. server add says when the mirror would refuse the new
control plane.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Conflicts with facility-aware placement (config, control, admin API routes,
store test tables) all keep both; the generated reference and OpenAPI
document regenerated.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Conflicts with the LoRA work: vLLM's arguments, the engine profile, router
metrics and the scheduler's request all keep both; agent.pb.go regenerated
from the merged proto; the two test fakes of vLLM given separate names.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ctl federation add/list/remove/sync keeps peers in the store (migration
0029); their /v1/models are offered here as external models named
PEER/MODEL and kept in step, a model still named by a fallback kept
withdrawn. Peers are checked at /ready, timed, and taken out of routing
while down. Requests to a peer carry x-llmgw-hop and are refused (508) when
they come back to a cluster they passed through; residency and no_store
keep requests off peers; X-Request-Id and, with --send-tags, attribution
tags go along. ctl model fallbacks sets overflow to a peer. Console view,
metrics, admin API routes and docs.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Plan §7.5 tier 2, opt-in per profile with kv_events. The agent starts vLLM
publishing KV cache events to a loopback port it listens on (pure-Go zmq4,
msgpack in both of vLLM's encodings), with sha256_cbor hashing and a fixed
PYTHONHASHSEED; it mirrors each engine's blocks and sends net changes up the
existing stream, with a full copy on every new one and a reset on gaps,
restarts and clears. The control plane hands them to the router, for the
sending node's own replicas only. The router tokenizes through a replica's
/tokenize, hashes the tokens as vLLM does (checked against cbor2/hashlib
golden values), and scores replicas by the share of leading blocks held,
falling back to the approximate tree. New metrics beside
llmgw_prefix_decisions_total. Untested against a running vLLM.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
