Measured on three VMs: cutting the store primary's power cost about 45 s of
hanging requests. New connections tried the dead member first with no
connect timeout, and Patroni kept its 30 s leader key. Cutting the leader of
both cost a further 17 s of 503s while the new leader knew no judge.
The store dials each member for 3 s and closes connections to a machine that
is gone within about 10 s (keepalive, TCP_USER_TIMEOUT). Patroni runs with
ttl 20, loop_wait 5 and retry_timeout 5, applied in place to existing
clusters through its API. Judges that all fail at once are asked again
every 250 ms while the policy's budget lasts, and reported once.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Llama Guard 3 1B could not tell ordinary coding requests from harmful ones:
on the PoC it scored "continue" 0.51, "continue the code above" 0.89 and
"how do I kill all child processes in bash" 0.92 unsafe, against 0.97 to
1.00 for harmful requests, at Q4_K_M and Q8_0 alike, with its own template.
A site had "continue" refused. Qwen3Guard-Gen 0.6B gave the ordinary ones
0.15 at the most and the harmful ones 1.00, in 0.8 GB on CPUs.
- format qwen3guard: its prompt written out as its template writes it, for
requests and answers, and its Safe / Unsafe / Controversial verdict read;
controversial counts as the policy's judge.controversial says, safe by
default.
- judge.models: judges asked in turn, the first that answers deciding; a
failure is recorded against the judge that failed. guard judge use sets
the order, replacing categories none of the judges names.
- known judges (Qwen3Guard 0.6B/4B/8B, Llama Guard 1B/8B): guard judge
enable serves any of them, --gpu the 4B on the roomiest GPU class as a
standard model; the leader fetches and serves one a policy names that the
catalog lacks.
- server init installs qwen3guard-0.6b with the baseline's categories.
- a judge reads a long text by its start and end, which fit its context,
rather than overflowing it and refusing the request under fail: closed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
guard.enabled (unset is on) makes every policy check nothing while false,
live: llmgw ctl guard off/on set it in the cluster's settings, and ctl
status, ctl guard list and the guardrails page say when they are off.
A refusal by a judge names the hazard it saw - judge: S6,Specialized
Advice - and the verdict is kept with the request in a new usage column,
guard_rule, shown by ctl request, the request log and the guardrails page.
llmgw ctl guard judge ask has a judge read a text, past the cache, and shows
its raw answer and how it is read; the endpoint is kept out of the audit so
no text is recorded.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
llmgw ctl guard judge status lists each judge model the policies ask:
in the catalog, as a judge, replicas ready, its last failure to answer, and
what is wrong in words. enable serves the missing one through model serve -
Llama Guard 3 1B on CPUs, role guard, under the name the policies give it -
with --replace for a model of that name that is not a judge; the console has
it under Do. GET /api/v1/guard/judges, and the summary's judges, put the
problem on ctl status and the console's Overview.
A Llama Guard file served from the mirror page is added as a judge, under
the name the policies ask, rather than as a model clients use the policies
never find. ctl says plainly when the API answers with a page.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Everything in server.yaml that is not one machine's own is now a versioned
document in the store (cluster_settings), the same on every control plane.
The first start against a store imports the file's cluster keys, with the
baseline guard policy when there is none; after that those keys in a file
are ignored, with a warning.
Each control plane polls the document's version and its own server.yaml's
hash. A change is validated and applied: guard live (Gate.Replace), anything
else by a graceful restart that systemd brings back, standbys first and the
leader 30s later. A version that does not load on a machine is not applied;
at start the newest one that loads is used, so a bad edit cannot keep every
control plane down.
llmgw ctl config show/get/set/unset/edit/history/rollback, the admin API
under /api/v1/settings, and the console's Settings screen, which layers the
store's document over the machine's file, write it. Commands beside the
control plane that read cluster keys (init, node add, server add, backup,
upgrade) read the layered configuration; init --bundle writes offline to
the store once it holds the settings.
A standby that becomes the leader re-reads models, engine choices and
facility holds from the store.
Retires appendGuardIfMissing: the import does it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Found on the PoC: llama.cpp takes a conversation ending on the assistant's
turn as a reply to continue, so Llama Guard answered "Rome.safe" and every
answer judged was served unjudged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Found on the PoC: a streamed list came out as "1. 2. 3." — each item cut at
one release and dropped at the next, which sent the engine's own events.
A property test now cuts an answer every way and checks it arrives whole.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
