Commit Graph
10 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 85f135311e How fast the judge is, measured, and said at install
CI / test (push) Successful in 5m29s
ctl guard judge check, and POST /api/v1/guard/judges/check, time three
judgments of a 280-token prompt, then a burst at once, against the
policies' wait: ok, tight or slow, about how many requests a second can
be judged, and what to do. server init runs it once the judge is up and
says a machine too slow before the next steps; the wait is not changed
without asking. BACKLOG reordered by what goes wrong where installed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-20 22:35:19 +02:00
Ilker ManapandClaude Opus 5.5 5a4b022dcd A lost store machine is given up on in seconds, and a new leader waits for its judge
CI / test (push) Failing after 30m6s
Measured on three VMs: cutting the store primary's power cost about 45 s of
hanging requests. New connections tried the dead member first with no
connect timeout, and Patroni kept its 30 s leader key. Cutting the leader of
both cost a further 17 s of 503s while the new leader knew no judge.

The store dials each member for 3 s and closes connections to a machine that
is gone within about 10 s (keepalive, TCP_USER_TIMEOUT). Patroni runs with
ttl 20, loop_wait 5 and retry_timeout 5, applied in place to existing
clusters through its API. Judges that all fail at once are asked again
every 250 ms while the policy's budget lasts, and reported once.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-15 20:33:13 +02:00
Ilker ManapandClaude Opus 5.5 b0273ffb5f Qwen3Guard is the judge, and judges on a GPU and on CPUs are asked in turn
Llama Guard 3 1B could not tell ordinary coding requests from harmful ones:
on the PoC it scored "continue" 0.51, "continue the code above" 0.89 and
"how do I kill all child processes in bash" 0.92 unsafe, against 0.97 to
1.00 for harmful requests, at Q4_K_M and Q8_0 alike, with its own template.
A site had "continue" refused. Qwen3Guard-Gen 0.6B gave the ordinary ones
0.15 at the most and the harmful ones 1.00, in 0.8 GB on CPUs.

- format qwen3guard: its prompt written out as its template writes it, for
  requests and answers, and its Safe / Unsafe / Controversial verdict read;
  controversial counts as the policy's judge.controversial says, safe by
  default.
- judge.models: judges asked in turn, the first that answers deciding; a
  failure is recorded against the judge that failed. guard judge use sets
  the order, replacing categories none of the judges names.
- known judges (Qwen3Guard 0.6B/4B/8B, Llama Guard 1B/8B): guard judge
  enable serves any of them, --gpu the 4B on the roomiest GPU class as a
  standard model; the leader fetches and serves one a policy names that the
  catalog lacks.
- server init installs qwen3guard-0.6b with the baseline's categories.
- a judge reads a long text by its start and end, which fit its context,
  rather than overflowing it and refusing the request under fail: closed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 10:22:47 +02:00
Ilker ManapandClaude Opus 5.5 4721b8e353 Llama Guard 3's verdicts carry their hazards' names without a categories list
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-06 12:44:53 +02:00
Ilker ManapandClaude Opus 5.5 249a91aeec Guardrails turn off and on; a judge's refusal says which hazard; ctl guard judge ask
guard.enabled (unset is on) makes every policy check nothing while false,
live: llmgw ctl guard off/on set it in the cluster's settings, and ctl
status, ctl guard list and the guardrails page say when they are off.

A refusal by a judge names the hazard it saw - judge: S6,Specialized
Advice - and the verdict is kept with the request in a new usage column,
guard_rule, shown by ctl request, the request log and the guardrails page.

llmgw ctl guard judge ask has a judge read a text, past the cache, and shows
its raw answer and how it is read; the endpoint is kept out of the audit so
no text is recorded.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-06 12:41:08 +02:00
Ilker ManapandClaude Opus 5.5 9136bc71c2 More of OpenAI's API: Responses structured output, images and previous_response_id; rerank, score and audio passed through; moderations by the judge
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 22:56:25 +02:00
Ilker ManapandClaude Opus 5.5 dbcecd1e85 Judge an answer with Llama Guard 3's own prompt, as a completion, not a conversation ending on the answer
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 15:35:57 +02:00
Ilker ManapandClaude Opus 5.5 742d6633e2 Read a judge's verdict after the answer llama.cpp goes on from, and log a verdict that cannot be read
Found on the PoC: llama.cpp takes a conversation ending on the assistant's
turn as a reply to continue, so Llama Guard answered "Rome.safe" and every
answer judged was served unjudged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 15:33:23 +02:00
Ilker ManapandClaude Opus 5.5 47e847423d Guardrail metrics and alerts: checks, latency, judge failures, fail-open, redactions, stopped streams, tenant blocks
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 15:11:50 +02:00
Ilker ManapandClaude Opus 5.5 43f6a585a6 A judge model on the cluster: role guard in the catalog, judge in a guard policy
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 15:09:31 +02:00