Qwen3Guard is the judge, and judges on a GPU and on CPUs are asked in turn

Llama Guard 3 1B could not tell ordinary coding requests from harmful ones:
on the PoC it scored "continue" 0.51, "continue the code above" 0.89 and
"how do I kill all child processes in bash" 0.92 unsafe, against 0.97 to
1.00 for harmful requests, at Q4_K_M and Q8_0 alike, with its own template.
A site had "continue" refused. Qwen3Guard-Gen 0.6B gave the ordinary ones
0.15 at the most and the harmful ones 1.00, in 0.8 GB on CPUs.

- format qwen3guard: its prompt written out as its template writes it, for
  requests and answers, and its Safe / Unsafe / Controversial verdict read;
  controversial counts as the policy's judge.controversial says, safe by
  default.
- judge.models: judges asked in turn, the first that answers deciding; a
  failure is recorded against the judge that failed. guard judge use sets
  the order, replacing categories none of the judges names.
- known judges (Qwen3Guard 0.6B/4B/8B, Llama Guard 1B/8B): guard judge
  enable serves any of them, --gpu the 4B on the roomiest GPU class as a
  standard model; the leader fetches and serves one a policy names that the
  catalog lacks.
- server init installs qwen3guard-0.6b with the baseline's categories.
- a judge reads a long text by its start and end, which fit its context,
  rather than overflowing it and refusing the request under fail: closed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-08 10:22:47 +02:00
1 parent 63f7c8cfd7
commit b0273ffb5f
24 files changed
+1313 -208

No files matched your search

+17 -2
View File
@@ -4573,6 +4573,13 @@
"type": "string"
}
},
"then": {
"type": "array",
"items": {
"type": "string"
},
"description": "Then are the judges its policies ask after it, when it gives no verdict: a CPU judge behind a GPU one."
},
"in_catalog": {
"type": "boolean",
"description": "InCatalog is whether a model of that name is in the catalog, and Judge whether it is there as a judge (role: guard)."
@@ -4830,6 +4837,10 @@
"source": {
"type": "string"
},
"role": {
"type": "string",
"description": "Role is \"guard\" for a guardrails judge, empty for a model clients use."
},
"min_replicas": {
"type": "integer"
},
@@ -6331,7 +6342,7 @@
"properties": {
"format": {
"type": "string",
"description": "Format is llamaguard3 (Meta's Llama Guard 3: \"safe\", or \"unsafe\" and the categories), shieldgemma (Google's ShieldGemma: Yes or No), or json ({\"safe\": false, \"categories\": [\"...\"]})."
"description": "Format is qwen3guard (Qwen3Guard-Gen: \"Safety: Safe\", \"Unsafe\" or \"Controversial\", and the categories), llamaguard3 (Meta's Llama Guard 3: \"safe\", or \"unsafe\" and the categories), shieldgemma (Google's ShieldGemma: Yes or No), or json ({\"safe\": false, \"categories\": [\"...\"]})."
},
"categories": {
"type": "array",
@@ -7511,7 +7522,11 @@
},
"guard_format": {
"type": "string",
"description": "GuardFormat makes it a judge model for the guardrails (role: guard): llamaguard3, shieldgemma or json. Empty serves clients."
"description": "GuardFormat makes it a judge model for the guardrails (role: guard): llamaguard3, qwen3guard, shieldgemma or json. Empty serves clients."
},
"class": {
"type": "string",
"description": "Class is what it may take from others and they from it when there is not room for both: critical, standard or opportunistic. Empty is the catalog's default for its role."
},
"ctx_pool": {
"type": "integer",
+95 -35
View File
@@ -4,7 +4,7 @@ Written for the operator deciding what the gateway lets through: what a
policy can check, what it can do, and how one is attached to a team.
`llmgw server init` writes a `baseline` policy that applies to every
request, with Llama Guard 3 1B as its judge on a CPU; see
request, with Qwen3Guard-Gen 0.6B as its judge on CPUs; see
[install.md](install.md#2-the-control-plane) for what it checks. The
policies are part of the cluster's settings, kept in the store: `llmgw ctl
config edit` (or the console's Settings) changes them, and the change is in
@@ -114,79 +114,136 @@ guard:
## A judge model
Patterns find what has a shape. For what does not — violence, self-harm,
abuse, whatever a moderation model was trained on — a policy can have a
**judge model on the cluster** read the text.
Patterns find what has a shape. For what does not — violence, crime,
sexual content, self-harm, jailbreaks, whatever a moderation model was
trained on — a policy can have a **judge model on the cluster** read the
text.
The judge must be in the catalog under the name the policy gives, as a
judge (`role: guard`), with a replica ready. When it is not, what the
policy judges is refused with `guard_unavailable` (`fail: closed`) or served
unjudged (`fail: open`), and `llmgw ctl status` and the console's Overview
say which judge and why. To serve it:
say which judge and why.
### Which judge
| judge | runs on | size | |
|---|---|---|---|
| `qwen3guard-0.6b` | CPUs | 0.8 GB | the default: about 2 s a check on two threads |
| `qwen3guard-4b` | a GPU | 2.5 GB | for a cluster with a card to spare |
| `qwen3guard-8b` | a GPU | 4.7 GB | |
| `llama-guard-3-1b` | CPUs | 0.9 GB | the earlier default: refuses ordinary coding requests |
| `llama-guard-3-8b` | a GPU | 4.6 GB | |
Measured on the reference cluster, as the judge's probability of "unsafe"
for ordinary coding requests against harmful ones, Llama Guard 3 1B could
not tell them apart: "continue" 0.51, "continue the code above" 0.89, "how
do I kill all child processes in bash" 0.92, against 0.97 to 1.00 for
harmful requests, at any quantisation. Qwen3Guard-Gen 0.6B gave the
ordinary ones 0.15 at the most and the harmful ones 1.00. Its verdicts are
safe, unsafe and **controversial**; controversial counts as safe unless the
policy says `controversial: unsafe`.
A policy that names one of these is enough: the leader fetches its weights
into the mirror and serves it as a judge, and serves it again if it is
removed from the catalog. Another judge is served by hand with `--repo`.
```
llmgw ctl guard judge status # each judge the policies ask, and what is wrong with it
llmgw ctl guard judge enable # Llama Guard 3 1B on CPUs, under the name the policies ask
llmgw ctl guard judge enable --replace # when a model of that name is there, but not as a judge
llmgw ctl guard judge status # each judge the policies ask, and what is wrong with it
llmgw ctl guard judge enable # Qwen3Guard-Gen 0.6B on CPUs, or the judge the policies ask that is missing
llmgw ctl guard judge enable --gpu # Qwen3Guard-Gen 4B on a GPU
llmgw ctl guard judge use qwen3guard-4b qwen3guard-0.6b # the order every policy asks them in
llmgw ctl guard judge enable --replace # when a model of that name is there, but not as a judge
```
or **Do → Serve the guardrails' judge** in the console.
**When it refuses ordinary requests.** A refusal by a judge names the hazard
it saw — the client is told `refused by the gateway's policy (judge:
S6,Specialized Advice)` — and the same is kept with the request: `llmgw ctl
request <id>`, the server's log (`guard_rule`) and the guardrails page show
it. To see what the judge says of any text, without anything being kept:
### A GPU and CPUs together
`judge.models` is a list asked in turn: the first judge that answers gives
the verdict. The next is asked when one has no replica serving, fails, or
does not answer within `latency_budget`; a judge with no replica at all is
passed over at once. A judge on a GPU first and one on CPUs after it is
the GPU's speed while it has its card, and the CPUs' when it does not.
```yaml
judge:
models: [qwen3guard-4b, qwen3guard-0.6b]
input: block
output: block
latency_budget: 5s # each judge's wait
```
A judge on a GPU is served as a `standard` model, on one card: it takes a
card only from what may give one up, and waits while every card is busy,
the judge on CPUs standing in. A judge on CPUs is `critical`, with two
replicas, because every judged request waits on one. `llmgw ctl guard judge
status` says, for each judge, which is asked when it gives no verdict.
### When it refuses ordinary requests
A refusal by a judge names the hazard it saw — the client is told `refused
by the gateway's policy (judge: Violent)` — and the same is kept with the
request: `llmgw ctl request <id>`, the server's log (`guard_rule`) and the
guardrails page show it. To see what the judge says of any text, without
anything being kept:
```
llmgw ctl guard judge ask 'How do I sort a list in Python?'
# llama-guard-3-1b: safe, 140 ms
# it answered: "safe"
# qwen3guard-0.6b: safe, 1900 ms
# it answered: "Safety: Safe\nCategories: None"
```
A judge that says `unsafe` to ordinary text is usually the wrong weights
(another repository's or quantisation's, without Llama Guard's template:
`judge enable --replace` serves the right ones), or a policy counting every
hazard: give its judge `categories`, leaving out S5, S6, S8 and S13 as the
baseline does. While finding out, `judge.input: flag` records instead of
refusing. A Llama Guard file
served from the mirror page is served as a judge too, under the name the
policies ask. By hand, the catalog entry is:
A judge that says `unsafe` to ordinary text is usually Llama Guard 3 1B
(`llmgw ctl guard judge use qwen3guard-0.6b`), the wrong weights (another
repository's, without the judge's template: `judge enable --replace` serves
the right ones), or a policy counting every category: give its judge
`categories`. The baseline counts Violent, Non-violent Illegal Acts, Sexual
Content or Sexual Acts, Suicide & Self-Harm and Jailbreak; PII is the
patterns', and copyright, unethical acts and politically sensitive topics
refuse too much ordinary work. While finding out, `judge.input: flag`
records instead of refusing.
A judge's file served from the mirror page — Llama Guard, Qwen3Guard,
ShieldGemma — is served as a judge too, under the name the policies ask.
By hand, the catalog entry is:
```yaml
# a catalog model (server.yaml models:, or ctl model add)
- name: llama-guard-3-1b
source: http://mirror:8081/llama-guard-3-1b/…/Llama-Guard-3-1B-Q4_K_M.gguf
- name: qwen3guard-0.6b
source: http://mirror:8081/qwen3guard-0.6b/…/Qwen3Guard-Gen-0.6B.Q8_0.gguf
role: guard
guard:
format: llamaguard3 # llamaguard3, shieldgemma or json
categories: [Violent Crimes, Non-Violent Crimes, Sex-Related Crimes, …] # names for S1, S2, …
format: qwen3guard # qwen3guard, llamaguard3, shieldgemma or json
scaling: {min: 2, max: 2}
profiles: [{name: cpu, engine: llamacpp}]
# in a guard policy
judge:
model: llama-guard-3-1b
models: [qwen3guard-0.6b] # or model: one judge
input: block # flag or block; empty does not ask about input
output: flag
categories: [S1, S10] # only these count; empty: any unsafe verdict
latency_budget: 250ms # then the policy's fail: open serves (flagged), closed refuses
categories: [Violent, Jailbreak] # only these count; empty: any unsafe verdict
controversial: safe # or unsafe
latency_budget: 250ms # each judge's wait; after the last, fail: open serves (flagged), closed refuses
```
A model with `role: guard` is the guardrails' alone:
- it is not in `/v1/models` or the chat page, and is refused (404) to
every key, as a name and as a fallback;
- it is `critical` by default and must always run — on demand is refused.
Give it two replicas on different nodes, because every judged request
waits on it.
- it must always run — on demand is refused. One on CPUs is `critical`
and should have two replicas on different nodes, because every judged
request waits on it.
The gateway's calls to it go past admission, accounting and the guardrails
themselves.
- **Formats:**
- `llamaguard3` reads Meta's "safe" / "unsafe
S1,S10".
- `qwen3guard` asks in Qwen3Guard's own words and reads "Safety: Safe |
Unsafe | Controversial" and its "Categories:".
- `llamaguard3` reads Meta's "safe" / "unsafe\nS1,S10"; give the catalog
entry `categories` to have S1 recorded as its name.
- `shieldgemma` asks Google's model whether the text breaks the listed
principles (`categories`, each a sentence) and reads Yes or No.
- `json` expects `{"safe": false, "categories": [...]}`.
@@ -196,6 +253,9 @@ S1,S10".
is read every 600 characters and at the end, so text between two
readings has gone out already; `stream: buffer_all` holds it until it
is read.
- **A long text** is read by its start and its end (about 6000
characters), so a message with a file pasted into it fits the judge's
context rather than overflowing it and going unjudged.
- **Cache:** the same text is not asked about twice within five minutes.
## Where a policy applies
+3 -2
View File
@@ -129,8 +129,9 @@ That is the whole step. In order, it:
`nvidia-smi` finds any, on CPU otherwise — and waits for it to install its
engine
9. closes llmgw's internal ports to everything but the cluster's networks
10. fetches Llama Guard 3 1B, the guardrails' judge, and runs two replicas
on the CPUs of nodes with room — this one, or the first that join
10. fetches Qwen3Guard-Gen 0.6B, the guardrails' judge, and runs two
replicas on the CPUs of nodes with room — this one, or the first that
join ([guardrails.md](guardrails.md#a-judge-model) has one on a GPU too)
11. fetches Qwen2.5 0.5B into the mirror and serves it on this node, so the
first thing anyone opens `/chat` to find is a model that answers. A
machine without room for it is told so within two minutes, and why.
+1 -1
View File
@@ -52,7 +52,7 @@ install:
- your administrator account. **The password is shown once. Copy it now.**
- a model mirror, which downloads model files once for the whole cluster;
- this machine as a first node, and a small model (Qwen2.5 0.5B) on it;
- guardrails on every request, with a judge model (Llama Guard 3 1B, two
- guardrails on every request, with a judge model (Qwen3Guard-Gen 0.6B, two
replicas) on CPUs — a request it cannot check is refused, not let through — and API keys that expire within 90 days. What is hardened, and how
to loosen it: [install.md](install.md#2-the-control-plane).
+25 -5
View File
@@ -102,9 +102,11 @@ rather than repeated.
| `format` | `string` | [GuardModel](#guardmodel) |
| `categories` | `[]string` | [GuardModel](#guardmodel) |
| `model` | `string` | [JudgePolicy](#judgepolicy) |
| `models` | `[]string` | [JudgePolicy](#judgepolicy) |
| `input` | `string` | [JudgePolicy](#judgepolicy) |
| `output` | `string` | [JudgePolicy](#judgepolicy) |
| `categories` | `[]string` | [JudgePolicy](#judgepolicy) |
| `controversial` | `string` | [JudgePolicy](#judgepolicy) |
| `latency_budget` | `time.Duration` | [JudgePolicy](#judgepolicy) |
| `tokenize_timeout` | `time.Duration` | [KVIndex](#kvindex) |
| `min_blocks` | `int` | [KVIndex](#kvindex) |
@@ -994,9 +996,11 @@ every node and control plane within seconds.
`string`
Format is llamaguard3 (Meta's Llama Guard 3: "safe", or "unsafe" and
the categories), shieldgemma (Google's ShieldGemma: Yes or No), or
json ({"safe": false, "categories": ["..."]}).
Format is qwen3guard (Qwen3Guard-Gen: "Safety: Safe", "Unsafe" or
"Controversial", and the categories), llamaguard3 (Meta's Llama Guard
3: "safe", or "unsafe" and the categories), shieldgemma (Google's
ShieldGemma: Yes or No), or json ({"safe": false, "categories":
["..."]}).
### `categories`
@@ -1013,6 +1017,15 @@ sentence — and the names reported for llamaguard3's S1, S2, ….
Model is a catalog model with role: guard.
### `models`
`[]string`
Models are judges asked in turn, instead of Model: the first that
answers gives the verdict. A judge on a GPU first and one on CPUs
after it is the GPU's speed while it is there, and the CPUs' when it
is not — busy, failed, or the node down.
### `input`
`string`
@@ -1031,12 +1044,19 @@ Empty does not ask the judge about that direction.
Categories, when set, are the only ones that count: Llama Guard's
codes (S1, S10) or the names the model reports.
### `controversial`
`string`
Controversial is what Qwen3Guard's third verdict counts as: safe
(the default) or unsafe.
### `latency_budget`
`time.Duration`
LatencyBudget is how long the judge is waited for before the policy's
fail applies. Zero is 250ms.
LatencyBudget is how long each judge is waited for before the next is
asked, or, after the last, the policy's fail applies. Zero is 250ms.
## KVIndex