Qwen3Guard is the judge, and judges on a GPU and on CPUs are asked in turn

Llama Guard 3 1B could not tell ordinary coding requests from harmful ones:
on the PoC it scored "continue" 0.51, "continue the code above" 0.89 and
"how do I kill all child processes in bash" 0.92 unsafe, against 0.97 to
1.00 for harmful requests, at Q4_K_M and Q8_0 alike, with its own template.
A site had "continue" refused. Qwen3Guard-Gen 0.6B gave the ordinary ones
0.15 at the most and the harmful ones 1.00, in 0.8 GB on CPUs.

- format qwen3guard: its prompt written out as its template writes it, for
  requests and answers, and its Safe / Unsafe / Controversial verdict read;
  controversial counts as the policy's judge.controversial says, safe by
  default.
- judge.models: judges asked in turn, the first that answers deciding; a
  failure is recorded against the judge that failed. guard judge use sets
  the order, replacing categories none of the judges names.
- known judges (Qwen3Guard 0.6B/4B/8B, Llama Guard 1B/8B): guard judge
  enable serves any of them, --gpu the 4B on the roomiest GPU class as a
  standard model; the leader fetches and serves one a policy names that the
  catalog lacks.
- server init installs qwen3guard-0.6b with the baseline's categories.
- a judge reads a long text by its start and end, which fit its context,
  rather than overflowing it and refusing the request under fail: closed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-08 10:22:47 +02:00
1 parent 63f7c8cfd7
commit b0273ffb5f
24 files changed
+1313 -208

No files matched your search

+17 -2
View File
@@ -4573,6 +4573,13 @@
"type": "string"
}
},
"then": {
"type": "array",
"items": {
"type": "string"
},
"description": "Then are the judges its policies ask after it, when it gives no verdict: a CPU judge behind a GPU one."
},
"in_catalog": {
"type": "boolean",
"description": "InCatalog is whether a model of that name is in the catalog, and Judge whether it is there as a judge (role: guard)."
@@ -4830,6 +4837,10 @@
"source": {
"type": "string"
},
"role": {
"type": "string",
"description": "Role is \"guard\" for a guardrails judge, empty for a model clients use."
},
"min_replicas": {
"type": "integer"
},
@@ -6331,7 +6342,7 @@
"properties": {
"format": {
"type": "string",
"description": "Format is llamaguard3 (Meta's Llama Guard 3: \"safe\", or \"unsafe\" and the categories), shieldgemma (Google's ShieldGemma: Yes or No), or json ({\"safe\": false, \"categories\": [\"...\"]})."
"description": "Format is qwen3guard (Qwen3Guard-Gen: \"Safety: Safe\", \"Unsafe\" or \"Controversial\", and the categories), llamaguard3 (Meta's Llama Guard 3: \"safe\", or \"unsafe\" and the categories), shieldgemma (Google's ShieldGemma: Yes or No), or json ({\"safe\": false, \"categories\": [\"...\"]})."
},
"categories": {
"type": "array",
@@ -7511,7 +7522,11 @@
},
"guard_format": {
"type": "string",
"description": "GuardFormat makes it a judge model for the guardrails (role: guard): llamaguard3, shieldgemma or json. Empty serves clients."
"description": "GuardFormat makes it a judge model for the guardrails (role: guard): llamaguard3, qwen3guard, shieldgemma or json. Empty serves clients."
},
"class": {
"type": "string",
"description": "Class is what it may take from others and they from it when there is not room for both: critical, standard or opportunistic. Empty is the catalog's default for its role."
},
"ctx_pool": {
"type": "integer",