Qwen3Guard is the judge, and judges on a GPU and on CPUs are asked in turn

Llama Guard 3 1B could not tell ordinary coding requests from harmful ones:
on the PoC it scored "continue" 0.51, "continue the code above" 0.89 and
"how do I kill all child processes in bash" 0.92 unsafe, against 0.97 to
1.00 for harmful requests, at Q4_K_M and Q8_0 alike, with its own template.
A site had "continue" refused. Qwen3Guard-Gen 0.6B gave the ordinary ones
0.15 at the most and the harmful ones 1.00, in 0.8 GB on CPUs.

- format qwen3guard: its prompt written out as its template writes it, for
  requests and answers, and its Safe / Unsafe / Controversial verdict read;
  controversial counts as the policy's judge.controversial says, safe by
  default.
- judge.models: judges asked in turn, the first that answers deciding; a
  failure is recorded against the judge that failed. guard judge use sets
  the order, replacing categories none of the judges names.
- known judges (Qwen3Guard 0.6B/4B/8B, Llama Guard 1B/8B): guard judge
  enable serves any of them, --gpu the 4B on the roomiest GPU class as a
  standard model; the leader fetches and serves one a policy names that the
  catalog lacks.
- server init installs qwen3guard-0.6b with the baseline's categories.
- a judge reads a long text by its start and end, which fit its context,
  rather than overflowing it and refusing the request under fail: closed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-08 10:22:47 +02:00
1 parent 63f7c8cfd7
commit b0273ffb5f
24 files changed
+1313 -208

No files matched your search

+25 -5
View File
@@ -102,9 +102,11 @@ rather than repeated.
| `format` | `string` | [GuardModel](#guardmodel) |
| `categories` | `[]string` | [GuardModel](#guardmodel) |
| `model` | `string` | [JudgePolicy](#judgepolicy) |
| `models` | `[]string` | [JudgePolicy](#judgepolicy) |
| `input` | `string` | [JudgePolicy](#judgepolicy) |
| `output` | `string` | [JudgePolicy](#judgepolicy) |
| `categories` | `[]string` | [JudgePolicy](#judgepolicy) |
| `controversial` | `string` | [JudgePolicy](#judgepolicy) |
| `latency_budget` | `time.Duration` | [JudgePolicy](#judgepolicy) |
| `tokenize_timeout` | `time.Duration` | [KVIndex](#kvindex) |
| `min_blocks` | `int` | [KVIndex](#kvindex) |
@@ -994,9 +996,11 @@ every node and control plane within seconds.
`string`
Format is llamaguard3 (Meta's Llama Guard 3: "safe", or "unsafe" and
the categories), shieldgemma (Google's ShieldGemma: Yes or No), or
json ({"safe": false, "categories": ["..."]}).
Format is qwen3guard (Qwen3Guard-Gen: "Safety: Safe", "Unsafe" or
"Controversial", and the categories), llamaguard3 (Meta's Llama Guard
3: "safe", or "unsafe" and the categories), shieldgemma (Google's
ShieldGemma: Yes or No), or json ({"safe": false, "categories":
["..."]}).
### `categories`
@@ -1013,6 +1017,15 @@ sentence — and the names reported for llamaguard3's S1, S2, ….
Model is a catalog model with role: guard.
### `models`
`[]string`
Models are judges asked in turn, instead of Model: the first that
answers gives the verdict. A judge on a GPU first and one on CPUs
after it is the GPU's speed while it is there, and the CPUs' when it
is not — busy, failed, or the node down.
### `input`
`string`
@@ -1031,12 +1044,19 @@ Empty does not ask the judge about that direction.
Categories, when set, are the only ones that count: Llama Guard's
codes (S1, S10) or the names the model reports.
### `controversial`
`string`
Controversial is what Qwen3Guard's third verdict counts as: safe
(the default) or unsafe.
### `latency_budget`
`time.Duration`
LatencyBudget is how long the judge is waited for before the policy's
fail applies. Zero is 250ms.
LatencyBudget is how long each judge is waited for before the next is
asked, or, after the last, the policy's fail applies. Zero is 250ms.
## KVIndex