Qwen3Guard is the judge, and judges on a GPU and on CPUs are asked in turn
Llama Guard 3 1B could not tell ordinary coding requests from harmful ones: on the PoC it scored "continue" 0.51, "continue the code above" 0.89 and "how do I kill all child processes in bash" 0.92 unsafe, against 0.97 to 1.00 for harmful requests, at Q4_K_M and Q8_0 alike, with its own template. A site had "continue" refused. Qwen3Guard-Gen 0.6B gave the ordinary ones 0.15 at the most and the harmful ones 1.00, in 0.8 GB on CPUs. - format qwen3guard: its prompt written out as its template writes it, for requests and answers, and its Safe / Unsafe / Controversial verdict read; controversial counts as the policy's judge.controversial says, safe by default. - judge.models: judges asked in turn, the first that answers deciding; a failure is recorded against the judge that failed. guard judge use sets the order, replacing categories none of the judges names. - known judges (Qwen3Guard 0.6B/4B/8B, Llama Guard 1B/8B): guard judge enable serves any of them, --gpu the 4B on the roomiest GPU class as a standard model; the leader fetches and serves one a policy names that the catalog lacks. - server init installs qwen3guard-0.6b with the baseline's categories. - a judge reads a long text by its start and end, which fit its context, rather than overflowing it and refusing the request under fail: closed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
63f7c8cfd7
commit
b0273ffb5f
24 files changed
+1313
-208
No files matched your search
+25
-5
@@ -102,9 +102,11 @@ rather than repeated.
|
||||
| `format` | `string` | [GuardModel](#guardmodel) |
|
||||
| `categories` | `[]string` | [GuardModel](#guardmodel) |
|
||||
| `model` | `string` | [JudgePolicy](#judgepolicy) |
|
||||
| `models` | `[]string` | [JudgePolicy](#judgepolicy) |
|
||||
| `input` | `string` | [JudgePolicy](#judgepolicy) |
|
||||
| `output` | `string` | [JudgePolicy](#judgepolicy) |
|
||||
| `categories` | `[]string` | [JudgePolicy](#judgepolicy) |
|
||||
| `controversial` | `string` | [JudgePolicy](#judgepolicy) |
|
||||
| `latency_budget` | `time.Duration` | [JudgePolicy](#judgepolicy) |
|
||||
| `tokenize_timeout` | `time.Duration` | [KVIndex](#kvindex) |
|
||||
| `min_blocks` | `int` | [KVIndex](#kvindex) |
|
||||
@@ -994,9 +996,11 @@ every node and control plane within seconds.
|
||||
|
||||
`string`
|
||||
|
||||
Format is llamaguard3 (Meta's Llama Guard 3: "safe", or "unsafe" and
|
||||
the categories), shieldgemma (Google's ShieldGemma: Yes or No), or
|
||||
json ({"safe": false, "categories": ["..."]}).
|
||||
Format is qwen3guard (Qwen3Guard-Gen: "Safety: Safe", "Unsafe" or
|
||||
"Controversial", and the categories), llamaguard3 (Meta's Llama Guard
|
||||
3: "safe", or "unsafe" and the categories), shieldgemma (Google's
|
||||
ShieldGemma: Yes or No), or json ({"safe": false, "categories":
|
||||
["..."]}).
|
||||
|
||||
### `categories`
|
||||
|
||||
@@ -1013,6 +1017,15 @@ sentence — and the names reported for llamaguard3's S1, S2, ….
|
||||
|
||||
Model is a catalog model with role: guard.
|
||||
|
||||
### `models`
|
||||
|
||||
`[]string`
|
||||
|
||||
Models are judges asked in turn, instead of Model: the first that
|
||||
answers gives the verdict. A judge on a GPU first and one on CPUs
|
||||
after it is the GPU's speed while it is there, and the CPUs' when it
|
||||
is not — busy, failed, or the node down.
|
||||
|
||||
### `input`
|
||||
|
||||
`string`
|
||||
@@ -1031,12 +1044,19 @@ Empty does not ask the judge about that direction.
|
||||
Categories, when set, are the only ones that count: Llama Guard's
|
||||
codes (S1, S10) or the names the model reports.
|
||||
|
||||
### `controversial`
|
||||
|
||||
`string`
|
||||
|
||||
Controversial is what Qwen3Guard's third verdict counts as: safe
|
||||
(the default) or unsafe.
|
||||
|
||||
### `latency_budget`
|
||||
|
||||
`time.Duration`
|
||||
|
||||
LatencyBudget is how long the judge is waited for before the policy's
|
||||
fail applies. Zero is 250ms.
|
||||
LatencyBudget is how long each judge is waited for before the next is
|
||||
asked, or, after the last, the policy's fail applies. Zero is 250ms.
|
||||
|
||||
## KVIndex
|
||||
|
||||
|
||||
Reference in new issue
Block a user