Qwen3Guard is the judge, and judges on a GPU and on CPUs are asked in turn
Llama Guard 3 1B could not tell ordinary coding requests from harmful ones: on the PoC it scored "continue" 0.51, "continue the code above" 0.89 and "how do I kill all child processes in bash" 0.92 unsafe, against 0.97 to 1.00 for harmful requests, at Q4_K_M and Q8_0 alike, with its own template. A site had "continue" refused. Qwen3Guard-Gen 0.6B gave the ordinary ones 0.15 at the most and the harmful ones 1.00, in 0.8 GB on CPUs. - format qwen3guard: its prompt written out as its template writes it, for requests and answers, and its Safe / Unsafe / Controversial verdict read; controversial counts as the policy's judge.controversial says, safe by default. - judge.models: judges asked in turn, the first that answers deciding; a failure is recorded against the judge that failed. guard judge use sets the order, replacing categories none of the judges names. - known judges (Qwen3Guard 0.6B/4B/8B, Llama Guard 1B/8B): guard judge enable serves any of them, --gpu the 4B on the roomiest GPU class as a standard model; the leader fetches and serves one a policy names that the catalog lacks. - server init installs qwen3guard-0.6b with the baseline's categories. - a judge reads a long text by its start and end, which fit its context, rather than overflowing it and refusing the request under fail: closed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
63f7c8cfd7
commit
b0273ffb5f
24 files changed
+1313
-208
No files matched your search
@@ -4573,6 +4573,13 @@
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"then": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "string"
|
||||
},
|
||||
"description": "Then are the judges its policies ask after it, when it gives no verdict: a CPU judge behind a GPU one."
|
||||
},
|
||||
"in_catalog": {
|
||||
"type": "boolean",
|
||||
"description": "InCatalog is whether a model of that name is in the catalog, and Judge whether it is there as a judge (role: guard)."
|
||||
@@ -4830,6 +4837,10 @@
|
||||
"source": {
|
||||
"type": "string"
|
||||
},
|
||||
"role": {
|
||||
"type": "string",
|
||||
"description": "Role is \"guard\" for a guardrails judge, empty for a model clients use."
|
||||
},
|
||||
"min_replicas": {
|
||||
"type": "integer"
|
||||
},
|
||||
@@ -6331,7 +6342,7 @@
|
||||
"properties": {
|
||||
"format": {
|
||||
"type": "string",
|
||||
"description": "Format is llamaguard3 (Meta's Llama Guard 3: \"safe\", or \"unsafe\" and the categories), shieldgemma (Google's ShieldGemma: Yes or No), or json ({\"safe\": false, \"categories\": [\"...\"]})."
|
||||
"description": "Format is qwen3guard (Qwen3Guard-Gen: \"Safety: Safe\", \"Unsafe\" or \"Controversial\", and the categories), llamaguard3 (Meta's Llama Guard 3: \"safe\", or \"unsafe\" and the categories), shieldgemma (Google's ShieldGemma: Yes or No), or json ({\"safe\": false, \"categories\": [\"...\"]})."
|
||||
},
|
||||
"categories": {
|
||||
"type": "array",
|
||||
@@ -7511,7 +7522,11 @@
|
||||
},
|
||||
"guard_format": {
|
||||
"type": "string",
|
||||
"description": "GuardFormat makes it a judge model for the guardrails (role: guard): llamaguard3, shieldgemma or json. Empty serves clients."
|
||||
"description": "GuardFormat makes it a judge model for the guardrails (role: guard): llamaguard3, qwen3guard, shieldgemma or json. Empty serves clients."
|
||||
},
|
||||
"class": {
|
||||
"type": "string",
|
||||
"description": "Class is what it may take from others and they from it when there is not room for both: critical, standard or opportunistic. Empty is the catalog's default for its role."
|
||||
},
|
||||
"ctx_pool": {
|
||||
"type": "integer",
|
||||
|
||||
+95
-35
@@ -4,7 +4,7 @@ Written for the operator deciding what the gateway lets through: what a
|
||||
policy can check, what it can do, and how one is attached to a team.
|
||||
|
||||
`llmgw server init` writes a `baseline` policy that applies to every
|
||||
request, with Llama Guard 3 1B as its judge on a CPU; see
|
||||
request, with Qwen3Guard-Gen 0.6B as its judge on CPUs; see
|
||||
[install.md](install.md#2-the-control-plane) for what it checks. The
|
||||
policies are part of the cluster's settings, kept in the store: `llmgw ctl
|
||||
config edit` (or the console's Settings) changes them, and the change is in
|
||||
@@ -114,79 +114,136 @@ guard:
|
||||
|
||||
## A judge model
|
||||
|
||||
Patterns find what has a shape. For what does not — violence, self-harm,
|
||||
abuse, whatever a moderation model was trained on — a policy can have a
|
||||
**judge model on the cluster** read the text.
|
||||
Patterns find what has a shape. For what does not — violence, crime,
|
||||
sexual content, self-harm, jailbreaks, whatever a moderation model was
|
||||
trained on — a policy can have a **judge model on the cluster** read the
|
||||
text.
|
||||
|
||||
The judge must be in the catalog under the name the policy gives, as a
|
||||
judge (`role: guard`), with a replica ready. When it is not, what the
|
||||
policy judges is refused with `guard_unavailable` (`fail: closed`) or served
|
||||
unjudged (`fail: open`), and `llmgw ctl status` and the console's Overview
|
||||
say which judge and why. To serve it:
|
||||
say which judge and why.
|
||||
|
||||
### Which judge
|
||||
|
||||
| judge | runs on | size | |
|
||||
|---|---|---|---|
|
||||
| `qwen3guard-0.6b` | CPUs | 0.8 GB | the default: about 2 s a check on two threads |
|
||||
| `qwen3guard-4b` | a GPU | 2.5 GB | for a cluster with a card to spare |
|
||||
| `qwen3guard-8b` | a GPU | 4.7 GB | |
|
||||
| `llama-guard-3-1b` | CPUs | 0.9 GB | the earlier default: refuses ordinary coding requests |
|
||||
| `llama-guard-3-8b` | a GPU | 4.6 GB | |
|
||||
|
||||
Measured on the reference cluster, as the judge's probability of "unsafe"
|
||||
for ordinary coding requests against harmful ones, Llama Guard 3 1B could
|
||||
not tell them apart: "continue" 0.51, "continue the code above" 0.89, "how
|
||||
do I kill all child processes in bash" 0.92, against 0.97 to 1.00 for
|
||||
harmful requests, at any quantisation. Qwen3Guard-Gen 0.6B gave the
|
||||
ordinary ones 0.15 at the most and the harmful ones 1.00. Its verdicts are
|
||||
safe, unsafe and **controversial**; controversial counts as safe unless the
|
||||
policy says `controversial: unsafe`.
|
||||
|
||||
A policy that names one of these is enough: the leader fetches its weights
|
||||
into the mirror and serves it as a judge, and serves it again if it is
|
||||
removed from the catalog. Another judge is served by hand with `--repo`.
|
||||
|
||||
```
|
||||
llmgw ctl guard judge status # each judge the policies ask, and what is wrong with it
|
||||
llmgw ctl guard judge enable # Llama Guard 3 1B on CPUs, under the name the policies ask
|
||||
llmgw ctl guard judge enable --replace # when a model of that name is there, but not as a judge
|
||||
llmgw ctl guard judge status # each judge the policies ask, and what is wrong with it
|
||||
llmgw ctl guard judge enable # Qwen3Guard-Gen 0.6B on CPUs, or the judge the policies ask that is missing
|
||||
llmgw ctl guard judge enable --gpu # Qwen3Guard-Gen 4B on a GPU
|
||||
llmgw ctl guard judge use qwen3guard-4b qwen3guard-0.6b # the order every policy asks them in
|
||||
llmgw ctl guard judge enable --replace # when a model of that name is there, but not as a judge
|
||||
```
|
||||
|
||||
or **Do → Serve the guardrails' judge** in the console.
|
||||
|
||||
**When it refuses ordinary requests.** A refusal by a judge names the hazard
|
||||
it saw — the client is told `refused by the gateway's policy (judge:
|
||||
S6,Specialized Advice)` — and the same is kept with the request: `llmgw ctl
|
||||
request <id>`, the server's log (`guard_rule`) and the guardrails page show
|
||||
it. To see what the judge says of any text, without anything being kept:
|
||||
### A GPU and CPUs together
|
||||
|
||||
`judge.models` is a list asked in turn: the first judge that answers gives
|
||||
the verdict. The next is asked when one has no replica serving, fails, or
|
||||
does not answer within `latency_budget`; a judge with no replica at all is
|
||||
passed over at once. A judge on a GPU first and one on CPUs after it is
|
||||
the GPU's speed while it has its card, and the CPUs' when it does not.
|
||||
|
||||
```yaml
|
||||
judge:
|
||||
models: [qwen3guard-4b, qwen3guard-0.6b]
|
||||
input: block
|
||||
output: block
|
||||
latency_budget: 5s # each judge's wait
|
||||
```
|
||||
|
||||
A judge on a GPU is served as a `standard` model, on one card: it takes a
|
||||
card only from what may give one up, and waits while every card is busy,
|
||||
the judge on CPUs standing in. A judge on CPUs is `critical`, with two
|
||||
replicas, because every judged request waits on one. `llmgw ctl guard judge
|
||||
status` says, for each judge, which is asked when it gives no verdict.
|
||||
|
||||
### When it refuses ordinary requests
|
||||
|
||||
A refusal by a judge names the hazard it saw — the client is told `refused
|
||||
by the gateway's policy (judge: Violent)` — and the same is kept with the
|
||||
request: `llmgw ctl request <id>`, the server's log (`guard_rule`) and the
|
||||
guardrails page show it. To see what the judge says of any text, without
|
||||
anything being kept:
|
||||
|
||||
```
|
||||
llmgw ctl guard judge ask 'How do I sort a list in Python?'
|
||||
# llama-guard-3-1b: safe, 140 ms
|
||||
# it answered: "safe"
|
||||
# qwen3guard-0.6b: safe, 1900 ms
|
||||
# it answered: "Safety: Safe\nCategories: None"
|
||||
```
|
||||
|
||||
A judge that says `unsafe` to ordinary text is usually the wrong weights
|
||||
(another repository's or quantisation's, without Llama Guard's template:
|
||||
`judge enable --replace` serves the right ones), or a policy counting every
|
||||
hazard: give its judge `categories`, leaving out S5, S6, S8 and S13 as the
|
||||
baseline does. While finding out, `judge.input: flag` records instead of
|
||||
refusing. A Llama Guard file
|
||||
served from the mirror page is served as a judge too, under the name the
|
||||
policies ask. By hand, the catalog entry is:
|
||||
A judge that says `unsafe` to ordinary text is usually Llama Guard 3 1B
|
||||
(`llmgw ctl guard judge use qwen3guard-0.6b`), the wrong weights (another
|
||||
repository's, without the judge's template: `judge enable --replace` serves
|
||||
the right ones), or a policy counting every category: give its judge
|
||||
`categories`. The baseline counts Violent, Non-violent Illegal Acts, Sexual
|
||||
Content or Sexual Acts, Suicide & Self-Harm and Jailbreak; PII is the
|
||||
patterns', and copyright, unethical acts and politically sensitive topics
|
||||
refuse too much ordinary work. While finding out, `judge.input: flag`
|
||||
records instead of refusing.
|
||||
|
||||
A judge's file served from the mirror page — Llama Guard, Qwen3Guard,
|
||||
ShieldGemma — is served as a judge too, under the name the policies ask.
|
||||
By hand, the catalog entry is:
|
||||
|
||||
```yaml
|
||||
# a catalog model (server.yaml models:, or ctl model add)
|
||||
- name: llama-guard-3-1b
|
||||
source: http://mirror:8081/llama-guard-3-1b/…/Llama-Guard-3-1B-Q4_K_M.gguf
|
||||
- name: qwen3guard-0.6b
|
||||
source: http://mirror:8081/qwen3guard-0.6b/…/Qwen3Guard-Gen-0.6B.Q8_0.gguf
|
||||
role: guard
|
||||
guard:
|
||||
format: llamaguard3 # llamaguard3, shieldgemma or json
|
||||
categories: [Violent Crimes, Non-Violent Crimes, Sex-Related Crimes, …] # names for S1, S2, …
|
||||
format: qwen3guard # qwen3guard, llamaguard3, shieldgemma or json
|
||||
scaling: {min: 2, max: 2}
|
||||
profiles: [{name: cpu, engine: llamacpp}]
|
||||
|
||||
# in a guard policy
|
||||
judge:
|
||||
model: llama-guard-3-1b
|
||||
models: [qwen3guard-0.6b] # or model: one judge
|
||||
input: block # flag or block; empty does not ask about input
|
||||
output: flag
|
||||
categories: [S1, S10] # only these count; empty: any unsafe verdict
|
||||
latency_budget: 250ms # then the policy's fail: open serves (flagged), closed refuses
|
||||
categories: [Violent, Jailbreak] # only these count; empty: any unsafe verdict
|
||||
controversial: safe # or unsafe
|
||||
latency_budget: 250ms # each judge's wait; after the last, fail: open serves (flagged), closed refuses
|
||||
```
|
||||
|
||||
A model with `role: guard` is the guardrails' alone:
|
||||
|
||||
- it is not in `/v1/models` or the chat page, and is refused (404) to
|
||||
every key, as a name and as a fallback;
|
||||
- it is `critical` by default and must always run — on demand is refused.
|
||||
Give it two replicas on different nodes, because every judged request
|
||||
waits on it.
|
||||
- it must always run — on demand is refused. One on CPUs is `critical`
|
||||
and should have two replicas on different nodes, because every judged
|
||||
request waits on it.
|
||||
|
||||
The gateway's calls to it go past admission, accounting and the guardrails
|
||||
themselves.
|
||||
|
||||
- **Formats:**
|
||||
- `llamaguard3` reads Meta's "safe" / "unsafe
|
||||
S1,S10".
|
||||
- `qwen3guard` asks in Qwen3Guard's own words and reads "Safety: Safe |
|
||||
Unsafe | Controversial" and its "Categories:".
|
||||
- `llamaguard3` reads Meta's "safe" / "unsafe\nS1,S10"; give the catalog
|
||||
entry `categories` to have S1 recorded as its name.
|
||||
- `shieldgemma` asks Google's model whether the text breaks the listed
|
||||
principles (`categories`, each a sentence) and reads Yes or No.
|
||||
- `json` expects `{"safe": false, "categories": [...]}`.
|
||||
@@ -196,6 +253,9 @@ S1,S10".
|
||||
is read every 600 characters and at the end, so text between two
|
||||
readings has gone out already; `stream: buffer_all` holds it until it
|
||||
is read.
|
||||
- **A long text** is read by its start and its end (about 6000
|
||||
characters), so a message with a file pasted into it fits the judge's
|
||||
context rather than overflowing it and going unjudged.
|
||||
- **Cache:** the same text is not asked about twice within five minutes.
|
||||
|
||||
## Where a policy applies
|
||||
|
||||
+3
-2
@@ -129,8 +129,9 @@ That is the whole step. In order, it:
|
||||
`nvidia-smi` finds any, on CPU otherwise — and waits for it to install its
|
||||
engine
|
||||
9. closes llmgw's internal ports to everything but the cluster's networks
|
||||
10. fetches Llama Guard 3 1B, the guardrails' judge, and runs two replicas
|
||||
on the CPUs of nodes with room — this one, or the first that join
|
||||
10. fetches Qwen3Guard-Gen 0.6B, the guardrails' judge, and runs two
|
||||
replicas on the CPUs of nodes with room — this one, or the first that
|
||||
join ([guardrails.md](guardrails.md#a-judge-model) has one on a GPU too)
|
||||
11. fetches Qwen2.5 0.5B into the mirror and serves it on this node, so the
|
||||
first thing anyone opens `/chat` to find is a model that answers. A
|
||||
machine without room for it is told so within two minutes, and why.
|
||||
|
||||
@@ -52,7 +52,7 @@ install:
|
||||
- your administrator account. **The password is shown once. Copy it now.**
|
||||
- a model mirror, which downloads model files once for the whole cluster;
|
||||
- this machine as a first node, and a small model (Qwen2.5 0.5B) on it;
|
||||
- guardrails on every request, with a judge model (Llama Guard 3 1B, two
|
||||
- guardrails on every request, with a judge model (Qwen3Guard-Gen 0.6B, two
|
||||
replicas) on CPUs — a request it cannot check is refused, not let through — and API keys that expire within 90 days. What is hardened, and how
|
||||
to loosen it: [install.md](install.md#2-the-control-plane).
|
||||
|
||||
|
||||
+25
-5
@@ -102,9 +102,11 @@ rather than repeated.
|
||||
| `format` | `string` | [GuardModel](#guardmodel) |
|
||||
| `categories` | `[]string` | [GuardModel](#guardmodel) |
|
||||
| `model` | `string` | [JudgePolicy](#judgepolicy) |
|
||||
| `models` | `[]string` | [JudgePolicy](#judgepolicy) |
|
||||
| `input` | `string` | [JudgePolicy](#judgepolicy) |
|
||||
| `output` | `string` | [JudgePolicy](#judgepolicy) |
|
||||
| `categories` | `[]string` | [JudgePolicy](#judgepolicy) |
|
||||
| `controversial` | `string` | [JudgePolicy](#judgepolicy) |
|
||||
| `latency_budget` | `time.Duration` | [JudgePolicy](#judgepolicy) |
|
||||
| `tokenize_timeout` | `time.Duration` | [KVIndex](#kvindex) |
|
||||
| `min_blocks` | `int` | [KVIndex](#kvindex) |
|
||||
@@ -994,9 +996,11 @@ every node and control plane within seconds.
|
||||
|
||||
`string`
|
||||
|
||||
Format is llamaguard3 (Meta's Llama Guard 3: "safe", or "unsafe" and
|
||||
the categories), shieldgemma (Google's ShieldGemma: Yes or No), or
|
||||
json ({"safe": false, "categories": ["..."]}).
|
||||
Format is qwen3guard (Qwen3Guard-Gen: "Safety: Safe", "Unsafe" or
|
||||
"Controversial", and the categories), llamaguard3 (Meta's Llama Guard
|
||||
3: "safe", or "unsafe" and the categories), shieldgemma (Google's
|
||||
ShieldGemma: Yes or No), or json ({"safe": false, "categories":
|
||||
["..."]}).
|
||||
|
||||
### `categories`
|
||||
|
||||
@@ -1013,6 +1017,15 @@ sentence — and the names reported for llamaguard3's S1, S2, ….
|
||||
|
||||
Model is a catalog model with role: guard.
|
||||
|
||||
### `models`
|
||||
|
||||
`[]string`
|
||||
|
||||
Models are judges asked in turn, instead of Model: the first that
|
||||
answers gives the verdict. A judge on a GPU first and one on CPUs
|
||||
after it is the GPU's speed while it is there, and the CPUs' when it
|
||||
is not — busy, failed, or the node down.
|
||||
|
||||
### `input`
|
||||
|
||||
`string`
|
||||
@@ -1031,12 +1044,19 @@ Empty does not ask the judge about that direction.
|
||||
Categories, when set, are the only ones that count: Llama Guard's
|
||||
codes (S1, S10) or the names the model reports.
|
||||
|
||||
### `controversial`
|
||||
|
||||
`string`
|
||||
|
||||
Controversial is what Qwen3Guard's third verdict counts as: safe
|
||||
(the default) or unsafe.
|
||||
|
||||
### `latency_budget`
|
||||
|
||||
`time.Duration`
|
||||
|
||||
LatencyBudget is how long the judge is waited for before the policy's
|
||||
fail applies. Zero is 250ms.
|
||||
LatencyBudget is how long each judge is waited for before the next is
|
||||
asked, or, after the last, the policy's fail applies. Zero is 250ms.
|
||||
|
||||
## KVIndex
|
||||
|
||||
|
||||
Reference in new issue
Block a user