The baseline fails closed with two judges, and a judge not answering is a notification
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
07af03aa93
commit
c553f2041a
11 files changed
+67
-22
No files matched your search
+12
-7
@@ -129,8 +129,8 @@ That is the whole step. In order, it:
|
||||
`nvidia-smi` finds any, on CPU otherwise — and waits for it to install its
|
||||
engine
|
||||
9. closes llmgw's internal ports to everything but the cluster's networks
|
||||
10. fetches Llama Guard 3 1B, the guardrails' judge, and runs it on the CPU
|
||||
of a node with room — this one, or the first that joins
|
||||
10. fetches Llama Guard 3 1B, the guardrails' judge, and runs two replicas
|
||||
on the CPUs of nodes with room — this one, or the first that join
|
||||
11. fetches Qwen2.5 0.5B into the mirror and serves it on this node, so the
|
||||
first thing anyone opens `/chat` to find is a model that answers. A
|
||||
machine without room for it is told so within two minutes, and why.
|
||||
@@ -154,11 +154,15 @@ demonstration one:
|
||||
| Requests | every key limited (60 a minute, 200 000 tokens a minute) and priced, so budgets bind |
|
||||
| Guardrails | the `baseline` policy on every request: secrets in prompts and answers refused, card numbers masked, injection attempts recorded, and the judge refusing violence, crime, weapons, child abuse, privacy violations, hate, self-harm and sexual content, in prompts and answers |
|
||||
|
||||
The judge runs on a CPU, about half a second for a short text, and reads a
|
||||
streamed answer every 600 characters. When it is slower than 5 seconds or
|
||||
not running, the request is served and recorded as flagged rather than
|
||||
refused (`fail: open`), so a judge that is down does not take the gateway
|
||||
with it; `fail: closed` turns that round. Defamation, specialized advice,
|
||||
The judge runs as two replicas, each on a CPU — about half a second for a
|
||||
short text on any CPU with AVX2 — and reads a streamed answer every 600
|
||||
characters. The policy fails **closed**: a request the judge cannot read
|
||||
within 5 seconds is refused with 503 rather than served unjudged, and
|
||||
`judge_unavailable` goes to the notification webhooks. Two replicas are what
|
||||
make that bearable; on one small machine, or a VM without AVX2 (see
|
||||
[step 6](#6-setting-the-nodes-cpu-model-if-it-is-a-vm)), the judge is the
|
||||
first thing to fail, so `fail: open` — serve it, recorded as flagged — or
|
||||
`--no-guard-model` is the choice there. Defamation, specialized advice,
|
||||
intellectual property and elections are left out of what it refuses: they
|
||||
catch too much ordinary work. [guardrails.md](guardrails.md) has the rest.
|
||||
|
||||
@@ -977,6 +981,7 @@ control plane sends those itself:
|
||||
| `node_lost` | A node was silent past `control.lost_after`; its replicas are rebuilt elsewhere. |
|
||||
| `budget_spent` | A key, project, team or organization reached its budget; priced models are refused. |
|
||||
| `update_available` | A model's registry repository published different weights. Once per revision. |
|
||||
| `judge_unavailable` | A guardrails judge gave no verdict: with `fail: closed` requests are being refused, with `open` served unjudged. |
|
||||
|
||||
```yaml
|
||||
notify:
|
||||
|
||||
Reference in new issue
Block a user