The baseline fails closed with two judges, and a judge not answering is a notification

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-04 19:19:49 +02:00
1 parent 07af03aa93
commit c553f2041a
11 files changed
+67 -22

No files matched your search

+12 -7
View File
@@ -129,8 +129,8 @@ That is the whole step. In order, it:
`nvidia-smi` finds any, on CPU otherwise — and waits for it to install its
engine
9. closes llmgw's internal ports to everything but the cluster's networks
10. fetches Llama Guard 3 1B, the guardrails' judge, and runs it on the CPU
of a node with room — this one, or the first that joins
10. fetches Llama Guard 3 1B, the guardrails' judge, and runs two replicas
on the CPUs of nodes with room — this one, or the first that join
11. fetches Qwen2.5 0.5B into the mirror and serves it on this node, so the
first thing anyone opens `/chat` to find is a model that answers. A
machine without room for it is told so within two minutes, and why.
@@ -154,11 +154,15 @@ demonstration one:
| Requests | every key limited (60 a minute, 200 000 tokens a minute) and priced, so budgets bind |
| Guardrails | the `baseline` policy on every request: secrets in prompts and answers refused, card numbers masked, injection attempts recorded, and the judge refusing violence, crime, weapons, child abuse, privacy violations, hate, self-harm and sexual content, in prompts and answers |
The judge runs on a CPU, about half a second for a short text, and reads a
streamed answer every 600 characters. When it is slower than 5 seconds or
not running, the request is served and recorded as flagged rather than
refused (`fail: open`), so a judge that is down does not take the gateway
with it; `fail: closed` turns that round. Defamation, specialized advice,
The judge runs as two replicas, each on a CPU — about half a second for a
short text on any CPU with AVX2 — and reads a streamed answer every 600
characters. The policy fails **closed**: a request the judge cannot read
within 5 seconds is refused with 503 rather than served unjudged, and
`judge_unavailable` goes to the notification webhooks. Two replicas are what
make that bearable; on one small machine, or a VM without AVX2 (see
[step 6](#6-setting-the-nodes-cpu-model-if-it-is-a-vm)), the judge is the
first thing to fail, so `fail: open` — serve it, recorded as flagged — or
`--no-guard-model` is the choice there. Defamation, specialized advice,
intellectual property and elections are left out of what it refuses: they
catch too much ordinary work. [guardrails.md](guardrails.md) has the rest.
@@ -977,6 +981,7 @@ control plane sends those itself:
| `node_lost` | A node was silent past `control.lost_after`; its replicas are rebuilt elsewhere. |
| `budget_spent` | A key, project, team or organization reached its budget; priced models are refused. |
| `update_available` | A model's registry repository published different weights. Once per revision. |
| `judge_unavailable` | A guardrails judge gave no verdict: with `fail: closed` requests are being refused, with `open` served unjudged. |
```yaml
notify: