The baseline fails closed with two judges, and a judge not answering is a notification

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-04 19:19:49 +02:00
1 parent 07af03aa93
commit c553f2041a
11 files changed
+67 -22

No files matched your search

+7 -4
View File
@@ -77,10 +77,12 @@ APIs change between minors without a deprecation period.
- **Guardrails:** a `baseline` policy on every request. It refuses
secrets in prompts and answers, masks card numbers and records injection
attempts.
- **The judge:** Llama Guard 3 1B, fetched and run on a CPU (about half a
second for a short text). It refuses violence, crime, weapons, child
abuse, privacy violations, hate, self-harm and sexual content. If it is
slower than 5 seconds the request is served, flagged.
- **The judge:** Llama Guard 3 1B, two replicas, fetched and run on CPUs
(about half a second for a short text). It refuses violence, crime,
weapons, child abuse, privacy violations, hate, self-harm and sexual
content. The policy fails closed: a request it cannot read within 5
seconds is refused (503), and `judge_unavailable` goes to the
notification webhooks.
- **Keys:** they live at most 90 days.
- **Opting out:** `--no-guard-model` (or unticking it on the setup screen)
leaves the judge out and keeps the pattern checks. A `server.yaml`
@@ -100,6 +102,7 @@ APIs change between minors without a deprecation period.
times; the same event about the same thing at most once an hour, and new
weights once per revision.
- **Checking:** `llmgw ctl notify test`.
- **A judge not answering** (`judge_unavailable`) is one of the events.
- **Alertmanager receiver templates** for Slack, e-mail, PagerDuty and a
webhook in `deploy/monitoring/alertmanager.yml`, commented out until
chosen.
+12 -7
View File
@@ -129,8 +129,8 @@ That is the whole step. In order, it:
`nvidia-smi` finds any, on CPU otherwise — and waits for it to install its
engine
9. closes llmgw's internal ports to everything but the cluster's networks
10. fetches Llama Guard 3 1B, the guardrails' judge, and runs it on the CPU
of a node with room — this one, or the first that joins
10. fetches Llama Guard 3 1B, the guardrails' judge, and runs two replicas
on the CPUs of nodes with room — this one, or the first that join
11. fetches Qwen2.5 0.5B into the mirror and serves it on this node, so the
first thing anyone opens `/chat` to find is a model that answers. A
machine without room for it is told so within two minutes, and why.
@@ -154,11 +154,15 @@ demonstration one:
| Requests | every key limited (60 a minute, 200 000 tokens a minute) and priced, so budgets bind |
| Guardrails | the `baseline` policy on every request: secrets in prompts and answers refused, card numbers masked, injection attempts recorded, and the judge refusing violence, crime, weapons, child abuse, privacy violations, hate, self-harm and sexual content, in prompts and answers |
The judge runs on a CPU, about half a second for a short text, and reads a
streamed answer every 600 characters. When it is slower than 5 seconds or
not running, the request is served and recorded as flagged rather than
refused (`fail: open`), so a judge that is down does not take the gateway
with it; `fail: closed` turns that round. Defamation, specialized advice,
The judge runs as two replicas, each on a CPU — about half a second for a
short text on any CPU with AVX2 — and reads a streamed answer every 600
characters. The policy fails **closed**: a request the judge cannot read
within 5 seconds is refused with 503 rather than served unjudged, and
`judge_unavailable` goes to the notification webhooks. Two replicas are what
make that bearable; on one small machine, or a VM without AVX2 (see
[step 6](#6-setting-the-nodes-cpu-model-if-it-is-a-vm)), the judge is the
first thing to fail, so `fail: open` — serve it, recorded as flagged — or
`--no-guard-model` is the choice there. Defamation, specialized advice,
intellectual property and elections are left out of what it refuses: they
catch too much ordinary work. [guardrails.md](guardrails.md) has the rest.
@@ -977,6 +981,7 @@ control plane sends those itself:
| `node_lost` | A node was silent past `control.lost_after`; its replicas are rebuilt elsewhere. |
| `budget_spent` | A key, project, team or organization reached its budget; priced models are refused. |
| `update_available` | A model's registry repository published different weights. Once per revision. |
| `judge_unavailable` | A guardrails judge gave no verdict: with `fail: closed` requests are being refused, with `open` served unjudged. |
```yaml
notify:
+3 -3
View File
@@ -8,7 +8,7 @@ explains every choice.
| | |
|---|---|
| **A control plane** | One machine, or three for high availability (the store replicated, either one able to take over). Linux, x86-64 or ARM64: Debian 12, Ubuntu 22.04+ or RHEL 10. No GPU needed; 2 GB RAM, 20 GB disk each — 6 GB RAM when it is also a node running the starter model and the judge. |
| **A control plane** | One machine, or three for high availability (the store replicated, either one able to take over). Linux, x86-64 or ARM64: Debian 12, Ubuntu 22.04+ or RHEL 10. No GPU needed; 2 GB RAM, 20 GB disk each — 8 GB RAM when it is also a node running the starter model and the two judges. |
| **GPU nodes** (optional to start) | The same Linux, with the NVIDIA driver installed. vLLM needs driver 525 or newer and a Turing GPU or newer; a V100 gets an older vLLM. |
| **root** | on the control plane. On the other two of a three-machine setup, `sudo` (its password is asked for if it needs one); on nodes, `sudo` without a password. |
| **SSH** | from the control plane to the other machines. The setup screens take a key, the ssh agent or a password; `node add` uses ssh keys. A machine nothing may log into joins with a code instead (`llmgw join`). |
@@ -52,8 +52,8 @@ install:
- your administrator account. **The password is shown once. Copy it now.**
- a model mirror, which downloads model files once for the whole cluster;
- this machine as a first node, and a small model (Qwen2.5 0.5B) on it;
- guardrails on every request, with a judge model (Llama Guard 3 1B) on a
CPU, and API keys that expire within 90 days. What is hardened, and how
- guardrails on every request, with a judge model (Llama Guard 3 1B, two
replicas) on CPUs — a request it cannot check is refused, not let through — and API keys that expire within 90 days. What is hardened, and how
to loosen it: [install.md](install.md#2-the-control-plane).
Run again later, `sudo llmgw` is the console: the cluster's state and
+8
View File
@@ -160,6 +160,14 @@ func newServerCmd() *cobra.Command {
return err
}
rt.SetNotifier(notifier)
if gate != nil && notifier != nil {
gate.SetJudgeAlarm(func(model string, err error) {
notifier.Notify(notify.JudgeUnavailable, model,
fmt.Sprintf("the guardrails judge %s gives no verdict (%v): policies with fail: closed "+
"refuse what it should read, fail: open serve it unjudged", model, err),
map[string]any{"model": model, "error": err.Error()})
})
}
if notifier != nil {
log.Info("notifications on", "webhooks", len(cfg.Notify.Webhooks))
}
+5 -4
View File
@@ -285,9 +285,10 @@ func guardSection(judge bool) string {
`
if judge {
j = ` # A judge model reads what patterns cannot: violence, weapons, abuse,
# self-harm. It runs on a CPU, about half a second for a short text;
# an answer is read every 600 characters as it streams. Slower than
# latency_budget, the request is served and recorded as flagged.
# self-harm. Two replicas run on CPUs with AVX2, about half a second
# for a short text; an answer is read every 600 characters as it
# streams. With fail: closed, a request the judge cannot read within
# latency_budget is refused (503); open serves it, flagged, unjudged.
# Defamation (S5), specialized advice (S6), intellectual property (S8)
# and elections (S13) are left out: they refuse too much ordinary work.
judge:
@@ -305,7 +306,7 @@ guard:
default: [baseline]
policies:
baseline:
fail: open
fail: closed
input:
pii: {secret: block, card: redact}
injection: flag
+3 -1
View File
@@ -458,7 +458,9 @@ func guardModel(source string) catalog.Model {
Source: source,
Role: catalog.RoleGuard,
Guard: &catalog.GuardModel{Format: "llamaguard3", Categories: llamaGuard3Categories},
Scaling: catalog.Scaling{Min: 1, Max: 1},
// Two: every judged request waits on it, and with fail: closed a
// judge that is not there refuses everything.
Scaling: catalog.Scaling{Min: 2, Max: 2},
Profiles: []catalog.Profile{{
Name: "cpu", Engine: "llamacpp", Priority: 100, MinMemoryGB: 2,
Args: []string{"--ctx-size", "8192"},
+4 -1
View File
@@ -144,6 +144,9 @@ func TestTheInstalledConfigurationIsHardenedAndLoads(t *testing.T) {
t.Errorf("judge=%v: guard.default %v", judge, cfg.Guard.Default)
}
p := cfg.Guard.Policies["baseline"]
if p.Fail != "closed" {
t.Errorf("judge=%v: the baseline fails %q", judge, p.Fail)
}
if (p.Judge != nil) != judge {
t.Errorf("judge=%v: the policy's judge is %+v", judge, p.Judge)
}
@@ -164,7 +167,7 @@ func TestTheGuardModelIsAValidJudge(t *testing.T) {
t.Fatalf("not a valid catalog entry: %v", err)
}
got, _ := c.Resolve(guardModelName)
if !got.IsGuard() || got.Guard.Format != "llamaguard3" || len(got.Guard.Categories) != 13 {
if !got.IsGuard() || got.Guard.Format != "llamaguard3" || len(got.Guard.Categories) != 13 || got.Scaling.Min != 2 {
t.Errorf("%+v", got)
}
for _, p := range got.Profiles {
+13
View File
@@ -40,6 +40,8 @@ type Gate struct {
tools toolMemo
judger *judge
m *gateMetrics
// alarm is told each time a judge gives no verdict; nil tells no one.
alarm func(model string, err error)
mu sync.Mutex
attached map[string][]string // "scope/id" → policy names
@@ -363,6 +365,7 @@ func (g *Gate) Input(ctx context.Context, s Subject, body []byte) (Verdict, erro
res, err := g.judger.ask(ctx, jp, user, "")
if err != nil {
g.log.Warn("the judge gave no verdict on a request", "judge", jp.Model, "err", err)
g.judgeFailed(jp.Model, err)
}
if err != nil && p.Fail == "closed" && !v.Shadow {
v.Took = g.now().Sub(start)
@@ -402,6 +405,16 @@ func (g *Gate) Input(ctx context.Context, s Subject, body []byte) (Verdict, erro
// judgeFinding is what a judge's answer, or its failing to give one, comes
// to under a policy.
// SetJudgeAlarm says whom to tell when a judge gives no verdict: set before
// the gate serves.
func (g *Gate) SetJudgeAlarm(f func(model string, err error)) { g.alarm = f }
func (g *Gate) judgeFailed(model string, err error) {
if g.alarm != nil {
g.alarm(model, err)
}
}
func judgeFinding(p Policy, action string, res judgment, err error) finding {
if err != nil {
// Unanswered: closed stops it, open lets it through, said.
+5
View File
@@ -105,9 +105,14 @@ func TestAJudgeThatDoesNotAnswerFailsAsThePolicySays(t *testing.T) {
t.Errorf("fail open: %+v %v after %s", v, err, time.Since(start))
}
closed := judged(t, Policy{Fail: "closed", Judge: &JudgePolicy{Model: "guard", Input: "block"}}, &fakeJudge{err: errors.New("down")})
var alarmed []string
closed.SetJudgeAlarm(func(model string, _ error) { alarmed = append(alarmed, model) })
if _, err := closed.Input(ctx, Subject{}, chat("user", "hi")); err == nil {
t.Error("fail closed served unjudged")
}
if len(alarmed) != 1 || alarmed[0] != "guard" {
t.Errorf("alarm told %v", alarmed)
}
// A judge that is not a guard model is a judge that cannot be asked.
bad := judged(t, Policy{Fail: "closed", Judge: &JudgePolicy{Model: "chat-model", Input: "block"}}, &fakeJudge{})
if _, err := bad.Input(ctx, Subject{}, chat("user", "hi")); err == nil {
+1
View File
@@ -194,6 +194,7 @@ func (f *outFilter) askJudge(text string, final bool) finding {
f.took += time.Since(start)
if err != nil {
f.g.log.Warn("the judge gave no verdict on an answer", "judge", jp.Model, "err", err)
f.g.judgeFailed(jp.Model, err)
}
jf := judgeFinding(f.p, jp.Output, res, err)
if jf.failOpen {
+6 -2
View File
@@ -1,6 +1,7 @@
// Package notify tells the people running the gateway about what needs them,
// without a Prometheus in between: a model parked after failing every start,
// a node lost, a budget spent, a model's repository with new weights. Each
// a node lost, a budget spent, a model's repository with new weights, a
// guardrails judge not answering. Each
// event goes to the webhooks configured for it — JSON signed with a shared
// secret, or a Slack-compatible message.
//
@@ -34,10 +35,13 @@ const (
NodeLost = "node_lost"
BudgetSpent = "budget_spent"
UpdateAvailable = "update_available"
// JudgeUnavailable is a guardrails judge giving no verdict: with
// fail: closed, requests are being refused; with open, served unjudged.
JudgeUnavailable = "judge_unavailable"
)
// Kinds are every event there is.
var Kinds = []string{ModelParked, NodeLost, BudgetSpent, UpdateAvailable}
var Kinds = []string{ModelParked, NodeLost, BudgetSpent, UpdateAvailable, JudgeUnavailable}
// Event is one thing that happened.
type Event struct {