The baseline fails closed with two judges, and a judge not answering is a notification
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
07af03aa93
commit
c553f2041a
11 files changed
+67
-22
No files matched your search
+7
-4
@@ -77,10 +77,12 @@ APIs change between minors without a deprecation period.
|
||||
- **Guardrails:** a `baseline` policy on every request. It refuses
|
||||
secrets in prompts and answers, masks card numbers and records injection
|
||||
attempts.
|
||||
- **The judge:** Llama Guard 3 1B, fetched and run on a CPU (about half a
|
||||
second for a short text). It refuses violence, crime, weapons, child
|
||||
abuse, privacy violations, hate, self-harm and sexual content. If it is
|
||||
slower than 5 seconds the request is served, flagged.
|
||||
- **The judge:** Llama Guard 3 1B, two replicas, fetched and run on CPUs
|
||||
(about half a second for a short text). It refuses violence, crime,
|
||||
weapons, child abuse, privacy violations, hate, self-harm and sexual
|
||||
content. The policy fails closed: a request it cannot read within 5
|
||||
seconds is refused (503), and `judge_unavailable` goes to the
|
||||
notification webhooks.
|
||||
- **Keys:** they live at most 90 days.
|
||||
- **Opting out:** `--no-guard-model` (or unticking it on the setup screen)
|
||||
leaves the judge out and keeps the pattern checks. A `server.yaml`
|
||||
@@ -100,6 +102,7 @@ APIs change between minors without a deprecation period.
|
||||
times; the same event about the same thing at most once an hour, and new
|
||||
weights once per revision.
|
||||
- **Checking:** `llmgw ctl notify test`.
|
||||
- **A judge not answering** (`judge_unavailable`) is one of the events.
|
||||
- **Alertmanager receiver templates** for Slack, e-mail, PagerDuty and a
|
||||
webhook in `deploy/monitoring/alertmanager.yml`, commented out until
|
||||
chosen.
|
||||
|
||||
+12
-7
@@ -129,8 +129,8 @@ That is the whole step. In order, it:
|
||||
`nvidia-smi` finds any, on CPU otherwise — and waits for it to install its
|
||||
engine
|
||||
9. closes llmgw's internal ports to everything but the cluster's networks
|
||||
10. fetches Llama Guard 3 1B, the guardrails' judge, and runs it on the CPU
|
||||
of a node with room — this one, or the first that joins
|
||||
10. fetches Llama Guard 3 1B, the guardrails' judge, and runs two replicas
|
||||
on the CPUs of nodes with room — this one, or the first that join
|
||||
11. fetches Qwen2.5 0.5B into the mirror and serves it on this node, so the
|
||||
first thing anyone opens `/chat` to find is a model that answers. A
|
||||
machine without room for it is told so within two minutes, and why.
|
||||
@@ -154,11 +154,15 @@ demonstration one:
|
||||
| Requests | every key limited (60 a minute, 200 000 tokens a minute) and priced, so budgets bind |
|
||||
| Guardrails | the `baseline` policy on every request: secrets in prompts and answers refused, card numbers masked, injection attempts recorded, and the judge refusing violence, crime, weapons, child abuse, privacy violations, hate, self-harm and sexual content, in prompts and answers |
|
||||
|
||||
The judge runs on a CPU, about half a second for a short text, and reads a
|
||||
streamed answer every 600 characters. When it is slower than 5 seconds or
|
||||
not running, the request is served and recorded as flagged rather than
|
||||
refused (`fail: open`), so a judge that is down does not take the gateway
|
||||
with it; `fail: closed` turns that round. Defamation, specialized advice,
|
||||
The judge runs as two replicas, each on a CPU — about half a second for a
|
||||
short text on any CPU with AVX2 — and reads a streamed answer every 600
|
||||
characters. The policy fails **closed**: a request the judge cannot read
|
||||
within 5 seconds is refused with 503 rather than served unjudged, and
|
||||
`judge_unavailable` goes to the notification webhooks. Two replicas are what
|
||||
make that bearable; on one small machine, or a VM without AVX2 (see
|
||||
[step 6](#6-setting-the-nodes-cpu-model-if-it-is-a-vm)), the judge is the
|
||||
first thing to fail, so `fail: open` — serve it, recorded as flagged — or
|
||||
`--no-guard-model` is the choice there. Defamation, specialized advice,
|
||||
intellectual property and elections are left out of what it refuses: they
|
||||
catch too much ordinary work. [guardrails.md](guardrails.md) has the rest.
|
||||
|
||||
@@ -977,6 +981,7 @@ control plane sends those itself:
|
||||
| `node_lost` | A node was silent past `control.lost_after`; its replicas are rebuilt elsewhere. |
|
||||
| `budget_spent` | A key, project, team or organization reached its budget; priced models are refused. |
|
||||
| `update_available` | A model's registry repository published different weights. Once per revision. |
|
||||
| `judge_unavailable` | A guardrails judge gave no verdict: with `fail: closed` requests are being refused, with `open` served unjudged. |
|
||||
|
||||
```yaml
|
||||
notify:
|
||||
|
||||
@@ -8,7 +8,7 @@ explains every choice.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **A control plane** | One machine, or three for high availability (the store replicated, either one able to take over). Linux, x86-64 or ARM64: Debian 12, Ubuntu 22.04+ or RHEL 10. No GPU needed; 2 GB RAM, 20 GB disk each — 6 GB RAM when it is also a node running the starter model and the judge. |
|
||||
| **A control plane** | One machine, or three for high availability (the store replicated, either one able to take over). Linux, x86-64 or ARM64: Debian 12, Ubuntu 22.04+ or RHEL 10. No GPU needed; 2 GB RAM, 20 GB disk each — 8 GB RAM when it is also a node running the starter model and the two judges. |
|
||||
| **GPU nodes** (optional to start) | The same Linux, with the NVIDIA driver installed. vLLM needs driver 525 or newer and a Turing GPU or newer; a V100 gets an older vLLM. |
|
||||
| **root** | on the control plane. On the other two of a three-machine setup, `sudo` (its password is asked for if it needs one); on nodes, `sudo` without a password. |
|
||||
| **SSH** | from the control plane to the other machines. The setup screens take a key, the ssh agent or a password; `node add` uses ssh keys. A machine nothing may log into joins with a code instead (`llmgw join`). |
|
||||
@@ -52,8 +52,8 @@ install:
|
||||
- your administrator account. **The password is shown once. Copy it now.**
|
||||
- a model mirror, which downloads model files once for the whole cluster;
|
||||
- this machine as a first node, and a small model (Qwen2.5 0.5B) on it;
|
||||
- guardrails on every request, with a judge model (Llama Guard 3 1B) on a
|
||||
CPU, and API keys that expire within 90 days. What is hardened, and how
|
||||
- guardrails on every request, with a judge model (Llama Guard 3 1B, two
|
||||
replicas) on CPUs — a request it cannot check is refused, not let through — and API keys that expire within 90 days. What is hardened, and how
|
||||
to loosen it: [install.md](install.md#2-the-control-plane).
|
||||
|
||||
Run again later, `sudo llmgw` is the console: the cluster's state and
|
||||
|
||||
@@ -160,6 +160,14 @@ func newServerCmd() *cobra.Command {
|
||||
return err
|
||||
}
|
||||
rt.SetNotifier(notifier)
|
||||
if gate != nil && notifier != nil {
|
||||
gate.SetJudgeAlarm(func(model string, err error) {
|
||||
notifier.Notify(notify.JudgeUnavailable, model,
|
||||
fmt.Sprintf("the guardrails judge %s gives no verdict (%v): policies with fail: closed "+
|
||||
"refuse what it should read, fail: open serve it unjudged", model, err),
|
||||
map[string]any{"model": model, "error": err.Error()})
|
||||
})
|
||||
}
|
||||
if notifier != nil {
|
||||
log.Info("notifications on", "webhooks", len(cfg.Notify.Webhooks))
|
||||
}
|
||||
|
||||
@@ -285,9 +285,10 @@ func guardSection(judge bool) string {
|
||||
`
|
||||
if judge {
|
||||
j = ` # A judge model reads what patterns cannot: violence, weapons, abuse,
|
||||
# self-harm. It runs on a CPU, about half a second for a short text;
|
||||
# an answer is read every 600 characters as it streams. Slower than
|
||||
# latency_budget, the request is served and recorded as flagged.
|
||||
# self-harm. Two replicas run on CPUs with AVX2, about half a second
|
||||
# for a short text; an answer is read every 600 characters as it
|
||||
# streams. With fail: closed, a request the judge cannot read within
|
||||
# latency_budget is refused (503); open serves it, flagged, unjudged.
|
||||
# Defamation (S5), specialized advice (S6), intellectual property (S8)
|
||||
# and elections (S13) are left out: they refuse too much ordinary work.
|
||||
judge:
|
||||
@@ -305,7 +306,7 @@ guard:
|
||||
default: [baseline]
|
||||
policies:
|
||||
baseline:
|
||||
fail: open
|
||||
fail: closed
|
||||
input:
|
||||
pii: {secret: block, card: redact}
|
||||
injection: flag
|
||||
|
||||
@@ -458,7 +458,9 @@ func guardModel(source string) catalog.Model {
|
||||
Source: source,
|
||||
Role: catalog.RoleGuard,
|
||||
Guard: &catalog.GuardModel{Format: "llamaguard3", Categories: llamaGuard3Categories},
|
||||
Scaling: catalog.Scaling{Min: 1, Max: 1},
|
||||
// Two: every judged request waits on it, and with fail: closed a
|
||||
// judge that is not there refuses everything.
|
||||
Scaling: catalog.Scaling{Min: 2, Max: 2},
|
||||
Profiles: []catalog.Profile{{
|
||||
Name: "cpu", Engine: "llamacpp", Priority: 100, MinMemoryGB: 2,
|
||||
Args: []string{"--ctx-size", "8192"},
|
||||
|
||||
@@ -144,6 +144,9 @@ func TestTheInstalledConfigurationIsHardenedAndLoads(t *testing.T) {
|
||||
t.Errorf("judge=%v: guard.default %v", judge, cfg.Guard.Default)
|
||||
}
|
||||
p := cfg.Guard.Policies["baseline"]
|
||||
if p.Fail != "closed" {
|
||||
t.Errorf("judge=%v: the baseline fails %q", judge, p.Fail)
|
||||
}
|
||||
if (p.Judge != nil) != judge {
|
||||
t.Errorf("judge=%v: the policy's judge is %+v", judge, p.Judge)
|
||||
}
|
||||
@@ -164,7 +167,7 @@ func TestTheGuardModelIsAValidJudge(t *testing.T) {
|
||||
t.Fatalf("not a valid catalog entry: %v", err)
|
||||
}
|
||||
got, _ := c.Resolve(guardModelName)
|
||||
if !got.IsGuard() || got.Guard.Format != "llamaguard3" || len(got.Guard.Categories) != 13 {
|
||||
if !got.IsGuard() || got.Guard.Format != "llamaguard3" || len(got.Guard.Categories) != 13 || got.Scaling.Min != 2 {
|
||||
t.Errorf("%+v", got)
|
||||
}
|
||||
for _, p := range got.Profiles {
|
||||
|
||||
@@ -40,6 +40,8 @@ type Gate struct {
|
||||
tools toolMemo
|
||||
judger *judge
|
||||
m *gateMetrics
|
||||
// alarm is told each time a judge gives no verdict; nil tells no one.
|
||||
alarm func(model string, err error)
|
||||
|
||||
mu sync.Mutex
|
||||
attached map[string][]string // "scope/id" → policy names
|
||||
@@ -363,6 +365,7 @@ func (g *Gate) Input(ctx context.Context, s Subject, body []byte) (Verdict, erro
|
||||
res, err := g.judger.ask(ctx, jp, user, "")
|
||||
if err != nil {
|
||||
g.log.Warn("the judge gave no verdict on a request", "judge", jp.Model, "err", err)
|
||||
g.judgeFailed(jp.Model, err)
|
||||
}
|
||||
if err != nil && p.Fail == "closed" && !v.Shadow {
|
||||
v.Took = g.now().Sub(start)
|
||||
@@ -402,6 +405,16 @@ func (g *Gate) Input(ctx context.Context, s Subject, body []byte) (Verdict, erro
|
||||
|
||||
// judgeFinding is what a judge's answer, or its failing to give one, comes
|
||||
// to under a policy.
|
||||
// SetJudgeAlarm says whom to tell when a judge gives no verdict: set before
|
||||
// the gate serves.
|
||||
func (g *Gate) SetJudgeAlarm(f func(model string, err error)) { g.alarm = f }
|
||||
|
||||
func (g *Gate) judgeFailed(model string, err error) {
|
||||
if g.alarm != nil {
|
||||
g.alarm(model, err)
|
||||
}
|
||||
}
|
||||
|
||||
func judgeFinding(p Policy, action string, res judgment, err error) finding {
|
||||
if err != nil {
|
||||
// Unanswered: closed stops it, open lets it through, said.
|
||||
|
||||
@@ -105,9 +105,14 @@ func TestAJudgeThatDoesNotAnswerFailsAsThePolicySays(t *testing.T) {
|
||||
t.Errorf("fail open: %+v %v after %s", v, err, time.Since(start))
|
||||
}
|
||||
closed := judged(t, Policy{Fail: "closed", Judge: &JudgePolicy{Model: "guard", Input: "block"}}, &fakeJudge{err: errors.New("down")})
|
||||
var alarmed []string
|
||||
closed.SetJudgeAlarm(func(model string, _ error) { alarmed = append(alarmed, model) })
|
||||
if _, err := closed.Input(ctx, Subject{}, chat("user", "hi")); err == nil {
|
||||
t.Error("fail closed served unjudged")
|
||||
}
|
||||
if len(alarmed) != 1 || alarmed[0] != "guard" {
|
||||
t.Errorf("alarm told %v", alarmed)
|
||||
}
|
||||
// A judge that is not a guard model is a judge that cannot be asked.
|
||||
bad := judged(t, Policy{Fail: "closed", Judge: &JudgePolicy{Model: "chat-model", Input: "block"}}, &fakeJudge{})
|
||||
if _, err := bad.Input(ctx, Subject{}, chat("user", "hi")); err == nil {
|
||||
|
||||
@@ -194,6 +194,7 @@ func (f *outFilter) askJudge(text string, final bool) finding {
|
||||
f.took += time.Since(start)
|
||||
if err != nil {
|
||||
f.g.log.Warn("the judge gave no verdict on an answer", "judge", jp.Model, "err", err)
|
||||
f.g.judgeFailed(jp.Model, err)
|
||||
}
|
||||
jf := judgeFinding(f.p, jp.Output, res, err)
|
||||
if jf.failOpen {
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
// Package notify tells the people running the gateway about what needs them,
|
||||
// without a Prometheus in between: a model parked after failing every start,
|
||||
// a node lost, a budget spent, a model's repository with new weights. Each
|
||||
// a node lost, a budget spent, a model's repository with new weights, a
|
||||
// guardrails judge not answering. Each
|
||||
// event goes to the webhooks configured for it — JSON signed with a shared
|
||||
// secret, or a Slack-compatible message.
|
||||
//
|
||||
@@ -34,10 +35,13 @@ const (
|
||||
NodeLost = "node_lost"
|
||||
BudgetSpent = "budget_spent"
|
||||
UpdateAvailable = "update_available"
|
||||
// JudgeUnavailable is a guardrails judge giving no verdict: with
|
||||
// fail: closed, requests are being refused; with open, served unjudged.
|
||||
JudgeUnavailable = "judge_unavailable"
|
||||
)
|
||||
|
||||
// Kinds are every event there is.
|
||||
var Kinds = []string{ModelParked, NodeLost, BudgetSpent, UpdateAvailable}
|
||||
var Kinds = []string{ModelParked, NodeLost, BudgetSpent, UpdateAvailable, JudgeUnavailable}
|
||||
|
||||
// Event is one thing that happened.
|
||||
type Event struct {
|
||||
|
||||
Reference in new issue
Block a user