diff --git a/CHANGELOG.md b/CHANGELOG.md index 3918d91..cf7b0f4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -77,10 +77,12 @@ APIs change between minors without a deprecation period. - **Guardrails:** a `baseline` policy on every request. It refuses secrets in prompts and answers, masks card numbers and records injection attempts. - - **The judge:** Llama Guard 3 1B, fetched and run on a CPU (about half a - second for a short text). It refuses violence, crime, weapons, child - abuse, privacy violations, hate, self-harm and sexual content. If it is - slower than 5 seconds the request is served, flagged. + - **The judge:** Llama Guard 3 1B, two replicas, fetched and run on CPUs + (about half a second for a short text). It refuses violence, crime, + weapons, child abuse, privacy violations, hate, self-harm and sexual + content. The policy fails closed: a request it cannot read within 5 + seconds is refused (503), and `judge_unavailable` goes to the + notification webhooks. - **Keys:** they live at most 90 days. - **Opting out:** `--no-guard-model` (or unticking it on the setup screen) leaves the judge out and keeps the pattern checks. A `server.yaml` @@ -100,6 +102,7 @@ APIs change between minors without a deprecation period. times; the same event about the same thing at most once an hour, and new weights once per revision. - **Checking:** `llmgw ctl notify test`. + - **A judge not answering** (`judge_unavailable`) is one of the events. - **Alertmanager receiver templates** for Slack, e-mail, PagerDuty and a webhook in `deploy/monitoring/alertmanager.yml`, commented out until chosen. diff --git a/docs/install.md b/docs/install.md index 41efc5a..9059a9b 100644 --- a/docs/install.md +++ b/docs/install.md @@ -129,8 +129,8 @@ That is the whole step. In order, it: `nvidia-smi` finds any, on CPU otherwise — and waits for it to install its engine 9. closes llmgw's internal ports to everything but the cluster's networks -10. fetches Llama Guard 3 1B, the guardrails' judge, and runs it on the CPU - of a node with room — this one, or the first that joins +10. fetches Llama Guard 3 1B, the guardrails' judge, and runs two replicas + on the CPUs of nodes with room — this one, or the first that join 11. fetches Qwen2.5 0.5B into the mirror and serves it on this node, so the first thing anyone opens `/chat` to find is a model that answers. A machine without room for it is told so within two minutes, and why. @@ -154,11 +154,15 @@ demonstration one: | Requests | every key limited (60 a minute, 200 000 tokens a minute) and priced, so budgets bind | | Guardrails | the `baseline` policy on every request: secrets in prompts and answers refused, card numbers masked, injection attempts recorded, and the judge refusing violence, crime, weapons, child abuse, privacy violations, hate, self-harm and sexual content, in prompts and answers | -The judge runs on a CPU, about half a second for a short text, and reads a -streamed answer every 600 characters. When it is slower than 5 seconds or -not running, the request is served and recorded as flagged rather than -refused (`fail: open`), so a judge that is down does not take the gateway -with it; `fail: closed` turns that round. Defamation, specialized advice, +The judge runs as two replicas, each on a CPU — about half a second for a +short text on any CPU with AVX2 — and reads a streamed answer every 600 +characters. The policy fails **closed**: a request the judge cannot read +within 5 seconds is refused with 503 rather than served unjudged, and +`judge_unavailable` goes to the notification webhooks. Two replicas are what +make that bearable; on one small machine, or a VM without AVX2 (see +[step 6](#6-setting-the-nodes-cpu-model-if-it-is-a-vm)), the judge is the +first thing to fail, so `fail: open` — serve it, recorded as flagged — or +`--no-guard-model` is the choice there. Defamation, specialized advice, intellectual property and elections are left out of what it refuses: they catch too much ordinary work. [guardrails.md](guardrails.md) has the rest. @@ -977,6 +981,7 @@ control plane sends those itself: | `node_lost` | A node was silent past `control.lost_after`; its replicas are rebuilt elsewhere. | | `budget_spent` | A key, project, team or organization reached its budget; priced models are refused. | | `update_available` | A model's registry repository published different weights. Once per revision. | +| `judge_unavailable` | A guardrails judge gave no verdict: with `fail: closed` requests are being refused, with `open` served unjudged. | ```yaml notify: diff --git a/docs/quick-install.md b/docs/quick-install.md index cfd8c7c..c97037e 100644 --- a/docs/quick-install.md +++ b/docs/quick-install.md @@ -8,7 +8,7 @@ explains every choice. | | | |---|---| -| **A control plane** | One machine, or three for high availability (the store replicated, either one able to take over). Linux, x86-64 or ARM64: Debian 12, Ubuntu 22.04+ or RHEL 10. No GPU needed; 2 GB RAM, 20 GB disk each — 6 GB RAM when it is also a node running the starter model and the judge. | +| **A control plane** | One machine, or three for high availability (the store replicated, either one able to take over). Linux, x86-64 or ARM64: Debian 12, Ubuntu 22.04+ or RHEL 10. No GPU needed; 2 GB RAM, 20 GB disk each — 8 GB RAM when it is also a node running the starter model and the two judges. | | **GPU nodes** (optional to start) | The same Linux, with the NVIDIA driver installed. vLLM needs driver 525 or newer and a Turing GPU or newer; a V100 gets an older vLLM. | | **root** | on the control plane. On the other two of a three-machine setup, `sudo` (its password is asked for if it needs one); on nodes, `sudo` without a password. | | **SSH** | from the control plane to the other machines. The setup screens take a key, the ssh agent or a password; `node add` uses ssh keys. A machine nothing may log into joins with a code instead (`llmgw join`). | @@ -52,8 +52,8 @@ install: - your administrator account. **The password is shown once. Copy it now.** - a model mirror, which downloads model files once for the whole cluster; - this machine as a first node, and a small model (Qwen2.5 0.5B) on it; -- guardrails on every request, with a judge model (Llama Guard 3 1B) on a - CPU, and API keys that expire within 90 days. What is hardened, and how +- guardrails on every request, with a judge model (Llama Guard 3 1B, two + replicas) on CPUs — a request it cannot check is refused, not let through — and API keys that expire within 90 days. What is hardened, and how to loosen it: [install.md](install.md#2-the-control-plane). Run again later, `sudo llmgw` is the console: the cluster's state and diff --git a/internal/cli/server.go b/internal/cli/server.go index db7a35a..aa783ce 100644 --- a/internal/cli/server.go +++ b/internal/cli/server.go @@ -160,6 +160,14 @@ func newServerCmd() *cobra.Command { return err } rt.SetNotifier(notifier) + if gate != nil && notifier != nil { + gate.SetJudgeAlarm(func(model string, err error) { + notifier.Notify(notify.JudgeUnavailable, model, + fmt.Sprintf("the guardrails judge %s gives no verdict (%v): policies with fail: closed "+ + "refuse what it should read, fail: open serve it unjudged", model, err), + map[string]any{"model": model, "error": err.Error()}) + }) + } if notifier != nil { log.Info("notifications on", "webhooks", len(cfg.Notify.Webhooks)) } diff --git a/internal/cli/serverinit.go b/internal/cli/serverinit.go index 6ddc667..03e6f5b 100644 --- a/internal/cli/serverinit.go +++ b/internal/cli/serverinit.go @@ -285,9 +285,10 @@ func guardSection(judge bool) string { ` if judge { j = ` # A judge model reads what patterns cannot: violence, weapons, abuse, - # self-harm. It runs on a CPU, about half a second for a short text; - # an answer is read every 600 characters as it streams. Slower than - # latency_budget, the request is served and recorded as flagged. + # self-harm. Two replicas run on CPUs with AVX2, about half a second + # for a short text; an answer is read every 600 characters as it + # streams. With fail: closed, a request the judge cannot read within + # latency_budget is refused (503); open serves it, flagged, unjudged. # Defamation (S5), specialized advice (S6), intellectual property (S8) # and elections (S13) are left out: they refuse too much ordinary work. judge: @@ -305,7 +306,7 @@ guard: default: [baseline] policies: baseline: - fail: open + fail: closed input: pii: {secret: block, card: redact} injection: flag diff --git a/internal/cli/serverinit_cluster.go b/internal/cli/serverinit_cluster.go index 058038f..9b162b8 100644 --- a/internal/cli/serverinit_cluster.go +++ b/internal/cli/serverinit_cluster.go @@ -458,7 +458,9 @@ func guardModel(source string) catalog.Model { Source: source, Role: catalog.RoleGuard, Guard: &catalog.GuardModel{Format: "llamaguard3", Categories: llamaGuard3Categories}, - Scaling: catalog.Scaling{Min: 1, Max: 1}, + // Two: every judged request waits on it, and with fail: closed a + // judge that is not there refuses everything. + Scaling: catalog.Scaling{Min: 2, Max: 2}, Profiles: []catalog.Profile{{ Name: "cpu", Engine: "llamacpp", Priority: 100, MinMemoryGB: 2, Args: []string{"--ctx-size", "8192"}, diff --git a/internal/cli/serverinit_cluster_test.go b/internal/cli/serverinit_cluster_test.go index f04a522..837a4fe 100644 --- a/internal/cli/serverinit_cluster_test.go +++ b/internal/cli/serverinit_cluster_test.go @@ -144,6 +144,9 @@ func TestTheInstalledConfigurationIsHardenedAndLoads(t *testing.T) { t.Errorf("judge=%v: guard.default %v", judge, cfg.Guard.Default) } p := cfg.Guard.Policies["baseline"] + if p.Fail != "closed" { + t.Errorf("judge=%v: the baseline fails %q", judge, p.Fail) + } if (p.Judge != nil) != judge { t.Errorf("judge=%v: the policy's judge is %+v", judge, p.Judge) } @@ -164,7 +167,7 @@ func TestTheGuardModelIsAValidJudge(t *testing.T) { t.Fatalf("not a valid catalog entry: %v", err) } got, _ := c.Resolve(guardModelName) - if !got.IsGuard() || got.Guard.Format != "llamaguard3" || len(got.Guard.Categories) != 13 { + if !got.IsGuard() || got.Guard.Format != "llamaguard3" || len(got.Guard.Categories) != 13 || got.Scaling.Min != 2 { t.Errorf("%+v", got) } for _, p := range got.Profiles { diff --git a/internal/guard/gate.go b/internal/guard/gate.go index 20de9fc..217c44a 100644 --- a/internal/guard/gate.go +++ b/internal/guard/gate.go @@ -40,6 +40,8 @@ type Gate struct { tools toolMemo judger *judge m *gateMetrics + // alarm is told each time a judge gives no verdict; nil tells no one. + alarm func(model string, err error) mu sync.Mutex attached map[string][]string // "scope/id" → policy names @@ -363,6 +365,7 @@ func (g *Gate) Input(ctx context.Context, s Subject, body []byte) (Verdict, erro res, err := g.judger.ask(ctx, jp, user, "") if err != nil { g.log.Warn("the judge gave no verdict on a request", "judge", jp.Model, "err", err) + g.judgeFailed(jp.Model, err) } if err != nil && p.Fail == "closed" && !v.Shadow { v.Took = g.now().Sub(start) @@ -402,6 +405,16 @@ func (g *Gate) Input(ctx context.Context, s Subject, body []byte) (Verdict, erro // judgeFinding is what a judge's answer, or its failing to give one, comes // to under a policy. +// SetJudgeAlarm says whom to tell when a judge gives no verdict: set before +// the gate serves. +func (g *Gate) SetJudgeAlarm(f func(model string, err error)) { g.alarm = f } + +func (g *Gate) judgeFailed(model string, err error) { + if g.alarm != nil { + g.alarm(model, err) + } +} + func judgeFinding(p Policy, action string, res judgment, err error) finding { if err != nil { // Unanswered: closed stops it, open lets it through, said. diff --git a/internal/guard/judge_test.go b/internal/guard/judge_test.go index 6b36526..81f0baa 100644 --- a/internal/guard/judge_test.go +++ b/internal/guard/judge_test.go @@ -105,9 +105,14 @@ func TestAJudgeThatDoesNotAnswerFailsAsThePolicySays(t *testing.T) { t.Errorf("fail open: %+v %v after %s", v, err, time.Since(start)) } closed := judged(t, Policy{Fail: "closed", Judge: &JudgePolicy{Model: "guard", Input: "block"}}, &fakeJudge{err: errors.New("down")}) + var alarmed []string + closed.SetJudgeAlarm(func(model string, _ error) { alarmed = append(alarmed, model) }) if _, err := closed.Input(ctx, Subject{}, chat("user", "hi")); err == nil { t.Error("fail closed served unjudged") } + if len(alarmed) != 1 || alarmed[0] != "guard" { + t.Errorf("alarm told %v", alarmed) + } // A judge that is not a guard model is a judge that cannot be asked. bad := judged(t, Policy{Fail: "closed", Judge: &JudgePolicy{Model: "chat-model", Input: "block"}}, &fakeJudge{}) if _, err := bad.Input(ctx, Subject{}, chat("user", "hi")); err == nil { diff --git a/internal/guard/output.go b/internal/guard/output.go index 241fb9f..b528d70 100644 --- a/internal/guard/output.go +++ b/internal/guard/output.go @@ -194,6 +194,7 @@ func (f *outFilter) askJudge(text string, final bool) finding { f.took += time.Since(start) if err != nil { f.g.log.Warn("the judge gave no verdict on an answer", "judge", jp.Model, "err", err) + f.g.judgeFailed(jp.Model, err) } jf := judgeFinding(f.p, jp.Output, res, err) if jf.failOpen { diff --git a/internal/notify/notify.go b/internal/notify/notify.go index 73ccf37..8be77a1 100644 --- a/internal/notify/notify.go +++ b/internal/notify/notify.go @@ -1,6 +1,7 @@ // Package notify tells the people running the gateway about what needs them, // without a Prometheus in between: a model parked after failing every start, -// a node lost, a budget spent, a model's repository with new weights. Each +// a node lost, a budget spent, a model's repository with new weights, a +// guardrails judge not answering. Each // event goes to the webhooks configured for it — JSON signed with a shared // secret, or a Slack-compatible message. // @@ -34,10 +35,13 @@ const ( NodeLost = "node_lost" BudgetSpent = "budget_spent" UpdateAvailable = "update_available" + // JudgeUnavailable is a guardrails judge giving no verdict: with + // fail: closed, requests are being refused; with open, served unjudged. + JudgeUnavailable = "judge_unavailable" ) // Kinds are every event there is. -var Kinds = []string{ModelParked, NodeLost, BudgetSpent, UpdateAvailable} +var Kinds = []string{ModelParked, NodeLost, BudgetSpent, UpdateAvailable, JudgeUnavailable} // Event is one thing that happened. type Event struct {