Guardrails turn off and on; a judge's refusal says which hazard; ctl guard judge ask

guard.enabled (unset is on) makes every policy check nothing while false,
live: llmgw ctl guard off/on set it in the cluster's settings, and ctl
status, ctl guard list and the guardrails page say when they are off.

A refusal by a judge names the hazard it saw - judge: S6,Specialized
Advice - and the verdict is kept with the request in a new usage column,
guard_rule, shown by ctl request, the request log and the guardrails page.

llmgw ctl guard judge ask has a judge read a text, past the cache, and shows
its raw answer and how it is read; the endpoint is kept out of the audit so
no text is recorded.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-06 12:41:08 +02:00
1 parent 3f5a809373
commit 249a91aeec
34 files changed
+505 -33

No files matched your search

+19
View File
@@ -22,6 +22,25 @@ is only released once its **exit criterion** is met, not when its code lands.
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
APIs change between minors without a deprecation period.
## Unreleased
### Added
- **The guardrails turn off and on as a whole**: `llmgw ctl guard off` /
`on`, which is `guard.enabled` in the cluster's settings — in force within
seconds, the policies and their attachments kept. `ctl status`, `ctl guard
list` and the guardrails page say when they are off.
- **`llmgw ctl guard judge ask TEXT`** has the judge read a text and shows
its answer as given and as read — for finding out why ordinary requests
are refused. Nothing of the text is kept, not even in the audit.
### Changed
- **A judge's refusal names the hazard**: the client is told `(judge:
S6,Specialized Advice)` rather than `(judge)`, and the verdict is kept with
the request (`guard_rule`): `ctl request`, the server's log and the
guardrails page show it.
## v0.12.1 — 2026-10-06
The guardrails' judge, served and watched without server init: a judge
+101
View File
@@ -924,6 +924,48 @@
}
}
},
"/api/v1/guard/judges/ask": {
"post": {
"operationId": "postGuardJudgesAsk",
"summary": "Have a judge read a text and say what it answered; nothing is kept",
"tags": [
"guard"
],
"parameters": [
{
"$ref": "#/components/parameters/Actor"
}
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/JudgeAskRequest"
}
}
}
},
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/JudgeAskView"
}
}
}
},
"401": {
"$ref": "#/components/responses/Unauthorized"
},
"default": {
"$ref": "#/components/responses/Error"
}
}
}
},
"/api/v1/jobs": {
"get": {
"operationId": "getJobs",
@@ -4129,6 +4171,10 @@
"category": {
"type": "string"
},
"rule": {
"type": "string",
"description": "Rule is what decided it within the category: a pattern's name, or a judge's verdict, \"S6,Specialized Advice\"."
},
"guard_ms": {
"type": "integer",
"format": "int64"
@@ -4191,6 +4237,10 @@
"type": "object",
"description": "GuardView is the library and where its policies are attached.",
"properties": {
"enabled": {
"type": "boolean",
"description": "Enabled is whether the guardrails check anything (guard.enabled)."
},
"policies": {
"type": "array",
"items": {
@@ -4389,6 +4439,50 @@
}
}
},
"JudgeAskRequest": {
"type": "object",
"description": "JudgeAskRequest is a text to have a judge read.",
"properties": {
"model": {
"type": "string",
"description": "Model is the judge; empty is the one the policies ask, when they ask one."
},
"text": {
"type": "string"
},
"answer": {
"type": "string",
"description": "Answer, when given, is read as the assistant's reply to Text."
}
}
},
"JudgeAskView": {
"type": "object",
"description": "JudgeAskView is what the judge answered.",
"properties": {
"model": {
"type": "string"
},
"verdict": {
"type": "string",
"description": "Verdict is safe or unsafe, as the gateway reads the answer."
},
"categories": {
"type": "array",
"items": {
"type": "string"
}
},
"raw": {
"type": "string",
"description": "Raw is the judge's answer as it gave it."
},
"took_ms": {
"type": "integer",
"format": "int64"
}
}
},
"JudgeView": {
"type": "object",
"description": "JudgeView is one judge model the guardrail policies ask, and whether it can answer them.",
@@ -5401,6 +5495,9 @@
"guard_category": {
"type": "string"
},
"guard_rule": {
"type": "string"
},
"guard_ms": {
"type": "integer",
"format": "int64"
@@ -5779,6 +5876,10 @@
"$ref": "#/components/schemas/JudgeView"
},
"description": "Judges are the guardrails' judge models with something wrong."
},
"guard_off": {
"type": "boolean",
"description": "GuardOff says the guardrails are turned off (guard.enabled: false): no request is checked, whatever the policies say."
}
}
},
+33 -1
View File
@@ -15,6 +15,19 @@ checked and nothing costs anything. With one, every request is judged once it is
within its limits, with its stored conversation spliced in, and before it is
sent to a model; and the answer is judged before the client sees it.
## Turning them off and on
```
llmgw ctl guard off --note "why" # no request is checked, whatever the policies say
llmgw ctl guard on # every policy checks again, as it was
```
That is `guard.enabled` in the cluster's settings, in force on every control
plane within seconds without a restart. The policies and where they are
attached are kept. While off, `llmgw ctl status`, `llmgw ctl guard list` and
the guardrails page say so. To stop one policy rather than all, give it
`mode: off`, or `mode: shadow` to have it record what it would do.
## A policy
```yaml
@@ -117,7 +130,26 @@ llmgw ctl guard judge enable # Llama Guard 3 1B on CPUs, under the nam
llmgw ctl guard judge enable --replace # when a model of that name is there, but not as a judge
```
or **Do → Serve the guardrails' judge** in the console. A Llama Guard file
or **Do → Serve the guardrails' judge** in the console.
**When it refuses ordinary requests.** A refusal by a judge names the hazard
it saw — the client is told `refused by the gateway's policy (judge:
S6,Specialized Advice)` — and the same is kept with the request: `llmgw ctl
request <id>`, the server's log (`guard_rule`) and the guardrails page show
it. To see what the judge says of any text, without anything being kept:
```
llmgw ctl guard judge ask 'How do I sort a list in Python?'
# llama-guard-3-1b: safe, 140 ms
# it answered: "safe"
```
A judge that says `unsafe` to ordinary text is usually the wrong weights
(another repository's or quantisation's, without Llama Guard's template:
`judge enable --replace` serves the right ones), or a policy counting every
hazard: give its judge `categories`, leaving out S5, S6, S8 and S13 as the
baseline does. While finding out, `judge.input: flag` records instead of
refusing. A Llama Guard file
served from the mirror page is served as a judge too, under the name the
policies ask. By hand, the catalog entry is:
+8
View File
@@ -372,6 +372,7 @@ rather than repeated.
| `llamacpp` | `LlamaCppConfig` | [engines.Config](#enginesconfig) |
| `trtllm` | `TRTLLMConfig` | [engines.Config](#enginesconfig) |
| `sim` | `SimConfig` | [engines.Config](#enginesconfig) |
| `enabled` | `*bool` | [guard.Config](#guardconfig) |
| `default` | `[]string` | [guard.Config](#guardconfig) |
| `policies` | `map[string]Policy` | [guard.Config](#guardconfig) |
| `webhooks` | `[]Webhook` | [notify.Config](#notifyconfig) |
@@ -3067,6 +3068,13 @@ Takes a [SimConfig](#simconfig) block.
## guard.Config
### `enabled`
`*bool`
Enabled turns every policy on or off at once: false checks nothing,
whatever is defined or attached. Unset is on.
### `default`
`[]string`
+7 -1
View File
@@ -12,6 +12,7 @@ import (
"net/http"
"net/url"
"regexp"
"slices"
"sort"
"strings"
"time"
@@ -110,11 +111,16 @@ func (w *statusWriter) Flush() {
}
}
// notChanges are POSTs that change nothing and carry text that must not be
// kept: asking a judge about a sentence.
var notChanges = []string{"/api/v1/guard/judges/ask"}
// API records every change to the admin API: a request that is not a GET or
// HEAD, under /api/. actor says who made it.
func API(sink Sink, actor func(*http.Request) string, log *slog.Logger, next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method == http.MethodGet || r.Method == http.MethodHead || !strings.HasPrefix(r.URL.Path, "/api/") {
if r.Method == http.MethodGet || r.Method == http.MethodHead || !strings.HasPrefix(r.URL.Path, "/api/") ||
slices.Contains(notChanges, r.URL.Path) {
next.ServeHTTP(w, r)
return
}
+10
View File
@@ -72,3 +72,13 @@ func TestABodyLargerThanWhatIsRecordedReachesTheHandlerWhole(t *testing.T) {
t.Errorf("the handler was given %d of %d bytes", got, len(big))
}
}
func TestAskingAJudgeKeepsNoText(t *testing.T) {
sink := &memSink{}
h := API(sink, func(*http.Request) string { return "ilker" }, nil, http.HandlerFunc(func(http.ResponseWriter, *http.Request) {}))
h.ServeHTTP(httptest.NewRecorder(), httptest.NewRequest("POST", "/api/v1/guard/judges/ask",
strings.NewReader(`{"text":"something a user wrote"}`)))
if len(sink.got) != 0 {
t.Errorf("a judge's question was recorded: %+v", sink.got)
}
}
+5 -2
View File
@@ -76,8 +76,11 @@ func newRequestCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
fmt.Fprintf(out, "tags %s\n", strings.Join(kv, ", "))
}
if v.GuardAction != "" {
fmt.Fprintf(out, "guard %s by policy %s (%s), %d ms\n", v.GuardAction, v.GuardPolicy,
or(v.GuardCategory, "-"), v.GuardMs)
what := or(v.GuardCategory, "-")
if v.GuardRule != "" {
what += ": " + v.GuardRule
}
fmt.Fprintf(out, "guard %s by policy %s (%s), %d ms\n", v.GuardAction, v.GuardPolicy, what, v.GuardMs)
} else if v.GuardMs > 0 {
fmt.Fprintf(out, "guard allowed, %d ms\n", v.GuardMs)
}
+33
View File
@@ -3,11 +3,13 @@ package cli
import (
"encoding/json"
"fmt"
"strconv"
"strings"
"github.com/spf13/cobra"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
)
func newGuardCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
@@ -25,6 +27,34 @@ func newGuardCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
"with IDs), key:KEY-ID or user:USER-ID. An attachment applies at once.",
}
c.AddCommand(newGuardJudgeCmd(client, asJSON))
for _, sw := range []struct {
use, short string
on bool
}{
{"on", "Turn the guardrails on: every policy checks again", true},
{"off", "Turn the guardrails off: no request is checked, whatever the policies say", false},
} {
var note string
cmd := &cobra.Command{
Use: sw.use,
Short: sw.short,
Long: "guard.enabled in the cluster's settings: in force on every control plane\n" +
"within seconds, without a restart. The policies and where they are attached\n" +
"stay as they are, and come back as they were when turned on again. While\n" +
"off, llmgw ctl status says so.",
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error {
if note == "" {
note = "guardrails " + sw.use
}
return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
return doc.SetYAML([]string{"guard", "enabled"}, strconv.FormatBool(sw.on))
})
},
}
cmd.Flags().StringVar(&note, "note", "", "why, kept with the settings' version")
c.AddCommand(cmd)
}
c.AddCommand(&cobra.Command{
Use: "list",
Short: "The policies, and where each is attached",
@@ -38,6 +68,9 @@ func newGuardCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
return json.NewEncoder(cmd.OutOrStdout()).Encode(v)
}
out := cmd.OutOrStdout()
if !v.Enabled {
fmt.Fprintln(out, "The guardrails are OFF: no request is checked (llmgw ctl guard on).")
}
fmt.Fprintf(out, "Policies: %s\n", strings.Join(v.Policies, ", "))
def := strings.Join(v.Default, ", ")
if def == "" {
+37 -1
View File
@@ -29,7 +29,7 @@ func newGuardJudgeCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
"guard), with a replica ready; otherwise what the policy judges is refused\n" +
"(fail: closed, guard_unavailable) or served unjudged (fail: open).",
}
c.AddCommand(newGuardJudgeStatusCmd(client, asJSON), newGuardJudgeEnableCmd(client))
c.AddCommand(newGuardJudgeStatusCmd(client, asJSON), newGuardJudgeEnableCmd(client), newGuardJudgeAskCmd(client, asJSON))
return c
}
@@ -187,3 +187,39 @@ func newGuardJudgeEnableCmd(client func() *ctlClient) *cobra.Command {
f.BoolVar(&detach, "detach", false, "start it and return; llmgw ctl jobs follows it")
return c
}
func newGuardJudgeAskCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
var model, answer string
c := &cobra.Command{
Use: "ask TEXT",
Short: "Have a judge read a text and say what it answered",
Long: "For finding out why a request was refused: the judge reads TEXT as a user's\n" +
"message (with --answer, ANSWER as the reply to it), past the cache, and its\n" +
"answer is shown as it gave it and as the gateway reads it — \"unsafe\" and\n" +
"the hazards it named, S6 Specialized Advice and the like. Nothing is kept:\n" +
"not in the audit, the usage or the logs.",
Example: " llmgw ctl guard judge ask 'How do I sort a list in Python?'\n" +
" llmgw ctl guard judge ask 'Write a poem' --answer 'Roses are red'",
Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
body, _ := json.Marshal(adminapi.JudgeAskRequest{Model: model, Text: args[0], Answer: answer})
var v adminapi.JudgeAskView
if err := client().patient(45*time.Second).post(cmd.Context(), "/api/v1/guard/judges/ask", body, &v); err != nil {
return err
}
if *asJSON {
return emitJSON(v)
}
out := cmd.OutOrStdout()
fmt.Fprintf(out, "%s: %s", v.Model, v.Verdict)
if len(v.Categories) > 0 {
fmt.Fprintf(out, " (%s)", strings.Join(v.Categories, ", "))
}
fmt.Fprintf(out, ", %d ms\nit answered: %q\n", v.TookMs, v.Raw)
return nil
},
}
c.Flags().StringVar(&model, "model", "", "the judge (default: the one the policies ask)")
c.Flags().StringVar(&answer, "answer", "", "an assistant's reply to TEXT, judged as output is")
return c
}
+1 -1
View File
@@ -362,7 +362,7 @@ func newServerCmd() *cobra.Command {
api.SetEval(evals)
if st := rt.Store(); st != nil && gate != nil {
api.SetGuard(&adminapi.GuardAdmin{Library: gate.Library, Store: st, Invalidate: gate.Invalidate,
JudgeErrors: gate.JudgeErrors})
JudgeErrors: gate.JudgeErrors, Ask: gate.AskJudge})
}
if st := rt.Store(); st != nil {
// Removing a node revokes its certificate.
+3
View File
@@ -153,6 +153,9 @@ func printSummary(cmd *cobra.Command, s adminapi.Summary) {
fmt.Fprintf(out, "STORE %s\n", strings.Join(parts, ", "))
}
if s.GuardOff {
fmt.Fprintln(out, "GUARD off: no request is checked, whatever the policies say (llmgw ctl guard on)")
}
if len(s.Judges) > 0 {
fmt.Fprintln(out, "\nGUARDRAILS' JUDGE")
for _, j := range s.Judges {
+2 -1
View File
@@ -1545,6 +1545,7 @@ type RequestView struct {
GuardAction string `json:"guard_action,omitempty"`
GuardPolicy string `json:"guard_policy,omitempty"`
GuardCategory string `json:"guard_category,omitempty"`
GuardRule string `json:"guard_rule,omitempty"`
GuardMs int64 `json:"guard_ms,omitempty"`
// What the client said about it: OpenAI's user field, and its tags.
EndUser string `json:"end_user,omitempty"`
@@ -1573,7 +1574,7 @@ func (a *API) request(w http.ResponseWriter, r *http.Request) {
Model: u.Model, ServedModel: u.ServedModel, Replica: u.Replica, Status: u.Status,
LatencyMs: u.LatencyMs, QueueMs: u.QueueMs, TTFTMs: u.TTFTMs,
PromptTokens: u.PromptTokens, CompletionTokens: u.CompletionTokens, CostMicros: u.CostMicros,
GuardAction: u.GuardAction, GuardPolicy: u.GuardPolicy, GuardCategory: u.GuardCategory, GuardMs: u.GuardMs,
GuardAction: u.GuardAction, GuardPolicy: u.GuardPolicy, GuardCategory: u.GuardCategory, GuardRule: u.GuardRule, GuardMs: u.GuardMs,
EndUser: u.EndUser, Tags: u.Tags, CacheHit: u.CacheHit,
})
}
+11 -3
View File
@@ -30,6 +30,8 @@ type GuardAdmin struct {
Invalidate func()
// JudgeErrors are each judge's last failure to give a verdict.
JudgeErrors func() map[string]guard.JudgeError
// Ask has a judge read a text, past the cache.
Ask func(ctx context.Context, model, text, answer string) (guard.JudgeAnswer, error)
}
// SetGuard enables the guard endpoints.
@@ -37,6 +39,8 @@ func (a *API) SetGuard(g *GuardAdmin) { a.guard = g }
// GuardView is the library and where its policies are attached.
type GuardView struct {
// Enabled is whether the guardrails check anything (guard.enabled).
Enabled bool `json:"enabled"`
Policies []string `json:"policies"`
Default []string `json:"default"`
Attachments []GuardAttachmentView `json:"attachments"`
@@ -91,7 +95,10 @@ type GuardEventView struct {
Action string `json:"action"`
Policy string `json:"policy"`
Category string `json:"category,omitempty"`
GuardMs int64 `json:"guard_ms"`
// Rule is what decided it within the category: a pattern's name, or a
// judge's verdict, "S6,Specialized Advice".
Rule string `json:"rule,omitempty"`
GuardMs int64 `json:"guard_ms"`
}
// policyView sums one policy up.
@@ -201,6 +208,7 @@ type GuardAttachmentRequest struct {
func (a *API) registerGuard(mux Mux) {
mux.HandleFunc("GET /api/v1/guard/judges", a.listJudges)
mux.HandleFunc("POST /api/v1/guard/judges/ask", a.askJudge)
mux.HandleFunc("GET /api/v1/guard", func(w http.ResponseWriter, r *http.Request) {
if a.guard == nil {
writeError(w, http.StatusNotImplemented, "no guard policies: define them under guard in the cluster's settings (llmgw ctl config edit)")
@@ -211,7 +219,7 @@ func (a *API) registerGuard(mux Mux) {
writeError(w, http.StatusInternalServerError, err.Error())
return
}
v := GuardView{Default: a.guard.Library().Default, Attachments: []GuardAttachmentView{}}
v := GuardView{Enabled: a.guard.Library().On(), Default: a.guard.Library().Default, Attachments: []GuardAttachmentView{}}
var names map[string]string
if a.quotas != nil {
names, _ = a.scopeNames(r.Context())
@@ -272,7 +280,7 @@ func (a *API) registerGuard(mux Mux) {
slices.SortStableFunc(v.Counts, func(a, b GuardCountView) int { return int(b.Count - a.Count) })
for _, e := range events {
v.Events = append(v.Events, GuardEventView{At: e.At, RequestID: e.RequestID, Project: project(e.ProjectID),
Model: e.Model, Status: e.Status, Action: e.Action, Policy: e.Policy, Category: e.Category, GuardMs: e.GuardMs})
Model: e.Model, Status: e.Status, Action: e.Action, Policy: e.Policy, Category: e.Category, Rule: e.Rule, GuardMs: e.GuardMs})
}
writeJSON(w, http.StatusOK, v)
})
+57
View File
@@ -1,10 +1,12 @@
package adminapi
import (
"encoding/json"
"fmt"
"net/http"
"slices"
"sort"
"strings"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
@@ -44,6 +46,10 @@ func (a *API) Judges() []JudgeView {
return nil
}
lib := a.guard.Library()
if !lib.On() {
// Off: no judge is asked, so none is missing.
return nil
}
byModel := map[string]*JudgeView{}
for name, p := range lib.Policies {
if p.Judge == nil || p.Judge.Model == "" {
@@ -122,3 +128,54 @@ func (a *API) listJudges(w http.ResponseWriter, _ *http.Request) {
}
writeJSON(w, http.StatusOK, a.Judges())
}
// JudgeAskRequest is a text to have a judge read.
type JudgeAskRequest struct {
// Model is the judge; empty is the one the policies ask, when they ask one.
Model string `json:"model,omitempty"`
Text string `json:"text"`
// Answer, when given, is read as the assistant's reply to Text.
Answer string `json:"answer,omitempty"`
}
// JudgeAskView is what the judge answered.
type JudgeAskView struct {
Model string `json:"model"`
// Verdict is safe or unsafe, as the gateway reads the answer.
Verdict string `json:"verdict"`
Categories []string `json:"categories,omitempty"`
// Raw is the judge's answer as it gave it.
Raw string `json:"raw"`
TookMs int64 `json:"took_ms"`
}
func (a *API) askJudge(w http.ResponseWriter, r *http.Request) {
if a.guard == nil || a.guard.Ask == nil {
writeError(w, http.StatusNotImplemented, "no guard policies: define them under guard in the cluster's settings (llmgw ctl config edit)")
return
}
var req JudgeAskRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil || strings.TrimSpace(req.Text) == "" {
writeError(w, http.StatusBadRequest, "give the text for the judge to read")
return
}
if req.Model == "" {
judges := a.Judges()
if len(judges) == 0 {
writeError(w, http.StatusBadRequest, "no policy asks a judge: give the model")
return
}
req.Model = judges[0].Model
}
ans, err := a.guard.Ask(r.Context(), req.Model, req.Text, req.Answer)
v := JudgeAskView{Model: req.Model, Raw: ans.Raw, Categories: ans.Categories, TookMs: ans.Took.Milliseconds()}
if err != nil {
writeError(w, http.StatusBadGateway, fmt.Sprintf("the judge %s gave no verdict: %v (it answered %q)", req.Model, err, ans.Raw))
return
}
v.Verdict = "safe"
if ans.Unsafe {
v.Verdict = "unsafe"
}
writeJSON(w, http.StatusOK, v)
}
+25
View File
@@ -1,6 +1,7 @@
package adminapi
import (
"context"
"strings"
"testing"
"time"
@@ -64,3 +65,27 @@ func TestEachJudgeSaysWhatIsWrongWithIt(t *testing.T) {
t.Errorf("an old failure still counts: %q", p)
}
}
func TestAJudgeIsAskedAndItsWordsShown(t *testing.T) {
a, _ := setup(t, testModels())
lib := guard.Config{Policies: map[string]guard.Policy{
"baseline": {Judge: &guard.JudgePolicy{Model: "llama-guard-3-1b", Input: "block"}}}}
var asked string
a.SetGuard(&GuardAdmin{Library: func() guard.Config { return lib },
Ask: func(_ context.Context, model, text, _ string) (guard.JudgeAnswer, error) {
asked = model + "|" + text
return guard.JudgeAnswer{Model: model, Raw: "unsafe\nS6", Unsafe: true,
Categories: []string{"S6", "Specialized Advice"}}, nil
}})
var v JudgeAskView
if code := call(t, a, "POST", "/api/v1/guard/judges/ask", `{"text":"how do I sort a list"}`, &v); code != 200 {
t.Fatalf("%d", code)
}
if asked != "llama-guard-3-1b|how do I sort a list" || v.Verdict != "unsafe" || v.Raw != "unsafe\nS6" ||
len(v.Categories) != 2 {
t.Errorf("asked %q, got %+v", asked, v)
}
if code := call(t, a, "POST", "/api/v1/guard/judges/ask", `{"text":" "}`, nil); code != 400 {
t.Errorf("no text: %d", code)
}
}
+2
View File
@@ -256,6 +256,8 @@ var Routes = []Route{
Response: GuardEventsView{}},
{Method: "GET", Path: "/api/v1/guard/judges", Summary: "The judge models the policies ask, and what is wrong with each",
Response: []JudgeView{}},
{Method: "POST", Path: "/api/v1/guard/judges/ask", Summary: "Have a judge read a text and say what it answered; nothing is kept",
Request: JudgeAskRequest{}, Response: JudgeAskView{}},
{Method: "POST", Path: "/api/v1/guard/attachments", Summary: "Attach a guardrail policy",
Request: GuardAttachmentRequest{}, Response: guard.Attachment{}},
{Method: "DELETE", Path: "/api/v1/guard/attachments", Summary: "Detach a guardrail policy",
+6
View File
@@ -44,6 +44,9 @@ type Summary struct {
ControlPlanes ControlPlanes `json:"control_planes,omitzero"`
// Judges are the guardrails' judge models with something wrong.
Judges []JudgeView `json:"judges,omitempty"`
// GuardOff says the guardrails are turned off (guard.enabled: false):
// no request is checked, whatever the policies say.
GuardOff bool `json:"guard_off,omitempty"`
}
// StoreMember is one member of a replicated store, as its Patroni reports it.
@@ -151,6 +154,9 @@ const idleThreshold = 10 * time.Minute
// Summary builds the fleet summary.
func (a *API) Summary() Summary {
s := Summary{Version: version.Version}
if a.guard != nil && a.guard.Library != nil && !a.guard.Library().On() {
s.GuardOff = true
}
for _, j := range a.Judges() {
if j.Problem != "" {
s.Judges = append(s.Judges, j)
+1
View File
@@ -484,6 +484,7 @@ var Reference = []Block{
{Key: "sim", Type: "SimConfig", Doc: "", Nested: "engines.SimConfig"},
}},
{Package: "guard", Name: "Config", Title: "guard.Config", Doc: "", Fields: []Field{
{Key: "enabled", Type: "*bool", Doc: "Enabled turns every policy on or off at once: false checks nothing,\nwhatever is defined or attached. Unset is on.", Nested: ""},
{Key: "default", Type: "[]string", Doc: "", Nested: ""},
{Key: "policies", Type: "map[string]Policy", Doc: "", Nested: ""},
}},
+16 -1
View File
@@ -156,6 +156,10 @@ func (g *Gate) Invalidate() { g.reload(context.Background()) }
// from.
func (g *Gate) Policy(s Subject) (Policy, string) {
g.mu.Lock()
if !g.cfg.On() {
g.mu.Unlock()
return Policy{}, ""
}
if !g.loading && g.store != nil && g.now().Sub(g.loadedAt) > attachmentsTTL {
g.loading = true
go g.reload(context.WithoutCancel(context.Background()))
@@ -328,7 +332,7 @@ func (g *Gate) Input(ctx context.Context, s Subject, body []byte) (Verdict, erro
done := func(f finding) Verdict {
v.Action, v.Category, v.Rule = f.action, f.category, f.rule
if f.action == Block {
v.Message = fmt.Sprintf("this request was refused by the gateway's policy (%s)", f.category)
v.Message = fmt.Sprintf("this request was refused by the gateway's policy (%s)", refusedFor(f))
v.Body = nil
}
v.Took = g.now().Sub(start)
@@ -524,3 +528,14 @@ func toolAllowed(t Tools, name string) bool {
}
return false
}
// refusedFor is what a refusal names: the category, and for a judge the
// hazard it saw — "judge: S6,Specialized Advice" — so whoever is refused
// can say why, and a judge refusing ordinary questions can be told apart.
// Never the text.
func refusedFor(f finding) string {
if f.category == "judge" && f.rule != "" && f.rule != "unavailable" {
return "judge: " + f.rule
}
return f.category
}
+31
View File
@@ -278,3 +278,34 @@ func TestCapabilitiesAndStructuredAnswersWhenAskedFor(t *testing.T) {
}
}
}
func TestTheGuardrailsTurnOffAndOnAsAWhole(t *testing.T) {
off := false
cfg := lib(map[string]Policy{"base": {Input: Rules{PII: map[string]string{"secret": "block"}}}}, "base")
cfg.Enabled = &off
g := gate(t, cfg)
secret := chat("user", "my key is sk-proj-abcdefghijklmnopqrstuvwx1234")
if v, err := g.Input(context.Background(), Subject{}, secret); err != nil || v.Action != "" {
t.Errorf("off, a secret was acted on: %+v %v", v, err)
}
on := true
cfg.Enabled = &on
if err := g.Replace(cfg); err != nil {
t.Fatal(err)
}
if v, _ := g.Input(context.Background(), Subject{}, secret); v.Action != Block {
t.Errorf("on again, a secret went through: %+v", v)
}
if !(Config{}).On() {
t.Error("unset is not on")
}
}
func TestAJudgesRefusalNamesTheHazard(t *testing.T) {
if got := refusedFor(finding{category: "judge", rule: "S6,Specialized Advice"}); got != "judge: S6,Specialized Advice" {
t.Errorf("%q", got)
}
if got := refusedFor(finding{category: "secret", rule: "pii:secret"}); got != "secret" {
t.Errorf("%q", got)
}
}
+50
View File
@@ -369,3 +369,53 @@ func DefaultCategories(format string) []string {
}
return nil
}
// JudgeAnswer is what a judge said of a text: its words as it gave them,
// and as the gateway reads them.
type JudgeAnswer struct {
Model string
Raw string
Unsafe bool
Categories []string
Took time.Duration
}
// AskJudge has a judge read a text, past the cache, and says what it
// answered: for an operator finding out why ordinary requests are refused.
// With answer, the judge reads it as the assistant's reply to text.
func (g *Gate) AskJudge(ctx context.Context, model, text, answer string) (JudgeAnswer, error) {
out := JudgeAnswer{Model: model}
if g.judger == nil {
return out, errNoJudge
}
format, names, ok := g.judger.format(model)
if !ok {
return out, fmt.Errorf("%w: %q is not a model with role: guard", errNoJudge, model)
}
ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
start := time.Now()
path, body := judgeRequest(model, format, names, text, answer)
reply, err := g.judger.call(ctx, model, path, body)
out.Took = time.Since(start)
if err != nil {
return out, err
}
var resp struct {
Choices []struct {
Message struct {
Content string `json:"content"`
} `json:"message"`
Text string `json:"text"`
} `json:"choices"`
}
if json.Unmarshal(reply, &resp) == nil && len(resp.Choices) > 0 {
out.Raw = strings.TrimSpace(resp.Choices[0].Message.Content + resp.Choices[0].Text)
}
res, err := parseJudgment(format, names, reply, answer)
if err != nil {
return out, err
}
out.Unsafe, out.Categories = res.unsafe, res.categories
return out, nil
}
+6
View File
@@ -25,10 +25,16 @@ import (
// Policies are attached to parts of the tenancy tree in the store; Default
// applies to every request.
type Config struct {
// Enabled turns every policy on or off at once: false checks nothing,
// whatever is defined or attached. Unset is on.
Enabled *bool `koanf:"enabled"`
Default []string `koanf:"default"`
Policies map[string]Policy `koanf:"policies"`
}
// On is whether the guardrails check anything.
func (c Config) On() bool { return c.Enabled == nil || *c.Enabled }
// Policy is one named set of rules.
type Policy struct {
// Mode: enforce acts, shadow only records what it would have done, off
+1 -1
View File
@@ -191,7 +191,7 @@ func guardFields(u *store.Usage, in, out guard.Verdict) {
if out.Stricter(in) {
v = out
}
u.GuardAction, u.GuardPolicy, u.GuardCategory = string(v.Action), v.Policy, v.Category
u.GuardAction, u.GuardPolicy, u.GuardCategory, u.GuardRule = string(v.Action), v.Policy, v.Category, v.Rule
if v.Shadow && v.Action != guard.Allow {
u.GuardAction = "shadow-" + u.GuardAction
}
+2 -1
View File
@@ -34,7 +34,7 @@ func (g *wordGuard) Input(_ context.Context, s guard.Subject, body []byte) (guar
case g.err != nil:
return guard.Verdict{}, g.err
case bytes.Contains(body, []byte("forbidden")):
return guard.Verdict{Action: guard.Block, Policy: "words", Category: "test"}, nil
return guard.Verdict{Action: guard.Block, Policy: "words", Category: "test", Rule: "S6,Specialized Advice"}, nil
case bytes.Contains(body, []byte("alice@example.com")):
return guard.Verdict{Action: guard.Redact, Policy: "pii", Category: "email",
Body: bytes.ReplaceAll(body, []byte("alice@example.com"), []byte("[EMAIL_1]"))}, nil
@@ -157,6 +157,7 @@ func TestABlockedRequestIsRecordedWithItsVerdictAndCostsNothing(t *testing.T) {
}
if ok {
if u.GuardAction != "blocked" || u.GuardPolicy != "words" || u.GuardCategory != "test" ||
u.GuardRule != "S6,Specialized Advice" ||
u.CostMicros != 0 || u.PromptTokens != 0 || u.Status != http.StatusBadRequest {
t.Errorf("recorded %+v", u)
}
+4 -2
View File
@@ -1217,9 +1217,11 @@ func (r *Router) accessLog(req *parsedRequest, res dispatchResult, status int, t
}
// What a guard did, by name: never what it matched.
if v := req.guardIn; res.guard.Stricter(v) {
attrs = append(attrs, "guard", string(res.guard.Action), "guard_stage", "output", "guard_policy", res.guard.Policy)
attrs = append(attrs, "guard", string(res.guard.Action), "guard_stage", "output", "guard_policy", res.guard.Policy,
"guard_category", res.guard.Category, "guard_rule", res.guard.Rule)
} else if v.Action != "" {
attrs = append(attrs, "guard", string(v.Action), "guard_stage", "input", "guard_policy", v.Policy)
attrs = append(attrs, "guard", string(v.Action), "guard_stage", "input", "guard_policy", v.Policy,
"guard_category", v.Category, "guard_rule", v.Rule)
}
if res.outcome.Stopped {
attrs = append(attrs, "stopped_by_guard", true)
+3 -2
View File
@@ -50,6 +50,7 @@ type GuardEvent struct {
Action string
Policy string
Category string
Rule string
GuardMs int64
}
@@ -57,14 +58,14 @@ type GuardEvent struct {
// flagged, redacted, blocked, or would have in shadow — since a moment.
func (s *DB) GuardEvents(ctx context.Context, since time.Time, limit int) ([]GuardEvent, error) {
return queryAll(ctx, s.db, `SELECT ts, request_id, key_id, COALESCE(user_id, ''), project_id, model, status,
guard_action, guard_policy, guard_category, guard_ms
guard_action, guard_policy, guard_category, guard_rule, guard_ms
FROM usage WHERE ts >= ? AND guard_action <> '' ORDER BY ts DESC LIMIT ?`,
[]any{millis(since), limit},
func(rows *sql.Rows) (GuardEvent, error) {
var e GuardEvent
var ts int64
err := rows.Scan(&ts, &e.RequestID, &e.KeyID, &e.UserID, &e.ProjectID, &e.Model, &e.Status,
&e.Action, &e.Policy, &e.Category, &e.GuardMs)
&e.Action, &e.Policy, &e.Category, &e.Rule, &e.GuardMs)
e.At = fromMillis(ts)
return e, err
})
@@ -0,0 +1,2 @@
-- Ported from the SQLite migration of the same number.
ALTER TABLE usage ADD COLUMN guard_rule TEXT NOT NULL DEFAULT '';
@@ -0,0 +1,5 @@
-- What in a guard policy decided a request, beside its category: the
-- pattern's name, or the judge's verdict — "S6,Specialized Advice" — so a
-- request a judge refused says which hazard it saw, and a judge that
-- refuses ordinary questions can be told from one that is right to.
ALTER TABLE usage ADD COLUMN guard_rule TEXT NOT NULL DEFAULT '';
+5 -5
View File
@@ -435,8 +435,8 @@ func (s *DB) WriteUsage(ctx context.Context, records []Usage) error {
`INSERT INTO usage (ts, key_id, project_id, model, served_model,
prompt_tokens, completion_tokens, cost_micros, status, latency_ms, user_id,
request_id, replica, queue_ms, ttft_ms,
guard_action, guard_policy, guard_category, guard_ms, end_user, tags, cache_hit)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`)
guard_action, guard_policy, guard_category, guard_rule, guard_ms, end_user, tags, cache_hit)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`)
if err != nil {
return err
}
@@ -450,7 +450,7 @@ func (s *DB) WriteUsage(ctx context.Context, records []Usage) error {
r.Model, r.ServedModel, r.PromptTokens, r.CompletionTokens,
r.CostMicros, r.Status, r.LatencyMs, r.UserID,
r.RequestID, r.Replica, r.QueueMs, r.TTFTMs,
r.GuardAction, r.GuardPolicy, r.GuardCategory, r.GuardMs, r.EndUser, tagsJSON(r.Tags), hit); err != nil {
r.GuardAction, r.GuardPolicy, r.GuardCategory, r.GuardRule, r.GuardMs, r.EndUser, tagsJSON(r.Tags), hit); err != nil {
return err
}
}
@@ -864,11 +864,11 @@ func (s *DB) UsageByRequest(ctx context.Context, requestID string) (Usage, bool,
err := s.db.QueryRowContext(ctx,
`SELECT ts, key_id, project_id, COALESCE(user_id, ''), model, served_model, prompt_tokens,
completion_tokens, cost_micros, status, latency_ms, request_id, replica, queue_ms, ttft_ms,
guard_action, guard_policy, guard_category, guard_ms, end_user, tags, cache_hit
guard_action, guard_policy, guard_category, guard_rule, guard_ms, end_user, tags, cache_hit
FROM usage WHERE request_id = ? ORDER BY ts DESC LIMIT 1`, requestID).
Scan(&ts, &u.KeyID, &u.ProjectID, &u.UserID, &u.Model, &u.ServedModel, &u.PromptTokens,
&u.CompletionTokens, &u.CostMicros, &u.Status, &u.LatencyMs, &u.RequestID, &u.Replica,
&u.QueueMs, &u.TTFTMs, &u.GuardAction, &u.GuardPolicy, &u.GuardCategory, &u.GuardMs,
&u.QueueMs, &u.TTFTMs, &u.GuardAction, &u.GuardPolicy, &u.GuardCategory, &u.GuardRule, &u.GuardMs,
&u.EndUser, &tags, &hit)
if errors.Is(err, sql.ErrNoRows) {
return Usage{}, false, nil
+3 -1
View File
@@ -257,11 +257,13 @@ type Usage struct {
// GuardAction is what a guard did — flagged, redacted, blocked — empty
// when it let the request through untouched or there is none;
// GuardPolicy and GuardCategory what decided it, GuardMs how long
// GuardPolicy, GuardCategory and GuardRule what decided it — a judge's
// rule is its verdict, "S6,Specialized Advice" — GuardMs how long
// judging took. Never the text.
GuardAction string
GuardPolicy string
GuardCategory string
GuardRule string
GuardMs int64
// EndUser is OpenAI's user field as the client sent it: its own name
+10 -7
View File
@@ -24,8 +24,10 @@ type guardPageData struct {
Window string
Windows []string
// Policies are the library, Default those applied to every request.
Policies []string
Default []string
Policies []string
Default []string
// Off is the guardrails turned off (guard.enabled: false).
Off bool
Attachments []guardAttachmentRow
Counts []guardCountRow
Events []guardEventRow
@@ -40,9 +42,9 @@ type guardCountRow struct {
}
type guardEventRow struct {
At, RequestID, Where, Model, Action, Policy, Category string
Status int
Ms int64
At, RequestID, Where, Model, Action, Policy, Category, Rule string
Status int
Ms int64
}
var guardWindows = map[string]time.Duration{"24h": 24 * time.Hour, "7d": 7 * 24 * time.Hour, "30d": 30 * 24 * time.Hour}
@@ -54,8 +56,9 @@ func (p *AccountPages) RegisterAdminGuard(mux *http.ServeMux, s GuardAdminSource
mux.HandleFunc("GET /admin/guard", p.adminOnly(func(w http.ResponseWriter, r *http.Request, admin store.User) {
ctx, l := r.Context(), langOf(r)
lib := library()
off := !lib.On()
d := guardPageData{Nav: p.navFor(r, admin, "guard"), Version: p.version, Window: r.URL.Query().Get("window"),
Windows: []string{"24h", "7d", "30d"}, Default: lib.Default}
Windows: []string{"24h", "7d", "30d"}, Default: lib.Default, Off: off}
since, ok := guardWindows[d.Window]
if !ok {
d.Window, since = "7d", guardWindows["7d"]
@@ -104,7 +107,7 @@ func (p *AccountPages) RegisterAdminGuard(mux *http.ServeMux, s GuardAdminSource
at := e.At.UTC()
d.Events = append(d.Events, guardEventRow{At: l.Date(at) + " " + at.Format("15:04:05"),
RequestID: e.RequestID, Where: name(e.ProjectID), Model: e.Model, Action: guardActionWord(l, e.Action),
Policy: e.Policy, Category: e.Category, Status: e.Status, Ms: e.GuardMs})
Policy: e.Policy, Category: e.Category, Rule: e.Rule, Status: e.Status, Ms: e.GuardMs})
}
noStore(w)
render(w, r, "admin_guard", d)
+2 -1
View File
@@ -32,5 +32,6 @@
"guard.col.request": "Request",
"guard.col.model": "Model",
"guard.col.status": "Status",
"guard.request_hint": "<code>llmgw ctl request ID</code> says what else a request went through."
"guard.request_hint": "<code>llmgw ctl request ID</code> says what else a request went through.",
"guard.off": "The guardrails are <strong>off</strong>: no request is checked, whatever the policies below say. <code>llmgw ctl guard on</code> turns them back on."
}
+2 -1
View File
@@ -32,5 +32,6 @@
"guard.col.request": "İstek",
"guard.col.model": "Model",
"guard.col.status": "Durum",
"guard.request_hint": "Bir isteğin başka nelerden geçtiğini <code>llmgw ctl request ID</code> söyler."
"guard.request_hint": "Bir isteğin başka nelerden geçtiğini <code>llmgw ctl request ID</code> söyler.",
"guard.off": "Guardrail'ler <strong>kapalı</strong>: aşağıdaki politikalar ne derse desin hiçbir istek kontrol edilmiyor. <code>llmgw ctl guard on</code> yeniden açar."
}
+2 -1
View File
@@ -9,6 +9,7 @@
{{template "nav" .Nav}}
</header>
{{if .Error}}<p class="error" role="alert">{{.Error}}</p>{{end}}
{{if .Off}}<p class="error" role="alert">{{th "guard.off"}}</p>{{end}}
<h2>{{t "guard.policies"}}</h2>
{{if .Policies}}
@@ -39,7 +40,7 @@
<table class="rows">
<tr><th>{{t "guard.col.when"}}</th><th>{{t "guard.col.request"}}</th><th>{{t "guard.col.where"}}</th><th>{{t "guard.col.model"}}</th><th>{{t "guard.col.action"}}</th><th>{{t "guard.col.policy"}}</th><th>{{t "guard.col.category"}}</th><th class="num">{{t "guard.col.status"}}</th><th class="num">ms</th></tr>
{{range .Events}}<tr><td>{{.At}}</td><td><code>{{.RequestID}}</code></td><td>{{.Where}}</td><td>{{.Model}}</td>
<td>{{.Action}}</td><td><code>{{.Policy}}</code></td><td>{{.Category}}</td><td class="num">{{.Status}}</td><td class="num">{{count .Ms}}</td></tr>{{end}}
<td>{{.Action}}</td><td><code>{{.Policy}}</code></td><td>{{.Category}}{{if .Rule}} · {{.Rule}}{{end}}</td><td class="num">{{.Status}}</td><td class="num">{{count .Ms}}</td></tr>{{end}}
</table>
<p class="sub">{{th "guard.request_hint"}}</p>
{{end}}