Compare commits

..
5 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 f285aa0548 v0.12.2
Release / build (push) Successful in 4m41s
CI / test (push) Successful in 4m43s
Release / notes (push) Successful in 3s
Guardrails off and on; a judge's refusal names the hazard; ctl guard judge ask.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-06 13:35:16 +02:00
Ilker ManapandClaude Opus 5.5 4721b8e353 Llama Guard 3's verdicts carry their hazards' names without a categories list
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-06 12:44:53 +02:00
Ilker ManapandClaude Opus 5.5 249a91aeec Guardrails turn off and on; a judge's refusal says which hazard; ctl guard judge ask
guard.enabled (unset is on) makes every policy check nothing while false,
live: llmgw ctl guard off/on set it in the cluster's settings, and ctl
status, ctl guard list and the guardrails page say when they are off.

A refusal by a judge names the hazard it saw - judge: S6,Specialized
Advice - and the verdict is kept with the request in a new usage column,
guard_rule, shown by ctl request, the request log and the guardrails page.

llmgw ctl guard judge ask has a judge read a text, past the cache, and shows
its raw answer and how it is read; the endpoint is kept out of the audit so
no text is recorded.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-06 12:41:08 +02:00
Ilker ManapandClaude Opus 5.5 3f5a809373 v0.12.1
CI / test (push) Failing after 4m20s
Release / build (push) Successful in 4m36s
Release / notes (push) Successful in 3s
The guardrails' judge served and watched without server init.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-06 10:52:24 +02:00
Ilker ManapandClaude Opus 5.5 a8bb417618 The guardrails' judge has its own command, and its trouble is shown
llmgw ctl guard judge status lists each judge model the policies ask:
in the catalog, as a judge, replicas ready, its last failure to answer, and
what is wrong in words. enable serves the missing one through model serve -
Llama Guard 3 1B on CPUs, role guard, under the name the policies give it -
with --replace for a model of that name that is not a judge; the console has
it under Do. GET /api/v1/guard/judges, and the summary's judges, put the
problem on ctl status and the console's Overview.

A Llama Guard file served from the mirror page is added as a judge, under
the name the policies ask, rather than as a model clients use the policies
never find. ctl says plainly when the API answers with a page.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-06 10:47:37 +02:00
44 changed files with 1184 additions and 43 deletions

No files matched your search

+56
View File
@@ -22,6 +22,62 @@ is only released once its **exit criterion** is met, not when its code lands.
Until v1.0.0 the minor version is the breaking one: configuration formats and Go Until v1.0.0 the minor version is the breaking one: configuration formats and Go
APIs change between minors without a deprecation period. APIs change between minors without a deprecation period.
## v0.12.2 — 2026-10-06
The guardrails turned off and on as a whole, and a judge that refuses
ordinary requests explained: the hazard it saw is told to the client and
kept with the request, and `llmgw ctl guard judge ask` shows what it says of
any text. Nothing to do to upgrade; the store gains a column.
### Added
- **The guardrails turn off and on as a whole**: `llmgw ctl guard off` /
`on`, which is `guard.enabled` in the cluster's settings — in force within
seconds, the policies and their attachments kept. `ctl status`, `ctl guard
list` and the guardrails page say when they are off.
- **`llmgw ctl guard judge ask TEXT`** has the judge read a text and shows
its answer as given and as read — for finding out why ordinary requests
are refused. Nothing of the text is kept, not even in the audit.
### Changed
- **A judge's refusal names the hazard**: the client is told `(judge:
S6,Specialized Advice)` rather than `(judge)`, and the verdict is kept with
the request (`guard_rule`): `ctl request`, the server's log and the
guardrails page show it. Llama Guard 3's codes come with their names,
S9 Indiscriminate Weapons, without a categories list.
## v0.12.1 — 2026-10-06
The guardrails' judge, served and watched without server init: a judge
missing, served as the wrong kind of model, or not answering is said on
`ctl status` and the console, and `llmgw ctl guard judge enable` puts it
right. Nothing to do to upgrade.
### Added
- **`llmgw ctl guard judge status` and `enable`**: each judge model the
policies ask, whether it is in the catalog, as a judge, with replicas
ready and answering — and serving the missing one, Llama Guard 3 1B on
CPUs under the name the policies give it, with `--replace` for a model of
that name that is not a judge. No more `server init` to put a judge back.
Also **Do → Serve the guardrails' judge** in the console.
- **A missing or broken judge is said where it is looked for**: `llmgw ctl
status` and the console's Overview list each judge with a problem — not in
the catalog, not a judge, no replica ready, or failing to answer — and
what that does to requests (refused or unjudged).
### Fixed
- **A Llama Guard file served from the mirror page is the judge**: it was
added as a model clients use, under whatever name the form had, so the
policies never found it and refused everything they judged
(`guard_unavailable`). It is now added as a judge, under the name the
policies ask.
- **`ctl` says so when the API answers with a page**: a command a control
plane does not know — the leader on an older release, through a standby —
failed with "invalid character '<'"; it now says which and why.
## v0.12.0 — 2026-10-06 ## v0.12.0 — 2026-10-06
A minor version, because where the configuration lives changes: everything in A minor version, because where the configuration lives changes: everything in
+184
View File
@@ -894,6 +894,78 @@
} }
} }
}, },
"/api/v1/guard/judges": {
"get": {
"operationId": "getGuardJudges",
"summary": "The judge models the policies ask, and what is wrong with each",
"tags": [
"guard"
],
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"type": "array",
"items": {
"$ref": "#/components/schemas/JudgeView"
}
}
}
}
},
"401": {
"$ref": "#/components/responses/Unauthorized"
},
"default": {
"$ref": "#/components/responses/Error"
}
}
}
},
"/api/v1/guard/judges/ask": {
"post": {
"operationId": "postGuardJudgesAsk",
"summary": "Have a judge read a text and say what it answered; nothing is kept",
"tags": [
"guard"
],
"parameters": [
{
"$ref": "#/components/parameters/Actor"
}
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/JudgeAskRequest"
}
}
}
},
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/JudgeAskView"
}
}
}
},
"401": {
"$ref": "#/components/responses/Unauthorized"
},
"default": {
"$ref": "#/components/responses/Error"
}
}
}
},
"/api/v1/jobs": { "/api/v1/jobs": {
"get": { "get": {
"operationId": "getJobs", "operationId": "getJobs",
@@ -4099,6 +4171,10 @@
"category": { "category": {
"type": "string" "type": "string"
}, },
"rule": {
"type": "string",
"description": "Rule is what decided it within the category: a pattern's name, or a judge's verdict, \"S6,Specialized Advice\"."
},
"guard_ms": { "guard_ms": {
"type": "integer", "type": "integer",
"format": "int64" "format": "int64"
@@ -4161,6 +4237,10 @@
"type": "object", "type": "object",
"description": "GuardView is the library and where its policies are attached.", "description": "GuardView is the library and where its policies are attached.",
"properties": { "properties": {
"enabled": {
"type": "boolean",
"description": "Enabled is whether the guardrails check anything (guard.enabled)."
},
"policies": { "policies": {
"type": "array", "type": "array",
"items": { "items": {
@@ -4359,6 +4439,96 @@
} }
} }
}, },
"JudgeAskRequest": {
"type": "object",
"description": "JudgeAskRequest is a text to have a judge read.",
"properties": {
"model": {
"type": "string",
"description": "Model is the judge; empty is the one the policies ask, when they ask one."
},
"text": {
"type": "string"
},
"answer": {
"type": "string",
"description": "Answer, when given, is read as the assistant's reply to Text."
}
}
},
"JudgeAskView": {
"type": "object",
"description": "JudgeAskView is what the judge answered.",
"properties": {
"model": {
"type": "string"
},
"verdict": {
"type": "string",
"description": "Verdict is safe or unsafe, as the gateway reads the answer."
},
"categories": {
"type": "array",
"items": {
"type": "string"
}
},
"raw": {
"type": "string",
"description": "Raw is the judge's answer as it gave it."
},
"took_ms": {
"type": "integer",
"format": "int64"
}
}
},
"JudgeView": {
"type": "object",
"description": "JudgeView is one judge model the guardrail policies ask, and whether it can answer them.",
"properties": {
"model": {
"type": "string"
},
"policies": {
"type": "array",
"items": {
"type": "string"
},
"description": "Policies are those that ask it; Closed those that refuse what it cannot read (fail: closed)."
},
"closed": {
"type": "array",
"items": {
"type": "string"
}
},
"in_catalog": {
"type": "boolean",
"description": "InCatalog is whether a model of that name is in the catalog, and Judge whether it is there as a judge (role: guard)."
},
"judge": {
"type": "boolean"
},
"ready_replicas": {
"type": "integer"
},
"wanted_replicas": {
"type": "integer"
},
"last_error": {
"type": "string",
"description": "LastError is its last failure to give a verdict, if any."
},
"last_error_at": {
"type": "string"
},
"problem": {
"type": "string",
"description": "Problem says, in words, what is wrong and what to do; empty when nothing is."
}
}
},
"Live": { "Live": {
"type": "object", "type": "object",
"description": "Live is one of the cluster's adapters as its node last reported it.", "description": "Live is one of the cluster's adapters as its node last reported it.",
@@ -5325,6 +5495,9 @@
"guard_category": { "guard_category": {
"type": "string" "type": "string"
}, },
"guard_rule": {
"type": "string"
},
"guard_ms": { "guard_ms": {
"type": "integer", "type": "integer",
"format": "int64" "format": "int64"
@@ -5696,6 +5869,17 @@
"control_planes": { "control_planes": {
"$ref": "#/components/schemas/ControlPlanes", "$ref": "#/components/schemas/ControlPlanes",
"description": "ControlPlanes is the cluster's control planes when there is more than one: which leads, which stand by, and the floating address in front." "description": "ControlPlanes is the cluster's control planes when there is more than one: which leads, which stand by, and the floating address in front."
},
"judges": {
"type": "array",
"items": {
"$ref": "#/components/schemas/JudgeView"
},
"description": "Judges are the guardrails' judge models with something wrong."
},
"guard_off": {
"type": "boolean",
"description": "GuardOff says the guardrails are turned off (guard.enabled: false): no request is checked, whatever the policies say."
} }
} }
}, },
+49 -1
View File
@@ -15,6 +15,19 @@ checked and nothing costs anything. With one, every request is judged once it is
within its limits, with its stored conversation spliced in, and before it is within its limits, with its stored conversation spliced in, and before it is
sent to a model; and the answer is judged before the client sees it. sent to a model; and the answer is judged before the client sees it.
## Turning them off and on
```
llmgw ctl guard off --note "why" # no request is checked, whatever the policies say
llmgw ctl guard on # every policy checks again, as it was
```
That is `guard.enabled` in the cluster's settings, in force on every control
plane within seconds without a restart. The policies and where they are
attached are kept. While off, `llmgw ctl status`, `llmgw ctl guard list` and
the guardrails page say so. To stop one policy rather than all, give it
`mode: off`, or `mode: shadow` to have it record what it would do.
## A policy ## A policy
```yaml ```yaml
@@ -103,7 +116,42 @@ guard:
Patterns find what has a shape. For what does not — violence, self-harm, Patterns find what has a shape. For what does not — violence, self-harm,
abuse, whatever a moderation model was trained on — a policy can have a abuse, whatever a moderation model was trained on — a policy can have a
**judge model on the cluster** read the text: **judge model on the cluster** read the text.
The judge must be in the catalog under the name the policy gives, as a
judge (`role: guard`), with a replica ready. When it is not, what the
policy judges is refused with `guard_unavailable` (`fail: closed`) or served
unjudged (`fail: open`), and `llmgw ctl status` and the console's Overview
say which judge and why. To serve it:
```
llmgw ctl guard judge status # each judge the policies ask, and what is wrong with it
llmgw ctl guard judge enable # Llama Guard 3 1B on CPUs, under the name the policies ask
llmgw ctl guard judge enable --replace # when a model of that name is there, but not as a judge
```
or **Do → Serve the guardrails' judge** in the console.
**When it refuses ordinary requests.** A refusal by a judge names the hazard
it saw — the client is told `refused by the gateway's policy (judge:
S6,Specialized Advice)` — and the same is kept with the request: `llmgw ctl
request <id>`, the server's log (`guard_rule`) and the guardrails page show
it. To see what the judge says of any text, without anything being kept:
```
llmgw ctl guard judge ask 'How do I sort a list in Python?'
# llama-guard-3-1b: safe, 140 ms
# it answered: "safe"
```
A judge that says `unsafe` to ordinary text is usually the wrong weights
(another repository's or quantisation's, without Llama Guard's template:
`judge enable --replace` serves the right ones), or a policy counting every
hazard: give its judge `categories`, leaving out S5, S6, S8 and S13 as the
baseline does. While finding out, `judge.input: flag` records instead of
refusing. A Llama Guard file
served from the mirror page is served as a judge too, under the name the
policies ask. By hand, the catalog entry is:
```yaml ```yaml
# a catalog model (server.yaml models:, or ctl model add) # a catalog model (server.yaml models:, or ctl model add)
+8
View File
@@ -372,6 +372,7 @@ rather than repeated.
| `llamacpp` | `LlamaCppConfig` | [engines.Config](#enginesconfig) | | `llamacpp` | `LlamaCppConfig` | [engines.Config](#enginesconfig) |
| `trtllm` | `TRTLLMConfig` | [engines.Config](#enginesconfig) | | `trtllm` | `TRTLLMConfig` | [engines.Config](#enginesconfig) |
| `sim` | `SimConfig` | [engines.Config](#enginesconfig) | | `sim` | `SimConfig` | [engines.Config](#enginesconfig) |
| `enabled` | `*bool` | [guard.Config](#guardconfig) |
| `default` | `[]string` | [guard.Config](#guardconfig) | | `default` | `[]string` | [guard.Config](#guardconfig) |
| `policies` | `map[string]Policy` | [guard.Config](#guardconfig) | | `policies` | `map[string]Policy` | [guard.Config](#guardconfig) |
| `webhooks` | `[]Webhook` | [notify.Config](#notifyconfig) | | `webhooks` | `[]Webhook` | [notify.Config](#notifyconfig) |
@@ -3067,6 +3068,13 @@ Takes a [SimConfig](#simconfig) block.
## guard.Config ## guard.Config
### `enabled`
`*bool`
Enabled turns every policy on or off at once: false checks nothing,
whatever is defined or attached. Unset is on.
### `default` ### `default`
`[]string` `[]string`
+7 -1
View File
@@ -12,6 +12,7 @@ import (
"net/http" "net/http"
"net/url" "net/url"
"regexp" "regexp"
"slices"
"sort" "sort"
"strings" "strings"
"time" "time"
@@ -110,11 +111,16 @@ func (w *statusWriter) Flush() {
} }
} }
// notChanges are POSTs that change nothing and carry text that must not be
// kept: asking a judge about a sentence.
var notChanges = []string{"/api/v1/guard/judges/ask"}
// API records every change to the admin API: a request that is not a GET or // API records every change to the admin API: a request that is not a GET or
// HEAD, under /api/. actor says who made it. // HEAD, under /api/. actor says who made it.
func API(sink Sink, actor func(*http.Request) string, log *slog.Logger, next http.Handler) http.Handler { func API(sink Sink, actor func(*http.Request) string, log *slog.Logger, next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method == http.MethodGet || r.Method == http.MethodHead || !strings.HasPrefix(r.URL.Path, "/api/") { if r.Method == http.MethodGet || r.Method == http.MethodHead || !strings.HasPrefix(r.URL.Path, "/api/") ||
slices.Contains(notChanges, r.URL.Path) {
next.ServeHTTP(w, r) next.ServeHTTP(w, r)
return return
} }
+10
View File
@@ -72,3 +72,13 @@ func TestABodyLargerThanWhatIsRecordedReachesTheHandlerWhole(t *testing.T) {
t.Errorf("the handler was given %d of %d bytes", got, len(big)) t.Errorf("the handler was given %d of %d bytes", got, len(big))
} }
} }
func TestAskingAJudgeKeepsNoText(t *testing.T) {
sink := &memSink{}
h := API(sink, func(*http.Request) string { return "ilker" }, nil, http.HandlerFunc(func(http.ResponseWriter, *http.Request) {}))
h.ServeHTTP(httptest.NewRecorder(), httptest.NewRequest("POST", "/api/v1/guard/judges/ask",
strings.NewReader(`{"text":"something a user wrote"}`)))
if len(sink.got) != 0 {
t.Errorf("a judge's question was recorded: %+v", sink.got)
}
}
+7
View File
@@ -116,6 +116,13 @@ func (c *ctlClient) do(ctx context.Context, method, path string, body, out any)
if out == nil { if out == nil {
return nil return nil
} }
// An older control plane answers a path it does not know with its
// dashboard: a page, which read as JSON said only "invalid character '<'".
if strings.HasPrefix(resp.Header.Get("Content-Type"), "text/html") {
return fmt.Errorf("%s %s answered with a page, not the API: the control plane answering "+
"(the leader, through a standby) does not know it — is it on an older release? "+
"llmgw ctl status says which it runs", method, path)
}
return json.NewDecoder(resp.Body).Decode(out) return json.NewDecoder(resp.Body).Decode(out)
} }
+22
View File
@@ -0,0 +1,22 @@
package cli
import (
"net/http"
"net/http/httptest"
"strings"
"testing"
)
func TestAPageInsteadOfTheAPIIsSaidPlainly(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Type", "text/html; charset=utf-8")
_, _ = w.Write([]byte("<!doctype html><title>llmgw</title>"))
}))
defer srv.Close()
c := newCtlClient(strings.TrimPrefix(srv.URL, "http://"), "")
var out map[string]any
err := c.get(t.Context(), "/api/v1/settings", &out)
if err == nil || !strings.Contains(err.Error(), "older release") || strings.Contains(err.Error(), "invalid character") {
t.Errorf("%v", err)
}
}
+5 -2
View File
@@ -76,8 +76,11 @@ func newRequestCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
fmt.Fprintf(out, "tags %s\n", strings.Join(kv, ", ")) fmt.Fprintf(out, "tags %s\n", strings.Join(kv, ", "))
} }
if v.GuardAction != "" { if v.GuardAction != "" {
fmt.Fprintf(out, "guard %s by policy %s (%s), %d ms\n", v.GuardAction, v.GuardPolicy, what := or(v.GuardCategory, "-")
or(v.GuardCategory, "-"), v.GuardMs) if v.GuardRule != "" {
what += ": " + v.GuardRule
}
fmt.Fprintf(out, "guard %s by policy %s (%s), %d ms\n", v.GuardAction, v.GuardPolicy, what, v.GuardMs)
} else if v.GuardMs > 0 { } else if v.GuardMs > 0 {
fmt.Fprintf(out, "guard allowed, %d ms\n", v.GuardMs) fmt.Fprintf(out, "guard allowed, %d ms\n", v.GuardMs)
} }
+34
View File
@@ -3,11 +3,13 @@ package cli
import ( import (
"encoding/json" "encoding/json"
"fmt" "fmt"
"strconv"
"strings" "strings"
"github.com/spf13/cobra" "github.com/spf13/cobra"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi" "gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
) )
func newGuardCmd(client func() *ctlClient, asJSON *bool) *cobra.Command { func newGuardCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
@@ -24,6 +26,35 @@ func newGuardCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
"Targets are written org:Acme, team:Acme/ml, project:Acme/ml/training (or\n" + "Targets are written org:Acme, team:Acme/ml, project:Acme/ml/training (or\n" +
"with IDs), key:KEY-ID or user:USER-ID. An attachment applies at once.", "with IDs), key:KEY-ID or user:USER-ID. An attachment applies at once.",
} }
c.AddCommand(newGuardJudgeCmd(client, asJSON))
for _, sw := range []struct {
use, short string
on bool
}{
{"on", "Turn the guardrails on: every policy checks again", true},
{"off", "Turn the guardrails off: no request is checked, whatever the policies say", false},
} {
var note string
cmd := &cobra.Command{
Use: sw.use,
Short: sw.short,
Long: "guard.enabled in the cluster's settings: in force on every control plane\n" +
"within seconds, without a restart. The policies and where they are attached\n" +
"stay as they are, and come back as they were when turned on again. While\n" +
"off, llmgw ctl status says so.",
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error {
if note == "" {
note = "guardrails " + sw.use
}
return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
return doc.SetYAML([]string{"guard", "enabled"}, strconv.FormatBool(sw.on))
})
},
}
cmd.Flags().StringVar(&note, "note", "", "why, kept with the settings' version")
c.AddCommand(cmd)
}
c.AddCommand(&cobra.Command{ c.AddCommand(&cobra.Command{
Use: "list", Use: "list",
Short: "The policies, and where each is attached", Short: "The policies, and where each is attached",
@@ -37,6 +68,9 @@ func newGuardCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
return json.NewEncoder(cmd.OutOrStdout()).Encode(v) return json.NewEncoder(cmd.OutOrStdout()).Encode(v)
} }
out := cmd.OutOrStdout() out := cmd.OutOrStdout()
if !v.Enabled {
fmt.Fprintln(out, "The guardrails are OFF: no request is checked (llmgw ctl guard on).")
}
fmt.Fprintf(out, "Policies: %s\n", strings.Join(v.Policies, ", ")) fmt.Fprintf(out, "Policies: %s\n", strings.Join(v.Policies, ", "))
def := strings.Join(v.Default, ", ") def := strings.Join(v.Default, ", ")
if def == "" { if def == "" {
+225
View File
@@ -0,0 +1,225 @@
package cli
import (
"encoding/json"
"errors"
"fmt"
"strings"
"time"
"github.com/spf13/cobra"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
)
// The judge the guardrails are installed with: Meta's Llama Guard 3 1B,
// from the repository server init fetches it from.
const (
guardModelRepo = "QuantFactory/Llama-Guard-3-1B-GGUF"
guardModelFile = "Llama-Guard-3-1B.Q4_K_M.gguf"
)
func newGuardJudgeCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
c := &cobra.Command{
Use: "judge",
Short: "The judge models the policies ask: whether each can answer, and serving one",
Long: "A policy with a judge has a model read what patterns cannot. The judge must\n" +
"be in the catalog under the name the policy gives, as a judge (role:\n" +
"guard), with a replica ready; otherwise what the policy judges is refused\n" +
"(fail: closed, guard_unavailable) or served unjudged (fail: open).",
}
c.AddCommand(newGuardJudgeStatusCmd(client, asJSON), newGuardJudgeEnableCmd(client), newGuardJudgeAskCmd(client, asJSON))
return c
}
func newGuardJudgeStatusCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
return &cobra.Command{
Use: "status",
Short: "Each judge the policies ask, and what is wrong with it",
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error {
var js []adminapi.JudgeView
if err := client().get(cmd.Context(), "/api/v1/guard/judges", &js); err != nil {
return err
}
if *asJSON {
return emitJSON(js)
}
out := cmd.OutOrStdout()
if len(js) == 0 {
fmt.Fprintln(out, "no policy asks a judge")
return nil
}
w := tabTo(out)
fmt.Fprintln(w, "JUDGE\tPOLICIES\tIN CATALOG\tAS JUDGE\tREADY\tSTATE")
for _, j := range js {
state := "answering"
if j.Problem != "" {
state = j.Problem
}
fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%d/%d\t%s\n", j.Model, strings.Join(j.Policies, ","),
yesNo(j.InCatalog), yesNo(j.Judge), j.Ready, j.Wanted, state)
}
if err := w.Flush(); err != nil {
return err
}
strays, _ := strayJudges(cmd, client, js)
for _, s := range strays {
fmt.Fprintf(out, "\n%s looks like a judge's weights but is served as a model clients use, and no "+
"policy asks it: llmgw ctl model remove %s\n", s, s)
}
return nil
},
}
}
// strayJudges are models whose weights are a judge's but which are in the
// catalog as models clients use, under a name no policy asks: what serving
// Llama Guard from the mirror page did before it knew better.
func strayJudges(cmd *cobra.Command, client func() *ctlClient, js []adminapi.JudgeView) ([]string, error) {
var models []adminapi.ModelView
if err := client().get(cmd.Context(), "/api/v1/models", &models); err != nil {
return nil, err
}
asked := map[string]bool{}
for _, j := range js {
asked[j.Model] = true
}
var out []string
for _, m := range models {
if !asked[m.Name] && strings.Contains(strings.ToLower(m.Source), "llama-guard") {
out = append(out, m.Name)
}
}
return out, nil
}
func newGuardJudgeEnableCmd(client func() *ctlClient) *cobra.Command {
var (
model, repo, file string
replicas, cores int
replace, detach bool
)
c := &cobra.Command{
Use: "enable",
Short: "Serve the judge the policies ask, as a judge: Llama Guard 3 1B on CPUs",
Long: "Fetches Llama Guard 3 1B into the mirror (once), adds it to the catalog under\n" +
"the name the policies give its judge, as a judge (role: guard,\n" +
"format llamaguard3), on CPUs so it takes no card, and waits for a replica\n" +
"to answer. With no --model, the judge the policies ask that is missing.\n\n" +
"A model of that name already in the catalog as a model clients use is\n" +
"refused; --replace removes it first. An offline cluster gets its judge\n" +
"from its bundle with server init.",
Example: " llmgw ctl guard judge enable\n" +
" llmgw ctl guard judge enable --replace",
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error {
ctx := cmd.Context()
out := cmd.OutOrStdout()
var js []adminapi.JudgeView
if err := client().get(ctx, "/api/v1/guard/judges", &js); err != nil {
return err
}
var target *adminapi.JudgeView
for i, j := range js {
if (model != "" && j.Model == model) || (model == "" && (!j.InCatalog || !j.Judge)) {
target = &js[i]
break
}
}
switch {
case target == nil && model != "":
target = &adminapi.JudgeView{Model: model}
fmt.Fprintf(out, "no policy asks %s yet; it is served, and a policy's judge.model names it\n", model)
case target == nil && len(js) == 0:
return errors.New("no policy asks a judge: give one a judge (llmgw ctl config edit, guard.policies.<name>.judge.model)")
case target == nil:
fmt.Fprintln(out, "every judge the policies ask is in the catalog, as a judge")
return nil
}
if target.InCatalog && target.Judge {
fmt.Fprintf(out, "%s is in the catalog as a judge already; llmgw ctl guard judge status says how it is\n", target.Model)
return nil
}
if target.InCatalog && !target.Judge {
if !replace {
return fmt.Errorf("%s is in the catalog as a model clients use, not as a judge: "+
"--replace removes it and serves it again as a judge", target.Model)
}
var res map[string]string
if err := client().delete(ctx, "/api/v1/models/"+target.Model, &res); err != nil {
return fmt.Errorf("removing %s: %w", target.Model, err)
}
fmt.Fprintf(out, "%s removed\n", target.Model)
}
req := modelserve.Request{
Repo: repo, File: file, Name: target.Model, GuardFormat: "llamaguard3",
Description: "The guardrails' judge: reads prompts and answers for unsafe content. Not offered to clients.",
CPU: true, Replicas: replicas, CPUCores: cores,
CtxPool: 8192, ContextLength: 4096,
}
body, _ := json.Marshal(req)
var job modelserve.Job
if err := client().patient(30*time.Second).post(ctx, "/api/v1/models/serve", body, &job); err != nil {
return err
}
if detach {
return detached(cmd, job)
}
if err := followJob(cmd, client, job, "is the judge"); err != nil {
return err
}
if strays, _ := strayJudges(cmd, client, js); len(strays) > 0 {
fmt.Fprintf(out, "\nno policy asks %s, served from a judge's weights as a model clients use: "+
"llmgw ctl model remove %s\n", strings.Join(strays, ", "), strings.Join(strays, " "))
}
return nil
},
}
f := c.Flags()
f.StringVar(&model, "model", "", "the judge to serve (default: the one the policies ask that is missing)")
f.StringVar(&repo, "repo", guardModelRepo, "the registry repository its weights come from")
f.StringVar(&file, "file", guardModelFile, "the file in it")
f.IntVar(&replicas, "replicas", 2, "replicas: every judged request waits on one")
f.IntVar(&cores, "cores", 4, "CPU cores a replica computes on")
f.BoolVar(&replace, "replace", false, "remove a model of that name that is not a judge first")
f.BoolVar(&detach, "detach", false, "start it and return; llmgw ctl jobs follows it")
return c
}
func newGuardJudgeAskCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
var model, answer string
c := &cobra.Command{
Use: "ask TEXT",
Short: "Have a judge read a text and say what it answered",
Long: "For finding out why a request was refused: the judge reads TEXT as a user's\n" +
"message (with --answer, ANSWER as the reply to it), past the cache, and its\n" +
"answer is shown as it gave it and as the gateway reads it — \"unsafe\" and\n" +
"the hazards it named, S6 Specialized Advice and the like. Nothing is kept:\n" +
"not in the audit, the usage or the logs.",
Example: " llmgw ctl guard judge ask 'How do I sort a list in Python?'\n" +
" llmgw ctl guard judge ask 'Write a poem' --answer 'Roses are red'",
Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
body, _ := json.Marshal(adminapi.JudgeAskRequest{Model: model, Text: args[0], Answer: answer})
var v adminapi.JudgeAskView
if err := client().patient(45*time.Second).post(cmd.Context(), "/api/v1/guard/judges/ask", body, &v); err != nil {
return err
}
if *asJSON {
return emitJSON(v)
}
out := cmd.OutOrStdout()
fmt.Fprintf(out, "%s: %s", v.Model, v.Verdict)
if len(v.Categories) > 0 {
fmt.Fprintf(out, " (%s)", strings.Join(v.Categories, ", "))
}
fmt.Fprintf(out, ", %d ms\nit answered: %q\n", v.TookMs, v.Raw)
return nil
},
}
c.Flags().StringVar(&model, "model", "", "the judge (default: the one the policies ask)")
c.Flags().StringVar(&answer, "answer", "", "an assistant's reply to TEXT, judged as output is")
return c
}
+2 -1
View File
@@ -361,7 +361,8 @@ func newServerCmd() *cobra.Command {
evals = newEvalService(cfg, rt, admin, log) evals = newEvalService(cfg, rt, admin, log)
api.SetEval(evals) api.SetEval(evals)
if st := rt.Store(); st != nil && gate != nil { if st := rt.Store(); st != nil && gate != nil {
api.SetGuard(&adminapi.GuardAdmin{Library: gate.Library, Store: st, Invalidate: gate.Invalidate}) api.SetGuard(&adminapi.GuardAdmin{Library: gate.Library, Store: st, Invalidate: gate.Invalidate,
JudgeErrors: gate.JudgeErrors, Ask: gate.AskJudge})
} }
if st := rt.Store(); st != nil { if st := rt.Store(); st != nil {
// Removing a node revokes its certificate. // Removing a node revokes its certificate.
+10
View File
@@ -153,6 +153,16 @@ func printSummary(cmd *cobra.Command, s adminapi.Summary) {
fmt.Fprintf(out, "STORE %s\n", strings.Join(parts, ", ")) fmt.Fprintf(out, "STORE %s\n", strings.Join(parts, ", "))
} }
if s.GuardOff {
fmt.Fprintln(out, "GUARD off: no request is checked, whatever the policies say (llmgw ctl guard on)")
}
if len(s.Judges) > 0 {
fmt.Fprintln(out, "\nGUARDRAILS' JUDGE")
for _, j := range s.Judges {
fmt.Fprintf(out, " %s: %s\n", j.Model, j.Problem)
}
}
if len(s.Parked) > 0 { if len(s.Parked) > 0 {
fmt.Fprintln(out, "\nPARKED — failed every time they started; not retried until fixed") fmt.Fprintln(out, "\nPARKED — failed every time they started; not retried until fixed")
for _, p := range s.Parked { for _, p := range s.Parked {
+2
View File
@@ -679,6 +679,8 @@ func (m *consoleModel) doEntry() tea.Cmd {
}} }}
case cBackup: case cBackup:
return m.runAfter("server", "backup", "--config="+m.file) return m.runAfter("server", "backup", "--config="+m.file)
case cJudge:
return m.runAfter("ctl", server, "guard", "judge", "enable", "--replace")
} }
return nil return nil
} }
+6 -2
View File
@@ -59,6 +59,7 @@ const (
cAddControl cAddControl
cReplicate cReplicate
cServe cServe
cJudge
cUpgrade cUpgrade
cRestartAll cRestartAll
cBackup cBackup
@@ -68,7 +69,7 @@ const (
) )
var consoleLabels = [cCount]string{"Overview", "Monitor", "Nodes", "GPUs", "Models", "Replicas", "Usage", "Quotas", "Guardrails", "Federation", "Mirror", "Jobs", "Logs", "Accounts", "API keys", "Organizations", "Audit", "Manual", var consoleLabels = [cCount]string{"Overview", "Monitor", "Nodes", "GPUs", "Models", "Replicas", "Usage", "Quotas", "Guardrails", "Federation", "Mirror", "Jobs", "Logs", "Accounts", "API keys", "Organizations", "Audit", "Manual",
"server.yaml", "Add a node", "Add a control plane", "Replicate the store", "Serve a model", "Upgrade the cluster", "server.yaml", "Add a node", "Add a control plane", "Replicate the store", "Serve a model", "Serve the guardrails' judge", "Upgrade the cluster",
"Restart all replicas", "Back up now", "Wipe this machine", "Leave"} "Restart all replicas", "Back up now", "Wipe this machine", "Leave"}
// consoleAction is what is done once the screen closes: each runs its own // consoleAction is what is done once the screen closes: each runs its own
@@ -496,7 +497,7 @@ func (m consoleModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) {
return m.leave() return m.leave()
case cWipe: case cWipe:
return m.askWipe() return m.askWipe()
case cUpgrade, cRestartAll, cBackup: case cUpgrade, cRestartAll, cBackup, cJudge:
return m, m.doEntry() return m, m.doEntry()
case cOverview: case cOverview:
return m, nil return m, nil
@@ -1299,6 +1300,9 @@ func (m consoleModel) panel() (title, body, keys string) {
item = "\n Moves the store onto two machines under Patroni, replicated,\n so losing one loses nothing.\n\n " + item = "\n Moves the store onto two machines under Patroni, replicated,\n so losing one loses nothing.\n\n " +
sDim.Render("On the command line:") + sDim.Render("On the command line:") +
"\n sudo --preserve-env=SSH_AUTH_SOCK llmgw server init \\\n --patroni <user>@<a> --patroni <user>@<b> --no-mirror --no-node --no-model" "\n sudo --preserve-env=SSH_AUTH_SOCK llmgw server init \\\n --patroni <user>@<a> --patroni <user>@<b> --no-mirror --no-node --no-model"
case cJudge:
item = "\n Serves the model the guardrail policies ask as their judge — Llama\n Guard 3 1B on CPUs, under the name the policies give it — so what they\n judge is not refused (guard_unavailable). A model of that name that is\n not a judge is replaced. Overview says when a judge is missing.\n\n " +
sDim.Render("On the command line:") + "\n llmgw ctl guard judge status\n llmgw ctl guard judge enable --replace"
case cUpgrade: case cUpgrade:
item = "\n Moves the cluster to a release: this machine first, then each node\n in turn. Engines keep serving; a node that does not come back stops it.\n\n " + item = "\n Moves the cluster to a release: this machine first, then each node\n in turn. Engines keep serving; a node that does not come back stops it.\n\n " +
sDim.Render("On the command line:") + "\n sudo llmgw ctl upgrade VERSION" sDim.Render("On the command line:") + "\n sudo llmgw ctl upgrade VERSION"
+2 -1
View File
@@ -1545,6 +1545,7 @@ type RequestView struct {
GuardAction string `json:"guard_action,omitempty"` GuardAction string `json:"guard_action,omitempty"`
GuardPolicy string `json:"guard_policy,omitempty"` GuardPolicy string `json:"guard_policy,omitempty"`
GuardCategory string `json:"guard_category,omitempty"` GuardCategory string `json:"guard_category,omitempty"`
GuardRule string `json:"guard_rule,omitempty"`
GuardMs int64 `json:"guard_ms,omitempty"` GuardMs int64 `json:"guard_ms,omitempty"`
// What the client said about it: OpenAI's user field, and its tags. // What the client said about it: OpenAI's user field, and its tags.
EndUser string `json:"end_user,omitempty"` EndUser string `json:"end_user,omitempty"`
@@ -1573,7 +1574,7 @@ func (a *API) request(w http.ResponseWriter, r *http.Request) {
Model: u.Model, ServedModel: u.ServedModel, Replica: u.Replica, Status: u.Status, Model: u.Model, ServedModel: u.ServedModel, Replica: u.Replica, Status: u.Status,
LatencyMs: u.LatencyMs, QueueMs: u.QueueMs, TTFTMs: u.TTFTMs, LatencyMs: u.LatencyMs, QueueMs: u.QueueMs, TTFTMs: u.TTFTMs,
PromptTokens: u.PromptTokens, CompletionTokens: u.CompletionTokens, CostMicros: u.CostMicros, PromptTokens: u.PromptTokens, CompletionTokens: u.CompletionTokens, CostMicros: u.CostMicros,
GuardAction: u.GuardAction, GuardPolicy: u.GuardPolicy, GuardCategory: u.GuardCategory, GuardMs: u.GuardMs, GuardAction: u.GuardAction, GuardPolicy: u.GuardPolicy, GuardCategory: u.GuardCategory, GuardRule: u.GuardRule, GuardMs: u.GuardMs,
EndUser: u.EndUser, Tags: u.Tags, CacheHit: u.CacheHit, EndUser: u.EndUser, Tags: u.Tags, CacheHit: u.CacheHit,
}) })
} }
+14 -3
View File
@@ -28,6 +28,10 @@ type GuardAdmin struct {
Library func() guard.Config Library func() guard.Config
Store GuardStore Store GuardStore
Invalidate func() Invalidate func()
// JudgeErrors are each judge's last failure to give a verdict.
JudgeErrors func() map[string]guard.JudgeError
// Ask has a judge read a text, past the cache.
Ask func(ctx context.Context, model, text, answer string) (guard.JudgeAnswer, error)
} }
// SetGuard enables the guard endpoints. // SetGuard enables the guard endpoints.
@@ -35,6 +39,8 @@ func (a *API) SetGuard(g *GuardAdmin) { a.guard = g }
// GuardView is the library and where its policies are attached. // GuardView is the library and where its policies are attached.
type GuardView struct { type GuardView struct {
// Enabled is whether the guardrails check anything (guard.enabled).
Enabled bool `json:"enabled"`
Policies []string `json:"policies"` Policies []string `json:"policies"`
Default []string `json:"default"` Default []string `json:"default"`
Attachments []GuardAttachmentView `json:"attachments"` Attachments []GuardAttachmentView `json:"attachments"`
@@ -89,7 +95,10 @@ type GuardEventView struct {
Action string `json:"action"` Action string `json:"action"`
Policy string `json:"policy"` Policy string `json:"policy"`
Category string `json:"category,omitempty"` Category string `json:"category,omitempty"`
GuardMs int64 `json:"guard_ms"` // Rule is what decided it within the category: a pattern's name, or a
// judge's verdict, "S6,Specialized Advice".
Rule string `json:"rule,omitempty"`
GuardMs int64 `json:"guard_ms"`
} }
// policyView sums one policy up. // policyView sums one policy up.
@@ -198,6 +207,8 @@ type GuardAttachmentRequest struct {
} }
func (a *API) registerGuard(mux Mux) { func (a *API) registerGuard(mux Mux) {
mux.HandleFunc("GET /api/v1/guard/judges", a.listJudges)
mux.HandleFunc("POST /api/v1/guard/judges/ask", a.askJudge)
mux.HandleFunc("GET /api/v1/guard", func(w http.ResponseWriter, r *http.Request) { mux.HandleFunc("GET /api/v1/guard", func(w http.ResponseWriter, r *http.Request) {
if a.guard == nil { if a.guard == nil {
writeError(w, http.StatusNotImplemented, "no guard policies: define them under guard in the cluster's settings (llmgw ctl config edit)") writeError(w, http.StatusNotImplemented, "no guard policies: define them under guard in the cluster's settings (llmgw ctl config edit)")
@@ -208,7 +219,7 @@ func (a *API) registerGuard(mux Mux) {
writeError(w, http.StatusInternalServerError, err.Error()) writeError(w, http.StatusInternalServerError, err.Error())
return return
} }
v := GuardView{Default: a.guard.Library().Default, Attachments: []GuardAttachmentView{}} v := GuardView{Enabled: a.guard.Library().On(), Default: a.guard.Library().Default, Attachments: []GuardAttachmentView{}}
var names map[string]string var names map[string]string
if a.quotas != nil { if a.quotas != nil {
names, _ = a.scopeNames(r.Context()) names, _ = a.scopeNames(r.Context())
@@ -269,7 +280,7 @@ func (a *API) registerGuard(mux Mux) {
slices.SortStableFunc(v.Counts, func(a, b GuardCountView) int { return int(b.Count - a.Count) }) slices.SortStableFunc(v.Counts, func(a, b GuardCountView) int { return int(b.Count - a.Count) })
for _, e := range events { for _, e := range events {
v.Events = append(v.Events, GuardEventView{At: e.At, RequestID: e.RequestID, Project: project(e.ProjectID), v.Events = append(v.Events, GuardEventView{At: e.At, RequestID: e.RequestID, Project: project(e.ProjectID),
Model: e.Model, Status: e.Status, Action: e.Action, Policy: e.Policy, Category: e.Category, GuardMs: e.GuardMs}) Model: e.Model, Status: e.Status, Action: e.Action, Policy: e.Policy, Category: e.Category, Rule: e.Rule, GuardMs: e.GuardMs})
} }
writeJSON(w, http.StatusOK, v) writeJSON(w, http.StatusOK, v)
}) })
+181
View File
@@ -0,0 +1,181 @@
package adminapi
import (
"encoding/json"
"fmt"
"net/http"
"slices"
"sort"
"strings"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
)
// JudgeView is one judge model the guardrail policies ask, and whether it
// can answer them.
type JudgeView struct {
Model string `json:"model"`
// Policies are those that ask it; Closed those that refuse what it
// cannot read (fail: closed).
Policies []string `json:"policies"`
Closed []string `json:"closed,omitempty"`
// InCatalog is whether a model of that name is in the catalog, and
// Judge whether it is there as a judge (role: guard).
InCatalog bool `json:"in_catalog"`
Judge bool `json:"judge"`
Ready int `json:"ready_replicas"`
Wanted int `json:"wanted_replicas"`
// LastError is its last failure to give a verdict, if any.
LastError string `json:"last_error,omitempty"`
LastErrorAt *time.Time `json:"last_error_at,omitempty"`
// Problem says, in words, what is wrong and what to do; empty when
// nothing is.
Problem string `json:"problem,omitempty"`
}
// judgeErrorFresh is how recent a judge's failure must be to count as a
// problem now.
const judgeErrorFresh = 5 * time.Minute
// Judges are the judge models the policies in force ask, each with what is
// wrong with it, if anything.
func (a *API) Judges() []JudgeView {
if a.guard == nil || a.guard.Library == nil {
return nil
}
lib := a.guard.Library()
if !lib.On() {
// Off: no judge is asked, so none is missing.
return nil
}
byModel := map[string]*JudgeView{}
for name, p := range lib.Policies {
if p.Judge == nil || p.Judge.Model == "" {
continue
}
v := byModel[p.Judge.Model]
if v == nil {
v = &JudgeView{Model: p.Judge.Model}
byModel[p.Judge.Model] = v
}
v.Policies = append(v.Policies, name)
if p.Fail == "closed" && p.Mode != "shadow" {
v.Closed = append(v.Closed, name)
}
}
ready := map[string]int{}
for _, rep := range a.ctl.Reconciler().Replicas() {
if rep.State == reconciler.Ready {
ready[rep.Model]++
}
}
var errs map[string]guard.JudgeError
if a.guard.JudgeErrors != nil {
errs = a.guard.JudgeErrors()
}
out := make([]JudgeView, 0, len(byModel))
for _, v := range byModel {
sort.Strings(v.Policies)
sort.Strings(v.Closed)
if m, ok := a.ctl.Catalog().Resolve(v.Model); ok && m.Name == v.Model {
v.InCatalog, v.Judge = true, m.IsGuard()
v.Wanted, v.Ready = m.Scaling.Min, ready[m.Name]
}
if e, ok := errs[v.Model]; ok {
at := e.At
v.LastError, v.LastErrorAt = e.Err, &at
}
v.Problem = judgeProblem(*v)
out = append(out, *v)
}
slices.SortFunc(out, func(x, y JudgeView) int {
switch {
case x.Model < y.Model:
return -1
case x.Model > y.Model:
return 1
}
return 0
})
return out
}
func judgeProblem(v JudgeView) string {
consequence := "requests its policies judge are served unjudged"
if len(v.Closed) > 0 {
consequence = "requests its policies judge are refused (guard_unavailable)"
}
switch {
case !v.InCatalog:
return fmt.Sprintf("not in the catalog, so %s: llmgw ctl guard judge enable", consequence)
case !v.Judge:
return fmt.Sprintf("in the catalog as a model clients use, not as a judge (role: guard), so %s: "+
"llmgw ctl guard judge enable --replace", consequence)
case v.Ready == 0:
return fmt.Sprintf("no replica ready, so %s: llmgw ctl replicas --model %s", consequence, v.Model)
case v.LastErrorAt != nil && time.Since(*v.LastErrorAt) < judgeErrorFresh:
return fmt.Sprintf("gave no verdict %s ago (%s)", time.Since(*v.LastErrorAt).Round(time.Second), v.LastError)
}
return ""
}
func (a *API) listJudges(w http.ResponseWriter, _ *http.Request) {
if a.guard == nil {
writeError(w, http.StatusNotImplemented, "no guard policies: define them under guard in the cluster's settings (llmgw ctl config edit)")
return
}
writeJSON(w, http.StatusOK, a.Judges())
}
// JudgeAskRequest is a text to have a judge read.
type JudgeAskRequest struct {
// Model is the judge; empty is the one the policies ask, when they ask one.
Model string `json:"model,omitempty"`
Text string `json:"text"`
// Answer, when given, is read as the assistant's reply to Text.
Answer string `json:"answer,omitempty"`
}
// JudgeAskView is what the judge answered.
type JudgeAskView struct {
Model string `json:"model"`
// Verdict is safe or unsafe, as the gateway reads the answer.
Verdict string `json:"verdict"`
Categories []string `json:"categories,omitempty"`
// Raw is the judge's answer as it gave it.
Raw string `json:"raw"`
TookMs int64 `json:"took_ms"`
}
func (a *API) askJudge(w http.ResponseWriter, r *http.Request) {
if a.guard == nil || a.guard.Ask == nil {
writeError(w, http.StatusNotImplemented, "no guard policies: define them under guard in the cluster's settings (llmgw ctl config edit)")
return
}
var req JudgeAskRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil || strings.TrimSpace(req.Text) == "" {
writeError(w, http.StatusBadRequest, "give the text for the judge to read")
return
}
if req.Model == "" {
judges := a.Judges()
if len(judges) == 0 {
writeError(w, http.StatusBadRequest, "no policy asks a judge: give the model")
return
}
req.Model = judges[0].Model
}
ans, err := a.guard.Ask(r.Context(), req.Model, req.Text, req.Answer)
v := JudgeAskView{Model: req.Model, Raw: ans.Raw, Categories: ans.Categories, TookMs: ans.Took.Milliseconds()}
if err != nil {
writeError(w, http.StatusBadGateway, fmt.Sprintf("the judge %s gave no verdict: %v (it answered %q)", req.Model, err, ans.Raw))
return
}
v.Verdict = "safe"
if ans.Unsafe {
v.Verdict = "unsafe"
}
writeJSON(w, http.StatusOK, v)
}
+91
View File
@@ -0,0 +1,91 @@
package adminapi
import (
"context"
"strings"
"testing"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
)
func TestEachJudgeSaysWhatIsWrongWithIt(t *testing.T) {
models := append(testModels(),
// Served from the mirror page before it knew better: a model
// clients use, under the judge's name.
catalog.Model{Name: "shield", Source: "/s", Scaling: catalog.Scaling{Min: 1, Max: 1},
Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp"}}},
catalog.Model{Name: "llama-guard-3-1b", Source: "/g", Role: catalog.RoleGuard,
Guard: &catalog.GuardModel{Format: "llamaguard3"}, Scaling: catalog.Scaling{Min: 2, Max: 2},
Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp"}}})
a, _ := setup(t, models)
lib := guard.Config{Default: []string{"baseline"}, Policies: map[string]guard.Policy{
"baseline": {Fail: "closed", Judge: &guard.JudgePolicy{Model: "llama-guard-3-1b", Input: "block"}},
"strict": {Fail: "closed", Judge: &guard.JudgePolicy{Model: "missing-judge", Input: "block"}},
"shield": {Fail: "open", Judge: &guard.JudgePolicy{Model: "shield", Input: "flag"}},
}}
a.SetGuard(&GuardAdmin{Library: func() guard.Config { return lib }})
var js []JudgeView
if code := call(t, a, "GET", "/api/v1/guard/judges", "", &js); code != 200 || len(js) != 3 {
t.Fatalf("%d %+v", code, js)
}
by := map[string]JudgeView{}
for _, j := range js {
by[j.Model] = j
}
if j := by["missing-judge"]; j.InCatalog || !strings.Contains(j.Problem, "not in the catalog") ||
!strings.Contains(j.Problem, "refused") {
t.Errorf("missing: %+v", j)
}
if j := by["shield"]; !j.InCatalog || j.Judge || !strings.Contains(j.Problem, "--replace") ||
!strings.Contains(j.Problem, "unjudged") {
t.Errorf("not a judge: %+v", j)
}
if j := by["llama-guard-3-1b"]; !j.Judge || j.Wanted != 2 || !strings.Contains(j.Problem, "no replica ready") {
t.Errorf("not running: %+v", j)
}
var s Summary
call(t, a, "GET", "/api/v1/summary", "", &s)
if len(s.Judges) != 3 {
t.Errorf("the summary does not carry the judges' problems: %+v", s.Judges)
}
// A judge that failed lately says so, once it is otherwise fine.
at := time.Now()
v := JudgeView{Model: "x", InCatalog: true, Judge: true, Ready: 1, LastError: "deadline exceeded", LastErrorAt: &at}
if p := judgeProblem(v); !strings.Contains(p, "deadline exceeded") {
t.Errorf("a recent failure: %q", p)
}
old := at.Add(-time.Hour)
v.LastErrorAt = &old
if p := judgeProblem(v); p != "" {
t.Errorf("an old failure still counts: %q", p)
}
}
func TestAJudgeIsAskedAndItsWordsShown(t *testing.T) {
a, _ := setup(t, testModels())
lib := guard.Config{Policies: map[string]guard.Policy{
"baseline": {Judge: &guard.JudgePolicy{Model: "llama-guard-3-1b", Input: "block"}}}}
var asked string
a.SetGuard(&GuardAdmin{Library: func() guard.Config { return lib },
Ask: func(_ context.Context, model, text, _ string) (guard.JudgeAnswer, error) {
asked = model + "|" + text
return guard.JudgeAnswer{Model: model, Raw: "unsafe\nS6", Unsafe: true,
Categories: []string{"S6", "Specialized Advice"}}, nil
}})
var v JudgeAskView
if code := call(t, a, "POST", "/api/v1/guard/judges/ask", `{"text":"how do I sort a list"}`, &v); code != 200 {
t.Fatalf("%d", code)
}
if asked != "llama-guard-3-1b|how do I sort a list" || v.Verdict != "unsafe" || v.Raw != "unsafe\nS6" ||
len(v.Categories) != 2 {
t.Errorf("asked %q, got %+v", asked, v)
}
if code := call(t, a, "POST", "/api/v1/guard/judges/ask", `{"text":" "}`, nil); code != 400 {
t.Errorf("no text: %d", code)
}
}
+4
View File
@@ -254,6 +254,10 @@ var Routes = []Route{
{Name: "since", Description: "How far back: 90m, 24h, 7d; 24h when absent."}, {Name: "since", Description: "How far back: 90m, 24h, 7d; 24h when absent."},
}, },
Response: GuardEventsView{}}, Response: GuardEventsView{}},
{Method: "GET", Path: "/api/v1/guard/judges", Summary: "The judge models the policies ask, and what is wrong with each",
Response: []JudgeView{}},
{Method: "POST", Path: "/api/v1/guard/judges/ask", Summary: "Have a judge read a text and say what it answered; nothing is kept",
Request: JudgeAskRequest{}, Response: JudgeAskView{}},
{Method: "POST", Path: "/api/v1/guard/attachments", Summary: "Attach a guardrail policy", {Method: "POST", Path: "/api/v1/guard/attachments", Summary: "Attach a guardrail policy",
Request: GuardAttachmentRequest{}, Response: guard.Attachment{}}, Request: GuardAttachmentRequest{}, Response: guard.Attachment{}},
{Method: "DELETE", Path: "/api/v1/guard/attachments", Summary: "Detach a guardrail policy", {Method: "DELETE", Path: "/api/v1/guard/attachments", Summary: "Detach a guardrail policy",
+13
View File
@@ -42,6 +42,11 @@ type Summary struct {
// ControlPlanes is the cluster's control planes when there is more than // ControlPlanes is the cluster's control planes when there is more than
// one: which leads, which stand by, and the floating address in front. // one: which leads, which stand by, and the floating address in front.
ControlPlanes ControlPlanes `json:"control_planes,omitzero"` ControlPlanes ControlPlanes `json:"control_planes,omitzero"`
// Judges are the guardrails' judge models with something wrong.
Judges []JudgeView `json:"judges,omitempty"`
// GuardOff says the guardrails are turned off (guard.enabled: false):
// no request is checked, whatever the policies say.
GuardOff bool `json:"guard_off,omitempty"`
} }
// StoreMember is one member of a replicated store, as its Patroni reports it. // StoreMember is one member of a replicated store, as its Patroni reports it.
@@ -149,6 +154,14 @@ const idleThreshold = 10 * time.Minute
// Summary builds the fleet summary. // Summary builds the fleet summary.
func (a *API) Summary() Summary { func (a *API) Summary() Summary {
s := Summary{Version: version.Version} s := Summary{Version: version.Version}
if a.guard != nil && a.guard.Library != nil && !a.guard.Library().On() {
s.GuardOff = true
}
for _, j := range a.Judges() {
if j.Problem != "" {
s.Judges = append(s.Judges, j)
}
}
if a.storeStatus != nil { if a.storeStatus != nil {
members, err := a.storeStatus() members, err := a.storeStatus()
s.Store = members s.Store = members
+1
View File
@@ -484,6 +484,7 @@ var Reference = []Block{
{Key: "sim", Type: "SimConfig", Doc: "", Nested: "engines.SimConfig"}, {Key: "sim", Type: "SimConfig", Doc: "", Nested: "engines.SimConfig"},
}}, }},
{Package: "guard", Name: "Config", Title: "guard.Config", Doc: "", Fields: []Field{ {Package: "guard", Name: "Config", Title: "guard.Config", Doc: "", Fields: []Field{
{Key: "enabled", Type: "*bool", Doc: "Enabled turns every policy on or off at once: false checks nothing,\nwhatever is defined or attached. Unset is on.", Nested: ""},
{Key: "default", Type: "[]string", Doc: "", Nested: ""}, {Key: "default", Type: "[]string", Doc: "", Nested: ""},
{Key: "policies", Type: "map[string]Policy", Doc: "", Nested: ""}, {Key: "policies", Type: "map[string]Policy", Doc: "", Nested: ""},
}}, }},
+39 -1
View File
@@ -5,6 +5,7 @@ import (
"encoding/json" "encoding/json"
"fmt" "fmt"
"log/slog" "log/slog"
"maps"
"regexp" "regexp"
"sort" "sort"
"strings" "strings"
@@ -43,6 +44,9 @@ type Gate struct {
m *gateMetrics m *gateMetrics
// alarm is told each time a judge gives no verdict; nil tells no one. // alarm is told each time a judge gives no verdict; nil tells no one.
alarm func(model string, err error) alarm func(model string, err error)
// lastErr is each judge's last failure to give a verdict.
errMu sync.Mutex
lastErr map[string]JudgeError
mu sync.Mutex mu sync.Mutex
attached map[string][]string // "scope/id" → policy names attached map[string][]string // "scope/id" → policy names
@@ -152,6 +156,10 @@ func (g *Gate) Invalidate() { g.reload(context.Background()) }
// from. // from.
func (g *Gate) Policy(s Subject) (Policy, string) { func (g *Gate) Policy(s Subject) (Policy, string) {
g.mu.Lock() g.mu.Lock()
if !g.cfg.On() {
g.mu.Unlock()
return Policy{}, ""
}
if !g.loading && g.store != nil && g.now().Sub(g.loadedAt) > attachmentsTTL { if !g.loading && g.store != nil && g.now().Sub(g.loadedAt) > attachmentsTTL {
g.loading = true g.loading = true
go g.reload(context.WithoutCancel(context.Background())) go g.reload(context.WithoutCancel(context.Background()))
@@ -324,7 +332,7 @@ func (g *Gate) Input(ctx context.Context, s Subject, body []byte) (Verdict, erro
done := func(f finding) Verdict { done := func(f finding) Verdict {
v.Action, v.Category, v.Rule = f.action, f.category, f.rule v.Action, v.Category, v.Rule = f.action, f.category, f.rule
if f.action == Block { if f.action == Block {
v.Message = fmt.Sprintf("this request was refused by the gateway's policy (%s)", f.category) v.Message = fmt.Sprintf("this request was refused by the gateway's policy (%s)", refusedFor(f))
v.Body = nil v.Body = nil
} }
v.Took = g.now().Sub(start) v.Took = g.now().Sub(start)
@@ -448,11 +456,30 @@ func (g *Gate) Input(ctx context.Context, s Subject, body []byte) (Verdict, erro
func (g *Gate) SetJudgeAlarm(f func(model string, err error)) { g.alarm = f } func (g *Gate) SetJudgeAlarm(f func(model string, err error)) { g.alarm = f }
func (g *Gate) judgeFailed(model string, err error) { func (g *Gate) judgeFailed(model string, err error) {
g.errMu.Lock()
if g.lastErr == nil {
g.lastErr = map[string]JudgeError{}
}
g.lastErr[model] = JudgeError{Err: err.Error(), At: g.now()}
g.errMu.Unlock()
if g.alarm != nil { if g.alarm != nil {
g.alarm(model, err) g.alarm(model, err)
} }
} }
// JudgeError is a judge's last failure to give a verdict.
type JudgeError struct {
Err string
At time.Time
}
// JudgeErrors are each judge's last failure to give a verdict, by model.
func (g *Gate) JudgeErrors() map[string]JudgeError {
g.errMu.Lock()
defer g.errMu.Unlock()
return maps.Clone(g.lastErr)
}
func judgeFinding(p Policy, action string, res judgment, err error) finding { func judgeFinding(p Policy, action string, res judgment, err error) finding {
if err != nil { if err != nil {
// Unanswered: closed stops it, open lets it through, said. // Unanswered: closed stops it, open lets it through, said.
@@ -501,3 +528,14 @@ func toolAllowed(t Tools, name string) bool {
} }
return false return false
} }
// refusedFor is what a refusal names: the category, and for a judge the
// hazard it saw — "judge: S6,Specialized Advice" — so whoever is refused
// can say why, and a judge refusing ordinary questions can be told apart.
// Never the text.
func refusedFor(f finding) string {
if f.category == "judge" && f.rule != "" && f.rule != "unavailable" {
return "judge: " + f.rule
}
return f.category
}
+31
View File
@@ -278,3 +278,34 @@ func TestCapabilitiesAndStructuredAnswersWhenAskedFor(t *testing.T) {
} }
} }
} }
func TestTheGuardrailsTurnOffAndOnAsAWhole(t *testing.T) {
off := false
cfg := lib(map[string]Policy{"base": {Input: Rules{PII: map[string]string{"secret": "block"}}}}, "base")
cfg.Enabled = &off
g := gate(t, cfg)
secret := chat("user", "my key is sk-proj-abcdefghijklmnopqrstuvwx1234")
if v, err := g.Input(context.Background(), Subject{}, secret); err != nil || v.Action != "" {
t.Errorf("off, a secret was acted on: %+v %v", v, err)
}
on := true
cfg.Enabled = &on
if err := g.Replace(cfg); err != nil {
t.Fatal(err)
}
if v, _ := g.Input(context.Background(), Subject{}, secret); v.Action != Block {
t.Errorf("on again, a secret went through: %+v", v)
}
if !(Config{}).On() {
t.Error("unset is not on")
}
}
func TestAJudgesRefusalNamesTheHazard(t *testing.T) {
if got := refusedFor(finding{category: "judge", rule: "S6,Specialized Advice"}); got != "judge: S6,Specialized Advice" {
t.Errorf("%q", got)
}
if got := refusedFor(finding{category: "secret", rule: "pii:secret"}); got != "secret" {
t.Errorf("%q", got)
}
}
+54
View File
@@ -294,6 +294,10 @@ func parseJudgment(format string, names []string, body []byte, answer string) (j
} }
switch format { switch format {
case "llamaguard3": case "llamaguard3":
if len(names) == 0 {
// Its own: S1 is Violent Crimes, S9 Indiscriminate Weapons.
names = llamaGuard3Categories
}
lines := strings.Split(text, "\n") lines := strings.Split(text, "\n")
switch strings.ToLower(strings.TrimSpace(lines[0])) { switch strings.ToLower(strings.TrimSpace(lines[0])) {
case "safe": case "safe":
@@ -369,3 +373,53 @@ func DefaultCategories(format string) []string {
} }
return nil return nil
} }
// JudgeAnswer is what a judge said of a text: its words as it gave them,
// and as the gateway reads them.
type JudgeAnswer struct {
Model string
Raw string
Unsafe bool
Categories []string
Took time.Duration
}
// AskJudge has a judge read a text, past the cache, and says what it
// answered: for an operator finding out why ordinary requests are refused.
// With answer, the judge reads it as the assistant's reply to text.
func (g *Gate) AskJudge(ctx context.Context, model, text, answer string) (JudgeAnswer, error) {
out := JudgeAnswer{Model: model}
if g.judger == nil {
return out, errNoJudge
}
format, names, ok := g.judger.format(model)
if !ok {
return out, fmt.Errorf("%w: %q is not a model with role: guard", errNoJudge, model)
}
ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
start := time.Now()
path, body := judgeRequest(model, format, names, text, answer)
reply, err := g.judger.call(ctx, model, path, body)
out.Took = time.Since(start)
if err != nil {
return out, err
}
var resp struct {
Choices []struct {
Message struct {
Content string `json:"content"`
} `json:"message"`
Text string `json:"text"`
} `json:"choices"`
}
if json.Unmarshal(reply, &resp) == nil && len(resp.Choices) > 0 {
out.Raw = strings.TrimSpace(resp.Choices[0].Message.Content + resp.Choices[0].Text)
}
res, err := parseJudgment(format, names, reply, answer)
if err != nil {
return out, err
}
out.Unsafe, out.Categories = res.unsafe, res.categories
return out, nil
}
+6
View File
@@ -25,10 +25,16 @@ import (
// Policies are attached to parts of the tenancy tree in the store; Default // Policies are attached to parts of the tenancy tree in the store; Default
// applies to every request. // applies to every request.
type Config struct { type Config struct {
// Enabled turns every policy on or off at once: false checks nothing,
// whatever is defined or attached. Unset is on.
Enabled *bool `koanf:"enabled"`
Default []string `koanf:"default"` Default []string `koanf:"default"`
Policies map[string]Policy `koanf:"policies"` Policies map[string]Policy `koanf:"policies"`
} }
// On is whether the guardrails check anything.
func (c Config) On() bool { return c.Enabled == nil || *c.Enabled }
// Policy is one named set of rules. // Policy is one named set of rules.
type Policy struct { type Policy struct {
// Mode: enforce acts, shadow only records what it would have done, off // Mode: enforce acts, shadow only records what it would have done, off
+1 -1
View File
@@ -191,7 +191,7 @@ func guardFields(u *store.Usage, in, out guard.Verdict) {
if out.Stricter(in) { if out.Stricter(in) {
v = out v = out
} }
u.GuardAction, u.GuardPolicy, u.GuardCategory = string(v.Action), v.Policy, v.Category u.GuardAction, u.GuardPolicy, u.GuardCategory, u.GuardRule = string(v.Action), v.Policy, v.Category, v.Rule
if v.Shadow && v.Action != guard.Allow { if v.Shadow && v.Action != guard.Allow {
u.GuardAction = "shadow-" + u.GuardAction u.GuardAction = "shadow-" + u.GuardAction
} }
+2 -1
View File
@@ -34,7 +34,7 @@ func (g *wordGuard) Input(_ context.Context, s guard.Subject, body []byte) (guar
case g.err != nil: case g.err != nil:
return guard.Verdict{}, g.err return guard.Verdict{}, g.err
case bytes.Contains(body, []byte("forbidden")): case bytes.Contains(body, []byte("forbidden")):
return guard.Verdict{Action: guard.Block, Policy: "words", Category: "test"}, nil return guard.Verdict{Action: guard.Block, Policy: "words", Category: "test", Rule: "S6,Specialized Advice"}, nil
case bytes.Contains(body, []byte("alice@example.com")): case bytes.Contains(body, []byte("alice@example.com")):
return guard.Verdict{Action: guard.Redact, Policy: "pii", Category: "email", return guard.Verdict{Action: guard.Redact, Policy: "pii", Category: "email",
Body: bytes.ReplaceAll(body, []byte("alice@example.com"), []byte("[EMAIL_1]"))}, nil Body: bytes.ReplaceAll(body, []byte("alice@example.com"), []byte("[EMAIL_1]"))}, nil
@@ -157,6 +157,7 @@ func TestABlockedRequestIsRecordedWithItsVerdictAndCostsNothing(t *testing.T) {
} }
if ok { if ok {
if u.GuardAction != "blocked" || u.GuardPolicy != "words" || u.GuardCategory != "test" || if u.GuardAction != "blocked" || u.GuardPolicy != "words" || u.GuardCategory != "test" ||
u.GuardRule != "S6,Specialized Advice" ||
u.CostMicros != 0 || u.PromptTokens != 0 || u.Status != http.StatusBadRequest { u.CostMicros != 0 || u.PromptTokens != 0 || u.Status != http.StatusBadRequest {
t.Errorf("recorded %+v", u) t.Errorf("recorded %+v", u)
} }
+4 -2
View File
@@ -1217,9 +1217,11 @@ func (r *Router) accessLog(req *parsedRequest, res dispatchResult, status int, t
} }
// What a guard did, by name: never what it matched. // What a guard did, by name: never what it matched.
if v := req.guardIn; res.guard.Stricter(v) { if v := req.guardIn; res.guard.Stricter(v) {
attrs = append(attrs, "guard", string(res.guard.Action), "guard_stage", "output", "guard_policy", res.guard.Policy) attrs = append(attrs, "guard", string(res.guard.Action), "guard_stage", "output", "guard_policy", res.guard.Policy,
"guard_category", res.guard.Category, "guard_rule", res.guard.Rule)
} else if v.Action != "" { } else if v.Action != "" {
attrs = append(attrs, "guard", string(v.Action), "guard_stage", "input", "guard_policy", v.Policy) attrs = append(attrs, "guard", string(v.Action), "guard_stage", "input", "guard_policy", v.Policy,
"guard_category", v.Category, "guard_rule", v.Rule)
} }
if res.outcome.Stopped { if res.outcome.Stopped {
attrs = append(attrs, "stopped_by_guard", true) attrs = append(attrs, "stopped_by_guard", true)
+3 -2
View File
@@ -50,6 +50,7 @@ type GuardEvent struct {
Action string Action string
Policy string Policy string
Category string Category string
Rule string
GuardMs int64 GuardMs int64
} }
@@ -57,14 +58,14 @@ type GuardEvent struct {
// flagged, redacted, blocked, or would have in shadow — since a moment. // flagged, redacted, blocked, or would have in shadow — since a moment.
func (s *DB) GuardEvents(ctx context.Context, since time.Time, limit int) ([]GuardEvent, error) { func (s *DB) GuardEvents(ctx context.Context, since time.Time, limit int) ([]GuardEvent, error) {
return queryAll(ctx, s.db, `SELECT ts, request_id, key_id, COALESCE(user_id, ''), project_id, model, status, return queryAll(ctx, s.db, `SELECT ts, request_id, key_id, COALESCE(user_id, ''), project_id, model, status,
guard_action, guard_policy, guard_category, guard_ms guard_action, guard_policy, guard_category, guard_rule, guard_ms
FROM usage WHERE ts >= ? AND guard_action <> '' ORDER BY ts DESC LIMIT ?`, FROM usage WHERE ts >= ? AND guard_action <> '' ORDER BY ts DESC LIMIT ?`,
[]any{millis(since), limit}, []any{millis(since), limit},
func(rows *sql.Rows) (GuardEvent, error) { func(rows *sql.Rows) (GuardEvent, error) {
var e GuardEvent var e GuardEvent
var ts int64 var ts int64
err := rows.Scan(&ts, &e.RequestID, &e.KeyID, &e.UserID, &e.ProjectID, &e.Model, &e.Status, err := rows.Scan(&ts, &e.RequestID, &e.KeyID, &e.UserID, &e.ProjectID, &e.Model, &e.Status,
&e.Action, &e.Policy, &e.Category, &e.GuardMs) &e.Action, &e.Policy, &e.Category, &e.Rule, &e.GuardMs)
e.At = fromMillis(ts) e.At = fromMillis(ts)
return e, err return e, err
}) })
@@ -0,0 +1,2 @@
-- Ported from the SQLite migration of the same number.
ALTER TABLE usage ADD COLUMN guard_rule TEXT NOT NULL DEFAULT '';
@@ -0,0 +1,5 @@
-- What in a guard policy decided a request, beside its category: the
-- pattern's name, or the judge's verdict — "S6,Specialized Advice" — so a
-- request a judge refused says which hazard it saw, and a judge that
-- refuses ordinary questions can be told from one that is right to.
ALTER TABLE usage ADD COLUMN guard_rule TEXT NOT NULL DEFAULT '';
+5 -5
View File
@@ -435,8 +435,8 @@ func (s *DB) WriteUsage(ctx context.Context, records []Usage) error {
`INSERT INTO usage (ts, key_id, project_id, model, served_model, `INSERT INTO usage (ts, key_id, project_id, model, served_model,
prompt_tokens, completion_tokens, cost_micros, status, latency_ms, user_id, prompt_tokens, completion_tokens, cost_micros, status, latency_ms, user_id,
request_id, replica, queue_ms, ttft_ms, request_id, replica, queue_ms, ttft_ms,
guard_action, guard_policy, guard_category, guard_ms, end_user, tags, cache_hit) guard_action, guard_policy, guard_category, guard_rule, guard_ms, end_user, tags, cache_hit)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`)
if err != nil { if err != nil {
return err return err
} }
@@ -450,7 +450,7 @@ func (s *DB) WriteUsage(ctx context.Context, records []Usage) error {
r.Model, r.ServedModel, r.PromptTokens, r.CompletionTokens, r.Model, r.ServedModel, r.PromptTokens, r.CompletionTokens,
r.CostMicros, r.Status, r.LatencyMs, r.UserID, r.CostMicros, r.Status, r.LatencyMs, r.UserID,
r.RequestID, r.Replica, r.QueueMs, r.TTFTMs, r.RequestID, r.Replica, r.QueueMs, r.TTFTMs,
r.GuardAction, r.GuardPolicy, r.GuardCategory, r.GuardMs, r.EndUser, tagsJSON(r.Tags), hit); err != nil { r.GuardAction, r.GuardPolicy, r.GuardCategory, r.GuardRule, r.GuardMs, r.EndUser, tagsJSON(r.Tags), hit); err != nil {
return err return err
} }
} }
@@ -864,11 +864,11 @@ func (s *DB) UsageByRequest(ctx context.Context, requestID string) (Usage, bool,
err := s.db.QueryRowContext(ctx, err := s.db.QueryRowContext(ctx,
`SELECT ts, key_id, project_id, COALESCE(user_id, ''), model, served_model, prompt_tokens, `SELECT ts, key_id, project_id, COALESCE(user_id, ''), model, served_model, prompt_tokens,
completion_tokens, cost_micros, status, latency_ms, request_id, replica, queue_ms, ttft_ms, completion_tokens, cost_micros, status, latency_ms, request_id, replica, queue_ms, ttft_ms,
guard_action, guard_policy, guard_category, guard_ms, end_user, tags, cache_hit guard_action, guard_policy, guard_category, guard_rule, guard_ms, end_user, tags, cache_hit
FROM usage WHERE request_id = ? ORDER BY ts DESC LIMIT 1`, requestID). FROM usage WHERE request_id = ? ORDER BY ts DESC LIMIT 1`, requestID).
Scan(&ts, &u.KeyID, &u.ProjectID, &u.UserID, &u.Model, &u.ServedModel, &u.PromptTokens, Scan(&ts, &u.KeyID, &u.ProjectID, &u.UserID, &u.Model, &u.ServedModel, &u.PromptTokens,
&u.CompletionTokens, &u.CostMicros, &u.Status, &u.LatencyMs, &u.RequestID, &u.Replica, &u.CompletionTokens, &u.CostMicros, &u.Status, &u.LatencyMs, &u.RequestID, &u.Replica,
&u.QueueMs, &u.TTFTMs, &u.GuardAction, &u.GuardPolicy, &u.GuardCategory, &u.GuardMs, &u.QueueMs, &u.TTFTMs, &u.GuardAction, &u.GuardPolicy, &u.GuardCategory, &u.GuardRule, &u.GuardMs,
&u.EndUser, &tags, &hit) &u.EndUser, &tags, &hit)
if errors.Is(err, sql.ErrNoRows) { if errors.Is(err, sql.ErrNoRows) {
return Usage{}, false, nil return Usage{}, false, nil
+3 -1
View File
@@ -257,11 +257,13 @@ type Usage struct {
// GuardAction is what a guard did — flagged, redacted, blocked — empty // GuardAction is what a guard did — flagged, redacted, blocked — empty
// when it let the request through untouched or there is none; // when it let the request through untouched or there is none;
// GuardPolicy and GuardCategory what decided it, GuardMs how long // GuardPolicy, GuardCategory and GuardRule what decided it — a judge's
// rule is its verdict, "S6,Specialized Advice" — GuardMs how long
// judging took. Never the text. // judging took. Never the text.
GuardAction string GuardAction string
GuardPolicy string GuardPolicy string
GuardCategory string GuardCategory string
GuardRule string
GuardMs int64 GuardMs int64
// EndUser is OpenAI's user field as the client sent it: its own name // EndUser is OpenAI's user field as the client sent it: its own name
+4 -2
View File
@@ -88,8 +88,10 @@ type AccountPages struct {
oidc *oidcLogin oidc *oidcLogin
// audit records every change made through these pages; nil records none. // audit records every change made through these pages; nil records none.
audit audit.Sink audit audit.Sink
// hasGuard is set when /admin/guard is registered. // hasGuard is set when /admin/guard is registered, and guardLibrary
hasGuard bool // with it: the policies in force, whose judges the mirror page serves.
hasGuard bool
guardLibrary func() guard.Config
// guardFor names the guardrail policies in force for someone, and // guardFor names the guardrail policies in force for someone, and
// whether they keep nothing; nil when there are none. // whether they keep nothing; nil when there are none.
guardFor func(guard.Subject) (string, bool) guardFor func(guard.Subject) (string, bool)
+11 -8
View File
@@ -24,8 +24,10 @@ type guardPageData struct {
Window string Window string
Windows []string Windows []string
// Policies are the library, Default those applied to every request. // Policies are the library, Default those applied to every request.
Policies []string Policies []string
Default []string Default []string
// Off is the guardrails turned off (guard.enabled: false).
Off bool
Attachments []guardAttachmentRow Attachments []guardAttachmentRow
Counts []guardCountRow Counts []guardCountRow
Events []guardEventRow Events []guardEventRow
@@ -40,9 +42,9 @@ type guardCountRow struct {
} }
type guardEventRow struct { type guardEventRow struct {
At, RequestID, Where, Model, Action, Policy, Category string At, RequestID, Where, Model, Action, Policy, Category, Rule string
Status int Status int
Ms int64 Ms int64
} }
var guardWindows = map[string]time.Duration{"24h": 24 * time.Hour, "7d": 7 * 24 * time.Hour, "30d": 30 * 24 * time.Hour} var guardWindows = map[string]time.Duration{"24h": 24 * time.Hour, "7d": 7 * 24 * time.Hour, "30d": 30 * 24 * time.Hour}
@@ -50,12 +52,13 @@ var guardWindows = map[string]time.Duration{"24h": 24 * time.Hour, "7d": 7 * 24
// RegisterAdminGuard mounts /admin/guard: what the guardrails did, where, by // RegisterAdminGuard mounts /admin/guard: what the guardrails did, where, by
// which policy — names and counts, never text. // which policy — names and counts, never text.
func (p *AccountPages) RegisterAdminGuard(mux *http.ServeMux, s GuardAdminSource, library func() guard.Config) { func (p *AccountPages) RegisterAdminGuard(mux *http.ServeMux, s GuardAdminSource, library func() guard.Config) {
p.hasGuard = true p.hasGuard, p.guardLibrary = true, library
mux.HandleFunc("GET /admin/guard", p.adminOnly(func(w http.ResponseWriter, r *http.Request, admin store.User) { mux.HandleFunc("GET /admin/guard", p.adminOnly(func(w http.ResponseWriter, r *http.Request, admin store.User) {
ctx, l := r.Context(), langOf(r) ctx, l := r.Context(), langOf(r)
lib := library() lib := library()
off := !lib.On()
d := guardPageData{Nav: p.navFor(r, admin, "guard"), Version: p.version, Window: r.URL.Query().Get("window"), d := guardPageData{Nav: p.navFor(r, admin, "guard"), Version: p.version, Window: r.URL.Query().Get("window"),
Windows: []string{"24h", "7d", "30d"}, Default: lib.Default} Windows: []string{"24h", "7d", "30d"}, Default: lib.Default, Off: off}
since, ok := guardWindows[d.Window] since, ok := guardWindows[d.Window]
if !ok { if !ok {
d.Window, since = "7d", guardWindows["7d"] d.Window, since = "7d", guardWindows["7d"]
@@ -104,7 +107,7 @@ func (p *AccountPages) RegisterAdminGuard(mux *http.ServeMux, s GuardAdminSource
at := e.At.UTC() at := e.At.UTC()
d.Events = append(d.Events, guardEventRow{At: l.Date(at) + " " + at.Format("15:04:05"), d.Events = append(d.Events, guardEventRow{At: l.Date(at) + " " + at.Format("15:04:05"),
RequestID: e.RequestID, Where: name(e.ProjectID), Model: e.Model, Action: guardActionWord(l, e.Action), RequestID: e.RequestID, Where: name(e.ProjectID), Model: e.Model, Action: guardActionWord(l, e.Action),
Policy: e.Policy, Category: e.Category, Status: e.Status, Ms: e.GuardMs}) Policy: e.Policy, Category: e.Category, Rule: e.Rule, Status: e.Status, Ms: e.GuardMs})
} }
noStore(w) noStore(w)
render(w, r, "admin_guard", d) render(w, r, "admin_guard", d)
+15 -4
View File
@@ -324,10 +324,14 @@ func (p *AccountPages) RegisterAdminMirror(mux *http.ServeMux, d MirrorDeps) {
if err != nil { if err != nil {
return "", err return "", err
} }
// A judge's weights are served as a judge, under the name the
// policies ask: served as a model clients use, the policies
// never find it and refuse every request they judge.
judge, renamed := p.asJudge(&model)
// The CPU while GPUs sit free is a choice to make on purpose: // The CPU while GPUs sit free is a choice to make on purpose:
// made by default, it was a model running at a tenth of the // made by default, it was a model running at a tenth of the
// speed with nothing saying why. // speed with nothing saying why. A judge is meant for the CPU.
if model.Profiles[0].CPUOnly() && r.FormValue("cpu_ok") == "" { if !judge && model.Profiles[0].CPUOnly() && r.FormValue("cpu_ok") == "" {
if n := freeGPUs(p.fleet); n > 0 { if n := freeGPUs(p.fleet); n > 0 {
return "", errors.New(l.N("serve.err.cpu_with_free_gpus", int64(n), model.Name)) return "", errors.New(l.N("serve.err.cpu_with_free_gpus", int64(n), model.Name))
} }
@@ -346,8 +350,15 @@ func (p *AccountPages) RegisterAdminMirror(mux *http.ServeMux, d MirrorDeps) {
} }
// Placed on the next pass, not now: saying it is serving would // Placed on the next pass, not now: saying it is serving would
// be wrong for however long the weights take to reach a node. // be wrong for however long the weights take to reach a node.
return l.T("serve.msg.added", model.Name, msg := l.T("serve.msg.added", model.Name,
l.N("serve.replicas_n", int64(model.Scaling.Min)), describeServe(l, model)), nil l.N("serve.replicas_n", int64(model.Scaling.Min)), describeServe(l, model))
switch {
case renamed != "":
msg += " " + l.T("serve.msg.judge_renamed", renamed, model.Name)
case judge:
msg += " " + l.T("serve.msg.judge", model.Name)
}
return msg, nil
})) }))
} }
+21
View File
@@ -4,6 +4,7 @@ import (
"context" "context"
"errors" "errors"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog" "gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
"io" "io"
"log/slog" "log/slog"
"net/http" "net/http"
@@ -745,3 +746,23 @@ func TestASafetensorsSetIsFetchedWithWhatTheEngineReadsBesideIt(t *testing.T) {
t.Errorf("the companions are not said; got %s", body) t.Errorf("the companions are not said; got %s", body)
} }
} }
func TestAJudgesWeightsAreServedAsTheJudgeThePoliciesAsk(t *testing.T) {
p := &AccountPages{guardLibrary: func() guard.Config {
return guard.Config{Policies: map[string]guard.Policy{
"baseline": {Judge: &guard.JudgePolicy{Model: "llama-guard-3-1b"}},
}}
}}
m := catalog.Model{Name: "llama-guard", Source: "http://10.0.0.146:8081/llama-guard/x/Llama-Guard-3-1B.Q4_K_M.gguf",
RoutingProfile: "chat"}
judge, renamed := p.asJudge(&m)
if !judge || renamed != "llama-guard" || m.Name != "llama-guard-3-1b" || !m.IsGuard() ||
m.Guard.Format != "llamaguard3" || m.RoutingProfile != "" {
t.Errorf("judge %v renamed %q: %+v", judge, renamed, m)
}
plain := catalog.Model{Name: "qwen", Source: "http://m/qwen.gguf", RoutingProfile: "chat"}
if judge, _ := p.asJudge(&plain); judge || plain.IsGuard() || plain.Name != "qwen" {
t.Errorf("a model clients use became a judge: %+v", plain)
}
}
+33
View File
@@ -585,3 +585,36 @@ func argInt(args []string, flag string) int {
} }
return 0 return 0
} }
// isJudgeWeights is whether a source is a guard model's weights: Llama
// Guard, which reads a conversation and says whether it is safe.
func isJudgeWeights(source string) bool {
return strings.Contains(strings.ToLower(source), "llama-guard")
}
// asJudge makes a model built from the serve form a judge when its weights
// are one's: role guard, Llama Guard's format, under the name the policies
// give their judge when they give one and the form gave another. It says
// whether it did, and the name the form gave when it was changed.
func (p *AccountPages) asJudge(m *catalog.Model) (judge bool, renamed string) {
if !isJudgeWeights(m.Source) {
return false, ""
}
m.Role, m.Guard, m.RoutingProfile = catalog.RoleGuard, &catalog.GuardModel{Format: "llamaguard3"}, ""
if p.guardLibrary == nil {
return true, ""
}
asked := map[string]bool{}
for _, pol := range p.guardLibrary().Policies {
if pol.Judge != nil && pol.Judge.Model != "" {
asked[pol.Judge.Model] = true
}
}
if asked[m.Name] || len(asked) != 1 {
return true, ""
}
for name := range asked {
renamed, m.Name = m.Name, name
}
return true, renamed
}
+2 -1
View File
@@ -32,5 +32,6 @@
"guard.col.request": "Request", "guard.col.request": "Request",
"guard.col.model": "Model", "guard.col.model": "Model",
"guard.col.status": "Status", "guard.col.status": "Status",
"guard.request_hint": "<code>llmgw ctl request ID</code> says what else a request went through." "guard.request_hint": "<code>llmgw ctl request ID</code> says what else a request went through.",
"guard.off": "The guardrails are <strong>off</strong>: no request is checked, whatever the policies below say. <code>llmgw ctl guard on</code> turns them back on."
} }
+3 -1
View File
@@ -77,5 +77,7 @@
"serve.err.cpu_with_free_gpus.other": "this fleet has %d GPUs free and %s was set to run on the CPU; choose a GPU placement, or tick \"run on the CPU anyway\"", "serve.err.cpu_with_free_gpus.other": "this fleet has %d GPUs free and %s was set to run on the CPU; choose a GPU placement, or tick \"run on the CPU anyway\"",
"serve.msg.added": "%s added to the catalog, %s: %s; it is placed on the next pass", "serve.msg.added": "%s added to the catalog, %s: %s; it is placed on the next pass",
"serve.replicas_n.one": "%d replica", "serve.replicas_n.one": "%d replica",
"serve.replicas_n.other": "%d replicas" "serve.replicas_n.other": "%d replicas",
"serve.msg.judge": "It is the guardrails' judge (role: guard): policies naming %s ask it; clients do not see it.",
"serve.msg.judge_renamed": "Its weights are a judge's, so it is the guardrails' judge (role: guard), named %[2]s as the policies ask rather than %[1]s."
} }
+2 -1
View File
@@ -32,5 +32,6 @@
"guard.col.request": "İstek", "guard.col.request": "İstek",
"guard.col.model": "Model", "guard.col.model": "Model",
"guard.col.status": "Durum", "guard.col.status": "Durum",
"guard.request_hint": "Bir isteğin başka nelerden geçtiğini <code>llmgw ctl request ID</code> söyler." "guard.request_hint": "Bir isteğin başka nelerden geçtiğini <code>llmgw ctl request ID</code> söyler.",
"guard.off": "Guardrail'ler <strong>kapalı</strong>: aşağıdaki politikalar ne derse desin hiçbir istek kontrol edilmiyor. <code>llmgw ctl guard on</code> yeniden açar."
} }
+3 -1
View File
@@ -74,5 +74,7 @@
"serve.err.placement": "%q geçerli bir yerleşim değil", "serve.err.placement": "%q geçerli bir yerleşim değil",
"serve.err.cpu_with_free_gpus.other": "bu kümede %d GPU boşta, %s ise CPU üzerinde çalışacak biçimde ayarlandı; bir GPU yerleşimi seçin ya da \"yine de CPU üzerinde çalıştır\" kutusunu işaretleyin", "serve.err.cpu_with_free_gpus.other": "bu kümede %d GPU boşta, %s ise CPU üzerinde çalışacak biçimde ayarlandı; bir GPU yerleşimi seçin ya da \"yine de CPU üzerinde çalıştır\" kutusunu işaretleyin",
"serve.msg.added": "%s kataloğa eklendi, %s: %s; bir sonraki geçişte yerleştirilecek", "serve.msg.added": "%s kataloğa eklendi, %s: %s; bir sonraki geçişte yerleştirilecek",
"serve.replicas_n.other": "%d replika" "serve.replicas_n.other": "%d replika",
"serve.msg.judge": "Guardrail'lerin judge modeli olarak eklendi (role: guard): %s adını veren politikalar ona soruyor; istemciler görmüyor.",
"serve.msg.judge_renamed": "Ağırlıkları bir judge modelinin olduğu için guardrail judge'ı olarak eklendi (role: guard); adı %[1]s yerine politikaların istediği gibi %[2]s."
} }
+2 -1
View File
@@ -9,6 +9,7 @@
{{template "nav" .Nav}} {{template "nav" .Nav}}
</header> </header>
{{if .Error}}<p class="error" role="alert">{{.Error}}</p>{{end}} {{if .Error}}<p class="error" role="alert">{{.Error}}</p>{{end}}
{{if .Off}}<p class="error" role="alert">{{th "guard.off"}}</p>{{end}}
<h2>{{t "guard.policies"}}</h2> <h2>{{t "guard.policies"}}</h2>
{{if .Policies}} {{if .Policies}}
@@ -39,7 +40,7 @@
<table class="rows"> <table class="rows">
<tr><th>{{t "guard.col.when"}}</th><th>{{t "guard.col.request"}}</th><th>{{t "guard.col.where"}}</th><th>{{t "guard.col.model"}}</th><th>{{t "guard.col.action"}}</th><th>{{t "guard.col.policy"}}</th><th>{{t "guard.col.category"}}</th><th class="num">{{t "guard.col.status"}}</th><th class="num">ms</th></tr> <tr><th>{{t "guard.col.when"}}</th><th>{{t "guard.col.request"}}</th><th>{{t "guard.col.where"}}</th><th>{{t "guard.col.model"}}</th><th>{{t "guard.col.action"}}</th><th>{{t "guard.col.policy"}}</th><th>{{t "guard.col.category"}}</th><th class="num">{{t "guard.col.status"}}</th><th class="num">ms</th></tr>
{{range .Events}}<tr><td>{{.At}}</td><td><code>{{.RequestID}}</code></td><td>{{.Where}}</td><td>{{.Model}}</td> {{range .Events}}<tr><td>{{.At}}</td><td><code>{{.RequestID}}</code></td><td>{{.Where}}</td><td>{{.Model}}</td>
<td>{{.Action}}</td><td><code>{{.Policy}}</code></td><td>{{.Category}}</td><td class="num">{{.Status}}</td><td class="num">{{count .Ms}}</td></tr>{{end}} <td>{{.Action}}</td><td><code>{{.Policy}}</code></td><td>{{.Category}}{{if .Rule}} · {{.Rule}}{{end}}</td><td class="num">{{.Status}}</td><td class="num">{{count .Ms}}</td></tr>{{end}}
</table> </table>
<p class="sub">{{th "guard.request_hint"}}</p> <p class="sub">{{th "guard.request_hint"}}</p>
{{end}} {{end}}