diff --git a/CHANGELOG.md b/CHANGELOG.md
index ff6d00be..d9dc987d 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -22,6 +22,32 @@ is only released once its **exit criterion** is met, not when its code lands.
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
APIs change between minors without a deprecation period.
+## Unreleased
+
+### Added
+
+- **`llmgw ctl guard judge status` and `enable`**: each judge model the
+ policies ask, whether it is in the catalog, as a judge, with replicas
+ ready and answering — and serving the missing one, Llama Guard 3 1B on
+ CPUs under the name the policies give it, with `--replace` for a model of
+ that name that is not a judge. No more `server init` to put a judge back.
+ Also **Do → Serve the guardrails' judge** in the console.
+- **A missing or broken judge is said where it is looked for**: `llmgw ctl
+ status` and the console's Overview list each judge with a problem — not in
+ the catalog, not a judge, no replica ready, or failing to answer — and
+ what that does to requests (refused or unjudged).
+
+### Fixed
+
+- **A Llama Guard file served from the mirror page is the judge**: it was
+ added as a model clients use, under whatever name the form had, so the
+ policies never found it and refused everything they judged
+ (`guard_unavailable`). It is now added as a judge, under the name the
+ policies ask.
+- **`ctl` says so when the API answers with a page**: a command a control
+ plane does not know — the leader on an older release, through a standby —
+ failed with "invalid character '<'"; it now says which and why.
+
## v0.12.0 — 2026-10-06
A minor version, because where the configuration lives changes: everything in
diff --git a/docs/admin-api.openapi.json b/docs/admin-api.openapi.json
index 2b38790c..4007b34d 100644
--- a/docs/admin-api.openapi.json
+++ b/docs/admin-api.openapi.json
@@ -894,6 +894,36 @@
}
}
},
+ "/api/v1/guard/judges": {
+ "get": {
+ "operationId": "getGuardJudges",
+ "summary": "The judge models the policies ask, and what is wrong with each",
+ "tags": [
+ "guard"
+ ],
+ "responses": {
+ "200": {
+ "description": "OK",
+ "content": {
+ "application/json": {
+ "schema": {
+ "type": "array",
+ "items": {
+ "$ref": "#/components/schemas/JudgeView"
+ }
+ }
+ }
+ }
+ },
+ "401": {
+ "$ref": "#/components/responses/Unauthorized"
+ },
+ "default": {
+ "$ref": "#/components/responses/Error"
+ }
+ }
+ }
+ },
"/api/v1/jobs": {
"get": {
"operationId": "getJobs",
@@ -4359,6 +4389,52 @@
}
}
},
+ "JudgeView": {
+ "type": "object",
+ "description": "JudgeView is one judge model the guardrail policies ask, and whether it can answer them.",
+ "properties": {
+ "model": {
+ "type": "string"
+ },
+ "policies": {
+ "type": "array",
+ "items": {
+ "type": "string"
+ },
+ "description": "Policies are those that ask it; Closed those that refuse what it cannot read (fail: closed)."
+ },
+ "closed": {
+ "type": "array",
+ "items": {
+ "type": "string"
+ }
+ },
+ "in_catalog": {
+ "type": "boolean",
+ "description": "InCatalog is whether a model of that name is in the catalog, and Judge whether it is there as a judge (role: guard)."
+ },
+ "judge": {
+ "type": "boolean"
+ },
+ "ready_replicas": {
+ "type": "integer"
+ },
+ "wanted_replicas": {
+ "type": "integer"
+ },
+ "last_error": {
+ "type": "string",
+ "description": "LastError is its last failure to give a verdict, if any."
+ },
+ "last_error_at": {
+ "type": "string"
+ },
+ "problem": {
+ "type": "string",
+ "description": "Problem says, in words, what is wrong and what to do; empty when nothing is."
+ }
+ }
+ },
"Live": {
"type": "object",
"description": "Live is one of the cluster's adapters as its node last reported it.",
@@ -5696,6 +5772,13 @@
"control_planes": {
"$ref": "#/components/schemas/ControlPlanes",
"description": "ControlPlanes is the cluster's control planes when there is more than one: which leads, which stand by, and the floating address in front."
+ },
+ "judges": {
+ "type": "array",
+ "items": {
+ "$ref": "#/components/schemas/JudgeView"
+ },
+ "description": "Judges are the guardrails' judge models with something wrong."
}
}
},
diff --git a/docs/guardrails.md b/docs/guardrails.md
index ad5838df..b05b7fc3 100644
--- a/docs/guardrails.md
+++ b/docs/guardrails.md
@@ -103,7 +103,23 @@ guard:
Patterns find what has a shape. For what does not — violence, self-harm,
abuse, whatever a moderation model was trained on — a policy can have a
-**judge model on the cluster** read the text:
+**judge model on the cluster** read the text.
+
+The judge must be in the catalog under the name the policy gives, as a
+judge (`role: guard`), with a replica ready. When it is not, what the
+policy judges is refused with `guard_unavailable` (`fail: closed`) or served
+unjudged (`fail: open`), and `llmgw ctl status` and the console's Overview
+say which judge and why. To serve it:
+
+```
+llmgw ctl guard judge status # each judge the policies ask, and what is wrong with it
+llmgw ctl guard judge enable # Llama Guard 3 1B on CPUs, under the name the policies ask
+llmgw ctl guard judge enable --replace # when a model of that name is there, but not as a judge
+```
+
+or **Do → Serve the guardrails' judge** in the console. A Llama Guard file
+served from the mirror page is served as a judge too, under the name the
+policies ask. By hand, the catalog entry is:
```yaml
# a catalog model (server.yaml models:, or ctl model add)
diff --git a/internal/cli/ctl.go b/internal/cli/ctl.go
index fe35a0c7..10c25643 100644
--- a/internal/cli/ctl.go
+++ b/internal/cli/ctl.go
@@ -116,6 +116,13 @@ func (c *ctlClient) do(ctx context.Context, method, path string, body, out any)
if out == nil {
return nil
}
+ // An older control plane answers a path it does not know with its
+ // dashboard: a page, which read as JSON said only "invalid character '<'".
+ if strings.HasPrefix(resp.Header.Get("Content-Type"), "text/html") {
+ return fmt.Errorf("%s %s answered with a page, not the API: the control plane answering "+
+ "(the leader, through a standby) does not know it — is it on an older release? "+
+ "llmgw ctl status says which it runs", method, path)
+ }
return json.NewDecoder(resp.Body).Decode(out)
}
diff --git a/internal/cli/ctl_config_test.go b/internal/cli/ctl_config_test.go
new file mode 100644
index 00000000..6f5afdb0
--- /dev/null
+++ b/internal/cli/ctl_config_test.go
@@ -0,0 +1,22 @@
+package cli
+
+import (
+ "net/http"
+ "net/http/httptest"
+ "strings"
+ "testing"
+)
+
+func TestAPageInsteadOfTheAPIIsSaidPlainly(t *testing.T) {
+ srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+ w.Header().Set("Content-Type", "text/html; charset=utf-8")
+ _, _ = w.Write([]byte("llmgw"))
+ }))
+ defer srv.Close()
+ c := newCtlClient(strings.TrimPrefix(srv.URL, "http://"), "")
+ var out map[string]any
+ err := c.get(t.Context(), "/api/v1/settings", &out)
+ if err == nil || !strings.Contains(err.Error(), "older release") || strings.Contains(err.Error(), "invalid character") {
+ t.Errorf("%v", err)
+ }
+}
diff --git a/internal/cli/guard.go b/internal/cli/guard.go
index d99ab6ca..55f16e40 100644
--- a/internal/cli/guard.go
+++ b/internal/cli/guard.go
@@ -24,6 +24,7 @@ func newGuardCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
"Targets are written org:Acme, team:Acme/ml, project:Acme/ml/training (or\n" +
"with IDs), key:KEY-ID or user:USER-ID. An attachment applies at once.",
}
+ c.AddCommand(newGuardJudgeCmd(client, asJSON))
c.AddCommand(&cobra.Command{
Use: "list",
Short: "The policies, and where each is attached",
diff --git a/internal/cli/guard_judge.go b/internal/cli/guard_judge.go
new file mode 100644
index 00000000..c59a9880
--- /dev/null
+++ b/internal/cli/guard_judge.go
@@ -0,0 +1,189 @@
+package cli
+
+import (
+ "encoding/json"
+ "errors"
+ "fmt"
+ "strings"
+ "time"
+
+ "github.com/spf13/cobra"
+
+ "gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+ "gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
+)
+
+// The judge the guardrails are installed with: Meta's Llama Guard 3 1B,
+// from the repository server init fetches it from.
+const (
+ guardModelRepo = "QuantFactory/Llama-Guard-3-1B-GGUF"
+ guardModelFile = "Llama-Guard-3-1B.Q4_K_M.gguf"
+)
+
+func newGuardJudgeCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
+ c := &cobra.Command{
+ Use: "judge",
+ Short: "The judge models the policies ask: whether each can answer, and serving one",
+ Long: "A policy with a judge has a model read what patterns cannot. The judge must\n" +
+ "be in the catalog under the name the policy gives, as a judge (role:\n" +
+ "guard), with a replica ready; otherwise what the policy judges is refused\n" +
+ "(fail: closed, guard_unavailable) or served unjudged (fail: open).",
+ }
+ c.AddCommand(newGuardJudgeStatusCmd(client, asJSON), newGuardJudgeEnableCmd(client))
+ return c
+}
+
+func newGuardJudgeStatusCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
+ return &cobra.Command{
+ Use: "status",
+ Short: "Each judge the policies ask, and what is wrong with it",
+ Args: cobra.NoArgs,
+ RunE: func(cmd *cobra.Command, _ []string) error {
+ var js []adminapi.JudgeView
+ if err := client().get(cmd.Context(), "/api/v1/guard/judges", &js); err != nil {
+ return err
+ }
+ if *asJSON {
+ return emitJSON(js)
+ }
+ out := cmd.OutOrStdout()
+ if len(js) == 0 {
+ fmt.Fprintln(out, "no policy asks a judge")
+ return nil
+ }
+ w := tabTo(out)
+ fmt.Fprintln(w, "JUDGE\tPOLICIES\tIN CATALOG\tAS JUDGE\tREADY\tSTATE")
+ for _, j := range js {
+ state := "answering"
+ if j.Problem != "" {
+ state = j.Problem
+ }
+ fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%d/%d\t%s\n", j.Model, strings.Join(j.Policies, ","),
+ yesNo(j.InCatalog), yesNo(j.Judge), j.Ready, j.Wanted, state)
+ }
+ if err := w.Flush(); err != nil {
+ return err
+ }
+ strays, _ := strayJudges(cmd, client, js)
+ for _, s := range strays {
+ fmt.Fprintf(out, "\n%s looks like a judge's weights but is served as a model clients use, and no "+
+ "policy asks it: llmgw ctl model remove %s\n", s, s)
+ }
+ return nil
+ },
+ }
+}
+
+// strayJudges are models whose weights are a judge's but which are in the
+// catalog as models clients use, under a name no policy asks: what serving
+// Llama Guard from the mirror page did before it knew better.
+func strayJudges(cmd *cobra.Command, client func() *ctlClient, js []adminapi.JudgeView) ([]string, error) {
+ var models []adminapi.ModelView
+ if err := client().get(cmd.Context(), "/api/v1/models", &models); err != nil {
+ return nil, err
+ }
+ asked := map[string]bool{}
+ for _, j := range js {
+ asked[j.Model] = true
+ }
+ var out []string
+ for _, m := range models {
+ if !asked[m.Name] && strings.Contains(strings.ToLower(m.Source), "llama-guard") {
+ out = append(out, m.Name)
+ }
+ }
+ return out, nil
+}
+
+func newGuardJudgeEnableCmd(client func() *ctlClient) *cobra.Command {
+ var (
+ model, repo, file string
+ replicas, cores int
+ replace, detach bool
+ )
+ c := &cobra.Command{
+ Use: "enable",
+ Short: "Serve the judge the policies ask, as a judge: Llama Guard 3 1B on CPUs",
+ Long: "Fetches Llama Guard 3 1B into the mirror (once), adds it to the catalog under\n" +
+ "the name the policies give its judge, as a judge (role: guard,\n" +
+ "format llamaguard3), on CPUs so it takes no card, and waits for a replica\n" +
+ "to answer. With no --model, the judge the policies ask that is missing.\n\n" +
+ "A model of that name already in the catalog as a model clients use is\n" +
+ "refused; --replace removes it first. An offline cluster gets its judge\n" +
+ "from its bundle with server init.",
+ Example: " llmgw ctl guard judge enable\n" +
+ " llmgw ctl guard judge enable --replace",
+ Args: cobra.NoArgs,
+ RunE: func(cmd *cobra.Command, _ []string) error {
+ ctx := cmd.Context()
+ out := cmd.OutOrStdout()
+ var js []adminapi.JudgeView
+ if err := client().get(ctx, "/api/v1/guard/judges", &js); err != nil {
+ return err
+ }
+ var target *adminapi.JudgeView
+ for i, j := range js {
+ if (model != "" && j.Model == model) || (model == "" && (!j.InCatalog || !j.Judge)) {
+ target = &js[i]
+ break
+ }
+ }
+ switch {
+ case target == nil && model != "":
+ target = &adminapi.JudgeView{Model: model}
+ fmt.Fprintf(out, "no policy asks %s yet; it is served, and a policy's judge.model names it\n", model)
+ case target == nil && len(js) == 0:
+ return errors.New("no policy asks a judge: give one a judge (llmgw ctl config edit, guard.policies..judge.model)")
+ case target == nil:
+ fmt.Fprintln(out, "every judge the policies ask is in the catalog, as a judge")
+ return nil
+ }
+ if target.InCatalog && target.Judge {
+ fmt.Fprintf(out, "%s is in the catalog as a judge already; llmgw ctl guard judge status says how it is\n", target.Model)
+ return nil
+ }
+ if target.InCatalog && !target.Judge {
+ if !replace {
+ return fmt.Errorf("%s is in the catalog as a model clients use, not as a judge: "+
+ "--replace removes it and serves it again as a judge", target.Model)
+ }
+ var res map[string]string
+ if err := client().delete(ctx, "/api/v1/models/"+target.Model, &res); err != nil {
+ return fmt.Errorf("removing %s: %w", target.Model, err)
+ }
+ fmt.Fprintf(out, "%s removed\n", target.Model)
+ }
+ req := modelserve.Request{
+ Repo: repo, File: file, Name: target.Model, GuardFormat: "llamaguard3",
+ Description: "The guardrails' judge: reads prompts and answers for unsafe content. Not offered to clients.",
+ CPU: true, Replicas: replicas, CPUCores: cores,
+ CtxPool: 8192, ContextLength: 4096,
+ }
+ body, _ := json.Marshal(req)
+ var job modelserve.Job
+ if err := client().patient(30*time.Second).post(ctx, "/api/v1/models/serve", body, &job); err != nil {
+ return err
+ }
+ if detach {
+ return detached(cmd, job)
+ }
+ if err := followJob(cmd, client, job, "is the judge"); err != nil {
+ return err
+ }
+ if strays, _ := strayJudges(cmd, client, js); len(strays) > 0 {
+ fmt.Fprintf(out, "\nno policy asks %s, served from a judge's weights as a model clients use: "+
+ "llmgw ctl model remove %s\n", strings.Join(strays, ", "), strings.Join(strays, " "))
+ }
+ return nil
+ },
+ }
+ f := c.Flags()
+ f.StringVar(&model, "model", "", "the judge to serve (default: the one the policies ask that is missing)")
+ f.StringVar(&repo, "repo", guardModelRepo, "the registry repository its weights come from")
+ f.StringVar(&file, "file", guardModelFile, "the file in it")
+ f.IntVar(&replicas, "replicas", 2, "replicas: every judged request waits on one")
+ f.IntVar(&cores, "cores", 4, "CPU cores a replica computes on")
+ f.BoolVar(&replace, "replace", false, "remove a model of that name that is not a judge first")
+ f.BoolVar(&detach, "detach", false, "start it and return; llmgw ctl jobs follows it")
+ return c
+}
diff --git a/internal/cli/server.go b/internal/cli/server.go
index 0dc97981..d0ed5657 100644
--- a/internal/cli/server.go
+++ b/internal/cli/server.go
@@ -361,7 +361,8 @@ func newServerCmd() *cobra.Command {
evals = newEvalService(cfg, rt, admin, log)
api.SetEval(evals)
if st := rt.Store(); st != nil && gate != nil {
- api.SetGuard(&adminapi.GuardAdmin{Library: gate.Library, Store: st, Invalidate: gate.Invalidate})
+ api.SetGuard(&adminapi.GuardAdmin{Library: gate.Library, Store: st, Invalidate: gate.Invalidate,
+ JudgeErrors: gate.JudgeErrors})
}
if st := rt.Store(); st != nil {
// Removing a node revokes its certificate.
diff --git a/internal/cli/status.go b/internal/cli/status.go
index 93e48839..6b99a0cc 100644
--- a/internal/cli/status.go
+++ b/internal/cli/status.go
@@ -153,6 +153,13 @@ func printSummary(cmd *cobra.Command, s adminapi.Summary) {
fmt.Fprintf(out, "STORE %s\n", strings.Join(parts, ", "))
}
+ if len(s.Judges) > 0 {
+ fmt.Fprintln(out, "\nGUARDRAILS' JUDGE")
+ for _, j := range s.Judges {
+ fmt.Fprintf(out, " %s: %s\n", j.Model, j.Problem)
+ }
+ }
+
if len(s.Parked) > 0 {
fmt.Fprintln(out, "\nPARKED — failed every time they started; not retried until fixed")
for _, p := range s.Parked {
diff --git a/internal/cli/tui_actions.go b/internal/cli/tui_actions.go
index 66dc17aa..e7862029 100644
--- a/internal/cli/tui_actions.go
+++ b/internal/cli/tui_actions.go
@@ -679,6 +679,8 @@ func (m *consoleModel) doEntry() tea.Cmd {
}}
case cBackup:
return m.runAfter("server", "backup", "--config="+m.file)
+ case cJudge:
+ return m.runAfter("ctl", server, "guard", "judge", "enable", "--replace")
}
return nil
}
diff --git a/internal/cli/tui_console.go b/internal/cli/tui_console.go
index 3d416701..0dda53b1 100644
--- a/internal/cli/tui_console.go
+++ b/internal/cli/tui_console.go
@@ -59,6 +59,7 @@ const (
cAddControl
cReplicate
cServe
+ cJudge
cUpgrade
cRestartAll
cBackup
@@ -68,7 +69,7 @@ const (
)
var consoleLabels = [cCount]string{"Overview", "Monitor", "Nodes", "GPUs", "Models", "Replicas", "Usage", "Quotas", "Guardrails", "Federation", "Mirror", "Jobs", "Logs", "Accounts", "API keys", "Organizations", "Audit", "Manual",
- "server.yaml", "Add a node", "Add a control plane", "Replicate the store", "Serve a model", "Upgrade the cluster",
+ "server.yaml", "Add a node", "Add a control plane", "Replicate the store", "Serve a model", "Serve the guardrails' judge", "Upgrade the cluster",
"Restart all replicas", "Back up now", "Wipe this machine", "Leave"}
// consoleAction is what is done once the screen closes: each runs its own
@@ -496,7 +497,7 @@ func (m consoleModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) {
return m.leave()
case cWipe:
return m.askWipe()
- case cUpgrade, cRestartAll, cBackup:
+ case cUpgrade, cRestartAll, cBackup, cJudge:
return m, m.doEntry()
case cOverview:
return m, nil
@@ -1299,6 +1300,9 @@ func (m consoleModel) panel() (title, body, keys string) {
item = "\n Moves the store onto two machines under Patroni, replicated,\n so losing one loses nothing.\n\n " +
sDim.Render("On the command line:") +
"\n sudo --preserve-env=SSH_AUTH_SOCK llmgw server init \\\n --patroni @ --patroni @ --no-mirror --no-node --no-model"
+ case cJudge:
+ item = "\n Serves the model the guardrail policies ask as their judge — Llama\n Guard 3 1B on CPUs, under the name the policies give it — so what they\n judge is not refused (guard_unavailable). A model of that name that is\n not a judge is replaced. Overview says when a judge is missing.\n\n " +
+ sDim.Render("On the command line:") + "\n llmgw ctl guard judge status\n llmgw ctl guard judge enable --replace"
case cUpgrade:
item = "\n Moves the cluster to a release: this machine first, then each node\n in turn. Engines keep serving; a node that does not come back stops it.\n\n " +
sDim.Render("On the command line:") + "\n sudo llmgw ctl upgrade VERSION"
diff --git a/internal/control/adminapi/guard.go b/internal/control/adminapi/guard.go
index 8dc128e0..60b14f63 100644
--- a/internal/control/adminapi/guard.go
+++ b/internal/control/adminapi/guard.go
@@ -28,6 +28,8 @@ type GuardAdmin struct {
Library func() guard.Config
Store GuardStore
Invalidate func()
+ // JudgeErrors are each judge's last failure to give a verdict.
+ JudgeErrors func() map[string]guard.JudgeError
}
// SetGuard enables the guard endpoints.
@@ -198,6 +200,7 @@ type GuardAttachmentRequest struct {
}
func (a *API) registerGuard(mux Mux) {
+ mux.HandleFunc("GET /api/v1/guard/judges", a.listJudges)
mux.HandleFunc("GET /api/v1/guard", func(w http.ResponseWriter, r *http.Request) {
if a.guard == nil {
writeError(w, http.StatusNotImplemented, "no guard policies: define them under guard in the cluster's settings (llmgw ctl config edit)")
diff --git a/internal/control/adminapi/judges.go b/internal/control/adminapi/judges.go
new file mode 100644
index 00000000..9cd01c9e
--- /dev/null
+++ b/internal/control/adminapi/judges.go
@@ -0,0 +1,124 @@
+package adminapi
+
+import (
+ "fmt"
+ "net/http"
+ "slices"
+ "sort"
+ "time"
+
+ "gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
+ "gitea.mynodes.xyz/ilker/llmgw/internal/guard"
+)
+
+// JudgeView is one judge model the guardrail policies ask, and whether it
+// can answer them.
+type JudgeView struct {
+ Model string `json:"model"`
+ // Policies are those that ask it; Closed those that refuse what it
+ // cannot read (fail: closed).
+ Policies []string `json:"policies"`
+ Closed []string `json:"closed,omitempty"`
+ // InCatalog is whether a model of that name is in the catalog, and
+ // Judge whether it is there as a judge (role: guard).
+ InCatalog bool `json:"in_catalog"`
+ Judge bool `json:"judge"`
+ Ready int `json:"ready_replicas"`
+ Wanted int `json:"wanted_replicas"`
+ // LastError is its last failure to give a verdict, if any.
+ LastError string `json:"last_error,omitempty"`
+ LastErrorAt *time.Time `json:"last_error_at,omitempty"`
+ // Problem says, in words, what is wrong and what to do; empty when
+ // nothing is.
+ Problem string `json:"problem,omitempty"`
+}
+
+// judgeErrorFresh is how recent a judge's failure must be to count as a
+// problem now.
+const judgeErrorFresh = 5 * time.Minute
+
+// Judges are the judge models the policies in force ask, each with what is
+// wrong with it, if anything.
+func (a *API) Judges() []JudgeView {
+ if a.guard == nil || a.guard.Library == nil {
+ return nil
+ }
+ lib := a.guard.Library()
+ byModel := map[string]*JudgeView{}
+ for name, p := range lib.Policies {
+ if p.Judge == nil || p.Judge.Model == "" {
+ continue
+ }
+ v := byModel[p.Judge.Model]
+ if v == nil {
+ v = &JudgeView{Model: p.Judge.Model}
+ byModel[p.Judge.Model] = v
+ }
+ v.Policies = append(v.Policies, name)
+ if p.Fail == "closed" && p.Mode != "shadow" {
+ v.Closed = append(v.Closed, name)
+ }
+ }
+ ready := map[string]int{}
+ for _, rep := range a.ctl.Reconciler().Replicas() {
+ if rep.State == reconciler.Ready {
+ ready[rep.Model]++
+ }
+ }
+ var errs map[string]guard.JudgeError
+ if a.guard.JudgeErrors != nil {
+ errs = a.guard.JudgeErrors()
+ }
+ out := make([]JudgeView, 0, len(byModel))
+ for _, v := range byModel {
+ sort.Strings(v.Policies)
+ sort.Strings(v.Closed)
+ if m, ok := a.ctl.Catalog().Resolve(v.Model); ok && m.Name == v.Model {
+ v.InCatalog, v.Judge = true, m.IsGuard()
+ v.Wanted, v.Ready = m.Scaling.Min, ready[m.Name]
+ }
+ if e, ok := errs[v.Model]; ok {
+ at := e.At
+ v.LastError, v.LastErrorAt = e.Err, &at
+ }
+ v.Problem = judgeProblem(*v)
+ out = append(out, *v)
+ }
+ slices.SortFunc(out, func(x, y JudgeView) int {
+ switch {
+ case x.Model < y.Model:
+ return -1
+ case x.Model > y.Model:
+ return 1
+ }
+ return 0
+ })
+ return out
+}
+
+func judgeProblem(v JudgeView) string {
+ consequence := "requests its policies judge are served unjudged"
+ if len(v.Closed) > 0 {
+ consequence = "requests its policies judge are refused (guard_unavailable)"
+ }
+ switch {
+ case !v.InCatalog:
+ return fmt.Sprintf("not in the catalog, so %s: llmgw ctl guard judge enable", consequence)
+ case !v.Judge:
+ return fmt.Sprintf("in the catalog as a model clients use, not as a judge (role: guard), so %s: "+
+ "llmgw ctl guard judge enable --replace", consequence)
+ case v.Ready == 0:
+ return fmt.Sprintf("no replica ready, so %s: llmgw ctl replicas --model %s", consequence, v.Model)
+ case v.LastErrorAt != nil && time.Since(*v.LastErrorAt) < judgeErrorFresh:
+ return fmt.Sprintf("gave no verdict %s ago (%s)", time.Since(*v.LastErrorAt).Round(time.Second), v.LastError)
+ }
+ return ""
+}
+
+func (a *API) listJudges(w http.ResponseWriter, _ *http.Request) {
+ if a.guard == nil {
+ writeError(w, http.StatusNotImplemented, "no guard policies: define them under guard in the cluster's settings (llmgw ctl config edit)")
+ return
+ }
+ writeJSON(w, http.StatusOK, a.Judges())
+}
diff --git a/internal/control/adminapi/judges_test.go b/internal/control/adminapi/judges_test.go
new file mode 100644
index 00000000..941dd5a1
--- /dev/null
+++ b/internal/control/adminapi/judges_test.go
@@ -0,0 +1,66 @@
+package adminapi
+
+import (
+ "strings"
+ "testing"
+ "time"
+
+ "gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+ "gitea.mynodes.xyz/ilker/llmgw/internal/guard"
+)
+
+func TestEachJudgeSaysWhatIsWrongWithIt(t *testing.T) {
+ models := append(testModels(),
+ // Served from the mirror page before it knew better: a model
+ // clients use, under the judge's name.
+ catalog.Model{Name: "shield", Source: "/s", Scaling: catalog.Scaling{Min: 1, Max: 1},
+ Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp"}}},
+ catalog.Model{Name: "llama-guard-3-1b", Source: "/g", Role: catalog.RoleGuard,
+ Guard: &catalog.GuardModel{Format: "llamaguard3"}, Scaling: catalog.Scaling{Min: 2, Max: 2},
+ Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp"}}})
+ a, _ := setup(t, models)
+ lib := guard.Config{Default: []string{"baseline"}, Policies: map[string]guard.Policy{
+ "baseline": {Fail: "closed", Judge: &guard.JudgePolicy{Model: "llama-guard-3-1b", Input: "block"}},
+ "strict": {Fail: "closed", Judge: &guard.JudgePolicy{Model: "missing-judge", Input: "block"}},
+ "shield": {Fail: "open", Judge: &guard.JudgePolicy{Model: "shield", Input: "flag"}},
+ }}
+ a.SetGuard(&GuardAdmin{Library: func() guard.Config { return lib }})
+
+ var js []JudgeView
+ if code := call(t, a, "GET", "/api/v1/guard/judges", "", &js); code != 200 || len(js) != 3 {
+ t.Fatalf("%d %+v", code, js)
+ }
+ by := map[string]JudgeView{}
+ for _, j := range js {
+ by[j.Model] = j
+ }
+ if j := by["missing-judge"]; j.InCatalog || !strings.Contains(j.Problem, "not in the catalog") ||
+ !strings.Contains(j.Problem, "refused") {
+ t.Errorf("missing: %+v", j)
+ }
+ if j := by["shield"]; !j.InCatalog || j.Judge || !strings.Contains(j.Problem, "--replace") ||
+ !strings.Contains(j.Problem, "unjudged") {
+ t.Errorf("not a judge: %+v", j)
+ }
+ if j := by["llama-guard-3-1b"]; !j.Judge || j.Wanted != 2 || !strings.Contains(j.Problem, "no replica ready") {
+ t.Errorf("not running: %+v", j)
+ }
+
+ var s Summary
+ call(t, a, "GET", "/api/v1/summary", "", &s)
+ if len(s.Judges) != 3 {
+ t.Errorf("the summary does not carry the judges' problems: %+v", s.Judges)
+ }
+
+ // A judge that failed lately says so, once it is otherwise fine.
+ at := time.Now()
+ v := JudgeView{Model: "x", InCatalog: true, Judge: true, Ready: 1, LastError: "deadline exceeded", LastErrorAt: &at}
+ if p := judgeProblem(v); !strings.Contains(p, "deadline exceeded") {
+ t.Errorf("a recent failure: %q", p)
+ }
+ old := at.Add(-time.Hour)
+ v.LastErrorAt = &old
+ if p := judgeProblem(v); p != "" {
+ t.Errorf("an old failure still counts: %q", p)
+ }
+}
diff --git a/internal/control/adminapi/routes.go b/internal/control/adminapi/routes.go
index 0ea0d56e..4a3e05fd 100644
--- a/internal/control/adminapi/routes.go
+++ b/internal/control/adminapi/routes.go
@@ -254,6 +254,8 @@ var Routes = []Route{
{Name: "since", Description: "How far back: 90m, 24h, 7d; 24h when absent."},
},
Response: GuardEventsView{}},
+ {Method: "GET", Path: "/api/v1/guard/judges", Summary: "The judge models the policies ask, and what is wrong with each",
+ Response: []JudgeView{}},
{Method: "POST", Path: "/api/v1/guard/attachments", Summary: "Attach a guardrail policy",
Request: GuardAttachmentRequest{}, Response: guard.Attachment{}},
{Method: "DELETE", Path: "/api/v1/guard/attachments", Summary: "Detach a guardrail policy",
diff --git a/internal/control/adminapi/summary.go b/internal/control/adminapi/summary.go
index 9c0d9e3d..600aa12e 100644
--- a/internal/control/adminapi/summary.go
+++ b/internal/control/adminapi/summary.go
@@ -42,6 +42,8 @@ type Summary struct {
// ControlPlanes is the cluster's control planes when there is more than
// one: which leads, which stand by, and the floating address in front.
ControlPlanes ControlPlanes `json:"control_planes,omitzero"`
+ // Judges are the guardrails' judge models with something wrong.
+ Judges []JudgeView `json:"judges,omitempty"`
}
// StoreMember is one member of a replicated store, as its Patroni reports it.
@@ -149,6 +151,11 @@ const idleThreshold = 10 * time.Minute
// Summary builds the fleet summary.
func (a *API) Summary() Summary {
s := Summary{Version: version.Version}
+ for _, j := range a.Judges() {
+ if j.Problem != "" {
+ s.Judges = append(s.Judges, j)
+ }
+ }
if a.storeStatus != nil {
members, err := a.storeStatus()
s.Store = members
diff --git a/internal/guard/gate.go b/internal/guard/gate.go
index 620f0c15..56a7d4f2 100644
--- a/internal/guard/gate.go
+++ b/internal/guard/gate.go
@@ -5,6 +5,7 @@ import (
"encoding/json"
"fmt"
"log/slog"
+ "maps"
"regexp"
"sort"
"strings"
@@ -43,6 +44,9 @@ type Gate struct {
m *gateMetrics
// alarm is told each time a judge gives no verdict; nil tells no one.
alarm func(model string, err error)
+ // lastErr is each judge's last failure to give a verdict.
+ errMu sync.Mutex
+ lastErr map[string]JudgeError
mu sync.Mutex
attached map[string][]string // "scope/id" → policy names
@@ -448,11 +452,30 @@ func (g *Gate) Input(ctx context.Context, s Subject, body []byte) (Verdict, erro
func (g *Gate) SetJudgeAlarm(f func(model string, err error)) { g.alarm = f }
func (g *Gate) judgeFailed(model string, err error) {
+ g.errMu.Lock()
+ if g.lastErr == nil {
+ g.lastErr = map[string]JudgeError{}
+ }
+ g.lastErr[model] = JudgeError{Err: err.Error(), At: g.now()}
+ g.errMu.Unlock()
if g.alarm != nil {
g.alarm(model, err)
}
}
+// JudgeError is a judge's last failure to give a verdict.
+type JudgeError struct {
+ Err string
+ At time.Time
+}
+
+// JudgeErrors are each judge's last failure to give a verdict, by model.
+func (g *Gate) JudgeErrors() map[string]JudgeError {
+ g.errMu.Lock()
+ defer g.errMu.Unlock()
+ return maps.Clone(g.lastErr)
+}
+
func judgeFinding(p Policy, action string, res judgment, err error) finding {
if err != nil {
// Unanswered: closed stops it, open lets it through, said.
diff --git a/internal/web/account.go b/internal/web/account.go
index d37b1288..33002427 100644
--- a/internal/web/account.go
+++ b/internal/web/account.go
@@ -88,8 +88,10 @@ type AccountPages struct {
oidc *oidcLogin
// audit records every change made through these pages; nil records none.
audit audit.Sink
- // hasGuard is set when /admin/guard is registered.
- hasGuard bool
+ // hasGuard is set when /admin/guard is registered, and guardLibrary
+ // with it: the policies in force, whose judges the mirror page serves.
+ hasGuard bool
+ guardLibrary func() guard.Config
// guardFor names the guardrail policies in force for someone, and
// whether they keep nothing; nil when there are none.
guardFor func(guard.Subject) (string, bool)
diff --git a/internal/web/admin_guard.go b/internal/web/admin_guard.go
index f9119d59..46621189 100644
--- a/internal/web/admin_guard.go
+++ b/internal/web/admin_guard.go
@@ -50,7 +50,7 @@ var guardWindows = map[string]time.Duration{"24h": 24 * time.Hour, "7d": 7 * 24
// RegisterAdminGuard mounts /admin/guard: what the guardrails did, where, by
// which policy — names and counts, never text.
func (p *AccountPages) RegisterAdminGuard(mux *http.ServeMux, s GuardAdminSource, library func() guard.Config) {
- p.hasGuard = true
+ p.hasGuard, p.guardLibrary = true, library
mux.HandleFunc("GET /admin/guard", p.adminOnly(func(w http.ResponseWriter, r *http.Request, admin store.User) {
ctx, l := r.Context(), langOf(r)
lib := library()
diff --git a/internal/web/admin_mirror.go b/internal/web/admin_mirror.go
index e84ebbd5..0c7bd271 100644
--- a/internal/web/admin_mirror.go
+++ b/internal/web/admin_mirror.go
@@ -324,10 +324,14 @@ func (p *AccountPages) RegisterAdminMirror(mux *http.ServeMux, d MirrorDeps) {
if err != nil {
return "", err
}
+ // A judge's weights are served as a judge, under the name the
+ // policies ask: served as a model clients use, the policies
+ // never find it and refuse every request they judge.
+ judge, renamed := p.asJudge(&model)
// The CPU while GPUs sit free is a choice to make on purpose:
// made by default, it was a model running at a tenth of the
- // speed with nothing saying why.
- if model.Profiles[0].CPUOnly() && r.FormValue("cpu_ok") == "" {
+ // speed with nothing saying why. A judge is meant for the CPU.
+ if !judge && model.Profiles[0].CPUOnly() && r.FormValue("cpu_ok") == "" {
if n := freeGPUs(p.fleet); n > 0 {
return "", errors.New(l.N("serve.err.cpu_with_free_gpus", int64(n), model.Name))
}
@@ -346,8 +350,15 @@ func (p *AccountPages) RegisterAdminMirror(mux *http.ServeMux, d MirrorDeps) {
}
// Placed on the next pass, not now: saying it is serving would
// be wrong for however long the weights take to reach a node.
- return l.T("serve.msg.added", model.Name,
- l.N("serve.replicas_n", int64(model.Scaling.Min)), describeServe(l, model)), nil
+ msg := l.T("serve.msg.added", model.Name,
+ l.N("serve.replicas_n", int64(model.Scaling.Min)), describeServe(l, model))
+ switch {
+ case renamed != "":
+ msg += " " + l.T("serve.msg.judge_renamed", renamed, model.Name)
+ case judge:
+ msg += " " + l.T("serve.msg.judge", model.Name)
+ }
+ return msg, nil
}))
}
diff --git a/internal/web/admin_mirror_test.go b/internal/web/admin_mirror_test.go
index 2dec8b03..a91242d0 100644
--- a/internal/web/admin_mirror_test.go
+++ b/internal/web/admin_mirror_test.go
@@ -4,6 +4,7 @@ import (
"context"
"errors"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+ "gitea.mynodes.xyz/ilker/llmgw/internal/guard"
"io"
"log/slog"
"net/http"
@@ -745,3 +746,23 @@ func TestASafetensorsSetIsFetchedWithWhatTheEngineReadsBesideIt(t *testing.T) {
t.Errorf("the companions are not said; got %s", body)
}
}
+
+func TestAJudgesWeightsAreServedAsTheJudgeThePoliciesAsk(t *testing.T) {
+ p := &AccountPages{guardLibrary: func() guard.Config {
+ return guard.Config{Policies: map[string]guard.Policy{
+ "baseline": {Judge: &guard.JudgePolicy{Model: "llama-guard-3-1b"}},
+ }}
+ }}
+ m := catalog.Model{Name: "llama-guard", Source: "http://10.0.0.146:8081/llama-guard/x/Llama-Guard-3-1B.Q4_K_M.gguf",
+ RoutingProfile: "chat"}
+ judge, renamed := p.asJudge(&m)
+ if !judge || renamed != "llama-guard" || m.Name != "llama-guard-3-1b" || !m.IsGuard() ||
+ m.Guard.Format != "llamaguard3" || m.RoutingProfile != "" {
+ t.Errorf("judge %v renamed %q: %+v", judge, renamed, m)
+ }
+
+ plain := catalog.Model{Name: "qwen", Source: "http://m/qwen.gguf", RoutingProfile: "chat"}
+ if judge, _ := p.asJudge(&plain); judge || plain.IsGuard() || plain.Name != "qwen" {
+ t.Errorf("a model clients use became a judge: %+v", plain)
+ }
+}
diff --git a/internal/web/admin_serve.go b/internal/web/admin_serve.go
index dc211599..e9889147 100644
--- a/internal/web/admin_serve.go
+++ b/internal/web/admin_serve.go
@@ -585,3 +585,36 @@ func argInt(args []string, flag string) int {
}
return 0
}
+
+// isJudgeWeights is whether a source is a guard model's weights: Llama
+// Guard, which reads a conversation and says whether it is safe.
+func isJudgeWeights(source string) bool {
+ return strings.Contains(strings.ToLower(source), "llama-guard")
+}
+
+// asJudge makes a model built from the serve form a judge when its weights
+// are one's: role guard, Llama Guard's format, under the name the policies
+// give their judge when they give one and the form gave another. It says
+// whether it did, and the name the form gave when it was changed.
+func (p *AccountPages) asJudge(m *catalog.Model) (judge bool, renamed string) {
+ if !isJudgeWeights(m.Source) {
+ return false, ""
+ }
+ m.Role, m.Guard, m.RoutingProfile = catalog.RoleGuard, &catalog.GuardModel{Format: "llamaguard3"}, ""
+ if p.guardLibrary == nil {
+ return true, ""
+ }
+ asked := map[string]bool{}
+ for _, pol := range p.guardLibrary().Policies {
+ if pol.Judge != nil && pol.Judge.Model != "" {
+ asked[pol.Judge.Model] = true
+ }
+ }
+ if asked[m.Name] || len(asked) != 1 {
+ return true, ""
+ }
+ for name := range asked {
+ renamed, m.Name = m.Name, name
+ }
+ return true, renamed
+}
diff --git a/internal/web/locales/en/serve.json b/internal/web/locales/en/serve.json
index 9a4b17ba..d36fa626 100644
--- a/internal/web/locales/en/serve.json
+++ b/internal/web/locales/en/serve.json
@@ -77,5 +77,7 @@
"serve.err.cpu_with_free_gpus.other": "this fleet has %d GPUs free and %s was set to run on the CPU; choose a GPU placement, or tick \"run on the CPU anyway\"",
"serve.msg.added": "%s added to the catalog, %s: %s; it is placed on the next pass",
"serve.replicas_n.one": "%d replica",
- "serve.replicas_n.other": "%d replicas"
+ "serve.replicas_n.other": "%d replicas",
+ "serve.msg.judge": "It is the guardrails' judge (role: guard): policies naming %s ask it; clients do not see it.",
+ "serve.msg.judge_renamed": "Its weights are a judge's, so it is the guardrails' judge (role: guard), named %[2]s as the policies ask rather than %[1]s."
}
diff --git a/internal/web/locales/tr/serve.json b/internal/web/locales/tr/serve.json
index 2798b719..6113f7f1 100644
--- a/internal/web/locales/tr/serve.json
+++ b/internal/web/locales/tr/serve.json
@@ -74,5 +74,7 @@
"serve.err.placement": "%q geçerli bir yerleşim değil",
"serve.err.cpu_with_free_gpus.other": "bu kümede %d GPU boşta, %s ise CPU üzerinde çalışacak biçimde ayarlandı; bir GPU yerleşimi seçin ya da \"yine de CPU üzerinde çalıştır\" kutusunu işaretleyin",
"serve.msg.added": "%s kataloğa eklendi, %s: %s; bir sonraki geçişte yerleştirilecek",
- "serve.replicas_n.other": "%d replika"
+ "serve.replicas_n.other": "%d replika",
+ "serve.msg.judge": "Guardrail'lerin judge modeli olarak eklendi (role: guard): %s adını veren politikalar ona soruyor; istemciler görmüyor.",
+ "serve.msg.judge_renamed": "Ağırlıkları bir judge modelinin olduğu için guardrail judge'ı olarak eklendi (role: guard); adı %[1]s yerine politikaların istediği gibi %[2]s."
}