From f0477f140d0f8c51f958036f3ba6547a10fd60cc Mon Sep 17 00:00:00 2001 From: Ilker Manap Date: Mon, 5 Oct 2026 00:04:43 +0200 Subject: [PATCH] Quality suites, and new weights held to one: ctl eval, model update/promote --eval, model gate A suite is a set of cases in YAML or JSON lines, each scored by exact, contains, regex, a number within a tolerance, a JSON Schema subset, or a judge model with a rubric; weighed into one score against a threshold. internal/bench/eval runs one on the control plane through Router.Replay, as the gateway's own requests, and keeps every run and its cases in the store (migration 0028: eval_runs, eval_cases). - llmgw ctl eval MODEL --suite FILE|NAME, --history, --run, --suites; exits non-zero below the score. Suites by name from control.eval_dir, or the two built in from deploy/eval (arithmetic, json-format). - model update and promote take --eval SUITE --min-score N, or a model's standing gate (ctl model gate, eval: in the catalog): a promote asks the canary first and leaves it below the score; an update tries the new weights as a canary given no share, then promotes or removes it. - The console's model menu: ask a suite with its progress, its runs, its gate. - The audit middleware no longer hands a handler a body cut off at 64 KiB. Co-Authored-By: Claude Opus 5.5 (1M context) --- deploy/eval/arithmetic.yaml | 58 +++ deploy/eval/json-format.yaml | 94 ++++ docs.go | 9 +- docs/console.md | 10 + docs/install.md | 112 +++++ docs/reference.md | 46 ++ internal/audit/audit.go | 8 +- internal/audit/audit_test.go | 14 + internal/bench/eval/run.go | 260 +++++++++++ internal/bench/eval/run_test.go | 292 ++++++++++++ internal/bench/eval/schema.go | 338 ++++++++++++++ internal/bench/eval/score.go | 214 +++++++++ internal/bench/eval/score_test.go | 133 ++++++ internal/bench/eval/service.go | 434 ++++++++++++++++++ internal/bench/eval/suite.go | 332 ++++++++++++++ internal/cli/ctl.go | 1 + internal/cli/ctl_eval.go | 408 ++++++++++++++++ internal/cli/ctl_eval_test.go | 123 +++++ internal/cli/model.go | 3 +- internal/cli/model_canary.go | 19 +- internal/cli/model_update.go | 19 +- internal/cli/server.go | 7 + internal/cli/tui_actions.go | 7 +- internal/cli/tui_console.go | 20 +- internal/cli/tui_quality.go | 291 ++++++++++++ internal/cli/tui_quality_test.go | 116 +++++ internal/config/config.go | 6 + internal/control/adminapi/api.go | 14 +- internal/control/adminapi/canary.go | 6 +- internal/control/adminapi/eval.go | 179 ++++++++ internal/control/adminapi/eval_test.go | 84 ++++ internal/control/catalog/catalog.go | 24 + internal/control/modelserve/canary.go | 70 ++- internal/control/modelserve/canary_test.go | 2 +- internal/control/modelserve/gate.go | 96 ++++ internal/control/modelserve/gate_test.go | 184 ++++++++ internal/control/modelserve/modelserve.go | 7 + internal/control/modelserve/updates.go | 56 ++- internal/control/modelserve/updates_test.go | 10 +- internal/control/modelsvc/modelsvc.go | 17 + internal/docgen/reference_gen.go | 7 + internal/router/quality_test.go | 131 ++++++ internal/store/copy.go | 3 + internal/store/eval.go | 136 ++++++ internal/store/eval_test.go | 59 +++ .../migrations/postgres/0028_eval_runs.sql | 43 ++ .../migrations/sqlite/0028_eval_runs.sql | 42 ++ internal/store/postgres_test.go | 3 +- notes/c21.md | 62 +++ 49 files changed, 4563 insertions(+), 46 deletions(-) create mode 100644 deploy/eval/arithmetic.yaml create mode 100644 deploy/eval/json-format.yaml create mode 100644 internal/bench/eval/run.go create mode 100644 internal/bench/eval/run_test.go create mode 100644 internal/bench/eval/schema.go create mode 100644 internal/bench/eval/score.go create mode 100644 internal/bench/eval/score_test.go create mode 100644 internal/bench/eval/service.go create mode 100644 internal/bench/eval/suite.go create mode 100644 internal/cli/ctl_eval.go create mode 100644 internal/cli/ctl_eval_test.go create mode 100644 internal/cli/tui_quality.go create mode 100644 internal/cli/tui_quality_test.go create mode 100644 internal/control/adminapi/eval.go create mode 100644 internal/control/adminapi/eval_test.go create mode 100644 internal/control/modelserve/gate.go create mode 100644 internal/control/modelserve/gate_test.go create mode 100644 internal/router/quality_test.go create mode 100644 internal/store/eval.go create mode 100644 internal/store/eval_test.go create mode 100644 internal/store/migrations/postgres/0028_eval_runs.sql create mode 100644 internal/store/migrations/sqlite/0028_eval_runs.sql create mode 100644 notes/c21.md diff --git a/deploy/eval/arithmetic.yaml b/deploy/eval/arithmetic.yaml new file mode 100644 index 0000000..c1bf1ae --- /dev/null +++ b/deploy/eval/arithmetic.yaml @@ -0,0 +1,58 @@ +# Arithmetic a small instruction model gets right: sums, products and a +# little reading, each answer scored by the last number in it. A quantisation +# that has gone wrong, or a chat template that no longer matches the +# weights, shows up here as a falling score long before anyone notices it in +# conversation. +# +# llmgw ctl eval MODEL --suite arithmetic +# llmgw ctl model update MODEL --eval arithmetic --min-score 0.8 +# +# Copy it into control.eval_dir (/etc/llmgw/eval) to change it; a suite +# there of the same name replaces this one. +name: arithmetic +description: Sums, products and word problems, each scored by the number in its answer. +threshold: 0.7 +system: You are a careful calculator. Answer with the number only, without words or units. +max_tokens: 64 +cases: + - id: add-small + prompt: What is 17 + 25? + number: 42 + - id: add-carry + prompt: What is 389 + 476? + number: 865 + - id: subtract + prompt: What is 1000 - 357? + number: 643 + - id: subtract-negative + prompt: What is 12 - 30? + number: -18 + - id: multiply + prompt: What is 12 * 12? + number: 144 + - id: multiply-larger + prompt: What is 23 * 17? + number: 391 + weight: 2 + - id: divide + prompt: What is 144 / 6? + number: 24 + - id: decimal + prompt: What is 2.5 * 4? + number: 10 + - id: percent + prompt: What is 15% of 200? + number: 30 + - id: words-apples + prompt: Ana has 7 apples and buys 5 more, then gives 3 away. How many apples does she have? + number: 9 + weight: 2 + - id: words-time + prompt: A train leaves at 14:00 and the trip takes 150 minutes. How many hours does the trip take? + number: 2.5 + tolerance: 0.01 + weight: 2 + - id: order-of-operations + prompt: What is 2 + 3 * 4? + number: 14 + weight: 2 diff --git a/deploy/eval/json-format.yaml b/deploy/eval/json-format.yaml new file mode 100644 index 0000000..f23e5a4 --- /dev/null +++ b/deploy/eval/json-format.yaml @@ -0,0 +1,94 @@ +# Following a format: asked for JSON of a given shape, does the model give +# it — the thing an application parsing its answers depends on, and the +# first to go when a chat template drifts. Each answer is read as JSON (a +# code fence or a sentence around it is forgiven) and held to a schema. +# +# llmgw ctl eval MODEL --suite json-format +# +# Copy it into control.eval_dir (/etc/llmgw/eval) to change it; a suite +# there of the same name replaces this one. +name: json-format +description: Answers asked for as JSON of a given shape, held to a schema. +threshold: 0.7 +system: You answer with a single JSON value and nothing else. +max_tokens: 200 +cases: + - id: person + prompt: >- + Give a JSON object for a person named Ada Lovelace, born in 1815, with the + keys "name" (a string) and "born" (a number). + json_schema: + type: object + required: [name, born] + properties: + name: {type: string, pattern: "Ada"} + born: {type: integer, minimum: 1815, maximum: 1815} + + - id: list-of-colours + prompt: Give a JSON array of exactly three colour names, as strings. + json_schema: + type: array + minItems: 3 + maxItems: 3 + items: {type: string, minLength: 1} + + - id: boolean-answer + prompt: >- + Is water wet? Answer as a JSON object with one key, "answer", whose + value is true or false. + json_schema: + type: object + required: [answer] + properties: + answer: {type: boolean} + + - id: nested + prompt: >- + Give a JSON object describing a book: "title" (a string), "author" (an + object with "first" and "last", both strings) and "pages" (a whole number + above zero). Use any book. + json_schema: + type: object + required: [title, author, pages] + properties: + title: {type: string, minLength: 1} + author: + type: object + required: [first, last] + properties: + first: {type: string} + last: {type: string} + pages: {type: integer, exclusiveMinimum: 0} + weight: 2 + + - id: enum-choice + prompt: >- + Classify the sentence "I love this product!" as positive, negative or + neutral. Answer as a JSON object {"sentiment": "..."} using one of those + three words in lower case. + json_schema: + type: object + required: [sentiment] + properties: + sentiment: {enum: [positive]} + weight: 2 + + - id: numbers + prompt: 'Give a JSON object {"sum": N} where N is the sum of 4, 5 and 6.' + json_schema: + type: object + required: [sum] + properties: + sum: {const: 15} + + - id: no-extra-keys + prompt: >- + Give a JSON object with exactly two keys, "city" and "country", for + Paris. No other keys. + json_schema: + type: object + required: [city, country] + additionalProperties: false + properties: + city: {type: string} + country: {type: string} diff --git a/docs.go b/docs.go index 11e35bc..8dcc249 100644 --- a/docs.go +++ b/docs.go @@ -1,4 +1,5 @@ -// Package llmgw holds what the binary carries besides code: its manual. +// Package llmgw holds what the binary carries besides code: its manual, +// and the example quality suites. package llmgw import "embed" @@ -8,3 +9,9 @@ import "embed" // //go:embed README.md CHANGELOG.md docs/*.md var Docs embed.FS + +// EvalSuites are the example quality suites under deploy/eval, which the +// control plane runs by name when its eval directory has none of the name. +// +//go:embed deploy/eval/*.yaml +var EvalSuites embed.FS diff --git a/docs/console.md b/docs/console.md index f891558..8bbc37a 100644 --- a/docs/console.md +++ b/docs/console.md @@ -117,6 +117,16 @@ and **UPDATE** when the registry has new weights for it. On a row: canary is removed) and **Take the canary out…**. A canary that errs or answers more slowly than the model is taken out by itself. The MODEL column shows `(10% to canary)` and `(canary of …)`. +- **Ask it a quality suite…** — one of the suites the control plane has (in + `control.eval_dir`, or built in: `arithmetic`, `json-format`), asked + through the router; the dialog counts the answers as they come in, then + shows the score, each case and the model's last runs. Esc stops watching; + the run carries on and is kept. **Its quality runs** lists them, as + `llmgw ctl eval --history` does. +- **Hold its updates to a quality suite…** — the suite and score its new + weights must reach before an update or a promotion moves it to them + (`llmgw ctl model gate`); **none** takes it away. With one set, **Update + to …** and **Promote the canary…** say so. - **Why is it placed where it is** — `llmgw ctl explain-placement`. - **Try again**, for a model parked after failing every start. - **Remove** — its replicas stop and it leaves the catalog. A model from diff --git a/docs/install.md b/docs/install.md index c5923ce..ced0bb2 100644 --- a/docs/install.md +++ b/docs/install.md @@ -911,6 +911,118 @@ llmgw ctl model canary qwen-7b --abort # or not - **Promote** is the update the canary was a trial of: the model's replicas are replaced one at a time, and `llmgw ctl model rollback` goes back. +### Quality suites, and holding new weights to one + +A canary says whether new weights serve as fast and as reliably; it does not +say whether their answers are as good. A republished quantisation or a chat +template that no longer matches can make a model faster and worse at once. A +quality suite is a handful of questions with known answers, asked of a model +and scored: + +``` +llmgw ctl eval --suites # what the control plane can run +llmgw ctl eval qwen-7b --suite arithmetic # by name, on the control plane +llmgw ctl eval qwen-7b --suite ./support.yaml --judge llama-70b --concurrency 8 +llmgw ctl eval --history qwen-7b # the runs kept +llmgw ctl eval --run eval-20261004-101500-1a2b3c4d # one, case by case +``` + +It prints each case's score and why, then the suite's: the cases' scores +weighed, 0 to 1, and whether it reached the suite's `threshold` (or +`--min-score`); it exits non-zero below it, so a script can stop on it. + +- **How it is asked:** on the control plane, through the router as the + gateway's own requests, the way a replay is — past admission, quotas, + accounting and the guardrails, so the score is the weights' and not a + tenant's limits'. It reaches the model exactly as a client's request + would, a canary or a model loading on demand included. Answers are asked + at temperature 0 unless a case says otherwise, and a reasoning model's + `` block is not part of its answer. +- **Where suites come from:** `--suite` is a file on the machine running + `ctl`, sent whole, or the name of one in the control plane's + `control.eval_dir` (`/etc/llmgw/eval`). Two are built in and run on small + models: `arithmetic` (sums and word problems, each scored by the number in + the answer) and `json-format` (answers asked for as JSON of a given shape); + their files are in `deploy/eval/`, to copy and change. A file in the + directory with the same name replaces the built-in one. +- **What is kept:** every run, in the store, with the model, the repository + and revision the answering weights came from (when known), the suite's name + and sha256, when, the score, and each case's answer (up to 2,000 + characters), score and reason. A run on a canary is filed under the model + it is a canary of. Runs are not pruned. + +A suite is YAML (or JSON lines: one case on each line, and optionally a line +without a prompt that holds the suite's own settings): + +```yaml +name: support # the file's name unless given +threshold: 0.8 # the score to pass at, 0 to 1; 0.8 unless given +judge: llama-70b # grades the cases scored by a rubric +system: You are a support agent for Acme. # every case's, unless it has one +max_tokens: 256 # every case's; 512 unless given +cases: + - id: refund-window + prompt: How many days do customers have to ask for a refund? + number: 30 # the last number in the answer, ±tolerance + tolerance: 0 + - prompt: Which plan includes SSO? + contains: [Enterprise] # every string must appear + ignore_case: true + - prompt: Reply with only the word OK. + exact: OK # the whole answer, white space around it aside + - prompt: What is today's ISO date format? + regex: '\d{4}-\d{2}-\d{2}' # matches somewhere in the answer + - prompt: Give the order as JSON with "id" and "total". + json_schema: # the answer, read as JSON, satisfies it + type: object + required: [id, total] + properties: { id: {type: string}, total: {type: number, minimum: 0} } + - messages: # a conversation instead of a prompt + - { role: user, content: My card was charged twice. } + judge: Apologises, and says the duplicate charge is refunded within 5 days. + weight: 3 # counts three times; 1 unless given +``` + +Each case is scored by exactly one of `exact`, `contains`, `regex`, `number`, +`json_schema` or `judge`. A JSON answer in a code fence or inside a sentence is +still read. The schema supports `type`, `properties`, `required`, +`additionalProperties`, `items`, `enum`, `const`, the numeric and length bounds, +`pattern`, `anyOf` and `allOf`; a suite using anything else is refused when it +is read rather than passed unchecked. A `judge` case asks another model on the +cluster (`judge_model:` on the case, else the suite's `judge:`, else `--judge`) +to grade the answer by the rubric, 0 to 1. A case passes at 0.5; a case that +got no answer scores 0 and says why. + +**Holding updates to a suite.** `model update` and `model promote` take +`--eval SUITE --min-score N` (and `--judge`); or a model carries one — set +with `ctl model gate`, or `eval: { suite: arithmetic, min_score: 0.8 }` in a +definition added with `ctl model add` — so every update and promotion of it +is held to it, from `ctl`, the console or anywhere: + +``` +llmgw ctl model gate qwen-7b --eval arithmetic --min-score 0.8 +llmgw ctl model update qwen-7b # asked the suite first +llmgw ctl model update qwen-7b --skip-eval # once, without it; the job says so +llmgw ctl model gate qwen-7b --off +``` + +- **Promote** asks the canary the suite before anything moves; below the + score the job fails with the score, and the canary stays, with its share, + until you take it out. +- **Update** runs the new weights first as `MODEL-canary` given none of the + model's requests (hidden from clients like any canary), asks it the suite, + then either promotes it — the model's replicas are replaced one at a time, + as for any update — or takes it out again, leaving the model on its old + weights. That needs room for one more replica while it runs; a model that + already has a canary is promoted or aborted first. Without a suite, an + update swaps the weights in as it always has. +- The job's `eval` step says the score (`llmgw ctl jobs JOB`), and the run is + kept with the job's id beside it. + +The console has it on a model's menu: **Ask it a quality suite…** (a suite the +control plane has, the score as it comes in), **Its quality runs**, and **Hold +its updates to a quality suite…**. + ### How many replicas, and how much each one carries Replica count is changed while the cluster runs, without editing the diff --git a/docs/reference.md b/docs/reference.md index 67325c4..ad3ef98 100644 --- a/docs/reference.md +++ b/docs/reference.md @@ -63,6 +63,7 @@ rather than repeated. | `port_range` | `[]int` | [Control](#control) | | `mgmt_interface` | `string` | [Control](#control) | | `timezone` | `string` | [Control](#control) | +| `eval_dir` | `string` | [Control](#control) | | `enabled` | `bool` | [Control](#control) | | `lease_ttl` | `time.Duration` | [Control](#control) | | `id` | `string` | [Endpoint](#endpoint) | @@ -77,6 +78,9 @@ rather than repeated. | `nodes` | `[]string` | [EngineOverride](#engineoverride) | | `labels` | `map[string]string` | [EngineOverride](#engineoverride) | | `engines` | `map[string]string` | [EngineOverride](#engineoverride) | +| `suite` | `string` | [EvalGate](#evalgate) | +| `min_score` | `float64` | [EvalGate](#evalgate) | +| `judge` | `string` | [EvalGate](#evalgate) | | `url` | `string` | [External](#external) | | `model` | `string` | [External](#external) | | `api_key_file` | `string` | [External](#external) | @@ -109,6 +113,7 @@ rather than repeated. | `adapters` | `[]Adapter` | [Model](#model) | | `fallbacks` | `[]string` | [Model](#model) | | `canary` | `*Canary` | [Model](#model) | +| `eval` | `*EvalGate` | [Model](#model) | | `routing_profile` | `string` | [Model](#model) | | `slo` | `SLO` | [Model](#model) | | `context_length` | `int` | [Model](#model) | @@ -665,6 +670,15 @@ MgmtInterface is the ethernet interface NCCL bootstraps over. Timezone is where models' scaling schedules are read, an IANA name like Europe/Istanbul; empty is the control plane's own. +### `eval_dir` + +`string` + +EvalDir is where quality suites are read from by name, for `llmgw ctl +eval MODEL --suite NAME`, a model's eval gate and the console. The +suites the binary carries (arithmetic, json-format) are found when it +has none of the name; a suite here of the same name replaces one. + ### `enabled` `bool` @@ -758,6 +772,28 @@ Labels picks nodes carrying all of these. Engines are laid over the cluster's: a version, or "" for none. +## EvalGate + +### `suite` + +`string` + +Suite is a suite's name in control.eval_dir, or one the binary +carries (arithmetic, json-format). + +### `min_score` + +`float64` + +MinScore is the score, 0 to 1, to pass at; zero is the suite's own +threshold. + +### `judge` + +`string` + +Judge grades the suite's rubric cases, instead of the judge it names. + ## External ### `url` @@ -1024,6 +1060,16 @@ moving to it (`llmgw ctl model canary`). Takes a [Canary](#canary) block. +### `eval` + +`*EvalGate` + +Eval is a quality suite new weights must pass before they replace +this model's: `llmgw ctl model update` and `model promote` run it +against them first, and stop below its score. + +Takes a [EvalGate](#evalgate) block. + ### `routing_profile` `string` diff --git a/internal/audit/audit.go b/internal/audit/audit.go index 1270003..e51a732 100644 --- a/internal/audit/audit.go +++ b/internal/audit/audit.go @@ -120,8 +120,14 @@ func API(sink Sink, actor func(*http.Request) string, log *slog.Logger, next htt } var body []byte if r.Body != nil { + // The first 64 KiB are what is recorded; the handler is given the + // whole body, or a larger one — a quality suite sent whole — would + // reach it cut off. body, _ = io.ReadAll(io.LimitReader(r.Body, 64<<10)) - r.Body = io.NopCloser(bytes.NewReader(body)) + r.Body = struct { + io.Reader + io.Closer + }{io.MultiReader(bytes.NewReader(body), r.Body), r.Body} } sw := &statusWriter{ResponseWriter: w} next.ServeHTTP(sw, r) diff --git a/internal/audit/audit_test.go b/internal/audit/audit_test.go index 8286806..f0687bf 100644 --- a/internal/audit/audit_test.go +++ b/internal/audit/audit_test.go @@ -2,6 +2,7 @@ package audit import ( "context" + "io" "net/http" "net/http/httptest" "net/url" @@ -58,3 +59,16 @@ func TestEveryChangeToTheAdminAPIIsRecorded(t *testing.T) { t.Errorf("%+v", e) } } + +func TestABodyLargerThanWhatIsRecordedReachesTheHandlerWhole(t *testing.T) { + big := `{"content":"` + strings.Repeat("x", 200<<10) + `"}` + var got int + h := API(&memSink{}, func(*http.Request) string { return "ilker" }, nil, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + b, _ := io.ReadAll(r.Body) + got = len(b) + })) + h.ServeHTTP(httptest.NewRecorder(), httptest.NewRequest("POST", "/api/v1/models/m/eval", strings.NewReader(big))) + if got != len(big) { + t.Errorf("the handler was given %d of %d bytes", got, len(big)) + } +} diff --git a/internal/bench/eval/run.go b/internal/bench/eval/run.go new file mode 100644 index 0000000..c02455e --- /dev/null +++ b/internal/bench/eval/run.go @@ -0,0 +1,260 @@ +package eval + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "net/http" + "strings" + "sync" + "time" + + "gitea.mynodes.xyz/ilker/llmgw/internal/store" +) + +// Caller sends a chat completion to a model as the gateway does for itself: +// the router's Replay, past admission, accounting and the guardrails — a +// suite measures the weights, and a tenant's quota or a guardrail's refusal +// would measure something else. It answers the status, the body, and how +// long it took. +type Caller func(ctx context.Context, path, model string, body []byte) (int, []byte, time.Duration, error) + +// Options are how a suite is run. +type Options struct { + // Model is the catalog model asked. + Model string + // Concurrency is how many cases are asked at once; 4 unless given. + Concurrency int + // Judge grades the cases scored by a rubric, instead of the suite's. + Judge string + // MinScore is the score to pass at, instead of the suite's threshold. + MinScore float64 + // CaseTimeout bounds one case, its judging included; five minutes + // unless given, which a model loading on demand needs. + CaseTimeout time.Duration + // Progress is told as each case finishes. + Progress func(done, total int) +} + +// PassAt is the score a case passes at: a judge's 0.5 is a coin toss, and +// counts as passed so a lenient rubric is not a hidden zero. +const PassAt = 0.5 + +// keptAnswer is how much of an answer is kept with its result. +const keptAnswer = 2000 + +// Evaluate asks a model every case of a suite and scores the answers. A +// case that got no answer scores 0 with its error; a suite that cannot be +// run at all — a rubric and no judge to read it — is an error before +// anything is asked. +func Evaluate(ctx context.Context, call Caller, s Suite, o Options) (store.EvalRun, error) { + if o.Model == "" { + return store.EvalRun{}, errors.New("no model to ask") + } + judge := o.Judge + if judge == "" { + judge = s.Judge + } + for _, c := range s.Cases { + if c.Judge != "" && c.JudgeModel == "" && judge == "" { + return store.EvalRun{}, fmt.Errorf("case %s is graded by a judge and the suite names none: "+ + "give one with --judge MODEL, or judge: in the suite", c.ID) + } + } + threshold := s.Threshold + if o.MinScore > 0 { + threshold = o.MinScore + } + if threshold <= 0 { + threshold = DefaultThreshold + } + workers := o.Concurrency + if workers <= 0 { + workers = 4 + } + timeout := o.CaseTimeout + if timeout <= 0 { + timeout = 5 * time.Minute + } + + run := store.EvalRun{Model: o.Model, Suite: s.Name, SuiteHash: s.Hash, At: time.Now().UTC(), + Threshold: threshold, Cases: len(s.Cases), Results: make([]store.EvalCase, len(s.Cases))} + if s.NeedsJudge() { + run.Judge = judge + } + todo := make(chan int) + var wg sync.WaitGroup + var mu sync.Mutex + done := 0 + for range min(workers, len(s.Cases)) { + wg.Go(func() { + for i := range todo { + cctx, cancel := context.WithTimeout(ctx, timeout) + res := askOne(cctx, call, s, &s.Cases[i], o.Model, judge) + cancel() + mu.Lock() + run.Results[i] = res + done++ + if o.Progress != nil { + o.Progress(done, len(s.Cases)) + } + mu.Unlock() + } + }) + } + for i := range s.Cases { + select { + case todo <- i: + case <-ctx.Done(): + } + if ctx.Err() != nil { + break + } + } + close(todo) + wg.Wait() + run.TookMs = time.Since(run.At).Milliseconds() + + var sum, weights float64 + firstErr := "" + for i, r := range run.Results { + if r.ID == "" { + // Not asked: the run was stopped first. + r = store.EvalCase{ID: s.Cases[i].ID, Kind: s.Cases[i].Kind(), Weight: s.Cases[i].Weight, + Error: "not asked: the run was stopped"} + run.Results[i] = r + } + sum += r.Weight * r.Score + weights += r.Weight + if !r.Passed { + run.Below++ + } + if r.Error != "" { + run.Errors++ + if firstErr == "" { + firstErr = r.Error + } + } + } + if weights > 0 { + run.Score = sum / weights + } + run.Passed = run.Score >= threshold + if run.Errors == run.Cases { + run.Passed = false + run.Error = "no case was answered: " + firstErr + } + return run, nil +} + +// askedOf is the conversation a case sends. +func askedOf(c *Case, suiteSystem string) []Message { + var out []Message + system := c.System + if system == "" { + system = suiteSystem + } + if system != "" && (len(c.Messages) == 0 || c.Messages[0].Role != "system") { + out = append(out, Message{Role: "system", Content: system}) + } + if len(c.Messages) > 0 { + return append(out, c.Messages...) + } + return append(out, Message{Role: "user", Content: c.Prompt}) +} + +// askOne asks one case and scores its answer. +func askOne(ctx context.Context, call Caller, s Suite, c *Case, model, judge string) store.EvalCase { + res := store.EvalCase{ID: c.ID, Kind: c.Kind(), Weight: c.Weight} + start := time.Now() + defer func() { res.TookMs = time.Since(start).Milliseconds() }() + + maxTokens := c.MaxTokens + if maxTokens <= 0 { + maxTokens = s.MaxTokens + } + if maxTokens <= 0 { + maxTokens = defaultMaxTokens + } + temp := 0.0 + if c.Temperature != nil { + temp = *c.Temperature + } + answer, finish, err := chat(ctx, call, model, askedOf(c, s.System), maxTokens, temp) + if err != nil { + res.Error = err.Error() + return res + } + answer = cleanAnswer(answer) + res.Answer = answer + if len(res.Answer) > keptAnswer { + res.Answer = res.Answer[:keptAnswer] + } + + var v verdict + if c.Judge != "" { + jm := c.JudgeModel + if jm == "" { + jm = judge + } + reply, _, err := chat(ctx, call, jm, judgeRequest(c, answer), 256, 0) + if err != nil { + res.Error = "the judge " + jm + ": " + err.Error() + return res + } + if v, err = readJudgement(reply); err != nil { + res.Error = err.Error() + return res + } + } else { + v = scoreRule(c, answer) + } + res.Score, res.Detail = v.score, v.detail + res.Passed = res.Score >= PassAt + if finish == "length" && !res.Passed { + res.Detail += fmt.Sprintf("; the answer was cut off at %d tokens", maxTokens) + } + if answer == "" && !res.Passed { + res.Detail += "; it answered nothing" + } + return res +} + +// chat asks a model one conversation and says what it answered and why it +// stopped. +func chat(ctx context.Context, call Caller, model string, msgs []Message, maxTokens int, temp float64) (string, string, error) { + body, err := json.Marshal(map[string]any{"model": model, "messages": msgs, "max_tokens": maxTokens, + "temperature": temp, "stream": false}) + if err != nil { + return "", "", err + } + status, out, _, err := call(ctx, "/v1/chat/completions", model, body) + if err != nil { + return "", "", err + } + if status != http.StatusOK { + var e struct { + Error struct { + Message string `json:"message"` + } `json:"error"` + } + msg := strings.TrimSpace(string(out)) + if json.Unmarshal(out, &e) == nil && e.Error.Message != "" { + msg = e.Error.Message + } + return "", "", fmt.Errorf("%s answered %d: %s", model, status, clip(msg, 200)) + } + var resp struct { + Choices []struct { + Message struct { + Content string `json:"content"` + } `json:"message"` + FinishReason string `json:"finish_reason"` + } `json:"choices"` + } + if err := json.Unmarshal(out, &resp); err != nil || len(resp.Choices) == 0 { + return "", "", fmt.Errorf("%s's answer is not a chat completion: %s", model, clip(string(out), 120)) + } + return resp.Choices[0].Message.Content, resp.Choices[0].FinishReason, nil +} diff --git a/internal/bench/eval/run_test.go b/internal/bench/eval/run_test.go new file mode 100644 index 0000000..be5dc18 --- /dev/null +++ b/internal/bench/eval/run_test.go @@ -0,0 +1,292 @@ +package eval + +import ( + "context" + "encoding/json" + "errors" + "log/slog" + "os" + "path/filepath" + "strings" + "sync" + "sync/atomic" + "testing" + "testing/fstest" + "time" + + "gitea.mynodes.xyz/ilker/llmgw/internal/store" +) + +const arith = ` +name: arith +threshold: 0.75 +judge: grader +cases: + - id: add + prompt: "What is 17 + 25? Answer with the number only." + number: 42 + - id: mul + prompt: "What is 6 * 7?" + number: 42 + weight: 2 + - id: capital + prompt: "What is the capital of France?" + contains: Paris + - id: polite + prompt: "Say hello." + judge: "A greeting of any kind passes." +` + +// fakeModel answers by the prompt, and as a judge by what it was asked to +// grade; it counts what it was asked. +type fakeModel struct { + mu sync.Mutex + answers map[string]string + asked []string + inFlight, most atomic.Int32 +} + +func (f *fakeModel) call(_ context.Context, path, model string, body []byte) (int, []byte, time.Duration, error) { + n := f.inFlight.Add(1) + defer f.inFlight.Add(-1) + for { + m := f.most.Load() + if n <= m || f.most.CompareAndSwap(m, n) { + break + } + } + time.Sleep(5 * time.Millisecond) + var req struct { + Model string `json:"model"` + Messages []Message `json:"messages"` + } + if err := json.Unmarshal(body, &req); err != nil || path != "/v1/chat/completions" || req.Model != model { + return 400, []byte(`{"error":{"message":"bad request"}}`), 0, nil + } + f.mu.Lock() + f.asked = append(f.asked, model) + f.mu.Unlock() + last := req.Messages[len(req.Messages)-1].Content + answer := "I do not know" + switch { + case model == "grader": + answer = `{"score": 0, "reason": "no greeting"}` + if strings.Contains(last, "Answer:\nHello") { + answer = `{"score": 1, "reason": "a greeting"}` + } + case model == "missing": + return 404, []byte(`{"error":{"message":"the model \"missing\" does not exist"}}`), 0, nil + default: + for k, v := range f.answers { + if strings.Contains(last, k) { + answer = v + } + } + } + out, _ := json.Marshal(map[string]any{"choices": []any{map[string]any{ + "message": map[string]any{"content": answer}, "finish_reason": "stop"}}}) + return 200, out, 0, nil +} + +func TestASuiteIsAskedAndScoredByWeight(t *testing.T) { + s, err := Parse("arith.yaml", []byte(arith)) + if err != nil { + t.Fatal(err) + } + f := &fakeModel{answers: map[string]string{"17 + 25": "carry42", "6 * 7": "41", + "France": "Paris.", "hello": "Hello there!"}} + var progress atomic.Int32 + run, err := Evaluate(context.Background(), f.call, s, Options{Model: "qwen", Concurrency: 2, + Progress: func(done, total int) { progress.Store(int32(done)) }}) + if err != nil { + t.Fatal(err) + } + // add 1, mul 0 at weight 2, capital 1, polite 1: 3 of 5. + if run.Score != 0.6 || run.Passed || run.Threshold != 0.75 || run.Below != 1 || run.Cases != 4 || + run.Suite != "arith" || run.Judge != "grader" || len(run.SuiteHash) != 64 { + t.Fatalf("%+v", run) + } + byID := map[string]store.EvalCase{} + for _, r := range run.Results { + byID[r.ID] = r + } + if r := byID["add"]; !r.Passed || r.Answer != "42" || r.Kind != "number" { + t.Errorf("add: %+v", r) + } + if r := byID["mul"]; r.Passed || !strings.Contains(r.Detail, "read 41, wanted 42") || r.Weight != 2 { + t.Errorf("mul: %+v", r) + } + if r := byID["polite"]; !r.Passed || !strings.Contains(r.Detail, "the judge: a greeting") { + t.Errorf("polite: %+v", r) + } + if progress.Load() != 4 || f.most.Load() > 2 { + t.Errorf("progress %d, %d at once with a concurrency of 2", progress.Load(), f.most.Load()) + } + + // A lower bar passes the same answers. + run, _ = Evaluate(context.Background(), f.call, s, Options{Model: "qwen", MinScore: 0.5}) + if !run.Passed || run.Threshold != 0.5 { + t.Errorf("at 0.5: %+v", run) + } +} + +func TestAModelThatAnswersNothingFailsWithWhy(t *testing.T) { + s, _ := Parse("arith.yaml", []byte(arith)) + run, err := Evaluate(context.Background(), (&fakeModel{}).call, s, Options{Model: "missing"}) + if err != nil { + t.Fatal(err) + } + if run.Passed || run.Errors != 4 || !strings.Contains(run.Error, `"missing" does not exist`) { + t.Fatalf("%+v", run) + } +} + +func TestARubricWithNoJudgeIsRefusedBeforeAnythingIsAsked(t *testing.T) { + s, _ := Parse("arith.yaml", []byte(strings.Replace(arith, "judge: grader\n", "", 1))) + f := &fakeModel{} + if _, err := Evaluate(context.Background(), f.call, s, Options{Model: "qwen"}); err == nil || + !strings.Contains(err.Error(), "--judge") { + t.Fatalf("%v", err) + } + if len(f.asked) != 0 { + t.Errorf("asked %v first", f.asked) + } +} + +func TestASuiteIsReadFromJSONLinesAsFromYAML(t *testing.T) { + jsonl := `{"name": "fmt", "threshold": 0.5} +{"id": "a", "prompt": "Give JSON", "json_schema": {"type": "object", "required": ["ok"]}} +{"prompt": "Say yes", "regex": "^yes", "ignore_case": true, "weight": 3} +` + s, err := Parse("format.jsonl", []byte(jsonl)) + if err != nil { + t.Fatal(err) + } + if s.Name != "fmt" || s.Threshold != 0.5 || len(s.Cases) != 2 || s.Cases[1].ID != "case-2" || + s.Cases[1].Weight != 3 || s.Cases[0].schema == nil || s.Cases[1].re == nil { + t.Fatalf("%+v", s) + } + // Named by its file when it names nothing, and 0.8 to pass. + s, err = Parse("dir/plain.jsonl", []byte(`{"prompt": "q", "exact": "a"}`)) + if err != nil || s.Name != "plain" || s.Threshold != DefaultThreshold { + t.Fatalf("%+v %v", s, err) + } +} + +func TestASuiteThatCannotBeScoredIsRefusedSayingWhy(t *testing.T) { + for _, tc := range []struct{ suite, why string }{ + {"cases: []", "no cases"}, + {"cases:\n - prompt: q", "no way to score it"}, + {"cases:\n - prompt: q\n exact: a\n regex: b", "exact and regex"}, + {"cases:\n - exact: a", "nothing to ask"}, + {"cases:\n - prompt: q\n regex: '('", "the regex"}, + {"cases:\n - id: x\n prompt: q\n exact: a\n - id: x\n prompt: q\n exact: a", "two cases are called x"}, + {"threshold: 2\ncases:\n - prompt: q\n exact: a", "not 0 to 1"}, + {"cases:\n - prompt: q\n exactly: a", "exactly"}, + {"cases:\n - messages: [{role: robot, content: hi}]\n exact: a", "role"}, + } { + if _, err := Parse("s.yaml", []byte(tc.suite)); err == nil || !strings.Contains(err.Error(), tc.why) { + t.Errorf("%q: %v, wanted it to say %q", tc.suite, err, tc.why) + } + } +} + +type memStore struct { + mu sync.Mutex + runs []store.EvalRun +} + +func (m *memStore) PutEvalRun(_ context.Context, r store.EvalRun) error { + m.mu.Lock() + defer m.mu.Unlock() + m.runs = append(m.runs, r) + return nil +} +func (m *memStore) EvalRuns(_ context.Context, model string, _ int) ([]store.EvalRun, error) { + m.mu.Lock() + defer m.mu.Unlock() + var out []store.EvalRun + for i := len(m.runs) - 1; i >= 0; i-- { + if r := m.runs[i]; model == "" || r.Model == model || r.Target == model { + r.Results = nil + out = append(out, r) + } + } + return out, nil +} +func (m *memStore) EvalRunByID(_ context.Context, id string) (store.EvalRun, error) { + m.mu.Lock() + defer m.mu.Unlock() + for _, r := range m.runs { + if r.ID == id { + return r, nil + } + } + return store.EvalRun{}, store.ErrNotFound +} + +func TestTheControlPlaneRunsSuitesByNameAndFilesACanarysRunUnderItsModel(t *testing.T) { + dir := t.TempDir() + if err := os.WriteFile(filepath.Join(dir, "arith.yaml"), []byte(arith), 0o600); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(dir, "broken.yaml"), []byte("cases: [}"), 0o600); err != nil { + t.Fatal(err) + } + builtin := fstest.MapFS{ + "arith.yaml": {Data: []byte("cases:\n - prompt: shadowed\n exact: x\n")}, + "json-format.yaml": {Data: []byte("description: JSON\ncases:\n - prompt: q\n json_schema: {type: object}\n")}, + } + f := &fakeModel{answers: map[string]string{"17 + 25": "42", "6 * 7": "42", "France": "Paris", "hello": "Hello"}} + st := &memStore{} + svc := NewService(Deps{Call: f.call, Store: st, Dir: dir, Builtin: builtin, Log: slog.New(slog.DiscardHandler), + Origin: func(m string) (string, string) { return "o/r", "rev-of-" + m }, + CanaryOf: func(n string) string { + if parent, ok := strings.CutSuffix(n, "-canary"); ok { + return parent + } + return "" + }, + }) + + suites := svc.Suites() + if len(suites) != 3 || suites[0].Name != "arith" || suites[0].From != filepath.Join(dir, "arith.yaml") || + suites[0].Cases != 4 || !suites[0].Judge || suites[1].Error == "" || suites[2].From != "built in" { + t.Fatalf("%+v", suites) + } + if _, err := svc.Load(Request{Suite: "../etc/passwd"}); err == nil { + t.Error("a path outside the eval directory was read") + } + + live, err := svc.Start("qwen-canary", Request{Suite: "arith"}, "tester") + if err != nil { + t.Fatal(err) + } + if live.State != Running || live.Model != "qwen" || live.Target != "qwen-canary" || live.Cases != 4 { + t.Fatalf("%+v", live) + } + deadline := time.Now().Add(5 * time.Second) + for live.State == Running && time.Now().Before(deadline) { + time.Sleep(10 * time.Millisecond) + live, _ = svc.Get(context.Background(), live.ID) + } + if live.State != Done || live.Score != 1 || !live.Passed || live.Revision != "rev-of-qwen-canary" || live.By != "tester" { + t.Fatalf("%+v", live) + } + if kept, err := st.EvalRunByID(context.Background(), live.ID); err != nil || len(kept.Results) != 4 { + t.Fatalf("kept %+v %v", kept, err) + } + + // A gate waits for its run, and files it with the job. + run, err := svc.Run(context.Background(), "qwen", Request{Suite: "json-format"}, "tester", "update-1") + if err != nil || run.Job != "update-1" || run.Passed || run.Target != "" { + t.Fatalf("%+v %v", run, err) + } + hist, err := svc.History(context.Background(), "qwen", 10) + if err != nil || len(hist) != 2 || hist[0].ID != run.ID || hist[1].ID != live.ID { + t.Fatalf("%+v %v", hist, err) + } + if _, err := svc.Get(context.Background(), "eval-nope"); !errors.Is(err, store.ErrNotFound) { + t.Errorf("%v", err) + } +} diff --git a/internal/bench/eval/schema.go b/internal/bench/eval/schema.go new file mode 100644 index 0000000..ff7cbf7 --- /dev/null +++ b/internal/bench/eval/schema.go @@ -0,0 +1,338 @@ +package eval + +import ( + "encoding/json" + "fmt" + "math" + "reflect" + "regexp" + "slices" + "sort" + "strings" + "unicode/utf8" +) + +// A JSON Schema validator for what a suite asks of an answer's shape: types, +// properties, required ones, items, enums, bounds and patterns. Not the whole +// standard — no $ref, no oneOf — and a schema using what is not here is +// refused when the suite is read rather than passed by an answer it never +// checked. + +type schema struct { + types []string + properties map[string]*schema + required []string + additional *schema // nil: anything; noneAllowed: nothing + items *schema + enum []any + constant any + hasConst bool + minimum, maximum *float64 + exclMin, exclMax *float64 + minLength, maxLength *int + minItems, maxItems *int + pattern *regexp.Regexp + anyOf, allOf []*schema + never bool +} + +// noneAllowed is additionalProperties: false. +var noneAllowed = &schema{never: true} + +// annotations are keywords that describe rather than constrain. +var annotations = map[string]bool{"$schema": true, "$id": true, "title": true, "description": true, + "default": true, "examples": true, "format": true, "$comment": true} + +func compileSchema(v any) (*schema, error) { + // Through JSON, so a schema written in YAML and one in JSON are the same + // thing by the time they are read: numbers as float64, maps as map[string]any. + b, err := json.Marshal(v) + if err != nil { + return nil, err + } + var raw any + if err := json.Unmarshal(b, &raw); err != nil { + return nil, err + } + return compile(raw, "") +} + +func compile(raw any, at string) (*schema, error) { + if b, ok := raw.(bool); ok { + if b { + return &schema{}, nil + } + return noneAllowed, nil + } + m, ok := raw.(map[string]any) + if !ok { + return nil, fmt.Errorf("%s is not a schema object", where(at)) + } + s := &schema{} + keys := make([]string, 0, len(m)) + for k := range m { + keys = append(keys, k) + } + sort.Strings(keys) + for _, k := range keys { + v := m[k] + var err error + switch k { + case "type": + switch t := v.(type) { + case string: + s.types = []string{t} + case []any: + for _, x := range t { + if str, ok := x.(string); ok { + s.types = append(s.types, str) + } + } + } + for _, t := range s.types { + if !slices.Contains([]string{"object", "array", "string", "number", "integer", "boolean", "null"}, t) { + return nil, fmt.Errorf("%s: %q is not a JSON type", where(at), t) + } + } + case "properties": + props, ok := v.(map[string]any) + if !ok { + return nil, fmt.Errorf("%s: properties is not an object", where(at)) + } + s.properties = map[string]*schema{} + for name, p := range props { + if s.properties[name], err = compile(p, at+"."+name); err != nil { + return nil, err + } + } + case "required": + list, _ := v.([]any) + for _, x := range list { + if str, ok := x.(string); ok { + s.required = append(s.required, str) + } + } + case "additionalProperties": + if s.additional, err = compile(v, at+".*"); err != nil { + return nil, err + } + case "items": + if s.items, err = compile(v, at+"[]"); err != nil { + return nil, err + } + case "enum": + s.enum, _ = v.([]any) + case "const": + s.constant, s.hasConst = v, true + case "minimum", "maximum", "exclusiveMinimum", "exclusiveMaximum": + f, ok := v.(float64) + if !ok { + return nil, fmt.Errorf("%s: %s is not a number", where(at), k) + } + *map[string]**float64{"minimum": &s.minimum, "maximum": &s.maximum, + "exclusiveMinimum": &s.exclMin, "exclusiveMaximum": &s.exclMax}[k] = &f + case "minLength", "maxLength", "minItems", "maxItems": + f, ok := v.(float64) + if !ok || f < 0 || f != math.Trunc(f) { + return nil, fmt.Errorf("%s: %s is not a whole number", where(at), k) + } + n := int(f) + *map[string]**int{"minLength": &s.minLength, "maxLength": &s.maxLength, + "minItems": &s.minItems, "maxItems": &s.maxItems}[k] = &n + case "pattern": + str, _ := v.(string) + if s.pattern, err = regexp.Compile(str); err != nil { + return nil, fmt.Errorf("%s: pattern: %w", where(at), err) + } + case "anyOf", "allOf": + list, ok := v.([]any) + if !ok || len(list) == 0 { + return nil, fmt.Errorf("%s: %s is not a list of schemas", where(at), k) + } + for i, x := range list { + sub, err := compile(x, fmt.Sprintf("%s(%s %d)", at, k, i+1)) + if err != nil { + return nil, err + } + if k == "anyOf" { + s.anyOf = append(s.anyOf, sub) + } else { + s.allOf = append(s.allOf, sub) + } + } + default: + if !annotations[k] { + return nil, fmt.Errorf("%s: %s is not supported here (type, properties, required, "+ + "additionalProperties, items, enum, const, minimum, maximum, exclusiveMinimum, exclusiveMaximum, "+ + "minLength, maxLength, minItems, maxItems, pattern, anyOf and allOf are)", where(at), k) + } + } + } + return s, nil +} + +func where(at string) string { + if at == "" { + return "the schema" + } + return "the schema at " + strings.TrimPrefix(at, ".") +} + +// validate says what is wrong with v, or "". +func (s *schema) validate(v any, at string) string { + if s.never { + return fmt.Sprintf("%s is not allowed", place(at)) + } + if len(s.types) > 0 && !slices.ContainsFunc(s.types, func(t string) bool { return isType(v, t) }) { + return fmt.Sprintf("%s is %s, not %s", place(at), typeOf(v), strings.Join(s.types, " or ")) + } + if s.hasConst && !equal(v, s.constant) { + return fmt.Sprintf("%s is %s, not %s", place(at), short(v), short(s.constant)) + } + if s.enum != nil && !slices.ContainsFunc(s.enum, func(e any) bool { return equal(v, e) }) { + return fmt.Sprintf("%s is %s, not one of %s", place(at), short(v), short(s.enum)) + } + switch x := v.(type) { + case float64: + switch { + case s.minimum != nil && x < *s.minimum: + return fmt.Sprintf("%s is %v, below %v", place(at), x, *s.minimum) + case s.maximum != nil && x > *s.maximum: + return fmt.Sprintf("%s is %v, above %v", place(at), x, *s.maximum) + case s.exclMin != nil && x <= *s.exclMin: + return fmt.Sprintf("%s is %v, not above %v", place(at), x, *s.exclMin) + case s.exclMax != nil && x >= *s.exclMax: + return fmt.Sprintf("%s is %v, not below %v", place(at), x, *s.exclMax) + } + case string: + n := utf8.RuneCountInString(x) + switch { + case s.minLength != nil && n < *s.minLength: + return fmt.Sprintf("%s is %d characters, fewer than %d", place(at), n, *s.minLength) + case s.maxLength != nil && n > *s.maxLength: + return fmt.Sprintf("%s is %d characters, more than %d", place(at), n, *s.maxLength) + case s.pattern != nil && !s.pattern.MatchString(x): + return fmt.Sprintf("%s, %s, does not match %s", place(at), short(x), s.pattern) + } + case []any: + switch { + case s.minItems != nil && len(x) < *s.minItems: + return fmt.Sprintf("%s has %d items, fewer than %d", place(at), len(x), *s.minItems) + case s.maxItems != nil && len(x) > *s.maxItems: + return fmt.Sprintf("%s has %d items, more than %d", place(at), len(x), *s.maxItems) + } + if s.items != nil { + for i, e := range x { + if why := s.items.validate(e, fmt.Sprintf("%s[%d]", at, i)); why != "" { + return why + } + } + } + case map[string]any: + for _, r := range s.required { + if _, ok := x[r]; !ok { + return fmt.Sprintf("%s has no %s", place(at), r) + } + } + keys := make([]string, 0, len(x)) + for k := range x { + keys = append(keys, k) + } + sort.Strings(keys) + for _, k := range keys { + if p, ok := s.properties[k]; ok { + if why := p.validate(x[k], at+"."+k); why != "" { + return why + } + } else if s.additional != nil { + if why := s.additional.validate(x[k], at+"."+k); why != "" { + return why + } + } + } + } + for _, sub := range s.allOf { + if why := sub.validate(v, at); why != "" { + return why + } + } + if len(s.anyOf) > 0 { + var first string + for _, sub := range s.anyOf { + why := sub.validate(v, at) + if why == "" { + return "" + } + if first == "" { + first = why + } + } + return first + " (nor any other choice)" + } + return "" +} + +func place(at string) string { + if at == "" { + return "the answer" + } + return strings.TrimPrefix(at, ".") +} + +func isType(v any, t string) bool { + switch t { + case "object": + _, ok := v.(map[string]any) + return ok + case "array": + _, ok := v.([]any) + return ok + case "string": + _, ok := v.(string) + return ok + case "number": + _, ok := v.(float64) + return ok + case "integer": + f, ok := v.(float64) + return ok && f == math.Trunc(f) + case "boolean": + _, ok := v.(bool) + return ok + case "null": + return v == nil + } + return false +} + +func typeOf(v any) string { + switch x := v.(type) { + case map[string]any: + return "an object" + case []any: + return "a list" + case string: + return "a string" + case float64: + if x == math.Trunc(x) { + return "an integer" + } + return "a number" + case bool: + return "a boolean" + case nil: + return "null" + } + return fmt.Sprintf("%T", v) +} + +func equal(a, b any) bool { return reflect.DeepEqual(a, b) } + +func short(v any) string { + b, _ := json.Marshal(v) + s := string(b) + if len(s) > 60 { + s = s[:57] + "..." + } + return s +} diff --git a/internal/bench/eval/score.go b/internal/bench/eval/score.go new file mode 100644 index 0000000..947e41d --- /dev/null +++ b/internal/bench/eval/score.go @@ -0,0 +1,214 @@ +package eval + +import ( + "encoding/json" + "fmt" + "math" + "regexp" + "strconv" + "strings" +) + +// verdict is how one answer scored, 0 to 1, and why. +type verdict struct { + score float64 + detail string +} + +func pass(detail string) verdict { return verdict{1, detail} } +func fail(detail string) verdict { return verdict{0, detail} } + +var thinkBlock = regexp.MustCompile(`(?s).*?`) + +// cleanAnswer is what a model said, without a reasoning model's thinking — +// what a reader is shown — and without the white space around it. +func cleanAnswer(s string) string { + s = thinkBlock.ReplaceAllString(s, "") + // A reply cut off while still thinking has an opening tag and no close. + if i := strings.Index(s, ""); i >= 0 { + s = s[:i] + } + return strings.TrimSpace(s) +} + +// scoreRule scores an answer by a case's rule; a case graded by a judge is +// scored elsewhere, since it takes another request. +func scoreRule(c *Case, answer string) verdict { + fold := func(s string) string { + if c.IgnoreCase { + return strings.ToLower(s) + } + return s + } + switch { + case c.Exact != nil: + if fold(answer) == fold(strings.TrimSpace(*c.Exact)) { + return pass("the answer is exactly that") + } + return fail(fmt.Sprintf("wanted exactly %q", strings.TrimSpace(*c.Exact))) + case len(c.Contains) > 0: + var missing []string + for _, want := range c.Contains { + if !strings.Contains(fold(answer), fold(want)) { + missing = append(missing, strconv.Quote(want)) + } + } + if len(missing) == 0 { + return pass("has everything asked for") + } + return fail("has no " + strings.Join(missing, ", ")) + case c.re != nil: + if c.re.MatchString(answer) { + return pass("matches") + } + return fail("does not match " + c.re.String()) + case c.Number != nil: + got, ok := lastNumber(answer) + if !ok { + return fail(fmt.Sprintf("no number in the answer; wanted %v", *c.Number)) + } + if math.Abs(got-*c.Number) <= c.Tolerance { + return pass(fmt.Sprintf("read %v", got)) + } + want := fmt.Sprintf("%v", *c.Number) + if c.Tolerance > 0 { + want += fmt.Sprintf(" ± %v", c.Tolerance) + } + return fail(fmt.Sprintf("read %v, wanted %s", got, want)) + case c.schema != nil: + v, err := readJSON(answer) + if err != nil { + return fail(err.Error()) + } + if why := c.schema.validate(v, ""); why != "" { + return fail(why) + } + return pass("JSON of the shape asked for") + } + return fail("no rule to score it by") +} + +// number is a number as a model writes one: a sign, thousands separated by +// commas, a decimal part, an exponent. +var number = regexp.MustCompile(`[-+]?(?:\d{1,3}(?:,\d{3})+|\d+)(?:\.\d+)?(?:[eE][-+]?\d+)?`) + +// lastNumber is the last number in a text: a model that shows its working +// ends with the result, and one that does not says only the result. +func lastNumber(s string) (float64, bool) { + all := number.FindAllStringIndex(s, -1) + if len(all) == 0 { + return 0, false + } + i, j := all[len(all)-1][0], all[len(all)-1][1] + // A hyphen after a digit or letter joins (3-5, 2026-10-04); it is not a + // minus sign. + if (s[i] == '-' || s[i] == '+') && i > 0 && isWordByte(s[i-1]) { + i++ + } + f, err := strconv.ParseFloat(strings.ReplaceAll(s[i:j], ",", ""), 64) + return f, err == nil +} + +func isWordByte(b byte) bool { + return b >= '0' && b <= '9' || b >= 'a' && b <= 'z' || b >= 'A' && b <= 'Z' +} + +var fence = regexp.MustCompile("(?s)```(?:json|JSON)?\\s*(.*?)```") + +// readJSON reads an answer as JSON: the whole of it, else what is inside a +// code fence, else from the first brace or bracket to the last — models +// asked for JSON wrap it in a sentence often enough that refusing that would +// measure manners rather than the shape. +func readJSON(answer string) (any, error) { + try := []string{answer} + if m := fence.FindStringSubmatch(answer); m != nil { + try = append(try, m[1]) + } + for _, pair := range [][2]string{{"{", "}"}, {"[", "]"}} { + i, j := strings.Index(answer, pair[0]), strings.LastIndex(answer, pair[1]) + if i >= 0 && j > i { + try = append(try, answer[i:j+1]) + } + } + for _, t := range try { + var v any + if json.Unmarshal([]byte(strings.TrimSpace(t)), &v) == nil { + return v, nil + } + } + if answer == "" { + return nil, fmt.Errorf("the answer is empty, not JSON") + } + return nil, fmt.Errorf("the answer is not JSON: %s", clip(answer, 80)) +} + +func clip(s string, n int) string { + s = strings.Join(strings.Fields(s), " ") + if len(s) <= n { + return s + } + return s[:n-3] + "..." +} + +// judgePrompt asks a model to grade an answer by a rubric. +const judgePrompt = "You grade an answer to a question against a rubric. Read the question, the answer " + + "and the rubric, then reply with only a JSON object: {\"score\": a number from 0 to 1, " + + "\"reason\": \"one short sentence\"}. 1 means the answer meets the rubric fully, 0 not at all." + +// judgeRequest is what a judge is asked about one answer. +func judgeRequest(c *Case, answer string) []Message { + var q strings.Builder + for _, m := range askedOf(c, "") { + if m.Role == "system" { + continue + } + fmt.Fprintf(&q, "%s: %s\n", m.Role, m.Content) + } + return []Message{ + {Role: "system", Content: judgePrompt}, + {Role: "user", Content: fmt.Sprintf("Question:\n%s\nAnswer:\n%s\n\nRubric:\n%s", q.String(), answer, c.Judge)}, + } +} + +var ( + passWord = regexp.MustCompile(`(?i)\b(pass|passed|yes|correct)\b`) + failWord = regexp.MustCompile(`(?i)\b(fail|failed|no|incorrect|wrong)\b`) +) + +// readJudgement reads a judge's reply: a JSON object with a score (or a +// pass true or false), else a lone number, else the words pass or fail. +func readJudgement(reply string) (verdict, error) { + reply = cleanAnswer(reply) + if v, err := readJSON(reply); err == nil { + if m, ok := v.(map[string]any); ok { + reason, _ := m["reason"].(string) + if f, ok := m["score"].(float64); ok { + return verdict{clamp(f), "the judge: " + clip(reason, 120)}, nil + } + if s, ok := m["score"].(string); ok { + if f, err := strconv.ParseFloat(strings.TrimSpace(s), 64); err == nil { + return verdict{clamp(f), "the judge: " + clip(reason, 120)}, nil + } + } + if b, ok := m["pass"].(bool); ok { + if b { + return pass("the judge: " + clip(reason, 120)), nil + } + return fail("the judge: " + clip(reason, 120)), nil + } + } + } + if f, err := strconv.ParseFloat(strings.TrimSpace(reply), 64); err == nil { + return verdict{clamp(f), "the judge gave " + strings.TrimSpace(reply)}, nil + } + p, f := passWord.MatchString(reply), failWord.MatchString(reply) + switch { + case p && !f: + return pass("the judge: " + clip(reply, 120)), nil + case f && !p: + return fail("the judge: " + clip(reply, 120)), nil + } + return verdict{}, fmt.Errorf("the judge's answer could not be read as a score: %s", clip(reply, 120)) +} + +func clamp(f float64) float64 { return math.Max(0, math.Min(1, f)) } diff --git a/internal/bench/eval/score_test.go b/internal/bench/eval/score_test.go new file mode 100644 index 0000000..af8d8ce --- /dev/null +++ b/internal/bench/eval/score_test.go @@ -0,0 +1,133 @@ +package eval + +import ( + "strings" + "testing" +) + +func caseOf(t *testing.T, yamlCase string) *Case { + t.Helper() + s, err := Parse("t.yaml", []byte("cases:\n - prompt: q\n"+indent(yamlCase))) + if err != nil { + t.Fatal(err) + } + return &s.Cases[0] +} + +func indent(s string) string { + var b strings.Builder + for _, l := range strings.Split(strings.TrimSpace(s), "\n") { + b.WriteString(" " + l + "\n") + } + return b.String() +} + +func TestEachRuleScoresTheAnswersItShouldAndRefusesTheRest(t *testing.T) { + for _, tc := range []struct { + rule string + answer string + want float64 + }{ + {`exact: "42"`, "42", 1}, + {`exact: "42"`, " 42\n", 1}, + {`exact: "42"`, "42.", 0}, + {"exact: Paris\nignore_case: true", "PARIS", 1}, + {`contains: Paris`, "The capital is Paris.", 1}, + {`contains: [Paris, France]`, "Paris is in France", 1}, + {`contains: [Paris, France]`, "Paris", 0}, + {"contains: paris\nignore_case: true", "PARIS", 1}, + {`regex: '^\s*\d{4}-\d{2}-\d{2}\s*$'`, "2026-10-04", 1}, + {`regex: '^\d+$'`, "about 12", 0}, + {"regex: '^yes'\nignore_case: true", "Yes, it is", 1}, + {`number: 42`, "17 + 25 = 42", 1}, + {`number: 42`, "The answer is 42.", 1}, + {`number: 42`, "It is 41", 0}, + {`number: 1234567`, "1,234,567", 1}, + {"number: 3.14\ntolerance: 0.01", "pi is about 3.1416", 1}, + {"number: 3.14\ntolerance: 0.001", "pi is about 3.1416", 0}, + {`number: -5`, "3 - 8 = -5", 1}, + // A date's last part is a number, not a negative one. + {`number: 4`, "from 2026-10-04", 1}, + {`number: 1`, "no number here", 0}, + } { + c := caseOf(t, tc.rule) + v := scoreRule(c, cleanAnswer(tc.answer)) + if v.score != tc.want { + t.Errorf("%s against %q scored %v (%s), wanted %v", strings.ReplaceAll(tc.rule, "\n", "; "), tc.answer, v.score, v.detail, tc.want) + } + } +} + +func TestAnAnswerIsReadAsJSONAndHeldToItsSchema(t *testing.T) { + c := caseOf(t, ` +json_schema: + type: object + required: [name, age, tags] + additionalProperties: false + properties: + name: {type: string, minLength: 1} + age: {type: integer, minimum: 0, maximum: 150} + tags: {type: array, items: {type: string}, maxItems: 3} + kind: {enum: [cat, dog]}`) + for _, tc := range []struct { + answer string + pass bool + why string + }{ + {`{"name": "Ada", "age": 36, "tags": ["math"]}`, true, ""}, + {"Here it is:\n```json\n{\"name\": \"Ada\", \"age\": 36, \"tags\": []}\n```", true, ""}, + {`Sure! {"name": "Ada", "age": 36, "tags": []} Hope that helps.`, true, ""}, + {`{"name": "Ada", "age": 36.5, "tags": []}`, false, "age is a number, not integer"}, + {`{"name": "Ada", "tags": []}`, false, "has no age"}, + {`{"name": "Ada", "age": 36, "tags": [], "extra": 1}`, false, "extra is not allowed"}, + {`{"name": "Ada", "age": 200, "tags": []}`, false, "above 150"}, + {`{"name": "Ada", "age": 3, "tags": [1]}`, false, "tags[0] is an integer, not string"}, + {`{"name": "Ada", "age": 3, "tags": [], "kind": "fish"}`, false, "not one of"}, + {`{"name": "", "age": 3, "tags": []}`, false, "fewer than 1"}, + {`not json at all`, false, "not JSON"}, + } { + v := scoreRule(c, tc.answer) + if (v.score == 1) != tc.pass || (tc.why != "" && !strings.Contains(v.detail, tc.why)) { + t.Errorf("%q: %v %q, wanted pass=%v saying %q", tc.answer, v.score, v.detail, tc.pass, tc.why) + } + } +} + +func TestASchemaUsingWhatIsNotCheckedIsRefusedWhenTheSuiteIsRead(t *testing.T) { + _, err := Parse("t.yaml", []byte("cases:\n - prompt: q\n json_schema: {oneOf: [{type: string}]}\n")) + if err == nil || !strings.Contains(err.Error(), "oneOf is not supported") { + t.Fatalf("a schema with oneOf was accepted: %v", err) + } +} + +func TestAJudgesReplyIsReadAsAScore(t *testing.T) { + for _, tc := range []struct { + reply string + want float64 + bad bool + }{ + {`{"score": 0.8, "reason": "mostly right"}`, 0.8, false}, + {"hmm\n```json\n{\"score\": 1, \"reason\": \"ok\"}\n```", 1, false}, + {`{"score": "0.25"}`, 0.25, false}, + {`{"pass": false, "reason": "wrong city"}`, 0, false}, + {`{"score": 7}`, 1, false}, + {`0.6`, 0.6, false}, + {`PASS`, 1, false}, + {`Fail: it named the wrong city.`, 0, false}, + {`I cannot say.`, 0, true}, + } { + v, err := readJudgement(tc.reply) + if (err != nil) != tc.bad || v.score != tc.want { + t.Errorf("%q: %v %v, wanted %v (unreadable: %v)", tc.reply, v.score, err, tc.want, tc.bad) + } + } +} + +func TestAReasoningModelsThinkingIsNotPartOfItsAnswer(t *testing.T) { + if got := cleanAnswer("\n17+25... carry the one\n\n\n42"); got != "42" { + t.Errorf("%q", got) + } + if got := cleanAnswer("still thinking when it was cut off"); got != "" { + t.Errorf("%q", got) + } +} diff --git a/internal/bench/eval/service.go b/internal/bench/eval/service.go new file mode 100644 index 0000000..ef4205f --- /dev/null +++ b/internal/bench/eval/service.go @@ -0,0 +1,434 @@ +package eval + +import ( + "context" + "crypto/rand" + "encoding/hex" + "errors" + "fmt" + "io/fs" + "log/slog" + "os" + "path/filepath" + "slices" + "sort" + "strings" + "sync" + "time" + + "gitea.mynodes.xyz/ilker/llmgw/internal/store" +) + +// The control plane's side: suites found on its disk or carried in the +// binary, runs started and followed, and kept in the store. + +// Store keeps finished runs. +type Store interface { + PutEvalRun(ctx context.Context, r store.EvalRun) error + EvalRuns(ctx context.Context, model string, limit int) ([]store.EvalRun, error) + EvalRunByID(ctx context.Context, id string) (store.EvalRun, error) +} + +// Deps are what the service runs suites with. +type Deps struct { + // Call asks a model: the router's Replay. + Call Caller + // Store keeps runs; without one they are kept here until a restart. + Store Store + // Dir is where suites are read from by name: control.eval_dir. + Dir string + // Builtin are the suites the binary carries, found when Dir has none + // of the name. + Builtin fs.FS + // Origin is where a catalog model's weights came from, when known. + Origin func(model string) (repo, revision string) + // CanaryOf is the model a name is the canary of, or "": a run on a + // canary is filed under the model it is a canary of. + CanaryOf func(name string) string + Log *slog.Logger +} + +// Request is a suite to run against a model: one on the control plane, by +// its name, or one sent whole. +type Request struct { + // Suite is a suite's name in the eval directory, or carried in the + // binary (arithmetic, json-format). + Suite string `json:"suite,omitempty"` + // Content is a suite's text instead, and File the name of the file it + // was read from, which names the suite unless it names itself. + Content string `json:"content,omitempty"` + File string `json:"file,omitempty"` + // Concurrency, Judge and MinScore are as Options has them. + Concurrency int `json:"concurrency,omitempty"` + Judge string `json:"judge,omitempty"` + MinScore float64 `json:"min_score,omitempty"` +} + +// Named says which suite a request is, for a step's detail. +func (r Request) Named() string { + if r.Suite != "" { + return r.Suite + } + return r.File +} + +// The states a run can be in. +const ( + Running = "running" + Done = "done" + Failed = "failed" +) + +// Live is a run as it stands: one running, with how many of its cases are +// done, or one finished. +type Live struct { + store.EvalRun + State string `json:"state"` + Done int `json:"done"` +} + +// SuiteInfo is a suite the control plane can run by name. +type SuiteInfo struct { + Name string `json:"name"` + // From is the file it is read from, or "built in". + From string `json:"from"` + Description string `json:"description,omitempty"` + Cases int `json:"cases"` + Threshold float64 `json:"threshold"` + Judge bool `json:"judge,omitempty"` + Hash string `json:"hash,omitempty"` + // Error is why the file cannot be run. + Error string `json:"error,omitempty"` +} + +// Service runs suites and remembers them. +type Service struct { + deps Deps + mu sync.Mutex + live map[string]*Live + // order is the live runs oldest first, so the finished ones past + // keepLive are forgotten — the store has them. + order []string +} + +const keepLive = 50 + +// NewService builds a service. +func NewService(d Deps) *Service { + if d.Log == nil { + d.Log = slog.Default() + } + return &Service{deps: d, live: map[string]*Live{}} +} + +var suiteExts = []string{".yaml", ".yml", ".jsonl", ".json"} + +// Load reads the suite a request names or carries. +func (s *Service) Load(r Request) (Suite, error) { + if r.Content != "" { + file := r.File + if file == "" { + file = "suite" + } + return Parse(file, []byte(r.Content)) + } + name := strings.TrimSpace(r.Suite) + if name == "" { + return Suite{}, errors.New("no suite: name one on the control plane, or send one's text") + } + // A name, not a path: this reads the control plane's disk on behalf of + // whoever holds the admin API, and only inside the eval directory. + if strings.ContainsAny(name, `/\`) || name == "." || name == ".." { + return Suite{}, fmt.Errorf("%q is not a suite's name: the control plane reads suites from %s by name; "+ + "send a file from elsewhere with its text", name, s.dirOrNone()) + } + tries := []string{name} + if !slices.Contains(suiteExts, filepath.Ext(name)) { + tries = nil + for _, e := range suiteExts { + tries = append(tries, name+e) + } + } + if s.deps.Dir != "" { + for _, t := range tries { + b, err := os.ReadFile(filepath.Join(s.deps.Dir, t)) + if err == nil { + return Parse(t, b) + } + if !errors.Is(err, fs.ErrNotExist) { + return Suite{}, err + } + } + } + if s.deps.Builtin != nil { + for _, t := range tries { + if b, err := fs.ReadFile(s.deps.Builtin, t); err == nil { + return Parse(t, b) + } + } + } + return Suite{}, fmt.Errorf("no suite called %s in %s or built in: `llmgw ctl eval --suites` lists them", name, s.dirOrNone()) +} + +func (s *Service) dirOrNone() string { + if s.deps.Dir == "" { + return "no eval directory" + } + return s.deps.Dir +} + +// Suites are those the control plane can run by name: its directory's, and +// the binary's that the directory does not replace. +func (s *Service) Suites() []SuiteInfo { + var out []SuiteInfo + seen := map[string]bool{} + add := func(fsys fs.FS, from func(string) string) { + entries, err := fs.ReadDir(fsys, ".") + if err != nil { + return + } + for _, e := range entries { + ext := filepath.Ext(e.Name()) + if e.IsDir() || !slices.Contains(suiteExts, ext) { + continue + } + name := strings.TrimSuffix(e.Name(), ext) + if seen[name] { + continue + } + seen[name] = true + info := SuiteInfo{Name: name, From: from(e.Name())} + b, err := fs.ReadFile(fsys, e.Name()) + var su Suite + if err == nil { + su, err = Parse(e.Name(), b) + } + if err != nil { + info.Error = err.Error() + } else { + info.Description, info.Cases, info.Threshold = su.Description, len(su.Cases), su.Threshold + info.Judge, info.Hash = su.NeedsJudge(), su.Hash + } + out = append(out, info) + } + } + if s.deps.Dir != "" { + add(os.DirFS(s.deps.Dir), func(n string) string { return filepath.Join(s.deps.Dir, n) }) + } + if s.deps.Builtin != nil { + add(s.deps.Builtin, func(string) string { return "built in" }) + } + sort.Slice(out, func(i, j int) bool { return out[i].Name < out[j].Name }) + return out +} + +func newID() string { + b := make([]byte, 4) + _, _ = rand.Read(b) + return fmt.Sprintf("eval-%s-%s", time.Now().UTC().Format("20060102-150405"), hex.EncodeToString(b)) +} + +// prepare reads the suite and checks it can be run against model, before +// anything is asked. +func (s *Service) prepare(model string, r Request) (Suite, error) { + if s.deps.Call == nil { + return Suite{}, errors.New("this control plane does not route requests, so it cannot ask a model") + } + if strings.TrimSpace(model) == "" { + return Suite{}, errors.New("no model to ask") + } + su, err := s.Load(r) + if err != nil { + return Suite{}, err + } + if r.MinScore < 0 || r.MinScore > 1 { + return Suite{}, fmt.Errorf("the minimum score %v is not 0 to 1", r.MinScore) + } + if su.NeedsJudge() && r.Judge == "" && su.Judge == "" { + for _, c := range su.Cases { + if c.Judge != "" && c.JudgeModel == "" { + return Suite{}, fmt.Errorf("case %s of %s is graded by a judge and the suite names none: give one "+ + "with --judge MODEL", c.ID, su.Name) + } + } + } + return su, nil +} + +// Check says whether a suite can be run against a model, without asking it +// anything: for a gate, refused before its job starts rather than after the +// new weights are fetched. +func (s *Service) Check(model string, r Request) error { + _, err := s.prepare(model, r) + return err +} + +// file fills in what a run is filed under: the model it is about, the one +// that answered, and where its weights came from. +func (s *Service) file(run *store.EvalRun, asked string) { + run.Model = asked + if s.deps.CanaryOf != nil { + if parent := s.deps.CanaryOf(asked); parent != "" { + run.Model, run.Target = parent, asked + } + } + if s.deps.Origin != nil { + run.Repo, run.Revision = s.deps.Origin(asked) + } +} + +// Start runs a suite against a model in the background and returns at once; +// Get follows it. +func (s *Service) Start(model string, r Request, by string) (Live, error) { + su, err := s.prepare(model, r) + if err != nil { + return Live{}, err + } + l := &Live{State: Running} + l.ID, l.Suite, l.SuiteHash, l.Cases, l.By, l.At = newID(), su.Name, su.Hash, len(su.Cases), by, time.Now().UTC() + s.file(&l.EvalRun, model) + s.mu.Lock() + s.keep(l) + s.mu.Unlock() + go func() { + _, _ = s.run(context.Background(), l, model, su, r) + }() + return s.snapshot(l), nil +} + +// Run runs a suite against a model and waits for it: for a gate, which +// cannot go on until it knows. The run is kept like any other; job is the +// update or promotion it gates. +func (s *Service) Run(ctx context.Context, model string, r Request, by, job string) (store.EvalRun, error) { + su, err := s.prepare(model, r) + if err != nil { + return store.EvalRun{}, err + } + l := &Live{State: Running} + l.ID, l.Suite, l.SuiteHash, l.Cases, l.By, l.Job, l.At = newID(), su.Name, su.Hash, len(su.Cases), by, job, time.Now().UTC() + s.file(&l.EvalRun, model) + s.mu.Lock() + s.keep(l) + s.mu.Unlock() + return s.run(ctx, l, model, su, r) +} + +func (s *Service) run(ctx context.Context, l *Live, model string, su Suite, r Request) (store.EvalRun, error) { + run, err := Evaluate(ctx, s.deps.Call, su, Options{Model: model, Concurrency: r.Concurrency, Judge: r.Judge, + MinScore: r.MinScore, Progress: func(done, _ int) { + s.mu.Lock() + l.Done = done + s.mu.Unlock() + }}) + s.mu.Lock() + if err != nil { + l.State, l.Error = Failed, err.Error() + out := l.EvalRun + s.mu.Unlock() + return out, err + } + run.ID, run.By, run.Job = l.ID, l.By, l.Job + run.Model, run.Target, run.Repo, run.Revision = l.Model, l.Target, l.Repo, l.Revision + s.mu.Unlock() + + // Kept before it is said to be done, so a history read in between does + // not find it in neither place. + if s.deps.Store != nil { + // Its own context: a gate's caller giving up does not unkeep a run + // that finished. + sctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) + defer cancel() + if err := s.deps.Store.PutEvalRun(sctx, run); err != nil { + s.deps.Log.Warn("could not keep an eval run", "run", run.ID, "model", run.Model, "err", err) + } + } + s.mu.Lock() + l.EvalRun, l.State, l.Done = run, Done, run.Cases + s.mu.Unlock() + s.deps.Log.Info("eval run finished", "run", run.ID, "model", run.Model, "target", run.Target, + "suite", run.Suite, "score", fmt.Sprintf("%.3f", run.Score), "threshold", run.Threshold, "passed", run.Passed) + return run, nil +} + +// keep remembers a run, and forgets the oldest finished ones past keepLive. +// Called with mu held. +func (s *Service) keep(l *Live) { + s.live[l.ID] = l + s.order = append(s.order, l.ID) + for len(s.order) > keepLive { + oldest := s.live[s.order[0]] + if oldest != nil && oldest.State == Running { + break + } + delete(s.live, s.order[0]) + s.order = s.order[1:] + } +} + +func (s *Service) snapshot(l *Live) Live { + s.mu.Lock() + defer s.mu.Unlock() + out := *l + out.Results = slices.Clone(l.Results) + return out +} + +// Get is a run by its id: one running or finished here, else one kept. +func (s *Service) Get(ctx context.Context, id string) (Live, error) { + s.mu.Lock() + l, ok := s.live[id] + s.mu.Unlock() + if ok { + return s.snapshot(l), nil + } + if s.deps.Store == nil { + return Live{}, store.ErrNotFound + } + r, err := s.deps.Store.EvalRunByID(ctx, id) + if err != nil { + return Live{}, err + } + return Live{EvalRun: r, State: Done, Done: r.Cases}, nil +} + +// History is a model's runs, newest first, without their cases: those +// running here, then those kept. +func (s *Service) History(ctx context.Context, model string, limit int) ([]Live, error) { + if limit <= 0 { + limit = 20 + } + var out []Live + seen := map[string]bool{} + s.mu.Lock() + for i := len(s.order) - 1; i >= 0; i-- { + l := s.live[s.order[i]] + if l == nil || (model != "" && l.Model != model && l.Target != model) { + continue + } + // Finished ones are listed from the store, when there is one, in + // its order. + if l.State != Running && s.deps.Store != nil { + continue + } + v := *l + v.Results = nil + out = append(out, v) + seen[l.ID] = true + } + s.mu.Unlock() + if s.deps.Store != nil { + runs, err := s.deps.Store.EvalRuns(ctx, model, limit) + if err != nil { + return out, err + } + for _, r := range runs { + if !seen[r.ID] { + out = append(out, Live{EvalRun: r, State: Done, Done: r.Cases}) + } + } + } + if len(out) > limit { + out = out[:limit] + } + return out, nil +} diff --git a/internal/bench/eval/suite.go b/internal/bench/eval/suite.go new file mode 100644 index 0000000..5dfae83 --- /dev/null +++ b/internal/bench/eval/suite.go @@ -0,0 +1,332 @@ +// Package eval asks a model a suite of questions and scores its answers, so +// new weights can be held to the quality of the old before they replace them. +// +// internal/bench measures how fast a replica answers; this measures whether +// the answers are any good. A repository that republishes its weights — a +// new quantisation, a fixed chat template — can make a model faster and +// worse at once, and nothing in the latency numbers says so. A suite is a +// handful of cases with known answers, scored by rules where a rule can +// decide (an exact string, a number, JSON of a shape) and by another model +// with a rubric where only judgement can. +package eval + +import ( + "bytes" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "io" + "path" + "regexp" + "strings" + + "go.yaml.in/yaml/v3" +) + +// Suite is a set of cases a model is asked, and the score it must reach. +type Suite struct { + // Name is what runs are filed under; the file's name without its + // extension unless the suite gives one. + Name string `json:"name,omitempty" yaml:"name"` + Description string `json:"description,omitempty" yaml:"description"` + // Threshold is the score, 0 to 1, the suite is passed at. Zero is 0.8. + Threshold float64 `json:"threshold,omitempty" yaml:"threshold"` + // Judge is the model asked to grade the cases scored by a rubric, unless + // a case names its own. + Judge string `json:"judge,omitempty" yaml:"judge"` + // System and MaxTokens are every case's unless it gives its own. + System string `json:"system,omitempty" yaml:"system"` + MaxTokens int `json:"max_tokens,omitempty" yaml:"max_tokens"` + Cases []Case `json:"cases" yaml:"cases"` + + // Hash is the sha256 of the text the suite was read from: two runs + // under one name compare only when they asked the same questions. + Hash string `json:"-" yaml:"-"` +} + +// Message is one turn of a conversation a case sends. +type Message struct { + Role string `json:"role" yaml:"role"` + Content string `json:"content" yaml:"content"` +} + +// Case is one question and how its answer is scored: by exactly one of +// Exact, Contains, Regex, Number, JSONSchema or Judge. +type Case struct { + // ID names the case in results; case-N, by its place, unless given. + ID string `json:"id,omitempty" yaml:"id"` + // Prompt is the user's message; Messages a whole conversation instead. + Prompt string `json:"prompt,omitempty" yaml:"prompt"` + Messages []Message `json:"messages,omitempty" yaml:"messages"` + System string `json:"system,omitempty" yaml:"system"` + // MaxTokens bounds the answer; the suite's, else 512. + MaxTokens int `json:"max_tokens,omitempty" yaml:"max_tokens"` + // Temperature is 0 unless given: a suite is a measurement, and sampling + // would make two runs of the same weights disagree. + Temperature *float64 `json:"temperature,omitempty" yaml:"temperature"` + // Weight is how much the case counts against the others; 1 unless given. + Weight float64 `json:"weight,omitempty" yaml:"weight"` + + // Exact is the whole answer, surrounding white space aside. + Exact *string `json:"exact,omitempty" yaml:"exact"` + // Contains are strings that must all be in the answer. + Contains StringList `json:"contains,omitempty" yaml:"contains"` + // Regex must match somewhere in the answer. + Regex string `json:"regex,omitempty" yaml:"regex"` + // IgnoreCase makes Exact, Contains and Regex blind to case. + IgnoreCase bool `json:"ignore_case,omitempty" yaml:"ignore_case"` + // Number is the value the last number in the answer must be, within + // Tolerance either side. + Number *float64 `json:"number,omitempty" yaml:"number"` + Tolerance float64 `json:"tolerance,omitempty" yaml:"tolerance"` + // JSONSchema is a schema the answer, read as JSON, must satisfy. + JSONSchema any `json:"json_schema,omitempty" yaml:"json_schema"` + // Judge is a rubric another model grades the answer by, 0 to 1; + // JudgeModel names that model, else the suite's judge. + Judge string `json:"judge,omitempty" yaml:"judge"` + JudgeModel string `json:"judge_model,omitempty" yaml:"judge_model"` + + re *regexp.Regexp + schema *schema +} + +// StringList is one string or a list of them. +type StringList []string + +// UnmarshalYAML takes a string or a list. +func (l *StringList) UnmarshalYAML(n *yaml.Node) error { + if n.Kind == yaml.ScalarNode { + *l = StringList{n.Value} + return nil + } + var v []string + if err := n.Decode(&v); err != nil { + return err + } + *l = v + return nil +} + +// UnmarshalJSON takes a string or a list. +func (l *StringList) UnmarshalJSON(b []byte) error { + var s string + if json.Unmarshal(b, &s) == nil { + *l = StringList{s} + return nil + } + var v []string + if err := json.Unmarshal(b, &v); err != nil { + return errors.New("contains is a string or a list of strings") + } + *l = v + return nil +} + +// Kind is how a case is scored: exact, contains, regex, number, +// json_schema or judge. +func (c Case) Kind() string { + var kinds []string + if c.Exact != nil { + kinds = append(kinds, "exact") + } + if len(c.Contains) > 0 { + kinds = append(kinds, "contains") + } + if c.Regex != "" { + kinds = append(kinds, "regex") + } + if c.Number != nil { + kinds = append(kinds, "number") + } + if c.JSONSchema != nil { + kinds = append(kinds, "json_schema") + } + if c.Judge != "" { + kinds = append(kinds, "judge") + } + return strings.Join(kinds, "+") +} + +// DefaultThreshold is the score a suite that names none is passed at. +const DefaultThreshold = 0.8 + +// defaultMaxTokens bounds an answer when neither the case nor the suite do. +const defaultMaxTokens = 512 + +// Parse reads a suite: YAML (a mapping with its cases under cases:), or +// JSON lines (one case on each, and optionally a line without a prompt or +// messages that holds the suite's own settings). The name is the file's, +// used when the suite gives none. +func Parse(file string, data []byte) (Suite, error) { + var s Suite + var err error + if isJSONLines(data) { + s, err = parseJSONLines(data) + } else { + dec := yaml.NewDecoder(bytes.NewReader(data)) + dec.KnownFields(true) + err = dec.Decode(&s) + if errors.Is(err, io.EOF) { + err = errors.New("the file is empty") + } + } + if err != nil { + return Suite{}, fmt.Errorf("%s: %w", file, err) + } + if s.Name == "" { + base := path.Base(strings.ReplaceAll(file, "\\", "/")) + s.Name = strings.TrimSuffix(base, path.Ext(base)) + } + sum := sha256.Sum256(data) + s.Hash = hex.EncodeToString(sum[:]) + if err := s.check(); err != nil { + return Suite{}, fmt.Errorf("%s: %w", file, err) + } + return s, nil +} + +// isJSONLines is whether every line that is not blank is a JSON object, and +// there is more than one, or the one there is is a case rather than a whole +// suite. +func isJSONLines(data []byte) bool { + n := 0 + for _, line := range bytes.Split(data, []byte("\n")) { + line = bytes.TrimSpace(line) + if len(line) == 0 { + continue + } + if line[0] != '{' || !json.Valid(line) { + return false + } + n++ + } + if n == 1 { + var probe struct { + Cases json.RawMessage `json:"cases"` + } + _ = json.Unmarshal(bytes.TrimSpace(data), &probe) + return probe.Cases == nil + } + return n > 0 +} + +func parseJSONLines(data []byte) (Suite, error) { + var s Suite + for i, line := range bytes.Split(data, []byte("\n")) { + line = bytes.TrimSpace(line) + if len(line) == 0 { + continue + } + var probe struct { + Prompt *string `json:"prompt"` + Messages json.RawMessage `json:"messages"` + } + _ = json.Unmarshal(line, &probe) + dec := json.NewDecoder(bytes.NewReader(line)) + dec.DisallowUnknownFields() + if probe.Prompt == nil && probe.Messages == nil { + cases := s.Cases + if err := dec.Decode(&s); err != nil { + return Suite{}, fmt.Errorf("line %d: %w", i+1, err) + } + s.Cases = append(cases, s.Cases...) + continue + } + var c Case + if err := dec.Decode(&c); err != nil { + return Suite{}, fmt.Errorf("line %d: %w", i+1, err) + } + s.Cases = append(s.Cases, c) + } + return s, nil +} + +// check fills in what a suite left out and refuses what cannot be run. +func (s *Suite) check() error { + if len(s.Cases) == 0 { + return errors.New("no cases: a suite has them under cases:, or one on each line of JSON") + } + if s.Threshold < 0 || s.Threshold > 1 { + return fmt.Errorf("threshold %v is not 0 to 1", s.Threshold) + } + if s.Threshold == 0 { + s.Threshold = DefaultThreshold + } + seen := map[string]bool{} + for i := range s.Cases { + c := &s.Cases[i] + if c.ID == "" { + c.ID = fmt.Sprintf("case-%d", i+1) + } + if seen[c.ID] { + return fmt.Errorf("two cases are called %s", c.ID) + } + seen[c.ID] = true + if err := c.check(); err != nil { + return fmt.Errorf("case %s: %w", c.ID, err) + } + } + return nil +} + +func (c *Case) check() error { + if strings.TrimSpace(c.Prompt) == "" && len(c.Messages) == 0 { + return errors.New("no prompt and no messages: nothing to ask") + } + if c.Prompt != "" && len(c.Messages) > 0 { + return errors.New("a prompt and messages both: give one") + } + for _, m := range c.Messages { + switch m.Role { + case "system", "user", "assistant": + default: + return fmt.Errorf("a message's role is system, user or assistant, not %q", m.Role) + } + } + switch k := c.Kind(); { + case k == "": + return errors.New("no way to score it: give one of exact, contains, regex, number, json_schema or judge") + case strings.Contains(k, "+"): + return fmt.Errorf("scored by %s: give one", strings.ReplaceAll(k, "+", " and ")) + } + if c.Weight < 0 { + return fmt.Errorf("weight %v is negative", c.Weight) + } + if c.Weight == 0 { + c.Weight = 1 + } + if c.Tolerance < 0 { + return fmt.Errorf("tolerance %v is negative", c.Tolerance) + } + if c.Regex != "" { + expr := c.Regex + if c.IgnoreCase { + expr = "(?i)" + expr + } + re, err := regexp.Compile(expr) + if err != nil { + return fmt.Errorf("the regex: %w", err) + } + c.re = re + } + if c.JSONSchema != nil { + sc, err := compileSchema(c.JSONSchema) + if err != nil { + return fmt.Errorf("the JSON schema: %w", err) + } + c.schema = sc + } + return nil +} + +// NeedsJudge is whether any case is graded by a model. +func (s Suite) NeedsJudge() bool { + for _, c := range s.Cases { + if c.Judge != "" { + return true + } + } + return false +} diff --git a/internal/cli/ctl.go b/internal/cli/ctl.go index 4cf0b82..91d6062 100644 --- a/internal/cli/ctl.go +++ b/internal/cli/ctl.go @@ -147,6 +147,7 @@ func newCtlCmd() *cobra.Command { // cert works on files rather than the admin API, so --server does not // apply to it; it lives here because this is where operators look. newBenchCmd(client, &asJSON), + newEvalCmd(client, &asJSON), newStatusCmd(client, &asJSON), newModelCmd(client, &asJSON), newScaleCmd(client), diff --git a/internal/cli/ctl_eval.go b/internal/cli/ctl_eval.go new file mode 100644 index 0000000..86a533f --- /dev/null +++ b/internal/cli/ctl_eval.go @@ -0,0 +1,408 @@ +package cli + +import ( + "encoding/json" + "errors" + "fmt" + "io" + "io/fs" + "log/slog" + "net/url" + "os" + "path/filepath" + "strconv" + "strings" + "time" + + "github.com/spf13/cobra" + + llmgw "gitea.mynodes.xyz/ilker/llmgw" + "gitea.mynodes.xyz/ilker/llmgw/internal/bench/eval" + "gitea.mynodes.xyz/ilker/llmgw/internal/config" + "gitea.mynodes.xyz/ilker/llmgw/internal/control" + "gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi" + "gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog" + "gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve" + "gitea.mynodes.xyz/ilker/llmgw/internal/router" +) + +// newEvalService is the control plane's runner of quality suites: through +// the router as its own requests, with the suites of control.eval_dir and +// those the binary carries, kept in the store when there is one. +func newEvalService(cfg config.Server, rt *router.Router, admin *control.Controller, log *slog.Logger) *eval.Service { + builtin, err := fs.Sub(llmgw.EvalSuites, "deploy/eval") + if err != nil { + builtin = nil + } + d := eval.Deps{Call: rt.Replay, Dir: cfg.Control.EvalDir, Builtin: builtin, Log: log, + Origin: func(name string) (string, string) { + if m, ok := admin.Catalog().Resolve(name); ok && m.Origin != nil { + return m.Origin.Repo, m.Origin.Revision + } + return "", "" + }, + CanaryOf: func(name string) string { return admin.Catalog().CanaryOf(name) }, + } + // Only a store that is there: a nil one in the interface would look + // like one. + if st := rt.Store(); st != nil { + d.Store = st + } + return eval.NewService(d) +} + +// suiteRequest is a suite named on the command line: a file here, sent +// whole, or else the name of one on the control plane. +func suiteRequest(arg string) (eval.Request, error) { + if st, err := os.Stat(arg); err == nil && !st.IsDir() { + b, err := os.ReadFile(arg) + if err != nil { + return eval.Request{}, err + } + // Read here first, so a mistake in it is said before anything is sent. + if _, err := eval.Parse(filepath.Base(arg), b); err != nil { + return eval.Request{}, err + } + return eval.Request{Content: string(b), File: filepath.Base(arg)}, nil + } + if strings.ContainsAny(arg, `/\`) { + return eval.Request{}, fmt.Errorf("%s: no such file here", arg) + } + return eval.Request{Suite: arg}, nil +} + +func newEvalCmd(client func() *ctlClient, asJSON *bool) *cobra.Command { + var ( + suite, judge, run string + history, suites, detach bool + concurrency, limit int + minScore float64 + ) + c := &cobra.Command{ + Use: "eval MODEL --suite FILE|NAME", + Short: "Ask a model a quality suite and score its answers", + Long: "A suite is a set of cases — a prompt or a conversation, and how its answer\n" + + "is scored: exact, contains, regex, a number within a tolerance, JSON that\n" + + "satisfies a schema, or a rubric another model on the cluster grades — with\n" + + "a weight each and a score, 0 to 1, the suite is passed at (docs/install.md,\n" + + "\"Quality suites\").\n\n" + + "It runs on the control plane, which asks the model through the router as\n" + + "its own requests, as a replay does: past admission, quotas, accounting and\n" + + "the guardrails, so the score is the weights' and not a tenant's limits'.\n" + + "--suite is a file here, sent whole, or the name of a suite in the control\n" + + "plane's control.eval_dir (/etc/llmgw/eval) or built in: arithmetic,\n" + + "json-format. Every run is kept, with the revision of the weights that\n" + + "answered; --history lists them.\n\n" + + "It exits non-zero when the model scores below the suite's threshold, or\n" + + "--min-score.", + Example: " llmgw ctl eval qwen-7b --suite arithmetic\n" + + " llmgw ctl eval qwen-7b --suite ./support.yaml --judge llama-70b --concurrency 8\n" + + " llmgw ctl eval --history qwen-7b\n" + + " llmgw ctl eval --run eval-20261004-101500-1a2b3c4d\n" + + " llmgw ctl eval --suites", + Args: cobra.MaximumNArgs(1), + RunE: func(cmd *cobra.Command, args []string) error { + out := cmd.OutOrStdout() + switch { + case suites: + var list []eval.SuiteInfo + if err := client().get(cmd.Context(), "/api/v1/eval/suites", &list); err != nil { + return err + } + if *asJSON { + return emitJSON(list) + } + w := tabTo(out) + fmt.Fprintln(w, "SUITE\tCASES\tPASS AT\tJUDGE\tFROM\tABOUT") + for _, s := range list { + if s.Error != "" { + fmt.Fprintf(w, "%s\t-\t-\t-\t%s\tcannot be run: %s\n", s.Name, s.From, truncate(s.Error, 70)) + continue + } + fmt.Fprintf(w, "%s\t%d\t%.2f\t%s\t%s\t%s\n", s.Name, s.Cases, s.Threshold, yesNo(s.Judge), s.From, + truncate(s.Description, 60)) + } + return w.Flush() + case run != "": + var l eval.Live + if err := client().get(cmd.Context(), "/api/v1/eval/runs/"+url.PathEscape(run), &l); err != nil { + return err + } + if l.State == eval.Running && !*asJSON { + return followEval(cmd, client, l, false) + } + return showEval(cmd, l, *asJSON) + case history: + q := url.Values{} + if len(args) == 1 { + q.Set("model", args[0]) + } + if limit > 0 { + q.Set("limit", strconv.Itoa(limit)) + } + var runs []eval.Live + if err := client().get(cmd.Context(), "/api/v1/eval/runs?"+q.Encode(), &runs); err != nil { + return err + } + if *asJSON { + return emitJSON(runs) + } + return printEvalHistory(out, runs) + } + if len(args) != 1 { + return errors.New("name the model to ask: llmgw ctl eval MODEL --suite FILE|NAME") + } + if suite == "" { + return errors.New("--suite is required: a file, or a suite's name on the control plane (`llmgw ctl eval --suites`)") + } + req, err := suiteRequest(suite) + if err != nil { + return err + } + req.Concurrency, req.Judge, req.MinScore = concurrency, judge, minScore + body, _ := json.Marshal(req) + var l eval.Live + if err := client().post(cmd.Context(), "/api/v1/models/"+url.PathEscape(args[0])+"/eval", body, &l); err != nil { + return err + } + if detach { + fmt.Fprintf(out, "started %s on the control plane\nfollow it: llmgw ctl eval --run %s\n", l.ID, l.ID) + return nil + } + return followEval(cmd, client, l, *asJSON) + }, + } + f := c.Flags() + f.StringVar(&suite, "suite", "", "the suite: a file here, or a suite's name on the control plane") + f.IntVar(&concurrency, "concurrency", 4, "cases asked at once") + f.StringVar(&judge, "judge", "", "the model that grades rubric cases, instead of the suite's judge") + f.Float64Var(&minScore, "min-score", 0, "the score to pass at, 0 to 1, instead of the suite's threshold") + f.BoolVar(&detach, "detach", false, "start it and return: it runs on the control plane") + f.BoolVar(&history, "history", false, "list the runs kept, of MODEL or of every model") + f.IntVar(&limit, "limit", 20, "with --history, how many runs") + f.StringVar(&run, "run", "", "show one run, by its id, with each case") + f.BoolVar(&suites, "suites", false, "list the suites the control plane can run by name") + return c +} + +func yesNo(b bool) string { + if b { + return "yes" + } + return "-" +} + +// followEval waits for a run, saying how far it has got, then shows it. +func followEval(cmd *cobra.Command, client func() *ctlClient, l eval.Live, asJSON bool) error { + out := cmd.OutOrStdout() + if !asJSON { + fmt.Fprintf(out, "run %s: %s asked %s, %d cases\n", l.ID, evalTarget(l), l.Suite, l.Cases) + } + shown := -1 + for l.State == eval.Running { + if !asJSON && l.Done != shown { + fmt.Fprintf(out, " %d of %d answered\n", l.Done, l.Cases) + shown = l.Done + } + select { + case <-cmd.Context().Done(): + return cmd.Context().Err() + case <-time.After(time.Second): + } + if err := client().get(cmd.Context(), "/api/v1/eval/runs/"+url.PathEscape(l.ID), &l); err != nil { + return err + } + } + if !asJSON { + fmt.Fprintln(out) + } + return showEval(cmd, l, asJSON) +} + +func evalTarget(l eval.Live) string { + if l.Target != "" { + return l.Target + } + return l.Model +} + +// showEval prints a finished run's cases and score, and fails below it. +func showEval(cmd *cobra.Command, l eval.Live, asJSON bool) error { + out := cmd.OutOrStdout() + if asJSON { + if err := emitJSON(l); err != nil { + return err + } + } else { + if l.State == eval.Failed { + return fmt.Errorf("the run could not be scored: %s", l.Error) + } + w := tabTo(out) + fmt.Fprintln(w, "CASE\tKIND\tWEIGHT\tSCORE\tTIME\tANSWER\tWHY") + for _, r := range l.Results { + score := fmt.Sprintf("%.2f", r.Score) + if !r.Passed { + score += " ✗" + } + why := r.Detail + if r.Error != "" { + why = "error: " + r.Error + } + fmt.Fprintf(w, "%s\t%s\t%g\t%s\t%s\t%s\t%s\n", r.ID, r.Kind, r.Weight, score, + (time.Duration(r.TookMs) * time.Millisecond).Round(10*time.Millisecond), dash(truncate(oneLine(r.Answer), 30)), + truncate(why, 70)) + } + if err := w.Flush(); err != nil { + return err + } + fmt.Fprintln(out) + fmt.Fprintln(out, evalSummary(l)) + if l.Error != "" { + fmt.Fprintln(out, l.Error) + } + } + if !l.Passed { + return fmt.Errorf("%s scored %.2f on %s, below %.2f", evalTarget(l), l.Score, l.Suite, l.Threshold) + } + return nil +} + +func oneLine(s string) string { return strings.Join(strings.Fields(s), " ") } + +// evalSummary is a run in a sentence. +func evalSummary(l eval.Live) string { + verdict := "passed" + if !l.Passed { + verdict = "failed" + } + s := fmt.Sprintf("%s scored %.2f on %s: %d of %d cases passed, %.2f asked: %s", evalTarget(l), l.Score, l.Suite, + l.Cases-l.Below, l.Cases, l.Threshold, verdict) + if l.Revision != "" { + s += fmt.Sprintf(" (weights %s at %s)", dash(l.Repo), catalog.ShortRevision(l.Revision)) + } + return s + "." +} + +func printEvalHistory(out io.Writer, runs []eval.Live) error { + if len(runs) == 0 { + fmt.Fprintln(out, "No runs kept: `llmgw ctl eval MODEL --suite NAME` makes one.") + return nil + } + w := tabTo(out) + fmt.Fprintln(w, "WHEN\tRUN\tMODEL\tWEIGHTS\tSUITE\tSCORE\tPASS AT\tRESULT\tFOR") + for _, r := range runs { + model := r.Model + if r.Target != "" { + model += " (" + r.Target + ")" + } + result := "passed" + switch { + case r.State == eval.Running: + result = fmt.Sprintf("running, %d of %d", r.Done, r.Cases) + case r.State == eval.Failed: + result = "could not run" + case !r.Passed: + result = "failed" + } + score := fmt.Sprintf("%.2f", r.Score) + if r.State == eval.Running { + score = "-" + } + forWhat := r.Job + if forWhat == "" { + forWhat = r.By + } + fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\t%.2f\t%s\t%s\n", ago(r.At, time.Now()), r.ID, model, + dash(catalog.ShortRevision(r.Revision)), r.Suite, score, r.Threshold, result, dash(forWhat)) + } + return w.Flush() +} + +// gateFlags are how an update or a promotion is held to a quality suite. +type gateFlags struct { + suite, judge string + minScore float64 + skip bool +} + +func (g *gateFlags) add(c *cobra.Command) { + f := c.Flags() + f.StringVar(&g.suite, "eval", "", "a quality suite the new weights must pass first: a file here, or a suite's name on the control plane") + f.Float64Var(&g.minScore, "min-score", 0, "with --eval, the score to pass at, 0 to 1; the suite's threshold otherwise") + f.StringVar(&g.judge, "judge", "", "with --eval, the model that grades rubric cases") + f.BoolVar(&g.skip, "skip-eval", false, "move without the model's own quality suite (`llmgw ctl model gate`)") +} + +// body is the change, as the API takes it. +func (g gateFlags) body() ([]byte, error) { + var ch modelserve.Change + if g.suite != "" { + if g.skip { + return nil, errors.New("--eval and --skip-eval together: give one") + } + r, err := suiteRequest(g.suite) + if err != nil { + return nil, err + } + r.MinScore, r.Judge = g.minScore, g.judge + ch.Eval = &r + } else if g.minScore != 0 || g.judge != "" { + return nil, errors.New("--min-score and --judge go with --eval") + } + ch.SkipEval = g.skip + return json.Marshal(ch) +} + +func newModelGateCmd(client func() *ctlClient) *cobra.Command { + var ( + g adminapi.EvalGateRequest + off bool + ) + c := &cobra.Command{ + Use: "gate MODEL --eval SUITE [--min-score N]", + Short: "Hold a model's updates and promotions to a quality suite", + Long: "From then on `llmgw ctl model update` and `model promote` (and the\n" + + "console's) ask the new weights the suite before they replace the old,\n" + + "and stop below its score: an update runs them first as a canary given\n" + + "none of the model's requests, and takes it out again when it fails. The\n" + + "suite is one on the control plane, by name: in control.eval_dir, or\n" + + "built in. --skip-eval on an update moves without it, once; --off takes it\n" + + "away.", + Example: " llmgw ctl model gate qwen-7b --eval arithmetic --min-score 0.8\n" + + " llmgw ctl model gate qwen-7b --off", + Args: cobra.ExactArgs(1), + RunE: func(cmd *cobra.Command, args []string) error { + path := "/api/v1/models/" + url.PathEscape(args[0]) + "/eval-gate" + var out struct { + Model string `json:"model"` + Eval *catalog.EvalGate `json:"eval"` + } + if off { + if err := client().delete(cmd.Context(), path, &out); err != nil { + return err + } + fmt.Fprintf(cmd.OutOrStdout(), "%s's updates are no longer held to a quality suite\n", args[0]) + return nil + } + if g.Suite == "" { + return errors.New("--eval names the suite, or --off takes the gate away") + } + body, _ := json.Marshal(g) + if err := client().put(cmd.Context(), path, body, &out); err != nil { + return err + } + at := "the suite's threshold" + if g.MinScore > 0 { + at = strconv.FormatFloat(g.MinScore, 'f', -1, 64) + } + fmt.Fprintf(cmd.OutOrStdout(), "%s's new weights must score at least %s on %s before they replace the old\n", + args[0], at, g.Suite) + return nil + }, + } + c.Flags().StringVar(&g.Suite, "eval", "", "the suite: its name on the control plane") + c.Flags().Float64Var(&g.MinScore, "min-score", 0, "the score to pass at, 0 to 1; the suite's threshold otherwise") + c.Flags().StringVar(&g.Judge, "judge", "", "the model that grades the suite's rubric cases") + c.Flags().BoolVar(&off, "off", false, "take the gate away") + return c +} diff --git a/internal/cli/ctl_eval_test.go b/internal/cli/ctl_eval_test.go new file mode 100644 index 0000000..19046b3 --- /dev/null +++ b/internal/cli/ctl_eval_test.go @@ -0,0 +1,123 @@ +package cli + +import ( + "bytes" + "context" + "encoding/json" + "io" + "io/fs" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "strings" + "sync" + "testing" + + llmgw "gitea.mynodes.xyz/ilker/llmgw" + "gitea.mynodes.xyz/ilker/llmgw/internal/bench/eval" + "gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve" + "gitea.mynodes.xyz/ilker/llmgw/internal/store" +) + +func TestTheSuitesTheBinaryCarriesCanBeRun(t *testing.T) { + sub, err := fs.Sub(llmgw.EvalSuites, "deploy/eval") + if err != nil { + t.Fatal(err) + } + got := eval.NewService(eval.Deps{Builtin: sub}).Suites() + var names []string + for _, s := range got { + if s.Error != "" || s.Cases < 5 || s.Judge { + t.Errorf("%s: %+v", s.Name, s) + } + names = append(names, s.Name) + } + if strings.Join(names, " ") != "arithmetic json-format" { + t.Errorf("built in: %v", names) + } +} + +func TestCtlEvalSendsAFileAndFailsBelowTheScore(t *testing.T) { + var mu sync.Mutex + var sent []string + finished := eval.Live{EvalRun: store.EvalRun{ID: "eval-1", Model: "qwen", Suite: "mine", Score: 0.5, + Threshold: 0.8, Cases: 2, Below: 1, Revision: "abcdef1234567", Repo: "o/r", Results: []store.EvalCase{ + {ID: "add", Kind: "number", Weight: 1, Score: 1, Passed: true, Answer: "42", Detail: "read 42"}, + {ID: "mul", Kind: "number", Weight: 1, Score: 0, Answer: "43", Detail: "read 43, wanted 42"}}}, + State: eval.Done, Done: 2} + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + b, _ := io.ReadAll(r.Body) + mu.Lock() + sent = append(sent, r.Method+" "+r.URL.Path+" "+string(b)) + mu.Unlock() + switch r.URL.Path { + case "/api/v1/models/qwen/eval": + w.WriteHeader(http.StatusAccepted) + _ = json.NewEncoder(w).Encode(eval.Live{EvalRun: store.EvalRun{ID: "eval-1", Model: "qwen", Cases: 2}, + State: eval.Running}) + case "/api/v1/eval/runs/eval-1": + _ = json.NewEncoder(w).Encode(finished) + case "/api/v1/models/qwen/update": + w.WriteHeader(http.StatusAccepted) + _ = json.NewEncoder(w).Encode(modelserve.Job{ID: "update-1", Model: "qwen", State: modelserve.Done}) + default: + http.NotFound(w, r) + } + })) + t.Cleanup(srv.Close) + file := filepath.Join(t.TempDir(), "mine.yaml") + if err := os.WriteFile(file, []byte("cases:\n - prompt: 40 + 2?\n number: 42\n"), 0o600); err != nil { + t.Fatal(err) + } + ctl := func(args ...string) (string, error) { + cmd := newCtlCmd() + var out bytes.Buffer + cmd.SetOut(&out) + cmd.SetErr(&out) + cmd.SetArgs(append([]string{"--server", srv.URL}, args...)) + err := cmd.ExecuteContext(context.Background()) + return out.String(), err + } + + out, err := ctl("eval", "qwen", "--suite", file, "--concurrency", "2") + if err == nil || !strings.Contains(err.Error(), "qwen scored 0.50 on mine, below 0.80") { + t.Fatalf("a run below its score did not fail: %v\n%s", err, out) + } + for _, want := range []string{"read 43, wanted 42", "1 of 2 cases passed", "weights o/r at abcdef1"} { + if !strings.Contains(out, want) { + t.Errorf("the output lacks %q:\n%s", want, out) + } + } + mu.Lock() + var req eval.Request + _ = json.Unmarshal([]byte(strings.SplitN(sent[0], " ", 3)[2]), &req) + mu.Unlock() + if req.File != "mine.yaml" || !strings.Contains(req.Content, "40 + 2?") || req.Concurrency != 2 { + t.Errorf("sent %+v", req) + } + + // A broken file is said here, before anything is sent. + if err := os.WriteFile(file, []byte("cases: []\n"), 0o600); err != nil { + t.Fatal(err) + } + if _, err := ctl("eval", "qwen", "--suite", file); err == nil || !strings.Contains(err.Error(), "no cases") { + t.Errorf("%v", err) + } + + // An update held to a suite by name carries it. + if _, err := ctl("model", "update", "qwen", "--eval", "arithmetic", "--min-score", "0.9"); err != nil { + t.Fatal(err) + } + mu.Lock() + last := sent[len(sent)-1] + mu.Unlock() + var ch modelserve.Change + if err := json.Unmarshal([]byte(strings.SplitN(last, " ", 3)[2]), &ch); err != nil || ch.Eval == nil || + ch.Eval.Suite != "arithmetic" || ch.Eval.MinScore != 0.9 { + t.Errorf("sent %s", last) + } + if _, err := ctl("model", "update", "qwen", "--min-score", "0.9"); err == nil { + t.Error("--min-score without --eval was sent") + } +} diff --git a/internal/cli/model.go b/internal/cli/model.go index 2a19bff..579e6af 100644 --- a/internal/cli/model.go +++ b/internal/cli/model.go @@ -37,7 +37,8 @@ func newModelCmd(client func() *ctlClient, asJSON *bool) *cobra.Command { newTargetCmd(client, asJSON), newAdaptersCmd(client), newModelServeCmd(client), newModelUpdatesCmd(client, asJSON), newModelUpdateCmd(client), newModelRollbackCmd(client), newModelOriginCmd(client), - newModelCanaryCmd(client), newModelPromoteCmd(client), newModelCanariesCmd(client, asJSON)) + newModelCanaryCmd(client), newModelPromoteCmd(client), newModelCanariesCmd(client, asJSON), + newModelGateCmd(client)) return c } diff --git a/internal/cli/model_canary.go b/internal/cli/model_canary.go index ad353e9..0fa9f91 100644 --- a/internal/cli/model_canary.go +++ b/internal/cli/model_canary.go @@ -83,17 +83,29 @@ func newModelCanaryCmd(client func() *ctlClient) *cobra.Command { } func newModelPromoteCmd(client func() *ctlClient) *cobra.Command { - var detach bool + var ( + detach bool + gate gateFlags + ) c := &cobra.Command{ Use: "promote MODEL", Short: "Move a model to its canary's weights, and take the canary out", Long: "The update the canary was a trial of: the catalog names the canary's\n" + "files, the model's replicas are replaced one at a time, and the canary\n" + - "is removed. `llmgw ctl model rollback MODEL` goes back.", + "is removed. `llmgw ctl model rollback MODEL` goes back.\n\n" + + "Held to a quality suite (--eval, or the model's own: `llmgw ctl model\n" + + "gate`), the canary is asked it first, and below its score nothing\n" + + "moves: the canary stays, with its share, until it is taken out.", + Example: " llmgw ctl model promote qwen-7b\n" + + " llmgw ctl model promote qwen-7b --eval ./support.yaml --min-score 0.9 --judge llama-70b", Args: cobra.ExactArgs(1), RunE: func(cmd *cobra.Command, args []string) error { + body, err := gate.body() + if err != nil { + return err + } var job modelserve.Job - if err := client().post(cmd.Context(), "/api/v1/models/"+args[0]+"/promote", []byte("{}"), &job); err != nil { + if err := client().post(cmd.Context(), "/api/v1/models/"+args[0]+"/promote", body, &job); err != nil { return err } if detach { @@ -103,6 +115,7 @@ func newModelPromoteCmd(client func() *ctlClient) *cobra.Command { }, } c.Flags().BoolVar(&detach, "detach", false, "start it and return: it runs on the control plane; `llmgw ctl jobs` follows it") + gate.add(c) return c } diff --git a/internal/cli/model_update.go b/internal/cli/model_update.go index 1795956..74aa6f1 100644 --- a/internal/cli/model_update.go +++ b/internal/cli/model_update.go @@ -82,7 +82,10 @@ func updateCell(u modelserve.UpdateInfo) string { } func newModelUpdateCmd(client func() *ctlClient) *cobra.Command { - var detach bool + var ( + detach bool + gate gateFlags + ) c := &cobra.Command{ Use: "update MODEL", Short: "Move a model to the weights its repository publishes now", @@ -90,12 +93,23 @@ func newModelUpdateCmd(client func() *ctlClient) *cobra.Command { "checked against the registry's checksums and the nodes' llama.cpp; then\n" + "the catalog names them and the replicas are replaced one at a time, each\n" + "new one up before an old one stops, and one is asked to say hello.\n\n" + + "Held to a quality suite (--eval, or the model's own: `llmgw ctl model\n" + + "gate`), the new weights first run beside the old as MODEL-canary, given\n" + + "none of its requests, and are asked the suite; below its score they are\n" + + "taken out again and the model keeps its weights. That needs room for one\n" + + "more replica while it runs.\n\n" + "The old files stay on the mirror: `llmgw ctl model rollback MODEL` goes\n" + "back to them. A model from server.yaml is changed there instead.", + Example: " llmgw ctl model update qwen-7b\n" + + " llmgw ctl model update qwen-7b --eval arithmetic --min-score 0.8", Args: cobra.ExactArgs(1), RunE: func(cmd *cobra.Command, args []string) error { + body, err := gate.body() + if err != nil { + return err + } var job modelserve.Job - if err := client().post(cmd.Context(), "/api/v1/models/"+args[0]+"/update", []byte("{}"), &job); err != nil { + if err := client().post(cmd.Context(), "/api/v1/models/"+args[0]+"/update", body, &job); err != nil { return err } if detach { @@ -105,6 +119,7 @@ func newModelUpdateCmd(client func() *ctlClient) *cobra.Command { }, } c.Flags().BoolVar(&detach, "detach", false, "start it and return: it runs on the control plane; `llmgw ctl jobs` follows it") + gate.add(c) return c } diff --git a/internal/cli/server.go b/internal/cli/server.go index d155582..0a4d197 100644 --- a/internal/cli/server.go +++ b/internal/cli/server.go @@ -32,6 +32,7 @@ import ( llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1" "gitea.mynodes.xyz/ilker/llmgw/internal/audit" + "gitea.mynodes.xyz/ilker/llmgw/internal/bench/eval" "gitea.mynodes.xyz/ilker/llmgw/internal/config" "gitea.mynodes.xyz/ilker/llmgw/internal/control" "gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi" @@ -286,6 +287,7 @@ func newServerCmd() *cobra.Command { var ( adminAPI *adminapi.API catalogSvc *modelsvc.Service + evals *eval.Service ) if admin != nil { api := adminapi.New(admin) @@ -303,6 +305,9 @@ func newServerCmd() *cobra.Command { api.SetReleases(cfg.Releases) api.SetNotifier(notifier) api.SetReplayer(rt.Replay) + // Quality suites, asked of a model the way a replay is. + evals = newEvalService(cfg, rt, admin, log) + api.SetEval(evals) if st := rt.Store(); st != nil && gate != nil { api.SetGuard(&adminapi.GuardAdmin{Library: cfg.Guard, Store: st, Invalidate: gate.Invalidate}) } @@ -418,6 +423,8 @@ func newServerCmd() *cobra.Command { Pool: adminAPI.PoolNodes, Notify: notifier, // Canaries: how the router saw each side serve. CanaryStats: rt.CanaryStats, ResetCanary: rt.ResetCanary, + // Updates and promotions held to a quality suite. + Eval: evals, }) go serve.WatchCanaries(ctx, 30*time.Second) adminAPI.SetServe(serve) diff --git a/internal/cli/tui_actions.go b/internal/cli/tui_actions.go index b73c109..0d88fbc 100644 --- a/internal/cli/tui_actions.go +++ b/internal/cli/tui_actions.go @@ -423,6 +423,7 @@ func (m consoleModel) modelMenu(md adminapi.ModelView) *menuDialog { }}, } items = append(items, m.updateItems(md)...) + items = append(items, m.evalItems(md)...) if md.Parked { items = append(items, menuItem{"Try again (it is parked)", func(m *consoleModel) tea.Cmd { return m.call("Retry "+md.Name, http.MethodPost, path+"/retry", map[string]string{}, nil) @@ -706,7 +707,7 @@ func (m consoleModel) canaryItems(md adminapi.ModelView) []menuItem { {"Promote the canary…", func(m *consoleModel) tea.Cmd { m.confirmD = &confirmDialog{title: "Promote " + cn.Model, text: md.Name + " moves to the canary's weights: its replicas are replaced one at a time,\n" + - "and the canary is removed. Going back is \"Go back to\" on this menu.", + "and the canary is removed. Going back is \"Go back to\" on this menu." + gateNote(md), yes: func(m *consoleModel) tea.Cmd { return m.call("Promote "+cn.Model, http.MethodPost, path+"/promote", map[string]string{}, startedSays) }} @@ -780,8 +781,8 @@ func (m consoleModel) updateItems(md adminapi.ModelView) []menuItem { m.confirmD = &confirmDialog{title: "Update " + md.Name, text: fmt.Sprintf("%s has new weights at %s: %s.\n\n"+ "They are fetched into the mirror beside the ones in use and checked, then the\n"+ - "replicas are replaced one at a time. The old files stay, to go back to.", - u.Repo, catalog.ShortRevision(u.Latest), strings.Join(u.Changed, ", ")), + "replicas are replaced one at a time. The old files stay, to go back to.%s", + u.Repo, catalog.ShortRevision(u.Latest), strings.Join(u.Changed, ", "), gateNote(md)), yes: func(m *consoleModel) tea.Cmd { return m.call("Update "+md.Name, http.MethodPost, "/api/v1/models/"+url.PathEscape(md.Name)+"/update", map[string]string{}, startedSays) diff --git a/internal/cli/tui_console.go b/internal/cli/tui_console.go index e96af5e..98adad2 100644 --- a/internal/cli/tui_console.go +++ b/internal/cli/tui_console.go @@ -154,6 +154,11 @@ type consoleModel struct { formD *formDialog confirmD *confirmDialog busy string + // busyNote says how far the request under way has got, when it can; + // evalWatch is the quality run being followed, which Esc stops + // following while it carries on. + busyNote string + evalWatch string quotas []adminapi.QuotaView quotasErr error @@ -337,8 +342,12 @@ func (m consoleModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) { m.people = peopleData(msg) case consoleQuotasMsg: m.quotas, m.quotasErr = msg.rows, msg.err + case evalSuitesMsg: + return m.onEvalSuites(msg) + case evalPollMsg: + return m.onEvalPoll(msg) case actionDoneMsg: - m.busy = "" + m.busy, m.busyNote, m.evalWatch = "", "", "" m.dialog, m.dialogTitle = "error", msg.title if msg.err != nil { m.dialogText = sBad.Render("✗ ") + msg.err.Error() @@ -367,6 +376,9 @@ func (m consoleModel) storeReplicated() bool { return len(m.data.sum.Store) > 0 func (m consoleModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) { if m.busy != "" { + if m.evalWatch != "" && k.Type == tea.KeyEsc { + m.busy, m.busyNote, m.evalWatch = "", "", "" + } return m, nil } if m.dialog != "" { @@ -973,7 +985,11 @@ func (m consoleModel) View() string { } else if t, d := m.docs.dialog(); d != "" && m.menu == cManual { f.DialogTitle, f.Dialog = t, d } else if m.busy != "" { - f.DialogTitle, f.Dialog = m.busy, m.spin.View()+" asking the control plane…" + note := "asking the control plane…" + if m.busyNote != "" { + note = m.busyNote + } + f.DialogTitle, f.Dialog = m.busy, m.spin.View()+" "+note } else if t, d := m.actionDialog(); d != "" { f.DialogTitle, f.Dialog = t, d } diff --git a/internal/cli/tui_quality.go b/internal/cli/tui_quality.go new file mode 100644 index 0000000..27bb955 --- /dev/null +++ b/internal/cli/tui_quality.go @@ -0,0 +1,291 @@ +package cli + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "net/http" + "net/url" + "slices" + "strconv" + "strings" + "time" + + tea "github.com/charmbracelet/bubbletea" + + "gitea.mynodes.xyz/ilker/llmgw/internal/bench/eval" + "gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi" +) + +// Quality suites from the model menu: ask a model one of the suites the +// control plane has (its eval directory's, and those built in) and watch +// its score come in; list its past runs; hold its updates to one — the same +// as `llmgw ctl eval` and `llmgw ctl model gate`. + +// evalSuitesMsg is the suites the control plane can run, fetched to open +// the form that runs one (or, with gate, the one that sets a model's). +type evalSuitesMsg struct { + model adminapi.ModelView + suites []eval.SuiteInfo + gate bool + err error +} + +// evalPollMsg is a run still going, to be asked about again. +type evalPollMsg struct { + title string + live eval.Live + err error +} + +// gateNote says, in an update's or a promotion's question, that the new +// weights are asked the model's suite first. +func gateNote(md adminapi.ModelView) string { + g := md.Eval + if g == nil { + return "" + } + at := "its threshold" + if g.MinScore > 0 { + at = fmt.Sprintf("%.2f", g.MinScore) + } + return fmt.Sprintf("\n\nFirst the new weights are asked the quality suite %s, and must score %s;\n"+ + "below it nothing moves.", g.Suite, at) +} + +// evalItems are the model menu's quality entries. +func (m consoleModel) evalItems(md adminapi.ModelView) []menuItem { + gate := "Hold its updates to a quality suite…" + if g := md.Eval; g != nil { + gate = "Its updates' quality suite (" + g.Suite + ")…" + } + return []menuItem{ + {"Ask it a quality suite…", func(m *consoleModel) tea.Cmd { return m.fetchSuites(md, false) }}, + {"Its quality runs", func(m *consoleModel) tea.Cmd { + return m.call("Quality runs of "+md.Name, http.MethodGet, + "/api/v1/eval/runs?limit=15&model="+url.QueryEscape(md.Name), nil, func(raw json.RawMessage) string { + var runs []eval.Live + if err := json.Unmarshal(raw, &runs); err != nil { + return string(raw) + } + return evalRunsText(runs) + }) + }}, + {gate, func(m *consoleModel) tea.Cmd { return m.fetchSuites(md, true) }}, + } +} + +func (m *consoleModel) fetchSuites(md adminapi.ModelView, gate bool) tea.Cmd { + m.busy = "Quality suites" + c := m.client + return func() tea.Msg { + ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) + defer cancel() + var list []eval.SuiteInfo + err := c.get(ctx, "/api/v1/eval/suites", &list) + return evalSuitesMsg{model: md, suites: list, gate: gate, err: err} + } +} + +func (m consoleModel) onEvalSuites(msg evalSuitesMsg) (tea.Model, tea.Cmd) { + m.busy = "" + md := msg.model + var names []string + for _, s := range msg.suites { + if s.Error == "" { + names = append(names, s.Name) + } + } + if msg.err == nil && len(names) == 0 { + msg.err = errors.New("the control plane has no suite it can run: put one in its control.eval_dir " + + "(/etc/llmgw/eval); docs/install.md, \"Quality suites\", says how one is written") + } + if msg.err != nil { + m.dialog, m.dialogTitle = "error", "Quality suites" + m.dialogText = sBad.Render("✗ ") + msg.err.Error() + "\n\n" + sDim.Render("any key") + return m, nil + } + about := func(name string) string { + for _, s := range msg.suites { + if s.Name == name { + return fmt.Sprintf("%s: %d cases, passed at %.2f, %s", s.Name, s.Cases, s.Threshold, s.From) + } + } + return "" + } + path := "/api/v1/models/" + url.PathEscape(md.Name) + if msg.gate { + current, minScore := "none", "" + if g := md.Eval; g != nil { + current = g.Suite + if g.MinScore > 0 { + minScore = strconv.FormatFloat(g.MinScore, 'f', -1, 64) + } + if !slices.Contains(names, current) { + names = append(names, current) + } + } + m.openForm(&formDialog{title: "Quality gate of " + md.Name, + note: "An update or a promotion asks the new weights this suite first, and stops below its score;\n" + + "an update tries them as a canary given none of " + md.Name + "'s requests, and takes it out again.", + fields: []formField{ + choiceF("suite", append([]string{"none"}, names...), current), + textF("min score", minScore, "0 to 1; empty is the suite's own threshold"), + }, + submit: func(m *consoleModel, v []string) (tea.Cmd, error) { + if v[0] == "none" { + return m.call("Quality gate of "+md.Name, http.MethodDelete, path+"/eval-gate", nil, + func(json.RawMessage) string { return md.Name + "'s updates are no longer held to a suite." }), nil + } + score, err := minScoreOf(v[1]) + if err != nil { + return nil, err + } + return m.call("Quality gate of "+md.Name, http.MethodPut, path+"/eval-gate", + adminapi.EvalGateRequest{Suite: v[0], MinScore: score}, func(json.RawMessage) string { + return fmt.Sprintf("%s's new weights must pass %s before they replace the old.", md.Name, v[0]) + }), nil + }}) + return m, nil + } + m.openForm(&formDialog{title: "Ask " + md.Name + " a quality suite", + note: "Asked on the control plane, through the router as its own requests: past quotas and guardrails.\n" + + about(names[0]), + fields: []formField{ + choiceF("suite", names, names[0]), + textF("at once", "4", "cases asked at the same time"), + textF("min score", "", "0 to 1; empty is the suite's own threshold"), + }, + submit: func(m *consoleModel, v []string) (tea.Cmd, error) { + n, err := strconv.Atoi(v[1]) + if err != nil || n < 1 { + return nil, fmt.Errorf("at once is a whole number, 1 or more") + } + score, err := minScoreOf(v[2]) + if err != nil { + return nil, err + } + return m.startEval(md.Name, eval.Request{Suite: v[0], Concurrency: n, MinScore: score}), nil + }}) + return m, nil +} + +func minScoreOf(s string) (float64, error) { + if s == "" { + return 0, nil + } + f, err := strconv.ParseFloat(s, 64) + if err != nil || f < 0 || f > 1 { + return 0, fmt.Errorf("the min score is a number from 0 to 1") + } + return f, nil +} + +// startEval starts a run and follows it. +func (m *consoleModel) startEval(model string, req eval.Request) tea.Cmd { + title := model + ": " + req.Suite + m.busy, m.busyNote = title, "starting…" + c := m.client + return func() tea.Msg { + ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) + defer cancel() + var l eval.Live + err := c.do(ctx, http.MethodPost, "/api/v1/models/"+url.PathEscape(model)+"/eval", req, &l) + if err != nil { + return actionDoneMsg{title: title, err: err} + } + return evalPollMsg{title: title, live: l} + } +} + +func (m consoleModel) onEvalPoll(msg evalPollMsg) (tea.Model, tea.Cmd) { + if m.evalWatch != "" && m.evalWatch != msg.live.ID { + return m, nil + } + if m.evalWatch == "" && m.busy != msg.title { + // Esc: no longer followed here; it carries on on the control plane. + return m, nil + } + m.evalWatch = msg.live.ID + m.busy = msg.title + m.busyNote = fmt.Sprintf("%d of %d answered · Esc stops watching (it carries on)", msg.live.Done, msg.live.Cases) + c, id, title := m.client, msg.live.ID, msg.title + return m, func() tea.Msg { + time.Sleep(time.Second) + ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) + defer cancel() + var l eval.Live + if err := c.get(ctx, "/api/v1/eval/runs/"+url.PathEscape(id), &l); err != nil { + return actionDoneMsg{title: title, err: err} + } + if l.State == eval.Running { + return evalPollMsg{title: title, live: l} + } + var runs []eval.Live + _ = c.get(ctx, "/api/v1/eval/runs?limit=6&model="+url.QueryEscape(l.Model), &runs) + text := evalRunText(l) + if len(runs) > 1 { + text += "\n\nIts last runs:\n" + evalRunsText(runs) + } + if l.State == eval.Failed || !l.Passed { + return actionDoneMsg{title: title, err: errors.New(text)} + } + return actionDoneMsg{title: title, text: text} + } +} + +// evalRunText is a finished run: its score, then each case. +func evalRunText(l eval.Live) string { + if l.State == eval.Failed { + return "The run could not be scored: " + l.Error + } + var b strings.Builder + b.WriteString(evalSummary(l) + "\n") + if l.Error != "" { + b.WriteString(l.Error + "\n") + } + b.WriteString("\n") + shown := min(len(l.Results), 14) + for _, r := range l.Results[:shown] { + mark := sOK.Render("✓") + if !r.Passed { + mark = sBad.Render("✗") + } + why := r.Detail + if r.Error != "" { + why = r.Error + } + fmt.Fprintf(&b, "%s %-18s %4.2f %s\n", mark, truncate(r.ID, 18), r.Score, truncate(why, 60)) + } + if len(l.Results) > shown { + fmt.Fprintf(&b, " and %d more: llmgw ctl eval --run %s\n", len(l.Results)-shown, l.ID) + } + return strings.TrimRight(b.String(), "\n") +} + +// evalRunsText is a model's runs, one a line. +func evalRunsText(runs []eval.Live) string { + if len(runs) == 0 { + return "No runs kept yet: \"Ask it a quality suite\" on this menu makes one." + } + var b strings.Builder + for _, r := range runs { + result := sOK.Render("passed") + switch { + case r.State == eval.Running: + result = fmt.Sprintf("running, %d of %d", r.Done, r.Cases) + case r.State == eval.Failed: + result = sBad.Render("could not run") + case !r.Passed: + result = sBad.Render("failed") + } + who := r.Model + if r.Target != "" { + who = r.Target + } + fmt.Fprintf(&b, "%-9s %-14s %-16s %4.2f of %4.2f %s\n", ageShort(r.At), truncate(r.Suite, 14), + truncate(who, 16), r.Score, r.Threshold, result) + } + return strings.TrimRight(b.String(), "\n") +} diff --git a/internal/cli/tui_quality_test.go b/internal/cli/tui_quality_test.go new file mode 100644 index 0000000..ae9c8c3 --- /dev/null +++ b/internal/cli/tui_quality_test.go @@ -0,0 +1,116 @@ +package cli + +import ( + "encoding/json" + "io" + "net/http" + "net/http/httptest" + "strings" + "sync" + "testing" + + tea "github.com/charmbracelet/bubbletea" + + "gitea.mynodes.xyz/ilker/llmgw/internal/bench/eval" + "gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi" + "gitea.mynodes.xyz/ilker/llmgw/internal/store" +) + +func TestAModelIsAskedASuiteFromItsMenuAndItsScoreShown(t *testing.T) { + var mu sync.Mutex + var asked eval.Request + polls := 0 + done := eval.Live{EvalRun: store.EvalRun{ID: "eval-1", Model: "qwen", Suite: "arithmetic", Score: 0.75, + Threshold: 0.7, Passed: true, Cases: 2, Below: 1, Results: []store.EvalCase{ + {ID: "add", Kind: "number", Score: 1, Passed: true, Detail: "read 42"}, + {ID: "mul", Kind: "number", Score: 0, Detail: "read 43, wanted 42"}}}, State: eval.Done, Done: 2} + admin := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() + defer mu.Unlock() + switch { + case r.URL.Path == "/api/v1/eval/suites": + _ = json.NewEncoder(w).Encode([]eval.SuiteInfo{ + {Name: "arithmetic", From: "built in", Cases: 2, Threshold: 0.7}, + {Name: "broken", From: "/etc/llmgw/eval/broken.yaml", Error: "no cases"}, + {Name: "json-format", From: "built in", Cases: 7, Threshold: 0.7}}) + case r.Method == http.MethodPost && r.URL.Path == "/api/v1/models/qwen/eval": + b, _ := io.ReadAll(r.Body) + _ = json.Unmarshal(b, &asked) + w.WriteHeader(http.StatusAccepted) + _ = json.NewEncoder(w).Encode(eval.Live{EvalRun: store.EvalRun{ID: "eval-1", Model: "qwen", + Suite: "arithmetic", Cases: 2}, State: eval.Running}) + case r.URL.Path == "/api/v1/eval/runs/eval-1": + polls++ + if polls == 1 { + _ = json.NewEncoder(w).Encode(eval.Live{EvalRun: store.EvalRun{ID: "eval-1", Model: "qwen", + Suite: "arithmetic", Cases: 2}, State: eval.Running, Done: 1}) + return + } + _ = json.NewEncoder(w).Encode(done) + case r.URL.Path == "/api/v1/eval/runs": + older := done + older.ID, older.Score, older.Passed = "eval-0", 0.5, false + _ = json.NewEncoder(w).Encode([]eval.Live{done, older}) + default: + http.NotFound(w, r) + } + })) + t.Cleanup(admin.Close) + m := newConsoleModel(admin.URL, "/nonexistent/server.yaml") + m.loaded, m.menu, m.inPanel = true, cModels, true + md := adminapi.ModelView{Name: "qwen"} + + // run plays a command and what it leads to, the way the program would. + run := func(tm tea.Model, cmd tea.Cmd) tea.Model { + for cmd != nil { + msg := cmd() + tm, cmd = tm.Update(msg) + if _, ok := msg.(actionDoneMsg); ok { + break + } + } + return tm + } + var tm tea.Model = m + items := m.evalItems(md) + if len(items) != 3 || !strings.Contains(items[0].label, "quality suite") { + t.Fatalf("%+v", items) + } + tm = run(tm, items[0].act(&m)) + f := tm.(consoleModel).formD + if f == nil || f.fields[0].options == nil || len(f.fields[0].options) != 2 { + t.Fatalf("no form offering the suites that can run: %+v", f) + } + tm = press(tm, "tab") + f.fields[1].text.SetValue("2") + tm = press(tm, "tab", "tab") + tm, cmd := tm.Update(tea.KeyMsg{Type: tea.KeyEnter}) + if busy := tm.(consoleModel).busy; busy != "qwen: arithmetic" { + t.Fatalf("busy %q", busy) + } + // Started, then followed: one poll still running, then its score. + tm, next := tm.Update(cmd()) + if note := tm.(consoleModel).busyNote; !strings.Contains(note, "0 of 2 answered") { + t.Errorf("progress %q", note) + } + tm, next = tm.Update(next()) + if note := tm.(consoleModel).busyNote; !strings.Contains(note, "1 of 2 answered") { + t.Errorf("progress %q", note) + } + tm, _ = tm.Update(next()) + view := tm.View() + mu.Lock() + sent := asked + mu.Unlock() + if sent.Suite != "arithmetic" || sent.Concurrency != 2 { + t.Errorf("asked %+v", sent) + } + for _, want := range []string{"qwen scored 0.75 on arithmetic", "1 of 2 cases passed", "read 43, wanted 42", "Its last runs"} { + if !strings.Contains(view, want) { + t.Errorf("the result does not say %q:\n%s", want, view) + } + } + if c := tm.(consoleModel); c.busy != "" || c.evalWatch != "" { + t.Errorf("still busy: %q %q", c.busy, c.evalWatch) + } +} diff --git a/internal/config/config.go b/internal/config/config.go index 23c0f55..7c8a220 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -392,6 +392,11 @@ type Control struct { // Timezone is where models' scaling schedules are read, an IANA name // like Europe/Istanbul; empty is the control plane's own. Timezone string `koanf:"timezone"` + // EvalDir is where quality suites are read from by name, for `llmgw ctl + // eval MODEL --suite NAME`, a model's eval gate and the console. The + // suites the binary carries (arithmetic, json-format) are found when it + // has none of the name; a suite here of the same name replaces one. + EvalDir string `koanf:"eval_dir"` // Enabled runs the control plane even when configuration declares no // models. @@ -625,6 +630,7 @@ func Default() Server { LostAfter: 60 * time.Second, PortRange: []int{20000, 20999}, MgmtInterface: "eth0", + EvalDir: "/etc/llmgw/eval", }, Router: Router{ ScrapeInterval: 2 * time.Second, diff --git a/internal/control/adminapi/api.go b/internal/control/adminapi/api.go index 920b88b..c3f7be6 100644 --- a/internal/control/adminapi/api.go +++ b/internal/control/adminapi/api.go @@ -28,6 +28,7 @@ import ( "gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery" "gitea.mynodes.xyz/ilker/llmgw/internal/bench" + "gitea.mynodes.xyz/ilker/llmgw/internal/bench/eval" "gitea.mynodes.xyz/ilker/llmgw/internal/control" "gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog" "gitea.mynodes.xyz/ilker/llmgw/internal/control/fabrics" @@ -43,6 +44,8 @@ import ( type API struct { // replay sends a kept request again; nil without a router here. replay Replayer + // evals runs quality suites against models; nil without a router here. + evals *eval.Service // notifier is the webhooks `ctl notify test` reaches; nil without any. notifier *notify.Notifier // releases is where release binaries are fetched from, for upgrades. @@ -157,6 +160,7 @@ func (a *API) Register(mux *http.ServeMux) { mux.HandleFunc("POST /api/v1/models/{name}/update", a.updateModel) mux.HandleFunc("POST /api/v1/models/{name}/rollback", a.rollbackModel) a.registerCanary(mux) + a.registerEval(mux) mux.HandleFunc("POST /api/v1/models/{name}/origin", a.setOrigin) mux.HandleFunc("GET /api/v1/plan", a.planModel) mux.HandleFunc("POST /api/v1/metrics/range", a.metricsBatch) @@ -339,6 +343,8 @@ type ModelView struct { // set on the candidate, naming the model it is a canary of. Canary *catalog.Canary `json:"canary,omitempty"` CanaryOf string `json:"canary_of,omitempty"` + // Eval is the quality suite its new weights must pass, when it has one. + Eval *catalog.EvalGate `json:"eval,omitempty"` // Description is what the model is for, when somebody said. Description string `json:"description,omitempty"` Aliases []string `json:"aliases,omitempty"` @@ -660,7 +666,7 @@ func (a *API) Models() []ModelView { Parked: h.Parked, Reason: h.Reason, Unplaceable: h.Unplaceable, Class: m.EffectiveClass(), Owner: m.Owner, Preempted: h.Preempted, HeldFor: h.HeldFor, External: externalLabel(m), Adapters: m.Adapters, - Canary: m.Canary, CanaryOf: a.ctl.Catalog().CanaryOf(m.Name), + Canary: m.Canary, CanaryOf: a.ctl.Catalog().CanaryOf(m.Name), Eval: m.Eval, } for _, p := range m.Profiles { v.Profiles = append(v.Profiles, p.Name) @@ -1559,7 +1565,11 @@ func (a *API) updateModel(w http.ResponseWriter, r *http.Request) { writeError(w, http.StatusNotImplemented, "updates need a mirror and a store, and this control plane lacks one") return } - job, err := a.serve.StartUpdate(r.PathValue("name"), actorOf(r)) + ch, ok := readChange(w, r) + if !ok { + return + } + job, err := a.serve.StartUpdate(r.PathValue("name"), actorOf(r), ch) if err != nil { writeError(w, http.StatusBadRequest, err.Error()) return diff --git a/internal/control/adminapi/canary.go b/internal/control/adminapi/canary.go index bba0821..840bae2 100644 --- a/internal/control/adminapi/canary.go +++ b/internal/control/adminapi/canary.go @@ -74,7 +74,11 @@ func (a *API) registerCanary(mux *http.ServeMux) { if !need(w) { return } - job, err := a.serve.StartPromote(r.PathValue("name"), actorOf(r)) + ch, ok := readChange(w, r) + if !ok { + return + } + job, err := a.serve.StartPromote(r.PathValue("name"), actorOf(r), ch) if err != nil { writeError(w, http.StatusBadRequest, err.Error()) return diff --git a/internal/control/adminapi/eval.go b/internal/control/adminapi/eval.go new file mode 100644 index 0000000..7488cb4 --- /dev/null +++ b/internal/control/adminapi/eval.go @@ -0,0 +1,179 @@ +package adminapi + +import ( + "context" + "encoding/json" + "errors" + "io" + "net/http" + "strconv" + "strings" + + "gitea.mynodes.xyz/ilker/llmgw/internal/bench/eval" + "gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog" + "gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve" + "gitea.mynodes.xyz/ilker/llmgw/internal/store" +) + +// Quality suites: run against a model on the control plane, through the +// router as the gateway's own requests (like a replay), followed while they +// run, and kept; and a model's standing suite that its updates must pass. + +// SetEval gives the API the service that runs quality suites. +func (a *API) SetEval(s *eval.Service) { a.evals = s } + +// maxSuiteBytes bounds a suite sent whole: a few hundred cases with their +// prompts, which is more than a gate should take to run. +const maxSuiteBytes = 4 << 20 + +// EvalGateRequest sets a model's standing quality suite; an empty suite +// takes it away. +type EvalGateRequest struct { + Suite string `json:"suite"` + MinScore float64 `json:"min_score,omitempty"` + Judge string `json:"judge,omitempty"` +} + +type evalGater interface { + SetEval(ctx context.Context, name string, g *catalog.EvalGate, by string) (catalog.Model, error) +} + +// readChange reads how an update or a promotion is held to quality; an +// empty body is no change to how. +func readChange(w http.ResponseWriter, r *http.Request) (modelserve.Change, bool) { + var ch modelserve.Change + b, err := io.ReadAll(io.LimitReader(r.Body, maxSuiteBytes+1)) + if err != nil { + writeError(w, http.StatusBadRequest, err.Error()) + return ch, false + } + if len(b) > maxSuiteBytes { + writeError(w, http.StatusRequestEntityTooLarge, "the suite is over 4 MiB") + return ch, false + } + if len(strings.TrimSpace(string(b))) == 0 { + return ch, true + } + if err := json.Unmarshal(b, &ch); err != nil { + writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error()) + return ch, false + } + return ch, true +} + +func (a *API) registerEval(mux *http.ServeMux) { + need := func(w http.ResponseWriter) bool { + if a.evals == nil { + writeError(w, http.StatusNotImplemented, "this control plane does not route requests, so it cannot ask a model a suite") + return false + } + return true + } + mux.HandleFunc("GET /api/v1/eval/suites", func(w http.ResponseWriter, r *http.Request) { + if !need(w) { + return + } + out := a.evals.Suites() + if out == nil { + out = []eval.SuiteInfo{} + } + writeJSON(w, http.StatusOK, out) + }) + // Started and answered at once: a suite against a model on CPUs takes + // minutes, longer than a client waits on one request. + mux.HandleFunc("POST /api/v1/models/{name}/eval", func(w http.ResponseWriter, r *http.Request) { + if !need(w) { + return + } + var req eval.Request + if err := json.NewDecoder(io.LimitReader(r.Body, maxSuiteBytes)).Decode(&req); err != nil { + writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error()) + return + } + name := r.PathValue("name") + if a.ctl != nil { + // By its own name, so an alias's runs are filed with the rest. + m, ok := a.ctl.Catalog().Resolve(name) + if !ok { + writeError(w, http.StatusNotFound, "no model called "+name+": `llmgw ctl models` lists them") + return + } + name = m.Name + } + live, err := a.evals.Start(name, req, actorOf(r)) + if err != nil { + writeError(w, http.StatusBadRequest, err.Error()) + return + } + writeJSON(w, http.StatusAccepted, live) + }) + mux.HandleFunc("GET /api/v1/eval/runs", func(w http.ResponseWriter, r *http.Request) { + if !need(w) { + return + } + limit, _ := strconv.Atoi(r.URL.Query().Get("limit")) + out, err := a.evals.History(r.Context(), r.URL.Query().Get("model"), limit) + if err != nil { + writeError(w, http.StatusInternalServerError, err.Error()) + return + } + if out == nil { + out = []eval.Live{} + } + writeJSON(w, http.StatusOK, out) + }) + mux.HandleFunc("GET /api/v1/eval/runs/{id}", func(w http.ResponseWriter, r *http.Request) { + if !need(w) { + return + } + live, err := a.evals.Get(r.Context(), r.PathValue("id")) + if errors.Is(err, store.ErrNotFound) { + writeError(w, http.StatusNotFound, "no eval run "+r.PathValue("id")) + return + } + if err != nil { + writeError(w, http.StatusInternalServerError, err.Error()) + return + } + writeJSON(w, http.StatusOK, live) + }) + setGate := func(w http.ResponseWriter, r *http.Request, g *catalog.EvalGate) { + gater, ok := a.models.(evalGater) + if !ok { + writeError(w, http.StatusNotImplemented, "this control plane has no store to keep the setting in") + return + } + name := r.PathValue("name") + if g != nil && a.evals != nil { + if err := a.evals.Check(name, eval.Request{Suite: g.Suite, Judge: g.Judge, MinScore: g.MinScore}); err != nil { + writeError(w, http.StatusBadRequest, err.Error()) + return + } + } + m, err := gater.SetEval(r.Context(), name, g, actorOf(r)) + if err != nil { + writeError(w, http.StatusBadRequest, err.Error()) + return + } + writeJSON(w, http.StatusOK, map[string]any{"model": m.Name, "eval": m.Eval}) + } + mux.HandleFunc("PUT /api/v1/models/{name}/eval-gate", func(w http.ResponseWriter, r *http.Request) { + var req EvalGateRequest + if err := json.NewDecoder(io.LimitReader(r.Body, 1<<12)).Decode(&req); err != nil { + writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error()) + return + } + if strings.TrimSpace(req.Suite) == "" { + writeError(w, http.StatusBadRequest, "name a suite: one in the control plane's eval directory, or built in") + return + } + if req.MinScore < 0 || req.MinScore > 1 { + writeError(w, http.StatusBadRequest, "the minimum score is 0 to 1") + return + } + setGate(w, r, &catalog.EvalGate{Suite: strings.TrimSpace(req.Suite), MinScore: req.MinScore, Judge: req.Judge}) + }) + mux.HandleFunc("DELETE /api/v1/models/{name}/eval-gate", func(w http.ResponseWriter, r *http.Request) { + setGate(w, r, nil) + }) +} diff --git a/internal/control/adminapi/eval_test.go b/internal/control/adminapi/eval_test.go new file mode 100644 index 0000000..ca7c8aa --- /dev/null +++ b/internal/control/adminapi/eval_test.go @@ -0,0 +1,84 @@ +package adminapi + +import ( + "context" + "encoding/json" + "log/slog" + "net/http" + "net/http/httptest" + "strings" + "testing" + "testing/fstest" + "time" + + "gitea.mynodes.xyz/ilker/llmgw/internal/bench/eval" + "gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog" +) + +// gatingModels is the catalog's service, keeping the gate it is given. +type gatingModels struct { + *fakeModels + gate *catalog.EvalGate + set bool +} + +func (g *gatingModels) SetEval(_ context.Context, name string, gate *catalog.EvalGate, _ string) (catalog.Model, error) { + g.gate, g.set = gate, true + return catalog.Model{Name: name, Eval: gate}, nil +} + +func TestASuiteIsStartedByAModelsAliasAndAGateNeedsASuiteThatIsThere(t *testing.T) { + a, _ := setup(t, []catalog.Model{{Name: "qwen", Aliases: []string{"q"}, Source: "/m.gguf", + Scaling: catalog.Scaling{Min: 1, Max: 1}, Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp"}}}}) + answer := func(_ context.Context, _, _ string, _ []byte) (int, []byte, time.Duration, error) { + return 200, []byte(`{"choices":[{"message":{"content":"42"}}]}`), 0, nil + } + a.SetEval(eval.NewService(eval.Deps{Call: answer, Log: slog.New(slog.DiscardHandler), + Builtin: fstest.MapFS{"arith.yaml": {Data: []byte("cases:\n - prompt: 40 + 2?\n number: 42\n")}}})) + models := &gatingModels{fakeModels: &fakeModels{}} + a.SetModels(models) + mux := http.NewServeMux() + a.Register(mux) + do := func(method, path, body string) *httptest.ResponseRecorder { + req := httptest.NewRequest(method, path, strings.NewReader(body)) + rec := httptest.NewRecorder() + mux.ServeHTTP(rec, req) + return rec + } + + rec := do(http.MethodPost, "/api/v1/models/q/eval", `{"suite": "arith"}`) + var live eval.Live + if rec.Code != http.StatusAccepted || json.Unmarshal(rec.Body.Bytes(), &live) != nil || live.Model != "qwen" { + t.Fatalf("%d %s", rec.Code, rec.Body.String()) + } + if rec := do(http.MethodPost, "/api/v1/models/nobody/eval", `{"suite": "arith"}`); rec.Code != http.StatusNotFound { + t.Errorf("a model that is not there: %d %s", rec.Code, rec.Body.String()) + } + if rec := do(http.MethodPost, "/api/v1/models/qwen/eval", `{"suite": "nope"}`); rec.Code != http.StatusBadRequest || + !strings.Contains(rec.Body.String(), "--suites") { + t.Errorf("a suite that is not there: %d %s", rec.Code, rec.Body.String()) + } + if rec := do(http.MethodGet, "/api/v1/eval/suites", ""); !strings.Contains(rec.Body.String(), `"name":"arith"`) { + t.Errorf("suites: %s", rec.Body.String()) + } + + if rec := do(http.MethodPut, "/api/v1/models/qwen/eval-gate", `{"suite": "nope"}`); rec.Code != http.StatusBadRequest || models.set { + t.Errorf("a gate on a suite that is not there was set: %d %s", rec.Code, rec.Body.String()) + } + if rec := do(http.MethodPut, "/api/v1/models/qwen/eval-gate", `{"suite": "arith", "min_score": 0.9}`); rec.Code != http.StatusOK || + models.gate == nil || models.gate.Suite != "arith" || models.gate.MinScore != 0.9 { + t.Errorf("%d %s %+v", rec.Code, rec.Body.String(), models.gate) + } + if rec := do(http.MethodDelete, "/api/v1/models/qwen/eval-gate", ""); rec.Code != http.StatusOK || models.gate != nil { + t.Errorf("%d %s", rec.Code, rec.Body.String()) + } + + deadline := time.Now().Add(5 * time.Second) + for live.State == eval.Running && time.Now().Before(deadline) { + time.Sleep(10 * time.Millisecond) + _ = json.Unmarshal(do(http.MethodGet, "/api/v1/eval/runs/"+live.ID, "").Body.Bytes(), &live) + } + if live.State != eval.Done || live.Score != 1 || len(live.Results) != 1 { + t.Fatalf("%+v", live) + } +} diff --git a/internal/control/catalog/catalog.go b/internal/control/catalog/catalog.go index 602082b..a93c5be 100644 --- a/internal/control/catalog/catalog.go +++ b/internal/control/catalog/catalog.go @@ -294,6 +294,10 @@ type Model struct { // model — the same model at new weights — to compare the two before // moving to it (`llmgw ctl model canary`). Canary *Canary `json:"canary,omitempty" koanf:"canary"` + // Eval is a quality suite new weights must pass before they replace + // this model's: `llmgw ctl model update` and `model promote` run it + // against them first, and stop below its score. + Eval *EvalGate `json:"eval,omitempty" koanf:"eval"` // RoutingProfile selects how the router scores endpoints for this model. RoutingProfile string `json:"routing_profile,omitempty" koanf:"routing_profile"` SLO SLO `json:"slo,omitempty" koanf:"slo"` @@ -512,6 +516,18 @@ type Canary struct { MaxTTFTRatio float64 `json:"max_ttft_ratio,omitempty" koanf:"max_ttft_ratio"` } +// EvalGate is a quality suite a model's new weights must pass. +type EvalGate struct { + // Suite is a suite's name in control.eval_dir, or one the binary + // carries (arithmetic, json-format). + Suite string `json:"suite" koanf:"suite"` + // MinScore is the score, 0 to 1, to pass at; zero is the suite's own + // threshold. + MinScore float64 `json:"min_score,omitempty" koanf:"min_score"` + // Judge grades the suite's rubric cases, instead of the judge it names. + Judge string `json:"judge,omitempty" koanf:"judge"` +} + // Limits are the rollback thresholds with their defaults filled in. func (c Canary) Limits() (minRequests int, maxErrorIncrease, maxTTFTRatio float64) { minRequests, maxErrorIncrease, maxTTFTRatio = c.MinRequests, c.MaxErrorIncrease, c.MaxTTFTRatio @@ -633,6 +649,14 @@ func (m Model) validate() error { if !validClass(m.Class) { return fmt.Errorf("model %q: class %q is not one of critical, standard, opportunistic", m.Name, m.Class) } + if g := m.Eval; g != nil { + if strings.TrimSpace(g.Suite) == "" { + return fmt.Errorf("model %q: eval names no suite", m.Name) + } + if g.MinScore < 0 || g.MinScore > 1 { + return fmt.Errorf("model %q: eval min_score %v is not 0 to 1", m.Name, g.MinScore) + } + } switch m.Role { case "": if m.Guard != nil { diff --git a/internal/control/modelserve/canary.go b/internal/control/modelserve/canary.go index 4eea4bc..aff78e3 100644 --- a/internal/control/modelserve/canary.go +++ b/internal/control/modelserve/canary.go @@ -56,25 +56,33 @@ func (s *Server) StartCanary(name string, cn catalog.Canary, by string) (Job, er func (s *Server) runCanary(j *Job, cn catalog.Canary) { ctx := context.Background() - name := j.Model - p, ok := s.prepare(ctx, j, name) + p, ok := s.prepare(ctx, j, j.Model) if !ok { return } + if _, ok := s.addCandidate(ctx, j, p, cn); ok { + s.finish(j) + } +} + +// addCandidate runs a model's prepared new weights beside it as its canary +// and gives it its share; it fails the job and says false when it cannot. +func (s *Server) addCandidate(ctx context.Context, j *Job, p prepared, cn catalog.Canary) (string, bool) { + name := j.Model // 4. The candidate, beside the model: the same in everything but its // weights, and one replica. cand := canaryName(name) s.step(j, "catalog", Running, "adding "+cand) c := p.m - c.Name, c.Aliases, c.Fallbacks, c.Canary, c.Adapters = cand, nil, nil, nil, nil + c.Name, c.Aliases, c.Fallbacks, c.Canary, c.Adapters, c.Eval = cand, nil, nil, nil, nil, nil c.Source = p.source c.Origin = &catalog.Origin{Repo: p.repo.ID, Revision: p.repo.Revision, Files: originFiles(p.set)} c.Scaling = catalog.Scaling{Min: 1, Max: 1} c.Description = fmt.Sprintf("Canary of %s at %s.", name, p.repo.ShortRevision()) if err := s.deps.Models.Add(ctx, c, j.Request.By); err != nil { s.fail(j, "catalog", err) - return + return "", false } s.step(j, "catalog", Done, cand+" runs "+p.repo.ID+" at "+p.repo.ShortRevision()) @@ -93,21 +101,22 @@ func (s *Server) runCanary(j *Job, cn catalog.Canary) { if err != nil { _ = s.deps.Models.Remove(ctx, cand) s.fail(j, "placement", fmt.Errorf("%w; %s was taken out again", err, cand)) - return + return "", false } - // 6. Its share. + // 6. Its share: none, for new weights only being asked a suite, which + // hides it from clients all the same. cn.Model = cand if _, err := s.deps.Models.SetCanary(ctx, name, &cn, j.Request.By); err != nil { _ = s.deps.Models.Remove(ctx, cand) s.fail(j, "split", err) - return + return "", false } if s.deps.ResetCanary != nil { s.deps.ResetCanary(name) } s.step(j, "split", Done, fmt.Sprintf("%d%% of %s's requests go to %s", cn.Percent, name, cand)) - s.finish(j) + return cand, true } // SetCanaryPercent changes the share of a model's requests its canary is @@ -152,8 +161,9 @@ func (s *Server) AbortCanary(ctx context.Context, name, by string) (string, erro } // StartPromote moves a model to its canary's weights — the update a canary -// was the trial of — and takes the canary out. -func (s *Server) StartPromote(name, by string) (Job, error) { +// was the trial of — and takes the canary out. Held to a quality suite, the +// canary is asked it first, and stays a canary below its score. +func (s *Server) StartPromote(name, by string, ch Change) (Job, error) { m, err := s.current(name) if err != nil { return Job{}, err @@ -161,28 +171,50 @@ func (s *Server) StartPromote(name, by string) (Job, error) { if m.Canary == nil { return Job{}, fmt.Errorf("%s has no canary to promote", name) } + g, skipped := gateFor(m, ch) + if err := s.checkGate(m.Canary.Model, g); err != nil { + return Job{}, err + } s.mu.Lock() + for _, j := range s.jobs { + if (j.Kind == "update" || j.Kind == "canary" || j.Kind == "promote") && j.Model == name && j.State == Running { + s.mu.Unlock() + return Job{}, fmt.Errorf("%s has a job running: %s", name, j.ID) + } + } s.seq++ j := &Job{ID: fmt.Sprintf("promote-%d-%d", time.Now().Unix(), s.seq), Kind: "promote", Model: name, Request: Request{Name: name, By: by}, State: Running, Started: time.Now()} s.keepJob(j) s.mu.Unlock() - go s.runPromote(j) + go func() { + ctx := context.Background() + cand := m.Canary.Model + if !s.gate(ctx, j, cand, g, skipped, func() string { + return fmt.Sprintf("%s stays a canary: `llmgw ctl model canary %s --abort` takes it out", cand, name) + }) { + return + } + if s.promoteTo(ctx, j) { + s.finish(j) + } + }() return s.snapshot(j), nil } -func (s *Server) runPromote(j *Job) { - ctx := context.Background() +// promoteTo moves a model to its canary's weights, replaces its replicas and +// takes the canary out; it fails the job and says false when it cannot. +func (s *Server) promoteTo(ctx context.Context, j *Job) bool { name := j.Model m, err := s.current(name) if err != nil || m.Canary == nil { s.fail(j, "catalog", fmt.Errorf("%s no longer has a canary", name)) - return + return false } c, err := s.current(m.Canary.Model) if err != nil { s.fail(j, "catalog", err) - return + return false } s.step(j, "catalog", Running, fmt.Sprintf("moving %s to %s's weights", name, c.Name)) next := &catalog.Origin{} @@ -195,18 +227,18 @@ func (s *Server) runPromote(j *Job) { } if _, err := s.deps.Models.SetSource(ctx, name, c.Source, next, j.Request.By); err != nil { s.fail(j, "catalog", err) - return + return false } // Its share back to the model, which is about to run the same weights. if _, err := s.deps.Models.SetCanary(ctx, name, nil, j.Request.By); err != nil { s.fail(j, "catalog", err) - return + return false } s.step(j, "catalog", Done, "the catalog names "+c.Source) if err := s.replace(ctx, j, name); err != nil { s.fail(j, "replicas", fmt.Errorf("%w. `llmgw ctl model rollback %s` goes back", err, name)) - return + return false } if err := s.deps.Models.Remove(ctx, c.Name); err != nil { s.step(j, "canary", Failed, fmt.Sprintf("%s could not be removed: %v", c.Name, err)) @@ -222,7 +254,7 @@ func (s *Server) runPromote(j *Job) { Latest: c.Origin.Revision, Previous: next.Previous.Revision, CheckedAt: time.Now()} s.mu.Unlock() } - s.finish(j) + return true } // WatchCanaries takes out, every so often, each canary serving worse than diff --git a/internal/control/modelserve/canary_test.go b/internal/control/modelserve/canary_test.go index 6eedadf..7411129 100644 --- a/internal/control/modelserve/canary_test.go +++ b/internal/control/modelserve/canary_test.go @@ -49,7 +49,7 @@ func TestACanaryRunsTheNewWeightsBesideAndIsPromoted(t *testing.T) { t.Errorf("share: %v", err) } - job, err = s.StartPromote("qwen", "tester") + job, err = s.StartPromote("qwen", "tester", Change{}) if err != nil { t.Fatal(err) } diff --git a/internal/control/modelserve/gate.go b/internal/control/modelserve/gate.go new file mode 100644 index 0000000..d0b3247 --- /dev/null +++ b/internal/control/modelserve/gate.go @@ -0,0 +1,96 @@ +package modelserve + +import ( + "context" + "errors" + "fmt" + + "gitea.mynodes.xyz/ilker/llmgw/internal/bench/eval" + "gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog" + "gitea.mynodes.xyz/ilker/llmgw/internal/store" +) + +// A gate: new weights asked a quality suite before they replace a model's, +// and refused below its score. A promotion asks the canary, which is +// already running; an update starts the new weights as a canary given none +// of the model's requests, asks it, and promotes it or takes it out — so +// the old weights serve until the new have passed. + +// Evaluator runs quality suites: the control plane's eval service. +type Evaluator interface { + // Check says whether a suite can be run, before anything is started + // for it. + Check(model string, r eval.Request) error + // Run asks a model a suite and waits for its score. + Run(ctx context.Context, model string, r eval.Request, by, job string) (store.EvalRun, error) +} + +// Change is how an update or a promotion is held to quality. +type Change struct { + // Eval is a suite the new weights must pass first. Nil takes the + // model's own (eval: in its catalog entry), if it has one. + Eval *eval.Request `json:"eval,omitempty"` + // SkipEval moves without the model's own suite; the job says so. + SkipEval bool `json:"skip_eval,omitempty"` +} + +// gateFor is the suite a change is held to, and whether one was skipped. +func gateFor(m catalog.Model, ch Change) (r *eval.Request, skipped bool) { + switch { + case ch.Eval != nil: + return ch.Eval, false + case m.Eval == nil: + return nil, false + case ch.SkipEval: + return nil, true + } + return &eval.Request{Suite: m.Eval.Suite, MinScore: m.Eval.MinScore, Judge: m.Eval.Judge}, false +} + +// checkGate refuses, before a job starts, a gate that cannot be run. +func (s *Server) checkGate(name string, r *eval.Request) error { + if r == nil { + return nil + } + if s.deps.Eval == nil { + return errors.New("this control plane cannot run quality suites: it routes no requests") + } + if err := s.deps.Eval.Check(name, *r); err != nil { + return fmt.Errorf("the quality suite: %w", err) + } + return nil +} + +// gate asks the new weights, running as target, a suite, and says whether +// they may replace the old; when they may not, the job fails saying why, +// and what refused does about the new weights, which it says too. +func (s *Server) gate(ctx context.Context, j *Job, target string, r *eval.Request, skipped bool, refused func() string) bool { + if skipped { + s.step(j, "eval", Skipped, "the model's quality suite was skipped by "+orUnknown(j.Request.By)) + return true + } + if r == nil { + return true + } + s.step(j, "eval", Running, fmt.Sprintf("asking %s the cases of %s", target, r.Named())) + run, err := s.deps.Eval.Run(ctx, target, *r, j.Request.By, j.ID) + if err != nil { + s.fail(j, "eval", fmt.Errorf("%w; %s", err, refused())) + return false + } + run.Results = nil + s.mu.Lock() + j.Eval = &run + s.mu.Unlock() + detail := fmt.Sprintf("%s scored %.2f on %s, %d of %d cases passed; at least %.2f is asked", target, + run.Score, run.Suite, run.Cases-run.Below, run.Cases, run.Threshold) + if run.Error != "" { + detail += "; " + run.Error + } + if !run.Passed { + s.fail(j, "eval", fmt.Errorf("%s; %s. `llmgw ctl eval --run %s` shows each case", detail, refused(), run.ID)) + return false + } + s.step(j, "eval", Done, detail) + return true +} diff --git a/internal/control/modelserve/gate_test.go b/internal/control/modelserve/gate_test.go new file mode 100644 index 0000000..9cd2247 --- /dev/null +++ b/internal/control/modelserve/gate_test.go @@ -0,0 +1,184 @@ +package modelserve + +import ( + "context" + "errors" + "strings" + "sync" + "testing" + + "gitea.mynodes.xyz/ilker/llmgw/internal/bench/eval" + "gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog" + "gitea.mynodes.xyz/ilker/llmgw/internal/store" +) + +// fakeEval scores whatever it is asked what it is told to. +type fakeEval struct { + mu sync.Mutex + score float64 + asked []string +} + +func (f *fakeEval) Check(_ string, r eval.Request) error { + if r.Suite == "missing" { + return errors.New("no suite called missing") + } + return nil +} + +func (f *fakeEval) Run(_ context.Context, model string, r eval.Request, by, job string) (store.EvalRun, error) { + f.mu.Lock() + defer f.mu.Unlock() + f.asked = append(f.asked, model+" "+r.Named()) + threshold := r.MinScore + if threshold == 0 { + threshold = 0.8 + } + below := 0 + if f.score < 1 { + below = 1 + } + return store.EvalRun{ID: "eval-1", Model: "qwen", Target: model, Suite: r.Named(), Score: f.score, + Threshold: threshold, Passed: f.score >= threshold, Cases: 4, Below: below, By: by, Job: job}, nil +} + +func (f *fakeEval) said() []string { + f.mu.Lock() + defer f.mu.Unlock() + return append([]string{}, f.asked...) +} + +func stepOf(j Job, name string) Step { + for _, s := range j.Steps { + if s.Name == name { + return s + } + } + return Step{} +} + +func TestAPromoteBelowTheSuitesScoreIsRefusedAndTheCanaryStays(t *testing.T) { + s, models, _ := canarySetup(t) + ev := &fakeEval{score: 0.5} + s.deps.Eval = ev + job, _ := s.StartCanary("qwen", catalog.Canary{Percent: 10}, "tester") + if j := wait(t, s, job.ID); j.State != Done { + t.Fatalf("%s: %+v", j.Error, j.Steps) + } + oldSource := models.models["qwen"].Source + gate := Change{Eval: &eval.Request{Suite: "arithmetic", MinScore: 0.8}} + + job, err := s.StartPromote("qwen", "tester", gate) + if err != nil { + t.Fatal(err) + } + j := wait(t, s, job.ID) + if j.State != Failed || stepOf(j, "eval").State != Failed || + !strings.Contains(j.Error, "qwen-canary scored 0.50 on arithmetic") || + !strings.Contains(j.Error, "stays a canary") || stepOf(j, "catalog").Name != "" { + t.Fatalf("a canary scoring 0.5 of 0.8 was promoted, or not said so: %s %+v", j.Error, j.Steps) + } + if j.Eval == nil || j.Eval.Score != 0.5 || j.Eval.Job != j.ID { + t.Errorf("the job does not carry the score: %+v", j.Eval) + } + m := models.models["qwen"] + if m.Source != oldSource || m.Canary == nil || m.Canary.Percent != 10 { + t.Fatalf("the model moved, or lost its canary: %s %+v", m.Source, m.Canary) + } + if _, ok := models.models["qwen-canary"]; !ok { + t.Fatal("the canary was taken out; a refused promotion leaves that to its owner") + } + if got := ev.said(); len(got) != 1 || got[0] != "qwen-canary arithmetic" { + t.Errorf("asked %v", got) + } + + ev.score = 0.9 + job, _ = s.StartPromote("qwen", "tester", gate) + if j := wait(t, s, job.ID); j.State != Done || !strings.Contains(stepOf(j, "eval").Detail, "scored 0.90") { + t.Fatalf("%s: %+v", j.Error, j.Steps) + } + if m := models.models["qwen"]; m.Source == oldSource || m.Canary != nil { + t.Fatalf("a canary that passed was not promoted: %s %+v", m.Source, m.Canary) + } +} + +func TestAnUpdateHeldToTheModelsSuiteTriesTheNewWeightsFirstAndKeepsTheOldBelowIt(t *testing.T) { + s, models, fleet := canarySetup(t) + ev := &fakeEval{score: 0.4} + s.deps.Eval = ev + m := models.models["qwen"] + m.Eval = &catalog.EvalGate{Suite: "arithmetic", MinScore: 0.75} + models.models["qwen"] = m + oldSource := m.Source + + job, err := s.StartUpdate("qwen", "tester", Change{}) + if err != nil { + t.Fatal(err) + } + j := wait(t, s, job.ID) + if j.State != Failed || !strings.Contains(j.Error, "scored 0.40") || !strings.Contains(j.Error, "taken out again") { + t.Fatalf("%s: %+v", j.Error, j.Steps) + } + if got := ev.said(); len(got) != 1 || got[0] != "qwen-canary arithmetic" { + t.Fatalf("the new weights were not what was asked: %v", got) + } + m = models.models["qwen"] + if m.Source != oldSource || m.Canary != nil { + t.Fatalf("the model moved to weights that failed, or kept their canary: %s %+v", m.Source, m.Canary) + } + if _, ok := models.models["qwen-canary"]; ok { + t.Fatal("the failed weights were left in the catalog") + } + for _, r := range fleet.Replicas() { + if r.Model == "qwen" && strings.Contains(r.ID, "new") { + t.Fatalf("%s was replaced onto weights that failed", r.ID) + } + } + + // Scoring enough, they replace the old. + ev.score = 1 + job, _ = s.StartUpdate("qwen", "tester", Change{}) + j = wait(t, s, job.ID) + if j.State != Done || stepOf(j, "eval").State != Done || stepOf(j, "split").Detail != "0% of qwen's requests go to qwen-canary" { + t.Fatalf("%s: %+v", j.Error, j.Steps) + } + if m := models.models["qwen"]; m.Source == oldSource || !strings.HasSuffix(m.Source, "/def98765/q4.gguf") || + m.Canary != nil || m.Origin.Previous == nil || m.Origin.Previous.Source != oldSource || m.Eval == nil { + t.Fatalf("not moved to the new weights, or the gate was lost: %s %+v %+v", m.Source, m.Canary, m.Origin) + } + if _, ok := models.models["qwen-canary"]; ok { + t.Error("the canary was left in the catalog") + } +} + +func TestAModelsSuiteCanBeSkippedOnceAndAMissingOneIsRefusedAtOnce(t *testing.T) { + s, models, _ := canarySetup(t) + ev := &fakeEval{score: 0} + s.deps.Eval = ev + m := models.models["qwen"] + m.Eval = &catalog.EvalGate{Suite: "arithmetic"} + models.models["qwen"] = m + + if _, err := s.StartUpdate("qwen", "tester", Change{Eval: &eval.Request{Suite: "missing"}}); err == nil || + !strings.Contains(err.Error(), "no suite called missing") { + t.Fatalf("an update held to a suite that is not there was started: %v", err) + } + + job, err := s.StartUpdate("qwen", "tester", Change{SkipEval: true}) + if err != nil { + t.Fatal(err) + } + j := wait(t, s, job.ID) + if j.State != Done || stepOf(j, "eval").State != Skipped || !strings.Contains(stepOf(j, "eval").Detail, "tester") { + t.Fatalf("%s: %+v", j.Error, j.Steps) + } + if len(ev.said()) != 0 { + t.Errorf("asked %v while skipped", ev.said()) + } + + // Without anything to run suites, a gate is refused rather than ignored. + s.deps.Eval = nil + if _, err := s.StartUpdate("qwen", "tester", Change{}); err == nil { + t.Error("an update held to the model's suite started with nothing to run it") + } +} diff --git a/internal/control/modelserve/modelserve.go b/internal/control/modelserve/modelserve.go index 0676d4a..156c5ac 100644 --- a/internal/control/modelserve/modelserve.go +++ b/internal/control/modelserve/modelserve.go @@ -33,6 +33,7 @@ import ( "gitea.mynodes.xyz/ilker/llmgw/internal/mirror" "gitea.mynodes.xyz/ilker/llmgw/internal/modelplan" "gitea.mynodes.xyz/ilker/llmgw/internal/notify" + "gitea.mynodes.xyz/ilker/llmgw/internal/store" ) // Registry reads a repository. @@ -95,6 +96,9 @@ type Deps struct { // canary is never rolled back by itself. CanaryStats func() map[string]canary.Stat ResetCanary func(model string) + // Eval runs the quality suites an update or a promotion is held to; + // without it, one held to a suite is refused. + Eval Evaluator } // errBusy is a model the cluster has the hardware for, all of it in use now. @@ -178,6 +182,9 @@ type Job struct { // how fast it said it. Reply string `json:"reply,omitempty"` TokensPerSecond float64 `json:"tokens_per_second,omitempty"` + // Eval is the quality suite the new weights were asked before they + // replaced the old, and how they scored, without each case. + Eval *store.EvalRun `json:"eval,omitempty"` } // Server runs jobs and remembers them. diff --git a/internal/control/modelserve/updates.go b/internal/control/modelserve/updates.go index 18b1f4d..3a85529 100644 --- a/internal/control/modelserve/updates.go +++ b/internal/control/modelserve/updates.go @@ -11,6 +11,7 @@ import ( "sync/atomic" "time" + "gitea.mynodes.xyz/ilker/llmgw/internal/bench/eval" "gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog" "gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler" "gitea.mynodes.xyz/ilker/llmgw/internal/gguf" @@ -429,33 +430,79 @@ func (s *Server) current(name string) (catalog.Model, error) { } // StartUpdate moves a model to its repository's latest weights, as a job. -func (s *Server) StartUpdate(name, by string) (Job, error) { +// Held to a quality suite, the new weights run first as a canary given none +// of its requests, are asked the suite, and replace the old only at its +// score. +func (s *Server) StartUpdate(name, by string, ch Change) (Job, error) { if s.deps.Models == nil || s.deps.Restart == nil { return Job{}, errors.New("updating needs the store, which this control plane lacks") } - if _, err := s.current(name); err != nil { + m, err := s.current(name) + if err != nil { return Job{}, err } if s.deps.Models.Declared(name) { return Job{}, fmt.Errorf("%s is declared in server.yaml: serve the new weights there, by its source, "+ "or they would revert on the next restart", name) } + g, skipped := gateFor(m, ch) + if g != nil && m.Canary != nil { + return Job{}, fmt.Errorf("%s has a canary, %s: an update held to a quality suite tries the new weights as "+ + "one. Promote it (`llmgw ctl model promote %s --eval ...`) or take it out first", name, m.Canary.Model, name) + } + if err := s.checkGate(canaryName(name), g); err != nil { + return Job{}, err + } s.mu.Lock() for _, j := range s.jobs { if j.Kind == "update" && j.Model == name && j.State == Running { s.mu.Unlock() return Job{}, fmt.Errorf("%s is already being updated: job %s", name, j.ID) } + if g != nil && (j.Kind == "canary" || j.Kind == "promote") && j.Model == name && j.State == Running { + s.mu.Unlock() + return Job{}, fmt.Errorf("%s has a job running: %s", name, j.ID) + } } s.seq++ j := &Job{ID: fmt.Sprintf("update-%d-%d", time.Now().Unix(), s.seq), Kind: "update", Model: name, Request: Request{Name: name, By: by}, State: Running, Started: time.Now()} s.keepJob(j) s.mu.Unlock() - go s.runUpdate(j) + if g != nil { + go s.runGatedUpdate(j, g) + } else { + go s.runUpdate(j, skipped) + } return s.snapshot(j), nil } +// runGatedUpdate is an update held to a quality suite: the new weights as a +// canary with no share, asked the suite, then promoted or taken out. +func (s *Server) runGatedUpdate(j *Job, g *eval.Request) { + ctx := context.Background() + name := j.Model + p, ok := s.prepare(ctx, j, name) + if !ok { + return + } + cand, ok := s.addCandidate(ctx, j, p, catalog.Canary{Percent: 0}) + if !ok { + return + } + if !s.gate(ctx, j, cand, g, false, func() string { + if _, err := s.AbortCanary(ctx, name, "llmgw"); err != nil { + return fmt.Sprintf("the new weights could not be taken out (%v): `llmgw ctl model canary %s --abort`", err, name) + } + return fmt.Sprintf("the new weights were taken out again; %s still runs %s", name, catalog.ShortRevision(p.o.Revision)) + }) { + return + } + if s.promoteTo(ctx, j) { + s.finish(j) + } +} + // prepared is a model's new weights, fetched and checked: what an update and // a canary both start from. type prepared struct { @@ -559,13 +606,14 @@ func (s *Server) prepare(ctx context.Context, j *Job, name string) (prepared, bo return prepared{m: m, o: o, repo: repo, set: set, source: source}, true } -func (s *Server) runUpdate(j *Job) { +func (s *Server) runUpdate(j *Job, skipped bool) { ctx := context.Background() name := j.Model p, ok := s.prepare(ctx, j, name) if !ok { return } + s.gate(ctx, j, name, nil, skipped, nil) m, o, repo, set, source := p.m, p.o, p.repo, p.set, p.source // 4. The catalog: the new source, and the old one kept to go back to. diff --git a/internal/control/modelserve/updates_test.go b/internal/control/modelserve/updates_test.go index 81c9430..f723caa 100644 --- a/internal/control/modelserve/updates_test.go +++ b/internal/control/modelserve/updates_test.go @@ -205,7 +205,7 @@ func updateSays(u UpdateInfo) string { func TestAnUpdateFetchesBesideTheOldReplacesTheReplicasAndCanGoBack(t *testing.T) { s, mir, models, _, fleet := updSetup(t) oldSource := models.models["qwen"].Source - job, err := s.StartUpdate("qwen", "tester") + job, err := s.StartUpdate("qwen", "tester", Change{}) if err != nil { t.Fatal(err) } @@ -243,7 +243,7 @@ func TestAnUpdateFetchesBesideTheOldReplacesTheReplicasAndCanGoBack(t *testing.T } // And forward again, from files already on the mirror. - job, err = s.StartUpdate("qwen", "tester") + job, err = s.StartUpdate("qwen", "tester", Change{}) if err != nil { t.Fatal(err) } @@ -251,7 +251,7 @@ func TestAnUpdateFetchesBesideTheOldReplacesTheReplicasAndCanGoBack(t *testing.T t.Fatalf("%s: %+v", j.Error, j.Steps) } // Then there is nothing newer: refused, saying why. - job, _ = s.StartUpdate("qwen", "tester") + job, _ = s.StartUpdate("qwen", "tester", Change{}) if j := wait(t, s, job.ID); j.State != Failed || !strings.Contains(j.Error, "already runs") { t.Fatalf("%+v", j) } @@ -260,7 +260,7 @@ func TestAnUpdateFetchesBesideTheOldReplacesTheReplicasAndCanGoBack(t *testing.T func TestAModelFromConfigurationIsNotUpdatedBehindItsBack(t *testing.T) { s, _, models, _, _ := updSetup(t) models.declared["qwen"] = true - if _, err := s.StartUpdate("qwen", "t"); err == nil || !strings.Contains(err.Error(), "server.yaml") { + if _, err := s.StartUpdate("qwen", "t", Change{}); err == nil || !strings.Contains(err.Error(), "server.yaml") { t.Fatalf("%v", err) } } @@ -268,7 +268,7 @@ func TestAModelFromConfigurationIsNotUpdatedBehindItsBack(t *testing.T) { func TestAnUpdateToFilesTheRepositoryDroppedIsRefused(t *testing.T) { s, _, _, reg, _ := updSetup(t) reg.repo.Files = []hf.File{{Path: "q4-v2.gguf", SHA256: "cccc"}} - job, _ := s.StartUpdate("qwen", "t") + job, _ := s.StartUpdate("qwen", "t", Change{}) if j := wait(t, s, job.ID); j.State != Failed || !strings.Contains(j.Error, "no longer has q4.gguf") { t.Fatalf("%+v", j) } diff --git a/internal/control/modelsvc/modelsvc.go b/internal/control/modelsvc/modelsvc.go index 36fa40b..392b67d 100644 --- a/internal/control/modelsvc/modelsvc.go +++ b/internal/control/modelsvc/modelsvc.go @@ -323,6 +323,23 @@ func (s *Service) SetCanary(ctx context.Context, name string, c *catalog.Canary, return final, err } +// SetEval sets the quality suite a model's new weights must pass before an +// update or a promotion moves it to them; nil takes it away. +func (s *Service) SetEval(ctx context.Context, name string, g *catalog.EvalGate, by string) (catalog.Model, error) { + final, err := s.update(ctx, name, by, "set its eval gate", func(m *catalog.Model) error { + m.Eval = g + return nil + }) + if err == nil { + if g == nil { + s.log.Info("model eval gate removed", "model", name, "by", by) + } else { + s.log.Info("model eval gate set", "model", name, "suite", g.Suite, "min_score", g.MinScore, "by", by) + } + } + return final, err +} + // SetOrigin records where a model's weights came from without changing them: // for a model added before origins were kept. A model from configuration is // left alone, its origin worked out again when it is needed. diff --git a/internal/docgen/reference_gen.go b/internal/docgen/reference_gen.go index 4503a72..74bf4fa 100644 --- a/internal/docgen/reference_gen.go +++ b/internal/docgen/reference_gen.go @@ -65,6 +65,7 @@ var Reference = []Block{ {Key: "port_range", Type: "[]int", Doc: "PortRange is the per-node range for engine HTTP and dist-init ports.", Nested: ""}, {Key: "mgmt_interface", Type: "string", Doc: "MgmtInterface is the ethernet interface NCCL bootstraps over.", Nested: ""}, {Key: "timezone", Type: "string", Doc: "Timezone is where models' scaling schedules are read, an IANA name\nlike Europe/Istanbul; empty is the control plane's own.", Nested: ""}, + {Key: "eval_dir", Type: "string", Doc: "EvalDir is where quality suites are read from by name, for `llmgw ctl\neval MODEL --suite NAME`, a model's eval gate and the console. The\nsuites the binary carries (arithmetic, json-format) are found when it\nhas none of the name; a suite here of the same name replaces one.", Nested: ""}, {Key: "enabled", Type: "bool", Doc: "Enabled runs the control plane even when configuration declares no\nmodels.\n\nWithout it, a deployment whose catalog lives entirely in the database\ncannot start: control plane mode is inferred from the config file, so an\nempty models block means \"router only\", the process refuses to start for\nwant of anything to serve, and the models it would have loaded are never\nread. That is a cluster that cannot be managed at runtime because it is\nmanaged at runtime. AgentListen cannot carry this signal — it has a\ndefault, so every deployment would become a control plane.", Nested: ""}, {Key: "lease_ttl", Type: "time.Duration", Doc: "LeaseTTL is how long this control plane's claim to be the one placing\nwork stands without being renewed.\n\nOnly meaningful with a shared store, where a second control plane can\nsee the claim. Long enough to survive a slow query or a brief network\nstall, short enough that a standby takes over before anyone has finished\nreading the alert. Zero takes the default.", Nested: ""}, }}, @@ -84,6 +85,11 @@ var Reference = []Block{ {Key: "labels", Type: "map[string]string", Doc: "Labels picks nodes carrying all of these.", Nested: ""}, {Key: "engines", Type: "map[string]string", Doc: "Engines are laid over the cluster's: a version, or \"\" for none.", Nested: ""}, }}, + {Package: "catalog", Name: "EvalGate", Title: "EvalGate", Doc: "", Fields: []Field{ + {Key: "suite", Type: "string", Doc: "Suite is a suite's name in control.eval_dir, or one the binary\ncarries (arithmetic, json-format).", Nested: ""}, + {Key: "min_score", Type: "float64", Doc: "MinScore is the score, 0 to 1, to pass at; zero is the suite's own\nthreshold.", Nested: ""}, + {Key: "judge", Type: "string", Doc: "Judge grades the suite's rubric cases, instead of the judge it names.", Nested: ""}, + }}, {Package: "catalog", Name: "External", Title: "External", Doc: "", Fields: []Field{ {Key: "url", Type: "string", Doc: "URL is the provider's base address, without /v1: https://api.openai.com.", Nested: ""}, {Key: "model", Type: "string", Doc: "Model is what the provider calls it; requests are rewritten to it.", Nested: ""}, @@ -129,6 +135,7 @@ var Reference = []Block{ {Key: "adapters", Type: "[]Adapter", Doc: "Adapters are LoRA adapters served on this model's replicas, each asked\nfor by its own name as though it were a model. Every replica loads\nevery adapter, applying one only to a request that names it.", Nested: ""}, {Key: "fallbacks", Type: "[]string", Doc: "", Nested: ""}, {Key: "canary", Type: "*Canary", Doc: "Canary sends a share of this model's requests to another catalog\nmodel — the same model at new weights — to compare the two before\nmoving to it (`llmgw ctl model canary`).", Nested: "catalog.Canary"}, + {Key: "eval", Type: "*EvalGate", Doc: "Eval is a quality suite new weights must pass before they replace\nthis model's: `llmgw ctl model update` and `model promote` run it\nagainst them first, and stop below its score.", Nested: "catalog.EvalGate"}, {Key: "routing_profile", Type: "string", Doc: "RoutingProfile selects how the router scores endpoints for this model.", Nested: ""}, {Key: "slo", Type: "SLO", Doc: "", Nested: "catalog.SLO"}, {Key: "context_length", Type: "int", Doc: "ContextLength is how many tokens one request may use, prompt and\ncompletion together, and MaxOutput bounds the completion alone. Zero\nmeans unknown, which is what every model was until now.\n\nThe gateway knew neither, so an over-long prompt was not refused: it was\nadmitted, given a replica, and failed at the engine, and a client sizing\nits prompts from /v1/models had nothing to read. Worse, the number is not\nthe one an operator writes in the engine's arguments — llama.cpp divides\n--ctx-size across its parallel slots, so a pool of 8192 across four slots\nis 2048 a request, and nothing said so.", Nested: ""}, diff --git a/internal/router/quality_test.go b/internal/router/quality_test.go new file mode 100644 index 0000000..8893cd4 --- /dev/null +++ b/internal/router/quality_test.go @@ -0,0 +1,131 @@ +package router + +import ( + "encoding/json" + "fmt" + "log/slog" + "net/http" + "net/http/httptest" + "path/filepath" + "strings" + "sync/atomic" + "testing" + "testing/fstest" + "time" + + "gitea.mynodes.xyz/ilker/llmgw/internal/bench/eval" + "gitea.mynodes.xyz/ilker/llmgw/internal/config" + "gitea.mynodes.xyz/ilker/llmgw/internal/store" +) + +// answeringEngine is an engine that answers a few questions right, one +// wrong, and grades as a judge would. +func answeringEngine(t *testing.T, served string) (*httptest.Server, *atomic.Int32) { + t.Helper() + var calls atomic.Int32 + mux := http.NewServeMux() + mux.HandleFunc("GET /metrics", func(w http.ResponseWriter, _ *http.Request) { + fmt.Fprintf(w, "# TYPE vllm:num_requests_running gauge\nvllm:num_requests_running 0\n") + }) + mux.HandleFunc("POST /v1/chat/completions", func(w http.ResponseWriter, r *http.Request) { + calls.Add(1) + var req struct { + Model string `json:"model"` + Messages []struct { + Content string `json:"content"` + } `json:"messages"` + Stream bool `json:"stream"` + } + if err := json.NewDecoder(r.Body).Decode(&req); err != nil || req.Stream || len(req.Messages) == 0 { + http.Error(w, "bad request", http.StatusBadRequest) + return + } + q := req.Messages[len(req.Messages)-1].Content + answer := "I am not sure." + switch { + case served == "grader" && strings.Contains(q, "Answer:\nParis"): + answer = `{"score": 1, "reason": "names Paris"}` + case served == "grader": + answer = `{"score": 0, "reason": "does not name Paris"}` + case strings.Contains(q, "17 + 25"): + answer = "17 and 2542" + case strings.Contains(q, "6 * 7"): + answer = "6 * 7 = 43" + case strings.Contains(q, "JSON"): + answer = "```json\n{\"ok\": true}\n```" + case strings.Contains(q, "capital"): + answer = "Paris" + } + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(map[string]any{"model": req.Model, "choices": []any{map[string]any{ + "message": map[string]any{"role": "assistant", "content": answer}, "finish_reason": "stop"}}, + "usage": map[string]any{"prompt_tokens": 5, "completion_tokens": 3}}) + }) + srv := httptest.NewServer(mux) + t.Cleanup(srv.Close) + return srv, &calls +} + +const qualitySuite = ` +name: smoke +threshold: 0.7 +judge: grader +cases: + - id: add + prompt: What is 17 + 25? + number: 42 + - id: mul + prompt: What is 6 * 7? + number: 42 + - id: json + prompt: Give a JSON object with ok true. + json_schema: {type: object, required: [ok], properties: {ok: {const: true}}} + - id: capital + prompt: What is the capital of France? + judge: It names Paris. + weight: 2 +` + +func TestASuiteIsAskedThroughTheRouterAndScoredAndKept(t *testing.T) { + model, modelCalls := answeringEngine(t, "qwen") + grader, graderCalls := answeringEngine(t, "grader") + rt := newTestRouter(t, baseConfig( + config.Endpoint{ID: "a", URL: model.URL, Models: []string{"qwen"}}, + config.Endpoint{ID: "j", URL: grader.URL, Models: []string{"grader"}}, + )) + db, err := store.OpenSQLite(t.Context(), filepath.Join(t.TempDir(), "store.db")) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = db.Close() }) + svc := eval.NewService(eval.Deps{Call: rt.Replay, Store: db, Log: slog.New(slog.DiscardHandler), + Builtin: fstest.MapFS{"smoke.yaml": {Data: []byte(qualitySuite)}}}) + + live, err := svc.Start("qwen", eval.Request{Suite: "smoke", Concurrency: 2}, "tester") + if err != nil { + t.Fatal(err) + } + deadline := time.Now().Add(10 * time.Second) + for live.State == eval.Running && time.Now().Before(deadline) { + time.Sleep(20 * time.Millisecond) + live, _ = svc.Get(t.Context(), live.ID) + } + // add 1, mul 0, json 1, capital 1 at weight 2: 4 of 5. + if live.State != eval.Done || live.Score != 0.8 || !live.Passed || live.Below != 1 || live.Errors != 0 { + t.Fatalf("%+v", live) + } + if modelCalls.Load() != 4 || graderCalls.Load() != 1 { + t.Errorf("the model was asked %d times and the judge %d", modelCalls.Load(), graderCalls.Load()) + } + kept, err := db.EvalRunByID(t.Context(), live.ID) + if err != nil || len(kept.Results) != 4 || kept.Results[0].Answer != "42" || + !strings.Contains(kept.Results[1].Detail, "read 43") || kept.Results[3].Detail != "the judge: names Paris" { + t.Fatalf("kept %+v %v", kept, err) + } + + // A model nothing serves scores nothing, and says why. + run, err := svc.Run(t.Context(), "nothing", eval.Request{Suite: "smoke"}, "tester", "") + if err != nil || run.Passed || run.Errors != 4 || !strings.Contains(run.Error, "no case was answered") { + t.Fatalf("%+v %v", run, err) + } +} diff --git a/internal/store/copy.go b/internal/store/copy.go index 535ca89..f56e9d5 100644 --- a/internal/store/copy.go +++ b/internal/store/copy.go @@ -62,6 +62,9 @@ var copyOrder = []string{ "batches", "batch_results", "request_bodies", + // Runs before their cases, which point at them. + "eval_runs", + "eval_cases", } // notCopied are tables deliberately left behind. diff --git a/internal/store/eval.go b/internal/store/eval.go new file mode 100644 index 0000000..911bf8e --- /dev/null +++ b/internal/store/eval.go @@ -0,0 +1,136 @@ +package store + +import ( + "context" + "database/sql" + "errors" + "time" +) + +// EvalRun is one quality suite run against a model, as `llmgw ctl eval` +// and the gate on an update keep it. +type EvalRun struct { + ID string `json:"id"` + // Model is the model the run is about; Target the catalog model that + // answered, when that was another — its canary, or the new weights an + // update tried before moving to them. + Model string `json:"model"` + Target string `json:"target,omitempty"` + // Repo and Revision are where the weights that answered came from, + // when that is known. + Repo string `json:"repo,omitempty"` + Revision string `json:"revision,omitempty"` + // Suite is its name, SuiteHash the sha256 of its text. + Suite string `json:"suite"` + SuiteHash string `json:"suite_hash"` + At time.Time `json:"at"` + TookMs int64 `json:"took_ms"` + // Score is the cases' scores weighed, 0 to 1; Passed whether it + // reached Threshold. + Score float64 `json:"score"` + Threshold float64 `json:"threshold"` + Passed bool `json:"passed"` + // Cases were asked; Below scored under half; Errors got no answer. + Cases int `json:"cases"` + Below int `json:"below"` + Errors int `json:"errors"` + Judge string `json:"judge,omitempty"` + By string `json:"by,omitempty"` + // Job is the update or promotion the run gated, if one did. + Job string `json:"job,omitempty"` + // Error is why the run could not be scored at all. + Error string `json:"error,omitempty"` + Results []EvalCase `json:"results,omitempty"` +} + +// EvalCase is one case of a run: how its answer scored, and why. +type EvalCase struct { + ID string `json:"id"` + Kind string `json:"kind"` + Weight float64 `json:"weight"` + Score float64 `json:"score"` + Passed bool `json:"passed"` + Answer string `json:"answer,omitempty"` + Detail string `json:"detail,omitempty"` + Error string `json:"error,omitempty"` + TookMs int64 `json:"took_ms"` +} + +// PutEvalRun keeps a finished run and its cases. +func (s *DB) PutEvalRun(ctx context.Context, r EvalRun) error { + return s.inTx(ctx, func(q conn) error { + if _, err := q.ExecContext(ctx, `INSERT INTO eval_runs (id, model, target, repo, revision, suite, + suite_hash, started_at, took_ms, score, threshold, passed, cases, below, errors, judge, actor, job, error) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, + r.ID, r.Model, r.Target, r.Repo, r.Revision, r.Suite, r.SuiteHash, millis(r.At), r.TookMs, + r.Score, r.Threshold, boolInt(r.Passed), r.Cases, r.Below, r.Errors, r.Judge, r.By, r.Job, r.Error); err != nil { + return err + } + for i, c := range r.Results { + if _, err := q.ExecContext(ctx, `INSERT INTO eval_cases (run_id, idx, case_id, kind, weight, score, + passed, answer, detail, error, took_ms) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, + r.ID, i, c.ID, c.Kind, c.Weight, c.Score, boolInt(c.Passed), c.Answer, c.Detail, c.Error, + c.TookMs); err != nil { + return err + } + } + return nil + }) +} + +func boolInt(b bool) int { + if b { + return 1 + } + return 0 +} + +const evalRunColumns = `id, model, target, repo, revision, suite, suite_hash, started_at, took_ms, score, + threshold, passed, cases, below, errors, judge, actor, job, error` + +func scanEvalRun(row interface{ Scan(...any) error }) (EvalRun, error) { + var r EvalRun + var at int64 + var passed int + err := row.Scan(&r.ID, &r.Model, &r.Target, &r.Repo, &r.Revision, &r.Suite, &r.SuiteHash, &at, &r.TookMs, + &r.Score, &r.Threshold, &passed, &r.Cases, &r.Below, &r.Errors, &r.Judge, &r.By, &r.Job, &r.Error) + r.At, r.Passed = fromMillis(at), passed == 1 + return r, err +} + +// EvalRuns are a model's runs, newest first and without their cases; every +// model's for an empty name. +func (s *DB) EvalRuns(ctx context.Context, model string, limit int) ([]EvalRun, error) { + if limit <= 0 || limit > 1000 { + limit = 20 + } + q := `SELECT ` + evalRunColumns + ` FROM eval_runs` + var args []any + if model != "" { + q += ` WHERE model = ? OR target = ?` + args = append(args, model, model) + } + q += ` ORDER BY started_at DESC, id DESC LIMIT ?` + args = append(args, limit) + return queryAll(ctx, s.db, q, args, func(rows *sql.Rows) (EvalRun, error) { return scanEvalRun(rows) }) +} + +// EvalRunByID is one run with its cases; ErrNotFound when there is none. +func (s *DB) EvalRunByID(ctx context.Context, id string) (EvalRun, error) { + r, err := scanEvalRun(s.db.QueryRowContext(ctx, `SELECT `+evalRunColumns+` FROM eval_runs WHERE id = ?`, id)) + if errors.Is(err, sql.ErrNoRows) { + return EvalRun{}, ErrNotFound + } + if err != nil { + return EvalRun{}, err + } + r.Results, err = queryAll(ctx, s.db, `SELECT case_id, kind, weight, score, passed, answer, detail, error, took_ms + FROM eval_cases WHERE run_id = ? ORDER BY idx`, []any{id}, func(rows *sql.Rows) (EvalCase, error) { + var c EvalCase + var passed int + err := rows.Scan(&c.ID, &c.Kind, &c.Weight, &c.Score, &passed, &c.Answer, &c.Detail, &c.Error, &c.TookMs) + c.Passed = passed == 1 + return c, err + }) + return r, err +} diff --git a/internal/store/eval_test.go b/internal/store/eval_test.go new file mode 100644 index 0000000..a9a5401 --- /dev/null +++ b/internal/store/eval_test.go @@ -0,0 +1,59 @@ +package store + +import ( + "errors" + "testing" + "time" +) + +func TestEvalRunsAreKeptWithTheirCases(t *testing.T) { + checkEvalRuns(t, capacityStore(t)) +} + +func TestPostgresEvalRunsAreKeptWithTheirCases(t *testing.T) { + checkEvalRuns(t, openPG(t)) +} + +func checkEvalRuns(t *testing.T, db *DB) { + ctx := t.Context() + now := time.Now().UTC().Truncate(time.Millisecond) + first := EvalRun{ID: "eval-1", Model: "qwen", Repo: "o/r", Revision: "abc", Suite: "arith", SuiteHash: "h1", + At: now.Add(-time.Hour), TookMs: 1200, Score: 0.9, Threshold: 0.8, Passed: true, Cases: 2, Below: 0, + By: "ilker", Results: []EvalCase{ + {ID: "add", Kind: "number", Weight: 1, Score: 1, Passed: true, Answer: "42", Detail: "read 42", TookMs: 300}, + {ID: "json", Kind: "json_schema", Weight: 2, Score: 0.85, Passed: true, Answer: "{}", TookMs: 900}, + }} + second := EvalRun{ID: "eval-2", Model: "qwen", Target: "qwen-canary", Revision: "def", Suite: "arith", + SuiteHash: "h1", At: now, Score: 0.5, Threshold: 0.8, Cases: 1, Below: 1, Errors: 1, Job: "promote-1", + Results: []EvalCase{{ID: "add", Kind: "number", Weight: 1, Error: "qwen-canary answered 503"}}} + other := EvalRun{ID: "eval-3", Model: "llama", Suite: "fmt", SuiteHash: "h2", At: now, Score: 1, Threshold: 0.8, + Passed: true, Cases: 0} + for _, r := range []EvalRun{first, second, other} { + if err := db.PutEvalRun(ctx, r); err != nil { + t.Fatal(err) + } + } + runs, err := db.EvalRuns(ctx, "qwen", 10) + if err != nil || len(runs) != 2 || runs[0].ID != "eval-2" || runs[1].ID != "eval-1" { + t.Fatalf("%+v %v", runs, err) + } + if r := runs[0]; r.Target != "qwen-canary" || r.Passed || r.Job != "promote-1" || r.Errors != 1 || + !r.At.Equal(now) || r.Results != nil { + t.Errorf("%+v", r) + } + // Asked by the canary's name, its runs are found too. + if got, _ := db.EvalRuns(ctx, "qwen-canary", 10); len(got) != 1 || got[0].ID != "eval-2" { + t.Errorf("by target: %+v", got) + } + if all, _ := db.EvalRuns(ctx, "", 2); len(all) != 2 { + t.Errorf("limit: %d", len(all)) + } + got, err := db.EvalRunByID(ctx, "eval-1") + if err != nil || got.Score != 0.9 || !got.Passed || got.Repo != "o/r" || got.By != "ilker" || len(got.Results) != 2 || + got.Results[1].ID != "json" || got.Results[1].Weight != 2 || got.Results[0].Answer != "42" || !got.Results[0].Passed { + t.Fatalf("%+v %v", got, err) + } + if _, err := db.EvalRunByID(ctx, "eval-9"); !errors.Is(err, ErrNotFound) { + t.Errorf("%v", err) + } +} diff --git a/internal/store/migrations/postgres/0028_eval_runs.sql b/internal/store/migrations/postgres/0028_eval_runs.sql new file mode 100644 index 0000000..1f621ba --- /dev/null +++ b/internal/store/migrations/postgres/0028_eval_runs.sql @@ -0,0 +1,43 @@ +-- Ported from the SQLite migration of the same number. +-- Quality suites run against a model (`llmgw ctl eval`, and the gate on +-- `model update` and `model promote`): one row a run, filed under the model +-- it is about, with the revision its weights came from when that is known, +-- and the suite's hash, so two runs compare only when they asked the same +-- questions. Each case's score and answer beside it. +CREATE TABLE eval_runs ( + id TEXT PRIMARY KEY, + model TEXT NOT NULL, + target TEXT NOT NULL DEFAULT '', + repo TEXT NOT NULL DEFAULT '', + revision TEXT NOT NULL DEFAULT '', + suite TEXT NOT NULL, + suite_hash TEXT NOT NULL, + started_at BIGINT NOT NULL, + took_ms BIGINT NOT NULL DEFAULT 0, + score DOUBLE PRECISION NOT NULL, + threshold DOUBLE PRECISION NOT NULL, + passed BIGINT NOT NULL, + cases BIGINT NOT NULL, + below BIGINT NOT NULL DEFAULT 0, + errors BIGINT NOT NULL DEFAULT 0, + judge TEXT NOT NULL DEFAULT '', + actor TEXT NOT NULL DEFAULT '', + job TEXT NOT NULL DEFAULT '', + error TEXT NOT NULL DEFAULT '' +); +CREATE INDEX eval_runs_model ON eval_runs (model, started_at); + +CREATE TABLE eval_cases ( + run_id TEXT NOT NULL REFERENCES eval_runs (id) ON DELETE CASCADE, + idx BIGINT NOT NULL, + case_id TEXT NOT NULL, + kind TEXT NOT NULL, + weight DOUBLE PRECISION NOT NULL, + score DOUBLE PRECISION NOT NULL, + passed BIGINT NOT NULL, + answer TEXT NOT NULL DEFAULT '', + detail TEXT NOT NULL DEFAULT '', + error TEXT NOT NULL DEFAULT '', + took_ms BIGINT NOT NULL DEFAULT 0, + PRIMARY KEY (run_id, idx) +); diff --git a/internal/store/migrations/sqlite/0028_eval_runs.sql b/internal/store/migrations/sqlite/0028_eval_runs.sql new file mode 100644 index 0000000..701843f --- /dev/null +++ b/internal/store/migrations/sqlite/0028_eval_runs.sql @@ -0,0 +1,42 @@ +-- Quality suites run against a model (`llmgw ctl eval`, and the gate on +-- `model update` and `model promote`): one row a run, filed under the model +-- it is about, with the revision its weights came from when that is known, +-- and the suite's hash, so two runs compare only when they asked the same +-- questions. Each case's score and answer beside it. +CREATE TABLE eval_runs ( + id TEXT PRIMARY KEY, + model TEXT NOT NULL, + target TEXT NOT NULL DEFAULT '', + repo TEXT NOT NULL DEFAULT '', + revision TEXT NOT NULL DEFAULT '', + suite TEXT NOT NULL, + suite_hash TEXT NOT NULL, + started_at INTEGER NOT NULL, + took_ms INTEGER NOT NULL DEFAULT 0, + score REAL NOT NULL, + threshold REAL NOT NULL, + passed INTEGER NOT NULL, + cases INTEGER NOT NULL, + below INTEGER NOT NULL DEFAULT 0, + errors INTEGER NOT NULL DEFAULT 0, + judge TEXT NOT NULL DEFAULT '', + actor TEXT NOT NULL DEFAULT '', + job TEXT NOT NULL DEFAULT '', + error TEXT NOT NULL DEFAULT '' +); +CREATE INDEX eval_runs_model ON eval_runs (model, started_at); + +CREATE TABLE eval_cases ( + run_id TEXT NOT NULL REFERENCES eval_runs (id) ON DELETE CASCADE, + idx INTEGER NOT NULL, + case_id TEXT NOT NULL, + kind TEXT NOT NULL, + weight REAL NOT NULL, + score REAL NOT NULL, + passed INTEGER NOT NULL, + answer TEXT NOT NULL DEFAULT '', + detail TEXT NOT NULL DEFAULT '', + error TEXT NOT NULL DEFAULT '', + took_ms INTEGER NOT NULL DEFAULT 0, + PRIMARY KEY (run_id, idx) +); diff --git a/internal/store/postgres_test.go b/internal/store/postgres_test.go index 010df30..c78b7e2 100644 --- a/internal/store/postgres_test.go +++ b/internal/store/postgres_test.go @@ -43,7 +43,8 @@ func openPG(t *testing.T) *DB { // every subtest gets a fresh file, and failed only here. `TRUNCATE thread_messages, threads, memory_notes, usage, usage_5m, usage_1h, usage_1d, api_keys, memberships, key_claims, limits, models, sessions, users, projects, teams, - organizations, leadership, control_planes, files, batches, batch_results, request_bodies RESTART IDENTITY CASCADE`) + organizations, leadership, control_planes, files, batches, batch_results, request_bodies, + eval_runs, eval_cases RESTART IDENTITY CASCADE`) if err != nil { t.Fatalf("clearing the test database: %v", err) } diff --git a/notes/c21.md b/notes/c21.md new file mode 100644 index 0000000..baa30a3 --- /dev/null +++ b/notes/c21.md @@ -0,0 +1,62 @@ +## CHANGELOG + +### Quality suites, and new weights held to one + +- **`llmgw ctl eval MODEL --suite FILE|NAME`** asks a model a quality suite + and scores its answers: per case, then the cases weighed into one score, 0 + to 1, against the suite's `threshold` (or `--min-score`); it exits non-zero + below it. `--concurrency`, `--judge`, `--detach`, `--json`. + - **Suites** are YAML or JSON lines: a prompt or a conversation, a weight, + and one way to score it — `exact`, `contains`, `regex`, `number` within + a `tolerance` (the last number in the answer), `json_schema` (the answer + read as JSON, in a code fence or a sentence or not, held to a schema), or + `judge`: a rubric another model on the cluster grades 0 to 1. A + reasoning model's `` is not part of its answer. + - **Run on the control plane**, through the router as its own requests, + as a replay is: past admission, quotas, accounting and the guardrails. + `--suite` is a file sent whole, or a suite's name in `control.eval_dir` + (`/etc/llmgw/eval`) or built in: `arithmetic` and `json-format`, in + `deploy/eval/`, sized for small models. `--suites` lists them. + - **Kept** in the store (migration 0028): every run with its model, the + repository and revision of the weights that answered, the suite's name + and hash, its score and each case's answer and reason. + `llmgw ctl eval --history [MODEL]` lists them, `--run ID` shows one. +- **Updates and promotions held to a suite.** `llmgw ctl model update` and + `model promote` take `--eval SUITE --min-score N`, or a model carries one + (`llmgw ctl model gate MODEL --eval SUITE`, `eval:` in its definition). + A promotion asks the canary first and leaves it in place below the score; + an update runs the new weights first as a canary given none of the + model's requests, asks it, and promotes it or takes it out again, so the + old weights serve until the new have passed. The job's `eval` step says + the score; `--skip-eval` moves once without the model's suite. +- **The console** has **Ask it a quality suite…** (the score counted in as it + comes), **Its quality runs** and **Hold its updates to a quality suite…** + on a model's menu. +- **Fixed:** a change to the admin API with a body over 64 KiB reached its + handler cut off at 64 KiB, by the audit log's reading of it. + +## BACKLOG + +**Done (branch `features-c2`):** `internal/bench/eval` (suites in YAML or +JSON lines; exact, contains, regex, number, a JSON Schema subset, and a judge +model with a rubric; weights and a threshold), run on the control plane +through `Router.Replay`; `llmgw ctl eval MODEL --suite FILE|NAME`, `--history`, +`--run`, `--suites`; runs and their cases kept (migration 0028, `eval_runs` +and `eval_cases`); `model update`/`promote --eval SUITE --min-score N`, a +model's standing gate (`ctl model gate`, `eval:` in the catalog) — a promote +asks the canary first, an update tries the new weights as a 0% canary and +promotes or removes it; the console's model menu; `arithmetic` and +`json-format` in `deploy/eval/`, carried in the binary. Left: the suite of a +model's standing gate is read by name on the control plane at the time of +the update, so two control planes need the same `control.eval_dir`; a gated +update needs room for one more replica while it runs (there is no in-place +gate, and a declared model in server.yaml cannot be updated at all); no +comparison against the old weights' score (the gate is an absolute +threshold, not "no worse than before"); a run is not stopped once started +(Esc in the console only stops watching it), and the in-memory progress of a +run lives on the control plane that started it; runs are never pruned; the +JSON Schema support is a subset (no `$ref`, `oneOf`, `not`, `uniqueItems`); +judge cases go through the same concurrency as the model's, and a judge's +answers are not kept, only its score and reason; a gateway without a +control plane cannot run suites (there is no mode that asks through the +client API with a key).