Quality suites, and new weights held to one: ctl eval, model update/promote --eval, model gate

A suite is a set of cases in YAML or JSON lines, each scored by exact,
contains, regex, a number within a tolerance, a JSON Schema subset, or a
judge model with a rubric; weighed into one score against a threshold.
internal/bench/eval runs one on the control plane through Router.Replay, as
the gateway's own requests, and keeps every run and its cases in the store
(migration 0028: eval_runs, eval_cases).

- llmgw ctl eval MODEL --suite FILE|NAME, --history, --run, --suites; exits
  non-zero below the score. Suites by name from control.eval_dir, or the
  two built in from deploy/eval (arithmetic, json-format).
- model update and promote take --eval SUITE --min-score N, or a model's
  standing gate (ctl model gate, eval: in the catalog): a promote asks the
  canary first and leaves it below the score; an update tries the new
  weights as a canary given no share, then promotes or removes it.
- The console's model menu: ask a suite with its progress, its runs, its
  gate.
- The audit middleware no longer hands a handler a body cut off at 64 KiB.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-05 00:04:43 +02:00
1 parent 30f7f3474a
commit f0477f140d
49 files changed
+4563 -46

No files matched your search

+112
View File
@@ -911,6 +911,118 @@ llmgw ctl model canary qwen-7b --abort # or not
- **Promote** is the update the canary was a trial of: the model's replicas
are replaced one at a time, and `llmgw ctl model rollback` goes back.
### Quality suites, and holding new weights to one
A canary says whether new weights serve as fast and as reliably; it does not
say whether their answers are as good. A republished quantisation or a chat
template that no longer matches can make a model faster and worse at once. A
quality suite is a handful of questions with known answers, asked of a model
and scored:
```
llmgw ctl eval --suites # what the control plane can run
llmgw ctl eval qwen-7b --suite arithmetic # by name, on the control plane
llmgw ctl eval qwen-7b --suite ./support.yaml --judge llama-70b --concurrency 8
llmgw ctl eval --history qwen-7b # the runs kept
llmgw ctl eval --run eval-20261004-101500-1a2b3c4d # one, case by case
```
It prints each case's score and why, then the suite's: the cases' scores
weighed, 0 to 1, and whether it reached the suite's `threshold` (or
`--min-score`); it exits non-zero below it, so a script can stop on it.
- **How it is asked:** on the control plane, through the router as the
gateway's own requests, the way a replay is — past admission, quotas,
accounting and the guardrails, so the score is the weights' and not a
tenant's limits'. It reaches the model exactly as a client's request
would, a canary or a model loading on demand included. Answers are asked
at temperature 0 unless a case says otherwise, and a reasoning model's
`<think>` block is not part of its answer.
- **Where suites come from:** `--suite` is a file on the machine running
`ctl`, sent whole, or the name of one in the control plane's
`control.eval_dir` (`/etc/llmgw/eval`). Two are built in and run on small
models: `arithmetic` (sums and word problems, each scored by the number in
the answer) and `json-format` (answers asked for as JSON of a given shape);
their files are in `deploy/eval/`, to copy and change. A file in the
directory with the same name replaces the built-in one.
- **What is kept:** every run, in the store, with the model, the repository
and revision the answering weights came from (when known), the suite's name
and sha256, when, the score, and each case's answer (up to 2,000
characters), score and reason. A run on a canary is filed under the model
it is a canary of. Runs are not pruned.
A suite is YAML (or JSON lines: one case on each line, and optionally a line
without a prompt that holds the suite's own settings):
```yaml
name: support # the file's name unless given
threshold: 0.8 # the score to pass at, 0 to 1; 0.8 unless given
judge: llama-70b # grades the cases scored by a rubric
system: You are a support agent for Acme. # every case's, unless it has one
max_tokens: 256 # every case's; 512 unless given
cases:
- id: refund-window
prompt: How many days do customers have to ask for a refund?
number: 30 # the last number in the answer, ±tolerance
tolerance: 0
- prompt: Which plan includes SSO?
contains: [Enterprise] # every string must appear
ignore_case: true
- prompt: Reply with only the word OK.
exact: OK # the whole answer, white space around it aside
- prompt: What is today's ISO date format?
regex: '\d{4}-\d{2}-\d{2}' # matches somewhere in the answer
- prompt: Give the order as JSON with "id" and "total".
json_schema: # the answer, read as JSON, satisfies it
type: object
required: [id, total]
properties: { id: {type: string}, total: {type: number, minimum: 0} }
- messages: # a conversation instead of a prompt
- { role: user, content: My card was charged twice. }
judge: Apologises, and says the duplicate charge is refunded within 5 days.
weight: 3 # counts three times; 1 unless given
```
Each case is scored by exactly one of `exact`, `contains`, `regex`, `number`,
`json_schema` or `judge`. A JSON answer in a code fence or inside a sentence is
still read. The schema supports `type`, `properties`, `required`,
`additionalProperties`, `items`, `enum`, `const`, the numeric and length bounds,
`pattern`, `anyOf` and `allOf`; a suite using anything else is refused when it
is read rather than passed unchecked. A `judge` case asks another model on the
cluster (`judge_model:` on the case, else the suite's `judge:`, else `--judge`)
to grade the answer by the rubric, 0 to 1. A case passes at 0.5; a case that
got no answer scores 0 and says why.
**Holding updates to a suite.** `model update` and `model promote` take
`--eval SUITE --min-score N` (and `--judge`); or a model carries one — set
with `ctl model gate`, or `eval: { suite: arithmetic, min_score: 0.8 }` in a
definition added with `ctl model add` — so every update and promotion of it
is held to it, from `ctl`, the console or anywhere:
```
llmgw ctl model gate qwen-7b --eval arithmetic --min-score 0.8
llmgw ctl model update qwen-7b # asked the suite first
llmgw ctl model update qwen-7b --skip-eval # once, without it; the job says so
llmgw ctl model gate qwen-7b --off
```
- **Promote** asks the canary the suite before anything moves; below the
score the job fails with the score, and the canary stays, with its share,
until you take it out.
- **Update** runs the new weights first as `MODEL-canary` given none of the
model's requests (hidden from clients like any canary), asks it the suite,
then either promotes it — the model's replicas are replaced one at a time,
as for any update — or takes it out again, leaving the model on its old
weights. That needs room for one more replica while it runs; a model that
already has a canary is promoted or aborted first. Without a suite, an
update swaps the weights in as it always has.
- The job's `eval` step says the score (`llmgw ctl jobs JOB`), and the run is
kept with the job's id beside it.
The console has it on a model's menu: **Ask it a quality suite…** (a suite the
control plane has, the score as it comes in), **Its quality runs**, and **Hold
its updates to a quality suite…**.
### How many replicas, and how much each one carries
Replica count is changed while the cluster runs, without editing the