Quality suites, and new weights held to one: ctl eval, model update/promote --eval, model gate
A suite is a set of cases in YAML or JSON lines, each scored by exact, contains, regex, a number within a tolerance, a JSON Schema subset, or a judge model with a rubric; weighed into one score against a threshold. internal/bench/eval runs one on the control plane through Router.Replay, as the gateway's own requests, and keeps every run and its cases in the store (migration 0028: eval_runs, eval_cases). - llmgw ctl eval MODEL --suite FILE|NAME, --history, --run, --suites; exits non-zero below the score. Suites by name from control.eval_dir, or the two built in from deploy/eval (arithmetic, json-format). - model update and promote take --eval SUITE --min-score N, or a model's standing gate (ctl model gate, eval: in the catalog): a promote asks the canary first and leaves it below the score; an update tries the new weights as a canary given no share, then promotes or removes it. - The console's model menu: ask a suite with its progress, its runs, its gate. - The audit middleware no longer hands a handler a body cut off at 64 KiB. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
30f7f3474a
commit
f0477f140d
49 files changed
+4563
-46
No files matched your search
+112
@@ -911,6 +911,118 @@ llmgw ctl model canary qwen-7b --abort # or not
|
||||
- **Promote** is the update the canary was a trial of: the model's replicas
|
||||
are replaced one at a time, and `llmgw ctl model rollback` goes back.
|
||||
|
||||
### Quality suites, and holding new weights to one
|
||||
|
||||
A canary says whether new weights serve as fast and as reliably; it does not
|
||||
say whether their answers are as good. A republished quantisation or a chat
|
||||
template that no longer matches can make a model faster and worse at once. A
|
||||
quality suite is a handful of questions with known answers, asked of a model
|
||||
and scored:
|
||||
|
||||
```
|
||||
llmgw ctl eval --suites # what the control plane can run
|
||||
llmgw ctl eval qwen-7b --suite arithmetic # by name, on the control plane
|
||||
llmgw ctl eval qwen-7b --suite ./support.yaml --judge llama-70b --concurrency 8
|
||||
llmgw ctl eval --history qwen-7b # the runs kept
|
||||
llmgw ctl eval --run eval-20261004-101500-1a2b3c4d # one, case by case
|
||||
```
|
||||
|
||||
It prints each case's score and why, then the suite's: the cases' scores
|
||||
weighed, 0 to 1, and whether it reached the suite's `threshold` (or
|
||||
`--min-score`); it exits non-zero below it, so a script can stop on it.
|
||||
|
||||
- **How it is asked:** on the control plane, through the router as the
|
||||
gateway's own requests, the way a replay is — past admission, quotas,
|
||||
accounting and the guardrails, so the score is the weights' and not a
|
||||
tenant's limits'. It reaches the model exactly as a client's request
|
||||
would, a canary or a model loading on demand included. Answers are asked
|
||||
at temperature 0 unless a case says otherwise, and a reasoning model's
|
||||
`<think>` block is not part of its answer.
|
||||
- **Where suites come from:** `--suite` is a file on the machine running
|
||||
`ctl`, sent whole, or the name of one in the control plane's
|
||||
`control.eval_dir` (`/etc/llmgw/eval`). Two are built in and run on small
|
||||
models: `arithmetic` (sums and word problems, each scored by the number in
|
||||
the answer) and `json-format` (answers asked for as JSON of a given shape);
|
||||
their files are in `deploy/eval/`, to copy and change. A file in the
|
||||
directory with the same name replaces the built-in one.
|
||||
- **What is kept:** every run, in the store, with the model, the repository
|
||||
and revision the answering weights came from (when known), the suite's name
|
||||
and sha256, when, the score, and each case's answer (up to 2,000
|
||||
characters), score and reason. A run on a canary is filed under the model
|
||||
it is a canary of. Runs are not pruned.
|
||||
|
||||
A suite is YAML (or JSON lines: one case on each line, and optionally a line
|
||||
without a prompt that holds the suite's own settings):
|
||||
|
||||
```yaml
|
||||
name: support # the file's name unless given
|
||||
threshold: 0.8 # the score to pass at, 0 to 1; 0.8 unless given
|
||||
judge: llama-70b # grades the cases scored by a rubric
|
||||
system: You are a support agent for Acme. # every case's, unless it has one
|
||||
max_tokens: 256 # every case's; 512 unless given
|
||||
cases:
|
||||
- id: refund-window
|
||||
prompt: How many days do customers have to ask for a refund?
|
||||
number: 30 # the last number in the answer, ±tolerance
|
||||
tolerance: 0
|
||||
- prompt: Which plan includes SSO?
|
||||
contains: [Enterprise] # every string must appear
|
||||
ignore_case: true
|
||||
- prompt: Reply with only the word OK.
|
||||
exact: OK # the whole answer, white space around it aside
|
||||
- prompt: What is today's ISO date format?
|
||||
regex: '\d{4}-\d{2}-\d{2}' # matches somewhere in the answer
|
||||
- prompt: Give the order as JSON with "id" and "total".
|
||||
json_schema: # the answer, read as JSON, satisfies it
|
||||
type: object
|
||||
required: [id, total]
|
||||
properties: { id: {type: string}, total: {type: number, minimum: 0} }
|
||||
- messages: # a conversation instead of a prompt
|
||||
- { role: user, content: My card was charged twice. }
|
||||
judge: Apologises, and says the duplicate charge is refunded within 5 days.
|
||||
weight: 3 # counts three times; 1 unless given
|
||||
```
|
||||
|
||||
Each case is scored by exactly one of `exact`, `contains`, `regex`, `number`,
|
||||
`json_schema` or `judge`. A JSON answer in a code fence or inside a sentence is
|
||||
still read. The schema supports `type`, `properties`, `required`,
|
||||
`additionalProperties`, `items`, `enum`, `const`, the numeric and length bounds,
|
||||
`pattern`, `anyOf` and `allOf`; a suite using anything else is refused when it
|
||||
is read rather than passed unchecked. A `judge` case asks another model on the
|
||||
cluster (`judge_model:` on the case, else the suite's `judge:`, else `--judge`)
|
||||
to grade the answer by the rubric, 0 to 1. A case passes at 0.5; a case that
|
||||
got no answer scores 0 and says why.
|
||||
|
||||
**Holding updates to a suite.** `model update` and `model promote` take
|
||||
`--eval SUITE --min-score N` (and `--judge`); or a model carries one — set
|
||||
with `ctl model gate`, or `eval: { suite: arithmetic, min_score: 0.8 }` in a
|
||||
definition added with `ctl model add` — so every update and promotion of it
|
||||
is held to it, from `ctl`, the console or anywhere:
|
||||
|
||||
```
|
||||
llmgw ctl model gate qwen-7b --eval arithmetic --min-score 0.8
|
||||
llmgw ctl model update qwen-7b # asked the suite first
|
||||
llmgw ctl model update qwen-7b --skip-eval # once, without it; the job says so
|
||||
llmgw ctl model gate qwen-7b --off
|
||||
```
|
||||
|
||||
- **Promote** asks the canary the suite before anything moves; below the
|
||||
score the job fails with the score, and the canary stays, with its share,
|
||||
until you take it out.
|
||||
- **Update** runs the new weights first as `MODEL-canary` given none of the
|
||||
model's requests (hidden from clients like any canary), asks it the suite,
|
||||
then either promotes it — the model's replicas are replaced one at a time,
|
||||
as for any update — or takes it out again, leaving the model on its old
|
||||
weights. That needs room for one more replica while it runs; a model that
|
||||
already has a canary is promoted or aborted first. Without a suite, an
|
||||
update swaps the weights in as it always has.
|
||||
- The job's `eval` step says the score (`llmgw ctl jobs JOB`), and the run is
|
||||
kept with the job's id beside it.
|
||||
|
||||
The console has it on a model's menu: **Ask it a quality suite…** (a suite the
|
||||
control plane has, the score as it comes in), **Its quality runs**, and **Hold
|
||||
its updates to a quality suite…**.
|
||||
|
||||
### How many replicas, and how much each one carries
|
||||
|
||||
Replica count is changed while the cluster runs, without editing the
|
||||
|
||||
Reference in new issue
Block a user