Quality suites, and new weights held to one: ctl eval, model update/promote --eval, model gate
A suite is a set of cases in YAML or JSON lines, each scored by exact, contains, regex, a number within a tolerance, a JSON Schema subset, or a judge model with a rubric; weighed into one score against a threshold. internal/bench/eval runs one on the control plane through Router.Replay, as the gateway's own requests, and keeps every run and its cases in the store (migration 0028: eval_runs, eval_cases). - llmgw ctl eval MODEL --suite FILE|NAME, --history, --run, --suites; exits non-zero below the score. Suites by name from control.eval_dir, or the two built in from deploy/eval (arithmetic, json-format). - model update and promote take --eval SUITE --min-score N, or a model's standing gate (ctl model gate, eval: in the catalog): a promote asks the canary first and leaves it below the score; an update tries the new weights as a canary given no share, then promotes or removes it. - The console's model menu: ask a suite with its progress, its runs, its gate. - The audit middleware no longer hands a handler a body cut off at 64 KiB. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
30f7f3474a
commit
f0477f140d
49 files changed
+4563
-46
No files matched your search
@@ -117,6 +117,16 @@ and **UPDATE** when the registry has new weights for it. On a row:
|
||||
canary is removed) and **Take the canary out…**. A canary that errs or
|
||||
answers more slowly than the model is taken out by itself. The MODEL
|
||||
column shows `(10% to canary)` and `(canary of …)`.
|
||||
- **Ask it a quality suite…** — one of the suites the control plane has (in
|
||||
`control.eval_dir`, or built in: `arithmetic`, `json-format`), asked
|
||||
through the router; the dialog counts the answers as they come in, then
|
||||
shows the score, each case and the model's last runs. Esc stops watching;
|
||||
the run carries on and is kept. **Its quality runs** lists them, as
|
||||
`llmgw ctl eval --history` does.
|
||||
- **Hold its updates to a quality suite…** — the suite and score its new
|
||||
weights must reach before an update or a promotion moves it to them
|
||||
(`llmgw ctl model gate`); **none** takes it away. With one set, **Update
|
||||
to …** and **Promote the canary…** say so.
|
||||
- **Why is it placed where it is** — `llmgw ctl explain-placement`.
|
||||
- **Try again**, for a model parked after failing every start.
|
||||
- **Remove** — its replicas stop and it leaves the catalog. A model from
|
||||
|
||||
+112
@@ -911,6 +911,118 @@ llmgw ctl model canary qwen-7b --abort # or not
|
||||
- **Promote** is the update the canary was a trial of: the model's replicas
|
||||
are replaced one at a time, and `llmgw ctl model rollback` goes back.
|
||||
|
||||
### Quality suites, and holding new weights to one
|
||||
|
||||
A canary says whether new weights serve as fast and as reliably; it does not
|
||||
say whether their answers are as good. A republished quantisation or a chat
|
||||
template that no longer matches can make a model faster and worse at once. A
|
||||
quality suite is a handful of questions with known answers, asked of a model
|
||||
and scored:
|
||||
|
||||
```
|
||||
llmgw ctl eval --suites # what the control plane can run
|
||||
llmgw ctl eval qwen-7b --suite arithmetic # by name, on the control plane
|
||||
llmgw ctl eval qwen-7b --suite ./support.yaml --judge llama-70b --concurrency 8
|
||||
llmgw ctl eval --history qwen-7b # the runs kept
|
||||
llmgw ctl eval --run eval-20261004-101500-1a2b3c4d # one, case by case
|
||||
```
|
||||
|
||||
It prints each case's score and why, then the suite's: the cases' scores
|
||||
weighed, 0 to 1, and whether it reached the suite's `threshold` (or
|
||||
`--min-score`); it exits non-zero below it, so a script can stop on it.
|
||||
|
||||
- **How it is asked:** on the control plane, through the router as the
|
||||
gateway's own requests, the way a replay is — past admission, quotas,
|
||||
accounting and the guardrails, so the score is the weights' and not a
|
||||
tenant's limits'. It reaches the model exactly as a client's request
|
||||
would, a canary or a model loading on demand included. Answers are asked
|
||||
at temperature 0 unless a case says otherwise, and a reasoning model's
|
||||
`<think>` block is not part of its answer.
|
||||
- **Where suites come from:** `--suite` is a file on the machine running
|
||||
`ctl`, sent whole, or the name of one in the control plane's
|
||||
`control.eval_dir` (`/etc/llmgw/eval`). Two are built in and run on small
|
||||
models: `arithmetic` (sums and word problems, each scored by the number in
|
||||
the answer) and `json-format` (answers asked for as JSON of a given shape);
|
||||
their files are in `deploy/eval/`, to copy and change. A file in the
|
||||
directory with the same name replaces the built-in one.
|
||||
- **What is kept:** every run, in the store, with the model, the repository
|
||||
and revision the answering weights came from (when known), the suite's name
|
||||
and sha256, when, the score, and each case's answer (up to 2,000
|
||||
characters), score and reason. A run on a canary is filed under the model
|
||||
it is a canary of. Runs are not pruned.
|
||||
|
||||
A suite is YAML (or JSON lines: one case on each line, and optionally a line
|
||||
without a prompt that holds the suite's own settings):
|
||||
|
||||
```yaml
|
||||
name: support # the file's name unless given
|
||||
threshold: 0.8 # the score to pass at, 0 to 1; 0.8 unless given
|
||||
judge: llama-70b # grades the cases scored by a rubric
|
||||
system: You are a support agent for Acme. # every case's, unless it has one
|
||||
max_tokens: 256 # every case's; 512 unless given
|
||||
cases:
|
||||
- id: refund-window
|
||||
prompt: How many days do customers have to ask for a refund?
|
||||
number: 30 # the last number in the answer, ±tolerance
|
||||
tolerance: 0
|
||||
- prompt: Which plan includes SSO?
|
||||
contains: [Enterprise] # every string must appear
|
||||
ignore_case: true
|
||||
- prompt: Reply with only the word OK.
|
||||
exact: OK # the whole answer, white space around it aside
|
||||
- prompt: What is today's ISO date format?
|
||||
regex: '\d{4}-\d{2}-\d{2}' # matches somewhere in the answer
|
||||
- prompt: Give the order as JSON with "id" and "total".
|
||||
json_schema: # the answer, read as JSON, satisfies it
|
||||
type: object
|
||||
required: [id, total]
|
||||
properties: { id: {type: string}, total: {type: number, minimum: 0} }
|
||||
- messages: # a conversation instead of a prompt
|
||||
- { role: user, content: My card was charged twice. }
|
||||
judge: Apologises, and says the duplicate charge is refunded within 5 days.
|
||||
weight: 3 # counts three times; 1 unless given
|
||||
```
|
||||
|
||||
Each case is scored by exactly one of `exact`, `contains`, `regex`, `number`,
|
||||
`json_schema` or `judge`. A JSON answer in a code fence or inside a sentence is
|
||||
still read. The schema supports `type`, `properties`, `required`,
|
||||
`additionalProperties`, `items`, `enum`, `const`, the numeric and length bounds,
|
||||
`pattern`, `anyOf` and `allOf`; a suite using anything else is refused when it
|
||||
is read rather than passed unchecked. A `judge` case asks another model on the
|
||||
cluster (`judge_model:` on the case, else the suite's `judge:`, else `--judge`)
|
||||
to grade the answer by the rubric, 0 to 1. A case passes at 0.5; a case that
|
||||
got no answer scores 0 and says why.
|
||||
|
||||
**Holding updates to a suite.** `model update` and `model promote` take
|
||||
`--eval SUITE --min-score N` (and `--judge`); or a model carries one — set
|
||||
with `ctl model gate`, or `eval: { suite: arithmetic, min_score: 0.8 }` in a
|
||||
definition added with `ctl model add` — so every update and promotion of it
|
||||
is held to it, from `ctl`, the console or anywhere:
|
||||
|
||||
```
|
||||
llmgw ctl model gate qwen-7b --eval arithmetic --min-score 0.8
|
||||
llmgw ctl model update qwen-7b # asked the suite first
|
||||
llmgw ctl model update qwen-7b --skip-eval # once, without it; the job says so
|
||||
llmgw ctl model gate qwen-7b --off
|
||||
```
|
||||
|
||||
- **Promote** asks the canary the suite before anything moves; below the
|
||||
score the job fails with the score, and the canary stays, with its share,
|
||||
until you take it out.
|
||||
- **Update** runs the new weights first as `MODEL-canary` given none of the
|
||||
model's requests (hidden from clients like any canary), asks it the suite,
|
||||
then either promotes it — the model's replicas are replaced one at a time,
|
||||
as for any update — or takes it out again, leaving the model on its old
|
||||
weights. That needs room for one more replica while it runs; a model that
|
||||
already has a canary is promoted or aborted first. Without a suite, an
|
||||
update swaps the weights in as it always has.
|
||||
- The job's `eval` step says the score (`llmgw ctl jobs JOB`), and the run is
|
||||
kept with the job's id beside it.
|
||||
|
||||
The console has it on a model's menu: **Ask it a quality suite…** (a suite the
|
||||
control plane has, the score as it comes in), **Its quality runs**, and **Hold
|
||||
its updates to a quality suite…**.
|
||||
|
||||
### How many replicas, and how much each one carries
|
||||
|
||||
Replica count is changed while the cluster runs, without editing the
|
||||
|
||||
@@ -63,6 +63,7 @@ rather than repeated.
|
||||
| `port_range` | `[]int` | [Control](#control) |
|
||||
| `mgmt_interface` | `string` | [Control](#control) |
|
||||
| `timezone` | `string` | [Control](#control) |
|
||||
| `eval_dir` | `string` | [Control](#control) |
|
||||
| `enabled` | `bool` | [Control](#control) |
|
||||
| `lease_ttl` | `time.Duration` | [Control](#control) |
|
||||
| `id` | `string` | [Endpoint](#endpoint) |
|
||||
@@ -77,6 +78,9 @@ rather than repeated.
|
||||
| `nodes` | `[]string` | [EngineOverride](#engineoverride) |
|
||||
| `labels` | `map[string]string` | [EngineOverride](#engineoverride) |
|
||||
| `engines` | `map[string]string` | [EngineOverride](#engineoverride) |
|
||||
| `suite` | `string` | [EvalGate](#evalgate) |
|
||||
| `min_score` | `float64` | [EvalGate](#evalgate) |
|
||||
| `judge` | `string` | [EvalGate](#evalgate) |
|
||||
| `url` | `string` | [External](#external) |
|
||||
| `model` | `string` | [External](#external) |
|
||||
| `api_key_file` | `string` | [External](#external) |
|
||||
@@ -109,6 +113,7 @@ rather than repeated.
|
||||
| `adapters` | `[]Adapter` | [Model](#model) |
|
||||
| `fallbacks` | `[]string` | [Model](#model) |
|
||||
| `canary` | `*Canary` | [Model](#model) |
|
||||
| `eval` | `*EvalGate` | [Model](#model) |
|
||||
| `routing_profile` | `string` | [Model](#model) |
|
||||
| `slo` | `SLO` | [Model](#model) |
|
||||
| `context_length` | `int` | [Model](#model) |
|
||||
@@ -665,6 +670,15 @@ MgmtInterface is the ethernet interface NCCL bootstraps over.
|
||||
Timezone is where models' scaling schedules are read, an IANA name
|
||||
like Europe/Istanbul; empty is the control plane's own.
|
||||
|
||||
### `eval_dir`
|
||||
|
||||
`string`
|
||||
|
||||
EvalDir is where quality suites are read from by name, for `llmgw ctl
|
||||
eval MODEL --suite NAME`, a model's eval gate and the console. The
|
||||
suites the binary carries (arithmetic, json-format) are found when it
|
||||
has none of the name; a suite here of the same name replaces one.
|
||||
|
||||
### `enabled`
|
||||
|
||||
`bool`
|
||||
@@ -758,6 +772,28 @@ Labels picks nodes carrying all of these.
|
||||
|
||||
Engines are laid over the cluster's: a version, or "" for none.
|
||||
|
||||
## EvalGate
|
||||
|
||||
### `suite`
|
||||
|
||||
`string`
|
||||
|
||||
Suite is a suite's name in control.eval_dir, or one the binary
|
||||
carries (arithmetic, json-format).
|
||||
|
||||
### `min_score`
|
||||
|
||||
`float64`
|
||||
|
||||
MinScore is the score, 0 to 1, to pass at; zero is the suite's own
|
||||
threshold.
|
||||
|
||||
### `judge`
|
||||
|
||||
`string`
|
||||
|
||||
Judge grades the suite's rubric cases, instead of the judge it names.
|
||||
|
||||
## External
|
||||
|
||||
### `url`
|
||||
@@ -1024,6 +1060,16 @@ moving to it (`llmgw ctl model canary`).
|
||||
|
||||
Takes a [Canary](#canary) block.
|
||||
|
||||
### `eval`
|
||||
|
||||
`*EvalGate`
|
||||
|
||||
Eval is a quality suite new weights must pass before they replace
|
||||
this model's: `llmgw ctl model update` and `model promote` run it
|
||||
against them first, and stop below its score.
|
||||
|
||||
Takes a [EvalGate](#evalgate) block.
|
||||
|
||||
### `routing_profile`
|
||||
|
||||
`string`
|
||||
|
||||
Reference in new issue
Block a user