Quality suites, and new weights held to one: ctl eval, model update/promote --eval, model gate
A suite is a set of cases in YAML or JSON lines, each scored by exact, contains, regex, a number within a tolerance, a JSON Schema subset, or a judge model with a rubric; weighed into one score against a threshold. internal/bench/eval runs one on the control plane through Router.Replay, as the gateway's own requests, and keeps every run and its cases in the store (migration 0028: eval_runs, eval_cases). - llmgw ctl eval MODEL --suite FILE|NAME, --history, --run, --suites; exits non-zero below the score. Suites by name from control.eval_dir, or the two built in from deploy/eval (arithmetic, json-format). - model update and promote take --eval SUITE --min-score N, or a model's standing gate (ctl model gate, eval: in the catalog): a promote asks the canary first and leaves it below the score; an update tries the new weights as a canary given no share, then promotes or removes it. - The console's model menu: ask a suite with its progress, its runs, its gate. - The audit middleware no longer hands a handler a body cut off at 64 KiB. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
30f7f3474a
commit
f0477f140d
49 files changed
+4563
-46
No files matched your search
@@ -117,6 +117,16 @@ and **UPDATE** when the registry has new weights for it. On a row:
|
||||
canary is removed) and **Take the canary out…**. A canary that errs or
|
||||
answers more slowly than the model is taken out by itself. The MODEL
|
||||
column shows `(10% to canary)` and `(canary of …)`.
|
||||
- **Ask it a quality suite…** — one of the suites the control plane has (in
|
||||
`control.eval_dir`, or built in: `arithmetic`, `json-format`), asked
|
||||
through the router; the dialog counts the answers as they come in, then
|
||||
shows the score, each case and the model's last runs. Esc stops watching;
|
||||
the run carries on and is kept. **Its quality runs** lists them, as
|
||||
`llmgw ctl eval --history` does.
|
||||
- **Hold its updates to a quality suite…** — the suite and score its new
|
||||
weights must reach before an update or a promotion moves it to them
|
||||
(`llmgw ctl model gate`); **none** takes it away. With one set, **Update
|
||||
to …** and **Promote the canary…** say so.
|
||||
- **Why is it placed where it is** — `llmgw ctl explain-placement`.
|
||||
- **Try again**, for a model parked after failing every start.
|
||||
- **Remove** — its replicas stop and it leaves the catalog. A model from
|
||||
|
||||
Reference in new issue
Block a user