A suite is a set of cases in YAML or JSON lines, each scored by exact, contains, regex, a number within a tolerance, a JSON Schema subset, or a judge model with a rubric; weighed into one score against a threshold. internal/bench/eval runs one on the control plane through Router.Replay, as the gateway's own requests, and keeps every run and its cases in the store (migration 0028: eval_runs, eval_cases). - llmgw ctl eval MODEL --suite FILE|NAME, --history, --run, --suites; exits non-zero below the score. Suites by name from control.eval_dir, or the two built in from deploy/eval (arithmetic, json-format). - model update and promote take --eval SUITE --min-score N, or a model's standing gate (ctl model gate, eval: in the catalog): a promote asks the canary first and leaves it below the score; an update tries the new weights as a canary given no share, then promotes or removes it. - The console's model menu: ask a suite with its progress, its runs, its gate. - The audit middleware no longer hands a handler a body cut off at 64 KiB. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
63 lines
3.9 KiB
Markdown
63 lines
3.9 KiB
Markdown
## CHANGELOG
|
|
|
|
### Quality suites, and new weights held to one
|
|
|
|
- **`llmgw ctl eval MODEL --suite FILE|NAME`** asks a model a quality suite
|
|
and scores its answers: per case, then the cases weighed into one score, 0
|
|
to 1, against the suite's `threshold` (or `--min-score`); it exits non-zero
|
|
below it. `--concurrency`, `--judge`, `--detach`, `--json`.
|
|
- **Suites** are YAML or JSON lines: a prompt or a conversation, a weight,
|
|
and one way to score it — `exact`, `contains`, `regex`, `number` within
|
|
a `tolerance` (the last number in the answer), `json_schema` (the answer
|
|
read as JSON, in a code fence or a sentence or not, held to a schema), or
|
|
`judge`: a rubric another model on the cluster grades 0 to 1. A
|
|
reasoning model's `<think>` is not part of its answer.
|
|
- **Run on the control plane**, through the router as its own requests,
|
|
as a replay is: past admission, quotas, accounting and the guardrails.
|
|
`--suite` is a file sent whole, or a suite's name in `control.eval_dir`
|
|
(`/etc/llmgw/eval`) or built in: `arithmetic` and `json-format`, in
|
|
`deploy/eval/`, sized for small models. `--suites` lists them.
|
|
- **Kept** in the store (migration 0028): every run with its model, the
|
|
repository and revision of the weights that answered, the suite's name
|
|
and hash, its score and each case's answer and reason.
|
|
`llmgw ctl eval --history [MODEL]` lists them, `--run ID` shows one.
|
|
- **Updates and promotions held to a suite.** `llmgw ctl model update` and
|
|
`model promote` take `--eval SUITE --min-score N`, or a model carries one
|
|
(`llmgw ctl model gate MODEL --eval SUITE`, `eval:` in its definition).
|
|
A promotion asks the canary first and leaves it in place below the score;
|
|
an update runs the new weights first as a canary given none of the
|
|
model's requests, asks it, and promotes it or takes it out again, so the
|
|
old weights serve until the new have passed. The job's `eval` step says
|
|
the score; `--skip-eval` moves once without the model's suite.
|
|
- **The console** has **Ask it a quality suite…** (the score counted in as it
|
|
comes), **Its quality runs** and **Hold its updates to a quality suite…**
|
|
on a model's menu.
|
|
- **Fixed:** a change to the admin API with a body over 64 KiB reached its
|
|
handler cut off at 64 KiB, by the audit log's reading of it.
|
|
|
|
## BACKLOG
|
|
|
|
**Done (branch `features-c2`):** `internal/bench/eval` (suites in YAML or
|
|
JSON lines; exact, contains, regex, number, a JSON Schema subset, and a judge
|
|
model with a rubric; weights and a threshold), run on the control plane
|
|
through `Router.Replay`; `llmgw ctl eval MODEL --suite FILE|NAME`, `--history`,
|
|
`--run`, `--suites`; runs and their cases kept (migration 0028, `eval_runs`
|
|
and `eval_cases`); `model update`/`promote --eval SUITE --min-score N`, a
|
|
model's standing gate (`ctl model gate`, `eval:` in the catalog) — a promote
|
|
asks the canary first, an update tries the new weights as a 0% canary and
|
|
promotes or removes it; the console's model menu; `arithmetic` and
|
|
`json-format` in `deploy/eval/`, carried in the binary. Left: the suite of a
|
|
model's standing gate is read by name on the control plane at the time of
|
|
the update, so two control planes need the same `control.eval_dir`; a gated
|
|
update needs room for one more replica while it runs (there is no in-place
|
|
gate, and a declared model in server.yaml cannot be updated at all); no
|
|
comparison against the old weights' score (the gate is an absolute
|
|
threshold, not "no worse than before"); a run is not stopped once started
|
|
(Esc in the console only stops watching it), and the in-memory progress of a
|
|
run lives on the control plane that started it; runs are never pruned; the
|
|
JSON Schema support is a subset (no `$ref`, `oneOf`, `not`, `uniqueItems`);
|
|
judge cases go through the same concurrency as the model's, and a judge's
|
|
answers are not kept, only its score and reason; a gateway without a
|
|
control plane cannot run suites (there is no mode that asks through the
|
|
client API with a key).
|