Files
llmgw/notes/c21.md
T
Ilker ManapandClaude Opus 5.5 f0477f140d Quality suites, and new weights held to one: ctl eval, model update/promote --eval, model gate
A suite is a set of cases in YAML or JSON lines, each scored by exact,
contains, regex, a number within a tolerance, a JSON Schema subset, or a
judge model with a rubric; weighed into one score against a threshold.
internal/bench/eval runs one on the control plane through Router.Replay, as
the gateway's own requests, and keeps every run and its cases in the store
(migration 0028: eval_runs, eval_cases).

- llmgw ctl eval MODEL --suite FILE|NAME, --history, --run, --suites; exits
  non-zero below the score. Suites by name from control.eval_dir, or the
  two built in from deploy/eval (arithmetic, json-format).
- model update and promote take --eval SUITE --min-score N, or a model's
  standing gate (ctl model gate, eval: in the catalog): a promote asks the
  canary first and leaves it below the score; an update tries the new
  weights as a canary given no share, then promotes or removes it.
- The console's model menu: ask a suite with its progress, its runs, its
  gate.
- The audit middleware no longer hands a handler a body cut off at 64 KiB.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 00:04:43 +02:00

63 lines
3.9 KiB
Markdown

## CHANGELOG
### Quality suites, and new weights held to one
- **`llmgw ctl eval MODEL --suite FILE|NAME`** asks a model a quality suite
and scores its answers: per case, then the cases weighed into one score, 0
to 1, against the suite's `threshold` (or `--min-score`); it exits non-zero
below it. `--concurrency`, `--judge`, `--detach`, `--json`.
- **Suites** are YAML or JSON lines: a prompt or a conversation, a weight,
and one way to score it — `exact`, `contains`, `regex`, `number` within
a `tolerance` (the last number in the answer), `json_schema` (the answer
read as JSON, in a code fence or a sentence or not, held to a schema), or
`judge`: a rubric another model on the cluster grades 0 to 1. A
reasoning model's `<think>` is not part of its answer.
- **Run on the control plane**, through the router as its own requests,
as a replay is: past admission, quotas, accounting and the guardrails.
`--suite` is a file sent whole, or a suite's name in `control.eval_dir`
(`/etc/llmgw/eval`) or built in: `arithmetic` and `json-format`, in
`deploy/eval/`, sized for small models. `--suites` lists them.
- **Kept** in the store (migration 0028): every run with its model, the
repository and revision of the weights that answered, the suite's name
and hash, its score and each case's answer and reason.
`llmgw ctl eval --history [MODEL]` lists them, `--run ID` shows one.
- **Updates and promotions held to a suite.** `llmgw ctl model update` and
`model promote` take `--eval SUITE --min-score N`, or a model carries one
(`llmgw ctl model gate MODEL --eval SUITE`, `eval:` in its definition).
A promotion asks the canary first and leaves it in place below the score;
an update runs the new weights first as a canary given none of the
model's requests, asks it, and promotes it or takes it out again, so the
old weights serve until the new have passed. The job's `eval` step says
the score; `--skip-eval` moves once without the model's suite.
- **The console** has **Ask it a quality suite…** (the score counted in as it
comes), **Its quality runs** and **Hold its updates to a quality suite…**
on a model's menu.
- **Fixed:** a change to the admin API with a body over 64 KiB reached its
handler cut off at 64 KiB, by the audit log's reading of it.
## BACKLOG
**Done (branch `features-c2`):** `internal/bench/eval` (suites in YAML or
JSON lines; exact, contains, regex, number, a JSON Schema subset, and a judge
model with a rubric; weights and a threshold), run on the control plane
through `Router.Replay`; `llmgw ctl eval MODEL --suite FILE|NAME`, `--history`,
`--run`, `--suites`; runs and their cases kept (migration 0028, `eval_runs`
and `eval_cases`); `model update`/`promote --eval SUITE --min-score N`, a
model's standing gate (`ctl model gate`, `eval:` in the catalog) — a promote
asks the canary first, an update tries the new weights as a 0% canary and
promotes or removes it; the console's model menu; `arithmetic` and
`json-format` in `deploy/eval/`, carried in the binary. Left: the suite of a
model's standing gate is read by name on the control plane at the time of
the update, so two control planes need the same `control.eval_dir`; a gated
update needs room for one more replica while it runs (there is no in-place
gate, and a declared model in server.yaml cannot be updated at all); no
comparison against the old weights' score (the gate is an absolute
threshold, not "no worse than before"); a run is not stopped once started
(Esc in the console only stops watching it), and the in-memory progress of a
run lives on the control plane that started it; runs are never pruned; the
JSON Schema support is a subset (no `$ref`, `oneOf`, `not`, `uniqueItems`);
judge cases go through the same concurrency as the model's, and a judge's
answers are not kept, only its score and reason; a gateway without a
control plane cannot run suites (there is no mode that asks through the
client API with a key).