## CHANGELOG ### Quality suites, and new weights held to one - **`llmgw ctl eval MODEL --suite FILE|NAME`** asks a model a quality suite and scores its answers: per case, then the cases weighed into one score, 0 to 1, against the suite's `threshold` (or `--min-score`); it exits non-zero below it. `--concurrency`, `--judge`, `--detach`, `--json`. - **Suites** are YAML or JSON lines: a prompt or a conversation, a weight, and one way to score it — `exact`, `contains`, `regex`, `number` within a `tolerance` (the last number in the answer), `json_schema` (the answer read as JSON, in a code fence or a sentence or not, held to a schema), or `judge`: a rubric another model on the cluster grades 0 to 1. A reasoning model's `` is not part of its answer. - **Run on the control plane**, through the router as its own requests, as a replay is: past admission, quotas, accounting and the guardrails. `--suite` is a file sent whole, or a suite's name in `control.eval_dir` (`/etc/llmgw/eval`) or built in: `arithmetic` and `json-format`, in `deploy/eval/`, sized for small models. `--suites` lists them. - **Kept** in the store (migration 0028): every run with its model, the repository and revision of the weights that answered, the suite's name and hash, its score and each case's answer and reason. `llmgw ctl eval --history [MODEL]` lists them, `--run ID` shows one. - **Updates and promotions held to a suite.** `llmgw ctl model update` and `model promote` take `--eval SUITE --min-score N`, or a model carries one (`llmgw ctl model gate MODEL --eval SUITE`, `eval:` in its definition). A promotion asks the canary first and leaves it in place below the score; an update runs the new weights first as a canary given none of the model's requests, asks it, and promotes it or takes it out again, so the old weights serve until the new have passed. The job's `eval` step says the score; `--skip-eval` moves once without the model's suite. - **The console** has **Ask it a quality suite…** (the score counted in as it comes), **Its quality runs** and **Hold its updates to a quality suite…** on a model's menu. - **Fixed:** a change to the admin API with a body over 64 KiB reached its handler cut off at 64 KiB, by the audit log's reading of it. ## BACKLOG **Done (branch `features-c2`):** `internal/bench/eval` (suites in YAML or JSON lines; exact, contains, regex, number, a JSON Schema subset, and a judge model with a rubric; weights and a threshold), run on the control plane through `Router.Replay`; `llmgw ctl eval MODEL --suite FILE|NAME`, `--history`, `--run`, `--suites`; runs and their cases kept (migration 0028, `eval_runs` and `eval_cases`); `model update`/`promote --eval SUITE --min-score N`, a model's standing gate (`ctl model gate`, `eval:` in the catalog) — a promote asks the canary first, an update tries the new weights as a 0% canary and promotes or removes it; the console's model menu; `arithmetic` and `json-format` in `deploy/eval/`, carried in the binary. Left: the suite of a model's standing gate is read by name on the control plane at the time of the update, so two control planes need the same `control.eval_dir`; a gated update needs room for one more replica while it runs (there is no in-place gate, and a declared model in server.yaml cannot be updated at all); no comparison against the old weights' score (the gate is an absolute threshold, not "no worse than before"); a run is not stopped once started (Esc in the console only stops watching it), and the in-memory progress of a run lives on the control plane that started it; runs are never pruned; the JSON Schema support is a subset (no `$ref`, `oneOf`, `not`, `uniqueItems`); judge cases go through the same concurrency as the model's, and a judge's answers are not kept, only its score and reason; a gateway without a control plane cannot run suites (there is no mode that asks through the client API with a key).