llmgw/notes/c21.md

CHANGELOG

Quality suites, and new weights held to one

• llmgw ctl eval MODEL --suite FILE|NAME asks a model a quality suite and scores its answers: per case, then the cases weighed into one score, 0 to 1, against the suite's threshold (or --min-score); it exits non-zero below it. --concurrency, --judge, --detach, --json.
  • Suites are YAML or JSON lines: a prompt or a conversation, a weight, and one way to score it — exact, contains, regex, number within a tolerance (the last number in the answer), json_schema (the answer read as JSON, in a code fence or a sentence or not, held to a schema), or judge: a rubric another model on the cluster grades 0 to 1. A reasoning model's <think> is not part of its answer.
  • Run on the control plane, through the router as its own requests, as a replay is: past admission, quotas, accounting and the guardrails. --suite is a file sent whole, or a suite's name in control.eval_dir (/etc/llmgw/eval) or built in: arithmetic and json-format, in deploy/eval/, sized for small models. --suites lists them.
  • Kept in the store (migration 0028): every run with its model, the repository and revision of the weights that answered, the suite's name and hash, its score and each case's answer and reason. llmgw ctl eval --history [MODEL] lists them, --run ID shows one.
• Updates and promotions held to a suite. llmgw ctl model update and model promote take --eval SUITE --min-score N, or a model carries one (llmgw ctl model gate MODEL --eval SUITE, eval: in its definition). A promotion asks the canary first and leaves it in place below the score; an update runs the new weights first as a canary given none of the model's requests, asks it, and promotes it or takes it out again, so the old weights serve until the new have passed. The job's eval step says the score; --skip-eval moves once without the model's suite.
• The console has Ask it a quality suite… (the score counted in as it comes), Its quality runs and Hold its updates to a quality suite… on a model's menu.
• Fixed: a change to the admin API with a body over 64 KiB reached its handler cut off at 64 KiB, by the audit log's reading of it.

BACKLOG

Done (branch features-c2): internal/bench/eval (suites in YAML or JSON lines; exact, contains, regex, number, a JSON Schema subset, and a judge model with a rubric; weights and a threshold), run on the control plane through Router.Replay; llmgw ctl eval MODEL --suite FILE|NAME, --history, --run, --suites; runs and their cases kept (migration 0028, eval_runs and eval_cases); model update/promote --eval SUITE --min-score N, a model's standing gate (ctl model gate, eval: in the catalog) — a promote asks the canary first, an update tries the new weights as a 0% canary and promotes or removes it; the console's model menu; arithmetic and json-format in deploy/eval/, carried in the binary. Left: the suite of a model's standing gate is read by name on the control plane at the time of the update, so two control planes need the same control.eval_dir; a gated update needs room for one more replica while it runs (there is no in-place gate, and a declared model in server.yaml cannot be updated at all); no comparison against the old weights' score (the gate is an absolute threshold, not "no worse than before"); a run is not stopped once started (Esc in the console only stops watching it), and the in-memory progress of a run lives on the control plane that started it; runs are never pruned; the JSON Schema support is a subset (no $ref, oneOf, not, uniqueItems); judge cases go through the same concurrency as the model's, and a judge's answers are not kept, only its score and reason; a gateway without a control plane cannot run suites (there is no mode that asks through the client API with a key).