Quality suites, and new weights held to one: ctl eval, model update/promote --eval, model gate

A suite is a set of cases in YAML or JSON lines, each scored by exact,
contains, regex, a number within a tolerance, a JSON Schema subset, or a
judge model with a rubric; weighed into one score against a threshold.
internal/bench/eval runs one on the control plane through Router.Replay, as
the gateway's own requests, and keeps every run and its cases in the store
(migration 0028: eval_runs, eval_cases).

- llmgw ctl eval MODEL --suite FILE|NAME, --history, --run, --suites; exits
  non-zero below the score. Suites by name from control.eval_dir, or the
  two built in from deploy/eval (arithmetic, json-format).
- model update and promote take --eval SUITE --min-score N, or a model's
  standing gate (ctl model gate, eval: in the catalog): a promote asks the
  canary first and leaves it below the score; an update tries the new
  weights as a canary given no share, then promotes or removes it.
- The console's model menu: ask a suite with its progress, its runs, its
  gate.
- The audit middleware no longer hands a handler a body cut off at 64 KiB.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-05 00:04:43 +02:00
1 parent 30f7f3474a
commit f0477f140d
49 files changed
+4563 -46

No files matched your search

+17 -2
View File
@@ -82,7 +82,10 @@ func updateCell(u modelserve.UpdateInfo) string {
}
func newModelUpdateCmd(client func() *ctlClient) *cobra.Command {
var detach bool
var (
detach bool
gate gateFlags
)
c := &cobra.Command{
Use: "update MODEL",
Short: "Move a model to the weights its repository publishes now",
@@ -90,12 +93,23 @@ func newModelUpdateCmd(client func() *ctlClient) *cobra.Command {
"checked against the registry's checksums and the nodes' llama.cpp; then\n" +
"the catalog names them and the replicas are replaced one at a time, each\n" +
"new one up before an old one stops, and one is asked to say hello.\n\n" +
"Held to a quality suite (--eval, or the model's own: `llmgw ctl model\n" +
"gate`), the new weights first run beside the old as MODEL-canary, given\n" +
"none of its requests, and are asked the suite; below its score they are\n" +
"taken out again and the model keeps its weights. That needs room for one\n" +
"more replica while it runs.\n\n" +
"The old files stay on the mirror: `llmgw ctl model rollback MODEL` goes\n" +
"back to them. A model from server.yaml is changed there instead.",
Example: " llmgw ctl model update qwen-7b\n" +
" llmgw ctl model update qwen-7b --eval arithmetic --min-score 0.8",
Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
body, err := gate.body()
if err != nil {
return err
}
var job modelserve.Job
if err := client().post(cmd.Context(), "/api/v1/models/"+args[0]+"/update", []byte("{}"), &job); err != nil {
if err := client().post(cmd.Context(), "/api/v1/models/"+args[0]+"/update", body, &job); err != nil {
return err
}
if detach {
@@ -105,6 +119,7 @@ func newModelUpdateCmd(client func() *ctlClient) *cobra.Command {
},
}
c.Flags().BoolVar(&detach, "detach", false, "start it and return: it runs on the control plane; `llmgw ctl jobs` follows it")
gate.add(c)
return c
}