Quality suites, and new weights held to one: ctl eval, model update/promote --eval, model gate
A suite is a set of cases in YAML or JSON lines, each scored by exact, contains, regex, a number within a tolerance, a JSON Schema subset, or a judge model with a rubric; weighed into one score against a threshold. internal/bench/eval runs one on the control plane through Router.Replay, as the gateway's own requests, and keeps every run and its cases in the store (migration 0028: eval_runs, eval_cases). - llmgw ctl eval MODEL --suite FILE|NAME, --history, --run, --suites; exits non-zero below the score. Suites by name from control.eval_dir, or the two built in from deploy/eval (arithmetic, json-format). - model update and promote take --eval SUITE --min-score N, or a model's standing gate (ctl model gate, eval: in the catalog): a promote asks the canary first and leaves it below the score; an update tries the new weights as a canary given no share, then promotes or removes it. - The console's model menu: ask a suite with its progress, its runs, its gate. - The audit middleware no longer hands a handler a body cut off at 64 KiB. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
30f7f3474a
commit
f0477f140d
49 files changed
+4563
-46
No files matched your search
@@ -82,7 +82,10 @@ func updateCell(u modelserve.UpdateInfo) string {
|
||||
}
|
||||
|
||||
func newModelUpdateCmd(client func() *ctlClient) *cobra.Command {
|
||||
var detach bool
|
||||
var (
|
||||
detach bool
|
||||
gate gateFlags
|
||||
)
|
||||
c := &cobra.Command{
|
||||
Use: "update MODEL",
|
||||
Short: "Move a model to the weights its repository publishes now",
|
||||
@@ -90,12 +93,23 @@ func newModelUpdateCmd(client func() *ctlClient) *cobra.Command {
|
||||
"checked against the registry's checksums and the nodes' llama.cpp; then\n" +
|
||||
"the catalog names them and the replicas are replaced one at a time, each\n" +
|
||||
"new one up before an old one stops, and one is asked to say hello.\n\n" +
|
||||
"Held to a quality suite (--eval, or the model's own: `llmgw ctl model\n" +
|
||||
"gate`), the new weights first run beside the old as MODEL-canary, given\n" +
|
||||
"none of its requests, and are asked the suite; below its score they are\n" +
|
||||
"taken out again and the model keeps its weights. That needs room for one\n" +
|
||||
"more replica while it runs.\n\n" +
|
||||
"The old files stay on the mirror: `llmgw ctl model rollback MODEL` goes\n" +
|
||||
"back to them. A model from server.yaml is changed there instead.",
|
||||
Example: " llmgw ctl model update qwen-7b\n" +
|
||||
" llmgw ctl model update qwen-7b --eval arithmetic --min-score 0.8",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
body, err := gate.body()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var job modelserve.Job
|
||||
if err := client().post(cmd.Context(), "/api/v1/models/"+args[0]+"/update", []byte("{}"), &job); err != nil {
|
||||
if err := client().post(cmd.Context(), "/api/v1/models/"+args[0]+"/update", body, &job); err != nil {
|
||||
return err
|
||||
}
|
||||
if detach {
|
||||
@@ -105,6 +119,7 @@ func newModelUpdateCmd(client func() *ctlClient) *cobra.Command {
|
||||
},
|
||||
}
|
||||
c.Flags().BoolVar(&detach, "detach", false, "start it and return: it runs on the control plane; `llmgw ctl jobs` follows it")
|
||||
gate.add(c)
|
||||
return c
|
||||
}
|
||||
|
||||
|
||||
Reference in new issue
Block a user