Quality suites, and new weights held to one: ctl eval, model update/promote --eval, model gate

A suite is a set of cases in YAML or JSON lines, each scored by exact,
contains, regex, a number within a tolerance, a JSON Schema subset, or a
judge model with a rubric; weighed into one score against a threshold.
internal/bench/eval runs one on the control plane through Router.Replay, as
the gateway's own requests, and keeps every run and its cases in the store
(migration 0028: eval_runs, eval_cases).

- llmgw ctl eval MODEL --suite FILE|NAME, --history, --run, --suites; exits
  non-zero below the score. Suites by name from control.eval_dir, or the
  two built in from deploy/eval (arithmetic, json-format).
- model update and promote take --eval SUITE --min-score N, or a model's
  standing gate (ctl model gate, eval: in the catalog): a promote asks the
  canary first and leaves it below the score; an update tries the new
  weights as a canary given no share, then promotes or removes it.
- The console's model menu: ask a suite with its progress, its runs, its
  gate.
- The audit middleware no longer hands a handler a body cut off at 64 KiB.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-05 00:04:43 +02:00
1 parent 30f7f3474a
commit f0477f140d
49 files changed
+4563 -46

No files matched your search

+46
View File
@@ -63,6 +63,7 @@ rather than repeated.
| `port_range` | `[]int` | [Control](#control) |
| `mgmt_interface` | `string` | [Control](#control) |
| `timezone` | `string` | [Control](#control) |
| `eval_dir` | `string` | [Control](#control) |
| `enabled` | `bool` | [Control](#control) |
| `lease_ttl` | `time.Duration` | [Control](#control) |
| `id` | `string` | [Endpoint](#endpoint) |
@@ -77,6 +78,9 @@ rather than repeated.
| `nodes` | `[]string` | [EngineOverride](#engineoverride) |
| `labels` | `map[string]string` | [EngineOverride](#engineoverride) |
| `engines` | `map[string]string` | [EngineOverride](#engineoverride) |
| `suite` | `string` | [EvalGate](#evalgate) |
| `min_score` | `float64` | [EvalGate](#evalgate) |
| `judge` | `string` | [EvalGate](#evalgate) |
| `url` | `string` | [External](#external) |
| `model` | `string` | [External](#external) |
| `api_key_file` | `string` | [External](#external) |
@@ -109,6 +113,7 @@ rather than repeated.
| `adapters` | `[]Adapter` | [Model](#model) |
| `fallbacks` | `[]string` | [Model](#model) |
| `canary` | `*Canary` | [Model](#model) |
| `eval` | `*EvalGate` | [Model](#model) |
| `routing_profile` | `string` | [Model](#model) |
| `slo` | `SLO` | [Model](#model) |
| `context_length` | `int` | [Model](#model) |
@@ -665,6 +670,15 @@ MgmtInterface is the ethernet interface NCCL bootstraps over.
Timezone is where models' scaling schedules are read, an IANA name
like Europe/Istanbul; empty is the control plane's own.
### `eval_dir`
`string`
EvalDir is where quality suites are read from by name, for `llmgw ctl
eval MODEL --suite NAME`, a model's eval gate and the console. The
suites the binary carries (arithmetic, json-format) are found when it
has none of the name; a suite here of the same name replaces one.
### `enabled`
`bool`
@@ -758,6 +772,28 @@ Labels picks nodes carrying all of these.
Engines are laid over the cluster's: a version, or "" for none.
## EvalGate
### `suite`
`string`
Suite is a suite's name in control.eval_dir, or one the binary
carries (arithmetic, json-format).
### `min_score`
`float64`
MinScore is the score, 0 to 1, to pass at; zero is the suite's own
threshold.
### `judge`
`string`
Judge grades the suite's rubric cases, instead of the judge it names.
## External
### `url`
@@ -1024,6 +1060,16 @@ moving to it (`llmgw ctl model canary`).
Takes a [Canary](#canary) block.
### `eval`
`*EvalGate`
Eval is a quality suite new weights must pass before they replace
this model's: `llmgw ctl model update` and `model promote` run it
against them first, and stop below its score.
Takes a [EvalGate](#evalgate) block.
### `routing_profile`
`string`