Quality suites, and new weights held to one: ctl eval, model update/promote --eval, model gate
A suite is a set of cases in YAML or JSON lines, each scored by exact, contains, regex, a number within a tolerance, a JSON Schema subset, or a judge model with a rubric; weighed into one score against a threshold. internal/bench/eval runs one on the control plane through Router.Replay, as the gateway's own requests, and keeps every run and its cases in the store (migration 0028: eval_runs, eval_cases). - llmgw ctl eval MODEL --suite FILE|NAME, --history, --run, --suites; exits non-zero below the score. Suites by name from control.eval_dir, or the two built in from deploy/eval (arithmetic, json-format). - model update and promote take --eval SUITE --min-score N, or a model's standing gate (ctl model gate, eval: in the catalog): a promote asks the canary first and leaves it below the score; an update tries the new weights as a canary given no share, then promotes or removes it. - The console's model menu: ask a suite with its progress, its runs, its gate. - The audit middleware no longer hands a handler a body cut off at 64 KiB. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
30f7f3474a
commit
f0477f140d
49 files changed
+4563
-46
No files matched your search
@@ -63,6 +63,7 @@ rather than repeated.
|
||||
| `port_range` | `[]int` | [Control](#control) |
|
||||
| `mgmt_interface` | `string` | [Control](#control) |
|
||||
| `timezone` | `string` | [Control](#control) |
|
||||
| `eval_dir` | `string` | [Control](#control) |
|
||||
| `enabled` | `bool` | [Control](#control) |
|
||||
| `lease_ttl` | `time.Duration` | [Control](#control) |
|
||||
| `id` | `string` | [Endpoint](#endpoint) |
|
||||
@@ -77,6 +78,9 @@ rather than repeated.
|
||||
| `nodes` | `[]string` | [EngineOverride](#engineoverride) |
|
||||
| `labels` | `map[string]string` | [EngineOverride](#engineoverride) |
|
||||
| `engines` | `map[string]string` | [EngineOverride](#engineoverride) |
|
||||
| `suite` | `string` | [EvalGate](#evalgate) |
|
||||
| `min_score` | `float64` | [EvalGate](#evalgate) |
|
||||
| `judge` | `string` | [EvalGate](#evalgate) |
|
||||
| `url` | `string` | [External](#external) |
|
||||
| `model` | `string` | [External](#external) |
|
||||
| `api_key_file` | `string` | [External](#external) |
|
||||
@@ -109,6 +113,7 @@ rather than repeated.
|
||||
| `adapters` | `[]Adapter` | [Model](#model) |
|
||||
| `fallbacks` | `[]string` | [Model](#model) |
|
||||
| `canary` | `*Canary` | [Model](#model) |
|
||||
| `eval` | `*EvalGate` | [Model](#model) |
|
||||
| `routing_profile` | `string` | [Model](#model) |
|
||||
| `slo` | `SLO` | [Model](#model) |
|
||||
| `context_length` | `int` | [Model](#model) |
|
||||
@@ -665,6 +670,15 @@ MgmtInterface is the ethernet interface NCCL bootstraps over.
|
||||
Timezone is where models' scaling schedules are read, an IANA name
|
||||
like Europe/Istanbul; empty is the control plane's own.
|
||||
|
||||
### `eval_dir`
|
||||
|
||||
`string`
|
||||
|
||||
EvalDir is where quality suites are read from by name, for `llmgw ctl
|
||||
eval MODEL --suite NAME`, a model's eval gate and the console. The
|
||||
suites the binary carries (arithmetic, json-format) are found when it
|
||||
has none of the name; a suite here of the same name replaces one.
|
||||
|
||||
### `enabled`
|
||||
|
||||
`bool`
|
||||
@@ -758,6 +772,28 @@ Labels picks nodes carrying all of these.
|
||||
|
||||
Engines are laid over the cluster's: a version, or "" for none.
|
||||
|
||||
## EvalGate
|
||||
|
||||
### `suite`
|
||||
|
||||
`string`
|
||||
|
||||
Suite is a suite's name in control.eval_dir, or one the binary
|
||||
carries (arithmetic, json-format).
|
||||
|
||||
### `min_score`
|
||||
|
||||
`float64`
|
||||
|
||||
MinScore is the score, 0 to 1, to pass at; zero is the suite's own
|
||||
threshold.
|
||||
|
||||
### `judge`
|
||||
|
||||
`string`
|
||||
|
||||
Judge grades the suite's rubric cases, instead of the judge it names.
|
||||
|
||||
## External
|
||||
|
||||
### `url`
|
||||
@@ -1024,6 +1060,16 @@ moving to it (`llmgw ctl model canary`).
|
||||
|
||||
Takes a [Canary](#canary) block.
|
||||
|
||||
### `eval`
|
||||
|
||||
`*EvalGate`
|
||||
|
||||
Eval is a quality suite new weights must pass before they replace
|
||||
this model's: `llmgw ctl model update` and `model promote` run it
|
||||
against them first, and stop below its score.
|
||||
|
||||
Takes a [EvalGate](#evalgate) block.
|
||||
|
||||
### `routing_profile`
|
||||
|
||||
`string`
|
||||
|
||||
Reference in new issue
Block a user