Merge C21 quality suites and gating

Conflicts: docs.go (manual, OpenAPI and suites all kept), modelserve Job
(trace and eval both kept). The eval routes are added to the OpenAPI route
table, update and promote documented with their optional body, and the
document regenerated.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-05 00:08:33 +02:00
commit d41104a452
51 files changed
+5204 -49

No files matched your search

+46
View File
@@ -65,6 +65,7 @@ rather than repeated.
| `port_range` | `[]int` | [Control](#control) |
| `mgmt_interface` | `string` | [Control](#control) |
| `timezone` | `string` | [Control](#control) |
| `eval_dir` | `string` | [Control](#control) |
| `enabled` | `bool` | [Control](#control) |
| `lease_ttl` | `time.Duration` | [Control](#control) |
| `id` | `string` | [Endpoint](#endpoint) |
@@ -79,6 +80,9 @@ rather than repeated.
| `nodes` | `[]string` | [EngineOverride](#engineoverride) |
| `labels` | `map[string]string` | [EngineOverride](#engineoverride) |
| `engines` | `map[string]string` | [EngineOverride](#engineoverride) |
| `suite` | `string` | [EvalGate](#evalgate) |
| `min_score` | `float64` | [EvalGate](#evalgate) |
| `judge` | `string` | [EvalGate](#evalgate) |
| `url` | `string` | [External](#external) |
| `model` | `string` | [External](#external) |
| `api_key_file` | `string` | [External](#external) |
@@ -112,6 +116,7 @@ rather than repeated.
| `adapters` | `[]Adapter` | [Model](#model) |
| `fallbacks` | `[]string` | [Model](#model) |
| `canary` | `*Canary` | [Model](#model) |
| `eval` | `*EvalGate` | [Model](#model) |
| `routing_profile` | `string` | [Model](#model) |
| `slo` | `SLO` | [Model](#model) |
| `context_length` | `int` | [Model](#model) |
@@ -688,6 +693,15 @@ MgmtInterface is the ethernet interface NCCL bootstraps over.
Timezone is where models' scaling schedules are read, an IANA name
like Europe/Istanbul; empty is the control plane's own.
### `eval_dir`
`string`
EvalDir is where quality suites are read from by name, for `llmgw ctl
eval MODEL --suite NAME`, a model's eval gate and the console. The
suites the binary carries (arithmetic, json-format) are found when it
has none of the name; a suite here of the same name replaces one.
### `enabled`
`bool`
@@ -781,6 +795,28 @@ Labels picks nodes carrying all of these.
Engines are laid over the cluster's: a version, or "" for none.
## EvalGate
### `suite`
`string`
Suite is a suite's name in control.eval_dir, or one the binary
carries (arithmetic, json-format).
### `min_score`
`float64`
MinScore is the score, 0 to 1, to pass at; zero is the suite's own
threshold.
### `judge`
`string`
Judge grades the suite's rubric cases, instead of the judge it names.
## External
### `url`
@@ -1058,6 +1094,16 @@ moving to it (`llmgw ctl model canary`).
Takes a [Canary](#canary) block.
### `eval`
`*EvalGate`
Eval is a quality suite new weights must pass before they replace
this model's: `llmgw ctl model update` and `model promote` run it
against them first, and stop below its score.
Takes a [EvalGate](#evalgate) block.
### `routing_profile`
`string`