Quality suites, and new weights held to one: ctl eval, model update/promote --eval, model gate
A suite is a set of cases in YAML or JSON lines, each scored by exact, contains, regex, a number within a tolerance, a JSON Schema subset, or a judge model with a rubric; weighed into one score against a threshold. internal/bench/eval runs one on the control plane through Router.Replay, as the gateway's own requests, and keeps every run and its cases in the store (migration 0028: eval_runs, eval_cases). - llmgw ctl eval MODEL --suite FILE|NAME, --history, --run, --suites; exits non-zero below the score. Suites by name from control.eval_dir, or the two built in from deploy/eval (arithmetic, json-format). - model update and promote take --eval SUITE --min-score N, or a model's standing gate (ctl model gate, eval: in the catalog): a promote asks the canary first and leaves it below the score; an update tries the new weights as a canary given no share, then promotes or removes it. - The console's model menu: ask a suite with its progress, its runs, its gate. - The audit middleware no longer hands a handler a body cut off at 64 KiB. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
30f7f3474a
commit
f0477f140d
49 files changed
+4563
-46
No files matched your search
@@ -32,6 +32,7 @@ import (
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/audit"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/bench/eval"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
@@ -286,6 +287,7 @@ func newServerCmd() *cobra.Command {
|
||||
var (
|
||||
adminAPI *adminapi.API
|
||||
catalogSvc *modelsvc.Service
|
||||
evals *eval.Service
|
||||
)
|
||||
if admin != nil {
|
||||
api := adminapi.New(admin)
|
||||
@@ -303,6 +305,9 @@ func newServerCmd() *cobra.Command {
|
||||
api.SetReleases(cfg.Releases)
|
||||
api.SetNotifier(notifier)
|
||||
api.SetReplayer(rt.Replay)
|
||||
// Quality suites, asked of a model the way a replay is.
|
||||
evals = newEvalService(cfg, rt, admin, log)
|
||||
api.SetEval(evals)
|
||||
if st := rt.Store(); st != nil && gate != nil {
|
||||
api.SetGuard(&adminapi.GuardAdmin{Library: cfg.Guard, Store: st, Invalidate: gate.Invalidate})
|
||||
}
|
||||
@@ -418,6 +423,8 @@ func newServerCmd() *cobra.Command {
|
||||
Pool: adminAPI.PoolNodes, Notify: notifier,
|
||||
// Canaries: how the router saw each side serve.
|
||||
CanaryStats: rt.CanaryStats, ResetCanary: rt.ResetCanary,
|
||||
// Updates and promotions held to a quality suite.
|
||||
Eval: evals,
|
||||
})
|
||||
go serve.WatchCanaries(ctx, 30*time.Second)
|
||||
adminAPI.SetServe(serve)
|
||||
|
||||
Reference in new issue
Block a user