Ilker ManapandClaude Opus 5.5
f0477f140d
A suite is a set of cases in YAML or JSON lines, each scored by exact, contains, regex, a number within a tolerance, a JSON Schema subset, or a judge model with a rubric; weighed into one score against a threshold. internal/bench/eval runs one on the control plane through Router.Replay, as the gateway's own requests, and keeps every run and its cases in the store (migration 0028: eval_runs, eval_cases). - llmgw ctl eval MODEL --suite FILE|NAME, --history, --run, --suites; exits non-zero below the score. Suites by name from control.eval_dir, or the two built in from deploy/eval (arithmetic, json-format). - model update and promote take --eval SUITE --min-score N, or a model's standing gate (ctl model gate, eval: in the catalog): a promote asks the canary first and leaves it below the score; an update tries the new weights as a canary given no share, then promotes or removes it. - The console's model menu: ask a suite with its progress, its runs, its gate. - The audit middleware no longer hands a handler a body cut off at 64 KiB. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
44 lines
1.6 KiB
SQL
44 lines
1.6 KiB
SQL
-- Ported from the SQLite migration of the same number.
|
|
-- Quality suites run against a model (`llmgw ctl eval`, and the gate on
|
|
-- `model update` and `model promote`): one row a run, filed under the model
|
|
-- it is about, with the revision its weights came from when that is known,
|
|
-- and the suite's hash, so two runs compare only when they asked the same
|
|
-- questions. Each case's score and answer beside it.
|
|
CREATE TABLE eval_runs (
|
|
id TEXT PRIMARY KEY,
|
|
model TEXT NOT NULL,
|
|
target TEXT NOT NULL DEFAULT '',
|
|
repo TEXT NOT NULL DEFAULT '',
|
|
revision TEXT NOT NULL DEFAULT '',
|
|
suite TEXT NOT NULL,
|
|
suite_hash TEXT NOT NULL,
|
|
started_at BIGINT NOT NULL,
|
|
took_ms BIGINT NOT NULL DEFAULT 0,
|
|
score DOUBLE PRECISION NOT NULL,
|
|
threshold DOUBLE PRECISION NOT NULL,
|
|
passed BIGINT NOT NULL,
|
|
cases BIGINT NOT NULL,
|
|
below BIGINT NOT NULL DEFAULT 0,
|
|
errors BIGINT NOT NULL DEFAULT 0,
|
|
judge TEXT NOT NULL DEFAULT '',
|
|
actor TEXT NOT NULL DEFAULT '',
|
|
job TEXT NOT NULL DEFAULT '',
|
|
error TEXT NOT NULL DEFAULT ''
|
|
);
|
|
CREATE INDEX eval_runs_model ON eval_runs (model, started_at);
|
|
|
|
CREATE TABLE eval_cases (
|
|
run_id TEXT NOT NULL REFERENCES eval_runs (id) ON DELETE CASCADE,
|
|
idx BIGINT NOT NULL,
|
|
case_id TEXT NOT NULL,
|
|
kind TEXT NOT NULL,
|
|
weight DOUBLE PRECISION NOT NULL,
|
|
score DOUBLE PRECISION NOT NULL,
|
|
passed BIGINT NOT NULL,
|
|
answer TEXT NOT NULL DEFAULT '',
|
|
detail TEXT NOT NULL DEFAULT '',
|
|
error TEXT NOT NULL DEFAULT '',
|
|
took_ms BIGINT NOT NULL DEFAULT 0,
|
|
PRIMARY KEY (run_id, idx)
|
|
);
|