Quality suites, and new weights held to one: ctl eval, model update/promote --eval, model gate
A suite is a set of cases in YAML or JSON lines, each scored by exact, contains, regex, a number within a tolerance, a JSON Schema subset, or a judge model with a rubric; weighed into one score against a threshold. internal/bench/eval runs one on the control plane through Router.Replay, as the gateway's own requests, and keeps every run and its cases in the store (migration 0028: eval_runs, eval_cases). - llmgw ctl eval MODEL --suite FILE|NAME, --history, --run, --suites; exits non-zero below the score. Suites by name from control.eval_dir, or the two built in from deploy/eval (arithmetic, json-format). - model update and promote take --eval SUITE --min-score N, or a model's standing gate (ctl model gate, eval: in the catalog): a promote asks the canary first and leaves it below the score; an update tries the new weights as a canary given no share, then promotes or removes it. - The console's model menu: ask a suite with its progress, its runs, its gate. - The audit middleware no longer hands a handler a body cut off at 64 KiB. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
30f7f3474a
commit
f0477f140d
49 files changed
+4563
-46
No files matched your search
@@ -0,0 +1,58 @@
|
||||
# Arithmetic a small instruction model gets right: sums, products and a
|
||||
# little reading, each answer scored by the last number in it. A quantisation
|
||||
# that has gone wrong, or a chat template that no longer matches the
|
||||
# weights, shows up here as a falling score long before anyone notices it in
|
||||
# conversation.
|
||||
#
|
||||
# llmgw ctl eval MODEL --suite arithmetic
|
||||
# llmgw ctl model update MODEL --eval arithmetic --min-score 0.8
|
||||
#
|
||||
# Copy it into control.eval_dir (/etc/llmgw/eval) to change it; a suite
|
||||
# there of the same name replaces this one.
|
||||
name: arithmetic
|
||||
description: Sums, products and word problems, each scored by the number in its answer.
|
||||
threshold: 0.7
|
||||
system: You are a careful calculator. Answer with the number only, without words or units.
|
||||
max_tokens: 64
|
||||
cases:
|
||||
- id: add-small
|
||||
prompt: What is 17 + 25?
|
||||
number: 42
|
||||
- id: add-carry
|
||||
prompt: What is 389 + 476?
|
||||
number: 865
|
||||
- id: subtract
|
||||
prompt: What is 1000 - 357?
|
||||
number: 643
|
||||
- id: subtract-negative
|
||||
prompt: What is 12 - 30?
|
||||
number: -18
|
||||
- id: multiply
|
||||
prompt: What is 12 * 12?
|
||||
number: 144
|
||||
- id: multiply-larger
|
||||
prompt: What is 23 * 17?
|
||||
number: 391
|
||||
weight: 2
|
||||
- id: divide
|
||||
prompt: What is 144 / 6?
|
||||
number: 24
|
||||
- id: decimal
|
||||
prompt: What is 2.5 * 4?
|
||||
number: 10
|
||||
- id: percent
|
||||
prompt: What is 15% of 200?
|
||||
number: 30
|
||||
- id: words-apples
|
||||
prompt: Ana has 7 apples and buys 5 more, then gives 3 away. How many apples does she have?
|
||||
number: 9
|
||||
weight: 2
|
||||
- id: words-time
|
||||
prompt: A train leaves at 14:00 and the trip takes 150 minutes. How many hours does the trip take?
|
||||
number: 2.5
|
||||
tolerance: 0.01
|
||||
weight: 2
|
||||
- id: order-of-operations
|
||||
prompt: What is 2 + 3 * 4?
|
||||
number: 14
|
||||
weight: 2
|
||||
@@ -0,0 +1,94 @@
|
||||
# Following a format: asked for JSON of a given shape, does the model give
|
||||
# it — the thing an application parsing its answers depends on, and the
|
||||
# first to go when a chat template drifts. Each answer is read as JSON (a
|
||||
# code fence or a sentence around it is forgiven) and held to a schema.
|
||||
#
|
||||
# llmgw ctl eval MODEL --suite json-format
|
||||
#
|
||||
# Copy it into control.eval_dir (/etc/llmgw/eval) to change it; a suite
|
||||
# there of the same name replaces this one.
|
||||
name: json-format
|
||||
description: Answers asked for as JSON of a given shape, held to a schema.
|
||||
threshold: 0.7
|
||||
system: You answer with a single JSON value and nothing else.
|
||||
max_tokens: 200
|
||||
cases:
|
||||
- id: person
|
||||
prompt: >-
|
||||
Give a JSON object for a person named Ada Lovelace, born in 1815, with the
|
||||
keys "name" (a string) and "born" (a number).
|
||||
json_schema:
|
||||
type: object
|
||||
required: [name, born]
|
||||
properties:
|
||||
name: {type: string, pattern: "Ada"}
|
||||
born: {type: integer, minimum: 1815, maximum: 1815}
|
||||
|
||||
- id: list-of-colours
|
||||
prompt: Give a JSON array of exactly three colour names, as strings.
|
||||
json_schema:
|
||||
type: array
|
||||
minItems: 3
|
||||
maxItems: 3
|
||||
items: {type: string, minLength: 1}
|
||||
|
||||
- id: boolean-answer
|
||||
prompt: >-
|
||||
Is water wet? Answer as a JSON object with one key, "answer", whose
|
||||
value is true or false.
|
||||
json_schema:
|
||||
type: object
|
||||
required: [answer]
|
||||
properties:
|
||||
answer: {type: boolean}
|
||||
|
||||
- id: nested
|
||||
prompt: >-
|
||||
Give a JSON object describing a book: "title" (a string), "author" (an
|
||||
object with "first" and "last", both strings) and "pages" (a whole number
|
||||
above zero). Use any book.
|
||||
json_schema:
|
||||
type: object
|
||||
required: [title, author, pages]
|
||||
properties:
|
||||
title: {type: string, minLength: 1}
|
||||
author:
|
||||
type: object
|
||||
required: [first, last]
|
||||
properties:
|
||||
first: {type: string}
|
||||
last: {type: string}
|
||||
pages: {type: integer, exclusiveMinimum: 0}
|
||||
weight: 2
|
||||
|
||||
- id: enum-choice
|
||||
prompt: >-
|
||||
Classify the sentence "I love this product!" as positive, negative or
|
||||
neutral. Answer as a JSON object {"sentiment": "..."} using one of those
|
||||
three words in lower case.
|
||||
json_schema:
|
||||
type: object
|
||||
required: [sentiment]
|
||||
properties:
|
||||
sentiment: {enum: [positive]}
|
||||
weight: 2
|
||||
|
||||
- id: numbers
|
||||
prompt: 'Give a JSON object {"sum": N} where N is the sum of 4, 5 and 6.'
|
||||
json_schema:
|
||||
type: object
|
||||
required: [sum]
|
||||
properties:
|
||||
sum: {const: 15}
|
||||
|
||||
- id: no-extra-keys
|
||||
prompt: >-
|
||||
Give a JSON object with exactly two keys, "city" and "country", for
|
||||
Paris. No other keys.
|
||||
json_schema:
|
||||
type: object
|
||||
required: [city, country]
|
||||
additionalProperties: false
|
||||
properties:
|
||||
city: {type: string}
|
||||
country: {type: string}
|
||||
Reference in new issue
Block a user