Canary on the console's model menu; docs
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
a1dfd4df1d
commit
889be70b8f
8 files changed
+209
-1
No files matched your search
@@ -43,6 +43,11 @@ rather than repeated.
|
||||
| `batch_size` | `int` | [Auth](#auth) |
|
||||
| `buffer_size` | `int` | [Auth](#auth) |
|
||||
| `usage_retention` | `time.Duration` | [Auth](#auth) |
|
||||
| `model` | `string` | [Canary](#canary) |
|
||||
| `percent` | `int` | [Canary](#canary) |
|
||||
| `min_requests` | `int` | [Canary](#canary) |
|
||||
| `max_error_increase` | `float64` | [Canary](#canary) |
|
||||
| `max_ttft_ratio` | `float64` | [Canary](#canary) |
|
||||
| `tools` | `bool` | [Capabilities](#capabilities) |
|
||||
| `vision` | `bool` | [Capabilities](#capabilities) |
|
||||
| `embeddings` | `bool` | [Capabilities](#capabilities) |
|
||||
@@ -98,6 +103,7 @@ rather than repeated.
|
||||
| `external` | `*External` | [Model](#model) |
|
||||
| `adapters` | `[]Adapter` | [Model](#model) |
|
||||
| `fallbacks` | `[]string` | [Model](#model) |
|
||||
| `canary` | `*Canary` | [Model](#model) |
|
||||
| `routing_profile` | `string` | [Model](#model) |
|
||||
| `slo` | `SLO` | [Model](#model) |
|
||||
| `context_length` | `int` | [Model](#model) |
|
||||
@@ -511,6 +517,38 @@ five-minute sums kept beside the records, which are not pruned, so this
|
||||
bounds the table that grows by a row a request without changing any
|
||||
figure a page shows. At least a day when set.
|
||||
|
||||
## Canary
|
||||
|
||||
### `model`
|
||||
|
||||
`string`
|
||||
|
||||
Model is the candidate: a catalog model, usually this one at newer
|
||||
weights, which clients cannot name themselves.
|
||||
|
||||
### `percent`
|
||||
|
||||
`int`
|
||||
|
||||
Percent of requests go to it, 0 to 100.
|
||||
|
||||
### `min_requests`
|
||||
|
||||
`int`
|
||||
|
||||
The candidate is taken out — rolled back — when, over at least
|
||||
MinRequests of its own, its error rate is MaxErrorIncrease above
|
||||
this model's (0.05 is five points), or its mean time to the first
|
||||
token is MaxTTFTRatio times this model's. Zero is 50, 0.05 and 1.5.
|
||||
|
||||
### `max_error_increase`
|
||||
|
||||
`float64`
|
||||
|
||||
### `max_ttft_ratio`
|
||||
|
||||
`float64`
|
||||
|
||||
## Capabilities
|
||||
|
||||
### `tools`
|
||||
@@ -916,6 +954,16 @@ every adapter, applying one only to a request that names it.
|
||||
|
||||
`[]string`
|
||||
|
||||
### `canary`
|
||||
|
||||
`*Canary`
|
||||
|
||||
Canary sends a share of this model's requests to another catalog
|
||||
model — the same model at new weights — to compare the two before
|
||||
moving to it (`llmgw ctl model canary`).
|
||||
|
||||
Takes a [Canary](#canary) block.
|
||||
|
||||
### `routing_profile`
|
||||
|
||||
`string`
|
||||
|
||||
Reference in new issue
Block a user