Canary on the console's model menu; docs
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
a1dfd4df1d
commit
889be70b8f
8 files changed
+209
-1
No files matched your search
@@ -110,6 +110,13 @@ and **UPDATE** when the registry has new weights for it. On a row:
|
||||
**Go back to …** returns to the weights the update replaced.
|
||||
**Ask the registry for new weights now** checks without waiting for the
|
||||
next scheduled look (`mirror.update_check`, six hours by default).
|
||||
- **Try the new weights as a canary…** — instead of swapping them in, runs
|
||||
them beside the model as `MODEL-canary`, one replica, given the share of
|
||||
its requests you choose. With a canary running the menu has **Canary's
|
||||
share…**, **Promote the canary…** (the model moves to its weights, the
|
||||
canary is removed) and **Take the canary out…**. A canary that errs or
|
||||
answers more slowly than the model is taken out by itself. The MODEL
|
||||
column shows `(10% to canary)` and `(canary of …)`.
|
||||
- **Why is it placed where it is** — `llmgw ctl explain-placement`.
|
||||
- **Try again**, for a model parked after failing every start.
|
||||
- **Remove** — its replicas stop and it leaves the catalog. A model from
|
||||
|
||||
@@ -879,6 +879,38 @@ prices and budgets, rate limits, usage records, and key allowlists.
|
||||
whose allowlist leaves the external model out cannot reach it, as a fallback or
|
||||
by name. The cluster page and `ctl models` mark it `external`.
|
||||
|
||||
### Trying new weights on some of the traffic first
|
||||
|
||||
When a model's repository publishes new weights, `llmgw ctl model update`
|
||||
swaps them in. To see how they serve first, run them as a canary:
|
||||
|
||||
```
|
||||
llmgw ctl model canary qwen-7b --percent 10 # fetched, started beside, given 10%
|
||||
llmgw ctl model canaries # how each side has served
|
||||
llmgw ctl model canary qwen-7b --percent 50 # more
|
||||
llmgw ctl model promote qwen-7b # the model moves to them
|
||||
llmgw ctl model canary qwen-7b --abort # or not
|
||||
```
|
||||
|
||||
- **What runs:** the new files are fetched into the mirror and checked as
|
||||
for an update, then run as `qwen-7b-canary` — the same model in everything
|
||||
but its weights, one replica. Clients cannot name it; it gets its share
|
||||
of `qwen-7b`'s requests.
|
||||
- **Who goes where:** a session (`x-session-id`, or the `user` field)
|
||||
stays on one side, so a conversation does not change weights halfway.
|
||||
`x-llmgw-variant: canary` or `stable` pins a request to one side; the
|
||||
response says which served it in the same header, and the usage record
|
||||
names `qwen-7b-canary` as the model that served it.
|
||||
- **Rolled back by itself:** once the canary has served `--min-requests`
|
||||
(50), it is taken out if its error rate is `--max-error-increase` (0.05,
|
||||
five points) above the model's, or its mean time to the first token — or
|
||||
to the answer, unstreamed — is `--max-ttft-ratio` (1.5) times the
|
||||
model's. `canary_rolled_back` goes to the notification webhooks. The
|
||||
comparison is the router's own count since the canary began, kept per
|
||||
control plane.
|
||||
- **Promote** is the update the canary was a trial of: the model's replicas
|
||||
are replaced one at a time, and `llmgw ctl model rollback` goes back.
|
||||
|
||||
### How many replicas, and how much each one carries
|
||||
|
||||
Replica count is changed while the cluster runs, without editing the
|
||||
@@ -981,6 +1013,7 @@ control plane sends those itself:
|
||||
| `node_lost` | A node was silent past `control.lost_after`; its replicas are rebuilt elsewhere. |
|
||||
| `budget_spent` | A key, project, team or organization reached its budget; priced models are refused. |
|
||||
| `update_available` | A model's registry repository published different weights. Once per revision. |
|
||||
| `canary_rolled_back` | A model's canary served worse than the model and was taken out. |
|
||||
| `judge_unavailable` | A guardrails judge gave no verdict: with `fail: closed` requests are being refused, with `open` served unjudged. |
|
||||
|
||||
```yaml
|
||||
|
||||
@@ -43,6 +43,11 @@ rather than repeated.
|
||||
| `batch_size` | `int` | [Auth](#auth) |
|
||||
| `buffer_size` | `int` | [Auth](#auth) |
|
||||
| `usage_retention` | `time.Duration` | [Auth](#auth) |
|
||||
| `model` | `string` | [Canary](#canary) |
|
||||
| `percent` | `int` | [Canary](#canary) |
|
||||
| `min_requests` | `int` | [Canary](#canary) |
|
||||
| `max_error_increase` | `float64` | [Canary](#canary) |
|
||||
| `max_ttft_ratio` | `float64` | [Canary](#canary) |
|
||||
| `tools` | `bool` | [Capabilities](#capabilities) |
|
||||
| `vision` | `bool` | [Capabilities](#capabilities) |
|
||||
| `embeddings` | `bool` | [Capabilities](#capabilities) |
|
||||
@@ -98,6 +103,7 @@ rather than repeated.
|
||||
| `external` | `*External` | [Model](#model) |
|
||||
| `adapters` | `[]Adapter` | [Model](#model) |
|
||||
| `fallbacks` | `[]string` | [Model](#model) |
|
||||
| `canary` | `*Canary` | [Model](#model) |
|
||||
| `routing_profile` | `string` | [Model](#model) |
|
||||
| `slo` | `SLO` | [Model](#model) |
|
||||
| `context_length` | `int` | [Model](#model) |
|
||||
@@ -511,6 +517,38 @@ five-minute sums kept beside the records, which are not pruned, so this
|
||||
bounds the table that grows by a row a request without changing any
|
||||
figure a page shows. At least a day when set.
|
||||
|
||||
## Canary
|
||||
|
||||
### `model`
|
||||
|
||||
`string`
|
||||
|
||||
Model is the candidate: a catalog model, usually this one at newer
|
||||
weights, which clients cannot name themselves.
|
||||
|
||||
### `percent`
|
||||
|
||||
`int`
|
||||
|
||||
Percent of requests go to it, 0 to 100.
|
||||
|
||||
### `min_requests`
|
||||
|
||||
`int`
|
||||
|
||||
The candidate is taken out — rolled back — when, over at least
|
||||
MinRequests of its own, its error rate is MaxErrorIncrease above
|
||||
this model's (0.05 is five points), or its mean time to the first
|
||||
token is MaxTTFTRatio times this model's. Zero is 50, 0.05 and 1.5.
|
||||
|
||||
### `max_error_increase`
|
||||
|
||||
`float64`
|
||||
|
||||
### `max_ttft_ratio`
|
||||
|
||||
`float64`
|
||||
|
||||
## Capabilities
|
||||
|
||||
### `tools`
|
||||
@@ -916,6 +954,16 @@ every adapter, applying one only to a request that names it.
|
||||
|
||||
`[]string`
|
||||
|
||||
### `canary`
|
||||
|
||||
`*Canary`
|
||||
|
||||
Canary sends a share of this model's requests to another catalog
|
||||
model — the same model at new weights — to compare the two before
|
||||
moving to it (`llmgw ctl model canary`).
|
||||
|
||||
Takes a [Canary](#canary) block.
|
||||
|
||||
### `routing_profile`
|
||||
|
||||
`string`
|
||||
|
||||
Reference in new issue
Block a user