Canary on the console's model menu; docs

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-04 20:33:59 +02:00
1 parent a1dfd4df1d
commit 889be70b8f
8 files changed
+209 -1

No files matched your search

+7
View File
@@ -110,6 +110,13 @@ and **UPDATE** when the registry has new weights for it. On a row:
**Go back to …** returns to the weights the update replaced.
**Ask the registry for new weights now** checks without waiting for the
next scheduled look (`mirror.update_check`, six hours by default).
- **Try the new weights as a canary…** — instead of swapping them in, runs
them beside the model as `MODEL-canary`, one replica, given the share of
its requests you choose. With a canary running the menu has **Canary's
share…**, **Promote the canary…** (the model moves to its weights, the
canary is removed) and **Take the canary out…**. A canary that errs or
answers more slowly than the model is taken out by itself. The MODEL
column shows `(10% to canary)` and `(canary of …)`.
- **Why is it placed where it is** — `llmgw ctl explain-placement`.
- **Try again**, for a model parked after failing every start.
- **Remove** — its replicas stop and it leaves the catalog. A model from
+33
View File
@@ -879,6 +879,38 @@ prices and budgets, rate limits, usage records, and key allowlists.
whose allowlist leaves the external model out cannot reach it, as a fallback or
by name. The cluster page and `ctl models` mark it `external`.
### Trying new weights on some of the traffic first
When a model's repository publishes new weights, `llmgw ctl model update`
swaps them in. To see how they serve first, run them as a canary:
```
llmgw ctl model canary qwen-7b --percent 10 # fetched, started beside, given 10%
llmgw ctl model canaries # how each side has served
llmgw ctl model canary qwen-7b --percent 50 # more
llmgw ctl model promote qwen-7b # the model moves to them
llmgw ctl model canary qwen-7b --abort # or not
```
- **What runs:** the new files are fetched into the mirror and checked as
for an update, then run as `qwen-7b-canary` — the same model in everything
but its weights, one replica. Clients cannot name it; it gets its share
of `qwen-7b`'s requests.
- **Who goes where:** a session (`x-session-id`, or the `user` field)
stays on one side, so a conversation does not change weights halfway.
`x-llmgw-variant: canary` or `stable` pins a request to one side; the
response says which served it in the same header, and the usage record
names `qwen-7b-canary` as the model that served it.
- **Rolled back by itself:** once the canary has served `--min-requests`
(50), it is taken out if its error rate is `--max-error-increase` (0.05,
five points) above the model's, or its mean time to the first token — or
to the answer, unstreamed — is `--max-ttft-ratio` (1.5) times the
model's. `canary_rolled_back` goes to the notification webhooks. The
comparison is the router's own count since the canary began, kept per
control plane.
- **Promote** is the update the canary was a trial of: the model's replicas
are replaced one at a time, and `llmgw ctl model rollback` goes back.
### How many replicas, and how much each one carries
Replica count is changed while the cluster runs, without editing the
@@ -981,6 +1013,7 @@ control plane sends those itself:
| `node_lost` | A node was silent past `control.lost_after`; its replicas are rebuilt elsewhere. |
| `budget_spent` | A key, project, team or organization reached its budget; priced models are refused. |
| `update_available` | A model's registry repository published different weights. Once per revision. |
| `canary_rolled_back` | A model's canary served worse than the model and was taken out. |
| `judge_unavailable` | A guardrails judge gave no verdict: with `fail: closed` requests are being refused, with `open` served unjudged. |
```yaml
+48
View File
@@ -43,6 +43,11 @@ rather than repeated.
| `batch_size` | `int` | [Auth](#auth) |
| `buffer_size` | `int` | [Auth](#auth) |
| `usage_retention` | `time.Duration` | [Auth](#auth) |
| `model` | `string` | [Canary](#canary) |
| `percent` | `int` | [Canary](#canary) |
| `min_requests` | `int` | [Canary](#canary) |
| `max_error_increase` | `float64` | [Canary](#canary) |
| `max_ttft_ratio` | `float64` | [Canary](#canary) |
| `tools` | `bool` | [Capabilities](#capabilities) |
| `vision` | `bool` | [Capabilities](#capabilities) |
| `embeddings` | `bool` | [Capabilities](#capabilities) |
@@ -98,6 +103,7 @@ rather than repeated.
| `external` | `*External` | [Model](#model) |
| `adapters` | `[]Adapter` | [Model](#model) |
| `fallbacks` | `[]string` | [Model](#model) |
| `canary` | `*Canary` | [Model](#model) |
| `routing_profile` | `string` | [Model](#model) |
| `slo` | `SLO` | [Model](#model) |
| `context_length` | `int` | [Model](#model) |
@@ -511,6 +517,38 @@ five-minute sums kept beside the records, which are not pruned, so this
bounds the table that grows by a row a request without changing any
figure a page shows. At least a day when set.
## Canary
### `model`
`string`
Model is the candidate: a catalog model, usually this one at newer
weights, which clients cannot name themselves.
### `percent`
`int`
Percent of requests go to it, 0 to 100.
### `min_requests`
`int`
The candidate is taken out — rolled back — when, over at least
MinRequests of its own, its error rate is MaxErrorIncrease above
this model's (0.05 is five points), or its mean time to the first
token is MaxTTFTRatio times this model's. Zero is 50, 0.05 and 1.5.
### `max_error_increase`
`float64`
### `max_ttft_ratio`
`float64`
## Capabilities
### `tools`
@@ -916,6 +954,16 @@ every adapter, applying one only to a request that names it.
`[]string`
### `canary`
`*Canary`
Canary sends a share of this model's requests to another catalog
model — the same model at new weights — to compare the two before
moving to it (`llmgw ctl model canary`).
Takes a [Canary](#canary) block.
### `routing_profile`
`string`