Canary on the console's model menu; docs

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-04 20:33:59 +02:00
1 parent a1dfd4df1d
commit 889be70b8f
8 files changed
+209 -1

No files matched your search

+33
View File
@@ -879,6 +879,38 @@ prices and budgets, rate limits, usage records, and key allowlists.
whose allowlist leaves the external model out cannot reach it, as a fallback or
by name. The cluster page and `ctl models` mark it `external`.
### Trying new weights on some of the traffic first
When a model's repository publishes new weights, `llmgw ctl model update`
swaps them in. To see how they serve first, run them as a canary:
```
llmgw ctl model canary qwen-7b --percent 10 # fetched, started beside, given 10%
llmgw ctl model canaries # how each side has served
llmgw ctl model canary qwen-7b --percent 50 # more
llmgw ctl model promote qwen-7b # the model moves to them
llmgw ctl model canary qwen-7b --abort # or not
```
- **What runs:** the new files are fetched into the mirror and checked as
for an update, then run as `qwen-7b-canary` — the same model in everything
but its weights, one replica. Clients cannot name it; it gets its share
of `qwen-7b`'s requests.
- **Who goes where:** a session (`x-session-id`, or the `user` field)
stays on one side, so a conversation does not change weights halfway.
`x-llmgw-variant: canary` or `stable` pins a request to one side; the
response says which served it in the same header, and the usage record
names `qwen-7b-canary` as the model that served it.
- **Rolled back by itself:** once the canary has served `--min-requests`
(50), it is taken out if its error rate is `--max-error-increase` (0.05,
five points) above the model's, or its mean time to the first token — or
to the answer, unstreamed — is `--max-ttft-ratio` (1.5) times the
model's. `canary_rolled_back` goes to the notification webhooks. The
comparison is the router's own count since the canary began, kept per
control plane.
- **Promote** is the update the canary was a trial of: the model's replicas
are replaced one at a time, and `llmgw ctl model rollback` goes back.
### How many replicas, and how much each one carries
Replica count is changed while the cluster runs, without editing the
@@ -981,6 +1013,7 @@ control plane sends those itself:
| `node_lost` | A node was silent past `control.lost_after`; its replicas are rebuilt elsewhere. |
| `budget_spent` | A key, project, team or organization reached its budget; priced models are refused. |
| `update_available` | A model's registry repository published different weights. Once per revision. |
| `canary_rolled_back` | A model's canary served worse than the model and was taken out. |
| `judge_unavailable` | A guardrails judge gave no verdict: with `fail: closed` requests are being refused, with `open` served unjudged. |
```yaml