Canary on the console's model menu; docs
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
a1dfd4df1d
commit
889be70b8f
8 files changed
+209
-1
No files matched your search
@@ -879,6 +879,38 @@ prices and budgets, rate limits, usage records, and key allowlists.
|
||||
whose allowlist leaves the external model out cannot reach it, as a fallback or
|
||||
by name. The cluster page and `ctl models` mark it `external`.
|
||||
|
||||
### Trying new weights on some of the traffic first
|
||||
|
||||
When a model's repository publishes new weights, `llmgw ctl model update`
|
||||
swaps them in. To see how they serve first, run them as a canary:
|
||||
|
||||
```
|
||||
llmgw ctl model canary qwen-7b --percent 10 # fetched, started beside, given 10%
|
||||
llmgw ctl model canaries # how each side has served
|
||||
llmgw ctl model canary qwen-7b --percent 50 # more
|
||||
llmgw ctl model promote qwen-7b # the model moves to them
|
||||
llmgw ctl model canary qwen-7b --abort # or not
|
||||
```
|
||||
|
||||
- **What runs:** the new files are fetched into the mirror and checked as
|
||||
for an update, then run as `qwen-7b-canary` — the same model in everything
|
||||
but its weights, one replica. Clients cannot name it; it gets its share
|
||||
of `qwen-7b`'s requests.
|
||||
- **Who goes where:** a session (`x-session-id`, or the `user` field)
|
||||
stays on one side, so a conversation does not change weights halfway.
|
||||
`x-llmgw-variant: canary` or `stable` pins a request to one side; the
|
||||
response says which served it in the same header, and the usage record
|
||||
names `qwen-7b-canary` as the model that served it.
|
||||
- **Rolled back by itself:** once the canary has served `--min-requests`
|
||||
(50), it is taken out if its error rate is `--max-error-increase` (0.05,
|
||||
five points) above the model's, or its mean time to the first token — or
|
||||
to the answer, unstreamed — is `--max-ttft-ratio` (1.5) times the
|
||||
model's. `canary_rolled_back` goes to the notification webhooks. The
|
||||
comparison is the router's own count since the canary began, kept per
|
||||
control plane.
|
||||
- **Promote** is the update the canary was a trial of: the model's replicas
|
||||
are replaced one at a time, and `llmgw ctl model rollback` goes back.
|
||||
|
||||
### How many replicas, and how much each one carries
|
||||
|
||||
Replica count is changed while the cluster runs, without editing the
|
||||
@@ -981,6 +1013,7 @@ control plane sends those itself:
|
||||
| `node_lost` | A node was silent past `control.lost_after`; its replicas are rebuilt elsewhere. |
|
||||
| `budget_spent` | A key, project, team or organization reached its budget; priced models are refused. |
|
||||
| `update_available` | A model's registry repository published different weights. Once per revision. |
|
||||
| `canary_rolled_back` | A model's canary served worse than the model and was taken out. |
|
||||
| `judge_unavailable` | A guardrails judge gave no verdict: with `fail: closed` requests are being refused, with `open` served unjudged. |
|
||||
|
||||
```yaml
|
||||
|
||||
Reference in new issue
Block a user