Canary on the console's model menu; docs

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-04 20:33:59 +02:00
1 parent a1dfd4df1d
commit 889be70b8f
8 files changed
+209 -1

No files matched your search

+7 -1
View File
@@ -420,7 +420,13 @@ portal is not done — the page shows; the console and ctl change.
**Request metadata and tags (S).** Accept `metadata`/`user` from the
body or `x-llmgw-tags`; store on the usage row; tenant metric labels
behind an allowlist. `router/handlers.go` parse, `accounting/recorder.go`.
7. **Canary and weighted A/B between model revisions (M).**
7. **Done (branch `features-c2`):** `ctl model canary/promote/canaries`,
session-sticky split with `x-llmgw-variant` pins, rollback on error rate
or TTFT, `canary_rolled_back` notification, console menu. Left: A/B
between two different models (the split is between a model and its
newer weights); the comparison is per control plane and lives in memory,
so a restart begins it again; a key-level pin.
**Canary and weighted A/B between model revisions (M).**
`control/modelserve/updates.go` `replace` swaps every replica; add
weighted routing across two revisions/aliases (scheduler filter, alias
resolution in `router/router.go:569`), a header/key pin, automatic
+17
View File
@@ -71,6 +71,23 @@ APIs change between minors without a deprecation period.
- **Other ways in:** `GET /api/v1/usage/export` (CSV, or `format=json`)
and **E** on the console's Usage view.
### New weights tried on a share of the traffic first
- **Canaries.** `llmgw ctl model canary MODEL --percent 10` fetches the newer
weights of a model's repository and runs them beside it as `MODEL-canary`,
given that share of its requests.
- **Routing:** a session stays on one side; `x-llmgw-variant` pins a
request to `canary` or `stable` and says which served it. Clients cannot
name the canary.
- **Rolled back by itself** when, over 50 of its requests or more, its
error rate is five points above the model's or its time to the first
token half again as long; `canary_rolled_back` is sent to the
notification webhooks.
- **Then:** `--percent` changes its share, `llmgw ctl model promote`
moves the model to it, `--abort` takes it out, and `llmgw ctl model
canaries` compares the two sides. All of it is also on the console's
model menu.
### Requests can be tagged
- **Tags and end users on every usage record.** OpenAI's `metadata` and the
+7
View File
@@ -110,6 +110,13 @@ and **UPDATE** when the registry has new weights for it. On a row:
**Go back to …** returns to the weights the update replaced.
**Ask the registry for new weights now** checks without waiting for the
next scheduled look (`mirror.update_check`, six hours by default).
- **Try the new weights as a canary…** — instead of swapping them in, runs
them beside the model as `MODEL-canary`, one replica, given the share of
its requests you choose. With a canary running the menu has **Canary's
share…**, **Promote the canary…** (the model moves to its weights, the
canary is removed) and **Take the canary out…**. A canary that errs or
answers more slowly than the model is taken out by itself. The MODEL
column shows `(10% to canary)` and `(canary of …)`.
- **Why is it placed where it is** — `llmgw ctl explain-placement`.
- **Try again**, for a model parked after failing every start.
- **Remove** — its replicas stop and it leaves the catalog. A model from
+33
View File
@@ -879,6 +879,38 @@ prices and budgets, rate limits, usage records, and key allowlists.
whose allowlist leaves the external model out cannot reach it, as a fallback or
by name. The cluster page and `ctl models` mark it `external`.
### Trying new weights on some of the traffic first
When a model's repository publishes new weights, `llmgw ctl model update`
swaps them in. To see how they serve first, run them as a canary:
```
llmgw ctl model canary qwen-7b --percent 10 # fetched, started beside, given 10%
llmgw ctl model canaries # how each side has served
llmgw ctl model canary qwen-7b --percent 50 # more
llmgw ctl model promote qwen-7b # the model moves to them
llmgw ctl model canary qwen-7b --abort # or not
```
- **What runs:** the new files are fetched into the mirror and checked as
for an update, then run as `qwen-7b-canary` — the same model in everything
but its weights, one replica. Clients cannot name it; it gets its share
of `qwen-7b`'s requests.
- **Who goes where:** a session (`x-session-id`, or the `user` field)
stays on one side, so a conversation does not change weights halfway.
`x-llmgw-variant: canary` or `stable` pins a request to one side; the
response says which served it in the same header, and the usage record
names `qwen-7b-canary` as the model that served it.
- **Rolled back by itself:** once the canary has served `--min-requests`
(50), it is taken out if its error rate is `--max-error-increase` (0.05,
five points) above the model's, or its mean time to the first token — or
to the answer, unstreamed — is `--max-ttft-ratio` (1.5) times the
model's. `canary_rolled_back` goes to the notification webhooks. The
comparison is the router's own count since the canary began, kept per
control plane.
- **Promote** is the update the canary was a trial of: the model's replicas
are replaced one at a time, and `llmgw ctl model rollback` goes back.
### How many replicas, and how much each one carries
Replica count is changed while the cluster runs, without editing the
@@ -981,6 +1013,7 @@ control plane sends those itself:
| `node_lost` | A node was silent past `control.lost_after`; its replicas are rebuilt elsewhere. |
| `budget_spent` | A key, project, team or organization reached its budget; priced models are refused. |
| `update_available` | A model's registry repository published different weights. Once per revision. |
| `canary_rolled_back` | A model's canary served worse than the model and was taken out. |
| `judge_unavailable` | A guardrails judge gave no verdict: with `fail: closed` requests are being refused, with `open` served unjudged. |
```yaml
+48
View File
@@ -43,6 +43,11 @@ rather than repeated.
| `batch_size` | `int` | [Auth](#auth) |
| `buffer_size` | `int` | [Auth](#auth) |
| `usage_retention` | `time.Duration` | [Auth](#auth) |
| `model` | `string` | [Canary](#canary) |
| `percent` | `int` | [Canary](#canary) |
| `min_requests` | `int` | [Canary](#canary) |
| `max_error_increase` | `float64` | [Canary](#canary) |
| `max_ttft_ratio` | `float64` | [Canary](#canary) |
| `tools` | `bool` | [Capabilities](#capabilities) |
| `vision` | `bool` | [Capabilities](#capabilities) |
| `embeddings` | `bool` | [Capabilities](#capabilities) |
@@ -98,6 +103,7 @@ rather than repeated.
| `external` | `*External` | [Model](#model) |
| `adapters` | `[]Adapter` | [Model](#model) |
| `fallbacks` | `[]string` | [Model](#model) |
| `canary` | `*Canary` | [Model](#model) |
| `routing_profile` | `string` | [Model](#model) |
| `slo` | `SLO` | [Model](#model) |
| `context_length` | `int` | [Model](#model) |
@@ -511,6 +517,38 @@ five-minute sums kept beside the records, which are not pruned, so this
bounds the table that grows by a row a request without changing any
figure a page shows. At least a day when set.
## Canary
### `model`
`string`
Model is the candidate: a catalog model, usually this one at newer
weights, which clients cannot name themselves.
### `percent`
`int`
Percent of requests go to it, 0 to 100.
### `min_requests`
`int`
The candidate is taken out — rolled back — when, over at least
MinRequests of its own, its error rate is MaxErrorIncrease above
this model's (0.05 is five points), or its mean time to the first
token is MaxTTFTRatio times this model's. Zero is 50, 0.05 and 1.5.
### `max_error_increase`
`float64`
### `max_ttft_ratio`
`float64`
## Capabilities
### `tools`
@@ -916,6 +954,16 @@ every adapter, applying one only to a request that names it.
`[]string`
### `canary`
`*Canary`
Canary sends a share of this model's requests to another catalog
model — the same model at new weights — to compare the two before
moving to it (`llmgw ctl model canary`).
Takes a [Canary](#canary) block.
### `routing_profile`
`string`
+62
View File
@@ -680,6 +680,67 @@ func (m *consoleModel) doEntry() tea.Cmd {
return nil
}
// canaryItems try a model's newer weights on a share of its requests: start
// one, change its share, promote it, or take it out.
func (m consoleModel) canaryItems(md adminapi.ModelView) []menuItem {
path := "/api/v1/models/" + url.PathEscape(md.Name)
result := func(raw json.RawMessage) string {
var out map[string]string
_ = json.Unmarshal(raw, &out)
return out["result"] + "."
}
if cn := md.Canary; cn != nil {
return []menuItem{
{fmt.Sprintf("Canary's share (%d%%)…", cn.Percent), func(m *consoleModel) tea.Cmd {
m.openForm(&formDialog{title: "Share of " + md.Name + " for " + cn.Model,
fields: []formField{textF("percent", strconv.Itoa(cn.Percent), "0 to 100; 0 keeps it running and gives it none")},
submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
n, err := strconv.Atoi(v[0])
if err != nil || n < 0 || n > 100 {
return nil, fmt.Errorf("a whole number from 0 to 100")
}
return m.call("Canary of "+md.Name, http.MethodPut, path+"/canary", map[string]int{"percent": n}, result), nil
}})
return nil
}},
{"Promote the canary…", func(m *consoleModel) tea.Cmd {
m.confirmD = &confirmDialog{title: "Promote " + cn.Model,
text: md.Name + " moves to the canary's weights: its replicas are replaced one at a time,\n" +
"and the canary is removed. Going back is \"Go back to\" on this menu.",
yes: func(m *consoleModel) tea.Cmd {
return m.call("Promote "+cn.Model, http.MethodPost, path+"/promote", map[string]string{}, startedSays)
}}
return nil
}},
{"Take the canary out…", func(m *consoleModel) tea.Cmd {
m.confirmD = &confirmDialog{title: "Take " + cn.Model + " out",
text: md.Name + " serves all of its requests again, and the canary is removed.",
yes: func(m *consoleModel) tea.Cmd {
return m.call("Canary of "+md.Name, http.MethodDelete, path+"/canary", nil, result)
}}
return nil
}},
}
}
if u := md.Update; u == nil || !u.Available || md.CanaryOf != "" {
return nil
}
return []menuItem{{"Try the new weights as a canary…", func(m *consoleModel) tea.Cmd {
m.openForm(&formDialog{title: "Canary of " + md.Name,
note: "The new weights run beside it, one replica, given this share of its requests.\n" +
"Taken out by itself if it errs or answers more slowly than " + md.Name + ".",
fields: []formField{textF("percent", "10", "share of its requests, 0 to 100")},
submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
n, err := strconv.Atoi(v[0])
if err != nil || n < 0 || n > 100 {
return nil, fmt.Errorf("a whole number from 0 to 100")
}
return m.call("Canary of "+md.Name, http.MethodPost, path+"/canary", catalog.Canary{Percent: n}, startedSays), nil
}})
return nil
}}}
}
// updateItems are what can be done about a model's repository publishing new
// weights: move to them, ask now, or go back.
func (m consoleModel) updateItems(md adminapi.ModelView) []menuItem {
@@ -728,6 +789,7 @@ func (m consoleModel) updateItems(md adminapi.ModelView) []menuItem {
return nil
}})
}
items = append(items, m.canaryItems(md)...)
items = append(items, menuItem{"Ask the registry for new weights now", func(m *consoleModel) tea.Cmd {
return m.call("New weights for "+md.Name, http.MethodPost, "/api/v1/updates/check", map[string]string{"model": md.Name},
func(raw json.RawMessage) string {
+27
View File
@@ -17,6 +17,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
"gitea.mynodes.xyz/ilker/llmgw/internal/modelplan"
@@ -669,3 +670,29 @@ func TestTheAuditViewListsChangesAndOpensOne(t *testing.T) {
t.Errorf("not opened:\n%s", v)
}
}
func TestAModelsMenuOffersACanaryAndThenRunsIt(t *testing.T) {
m, _ := consoleWith(t, cModels)
labels := func(md adminapi.ModelView) []string {
var out []string
for _, it := range m.canaryItems(md) {
out = append(out, it.label)
}
return out
}
md := adminapi.ModelView{Name: "qwen", Update: &modelserve.UpdateInfo{Available: true, Latest: "def98765"}}
if got := labels(md); len(got) != 1 || !strings.Contains(got[0], "canary") {
t.Errorf("with new weights: %v", got)
}
md.Canary = &catalog.Canary{Model: "qwen-canary", Percent: 10}
got := strings.Join(labels(md), " | ")
for _, want := range []string{"share (10%)", "Promote", "Take the canary out"} {
if !strings.Contains(got, want) {
t.Errorf("with a canary, %q lacks %q", got, want)
}
}
if got := labels(adminapi.ModelView{Name: "qwen-canary", CanaryOf: "qwen",
Update: &modelserve.UpdateInfo{Available: true}}); len(got) != 0 {
t.Errorf("a canary offered a canary of its own: %v", got)
}
}
+8
View File
@@ -38,6 +38,13 @@ var Reference = []Block{
{Key: "buffer_size", Type: "int", Doc: "BufferSize bounds the queue. Beyond it records are dropped rather than\nblocking a request: losing accounting is bad, turning a stalled database\ninto an outage is worse.", Nested: ""},
{Key: "usage_retention", Type: "time.Duration", Doc: "UsageRetention is how long the record of each request is kept: who sent\nit, which replica answered, how long it queued — what `ctl request`\nreads. Zero keeps it for ever. Totals, charts and budgets are read from\nfive-minute sums kept beside the records, which are not pruned, so this\nbounds the table that grows by a row a request without changing any\nfigure a page shows. At least a day when set.", Nested: ""},
}},
{Package: "catalog", Name: "Canary", Title: "Canary", Doc: "", Fields: []Field{
{Key: "model", Type: "string", Doc: "Model is the candidate: a catalog model, usually this one at newer\nweights, which clients cannot name themselves.", Nested: ""},
{Key: "percent", Type: "int", Doc: "Percent of requests go to it, 0 to 100.", Nested: ""},
{Key: "min_requests", Type: "int", Doc: "The candidate is taken out — rolled back — when, over at least\nMinRequests of its own, its error rate is MaxErrorIncrease above\nthis model's (0.05 is five points), or its mean time to the first\ntoken is MaxTTFTRatio times this model's. Zero is 50, 0.05 and 1.5.", Nested: ""},
{Key: "max_error_increase", Type: "float64", Doc: "", Nested: ""},
{Key: "max_ttft_ratio", Type: "float64", Doc: "", Nested: ""},
}},
{Package: "catalog", Name: "Capabilities", Title: "Capabilities", Doc: "", Fields: []Field{
{Key: "tools", Type: "bool", Doc: "", Nested: ""},
{Key: "vision", Type: "bool", Doc: "", Nested: ""},
@@ -114,6 +121,7 @@ var Reference = []Block{
{Key: "external", Type: "*External", Doc: "External is a model served by someone else — an OpenAI-compatible API\noutside the cluster — rather than by replicas placed here. Such a model\nhas no profiles and nothing is placed for it; the router sends its\nrequests to the provider. Most useful named in another model's\nfallbacks: used only when the cluster has no capacity left. Its\nprompts leave the cluster, so which keys may name it is worth an\nallowlist.", Nested: "catalog.External"},
{Key: "adapters", Type: "[]Adapter", Doc: "Adapters are LoRA adapters served on this model's replicas, each asked\nfor by its own name as though it were a model. Every replica loads\nevery adapter, applying one only to a request that names it.", Nested: ""},
{Key: "fallbacks", Type: "[]string", Doc: "", Nested: ""},
{Key: "canary", Type: "*Canary", Doc: "Canary sends a share of this model's requests to another catalog\nmodel — the same model at new weights — to compare the two before\nmoving to it (`llmgw ctl model canary`).", Nested: "catalog.Canary"},
{Key: "routing_profile", Type: "string", Doc: "RoutingProfile selects how the router scores endpoints for this model.", Nested: ""},
{Key: "slo", Type: "SLO", Doc: "", Nested: "catalog.SLO"},
{Key: "context_length", Type: "int", Doc: "ContextLength is how many tokens one request may use, prompt and\ncompletion together, and MaxOutput bounds the completion alone. Zero\nmeans unknown, which is what every model was until now.\n\nThe gateway knew neither, so an over-long prompt was not refused: it was\nadmitted, given a replica, and failed at the engine, and a client sizing\nits prompts from /v1/models had nothing to read. Worse, the number is not\nthe one an operator writes in the engine's arguments — llama.cpp divides\n--ctx-size across its parallel slots, so a pool of 8192 across four slots\nis 2048 a request, and nothing said so.", Nested: ""},