From 889be70b8f8e279d9c6e1e4cbc599751f6659c00 Mon Sep 17 00:00:00 2001 From: Ilker Manap Date: Sun, 4 Oct 2026 20:33:59 +0200 Subject: [PATCH] Canary on the console's model menu; docs Co-Authored-By: Claude Opus 5.5 (1M context) --- BACKLOG.md | 8 ++++- CHANGELOG.md | 17 +++++++++ docs/console.md | 7 ++++ docs/install.md | 33 +++++++++++++++++ docs/reference.md | 48 +++++++++++++++++++++++++ internal/cli/tui_actions.go | 62 ++++++++++++++++++++++++++++++++ internal/cli/tui_actions_test.go | 27 ++++++++++++++ internal/docgen/reference_gen.go | 8 +++++ 8 files changed, 209 insertions(+), 1 deletion(-) diff --git a/BACKLOG.md b/BACKLOG.md index 4d19671..90ca888 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -420,7 +420,13 @@ portal is not done — the page shows; the console and ctl change. **Request metadata and tags (S).** Accept `metadata`/`user` from the body or `x-llmgw-tags`; store on the usage row; tenant metric labels behind an allowlist. `router/handlers.go` parse, `accounting/recorder.go`. -7. **Canary and weighted A/B between model revisions (M).** +7. **Done (branch `features-c2`):** `ctl model canary/promote/canaries`, + session-sticky split with `x-llmgw-variant` pins, rollback on error rate + or TTFT, `canary_rolled_back` notification, console menu. Left: A/B + between two different models (the split is between a model and its + newer weights); the comparison is per control plane and lives in memory, + so a restart begins it again; a key-level pin. + **Canary and weighted A/B between model revisions (M).** `control/modelserve/updates.go` `replace` swaps every replica; add weighted routing across two revisions/aliases (scheduler filter, alias resolution in `router/router.go:569`), a header/key pin, automatic diff --git a/CHANGELOG.md b/CHANGELOG.md index 2a22482..7bda153 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -71,6 +71,23 @@ APIs change between minors without a deprecation period. - **Other ways in:** `GET /api/v1/usage/export` (CSV, or `format=json`) and **E** on the console's Usage view. +### New weights tried on a share of the traffic first + +- **Canaries.** `llmgw ctl model canary MODEL --percent 10` fetches the newer + weights of a model's repository and runs them beside it as `MODEL-canary`, + given that share of its requests. + - **Routing:** a session stays on one side; `x-llmgw-variant` pins a + request to `canary` or `stable` and says which served it. Clients cannot + name the canary. + - **Rolled back by itself** when, over 50 of its requests or more, its + error rate is five points above the model's or its time to the first + token half again as long; `canary_rolled_back` is sent to the + notification webhooks. + - **Then:** `--percent` changes its share, `llmgw ctl model promote` + moves the model to it, `--abort` takes it out, and `llmgw ctl model + canaries` compares the two sides. All of it is also on the console's + model menu. + ### Requests can be tagged - **Tags and end users on every usage record.** OpenAI's `metadata` and the diff --git a/docs/console.md b/docs/console.md index 8a8b047..44fcbdc 100644 --- a/docs/console.md +++ b/docs/console.md @@ -110,6 +110,13 @@ and **UPDATE** when the registry has new weights for it. On a row: **Go back to …** returns to the weights the update replaced. **Ask the registry for new weights now** checks without waiting for the next scheduled look (`mirror.update_check`, six hours by default). +- **Try the new weights as a canary…** — instead of swapping them in, runs + them beside the model as `MODEL-canary`, one replica, given the share of + its requests you choose. With a canary running the menu has **Canary's + share…**, **Promote the canary…** (the model moves to its weights, the + canary is removed) and **Take the canary out…**. A canary that errs or + answers more slowly than the model is taken out by itself. The MODEL + column shows `(10% to canary)` and `(canary of …)`. - **Why is it placed where it is** — `llmgw ctl explain-placement`. - **Try again**, for a model parked after failing every start. - **Remove** — its replicas stop and it leaves the catalog. A model from diff --git a/docs/install.md b/docs/install.md index 4eb4b71..fae8cd8 100644 --- a/docs/install.md +++ b/docs/install.md @@ -879,6 +879,38 @@ prices and budgets, rate limits, usage records, and key allowlists. whose allowlist leaves the external model out cannot reach it, as a fallback or by name. The cluster page and `ctl models` mark it `external`. +### Trying new weights on some of the traffic first + +When a model's repository publishes new weights, `llmgw ctl model update` +swaps them in. To see how they serve first, run them as a canary: + +``` +llmgw ctl model canary qwen-7b --percent 10 # fetched, started beside, given 10% +llmgw ctl model canaries # how each side has served +llmgw ctl model canary qwen-7b --percent 50 # more +llmgw ctl model promote qwen-7b # the model moves to them +llmgw ctl model canary qwen-7b --abort # or not +``` + +- **What runs:** the new files are fetched into the mirror and checked as + for an update, then run as `qwen-7b-canary` — the same model in everything + but its weights, one replica. Clients cannot name it; it gets its share + of `qwen-7b`'s requests. +- **Who goes where:** a session (`x-session-id`, or the `user` field) + stays on one side, so a conversation does not change weights halfway. + `x-llmgw-variant: canary` or `stable` pins a request to one side; the + response says which served it in the same header, and the usage record + names `qwen-7b-canary` as the model that served it. +- **Rolled back by itself:** once the canary has served `--min-requests` + (50), it is taken out if its error rate is `--max-error-increase` (0.05, + five points) above the model's, or its mean time to the first token — or + to the answer, unstreamed — is `--max-ttft-ratio` (1.5) times the + model's. `canary_rolled_back` goes to the notification webhooks. The + comparison is the router's own count since the canary began, kept per + control plane. +- **Promote** is the update the canary was a trial of: the model's replicas + are replaced one at a time, and `llmgw ctl model rollback` goes back. + ### How many replicas, and how much each one carries Replica count is changed while the cluster runs, without editing the @@ -981,6 +1013,7 @@ control plane sends those itself: | `node_lost` | A node was silent past `control.lost_after`; its replicas are rebuilt elsewhere. | | `budget_spent` | A key, project, team or organization reached its budget; priced models are refused. | | `update_available` | A model's registry repository published different weights. Once per revision. | +| `canary_rolled_back` | A model's canary served worse than the model and was taken out. | | `judge_unavailable` | A guardrails judge gave no verdict: with `fail: closed` requests are being refused, with `open` served unjudged. | ```yaml diff --git a/docs/reference.md b/docs/reference.md index ceb5a02..575d192 100644 --- a/docs/reference.md +++ b/docs/reference.md @@ -43,6 +43,11 @@ rather than repeated. | `batch_size` | `int` | [Auth](#auth) | | `buffer_size` | `int` | [Auth](#auth) | | `usage_retention` | `time.Duration` | [Auth](#auth) | +| `model` | `string` | [Canary](#canary) | +| `percent` | `int` | [Canary](#canary) | +| `min_requests` | `int` | [Canary](#canary) | +| `max_error_increase` | `float64` | [Canary](#canary) | +| `max_ttft_ratio` | `float64` | [Canary](#canary) | | `tools` | `bool` | [Capabilities](#capabilities) | | `vision` | `bool` | [Capabilities](#capabilities) | | `embeddings` | `bool` | [Capabilities](#capabilities) | @@ -98,6 +103,7 @@ rather than repeated. | `external` | `*External` | [Model](#model) | | `adapters` | `[]Adapter` | [Model](#model) | | `fallbacks` | `[]string` | [Model](#model) | +| `canary` | `*Canary` | [Model](#model) | | `routing_profile` | `string` | [Model](#model) | | `slo` | `SLO` | [Model](#model) | | `context_length` | `int` | [Model](#model) | @@ -511,6 +517,38 @@ five-minute sums kept beside the records, which are not pruned, so this bounds the table that grows by a row a request without changing any figure a page shows. At least a day when set. +## Canary + +### `model` + +`string` + +Model is the candidate: a catalog model, usually this one at newer +weights, which clients cannot name themselves. + +### `percent` + +`int` + +Percent of requests go to it, 0 to 100. + +### `min_requests` + +`int` + +The candidate is taken out — rolled back — when, over at least +MinRequests of its own, its error rate is MaxErrorIncrease above +this model's (0.05 is five points), or its mean time to the first +token is MaxTTFTRatio times this model's. Zero is 50, 0.05 and 1.5. + +### `max_error_increase` + +`float64` + +### `max_ttft_ratio` + +`float64` + ## Capabilities ### `tools` @@ -916,6 +954,16 @@ every adapter, applying one only to a request that names it. `[]string` +### `canary` + +`*Canary` + +Canary sends a share of this model's requests to another catalog +model — the same model at new weights — to compare the two before +moving to it (`llmgw ctl model canary`). + +Takes a [Canary](#canary) block. + ### `routing_profile` `string` diff --git a/internal/cli/tui_actions.go b/internal/cli/tui_actions.go index 4c8f251..b73c109 100644 --- a/internal/cli/tui_actions.go +++ b/internal/cli/tui_actions.go @@ -680,6 +680,67 @@ func (m *consoleModel) doEntry() tea.Cmd { return nil } +// canaryItems try a model's newer weights on a share of its requests: start +// one, change its share, promote it, or take it out. +func (m consoleModel) canaryItems(md adminapi.ModelView) []menuItem { + path := "/api/v1/models/" + url.PathEscape(md.Name) + result := func(raw json.RawMessage) string { + var out map[string]string + _ = json.Unmarshal(raw, &out) + return out["result"] + "." + } + if cn := md.Canary; cn != nil { + return []menuItem{ + {fmt.Sprintf("Canary's share (%d%%)…", cn.Percent), func(m *consoleModel) tea.Cmd { + m.openForm(&formDialog{title: "Share of " + md.Name + " for " + cn.Model, + fields: []formField{textF("percent", strconv.Itoa(cn.Percent), "0 to 100; 0 keeps it running and gives it none")}, + submit: func(m *consoleModel, v []string) (tea.Cmd, error) { + n, err := strconv.Atoi(v[0]) + if err != nil || n < 0 || n > 100 { + return nil, fmt.Errorf("a whole number from 0 to 100") + } + return m.call("Canary of "+md.Name, http.MethodPut, path+"/canary", map[string]int{"percent": n}, result), nil + }}) + return nil + }}, + {"Promote the canary…", func(m *consoleModel) tea.Cmd { + m.confirmD = &confirmDialog{title: "Promote " + cn.Model, + text: md.Name + " moves to the canary's weights: its replicas are replaced one at a time,\n" + + "and the canary is removed. Going back is \"Go back to\" on this menu.", + yes: func(m *consoleModel) tea.Cmd { + return m.call("Promote "+cn.Model, http.MethodPost, path+"/promote", map[string]string{}, startedSays) + }} + return nil + }}, + {"Take the canary out…", func(m *consoleModel) tea.Cmd { + m.confirmD = &confirmDialog{title: "Take " + cn.Model + " out", + text: md.Name + " serves all of its requests again, and the canary is removed.", + yes: func(m *consoleModel) tea.Cmd { + return m.call("Canary of "+md.Name, http.MethodDelete, path+"/canary", nil, result) + }} + return nil + }}, + } + } + if u := md.Update; u == nil || !u.Available || md.CanaryOf != "" { + return nil + } + return []menuItem{{"Try the new weights as a canary…", func(m *consoleModel) tea.Cmd { + m.openForm(&formDialog{title: "Canary of " + md.Name, + note: "The new weights run beside it, one replica, given this share of its requests.\n" + + "Taken out by itself if it errs or answers more slowly than " + md.Name + ".", + fields: []formField{textF("percent", "10", "share of its requests, 0 to 100")}, + submit: func(m *consoleModel, v []string) (tea.Cmd, error) { + n, err := strconv.Atoi(v[0]) + if err != nil || n < 0 || n > 100 { + return nil, fmt.Errorf("a whole number from 0 to 100") + } + return m.call("Canary of "+md.Name, http.MethodPost, path+"/canary", catalog.Canary{Percent: n}, startedSays), nil + }}) + return nil + }}} +} + // updateItems are what can be done about a model's repository publishing new // weights: move to them, ask now, or go back. func (m consoleModel) updateItems(md adminapi.ModelView) []menuItem { @@ -728,6 +789,7 @@ func (m consoleModel) updateItems(md adminapi.ModelView) []menuItem { return nil }}) } + items = append(items, m.canaryItems(md)...) items = append(items, menuItem{"Ask the registry for new weights now", func(m *consoleModel) tea.Cmd { return m.call("New weights for "+md.Name, http.MethodPost, "/api/v1/updates/check", map[string]string{"model": md.Name}, func(raw json.RawMessage) string { diff --git a/internal/cli/tui_actions_test.go b/internal/cli/tui_actions_test.go index 859d471..9beebbe 100644 --- a/internal/cli/tui_actions_test.go +++ b/internal/cli/tui_actions_test.go @@ -17,6 +17,7 @@ import ( "gitea.mynodes.xyz/ilker/llmgw/internal/config" "gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi" + "gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog" "gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve" "gitea.mynodes.xyz/ilker/llmgw/internal/guard" "gitea.mynodes.xyz/ilker/llmgw/internal/modelplan" @@ -669,3 +670,29 @@ func TestTheAuditViewListsChangesAndOpensOne(t *testing.T) { t.Errorf("not opened:\n%s", v) } } + +func TestAModelsMenuOffersACanaryAndThenRunsIt(t *testing.T) { + m, _ := consoleWith(t, cModels) + labels := func(md adminapi.ModelView) []string { + var out []string + for _, it := range m.canaryItems(md) { + out = append(out, it.label) + } + return out + } + md := adminapi.ModelView{Name: "qwen", Update: &modelserve.UpdateInfo{Available: true, Latest: "def98765"}} + if got := labels(md); len(got) != 1 || !strings.Contains(got[0], "canary") { + t.Errorf("with new weights: %v", got) + } + md.Canary = &catalog.Canary{Model: "qwen-canary", Percent: 10} + got := strings.Join(labels(md), " | ") + for _, want := range []string{"share (10%)", "Promote", "Take the canary out"} { + if !strings.Contains(got, want) { + t.Errorf("with a canary, %q lacks %q", got, want) + } + } + if got := labels(adminapi.ModelView{Name: "qwen-canary", CanaryOf: "qwen", + Update: &modelserve.UpdateInfo{Available: true}}); len(got) != 0 { + t.Errorf("a canary offered a canary of its own: %v", got) + } +} diff --git a/internal/docgen/reference_gen.go b/internal/docgen/reference_gen.go index 39f81dc..9d01106 100644 --- a/internal/docgen/reference_gen.go +++ b/internal/docgen/reference_gen.go @@ -38,6 +38,13 @@ var Reference = []Block{ {Key: "buffer_size", Type: "int", Doc: "BufferSize bounds the queue. Beyond it records are dropped rather than\nblocking a request: losing accounting is bad, turning a stalled database\ninto an outage is worse.", Nested: ""}, {Key: "usage_retention", Type: "time.Duration", Doc: "UsageRetention is how long the record of each request is kept: who sent\nit, which replica answered, how long it queued — what `ctl request`\nreads. Zero keeps it for ever. Totals, charts and budgets are read from\nfive-minute sums kept beside the records, which are not pruned, so this\nbounds the table that grows by a row a request without changing any\nfigure a page shows. At least a day when set.", Nested: ""}, }}, + {Package: "catalog", Name: "Canary", Title: "Canary", Doc: "", Fields: []Field{ + {Key: "model", Type: "string", Doc: "Model is the candidate: a catalog model, usually this one at newer\nweights, which clients cannot name themselves.", Nested: ""}, + {Key: "percent", Type: "int", Doc: "Percent of requests go to it, 0 to 100.", Nested: ""}, + {Key: "min_requests", Type: "int", Doc: "The candidate is taken out — rolled back — when, over at least\nMinRequests of its own, its error rate is MaxErrorIncrease above\nthis model's (0.05 is five points), or its mean time to the first\ntoken is MaxTTFTRatio times this model's. Zero is 50, 0.05 and 1.5.", Nested: ""}, + {Key: "max_error_increase", Type: "float64", Doc: "", Nested: ""}, + {Key: "max_ttft_ratio", Type: "float64", Doc: "", Nested: ""}, + }}, {Package: "catalog", Name: "Capabilities", Title: "Capabilities", Doc: "", Fields: []Field{ {Key: "tools", Type: "bool", Doc: "", Nested: ""}, {Key: "vision", Type: "bool", Doc: "", Nested: ""}, @@ -114,6 +121,7 @@ var Reference = []Block{ {Key: "external", Type: "*External", Doc: "External is a model served by someone else — an OpenAI-compatible API\noutside the cluster — rather than by replicas placed here. Such a model\nhas no profiles and nothing is placed for it; the router sends its\nrequests to the provider. Most useful named in another model's\nfallbacks: used only when the cluster has no capacity left. Its\nprompts leave the cluster, so which keys may name it is worth an\nallowlist.", Nested: "catalog.External"}, {Key: "adapters", Type: "[]Adapter", Doc: "Adapters are LoRA adapters served on this model's replicas, each asked\nfor by its own name as though it were a model. Every replica loads\nevery adapter, applying one only to a request that names it.", Nested: ""}, {Key: "fallbacks", Type: "[]string", Doc: "", Nested: ""}, + {Key: "canary", Type: "*Canary", Doc: "Canary sends a share of this model's requests to another catalog\nmodel — the same model at new weights — to compare the two before\nmoving to it (`llmgw ctl model canary`).", Nested: "catalog.Canary"}, {Key: "routing_profile", Type: "string", Doc: "RoutingProfile selects how the router scores endpoints for this model.", Nested: ""}, {Key: "slo", Type: "SLO", Doc: "", Nested: "catalog.SLO"}, {Key: "context_length", Type: "int", Doc: "ContextLength is how many tokens one request may use, prompt and\ncompletion together, and MaxOutput bounds the completion alone. Zero\nmeans unknown, which is what every model was until now.\n\nThe gateway knew neither, so an over-long prompt was not refused: it was\nadmitted, given a replica, and failed at the engine, and a client sizing\nits prompts from /v1/models had nothing to read. Worse, the number is not\nthe one an operator writes in the engine's arguments — llama.cpp divides\n--ctx-size across its parallel slots, so a pool of 8192 across four slots\nis 2048 a request, and nothing said so.", Nested: ""},