From 29a7347df2a0aaee4c91c8621290ec7cb81f1094 Mon Sep 17 00:00:00 2001 From: Ilker Manap Date: Sun, 4 Oct 2026 19:45:37 +0200 Subject: [PATCH 01/38] Requests can be tagged: metadata, x-llmgw-tags and the user field on the usage record, and counted by the tags router.tag_labels names Co-Authored-By: Claude Opus 5.5 (1M context) --- BACKLOG.md | 7 +- CHANGELOG.md | 11 ++ docs/data.md | 7 +- docs/install.md | 35 ++++++ docs/reference.md | 11 ++ internal/cli/ctl_request.go | 17 +++ internal/config/config.go | 7 ++ internal/control/adminapi/api.go | 4 + internal/docgen/reference_gen.go | 1 + internal/router/guard.go | 1 + internal/router/handlers.go | 28 ++++- internal/router/router.go | 1 + internal/router/tags.go | 91 ++++++++++++++++ internal/router/tags_test.go | 101 ++++++++++++++++++ internal/router/tenantmetrics.go | 77 +++++++++++-- .../migrations/postgres/0023_usage_tags.sql | 7 ++ .../migrations/sqlite/0023_usage_tags.sql | 6 ++ internal/store/sqlite.go | 13 ++- internal/store/store.go | 28 +++++ 19 files changed, 436 insertions(+), 17 deletions(-) create mode 100644 internal/router/tags.go create mode 100644 internal/router/tags_test.go create mode 100644 internal/store/migrations/postgres/0023_usage_tags.sql create mode 100644 internal/store/migrations/sqlite/0023_usage_tags.sql diff --git a/BACKLOG.md b/BACKLOG.md index c61dfe03..4d196713 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -412,7 +412,12 @@ portal is not done — the page shows; the console and ctl change. templates driven by `server init` or the console; llmgw-native webhooks for model parked, budget spent, update available (from the reconciler and `modelserve`). -6. **Request metadata and tags (S).** Accept `metadata`/`user` from the +6. **Done (branch `features-c2`):** `metadata`, `x-llmgw-tags` and `user` + on the usage record (migration 0023), `ctl request` shows them, + `router.tag_labels` metrics with 100 values per tag. Left: export and + `ctl usage` by tag or end user (the hourly and daily sums do not carry + tags; it would read the request records, within their retention). + **Request metadata and tags (S).** Accept `metadata`/`user` from the body or `x-llmgw-tags`; store on the usage row; tenant metric labels behind an allowlist. `router/handlers.go` parse, `accounting/recorder.go`. 7. **Canary and weighted A/B between model revisions (M).** diff --git a/CHANGELOG.md b/CHANGELOG.md index cf7b0f41..2a224821 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -71,6 +71,17 @@ APIs change between minors without a deprecation period. - **Other ways in:** `GET /api/v1/usage/export` (CSV, or `format=json`) and **E** on the console's Usage view. +### Requests can be tagged + +- **Tags and end users on every usage record.** OpenAI's `metadata` and the + `x-llmgw-tags` header (`feature=search, env=prod`) tag a request; OpenAI's + `user` field names the client's own end user. Both are kept with the + request's usage and shown by `llmgw ctl request`. Tags are not sent to the + engine. A malformed one is refused with 400. +- **Counted by tag, for the tags named.** `router.tag_labels` lists the tags + counted in `llmgw_tagged_requests_total` and `llmgw_tagged_tokens_total`, + the first 100 values of each apart and the rest as `other`. + ### A hardened install by default - **`server init` and the setup screens write a hardened configuration.** diff --git a/docs/data.md b/docs/data.md index 77ed63a4..c103c23b 100644 --- a/docs/data.md +++ b/docs/data.md @@ -32,7 +32,12 @@ the text, the value that matched, or the placeholder map of a redaction. See **Usage records**, one per request: the key, the project, the model asked for, the model that answered, prompt and completion token counts, cost, HTTP status and latency. No message content. The row is in `internal/store/store.go` as -`Usage` and there is no field for text in it. +`Usage` and there is no field for text in it. What the client chose to say +about the request is kept with it: OpenAI's `user` field and its `metadata` +or `x-llmgw-tags` — short strings, set by the client, for attributing spend. +A client that puts a person's name or address there puts it in the usage +records; the field is the client's, and so is that choice. Tags are not sent +on to the engine. **API keys**, as SHA-256 hashes. The key itself is shown once, at creation, and never stored — so a leaked database hands over no working credential, and diff --git a/docs/install.md b/docs/install.md index 9059a9ba..4eb4b71e 100644 --- a/docs/install.md +++ b/docs/install.md @@ -1062,6 +1062,41 @@ outcome than a smaller cache, so cap it: `shared_buffers = 96MB` and `max_connections = 40` are enough for a store whose working set is measured in megabytes. +## Tagging requests + +A client can say what a request was for — which feature, which customer, +which environment — and the spend can then be found by it inside a project: + +``` +curl http://gateway:8080/v1/chat/completions \ + -H "Authorization: Bearer $LLMGW_API_KEY" \ + -H "x-llmgw-tags: feature=search, env=prod" \ + -d '{"model": "qwen-1.5b", "user": "customer-42", + "metadata": {"feature": "search"}, + "messages": [{"role": "user", "content": "hello"}]}' +``` + +- **`metadata`** is OpenAI's: an object of strings, up to 16 keys. **`x-llmgw-tags`** + is the same as a header, for clients that cannot change the body; for a + key both give, the header wins. A key is 1 to 64 letters, digits, `_`, + `.` or `-`, a value at most 512 bytes; anything else is refused with 400 + rather than dropped, since a tag lost silently is spend nobody finds. +- **`user`** is OpenAI's end-user field: the client's own name for whoever + it serves, not an account here. It is passed on to the engine, as OpenAI + does; tags are not. +- **Where they show:** every usage record keeps them, and `llmgw ctl request + ` prints them. For Prometheus, name the tags worth a series: + + ```yaml + router: + tag_labels: [feature, env] + ``` + + `llmgw_tagged_requests_total{tag,value,outcome}` and + `llmgw_tagged_tokens_total{tag,value,direction}` count those, the first + 100 values of each apart and the rest as `other`: a tag is the client's + to set, and every value is a series. + ## Conversations across models Off everywhere until a project asks. llmgw otherwise stores no message content diff --git a/docs/reference.md b/docs/reference.md index df96ec45..ceb5a022 100644 --- a/docs/reference.md +++ b/docs/reference.md @@ -190,6 +190,7 @@ rather than repeated. | `session_header` | `string` | [Router](#router) | | `inject_usage` | `bool` | [Router](#router) | | `tenant_label` | `string` | [Router](#router) | +| `tag_labels` | `[]string` | [Router](#router) | | `max_body_bytes` | `int64` | [Router](#router) | | `trusted_proxies` | `[]string` | [Router](#router) | | `timeouts` | `Timeouts` | [Router](#router) | @@ -1557,6 +1558,16 @@ labelled by — llmgw_tenant_requests_total and its siblings: "org", on a cluster of thousands is more series than Prometheus wants, and a team is what an alert is usually about; "team" is the default. +### `tag_labels` + +`[]string` + +TagLabels are the request tags (metadata, x-llmgw-tags) counted in +llmgw_tagged_requests_total and llmgw_tagged_tokens_total, by value. +Only these: a tag is the client's to set, and every value of one is +a series. The first 100 values of each are counted apart, the rest as +"other". Empty counts none; the usage records keep every tag anyway. + ### `max_body_bytes` `int64` diff --git a/internal/cli/ctl_request.go b/internal/cli/ctl_request.go index 036b888e..d0c8e5fc 100644 --- a/internal/cli/ctl_request.go +++ b/internal/cli/ctl_request.go @@ -4,6 +4,8 @@ import ( "encoding/json" "fmt" "net/url" + "slices" + "strings" "github.com/spf13/cobra" @@ -39,6 +41,21 @@ func newRequestCmd(client func() *ctlClient, asJSON *bool) *cobra.Command { fmt.Fprintf(out, "time %d ms total: %d queued, %d to the first token\n", v.LatencyMs, v.QueueMs, v.TTFTMs) fmt.Fprintf(out, "tokens %d in, %d out\n", v.PromptTokens, v.CompletionTokens) fmt.Fprintf(out, "who key %s, project %s%s\n", v.KeyID, v.ProjectID, prefixed(", user ", v.UserID)) + if v.EndUser != "" { + fmt.Fprintf(out, "end user %s (the client's own name for them)\n", v.EndUser) + } + if len(v.Tags) > 0 { + keys := make([]string, 0, len(v.Tags)) + for k := range v.Tags { + keys = append(keys, k) + } + slices.Sort(keys) + var kv []string + for _, k := range keys { + kv = append(kv, k+"="+v.Tags[k]) + } + fmt.Fprintf(out, "tags %s\n", strings.Join(kv, ", ")) + } if v.GuardAction != "" { fmt.Fprintf(out, "guard %s by policy %s (%s), %d ms\n", v.GuardAction, v.GuardPolicy, or(v.GuardCategory, "-"), v.GuardMs) diff --git a/internal/config/config.go b/internal/config/config.go index 5d7050cf..d19a2e20 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -443,6 +443,13 @@ type Router struct { // team is what an alert is usually about; "team" is the default. TenantLabel string `koanf:"tenant_label"` + // TagLabels are the request tags (metadata, x-llmgw-tags) counted in + // llmgw_tagged_requests_total and llmgw_tagged_tokens_total, by value. + // Only these: a tag is the client's to set, and every value of one is + // a series. The first 100 values of each are counted apart, the rest as + // "other". Empty counts none; the usage records keep every tag anyway. + TagLabels []string `koanf:"tag_labels"` + // MaxBodyBytes bounds the request body the router will read. MaxBodyBytes int64 `koanf:"max_body_bytes"` diff --git a/internal/control/adminapi/api.go b/internal/control/adminapi/api.go index 12a217e4..19aecd00 100644 --- a/internal/control/adminapi/api.go +++ b/internal/control/adminapi/api.go @@ -1406,6 +1406,9 @@ type RequestView struct { GuardPolicy string `json:"guard_policy,omitempty"` GuardCategory string `json:"guard_category,omitempty"` GuardMs int64 `json:"guard_ms,omitempty"` + // What the client said about it: OpenAI's user field, and its tags. + EndUser string `json:"end_user,omitempty"` + Tags map[string]string `json:"tags,omitempty"` } func (a *API) request(w http.ResponseWriter, r *http.Request) { @@ -1429,6 +1432,7 @@ func (a *API) request(w http.ResponseWriter, r *http.Request) { LatencyMs: u.LatencyMs, QueueMs: u.QueueMs, TTFTMs: u.TTFTMs, PromptTokens: u.PromptTokens, CompletionTokens: u.CompletionTokens, CostMicros: u.CostMicros, GuardAction: u.GuardAction, GuardPolicy: u.GuardPolicy, GuardCategory: u.GuardCategory, GuardMs: u.GuardMs, + EndUser: u.EndUser, Tags: u.Tags, }) } diff --git a/internal/docgen/reference_gen.go b/internal/docgen/reference_gen.go index 839dea7e..39f81dc0 100644 --- a/internal/docgen/reference_gen.go +++ b/internal/docgen/reference_gen.go @@ -230,6 +230,7 @@ var Reference = []Block{ {Key: "session_header", Type: "string", Doc: "SessionHeader carries the session id for affinity (plan §7.5 tier 0).", Nested: ""}, {Key: "inject_usage", Type: "bool", Doc: "InjectUsage asks engines for usage even when the client did not, so token\naccounting works regardless of client behaviour.", Nested: ""}, {Key: "tenant_label", Type: "string", Doc: "TenantLabel is the level of the tree the per-tenant request series are\nlabelled by — llmgw_tenant_requests_total and its siblings: \"org\",\n\"team\" or \"project\", or \"none\" for no such series. A label per project\non a cluster of thousands is more series than Prometheus wants, and a\nteam is what an alert is usually about; \"team\" is the default.", Nested: ""}, + {Key: "tag_labels", Type: "[]string", Doc: "TagLabels are the request tags (metadata, x-llmgw-tags) counted in\nllmgw_tagged_requests_total and llmgw_tagged_tokens_total, by value.\nOnly these: a tag is the client's to set, and every value of one is\na series. The first 100 values of each are counted apart, the rest as\n\"other\". Empty counts none; the usage records keep every tag anyway.", Nested: ""}, {Key: "max_body_bytes", Type: "int64", Doc: "MaxBodyBytes bounds the request body the router will read.", Nested: ""}, {Key: "trusted_proxies", Type: "[]string", Doc: "TrustedProxies are the reverse proxies in front of the client port,\nas addresses or CIDR prefixes. Their X-Forwarded-For, -Proto and -Host\nare believed; anyone else's are not, since they decide the address\nwritten into the setup scripts and whose attempts are counted where.", Nested: ""}, {Key: "timeouts", Type: "Timeouts", Doc: "", Nested: "config.Timeouts"}, diff --git a/internal/router/guard.go b/internal/router/guard.go index 114e467b..6df9d144 100644 --- a/internal/router/guard.go +++ b/internal/router/guard.go @@ -135,6 +135,7 @@ func (r *Router) refuseGuarded(w http.ResponseWriter, req *parsedRequest, v guar At: time.Now(), KeyID: req.KeyID, UserID: req.UserID, ProjectID: req.ProjectID, Model: req.Model, ServedModel: req.Model, Status: status, LatencyMs: time.Since(start).Milliseconds(), RequestID: req.RequestID, QueueMs: req.QueueMs, + EndUser: req.EndUser, Tags: req.Tags, } guardFields(&u, v, guard.Verdict{}) // Settled at nothing: the estimate the limit check charged is diff --git a/internal/router/handlers.go b/internal/router/handlers.go index 5e6a5e43..f18a05c0 100644 --- a/internal/router/handlers.go +++ b/internal/router/handlers.go @@ -280,6 +280,10 @@ func (r *Router) Handler() http.Handler { // business interpreting. type parsedRequest struct { Path string + // Tags are the client's metadata and x-llmgw-tags; EndUser is OpenAI's + // user field. Both are recorded with the usage, never sent on. + Tags map[string]string + EndUser string // Model is what will be dispatched, which is not always what was asked // for: a caller out of budget may be moved to a model they can afford. Model string @@ -338,9 +342,10 @@ type parsedRequest struct { // inferenceBody covers the fields the router reads across all inference // endpoints. Unknown fields are ignored here and preserved in UpstreamBody. type inferenceBody struct { - Model string `json:"model"` - Stream bool `json:"stream"` - User string `json:"user"` + Model string `json:"model"` + Stream bool `json:"stream"` + User string `json:"user"` + Metadata json.RawMessage `json:"metadata"` Messages []struct { Role string `json:"role"` Content json.RawMessage `json:"content"` @@ -587,6 +592,7 @@ func (r *Router) account(req *parsedRequest, res dispatchResult, status int, too LatencyMs: took.Milliseconds(), RequestID: req.RequestID, Replica: res.endpointID, QueueMs: req.QueueMs, TTFTMs: res.outcome.TTFT.Milliseconds(), + EndUser: req.EndUser, Tags: req.Tags, } guardFields(&u, req.guardIn, res.guard) if u.ServedModel == "" { @@ -625,6 +631,7 @@ func (r *Router) rejectLimit(w http.ResponseWriter, req *parsedRequest, err erro At: time.Now(), KeyID: req.KeyID, UserID: req.UserID, ProjectID: req.ProjectID, Model: req.Model, ServedModel: req.Model, Status: http.StatusTooManyRequests, LatencyMs: time.Since(start).Milliseconds(), RequestID: req.RequestID, + EndUser: req.EndUser, Tags: req.Tags, }) r.accessLog(req, dispatchResult{err: err}, http.StatusTooManyRequests, time.Since(start)) } @@ -644,7 +651,14 @@ func (r *Router) parse(req *http.Request, path string) (*parsedRequest, error) { return nil, errors.New("model is required") } + tags, err := requestTags(b.Metadata, req.Header.Get(tagsHeader)) + if err != nil { + return nil, err + } + p := &parsedRequest{ + Tags: tags, + EndUser: endUser(b.User), Path: path, Model: b.Model, RequestedModel: b.Model, @@ -680,10 +694,16 @@ func (r *Router) parse(req *http.Request, path string) (*parsedRequest, error) { // see promptBytes. p.PromptTokens = promptBytes(&b) / 4 + // Tags are the gateway's: an engine has no use for them, and its logs + // are no place for what a client wrote there. + if len(b.Metadata) > 0 { + p.UpstreamBody = withoutField(p.UpstreamBody, "metadata") + } + // Ask the engine for usage the client did not request, so accounting works // regardless of client behaviour; the extra chunk is dropped on the way back. if r.cfg.InjectUsage && b.Stream && (b.StreamOptions == nil || !b.StreamOptions.IncludeUsage) { - if injected, ok := injectIncludeUsage(body); ok { + if injected, ok := injectIncludeUsage(p.UpstreamBody); ok { p.UpstreamBody = injected p.SuppressUsageChunk = true } diff --git a/internal/router/router.go b/internal/router/router.go index b94ede70..52ed428d 100644 --- a/internal/router/router.go +++ b/internal/router/router.go @@ -209,6 +209,7 @@ func New(cfg config.Server, log *slog.Logger) (*Router, error) { // guessing at it. metrics := newMetrics() metrics.tenant.level = cfg.Router.TenantLabel + metrics.tenant.setTagLabels(cfg.Router.TagLabels) var prefixScorer *scheduler.PrefixScorer if cfg.Router.Prefix.Enabled { prefixScorer = scheduler.NewPrefixScorer( diff --git a/internal/router/tags.go b/internal/router/tags.go new file mode 100644 index 00000000..481698e3 --- /dev/null +++ b/internal/router/tags.go @@ -0,0 +1,91 @@ +package router + +import ( + "encoding/json" + "fmt" + "regexp" + "strings" +) + +// What a client may say about a request, for attributing spend inside a +// project: OpenAI's metadata (at most 16 keys, as OpenAI allows) and the +// x-llmgw-tags header, which wins for a key both give. +const ( + tagsHeader = "x-llmgw-tags" + maxTags = 16 + maxTagValue = 512 + maxEndUser = 256 +) + +var tagKey = regexp.MustCompile(`^[A-Za-z0-9_.-]{1,64}$`) + +// requestTags reads metadata from the body and x-llmgw-tags ("team=search, +// feature=autocomplete") from the header. An invalid one is the client's +// mistake and is said, not dropped: a tag silently lost is spend that +// cannot be found again. +func requestTags(metadata json.RawMessage, header string) (map[string]string, error) { + tags := map[string]string{} + if len(metadata) > 0 && string(metadata) != "null" { + var m map[string]any + if err := json.Unmarshal(metadata, &m); err != nil { + return nil, fmt.Errorf("metadata is an object of strings") + } + for k, v := range m { + s, ok := v.(string) + if !ok { + return nil, fmt.Errorf("metadata.%s is not a string", k) + } + tags[k] = s + } + } + if header = strings.TrimSpace(header); header != "" { + for _, part := range strings.Split(header, ",") { + k, v, ok := strings.Cut(strings.TrimSpace(part), "=") + if !ok { + return nil, fmt.Errorf("%s: %q is not key=value", tagsHeader, part) + } + tags[strings.TrimSpace(k)] = strings.TrimSpace(v) + } + } + if len(tags) > maxTags { + return nil, fmt.Errorf("%d tags; at most %d", len(tags), maxTags) + } + for k, v := range tags { + if !tagKey.MatchString(k) { + return nil, fmt.Errorf("tag %q: a key is 1 to 64 letters, digits, '_', '.' or '-'", k) + } + if len(v) > maxTagValue { + return nil, fmt.Errorf("tag %s: the value is longer than %d bytes", k, maxTagValue) + } + } + if len(tags) == 0 { + return nil, nil + } + return tags, nil +} + +// endUser is OpenAI's user field as kept: bounded, since it is the client's. +func endUser(u string) string { + if len(u) > maxEndUser { + return u[:maxEndUser] + } + return u +} + +// withoutField removes a top-level field from a JSON body, or returns it +// unchanged when the field is not there or the body is not an object. +func withoutField(body []byte, field string) []byte { + var m map[string]json.RawMessage + if err := json.Unmarshal(body, &m); err != nil { + return body + } + if _, ok := m[field]; !ok { + return body + } + delete(m, field) + out, err := json.Marshal(m) + if err != nil { + return body + } + return out +} diff --git a/internal/router/tags_test.go b/internal/router/tags_test.go new file mode 100644 index 00000000..b551086e --- /dev/null +++ b/internal/router/tags_test.go @@ -0,0 +1,101 @@ +package router + +import ( + "encoding/json" + "strings" + "testing" + "time" + + "github.com/prometheus/client_golang/prometheus/testutil" +) + +func TestTagsComeFromMetadataAndTheHeader(t *testing.T) { + got, err := requestTags(json.RawMessage(`{"feature":"search","team":"a"}`), "team=b, env=prod") + if err != nil { + t.Fatal(err) + } + if len(got) != 3 || got["team"] != "b" || got["feature"] != "search" || got["env"] != "prod" { + t.Errorf("%v", got) + } + if got, err := requestTags(nil, ""); got != nil || err != nil { + t.Errorf("none: %v %v", got, err) + } + many := map[string]string{} + for i := range 17 { + many[strings.Repeat("k", i+1)] = "v" + } + b, _ := json.Marshal(many) + for name, c := range map[string]struct { + meta string + header string + }{ + "not strings": {`{"n": 1}`, ""}, + "not an object": {`["a"]`, ""}, + "no equals": {"", "feature"}, + "bad key": {`{"a b": "x"}`, ""}, + "long value": {`{"a": "` + strings.Repeat("x", 513) + `"}`, ""}, + "too many": {string(b), ""}, + } { + if _, err := requestTags(json.RawMessage(c.meta), c.header); err == nil { + t.Errorf("%s: accepted", name) + } + } +} + +func TestTagsAreRecordedAndNotSentToTheEngine(t *testing.T) { + e := newFakeEngine(t, "qwen2.5-14b-instruct") + rt, secret := pricedRouter(t, e.srv.URL) + rt.m.tenant.setTagLabels([]string{"feature"}) + + rec := post(t, rt.Handler(), + map[string]any{"model": "q14", "user": "customer-42", "metadata": map[string]string{"feature": "search"}, + "messages": []any{map[string]any{"role": "user", "content": "hello"}}}, + map[string]string{"Authorization": "Bearer " + secret, "X-Request-Id": "tagged-1", "X-Llmgw-Tags": "env=prod"}) + if rec.Code != 200 { + t.Fatalf("status %d: %s", rec.Code, rec.Body.String()) + } + sent, _ := e.last.Load().([]byte) + if strings.Contains(string(sent), "metadata") || strings.Contains(string(sent), "search") { + t.Errorf("the engine was sent the tags: %s", sent) + } + if !strings.Contains(string(sent), "customer-42") { + t.Errorf("the user field, which engines know, was not passed on: %s", sent) + } + + deadline := time.Now().Add(5 * time.Second) + for { + u, ok, err := rt.store.UsageByRequest(t.Context(), "tagged-1") + if err != nil { + t.Fatal(err) + } + if ok { + if u.EndUser != "customer-42" || u.Tags["feature"] != "search" || u.Tags["env"] != "prod" { + t.Errorf("recorded end user %q tags %v", u.EndUser, u.Tags) + } + break + } + if time.Now().After(deadline) { + t.Fatal("no usage recorded") + } + time.Sleep(20 * time.Millisecond) + } + + bad := post(t, rt.Handler(), + map[string]any{"model": "q14", "metadata": map[string]any{"n": 1}, + "messages": []any{map[string]any{"role": "user", "content": "hello"}}}, + map[string]string{"Authorization": "Bearer " + secret}) + if bad.Code != 400 || !strings.Contains(bad.Body.String(), "metadata.n") { + t.Errorf("a bad tag: %d %s", bad.Code, bad.Body.String()) + } +} + +func TestOnlyNamedTagsAreCountedAndTheirValuesBounded(t *testing.T) { + m := newTenantMetrics() + m.setTagLabels([]string{"feature"}) + for i := range maxTagValues + 20 { + m.observe(&parsedRequest{Tags: map[string]string{"feature": strings.Repeat("f", i+1), "secret": "x"}}, 200, 1, 1, 0) + } + if n := testutil.CollectAndCount(m.tagRequests); n != maxTagValues+1 { + t.Errorf("%d series; want %d values and other", testutil.CollectAndCount(m.tagRequests), maxTagValues+1) + } +} diff --git a/internal/router/tenantmetrics.go b/internal/router/tenantmetrics.go index 7202b7b2..8f855a28 100644 --- a/internal/router/tenantmetrics.go +++ b/internal/router/tenantmetrics.go @@ -28,6 +28,12 @@ type tenantMetrics struct { ttft *prometheus.HistogramVec budget *prometheus.GaugeVec + // Request tags, for the keys router.tag_labels names. + tagKeys map[string]bool + tagRequests *prometheus.CounterVec + tagTokens *prometheus.CounterVec + tagValues map[string]map[string]bool // tag -> values counted apart + mu sync.RWMutex names map[string]string // "team:" -> "acme/platform" } @@ -52,12 +58,68 @@ func newTenantMetrics() *tenantMetrics { Name: "llmgw_budget_spent_ratio", Help: "Spend this period over the budget, for every organization, team, project and person given one. 1 is spent.", }, []string{"scope", "id", "name", "period"}), - names: map[string]string{}, + tagRequests: prometheus.NewCounterVec(prometheus.CounterOpts{ + Name: "llmgw_tagged_requests_total", + Help: "Requests by a request tag router.tag_labels names, its value, and outcome: ok, error, or limited.", + }, []string{"tag", "value", "outcome"}), + tagTokens: prometheus.NewCounterVec(prometheus.CounterOpts{ + Name: "llmgw_tagged_tokens_total", + Help: "Tokens by a request tag router.tag_labels names, its value, and direction.", + }, []string{"tag", "value", "direction"}), + tagValues: map[string]map[string]bool{}, + names: map[string]string{}, + } +} + +// maxTagValues is how many values of one tag are counted apart. +const maxTagValues = 100 + +// setTagLabels names the tags counted. +func (t *tenantMetrics) setTagLabels(keys []string) { + t.tagKeys = map[string]bool{} + for _, k := range keys { + t.tagKeys[k] = true + } +} + +// tagValue is the label a tag's value is counted under: itself while there +// is room, "other" after. +func (t *tenantMetrics) tagValue(tag, value string) string { + t.mu.Lock() + defer t.mu.Unlock() + seen := t.tagValues[tag] + if seen == nil { + seen = map[string]bool{} + t.tagValues[tag] = seen + } + if seen[value] { + return value + } + if len(seen) >= maxTagValues { + return "other" + } + seen[value] = true + return value +} + +func (t *tenantMetrics) observeTags(req *parsedRequest, outcome string, prompt, completion int64) { + for k, v := range req.Tags { + if !t.tagKeys[k] { + continue + } + v = t.tagValue(k, v) + t.tagRequests.WithLabelValues(k, v, outcome).Inc() + if prompt > 0 { + t.tagTokens.WithLabelValues(k, v, "prompt").Add(float64(prompt)) + } + if completion > 0 { + t.tagTokens.WithLabelValues(k, v, "completion").Add(float64(completion)) + } } } func (t *tenantMetrics) collectors() []prometheus.Collector { - return []prometheus.Collector{t.requests, t.tokens, t.ttft, t.budget} + return []prometheus.Collector{t.requests, t.tokens, t.ttft, t.budget, t.tagRequests, t.tagTokens} } // tenantOf is the label a request is counted under: the name of its @@ -91,10 +153,6 @@ func (t *tenantMetrics) tenantOf(req *parsedRequest) string { // observe counts one finished or refused request. func (t *tenantMetrics) observe(req *parsedRequest, status int, prompt, completion int64, ttft time.Duration) { - tenant := t.tenantOf(req) - if tenant == "" { - return - } outcome := "ok" switch { case status == 429: @@ -102,6 +160,13 @@ func (t *tenantMetrics) observe(req *parsedRequest, status int, prompt, completi case status >= 400: outcome = "error" } + if len(t.tagKeys) > 0 { + t.observeTags(req, outcome, prompt, completion) + } + tenant := t.tenantOf(req) + if tenant == "" { + return + } t.requests.WithLabelValues(tenant, outcome).Inc() if prompt > 0 { t.tokens.WithLabelValues(tenant, "prompt").Add(float64(prompt)) diff --git a/internal/store/migrations/postgres/0023_usage_tags.sql b/internal/store/migrations/postgres/0023_usage_tags.sql new file mode 100644 index 00000000..433ca5ad --- /dev/null +++ b/internal/store/migrations/postgres/0023_usage_tags.sql @@ -0,0 +1,7 @@ +-- Ported from the SQLite migration of the same number. +-- What a client said about a request: OpenAI's user field (end_user, the +-- caller's own name for whoever it serves) and its metadata or +-- x-llmgw-tags (tags, a JSON object of strings, keys sorted). For +-- attributing spend inside a project; never the text. +ALTER TABLE usage ADD COLUMN end_user TEXT NOT NULL DEFAULT ''; +ALTER TABLE usage ADD COLUMN tags TEXT NOT NULL DEFAULT ''; diff --git a/internal/store/migrations/sqlite/0023_usage_tags.sql b/internal/store/migrations/sqlite/0023_usage_tags.sql new file mode 100644 index 00000000..b7a822ad --- /dev/null +++ b/internal/store/migrations/sqlite/0023_usage_tags.sql @@ -0,0 +1,6 @@ +-- What a client said about a request: OpenAI's user field (end_user, the +-- caller's own name for whoever it serves) and its metadata or +-- x-llmgw-tags (tags, a JSON object of strings, keys sorted). For +-- attributing spend inside a project; never the text. +ALTER TABLE usage ADD COLUMN end_user TEXT NOT NULL DEFAULT ''; +ALTER TABLE usage ADD COLUMN tags TEXT NOT NULL DEFAULT ''; diff --git a/internal/store/sqlite.go b/internal/store/sqlite.go index 4da30c75..85b2ed55 100644 --- a/internal/store/sqlite.go +++ b/internal/store/sqlite.go @@ -430,8 +430,8 @@ func (s *DB) WriteUsage(ctx context.Context, records []Usage) error { `INSERT INTO usage (ts, key_id, project_id, model, served_model, prompt_tokens, completion_tokens, cost_micros, status, latency_ms, user_id, request_id, replica, queue_ms, ttft_ms, - guard_action, guard_policy, guard_category, guard_ms) - VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`) + guard_action, guard_policy, guard_category, guard_ms, end_user, tags) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`) if err != nil { return err } @@ -441,7 +441,7 @@ func (s *DB) WriteUsage(ctx context.Context, records []Usage) error { r.Model, r.ServedModel, r.PromptTokens, r.CompletionTokens, r.CostMicros, r.Status, r.LatencyMs, r.UserID, r.RequestID, r.Replica, r.QueueMs, r.TTFTMs, - r.GuardAction, r.GuardPolicy, r.GuardCategory, r.GuardMs); err != nil { + r.GuardAction, r.GuardPolicy, r.GuardCategory, r.GuardMs, r.EndUser, tagsJSON(r.Tags)); err != nil { return err } } @@ -850,14 +850,16 @@ func (s *DB) SnapshotTo(ctx context.Context, path string) error { func (s *DB) UsageByRequest(ctx context.Context, requestID string) (Usage, bool, error) { var u Usage var ts int64 + var tags string err := s.db.QueryRowContext(ctx, `SELECT ts, key_id, project_id, COALESCE(user_id, ''), model, served_model, prompt_tokens, completion_tokens, cost_micros, status, latency_ms, request_id, replica, queue_ms, ttft_ms, - guard_action, guard_policy, guard_category, guard_ms + guard_action, guard_policy, guard_category, guard_ms, end_user, tags FROM usage WHERE request_id = ? ORDER BY ts DESC LIMIT 1`, requestID). Scan(&ts, &u.KeyID, &u.ProjectID, &u.UserID, &u.Model, &u.ServedModel, &u.PromptTokens, &u.CompletionTokens, &u.CostMicros, &u.Status, &u.LatencyMs, &u.RequestID, &u.Replica, - &u.QueueMs, &u.TTFTMs, &u.GuardAction, &u.GuardPolicy, &u.GuardCategory, &u.GuardMs) + &u.QueueMs, &u.TTFTMs, &u.GuardAction, &u.GuardPolicy, &u.GuardCategory, &u.GuardMs, + &u.EndUser, &tags) if errors.Is(err, sql.ErrNoRows) { return Usage{}, false, nil } @@ -865,5 +867,6 @@ func (s *DB) UsageByRequest(ctx context.Context, requestID string) (Usage, bool, return Usage{}, false, err } u.At = time.UnixMilli(ts) + u.Tags = parseTags(tags) return u, true, nil } diff --git a/internal/store/store.go b/internal/store/store.go index 5183a25a..d1d338db 100644 --- a/internal/store/store.go +++ b/internal/store/store.go @@ -16,6 +16,7 @@ package store import ( "context" + "encoding/json" "errors" "strings" "time" @@ -262,6 +263,33 @@ type Usage struct { GuardPolicy string GuardCategory string GuardMs int64 + + // EndUser is OpenAI's user field as the client sent it: its own name + // for whoever it serves, not an account here. Tags are its metadata + // and x-llmgw-tags. Both for attributing spend inside a project. + EndUser string + Tags map[string]string +} + +// tagsJSON is how tags are kept: a JSON object, keys sorted, or empty. +func tagsJSON(t map[string]string) string { + if len(t) == 0 { + return "" + } + b, _ := json.Marshal(t) // encoding/json sorts map keys + return string(b) +} + +// parseTags reads tagsJSON's form back. +func parseTags(s string) map[string]string { + if s == "" { + return nil + } + var t map[string]string + if json.Unmarshal([]byte(s), &t) != nil { + return nil + } + return t } // Store is the persistent half of tenancy and accounting. From 0a4076928a95b57b2532465bad2d47f0949cfc99 Mon Sep 17 00:00:00 2001 From: Ilker Manap Date: Sun, 4 Oct 2026 19:56:16 +0200 Subject: [PATCH 02/38] Canary routing: a model's canary takes its share of requests, a session stays on one side, x-llmgw-variant pins one, and the two are compared Co-Authored-By: Claude Opus 5.5 (1M context) --- internal/canary/canary.go | 79 ++++++++++++++++++ internal/canary/canary_test.go | 42 ++++++++++ internal/control/catalog/catalog.go | 74 +++++++++++++++++ internal/router/canary.go | 124 ++++++++++++++++++++++++++++ internal/router/canary_test.go | 82 ++++++++++++++++++ internal/router/handlers.go | 12 +-- internal/router/router.go | 52 +++++++++++- 7 files changed, 459 insertions(+), 6 deletions(-) create mode 100644 internal/canary/canary.go create mode 100644 internal/canary/canary_test.go create mode 100644 internal/router/canary.go create mode 100644 internal/router/canary_test.go diff --git a/internal/canary/canary.go b/internal/canary/canary.go new file mode 100644 index 00000000..bc16bb37 --- /dev/null +++ b/internal/canary/canary.go @@ -0,0 +1,79 @@ +// Package canary compares a model's candidate with the model it may +// replace, from what each side served, and says when to give up on it. +package canary + +import ( + "fmt" + "time" + + "gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog" +) + +// Side is what one side of a canary served since it began. +type Side struct { + Requests int64 `json:"requests"` + // Errors are answers of 500 and up: the engine's failures, not the + // client's mistakes. + Errors int64 `json:"errors"` + // TTFT is over streamed requests, Latency over the rest, both means in + // milliseconds with how many they are over. + TTFTMs float64 `json:"ttft_ms"` + TTFTCount int64 `json:"ttft_count"` + LatencyMs float64 `json:"latency_ms"` + LatencyCount int64 `json:"latency_count"` +} + +// ErrorRate is errors over requests, 0 with none. +func (v Side) ErrorRate() float64 { + if v.Requests == 0 { + return 0 + } + return float64(v.Errors) / float64(v.Requests) +} + +// Add counts one finished request. +func (v *Side) Add(status int, streamed bool, ttft, took time.Duration) { + v.Requests++ + if status >= 500 { + v.Errors++ + } + switch { + case streamed && ttft > 0: + v.TTFTMs += (float64(ttft.Milliseconds()) - v.TTFTMs) / float64(v.TTFTCount+1) + v.TTFTCount++ + case !streamed && status < 400: + v.LatencyMs += (float64(took.Milliseconds()) - v.LatencyMs) / float64(v.LatencyCount+1) + v.LatencyCount++ + } +} + +// Stat compares the two sides of one model's canary. +type Stat struct { + Canary string `json:"canary"` + Since time.Time `json:"since"` + Stable Side `json:"stable"` + Candidate Side `json:"candidate"` +} + +// minTimed is how many timed requests a side needs before its mean counts. +const minTimed = 10 + +// Verdict says whether the candidate has done worse than the canary allows, +// and how. Nothing is decided before it has served MinRequests. +func Verdict(s Stat, cn catalog.Canary) (rollback bool, why string) { + minReq, maxErr, maxRatio := cn.Limits() + c, st := s.Candidate, s.Stable + if c.Requests < int64(minReq) { + return false, "" + } + if d := c.ErrorRate() - st.ErrorRate(); d > maxErr { + return true, fmt.Sprintf("errors in %.1f%% of its requests against %.1f%%", 100*c.ErrorRate(), 100*st.ErrorRate()) + } + if c.TTFTCount >= minTimed && st.TTFTCount >= minTimed && st.TTFTMs > 0 && c.TTFTMs > maxRatio*st.TTFTMs { + return true, fmt.Sprintf("first token after %.0f ms on average against %.0f ms", c.TTFTMs, st.TTFTMs) + } + if c.LatencyCount >= minTimed && st.LatencyCount >= minTimed && st.LatencyMs > 0 && c.LatencyMs > maxRatio*st.LatencyMs { + return true, fmt.Sprintf("answers in %.0f ms on average against %.0f ms", c.LatencyMs, st.LatencyMs) + } + return false, "" +} diff --git a/internal/canary/canary_test.go b/internal/canary/canary_test.go new file mode 100644 index 00000000..f9008dc3 --- /dev/null +++ b/internal/canary/canary_test.go @@ -0,0 +1,42 @@ +package canary + +import ( + "testing" + "time" + + "gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog" +) + +func side(n, errs int, ms time.Duration, streamed bool) Side { + var s Side + for i := range n { + status := 200 + if i < errs { + status = 500 + } + s.Add(status, streamed, ms, ms) + } + return s +} + +func TestTheVerdict(t *testing.T) { + cn := catalog.Canary{Model: "c", Percent: 10} + for name, c := range map[string]struct { + stable, cand Side + rollback bool + }{ + "too few to say": {side(500, 0, 100*time.Millisecond, true), side(49, 49, 100*time.Millisecond, true), false}, + "as good": {side(500, 5, 100*time.Millisecond, true), side(60, 1, 110*time.Millisecond, true), false}, + "more errors": {side(500, 5, 100*time.Millisecond, true), side(60, 6, 100*time.Millisecond, true), true}, + "slower first token": {side(500, 0, 100*time.Millisecond, true), side(60, 0, 200*time.Millisecond, true), true}, + "slower answers": {side(500, 0, time.Second, false), side(60, 0, 2*time.Second, false), true}, + } { + got, why := Verdict(Stat{Stable: c.stable, Candidate: c.cand}, cn) + if got != c.rollback { + t.Errorf("%s: rollback=%v (%s)", name, got, why) + } + if got && why == "" { + t.Errorf("%s: no reason given", name) + } + } +} diff --git a/internal/control/catalog/catalog.go b/internal/control/catalog/catalog.go index 9564fc50..9f4429c1 100644 --- a/internal/control/catalog/catalog.go +++ b/internal/control/catalog/catalog.go @@ -175,6 +175,10 @@ type Model struct { // every adapter, applying one only to a request that names it. Adapters []Adapter `json:"adapters,omitempty" koanf:"adapters"` Fallbacks []string `json:"fallbacks,omitempty" koanf:"fallbacks"` + // Canary sends a share of this model's requests to another catalog + // model — the same model at new weights — to compare the two before + // moving to it (`llmgw ctl model canary`). + Canary *Canary `json:"canary,omitempty" koanf:"canary"` // RoutingProfile selects how the router scores endpoints for this model. RoutingProfile string `json:"routing_profile,omitempty" koanf:"routing_profile"` SLO SLO `json:"slo,omitempty" koanf:"slo"` @@ -347,9 +351,79 @@ func New(models []Model) (*Catalog, error) { } } } + // A canary is another model in the catalog, a candidate for this one's + // place: not itself, not a judge, not a canary of its own, and the + // candidate of one model only. + canaryOf := map[string]string{} + for _, name := range c.order { + cn := c.models[name].Canary + if cn == nil { + continue + } + target, ok := c.Resolve(cn.Model) + switch { + case !ok: + return nil, fmt.Errorf("model %q: canary %q is not in the catalog", name, cn.Model) + case target.Name == name: + return nil, fmt.Errorf("model %q: a model cannot be its own canary", name) + case target.IsGuard(): + return nil, fmt.Errorf("model %q: canary %q is a guard model", name, cn.Model) + case target.Canary != nil: + return nil, fmt.Errorf("model %q: canary %q has a canary of its own", name, cn.Model) + case canaryOf[target.Name] != "": + return nil, fmt.Errorf("model %q: %q is already the canary of %q", name, cn.Model, canaryOf[target.Name]) + case cn.Percent < 0 || cn.Percent > 100: + return nil, fmt.Errorf("model %q: canary percent %d is not 0 to 100", name, cn.Percent) + } + canaryOf[target.Name] = name + } return c, nil } +// Canary is a share of a model's requests sent to a candidate, and when to +// give up on it. +type Canary struct { + // Model is the candidate: a catalog model, usually this one at newer + // weights, which clients cannot name themselves. + Model string `json:"model" koanf:"model"` + // Percent of requests go to it, 0 to 100. + Percent int `json:"percent" koanf:"percent"` + // The candidate is taken out — rolled back — when, over at least + // MinRequests of its own, its error rate is MaxErrorIncrease above + // this model's (0.05 is five points), or its mean time to the first + // token is MaxTTFTRatio times this model's. Zero is 50, 0.05 and 1.5. + MinRequests int `json:"min_requests,omitempty" koanf:"min_requests"` + MaxErrorIncrease float64 `json:"max_error_increase,omitempty" koanf:"max_error_increase"` + MaxTTFTRatio float64 `json:"max_ttft_ratio,omitempty" koanf:"max_ttft_ratio"` +} + +// Limits are the rollback thresholds with their defaults filled in. +func (c Canary) Limits() (minRequests int, maxErrorIncrease, maxTTFTRatio float64) { + minRequests, maxErrorIncrease, maxTTFTRatio = c.MinRequests, c.MaxErrorIncrease, c.MaxTTFTRatio + if minRequests <= 0 { + minRequests = 50 + } + if maxErrorIncrease <= 0 { + maxErrorIncrease = 0.05 + } + if maxTTFTRatio <= 0 { + maxTTFTRatio = 1.5 + } + return +} + +// CanaryOf is the model whose canary name is, or "". +func (c *Catalog) CanaryOf(name string) string { + for _, n := range c.order { + if cn := c.models[n].Canary; cn != nil { + if t, ok := c.Resolve(cn.Model); ok && t.Name == name { + return n + } + } + } + return "" +} + func (m Model) normalize() Model { if m.Scaling.Min < 0 { m.Scaling.Min = 0 diff --git a/internal/router/canary.go b/internal/router/canary.go new file mode 100644 index 00000000..99f1a6f4 --- /dev/null +++ b/internal/router/canary.go @@ -0,0 +1,124 @@ +package router + +import ( + "hash/fnv" + "net/http" + "strings" + "sync" + "time" + + "gitea.mynodes.xyz/ilker/llmgw/internal/canary" + "gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog" +) + +// A canary: a share of a model's requests sent to a candidate, usually the +// same model at newer weights, and the two compared as they are served. + +// variantHeader pins a request to one side, for testing either on purpose: +// "canary" or "stable". Either is the model the key asked for, so pinning +// grants nothing the key did not have. +const variantHeader = "x-llmgw-variant" + +// chooseVariant says whether a request goes to the canary. A session — the +// x-session-id header or OpenAI's user field — always lands on the same +// side, so one conversation does not hop between weights; a request without +// one is placed by its own id. +func chooseVariant(req *parsedRequest, cn *catalog.Canary) bool { + switch strings.ToLower(req.Header.Get(variantHeader)) { + case "canary": + return true + case "stable": + return false + } + if cn.Percent <= 0 { + return false + } + if cn.Percent >= 100 { + return true + } + key := req.SessionID + if key == "" { + key = req.RequestID + } + h := fnv.New32a() + h.Write([]byte(key)) + return int(h.Sum32()%100) < cn.Percent +} + +type canaryStats struct { + mu sync.Mutex + by map[string]*canary.Stat // by the model the canary is of +} + +func (c *canaryStats) observe(parent, cand string, onCanary bool, status int, streamed bool, ttft, took time.Duration) { + c.mu.Lock() + defer c.mu.Unlock() + if c.by == nil { + c.by = map[string]*canary.Stat{} + } + s := c.by[parent] + if s == nil || s.Canary != cand { + // A new candidate starts from nothing: the last one's numbers are + // not this one's. + s = &canary.Stat{Canary: cand, Since: time.Now()} + c.by[parent] = s + } + if onCanary { + s.Candidate.Add(status, streamed, ttft, took) + } else { + s.Stable.Add(status, streamed, ttft, took) + } +} + +// CanaryStats is how each model's canary and its stable side have served +// since the canary began, kept by this router. +func (r *Router) CanaryStats() map[string]canary.Stat { + r.canaries.mu.Lock() + defer r.canaries.mu.Unlock() + out := map[string]canary.Stat{} + for k, v := range r.canaries.by { + out[k] = *v + } + return out +} + +// ResetCanary forgets what a model's canary served, for a canary that has +// been taken out or promoted. +func (r *Router) ResetCanary(model string) { + r.canaries.mu.Lock() + defer r.canaries.mu.Unlock() + delete(r.canaries.by, model) +} + +// observeCanary counts a finished request against its side of a canary. +func (r *Router) observeCanary(req *parsedRequest, res dispatchResult, status int, took time.Duration) { + if res.canaryOf == "" || status == 499 { + return + } + r.canaries.observe(res.canaryOf, res.canary, res.onCanary, status, req.Stream, res.outcome.TTFT, took) +} + +// isCanaryTarget reports whether a name is some model's canary: reached +// through that model, never by its own name. +func (r *Router) isCanaryTarget(name string) bool { + canonical, _ := r.resolve(name) + if p := r.canaryTargets.Load(); p != nil { + _, ok := (*p)[canonical] + return ok + } + return false +} + +// hiddenModel is a model clients cannot name: a judge, or a canary. +func (r *Router) hiddenModel(name string) bool { + return r.isGuardModel(name) || r.isCanaryTarget(name) +} + +// setVariantHeader tells the client which side served it. +func setVariantHeader(h http.Header, onCanary bool) { + if onCanary { + h.Set(variantHeader, "canary") + } else { + h.Set(variantHeader, "stable") + } +} diff --git a/internal/router/canary_test.go b/internal/router/canary_test.go new file mode 100644 index 00000000..a4bb51f2 --- /dev/null +++ b/internal/router/canary_test.go @@ -0,0 +1,82 @@ +package router + +import ( + "fmt" + "net/http" + "strings" + "testing" + + "gitea.mynodes.xyz/ilker/llmgw/internal/config" + "gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog" +) + +func canaryRouter(t *testing.T, percent int) (*Router, *fakeEngine, *fakeEngine) { + t.Helper() + stable := newFakeEngine(t, "qwen") + cand := newFakeEngine(t, "qwen-canary") + cfg := baseConfig( + config.Endpoint{ID: "s", URL: stable.srv.URL, Models: []string{"qwen"}}, + config.Endpoint{ID: "c", URL: cand.srv.URL, Models: []string{"qwen-canary"}}, + ) + m := catModel("qwen", nil) + m.Canary = &catalog.Canary{Model: "qwen-canary", Percent: percent} + cfg.Models = []catalog.Model{m, catModel("qwen-canary", nil)} + return newTestRouter(t, cfg), stable, cand +} + +func TestACanaryTakesItsShareAndIsComparedWithTheRest(t *testing.T) { + rt, stable, cand := canaryRouter(t, 30) + for i := range 400 { + rec := post(t, rt.Handler(), map[string]any{"model": "qwen", "messages": []any{}}, + map[string]string{"X-Request-Id": fmt.Sprintf("r-%d", i)}) + if rec.Code != http.StatusOK { + t.Fatalf("status %d: %s", rec.Code, rec.Body.String()) + } + } + c, s := cand.calls.Load(), stable.calls.Load() + if c < 80 || c > 160 || c+s != 400 { + t.Errorf("canary %d, stable %d of 400 at 30%%", c, s) + } + if b, _ := cand.last.Load().([]byte); !strings.Contains(string(b), `"qwen-canary"`) { + t.Errorf("the candidate was asked for another name: %s", b) + } + st := rt.CanaryStats()["qwen"] + if st.Canary != "qwen-canary" || st.Candidate.Requests != c || st.Stable.Requests != s { + t.Errorf("stats %+v", st) + } +} + +func TestASessionStaysOnOneSideAndAHeaderPinsIt(t *testing.T) { + rt, _, cand := canaryRouter(t, 50) + for range 20 { + post(t, rt.Handler(), map[string]any{"model": "qwen", "user": "alice", "messages": []any{}}, nil) + } + if n := cand.calls.Load(); n != 0 && n != 20 { + t.Errorf("one session went to the canary %d times of 20", n) + } + before := cand.calls.Load() + rec := post(t, rt.Handler(), map[string]any{"model": "qwen", "user": "alice", "messages": []any{}}, + map[string]string{"X-Llmgw-Variant": "canary"}) + if cand.calls.Load() != before+1 || rec.Header().Get("x-llmgw-variant") != "canary" { + t.Errorf("the pin was not followed: %q", rec.Header().Get("x-llmgw-variant")) + } +} + +func TestACanaryCannotBeNamedAndFallsBackToStable(t *testing.T) { + rt, _, _ := canaryRouter(t, 100) + if rec := post(t, rt.Handler(), map[string]any{"model": "qwen-canary", "messages": []any{}}, nil); rec.Code != http.StatusNotFound { + t.Errorf("a canary named directly: %d", rec.Code) + } + + // With nothing ready on the canary side, the stable side serves. + stable := newFakeEngine(t, "qwen") + cfg := baseConfig(config.Endpoint{ID: "s", URL: stable.srv.URL, Models: []string{"qwen"}}) + m := catModel("qwen", nil) + m.Canary = &catalog.Canary{Model: "qwen-canary", Percent: 100} + cfg.Models = []catalog.Model{m, catModel("qwen-canary", nil)} + rt = newTestRouter(t, cfg) + rec := post(t, rt.Handler(), map[string]any{"model": "qwen", "messages": []any{}}, nil) + if rec.Code != http.StatusOK || stable.calls.Load() != 1 || rec.Header().Get("x-llmgw-variant") != "stable" { + t.Errorf("status %d, stable calls %d, variant %q", rec.Code, stable.calls.Load(), rec.Header().Get("x-llmgw-variant")) + } +} diff --git a/internal/router/handlers.go b/internal/router/handlers.go index f18a05c0..7cc1a4c8 100644 --- a/internal/router/handlers.go +++ b/internal/router/handlers.go @@ -396,7 +396,7 @@ func (r *Router) handleInference(path string) http.HandlerFunc { } } - if r.isGuardModel(parsed.Model) { + if r.hiddenModel(parsed.Model) { // Said as a model that is not there: it is not, for clients. writeErrorCode(w, http.StatusNotFound, "invalid_request_error", "model_not_found", fmt.Sprintf("the model %q does not exist", parsed.Model)) @@ -491,6 +491,7 @@ func (r *Router) handleInference(path string) http.HandlerFunc { res := r.dispatch(req.Context(), w, parsed) status := r.finish(w, parsed, res, start) + r.observeCanary(parsed, res, status, time.Since(start)) r.observeOutput(parsed, res, res.judged) span.SetAttributes(attribute.Int("http.response.status_code", status), attribute.String("llmgw.served_model", res.model), @@ -516,8 +517,9 @@ func (r *Router) handleInference(path string) http.HandlerFunc { // around the allowlist; matching only the canonical name would silently ignore // an allowlist written in the names clients actually use. func (r *Router) permits(p auth.Principal, requested string) bool { - // A judge model is the guardrails', not anybody's to ask. - if r.isGuardModel(requested) { + // A judge model is the guardrails', not anybody's to ask; a canary is + // reached through the model it is a canary of. + if r.hiddenModel(requested) { return false } // An adapter named in the allowlist allows that adapter: not its model, @@ -852,7 +854,7 @@ func noEndpointFor(model string, served []string) string { // handing it a simulated model would send someone's first request to an engine // that replies in Greek letters. func (r *Router) exampleModels() []string { - names := slices.DeleteFunc(r.servedModels(), r.isGuardModel) + names := slices.DeleteFunc(r.servedModels(), r.hiddenModel) slices.SortStableFunc(names, func(a, b string) int { as, bs := r.modelMap()[a].Simulated(), r.modelMap()[b].Simulated() switch { @@ -910,7 +912,7 @@ func (r *Router) handleModels(w http.ResponseWriter, req *http.Request) { principal, authed := auth.FromContext(req.Context()) names := make([]string, 0, len(seen)) for n := range seen { - if authed && !r.permits(principal, n) || r.isGuardModel(n) { + if authed && !r.permits(principal, n) || r.hiddenModel(n) { continue } names = append(names, n) diff --git a/internal/router/router.go b/internal/router/router.go index 52ed428d..77fa7ac1 100644 --- a/internal/router/router.go +++ b/internal/router/router.go @@ -64,6 +64,10 @@ type Router struct { models atomic.Pointer[map[string]catalog.Model] // adapters are the LoRA adapters by the name clients ask for them by. adapters atomic.Pointer[map[string]adapterRef] + // canaryTargets maps each model that is another's canary to that one; + // canaries counts how the two sides served. + canaryTargets atomic.Pointer[map[string]string] + canaries canaryStats // extraMetrics are collectors other components publish on /metrics. extraMetrics []prometheus.Collector @@ -635,6 +639,15 @@ func (r *Router) storeModels(models []catalog.Model) { } r.models.Store(&m) r.adapters.Store(&ads) + targets := map[string]string{} + for _, entry := range models { + if entry.Canary != nil { + if t, ok := m[entry.Canary.Model]; ok { + targets[t.Name] = entry.Name + } + } + } + r.canaryTargets.Store(&targets) } // adapterRef is where an adapter runs and how it is applied. @@ -732,6 +745,10 @@ type dispatchResult struct { // judged whether there was. guard guard.Verdict judged bool + // canaryOf is the model whose canary this request was split by, canary + // the candidate, and onCanary whether the candidate served it. + canaryOf, canary string + onCanary bool } type servedModel struct { @@ -749,6 +766,39 @@ type servedModel struct { func (r *Router) dispatch(ctx context.Context, w http.ResponseWriter, req *parsedRequest) dispatchResult { canonical, entry := r.resolve(req.Model) + // A canary takes its share before anything else, and falls back to the + // stable side, not to a fallback, when it has nothing ready. + // Not an adapter: the candidate's replicas do not run it under the + // stable model's name. + _, isAdapter := r.adapter(req.Model) + if cn := entry.Canary; cn != nil && !isAdapter { + onCanary := chooseVariant(req, cn) + target, _ := r.resolve(cn.Model) + if onCanary && len(r.reg.Snapshots(target)) == 0 { + onCanary = false + } + setVariantHeader(w.Header(), onCanary) + if onCanary { + // The engine is asked for the model by the name it serves. + creq := *req + creq.UpstreamBody = withModel(req.UpstreamBody, target) + res := r.dispatchOne(ctx, w, &creq, target, entry) + res.model, res.canaryOf, res.canary, res.onCanary = target, canonical, target, true + if res.err == nil || res.outcome.Wrote || errors.Is(res.err, proxy.ErrClientGone) || + !errors.Is(res.err, errNoCapacity) { + return res + } + setVariantHeader(w.Header(), false) + } + res := r.dispatchStable(ctx, w, req, canonical, entry) + res.canaryOf, res.canary = canonical, target + return res + } + return r.dispatchStable(ctx, w, req, canonical, entry) +} + +// dispatchStable serves a model as asked, then its fallbacks. +func (r *Router) dispatchStable(ctx context.Context, w http.ResponseWriter, req *parsedRequest, canonical string, entry catalog.Model) dispatchResult { // The requested model first, then its fallbacks. A fallback is a different // model, so the client is told about it via x-llmgw-served-model. chain := []string{canonical} @@ -824,7 +874,7 @@ func (r *Router) dispatchOne(ctx context.Context, w http.ResponseWriter, req *pa return dispatchResult{err: fmt.Errorf("%w: %s", errLoading, "the model is loading; retry shortly")} } - return dispatchResult{err: fmt.Errorf("%w: %s", errNoCapacity, noEndpointFor(model, slices.DeleteFunc(r.servedModels(), r.isGuardModel)))} + return dispatchResult{err: fmt.Errorf("%w: %s", errNoCapacity, noEndpointFor(model, slices.DeleteFunc(r.servedModels(), r.hiddenModel)))} } profile := r.profileFor(entry, req.ProfileOverride) From a1dfd4df1dcc4eee1253f2f66ff3b6be3af5f97f Mon Sep 17 00:00:00 2001 From: Ilker Manap Date: Sun, 4 Oct 2026 20:31:26 +0200 Subject: [PATCH 03/38] Canary of a model's newer weights: ctl model canary, promote and canaries, rolled back by itself when it serves worse Co-Authored-By: Claude Opus 5.5 (1M context) --- internal/cli/ctl.go | 9 +- internal/cli/model.go | 3 +- internal/cli/model_canary.go | 148 ++++++++++ internal/cli/server.go | 3 + internal/control/adminapi/api.go | 6 + internal/control/adminapi/canary.go | 84 ++++++ internal/control/modelserve/canary.go | 304 ++++++++++++++++++++ internal/control/modelserve/canary_test.go | 101 +++++++ internal/control/modelserve/modelserve.go | 10 +- internal/control/modelserve/updates.go | 53 +++- internal/control/modelserve/updates_test.go | 25 ++ internal/control/modelsvc/modelsvc.go | 17 ++ internal/notify/notify.go | 5 +- 13 files changed, 750 insertions(+), 18 deletions(-) create mode 100644 internal/cli/model_canary.go create mode 100644 internal/control/adminapi/canary.go create mode 100644 internal/control/modelserve/canary.go create mode 100644 internal/control/modelserve/canary_test.go diff --git a/internal/cli/ctl.go b/internal/cli/ctl.go index def0aed2..4cf0b822 100644 --- a/internal/cli/ctl.go +++ b/internal/cli/ctl.go @@ -351,8 +351,15 @@ func newModelsCmd(client func() *ctlClient, asJSON *bool) *cobra.Command { if m.Ready > 0 { serving = "yes" } + name := m.Name + if m.Canary != nil { + name += fmt.Sprintf(" (%d%% to canary)", m.Canary.Percent) + } + if m.CanaryOf != "" { + name += " (canary of " + m.CanaryOf + ")" + } fmt.Fprintf(w, "%s\t%s\t%d/%d\t%s\t%s\t%s\t%s\t%s\t%s\n", - m.Name, serving, m.Ready, m.Min, modelStatus(m), m.Class, reserves(m), + name, serving, m.Ready, m.Min, modelStatus(m), m.Class, reserves(m), join(m.Aliases), join(m.Profiles), join(m.Fallbacks)) for _, x := range m.Warnings { warnings = append(warnings, m.Name+": "+x) diff --git a/internal/cli/model.go b/internal/cli/model.go index 66c6d283..2a19bffb 100644 --- a/internal/cli/model.go +++ b/internal/cli/model.go @@ -36,7 +36,8 @@ func newModelCmd(client func() *ctlClient, asJSON *bool) *cobra.Command { newResourcesCmd(client), newClassCmd(client), newOwnerCmd(client), newTargetCmd(client, asJSON), newAdaptersCmd(client), newModelServeCmd(client), newModelUpdatesCmd(client, asJSON), newModelUpdateCmd(client), - newModelRollbackCmd(client), newModelOriginCmd(client)) + newModelRollbackCmd(client), newModelOriginCmd(client), + newModelCanaryCmd(client), newModelPromoteCmd(client), newModelCanariesCmd(client, asJSON)) return c } diff --git a/internal/cli/model_canary.go b/internal/cli/model_canary.go new file mode 100644 index 00000000..ad353e98 --- /dev/null +++ b/internal/cli/model_canary.go @@ -0,0 +1,148 @@ +package cli + +import ( + "encoding/json" + "fmt" + "slices" + "text/tabwriter" + + "github.com/spf13/cobra" + + "gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog" + "gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve" +) + +func newModelCanaryCmd(client func() *ctlClient) *cobra.Command { + var ( + cn catalog.Canary + abort, detach bool + ) + c := &cobra.Command{ + Use: "canary MODEL", + Short: "Try a model's newer weights on a share of its requests before moving to them", + Long: "The newer files of the model's repository are fetched and run beside it\n" + + "as MODEL-canary, one replica, and given --percent of MODEL's requests. A\n" + + "session (x-session-id, or the user field) stays on one side; the\n" + + "x-llmgw-variant header (canary or stable) pins a request to one. Clients\n" + + "cannot name the canary itself.\n\n" + + "The two sides are compared as they serve. Once the canary has served\n" + + "--min-requests, it is taken out by itself if its error rate is\n" + + "--max-error-increase above the model's, or its time to the first token\n" + + "(or, unstreamed, to the answer) --max-ttft-ratio times the model's, and\n" + + "canary_rolled_back goes to the notification webhooks.\n\n" + + "Run again with --percent to change its share, --abort to take it out;\n" + + "`llmgw ctl model promote MODEL` moves the model to it.", + Example: " llmgw ctl model canary qwen-7b --percent 10\n" + + " llmgw ctl model canary qwen-7b --percent 50\n" + + " llmgw ctl model canary qwen-7b --abort", + Args: cobra.ExactArgs(1), + RunE: func(cmd *cobra.Command, args []string) error { + name := args[0] + var out map[string]string + if abort { + if err := client().delete(cmd.Context(), "/api/v1/models/"+name+"/canary", &out); err != nil { + return err + } + fmt.Fprintln(cmd.OutOrStdout(), out["result"]) + return nil + } + var have []modelserve.CanaryView + if err := client().get(cmd.Context(), "/api/v1/canaries", &have); err != nil { + return err + } + if slices.ContainsFunc(have, func(v modelserve.CanaryView) bool { return v.Model == name }) { + if !cmd.Flags().Changed("percent") { + return fmt.Errorf("%s already has a canary: give --percent to change its share, or --abort", name) + } + body, _ := json.Marshal(map[string]int{"percent": cn.Percent}) + if err := client().put(cmd.Context(), "/api/v1/models/"+name+"/canary", body, &out); err != nil { + return err + } + fmt.Fprintln(cmd.OutOrStdout(), out["result"]) + return nil + } + body, _ := json.Marshal(cn) + var job modelserve.Job + if err := client().post(cmd.Context(), "/api/v1/models/"+name+"/canary", body, &job); err != nil { + return err + } + if detach { + return detached(cmd, job) + } + return followJob(cmd, client, job, "has a canary") + }, + } + f := c.Flags() + f.IntVar(&cn.Percent, "percent", 10, "share of the model's requests the canary is given, 0 to 100") + f.IntVar(&cn.MinRequests, "min-requests", 50, "requests the canary serves before it is judged") + f.Float64Var(&cn.MaxErrorIncrease, "max-error-increase", 0.05, "error rate above the model's that rolls it back (0.05 is five points)") + f.Float64Var(&cn.MaxTTFTRatio, "max-ttft-ratio", 1.5, "times the model's time to the first token that rolls it back") + f.BoolVar(&abort, "abort", false, "take the canary out; the model serves all its requests again") + f.BoolVar(&detach, "detach", false, "start it and return: it runs on the control plane; `llmgw ctl jobs` follows it") + return c +} + +func newModelPromoteCmd(client func() *ctlClient) *cobra.Command { + var detach bool + c := &cobra.Command{ + Use: "promote MODEL", + Short: "Move a model to its canary's weights, and take the canary out", + Long: "The update the canary was a trial of: the catalog names the canary's\n" + + "files, the model's replicas are replaced one at a time, and the canary\n" + + "is removed. `llmgw ctl model rollback MODEL` goes back.", + Args: cobra.ExactArgs(1), + RunE: func(cmd *cobra.Command, args []string) error { + var job modelserve.Job + if err := client().post(cmd.Context(), "/api/v1/models/"+args[0]+"/promote", []byte("{}"), &job); err != nil { + return err + } + if detach { + return detached(cmd, job) + } + return followJob(cmd, client, job, "runs the canary's weights") + }, + } + c.Flags().BoolVar(&detach, "detach", false, "start it and return: it runs on the control plane; `llmgw ctl jobs` follows it") + return c +} + +func newModelCanariesCmd(client func() *ctlClient, asJSON *bool) *cobra.Command { + return &cobra.Command{ + Use: "canaries", + Short: "Models trying a canary, and how each side has served since it began", + Args: cobra.NoArgs, + RunE: func(cmd *cobra.Command, _ []string) error { + var vs []modelserve.CanaryView + if err := client().get(cmd.Context(), "/api/v1/canaries", &vs); err != nil { + return err + } + if *asJSON { + return json.NewEncoder(cmd.OutOrStdout()).Encode(vs) + } + if len(vs) == 0 { + fmt.Fprintln(cmd.OutOrStdout(), "No model has a canary.") + return nil + } + w := tabwriter.NewWriter(cmd.OutOrStdout(), 0, 0, 2, ' ', 0) + fmt.Fprintln(w, "MODEL\tCANARY\tSHARE\tREQUESTS\tERRORS\tFIRST TOKEN\tANSWER") + for _, v := range vs { + side := func(label string, req int64, errRate, ttft, lat float64) { + fmt.Fprintf(w, "%s\t%s\t\t%d\t%.1f%%\t%s\t%s\n", "", label, req, 100*errRate, meanMs(ttft), meanMs(lat)) + } + fmt.Fprintf(w, "%s\t%s\t%d%%\t\t\t\t\n", v.Model, v.Canary.Model, v.Canary.Percent) + if s := v.Serving; s != nil { + side(" canary", s.Candidate.Requests, s.Candidate.ErrorRate(), s.Candidate.TTFTMs, s.Candidate.LatencyMs) + side(" stable", s.Stable.Requests, s.Stable.ErrorRate(), s.Stable.TTFTMs, s.Stable.LatencyMs) + } + } + return w.Flush() + }, + } +} + +func meanMs(v float64) string { + if v <= 0 { + return "-" + } + return fmt.Sprintf("%.0f ms", v) +} diff --git a/internal/cli/server.go b/internal/cli/server.go index aa783ce0..e9aadfea 100644 --- a/internal/cli/server.go +++ b/internal/cli/server.go @@ -405,7 +405,10 @@ func newServerCmd() *cobra.Command { Current: func() []catalog.Model { return admin.Catalog().Models() }, Models: catalogSvc, Restart: admin.Reconciler().Restart, Pool: adminAPI.PoolNodes, Notify: notifier, + // Canaries: how the router saw each side serve. + CanaryStats: rt.CanaryStats, ResetCanary: rt.ResetCanary, }) + go serve.WatchCanaries(ctx, 30*time.Second) adminAPI.SetServe(serve) updateMetrics.Set(serve) go serve.WatchUpdates(ctx, cfg.Mirror.UpdateCheck) diff --git a/internal/control/adminapi/api.go b/internal/control/adminapi/api.go index 19aecd00..c880ad2d 100644 --- a/internal/control/adminapi/api.go +++ b/internal/control/adminapi/api.go @@ -154,6 +154,7 @@ func (a *API) Register(mux *http.ServeMux) { mux.HandleFunc("POST /api/v1/updates/check", a.checkUpdates) mux.HandleFunc("POST /api/v1/models/{name}/update", a.updateModel) mux.HandleFunc("POST /api/v1/models/{name}/rollback", a.rollbackModel) + a.registerCanary(mux) mux.HandleFunc("POST /api/v1/models/{name}/origin", a.setOrigin) mux.HandleFunc("GET /api/v1/plan", a.planModel) mux.HandleFunc("POST /api/v1/metrics/range", a.metricsBatch) @@ -330,6 +331,10 @@ func loadView(l *inventory.Load, now time.Time) *LoadView { // ModelView is a model as `ctl models` shows it. type ModelView struct { Name string `json:"name"` + // Canary is a share of its requests sent to a candidate; CanaryOf is + // set on the candidate, naming the model it is a canary of. + Canary *catalog.Canary `json:"canary,omitempty"` + CanaryOf string `json:"canary_of,omitempty"` // Description is what the model is for, when somebody said. Description string `json:"description,omitempty"` Aliases []string `json:"aliases,omitempty"` @@ -651,6 +656,7 @@ func (a *API) Models() []ModelView { Parked: h.Parked, Reason: h.Reason, Unplaceable: h.Unplaceable, Class: m.EffectiveClass(), Owner: m.Owner, Preempted: h.Preempted, HeldFor: h.HeldFor, External: externalLabel(m), Adapters: m.Adapters, + Canary: m.Canary, CanaryOf: a.ctl.Catalog().CanaryOf(m.Name), } for _, p := range m.Profiles { v.Profiles = append(v.Profiles, p.Name) diff --git a/internal/control/adminapi/canary.go b/internal/control/adminapi/canary.go new file mode 100644 index 00000000..bba08211 --- /dev/null +++ b/internal/control/adminapi/canary.go @@ -0,0 +1,84 @@ +package adminapi + +import ( + "encoding/json" + "io" + "net/http" + + "gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog" +) + +func (a *API) registerCanary(mux *http.ServeMux) { + need := func(w http.ResponseWriter) bool { + if a.serve == nil { + writeError(w, http.StatusNotImplemented, "a canary needs a mirror and a store, and this control plane lacks one") + return false + } + return true + } + mux.HandleFunc("GET /api/v1/canaries", func(w http.ResponseWriter, r *http.Request) { + if !need(w) { + return + } + writeJSON(w, http.StatusOK, a.serve.Canaries()) + }) + // Start one: the newer weights fetched and run beside, then given + // their share. + mux.HandleFunc("POST /api/v1/models/{name}/canary", func(w http.ResponseWriter, r *http.Request) { + if !need(w) { + return + } + var cn catalog.Canary + if err := json.NewDecoder(io.LimitReader(r.Body, 1<<12)).Decode(&cn); err != nil { + writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error()) + return + } + job, err := a.serve.StartCanary(r.PathValue("name"), cn, actorOf(r)) + if err != nil { + writeError(w, http.StatusBadRequest, err.Error()) + return + } + writeJSON(w, http.StatusAccepted, job) + }) + // Change its share. + mux.HandleFunc("PUT /api/v1/models/{name}/canary", func(w http.ResponseWriter, r *http.Request) { + if !need(w) { + return + } + var req struct { + Percent int `json:"percent"` + } + if err := json.NewDecoder(io.LimitReader(r.Body, 1<<12)).Decode(&req); err != nil { + writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error()) + return + } + msg, err := a.serve.SetCanaryPercent(r.Context(), r.PathValue("name"), req.Percent, actorOf(r)) + if err != nil { + writeError(w, http.StatusBadRequest, err.Error()) + return + } + writeJSON(w, http.StatusOK, map[string]string{"result": msg}) + }) + mux.HandleFunc("DELETE /api/v1/models/{name}/canary", func(w http.ResponseWriter, r *http.Request) { + if !need(w) { + return + } + msg, err := a.serve.AbortCanary(r.Context(), r.PathValue("name"), actorOf(r)) + if err != nil { + writeError(w, http.StatusBadRequest, err.Error()) + return + } + writeJSON(w, http.StatusOK, map[string]string{"result": msg}) + }) + mux.HandleFunc("POST /api/v1/models/{name}/promote", func(w http.ResponseWriter, r *http.Request) { + if !need(w) { + return + } + job, err := a.serve.StartPromote(r.PathValue("name"), actorOf(r)) + if err != nil { + writeError(w, http.StatusBadRequest, err.Error()) + return + } + writeJSON(w, http.StatusAccepted, job) + }) +} diff --git a/internal/control/modelserve/canary.go b/internal/control/modelserve/canary.go new file mode 100644 index 00000000..4eea4bc1 --- /dev/null +++ b/internal/control/modelserve/canary.go @@ -0,0 +1,304 @@ +package modelserve + +import ( + "context" + "errors" + "fmt" + "time" + + "gitea.mynodes.xyz/ilker/llmgw/internal/canary" + "gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog" + "gitea.mynodes.xyz/ilker/llmgw/internal/notify" +) + +// A canary: a model's repository's newer weights run beside it as another +// model, given a share of its requests, compared, and then promoted to its +// place or taken out — by hand, or by itself when it serves worse. + +// canaryName is the candidate's name in the catalog. +func canaryName(model string) string { return model + "-canary" } + +// StartCanary fetches a model's newer weights and runs them as its canary, +// with the share and limits given. +func (s *Server) StartCanary(name string, cn catalog.Canary, by string) (Job, error) { + if s.deps.Models == nil || s.deps.Restart == nil { + return Job{}, errors.New("a canary needs the store, which this control plane lacks") + } + m, err := s.current(name) + if err != nil { + return Job{}, err + } + if s.deps.Models.Declared(name) { + return Job{}, fmt.Errorf("%s is declared in server.yaml; a canary is for a model added at runtime", name) + } + if m.Canary != nil { + return Job{}, fmt.Errorf("%s already has a canary, %s at %d%%: change its share with --percent, "+ + "or promote or abort it first", name, m.Canary.Model, m.Canary.Percent) + } + if cn.Percent < 0 || cn.Percent > 100 { + return Job{}, fmt.Errorf("percent %d is not 0 to 100", cn.Percent) + } + s.mu.Lock() + for _, j := range s.jobs { + if (j.Kind == "update" || j.Kind == "canary" || j.Kind == "promote") && j.Model == name && j.State == Running { + s.mu.Unlock() + return Job{}, fmt.Errorf("%s has a job running: %s", name, j.ID) + } + } + s.seq++ + j := &Job{ID: fmt.Sprintf("canary-%d-%d", time.Now().Unix(), s.seq), Kind: "canary", Model: name, + Request: Request{Name: name, By: by}, State: Running, Started: time.Now()} + s.keepJob(j) + s.mu.Unlock() + go s.runCanary(j, cn) + return s.snapshot(j), nil +} + +func (s *Server) runCanary(j *Job, cn catalog.Canary) { + ctx := context.Background() + name := j.Model + p, ok := s.prepare(ctx, j, name) + if !ok { + return + } + + // 4. The candidate, beside the model: the same in everything but its + // weights, and one replica. + cand := canaryName(name) + s.step(j, "catalog", Running, "adding "+cand) + c := p.m + c.Name, c.Aliases, c.Fallbacks, c.Canary, c.Adapters = cand, nil, nil, nil, nil + c.Source = p.source + c.Origin = &catalog.Origin{Repo: p.repo.ID, Revision: p.repo.Revision, Files: originFiles(p.set)} + c.Scaling = catalog.Scaling{Min: 1, Max: 1} + c.Description = fmt.Sprintf("Canary of %s at %s.", name, p.repo.ShortRevision()) + if err := s.deps.Models.Add(ctx, c, j.Request.By); err != nil { + s.fail(j, "catalog", err) + return + } + s.step(j, "catalog", Done, cand+" runs "+p.repo.ID+" at "+p.repo.ShortRevision()) + + // 5. Serving, before it is given anyone's requests. + r, err := s.waitReady(ctx, j, cand, false) + if err == nil { + s.step(j, "smoke test", Running, "asking "+r.ID+" to say hello") + var reply string + var tps float64 + if reply, tps, err = s.smoke(ctx, r, cand); err == nil { + s.step(j, "smoke test", Done, fmt.Sprintf("answered at %.1f tokens/s: %q", tps, reply)) + } else { + s.step(j, "smoke test", Failed, err.Error()) + } + } + if err != nil { + _ = s.deps.Models.Remove(ctx, cand) + s.fail(j, "placement", fmt.Errorf("%w; %s was taken out again", err, cand)) + return + } + + // 6. Its share. + cn.Model = cand + if _, err := s.deps.Models.SetCanary(ctx, name, &cn, j.Request.By); err != nil { + _ = s.deps.Models.Remove(ctx, cand) + s.fail(j, "split", err) + return + } + if s.deps.ResetCanary != nil { + s.deps.ResetCanary(name) + } + s.step(j, "split", Done, fmt.Sprintf("%d%% of %s's requests go to %s", cn.Percent, name, cand)) + s.finish(j) +} + +// SetCanaryPercent changes the share of a model's requests its canary is +// given; 0 keeps it running and gives it none. +func (s *Server) SetCanaryPercent(ctx context.Context, name string, percent int, by string) (string, error) { + m, err := s.current(name) + if err != nil { + return "", err + } + if m.Canary == nil { + return "", fmt.Errorf("%s has no canary; start one with `llmgw ctl model canary %s --percent N`", name, name) + } + cn := *m.Canary + cn.Percent = percent + if _, err := s.deps.Models.SetCanary(ctx, name, &cn, by); err != nil { + return "", err + } + return fmt.Sprintf("%d%% of %s's requests go to %s", percent, name, cn.Model), nil +} + +// AbortCanary takes a model's canary out: its share first, then the +// candidate. +func (s *Server) AbortCanary(ctx context.Context, name, by string) (string, error) { + m, err := s.current(name) + if err != nil { + return "", err + } + if m.Canary == nil { + return "", fmt.Errorf("%s has no canary", name) + } + cand := m.Canary.Model + if _, err := s.deps.Models.SetCanary(ctx, name, nil, by); err != nil { + return "", err + } + if err := s.deps.Models.Remove(ctx, cand); err != nil { + return "", fmt.Errorf("%s gets no more requests, but could not be removed: %w", cand, err) + } + if s.deps.ResetCanary != nil { + s.deps.ResetCanary(name) + } + return fmt.Sprintf("%s is out; %s serves all of %s's requests", cand, name, name), nil +} + +// StartPromote moves a model to its canary's weights — the update a canary +// was the trial of — and takes the canary out. +func (s *Server) StartPromote(name, by string) (Job, error) { + m, err := s.current(name) + if err != nil { + return Job{}, err + } + if m.Canary == nil { + return Job{}, fmt.Errorf("%s has no canary to promote", name) + } + s.mu.Lock() + s.seq++ + j := &Job{ID: fmt.Sprintf("promote-%d-%d", time.Now().Unix(), s.seq), Kind: "promote", Model: name, + Request: Request{Name: name, By: by}, State: Running, Started: time.Now()} + s.keepJob(j) + s.mu.Unlock() + go s.runPromote(j) + return s.snapshot(j), nil +} + +func (s *Server) runPromote(j *Job) { + ctx := context.Background() + name := j.Model + m, err := s.current(name) + if err != nil || m.Canary == nil { + s.fail(j, "catalog", fmt.Errorf("%s no longer has a canary", name)) + return + } + c, err := s.current(m.Canary.Model) + if err != nil { + s.fail(j, "catalog", err) + return + } + s.step(j, "catalog", Running, fmt.Sprintf("moving %s to %s's weights", name, c.Name)) + next := &catalog.Origin{} + if c.Origin != nil { + *next = *c.Origin + } + next.Previous = &catalog.PreviousOrigin{Source: m.Source} + if m.Origin != nil { + next.Previous.Revision, next.Previous.Files = m.Origin.Revision, m.Origin.Files + } + if _, err := s.deps.Models.SetSource(ctx, name, c.Source, next, j.Request.By); err != nil { + s.fail(j, "catalog", err) + return + } + // Its share back to the model, which is about to run the same weights. + if _, err := s.deps.Models.SetCanary(ctx, name, nil, j.Request.By); err != nil { + s.fail(j, "catalog", err) + return + } + s.step(j, "catalog", Done, "the catalog names "+c.Source) + + if err := s.replace(ctx, j, name); err != nil { + s.fail(j, "replicas", fmt.Errorf("%w. `llmgw ctl model rollback %s` goes back", err, name)) + return + } + if err := s.deps.Models.Remove(ctx, c.Name); err != nil { + s.step(j, "canary", Failed, fmt.Sprintf("%s could not be removed: %v", c.Name, err)) + } else { + s.step(j, "canary", Done, c.Name+" removed") + } + if s.deps.ResetCanary != nil { + s.deps.ResetCanary(name) + } + if c.Origin != nil { + s.mu.Lock() + s.updates[name] = UpdateInfo{Model: name, Repo: c.Origin.Repo, Current: c.Origin.Revision, + Latest: c.Origin.Revision, Previous: next.Previous.Revision, CheckedAt: time.Now()} + s.mu.Unlock() + } + s.finish(j) +} + +// WatchCanaries takes out, every so often, each canary serving worse than +// its model by the limits it was given. Until ctx ends. +func (s *Server) WatchCanaries(ctx context.Context, every time.Duration) { + if s.deps.CanaryStats == nil || s.deps.Current == nil { + return + } + if every <= 0 { + every = 30 * time.Second + } + t := time.NewTicker(every) + defer t.Stop() + for { + select { + case <-ctx.Done(): + return + case <-t.C: + s.checkCanaries(ctx) + } + } +} + +func (s *Server) checkCanaries(ctx context.Context) { + stats := s.deps.CanaryStats() + for _, m := range s.deps.Current() { + if m.Canary == nil || m.Canary.Percent == 0 { + continue + } + st, ok := stats[m.Name] + if !ok || st.Canary != m.Canary.Model { + continue + } + bad, why := canary.Verdict(st, *m.Canary) + if !bad { + continue + } + cand := m.Canary.Model + if _, err := s.AbortCanary(ctx, m.Name, "llmgw"); err != nil { + s.deps.Log.Error("a canary serving worse could not be taken out", "model", m.Name, "canary", cand, "err", err) + continue + } + s.deps.Log.Warn("canary rolled back: it served worse than the model", "model", m.Name, "canary", cand, "why", why) + s.deps.Notify.Notify(notify.CanaryRolledBack, m.Name+"/"+cand, + fmt.Sprintf("the canary of %s was rolled back: %s, over %d requests", m.Name, why, st.Candidate.Requests), + map[string]any{"model": m.Name, "canary": cand, "reason": why, + "canary_requests": st.Candidate.Requests, "stable_requests": st.Stable.Requests}) + } +} + +// CanaryView is one model's canary as it stands. +type CanaryView struct { + Model string `json:"model"` + Canary catalog.Canary `json:"canary"` + Serving *canary.Stat `json:"serving,omitempty"` +} + +// Canaries are the models with a canary, and how each side has served. +func (s *Server) Canaries() []CanaryView { + if s.deps.Current == nil { + return nil + } + var stats map[string]canary.Stat + if s.deps.CanaryStats != nil { + stats = s.deps.CanaryStats() + } + var out []CanaryView + for _, m := range s.deps.Current() { + if m.Canary == nil { + continue + } + v := CanaryView{Model: m.Name, Canary: *m.Canary} + if st, ok := stats[m.Name]; ok && st.Canary == m.Canary.Model { + v.Serving = &st + } + out = append(out, v) + } + return out +} diff --git a/internal/control/modelserve/canary_test.go b/internal/control/modelserve/canary_test.go new file mode 100644 index 00000000..6eedadfc --- /dev/null +++ b/internal/control/modelserve/canary_test.go @@ -0,0 +1,101 @@ +package modelserve + +import ( + "context" + "log/slog" + "strings" + "testing" + "time" + + "gitea.mynodes.xyz/ilker/llmgw/internal/canary" + "gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog" + "gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler" +) + +func canarySetup(t *testing.T) (*Server, *updModels, *restartFleet) { + t.Helper() + s, _, models, _, fleet := updSetup(t) + models.onAdd = func(m catalog.Model) { + fleet.mu.Lock() + fleet.reps = append(fleet.reps, reconciler.Replica{ID: m.Name + "-1", Model: m.Name, + State: reconciler.Ready, Endpoint: fleet.engine}) + fleet.mu.Unlock() + } + s.deps.Log = slog.New(slog.DiscardHandler) + return s, models, fleet +} + +func TestACanaryRunsTheNewWeightsBesideAndIsPromoted(t *testing.T) { + s, models, fleet := canarySetup(t) + oldSource := models.models["qwen"].Source + job, err := s.StartCanary("qwen", catalog.Canary{Percent: 10}, "tester") + if err != nil { + t.Fatal(err) + } + if j := wait(t, s, job.ID); j.State != Done { + t.Fatalf("%s: %+v", j.Error, j.Steps) + } + m, c := models.models["qwen"], models.models["qwen-canary"] + if m.Source != oldSource || m.Canary == nil || m.Canary.Model != "qwen-canary" || m.Canary.Percent != 10 { + t.Fatalf("the model was changed, or given no canary: %s %+v", m.Source, m.Canary) + } + if !strings.HasSuffix(c.Source, "/qwen/def98765/q4.gguf") || c.Origin.Revision != newRev || c.Scaling.Max != 1 { + t.Fatalf("candidate: %s %+v %+v", c.Source, c.Origin, c.Scaling) + } + if _, err := s.StartCanary("qwen", catalog.Canary{Percent: 10}, "tester"); err == nil { + t.Error("a second canary was started") + } + if _, err := s.SetCanaryPercent(context.Background(), "qwen", 50, "tester"); err != nil || models.models["qwen"].Canary.Percent != 50 { + t.Errorf("share: %v", err) + } + + job, err = s.StartPromote("qwen", "tester") + if err != nil { + t.Fatal(err) + } + if j := wait(t, s, job.ID); j.State != Done { + t.Fatalf("%s: %+v", j.Error, j.Steps) + } + m = models.models["qwen"] + if m.Source != c.Source || m.Canary != nil || m.Origin.Previous == nil || m.Origin.Previous.Source != oldSource { + t.Fatalf("promoted: %s %+v %+v", m.Source, m.Canary, m.Origin) + } + if _, ok := models.models["qwen-canary"]; ok { + t.Error("the canary was left in the catalog") + } + for _, r := range fleet.Replicas() { + if r.Model == "qwen" && !strings.Contains(r.ID, "new") { + t.Errorf("%s was not replaced", r.ID) + } + } +} + +func TestACanaryServingWorseIsRolledBackByItself(t *testing.T) { + s, models, _ := canarySetup(t) + job, _ := s.StartCanary("qwen", catalog.Canary{Percent: 10, MinRequests: 20}, "tester") + if j := wait(t, s, job.ID); j.State != Done { + t.Fatalf("%s: %+v", j.Error, j.Steps) + } + var stat canary.Stat + stat.Canary = "qwen-canary" + for i := range 200 { + stat.Stable.Add(200, true, 100*time.Millisecond, time.Second) + if i < 30 { + status := 200 + if i%3 == 0 { + status = 502 + } + stat.Candidate.Add(status, true, 100*time.Millisecond, time.Second) + } + } + reset := "" + s.deps.CanaryStats = func() map[string]canary.Stat { return map[string]canary.Stat{"qwen": stat} } + s.deps.ResetCanary = func(m string) { reset = m } + s.checkCanaries(context.Background()) + if models.models["qwen"].Canary != nil || reset != "qwen" { + t.Fatalf("a canary failing a third of its requests was kept: %+v", models.models["qwen"].Canary) + } + if _, ok := models.models["qwen-canary"]; ok { + t.Error("the candidate was left in the catalog") + } +} diff --git a/internal/control/modelserve/modelserve.go b/internal/control/modelserve/modelserve.go index 294d14ec..0676d4ac 100644 --- a/internal/control/modelserve/modelserve.go +++ b/internal/control/modelserve/modelserve.go @@ -25,6 +25,7 @@ import ( "sync" "time" + "gitea.mynodes.xyz/ilker/llmgw/internal/canary" "gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog" "gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler" "gitea.mynodes.xyz/ilker/llmgw/internal/gguf" @@ -86,9 +87,14 @@ type Deps struct { // UnplacedGrace is how long a model may go unplaced before the job // stops waiting for it: a minute unless set. UnplacedGrace time.Duration - // Notify is told when a model's repository publishes new weights; nil - // tells no one. + // Notify is told when a model's repository publishes new weights or a + // canary is rolled back; nil tells no one. Notify *notify.Notifier + // CanaryStats is how each model's canary and stable side served, by + // model; ResetCanary forgets one. From the router; without them a + // canary is never rolled back by itself. + CanaryStats func() map[string]canary.Stat + ResetCanary func(model string) } // errBusy is a model the cluster has the hardware for, all of it in use now. diff --git a/internal/control/modelserve/updates.go b/internal/control/modelserve/updates.go index d3a6a28c..18b1f4db 100644 --- a/internal/control/modelserve/updates.go +++ b/internal/control/modelserve/updates.go @@ -33,6 +33,11 @@ type Models interface { Declared(name string) bool SetSource(ctx context.Context, name, source string, origin *catalog.Origin, by string) (catalog.Model, error) SetOrigin(ctx context.Context, name string, origin *catalog.Origin) error + // For a canary: the candidate added beside the model and taken out + // again, and the share of requests it is given. + Add(ctx context.Context, m catalog.Model, by string) error + Remove(ctx context.Context, name string) error + SetCanary(ctx context.Context, name string, c *catalog.Canary, by string) (catalog.Model, error) } // UpdateInfo is what the registry has for a model against what it runs. @@ -451,32 +456,43 @@ func (s *Server) StartUpdate(name, by string) (Job, error) { return s.snapshot(j), nil } -func (s *Server) runUpdate(j *Job) { - ctx := context.Background() - name := j.Model +// prepared is a model's new weights, fetched and checked: what an update and +// a canary both start from. +type prepared struct { + m catalog.Model + o *catalog.Origin + repo hf.Repo + set hf.FileSet + source string +} + +// prepare reads where a model came from, fetches its repository's newer +// files into the mirror beside the ones in use, and checks the nodes can +// load them. It fails the job and says false when any of that cannot be. +func (s *Server) prepare(ctx context.Context, j *Job, name string) (prepared, bool) { // 1. The registry: which files, at which commit. s.step(j, "registry", Running, "reading where "+name+" came from") m, err := s.current(name) if err != nil { s.fail(j, "registry", err) - return + return prepared{}, false } list, err := s.deps.Mirror.List(ctx) if err != nil { s.fail(j, "registry", fmt.Errorf("the mirror: %w", err)) - return + return prepared{}, false } o := s.originOf(ctx, m, list) p, onMirror := placeOf(m.Source, list) if o == nil || !onMirror { s.fail(j, "registry", fmt.Errorf("%s did not come from the registry through the mirror", name)) - return + return prepared{}, false } repo, err := s.deps.Registry.Repo(ctx, o.Repo) if err != nil { s.fail(j, "registry", err) - return + return prepared{}, false } s.mu.Lock() j.Request.Repo = repo.ID @@ -485,18 +501,18 @@ func (s *Server) runUpdate(j *Job) { switch { case sameRevision(repo.Revision, o.Revision): s.fail(j, "registry", fmt.Errorf("%s already runs %s's latest commit, %s", name, repo.ID, catalog.ShortRevision(repo.Revision))) - return + return prepared{}, false case len(u.Missing) > 0: s.fail(j, "registry", fmt.Errorf("%s at %s no longer has %s; choose one of its files and serve it under a new name", repo.ID, repo.ShortRevision(), strings.Join(u.Missing, ", "))) - return + return prepared{}, false case len(u.Changed) == 0: s.fail(j, "registry", fmt.Errorf("%s has a newer commit, %s, but the files %s runs are the same in it", repo.ID, repo.ShortRevision(), name)) - return + return prepared{}, false case repo.Gated.Is(): s.fail(j, "registry", fmt.Errorf("%s is gated: it needs an account's approval the mirror does not have", repo.ID)) - return + return prepared{}, false } set := hf.FileSet{} for _, f := range o.Files { @@ -512,7 +528,7 @@ func (s *Server) runUpdate(j *Job) { source, err := s.mirror(ctx, j, repo, set, p.model) if err != nil { s.fail(j, "mirror", err) - return + return prepared{}, false } // 3. Whether the nodes load it, and whether it still fits what is @@ -523,7 +539,7 @@ func (s *Server) runUpdate(j *Job) { v := gguf.Check(ctx, u, s.deps.LlamaVersion) if !v.Servable { s.fail(j, "check", fmt.Errorf("llama.cpp %s on the nodes cannot run the new weights: %s", s.deps.LlamaVersion, v.Reason)) - return + return prepared{}, false } detail = orUnknown(v.Header.Architecture) if len(m.Profiles) > 0 { @@ -540,6 +556,17 @@ func (s *Server) runUpdate(j *Job) { } } s.step(j, "check", Done, detail) + return prepared{m: m, o: o, repo: repo, set: set, source: source}, true +} + +func (s *Server) runUpdate(j *Job) { + ctx := context.Background() + name := j.Model + p, ok := s.prepare(ctx, j, name) + if !ok { + return + } + m, o, repo, set, source := p.m, p.o, p.repo, p.set, p.source // 4. The catalog: the new source, and the old one kept to go back to. s.step(j, "catalog", Running, "moving "+name+" to "+repo.ShortRevision()) diff --git a/internal/control/modelserve/updates_test.go b/internal/control/modelserve/updates_test.go index ee3b22ff..81c94303 100644 --- a/internal/control/modelserve/updates_test.go +++ b/internal/control/modelserve/updates_test.go @@ -60,6 +60,8 @@ type updModels struct { mu sync.Mutex models map[string]catalog.Model declared map[string]bool + // onAdd starts what is added, as the reconciler would. + onAdd func(catalog.Model) } func (c *updModels) Declared(n string) bool { return c.declared[n] } @@ -79,6 +81,29 @@ func (c *updModels) SetOrigin(_ context.Context, n string, o *catalog.Origin) er c.models[n] = m return nil } +func (c *updModels) Add(_ context.Context, m catalog.Model, _ string) error { + c.mu.Lock() + defer c.mu.Unlock() + c.models[m.Name] = m + if c.onAdd != nil { + c.onAdd(m) + } + return nil +} +func (c *updModels) Remove(_ context.Context, n string) error { + c.mu.Lock() + defer c.mu.Unlock() + delete(c.models, n) + return nil +} +func (c *updModels) SetCanary(_ context.Context, n string, cn *catalog.Canary, _ string) (catalog.Model, error) { + c.mu.Lock() + defer c.mu.Unlock() + m := c.models[n] + m.Canary = cn + c.models[n] = m + return m, nil +} func (c *updModels) all() []catalog.Model { c.mu.Lock() defer c.mu.Unlock() diff --git a/internal/control/modelsvc/modelsvc.go b/internal/control/modelsvc/modelsvc.go index b45be61d..36fa40bc 100644 --- a/internal/control/modelsvc/modelsvc.go +++ b/internal/control/modelsvc/modelsvc.go @@ -306,6 +306,23 @@ func (s *Service) SetSource(ctx context.Context, name, source string, origin *ca return final, err } +// SetCanary gives a model added at runtime a canary, changes its share, or +// with nil takes it away. The candidate must already be in the catalog. +func (s *Service) SetCanary(ctx context.Context, name string, c *catalog.Canary, by string) (catalog.Model, error) { + final, err := s.update(ctx, name, by, "set its canary", func(m *catalog.Model) error { + m.Canary = c + return nil + }) + if err == nil { + if c == nil { + s.log.Info("model canary removed", "model", name, "by", by) + } else { + s.log.Info("model canary set", "model", name, "canary", c.Model, "percent", c.Percent, "by", by) + } + } + return final, err +} + // SetOrigin records where a model's weights came from without changing them: // for a model added before origins were kept. A model from configuration is // left alone, its origin worked out again when it is needed. diff --git a/internal/notify/notify.go b/internal/notify/notify.go index 8be77a15..0eca9b56 100644 --- a/internal/notify/notify.go +++ b/internal/notify/notify.go @@ -38,10 +38,13 @@ const ( // JudgeUnavailable is a guardrails judge giving no verdict: with // fail: closed, requests are being refused; with open, served unjudged. JudgeUnavailable = "judge_unavailable" + // CanaryRolledBack is a model's canary taken out because it served + // worse than the model it was a candidate to replace. + CanaryRolledBack = "canary_rolled_back" ) // Kinds are every event there is. -var Kinds = []string{ModelParked, NodeLost, BudgetSpent, UpdateAvailable, JudgeUnavailable} +var Kinds = []string{ModelParked, NodeLost, BudgetSpent, UpdateAvailable, JudgeUnavailable, CanaryRolledBack} // Event is one thing that happened. type Event struct { From 889be70b8f8e279d9c6e1e4cbc599751f6659c00 Mon Sep 17 00:00:00 2001 From: Ilker Manap Date: Sun, 4 Oct 2026 20:33:59 +0200 Subject: [PATCH 04/38] Canary on the console's model menu; docs Co-Authored-By: Claude Opus 5.5 (1M context) --- BACKLOG.md | 8 ++++- CHANGELOG.md | 17 +++++++++ docs/console.md | 7 ++++ docs/install.md | 33 +++++++++++++++++ docs/reference.md | 48 +++++++++++++++++++++++++ internal/cli/tui_actions.go | 62 ++++++++++++++++++++++++++++++++ internal/cli/tui_actions_test.go | 27 ++++++++++++++ internal/docgen/reference_gen.go | 8 +++++ 8 files changed, 209 insertions(+), 1 deletion(-) diff --git a/BACKLOG.md b/BACKLOG.md index 4d196713..90ca8880 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -420,7 +420,13 @@ portal is not done — the page shows; the console and ctl change. **Request metadata and tags (S).** Accept `metadata`/`user` from the body or `x-llmgw-tags`; store on the usage row; tenant metric labels behind an allowlist. `router/handlers.go` parse, `accounting/recorder.go`. -7. **Canary and weighted A/B between model revisions (M).** +7. **Done (branch `features-c2`):** `ctl model canary/promote/canaries`, + session-sticky split with `x-llmgw-variant` pins, rollback on error rate + or TTFT, `canary_rolled_back` notification, console menu. Left: A/B + between two different models (the split is between a model and its + newer weights); the comparison is per control plane and lives in memory, + so a restart begins it again; a key-level pin. + **Canary and weighted A/B between model revisions (M).** `control/modelserve/updates.go` `replace` swaps every replica; add weighted routing across two revisions/aliases (scheduler filter, alias resolution in `router/router.go:569`), a header/key pin, automatic diff --git a/CHANGELOG.md b/CHANGELOG.md index 2a224821..7bda1537 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -71,6 +71,23 @@ APIs change between minors without a deprecation period. - **Other ways in:** `GET /api/v1/usage/export` (CSV, or `format=json`) and **E** on the console's Usage view. +### New weights tried on a share of the traffic first + +- **Canaries.** `llmgw ctl model canary MODEL --percent 10` fetches the newer + weights of a model's repository and runs them beside it as `MODEL-canary`, + given that share of its requests. + - **Routing:** a session stays on one side; `x-llmgw-variant` pins a + request to `canary` or `stable` and says which served it. Clients cannot + name the canary. + - **Rolled back by itself** when, over 50 of its requests or more, its + error rate is five points above the model's or its time to the first + token half again as long; `canary_rolled_back` is sent to the + notification webhooks. + - **Then:** `--percent` changes its share, `llmgw ctl model promote` + moves the model to it, `--abort` takes it out, and `llmgw ctl model + canaries` compares the two sides. All of it is also on the console's + model menu. + ### Requests can be tagged - **Tags and end users on every usage record.** OpenAI's `metadata` and the diff --git a/docs/console.md b/docs/console.md index 8a8b0476..44fcbdc7 100644 --- a/docs/console.md +++ b/docs/console.md @@ -110,6 +110,13 @@ and **UPDATE** when the registry has new weights for it. On a row: **Go back to …** returns to the weights the update replaced. **Ask the registry for new weights now** checks without waiting for the next scheduled look (`mirror.update_check`, six hours by default). +- **Try the new weights as a canary…** — instead of swapping them in, runs + them beside the model as `MODEL-canary`, one replica, given the share of + its requests you choose. With a canary running the menu has **Canary's + share…**, **Promote the canary…** (the model moves to its weights, the + canary is removed) and **Take the canary out…**. A canary that errs or + answers more slowly than the model is taken out by itself. The MODEL + column shows `(10% to canary)` and `(canary of …)`. - **Why is it placed where it is** — `llmgw ctl explain-placement`. - **Try again**, for a model parked after failing every start. - **Remove** — its replicas stop and it leaves the catalog. A model from diff --git a/docs/install.md b/docs/install.md index 4eb4b71e..fae8cd8f 100644 --- a/docs/install.md +++ b/docs/install.md @@ -879,6 +879,38 @@ prices and budgets, rate limits, usage records, and key allowlists. whose allowlist leaves the external model out cannot reach it, as a fallback or by name. The cluster page and `ctl models` mark it `external`. +### Trying new weights on some of the traffic first + +When a model's repository publishes new weights, `llmgw ctl model update` +swaps them in. To see how they serve first, run them as a canary: + +``` +llmgw ctl model canary qwen-7b --percent 10 # fetched, started beside, given 10% +llmgw ctl model canaries # how each side has served +llmgw ctl model canary qwen-7b --percent 50 # more +llmgw ctl model promote qwen-7b # the model moves to them +llmgw ctl model canary qwen-7b --abort # or not +``` + +- **What runs:** the new files are fetched into the mirror and checked as + for an update, then run as `qwen-7b-canary` — the same model in everything + but its weights, one replica. Clients cannot name it; it gets its share + of `qwen-7b`'s requests. +- **Who goes where:** a session (`x-session-id`, or the `user` field) + stays on one side, so a conversation does not change weights halfway. + `x-llmgw-variant: canary` or `stable` pins a request to one side; the + response says which served it in the same header, and the usage record + names `qwen-7b-canary` as the model that served it. +- **Rolled back by itself:** once the canary has served `--min-requests` + (50), it is taken out if its error rate is `--max-error-increase` (0.05, + five points) above the model's, or its mean time to the first token — or + to the answer, unstreamed — is `--max-ttft-ratio` (1.5) times the + model's. `canary_rolled_back` goes to the notification webhooks. The + comparison is the router's own count since the canary began, kept per + control plane. +- **Promote** is the update the canary was a trial of: the model's replicas + are replaced one at a time, and `llmgw ctl model rollback` goes back. + ### How many replicas, and how much each one carries Replica count is changed while the cluster runs, without editing the @@ -981,6 +1013,7 @@ control plane sends those itself: | `node_lost` | A node was silent past `control.lost_after`; its replicas are rebuilt elsewhere. | | `budget_spent` | A key, project, team or organization reached its budget; priced models are refused. | | `update_available` | A model's registry repository published different weights. Once per revision. | +| `canary_rolled_back` | A model's canary served worse than the model and was taken out. | | `judge_unavailable` | A guardrails judge gave no verdict: with `fail: closed` requests are being refused, with `open` served unjudged. | ```yaml diff --git a/docs/reference.md b/docs/reference.md index ceb5a022..575d1927 100644 --- a/docs/reference.md +++ b/docs/reference.md @@ -43,6 +43,11 @@ rather than repeated. | `batch_size` | `int` | [Auth](#auth) | | `buffer_size` | `int` | [Auth](#auth) | | `usage_retention` | `time.Duration` | [Auth](#auth) | +| `model` | `string` | [Canary](#canary) | +| `percent` | `int` | [Canary](#canary) | +| `min_requests` | `int` | [Canary](#canary) | +| `max_error_increase` | `float64` | [Canary](#canary) | +| `max_ttft_ratio` | `float64` | [Canary](#canary) | | `tools` | `bool` | [Capabilities](#capabilities) | | `vision` | `bool` | [Capabilities](#capabilities) | | `embeddings` | `bool` | [Capabilities](#capabilities) | @@ -98,6 +103,7 @@ rather than repeated. | `external` | `*External` | [Model](#model) | | `adapters` | `[]Adapter` | [Model](#model) | | `fallbacks` | `[]string` | [Model](#model) | +| `canary` | `*Canary` | [Model](#model) | | `routing_profile` | `string` | [Model](#model) | | `slo` | `SLO` | [Model](#model) | | `context_length` | `int` | [Model](#model) | @@ -511,6 +517,38 @@ five-minute sums kept beside the records, which are not pruned, so this bounds the table that grows by a row a request without changing any figure a page shows. At least a day when set. +## Canary + +### `model` + +`string` + +Model is the candidate: a catalog model, usually this one at newer +weights, which clients cannot name themselves. + +### `percent` + +`int` + +Percent of requests go to it, 0 to 100. + +### `min_requests` + +`int` + +The candidate is taken out — rolled back — when, over at least +MinRequests of its own, its error rate is MaxErrorIncrease above +this model's (0.05 is five points), or its mean time to the first +token is MaxTTFTRatio times this model's. Zero is 50, 0.05 and 1.5. + +### `max_error_increase` + +`float64` + +### `max_ttft_ratio` + +`float64` + ## Capabilities ### `tools` @@ -916,6 +954,16 @@ every adapter, applying one only to a request that names it. `[]string` +### `canary` + +`*Canary` + +Canary sends a share of this model's requests to another catalog +model — the same model at new weights — to compare the two before +moving to it (`llmgw ctl model canary`). + +Takes a [Canary](#canary) block. + ### `routing_profile` `string` diff --git a/internal/cli/tui_actions.go b/internal/cli/tui_actions.go index 4c8f2511..b73c1091 100644 --- a/internal/cli/tui_actions.go +++ b/internal/cli/tui_actions.go @@ -680,6 +680,67 @@ func (m *consoleModel) doEntry() tea.Cmd { return nil } +// canaryItems try a model's newer weights on a share of its requests: start +// one, change its share, promote it, or take it out. +func (m consoleModel) canaryItems(md adminapi.ModelView) []menuItem { + path := "/api/v1/models/" + url.PathEscape(md.Name) + result := func(raw json.RawMessage) string { + var out map[string]string + _ = json.Unmarshal(raw, &out) + return out["result"] + "." + } + if cn := md.Canary; cn != nil { + return []menuItem{ + {fmt.Sprintf("Canary's share (%d%%)…", cn.Percent), func(m *consoleModel) tea.Cmd { + m.openForm(&formDialog{title: "Share of " + md.Name + " for " + cn.Model, + fields: []formField{textF("percent", strconv.Itoa(cn.Percent), "0 to 100; 0 keeps it running and gives it none")}, + submit: func(m *consoleModel, v []string) (tea.Cmd, error) { + n, err := strconv.Atoi(v[0]) + if err != nil || n < 0 || n > 100 { + return nil, fmt.Errorf("a whole number from 0 to 100") + } + return m.call("Canary of "+md.Name, http.MethodPut, path+"/canary", map[string]int{"percent": n}, result), nil + }}) + return nil + }}, + {"Promote the canary…", func(m *consoleModel) tea.Cmd { + m.confirmD = &confirmDialog{title: "Promote " + cn.Model, + text: md.Name + " moves to the canary's weights: its replicas are replaced one at a time,\n" + + "and the canary is removed. Going back is \"Go back to\" on this menu.", + yes: func(m *consoleModel) tea.Cmd { + return m.call("Promote "+cn.Model, http.MethodPost, path+"/promote", map[string]string{}, startedSays) + }} + return nil + }}, + {"Take the canary out…", func(m *consoleModel) tea.Cmd { + m.confirmD = &confirmDialog{title: "Take " + cn.Model + " out", + text: md.Name + " serves all of its requests again, and the canary is removed.", + yes: func(m *consoleModel) tea.Cmd { + return m.call("Canary of "+md.Name, http.MethodDelete, path+"/canary", nil, result) + }} + return nil + }}, + } + } + if u := md.Update; u == nil || !u.Available || md.CanaryOf != "" { + return nil + } + return []menuItem{{"Try the new weights as a canary…", func(m *consoleModel) tea.Cmd { + m.openForm(&formDialog{title: "Canary of " + md.Name, + note: "The new weights run beside it, one replica, given this share of its requests.\n" + + "Taken out by itself if it errs or answers more slowly than " + md.Name + ".", + fields: []formField{textF("percent", "10", "share of its requests, 0 to 100")}, + submit: func(m *consoleModel, v []string) (tea.Cmd, error) { + n, err := strconv.Atoi(v[0]) + if err != nil || n < 0 || n > 100 { + return nil, fmt.Errorf("a whole number from 0 to 100") + } + return m.call("Canary of "+md.Name, http.MethodPost, path+"/canary", catalog.Canary{Percent: n}, startedSays), nil + }}) + return nil + }}} +} + // updateItems are what can be done about a model's repository publishing new // weights: move to them, ask now, or go back. func (m consoleModel) updateItems(md adminapi.ModelView) []menuItem { @@ -728,6 +789,7 @@ func (m consoleModel) updateItems(md adminapi.ModelView) []menuItem { return nil }}) } + items = append(items, m.canaryItems(md)...) items = append(items, menuItem{"Ask the registry for new weights now", func(m *consoleModel) tea.Cmd { return m.call("New weights for "+md.Name, http.MethodPost, "/api/v1/updates/check", map[string]string{"model": md.Name}, func(raw json.RawMessage) string { diff --git a/internal/cli/tui_actions_test.go b/internal/cli/tui_actions_test.go index 859d4713..9beebbe7 100644 --- a/internal/cli/tui_actions_test.go +++ b/internal/cli/tui_actions_test.go @@ -17,6 +17,7 @@ import ( "gitea.mynodes.xyz/ilker/llmgw/internal/config" "gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi" + "gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog" "gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve" "gitea.mynodes.xyz/ilker/llmgw/internal/guard" "gitea.mynodes.xyz/ilker/llmgw/internal/modelplan" @@ -669,3 +670,29 @@ func TestTheAuditViewListsChangesAndOpensOne(t *testing.T) { t.Errorf("not opened:\n%s", v) } } + +func TestAModelsMenuOffersACanaryAndThenRunsIt(t *testing.T) { + m, _ := consoleWith(t, cModels) + labels := func(md adminapi.ModelView) []string { + var out []string + for _, it := range m.canaryItems(md) { + out = append(out, it.label) + } + return out + } + md := adminapi.ModelView{Name: "qwen", Update: &modelserve.UpdateInfo{Available: true, Latest: "def98765"}} + if got := labels(md); len(got) != 1 || !strings.Contains(got[0], "canary") { + t.Errorf("with new weights: %v", got) + } + md.Canary = &catalog.Canary{Model: "qwen-canary", Percent: 10} + got := strings.Join(labels(md), " | ") + for _, want := range []string{"share (10%)", "Promote", "Take the canary out"} { + if !strings.Contains(got, want) { + t.Errorf("with a canary, %q lacks %q", got, want) + } + } + if got := labels(adminapi.ModelView{Name: "qwen-canary", CanaryOf: "qwen", + Update: &modelserve.UpdateInfo{Available: true}}); len(got) != 0 { + t.Errorf("a canary offered a canary of its own: %v", got) + } +} diff --git a/internal/docgen/reference_gen.go b/internal/docgen/reference_gen.go index 39f81dc0..9d01106d 100644 --- a/internal/docgen/reference_gen.go +++ b/internal/docgen/reference_gen.go @@ -38,6 +38,13 @@ var Reference = []Block{ {Key: "buffer_size", Type: "int", Doc: "BufferSize bounds the queue. Beyond it records are dropped rather than\nblocking a request: losing accounting is bad, turning a stalled database\ninto an outage is worse.", Nested: ""}, {Key: "usage_retention", Type: "time.Duration", Doc: "UsageRetention is how long the record of each request is kept: who sent\nit, which replica answered, how long it queued — what `ctl request`\nreads. Zero keeps it for ever. Totals, charts and budgets are read from\nfive-minute sums kept beside the records, which are not pruned, so this\nbounds the table that grows by a row a request without changing any\nfigure a page shows. At least a day when set.", Nested: ""}, }}, + {Package: "catalog", Name: "Canary", Title: "Canary", Doc: "", Fields: []Field{ + {Key: "model", Type: "string", Doc: "Model is the candidate: a catalog model, usually this one at newer\nweights, which clients cannot name themselves.", Nested: ""}, + {Key: "percent", Type: "int", Doc: "Percent of requests go to it, 0 to 100.", Nested: ""}, + {Key: "min_requests", Type: "int", Doc: "The candidate is taken out — rolled back — when, over at least\nMinRequests of its own, its error rate is MaxErrorIncrease above\nthis model's (0.05 is five points), or its mean time to the first\ntoken is MaxTTFTRatio times this model's. Zero is 50, 0.05 and 1.5.", Nested: ""}, + {Key: "max_error_increase", Type: "float64", Doc: "", Nested: ""}, + {Key: "max_ttft_ratio", Type: "float64", Doc: "", Nested: ""}, + }}, {Package: "catalog", Name: "Capabilities", Title: "Capabilities", Doc: "", Fields: []Field{ {Key: "tools", Type: "bool", Doc: "", Nested: ""}, {Key: "vision", Type: "bool", Doc: "", Nested: ""}, @@ -114,6 +121,7 @@ var Reference = []Block{ {Key: "external", Type: "*External", Doc: "External is a model served by someone else — an OpenAI-compatible API\noutside the cluster — rather than by replicas placed here. Such a model\nhas no profiles and nothing is placed for it; the router sends its\nrequests to the provider. Most useful named in another model's\nfallbacks: used only when the cluster has no capacity left. Its\nprompts leave the cluster, so which keys may name it is worth an\nallowlist.", Nested: "catalog.External"}, {Key: "adapters", Type: "[]Adapter", Doc: "Adapters are LoRA adapters served on this model's replicas, each asked\nfor by its own name as though it were a model. Every replica loads\nevery adapter, applying one only to a request that names it.", Nested: ""}, {Key: "fallbacks", Type: "[]string", Doc: "", Nested: ""}, + {Key: "canary", Type: "*Canary", Doc: "Canary sends a share of this model's requests to another catalog\nmodel — the same model at new weights — to compare the two before\nmoving to it (`llmgw ctl model canary`).", Nested: "catalog.Canary"}, {Key: "routing_profile", Type: "string", Doc: "RoutingProfile selects how the router scores endpoints for this model.", Nested: ""}, {Key: "slo", Type: "SLO", Doc: "", Nested: "catalog.SLO"}, {Key: "context_length", Type: "int", Doc: "ContextLength is how many tokens one request may use, prompt and\ncompletion together, and MaxOutput bounds the completion alone. Zero\nmeans unknown, which is what every model was until now.\n\nThe gateway knew neither, so an over-long prompt was not refused: it was\nadmitted, given a replica, and failed at the engine, and a client sizing\nits prompts from /v1/models had nothing to read. Worse, the number is not\nthe one an operator writes in the engine's arguments — llama.cpp divides\n--ctx-size across its parallel slots, so a pool of 8192 across four slots\nis 2048 a request, and nothing said so.", Nested: ""}, From 841ee21c12d3a6f7cf933d52906d0ab663e83f45 Mon Sep 17 00:00:00 2001 From: Ilker Manap Date: Sun, 4 Oct 2026 21:00:01 +0200 Subject: [PATCH 05/38] Batches: OpenAI's Files and Batch APIs, served at the lowest priority under a store lease, each line a request of the key that made it Co-Authored-By: Claude Opus 5.5 (1M context) --- BACKLOG.md | 9 +- CHANGELOG.md | 15 + docs/data.md | 9 +- docs/install.md | 36 ++ docs/reference.md | 44 +++ internal/config/config.go | 20 + internal/docgen/reference_gen.go | 7 + internal/router/batches.go | 255 ++++++++++++ internal/router/batches_test.go | 241 +++++++++++ internal/router/batchworker.go | 373 ++++++++++++++++++ internal/router/files.go | 207 ++++++++++ internal/router/handlers.go | 10 + internal/router/router.go | 18 +- internal/store/batches.go | 345 ++++++++++++++++ internal/store/batches_test.go | 80 ++++ internal/store/copy.go | 3 + .../migrations/postgres/0024_batches.sql | 56 +++ .../store/migrations/sqlite/0024_batches.sql | 55 +++ internal/store/postgres_test.go | 2 +- 19 files changed, 1775 insertions(+), 10 deletions(-) create mode 100644 internal/router/batches.go create mode 100644 internal/router/batches_test.go create mode 100644 internal/router/batchworker.go create mode 100644 internal/router/files.go create mode 100644 internal/store/batches.go create mode 100644 internal/store/batches_test.go create mode 100644 internal/store/migrations/postgres/0024_batches.sql create mode 100644 internal/store/migrations/sqlite/0024_batches.sql diff --git a/BACKLOG.md b/BACKLOG.md index 90ca8880..95631cb7 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -431,7 +431,14 @@ portal is not done — the page shows; the console and ctl change. weighted routing across two revisions/aliases (scheduler filter, alias resolution in `router/router.go:569`), a header/key pin, automatic rollback on error-rate or TTFT regression from per-replica metrics. -8. **Batch API and Files (M–L).** `/v1/files`, `/v1/batches`, JSONL +8. **Done (branch `features-c2`):** `/v1/files` and `/v1/batches` + (OpenAI's shape), worker under a store lease, lines kept as served, + lowest band, usage tagged `batch=`, output and error files with + retention, `no_store` refused. Left: the Responses API (`/v1/responses`) + as a batch endpoint; the console and `ctl` do not list batches (an + operator sees them through usage tags); a file is one row in the store, + so a store backup carries batch files with it. + **Batch API and Files (M–L).** `/v1/files`, `/v1/batches`, JSONL storage, a worker feeding the bronze band, accounting. New `internal/router/batch`. 9. **Request/response logging and replay (M).** Opt-in per project with diff --git a/CHANGELOG.md b/CHANGELOG.md index 7bda1537..b8c411b9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -71,6 +71,21 @@ APIs change between minors without a deprecation period. - **Other ways in:** `GET /api/v1/usage/export` (CSV, or `format=json`) and **E** on the console's Usage view. +### Batches + +- **OpenAI's Batch and Files APIs.** `POST /v1/files` (purpose batch), + `POST /v1/batches`, and reading, listing, cancelling and deleting them, so + OpenAI's clients send batches here unchanged. + - **Serving:** each line goes as a request of the key that made the batch, + in the lowest admission band, a few at a time; rate limits are waited + out. Answers and failures are written to files kept 30 days. + - **Accounting:** each line is an ordinary usage record tagged + `batch=`. + - **Restarts:** answered lines are kept as they are answered, and any + gateway on the store carries a batch on under a lease. + - **Limits:** `router.batch` bounds the file, the lines and the + concurrency. Refused to a project under a `no_store` policy. + ### New weights tried on a share of the traffic first - **Canaries.** `llmgw ctl model canary MODEL --percent 10` fetches the newer diff --git a/docs/data.md b/docs/data.md index c103c23b..6d3cf02e 100644 --- a/docs/data.md +++ b/docs/data.md @@ -51,10 +51,17 @@ limits attached to it. Passwords are hashed. **Conversations and agent notes** — only for a project that asked. See below. +**Batch files** — when a client uses the Batch API: the file of requests it +uploaded (prompts) and the files the batch wrote (answers), in the store, +until deleted (`DELETE /v1/files/{id}`) or, for what a batch wrote, until +`router.batch.retain_days` (30) pass. Only keys of the project that made +them can read them, and a project under a `no_store` policy cannot make +them. + ## What is not stored Prompts. Completions. Anything the model was shown or produced, unless the -project turned on conversations or notes. +project turned on conversations or notes, or sent a batch. The chat page in llmgw's own web interface keeps the conversation in a JavaScript variable in the browser. Refresh it and the conversation is gone, diff --git a/docs/install.md b/docs/install.md index fae8cd8f..ae7b9f69 100644 --- a/docs/install.md +++ b/docs/install.md @@ -1130,6 +1130,42 @@ curl http://gateway:8080/v1/chat/completions \ 100 values of each apart and the rest as `other`: a tag is the client's to set, and every value is a series. +## Batches + +Work that can wait — a night's classification, embeddings for a document +store — goes as a batch: OpenAI's Batch API, so its clients work unchanged. + +``` +curl http://gateway:8080/v1/files -H "Authorization: Bearer $LLMGW_API_KEY" \ + -F purpose=batch -F file=@requests.jsonl +curl http://gateway:8080/v1/batches -H "Authorization: Bearer $LLMGW_API_KEY" \ + -d '{"input_file_id": "file-…", "endpoint": "/v1/chat/completions", "completion_window": "24h"}' +curl http://gateway:8080/v1/batches/batch_… -H "Authorization: Bearer $LLMGW_API_KEY" +curl http://gateway:8080/v1/files/file-…/content -H "Authorization: Bearer $LLMGW_API_KEY" +``` + +Each line of the file is `{"custom_id": …, "method": "POST", "url": +"/v1/chat/completions", "body": {…}}`, at most 50 000 of them +(`router.batch.max_lines`) in at most 100 MB (`router.batch.max_file_bytes`). + +- **How it is served:** each line as a request of the key that made the + batch — its limits, its budget, its guardrails — in the lowest admission + band, behind every interactive request, `router.batch.concurrency` (4) at + a time. A rate limit or a model still loading is waited out; a spent + budget fails the lines left. +- **What comes back:** answers in an output file, failures in an error + file, both OpenAI's form, kept `router.batch.retain_days` (30). Each + request is a usage record tagged `batch=`, so `llmgw ctl request` + finds it. +- **Who sees it:** files and batches belong to the key's project; every key + of the project sees them, no other does. +- **Surviving a restart:** every answered line is kept as it is answered, + and whichever gateway holds the batch's lease carries on from there; a + control plane that stops hands its batches to another within two minutes. +- **What it keeps:** a batch is prompts and answers stored until deleted + or past their retention — a project under a guardrails policy with + `no_store` is refused it. + ## Conversations across models Off everywhere until a project asks. llmgw otherwise stores no message content diff --git a/docs/reference.md b/docs/reference.md index 575d1927..03606276 100644 --- a/docs/reference.md +++ b/docs/reference.md @@ -43,6 +43,10 @@ rather than repeated. | `batch_size` | `int` | [Auth](#auth) | | `buffer_size` | `int` | [Auth](#auth) | | `usage_retention` | `time.Duration` | [Auth](#auth) | +| `max_file_bytes` | `int64` | [Batch](#batch) | +| `max_lines` | `int` | [Batch](#batch) | +| `concurrency` | `int` | [Batch](#batch) | +| `retain_days` | `int` | [Batch](#batch) | | `model` | `string` | [Canary](#canary) | | `percent` | `int` | [Canary](#canary) | | `min_requests` | `int` | [Canary](#canary) | @@ -198,6 +202,7 @@ rather than repeated. | `tenant_label` | `string` | [Router](#router) | | `tag_labels` | `[]string` | [Router](#router) | | `max_body_bytes` | `int64` | [Router](#router) | +| `batch` | `Batch` | [Router](#router) | | `trusted_proxies` | `[]string` | [Router](#router) | | `timeouts` | `Timeouts` | [Router](#router) | | `scoring` | `Scoring` | [Router](#router) | @@ -517,6 +522,35 @@ five-minute sums kept beside the records, which are not pruned, so this bounds the table that grows by a row a request without changing any figure a page shows. At least a day when set. +## Batch + +### `max_file_bytes` + +`int64` + +MaxFileBytes bounds one uploaded file; 100 MB unless set. + +### `max_lines` + +`int` + +MaxLines bounds the requests in one batch; 50 000 unless set. + +### `concurrency` + +`int` + +Concurrency is how many of one batch's requests are in flight at +once; 4 unless set. They queue in the lowest admission band, behind +every interactive request. + +### `retain_days` + +`int` + +RetainDays is how long a batch's answer and error files are kept; +30 unless set. Uploaded files are kept until deleted. + ## Canary ### `model` @@ -1622,6 +1656,16 @@ a series. The first 100 values of each are counted apart, the rest as MaxBodyBytes bounds the request body the router will read. +### `batch` + +`Batch` + +Batch is the Files and Batch APIs (/v1/files, /v1/batches): a file of +requests served at the lowest priority, its answers written to +another. Needs the store. + +Takes a [Batch](#batch) block. + ### `trusted_proxies` `[]string` diff --git a/internal/config/config.go b/internal/config/config.go index d19a2e20..fcc33794 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -453,6 +453,11 @@ type Router struct { // MaxBodyBytes bounds the request body the router will read. MaxBodyBytes int64 `koanf:"max_body_bytes"` + // Batch is the Files and Batch APIs (/v1/files, /v1/batches): a file of + // requests served at the lowest priority, its answers written to + // another. Needs the store. + Batch Batch `koanf:"batch"` + // TrustedProxies are the reverse proxies in front of the client port, // as addresses or CIDR prefixes. Their X-Forwarded-For, -Proto and -Host // are believed; anyone else's are not, since they decide the address @@ -553,6 +558,21 @@ type Endpoint struct { Priority int `koanf:"priority"` } +// Batch is how the Files and Batch APIs are bounded. +type Batch struct { + // MaxFileBytes bounds one uploaded file; 100 MB unless set. + MaxFileBytes int64 `koanf:"max_file_bytes"` + // MaxLines bounds the requests in one batch; 50 000 unless set. + MaxLines int `koanf:"max_lines"` + // Concurrency is how many of one batch's requests are in flight at + // once; 4 unless set. They queue in the lowest admission band, behind + // every interactive request. + Concurrency int `koanf:"concurrency"` + // RetainDays is how long a batch's answer and error files are kept; + // 30 unless set. Uploaded files are kept until deleted. + RetainDays int `koanf:"retain_days"` +} + // Default returns the configuration used when nothing is specified. func Default() Server { return Server{ diff --git a/internal/docgen/reference_gen.go b/internal/docgen/reference_gen.go index 9d01106d..3cdf0786 100644 --- a/internal/docgen/reference_gen.go +++ b/internal/docgen/reference_gen.go @@ -38,6 +38,12 @@ var Reference = []Block{ {Key: "buffer_size", Type: "int", Doc: "BufferSize bounds the queue. Beyond it records are dropped rather than\nblocking a request: losing accounting is bad, turning a stalled database\ninto an outage is worse.", Nested: ""}, {Key: "usage_retention", Type: "time.Duration", Doc: "UsageRetention is how long the record of each request is kept: who sent\nit, which replica answered, how long it queued — what `ctl request`\nreads. Zero keeps it for ever. Totals, charts and budgets are read from\nfive-minute sums kept beside the records, which are not pruned, so this\nbounds the table that grows by a row a request without changing any\nfigure a page shows. At least a day when set.", Nested: ""}, }}, + {Package: "config", Name: "Batch", Title: "Batch", Doc: "", Fields: []Field{ + {Key: "max_file_bytes", Type: "int64", Doc: "MaxFileBytes bounds one uploaded file; 100 MB unless set.", Nested: ""}, + {Key: "max_lines", Type: "int", Doc: "MaxLines bounds the requests in one batch; 50 000 unless set.", Nested: ""}, + {Key: "concurrency", Type: "int", Doc: "Concurrency is how many of one batch's requests are in flight at\nonce; 4 unless set. They queue in the lowest admission band, behind\nevery interactive request.", Nested: ""}, + {Key: "retain_days", Type: "int", Doc: "RetainDays is how long a batch's answer and error files are kept;\n30 unless set. Uploaded files are kept until deleted.", Nested: ""}, + }}, {Package: "catalog", Name: "Canary", Title: "Canary", Doc: "", Fields: []Field{ {Key: "model", Type: "string", Doc: "Model is the candidate: a catalog model, usually this one at newer\nweights, which clients cannot name themselves.", Nested: ""}, {Key: "percent", Type: "int", Doc: "Percent of requests go to it, 0 to 100.", Nested: ""}, @@ -240,6 +246,7 @@ var Reference = []Block{ {Key: "tenant_label", Type: "string", Doc: "TenantLabel is the level of the tree the per-tenant request series are\nlabelled by — llmgw_tenant_requests_total and its siblings: \"org\",\n\"team\" or \"project\", or \"none\" for no such series. A label per project\non a cluster of thousands is more series than Prometheus wants, and a\nteam is what an alert is usually about; \"team\" is the default.", Nested: ""}, {Key: "tag_labels", Type: "[]string", Doc: "TagLabels are the request tags (metadata, x-llmgw-tags) counted in\nllmgw_tagged_requests_total and llmgw_tagged_tokens_total, by value.\nOnly these: a tag is the client's to set, and every value of one is\na series. The first 100 values of each are counted apart, the rest as\n\"other\". Empty counts none; the usage records keep every tag anyway.", Nested: ""}, {Key: "max_body_bytes", Type: "int64", Doc: "MaxBodyBytes bounds the request body the router will read.", Nested: ""}, + {Key: "batch", Type: "Batch", Doc: "Batch is the Files and Batch APIs (/v1/files, /v1/batches): a file of\nrequests served at the lowest priority, its answers written to\nanother. Needs the store.", Nested: "config.Batch"}, {Key: "trusted_proxies", Type: "[]string", Doc: "TrustedProxies are the reverse proxies in front of the client port,\nas addresses or CIDR prefixes. Their X-Forwarded-For, -Proto and -Host\nare believed; anyone else's are not, since they decide the address\nwritten into the setup scripts and whose attempts are counted where.", Nested: ""}, {Key: "timeouts", Type: "Timeouts", Doc: "", Nested: "config.Timeouts"}, {Key: "scoring", Type: "Scoring", Doc: "", Nested: "config.Scoring"}, diff --git a/internal/router/batches.go b/internal/router/batches.go new file mode 100644 index 00000000..55373ec8 --- /dev/null +++ b/internal/router/batches.go @@ -0,0 +1,255 @@ +package router + +import ( + "bytes" + "encoding/json" + "errors" + "fmt" + "io" + "net/http" + "strconv" + "time" + + "gitea.mynodes.xyz/ilker/llmgw/internal/store" +) + +// The Batch API: a file of requests, served one by one at the lowest +// priority, its answers written to another file. OpenAI's shape, so its +// clients work unchanged. + +// batchEndpoints are what a batch may be of. +var batchEndpoints = map[string]bool{ + "/v1/chat/completions": true, + "/v1/completions": true, + "/v1/embeddings": true, +} + +// batchWindow is the only completion window, as OpenAI has it. +const batchWindow = "24h" + +func unixOrNil(t time.Time) any { + if t.IsZero() { + return nil + } + return t.Unix() +} + +func strOrNil(s string) any { + if s == "" { + return nil + } + return s +} + +func batchJSON(b store.Batch) map[string]any { + out := map[string]any{ + "id": b.ID, "object": "batch", "endpoint": b.Endpoint, "input_file_id": b.InputFileID, + "completion_window": b.CompletionWindow, "status": b.Status, + "output_file_id": strOrNil(b.OutputFileID), "error_file_id": strOrNil(b.ErrorFileID), + "created_at": b.CreatedAt.Unix(), "in_progress_at": unixOrNil(b.InProgressAt), + "expires_at": unixOrNil(b.ExpiresAt), "finalizing_at": unixOrNil(b.FinalizingAt), + "completed_at": unixOrNil(b.CompletedAt), "failed_at": unixOrNil(b.FailedAt), + "expired_at": unixOrNil(b.ExpiredAt), "cancelling_at": unixOrNil(b.CancellingAt), + "cancelled_at": unixOrNil(b.CancelledAt), + "request_counts": map[string]int{"total": b.Total, "completed": b.Completed, "failed": b.Failed}, + "errors": nil, "metadata": nil, + } + if b.Errors != "" { + out["errors"] = json.RawMessage(b.Errors) + } + if b.Metadata != "" { + out["metadata"] = json.RawMessage(b.Metadata) + } + return out +} + +func (r *Router) handleCreateBatch(w http.ResponseWriter, req *http.Request) { + p, ok := r.batchPrincipal(w, req) + if !ok { + return + } + var in struct { + InputFileID string `json:"input_file_id"` + Endpoint string `json:"endpoint"` + CompletionWindow string `json:"completion_window"` + Metadata json.RawMessage `json:"metadata"` + } + if err := json.NewDecoder(io.LimitReader(req.Body, 1<<16)).Decode(&in); err != nil { + writeError(w, http.StatusBadRequest, "invalid_request_error", "the request is not valid JSON: "+err.Error()) + return + } + if !batchEndpoints[in.Endpoint] { + writeError(w, http.StatusBadRequest, "invalid_request_error", + fmt.Sprintf("endpoint %q: a batch is of /v1/chat/completions, /v1/completions or /v1/embeddings", in.Endpoint)) + return + } + if in.CompletionWindow != batchWindow { + writeError(w, http.StatusBadRequest, "invalid_request_error", "completion_window is 24h") + return + } + meta := "" + if len(in.Metadata) > 0 && string(in.Metadata) != "null" { + if _, err := requestTags(in.Metadata, ""); err != nil { + writeError(w, http.StatusBadRequest, "invalid_request_error", err.Error()) + return + } + meta = string(in.Metadata) + } + f, err := r.store.FileByID(req.Context(), in.InputFileID) + if errors.Is(err, store.ErrNotFound) || err == nil && (f.Owner != ownerOf(p) || f.Purpose != purposeBatch) { + writeErrorCode(w, http.StatusBadRequest, "invalid_request_error", "file_not_found", + fmt.Sprintf("no batch file %q", in.InputFileID)) + return + } + if err != nil { + writeError(w, http.StatusInternalServerError, "server_error", err.Error()) + return + } + now := time.Now() + b := store.Batch{ID: newID("batch_"), Owner: ownerOf(p), KeyID: p.KeyID, Endpoint: in.Endpoint, + InputFileID: f.ID, CompletionWindow: batchWindow, Status: store.BatchValidating, + CreatedAt: now, ExpiresAt: now.Add(24 * time.Hour), Metadata: meta} + if err := r.store.PutBatch(req.Context(), b); err != nil { + writeError(w, http.StatusInternalServerError, "server_error", err.Error()) + return + } + r.wakeBatches() + writeJSON(w, http.StatusOK, batchJSON(b)) +} + +// ownedBatch is a batch the caller owns, or a 404 said. +func (r *Router) ownedBatch(w http.ResponseWriter, req *http.Request, owner string) (store.Batch, bool) { + b, err := r.store.BatchByID(req.Context(), req.PathValue("id")) + if errors.Is(err, store.ErrNotFound) || err == nil && b.Owner != owner { + writeErrorCode(w, http.StatusNotFound, "invalid_request_error", "batch_not_found", + fmt.Sprintf("no batch %q", req.PathValue("id"))) + return store.Batch{}, false + } + if err != nil { + writeError(w, http.StatusInternalServerError, "server_error", err.Error()) + return store.Batch{}, false + } + return b, true +} + +func (r *Router) handleGetBatch(w http.ResponseWriter, req *http.Request) { + p, ok := r.batchPrincipal(w, req) + if !ok { + return + } + if b, ok := r.ownedBatch(w, req, ownerOf(p)); ok { + writeJSON(w, http.StatusOK, batchJSON(b)) + } +} + +func (r *Router) handleCancelBatch(w http.ResponseWriter, req *http.Request) { + p, ok := r.batchPrincipal(w, req) + if !ok { + return + } + b, ok := r.ownedBatch(w, req, ownerOf(p)) + if !ok { + return + } + b, err := r.store.CancelBatch(req.Context(), b.ID, time.Now()) + if err != nil { + writeError(w, http.StatusInternalServerError, "server_error", err.Error()) + return + } + r.wakeBatches() + writeJSON(w, http.StatusOK, batchJSON(b)) +} + +func (r *Router) handleListBatches(w http.ResponseWriter, req *http.Request) { + p, ok := r.batchPrincipal(w, req) + if !ok { + return + } + limit, _ := strconv.Atoi(req.URL.Query().Get("limit")) + if limit <= 0 || limit > 100 { + limit = 20 + } + bs, err := r.store.Batches(req.Context(), ownerOf(p), req.URL.Query().Get("after"), limit+1) + if err != nil { + writeError(w, http.StatusInternalServerError, "server_error", err.Error()) + return + } + more := len(bs) > limit + if more { + bs = bs[:limit] + } + data := make([]map[string]any, 0, len(bs)) + for _, b := range bs { + data = append(data, batchJSON(b)) + } + out := map[string]any{"object": "list", "data": data, "has_more": more, "first_id": nil, "last_id": nil} + if len(bs) > 0 { + out["first_id"], out["last_id"] = bs[0].ID, bs[len(bs)-1].ID + } + writeJSON(w, http.StatusOK, out) +} + +// batchLine is one line of a batch's input file. +type batchLine struct { + CustomID string `json:"custom_id"` + Method string `json:"method"` + URL string `json:"url"` + Body json.RawMessage `json:"body"` +} + +// lineError is one reason a batch's input was refused, as OpenAI lists them. +type lineError struct { + Code string `json:"code"` + Message string `json:"message"` + Line int `json:"line,omitempty"` +} + +// parseBatchInput reads a batch's input file, or says every line that is +// wrong with it. +func parseBatchInput(content []byte, endpoint string, maxLines int) ([]batchLine, []lineError) { + var lines []batchLine + var errs []lineError + seen := map[string]bool{} + n := 0 + for raw := range bytes.SplitSeq(content, []byte("\n")) { + n++ + raw = bytes.TrimSpace(raw) + if len(raw) == 0 { + continue + } + bad := func(code, msg string) { + if len(errs) < 100 { + errs = append(errs, lineError{Code: code, Message: msg, Line: n}) + } + } + var l batchLine + if err := json.Unmarshal(raw, &l); err != nil { + bad("invalid_json_line", "not a JSON object: "+err.Error()) + continue + } + var body map[string]any + switch { + case l.CustomID == "": + bad("missing_custom_id", "custom_id is required") + case seen[l.CustomID]: + bad("duplicate_custom_id", fmt.Sprintf("custom_id %q is used twice", l.CustomID)) + case l.Method != http.MethodPost: + bad("invalid_method", "method is POST") + case l.URL != endpoint: + bad("mismatched_endpoint", fmt.Sprintf("url %q is not the batch's endpoint, %s", l.URL, endpoint)) + case json.Unmarshal(l.Body, &body) != nil || body["model"] == nil: + bad("invalid_body", "body is an object naming a model") + default: + seen[l.CustomID] = true + lines = append(lines, l) + } + } + if len(lines)+len(errs) == 0 { + errs = append(errs, lineError{Code: "empty_file", Message: "the file has no requests"}) + } + if len(lines) > maxLines { + errs = append(errs, lineError{Code: "too_many_requests", + Message: fmt.Sprintf("%d requests; a batch holds at most %d", len(lines), maxLines)}) + } + return lines, errs +} diff --git a/internal/router/batches_test.go b/internal/router/batches_test.go new file mode 100644 index 00000000..7048a4fc --- /dev/null +++ b/internal/router/batches_test.go @@ -0,0 +1,241 @@ +package router + +import ( + "bytes" + "crypto/sha256" + "encoding/json" + "fmt" + "mime/multipart" + "net/http" + "net/http/httptest" + "strings" + "testing" + "time" + + "gitea.mynodes.xyz/ilker/llmgw/internal/store" +) + +func upload(t *testing.T, h http.Handler, secret, content string) (int, map[string]any) { + t.Helper() + var buf bytes.Buffer + mw := multipart.NewWriter(&buf) + _ = mw.WriteField("purpose", "batch") + fw, _ := mw.CreateFormFile("file", "in.jsonl") + _, _ = fw.Write([]byte(content)) + _ = mw.Close() + req := httptest.NewRequest(http.MethodPost, "/v1/files", &buf) + req.Header.Set("Content-Type", mw.FormDataContentType()) + req.Header.Set("Authorization", "Bearer "+secret) + rec := httptest.NewRecorder() + h.ServeHTTP(rec, req) + var out map[string]any + _ = json.Unmarshal(rec.Body.Bytes(), &out) + return rec.Code, out +} + +func call(t *testing.T, h http.Handler, secret, method, path string, body any) (int, map[string]any, string) { + t.Helper() + var rd *bytes.Reader + if body != nil { + b, _ := json.Marshal(body) + rd = bytes.NewReader(b) + } else { + rd = bytes.NewReader(nil) + } + req := httptest.NewRequest(method, path, rd) + req.Header.Set("Authorization", "Bearer "+secret) + req.Header.Set("Content-Type", "application/json") + rec := httptest.NewRecorder() + h.ServeHTTP(rec, req) + var out map[string]any + _ = json.Unmarshal(rec.Body.Bytes(), &out) + return rec.Code, out, rec.Body.String() +} + +func waitBatch(t *testing.T, h http.Handler, secret, id string) map[string]any { + t.Helper() + deadline := time.Now().Add(10 * time.Second) + for { + _, b, _ := call(t, h, secret, http.MethodGet, "/v1/batches/"+id, nil) + if store.BatchDone(fmt.Sprint(b["status"])) { + return b + } + if time.Now().After(deadline) { + t.Fatalf("the batch did not end: %v", b) + } + time.Sleep(50 * time.Millisecond) + } +} + +func TestABatchIsServedAndItsAnswersWritten(t *testing.T) { + e := newFakeEngine(t, "qwen2.5-14b-instruct") + rt, secret := pricedRouter(t, e.srv.URL) + h := rt.Handler() + + var in strings.Builder + for i := range 5 { + fmt.Fprintf(&in, `{"custom_id":"r%d","method":"POST","url":"/v1/chat/completions","body":{"model":"q14","messages":[{"role":"user","content":"hi"}]}}`+"\n", i) + } + // One the engine cannot serve: a model that is not there. + in.WriteString(`{"custom_id":"bad","method":"POST","url":"/v1/chat/completions","body":{"model":"nope","messages":[]}}` + "\n") + code, f := upload(t, h, secret, in.String()) + if code != 200 || f["purpose"] != "batch" { + t.Fatalf("upload: %d %v", code, f) + } + code, b, raw := call(t, h, secret, http.MethodPost, "/v1/batches", map[string]any{ + "input_file_id": f["id"], "endpoint": "/v1/chat/completions", "completion_window": "24h", + "metadata": map[string]string{"job": "nightly"}}) + if code != 200 || b["status"] != "validating" { + t.Fatalf("create: %d %s", code, raw) + } + b = waitBatch(t, h, secret, b["id"].(string)) + counts := b["request_counts"].(map[string]any) + if b["status"] != "completed" || counts["total"] != 6.0 || counts["completed"] != 5.0 || counts["failed"] != 1.0 { + t.Fatalf("ended %v", b) + } + if e.calls.Load() != 5 { + t.Errorf("the engine served %d", e.calls.Load()) + } + _, _, out := call(t, h, secret, http.MethodGet, "/v1/files/"+b["output_file_id"].(string)+"/content", nil) + lines := strings.Split(strings.TrimSpace(out), "\n") + if len(lines) != 5 || !strings.Contains(lines[0], `"custom_id":"r`) || !strings.Contains(lines[0], `"status_code":200`) { + t.Errorf("output:\n%s", out) + } + _, _, errs := call(t, h, secret, http.MethodGet, "/v1/files/"+b["error_file_id"].(string)+"/content", nil) + if !strings.Contains(errs, `"custom_id":"bad"`) { + t.Errorf("errors:\n%s", errs) + } + // Charged like any request, and findable by the batch. + var rid string + for _, l := range lines { + var o struct { + Response struct { + RequestID string `json:"request_id"` + } `json:"response"` + } + _ = json.Unmarshal([]byte(l), &o) + rid = o.Response.RequestID + } + deadline := time.Now().Add(5 * time.Second) + for { + u, ok, _ := rt.store.UsageByRequest(t.Context(), rid) + if ok { + if u.Tags["batch"] != b["id"] || u.CostMicros == 0 { + t.Errorf("usage %+v", u) + } + break + } + if time.Now().After(deadline) { + t.Fatal("a batch request left no usage") + } + time.Sleep(20 * time.Millisecond) + } + _, list, _ := call(t, h, secret, http.MethodGet, "/v1/batches", nil) + if d := list["data"].([]any); len(d) != 1 { + t.Errorf("listed %v", list) + } +} + +func TestABatchWithBadLinesFailsAndSaysWhich(t *testing.T) { + e := newFakeEngine(t, "qwen2.5-14b-instruct") + rt, secret := pricedRouter(t, e.srv.URL) + h := rt.Handler() + _, f := upload(t, h, secret, `{"custom_id":"a","method":"POST","url":"/v1/embeddings","body":{"model":"q14"}} +not json +{"custom_id":"a","method":"POST","url":"/v1/chat/completions","body":{"model":"q14"}} +`) + _, b, _ := call(t, h, secret, http.MethodPost, "/v1/batches", map[string]any{ + "input_file_id": f["id"], "endpoint": "/v1/chat/completions", "completion_window": "24h"}) + b = waitBatch(t, h, secret, b["id"].(string)) + j, _ := json.Marshal(b["errors"]) + if b["status"] != "failed" || !strings.Contains(string(j), "mismatched_endpoint") || !strings.Contains(string(j), "invalid_json_line") { + t.Errorf("%v %s", b["status"], j) + } + if e.calls.Load() != 0 { + t.Error("a batch that failed validation was served") + } +} + +func TestBatchesAndFilesAreTheirOwners(t *testing.T) { + e := newFakeEngine(t, "qwen2.5-14b-instruct") + rt, secret := pricedRouter(t, e.srv.URL) + h := rt.Handler() + _, f := upload(t, h, secret, `{"custom_id":"a","method":"POST","url":"/v1/chat/completions","body":{"model":"q14"}}`) + + // Another project's key sees none of it. + org, _ := rt.store.CreateOrg(t.Context(), "other") + team, _ := rt.store.CreateTeam(t.Context(), org, "t") + project, _ := rt.store.CreateProject(t.Context(), team, "p") + other := store.KeyPrefix + strings.Repeat("b", 43) + sum := sha256.Sum256([]byte(other)) + if err := rt.store.CreateKey(t.Context(), store.Key{ID: "key-2", Hash: sum[:], Prefix: "bbbb", Name: "other", + ProjectID: project, CreatedAt: time.Now()}); err != nil { + t.Fatal(err) + } + if code, _, _ := call(t, h, other, http.MethodGet, "/v1/files/"+f["id"].(string), nil); code != 404 { + t.Errorf("another project read a file: %d", code) + } + if code, _, _ := call(t, h, other, http.MethodPost, "/v1/batches", map[string]any{ + "input_file_id": f["id"], "endpoint": "/v1/chat/completions", "completion_window": "24h"}); code != 400 { + t.Errorf("another project batched a file: %d", code) + } + if code, _, _ := call(t, h, secret, http.MethodDelete, "/v1/files/"+f["id"].(string), nil); code != 200 { + t.Errorf("delete: %d", code) + } +} + +func TestABatchPicksUpWhereItStoppedAndCanBeCancelled(t *testing.T) { + e := newFakeEngine(t, "qwen2.5-14b-instruct") + rt, secret := pricedRouter(t, e.srv.URL) + h := rt.Handler() + var in strings.Builder + for i := range 4 { + fmt.Fprintf(&in, `{"custom_id":"r%d","method":"POST","url":"/v1/chat/completions","body":{"model":"q14","messages":[]}}`+"\n", i) + } + _, f := upload(t, h, secret, in.String()) + + // As a gateway that died after serving the first line left it. + now := time.Now() + b := store.Batch{ID: "batch_resumed", Owner: "project:" + mustProjectOf(t, rt, "key-1"), KeyID: "key-1", + Endpoint: "/v1/chat/completions", InputFileID: f["id"].(string), CompletionWindow: "24h", + Status: store.BatchInProgress, CreatedAt: now, ExpiresAt: now.Add(24 * time.Hour)} + if err := rt.store.PutBatch(t.Context(), b); err != nil { + t.Fatal(err) + } + b.Total, b.Completed, b.InProgressAt = 4, 1, now + _ = rt.store.UpdateBatch(t.Context(), b) + _ = rt.store.PutBatchResult(t.Context(), b.ID, store.BatchResult{Line: 0, OK: true, + Data: `{"id":"batch_req_x","custom_id":"r0","response":{"status_code":200},"error":null}`}) + rt.wakeBatches() + got := waitBatch(t, h, secret, b.ID) + if c := got["request_counts"].(map[string]any); got["status"] != "completed" || c["completed"] != 4.0 { + t.Fatalf("%v", got) + } + if e.calls.Load() != 3 { + t.Errorf("served %d; the line kept already was served again", e.calls.Load()) + } + + // Cancelled before it is served: nothing is sent. + b2 := b + b2.ID, b2.Status = "batch_cancelled", store.BatchValidating + if err := rt.store.PutBatch(t.Context(), b2); err != nil { + t.Fatal(err) + } + _, _ = rt.store.CancelBatch(t.Context(), b2.ID, time.Now()) + rt.wakeBatches() + if got := waitBatch(t, h, secret, b2.ID); got["status"] != "cancelled" { + t.Errorf("%v", got) + } + if e.calls.Load() != 3 { + t.Errorf("a cancelled batch was served: %d", e.calls.Load()) + } +} + +func mustProjectOf(t *testing.T, rt *Router, keyID string) string { + t.Helper() + k, err := rt.store.KeyByID(t.Context(), keyID) + if err != nil { + t.Fatal(err) + } + return k.ProjectID +} diff --git a/internal/router/batchworker.go b/internal/router/batchworker.go new file mode 100644 index 00000000..1ba07711 --- /dev/null +++ b/internal/router/batchworker.go @@ -0,0 +1,373 @@ +package router + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "fmt" + "net/http" + "os" + "strconv" + "sync" + "time" + + "gitea.mynodes.xyz/ilker/llmgw/internal/auth" + "gitea.mynodes.xyz/ilker/llmgw/internal/store" +) + +// The worker that serves batches. Any gateway on the store may run one: a +// batch is worked on by whichever holds its lease, and each finished line +// is kept as it is served, so another picks up where one stopped. + +const ( + batchLease = 2 * time.Minute + batchPoll = 5 * time.Second +) + +// wakeBatches has the worker look for work now rather than at its next poll. +func (r *Router) wakeBatches() { + select { + case r.batchWake <- struct{}{}: + default: + } +} + +func (r *Router) runBatches(ctx context.Context) { + host, _ := os.Hostname() + owner := fmt.Sprintf("%s:%d:%s", host, os.Getpid(), newID("")) + t := time.NewTicker(batchPoll) + defer t.Stop() + for { + for { + now := time.Now() + b, ok, err := r.store.ClaimBatch(ctx, owner, now, now.Add(batchLease)) + if err != nil { + if ctx.Err() == nil { + r.log.Warn("could not look for batches to serve", "err", err) + } + break + } + if !ok { + break + } + r.workBatch(ctx, owner, b) + } + if _, _, _, days := r.batchCfg(); days > 0 { + if n, err := r.store.DeleteExpiredFiles(ctx, time.Now()); err == nil && n > 0 { + r.log.Info("batch files past their retention deleted", "files", n) + } + } + select { + case <-ctx.Done(): + return + case <-t.C: + case <-r.batchWake: + } + } +} + +// workBatch takes one batch as far as it goes: validated, served, written. +func (r *Router) workBatch(ctx context.Context, owner string, b store.Batch) { + bctx, stop := context.WithCancel(ctx) + defer stop() + defer func() { _ = r.store.ReleaseBatch(context.WithoutCancel(ctx), b.ID, owner) }() + // The lease kept while working; lost, the work stops. + go func() { + t := time.NewTicker(batchLease / 4) + defer t.Stop() + for { + select { + case <-bctx.Done(): + return + case <-t.C: + if held, err := r.store.RenewBatchLease(bctx, b.ID, owner, time.Now().Add(batchLease)); err == nil && !held { + stop() + return + } + } + } + }() + log := r.log.With("batch", b.ID) + if err := r.serveBatch(bctx, &b); err != nil && bctx.Err() == nil { + log.Warn("a batch stopped with an error; it is picked up again", "err", err) + } +} + +func (r *Router) serveBatch(ctx context.Context, b *store.Batch) error { + content, err := r.store.FileContent(ctx, b.InputFileID) + if err != nil && !errors.Is(err, store.ErrNotFound) { + return err + } + _, maxLines, concurrency, _ := r.batchCfg() + lines, errs := parseBatchInput(content, b.Endpoint, maxLines) + if errors.Is(err, store.ErrNotFound) { + errs = []lineError{{Code: "file_not_found", Message: "the input file was deleted"}} + } + + if b.Status == store.BatchValidating { + if len(errs) > 0 { + ej, _ := json.Marshal(map[string]any{"object": "list", "data": errs}) + b.Status, b.FailedAt, b.Errors = store.BatchFailed, time.Now(), string(ej) + return r.store.UpdateBatch(ctx, *b) + } + b.Status, b.Total, b.InProgressAt = store.BatchInProgress, len(lines), time.Now() + if err := r.store.UpdateBatch(ctx, *b); err != nil { + return err + } + } + + if b.Status == store.BatchInProgress { + end, err := r.serveLines(ctx, b, lines, concurrency) + if err != nil { + return err + } + switch end { + case store.BatchCancelling, store.BatchExpired: + b.Status = end + default: + b.Status, b.FinalizingAt = store.BatchFinalizing, time.Now() + if err := r.store.UpdateBatch(ctx, *b); err != nil { + return err + } + } + } + + // Finalizing, cancelling or expired: what was served is written out. + return r.finishBatch(ctx, b, lines) +} + +// serveLines serves the lines not yet served, a few at a time, and says why +// it stopped: "" when every line is served. +func (r *Router) serveLines(ctx context.Context, b *store.Batch, lines []batchLine, concurrency int) (string, error) { + done, err := r.store.BatchResults(ctx, b.ID) + if err != nil { + return "", err + } + served := map[int]bool{} + for _, d := range done { + served[d.Line] = true + } + p, perr := r.batchPrincipalOf(ctx, b.KeyID) + + var mu sync.Mutex + stopWith := "" + lastCheck := time.Now() + sem := make(chan struct{}, concurrency) + var wg sync.WaitGroup + for i, l := range lines { + if served[i] { + continue + } + mu.Lock() + stopped := stopWith != "" + mu.Unlock() + if stopped || ctx.Err() != nil { + break + } + // Asked to stop, or out of time: looked at now and then, not per line. + if time.Since(lastCheck) > 2*time.Second { + lastCheck = time.Now() + if cur, err := r.store.BatchByID(ctx, b.ID); err == nil && cur.Status == store.BatchCancelling { + mu.Lock() + stopWith = store.BatchCancelling + mu.Unlock() + break + } + if time.Now().After(b.ExpiresAt) { + mu.Lock() + stopWith = store.BatchExpired + mu.Unlock() + break + } + } + sem <- struct{}{} + wg.Add(1) + go func(i int, l batchLine) { + defer wg.Done() + defer func() { <-sem }() + res := r.serveLine(ctx, b, p, perr, l) + if ctx.Err() != nil { + return + } + if err := r.store.PutBatchResult(ctx, b.ID, res.withLine(i)); err != nil { + r.log.Warn("a batch's answer could not be kept", "batch", b.ID, "err", err) + return + } + mu.Lock() + if res.OK { + b.Completed++ + } else { + b.Failed++ + } + _ = r.store.UpdateBatch(ctx, *b) + mu.Unlock() + }(i, l) + } + wg.Wait() + if ctx.Err() != nil { + return "", ctx.Err() + } + if stopWith == "" { + // Counted from what is kept, which is the truth after a restart. + rs, err := r.store.BatchResults(ctx, b.ID) + if err != nil { + return "", err + } + if len(rs) < len(lines) { + return "", fmt.Errorf("%d of %d lines served", len(rs), len(lines)) + } + } + return stopWith, nil +} + +// batchPrincipalOf is the key a batch was made with, as it stands now: a key +// revoked since serves no more of it. +func (r *Router) batchPrincipalOf(ctx context.Context, keyID string) (auth.Principal, error) { + k, err := r.store.KeyByID(ctx, keyID) + if err != nil { + return auth.Principal{}, err + } + switch { + case !k.RevokedAt.IsZero(): + return auth.Principal{}, auth.ErrRevoked + case !k.ExpiresAt.IsZero() && time.Now().After(k.ExpiresAt): + return auth.Principal{}, auth.ErrExpired + } + eff, err := r.store.Resolve(ctx, k.ID) + if err != nil { + return auth.Principal{}, err + } + return auth.Principal{ + KeyID: k.ID, KeyPrefix: k.Prefix, UserID: eff.UserID, ProjectID: eff.ProjectID, + TeamID: eff.TeamID, OrgID: eff.OrgID, + Band: r.batchBand(), RoutingProfile: eff.RoutingProfile, Limits: eff, + }, nil +} + +// batchBand is the lowest admission band: a batch waits behind everyone. +func (r *Router) batchBand() string { + band, prio := "", 0 + for _, b := range r.cfg.Admission.Bands { + if band == "" || b.Priority < prio { + band, prio = b.Name, b.Priority + } + } + return band +} + +// lineResult is a served line as it is kept. +type lineResult struct { + OK bool + Data string +} + +func (l lineResult) withLine(i int) store.BatchResult { + return store.BatchResult{Line: i, OK: l.OK, Data: l.Data} +} + +// serveLine sends one request the way a client's is sent — limits, +// accounting, guardrails, admission in the lowest band — and keeps its +// answer in OpenAI's batch output form. A limit that lifts with time is +// waited out; one that does not fails the line. +func (r *Router) serveLine(ctx context.Context, b *store.Batch, p auth.Principal, perr error, l batchLine) lineResult { + id := newID("batch_req_") + out := map[string]any{"id": id, "custom_id": l.CustomID, "response": nil, "error": nil} + fail := func(code, msg string) lineResult { + out["error"] = map[string]string{"code": code, "message": msg} + j, _ := json.Marshal(out) + return lineResult{Data: string(j)} + } + if perr != nil { + return fail("key_unusable", "the key this batch was made with can no longer be used: "+perr.Error()) + } + body := withoutField(l.Body, "stream") + handler := r.handleInference(b.Endpoint) + for attempt := 1; ; attempt++ { + hreq, err := http.NewRequestWithContext(auth.WithPrincipal(ctx, p), http.MethodPost, b.Endpoint, bytes.NewReader(body)) + if err != nil { + return fail("invalid_request", err.Error()) + } + hreq.Header.Set("Content-Type", "application/json") + hreq.Header.Set("X-Request-Id", id) + hreq.Header.Set(tagsHeader, "batch="+b.ID) + w := &bufferWriter{header: http.Header{}} + handler(w, hreq) + status := w.status + if status == 0 { + status = http.StatusOK + } + // Waited out: a rate limit, a model loading, a queue full. Not a + // spent budget, which says no Retry-After because waiting does not + // help, and not past the batch's time. + if wait := w.header.Get("Retry-After"); wait != "" && (status == http.StatusTooManyRequests || status >= 500) && + time.Now().Before(b.ExpiresAt) && attempt < 1000 { + secs, _ := strconv.Atoi(wait) + select { + case <-ctx.Done(): + return fail("cancelled", "the gateway stopped") + case <-time.After(time.Duration(max(secs, 1)) * time.Second): + } + continue + } + var parsed any = json.RawMessage(bytes.TrimSpace(w.buf.Bytes())) + if !json.Valid(w.buf.Bytes()) { + parsed = w.buf.String() + } + out["response"] = map[string]any{"status_code": status, "request_id": id, "body": parsed} + j, _ := json.Marshal(out) + return lineResult{OK: status < 300, Data: string(j)} + } +} + +// finishBatch writes what was served — answers to one file, failures to +// another — and ends the batch: completed, cancelled or expired. +func (r *Router) finishBatch(ctx context.Context, b *store.Batch, lines []batchLine) error { + rs, err := r.store.BatchResults(ctx, b.ID) + if err != nil { + return err + } + var okBuf, errBuf bytes.Buffer + b.Completed, b.Failed = 0, 0 + for _, res := range rs { + if res.OK { + okBuf.WriteString(res.Data + "\n") + b.Completed++ + } else { + errBuf.WriteString(res.Data + "\n") + b.Failed++ + } + } + _, _, _, days := r.batchCfg() + now := time.Now() + keep := func(buf *bytes.Buffer, name string) (string, error) { + if buf.Len() == 0 { + return "", nil + } + f := store.File{ID: newID("file-"), Owner: b.Owner, Purpose: purposeBatchOutput, + Filename: b.ID + "_" + name + ".jsonl", CreatedAt: now, ExpiresAt: now.AddDate(0, 0, days)} + return f.ID, r.store.PutFile(ctx, f, buf.Bytes()) + } + if b.OutputFileID == "" { + if b.OutputFileID, err = keep(&okBuf, "output"); err != nil { + return err + } + } + if b.ErrorFileID == "" { + if b.ErrorFileID, err = keep(&errBuf, "error"); err != nil { + return err + } + } + switch b.Status { + case store.BatchCancelling: + b.Status, b.CancelledAt = store.BatchCancelled, now + case store.BatchExpired: + b.ExpiredAt = now + default: + b.Status, b.CompletedAt = store.BatchCompleted, now + } + if err := r.store.UpdateBatch(ctx, *b); err != nil { + return err + } + return r.store.DeleteBatchResults(ctx, b.ID) +} diff --git a/internal/router/files.go b/internal/router/files.go new file mode 100644 index 00000000..f4292293 --- /dev/null +++ b/internal/router/files.go @@ -0,0 +1,207 @@ +package router + +import ( + "crypto/rand" + "encoding/hex" + "errors" + "fmt" + "io" + "net/http" + "time" + + "gitea.mynodes.xyz/ilker/llmgw/internal/auth" + "gitea.mynodes.xyz/ilker/llmgw/internal/guard" + "gitea.mynodes.xyz/ilker/llmgw/internal/store" +) + +// The Files API, as far as batches need it: a client uploads a file of +// requests, and reads back the files a batch writes. + +const ( + purposeBatch = "batch" + purposeBatchOutput = "batch_output" +) + +func newID(prefix string) string { + var b [12]byte + _, _ = rand.Read(b[:]) + return prefix + hex.EncodeToString(b[:]) +} + +// ownerOf is who a file or a batch belongs to: the key's project, so every +// key of it sees them, or the person for a personal key. +func ownerOf(p auth.Principal) string { + switch { + case p.ProjectID != "": + return "project:" + p.ProjectID + case p.UserID != "": + return "user:" + p.UserID + } + return "key:" + p.KeyID +} + +// batchPrincipal is the caller, or a refusal when there is no store to keep +// files and batches in. +func (r *Router) batchPrincipal(w http.ResponseWriter, req *http.Request) (auth.Principal, bool) { + p, ok := auth.FromContext(req.Context()) + if !ok || r.store == nil { + writeError(w, http.StatusNotImplemented, "invalid_request_error", + "files and batches need authentication and the store, which this gateway runs without") + return auth.Principal{}, false + } + // A batch's files are prompts and answers kept: not under a policy + // that keeps nothing. + if r.noStore(guard.Subject{KeyID: p.KeyID, UserID: p.UserID, ProjectID: p.ProjectID, + TeamID: p.TeamID, OrgID: p.OrgID, Path: req.URL.Path}) { + writeError(w, http.StatusForbidden, "invalid_request_error", + "the gateway's policy for this caller keeps nothing, and a batch is prompts and answers kept") + return auth.Principal{}, false + } + return p, true +} + +func (r *Router) batchCfg() (maxBytes int64, maxLines, concurrency, retainDays int) { + b := r.cfg.Batch + maxBytes, maxLines, concurrency, retainDays = b.MaxFileBytes, b.MaxLines, b.Concurrency, b.RetainDays + if maxBytes <= 0 { + maxBytes = 100 << 20 + } + if maxLines <= 0 { + maxLines = 50000 + } + if concurrency <= 0 { + concurrency = 4 + } + if retainDays <= 0 { + retainDays = 30 + } + return +} + +func fileJSON(f store.File) map[string]any { + out := map[string]any{"id": f.ID, "object": "file", "bytes": f.Bytes, "created_at": f.CreatedAt.Unix(), + "filename": f.Filename, "purpose": f.Purpose, "status": "processed"} + if !f.ExpiresAt.IsZero() { + out["expires_at"] = f.ExpiresAt.Unix() + } + return out +} + +func (r *Router) handleUploadFile(w http.ResponseWriter, req *http.Request) { + p, ok := r.batchPrincipal(w, req) + if !ok { + return + } + maxBytes, _, _, _ := r.batchCfg() + req.Body = http.MaxBytesReader(w, req.Body, maxBytes+1<<20) + if err := req.ParseMultipartForm(32 << 20); err != nil { + writeError(w, http.StatusBadRequest, "invalid_request_error", + fmt.Sprintf("a multipart form with purpose and file, at most %d MB: %v", maxBytes>>20, err)) + return + } + if purpose := req.FormValue("purpose"); purpose != purposeBatch { + writeError(w, http.StatusBadRequest, "invalid_request_error", + fmt.Sprintf("purpose %q: this gateway keeps files for batches only (purpose batch)", purpose)) + return + } + file, header, err := req.FormFile("file") + if err != nil { + writeError(w, http.StatusBadRequest, "invalid_request_error", "no file in the form: "+err.Error()) + return + } + defer file.Close() + content, err := io.ReadAll(io.LimitReader(file, maxBytes+1)) + if err != nil { + writeError(w, http.StatusBadRequest, "invalid_request_error", err.Error()) + return + } + if int64(len(content)) > maxBytes { + writeError(w, http.StatusRequestEntityTooLarge, "invalid_request_error", + fmt.Sprintf("the file is larger than %d MB", maxBytes>>20)) + return + } + f := store.File{ID: newID("file-"), Owner: ownerOf(p), Purpose: purposeBatch, + Filename: header.Filename, Bytes: int64(len(content)), CreatedAt: time.Now()} + if err := r.store.PutFile(req.Context(), f, content); err != nil { + writeError(w, http.StatusInternalServerError, "server_error", err.Error()) + return + } + writeJSON(w, http.StatusOK, fileJSON(f)) +} + +// ownedFile is a file the caller owns, or a 404 said. +func (r *Router) ownedFile(w http.ResponseWriter, req *http.Request, p auth.Principal) (store.File, bool) { + f, err := r.store.FileByID(req.Context(), req.PathValue("id")) + if errors.Is(err, store.ErrNotFound) || err == nil && f.Owner != ownerOf(p) { + writeErrorCode(w, http.StatusNotFound, "invalid_request_error", "file_not_found", + fmt.Sprintf("no file %q", req.PathValue("id"))) + return store.File{}, false + } + if err != nil { + writeError(w, http.StatusInternalServerError, "server_error", err.Error()) + return store.File{}, false + } + return f, true +} + +func (r *Router) handleListFiles(w http.ResponseWriter, req *http.Request) { + p, ok := r.batchPrincipal(w, req) + if !ok { + return + } + fs, err := r.store.Files(req.Context(), ownerOf(p), req.URL.Query().Get("purpose"), 0) + if err != nil { + writeError(w, http.StatusInternalServerError, "server_error", err.Error()) + return + } + data := make([]map[string]any, 0, len(fs)) + for _, f := range fs { + data = append(data, fileJSON(f)) + } + writeJSON(w, http.StatusOK, map[string]any{"object": "list", "data": data}) +} + +func (r *Router) handleGetFile(w http.ResponseWriter, req *http.Request) { + p, ok := r.batchPrincipal(w, req) + if !ok { + return + } + if f, ok := r.ownedFile(w, req, p); ok { + writeJSON(w, http.StatusOK, fileJSON(f)) + } +} + +func (r *Router) handleFileContent(w http.ResponseWriter, req *http.Request) { + p, ok := r.batchPrincipal(w, req) + if !ok { + return + } + f, ok := r.ownedFile(w, req, p) + if !ok { + return + } + b, err := r.store.FileContent(req.Context(), f.ID) + if err != nil { + writeError(w, http.StatusInternalServerError, "server_error", err.Error()) + return + } + w.Header().Set("Content-Type", "application/octet-stream") + w.Header().Set("Content-Disposition", fmt.Sprintf("attachment; filename=%q", f.Filename)) + _, _ = w.Write(b) +} + +func (r *Router) handleDeleteFile(w http.ResponseWriter, req *http.Request) { + p, ok := r.batchPrincipal(w, req) + if !ok { + return + } + f, ok := r.ownedFile(w, req, p) + if !ok { + return + } + if _, err := r.store.DeleteFile(req.Context(), f.ID); err != nil { + writeError(w, http.StatusInternalServerError, "server_error", err.Error()) + return + } + writeJSON(w, http.StatusOK, map[string]any{"id": f.ID, "object": "file", "deleted": true}) +} diff --git a/internal/router/handlers.go b/internal/router/handlers.go index 7cc1a4c8..790a2bd6 100644 --- a/internal/router/handlers.go +++ b/internal/router/handlers.go @@ -100,6 +100,16 @@ func (r *Router) Handler() http.Handler { // chat completions on the way in and back on the way out. mux.HandleFunc("POST /v1/responses", r.handleResponses) mux.HandleFunc("GET /v1/models", r.handleModels) + // Files and batches: a file of requests served at the lowest priority. + mux.HandleFunc("POST /v1/files", r.handleUploadFile) + mux.HandleFunc("GET /v1/files", r.handleListFiles) + mux.HandleFunc("GET /v1/files/{id}", r.handleGetFile) + mux.HandleFunc("GET /v1/files/{id}/content", r.handleFileContent) + mux.HandleFunc("DELETE /v1/files/{id}", r.handleDeleteFile) + mux.HandleFunc("POST /v1/batches", r.handleCreateBatch) + mux.HandleFunc("GET /v1/batches", r.handleListBatches) + mux.HandleFunc("GET /v1/batches/{id}", r.handleGetBatch) + mux.HandleFunc("POST /v1/batches/{id}/cancel", r.handleCancelBatch) // Liveness: this process is up. It says nothing about whether it can // serve, which is why it is not the thing to put a load balancer on. mux.HandleFunc("GET /health", func(w http.ResponseWriter, _ *http.Request) { diff --git a/internal/router/router.go b/internal/router/router.go index 77fa7ac1..f6739853 100644 --- a/internal/router/router.go +++ b/internal/router/router.go @@ -68,6 +68,8 @@ type Router struct { // canaries counts how the two sides served. canaryTargets atomic.Pointer[map[string]string] canaries canaryStats + // batchWake has the batch worker look for work at once. + batchWake chan struct{} // extraMetrics are collectors other components publish on /metrics. extraMetrics []prometheus.Collector @@ -240,13 +242,14 @@ func New(cfg config.Server, log *slog.Logger) (*Router, error) { } r := &Router{ - cfg: cfg.Router, - landing: cfg.Landing, - log: log, - reg: reg, - profiles: profiles, - prefix: prefixScorer, - m: metrics, + batchWake: make(chan struct{}, 1), + cfg: cfg.Router, + landing: cfg.Landing, + log: log, + reg: reg, + profiles: profiles, + prefix: prefixScorer, + m: metrics, px: proxy.New(proxy.Config{ DialTimeout: cfg.Router.Timeouts.Dial, ResponseHeaderTimeout: cfg.Router.Timeouts.ResponseHeader, @@ -444,6 +447,7 @@ func (r *Router) Start(ctx context.Context) { go r.pruneUsage(ctx, r.usageRetention) } if r.store != nil { + go r.runBatches(ctx) go r.refreshTenants(ctx) go r.purgeSessions(ctx) } diff --git a/internal/store/batches.go b/internal/store/batches.go new file mode 100644 index 00000000..cbe37d83 --- /dev/null +++ b/internal/store/batches.go @@ -0,0 +1,345 @@ +package store + +import ( + "context" + "database/sql" + "errors" + "time" +) + +// File is an uploaded or written file of the Files API, without its bytes. +type File struct { + ID string + Owner string + Purpose string + Filename string + Bytes int64 + CreatedAt time.Time + // ExpiresAt is when it is deleted by itself; zero is never. + ExpiresAt time.Time +} + +// PutFile keeps a file and its bytes. +func (s *DB) PutFile(ctx context.Context, f File, content []byte) error { + _, err := s.db.ExecContext(ctx, `INSERT INTO files (id, owner, purpose, filename, bytes, content, created_at, expires_at) + VALUES (?, ?, ?, ?, ?, ?, ?, ?)`, f.ID, f.Owner, f.Purpose, f.Filename, int64(len(content)), content, + millis(f.CreatedAt), optMillis(f.ExpiresAt)) + return err +} + +const fileCols = `id, owner, purpose, filename, bytes, created_at, expires_at` + +func scanFile(sc interface{ Scan(...any) error }) (File, error) { + var f File + var created, expires int64 + if err := sc.Scan(&f.ID, &f.Owner, &f.Purpose, &f.Filename, &f.Bytes, &created, &expires); err != nil { + return File{}, err + } + f.CreatedAt = time.UnixMilli(created) + if expires > 0 { + f.ExpiresAt = time.UnixMilli(expires) + } + return f, nil +} + +// FileByID finds a file, ErrNotFound when there is none. +func (s *DB) FileByID(ctx context.Context, id string) (File, error) { + f, err := scanFile(s.db.QueryRowContext(ctx, `SELECT `+fileCols+` FROM files WHERE id = ?`, id)) + if errors.Is(err, sql.ErrNoRows) { + return File{}, ErrNotFound + } + return f, err +} + +// FileContent is a file's bytes. +func (s *DB) FileContent(ctx context.Context, id string) ([]byte, error) { + var b []byte + err := s.db.QueryRowContext(ctx, `SELECT content FROM files WHERE id = ?`, id).Scan(&b) + if errors.Is(err, sql.ErrNoRows) { + return nil, ErrNotFound + } + return b, err +} + +// Files lists an owner's files, the newest first; purpose narrows it. +func (s *DB) Files(ctx context.Context, owner, purpose string, limit int) ([]File, error) { + q := `SELECT ` + fileCols + ` FROM files WHERE owner = ?` + args := []any{owner} + if purpose != "" { + q += ` AND purpose = ?` + args = append(args, purpose) + } + if limit <= 0 || limit > 10000 { + limit = 10000 + } + q += ` ORDER BY created_at DESC LIMIT ?` + args = append(args, limit) + rows, err := s.db.QueryContext(ctx, q, args...) + if err != nil { + return nil, err + } + defer rows.Close() + var out []File + for rows.Next() { + f, err := scanFile(rows) + if err != nil { + return nil, err + } + out = append(out, f) + } + return out, rows.Err() +} + +// DeleteFile removes a file; false when there was none. +func (s *DB) DeleteFile(ctx context.Context, id string) (bool, error) { + res, err := s.db.ExecContext(ctx, `DELETE FROM files WHERE id = ?`, id) + if err != nil { + return false, err + } + n, _ := res.RowsAffected() + return n > 0, nil +} + +// DeleteExpiredFiles removes the files past their expiry. +func (s *DB) DeleteExpiredFiles(ctx context.Context, now time.Time) (int64, error) { + res, err := s.db.ExecContext(ctx, `DELETE FROM files WHERE expires_at > 0 AND expires_at < ?`, millis(now)) + if err != nil { + return 0, err + } + return res.RowsAffected() +} + +// Batch is one batch of the Batch API. +type Batch struct { + ID string + Owner string + KeyID string + Endpoint string + InputFileID string + CompletionWindow string + Status string + OutputFileID string + ErrorFileID string + Total int + Completed int + Failed int + CreatedAt time.Time + InProgressAt time.Time + FinalizingAt time.Time + CompletedAt time.Time + FailedAt time.Time + ExpiredAt time.Time + CancellingAt time.Time + CancelledAt time.Time + ExpiresAt time.Time + // Metadata is the client's, as JSON; Errors why validation failed, as + // JSON. + Metadata string + Errors string +} + +// Batch states, as OpenAI names them. +const ( + BatchValidating = "validating" + BatchFailed = "failed" + BatchInProgress = "in_progress" + BatchFinalizing = "finalizing" + BatchCompleted = "completed" + BatchExpired = "expired" + BatchCancelling = "cancelling" + BatchCancelled = "cancelled" +) + +// BatchDone is whether a state is an end. +func BatchDone(status string) bool { + switch status { + case BatchFailed, BatchCompleted, BatchExpired, BatchCancelled: + return true + } + return false +} + +const batchCols = `id, owner, key_id, endpoint, input_file_id, completion_window, status, output_file_id, + error_file_id, total, completed, failed, created_at, in_progress_at, finalizing_at, completed_at, + failed_at, expired_at, cancelling_at, cancelled_at, expires_at, metadata, errors` + +func scanBatch(sc interface{ Scan(...any) error }) (Batch, error) { + var b Batch + var t [10]int64 + err := sc.Scan(&b.ID, &b.Owner, &b.KeyID, &b.Endpoint, &b.InputFileID, &b.CompletionWindow, &b.Status, + &b.OutputFileID, &b.ErrorFileID, &b.Total, &b.Completed, &b.Failed, + &t[0], &t[1], &t[2], &t[3], &t[4], &t[5], &t[6], &t[7], &t[8], &b.Metadata, &b.Errors) + if err != nil { + return Batch{}, err + } + at := func(v int64) time.Time { + if v == 0 { + return time.Time{} + } + return time.UnixMilli(v) + } + b.CreatedAt, b.InProgressAt, b.FinalizingAt, b.CompletedAt = at(t[0]), at(t[1]), at(t[2]), at(t[3]) + b.FailedAt, b.ExpiredAt, b.CancellingAt, b.CancelledAt, b.ExpiresAt = at(t[4]), at(t[5]), at(t[6]), at(t[7]), at(t[8]) + return b, nil +} + +// PutBatch records a new batch. +func (s *DB) PutBatch(ctx context.Context, b Batch) error { + _, err := s.db.ExecContext(ctx, `INSERT INTO batches (id, owner, key_id, endpoint, input_file_id, + completion_window, status, created_at, expires_at, metadata) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, + b.ID, b.Owner, b.KeyID, b.Endpoint, b.InputFileID, b.CompletionWindow, b.Status, + millis(b.CreatedAt), millis(b.ExpiresAt), b.Metadata) + return err +} + +// UpdateBatch writes a batch's progress: its state, counts, files, times +// and errors. +func (s *DB) UpdateBatch(ctx context.Context, b Batch) error { + _, err := s.db.ExecContext(ctx, `UPDATE batches SET status = ?, output_file_id = ?, error_file_id = ?, + total = ?, completed = ?, failed = ?, in_progress_at = ?, finalizing_at = ?, completed_at = ?, + failed_at = ?, expired_at = ?, cancelling_at = ?, cancelled_at = ?, errors = ? WHERE id = ?`, + b.Status, b.OutputFileID, b.ErrorFileID, b.Total, b.Completed, b.Failed, + optMillis(b.InProgressAt), optMillis(b.FinalizingAt), optMillis(b.CompletedAt), optMillis(b.FailedAt), + optMillis(b.ExpiredAt), optMillis(b.CancellingAt), optMillis(b.CancelledAt), b.Errors, b.ID) + return err +} + +// CancelBatch asks for a batch to stop: cancelling, unless it has ended +// already. It says the batch as it stands. +func (s *DB) CancelBatch(ctx context.Context, id string, now time.Time) (Batch, error) { + if _, err := s.db.ExecContext(ctx, `UPDATE batches SET status = ?, cancelling_at = ? + WHERE id = ? AND status IN (?, ?, ?)`, BatchCancelling, millis(now), id, + BatchValidating, BatchInProgress, BatchFinalizing); err != nil { + return Batch{}, err + } + return s.BatchByID(ctx, id) +} + +// BatchByID finds a batch, ErrNotFound when there is none. +func (s *DB) BatchByID(ctx context.Context, id string) (Batch, error) { + b, err := scanBatch(s.db.QueryRowContext(ctx, `SELECT `+batchCols+` FROM batches WHERE id = ?`, id)) + if errors.Is(err, sql.ErrNoRows) { + return Batch{}, ErrNotFound + } + return b, err +} + +// Batches lists an owner's batches, the newest first, those made before +// the batch after when it is given. +func (s *DB) Batches(ctx context.Context, owner, after string, limit int) ([]Batch, error) { + q := `SELECT ` + batchCols + ` FROM batches WHERE owner = ?` + args := []any{owner} + if after != "" { + q += ` AND created_at < (SELECT created_at FROM batches WHERE id = ?)` + args = append(args, after) + } + if limit <= 0 || limit > 100 { + limit = 20 + } + q += ` ORDER BY created_at DESC LIMIT ?` + args = append(args, limit) + rows, err := s.db.QueryContext(ctx, q, args...) + if err != nil { + return nil, err + } + defer rows.Close() + var out []Batch + for rows.Next() { + b, err := scanBatch(rows) + if err != nil { + return nil, err + } + out = append(out, b) + } + return out, rows.Err() +} + +// ClaimBatch takes the lease on one batch that has work left and that no +// gateway holds, for owner until the time given. ok is false when there is +// none. +func (s *DB) ClaimBatch(ctx context.Context, owner string, now, until time.Time) (Batch, bool, error) { + var id string + err := s.db.QueryRowContext(ctx, `UPDATE batches SET lease_owner = ?, lease_until = ? + WHERE id = (SELECT id FROM batches WHERE status IN (?, ?, ?, ?) AND lease_until < ? + ORDER BY created_at LIMIT 1) + AND lease_until < ? + RETURNING id`, owner, millis(until), BatchValidating, BatchInProgress, BatchFinalizing, BatchCancelling, + millis(now), millis(now)).Scan(&id) + if errors.Is(err, sql.ErrNoRows) { + return Batch{}, false, nil + } + if err != nil { + return Batch{}, false, err + } + b, err := s.BatchByID(ctx, id) + return b, err == nil, err +} + +// RenewBatchLease keeps a lease held; false when it is held no longer. +func (s *DB) RenewBatchLease(ctx context.Context, id, owner string, until time.Time) (bool, error) { + res, err := s.db.ExecContext(ctx, `UPDATE batches SET lease_until = ? WHERE id = ? AND lease_owner = ?`, + millis(until), id, owner) + if err != nil { + return false, err + } + n, _ := res.RowsAffected() + return n > 0, nil +} + +// ReleaseBatch gives a lease up. +func (s *DB) ReleaseBatch(ctx context.Context, id, owner string) error { + _, err := s.db.ExecContext(ctx, `UPDATE batches SET lease_until = 0, lease_owner = '' WHERE id = ? AND lease_owner = ?`, id, owner) + return err +} + +// BatchResult is one line of a batch, served. +type BatchResult struct { + Line int + OK bool + Data string +} + +// PutBatchResult keeps one served line; a line kept already is left as it +// was. +func (s *DB) PutBatchResult(ctx context.Context, batchID string, r BatchResult) error { + ok := 0 + if r.OK { + ok = 1 + } + _, err := s.db.ExecContext(ctx, `INSERT INTO batch_results (batch_id, line, ok, data) VALUES (?, ?, ?, ?) + ON CONFLICT (batch_id, line) DO NOTHING`, batchID, r.Line, ok, r.Data) + return err +} + +// BatchResults are a batch's served lines, in order. +func (s *DB) BatchResults(ctx context.Context, batchID string) ([]BatchResult, error) { + rows, err := s.db.QueryContext(ctx, `SELECT line, ok, data FROM batch_results WHERE batch_id = ? ORDER BY line`, batchID) + if err != nil { + return nil, err + } + defer rows.Close() + var out []BatchResult + for rows.Next() { + var r BatchResult + var ok int + if err := rows.Scan(&r.Line, &ok, &r.Data); err != nil { + return nil, err + } + r.OK = ok == 1 + out = append(out, r) + } + return out, rows.Err() +} + +// DeleteBatchResults forgets a batch's lines, once its files are written. +func (s *DB) DeleteBatchResults(ctx context.Context, batchID string) error { + _, err := s.db.ExecContext(ctx, `DELETE FROM batch_results WHERE batch_id = ?`, batchID) + return err +} + +func optMillis(t time.Time) int64 { + if t.IsZero() { + return 0 + } + return t.UnixMilli() +} diff --git a/internal/store/batches_test.go b/internal/store/batches_test.go new file mode 100644 index 00000000..45dd72d5 --- /dev/null +++ b/internal/store/batches_test.go @@ -0,0 +1,80 @@ +package store + +import ( + "errors" + "testing" + "time" +) + +func TestFilesAndBatches(t *testing.T) { + checkFilesAndBatches(t, capacityStore(t)) +} + +func TestPostgresFilesAndBatches(t *testing.T) { + checkFilesAndBatches(t, openPG(t)) +} + +func checkFilesAndBatches(t *testing.T, db *DB) { + ctx := t.Context() + now := time.Now() + if err := db.PutFile(ctx, File{ID: "file-1", Owner: "p1", Purpose: "batch", Filename: "in.jsonl", CreatedAt: now}, + []byte("{\"a\":1}\n")); err != nil { + t.Fatal(err) + } + if err := db.PutFile(ctx, File{ID: "file-old", Owner: "p1", Purpose: "batch_output", Filename: "out.jsonl", + CreatedAt: now, ExpiresAt: now.Add(-time.Minute)}, []byte("x")); err != nil { + t.Fatal(err) + } + f, err := db.FileByID(ctx, "file-1") + if err != nil || f.Bytes != 8 || f.Owner != "p1" { + t.Fatalf("%+v %v", f, err) + } + if b, _ := db.FileContent(ctx, "file-1"); string(b) != "{\"a\":1}\n" { + t.Errorf("content %q", b) + } + if fs, _ := db.Files(ctx, "p1", "batch", 0); len(fs) != 1 { + t.Errorf("listed %v", fs) + } + if n, _ := db.DeleteExpiredFiles(ctx, now); n != 1 { + t.Errorf("expired %d", n) + } + + b := Batch{ID: "batch-1", Owner: "p1", KeyID: "k", Endpoint: "/v1/chat/completions", InputFileID: "file-1", + CompletionWindow: "24h", Status: BatchValidating, CreatedAt: now, ExpiresAt: now.Add(24 * time.Hour)} + if err := db.PutBatch(ctx, b); err != nil { + t.Fatal(err) + } + got, ok, err := db.ClaimBatch(ctx, "gw-a", now, now.Add(time.Minute)) + if err != nil || !ok || got.ID != "batch-1" { + t.Fatalf("claim: %+v %v %v", got, ok, err) + } + if _, ok, _ := db.ClaimBatch(ctx, "gw-b", now, now.Add(time.Minute)); ok { + t.Error("a leased batch was claimed twice") + } + if held, _ := db.RenewBatchLease(ctx, "batch-1", "gw-b", now.Add(time.Hour)); held { + t.Error("another gateway renewed the lease") + } + got.Status, got.Total, got.InProgressAt = BatchInProgress, 2, now + if err := db.UpdateBatch(ctx, got); err != nil { + t.Fatal(err) + } + for _, r := range []BatchResult{{Line: 1, OK: false, Data: "e"}, {Line: 0, OK: true, Data: "a"}, {Line: 0, OK: false, Data: "dup"}} { + if err := db.PutBatchResult(ctx, "batch-1", r); err != nil { + t.Fatal(err) + } + } + rs, _ := db.BatchResults(ctx, "batch-1") + if len(rs) != 2 || rs[0].Data != "a" || !rs[0].OK || rs[1].OK { + t.Errorf("results %+v", rs) + } + c, err := db.CancelBatch(ctx, "batch-1", now) + if err != nil || c.Status != BatchCancelling || c.Total != 2 || c.CancellingAt.IsZero() { + t.Errorf("cancel %+v %v", c, err) + } + if bs, _ := db.Batches(ctx, "p1", "", 0); len(bs) != 1 { + t.Errorf("listed %d", len(bs)) + } + if _, err := db.BatchByID(ctx, "nope"); !errors.Is(err, ErrNotFound) { + t.Errorf("missing: %v", err) + } +} diff --git a/internal/store/copy.go b/internal/store/copy.go index 8dbea3d2..a7a630c5 100644 --- a/internal/store/copy.go +++ b/internal/store/copy.go @@ -58,6 +58,9 @@ var copyOrder = []string{ // Who changed what is not lost by moving the store. "audit_events", "key_ttl", + "files", + "batches", + "batch_results", } // notCopied are tables deliberately left behind. diff --git a/internal/store/migrations/postgres/0024_batches.sql b/internal/store/migrations/postgres/0024_batches.sql new file mode 100644 index 00000000..60e7d247 --- /dev/null +++ b/internal/store/migrations/postgres/0024_batches.sql @@ -0,0 +1,56 @@ +-- Ported from the SQLite migration of the same number. +-- The Files and Batch APIs. A file is what a client uploaded (a batch's +-- requests) or what a batch wrote (its answers, its errors), owned by the +-- project of the key that made it. A batch is a file of requests served one +-- by one at the lowest priority; each finished line is kept as it is +-- served, so a batch survives the control plane restarting under it, and +-- the lease says which gateway is working on it. +CREATE TABLE files ( + id TEXT PRIMARY KEY, + owner TEXT NOT NULL, + purpose TEXT NOT NULL, + filename TEXT NOT NULL, + bytes BIGINT NOT NULL, + content BYTEA NOT NULL, + created_at BIGINT NOT NULL, + expires_at BIGINT NOT NULL DEFAULT 0 +); +CREATE INDEX files_owner ON files (owner, created_at); + +CREATE TABLE batches ( + id TEXT PRIMARY KEY, + owner TEXT NOT NULL, + key_id TEXT NOT NULL, + endpoint TEXT NOT NULL, + input_file_id TEXT NOT NULL, + completion_window TEXT NOT NULL, + status TEXT NOT NULL, + output_file_id TEXT NOT NULL DEFAULT '', + error_file_id TEXT NOT NULL DEFAULT '', + total BIGINT NOT NULL DEFAULT 0, + completed BIGINT NOT NULL DEFAULT 0, + failed BIGINT NOT NULL DEFAULT 0, + created_at BIGINT NOT NULL, + in_progress_at BIGINT NOT NULL DEFAULT 0, + finalizing_at BIGINT NOT NULL DEFAULT 0, + completed_at BIGINT NOT NULL DEFAULT 0, + failed_at BIGINT NOT NULL DEFAULT 0, + expired_at BIGINT NOT NULL DEFAULT 0, + cancelling_at BIGINT NOT NULL DEFAULT 0, + cancelled_at BIGINT NOT NULL DEFAULT 0, + expires_at BIGINT NOT NULL, + metadata TEXT NOT NULL DEFAULT '', + errors TEXT NOT NULL DEFAULT '', + lease_owner TEXT NOT NULL DEFAULT '', + lease_until BIGINT NOT NULL DEFAULT 0 +); +CREATE INDEX batches_owner ON batches (owner, created_at); +CREATE INDEX batches_status ON batches (status, lease_until); + +CREATE TABLE batch_results ( + batch_id TEXT NOT NULL, + line BIGINT NOT NULL, + ok BIGINT NOT NULL, + data TEXT NOT NULL, + PRIMARY KEY (batch_id, line) +); diff --git a/internal/store/migrations/sqlite/0024_batches.sql b/internal/store/migrations/sqlite/0024_batches.sql new file mode 100644 index 00000000..12931212 --- /dev/null +++ b/internal/store/migrations/sqlite/0024_batches.sql @@ -0,0 +1,55 @@ +-- The Files and Batch APIs. A file is what a client uploaded (a batch's +-- requests) or what a batch wrote (its answers, its errors), owned by the +-- project of the key that made it. A batch is a file of requests served one +-- by one at the lowest priority; each finished line is kept as it is +-- served, so a batch survives the control plane restarting under it, and +-- the lease says which gateway is working on it. +CREATE TABLE files ( + id TEXT PRIMARY KEY, + owner TEXT NOT NULL, + purpose TEXT NOT NULL, + filename TEXT NOT NULL, + bytes INTEGER NOT NULL, + content BLOB NOT NULL, + created_at INTEGER NOT NULL, + expires_at INTEGER NOT NULL DEFAULT 0 +); +CREATE INDEX files_owner ON files (owner, created_at); + +CREATE TABLE batches ( + id TEXT PRIMARY KEY, + owner TEXT NOT NULL, + key_id TEXT NOT NULL, + endpoint TEXT NOT NULL, + input_file_id TEXT NOT NULL, + completion_window TEXT NOT NULL, + status TEXT NOT NULL, + output_file_id TEXT NOT NULL DEFAULT '', + error_file_id TEXT NOT NULL DEFAULT '', + total INTEGER NOT NULL DEFAULT 0, + completed INTEGER NOT NULL DEFAULT 0, + failed INTEGER NOT NULL DEFAULT 0, + created_at INTEGER NOT NULL, + in_progress_at INTEGER NOT NULL DEFAULT 0, + finalizing_at INTEGER NOT NULL DEFAULT 0, + completed_at INTEGER NOT NULL DEFAULT 0, + failed_at INTEGER NOT NULL DEFAULT 0, + expired_at INTEGER NOT NULL DEFAULT 0, + cancelling_at INTEGER NOT NULL DEFAULT 0, + cancelled_at INTEGER NOT NULL DEFAULT 0, + expires_at INTEGER NOT NULL, + metadata TEXT NOT NULL DEFAULT '', + errors TEXT NOT NULL DEFAULT '', + lease_owner TEXT NOT NULL DEFAULT '', + lease_until INTEGER NOT NULL DEFAULT 0 +); +CREATE INDEX batches_owner ON batches (owner, created_at); +CREATE INDEX batches_status ON batches (status, lease_until); + +CREATE TABLE batch_results ( + batch_id TEXT NOT NULL, + line INTEGER NOT NULL, + ok INTEGER NOT NULL, + data TEXT NOT NULL, + PRIMARY KEY (batch_id, line) +); diff --git a/internal/store/postgres_test.go b/internal/store/postgres_test.go index 47bf6613..13e733c0 100644 --- a/internal/store/postgres_test.go +++ b/internal/store/postgres_test.go @@ -43,7 +43,7 @@ func openPG(t *testing.T) *DB { // every subtest gets a fresh file, and failed only here. `TRUNCATE thread_messages, threads, memory_notes, usage, usage_5m, usage_1h, usage_1d, api_keys, memberships, key_claims, limits, models, sessions, users, projects, teams, - organizations, leadership, control_planes RESTART IDENTITY CASCADE`) + organizations, leadership, control_planes, files, batches, batch_results RESTART IDENTITY CASCADE`) if err != nil { t.Fatalf("clearing the test database: %v", err) } From 32ff1dbd6f7ed04bf7944fc04a68d33efd05d1f0 Mon Sep 17 00:00:00 2001 From: Ilker Manap Date: Sun, 4 Oct 2026 22:51:21 +0200 Subject: [PATCH 06/38] Requests kept and replayed: keys requests per project with retention and a share, ctl request --body and --replay Co-Authored-By: Claude Opus 5.5 (1M context) --- BACKLOG.md | 6 +- CHANGELOG.md | 9 ++ docs/console.md | 3 +- docs/data.md | 7 +- docs/install.md | 23 +++ internal/cli/ctl_request.go | 88 +++++++++- internal/cli/keys.go | 1 + internal/cli/requestlog.go | 74 +++++++++ internal/cli/server.go | 1 + internal/cli/tui_people.go | 35 +++- internal/control/adminapi/api.go | 4 + internal/control/adminapi/requestbody.go | 86 ++++++++++ internal/router/handlers.go | 7 + internal/router/requestlog.go | 152 ++++++++++++++++++ internal/router/requestlog_test.go | 96 +++++++++++ internal/router/router.go | 3 + internal/router/threads.go | 2 +- internal/store/copy.go | 1 + .../migrations/postgres/0025_request_log.sql | 24 +++ .../migrations/sqlite/0025_request_log.sql | 23 +++ internal/store/postgres_test.go | 2 +- internal/store/requestlog.go | 120 ++++++++++++++ 22 files changed, 757 insertions(+), 10 deletions(-) create mode 100644 internal/cli/requestlog.go create mode 100644 internal/control/adminapi/requestbody.go create mode 100644 internal/router/requestlog.go create mode 100644 internal/router/requestlog_test.go create mode 100644 internal/store/migrations/postgres/0025_request_log.sql create mode 100644 internal/store/migrations/sqlite/0025_request_log.sql create mode 100644 internal/store/requestlog.go diff --git a/BACKLOG.md b/BACKLOG.md index 95631cb7..e30e132b 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -441,7 +441,11 @@ portal is not done — the page shows; the console and ctl change. **Batch API and Files (M–L).** `/v1/files`, `/v1/batches`, JSONL storage, a worker feeding the bronze band, accounting. New `internal/router/batch`. -9. **Request/response logging and replay (M).** Opt-in per project with +9. **Done (branch `features-c2`):** `keys requests` (retention, sample), + kept after redaction, `ctl request --body/--replay`, console, pruning. + Left: a streamed answer is kept as its text, not its chunks; replay is + one request at a time (a set of them against a model is item 21's eval). + **Request/response logging and replay (M).** Opt-in per project with retention like threads; sampled or full; redaction hooks from B; `ctl request --body`; replay against another model. 10. **OpenAI compatibility gaps (S–M each).** Responses: `text.format` → diff --git a/CHANGELOG.md b/CHANGELOG.md index b8c411b9..a80c6d13 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -71,6 +71,15 @@ APIs change between minors without a deprecation period. - **Other ways in:** `GET /api/v1/usage/export` (CSV, or `format=json`) and **E** on the console's Usage view. +### Requests kept, and replayed + +- **A project can keep its requests and answers** (`llmgw keys requests + --project P --retain-hours N [--sample 0.05]`, or **Keep requests…** in the + console): what the model was sent, after redaction, and what the client + got. Off until asked; never under `no_store`; pruned hourly. +- **`llmgw ctl request ID --body`** shows one; **`--replay MODEL`** sends it + to another model and shows both answers. + ### Batches - **OpenAI's Batch and Files APIs.** `POST /v1/files` (purpose batch), diff --git a/docs/console.md b/docs/console.md index 44fcbdc7..cfab53f4 100644 --- a/docs/console.md +++ b/docs/console.md @@ -192,7 +192,8 @@ with anything secret taken out. `llmgw ctl audit` is the same, with filters. projects, with members, keys and limits. **A** adds an organization. On a row: add a team or project inside it, rename, move, limits for the level and for each of its members, its capacity quota, and for a project how long -conversations and agent notes are kept. **Longest key lifetime…** caps how +conversations and agent notes are kept, and whether its requests are +(**Keep requests…**: for how long, and what share). **Longest key lifetime…** caps how long a key made there may live, and is the expiry of one made without any (`llmgw keys ttl`; `auth.max_key_ttl` in server.yaml caps every key). **Remove** takes away only something empty. diff --git a/docs/data.md b/docs/data.md index 6d3cf02e..1b6d4271 100644 --- a/docs/data.md +++ b/docs/data.md @@ -51,6 +51,11 @@ limits attached to it. Passwords are hashed. **Conversations and agent notes** — only for a project that asked. See below. +**Kept requests** — only for a project that asked (`llmgw keys requests`): +what each request sent the model, after the guardrails' redaction and with +any stored conversation in it, and the answer the client got, for the hours +it said, all of them or a share. Never under a `no_store` policy. + **Batch files** — when a client uses the Batch API: the file of requests it uploaded (prompts) and the files the batch wrote (answers), in the store, until deleted (`DELETE /v1/files/{id}`) or, for what a batch wrote, until @@ -61,7 +66,7 @@ them. ## What is not stored Prompts. Completions. Anything the model was shown or produced, unless the -project turned on conversations or notes, or sent a batch. +project turned on conversations, notes or kept requests, or sent a batch. The chat page in llmgw's own web interface keeps the conversation in a JavaScript variable in the browser. Refresh it and the conversation is gone, diff --git a/docs/install.md b/docs/install.md index ae7b9f69..984840b7 100644 --- a/docs/install.md +++ b/docs/install.md @@ -1130,6 +1130,29 @@ curl http://gateway:8080/v1/chat/completions \ 100 values of each apart and the rest as `other`: a tag is the client's to set, and every value is a series. +## Keeping requests, and replaying them + +To follow a request up — what exactly was sent, what came back — or to try +real traffic on another model, a project can keep its requests: + +``` +llmgw keys requests --project Acme/ml/train --retain-hours 72 # every one +llmgw keys requests --project Acme/ml/train --retain-hours 168 --sample 0.05 +llmgw ctl request req_3f2a… --body # what was sent and answered +llmgw ctl request req_3f2a… --replay qwen2.5-14b-instruct # the same, to another model +llmgw keys requests --project Acme/ml/train --off +``` + +- **What is kept:** the request as the model was sent it — after the + guardrails' redaction, with any stored conversation spliced in — and the + answer as the client got it, up to 128 KB of it. A share (`--sample`) is + the same requests every time for the same ids. +- **Replay** sends it as the gateway itself: past the limits, the budget and + the guardrails, which judged it the first time, and charged to nobody. +- **Never** for a project under a guardrails policy with `no_store`. Kept + requests past their hours, or of a project turned off, are removed hourly. + The console has it as **Keep requests…** on a project. + ## Batches Work that can wait — a night's classification, embeddings for a document diff --git a/internal/cli/ctl_request.go b/internal/cli/ctl_request.go index d0c8e5fc..74cba2ee 100644 --- a/internal/cli/ctl_request.go +++ b/internal/cli/ctl_request.go @@ -10,18 +10,34 @@ import ( "github.com/spf13/cobra" "gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi" + "gitea.mynodes.xyz/ilker/llmgw/internal/store" ) func newRequestCmd(client func() *ctlClient, asJSON *bool) *cobra.Command { - return &cobra.Command{ + var ( + body bool + replay string + ) + c := &cobra.Command{ Use: "request ID", Short: "What one request went through: replica, queue, first token, tokens, guardrails", Long: "Every response carries its id in X-Request-Id, and a client may send its\n" + "own. Given one, this says which replica answered, how long admission held\n" + "it, how long the first token took and what it cost — the answer to \"why\n" + - "was this slow\". The access log has the same fields under \"request\".", - Args: cobra.ExactArgs(1), + "was this slow\". The access log has the same fields under \"request\".\n\n" + + "For a project that keeps its requests (`llmgw keys requests`), --body\n" + + "shows what the model was sent and what came back, and --replay MODEL\n" + + "sends the same request to another model, past the limits and the\n" + + "guardrails, and shows its answer beside the first.", + Example: " llmgw ctl request req_3f2a…\n llmgw ctl request req_3f2a… --body\n llmgw ctl request req_3f2a… --replay qwen2.5-14b-instruct", + Args: cobra.ExactArgs(1), RunE: func(cmd *cobra.Command, args []string) error { + if replay != "" { + return replayRequest(cmd, client, args[0], replay, *asJSON) + } + if body { + return showRequestBody(cmd, client, args[0], *asJSON) + } var v adminapi.RequestView if err := client().get(cmd.Context(), "/api/v1/requests/"+url.PathEscape(args[0]), &v); err != nil { return err @@ -65,4 +81,70 @@ func newRequestCmd(client func() *ctlClient, asJSON *bool) *cobra.Command { return nil }, } + c.Flags().BoolVar(&body, "body", false, "show the request and its answer, for a project that keeps them") + c.Flags().StringVar(&replay, "replay", "", "send the kept request to this model and show both answers") + return c +} + +func showRequestBody(cmd *cobra.Command, client func() *ctlClient, id string, asJSON bool) error { + var b store.RequestBody + if err := client().get(cmd.Context(), "/api/v1/requests/"+url.PathEscape(id)+"/body", &b); err != nil { + return err + } + if asJSON { + return json.NewEncoder(cmd.OutOrStdout()).Encode(b) + } + out := cmd.OutOrStdout() + fmt.Fprintf(out, "request %s at %s, %s to %s (served by %s), status %d\n", b.RequestID, + b.At.UTC().Format("2006-01-02 15:04:05Z"), b.Path, b.Model, b.ServedModel, b.Status) + fmt.Fprintf(out, "kept until %s\n\n", b.ExpiresAt.UTC().Format("2006-01-02 15:04Z")) + fmt.Fprintln(out, "--- sent") + fmt.Fprintln(out, indentJSON(b.Request)) + fmt.Fprintln(out, "--- answered") + fmt.Fprintln(out, or(b.Response, "(no text: an error, or not a completion)")) + if b.Truncated { + fmt.Fprintln(out, "(longer than what is kept; cut here)") + } + return nil +} + +func replayRequest(cmd *cobra.Command, client func() *ctlClient, id, model string, asJSON bool) error { + var v adminapi.ReplayView + body, _ := json.Marshal(map[string]string{"model": model}) + if err := client().post(cmd.Context(), "/api/v1/requests/"+url.PathEscape(id)+"/replay", body, &v); err != nil { + return err + } + if asJSON { + return json.NewEncoder(cmd.OutOrStdout()).Encode(v) + } + out := cmd.OutOrStdout() + fmt.Fprintf(out, "--- %s answered (status %d)\n%s\n\n", v.Original.ServedModel, v.Original.Status, + or(v.Original.Response, "(no text)")) + fmt.Fprintf(out, "--- %s answers (status %d, %d ms)\n%s\n", v.Model, v.Status, v.TookMs, or(replyText(v.Body), string(v.Body))) + return nil +} + +// replyText is a completion's text, or "" when it is not one. +func replyText(body json.RawMessage) string { + var c struct { + Choices []struct { + Message struct { + Content string `json:"content"` + } `json:"message"` + Text string `json:"text"` + } `json:"choices"` + } + if json.Unmarshal(body, &c) != nil || len(c.Choices) == 0 { + return "" + } + return c.Choices[0].Message.Content + c.Choices[0].Text +} + +func indentJSON(s string) string { + var v any + if json.Unmarshal([]byte(s), &v) != nil { + return s + } + b, _ := json.MarshalIndent(v, "", " ") + return string(b) } diff --git a/internal/cli/keys.go b/internal/cli/keys.go index 717079f0..24c3d585 100644 --- a/internal/cli/keys.go +++ b/internal/cli/keys.go @@ -42,6 +42,7 @@ func newTenancyCmd() *cobra.Command { newKeyTTLCmd(open), newLimitsCmd(open), newThreadsCmd(open), + newRequestLogCmd(open), newMemoryCmd(open), newMoveStoreCmd(), ) diff --git a/internal/cli/requestlog.go b/internal/cli/requestlog.go new file mode 100644 index 00000000..fa8d0fb2 --- /dev/null +++ b/internal/cli/requestlog.go @@ -0,0 +1,74 @@ +package cli + +import ( + "fmt" + + "github.com/spf13/cobra" +) + +func newRequestLogCmd(open opener) *cobra.Command { + var ( + project string + hours int + sample float64 + off bool + ) + cmd := &cobra.Command{ + Use: "requests", + Short: "Keep a project's requests and answers, to follow one up or replay it", + Long: "With this on, what each request of the project sent the model — after the\n" + + "guardrails' redaction, with any stored conversation in it — and the answer\n" + + "the client got are kept for --retain-hours. `llmgw ctl request ID --body`\n" + + "shows one, and `--replay MODEL` sends it to another model.\n\n" + + "Off for every project until this is run, like conversations: llmgw\n" + + "otherwise keeps no text. --sample keeps a share of the requests rather\n" + + "than all of them. A project under a guardrails policy with no_store keeps\n" + + "nothing whatever this says.", + Example: " llmgw keys requests --project Acme/ml/train --retain-hours 72\n" + + " llmgw keys requests --project Acme/ml/train --retain-hours 168 --sample 0.05\n" + + " llmgw keys requests --project Acme/ml/train --off", + Args: cobra.NoArgs, + RunE: func(cmd *cobra.Command, _ []string) error { + if !off && hours <= 0 { + return fmt.Errorf("--retain-hours is required: say how long requests are kept, or pass --off") + } + if sample <= 0 || sample > 1 { + return fmt.Errorf("--sample is a share: above 0, at most 1") + } + s, err := open(cmd.Context()) + if err != nil { + return err + } + defer s.Close() + p, err := resolveProject(cmd.Context(), s, project) + if err != nil { + return err + } + if off { + hours = 0 + } + if err := s.SetRequestLog(cmd.Context(), p.ID, hours, sample); err != nil { + return err + } + if hours == 0 { + fmt.Fprintf(cmd.OutOrStdout(), "%s keeps no requests. Those kept are removed at the next prune.\n", p.Path) + return nil + } + fmt.Fprintf(cmd.OutOrStdout(), "%s keeps %s of its requests and answers for %d hours.\n", + p.Path, share(sample), hours) + return nil + }, + } + cmd.Flags().StringVar(&project, "project", "", "project: id, path or unique name") + cmd.Flags().IntVar(&hours, "retain-hours", 0, "how long a request and its answer are kept") + cmd.Flags().Float64Var(&sample, "sample", 1, "the share of requests kept, above 0 and at most 1") + cmd.Flags().BoolVar(&off, "off", false, "stop keeping requests for this project") + return cmd +} + +func share(f float64) string { + if f >= 1 { + return "every one" + } + return fmt.Sprintf("%g%%", f*100) +} diff --git a/internal/cli/server.go b/internal/cli/server.go index e9aadfea..3555ccbf 100644 --- a/internal/cli/server.go +++ b/internal/cli/server.go @@ -292,6 +292,7 @@ func newServerCmd() *cobra.Command { adminAPI = api api.SetReleases(cfg.Releases) api.SetNotifier(notifier) + api.SetReplayer(rt.Replay) if st := rt.Store(); st != nil && gate != nil { api.SetGuard(&adminapi.GuardAdmin{Library: cfg.Guard, Store: st, Invalidate: gate.Invalidate}) } diff --git a/internal/cli/tui_people.go b/internal/cli/tui_people.go index d42a981f..66ccb914 100644 --- a/internal/cli/tui_people.go +++ b/internal/cli/tui_people.go @@ -70,6 +70,8 @@ type peopleData struct { limits map[string]store.Limits threads map[string]int // project id → hours kept, conversations notes map[string]int // project id → hours kept, agent notes + // requests is how each project keeps its requests and answers. + requests map[string]store.RequestLog // guardPolicies are server.yaml's guardrail policies, guardDefault those // applied everywhere, and guardAt which are attached where. guardPolicies []string @@ -122,7 +124,7 @@ func (m consoleModel) fetchPeople() tea.Cmd { d.guardDefault = cfg.Guard.Default d.guardAt, _ = s.GuardAttachments(ctx) } - d.threads, d.notes = map[string]int{}, map[string]int{} + d.threads, d.notes, d.requests = map[string]int{}, map[string]int{}, map[string]store.RequestLog{} for _, r := range d.orgs { if r.level != "project" { continue @@ -133,6 +135,9 @@ func (m consoleModel) fetchPeople() tea.Cmd { if t, err := s.MemoryRetention(ctx, r.id); err == nil { d.notes[r.id] = int(t.Hours()) } + if l, err := s.RequestLogOf(ctx, r.id); err == nil { + d.requests[r.id] = l + } } return consolePeopleMsg(d) } @@ -502,7 +507,33 @@ func (m consoleModel) orgMenu(r orgRow) *menuDialog { } items = append(items, retention("Keep conversations", "threads", "conversations", m.people.threads[r.id]), - retention("Keep agent notes", "memory", "notes", m.people.notes[r.id])) + retention("Keep agent notes", "memory", "notes", m.people.notes[r.id]), + menuItem{"Keep requests…", func(m *consoleModel) tea.Cmd { + l := m.people.requests[r.id] + sample := "1" + if l.Sample > 0 { + sample = strconv.FormatFloat(l.Sample, 'f', -1, 64) + } + m.openForm(&formDialog{title: "Keep requests · " + r.path, + note: "Off by default: llmgw keeps no requests unless a project asks for it.\n" + + "Kept, `llmgw ctl request ID --body` shows one and --replay sends it to another model.", + fields: []formField{ + textF("hours", strconv.Itoa(int(l.Retain.Hours())), "how long they are kept; 0 keeps none"), + textF("share", sample, "of requests kept, above 0 and at most 1"), + }, + submit: func(m *consoleModel, v []string) (tea.Cmd, error) { + h, err := strconv.Atoi(v[0]) + if err != nil || h < 0 { + return nil, fmt.Errorf("hours is a whole number") + } + args := []string{"keys", "requests", "--project=" + r.id} + if h == 0 { + return m.runCLI("Keep requests", append(args, "--off")...), nil + } + return m.runCLI("Keep requests", append(args, "--retain-hours="+v[0], "--sample="+v[1])...), nil + }}) + return nil + }}) } items = append(items, menuItem{"Remove…", func(m *consoleModel) tea.Cmd { m.confirmD = &confirmDialog{title: "Remove " + r.path, diff --git a/internal/control/adminapi/api.go b/internal/control/adminapi/api.go index c880ad2d..920b88b1 100644 --- a/internal/control/adminapi/api.go +++ b/internal/control/adminapi/api.go @@ -41,6 +41,8 @@ import ( // API serves the management endpoints. type API struct { + // replay sends a kept request again; nil without a router here. + replay Replayer // notifier is the webhooks `ctl notify test` reaches; nil without any. notifier *notify.Notifier // releases is where release binaries are fetched from, for upgrades. @@ -159,6 +161,8 @@ func (a *API) Register(mux *http.ServeMux) { mux.HandleFunc("GET /api/v1/plan", a.planModel) mux.HandleFunc("POST /api/v1/metrics/range", a.metricsBatch) mux.HandleFunc("GET /api/v1/requests/{id}", a.request) + mux.HandleFunc("GET /api/v1/requests/{id}/body", a.requestBody) + mux.HandleFunc("POST /api/v1/requests/{id}/replay", a.replayRequest) mux.HandleFunc("POST /api/v1/models/{name}/retry", a.retryModel) mux.HandleFunc("GET /api/v1/models/{name}/headroom", func(w http.ResponseWriter, r *http.Request) { limit, _ := strconv.Atoi(r.URL.Query().Get("limit")) diff --git a/internal/control/adminapi/requestbody.go b/internal/control/adminapi/requestbody.go new file mode 100644 index 00000000..edfbe2cf --- /dev/null +++ b/internal/control/adminapi/requestbody.go @@ -0,0 +1,86 @@ +package adminapi + +import ( + "context" + "encoding/json" + "errors" + "io" + "net/http" + "time" + + "gitea.mynodes.xyz/ilker/llmgw/internal/store" +) + +type bodyFinder interface { + RequestBodyByID(ctx context.Context, id string) (store.RequestBody, error) +} + +// Replayer sends a kept request again to a model: the router's Replay. +type Replayer func(ctx context.Context, path, model string, body []byte) (int, []byte, time.Duration, error) + +// SetReplayer gives the API the router to replay kept requests through. +func (a *API) SetReplayer(f Replayer) { a.replay = f } + +// ReplayView is a kept request sent again, beside what it got the first time. +type ReplayView struct { + Original store.RequestBody `json:"original"` + Model string `json:"model"` + Status int `json:"status"` + Body json.RawMessage `json:"body"` + TookMs int64 `json:"took_ms"` +} + +func (a *API) keptBody(w http.ResponseWriter, r *http.Request) (store.RequestBody, bool) { + f, ok := a.usage.(bodyFinder) + if !ok { + writeError(w, http.StatusNotImplemented, "this control plane keeps no requests") + return store.RequestBody{}, false + } + b, err := f.RequestBodyByID(r.Context(), r.PathValue("id")) + if errors.Is(err, store.ErrNotFound) { + writeError(w, http.StatusNotFound, "no request with that id is kept: its project keeps none "+ + "(llmgw keys requests --project P --retain-hours N), it fell outside the share kept, or it has expired") + return store.RequestBody{}, false + } + if err != nil { + writeError(w, http.StatusInternalServerError, err.Error()) + return store.RequestBody{}, false + } + return b, true +} + +func (a *API) requestBody(w http.ResponseWriter, r *http.Request) { + if b, ok := a.keptBody(w, r); ok { + writeJSON(w, http.StatusOK, b) + } +} + +func (a *API) replayRequest(w http.ResponseWriter, r *http.Request) { + if a.replay == nil { + writeError(w, http.StatusNotImplemented, "this control plane does not route requests") + return + } + var req struct { + Model string `json:"model"` + } + if err := json.NewDecoder(io.LimitReader(r.Body, 1<<12)).Decode(&req); err != nil || req.Model == "" { + writeError(w, http.StatusBadRequest, "say which model: {\"model\": \"...\"}") + return + } + b, ok := a.keptBody(w, r) + if !ok { + return + } + status, body, took, err := a.replay(r.Context(), b.Path, req.Model, []byte(b.Request)) + if err != nil { + writeError(w, http.StatusBadGateway, err.Error()) + return + } + v := ReplayView{Original: b, Model: req.Model, Status: status, TookMs: took.Milliseconds()} + if json.Valid(body) { + v.Body = body + } else { + v.Body, _ = json.Marshal(string(body)) + } + writeJSON(w, http.StatusOK, v) +} diff --git a/internal/router/handlers.go b/internal/router/handlers.go index 790a2bd6..e5c32274 100644 --- a/internal/router/handlers.go +++ b/internal/router/handlers.go @@ -294,6 +294,9 @@ type parsedRequest struct { // user field. Both are recorded with the usage, never sent on. Tags map[string]string EndUser string + // logRetain is how long the request and its answer are kept, when its + // project keeps them and it fell in the share kept. + logRetain time.Duration // Model is what will be dispatched, which is not always what was asked // for: a caller out of budget may be moved to a model they can afford. Model string @@ -499,9 +502,13 @@ func (r *Router) handleInference(path string) http.HandlerFunc { return } + r.decideRequestLog(req.Context(), parsed) res := r.dispatch(req.Context(), w, parsed) status := r.finish(w, parsed, res, start) r.observeCanary(parsed, res, status, time.Since(start)) + if parsed.logRetain > 0 { + go r.keepRequest(context.WithoutCancel(req.Context()), parsed, res, status) + } r.observeOutput(parsed, res, res.judged) span.SetAttributes(attribute.Int("http.response.status_code", status), attribute.String("llmgw.served_model", res.model), diff --git a/internal/router/requestlog.go b/internal/router/requestlog.go new file mode 100644 index 00000000..105dda47 --- /dev/null +++ b/internal/router/requestlog.go @@ -0,0 +1,152 @@ +package router + +import ( + "bytes" + "context" + "fmt" + "hash/fnv" + "net/http" + "sync" + "time" + + storepkg "gitea.mynodes.xyz/ilker/llmgw/internal/store" +) + +// Keeping requests and answers, for a project that asks: what was sent to +// the model and what came back, for following one request up and for +// sending it again to another model. + +// requestLogTTL is how long a project's setting is believed before it is +// read again. +const requestLogTTL = 30 * time.Second + +type requestLogs struct { + mu sync.Mutex + by map[string]cachedLog +} + +type cachedLog struct { + l storepkg.RequestLog + at time.Time +} + +// requestLogOf is a project's setting, read at most every requestLogTTL. +func (r *Router) requestLogOf(ctx context.Context, project string) storepkg.RequestLog { + if r.store == nil || project == "" { + return storepkg.RequestLog{} + } + r.reqLogs.mu.Lock() + c, ok := r.reqLogs.by[project] + r.reqLogs.mu.Unlock() + if ok && time.Since(c.at) < requestLogTTL { + return c.l + } + l, err := r.store.RequestLogOf(ctx, project) + if err != nil { + r.log.Warn("could not read whether a project keeps its requests", "project", project, "err", err) + return c.l + } + r.reqLogs.mu.Lock() + if r.reqLogs.by == nil { + r.reqLogs.by = map[string]cachedLog{} + } + r.reqLogs.by[project] = cachedLog{l: l, at: time.Now()} + r.reqLogs.mu.Unlock() + return l +} + +// sampled says whether a request falls in a share of them, by its id, so +// the same id always says the same. +func sampled(id string, share float64) bool { + if share >= 1 { + return true + } + h := fnv.New32a() + h.Write([]byte(id)) + return float64(h.Sum32()%10000) < share*10000 +} + +// decideRequestLog marks a request to be kept, once it is known who sent +// it and that no policy forbids keeping it. +func (r *Router) decideRequestLog(ctx context.Context, req *parsedRequest) { + if req.noStore || req.internal { + return + } + l := r.requestLogOf(ctx, req.ProjectID) + if l.Retain > 0 && sampled(req.RequestID, l.Sample) { + req.logRetain = l.Retain + } +} + +// keepRequest writes a finished request and its answer, for one marked. +func (r *Router) keepRequest(ctx context.Context, req *parsedRequest, res dispatchResult, status int) { + if req.logRetain <= 0 { + return + } + served := res.model + if served == "" { + served = req.Model + } + now := time.Now() + b := storepkg.RequestBody{RequestID: req.RequestID, ProjectID: req.ProjectID, At: now, + ExpiresAt: now.Add(req.logRetain), Path: req.Path, Model: req.Model, ServedModel: served, + Status: status, Request: string(req.UpstreamBody), Response: res.outcome.Reply, + Truncated: res.outcome.ReplyTruncated} + if err := r.store.PutRequestBody(ctx, b); err != nil { + r.log.Warn("could not keep a request", "request_id", req.RequestID, "err", err) + } +} + +// pruneRequestBodies removes kept requests past their retention, hourly. +func (r *Router) pruneRequestBodies(ctx context.Context) { + t := time.NewTicker(time.Hour) + defer t.Stop() + for { + if n, err := r.store.PruneRequestBodies(ctx, time.Now()); err != nil { + if ctx.Err() == nil { + r.log.Warn("could not prune kept requests", "err", err) + } + } else if n > 0 { + r.log.Info("removed kept requests past their retention", "requests", n) + } + select { + case <-ctx.Done(): + return + case <-t.C: + } + } +} + +// Replay sends a kept request again to a model, as the gateway itself: past +// admission, accounting and the guardrails, which judged it the first time. +// For comparing models on real traffic. It answers the status, the body, +// and how long it took. +func (r *Router) Replay(ctx context.Context, path, model string, body []byte) (int, []byte, time.Duration, error) { + if !batchEndpoints[path] { + return 0, nil, 0, fmt.Errorf("a request to %s cannot be replayed", path) + } + body = withoutField(withModel(body, model), "stream") + hreq, err := http.NewRequestWithContext(ctx, http.MethodPost, path, bytes.NewReader(body)) + if err != nil { + return 0, nil, 0, err + } + req, err := r.parse(hreq, path) + if err != nil { + return 0, nil, 0, err + } + req.internal, req.Stream = true, false + req.RequestID = requestID("") + req.Header = http.Header{"X-Request-Id": []string{req.RequestID}} + w := &bufferWriter{header: http.Header{}} + start := time.Now() + res := r.dispatch(ctx, w, req) + took := time.Since(start) + if res.err != nil { + return 0, nil, took, res.err + } + status := w.status + if status == 0 { + status = http.StatusOK + } + return status, w.buf.Bytes(), took, nil +} diff --git a/internal/router/requestlog_test.go b/internal/router/requestlog_test.go new file mode 100644 index 00000000..fa0f7f04 --- /dev/null +++ b/internal/router/requestlog_test.go @@ -0,0 +1,96 @@ +package router + +import ( + "fmt" + "strings" + "testing" + "time" + + "gitea.mynodes.xyz/ilker/llmgw/internal/store" +) + +func keptBody(t *testing.T, rt *Router, id string) (store.RequestBody, bool) { + t.Helper() + deadline := time.Now().Add(3 * time.Second) + for { + b, err := rt.store.RequestBodyByID(t.Context(), id) + if err == nil { + return b, true + } + if time.Now().After(deadline) { + return store.RequestBody{}, false + } + time.Sleep(20 * time.Millisecond) + } +} + +func TestAProjectThatAsksKeepsItsRequestsAndTheyReplay(t *testing.T) { + e := newFakeEngine(t, "qwen2.5-14b-instruct") + rt, secret := pricedRouter(t, e.srv.URL) + h := rt.Handler() + project := mustProjectOf(t, rt, "key-1") + + send := func(id string) { + rec := post(t, h, map[string]any{"model": "q14", "messages": []any{ + map[string]any{"role": "user", "content": "what is 2+2"}}}, + map[string]string{"Authorization": "Bearer " + secret, "X-Request-Id": id}) + if rec.Code != 200 { + t.Fatalf("%d %s", rec.Code, rec.Body.String()) + } + } + send("before") + if _, ok := keptBody(t, rt, "before"); ok { + t.Fatal("a request was kept by a project that keeps none") + } + + if err := rt.store.SetRequestLog(t.Context(), project, 24, 1); err != nil { + t.Fatal(err) + } + rt.reqLogs.by = nil + send("kept-1") + b, ok := keptBody(t, rt, "kept-1") + if !ok { + t.Fatal("not kept") + } + if !strings.Contains(b.Request, "what is 2+2") || b.Response != "hi" || b.Status != 200 || + b.ServedModel != "qwen2.5-14b-instruct" || b.Path != "/v1/chat/completions" { + t.Errorf("kept %+v", b) + } + if b.ExpiresAt.Sub(b.At) != 24*time.Hour { + t.Errorf("kept for %s", b.ExpiresAt.Sub(b.At)) + } + + status, body, _, err := rt.Replay(t.Context(), b.Path, "q14", []byte(b.Request)) + if err != nil || status != 200 || !strings.Contains(string(body), "hi") { + t.Errorf("replay: %d %s %v", status, body, err) + } + if _, _, _, err := rt.Replay(t.Context(), "/v1/files", "q14", nil); err == nil { + t.Error("a non-inference path was replayed") + } + + // A small share keeps few. + _ = rt.store.SetRequestLog(t.Context(), project, 24, 0.05) + rt.reqLogs.by = nil + for i := range 100 { + send(fmt.Sprintf("s-%d", i)) + } + time.Sleep(200 * time.Millisecond) + n := 0 + for i := range 100 { + if _, err := rt.store.RequestBodyByID(t.Context(), fmt.Sprintf("s-%d", i)); err == nil { + n++ + } + } + if n == 0 || n > 20 { + t.Errorf("kept %d of 100 at 5%%", n) + } + + // Off: what was kept goes at the next prune. + _ = rt.store.SetRequestLog(t.Context(), project, 0, 0) + if _, err := rt.store.PruneRequestBodies(t.Context(), time.Now()); err != nil { + t.Fatal(err) + } + if _, err := rt.store.RequestBodyByID(t.Context(), "kept-1"); err == nil { + t.Error("a project that stopped keeping requests still has them after a prune") + } +} diff --git a/internal/router/router.go b/internal/router/router.go index f6739853..e5e7f03b 100644 --- a/internal/router/router.go +++ b/internal/router/router.go @@ -70,6 +70,8 @@ type Router struct { canaries canaryStats // batchWake has the batch worker look for work at once. batchWake chan struct{} + // reqLogs is each project's keeping of requests, as last read. + reqLogs requestLogs // extraMetrics are collectors other components publish on /metrics. extraMetrics []prometheus.Collector @@ -448,6 +450,7 @@ func (r *Router) Start(ctx context.Context) { } if r.store != nil { go r.runBatches(ctx) + go r.pruneRequestBodies(ctx) go r.refreshTenants(ctx) go r.purgeSessions(ctx) } diff --git a/internal/router/threads.go b/internal/router/threads.go index 04c12a2a..c7d5c374 100644 --- a/internal/router/threads.go +++ b/internal/router/threads.go @@ -221,7 +221,7 @@ const maxStoredReply = 128 << 10 // Zero for every request not continuing a conversation, which is what keeps // the cost of this feature off the path of requests that do not use it. func (p *parsedRequest) captureBytes() int { - if p.ThreadID == "" || p.noStore { + if p.ThreadID == "" && p.logRetain <= 0 || p.noStore { return 0 } return maxStoredReply diff --git a/internal/store/copy.go b/internal/store/copy.go index a7a630c5..535ca893 100644 --- a/internal/store/copy.go +++ b/internal/store/copy.go @@ -61,6 +61,7 @@ var copyOrder = []string{ "files", "batches", "batch_results", + "request_bodies", } // notCopied are tables deliberately left behind. diff --git a/internal/store/migrations/postgres/0025_request_log.sql b/internal/store/migrations/postgres/0025_request_log.sql new file mode 100644 index 00000000..7f38b57f --- /dev/null +++ b/internal/store/migrations/postgres/0025_request_log.sql @@ -0,0 +1,24 @@ +-- Ported from the SQLite migration of the same number. +-- Keeping requests and answers, for a project that asks: how long, and +-- what share of its requests. Off for every project until set, like +-- conversations: a gateway that keeps no text is the default. +ALTER TABLE projects ADD COLUMN request_log_hours BIGINT; +ALTER TABLE projects ADD COLUMN request_log_sample DOUBLE PRECISION; + +-- The request as the model was sent it — after the guardrails' redaction +-- and with any stored conversation spliced in — and the answer as the +-- client was sent it. +CREATE TABLE request_bodies ( + request_id TEXT PRIMARY KEY, + project_id TEXT NOT NULL, + ts BIGINT NOT NULL, + expires_at BIGINT NOT NULL, + path TEXT NOT NULL, + model TEXT NOT NULL, + served_model TEXT NOT NULL, + status BIGINT NOT NULL, + request TEXT NOT NULL, + response TEXT NOT NULL, + truncated BIGINT NOT NULL DEFAULT 0 +); +CREATE INDEX request_bodies_expiry ON request_bodies (expires_at); diff --git a/internal/store/migrations/sqlite/0025_request_log.sql b/internal/store/migrations/sqlite/0025_request_log.sql new file mode 100644 index 00000000..658ef214 --- /dev/null +++ b/internal/store/migrations/sqlite/0025_request_log.sql @@ -0,0 +1,23 @@ +-- Keeping requests and answers, for a project that asks: how long, and +-- what share of its requests. Off for every project until set, like +-- conversations: a gateway that keeps no text is the default. +ALTER TABLE projects ADD COLUMN request_log_hours INTEGER; +ALTER TABLE projects ADD COLUMN request_log_sample REAL; + +-- The request as the model was sent it — after the guardrails' redaction +-- and with any stored conversation spliced in — and the answer as the +-- client was sent it. +CREATE TABLE request_bodies ( + request_id TEXT PRIMARY KEY, + project_id TEXT NOT NULL, + ts INTEGER NOT NULL, + expires_at INTEGER NOT NULL, + path TEXT NOT NULL, + model TEXT NOT NULL, + served_model TEXT NOT NULL, + status INTEGER NOT NULL, + request TEXT NOT NULL, + response TEXT NOT NULL, + truncated INTEGER NOT NULL DEFAULT 0 +); +CREATE INDEX request_bodies_expiry ON request_bodies (expires_at); diff --git a/internal/store/postgres_test.go b/internal/store/postgres_test.go index 13e733c0..010df303 100644 --- a/internal/store/postgres_test.go +++ b/internal/store/postgres_test.go @@ -43,7 +43,7 @@ func openPG(t *testing.T) *DB { // every subtest gets a fresh file, and failed only here. `TRUNCATE thread_messages, threads, memory_notes, usage, usage_5m, usage_1h, usage_1d, api_keys, memberships, key_claims, limits, models, sessions, users, projects, teams, - organizations, leadership, control_planes, files, batches, batch_results RESTART IDENTITY CASCADE`) + organizations, leadership, control_planes, files, batches, batch_results, request_bodies RESTART IDENTITY CASCADE`) if err != nil { t.Fatalf("clearing the test database: %v", err) } diff --git a/internal/store/requestlog.go b/internal/store/requestlog.go new file mode 100644 index 00000000..3d19ffd9 --- /dev/null +++ b/internal/store/requestlog.go @@ -0,0 +1,120 @@ +package store + +import ( + "context" + "database/sql" + "errors" + "fmt" + "time" +) + +// RequestLog is how a project keeps its requests and answers: for how long, +// and what share of them. Zero Retain keeps none. +type RequestLog struct { + Retain time.Duration + // Sample is the share kept, above 0 and at most 1. + Sample float64 +} + +// SetRequestLog turns keeping requests and answers on for a project, or off +// with hours 0. A sample outside (0, 1] keeps every request. +func (s *DB) SetRequestLog(ctx context.Context, projectID string, hours int, sample float64) error { + var h, sm any + if hours > 0 { + h = hours + if sample <= 0 || sample > 1 { + sample = 1 + } + sm = sample + } + res, err := s.db.ExecContext(ctx, + `UPDATE projects SET request_log_hours = ?, request_log_sample = ? WHERE id = ?`, h, sm, projectID) + if err != nil { + return err + } + if n, _ := res.RowsAffected(); n == 0 { + return fmt.Errorf("store: no project %q", projectID) + } + return nil +} + +// RequestLogOf is a project's setting; zero when it keeps none. +func (s *DB) RequestLogOf(ctx context.Context, projectID string) (RequestLog, error) { + var hours sql.NullInt64 + var sample sql.NullFloat64 + err := s.db.QueryRowContext(ctx, + `SELECT request_log_hours, request_log_sample FROM projects WHERE id = ?`, projectID).Scan(&hours, &sample) + if errors.Is(err, sql.ErrNoRows) || err == nil && (!hours.Valid || hours.Int64 <= 0) { + return RequestLog{}, nil + } + if err != nil { + return RequestLog{}, err + } + l := RequestLog{Retain: time.Duration(hours.Int64) * time.Hour, Sample: 1} + if sample.Valid && sample.Float64 > 0 && sample.Float64 <= 1 { + l.Sample = sample.Float64 + } + return l, nil +} + +// RequestBody is one kept request and its answer. +type RequestBody struct { + RequestID string `json:"request_id"` + ProjectID string `json:"project_id"` + At time.Time `json:"at"` + ExpiresAt time.Time `json:"expires_at"` + Path string `json:"path"` + Model string `json:"model"` + ServedModel string `json:"served_model"` + Status int `json:"status"` + Request string `json:"request"` + // Response is the answer's text as the client was sent it; Truncated + // says it was longer than what is kept. + Response string `json:"response"` + Truncated bool `json:"truncated,omitempty"` +} + +// PutRequestBody keeps one. +func (s *DB) PutRequestBody(ctx context.Context, b RequestBody) error { + tr := 0 + if b.Truncated { + tr = 1 + } + _, err := s.db.ExecContext(ctx, `INSERT INTO request_bodies (request_id, project_id, ts, expires_at, path, + model, served_model, status, request, response, truncated) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + ON CONFLICT (request_id) DO NOTHING`, + b.RequestID, b.ProjectID, millis(b.At), millis(b.ExpiresAt), b.Path, b.Model, b.ServedModel, b.Status, + b.Request, b.Response, tr) + return err +} + +// RequestBodyByID finds one, ErrNotFound when none is kept. +func (s *DB) RequestBodyByID(ctx context.Context, id string) (RequestBody, error) { + var b RequestBody + var ts, exp int64 + var tr int + err := s.db.QueryRowContext(ctx, `SELECT request_id, project_id, ts, expires_at, path, model, served_model, + status, request, response, truncated FROM request_bodies WHERE request_id = ?`, id). + Scan(&b.RequestID, &b.ProjectID, &ts, &exp, &b.Path, &b.Model, &b.ServedModel, &b.Status, + &b.Request, &b.Response, &tr) + if errors.Is(err, sql.ErrNoRows) { + return RequestBody{}, ErrNotFound + } + if err != nil { + return RequestBody{}, err + } + b.At, b.ExpiresAt, b.Truncated = time.UnixMilli(ts), time.UnixMilli(exp), tr == 1 + return b, nil +} + +// PruneRequestBodies removes those past their retention, and those of +// projects that keep none any more. +func (s *DB) PruneRequestBodies(ctx context.Context, now time.Time) (int64, error) { + res, err := s.db.ExecContext(ctx, `DELETE FROM request_bodies WHERE expires_at < ? + OR project_id IN (SELECT id FROM projects WHERE request_log_hours IS NULL OR request_log_hours <= 0)`, + millis(now)) + if err != nil { + return 0, err + } + return res.RowsAffected() +} From 9136bc71c269ddb4f14bee808bac922dc305e9c7 Mon Sep 17 00:00:00 2001 From: Ilker Manap Date: Sun, 4 Oct 2026 22:56:25 +0200 Subject: [PATCH 07/38] More of OpenAI's API: Responses structured output, images and previous_response_id; rerank, score and audio passed through; moderations by the judge Co-Authored-By: Claude Opus 5.5 (1M context) --- BACKLOG.md | 8 +- CHANGELOG.md | 10 ++ docs/install.md | 12 +++ internal/guard/judge.go | 25 +++++ internal/router/audio.go | 62 ++++++++++++ internal/router/compat_test.go | 178 +++++++++++++++++++++++++++++++++ internal/router/guard.go | 4 + internal/router/handlers.go | 14 +++ internal/router/moderations.go | 95 ++++++++++++++++++ internal/router/proxy/proxy.go | 9 +- internal/router/responses.go | 134 ++++++++++++++++++++++++- internal/router/router.go | 3 +- 12 files changed, 549 insertions(+), 5 deletions(-) create mode 100644 internal/router/audio.go create mode 100644 internal/router/compat_test.go create mode 100644 internal/router/moderations.go diff --git a/BACKLOG.md b/BACKLOG.md index e30e132b..1bba26a8 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -448,7 +448,13 @@ portal is not done — the page shows; the console and ctl change. **Request/response logging and replay (M).** Opt-in per project with retention like threads; sampled or full; redaction hooks from B; `ctl request --body`; replay against another model. -10. **OpenAI compatibility gaps (S–M each).** Responses: `text.format` → +10. **Done (branch `features-c2`):** Responses `text.format`, + `input_image`, `previous_response_id` (thread in the id); rerank and + score passthrough; transcription and translation as multipart; + `/v1/moderations` by the judge. Left: audio is charged no tokens (an + engine's duration usage is not read); `input_image` by `file_id`; + moderation scores are 0 or 1, as a judge says yes or no. + **OpenAI compatibility gaps (S–M each).** Responses: `text.format` → `response_format`, `input_image`, `previous_response_id` via threads (`router/responses.go`). Rerank/score passthrough for vLLM, `/v1/audio/transcriptions` for a whisper engine, `/v1/moderations` diff --git a/CHANGELOG.md b/CHANGELOG.md index a80c6d13..f708eb1b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -71,6 +71,16 @@ APIs change between minors without a deprecation period. - **Other ways in:** `GET /api/v1/usage/export` (CSV, or `format=json`) and **E** on the console's Usage view. +### More of OpenAI's API + +- **Responses:** `text.format` asks for structured output, `input_image` + sends images, and `previous_response_id` continues a conversation the + gateway keeps as a thread (for a project that keeps conversations). +- **Passed through:** `/v1/rerank` and `/v1/score` for vLLM's rerankers, and + `/v1/audio/transcriptions` and `/translations` as the multipart form came, + for an engine serving Whisper. +- **`/v1/moderations`** is answered by the guardrails' judge model. + ### Requests kept, and replayed - **A project can keep its requests and answers** (`llmgw keys requests diff --git a/docs/install.md b/docs/install.md index 984840b7..01d4259c 100644 --- a/docs/install.md +++ b/docs/install.md @@ -959,6 +959,18 @@ Or open `/chat` in a browser and use it without a key at all — the page carrie your session, and the request still goes through admission, rate limits, budget and accounting exactly as an API call does. +### What the client port answers + +| Path | What | +|---|---| +| `POST /v1/chat/completions`, `/v1/completions`, `/v1/embeddings` | OpenAI's, served by the models in the catalog | +| `POST /v1/responses` | OpenAI's Responses API, translated to chat completions: tools, `text.format` (structured output), `input_image` (by URL or `data:` URL), and `previous_response_id` — the conversation is kept as a thread, so the project must keep conversations (`llmgw keys threads`); `store: false` keeps none | +| `POST /v1/rerank`, `/v1/score` | passed through to an engine that serves them: vLLM with a reranker or a cross-encoder | +| `POST /v1/audio/transcriptions`, `/v1/audio/translations` | the multipart form passed through as it came, to an engine that serves speech to text: vLLM with Whisper | +| `POST /v1/moderations` | answered by the guardrails' judge model (`role: guard`): `flagged`, and the judge's own categories | +| `/v1/files`, `/v1/batches` | batches; see [Batches](#batches) | +| `GET /v1/models` | the models this key may use | + ## Watching it The control plane exports both halves on one endpoint: what the cluster is diff --git a/internal/guard/judge.go b/internal/guard/judge.go index 0096c0d4..74d1b671 100644 --- a/internal/guard/judge.go +++ b/internal/guard/judge.go @@ -344,3 +344,28 @@ func parseJudgment(format string, names []string, body []byte, answer string) (j } return judgment{}, fmt.Errorf("%w: not in the form %s has: %q", errJudgeFormat, format, string(seen)) } + +// Moderate asks a judge model about one text, as a user's message, outside +// any policy: for /v1/moderations. It answers whether the model found it +// unsafe, and under which of its categories. +func Moderate(ctx context.Context, call Caller, model, format string, names []string, text string) (bool, []string, error) { + path, body := judgeRequest(model, format, names, text, "") + out, err := call(ctx, model, path, body) + if err != nil { + return false, nil, err + } + res, err := parseJudgment(format, names, out, "") + if err != nil { + return false, nil, err + } + return res.unsafe, res.categories, nil +} + +// DefaultCategories are the names a judge format reports its categories by +// when the catalog gives none: Llama Guard 3's S1 to S13. +func DefaultCategories(format string) []string { + if format == "llamaguard3" { + return slices.Clone(llamaGuard3Categories) + } + return nil +} diff --git a/internal/router/audio.go b/internal/router/audio.go new file mode 100644 index 00000000..0d40ef80 --- /dev/null +++ b/internal/router/audio.go @@ -0,0 +1,62 @@ +package router + +import ( + "bytes" + "errors" + "mime" + "mime/multipart" + "net/http" + "strings" + + "gitea.mynodes.xyz/ilker/llmgw/internal/auth" +) + +// parseForm reads a multipart request — audio sent for transcription — for +// what routing needs, the model, and keeps the body to send on unchanged. +func (r *Router) parseForm(req *http.Request, path, ct string, body []byte) (*parsedRequest, error) { + _, params, err := mime.ParseMediaType(ct) + if err != nil || params["boundary"] == "" { + return nil, errors.New("a multipart form needs its boundary") + } + mr := multipart.NewReader(bytes.NewReader(body), params["boundary"]) + model, stream := "", false + for { + part, err := mr.NextPart() + if err != nil { + break + } + switch part.FormName() { + case "model": + var b bytes.Buffer + _, _ = b.ReadFrom(part) + model = strings.TrimSpace(b.String()) + case "stream": + var b bytes.Buffer + _, _ = b.ReadFrom(part) + stream = strings.TrimSpace(b.String()) == "true" + } + part.Close() + } + if model == "" { + return nil, errors.New("model is required") + } + tags, err := requestTags(nil, req.Header.Get(tagsHeader)) + if err != nil { + return nil, err + } + p := &parsedRequest{ + Path: path, ContentType: ct, Model: model, RequestedModel: model, Stream: stream, + UpstreamBody: body, Header: req.Header, Tags: tags, + SessionID: sessionID(req, r.cfg.SessionHeader, ""), + Band: req.Header.Get(r.cfg.Admission.BandHeader), + ProfileOverride: req.Header.Get("x-llmgw-profile"), + } + if principal, ok := auth.FromContext(req.Context()); ok { + p.Band = principal.Band + p.ProfileOverride = principal.RoutingProfile + p.KeyID, p.UserID, p.ProjectID = principal.KeyID, principal.UserID, principal.ProjectID + p.TeamID, p.OrgID = principal.TeamID, principal.OrgID + p.Tenant = principal.KeyID + } + return p, nil +} diff --git a/internal/router/compat_test.go b/internal/router/compat_test.go new file mode 100644 index 00000000..bbf07268 --- /dev/null +++ b/internal/router/compat_test.go @@ -0,0 +1,178 @@ +package router + +import ( + "bytes" + "encoding/json" + "io" + "mime/multipart" + "net/http" + "net/http/httptest" + "strings" + "testing" + + "gitea.mynodes.xyz/ilker/llmgw/internal/config" + "gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog" +) + +func TestResponsesAsksForStructuredOutputAndImages(t *testing.T) { + var in respRequest + _ = json.Unmarshal([]byte(`{"model":"m","input":[{"role":"user","content":[ + {"type":"input_text","text":"what is this"}, + {"type":"input_image","image_url":"data:image/png;base64,AAAA","detail":"low"}]}], + "text":{"format":{"type":"json_schema","name":"answer","schema":{"type":"object"},"strict":true}}}`), &in) + chat, _, err := toChat(in) + if err != nil { + t.Fatal(err) + } + b, _ := json.Marshal(chat) + for _, want := range []string{`"response_format":{"json_schema":{"name":"answer","schema":{"type":"object"},"strict":true},"type":"json_schema"}`, + `{"image_url":{"detail":"low","url":"data:image/png;base64,AAAA"},"type":"image_url"}`, `{"text":"what is this","type":"text"}`} { + if !strings.Contains(string(b), want) { + t.Errorf("%s lacks %s", b, want) + } + } + _ = json.Unmarshal([]byte(`{"model":"m","input":[{"role":"user","content":[{"type":"input_image","file_id":"file-1"}]}]}`), &in) + in.Text = nil + if _, _, err := toChat(in); err == nil { + t.Error("an image by file_id was taken") + } + if th, ok := responseThread("resp_abc123.def"); !ok || th != "abc123" { + t.Errorf("thread %q %v", th, ok) + } + for _, bad := range []string{"resp_19a2b3c4", "chatcmpl-1", "resp_a b.c"} { + if _, ok := responseThread(bad); ok { + t.Errorf("%q read as a kept response", bad) + } + } +} + +func TestAResponseIsContinuedByItsID(t *testing.T) { + e := newFakeEngine(t, "qwen2.5-14b-instruct") + rt, secret := pricedRouter(t, e.srv.URL) + if err := rt.store.SetThreadRetention(t.Context(), mustProjectOf(t, rt, "key-1"), 24); err != nil { + t.Fatal(err) + } + h := rt.Handler() + ask := func(body map[string]any) map[string]any { + b, _ := json.Marshal(body) + req := httptest.NewRequest(http.MethodPost, "/v1/responses", bytes.NewReader(b)) + req.Header.Set("Authorization", "Bearer "+secret) + req.Header.Set("Content-Type", "application/json") + rec := httptest.NewRecorder() + h.ServeHTTP(rec, req) + var out map[string]any + _ = json.Unmarshal(rec.Body.Bytes(), &out) + if rec.Code != 200 { + t.Fatalf("%d %s", rec.Code, rec.Body.String()) + } + return out + } + first := ask(map[string]any{"model": "q14", "input": "my name is Ayşe"}) + id, _ := first["id"].(string) + if _, ok := responseThread(id); !ok { + t.Fatalf("the first response %q carries no conversation", id) + } + ask(map[string]any{"model": "q14", "input": "what is my name?", "previous_response_id": id}) + sent, _ := e.last.Load().([]byte) + if !strings.Contains(string(sent), "my name is Ayşe") || !strings.Contains(string(sent), "what is my name?") { + t.Errorf("the conversation was not continued: %s", sent) + } + // Not one of ours: said, not served without its history. + b, _ := json.Marshal(map[string]any{"model": "q14", "input": "x", "previous_response_id": "resp_nope"}) + req := httptest.NewRequest(http.MethodPost, "/v1/responses", bytes.NewReader(b)) + req.Header.Set("Authorization", "Bearer "+secret) + rec := httptest.NewRecorder() + h.ServeHTTP(rec, req) + if rec.Code != 400 || !strings.Contains(rec.Body.String(), "previous_response_not_found") { + t.Errorf("%d %s", rec.Code, rec.Body.String()) + } +} + +// engineFor serves a few paths with fixed answers and records what came. +func engineFor(t *testing.T, model string, answers map[string]string) (*httptest.Server, *[]*http.Request, *[][]byte) { + t.Helper() + var reqs []*http.Request + var bodies [][]byte + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path == "/metrics" { + return + } + b, _ := io.ReadAll(r.Body) + reqs, bodies = append(reqs, r), append(bodies, b) + a, ok := answers[r.URL.Path] + if !ok { + http.NotFound(w, r) + return + } + w.Header().Set("Content-Type", "application/json") + _, _ = w.Write([]byte(a)) + })) + t.Cleanup(srv.Close) + return srv, &reqs, &bodies +} + +func TestRerankScoreAndAudioArePassedThrough(t *testing.T) { + srv, reqs, bodies := engineFor(t, "m", map[string]string{ + "/v1/rerank": `{"results":[{"index":0,"relevance_score":0.9}],"usage":{"total_tokens":7}}`, + "/v1/score": `{"data":[{"score":0.5}]}`, + "/v1/audio/transcriptions": `{"text":"merhaba dünya"}`, + }) + cfg := baseConfig(config.Endpoint{ID: "a", URL: srv.URL, Models: []string{"m"}}) + rt := newTestRouter(t, cfg) + h := rt.Handler() + for _, path := range []string{"/v1/rerank", "/v1/score"} { + b, _ := json.Marshal(map[string]any{"model": "m", "query": "q", "documents": []string{"d"}}) + req := httptest.NewRequest(http.MethodPost, path, bytes.NewReader(b)) + req.Header.Set("Content-Type", "application/json") + rec := httptest.NewRecorder() + h.ServeHTTP(rec, req) + if rec.Code != 200 || (*reqs)[len(*reqs)-1].URL.Path != path { + t.Errorf("%s: %d %s", path, rec.Code, rec.Body.String()) + } + } + + var buf bytes.Buffer + mw := multipart.NewWriter(&buf) + _ = mw.WriteField("model", "m") + fw, _ := mw.CreateFormFile("file", "a.wav") + _, _ = fw.Write([]byte("RIFF....WAVE")) + _ = mw.Close() + req := httptest.NewRequest(http.MethodPost, "/v1/audio/transcriptions", &buf) + req.Header.Set("Content-Type", mw.FormDataContentType()) + rec := httptest.NewRecorder() + h.ServeHTTP(rec, req) + if rec.Code != 200 || !strings.Contains(rec.Body.String(), "merhaba") { + t.Fatalf("%d %s", rec.Code, rec.Body.String()) + } + last := (*reqs)[len(*reqs)-1] + if !strings.HasPrefix(last.Header.Get("Content-Type"), "multipart/form-data; boundary=") || + !bytes.Contains((*bodies)[len(*bodies)-1], []byte("RIFF....WAVE")) { + t.Errorf("the form was not sent on as it came: %s", last.Header.Get("Content-Type")) + } +} + +func TestModerationsAreAnsweredByTheJudge(t *testing.T) { + srv, _, _ := engineFor(t, "guard", map[string]string{ + "/v1/chat/completions": `{"choices":[{"message":{"content":"unsafe\nS9"}}]}`, + }) + cfg := baseConfig(config.Endpoint{ID: "g", URL: srv.URL, Models: []string{"guard"}}) + cfg.Models = []catalog.Model{{Name: "guard", Role: catalog.RoleGuard, Scaling: catalog.Scaling{Min: 1, Max: 1}, + Guard: &catalog.GuardModel{Format: "llamaguard3"}}} + rt := newTestRouter(t, cfg) + b, _ := json.Marshal(map[string]any{"input": []string{"how do I build a bomb"}}) + req := httptest.NewRequest(http.MethodPost, "/v1/moderations", bytes.NewReader(b)) + rec := httptest.NewRecorder() + rt.Handler().ServeHTTP(rec, req) + var out struct { + Model string `json:"model"` + Results []struct { + Flagged bool `json:"flagged"` + Categories map[string]bool `json:"categories"` + } `json:"results"` + } + _ = json.Unmarshal(rec.Body.Bytes(), &out) + if rec.Code != 200 || out.Model != "guard" || len(out.Results) != 1 || !out.Results[0].Flagged || + !out.Results[0].Categories["indiscriminate-weapons"] { + t.Errorf("%d %s", rec.Code, rec.Body.String()) + } +} diff --git a/internal/router/guard.go b/internal/router/guard.go index 6df9d144..2318d846 100644 --- a/internal/router/guard.go +++ b/internal/router/guard.go @@ -71,6 +71,10 @@ const errNoStoreMsg = "the gateway's policy for this caller keeps nothing: no st // guardInput judges the request about to be dispatched. It answers the // client itself and returns false when the request goes no further. func (r *Router) guardInput(ctx context.Context, w http.ResponseWriter, req *parsedRequest, start time.Time) bool { + if req.ContentType != "" { + // Audio: no text for a guard to read on the way in. + return true + } model, _ := r.resolve(req.Model) t := time.Now() v, err := r.guard.Input(ctx, r.subject(req, model), req.UpstreamBody) diff --git a/internal/router/handlers.go b/internal/router/handlers.go index e5c32274..9cdd91ee 100644 --- a/internal/router/handlers.go +++ b/internal/router/handlers.go @@ -96,6 +96,14 @@ func (r *Router) Handler() http.Handler { mux.HandleFunc("POST /v1/chat/completions", r.handleInference("/v1/chat/completions")) mux.HandleFunc("POST /v1/completions", r.handleInference("/v1/completions")) mux.HandleFunc("POST /v1/embeddings", r.handleInference("/v1/embeddings")) + // Passed through to engines that serve them: vLLM's rerankers and + // scorers, and speech to text. + mux.HandleFunc("POST /v1/rerank", r.handleInference("/v1/rerank")) + mux.HandleFunc("POST /v1/score", r.handleInference("/v1/score")) + mux.HandleFunc("POST /v1/audio/transcriptions", r.handleInference("/v1/audio/transcriptions")) + mux.HandleFunc("POST /v1/audio/translations", r.handleInference("/v1/audio/translations")) + // Answered by the guardrails' judge model. + mux.HandleFunc("POST /v1/moderations", r.handleModerations) // Codex, and whatever else speaks only the Responses API, translated to // chat completions on the way in and back on the way out. mux.HandleFunc("POST /v1/responses", r.handleResponses) @@ -290,6 +298,9 @@ func (r *Router) Handler() http.Handler { // business interpreting. type parsedRequest struct { Path string + // ContentType is the body's when it is a form rather than JSON — audio + // for transcription — and is sent on as it came. + ContentType string // Tags are the client's metadata and x-llmgw-tags; EndUser is OpenAI's // user field. Both are recorded with the usage, never sent on. Tags map[string]string @@ -661,6 +672,9 @@ func (r *Router) parse(req *http.Request, path string) (*parsedRequest, error) { if err != nil { return nil, err } + if ct := req.Header.Get("Content-Type"); strings.HasPrefix(ct, "multipart/form-data") { + return r.parseForm(req, path, ct, body) + } var b inferenceBody if err := json.Unmarshal(body, &b); err != nil { diff --git a/internal/router/moderations.go b/internal/router/moderations.go new file mode 100644 index 00000000..637ccd2b --- /dev/null +++ b/internal/router/moderations.go @@ -0,0 +1,95 @@ +package router + +import ( + "encoding/json" + "fmt" + "io" + "net/http" + "strings" + + "gitea.mynodes.xyz/ilker/llmgw/internal/guard" +) + +// /v1/moderations, answered by a judge model on the cluster: the same model +// the guardrails ask. OpenAI's shape — flagged, and the categories — with the +// judge's own categories named as it names them. + +func (r *Router) handleModerations(w http.ResponseWriter, req *http.Request) { + var in struct { + Input json.RawMessage `json:"input"` + Model string `json:"model"` + } + if err := json.NewDecoder(io.LimitReader(req.Body, r.cfg.MaxBodyBytes)).Decode(&in); err != nil { + writeError(w, http.StatusBadRequest, "invalid_request_error", "the body is not JSON: "+err.Error()) + return + } + var texts []string + var one string + if json.Unmarshal(in.Input, &one) == nil { + texts = []string{one} + } else if err := json.Unmarshal(in.Input, &texts); err != nil || len(texts) == 0 { + writeError(w, http.StatusBadRequest, "invalid_request_error", "input is a string or a list of strings") + return + } + if len(texts) > 32 { + writeError(w, http.StatusBadRequest, "invalid_request_error", "at most 32 inputs at once") + return + } + model := r.moderationModel(in.Model) + if model == "" { + writeErrorCode(w, http.StatusNotFound, "invalid_request_error", "model_not_found", + "no moderation model: the catalog has no model with role: guard") + return + } + format, names, _ := r.GuardFormat(model) + if len(names) == 0 { + names = guard.DefaultCategories(format) + } + results := make([]map[string]any, 0, len(texts)) + for _, t := range texts { + unsafe, cats, err := guard.Moderate(req.Context(), r.Judge, model, format, names, t) + if err != nil { + writeError(w, http.StatusServiceUnavailable, "server_error", "the moderation model gave no verdict: "+err.Error()) + return + } + categories, scores := map[string]bool{}, map[string]float64{} + for _, n := range names { + categories[moderationKey(n)], scores[moderationKey(n)] = false, 0 + } + for _, c := range cats { + categories[moderationKey(c)], scores[moderationKey(c)] = true, 1 + } + results = append(results, map[string]any{"flagged": unsafe, "categories": categories, "category_scores": scores}) + } + writeJSON(w, http.StatusOK, map[string]any{"id": newID("modr-"), "model": model, "results": results}) +} + +// moderationModel is the judge asked: the one named, if it is one, or the +// first in the catalog. +func (r *Router) moderationModel(asked string) string { + if asked != "" { + if canonical, e := r.resolve(asked); e.IsGuard() { + return canonical + } + } + var first string + for _, m := range r.modelMap() { + if m.IsGuard() && (first == "" || m.Name < first) { + first = m.Name + } + } + return first +} + +// moderationKey is a category's name as a JSON key, as OpenAI writes them. +func moderationKey(name string) string { + return strings.Map(func(c rune) rune { + switch { + case c >= 'A' && c <= 'Z': + return c + 'a' - 'A' + case c >= 'a' && c <= 'z', c >= '0' && c <= '9', c == '/': + return c + } + return '-' + }, fmt.Sprint(name)) +} diff --git a/internal/router/proxy/proxy.go b/internal/router/proxy/proxy.go index 0be6249e..7a8e864e 100644 --- a/internal/router/proxy/proxy.go +++ b/internal/router/proxy/proxy.go @@ -165,6 +165,9 @@ type Request struct { // and a gateway that accumulated every reply in memory would be one long // answer away from trouble; a request that stores nothing must pay nothing. CaptureReply int + // ContentType is the body's, when it is not JSON: an audio upload's + // multipart form. + ContentType string // Filter, when set, sees a whole non-streamed answer and the headers // about to be sent with it, and returns the body to send: a guard's @@ -241,7 +244,11 @@ func (p *Proxy) Do(ctx context.Context, w http.ResponseWriter, r Request) (Outco return Outcome{Retryable: false}, err } req.Header = cloneForUpstream(r.Header) - req.Header.Set("Content-Type", "application/json") + ct := r.ContentType + if ct == "" { + ct = "application/json" + } + req.Header.Set("Content-Type", ct) if r.APIKey != "" { req.Header.Set("Authorization", "Bearer "+r.APIKey) } diff --git a/internal/router/responses.go b/internal/router/responses.go index fc7d2022..2ae5160b 100644 --- a/internal/router/responses.go +++ b/internal/router/responses.go @@ -9,6 +9,8 @@ import ( "strings" "sync/atomic" "time" + + "gitea.mynodes.xyz/ilker/llmgw/internal/auth" ) // The Responses API, as a translation of chat completions. @@ -38,6 +40,13 @@ type respRequest struct { MaxOutputTokens *int `json:"max_output_tokens"` ParallelToolCalls *bool `json:"parallel_tool_calls"` Metadata map[string]string `json:"metadata"` + // Text.Format asks for structured output; PreviousResponseID continues + // a conversation kept by the gateway, which Store false declines. + Text *struct { + Format json.RawMessage `json:"format"` + } `json:"text"` + PreviousResponseID string `json:"previous_response_id"` + Store *bool `json:"store"` } type respTool struct { @@ -135,7 +144,11 @@ func toChat(in respRequest) (map[string]any, map[string]bool, error) { if role == "" { role = "user" } - msgs = append(msgs, chatMsg{Role: role, Content: partsText(it.Content)}) + content, err := chatContent(it.Content) + if err != nil { + return nil, nil, err + } + msgs = append(msgs, chatMsg{Role: role, Content: content}) case "function_call", "custom_tool_call": args := it.Arguments if it.Type == "custom_tool_call" { @@ -206,9 +219,89 @@ func toChat(in respRequest) (map[string]any, map[string]bool, error) { if in.MaxOutputTokens != nil { out["max_tokens"] = *in.MaxOutputTokens } + if in.Text != nil && len(in.Text.Format) > 0 { + var f struct { + Type string `json:"type"` + Name string `json:"name"` + Description string `json:"description"` + Schema json.RawMessage `json:"schema"` + Strict *bool `json:"strict"` + } + if err := json.Unmarshal(in.Text.Format, &f); err != nil { + return nil, nil, fmt.Errorf("text.format: %v", err) + } + switch f.Type { + case "json_schema": + js := map[string]any{"name": f.Name, "schema": f.Schema} + if f.Description != "" { + js["description"] = f.Description + } + if f.Strict != nil { + js["strict"] = *f.Strict + } + out["response_format"] = map[string]any{"type": "json_schema", "json_schema": js} + case "json_object": + out["response_format"] = map[string]any{"type": "json_object"} + case "", "text": + default: + return nil, nil, fmt.Errorf("text.format.type %q is not text, json_object or json_schema", f.Type) + } + } return out, custom, nil } +// chatContent is a message's content for a chat completion: its text, or, +// with images among its parts, the parts as a chat completion writes them. +func chatContent(raw json.RawMessage) (any, error) { + var parts []struct { + Type string `json:"type"` + Text string `json:"text"` + ImageURL string `json:"image_url"` + FileID string `json:"file_id"` + Detail string `json:"detail"` + } + if json.Unmarshal(raw, &parts) != nil { + return partsText(raw), nil + } + images := false + for _, p := range parts { + images = images || p.Type == "input_image" + } + if !images { + return partsText(raw), nil + } + var out []map[string]any + for _, p := range parts { + switch p.Type { + case "input_text", "output_text", "text": + out = append(out, map[string]any{"type": "text", "text": p.Text}) + case "input_image": + if p.ImageURL == "" { + return nil, fmt.Errorf("an input_image by file_id is not supported here: send image_url (a data: URL works)") + } + img := map[string]any{"url": p.ImageURL} + if p.Detail != "" && p.Detail != "auto" { + img["detail"] = p.Detail + } + out = append(out, map[string]any{"type": "image_url", "image_url": img}) + } + } + return out, nil +} + +// A response continued by previous_response_id is a stored conversation: its +// thread is in its id, as resp_., so nothing more is kept to find +// it. + +func responseThread(id string) (string, bool) { + rest, ok := strings.CutPrefix(id, "resp_") + i := strings.LastIndexByte(rest, '.') + if !ok || i <= 0 || validThreadID(rest[:i]) != nil { + return "", false + } + return rest[:i], true +} + var respSeq atomic.Int64 func respID(prefix string) string { @@ -239,8 +332,30 @@ func (r *Router) handleResponses(w http.ResponseWriter, req *http.Request) { next.ContentLength = int64(len(upstream)) next.Header.Set("Content-Type", "application/json") + id := respID("resp") + // A conversation the gateway keeps: continued, or begun when the + // project keeps conversations and the client did not say store false. + thread := "" + switch { + case in.PreviousResponseID != "": + var ok bool + if thread, ok = responseThread(in.PreviousResponseID); !ok { + writeErrorCode(w, http.StatusBadRequest, "invalid_request_error", "previous_response_not_found", + fmt.Sprintf("previous_response_id %q is not a response this gateway kept", in.PreviousResponseID)) + return + } + case in.Store == nil || *in.Store: + if r.keepsThreads(req) { + thread = newID("") + } + } + if thread != "" { + next.Header.Set(threadHeader, thread) + id = "resp_" + thread + "." + newID("")[:8] + } + t := &respTranslator{w: w, model: in.Model, stream: in.Stream, custom: custom, - id: respID("resp"), created: time.Now().Unix(), metadata: in.Metadata} + id: id, created: time.Now().Unix(), metadata: in.Metadata} r.handleInference("/v1/chat/completions")(t, next) t.finish() } @@ -590,3 +705,18 @@ func (t *respTranslator) fromChatJSON(b []byte) { out, _ := json.Marshal(t.object(status)) _, _ = t.w.Write(out) } + +// keepsThreads says whether the caller's project keeps conversations, so a +// Responses request can begin one its next turn continues. +func (r *Router) keepsThreads(req *http.Request) bool { + if r.threads == nil { + return false + } + p, _ := auth.FromContext(req.Context()) + project, err := r.threadProject(p.ProjectID) + if err != nil { + return false + } + retention, err := r.threads.ThreadRetention(req.Context(), project) + return err == nil && retention > 0 +} diff --git a/internal/router/router.go b/internal/router/router.go index e5e7f03b..6776e41e 100644 --- a/internal/router/router.go +++ b/internal/router/router.go @@ -927,7 +927,7 @@ func (r *Router) dispatchOne(ctx context.Context, w http.ResponseWriter, req *pa trace.WithAttributes(attribute.String("llmgw.endpoint", dec.Endpoint.ID), attribute.String("url.full", dec.Endpoint.URL+req.Path))) body := req.UpstreamBody - if dec.Endpoint.UpstreamModel != "" { + if dec.Endpoint.UpstreamModel != "" && req.ContentType == "" { body = withModel(body, dec.Endpoint.UpstreamModel) } // An adapter asked for by name, served by its model's replicas: the @@ -954,6 +954,7 @@ func (r *Router) dispatchOne(ctx context.Context, w http.ResponseWriter, req *pa Stream: req.Stream, SuppressUsageChunk: req.SuppressUsageChunk, CaptureReply: req.captureBytes(), + ContentType: req.ContentType, } // A filter per attempt, for the model this attempt is served by. filter := r.outputFilter(uctx, req, model, &preq) From e5029961985e2f27e4306334a2fb611dbb89d662 Mon Sep 17 00:00:00 2001 From: Ilker Manap Date: Sun, 4 Oct 2026 22:59:55 +0200 Subject: [PATCH 08/38] Anthropic's Messages API: /v1/messages translated to chat completions and back, streamed or not, with tools and images; keys in x-api-key Co-Authored-By: Claude Opus 5.5 (1M context) --- BACKLOG.md | 7 +- CHANGELOG.md | 8 + docs/install.md | 1 + docs/onboarding.md | 5 + internal/auth/auth.go | 6 +- internal/router/handlers.go | 3 + internal/router/messages.go | 600 +++++++++++++++++++++++++++++++ internal/router/messages_test.go | 141 ++++++++ 8 files changed, 769 insertions(+), 2 deletions(-) create mode 100644 internal/router/messages.go create mode 100644 internal/router/messages_test.go diff --git a/BACKLOG.md b/BACKLOG.md index 1bba26a8..6ae5f0d6 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -459,7 +459,12 @@ portal is not done — the page shows; the console and ctl change. (`router/responses.go`). Rerank/score passthrough for vLLM, `/v1/audio/transcriptions` for a whisper engine, `/v1/moderations` mapped to the guard model. -11. **Anthropic `/v1/messages` (M).** Translate like `responses.go`. +11. **Done (branch `features-c2`):** `/v1/messages` and `count_tokens`, + streamed and not, tools, images, `x-api-key`, Anthropic error shape. + Left: extended thinking (a reasoning model's thoughts are not mapped to + thinking blocks); prompt caching; the account page's "Connect your + editor" does not offer Claude Code yet. + **Anthropic `/v1/messages` (M).** Translate like `responses.go`. 12. **`server restore` (S).** Restore is manual shell today (`docs/install.md:1187`); add the command with version check, dry run, off-host copy target; a documented DR drill. diff --git a/CHANGELOG.md b/CHANGELOG.md index f708eb1b..21d1683b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -71,6 +71,14 @@ APIs change between minors without a deprecation period. - **Other ways in:** `GET /api/v1/usage/export` (CSV, or `format=json`) and **E** on the console's Usage view. +### Anthropic's Messages API + +- **`/v1/messages`** (and `/v1/messages/count_tokens`) translated to chat + completions and back, streamed or not: text, images, tools and their + results, the system prompt, stop sequences. Claude Code and Anthropic's + SDKs reach the models here; the key may come in `x-api-key`. Errors come + back in Anthropic's shape. + ### More of OpenAI's API - **Responses:** `text.format` asks for structured output, `input_image` diff --git a/docs/install.md b/docs/install.md index 01d4259c..ba2615ec 100644 --- a/docs/install.md +++ b/docs/install.md @@ -965,6 +965,7 @@ and accounting exactly as an API call does. |---|---| | `POST /v1/chat/completions`, `/v1/completions`, `/v1/embeddings` | OpenAI's, served by the models in the catalog | | `POST /v1/responses` | OpenAI's Responses API, translated to chat completions: tools, `text.format` (structured output), `input_image` (by URL or `data:` URL), and `previous_response_id` — the conversation is kept as a thread, so the project must keep conversations (`llmgw keys threads`); `store: false` keeps none | +| `POST /v1/messages`, `/v1/messages/count_tokens` | Anthropic's Messages API, translated to chat completions: text, images, tools and their results, the system prompt, streaming; the key in `x-api-key` or `Authorization`. Extended thinking, prompt caching and server tools are ignored | | `POST /v1/rerank`, `/v1/score` | passed through to an engine that serves them: vLLM with a reranker or a cross-encoder | | `POST /v1/audio/transcriptions`, `/v1/audio/translations` | the multipart form passed through as it came, to an engine that serves speech to text: vLLM with Whisper | | `POST /v1/moderations` | answered by the guardrails' judge model (`role: guard`): `flagged`, and the judge's own categories | diff --git a/docs/onboarding.md b/docs/onboarding.md index 66624355..6212daed 100644 --- a/docs/onboarding.md +++ b/docs/onboarding.md @@ -65,6 +65,11 @@ model already in it: Responses API, which the gateway translates at `/v1/responses`. - **Cline** — four settings to paste into its own screen, since it keeps them in VS Code's secret storage where no file reaches. +- **Claude Code** — not on the page yet; it speaks Anthropic's Messages API, + which the gateway translates at `/v1/messages`: + `ANTHROPIC_BASE_URL=https:// ANTHROPIC_AUTH_TOKEN= + ANTHROPIC_MODEL= ANTHROPIC_SMALL_FAST_MODEL= claude`. It + needs a model that calls tools well. For Continue and Codex there is a one-line command, `curl -fsSL /setup/?model=… | sh`. Its script is served without diff --git a/internal/auth/auth.go b/internal/auth/auth.go index 01335512..c4255544 100644 --- a/internal/auth/auth.go +++ b/internal/auth/auth.go @@ -314,5 +314,9 @@ func BearerToken(r *http.Request) string { // A bare value is common enough in hand-written clients to accept. return strings.TrimSpace(h) } - return strings.TrimSpace(r.Header.Get("api-key")) + if k := r.Header.Get("api-key"); k != "" { + return strings.TrimSpace(k) + } + // Anthropic's clients send their key here. + return strings.TrimSpace(r.Header.Get("x-api-key")) } diff --git a/internal/router/handlers.go b/internal/router/handlers.go index 9cdd91ee..21fe18f1 100644 --- a/internal/router/handlers.go +++ b/internal/router/handlers.go @@ -107,6 +107,9 @@ func (r *Router) Handler() http.Handler { // Codex, and whatever else speaks only the Responses API, translated to // chat completions on the way in and back on the way out. mux.HandleFunc("POST /v1/responses", r.handleResponses) + // Claude Code and Anthropic's SDKs, translated the same way. + mux.HandleFunc("POST /v1/messages", r.handleMessages) + mux.HandleFunc("POST /v1/messages/count_tokens", r.handleCountTokens) mux.HandleFunc("GET /v1/models", r.handleModels) // Files and batches: a file of requests served at the lowest priority. mux.HandleFunc("POST /v1/files", r.handleUploadFile) diff --git a/internal/router/messages.go b/internal/router/messages.go new file mode 100644 index 00000000..00690003 --- /dev/null +++ b/internal/router/messages.go @@ -0,0 +1,600 @@ +package router + +import ( + "bytes" + "encoding/json" + "fmt" + "io" + "net/http" + "strings" +) + +// Anthropic's Messages API (/v1/messages), translated to a chat completion +// on the way in and back on the way out, like the Responses API: so Claude +// Code and Anthropic's SDKs reach the models here. Text, images, tools and +// their results, the system prompt and streaming are translated; extended +// thinking, prompt caching and server tools are not, and are ignored rather +// than refused. + +type msgRequest struct { + Model string `json:"model"` + System json.RawMessage `json:"system"` + Messages []msgMessage `json:"messages"` + MaxTokens int `json:"max_tokens"` + StopSequences []string `json:"stop_sequences"` + Stream bool `json:"stream"` + Temperature *float64 `json:"temperature"` + TopP *float64 `json:"top_p"` + TopK *int `json:"top_k"` + Tools []msgTool `json:"tools"` + ToolChoice *struct { + Type string `json:"type"` + Name string `json:"name"` + } `json:"tool_choice"` + Metadata *struct { + UserID string `json:"user_id"` + } `json:"metadata"` +} + +type msgMessage struct { + Role string `json:"role"` + Content json.RawMessage `json:"content"` +} + +type msgTool struct { + Name string `json:"name"` + Description string `json:"description"` + InputSchema json.RawMessage `json:"input_schema"` + Type string `json:"type"` +} + +type msgBlock struct { + Type string `json:"type"` + Text string `json:"text"` + Source *struct { + Type string `json:"type"` + MediaType string `json:"media_type"` + Data string `json:"data"` + URL string `json:"url"` + } `json:"source"` + // tool_use + ID string `json:"id"` + Name string `json:"name"` + Input json.RawMessage `json:"input"` + // tool_result + ToolUseID string `json:"tool_use_id"` + Content json.RawMessage `json:"content"` + IsError bool `json:"is_error"` +} + +// msgBlocks reads content that is a string or a list of blocks. +func msgBlocks(raw json.RawMessage) ([]msgBlock, error) { + var s string + if json.Unmarshal(raw, &s) == nil { + return []msgBlock{{Type: "text", Text: s}}, nil + } + var bs []msgBlock + if err := json.Unmarshal(raw, &bs); err != nil { + return nil, fmt.Errorf("content is neither text nor a list of blocks") + } + return bs, nil +} + +func blocksText(raw json.RawMessage) string { + bs, err := msgBlocks(raw) + if err != nil { + return "" + } + var b strings.Builder + for _, x := range bs { + if x.Type == "text" { + b.WriteString(x.Text) + } + } + return b.String() +} + +// messagesToChat rewrites a Messages request as a chat completion. +func messagesToChat(in msgRequest) (map[string]any, error) { + var msgs []map[string]any + if sys := blocksText(in.System); sys != "" { + msgs = append(msgs, map[string]any{"role": "system", "content": sys}) + } + for _, m := range in.Messages { + bs, err := msgBlocks(m.Content) + if err != nil { + return nil, err + } + var parts []map[string]any + images := false + var calls []map[string]any + flush := func() { + if len(parts) == 0 && len(calls) == 0 { + return + } + msg := map[string]any{"role": m.Role} + if images { + msg["content"] = parts + } else { + var b strings.Builder + for _, p := range parts { + b.WriteString(fmt.Sprint(p["text"])) + } + msg["content"] = b.String() + } + if len(calls) > 0 { + msg["tool_calls"] = calls + } + msgs = append(msgs, msg) + parts, calls, images = nil, nil, false + } + for _, b := range bs { + switch b.Type { + case "text": + parts = append(parts, map[string]any{"type": "text", "text": b.Text}) + case "image": + if b.Source == nil { + continue + } + url := b.Source.URL + if b.Source.Type == "base64" { + url = "data:" + b.Source.MediaType + ";base64," + b.Source.Data + } + if url == "" { + return nil, fmt.Errorf("an image by %q source is not supported here", b.Source.Type) + } + images = true + parts = append(parts, map[string]any{"type": "image_url", "image_url": map[string]any{"url": url}}) + case "tool_use": + args := string(b.Input) + if args == "" { + args = "{}" + } + calls = append(calls, map[string]any{"id": b.ID, "type": "function", + "function": map[string]any{"name": b.Name, "arguments": args}}) + case "tool_result": + // A tool's answer is its own turn in a chat completion, + // after whatever the user said before it in this message. + flush() + out := blocksText(b.Content) + if out == "" && len(b.Content) > 0 { + var s string + if json.Unmarshal(b.Content, &s) != nil { + out = string(b.Content) + } + } + if b.IsError { + out = "Error: " + out + } + msgs = append(msgs, map[string]any{"role": "tool", "tool_call_id": b.ToolUseID, "content": out}) + } + // thinking, redacted_thinking, documents: nothing an engine here reads. + } + flush() + } + if len(msgs) == 0 { + return nil, fmt.Errorf("the request has no messages") + } + out := map[string]any{"model": in.Model, "messages": msgs, "stream": in.Stream} + if in.Stream { + out["stream_options"] = map[string]any{"include_usage": true} + } + if in.MaxTokens > 0 { + out["max_tokens"] = in.MaxTokens + } + if len(in.StopSequences) > 0 { + out["stop"] = in.StopSequences + } + if in.Temperature != nil { + out["temperature"] = *in.Temperature + } + if in.TopP != nil { + out["top_p"] = *in.TopP + } + if in.TopK != nil { + out["top_k"] = *in.TopK + } + if in.Metadata != nil && in.Metadata.UserID != "" { + out["user"] = in.Metadata.UserID + } + var tools []map[string]any + for _, t := range in.Tools { + if t.Type != "" && t.Type != "custom" { + continue // server tools: nothing here runs them + } + params := t.InputSchema + if len(params) == 0 { + params = json.RawMessage(`{"type":"object","properties":{}}`) + } + tools = append(tools, map[string]any{"type": "function", "function": map[string]any{ + "name": t.Name, "description": t.Description, "parameters": params}}) + } + if len(tools) > 0 { + out["tools"] = tools + if c := in.ToolChoice; c != nil { + switch c.Type { + case "auto": + out["tool_choice"] = "auto" + case "any": + out["tool_choice"] = "required" + case "none": + out["tool_choice"] = "none" + case "tool": + out["tool_choice"] = map[string]any{"type": "function", "function": map[string]any{"name": c.Name}} + } + } + } + return out, nil +} + +func (r *Router) handleMessages(w http.ResponseWriter, req *http.Request) { + body, err := io.ReadAll(io.LimitReader(req.Body, r.cfg.MaxBodyBytes)) + if err != nil { + writeAnthropicError(w, http.StatusBadRequest, "invalid_request_error", "could not read the request") + return + } + var in msgRequest + if err := json.Unmarshal(body, &in); err != nil { + writeAnthropicError(w, http.StatusBadRequest, "invalid_request_error", "the body is not JSON: "+err.Error()) + return + } + chat, err := messagesToChat(in) + if err != nil { + writeAnthropicError(w, http.StatusBadRequest, "invalid_request_error", err.Error()) + return + } + upstream, _ := json.Marshal(chat) + next := req.Clone(req.Context()) + next.URL.Path = "/v1/chat/completions" + next.Body = io.NopCloser(bytes.NewReader(upstream)) + next.ContentLength = int64(len(upstream)) + next.Header.Set("Content-Type", "application/json") + + t := &msgTranslator{w: w, model: in.Model, stream: in.Stream, id: newID("msg_")} + r.handleInference("/v1/chat/completions")(t, next) + t.finish() +} + +// handleCountTokens answers /v1/messages/count_tokens with the gateway's +// own estimate, bytes over four: what it limits by. +func (r *Router) handleCountTokens(w http.ResponseWriter, req *http.Request) { + body, err := io.ReadAll(io.LimitReader(req.Body, r.cfg.MaxBodyBytes)) + if err != nil { + writeAnthropicError(w, http.StatusBadRequest, "invalid_request_error", "could not read the request") + return + } + var in msgRequest + if err := json.Unmarshal(body, &in); err != nil { + writeAnthropicError(w, http.StatusBadRequest, "invalid_request_error", "the body is not JSON: "+err.Error()) + return + } + n := len(blocksText(in.System)) + for _, m := range in.Messages { + n += len(m.Content) + } + for _, t := range in.Tools { + n += len(t.Name) + len(t.Description) + len(t.InputSchema) + } + writeJSON(w, http.StatusOK, map[string]int{"input_tokens": max(n/4, 1)}) +} + +// writeAnthropicError writes an error the way Anthropic's API does. +func writeAnthropicError(w http.ResponseWriter, status int, kind, msg string) { + writeJSON(w, status, map[string]any{"type": "error", "error": map[string]string{"type": kind, "message": msg}}) +} + +// anthropicErrorKind is the Anthropic name for a status. +func anthropicErrorKind(status int) string { + switch { + case status == 400 || status == 413 || status == 422: + return "invalid_request_error" + case status == 401: + return "authentication_error" + case status == 403: + return "permission_error" + case status == 404: + return "not_found_error" + case status == 429: + return "rate_limit_error" + case status == 503 || status == 529: + return "overloaded_error" + } + return "api_error" +} + +// msgTranslator stands between the chat handler and the client, rewriting +// what it writes into the Messages API's shape. +type msgTranslator struct { + w http.ResponseWriter + model string + stream bool + id string + + status int + failed bool + errBuf bytes.Buffer + buf bytes.Buffer + started bool + + // The content blocks as the stream builds them: the text block, if any, + // and the tool calls, each its own block, in the order they began. + textIndex int + textOpen bool + text strings.Builder + calls []*msgCall + byIndex map[int]*msgCall + nextBlock int + stopReason string + usage [2]int + finished bool +} + +type msgCall struct { + block int + id, name string + args strings.Builder + open bool +} + +func (t *msgTranslator) Header() http.Header { return t.w.Header() } + +func (t *msgTranslator) WriteHeader(code int) { + t.status = code + if code >= 300 { + // Rewritten at the end as Anthropic's error shape. + t.failed = true + return + } + if t.stream { + t.w.Header().Set("Content-Type", "text/event-stream") + } else { + t.w.Header().Set("Content-Type", "application/json") + } + t.w.Header().Del("Content-Length") + t.w.WriteHeader(code) +} + +func (t *msgTranslator) Flush() { + if f, ok := t.w.(http.Flusher); ok { + f.Flush() + } +} + +func (t *msgTranslator) Write(p []byte) (int, error) { + if t.status == 0 { + t.WriteHeader(http.StatusOK) + } + if t.failed { + t.errBuf.Write(p) + return len(p), nil + } + t.buf.Write(p) + if !t.stream { + return len(p), nil + } + for { + line, err := t.buf.ReadString('\n') + if err != nil { + t.buf.Reset() + t.buf.WriteString(line) + break + } + t.chatLine(strings.TrimSpace(line)) + } + return len(p), nil +} + +func (t *msgTranslator) event(typ string, fields map[string]any) { + fields["type"] = typ + b, _ := json.Marshal(fields) + fmt.Fprintf(t.w, "event: %s\ndata: %s\n\n", typ, b) +} + +func (t *msgTranslator) begin() { + if t.started { + return + } + t.started = true + t.byIndex = map[int]*msgCall{} + t.event("message_start", map[string]any{"message": map[string]any{ + "id": t.id, "type": "message", "role": "assistant", "model": t.model, "content": []any{}, + "stop_reason": nil, "stop_sequence": nil, "usage": map[string]int{"input_tokens": 0, "output_tokens": 0}}}) +} + +func stopReason(finish string) string { + switch finish { + case "length": + return "max_tokens" + case "tool_calls", "function_call": + return "tool_use" + case "stop_sequence": + return "stop_sequence" + case "content_filter": + return "refusal" + } + return "end_turn" +} + +func (t *msgTranslator) closeText() { + if t.textOpen { + t.event("content_block_stop", map[string]any{"index": t.textIndex}) + t.textOpen = false + } +} + +func (t *msgTranslator) chatLine(line string) { + if !strings.HasPrefix(line, "data:") { + return + } + payload := strings.TrimSpace(strings.TrimPrefix(line, "data:")) + if payload == "[DONE]" { + return + } + var chunk struct { + Choices []struct { + Delta struct { + Content string `json:"content"` + ToolCalls []struct { + Index int `json:"index"` + ID string `json:"id"` + Function struct { + Name string `json:"name"` + Arguments string `json:"arguments"` + } `json:"function"` + } `json:"tool_calls"` + } `json:"delta"` + FinishReason string `json:"finish_reason"` + } `json:"choices"` + Usage *struct { + Prompt int `json:"prompt_tokens"` + Completion int `json:"completion_tokens"` + } `json:"usage"` + } + if json.Unmarshal([]byte(payload), &chunk) != nil { + return + } + t.begin() + if chunk.Usage != nil { + t.usage = [2]int{chunk.Usage.Prompt, chunk.Usage.Completion} + } + for _, ch := range chunk.Choices { + if ch.FinishReason != "" { + t.stopReason = stopReason(ch.FinishReason) + } + if d := ch.Delta.Content; d != "" { + if !t.textOpen && t.text.Len() == 0 { + t.textIndex, t.textOpen = t.nextBlock, true + t.nextBlock++ + t.event("content_block_start", map[string]any{"index": t.textIndex, + "content_block": map[string]any{"type": "text", "text": ""}}) + } + t.text.WriteString(d) + if t.textOpen { + t.event("content_block_delta", map[string]any{"index": t.textIndex, + "delta": map[string]any{"type": "text_delta", "text": d}}) + } + } + for _, tc := range ch.Delta.ToolCalls { + c := t.byIndex[tc.Index] + if c == nil { + t.closeText() + for _, o := range t.calls { + if o.open { + t.event("content_block_stop", map[string]any{"index": o.block}) + o.open = false + } + } + c = &msgCall{block: t.nextBlock, id: tc.ID, name: tc.Function.Name, open: true} + if c.id == "" { + c.id = newID("toolu_") + } + t.nextBlock++ + t.byIndex[tc.Index] = c + t.calls = append(t.calls, c) + t.event("content_block_start", map[string]any{"index": c.block, "content_block": map[string]any{ + "type": "tool_use", "id": c.id, "name": c.name, "input": map[string]any{}}}) + } + if a := tc.Function.Arguments; a != "" { + c.args.WriteString(a) + t.event("content_block_delta", map[string]any{"index": c.block, + "delta": map[string]any{"type": "input_json_delta", "partial_json": a}}) + } + } + } +} + +func (t *msgTranslator) finish() { + if t.finished { + return + } + t.finished = true + if t.failed { + msg := strings.TrimSpace(t.errBuf.String()) + var e struct { + Error struct { + Message string `json:"message"` + } `json:"error"` + } + if json.Unmarshal(t.errBuf.Bytes(), &e) == nil && e.Error.Message != "" { + msg = e.Error.Message + } + if ra := t.w.Header().Get("Retry-After"); ra != "" { + t.w.Header().Set("Retry-After", ra) + } + writeAnthropicError(t.w, t.status, anthropicErrorKind(t.status), msg) + return + } + if !t.stream { + t.fromChatJSON(t.buf.Bytes()) + return + } + if rest := strings.TrimSpace(t.buf.String()); rest != "" { + t.chatLine(rest) + } + t.begin() + t.closeText() + for _, c := range t.calls { + if c.open { + t.event("content_block_stop", map[string]any{"index": c.block}) + } + } + reason := t.stopReason + if reason == "" { + reason = "end_turn" + } + t.event("message_delta", map[string]any{"delta": map[string]any{"stop_reason": reason, "stop_sequence": nil}, + "usage": map[string]int{"input_tokens": t.usage[0], "output_tokens": t.usage[1]}}) + t.event("message_stop", map[string]any{}) + t.Flush() +} + +// fromChatJSON rewrites a whole chat completion as a message. +func (t *msgTranslator) fromChatJSON(b []byte) { + var c struct { + Choices []struct { + Message struct { + Content string `json:"content"` + ToolCalls []struct { + ID string `json:"id"` + Function struct { + Name string `json:"name"` + Arguments string `json:"arguments"` + } `json:"function"` + } `json:"tool_calls"` + } `json:"message"` + FinishReason string `json:"finish_reason"` + } `json:"choices"` + Usage struct { + Prompt int `json:"prompt_tokens"` + Completion int `json:"completion_tokens"` + } `json:"usage"` + } + if err := json.Unmarshal(b, &c); err != nil { + _, _ = t.w.Write(b) + return + } + content := []map[string]any{} + reason := "end_turn" + if len(c.Choices) > 0 { + ch := c.Choices[0] + reason = stopReason(ch.FinishReason) + if ch.Message.Content != "" { + content = append(content, map[string]any{"type": "text", "text": ch.Message.Content}) + } + for _, tc := range ch.Message.ToolCalls { + input := json.RawMessage(tc.Function.Arguments) + if !json.Valid(input) { + input = json.RawMessage(`{}`) + } + id := tc.ID + if id == "" { + id = newID("toolu_") + } + content = append(content, map[string]any{"type": "tool_use", "id": id, "name": tc.Function.Name, "input": input}) + } + } + out, _ := json.Marshal(map[string]any{"id": t.id, "type": "message", "role": "assistant", "model": t.model, + "content": content, "stop_reason": reason, "stop_sequence": nil, + "usage": map[string]int{"input_tokens": c.Usage.Prompt, "output_tokens": c.Usage.Completion}}) + _, _ = t.w.Write(out) +} diff --git a/internal/router/messages_test.go b/internal/router/messages_test.go new file mode 100644 index 00000000..10192ba1 --- /dev/null +++ b/internal/router/messages_test.go @@ -0,0 +1,141 @@ +package router + +import ( + "bytes" + "encoding/json" + "net/http" + "net/http/httptest" + "strings" + "testing" + + "gitea.mynodes.xyz/ilker/llmgw/internal/config" +) + +func TestAMessagesRequestBecomesAChatCompletion(t *testing.T) { + var in msgRequest + _ = json.Unmarshal([]byte(`{"model":"m","max_tokens":100,"system":[{"type":"text","text":"be brief"}], + "stop_sequences":["END"],"tool_choice":{"type":"any"}, + "tools":[{"name":"get_weather","description":"w","input_schema":{"type":"object"}}], + "messages":[ + {"role":"user","content":[{"type":"text","text":"look"},{"type":"image","source":{"type":"base64","media_type":"image/png","data":"AAAA"}}]}, + {"role":"assistant","content":[{"type":"text","text":"checking"},{"type":"tool_use","id":"toolu_1","name":"get_weather","input":{"city":"Ankara"}}]}, + {"role":"user","content":[{"type":"tool_result","tool_use_id":"toolu_1","content":"sunny"},{"type":"text","text":"and tomorrow?"}]}]}`), &in) + chat, err := messagesToChat(in) + if err != nil { + t.Fatal(err) + } + b, _ := json.Marshal(chat) + for _, want := range []string{ + `{"content":"be brief","role":"system"}`, + `"image_url":{"url":"data:image/png;base64,AAAA"}`, + `"tool_calls":[{"function":{"arguments":"{\"city\":\"Ankara\"}","name":"get_weather"},"id":"toolu_1","type":"function"}]`, + `{"content":"sunny","role":"tool","tool_call_id":"toolu_1"}`, + `{"content":"and tomorrow?","role":"user"}`, + `"max_tokens":100`, `"stop":["END"]`, `"tool_choice":"required"`, + } { + if !strings.Contains(string(b), want) { + t.Errorf("lacks %s:\n%s", want, b) + } + } +} + +func messagesRouter(t *testing.T, answer string) (*Router, http.Handler) { + t.Helper() + srv, _, _ := engineFor(t, "m", map[string]string{"/v1/chat/completions": answer}) + cfg := baseConfig(config.Endpoint{ID: "a", URL: srv.URL, Models: []string{"m"}}) + rt := newTestRouter(t, cfg) + return rt, rt.Handler() +} + +func sendMessages(t *testing.T, h http.Handler, body map[string]any) *httptest.ResponseRecorder { + t.Helper() + b, _ := json.Marshal(body) + req := httptest.NewRequest(http.MethodPost, "/v1/messages", bytes.NewReader(b)) + req.Header.Set("Content-Type", "application/json") + rec := httptest.NewRecorder() + h.ServeHTTP(rec, req) + return rec +} + +func TestAMessageComesBackInAnthropicsShape(t *testing.T) { + _, h := messagesRouter(t, `{"choices":[{"message":{"content":"Hello","tool_calls":[{"id":"call_9","function":{"name":"f","arguments":"{\"a\":1}"}}]},"finish_reason":"tool_calls"}],"usage":{"prompt_tokens":5,"completion_tokens":3}}`) + rec := sendMessages(t, h, map[string]any{"model": "m", "max_tokens": 10, + "messages": []any{map[string]any{"role": "user", "content": "hi"}}}) + var out struct { + Type string `json:"type"` + Role string `json:"role"` + Content []map[string]any `json:"content"` + StopReason string `json:"stop_reason"` + Usage map[string]int `json:"usage"` + } + _ = json.Unmarshal(rec.Body.Bytes(), &out) + if rec.Code != 200 || out.Type != "message" || out.Role != "assistant" || len(out.Content) != 2 || + out.Content[0]["text"] != "Hello" || out.Content[1]["type"] != "tool_use" || out.StopReason != "tool_use" || + out.Usage["input_tokens"] != 5 { + t.Errorf("%d %s", rec.Code, rec.Body.String()) + } + + rec = sendMessages(t, h, map[string]any{"model": "nope", "max_tokens": 10, + "messages": []any{map[string]any{"role": "user", "content": "hi"}}}) + var e struct { + Type string `json:"type"` + Error struct { + Type, Message string + } `json:"error"` + } + _ = json.Unmarshal(rec.Body.Bytes(), &e) + if rec.Code < 400 || e.Type != "error" || e.Error.Type == "" || e.Error.Message == "" { + t.Errorf("error shape: %d %s", rec.Code, rec.Body.String()) + } +} + +func TestAStreamedMessageIsAnthropicsEvents(t *testing.T) { + stream := "data: {\"choices\":[{\"delta\":{\"content\":\"Hel\"}}]}\n\n" + + "data: {\"choices\":[{\"delta\":{\"content\":\"lo\"}}]}\n\n" + + "data: {\"choices\":[{\"delta\":{\"tool_calls\":[{\"index\":0,\"id\":\"call_1\",\"function\":{\"name\":\"f\",\"arguments\":\"{\\\"a\\\"\"}}]}}]}\n\n" + + "data: {\"choices\":[{\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\":1}\"}}]},\"finish_reason\":\"tool_calls\"}]}\n\n" + + "data: {\"choices\":[],\"usage\":{\"prompt_tokens\":4,\"completion_tokens\":6}}\n\n" + + "data: [DONE]\n\n" + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/v1/chat/completions" { + return + } + w.Header().Set("Content-Type", "text/event-stream") + _, _ = w.Write([]byte(stream)) + })) + t.Cleanup(srv.Close) + rt := newTestRouter(t, baseConfig(config.Endpoint{ID: "a", URL: srv.URL, Models: []string{"m"}})) + rec := sendMessages(t, rt.Handler(), map[string]any{"model": "m", "max_tokens": 10, "stream": true, + "messages": []any{map[string]any{"role": "user", "content": "hi"}}}) + got := rec.Body.String() + var events []string + for _, l := range strings.Split(got, "\n") { + if e, ok := strings.CutPrefix(l, "event: "); ok { + events = append(events, e) + } + } + want := "message_start content_block_start content_block_delta content_block_delta content_block_stop " + + "content_block_start content_block_delta content_block_delta content_block_stop message_delta message_stop" + if strings.Join(events, " ") != want { + t.Errorf("events:\n%s\nwant:\n%s\n%s", strings.Join(events, " "), want, got) + } + for _, s := range []string{`"text":"Hel","type":"text_delta"`, `"partial_json":":1}"`, `"stop_reason":"tool_use"`, `"output_tokens":6`} { + if !strings.Contains(got, s) { + t.Errorf("lacks %s", s) + } + } +} + +func TestAnAnthropicClientsKeyIsAccepted(t *testing.T) { + e := newFakeEngine(t, "qwen2.5-14b-instruct") + rt, secret := pricedRouter(t, e.srv.URL) + b, _ := json.Marshal(map[string]any{"model": "q14", "max_tokens": 5, "messages": []any{map[string]any{"role": "user", "content": "hi"}}}) + req := httptest.NewRequest(http.MethodPost, "/v1/messages", bytes.NewReader(b)) + req.Header.Set("x-api-key", secret) + req.Header.Set("anthropic-version", "2023-06-01") + rec := httptest.NewRecorder() + rt.Handler().ServeHTTP(rec, req) + if rec.Code != 200 || !strings.Contains(rec.Body.String(), `"type":"message"`) { + t.Errorf("%d %s", rec.Code, rec.Body.String()) + } +} From d29f694aaf205d47e4fb460b9572e5fa978c73a5 Mon Sep 17 00:00:00 2001 From: Ilker Manap Date: Sun, 4 Oct 2026 23:03:35 +0200 Subject: [PATCH 09/38] Moderations name a judge's categories once, by name Co-Authored-By: Claude Opus 5.5 (1M context) --- internal/router/compat_test.go | 2 +- internal/router/moderations.go | 6 ++++++ 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/internal/router/compat_test.go b/internal/router/compat_test.go index bbf07268..043453b4 100644 --- a/internal/router/compat_test.go +++ b/internal/router/compat_test.go @@ -172,7 +172,7 @@ func TestModerationsAreAnsweredByTheJudge(t *testing.T) { } _ = json.Unmarshal(rec.Body.Bytes(), &out) if rec.Code != 200 || out.Model != "guard" || len(out.Results) != 1 || !out.Results[0].Flagged || - !out.Results[0].Categories["indiscriminate-weapons"] { + !out.Results[0].Categories["indiscriminate-weapons"] || out.Results[0].Categories["s9"] { t.Errorf("%d %s", rec.Code, rec.Body.String()) } } diff --git a/internal/router/moderations.go b/internal/router/moderations.go index 637ccd2b..bb984de8 100644 --- a/internal/router/moderations.go +++ b/internal/router/moderations.go @@ -57,6 +57,12 @@ func (r *Router) handleModerations(w http.ResponseWriter, req *http.Request) { categories[moderationKey(n)], scores[moderationKey(n)] = false, 0 } for _, c := range cats { + // The judge says both a code and the name it stands for; the + // name is what is shown, when there is one. + var n int + if _, err := fmt.Sscanf(c, "S%d", &n); err == nil && n >= 1 && n <= len(names) { + continue + } categories[moderationKey(c)], scores[moderationKey(c)] = true, 1 } results = append(results, map[string]any{"flagged": unsafe, "categories": categories, "category_scores": scores}) From 10466416df87995779e1133e4dd55704103a89f8 Mon Sep 17 00:00:00 2001 From: Ilker Manap Date: Sun, 4 Oct 2026 23:09:59 +0200 Subject: [PATCH 10/38] =?UTF-8?q?server=20restore:=20an=20archive=20checke?= =?UTF-8?q?d,=20dry-run,=20and=20put=20back=20=E2=80=94=20configuration=20?= =?UTF-8?q?with=20modes=20and=20owners,=20the=20store=20in=20SQLite=20or?= =?UTF-8?q?=20PostgreSQL;=20backup=5Fcopy=5Fto=20copies=20archives=20off?= =?UTF-8?q?=20the=20machine?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 (1M context) --- BACKLOG.md | 8 +- CHANGELOG.md | 14 + docs/install.md | 66 ++-- docs/reference.md | 19 ++ internal/cli/backup.go | 31 +- internal/cli/backupcopy.go | 84 +++++ internal/cli/restore.go | 546 +++++++++++++++++++++++++++++++ internal/cli/restore_test.go | 118 +++++++ internal/cli/server.go | 2 +- internal/config/config.go | 24 ++ internal/docgen/reference_gen.go | 2 + 11 files changed, 889 insertions(+), 25 deletions(-) create mode 100644 internal/cli/backupcopy.go create mode 100644 internal/cli/restore.go create mode 100644 internal/cli/restore_test.go diff --git a/BACKLOG.md b/BACKLOG.md index 6ae5f0d6..3b46c912 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -465,7 +465,13 @@ portal is not done — the page shows; the console and ctl change. thinking blocks); prompt caching; the account page's "Connect your editor" does not offer Claude Code yet. **Anthropic `/v1/messages` (M).** Translate like `responses.go`. -12. **`server restore` (S).** Restore is manual shell today +12. **Done (branch `features-c2`):** `server restore` (dry run, version + check, decryption, modes and owners, SQLite and PostgreSQL), + `backup_copy_to` (directory or ssh), the drill in docs/install.md. + Left: restoring a replicated store's members (restore into the primary; + Patroni replicates it); the console's Back up now does not offer + restore; an ssh copy holds the archive in memory while it sends it. + **`server restore` (S).** Restore is manual shell today (`docs/install.md:1187`); add the command with version check, dry run, off-host copy target; a documented DR drill. 13. **SLO-based admission and token-level metrics (S–M).** TPOT and diff --git a/CHANGELOG.md b/CHANGELOG.md index 21d1683b..0b537b86 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -71,6 +71,20 @@ APIs change between minors without a deprecation period. - **Other ways in:** `GET /api/v1/usage/export` (CSV, or `format=json`) and **E** on the console's Usage view. +### Restoring a backup, and copies off the machine + +- **`llmgw server restore ARCHIVE`** puts a backup back: checks it first + (whole, opens with `--identity`, not newer than this llmgw, where the store + goes), says so with `--dry-run`, then moves `/etc/llmgw` aside, restores it + with modes and owners, restores the store (SQLite, or PostgreSQL with + `pg_restore --clean`), and starts the server. A drill is in + docs/install.md. +- **`backup_copy_to`** copies each archive to a directory or over ssh as it + is written, keeping as many there as here; refused without + `backup_recipients`. +- Archives now record each file's owner by name, so a restore on another + machine gives the llmgw user its files back. + ### Anthropic's Messages API - **`/v1/messages`** (and `/v1/messages/count_tokens`) translated to chat diff --git a/docs/install.md b/docs/install.md index ba2615ec..aaf8fdce 100644 --- a/docs/install.md +++ b/docs/install.md @@ -1529,35 +1529,63 @@ anywhere else.** Restoring an old store restores which keys exist and who owns them, not their secrets. A key created after the backup is gone and has to be reissued. +### Copying archives off the machine + +```yaml +backup_recipients: [age1…] # required: an archive leaves only encrypted +backup_copy_to: + - /mnt/backups/llmgw # a mounted share or another disk + - ssh://backup@10.0.0.20/srv/llmgw-backups +backup_copy_key: /etc/llmgw/secrets/backup-ssh-key +``` + +Each archive is copied there as soon as it is written, and the same number is +kept there as here (`--keep`, 14). An ssh host's key is learnt the first time +and checked every time after, in `/etc/llmgw/backup_known_hosts`. A copy that +fails makes the backup fail, so the timer's unit shows it. + ### Restoring -On a machine with llmgw installed (the same version or newer) and, for -PostgreSQL, an empty database and role as `server init --postgres` would create -them: +On a machine with llmgw installed — the same version as the archive's, or +newer — and, for PostgreSQL, the database and role the store's address names: ``` -sudo systemctl stop llmgw-server -mkdir /tmp/restore && sudo tar -xzf /var/backups/llmgw-server/llmgw-backup--