diff --git a/BACKLOG.md b/BACKLOG.md index 446ebecc..2d061dbc 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -1426,9 +1426,92 @@ console show and add them. `on_demand.evict_after_idle` / `min_lifetime` as live settings; an upgrade that leaves nodes behind says which; `ctl model serve` serves an embedding model for embedding; refreshes wait while a form is filled in. -- Open: llmgw's `/mcp` with picture, 3D, VLM and workflow tools, so a - client needs no server of its own (asked by the game team); the shared - ComfyUI replica (H3's placement side). +- Done 2026-10-15: llmgw's `/mcp` with picture, 3D, VLM and workflow tools, + so a client needs no server of its own (asked by the game team); the + shared ComfyUI replica (H3's placement side). + +## I. What other gateways do that llmgw does not (written 2026-10-16) + +From a comparison with Bifrost (maximhq/bifrost, Apache 2.0, Go), with +LiteLLM and Portkey in mind. Bifrost routes to 23+ cloud providers and runs +no model; llmgw runs the models, so most of Bifrost is either in llmgw +already (keys, budgets, fallbacks, guardrails, tracing, SSO, HA — the last +three paid in Bifrost) or does not apply. These four are worth taking for +a site that runs its own models. Left until a customer asks: native +Azure/Anthropic/Bedrock/Vertex adapters, request hooks a customer plugs in, +routing by what a request holds. Not taken: a realtime (WebSocket audio) +API, prompt management, an `npx` install. + +### I1. Answers to similar questions, not only the same one (M) +The answer cache (`internal/router/cache.go`, C17) answers a request whose +body is the same byte for byte. A help desk asked "how do I reset my +password" in forty wordings asks the model forty times. +- Per project, off by default, beside the exact cache: `llmgw keys cache + --project P --similar 0.95` (the cosine similarity a question must reach; + higher is stricter). The exact cache is asked first. +- The question embedded by a model the operator names + (`router.cache.embedding_model`, an embedding model in the catalog), into a + pgvector table per project with the answer, the model and the TTL; the + store's halfvec and text index work from F applies. SQLite stores keep the + exact cache only. +- What counts as the question: the last user turn, with the system prompt + and the earlier turns hashed and required to match exactly — a similar + question in another conversation is not the same question. Tools, + `response_format` and the sampling settings also exact. +- Said: `x-llmgw-cache: similar` with the similarity, and the usage row + says so; guardrails judge the request before a hit, as for the exact + cache. A project with `no_store` gets neither. +- Done when: forty rewordings of one question on the PoC cost one model call + and 39 lookups, and an eval suite run with the cache on scores within a + point of the run with it off. + +### I2. MCP gateway: the customer's tool servers behind llmgw (L) +`/mcp` serves llmgw's own tools. An agent inside a company also wants Jira, +the file server, a database — each an MCP server, each today configured +in every client with its own credentials and nobody seeing what was called. +- Servers registered once, by an administrator: `llmgw ctl mcp add jira + --url https://… --header-file /etc/llmgw/secrets/jira` (HTTP transport; + stdio servers run on the control plane under their own user, like an + engine). Their credentials stay on the control plane, never in a client. +- `/mcp` lists every server's tools, prefixed by its name (`jira.search`), + filtered by what the key may use: an allowlist per org, team, project or + key, by server or by tool, in the tenancy tree as model allowlists are. +- Each call goes to the audit log — who, which tool, the arguments' size, + the outcome and how long — and through the guardrails both ways (a tool's + answer is prompt input). A per-tool rate limit. +- Later, if it earns it: Bifrost's "code mode" (the tools as one typed API + the model writes code against), which cuts the tokens of a long tool list. +- Done when: Claude Code given only llmgw's `/mcp` reaches a test MCP server + through it, a key without the grant does not see the server's tools, and + every call is in `ctl audit`. + +### I3. What llmgw adds to a request, measured (S) +Bifrost's headline is "11 µs at 5k RPS" (its own figure, and its own pages +disagree with it). llmgw has none, so any question about its overhead is +answered with a guess. +- `llmgw ctl bench --overhead`: the same load through the gateway and straight + at a `sim-engine` (`internal/cli/simengine.go`) that answers at once, at + rising request rates, with auth, accounting and the guardrails' pattern + checks on and off. Reports the added latency at p50/p99 and the gateway's + CPU per thousand requests. +- On the PoC the CPU is oversubscribed (pve1, 28 threads for ~125 vCPUs): + report it as a ratio and record the host's load beside it, and repeat on + an idle machine before a figure goes into the README. +- Done when: the README states the overhead, with how it was measured, and + CI runs the short version so a change that doubles it is seen. + +### I4. Several keys for one outside provider (S) +An external model has one `api_key_file` (`internal/control/control.go`, +where external endpoints are made). A provider's limit is per key; one key +answering 429 makes the whole fallback answer 429. +- `api_key_files: [a, b, c]`, each with an optional `weight`: one endpoint + per key (`external:#`), so the router's scoring, retries and + breakers work per key as they already do per replica. A key answering 429 + or 401 is passed over while its breaker is open; a 401 also notifies + (`external_key_refused`). +- The usage row says which key answered (by its index, never the key). +- Done when: with two keys and one returning 429, every request is answered + by the other, and the console shows the one breaker open. ## D. Facts worth knowing before starting