Backlog I: what other gateways do that llmgw does not

From a comparison with Bifrost: answers to similar questions, the
customer's MCP servers behind llmgw, the gateway's overhead measured, and
several keys for one outside provider. H9's /mcp item is done.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-16 13:29:20 +02:00
1 parent 63b2c6d9a4
commit b8fc1de822
1 file changed
+86 -3
+86 -3
View File
@@ -1426,9 +1426,92 @@ console show and add them.
`on_demand.evict_after_idle` / `min_lifetime` as live settings; an
upgrade that leaves nodes behind says which; `ctl model serve` serves an
embedding model for embedding; refreshes wait while a form is filled in.
- Open: llmgw's `/mcp` with picture, 3D, VLM and workflow tools, so a
client needs no server of its own (asked by the game team); the shared
ComfyUI replica (H3's placement side).
- Done 2026-10-15: llmgw's `/mcp` with picture, 3D, VLM and workflow tools,
so a client needs no server of its own (asked by the game team); the
shared ComfyUI replica (H3's placement side).
## I. What other gateways do that llmgw does not (written 2026-10-16)
From a comparison with Bifrost (maximhq/bifrost, Apache 2.0, Go), with
LiteLLM and Portkey in mind. Bifrost routes to 23+ cloud providers and runs
no model; llmgw runs the models, so most of Bifrost is either in llmgw
already (keys, budgets, fallbacks, guardrails, tracing, SSO, HA — the last
three paid in Bifrost) or does not apply. These four are worth taking for
a site that runs its own models. Left until a customer asks: native
Azure/Anthropic/Bedrock/Vertex adapters, request hooks a customer plugs in,
routing by what a request holds. Not taken: a realtime (WebSocket audio)
API, prompt management, an `npx` install.
### I1. Answers to similar questions, not only the same one (M)
The answer cache (`internal/router/cache.go`, C17) answers a request whose
body is the same byte for byte. A help desk asked "how do I reset my
password" in forty wordings asks the model forty times.
- Per project, off by default, beside the exact cache: `llmgw keys cache
--project P --similar 0.95` (the cosine similarity a question must reach;
higher is stricter). The exact cache is asked first.
- The question embedded by a model the operator names
(`router.cache.embedding_model`, an embedding model in the catalog), into a
pgvector table per project with the answer, the model and the TTL; the
store's halfvec and text index work from F applies. SQLite stores keep the
exact cache only.
- What counts as the question: the last user turn, with the system prompt
and the earlier turns hashed and required to match exactly — a similar
question in another conversation is not the same question. Tools,
`response_format` and the sampling settings also exact.
- Said: `x-llmgw-cache: similar` with the similarity, and the usage row
says so; guardrails judge the request before a hit, as for the exact
cache. A project with `no_store` gets neither.
- Done when: forty rewordings of one question on the PoC cost one model call
and 39 lookups, and an eval suite run with the cache on scores within a
point of the run with it off.
### I2. MCP gateway: the customer's tool servers behind llmgw (L)
`/mcp` serves llmgw's own tools. An agent inside a company also wants Jira,
the file server, a database — each an MCP server, each today configured
in every client with its own credentials and nobody seeing what was called.
- Servers registered once, by an administrator: `llmgw ctl mcp add jira
--url https://… --header-file /etc/llmgw/secrets/jira` (HTTP transport;
stdio servers run on the control plane under their own user, like an
engine). Their credentials stay on the control plane, never in a client.
- `/mcp` lists every server's tools, prefixed by its name (`jira.search`),
filtered by what the key may use: an allowlist per org, team, project or
key, by server or by tool, in the tenancy tree as model allowlists are.
- Each call goes to the audit log — who, which tool, the arguments' size,
the outcome and how long — and through the guardrails both ways (a tool's
answer is prompt input). A per-tool rate limit.
- Later, if it earns it: Bifrost's "code mode" (the tools as one typed API
the model writes code against), which cuts the tokens of a long tool list.
- Done when: Claude Code given only llmgw's `/mcp` reaches a test MCP server
through it, a key without the grant does not see the server's tools, and
every call is in `ctl audit`.
### I3. What llmgw adds to a request, measured (S)
Bifrost's headline is "11 µs at 5k RPS" (its own figure, and its own pages
disagree with it). llmgw has none, so any question about its overhead is
answered with a guess.
- `llmgw ctl bench --overhead`: the same load through the gateway and straight
at a `sim-engine` (`internal/cli/simengine.go`) that answers at once, at
rising request rates, with auth, accounting and the guardrails' pattern
checks on and off. Reports the added latency at p50/p99 and the gateway's
CPU per thousand requests.
- On the PoC the CPU is oversubscribed (pve1, 28 threads for ~125 vCPUs):
report it as a ratio and record the host's load beside it, and repeat on
an idle machine before a figure goes into the README.
- Done when: the README states the overhead, with how it was measured, and
CI runs the short version so a change that doubles it is seen.
### I4. Several keys for one outside provider (S)
An external model has one `api_key_file` (`internal/control/control.go`,
where external endpoints are made). A provider's limit is per key; one key
answering 429 makes the whole fallback answer 429.
- `api_key_files: [a, b, c]`, each with an optional `weight`: one endpoint
per key (`external:<model>#<n>`), so the router's scoring, retries and
breakers work per key as they already do per replica. A key answering 429
or 401 is passed over while its breaker is open; a 401 also notifies
(`external_key_refused`).
- The usage row says which key answered (by its index, never the key).
- Done when: with two keys and one returning 429, every request is answered
by the other, and the console shows the one breaker open.
## D. Facts worth knowing before starting