From a comparison with Bifrost: answers to similar questions, the customer's MCP servers behind llmgw, the gateway's overhead measured, and several keys for one outside provider. H9's /mcp item is done. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
63b2c6d9a4
commit
b8fc1de822
1 file changed
+86
-3
+86
-3
@@ -1426,9 +1426,92 @@ console show and add them.
|
||||
`on_demand.evict_after_idle` / `min_lifetime` as live settings; an
|
||||
upgrade that leaves nodes behind says which; `ctl model serve` serves an
|
||||
embedding model for embedding; refreshes wait while a form is filled in.
|
||||
- Open: llmgw's `/mcp` with picture, 3D, VLM and workflow tools, so a
|
||||
client needs no server of its own (asked by the game team); the shared
|
||||
ComfyUI replica (H3's placement side).
|
||||
- Done 2026-10-15: llmgw's `/mcp` with picture, 3D, VLM and workflow tools,
|
||||
so a client needs no server of its own (asked by the game team); the
|
||||
shared ComfyUI replica (H3's placement side).
|
||||
|
||||
## I. What other gateways do that llmgw does not (written 2026-10-16)
|
||||
|
||||
From a comparison with Bifrost (maximhq/bifrost, Apache 2.0, Go), with
|
||||
LiteLLM and Portkey in mind. Bifrost routes to 23+ cloud providers and runs
|
||||
no model; llmgw runs the models, so most of Bifrost is either in llmgw
|
||||
already (keys, budgets, fallbacks, guardrails, tracing, SSO, HA — the last
|
||||
three paid in Bifrost) or does not apply. These four are worth taking for
|
||||
a site that runs its own models. Left until a customer asks: native
|
||||
Azure/Anthropic/Bedrock/Vertex adapters, request hooks a customer plugs in,
|
||||
routing by what a request holds. Not taken: a realtime (WebSocket audio)
|
||||
API, prompt management, an `npx` install.
|
||||
|
||||
### I1. Answers to similar questions, not only the same one (M)
|
||||
The answer cache (`internal/router/cache.go`, C17) answers a request whose
|
||||
body is the same byte for byte. A help desk asked "how do I reset my
|
||||
password" in forty wordings asks the model forty times.
|
||||
- Per project, off by default, beside the exact cache: `llmgw keys cache
|
||||
--project P --similar 0.95` (the cosine similarity a question must reach;
|
||||
higher is stricter). The exact cache is asked first.
|
||||
- The question embedded by a model the operator names
|
||||
(`router.cache.embedding_model`, an embedding model in the catalog), into a
|
||||
pgvector table per project with the answer, the model and the TTL; the
|
||||
store's halfvec and text index work from F applies. SQLite stores keep the
|
||||
exact cache only.
|
||||
- What counts as the question: the last user turn, with the system prompt
|
||||
and the earlier turns hashed and required to match exactly — a similar
|
||||
question in another conversation is not the same question. Tools,
|
||||
`response_format` and the sampling settings also exact.
|
||||
- Said: `x-llmgw-cache: similar` with the similarity, and the usage row
|
||||
says so; guardrails judge the request before a hit, as for the exact
|
||||
cache. A project with `no_store` gets neither.
|
||||
- Done when: forty rewordings of one question on the PoC cost one model call
|
||||
and 39 lookups, and an eval suite run with the cache on scores within a
|
||||
point of the run with it off.
|
||||
|
||||
### I2. MCP gateway: the customer's tool servers behind llmgw (L)
|
||||
`/mcp` serves llmgw's own tools. An agent inside a company also wants Jira,
|
||||
the file server, a database — each an MCP server, each today configured
|
||||
in every client with its own credentials and nobody seeing what was called.
|
||||
- Servers registered once, by an administrator: `llmgw ctl mcp add jira
|
||||
--url https://… --header-file /etc/llmgw/secrets/jira` (HTTP transport;
|
||||
stdio servers run on the control plane under their own user, like an
|
||||
engine). Their credentials stay on the control plane, never in a client.
|
||||
- `/mcp` lists every server's tools, prefixed by its name (`jira.search`),
|
||||
filtered by what the key may use: an allowlist per org, team, project or
|
||||
key, by server or by tool, in the tenancy tree as model allowlists are.
|
||||
- Each call goes to the audit log — who, which tool, the arguments' size,
|
||||
the outcome and how long — and through the guardrails both ways (a tool's
|
||||
answer is prompt input). A per-tool rate limit.
|
||||
- Later, if it earns it: Bifrost's "code mode" (the tools as one typed API
|
||||
the model writes code against), which cuts the tokens of a long tool list.
|
||||
- Done when: Claude Code given only llmgw's `/mcp` reaches a test MCP server
|
||||
through it, a key without the grant does not see the server's tools, and
|
||||
every call is in `ctl audit`.
|
||||
|
||||
### I3. What llmgw adds to a request, measured (S)
|
||||
Bifrost's headline is "11 µs at 5k RPS" (its own figure, and its own pages
|
||||
disagree with it). llmgw has none, so any question about its overhead is
|
||||
answered with a guess.
|
||||
- `llmgw ctl bench --overhead`: the same load through the gateway and straight
|
||||
at a `sim-engine` (`internal/cli/simengine.go`) that answers at once, at
|
||||
rising request rates, with auth, accounting and the guardrails' pattern
|
||||
checks on and off. Reports the added latency at p50/p99 and the gateway's
|
||||
CPU per thousand requests.
|
||||
- On the PoC the CPU is oversubscribed (pve1, 28 threads for ~125 vCPUs):
|
||||
report it as a ratio and record the host's load beside it, and repeat on
|
||||
an idle machine before a figure goes into the README.
|
||||
- Done when: the README states the overhead, with how it was measured, and
|
||||
CI runs the short version so a change that doubles it is seen.
|
||||
|
||||
### I4. Several keys for one outside provider (S)
|
||||
An external model has one `api_key_file` (`internal/control/control.go`,
|
||||
where external endpoints are made). A provider's limit is per key; one key
|
||||
answering 429 makes the whole fallback answer 429.
|
||||
- `api_key_files: [a, b, c]`, each with an optional `weight`: one endpoint
|
||||
per key (`external:<model>#<n>`), so the router's scoring, retries and
|
||||
breakers work per key as they already do per replica. A key answering 429
|
||||
or 401 is passed over while its breaker is open; a 401 also notifies
|
||||
(`external_key_refused`).
|
||||
- The usage row says which key answered (by its index, never the key).
|
||||
- Done when: with two keys and one returning 429, every request is answered
|
||||
by the other, and the console shows the one breaker open.
|
||||
|
||||
## D. Facts worth knowing before starting
|
||||
|
||||
|
||||
Reference in new issue
Block a user