llmgw keys cache --similar 0.95 also gives a chat question the answer to
one like it: the last user message embedded by router.cache.embedding_model
(asked on the gateway's account, as a judge is, within 2 s), compared by
cosine with the questions answered in the same conversation — system
prompt, earlier turns, tools and sampling the same. The exact cache is
asked first; an exact hit embeds nothing. Said as x-llmgw-cache: similar
with x-llmgw-cache-similarity, charged as a hit.
The vectors live with their answers in the gateway's memory, counted in
max_bytes and evicted with them, not in a pgvector table as BACKLOG I1
planned: the cache keeps answer text out of the store on purpose. A
question over 2,000 characters is reused only when the same. Migration
0051 holds the setting. BACKLOG I1.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cached_per_1k in a price charges the prompt tokens the answer counts as
cached_tokens at their own rate; zero keeps charging them as prompt tokens.
The count reaches the cost and not a column: CostMicros holds what it
changed. Cached beyond the prompt is clamped, an engine's miscount rather
than a refund. BACKLOG L5.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The router keeps each endpoint's time to first token, smoothed over its last
streamed answers, and the chat profile scores by it beside session, queue,
KV and inflight: the fastest endpoint 1, one twice as slow 0.5, one not yet
measured 1 so a new replica is tried. Load still sees a slow drain from the
engine's own counters; this sees what answers actually took. BACKLOG L4.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docs/kong.md: a plain route or ai-proxy with llmgw as the upstream, the
client's llmgw key passed on so per-team budgets hold, Kong as a provider
behind llmgw, and Kong 3.9's streamed answers counted without completion
tokens. Closes BACKLOG L9.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A site may be required to use a particular guardrail: Azure Content Safety,
Lakera, NeMo, a classifier of its own. A policy's services asks one over
HTTP — a body template given the text, header values from files read each
call, dotted paths to where the answer says unsafe and why — on input,
output or both, to block or flag. On input it reads the latest user turn
before the judge; on output the whole answer once at its end. A service
that does not answer is what the policy's fail says. It is told the text
only, never who sent it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With restore: true on a guardrail policy's input, what redact rewrote as
placeholders comes back in the answer: the model reads "[EMAIL_1]", the
client reads the address it sent. A streamed placeholder split across
events is held until it is whole, and what is held is sent at the end.
The swap runs before the output filter, so the output rules judge the
answer as the client reads it and a placeholder they make of a value the
model invented is never taken for the request's own. The map lives in the
Verdict for the request only; nothing stores, logs or traces it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A control plane that is also a node, serving the starter model and two
judges, needs about 10.5 GB: 3.5 for itself, 2 for the starter, 2 for each
judge, and one more because the scheduler counts whole gigabytes. On 8 GB
the install did every step and then had nowhere to run the starter. The
setup's checks now fail this machine's memory and name what it would serve,
and server init refuses before it installs anything, naming the flags that
leave those models off it. quick-install.md said 8 GB; it says 12.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An inference request and each attempt at an engine carry gen_ai.*: the
operation, the requested and answering model, max_tokens, temperature and
top_p when the client set them, the session as the conversation id, the
input, output and cached tokens, and on an attempt the engine that answered.
A customer's Grafana, Jaeger or Langfuse then shows llmgw's calls as model
calls. No prompt or answer text is ever put in a span, and a test says so.
BACKLOG gains section L: what is worth taking from Kong's AI Gateway, of
which this is the first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
llmgw's own extension for VS Code, plain JavaScript with no dependencies,
packed into a .vsix by the binary at /setup/vscode/llmgw.vsix with the
serving gateway's address in it, so it needs no build, no Node and no
marketplace. It registers llmgw as a language-model provider: the chat's
model picker, agent mode and any extension that asks VS Code for a model
get every chat model the key may use, sized and with tool and vision
capabilities as the catalog says, streamed, tool calls both ways.
VS Code is a tool on the account page's Connect your editor: a key, the
install command, and an Open in VS Code link that hands the key to the
extension's secret storage. The extension's Sign In opens the page with a
state of its own; a link without it is confirmed, the gateway named, since
it decides where code is sent.
Tested in VS Code 1.138 against the PoC: 27 models listed, an answer from
qwen2.5-0.5b-instruct, and qwen3-coder-30b calling a tool and using its
result (internal/vscodeext/e2e).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
/install takes a reader from three bare Ubuntu machines to a first answer:
what each machine needs and the commands that check it, the release fetched
and its checksum checked, the eight setup screens, the system's state and
the console, and a first request with a key from the account page. Every
screen is from an install of v0.14.9 on three fresh VMs, recorded with tmux
and kept as terminal frames: the page shows them as text, and plays a
session back when asked, fetching it compressed from /install/cast/. The
administrator's password and the API key were masked before the recording
was kept, and a test fails if either shape reaches the page.
/about gains what llmgw now does beyond routing, and its install steps are
today's: the setup screens and node add, not onboard-node.sh and a model
file, with a link into the guide.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The errors other packages give a page (store, catalog, mirror) are
shown in its language; a percentage is written as the language writes
it (%93 in Turkish); the footer's time in its date form; and a stale
form is refused with a sentence rather than "bad request".
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A tool's answer is read by the caller's policies as the prompt it
becomes, so an instruction planted in a ticket is withheld and personal
data redacted; its arguments are read before they leave. ctl mcp add
--rate bounds each caller's calls a minute to a server (migration 0050),
and five failures in a row take a server out for a minute, its tools
off the list.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The same text again was answered from what the engine kept of the last
reading, in a tenth of the time: the check read 4.9 s, then 0.5 and 0.5.
Each judgment now begins with a line of its own.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ctl guard judge check, and POST /api/v1/guard/judges/check, time three
judgments of a 280-token prompt, then a burst at once, against the
policies' wait: ok, tight or slow, about how many requests a second can
be judged, and what to do. server init runs it once the judge is up and
says a machine too slow before the next steps; the wait is not changed
without asking. BACKLOG reordered by what goes wrong where installed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Setting one limit at a scope wrote the whole row, so a tool grant given
alone cleared the scope's rate, budget and models. --replace writes it
whole, as the console's form, which shows every field, now asks for.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Its tools say what llmgw ctl does: cluster_status, list_nodes,
list_models, list_replicas, explain_placement, node_logs; none changes
anything. The admin API serves it at POST /api/v1/mcp, and the gateway
answers a server registered as llmgw:admin in its own process, so no
token is kept and each control plane serves it. ctl mcp add takes
--bearer-file for a token sent as Authorization: Bearer.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
llmgw ctl mcp add|list|remove|test and /api/v1/mcp/servers keep the
servers; a token given with --secret-file is kept in the store, sealed,
and named by reference. keys limits --tools, and the console's limits
form, grant tools as patterns. Documented in install.md and clients.md.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Each registered server's tools are listed as server.tool beside llmgw's
own, to a key whose tool grants match, and called with the server's
credentials, read from their secret references at each call; the
server is told the key, person and project calling. A call is bounded
in time and size, its progress passed on to a streaming client, and
recorded in the audit log with its arguments' size and digest, never
their text. A key granted nothing sees none of them.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
internal/mcpclient speaks streamable HTTP: initializes once and keeps
the session the server gives, beginning it again once if the server
forgot it; lists tools page by page; calls one with headers of its own,
taking the answer as JSON or as events with progress before it; and
bounds what it reads. mcptest is a server for tests.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
mcp_servers holds each server: its address, headers, and secret
references for the headers that carry credentials. limits gains
allowed_tools, patterns such as jira.* resolved down the tenancy tree;
unlike models, a tool no level grants is not granted.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The startup timeout runs from when a node's download last moved, so a
model of hundreds of gigabytes on a slow mirror is not failed while it
arrives. ctl model serve --env sets engine variables (NCCL_DEBUG=INFO)
on every node, and a layout given by hand keeps expert parallelism for a
model of experts. docs/large-models.md: how a model is split, what each
way needs of a card, and a checklist for bringing one up on a fabric.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A mixture-of-experts model on more than one card is served with vLLM's
--enable-expert-parallel; config.json's experts are read by any of their
names, and a model that also reads pictures is sized by its text_config.
Tensor parallelism past the KV heads is allowed when it is a multiple of
them (Qwen3-235B, four KV heads, runs on eight cards), which the plan
had refused. Several nodes are offered in any number, not only powers of
two. Each way says what it needs of a card, part by part, and of all its
cards; the serve form checks a context longer than planned still fits.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Found on a fresh HA install with a judge machine: the leader saw the
policy's judge missing while init was still fetching it, served one
itself, and its entry replaced init's, so a second judge ran off the
judge machine. The keeper now waits ten minutes after it starts, and a
serve job it starts gives way to a model added meanwhile (AddNew).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The judges' machine was selected by the judge's profile, but any other
model could still be placed beside it. A node labelled llmgw.role now
takes only profiles whose selector asks for that role.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The setup screen asks for up to two judge machines after the store's
two, and server init takes --judge-node for each. Each is added as a
node labelled llmgw.role: judge, and the judge's profile selects that
label, with cores by init's rule over those machines. A judge beside a
busy control plane refused 38 of 60 requests under load; alone, 2.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A judge a policy names and the catalog lacked was served with 3 GB, no
cores and llama.cpp's prompt cache; on a busy node its judgments missed
their budget and fail: closed refused the requests. It now gets init's
2 GB and cores, and every judge modelserve serves runs with
--cache-ram 0.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
sudo llmgw on a node showed only agent.yaml, the manual, wipe and leave.
It now opens on an Overview read on the node itself: the agent's state and
version, the control planes it dials and whether they answer, each GPU with
the models on it, the engines from the supervisor's state files, the free
space for weights and engines, and the agent's warnings of the last hour.
No admin token is needed, every command is bounded by two seconds, and a
missing tool is said in a line.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
People → Sign-in shows single sign-on, Active Directory and SCIM, and
the directory's last sync. E sets up Entra ID from the tenant and client
IDs; D sets up Active Directory, tried from the control plane first:
each server bound, people who may sign in, members of each mapped group.
The secrets go into the store as store:auth/..., sealed by the master
key, and the CA into the settings, so no control plane needs a file.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Found trying auth.ldap against a Samba DC on the PoC. A CA file a
control plane could not use made every control plane exit, and no
ctl config could then put the setting right: directory sign-in is now
turned off there, and said. `llmgw server check` ran a whole server
that, its ports taken, held the leader's lease for three hours: server
refuses words it does not know and takes its ports before anything else.
The guides now say auth is set with ctl config, not server.yaml.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
auth.ldap checks a username and password against the domain controllers
over LDAPS or StartTLS: found with a service account, checked by binding
as the person. Groups, nested ones included, give staff roles and places
in the tree. Accounts are tied to the objectGUID; the leader reads the
directory every 15 minutes, suspends whom it disabled or removed and
reinstates them after, and refuses a sync that would suspend most.
Entra ID sends no groups for someone in more than 200, only where to
fetch them; that was read as none and took an administrator's role away.
It is now left as it was, and logged. An account made by SCIM or OIDC
can be reinstated again.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
An external model takes api_key_files, each a file and an optional weight,
in place of api_key_file. Each key is an endpoint of its own
(external:MODEL#1, #2), so the router's scoring, retries and breakers work
per key. A key answering 429 is passed over while its breaker is open. A
key answering 401 is passed over too, and external_key_refused is sent,
naming the endpoint and never the key; with a single key the 401 is still
the client's. A key whose file cannot be read is left out, and usage
records which key answered by its endpoint.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
An engine that answers at once and the gateway in front of it, in one
process on loopback, given the same load straight and through the gateway:
bare, with authentication and accounting, and with the guardrails' pattern
checks. It reports p50 and p99, what the gateway added, and a rough CPU
per thousand requests.
On an Apple M5 at 100 to 2,000 requests a second (10 s a step, 72,000
requests, none failed), the gateway added 0.05–0.42 ms at p50 and at most
0.49 ms at p99. That is now in the README, with how it was measured and
why the CPU figure is rough. CI runs a one-second version.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
CI run 394, on a commit that changed only BACKLOG.md, failed with a data
race in TestAVisionModelIsServedWithItsProjectorBesideIt. The serve job's
goroutine wrote j.Request without the lock (since efffdb6, which sized
embedding slots), while Get's snapshot read the job under it. Every other
write to a running job holds the lock. It did not reproduce in 30 runs
here; the race detector's trace names this write and that read.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A v0.14.6 install on fewer than eight cores refused every request until
its judge was given cores by hand, and upgrading did not change that. The
judge keeper, on the leader, now finds a judge whose entry is exactly what
init before v0.14.7 wrote: one llama.cpp CPU profile, an 8192 context and
nothing else, 2 or 4 GB, none to two cores. It gives the judge 2 GB,
--cache-ram 0 and cores by init's rule through modelsvc.SetProfiles, then
restarts its ready replicas. The replica count is left as it is. A judge
whose entry differs is left alone, and the log says so once.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On a 4-core single-machine install, `ctl model resources` was run on the
judge (to two cores) and on the starter (to none), then both restarted.
The judge's restart was refused: its replacement fit on no node even with
its old replica stopped, because the starter still held its two cores. A
restart now waits for other restarts under way (queued, retiring or
draining) when their going would make room, up to DrainGrace.
A drained replica was also kept until its model had enough live replicas,
and a replica being restarted that still answers did not count. So with
replicas restarted one after another, each drained one waited out the
five-minute grace, holding its room.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A second judge replica on the same machine shares its cores and is lost
with it, so it bought nothing but half the speed of each. On a 4-core,
8 GB single-machine install, one judge on four cores passed 20 of 20
requests at about 3.5 s each, judged in and out. Two judges on two cores
each had passed 19 of 20.
server init now serves one judge when this machine is the only node, and
two, on two cores each and spread over the nodes, where there will be
others (HA setups, or a control plane that is not a node).
--judge-replicas and the setup's Copies still choose. install.md gives
the measured times in place of "half a second".
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Checked by a single-machine install of v0.14.6 on a 4-core, 8 GB VM. The
second judge fitted nowhere: v0.14.6 reserved 4 GB a judge, a figure from
one replica that grew to 3.2 GB through llama.cpp's prompt cache (up to
8 GB by default), which a judge never reuses. Once it fitted, every
judgment timed out. v0.14.6 gave a judge one core below 8 CPUs, and a
judgment of about 280 tokens takes 8.0 s on one core, 4.2 s on two and
2.3 s on four, against a 5 s budget: 0 of 20 requests went through.
A judge now runs with --cache-ram 0, uses about 1.6 GB and reserves 2,
and reserves two cores (one only below four CPUs). On the same VM, 19 of
20 went through.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every test run since c22528c ended at the job's 30-minute deadline with
nothing after internal/release, while the same tests pass with -race on
Linux, as a user and as root, in 5 minutes. A 12-minute test timeout makes
the hang name itself.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
From a comparison with Bifrost: answers to similar questions, the
customer's MCP servers behind llmgw, the gateway's overhead measured, and
several keys for one outside provider. H9's /mcp item is done.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
