mcp_servers holds each server: its address, headers, and secret
references for the headers that carry credentials. limits gains
allowed_tools, patterns such as jira.* resolved down the tenancy tree;
unlike models, a tool no level grants is not granted.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
G3, the part that needs no model: a ComfyUI model's comfyui.output (or
external.output) says pictures or meshes. A request for a 3D model — a
job at /v1/3d/generations — carries the picture as base64; at the replica
it is uploaded to ComfyUI's input directory (the gate lets /upload/image
through), the workflow is filled in with its name, the seed and the
resolution, and the mesh the workflow saves comes back with its type,
which is the file the job keeps. comfy.Client reads every list of files an
output names, not "images" only, and types a file by its extension where
ComfyUI sends octet-stream.
A model's license (name, url, notice) is said in /v1/models and in the
job, and written into a GLB's asset.extras, since the file travels without
the API; a mesh it cannot be written into is refused. per_mesh in
auth.pricing, usage.meshes in the rows, the sums and the export
(migration 0045). A picture asked of a mesh model, or a mesh of a picture
model, is refused saying which it makes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
auth.pricing takes per_image, charged on top of a prompt's tokens; a model
priced only per picture is not free, so a spent budget still stops it.
Each usage row keeps how many pictures were made (migration 0041), and
ctl request and the admin API's request view show them.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A search asking for more than 10 passages got at most 40 candidates from
HNSW, pgvector's default ef_search. The full-text arm scored every chunk
holding a word of the question before limiting, over the whole
organization's GIN index; it now takes candidates by ts_rank from the
collection's own partial index and scores those. New collections index
halfvec on pgvector 0.7 and later (up to 4000 dimensions).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Collections owned by an organization, team, project or person; grants that
add up down the tree and never cross an organization; chunks in PostgreSQL
with pgvector, partitioned by organization, an HNSW index per collection, and
row-level security as a second line behind each query's filter. Search fuses
nearest-neighbour and full-text ranks. Off on SQLite and without pgvector,
saying why. server init and the HA database setup install and enable it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
guard.enabled (unset is on) makes every policy check nothing while false,
live: llmgw ctl guard off/on set it in the cluster's settings, and ctl
status, ctl guard list and the guardrails page say when they are off.
A refusal by a judge names the hazard it saw - judge: S6,Specialized
Advice - and the verdict is kept with the request in a new usage column,
guard_rule, shown by ctl request, the request log and the guardrails page.
llmgw ctl guard judge ask has a judge read a text, past the cache, and shows
its raw answer and how it is read; the endpoint is kept out of the audit so
no text is recorded.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
llmgw keys cache --project P --ttl 1h [--any] | --off, and Cache answers… on
the console's project menu. The same request — project, endpoint, canonical
model and the body without stream, stream_options, user, metadata, store and
service_tier, keys sorted — is answered from the gateway's memory for the
TTL. Only reproducible answers (temperature 0 or a seed; embeddings, rerank,
score) unless --any; only a whole 200 from the model asked for, not a
fallback's, a canary's, or one a guardrail changed.
Looked up after the limits and the input guardrails; the output policy
judges a reused answer; no_store and stored conversations bypass it.
Streamed answers are kept put back together and streamed back as chunks.
x-llmgw-cache: hit|miss|bypass, Age on a hit, Cache-Control: no-cache or
x-llmgw-cache: no to bypass. A hit is recorded with no tokens and no cost,
marked cache_hit (migration 0027) and shown by ctl request. An LRU bounded
by router.cache.max_bytes and max_entry_bytes; llmgw_cache_requests_total,
llmgw_cache_bytes, llmgw_cache_entries.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Charts, totals and budgets summed the request table on every view. Usage is
now also summed in the writing transaction into five-minute, hourly and daily
tables per key, person, project and model, with latency and first-token
histograms. Charts read the coarsest tier their slices allow; totals and
budgets compose rows, slices, hours and days, and stay exact. Request records
can be given a retention without moving any total.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
"Why was this request slow?" had no answer. Nothing tied a request to the
replica that served it, the time admission held it went out in a response
header and was never kept, and the only per-request log lines were for
disconnects, early stream ends and upstream retries.
Every request now has an id: the client's X-Request-Id when it is a plain
string of reasonable length, otherwise one the gateway makes. It is set on
the response before anything else, so a request refused as malformed can be
named too, and forwarded to the engine. One access log line per request
carries it with the model asked for and served, the replica, status, time in
the queue, time to the first token, total time, tokens, and the key,
project and user.
Usage records gain the id, the replica, queue time and time to first token,
tried against a restored copy of the live store before being shipped: 591
rows kept, columns and index in place. `llmgw ctl request ID` reads one back
through the admin API.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
There were no backups: docs/install.md gave a pg_dump command, and nothing
ran it. The store holds every key, person, quota and usage record, the models
added at runtime, and now the secret each engine's key is derived from; the
certificate authority's key sits in /etc/llmgw/pki beside it. Losing the disk
lost all of it.
`llmgw server backup` writes both to one archive: pg_dump's custom format, or
a VACUUM INTO snapshot for SQLite, plus /etc/llmgw. The archive holds the CA's
key, so it is 0600; the directory is listable, so the control plane — not
root — can see when the newest was written. pg_dump is given the connection
through its environment, since a DSN carries the password and a process's
arguments are readable by every user. Written beside its name and renamed,
so an interrupted backup is never the newest; the newest 14 are kept, and
nothing the command did not write is ever removed.
llmgw-backup.timer runs it nightly and catches up after downtime. The control
plane publishes the newest archive's time, and BackupTooOld fires past 26
hours — or when there has never been one. The install guide now covers
restoring, and says to copy archives off the machine, since they share its
disk.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The cluster page's usage table showed session:usr-... and proj-... ids;
it now shows the person's email, whether it came through a key or the
chat page, and the project's path or "personal". Times read "3 hours
ago" rather than "3h7m25s ago", everywhere.
The usage page's legend gave each line's latest slice, which was 0
beside a table of hundreds; it gives the window's total, and the table's
numbers are right-aligned like everywhere else.
The PostgreSQL placeholder rewriting skips line comments. An apostrophe
in one opened a string that swallowed every placeholder after it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A person was in exactly one project, chosen at approval, and could not
use the gateway before that. People belong to several teams and to
none, so:
- memberships: a person is a member of any number of organizations,
teams and projects, or of none; a member of a team or organization
may use every project in it. users join/leave, and the people page
adds and takes out from either side. Approval no longer needs a
project.
- personal keys: a key may have an owner and no project. /profile asks
what a new key spends against; keys create --for does the same.
auth.personal sets default limits for personal use or turns it off.
- limits for people: --scope user holds a person across every key,
project and the chat page; --per-member on an org, team or project
gives each member their own share there. Resolution folds the tree,
each per-member limit for the key's owner, and the owner's own, each
in its own pool; the quotas page shows and sets both.
- the chat page asks what to spend against and checks it per message.
- usage records who spent it, so a person's budget survives a restart,
personal use and chat included; restore no longer drops rows with no
project.
Leaving a project revokes the keys made there unless another membership
still reaches it. A personal key keeps no threads or notes, which are
per project, and is refused rather than dropped into the anonymous pool.
Migration 0010 turns each person's project into a membership, adds
usage.user_id and backfills it, and makes api_keys.project_id nullable.
SQLite cannot drop NOT NULL, so the table is rebuilt; that needs foreign
keys off, which only works outside a transaction, so migrations marked
for it run on one connection with foreign_key_check before commit.
Tested on a populated SQLite store with claims, and on a copy of the
cluster's PostgreSQL store: 5 people became members, 17 keys and 3
claims intact, 145 of 577 usage rows and all 69 chat rows attributed.
The PostgreSQL store tests, including new membership cases, pass on 16.
users move is gone; join and leave replace it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Nothing in the tree could be renamed, moved or removed, and an approved
person stayed in the project they were first put in — from the web or
the CLI. Now:
- /admin/users is the tree, each project opening onto its people and
keys, with approve, move, suspend, reinstate and remove for people,
revoke for keys, and add, rename, move and remove at every level,
plus a search over every account. Forms post and get the page back,
so it works with no script; htmx 2.0.11 (0BSD, embedded, checked
against npm's integrity hash) swaps only the changed part, keeping
open projects open.
- llmgw org tree/add/rename/move/remove, users move/delete, and
keys list --all.
- Projects, teams and organizations are named by id, path
(Acme/platform/gateway) or unique name; users by id or email. With a
single project --project can be left out, instead of defaulting to
"default", which a tree built from the web does not have.
Moving a person moves the keys they made; operator keys stay with the
project. Only empty things are removed, and never a project that had a
key, since usage history refers to it.
Also: keys init no longer reparents an existing team or project;
reinstating an account with no project is refused (a rejected signup
used to come back active and fail at its first key); rejecting a
signup removes it; keys limits --budget takes whole units like the
quotas page.
Checked in headless Chrome: a move from an open project updates the
page in place, keeps the project open, and the search still filters.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Resolution took the most restrictive value along the chain and enforced only
that one. For a single caller's ceiling that is right: a team cannot raise what
its organization set. For a pool it is not, because two levels are two pools.
An organization given 1000 with four teams given 500 each spent 2000. Each team
filled its own counter, the organization's value lost to the tighter one and
was never looked at again. The number was written, displayed, reported against
— and enforced by nothing.
Every level that sets a limit is now checked against its own counter, and
settled onto all of them, so an organization's pool fills from its teams'
spending rather than only from requests that somehow named it. A team still
stops before its organization when its own pool runs out first, which is what a
sub-allocation is for.
Effective carries the chain as well as the collapsed values. The collapsed ones
are still what a header or a profile page should show — they are the ceiling a
single caller faces — they are just not enough to enforce with.
Rate limits charge as they check, deliberately, so that two concurrent requests
cannot both see room. With several levels that means a refusal at one has
already taken from the levels before it, so those are handed back. Without the
refund a team sitting at its organization's ceiling would quietly eat its own
rate limit with requests that were never served.
Budgets are checked before any of that, and only read, so a refusal on a budget
charges nothing at all.
Status reports the fullest pool rather than the smallest number. A team inside
an organization with the same budget is bounded by whichever is nearer its
ceiling, and that is the organization as soon as a second team spends anything;
a header showing the team's would disagree with the refusal after it.
A cluster default joins the chain as a key-level entry. Left out, it would be
reported as the ceiling and then never checked.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two things that were only ever confusing, and one of them stopped a command
working on the cluster.
The store type was named SQLite and kept the name when PostgreSQL was added, so
OpenPostgres returned a *SQLite and all seventy-odd methods on the production
store were spelled after the database it was not running on. Nothing broke;
people did. Earlier today I added three methods to *SQLite for a table that on
this cluster only ever lives in PostgreSQL, and the name did not once look
wrong. It is a field, which is all it ever was.
`llmgw users` and `llmgw keys` defaulted --store to a SQLite path. On a
deployment whose store is PostgreSQL every one of them failed with "unable to
open database file" while the right answer sat in auth.store in the config file
beside them; the conclusion an operator draws is that the tool is broken, and
they are not far off. They now ask the server's own configuration before
falling back, and the error names which of the three sources chose the address
— the failure was never the path, it was which one picked it.
The single-machine default still applies last, so a deployment with no config
file keeps working with no flags.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The tenancy tree handed limits down and never pooled what was spent against
them. A limit written on a team was inherited by every key beneath it, and then
enforced with the key as the counter — so a project given a budget of 1000 and
ten keys permitted ten thousand, and a team given 60 RPM got 60 per member.
That is "each of you may spend this much", which is a useful thing to be able
to say. It is not what somebody writing a budget on a team means, and nothing
in the system said which one you were getting.
Resolution now records which scope set each value that won, and the counter is
keyed by that scope. The value and the counter come from the same place, so a
project tightening its team's limit is enforced against the project's own pool
rather than eating into a quota another project never agreed to.
Each limit is owned separately, because they resolve separately: a team budget
with a per-key rate limit — the group's money, the individual's throughput — is
the ordinary arrangement and has to stay expressible.
Cluster defaults are owned by the key. Read as a shared pool, one default would
become a single cluster-wide rate limit that every key in every organization
competes for: not a default, an outage.
Restoring after a restart had to change with it. Per-key totals rebuilt a
counter named after the key while the check read one named after the team, so
the control plane came back believing the team had spent nothing — a budget
that reset on every restart, for exactly the scopes where it matters most.
SpendSinceByScope returns each key's place in the tree and the accountant folds
it into every scope that might own a budget.
Counters are namespaced by level throughout. A bare key id in one path and a
"key:"-prefixed one in another are two counters for the same key, which is how
the restore was silently doing nothing.
Still missing, and worth naming: no usage report rolls up by project, team or
org, so a shared quota can now be enforced but not yet shown.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Groundwork for PostgreSQL, with no PostgreSQL in it yet: the store still opens
SQLite and behaves exactly as it did.
One set of queries rather than one per database. The SQL here is almost
entirely portable — ON CONFLICT upserts work in both, and the only construct
that genuinely differs is the placeholder — so two copies of every query would
buy nothing and drift, one query at a time, discovered in production.
The rewrite is a wrapper around the handle rather than a change at each call
site. There are sixty of those and sixty chances to forget one; this way the
only route to the driver goes through the rewrite. Making it a wrapper found
six places that reached past it, including all three manual transactions and
two prepared statements — every one of which would have worked on SQLite and
failed on PostgreSQL, on a path nobody exercises before deploying.
The rewriter skips quoted sections. No query here contains a question mark in
a literal today; this is so that adding one is a test failure rather than a
statement that silently binds the wrong argument.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The dashboard could say what the cluster was made of and not who was
spending it. Usage was per key, which is the right unit for revoking
something and the wrong one for the question actually being asked: one
person holds a laptop key, a CI key and a notebook key, and adding three
rows up by eye is a chore that gets skipped.
Per person now, joined through the keys they own, and per model — counted
by the model that answered rather than the one requested, since those differ
whenever a fallback serves and what actually ran is the question. Headline
figures are stat tiles with a sparkline: a single current value is a number,
and plotting one is how a chart ends up with a single bar in it. The
sparkline says only rising, falling or flat, which is the one thing the
number cannot.
Ranked horizontal bars, because the labels are email addresses and model
names and the label is the point — the question is which person, not what
shape the distribution has. Rotated or truncated axis labels say the chart
matters more than reading it. Trimmed where drawn and complete on hover, so
the plot keeps its geometry and a reader can still find out whose row it is.
The palette is validated rather than chosen: every adjacent pair clears the
colour-vision separation threshold, and the dark values are the same hues
re-stepped for the dark surface instead of an automatic flip. Run through a
validator, because that is the only way anyone can honestly claim a palette
is colourblind-safe.
Which turned up a bug in the existing chart code. cssVar is an allowlist of
four names that silently returns --accent for anything else, and callers
wrap it in var() themselves — so every new series would have rendered in one
colour with accent-coloured text. Caught by a test written from the rule
that chart text wears text tokens, never the series colour.
A key with no owner is labelled "operator-issued keys" rather than unknown:
it predates accounts and is doing its job, and "unknown" would imply
something is wrong.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Unlimited was the default. Creating a project without limits is the easy
path and the common one — the "default" project on this cluster is one, and
the person using it is metered by nothing at all. The first sign of that is
the bill or the outage.
auth.default_rpm and default_tpm now apply to a key no scope gives a limit
to. Zero keeps the old behaviour, which is still a reasonable choice; the
point is that it becomes a choice rather than an omission.
Applied after the chain rather than folded in as its widest scope, because
a default is a fallback and not a ceiling: a project deliberately given 600
rpm must keep it, and tightening would silently cap it at the default. An
explicit limit wins in both directions.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two things the dashboard was missing: it could not act, and it was a column of
numbers.
Account administration moves to /admin/users — approving into a project,
suspending, reinstating. That is the work an operator does constantly and it
should not need a terminal. Three things stay on the CLI, and the line is not
arbitrary: promoting someone to admin, resetting a password, and issuing a key
on another person's behalf each turn a stolen admin session into more than it
was — permanent access, or somebody else's account. Approving and suspending
do not. Suspending signs the account out everywhere, or suspension is advisory
until the session happens to expire, and an admin cannot suspend themselves,
which is not a thing an interface should let you do by misclicking.
The charts are inline SVG generated in Go. A charting library would mean a
JavaScript toolchain in CI and a megabyte of bundle for what is a list of
rectangles, and the single-binary goal rules that out. Hourly over a day: fine
enough to show a working day, coarse enough that one slow minute is not an
incident. Empty hours are filled in, because a chart with hours missing where
nothing happened reads as a chart with hours missing from the data.
The SVG is marked as trusted HTML, so everything that reaches it is either a
number this package computed or escaped explicitly, and a bar's colour can only
be one of four names — a colour that could be anything is a way to write CSS
into every page the chart appears on. There are tests for both.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
SQLite creates its file with the process umask, which on a stock system means
every account on the machine can read it. What is in there is not a working
credential — keys are stored hashed — but it is the whole tenancy tree and
every request anyone made, and that should not depend on an operator
remembering to chmod it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Pasting the gateway's address into a browser returned 404, which looks like a
broken endpoint rather than one that tells you how to use it.
Two pages, and the split between them is a security boundary rather than a
layout choice. The landing page is served without a key — it has to be,
because the person reading it is the one who does not have one yet — so it
contains no data at all: it says how to connect, not what is here. Model names
stay behind /v1/models. It reconstructs the address from the forwarded headers
so the copy-pasteable example names the address the client actually used
rather than the internal one; those headers are reflected into displayed text
and never used for a decision.
The dashboard shows nodes, models, replicas, placement failures and usage, and
lives on the admin listener only. That listener is loopback-bound and also
serves pprof, which can dump the process heap — where API keys sit in
plaintext — so reaching the page from elsewhere is an SSH tunnel's job. It is
read-only: anything that mints a credential stays in `llmgw keys`, where it
needs a shell rather than a browser session this project has no way to
authenticate.
Mounting the landing page uncovered a hole in the exemption list. An entry
ending in "/" was a prefix rule, and "/" ends in "/", so exempting the landing
page would have exempted every path there is — authentication still installed,
still logging, and letting the whole API through. "/" is now exact, with a
test enumerating what must and must not be exempt.
Server-rendered templates and one stylesheet, embedded. The single-binary goal
rules out a JavaScript toolchain and nothing on either page needs one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Phase 5 begins. This is the persistence layer for the things the cluster
cannot reconstruct — who may call, what they are allowed, what they used.
Everything else llmgw knows is still derived from the fleet, which cannot be
wrong about itself the way a database can (§6.5).
SQLite behind an interface. The plan justified that by write volume, which was
the wrong argument: the write path is batched anyway, and a 500-row
transaction every 200ms is nothing for either engine. The real argument is the
failure mode. Postgres is a second service, and a gateway that stops serving
because its accounting database is unreachable is worse than one that buffers
some accounting. Embedded means the store shares the gateway's fate instead of
adding an independent way to be down. That reasoning stops holding at two
instances, where budgets have to be counted somewhere shared — so the
interface is here from the start, the SQL is portable, and migrations are per
dialect with shared version numbers.
Keys are stored as sha256 and nothing else, so a database leak hands over no
working credential. The fast hash is deliberate: the input is 256 random bits
with no dictionary to attack, so argon2 would buy nothing and charge tens of
milliseconds per request. That assumption is written down next to the code
that depends on it.
Two things the schema now says that it did not: budgets are integer
micro-units, because repeated addition of floats drifts; and an empty model
allowlist forbids everything while an absent one permits everything, which the
resolver has to keep distinct.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
