Resolve now checks the owner's state for a personal key; the authenticator
treats an inactive owner as a revoked key. Suspension stops the keys within
the cache TTL, reinstatement brings them back.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
server add gives another machine this llmgw, a certificate, the
configuration and store address, lets it into the store through Patroni, and
starts it; --vip keeps a floating address on the leader with keepalived,
following a new /leader endpoint. Nodes learn every control plane from the
leader and notice a dead one by HTTP/2 ping instead of after the kernel's
fifteen minutes. The setup, console, node and init screens are drawn full
screen in one frame: title bar, sidebar, bordered panel, key bar, dialogs.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
init --patroni user@host ... installs PostgreSQL, Patroni and etcd on the
given machines over ssh, with this one as the third etcd vote, makes the role
and database, and moves the store in use into the cluster. Each step checks
first, so it changes nothing on a cluster that is up. For a store of several
hosts the control plane publishes each member's role, state and lag from
Patroni, a stopped member as missing, prints it in ctl status, and alerts on
it. Making an administrator on PostgreSQL sent a bool to an integer column;
it sends an integer now.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Percentiles are read from the histograms kept with the usage sums and shown
per group on the usage tab and in the people page's panels. Requests refused
by a limit are recorded as 429s against their owner, free of charge, so a team
hitting its limits shows as such. Usage can be grouped by model.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Charts, totals and budgets summed the request table on every view. Usage is
now also summed in the writing transaction into five-minute, hourly and daily
tables per key, person, project and model, with latency and first-token
histograms. Charts read the coarsest tier their slices allow; totals and
budgets compose rows, slices, hours and days, and stay exact. Request records
can be given a retention without moving any total.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A node that cannot run the CUDA 13 build installs the same version's
CUDA 12.9 build from vLLM's GitHub release, with PyTorch's cu129 torch;
the install computes on the GPU before it counts. The build can be
chosen per node — auto, cu13, cu129, none — on the cluster page, with
ctl engine-build, or over the admin API; the control plane keeps the
choice and tells the node at once.
Refs #2.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A taken address on PostgreSQL fell through to a generic error, because
the store looked for SQLite's "UNIQUE". It checks code 23505 too. The
done page tells someone whose address had an account to sign in or ask
for a reset, without saying which case it was; other failures give a
logged reference.
Refs #2 (item 6).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A model has an owner (org, team or project). Quotas guarantee a scope
GPUs, cores and memory and may limit it; what a scope holds is summed from
its models' replicas and those beneath it. A replica within its owner's
guarantee is not taken for another scope; what is held beyond a guarantee
is borrowed and taken back for a scope within its own. A limit stops a
model before it starts, and headroom stops there too. ctl quota and ctl
model owner set them; metrics and a CapacityAtLimit alert report them.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
"Why was this request slow?" had no answer. Nothing tied a request to the
replica that served it, the time admission held it went out in a response
header and was never kept, and the only per-request log lines were for
disconnects, early stream ends and upstream retries.
Every request now has an id: the client's X-Request-Id when it is a plain
string of reasonable length, otherwise one the gateway makes. It is set on
the response before anything else, so a request refused as malformed can be
named too, and forwarded to the engine. One access log line per request
carries it with the model asked for and served, the replica, status, time in
the queue, time to the first token, total time, tokens, and the key,
project and user.
Usage records gain the id, the replica, queue time and time to first token,
tried against a restored copy of the live store before being shipped: 591
rows kept, columns and index in place. `llmgw ctl request ID` reads one back
through the admin API.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
There were no backups: docs/install.md gave a pg_dump command, and nothing
ran it. The store holds every key, person, quota and usage record, the models
added at runtime, and now the secret each engine's key is derived from; the
certificate authority's key sits in /etc/llmgw/pki beside it. Losing the disk
lost all of it.
`llmgw server backup` writes both to one archive: pg_dump's custom format, or
a VACUUM INTO snapshot for SQLite, plus /etc/llmgw. The archive holds the CA's
key, so it is 0600; the directory is listable, so the control plane — not
root — can see when the newest was written. pg_dump is given the connection
through its environment, since a DSN carries the password and a process's
arguments are readable by every user. Written beside its name and renamed,
so an interrupted backup is never the newest; the newest 14 are kept, and
nothing the command did not write is ever removed.
llmgw-backup.timer runs it nightly and catches up after downtime. The control
plane publishes the newest archive's time, and BackupTooOld fires past 26
hours — or when there has never been one. The install guide now covers
restoring, and says to copy archives off the machine, since they share its
disk.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Engines listen on every address their node has, with no key. Checked on the
live cluster: from a neighbouring node, a request straight to an engine's
port was answered — past the gateway's authentication, quotas and
accounting, by anyone who could reach the node.
Each replica's engine is now started with a key and refuses requests without
it. The key is derived from the replica id with an HMAC over a secret kept
in the store, rather than stored per replica: the router has to send the
right key to replicas this process never started — adopted after a restart,
or placed by the other control plane — and a function of a shared secret
gives that without a table to keep in step. A key taken from one node opens
that replica only. The secret is made on first use and copied with the rest
of the store when moving to PostgreSQL; a new one would lock the router out
of every running engine at the moment of the move.
The key reaches llama.cpp and vLLM through their environment
(LLAMA_API_KEY, VLLM_API_KEY) rather than their command line, which every
user on the node can read; SGLang only takes it as an argument. The agent's
readiness check, the router's proxy and its metrics scrape send it, the
client's own Authorization still never reaches the engine, and `ctl bench`
fetches the key of the replica it is pointed at from the admin API.
Engines started before this keep running without a key until their replica
is next started. With no store there is nowhere to keep the secret, and
engines start without one, with a warning.
Also formats two files that went in unformatted.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The account page (was "Your keys") shows the last thirty days of the
person's own use - requests, tokens, spend and tokens per day - and, for
each place they can spend, what they used there and a bar for every
budget that applies: their own, a per-member one, or a project's, team's
or organization's shared with others, named as such. Bars turn amber past
four-fifths and red when spent. It sits under the keys now, above the
password form.
Spend is read from the usage table for the pool each limit counts
against, from the start of its period.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The cluster page's usage table showed session:usr-... and proj-... ids;
it now shows the person's email, whether it came through a key or the
chat page, and the project's path or "personal". Times read "3 hours
ago" rather than "3h7m25s ago", everywhere.
The usage page's legend gave each line's latest slice, which was 0
beside a table of hundreds; it gives the window's total, and the table's
numbers are right-aligned like everywhere else.
The PostgreSQL placeholder rewriting skips line comments. An apostrophe
in one opened a string that swallowed every placeholder after it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A member of an organization can be its manager (users join --org X
--role manager, or the star on the people page). /org shows a manager
their organization: usage over time by team, project or person, its
teams and projects with limits and 30-day spend, and its members, each
once with every place they belong. Read-only; administrators see every
organization. Only what was spent in the organization's projects is
shown — UsageBucketsIn filters by where a request was spent.
Found on the way:
- Usage by org, team and project, and the quotas page's spend, joined
through api_keys and so dropped every chat-page request, which has no
key row. They now read usage.project_id; keys no longer move between
projects, so the two agree.
- /org was not exempt from API-key auth and answered browsers with a
JSON error. A router test now drives the session pages through the
middleware; it fails without the fix.
Membership queries, org-scoped usage and the reworked joins pass the
PostgreSQL store tests on 16.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A person was in exactly one project, chosen at approval, and could not
use the gateway before that. People belong to several teams and to
none, so:
- memberships: a person is a member of any number of organizations,
teams and projects, or of none; a member of a team or organization
may use every project in it. users join/leave, and the people page
adds and takes out from either side. Approval no longer needs a
project.
- personal keys: a key may have an owner and no project. /profile asks
what a new key spends against; keys create --for does the same.
auth.personal sets default limits for personal use or turns it off.
- limits for people: --scope user holds a person across every key,
project and the chat page; --per-member on an org, team or project
gives each member their own share there. Resolution folds the tree,
each per-member limit for the key's owner, and the owner's own, each
in its own pool; the quotas page shows and sets both.
- the chat page asks what to spend against and checks it per message.
- usage records who spent it, so a person's budget survives a restart,
personal use and chat included; restore no longer drops rows with no
project.
Leaving a project revokes the keys made there unless another membership
still reaches it. A personal key keeps no threads or notes, which are
per project, and is refused rather than dropped into the anonymous pool.
Migration 0010 turns each person's project into a membership, adds
usage.user_id and backfills it, and makes api_keys.project_id nullable.
SQLite cannot drop NOT NULL, so the table is rebuilt; that needs foreign
keys off, which only works outside a transaction, so migrations marked
for it run on one connection with foreign_key_check before commit.
Tested on a populated SQLite store with claims, and on a copy of the
cluster's PostgreSQL store: 5 people became members, 17 keys and 3
claims intact, 145 of 577 usage rows and all 69 chat rows attributed.
The PostgreSQL store tests, including new membership cases, pass on 16.
users move is gone; join and leave replace it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Nothing in the tree could be renamed, moved or removed, and an approved
person stayed in the project they were first put in — from the web or
the CLI. Now:
- /admin/users is the tree, each project opening onto its people and
keys, with approve, move, suspend, reinstate and remove for people,
revoke for keys, and add, rename, move and remove at every level,
plus a search over every account. Forms post and get the page back,
so it works with no script; htmx 2.0.11 (0BSD, embedded, checked
against npm's integrity hash) swaps only the changed part, keeping
open projects open.
- llmgw org tree/add/rename/move/remove, users move/delete, and
keys list --all.
- Projects, teams and organizations are named by id, path
(Acme/platform/gateway) or unique name; users by id or email. With a
single project --project can be left out, instead of defaulting to
"default", which a tree built from the web does not have.
Moving a person moves the keys they made; operator keys stay with the
project. Only empty things are removed, and never a project that had a
key, since usage history refers to it.
Also: keys init no longer reparents an existing team or project;
reinstating an account with no project is refused (a rejected signup
used to come back active and fail at its first key); rejecting a
signup removes it; keys limits --budget takes whole units like the
quotas page.
Checked in headless Chrome: a move from an open project updates the
page in place, keeps the project open, and the search still filters.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Installs PostgreSQL with the machine's own package manager when it has
none (initdb with password auth on the RHEL family, whose default is
ident), creates the llmgw role and database with a generated password
kept in /etc/llmgw/secrets/store-dsn, and checks the address connects.
On a machine already on SQLite it stops the gateway, copies the store
so accounts and keys survive, points auth.store at the new address and
starts it again. A move that fails starts the gateway on SQLite again.
server.yaml is edited as text so its comments survive, and kept beside
it. A PostgreSQL init did not install is not resized: it may serve
something else. It refuses to move one PostgreSQL store into another.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Usage could be read as a total — per key, per user, per model, per group —
and the cluster's own charts covered the fleet's hardware. What nobody could
see was when a group used it. "Research doubled its traffic three weeks ago"
is not a question a column of totals can be asked, and it is the question
somebody asks before a quota conversation.
/admin/usage draws requests, tokens and spend over a chosen window, grouped
by any level of the tenancy tree or by person. Person is not a level of that
tree — one person holds keys in several projects — so the grouping is its
own type rather than store.Scope with a fourth value nothing else could use.
Three things learned from the charts already here. Buckets are gap-filled
per group, because a line drawn straight across an absence claims traffic
that did not happen, and because the chart sums its tail by timestamp and
can only do that if every series shares a clock. Series are ranked by total
rather than by peak — a counter's question is "who used the most", and
ranking by peak would colour the team that made one burst over the team that
ran steadily all week. And the spend chart is replaced by a sentence when no
model has a price, rather than drawn flat along the bottom.
The bucket query groups by ordinal and was planned against the cluster's
PostgreSQL before being believed, with a test alongside the others that run
there: three reports have shipped working on SQLite and failing on it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Resolution took the most restrictive value along the chain and enforced only
that one. For a single caller's ceiling that is right: a team cannot raise what
its organization set. For a pool it is not, because two levels are two pools.
An organization given 1000 with four teams given 500 each spent 2000. Each team
filled its own counter, the organization's value lost to the tighter one and
was never looked at again. The number was written, displayed, reported against
— and enforced by nothing.
Every level that sets a limit is now checked against its own counter, and
settled onto all of them, so an organization's pool fills from its teams'
spending rather than only from requests that somehow named it. A team still
stops before its organization when its own pool runs out first, which is what a
sub-allocation is for.
Effective carries the chain as well as the collapsed values. The collapsed ones
are still what a header or a profile page should show — they are the ceiling a
single caller faces — they are just not enough to enforce with.
Rate limits charge as they check, deliberately, so that two concurrent requests
cannot both see room. With several levels that means a refusal at one has
already taken from the levels before it, so those are handed back. Without the
refund a team sitting at its organization's ceiling would quietly eat its own
rate limit with requests that were never served.
Budgets are checked before any of that, and only read, so a refusal on a budget
charges nothing at all.
Status reports the fullest pool rather than the smallest number. A team inside
an organization with the same budget is bounded by whichever is nearer its
ceiling, and that is the organization as soon as a second team spends anything;
a header showing the team's would disagree with the refusal after it.
A cluster default joins the chain as a key-level entry. Left out, it would be
reported as the ceiling and then never checked.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A quota could be enforced and reported, and set nowhere but a shell. Nothing in
the browser showed one at all — not what a team may spend, not what it has
spent, not that it had no limit. In practice that means quotas nobody sets.
/admin/quotas lists the whole tree, whether or not a group has a limit: the one
without a quota is the one most likely to need one, so it cannot be the row the
page leaves out. Spend over the last thirty days sits beside the budget, with
the fraction used.
Setting a quota here is a change of position. It was kept to the shell because
raising a budget is a financial act and a stolen admin session should not
perform one. That argument holds for exactly one row on this page: resolution
takes the most restrictive value along the chain, so raising a team's or a
project's limit cannot exceed what the organization allows. Only an
organization's own limit is bounded by nothing. Every change is logged with who
made it and what it was before.
Per-key limits stay on the CLI. A key belongs to a person and is already
bounded by its project; letting an admin session reach individual keys would
put every caller's throughput one click away, and the page refuses a key scope
rather than rendering it and failing later.
A limit an ancestor already caps lower is marked as written-but-unreachable.
Resolution means a project given more than its team is not a bigger budget, it
is a number nobody will reach — and without saying so the page would show it as
though it applied, with the first sign being a refusal that disagrees with the
screen.
The group is chosen, not typed. Ids became random in the previous commit so two
organizations could both have a "platform" team, which also made them
impossible to remember and a typo impossible to notice: it would write a quota
onto nothing and report success.
A blank field is no limit of that kind, not zero — zero is a limit of nothing,
which refuses every request. Clearing removes the row rather than writing nulls
across it: both resolve the same, only one tells a reader nobody has set a limit
here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An account is approved into a project, a project belongs to a team, a team to
an organization — and until now the only way to make any of them was `llmgw
keys init` on the control plane. So the first thing to do after installing was
leave the interface you were already in, and an approval form with an empty
dropdown gave no hint why.
The accounts page now creates all three.
Creating a container is not granting access: nobody can use one until an
account is approved into it, which is the step already on this page. Quotas stay
on the CLI. Making somewhere to put people is structural; raising what they may
spend is financial, and a stolen admin session should be able to do the first
and not the second.
Ids are random now instead of slugs of the name, which fixes a bug that only
appears when you have a second organization — which is exactly what this
feature invites. Team ids were slugs in one flat namespace and PutTeam upserts
on id, so creating "platform" under a second organization moved the first one's
team into it. Nothing failed. The team changed owner.
Create* is separate from Put* rather than replacing it. The upserts exist so
`keys init` can be re-run while provisioning a machine; a person filling in a
form means something else by a name that already exists, and gets told.
A duplicate is refused among siblings only. Two companies both having a
"platform" team is ordinary, and refusing the second would be the flat namespace
again wearing a better error message.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Quotas can be written on an organization, a team or a project, and nothing
could show what any of them had used: usage came back per key, per user and per
model only. A quota whose consumption cannot be seen is one nobody reviews
until the day it refuses somebody — and since the previous commit made a group
quota a shared pool, it is now a number several people are spending against
without being able to look at it.
`llmgw ctl usage --by team` sums every key beneath each group, with the budget
that level sets beside it.
Beside it, and only when that level sets one. A team bounded by its
organization's budget has none of its own, and printing the organization's
against each of its teams would read as though each had that much — the exact
misreading this work exists to remove.
Joined through api_keys rather than read from usage.project_id: that column
records where the key was when the request was served, while the quota is
written against the tree. They agree until a key is moved, and a report that
disagrees with the limit shown beside it is the useless one.
Three existing reports were fixed on the way, all failing on PostgreSQL and all
passing in tests, because the tests run on SQLite:
column "k.name" must appear in the GROUP BY clause (UsageSince)
column "us.email" must appear in the GROUP BY clause (UsageByUser)
column "usage.ts" must appear in the GROUP BY clause (UsageBuckets)
SQLite picks an arbitrary row for an ungrouped column and carries on; PostgreSQL
refuses. The first two were selecting names they had not grouped. The third
grouped "ts / ?" while selecting "(ts / ?) * ?", which PostgreSQL matches as
text and so saw a bare ts — it is grouped by ordinal now, the same expression by
construction. The dashboard's usage charts have been empty on the cluster this
whole time, with the reason only in the log.
All four queries were run against the cluster's own PostgreSQL before this was
committed, because passing on SQLite is what they were already doing.
The grouping level picks from a fixed table of levels rather than being
interpolated, so an unknown one is an error and not a fragment of SQL.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two things that were only ever confusing, and one of them stopped a command
working on the cluster.
The store type was named SQLite and kept the name when PostgreSQL was added, so
OpenPostgres returned a *SQLite and all seventy-odd methods on the production
store were spelled after the database it was not running on. Nothing broke;
people did. Earlier today I added three methods to *SQLite for a table that on
this cluster only ever lives in PostgreSQL, and the name did not once look
wrong. It is a field, which is all it ever was.
`llmgw users` and `llmgw keys` defaulted --store to a SQLite path. On a
deployment whose store is PostgreSQL every one of them failed with "unable to
open database file" while the right answer sat in auth.store in the config file
beside them; the conclusion an operator draws is that the tool is broken, and
they are not far off. They now ask the server's own configuration before
falling back, and the error names which of the three sources chose the address
— the failure was never the path, it was which one picked it.
The single-machine default still applies last, so a deployment with no config
file keeps working with no flags.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The tenancy tree handed limits down and never pooled what was spent against
them. A limit written on a team was inherited by every key beneath it, and then
enforced with the key as the counter — so a project given a budget of 1000 and
ten keys permitted ten thousand, and a team given 60 RPM got 60 per member.
That is "each of you may spend this much", which is a useful thing to be able
to say. It is not what somebody writing a budget on a team means, and nothing
in the system said which one you were getting.
Resolution now records which scope set each value that won, and the counter is
keyed by that scope. The value and the counter come from the same place, so a
project tightening its team's limit is enforced against the project's own pool
rather than eating into a quota another project never agreed to.
Each limit is owned separately, because they resolve separately: a team budget
with a per-key rate limit — the group's money, the individual's throughput — is
the ordinary arrangement and has to stay expressible.
Cluster defaults are owned by the key. Read as a shared pool, one default would
become a single cluster-wide rate limit that every key in every organization
competes for: not a default, an outage.
Restoring after a restart had to change with it. Per-key totals rebuilt a
counter named after the key while the check read one named after the team, so
the control plane came back believing the team had spent nothing — a budget
that reset on every restart, for exactly the scopes where it matters most.
SpendSinceByScope returns each key's place in the tree and the accountant folds
it into every scope that might own a budget.
Counters are namespaced by level throughout. A bare key id in one path and a
"key:"-prefixed one in another are two counters for the same key, which is how
the restore was silently doing nothing.
Still missing, and worth naming: no usage report rolls up by project, team or
org, so a shared quota can now be enforced but not yet shown.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`ctl bench` has always been able to work out what one replica can serve — it
ramps concurrency until the SLO breaks and reports the last level that held.
It printed that and nothing kept it, so the one number saying what a replica is
worth lived in whoever ran it's scrollback. Everything downstream then had to
guess: admission applies a single global default for healthy queue depth, and
sizing a model against real demand had nothing to size against.
`ctl bench --save` records it; `ctl capacity` reads it back.
Keyed by model and profile. Capacity belongs to the hardware and the engine
arguments as much as to the weights, so the same model on CPU and on an H100
are two numbers, and one key would overwrite the other on every run.
The profile is looked up from the replica rather than asked for, because the
control plane already knows which profile answers on which endpoint and a
measurement filed under the wrong one is worse than none. A target that is not
a replica — the gateway, usually — is refused with that said: benchmarking the
whole path is worth doing and says nothing about the hardware.
The conclusion is stored, not the ramp. Every level and its latencies are worth
reading once while judging whether the run was sound; keeping them would grow
without bound as a model is re-benchmarked.
Re-measuring replaces. A second run is a better answer to the same question,
not another data point, and keeping both would leave a reader choosing between
numbers with no basis for the choice.
The SLO and the date are stored with it. Capacity means nothing without the
target it was measured against, and hardware and engine builds change — a
measurement is an observation with a date on it, not a fact about the model.
A failed save is reported and not fatal: the run has already cost the GPU time.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The choice is unchanged and now better argued: a deployment does not announce
which language its notes are in, so any stemmer picked here is the wrong one
somewhere — and applying the wrong one does not fail loudly, it quietly stops
finding things.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
These TRUNCATE every table before each test. Pointed at a deployment that is a
silent and complete data loss — every key, every user, every conversation.
Not hypothetical. I did it: an SSH tunnel to the production database had been
opened minutes earlier for a perfectly good reason, and the same DSN was then
handed to the test suite. Twenty-one users, nine API keys, eighteen sessions,
four hundred and sixty-one usage records and three runtime-added models, gone
between one command and the next. Recovered only because the pre-migration
SQLite file was still there.
The guard is the database name. A test database has to say so, because a check
that could be satisfied by accident would not have stopped this one — the DSN
was entirely correct for what it had been opened for, and catastrophic for
what it was then used for.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The first thing HA needs that a shared store does not give. Two control planes
against one store both reconcile: both see a model wanting two replicas and
none running, and both place. The model ends up with twice what it asked for,
each replica holding GPUs the other's inventory believes are free — and nothing
crashes to say so.
A lease in a table rather than a lock held by a connection. PostgreSQL has
advisory locks and SQLite has nothing like them, and this store keeps one set
of queries for both; a row with an expiry works everywhere, and it survives a
leader being powered off rather than asked to stop, which is the case that
matters. Taking and renewing are the same statement, so the two cannot disagree
about what a held lease looks like, and it is one statement, so two processes
racing cannot both believe they won.
The owner id is random rather than the hostname. A hostname would let a
restarted process match its own stale lease and believe it had been leading all
along — including through the gap that made it restart, during which somebody
else may have taken over and started placing.
A store that cannot be reached is not evidence that somebody else is leading,
so an unreachable store keeps the claim. Two leaders is worse than one, but no
leader means nothing is placed and nothing is evicted, and a brief database
blip should not stop a cluster healing itself.
Unset means leader. Every deployment that existed before this, every
single-control-plane install and every test acts unconditionally, with nothing
to configure.
The same seven assertions run against both backends, which is how the test
harness was caught inheriting a lease between subtests — it passed on SQLite,
where each subtest gets a fresh file, and failed only on PostgreSQL.
What this does not yet do is move agents. A standby holding an agent's stream
is one the leader cannot command, so failover is degraded rather than clean
until agents know more than one address.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Keys are stored hashed, so a migration that drops them cannot recreate them:
every person and every script needs a new one issued, and the old ones cannot
even be looked up to say which they were. `llmgw keys move-store` copies
instead.
Found by running it against the real cluster, twice, after the gateway had
already been stopped.
The first order came from the order the migrations create tables in, and
users is created in a later migration than api_keys even though api_keys
references it — so the copy died three tables in. The order is now checked by
a test that derives the foreign keys from the migrations rather than restating
them, so a new reference fails there instead of during somebody's migration.
Checked by putting the wrong order back and watching it fail.
Then key_claims. Its sealed column is emptied the moment a claim is used, so
every claimed row holds an empty blob in a NOT NULL column. An empty blob
scans as a typed nil slice and a genuine NULL as an untyped one — a
distinction this side can see and pgx cannot, since it encodes any nil slice
as NULL. Restored before the insert, which is the difference between a copy
that completes and one that stops four tables in.
One transaction per table rather than one for the whole copy: a single
transaction holding every row of a large usage table is how a migration runs
out of memory on the machine least able to spare it. A failure therefore
leaves a partial copy, which is why the destination must have been empty —
starting again is dropping and recreating it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A path is still SQLite and a postgres:// URL is PostgreSQL, so every existing
configuration keeps working untouched and moving is one line changed rather
than a new setting to find. An address of the form @/path/to/file reads the
real one out of that file, which is how a DSN carrying a password stays out of
a configuration that gets copied, diffed and pasted into issues.
The driver is pgx, which is pure Go. That is not a preference: the whole binary
is built with CGO_ENABLED=0 so it runs on a node whose libc is older than the
builder's, and a cgo driver would take that away from every deployment,
including the ones still on SQLite.
Tested against a real PostgreSQL, which found two things nothing else could.
The schema_migrations table is created in Go rather than in a migration file,
so it escaped the port and still said INTEGER — 64 bits in SQLite, 32 in
PostgreSQL, and a millisecond timestamp does not fit, so the very first
migration failed to record that it had run. And the memory migration uses an
FTS5 virtual table with three triggers keeping it in step, which has no
counterpart: PostgreSQL derives a tsvector from the row, so there is nothing
to keep in step and no trigger to forget.
That leaves exactly one query that is not the same sentence in both dialects —
the note search, MATCH against @@ — and it is written out twice rather than
hidden behind a shared spelling that would mean something subtly different in
each. Everything else is one set of queries.
The text search uses the 'simple' configuration rather than English. These are
an agent's own notes, in whatever language it chose, and stemming English
rules over Turkish would quietly stop finding things.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Groundwork for PostgreSQL, with no PostgreSQL in it yet: the store still opens
SQLite and behaves exactly as it did.
One set of queries rather than one per database. The SQL here is almost
entirely portable — ON CONFLICT upserts work in both, and the only construct
that genuinely differs is the placeholder — so two copies of every query would
buy nothing and drift, one query at a time, discovered in production.
The rewrite is a wrapper around the handle rather than a change at each call
site. There are sixty of those and sixty chances to forget one; this way the
only route to the driver goes through the rewrite. Making it a wrapper found
six places that reached past it, including all three manual transactions and
two prepared statements — every one of which would have worked on SQLite and
failed on PostgreSQL, on a path nobody exercises before deploying.
The rewriter skips quoted sections. No query here contains a question mark in
a literal today; this is so that adding one is a test failure rather than a
statement that silently binds the wrong argument.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The storage half of continuing a thread on a different model than it started
on. Nothing uses it yet.
Until now llmgw stored no message content anywhere: usage rows carry token
counts, costs and latencies, never text, and even its own chat page keeps the
conversation in a JavaScript variable that a refresh discards. That property
is worth more than the feature, so it is the default and stays the default.
A project stores nothing until someone turns it on for that project.
Retention is a number of hours rather than a boolean, because a thread store
with no expiry is a transcript of everything a team ever typed, kept forever
because nobody chose a number. Turning it on means saying for how long. The
window measures idleness rather than age — expiring a conversation someone is
still having would be a strange way to keep that promise.
Threads are scoped to a project and the scope is part of every lookup, not a
check afterwards. Reading a thread from another project and reading one that
does not exist return the same error: two errors would let a caller learn
which ids are real in projects they cannot read. Writing across projects is
refused for the same reason in the other direction — append, then read your
own reply back.
Deleting a thread takes its messages with it, which the test checks by
counting rows rather than by trusting the cascade. Someone has to be able to
say "forget that", and a retention window is no answer to a message sent by
mistake.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The dashboard could say what the cluster was made of and not who was
spending it. Usage was per key, which is the right unit for revoking
something and the wrong one for the question actually being asked: one
person holds a laptop key, a CI key and a notebook key, and adding three
rows up by eye is a chore that gets skipped.
Per person now, joined through the keys they own, and per model — counted
by the model that answered rather than the one requested, since those differ
whenever a fallback serves and what actually ran is the question. Headline
figures are stat tiles with a sparkline: a single current value is a number,
and plotting one is how a chart ends up with a single bar in it. The
sparkline says only rising, falling or flat, which is the one thing the
number cannot.
Ranked horizontal bars, because the labels are email addresses and model
names and the label is the point — the question is which person, not what
shape the distribution has. Rotated or truncated axis labels say the chart
matters more than reading it. Trimmed where drawn and complete on hover, so
the plot keeps its geometry and a reader can still find out whose row it is.
The palette is validated rather than chosen: every adjacent pair clears the
colour-vision separation threshold, and the dark values are the same hues
re-stepped for the dark surface instead of an automatic flip. Run through a
validator, because that is the only way anyone can honestly claim a palette
is colourblind-safe.
Which turned up a bug in the existing chart code. cssVar is an allowlist of
four names that silently returns --accent for anything else, and callers
wrap it in var() themselves — so every new series would have rendered in one
colour with accent-coloured text. Caught by a test written from the rule
that chart text wears text tokens, never the series colour.
A key with no owner is labelled "operator-issued keys" rather than unknown:
it predates accounts and is doing its job, and "unknown" would imply
something is wrong.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The catalog can change while the cluster runs; this is where a change goes
so it is still there tomorrow. Models declared in configuration and models
added at runtime are two sources on purpose: config-as-code stays
declarative and under review, and the database holds what someone added to
try it.
Configuration wins every conflict, and the direction matters. A request for
a reviewed production model must never be answered by something added from
a laptop, so a stored model colliding with a declared name is refused when
it is added. A declared model cannot be removed at runtime either — it
would return on the next restart, leaving the cluster disagreeing with its
own config file until then, which is a worse answer than refusing and
saying where to edit.
The whole catalog is built before anything is written. A model naming a
missing fallback, or colliding, or malformed in any of the ways catalog.New
refuses, is rejected rather than persisted and then skipped forever after —
a model that looks added and never appears is the worst outcome available.
Removal is checked the same way, because another model may fall back to the
one being removed, and a fallback pointing at nothing is discovered during
the outage it was meant to cover.
At startup each stored model is validated on its own against what is
already accepted, so one bad row is skipped and named rather than taking
the cluster with it. The declared models are precisely the ones that must
survive that.
Catalog types now carry json tags mirroring their koanf ones, so a stored
definition is the same shape as the YAML someone writes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Unlimited was the default. Creating a project without limits is the easy
path and the common one — the "default" project on this cluster is one, and
the person using it is metered by nothing at all. The first sign of that is
the bill or the outage.
auth.default_rpm and default_tpm now apply to a key no scope gives a limit
to. Zero keeps the old behaviour, which is still a reasonable choice; the
point is that it becomes a choice rather than an omission.
Applied after the chain rather than folded in as its widest scope, because
a default is a fallback and not a ceiling: a project deliberately given 600
rpm must keep it, and tightening would silently cap it at the default. An
explicit limit wins in both directions.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
So someone can try a model without setting up a client, and — more often —
tell whether a problem is the gateway or their code.
The page carries a session, and the server turns that into the identity an API
key would have produced, resolved from the person's project. The request then
goes down the ordinary path by being handed to the same handler /v1 uses:
admission, model allowlist, rate limits, budget, accounting. There is exactly
one request path, so there is no second place for its rules to drift — and
without that, a playground is a hole in every limit the rest of the gateway
enforces. The allowlist test posts a forbidden model and expects 403 for that
reason.
The body is rebuilt rather than forwarded, so the page cannot set fields it has
no business setting: a stream:false that would hang the reader, a max_tokens
large enough to occupy a replica for an hour. One turn is capped at 512 tokens.
The picker offers what is actually served and allowed, not what is catalogued,
because offering a model with no replica produces a 503 the reader cannot
interpret.
Inline vanilla JavaScript, for the same reason the charts are hand-drawn SVG.
It streams, because on CPU inference ten seconds is normal and a chat that
hangs and then dumps a paragraph feels broken. Replies are written with
textContent throughout: model output and user input have no business being
parsed as markup. Its POSTs carry the CSRF token in a header, which a
cross-site form cannot set at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two things the dashboard was missing: it could not act, and it was a column of
numbers.
Account administration moves to /admin/users — approving into a project,
suspending, reinstating. That is the work an operator does constantly and it
should not need a terminal. Three things stay on the CLI, and the line is not
arbitrary: promoting someone to admin, resetting a password, and issuing a key
on another person's behalf each turn a stolen admin session into more than it
was — permanent access, or somebody else's account. Approving and suspending
do not. Suspending signs the account out everywhere, or suspension is advisory
until the session happens to expire, and an admin cannot suspend themselves,
which is not a thing an interface should let you do by misclicking.
The charts are inline SVG generated in Go. A charting library would mean a
JavaScript toolchain in CI and a megabyte of bundle for what is a list of
rectangles, and the single-binary goal rules that out. Hourly over a day: fine
enough to show a working day, coarse enough that one slow minute is not an
incident. Empty hours are filled in, because a chart with hours missing where
nothing happened reads as a chart with hours missing from the data.
The SVG is marked as trusted HTML, so everything that reaches it is either a
number this package computed or escaped explicitly, and a bar's colour can only
be one of four names — a colour that could be anything is a way to write CSS
into every page the chart appears on. There are tests for both.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
