/forgot mails a link to set a new password when the cluster's mail settings
say how - SMTP relay, sender, and the public base_url the link points at,
never the request's Host - and says the same whether or not the address has
an account. /reset sets it once, closing every session. Links are kept by
their hash in password_resets, good for an hour, a newer one replacing an
older. Without mail, /forgot says to ask an operator, and llmgw users
reset-link makes the link to hand over. Single sign-on accounts get none.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
guard.enabled (unset is on) makes every policy check nothing while false,
live: llmgw ctl guard off/on set it in the cluster's settings, and ctl
status, ctl guard list and the guardrails page say when they are off.
A refusal by a judge names the hazard it saw - judge: S6,Specialized
Advice - and the verdict is kept with the request in a new usage column,
guard_rule, shown by ctl request, the request log and the guardrails page.
llmgw ctl guard judge ask has a judge read a text, past the cache, and shows
its raw answer and how it is read; the endpoint is kept out of the audit so
no text is recorded.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Everything in server.yaml that is not one machine's own is now a versioned
document in the store (cluster_settings), the same on every control plane.
The first start against a store imports the file's cluster keys, with the
baseline guard policy when there is none; after that those keys in a file
are ignored, with a warning.
Each control plane polls the document's version and its own server.yaml's
hash. A change is validated and applied: guard live (Gate.Replace), anything
else by a graceful restart that systemd brings back, standbys first and the
leader 30s later. A version that does not load on a machine is not applied;
at start the newest one that loads is used, so a bad edit cannot keep every
control plane down.
llmgw ctl config show/get/set/unset/edit/history/rollback, the admin API
under /api/v1/settings, and the console's Settings screen, which layers the
store's document over the machine's file, write it. Commands beside the
control plane that read cluster keys (init, node add, server add, backup,
upgrade) read the layered configuration; init --bundle writes offline to
the store once it holds the settings.
A standby that becomes the leader re-reads models, engine choices and
facility holds from the store.
Retires appendGuardIfMissing: the import does it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Conflicts with facility-aware placement (config, control, admin API routes,
store test tables) all keep both; the generated reference and OpenAPI
document regenerated.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ctl federation add/list/remove/sync keeps peers in the store (migration
0029); their /v1/models are offered here as external models named
PEER/MODEL and kept in step, a model still named by a fallback kept
withdrawn. Peers are checked at /ready, timed, and taken out of routing
while down. Requests to a peer carry x-llmgw-hop and are refused (508) when
they come back to a cluster they passed through; residency and no_store
keep requests off peers; X-Request-Id and, with --send-tags, attribution
tags go along. ctl model fallbacks sets overflow to a peer. Console view,
metrics, admin API routes and docs.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Nodes say where they are with llmgw.io/site, room, row, rack and pdu labels,
from agent.yaml or `llmgw ctl label` (kept in the store, over the agent's).
Placement spreads a model's replicas across racks and PDUs, keeps a
multi-node replica within one domain when its profile says keep_within,
places nothing in a domain held with `llmgw ctl facility hold` (optionally
draining it), and refuses a shard that would take a domain past its power
cap from facility.budgets, counting declared, derived and measured draw.
Explain-placement names the domain, and keeps the rejections of a placed
replica; `ctl nodes` gains LOCATION and --domain; `ctl facility` lists the
domains; the console's Nodes view shows and sets location and holds.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Conflicts: docs.go (manual, OpenAPI and suites all kept), modelserve Job
(trace and eval both kept). The eval routes are added to the OpenAPI route
table, update and promote documented with their optional body, and the
document regenerated.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A suite is a set of cases in YAML or JSON lines, each scored by exact,
contains, regex, a number within a tolerance, a JSON Schema subset, or a
judge model with a rubric; weighed into one score against a threshold.
internal/bench/eval runs one on the control plane through Router.Replay, as
the gateway's own requests, and keeps every run and its cases in the store
(migration 0028: eval_runs, eval_cases).
- llmgw ctl eval MODEL --suite FILE|NAME, --history, --run, --suites; exits
non-zero below the score. Suites by name from control.eval_dir, or the
two built in from deploy/eval (arithmetic, json-format).
- model update and promote take --eval SUITE --min-score N, or a model's
standing gate (ctl model gate, eval: in the catalog): a promote asks the
canary first and leaves it below the score; an update tries the new
weights as a canary given no share, then promotes or removes it.
- The console's model menu: ask a suite with its progress, its runs, its
gate.
- The audit middleware no longer hands a handler a body cut off at 64 KiB.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
llmgw keys cache --project P --ttl 1h [--any] | --off, and Cache answers… on
the console's project menu. The same request — project, endpoint, canonical
model and the body without stream, stream_options, user, metadata, store and
service_tier, keys sorted — is answered from the gateway's memory for the
TTL. Only reproducible answers (temperature 0 or a seed; embeddings, rerank,
score) unless --any; only a whole 200 from the model asked for, not a
fallback's, a canary's, or one a guardrail changed.
Looked up after the limits and the input guardrails; the output policy
judges a reused answer; no_store and stored conversations bypass it.
Streamed answers are kept put back together and streamed back as chunks.
x-llmgw-cache: hit|miss|bypass, Age on a hit, Cache-Control: no-cache or
x-llmgw-cache: no to bypass. A hit is recorded with no tokens and no cost,
marked cache_hit (migration 0027) and shown by ctl request. An LRU bounded
by router.cache.max_bytes and max_entry_bytes; llmgw_cache_requests_total,
llmgw_cache_bytes, llmgw_cache_entries.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Resolve now checks the owner's state for a personal key; the authenticator
treats an inactive owner as a revoked key. Suspension stops the keys within
the cache TTL, reinstatement brings them back.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
server add gives another machine this llmgw, a certificate, the
configuration and store address, lets it into the store through Patroni, and
starts it; --vip keeps a floating address on the leader with keepalived,
following a new /leader endpoint. Nodes learn every control plane from the
leader and notice a dead one by HTTP/2 ping instead of after the kernel's
fifteen minutes. The setup, console, node and init screens are drawn full
screen in one frame: title bar, sidebar, bordered panel, key bar, dialogs.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
init --patroni user@host ... installs PostgreSQL, Patroni and etcd on the
given machines over ssh, with this one as the third etcd vote, makes the role
and database, and moves the store in use into the cluster. Each step checks
first, so it changes nothing on a cluster that is up. For a store of several
hosts the control plane publishes each member's role, state and lag from
Patroni, a stopped member as missing, prints it in ctl status, and alerts on
it. Making an administrator on PostgreSQL sent a bool to an integer column;
it sends an integer now.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Percentiles are read from the histograms kept with the usage sums and shown
per group on the usage tab and in the people page's panels. Requests refused
by a limit are recorded as 429s against their owner, free of charge, so a team
hitting its limits shows as such. Usage can be grouped by model.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Charts, totals and budgets summed the request table on every view. Usage is
now also summed in the writing transaction into five-minute, hourly and daily
tables per key, person, project and model, with latency and first-token
histograms. Charts read the coarsest tier their slices allow; totals and
budgets compose rows, slices, hours and days, and stay exact. Request records
can be given a retention without moving any total.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A node that cannot run the CUDA 13 build installs the same version's
CUDA 12.9 build from vLLM's GitHub release, with PyTorch's cu129 torch;
the install computes on the GPU before it counts. The build can be
chosen per node — auto, cu13, cu129, none — on the cluster page, with
ctl engine-build, or over the admin API; the control plane keeps the
choice and tells the node at once.
Refs #2.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A taken address on PostgreSQL fell through to a generic error, because
the store looked for SQLite's "UNIQUE". It checks code 23505 too. The
done page tells someone whose address had an account to sign in or ask
for a reset, without saying which case it was; other failures give a
logged reference.
Refs #2 (item 6).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A model has an owner (org, team or project). Quotas guarantee a scope
GPUs, cores and memory and may limit it; what a scope holds is summed from
its models' replicas and those beneath it. A replica within its owner's
guarantee is not taken for another scope; what is held beyond a guarantee
is borrowed and taken back for a scope within its own. A limit stops a
model before it starts, and headroom stops there too. ctl quota and ctl
model owner set them; metrics and a CapacityAtLimit alert report them.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
"Why was this request slow?" had no answer. Nothing tied a request to the
replica that served it, the time admission held it went out in a response
header and was never kept, and the only per-request log lines were for
disconnects, early stream ends and upstream retries.
Every request now has an id: the client's X-Request-Id when it is a plain
string of reasonable length, otherwise one the gateway makes. It is set on
the response before anything else, so a request refused as malformed can be
named too, and forwarded to the engine. One access log line per request
carries it with the model asked for and served, the replica, status, time in
the queue, time to the first token, total time, tokens, and the key,
project and user.
Usage records gain the id, the replica, queue time and time to first token,
tried against a restored copy of the live store before being shipped: 591
rows kept, columns and index in place. `llmgw ctl request ID` reads one back
through the admin API.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
There were no backups: docs/install.md gave a pg_dump command, and nothing
ran it. The store holds every key, person, quota and usage record, the models
added at runtime, and now the secret each engine's key is derived from; the
certificate authority's key sits in /etc/llmgw/pki beside it. Losing the disk
lost all of it.
`llmgw server backup` writes both to one archive: pg_dump's custom format, or
a VACUUM INTO snapshot for SQLite, plus /etc/llmgw. The archive holds the CA's
key, so it is 0600; the directory is listable, so the control plane — not
root — can see when the newest was written. pg_dump is given the connection
through its environment, since a DSN carries the password and a process's
arguments are readable by every user. Written beside its name and renamed,
so an interrupted backup is never the newest; the newest 14 are kept, and
nothing the command did not write is ever removed.
llmgw-backup.timer runs it nightly and catches up after downtime. The control
plane publishes the newest archive's time, and BackupTooOld fires past 26
hours — or when there has never been one. The install guide now covers
restoring, and says to copy archives off the machine, since they share its
disk.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Engines listen on every address their node has, with no key. Checked on the
live cluster: from a neighbouring node, a request straight to an engine's
port was answered — past the gateway's authentication, quotas and
accounting, by anyone who could reach the node.
Each replica's engine is now started with a key and refuses requests without
it. The key is derived from the replica id with an HMAC over a secret kept
in the store, rather than stored per replica: the router has to send the
right key to replicas this process never started — adopted after a restart,
or placed by the other control plane — and a function of a shared secret
gives that without a table to keep in step. A key taken from one node opens
that replica only. The secret is made on first use and copied with the rest
of the store when moving to PostgreSQL; a new one would lock the router out
of every running engine at the moment of the move.
The key reaches llama.cpp and vLLM through their environment
(LLAMA_API_KEY, VLLM_API_KEY) rather than their command line, which every
user on the node can read; SGLang only takes it as an argument. The agent's
readiness check, the router's proxy and its metrics scrape send it, the
client's own Authorization still never reaches the engine, and `ctl bench`
fetches the key of the replica it is pointed at from the admin API.
Engines started before this keep running without a key until their replica
is next started. With no store there is nowhere to keep the secret, and
engines start without one, with a warning.
Also formats two files that went in unformatted.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The account page (was "Your keys") shows the last thirty days of the
person's own use - requests, tokens, spend and tokens per day - and, for
each place they can spend, what they used there and a bar for every
budget that applies: their own, a per-member one, or a project's, team's
or organization's shared with others, named as such. Bars turn amber past
four-fifths and red when spent. It sits under the keys now, above the
password form.
Spend is read from the usage table for the pool each limit counts
against, from the start of its period.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The cluster page's usage table showed session:usr-... and proj-... ids;
it now shows the person's email, whether it came through a key or the
chat page, and the project's path or "personal". Times read "3 hours
ago" rather than "3h7m25s ago", everywhere.
The usage page's legend gave each line's latest slice, which was 0
beside a table of hundreds; it gives the window's total, and the table's
numbers are right-aligned like everywhere else.
The PostgreSQL placeholder rewriting skips line comments. An apostrophe
in one opened a string that swallowed every placeholder after it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A member of an organization can be its manager (users join --org X
--role manager, or the star on the people page). /org shows a manager
their organization: usage over time by team, project or person, its
teams and projects with limits and 30-day spend, and its members, each
once with every place they belong. Read-only; administrators see every
organization. Only what was spent in the organization's projects is
shown — UsageBucketsIn filters by where a request was spent.
Found on the way:
- Usage by org, team and project, and the quotas page's spend, joined
through api_keys and so dropped every chat-page request, which has no
key row. They now read usage.project_id; keys no longer move between
projects, so the two agree.
- /org was not exempt from API-key auth and answered browsers with a
JSON error. A router test now drives the session pages through the
middleware; it fails without the fix.
Membership queries, org-scoped usage and the reworked joins pass the
PostgreSQL store tests on 16.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A person was in exactly one project, chosen at approval, and could not
use the gateway before that. People belong to several teams and to
none, so:
- memberships: a person is a member of any number of organizations,
teams and projects, or of none; a member of a team or organization
may use every project in it. users join/leave, and the people page
adds and takes out from either side. Approval no longer needs a
project.
- personal keys: a key may have an owner and no project. /profile asks
what a new key spends against; keys create --for does the same.
auth.personal sets default limits for personal use or turns it off.
- limits for people: --scope user holds a person across every key,
project and the chat page; --per-member on an org, team or project
gives each member their own share there. Resolution folds the tree,
each per-member limit for the key's owner, and the owner's own, each
in its own pool; the quotas page shows and sets both.
- the chat page asks what to spend against and checks it per message.
- usage records who spent it, so a person's budget survives a restart,
personal use and chat included; restore no longer drops rows with no
project.
Leaving a project revokes the keys made there unless another membership
still reaches it. A personal key keeps no threads or notes, which are
per project, and is refused rather than dropped into the anonymous pool.
Migration 0010 turns each person's project into a membership, adds
usage.user_id and backfills it, and makes api_keys.project_id nullable.
SQLite cannot drop NOT NULL, so the table is rebuilt; that needs foreign
keys off, which only works outside a transaction, so migrations marked
for it run on one connection with foreign_key_check before commit.
Tested on a populated SQLite store with claims, and on a copy of the
cluster's PostgreSQL store: 5 people became members, 17 keys and 3
claims intact, 145 of 577 usage rows and all 69 chat rows attributed.
The PostgreSQL store tests, including new membership cases, pass on 16.
users move is gone; join and leave replace it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Nothing in the tree could be renamed, moved or removed, and an approved
person stayed in the project they were first put in — from the web or
the CLI. Now:
- /admin/users is the tree, each project opening onto its people and
keys, with approve, move, suspend, reinstate and remove for people,
revoke for keys, and add, rename, move and remove at every level,
plus a search over every account. Forms post and get the page back,
so it works with no script; htmx 2.0.11 (0BSD, embedded, checked
against npm's integrity hash) swaps only the changed part, keeping
open projects open.
- llmgw org tree/add/rename/move/remove, users move/delete, and
keys list --all.
- Projects, teams and organizations are named by id, path
(Acme/platform/gateway) or unique name; users by id or email. With a
single project --project can be left out, instead of defaulting to
"default", which a tree built from the web does not have.
Moving a person moves the keys they made; operator keys stay with the
project. Only empty things are removed, and never a project that had a
key, since usage history refers to it.
Also: keys init no longer reparents an existing team or project;
reinstating an account with no project is refused (a rejected signup
used to come back active and fail at its first key); rejecting a
signup removes it; keys limits --budget takes whole units like the
quotas page.
Checked in headless Chrome: a move from an open project updates the
page in place, keeps the project open, and the search still filters.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Installs PostgreSQL with the machine's own package manager when it has
none (initdb with password auth on the RHEL family, whose default is
ident), creates the llmgw role and database with a generated password
kept in /etc/llmgw/secrets/store-dsn, and checks the address connects.
On a machine already on SQLite it stops the gateway, copies the store
so accounts and keys survive, points auth.store at the new address and
starts it again. A move that fails starts the gateway on SQLite again.
server.yaml is edited as text so its comments survive, and kept beside
it. A PostgreSQL init did not install is not resized: it may serve
something else. It refuses to move one PostgreSQL store into another.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Usage could be read as a total — per key, per user, per model, per group —
and the cluster's own charts covered the fleet's hardware. What nobody could
see was when a group used it. "Research doubled its traffic three weeks ago"
is not a question a column of totals can be asked, and it is the question
somebody asks before a quota conversation.
/admin/usage draws requests, tokens and spend over a chosen window, grouped
by any level of the tenancy tree or by person. Person is not a level of that
tree — one person holds keys in several projects — so the grouping is its
own type rather than store.Scope with a fourth value nothing else could use.
Three things learned from the charts already here. Buckets are gap-filled
per group, because a line drawn straight across an absence claims traffic
that did not happen, and because the chart sums its tail by timestamp and
can only do that if every series shares a clock. Series are ranked by total
rather than by peak — a counter's question is "who used the most", and
ranking by peak would colour the team that made one burst over the team that
ran steadily all week. And the spend chart is replaced by a sentence when no
model has a price, rather than drawn flat along the bottom.
The bucket query groups by ordinal and was planned against the cluster's
PostgreSQL before being believed, with a test alongside the others that run
there: three reports have shipped working on SQLite and failing on it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Resolution took the most restrictive value along the chain and enforced only
that one. For a single caller's ceiling that is right: a team cannot raise what
its organization set. For a pool it is not, because two levels are two pools.
An organization given 1000 with four teams given 500 each spent 2000. Each team
filled its own counter, the organization's value lost to the tighter one and
was never looked at again. The number was written, displayed, reported against
— and enforced by nothing.
Every level that sets a limit is now checked against its own counter, and
settled onto all of them, so an organization's pool fills from its teams'
spending rather than only from requests that somehow named it. A team still
stops before its organization when its own pool runs out first, which is what a
sub-allocation is for.
Effective carries the chain as well as the collapsed values. The collapsed ones
are still what a header or a profile page should show — they are the ceiling a
single caller faces — they are just not enough to enforce with.
Rate limits charge as they check, deliberately, so that two concurrent requests
cannot both see room. With several levels that means a refusal at one has
already taken from the levels before it, so those are handed back. Without the
refund a team sitting at its organization's ceiling would quietly eat its own
rate limit with requests that were never served.
Budgets are checked before any of that, and only read, so a refusal on a budget
charges nothing at all.
Status reports the fullest pool rather than the smallest number. A team inside
an organization with the same budget is bounded by whichever is nearer its
ceiling, and that is the organization as soon as a second team spends anything;
a header showing the team's would disagree with the refusal after it.
A cluster default joins the chain as a key-level entry. Left out, it would be
reported as the ceiling and then never checked.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A quota could be enforced and reported, and set nowhere but a shell. Nothing in
the browser showed one at all — not what a team may spend, not what it has
spent, not that it had no limit. In practice that means quotas nobody sets.
/admin/quotas lists the whole tree, whether or not a group has a limit: the one
without a quota is the one most likely to need one, so it cannot be the row the
page leaves out. Spend over the last thirty days sits beside the budget, with
the fraction used.
Setting a quota here is a change of position. It was kept to the shell because
raising a budget is a financial act and a stolen admin session should not
perform one. That argument holds for exactly one row on this page: resolution
takes the most restrictive value along the chain, so raising a team's or a
project's limit cannot exceed what the organization allows. Only an
organization's own limit is bounded by nothing. Every change is logged with who
made it and what it was before.
Per-key limits stay on the CLI. A key belongs to a person and is already
bounded by its project; letting an admin session reach individual keys would
put every caller's throughput one click away, and the page refuses a key scope
rather than rendering it and failing later.
A limit an ancestor already caps lower is marked as written-but-unreachable.
Resolution means a project given more than its team is not a bigger budget, it
is a number nobody will reach — and without saying so the page would show it as
though it applied, with the first sign being a refusal that disagrees with the
screen.
The group is chosen, not typed. Ids became random in the previous commit so two
organizations could both have a "platform" team, which also made them
impossible to remember and a typo impossible to notice: it would write a quota
onto nothing and report success.
A blank field is no limit of that kind, not zero — zero is a limit of nothing,
which refuses every request. Clearing removes the row rather than writing nulls
across it: both resolve the same, only one tells a reader nobody has set a limit
here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An account is approved into a project, a project belongs to a team, a team to
an organization — and until now the only way to make any of them was `llmgw
keys init` on the control plane. So the first thing to do after installing was
leave the interface you were already in, and an approval form with an empty
dropdown gave no hint why.
The accounts page now creates all three.
Creating a container is not granting access: nobody can use one until an
account is approved into it, which is the step already on this page. Quotas stay
on the CLI. Making somewhere to put people is structural; raising what they may
spend is financial, and a stolen admin session should be able to do the first
and not the second.
Ids are random now instead of slugs of the name, which fixes a bug that only
appears when you have a second organization — which is exactly what this
feature invites. Team ids were slugs in one flat namespace and PutTeam upserts
on id, so creating "platform" under a second organization moved the first one's
team into it. Nothing failed. The team changed owner.
Create* is separate from Put* rather than replacing it. The upserts exist so
`keys init` can be re-run while provisioning a machine; a person filling in a
form means something else by a name that already exists, and gets told.
A duplicate is refused among siblings only. Two companies both having a
"platform" team is ordinary, and refusing the second would be the flat namespace
again wearing a better error message.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Quotas can be written on an organization, a team or a project, and nothing
could show what any of them had used: usage came back per key, per user and per
model only. A quota whose consumption cannot be seen is one nobody reviews
until the day it refuses somebody — and since the previous commit made a group
quota a shared pool, it is now a number several people are spending against
without being able to look at it.
`llmgw ctl usage --by team` sums every key beneath each group, with the budget
that level sets beside it.
Beside it, and only when that level sets one. A team bounded by its
organization's budget has none of its own, and printing the organization's
against each of its teams would read as though each had that much — the exact
misreading this work exists to remove.
Joined through api_keys rather than read from usage.project_id: that column
records where the key was when the request was served, while the quota is
written against the tree. They agree until a key is moved, and a report that
disagrees with the limit shown beside it is the useless one.
Three existing reports were fixed on the way, all failing on PostgreSQL and all
passing in tests, because the tests run on SQLite:
column "k.name" must appear in the GROUP BY clause (UsageSince)
column "us.email" must appear in the GROUP BY clause (UsageByUser)
column "usage.ts" must appear in the GROUP BY clause (UsageBuckets)
SQLite picks an arbitrary row for an ungrouped column and carries on; PostgreSQL
refuses. The first two were selecting names they had not grouped. The third
grouped "ts / ?" while selecting "(ts / ?) * ?", which PostgreSQL matches as
text and so saw a bare ts — it is grouped by ordinal now, the same expression by
construction. The dashboard's usage charts have been empty on the cluster this
whole time, with the reason only in the log.
All four queries were run against the cluster's own PostgreSQL before this was
committed, because passing on SQLite is what they were already doing.
The grouping level picks from a fixed table of levels rather than being
interpolated, so an unknown one is an error and not a fragment of SQL.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two things that were only ever confusing, and one of them stopped a command
working on the cluster.
The store type was named SQLite and kept the name when PostgreSQL was added, so
OpenPostgres returned a *SQLite and all seventy-odd methods on the production
store were spelled after the database it was not running on. Nothing broke;
people did. Earlier today I added three methods to *SQLite for a table that on
this cluster only ever lives in PostgreSQL, and the name did not once look
wrong. It is a field, which is all it ever was.
`llmgw users` and `llmgw keys` defaulted --store to a SQLite path. On a
deployment whose store is PostgreSQL every one of them failed with "unable to
open database file" while the right answer sat in auth.store in the config file
beside them; the conclusion an operator draws is that the tool is broken, and
they are not far off. They now ask the server's own configuration before
falling back, and the error names which of the three sources chose the address
— the failure was never the path, it was which one picked it.
The single-machine default still applies last, so a deployment with no config
file keeps working with no flags.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The tenancy tree handed limits down and never pooled what was spent against
them. A limit written on a team was inherited by every key beneath it, and then
enforced with the key as the counter — so a project given a budget of 1000 and
ten keys permitted ten thousand, and a team given 60 RPM got 60 per member.
That is "each of you may spend this much", which is a useful thing to be able
to say. It is not what somebody writing a budget on a team means, and nothing
in the system said which one you were getting.
Resolution now records which scope set each value that won, and the counter is
keyed by that scope. The value and the counter come from the same place, so a
project tightening its team's limit is enforced against the project's own pool
rather than eating into a quota another project never agreed to.
Each limit is owned separately, because they resolve separately: a team budget
with a per-key rate limit — the group's money, the individual's throughput — is
the ordinary arrangement and has to stay expressible.
Cluster defaults are owned by the key. Read as a shared pool, one default would
become a single cluster-wide rate limit that every key in every organization
competes for: not a default, an outage.
Restoring after a restart had to change with it. Per-key totals rebuilt a
counter named after the key while the check read one named after the team, so
the control plane came back believing the team had spent nothing — a budget
that reset on every restart, for exactly the scopes where it matters most.
SpendSinceByScope returns each key's place in the tree and the accountant folds
it into every scope that might own a budget.
Counters are namespaced by level throughout. A bare key id in one path and a
"key:"-prefixed one in another are two counters for the same key, which is how
the restore was silently doing nothing.
Still missing, and worth naming: no usage report rolls up by project, team or
org, so a shared quota can now be enforced but not yet shown.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`ctl bench` has always been able to work out what one replica can serve — it
ramps concurrency until the SLO breaks and reports the last level that held.
It printed that and nothing kept it, so the one number saying what a replica is
worth lived in whoever ran it's scrollback. Everything downstream then had to
guess: admission applies a single global default for healthy queue depth, and
sizing a model against real demand had nothing to size against.
`ctl bench --save` records it; `ctl capacity` reads it back.
Keyed by model and profile. Capacity belongs to the hardware and the engine
arguments as much as to the weights, so the same model on CPU and on an H100
are two numbers, and one key would overwrite the other on every run.
The profile is looked up from the replica rather than asked for, because the
control plane already knows which profile answers on which endpoint and a
measurement filed under the wrong one is worse than none. A target that is not
a replica — the gateway, usually — is refused with that said: benchmarking the
whole path is worth doing and says nothing about the hardware.
The conclusion is stored, not the ramp. Every level and its latencies are worth
reading once while judging whether the run was sound; keeping them would grow
without bound as a model is re-benchmarked.
Re-measuring replaces. A second run is a better answer to the same question,
not another data point, and keeping both would leave a reader choosing between
numbers with no basis for the choice.
The SLO and the date are stored with it. Capacity means nothing without the
target it was measured against, and hardware and engine builds change — a
measurement is an observation with a date on it, not a fact about the model.
A failed save is reported and not fatal: the run has already cost the GPU time.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
