Two things that were only ever confusing, and one of them stopped a command
working on the cluster.
The store type was named SQLite and kept the name when PostgreSQL was added, so
OpenPostgres returned a *SQLite and all seventy-odd methods on the production
store were spelled after the database it was not running on. Nothing broke;
people did. Earlier today I added three methods to *SQLite for a table that on
this cluster only ever lives in PostgreSQL, and the name did not once look
wrong. It is a field, which is all it ever was.
`llmgw users` and `llmgw keys` defaulted --store to a SQLite path. On a
deployment whose store is PostgreSQL every one of them failed with "unable to
open database file" while the right answer sat in auth.store in the config file
beside them; the conclusion an operator draws is that the tool is broken, and
they are not far off. They now ask the server's own configuration before
falling back, and the error names which of the three sources chose the address
— the failure was never the path, it was which one picked it.
The single-machine default still applies last, so a deployment with no config
file keeps working with no flags.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Keys are stored hashed, so a migration that drops them cannot recreate them:
every person and every script needs a new one issued, and the old ones cannot
even be looked up to say which they were. `llmgw keys move-store` copies
instead.
Found by running it against the real cluster, twice, after the gateway had
already been stopped.
The first order came from the order the migrations create tables in, and
users is created in a later migration than api_keys even though api_keys
references it — so the copy died three tables in. The order is now checked by
a test that derives the foreign keys from the migrations rather than restating
them, so a new reference fails there instead of during somebody's migration.
Checked by putting the wrong order back and watching it fail.
Then key_claims. Its sealed column is emptied the moment a claim is used, so
every claimed row holds an empty blob in a NOT NULL column. An empty blob
scans as a typed nil slice and a genuine NULL as an untyped one — a
distinction this side can see and pgx cannot, since it encodes any nil slice
as NULL. Restored before the insert, which is the difference between a copy
that completes and one that stops four tables in.
One transaction per table rather than one for the whole copy: a single
transaction holding every row of a large usage table is how a migration runs
out of memory on the machine least able to spare it. A failure therefore
leaves a partial copy, which is why the destination must have been empty —
starting again is dropping and recreating it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A path is still SQLite and a postgres:// URL is PostgreSQL, so every existing
configuration keeps working untouched and moving is one line changed rather
than a new setting to find. An address of the form @/path/to/file reads the
real one out of that file, which is how a DSN carrying a password stays out of
a configuration that gets copied, diffed and pasted into issues.
The driver is pgx, which is pure Go. That is not a preference: the whole binary
is built with CGO_ENABLED=0 so it runs on a node whose libc is older than the
builder's, and a cgo driver would take that away from every deployment,
including the ones still on SQLite.
Tested against a real PostgreSQL, which found two things nothing else could.
The schema_migrations table is created in Go rather than in a migration file,
so it escaped the port and still said INTEGER — 64 bits in SQLite, 32 in
PostgreSQL, and a millisecond timestamp does not fit, so the very first
migration failed to record that it had run. And the memory migration uses an
FTS5 virtual table with three triggers keeping it in step, which has no
counterpart: PostgreSQL derives a tsvector from the row, so there is nothing
to keep in step and no trigger to forget.
That leaves exactly one query that is not the same sentence in both dialects —
the note search, MATCH against @@ — and it is written out twice rather than
hidden behind a shared spelling that would mean something subtly different in
each. Everything else is one set of queries.
The text search uses the 'simple' configuration rather than English. These are
an agent's own notes, in whatever language it chose, and stemming English
rules over Turkish would quietly stop finding things.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
SetThreadRetention existed and nothing called it, so the feature was
unreachable: a project could never opt in and every request carrying a thread
was refused.
A command rather than a configuration key, because it is a decision about data
rather than about deployment. Someone has to choose to keep what a team types,
and choosing it should look like an act rather than like a default nobody
read.
--retain-hours is required. Without a number the command would create exactly
the thing the design avoids: a conversation store with no expiry, holding
everything a team ever typed because nobody chose one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Issuing a key worked; getting it to the person it was issued for did not. The
operator had a secret on their terminal and no way to send it that did not
leave a bearer credential in a chat log forever.
`keys create --claim` prints a link instead. GET shows a button and consumes
nothing; POST reveals the key and burns the link. That split is not politeness
— chat clients, mail scanners and link previewers fetch URLs the moment they
are posted, so a GET that consumed the claim would mean the first thing to
unfurl the link destroys the key before anyone reads it.
A pending claim has to hold the key somewhere, which would otherwise put a
working credential in the database and undo what hashing bought. So the key is
sealed with a key derived from the link's own token, and the token is never
stored — only its hash. A database leak yields two useless hashes and a blob
nothing in the database can open. HKDF rather than the raw token, so the value
used for lookup can never become the value that decrypts.
Consuming is one UPDATE moving claimed_at off NULL, so two people opening the
same link produce one key rather than two copies. A used link, an expired one
and a token that was never real all render identically, because saying which
confirms that a token was once valid. The page is no-store, no-referrer and
noindex: the URL is itself the credential.
`GET /me` reports a key's own limits and spend, so nobody has to ask an
operator to read a dashboard they cannot reach. Outside /v1, because it is not
part of the OpenAI API.
docs/onboarding.md is written for whoever operates this, including the parts
that are deliberately absent — no self-service, no SSO, and why.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Until now anything that could reach the client port could use every model in
the catalog. A key now resolves to an identity, and that identity decides what
it may do.
Authentication wraps the whole mux rather than checking inside each handler,
so adding a route cannot add an unauthenticated one by omission. Health stays
open, because a load balancer has no key and a health check that needs one
reports the gateway down.
Nothing on the request path reads the database. A key is resolved once and
answered from memory until the entry expires; rejections are cached too, so a
client looping with a bad key cannot turn into a database read per request.
The price is that a revocation takes up to cache_ttl to take effect, which is
the trade stated in the configuration next to the knob. A store failure is
not an authentication answer: it is never cached as one and returns 503, not
401 — sending someone hunting for a key problem that does not exist wastes the
one thing an incident is short of. A revoked key is refused with the same
message as an unknown one, because saying otherwise confirms the key was real.
With auth on the band comes from the key and the band header is ignored. The
header was trusted only while there was nothing to consult, and it let any
client promote itself into the top priority band. Tenant fairness within a
band is now per key rather than per session, so one caller cannot take a band
by opening many sessions.
Allowlists are decided per model, not per string: a model's canonical name and
every alias name the same thing, so either form works in an allowlist and no
alias can reach a model that was not allowed. The first version of that test
passed without the resolution being there at all, which is why it now checks
all four combinations.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
deniz