Commit Graph
13 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 4a4fe8fa16 Pictures are priced by the picture
auth.pricing takes per_image, charged on top of a prompt's tokens; a model
priced only per picture is not free, so a spent budget still stops it.
Each usage row keeps how many pictures were made (migration 0041), and
ctl request and the admin API's request view show them.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-09 21:48:47 +02:00
Ilker ManapandClaude Opus 5.5 249a91aeec Guardrails turn off and on; a judge's refusal says which hazard; ctl guard judge ask
guard.enabled (unset is on) makes every policy check nothing while false,
live: llmgw ctl guard off/on set it in the cluster's settings, and ctl
status, ctl guard list and the guardrails page say when they are off.

A refusal by a judge names the hazard it saw - judge: S6,Specialized
Advice - and the verdict is kept with the request in a new usage column,
guard_rule, shown by ctl request, the request log and the guardrails page.

llmgw ctl guard judge ask has a judge read a text, past the cache, and shows
its raw answer and how it is read; the endpoint is kept out of the audit so
no text is recorded.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-06 12:41:08 +02:00
Ilker ManapandClaude Opus 5.5 44f3a9e31d Reusing answers: a response cache per project, exact match, opt-in with a TTL
llmgw keys cache --project P --ttl 1h [--any] | --off, and Cache answers… on
the console's project menu. The same request — project, endpoint, canonical
model and the body without stream, stream_options, user, metadata, store and
service_tier, keys sorted — is answered from the gateway's memory for the
TTL. Only reproducible answers (temperature 0 or a seed; embeddings, rerank,
score) unless --any; only a whole 200 from the model asked for, not a
fallback's, a canary's, or one a guardrail changed.

Looked up after the limits and the input guardrails; the output policy
judges a reused answer; no_store and stored conversations bypass it.
Streamed answers are kept put back together and streamed back as chunks.
x-llmgw-cache: hit|miss|bypass, Age on a hit, Cache-Control: no-cache or
x-llmgw-cache: no to bypass. A hit is recorded with no tokens and no cost,
marked cache_hit (migration 0027) and shown by ctl request. An LRU bounded
by router.cache.max_bytes and max_entry_bytes; llmgw_cache_requests_total,
llmgw_cache_bytes, llmgw_cache_entries.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 23:50:53 +02:00
Ilker ManapandClaude Opus 5.5 29a7347df2 Requests can be tagged: metadata, x-llmgw-tags and the user field on the usage record, and counted by the tags router.tag_labels names
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 19:45:37 +02:00
Ilker ManapandClaude Opus 5.5 979aa44804 Hook a guard into the router: judge requests before dispatch and filter answers as they are relayed
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 12:20:24 +02:00
Ilker ManapandClaude Opus 5.5 f8572fb5ee Follow one request from its id to the replica that served it
"Why was this request slow?" had no answer. Nothing tied a request to the
replica that served it, the time admission held it went out in a response
header and was never kept, and the only per-request log lines were for
disconnects, early stream ends and upstream retries.

Every request now has an id: the client's X-Request-Id when it is a plain
string of reasonable length, otherwise one the gateway makes. It is set on
the response before anything else, so a request refused as malformed can be
named too, and forwarded to the engine. One access log line per request
carries it with the model asked for and served, the replica, status, time in
the queue, time to the first token, total time, tokens, and the key,
project and user.

Usage records gain the id, the replica, queue time and time to first token,
tried against a restored copy of the live store before being shipped: 591
rows kept, columns and index in place. `llmgw ctl request ID` reads one back
through the admin API.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 02:36:45 +02:00
Ilker ManapandClaude Opus 5.5 9b663eb23c Name people and projects in usage reports, and total a usage legend
The cluster page's usage table showed session:usr-... and proj-... ids;
it now shows the person's email, whether it came through a key or the
chat page, and the project's path or "personal". Times read "3 hours
ago" rather than "3h7m25s ago", everywhere.

The usage page's legend gave each line's latest slice, which was 0
beside a table of hundreds; it gives the window's total, and the table's
numbers are right-aligned like everywhere else.

The PostgreSQL placeholder rewriting skips line comments. An apostrophe
in one opened a string that swallowed every placeholder after it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 22:37:21 +02:00
Ilker ManapandClaude Opus 5.5 098886e0e4 Let a person belong to many places, or to none
A person was in exactly one project, chosen at approval, and could not
use the gateway before that. People belong to several teams and to
none, so:

- memberships: a person is a member of any number of organizations,
  teams and projects, or of none; a member of a team or organization
  may use every project in it. users join/leave, and the people page
  adds and takes out from either side. Approval no longer needs a
  project.
- personal keys: a key may have an owner and no project. /profile asks
  what a new key spends against; keys create --for does the same.
  auth.personal sets default limits for personal use or turns it off.
- limits for people: --scope user holds a person across every key,
  project and the chat page; --per-member on an org, team or project
  gives each member their own share there. Resolution folds the tree,
  each per-member limit for the key's owner, and the owner's own, each
  in its own pool; the quotas page shows and sets both.
- the chat page asks what to spend against and checks it per message.
- usage records who spent it, so a person's budget survives a restart,
  personal use and chat included; restore no longer drops rows with no
  project.

Leaving a project revokes the keys made there unless another membership
still reaches it. A personal key keeps no threads or notes, which are
per project, and is refused rather than dropped into the anonymous pool.

Migration 0010 turns each person's project into a membership, adds
usage.user_id and backfills it, and makes api_keys.project_id nullable.
SQLite cannot drop NOT NULL, so the table is rebuilt; that needs foreign
keys off, which only works outside a transaction, so migrations marked
for it run on one connection with foreign_key_check before commit.
Tested on a populated SQLite store with claims, and on a copy of the
cluster's PostgreSQL store: 5 people became members, 17 keys and 3
claims intact, 145 of 577 usage rows and all 69 chat rows attributed.
The PostgreSQL store tests, including new membership cases, pass on 16.

users move is gone; join and leave replace it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 19:03:46 +02:00
Ilker ManapandClaude Opus 5 f999ee2fb2 Make an organization's budget bound the sum of its teams
CI / test (push) Successful in 1m40s
Resolution took the most restrictive value along the chain and enforced only
that one. For a single caller's ceiling that is right: a team cannot raise what
its organization set. For a pool it is not, because two levels are two pools.

An organization given 1000 with four teams given 500 each spent 2000. Each team
filled its own counter, the organization's value lost to the tighter one and
was never looked at again. The number was written, displayed, reported against
— and enforced by nothing.

Every level that sets a limit is now checked against its own counter, and
settled onto all of them, so an organization's pool fills from its teams'
spending rather than only from requests that somehow named it. A team still
stops before its organization when its own pool runs out first, which is what a
sub-allocation is for.

Effective carries the chain as well as the collapsed values. The collapsed ones
are still what a header or a profile page should show — they are the ceiling a
single caller faces — they are just not enough to enforce with.

Rate limits charge as they check, deliberately, so that two concurrent requests
cannot both see room. With several levels that means a refusal at one has
already taken from the levels before it, so those are handed back. Without the
refund a team sitting at its organization's ceiling would quietly eat its own
rate limit with requests that were never served.

Budgets are checked before any of that, and only read, so a refusal on a budget
charges nothing at all.

Status reports the fullest pool rather than the smallest number. A team inside
an organization with the same budget is bounded by whichever is nearer its
ceiling, and that is the organization as soon as a second team spends anything;
a header showing the team's would disagree with the refusal after it.

A cluster default joins the chain as a key-level entry. Left out, it would be
reported as the ceiling and then never checked.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-27 15:08:30 +02:00
Ilker ManapandClaude Opus 5 3febb6ef63 Count a group's quota against the group, not against each of its keys
The tenancy tree handed limits down and never pooled what was spent against
them. A limit written on a team was inherited by every key beneath it, and then
enforced with the key as the counter — so a project given a budget of 1000 and
ten keys permitted ten thousand, and a team given 60 RPM got 60 per member.

That is "each of you may spend this much", which is a useful thing to be able
to say. It is not what somebody writing a budget on a team means, and nothing
in the system said which one you were getting.

Resolution now records which scope set each value that won, and the counter is
keyed by that scope. The value and the counter come from the same place, so a
project tightening its team's limit is enforced against the project's own pool
rather than eating into a quota another project never agreed to.

Each limit is owned separately, because they resolve separately: a team budget
with a per-key rate limit — the group's money, the individual's throughput — is
the ordinary arrangement and has to stay expressible.

Cluster defaults are owned by the key. Read as a shared pool, one default would
become a single cluster-wide rate limit that every key in every organization
competes for: not a default, an outage.

Restoring after a restart had to change with it. Per-key totals rebuilt a
counter named after the key while the check read one named after the team, so
the control plane came back believing the team had spent nothing — a budget
that reset on every restart, for exactly the scopes where it matters most.
SpendSinceByScope returns each key's place in the tree and the accountant folds
it into every scope that might own a budget.

Counters are namespaced by level throughout. A bare key id in one path and a
"key:"-prefixed one in another are two counters for the same key, which is how
the restore was silently doing nothing.

Still missing, and worth naming: no usage report rolls up by project, team or
org, so a shared quota can now be enforced but not yet shown.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-27 12:10:46 +02:00
Ilker ManapandClaude Opus 5 859f08da9b Approve accounts from the browser, and draw the traffic
CI / test (push) Successful in 1m19s
Two things the dashboard was missing: it could not act, and it was a column of
numbers.

Account administration moves to /admin/users — approving into a project,
suspending, reinstating. That is the work an operator does constantly and it
should not need a terminal. Three things stay on the CLI, and the line is not
arbitrary: promoting someone to admin, resetting a password, and issuing a key
on another person's behalf each turn a stolen admin session into more than it
was — permanent access, or somebody else's account. Approving and suspending
do not. Suspending signs the account out everywhere, or suspension is advisory
until the session happens to expire, and an admin cannot suspend themselves,
which is not a thing an interface should let you do by misclicking.

The charts are inline SVG generated in Go. A charting library would mean a
JavaScript toolchain in CI and a megabyte of bundle for what is a list of
rectangles, and the single-binary goal rules that out. Hourly over a day: fine
enough to show a working day, coarse enough that one slow minute is not an
incident. Empty hours are filled in, because a chart with hours missing where
nothing happened reads as a chart with hours missing from the data.

The SVG is marked as trusted HTML, so everything that reaches it is either a
number this package computed or escaped explicitly, and a bar's colour can only
be one of four names — a colour that could be anything is a way to write CSS
into every page the chart appears on. There are tests for both.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 12:01:42 +02:00
Ilker ManapandClaude Opus 5 1839832b8b Add a landing page and a read-only cluster dashboard
CI / test (push) Successful in 1m2s
Pasting the gateway's address into a browser returned 404, which looks like a
broken endpoint rather than one that tells you how to use it.

Two pages, and the split between them is a security boundary rather than a
layout choice. The landing page is served without a key — it has to be,
because the person reading it is the one who does not have one yet — so it
contains no data at all: it says how to connect, not what is here. Model names
stay behind /v1/models. It reconstructs the address from the forwarded headers
so the copy-pasteable example names the address the client actually used
rather than the internal one; those headers are reflected into displayed text
and never used for a decision.

The dashboard shows nodes, models, replicas, placement failures and usage, and
lives on the admin listener only. That listener is loopback-bound and also
serves pprof, which can dump the process heap — where API keys sit in
plaintext — so reaching the page from elsewhere is an SSH tunnel's job. It is
read-only: anything that mints a credential stays in `llmgw keys`, where it
needs a shell rather than a browser session this project has no way to
authenticate.

Mounting the landing page uncovered a hole in the exemption list. An entry
ending in "/" was a prefix rule, and "/" ends in "/", so exempting the landing
page would have exempted every path there is — authentication still installed,
still logging, and letting the whole API through. "/" is now exact, with a
test enumerating what must and must not be exempt.

Server-rendered templates and one stylesheet, embedded. The single-binary goal
rules out a JavaScript toolchain and nothing on either page needs one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 08:58:05 +02:00
Ilker ManapandClaude Opus 5 29617f8cb1 Add the store: tenancy, keys and usage on SQLite
Phase 5 begins. This is the persistence layer for the things the cluster
cannot reconstruct — who may call, what they are allowed, what they used.
Everything else llmgw knows is still derived from the fleet, which cannot be
wrong about itself the way a database can (§6.5).

SQLite behind an interface. The plan justified that by write volume, which was
the wrong argument: the write path is batched anyway, and a 500-row
transaction every 200ms is nothing for either engine. The real argument is the
failure mode. Postgres is a second service, and a gateway that stops serving
because its accounting database is unreachable is worse than one that buffers
some accounting. Embedded means the store shares the gateway's fate instead of
adding an independent way to be down. That reasoning stops holding at two
instances, where budgets have to be counted somewhere shared — so the
interface is here from the start, the SQL is portable, and migrations are per
dialect with shared version numbers.

Keys are stored as sha256 and nothing else, so a database leak hands over no
working credential. The fast hash is deliberate: the input is 256 random bits
with no dictionary to attack, so argon2 would buy nothing and charge tens of
milliseconds per request. That assumption is written down next to the code
that depends on it.

Two things the schema now says that it did not: budgets are integer
micro-units, because repeated addition of floats drifts; and an empty model
allowlist forbids everything while an absent one permits everything, which the
resolver has to keep distinct.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 08:37:40 +02:00