bf2d171e8941bbff7cbbfba078a5ceb18c388f9e
37
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
daed835d88 |
Measure each node's CPU and memory, and give every node a page
The nodes are CPU machines, and nothing measured how busy one was: the heartbeat carried GPU memory and disk, and llmgw_node_memory_bytes is the total. The agent now reads /proc between heartbeats - CPU across all cores, load averages, memory used and available, and each engine's process group's cores and resident memory - and sends it with the heartbeat. Nothing new runs on the nodes and Prometheus needs no new target: the control plane exports it with the rest of the fleet, as llmgw_node_cpu_utilization_percent, llmgw_node_cpus, llmgw_node_load, llmgw_node_memory_used_bytes, llmgw_node_memory_available_bytes, llmgw_shard_cpu_cores and llmgw_shard_memory_bytes. A measurement more than a minute old is left out rather than repeated. The cluster page's node table shows CPU and RAM as bars, and each node links to a page of its own: current CPU, RAM, load and disk, the engines on it with their cores and memory, and an hour of CPU, RAM, load, per-model cores and memory and queue depth from Prometheus. It refreshes in place. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
5aa5675a9c |
Show a model loading in the chat, and send once it is up
A message to a model that was not running waited 45 seconds and came back as "the model is loading; retry shortly". The page now asks for the model with /chat/wake, shows where it stands from /chat/model-status - waiting for a node, downloading with bytes and a percentage, starting the engine - and sends the message when it answers. Also passes the nodes' llama.cpp build to the pages, which the mirror's checks for loadable weights needed and were never given. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
e493b2f53d |
Give an organization's managers a page of their own
A member of an organization can be its manager (users join --org X --role manager, or the star on the people page). /org shows a manager their organization: usage over time by team, project or person, its teams and projects with limits and 30-day spend, and its members, each once with every place they belong. Read-only; administrators see every organization. Only what was spent in the organization's projects is shown — UsageBucketsIn filters by where a request was spent. Found on the way: - Usage by org, team and project, and the quotas page's spend, joined through api_keys and so dropped every chat-page request, which has no key row. They now read usage.project_id; keys no longer move between projects, so the two agree. - /org was not exempt from API-key auth and answered browsers with a JSON error. A router test now drives the session pages through the middleware; it fails without the fix. Membership queries, org-scoped usage and the reworked joins pass the PostgreSQL store tests on 16. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
098886e0e4 |
Let a person belong to many places, or to none
A person was in exactly one project, chosen at approval, and could not use the gateway before that. People belong to several teams and to none, so: - memberships: a person is a member of any number of organizations, teams and projects, or of none; a member of a team or organization may use every project in it. users join/leave, and the people page adds and takes out from either side. Approval no longer needs a project. - personal keys: a key may have an owner and no project. /profile asks what a new key spends against; keys create --for does the same. auth.personal sets default limits for personal use or turns it off. - limits for people: --scope user holds a person across every key, project and the chat page; --per-member on an org, team or project gives each member their own share there. Resolution folds the tree, each per-member limit for the key's owner, and the owner's own, each in its own pool; the quotas page shows and sets both. - the chat page asks what to spend against and checks it per message. - usage records who spent it, so a person's budget survives a restart, personal use and chat included; restore no longer drops rows with no project. Leaving a project revokes the keys made there unless another membership still reaches it. A personal key keeps no threads or notes, which are per project, and is refused rather than dropped into the anonymous pool. Migration 0010 turns each person's project into a membership, adds usage.user_id and backfills it, and makes api_keys.project_id nullable. SQLite cannot drop NOT NULL, so the table is rebuilt; that needs foreign keys off, which only works outside a transaction, so migrations marked for it run on one connection with foreign_key_check before commit. Tested on a populated SQLite store with claims, and on a copy of the cluster's PostgreSQL store: 5 people became members, 17 keys and 3 claims intact, 145 of 577 usage rows and all 69 chat rows attributed. The PostgreSQL store tests, including new membership cases, pass on 16. users move is gone; join and leave replace it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
33e678ab84 |
Show a node's logs on the admin page
Seeing what a node did meant ssh through a jump host, to a machine whose address nobody remembers, to read a journal only root may open. When an engine install failed on a new node today, the reason was one line — and it was four steps away from the page that said the install had failed. Each node's row on the cluster page now links to its logs: the agent's own, and every engine running there, chosen from the list the node itself sends. The page can follow, refreshing every five seconds, and is on both the admin listener and /admin. `llmgw ctl logs NODE [--source SHARD] [-n N]` reads the same. They travel over the stream the agent already holds, so they need nothing the control plane does not already have: a request goes down, the lines come back, and a reply that does not come in ten seconds is an agent too old to answer. The agent keeps its last 2000 lines in memory as well as writing them wherever it did before; the engines' last few hundred lines were already kept and sent only when one crashed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
07809490e3 |
Show usage over time, per organization, team, project and person
CI / test (push) Has been cancelled
Usage could be read as a total — per key, per user, per model, per group — and the cluster's own charts covered the fleet's hardware. What nobody could see was when a group used it. "Research doubled its traffic three weeks ago" is not a question a column of totals can be asked, and it is the question somebody asks before a quota conversation. /admin/usage draws requests, tokens and spend over a chosen window, grouped by any level of the tenancy tree or by person. Person is not a level of that tree — one person holds keys in several projects — so the grouping is its own type rather than store.Scope with a fourth value nothing else could use. Three things learned from the charts already here. Buckets are gap-filled per group, because a line drawn straight across an absence claims traffic that did not happen, and because the chart sums its tail by timestamp and can only do that if every series shares a clock. Series are ranked by total rather than by peak — a counter's question is "who used the most", and ranking by peak would colour the team that made one burst over the team that ran steadily all week. And the spend chart is replaced by a sentence when no model has a price, rather than drawn flat along the bottom. The bucket query groups by ordinal and was planned against the cluster's PostgreSQL before being believed, with a test alongside the others that run there: three reports have shipped working on SQLite and failing on it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
4535bf362f |
Price a request by the model that serves it
CI / test (push) Successful in 1m38s
The price table is written against the catalog. The name reaching the accountant was the one the caller typed, so every request made by an alias missed the table and fell to the default price — the same request charged two different rates depending on which of a model's names the client happened to be given. With no default price that is zero, which is silent. With one it is a wrong number, which is worse: a budget spent at the rate of whatever the default says rather than at the model's own. The router already resolves the alias, two lines above, to decide whether the key is allowed the model at all. It now passes the resolved name to the accountant as well, in the ordinary path and in the budget fallback. The test charges a real request through an aliased name and compares the figure with arithmetic it states itself, because the first version of it asserted "more than the fallback" and passed with the bug in place. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
832d0a652c |
Say when a budget cannot be spent against
CI / test (push) Successful in 1m45s
A team was given a budget of a million, the quota page showed it, and after sixty requests it read zero percent used. Not because the counting was wrong: cost comes from a price table, no model had a price, and every request therefore cost nothing. The number was a limit in appearance and bound on nothing. An empty price table is the ordinary case — self-hosted models have no per-token price, which the schema already says. Showing somebody a budget without mentioning it is not. The page says so, and says it precisely: rate and token limits are unaffected, because they count requests and tokens rather than money. A warning that read as "no quota works here" would be worse than the silence it replaces — those two were enforcing perfectly well, as twenty served and ten refused had just shown. AnythingPriced is on the accountant rather than the price table, because what a caller is asking is whether the budgets it is about to display mean anything, and the accountant is what would enforce them. A model priced at zero is not priced; a fallback price prices everything. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
c6df0659af |
Put the cluster's own charts on its own admin page
CI / test (push) Successful in 1m50s
The dashboard charted what the cluster had been billed for — requests, tokens, spend, all from llmgw's own store — and nothing about whether it was healthy. Saturation, queue depth, cache occupancy and replica counts were numbers in a table at the instant the page loaded, with no way to see that one of them had been climbing for an hour. Four line charts now, read back from wherever the metrics were scraped to. Read back rather than kept twice. llmgw publishes these already; what a gauge cannot do is remember, and a ring buffer in this process would be a rival copy of the same numbers with its own retention, lost on every restart and disagreeing with the one an operator has on screen elsewhere. monitoring.prometheus is empty by default and the panels are then absent rather than empty. One axis per chart, always. Two measures of different scale belong in two charts: a second y-axis lets any two lines be made to cross wherever the author likes and the reader cannot tell it happened. Four colours, handed out in fixed order, never cycled. A fifth series is summed into "other" and drawn in grey — dropping it would make a chart that does not add up, and inventing a hue for it would put two lines on the page that nobody can tell apart. The fold is by timestamp, not by position: the tail's series do not share sample times, and adding them by index slides points sideways onto each other's clock. A legend for every chart with more than one line, because colour alone is not an encoding a reader can rely on. A flat zero is drawn along the bottom rather than left blank: nothing happening is a real answer, and an empty frame reads as a broken chart. A single sample is not a line and says so instead of putting a dot in the corner. One failed query fails all four panels. Three drawn and one blank invites reading the blank one as "nothing is happening there". NaN is dropped rather than plotted. Prometheus sends it as a string for a sample it has no value for, Go's ParseFloat accepts it happily, and one NaN coordinate is an SVG path that does not draw at all — taking the rest of the line with it. Axis maxima are rounded to a number a person would pick. An axis topping out at 0.8347 makes every gridline unreadable, and the shape is what a chart is for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
d998ea7ef5 |
Let a model say what it is for
CI / test (push) Successful in 1m42s
A gateway serving nine models offered nine names and nothing to choose between them. /v1/models is the only view most clients ever have, and it listed ids. Models carry a description now: written when the model is added, shown on the dashboard and sent to clients in /v1/models. Written rather than derived. A registry offers a pipeline tag and a wall of README, and neither answers the question somebody picking from a list is asking — which of these should I send this request to. "General chat, strong Swedish; slower than the 8B" is not in any metadata field. It is optional, and absent means absent rather than an empty string in the response: a model nobody described should read as undescribed, not as described with nothing. On the dashboard it sits under its row rather than in a column. A sentence in a table cell either wraps every other column into a stripe or gets cut off. Co-Authored-By: Claude Opus 5 (1M context) <moreply@anthropic.com> |
||
|
|
e034862b79 |
Serve a mirrored model from the list, instead of writing YAML for it
CI / test (push) Successful in 1m35s
Fetching a model put it on the mirror and left the operator to hand-write a catalogue entry and run `ctl model add` on the control plane — for a source the page was already displaying, with an engine decided by the file extension it had just shown them. A present row now offers to serve it. The form opens on that row with the name, the mirror's own URL and the engine filled in, and asks for the two things nobody can infer: the context pool and how many replicas. What it refuses is the point. A version of several files is refused with the reason. A catalogue entry names one source, so an entry for a sharded model is an entry for one shard, which loads as a broken model rather than failing to load. Nothing said so before — the mirror happily holds a four-shard model that no catalogue entry can describe. Safetensors on a fleet with no GPU is warned about before anything is added. This is the failure we hit today: fifteen gigabytes of shards arrived, and only then did looking at a node show no card, no vLLM and no SGLang. The control plane is not told which engines a node has installed, so the check is on cards, which it does know — a warning rather than a refusal, because a card can be added tomorrow. A name already declared in configuration is refused here too, matching what the model service would say anyway, but before the form is filled in rather than after. Without a store there is no serve action at all: a model added at runtime would not survive a restart, and offering it then refusing is worse than not offering. The mirror page's dependencies became a struct. Three are optional interfaces and two are the same shape, which at a call site is a swap nobody notices until a page shows the wrong thing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
ed530133b6 |
Find a model in the registry instead of pasting a URL into a box
CI / test (push) Successful in 1m41s
Mirroring anything meant knowing a repository existed, knowing which of its thirty files were the weights, and assembling a resolve URL by hand. The mirror page had a text box for that URL and no way to find one. It now searches Hugging Face and lists a repository's weight files with their sizes and published checksums, each with a button that fills the fetch form. Only weight files. A repository is mostly a README and a tokenizer, and listing those buries the four entries somebody came for. The size shown is the LFS size, not the plain one. For an LFS-stored file — every weights file — the plain field is the size of the pointer, so the other one reports a gigabyte model as 133 bytes, and that is the number somebody would then plan disk against. The source URL is pinned to the commit the listing came from rather than to a branch, so what the mirror fetches is what was on screen. A repository re-uploaded under the same branch otherwise leaves two different sets of weights with identical provenance and no way to tell them apart. The version defaults to that revision for the same reason. The registry's own failures are reported on their own line. It is somebody else's server on the other side of the internet, and it being down says nothing about the mirror — which is what the page is for and has to keep working regardless. With no registry configured there is no search box at all, rather than one that can only time out. Gating decodes from a bool or a string, because the field is documented as the first and is the second in practice: strict decoding fails the whole response, so one gated model would hide the other nineteen in a page of results. And a bug this made obvious: downloads were keyed by model and version, so the second shard of a sharded model was refused as "already fetching" until the first had finished. A fifteen-shard model had to be pulled one file at a time, each waiting on the last. Keyed by file now — which is what the single-flight was ever protecting, since the race is two writers on one .part. A version made of several files reports as the worst of them: it is not present until every shard is, and one shard failing makes the version unusable however many others arrived. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
55262418c9 |
Put quotas somewhere they can be seen and changed
CI / test (push) Successful in 1m35s
A quota could be enforced and reported, and set nowhere but a shell. Nothing in the browser showed one at all — not what a team may spend, not what it has spent, not that it had no limit. In practice that means quotas nobody sets. /admin/quotas lists the whole tree, whether or not a group has a limit: the one without a quota is the one most likely to need one, so it cannot be the row the page leaves out. Spend over the last thirty days sits beside the budget, with the fraction used. Setting a quota here is a change of position. It was kept to the shell because raising a budget is a financial act and a stolen admin session should not perform one. That argument holds for exactly one row on this page: resolution takes the most restrictive value along the chain, so raising a team's or a project's limit cannot exceed what the organization allows. Only an organization's own limit is bounded by nothing. Every change is logged with who made it and what it was before. Per-key limits stay on the CLI. A key belongs to a person and is already bounded by its project; letting an admin session reach individual keys would put every caller's throughput one click away, and the page refuses a key scope rather than rendering it and failing later. A limit an ancestor already caps lower is marked as written-but-unreachable. Resolution means a project given more than its team is not a bigger budget, it is a number nobody will reach — and without saying so the page would show it as though it applied, with the first sign being a refusal that disagrees with the screen. The group is chosen, not typed. Ids became random in the previous commit so two organizations could both have a "platform" team, which also made them impossible to remember and a typo impossible to notice: it would write a quota onto nothing and report success. A blank field is no limit of that kind, not zero — zero is a limit of nothing, which refuses every request. Clearing removes the row rather than writing nulls across it: both resolve the same, only one tells a reader nobody has set a limit here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
cd7eedf09e |
Give the mirror a page, so managing it is not a curl command
CI / test (push) Successful in 1m49s
/admin/mirror, behind the same admin session as the other admin pages. What is held, what is arriving with its byte count, what failed and why; start a fetch, retry a failed one, remove a version. Not mounted at all without a mirror configured. The right page for a deployment with no mirror is no page, rather than an empty one pointing at nothing. It reloads itself only while something is downloading. A reload at any other moment takes with it whatever is half-typed into the fetch form, and there is nothing to watch when nothing is moving. A failed row offers Retry, not Remove. A fetch that failed left nothing on disk, so the remove could only answer "the mirror holds nothing for this" — a button whose single outcome is a confusing error. Starting the same download again is what somebody looking at that row wants, and the mirror allows it: only a download still running is refused. An unreachable mirror is said outright instead of rendering as an empty table. The two look nearly identical and mean opposite things: one is a mirror waiting to be filled, the other is a machine to go and look at. Dashboard takes NavExtras rather than a second bool. Two adjacent booleans at a call site get swapped silently and are discovered by somebody clicking the wrong link. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
2bf7de1a43 |
Give two control planes a shared address, and a probe that means something
CI / test (push) Successful in 1m36s
The last piece of HA, and it is a deployment pattern rather than code: llmgw picks one leader and agents follow it, but client traffic arrives in front of both and llmgw is not in front of anything. Writing the pattern found a bug in the pattern. The first draft said to health-check `/health` and claimed a standby still routes to replicas the leader placed, so sending it traffic was fine. It does not. A standby's reconciler places nothing and it holds no agent streams to adopt from, so it publishes an empty endpoint list — a load balancer following that advice would have sent half the traffic to something that answers every request with 503. So there is now a `/ready` that means "this process has somewhere to send a request": 503 on a standby, 503 on a leader whose replicas have all failed, 200 when there is something to serve. `/health` keeps meaning the process is up, which is a different and less useful question. Both are open, because a load balancer has no key. There is a gap during failover where neither passes — three seconds when the leader stops cleanly, up to the lease TTL when it does not — and the document says so rather than implying otherwise. Traffic cannot go to a control plane that has nothing to route to. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
2eb79f3400 |
agenta memory eklendi
CI / test (pull_request) Successful in 2m31s
|
||
|
|
d26a30471e |
Let people forget a conversation, and make retention actually expire things
CI / test (push) Successful in 1m38s
Two halves of keeping the promise that storing makes.
DELETE /v1/threads/{id} is a client endpoint, not an admin one: the person who
said the thing should be able to unsay it, and making them ask an operator
would mean the operator reads it first. Deleting a thread in another project
answers exactly as deleting one that never existed does — two answers would
let a caller map which ids are real in projects they cannot read, and this one
would do it by trying to destroy them.
And something now runs the pruner. Without it "we keep conversations for 24
hours" was a comment rather than a promise: the rows would accumulate and the
number a project chose would describe nothing. Hourly, and once at startup,
because a gateway that has been down for a week should not wait another hour
before honouring a window that expired while it was off.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
|
9eb321ae3e |
Let a conversation continue on a different model than it started on
CI / test (push) Has been cancelled
The request path half. A caller sets x-llmgw-thread and sends only the new turn; llmgw puts the stored history in front of it, sends that to whichever model is serving, and stores the exchange afterwards. A header rather than a body field because some SDKs drop fields they do not know and all of them let a caller set a header. Capturing the reply is opt-in and bounded. This is the hottest path in the system, so a request without a thread accumulates nothing at all, and one with a thread stops accumulating at a cap rather than growing to it — a gateway that kept every reply in memory would be one long answer away from trouble. A reply is truncated on a rune boundary rather than dropped: a shortened turn replays as a shortened turn, while nothing at all replays as the assistant having said nothing, and invalid UTF-8 would break the next turn's body for good rather than for one turn. The streamed path sets the captured reply on every exit, not only the one that looks like the end. A stream can also finish with the client gone or the engine stalled, and what the assistant said up to that point is still what it said. A thread on a project that keeps no conversations is refused. The e2e test asserted that and got a cheerful 200: the request was served, nothing was stored, and the caller would have discovered it only when the model had no idea what they were referring to — which reads as the model being broken. It now asks whether storage is on before doing anything else. History is spliced after the model is finally settled, because a budget fallback may have changed it and trimming is measured against the context of the model that will actually serve. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
28967edab3 |
Serve a spent budget something it can afford
CI / test (push) Successful in 1m38s
The accounting layer has always exempted a model that costs nothing, with a comment saying it is "what lets a key with no budget left still be served a cheap fallback". That described a caller nobody had written: the budget was checked once, against the requested model, and a refusal was the end of it, so the exemption could never fire. A caller that has run out is now offered the first model in its fallbacks it can still afford. The allowlist still applies — running out of money is not a way to reach a model a key was never allowed — and a model with no fallback is refused exactly as before, or a budget would stop meaning anything. Only a spent budget falls back: a rate limit is answered by waiting a second, and moving the request elsewhere would spend someone else's capacity to avoid a delay nobody asked to avoid. Which uncovered a header that has never worked. x-llmgw-served-model was set in finish, and finish runs after the proxy has written the response — so the one mechanism for telling a client that something else answered has been announcing it to nobody since it was written. Nothing failed, because the only thing asserting it was a comment. It is now set before dispatch, for the capacity fallback as well as this one. Finding that took bisecting a test I had written wrong: it polled for a 429 that never came, because the budget had run out and the fallback had served the request — the feature working, reported as the feature failing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
7fe59bd5fb |
Load a model when someone asks for it, and stop it when nobody does
CI / test (push) Successful in 1m29s
The feature the idle tracking was for. A cluster with more models than GPUs can now hold every model in its catalog and only the wanted ones in its cards: min 0 with a max above it means load on demand, and once nobody has asked for idle_timeout the replicas stop and the cards go back. No new field, because min 0 max 1 already reads correctly — keep none running, allow up to one. Which made min 0 max 0 meaningful for the first time: configured and deliberately not running. normalize() used to fill a zero max in, on the reasonable ground that a model with no maximum cannot run; that would now silently turn every switched-off model into an on-demand one, including the ones already written that way in this cluster's own configuration. A model with no scaling stanza stays off for the same reason — defaulting it to on-demand would let any client request start a 70B nobody intended to run. The one real decision is what happens to the request that triggers a load. Refusing makes the feature useless, since the first request for every model fails; waiting without a bound turns a request into a job, and a client that budgeted thirty seconds holds a connection for ten minutes. So it waits briefly and then says what is happening — 503 with Retry-After and "the model is loading", which is a different answer from "no endpoint serves this" and produces different client behaviour. An unknown model still gets the unknown-model error, or on-demand would turn every typo into "loading" and the useful message would disappear. Use keeps a model alive rather than only the request that started it, without which the timer measures time since the load and a busy model is evicted on schedule however hard it is working. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
1c57a4e378 |
Let the catalog change while the cluster is running
CI / test (push) Successful in 1m29s
Adding a model was an operator procedure with downtime: edit a file on the control plane, restart it, wait out the settling window, watch replicas churn while adoption catches up. I did that eight times in a day, and one of the restarts cost two minutes of outage while stalled replicas timed out. None of it was necessary. The reconciler already compares desired against actual on every pass, so a new catalog is simply a different desired. A Catalog stays immutable — New is the only way to build one and it refuses an invalid one — and what changes is which immutable catalog is current. Readers take the pointer and finish their operation against it, so a swap mid-request is invisible rather than a race. The router is told rather than asked, the same shape as the endpoint list and for the same reason: a push happens once per change, a pull would happen once per request. The router still seeds from configuration as well as being pushed to. A router with a static endpoint list and no control plane is a supported mode and has nothing to push it a catalog; without seeding it would resolve no model at all. Which turned up a bug that was latent while catalogs were fixed. reconcileModel only ever visits models that are in the catalog, so a replica whose model had been removed was visited by nothing: it kept its GPUs, kept counting, and kept running an engine for a name no client could ask for. The endpoint publisher skipped such a replica with the comment "the reconciler will tear it down", which was not true. It is now, and the comment says what actually happens instead of what was assumed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
e9fe3ad379 |
Know how big a model's context is, and what it can do
CI / test (push) Successful in 1m32s
Three things the gateway did not know about its own models, all of which surfaced this week as somebody else's bug. Context length. Nothing read it, /v1/models did not publish it, and an over-long prompt was not refused: it was admitted, scheduled, given a replica and a slot, and rejected by the engine in whatever shape that engine chose. A client sizing prompts from /v1/models had nothing to read. It is now declared per model, published when known, and checked before authentication work or a queue slot is spent — with a ten percent margin, because the estimate is bytes over four and being a few percent out must not turn a valid request into a refusal. Capabilities. The gateway forwards a tools array to any model, and one that cannot use them answers in prose; every OpenAI-compatible client then behaves as though the model is broken, and the only way to find out was to try it. Declared rather than discovered, because no engine will say — which is why these are claims, and why the next commit makes engine-check verify the tool-calling one instead of trusting it. And a bug found while wiring the first: the token estimate was taken from the routing prefix. That prefix stops at 2 KB by design, so every prompt above it estimated as 512 tokens however long it really was, and TPM limits undercounted exactly the requests that consume the most. Prefix routing wants a bounded head of the conversation; a token estimate wants all of it. They are now separate functions, and structured multimodal content counts rather than measuring zero. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
082d52bc97 |
Stop shipping a placeholder that looks like a value, and say what is served
CI / test (push) Successful in 1m23s
A user copied the curl from the profile page and sent it as it stood. Every other field in that snippet is filled in for them — the base URL, the key they had just been issued — so "model": "MODEL" reads as a value rather than a blank, and they got: no endpoint serves "MODEL" which is true and useless. It reads the same whether the name was a typo, a placeholder left in from the documentation, or a real outage, and it never says what would have worked. The message now lists what is being served, capped at ten with a count for the rest, and says plainly when nothing is. That list is what /v1/models already returns to the same caller, so it discloses nothing new. The profile page's example now names a model that is actually running, real ones before simulated — the point of an example someone can run is that it runs, and sending a first request to the sim engine would answer in Greek letters. The landing and claim pages keep a placeholder, because they are served without a key and deliberately say nothing about what is here; theirs is <model-name>, which cannot be mistaken for a value, with a line saying where to get the real one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
72b03a1a1c |
Say which model is simulated instead of letting it look broken
CI / test (push) Successful in 1m31s
big-model runs on the sim engine, whose whole vocabulary is the Greek alphabet, cycled at a fixed rate until max_tokens. That is what it is for: with no H100s it is the only thing exercising the TP8 two-node path, the fabric invariant and rebuild-after-node-loss. The output is the point. What was wrong is that nothing said so. /v1/models returned it with owned_by "llmgw", identical to the real model, and the chat picker sorted purely by name — so "big-model" led the list, was preselected by the browser, and anyone who typed a question and pressed Send got Greek letters back with no explanation. Reported as the model spitting out crap, which is a fair reading of what it looks like from outside. Simulated is derived from the profiles rather than configured: an operator who adds a sim profile and forgets a flag would be shipping placeholder text under a real model's name. A model counts as simulated only when every profile is, because one real profile means the request can still come back as inference. The marker on /v1/models is owned_by "llmgw-sim", which is already what the sim engine calls itself on its own /v1/models — the gateway was flattening a distinction that existed one layer down. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
3b67b1fe42 |
Stop the dashboard linking to pages that are not where it is
CI / test (push) Successful in 1m25s
Clicking "metrics" on the web dashboard answered "no api key was provided". The page is served twice and its navigation assumed one of them. Metrics, pprof and the management API exist on the admin listener; account administration, the profile and the chat exist on the client-facing port. The nav offered the first set in both places, so on the public port those links fell through to the API-key middleware — which is a far more confusing way to say "that is not here" than a 404 would have been. Each page now offers what is actually reachable from it, and the public one says where the rest lives rather than pretending it does not exist, including why that port is not exposed: pprof can dump the process heap, and the heap holds every API key in plaintext. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
de21649ad6 |
Add an in-browser chat, without putting a key in the browser
CI / test (push) Successful in 1m13s
So someone can try a model without setting up a client, and — more often — tell whether a problem is the gateway or their code. The page carries a session, and the server turns that into the identity an API key would have produced, resolved from the person's project. The request then goes down the ordinary path by being handed to the same handler /v1 uses: admission, model allowlist, rate limits, budget, accounting. There is exactly one request path, so there is no second place for its rules to drift — and without that, a playground is a hole in every limit the rest of the gateway enforces. The allowlist test posts a forbidden model and expects 403 for that reason. The body is rebuilt rather than forwarded, so the page cannot set fields it has no business setting: a stream:false that would hang the reader, a max_tokens large enough to occupy a replica for an hour. One turn is capped at 512 tokens. The picker offers what is actually served and allowed, not what is catalogued, because offering a model with no replica produces a 503 the reader cannot interpret. Inline vanilla JavaScript, for the same reason the charts are hand-drawn SVG. It streams, because on CPU inference ten seconds is normal and a chat that hangs and then dumps a paragraph feels broken. Replies are written with textContent throughout: model output and user input have no business being parsed as markup. Its POSTs carry the CSRF token in a header, which a cross-site form cannot set at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
859f08da9b |
Approve accounts from the browser, and draw the traffic
CI / test (push) Successful in 1m19s
Two things the dashboard was missing: it could not act, and it was a column of numbers. Account administration moves to /admin/users — approving into a project, suspending, reinstating. That is the work an operator does constantly and it should not need a terminal. Three things stay on the CLI, and the line is not arbitrary: promoting someone to admin, resetting a password, and issuing a key on another person's behalf each turn a stolen admin session into more than it was — permanent access, or somebody else's account. Approving and suspending do not. Suspending signs the account out everywhere, or suspension is advisory until the session happens to expire, and an admin cannot suspend themselves, which is not a thing an interface should let you do by misclicking. The charts are inline SVG generated in Go. A charting library would mean a JavaScript toolchain in CI and a megabyte of bundle for what is a list of rectangles, and the single-binary goal rules that out. Hourly over a day: fine enough to show a working day, coarse enough that one slow minute is not an incident. Empty hours are filled in, because a chart with hours missing where nothing happened reads as a chart with hours missing from the data. The SVG is marked as trusted HTML, so everything that reaches it is either a number this package computed or escaped explicitly, and a bar's colour can only be one of four names — a colour that could be anything is a way to write CSS into every page the chart appears on. There are tests for both. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
e519dc587c |
Serve the cluster dashboard over the web, without serving pprof with it
CI / test (push) Successful in 1m21s
Reaching it meant an SSH tunnel, because the page lived on the admin listener and that listener is bound to loopback. Opening the port was never the way to fix that: it also serves pprof, and pprof can dump the process heap, where every API key sits in plaintext. Exposing the port would have been exposing the key store. So the page moves rather than the port. The same dashboard is now mounted at /admin on the client-facing port, behind the session and CSRF machinery that already guards /profile, gated on a flag on the account. The admin listener keeps pprof and metrics and stays on loopback. One bit, not a role system: it grants looking. Everything that changes the cluster or issues a credential stays on the CLI, where it needs a shell rather than a session — which keeps the worst case of a stolen admin cookie to "saw the cluster" rather than "drained your fleet". Someone without the flag gets 404, not 403. A 403 confirms the page is there and that admin accounts exist here, which is free reconnaissance; 404 is what every other unknown path returns. The profile page does not show a link to a door the visitor cannot open. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
79e459d9ff |
Let people change their own password, and operators reset a lost one
CI / test (push) Failing after 1m11s
The account system shipped without either, which made the first forgotten password an unanswerable support request. Changing it requires the current one. A session is what an attacker gets from a borrowed laptop or a stolen cookie; the password is what they do not have, and without the check the first thing they would do is take the account. Every session is closed, including the browser doing the changing — leaving the others open would mean a password change does nothing about the reason someone is changing it. A new session is then opened for the browser in front of us, so the person who just proved their password is not thrown back to the sign-in page. That response also carries a new CSRF cookie, so the forms it renders take the new token rather than the one the request arrived with; there is a test that posts a form from that exact page. Guessing the current password is bounded the same way sign-in is, because someone holding a stolen session should not get unlimited attempts at the password that would let them keep it. There is no self-service reset: this gateway sends no email, and a reset link with nowhere to send it is not a reset. `llmgw users reset-password` is the operator's answer, and it closes their sessions too — if someone lost their password, the reason might be that somebody else has it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
216fc9bb03 |
Let people sign up, be approved, and issue their own keys
CI / test (push) Successful in 1m9s
Every key needed an operator with a shell. This adds the login system that made that necessary, which is the thing I argued against building until it was asked for — so it is built as if it will be attacked, because a public login form is. Signing up produces an account that can do nothing. Approval is what makes it work and stays on the CLI: having a shell on the control plane is what makes someone an administrator, and that is a smaller thing to get right than roles and an admin UI that mints credentials. The details that matter: Passwords are argon2id, where a slow hash is correct — the input is chosen by a human and has a dictionary, which is the opposite of an API key. Both kinds of authentication now live in this codebase and they are in separate packages so neither set of assumptions leaks into the other. Length is the only rule: composition rules push people towards Password1! and cost more entropy than they buy. Signup never says whether an address already has an account, and sign-in answers the same way for a wrong password and a missing account — verifying against a dummy hash when there is no user, so a stopwatch cannot enumerate accounts. Account state is checked after the password, not before, so guessing an address reveals nothing about it. Guessing is bounded per address rather than per IP: an attacker has many addresses to come from and one to guess against, and limiting by IP locks out an office behind one NAT. Sessions are server-side, so signing out and suspending take effect immediately rather than whenever a token expires; suspension is checked in the same query that resolves the session, because a second round trip is a check somebody eventually forgets. Every state-changing form carries a CSRF token verified against a hash in the session — a cookie authenticates the browser, not the page that asked. Key ownership is enforced in the UPDATE rather than by reading first, so a guessed key id from another account changes nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
8d35faf6f8 |
Give the landing page somewhere to send people
CI / test (push) Successful in 56s
The page explained the API, said keys come from an operator, and left the reader there. Someone who reaches the address without a key had no route to getting one, which is where onboarding stopped. `landing.contact` is that route, because the gateway cannot guess who to ask. An address becomes a mailto link, an http(s) URL stays a link, and anything else — a channel, a queue, a person — is shown as text. Other schemes are never linked: this string comes from configuration, configuration is not always written by whoever runs the gateway, and a javascript: href on a page served to every visitor is a short path to a stored XSS. Escaping the text does not make a scheme safe. Unconfigured, it now says something rather than nothing, and the page explains that a key arrives as a one-time link so the reader knows what to expect before it lands. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
3ed211b2f4 |
Onboard users with one-time links instead of pasted keys
CI / test (push) Successful in 2m43s
Issuing a key worked; getting it to the person it was issued for did not. The operator had a secret on their terminal and no way to send it that did not leave a bearer credential in a chat log forever. `keys create --claim` prints a link instead. GET shows a button and consumes nothing; POST reveals the key and burns the link. That split is not politeness — chat clients, mail scanners and link previewers fetch URLs the moment they are posted, so a GET that consumed the claim would mean the first thing to unfurl the link destroys the key before anyone reads it. A pending claim has to hold the key somewhere, which would otherwise put a working credential in the database and undo what hashing bought. So the key is sealed with a key derived from the link's own token, and the token is never stored — only its hash. A database leak yields two useless hashes and a blob nothing in the database can open. HKDF rather than the raw token, so the value used for lookup can never become the value that decrypts. Consuming is one UPDATE moving claimed_at off NULL, so two people opening the same link produce one key rather than two copies. A used link, an expired one and a token that was never real all render identically, because saying which confirms that a token was once valid. The page is no-store, no-referrer and noindex: the URL is itself the credential. `GET /me` reports a key's own limits and spend, so nobody has to ask an operator to read a dashboard they cannot reach. Outside /v1, because it is not part of the OpenAI API. docs/onboarding.md is written for whoever operates this, including the parts that are deliberately absent — no self-service, no SSO, and why. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
1839832b8b |
Add a landing page and a read-only cluster dashboard
CI / test (push) Successful in 1m2s
Pasting the gateway's address into a browser returned 404, which looks like a broken endpoint rather than one that tells you how to use it. Two pages, and the split between them is a security boundary rather than a layout choice. The landing page is served without a key — it has to be, because the person reading it is the one who does not have one yet — so it contains no data at all: it says how to connect, not what is here. Model names stay behind /v1/models. It reconstructs the address from the forwarded headers so the copy-pasteable example names the address the client actually used rather than the internal one; those headers are reflected into displayed text and never used for a decision. The dashboard shows nodes, models, replicas, placement failures and usage, and lives on the admin listener only. That listener is loopback-bound and also serves pprof, which can dump the process heap — where API keys sit in plaintext — so reaching the page from elsewhere is an SSH tunnel's job. It is read-only: anything that mints a credential stays in `llmgw keys`, where it needs a shell rather than a browser session this project has no way to authenticate. Mounting the landing page uncovered a hole in the exemption list. An entry ending in "/" was a prefix rule, and "/" ends in "/", so exempting the landing page would have exempted every path there is — authentication still installed, still logging, and letting the whole API through. "/" is now exact, with a test enumerating what must and must not be exempt. Server-rendered templates and one stylesheet, embedded. The single-binary goal rules out a JavaScript toolchain and nothing on either page needs one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
0b93fb2883 |
Enforce budgets and rate limits, and write usage off the request path
CI / test (push) Successful in 1m7s
The second half of phase 5. A key now has a cost, a quota and a bill. Rate limits use a sliding window rather than a fixed one. A fixed window lets a caller send its whole minute's allowance in the last second of one window and again in the first second of the next, so the observed rate is twice the configured limit exactly when the system is least able to take it. Checking and charging are one operation, or two concurrent requests both see room and both take it. Token limits are checked against an estimate, because the real count only exists once the request has run; the difference is charged afterwards, so a caller who consistently underestimates is still bounded, one window late. A request with no max_tokens gets a modest assumption rather than zero — assuming zero would let an unbounded request through every token limit. Limits are checked before admission, not after: a request over its quota should be refused rather than queued, where it would occupy a band's capacity to no purpose. All refusals are 429, but a spent budget carries no Retry-After because waiting does not help until the period rolls over, and saying otherwise turns one client into a retry loop. Usage goes to a buffered channel and is written in batches. Record never blocks: if the buffer fills — a database stalled long enough to lose eight thousand records — records are dropped and counted, because losing accounting is bad and turning a stalled database into an outage is worse. A clean shutdown drains what is queued; those are requests that were already served. Failures are logged at error, since silently losing billing data is the kind of thing discovered at the end of a month. Requests are accounted whatever their outcome. A request that used a GPU and then failed still used it, and a key that could spend nothing by failing would be a way around a budget. Budgets are counted in memory and restored from the store at startup, so a restart does not hand every key a fresh allowance. The budget period travels with the reservation rather than being re-derived, because a monthly budget credited to a daily counter is a budget that never binds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
45c8cd23a3 |
Close the gateway: API keys, tenancy and model allowlists
CI / test (push) Successful in 2m34s
Until now anything that could reach the client port could use every model in the catalog. A key now resolves to an identity, and that identity decides what it may do. Authentication wraps the whole mux rather than checking inside each handler, so adding a route cannot add an unauthenticated one by omission. Health stays open, because a load balancer has no key and a health check that needs one reports the gateway down. Nothing on the request path reads the database. A key is resolved once and answered from memory until the entry expires; rejections are cached too, so a client looping with a bad key cannot turn into a database read per request. The price is that a revocation takes up to cache_ttl to take effect, which is the trade stated in the configuration next to the knob. A store failure is not an authentication answer: it is never cached as one and returns 503, not 401 — sending someone hunting for a key problem that does not exist wastes the one thing an incident is short of. A revoked key is refused with the same message as an unknown one, because saying otherwise confirms the key was real. With auth on the band comes from the key and the band header is ignored. The header was trusted only while there was nothing to consult, and it let any client promote itself into the top priority band. Tenant fairness within a band is now per key rather than per session, so one caller cannot take a band by opening many sessions. Allowlists are decided per model, not per string: a model's canonical name and every alias name the same thing, so either form works in an allowlist and no alias can reach a model that was not allowed. The first version of that test passed without the resolution being there at all, which is why it now checks all four combinations. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
dc38dd6096 |
Phase 1.5: admission control
CI / test (push) Successful in 48s
Inserted between quota checks and scheduling: admission answers when a request may proceed, the scheduler still answers where it goes. Keeping them apart is the point — an engine queue is FIFO and knows nothing about tenants, so once a batch job is in it, no endpoint selection rescues the request behind it. - saturation detector: one number per pool, the worse of queue depth and KV occupancy, since either alone misleads. - priority bands with per-band saturation ceilings, so the cluster degrades from the bottom up. - tenant round-robin within a band: one caller sending a thousand requests takes one slot in the rotation, not a thousand. - bounded queues with capacity (503) and TTL (504); cancelled requests leave the queue instead of holding capacity. - metrics for queue depth, wait time, rejections and pool saturation. Exit criterion, measured in test/e2e: with 500 concurrent bronze requests running continuously, interactive p99 in the platinum band stays at 1.04-1.24x idle against a 1.5x budget. With admission off the same load pushes it to 55-60x. Both directions are asserted, so the test fails if admission regresses and also if the load generator stops being a real load. Two lags had to be closed, and they needed different answers. Engine metrics are scraped and lag an interval; router inflight does not, but it counts running requests, and treating those as queue makes a healthy engine look overloaded every interval. What is actually queued is inflight minus batch capacity, and capacity is not in the metric contract, so it is learned as the high-water mark of observed running. Separately, between admission granting a request and the router dispatching it the request is invisible everywhere, so a simultaneous burst passed the check en masse; the controller now counts its own outstanding leases against dispatched inflight. One plan correction: a saturation ceiling of 1.0 was documented as "never held back". It is not — it means "flow up to the healthy limit". A band that must never wait needs a ceiling above 1. Plan §7.2.2 updated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
a3f71aa5ee |
Phase 1: the router serves traffic
CI / test (push) Successful in 1m1s
llmgw server is real now. It fronts a static set of engine replicas with an OpenAI-compatible API, scoring endpoints through the pipeline added in the previous commit. - proxy: SSE relayed event by event with a flush after each, cancellation propagated upstream, and separate deadlines for dial, headers, first token, inter-token idle and total. The idle watchdog is what distinguishes a wedged engine from a slow one. - router: dispatch with retry and model fallback. Retry only happens while nothing has reached the client; after that, splicing two responses together would be worse than the short stream the client already sees. - config: YAML via koanf overlaid on defaults, with LLMGW_ env overrides. Validation rejects duplicate endpoint ids and aliases that shadow a model, because both misroute silently at request time rather than failing loudly. - metrics: requests, duration, TTFT, tokens, plus per-scorer values and per-filter drops on a separate admin listener with health and pprof. Phase 1 exit criterion, measured in test/e2e: 1000 concurrent streams across 3 sim-engines complete at ~6800 req/s; every client disconnect frees the KV blocks on every engine; router overhead p99 is about 0.9 ms against the plan's 2 ms budget. The handlers live in internal/router rather than an internal/router/openai package as the plan's tree sketched: they need the dispatch loop, and splitting them apart would have meant an interface with exactly one implementation. Plan §4 updated to match. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |