ea4dd37f80af44432cedc856b67b8f5fba577366
17
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
1c57a4e378 |
Let the catalog change while the cluster is running
CI / test (push) Successful in 1m29s
Adding a model was an operator procedure with downtime: edit a file on the control plane, restart it, wait out the settling window, watch replicas churn while adoption catches up. I did that eight times in a day, and one of the restarts cost two minutes of outage while stalled replicas timed out. None of it was necessary. The reconciler already compares desired against actual on every pass, so a new catalog is simply a different desired. A Catalog stays immutable — New is the only way to build one and it refuses an invalid one — and what changes is which immutable catalog is current. Readers take the pointer and finish their operation against it, so a swap mid-request is invisible rather than a race. The router is told rather than asked, the same shape as the endpoint list and for the same reason: a push happens once per change, a pull would happen once per request. The router still seeds from configuration as well as being pushed to. A router with a static endpoint list and no control plane is a supported mode and has nothing to push it a catalog; without seeding it would resolve no model at all. Which turned up a bug that was latent while catalogs were fixed. reconcileModel only ever visits models that are in the catalog, so a replica whose model had been removed was visited by nothing: it kept its GPUs, kept counting, and kept running an engine for a name no client could ask for. The endpoint publisher skipped such a replica with the comment "the reconciler will tear it down", which was not true. It is now, and the comment says what actually happens instead of what was assumed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
e9fe3ad379 |
Know how big a model's context is, and what it can do
CI / test (push) Successful in 1m32s
Three things the gateway did not know about its own models, all of which surfaced this week as somebody else's bug. Context length. Nothing read it, /v1/models did not publish it, and an over-long prompt was not refused: it was admitted, scheduled, given a replica and a slot, and rejected by the engine in whatever shape that engine chose. A client sizing prompts from /v1/models had nothing to read. It is now declared per model, published when known, and checked before authentication work or a queue slot is spent — with a ten percent margin, because the estimate is bytes over four and being a few percent out must not turn a valid request into a refusal. Capabilities. The gateway forwards a tools array to any model, and one that cannot use them answers in prose; every OpenAI-compatible client then behaves as though the model is broken, and the only way to find out was to try it. Declared rather than discovered, because no engine will say — which is why these are claims, and why the next commit makes engine-check verify the tool-calling one instead of trusting it. And a bug found while wiring the first: the token estimate was taken from the routing prefix. That prefix stops at 2 KB by design, so every prompt above it estimated as 512 tokens however long it really was, and TPM limits undercounted exactly the requests that consume the most. Prefix routing wants a bounded head of the conversation; a token estimate wants all of it. They are now separate functions, and structured multimodal content counts rather than measuring zero. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
082d52bc97 |
Stop shipping a placeholder that looks like a value, and say what is served
CI / test (push) Successful in 1m23s
A user copied the curl from the profile page and sent it as it stood. Every other field in that snippet is filled in for them — the base URL, the key they had just been issued — so "model": "MODEL" reads as a value rather than a blank, and they got: no endpoint serves "MODEL" which is true and useless. It reads the same whether the name was a typo, a placeholder left in from the documentation, or a real outage, and it never says what would have worked. The message now lists what is being served, capped at ten with a count for the rest, and says plainly when nothing is. That list is what /v1/models already returns to the same caller, so it discloses nothing new. The profile page's example now names a model that is actually running, real ones before simulated — the point of an example someone can run is that it runs, and sending a first request to the sim engine would answer in Greek letters. The landing and claim pages keep a placeholder, because they are served without a key and deliberately say nothing about what is here; theirs is <model-name>, which cannot be mistaken for a value, with a line saying where to get the real one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
72b03a1a1c |
Say which model is simulated instead of letting it look broken
CI / test (push) Successful in 1m31s
big-model runs on the sim engine, whose whole vocabulary is the Greek alphabet, cycled at a fixed rate until max_tokens. That is what it is for: with no H100s it is the only thing exercising the TP8 two-node path, the fabric invariant and rebuild-after-node-loss. The output is the point. What was wrong is that nothing said so. /v1/models returned it with owned_by "llmgw", identical to the real model, and the chat picker sorted purely by name — so "big-model" led the list, was preselected by the browser, and anyone who typed a question and pressed Send got Greek letters back with no explanation. Reported as the model spitting out crap, which is a fair reading of what it looks like from outside. Simulated is derived from the profiles rather than configured: an operator who adds a sim profile and forgets a flag would be shipping placeholder text under a real model's name. A model counts as simulated only when every profile is, because one real profile means the request can still come back as inference. The marker on /v1/models is owned_by "llmgw-sim", which is already what the sim engine calls itself on its own /v1/models — the gateway was flattening a distinction that existed one layer down. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
3b67b1fe42 |
Stop the dashboard linking to pages that are not where it is
CI / test (push) Successful in 1m25s
Clicking "metrics" on the web dashboard answered "no api key was provided". The page is served twice and its navigation assumed one of them. Metrics, pprof and the management API exist on the admin listener; account administration, the profile and the chat exist on the client-facing port. The nav offered the first set in both places, so on the public port those links fell through to the API-key middleware — which is a far more confusing way to say "that is not here" than a 404 would have been. Each page now offers what is actually reachable from it, and the public one says where the rest lives rather than pretending it does not exist, including why that port is not exposed: pprof can dump the process heap, and the heap holds every API key in plaintext. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
de21649ad6 |
Add an in-browser chat, without putting a key in the browser
CI / test (push) Successful in 1m13s
So someone can try a model without setting up a client, and — more often — tell whether a problem is the gateway or their code. The page carries a session, and the server turns that into the identity an API key would have produced, resolved from the person's project. The request then goes down the ordinary path by being handed to the same handler /v1 uses: admission, model allowlist, rate limits, budget, accounting. There is exactly one request path, so there is no second place for its rules to drift — and without that, a playground is a hole in every limit the rest of the gateway enforces. The allowlist test posts a forbidden model and expects 403 for that reason. The body is rebuilt rather than forwarded, so the page cannot set fields it has no business setting: a stream:false that would hang the reader, a max_tokens large enough to occupy a replica for an hour. One turn is capped at 512 tokens. The picker offers what is actually served and allowed, not what is catalogued, because offering a model with no replica produces a 503 the reader cannot interpret. Inline vanilla JavaScript, for the same reason the charts are hand-drawn SVG. It streams, because on CPU inference ten seconds is normal and a chat that hangs and then dumps a paragraph feels broken. Replies are written with textContent throughout: model output and user input have no business being parsed as markup. Its POSTs carry the CSRF token in a header, which a cross-site form cannot set at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
859f08da9b |
Approve accounts from the browser, and draw the traffic
CI / test (push) Successful in 1m19s
Two things the dashboard was missing: it could not act, and it was a column of numbers. Account administration moves to /admin/users — approving into a project, suspending, reinstating. That is the work an operator does constantly and it should not need a terminal. Three things stay on the CLI, and the line is not arbitrary: promoting someone to admin, resetting a password, and issuing a key on another person's behalf each turn a stolen admin session into more than it was — permanent access, or somebody else's account. Approving and suspending do not. Suspending signs the account out everywhere, or suspension is advisory until the session happens to expire, and an admin cannot suspend themselves, which is not a thing an interface should let you do by misclicking. The charts are inline SVG generated in Go. A charting library would mean a JavaScript toolchain in CI and a megabyte of bundle for what is a list of rectangles, and the single-binary goal rules that out. Hourly over a day: fine enough to show a working day, coarse enough that one slow minute is not an incident. Empty hours are filled in, because a chart with hours missing where nothing happened reads as a chart with hours missing from the data. The SVG is marked as trusted HTML, so everything that reaches it is either a number this package computed or escaped explicitly, and a bar's colour can only be one of four names — a colour that could be anything is a way to write CSS into every page the chart appears on. There are tests for both. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
e519dc587c |
Serve the cluster dashboard over the web, without serving pprof with it
CI / test (push) Successful in 1m21s
Reaching it meant an SSH tunnel, because the page lived on the admin listener and that listener is bound to loopback. Opening the port was never the way to fix that: it also serves pprof, and pprof can dump the process heap, where every API key sits in plaintext. Exposing the port would have been exposing the key store. So the page moves rather than the port. The same dashboard is now mounted at /admin on the client-facing port, behind the session and CSRF machinery that already guards /profile, gated on a flag on the account. The admin listener keeps pprof and metrics and stays on loopback. One bit, not a role system: it grants looking. Everything that changes the cluster or issues a credential stays on the CLI, where it needs a shell rather than a session — which keeps the worst case of a stolen admin cookie to "saw the cluster" rather than "drained your fleet". Someone without the flag gets 404, not 403. A 403 confirms the page is there and that admin accounts exist here, which is free reconnaissance; 404 is what every other unknown path returns. The profile page does not show a link to a door the visitor cannot open. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
79e459d9ff |
Let people change their own password, and operators reset a lost one
CI / test (push) Failing after 1m11s
The account system shipped without either, which made the first forgotten password an unanswerable support request. Changing it requires the current one. A session is what an attacker gets from a borrowed laptop or a stolen cookie; the password is what they do not have, and without the check the first thing they would do is take the account. Every session is closed, including the browser doing the changing — leaving the others open would mean a password change does nothing about the reason someone is changing it. A new session is then opened for the browser in front of us, so the person who just proved their password is not thrown back to the sign-in page. That response also carries a new CSRF cookie, so the forms it renders take the new token rather than the one the request arrived with; there is a test that posts a form from that exact page. Guessing the current password is bounded the same way sign-in is, because someone holding a stolen session should not get unlimited attempts at the password that would let them keep it. There is no self-service reset: this gateway sends no email, and a reset link with nowhere to send it is not a reset. `llmgw users reset-password` is the operator's answer, and it closes their sessions too — if someone lost their password, the reason might be that somebody else has it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
216fc9bb03 |
Let people sign up, be approved, and issue their own keys
CI / test (push) Successful in 1m9s
Every key needed an operator with a shell. This adds the login system that made that necessary, which is the thing I argued against building until it was asked for — so it is built as if it will be attacked, because a public login form is. Signing up produces an account that can do nothing. Approval is what makes it work and stays on the CLI: having a shell on the control plane is what makes someone an administrator, and that is a smaller thing to get right than roles and an admin UI that mints credentials. The details that matter: Passwords are argon2id, where a slow hash is correct — the input is chosen by a human and has a dictionary, which is the opposite of an API key. Both kinds of authentication now live in this codebase and they are in separate packages so neither set of assumptions leaks into the other. Length is the only rule: composition rules push people towards Password1! and cost more entropy than they buy. Signup never says whether an address already has an account, and sign-in answers the same way for a wrong password and a missing account — verifying against a dummy hash when there is no user, so a stopwatch cannot enumerate accounts. Account state is checked after the password, not before, so guessing an address reveals nothing about it. Guessing is bounded per address rather than per IP: an attacker has many addresses to come from and one to guess against, and limiting by IP locks out an office behind one NAT. Sessions are server-side, so signing out and suspending take effect immediately rather than whenever a token expires; suspension is checked in the same query that resolves the session, because a second round trip is a check somebody eventually forgets. Every state-changing form carries a CSRF token verified against a hash in the session — a cookie authenticates the browser, not the page that asked. Key ownership is enforced in the UPDATE rather than by reading first, so a guessed key id from another account changes nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
8d35faf6f8 |
Give the landing page somewhere to send people
CI / test (push) Successful in 56s
The page explained the API, said keys come from an operator, and left the reader there. Someone who reaches the address without a key had no route to getting one, which is where onboarding stopped. `landing.contact` is that route, because the gateway cannot guess who to ask. An address becomes a mailto link, an http(s) URL stays a link, and anything else — a channel, a queue, a person — is shown as text. Other schemes are never linked: this string comes from configuration, configuration is not always written by whoever runs the gateway, and a javascript: href on a page served to every visitor is a short path to a stored XSS. Escaping the text does not make a scheme safe. Unconfigured, it now says something rather than nothing, and the page explains that a key arrives as a one-time link so the reader knows what to expect before it lands. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
3ed211b2f4 |
Onboard users with one-time links instead of pasted keys
CI / test (push) Successful in 2m43s
Issuing a key worked; getting it to the person it was issued for did not. The operator had a secret on their terminal and no way to send it that did not leave a bearer credential in a chat log forever. `keys create --claim` prints a link instead. GET shows a button and consumes nothing; POST reveals the key and burns the link. That split is not politeness — chat clients, mail scanners and link previewers fetch URLs the moment they are posted, so a GET that consumed the claim would mean the first thing to unfurl the link destroys the key before anyone reads it. A pending claim has to hold the key somewhere, which would otherwise put a working credential in the database and undo what hashing bought. So the key is sealed with a key derived from the link's own token, and the token is never stored — only its hash. A database leak yields two useless hashes and a blob nothing in the database can open. HKDF rather than the raw token, so the value used for lookup can never become the value that decrypts. Consuming is one UPDATE moving claimed_at off NULL, so two people opening the same link produce one key rather than two copies. A used link, an expired one and a token that was never real all render identically, because saying which confirms that a token was once valid. The page is no-store, no-referrer and noindex: the URL is itself the credential. `GET /me` reports a key's own limits and spend, so nobody has to ask an operator to read a dashboard they cannot reach. Outside /v1, because it is not part of the OpenAI API. docs/onboarding.md is written for whoever operates this, including the parts that are deliberately absent — no self-service, no SSO, and why. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
1839832b8b |
Add a landing page and a read-only cluster dashboard
CI / test (push) Successful in 1m2s
Pasting the gateway's address into a browser returned 404, which looks like a broken endpoint rather than one that tells you how to use it. Two pages, and the split between them is a security boundary rather than a layout choice. The landing page is served without a key — it has to be, because the person reading it is the one who does not have one yet — so it contains no data at all: it says how to connect, not what is here. Model names stay behind /v1/models. It reconstructs the address from the forwarded headers so the copy-pasteable example names the address the client actually used rather than the internal one; those headers are reflected into displayed text and never used for a decision. The dashboard shows nodes, models, replicas, placement failures and usage, and lives on the admin listener only. That listener is loopback-bound and also serves pprof, which can dump the process heap — where API keys sit in plaintext — so reaching the page from elsewhere is an SSH tunnel's job. It is read-only: anything that mints a credential stays in `llmgw keys`, where it needs a shell rather than a browser session this project has no way to authenticate. Mounting the landing page uncovered a hole in the exemption list. An entry ending in "/" was a prefix rule, and "/" ends in "/", so exempting the landing page would have exempted every path there is — authentication still installed, still logging, and letting the whole API through. "/" is now exact, with a test enumerating what must and must not be exempt. Server-rendered templates and one stylesheet, embedded. The single-binary goal rules out a JavaScript toolchain and nothing on either page needs one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
0b93fb2883 |
Enforce budgets and rate limits, and write usage off the request path
CI / test (push) Successful in 1m7s
The second half of phase 5. A key now has a cost, a quota and a bill. Rate limits use a sliding window rather than a fixed one. A fixed window lets a caller send its whole minute's allowance in the last second of one window and again in the first second of the next, so the observed rate is twice the configured limit exactly when the system is least able to take it. Checking and charging are one operation, or two concurrent requests both see room and both take it. Token limits are checked against an estimate, because the real count only exists once the request has run; the difference is charged afterwards, so a caller who consistently underestimates is still bounded, one window late. A request with no max_tokens gets a modest assumption rather than zero — assuming zero would let an unbounded request through every token limit. Limits are checked before admission, not after: a request over its quota should be refused rather than queued, where it would occupy a band's capacity to no purpose. All refusals are 429, but a spent budget carries no Retry-After because waiting does not help until the period rolls over, and saying otherwise turns one client into a retry loop. Usage goes to a buffered channel and is written in batches. Record never blocks: if the buffer fills — a database stalled long enough to lose eight thousand records — records are dropped and counted, because losing accounting is bad and turning a stalled database into an outage is worse. A clean shutdown drains what is queued; those are requests that were already served. Failures are logged at error, since silently losing billing data is the kind of thing discovered at the end of a month. Requests are accounted whatever their outcome. A request that used a GPU and then failed still used it, and a key that could spend nothing by failing would be a way around a budget. Budgets are counted in memory and restored from the store at startup, so a restart does not hand every key a fresh allowance. The budget period travels with the reservation rather than being re-derived, because a monthly budget credited to a daily counter is a budget that never binds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
45c8cd23a3 |
Close the gateway: API keys, tenancy and model allowlists
CI / test (push) Successful in 2m34s
Until now anything that could reach the client port could use every model in the catalog. A key now resolves to an identity, and that identity decides what it may do. Authentication wraps the whole mux rather than checking inside each handler, so adding a route cannot add an unauthenticated one by omission. Health stays open, because a load balancer has no key and a health check that needs one reports the gateway down. Nothing on the request path reads the database. A key is resolved once and answered from memory until the entry expires; rejections are cached too, so a client looping with a bad key cannot turn into a database read per request. The price is that a revocation takes up to cache_ttl to take effect, which is the trade stated in the configuration next to the knob. A store failure is not an authentication answer: it is never cached as one and returns 503, not 401 — sending someone hunting for a key problem that does not exist wastes the one thing an incident is short of. A revoked key is refused with the same message as an unknown one, because saying otherwise confirms the key was real. With auth on the band comes from the key and the band header is ignored. The header was trusted only while there was nothing to consult, and it let any client promote itself into the top priority band. Tenant fairness within a band is now per key rather than per session, so one caller cannot take a band by opening many sessions. Allowlists are decided per model, not per string: a model's canonical name and every alias name the same thing, so either form works in an allowlist and no alias can reach a model that was not allowed. The first version of that test passed without the resolution being there at all, which is why it now checks all four combinations. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
dc38dd6096 |
Phase 1.5: admission control
CI / test (push) Successful in 48s
Inserted between quota checks and scheduling: admission answers when a request may proceed, the scheduler still answers where it goes. Keeping them apart is the point — an engine queue is FIFO and knows nothing about tenants, so once a batch job is in it, no endpoint selection rescues the request behind it. - saturation detector: one number per pool, the worse of queue depth and KV occupancy, since either alone misleads. - priority bands with per-band saturation ceilings, so the cluster degrades from the bottom up. - tenant round-robin within a band: one caller sending a thousand requests takes one slot in the rotation, not a thousand. - bounded queues with capacity (503) and TTL (504); cancelled requests leave the queue instead of holding capacity. - metrics for queue depth, wait time, rejections and pool saturation. Exit criterion, measured in test/e2e: with 500 concurrent bronze requests running continuously, interactive p99 in the platinum band stays at 1.04-1.24x idle against a 1.5x budget. With admission off the same load pushes it to 55-60x. Both directions are asserted, so the test fails if admission regresses and also if the load generator stops being a real load. Two lags had to be closed, and they needed different answers. Engine metrics are scraped and lag an interval; router inflight does not, but it counts running requests, and treating those as queue makes a healthy engine look overloaded every interval. What is actually queued is inflight minus batch capacity, and capacity is not in the metric contract, so it is learned as the high-water mark of observed running. Separately, between admission granting a request and the router dispatching it the request is invisible everywhere, so a simultaneous burst passed the check en masse; the controller now counts its own outstanding leases against dispatched inflight. One plan correction: a saturation ceiling of 1.0 was documented as "never held back". It is not — it means "flow up to the healthy limit". A band that must never wait needs a ceiling above 1. Plan §7.2.2 updated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
a3f71aa5ee |
Phase 1: the router serves traffic
CI / test (push) Successful in 1m1s
llmgw server is real now. It fronts a static set of engine replicas with an OpenAI-compatible API, scoring endpoints through the pipeline added in the previous commit. - proxy: SSE relayed event by event with a flush after each, cancellation propagated upstream, and separate deadlines for dial, headers, first token, inter-token idle and total. The idle watchdog is what distinguishes a wedged engine from a slow one. - router: dispatch with retry and model fallback. Retry only happens while nothing has reached the client; after that, splicing two responses together would be worse than the short stream the client already sees. - config: YAML via koanf overlaid on defaults, with LLMGW_ env overrides. Validation rejects duplicate endpoint ids and aliases that shadow a model, because both misroute silently at request time rather than failing loudly. - metrics: requests, duration, TTFT, tokens, plus per-scorer values and per-filter drops on a separate admin listener with health and pprof. Phase 1 exit criterion, measured in test/e2e: 1000 concurrent streams across 3 sim-engines complete at ~6800 req/s; every client disconnect frees the KV blocks on every engine; router overhead p99 is about 0.9 ms against the plan's 2 ms budget. The handlers live in internal/router rather than an internal/router/openai package as the plan's tree sketched: they need the dispatch loop, and splitting them apart would have meant an interface with exactly one implementation. Plan §4 updated to match. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |