b5d650af16fe0631c5b8a85c5aebf450d8a9577d
23
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Ilker ManapandClaude Opus 5.5
|
abaf086fd5 |
A standby's portal is the leader's; the changelog as the release notes read it
A proxy left pointing at a standby showed an empty cluster: the portal and the admin pages read the standby's own memory. A standby now passes its client port to the leader's, all but /health, /ready and /leader, which must describe the machine a load balancer is choosing; and its admin port passes the dashboard as well as the API. The leader trusts the control planes' X-Forwarded-For, so clients keep their own address. The changelog's candidate section is "Unreleased" again: release notes for an rc read that section, and v0.11.3-rc.1 got none. The e2e wait for two endpoints is 30s: under the race detector on a busy runner, 15s was missed once. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5.5
|
44cd58005a |
Count sign-in attempts per address and client, guard the sign-in form with a token, ration both forms per address
A wrong password from anywhere counted against the address, so ten locked its owner out. Attempts are now counted per address and client, with a looser bound per address alone. The sign-in form carries a token matched by a cookie (login CSRF). POST /login and /signup are rationed per client address, a hash waits at most ten seconds for a turn, and server init writes signup_open: false. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
ebce41e22d |
Stop calling the PostgreSQL store SQLite, and stop guessing where it is
Two things that were only ever confusing, and one of them stopped a command working on the cluster. The store type was named SQLite and kept the name when PostgreSQL was added, so OpenPostgres returned a *SQLite and all seventy-odd methods on the production store were spelled after the database it was not running on. Nothing broke; people did. Earlier today I added three methods to *SQLite for a table that on this cluster only ever lives in PostgreSQL, and the name did not once look wrong. It is a field, which is all it ever was. `llmgw users` and `llmgw keys` defaulted --store to a SQLite path. On a deployment whose store is PostgreSQL every one of them failed with "unable to open database file" while the right answer sat in auth.store in the config file beside them; the conclusion an operator draws is that the tool is broken, and they are not far off. They now ask the server's own configuration before falling back, and the error names which of the three sources chose the address — the failure was never the path, it was which one picked it. The single-machine default still applies last, so a deployment with no config file keeps working with no flags. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
2bf7de1a43 |
Give two control planes a shared address, and a probe that means something
CI / test (push) Successful in 1m36s
The last piece of HA, and it is a deployment pattern rather than code: llmgw picks one leader and agents follow it, but client traffic arrives in front of both and llmgw is not in front of anything. Writing the pattern found a bug in the pattern. The first draft said to health-check `/health` and claimed a standby still routes to replicas the leader placed, so sending it traffic was fine. It does not. A standby's reconciler places nothing and it holds no agent streams to adopt from, so it publishes an empty endpoint list — a load balancer following that advice would have sent half the traffic to something that answers every request with 503. So there is now a `/ready` that means "this process has somewhere to send a request": 503 on a standby, 503 on a leader whose replicas have all failed, 200 when there is something to serve. `/health` keeps meaning the process is up, which is a different and less useful question. Both are open, because a load balancer has no key. There is a gap during failover where neither passes — three seconds when the leader stops cleanly, up to the lease TTL when it does not — and the document says so rather than implying otherwise. Traffic cannot go to a control plane that has nothing to route to. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
d26a30471e |
Let people forget a conversation, and make retention actually expire things
CI / test (push) Successful in 1m38s
Two halves of keeping the promise that storing makes.
DELETE /v1/threads/{id} is a client endpoint, not an admin one: the person who
said the thing should be able to unsay it, and making them ask an operator
would mean the operator reads it first. Deleting a thread in another project
answers exactly as deleting one that never existed does — two answers would
let a caller map which ids are real in projects they cannot read, and this one
would do it by trying to destroy them.
And something now runs the pruner. Without it "we keep conversations for 24
hours" was a comment rather than a promise: the rows would accumulate and the
number a project chose would describe nothing. Hourly, and once at startup,
because a gateway that has been down for a week should not wait another hour
before honouring a window that expired while it was off.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Ilker ManapandClaude Opus 5
|
9eb321ae3e |
Let a conversation continue on a different model than it started on
CI / test (push) Has been cancelled
The request path half. A caller sets x-llmgw-thread and sends only the new turn; llmgw puts the stored history in front of it, sends that to whichever model is serving, and stores the exchange afterwards. A header rather than a body field because some SDKs drop fields they do not know and all of them let a caller set a header. Capturing the reply is opt-in and bounded. This is the hottest path in the system, so a request without a thread accumulates nothing at all, and one with a thread stops accumulating at a cap rather than growing to it — a gateway that kept every reply in memory would be one long answer away from trouble. A reply is truncated on a rune boundary rather than dropped: a shortened turn replays as a shortened turn, while nothing at all replays as the assistant having said nothing, and invalid UTF-8 would break the next turn's body for good rather than for one turn. The streamed path sets the captured reply on every exit, not only the one that looks like the end. A stream can also finish with the client gone or the engine stalled, and what the assistant said up to that point is still what it said. A thread on a project that keeps no conversations is refused. The e2e test asserted that and got a cheerful 200: the request was served, nothing was stored, and the caller would have discovered it only when the model had no idea what they were referring to — which reads as the model being broken. It now asks whether storage is on before doing anything else. History is spliced after the model is finally settled, because a budget fallback may have changed it and trimming is measured against the context of the model that will actually serve. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
c9c841387a |
Check the prompt fits the model that will actually serve it
CI / test (push) Successful in 1m40s
The context check ran against the model the caller asked for. The budget fallback then swapped in a different one and nothing looked again, so a prompt sized for a 32k model could be sent to a 512-token one. Measured before the fix: 10002 prompt tokens delivered to a model declaring 512. The caller sees that as a truncated reply or an engine error naming a model they never chose, which is a worse outcome than the refusal the fallback exists to avoid — it fails later and explains less. Candidates that cannot hold the prompt are now skipped, and a fallback further down the list that can hold it is used instead. When none fit, the request is refused as it was before the fallback existed. The history is deliberately not trimmed to make it fit. Dropping a caller's messages to stay under a limit they never asked about would be a silent loss, inside a request that was already being quietly moved to another model. The tests drive the real request path, because a unit test on the fallback would have asserted the same wrong thing the code did. Both were checked against the unfixed code and fail there. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
28967edab3 |
Serve a spent budget something it can afford
CI / test (push) Successful in 1m38s
The accounting layer has always exempted a model that costs nothing, with a comment saying it is "what lets a key with no budget left still be served a cheap fallback". That described a caller nobody had written: the budget was checked once, against the requested model, and a refusal was the end of it, so the exemption could never fire. A caller that has run out is now offered the first model in its fallbacks it can still afford. The allowlist still applies — running out of money is not a way to reach a model a key was never allowed — and a model with no fallback is refused exactly as before, or a budget would stop meaning anything. Only a spent budget falls back: a rate limit is answered by waiting a second, and moving the request elsewhere would spend someone else's capacity to avoid a delay nobody asked to avoid. Which uncovered a header that has never worked. x-llmgw-served-model was set in finish, and finish runs after the proxy has written the response — so the one mechanism for telling a client that something else answered has been announcing it to nobody since it was written. Nothing failed, because the only thing asserting it was a comment. It is now set before dispatch, for the capacity fallback as well as this one. Finding that took bisecting a test I had written wrong: it polled for a 429 that never came, because the budget had run out and the fallback had served the request — the feature working, reported as the feature failing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
c232f41a66 |
Put the fleet view in the dashboard, so nothing else is needed to see it
CI / test (push) Successful in 1m32s
The metrics were exported and the only way to look at them was to stand up Prometheus beside the binary. For a project whose whole shape is one static binary with no dependencies, that is the wrong default: the common case is somebody wanting to know how full their cluster is, and that should not cost a monitoring stack. /admin now shows capacity per GPU class with a usage bar, free whole nodes per fabric — the number that decides whether a replica spanning nodes can land, since free cards on the wrong fabric are not capacity for a model that needs this one — and which replicas are holding hardware nobody is asking for. Nodes gain their memory and how many model files they can see. Taken from the same Summary that ctl status uses rather than derived in the page, because two implementations of the same arithmetic drift apart and the one on the screen is the one people would believe. A faulty card counts as neither used nor free, matching the scheduler. Counting it free is how a cluster looks like it has room it does not, and a page that disagreed with placement about that would be worse than no page. Prometheus stays supported and is now the answer to a narrower question: history, alerting, several clusters at once. The dashboard shows what is true now and keeps nothing, which is the honest division between them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
ad37bee081 |
Drive on-demand loading end to end, not just per component
CI / test (push) Successful in 1m30s
On-demand loading is four components agreeing: the router notices nothing is serving, the control plane decides the model may be started, the reconciler places it, and the endpoint list carries it back. Each half was tested alone, and the wiring between them is where this session's real bugs have been — a metrics collector registered after the registry was built, an adoption path that never health-checked — neither of which failed a unit test. Three tests through the real request path: one request loads a model that was not running and is answered by it, an idle one is stopped with the node still heartbeating so it is eviction rather than node loss, and an unknown model still gets the unknown-model error, because on-demand must not turn every typo into "loading". Writing them was itself the check. The first version heartbeated once and then slept through an UnreachableAfter of 300 milliseconds, so the node went quiet and nothing was placed — which looks exactly like the feature failing. A wrong assertion too: with an on-demand-only cluster and nothing loaded, naming what is served means naming nothing, and "no model is currently being served" is the honest answer rather than a lost error message. This matters more now that there is no cluster to deploy to. Live verification found several things today that no unit test would have, and these tests are what has to replace it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
1c57a4e378 |
Let the catalog change while the cluster is running
CI / test (push) Successful in 1m29s
Adding a model was an operator procedure with downtime: edit a file on the control plane, restart it, wait out the settling window, watch replicas churn while adoption catches up. I did that eight times in a day, and one of the restarts cost two minutes of outage while stalled replicas timed out. None of it was necessary. The reconciler already compares desired against actual on every pass, so a new catalog is simply a different desired. A Catalog stays immutable — New is the only way to build one and it refuses an invalid one — and what changes is which immutable catalog is current. Readers take the pointer and finish their operation against it, so a swap mid-request is invisible rather than a race. The router is told rather than asked, the same shape as the endpoint list and for the same reason: a push happens once per change, a pull would happen once per request. The router still seeds from configuration as well as being pushed to. A router with a static endpoint list and no control plane is a supported mode and has nothing to push it a catalog; without seeding it would resolve no model at all. Which turned up a bug that was latent while catalogs were fixed. reconcileModel only ever visits models that are in the catalog, so a replica whose model had been removed was visited by nothing: it kept its GPUs, kept counting, and kept running an engine for a name no client could ask for. The endpoint publisher skipped such a replica with the comment "the reconciler will tear it down", which was not true. It is now, and the comment says what actually happens instead of what was assumed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
aa2a1fe08f |
Scale the settling window in the e2e tests instead of leaving them failing
CI / test (push) Successful in 1m26s
|
||
|
Ilker ManapandClaude Opus 5
|
a075ff4120 |
Add approximate prefix routing, and find out it did nothing
CI / test (push) Successful in 1m19s
Tier 1 of plan §7.5: a radix tree of request prefixes that estimates which replica has already prefilled one. Bytes, not tokens — needing the engine's exact tokenizer is what makes tier 2 a separate project. It went in inert, and the decision metric is what caught it: 144 lookups, zero hits. The cause was a rule I had written and tested for. A node's label is the first request that created it, so the second request for the same system prompt diverges in its last few bytes — a new question after the same preamble, the single most common shape there is. Requiring the whole label to match threw away the 700 bytes they shared. What matters is the shared head, because that is what the engine cached, and a partial label match now counts. Two unit tests asserted the wrong rule and have been inverted. With that fixed, measured end to end against engines whose caches are too small for the working set: 22.5% cache hits without, 81.7% with, and the breakdown shows the misses are exactly the first round, when the tree is empty. The scenario matters as much as the number. With a handful of prompts and a large cache, scattering is free — every replica ends up holding everything — and prefix routing can only add constraint. The first version of the test measured that case and showed a loss, which was a true statement about a situation the feature is not for. Two things keep it from being the disaster prefix routing usually is. The imbalance guard, which needs both an absolute and a relative gap before it fires, because either alone misfires — absolute on an idle cluster, relative on a busy one. And it is a scorer rather than a filter: affinity is a vote the queue and inflight scorers can outvote, not a rule. The tree records the replica the picker actually chose, not the one the scorer proposed, and a node prefers the replica it has seen most rather than most recently — a count is stable under a randomizing picker, where recency just chases whatever the picker last did. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
de21649ad6 |
Add an in-browser chat, without putting a key in the browser
CI / test (push) Successful in 1m13s
So someone can try a model without setting up a client, and — more often — tell whether a problem is the gateway or their code. The page carries a session, and the server turns that into the identity an API key would have produced, resolved from the person's project. The request then goes down the ordinary path by being handed to the same handler /v1 uses: admission, model allowlist, rate limits, budget, accounting. There is exactly one request path, so there is no second place for its rules to drift — and without that, a playground is a hole in every limit the rest of the gateway enforces. The allowlist test posts a forbidden model and expects 403 for that reason. The body is rebuilt rather than forwarded, so the page cannot set fields it has no business setting: a stream:false that would hang the reader, a max_tokens large enough to occupy a replica for an hour. One turn is capped at 512 tokens. The picker offers what is actually served and allowed, not what is catalogued, because offering a model with no replica produces a 503 the reader cannot interpret. Inline vanilla JavaScript, for the same reason the charts are hand-drawn SVG. It streams, because on CPU inference ten seconds is normal and a chat that hangs and then dumps a paragraph feels broken. Replies are written with textContent throughout: model output and user input have no business being parsed as markup. Its POSTs carry the CSRF token in a header, which a cross-site form cannot set at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
e519dc587c |
Serve the cluster dashboard over the web, without serving pprof with it
CI / test (push) Successful in 1m21s
Reaching it meant an SSH tunnel, because the page lived on the admin listener and that listener is bound to loopback. Opening the port was never the way to fix that: it also serves pprof, and pprof can dump the process heap, where every API key sits in plaintext. Exposing the port would have been exposing the key store. So the page moves rather than the port. The same dashboard is now mounted at /admin on the client-facing port, behind the session and CSRF machinery that already guards /profile, gated on a flag on the account. The admin listener keeps pprof and metrics and stays on loopback. One bit, not a role system: it grants looking. Everything that changes the cluster or issues a credential stays on the CLI, where it needs a shell rather than a session — which keeps the worst case of a stolen admin cookie to "saw the cluster" rather than "drained your fleet". Someone without the flag gets 404, not 403. A 403 confirms the page is there and that admin accounts exist here, which is free reconnaissance; 404 is what every other unknown path returns. The profile page does not show a link to a door the visitor cannot open. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
79e459d9ff |
Let people change their own password, and operators reset a lost one
CI / test (push) Failing after 1m11s
The account system shipped without either, which made the first forgotten password an unanswerable support request. Changing it requires the current one. A session is what an attacker gets from a borrowed laptop or a stolen cookie; the password is what they do not have, and without the check the first thing they would do is take the account. Every session is closed, including the browser doing the changing — leaving the others open would mean a password change does nothing about the reason someone is changing it. A new session is then opened for the browser in front of us, so the person who just proved their password is not thrown back to the sign-in page. That response also carries a new CSRF cookie, so the forms it renders take the new token rather than the one the request arrived with; there is a test that posts a form from that exact page. Guessing the current password is bounded the same way sign-in is, because someone holding a stolen session should not get unlimited attempts at the password that would let them keep it. There is no self-service reset: this gateway sends no email, and a reset link with nowhere to send it is not a reset. `llmgw users reset-password` is the operator's answer, and it closes their sessions too — if someone lost their password, the reason might be that somebody else has it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
216fc9bb03 |
Let people sign up, be approved, and issue their own keys
CI / test (push) Successful in 1m9s
Every key needed an operator with a shell. This adds the login system that made that necessary, which is the thing I argued against building until it was asked for — so it is built as if it will be attacked, because a public login form is. Signing up produces an account that can do nothing. Approval is what makes it work and stays on the CLI: having a shell on the control plane is what makes someone an administrator, and that is a smaller thing to get right than roles and an admin UI that mints credentials. The details that matter: Passwords are argon2id, where a slow hash is correct — the input is chosen by a human and has a dictionary, which is the opposite of an API key. Both kinds of authentication now live in this codebase and they are in separate packages so neither set of assumptions leaks into the other. Length is the only rule: composition rules push people towards Password1! and cost more entropy than they buy. Signup never says whether an address already has an account, and sign-in answers the same way for a wrong password and a missing account — verifying against a dummy hash when there is no user, so a stopwatch cannot enumerate accounts. Account state is checked after the password, not before, so guessing an address reveals nothing about it. Guessing is bounded per address rather than per IP: an attacker has many addresses to come from and one to guess against, and limiting by IP locks out an office behind one NAT. Sessions are server-side, so signing out and suspending take effect immediately rather than whenever a token expires; suspension is checked in the same query that resolves the session, because a second round trip is a check somebody eventually forgets. Every state-changing form carries a CSRF token verified against a hash in the session — a cookie authenticates the browser, not the page that asked. Key ownership is enforced in the UPDATE rather than by reading first, so a guessed key id from another account changes nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
3ed211b2f4 |
Onboard users with one-time links instead of pasted keys
CI / test (push) Successful in 2m43s
Issuing a key worked; getting it to the person it was issued for did not. The operator had a secret on their terminal and no way to send it that did not leave a bearer credential in a chat log forever. `keys create --claim` prints a link instead. GET shows a button and consumes nothing; POST reveals the key and burns the link. That split is not politeness — chat clients, mail scanners and link previewers fetch URLs the moment they are posted, so a GET that consumed the claim would mean the first thing to unfurl the link destroys the key before anyone reads it. A pending claim has to hold the key somewhere, which would otherwise put a working credential in the database and undo what hashing bought. So the key is sealed with a key derived from the link's own token, and the token is never stored — only its hash. A database leak yields two useless hashes and a blob nothing in the database can open. HKDF rather than the raw token, so the value used for lookup can never become the value that decrypts. Consuming is one UPDATE moving claimed_at off NULL, so two people opening the same link produce one key rather than two copies. A used link, an expired one and a token that was never real all render identically, because saying which confirms that a token was once valid. The page is no-store, no-referrer and noindex: the URL is itself the credential. `GET /me` reports a key's own limits and spend, so nobody has to ask an operator to read a dashboard they cannot reach. Outside /v1, because it is not part of the OpenAI API. docs/onboarding.md is written for whoever operates this, including the parts that are deliberately absent — no self-service, no SSO, and why. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
0b93fb2883 |
Enforce budgets and rate limits, and write usage off the request path
CI / test (push) Successful in 1m7s
The second half of phase 5. A key now has a cost, a quota and a bill. Rate limits use a sliding window rather than a fixed one. A fixed window lets a caller send its whole minute's allowance in the last second of one window and again in the first second of the next, so the observed rate is twice the configured limit exactly when the system is least able to take it. Checking and charging are one operation, or two concurrent requests both see room and both take it. Token limits are checked against an estimate, because the real count only exists once the request has run; the difference is charged afterwards, so a caller who consistently underestimates is still bounded, one window late. A request with no max_tokens gets a modest assumption rather than zero — assuming zero would let an unbounded request through every token limit. Limits are checked before admission, not after: a request over its quota should be refused rather than queued, where it would occupy a band's capacity to no purpose. All refusals are 429, but a spent budget carries no Retry-After because waiting does not help until the period rolls over, and saying otherwise turns one client into a retry loop. Usage goes to a buffered channel and is written in batches. Record never blocks: if the buffer fills — a database stalled long enough to lose eight thousand records — records are dropped and counted, because losing accounting is bad and turning a stalled database into an outage is worse. A clean shutdown drains what is queued; those are requests that were already served. Failures are logged at error, since silently losing billing data is the kind of thing discovered at the end of a month. Requests are accounted whatever their outcome. A request that used a GPU and then failed still used it, and a key that could spend nothing by failing would be a way around a budget. Budgets are counted in memory and restored from the store at startup, so a restart does not hand every key a fresh allowance. The budget period travels with the reservation rather than being re-derived, because a monthly budget credited to a daily counter is a budget that never binds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
45c8cd23a3 |
Close the gateway: API keys, tenancy and model allowlists
CI / test (push) Successful in 2m34s
Until now anything that could reach the client port could use every model in the catalog. A key now resolves to an identity, and that identity decides what it may do. Authentication wraps the whole mux rather than checking inside each handler, so adding a route cannot add an unauthenticated one by omission. Health stays open, because a load balancer has no key and a health check that needs one reports the gateway down. Nothing on the request path reads the database. A key is resolved once and answered from memory until the entry expires; rejections are cached too, so a client looping with a bad key cannot turn into a database read per request. The price is that a revocation takes up to cache_ttl to take effect, which is the trade stated in the configuration next to the knob. A store failure is not an authentication answer: it is never cached as one and returns 503, not 401 — sending someone hunting for a key problem that does not exist wastes the one thing an incident is short of. A revoked key is refused with the same message as an unknown one, because saying otherwise confirms the key was real. With auth on the band comes from the key and the band header is ignored. The header was trusted only while there was nothing to consult, and it let any client promote itself into the top priority band. Tenant fairness within a band is now per key rather than per session, so one caller cannot take a band by opening many sessions. Allowlists are decided per model, not per string: a model's canonical name and every alias name the same thing, so either form works in an allowlist and no alias can reach a model that was not allowed. The first version of that test passed without the resolution being there at all, which is why it now checks all four combinations. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
 ilkerandClaude Opus 5
|
c1b9ea1bc2 |
Phase 3 (3/3): wire the control plane to the router
CI / test (push) Successful in 50s
Ready replicas become router endpoints with no static configuration involved. Endpoints that survive an update keep their live state — inflight counts, breaker state, drain rate — so a replica that was already serving does not start from cold because another one moved. The sink is an interface, so when the router and control plane split into separate processes it becomes a watch stream and nothing above it changes. Phase 3's exit criterion now holds end to end. A node hosting a live replica is killed while four clients stream continuously; the control plane marks it unreachable, then lost, rebuilds the replica on a survivor and republishes. Across three runs ~370 requests completed with zero failures and no replica was left on the dead node. This proves the control logic, not the transport: the agent fleet is simulated and the real path still needs the stream. The race detector caught a genuine bug while this went in. startReplica read the replica's fields after publishing it to the map, and scaleUp walked replica pointers outside the lock; both raced with agent status reports, which arrive from another goroutine and mutate exactly those fields. Against the simulated fleet the reports came back fast enough to hit the window on every run. State is snapshotted under the lock now and those functions work from values. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
ilkerandClaude Opus 5
|
dc38dd6096 |
Phase 1.5: admission control
CI / test (push) Successful in 48s
Inserted between quota checks and scheduling: admission answers when a request may proceed, the scheduler still answers where it goes. Keeping them apart is the point — an engine queue is FIFO and knows nothing about tenants, so once a batch job is in it, no endpoint selection rescues the request behind it. - saturation detector: one number per pool, the worse of queue depth and KV occupancy, since either alone misleads. - priority bands with per-band saturation ceilings, so the cluster degrades from the bottom up. - tenant round-robin within a band: one caller sending a thousand requests takes one slot in the rotation, not a thousand. - bounded queues with capacity (503) and TTL (504); cancelled requests leave the queue instead of holding capacity. - metrics for queue depth, wait time, rejections and pool saturation. Exit criterion, measured in test/e2e: with 500 concurrent bronze requests running continuously, interactive p99 in the platinum band stays at 1.04-1.24x idle against a 1.5x budget. With admission off the same load pushes it to 55-60x. Both directions are asserted, so the test fails if admission regresses and also if the load generator stops being a real load. Two lags had to be closed, and they needed different answers. Engine metrics are scraped and lag an interval; router inflight does not, but it counts running requests, and treating those as queue makes a healthy engine look overloaded every interval. What is actually queued is inflight minus batch capacity, and capacity is not in the metric contract, so it is learned as the high-water mark of observed running. Separately, between admission granting a request and the router dispatching it the request is invisible everywhere, so a simultaneous burst passed the check en masse; the controller now counts its own outstanding leases against dispatched inflight. One plan correction: a saturation ceiling of 1.0 was documented as "never held back". It is not — it means "flow up to the healthy limit". A band that must never wait needs a ceiling above 1. Plan §7.2.2 updated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
ilkerandClaude Opus 5
|
a3f71aa5ee |
Phase 1: the router serves traffic
CI / test (push) Successful in 1m1s
llmgw server is real now. It fronts a static set of engine replicas with an OpenAI-compatible API, scoring endpoints through the pipeline added in the previous commit. - proxy: SSE relayed event by event with a flush after each, cancellation propagated upstream, and separate deadlines for dial, headers, first token, inter-token idle and total. The idle watchdog is what distinguishes a wedged engine from a slow one. - router: dispatch with retry and model fallback. Retry only happens while nothing has reached the client; after that, splicing two responses together would be worse than the short stream the client already sees. - config: YAML via koanf overlaid on defaults, with LLMGW_ env overrides. Validation rejects duplicate endpoint ids and aliases that shadow a model, because both misroute silently at request time rather than failing loudly. - metrics: requests, duration, TTFT, tokens, plus per-scorer values and per-filter drops on a separate admin listener with health and pprof. Phase 1 exit criterion, measured in test/e2e: 1000 concurrent streams across 3 sim-engines complete at ~6800 req/s; every client disconnect frees the KV blocks on every engine; router overhead p99 is about 0.9 ms against the plan's 2 ms budget. The handlers live in internal/router rather than an internal/router/openai package as the plan's tree sketched: they need the dispatch loop, and splitting them apart would have meant an interface with exactly one implementation. Plan §4 updated to match. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |