Commit Graph
30 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5 1c57a4e378 Let the catalog change while the cluster is running
CI / test (push) Successful in 1m29s
Adding a model was an operator procedure with downtime: edit a file on the
control plane, restart it, wait out the settling window, watch replicas
churn while adoption catches up. I did that eight times in a day, and one
of the restarts cost two minutes of outage while stalled replicas timed
out. None of it was necessary. The reconciler already compares desired
against actual on every pass, so a new catalog is simply a different
desired.

A Catalog stays immutable — New is the only way to build one and it refuses
an invalid one — and what changes is which immutable catalog is current.
Readers take the pointer and finish their operation against it, so a swap
mid-request is invisible rather than a race. The router is told rather than
asked, the same shape as the endpoint list and for the same reason: a push
happens once per change, a pull would happen once per request.

The router still seeds from configuration as well as being pushed to. A
router with a static endpoint list and no control plane is a supported mode
and has nothing to push it a catalog; without seeding it would resolve no
model at all.

Which turned up a bug that was latent while catalogs were fixed.
reconcileModel only ever visits models that are in the catalog, so a
replica whose model had been removed was visited by nothing: it kept its
GPUs, kept counting, and kept running an engine for a name no client could
ask for. The endpoint publisher skipped such a replica with the comment
"the reconciler will tear it down", which was not true. It is now, and the
comment says what actually happens instead of what was assumed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 01:08:21 +02:00
Ilker ManapandClaude Opus 5 9d8cd5e284 Track which replicas are actually being used, and summarise the fleet
CI / test (push) Successful in 1m28s
Two things a cluster with many nodes and many GPUs needs that were missing,
and they are the same thing seen from two distances.

Idle tracking. The control plane knew what was placed and whether it was
healthy, never whether anyone wanted it — so on a fleet with more models
than cards there was no way to answer the first question anyone asks, which
is what can I stop. The router is the only thing that knows, so it now
reports it: a timestamp written in Acquire, where a request already meets an
endpoint under a lock already held, and collected once a tick as a map. That
shape is the point. Per-request notification would be a firehose carrying
one bit — the newest timestamp wins — and this costs the same on four
replicas as on four hundred.

It travels through an interface in the direction opposite EndpointSink, for
the same reason that one exists: the control plane decides what should run,
the router knows whether anyone wants it, and neither should need to be in
the other's process to say so.

Unknown is kept distinct from idle throughout. A router that has just
restarted knows nothing, and a zero timestamp would be the oldest idle
replica in every comparison — so one restart would nominate an entire fleet
for eviction. Absent from the map, "-" in the table, negative in the JSON,
and IdleFor returns a second value rather than a duration anything could
act on by mistake.

ctl status. A per-node listing is right for four nodes and wrong for two
hundred: how full am I, what is broken, what is holding cards nobody is
using are aggregates, and reading them off a long table is a chore, which
means it gets skipped. Capacity per GPU class with usage bars, free whole
nodes per fabric — the number that decides whether a multi-node replica can
land, since free cards on the wrong fabric are not capacity for a model that
spans nodes — replica and model counts, models that cannot be placed, and
what has gone unasked-for. Derived from the same state the tables show, so
it cannot disagree with them.

Unhealthy GPUs count as neither used nor free, because counting them free is
how a cluster looks like it has room it does not.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 00:52:34 +02:00
Ilker ManapandClaude Opus 5 c115681259 Meter a project nobody set limits on
CI / test (push) Has been cancelled
Unlimited was the default. Creating a project without limits is the easy
path and the common one — the "default" project on this cluster is one, and
the person using it is metered by nothing at all. The first sign of that is
the bill or the outage.

auth.default_rpm and default_tpm now apply to a key no scope gives a limit
to. Zero keeps the old behaviour, which is still a reasonable choice; the
point is that it becomes a choice rather than an omission.

Applied after the chain rather than folded in as its widest scope, because
a default is a fallback and not a ceiling: a project deliberately given 600
rpm must keep it, and tightening would silently cap it at the default. An
explicit limit wins in both directions.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 23:28:28 +02:00
Ilker ManapandClaude Opus 5 db448a9e87 Stop using a liveness timeout as a generation budget
CI / test (push) Successful in 1m23s
response_header sounds like a check that the endpoint is alive, and for a
streaming request it is: headers arrive before generation starts. For a
non-streaming request the engine sends nothing until the completion is
finished, so the same setting is the total generation budget wearing a
liveness timeout's name.

At 60s that was invisible on GPUs and a hard cap of roughly a hundred
tokens on qwen2.5-coder-14b at 1.7 tokens a second. A request for 120
tokens failed at exactly one minute, and neither the setting's name nor the
error said why.

Two transports now: the short header timeout for streaming, where it still
catches a sick endpoint, and a generation budget for non-streaming,
defaulting to ten minutes. One number could not mean both things.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 23:15:00 +02:00
Ilker ManapandClaude Opus 5 e9fe3ad379 Know how big a model's context is, and what it can do
CI / test (push) Successful in 1m32s
Three things the gateway did not know about its own models, all of which
surfaced this week as somebody else's bug.

Context length. Nothing read it, /v1/models did not publish it, and an
over-long prompt was not refused: it was admitted, scheduled, given a
replica and a slot, and rejected by the engine in whatever shape that
engine chose. A client sizing prompts from /v1/models had nothing to read.
It is now declared per model, published when known, and checked before
authentication work or a queue slot is spent — with a ten percent margin,
because the estimate is bytes over four and being a few percent out must
not turn a valid request into a refusal.

Capabilities. The gateway forwards a tools array to any model, and one that
cannot use them answers in prose; every OpenAI-compatible client then
behaves as though the model is broken, and the only way to find out was to
try it. Declared rather than discovered, because no engine will say — which
is why these are claims, and why the next commit makes engine-check verify
the tool-calling one instead of trusting it.

And a bug found while wiring the first: the token estimate was taken from
the routing prefix. That prefix stops at 2 KB by design, so every prompt
above it estimated as 512 tokens however long it really was, and TPM limits
undercounted exactly the requests that consume the most. Prefix routing
wants a bounded head of the conversation; a token estimate wants all of it.
They are now separate functions, and structured multimodal content counts
rather than measuring zero.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 23:07:49 +02:00
Ilker ManapandClaude Opus 5 45d0dee470 Let a reply full of code be downloaded as a zip
CI / test (push) Successful in 1m25s
Asking a coding model for anything real produces a small tree, and the only
way to get it out of the page was to select each block and paste it
somewhere. A reply containing fenced code now offers a zip, with the
directory structure the model described.

Server-side, because the interesting part is the filenames. They are written
by a language model and they go into an archive that somebody extracts with
a tool that trusts archives, so "../../.ssh/authorized_keys" and
"/etc/cron.d/x" are inputs to expect rather than hope against. archive/zip
and a sanitiser with tests beat string handling in a browser. An absolute
path is made relative — it cannot escape once the leading separator is gone,
and refusing a file over a leading slash throws away work. Anything that
still escapes after cleaning is rejected rather than repaired, because
silently turning ../../evil.go into evil.go puts a file somewhere the reader
did not expect; it gets a generated name instead.

Naming follows what models actually do: on the fence (```go:cmd/main.go,
```main.go, ```go main.go, title="x"), or on the line above it as a heading
or comment. Only the two lines immediately above, so a sentence mentioning
main.go is not read as a filename. A block nobody named still comes back
under a generated name, because losing code because it was unlabelled is
worse than naming it badly, and two blocks claiming one path both survive —
models show a file and then show it again with one line changed.

Nothing is stored: the reply arrives, the zip goes back, neither is kept.
The endpoint sits behind the same session guard and CSRF check as
/chat/send.

Two of these tests asserted the wrong thing when first written — that an
absolute path must be refused, and that hyphens are stripped from the
archive name. Both were corrected to what the code should do rather than
the code changed to match them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 21:06:07 +02:00
Ilker ManapandClaude Opus 5 0a98d01361 Say when a chat reply was cut off, and stop cutting it so early
CI / test (push) Successful in 1m20s
Reported as the model not completing a request and stopping. It did
complete — it ran out of room. The chat page caps a turn at max_tokens and
then never looked at finish_reason, so a reply that hit the cap ended
mid-word and looked exactly like one that was finished.

The silence was the real defect. A reader has no way to tell "the model had
nothing more to say" from "the page stopped it", and the only clue is that
the last sentence ends abruptly.

512 was also too low. It was chosen when every model here was a 0.5B
answering in a paragraph; it is about a screenful of code, so a coding
model gets cut off inside the first function it writes. 2048 now, which is
still a bound — the reason for having one has not changed.

The notice is appended beside the reply, never into it: that element holds
model output placed with textContent, and building markup in the same node
is how the careful handling everywhere else gets undone.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 21:03:07 +02:00
Ilker ManapandClaude Opus 5 082d52bc97 Stop shipping a placeholder that looks like a value, and say what is served
CI / test (push) Successful in 1m23s
A user copied the curl from the profile page and sent it as it stood. Every
other field in that snippet is filled in for them — the base URL, the key
they had just been issued — so "model": "MODEL" reads as a value rather
than a blank, and they got:

  no endpoint serves "MODEL"

which is true and useless. It reads the same whether the name was a typo,
a placeholder left in from the documentation, or a real outage, and it
never says what would have worked.

The message now lists what is being served, capped at ten with a count for
the rest, and says plainly when nothing is. That list is what /v1/models
already returns to the same caller, so it discloses nothing new.

The profile page's example now names a model that is actually running,
real ones before simulated — the point of an example someone can run is
that it runs, and sending a first request to the sim engine would answer
in Greek letters. The landing and claim pages keep a placeholder, because
they are served without a key and deliberately say nothing about what is
here; theirs is <model-name>, which cannot be mistaken for a value, with a
line saying where to get the real one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 18:23:19 +02:00
Ilker ManapandClaude Opus 5 72b03a1a1c Say which model is simulated instead of letting it look broken
CI / test (push) Successful in 1m31s
big-model runs on the sim engine, whose whole vocabulary is the Greek
alphabet, cycled at a fixed rate until max_tokens. That is what it is for:
with no H100s it is the only thing exercising the TP8 two-node path, the
fabric invariant and rebuild-after-node-loss. The output is the point.

What was wrong is that nothing said so. /v1/models returned it with
owned_by "llmgw", identical to the real model, and the chat picker sorted
purely by name — so "big-model" led the list, was preselected by the
browser, and anyone who typed a question and pressed Send got Greek
letters back with no explanation. Reported as the model spitting out crap,
which is a fair reading of what it looks like from outside.

Simulated is derived from the profiles rather than configured: an operator
who adds a sim profile and forgets a flag would be shipping placeholder
text under a real model's name. A model counts as simulated only when
every profile is, because one real profile means the request can still
come back as inference.

The marker on /v1/models is owned_by "llmgw-sim", which is already what
the sim engine calls itself on its own /v1/models — the gateway was
flattening a distinction that existed one layer down.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 17:11:56 +02:00
Ilker ManapandClaude Opus 5 3b67b1fe42 Stop the dashboard linking to pages that are not where it is
CI / test (push) Successful in 1m25s
Clicking "metrics" on the web dashboard answered "no api key was provided".

The page is served twice and its navigation assumed one of them. Metrics,
pprof and the management API exist on the admin listener; account
administration, the profile and the chat exist on the client-facing port. The
nav offered the first set in both places, so on the public port those links
fell through to the API-key middleware — which is a far more confusing way to
say "that is not here" than a 404 would have been.

Each page now offers what is actually reachable from it, and the public one
says where the rest lives rather than pretending it does not exist, including
why that port is not exposed: pprof can dump the process heap, and the heap
holds every API key in plaintext.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 16:57:58 +02:00
Ilker ManapandClaude Opus 5 a075ff4120 Add approximate prefix routing, and find out it did nothing
CI / test (push) Successful in 1m19s
Tier 1 of plan §7.5: a radix tree of request prefixes that estimates which
replica has already prefilled one. Bytes, not tokens — needing the engine's
exact tokenizer is what makes tier 2 a separate project.

It went in inert, and the decision metric is what caught it: 144 lookups, zero
hits. The cause was a rule I had written and tested for. A node's label is the
first request that created it, so the second request for the same system
prompt diverges in its last few bytes — a new question after the same
preamble, the single most common shape there is. Requiring the whole label to
match threw away the 700 bytes they shared. What matters is the shared head,
because that is what the engine cached, and a partial label match now counts.
Two unit tests asserted the wrong rule and have been inverted.

With that fixed, measured end to end against engines whose caches are too
small for the working set: 22.5% cache hits without, 81.7% with, and the
breakdown shows the misses are exactly the first round, when the tree is
empty.

The scenario matters as much as the number. With a handful of prompts and a
large cache, scattering is free — every replica ends up holding everything —
and prefix routing can only add constraint. The first version of the test
measured that case and showed a loss, which was a true statement about a
situation the feature is not for.

Two things keep it from being the disaster prefix routing usually is. The
imbalance guard, which needs both an absolute and a relative gap before it
fires, because either alone misfires — absolute on an idle cluster, relative
on a busy one. And it is a scorer rather than a filter: affinity is a vote the
queue and inflight scorers can outvote, not a rule.

The tree records the replica the picker actually chose, not the one the scorer
proposed, and a node prefers the replica it has seen most rather than most
recently — a count is stable under a randomizing picker, where recency just
chases whatever the picker last did.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 12:13:19 +02:00
Ilker ManapandClaude Opus 5 de21649ad6 Add an in-browser chat, without putting a key in the browser
CI / test (push) Successful in 1m13s
So someone can try a model without setting up a client, and — more often —
tell whether a problem is the gateway or their code.

The page carries a session, and the server turns that into the identity an API
key would have produced, resolved from the person's project. The request then
goes down the ordinary path by being handed to the same handler /v1 uses:
admission, model allowlist, rate limits, budget, accounting. There is exactly
one request path, so there is no second place for its rules to drift — and
without that, a playground is a hole in every limit the rest of the gateway
enforces. The allowlist test posts a forbidden model and expects 403 for that
reason.

The body is rebuilt rather than forwarded, so the page cannot set fields it has
no business setting: a stream:false that would hang the reader, a max_tokens
large enough to occupy a replica for an hour. One turn is capped at 512 tokens.

The picker offers what is actually served and allowed, not what is catalogued,
because offering a model with no replica produces a 503 the reader cannot
interpret.

Inline vanilla JavaScript, for the same reason the charts are hand-drawn SVG.
It streams, because on CPU inference ten seconds is normal and a chat that
hangs and then dumps a paragraph feels broken. Replies are written with
textContent throughout: model output and user input have no business being
parsed as markup. Its POSTs carry the CSRF token in a header, which a
cross-site form cannot set at all.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 12:05:30 +02:00
Ilker ManapandClaude Opus 5 859f08da9b Approve accounts from the browser, and draw the traffic
CI / test (push) Successful in 1m19s
Two things the dashboard was missing: it could not act, and it was a column of
numbers.

Account administration moves to /admin/users — approving into a project,
suspending, reinstating. That is the work an operator does constantly and it
should not need a terminal. Three things stay on the CLI, and the line is not
arbitrary: promoting someone to admin, resetting a password, and issuing a key
on another person's behalf each turn a stolen admin session into more than it
was — permanent access, or somebody else's account. Approving and suspending
do not. Suspending signs the account out everywhere, or suspension is advisory
until the session happens to expire, and an admin cannot suspend themselves,
which is not a thing an interface should let you do by misclicking.

The charts are inline SVG generated in Go. A charting library would mean a
JavaScript toolchain in CI and a megabyte of bundle for what is a list of
rectangles, and the single-binary goal rules that out. Hourly over a day: fine
enough to show a working day, coarse enough that one slow minute is not an
incident. Empty hours are filled in, because a chart with hours missing where
nothing happened reads as a chart with hours missing from the data.

The SVG is marked as trusted HTML, so everything that reaches it is either a
number this package computed or escaped explicitly, and a bar's colour can only
be one of four names — a colour that could be anything is a way to write CSS
into every page the chart appears on. There are tests for both.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 12:01:42 +02:00
Ilker ManapandClaude Opus 5 e519dc587c Serve the cluster dashboard over the web, without serving pprof with it
CI / test (push) Successful in 1m21s
Reaching it meant an SSH tunnel, because the page lived on the admin listener
and that listener is bound to loopback. Opening the port was never the way to
fix that: it also serves pprof, and pprof can dump the process heap, where
every API key sits in plaintext. Exposing the port would have been exposing
the key store.

So the page moves rather than the port. The same dashboard is now mounted at
/admin on the client-facing port, behind the session and CSRF machinery that
already guards /profile, gated on a flag on the account. The admin listener
keeps pprof and metrics and stays on loopback.

One bit, not a role system: it grants looking. Everything that changes the
cluster or issues a credential stays on the CLI, where it needs a shell rather
than a session — which keeps the worst case of a stolen admin cookie to "saw
the cluster" rather than "drained your fleet".

Someone without the flag gets 404, not 403. A 403 confirms the page is there
and that admin accounts exist here, which is free reconnaissance; 404 is what
every other unknown path returns. The profile page does not show a link to a
door the visitor cannot open.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 10:46:45 +02:00
Ilker ManapandClaude Opus 5 79e459d9ff Let people change their own password, and operators reset a lost one
CI / test (push) Failing after 1m11s
The account system shipped without either, which made the first forgotten
password an unanswerable support request.

Changing it requires the current one. A session is what an attacker gets from
a borrowed laptop or a stolen cookie; the password is what they do not have,
and without the check the first thing they would do is take the account.

Every session is closed, including the browser doing the changing — leaving
the others open would mean a password change does nothing about the reason
someone is changing it. A new session is then opened for the browser in front
of us, so the person who just proved their password is not thrown back to the
sign-in page. That response also carries a new CSRF cookie, so the forms it
renders take the new token rather than the one the request arrived with; there
is a test that posts a form from that exact page.

Guessing the current password is bounded the same way sign-in is, because
someone holding a stolen session should not get unlimited attempts at the
password that would let them keep it.

There is no self-service reset: this gateway sends no email, and a reset link
with nowhere to send it is not a reset. `llmgw users reset-password` is the
operator's answer, and it closes their sessions too — if someone lost their
password, the reason might be that somebody else has it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 09:39:11 +02:00
Ilker ManapandClaude Opus 5 216fc9bb03 Let people sign up, be approved, and issue their own keys
CI / test (push) Successful in 1m9s
Every key needed an operator with a shell. This adds the login system that
made that necessary, which is the thing I argued against building until it was
asked for — so it is built as if it will be attacked, because a public login
form is.

Signing up produces an account that can do nothing. Approval is what makes it
work and stays on the CLI: having a shell on the control plane is what makes
someone an administrator, and that is a smaller thing to get right than roles
and an admin UI that mints credentials.

The details that matter:

Passwords are argon2id, where a slow hash is correct — the input is chosen by
a human and has a dictionary, which is the opposite of an API key. Both kinds
of authentication now live in this codebase and they are in separate packages
so neither set of assumptions leaks into the other. Length is the only rule:
composition rules push people towards Password1! and cost more entropy than
they buy.

Signup never says whether an address already has an account, and sign-in
answers the same way for a wrong password and a missing account — verifying
against a dummy hash when there is no user, so a stopwatch cannot enumerate
accounts. Account state is checked after the password, not before, so guessing
an address reveals nothing about it. Guessing is bounded per address rather
than per IP: an attacker has many addresses to come from and one to guess
against, and limiting by IP locks out an office behind one NAT.

Sessions are server-side, so signing out and suspending take effect
immediately rather than whenever a token expires; suspension is checked in the
same query that resolves the session, because a second round trip is a check
somebody eventually forgets. Every state-changing form carries a CSRF token
verified against a hash in the session — a cookie authenticates the browser,
not the page that asked.

Key ownership is enforced in the UPDATE rather than by reading first, so a
guessed key id from another account changes nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 09:32:35 +02:00
Ilker ManapandClaude Opus 5 8d35faf6f8 Give the landing page somewhere to send people
CI / test (push) Successful in 56s
The page explained the API, said keys come from an operator, and left the
reader there. Someone who reaches the address without a key had no route to
getting one, which is where onboarding stopped.

`landing.contact` is that route, because the gateway cannot guess who to ask.
An address becomes a mailto link, an http(s) URL stays a link, and anything
else — a channel, a queue, a person — is shown as text. Other schemes are
never linked: this string comes from configuration, configuration is not
always written by whoever runs the gateway, and a javascript: href on a page
served to every visitor is a short path to a stored XSS. Escaping the text
does not make a scheme safe.

Unconfigured, it now says something rather than nothing, and the page explains
that a key arrives as a one-time link so the reader knows what to expect
before it lands.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 09:22:42 +02:00
Ilker ManapandClaude Opus 5 3ed211b2f4 Onboard users with one-time links instead of pasted keys
CI / test (push) Successful in 2m43s
Issuing a key worked; getting it to the person it was issued for did not. The
operator had a secret on their terminal and no way to send it that did not
leave a bearer credential in a chat log forever.

`keys create --claim` prints a link instead. GET shows a button and consumes
nothing; POST reveals the key and burns the link. That split is not politeness
— chat clients, mail scanners and link previewers fetch URLs the moment they
are posted, so a GET that consumed the claim would mean the first thing to
unfurl the link destroys the key before anyone reads it.

A pending claim has to hold the key somewhere, which would otherwise put a
working credential in the database and undo what hashing bought. So the key is
sealed with a key derived from the link's own token, and the token is never
stored — only its hash. A database leak yields two useless hashes and a blob
nothing in the database can open. HKDF rather than the raw token, so the value
used for lookup can never become the value that decrypts.

Consuming is one UPDATE moving claimed_at off NULL, so two people opening the
same link produce one key rather than two copies. A used link, an expired one
and a token that was never real all render identically, because saying which
confirms that a token was once valid. The page is no-store, no-referrer and
noindex: the URL is itself the credential.

`GET /me` reports a key's own limits and spend, so nobody has to ask an
operator to read a dashboard they cannot reach. Outside /v1, because it is not
part of the OpenAI API.

docs/onboarding.md is written for whoever operates this, including the parts
that are deliberately absent — no self-service, no SSO, and why.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 09:11:28 +02:00
Ilker ManapandClaude Opus 5 1839832b8b Add a landing page and a read-only cluster dashboard
CI / test (push) Successful in 1m2s
Pasting the gateway's address into a browser returned 404, which looks like a
broken endpoint rather than one that tells you how to use it.

Two pages, and the split between them is a security boundary rather than a
layout choice. The landing page is served without a key — it has to be,
because the person reading it is the one who does not have one yet — so it
contains no data at all: it says how to connect, not what is here. Model names
stay behind /v1/models. It reconstructs the address from the forwarded headers
so the copy-pasteable example names the address the client actually used
rather than the internal one; those headers are reflected into displayed text
and never used for a decision.

The dashboard shows nodes, models, replicas, placement failures and usage, and
lives on the admin listener only. That listener is loopback-bound and also
serves pprof, which can dump the process heap — where API keys sit in
plaintext — so reaching the page from elsewhere is an SSH tunnel's job. It is
read-only: anything that mints a credential stays in `llmgw keys`, where it
needs a shell rather than a browser session this project has no way to
authenticate.

Mounting the landing page uncovered a hole in the exemption list. An entry
ending in "/" was a prefix rule, and "/" ends in "/", so exempting the landing
page would have exempted every path there is — authentication still installed,
still logging, and letting the whole API through. "/" is now exact, with a
test enumerating what must and must not be exempt.

Server-rendered templates and one stylesheet, embedded. The single-binary goal
rules out a JavaScript toolchain and nothing on either page needs one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 08:58:05 +02:00
Ilker ManapandClaude Opus 5 0b93fb2883 Enforce budgets and rate limits, and write usage off the request path
CI / test (push) Successful in 1m7s
The second half of phase 5. A key now has a cost, a quota and a bill.

Rate limits use a sliding window rather than a fixed one. A fixed window lets
a caller send its whole minute's allowance in the last second of one window
and again in the first second of the next, so the observed rate is twice the
configured limit exactly when the system is least able to take it. Checking
and charging are one operation, or two concurrent requests both see room and
both take it.

Token limits are checked against an estimate, because the real count only
exists once the request has run; the difference is charged afterwards, so a
caller who consistently underestimates is still bounded, one window late. A
request with no max_tokens gets a modest assumption rather than zero —
assuming zero would let an unbounded request through every token limit.

Limits are checked before admission, not after: a request over its quota
should be refused rather than queued, where it would occupy a band's capacity
to no purpose. All refusals are 429, but a spent budget carries no Retry-After
because waiting does not help until the period rolls over, and saying
otherwise turns one client into a retry loop.

Usage goes to a buffered channel and is written in batches. Record never
blocks: if the buffer fills — a database stalled long enough to lose eight
thousand records — records are dropped and counted, because losing accounting
is bad and turning a stalled database into an outage is worse. A clean
shutdown drains what is queued; those are requests that were already served.
Failures are logged at error, since silently losing billing data is the kind
of thing discovered at the end of a month.

Requests are accounted whatever their outcome. A request that used a GPU and
then failed still used it, and a key that could spend nothing by failing would
be a way around a budget.

Budgets are counted in memory and restored from the store at startup, so a
restart does not hand every key a fresh allowance. The budget period travels
with the reservation rather than being re-derived, because a monthly budget
credited to a daily counter is a budget that never binds.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 08:52:36 +02:00
Ilker ManapandClaude Opus 5 45c8cd23a3 Close the gateway: API keys, tenancy and model allowlists
CI / test (push) Successful in 2m34s
Until now anything that could reach the client port could use every model in
the catalog. A key now resolves to an identity, and that identity decides what
it may do.

Authentication wraps the whole mux rather than checking inside each handler,
so adding a route cannot add an unauthenticated one by omission. Health stays
open, because a load balancer has no key and a health check that needs one
reports the gateway down.

Nothing on the request path reads the database. A key is resolved once and
answered from memory until the entry expires; rejections are cached too, so a
client looping with a bad key cannot turn into a database read per request.
The price is that a revocation takes up to cache_ttl to take effect, which is
the trade stated in the configuration next to the knob. A store failure is
not an authentication answer: it is never cached as one and returns 503, not
401 — sending someone hunting for a key problem that does not exist wastes the
one thing an incident is short of. A revoked key is refused with the same
message as an unknown one, because saying otherwise confirms the key was real.

With auth on the band comes from the key and the band header is ignored. The
header was trusted only while there was nothing to consult, and it let any
client promote itself into the top priority band. Tenant fairness within a
band is now per key rather than per session, so one caller cannot take a band
by opening many sessions.

Allowlists are decided per model, not per string: a model's canonical name and
every alias name the same thing, so either form works in an allowlist and no
alias can reach a model that was not allowed. The first version of that test
passed without the resolution being there at all, which is why it now checks
all four combinations.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 08:44:33 +02:00
Ilker ManapandClaude Opus 5 f19db242ae Stop naming a metric llama.cpp does not have, and say so when it happens
Deployed against real llama.cpp, every scrape failed. The mapping named
llamacpp:kv_cache_usage_ratio; build b10991 serves fifteen metric families
and none is a cache ratio. Because KV usage was mandatory, that one absent
name threw away the two metrics the engine does publish, and the router
scheduled to all three replicas blind while the cluster looked healthy.

A mapping that leaves a metric unnamed now means the engine does not publish
it, and the field stays zero. Naming one the engine does not serve is still a
violation, so this does not become a way to tolerate a typo. Running and
waiting can never be omitted: an engine that reports neither what it is doing
nor what is queued cannot be scheduled against at all. For llama.cpp this
costs the KV half of the saturation signal, which is a real loss softened by
its fixed slot count — queue depth already says when it is full.

The second half is why it took a deployment to notice. A contract violation
was logged per scrape at debug level, so at the default level nothing said
anything. It is now a warning, once, with a matching line when it clears.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 08:18:27 +02:00
Ilker ManapandClaude Opus 5 501db3d3e8 Add the llama.cpp engine, and let a profile ask for no GPU
llama.cpp is what makes a cluster with no cards useful: it serves the same
OpenAI API on CPU, so the whole path — placement, supervision, scraping,
scheduling, admission — can be exercised end to end on ordinary machines.

Three things had to give way for it.

A profile may now omit its GPU class, and that is how it says it runs on the
CPU. Only engines that can actually serve without a card may omit it; for
vLLM an omitted class stays an error, because otherwise a typo becomes a
replica placed on a machine with no hardware and diagnosed minutes later
during model loading. Placement spreads CPU replicas instead of bin packing
them, since unlike GPUs there is no allocation to stop two of them sharing a
machine's cores.

Metric mappings are now per engine rather than per cluster. llama.cpp calls
the same three numbers requests_processing, requests_deferred and
kv_cache_usage_ratio, and a single global mapping would leave half a mixed
fleet unmeasurable. An empty mapping still defaults to vLLM's names; one that
names its own Running metric is taken as a complete statement, so the fields
it leaves blank stay blank rather than being filled with names that engine
cannot have.

The adapter refuses multi-node and tensor-parallel profiles. llama.cpp splits
across local GPUs by a different mechanism than this adapter drives, and
accepting either would produce a replica that starts and then serves from one
machine while the catalog believes it has several.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 08:14:05 +02:00
ilkerandClaude Opus 5 e8d90cf375 Treat a zombie as dead, and test fairness where it is deterministic
CI / test (push) Successful in 52s
CI went red on Linux with two failures that never appear on macOS. Both were
worth the trip.

The first is a real bug. processAlive used kill(pid, 0), which succeeds on a
zombie — a process that has exited and is only waiting to be reaped. That
happens for real whenever an engine's parent dies before it does and nothing
reaps the orphan, which in a container is the normal case: PID 1 is usually the
workload rather than an init. The supervisor would then report a dead adopted
engine as running forever, and hold its GPUs against every future placement.
Zombies are now detected through procfs on Linux, where the answer exists, and
the check degrades to its old behaviour elsewhere rather than to something worse.

The second was a bad test. TestTenantFairnessWithinBand asserted on the order
goroutines sent to a channel after being admitted, which is the order the Go
scheduler woke them, not the order the dispatcher chose. Under load it measured
the runtime. The rotation itself lives in bandQueue.pop, which is pure, so it is
tested there — including what happens when a tenant runs out mid-rotation and
when a cancelled waiter is removed. The controller-level test now asserts the
thing that is actually stable: nobody is starved.

Verified with three consecutive `go test -race ./...` runs in a Linux container.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 06:34:08 +02:00
ilkerandClaude Opus 5 e1eb1d4149 Phase 3: ctl, and make draining a rolling move
CI / test (push) Failing after 50s
ctl talks to a management API on the admin listener, not the client-facing one.
The two have different audiences: one serves inference to whoever holds an API
key, the other lets a caller cordon nodes and read the cluster's shape.

- nodes surfaces unhealthy GPUs and why they are excluded. That is usually the
  answer when a node looks idle but nothing will schedule onto it, and it was
  invisible before.
- explain-placement prints every candidate considered and why it was rejected.
- replicas shows the shards a replica is made of, with per-shard errors.
- cordon, uncordon and drain are one endpoint with different targets, because
  the difference between cordon and drain is exactly what an operator needs to
  be sure of before running either.

Draining had to be fixed first to make ctl drain worth having. A drained replica
was stopped on the next pass, which cuts in-flight requests — the thing draining
exists to avoid. Now a node marked Draining has its replicas drained, the model
falls below its desired count, a replacement is placed elsewhere, and the
original is stopped only once the replacement is serving. A grace period bounds
it, because draining the only node that can host a model would otherwise never
finish and the node could never be taken out of service.

Verified against a two-node cluster of real binaries: draining moved a replica
to the surviving node and traffic kept returning 200 throughout.

ctl logs is deliberately absent. It needs a request/response pair the agent
protocol does not have, and the failure path already ships an engine log tail
with every status report — so the case that matters most is covered, and
reading a healthy shard's output on demand is not worth a protocol change yet.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 06:27:21 +02:00
ilkerandClaude Opus 5 2373726efd Agent stream: connect the fleet to the control plane
CI / test (push) Failing after 1m29s
One bidirectional connect-go stream per node, carrying registration, heartbeats
and shard status up, and start, stop and drain down. The agent dials out: GPU
nodes sit behind firewalls far more often than a control plane does, and an
outbound connection needs no inbound rule on every node in the fleet.

- registration carries the whole hardware inventory, so there is no separate
  discovery call and a reconnecting agent refreshes what the scheduler knows.
  The fabric labels travel with it; they cannot be inferred from the wire, so
  losing them in transit would silently disable multi-node placement.
- agentsvc is both the stream handler and the reconciler's dispatcher, because
  it holds the open streams and is therefore the only thing that can deliver a
  command to a node.
- the agent validates a profile before launching anything, since an engine that
  refuses a profile does so after minutes of loading, then supervises the
  process and reports every transition with the engine's output tail on failure.
- reconnects forever with jittered backoff. A control plane restart must not
  require touching every GPU node, and a fleet must not reconnect in lockstep
  and knock it over again. Engines keep serving throughout, because the router
  talks to them directly.

llmgw agent is real now, and llmgw server runs a control plane whenever any
model declares a profile. The static endpoint list stays for router-only runs.

Verified with real binaries as separate processes: the agent registers, a
replica is placed, an engine launches through the vLLM adapter's actual argument
construction, streaming traffic is served, and killing the agent moves the node
through unreachable to lost with placement explaining why nothing can run.

config.Model and catalog.Model were the same thing described twice, so the
router now reads catalog entries directly and the validation rules live in one
place.

Still missing: mTLS. The stream is plaintext HTTP/2 and must stay on a trusted
network until that lands.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 02:48:21 +02:00
ilkerandClaude Opus 5 c1b9ea1bc2 Phase 3 (3/3): wire the control plane to the router
CI / test (push) Successful in 50s
Ready replicas become router endpoints with no static configuration involved.
Endpoints that survive an update keep their live state — inflight counts,
breaker state, drain rate — so a replica that was already serving does not start
from cold because another one moved. The sink is an interface, so when the
router and control plane split into separate processes it becomes a watch stream
and nothing above it changes.

Phase 3's exit criterion now holds end to end. A node hosting a live replica is
killed while four clients stream continuously; the control plane marks it
unreachable, then lost, rebuilds the replica on a survivor and republishes.
Across three runs ~370 requests completed with zero failures and no replica was
left on the dead node. This proves the control logic, not the transport: the
agent fleet is simulated and the real path still needs the stream.

The race detector caught a genuine bug while this went in. startReplica read the
replica's fields after publishing it to the map, and scaleUp walked replica
pointers outside the lock; both raced with agent status reports, which arrive
from another goroutine and mutate exactly those fields. Against the simulated
fleet the reports came back fast enough to hit the window on every run. State is
snapshotted under the lock now and those functions work from values.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 02:16:35 +02:00
ilkerandClaude Opus 5 dc38dd6096 Phase 1.5: admission control
CI / test (push) Successful in 48s
Inserted between quota checks and scheduling: admission answers when a request
may proceed, the scheduler still answers where it goes. Keeping them apart is
the point — an engine queue is FIFO and knows nothing about tenants, so once a
batch job is in it, no endpoint selection rescues the request behind it.

- saturation detector: one number per pool, the worse of queue depth and KV
  occupancy, since either alone misleads.
- priority bands with per-band saturation ceilings, so the cluster degrades from
  the bottom up.
- tenant round-robin within a band: one caller sending a thousand requests takes
  one slot in the rotation, not a thousand.
- bounded queues with capacity (503) and TTL (504); cancelled requests leave the
  queue instead of holding capacity.
- metrics for queue depth, wait time, rejections and pool saturation.

Exit criterion, measured in test/e2e: with 500 concurrent bronze requests
running continuously, interactive p99 in the platinum band stays at 1.04-1.24x
idle against a 1.5x budget. With admission off the same load pushes it to
55-60x. Both directions are asserted, so the test fails if admission regresses
and also if the load generator stops being a real load.

Two lags had to be closed, and they needed different answers. Engine metrics are
scraped and lag an interval; router inflight does not, but it counts running
requests, and treating those as queue makes a healthy engine look overloaded
every interval. What is actually queued is inflight minus batch capacity, and
capacity is not in the metric contract, so it is learned as the high-water mark
of observed running. Separately, between admission granting a request and the
router dispatching it the request is invisible everywhere, so a simultaneous
burst passed the check en masse; the controller now counts its own outstanding
leases against dispatched inflight.

One plan correction: a saturation ceiling of 1.0 was documented as "never held
back". It is not — it means "flow up to the healthy limit". A band that must
never wait needs a ceiling above 1. Plan §7.2.2 updated.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 01:53:38 +02:00
ilkerandClaude Opus 5 a3f71aa5ee Phase 1: the router serves traffic
CI / test (push) Successful in 1m1s
llmgw server is real now. It fronts a static set of engine replicas with an
OpenAI-compatible API, scoring endpoints through the pipeline added in the
previous commit.

- proxy: SSE relayed event by event with a flush after each, cancellation
  propagated upstream, and separate deadlines for dial, headers, first token,
  inter-token idle and total. The idle watchdog is what distinguishes a wedged
  engine from a slow one.
- router: dispatch with retry and model fallback. Retry only happens while
  nothing has reached the client; after that, splicing two responses together
  would be worse than the short stream the client already sees.
- config: YAML via koanf overlaid on defaults, with LLMGW_ env overrides.
  Validation rejects duplicate endpoint ids and aliases that shadow a model,
  because both misroute silently at request time rather than failing loudly.
- metrics: requests, duration, TTFT, tokens, plus per-scorer values and
  per-filter drops on a separate admin listener with health and pprof.

Phase 1 exit criterion, measured in test/e2e: 1000 concurrent streams across
3 sim-engines complete at ~6800 req/s; every client disconnect frees the KV
blocks on every engine; router overhead p99 is about 0.9 ms against the plan's
2 ms budget.

The handlers live in internal/router rather than an internal/router/openai
package as the plan's tree sketched: they need the dispatch loop, and splitting
them apart would have meant an interface with exactly one implementation. Plan
§4 updated to match.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 01:41:13 +02:00
ilkerandClaude Opus 5 b7417e1a69 Phase 1 (1/2): scheduler pipeline, endpoint registry, metric scraping
The code behind plan §7.3-7.6:

- breaker: folds cooldown and circuit breaking into one component
  (FailureThreshold=1 gives LiteLLM-style cooldown), with exponential backoff
  and half-open probes.
- endpoint: live state registry. Inflight counting, staleness detection, drain
  rate derived from counters. A scrape failure does not trip the breaker — the
  engine may be serving fine while only its metrics endpoint is down.
- scrape: parses the §7.6 metric contract. Errors when any of the mandatory
  three is missing; the metric name mapping is overridable, since names drift
  between versions (plan §14).
- scheduler: the filter -> score -> pick pipeline. A "strategy" is now just a
  set of scorer weights; chat, batch and long-context are the same code with
  different weights. Scorers: queue, kv (hard penalty above the ceiling),
  inflight, load, weight, session (bounded-load consistent hashing). Pickers:
  max-score, p2c, softmax, round-robin.

Decision carries the rationale — which scorer produced what, which filter
dropped how many. Tuning the weights without that is guesswork.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 01:15:21 +02:00