The commonest convention of all, and the one the extractor missed. Asked
for a two-file Go server, qwen2.5-coder-3b opened a plain ```go fence and
wrote "// cmd/server/main.go" as the first line of the file — not on the
fence, not above it. Both files downloaded as snippet-1.go and
snippet-2.go, which is the feature failing at the first real use of it.
Found by running it rather than by reading it: the unit tests covered the
conventions I had thought of, and passed throughout.
Only the first line or two, only when the line is a comment holding one
token that looks like a path. "// this reads config.json at startup" is a
sentence, "// Package main implements the server." is a doc comment, and a
path comment further down the file is just a comment. The markers cover the
languages people actually ask for, including block comments closed on the
same line.
The comment stays in the body. People write a path at the top of a file
anyway, and editing model output to make a filename look tidier is a
surprise nobody asked for.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Asking a coding model for anything real produces a small tree, and the only
way to get it out of the page was to select each block and paste it
somewhere. A reply containing fenced code now offers a zip, with the
directory structure the model described.
Server-side, because the interesting part is the filenames. They are written
by a language model and they go into an archive that somebody extracts with
a tool that trusts archives, so "../../.ssh/authorized_keys" and
"/etc/cron.d/x" are inputs to expect rather than hope against. archive/zip
and a sanitiser with tests beat string handling in a browser. An absolute
path is made relative — it cannot escape once the leading separator is gone,
and refusing a file over a leading slash throws away work. Anything that
still escapes after cleaning is rejected rather than repaired, because
silently turning ../../evil.go into evil.go puts a file somewhere the reader
did not expect; it gets a generated name instead.
Naming follows what models actually do: on the fence (```go:cmd/main.go,
```main.go, ```go main.go, title="x"), or on the line above it as a heading
or comment. Only the two lines immediately above, so a sentence mentioning
main.go is not read as a filename. A block nobody named still comes back
under a generated name, because losing code because it was unlabelled is
worse than naming it badly, and two blocks claiming one path both survive —
models show a file and then show it again with one line changed.
Nothing is stored: the reply arrives, the zip goes back, neither is kept.
The endpoint sits behind the same session guard and CSRF check as
/chat/send.
Two of these tests asserted the wrong thing when first written — that an
absolute path must be refused, and that hyphens are stripped from the
archive name. Both were corrected to what the code should do rather than
the code changed to match them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reported as the model not completing a request and stopping. It did
complete — it ran out of room. The chat page caps a turn at max_tokens and
then never looked at finish_reason, so a reply that hit the cap ended
mid-word and looked exactly like one that was finished.
The silence was the real defect. A reader has no way to tell "the model had
nothing more to say" from "the page stopped it", and the only clue is that
the last sentence ends abruptly.
512 was also too low. It was chosen when every model here was a 0.5B
answering in a paragraph; it is about a screenful of code, so a coding
model gets cut off inside the first function it writes. 2048 now, which is
still a bound — the reason for having one has not changed.
The notice is appended beside the reply, never into it: that element holds
model output placed with textContent, and building markup in the same node
is how the careful handling everywhere else gets undone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A user copied the curl from the profile page and sent it as it stood. Every
other field in that snippet is filled in for them — the base URL, the key
they had just been issued — so "model": "MODEL" reads as a value rather
than a blank, and they got:
no endpoint serves "MODEL"
which is true and useless. It reads the same whether the name was a typo,
a placeholder left in from the documentation, or a real outage, and it
never says what would have worked.
The message now lists what is being served, capped at ten with a count for
the rest, and says plainly when nothing is. That list is what /v1/models
already returns to the same caller, so it discloses nothing new.
The profile page's example now names a model that is actually running,
real ones before simulated — the point of an example someone can run is
that it runs, and sending a first request to the sim engine would answer
in Greek letters. The landing and claim pages keep a placeholder, because
they are served without a key and deliberately say nothing about what is
here; theirs is <model-name>, which cannot be mistaken for a value, with a
line saying where to get the real one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
big-model runs on the sim engine, whose whole vocabulary is the Greek
alphabet, cycled at a fixed rate until max_tokens. That is what it is for:
with no H100s it is the only thing exercising the TP8 two-node path, the
fabric invariant and rebuild-after-node-loss. The output is the point.
What was wrong is that nothing said so. /v1/models returned it with
owned_by "llmgw", identical to the real model, and the chat picker sorted
purely by name — so "big-model" led the list, was preselected by the
browser, and anyone who typed a question and pressed Send got Greek
letters back with no explanation. Reported as the model spitting out crap,
which is a fair reading of what it looks like from outside.
Simulated is derived from the profiles rather than configured: an operator
who adds a sim profile and forgets a flag would be shipping placeholder
text under a real model's name. A model counts as simulated only when
every profile is, because one real profile means the request can still
come back as inference.
The marker on /v1/models is owned_by "llmgw-sim", which is already what
the sim engine calls itself on its own /v1/models — the gateway was
flattening a distinction that existed one layer down.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replies are plain text, so a model asked about an integral answers with
\int_0^1 x^2 dx and the reader gets the source.
KaTeX is vendored into the binary rather than loaded from a CDN. This gateway
makes no outbound requests of its own and is expected to run where it could
not make one; a chat page that silently loses its maths on an air-gapped
cluster would be worse than one that never had it. 600 KB of JS, CSS and WOFF2
fonts, which the stripped binary absorbs without changing its rounded size.
The ordering is the safety argument, and there is a test for it rather than a
comment alone. The reply is put in place with textContent exactly as before,
and only then walked for maths: renderMathInElement replaces the spans it
recognises, so nothing the model wrote is ever parsed as HTML and the markup
that appears is KaTeX's. trust stays false — KaTeX's default, and what keeps
\href and \includegraphics out of a reply — and throwOnError is off so one
malformed expression does not take the rest with it.
Rendered once the reply is complete. Typesetting each chunk would try to set
half an expression on every token and flicker through the whole answer.
The static handler now serves a small embedded asset tree rather than one
file, so it moved to its own embed: a file server over the tree that also
carries the templates would serve those too. Directory listings are refused —
nothing links to one, so the only requests are from somebody looking around.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Clicking "metrics" on the web dashboard answered "no api key was provided".
The page is served twice and its navigation assumed one of them. Metrics,
pprof and the management API exist on the admin listener; account
administration, the profile and the chat exist on the client-facing port. The
nav offered the first set in both places, so on the public port those links
fell through to the API-key middleware — which is a far more confusing way to
say "that is not here" than a 404 would have been.
Each page now offers what is actually reachable from it, and the public one
says where the rest lives rather than pretending it does not exist, including
why that port is not exposed: pprof can dump the process heap, and the heap
holds every API key in plaintext.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
So someone can try a model without setting up a client, and — more often —
tell whether a problem is the gateway or their code.
The page carries a session, and the server turns that into the identity an API
key would have produced, resolved from the person's project. The request then
goes down the ordinary path by being handed to the same handler /v1 uses:
admission, model allowlist, rate limits, budget, accounting. There is exactly
one request path, so there is no second place for its rules to drift — and
without that, a playground is a hole in every limit the rest of the gateway
enforces. The allowlist test posts a forbidden model and expects 403 for that
reason.
The body is rebuilt rather than forwarded, so the page cannot set fields it has
no business setting: a stream:false that would hang the reader, a max_tokens
large enough to occupy a replica for an hour. One turn is capped at 512 tokens.
The picker offers what is actually served and allowed, not what is catalogued,
because offering a model with no replica produces a 503 the reader cannot
interpret.
Inline vanilla JavaScript, for the same reason the charts are hand-drawn SVG.
It streams, because on CPU inference ten seconds is normal and a chat that
hangs and then dumps a paragraph feels broken. Replies are written with
textContent throughout: model output and user input have no business being
parsed as markup. Its POSTs carry the CSRF token in a header, which a
cross-site form cannot set at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two things the dashboard was missing: it could not act, and it was a column of
numbers.
Account administration moves to /admin/users — approving into a project,
suspending, reinstating. That is the work an operator does constantly and it
should not need a terminal. Three things stay on the CLI, and the line is not
arbitrary: promoting someone to admin, resetting a password, and issuing a key
on another person's behalf each turn a stolen admin session into more than it
was — permanent access, or somebody else's account. Approving and suspending
do not. Suspending signs the account out everywhere, or suspension is advisory
until the session happens to expire, and an admin cannot suspend themselves,
which is not a thing an interface should let you do by misclicking.
The charts are inline SVG generated in Go. A charting library would mean a
JavaScript toolchain in CI and a megabyte of bundle for what is a list of
rectangles, and the single-binary goal rules that out. Hourly over a day: fine
enough to show a working day, coarse enough that one slow minute is not an
incident. Empty hours are filled in, because a chart with hours missing where
nothing happened reads as a chart with hours missing from the data.
The SVG is marked as trusted HTML, so everything that reaches it is either a
number this package computed or escaped explicitly, and a bar's colour can only
be one of four names — a colour that could be anything is a way to write CSS
into every page the chart appears on. There are tests for both.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reaching it meant an SSH tunnel, because the page lived on the admin listener
and that listener is bound to loopback. Opening the port was never the way to
fix that: it also serves pprof, and pprof can dump the process heap, where
every API key sits in plaintext. Exposing the port would have been exposing
the key store.
So the page moves rather than the port. The same dashboard is now mounted at
/admin on the client-facing port, behind the session and CSRF machinery that
already guards /profile, gated on a flag on the account. The admin listener
keeps pprof and metrics and stays on loopback.
One bit, not a role system: it grants looking. Everything that changes the
cluster or issues a credential stays on the CLI, where it needs a shell rather
than a session — which keeps the worst case of a stolen admin cookie to "saw
the cluster" rather than "drained your fleet".
Someone without the flag gets 404, not 403. A 403 confirms the page is there
and that admin accounts exist here, which is free reconnaissance; 404 is what
every other unknown path returns. The profile page does not show a link to a
door the visitor cannot open.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The account system shipped without either, which made the first forgotten
password an unanswerable support request.
Changing it requires the current one. A session is what an attacker gets from
a borrowed laptop or a stolen cookie; the password is what they do not have,
and without the check the first thing they would do is take the account.
Every session is closed, including the browser doing the changing — leaving
the others open would mean a password change does nothing about the reason
someone is changing it. A new session is then opened for the browser in front
of us, so the person who just proved their password is not thrown back to the
sign-in page. That response also carries a new CSRF cookie, so the forms it
renders take the new token rather than the one the request arrived with; there
is a test that posts a form from that exact page.
Guessing the current password is bounded the same way sign-in is, because
someone holding a stolen session should not get unlimited attempts at the
password that would let them keep it.
There is no self-service reset: this gateway sends no email, and a reset link
with nowhere to send it is not a reset. `llmgw users reset-password` is the
operator's answer, and it closes their sessions too — if someone lost their
password, the reason might be that somebody else has it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every key needed an operator with a shell. This adds the login system that
made that necessary, which is the thing I argued against building until it was
asked for — so it is built as if it will be attacked, because a public login
form is.
Signing up produces an account that can do nothing. Approval is what makes it
work and stays on the CLI: having a shell on the control plane is what makes
someone an administrator, and that is a smaller thing to get right than roles
and an admin UI that mints credentials.
The details that matter:
Passwords are argon2id, where a slow hash is correct — the input is chosen by
a human and has a dictionary, which is the opposite of an API key. Both kinds
of authentication now live in this codebase and they are in separate packages
so neither set of assumptions leaks into the other. Length is the only rule:
composition rules push people towards Password1! and cost more entropy than
they buy.
Signup never says whether an address already has an account, and sign-in
answers the same way for a wrong password and a missing account — verifying
against a dummy hash when there is no user, so a stopwatch cannot enumerate
accounts. Account state is checked after the password, not before, so guessing
an address reveals nothing about it. Guessing is bounded per address rather
than per IP: an attacker has many addresses to come from and one to guess
against, and limiting by IP locks out an office behind one NAT.
Sessions are server-side, so signing out and suspending take effect
immediately rather than whenever a token expires; suspension is checked in the
same query that resolves the session, because a second round trip is a check
somebody eventually forgets. Every state-changing form carries a CSRF token
verified against a hash in the session — a cookie authenticates the browser,
not the page that asked.
Key ownership is enforced in the UPDATE rather than by reading first, so a
guessed key id from another account changes nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The page explained the API, said keys come from an operator, and left the
reader there. Someone who reaches the address without a key had no route to
getting one, which is where onboarding stopped.
`landing.contact` is that route, because the gateway cannot guess who to ask.
An address becomes a mailto link, an http(s) URL stays a link, and anything
else — a channel, a queue, a person — is shown as text. Other schemes are
never linked: this string comes from configuration, configuration is not
always written by whoever runs the gateway, and a javascript: href on a page
served to every visitor is a short path to a stored XSS. Escaping the text
does not make a scheme safe.
Unconfigured, it now says something rather than nothing, and the page explains
that a key arrives as a one-time link so the reader knows what to expect
before it lands.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Issuing a key worked; getting it to the person it was issued for did not. The
operator had a secret on their terminal and no way to send it that did not
leave a bearer credential in a chat log forever.
`keys create --claim` prints a link instead. GET shows a button and consumes
nothing; POST reveals the key and burns the link. That split is not politeness
— chat clients, mail scanners and link previewers fetch URLs the moment they
are posted, so a GET that consumed the claim would mean the first thing to
unfurl the link destroys the key before anyone reads it.
A pending claim has to hold the key somewhere, which would otherwise put a
working credential in the database and undo what hashing bought. So the key is
sealed with a key derived from the link's own token, and the token is never
stored — only its hash. A database leak yields two useless hashes and a blob
nothing in the database can open. HKDF rather than the raw token, so the value
used for lookup can never become the value that decrypts.
Consuming is one UPDATE moving claimed_at off NULL, so two people opening the
same link produce one key rather than two copies. A used link, an expired one
and a token that was never real all render identically, because saying which
confirms that a token was once valid. The page is no-store, no-referrer and
noindex: the URL is itself the credential.
`GET /me` reports a key's own limits and spend, so nobody has to ask an
operator to read a dashboard they cannot reach. Outside /v1, because it is not
part of the OpenAI API.
docs/onboarding.md is written for whoever operates this, including the parts
that are deliberately absent — no self-service, no SSO, and why.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Pasting the gateway's address into a browser returned 404, which looks like a
broken endpoint rather than one that tells you how to use it.
Two pages, and the split between them is a security boundary rather than a
layout choice. The landing page is served without a key — it has to be,
because the person reading it is the one who does not have one yet — so it
contains no data at all: it says how to connect, not what is here. Model names
stay behind /v1/models. It reconstructs the address from the forwarded headers
so the copy-pasteable example names the address the client actually used
rather than the internal one; those headers are reflected into displayed text
and never used for a decision.
The dashboard shows nodes, models, replicas, placement failures and usage, and
lives on the admin listener only. That listener is loopback-bound and also
serves pprof, which can dump the process heap — where API keys sit in
plaintext — so reaching the page from elsewhere is an SSH tunnel's job. It is
read-only: anything that mints a credential stays in `llmgw keys`, where it
needs a shell rather than a browser session this project has no way to
authenticate.
Mounting the landing page uncovered a hole in the exemption list. An entry
ending in "/" was a prefix rule, and "/" ends in "/", so exempting the landing
page would have exempted every path there is — authentication still installed,
still logging, and letting the whole API through. "/" is now exact, with a
test enumerating what must and must not be exempt.
Server-rendered templates and one stylesheet, embedded. The single-binary goal
rules out a JavaScript toolchain and nothing on either page needs one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
