Commit Graph
3 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5 0a98d01361 Say when a chat reply was cut off, and stop cutting it so early
Reported as the model not completing a request and stopping. It did
complete — it ran out of room. The chat page caps a turn at max_tokens and
then never looked at finish_reason, so a reply that hit the cap ended
mid-word and looked exactly like one that was finished.

The silence was the real defect. A reader has no way to tell "the model had
nothing more to say" from "the page stopped it", and the only clue is that
the last sentence ends abruptly.

512 was also too low. It was chosen when every model here was a 0.5B
answering in a paragraph; it is about a screenful of code, so a coding
model gets cut off inside the first function it writes. 2048 now, which is
still a bound — the reason for having one has not changed.

The notice is appended beside the reply, never into it: that element holds
model output placed with textContent, and building markup in the same node
is how the careful handling everywhere else gets undone.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 21:03:07 +02:00
Ilker ManapandClaude Opus 5 72b03a1a1c Say which model is simulated instead of letting it look broken
big-model runs on the sim engine, whose whole vocabulary is the Greek
alphabet, cycled at a fixed rate until max_tokens. That is what it is for:
with no H100s it is the only thing exercising the TP8 two-node path, the
fabric invariant and rebuild-after-node-loss. The output is the point.

What was wrong is that nothing said so. /v1/models returned it with
owned_by "llmgw", identical to the real model, and the chat picker sorted
purely by name — so "big-model" led the list, was preselected by the
browser, and anyone who typed a question and pressed Send got Greek
letters back with no explanation. Reported as the model spitting out crap,
which is a fair reading of what it looks like from outside.

Simulated is derived from the profiles rather than configured: an operator
who adds a sim profile and forgets a flag would be shipping placeholder
text under a real model's name. A model counts as simulated only when
every profile is, because one real profile means the request can still
come back as inference.

The marker on /v1/models is owned_by "llmgw-sim", which is already what
the sim engine calls itself on its own /v1/models — the gateway was
flattening a distinction that existed one layer down.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 17:11:56 +02:00
Ilker ManapandClaude Opus 5 cbf276a41b Render maths in the chat, without letting the model write markup
Replies are plain text, so a model asked about an integral answers with
\int_0^1 x^2 dx and the reader gets the source.

KaTeX is vendored into the binary rather than loaded from a CDN. This gateway
makes no outbound requests of its own and is expected to run where it could
not make one; a chat page that silently loses its maths on an air-gapped
cluster would be worse than one that never had it. 600 KB of JS, CSS and WOFF2
fonts, which the stripped binary absorbs without changing its rounded size.

The ordering is the safety argument, and there is a test for it rather than a
comment alone. The reply is put in place with textContent exactly as before,
and only then walked for maths: renderMathInElement replaces the spans it
recognises, so nothing the model wrote is ever parsed as HTML and the markup
that appears is KaTeX's. trust stays false — KaTeX's default, and what keeps
\href and \includegraphics out of a reply — and throwOnError is off so one
malformed expression does not take the rest with it.

Rendered once the reply is complete. Typesetting each chunk would try to set
half an expression on every token and flicker through the whole answer.

The static handler now serves a small embedded asset tree rather than one
file, so it moved to its own embed: a file server over the tree that also
carries the templates would serve those too. Directory listings are refused —
nothing links to one, so the only requests are from somebody looking around.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 17:03:46 +02:00