The request path half. A caller sets x-llmgw-thread and sends only the new
turn; llmgw puts the stored history in front of it, sends that to whichever
model is serving, and stores the exchange afterwards.
A header rather than a body field because some SDKs drop fields they do not
know and all of them let a caller set a header.
Capturing the reply is opt-in and bounded. This is the hottest path in the
system, so a request without a thread accumulates nothing at all, and one with
a thread stops accumulating at a cap rather than growing to it — a gateway
that kept every reply in memory would be one long answer away from trouble. A
reply is truncated on a rune boundary rather than dropped: a shortened turn
replays as a shortened turn, while nothing at all replays as the assistant
having said nothing, and invalid UTF-8 would break the next turn's body for
good rather than for one turn.
The streamed path sets the captured reply on every exit, not only the one that
looks like the end. A stream can also finish with the client gone or the
engine stalled, and what the assistant said up to that point is still what it
said.
A thread on a project that keeps no conversations is refused. The e2e test
asserted that and got a cheerful 200: the request was served, nothing was
stored, and the caller would have discovered it only when the model had no
idea what they were referring to — which reads as the model being broken. It
now asks whether storage is on before doing anything else.
History is spliced after the model is finally settled, because a budget
fallback may have changed it and trimming is measured against the context of
the model that will actually serve.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The part that makes a thread work across models: the history goes in front of
what the caller just said, because a client using a thread sends the new turn
rather than the conversation again.
Trimming has three rules, and each is there because the alternative fails
quietly. The oldest stored turns go first, so a conversation that outgrew the
model now serving it stays usable — after a budget fallback that model was not
the caller's choice, and a permanently unusable thread would be a strange way
to punish them for running out of budget. The caller's own messages are never
dropped: answering something they did not ask is worse than refusing. And a
system message is kept wherever it sits, because losing one silently changes
what the model is for while it carries on answering.
A model that declares no context length has nothing trimmed. Guessing a limit
for it would drop turns for no reason and the caller would never learn why the
model forgot.
The body is rebuilt through a map so every field the client sent survives.
Forwarding fields llmgw does not understand is how engine-specific options
reach an engine at all, and a thread must not be what starts dropping them.
An empty history returns the body byte for byte, so a request without a thread
takes exactly the path it took before this existed.
Not yet wired to the request path.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>