Document conversations, starting with what is not kept

The section opens on the default rather than on the feature, because the
default is the part someone deploying this needs to be sure of: nothing anyone
types is stored until a project asks, and the command to ask is an act rather
than a setting.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-18 11:58:53 +02:00
1 parent eaf3382a6d
commit 21a502035f
2 files changed
+80

No files matched your search

+31
View File
@@ -18,6 +18,37 @@ APIs change between minors without a deprecation period.
## Unreleased
### Conversations that can move between models
A thread a client names with `x-llmgw-thread`, carrying only its new turn.
llmgw replays the stored history to whichever model is serving, which is what
lets a conversation begun on a large model continue on a small one — including
when a spent budget moved it there and the caller never chose.
Off for every project until someone runs `llmgw keys threads`. Until then llmgw
stores no message content anywhere, as it never has, and that default is worth
more than the feature. The retention is required rather than optional, because
a conversation store with no expiry is a transcript of everything a team ever
typed kept forever by omission, and it is measured from the last turn so a
conversation in progress does not expire underneath it.
History that does not fit the serving model is trimmed oldest-first and the
reply says how many turns went, in `x-llmgw-thread-trimmed`. System messages
are never dropped — losing one silently changes what the model is for while it
carries on answering — and neither is the turn the caller just sent.
Threads are scoped to a project, and reading, writing or deleting one that
belongs to another answers exactly as one that never existed: two answers would
let a caller map which ids are real in projects they cannot read.
Capturing the reply is opt-in and bounded, so a request without a thread
accumulates nothing on the hottest path in the system, and one with a thread
stops at a cap rather than growing to it.
`DELETE /v1/threads/{id}` forgets a conversation, as a client rather than as an
operator: the person who said the thing should not have to ask someone to read
it before it can go.
### Disk, and taking back only what llmgw put there
Disk was the one resource nothing reported. `statfs` was already being called
+49
View File
@@ -595,6 +595,55 @@ fleet means the measurement is missing, not that the fleet is idle. And
`llmgw_gpu_nvlink_peers` at zero everywhere means the topology is unknown,
which is what a binary built without the nvml tag reports.
## Conversations across models
Off everywhere until a project asks. llmgw otherwise stores no message content
at all — usage rows carry token counts, costs and latencies, never text, and
even its own chat page keeps the conversation in the browser. A gateway that
keeps nothing anyone typed is a much easier thing to have in front of a team,
so keeping it is a decision someone makes rather than a default they discover.
```
llmgw keys threads --project default --retain-hours 24
```
The retention is required. A conversation store with no expiry is a transcript
of everything a team ever typed, kept forever because nobody chose a number.
The window is measured from the last turn, so a conversation still in progress
does not expire underneath it.
A client then sends a thread id and only its new turn:
```
curl $BASE/v1/chat/completions \
-H "Authorization: Bearer $LLMGW_API_KEY" \
-H "x-llmgw-thread: my-conversation" \
-d '{"model": "<model-name>", "messages": [{"role":"user","content":"and then?"}]}'
```
llmgw puts the stored history in front of that turn and sends it to whichever
model is serving — **including a different one from the one the conversation
started on**. That is the point: a thread begun on a large model can continue
on a small one, and a caller dropped to a cheaper model by a spent budget keeps
their conversation.
When the history does not fit the model now serving, the oldest turns are
dropped until it does, and the reply carries `x-llmgw-thread-trimmed` with how
many. System messages are never dropped, and neither is the turn the caller
just sent.
`DELETE /v1/threads/{id}` forgets one. It is a client endpoint on purpose: the
person who said the thing should be able to unsay it without asking an
operator to read it first. A thread belonging to another project answers the
same as one that never existed.
Turning it off again stops new storage, and existing conversations go at the
next hourly prune:
```
llmgw keys threads --project default --off
```
## Reclaiming disk
`llmgw ctl nodes` has a DISK column: free space against the size it is free