Document conversations, starting with what is not kept
CI / test (push) Successful in 1m35s

The section opens on the default rather than on the feature, because the
default is the part someone deploying this needs to be sure of: nothing anyone
types is stored until a project asks, and the command to ask is an act rather
than a setting.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-18 11:58:53 +02:00
1 parent eaf3382a6d
commit 21a502035f
2 files changed
+80

No files matched your search

+31
View File
@@ -18,6 +18,37 @@ APIs change between minors without a deprecation period.
## Unreleased
### Conversations that can move between models
A thread a client names with `x-llmgw-thread`, carrying only its new turn.
llmgw replays the stored history to whichever model is serving, which is what
lets a conversation begun on a large model continue on a small one — including
when a spent budget moved it there and the caller never chose.
Off for every project until someone runs `llmgw keys threads`. Until then llmgw
stores no message content anywhere, as it never has, and that default is worth
more than the feature. The retention is required rather than optional, because
a conversation store with no expiry is a transcript of everything a team ever
typed kept forever by omission, and it is measured from the last turn so a
conversation in progress does not expire underneath it.
History that does not fit the serving model is trimmed oldest-first and the
reply says how many turns went, in `x-llmgw-thread-trimmed`. System messages
are never dropped — losing one silently changes what the model is for while it
carries on answering — and neither is the turn the caller just sent.
Threads are scoped to a project, and reading, writing or deleting one that
belongs to another answers exactly as one that never existed: two answers would
let a caller map which ids are real in projects they cannot read.
Capturing the reply is opt-in and bounded, so a request without a thread
accumulates nothing on the hottest path in the system, and one with a thread
stops at a cap rather than growing to it.
`DELETE /v1/threads/{id}` forgets a conversation, as a client rather than as an
operator: the person who said the thing should not have to ask someone to read
it before it can go.
### Disk, and taking back only what llmgw put there
Disk was the one resource nothing reported. `statfs` was already being called
+49
View File
@@ -595,6 +595,55 @@ fleet means the measurement is missing, not that the fleet is idle. And
`llmgw_gpu_nvlink_peers` at zero everywhere means the topology is unknown,
which is what a binary built without the nvml tag reports.
## Conversations across models
Off everywhere until a project asks. llmgw otherwise stores no message content
at all — usage rows carry token counts, costs and latencies, never text, and
even its own chat page keeps the conversation in the browser. A gateway that
keeps nothing anyone typed is a much easier thing to have in front of a team,
so keeping it is a decision someone makes rather than a default they discover.
```
llmgw keys threads --project default --retain-hours 24
```
The retention is required. A conversation store with no expiry is a transcript
of everything a team ever typed, kept forever because nobody chose a number.
The window is measured from the last turn, so a conversation still in progress
does not expire underneath it.
A client then sends a thread id and only its new turn:
```
curl $BASE/v1/chat/completions \
-H "Authorization: Bearer $LLMGW_API_KEY" \
-H "x-llmgw-thread: my-conversation" \
-d '{"model": "<model-name>", "messages": [{"role":"user","content":"and then?"}]}'
```
llmgw puts the stored history in front of that turn and sends it to whichever
model is serving — **including a different one from the one the conversation
started on**. That is the point: a thread begun on a large model can continue
on a small one, and a caller dropped to a cheaper model by a spent budget keeps
their conversation.
When the history does not fit the model now serving, the oldest turns are
dropped until it does, and the reply carries `x-llmgw-thread-trimmed` with how
many. System messages are never dropped, and neither is the turn the caller
just sent.
`DELETE /v1/threads/{id}` forgets one. It is a client endpoint on purpose: the
person who said the thing should be able to unsay it without asking an
operator to read it first. A thread belonging to another project answers the
same as one that never existed.
Turning it off again stops new storage, and existing conversations go at the
next hourly prune:
```
llmgw keys threads --project default --off
```
## Reclaiming disk
`llmgw ctl nodes` has a DISK column: free space against the size it is free