Document conversations, starting with what is not kept

The section opens on the default rather than on the feature, because the
default is the part someone deploying this needs to be sure of: nothing anyone
types is stored until a project asks, and the command to ask is an act rather
than a setting.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-18 11:58:53 +02:00
1 parent eaf3382a6d
commit 21a502035f
2 files changed
+80

No files matched your search

+49
View File
@@ -595,6 +595,55 @@ fleet means the measurement is missing, not that the fleet is idle. And
`llmgw_gpu_nvlink_peers` at zero everywhere means the topology is unknown,
which is what a binary built without the nvml tag reports.
## Conversations across models
Off everywhere until a project asks. llmgw otherwise stores no message content
at all — usage rows carry token counts, costs and latencies, never text, and
even its own chat page keeps the conversation in the browser. A gateway that
keeps nothing anyone typed is a much easier thing to have in front of a team,
so keeping it is a decision someone makes rather than a default they discover.
```
llmgw keys threads --project default --retain-hours 24
```
The retention is required. A conversation store with no expiry is a transcript
of everything a team ever typed, kept forever because nobody chose a number.
The window is measured from the last turn, so a conversation still in progress
does not expire underneath it.
A client then sends a thread id and only its new turn:
```
curl $BASE/v1/chat/completions \
-H "Authorization: Bearer $LLMGW_API_KEY" \
-H "x-llmgw-thread: my-conversation" \
-d '{"model": "<model-name>", "messages": [{"role":"user","content":"and then?"}]}'
```
llmgw puts the stored history in front of that turn and sends it to whichever
model is serving — **including a different one from the one the conversation
started on**. That is the point: a thread begun on a large model can continue
on a small one, and a caller dropped to a cheaper model by a spent budget keeps
their conversation.
When the history does not fit the model now serving, the oldest turns are
dropped until it does, and the reply carries `x-llmgw-thread-trimmed` with how
many. System messages are never dropped, and neither is the turn the caller
just sent.
`DELETE /v1/threads/{id}` forgets one. It is a client endpoint on purpose: the
person who said the thing should be able to unsay it without asking an
operator to read it first. A thread belonging to another project answers the
same as one that never existed.
Turning it off again stops new storage, and existing conversations go at the
next hourly prune:
```
llmgw keys threads --project default --off
```
## Reclaiming disk
`llmgw ctl nodes` has a DISK column: free space against the size it is free