Document conversations, starting with what is not kept
The section opens on the default rather than on the feature, because the default is the part someone deploying this needs to be sure of: nothing anyone types is stored until a project asks, and the command to ask is an act rather than a setting. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
eaf3382a6d
commit
21a502035f
2 files changed
+80
No files matched your search
@@ -595,6 +595,55 @@ fleet means the measurement is missing, not that the fleet is idle. And
|
||||
`llmgw_gpu_nvlink_peers` at zero everywhere means the topology is unknown,
|
||||
which is what a binary built without the nvml tag reports.
|
||||
|
||||
## Conversations across models
|
||||
|
||||
Off everywhere until a project asks. llmgw otherwise stores no message content
|
||||
at all — usage rows carry token counts, costs and latencies, never text, and
|
||||
even its own chat page keeps the conversation in the browser. A gateway that
|
||||
keeps nothing anyone typed is a much easier thing to have in front of a team,
|
||||
so keeping it is a decision someone makes rather than a default they discover.
|
||||
|
||||
```
|
||||
llmgw keys threads --project default --retain-hours 24
|
||||
```
|
||||
|
||||
The retention is required. A conversation store with no expiry is a transcript
|
||||
of everything a team ever typed, kept forever because nobody chose a number.
|
||||
The window is measured from the last turn, so a conversation still in progress
|
||||
does not expire underneath it.
|
||||
|
||||
A client then sends a thread id and only its new turn:
|
||||
|
||||
```
|
||||
curl $BASE/v1/chat/completions \
|
||||
-H "Authorization: Bearer $LLMGW_API_KEY" \
|
||||
-H "x-llmgw-thread: my-conversation" \
|
||||
-d '{"model": "<model-name>", "messages": [{"role":"user","content":"and then?"}]}'
|
||||
```
|
||||
|
||||
llmgw puts the stored history in front of that turn and sends it to whichever
|
||||
model is serving — **including a different one from the one the conversation
|
||||
started on**. That is the point: a thread begun on a large model can continue
|
||||
on a small one, and a caller dropped to a cheaper model by a spent budget keeps
|
||||
their conversation.
|
||||
|
||||
When the history does not fit the model now serving, the oldest turns are
|
||||
dropped until it does, and the reply carries `x-llmgw-thread-trimmed` with how
|
||||
many. System messages are never dropped, and neither is the turn the caller
|
||||
just sent.
|
||||
|
||||
`DELETE /v1/threads/{id}` forgets one. It is a client endpoint on purpose: the
|
||||
person who said the thing should be able to unsay it without asking an
|
||||
operator to read it first. A thread belonging to another project answers the
|
||||
same as one that never existed.
|
||||
|
||||
Turning it off again stops new storage, and existing conversations go at the
|
||||
next hourly prune:
|
||||
|
||||
```
|
||||
llmgw keys threads --project default --off
|
||||
```
|
||||
|
||||
## Reclaiming disk
|
||||
|
||||
`llmgw ctl nodes` has a DISK column: free space against the size it is free
|
||||
|
||||
Reference in new issue
Block a user