The section opens on the default rather than on the feature, because the default is the part someone deploying this needs to be sure of: nothing anyone types is stored until a project asks, and the command to ask is an act rather than a setting. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
eaf3382a6d
commit
21a502035f
2 files changed
+80
No files matched your search
@@ -595,6 +595,55 @@ fleet means the measurement is missing, not that the fleet is idle. And
|
||||
`llmgw_gpu_nvlink_peers` at zero everywhere means the topology is unknown,
|
||||
which is what a binary built without the nvml tag reports.
|
||||
|
||||
## Conversations across models
|
||||
|
||||
Off everywhere until a project asks. llmgw otherwise stores no message content
|
||||
at all — usage rows carry token counts, costs and latencies, never text, and
|
||||
even its own chat page keeps the conversation in the browser. A gateway that
|
||||
keeps nothing anyone typed is a much easier thing to have in front of a team,
|
||||
so keeping it is a decision someone makes rather than a default they discover.
|
||||
|
||||
```
|
||||
llmgw keys threads --project default --retain-hours 24
|
||||
```
|
||||
|
||||
The retention is required. A conversation store with no expiry is a transcript
|
||||
of everything a team ever typed, kept forever because nobody chose a number.
|
||||
The window is measured from the last turn, so a conversation still in progress
|
||||
does not expire underneath it.
|
||||
|
||||
A client then sends a thread id and only its new turn:
|
||||
|
||||
```
|
||||
curl $BASE/v1/chat/completions \
|
||||
-H "Authorization: Bearer $LLMGW_API_KEY" \
|
||||
-H "x-llmgw-thread: my-conversation" \
|
||||
-d '{"model": "<model-name>", "messages": [{"role":"user","content":"and then?"}]}'
|
||||
```
|
||||
|
||||
llmgw puts the stored history in front of that turn and sends it to whichever
|
||||
model is serving — **including a different one from the one the conversation
|
||||
started on**. That is the point: a thread begun on a large model can continue
|
||||
on a small one, and a caller dropped to a cheaper model by a spent budget keeps
|
||||
their conversation.
|
||||
|
||||
When the history does not fit the model now serving, the oldest turns are
|
||||
dropped until it does, and the reply carries `x-llmgw-thread-trimmed` with how
|
||||
many. System messages are never dropped, and neither is the turn the caller
|
||||
just sent.
|
||||
|
||||
`DELETE /v1/threads/{id}` forgets one. It is a client endpoint on purpose: the
|
||||
person who said the thing should be able to unsay it without asking an
|
||||
operator to read it first. A thread belonging to another project answers the
|
||||
same as one that never existed.
|
||||
|
||||
Turning it off again stops new storage, and existing conversations go at the
|
||||
next hourly prune:
|
||||
|
||||
```
|
||||
llmgw keys threads --project default --off
|
||||
```
|
||||
|
||||
## Reclaiming disk
|
||||
|
||||
`llmgw ctl nodes` has a DISK column: free space against the size it is free
|
||||
|
||||
Reference in new issue
Block a user