Document conversations, starting with what is not kept
The section opens on the default rather than on the feature, because the default is the part someone deploying this needs to be sure of: nothing anyone types is stored until a project asks, and the command to ask is an act rather than a setting. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
eaf3382a6d
commit
21a502035f
2 files changed
+80
No files matched your search
@@ -18,6 +18,37 @@ APIs change between minors without a deprecation period.
|
||||
|
||||
## Unreleased
|
||||
|
||||
### Conversations that can move between models
|
||||
|
||||
A thread a client names with `x-llmgw-thread`, carrying only its new turn.
|
||||
llmgw replays the stored history to whichever model is serving, which is what
|
||||
lets a conversation begun on a large model continue on a small one — including
|
||||
when a spent budget moved it there and the caller never chose.
|
||||
|
||||
Off for every project until someone runs `llmgw keys threads`. Until then llmgw
|
||||
stores no message content anywhere, as it never has, and that default is worth
|
||||
more than the feature. The retention is required rather than optional, because
|
||||
a conversation store with no expiry is a transcript of everything a team ever
|
||||
typed kept forever by omission, and it is measured from the last turn so a
|
||||
conversation in progress does not expire underneath it.
|
||||
|
||||
History that does not fit the serving model is trimmed oldest-first and the
|
||||
reply says how many turns went, in `x-llmgw-thread-trimmed`. System messages
|
||||
are never dropped — losing one silently changes what the model is for while it
|
||||
carries on answering — and neither is the turn the caller just sent.
|
||||
|
||||
Threads are scoped to a project, and reading, writing or deleting one that
|
||||
belongs to another answers exactly as one that never existed: two answers would
|
||||
let a caller map which ids are real in projects they cannot read.
|
||||
|
||||
Capturing the reply is opt-in and bounded, so a request without a thread
|
||||
accumulates nothing on the hottest path in the system, and one with a thread
|
||||
stops at a cap rather than growing to it.
|
||||
|
||||
`DELETE /v1/threads/{id}` forgets a conversation, as a client rather than as an
|
||||
operator: the person who said the thing should not have to ask someone to read
|
||||
it before it can go.
|
||||
|
||||
### Disk, and taking back only what llmgw put there
|
||||
|
||||
Disk was the one resource nothing reported. `statfs` was already being called
|
||||
|
||||
@@ -595,6 +595,55 @@ fleet means the measurement is missing, not that the fleet is idle. And
|
||||
`llmgw_gpu_nvlink_peers` at zero everywhere means the topology is unknown,
|
||||
which is what a binary built without the nvml tag reports.
|
||||
|
||||
## Conversations across models
|
||||
|
||||
Off everywhere until a project asks. llmgw otherwise stores no message content
|
||||
at all — usage rows carry token counts, costs and latencies, never text, and
|
||||
even its own chat page keeps the conversation in the browser. A gateway that
|
||||
keeps nothing anyone typed is a much easier thing to have in front of a team,
|
||||
so keeping it is a decision someone makes rather than a default they discover.
|
||||
|
||||
```
|
||||
llmgw keys threads --project default --retain-hours 24
|
||||
```
|
||||
|
||||
The retention is required. A conversation store with no expiry is a transcript
|
||||
of everything a team ever typed, kept forever because nobody chose a number.
|
||||
The window is measured from the last turn, so a conversation still in progress
|
||||
does not expire underneath it.
|
||||
|
||||
A client then sends a thread id and only its new turn:
|
||||
|
||||
```
|
||||
curl $BASE/v1/chat/completions \
|
||||
-H "Authorization: Bearer $LLMGW_API_KEY" \
|
||||
-H "x-llmgw-thread: my-conversation" \
|
||||
-d '{"model": "<model-name>", "messages": [{"role":"user","content":"and then?"}]}'
|
||||
```
|
||||
|
||||
llmgw puts the stored history in front of that turn and sends it to whichever
|
||||
model is serving — **including a different one from the one the conversation
|
||||
started on**. That is the point: a thread begun on a large model can continue
|
||||
on a small one, and a caller dropped to a cheaper model by a spent budget keeps
|
||||
their conversation.
|
||||
|
||||
When the history does not fit the model now serving, the oldest turns are
|
||||
dropped until it does, and the reply carries `x-llmgw-thread-trimmed` with how
|
||||
many. System messages are never dropped, and neither is the turn the caller
|
||||
just sent.
|
||||
|
||||
`DELETE /v1/threads/{id}` forgets one. It is a client endpoint on purpose: the
|
||||
person who said the thing should be able to unsay it without asking an
|
||||
operator to read it first. A thread belonging to another project answers the
|
||||
same as one that never existed.
|
||||
|
||||
Turning it off again stops new storage, and existing conversations go at the
|
||||
next hourly prune:
|
||||
|
||||
```
|
||||
llmgw keys threads --project default --off
|
||||
```
|
||||
|
||||
## Reclaiming disk
|
||||
|
||||
`llmgw ctl nodes` has a DISK column: free space against the size it is free
|
||||
|
||||
Reference in new issue
Block a user