The section opens on the default rather than on the feature, because the default is the part someone deploying this needs to be sure of: nothing anyone types is stored until a project asks, and the command to ask is an act rather than a setting. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
eaf3382a6d
commit
21a502035f
2 files changed
+80
No files matched your search
@@ -18,6 +18,37 @@ APIs change between minors without a deprecation period.
|
||||
|
||||
## Unreleased
|
||||
|
||||
### Conversations that can move between models
|
||||
|
||||
A thread a client names with `x-llmgw-thread`, carrying only its new turn.
|
||||
llmgw replays the stored history to whichever model is serving, which is what
|
||||
lets a conversation begun on a large model continue on a small one — including
|
||||
when a spent budget moved it there and the caller never chose.
|
||||
|
||||
Off for every project until someone runs `llmgw keys threads`. Until then llmgw
|
||||
stores no message content anywhere, as it never has, and that default is worth
|
||||
more than the feature. The retention is required rather than optional, because
|
||||
a conversation store with no expiry is a transcript of everything a team ever
|
||||
typed kept forever by omission, and it is measured from the last turn so a
|
||||
conversation in progress does not expire underneath it.
|
||||
|
||||
History that does not fit the serving model is trimmed oldest-first and the
|
||||
reply says how many turns went, in `x-llmgw-thread-trimmed`. System messages
|
||||
are never dropped — losing one silently changes what the model is for while it
|
||||
carries on answering — and neither is the turn the caller just sent.
|
||||
|
||||
Threads are scoped to a project, and reading, writing or deleting one that
|
||||
belongs to another answers exactly as one that never existed: two answers would
|
||||
let a caller map which ids are real in projects they cannot read.
|
||||
|
||||
Capturing the reply is opt-in and bounded, so a request without a thread
|
||||
accumulates nothing on the hottest path in the system, and one with a thread
|
||||
stops at a cap rather than growing to it.
|
||||
|
||||
`DELETE /v1/threads/{id}` forgets a conversation, as a client rather than as an
|
||||
operator: the person who said the thing should not have to ask someone to read
|
||||
it before it can go.
|
||||
|
||||
### Disk, and taking back only what llmgw put there
|
||||
|
||||
Disk was the one resource nothing reported. `statfs` was already being called
|
||||
|
||||
@@ -595,6 +595,55 @@ fleet means the measurement is missing, not that the fleet is idle. And
|
||||
`llmgw_gpu_nvlink_peers` at zero everywhere means the topology is unknown,
|
||||
which is what a binary built without the nvml tag reports.
|
||||
|
||||
## Conversations across models
|
||||
|
||||
Off everywhere until a project asks. llmgw otherwise stores no message content
|
||||
at all — usage rows carry token counts, costs and latencies, never text, and
|
||||
even its own chat page keeps the conversation in the browser. A gateway that
|
||||
keeps nothing anyone typed is a much easier thing to have in front of a team,
|
||||
so keeping it is a decision someone makes rather than a default they discover.
|
||||
|
||||
```
|
||||
llmgw keys threads --project default --retain-hours 24
|
||||
```
|
||||
|
||||
The retention is required. A conversation store with no expiry is a transcript
|
||||
of everything a team ever typed, kept forever because nobody chose a number.
|
||||
The window is measured from the last turn, so a conversation still in progress
|
||||
does not expire underneath it.
|
||||
|
||||
A client then sends a thread id and only its new turn:
|
||||
|
||||
```
|
||||
curl $BASE/v1/chat/completions \
|
||||
-H "Authorization: Bearer $LLMGW_API_KEY" \
|
||||
-H "x-llmgw-thread: my-conversation" \
|
||||
-d '{"model": "<model-name>", "messages": [{"role":"user","content":"and then?"}]}'
|
||||
```
|
||||
|
||||
llmgw puts the stored history in front of that turn and sends it to whichever
|
||||
model is serving — **including a different one from the one the conversation
|
||||
started on**. That is the point: a thread begun on a large model can continue
|
||||
on a small one, and a caller dropped to a cheaper model by a spent budget keeps
|
||||
their conversation.
|
||||
|
||||
When the history does not fit the model now serving, the oldest turns are
|
||||
dropped until it does, and the reply carries `x-llmgw-thread-trimmed` with how
|
||||
many. System messages are never dropped, and neither is the turn the caller
|
||||
just sent.
|
||||
|
||||
`DELETE /v1/threads/{id}` forgets one. It is a client endpoint on purpose: the
|
||||
person who said the thing should be able to unsay it without asking an
|
||||
operator to read it first. A thread belonging to another project answers the
|
||||
same as one that never existed.
|
||||
|
||||
Turning it off again stops new storage, and existing conversations go at the
|
||||
next hourly prune:
|
||||
|
||||
```
|
||||
llmgw keys threads --project default --off
|
||||
```
|
||||
|
||||
## Reclaiming disk
|
||||
|
||||
`llmgw ctl nodes` has a DISK column: free space against the size it is free
|
||||
|
||||
Reference in new issue
Block a user