diff --git a/CHANGELOG.md b/CHANGELOG.md index c564a46..e0f35f4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,37 @@ APIs change between minors without a deprecation period. ## Unreleased +### Conversations that can move between models + +A thread a client names with `x-llmgw-thread`, carrying only its new turn. +llmgw replays the stored history to whichever model is serving, which is what +lets a conversation begun on a large model continue on a small one — including +when a spent budget moved it there and the caller never chose. + +Off for every project until someone runs `llmgw keys threads`. Until then llmgw +stores no message content anywhere, as it never has, and that default is worth +more than the feature. The retention is required rather than optional, because +a conversation store with no expiry is a transcript of everything a team ever +typed kept forever by omission, and it is measured from the last turn so a +conversation in progress does not expire underneath it. + +History that does not fit the serving model is trimmed oldest-first and the +reply says how many turns went, in `x-llmgw-thread-trimmed`. System messages +are never dropped — losing one silently changes what the model is for while it +carries on answering — and neither is the turn the caller just sent. + +Threads are scoped to a project, and reading, writing or deleting one that +belongs to another answers exactly as one that never existed: two answers would +let a caller map which ids are real in projects they cannot read. + +Capturing the reply is opt-in and bounded, so a request without a thread +accumulates nothing on the hottest path in the system, and one with a thread +stops at a cap rather than growing to it. + +`DELETE /v1/threads/{id}` forgets a conversation, as a client rather than as an +operator: the person who said the thing should not have to ask someone to read +it before it can go. + ### Disk, and taking back only what llmgw put there Disk was the one resource nothing reported. `statfs` was already being called diff --git a/docs/install.md b/docs/install.md index 3b1b03d..200abd3 100644 --- a/docs/install.md +++ b/docs/install.md @@ -595,6 +595,55 @@ fleet means the measurement is missing, not that the fleet is idle. And `llmgw_gpu_nvlink_peers` at zero everywhere means the topology is unknown, which is what a binary built without the nvml tag reports. +## Conversations across models + +Off everywhere until a project asks. llmgw otherwise stores no message content +at all — usage rows carry token counts, costs and latencies, never text, and +even its own chat page keeps the conversation in the browser. A gateway that +keeps nothing anyone typed is a much easier thing to have in front of a team, +so keeping it is a decision someone makes rather than a default they discover. + +``` +llmgw keys threads --project default --retain-hours 24 +``` + +The retention is required. A conversation store with no expiry is a transcript +of everything a team ever typed, kept forever because nobody chose a number. +The window is measured from the last turn, so a conversation still in progress +does not expire underneath it. + +A client then sends a thread id and only its new turn: + +``` +curl $BASE/v1/chat/completions \ + -H "Authorization: Bearer $LLMGW_API_KEY" \ + -H "x-llmgw-thread: my-conversation" \ + -d '{"model": "", "messages": [{"role":"user","content":"and then?"}]}' +``` + +llmgw puts the stored history in front of that turn and sends it to whichever +model is serving — **including a different one from the one the conversation +started on**. That is the point: a thread begun on a large model can continue +on a small one, and a caller dropped to a cheaper model by a spent budget keeps +their conversation. + +When the history does not fit the model now serving, the oldest turns are +dropped until it does, and the reply carries `x-llmgw-thread-trimmed` with how +many. System messages are never dropped, and neither is the turn the caller +just sent. + +`DELETE /v1/threads/{id}` forgets one. It is a client endpoint on purpose: the +person who said the thing should be able to unsay it without asking an +operator to read it first. A thread belonging to another project answers the +same as one that never existed. + +Turning it off again stops new storage, and existing conversations go at the +next hourly prune: + +``` +llmgw keys threads --project default --off +``` + ## Reclaiming disk `llmgw ctl nodes` has a DISK column: free space against the size it is free