Ilker ManapandClaude Opus 5
b7a97468e0
Neither had a page. "We have GPUs, now what" was left to the reader, and what llmgw stores was scattered across four documents and a changelog — which is the same as not being written down when somebody is doing a security review. docs/data.md describes the behaviour and leaves the legal mapping to whoever holds the obligation. Every claim in it was checked against the source, and it names where to look for each: usage rows carry no text field, keys are stored hashed, conversations are off until a named project turns them on and cannot be turned on without a retention window. It also states the egress exactly, which took reading every outbound call to say honestly: nothing on the serving path leaves the cluster, and the two things that can reach outside — fetching weights, and ctl plan reading a model's architecture — are both started by an operator. Weight fetching points wherever you tell it, including an internal mirror. docs/choosing-models.md is the opinion llmgw itself does not have. Three jobs rather than one model, with the third — embeddings — being the one that gets forgotten and the one that decides whether the deployment can answer questions about your company or only questions in general. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
6.1 KiB
What llmgw stores, and what leaves the machine
Written for the review that happens before a gateway is allowed near anyone's documents. It describes what the software does. Mapping that onto an obligation — GDPR, a sector rule, an internal policy — is work for whoever holds the obligation; this page exists so that work starts from facts rather than from a vendor's summary.
Everything below is checkable in the source, and the places to look are named.
The short version
By default llmgw stores no prompt or completion text anywhere. It stores who made a request, when, how many tokens, and what it cost.
Two features change that, and both are off until somebody turns them on for a named project, and neither can be turned on without also saying for how long.
Nothing on the serving path leaves the cluster.
What is stored, exactly
Usage records, one per request: the key, the project, the model asked for,
the model that answered, prompt and completion token counts, cost, HTTP status
and latency. No message content. The row is in internal/store/store.go as
Usage and there is no field for text in it.
API keys, as SHA-256 hashes. The key itself is shown once, at creation, and never stored — so a leaked database hands over no working credential, and neither llmgw nor an administrator can recover a key somebody lost. A prefix is kept so a key can be identified in a listing without revealing it.
People and tenancy: accounts, the organisation/team/project tree, and the limits attached to it. Passwords are hashed.
Sessions for the browser pages, as tokens.
Conversations and agent notes — only for a project that asked. See below.
What is not stored
Prompts. Completions. Anything the model was shown or produced, unless the project turned on conversations or notes.
The chat page in llmgw's own web interface keeps the conversation in a JavaScript variable in the browser. Refresh it and the conversation is gone, because it was never anywhere else.
Conversations and notes, when they are turned on
Two features keep text, and both work the same way:
- Conversations (
x-llmgw-thread) let a client send only its new turn while llmgw replays the history — including to a different model than the one the conversation started on. - Agent notes (
/v1/memory, and MCP tools) let an agent keep something for itself across conversations.
Both are off for every project until an operator runs llmgw keys threads
or llmgw keys memory for that project by name. A request carrying a thread id
against a project that has not is refused with an explanation, not quietly
served without its history.
Both require a retention window. There is no way to turn either on without saying how many hours, because a store with no expiry is a transcript of everything a team ever typed, kept forever because nobody chose a number. The window is measured from the last turn, so an active conversation does not expire underneath it, and a background sweep removes what has passed it.
Both are scoped to a project, and the scope is part of every lookup rather than a check afterwards. Reading a conversation that belongs to another project returns the same answer as reading one that does not exist, so a leaked id cannot be used to discover what exists elsewhere.
DELETE /v1/threads/{id} removes a conversation and its messages immediately.
It is a client endpoint, not an administrative one: the person who said the
thing can unsay it without asking somebody else to read it first.
Turning retention off stops new storage, and existing conversations go at the next hourly sweep.
What leaves the machine
Nothing on the serving path. A request arrives, is routed to an engine on your own hardware, and the answer comes back. The router talks to engines, the engines talk to nothing, the agents talk to the control plane. All of it is inside the cluster.
Two things can reach outside, both started by an operator rather than by a request:
Fetching weights. When a model's source is a URL, the agent downloads it.
Point that at an internal mirror and nothing leaves at all — which is also the
faster arrangement, because eight nodes pulling forty gigabytes from the
internet is a day's work and from a local mirror it is minutes. A source that
is a path on disk is never fetched.
llmgw ctl plan. Reads a model's architecture from huggingface.co to work
out whether it fits your hardware. An operator command run at planning time,
not part of serving, and not run by anything automatically.
There is no telemetry, no licence check, and no call home. The binary is
static, built with CGO_ENABLED=0, and runs in a network with no route to the
internet as long as the weights are already on disk.
What the software cannot do for you
The admin port serves pprof, and pprof can dump the process heap — where API keys sit in plaintext while they are being checked. Bind it to loopback and reach it over SSH. The default does this; a deployment that changes it should know why.
The store holds everything. With conversations turned on it holds text as
well as accounting, so where it lives, who can read it and how it is backed up
are decisions with the same weight as the ones above. The PostgreSQL DSN
belongs in a file readable only by the user llmgw runs as, which is why the
configuration takes @/path/to/file rather than the address itself.
Model weights are yours. llmgw reclaims only files it downloaded, and it identifies those by a name it generated. Anything you put on a node it will not touch, whatever the catalog says.
Where to look
| Claim | Where |
|---|---|
| Usage rows carry no text | internal/store/store.go, type Usage |
| Keys are stored hashed | internal/store/keys.go |
| Conversations are off by default | internal/store/threads.go, ThreadRetention |
| Retention is required | llmgw keys threads --help |
| Scope is part of the lookup | internal/store/threads.go, Thread |
| Deletion is a client endpoint | internal/router/threads.go, handleDeleteThread |
| Weight fetching is the only serving-path egress | internal/agent/weights/fetch.go |