Ilker ManapandClaude Opus 5
b7a97468e0
Neither had a page. "We have GPUs, now what" was left to the reader, and what llmgw stores was scattered across four documents and a changelog — which is the same as not being written down when somebody is doing a security review. docs/data.md describes the behaviour and leaves the legal mapping to whoever holds the obligation. Every claim in it was checked against the source, and it names where to look for each: usage rows carry no text field, keys are stored hashed, conversations are off until a named project turns them on and cannot be turned on without a retention window. It also states the egress exactly, which took reading every outbound call to say honestly: nothing on the serving path leaves the cluster, and the two things that can reach outside — fetching weights, and ctl plan reading a model's architecture — are both started by an operator. Weight fetching points wherever you tell it, including an internal mirror. docs/choosing-models.md is the opinion llmgw itself does not have. Three jobs rather than one model, with the third — embeddings — being the one that gets forgotten and the one that decides whether the deployment can answer questions about your company or only questions in general. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
134 lines
6.1 KiB
Markdown
134 lines
6.1 KiB
Markdown
# What llmgw stores, and what leaves the machine
|
|
|
|
Written for the review that happens before a gateway is allowed near anyone's
|
|
documents. It describes what the software does. Mapping that onto an
|
|
obligation — GDPR, a sector rule, an internal policy — is work for whoever
|
|
holds the obligation; this page exists so that work starts from facts rather
|
|
than from a vendor's summary.
|
|
|
|
Everything below is checkable in the source, and the places to look are named.
|
|
|
|
## The short version
|
|
|
|
By default llmgw stores **no prompt or completion text anywhere**. It stores
|
|
who made a request, when, how many tokens, and what it cost.
|
|
|
|
Two features change that, and both are off until somebody turns them on for a
|
|
named project, and neither can be turned on without also saying for how long.
|
|
|
|
Nothing on the serving path leaves the cluster.
|
|
|
|
## What is stored, exactly
|
|
|
|
**Usage records**, one per request: the key, the project, the model asked for,
|
|
the model that answered, prompt and completion token counts, cost, HTTP status
|
|
and latency. No message content. The row is in `internal/store/store.go` as
|
|
`Usage` and there is no field for text in it.
|
|
|
|
**API keys**, as SHA-256 hashes. The key itself is shown once, at creation, and
|
|
never stored — so a leaked database hands over no working credential, and
|
|
neither llmgw nor an administrator can recover a key somebody lost. A prefix is
|
|
kept so a key can be identified in a listing without revealing it.
|
|
|
|
**People and tenancy**: accounts, the organisation/team/project tree, and the
|
|
limits attached to it. Passwords are hashed.
|
|
|
|
**Sessions** for the browser pages, as tokens.
|
|
|
|
**Conversations and agent notes** — only for a project that asked. See below.
|
|
|
|
## What is not stored
|
|
|
|
Prompts. Completions. Anything the model was shown or produced, unless the
|
|
project turned on conversations or notes.
|
|
|
|
The chat page in llmgw's own web interface keeps the conversation in a
|
|
JavaScript variable in the browser. Refresh it and the conversation is gone,
|
|
because it was never anywhere else.
|
|
|
|
## Conversations and notes, when they are turned on
|
|
|
|
Two features keep text, and both work the same way:
|
|
|
|
- **Conversations** (`x-llmgw-thread`) let a client send only its new turn while
|
|
llmgw replays the history — including to a different model than the one the
|
|
conversation started on.
|
|
- **Agent notes** (`/v1/memory`, and MCP tools) let an agent keep something for
|
|
itself across conversations.
|
|
|
|
Both are **off for every project** until an operator runs `llmgw keys threads`
|
|
or `llmgw keys memory` for that project by name. A request carrying a thread id
|
|
against a project that has not is refused with an explanation, not quietly
|
|
served without its history.
|
|
|
|
Both **require a retention window**. There is no way to turn either on without
|
|
saying how many hours, because a store with no expiry is a transcript of
|
|
everything a team ever typed, kept forever because nobody chose a number. The
|
|
window is measured from the last turn, so an active conversation does not
|
|
expire underneath it, and a background sweep removes what has passed it.
|
|
|
|
Both are **scoped to a project**, and the scope is part of every lookup rather
|
|
than a check afterwards. Reading a conversation that belongs to another project
|
|
returns the same answer as reading one that does not exist, so a leaked id
|
|
cannot be used to discover what exists elsewhere.
|
|
|
|
`DELETE /v1/threads/{id}` removes a conversation and its messages immediately.
|
|
It is a client endpoint, not an administrative one: the person who said the
|
|
thing can unsay it without asking somebody else to read it first.
|
|
|
|
Turning retention off stops new storage, and existing conversations go at the
|
|
next hourly sweep.
|
|
|
|
## What leaves the machine
|
|
|
|
**Nothing on the serving path.** A request arrives, is routed to an engine on
|
|
your own hardware, and the answer comes back. The router talks to engines, the
|
|
engines talk to nothing, the agents talk to the control plane. All of it is
|
|
inside the cluster.
|
|
|
|
Two things can reach outside, both started by an operator rather than by a
|
|
request:
|
|
|
|
**Fetching weights.** When a model's `source` is a URL, the agent downloads it.
|
|
Point that at an internal mirror and nothing leaves at all — which is also the
|
|
faster arrangement, because eight nodes pulling forty gigabytes from the
|
|
internet is a day's work and from a local mirror it is minutes. A `source` that
|
|
is a path on disk is never fetched.
|
|
|
|
**`llmgw ctl plan`.** Reads a model's architecture from huggingface.co to work
|
|
out whether it fits your hardware. An operator command run at planning time,
|
|
not part of serving, and not run by anything automatically.
|
|
|
|
There is no telemetry, no licence check, and no call home. The binary is
|
|
static, built with `CGO_ENABLED=0`, and runs in a network with no route to the
|
|
internet as long as the weights are already on disk.
|
|
|
|
## What the software cannot do for you
|
|
|
|
**The admin port serves pprof**, and pprof can dump the process heap — where
|
|
API keys sit in plaintext while they are being checked. Bind it to loopback and
|
|
reach it over SSH. The default does this; a deployment that changes it should
|
|
know why.
|
|
|
|
**The store holds everything.** With conversations turned on it holds text as
|
|
well as accounting, so where it lives, who can read it and how it is backed up
|
|
are decisions with the same weight as the ones above. The PostgreSQL DSN
|
|
belongs in a file readable only by the user llmgw runs as, which is why the
|
|
configuration takes `@/path/to/file` rather than the address itself.
|
|
|
|
**Model weights are yours.** llmgw reclaims only files it downloaded, and it
|
|
identifies those by a name it generated. Anything you put on a node it will not
|
|
touch, whatever the catalog says.
|
|
|
|
## Where to look
|
|
|
|
| Claim | Where |
|
|
|---|---|
|
|
| Usage rows carry no text | `internal/store/store.go`, type `Usage` |
|
|
| Keys are stored hashed | `internal/store/keys.go` |
|
|
| Conversations are off by default | `internal/store/threads.go`, `ThreadRetention` |
|
|
| Retention is required | `llmgw keys threads --help` |
|
|
| Scope is part of the lookup | `internal/store/threads.go`, `Thread` |
|
|
| Deletion is a client endpoint | `internal/router/threads.go`, `handleDeleteThread` |
|
|
| Weight fetching is the only serving-path egress | `internal/agent/weights/fetch.go` |
|