Answer the two questions asked before a gateway is allowed near anything
CI / test (push) Successful in 1m34s
CI / test (push) Successful in 1m34s
Neither had a page. "We have GPUs, now what" was left to the reader, and what llmgw stores was scattered across four documents and a changelog — which is the same as not being written down when somebody is doing a security review. docs/data.md describes the behaviour and leaves the legal mapping to whoever holds the obligation. Every claim in it was checked against the source, and it names where to look for each: usage rows carry no text field, keys are stored hashed, conversations are off until a named project turns them on and cannot be turned on without a retention window. It also states the egress exactly, which took reading every outbound call to say honestly: nothing on the serving path leaves the cluster, and the two things that can reach outside — fetching weights, and ctl plan reading a model's architecture — are both started by an operator. Weight fetching points wherever you tell it, including an internal mirror. docs/choosing-models.md is the opinion llmgw itself does not have. Three jobs rather than one model, with the third — embeddings — being the one that gets forgotten and the one that decides whether the deployment can answer questions about your company or only questions in general. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
f2f375d824
commit
b7a97468e0
4 files changed
+289
No files matched your search
+133
@@ -0,0 +1,133 @@
|
||||
# What llmgw stores, and what leaves the machine
|
||||
|
||||
Written for the review that happens before a gateway is allowed near anyone's
|
||||
documents. It describes what the software does. Mapping that onto an
|
||||
obligation — GDPR, a sector rule, an internal policy — is work for whoever
|
||||
holds the obligation; this page exists so that work starts from facts rather
|
||||
than from a vendor's summary.
|
||||
|
||||
Everything below is checkable in the source, and the places to look are named.
|
||||
|
||||
## The short version
|
||||
|
||||
By default llmgw stores **no prompt or completion text anywhere**. It stores
|
||||
who made a request, when, how many tokens, and what it cost.
|
||||
|
||||
Two features change that, and both are off until somebody turns them on for a
|
||||
named project, and neither can be turned on without also saying for how long.
|
||||
|
||||
Nothing on the serving path leaves the cluster.
|
||||
|
||||
## What is stored, exactly
|
||||
|
||||
**Usage records**, one per request: the key, the project, the model asked for,
|
||||
the model that answered, prompt and completion token counts, cost, HTTP status
|
||||
and latency. No message content. The row is in `internal/store/store.go` as
|
||||
`Usage` and there is no field for text in it.
|
||||
|
||||
**API keys**, as SHA-256 hashes. The key itself is shown once, at creation, and
|
||||
never stored — so a leaked database hands over no working credential, and
|
||||
neither llmgw nor an administrator can recover a key somebody lost. A prefix is
|
||||
kept so a key can be identified in a listing without revealing it.
|
||||
|
||||
**People and tenancy**: accounts, the organisation/team/project tree, and the
|
||||
limits attached to it. Passwords are hashed.
|
||||
|
||||
**Sessions** for the browser pages, as tokens.
|
||||
|
||||
**Conversations and agent notes** — only for a project that asked. See below.
|
||||
|
||||
## What is not stored
|
||||
|
||||
Prompts. Completions. Anything the model was shown or produced, unless the
|
||||
project turned on conversations or notes.
|
||||
|
||||
The chat page in llmgw's own web interface keeps the conversation in a
|
||||
JavaScript variable in the browser. Refresh it and the conversation is gone,
|
||||
because it was never anywhere else.
|
||||
|
||||
## Conversations and notes, when they are turned on
|
||||
|
||||
Two features keep text, and both work the same way:
|
||||
|
||||
- **Conversations** (`x-llmgw-thread`) let a client send only its new turn while
|
||||
llmgw replays the history — including to a different model than the one the
|
||||
conversation started on.
|
||||
- **Agent notes** (`/v1/memory`, and MCP tools) let an agent keep something for
|
||||
itself across conversations.
|
||||
|
||||
Both are **off for every project** until an operator runs `llmgw keys threads`
|
||||
or `llmgw keys memory` for that project by name. A request carrying a thread id
|
||||
against a project that has not is refused with an explanation, not quietly
|
||||
served without its history.
|
||||
|
||||
Both **require a retention window**. There is no way to turn either on without
|
||||
saying how many hours, because a store with no expiry is a transcript of
|
||||
everything a team ever typed, kept forever because nobody chose a number. The
|
||||
window is measured from the last turn, so an active conversation does not
|
||||
expire underneath it, and a background sweep removes what has passed it.
|
||||
|
||||
Both are **scoped to a project**, and the scope is part of every lookup rather
|
||||
than a check afterwards. Reading a conversation that belongs to another project
|
||||
returns the same answer as reading one that does not exist, so a leaked id
|
||||
cannot be used to discover what exists elsewhere.
|
||||
|
||||
`DELETE /v1/threads/{id}` removes a conversation and its messages immediately.
|
||||
It is a client endpoint, not an administrative one: the person who said the
|
||||
thing can unsay it without asking somebody else to read it first.
|
||||
|
||||
Turning retention off stops new storage, and existing conversations go at the
|
||||
next hourly sweep.
|
||||
|
||||
## What leaves the machine
|
||||
|
||||
**Nothing on the serving path.** A request arrives, is routed to an engine on
|
||||
your own hardware, and the answer comes back. The router talks to engines, the
|
||||
engines talk to nothing, the agents talk to the control plane. All of it is
|
||||
inside the cluster.
|
||||
|
||||
Two things can reach outside, both started by an operator rather than by a
|
||||
request:
|
||||
|
||||
**Fetching weights.** When a model's `source` is a URL, the agent downloads it.
|
||||
Point that at an internal mirror and nothing leaves at all — which is also the
|
||||
faster arrangement, because eight nodes pulling forty gigabytes from the
|
||||
internet is a day's work and from a local mirror it is minutes. A `source` that
|
||||
is a path on disk is never fetched.
|
||||
|
||||
**`llmgw ctl plan`.** Reads a model's architecture from huggingface.co to work
|
||||
out whether it fits your hardware. An operator command run at planning time,
|
||||
not part of serving, and not run by anything automatically.
|
||||
|
||||
There is no telemetry, no licence check, and no call home. The binary is
|
||||
static, built with `CGO_ENABLED=0`, and runs in a network with no route to the
|
||||
internet as long as the weights are already on disk.
|
||||
|
||||
## What the software cannot do for you
|
||||
|
||||
**The admin port serves pprof**, and pprof can dump the process heap — where
|
||||
API keys sit in plaintext while they are being checked. Bind it to loopback and
|
||||
reach it over SSH. The default does this; a deployment that changes it should
|
||||
know why.
|
||||
|
||||
**The store holds everything.** With conversations turned on it holds text as
|
||||
well as accounting, so where it lives, who can read it and how it is backed up
|
||||
are decisions with the same weight as the ones above. The PostgreSQL DSN
|
||||
belongs in a file readable only by the user llmgw runs as, which is why the
|
||||
configuration takes `@/path/to/file` rather than the address itself.
|
||||
|
||||
**Model weights are yours.** llmgw reclaims only files it downloaded, and it
|
||||
identifies those by a name it generated. Anything you put on a node it will not
|
||||
touch, whatever the catalog says.
|
||||
|
||||
## Where to look
|
||||
|
||||
| Claim | Where |
|
||||
|---|---|
|
||||
| Usage rows carry no text | `internal/store/store.go`, type `Usage` |
|
||||
| Keys are stored hashed | `internal/store/keys.go` |
|
||||
| Conversations are off by default | `internal/store/threads.go`, `ThreadRetention` |
|
||||
| Retention is required | `llmgw keys threads --help` |
|
||||
| Scope is part of the lookup | `internal/store/threads.go`, `Thread` |
|
||||
| Deletion is a client endpoint | `internal/router/threads.go`, `handleDeleteThread` |
|
||||
| Weight fetching is the only serving-path egress | `internal/agent/weights/fetch.go` |
|
||||
Reference in new issue
Block a user