llmgw/docs/data.md

# What llmgw stores, and what leaves the machine

Written for the review that happens before a gateway is allowed near anyone's
documents. It describes what the software does. Mapping that onto an
obligation — GDPR, a sector rule, an internal policy — is work for whoever
holds the obligation; this page exists so that work starts from facts rather
than from a vendor's summary.

Everything below is checkable in the source, and the places to look are named.

## The short version

By default llmgw stores **no prompt or completion text anywhere**. It stores
who made a request, when, how many tokens, and what it cost.

Two features change that, and both are off until somebody turns them on for a
named project, and neither can be turned on without also saying for how long.

Nothing on the serving path leaves the cluster — **unless the catalog has a
model served outside it** (`external:` in a model definition): a request for
that model, or one that falls back to it, sends its prompt to that provider.
A guardrail policy with `no_store` refuses both for the scopes it covers.

**Guardrails** read every prompt and answer they are configured for, in the
gateway's own memory, to judge them. What they record — in the usage row and
the log — is the action, the policy, the category and how long it took: never
the text, the value that matched, or the placeholder map of a redaction. See
[guardrails.md](guardrails.md).

## What is stored, exactly

**Usage records**, one per request: the key, the project, the model asked for,
the model that answered, prompt and completion token counts, cost, HTTP status
and latency. No message content. The row is in `internal/store/store.go` as
`Usage` and there is no field for text in it. What the client chose to say
about the request is kept with it: OpenAI's `user` field and its `metadata`
or `x-llmgw-tags` — short strings, set by the client, for attributing spend.
A client that puts a person's name or address there puts it in the usage
records; the field is the client's, and so is that choice. Tags are not sent
on to the engine.

**API keys**, as SHA-256 hashes. The key itself is shown once, at creation, and
never stored — so a leaked database hands over no working credential, and
neither llmgw nor an administrator can recover a key somebody lost. A prefix is
kept so a key can be identified in a listing without revealing it.

**The store's own secrets**, like the one engine keys are derived from, plain
unless `secrets.master_key` seals them — with Vault's transit engine, so that
neither the store nor its backups hold them.

**People and tenancy**: accounts, the organisation/team/project tree, and the
limits attached to it. Passwords are hashed.

**Sessions** for the browser pages, as tokens.

**Conversations and agent notes** — only for a project that asked. See below.

**Kept requests** — only for a project that asked (`llmgw keys requests`):
what each request sent the model, after the guardrails' redaction and with
any stored conversation in it, and the answer the client got, for the hours
it said, all of them or a share. Never under a `no_store` policy.

**Batch files** — when a client uses the Batch API: the file of requests it
uploaded (prompts) and the files the batch wrote (answers), in the store,
until deleted (`DELETE /v1/files/{id}`) or, for what a batch wrote, until
`router.batch.retain_days` (30) pass. Only keys of the project that made
them can read them, and a project under a `no_store` policy cannot make
them.

**Reused answers** — only for a project that asked (`llmgw keys cache`):
each answer that may be reused, as the client got it, and a hash of the
request it answers, in the gateway process's memory — never in the store,
never on disk — for the project's TTL or until pushed out by newer ones
(`router.cache.max_bytes`), and gone with a restart. Never under a `no_store`
policy. Forgetting a person does not reach into it; the answers age out by
the TTL. The store keeps only the setting, and a usage record's mark that it
was answered from the cache.

## What is not stored

Prompts. Completions. Anything the model was shown or produced, unless the
project turned on conversations, notes or kept requests, or sent a batch.
(Reused answers are held in memory only, as above.)

The chat page in llmgw's own web interface keeps the conversation in a
JavaScript variable in the browser. Refresh it and the conversation is gone,
because it was never anywhere else.

## Conversations and notes, when they are turned on

Two features keep text, and both work the same way:

- **Conversations** (`x-llmgw-thread`) let a client send only its new turn while
  llmgw replays the history — including to a different model than the one the
  conversation started on.
- **Agent notes** (`/v1/memory`, and MCP tools) let an agent keep something for
  itself across conversations.

Both are **off for every project** until an operator runs `llmgw keys threads`
or `llmgw keys memory` for that project by name. A request carrying a thread id
against a project that has not is refused with an explanation, not quietly
served without its history.

Both **require a retention window**. There is no way to turn either on without
saying how many hours, because a store with no expiry is a transcript of
everything a team ever typed, kept forever because nobody chose a number. The
window is measured from the last turn, so an active conversation does not
expire underneath it, and a background sweep removes what has passed it.

Both are **scoped to a project**, and the scope is part of every lookup rather
than a check afterwards. Reading a conversation that belongs to another project
returns the same answer as reading one that does not exist, so a leaked id
cannot be used to discover what exists elsewhere.

`DELETE /v1/threads/{id}` removes a conversation and its messages immediately.
It is a client endpoint, not an administrative one: the person who said the
thing can unsay it without asking somebody else to read it first.

Turning retention off stops new storage, and existing conversations go at the
next hourly sweep.

## What leaves the machine

**Nothing on the serving path.** A request arrives, is routed to an engine on
your own hardware, and the answer comes back. The router talks to engines, the
engines talk to nothing, the agents talk to the control plane. All of it is
inside the cluster.

Around the serving path, a cluster set up the ordinary way does reach out.
None of it carries a prompt, an answer, a key or a user — it is software and
public model files coming in — but a review should know each one:

**Installing engines.** Each node's agent downloads the llama.cpp build for its
hardware from GitHub, checked against the checksum GitHub publishes, and vLLM
(SGLang when asked) from PyPI, with `uv` from GitHub. On every node, when it
joins and when the engine version changes.

**Releases.** `llmgw node add`, `llmgw ctl upgrade` and `llmgw upgrade` fetch
llmgw's own release binaries from the project's release server, checked against
its signed checksums.

**Fetching weights.** When a model's `source` is a URL, the agent downloads it;
the mirror fetches what it is asked to, from Hugging Face or another URL, and
`server init` has it fetch the starter model and the guardrails' judge. Point
sources at the mirror and the nodes fetch only from it — which is also the
faster arrangement, because eight nodes pulling forty gigabytes from the
internet is a day's work and from a local mirror it is minutes. A `source` that
is a path on disk is never fetched.

**The registry.** With a mirror configured, the control plane asks Hugging Face
every six hours whether the models served from it have newer files
(`mirror.update_check`), and `/admin/mirror` searches it. `llmgw ctl plan`
reads a model's architecture from huggingface.co to work out whether it fits
your hardware.

There is no telemetry, no licence check, and no call home.

### A cluster with no way out

A site with no route to the internet installs from an **offline bundle**
instead, and then nothing at all leaves the cluster (docs/install.md, "Offline
sites"). The bundle is made on a machine that has internet — llmgw's release
binaries with their signed checksums, the llama.cpp builds with the checksums
GitHub publishes, the model files with the checksums Hugging Face publishes,
each checked as it is downloaded — carried in, and imported into the mirror.
With `offline.mirror` set in `server.yaml`, nodes install engines from the
mirror and never from GitHub, releases come from the mirror, and the registry
is not asked about newer weights.

Verification is not weakened by any of this. A release is held to the project's
signing key wherever it is fetched from. An engine file is held to the checksum
recorded when the bundle was made, which the control plane keeps and hands each
node over the agent's authenticated connection: the mirror serves plain HTTP,
and nothing a node runs is taken on the mirror's word. The bundle itself is
checked against the sha256 its maker printed, carried to the site apart from
the bundle.

What does not come in a bundle: vLLM and SGLang, which are Python environments
from PyPI — an offline node serves with llama.cpp unless one is installed on it
by hand — and the operating system's own packages (the OpenMP runtime the
engine needs, PostgreSQL when chosen), which come from the machines' package
sources: at an offline site, the distribution mirror it already has. Searching
the registry from `/admin/mirror` and `llmgw ctl plan` need the internet and
do not work offline; `mirror.registry` can point the first at an internal copy.

## What the software cannot do for you

**The admin port serves pprof**, and pprof can dump the process heap — where
API keys sit in plaintext while they are being checked. Bind it to loopback and
reach it over SSH. The default does this; a deployment that changes it should
know why.

**The store holds everything.** With conversations turned on it holds text as
well as accounting, so where it lives, who can read it and how it is backed up
are decisions with the same weight as the ones above. The PostgreSQL DSN
belongs in a file readable only by the user llmgw runs as, which is why the
configuration takes `@/path/to/file` rather than the address itself.

**Model weights are yours.** llmgw reclaims only files it downloaded, and it
identifies those by a name it generated. Anything you put on a node it will not
touch, whatever the catalog says.

## Where to look

| Claim | Where |
|---|---|
| Usage rows carry no text | `internal/store/store.go`, type `Usage` |
| Keys are stored hashed | `internal/store/keys.go` |
| Conversations are off by default | `internal/store/threads.go`, `ThreadRetention` |
| Retention is required | `llmgw keys threads --help` |
| Scope is part of the lookup | `internal/store/threads.go`, `Thread` |
| Deletion is a client endpoint | `internal/router/threads.go`, `handleDeleteThread` |
| Weight fetching is the only serving-path egress | `internal/agent/weights/fetch.go` |

## Forgetting a person

`llmgw users forget <email> --yes` (or **Forget them…** in the console) is for
a request to be forgotten:

- their usage — every request record and every sum — is kept, so totals and
  budgets stay true, but under a random pseudonym that links to nothing;
- their kept requests (`keys requests`), personal files and batches, and
  their own limits are deleted;
- their email and id are taken out of the audit log, which keeps what was
  done;
- then the account goes, as `users delete` removes it: sessions ended, keys
  revoked and left without an owner.

Conversations and agent notes belong to projects, not to the people in them,
and are not touched: a project forgets them by its retention, and a thread by
`DELETE /v1/threads/{id}`. Backups taken before still hold everything until
they age out; that is what their retention is for.

## Keeping a project's requests in the cluster

`llmgw keys residency --project <path> --in-cluster` refuses the project's
keys any model served by an outside provider, and never uses one for them as
a fallback or a canary: their prompts do not leave the machines the cluster
runs on. `--anywhere` lifts it.