Ilker ManapandClaude Opus 5.5
44f3a9e31d
llmgw keys cache --project P --ttl 1h [--any] | --off, and Cache answers… on the console's project menu. The same request — project, endpoint, canonical model and the body without stream, stream_options, user, metadata, store and service_tier, keys sorted — is answered from the gateway's memory for the TTL. Only reproducible answers (temperature 0 or a seed; embeddings, rerank, score) unless --any; only a whole 200 from the model asked for, not a fallback's, a canary's, or one a guardrail changed. Looked up after the limits and the input guardrails; the output policy judges a reused answer; no_store and stored conversations bypass it. Streamed answers are kept put back together and streamed back as chunks. x-llmgw-cache: hit|miss|bypass, Age on a hit, Cache-Control: no-cache or x-llmgw-cache: no to bypass. A hit is recorded with no tokens and no cost, marked cache_hit (migration 0027) and shown by ctl request. An LRU bounded by router.cache.max_bytes and max_entry_bytes; llmgw_cache_requests_total, llmgw_cache_bytes, llmgw_cache_entries. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
200 lines
9.7 KiB
Markdown
200 lines
9.7 KiB
Markdown
# What llmgw stores, and what leaves the machine
|
|
|
|
Written for the review that happens before a gateway is allowed near anyone's
|
|
documents. It describes what the software does. Mapping that onto an
|
|
obligation — GDPR, a sector rule, an internal policy — is work for whoever
|
|
holds the obligation; this page exists so that work starts from facts rather
|
|
than from a vendor's summary.
|
|
|
|
Everything below is checkable in the source, and the places to look are named.
|
|
|
|
## The short version
|
|
|
|
By default llmgw stores **no prompt or completion text anywhere**. It stores
|
|
who made a request, when, how many tokens, and what it cost.
|
|
|
|
Two features change that, and both are off until somebody turns them on for a
|
|
named project, and neither can be turned on without also saying for how long.
|
|
|
|
Nothing on the serving path leaves the cluster — **unless the catalog has a
|
|
model served outside it** (`external:` in a model definition): a request for
|
|
that model, or one that falls back to it, sends its prompt to that provider.
|
|
A guardrail policy with `no_store` refuses both for the scopes it covers.
|
|
|
|
**Guardrails** read every prompt and answer they are configured for, in the
|
|
gateway's own memory, to judge them. What they record — in the usage row and
|
|
the log — is the action, the policy, the category and how long it took: never
|
|
the text, the value that matched, or the placeholder map of a redaction. See
|
|
[guardrails.md](guardrails.md).
|
|
|
|
## What is stored, exactly
|
|
|
|
**Usage records**, one per request: the key, the project, the model asked for,
|
|
the model that answered, prompt and completion token counts, cost, HTTP status
|
|
and latency. No message content. The row is in `internal/store/store.go` as
|
|
`Usage` and there is no field for text in it. What the client chose to say
|
|
about the request is kept with it: OpenAI's `user` field and its `metadata`
|
|
or `x-llmgw-tags` — short strings, set by the client, for attributing spend.
|
|
A client that puts a person's name or address there puts it in the usage
|
|
records; the field is the client's, and so is that choice. Tags are not sent
|
|
on to the engine.
|
|
|
|
**API keys**, as SHA-256 hashes. The key itself is shown once, at creation, and
|
|
never stored — so a leaked database hands over no working credential, and
|
|
neither llmgw nor an administrator can recover a key somebody lost. A prefix is
|
|
kept so a key can be identified in a listing without revealing it.
|
|
|
|
**The store's own secrets**, like the one engine keys are derived from, plain
|
|
unless `secrets.master_key` seals them — with Vault's transit engine, so that
|
|
neither the store nor its backups hold them.
|
|
|
|
**People and tenancy**: accounts, the organisation/team/project tree, and the
|
|
limits attached to it. Passwords are hashed.
|
|
|
|
**Sessions** for the browser pages, as tokens.
|
|
|
|
**Conversations and agent notes** — only for a project that asked. See below.
|
|
|
|
**Kept requests** — only for a project that asked (`llmgw keys requests`):
|
|
what each request sent the model, after the guardrails' redaction and with
|
|
any stored conversation in it, and the answer the client got, for the hours
|
|
it said, all of them or a share. Never under a `no_store` policy.
|
|
|
|
**Batch files** — when a client uses the Batch API: the file of requests it
|
|
uploaded (prompts) and the files the batch wrote (answers), in the store,
|
|
until deleted (`DELETE /v1/files/{id}`) or, for what a batch wrote, until
|
|
`router.batch.retain_days` (30) pass. Only keys of the project that made
|
|
them can read them, and a project under a `no_store` policy cannot make
|
|
them.
|
|
|
|
**Reused answers** — only for a project that asked (`llmgw keys cache`):
|
|
each answer that may be reused, as the client got it, and a hash of the
|
|
request it answers, in the gateway process's memory — never in the store,
|
|
never on disk — for the project's TTL or until pushed out by newer ones
|
|
(`router.cache.max_bytes`), and gone with a restart. Never under a `no_store`
|
|
policy. Forgetting a person does not reach into it; the answers age out by
|
|
the TTL. The store keeps only the setting, and a usage record's mark that it
|
|
was answered from the cache.
|
|
|
|
## What is not stored
|
|
|
|
Prompts. Completions. Anything the model was shown or produced, unless the
|
|
project turned on conversations, notes or kept requests, or sent a batch.
|
|
(Reused answers are held in memory only, as above.)
|
|
|
|
The chat page in llmgw's own web interface keeps the conversation in a
|
|
JavaScript variable in the browser. Refresh it and the conversation is gone,
|
|
because it was never anywhere else.
|
|
|
|
## Conversations and notes, when they are turned on
|
|
|
|
Two features keep text, and both work the same way:
|
|
|
|
- **Conversations** (`x-llmgw-thread`) let a client send only its new turn while
|
|
llmgw replays the history — including to a different model than the one the
|
|
conversation started on.
|
|
- **Agent notes** (`/v1/memory`, and MCP tools) let an agent keep something for
|
|
itself across conversations.
|
|
|
|
Both are **off for every project** until an operator runs `llmgw keys threads`
|
|
or `llmgw keys memory` for that project by name. A request carrying a thread id
|
|
against a project that has not is refused with an explanation, not quietly
|
|
served without its history.
|
|
|
|
Both **require a retention window**. There is no way to turn either on without
|
|
saying how many hours, because a store with no expiry is a transcript of
|
|
everything a team ever typed, kept forever because nobody chose a number. The
|
|
window is measured from the last turn, so an active conversation does not
|
|
expire underneath it, and a background sweep removes what has passed it.
|
|
|
|
Both are **scoped to a project**, and the scope is part of every lookup rather
|
|
than a check afterwards. Reading a conversation that belongs to another project
|
|
returns the same answer as reading one that does not exist, so a leaked id
|
|
cannot be used to discover what exists elsewhere.
|
|
|
|
`DELETE /v1/threads/{id}` removes a conversation and its messages immediately.
|
|
It is a client endpoint, not an administrative one: the person who said the
|
|
thing can unsay it without asking somebody else to read it first.
|
|
|
|
Turning retention off stops new storage, and existing conversations go at the
|
|
next hourly sweep.
|
|
|
|
## What leaves the machine
|
|
|
|
**Nothing on the serving path.** A request arrives, is routed to an engine on
|
|
your own hardware, and the answer comes back. The router talks to engines, the
|
|
engines talk to nothing, the agents talk to the control plane. All of it is
|
|
inside the cluster.
|
|
|
|
Two things can reach outside, both started by an operator rather than by a
|
|
request:
|
|
|
|
**Fetching weights.** When a model's `source` is a URL, the agent downloads it.
|
|
Point that at an internal mirror and nothing leaves at all — which is also the
|
|
faster arrangement, because eight nodes pulling forty gigabytes from the
|
|
internet is a day's work and from a local mirror it is minutes. A `source` that
|
|
is a path on disk is never fetched.
|
|
|
|
**`llmgw ctl plan`.** Reads a model's architecture from huggingface.co to work
|
|
out whether it fits your hardware. An operator command run at planning time,
|
|
not part of serving, and not run by anything automatically.
|
|
|
|
There is no telemetry, no licence check, and no call home. The binary is
|
|
static, built with `CGO_ENABLED=0`, and runs in a network with no route to the
|
|
internet as long as the weights are already on disk.
|
|
|
|
## What the software cannot do for you
|
|
|
|
**The admin port serves pprof**, and pprof can dump the process heap — where
|
|
API keys sit in plaintext while they are being checked. Bind it to loopback and
|
|
reach it over SSH. The default does this; a deployment that changes it should
|
|
know why.
|
|
|
|
**The store holds everything.** With conversations turned on it holds text as
|
|
well as accounting, so where it lives, who can read it and how it is backed up
|
|
are decisions with the same weight as the ones above. The PostgreSQL DSN
|
|
belongs in a file readable only by the user llmgw runs as, which is why the
|
|
configuration takes `@/path/to/file` rather than the address itself.
|
|
|
|
**Model weights are yours.** llmgw reclaims only files it downloaded, and it
|
|
identifies those by a name it generated. Anything you put on a node it will not
|
|
touch, whatever the catalog says.
|
|
|
|
## Where to look
|
|
|
|
| Claim | Where |
|
|
|---|---|
|
|
| Usage rows carry no text | `internal/store/store.go`, type `Usage` |
|
|
| Keys are stored hashed | `internal/store/keys.go` |
|
|
| Conversations are off by default | `internal/store/threads.go`, `ThreadRetention` |
|
|
| Retention is required | `llmgw keys threads --help` |
|
|
| Scope is part of the lookup | `internal/store/threads.go`, `Thread` |
|
|
| Deletion is a client endpoint | `internal/router/threads.go`, `handleDeleteThread` |
|
|
| Weight fetching is the only serving-path egress | `internal/agent/weights/fetch.go` |
|
|
|
|
## Forgetting a person
|
|
|
|
`llmgw users forget <email> --yes` (or **Forget them…** in the console) is for
|
|
a request to be forgotten:
|
|
|
|
- their usage — every request record and every sum — is kept, so totals and
|
|
budgets stay true, but under a random pseudonym that links to nothing;
|
|
- their kept requests (`keys requests`), personal files and batches, and
|
|
their own limits are deleted;
|
|
- their email and id are taken out of the audit log, which keeps what was
|
|
done;
|
|
- then the account goes, as `users delete` removes it: sessions ended, keys
|
|
revoked and left without an owner.
|
|
|
|
Conversations and agent notes belong to projects, not to the people in them,
|
|
and are not touched: a project forgets them by its retention, and a thread by
|
|
`DELETE /v1/threads/{id}`. Backups taken before still hold everything until
|
|
they age out; that is what their retention is for.
|
|
|
|
## Keeping a project's requests in the cluster
|
|
|
|
`llmgw keys residency --project <path> --in-cluster` refuses the project's
|
|
keys any model served by an outside provider, and never uses one for them as
|
|
a fallback or a canary: their prompts do not leave the machines the cluster
|
|
runs on. `--anywhere` lifts it.
|