Answer the two questions asked before a gateway is allowed near anything
CI / test (push) Successful in 1m34s

Neither had a page. "We have GPUs, now what" was left to the reader, and what
llmgw stores was scattered across four documents and a changelog — which is
the same as not being written down when somebody is doing a security review.

docs/data.md describes the behaviour and leaves the legal mapping to whoever
holds the obligation. Every claim in it was checked against the source, and it
names where to look for each: usage rows carry no text field, keys are stored
hashed, conversations are off until a named project turns them on and cannot be
turned on without a retention window.

It also states the egress exactly, which took reading every outbound call to
say honestly: nothing on the serving path leaves the cluster, and the two
things that can reach outside — fetching weights, and ctl plan reading a model's
architecture — are both started by an operator. Weight fetching points wherever
you tell it, including an internal mirror.

docs/choosing-models.md is the opinion llmgw itself does not have. Three jobs
rather than one model, with the third — embeddings — being the one that gets
forgotten and the one that decides whether the deployment can answer questions
about your company or only questions in general.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-27 08:16:08 +02:00
1 parent f2f375d824
commit b7a97468e0
4 files changed
+289

No files matched your search

+4
View File
@@ -57,6 +57,10 @@ on a machine that has them.
- [docs/install.md](docs/install.md) — installing from nothing: control plane,
nodes, engines, weights, first request
- [docs/quickstart.md](docs/quickstart.md) — the short path
- [docs/choosing-models.md](docs/choosing-models.md) — which models to run,
how many of each, and where the weights should live
- [docs/data.md](docs/data.md) — what llmgw stores, what it does not, and
what leaves the machine; written for a security review
- [docs/reference.md](docs/reference.md) — every configuration key, generated
from the structs and checked by a test, so it cannot describe a setting that
does not exist or miss one that does
+148
View File
@@ -0,0 +1,148 @@
# Choosing what to run
llmgw serves whatever you give it and has no opinion about which models those
are. This page has an opinion, because "we have GPUs, now what" is the question
that stalls most of these projects — and the usual first answer, one large
model, is the wrong shape.
Model releases move faster than documentation. Treat the families named here as
a starting point and check what is current before committing; the reasoning is
what should outlive the names.
## Three jobs, not one model
The requests a company actually makes fall into three groups with different
needs, and no single model serves all three well.
**Interactive** — chat, autocomplete, short questions. Most of your traffic.
Here latency beats quality: a person waits two seconds and does not wait
twenty. A 7–14B class model, always resident.
**Heavy** — long document analysis, code generation, anything that needs
reasoning. Quality beats latency, and a queue is acceptable. A 30–70B class
model, loaded on demand:
```yaml
scaling: { min: 0, max: 1, idle_timeout: 30m }
```
llmgw starts it when somebody asks and stops it when nobody has, so it costs
nothing while it is not in use. The first request waits for the load, and the
client is told that rather than refused.
**Embeddings** — and this is the one that gets forgotten. Most "let us chat
with our documents" projects are retrieval projects, and retrieval runs on an
embedding model. It is small, it is cheap, and it carries the highest request
volume of the three: every document chunk once, every query forever. llmgw
routes `/v1/embeddings` like any other endpoint.
A deployment with the first two and not the third can answer questions from
what the model already knew. It cannot answer questions about your company.
## Swedish and English
The first thing to know is that this combination is harder than either alone.
Open models are strongest in English by a wide margin. Swedish is served —
better than most languages its size, because the Nordic languages appear in
multilingual training sets — but the gap is real, and it widens as models get
smaller. **A 7B model that feels fine in English can feel noticeably worse in
Swedish**, which means the interactive tier may need to be larger than an
English-only deployment would choose. Test before deciding; do not assume the
English benchmark transfers.
Two things matter more than the model's headline multilingual score:
**Cross-lingual retrieval.** A Swedish company writes internal documents in
Swedish and reads technical documentation in English. Somebody will ask a
question in Swedish whose answer is in an English document. An embedding model
that handles both languages separately may still fail to connect them, and it
fails silently — the model gives a confident answer from the wrong source, and
nobody notices because the answer reads well. Test this explicitly: ask in one
language, check that documents in the other are retrieved.
**Swedish compounds.** *Kundtjänstmedarbetare* is one token-hostile word where
English would use three. This hurts keyword search more than dense retrieval,
so if you run hybrid search, expect the lexical half to underperform on Swedish
and weight accordingly.
## How to actually decide
Benchmarks will not settle this. Take twenty real documents and twenty real
questions from the company — in both languages, including at least a few where
the question and the answer are in different languages — and compare candidates
on those. An afternoon of this is worth more than a week of reading leaderboards,
and it is the only way to find the failure modes that matter to you.
Do the same for the embedding model, separately. A good generation model on top
of bad retrieval produces fluent, confident, wrong answers, which is the worst
failure this stack has: it does not look like a failure.
## How many replicas
Not "how many employees". **Concurrency**, which is far lower than headcount
because people spend most of their time reading rather than waiting on a model.
Two hundred people generate perhaps five to fifteen concurrent requests at
peak, for interactive use.
vLLM batches continuously, so one replica on a capable card serves many
concurrent requests. Which means the replica count is usually decided by
**redundancy rather than throughput**:
| Tier | Replicas | Why |
|---|---|---|
| Embeddings | 2 | Not for capacity. If retrieval stops, every document feature stops. |
| Interactive | 2 | So a node can be rebooted during working hours. |
| Heavy | 0–1, on demand | Loaded when asked for, stopped when idle. |
Then measure rather than guess:
```
llmgw ctl bench <model>
```
It climbs concurrency until the SLO breaks and reports the last level that met
it. Peak throughput arrives long after latency has become unusable, so
throughput alone overstates what the hardware can serve.
## Where the weights live
The single highest-value piece of infrastructure here is **a local mirror**: one
internal HTTP server holding the model files. Hugging Face is slow, rate
limited, and blocked outright on plenty of corporate networks. Eight nodes each
pulling forty gigabytes from the internet is a day; from a mirror on the same
switch it is minutes. Point a model's `source` at it and llmgw fetches from
there.
**Pin the version.** Put it in the path — `/models/qwen2.5-14b/2026-03/q4_k_m.gguf`
— so a repository that changes upstream cannot change what you are running.
Without this, a prompt that worked last week gives a different answer this week
and nobody can explain why.
**Quantisation.** For llama.cpp, `q4_k_m` is the practical default: the quality
difference from `q8_0` is smaller than most people expect and the memory
difference is a factor of two. For vLLM on modern cards, FP8 or AWQ. Run
`llmgw ctl plan <model>` to see whether a given precision fits before
downloading anything.
**Disk.** Budget three to five times what you first estimate. Several
quantisations, several versions, and partial downloads all accumulate.
`llmgw ctl nodes` shows free space per node once `model_dirs` is configured,
and `llmgw ctl weights` reclaims files llmgw downloaded and no model needs —
never files you placed yourself.
Under ten nodes, let each keep its own copy; it is simpler and the disk is
cheap. Above that, consider a shared filesystem — llmgw notices one and will
not count the same free terabyte once per node.
## Before anyone starts using it
Set budgets and limits on day one, not later:
```
llmgw keys limits --scope project --id <project> --rpm 60 --budget 50000000
```
The reason is not technical. A resource that appears free and unlimited becomes,
three months later, a cost nobody can account for — and introducing limits at
that point is a much harder conversation than having had them from the start.
+133
View File
@@ -0,0 +1,133 @@
# What llmgw stores, and what leaves the machine
Written for the review that happens before a gateway is allowed near anyone's
documents. It describes what the software does. Mapping that onto an
obligation — GDPR, a sector rule, an internal policy — is work for whoever
holds the obligation; this page exists so that work starts from facts rather
than from a vendor's summary.
Everything below is checkable in the source, and the places to look are named.
## The short version
By default llmgw stores **no prompt or completion text anywhere**. It stores
who made a request, when, how many tokens, and what it cost.
Two features change that, and both are off until somebody turns them on for a
named project, and neither can be turned on without also saying for how long.
Nothing on the serving path leaves the cluster.
## What is stored, exactly
**Usage records**, one per request: the key, the project, the model asked for,
the model that answered, prompt and completion token counts, cost, HTTP status
and latency. No message content. The row is in `internal/store/store.go` as
`Usage` and there is no field for text in it.
**API keys**, as SHA-256 hashes. The key itself is shown once, at creation, and
never stored — so a leaked database hands over no working credential, and
neither llmgw nor an administrator can recover a key somebody lost. A prefix is
kept so a key can be identified in a listing without revealing it.
**People and tenancy**: accounts, the organisation/team/project tree, and the
limits attached to it. Passwords are hashed.
**Sessions** for the browser pages, as tokens.
**Conversations and agent notes** — only for a project that asked. See below.
## What is not stored
Prompts. Completions. Anything the model was shown or produced, unless the
project turned on conversations or notes.
The chat page in llmgw's own web interface keeps the conversation in a
JavaScript variable in the browser. Refresh it and the conversation is gone,
because it was never anywhere else.
## Conversations and notes, when they are turned on
Two features keep text, and both work the same way:
- **Conversations** (`x-llmgw-thread`) let a client send only its new turn while
llmgw replays the history — including to a different model than the one the
conversation started on.
- **Agent notes** (`/v1/memory`, and MCP tools) let an agent keep something for
itself across conversations.
Both are **off for every project** until an operator runs `llmgw keys threads`
or `llmgw keys memory` for that project by name. A request carrying a thread id
against a project that has not is refused with an explanation, not quietly
served without its history.
Both **require a retention window**. There is no way to turn either on without
saying how many hours, because a store with no expiry is a transcript of
everything a team ever typed, kept forever because nobody chose a number. The
window is measured from the last turn, so an active conversation does not
expire underneath it, and a background sweep removes what has passed it.
Both are **scoped to a project**, and the scope is part of every lookup rather
than a check afterwards. Reading a conversation that belongs to another project
returns the same answer as reading one that does not exist, so a leaked id
cannot be used to discover what exists elsewhere.
`DELETE /v1/threads/{id}` removes a conversation and its messages immediately.
It is a client endpoint, not an administrative one: the person who said the
thing can unsay it without asking somebody else to read it first.
Turning retention off stops new storage, and existing conversations go at the
next hourly sweep.
## What leaves the machine
**Nothing on the serving path.** A request arrives, is routed to an engine on
your own hardware, and the answer comes back. The router talks to engines, the
engines talk to nothing, the agents talk to the control plane. All of it is
inside the cluster.
Two things can reach outside, both started by an operator rather than by a
request:
**Fetching weights.** When a model's `source` is a URL, the agent downloads it.
Point that at an internal mirror and nothing leaves at all — which is also the
faster arrangement, because eight nodes pulling forty gigabytes from the
internet is a day's work and from a local mirror it is minutes. A `source` that
is a path on disk is never fetched.
**`llmgw ctl plan`.** Reads a model's architecture from huggingface.co to work
out whether it fits your hardware. An operator command run at planning time,
not part of serving, and not run by anything automatically.
There is no telemetry, no licence check, and no call home. The binary is
static, built with `CGO_ENABLED=0`, and runs in a network with no route to the
internet as long as the weights are already on disk.
## What the software cannot do for you
**The admin port serves pprof**, and pprof can dump the process heap — where
API keys sit in plaintext while they are being checked. Bind it to loopback and
reach it over SSH. The default does this; a deployment that changes it should
know why.
**The store holds everything.** With conversations turned on it holds text as
well as accounting, so where it lives, who can read it and how it is backed up
are decisions with the same weight as the ones above. The PostgreSQL DSN
belongs in a file readable only by the user llmgw runs as, which is why the
configuration takes `@/path/to/file` rather than the address itself.
**Model weights are yours.** llmgw reclaims only files it downloaded, and it
identifies those by a name it generated. Anything you put on a node it will not
touch, whatever the catalog says.
## Where to look
| Claim | Where |
|---|---|
| Usage rows carry no text | `internal/store/store.go`, type `Usage` |
| Keys are stored hashed | `internal/store/keys.go` |
| Conversations are off by default | `internal/store/threads.go`, `ThreadRetention` |
| Retention is required | `llmgw keys threads --help` |
| Scope is part of the lookup | `internal/store/threads.go`, `Thread` |
| Deletion is a client endpoint | `internal/router/threads.go`, `handleDeleteThread` |
| Weight fetching is the only serving-path egress | `internal/agent/weights/fetch.go` |
+4
View File
@@ -786,6 +786,10 @@ requests. On a slow model, set `timeouts.generation` higher.
## What to read next
- `docs/choosing-models.md` — which models to run and how many, if the answer
to "we have GPUs, now what" is not already obvious
- `docs/data.md` — what is stored and what leaves the machine, for the review
that happens before this is allowed near anyone's documents
- `docs/reference.md` — every configuration key, generated from the source, so
it describes the version you are running rather than the one somebody last
wrote about