Answer the two questions asked before a gateway is allowed near anything
CI / test (push) Successful in 1m34s
CI / test (push) Successful in 1m34s
Neither had a page. "We have GPUs, now what" was left to the reader, and what llmgw stores was scattered across four documents and a changelog — which is the same as not being written down when somebody is doing a security review. docs/data.md describes the behaviour and leaves the legal mapping to whoever holds the obligation. Every claim in it was checked against the source, and it names where to look for each: usage rows carry no text field, keys are stored hashed, conversations are off until a named project turns them on and cannot be turned on without a retention window. It also states the egress exactly, which took reading every outbound call to say honestly: nothing on the serving path leaves the cluster, and the two things that can reach outside — fetching weights, and ctl plan reading a model's architecture — are both started by an operator. Weight fetching points wherever you tell it, including an internal mirror. docs/choosing-models.md is the opinion llmgw itself does not have. Three jobs rather than one model, with the third — embeddings — being the one that gets forgotten and the one that decides whether the deployment can answer questions about your company or only questions in general. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
f2f375d824
commit
b7a97468e0
4 files changed
+289
No files matched your search
@@ -57,6 +57,10 @@ on a machine that has them.
|
||||
- [docs/install.md](docs/install.md) — installing from nothing: control plane,
|
||||
nodes, engines, weights, first request
|
||||
- [docs/quickstart.md](docs/quickstart.md) — the short path
|
||||
- [docs/choosing-models.md](docs/choosing-models.md) — which models to run,
|
||||
how many of each, and where the weights should live
|
||||
- [docs/data.md](docs/data.md) — what llmgw stores, what it does not, and
|
||||
what leaves the machine; written for a security review
|
||||
- [docs/reference.md](docs/reference.md) — every configuration key, generated
|
||||
from the structs and checked by a test, so it cannot describe a setting that
|
||||
does not exist or miss one that does
|
||||
|
||||
@@ -0,0 +1,148 @@
|
||||
# Choosing what to run
|
||||
|
||||
llmgw serves whatever you give it and has no opinion about which models those
|
||||
are. This page has an opinion, because "we have GPUs, now what" is the question
|
||||
that stalls most of these projects — and the usual first answer, one large
|
||||
model, is the wrong shape.
|
||||
|
||||
Model releases move faster than documentation. Treat the families named here as
|
||||
a starting point and check what is current before committing; the reasoning is
|
||||
what should outlive the names.
|
||||
|
||||
## Three jobs, not one model
|
||||
|
||||
The requests a company actually makes fall into three groups with different
|
||||
needs, and no single model serves all three well.
|
||||
|
||||
**Interactive** — chat, autocomplete, short questions. Most of your traffic.
|
||||
Here latency beats quality: a person waits two seconds and does not wait
|
||||
twenty. A 7–14B class model, always resident.
|
||||
|
||||
**Heavy** — long document analysis, code generation, anything that needs
|
||||
reasoning. Quality beats latency, and a queue is acceptable. A 30–70B class
|
||||
model, loaded on demand:
|
||||
|
||||
```yaml
|
||||
scaling: { min: 0, max: 1, idle_timeout: 30m }
|
||||
```
|
||||
|
||||
llmgw starts it when somebody asks and stops it when nobody has, so it costs
|
||||
nothing while it is not in use. The first request waits for the load, and the
|
||||
client is told that rather than refused.
|
||||
|
||||
**Embeddings** — and this is the one that gets forgotten. Most "let us chat
|
||||
with our documents" projects are retrieval projects, and retrieval runs on an
|
||||
embedding model. It is small, it is cheap, and it carries the highest request
|
||||
volume of the three: every document chunk once, every query forever. llmgw
|
||||
routes `/v1/embeddings` like any other endpoint.
|
||||
|
||||
A deployment with the first two and not the third can answer questions from
|
||||
what the model already knew. It cannot answer questions about your company.
|
||||
|
||||
## Swedish and English
|
||||
|
||||
The first thing to know is that this combination is harder than either alone.
|
||||
|
||||
Open models are strongest in English by a wide margin. Swedish is served —
|
||||
better than most languages its size, because the Nordic languages appear in
|
||||
multilingual training sets — but the gap is real, and it widens as models get
|
||||
smaller. **A 7B model that feels fine in English can feel noticeably worse in
|
||||
Swedish**, which means the interactive tier may need to be larger than an
|
||||
English-only deployment would choose. Test before deciding; do not assume the
|
||||
English benchmark transfers.
|
||||
|
||||
Two things matter more than the model's headline multilingual score:
|
||||
|
||||
**Cross-lingual retrieval.** A Swedish company writes internal documents in
|
||||
Swedish and reads technical documentation in English. Somebody will ask a
|
||||
question in Swedish whose answer is in an English document. An embedding model
|
||||
that handles both languages separately may still fail to connect them, and it
|
||||
fails silently — the model gives a confident answer from the wrong source, and
|
||||
nobody notices because the answer reads well. Test this explicitly: ask in one
|
||||
language, check that documents in the other are retrieved.
|
||||
|
||||
**Swedish compounds.** *Kundtjänstmedarbetare* is one token-hostile word where
|
||||
English would use three. This hurts keyword search more than dense retrieval,
|
||||
so if you run hybrid search, expect the lexical half to underperform on Swedish
|
||||
and weight accordingly.
|
||||
|
||||
## How to actually decide
|
||||
|
||||
Benchmarks will not settle this. Take twenty real documents and twenty real
|
||||
questions from the company — in both languages, including at least a few where
|
||||
the question and the answer are in different languages — and compare candidates
|
||||
on those. An afternoon of this is worth more than a week of reading leaderboards,
|
||||
and it is the only way to find the failure modes that matter to you.
|
||||
|
||||
Do the same for the embedding model, separately. A good generation model on top
|
||||
of bad retrieval produces fluent, confident, wrong answers, which is the worst
|
||||
failure this stack has: it does not look like a failure.
|
||||
|
||||
## How many replicas
|
||||
|
||||
Not "how many employees". **Concurrency**, which is far lower than headcount
|
||||
because people spend most of their time reading rather than waiting on a model.
|
||||
Two hundred people generate perhaps five to fifteen concurrent requests at
|
||||
peak, for interactive use.
|
||||
|
||||
vLLM batches continuously, so one replica on a capable card serves many
|
||||
concurrent requests. Which means the replica count is usually decided by
|
||||
**redundancy rather than throughput**:
|
||||
|
||||
| Tier | Replicas | Why |
|
||||
|---|---|---|
|
||||
| Embeddings | 2 | Not for capacity. If retrieval stops, every document feature stops. |
|
||||
| Interactive | 2 | So a node can be rebooted during working hours. |
|
||||
| Heavy | 0–1, on demand | Loaded when asked for, stopped when idle. |
|
||||
|
||||
Then measure rather than guess:
|
||||
|
||||
```
|
||||
llmgw ctl bench <model>
|
||||
```
|
||||
|
||||
It climbs concurrency until the SLO breaks and reports the last level that met
|
||||
it. Peak throughput arrives long after latency has become unusable, so
|
||||
throughput alone overstates what the hardware can serve.
|
||||
|
||||
## Where the weights live
|
||||
|
||||
The single highest-value piece of infrastructure here is **a local mirror**: one
|
||||
internal HTTP server holding the model files. Hugging Face is slow, rate
|
||||
limited, and blocked outright on plenty of corporate networks. Eight nodes each
|
||||
pulling forty gigabytes from the internet is a day; from a mirror on the same
|
||||
switch it is minutes. Point a model's `source` at it and llmgw fetches from
|
||||
there.
|
||||
|
||||
**Pin the version.** Put it in the path — `/models/qwen2.5-14b/2026-03/q4_k_m.gguf`
|
||||
— so a repository that changes upstream cannot change what you are running.
|
||||
Without this, a prompt that worked last week gives a different answer this week
|
||||
and nobody can explain why.
|
||||
|
||||
**Quantisation.** For llama.cpp, `q4_k_m` is the practical default: the quality
|
||||
difference from `q8_0` is smaller than most people expect and the memory
|
||||
difference is a factor of two. For vLLM on modern cards, FP8 or AWQ. Run
|
||||
`llmgw ctl plan <model>` to see whether a given precision fits before
|
||||
downloading anything.
|
||||
|
||||
**Disk.** Budget three to five times what you first estimate. Several
|
||||
quantisations, several versions, and partial downloads all accumulate.
|
||||
`llmgw ctl nodes` shows free space per node once `model_dirs` is configured,
|
||||
and `llmgw ctl weights` reclaims files llmgw downloaded and no model needs —
|
||||
never files you placed yourself.
|
||||
|
||||
Under ten nodes, let each keep its own copy; it is simpler and the disk is
|
||||
cheap. Above that, consider a shared filesystem — llmgw notices one and will
|
||||
not count the same free terabyte once per node.
|
||||
|
||||
## Before anyone starts using it
|
||||
|
||||
Set budgets and limits on day one, not later:
|
||||
|
||||
```
|
||||
llmgw keys limits --scope project --id <project> --rpm 60 --budget 50000000
|
||||
```
|
||||
|
||||
The reason is not technical. A resource that appears free and unlimited becomes,
|
||||
three months later, a cost nobody can account for — and introducing limits at
|
||||
that point is a much harder conversation than having had them from the start.
|
||||
+133
@@ -0,0 +1,133 @@
|
||||
# What llmgw stores, and what leaves the machine
|
||||
|
||||
Written for the review that happens before a gateway is allowed near anyone's
|
||||
documents. It describes what the software does. Mapping that onto an
|
||||
obligation — GDPR, a sector rule, an internal policy — is work for whoever
|
||||
holds the obligation; this page exists so that work starts from facts rather
|
||||
than from a vendor's summary.
|
||||
|
||||
Everything below is checkable in the source, and the places to look are named.
|
||||
|
||||
## The short version
|
||||
|
||||
By default llmgw stores **no prompt or completion text anywhere**. It stores
|
||||
who made a request, when, how many tokens, and what it cost.
|
||||
|
||||
Two features change that, and both are off until somebody turns them on for a
|
||||
named project, and neither can be turned on without also saying for how long.
|
||||
|
||||
Nothing on the serving path leaves the cluster.
|
||||
|
||||
## What is stored, exactly
|
||||
|
||||
**Usage records**, one per request: the key, the project, the model asked for,
|
||||
the model that answered, prompt and completion token counts, cost, HTTP status
|
||||
and latency. No message content. The row is in `internal/store/store.go` as
|
||||
`Usage` and there is no field for text in it.
|
||||
|
||||
**API keys**, as SHA-256 hashes. The key itself is shown once, at creation, and
|
||||
never stored — so a leaked database hands over no working credential, and
|
||||
neither llmgw nor an administrator can recover a key somebody lost. A prefix is
|
||||
kept so a key can be identified in a listing without revealing it.
|
||||
|
||||
**People and tenancy**: accounts, the organisation/team/project tree, and the
|
||||
limits attached to it. Passwords are hashed.
|
||||
|
||||
**Sessions** for the browser pages, as tokens.
|
||||
|
||||
**Conversations and agent notes** — only for a project that asked. See below.
|
||||
|
||||
## What is not stored
|
||||
|
||||
Prompts. Completions. Anything the model was shown or produced, unless the
|
||||
project turned on conversations or notes.
|
||||
|
||||
The chat page in llmgw's own web interface keeps the conversation in a
|
||||
JavaScript variable in the browser. Refresh it and the conversation is gone,
|
||||
because it was never anywhere else.
|
||||
|
||||
## Conversations and notes, when they are turned on
|
||||
|
||||
Two features keep text, and both work the same way:
|
||||
|
||||
- **Conversations** (`x-llmgw-thread`) let a client send only its new turn while
|
||||
llmgw replays the history — including to a different model than the one the
|
||||
conversation started on.
|
||||
- **Agent notes** (`/v1/memory`, and MCP tools) let an agent keep something for
|
||||
itself across conversations.
|
||||
|
||||
Both are **off for every project** until an operator runs `llmgw keys threads`
|
||||
or `llmgw keys memory` for that project by name. A request carrying a thread id
|
||||
against a project that has not is refused with an explanation, not quietly
|
||||
served without its history.
|
||||
|
||||
Both **require a retention window**. There is no way to turn either on without
|
||||
saying how many hours, because a store with no expiry is a transcript of
|
||||
everything a team ever typed, kept forever because nobody chose a number. The
|
||||
window is measured from the last turn, so an active conversation does not
|
||||
expire underneath it, and a background sweep removes what has passed it.
|
||||
|
||||
Both are **scoped to a project**, and the scope is part of every lookup rather
|
||||
than a check afterwards. Reading a conversation that belongs to another project
|
||||
returns the same answer as reading one that does not exist, so a leaked id
|
||||
cannot be used to discover what exists elsewhere.
|
||||
|
||||
`DELETE /v1/threads/{id}` removes a conversation and its messages immediately.
|
||||
It is a client endpoint, not an administrative one: the person who said the
|
||||
thing can unsay it without asking somebody else to read it first.
|
||||
|
||||
Turning retention off stops new storage, and existing conversations go at the
|
||||
next hourly sweep.
|
||||
|
||||
## What leaves the machine
|
||||
|
||||
**Nothing on the serving path.** A request arrives, is routed to an engine on
|
||||
your own hardware, and the answer comes back. The router talks to engines, the
|
||||
engines talk to nothing, the agents talk to the control plane. All of it is
|
||||
inside the cluster.
|
||||
|
||||
Two things can reach outside, both started by an operator rather than by a
|
||||
request:
|
||||
|
||||
**Fetching weights.** When a model's `source` is a URL, the agent downloads it.
|
||||
Point that at an internal mirror and nothing leaves at all — which is also the
|
||||
faster arrangement, because eight nodes pulling forty gigabytes from the
|
||||
internet is a day's work and from a local mirror it is minutes. A `source` that
|
||||
is a path on disk is never fetched.
|
||||
|
||||
**`llmgw ctl plan`.** Reads a model's architecture from huggingface.co to work
|
||||
out whether it fits your hardware. An operator command run at planning time,
|
||||
not part of serving, and not run by anything automatically.
|
||||
|
||||
There is no telemetry, no licence check, and no call home. The binary is
|
||||
static, built with `CGO_ENABLED=0`, and runs in a network with no route to the
|
||||
internet as long as the weights are already on disk.
|
||||
|
||||
## What the software cannot do for you
|
||||
|
||||
**The admin port serves pprof**, and pprof can dump the process heap — where
|
||||
API keys sit in plaintext while they are being checked. Bind it to loopback and
|
||||
reach it over SSH. The default does this; a deployment that changes it should
|
||||
know why.
|
||||
|
||||
**The store holds everything.** With conversations turned on it holds text as
|
||||
well as accounting, so where it lives, who can read it and how it is backed up
|
||||
are decisions with the same weight as the ones above. The PostgreSQL DSN
|
||||
belongs in a file readable only by the user llmgw runs as, which is why the
|
||||
configuration takes `@/path/to/file` rather than the address itself.
|
||||
|
||||
**Model weights are yours.** llmgw reclaims only files it downloaded, and it
|
||||
identifies those by a name it generated. Anything you put on a node it will not
|
||||
touch, whatever the catalog says.
|
||||
|
||||
## Where to look
|
||||
|
||||
| Claim | Where |
|
||||
|---|---|
|
||||
| Usage rows carry no text | `internal/store/store.go`, type `Usage` |
|
||||
| Keys are stored hashed | `internal/store/keys.go` |
|
||||
| Conversations are off by default | `internal/store/threads.go`, `ThreadRetention` |
|
||||
| Retention is required | `llmgw keys threads --help` |
|
||||
| Scope is part of the lookup | `internal/store/threads.go`, `Thread` |
|
||||
| Deletion is a client endpoint | `internal/router/threads.go`, `handleDeleteThread` |
|
||||
| Weight fetching is the only serving-path egress | `internal/agent/weights/fetch.go` |
|
||||
@@ -786,6 +786,10 @@ requests. On a slow model, set `timeouts.generation` higher.
|
||||
|
||||
## What to read next
|
||||
|
||||
- `docs/choosing-models.md` — which models to run and how many, if the answer
|
||||
to "we have GPUs, now what" is not already obvious
|
||||
- `docs/data.md` — what is stored and what leaves the machine, for the review
|
||||
that happens before this is allowed near anyone's documents
|
||||
- `docs/reference.md` — every configuration key, generated from the source, so
|
||||
it describes the version you are running rather than the one somebody last
|
||||
wrote about
|
||||
|
||||
Reference in new issue
Block a user