diff --git a/README.md b/README.md index 19c137d..e35c8a7 100644 --- a/README.md +++ b/README.md @@ -57,6 +57,10 @@ on a machine that has them. - [docs/install.md](docs/install.md) — installing from nothing: control plane, nodes, engines, weights, first request - [docs/quickstart.md](docs/quickstart.md) — the short path +- [docs/choosing-models.md](docs/choosing-models.md) — which models to run, + how many of each, and where the weights should live +- [docs/data.md](docs/data.md) — what llmgw stores, what it does not, and + what leaves the machine; written for a security review - [docs/reference.md](docs/reference.md) — every configuration key, generated from the structs and checked by a test, so it cannot describe a setting that does not exist or miss one that does diff --git a/docs/choosing-models.md b/docs/choosing-models.md new file mode 100644 index 0000000..327ef09 --- /dev/null +++ b/docs/choosing-models.md @@ -0,0 +1,148 @@ +# Choosing what to run + +llmgw serves whatever you give it and has no opinion about which models those +are. This page has an opinion, because "we have GPUs, now what" is the question +that stalls most of these projects — and the usual first answer, one large +model, is the wrong shape. + +Model releases move faster than documentation. Treat the families named here as +a starting point and check what is current before committing; the reasoning is +what should outlive the names. + +## Three jobs, not one model + +The requests a company actually makes fall into three groups with different +needs, and no single model serves all three well. + +**Interactive** — chat, autocomplete, short questions. Most of your traffic. +Here latency beats quality: a person waits two seconds and does not wait +twenty. A 7–14B class model, always resident. + +**Heavy** — long document analysis, code generation, anything that needs +reasoning. Quality beats latency, and a queue is acceptable. A 30–70B class +model, loaded on demand: + +```yaml +scaling: { min: 0, max: 1, idle_timeout: 30m } +``` + +llmgw starts it when somebody asks and stops it when nobody has, so it costs +nothing while it is not in use. The first request waits for the load, and the +client is told that rather than refused. + +**Embeddings** — and this is the one that gets forgotten. Most "let us chat +with our documents" projects are retrieval projects, and retrieval runs on an +embedding model. It is small, it is cheap, and it carries the highest request +volume of the three: every document chunk once, every query forever. llmgw +routes `/v1/embeddings` like any other endpoint. + +A deployment with the first two and not the third can answer questions from +what the model already knew. It cannot answer questions about your company. + +## Swedish and English + +The first thing to know is that this combination is harder than either alone. + +Open models are strongest in English by a wide margin. Swedish is served — +better than most languages its size, because the Nordic languages appear in +multilingual training sets — but the gap is real, and it widens as models get +smaller. **A 7B model that feels fine in English can feel noticeably worse in +Swedish**, which means the interactive tier may need to be larger than an +English-only deployment would choose. Test before deciding; do not assume the +English benchmark transfers. + +Two things matter more than the model's headline multilingual score: + +**Cross-lingual retrieval.** A Swedish company writes internal documents in +Swedish and reads technical documentation in English. Somebody will ask a +question in Swedish whose answer is in an English document. An embedding model +that handles both languages separately may still fail to connect them, and it +fails silently — the model gives a confident answer from the wrong source, and +nobody notices because the answer reads well. Test this explicitly: ask in one +language, check that documents in the other are retrieved. + +**Swedish compounds.** *Kundtjänstmedarbetare* is one token-hostile word where +English would use three. This hurts keyword search more than dense retrieval, +so if you run hybrid search, expect the lexical half to underperform on Swedish +and weight accordingly. + +## How to actually decide + +Benchmarks will not settle this. Take twenty real documents and twenty real +questions from the company — in both languages, including at least a few where +the question and the answer are in different languages — and compare candidates +on those. An afternoon of this is worth more than a week of reading leaderboards, +and it is the only way to find the failure modes that matter to you. + +Do the same for the embedding model, separately. A good generation model on top +of bad retrieval produces fluent, confident, wrong answers, which is the worst +failure this stack has: it does not look like a failure. + +## How many replicas + +Not "how many employees". **Concurrency**, which is far lower than headcount +because people spend most of their time reading rather than waiting on a model. +Two hundred people generate perhaps five to fifteen concurrent requests at +peak, for interactive use. + +vLLM batches continuously, so one replica on a capable card serves many +concurrent requests. Which means the replica count is usually decided by +**redundancy rather than throughput**: + +| Tier | Replicas | Why | +|---|---|---| +| Embeddings | 2 | Not for capacity. If retrieval stops, every document feature stops. | +| Interactive | 2 | So a node can be rebooted during working hours. | +| Heavy | 0–1, on demand | Loaded when asked for, stopped when idle. | + +Then measure rather than guess: + +``` +llmgw ctl bench +``` + +It climbs concurrency until the SLO breaks and reports the last level that met +it. Peak throughput arrives long after latency has become unusable, so +throughput alone overstates what the hardware can serve. + +## Where the weights live + +The single highest-value piece of infrastructure here is **a local mirror**: one +internal HTTP server holding the model files. Hugging Face is slow, rate +limited, and blocked outright on plenty of corporate networks. Eight nodes each +pulling forty gigabytes from the internet is a day; from a mirror on the same +switch it is minutes. Point a model's `source` at it and llmgw fetches from +there. + +**Pin the version.** Put it in the path — `/models/qwen2.5-14b/2026-03/q4_k_m.gguf` +— so a repository that changes upstream cannot change what you are running. +Without this, a prompt that worked last week gives a different answer this week +and nobody can explain why. + +**Quantisation.** For llama.cpp, `q4_k_m` is the practical default: the quality +difference from `q8_0` is smaller than most people expect and the memory +difference is a factor of two. For vLLM on modern cards, FP8 or AWQ. Run +`llmgw ctl plan ` to see whether a given precision fits before +downloading anything. + +**Disk.** Budget three to five times what you first estimate. Several +quantisations, several versions, and partial downloads all accumulate. +`llmgw ctl nodes` shows free space per node once `model_dirs` is configured, +and `llmgw ctl weights` reclaims files llmgw downloaded and no model needs — +never files you placed yourself. + +Under ten nodes, let each keep its own copy; it is simpler and the disk is +cheap. Above that, consider a shared filesystem — llmgw notices one and will +not count the same free terabyte once per node. + +## Before anyone starts using it + +Set budgets and limits on day one, not later: + +``` +llmgw keys limits --scope project --id --rpm 60 --budget 50000000 +``` + +The reason is not technical. A resource that appears free and unlimited becomes, +three months later, a cost nobody can account for — and introducing limits at +that point is a much harder conversation than having had them from the start. diff --git a/docs/data.md b/docs/data.md new file mode 100644 index 0000000..6c4561d --- /dev/null +++ b/docs/data.md @@ -0,0 +1,133 @@ +# What llmgw stores, and what leaves the machine + +Written for the review that happens before a gateway is allowed near anyone's +documents. It describes what the software does. Mapping that onto an +obligation — GDPR, a sector rule, an internal policy — is work for whoever +holds the obligation; this page exists so that work starts from facts rather +than from a vendor's summary. + +Everything below is checkable in the source, and the places to look are named. + +## The short version + +By default llmgw stores **no prompt or completion text anywhere**. It stores +who made a request, when, how many tokens, and what it cost. + +Two features change that, and both are off until somebody turns them on for a +named project, and neither can be turned on without also saying for how long. + +Nothing on the serving path leaves the cluster. + +## What is stored, exactly + +**Usage records**, one per request: the key, the project, the model asked for, +the model that answered, prompt and completion token counts, cost, HTTP status +and latency. No message content. The row is in `internal/store/store.go` as +`Usage` and there is no field for text in it. + +**API keys**, as SHA-256 hashes. The key itself is shown once, at creation, and +never stored — so a leaked database hands over no working credential, and +neither llmgw nor an administrator can recover a key somebody lost. A prefix is +kept so a key can be identified in a listing without revealing it. + +**People and tenancy**: accounts, the organisation/team/project tree, and the +limits attached to it. Passwords are hashed. + +**Sessions** for the browser pages, as tokens. + +**Conversations and agent notes** — only for a project that asked. See below. + +## What is not stored + +Prompts. Completions. Anything the model was shown or produced, unless the +project turned on conversations or notes. + +The chat page in llmgw's own web interface keeps the conversation in a +JavaScript variable in the browser. Refresh it and the conversation is gone, +because it was never anywhere else. + +## Conversations and notes, when they are turned on + +Two features keep text, and both work the same way: + +- **Conversations** (`x-llmgw-thread`) let a client send only its new turn while + llmgw replays the history — including to a different model than the one the + conversation started on. +- **Agent notes** (`/v1/memory`, and MCP tools) let an agent keep something for + itself across conversations. + +Both are **off for every project** until an operator runs `llmgw keys threads` +or `llmgw keys memory` for that project by name. A request carrying a thread id +against a project that has not is refused with an explanation, not quietly +served without its history. + +Both **require a retention window**. There is no way to turn either on without +saying how many hours, because a store with no expiry is a transcript of +everything a team ever typed, kept forever because nobody chose a number. The +window is measured from the last turn, so an active conversation does not +expire underneath it, and a background sweep removes what has passed it. + +Both are **scoped to a project**, and the scope is part of every lookup rather +than a check afterwards. Reading a conversation that belongs to another project +returns the same answer as reading one that does not exist, so a leaked id +cannot be used to discover what exists elsewhere. + +`DELETE /v1/threads/{id}` removes a conversation and its messages immediately. +It is a client endpoint, not an administrative one: the person who said the +thing can unsay it without asking somebody else to read it first. + +Turning retention off stops new storage, and existing conversations go at the +next hourly sweep. + +## What leaves the machine + +**Nothing on the serving path.** A request arrives, is routed to an engine on +your own hardware, and the answer comes back. The router talks to engines, the +engines talk to nothing, the agents talk to the control plane. All of it is +inside the cluster. + +Two things can reach outside, both started by an operator rather than by a +request: + +**Fetching weights.** When a model's `source` is a URL, the agent downloads it. +Point that at an internal mirror and nothing leaves at all — which is also the +faster arrangement, because eight nodes pulling forty gigabytes from the +internet is a day's work and from a local mirror it is minutes. A `source` that +is a path on disk is never fetched. + +**`llmgw ctl plan`.** Reads a model's architecture from huggingface.co to work +out whether it fits your hardware. An operator command run at planning time, +not part of serving, and not run by anything automatically. + +There is no telemetry, no licence check, and no call home. The binary is +static, built with `CGO_ENABLED=0`, and runs in a network with no route to the +internet as long as the weights are already on disk. + +## What the software cannot do for you + +**The admin port serves pprof**, and pprof can dump the process heap — where +API keys sit in plaintext while they are being checked. Bind it to loopback and +reach it over SSH. The default does this; a deployment that changes it should +know why. + +**The store holds everything.** With conversations turned on it holds text as +well as accounting, so where it lives, who can read it and how it is backed up +are decisions with the same weight as the ones above. The PostgreSQL DSN +belongs in a file readable only by the user llmgw runs as, which is why the +configuration takes `@/path/to/file` rather than the address itself. + +**Model weights are yours.** llmgw reclaims only files it downloaded, and it +identifies those by a name it generated. Anything you put on a node it will not +touch, whatever the catalog says. + +## Where to look + +| Claim | Where | +|---|---| +| Usage rows carry no text | `internal/store/store.go`, type `Usage` | +| Keys are stored hashed | `internal/store/keys.go` | +| Conversations are off by default | `internal/store/threads.go`, `ThreadRetention` | +| Retention is required | `llmgw keys threads --help` | +| Scope is part of the lookup | `internal/store/threads.go`, `Thread` | +| Deletion is a client endpoint | `internal/router/threads.go`, `handleDeleteThread` | +| Weight fetching is the only serving-path egress | `internal/agent/weights/fetch.go` | diff --git a/docs/install.md b/docs/install.md index da4400c..aece215 100644 --- a/docs/install.md +++ b/docs/install.md @@ -786,6 +786,10 @@ requests. On a slow model, set `timeouts.generation` higher. ## What to read next +- `docs/choosing-models.md` — which models to run and how many, if the answer + to "we have GPUs, now what" is not already obvious +- `docs/data.md` — what is stored and what leaves the machine, for the review + that happens before this is allowed near anyone's documents - `docs/reference.md` — every configuration key, generated from the source, so it describes the version you are running rather than the one somebody last wrote about