CI / test (push) Successful in 1m34s
Neither had a page. "We have GPUs, now what" was left to the reader, and what llmgw stores was scattered across four documents and a changelog — which is the same as not being written down when somebody is doing a security review. docs/data.md describes the behaviour and leaves the legal mapping to whoever holds the obligation. Every claim in it was checked against the source, and it names where to look for each: usage rows carry no text field, keys are stored hashed, conversations are off until a named project turns them on and cannot be turned on without a retention window. It also states the egress exactly, which took reading every outbound call to say honestly: nothing on the serving path leaves the cluster, and the two things that can reach outside — fetching weights, and ctl plan reading a model's architecture — are both started by an operator. Weight fetching points wherever you tell it, including an internal mirror. docs/choosing-models.md is the opinion llmgw itself does not have. Three jobs rather than one model, with the third — embeddings — being the one that gets forgotten and the one that decides whether the deployment can answer questions about your company or only questions in general. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
149 lines
6.7 KiB
Markdown
149 lines
6.7 KiB
Markdown
# Choosing what to run
|
||
|
||
llmgw serves whatever you give it and has no opinion about which models those
|
||
are. This page has an opinion, because "we have GPUs, now what" is the question
|
||
that stalls most of these projects — and the usual first answer, one large
|
||
model, is the wrong shape.
|
||
|
||
Model releases move faster than documentation. Treat the families named here as
|
||
a starting point and check what is current before committing; the reasoning is
|
||
what should outlive the names.
|
||
|
||
## Three jobs, not one model
|
||
|
||
The requests a company actually makes fall into three groups with different
|
||
needs, and no single model serves all three well.
|
||
|
||
**Interactive** — chat, autocomplete, short questions. Most of your traffic.
|
||
Here latency beats quality: a person waits two seconds and does not wait
|
||
twenty. A 7–14B class model, always resident.
|
||
|
||
**Heavy** — long document analysis, code generation, anything that needs
|
||
reasoning. Quality beats latency, and a queue is acceptable. A 30–70B class
|
||
model, loaded on demand:
|
||
|
||
```yaml
|
||
scaling: { min: 0, max: 1, idle_timeout: 30m }
|
||
```
|
||
|
||
llmgw starts it when somebody asks and stops it when nobody has, so it costs
|
||
nothing while it is not in use. The first request waits for the load, and the
|
||
client is told that rather than refused.
|
||
|
||
**Embeddings** — and this is the one that gets forgotten. Most "let us chat
|
||
with our documents" projects are retrieval projects, and retrieval runs on an
|
||
embedding model. It is small, it is cheap, and it carries the highest request
|
||
volume of the three: every document chunk once, every query forever. llmgw
|
||
routes `/v1/embeddings` like any other endpoint.
|
||
|
||
A deployment with the first two and not the third can answer questions from
|
||
what the model already knew. It cannot answer questions about your company.
|
||
|
||
## Swedish and English
|
||
|
||
The first thing to know is that this combination is harder than either alone.
|
||
|
||
Open models are strongest in English by a wide margin. Swedish is served —
|
||
better than most languages its size, because the Nordic languages appear in
|
||
multilingual training sets — but the gap is real, and it widens as models get
|
||
smaller. **A 7B model that feels fine in English can feel noticeably worse in
|
||
Swedish**, which means the interactive tier may need to be larger than an
|
||
English-only deployment would choose. Test before deciding; do not assume the
|
||
English benchmark transfers.
|
||
|
||
Two things matter more than the model's headline multilingual score:
|
||
|
||
**Cross-lingual retrieval.** A Swedish company writes internal documents in
|
||
Swedish and reads technical documentation in English. Somebody will ask a
|
||
question in Swedish whose answer is in an English document. An embedding model
|
||
that handles both languages separately may still fail to connect them, and it
|
||
fails silently — the model gives a confident answer from the wrong source, and
|
||
nobody notices because the answer reads well. Test this explicitly: ask in one
|
||
language, check that documents in the other are retrieved.
|
||
|
||
**Swedish compounds.** *Kundtjänstmedarbetare* is one token-hostile word where
|
||
English would use three. This hurts keyword search more than dense retrieval,
|
||
so if you run hybrid search, expect the lexical half to underperform on Swedish
|
||
and weight accordingly.
|
||
|
||
## How to actually decide
|
||
|
||
Benchmarks will not settle this. Take twenty real documents and twenty real
|
||
questions from the company — in both languages, including at least a few where
|
||
the question and the answer are in different languages — and compare candidates
|
||
on those. An afternoon of this is worth more than a week of reading leaderboards,
|
||
and it is the only way to find the failure modes that matter to you.
|
||
|
||
Do the same for the embedding model, separately. A good generation model on top
|
||
of bad retrieval produces fluent, confident, wrong answers, which is the worst
|
||
failure this stack has: it does not look like a failure.
|
||
|
||
## How many replicas
|
||
|
||
Not "how many employees". **Concurrency**, which is far lower than headcount
|
||
because people spend most of their time reading rather than waiting on a model.
|
||
Two hundred people generate perhaps five to fifteen concurrent requests at
|
||
peak, for interactive use.
|
||
|
||
vLLM batches continuously, so one replica on a capable card serves many
|
||
concurrent requests. Which means the replica count is usually decided by
|
||
**redundancy rather than throughput**:
|
||
|
||
| Tier | Replicas | Why |
|
||
|---|---|---|
|
||
| Embeddings | 2 | Not for capacity. If retrieval stops, every document feature stops. |
|
||
| Interactive | 2 | So a node can be rebooted during working hours. |
|
||
| Heavy | 0–1, on demand | Loaded when asked for, stopped when idle. |
|
||
|
||
Then measure rather than guess:
|
||
|
||
```
|
||
llmgw ctl bench <model>
|
||
```
|
||
|
||
It climbs concurrency until the SLO breaks and reports the last level that met
|
||
it. Peak throughput arrives long after latency has become unusable, so
|
||
throughput alone overstates what the hardware can serve.
|
||
|
||
## Where the weights live
|
||
|
||
The single highest-value piece of infrastructure here is **a local mirror**: one
|
||
internal HTTP server holding the model files. Hugging Face is slow, rate
|
||
limited, and blocked outright on plenty of corporate networks. Eight nodes each
|
||
pulling forty gigabytes from the internet is a day; from a mirror on the same
|
||
switch it is minutes. Point a model's `source` at it and llmgw fetches from
|
||
there.
|
||
|
||
**Pin the version.** Put it in the path — `/models/qwen2.5-14b/2026-03/q4_k_m.gguf`
|
||
— so a repository that changes upstream cannot change what you are running.
|
||
Without this, a prompt that worked last week gives a different answer this week
|
||
and nobody can explain why.
|
||
|
||
**Quantisation.** For llama.cpp, `q4_k_m` is the practical default: the quality
|
||
difference from `q8_0` is smaller than most people expect and the memory
|
||
difference is a factor of two. For vLLM on modern cards, FP8 or AWQ. Run
|
||
`llmgw ctl plan <model>` to see whether a given precision fits before
|
||
downloading anything.
|
||
|
||
**Disk.** Budget three to five times what you first estimate. Several
|
||
quantisations, several versions, and partial downloads all accumulate.
|
||
`llmgw ctl nodes` shows free space per node once `model_dirs` is configured,
|
||
and `llmgw ctl weights` reclaims files llmgw downloaded and no model needs —
|
||
never files you placed yourself.
|
||
|
||
Under ten nodes, let each keep its own copy; it is simpler and the disk is
|
||
cheap. Above that, consider a shared filesystem — llmgw notices one and will
|
||
not count the same free terabyte once per node.
|
||
|
||
## Before anyone starts using it
|
||
|
||
Set budgets and limits on day one, not later:
|
||
|
||
```
|
||
llmgw keys limits --scope project --id <project> --rpm 60 --budget 50000000
|
||
```
|
||
|
||
The reason is not technical. A resource that appears free and unlimited becomes,
|
||
three months later, a cost nobody can account for — and introducing limits at
|
||
that point is a much harder conversation than having had them from the start.
|