llmgw/docs/choosing-models.md

Choosing what to run

llmgw serves whatever you give it and has no opinion about which models those are. This page has an opinion, because "we have GPUs, now what" is the question that stalls most of these projects — and the usual first answer, one large model, is the wrong shape.

Model releases move faster than documentation. Treat the families named here as a starting point and check what is current before committing; the reasoning is what should outlive the names.

Three jobs, not one model

The requests a company actually makes fall into three groups with different needs, and no single model serves all three well.

Interactive — chat, autocomplete, short questions. Most of your traffic. Here latency beats quality: a person waits two seconds and does not wait twenty. A 7–14B class model, always resident.

Heavy — long document analysis, code generation, anything that needs reasoning. Quality beats latency, and a queue is acceptable. A 30–70B class model, loaded on demand:

scaling: { min: 0, max: 1, idle_timeout: 30m }

llmgw starts it when somebody asks and stops it when nobody has, so it costs nothing while it is not in use. The first request waits for the load, and the client is told that rather than refused.

Embeddings — and this is the one that gets forgotten. Most "let us chat with our documents" projects are retrieval projects, and retrieval runs on an embedding model. It is small, it is cheap, and it carries the highest request volume of the three: every document chunk once, every query forever. llmgw routes /v1/embeddings like any other endpoint.

A deployment with the first two and not the third can answer questions from what the model already knew. It cannot answer questions about your company.

Swedish and English

The first thing to know is that this combination is harder than either alone.

Open models are strongest in English by a wide margin. Swedish is served — better than most languages its size, because the Nordic languages appear in multilingual training sets — but the gap is real, and it widens as models get smaller. A 7B model that feels fine in English can feel noticeably worse in Swedish, which means the interactive tier may need to be larger than an English-only deployment would choose. Test before deciding; do not assume the English benchmark transfers.

Two things matter more than the model's headline multilingual score:

Cross-lingual retrieval. A Swedish company writes internal documents in Swedish and reads technical documentation in English. Somebody will ask a question in Swedish whose answer is in an English document. An embedding model that handles both languages separately may still fail to connect them, and it fails silently — the model gives a confident answer from the wrong source, and nobody notices because the answer reads well. Test this explicitly: ask in one language, check that documents in the other are retrieved.

Swedish compounds. Kundtjänstmedarbetare is one token-hostile word where English would use three. This hurts keyword search more than dense retrieval, so if you run hybrid search, expect the lexical half to underperform on Swedish and weight accordingly.

How to actually decide

Benchmarks will not settle this. Take twenty real documents and twenty real questions from the company — in both languages, including at least a few where the question and the answer are in different languages — and compare candidates on those. An afternoon of this is worth more than a week of reading leaderboards, and it is the only way to find the failure modes that matter to you.

Do the same for the embedding model, separately. A good generation model on top of bad retrieval produces fluent, confident, wrong answers, which is the worst failure this stack has: it does not look like a failure.

How many replicas

Not "how many employees". Concurrency, which is far lower than headcount because people spend most of their time reading rather than waiting on a model. Two hundred people generate perhaps five to fifteen concurrent requests at peak, for interactive use.

vLLM batches continuously, so one replica on a capable card serves many concurrent requests. Which means the replica count is usually decided by redundancy rather than throughput:

Tier	Replicas	Why
Embeddings	2	Not for capacity. If retrieval stops, every document feature stops.
Interactive	2	So a node can be rebooted during working hours.
Heavy	0–1, on demand	Loaded when asked for, stopped when idle.

Then measure rather than guess:

llmgw ctl bench <model>

It climbs concurrency until the SLO breaks and reports the last level that met it. Peak throughput arrives long after latency has become unusable, so throughput alone overstates what the hardware can serve.

Where the weights live

The single highest-value piece of infrastructure here is a local mirror: one internal HTTP server holding the model files. deploy/mirror/ has the nginx config and the directory layout. Hugging Face is slow, rate limited, and blocked outright on plenty of corporate networks. Eight nodes each pulling forty gigabytes from the internet is a day; from a mirror on the same switch it is minutes. Point a model's source at it and llmgw fetches from there.

Pin the version. Put it in the path — /models/qwen2.5-14b/2026-03/q4_k_m.gguf — so a repository that changes upstream cannot change what you are running. Without this, a prompt that worked last week gives a different answer this week and nobody can explain why.

Quantisation. For llama.cpp, q4_k_m is the practical default: the quality difference from q8_0 is smaller than most people expect and the memory difference is a factor of two. For vLLM on modern cards, FP8 or AWQ. Run llmgw ctl plan <model> to see whether a given precision fits before downloading anything.

Disk. Budget three to five times what you first estimate. Several quantisations, several versions, and partial downloads all accumulate. llmgw ctl nodes shows free space per node once model_dirs is configured, and llmgw ctl weights reclaims files llmgw downloaded and no model needs — never files you placed yourself.

Under ten nodes, let each keep its own copy; it is simpler and the disk is cheap. Above that, consider a shared filesystem — llmgw notices one and will not count the same free terabyte once per node.

Before anyone starts using it

Set budgets and limits on day one, not later:

llmgw keys limits --scope project --id <project> --rpm 60 --budget 50000000

The reason is not technical. A resource that appears free and unlimited becomes, three months later, a cost nobody can account for — and introducing limits at that point is a much harder conversation than having had them from the start.