diff --git a/README.md b/README.md index e35c8a78..d8cd2431 100644 --- a/README.md +++ b/README.md @@ -64,6 +64,8 @@ on a machine that has them. - [docs/reference.md](docs/reference.md) — every configuration key, generated from the structs and checked by a test, so it cannot describe a setting that does not exist or miss one that does +- [deploy/mirror/](deploy/mirror/) — a model mirror, which is the cheapest + and most useful thing to build around llmgw - [deploy/ha/](deploy/ha/) — running two control planes, and the one health check that has to be right - [docs/node-onboarding.md](docs/node-onboarding.md), diff --git a/deploy/mirror/README.md b/deploy/mirror/README.md new file mode 100644 index 00000000..4570a8df --- /dev/null +++ b/deploy/mirror/README.md @@ -0,0 +1,110 @@ +# A model mirror + +One machine that holds the model files, so nodes fetch over your own network +instead of over the internet. + +This is the highest-value piece of infrastructure around llmgw and the cheapest. +Eight nodes each pulling forty gigabytes from Hugging Face is a day's work, and +on plenty of corporate networks it is not possible at all. From a mirror on the +same switch it is minutes. + +llmgw needs nothing special for this: a model's `source` is a URL, and the URL +can be yours. + +## What it is + +An HTTP server serving a directory. That is the whole design. + +``` +/srv/models/ + qwen2.5-14b-instruct/ + 2026-03/ + q4_k_m.gguf + bge-m3/ + 2026-01/ + model.safetensors +``` + +The date in the path is the point of the layout, not decoration. A model +repository that changes upstream changes what you are running, and the first +sign is a prompt that worked last week answering differently this week with +nobody able to say why. Pin it here and that cannot happen. + +## nginx + +```nginx +server { + listen 80; + server_name models.internal; + + root /srv/models; + + # Read-only. This is deliberate: a mirror that can be written to over HTTP + # is a way to replace a model with another one, and the nodes would fetch + # it without complaint. + location / { + autoindex on; + # Large files, and the client is a machine with a resumable fetcher. + # Sendfile and no buffering keep memory flat regardless of size. + sendfile on; + tcp_nopush on; + } +} +``` + +`autoindex` is there so a person can browse what is available; llmgw does not +need it. + +## Filling it + +``` +# On the mirror, once per model and version: +mkdir -p /srv/models/qwen2.5-14b-instruct/2026-03 +curl -fL -o /srv/models/qwen2.5-14b-instruct/2026-03/q4_k_m.gguf \ + "https://huggingface.co//resolve//" +``` + +Use a revision rather than `main` in the source URL. `main` is what moves. + +## Pointing llmgw at it + +```yaml +- name: qwen2.5-14b-instruct + source: http://models.internal/qwen2.5-14b-instruct/2026-03/q4_k_m.gguf + scaling: { min: 2, max: 2 } + profiles: + - name: gpu + engine: vllm + gpu_class: h100-80g + tp: 1 +``` + +Each node fetches once, in the background, before its engine starts — and +`llmgw ctl replicas` shows how far along it is while it does. A killed download +resumes rather than restarting, and a partial file is never visible under the +model's name, so an interrupted fetch cannot be handed to an engine. + +## What this is not + +It is not a cache: nothing populates it automatically, and a model you did not +put there is not there. That is on purpose. A mirror that fetches on demand is +a mirror that reaches the internet at the least convenient moment, and in a +network where that is blocked it fails in a way that looks like llmgw being +broken. + +It is also not a replacement for a shared filesystem. If you already run one +that every node mounts, put the models on it and use a path as the `source` — +nothing is fetched at all, and llmgw notices the filesystem is shared and will +not count the same free space once per node. + +## When this stops being enough + +Somewhere above ten or twenty nodes, one mirror serving a forty gigabyte file +to all of them at once becomes the bottleneck instead of the fix. At that point +the options are a filer, a second mirror per rack, or distribution between the +nodes themselves. + +llmgw knows which nodes already hold which weights — it is in `ctl status` as +"N models on X/Y nodes" — so distributing between them is a feature it is +positioned to add. It has not been built, because a mirror is enough for +everyone who has needed it so far. diff --git a/docs/choosing-models.md b/docs/choosing-models.md index 327ef09a..0de296af 100644 --- a/docs/choosing-models.md +++ b/docs/choosing-models.md @@ -108,7 +108,8 @@ throughput alone overstates what the hardware can serve. ## Where the weights live The single highest-value piece of infrastructure here is **a local mirror**: one -internal HTTP server holding the model files. Hugging Face is slow, rate +internal HTTP server holding the model files. [deploy/mirror/](../deploy/mirror/) +has the nginx config and the directory layout. Hugging Face is slow, rate limited, and blocked outright on plenty of corporate networks. Eight nodes each pulling forty gigabytes from the internet is a day; from a mirror on the same switch it is minutes. Point a model's `source` at it and llmgw fetches from