Files
llmgw/deploy/mirror/README.md
T
Ilker ManapandClaude Opus 5 e86457cf02
CI / test (push) Successful in 1m39s
Make the model mirror a file you can copy, not an idea you have to have
"Set up a local mirror" was advice. This is the nginx config, the directory
layout and the reason the layout has a date in it — a repository that changes
upstream changes what you are running, and the first sign is a prompt that
worked last week answering differently this week.

It says what the mirror is not, because both mistakes are easy. It is not a
caching proxy: nothing populates it on demand, deliberately, since a mirror
that reaches the internet at the least convenient moment fails in a way that
looks like llmgw being broken. And it is not needed at all if you already run
a shared filesystem — put the models there, use a path, and nothing is fetched.

It also says when a mirror stops being enough, and that llmgw is positioned to
distribute between nodes because it already knows which of them hold which
weights. That is not built, and the page says so rather than implying it is
coming.

Every behaviour it relies on was checked against the source: an http:// source
is fetched and a path is not, a killed download resumes, a partial file is
never visible under the model's name, and a shared filesystem is not counted
once per node.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-27 08:20:50 +02:00

111 lines
3.6 KiB
Markdown

# A model mirror
One machine that holds the model files, so nodes fetch over your own network
instead of over the internet.
This is the highest-value piece of infrastructure around llmgw and the cheapest.
Eight nodes each pulling forty gigabytes from Hugging Face is a day's work, and
on plenty of corporate networks it is not possible at all. From a mirror on the
same switch it is minutes.
llmgw needs nothing special for this: a model's `source` is a URL, and the URL
can be yours.
## What it is
An HTTP server serving a directory. That is the whole design.
```
/srv/models/
qwen2.5-14b-instruct/
2026-03/
q4_k_m.gguf
bge-m3/
2026-01/
model.safetensors
```
The date in the path is the point of the layout, not decoration. A model
repository that changes upstream changes what you are running, and the first
sign is a prompt that worked last week answering differently this week with
nobody able to say why. Pin it here and that cannot happen.
## nginx
```nginx
server {
listen 80;
server_name models.internal;
root /srv/models;
# Read-only. This is deliberate: a mirror that can be written to over HTTP
# is a way to replace a model with another one, and the nodes would fetch
# it without complaint.
location / {
autoindex on;
# Large files, and the client is a machine with a resumable fetcher.
# Sendfile and no buffering keep memory flat regardless of size.
sendfile on;
tcp_nopush on;
}
}
```
`autoindex` is there so a person can browse what is available; llmgw does not
need it.
## Filling it
```
# On the mirror, once per model and version:
mkdir -p /srv/models/qwen2.5-14b-instruct/2026-03
curl -fL -o /srv/models/qwen2.5-14b-instruct/2026-03/q4_k_m.gguf \
"https://huggingface.co/<repo>/resolve/<revision>/<file>"
```
Use a revision rather than `main` in the source URL. `main` is what moves.
## Pointing llmgw at it
```yaml
- name: qwen2.5-14b-instruct
source: http://models.internal/qwen2.5-14b-instruct/2026-03/q4_k_m.gguf
scaling: { min: 2, max: 2 }
profiles:
- name: gpu
engine: vllm
gpu_class: h100-80g
tp: 1
```
Each node fetches once, in the background, before its engine starts — and
`llmgw ctl replicas` shows how far along it is while it does. A killed download
resumes rather than restarting, and a partial file is never visible under the
model's name, so an interrupted fetch cannot be handed to an engine.
## What this is not
It is not a cache: nothing populates it automatically, and a model you did not
put there is not there. That is on purpose. A mirror that fetches on demand is
a mirror that reaches the internet at the least convenient moment, and in a
network where that is blocked it fails in a way that looks like llmgw being
broken.
It is also not a replacement for a shared filesystem. If you already run one
that every node mounts, put the models on it and use a path as the `source` —
nothing is fetched at all, and llmgw notices the filesystem is shared and will
not count the same free space once per node.
## When this stops being enough
Somewhere above ten or twenty nodes, one mirror serving a forty gigabyte file
to all of them at once becomes the bottleneck instead of the fix. At that point
the options are a filer, a second mirror per rack, or distribution between the
nodes themselves.
llmgw knows which nodes already hold which weights — it is in `ctl status` as
"N models on X/Y nodes" — so distributing between them is a feature it is
positioned to add. It has not been built, because a mirror is enough for
everyone who has needed it so far.