The page described nginx and a curl loop, which was accurate when the mirror was a directory somebody kept by hand. It is a subcommand now, so it says so: the two listeners and why they are two, the API and what each endpoint answers, and the certificates coming from the authority the fleet already has. nginx stays, further down. It serves a directory perfectly well and anyone already running it should not be told to install something to replace it. The systemd unit spells out --base in a comment because it is the flag that goes wrong quietly: it ends up in catalog entries, and a wrong one makes `mirror unused` call files unreferenced while they are in use. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
231 lines
8.4 KiB
Markdown
231 lines
8.4 KiB
Markdown
# A model mirror
|
|
|
|
One machine that holds the model files, so nodes fetch over your own network
|
|
instead of over the internet.
|
|
|
|
This is the highest-value piece of infrastructure around llmgw and the cheapest.
|
|
Eight nodes each pulling forty gigabytes from Hugging Face is a day's work, and
|
|
on plenty of corporate networks it is not possible at all. From a mirror on the
|
|
same switch it is minutes.
|
|
|
|
llmgw needs nothing special for this: a model's `source` is a URL, and the URL
|
|
can be yours.
|
|
|
|
## What it is
|
|
|
|
An HTTP server serving a directory. That is the whole design.
|
|
|
|
```
|
|
/srv/models/
|
|
qwen2.5-14b-instruct/
|
|
2026-03/
|
|
q4_k_m.gguf
|
|
bge-m3/
|
|
2026-01/
|
|
model.safetensors
|
|
```
|
|
|
|
The date in the path is the point of the layout, not decoration. A model
|
|
repository that changes upstream changes what you are running, and the first
|
|
sign is a prompt that worked last week answering differently this week with
|
|
nobody able to say why. Pin it here and that cannot happen.
|
|
|
|
## Running it
|
|
|
|
`llmgw` is the mirror. Same binary as the control plane and the agents, one
|
|
subcommand:
|
|
|
|
```
|
|
sudo mkdir -p /srv/models && sudo chown llmgw:llmgw /srv/models
|
|
llmgw mirror serve --dir /srv/models --base http://models.internal:8081
|
|
```
|
|
|
|
There is a unit in `deploy/systemd/llmgw-mirror.service`. Set `--base` to the
|
|
address nodes reach this machine on — not `localhost`, and not a name only this
|
|
host knows itself by. It is the string that ends up in catalog entries, and a
|
|
wrong one makes `mirror unused` report files as unreferenced when they are in
|
|
use. The next step after that list is deletion.
|
|
|
|
### Two listeners
|
|
|
|
`--listen :8081` serves the weights. Read-only, unauthenticated, Range requests
|
|
supported so an interrupted fetch on a node resumes rather than restarts. This
|
|
is the path eight nodes pull forty gigabytes from.
|
|
|
|
`--api-listen :8082` is the management API. Separate on purpose: putting a
|
|
client certificate in front of the byte path would mean re-certifying every
|
|
agent's fetcher in order to change how models are administered.
|
|
|
|
The API is unauthenticated unless you give it the fleet's CA, and it says so at
|
|
startup rather than leaving you to find out. Anyone who can reach that port can
|
|
start downloads and delete models, so either keep it on a trusted network or
|
|
hand it certificates:
|
|
|
|
```
|
|
llmgw mirror serve --dir /srv/models --base http://models.internal:8081 \
|
|
--tls-ca /etc/llmgw/ca.pem \
|
|
--tls-cert /etc/llmgw/mirror.pem \
|
|
--tls-key /etc/llmgw/mirror-key.pem
|
|
```
|
|
|
|
That is the same CA the agents already use, so the certificates come from the
|
|
authority you have:
|
|
|
|
```
|
|
llmgw ctl cert issue-server mirror --host models.internal --host 10.0.0.146
|
|
llmgw ctl cert issue-client control-plane
|
|
```
|
|
|
|
`--host` has to cover every address the API is called on, because that is what
|
|
a client verifies against. Both sides are checked — the mirror will not accept a
|
|
caller without a certificate from this CA, the same rule the agent stream
|
|
follows.
|
|
|
|
### nginx instead
|
|
|
|
If you already run nginx, it serves a directory perfectly well and you do not
|
|
need `mirror serve` for the bytes:
|
|
|
|
```nginx
|
|
server {
|
|
listen 80;
|
|
server_name models.internal;
|
|
|
|
root /srv/models;
|
|
|
|
# Read-only. This is deliberate: a mirror that can be written to over HTTP
|
|
# is a way to replace a model with another one, and the nodes would fetch
|
|
# it without complaint.
|
|
location / {
|
|
autoindex on;
|
|
# Large files, and the client is a machine with a resumable fetcher.
|
|
# Sendfile and no buffering keep memory flat regardless of size.
|
|
sendfile on;
|
|
tcp_nopush on;
|
|
}
|
|
}
|
|
```
|
|
|
|
`autoindex` is there so a person can browse what is available; llmgw does not
|
|
need it.
|
|
|
|
## Filling it
|
|
|
|
From the mirror itself:
|
|
|
|
```
|
|
llmgw mirror add \
|
|
--model qwen2.5-14b-instruct --version 2026-03 \
|
|
"https://huggingface.co/<repo>/resolve/<revision>/q4_k_m.gguf"
|
|
```
|
|
|
|
Use a revision rather than `main` in the source URL. `main` is what moves.
|
|
|
|
It downloads to a temporary name and renames when complete, so an interrupted
|
|
download never leaves something a catalog could be pointed at, and it prints the
|
|
sha256 when it lands. Compare that against what upstream publishes — Hugging
|
|
Face gives it as `lfs.oid` from
|
|
`https://huggingface.co/api/models/<repo>/tree/<revision>`. A mirror is the one
|
|
place in this system where a corrupt file is copied to every node instead of
|
|
being caught once.
|
|
|
|
It refuses to overwrite a version that is already there. Add a new one beside
|
|
it: that refusal is what makes the old weights still available when the new ones
|
|
turn out to be worse.
|
|
|
|
## Driving it from elsewhere
|
|
|
|
The same fetch over the API, so the control plane — or anything else — can ask
|
|
for a model without anyone logging in:
|
|
|
|
```
|
|
curl -X POST http://models.internal:8082/v1/models -d '{
|
|
"source": "https://huggingface.co/<repo>/resolve/<revision>/q4_k_m.gguf",
|
|
"model": "qwen2.5-14b-instruct",
|
|
"version": "2026-03"
|
|
}'
|
|
```
|
|
|
|
The mirror does the downloading. A control plane that pulled the bytes itself
|
|
and pushed them here would carry every model twice over the network, through the
|
|
one machine least able to spare the bandwidth.
|
|
|
|
It answers `202` with the URL a catalog entry should use, and the bytes are not
|
|
here yet. Poll for them:
|
|
|
|
| | |
|
|
|---|---|
|
|
| `GET /v1/models` | everything held or arriving, and the base to build URLs from |
|
|
| `POST /v1/models` | fetch this source into model/version |
|
|
| `GET /v1/models/{model}/{version}` | `fetching` with a byte count, `present`, or `failed` with the reason |
|
|
| `DELETE /v1/models/{model}/{version}` | remove a version; refuses while it is downloading |
|
|
| `GET /v1/health` | up *and* able to read its own directory |
|
|
|
|
A failed fetch stays `failed` with what went wrong, rather than disappearing:
|
|
otherwise the thing waiting on that model waits forever with nothing to say why.
|
|
Asking twice for the same download joins the one already running instead of
|
|
starting a rival writer on the same file.
|
|
|
|
`DELETE` takes the whole version, never one file of it. A model split across
|
|
several files is useless with one missing, and half a version on disk looks
|
|
present and will not load.
|
|
|
|
## Retiring a model
|
|
|
|
```
|
|
llmgw mirror unused # what no catalog entry refers to
|
|
llmgw mirror remove --model <m> --version <v> --file <f>
|
|
```
|
|
|
|
`unused` compares against every `source` in the running catalog, so the answer
|
|
is what can go rather than what looks old. It refuses to answer when it cannot
|
|
reach the control plane: an empty catalog and an unreachable one produce the
|
|
same list — everything — and the obvious next step after that list is deletion.
|
|
|
|
Retire the catalog entry first, let the fleet settle, then remove. A node that
|
|
has to refetch a file you deleted cannot, and the failure arrives later, on the
|
|
node, looking like something else.
|
|
|
|
## Pointing llmgw at it
|
|
|
|
```yaml
|
|
- name: qwen2.5-14b-instruct
|
|
source: http://models.internal/qwen2.5-14b-instruct/2026-03/q4_k_m.gguf
|
|
scaling: { min: 2, max: 2 }
|
|
profiles:
|
|
- name: gpu
|
|
engine: vllm
|
|
gpu_class: h100-80g
|
|
tp: 1
|
|
```
|
|
|
|
Each node fetches once, in the background, before its engine starts — and
|
|
`llmgw ctl replicas` shows how far along it is while it does. A killed download
|
|
resumes rather than restarting, and a partial file is never visible under the
|
|
model's name, so an interrupted fetch cannot be handed to an engine.
|
|
|
|
## What this is not
|
|
|
|
It is not a cache. Nothing populates it on its own: a model is there because
|
|
somebody asked for it, by hand or over the API, and one nobody asked for is not
|
|
there. That is on purpose. A mirror that fetched on demand would reach the
|
|
internet at the least convenient moment, and in a network where that is blocked
|
|
it would fail in a way that looks like llmgw being broken.
|
|
|
|
It is also not a replacement for a shared filesystem. If you already run one
|
|
that every node mounts, put the models on it and use a path as the `source` —
|
|
nothing is fetched at all, and llmgw notices the filesystem is shared and will
|
|
not count the same free space once per node.
|
|
|
|
## When this stops being enough
|
|
|
|
Somewhere above ten or twenty nodes, one mirror serving a forty gigabyte file
|
|
to all of them at once becomes the bottleneck instead of the fix. At that point
|
|
the options are a filer, a second mirror per rack, or distribution between the
|
|
nodes themselves.
|
|
|
|
llmgw knows which nodes already hold which weights — it is in `ctl status` as
|
|
"N models on X/Y nodes" — so distributing between them is a feature it is
|
|
positioned to add. It has not been built, because a mirror is enough for
|
|
everyone who has needed it so far.
|