Document the mirror that exists rather than the one you assemble
The page described nginx and a curl loop, which was accurate when the mirror was a directory somebody kept by hand. It is a subcommand now, so it says so: the two listeners and why they are two, the API and what each endpoint answers, and the certificates coming from the authority the fleet already has. nginx stays, further down. It serves a directory perfectly well and anyone already running it should not be told to install something to replace it. The systemd unit spells out --base in a comment because it is the flag that goes wrong quietly: it ends up in catalog entries, and a wrong one makes `mirror unused` call files unreferenced while they are in use. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
dde2bcab19
commit
afb3797e57
2 files changed
+159
-10
No files matched your search
+130
-10
@@ -30,7 +30,61 @@ repository that changes upstream changes what you are running, and the first
|
||||
sign is a prompt that worked last week answering differently this week with
|
||||
nobody able to say why. Pin it here and that cannot happen.
|
||||
|
||||
## nginx
|
||||
## Running it
|
||||
|
||||
`llmgw` is the mirror. Same binary as the control plane and the agents, one
|
||||
subcommand:
|
||||
|
||||
```
|
||||
sudo mkdir -p /srv/models && sudo chown llmgw:llmgw /srv/models
|
||||
llmgw mirror serve --dir /srv/models --base http://models.internal:8081
|
||||
```
|
||||
|
||||
There is a unit in `deploy/systemd/llmgw-mirror.service`. Set `--base` to the
|
||||
address nodes reach this machine on — not `localhost`, and not a name only this
|
||||
host knows itself by. It is the string that ends up in catalog entries, and a
|
||||
wrong one makes `mirror unused` report files as unreferenced when they are in
|
||||
use. The next step after that list is deletion.
|
||||
|
||||
### Two listeners
|
||||
|
||||
`--listen :8081` serves the weights. Read-only, unauthenticated, Range requests
|
||||
supported so an interrupted fetch on a node resumes rather than restarts. This
|
||||
is the path eight nodes pull forty gigabytes from.
|
||||
|
||||
`--api-listen :8082` is the management API. Separate on purpose: putting a
|
||||
client certificate in front of the byte path would mean re-certifying every
|
||||
agent's fetcher in order to change how models are administered.
|
||||
|
||||
The API is unauthenticated unless you give it the fleet's CA, and it says so at
|
||||
startup rather than leaving you to find out. Anyone who can reach that port can
|
||||
start downloads and delete models, so either keep it on a trusted network or
|
||||
hand it certificates:
|
||||
|
||||
```
|
||||
llmgw mirror serve --dir /srv/models --base http://models.internal:8081 \
|
||||
--tls-ca /etc/llmgw/ca.pem \
|
||||
--tls-cert /etc/llmgw/mirror.pem \
|
||||
--tls-key /etc/llmgw/mirror-key.pem
|
||||
```
|
||||
|
||||
That is the same CA the agents already use, so the certificates come from the
|
||||
authority you have:
|
||||
|
||||
```
|
||||
llmgw ctl cert issue-server mirror --host models.internal --host 10.0.0.146
|
||||
llmgw ctl cert issue-client control-plane
|
||||
```
|
||||
|
||||
`--host` has to cover every address the API is called on, because that is what
|
||||
a client verifies against. Both sides are checked — the mirror will not accept a
|
||||
caller without a certificate from this CA, the same rule the agent stream
|
||||
follows.
|
||||
|
||||
### nginx instead
|
||||
|
||||
If you already run nginx, it serves a directory perfectly well and you do not
|
||||
need `mirror serve` for the bytes:
|
||||
|
||||
```nginx
|
||||
server {
|
||||
@@ -57,15 +111,81 @@ need it.
|
||||
|
||||
## Filling it
|
||||
|
||||
From the mirror itself:
|
||||
|
||||
```
|
||||
# On the mirror, once per model and version:
|
||||
mkdir -p /srv/models/qwen2.5-14b-instruct/2026-03
|
||||
curl -fL -o /srv/models/qwen2.5-14b-instruct/2026-03/q4_k_m.gguf \
|
||||
"https://huggingface.co/<repo>/resolve/<revision>/<file>"
|
||||
llmgw mirror add \
|
||||
--model qwen2.5-14b-instruct --version 2026-03 \
|
||||
"https://huggingface.co/<repo>/resolve/<revision>/q4_k_m.gguf"
|
||||
```
|
||||
|
||||
Use a revision rather than `main` in the source URL. `main` is what moves.
|
||||
|
||||
It downloads to a temporary name and renames when complete, so an interrupted
|
||||
download never leaves something a catalog could be pointed at, and it prints the
|
||||
sha256 when it lands. Compare that against what upstream publishes — Hugging
|
||||
Face gives it as `lfs.oid` from
|
||||
`https://huggingface.co/api/models/<repo>/tree/<revision>`. A mirror is the one
|
||||
place in this system where a corrupt file is copied to every node instead of
|
||||
being caught once.
|
||||
|
||||
It refuses to overwrite a version that is already there. Add a new one beside
|
||||
it: that refusal is what makes the old weights still available when the new ones
|
||||
turn out to be worse.
|
||||
|
||||
## Driving it from elsewhere
|
||||
|
||||
The same fetch over the API, so the control plane — or anything else — can ask
|
||||
for a model without anyone logging in:
|
||||
|
||||
```
|
||||
curl -X POST http://models.internal:8082/v1/models -d '{
|
||||
"source": "https://huggingface.co/<repo>/resolve/<revision>/q4_k_m.gguf",
|
||||
"model": "qwen2.5-14b-instruct",
|
||||
"version": "2026-03"
|
||||
}'
|
||||
```
|
||||
|
||||
The mirror does the downloading. A control plane that pulled the bytes itself
|
||||
and pushed them here would carry every model twice over the network, through the
|
||||
one machine least able to spare the bandwidth.
|
||||
|
||||
It answers `202` with the URL a catalog entry should use, and the bytes are not
|
||||
here yet. Poll for them:
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| `GET /v1/models` | everything held or arriving, and the base to build URLs from |
|
||||
| `POST /v1/models` | fetch this source into model/version |
|
||||
| `GET /v1/models/{model}/{version}` | `fetching` with a byte count, `present`, or `failed` with the reason |
|
||||
| `DELETE /v1/models/{model}/{version}` | remove a version; refuses while it is downloading |
|
||||
| `GET /v1/health` | up *and* able to read its own directory |
|
||||
|
||||
A failed fetch stays `failed` with what went wrong, rather than disappearing:
|
||||
otherwise the thing waiting on that model waits forever with nothing to say why.
|
||||
Asking twice for the same download joins the one already running instead of
|
||||
starting a rival writer on the same file.
|
||||
|
||||
`DELETE` takes the whole version, never one file of it. A model split across
|
||||
several files is useless with one missing, and half a version on disk looks
|
||||
present and will not load.
|
||||
|
||||
## Retiring a model
|
||||
|
||||
```
|
||||
llmgw mirror unused # what no catalog entry refers to
|
||||
llmgw mirror remove --model <m> --version <v> --file <f>
|
||||
```
|
||||
|
||||
`unused` compares against every `source` in the running catalog, so the answer
|
||||
is what can go rather than what looks old. It refuses to answer when it cannot
|
||||
reach the control plane: an empty catalog and an unreachable one produce the
|
||||
same list — everything — and the obvious next step after that list is deletion.
|
||||
|
||||
Retire the catalog entry first, let the fleet settle, then remove. A node that
|
||||
has to refetch a file you deleted cannot, and the failure arrives later, on the
|
||||
node, looking like something else.
|
||||
|
||||
## Pointing llmgw at it
|
||||
|
||||
```yaml
|
||||
@@ -86,11 +206,11 @@ model's name, so an interrupted fetch cannot be handed to an engine.
|
||||
|
||||
## What this is not
|
||||
|
||||
It is not a cache: nothing populates it automatically, and a model you did not
|
||||
put there is not there. That is on purpose. A mirror that fetches on demand is
|
||||
a mirror that reaches the internet at the least convenient moment, and in a
|
||||
network where that is blocked it fails in a way that looks like llmgw being
|
||||
broken.
|
||||
It is not a cache. Nothing populates it on its own: a model is there because
|
||||
somebody asked for it, by hand or over the API, and one nobody asked for is not
|
||||
there. That is on purpose. A mirror that fetched on demand would reach the
|
||||
internet at the least convenient moment, and in a network where that is blocked
|
||||
it would fail in a way that looks like llmgw being broken.
|
||||
|
||||
It is also not a replacement for a shared filesystem. If you already run one
|
||||
that every node mounts, put the models on it and use a path as the `source` —
|
||||
|
||||
@@ -0,0 +1,29 @@
|
||||
[Unit]
|
||||
Description=llmgw model mirror
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
# --base is what a catalog entry will say, so it has to be the address nodes
|
||||
# reach this machine on — not localhost, and not the name only this host knows
|
||||
# itself by. Getting it wrong makes `mirror unused` report files as unreferenced
|
||||
# when they are in use, and the next step after that list is deletion.
|
||||
ExecStart=/usr/local/bin/llmgw mirror serve \
|
||||
--dir /srv/models \
|
||||
--listen :8081 \
|
||||
--api-listen :8082 \
|
||||
--base http://models.internal:8081
|
||||
User=llmgw
|
||||
Group=llmgw
|
||||
Restart=on-failure
|
||||
RestartSec=2s
|
||||
NoNewPrivileges=true
|
||||
ProtectSystem=strict
|
||||
ProtectHome=true
|
||||
PrivateTmp=true
|
||||
# The one writable path. The mirror serves reads to the whole fleet and writes
|
||||
# only what its own API was asked to fetch, so nothing else needs to be.
|
||||
ReadWritePaths=/srv/models
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
Reference in new issue
Block a user