llmgw/deploy/mirror/README.md

A model mirror

One machine that holds the model files, so nodes fetch over your own network instead of over the internet.

This is the highest-value piece of infrastructure around llmgw and the cheapest. Eight nodes each pulling forty gigabytes from Hugging Face is a day's work, and on plenty of corporate networks it is not possible at all. From a mirror on the same switch it is minutes.

llmgw needs nothing special for this: a model's source is a URL, and the URL can be yours.

What it is

An HTTP server serving a directory. That is the whole design.

/srv/models/
  qwen2.5-14b-instruct/
    2026-03/
      q4_k_m.gguf
  bge-m3/
    2026-01/
      model.safetensors

The date in the path is the point of the layout, not decoration. A model repository that changes upstream changes what you are running, and the first sign is a prompt that worked last week answering differently this week with nobody able to say why. Pin it here and that cannot happen.

Running it

llmgw is the mirror. Same binary as the control plane and the agents, one subcommand:

sudo mkdir -p /srv/models && sudo chown llmgw:llmgw /srv/models
llmgw mirror serve --dir /srv/models --base http://models.internal:8081

There is a unit in deploy/systemd/llmgw-mirror.service. Set --base to the address nodes reach this machine on — not localhost, and not a name only this host knows itself by. It is the string that ends up in catalog entries, and a wrong one makes mirror unused report files as unreferenced when they are in use. The next step after that list is deletion.

Two listeners

--listen :8081 serves the weights. Read-only, unauthenticated, Range requests supported so an interrupted fetch on a node resumes rather than restarts. This is the path eight nodes pull forty gigabytes from.

--api-listen :8082 is the management API. Separate on purpose: putting a client certificate in front of the byte path would mean re-certifying every agent's fetcher in order to change how models are administered.

The API is unauthenticated unless you give it the fleet's CA, and it says so at startup rather than leaving you to find out. Anyone who can reach that port can start downloads and delete models, so either keep it on a trusted network or hand it certificates:

llmgw mirror serve --dir /srv/models --base http://models.internal:8081 \
  --tls-ca /etc/llmgw/ca.pem \
  --tls-cert /etc/llmgw/mirror.pem \
  --tls-key /etc/llmgw/mirror-key.pem \
  --api-client control-plane

--api-client is required with --tls-ca: it names the one certificate the API answers, because the CA signs every node's certificate as well.

That is the same CA the agents already use, so the certificates come from the authority you have:

llmgw ctl cert issue-server mirror --host models.internal --host 10.0.0.146
llmgw ctl cert issue-client control-plane

--host has to cover every address the API is called on, because that is what a client verifies against. Both sides are checked — the mirror will not accept a caller without a certificate from this CA, the same rule the agent stream follows.

nginx instead

If you already run nginx, it serves a directory perfectly well and you do not need mirror serve for the bytes:

server {
    listen 80;
    server_name models.internal;

    root /srv/models;

    # Read-only. This is deliberate: a mirror that can be written to over HTTP
    # is a way to replace a model with another one, and the nodes would fetch
    # it without complaint.
    location / {
        autoindex on;
        # Large files, and the client is a machine with a resumable fetcher.
        # Sendfile and no buffering keep memory flat regardless of size.
        sendfile on;
        tcp_nopush on;
    }
}

autoindex is there so a person can browse what is available; llmgw does not need it.

Filling it

From the mirror itself:

llmgw mirror add \
  --model qwen2.5-14b-instruct --version 2026-03 \
  "https://huggingface.co/<repo>/resolve/<revision>/q4_k_m.gguf"

Use a revision rather than main in the source URL. main is what moves.

It downloads to a temporary name and renames when complete, so an interrupted download never leaves something a catalog could be pointed at, and it prints the sha256 when it lands. Compare that against what upstream publishes — Hugging Face gives it as lfs.oid from https://huggingface.co/api/models/<repo>/tree/<revision>. A mirror is the one place in this system where a corrupt file is copied to every node instead of being caught once.

It refuses to overwrite a version that is already there. Add a new one beside it: that refusal is what makes the old weights still available when the new ones turn out to be worse.

Driving it from elsewhere

The same fetch over the API, so the control plane — or anything else — can ask for a model without anyone logging in:

curl -X POST http://models.internal:8082/v1/models -d '{
  "source": "https://huggingface.co/<repo>/resolve/<revision>/q4_k_m.gguf",
  "model": "qwen2.5-14b-instruct",
  "version": "2026-03"
}'

The mirror does the downloading. A control plane that pulled the bytes itself and pushed them here would carry every model twice over the network, through the one machine least able to spare the bandwidth.

It answers 202 with the URL a catalog entry should use, and the bytes are not here yet. Poll for them:

GET /v1/models	everything held or arriving, and the base to build URLs from
POST /v1/models	fetch this source into model/version
GET /v1/models/{model}/{version}	fetching with a byte count, present, or failed with the reason
DELETE /v1/models/{model}/{version}	remove a version; refuses while it is downloading
GET /v1/health	up and able to read its own directory

A failed fetch stays failed with what went wrong, rather than disappearing: otherwise the thing waiting on that model waits forever with nothing to say why. Asking twice for the same download joins the one already running instead of starting a rival writer on the same file.

DELETE takes the whole version, never one file of it. A model split across several files is useless with one missing, and half a version on disk looks present and will not load.

Retiring a model

llmgw mirror unused     # what no catalog entry refers to
llmgw mirror remove --model <m> --version <v> --file <f>

unused compares against every source in the running catalog, so the answer is what can go rather than what looks old. It refuses to answer when it cannot reach the control plane: an empty catalog and an unreachable one produce the same list — everything — and the obvious next step after that list is deletion.

Retire the catalog entry first, let the fleet settle, then remove. A node that has to refetch a file you deleted cannot, and the failure arrives later, on the node, looking like something else.

From the control plane's admin page

Configure the mirror on the control plane and the same operations appear at /admin/mirror, behind the admin session the other admin pages use:

mirror:
  api: https://models.internal:8082
  tls:
    ca: /etc/llmgw/ca.crt
    cert: /etc/llmgw/control-plane.crt
    key: /etc/llmgw/control-plane.key

Leave the block out and the page is not mounted at all, which is the right page for a deployment with no mirror — not an empty one pointing at nothing.

The page lists what is held, what is arriving with its byte count, and what failed with the reason; it starts a fetch, retries a failed one, and removes a version. It reloads itself only while something is downloading, because a reload at any other moment takes with it whatever is half-typed in the form.

Two things it deliberately will not do. It cannot point the control plane at a different mirror — that is configuration, and changing it needs access to the host rather than a session. And it does not write the catalog for you: it shows the URL an entry should use, and putting it there is still a deliberate act, because that is the moment a model starts being served.

The address is a mistake worth naming: api is the management port, not the one nodes fetch bytes from. They are two listeners so this one can be locked down without re-certifying every agent's fetcher.

Pointing llmgw at it

- name: qwen2.5-14b-instruct
  source: http://models.internal/qwen2.5-14b-instruct/2026-03/q4_k_m.gguf
  scaling: { min: 2, max: 2 }
  profiles:
    - name: gpu
      engine: vllm
      gpu_class: h100-80g
      tp: 1

Each node fetches once, in the background, before its engine starts — and llmgw ctl replicas shows how far along it is while it does. A killed download resumes rather than restarting, and a partial file is never visible under the model's name, so an interrupted fetch cannot be handed to an engine.

What this is not

It is not a cache. Nothing populates it on its own: a model is there because somebody asked for it, by hand or over the API, and one nobody asked for is not there. That is on purpose. A mirror that fetched on demand would reach the internet at the least convenient moment, and in a network where that is blocked it would fail in a way that looks like llmgw being broken.

It is also not a replacement for a shared filesystem. If you already run one that every node mounts, put the models on it and use a path as the source — nothing is fetched at all, and llmgw notices the filesystem is shared and will not count the same free space once per node.

When this stops being enough

Somewhere above ten or twenty nodes, one mirror serving a forty gigabyte file to all of them at once becomes the bottleneck instead of the fix. At that point the options are a filer, a second mirror per rack, or distribution between the nodes themselves.

llmgw knows which nodes already hold which weights — it is in ctl status as "N models on X/Y nodes" — so distributing between them is a feature it is positioned to add. It has not been built, because a mirror is enough for everyone who has needed it so far.