llmgw/deploy/mirror/README.md

# A model mirror

One machine that holds the model files, so nodes fetch over your own network
instead of over the internet.

This is the highest-value piece of infrastructure around llmgw and the cheapest.
Eight nodes each pulling forty gigabytes from Hugging Face is a day's work, and
on plenty of corporate networks it is not possible at all. From a mirror on the
same switch it is minutes.

llmgw needs nothing special for this: a model's `source` is a URL, and the URL
can be yours.

## What it is

An HTTP server serving a directory. That is the whole design.

```
/srv/models/
  qwen2.5-14b-instruct/
    2026-03/
      q4_k_m.gguf
  bge-m3/
    2026-01/
      model.safetensors
```

At an offline site the mirror also holds what the nodes would otherwise fetch
from the internet — llmgw's releases and the llama.cpp builds — under
`_llmgw/releases/` and `_llmgw/engines/`, put there by `llmgw bundle import`
(docs/install.md, "Offline sites"). They are not models: `mirror list` and
`mirror unused` pass over them, and a node checks each against what the
control plane says it must be, not against anything the mirror says.

The date in the path is the point of the layout, not decoration. A model
repository that changes upstream changes what you are running, and the first
sign is a prompt that worked last week answering differently this week with
nobody able to say why. Pin it here and that cannot happen.

## Running it

`llmgw` is the mirror. Same binary as the control plane and the agents, one
subcommand:

```
sudo mkdir -p /srv/models && sudo chown llmgw:llmgw /srv/models
llmgw mirror serve --dir /srv/models --base http://models.internal:8081
```

There is a unit in `deploy/systemd/llmgw-mirror.service`. Set `--base` to the
address nodes reach this machine on — not `localhost`, and not a name only this
host knows itself by. It is the string that ends up in catalog entries, and a
wrong one makes `mirror unused` report files as unreferenced when they are in
use. The next step after that list is deletion.

### Two listeners

`--listen :8081` serves the weights. Read-only, unauthenticated, Range requests
supported so an interrupted fetch on a node resumes rather than restarts. This
is the path eight nodes pull forty gigabytes from.

`--api-listen :8082` is the management API. Separate on purpose: putting a
client certificate in front of the byte path would mean re-certifying every
agent's fetcher in order to change how models are administered.

The API is unauthenticated unless you give it the fleet's CA, and it says so at
startup rather than leaving you to find out. Anyone who can reach that port can
start downloads and delete models, so either keep it on a trusted network or
hand it certificates:

```
llmgw mirror serve --dir /srv/models --base http://models.internal:8081 \
  --tls-ca /etc/llmgw/ca.pem \
  --tls-cert /etc/llmgw/mirror.pem \
  --tls-key /etc/llmgw/mirror-key.pem \
  --api-client control-plane
```

`--api-client` is required with `--tls-ca`: it names the one certificate the
API answers, because the CA signs every node's certificate as well.

That is the same CA the agents already use, so the certificates come from the
authority you have:

```
llmgw ctl cert issue-server mirror --host models.internal --host 10.0.0.146
llmgw ctl cert issue-client control-plane
```

`--host` has to cover every address the API is called on, because that is what
a client verifies against. Both sides are checked — the mirror will not accept a
caller without a certificate from this CA, the same rule the agent stream
follows.

### nginx instead

If you already run nginx, it serves a directory perfectly well and you do not
need `mirror serve` for the bytes:

```nginx
server {
    listen 80;
    server_name models.internal;

    root /srv/models;

    # Read-only. This is deliberate: a mirror that can be written to over HTTP
    # is a way to replace a model with another one, and the nodes would fetch
    # it without complaint.
    location / {
        autoindex on;
        # Large files, and the client is a machine with a resumable fetcher.
        # Sendfile and no buffering keep memory flat regardless of size.
        sendfile on;
        tcp_nopush on;
    }
}
```

`autoindex` is there so a person can browse what is available; llmgw does not
need it.

## Filling it

From the mirror itself:

```
llmgw mirror add \
  --model qwen2.5-14b-instruct --version 2026-03 \
  "https://huggingface.co/<repo>/resolve/<revision>/q4_k_m.gguf"
```

Use a revision rather than `main` in the source URL. `main` is what moves.

It downloads to a temporary name and renames when complete, so an interrupted
download never leaves something a catalog could be pointed at, and it prints the
sha256 when it lands. Compare that against what upstream publishes — Hugging
Face gives it as `lfs.oid` from
`https://huggingface.co/api/models/<repo>/tree/<revision>`. A mirror is the one
place in this system where a corrupt file is copied to every node instead of
being caught once.

It refuses to overwrite a version that is already there. Add a new one beside
it: that refusal is what makes the old weights still available when the new ones
turn out to be worse.

## Driving it from elsewhere

The same fetch over the API, so the control plane — or anything else — can ask
for a model without anyone logging in:

```
curl -X POST http://models.internal:8082/v1/models -d '{
  "source": "https://huggingface.co/<repo>/resolve/<revision>/q4_k_m.gguf",
  "model": "qwen2.5-14b-instruct",
  "version": "2026-03"
}'
```

The mirror does the downloading. A control plane that pulled the bytes itself
and pushed them here would carry every model twice over the network, through the
one machine least able to spare the bandwidth.

It answers `202` with the URL a catalog entry should use, and the bytes are not
here yet. Poll for them:

| | |
|---|---|
| `GET /v1/models` | everything held or arriving, and the base to build URLs from |
| `POST /v1/models` | fetch this source into model/version |
| `GET /v1/models/{model}/{version}` | `fetching` with a byte count, `present`, or `failed` with the reason |
| `DELETE /v1/models/{model}/{version}` | remove a version; refuses while it is downloading |
| `GET /v1/health` | up *and* able to read its own directory |

A failed fetch stays `failed` with what went wrong, rather than disappearing:
otherwise the thing waiting on that model waits forever with nothing to say why.
Asking twice for the same download joins the one already running instead of
starting a rival writer on the same file.

`DELETE` takes the whole version, never one file of it. A model split across
several files is useless with one missing, and half a version on disk looks
present and will not load.

## Retiring a model

```
llmgw mirror unused     # what no catalog entry refers to
llmgw mirror remove --model <m> --version <v> --file <f>
```

`unused` compares against every `source` in the running catalog, so the answer
is what can go rather than what looks old. It refuses to answer when it cannot
reach the control plane: an empty catalog and an unreachable one produce the
same list — everything — and the obvious next step after that list is deletion.

Retire the catalog entry first, let the fleet settle, then remove. A node that
has to refetch a file you deleted cannot, and the failure arrives later, on the
node, looking like something else.

## From the control plane's admin page

Configure the mirror on the control plane and the same operations appear at
`/admin/mirror`, behind the admin session the other admin pages use:

```yaml
mirror:
  api: https://models.internal:8082
  tls:
    ca: /etc/llmgw/ca.crt
    cert: /etc/llmgw/control-plane.crt
    key: /etc/llmgw/control-plane.key
```

Leave the block out and the page is not mounted at all, which is the right page
for a deployment with no mirror — not an empty one pointing at nothing.

The page lists what is held, what is arriving with its byte count, and what
failed with the reason; it starts a fetch, retries a failed one, and removes a
version. It reloads itself only while something is downloading, because a reload
at any other moment takes with it whatever is half-typed in the form.

Two things it deliberately will not do. It cannot point the control plane at a
different mirror — that is configuration, and changing it needs access to the
host rather than a session. And it does not write the catalog for you: it shows
the URL an entry should use, and putting it there is still a deliberate act,
because that is the moment a model starts being served.

The address is a mistake worth naming: `api` is the **management** port, not the
one nodes fetch bytes from. They are two listeners so this one can be locked
down without re-certifying every agent's fetcher.

## Pointing llmgw at it

```yaml
- name: qwen2.5-14b-instruct
  source: http://models.internal/qwen2.5-14b-instruct/2026-03/q4_k_m.gguf
  scaling: { min: 2, max: 2 }
  profiles:
    - name: gpu
      engine: vllm
      gpu_class: h100-80g
      tp: 1
```

Each node fetches once, in the background, before its engine starts — and
`llmgw ctl replicas` shows how far along it is while it does. A killed download
resumes rather than restarting, and a partial file is never visible under the
model's name, so an interrupted fetch cannot be handed to an engine.

## What this is not

It is not a cache. Nothing populates it on its own: a model is there because
somebody asked for it, by hand or over the API, and one nobody asked for is not
there. That is on purpose. A mirror that fetched on demand would reach the
internet at the least convenient moment, and in a network where that is blocked
it would fail in a way that looks like llmgw being broken.

It is also not a replacement for a shared filesystem. If you already run one
that every node mounts, put the models on it and use a path as the `source` —
nothing is fetched at all, and llmgw notices the filesystem is shared and will
not count the same free space once per node.

## When this stops being enough

Somewhere above ten or twenty nodes, one mirror serving a forty gigabyte file
to all of them at once becomes the bottleneck instead of the fix. At that point
the options are a filer, a second mirror per rack, or distribution between the
nodes themselves.

llmgw knows which nodes already hold which weights — it is in `ctl status` as
"N models on X/Y nodes" — so distributing between them is a feature it is
positioned to add. It has not been built, because a mirror is enough for
everyone who has needed it so far.